2020-11-20T11:42:47.899 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-20T11:42:47.901 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} --archive /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640308 --block --verbose --name pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi --lock --owner scheduled_pdonnell@teuthology -- /tmp/teuthology-worker.0462mmsx.tmp 2020-11-20T11:42:47.946 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:42:48.026 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640308 branch: wip-pdonnell-testing-20201120.053228 description: fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} email: pdonnell@redhat.com first_in_suite: false job_id: '5640308' kernel: client: branch: testing kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-pdonnell-testing-20201120.053228 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 15m rados osd op timeout: 15m global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20201120.053228 sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb owner: scheduled_pdonnell@teuthology priority: 95 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.1 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - client.2 - client.3 sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb sleep_before_teardown: 0 suite: fs suite_branch: wip-pdonnell-testing-20201120.053228 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology_branch: master timestamp: 2020-11-20_08:39:42 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.6030 2020-11-20T11:42:48.027 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa; will attempt to use it 2020-11-20T11:42:48.027 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa/tasks 2020-11-20T11:42:48.028 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-20T11:42:48.030 INFO:teuthology.task.internal:Checking packages... 2020-11-20T11:42:48.050 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'b86e691bf191566922a8f08e2a5abf64fde8cfcb' 2020-11-20T11:42:48.051 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-20T11:42:48.051 INFO:teuthology.packaging:ref: None 2020-11-20T11:42:48.052 INFO:teuthology.packaging:tag: None 2020-11-20T11:42:48.052 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20201120.053228 2020-11-20T11:42:48.052 INFO:teuthology.packaging:sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:42:48.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20201120.053228 2020-11-20T11:42:48.213 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7397.gb86e691b 2020-11-20T11:42:48.214 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-20T11:42:48.231 INFO:teuthology.task.internal:no buildpackages task found 2020-11-20T11:42:48.231 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-20T11:42:48.266 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-20T11:42:48.267 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:42:48.326 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:42:58.341 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:43:08.356 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:43:18.374 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:43:28.395 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:43:38.411 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:43:48.431 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:43:58.477 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:08.488 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:18.507 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:28.520 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:38.536 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:48.556 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:58.580 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:08.597 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:18.619 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:28.632 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:38.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:48.670 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:58.690 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:46:08.701 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:46:18.710 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:46:28.734 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:46:38.751 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:46:48.776 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:46:58.794 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:47:08.812 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:47:18.829 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:47:28.848 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:47:38.862 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:47:48.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:47:58.898 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:08.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:18.940 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:28.957 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:38.973 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640308'} 2020-11-20T11:48:38.993 DEBUG:teuthology.lock.ops:locked ubuntu@smithi187.front.sepia.ceph.com, ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T11:48:39.046 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe32611eb70>, signals=[15]) 2020-11-20T11:48:39.243 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of centos 8.1 2020-11-20T11:48:39.246 INFO:teuthology.provision.fog.smithi005:Scheduling deploy of centos 8.1 2020-11-20T11:48:39.598 INFO:teuthology.orchestra.console:Power off smithi187 2020-11-20T11:48:39.598 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-20T11:48:39.615 INFO:teuthology.orchestra.console:Power off smithi005 2020-11-20T11:48:39.615 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-20T11:48:39.634 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:39.643 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:43.659 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:43.669 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:47.773 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:47.783 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:47.995 INFO:teuthology.orchestra.console:Power off for smithi005 completed 2020-11-20T11:48:48.097 INFO:teuthology.orchestra.console:Power on smithi005 2020-11-20T11:48:48.097 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-20T11:48:48.117 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:51.885 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:52.107 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2020-11-20T11:48:52.145 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:52.209 INFO:teuthology.orchestra.console:Power on smithi187 2020-11-20T11:48:52.210 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-20T11:48:52.249 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:56.265 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:56.295 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:48:56.476 INFO:teuthology.orchestra.console:Power on for smithi005 completed 2020-11-20T11:48:56.578 INFO:teuthology.provision.fog.smithi005:Waiting for deploy to finish 2020-11-20T11:49:00.405 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:49:00.615 INFO:teuthology.orchestra.console:Power on for smithi187 completed 2020-11-20T11:49:00.717 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2020-11-20T11:51:12.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:21.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:30.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:31.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:40.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:41.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:49.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:50.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:58.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:51:59.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:52:08.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:52:17.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:52:22.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:52:26.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:52:31.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:52:33.834 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-20T11:52:33.836 INFO:teuthology.orchestra.run.smithi005:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-20T11:52:35.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:52:49.454 INFO:teuthology.provision.fog.smithi005:Node is ready 2020-11-20T11:52:49.455 INFO:teuthology.orchestra.run.smithi005:> hostname 2020-11-20T11:52:49.475 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005.front.sepia.ceph.com 2020-11-20T11:52:49.476 INFO:teuthology.orchestra.run.smithi005:> grep smithi005.front.sepia.ceph.com /etc/hosts 2020-11-20T11:52:49.532 INFO:teuthology.orchestra.run.smithi005.stdout:172.21.15.5 smithi005.front.sepia.ceph.com smithi005 2020-11-20T11:52:49.533 INFO:teuthology.orchestra.run.smithi005:> sudo hostname smithi005 2020-11-20T11:52:49.686 INFO:teuthology.orchestra.run.smithi005:> sudo sed -i -e 's/smithi005.front.sepia.ceph.com/smithi005/g' /etc/hosts 2020-11-20T11:52:49.722 INFO:teuthology.orchestra.run.smithi005:> sudo sed -i -e 's/172.21.15.5/172.21.15.5/g' /etc/hosts 2020-11-20T11:52:49.796 INFO:teuthology.orchestra.run.smithi005:> sudo sed -i -e 's/smithi005.front.sepia.ceph.com/smithi005/g' /etc/hostname 2020-11-20T11:52:49.869 INFO:teuthology.orchestra.run.smithi005:> sudo hostname smithi005 2020-11-20T11:52:49.940 INFO:teuthology.provision.fog.smithi005:Deploy complete! 2020-11-20T11:52:49.941 DEBUG:teuthology.parallel:result is None 2020-11-20T11:53:07.109 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-20T11:53:07.596 INFO:teuthology.orchestra.run.smithi187:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-20T11:53:27.922 INFO:teuthology.provision.fog.smithi187:Node is ready 2020-11-20T11:53:27.922 INFO:teuthology.orchestra.run.smithi187:> hostname 2020-11-20T11:53:27.942 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2020-11-20T11:53:27.943 INFO:teuthology.orchestra.run.smithi187:> grep smithi187.front.sepia.ceph.com /etc/hosts 2020-11-20T11:53:28.003 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2020-11-20T11:53:28.003 INFO:teuthology.orchestra.run.smithi187:> sudo hostname smithi187 2020-11-20T11:53:28.147 INFO:teuthology.orchestra.run.smithi187:> sudo sed -i -e 's/smithi187.front.sepia.ceph.com/smithi187/g' /etc/hosts 2020-11-20T11:53:28.182 INFO:teuthology.orchestra.run.smithi187:> sudo sed -i -e 's/172.21.15.187/172.21.15.187/g' /etc/hosts 2020-11-20T11:53:28.256 INFO:teuthology.orchestra.run.smithi187:> sudo sed -i -e 's/smithi187.front.sepia.ceph.com/smithi187/g' /etc/hostname 2020-11-20T11:53:28.329 INFO:teuthology.orchestra.run.smithi187:> sudo hostname smithi187 2020-11-20T11:53:28.399 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2020-11-20T11:53:28.399 DEBUG:teuthology.parallel:result is None 2020-11-20T11:53:28.733 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com 2020-11-20T11:53:28.734 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-20T11:53:28.760 INFO:teuthology.lock.ops:Checking smithi005.front.sepia.ceph.com 2020-11-20T11:53:28.761 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-20T11:53:28.785 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server 2020-11-20T11:53:28.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:53:28.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:53:28.911 INFO:teuthology.orchestra.run.smithi187:> true 2020-11-20T11:53:29.077 INFO:teuthology.orchestra.run.smithi187:> uname -m 2020-11-20T11:53:29.145 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2020-11-20T11:53:29.145 INFO:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2020-11-20T11:53:29.201 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Linux" 2020-11-20T11:53:29.203 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8 (Core)" 2020-11-20T11:53:29.203 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2020-11-20T11:53:29.203 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2020-11-20T11:53:29.204 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8" 2020-11-20T11:53:29.204 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2020-11-20T11:53:29.204 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-20T11:53:29.204 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2020-11-20T11:53:29.204 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-20T11:53:29.205 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.centos.org/" 2020-11-20T11:53:29.205 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-20T11:53:29.205 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T11:53:29.205 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-20T11:53:29.206 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-20T11:53:29.206 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-20T11:53:29.206 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-20T11:53:29.206 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T11:53:29.207 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2020-11-20T11:53:29.230 INFO:teuthology.lock.ops:Updating [smithi005.front.sepia.ceph.com]: set os type and version on server 2020-11-20T11:53:29.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:53:29.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:53:29.304 INFO:teuthology.orchestra.run.smithi005:> true 2020-11-20T11:53:29.488 INFO:teuthology.orchestra.run.smithi005:> uname -m 2020-11-20T11:53:29.548 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2020-11-20T11:53:29.549 INFO:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2020-11-20T11:53:29.607 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Linux" 2020-11-20T11:53:29.607 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="8 (Core)" 2020-11-20T11:53:29.608 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2020-11-20T11:53:29.608 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2020-11-20T11:53:29.608 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="8" 2020-11-20T11:53:29.608 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el8" 2020-11-20T11:53:29.608 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-20T11:53:29.609 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2020-11-20T11:53:29.609 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-20T11:53:29.609 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.centos.org/" 2020-11-20T11:53:29.609 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-20T11:53:29.609 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T11:53:29.610 INFO:teuthology.orchestra.run.smithi005.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-20T11:53:29.610 INFO:teuthology.orchestra.run.smithi005.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-20T11:53:29.610 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-20T11:53:29.610 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-20T11:53:29.610 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T11:53:29.611 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2020-11-20T11:53:29.630 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-20T11:53:29.668 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT6/H5StJqG+ZPdwMkhIuuh6om1MQcqs5XXVo+WKx5Cf9duUWjGUj8WWAfp/p0/iLH9FTfRZSYFDrzTevV6HLFNm3D57+RSHEnQSL7R3zWOsDvtG7teZ6O9d/kzF3ZdBNbqOsN020st1bIfgnPiz0awjJdC10s0LuTCKV9r4tRbHIbdxmekhA9GqjzUH9hByV2cdWkqbkqjnDlP5Tn8s13JRwA/+mVfnMUvoMslFOb+FMeZNQKKCfCnLLh70ZM6ZGiRtILlZAols5hqvCuDA7JYR7q2joUCIJL681c4nMx5CrHpB6peVw7b7r6OL4ZYP0j8YAL8ovIPoguvTBown7uL6XUgL+NkeQZV6l0fIT/coOMvIujx1P2N5kqYpZ5aYh+OYm68Ti3N6aklrjZnfmSxS8aBRNKdJUvRWwu8P0bztgWrX0b81QVT8FPBPQw1DmNBeGWY9ViKgb/RFY1kc81ENtZGkjPFF0dNvySM3SrooXJwkv1CroWvs72DhTRJks= smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmEJkuo87iHMn7jODntPiOZNQeC8JxqLEsXf5Kfr7f6P7wo+BNVAX9coacFXpI/cVZm/PrSN3FI++JGh6KkubLfxAfS289nNRM653HD6JmGu9FbGy0pCH2Ekd4Ff3RT6ViYWUs/dl2GXe04muM0zTGoAcz82C5FmQzGQgzN12esoHqF7UXR9qjb361GDe1lrvTY+M/U8V8+LcRuDsYUne8Q4DwNHJ+uNnCJmMzTdWMQyo+jclbb1Ctg//XhWA19yaZRA9wbdSjncfcPPi3APfk42BG2ANSQ1u98wPtlmM/KuSKrUvB2lIhllnpoBuNjJH69cxAFQ4tLaRQgHdvw188AzfGgQIHoXFUMOCKXQa1QPIaJU5uWY6TdKMLLyoLGJ941mAvFwac+SeQYi1qyX0RLtWBHELqKcZpT/GiSndwTgp5195d3xzoOsQOhGM76nnWWPfX8M8T6rw1P+yRK0zTtrJ3mYVTeA6qjC3LGD4Sobef9Fb+tfGxqbelB8r1Mh8= 2020-11-20T11:53:29.669 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:53:29.726 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-20T11:53:29.731 INFO:teuthology.task.internal:Saving configuration 2020-11-20T11:53:29.750 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-20T11:53:29.769 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-20T11:53:29.787 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-20 11:48:38.978738', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640308', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOD7LULfuPzxi/lt6v9/oVAytDrn+CCcwcHYSeRK72un/5f1JBBU92rWAp6bHKdYlZWp1xe9dq+OsnCa/f0jXG4=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi187.front.sepia.ceph.com'} 2020-11-20T11:53:29.812 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-20 11:48:38.978758', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640308', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNo6X09ebrDZ0Ldb2GutP/KSHekeEL7idj6h7al5exFv6gD1ZEs7MKzqJmL8rCqvwlVzXtzctjGGDzz+wrRWHhc=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi005.front.sepia.ceph.com'} 2020-11-20T11:53:29.813 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-20T11:53:29.842 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'] 2020-11-20T11:53:29.843 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3'] 2020-11-20T11:53:29.843 INFO:teuthology.run_tasks:Running task console_log... 2020-11-20T11:53:29.966 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe326163bf8>, signals=[15]) 2020-11-20T11:53:29.966 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-20T11:53:29.989 INFO:teuthology.task.internal:Opening connections... 2020-11-20T11:53:29.989 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T11:53:29.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:53:30.063 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T11:53:30.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:53:30.138 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-20T11:53:30.146 INFO:teuthology.orchestra.run.smithi187:> uname -m 2020-11-20T11:53:30.183 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2020-11-20T11:53:30.183 INFO:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2020-11-20T11:53:30.236 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Linux" 2020-11-20T11:53:30.236 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8 (Core)" 2020-11-20T11:53:30.237 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2020-11-20T11:53:30.237 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2020-11-20T11:53:30.237 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8" 2020-11-20T11:53:30.237 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2020-11-20T11:53:30.237 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-20T11:53:30.238 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2020-11-20T11:53:30.238 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-20T11:53:30.238 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.centos.org/" 2020-11-20T11:53:30.238 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-20T11:53:30.238 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T11:53:30.239 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-20T11:53:30.239 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-20T11:53:30.239 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-20T11:53:30.239 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-20T11:53:30.240 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T11:53:30.240 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2020-11-20T11:53:30.257 INFO:teuthology.orchestra.run.smithi005:> uname -m 2020-11-20T11:53:30.273 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2020-11-20T11:53:30.274 INFO:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2020-11-20T11:53:30.328 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Linux" 2020-11-20T11:53:30.328 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="8 (Core)" 2020-11-20T11:53:30.329 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2020-11-20T11:53:30.329 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2020-11-20T11:53:30.329 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="8" 2020-11-20T11:53:30.329 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el8" 2020-11-20T11:53:30.329 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-20T11:53:30.330 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2020-11-20T11:53:30.330 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-20T11:53:30.330 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.centos.org/" 2020-11-20T11:53:30.330 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-20T11:53:30.330 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T11:53:30.331 INFO:teuthology.orchestra.run.smithi005.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-20T11:53:30.331 INFO:teuthology.orchestra.run.smithi005.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-20T11:53:30.331 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-20T11:53:30.331 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-20T11:53:30.332 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T11:53:30.332 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2020-11-20T11:53:30.347 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-20T11:53:30.401 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-20T11:53:30.411 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-20T11:53:30.412 INFO:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2020-11-20T11:53:30.415 INFO:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2020-11-20T11:53:30.428 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-20T11:53:30.435 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-20T11:53:30.435 INFO:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2020-11-20T11:53:30.469 INFO:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2020-11-20T11:53:30.495 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-20T11:53:30.536 INFO:teuthology.run_tasks:Running task kernel... 2020-11-20T11:53:30.557 DEBUG:teuthology.task.kernel:normalized config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2020-11-20T11:53:30.558 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}}, timeout 300 2020-11-20T11:53:30.558 INFO:teuthology.orchestra.run.smithi187:> sudo lsb_release -is 2020-11-20T11:53:30.611 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS 2020-11-20T11:53:30.612 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T11:53:30.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2020-11-20T11:53:30.908 DEBUG:teuthology.task.kernel:sha1 for client.0 is ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 2020-11-20T11:53:30.909 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9"... 2020-11-20T11:53:30.909 INFO:teuthology.orchestra.run.smithi187:> uname -r 2020-11-20T11:53:30.924 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-193.28.1.el8_2.x86_64 2020-11-20T11:53:30.925 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-193.28.1.el8_2.x86_64 vs ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 2020-11-20T11:53:30.926 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2020-11-20T11:53:30.927 INFO:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2020-11-20T11:53:31.003 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS 2020-11-20T11:53:31.005 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T11:53:31.006 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2020-11-20T11:53:31.299 DEBUG:teuthology.task.kernel:sha1 for client.2 is ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 2020-11-20T11:53:31.300 INFO:teuthology.task.kernel:Checking kernel version of client.2, want "ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9"... 2020-11-20T11:53:31.300 INFO:teuthology.orchestra.run.smithi005:> uname -r 2020-11-20T11:53:31.317 INFO:teuthology.orchestra.run.smithi005.stdout:4.18.0-193.28.1.el8_2.x86_64 2020-11-20T11:53:31.317 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-193.28.1.el8_2.x86_64 vs ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 2020-11-20T11:53:31.318 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2020-11-20T11:53:31.318 INFO:teuthology.orchestra.run.smithi187:> sudo yum upgrade -y linux-firmware 2020-11-20T11:53:33.886 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - AppStream 22 MB/s | 7.0 MB 00:00 2020-11-20T11:53:35.075 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - Base 25 MB/s | 2.2 MB 00:00 2020-11-20T11:53:35.498 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - Extras 86 kB/s | 5.5 kB 00:00 2020-11-20T11:53:35.939 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 21 MB/s | 8.3 MB 00:00 2020-11-20T11:53:37.682 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 4.7 MB/s | 23 kB 00:00 2020-11-20T11:53:38.815 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2020-11-20T11:53:39.868 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2020-11-20T11:53:39.868 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2020-11-20T11:53:39.868 INFO:teuthology.orchestra.run.smithi005:> sudo yum upgrade -y linux-firmware 2020-11-20T11:53:42.189 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - AppStream 44 MB/s | 7.0 MB 00:00 2020-11-20T11:53:43.470 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - Base 23 MB/s | 2.2 MB 00:00 2020-11-20T11:53:44.346 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - Extras 85 kB/s | 5.5 kB 00:00 2020-11-20T11:53:44.346 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 22 MB/s | 8.3 MB 00:00 2020-11-20T11:53:46.162 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 4.6 MB/s | 23 kB 00:00 2020-11-20T11:53:47.263 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2020-11-20T11:53:47.264 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2020-11-20T11:53:47.265 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2020-11-20T11:53:47.380 INFO:teuthology.task.kernel:Downloading kernel ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 on client.0... 2020-11-20T11:53:47.383 INFO:teuthology.orchestra.run.smithi187:> sudo lsb_release -is 2020-11-20T11:53:47.448 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS 2020-11-20T11:53:47.450 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T11:53:47.451 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 2020-11-20T11:53:47.660 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9/centos/8/flavors/default/x86_64/ 2020-11-20T11:53:47.660 INFO:teuthology.orchestra.run.smithi187:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.10.0_rc2_ceph_gac4f7b07f403-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9/centos/8/flavors/default/x86_64/ --input-file=- 2020-11-20T11:53:47.676 INFO:teuthology.task.kernel:Downloading kernel ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 on client.2... 2020-11-20T11:53:47.677 INFO:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2020-11-20T11:53:47.724 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS 2020-11-20T11:53:47.725 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T11:53:47.725 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 2020-11-20T11:53:47.896 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9/centos/8/flavors/default/x86_64/ 2020-11-20T11:53:47.897 INFO:teuthology.orchestra.run.smithi005:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.10.0_rc2_ceph_gac4f7b07f403-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9/centos/8/flavors/default/x86_64/ --input-file=- 2020-11-20T11:53:47.900 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi187.front.sepia.ceph.com to complete... 2020-11-20T11:54:12.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20 11:54:12 URL:https://2.chacra.ceph.com/r/kernel/testing/ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9/centos/8/flavors/default/x86_64/kernel-5.10.0_rc2_ceph_gac4f7b07f403-1.x86_64.rpm [517054028/517054028] -> "/tmp/kernel.x86_64.rpm" [1] 2020-11-20T11:54:12.786 INFO:teuthology.orchestra.run.smithi187.stderr:FINISHED --2020-11-20 11:54:12-- 2020-11-20T11:54:12.786 INFO:teuthology.orchestra.run.smithi187.stderr:Total wall clock time: 25s 2020-11-20T11:54:12.786 INFO:teuthology.orchestra.run.smithi187.stderr:Downloaded: 1 files, 493M in 25s (19.9 MB/s) 2020-11-20T11:54:12.787 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi005.front.sepia.ceph.com to complete... 2020-11-20T11:54:22.668 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20 11:54:22 URL:https://2.chacra.ceph.com/r/kernel/testing/ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9/centos/8/flavors/default/x86_64/kernel-5.10.0_rc2_ceph_gac4f7b07f403-1.x86_64.rpm [517054028/517054028] -> "/tmp/kernel.x86_64.rpm" [1] 2020-11-20T11:54:22.669 INFO:teuthology.orchestra.run.smithi005.stderr:FINISHED --2020-11-20 11:54:22-- 2020-11-20T11:54:22.669 INFO:teuthology.orchestra.run.smithi005.stderr:Total wall clock time: 35s 2020-11-20T11:54:22.669 INFO:teuthology.orchestra.run.smithi005.stderr:Downloaded: 1 files, 493M in 34s (14.3 MB/s) 2020-11-20T11:54:22.722 INFO:teuthology.task.kernel:Installing kernel ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 on client.0... 2020-11-20T11:54:22.723 INFO:teuthology.orchestra.run.smithi187:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2020-11-20T11:54:24.822 INFO:teuthology.orchestra.run.smithi187.stdout:Verifying... ######################################## 2020-11-20T11:54:25.336 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing... ######################################## 2020-11-20T11:54:25.407 INFO:teuthology.orchestra.run.smithi187.stdout:Updating / installing... 2020-11-20T11:55:04.225 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.10.0_rc2_ceph_gac4f7b07f403-1######################################## 2020-11-20T11:55:25.646 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2020-11-20T11:55:27.589 INFO:teuthology.orchestra.run.smithi187:> rpm -qlp /tmp/kernel.x86_64.rpm 2020-11-20T11:55:27.643 INFO:teuthology.orchestra.run.smithi187.stdout:/boot/System.map-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:27.644 INFO:teuthology.orchestra.run.smithi187.stdout:/boot/config-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:27.644 INFO:teuthology.orchestra.run.smithi187.stdout:/boot/vmlinux-5.10.0-rc2-ceph-gac4f7b07f403.bz2 2020-11-20T11:55:27.644 INFO:teuthology.orchestra.run.smithi187.stdout:/boot/vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:27.644 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:27.644 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel 2020-11-20T11:55:27.645 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch 2020-11-20T11:55:27.645 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86 2020-11-20T11:55:27.645 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto 2020-11-20T11:55:27.645 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/blowfish-x86_64.ko 2020-11-20T11:55:27.645 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2020-11-20T11:55:27.646 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2020-11-20T11:55:27.646 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/camellia-x86_64.ko 2020-11-20T11:55:27.646 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2020-11-20T11:55:27.646 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2020-11-20T11:55:27.647 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/crc32-pclmul.ko 2020-11-20T11:55:27.647 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/crc32c-intel.ko 2020-11-20T11:55:27.647 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2020-11-20T11:55:27.647 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2020-11-20T11:55:27.647 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2020-11-20T11:55:27.648 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2020-11-20T11:55:27.648 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/serpent-avx2.ko 2020-11-20T11:55:27.648 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2020-11-20T11:55:27.648 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/sha1-ssse3.ko 2020-11-20T11:55:27.648 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/sha256-ssse3.ko 2020-11-20T11:55:27.649 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/sha512-ssse3.ko 2020-11-20T11:55:27.649 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2020-11-20T11:55:27.649 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2020-11-20T11:55:27.649 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/twofish-x86_64.ko 2020-11-20T11:55:27.649 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kernel 2020-11-20T11:55:27.650 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kernel/cpu 2020-11-20T11:55:27.651 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kernel/cpu/mce 2020-11-20T11:55:27.651 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2020-11-20T11:55:27.652 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kvm 2020-11-20T11:55:27.652 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kvm/kvm-amd.ko 2020-11-20T11:55:27.652 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kvm/kvm-intel.ko 2020-11-20T11:55:27.652 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kvm/kvm.ko 2020-11-20T11:55:27.652 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/oprofile 2020-11-20T11:55:27.653 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/oprofile/oprofile.ko 2020-11-20T11:55:27.653 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto 2020-11-20T11:55:27.653 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/ansi_cprng.ko 2020-11-20T11:55:27.653 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/anubis.ko 2020-11-20T11:55:27.654 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/arc4.ko 2020-11-20T11:55:27.654 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx 2020-11-20T11:55:27.654 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_memcpy.ko 2020-11-20T11:55:27.654 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_pq.ko 2020-11-20T11:55:27.654 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_raid6_recov.ko 2020-11-20T11:55:27.655 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_tx.ko 2020-11-20T11:55:27.655 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_xor.ko 2020-11-20T11:55:27.655 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/raid6test.ko 2020-11-20T11:55:27.655 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/authenc.ko 2020-11-20T11:55:27.655 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/authencesn.ko 2020-11-20T11:55:27.656 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/blake2b_generic.ko 2020-11-20T11:55:27.656 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/blowfish_common.ko 2020-11-20T11:55:27.656 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/blowfish_generic.ko 2020-11-20T11:55:27.656 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/camellia_generic.ko 2020-11-20T11:55:27.656 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cast5_generic.ko 2020-11-20T11:55:27.657 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cast6_generic.ko 2020-11-20T11:55:27.657 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cast_common.ko 2020-11-20T11:55:27.657 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/ccm.ko 2020-11-20T11:55:27.657 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cmac.ko 2020-11-20T11:55:27.657 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/crc32_generic.ko 2020-11-20T11:55:27.658 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/crypto_engine.ko 2020-11-20T11:55:27.658 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/crypto_user.ko 2020-11-20T11:55:27.658 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cts.ko 2020-11-20T11:55:27.658 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/des_generic.ko 2020-11-20T11:55:27.658 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/dh_generic.ko 2020-11-20T11:55:27.659 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/echainiv.ko 2020-11-20T11:55:27.659 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/essiv.ko 2020-11-20T11:55:27.659 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/fcrypt.ko 2020-11-20T11:55:27.659 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/khazad.ko 2020-11-20T11:55:27.660 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/lz4.ko 2020-11-20T11:55:27.660 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/lz4hc.ko 2020-11-20T11:55:27.660 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/md4.ko 2020-11-20T11:55:27.660 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/michael_mic.ko 2020-11-20T11:55:27.660 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/pcbc.ko 2020-11-20T11:55:27.661 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/pcrypt.ko 2020-11-20T11:55:27.661 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/rmd128.ko 2020-11-20T11:55:27.661 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/rmd160.ko 2020-11-20T11:55:27.661 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/rmd256.ko 2020-11-20T11:55:27.661 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/rmd320.ko 2020-11-20T11:55:27.662 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/salsa20_generic.ko 2020-11-20T11:55:27.662 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/seed.ko 2020-11-20T11:55:27.662 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/serpent_generic.ko 2020-11-20T11:55:27.662 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/sha512_generic.ko 2020-11-20T11:55:27.662 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/tcrypt.ko 2020-11-20T11:55:27.662 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/tea.ko 2020-11-20T11:55:27.663 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/tgr192.ko 2020-11-20T11:55:27.663 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/twofish_common.ko 2020-11-20T11:55:27.663 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/twofish_generic.ko 2020-11-20T11:55:27.663 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/vmac.ko 2020-11-20T11:55:27.663 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/wp512.ko 2020-11-20T11:55:27.664 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/xcbc.ko 2020-11-20T11:55:27.664 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/xor.ko 2020-11-20T11:55:27.664 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/xxhash_generic.ko 2020-11-20T11:55:27.664 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers 2020-11-20T11:55:27.664 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi 2020-11-20T11:55:27.665 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/acpi_ipmi.ko 2020-11-20T11:55:27.665 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/acpi_pad.ko 2020-11-20T11:55:27.665 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/custom_method.ko 2020-11-20T11:55:27.665 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/ec_sys.ko 2020-11-20T11:55:27.666 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/sbs.ko 2020-11-20T11:55:27.666 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/sbshc.ko 2020-11-20T11:55:27.666 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/video.ko 2020-11-20T11:55:27.666 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata 2020-11-20T11:55:27.666 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/acard-ahci.ko 2020-11-20T11:55:27.666 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/ahci_platform.ko 2020-11-20T11:55:27.667 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/ata_generic.ko 2020-11-20T11:55:27.667 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/libahci_platform.ko 2020-11-20T11:55:27.667 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_acpi.ko 2020-11-20T11:55:27.667 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_ali.ko 2020-11-20T11:55:27.669 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_amd.ko 2020-11-20T11:55:27.669 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_artop.ko 2020-11-20T11:55:27.669 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_atiixp.ko 2020-11-20T11:55:27.669 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_atp867x.ko 2020-11-20T11:55:27.670 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_cmd640.ko 2020-11-20T11:55:27.670 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_cmd64x.ko 2020-11-20T11:55:27.670 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_cypress.ko 2020-11-20T11:55:27.670 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_efar.ko 2020-11-20T11:55:27.670 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_hpt366.ko 2020-11-20T11:55:27.671 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_hpt37x.ko 2020-11-20T11:55:27.671 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_hpt3x2n.ko 2020-11-20T11:55:27.671 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_hpt3x3.ko 2020-11-20T11:55:27.671 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_it8213.ko 2020-11-20T11:55:27.672 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_it821x.ko 2020-11-20T11:55:27.672 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_jmicron.ko 2020-11-20T11:55:27.672 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_marvell.ko 2020-11-20T11:55:27.672 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_mpiix.ko 2020-11-20T11:55:27.672 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_netcell.ko 2020-11-20T11:55:27.673 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_ninja32.ko 2020-11-20T11:55:27.673 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_ns87410.ko 2020-11-20T11:55:27.673 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_ns87415.ko 2020-11-20T11:55:27.673 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_oldpiix.ko 2020-11-20T11:55:27.673 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_opti.ko 2020-11-20T11:55:27.674 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_optidma.ko 2020-11-20T11:55:27.674 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_pcmcia.ko 2020-11-20T11:55:27.674 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_pdc2027x.ko 2020-11-20T11:55:27.674 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_pdc202xx_old.ko 2020-11-20T11:55:27.674 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_piccolo.ko 2020-11-20T11:55:27.675 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_rdc.ko 2020-11-20T11:55:27.675 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_sch.ko 2020-11-20T11:55:27.675 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_serverworks.ko 2020-11-20T11:55:27.675 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_sil680.ko 2020-11-20T11:55:27.675 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_sis.ko 2020-11-20T11:55:27.676 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_sl82c105.ko 2020-11-20T11:55:27.676 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_triflex.ko 2020-11-20T11:55:27.677 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_via.ko 2020-11-20T11:55:27.677 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pdc_adma.ko 2020-11-20T11:55:27.677 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_inic162x.ko 2020-11-20T11:55:27.677 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_mv.ko 2020-11-20T11:55:27.677 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_nv.ko 2020-11-20T11:55:27.678 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_promise.ko 2020-11-20T11:55:27.678 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_qstor.ko 2020-11-20T11:55:27.678 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_sil.ko 2020-11-20T11:55:27.678 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_sil24.ko 2020-11-20T11:55:27.678 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_sis.ko 2020-11-20T11:55:27.679 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_svw.ko 2020-11-20T11:55:27.679 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_sx4.ko 2020-11-20T11:55:27.679 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_uli.ko 2020-11-20T11:55:27.679 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_via.ko 2020-11-20T11:55:27.679 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_vsc.ko 2020-11-20T11:55:27.680 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/auxdisplay 2020-11-20T11:55:27.680 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/auxdisplay/ks0108.ko 2020-11-20T11:55:27.680 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/base 2020-11-20T11:55:27.680 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/base/regmap 2020-11-20T11:55:27.680 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/base/regmap/regmap-i2c.ko 2020-11-20T11:55:27.681 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/bcma 2020-11-20T11:55:27.681 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/bcma/bcma.ko 2020-11-20T11:55:27.681 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block 2020-11-20T11:55:27.681 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/aoe 2020-11-20T11:55:27.681 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/aoe/aoe.ko 2020-11-20T11:55:27.682 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/brd.ko 2020-11-20T11:55:27.682 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/drbd 2020-11-20T11:55:27.682 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/drbd/drbd.ko 2020-11-20T11:55:27.682 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/floppy.ko 2020-11-20T11:55:27.682 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/loop.ko 2020-11-20T11:55:27.683 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/mtip32xx 2020-11-20T11:55:27.683 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/mtip32xx/mtip32xx.ko 2020-11-20T11:55:27.683 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/nbd.ko 2020-11-20T11:55:27.683 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/null_blk.ko 2020-11-20T11:55:27.683 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/pktcdvd.ko 2020-11-20T11:55:27.684 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/rbd.ko 2020-11-20T11:55:27.684 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/skd.ko 2020-11-20T11:55:27.684 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/sx8.ko 2020-11-20T11:55:27.684 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/umem.ko 2020-11-20T11:55:27.684 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/virtio_blk.ko 2020-11-20T11:55:27.685 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/xen-blkback 2020-11-20T11:55:27.685 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/xen-blkback/xen-blkback.ko 2020-11-20T11:55:27.685 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/xen-blkfront.ko 2020-11-20T11:55:27.685 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/zram 2020-11-20T11:55:27.685 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/zram/zram.ko 2020-11-20T11:55:27.686 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char 2020-11-20T11:55:27.686 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hangcheck-timer.ko 2020-11-20T11:55:27.686 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random 2020-11-20T11:55:27.686 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/amd-rng.ko 2020-11-20T11:55:27.686 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/intel-rng.ko 2020-11-20T11:55:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/timeriomem-rng.ko 2020-11-20T11:55:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/via-rng.ko 2020-11-20T11:55:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/virtio-rng.ko 2020-11-20T11:55:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi 2020-11-20T11:55:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_devintf.ko 2020-11-20T11:55:27.688 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2020-11-20T11:55:27.688 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2020-11-20T11:55:27.688 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_si.ko 2020-11-20T11:55:27.688 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_ssif.ko 2020-11-20T11:55:27.688 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2020-11-20T11:55:27.689 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/lp.ko 2020-11-20T11:55:27.689 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/mwave 2020-11-20T11:55:27.689 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/mwave/mwave.ko 2020-11-20T11:55:27.689 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/pcmcia 2020-11-20T11:55:27.689 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/pcmcia/cm4000_cs.ko 2020-11-20T11:55:27.690 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/pcmcia/cm4040_cs.ko 2020-11-20T11:55:27.690 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ppdev.ko 2020-11-20T11:55:27.690 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tlclk.ko 2020-11-20T11:55:27.690 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm 2020-11-20T11:55:27.690 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm.ko 2020-11-20T11:55:27.691 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_atmel.ko 2020-11-20T11:55:27.691 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_infineon.ko 2020-11-20T11:55:27.691 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_nsc.ko 2020-11-20T11:55:27.691 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_tis.ko 2020-11-20T11:55:27.692 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_tis_core.ko 2020-11-20T11:55:27.692 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/uv_mmtimer.ko 2020-11-20T11:55:27.692 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/virtio_console.ko 2020-11-20T11:55:27.692 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq 2020-11-20T11:55:27.692 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/acpi-cpufreq.ko 2020-11-20T11:55:27.693 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2020-11-20T11:55:27.693 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/p4-clockmod.ko 2020-11-20T11:55:27.693 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/pcc-cpufreq.ko 2020-11-20T11:55:27.693 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/powernow-k8.ko 2020-11-20T11:55:27.693 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/speedstep-lib.ko 2020-11-20T11:55:27.694 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto 2020-11-20T11:55:27.694 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/ccp 2020-11-20T11:55:27.694 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/ccp/ccp-crypto.ko 2020-11-20T11:55:27.694 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/ccp/ccp.ko 2020-11-20T11:55:27.694 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/padlock-aes.ko 2020-11-20T11:55:27.695 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/padlock-sha.ko 2020-11-20T11:55:27.695 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat 2020-11-20T11:55:27.695 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat/qat_common 2020-11-20T11:55:27.695 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2020-11-20T11:55:27.695 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat/qat_dh895xcc 2020-11-20T11:55:27.696 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2020-11-20T11:55:27.696 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/virtio 2020-11-20T11:55:27.696 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/virtio/virtio_crypto.ko 2020-11-20T11:55:27.696 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dca 2020-11-20T11:55:27.696 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dca/dca.ko 2020-11-20T11:55:27.697 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/devfreq 2020-11-20T11:55:27.697 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/devfreq/governor_simpleondemand.ko 2020-11-20T11:55:27.697 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma 2020-11-20T11:55:27.697 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma/dw 2020-11-20T11:55:27.697 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma/dw/dw_dmac.ko 2020-11-20T11:55:27.698 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma/ioat 2020-11-20T11:55:27.698 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma/ioat/ioatdma.ko 2020-11-20T11:55:27.698 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac 2020-11-20T11:55:27.698 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/amd64_edac_mod.ko 2020-11-20T11:55:27.698 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/e752x_edac.ko 2020-11-20T11:55:27.699 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/edac_mce_amd.ko 2020-11-20T11:55:27.699 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i3000_edac.ko 2020-11-20T11:55:27.699 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i3200_edac.ko 2020-11-20T11:55:27.699 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i5000_edac.ko 2020-11-20T11:55:27.699 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i5100_edac.ko 2020-11-20T11:55:27.700 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i5400_edac.ko 2020-11-20T11:55:27.700 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i7300_edac.ko 2020-11-20T11:55:27.700 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i7core_edac.ko 2020-11-20T11:55:27.700 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i82975x_edac.ko 2020-11-20T11:55:27.700 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/ie31200_edac.ko 2020-11-20T11:55:27.701 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/sb_edac.ko 2020-11-20T11:55:27.701 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/x38_edac.ko 2020-11-20T11:55:27.701 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/firmware 2020-11-20T11:55:27.701 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/firmware/edd.ko 2020-11-20T11:55:27.701 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/firmware/iscsi_ibft.ko 2020-11-20T11:55:27.702 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/gpio 2020-11-20T11:55:27.702 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/gpio/gpio-ich.ko 2020-11-20T11:55:27.702 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/gpio/gpio-viperboard.ko 2020-11-20T11:55:27.702 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid 2020-11-20T11:55:27.702 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-appleir.ko 2020-11-20T11:55:27.703 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-aureal.ko 2020-11-20T11:55:27.703 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-axff.ko 2020-11-20T11:55:27.703 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-dr.ko 2020-11-20T11:55:27.703 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-elecom.ko 2020-11-20T11:55:27.703 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-elo.ko 2020-11-20T11:55:27.704 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-emsff.ko 2020-11-20T11:55:27.704 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-gaff.ko 2020-11-20T11:55:27.704 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-gt683r.ko 2020-11-20T11:55:27.704 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-gyration.ko 2020-11-20T11:55:27.704 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-holtek-kbd.ko 2020-11-20T11:55:27.705 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-holtek-mouse.ko 2020-11-20T11:55:27.705 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-holtekff.ko 2020-11-20T11:55:27.705 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-hyperv.ko 2020-11-20T11:55:27.705 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-icade.ko 2020-11-20T11:55:27.705 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-keytouch.ko 2020-11-20T11:55:27.706 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-kye.ko 2020-11-20T11:55:27.706 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-lcpower.ko 2020-11-20T11:55:27.706 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-led.ko 2020-11-20T11:55:27.706 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-lenovo.ko 2020-11-20T11:55:27.707 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-logitech-dj.ko 2020-11-20T11:55:27.707 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-logitech-hidpp.ko 2020-11-20T11:55:27.707 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-multitouch.ko 2020-11-20T11:55:27.707 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-ortek.ko 2020-11-20T11:55:27.707 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-penmount.ko 2020-11-20T11:55:27.707 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-petalynx.ko 2020-11-20T11:55:27.708 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-picolcd.ko 2020-11-20T11:55:27.708 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-pl.ko 2020-11-20T11:55:27.708 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-plantronics.ko 2020-11-20T11:55:27.708 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-primax.ko 2020-11-20T11:55:27.708 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-rmi.ko 2020-11-20T11:55:27.709 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-arvo.ko 2020-11-20T11:55:27.709 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-common.ko 2020-11-20T11:55:27.709 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-isku.ko 2020-11-20T11:55:27.709 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-kone.ko 2020-11-20T11:55:27.709 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-koneplus.ko 2020-11-20T11:55:27.710 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-konepure.ko 2020-11-20T11:55:27.710 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-kovaplus.ko 2020-11-20T11:55:27.710 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-lua.ko 2020-11-20T11:55:27.710 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-pyra.ko 2020-11-20T11:55:27.710 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-ryos.ko 2020-11-20T11:55:27.711 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-savu.ko 2020-11-20T11:55:27.711 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat.ko 2020-11-20T11:55:27.711 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-saitek.ko 2020-11-20T11:55:27.711 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-samsung.ko 2020-11-20T11:55:27.711 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-sensor-hub.ko 2020-11-20T11:55:27.712 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-sjoy.ko 2020-11-20T11:55:27.712 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-sony.ko 2020-11-20T11:55:27.712 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-speedlink.ko 2020-11-20T11:55:27.712 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-steelseries.ko 2020-11-20T11:55:27.712 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-sunplus.ko 2020-11-20T11:55:27.713 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-tivo.ko 2020-11-20T11:55:27.713 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-tmff.ko 2020-11-20T11:55:27.713 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-topseed.ko 2020-11-20T11:55:27.713 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-twinhan.ko 2020-11-20T11:55:27.714 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-uclogic.ko 2020-11-20T11:55:27.714 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-waltop.ko 2020-11-20T11:55:27.714 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-wiimote.ko 2020-11-20T11:55:27.714 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-xinmo.ko 2020-11-20T11:55:27.714 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-zpff.ko 2020-11-20T11:55:27.715 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-zydacron.ko 2020-11-20T11:55:27.715 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/i2c-hid 2020-11-20T11:55:27.715 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/i2c-hid/i2c-hid.ko 2020-11-20T11:55:27.715 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/uhid.ko 2020-11-20T11:55:27.715 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/wacom.ko 2020-11-20T11:55:27.716 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hv 2020-11-20T11:55:27.716 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hv/hv_balloon.ko 2020-11-20T11:55:27.716 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hv/hv_utils.ko 2020-11-20T11:55:27.716 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hv/hv_vmbus.ko 2020-11-20T11:55:27.716 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon 2020-11-20T11:55:27.717 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/abituguru.ko 2020-11-20T11:55:27.717 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/abituguru3.ko 2020-11-20T11:55:27.717 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/acpi_power_meter.ko 2020-11-20T11:55:27.717 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ad7414.ko 2020-11-20T11:55:27.717 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ad7418.ko 2020-11-20T11:55:27.718 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adc128d818.ko 2020-11-20T11:55:27.718 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1021.ko 2020-11-20T11:55:27.718 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1025.ko 2020-11-20T11:55:27.718 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1026.ko 2020-11-20T11:55:27.718 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1029.ko 2020-11-20T11:55:27.719 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1031.ko 2020-11-20T11:55:27.719 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm9240.ko 2020-11-20T11:55:27.719 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ads7828.ko 2020-11-20T11:55:27.719 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7410.ko 2020-11-20T11:55:27.719 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7411.ko 2020-11-20T11:55:27.720 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7462.ko 2020-11-20T11:55:27.720 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7470.ko 2020-11-20T11:55:27.720 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7475.ko 2020-11-20T11:55:27.720 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7x10.ko 2020-11-20T11:55:27.720 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/amc6821.ko 2020-11-20T11:55:27.721 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/applesmc.ko 2020-11-20T11:55:27.721 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/asb100.ko 2020-11-20T11:55:27.721 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/asc7621.ko 2020-11-20T11:55:27.721 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/asus_atk0110.ko 2020-11-20T11:55:27.721 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/atxp1.ko 2020-11-20T11:55:27.722 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/coretemp.ko 2020-11-20T11:55:27.722 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/dell-smm-hwmon.ko 2020-11-20T11:55:27.722 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/dme1737.ko 2020-11-20T11:55:27.722 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ds1621.ko 2020-11-20T11:55:27.722 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ds620.ko 2020-11-20T11:55:27.723 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/emc1403.ko 2020-11-20T11:55:27.723 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/emc6w201.ko 2020-11-20T11:55:27.723 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/f71805f.ko 2020-11-20T11:55:27.723 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/f71882fg.ko 2020-11-20T11:55:27.723 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/f75375s.ko 2020-11-20T11:55:27.724 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/fam15h_power.ko 2020-11-20T11:55:27.724 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/fschmd.ko 2020-11-20T11:55:27.724 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/g760a.ko 2020-11-20T11:55:27.724 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/g762.ko 2020-11-20T11:55:27.724 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/gl518sm.ko 2020-11-20T11:55:27.725 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/gl520sm.ko 2020-11-20T11:55:27.725 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/hwmon-vid.ko 2020-11-20T11:55:27.725 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/i5500_temp.ko 2020-11-20T11:55:27.725 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/i5k_amb.ko 2020-11-20T11:55:27.725 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ibmaem.ko 2020-11-20T11:55:27.726 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ibmpex.ko 2020-11-20T11:55:27.726 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ina209.ko 2020-11-20T11:55:27.726 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ina2xx.ko 2020-11-20T11:55:27.726 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/it87.ko 2020-11-20T11:55:27.726 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/k10temp.ko 2020-11-20T11:55:27.727 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/k8temp.ko 2020-11-20T11:55:27.727 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lineage-pem.ko 2020-11-20T11:55:27.727 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm63.ko 2020-11-20T11:55:27.727 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm73.ko 2020-11-20T11:55:27.727 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm75.ko 2020-11-20T11:55:27.728 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm77.ko 2020-11-20T11:55:27.728 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm78.ko 2020-11-20T11:55:27.728 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm80.ko 2020-11-20T11:55:27.728 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm83.ko 2020-11-20T11:55:27.728 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm85.ko 2020-11-20T11:55:27.729 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm87.ko 2020-11-20T11:55:27.729 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm90.ko 2020-11-20T11:55:27.729 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm92.ko 2020-11-20T11:55:27.729 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm93.ko 2020-11-20T11:55:27.729 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm95234.ko 2020-11-20T11:55:27.730 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm95241.ko 2020-11-20T11:55:27.730 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm95245.ko 2020-11-20T11:55:27.730 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc2945.ko 2020-11-20T11:55:27.730 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4151.ko 2020-11-20T11:55:27.730 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4215.ko 2020-11-20T11:55:27.731 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4222.ko 2020-11-20T11:55:27.731 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4245.ko 2020-11-20T11:55:27.731 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4260.ko 2020-11-20T11:55:27.731 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4261.ko 2020-11-20T11:55:27.731 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max16065.ko 2020-11-20T11:55:27.732 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max1619.ko 2020-11-20T11:55:27.732 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max1668.ko 2020-11-20T11:55:27.732 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max197.ko 2020-11-20T11:55:27.732 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max6639.ko 2020-11-20T11:55:27.732 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max6642.ko 2020-11-20T11:55:27.733 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max6650.ko 2020-11-20T11:55:27.733 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max6697.ko 2020-11-20T11:55:27.733 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/mcp3021.ko 2020-11-20T11:55:27.733 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/nct6683.ko 2020-11-20T11:55:27.733 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/nct6775.ko 2020-11-20T11:55:27.734 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/nct7802.ko 2020-11-20T11:55:27.734 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ntc_thermistor.ko 2020-11-20T11:55:27.734 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pc87360.ko 2020-11-20T11:55:27.734 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pc87427.ko 2020-11-20T11:55:27.735 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pcf8591.ko 2020-11-20T11:55:27.735 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus 2020-11-20T11:55:27.735 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/adm1275.ko 2020-11-20T11:55:27.735 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/lm25066.ko 2020-11-20T11:55:27.735 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/ltc2978.ko 2020-11-20T11:55:27.736 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/max16064.ko 2020-11-20T11:55:27.736 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/max34440.ko 2020-11-20T11:55:27.736 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/max8688.ko 2020-11-20T11:55:27.736 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/pmbus.ko 2020-11-20T11:55:27.736 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2020-11-20T11:55:27.737 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/tps40422.ko 2020-11-20T11:55:27.737 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/ucd9000.ko 2020-11-20T11:55:27.737 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/ucd9200.ko 2020-11-20T11:55:27.737 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/zl6100.ko 2020-11-20T11:55:27.737 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/powr1220.ko 2020-11-20T11:55:27.738 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sch5627.ko 2020-11-20T11:55:27.738 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sch5636.ko 2020-11-20T11:55:27.738 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sch56xx-common.ko 2020-11-20T11:55:27.738 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sht15.ko 2020-11-20T11:55:27.738 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sht21.ko 2020-11-20T11:55:27.739 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/shtc1.ko 2020-11-20T11:55:27.739 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sis5595.ko 2020-11-20T11:55:27.739 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/smsc47b397.ko 2020-11-20T11:55:27.739 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/smsc47m1.ko 2020-11-20T11:55:27.739 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/smsc47m192.ko 2020-11-20T11:55:27.740 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/thmc50.ko 2020-11-20T11:55:27.740 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/tmp102.ko 2020-11-20T11:55:27.740 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/tmp103.ko 2020-11-20T11:55:27.740 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/tmp401.ko 2020-11-20T11:55:27.740 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/tmp421.ko 2020-11-20T11:55:27.741 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/via-cputemp.ko 2020-11-20T11:55:27.741 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/via686a.ko 2020-11-20T11:55:27.741 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/vt1211.ko 2020-11-20T11:55:27.741 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/vt8231.ko 2020-11-20T11:55:27.741 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83627ehf.ko 2020-11-20T11:55:27.742 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83627hf.ko 2020-11-20T11:55:27.742 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83781d.ko 2020-11-20T11:55:27.742 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83791d.ko 2020-11-20T11:55:27.742 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83792d.ko 2020-11-20T11:55:27.742 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83793.ko 2020-11-20T11:55:27.743 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83795.ko 2020-11-20T11:55:27.743 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83l785ts.ko 2020-11-20T11:55:27.743 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83l786ng.ko 2020-11-20T11:55:27.743 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c 2020-11-20T11:55:27.743 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/algos 2020-11-20T11:55:27.744 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2020-11-20T11:55:27.744 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2020-11-20T11:55:27.744 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses 2020-11-20T11:55:27.744 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2020-11-20T11:55:27.745 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-amd756.ko 2020-11-20T11:55:27.745 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-amd8111.ko 2020-11-20T11:55:27.745 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-designware-core.ko 2020-11-20T11:55:27.745 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2020-11-20T11:55:27.745 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2020-11-20T11:55:27.746 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2020-11-20T11:55:27.746 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-i801.ko 2020-11-20T11:55:27.746 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-isch.ko 2020-11-20T11:55:27.746 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-ismt.ko 2020-11-20T11:55:27.746 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2020-11-20T11:55:27.747 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-nforce2.ko 2020-11-20T11:55:27.747 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-parport.ko 2020-11-20T11:55:27.747 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2020-11-20T11:55:27.747 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-piix4.ko 2020-11-20T11:55:27.747 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-scmi.ko 2020-11-20T11:55:27.748 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-simtec.ko 2020-11-20T11:55:27.748 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-sis96x.ko 2020-11-20T11:55:27.748 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2020-11-20T11:55:27.748 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-via.ko 2020-11-20T11:55:27.748 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-viapro.ko 2020-11-20T11:55:27.749 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-viperboard.ko 2020-11-20T11:55:27.749 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-dev.ko 2020-11-20T11:55:27.749 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-mux.ko 2020-11-20T11:55:27.749 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-slave-eeprom.ko 2020-11-20T11:55:27.749 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-smbus.ko 2020-11-20T11:55:27.750 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-stub.ko 2020-11-20T11:55:27.750 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband 2020-11-20T11:55:27.750 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core 2020-11-20T11:55:27.750 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/ib_cm.ko 2020-11-20T11:55:27.750 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/ib_core.ko 2020-11-20T11:55:27.751 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/ib_umad.ko 2020-11-20T11:55:27.751 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/ib_uverbs.ko 2020-11-20T11:55:27.751 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/iw_cm.ko 2020-11-20T11:55:27.751 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/rdma_cm.ko 2020-11-20T11:55:27.751 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/rdma_ucm.ko 2020-11-20T11:55:27.752 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw 2020-11-20T11:55:27.752 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/cxgb4 2020-11-20T11:55:27.752 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2020-11-20T11:55:27.752 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mlx4 2020-11-20T11:55:27.752 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2020-11-20T11:55:27.753 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mlx5 2020-11-20T11:55:27.753 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2020-11-20T11:55:27.753 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mthca 2020-11-20T11:55:27.753 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2020-11-20T11:55:27.753 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp 2020-11-20T11:55:27.754 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/ipoib 2020-11-20T11:55:27.754 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2020-11-20T11:55:27.754 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/iser 2020-11-20T11:55:27.754 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2020-11-20T11:55:27.754 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/isert 2020-11-20T11:55:27.755 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2020-11-20T11:55:27.755 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/srp 2020-11-20T11:55:27.755 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2020-11-20T11:55:27.755 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/srpt 2020-11-20T11:55:27.755 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2020-11-20T11:55:27.756 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input 2020-11-20T11:55:27.756 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport 2020-11-20T11:55:27.756 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/emu10k1-gp.ko 2020-11-20T11:55:27.756 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/fm801-gp.ko 2020-11-20T11:55:27.756 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/gameport.ko 2020-11-20T11:55:27.757 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/lightning.ko 2020-11-20T11:55:27.757 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/ns558.ko 2020-11-20T11:55:27.757 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/input-polldev.ko 2020-11-20T11:55:27.757 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/joydev.ko 2020-11-20T11:55:27.757 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/keyboard 2020-11-20T11:55:27.758 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/keyboard/gpio_keys.ko 2020-11-20T11:55:27.758 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse 2020-11-20T11:55:27.758 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/appletouch.ko 2020-11-20T11:55:27.758 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/bcm5974.ko 2020-11-20T11:55:27.758 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/cyapatp.ko 2020-11-20T11:55:27.759 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/elan_i2c.ko 2020-11-20T11:55:27.759 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/sermouse.ko 2020-11-20T11:55:27.759 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/synaptics_i2c.ko 2020-11-20T11:55:27.759 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/synaptics_usb.ko 2020-11-20T11:55:27.759 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/vsxxxaa.ko 2020-11-20T11:55:27.760 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/rmi4 2020-11-20T11:55:27.760 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/rmi4/rmi_core.ko 2020-11-20T11:55:27.760 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio 2020-11-20T11:55:27.760 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio/altera_ps2.ko 2020-11-20T11:55:27.760 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio/arc_ps2.ko 2020-11-20T11:55:27.761 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio/hyperv-keyboard.ko 2020-11-20T11:55:27.761 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio/serio_raw.ko 2020-11-20T11:55:27.761 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/sparse-keymap.ko 2020-11-20T11:55:27.761 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/iommu 2020-11-20T11:55:27.761 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/iommu/amd 2020-11-20T11:55:27.762 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/iommu/amd/iommu_v2.ko 2020-11-20T11:55:27.762 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds 2020-11-20T11:55:27.762 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-blinkm.ko 2020-11-20T11:55:27.762 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-clevo-mail.ko 2020-11-20T11:55:27.762 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-lm3530.ko 2020-11-20T11:55:27.763 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-lp3944.ko 2020-11-20T11:55:27.763 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-ss4200.ko 2020-11-20T11:55:27.763 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger 2020-11-20T11:55:27.763 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-audio.ko 2020-11-20T11:55:27.763 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2020-11-20T11:55:27.764 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-camera.ko 2020-11-20T11:55:27.764 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2020-11-20T11:55:27.764 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2020-11-20T11:55:27.764 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2020-11-20T11:55:27.764 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2020-11-20T11:55:27.765 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-timer.ko 2020-11-20T11:55:27.765 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-transient.ko 2020-11-20T11:55:27.765 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md 2020-11-20T11:55:27.765 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/bcache 2020-11-20T11:55:27.765 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/bcache/bcache.ko 2020-11-20T11:55:27.766 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-bio-prison.ko 2020-11-20T11:55:27.766 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-cache-smq.ko 2020-11-20T11:55:27.766 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-cache.ko 2020-11-20T11:55:27.766 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-crypt.ko 2020-11-20T11:55:27.766 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-delay.ko 2020-11-20T11:55:27.767 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-flakey.ko 2020-11-20T11:55:27.767 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-log-userspace.ko 2020-11-20T11:55:27.767 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-multipath.ko 2020-11-20T11:55:27.767 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-queue-length.ko 2020-11-20T11:55:27.767 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-raid.ko 2020-11-20T11:55:27.768 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-round-robin.ko 2020-11-20T11:55:27.768 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-service-time.ko 2020-11-20T11:55:27.768 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-switch.ko 2020-11-20T11:55:27.768 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-thin-pool.ko 2020-11-20T11:55:27.769 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-verity.ko 2020-11-20T11:55:27.769 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/faulty.ko 2020-11-20T11:55:27.769 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/linear.ko 2020-11-20T11:55:27.769 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/multipath.ko 2020-11-20T11:55:27.769 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/persistent-data 2020-11-20T11:55:27.770 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2020-11-20T11:55:27.770 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/raid0.ko 2020-11-20T11:55:27.770 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/raid1.ko 2020-11-20T11:55:27.770 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/raid10.ko 2020-11-20T11:55:27.770 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/raid456.ko 2020-11-20T11:55:27.771 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message 2020-11-20T11:55:27.771 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion 2020-11-20T11:55:27.771 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptbase.ko 2020-11-20T11:55:27.771 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptctl.ko 2020-11-20T11:55:27.771 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptfc.ko 2020-11-20T11:55:27.772 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptlan.ko 2020-11-20T11:55:27.772 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptsas.ko 2020-11-20T11:55:27.772 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptscsih.ko 2020-11-20T11:55:27.772 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptspi.ko 2020-11-20T11:55:27.772 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd 2020-11-20T11:55:27.772 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/lpc_ich.ko 2020-11-20T11:55:27.773 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/lpc_sch.ko 2020-11-20T11:55:27.773 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/mfd-core.ko 2020-11-20T11:55:27.773 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/sm501.ko 2020-11-20T11:55:27.773 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/viperboard.ko 2020-11-20T11:55:27.774 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/vx855.ko 2020-11-20T11:55:27.774 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/wl1273-core.ko 2020-11-20T11:55:27.774 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc 2020-11-20T11:55:27.774 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/altera-stapl 2020-11-20T11:55:27.774 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2020-11-20T11:55:27.774 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/apds9802als.ko 2020-11-20T11:55:27.775 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/apds990x.ko 2020-11-20T11:55:27.775 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/bh1770glc.ko 2020-11-20T11:55:27.775 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/cb710 2020-11-20T11:55:27.775 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/cb710/cb710.ko 2020-11-20T11:55:27.775 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom 2020-11-20T11:55:27.776 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom/at24.ko 2020-11-20T11:55:27.776 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom/eeprom.ko 2020-11-20T11:55:27.776 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2020-11-20T11:55:27.776 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom/max6875.ko 2020-11-20T11:55:27.776 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/enclosure.ko 2020-11-20T11:55:27.777 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/hpilo.ko 2020-11-20T11:55:27.777 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/isl29003.ko 2020-11-20T11:55:27.777 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/isl29020.ko 2020-11-20T11:55:27.777 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/lis3lv02d 2020-11-20T11:55:27.777 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2020-11-20T11:55:27.778 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2020-11-20T11:55:27.778 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/mei 2020-11-20T11:55:27.778 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/mei/mei-me.ko 2020-11-20T11:55:27.778 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/mei/mei-txe.ko 2020-11-20T11:55:27.779 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/mei/mei.ko 2020-11-20T11:55:27.779 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/pvpanic.ko 2020-11-20T11:55:27.779 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-gru 2020-11-20T11:55:27.779 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-gru/gru.ko 2020-11-20T11:55:27.779 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-xp 2020-11-20T11:55:27.780 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-xp/xp.ko 2020-11-20T11:55:27.780 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-xp/xpc.ko 2020-11-20T11:55:27.780 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-xp/xpnet.ko 2020-11-20T11:55:27.780 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/tifm_7xx1.ko 2020-11-20T11:55:27.780 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/tifm_core.ko 2020-11-20T11:55:27.781 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/tsl2550.ko 2020-11-20T11:55:27.781 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/vmw_balloon.ko 2020-11-20T11:55:27.781 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/vmw_vmci 2020-11-20T11:55:27.781 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2020-11-20T11:55:27.781 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net 2020-11-20T11:55:27.782 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/bonding 2020-11-20T11:55:27.782 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/bonding/bonding.ko 2020-11-20T11:55:27.782 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa 2020-11-20T11:55:27.782 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/b53 2020-11-20T11:55:27.782 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/b53/b53_common.ko 2020-11-20T11:55:27.783 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/bcm-sf2.ko 2020-11-20T11:55:27.783 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/mv88e6060.ko 2020-11-20T11:55:27.783 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/mv88e6xxx 2020-11-20T11:55:27.783 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2020-11-20T11:55:27.783 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dummy.ko 2020-11-20T11:55:27.784 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/eql.ko 2020-11-20T11:55:27.784 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet 2020-11-20T11:55:27.784 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com 2020-11-20T11:55:27.784 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2020-11-20T11:55:27.784 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2020-11-20T11:55:27.785 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com/3c59x.ko 2020-11-20T11:55:27.785 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com/typhoon.ko 2020-11-20T11:55:27.785 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390 2020-11-20T11:55:27.785 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390/8390.ko 2020-11-20T11:55:27.785 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2020-11-20T11:55:27.786 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2020-11-20T11:55:27.786 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2020-11-20T11:55:27.786 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/adaptec 2020-11-20T11:55:27.786 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/adaptec/starfire.ko 2020-11-20T11:55:27.786 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/agere 2020-11-20T11:55:27.787 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/agere/et131x.ko 2020-11-20T11:55:27.787 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/alteon 2020-11-20T11:55:27.787 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/alteon/acenic.ko 2020-11-20T11:55:27.787 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/altera 2020-11-20T11:55:27.787 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/altera/altera_tse.ko 2020-11-20T11:55:27.788 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/amd 2020-11-20T11:55:27.788 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/amd/amd8111e.ko 2020-11-20T11:55:27.788 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2020-11-20T11:55:27.788 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/amd/pcnet32.ko 2020-11-20T11:55:27.788 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros 2020-11-20T11:55:27.789 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/alx 2020-11-20T11:55:27.789 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2020-11-20T11:55:27.789 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atl1c 2020-11-20T11:55:27.789 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2020-11-20T11:55:27.789 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atl1e 2020-11-20T11:55:27.790 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2020-11-20T11:55:27.790 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atlx 2020-11-20T11:55:27.790 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2020-11-20T11:55:27.790 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2020-11-20T11:55:27.790 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom 2020-11-20T11:55:27.791 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/b44.ko 2020-11-20T11:55:27.791 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2020-11-20T11:55:27.791 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/bnx2x 2020-11-20T11:55:27.791 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2020-11-20T11:55:27.791 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/cnic.ko 2020-11-20T11:55:27.792 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/genet 2020-11-20T11:55:27.792 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2020-11-20T11:55:27.792 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/tg3.ko 2020-11-20T11:55:27.792 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/brocade 2020-11-20T11:55:27.792 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/brocade/bna 2020-11-20T11:55:27.793 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2020-11-20T11:55:27.793 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio 2020-11-20T11:55:27.793 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb 2020-11-20T11:55:27.793 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2020-11-20T11:55:27.793 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb3 2020-11-20T11:55:27.794 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2020-11-20T11:55:27.794 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb4 2020-11-20T11:55:27.794 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2020-11-20T11:55:27.794 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2020-11-20T11:55:27.794 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2020-11-20T11:55:27.795 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/libcxgb 2020-11-20T11:55:27.795 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2020-11-20T11:55:27.795 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/cisco 2020-11-20T11:55:27.795 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/cisco/enic 2020-11-20T11:55:27.795 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2020-11-20T11:55:27.796 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec 2020-11-20T11:55:27.796 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip 2020-11-20T11:55:27.796 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2020-11-20T11:55:27.796 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/de4x5.ko 2020-11-20T11:55:27.796 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2020-11-20T11:55:27.797 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2020-11-20T11:55:27.797 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2020-11-20T11:55:27.797 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2020-11-20T11:55:27.797 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2020-11-20T11:55:27.797 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dlink 2020-11-20T11:55:27.798 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dlink/dl2k.ko 2020-11-20T11:55:27.798 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dlink/sundance.ko 2020-11-20T11:55:27.798 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dnet.ko 2020-11-20T11:55:27.798 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/emulex 2020-11-20T11:55:27.798 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/emulex/benet 2020-11-20T11:55:27.799 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2020-11-20T11:55:27.799 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/ethoc.ko 2020-11-20T11:55:27.799 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/fealnx.ko 2020-11-20T11:55:27.799 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel 2020-11-20T11:55:27.799 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e100.ko 2020-11-20T11:55:27.800 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e1000 2020-11-20T11:55:27.800 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2020-11-20T11:55:27.800 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e1000e 2020-11-20T11:55:27.800 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2020-11-20T11:55:27.800 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/fm10k 2020-11-20T11:55:27.801 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2020-11-20T11:55:27.801 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/i40e 2020-11-20T11:55:27.801 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2020-11-20T11:55:27.801 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/iavf 2020-11-20T11:55:27.801 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2020-11-20T11:55:27.802 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/igb 2020-11-20T11:55:27.802 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/igb/igb.ko 2020-11-20T11:55:27.802 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/igbvf 2020-11-20T11:55:27.802 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2020-11-20T11:55:27.802 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgb 2020-11-20T11:55:27.803 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2020-11-20T11:55:27.803 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgbe 2020-11-20T11:55:27.803 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2020-11-20T11:55:27.803 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgbevf 2020-11-20T11:55:27.803 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2020-11-20T11:55:27.804 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/jme.ko 2020-11-20T11:55:27.804 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/marvell 2020-11-20T11:55:27.804 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2020-11-20T11:55:27.804 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/marvell/skge.ko 2020-11-20T11:55:27.804 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/marvell/sky2.ko 2020-11-20T11:55:27.805 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox 2020-11-20T11:55:27.805 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx4 2020-11-20T11:55:27.805 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2020-11-20T11:55:27.805 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2020-11-20T11:55:27.805 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx5 2020-11-20T11:55:27.806 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx5/core 2020-11-20T11:55:27.806 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2020-11-20T11:55:27.806 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/micrel 2020-11-20T11:55:27.806 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2020-11-20T11:55:27.806 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/myricom 2020-11-20T11:55:27.807 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/myricom/myri10ge 2020-11-20T11:55:27.807 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2020-11-20T11:55:27.807 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/natsemi 2020-11-20T11:55:27.807 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2020-11-20T11:55:27.807 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2020-11-20T11:55:27.808 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/neterion 2020-11-20T11:55:27.808 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/neterion/s2io.ko 2020-11-20T11:55:27.808 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/neterion/vxge 2020-11-20T11:55:27.808 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/neterion/vxge/vxge.ko 2020-11-20T11:55:27.808 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/nvidia 2020-11-20T11:55:27.809 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2020-11-20T11:55:27.809 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/packetengines 2020-11-20T11:55:27.809 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2020-11-20T11:55:27.809 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2020-11-20T11:55:27.809 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic 2020-11-20T11:55:27.810 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/netxen 2020-11-20T11:55:27.810 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2020-11-20T11:55:27.810 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2020-11-20T11:55:27.810 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/qlcnic 2020-11-20T11:55:27.810 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2020-11-20T11:55:27.811 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/rdc 2020-11-20T11:55:27.811 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/rdc/r6040.ko 2020-11-20T11:55:27.811 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek 2020-11-20T11:55:27.811 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek/8139cp.ko 2020-11-20T11:55:27.811 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek/8139too.ko 2020-11-20T11:55:27.812 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek/atp.ko 2020-11-20T11:55:27.812 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek/r8169.ko 2020-11-20T11:55:27.812 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/rocker 2020-11-20T11:55:27.812 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/rocker/rocker.ko 2020-11-20T11:55:27.812 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sfc 2020-11-20T11:55:27.813 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sfc/sfc.ko 2020-11-20T11:55:27.813 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/silan 2020-11-20T11:55:27.813 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/silan/sc92031.ko 2020-11-20T11:55:27.813 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sis 2020-11-20T11:55:27.813 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sis/sis190.ko 2020-11-20T11:55:27.814 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sis/sis900.ko 2020-11-20T11:55:27.814 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc 2020-11-20T11:55:27.814 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc/epic100.ko 2020-11-20T11:55:27.814 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2020-11-20T11:55:27.814 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2020-11-20T11:55:27.815 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2020-11-20T11:55:27.815 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/stmicro 2020-11-20T11:55:27.815 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/stmicro/stmmac 2020-11-20T11:55:27.815 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2020-11-20T11:55:27.815 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2020-11-20T11:55:27.816 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun 2020-11-20T11:55:27.816 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun/cassini.ko 2020-11-20T11:55:27.816 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun/niu.ko 2020-11-20T11:55:27.816 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun/sungem.ko 2020-11-20T11:55:27.817 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun/sunhme.ko 2020-11-20T11:55:27.817 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/tehuti 2020-11-20T11:55:27.817 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2020-11-20T11:55:27.817 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/ti 2020-11-20T11:55:27.817 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/ti/tlan.ko 2020-11-20T11:55:27.818 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/via 2020-11-20T11:55:27.818 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/via/via-rhine.ko 2020-11-20T11:55:27.818 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/via/via-velocity.ko 2020-11-20T11:55:27.818 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/wiznet 2020-11-20T11:55:27.818 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/wiznet/w5100.ko 2020-11-20T11:55:27.819 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/wiznet/w5300.ko 2020-11-20T11:55:27.819 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/xircom 2020-11-20T11:55:27.819 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2020-11-20T11:55:27.819 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/geneve.ko 2020-11-20T11:55:27.819 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/hyperv 2020-11-20T11:55:27.820 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/hyperv/hv_netvsc.ko 2020-11-20T11:55:27.820 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ieee802154 2020-11-20T11:55:27.820 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ieee802154/fakelb.ko 2020-11-20T11:55:27.820 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ifb.ko 2020-11-20T11:55:27.820 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ipvlan 2020-11-20T11:55:27.821 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ipvlan/ipvlan.ko 2020-11-20T11:55:27.821 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/macvlan.ko 2020-11-20T11:55:27.821 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/macvtap.ko 2020-11-20T11:55:27.821 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mdio 2020-11-20T11:55:27.821 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mdio.ko 2020-11-20T11:55:27.822 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2020-11-20T11:55:27.822 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mdio/mdio-bitbang.ko 2020-11-20T11:55:27.822 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mii.ko 2020-11-20T11:55:27.822 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/net_failover.ko 2020-11-20T11:55:27.822 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/netconsole.ko 2020-11-20T11:55:27.823 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/nlmon.ko 2020-11-20T11:55:27.823 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/pcs 2020-11-20T11:55:27.823 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/pcs/pcs-xpcs.ko 2020-11-20T11:55:27.823 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy 2020-11-20T11:55:27.823 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/amd.ko 2020-11-20T11:55:27.824 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/bcm-phy-lib.ko 2020-11-20T11:55:27.824 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/bcm7xxx.ko 2020-11-20T11:55:27.824 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/bcm87xx.ko 2020-11-20T11:55:27.824 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/broadcom.ko 2020-11-20T11:55:27.824 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/cicada.ko 2020-11-20T11:55:27.825 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/davicom.ko 2020-11-20T11:55:27.825 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/dp83640.ko 2020-11-20T11:55:27.825 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/et1011c.ko 2020-11-20T11:55:27.825 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/icplus.ko 2020-11-20T11:55:27.825 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/lxt.ko 2020-11-20T11:55:27.826 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/marvell.ko 2020-11-20T11:55:27.826 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/micrel.ko 2020-11-20T11:55:27.826 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/national.ko 2020-11-20T11:55:27.826 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/phylink.ko 2020-11-20T11:55:27.826 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/qsemi.ko 2020-11-20T11:55:27.827 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/realtek.ko 2020-11-20T11:55:27.827 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/smsc.ko 2020-11-20T11:55:27.827 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/ste10Xp.ko 2020-11-20T11:55:27.827 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/vitesse.ko 2020-11-20T11:55:27.827 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp 2020-11-20T11:55:27.828 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/bsd_comp.ko 2020-11-20T11:55:27.828 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_async.ko 2020-11-20T11:55:27.828 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_deflate.ko 2020-11-20T11:55:27.828 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_generic.ko 2020-11-20T11:55:27.828 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_mppe.ko 2020-11-20T11:55:27.829 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_synctty.ko 2020-11-20T11:55:27.829 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/pppoe.ko 2020-11-20T11:55:27.829 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/pppox.ko 2020-11-20T11:55:27.829 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/pptp.ko 2020-11-20T11:55:27.829 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/slip 2020-11-20T11:55:27.830 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/slip/slhc.ko 2020-11-20T11:55:27.830 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/slip/slip.ko 2020-11-20T11:55:27.830 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/sungem_phy.ko 2020-11-20T11:55:27.830 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/tap.ko 2020-11-20T11:55:27.830 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team 2020-11-20T11:55:27.831 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team.ko 2020-11-20T11:55:27.831 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_activebackup.ko 2020-11-20T11:55:27.831 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_broadcast.ko 2020-11-20T11:55:27.831 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_loadbalance.ko 2020-11-20T11:55:27.831 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_random.ko 2020-11-20T11:55:27.832 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_roundrobin.ko 2020-11-20T11:55:27.832 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/tun.ko 2020-11-20T11:55:27.832 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/veth.ko 2020-11-20T11:55:27.832 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/virtio_net.ko 2020-11-20T11:55:27.832 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/vmxnet3 2020-11-20T11:55:27.833 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/vmxnet3/vmxnet3.ko 2020-11-20T11:55:27.833 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/vxlan.ko 2020-11-20T11:55:27.833 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/xen-netback 2020-11-20T11:55:27.833 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/xen-netback/xen-netback.ko 2020-11-20T11:55:27.833 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/xen-netfront.ko 2020-11-20T11:55:27.834 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ntb 2020-11-20T11:55:27.834 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ntb/ntb.ko 2020-11-20T11:55:27.834 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/nvme 2020-11-20T11:55:27.834 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/nvme/host 2020-11-20T11:55:27.834 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/nvme/host/nvme-core.ko 2020-11-20T11:55:27.835 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/nvme/host/nvme.ko 2020-11-20T11:55:27.835 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport 2020-11-20T11:55:27.835 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport/parport.ko 2020-11-20T11:55:27.835 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport/parport_cs.ko 2020-11-20T11:55:27.835 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport/parport_pc.ko 2020-11-20T11:55:27.836 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport/parport_serial.ko 2020-11-20T11:55:27.836 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci 2020-11-20T11:55:27.836 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/hotplug 2020-11-20T11:55:27.836 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2020-11-20T11:55:27.836 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/pcie 2020-11-20T11:55:27.837 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/pcie/aer_inject.ko 2020-11-20T11:55:27.837 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/xen-pcifront.ko 2020-11-20T11:55:27.837 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pcmcia 2020-11-20T11:55:27.837 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pcmcia/i82092.ko 2020-11-20T11:55:27.837 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pcmcia/pd6729.ko 2020-11-20T11:55:27.838 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pcmcia/yenta_socket.ko 2020-11-20T11:55:27.838 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pinctrl 2020-11-20T11:55:27.838 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pinctrl/intel 2020-11-20T11:55:27.838 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2020-11-20T11:55:27.838 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform 2020-11-20T11:55:27.839 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86 2020-11-20T11:55:27.839 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/acer-wmi.ko 2020-11-20T11:55:27.839 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/acerhdf.ko 2020-11-20T11:55:27.839 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/alienware-wmi.ko 2020-11-20T11:55:27.839 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/apple-gmux.ko 2020-11-20T11:55:27.840 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/asus-laptop.ko 2020-11-20T11:55:27.840 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/asus-nb-wmi.ko 2020-11-20T11:55:27.840 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/asus-wmi.ko 2020-11-20T11:55:27.840 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/classmate-laptop.ko 2020-11-20T11:55:27.840 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/dcdbas.ko 2020-11-20T11:55:27.841 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/dell-smo8800.ko 2020-11-20T11:55:27.841 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/dell-wmi-aio.ko 2020-11-20T11:55:27.841 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/eeepc-laptop.ko 2020-11-20T11:55:27.841 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/eeepc-wmi.ko 2020-11-20T11:55:27.841 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/fujitsu-laptop.ko 2020-11-20T11:55:27.842 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/fujitsu-tablet.ko 2020-11-20T11:55:27.842 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/hdaps.ko 2020-11-20T11:55:27.842 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/hp-wireless.ko 2020-11-20T11:55:27.842 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/hp-wmi.ko 2020-11-20T11:55:27.842 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/hp_accel.ko 2020-11-20T11:55:27.843 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/intel-rst.ko 2020-11-20T11:55:27.843 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/intel_ips.ko 2020-11-20T11:55:27.843 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/msi-wmi.ko 2020-11-20T11:55:27.843 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/mxm-wmi.ko 2020-11-20T11:55:27.843 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/panasonic-laptop.ko 2020-11-20T11:55:27.844 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/samsung-laptop.ko 2020-11-20T11:55:27.844 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/samsung-q10.ko 2020-11-20T11:55:27.844 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/thinkpad_acpi.ko 2020-11-20T11:55:27.844 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/topstar-laptop.ko 2020-11-20T11:55:27.844 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2020-11-20T11:55:27.845 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/toshiba_haps.ko 2020-11-20T11:55:27.845 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/wmi-bmof.ko 2020-11-20T11:55:27.845 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/wmi.ko 2020-11-20T11:55:27.845 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/power 2020-11-20T11:55:27.845 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/power/supply 2020-11-20T11:55:27.846 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/power/supply/smb347-charger.ko 2020-11-20T11:55:27.846 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/powercap 2020-11-20T11:55:27.846 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/powercap/intel_rapl_common.ko 2020-11-20T11:55:27.846 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/powercap/intel_rapl_msr.ko 2020-11-20T11:55:27.846 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps 2020-11-20T11:55:27.847 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/clients 2020-11-20T11:55:27.847 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/clients/pps-gpio.ko 2020-11-20T11:55:27.847 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/clients/pps-ldisc.ko 2020-11-20T11:55:27.847 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/clients/pps_parport.ko 2020-11-20T11:55:27.847 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/pps_core.ko 2020-11-20T11:55:27.848 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ptp 2020-11-20T11:55:27.848 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ptp/ptp.ko 2020-11-20T11:55:27.848 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ptp/ptp_kvm.ko 2020-11-20T11:55:27.848 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc 2020-11-20T11:55:27.848 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-bq32k.ko 2020-11-20T11:55:27.849 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-bq4802.ko 2020-11-20T11:55:27.849 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1286.ko 2020-11-20T11:55:27.849 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1307.ko 2020-11-20T11:55:27.849 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1374.ko 2020-11-20T11:55:27.849 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1511.ko 2020-11-20T11:55:27.850 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1553.ko 2020-11-20T11:55:27.850 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1672.ko 2020-11-20T11:55:27.850 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1742.ko 2020-11-20T11:55:27.850 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds2404.ko 2020-11-20T11:55:27.850 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds3232.ko 2020-11-20T11:55:27.851 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-em3027.ko 2020-11-20T11:55:27.851 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-fm3130.ko 2020-11-20T11:55:27.851 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-isl12022.ko 2020-11-20T11:55:27.851 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-isl1208.ko 2020-11-20T11:55:27.851 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-m41t80.ko 2020-11-20T11:55:27.852 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-m48t35.ko 2020-11-20T11:55:27.852 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-m48t59.ko 2020-11-20T11:55:27.852 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-max6900.ko 2020-11-20T11:55:27.852 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-msm6242.ko 2020-11-20T11:55:27.852 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf2127.ko 2020-11-20T11:55:27.853 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf85063.ko 2020-11-20T11:55:27.853 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf8523.ko 2020-11-20T11:55:27.853 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf8563.ko 2020-11-20T11:55:27.853 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf8583.ko 2020-11-20T11:55:27.853 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rp5c01.ko 2020-11-20T11:55:27.854 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rs5c372.ko 2020-11-20T11:55:27.854 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rv3029c2.ko 2020-11-20T11:55:27.854 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rx8025.ko 2020-11-20T11:55:27.854 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rx8581.ko 2020-11-20T11:55:27.854 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-stk17ta8.ko 2020-11-20T11:55:27.855 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-v3020.ko 2020-11-20T11:55:27.855 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-x1205.ko 2020-11-20T11:55:27.855 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi 2020-11-20T11:55:27.855 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/3w-9xxx.ko 2020-11-20T11:55:27.855 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/3w-sas.ko 2020-11-20T11:55:27.856 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/3w-xxxx.ko 2020-11-20T11:55:27.856 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/BusLogic.ko 2020-11-20T11:55:27.856 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/a100u2w.ko 2020-11-20T11:55:27.856 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aacraid 2020-11-20T11:55:27.856 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aacraid/aacraid.ko 2020-11-20T11:55:27.857 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/advansys.ko 2020-11-20T11:55:27.857 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aic7xxx 2020-11-20T11:55:27.857 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2020-11-20T11:55:27.857 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2020-11-20T11:55:27.857 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/am53c974.ko 2020-11-20T11:55:27.858 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/arcmsr 2020-11-20T11:55:27.858 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/arcmsr/arcmsr.ko 2020-11-20T11:55:27.858 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/atp870u.ko 2020-11-20T11:55:27.858 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/be2iscsi 2020-11-20T11:55:27.858 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2020-11-20T11:55:27.859 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bfa 2020-11-20T11:55:27.859 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bfa/bfa.ko 2020-11-20T11:55:27.859 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bnx2fc 2020-11-20T11:55:27.859 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2020-11-20T11:55:27.859 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bnx2i 2020-11-20T11:55:27.860 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bnx2i/bnx2i.ko 2020-11-20T11:55:27.860 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ch.ko 2020-11-20T11:55:27.860 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/csiostor 2020-11-20T11:55:27.860 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/csiostor/csiostor.ko 2020-11-20T11:55:27.860 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi 2020-11-20T11:55:27.861 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/cxgb3i 2020-11-20T11:55:27.861 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2020-11-20T11:55:27.861 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/cxgb4i 2020-11-20T11:55:27.861 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2020-11-20T11:55:27.861 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2020-11-20T11:55:27.862 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/dc395x.ko 2020-11-20T11:55:27.862 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler 2020-11-20T11:55:27.862 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2020-11-20T11:55:27.862 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2020-11-20T11:55:27.862 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2020-11-20T11:55:27.863 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2020-11-20T11:55:27.863 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/dmx3191d.ko 2020-11-20T11:55:27.863 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/esas2r 2020-11-20T11:55:27.863 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/esas2r/esas2r.ko 2020-11-20T11:55:27.863 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/esp_scsi.ko 2020-11-20T11:55:27.864 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fcoe 2020-11-20T11:55:27.864 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fcoe/fcoe.ko 2020-11-20T11:55:27.864 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fcoe/libfcoe.ko 2020-11-20T11:55:27.864 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fnic 2020-11-20T11:55:27.864 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fnic/fnic.ko 2020-11-20T11:55:27.865 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/gdth.ko 2020-11-20T11:55:27.865 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/hpsa.ko 2020-11-20T11:55:27.865 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/hptiop.ko 2020-11-20T11:55:27.865 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/hv_storvsc.ko 2020-11-20T11:55:27.865 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/initio.ko 2020-11-20T11:55:27.866 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ipr.ko 2020-11-20T11:55:27.866 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ips.ko 2020-11-20T11:55:27.866 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/isci 2020-11-20T11:55:27.866 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/isci/isci.ko 2020-11-20T11:55:27.866 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2020-11-20T11:55:27.867 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/iscsi_tcp.ko 2020-11-20T11:55:27.867 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libfc 2020-11-20T11:55:27.867 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libfc/libfc.ko 2020-11-20T11:55:27.867 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libiscsi.ko 2020-11-20T11:55:27.867 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libiscsi_tcp.ko 2020-11-20T11:55:27.868 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libsas 2020-11-20T11:55:27.868 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libsas/libsas.ko 2020-11-20T11:55:27.868 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/lpfc 2020-11-20T11:55:27.868 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/lpfc/lpfc.ko 2020-11-20T11:55:27.868 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid 2020-11-20T11:55:27.869 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid.ko 2020-11-20T11:55:27.869 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2020-11-20T11:55:27.869 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2020-11-20T11:55:27.869 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2020-11-20T11:55:27.869 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mpt3sas 2020-11-20T11:55:27.870 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2020-11-20T11:55:27.870 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mvsas 2020-11-20T11:55:27.870 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mvsas/mvsas.ko 2020-11-20T11:55:27.870 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mvumi.ko 2020-11-20T11:55:27.870 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/pm8001 2020-11-20T11:55:27.871 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/pm8001/pm80xx.ko 2020-11-20T11:55:27.871 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/pmcraid.ko 2020-11-20T11:55:27.871 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla1280.ko 2020-11-20T11:55:27.871 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla2xxx 2020-11-20T11:55:27.871 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2020-11-20T11:55:27.872 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2020-11-20T11:55:27.872 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla4xxx 2020-11-20T11:55:27.872 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2020-11-20T11:55:27.872 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/raid_class.ko 2020-11-20T11:55:27.872 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_debug.ko 2020-11-20T11:55:27.873 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_fc.ko 2020-11-20T11:55:27.873 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_iscsi.ko 2020-11-20T11:55:27.873 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_sas.ko 2020-11-20T11:55:27.873 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_spi.ko 2020-11-20T11:55:27.873 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_srp.ko 2020-11-20T11:55:27.874 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ses.ko 2020-11-20T11:55:27.874 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/st.ko 2020-11-20T11:55:27.874 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/stex.ko 2020-11-20T11:55:27.874 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/sym53c8xx_2 2020-11-20T11:55:27.874 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2020-11-20T11:55:27.875 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ufs 2020-11-20T11:55:27.875 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ufs/ufshcd-core.ko 2020-11-20T11:55:27.875 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ufs/ufshcd-pci.ko 2020-11-20T11:55:27.875 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/virtio_scsi.ko 2020-11-20T11:55:27.875 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/vmw_pvscsi.ko 2020-11-20T11:55:27.876 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/wd719x.ko 2020-11-20T11:55:27.876 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ssb 2020-11-20T11:55:27.876 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ssb/ssb.ko 2020-11-20T11:55:27.876 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target 2020-11-20T11:55:27.876 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/iscsi 2020-11-20T11:55:27.877 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2020-11-20T11:55:27.877 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/loopback 2020-11-20T11:55:27.877 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/loopback/tcm_loop.ko 2020-11-20T11:55:27.877 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_file.ko 2020-11-20T11:55:27.877 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_iblock.ko 2020-11-20T11:55:27.878 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_mod.ko 2020-11-20T11:55:27.878 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_pscsi.ko 2020-11-20T11:55:27.878 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_user.ko 2020-11-20T11:55:27.878 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/tcm_fc 2020-11-20T11:55:27.878 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/tcm_fc/tcm_fc.ko 2020-11-20T11:55:27.879 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal 2020-11-20T11:55:27.879 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel 2020-11-20T11:55:27.879 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal 2020-11-20T11:55:27.879 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2020-11-20T11:55:27.879 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2020-11-20T11:55:27.880 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2020-11-20T11:55:27.880 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2020-11-20T11:55:27.880 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2020-11-20T11:55:27.880 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2020-11-20T11:55:27.880 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2020-11-20T11:55:27.881 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2020-11-20T11:55:27.881 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2020-11-20T11:55:27.881 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty 2020-11-20T11:55:27.881 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/cyclades.ko 2020-11-20T11:55:27.881 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/ipwireless 2020-11-20T11:55:27.882 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/ipwireless/ipwireless.ko 2020-11-20T11:55:27.882 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/n_gsm.ko 2020-11-20T11:55:27.882 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/n_hdlc.ko 2020-11-20T11:55:27.882 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/nozomi.ko 2020-11-20T11:55:27.882 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/rocket.ko 2020-11-20T11:55:27.883 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial 2020-11-20T11:55:27.883 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/8250 2020-11-20T11:55:27.883 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/8250/serial_cs.ko 2020-11-20T11:55:27.883 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/arc_uart.ko 2020-11-20T11:55:27.883 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/jsm 2020-11-20T11:55:27.884 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/jsm/jsm.ko 2020-11-20T11:55:27.884 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/synclink.ko 2020-11-20T11:55:27.884 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/synclink_gt.ko 2020-11-20T11:55:27.884 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/synclinkmp.ko 2020-11-20T11:55:27.884 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio 2020-11-20T11:55:27.885 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio.ko 2020-11-20T11:55:27.885 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio_aec.ko 2020-11-20T11:55:27.885 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio_cif.ko 2020-11-20T11:55:27.885 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio_pci_generic.ko 2020-11-20T11:55:27.885 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio_sercos3.ko 2020-11-20T11:55:27.886 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb 2020-11-20T11:55:27.886 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class 2020-11-20T11:55:27.886 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class/cdc-acm.ko 2020-11-20T11:55:27.886 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class/cdc-wdm.ko 2020-11-20T11:55:27.886 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class/usblp.ko 2020-11-20T11:55:27.887 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class/usbtmc.ko 2020-11-20T11:55:27.887 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/host 2020-11-20T11:55:27.887 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/host/sl811-hcd.ko 2020-11-20T11:55:27.887 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/host/u132-hcd.ko 2020-11-20T11:55:27.887 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/image 2020-11-20T11:55:27.888 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/image/mdc800.ko 2020-11-20T11:55:27.888 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/image/microtek.ko 2020-11-20T11:55:27.888 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc 2020-11-20T11:55:27.888 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/adutux.ko 2020-11-20T11:55:27.888 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/appledisplay.ko 2020-11-20T11:55:27.889 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/emi26.ko 2020-11-20T11:55:27.889 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/emi62.ko 2020-11-20T11:55:27.889 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/ezusb.ko 2020-11-20T11:55:27.889 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/ftdi-elan.ko 2020-11-20T11:55:27.889 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/idmouse.ko 2020-11-20T11:55:27.890 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/iowarrior.ko 2020-11-20T11:55:27.890 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/isight_firmware.ko 2020-11-20T11:55:27.890 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/ldusb.ko 2020-11-20T11:55:27.890 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/legousbtower.ko 2020-11-20T11:55:27.890 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/sisusbvga 2020-11-20T11:55:27.891 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2020-11-20T11:55:27.891 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/trancevibrator.ko 2020-11-20T11:55:27.891 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/usb3503.ko 2020-11-20T11:55:27.891 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/usblcd.ko 2020-11-20T11:55:27.891 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/usbsevseg.ko 2020-11-20T11:55:27.892 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/uss720.ko 2020-11-20T11:55:27.892 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/yurex.ko 2020-11-20T11:55:27.892 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/phy 2020-11-20T11:55:27.892 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/phy/phy-generic.ko 2020-11-20T11:55:27.892 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial 2020-11-20T11:55:27.893 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/aircable.ko 2020-11-20T11:55:27.893 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ark3116.ko 2020-11-20T11:55:27.893 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/belkin_sa.ko 2020-11-20T11:55:27.893 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ch341.ko 2020-11-20T11:55:27.893 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/cp210x.ko 2020-11-20T11:55:27.894 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/cyberjack.ko 2020-11-20T11:55:27.894 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/cypress_m8.ko 2020-11-20T11:55:27.894 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/digi_acceleport.ko 2020-11-20T11:55:27.894 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/empeg.ko 2020-11-20T11:55:27.894 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ftdi_sio.ko 2020-11-20T11:55:27.895 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/garmin_gps.ko 2020-11-20T11:55:27.895 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/io_edgeport.ko 2020-11-20T11:55:27.895 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/io_ti.ko 2020-11-20T11:55:27.895 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ipaq.ko 2020-11-20T11:55:27.895 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ipw.ko 2020-11-20T11:55:27.896 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ir-usb.ko 2020-11-20T11:55:27.896 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/iuu_phoenix.ko 2020-11-20T11:55:27.896 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/keyspan.ko 2020-11-20T11:55:27.896 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/keyspan_pda.ko 2020-11-20T11:55:27.896 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/kl5kusb105.ko 2020-11-20T11:55:27.897 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/kobil_sct.ko 2020-11-20T11:55:27.897 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/mct_u232.ko 2020-11-20T11:55:27.897 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/mos7720.ko 2020-11-20T11:55:27.897 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/mos7840.ko 2020-11-20T11:55:27.898 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/navman.ko 2020-11-20T11:55:27.898 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/omninet.ko 2020-11-20T11:55:27.898 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/opticon.ko 2020-11-20T11:55:27.898 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/option.ko 2020-11-20T11:55:27.898 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/oti6858.ko 2020-11-20T11:55:27.899 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/pl2303.ko 2020-11-20T11:55:27.899 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/qcaux.ko 2020-11-20T11:55:27.899 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/qcserial.ko 2020-11-20T11:55:27.899 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/quatech2.ko 2020-11-20T11:55:27.899 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/safe_serial.ko 2020-11-20T11:55:27.900 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/sierra.ko 2020-11-20T11:55:27.900 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/spcp8x5.ko 2020-11-20T11:55:27.900 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ssu100.ko 2020-11-20T11:55:27.900 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/symbolserial.ko 2020-11-20T11:55:27.900 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2020-11-20T11:55:27.901 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/usb-serial-simple.ko 2020-11-20T11:55:27.901 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/usb_debug.ko 2020-11-20T11:55:27.901 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/usb_wwan.ko 2020-11-20T11:55:27.901 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/visor.ko 2020-11-20T11:55:27.901 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/whiteheat.ko 2020-11-20T11:55:27.902 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/xsens_mt.ko 2020-11-20T11:55:27.902 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage 2020-11-20T11:55:27.902 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/uas.ko 2020-11-20T11:55:27.902 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-alauda.ko 2020-11-20T11:55:27.902 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-cypress.ko 2020-11-20T11:55:27.903 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-datafab.ko 2020-11-20T11:55:27.903 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-eneub6250.ko 2020-11-20T11:55:27.903 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-freecom.ko 2020-11-20T11:55:27.903 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-isd200.ko 2020-11-20T11:55:27.903 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-jumpshot.ko 2020-11-20T11:55:27.904 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-karma.ko 2020-11-20T11:55:27.904 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-onetouch.ko 2020-11-20T11:55:27.904 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-realtek.ko 2020-11-20T11:55:27.904 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-sddr09.ko 2020-11-20T11:55:27.904 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-sddr55.ko 2020-11-20T11:55:27.905 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-usbat.ko 2020-11-20T11:55:27.905 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/usb-storage.ko 2020-11-20T11:55:27.905 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/usbip 2020-11-20T11:55:27.905 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/usbip/usbip-core.ko 2020-11-20T11:55:27.905 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/usbip/usbip-host.ko 2020-11-20T11:55:27.906 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/usbip/vhci-hcd.ko 2020-11-20T11:55:27.906 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio 2020-11-20T11:55:27.906 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/pci 2020-11-20T11:55:27.906 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/pci/vfio-pci.ko 2020-11-20T11:55:27.906 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/vfio.ko 2020-11-20T11:55:27.907 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/vfio_iommu_type1.ko 2020-11-20T11:55:27.907 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/vfio_virqfd.ko 2020-11-20T11:55:27.907 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost 2020-11-20T11:55:27.907 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost/vhost.ko 2020-11-20T11:55:27.907 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost/vhost_iotlb.ko 2020-11-20T11:55:27.908 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost/vhost_net.ko 2020-11-20T11:55:27.908 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost/vhost_scsi.ko 2020-11-20T11:55:27.908 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video 2020-11-20T11:55:27.908 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video/backlight 2020-11-20T11:55:27.908 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video/backlight/apple_bl.ko 2020-11-20T11:55:27.909 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video/backlight/lcd.ko 2020-11-20T11:55:27.909 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video/backlight/platform_lcd.ko 2020-11-20T11:55:27.909 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio 2020-11-20T11:55:27.909 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio.ko 2020-11-20T11:55:27.909 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio_balloon.ko 2020-11-20T11:55:27.910 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio_mmio.ko 2020-11-20T11:55:27.910 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio_pci.ko 2020-11-20T11:55:27.910 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio_ring.ko 2020-11-20T11:55:27.910 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1 2020-11-20T11:55:27.911 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/masters 2020-11-20T11:55:27.911 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/masters/ds1wm.ko 2020-11-20T11:55:27.911 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/masters/ds2482.ko 2020-11-20T11:55:27.911 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/masters/ds2490.ko 2020-11-20T11:55:27.911 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves 2020-11-20T11:55:27.912 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2406.ko 2020-11-20T11:55:27.912 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2408.ko 2020-11-20T11:55:27.912 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2413.ko 2020-11-20T11:55:27.912 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2423.ko 2020-11-20T11:55:27.912 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2431.ko 2020-11-20T11:55:27.913 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2433.ko 2020-11-20T11:55:27.913 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2780.ko 2020-11-20T11:55:27.913 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2781.ko 2020-11-20T11:55:27.913 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds28e04.ko 2020-11-20T11:55:27.913 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_smem.ko 2020-11-20T11:55:27.914 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_therm.ko 2020-11-20T11:55:27.914 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/wire.ko 2020-11-20T11:55:27.914 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog 2020-11-20T11:55:27.914 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/alim1535_wdt.ko 2020-11-20T11:55:27.914 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/alim7101_wdt.ko 2020-11-20T11:55:27.915 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/f71808e_wdt.ko 2020-11-20T11:55:27.915 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/hpwdt.ko 2020-11-20T11:55:27.915 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/i6300esb.ko 2020-11-20T11:55:27.915 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/iTCO_vendor_support.ko 2020-11-20T11:55:27.915 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/iTCO_wdt.ko 2020-11-20T11:55:27.916 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/ib700wdt.ko 2020-11-20T11:55:27.916 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/ibmasr.ko 2020-11-20T11:55:27.916 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/ie6xx_wdt.ko 2020-11-20T11:55:27.916 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/it8712f_wdt.ko 2020-11-20T11:55:27.916 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/it87_wdt.ko 2020-11-20T11:55:27.916 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/machzwd.ko 2020-11-20T11:55:27.917 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/nv_tco.ko 2020-11-20T11:55:27.917 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/pcwd_pci.ko 2020-11-20T11:55:27.917 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/pcwd_usb.ko 2020-11-20T11:55:27.917 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2020-11-20T11:55:27.917 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/sch311x_wdt.ko 2020-11-20T11:55:27.918 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/softdog.ko 2020-11-20T11:55:27.918 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/sp5100_tco.ko 2020-11-20T11:55:27.918 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/via_wdt.ko 2020-11-20T11:55:27.918 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/w83627hf_wdt.ko 2020-11-20T11:55:27.918 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/w83877f_wdt.ko 2020-11-20T11:55:27.919 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/w83977f_wdt.ko 2020-11-20T11:55:27.919 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/wdt_pci.ko 2020-11-20T11:55:27.919 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/xen_wdt.ko 2020-11-20T11:55:27.919 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen 2020-11-20T11:55:27.919 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-acpi-processor.ko 2020-11-20T11:55:27.920 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-evtchn.ko 2020-11-20T11:55:27.920 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-gntalloc.ko 2020-11-20T11:55:27.920 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-gntdev.ko 2020-11-20T11:55:27.920 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-pciback 2020-11-20T11:55:27.920 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2020-11-20T11:55:27.921 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-privcmd.ko 2020-11-20T11:55:27.921 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xenfs 2020-11-20T11:55:27.921 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xenfs/xenfs.ko 2020-11-20T11:55:27.921 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs 2020-11-20T11:55:27.921 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/9p 2020-11-20T11:55:27.922 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/9p/9p.ko 2020-11-20T11:55:27.922 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/affs 2020-11-20T11:55:27.922 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/affs/affs.ko 2020-11-20T11:55:27.922 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/befs 2020-11-20T11:55:27.922 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/befs/befs.ko 2020-11-20T11:55:27.923 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/binfmt_misc.ko 2020-11-20T11:55:27.923 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/btrfs 2020-11-20T11:55:27.923 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/btrfs/btrfs.ko 2020-11-20T11:55:27.923 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cachefiles 2020-11-20T11:55:27.923 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cachefiles/cachefiles.ko 2020-11-20T11:55:27.924 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ceph 2020-11-20T11:55:27.924 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ceph/ceph.ko 2020-11-20T11:55:27.924 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cifs 2020-11-20T11:55:27.924 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cifs/cifs.ko 2020-11-20T11:55:27.924 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/coda 2020-11-20T11:55:27.925 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/coda/coda.ko 2020-11-20T11:55:27.925 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cramfs 2020-11-20T11:55:27.925 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cramfs/cramfs.ko 2020-11-20T11:55:27.925 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/dlm 2020-11-20T11:55:27.925 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/dlm/dlm.ko 2020-11-20T11:55:27.926 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ecryptfs 2020-11-20T11:55:27.926 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ecryptfs/ecryptfs.ko 2020-11-20T11:55:27.926 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/f2fs 2020-11-20T11:55:27.926 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/f2fs/f2fs.ko 2020-11-20T11:55:27.926 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fat 2020-11-20T11:55:27.927 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fat/fat.ko 2020-11-20T11:55:27.927 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fat/msdos.ko 2020-11-20T11:55:27.927 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fat/vfat.ko 2020-11-20T11:55:27.927 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fscache 2020-11-20T11:55:27.927 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fscache/fscache.ko 2020-11-20T11:55:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fuse 2020-11-20T11:55:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fuse/cuse.ko 2020-11-20T11:55:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fuse/fuse.ko 2020-11-20T11:55:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/gfs2 2020-11-20T11:55:27.929 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/gfs2/gfs2.ko 2020-11-20T11:55:27.929 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/hfs 2020-11-20T11:55:27.929 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/hfs/hfs.ko 2020-11-20T11:55:27.929 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/hfsplus 2020-11-20T11:55:27.929 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/hfsplus/hfsplus.ko 2020-11-20T11:55:27.930 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/isofs 2020-11-20T11:55:27.930 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/isofs/isofs.ko 2020-11-20T11:55:27.930 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/jfs 2020-11-20T11:55:27.930 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/jfs/jfs.ko 2020-11-20T11:55:27.930 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/lockd 2020-11-20T11:55:27.931 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/lockd/lockd.ko 2020-11-20T11:55:27.931 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/minix 2020-11-20T11:55:27.931 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/minix/minix.ko 2020-11-20T11:55:27.931 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs 2020-11-20T11:55:27.931 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/blocklayout 2020-11-20T11:55:27.931 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2020-11-20T11:55:27.932 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/filelayout 2020-11-20T11:55:27.932 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2020-11-20T11:55:27.932 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/flexfilelayout 2020-11-20T11:55:27.932 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2020-11-20T11:55:27.932 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/nfs.ko 2020-11-20T11:55:27.933 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/nfsv3.ko 2020-11-20T11:55:27.933 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/nfsv4.ko 2020-11-20T11:55:27.933 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs_common 2020-11-20T11:55:27.933 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs_common/grace.ko 2020-11-20T11:55:27.934 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs_common/nfs_acl.ko 2020-11-20T11:55:27.934 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs_common/nfs_ssc.ko 2020-11-20T11:55:27.934 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfsd 2020-11-20T11:55:27.934 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfsd/nfsd.ko 2020-11-20T11:55:27.934 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nilfs2 2020-11-20T11:55:27.934 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nilfs2/nilfs2.ko 2020-11-20T11:55:27.935 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls 2020-11-20T11:55:27.935 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-celtic.ko 2020-11-20T11:55:27.935 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-centeuro.ko 2020-11-20T11:55:27.935 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-croatian.ko 2020-11-20T11:55:27.935 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-cyrillic.ko 2020-11-20T11:55:27.936 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-gaelic.ko 2020-11-20T11:55:27.936 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-greek.ko 2020-11-20T11:55:27.936 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-iceland.ko 2020-11-20T11:55:27.936 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-inuit.ko 2020-11-20T11:55:27.936 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-roman.ko 2020-11-20T11:55:27.937 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-romanian.ko 2020-11-20T11:55:27.937 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-turkish.ko 2020-11-20T11:55:27.937 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp1250.ko 2020-11-20T11:55:27.937 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp1251.ko 2020-11-20T11:55:27.937 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp1255.ko 2020-11-20T11:55:27.938 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp737.ko 2020-11-20T11:55:27.938 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp775.ko 2020-11-20T11:55:27.938 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp850.ko 2020-11-20T11:55:27.938 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp852.ko 2020-11-20T11:55:27.938 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp855.ko 2020-11-20T11:55:27.939 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp857.ko 2020-11-20T11:55:27.939 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp860.ko 2020-11-20T11:55:27.939 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp861.ko 2020-11-20T11:55:27.939 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp862.ko 2020-11-20T11:55:27.939 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp863.ko 2020-11-20T11:55:27.940 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp864.ko 2020-11-20T11:55:27.940 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp865.ko 2020-11-20T11:55:27.940 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp866.ko 2020-11-20T11:55:27.940 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp869.ko 2020-11-20T11:55:27.940 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp874.ko 2020-11-20T11:55:27.941 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp932.ko 2020-11-20T11:55:27.941 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp936.ko 2020-11-20T11:55:27.941 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp949.ko 2020-11-20T11:55:27.941 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp950.ko 2020-11-20T11:55:27.941 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_euc-jp.ko 2020-11-20T11:55:27.942 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-1.ko 2020-11-20T11:55:27.942 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-13.ko 2020-11-20T11:55:27.942 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-14.ko 2020-11-20T11:55:27.942 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-15.ko 2020-11-20T11:55:27.942 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-2.ko 2020-11-20T11:55:27.943 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-3.ko 2020-11-20T11:55:27.943 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-4.ko 2020-11-20T11:55:27.943 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-5.ko 2020-11-20T11:55:27.943 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-6.ko 2020-11-20T11:55:27.943 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-7.ko 2020-11-20T11:55:27.944 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-9.ko 2020-11-20T11:55:27.944 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_koi8-r.ko 2020-11-20T11:55:27.944 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_koi8-ru.ko 2020-11-20T11:55:27.944 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_koi8-u.ko 2020-11-20T11:55:27.944 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_utf8.ko 2020-11-20T11:55:27.944 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2 2020-11-20T11:55:27.945 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/cluster 2020-11-20T11:55:27.945 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2020-11-20T11:55:27.945 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/dlm 2020-11-20T11:55:27.945 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2020-11-20T11:55:27.946 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/dlmfs 2020-11-20T11:55:27.946 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2020-11-20T11:55:27.946 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/ocfs2.ko 2020-11-20T11:55:27.946 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2020-11-20T11:55:27.946 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/ocfs2_stack_user.ko 2020-11-20T11:55:27.947 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/ocfs2_stackglue.ko 2020-11-20T11:55:27.947 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/overlayfs 2020-11-20T11:55:27.947 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/overlayfs/overlay.ko 2020-11-20T11:55:27.947 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/pstore 2020-11-20T11:55:27.947 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/pstore/ramoops.ko 2020-11-20T11:55:27.947 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/reiserfs 2020-11-20T11:55:27.948 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/reiserfs/reiserfs.ko 2020-11-20T11:55:27.948 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/romfs 2020-11-20T11:55:27.948 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/romfs/romfs.ko 2020-11-20T11:55:27.948 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/squashfs 2020-11-20T11:55:27.948 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/squashfs/squashfs.ko 2020-11-20T11:55:27.949 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/sysv 2020-11-20T11:55:27.949 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/sysv/sysv.ko 2020-11-20T11:55:27.949 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/udf 2020-11-20T11:55:27.949 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/udf/udf.ko 2020-11-20T11:55:27.949 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ufs 2020-11-20T11:55:27.950 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ufs/ufs.ko 2020-11-20T11:55:27.950 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/xfs 2020-11-20T11:55:27.950 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/xfs/xfs.ko 2020-11-20T11:55:27.950 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel 2020-11-20T11:55:27.950 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/rcu 2020-11-20T11:55:27.951 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/rcu/rcutorture.ko 2020-11-20T11:55:27.951 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/torture.ko 2020-11-20T11:55:27.951 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/trace 2020-11-20T11:55:27.951 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/trace/ring_buffer_benchmark.ko 2020-11-20T11:55:27.951 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib 2020-11-20T11:55:27.952 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crc-itu-t.ko 2020-11-20T11:55:27.952 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crc64.ko 2020-11-20T11:55:27.952 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crc8.ko 2020-11-20T11:55:27.952 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crypto 2020-11-20T11:55:27.952 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crypto/libarc4.ko 2020-11-20T11:55:27.953 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crypto/libdes.ko 2020-11-20T11:55:27.953 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/libcrc32c.ko 2020-11-20T11:55:27.953 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/lru_cache.ko 2020-11-20T11:55:27.953 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/lz4 2020-11-20T11:55:27.953 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/lz4/lz4_compress.ko 2020-11-20T11:55:27.954 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/lz4/lz4hc_compress.ko 2020-11-20T11:55:27.954 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/math 2020-11-20T11:55:27.954 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/math/cordic.ko 2020-11-20T11:55:27.954 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/raid6 2020-11-20T11:55:27.954 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/raid6/raid6_pq.ko 2020-11-20T11:55:27.955 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/reed_solomon 2020-11-20T11:55:27.955 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/reed_solomon/reed_solomon.ko 2020-11-20T11:55:27.955 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/ts_bm.ko 2020-11-20T11:55:27.955 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/ts_fsm.ko 2020-11-20T11:55:27.955 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/ts_kmp.ko 2020-11-20T11:55:27.956 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/zstd 2020-11-20T11:55:27.956 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/zstd/zstd_compress.ko 2020-11-20T11:55:27.956 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/mm 2020-11-20T11:55:27.956 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/mm/hwpoison-inject.ko 2020-11-20T11:55:27.956 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net 2020-11-20T11:55:27.957 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan 2020-11-20T11:55:27.957 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/6lowpan.ko 2020-11-20T11:55:27.957 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_dest.ko 2020-11-20T11:55:27.957 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_fragment.ko 2020-11-20T11:55:27.957 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_hop.ko 2020-11-20T11:55:27.958 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_ipv6.ko 2020-11-20T11:55:27.958 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_mobility.ko 2020-11-20T11:55:27.958 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_routing.ko 2020-11-20T11:55:27.958 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_udp.ko 2020-11-20T11:55:27.958 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802 2020-11-20T11:55:27.959 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/garp.ko 2020-11-20T11:55:27.959 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/mrp.ko 2020-11-20T11:55:27.959 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/p8022.ko 2020-11-20T11:55:27.959 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/psnap.ko 2020-11-20T11:55:27.959 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/stp.ko 2020-11-20T11:55:27.960 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/8021q 2020-11-20T11:55:27.960 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/8021q/8021q.ko 2020-11-20T11:55:27.960 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/9p 2020-11-20T11:55:27.960 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/9p/9pnet.ko 2020-11-20T11:55:27.960 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/9p/9pnet_rdma.ko 2020-11-20T11:55:27.961 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/9p/9pnet_virtio.ko 2020-11-20T11:55:27.961 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/batman-adv 2020-11-20T11:55:27.961 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/batman-adv/batman-adv.ko 2020-11-20T11:55:27.961 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge 2020-11-20T11:55:27.961 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/br_netfilter.ko 2020-11-20T11:55:27.962 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/bridge.ko 2020-11-20T11:55:27.962 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter 2020-11-20T11:55:27.962 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_802_3.ko 2020-11-20T11:55:27.962 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_among.ko 2020-11-20T11:55:27.962 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_arp.ko 2020-11-20T11:55:27.963 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_arpreply.ko 2020-11-20T11:55:27.963 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_dnat.ko 2020-11-20T11:55:27.963 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_ip.ko 2020-11-20T11:55:27.963 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_ip6.ko 2020-11-20T11:55:27.963 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_limit.ko 2020-11-20T11:55:27.964 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_log.ko 2020-11-20T11:55:27.964 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_mark.ko 2020-11-20T11:55:27.964 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_mark_m.ko 2020-11-20T11:55:27.964 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_nflog.ko 2020-11-20T11:55:27.964 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_pkttype.ko 2020-11-20T11:55:27.964 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_redirect.ko 2020-11-20T11:55:27.965 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_snat.ko 2020-11-20T11:55:27.965 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_stp.ko 2020-11-20T11:55:27.965 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_vlan.ko 2020-11-20T11:55:27.965 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebtable_broute.ko 2020-11-20T11:55:27.965 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebtable_filter.ko 2020-11-20T11:55:27.966 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebtable_nat.ko 2020-11-20T11:55:27.966 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebtables.ko 2020-11-20T11:55:27.966 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2020-11-20T11:55:27.966 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/nf_log_bridge.ko 2020-11-20T11:55:27.966 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2020-11-20T11:55:27.967 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2020-11-20T11:55:27.967 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ceph 2020-11-20T11:55:27.967 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ceph/libceph.ko 2020-11-20T11:55:27.967 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/core 2020-11-20T11:55:27.967 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/core/failover.ko 2020-11-20T11:55:27.968 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/core/pktgen.ko 2020-11-20T11:55:27.968 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp 2020-11-20T11:55:27.968 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp/dccp.ko 2020-11-20T11:55:27.968 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp/dccp_diag.ko 2020-11-20T11:55:27.968 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp/dccp_ipv4.ko 2020-11-20T11:55:27.969 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp/dccp_ipv6.ko 2020-11-20T11:55:27.969 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dns_resolver 2020-11-20T11:55:27.969 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dns_resolver/dns_resolver.ko 2020-11-20T11:55:27.969 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa 2020-11-20T11:55:27.969 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/dsa_core.ko 2020-11-20T11:55:27.970 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/tag_brcm.ko 2020-11-20T11:55:27.970 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/tag_dsa.ko 2020-11-20T11:55:27.970 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/tag_edsa.ko 2020-11-20T11:55:27.970 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/tag_trailer.ko 2020-11-20T11:55:27.970 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154 2020-11-20T11:55:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154/6lowpan 2020-11-20T11:55:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2020-11-20T11:55:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154/ieee802154.ko 2020-11-20T11:55:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154/ieee802154_socket.ko 2020-11-20T11:55:27.971 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4 2020-11-20T11:55:27.972 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ah4.ko 2020-11-20T11:55:27.972 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/esp4.ko 2020-11-20T11:55:27.972 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/fou.ko 2020-11-20T11:55:27.972 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/gre.ko 2020-11-20T11:55:27.972 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/inet_diag.ko 2020-11-20T11:55:27.973 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ip_gre.ko 2020-11-20T11:55:27.973 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ip_tunnel.ko 2020-11-20T11:55:27.973 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ip_vti.ko 2020-11-20T11:55:27.973 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ipcomp.ko 2020-11-20T11:55:27.973 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ipip.ko 2020-11-20T11:55:27.974 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter 2020-11-20T11:55:27.974 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/arp_tables.ko 2020-11-20T11:55:27.974 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/arpt_mangle.ko 2020-11-20T11:55:27.974 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/arptable_filter.ko 2020-11-20T11:55:27.974 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2020-11-20T11:55:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_ECN.ko 2020-11-20T11:55:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2020-11-20T11:55:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_ah.ko 2020-11-20T11:55:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2020-11-20T11:55:27.975 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/iptable_mangle.ko 2020-11-20T11:55:27.976 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/iptable_nat.ko 2020-11-20T11:55:27.976 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/iptable_raw.ko 2020-11-20T11:55:27.976 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/iptable_security.ko 2020-11-20T11:55:27.976 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2020-11-20T11:55:27.976 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2020-11-20T11:55:27.977 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_log_arp.ko 2020-11-20T11:55:27.977 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_log_ipv4.ko 2020-11-20T11:55:27.977 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2020-11-20T11:55:27.977 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2020-11-20T11:55:27.977 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2020-11-20T11:55:27.978 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2020-11-20T11:55:27.978 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2020-11-20T11:55:27.978 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2020-11-20T11:55:27.978 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2020-11-20T11:55:27.978 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2020-11-20T11:55:27.979 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_bic.ko 2020-11-20T11:55:27.979 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_dctcp.ko 2020-11-20T11:55:27.979 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_diag.ko 2020-11-20T11:55:27.979 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_highspeed.ko 2020-11-20T11:55:27.979 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_htcp.ko 2020-11-20T11:55:27.980 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_hybla.ko 2020-11-20T11:55:27.980 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_illinois.ko 2020-11-20T11:55:27.980 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_lp.ko 2020-11-20T11:55:27.980 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_scalable.ko 2020-11-20T11:55:27.980 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_vegas.ko 2020-11-20T11:55:27.981 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_veno.ko 2020-11-20T11:55:27.981 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_westwood.ko 2020-11-20T11:55:27.981 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_yeah.ko 2020-11-20T11:55:27.981 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tunnel4.ko 2020-11-20T11:55:27.981 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/udp_diag.ko 2020-11-20T11:55:27.982 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/udp_tunnel.ko 2020-11-20T11:55:27.982 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/xfrm4_tunnel.ko 2020-11-20T11:55:27.982 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6 2020-11-20T11:55:27.982 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ah6.ko 2020-11-20T11:55:27.982 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/esp6.ko 2020-11-20T11:55:27.983 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/fou6.ko 2020-11-20T11:55:27.983 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ip6_tunnel.ko 2020-11-20T11:55:27.983 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ip6_udp_tunnel.ko 2020-11-20T11:55:27.983 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ip6_vti.ko 2020-11-20T11:55:27.983 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ipcomp6.ko 2020-11-20T11:55:27.984 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter 2020-11-20T11:55:27.984 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6_tables.ko 2020-11-20T11:55:27.984 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2020-11-20T11:55:27.984 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2020-11-20T11:55:27.984 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_ah.ko 2020-11-20T11:55:27.985 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2020-11-20T11:55:27.985 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_frag.ko 2020-11-20T11:55:27.985 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2020-11-20T11:55:27.985 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2020-11-20T11:55:27.985 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_mh.ko 2020-11-20T11:55:27.986 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2020-11-20T11:55:27.986 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_rt.ko 2020-11-20T11:55:27.986 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_filter.ko 2020-11-20T11:55:27.986 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2020-11-20T11:55:27.986 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_nat.ko 2020-11-20T11:55:27.987 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_raw.ko 2020-11-20T11:55:27.987 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_security.ko 2020-11-20T11:55:27.987 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2020-11-20T11:55:27.987 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2020-11-20T11:55:27.987 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_flow_table_ipv6.ko 2020-11-20T11:55:27.988 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_log_ipv6.ko 2020-11-20T11:55:27.988 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2020-11-20T11:55:27.988 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2020-11-20T11:55:27.988 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2020-11-20T11:55:27.988 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2020-11-20T11:55:27.989 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2020-11-20T11:55:27.989 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2020-11-20T11:55:27.989 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/sit.ko 2020-11-20T11:55:27.989 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/tunnel6.ko 2020-11-20T11:55:27.989 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/xfrm6_tunnel.ko 2020-11-20T11:55:27.990 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/key 2020-11-20T11:55:27.990 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/key/af_key.ko 2020-11-20T11:55:27.990 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp 2020-11-20T11:55:27.990 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_core.ko 2020-11-20T11:55:27.990 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_debugfs.ko 2020-11-20T11:55:27.991 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_eth.ko 2020-11-20T11:55:27.991 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_ip.ko 2020-11-20T11:55:27.991 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_ip6.ko 2020-11-20T11:55:27.991 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_netlink.ko 2020-11-20T11:55:27.991 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_ppp.ko 2020-11-20T11:55:27.992 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/llc 2020-11-20T11:55:27.992 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/llc/llc.ko 2020-11-20T11:55:27.992 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/mac802154 2020-11-20T11:55:27.992 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/mac802154/mac802154.ko 2020-11-20T11:55:27.992 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/mpls 2020-11-20T11:55:27.993 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/mpls/mpls_gso.ko 2020-11-20T11:55:27.993 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter 2020-11-20T11:55:27.993 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset 2020-11-20T11:55:27.993 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set.ko 2020-11-20T11:55:27.993 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2020-11-20T11:55:27.994 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2020-11-20T11:55:27.994 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2020-11-20T11:55:27.994 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2020-11-20T11:55:27.994 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2020-11-20T11:55:27.994 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2020-11-20T11:55:27.995 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2020-11-20T11:55:27.995 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2020-11-20T11:55:27.995 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2020-11-20T11:55:27.995 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2020-11-20T11:55:27.995 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2020-11-20T11:55:27.996 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2020-11-20T11:55:27.996 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2020-11-20T11:55:27.996 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2020-11-20T11:55:27.996 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_list_set.ko 2020-11-20T11:55:27.996 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs 2020-11-20T11:55:27.997 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs.ko 2020-11-20T11:55:27.997 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2020-11-20T11:55:27.997 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2020-11-20T11:55:27.997 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2020-11-20T11:55:27.997 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2020-11-20T11:55:27.998 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2020-11-20T11:55:27.998 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2020-11-20T11:55:27.998 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2020-11-20T11:55:27.998 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2020-11-20T11:55:27.998 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2020-11-20T11:55:27.999 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2020-11-20T11:55:27.999 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2020-11-20T11:55:27.999 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2020-11-20T11:55:27.999 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2020-11-20T11:55:27.999 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conncount.ko 2020-11-20T11:55:28.000 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack.ko 2020-11-20T11:55:28.000 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_amanda.ko 2020-11-20T11:55:28.000 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_broadcast.ko 2020-11-20T11:55:28.000 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_ftp.ko 2020-11-20T11:55:28.000 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_h323.ko 2020-11-20T11:55:28.001 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_irc.ko 2020-11-20T11:55:28.001 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2020-11-20T11:55:28.001 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_netlink.ko 2020-11-20T11:55:28.001 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_pptp.ko 2020-11-20T11:55:28.001 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_sane.ko 2020-11-20T11:55:28.002 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_sip.ko 2020-11-20T11:55:28.002 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_snmp.ko 2020-11-20T11:55:28.002 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_tftp.ko 2020-11-20T11:55:28.002 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_dup_netdev.ko 2020-11-20T11:55:28.002 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_flow_table.ko 2020-11-20T11:55:28.003 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_flow_table_inet.ko 2020-11-20T11:55:28.003 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_log_common.ko 2020-11-20T11:55:28.003 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat.ko 2020-11-20T11:55:28.003 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_amanda.ko 2020-11-20T11:55:28.003 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_ftp.ko 2020-11-20T11:55:28.004 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_irc.ko 2020-11-20T11:55:28.004 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_sip.ko 2020-11-20T11:55:28.004 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_tftp.ko 2020-11-20T11:55:28.004 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_synproxy_core.ko 2020-11-20T11:55:28.004 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_tables.ko 2020-11-20T11:55:28.005 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink.ko 2020-11-20T11:55:28.005 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink_acct.ko 2020-11-20T11:55:28.005 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink_log.ko 2020-11-20T11:55:28.005 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink_osf.ko 2020-11-20T11:55:28.005 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink_queue.ko 2020-11-20T11:55:28.006 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_chain_nat.ko 2020-11-20T11:55:28.006 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_compat.ko 2020-11-20T11:55:28.006 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_counter.ko 2020-11-20T11:55:28.006 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_ct.ko 2020-11-20T11:55:28.006 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_dup_netdev.ko 2020-11-20T11:55:28.007 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_fib.ko 2020-11-20T11:55:28.007 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_fwd_netdev.ko 2020-11-20T11:55:28.007 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_hash.ko 2020-11-20T11:55:28.007 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_limit.ko 2020-11-20T11:55:28.007 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_log.ko 2020-11-20T11:55:28.008 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_masq.ko 2020-11-20T11:55:28.008 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_nat.ko 2020-11-20T11:55:28.008 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_numgen.ko 2020-11-20T11:55:28.008 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_objref.ko 2020-11-20T11:55:28.008 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_queue.ko 2020-11-20T11:55:28.008 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_quota.ko 2020-11-20T11:55:28.009 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_redir.ko 2020-11-20T11:55:28.009 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_reject.ko 2020-11-20T11:55:28.009 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_reject_inet.ko 2020-11-20T11:55:28.009 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_socket.ko 2020-11-20T11:55:28.010 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_tunnel.ko 2020-11-20T11:55:28.010 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_AUDIT.ko 2020-11-20T11:55:28.010 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_CHECKSUM.ko 2020-11-20T11:55:28.010 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_CLASSIFY.ko 2020-11-20T11:55:28.010 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_CONNSECMARK.ko 2020-11-20T11:55:28.011 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_CT.ko 2020-11-20T11:55:28.011 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_DSCP.ko 2020-11-20T11:55:28.011 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_HL.ko 2020-11-20T11:55:28.011 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_HMARK.ko 2020-11-20T11:55:28.011 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_IDLETIMER.ko 2020-11-20T11:55:28.012 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_LED.ko 2020-11-20T11:55:28.012 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_LOG.ko 2020-11-20T11:55:28.012 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_MASQUERADE.ko 2020-11-20T11:55:28.012 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_NETMAP.ko 2020-11-20T11:55:28.012 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_NFLOG.ko 2020-11-20T11:55:28.013 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_NFQUEUE.ko 2020-11-20T11:55:28.013 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_RATEEST.ko 2020-11-20T11:55:28.013 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_REDIRECT.ko 2020-11-20T11:55:28.013 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_SECMARK.ko 2020-11-20T11:55:28.013 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TCPMSS.ko 2020-11-20T11:55:28.014 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2020-11-20T11:55:28.014 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TEE.ko 2020-11-20T11:55:28.014 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TPROXY.ko 2020-11-20T11:55:28.014 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TRACE.ko 2020-11-20T11:55:28.014 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_addrtype.ko 2020-11-20T11:55:28.014 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_bpf.ko 2020-11-20T11:55:28.015 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_cgroup.ko 2020-11-20T11:55:28.015 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_cluster.ko 2020-11-20T11:55:28.015 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_comment.ko 2020-11-20T11:55:28.015 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_connbytes.ko 2020-11-20T11:55:28.015 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_connlabel.ko 2020-11-20T11:55:28.016 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_connlimit.ko 2020-11-20T11:55:28.016 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_connmark.ko 2020-11-20T11:55:28.016 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_conntrack.ko 2020-11-20T11:55:28.016 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_cpu.ko 2020-11-20T11:55:28.016 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_dccp.ko 2020-11-20T11:55:28.017 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_devgroup.ko 2020-11-20T11:55:28.017 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_dscp.ko 2020-11-20T11:55:28.017 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_ecn.ko 2020-11-20T11:55:28.017 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_esp.ko 2020-11-20T11:55:28.017 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_hashlimit.ko 2020-11-20T11:55:28.018 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_helper.ko 2020-11-20T11:55:28.018 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_hl.ko 2020-11-20T11:55:28.018 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_ipcomp.ko 2020-11-20T11:55:28.018 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_iprange.ko 2020-11-20T11:55:28.018 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_ipvs.ko 2020-11-20T11:55:28.019 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_l2tp.ko 2020-11-20T11:55:28.019 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_length.ko 2020-11-20T11:55:28.019 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_limit.ko 2020-11-20T11:55:28.019 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_mac.ko 2020-11-20T11:55:28.019 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_mark.ko 2020-11-20T11:55:28.020 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_multiport.ko 2020-11-20T11:55:28.020 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_nat.ko 2020-11-20T11:55:28.020 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_nfacct.ko 2020-11-20T11:55:28.020 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_osf.ko 2020-11-20T11:55:28.020 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_owner.ko 2020-11-20T11:55:28.021 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_physdev.ko 2020-11-20T11:55:28.021 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_pkttype.ko 2020-11-20T11:55:28.021 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_policy.ko 2020-11-20T11:55:28.021 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_quota.ko 2020-11-20T11:55:28.021 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_rateest.ko 2020-11-20T11:55:28.022 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_realm.ko 2020-11-20T11:55:28.022 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_recent.ko 2020-11-20T11:55:28.022 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_sctp.ko 2020-11-20T11:55:28.022 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_set.ko 2020-11-20T11:55:28.022 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_socket.ko 2020-11-20T11:55:28.023 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_state.ko 2020-11-20T11:55:28.023 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_statistic.ko 2020-11-20T11:55:28.023 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_string.ko 2020-11-20T11:55:28.023 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_tcpmss.ko 2020-11-20T11:55:28.023 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_time.ko 2020-11-20T11:55:28.024 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_u32.ko 2020-11-20T11:55:28.024 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netlink 2020-11-20T11:55:28.024 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netlink/netlink_diag.ko 2020-11-20T11:55:28.024 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/nsh 2020-11-20T11:55:28.024 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/nsh/nsh.ko 2020-11-20T11:55:28.025 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch 2020-11-20T11:55:28.025 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch/openvswitch.ko 2020-11-20T11:55:28.025 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch/vport-geneve.ko 2020-11-20T11:55:28.025 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch/vport-gre.ko 2020-11-20T11:55:28.025 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch/vport-vxlan.ko 2020-11-20T11:55:28.026 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/packet 2020-11-20T11:55:28.026 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/packet/af_packet_diag.ko 2020-11-20T11:55:28.026 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/rds 2020-11-20T11:55:28.026 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/rds/rds.ko 2020-11-20T11:55:28.026 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/rds/rds_rdma.ko 2020-11-20T11:55:28.027 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/rds/rds_tcp.ko 2020-11-20T11:55:28.027 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched 2020-11-20T11:55:28.027 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_csum.ko 2020-11-20T11:55:28.027 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_gact.ko 2020-11-20T11:55:28.027 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_ipt.ko 2020-11-20T11:55:28.028 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_mirred.ko 2020-11-20T11:55:28.028 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_nat.ko 2020-11-20T11:55:28.028 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_pedit.ko 2020-11-20T11:55:28.028 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_police.ko 2020-11-20T11:55:28.028 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_simple.ko 2020-11-20T11:55:28.029 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_skbedit.ko 2020-11-20T11:55:28.029 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_vlan.ko 2020-11-20T11:55:28.029 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_basic.ko 2020-11-20T11:55:28.029 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_bpf.ko 2020-11-20T11:55:28.029 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_flow.ko 2020-11-20T11:55:28.030 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_fw.ko 2020-11-20T11:55:28.030 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_route.ko 2020-11-20T11:55:28.030 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_rsvp.ko 2020-11-20T11:55:28.030 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_rsvp6.ko 2020-11-20T11:55:28.030 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_tcindex.ko 2020-11-20T11:55:28.031 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_u32.ko 2020-11-20T11:55:28.031 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_cmp.ko 2020-11-20T11:55:28.031 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_ipset.ko 2020-11-20T11:55:28.031 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_meta.ko 2020-11-20T11:55:28.031 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_nbyte.ko 2020-11-20T11:55:28.032 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_text.ko 2020-11-20T11:55:28.032 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_u32.ko 2020-11-20T11:55:28.032 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_cbq.ko 2020-11-20T11:55:28.032 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_choke.ko 2020-11-20T11:55:28.032 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_codel.ko 2020-11-20T11:55:28.033 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_drr.ko 2020-11-20T11:55:28.033 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_dsmark.ko 2020-11-20T11:55:28.033 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_fq.ko 2020-11-20T11:55:28.033 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_gred.ko 2020-11-20T11:55:28.033 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_hfsc.ko 2020-11-20T11:55:28.034 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_hhf.ko 2020-11-20T11:55:28.034 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_htb.ko 2020-11-20T11:55:28.034 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_ingress.ko 2020-11-20T11:55:28.034 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_mqprio.ko 2020-11-20T11:55:28.034 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_multiq.ko 2020-11-20T11:55:28.035 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_netem.ko 2020-11-20T11:55:28.035 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_pie.ko 2020-11-20T11:55:28.035 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_plug.ko 2020-11-20T11:55:28.035 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_prio.ko 2020-11-20T11:55:28.035 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_qfq.ko 2020-11-20T11:55:28.036 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_red.ko 2020-11-20T11:55:28.036 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_sfb.ko 2020-11-20T11:55:28.036 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_sfq.ko 2020-11-20T11:55:28.036 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_tbf.ko 2020-11-20T11:55:28.036 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_teql.ko 2020-11-20T11:55:28.037 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sctp 2020-11-20T11:55:28.037 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sctp/sctp.ko 2020-11-20T11:55:28.037 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sctp/sctp_diag.ko 2020-11-20T11:55:28.037 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc 2020-11-20T11:55:28.037 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/auth_gss 2020-11-20T11:55:28.038 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2020-11-20T11:55:28.038 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2020-11-20T11:55:28.038 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/sunrpc.ko 2020-11-20T11:55:28.038 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/xprtrdma 2020-11-20T11:55:28.038 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2020-11-20T11:55:28.039 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/tipc 2020-11-20T11:55:28.039 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/tipc/diag.ko 2020-11-20T11:55:28.039 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/tipc/tipc.ko 2020-11-20T11:55:28.039 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/unix 2020-11-20T11:55:28.039 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/unix/unix_diag.ko 2020-11-20T11:55:28.040 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock 2020-11-20T11:55:28.040 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2020-11-20T11:55:28.040 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2020-11-20T11:55:28.040 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vsock.ko 2020-11-20T11:55:28.040 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vsock_diag.ko 2020-11-20T11:55:28.041 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vsock_loopback.ko 2020-11-20T11:55:28.041 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/xfrm 2020-11-20T11:55:28.041 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/xfrm/xfrm_ipcomp.ko 2020-11-20T11:55:28.041 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security 2020-11-20T11:55:28.041 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys 2020-11-20T11:55:28.042 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys/encrypted-keys 2020-11-20T11:55:28.042 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2020-11-20T11:55:28.042 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys/trusted-keys 2020-11-20T11:55:28.042 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys/trusted-keys/trusted.ko 2020-11-20T11:55:28.042 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/virt 2020-11-20T11:55:28.043 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/virt/lib 2020-11-20T11:55:28.043 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/virt/lib/irqbypass.ko 2020-11-20T11:55:28.043 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.alias 2020-11-20T11:55:28.043 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.alias.bin 2020-11-20T11:55:28.043 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.builtin 2020-11-20T11:55:28.044 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.builtin.bin 2020-11-20T11:55:28.044 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.builtin.modinfo 2020-11-20T11:55:28.044 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.dep 2020-11-20T11:55:28.044 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.dep.bin 2020-11-20T11:55:28.044 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.devname 2020-11-20T11:55:28.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.order 2020-11-20T11:55:28.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.softdep 2020-11-20T11:55:28.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.symbols 2020-11-20T11:55:28.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.symbols.bin 2020-11-20T11:55:28.046 DEBUG:teuthology.task.kernel:get_image_version: 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:28.046 INFO:teuthology.orchestra.run.smithi187:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2020-11-20T11:55:28.070 INFO:teuthology.orchestra.run.smithi187.stdout:postinstall scriptlet (using /bin/sh): 2020-11-20T11:55:28.071 INFO:teuthology.orchestra.run.smithi187.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403 -a -r /boot/System.map-5.10.0-rc2-ceph-gac4f7b07f403 ]; then 2020-11-20T11:55:28.071 INFO:teuthology.orchestra.run.smithi187.stdout:cp /boot/vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403 /boot/.vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403-rpm 2020-11-20T11:55:28.071 INFO:teuthology.orchestra.run.smithi187.stdout:cp /boot/System.map-5.10.0-rc2-ceph-gac4f7b07f403 /boot/.System.map-5.10.0-rc2-ceph-gac4f7b07f403-rpm 2020-11-20T11:55:28.071 INFO:teuthology.orchestra.run.smithi187.stdout:rm -f /boot/vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403 /boot/System.map-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:28.072 INFO:teuthology.orchestra.run.smithi187.stdout:/sbin/installkernel 5.10.0-rc2-ceph-gac4f7b07f403 /boot/.vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403-rpm /boot/.System.map-5.10.0-rc2-ceph-gac4f7b07f403-rpm 2020-11-20T11:55:28.072 INFO:teuthology.orchestra.run.smithi187.stdout:rm -f /boot/.vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403-rpm /boot/.System.map-5.10.0-rc2-ceph-gac4f7b07f403-rpm 2020-11-20T11:55:28.072 INFO:teuthology.orchestra.run.smithi187.stdout:fi 2020-11-20T11:55:28.072 INFO:teuthology.orchestra.run.smithi187.stdout:preuninstall scriptlet (using /bin/sh): 2020-11-20T11:55:28.072 INFO:teuthology.orchestra.run.smithi187.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2020-11-20T11:55:28.073 INFO:teuthology.orchestra.run.smithi187.stdout:new-kernel-pkg --remove 5.10.0-rc2-ceph-gac4f7b07f403 --rminitrd --initrdfile=/boot/initramfs-5.10.0-rc2-ceph-gac4f7b07f403.img 2020-11-20T11:55:28.073 INFO:teuthology.orchestra.run.smithi187.stdout:elif [ -x /usr/bin/kernel-install ]; then 2020-11-20T11:55:28.073 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-install remove 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:28.073 INFO:teuthology.orchestra.run.smithi187.stdout:fi 2020-11-20T11:55:28.073 INFO:teuthology.orchestra.run.smithi187.stdout:postuninstall scriptlet (using /bin/sh): 2020-11-20T11:55:28.074 INFO:teuthology.orchestra.run.smithi187.stdout:if [ -x /sbin/update-bootloader ]; then 2020-11-20T11:55:28.074 INFO:teuthology.orchestra.run.smithi187.stdout:/sbin/update-bootloader --remove 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:28.074 INFO:teuthology.orchestra.run.smithi187.stdout:fi 2020-11-20T11:55:28.075 INFO:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2020-11-20T11:55:28.154 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2020-11-20T11:55:28.154 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2020-11-20T11:55:28.154 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.02 2020-11-20T11:55:28.155 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 78.el8 2020-11-20T11:55:28.155 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2020-11-20T11:55:28.155 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Sun 15 Nov 2020 12:08:16 PM UTC 2020-11-20T11:55:28.155 INFO:teuthology.orchestra.run.smithi187.stdout:Group : System Environment/Base 2020-11-20T11:55:28.156 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 9192691 2020-11-20T11:55:28.156 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2020-11-20T11:55:28.156 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-11-20T11:55:28.156 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-11-20T11:55:28.156 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-11-20T11:55:28.157 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : kbuilder.bsys.centos.org 2020-11-20T11:55:28.157 INFO:teuthology.orchestra.run.smithi187.stdout:Relocations : (not relocatable) 2020-11-20T11:55:28.157 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : CentOS BuildSystem 2020-11-20T11:55:28.157 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2020-11-20T11:55:28.158 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2020-11-20T11:55:28.158 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2020-11-20T11:55:28.158 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2020-11-20T11:55:28.158 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T11:55:28.159 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-11-20T11:55:28.159 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2020-11-20T11:55:28.159 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2020-11-20T11:55:28.159 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2020-11-20T11:55:28.159 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T11:55:28.160 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2020-11-20T11:55:28.160 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-11-20T11:55:28.161 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:28.161 INFO:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-11-20T11:55:32.389 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2020-11-20T11:55:34.523 INFO:teuthology.orchestra.run.smithi187.stderr:done 2020-11-20T11:55:34.526 INFO:teuthology.orchestra.run.smithi187:> mktemp 2020-11-20T11:55:34.552 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.3Zzg1PXSa9 2020-11-20T11:55:34.553 INFO:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3Zzg1PXSa9 2020-11-20T11:55:34.647 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.3Zzg1PXSa9 2020-11-20T11:55:34.785 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.3Zzg1PXSa9 is 4KB 2020-11-20T11:55:34.816 INFO:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.3Zzg1PXSa9 2020-11-20T11:55:34.834 INFO:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2020-11-20T11:55:34.910 INFO:teuthology.orchestra.run.smithi187.stdout:558cda909eee45cd892f671baf08ec7a-0-rescue.conf 2020-11-20T11:55:34.911 INFO:teuthology.orchestra.run.smithi187.stdout:558cda909eee45cd892f671baf08ec7a-4.18.0-147.el8.x86_64.conf 2020-11-20T11:55:34.911 INFO:teuthology.orchestra.run.smithi187.stdout:558cda909eee45cd892f671baf08ec7a-4.18.0-193.28.1.el8_2.x86_64.conf 2020-11-20T11:55:34.911 INFO:teuthology.orchestra.run.smithi187.stdout:558cda909eee45cd892f671baf08ec7a-5.10.0-rc2-ceph-gac4f7b07f403.conf 2020-11-20T11:55:34.912 INFO:teuthology.orchestra.run.smithi187:> sudo grub2-set-default 558cda909eee45cd892f671baf08ec7a-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:55:35.043 INFO:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2020-11-20T11:55:35.045 INFO:teuthology.task.kernel:Installing kernel ac4f7b07f4039e2bfae0b119fa6b4810d4eaa4d9 on client.2... 2020-11-20T11:55:35.046 INFO:teuthology.orchestra.run.smithi005:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2020-11-20T11:55:37.246 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying... ######################################## 2020-11-20T11:55:37.632 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing... ######################################## 2020-11-20T11:55:38.528 INFO:teuthology.orchestra.run.smithi005.stdout:Updating / installing... 2020-11-20T11:56:18.877 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.10.0_rc2_ceph_gac4f7b07f403-1######################################## 2020-11-20T11:56:41.331 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2020-11-20T11:56:41.331 INFO:teuthology.orchestra.run.smithi005:> rpm -qlp /tmp/kernel.x86_64.rpm 2020-11-20T11:56:41.388 INFO:teuthology.orchestra.run.smithi005.stdout:/boot/System.map-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:41.388 INFO:teuthology.orchestra.run.smithi005.stdout:/boot/config-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:41.389 INFO:teuthology.orchestra.run.smithi005.stdout:/boot/vmlinux-5.10.0-rc2-ceph-gac4f7b07f403.bz2 2020-11-20T11:56:41.389 INFO:teuthology.orchestra.run.smithi005.stdout:/boot/vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:41.389 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:41.390 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel 2020-11-20T11:56:41.390 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch 2020-11-20T11:56:41.390 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86 2020-11-20T11:56:41.391 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto 2020-11-20T11:56:41.391 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/blowfish-x86_64.ko 2020-11-20T11:56:41.391 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2020-11-20T11:56:41.392 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2020-11-20T11:56:41.392 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/camellia-x86_64.ko 2020-11-20T11:56:41.392 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2020-11-20T11:56:41.392 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2020-11-20T11:56:41.393 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/crc32-pclmul.ko 2020-11-20T11:56:41.393 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/crc32c-intel.ko 2020-11-20T11:56:41.393 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2020-11-20T11:56:41.393 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2020-11-20T11:56:41.394 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2020-11-20T11:56:41.394 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2020-11-20T11:56:41.394 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/serpent-avx2.ko 2020-11-20T11:56:41.394 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2020-11-20T11:56:41.395 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/sha1-ssse3.ko 2020-11-20T11:56:41.395 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/sha256-ssse3.ko 2020-11-20T11:56:41.395 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/sha512-ssse3.ko 2020-11-20T11:56:41.396 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2020-11-20T11:56:41.396 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2020-11-20T11:56:41.396 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/crypto/twofish-x86_64.ko 2020-11-20T11:56:41.397 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kernel 2020-11-20T11:56:41.397 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kernel/cpu 2020-11-20T11:56:41.397 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kernel/cpu/mce 2020-11-20T11:56:41.397 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2020-11-20T11:56:41.398 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kvm 2020-11-20T11:56:41.398 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kvm/kvm-amd.ko 2020-11-20T11:56:41.398 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kvm/kvm-intel.ko 2020-11-20T11:56:41.399 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/kvm/kvm.ko 2020-11-20T11:56:41.399 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/oprofile 2020-11-20T11:56:41.399 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/arch/x86/oprofile/oprofile.ko 2020-11-20T11:56:41.400 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto 2020-11-20T11:56:41.400 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/ansi_cprng.ko 2020-11-20T11:56:41.400 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/anubis.ko 2020-11-20T11:56:41.400 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/arc4.ko 2020-11-20T11:56:41.401 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx 2020-11-20T11:56:41.401 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_memcpy.ko 2020-11-20T11:56:41.401 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_pq.ko 2020-11-20T11:56:41.401 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_raid6_recov.ko 2020-11-20T11:56:41.402 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_tx.ko 2020-11-20T11:56:41.402 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/async_xor.ko 2020-11-20T11:56:41.402 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/async_tx/raid6test.ko 2020-11-20T11:56:41.403 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/authenc.ko 2020-11-20T11:56:41.403 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/authencesn.ko 2020-11-20T11:56:41.403 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/blake2b_generic.ko 2020-11-20T11:56:41.403 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/blowfish_common.ko 2020-11-20T11:56:41.404 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/blowfish_generic.ko 2020-11-20T11:56:41.404 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/camellia_generic.ko 2020-11-20T11:56:41.404 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cast5_generic.ko 2020-11-20T11:56:41.404 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cast6_generic.ko 2020-11-20T11:56:41.405 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cast_common.ko 2020-11-20T11:56:41.405 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/ccm.ko 2020-11-20T11:56:41.405 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cmac.ko 2020-11-20T11:56:41.405 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/crc32_generic.ko 2020-11-20T11:56:41.406 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/crypto_engine.ko 2020-11-20T11:56:41.406 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/crypto_user.ko 2020-11-20T11:56:41.406 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/cts.ko 2020-11-20T11:56:41.407 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/des_generic.ko 2020-11-20T11:56:41.407 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/dh_generic.ko 2020-11-20T11:56:41.407 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/echainiv.ko 2020-11-20T11:56:41.408 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/essiv.ko 2020-11-20T11:56:41.408 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/fcrypt.ko 2020-11-20T11:56:41.408 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/khazad.ko 2020-11-20T11:56:41.408 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/lz4.ko 2020-11-20T11:56:41.409 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/lz4hc.ko 2020-11-20T11:56:41.409 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/md4.ko 2020-11-20T11:56:41.409 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/michael_mic.ko 2020-11-20T11:56:41.410 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/pcbc.ko 2020-11-20T11:56:41.410 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/pcrypt.ko 2020-11-20T11:56:41.410 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/rmd128.ko 2020-11-20T11:56:41.411 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/rmd160.ko 2020-11-20T11:56:41.411 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/rmd256.ko 2020-11-20T11:56:41.411 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/rmd320.ko 2020-11-20T11:56:41.411 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/salsa20_generic.ko 2020-11-20T11:56:41.412 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/seed.ko 2020-11-20T11:56:41.412 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/serpent_generic.ko 2020-11-20T11:56:41.412 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/sha512_generic.ko 2020-11-20T11:56:41.412 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/tcrypt.ko 2020-11-20T11:56:41.413 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/tea.ko 2020-11-20T11:56:41.413 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/tgr192.ko 2020-11-20T11:56:41.414 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/twofish_common.ko 2020-11-20T11:56:41.414 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/twofish_generic.ko 2020-11-20T11:56:41.414 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/vmac.ko 2020-11-20T11:56:41.414 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/wp512.ko 2020-11-20T11:56:41.415 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/xcbc.ko 2020-11-20T11:56:41.415 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/xor.ko 2020-11-20T11:56:41.415 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/crypto/xxhash_generic.ko 2020-11-20T11:56:41.415 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers 2020-11-20T11:56:41.416 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi 2020-11-20T11:56:41.416 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/acpi_ipmi.ko 2020-11-20T11:56:41.416 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/acpi_pad.ko 2020-11-20T11:56:41.417 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/custom_method.ko 2020-11-20T11:56:41.417 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/ec_sys.ko 2020-11-20T11:56:41.417 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/sbs.ko 2020-11-20T11:56:41.418 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/sbshc.ko 2020-11-20T11:56:41.418 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/acpi/video.ko 2020-11-20T11:56:41.418 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata 2020-11-20T11:56:41.418 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/acard-ahci.ko 2020-11-20T11:56:41.418 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/ahci_platform.ko 2020-11-20T11:56:41.419 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/ata_generic.ko 2020-11-20T11:56:41.419 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/libahci_platform.ko 2020-11-20T11:56:41.419 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_acpi.ko 2020-11-20T11:56:41.419 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_ali.ko 2020-11-20T11:56:41.420 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_amd.ko 2020-11-20T11:56:41.420 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_artop.ko 2020-11-20T11:56:41.420 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_atiixp.ko 2020-11-20T11:56:41.420 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_atp867x.ko 2020-11-20T11:56:41.421 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_cmd640.ko 2020-11-20T11:56:41.421 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_cmd64x.ko 2020-11-20T11:56:41.421 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_cypress.ko 2020-11-20T11:56:41.421 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_efar.ko 2020-11-20T11:56:41.422 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_hpt366.ko 2020-11-20T11:56:41.422 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_hpt37x.ko 2020-11-20T11:56:41.422 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_hpt3x2n.ko 2020-11-20T11:56:41.422 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_hpt3x3.ko 2020-11-20T11:56:41.423 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_it8213.ko 2020-11-20T11:56:41.423 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_it821x.ko 2020-11-20T11:56:41.423 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_jmicron.ko 2020-11-20T11:56:41.423 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_marvell.ko 2020-11-20T11:56:41.424 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_mpiix.ko 2020-11-20T11:56:41.424 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_netcell.ko 2020-11-20T11:56:41.424 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_ninja32.ko 2020-11-20T11:56:41.424 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_ns87410.ko 2020-11-20T11:56:41.425 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_ns87415.ko 2020-11-20T11:56:41.425 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_oldpiix.ko 2020-11-20T11:56:41.426 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_opti.ko 2020-11-20T11:56:41.426 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_optidma.ko 2020-11-20T11:56:41.426 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_pcmcia.ko 2020-11-20T11:56:41.426 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_pdc2027x.ko 2020-11-20T11:56:41.427 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_pdc202xx_old.ko 2020-11-20T11:56:41.427 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_piccolo.ko 2020-11-20T11:56:41.427 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_rdc.ko 2020-11-20T11:56:41.427 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_sch.ko 2020-11-20T11:56:41.427 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_serverworks.ko 2020-11-20T11:56:41.428 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_sil680.ko 2020-11-20T11:56:41.428 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_sis.ko 2020-11-20T11:56:41.428 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_sl82c105.ko 2020-11-20T11:56:41.428 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_triflex.ko 2020-11-20T11:56:41.429 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pata_via.ko 2020-11-20T11:56:41.429 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/pdc_adma.ko 2020-11-20T11:56:41.429 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_inic162x.ko 2020-11-20T11:56:41.429 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_mv.ko 2020-11-20T11:56:41.430 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_nv.ko 2020-11-20T11:56:41.430 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_promise.ko 2020-11-20T11:56:41.430 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_qstor.ko 2020-11-20T11:56:41.430 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_sil.ko 2020-11-20T11:56:41.431 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_sil24.ko 2020-11-20T11:56:41.431 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_sis.ko 2020-11-20T11:56:41.431 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_svw.ko 2020-11-20T11:56:41.431 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_sx4.ko 2020-11-20T11:56:41.432 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_uli.ko 2020-11-20T11:56:41.432 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_via.ko 2020-11-20T11:56:41.432 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ata/sata_vsc.ko 2020-11-20T11:56:41.432 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/auxdisplay 2020-11-20T11:56:41.433 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/auxdisplay/ks0108.ko 2020-11-20T11:56:41.433 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/base 2020-11-20T11:56:41.433 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/base/regmap 2020-11-20T11:56:41.433 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/base/regmap/regmap-i2c.ko 2020-11-20T11:56:41.434 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/bcma 2020-11-20T11:56:41.434 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/bcma/bcma.ko 2020-11-20T11:56:41.434 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block 2020-11-20T11:56:41.434 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/aoe 2020-11-20T11:56:41.435 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/aoe/aoe.ko 2020-11-20T11:56:41.435 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/brd.ko 2020-11-20T11:56:41.435 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/drbd 2020-11-20T11:56:41.435 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/drbd/drbd.ko 2020-11-20T11:56:41.435 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/floppy.ko 2020-11-20T11:56:41.436 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/loop.ko 2020-11-20T11:56:41.436 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/mtip32xx 2020-11-20T11:56:41.436 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/mtip32xx/mtip32xx.ko 2020-11-20T11:56:41.436 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/nbd.ko 2020-11-20T11:56:41.437 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/null_blk.ko 2020-11-20T11:56:41.437 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/pktcdvd.ko 2020-11-20T11:56:41.437 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/rbd.ko 2020-11-20T11:56:41.437 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/skd.ko 2020-11-20T11:56:41.438 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/sx8.ko 2020-11-20T11:56:41.438 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/umem.ko 2020-11-20T11:56:41.438 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/virtio_blk.ko 2020-11-20T11:56:41.438 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/xen-blkback 2020-11-20T11:56:41.439 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/xen-blkback/xen-blkback.ko 2020-11-20T11:56:41.439 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/xen-blkfront.ko 2020-11-20T11:56:41.439 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/zram 2020-11-20T11:56:41.439 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/block/zram/zram.ko 2020-11-20T11:56:41.440 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char 2020-11-20T11:56:41.440 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hangcheck-timer.ko 2020-11-20T11:56:41.440 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random 2020-11-20T11:56:41.440 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/amd-rng.ko 2020-11-20T11:56:41.440 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/intel-rng.ko 2020-11-20T11:56:41.441 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/timeriomem-rng.ko 2020-11-20T11:56:41.441 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/via-rng.ko 2020-11-20T11:56:41.441 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/hw_random/virtio-rng.ko 2020-11-20T11:56:41.441 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi 2020-11-20T11:56:41.442 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_devintf.ko 2020-11-20T11:56:41.442 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2020-11-20T11:56:41.442 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2020-11-20T11:56:41.442 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_si.ko 2020-11-20T11:56:41.443 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_ssif.ko 2020-11-20T11:56:41.443 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2020-11-20T11:56:41.443 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/lp.ko 2020-11-20T11:56:41.443 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/mwave 2020-11-20T11:56:41.443 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/mwave/mwave.ko 2020-11-20T11:56:41.444 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/pcmcia 2020-11-20T11:56:41.444 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/pcmcia/cm4000_cs.ko 2020-11-20T11:56:41.444 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/pcmcia/cm4040_cs.ko 2020-11-20T11:56:41.444 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/ppdev.ko 2020-11-20T11:56:41.445 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tlclk.ko 2020-11-20T11:56:41.445 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm 2020-11-20T11:56:41.445 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm.ko 2020-11-20T11:56:41.445 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_atmel.ko 2020-11-20T11:56:41.445 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_infineon.ko 2020-11-20T11:56:41.446 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_nsc.ko 2020-11-20T11:56:41.446 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_tis.ko 2020-11-20T11:56:41.446 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/tpm/tpm_tis_core.ko 2020-11-20T11:56:41.447 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/uv_mmtimer.ko 2020-11-20T11:56:41.447 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/char/virtio_console.ko 2020-11-20T11:56:41.447 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq 2020-11-20T11:56:41.447 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/acpi-cpufreq.ko 2020-11-20T11:56:41.448 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2020-11-20T11:56:41.448 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/p4-clockmod.ko 2020-11-20T11:56:41.448 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/pcc-cpufreq.ko 2020-11-20T11:56:41.449 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/powernow-k8.ko 2020-11-20T11:56:41.449 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/cpufreq/speedstep-lib.ko 2020-11-20T11:56:41.449 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto 2020-11-20T11:56:41.450 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/ccp 2020-11-20T11:56:41.450 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/ccp/ccp-crypto.ko 2020-11-20T11:56:41.450 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/ccp/ccp.ko 2020-11-20T11:56:41.450 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/padlock-aes.ko 2020-11-20T11:56:41.451 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/padlock-sha.ko 2020-11-20T11:56:41.451 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat 2020-11-20T11:56:41.451 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat/qat_common 2020-11-20T11:56:41.451 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2020-11-20T11:56:41.451 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat/qat_dh895xcc 2020-11-20T11:56:41.452 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2020-11-20T11:56:41.452 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/virtio 2020-11-20T11:56:41.452 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/crypto/virtio/virtio_crypto.ko 2020-11-20T11:56:41.453 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dca 2020-11-20T11:56:41.453 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dca/dca.ko 2020-11-20T11:56:41.453 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/devfreq 2020-11-20T11:56:41.453 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/devfreq/governor_simpleondemand.ko 2020-11-20T11:56:41.453 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma 2020-11-20T11:56:41.454 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma/dw 2020-11-20T11:56:41.454 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma/dw/dw_dmac.ko 2020-11-20T11:56:41.454 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma/ioat 2020-11-20T11:56:41.454 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/dma/ioat/ioatdma.ko 2020-11-20T11:56:41.455 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac 2020-11-20T11:56:41.455 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/amd64_edac_mod.ko 2020-11-20T11:56:41.455 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/e752x_edac.ko 2020-11-20T11:56:41.456 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/edac_mce_amd.ko 2020-11-20T11:56:41.456 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i3000_edac.ko 2020-11-20T11:56:41.456 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i3200_edac.ko 2020-11-20T11:56:41.456 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i5000_edac.ko 2020-11-20T11:56:41.457 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i5100_edac.ko 2020-11-20T11:56:41.457 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i5400_edac.ko 2020-11-20T11:56:41.457 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i7300_edac.ko 2020-11-20T11:56:41.457 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i7core_edac.ko 2020-11-20T11:56:41.458 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/i82975x_edac.ko 2020-11-20T11:56:41.458 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/ie31200_edac.ko 2020-11-20T11:56:41.458 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/sb_edac.ko 2020-11-20T11:56:41.458 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/edac/x38_edac.ko 2020-11-20T11:56:41.458 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/firmware 2020-11-20T11:56:41.459 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/firmware/edd.ko 2020-11-20T11:56:41.459 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/firmware/iscsi_ibft.ko 2020-11-20T11:56:41.459 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/gpio 2020-11-20T11:56:41.460 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/gpio/gpio-ich.ko 2020-11-20T11:56:41.460 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/gpio/gpio-viperboard.ko 2020-11-20T11:56:41.460 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid 2020-11-20T11:56:41.460 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-appleir.ko 2020-11-20T11:56:41.461 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-aureal.ko 2020-11-20T11:56:41.461 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-axff.ko 2020-11-20T11:56:41.461 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-dr.ko 2020-11-20T11:56:41.462 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-elecom.ko 2020-11-20T11:56:41.462 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-elo.ko 2020-11-20T11:56:41.462 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-emsff.ko 2020-11-20T11:56:41.462 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-gaff.ko 2020-11-20T11:56:41.463 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-gt683r.ko 2020-11-20T11:56:41.463 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-gyration.ko 2020-11-20T11:56:41.463 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-holtek-kbd.ko 2020-11-20T11:56:41.464 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-holtek-mouse.ko 2020-11-20T11:56:41.464 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-holtekff.ko 2020-11-20T11:56:41.464 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-hyperv.ko 2020-11-20T11:56:41.464 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-icade.ko 2020-11-20T11:56:41.464 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-keytouch.ko 2020-11-20T11:56:41.465 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-kye.ko 2020-11-20T11:56:41.465 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-lcpower.ko 2020-11-20T11:56:41.465 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-led.ko 2020-11-20T11:56:41.466 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-lenovo.ko 2020-11-20T11:56:41.466 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-logitech-dj.ko 2020-11-20T11:56:41.466 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-logitech-hidpp.ko 2020-11-20T11:56:41.466 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-multitouch.ko 2020-11-20T11:56:41.467 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-ortek.ko 2020-11-20T11:56:41.467 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-penmount.ko 2020-11-20T11:56:41.467 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-petalynx.ko 2020-11-20T11:56:41.467 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-picolcd.ko 2020-11-20T11:56:41.468 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-pl.ko 2020-11-20T11:56:41.468 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-plantronics.ko 2020-11-20T11:56:41.468 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-primax.ko 2020-11-20T11:56:41.469 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-rmi.ko 2020-11-20T11:56:41.469 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-arvo.ko 2020-11-20T11:56:41.469 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-common.ko 2020-11-20T11:56:41.469 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-isku.ko 2020-11-20T11:56:41.469 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-kone.ko 2020-11-20T11:56:41.470 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-koneplus.ko 2020-11-20T11:56:41.470 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-konepure.ko 2020-11-20T11:56:41.470 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-kovaplus.ko 2020-11-20T11:56:41.470 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-lua.ko 2020-11-20T11:56:41.471 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-pyra.ko 2020-11-20T11:56:41.471 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-ryos.ko 2020-11-20T11:56:41.471 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat-savu.ko 2020-11-20T11:56:41.471 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-roccat.ko 2020-11-20T11:56:41.472 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-saitek.ko 2020-11-20T11:56:41.472 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-samsung.ko 2020-11-20T11:56:41.472 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-sensor-hub.ko 2020-11-20T11:56:41.473 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-sjoy.ko 2020-11-20T11:56:41.473 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-sony.ko 2020-11-20T11:56:41.473 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-speedlink.ko 2020-11-20T11:56:41.473 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-steelseries.ko 2020-11-20T11:56:41.474 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-sunplus.ko 2020-11-20T11:56:41.474 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-tivo.ko 2020-11-20T11:56:41.474 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-tmff.ko 2020-11-20T11:56:41.475 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-topseed.ko 2020-11-20T11:56:41.475 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-twinhan.ko 2020-11-20T11:56:41.475 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-uclogic.ko 2020-11-20T11:56:41.475 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-waltop.ko 2020-11-20T11:56:41.476 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-wiimote.ko 2020-11-20T11:56:41.476 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-xinmo.ko 2020-11-20T11:56:41.476 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-zpff.ko 2020-11-20T11:56:41.476 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/hid-zydacron.ko 2020-11-20T11:56:41.477 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/i2c-hid 2020-11-20T11:56:41.477 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/i2c-hid/i2c-hid.ko 2020-11-20T11:56:41.477 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/uhid.ko 2020-11-20T11:56:41.477 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hid/wacom.ko 2020-11-20T11:56:41.478 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hv 2020-11-20T11:56:41.478 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hv/hv_balloon.ko 2020-11-20T11:56:41.478 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hv/hv_utils.ko 2020-11-20T11:56:41.478 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hv/hv_vmbus.ko 2020-11-20T11:56:41.479 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon 2020-11-20T11:56:41.479 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/abituguru.ko 2020-11-20T11:56:41.479 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/abituguru3.ko 2020-11-20T11:56:41.480 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/acpi_power_meter.ko 2020-11-20T11:56:41.480 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ad7414.ko 2020-11-20T11:56:41.480 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ad7418.ko 2020-11-20T11:56:41.481 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adc128d818.ko 2020-11-20T11:56:41.481 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1021.ko 2020-11-20T11:56:41.481 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1025.ko 2020-11-20T11:56:41.481 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1026.ko 2020-11-20T11:56:41.481 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1029.ko 2020-11-20T11:56:41.482 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm1031.ko 2020-11-20T11:56:41.482 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adm9240.ko 2020-11-20T11:56:41.482 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ads7828.ko 2020-11-20T11:56:41.482 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7410.ko 2020-11-20T11:56:41.483 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7411.ko 2020-11-20T11:56:41.483 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7462.ko 2020-11-20T11:56:41.483 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7470.ko 2020-11-20T11:56:41.484 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7475.ko 2020-11-20T11:56:41.484 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/adt7x10.ko 2020-11-20T11:56:41.484 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/amc6821.ko 2020-11-20T11:56:41.484 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/applesmc.ko 2020-11-20T11:56:41.485 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/asb100.ko 2020-11-20T11:56:41.485 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/asc7621.ko 2020-11-20T11:56:41.485 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/asus_atk0110.ko 2020-11-20T11:56:41.486 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/atxp1.ko 2020-11-20T11:56:41.486 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/coretemp.ko 2020-11-20T11:56:41.486 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/dell-smm-hwmon.ko 2020-11-20T11:56:41.486 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/dme1737.ko 2020-11-20T11:56:41.487 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ds1621.ko 2020-11-20T11:56:41.487 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ds620.ko 2020-11-20T11:56:41.487 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/emc1403.ko 2020-11-20T11:56:41.487 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/emc6w201.ko 2020-11-20T11:56:41.488 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/f71805f.ko 2020-11-20T11:56:41.488 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/f71882fg.ko 2020-11-20T11:56:41.488 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/f75375s.ko 2020-11-20T11:56:41.488 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/fam15h_power.ko 2020-11-20T11:56:41.489 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/fschmd.ko 2020-11-20T11:56:41.489 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/g760a.ko 2020-11-20T11:56:41.489 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/g762.ko 2020-11-20T11:56:41.490 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/gl518sm.ko 2020-11-20T11:56:41.490 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/gl520sm.ko 2020-11-20T11:56:41.490 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/hwmon-vid.ko 2020-11-20T11:56:41.490 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/i5500_temp.ko 2020-11-20T11:56:41.491 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/i5k_amb.ko 2020-11-20T11:56:41.491 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ibmaem.ko 2020-11-20T11:56:41.491 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ibmpex.ko 2020-11-20T11:56:41.491 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ina209.ko 2020-11-20T11:56:41.492 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ina2xx.ko 2020-11-20T11:56:41.492 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/it87.ko 2020-11-20T11:56:41.492 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/k10temp.ko 2020-11-20T11:56:41.492 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/k8temp.ko 2020-11-20T11:56:41.493 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lineage-pem.ko 2020-11-20T11:56:41.493 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm63.ko 2020-11-20T11:56:41.493 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm73.ko 2020-11-20T11:56:41.493 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm75.ko 2020-11-20T11:56:41.494 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm77.ko 2020-11-20T11:56:41.494 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm78.ko 2020-11-20T11:56:41.494 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm80.ko 2020-11-20T11:56:41.494 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm83.ko 2020-11-20T11:56:41.495 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm85.ko 2020-11-20T11:56:41.495 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm87.ko 2020-11-20T11:56:41.495 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm90.ko 2020-11-20T11:56:41.495 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm92.ko 2020-11-20T11:56:41.496 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm93.ko 2020-11-20T11:56:41.496 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm95234.ko 2020-11-20T11:56:41.496 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm95241.ko 2020-11-20T11:56:41.496 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/lm95245.ko 2020-11-20T11:56:41.497 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc2945.ko 2020-11-20T11:56:41.497 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4151.ko 2020-11-20T11:56:41.497 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4215.ko 2020-11-20T11:56:41.497 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4222.ko 2020-11-20T11:56:41.498 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4245.ko 2020-11-20T11:56:41.498 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4260.ko 2020-11-20T11:56:41.498 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ltc4261.ko 2020-11-20T11:56:41.498 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max16065.ko 2020-11-20T11:56:41.499 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max1619.ko 2020-11-20T11:56:41.499 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max1668.ko 2020-11-20T11:56:41.499 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max197.ko 2020-11-20T11:56:41.499 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max6639.ko 2020-11-20T11:56:41.499 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max6642.ko 2020-11-20T11:56:41.500 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max6650.ko 2020-11-20T11:56:41.500 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/max6697.ko 2020-11-20T11:56:41.500 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/mcp3021.ko 2020-11-20T11:56:41.500 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/nct6683.ko 2020-11-20T11:56:41.501 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/nct6775.ko 2020-11-20T11:56:41.501 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/nct7802.ko 2020-11-20T11:56:41.501 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/ntc_thermistor.ko 2020-11-20T11:56:41.502 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pc87360.ko 2020-11-20T11:56:41.502 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pc87427.ko 2020-11-20T11:56:41.502 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pcf8591.ko 2020-11-20T11:56:41.503 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus 2020-11-20T11:56:41.503 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/adm1275.ko 2020-11-20T11:56:41.503 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/lm25066.ko 2020-11-20T11:56:41.503 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/ltc2978.ko 2020-11-20T11:56:41.504 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/max16064.ko 2020-11-20T11:56:41.504 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/max34440.ko 2020-11-20T11:56:41.504 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/max8688.ko 2020-11-20T11:56:41.504 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/pmbus.ko 2020-11-20T11:56:41.505 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2020-11-20T11:56:41.505 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/tps40422.ko 2020-11-20T11:56:41.505 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/ucd9000.ko 2020-11-20T11:56:41.505 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/ucd9200.ko 2020-11-20T11:56:41.506 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/pmbus/zl6100.ko 2020-11-20T11:56:41.506 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/powr1220.ko 2020-11-20T11:56:41.506 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sch5627.ko 2020-11-20T11:56:41.506 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sch5636.ko 2020-11-20T11:56:41.506 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sch56xx-common.ko 2020-11-20T11:56:41.507 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sht15.ko 2020-11-20T11:56:41.507 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sht21.ko 2020-11-20T11:56:41.507 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/shtc1.ko 2020-11-20T11:56:41.508 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/sis5595.ko 2020-11-20T11:56:41.508 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/smsc47b397.ko 2020-11-20T11:56:41.508 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/smsc47m1.ko 2020-11-20T11:56:41.508 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/smsc47m192.ko 2020-11-20T11:56:41.508 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/thmc50.ko 2020-11-20T11:56:41.509 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/tmp102.ko 2020-11-20T11:56:41.509 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/tmp103.ko 2020-11-20T11:56:41.509 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/tmp401.ko 2020-11-20T11:56:41.509 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/tmp421.ko 2020-11-20T11:56:41.510 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/via-cputemp.ko 2020-11-20T11:56:41.510 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/via686a.ko 2020-11-20T11:56:41.510 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/vt1211.ko 2020-11-20T11:56:41.510 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/vt8231.ko 2020-11-20T11:56:41.510 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83627ehf.ko 2020-11-20T11:56:41.511 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83627hf.ko 2020-11-20T11:56:41.511 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83781d.ko 2020-11-20T11:56:41.511 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83791d.ko 2020-11-20T11:56:41.512 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83792d.ko 2020-11-20T11:56:41.512 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83793.ko 2020-11-20T11:56:41.512 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83795.ko 2020-11-20T11:56:41.512 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83l785ts.ko 2020-11-20T11:56:41.513 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/hwmon/w83l786ng.ko 2020-11-20T11:56:41.513 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c 2020-11-20T11:56:41.513 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/algos 2020-11-20T11:56:41.513 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2020-11-20T11:56:41.514 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2020-11-20T11:56:41.514 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses 2020-11-20T11:56:41.514 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2020-11-20T11:56:41.514 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-amd756.ko 2020-11-20T11:56:41.515 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-amd8111.ko 2020-11-20T11:56:41.515 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-designware-core.ko 2020-11-20T11:56:41.515 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2020-11-20T11:56:41.515 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2020-11-20T11:56:41.516 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2020-11-20T11:56:41.516 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-i801.ko 2020-11-20T11:56:41.516 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-isch.ko 2020-11-20T11:56:41.516 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-ismt.ko 2020-11-20T11:56:41.517 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2020-11-20T11:56:41.517 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-nforce2.ko 2020-11-20T11:56:41.517 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-parport.ko 2020-11-20T11:56:41.517 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2020-11-20T11:56:41.518 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-piix4.ko 2020-11-20T11:56:41.518 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-scmi.ko 2020-11-20T11:56:41.518 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-simtec.ko 2020-11-20T11:56:41.518 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-sis96x.ko 2020-11-20T11:56:41.518 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2020-11-20T11:56:41.519 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-via.ko 2020-11-20T11:56:41.519 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-viapro.ko 2020-11-20T11:56:41.519 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/busses/i2c-viperboard.ko 2020-11-20T11:56:41.519 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-dev.ko 2020-11-20T11:56:41.520 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-mux.ko 2020-11-20T11:56:41.520 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-slave-eeprom.ko 2020-11-20T11:56:41.520 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-smbus.ko 2020-11-20T11:56:41.520 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/i2c/i2c-stub.ko 2020-11-20T11:56:41.521 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband 2020-11-20T11:56:41.521 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core 2020-11-20T11:56:41.521 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/ib_cm.ko 2020-11-20T11:56:41.521 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/ib_core.ko 2020-11-20T11:56:41.522 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/ib_umad.ko 2020-11-20T11:56:41.522 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/ib_uverbs.ko 2020-11-20T11:56:41.522 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/iw_cm.ko 2020-11-20T11:56:41.522 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/rdma_cm.ko 2020-11-20T11:56:41.523 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/core/rdma_ucm.ko 2020-11-20T11:56:41.523 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw 2020-11-20T11:56:41.523 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/cxgb4 2020-11-20T11:56:41.523 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2020-11-20T11:56:41.523 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mlx4 2020-11-20T11:56:41.524 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2020-11-20T11:56:41.524 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mlx5 2020-11-20T11:56:41.524 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2020-11-20T11:56:41.524 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mthca 2020-11-20T11:56:41.525 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2020-11-20T11:56:41.525 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp 2020-11-20T11:56:41.525 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/ipoib 2020-11-20T11:56:41.525 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2020-11-20T11:56:41.525 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/iser 2020-11-20T11:56:41.526 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2020-11-20T11:56:41.526 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/isert 2020-11-20T11:56:41.526 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2020-11-20T11:56:41.526 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/srp 2020-11-20T11:56:41.527 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2020-11-20T11:56:41.527 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/srpt 2020-11-20T11:56:41.527 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2020-11-20T11:56:41.527 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input 2020-11-20T11:56:41.528 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport 2020-11-20T11:56:41.528 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/emu10k1-gp.ko 2020-11-20T11:56:41.528 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/fm801-gp.ko 2020-11-20T11:56:41.528 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/gameport.ko 2020-11-20T11:56:41.529 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/lightning.ko 2020-11-20T11:56:41.529 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/gameport/ns558.ko 2020-11-20T11:56:41.529 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/input-polldev.ko 2020-11-20T11:56:41.529 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/joydev.ko 2020-11-20T11:56:41.530 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/keyboard 2020-11-20T11:56:41.530 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/keyboard/gpio_keys.ko 2020-11-20T11:56:41.530 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse 2020-11-20T11:56:41.530 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/appletouch.ko 2020-11-20T11:56:41.531 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/bcm5974.ko 2020-11-20T11:56:41.531 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/cyapatp.ko 2020-11-20T11:56:41.531 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/elan_i2c.ko 2020-11-20T11:56:41.531 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/sermouse.ko 2020-11-20T11:56:41.532 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/synaptics_i2c.ko 2020-11-20T11:56:41.532 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/synaptics_usb.ko 2020-11-20T11:56:41.532 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/mouse/vsxxxaa.ko 2020-11-20T11:56:41.532 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/rmi4 2020-11-20T11:56:41.533 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/rmi4/rmi_core.ko 2020-11-20T11:56:41.533 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio 2020-11-20T11:56:41.533 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio/altera_ps2.ko 2020-11-20T11:56:41.534 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio/arc_ps2.ko 2020-11-20T11:56:41.534 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio/hyperv-keyboard.ko 2020-11-20T11:56:41.534 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/serio/serio_raw.ko 2020-11-20T11:56:41.534 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/input/sparse-keymap.ko 2020-11-20T11:56:41.534 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/iommu 2020-11-20T11:56:41.535 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/iommu/amd 2020-11-20T11:56:41.535 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/iommu/amd/iommu_v2.ko 2020-11-20T11:56:41.535 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds 2020-11-20T11:56:41.536 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-blinkm.ko 2020-11-20T11:56:41.536 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-clevo-mail.ko 2020-11-20T11:56:41.536 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-lm3530.ko 2020-11-20T11:56:41.536 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-lp3944.ko 2020-11-20T11:56:41.537 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/leds-ss4200.ko 2020-11-20T11:56:41.537 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger 2020-11-20T11:56:41.537 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-audio.ko 2020-11-20T11:56:41.538 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2020-11-20T11:56:41.538 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-camera.ko 2020-11-20T11:56:41.538 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2020-11-20T11:56:41.538 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2020-11-20T11:56:41.539 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2020-11-20T11:56:41.539 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2020-11-20T11:56:41.539 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-timer.ko 2020-11-20T11:56:41.539 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/leds/trigger/ledtrig-transient.ko 2020-11-20T11:56:41.540 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md 2020-11-20T11:56:41.540 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/bcache 2020-11-20T11:56:41.540 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/bcache/bcache.ko 2020-11-20T11:56:41.540 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-bio-prison.ko 2020-11-20T11:56:41.541 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-cache-smq.ko 2020-11-20T11:56:41.541 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-cache.ko 2020-11-20T11:56:41.541 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-crypt.ko 2020-11-20T11:56:41.542 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-delay.ko 2020-11-20T11:56:41.542 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-flakey.ko 2020-11-20T11:56:41.542 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-log-userspace.ko 2020-11-20T11:56:41.542 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-multipath.ko 2020-11-20T11:56:41.543 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-queue-length.ko 2020-11-20T11:56:41.543 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-raid.ko 2020-11-20T11:56:41.543 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-round-robin.ko 2020-11-20T11:56:41.543 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-service-time.ko 2020-11-20T11:56:41.543 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-switch.ko 2020-11-20T11:56:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-thin-pool.ko 2020-11-20T11:56:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/dm-verity.ko 2020-11-20T11:56:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/faulty.ko 2020-11-20T11:56:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/linear.ko 2020-11-20T11:56:41.545 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/multipath.ko 2020-11-20T11:56:41.545 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/persistent-data 2020-11-20T11:56:41.545 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2020-11-20T11:56:41.545 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/raid0.ko 2020-11-20T11:56:41.546 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/raid1.ko 2020-11-20T11:56:41.546 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/raid10.ko 2020-11-20T11:56:41.546 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/md/raid456.ko 2020-11-20T11:56:41.546 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message 2020-11-20T11:56:41.547 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion 2020-11-20T11:56:41.547 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptbase.ko 2020-11-20T11:56:41.547 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptctl.ko 2020-11-20T11:56:41.547 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptfc.ko 2020-11-20T11:56:41.548 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptlan.ko 2020-11-20T11:56:41.548 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptsas.ko 2020-11-20T11:56:41.548 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptscsih.ko 2020-11-20T11:56:41.548 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/message/fusion/mptspi.ko 2020-11-20T11:56:41.548 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd 2020-11-20T11:56:41.549 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/lpc_ich.ko 2020-11-20T11:56:41.549 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/lpc_sch.ko 2020-11-20T11:56:41.549 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/mfd-core.ko 2020-11-20T11:56:41.549 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/sm501.ko 2020-11-20T11:56:41.550 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/viperboard.ko 2020-11-20T11:56:41.550 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/vx855.ko 2020-11-20T11:56:41.550 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/mfd/wl1273-core.ko 2020-11-20T11:56:41.550 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc 2020-11-20T11:56:41.551 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/altera-stapl 2020-11-20T11:56:41.551 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2020-11-20T11:56:41.551 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/apds9802als.ko 2020-11-20T11:56:41.552 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/apds990x.ko 2020-11-20T11:56:41.552 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/bh1770glc.ko 2020-11-20T11:56:41.552 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/cb710 2020-11-20T11:56:41.552 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/cb710/cb710.ko 2020-11-20T11:56:41.553 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom 2020-11-20T11:56:41.553 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom/at24.ko 2020-11-20T11:56:41.553 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom/eeprom.ko 2020-11-20T11:56:41.553 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2020-11-20T11:56:41.554 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/eeprom/max6875.ko 2020-11-20T11:56:41.554 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/enclosure.ko 2020-11-20T11:56:41.554 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/hpilo.ko 2020-11-20T11:56:41.554 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/isl29003.ko 2020-11-20T11:56:41.555 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/isl29020.ko 2020-11-20T11:56:41.555 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/lis3lv02d 2020-11-20T11:56:41.555 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2020-11-20T11:56:41.555 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2020-11-20T11:56:41.556 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/mei 2020-11-20T11:56:41.556 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/mei/mei-me.ko 2020-11-20T11:56:41.556 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/mei/mei-txe.ko 2020-11-20T11:56:41.556 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/mei/mei.ko 2020-11-20T11:56:41.556 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/pvpanic.ko 2020-11-20T11:56:41.557 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-gru 2020-11-20T11:56:41.557 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-gru/gru.ko 2020-11-20T11:56:41.557 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-xp 2020-11-20T11:56:41.557 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-xp/xp.ko 2020-11-20T11:56:41.558 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-xp/xpc.ko 2020-11-20T11:56:41.558 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/sgi-xp/xpnet.ko 2020-11-20T11:56:41.558 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/tifm_7xx1.ko 2020-11-20T11:56:41.558 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/tifm_core.ko 2020-11-20T11:56:41.559 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/tsl2550.ko 2020-11-20T11:56:41.559 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/vmw_balloon.ko 2020-11-20T11:56:41.559 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/vmw_vmci 2020-11-20T11:56:41.559 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2020-11-20T11:56:41.560 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net 2020-11-20T11:56:41.560 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/bonding 2020-11-20T11:56:41.560 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/bonding/bonding.ko 2020-11-20T11:56:41.560 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa 2020-11-20T11:56:41.560 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/b53 2020-11-20T11:56:41.561 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/b53/b53_common.ko 2020-11-20T11:56:41.561 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/bcm-sf2.ko 2020-11-20T11:56:41.561 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/mv88e6060.ko 2020-11-20T11:56:41.562 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/mv88e6xxx 2020-11-20T11:56:41.562 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2020-11-20T11:56:41.562 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/dummy.ko 2020-11-20T11:56:41.562 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/eql.ko 2020-11-20T11:56:41.562 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet 2020-11-20T11:56:41.563 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com 2020-11-20T11:56:41.563 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2020-11-20T11:56:41.563 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2020-11-20T11:56:41.563 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com/3c59x.ko 2020-11-20T11:56:41.564 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/3com/typhoon.ko 2020-11-20T11:56:41.564 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390 2020-11-20T11:56:41.564 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390/8390.ko 2020-11-20T11:56:41.564 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2020-11-20T11:56:41.565 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2020-11-20T11:56:41.565 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2020-11-20T11:56:41.565 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/adaptec 2020-11-20T11:56:41.565 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/adaptec/starfire.ko 2020-11-20T11:56:41.566 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/agere 2020-11-20T11:56:41.566 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/agere/et131x.ko 2020-11-20T11:56:41.566 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/alteon 2020-11-20T11:56:41.566 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/alteon/acenic.ko 2020-11-20T11:56:41.567 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/altera 2020-11-20T11:56:41.567 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/altera/altera_tse.ko 2020-11-20T11:56:41.567 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/amd 2020-11-20T11:56:41.567 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/amd/amd8111e.ko 2020-11-20T11:56:41.567 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2020-11-20T11:56:41.568 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/amd/pcnet32.ko 2020-11-20T11:56:41.568 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros 2020-11-20T11:56:41.568 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/alx 2020-11-20T11:56:41.568 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2020-11-20T11:56:41.569 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atl1c 2020-11-20T11:56:41.569 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2020-11-20T11:56:41.569 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atl1e 2020-11-20T11:56:41.570 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2020-11-20T11:56:41.570 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atlx 2020-11-20T11:56:41.570 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2020-11-20T11:56:41.570 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2020-11-20T11:56:41.571 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom 2020-11-20T11:56:41.571 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/b44.ko 2020-11-20T11:56:41.571 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2020-11-20T11:56:41.571 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/bnx2x 2020-11-20T11:56:41.572 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2020-11-20T11:56:41.572 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/cnic.ko 2020-11-20T11:56:41.572 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/genet 2020-11-20T11:56:41.573 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2020-11-20T11:56:41.573 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/broadcom/tg3.ko 2020-11-20T11:56:41.573 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/brocade 2020-11-20T11:56:41.573 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/brocade/bna 2020-11-20T11:56:41.574 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2020-11-20T11:56:41.574 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio 2020-11-20T11:56:41.575 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb 2020-11-20T11:56:41.575 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2020-11-20T11:56:41.575 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb3 2020-11-20T11:56:41.575 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2020-11-20T11:56:41.576 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb4 2020-11-20T11:56:41.576 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2020-11-20T11:56:41.576 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2020-11-20T11:56:41.577 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2020-11-20T11:56:41.577 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/libcxgb 2020-11-20T11:56:41.577 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2020-11-20T11:56:41.577 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/cisco 2020-11-20T11:56:41.578 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/cisco/enic 2020-11-20T11:56:41.578 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2020-11-20T11:56:41.578 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec 2020-11-20T11:56:41.578 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip 2020-11-20T11:56:41.579 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2020-11-20T11:56:41.579 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/de4x5.ko 2020-11-20T11:56:41.579 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2020-11-20T11:56:41.579 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2020-11-20T11:56:41.579 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2020-11-20T11:56:41.580 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2020-11-20T11:56:41.580 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2020-11-20T11:56:41.580 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dlink 2020-11-20T11:56:41.580 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dlink/dl2k.ko 2020-11-20T11:56:41.581 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dlink/sundance.ko 2020-11-20T11:56:41.581 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/dnet.ko 2020-11-20T11:56:41.581 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/emulex 2020-11-20T11:56:41.581 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/emulex/benet 2020-11-20T11:56:41.582 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2020-11-20T11:56:41.582 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/ethoc.ko 2020-11-20T11:56:41.582 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/fealnx.ko 2020-11-20T11:56:41.582 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel 2020-11-20T11:56:41.582 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e100.ko 2020-11-20T11:56:41.583 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e1000 2020-11-20T11:56:41.583 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2020-11-20T11:56:41.583 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e1000e 2020-11-20T11:56:41.583 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2020-11-20T11:56:41.584 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/fm10k 2020-11-20T11:56:41.584 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2020-11-20T11:56:41.584 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/i40e 2020-11-20T11:56:41.584 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2020-11-20T11:56:41.585 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/iavf 2020-11-20T11:56:41.585 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2020-11-20T11:56:41.585 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/igb 2020-11-20T11:56:41.585 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/igb/igb.ko 2020-11-20T11:56:41.586 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/igbvf 2020-11-20T11:56:41.586 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2020-11-20T11:56:41.586 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgb 2020-11-20T11:56:41.586 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2020-11-20T11:56:41.587 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgbe 2020-11-20T11:56:41.587 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2020-11-20T11:56:41.587 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgbevf 2020-11-20T11:56:41.587 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2020-11-20T11:56:41.587 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/jme.ko 2020-11-20T11:56:41.588 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/marvell 2020-11-20T11:56:41.588 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2020-11-20T11:56:41.588 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/marvell/skge.ko 2020-11-20T11:56:41.588 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/marvell/sky2.ko 2020-11-20T11:56:41.589 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox 2020-11-20T11:56:41.589 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx4 2020-11-20T11:56:41.589 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2020-11-20T11:56:41.590 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2020-11-20T11:56:41.590 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx5 2020-11-20T11:56:41.590 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx5/core 2020-11-20T11:56:41.590 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2020-11-20T11:56:41.591 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/micrel 2020-11-20T11:56:41.591 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2020-11-20T11:56:41.591 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/myricom 2020-11-20T11:56:41.591 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/myricom/myri10ge 2020-11-20T11:56:41.592 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2020-11-20T11:56:41.592 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/natsemi 2020-11-20T11:56:41.592 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2020-11-20T11:56:41.592 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2020-11-20T11:56:41.593 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/neterion 2020-11-20T11:56:41.593 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/neterion/s2io.ko 2020-11-20T11:56:41.593 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/neterion/vxge 2020-11-20T11:56:41.594 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/neterion/vxge/vxge.ko 2020-11-20T11:56:41.594 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/nvidia 2020-11-20T11:56:41.594 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2020-11-20T11:56:41.595 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/packetengines 2020-11-20T11:56:41.595 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2020-11-20T11:56:41.595 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2020-11-20T11:56:41.595 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic 2020-11-20T11:56:41.595 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/netxen 2020-11-20T11:56:41.596 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2020-11-20T11:56:41.596 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2020-11-20T11:56:41.596 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/qlcnic 2020-11-20T11:56:41.596 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2020-11-20T11:56:41.597 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/rdc 2020-11-20T11:56:41.597 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/rdc/r6040.ko 2020-11-20T11:56:41.597 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek 2020-11-20T11:56:41.598 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek/8139cp.ko 2020-11-20T11:56:41.598 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek/8139too.ko 2020-11-20T11:56:41.598 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek/atp.ko 2020-11-20T11:56:41.598 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/realtek/r8169.ko 2020-11-20T11:56:41.598 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/rocker 2020-11-20T11:56:41.599 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/rocker/rocker.ko 2020-11-20T11:56:41.599 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sfc 2020-11-20T11:56:41.599 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sfc/sfc.ko 2020-11-20T11:56:41.599 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/silan 2020-11-20T11:56:41.600 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/silan/sc92031.ko 2020-11-20T11:56:41.600 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sis 2020-11-20T11:56:41.600 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sis/sis190.ko 2020-11-20T11:56:41.600 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sis/sis900.ko 2020-11-20T11:56:41.601 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc 2020-11-20T11:56:41.601 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc/epic100.ko 2020-11-20T11:56:41.601 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2020-11-20T11:56:41.601 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2020-11-20T11:56:41.601 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2020-11-20T11:56:41.602 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/stmicro 2020-11-20T11:56:41.602 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/stmicro/stmmac 2020-11-20T11:56:41.602 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2020-11-20T11:56:41.603 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2020-11-20T11:56:41.603 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun 2020-11-20T11:56:41.603 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun/cassini.ko 2020-11-20T11:56:41.603 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun/niu.ko 2020-11-20T11:56:41.604 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun/sungem.ko 2020-11-20T11:56:41.604 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/sun/sunhme.ko 2020-11-20T11:56:41.604 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/tehuti 2020-11-20T11:56:41.604 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2020-11-20T11:56:41.604 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/ti 2020-11-20T11:56:41.605 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/ti/tlan.ko 2020-11-20T11:56:41.605 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/via 2020-11-20T11:56:41.605 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/via/via-rhine.ko 2020-11-20T11:56:41.605 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/via/via-velocity.ko 2020-11-20T11:56:41.606 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/wiznet 2020-11-20T11:56:41.606 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/wiznet/w5100.ko 2020-11-20T11:56:41.606 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/wiznet/w5300.ko 2020-11-20T11:56:41.606 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/xircom 2020-11-20T11:56:41.607 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2020-11-20T11:56:41.607 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/geneve.ko 2020-11-20T11:56:41.607 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/hyperv 2020-11-20T11:56:41.607 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/hyperv/hv_netvsc.ko 2020-11-20T11:56:41.608 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ieee802154 2020-11-20T11:56:41.608 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ieee802154/fakelb.ko 2020-11-20T11:56:41.608 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ifb.ko 2020-11-20T11:56:41.608 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ipvlan 2020-11-20T11:56:41.608 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ipvlan/ipvlan.ko 2020-11-20T11:56:41.609 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/macvlan.ko 2020-11-20T11:56:41.609 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/macvtap.ko 2020-11-20T11:56:41.609 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mdio 2020-11-20T11:56:41.609 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mdio.ko 2020-11-20T11:56:41.610 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2020-11-20T11:56:41.610 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mdio/mdio-bitbang.ko 2020-11-20T11:56:41.610 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/mii.ko 2020-11-20T11:56:41.610 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/net_failover.ko 2020-11-20T11:56:41.611 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/netconsole.ko 2020-11-20T11:56:41.611 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/nlmon.ko 2020-11-20T11:56:41.611 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/pcs 2020-11-20T11:56:41.611 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/pcs/pcs-xpcs.ko 2020-11-20T11:56:41.612 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy 2020-11-20T11:56:41.612 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/amd.ko 2020-11-20T11:56:41.612 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/bcm-phy-lib.ko 2020-11-20T11:56:41.612 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/bcm7xxx.ko 2020-11-20T11:56:41.613 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/bcm87xx.ko 2020-11-20T11:56:41.613 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/broadcom.ko 2020-11-20T11:56:41.613 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/cicada.ko 2020-11-20T11:56:41.613 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/davicom.ko 2020-11-20T11:56:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/dp83640.ko 2020-11-20T11:56:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/et1011c.ko 2020-11-20T11:56:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/icplus.ko 2020-11-20T11:56:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/lxt.ko 2020-11-20T11:56:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/marvell.ko 2020-11-20T11:56:41.615 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/micrel.ko 2020-11-20T11:56:41.615 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/national.ko 2020-11-20T11:56:41.615 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/phylink.ko 2020-11-20T11:56:41.615 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/qsemi.ko 2020-11-20T11:56:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/realtek.ko 2020-11-20T11:56:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/smsc.ko 2020-11-20T11:56:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/ste10Xp.ko 2020-11-20T11:56:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/phy/vitesse.ko 2020-11-20T11:56:41.617 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp 2020-11-20T11:56:41.617 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/bsd_comp.ko 2020-11-20T11:56:41.617 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_async.ko 2020-11-20T11:56:41.617 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_deflate.ko 2020-11-20T11:56:41.618 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_generic.ko 2020-11-20T11:56:41.618 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_mppe.ko 2020-11-20T11:56:41.618 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/ppp_synctty.ko 2020-11-20T11:56:41.618 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/pppoe.ko 2020-11-20T11:56:41.619 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/pppox.ko 2020-11-20T11:56:41.619 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/ppp/pptp.ko 2020-11-20T11:56:41.619 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/slip 2020-11-20T11:56:41.620 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/slip/slhc.ko 2020-11-20T11:56:41.620 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/slip/slip.ko 2020-11-20T11:56:41.620 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/sungem_phy.ko 2020-11-20T11:56:41.620 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/tap.ko 2020-11-20T11:56:41.621 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team 2020-11-20T11:56:41.621 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team.ko 2020-11-20T11:56:41.621 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_activebackup.ko 2020-11-20T11:56:41.621 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_broadcast.ko 2020-11-20T11:56:41.622 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_loadbalance.ko 2020-11-20T11:56:41.622 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_random.ko 2020-11-20T11:56:41.622 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/team/team_mode_roundrobin.ko 2020-11-20T11:56:41.622 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/tun.ko 2020-11-20T11:56:41.623 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/veth.ko 2020-11-20T11:56:41.623 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/virtio_net.ko 2020-11-20T11:56:41.623 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/vmxnet3 2020-11-20T11:56:41.623 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/vmxnet3/vmxnet3.ko 2020-11-20T11:56:41.624 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/vxlan.ko 2020-11-20T11:56:41.624 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/xen-netback 2020-11-20T11:56:41.624 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/xen-netback/xen-netback.ko 2020-11-20T11:56:41.624 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/net/xen-netfront.ko 2020-11-20T11:56:41.625 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ntb 2020-11-20T11:56:41.625 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ntb/ntb.ko 2020-11-20T11:56:41.625 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/nvme 2020-11-20T11:56:41.625 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/nvme/host 2020-11-20T11:56:41.626 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/nvme/host/nvme-core.ko 2020-11-20T11:56:41.626 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/nvme/host/nvme.ko 2020-11-20T11:56:41.626 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport 2020-11-20T11:56:41.627 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport/parport.ko 2020-11-20T11:56:41.627 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport/parport_cs.ko 2020-11-20T11:56:41.627 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport/parport_pc.ko 2020-11-20T11:56:41.627 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/parport/parport_serial.ko 2020-11-20T11:56:41.628 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci 2020-11-20T11:56:41.628 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/hotplug 2020-11-20T11:56:41.628 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2020-11-20T11:56:41.628 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/pcie 2020-11-20T11:56:41.629 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/pcie/aer_inject.ko 2020-11-20T11:56:41.629 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pci/xen-pcifront.ko 2020-11-20T11:56:41.629 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pcmcia 2020-11-20T11:56:41.630 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pcmcia/i82092.ko 2020-11-20T11:56:41.630 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pcmcia/pd6729.ko 2020-11-20T11:56:41.630 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pcmcia/yenta_socket.ko 2020-11-20T11:56:41.630 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pinctrl 2020-11-20T11:56:41.631 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pinctrl/intel 2020-11-20T11:56:41.631 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2020-11-20T11:56:41.631 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform 2020-11-20T11:56:41.631 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86 2020-11-20T11:56:41.632 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/acer-wmi.ko 2020-11-20T11:56:41.632 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/acerhdf.ko 2020-11-20T11:56:41.632 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/alienware-wmi.ko 2020-11-20T11:56:41.633 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/apple-gmux.ko 2020-11-20T11:56:41.633 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/asus-laptop.ko 2020-11-20T11:56:41.633 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/asus-nb-wmi.ko 2020-11-20T11:56:41.633 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/asus-wmi.ko 2020-11-20T11:56:41.634 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/classmate-laptop.ko 2020-11-20T11:56:41.634 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/dcdbas.ko 2020-11-20T11:56:41.634 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/dell-smo8800.ko 2020-11-20T11:56:41.634 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/dell-wmi-aio.ko 2020-11-20T11:56:41.635 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/eeepc-laptop.ko 2020-11-20T11:56:41.635 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/eeepc-wmi.ko 2020-11-20T11:56:41.635 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/fujitsu-laptop.ko 2020-11-20T11:56:41.635 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/fujitsu-tablet.ko 2020-11-20T11:56:41.636 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/hdaps.ko 2020-11-20T11:56:41.636 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/hp-wireless.ko 2020-11-20T11:56:41.636 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/hp-wmi.ko 2020-11-20T11:56:41.636 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/hp_accel.ko 2020-11-20T11:56:41.637 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/intel-rst.ko 2020-11-20T11:56:41.637 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/intel_ips.ko 2020-11-20T11:56:41.637 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/msi-wmi.ko 2020-11-20T11:56:41.637 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/mxm-wmi.ko 2020-11-20T11:56:41.638 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/panasonic-laptop.ko 2020-11-20T11:56:41.638 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/samsung-laptop.ko 2020-11-20T11:56:41.638 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/samsung-q10.ko 2020-11-20T11:56:41.638 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/thinkpad_acpi.ko 2020-11-20T11:56:41.639 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/topstar-laptop.ko 2020-11-20T11:56:41.639 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2020-11-20T11:56:41.639 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/toshiba_haps.ko 2020-11-20T11:56:41.639 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/wmi-bmof.ko 2020-11-20T11:56:41.639 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/platform/x86/wmi.ko 2020-11-20T11:56:41.640 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/power 2020-11-20T11:56:41.640 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/power/supply 2020-11-20T11:56:41.640 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/power/supply/smb347-charger.ko 2020-11-20T11:56:41.641 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/powercap 2020-11-20T11:56:41.641 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/powercap/intel_rapl_common.ko 2020-11-20T11:56:41.641 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/powercap/intel_rapl_msr.ko 2020-11-20T11:56:41.641 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps 2020-11-20T11:56:41.642 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/clients 2020-11-20T11:56:41.642 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/clients/pps-gpio.ko 2020-11-20T11:56:41.642 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/clients/pps-ldisc.ko 2020-11-20T11:56:41.642 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/clients/pps_parport.ko 2020-11-20T11:56:41.643 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/pps/pps_core.ko 2020-11-20T11:56:41.643 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ptp 2020-11-20T11:56:41.643 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ptp/ptp.ko 2020-11-20T11:56:41.643 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ptp/ptp_kvm.ko 2020-11-20T11:56:41.644 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc 2020-11-20T11:56:41.644 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-bq32k.ko 2020-11-20T11:56:41.644 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-bq4802.ko 2020-11-20T11:56:41.644 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1286.ko 2020-11-20T11:56:41.645 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1307.ko 2020-11-20T11:56:41.645 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1374.ko 2020-11-20T11:56:41.645 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1511.ko 2020-11-20T11:56:41.645 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1553.ko 2020-11-20T11:56:41.646 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1672.ko 2020-11-20T11:56:41.646 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds1742.ko 2020-11-20T11:56:41.646 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds2404.ko 2020-11-20T11:56:41.646 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-ds3232.ko 2020-11-20T11:56:41.647 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-em3027.ko 2020-11-20T11:56:41.647 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-fm3130.ko 2020-11-20T11:56:41.647 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-isl12022.ko 2020-11-20T11:56:41.647 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-isl1208.ko 2020-11-20T11:56:41.647 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-m41t80.ko 2020-11-20T11:56:41.648 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-m48t35.ko 2020-11-20T11:56:41.648 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-m48t59.ko 2020-11-20T11:56:41.648 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-max6900.ko 2020-11-20T11:56:41.648 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-msm6242.ko 2020-11-20T11:56:41.649 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf2127.ko 2020-11-20T11:56:41.649 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf85063.ko 2020-11-20T11:56:41.649 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf8523.ko 2020-11-20T11:56:41.649 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf8563.ko 2020-11-20T11:56:41.650 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-pcf8583.ko 2020-11-20T11:56:41.650 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rp5c01.ko 2020-11-20T11:56:41.650 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rs5c372.ko 2020-11-20T11:56:41.650 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rv3029c2.ko 2020-11-20T11:56:41.651 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rx8025.ko 2020-11-20T11:56:41.651 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-rx8581.ko 2020-11-20T11:56:41.651 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-stk17ta8.ko 2020-11-20T11:56:41.652 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-v3020.ko 2020-11-20T11:56:41.652 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/rtc/rtc-x1205.ko 2020-11-20T11:56:41.652 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi 2020-11-20T11:56:41.652 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/3w-9xxx.ko 2020-11-20T11:56:41.652 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/3w-sas.ko 2020-11-20T11:56:41.653 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/3w-xxxx.ko 2020-11-20T11:56:41.653 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/BusLogic.ko 2020-11-20T11:56:41.653 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/a100u2w.ko 2020-11-20T11:56:41.654 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aacraid 2020-11-20T11:56:41.654 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aacraid/aacraid.ko 2020-11-20T11:56:41.654 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/advansys.ko 2020-11-20T11:56:41.654 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aic7xxx 2020-11-20T11:56:41.654 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2020-11-20T11:56:41.655 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2020-11-20T11:56:41.655 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/am53c974.ko 2020-11-20T11:56:41.655 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/arcmsr 2020-11-20T11:56:41.656 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/arcmsr/arcmsr.ko 2020-11-20T11:56:41.656 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/atp870u.ko 2020-11-20T11:56:41.656 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/be2iscsi 2020-11-20T11:56:41.656 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2020-11-20T11:56:41.657 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bfa 2020-11-20T11:56:41.657 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bfa/bfa.ko 2020-11-20T11:56:41.657 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bnx2fc 2020-11-20T11:56:41.657 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2020-11-20T11:56:41.658 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bnx2i 2020-11-20T11:56:41.658 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/bnx2i/bnx2i.ko 2020-11-20T11:56:41.658 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ch.ko 2020-11-20T11:56:41.659 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/csiostor 2020-11-20T11:56:41.659 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/csiostor/csiostor.ko 2020-11-20T11:56:41.659 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi 2020-11-20T11:56:41.660 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/cxgb3i 2020-11-20T11:56:41.660 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2020-11-20T11:56:41.660 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/cxgb4i 2020-11-20T11:56:41.660 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2020-11-20T11:56:41.660 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2020-11-20T11:56:41.661 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/dc395x.ko 2020-11-20T11:56:41.661 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler 2020-11-20T11:56:41.661 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2020-11-20T11:56:41.661 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2020-11-20T11:56:41.662 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2020-11-20T11:56:41.662 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2020-11-20T11:56:41.662 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/dmx3191d.ko 2020-11-20T11:56:41.662 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/esas2r 2020-11-20T11:56:41.663 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/esas2r/esas2r.ko 2020-11-20T11:56:41.663 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/esp_scsi.ko 2020-11-20T11:56:41.663 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fcoe 2020-11-20T11:56:41.663 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fcoe/fcoe.ko 2020-11-20T11:56:41.664 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fcoe/libfcoe.ko 2020-11-20T11:56:41.664 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fnic 2020-11-20T11:56:41.664 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/fnic/fnic.ko 2020-11-20T11:56:41.664 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/gdth.ko 2020-11-20T11:56:41.665 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/hpsa.ko 2020-11-20T11:56:41.665 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/hptiop.ko 2020-11-20T11:56:41.665 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/hv_storvsc.ko 2020-11-20T11:56:41.665 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/initio.ko 2020-11-20T11:56:41.666 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ipr.ko 2020-11-20T11:56:41.666 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ips.ko 2020-11-20T11:56:41.666 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/isci 2020-11-20T11:56:41.666 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/isci/isci.ko 2020-11-20T11:56:41.667 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2020-11-20T11:56:41.667 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/iscsi_tcp.ko 2020-11-20T11:56:41.667 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libfc 2020-11-20T11:56:41.668 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libfc/libfc.ko 2020-11-20T11:56:41.668 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libiscsi.ko 2020-11-20T11:56:41.668 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libiscsi_tcp.ko 2020-11-20T11:56:41.668 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libsas 2020-11-20T11:56:41.669 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/libsas/libsas.ko 2020-11-20T11:56:41.669 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/lpfc 2020-11-20T11:56:41.669 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/lpfc/lpfc.ko 2020-11-20T11:56:41.669 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid 2020-11-20T11:56:41.670 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid.ko 2020-11-20T11:56:41.670 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2020-11-20T11:56:41.670 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2020-11-20T11:56:41.671 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2020-11-20T11:56:41.671 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mpt3sas 2020-11-20T11:56:41.671 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2020-11-20T11:56:41.671 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mvsas 2020-11-20T11:56:41.672 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mvsas/mvsas.ko 2020-11-20T11:56:41.672 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/mvumi.ko 2020-11-20T11:56:41.672 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/pm8001 2020-11-20T11:56:41.672 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/pm8001/pm80xx.ko 2020-11-20T11:56:41.673 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/pmcraid.ko 2020-11-20T11:56:41.673 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla1280.ko 2020-11-20T11:56:41.673 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla2xxx 2020-11-20T11:56:41.673 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2020-11-20T11:56:41.674 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2020-11-20T11:56:41.674 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla4xxx 2020-11-20T11:56:41.674 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2020-11-20T11:56:41.675 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/raid_class.ko 2020-11-20T11:56:41.675 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_debug.ko 2020-11-20T11:56:41.675 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_fc.ko 2020-11-20T11:56:41.676 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_iscsi.ko 2020-11-20T11:56:41.676 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_sas.ko 2020-11-20T11:56:41.676 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_spi.ko 2020-11-20T11:56:41.676 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/scsi_transport_srp.ko 2020-11-20T11:56:41.676 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ses.ko 2020-11-20T11:56:41.677 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/st.ko 2020-11-20T11:56:41.677 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/stex.ko 2020-11-20T11:56:41.677 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/sym53c8xx_2 2020-11-20T11:56:41.677 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2020-11-20T11:56:41.678 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ufs 2020-11-20T11:56:41.678 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ufs/ufshcd-core.ko 2020-11-20T11:56:41.678 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/ufs/ufshcd-pci.ko 2020-11-20T11:56:41.678 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/virtio_scsi.ko 2020-11-20T11:56:41.679 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/vmw_pvscsi.ko 2020-11-20T11:56:41.679 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/scsi/wd719x.ko 2020-11-20T11:56:41.679 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ssb 2020-11-20T11:56:41.679 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/ssb/ssb.ko 2020-11-20T11:56:41.680 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target 2020-11-20T11:56:41.680 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/iscsi 2020-11-20T11:56:41.680 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2020-11-20T11:56:41.680 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/loopback 2020-11-20T11:56:41.680 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/loopback/tcm_loop.ko 2020-11-20T11:56:41.681 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_file.ko 2020-11-20T11:56:41.681 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_iblock.ko 2020-11-20T11:56:41.681 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_mod.ko 2020-11-20T11:56:41.681 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_pscsi.ko 2020-11-20T11:56:41.682 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/target_core_user.ko 2020-11-20T11:56:41.682 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/tcm_fc 2020-11-20T11:56:41.682 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/target/tcm_fc/tcm_fc.ko 2020-11-20T11:56:41.682 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal 2020-11-20T11:56:41.683 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel 2020-11-20T11:56:41.683 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal 2020-11-20T11:56:41.683 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2020-11-20T11:56:41.683 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2020-11-20T11:56:41.684 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2020-11-20T11:56:41.684 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2020-11-20T11:56:41.684 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2020-11-20T11:56:41.684 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2020-11-20T11:56:41.685 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2020-11-20T11:56:41.685 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2020-11-20T11:56:41.685 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2020-11-20T11:56:41.685 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty 2020-11-20T11:56:41.686 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/cyclades.ko 2020-11-20T11:56:41.686 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/ipwireless 2020-11-20T11:56:41.686 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/ipwireless/ipwireless.ko 2020-11-20T11:56:41.686 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/n_gsm.ko 2020-11-20T11:56:41.687 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/n_hdlc.ko 2020-11-20T11:56:41.687 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/nozomi.ko 2020-11-20T11:56:41.687 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/rocket.ko 2020-11-20T11:56:41.687 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial 2020-11-20T11:56:41.687 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/8250 2020-11-20T11:56:41.688 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/8250/serial_cs.ko 2020-11-20T11:56:41.688 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/arc_uart.ko 2020-11-20T11:56:41.688 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/jsm 2020-11-20T11:56:41.688 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/serial/jsm/jsm.ko 2020-11-20T11:56:41.689 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/synclink.ko 2020-11-20T11:56:41.689 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/synclink_gt.ko 2020-11-20T11:56:41.689 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/tty/synclinkmp.ko 2020-11-20T11:56:41.689 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio 2020-11-20T11:56:41.690 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio.ko 2020-11-20T11:56:41.690 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio_aec.ko 2020-11-20T11:56:41.690 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio_cif.ko 2020-11-20T11:56:41.690 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio_pci_generic.ko 2020-11-20T11:56:41.691 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/uio/uio_sercos3.ko 2020-11-20T11:56:41.691 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb 2020-11-20T11:56:41.691 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class 2020-11-20T11:56:41.691 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class/cdc-acm.ko 2020-11-20T11:56:41.692 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class/cdc-wdm.ko 2020-11-20T11:56:41.692 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class/usblp.ko 2020-11-20T11:56:41.692 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/class/usbtmc.ko 2020-11-20T11:56:41.692 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/host 2020-11-20T11:56:41.693 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/host/sl811-hcd.ko 2020-11-20T11:56:41.693 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/host/u132-hcd.ko 2020-11-20T11:56:41.693 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/image 2020-11-20T11:56:41.693 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/image/mdc800.ko 2020-11-20T11:56:41.694 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/image/microtek.ko 2020-11-20T11:56:41.694 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc 2020-11-20T11:56:41.694 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/adutux.ko 2020-11-20T11:56:41.694 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/appledisplay.ko 2020-11-20T11:56:41.694 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/emi26.ko 2020-11-20T11:56:41.695 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/emi62.ko 2020-11-20T11:56:41.695 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/ezusb.ko 2020-11-20T11:56:41.695 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/ftdi-elan.ko 2020-11-20T11:56:41.695 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/idmouse.ko 2020-11-20T11:56:41.696 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/iowarrior.ko 2020-11-20T11:56:41.696 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/isight_firmware.ko 2020-11-20T11:56:41.696 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/ldusb.ko 2020-11-20T11:56:41.696 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/legousbtower.ko 2020-11-20T11:56:41.697 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/sisusbvga 2020-11-20T11:56:41.697 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2020-11-20T11:56:41.697 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/trancevibrator.ko 2020-11-20T11:56:41.697 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/usb3503.ko 2020-11-20T11:56:41.698 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/usblcd.ko 2020-11-20T11:56:41.698 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/usbsevseg.ko 2020-11-20T11:56:41.698 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/uss720.ko 2020-11-20T11:56:41.698 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/misc/yurex.ko 2020-11-20T11:56:41.699 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/phy 2020-11-20T11:56:41.699 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/phy/phy-generic.ko 2020-11-20T11:56:41.699 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial 2020-11-20T11:56:41.699 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/aircable.ko 2020-11-20T11:56:41.700 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ark3116.ko 2020-11-20T11:56:41.700 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/belkin_sa.ko 2020-11-20T11:56:41.700 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ch341.ko 2020-11-20T11:56:41.700 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/cp210x.ko 2020-11-20T11:56:41.701 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/cyberjack.ko 2020-11-20T11:56:41.701 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/cypress_m8.ko 2020-11-20T11:56:41.701 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/digi_acceleport.ko 2020-11-20T11:56:41.701 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/empeg.ko 2020-11-20T11:56:41.702 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ftdi_sio.ko 2020-11-20T11:56:41.702 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/garmin_gps.ko 2020-11-20T11:56:41.702 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/io_edgeport.ko 2020-11-20T11:56:41.702 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/io_ti.ko 2020-11-20T11:56:41.702 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ipaq.ko 2020-11-20T11:56:41.703 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ipw.ko 2020-11-20T11:56:41.703 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ir-usb.ko 2020-11-20T11:56:41.703 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/iuu_phoenix.ko 2020-11-20T11:56:41.703 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/keyspan.ko 2020-11-20T11:56:41.704 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/keyspan_pda.ko 2020-11-20T11:56:41.704 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/kl5kusb105.ko 2020-11-20T11:56:41.704 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/kobil_sct.ko 2020-11-20T11:56:41.704 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/mct_u232.ko 2020-11-20T11:56:41.705 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/mos7720.ko 2020-11-20T11:56:41.705 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/mos7840.ko 2020-11-20T11:56:41.705 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/navman.ko 2020-11-20T11:56:41.705 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/omninet.ko 2020-11-20T11:56:41.706 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/opticon.ko 2020-11-20T11:56:41.706 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/option.ko 2020-11-20T11:56:41.706 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/oti6858.ko 2020-11-20T11:56:41.706 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/pl2303.ko 2020-11-20T11:56:41.706 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/qcaux.ko 2020-11-20T11:56:41.707 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/qcserial.ko 2020-11-20T11:56:41.707 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/quatech2.ko 2020-11-20T11:56:41.707 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/safe_serial.ko 2020-11-20T11:56:41.707 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/sierra.ko 2020-11-20T11:56:41.708 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/spcp8x5.ko 2020-11-20T11:56:41.708 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ssu100.ko 2020-11-20T11:56:41.708 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/symbolserial.ko 2020-11-20T11:56:41.708 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2020-11-20T11:56:41.709 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/usb-serial-simple.ko 2020-11-20T11:56:41.709 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/usb_debug.ko 2020-11-20T11:56:41.709 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/usb_wwan.ko 2020-11-20T11:56:41.710 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/visor.ko 2020-11-20T11:56:41.710 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/whiteheat.ko 2020-11-20T11:56:41.710 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/serial/xsens_mt.ko 2020-11-20T11:56:41.710 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage 2020-11-20T11:56:41.710 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/uas.ko 2020-11-20T11:56:41.711 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-alauda.ko 2020-11-20T11:56:41.711 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-cypress.ko 2020-11-20T11:56:41.711 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-datafab.ko 2020-11-20T11:56:41.711 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-eneub6250.ko 2020-11-20T11:56:41.712 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-freecom.ko 2020-11-20T11:56:41.712 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-isd200.ko 2020-11-20T11:56:41.712 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-jumpshot.ko 2020-11-20T11:56:41.712 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-karma.ko 2020-11-20T11:56:41.713 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-onetouch.ko 2020-11-20T11:56:41.713 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-realtek.ko 2020-11-20T11:56:41.713 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-sddr09.ko 2020-11-20T11:56:41.713 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-sddr55.ko 2020-11-20T11:56:41.714 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/ums-usbat.ko 2020-11-20T11:56:41.714 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/storage/usb-storage.ko 2020-11-20T11:56:41.714 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/usbip 2020-11-20T11:56:41.714 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/usbip/usbip-core.ko 2020-11-20T11:56:41.714 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/usbip/usbip-host.ko 2020-11-20T11:56:41.715 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/usb/usbip/vhci-hcd.ko 2020-11-20T11:56:41.715 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio 2020-11-20T11:56:41.715 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/pci 2020-11-20T11:56:41.715 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/pci/vfio-pci.ko 2020-11-20T11:56:41.716 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/vfio.ko 2020-11-20T11:56:41.716 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/vfio_iommu_type1.ko 2020-11-20T11:56:41.716 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vfio/vfio_virqfd.ko 2020-11-20T11:56:41.716 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost 2020-11-20T11:56:41.717 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost/vhost.ko 2020-11-20T11:56:41.717 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost/vhost_iotlb.ko 2020-11-20T11:56:41.717 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost/vhost_net.ko 2020-11-20T11:56:41.717 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/vhost/vhost_scsi.ko 2020-11-20T11:56:41.718 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video 2020-11-20T11:56:41.718 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video/backlight 2020-11-20T11:56:41.718 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video/backlight/apple_bl.ko 2020-11-20T11:56:41.718 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video/backlight/lcd.ko 2020-11-20T11:56:41.719 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/video/backlight/platform_lcd.ko 2020-11-20T11:56:41.719 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio 2020-11-20T11:56:41.719 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio.ko 2020-11-20T11:56:41.720 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio_balloon.ko 2020-11-20T11:56:41.720 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio_mmio.ko 2020-11-20T11:56:41.720 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio_pci.ko 2020-11-20T11:56:41.720 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/virtio/virtio_ring.ko 2020-11-20T11:56:41.721 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1 2020-11-20T11:56:41.721 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/masters 2020-11-20T11:56:41.721 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/masters/ds1wm.ko 2020-11-20T11:56:41.721 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/masters/ds2482.ko 2020-11-20T11:56:41.722 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/masters/ds2490.ko 2020-11-20T11:56:41.722 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves 2020-11-20T11:56:41.722 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2406.ko 2020-11-20T11:56:41.722 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2408.ko 2020-11-20T11:56:41.722 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2413.ko 2020-11-20T11:56:41.723 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2423.ko 2020-11-20T11:56:41.723 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2431.ko 2020-11-20T11:56:41.723 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2433.ko 2020-11-20T11:56:41.724 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2780.ko 2020-11-20T11:56:41.724 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds2781.ko 2020-11-20T11:56:41.724 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_ds28e04.ko 2020-11-20T11:56:41.724 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_smem.ko 2020-11-20T11:56:41.725 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/slaves/w1_therm.ko 2020-11-20T11:56:41.725 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/w1/wire.ko 2020-11-20T11:56:41.725 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog 2020-11-20T11:56:41.725 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/alim1535_wdt.ko 2020-11-20T11:56:41.726 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/alim7101_wdt.ko 2020-11-20T11:56:41.726 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/f71808e_wdt.ko 2020-11-20T11:56:41.726 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/hpwdt.ko 2020-11-20T11:56:41.726 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/i6300esb.ko 2020-11-20T11:56:41.726 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/iTCO_vendor_support.ko 2020-11-20T11:56:41.727 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/iTCO_wdt.ko 2020-11-20T11:56:41.727 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/ib700wdt.ko 2020-11-20T11:56:41.727 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/ibmasr.ko 2020-11-20T11:56:41.727 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/ie6xx_wdt.ko 2020-11-20T11:56:41.728 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/it8712f_wdt.ko 2020-11-20T11:56:41.728 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/it87_wdt.ko 2020-11-20T11:56:41.728 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/machzwd.ko 2020-11-20T11:56:41.728 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/nv_tco.ko 2020-11-20T11:56:41.729 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/pcwd_pci.ko 2020-11-20T11:56:41.729 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/pcwd_usb.ko 2020-11-20T11:56:41.729 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2020-11-20T11:56:41.729 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/sch311x_wdt.ko 2020-11-20T11:56:41.730 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/softdog.ko 2020-11-20T11:56:41.730 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/sp5100_tco.ko 2020-11-20T11:56:41.730 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/via_wdt.ko 2020-11-20T11:56:41.730 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/w83627hf_wdt.ko 2020-11-20T11:56:41.731 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/w83877f_wdt.ko 2020-11-20T11:56:41.731 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/w83977f_wdt.ko 2020-11-20T11:56:41.731 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/wdt_pci.ko 2020-11-20T11:56:41.731 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/watchdog/xen_wdt.ko 2020-11-20T11:56:41.732 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen 2020-11-20T11:56:41.732 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-acpi-processor.ko 2020-11-20T11:56:41.732 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-evtchn.ko 2020-11-20T11:56:41.732 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-gntalloc.ko 2020-11-20T11:56:41.733 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-gntdev.ko 2020-11-20T11:56:41.733 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-pciback 2020-11-20T11:56:41.733 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2020-11-20T11:56:41.733 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xen-privcmd.ko 2020-11-20T11:56:41.733 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xenfs 2020-11-20T11:56:41.734 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/drivers/xen/xenfs/xenfs.ko 2020-11-20T11:56:41.734 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs 2020-11-20T11:56:41.734 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/9p 2020-11-20T11:56:41.735 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/9p/9p.ko 2020-11-20T11:56:41.735 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/affs 2020-11-20T11:56:41.735 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/affs/affs.ko 2020-11-20T11:56:41.735 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/befs 2020-11-20T11:56:41.735 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/befs/befs.ko 2020-11-20T11:56:41.736 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/binfmt_misc.ko 2020-11-20T11:56:41.736 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/btrfs 2020-11-20T11:56:41.736 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/btrfs/btrfs.ko 2020-11-20T11:56:41.736 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cachefiles 2020-11-20T11:56:41.737 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cachefiles/cachefiles.ko 2020-11-20T11:56:41.737 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ceph 2020-11-20T11:56:41.737 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ceph/ceph.ko 2020-11-20T11:56:41.737 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cifs 2020-11-20T11:56:41.738 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cifs/cifs.ko 2020-11-20T11:56:41.738 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/coda 2020-11-20T11:56:41.738 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/coda/coda.ko 2020-11-20T11:56:41.738 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cramfs 2020-11-20T11:56:41.739 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/cramfs/cramfs.ko 2020-11-20T11:56:41.739 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/dlm 2020-11-20T11:56:41.739 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/dlm/dlm.ko 2020-11-20T11:56:41.740 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ecryptfs 2020-11-20T11:56:41.740 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ecryptfs/ecryptfs.ko 2020-11-20T11:56:41.740 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/f2fs 2020-11-20T11:56:41.740 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/f2fs/f2fs.ko 2020-11-20T11:56:41.741 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fat 2020-11-20T11:56:41.741 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fat/fat.ko 2020-11-20T11:56:41.741 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fat/msdos.ko 2020-11-20T11:56:41.741 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fat/vfat.ko 2020-11-20T11:56:41.742 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fscache 2020-11-20T11:56:41.742 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fscache/fscache.ko 2020-11-20T11:56:41.742 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fuse 2020-11-20T11:56:41.743 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fuse/cuse.ko 2020-11-20T11:56:41.743 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/fuse/fuse.ko 2020-11-20T11:56:41.743 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/gfs2 2020-11-20T11:56:41.743 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/gfs2/gfs2.ko 2020-11-20T11:56:41.744 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/hfs 2020-11-20T11:56:41.744 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/hfs/hfs.ko 2020-11-20T11:56:41.744 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/hfsplus 2020-11-20T11:56:41.744 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/hfsplus/hfsplus.ko 2020-11-20T11:56:41.745 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/isofs 2020-11-20T11:56:41.745 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/isofs/isofs.ko 2020-11-20T11:56:41.745 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/jfs 2020-11-20T11:56:41.745 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/jfs/jfs.ko 2020-11-20T11:56:41.746 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/lockd 2020-11-20T11:56:41.746 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/lockd/lockd.ko 2020-11-20T11:56:41.746 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/minix 2020-11-20T11:56:41.746 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/minix/minix.ko 2020-11-20T11:56:41.747 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs 2020-11-20T11:56:41.747 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/blocklayout 2020-11-20T11:56:41.747 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2020-11-20T11:56:41.747 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/filelayout 2020-11-20T11:56:41.748 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2020-11-20T11:56:41.748 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/flexfilelayout 2020-11-20T11:56:41.748 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2020-11-20T11:56:41.748 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/nfs.ko 2020-11-20T11:56:41.748 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/nfsv3.ko 2020-11-20T11:56:41.749 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs/nfsv4.ko 2020-11-20T11:56:41.749 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs_common 2020-11-20T11:56:41.749 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs_common/grace.ko 2020-11-20T11:56:41.749 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs_common/nfs_acl.ko 2020-11-20T11:56:41.750 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfs_common/nfs_ssc.ko 2020-11-20T11:56:41.750 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfsd 2020-11-20T11:56:41.750 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nfsd/nfsd.ko 2020-11-20T11:56:41.750 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nilfs2 2020-11-20T11:56:41.751 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nilfs2/nilfs2.ko 2020-11-20T11:56:41.751 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls 2020-11-20T11:56:41.751 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-celtic.ko 2020-11-20T11:56:41.751 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-centeuro.ko 2020-11-20T11:56:41.751 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-croatian.ko 2020-11-20T11:56:41.752 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-cyrillic.ko 2020-11-20T11:56:41.752 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-gaelic.ko 2020-11-20T11:56:41.752 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-greek.ko 2020-11-20T11:56:41.752 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-iceland.ko 2020-11-20T11:56:41.753 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-inuit.ko 2020-11-20T11:56:41.753 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-roman.ko 2020-11-20T11:56:41.753 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-romanian.ko 2020-11-20T11:56:41.753 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/mac-turkish.ko 2020-11-20T11:56:41.754 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp1250.ko 2020-11-20T11:56:41.754 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp1251.ko 2020-11-20T11:56:41.754 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp1255.ko 2020-11-20T11:56:41.754 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp737.ko 2020-11-20T11:56:41.755 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp775.ko 2020-11-20T11:56:41.755 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp850.ko 2020-11-20T11:56:41.755 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp852.ko 2020-11-20T11:56:41.755 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp855.ko 2020-11-20T11:56:41.756 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp857.ko 2020-11-20T11:56:41.756 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp860.ko 2020-11-20T11:56:41.756 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp861.ko 2020-11-20T11:56:41.756 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp862.ko 2020-11-20T11:56:41.757 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp863.ko 2020-11-20T11:56:41.757 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp864.ko 2020-11-20T11:56:41.757 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp865.ko 2020-11-20T11:56:41.757 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp866.ko 2020-11-20T11:56:41.758 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp869.ko 2020-11-20T11:56:41.758 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp874.ko 2020-11-20T11:56:41.758 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp932.ko 2020-11-20T11:56:41.759 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp936.ko 2020-11-20T11:56:41.759 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp949.ko 2020-11-20T11:56:41.759 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_cp950.ko 2020-11-20T11:56:41.759 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_euc-jp.ko 2020-11-20T11:56:41.760 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-1.ko 2020-11-20T11:56:41.760 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-13.ko 2020-11-20T11:56:41.760 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-14.ko 2020-11-20T11:56:41.760 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-15.ko 2020-11-20T11:56:41.761 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-2.ko 2020-11-20T11:56:41.761 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-3.ko 2020-11-20T11:56:41.761 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-4.ko 2020-11-20T11:56:41.761 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-5.ko 2020-11-20T11:56:41.762 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-6.ko 2020-11-20T11:56:41.762 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-7.ko 2020-11-20T11:56:41.762 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_iso8859-9.ko 2020-11-20T11:56:41.762 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_koi8-r.ko 2020-11-20T11:56:41.763 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_koi8-ru.ko 2020-11-20T11:56:41.763 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_koi8-u.ko 2020-11-20T11:56:41.763 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/nls/nls_utf8.ko 2020-11-20T11:56:41.763 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2 2020-11-20T11:56:41.764 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/cluster 2020-11-20T11:56:41.764 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2020-11-20T11:56:41.764 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/dlm 2020-11-20T11:56:41.765 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2020-11-20T11:56:41.765 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/dlmfs 2020-11-20T11:56:41.765 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2020-11-20T11:56:41.765 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/ocfs2.ko 2020-11-20T11:56:41.766 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2020-11-20T11:56:41.766 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/ocfs2_stack_user.ko 2020-11-20T11:56:41.766 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ocfs2/ocfs2_stackglue.ko 2020-11-20T11:56:41.766 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/overlayfs 2020-11-20T11:56:41.766 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/overlayfs/overlay.ko 2020-11-20T11:56:41.767 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/pstore 2020-11-20T11:56:41.767 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/pstore/ramoops.ko 2020-11-20T11:56:41.767 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/reiserfs 2020-11-20T11:56:41.767 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/reiserfs/reiserfs.ko 2020-11-20T11:56:41.768 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/romfs 2020-11-20T11:56:41.768 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/romfs/romfs.ko 2020-11-20T11:56:41.768 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/squashfs 2020-11-20T11:56:41.769 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/squashfs/squashfs.ko 2020-11-20T11:56:41.769 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/sysv 2020-11-20T11:56:41.769 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/sysv/sysv.ko 2020-11-20T11:56:41.769 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/udf 2020-11-20T11:56:41.770 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/udf/udf.ko 2020-11-20T11:56:41.770 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ufs 2020-11-20T11:56:41.770 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/ufs/ufs.ko 2020-11-20T11:56:41.770 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/xfs 2020-11-20T11:56:41.771 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/fs/xfs/xfs.ko 2020-11-20T11:56:41.771 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel 2020-11-20T11:56:41.771 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/rcu 2020-11-20T11:56:41.771 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/rcu/rcutorture.ko 2020-11-20T11:56:41.772 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/torture.ko 2020-11-20T11:56:41.772 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/trace 2020-11-20T11:56:41.772 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/kernel/trace/ring_buffer_benchmark.ko 2020-11-20T11:56:41.772 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib 2020-11-20T11:56:41.773 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crc-itu-t.ko 2020-11-20T11:56:41.773 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crc64.ko 2020-11-20T11:56:41.773 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crc8.ko 2020-11-20T11:56:41.773 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crypto 2020-11-20T11:56:41.774 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crypto/libarc4.ko 2020-11-20T11:56:41.774 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/crypto/libdes.ko 2020-11-20T11:56:41.774 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/libcrc32c.ko 2020-11-20T11:56:41.775 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/lru_cache.ko 2020-11-20T11:56:41.775 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/lz4 2020-11-20T11:56:41.775 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/lz4/lz4_compress.ko 2020-11-20T11:56:41.776 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/lz4/lz4hc_compress.ko 2020-11-20T11:56:41.776 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/math 2020-11-20T11:56:41.776 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/math/cordic.ko 2020-11-20T11:56:41.776 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/raid6 2020-11-20T11:56:41.777 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/raid6/raid6_pq.ko 2020-11-20T11:56:41.777 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/reed_solomon 2020-11-20T11:56:41.777 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/reed_solomon/reed_solomon.ko 2020-11-20T11:56:41.778 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/ts_bm.ko 2020-11-20T11:56:41.778 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/ts_fsm.ko 2020-11-20T11:56:41.778 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/ts_kmp.ko 2020-11-20T11:56:41.778 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/zstd 2020-11-20T11:56:41.779 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/lib/zstd/zstd_compress.ko 2020-11-20T11:56:41.779 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/mm 2020-11-20T11:56:41.779 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/mm/hwpoison-inject.ko 2020-11-20T11:56:41.779 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net 2020-11-20T11:56:41.780 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan 2020-11-20T11:56:41.780 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/6lowpan.ko 2020-11-20T11:56:41.780 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_dest.ko 2020-11-20T11:56:41.780 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_fragment.ko 2020-11-20T11:56:41.781 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_hop.ko 2020-11-20T11:56:41.781 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_ipv6.ko 2020-11-20T11:56:41.781 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_mobility.ko 2020-11-20T11:56:41.781 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_routing.ko 2020-11-20T11:56:41.782 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/6lowpan/nhc_udp.ko 2020-11-20T11:56:41.782 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802 2020-11-20T11:56:41.782 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/garp.ko 2020-11-20T11:56:41.782 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/mrp.ko 2020-11-20T11:56:41.783 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/p8022.ko 2020-11-20T11:56:41.783 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/psnap.ko 2020-11-20T11:56:41.783 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/802/stp.ko 2020-11-20T11:56:41.783 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/8021q 2020-11-20T11:56:41.784 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/8021q/8021q.ko 2020-11-20T11:56:41.784 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/9p 2020-11-20T11:56:41.784 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/9p/9pnet.ko 2020-11-20T11:56:41.784 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/9p/9pnet_rdma.ko 2020-11-20T11:56:41.785 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/9p/9pnet_virtio.ko 2020-11-20T11:56:41.785 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/batman-adv 2020-11-20T11:56:41.785 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/batman-adv/batman-adv.ko 2020-11-20T11:56:41.786 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge 2020-11-20T11:56:41.786 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/br_netfilter.ko 2020-11-20T11:56:41.786 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/bridge.ko 2020-11-20T11:56:41.786 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter 2020-11-20T11:56:41.787 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_802_3.ko 2020-11-20T11:56:41.787 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_among.ko 2020-11-20T11:56:41.787 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_arp.ko 2020-11-20T11:56:41.788 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_arpreply.ko 2020-11-20T11:56:41.788 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_dnat.ko 2020-11-20T11:56:41.788 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_ip.ko 2020-11-20T11:56:41.789 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_ip6.ko 2020-11-20T11:56:41.789 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_limit.ko 2020-11-20T11:56:41.789 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_log.ko 2020-11-20T11:56:41.789 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_mark.ko 2020-11-20T11:56:41.790 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_mark_m.ko 2020-11-20T11:56:41.790 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_nflog.ko 2020-11-20T11:56:41.790 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_pkttype.ko 2020-11-20T11:56:41.791 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_redirect.ko 2020-11-20T11:56:41.791 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_snat.ko 2020-11-20T11:56:41.791 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_stp.ko 2020-11-20T11:56:41.791 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebt_vlan.ko 2020-11-20T11:56:41.792 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebtable_broute.ko 2020-11-20T11:56:41.792 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebtable_filter.ko 2020-11-20T11:56:41.792 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebtable_nat.ko 2020-11-20T11:56:41.792 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/ebtables.ko 2020-11-20T11:56:41.793 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2020-11-20T11:56:41.793 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/nf_log_bridge.ko 2020-11-20T11:56:41.793 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2020-11-20T11:56:41.793 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2020-11-20T11:56:41.794 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ceph 2020-11-20T11:56:41.794 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ceph/libceph.ko 2020-11-20T11:56:41.794 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/core 2020-11-20T11:56:41.795 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/core/failover.ko 2020-11-20T11:56:41.795 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/core/pktgen.ko 2020-11-20T11:56:41.795 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp 2020-11-20T11:56:41.796 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp/dccp.ko 2020-11-20T11:56:41.796 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp/dccp_diag.ko 2020-11-20T11:56:41.796 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp/dccp_ipv4.ko 2020-11-20T11:56:41.797 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dccp/dccp_ipv6.ko 2020-11-20T11:56:41.797 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dns_resolver 2020-11-20T11:56:41.797 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dns_resolver/dns_resolver.ko 2020-11-20T11:56:41.797 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa 2020-11-20T11:56:41.798 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/dsa_core.ko 2020-11-20T11:56:41.798 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/tag_brcm.ko 2020-11-20T11:56:41.798 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/tag_dsa.ko 2020-11-20T11:56:41.798 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/tag_edsa.ko 2020-11-20T11:56:41.799 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/dsa/tag_trailer.ko 2020-11-20T11:56:41.799 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154 2020-11-20T11:56:41.799 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154/6lowpan 2020-11-20T11:56:41.799 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2020-11-20T11:56:41.800 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154/ieee802154.ko 2020-11-20T11:56:41.800 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ieee802154/ieee802154_socket.ko 2020-11-20T11:56:41.800 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4 2020-11-20T11:56:41.801 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ah4.ko 2020-11-20T11:56:41.801 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/esp4.ko 2020-11-20T11:56:41.801 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/fou.ko 2020-11-20T11:56:41.801 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/gre.ko 2020-11-20T11:56:41.802 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/inet_diag.ko 2020-11-20T11:56:41.802 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ip_gre.ko 2020-11-20T11:56:41.802 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ip_tunnel.ko 2020-11-20T11:56:41.802 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ip_vti.ko 2020-11-20T11:56:41.803 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ipcomp.ko 2020-11-20T11:56:41.803 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/ipip.ko 2020-11-20T11:56:41.803 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter 2020-11-20T11:56:41.803 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/arp_tables.ko 2020-11-20T11:56:41.803 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/arpt_mangle.ko 2020-11-20T11:56:41.804 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/arptable_filter.ko 2020-11-20T11:56:41.804 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2020-11-20T11:56:41.804 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_ECN.ko 2020-11-20T11:56:41.805 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2020-11-20T11:56:41.805 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_ah.ko 2020-11-20T11:56:41.806 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2020-11-20T11:56:41.806 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/iptable_mangle.ko 2020-11-20T11:56:41.806 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/iptable_nat.ko 2020-11-20T11:56:41.806 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/iptable_raw.ko 2020-11-20T11:56:41.807 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/iptable_security.ko 2020-11-20T11:56:41.807 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2020-11-20T11:56:41.807 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2020-11-20T11:56:41.807 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_log_arp.ko 2020-11-20T11:56:41.808 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_log_ipv4.ko 2020-11-20T11:56:41.808 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2020-11-20T11:56:41.808 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2020-11-20T11:56:41.809 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2020-11-20T11:56:41.809 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2020-11-20T11:56:41.809 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2020-11-20T11:56:41.809 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2020-11-20T11:56:41.810 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2020-11-20T11:56:41.810 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2020-11-20T11:56:41.810 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_bic.ko 2020-11-20T11:56:41.810 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_dctcp.ko 2020-11-20T11:56:41.811 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_diag.ko 2020-11-20T11:56:41.811 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_highspeed.ko 2020-11-20T11:56:41.811 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_htcp.ko 2020-11-20T11:56:41.811 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_hybla.ko 2020-11-20T11:56:41.812 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_illinois.ko 2020-11-20T11:56:41.812 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_lp.ko 2020-11-20T11:56:41.812 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_scalable.ko 2020-11-20T11:56:41.813 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_vegas.ko 2020-11-20T11:56:41.813 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_veno.ko 2020-11-20T11:56:41.813 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_westwood.ko 2020-11-20T11:56:41.814 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tcp_yeah.ko 2020-11-20T11:56:41.814 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/tunnel4.ko 2020-11-20T11:56:41.814 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/udp_diag.ko 2020-11-20T11:56:41.814 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/udp_tunnel.ko 2020-11-20T11:56:41.815 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv4/xfrm4_tunnel.ko 2020-11-20T11:56:41.815 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6 2020-11-20T11:56:41.815 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ah6.ko 2020-11-20T11:56:41.815 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/esp6.ko 2020-11-20T11:56:41.816 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/fou6.ko 2020-11-20T11:56:41.816 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ip6_tunnel.ko 2020-11-20T11:56:41.816 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ip6_udp_tunnel.ko 2020-11-20T11:56:41.817 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ip6_vti.ko 2020-11-20T11:56:41.817 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/ipcomp6.ko 2020-11-20T11:56:41.817 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter 2020-11-20T11:56:41.817 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6_tables.ko 2020-11-20T11:56:41.818 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2020-11-20T11:56:41.818 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2020-11-20T11:56:41.818 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_ah.ko 2020-11-20T11:56:41.818 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2020-11-20T11:56:41.819 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_frag.ko 2020-11-20T11:56:41.819 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2020-11-20T11:56:41.819 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2020-11-20T11:56:41.819 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_mh.ko 2020-11-20T11:56:41.820 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2020-11-20T11:56:41.820 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6t_rt.ko 2020-11-20T11:56:41.820 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_filter.ko 2020-11-20T11:56:41.820 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2020-11-20T11:56:41.821 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_nat.ko 2020-11-20T11:56:41.821 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_raw.ko 2020-11-20T11:56:41.821 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/ip6table_security.ko 2020-11-20T11:56:41.822 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2020-11-20T11:56:41.822 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2020-11-20T11:56:41.822 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_flow_table_ipv6.ko 2020-11-20T11:56:41.823 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_log_ipv6.ko 2020-11-20T11:56:41.823 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2020-11-20T11:56:41.823 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2020-11-20T11:56:41.823 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2020-11-20T11:56:41.824 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2020-11-20T11:56:41.824 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2020-11-20T11:56:41.824 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2020-11-20T11:56:41.825 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/sit.ko 2020-11-20T11:56:41.825 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/tunnel6.ko 2020-11-20T11:56:41.825 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/ipv6/xfrm6_tunnel.ko 2020-11-20T11:56:41.825 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/key 2020-11-20T11:56:41.826 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/key/af_key.ko 2020-11-20T11:56:41.826 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp 2020-11-20T11:56:41.826 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_core.ko 2020-11-20T11:56:41.826 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_debugfs.ko 2020-11-20T11:56:41.827 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_eth.ko 2020-11-20T11:56:41.827 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_ip.ko 2020-11-20T11:56:41.827 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_ip6.ko 2020-11-20T11:56:41.827 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_netlink.ko 2020-11-20T11:56:41.828 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/l2tp/l2tp_ppp.ko 2020-11-20T11:56:41.828 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/llc 2020-11-20T11:56:41.828 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/llc/llc.ko 2020-11-20T11:56:41.828 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/mac802154 2020-11-20T11:56:41.829 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/mac802154/mac802154.ko 2020-11-20T11:56:41.829 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/mpls 2020-11-20T11:56:41.829 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/mpls/mpls_gso.ko 2020-11-20T11:56:41.829 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter 2020-11-20T11:56:41.830 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset 2020-11-20T11:56:41.830 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set.ko 2020-11-20T11:56:41.831 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2020-11-20T11:56:41.831 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2020-11-20T11:56:41.831 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2020-11-20T11:56:41.831 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2020-11-20T11:56:41.832 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2020-11-20T11:56:41.832 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2020-11-20T11:56:41.832 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2020-11-20T11:56:41.832 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2020-11-20T11:56:41.833 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2020-11-20T11:56:41.833 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2020-11-20T11:56:41.833 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2020-11-20T11:56:41.833 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2020-11-20T11:56:41.833 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2020-11-20T11:56:41.834 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2020-11-20T11:56:41.834 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipset/ip_set_list_set.ko 2020-11-20T11:56:41.834 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs 2020-11-20T11:56:41.834 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs.ko 2020-11-20T11:56:41.835 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2020-11-20T11:56:41.835 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2020-11-20T11:56:41.835 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2020-11-20T11:56:41.835 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2020-11-20T11:56:41.836 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2020-11-20T11:56:41.836 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2020-11-20T11:56:41.836 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2020-11-20T11:56:41.836 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2020-11-20T11:56:41.837 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2020-11-20T11:56:41.837 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2020-11-20T11:56:41.837 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2020-11-20T11:56:41.837 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2020-11-20T11:56:41.838 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2020-11-20T11:56:41.838 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conncount.ko 2020-11-20T11:56:41.838 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack.ko 2020-11-20T11:56:41.838 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_amanda.ko 2020-11-20T11:56:41.839 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_broadcast.ko 2020-11-20T11:56:41.839 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_ftp.ko 2020-11-20T11:56:41.839 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_h323.ko 2020-11-20T11:56:41.840 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_irc.ko 2020-11-20T11:56:41.840 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2020-11-20T11:56:41.840 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_netlink.ko 2020-11-20T11:56:41.841 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_pptp.ko 2020-11-20T11:56:41.841 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_sane.ko 2020-11-20T11:56:41.841 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_sip.ko 2020-11-20T11:56:41.841 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_snmp.ko 2020-11-20T11:56:41.841 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_conntrack_tftp.ko 2020-11-20T11:56:41.842 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_dup_netdev.ko 2020-11-20T11:56:41.842 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_flow_table.ko 2020-11-20T11:56:41.842 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_flow_table_inet.ko 2020-11-20T11:56:41.842 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_log_common.ko 2020-11-20T11:56:41.843 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat.ko 2020-11-20T11:56:41.843 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_amanda.ko 2020-11-20T11:56:41.843 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_ftp.ko 2020-11-20T11:56:41.843 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_irc.ko 2020-11-20T11:56:41.844 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_sip.ko 2020-11-20T11:56:41.844 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_nat_tftp.ko 2020-11-20T11:56:41.844 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_synproxy_core.ko 2020-11-20T11:56:41.844 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nf_tables.ko 2020-11-20T11:56:41.845 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink.ko 2020-11-20T11:56:41.845 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink_acct.ko 2020-11-20T11:56:41.845 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink_log.ko 2020-11-20T11:56:41.846 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink_osf.ko 2020-11-20T11:56:41.846 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nfnetlink_queue.ko 2020-11-20T11:56:41.846 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_chain_nat.ko 2020-11-20T11:56:41.846 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_compat.ko 2020-11-20T11:56:41.847 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_counter.ko 2020-11-20T11:56:41.847 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_ct.ko 2020-11-20T11:56:41.847 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_dup_netdev.ko 2020-11-20T11:56:41.848 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_fib.ko 2020-11-20T11:56:41.848 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_fwd_netdev.ko 2020-11-20T11:56:41.848 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_hash.ko 2020-11-20T11:56:41.849 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_limit.ko 2020-11-20T11:56:41.849 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_log.ko 2020-11-20T11:56:41.849 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_masq.ko 2020-11-20T11:56:41.849 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_nat.ko 2020-11-20T11:56:41.850 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_numgen.ko 2020-11-20T11:56:41.850 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_objref.ko 2020-11-20T11:56:41.850 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_queue.ko 2020-11-20T11:56:41.850 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_quota.ko 2020-11-20T11:56:41.850 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_redir.ko 2020-11-20T11:56:41.851 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_reject.ko 2020-11-20T11:56:41.851 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_reject_inet.ko 2020-11-20T11:56:41.851 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_socket.ko 2020-11-20T11:56:41.851 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/nft_tunnel.ko 2020-11-20T11:56:41.852 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_AUDIT.ko 2020-11-20T11:56:41.852 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_CHECKSUM.ko 2020-11-20T11:56:41.852 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_CLASSIFY.ko 2020-11-20T11:56:41.852 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_CONNSECMARK.ko 2020-11-20T11:56:41.853 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_CT.ko 2020-11-20T11:56:41.853 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_DSCP.ko 2020-11-20T11:56:41.853 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_HL.ko 2020-11-20T11:56:41.853 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_HMARK.ko 2020-11-20T11:56:41.854 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_IDLETIMER.ko 2020-11-20T11:56:41.854 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_LED.ko 2020-11-20T11:56:41.854 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_LOG.ko 2020-11-20T11:56:41.854 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_MASQUERADE.ko 2020-11-20T11:56:41.855 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_NETMAP.ko 2020-11-20T11:56:41.855 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_NFLOG.ko 2020-11-20T11:56:41.855 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_NFQUEUE.ko 2020-11-20T11:56:41.856 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_RATEEST.ko 2020-11-20T11:56:41.856 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_REDIRECT.ko 2020-11-20T11:56:41.856 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_SECMARK.ko 2020-11-20T11:56:41.856 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TCPMSS.ko 2020-11-20T11:56:41.857 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2020-11-20T11:56:41.857 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TEE.ko 2020-11-20T11:56:41.857 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TPROXY.ko 2020-11-20T11:56:41.858 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_TRACE.ko 2020-11-20T11:56:41.858 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_addrtype.ko 2020-11-20T11:56:41.858 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_bpf.ko 2020-11-20T11:56:41.858 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_cgroup.ko 2020-11-20T11:56:41.859 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_cluster.ko 2020-11-20T11:56:41.859 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_comment.ko 2020-11-20T11:56:41.859 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_connbytes.ko 2020-11-20T11:56:41.859 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_connlabel.ko 2020-11-20T11:56:41.860 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_connlimit.ko 2020-11-20T11:56:41.860 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_connmark.ko 2020-11-20T11:56:41.860 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_conntrack.ko 2020-11-20T11:56:41.860 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_cpu.ko 2020-11-20T11:56:41.861 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_dccp.ko 2020-11-20T11:56:41.861 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_devgroup.ko 2020-11-20T11:56:41.861 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_dscp.ko 2020-11-20T11:56:41.861 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_ecn.ko 2020-11-20T11:56:41.862 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_esp.ko 2020-11-20T11:56:41.862 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_hashlimit.ko 2020-11-20T11:56:41.862 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_helper.ko 2020-11-20T11:56:41.863 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_hl.ko 2020-11-20T11:56:41.863 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_ipcomp.ko 2020-11-20T11:56:41.863 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_iprange.ko 2020-11-20T11:56:41.863 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_ipvs.ko 2020-11-20T11:56:41.864 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_l2tp.ko 2020-11-20T11:56:41.864 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_length.ko 2020-11-20T11:56:41.864 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_limit.ko 2020-11-20T11:56:41.864 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_mac.ko 2020-11-20T11:56:41.865 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_mark.ko 2020-11-20T11:56:41.865 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_multiport.ko 2020-11-20T11:56:41.865 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_nat.ko 2020-11-20T11:56:41.866 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_nfacct.ko 2020-11-20T11:56:41.866 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_osf.ko 2020-11-20T11:56:41.866 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_owner.ko 2020-11-20T11:56:41.867 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_physdev.ko 2020-11-20T11:56:41.867 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_pkttype.ko 2020-11-20T11:56:41.867 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_policy.ko 2020-11-20T11:56:41.867 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_quota.ko 2020-11-20T11:56:41.868 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_rateest.ko 2020-11-20T11:56:41.868 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_realm.ko 2020-11-20T11:56:41.868 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_recent.ko 2020-11-20T11:56:41.868 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_sctp.ko 2020-11-20T11:56:41.869 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_set.ko 2020-11-20T11:56:41.869 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_socket.ko 2020-11-20T11:56:41.869 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_state.ko 2020-11-20T11:56:41.869 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_statistic.ko 2020-11-20T11:56:41.870 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_string.ko 2020-11-20T11:56:41.870 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_tcpmss.ko 2020-11-20T11:56:41.870 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_time.ko 2020-11-20T11:56:41.871 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netfilter/xt_u32.ko 2020-11-20T11:56:41.871 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netlink 2020-11-20T11:56:41.871 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/netlink/netlink_diag.ko 2020-11-20T11:56:41.871 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/nsh 2020-11-20T11:56:41.872 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/nsh/nsh.ko 2020-11-20T11:56:41.872 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch 2020-11-20T11:56:41.872 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch/openvswitch.ko 2020-11-20T11:56:41.872 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch/vport-geneve.ko 2020-11-20T11:56:41.873 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch/vport-gre.ko 2020-11-20T11:56:41.873 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/openvswitch/vport-vxlan.ko 2020-11-20T11:56:41.873 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/packet 2020-11-20T11:56:41.873 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/packet/af_packet_diag.ko 2020-11-20T11:56:41.874 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/rds 2020-11-20T11:56:41.874 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/rds/rds.ko 2020-11-20T11:56:41.874 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/rds/rds_rdma.ko 2020-11-20T11:56:41.874 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/rds/rds_tcp.ko 2020-11-20T11:56:41.875 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched 2020-11-20T11:56:41.875 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_csum.ko 2020-11-20T11:56:41.876 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_gact.ko 2020-11-20T11:56:41.876 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_ipt.ko 2020-11-20T11:56:41.876 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_mirred.ko 2020-11-20T11:56:41.876 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_nat.ko 2020-11-20T11:56:41.876 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_pedit.ko 2020-11-20T11:56:41.877 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_police.ko 2020-11-20T11:56:41.877 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_simple.ko 2020-11-20T11:56:41.877 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_skbedit.ko 2020-11-20T11:56:41.877 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/act_vlan.ko 2020-11-20T11:56:41.878 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_basic.ko 2020-11-20T11:56:41.878 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_bpf.ko 2020-11-20T11:56:41.878 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_flow.ko 2020-11-20T11:56:41.878 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_fw.ko 2020-11-20T11:56:41.878 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_route.ko 2020-11-20T11:56:41.879 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_rsvp.ko 2020-11-20T11:56:41.879 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_rsvp6.ko 2020-11-20T11:56:41.879 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_tcindex.ko 2020-11-20T11:56:41.879 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/cls_u32.ko 2020-11-20T11:56:41.879 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_cmp.ko 2020-11-20T11:56:41.880 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_ipset.ko 2020-11-20T11:56:41.880 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_meta.ko 2020-11-20T11:56:41.880 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_nbyte.ko 2020-11-20T11:56:41.880 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_text.ko 2020-11-20T11:56:41.880 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/em_u32.ko 2020-11-20T11:56:41.881 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_cbq.ko 2020-11-20T11:56:41.881 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_choke.ko 2020-11-20T11:56:41.881 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_codel.ko 2020-11-20T11:56:41.881 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_drr.ko 2020-11-20T11:56:41.882 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_dsmark.ko 2020-11-20T11:56:41.882 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_fq.ko 2020-11-20T11:56:41.882 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_gred.ko 2020-11-20T11:56:41.882 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_hfsc.ko 2020-11-20T11:56:41.882 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_hhf.ko 2020-11-20T11:56:41.883 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_htb.ko 2020-11-20T11:56:41.883 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_ingress.ko 2020-11-20T11:56:41.883 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_mqprio.ko 2020-11-20T11:56:41.883 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_multiq.ko 2020-11-20T11:56:41.883 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_netem.ko 2020-11-20T11:56:41.884 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_pie.ko 2020-11-20T11:56:41.884 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_plug.ko 2020-11-20T11:56:41.884 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_prio.ko 2020-11-20T11:56:41.884 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_qfq.ko 2020-11-20T11:56:41.884 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_red.ko 2020-11-20T11:56:41.885 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_sfb.ko 2020-11-20T11:56:41.885 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_sfq.ko 2020-11-20T11:56:41.885 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_tbf.ko 2020-11-20T11:56:41.885 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sched/sch_teql.ko 2020-11-20T11:56:41.885 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sctp 2020-11-20T11:56:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sctp/sctp.ko 2020-11-20T11:56:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sctp/sctp_diag.ko 2020-11-20T11:56:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc 2020-11-20T11:56:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/auth_gss 2020-11-20T11:56:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2020-11-20T11:56:41.887 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2020-11-20T11:56:41.887 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/sunrpc.ko 2020-11-20T11:56:41.887 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/xprtrdma 2020-11-20T11:56:41.887 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2020-11-20T11:56:41.887 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/tipc 2020-11-20T11:56:41.888 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/tipc/diag.ko 2020-11-20T11:56:41.888 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/tipc/tipc.ko 2020-11-20T11:56:41.888 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/unix 2020-11-20T11:56:41.888 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/unix/unix_diag.ko 2020-11-20T11:56:41.888 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock 2020-11-20T11:56:41.889 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2020-11-20T11:56:41.889 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2020-11-20T11:56:41.889 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vsock.ko 2020-11-20T11:56:41.889 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vsock_diag.ko 2020-11-20T11:56:41.890 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/vmw_vsock/vsock_loopback.ko 2020-11-20T11:56:41.890 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/xfrm 2020-11-20T11:56:41.890 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/net/xfrm/xfrm_ipcomp.ko 2020-11-20T11:56:41.890 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security 2020-11-20T11:56:41.890 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys 2020-11-20T11:56:41.891 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys/encrypted-keys 2020-11-20T11:56:41.891 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2020-11-20T11:56:41.891 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys/trusted-keys 2020-11-20T11:56:41.891 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/security/keys/trusted-keys/trusted.ko 2020-11-20T11:56:41.891 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/virt 2020-11-20T11:56:41.892 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/virt/lib 2020-11-20T11:56:41.892 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/kernel/virt/lib/irqbypass.ko 2020-11-20T11:56:41.892 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.alias 2020-11-20T11:56:41.892 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.alias.bin 2020-11-20T11:56:41.892 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.builtin 2020-11-20T11:56:41.893 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.builtin.bin 2020-11-20T11:56:41.893 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.builtin.modinfo 2020-11-20T11:56:41.893 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.dep 2020-11-20T11:56:41.893 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.dep.bin 2020-11-20T11:56:41.893 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.devname 2020-11-20T11:56:41.894 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.order 2020-11-20T11:56:41.894 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.softdep 2020-11-20T11:56:41.894 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.symbols 2020-11-20T11:56:41.894 INFO:teuthology.orchestra.run.smithi005.stdout:/lib/modules/5.10.0-rc2-ceph-gac4f7b07f403/modules.symbols.bin 2020-11-20T11:56:41.895 DEBUG:teuthology.task.kernel:get_image_version: 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:41.895 INFO:teuthology.orchestra.run.smithi005:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2020-11-20T11:56:41.920 INFO:teuthology.orchestra.run.smithi005.stdout:postinstall scriptlet (using /bin/sh): 2020-11-20T11:56:41.921 INFO:teuthology.orchestra.run.smithi005.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403 -a -r /boot/System.map-5.10.0-rc2-ceph-gac4f7b07f403 ]; then 2020-11-20T11:56:41.921 INFO:teuthology.orchestra.run.smithi005.stdout:cp /boot/vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403 /boot/.vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403-rpm 2020-11-20T11:56:41.921 INFO:teuthology.orchestra.run.smithi005.stdout:cp /boot/System.map-5.10.0-rc2-ceph-gac4f7b07f403 /boot/.System.map-5.10.0-rc2-ceph-gac4f7b07f403-rpm 2020-11-20T11:56:41.922 INFO:teuthology.orchestra.run.smithi005.stdout:rm -f /boot/vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403 /boot/System.map-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:41.922 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/installkernel 5.10.0-rc2-ceph-gac4f7b07f403 /boot/.vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403-rpm /boot/.System.map-5.10.0-rc2-ceph-gac4f7b07f403-rpm 2020-11-20T11:56:41.922 INFO:teuthology.orchestra.run.smithi005.stdout:rm -f /boot/.vmlinuz-5.10.0-rc2-ceph-gac4f7b07f403-rpm /boot/.System.map-5.10.0-rc2-ceph-gac4f7b07f403-rpm 2020-11-20T11:56:41.922 INFO:teuthology.orchestra.run.smithi005.stdout:fi 2020-11-20T11:56:41.922 INFO:teuthology.orchestra.run.smithi005.stdout:preuninstall scriptlet (using /bin/sh): 2020-11-20T11:56:41.923 INFO:teuthology.orchestra.run.smithi005.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2020-11-20T11:56:41.923 INFO:teuthology.orchestra.run.smithi005.stdout:new-kernel-pkg --remove 5.10.0-rc2-ceph-gac4f7b07f403 --rminitrd --initrdfile=/boot/initramfs-5.10.0-rc2-ceph-gac4f7b07f403.img 2020-11-20T11:56:41.923 INFO:teuthology.orchestra.run.smithi005.stdout:elif [ -x /usr/bin/kernel-install ]; then 2020-11-20T11:56:41.923 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-install remove 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:41.923 INFO:teuthology.orchestra.run.smithi005.stdout:fi 2020-11-20T11:56:41.924 INFO:teuthology.orchestra.run.smithi005.stdout:postuninstall scriptlet (using /bin/sh): 2020-11-20T11:56:41.924 INFO:teuthology.orchestra.run.smithi005.stdout:if [ -x /sbin/update-bootloader ]; then 2020-11-20T11:56:41.924 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/update-bootloader --remove 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:41.924 INFO:teuthology.orchestra.run.smithi005.stdout:fi 2020-11-20T11:56:41.925 INFO:teuthology.orchestra.run.smithi005:> sudo rpm -qi grub2-tools 2020-11-20T11:56:42.007 INFO:teuthology.orchestra.run.smithi005.stdout:Name : grub2-tools 2020-11-20T11:56:42.008 INFO:teuthology.orchestra.run.smithi005.stdout:Epoch : 1 2020-11-20T11:56:42.008 INFO:teuthology.orchestra.run.smithi005.stdout:Version : 2.02 2020-11-20T11:56:42.008 INFO:teuthology.orchestra.run.smithi005.stdout:Release : 78.el8 2020-11-20T11:56:42.008 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: x86_64 2020-11-20T11:56:42.009 INFO:teuthology.orchestra.run.smithi005.stdout:Install Date: Sun 15 Nov 2020 12:08:16 PM UTC 2020-11-20T11:56:42.009 INFO:teuthology.orchestra.run.smithi005.stdout:Group : System Environment/Base 2020-11-20T11:56:42.009 INFO:teuthology.orchestra.run.smithi005.stdout:Size : 9192691 2020-11-20T11:56:42.009 INFO:teuthology.orchestra.run.smithi005.stdout:License : GPLv3+ 2020-11-20T11:56:42.009 INFO:teuthology.orchestra.run.smithi005.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-11-20T11:56:42.010 INFO:teuthology.orchestra.run.smithi005.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-11-20T11:56:42.010 INFO:teuthology.orchestra.run.smithi005.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-11-20T11:56:42.010 INFO:teuthology.orchestra.run.smithi005.stdout:Build Host : kbuilder.bsys.centos.org 2020-11-20T11:56:42.010 INFO:teuthology.orchestra.run.smithi005.stdout:Relocations : (not relocatable) 2020-11-20T11:56:42.010 INFO:teuthology.orchestra.run.smithi005.stdout:Packager : CentOS BuildSystem 2020-11-20T11:56:42.011 INFO:teuthology.orchestra.run.smithi005.stdout:Vendor : CentOS 2020-11-20T11:56:42.011 INFO:teuthology.orchestra.run.smithi005.stdout:URL : http://www.gnu.org/software/grub/ 2020-11-20T11:56:42.011 INFO:teuthology.orchestra.run.smithi005.stdout:Summary : Support tools for GRUB. 2020-11-20T11:56:42.011 INFO:teuthology.orchestra.run.smithi005.stdout:Description : 2020-11-20T11:56:42.011 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T11:56:42.012 INFO:teuthology.orchestra.run.smithi005.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-11-20T11:56:42.012 INFO:teuthology.orchestra.run.smithi005.stdout:customizable bootloader with modular architecture. It supports a rich 2020-11-20T11:56:42.012 INFO:teuthology.orchestra.run.smithi005.stdout:variety of kernel formats, file systems, computer architectures and 2020-11-20T11:56:42.012 INFO:teuthology.orchestra.run.smithi005.stdout:hardware devices. 2020-11-20T11:56:42.012 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T11:56:42.013 INFO:teuthology.orchestra.run.smithi005.stdout:This subpackage provides tools for support of all platforms. 2020-11-20T11:56:42.014 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-11-20T11:56:42.014 INFO:teuthology.task.kernel:Updating grub on smithi005 to boot 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:42.014 INFO:teuthology.orchestra.run.smithi005:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-11-20T11:56:46.746 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2020-11-20T11:56:48.669 INFO:teuthology.orchestra.run.smithi005.stderr:done 2020-11-20T11:56:48.670 INFO:teuthology.orchestra.run.smithi005:> mktemp 2020-11-20T11:56:48.724 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.BdDQtnLIP2 2020-11-20T11:56:48.724 INFO:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BdDQtnLIP2 2020-11-20T11:56:48.839 INFO:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.BdDQtnLIP2 2020-11-20T11:56:50.689 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.BdDQtnLIP2 is 4KB 2020-11-20T11:56:50.741 INFO:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.BdDQtnLIP2 2020-11-20T11:56:50.759 INFO:teuthology.orchestra.run.smithi005:> sudo /bin/ls /boot/loader/entries || true 2020-11-20T11:56:50.858 INFO:teuthology.orchestra.run.smithi005.stdout:558cda909eee45cd892f671baf08ec7a-0-rescue.conf 2020-11-20T11:56:50.858 INFO:teuthology.orchestra.run.smithi005.stdout:558cda909eee45cd892f671baf08ec7a-4.18.0-147.el8.x86_64.conf 2020-11-20T11:56:50.859 INFO:teuthology.orchestra.run.smithi005.stdout:558cda909eee45cd892f671baf08ec7a-4.18.0-193.28.1.el8_2.x86_64.conf 2020-11-20T11:56:50.859 INFO:teuthology.orchestra.run.smithi005.stdout:558cda909eee45cd892f671baf08ec7a-5.10.0-rc2-ceph-gac4f7b07f403.conf 2020-11-20T11:56:50.860 INFO:teuthology.orchestra.run.smithi005:> sudo grub2-set-default 558cda909eee45cd892f671baf08ec7a-5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:56:50.959 INFO:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2020-11-20T11:57:20.989 INFO:teuthology.misc:Re-opening connections... 2020-11-20T11:57:20.990 INFO:teuthology.misc:trying to connect to ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T11:57:20.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:57:20.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:57:21.299 INFO:teuthology.orchestra.run.smithi187:> true 2020-11-20T11:57:21.859 DEBUG:teuthology.misc:waited 0.869678258895874 2020-11-20T11:57:22.860 INFO:teuthology.misc:trying to connect to ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T11:57:22.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:57:22.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:58:22.863 DEBUG:teuthology.orchestra.remote:timed out 2020-11-20T11:58:24.040 DEBUG:teuthology.misc:waited 63.05021524429321 2020-11-20T11:58:25.042 INFO:teuthology.misc:trying to connect to ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T11:58:25.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:58:25.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:58:25.337 INFO:teuthology.orchestra.run.smithi005:> true 2020-11-20T11:58:25.926 DEBUG:teuthology.misc:waited 64.9359803199768 2020-11-20T11:58:26.927 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2020-11-20T11:58:26.927 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.10.0-rc2-ceph-gac4f7b07f403"... 2020-11-20T11:58:26.928 INFO:teuthology.orchestra.run.smithi187:> uname -r 2020-11-20T11:58:26.989 INFO:teuthology.orchestra.run.smithi187.stdout:5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:58:26.989 DEBUG:teuthology.task.kernel:current kernel version is 5.10.0-rc2-ceph-gac4f7b07f403 vs 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:58:26.990 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-20T11:58:26.990 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2020-11-20T11:58:26.990 INFO:teuthology.task.kernel:Checking client client.2 for new kernel version... 2020-11-20T11:58:26.991 INFO:teuthology.task.kernel:Checking kernel version of client.2, want "5.10.0-rc2-ceph-gac4f7b07f403"... 2020-11-20T11:58:26.991 INFO:teuthology.orchestra.run.smithi005:> uname -r 2020-11-20T11:58:27.052 INFO:teuthology.orchestra.run.smithi005.stdout:5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:58:27.053 DEBUG:teuthology.task.kernel:current kernel version is 5.10.0-rc2-ceph-gac4f7b07f403 vs 5.10.0-rc2-ceph-gac4f7b07f403 2020-11-20T11:58:27.053 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-20T11:58:27.054 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2020-11-20T11:58:28.055 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-20T11:58:28.063 INFO:teuthology.task.internal:Creating test directory... 2020-11-20T11:58:28.064 INFO:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-20T11:58:28.072 INFO:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-20T11:58:28.128 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-20T11:58:28.138 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-20T11:58:28.163 INFO:teuthology.task.internal:Creating archive directory... 2020-11-20T11:58:28.163 INFO:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-20T11:58:28.169 INFO:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-20T11:58:28.234 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-20T11:58:28.245 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-20T11:58:28.245 INFO:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-20T11:58:28.254 INFO:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-20T11:58:28.471 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:58:28.516 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:58:28.538 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:58:28.585 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:58:28.589 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-20T11:58:28.598 INFO:teuthology.task.internal:Configuring sudo... 2020-11-20T11:58:28.598 INFO:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-20T11:58:28.604 INFO:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-20T11:58:28.713 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-20T11:58:28.734 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-20T11:58:28.735 INFO:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-20T11:58:28.753 INFO:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-20T11:58:28.815 INFO:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:58:28.926 INFO:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:58:29.044 INFO:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:58:29.138 INFO:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:58:29.257 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T11:58:29.258 INFO:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:58:29.321 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:58:29.406 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T11:58:29.406 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T11:58:29.406 INFO:teuthology.orchestra.run.smithi187.stderr:152 bytes copied, 0.000152398 s, 997 kB/s 2020-11-20T11:58:29.407 INFO:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:58:29.515 INFO:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:58:29.650 INFO:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:58:29.746 INFO:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:58:29.862 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T11:58:29.863 INFO:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:58:29.925 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:58:30.017 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T11:58:30.017 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T11:58:30.018 INFO:teuthology.orchestra.run.smithi005.stderr:152 bytes copied, 0.000131873 s, 1.2 MB/s 2020-11-20T11:58:30.024 INFO:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2020-11-20T11:58:30.037 INFO:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2020-11-20T11:58:30.216 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-20T11:58:30.243 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-20T11:58:30.546 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-20T11:58:30.556 INFO:teuthology.task.internal:Starting timer... 2020-11-20T11:58:30.557 INFO:teuthology.run_tasks:Running task pcp... 2020-11-20T11:58:30.596 INFO:teuthology.run_tasks:Running task selinux... 2020-11-20T11:58:30.694 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-11-20T11:58:30.695 INFO:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2020-11-20T11:58:30.884 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: [ OK ] 2020-11-20T11:58:30.891 INFO:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2020-11-20T11:58:31.056 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: [ OK ] 2020-11-20T11:58:31.061 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-20T11:58:31.062 INFO:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2020-11-20T11:58:31.126 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2020-11-20T11:58:31.127 INFO:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2020-11-20T11:58:31.192 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2020-11-20T11:58:31.193 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi187.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi005.front.sepia.ceph.com': 'permissive'} 2020-11-20T11:58:31.194 INFO:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-20T11:58:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T11:58:31.302 INFO:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-20T11:58:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T11:58:31.415 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-20T11:58:31.415 INFO:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2020-11-20T11:58:31.520 INFO:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2020-11-20T11:58:31.634 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-20T11:58:31.643 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-11-20T11:58:31.644 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-20T11:58:31.658 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-20T11:58:31.659 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi187.front.sepia.ceph.com,smithi005.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-20T11:58:31.986 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-20T11:58:32.862 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-20T11:58:35.448 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-20T11:58:35.468 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:35 +0000 (0:00:00.071) 0:00:00.071 ******* 2020-11-20T11:58:35.481 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:35 +0000 (0:00:00.013) 0:00:00.084 ******* 2020-11-20T11:58:38.118 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi187.front.sepia.ceph.com] 2020-11-20T11:58:38.188 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:38 +0000 (0:00:02.707) 0:00:02.791 ******* 2020-11-20T11:58:38.240 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] 2020-11-20T11:58:38.313 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:38 +0000 (0:00:00.124) 0:00:02.916 ******* 2020-11-20T11:58:38.656 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-11-20T11:58:38.658 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-20T11:58:38.690 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:38 +0000 (0:00:00.376) 0:00:03.293 ******* 2020-11-20T11:58:38.696 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-20T11:58:38.731 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:38 +0000 (0:00:00.040) 0:00:03.334 ******* 2020-11-20T11:58:39.311 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-20T11:58:39.313 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-20T11:58:39.348 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:39 +0000 (0:00:00.618) 0:00:03.952 ******* 2020-11-20T11:58:39.355 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-20T11:58:39.389 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:39 +0000 (0:00:00.040) 0:00:03.992 ******* 2020-11-20T11:58:39.396 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-20T11:58:39.397 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:39.431 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:39 +0000 (0:00:00.042) 0:00:04.034 ******* 2020-11-20T11:58:39.437 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:39.474 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:39 +0000 (0:00:00.042) 0:00:04.077 ******* 2020-11-20T11:58:39.923 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-11-20T11:58:39.925 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:58:39.960 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:39 +0000 (0:00:00.486) 0:00:04.563 ******* 2020-11-20T11:58:39.968 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:58:40.010 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:40 +0000 (0:00:00.050) 0:00:04.613 ******* 2020-11-20T11:58:41.532 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-20T11:58:42.358 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605873519.974066-180915345484405/source", "state": "file", "uid": 0} Friday 20 November 2020 11:58:41 +0000 (0:00:01.569) 0:00:06.183 ******* changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605873520.023372-273488290528866/source", "state": "file", "uid": 0} Friday 20 November 2020 11:58:41 +0000 (0:00:00.045) 0:00:06.229 ******* TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoid 2020-11-20T11:58:42.360 INFO:teuthology.task.ansible.out:D3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVP 2020-11-20T11:58:42.362 INFO:teuthology.task.ansible.out:iUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7 2020-11-20T11:58:42.363 INFO:teuthology.task.ansible.out:WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-20T11:58:42.389 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-20T11:58:42.397 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-20T11:58:42.548 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:42 +0000 (0:00:00.921) 0:00:07.151 ******* 2020-11-20T11:58:42.580 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:42 +0000 (0:00:00.032) 0:00:07.183 ******* 2020-11-20T11:58:43.921 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-11-20T11:58:43.923 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] 2020-11-20T11:58:44.002 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-20T11:58:44.003 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:43 +0000 (0:00:01.422) 0:00:08.605 ******* 2020-11-20T11:58:44.013 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] 2020-11-20T11:58:44.111 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:44 +0000 (0:00:00.109) 0:00:08.714 ******* 2020-11-20T11:58:44.484 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-20T11:58:44.486 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.177143", "end": "2020-11-20 11:58:44.458169", "rc": 0, "start": "2020-11-20 11:58:44.281026", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} changed: [smithi187.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.166231", "end": "2020-11-20 11:58:44.464636", "rc": 0, "start": "2020-11-20 11:58:44.298405", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:58:44.518 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:44 +0000 (0:00:00.407) 0:00:09.121 ******* 2020-11-20T11:58:44.553 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:44 +0000 (0:00:00.034) 0:00:09.156 ******* 2020-11-20T11:58:44.895 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-20T11:58:44.896 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.020322", "end": "2020-11-20 11:58:44.852090", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:58:44.831768", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:58:44.931 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:44 +0000 (0:00:00.378) 0:00:09.534 ******* 2020-11-20T11:58:44.938 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013056", "end": "2020-11-20 11:58:44.867487", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:58:44.854431", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:58:44.974 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:44 +0000 (0:00:00.042) 0:00:09.577 ******* 2020-11-20T11:58:44.980 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-20T11:58:44.981 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-20T11:58:45.015 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.041) 0:00:09.618 ******* 2020-11-20T11:58:45.021 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-20T11:58:45.056 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.041) 0:00:09.659 ******* 2020-11-20T11:58:45.062 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-20T11:58:45.063 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.097 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.040) 0:00:09.700 ******* 2020-11-20T11:58:45.103 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.139 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.042) 0:00:09.742 ******* 2020-11-20T11:58:45.145 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-11-20T11:58:45.147 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.184 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.044) 0:00:09.787 ******* 2020-11-20T11:58:45.190 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.225 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.041) 0:00:09.828 ******* 2020-11-20T11:58:45.232 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.267 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.041) 0:00:09.870 ******* 2020-11-20T11:58:45.273 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.307 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.040) 0:00:09.911 ******* 2020-11-20T11:58:45.313 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-20T11:58:45.315 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.359 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.051) 0:00:09.962 ******* 2020-11-20T11:58:45.365 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.400 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.041) 0:00:10.003 ******* 2020-11-20T11:58:45.406 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-20T11:58:45.408 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.441 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.040) 0:00:10.044 ******* 2020-11-20T11:58:45.447 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.494 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:45 +0000 (0:00:00.052) 0:00:10.096 ******* 2020-11-20T11:58:45.500 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-20T11:58:45.502 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.504 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.515 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.518 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.530 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.531 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.532 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.543 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.555 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.557 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.557 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 2020-11-20T11:58:45.559 INFO:teuthology.task.ansible.out:8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dac 2020-11-20T11:58:45.561 INFO:teuthology.task.ansible.out:d78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.572 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.574 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.574 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+ 2020-11-20T11:58:45.576 INFO:teuthology.task.ansible.out:mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.586 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.587 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.597 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.599 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.599 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ov 2020-11-20T11:58:45.601 INFO:teuthology.task.ansible.out:pn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.611 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.613 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHs 2020-11-20T11:58:45.615 INFO:teuthology.task.ansible.out:nQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.624 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.626 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQA 2020-11-20T11:58:45.628 INFO:teuthology.task.ansible.out:BAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'} 2020-11-20T11:58:45.630 INFO:teuthology.task.ansible.out:) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.639 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.641 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa70 2020-11-20T11:58:45.642 INFO:teuthology.task.ansible.out:1ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.652 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.654 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.664 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.666 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d 2020-11-20T11:58:45.668 INFO:teuthology.task.ansible.out:2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.668 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.680 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.682 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.691 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.693 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'nam 2020-11-20T11:58:45.694 INFO:teuthology.task.ansible.out:e': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.706 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.708 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.718 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.729 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.731 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2 2020-11-20T11:58:45.733 INFO:teuthology.task.ansible.out:794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.742 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.744 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.744 INFO:teuthology.task.ansible.out: 2020-11-20T11:58:45.756 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.767 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.769 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.781 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzar 2020-11-20T11:58:45.783 INFO:teuthology.task.ansible.out:zyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.793 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af2 2020-11-20T11:58:45.795 INFO:teuthology.task.ansible.out:72c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.805 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.807 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.818 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.820 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.831 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.833 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.846 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.848 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.860 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.870 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.872 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.872 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.874 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.886 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.896 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.898 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.908 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.910 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.911 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@think 2020-11-20T11:58:45.912 INFO:teuthology.task.ansible.out:pad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.923 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.925 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.935 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.936 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.938 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.949 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.951 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.962 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.963 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.964 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.976 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.978 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.989 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.991 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:45.992 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.003 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.006 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.016 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.018 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.019 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.c 2020-11-20T11:58:46.020 INFO:teuthology.task.ansible.out:eph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.030 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.031 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.032 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.f 2020-11-20T11:58:46.033 INFO:teuthology.task.ansible.out:ront.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.044 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.046 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@mi 2020-11-20T11:58:46.047 INFO:teuthology.task.ansible.out:lestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.059 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.071 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.082 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.084 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.084 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.085 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.red 2020-11-20T11:58:46.086 INFO:teuthology.task.ansible.out:hat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.099 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.100 INFO:teuthology.task.ansible.out: skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.101 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.102 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.114 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.129 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": 2020-11-20T11:58:46.133 INFO:teuthology.task.ansible.out:{"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.143 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.145 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.155 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.156 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.157 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prate 2020-11-20T11:58:46.158 INFO:teuthology.task.ansible.out:ek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.169 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.171 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.171 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.172 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4 2020-11-20T11:58:46.173 INFO:teuthology.task.ansible.out:ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.183 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.185 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.195 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.198 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.207 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.209 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.210 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) 2020-11-20T11:58:46.211 INFO:teuthology.task.ansible.out: => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.222 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.236 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.237 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.239 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.241 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.243 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.254 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.256 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.256 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name 2020-11-20T11:58:46.258 INFO:teuthology.task.ansible.out:": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.268 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.270 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:46.286 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.780 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.830) 0:00:10.927 ******* skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0df 2020-11-20T11:58:47.782 INFO:teuthology.task.ansible.out:a39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, 2020-11-20T11:58:47.784 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} TASK [users : Create all users without sudo access.] *************************** Friday 20 November 2020 11:58:46 +0000 (0:00:00.059) 0:00:10.987 ******* Friday 20 November 2020 11:58:46 +0000 (0:00:00.032) 0:00:11.019 ******* TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.046) 0:00:11.066 ******* TASK [users : Create all users without sudo access.] *************************** Friday 20 November 2020 11:58:46 +0000 (0:00:00.043) 0:00:11.110 ******* TASK [users : Clone the keys repo] ********************************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.040) 0:00:11.150 ******* TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.048) 0:00:11.199 ******* TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.041) 0:00:11.240 ******* TASK [users : Clone the keys repo] ********************************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.043) 0:00:11.284 ******* TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi005.front.sepia.ceph.com] => {"changed": 2020-11-20T11:58:47.785 INFO:teuthology.task.ansible.out:false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.042) 0:00:11.327 ******* TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.043) 0:00:11.371 ******* TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.047) 0:00:11.419 ******* TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.043) 0:00:11.462 ******* TASK [users : Remove revoked users] ******************************************** skipping: [smithi005.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=kefu) => 2020-11-20T11:58:47.787 INFO:teuthology.task.ansible.out: {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi005.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:46 +0000 (0:00:00.067) 0:00:11.529 ******* skipping: [smithi005.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item" 2020-11-20T11:58:47.789 INFO:teuthology.task.ansible.out:, "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Condi 2020-11-20T11:58:47.790 INFO:teuthology.task.ansible.out:tional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [s 2020-11-20T11:58:47.792 INFO:teuthology.task.ansible.out:mithi005.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.792 INFO:teuthology.task.ansible.out: 2020-11-20T11:58:47.793 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.793 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.793 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.794 INFO:teuthology.task.ansible.out:skipping: [smithi187.fr 2020-11-20T11:58:47.795 INFO:teuthology.task.ansible.out:ont.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.796 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.796 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.797 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.797 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.798 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=icolle) = 2020-11-20T11:58:47.799 INFO:teuthology.task.ansible.out:> {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.800 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.800 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.800 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.801 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.801 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.801 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", 2020-11-20T11:58:47.803 INFO:teuthology.task.ansible.out:"skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.806 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.808 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.808 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.809 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.809 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.810 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.810 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.810 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.811 INFO:teuthology.task.ansible.out:skipping: 2020-11-20T11:58:47.812 INFO:teuthology.task.ansible.out: [smithi187.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.813 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.813 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.814 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.814 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-20T11:58:47.852 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:47 +0000 (0:00:00.925) 0:00:12.455 ******* 2020-11-20T11:58:47.884 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:47 +0000 (0:00:00.031) 0:00:12.487 ******* 2020-11-20T11:58:49.038 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-20T11:58:49.040 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-11-20T11:58:49.073 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:49 +0000 (0:00:01.189) 0:00:13.676 ******* 2020-11-20T11:58:49.079 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-11-20T11:58:49.114 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:49 +0000 (0:00:00.041) 0:00:13.717 ******* 2020-11-20T11:58:49.120 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-20T11:58:49.122 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => { "msg": "Host smithi187.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-11-20T11:58:49.157 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:49 +0000 (0:00:00.042) 0:00:13.760 ******* 2020-11-20T11:58:49.164 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => { "msg": "Host smithi005.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-11-20T11:58:49.202 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:49 +0000 (0:00:00.045) 0:00:13.805 ******* 2020-11-20T11:58:49.210 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-20T11:58:49.211 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"container_mirror_cert_path": "/etc/containers/certs.d/{{ container_mirror }}", "nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-11-20T11:58:49.246 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:49 +0000 (0:00:00.043) 0:00:13.849 ******* 2020-11-20T11:58:49.253 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"container_mirror_cert_path": "/etc/containers/certs.d/{{ container_mirror }}", "nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-11-20T11:58:49.291 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:49 +0000 (0:00:00.045) 0:00:13.894 ******* 2020-11-20T11:58:49.297 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-20T11:58:49.299 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:49.355 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:49 +0000 (0:00:00.064) 0:00:13.958 ******* 2020-11-20T11:58:49.364 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:49.404 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:49 +0000 (0:00:00.048) 0:00:14.007 ******* 2020-11-20T11:58:50.174 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-20T11:58:50.176 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.482775", "end": "2020-11-20 11:58:50.137774", "rc": 0, "start": "2020-11-20 11:58:49.654999", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-20T11:58:50.209 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.805) 0:00:14.812 ******* 2020-11-20T11:58:50.236 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-20T11:58:50.238 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.272 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.063) 0:00:14.875 ******* 2020-11-20T11:58:50.288 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-20T11:58:50.290 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.514253", "end": "2020-11-20 11:58:50.238473", "rc": 0, "start": "2020-11-20 11:58:49.724220", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-20T11:58:50.323 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.051) 0:00:14.927 ******* 2020-11-20T11:58:50.330 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-20T11:58:50.331 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.364 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.040) 0:00:14.968 ******* 2020-11-20T11:58:50.370 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-20T11:58:50.372 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.406 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.041) 0:00:15.010 ******* 2020-11-20T11:58:50.413 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-20T11:58:50.414 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.451 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.045) 0:00:15.055 ******* 2020-11-20T11:58:50.457 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-20T11:58:50.459 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.493 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.041) 0:00:15.096 ******* 2020-11-20T11:58:50.499 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-20T11:58:50.501 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.535 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.041) 0:00:15.138 ******* 2020-11-20T11:58:50.541 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-20T11:58:50.543 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.577 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.041) 0:00:15.180 ******* 2020-11-20T11:58:50.814 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:50 +0000 (0:00:00.050) 0:00:15.231 ******* TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:50 +0000 (0:00:00.052) 0:00:15.283 ******* TASK [common : Set entitlements_path] ****************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:50 +0000 (0:00:00.057) 0:00:15.341 ******* TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:58:50 +0000 (0:00:00.055) 0:00:15.396 ******* TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi187.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:50.835 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.042) 0:00:15.438 ******* 2020-11-20T11:58:50.841 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-20T11:58:50.843 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.877 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.041) 0:00:15.480 ******* 2020-11-20T11:58:50.883 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-20T11:58:50.885 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:50.921 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.043) 0:00:15.524 ******* 2020-11-20T11:58:50.926 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-20T11:58:50.928 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:50.964 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:50 +0000 (0:00:00.043) 0:00:15.567 ******* 2020-11-20T11:58:50.970 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-20T11:58:50.972 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.007 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.042) 0:00:15.610 ******* 2020-11-20T11:58:51.012 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-20T11:58:51.014 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.049 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.042) 0:00:15.652 ******* 2020-11-20T11:58:51.055 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-20T11:58:51.057 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:51.100 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.051) 0:00:15.704 ******* 2020-11-20T11:58:51.106 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-20T11:58:51.108 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.145 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.044) 0:00:15.748 ******* 2020-11-20T11:58:51.151 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-20T11:58:51.153 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.198 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.053) 0:00:15.801 ******* 2020-11-20T11:58:51.206 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-20T11:58:51.208 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:51.247 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.048) 0:00:15.850 ******* 2020-11-20T11:58:51.253 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:51.289 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.042) 0:00:15.892 ******* 2020-11-20T11:58:51.295 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-20T11:58:51.297 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.337 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.047) 0:00:15.940 ******* 2020-11-20T11:58:51.344 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-20T11:58:51.345 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.384 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.047) 0:00:15.988 ******* 2020-11-20T11:58:51.391 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-20T11:58:51.393 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:51.428 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.043) 0:00:16.031 ******* 2020-11-20T11:58:51.434 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-20T11:58:51.436 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.469 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.041) 0:00:16.072 ******* 2020-11-20T11:58:51.475 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-20T11:58:51.477 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.511 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.041) 0:00:16.114 ******* 2020-11-20T11:58:51.519 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.553 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.041) 0:00:16.156 ******* 2020-11-20T11:58:51.559 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-20T11:58:51.560 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.596 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.042) 0:00:16.199 ******* 2020-11-20T11:58:51.602 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-20T11:58:51.604 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.643 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.047) 0:00:16.246 ******* 2020-11-20T11:58:51.649 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-20T11:58:51.651 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.684 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.041) 0:00:16.288 ******* 2020-11-20T11:58:51.692 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-20T11:58:51.694 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:51.728 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.043) 0:00:16.331 ******* 2020-11-20T11:58:51.734 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-20T11:58:51.736 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.771 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.042) 0:00:16.374 ******* 2020-11-20T11:58:51.778 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-20T11:58:51.780 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.813 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.042) 0:00:16.417 ******* 2020-11-20T11:58:51.820 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-20T11:58:51.822 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:51.857 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.044) 0:00:16.461 ******* 2020-11-20T11:58:51.865 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-20T11:58:51.866 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.901 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.043) 0:00:16.504 ******* 2020-11-20T11:58:51.909 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-20T11:58:51.911 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.948 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.046) 0:00:16.551 ******* 2020-11-20T11:58:51.955 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-20T11:58:51.957 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:51.991 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:51 +0000 (0:00:00.043) 0:00:16.595 ******* 2020-11-20T11:58:51.998 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-20T11:58:52.000 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.036 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.043) 0:00:16.638 ******* 2020-11-20T11:58:52.043 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.093 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.050) 0:00:16.689 ******* 2020-11-20T11:58:52.100 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.154 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.068) 0:00:16.757 ******* 2020-11-20T11:58:52.163 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.215 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.061) 0:00:16.818 ******* 2020-11-20T11:58:52.223 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-20T11:58:52.225 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.312 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.096) 0:00:16.915 ******* 2020-11-20T11:58:52.314 INFO:teuthology.task.ansible.out: 2020-11-20T11:58:52.330 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.418 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.105) 0:00:17.021 ******* 2020-11-20T11:58:52.429 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-20T11:58:52.431 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.467 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.049) 0:00:17.070 ******* 2020-11-20T11:58:52.474 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-20T11:58:52.476 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:52.509 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.042) 0:00:17.113 ******* 2020-11-20T11:58:52.516 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-20T11:58:52.517 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.551 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.041) 0:00:17.155 ******* 2020-11-20T11:58:52.559 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-20T11:58:52.561 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.600 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.048) 0:00:17.204 ******* 2020-11-20T11:58:52.606 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-20T11:58:52.608 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:58:52.654 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.052) 0:00:17.256 ******* 2020-11-20T11:58:52.660 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-20T11:58:52.661 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.698 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.044) 0:00:17.301 ******* 2020-11-20T11:58:52.705 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.744 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.046) 0:00:17.348 ******* 2020-11-20T11:58:52.773 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-20T11:58:52.775 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:52.814 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:52 +0000 (0:00:00.069) 0:00:17.417 ******* 2020-11-20T11:58:53.094 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:58:53.128 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:53 +0000 (0:00:00.313) 0:00:17.731 ******* 2020-11-20T11:58:53.220 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:58:53.256 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:53 +0000 (0:00:00.127) 0:00:17.859 ******* 2020-11-20T11:58:54.145 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-20T11:58:54.147 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2020-11-20T11:58:54.263 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2020-11-20T11:58:55.031 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2020-11-20T11:58:55.071 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:55 +0000 (0:00:01.815) 0:00:19.674 ******* 2020-11-20T11:58:55.217 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2020-11-20T11:58:55.258 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:55 +0000 (0:00:00.187) 0:00:19.862 ******* 2020-11-20T11:58:55.992 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi187.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-11-20T11:58:56.217 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-11-20T11:58:56.908 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-11-20T11:58:56.956 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:56 +0000 (0:00:01.696) 0:00:21.559 ******* 2020-11-20T11:58:56.985 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-20T11:58:56.989 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.020 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.064) 0:00:21.624 ******* 2020-11-20T11:58:57.048 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T11:58:57.049 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.083 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.062) 0:00:21.686 ******* 2020-11-20T11:58:57.126 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.163 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.080) 0:00:21.767 ******* 2020-11-20T11:58:57.193 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-20T11:58:57.195 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-11-20T11:58:57.196 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.236 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.072) 0:00:21.839 ******* 2020-11-20T11:58:57.269 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.033) 0:00:21.873 ******* 2020-11-20T11:58:57.276 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-20T11:58:57.278 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.312 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.042) 0:00:21.915 ******* 2020-11-20T11:58:57.318 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T11:58:57.320 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.358 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.046) 0:00:21.961 ******* 2020-11-20T11:58:57.364 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T11:58:57.366 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.401 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.043) 0:00:22.005 ******* 2020-11-20T11:58:57.409 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.444 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.042) 0:00:22.047 ******* 2020-11-20T11:58:57.453 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.503 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.059) 0:00:22.106 ******* 2020-11-20T11:58:57.510 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-20T11:58:57.512 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.556 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.053) 0:00:22.159 ******* 2020-11-20T11:58:57.563 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-20T11:58:57.565 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.601 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.044) 0:00:22.204 ******* 2020-11-20T11:58:57.608 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.663 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.061) 0:00:22.266 ******* 2020-11-20T11:58:57.672 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T11:58:57.674 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.707 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.044) 0:00:22.310 ******* 2020-11-20T11:58:57.715 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-20T11:58:57.717 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.756 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.048) 0:00:22.359 ******* 2020-11-20T11:58:57.765 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-20T11:58:57.768 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.805 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.049) 0:00:22.408 ******* 2020-11-20T11:58:57.833 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-20T11:58:57.835 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.886 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.081) 0:00:22.490 ******* 2020-11-20T11:58:57.919 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-20T11:58:57.921 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:57.980 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:57 +0000 (0:00:00.093) 0:00:22.583 ******* 2020-11-20T11:58:58.011 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-20T11:58:58.012 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:58:58.049 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:58 +0000 (0:00:00.068) 0:00:22.652 ******* 2020-11-20T11:58:58.537 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-20T11:58:58.539 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.433184", "end": "2020-11-20 11:58:58.501352", "rc": 0, "start": "2020-11-20 11:58:58.068168", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.120156", "end": "2020-11-20 11:58:58.501237", "rc": 0, "start": "2020-11-20 11:58:58.381081", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:58:58.593 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:58 +0000 (0:00:00.544) 0:00:23.196 ******* 2020-11-20T11:58:58.637 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:58:58 +0000 (0:00:00.044) 0:00:23.240 ******* 2020-11-20T11:59:02.940 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-20T11:59:02.942 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-11-20T11:59:02.975 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:02 +0000 (0:00:04.338) 0:00:27.578 ******* 2020-11-20T11:59:02.992 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-11-20T11:59:02.994 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-20T11:59:02.995 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:03.028 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.052) 0:00:27.631 ******* 2020-11-20T11:59:03.060 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.032) 0:00:27.663 ******* 2020-11-20T11:59:03.067 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-20T11:59:03.068 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:03.105 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.045) 0:00:27.709 ******* 2020-11-20T11:59:03.112 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-20T11:59:03.114 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:03.149 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.043) 0:00:27.752 ******* 2020-11-20T11:59:03.155 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-20T11:59:03.157 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:03.190 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.040) 0:00:27.793 ******* 2020-11-20T11:59:03.196 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-20T11:59:03.197 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:03.233 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.042) 0:00:27.836 ******* 2020-11-20T11:59:03.239 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-20T11:59:03.241 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:03.273 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.040) 0:00:27.876 ******* 2020-11-20T11:59:03.279 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-20T11:59:03.281 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:03.317 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.043) 0:00:27.920 ******* 2020-11-20T11:59:03.344 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-20T11:59:03.345 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:03.378 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:03 +0000 (0:00:00.061) 0:00:27.981 ******* 2020-11-20T11:59:04.274 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-20T11:59:04.276 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-11-20T11:59:04.318 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:04 +0000 (0:00:00.940) 0:00:28.921 ******* 2020-11-20T11:59:04.326 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-11-20T11:59:04.363 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:04 +0000 (0:00:00.044) 0:00:28.966 ******* 2020-11-20T11:59:09.195 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-20T11:59:09.310 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools", "Installed: smartmontools-1:7.1-1.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} Friday 20 November 2020 11:59:09 +0000 (0:00:04.870) 0:00:33.837 ******* 2020-11-20T11:59:09.452 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools", "Installed: smartmontools-1:7.1-1.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} 2020-11-20T11:59:09.487 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:09 +0000 (0:00:00.253) 0:00:34.091 ******* 2020-11-20T11:59:10.120 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-20T11:59:10.122 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-11-20T11:59:10.374 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-11-20T11:59:10.972 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-11-20T11:59:11.278 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-11-20T11:59:11.849 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-11-20T11:59:11.890 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:11 +0000 (0:00:02.402) 0:00:36.493 ******* 2020-11-20T11:59:12.226 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-11-20T11:59:12.264 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:12 +0000 (0:00:00.374) 0:00:36.867 ******* 2020-11-20T11:59:12.273 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T11:59:12.317 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:12 +0000 (0:00:00.052) 0:00:36.920 ******* 2020-11-20T11:59:12.626 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T11:59:12.668 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:12 +0000 (0:00:00.351) 0:00:37.272 ******* 2020-11-20T11:59:13.270 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-20T11:59:13.271 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-11-20T11:59:13.613 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-11-20T11:59:14.183 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-11-20T11:59:14.568 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-11-20T11:59:15.098 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-11-20T11:59:15.137 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:15 +0000 (0:00:02.467) 0:00:39.739 ******* 2020-11-20T11:59:15.871 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:59:15 +0000 (0:00:00.093) 0:00:39.833 ******* TASK [common : Include yum_systems vars] *************************************** ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} Friday 20 November 2020 11:59:15 +0000 (0:00:00.094) 0:00:39.928 ******* TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi005.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} Friday 20 November 2020 11:59:15 +0000 (0:00:00.235) 0:00:40.163 ******* TASK [common : Include apt_system vars] **************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:59:15 +0000 (0:00:00.069) 0:00:40.233 ******* TASK [common : Include yum_systems vars] *************************************** ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", " 2020-11-20T11:59:15.873 INFO:teuthology.task.ansible.out:nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} Friday 20 November 2020 11:59:15 +0000 (0:00:00.073) 0:00:40.307 ******* TASK [common : Check for epel] ************************************************* changed: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} Friday 20 November 2020 11:59:15 +0000 (0:00:00.044) 0:00:40.351 ******* TASK [common : Install nrpe without epel] ************************************** skipping: [smithi187.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_lo 2020-11-20T11:59:15.874 INFO:teuthology.task.ansible.out:op_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:59:15 +0000 (0:00:00.118) 0:00:40.469 ******* 2020-11-20T11:59:16.075 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-11-20T11:59:16.111 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:16 +0000 (0:00:00.244) 0:00:40.714 ******* 2020-11-20T11:59:16.161 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi005.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:59:16.170 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:59:16.172 INFO:teuthology.task.ansible.out: 2020-11-20T11:59:16.197 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:59:16.209 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:59:16.242 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:16 +0000 (0:00:00.131) 0:00:40.845 ******* 2020-11-20T11:59:17.148 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-20T11:59:17.150 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-11-20T11:59:17.184 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:17 +0000 (0:00:00.942) 0:00:41.787 ******* 2020-11-20T11:59:17.212 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-20T11:59:17.214 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:17.250 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:17 +0000 (0:00:00.065) 0:00:41.853 ******* 2020-11-20T11:59:17.540 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-20T11:59:17.542 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-11-20T11:59:17.578 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:17 +0000 (0:00:00.328) 0:00:42.182 ******* 2020-11-20T11:59:17.605 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-20T11:59:17.607 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:17.646 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:17 +0000 (0:00:00.067) 0:00:42.249 ******* 2020-11-20T11:59:18.147 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-20T11:59:18.149 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-11-20T11:59:18.182 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:18 +0000 (0:00:00.536) 0:00:42.786 ******* 2020-11-20T11:59:18.593 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-11-20T11:59:18.628 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:18 +0000 (0:00:00.444) 0:00:43.231 ******* 2020-11-20T11:59:19.130 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-20T11:59:19.132 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-11-20T11:59:19.166 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:19 +0000 (0:00:00.538) 0:00:43.769 ******* 2020-11-20T11:59:19.205 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-20T11:59:19.206 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2020-11-20T11:59:19.242 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:19 +0000 (0:00:00.076) 0:00:43.846 ******* 2020-11-20T11:59:19.589 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-11-20T11:59:19.624 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:19 +0000 (0:00:00.381) 0:00:44.227 ******* 2020-11-20T11:59:19.630 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-20T11:59:19.632 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:59:19.672 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:19 +0000 (0:00:00.048) 0:00:44.275 ******* 2020-11-20T11:59:19.679 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi005.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2020-11-20T11:59:19.681 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2020-11-20T11:59:19.721 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:19 +0000 (0:00:00.049) 0:00:44.325 ******* 2020-11-20T11:59:20.080 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-20T11:59:20.082 INFO:teuthology.task.ansible.out:fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-11-20T11:59:20.117 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:20 +0000 (0:00:00.394) 0:00:44.719 ******* 2020-11-20T11:59:20.122 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-20T11:59:20.124 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:59:20.158 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:20 +0000 (0:00:00.042) 0:00:44.762 ******* 2020-11-20T11:59:20.166 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-20T11:59:20.167 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:20.206 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:20 +0000 (0:00:00.047) 0:00:44.809 ******* 2020-11-20T11:59:20.578 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi005.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-11-20T11:59:20.621 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:20 +0000 (0:00:00.414) 0:00:45.224 ******* 2020-11-20T11:59:20.672 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:20.704 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:20 +0000 (0:00:00.083) 0:00:45.307 ******* 2020-11-20T11:59:21.167 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2020-11-20T11:59:21.204 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:21 +0000 (0:00:00.499) 0:00:45.807 ******* 2020-11-20T11:59:21.702 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2020-11-20T11:59:21.748 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:21 +0000 (0:00:00.544) 0:00:46.351 ******* 2020-11-20T11:59:22.133 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service nrpe: host"} ...ignoring 2020-11-20T11:59:22.165 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:22 +0000 (0:00:00.417) 0:00:46.769 ******* 2020-11-20T11:59:22.298 INFO:teuthology.task.ansible.out:fatal: [smithi005.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service nrpe: host"} ...ignoring 2020-11-20T11:59:22.333 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:22 +0000 (0:00:00.167) 0:00:46.936 ******* 2020-11-20T11:59:22.513 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.007010", "end": "2020-11-20 11:59:22.471748", "rc": 0, "start": "2020-11-20 11:59:22.464738", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-11-20T11:59:22.552 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:22 +0000 (0:00:00.218) 0:00:47.155 ******* 2020-11-20T11:59:22.685 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.008393", "end": "2020-11-20 11:59:22.651531", "rc": 0, "start": "2020-11-20 11:59:22.643138", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-11-20T11:59:22.722 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:22 +0000 (0:00:00.168) 0:00:47.324 ******* 2020-11-20T11:59:23.815 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-20T11:59:23.816 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-11-20T11:59:23.854 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:23 +0000 (0:00:01.133) 0:00:48.457 ******* 2020-11-20T11:59:24.011 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-11-20T11:59:24.054 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:24 +0000 (0:00:00.200) 0:00:48.658 ******* 2020-11-20T11:59:25.137 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-20T11:59:25.139 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-11-20T11:59:25.173 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:25 +0000 (0:00:01.119) 0:00:49.777 ******* 2020-11-20T11:59:25.348 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-11-20T11:59:25.383 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:25 +0000 (0:00:00.209) 0:00:49.986 ******* 2020-11-20T11:59:27.014 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-11-20T11:59:27.218 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:27 +0000 (0:00:01.664) 0:00:51.651 ******* ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} Friday 20 November 2020 11:59:27 +0000 (0:00:00.095) 0:00:51.747 ******* 2020-11-20T11:59:43.805 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-20T11:59:43.807 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.304580", "end": "2020-11-20 11:59:43.748837", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:59:27.444257", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-11-20T11:59:43.847 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:43 +0000 (0:00:16.701) 0:01:08.448 ******* 2020-11-20T11:59:44.299 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.874999", "end": "2020-11-20 11:59:44.237129", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:59:27.362130", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-11-20T11:59:44.339 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:44 +0000 (0:00:00.494) 0:01:08.943 ******* 2020-11-20T11:59:44.751 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-20T11:59:44.753 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-11-20T11:59:44.787 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:44 +0000 (0:00:00.447) 0:01:09.390 ******* 2020-11-20T11:59:45.142 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-20T11:59:45.144 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013453", "end": "2020-11-20 11:59:45.103983", "rc": 0, "start": "2020-11-20 11:59:45.090530", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:59:45.186 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:45 +0000 (0:00:00.398) 0:01:09.789 ******* 2020-11-20T11:59:45.326 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-11-20T11:59:45.380 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:45 +0000 (0:00:00.193) 0:01:09.983 ******* 2020-11-20T11:59:45.537 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-20T11:59:45.539 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.016708", "end": "2020-11-20 11:59:45.499502", "rc": 0, "start": "2020-11-20 11:59:45.482794", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:59:45.572 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:45 +0000 (0:00:00.191) 0:01:10.175 ******* 2020-11-20T11:59:45.732 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.018824", "end": "2020-11-20 11:59:45.701104", "rc": 0, "start": "2020-11-20 11:59:45.682280", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:59:45.768 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:45 +0000 (0:00:00.194) 0:01:10.369 ******* 2020-11-20T11:59:46.133 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.014369", "end": "2020-11-20 11:59:46.092037", "rc": 0, "start": "2020-11-20 11:59:46.077668", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:59:46.168 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:46 +0000 (0:00:00.401) 0:01:10.771 ******* 2020-11-20T12:00:02.070 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-20T12:00:02.072 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.160098", "end": "2020-11-20 12:00:02.033075", "rc": 0, "start": "2020-11-20 11:59:45.872977", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:02.105 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:15.937) 0:01:26.708 ******* 2020-11-20T12:00:02.451 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-20T12:00:02.452 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-11-20T12:00:02.490 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.383) 0:01:27.092 ******* 2020-11-20T12:00:02.521 INFO:teuthology.task.ansible.out: TASK [common : Check for podman] *********************************************** 2020-11-20T12:00:02.527 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.577 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.087) 0:01:27.179 ******* 2020-11-20T12:00:02.633 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000"}, "changed": false} 2020-11-20T12:00:02.673 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.095) 0:01:27.275 ******* 2020-11-20T12:00:02.725 INFO:teuthology.task.ansible.out: TASK [common : Check for docker] *********************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.755 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.083) 0:01:27.358 ******* 2020-11-20T12:00:02.796 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.845 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.088) 0:01:27.447 ******* 2020-11-20T12:00:02.924 INFO:teuthology.task.ansible.out: TASK [common : Include encrypted variables] ************************************ ok: [smithi187.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:02.926 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:02.980 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.135) 0:01:27.583 ******* 2020-11-20T12:00:03.340 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-20T12:00:03.341 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.821292", "end": "2020-11-20 12:00:03.277167", "rc": 0, "start": "2020-11-20 11:59:46.455875", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:03.381 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.401) 0:01:27.984 ******* 2020-11-20T12:00:03.390 INFO:teuthology.task.ansible.out: TASK [common : Create /etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000] *** 2020-11-20T12:00:03.392 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:00:03.431 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.049) 0:01:28.034 ******* 2020-11-20T12:00:03.728 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-11-20T12:00:03.764 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.332) 0:01:28.367 ******* 2020-11-20T12:00:03.792 INFO:teuthology.task.ansible.out: TASK [common : Check for podman] *********************************************** 2020-11-20T12:00:03.794 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.833 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.066) 0:01:28.433 ******* 2020-11-20T12:00:03.885 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-20T12:00:03.886 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000"}, "changed": false} 2020-11-20T12:00:03.939 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.108) 0:01:28.542 ******* 2020-11-20T12:00:03.972 INFO:teuthology.task.ansible.out: TASK [common : Check for docker] *********************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.015 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.069) 0:01:28.612 ******* 2020-11-20T12:00:04.043 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-20T12:00:04.050 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.081 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.071) 0:01:28.684 ******* 2020-11-20T12:00:04.172 INFO:teuthology.task.ansible.out: TASK [common : Include encrypted variables] ************************************ ok: [smithi005.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:04.208 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.127) 0:01:28.811 ******* 2020-11-20T12:00:04.639 INFO:teuthology.task.ansible.out: TASK [common : Create /etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000] *** 2020-11-20T12:00:04.643 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:00:04.679 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.471) 0:01:29.282 ******* 2020-11-20T12:00:04.702 INFO:teuthology.task.ansible.out: TASK [common : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605873603.4725158-242074371266213/source", "state": "file", "uid": 0} 2020-11-20T12:00:04.751 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.071) 0:01:29.354 ******* 2020-11-20T12:00:04.793 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-11-20T12:00:04.828 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.076) 0:01:29.431 ******* 2020-11-20T12:00:04.884 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T12:00:04.952 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.124) 0:01:29.556 ******* 2020-11-20T12:00:04.994 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-20T12:00:05.001 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-11-20T12:00:05.032 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.079) 0:01:29.635 ******* 2020-11-20T12:00:05.082 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T12:00:05.117 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.083) 0:01:29.719 ******* 2020-11-20T12:00:05.478 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-20T12:00:05.481 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-20T12:00:05.520 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.405) 0:01:30.124 ******* 2020-11-20T12:00:05.878 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-20T12:00:05.924 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.403) 0:01:30.527 ******* 2020-11-20T12:00:05.930 INFO:teuthology.task.ansible.out: TASK [common : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2020-11-20T12:00:05.933 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605873604.7150304-131222269681341/source", "state": "file", "uid": 0} 2020-11-20T12:00:05.983 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.059) 0:01:30.587 ******* 2020-11-20T12:00:05.994 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:06.046 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.062) 0:01:30.649 ******* 2020-11-20T12:00:06.052 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-20T12:00:06.054 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-11-20T12:00:06.090 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.043) 0:01:30.693 ******* 2020-11-20T12:00:06.096 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-20T12:00:06.098 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:06.133 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.043) 0:01:30.736 ******* 2020-11-20T12:00:06.139 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-20T12:00:06.141 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T12:00:06.189 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.056) 0:01:30.793 ******* 2020-11-20T12:00:06.242 INFO:teuthology.task.ansible.out: 2020-11-20T12:00:06.244 INFO:teuthology.task.ansible.out:TASK [testnode : Including major version specific variables.] ****************** ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml 2020-11-20T12:00:06.246 INFO:teuthology.task.ansible.out:"} 2020-11-20T12:00:06.291 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.100) 0:01:30.894 ******* 2020-11-20T12:00:06.350 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-20T12:00:06.352 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T12:00:06.396 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.105) 0:01:30.999 ******* 2020-11-20T12:00:06.525 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-20T12:00:06.527 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-11-20T12:00:06.573 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.176) 0:01:31.176 ******* 2020-11-20T12:00:06.760 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-20T12:00:06.762 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-20T12:00:06.803 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.230) 0:01:31.406 ******* 2020-11-20T12:00:07.178 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-20T12:00:07.180 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-20T12:00:07.215 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:07 +0000 (0:00:00.412) 0:01:31.819 ******* 2020-11-20T12:00:07.242 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-20T12:00:07.244 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:07.278 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:07 +0000 (0:00:00.062) 0:01:31.881 ******* 2020-11-20T12:00:07.305 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-20T12:00:07.306 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:07.338 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:07 +0000 (0:00:00.060) 0:01:31.942 ******* 2020-11-20T12:00:07.745 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-20T12:00:07.747 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi005.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-11-20T12:00:07.787 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:07 +0000 (0:00:00.448) 0:01:32.390 ******* 2020-11-20T12:00:07.824 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:07 +0000 (0:00:00.036) 0:01:32.427 ******* 2020-11-20T12:00:07.864 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi187"}, "changed": false} 2020-11-20T12:00:07.923 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:07 +0000 (0:00:00.099) 0:01:32.526 ******* 2020-11-20T12:00:08.914 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-20T12:00:08.914 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-11-20T12:00:08.953 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:08 +0000 (0:00:01.029) 0:01:33.556 ******* 2020-11-20T12:00:08.992 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-20T12:00:08.994 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi005"}, "changed": false} 2020-11-20T12:00:09.028 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:09 +0000 (0:00:00.074) 0:01:33.631 ******* 2020-11-20T12:00:09.254 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-11-20T12:00:09.256 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi187.front.sepia.ceph.com", "ansible_hostname": "smithi187", "ansible_nodename": "smithi187"}, "changed": true, "name": "smithi187"} 2020-11-20T12:00:09.290 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:09 +0000 (0:00:00.262) 0:01:33.893 ******* 2020-11-20T12:00:09.991 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi005.front.sepia.ceph.com", "ansible_hostname": "smithi005", "ansible_nodename": "smithi005"}, "changed": true, "name": "smithi005"} 2020-11-20T12:00:10.048 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:10 +0000 (0:00:00.757) 0:01:34.651 ******* 2020-11-20T12:00:10.247 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-11-20T12:00:10.301 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:10 +0000 (0:00:00.253) 0:01:34.904 ******* 2020-11-20T12:00:11.050 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-11-20T12:00:11.085 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:11 +0000 (0:00:00.782) 0:01:35.687 ******* 2020-11-20T12:00:11.275 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-20T12:00:11.276 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-11-20T12:00:11.323 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:11 +0000 (0:00:00.238) 0:01:35.926 ******* 2020-11-20T12:00:12.075 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-11-20T12:00:12.110 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:12 +0000 (0:00:00.787) 0:01:36.714 ******* 2020-11-20T12:00:12.180 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-20T12:00:12.182 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-20T12:00:12.216 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:12 +0000 (0:00:00.105) 0:01:36.820 ******* 2020-11-20T12:00:13.011 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-20T12:00:13.052 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:13 +0000 (0:00:00.835) 0:01:37.656 ******* 2020-11-20T12:00:13.123 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-20T12:00:13.125 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-11-20T12:00:13.167 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:13 +0000 (0:00:00.114) 0:01:37.770 ******* 2020-11-20T12:00:13.519 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T12:00:13.520 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-20T12:00:13.552 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:13 +0000 (0:00:00.385) 0:01:38.155 ******* 2020-11-20T12:00:13.898 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T12:00:13.900 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-11-20T12:00:13.933 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:13 +0000 (0:00:00.381) 0:01:38.537 ******* 2020-11-20T12:00:14.045 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-20T12:00:14.047 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-11-20T12:00:14.082 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:14 +0000 (0:00:00.147) 0:01:38.684 ******* 2020-11-20T12:00:14.316 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-20T12:00:14.318 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-20T12:00:14.353 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:14 +0000 (0:00:00.271) 0:01:38.956 ******* 2020-11-20T12:00:14.454 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T12:00:14.457 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-20T12:00:14.492 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:14 +0000 (0:00:00.139) 0:01:39.096 ******* 2020-11-20T12:00:14.860 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-11-20T12:00:14.893 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:14 +0000 (0:00:00.400) 0:01:39.496 ******* 2020-11-20T12:00:15.293 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-11-20T12:00:15.328 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.435) 0:01:39.931 ******* 2020-11-20T12:00:15.335 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi005.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-20T12:00:15.373 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.044) 0:01:39.976 ******* 2020-11-20T12:00:16.293 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-20T12:00:16.295 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-11-20T12:00:16.331 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:16 +0000 (0:00:00.958) 0:01:40.934 ******* 2020-11-20T12:00:16.362 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-20T12:00:16.365 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-11-20T12:00:16.402 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:16 +0000 (0:00:00.071) 0:01:41.006 ******* 2020-11-20T12:00:17.450 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-20T12:00:17.452 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-11-20T12:00:17.489 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:01.083) 0:01:42.089 ******* 2020-11-20T12:00:18.969 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T12:00:18.971 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.263682", "end": "2020-11-20 12:00:18.921726", "rc": 0, "start": "2020-11-20 12:00:16.658044", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:19.010 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:01.524) 0:01:43.613 ******* 2020-11-20T12:00:19.948 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-11-20T12:00:19.951 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2020-11-20T12:00:20.260 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T12:00:20.262 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.405234", "end": "2020-11-20 12:00:20.218498", "rc": 0, "start": "2020-11-20 12:00:17.813264", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:20.307 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:20 +0000 (0:00:01.297) 0:01:44.910 ******* 2020-11-20T12:00:20.968 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** ok: [smithi187.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2020-11-20T12:00:21.351 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2020-11-20T12:00:21.904 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2020-11-20T12:00:21.940 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:21 +0000 (0:00:01.632) 0:01:46.543 ******* 2020-11-20T12:00:22.374 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2020-11-20T12:00:22.951 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-20T12:00:22.953 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-11-20T12:00:22.986 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:22 +0000 (0:00:01.045) 0:01:47.589 ******* 2020-11-20T12:00:23.354 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-11-20T12:00:23.355 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2020-11-20T12:00:23.392 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:23 +0000 (0:00:00.405) 0:01:47.995 ******* 2020-11-20T12:00:23.940 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-20T12:00:23.942 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-11-20T12:00:24.380 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi005.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-11-20T12:00:24.418 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:24 +0000 (0:00:01.025) 0:01:49.021 ******* 2020-11-20T12:00:24.864 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-11-20T12:00:25.421 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-11-20T12:00:25.783 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-11-20T12:00:25.824 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:01.406) 0:01:50.427 ******* 2020-11-20T12:00:25.851 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-20T12:00:25.886 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.061) 0:01:50.489 ******* 2020-11-20T12:00:25.914 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-11-20T12:00:25.915 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:25.948 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.062) 0:01:50.551 ******* 2020-11-20T12:00:25.986 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-20T12:00:25.987 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:25.998 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.032 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.084) 0:01:50.635 ******* 2020-11-20T12:00:26.059 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.095 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.063) 0:01:50.699 ******* 2020-11-20T12:00:26.133 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:00:26.135 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-11-20T12:00:26.176 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.080) 0:01:50.780 ******* 2020-11-20T12:00:26.216 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-20T12:00:26.218 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.264 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.087) 0:01:50.867 ******* 2020-11-20T12:00:26.305 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.583 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.075) 0:01:50.942 ******* TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi005.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-11-20T12:00:26.709 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.020790", "end": "2020-11-20 12:00:26.674196", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 12:00:26.653406", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-11-20T12:00:26.746 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.407) 0:01:51.350 ******* 2020-11-20T12:00:26.778 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.817 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.066) 0:01:51.416 ******* 2020-11-20T12:00:27.356 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi005.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-11-20T12:00:27.394 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.581) 0:01:51.997 ******* 2020-11-20T12:00:27.422 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-20T12:00:27.462 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.067) 0:01:52.065 ******* 2020-11-20T12:00:27.489 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-11-20T12:00:27.491 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:27.525 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.062) 0:01:52.128 ******* 2020-11-20T12:00:27.562 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-20T12:00:27.564 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:27.575 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:27.609 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.085) 0:01:52.213 ******* 2020-11-20T12:00:27.637 INFO:teuthology.task.ansible.out: 2020-11-20T12:00:27.638 INFO:teuthology.task.ansible.out:TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:27.677 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.067) 0:01:52.280 ******* 2020-11-20T12:00:27.704 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-11-20T12:00:27.738 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.061) 0:01:52.342 ******* 2020-11-20T12:00:27.776 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-20T12:00:27.779 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:27.815 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.076) 0:01:52.418 ******* 2020-11-20T12:00:27.853 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-20T12:00:27.854 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:27.889 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.073) 0:01:52.492 ******* 2020-11-20T12:00:28.284 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-20T12:00:28.286 INFO:teuthology.task.ansible.out:fatal: [smithi005.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.022138", "end": "2020-11-20 12:00:28.249676", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 12:00:28.227538", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-11-20T12:00:28.331 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:28 +0000 (0:00:00.442) 0:01:52.934 ******* 2020-11-20T12:00:28.361 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-20T12:00:28.363 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:28.399 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:28 +0000 (0:00:00.068) 0:01:53.002 ******* 2020-11-20T12:00:29.837 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-20T12:00:29.839 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-11-20T12:00:29.872 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:01.473) 0:01:54.476 ******* 2020-11-20T12:00:31.469 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-11-20T12:00:31.525 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:01.651) 0:01:56.128 ******* 2020-11-20T12:00:33.190 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} Friday 20 November 2020 12:00:32 +0000 (0:00:01.072) 0:01:57.201 ******* 2020-11-20T12:00:34.298 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2020-11-20T12:00:34.333 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:34 +0000 (0:00:01.735) 0:01:58.936 ******* 2020-11-20T12:00:36.711 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T12:00:36.713 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64"]} 2020-11-20T12:00:36.748 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:36 +0000 (0:00:02.414) 0:02:01.351 ******* 2020-11-20T12:00:36.775 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-20T12:00:36.777 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:36.809 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:36 +0000 (0:00:00.061) 0:02:01.413 ******* 2020-11-20T12:00:38.060 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T12:00:38.063 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-11-20T12:00:38.094 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:38 +0000 (0:00:01.284) 0:02:02.697 ******* 2020-11-20T12:00:38.121 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:38.155 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:38 +0000 (0:00:00.061) 0:02:02.759 ******* 2020-11-20T12:00:38.522 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64"]} 2020-11-20T12:00:38.557 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:38 +0000 (0:00:00.401) 0:02:03.160 ******* 2020-11-20T12:00:38.584 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-20T12:00:38.586 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:38.619 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:38 +0000 (0:00:00.061) 0:02:03.221 ******* 2020-11-20T12:00:38.677 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-20T12:00:38.679 INFO:teuthology.task.ansible.out:fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service firewalld: host"} ...ignoring 2020-11-20T12:00:38.713 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:38 +0000 (0:00:00.094) 0:02:03.316 ******* 2020-11-20T12:00:39.213 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-20T12:00:39.215 INFO:teuthology.task.ansible.out:fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-11-20T12:00:39.257 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:39 +0000 (0:00:00.543) 0:02:03.860 ******* 2020-11-20T12:00:39.966 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T12:00:39.967 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-11-20T12:00:40.005 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:39 +0000 (0:00:00.747) 0:02:04.608 ******* 2020-11-20T12:00:40.031 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T12:00:40.033 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.080 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.075) 0:02:04.683 ******* 2020-11-20T12:00:40.097 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-20T12:00:40.099 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-11-20T12:00:40.134 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.054) 0:02:04.737 ******* 2020-11-20T12:00:40.161 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.197 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.062) 0:02:04.800 ******* 2020-11-20T12:00:40.224 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-20T12:00:40.225 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.259 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.061) 0:02:04.862 ******* 2020-11-20T12:00:40.287 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-20T12:00:40.288 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.321 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.062) 0:02:04.924 ******* 2020-11-20T12:00:40.358 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.395 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.074) 0:02:04.998 ******* 2020-11-20T12:00:40.423 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.464 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.068) 0:02:05.067 ******* 2020-11-20T12:00:40.493 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-20T12:00:40.494 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.535 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.070) 0:02:05.138 ******* 2020-11-20T12:00:40.562 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-20T12:00:40.599 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.063) 0:02:05.202 ******* 2020-11-20T12:00:40.605 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-20T12:00:40.607 INFO:teuthology.task.ansible.out:fatal: [smithi005.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service firewalld: host"} ...ignoring 2020-11-20T12:00:40.643 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.043) 0:02:05.246 ******* 2020-11-20T12:00:40.649 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-20T12:00:40.651 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.687 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.044) 0:02:05.290 ******* 2020-11-20T12:00:40.714 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-20T12:00:40.716 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.749 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.061) 0:02:05.352 ******* 2020-11-20T12:00:40.787 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-20T12:00:40.789 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.823 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.073) 0:02:05.426 ******* 2020-11-20T12:00:40.849 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-20T12:00:40.851 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:40.885 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.061) 0:02:05.487 ******* 2020-11-20T12:00:40.910 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-20T12:00:40.944 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.060) 0:02:05.548 ******* 2020-11-20T12:00:40.971 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-20T12:00:40.973 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.009 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.064) 0:02:05.612 ******* 2020-11-20T12:00:41.036 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-20T12:00:41.038 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.070 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.061) 0:02:05.674 ******* 2020-11-20T12:00:41.098 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.131 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.060) 0:02:05.735 ******* 2020-11-20T12:00:41.158 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T12:00:41.160 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi005.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-11-20T12:00:41.192 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.060) 0:02:05.795 ******* 2020-11-20T12:00:41.226 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.034) 0:02:05.829 ******* 2020-11-20T12:00:41.253 INFO:teuthology.task.ansible.out: 2020-11-20T12:00:41.255 INFO:teuthology.task.ansible.out:TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.292 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.066) 0:02:05.896 ******* 2020-11-20T12:00:41.319 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T12:00:41.321 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.353 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.060) 0:02:05.956 ******* 2020-11-20T12:00:41.381 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-20T12:00:41.383 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.418 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.064) 0:02:06.021 ******* 2020-11-20T12:00:41.445 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-20T12:00:41.446 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.485 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.067) 0:02:06.089 ******* 2020-11-20T12:00:41.513 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-20T12:00:41.515 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.551 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.065) 0:02:06.154 ******* 2020-11-20T12:00:41.579 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-20T12:00:41.581 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.619 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.068) 0:02:06.222 ******* 2020-11-20T12:00:41.647 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-20T12:00:41.649 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.684 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.064) 0:02:06.287 ******* 2020-11-20T12:00:41.690 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-20T12:00:41.692 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-11-20T12:00:41.724 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.040) 0:02:06.327 ******* 2020-11-20T12:00:41.729 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-20T12:00:41.731 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.765 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.041) 0:02:06.368 ******* 2020-11-20T12:00:41.770 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-20T12:00:41.772 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.805 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.040) 0:02:06.409 ******* 2020-11-20T12:00:41.811 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-20T12:00:41.813 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.855 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.049) 0:02:06.459 ******* 2020-11-20T12:00:41.862 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-20T12:00:41.864 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.899 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.043) 0:02:06.502 ******* 2020-11-20T12:00:41.905 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-20T12:00:41.906 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.939 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.040) 0:02:06.542 ******* 2020-11-20T12:00:41.944 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-20T12:00:41.946 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:41.979 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:41 +0000 (0:00:00.039) 0:02:06.582 ******* 2020-11-20T12:00:41.984 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T12:00:41.986 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.019 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.039) 0:02:06.622 ******* 2020-11-20T12:00:42.024 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-20T12:00:42.026 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.060 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:06.663 ******* 2020-11-20T12:00:42.067 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T12:00:42.069 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.102 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.042) 0:02:06.706 ******* 2020-11-20T12:00:42.108 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-20T12:00:42.110 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.143 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.040) 0:02:06.746 ******* 2020-11-20T12:00:42.149 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-20T12:00:42.151 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.188 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.045) 0:02:06.792 ******* 2020-11-20T12:00:42.195 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-20T12:00:42.197 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.230 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:06.834 ******* 2020-11-20T12:00:42.237 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-20T12:00:42.238 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.271 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.040) 0:02:06.874 ******* 2020-11-20T12:00:42.277 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-20T12:00:42.312 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:06.915 ******* 2020-11-20T12:00:42.320 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.355 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.042) 0:02:06.958 ******* 2020-11-20T12:00:42.361 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-20T12:00:42.363 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.397 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.042) 0:02:07.000 ******* 2020-11-20T12:00:42.403 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T12:00:42.405 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.438 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:07.042 ******* 2020-11-20T12:00:42.444 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-20T12:00:42.446 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.479 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.040) 0:02:07.082 ******* 2020-11-20T12:00:42.486 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-20T12:00:42.488 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.524 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.044) 0:02:07.127 ******* 2020-11-20T12:00:42.530 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-20T12:00:42.532 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.565 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:07.169 ******* 2020-11-20T12:00:42.571 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-20T12:00:42.573 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.606 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:07.210 ******* 2020-11-20T12:00:42.612 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-20T12:00:42.614 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.648 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:07.251 ******* 2020-11-20T12:00:42.654 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:00:42.656 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.689 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.040) 0:02:07.291 ******* 2020-11-20T12:00:42.694 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-20T12:00:42.730 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:07.333 ******* 2020-11-20T12:00:42.736 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T12:00:42.738 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.771 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.041) 0:02:07.375 ******* 2020-11-20T12:00:42.777 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-20T12:00:42.779 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.817 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.045) 0:02:07.420 ******* 2020-11-20T12:00:42.907 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:42 +0000 (0:00:00.054) 0:02:07.475 ******* TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.927 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.054) 0:02:07.530 ******* 2020-11-20T12:00:42.934 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T12:00:42.936 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:42.970 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:42 +0000 (0:00:00.043) 0:02:07.573 ******* 2020-11-20T12:00:42.977 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.028 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.057) 0:02:07.631 ******* 2020-11-20T12:00:43.037 INFO:teuthology.task.ansible.out: 2020-11-20T12:00:43.039 INFO:teuthology.task.ansible.out:TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.084 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.055) 0:02:07.687 ******* 2020-11-20T12:00:43.092 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T12:00:43.097 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.139 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.055) 0:02:07.743 ******* 2020-11-20T12:00:43.169 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.214 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.074) 0:02:07.817 ******* 2020-11-20T12:00:43.242 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T12:00:43.244 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.277 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.063) 0:02:07.880 ******* 2020-11-20T12:00:43.305 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-20T12:00:43.307 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.352 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.075) 0:02:07.956 ******* 2020-11-20T12:00:43.382 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-20T12:00:43.384 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.418 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.065) 0:02:08.022 ******* 2020-11-20T12:00:43.436 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:00:43.438 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-11-20T12:00:43.471 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.052) 0:02:08.074 ******* 2020-11-20T12:00:43.477 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-20T12:00:43.479 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.513 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.041) 0:02:08.116 ******* 2020-11-20T12:00:43.540 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.575 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.062) 0:02:08.178 ******* 2020-11-20T12:00:43.603 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-20T12:00:43.605 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.637 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.062) 0:02:08.241 ******* 2020-11-20T12:00:43.664 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-20T12:00:43.666 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.700 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.061) 0:02:08.302 ******* 2020-11-20T12:00:43.726 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-20T12:00:43.728 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.761 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.062) 0:02:08.365 ******* 2020-11-20T12:00:43.789 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.823 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.061) 0:02:08.426 ******* 2020-11-20T12:00:43.830 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:00:43.832 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:00:43.867 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.043) 0:02:08.470 ******* 2020-11-20T12:00:43.873 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T12:00:43.875 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.911 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.043) 0:02:08.514 ******* 2020-11-20T12:00:43.949 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T12:00:43.951 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:43.990 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:43 +0000 (0:00:00.079) 0:02:08.594 ******* 2020-11-20T12:00:44.018 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-20T12:00:44.020 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.053 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.062) 0:02:08.656 ******* 2020-11-20T12:00:44.081 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-20T12:00:44.083 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.124 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.071) 0:02:08.728 ******* 2020-11-20T12:00:44.151 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.186 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.061) 0:02:08.789 ******* 2020-11-20T12:00:44.214 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T12:00:44.215 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-11-20T12:00:44.250 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.063) 0:02:08.853 ******* 2020-11-20T12:00:44.283 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.032) 0:02:08.886 ******* 2020-11-20T12:00:44.289 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-20T12:00:44.291 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.326 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.043) 0:02:08.929 ******* 2020-11-20T12:00:44.334 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T12:00:44.336 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.370 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.043) 0:02:08.973 ******* 2020-11-20T12:00:44.376 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-20T12:00:44.378 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.416 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.046) 0:02:09.019 ******* 2020-11-20T12:00:44.424 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T12:00:44.426 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.475 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.058) 0:02:09.078 ******* 2020-11-20T12:00:44.488 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:00:44.490 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.534 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.059) 0:02:09.137 ******* 2020-11-20T12:00:44.542 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T12:00:44.544 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.579 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.044) 0:02:09.182 ******* 2020-11-20T12:00:44.586 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T12:00:44.588 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.634 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.055) 0:02:09.237 ******* 2020-11-20T12:00:44.641 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.685 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.050) 0:02:09.288 ******* 2020-11-20T12:00:44.697 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.744 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.058) 0:02:09.347 ******* 2020-11-20T12:00:44.751 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.795 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.049) 0:02:09.396 ******* 2020-11-20T12:00:44.801 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T12:00:44.803 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.860 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.067) 0:02:09.464 ******* 2020-11-20T12:00:44.871 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T12:00:44.873 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.912 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.051) 0:02:09.515 ******* 2020-11-20T12:00:44.918 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T12:00:44.926 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:44.960 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.048) 0:02:09.563 ******* 2020-11-20T12:00:44.967 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:00:44.969 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.004 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:44 +0000 (0:00:00.043) 0:02:09.607 ******* 2020-11-20T12:00:45.029 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:00:45.031 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.063 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.059) 0:02:09.666 ******* 2020-11-20T12:00:45.089 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T12:00:45.091 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.125 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.061) 0:02:09.728 ******* 2020-11-20T12:00:45.154 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-20T12:00:45.156 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.191 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.066) 0:02:09.795 ******* 2020-11-20T12:00:45.229 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-20T12:00:45.231 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.264 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.072) 0:02:09.867 ******* 2020-11-20T12:00:45.291 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T12:00:45.293 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.326 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.062) 0:02:09.929 ******* 2020-11-20T12:00:45.343 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:00:45.345 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-11-20T12:00:45.378 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.051) 0:02:09.981 ******* 2020-11-20T12:00:45.385 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T12:00:45.386 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.423 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.045) 0:02:10.026 ******* 2020-11-20T12:00:45.451 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-20T12:00:45.453 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.513 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.088) 0:02:10.115 ******* 2020-11-20T12:00:45.542 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.590 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.078) 0:02:10.194 ******* 2020-11-20T12:00:45.619 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-20T12:00:45.620 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.653 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.062) 0:02:10.256 ******* 2020-11-20T12:00:45.680 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-20T12:00:45.682 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.723 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.069) 0:02:10.326 ******* 2020-11-20T12:00:45.754 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:00:45.755 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.790 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.066) 0:02:10.393 ******* 2020-11-20T12:00:45.822 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.031) 0:02:10.425 ******* 2020-11-20T12:00:45.860 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-20T12:00:45.862 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.911 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.089) 0:02:10.514 ******* 2020-11-20T12:00:45.939 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-20T12:00:45.941 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:45.983 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:45 +0000 (0:00:00.071) 0:02:10.586 ******* 2020-11-20T12:00:46.010 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-20T12:00:46.012 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.053 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.070) 0:02:10.657 ******* 2020-11-20T12:00:46.083 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-20T12:00:46.085 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.117 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.063) 0:02:10.721 ******* 2020-11-20T12:00:46.147 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:00:46.149 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-20T12:00:46.183 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.065) 0:02:10.786 ******* 2020-11-20T12:00:46.191 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T12:00:46.193 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-11-20T12:00:46.229 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.046) 0:02:10.832 ******* 2020-11-20T12:00:46.236 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-20T12:00:46.238 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.248 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-20T12:00:46.250 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.288 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.058) 0:02:10.891 ******* 2020-11-20T12:00:46.295 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-20T12:00:46.297 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.344 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.056) 0:02:10.947 ******* 2020-11-20T12:00:46.351 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T12:00:46.353 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.388 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.043) 0:02:10.991 ******* 2020-11-20T12:00:46.418 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T12:00:46.420 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.453 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.065) 0:02:11.057 ******* 2020-11-20T12:00:46.481 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:00:46.483 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.527 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.073) 0:02:11.130 ******* 2020-11-20T12:00:46.556 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-20T12:00:46.558 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003554", "end": "2020-11-20 12:00:46.525284", "rc": 0, "start": "2020-11-20 12:00:46.521730", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-20T12:00:46.603 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.075) 0:02:11.206 ******* 2020-11-20T12:00:46.612 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:00:46.614 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.654 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.051) 0:02:11.258 ******* 2020-11-20T12:00:46.664 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.704 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.050) 0:02:11.308 ******* 2020-11-20T12:00:46.711 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T12:00:46.713 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.752 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.047) 0:02:11.355 ******* 2020-11-20T12:00:46.761 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-20T12:00:46.763 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.799 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.046) 0:02:11.402 ******* 2020-11-20T12:00:46.805 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:00:46.807 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.865 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.066) 0:02:11.469 ******* 2020-11-20T12:00:46.875 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-20T12:00:46.879 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.917 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.051) 0:02:11.520 ******* 2020-11-20T12:00:46.925 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:00:46.928 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:46.964 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:46 +0000 (0:00:00.046) 0:02:11.567 ******* 2020-11-20T12:00:46.973 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-20T12:00:46.975 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.011 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.046) 0:02:11.614 ******* 2020-11-20T12:00:47.017 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T12:00:47.019 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.059 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.046) 0:02:11.661 ******* 2020-11-20T12:00:47.085 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-20T12:00:47.086 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.119 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.061) 0:02:11.723 ******* 2020-11-20T12:00:47.146 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-20T12:00:47.148 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.181 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.061) 0:02:11.784 ******* 2020-11-20T12:00:47.208 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T12:00:47.209 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.242 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.061) 0:02:11.845 ******* 2020-11-20T12:00:47.269 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T12:00:47.271 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.303 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.061) 0:02:11.906 ******* 2020-11-20T12:00:47.341 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.378 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.075) 0:02:11.981 ******* 2020-11-20T12:00:47.406 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-20T12:00:47.408 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.442 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.063) 0:02:12.045 ******* 2020-11-20T12:00:47.472 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-20T12:00:47.474 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.521 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.079) 0:02:12.124 ******* 2020-11-20T12:00:47.548 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-20T12:00:47.550 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.583 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.061) 0:02:12.186 ******* 2020-11-20T12:00:47.610 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-20T12:00:47.611 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.646 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.062) 0:02:12.249 ******* 2020-11-20T12:00:47.683 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-20T12:00:47.685 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.723 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.077) 0:02:12.326 ******* 2020-11-20T12:00:47.750 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-20T12:00:47.752 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.784 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.061) 0:02:12.387 ******* 2020-11-20T12:00:47.811 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-20T12:00:47.813 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.846 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.061) 0:02:12.449 ******* 2020-11-20T12:00:47.873 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:47.907 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.061) 0:02:12.510 ******* 2020-11-20T12:00:47.945 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:00:47.947 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-20T12:00:47.979 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:47 +0000 (0:00:00.071) 0:02:12.582 ******* 2020-11-20T12:00:48.016 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-20T12:00:48.018 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:48.028 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-20T12:00:48.039 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:48.050 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:48.062 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:48.101 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:48 +0000 (0:00:00.122) 0:02:12.704 ******* 2020-11-20T12:00:48.266 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-20T12:00:48.267 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-11-20T12:00:48.301 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:48 +0000 (0:00:00.200) 0:02:12.905 ******* 2020-11-20T12:00:48.308 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-20T12:00:48.310 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003110", "end": "2020-11-20 12:00:48.267470", "rc": 0, "start": "2020-11-20 12:00:48.264360", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-20T12:00:48.348 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:48 +0000 (0:00:00.046) 0:02:12.951 ******* 2020-11-20T12:00:48.354 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T12:00:48.356 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 927554727936, 'block_size': 4096, 'block_total': 240095108, 'block_available': 226453791, 'block_used': 13641317, 'inode_total': 61054976, 'inode_available': 60988135, 'inode_used': 66841, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453791, "block_size": 4096, "block_total": 240095108, "block_used": 13641317, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60988135, "inode_total": 61054976, "inode_used": 66841, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 927554727936, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}} 2020-11-20T12:00:48.391 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:48 +0000 (0:00:00.043) 0:02:12.994 ******* 2020-11-20T12:00:48.397 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-20T12:00:48.399 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:48.438 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:48 +0000 (0:00:00.046) 0:02:13.041 ******* 2020-11-20T12:00:48.465 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-20T12:00:48.467 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:48.499 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:48 +0000 (0:00:00.060) 0:02:13.102 ******* 2020-11-20T12:00:48.526 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-20T12:00:48.528 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:48.560 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:48 +0000 (0:00:00.061) 0:02:13.163 ******* 2020-11-20T12:00:48.587 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-20T12:00:48.590 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:48.622 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:48 +0000 (0:00:00.061) 0:02:13.225 ******* 2020-11-20T12:00:49.207 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.020197", "end": "2020-11-20 12:00:48.704778", "rc": 0, "start": "2020-11-20 12:00:48.684581", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Friday 20 November 2020 12:00:48 +0000 (0:00:00.151) 0:02:13.376 ******* TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi187.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 927554727936, 'block_size': 4096, 'block_total': 240095108, 'block_available': 226453791, 'block_used': 13641317, 'inode_total': 61054976, 'inode_available': 60988135, 'inode_used': 66841, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453791, "block_size": 4096, "block_total": 240095108, "block_used": 13641317, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60988135, "inode_total": 61054976, "inode_used": 66841, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 927554727936, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:48 +0000 (0:00:00.073) 0:02:13.449 ******* TASK [testnode : Wipe filesystems on non-root partitions] ********************** Friday 20 November 2020 12:00:48 +0000 (0:00:00.060) 0:02:13.510 ******* 2020-11-20T12:00:49.957 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-20T12:00:49.960 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-11-20T12:00:49.992 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:49 +0000 (0:00:01.085) 0:02:14.596 ******* 2020-11-20T12:00:50.030 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T12:00:50.032 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 927554650112, 'block_size': 4096, 'block_total': 240095108, 'block_available': 226453772, 'block_used': 13641336, 'inode_total': 61054976, 'inode_available': 60988127, 'inode_used': 66849, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453772, "block_size": 4096, "block_total": 240095108, "block_used": 13641336, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60988127, "inode_total": 61054976, "inode_used": 66849, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 927554650112, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}} 2020-11-20T12:00:50.065 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:50 +0000 (0:00:00.072) 0:02:14.668 ******* 2020-11-20T12:00:50.436 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.021440", "end": "2020-11-20 12:00:50.398837", "rc": 0, "start": "2020-11-20 12:00:50.377397", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:50.479 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:50 +0000 (0:00:00.414) 0:02:15.083 ******* 2020-11-20T12:00:50.517 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-20T12:00:50.519 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 927554650112, 'block_size': 4096, 'block_total': 240095108, 'block_available': 226453772, 'block_used': 13641336, 'inode_total': 61054976, 'inode_available': 60988127, 'inode_used': 66849, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453772, "block_size": 4096, "block_total": 240095108, "block_used": 13641336, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60988127, "inode_total": 61054976, "inode_used": 66849, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 927554650112, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:50.553 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:50 +0000 (0:00:00.073) 0:02:15.156 ******* 2020-11-20T12:00:50.580 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-20T12:00:50.614 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:50 +0000 (0:00:00.061) 0:02:15.217 ******* 2020-11-20T12:00:50.653 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-20T12:00:50.655 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwG9l5QNtFR0YRCHz8CEKjZY693HIfJnAr'], 'uuids': ['6b6f1bbd-1d69-4b33-9fd0-17ab2a1bd58a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.383654", "end": "2020-11-20 12:00:50.603375", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwG9l5QNtFR0YRCHz8CEKjZY693HIfJnAr"], "labels": [], "masters": [], "uuids": ["6b6f1bbd-1d69-4b33-9fd0-17ab2a1bd58a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:49.219721", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:52.054 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-FrVXG2-WhwF-lcFW-IiMe-BLzE-1mV9-eYsHRd', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500MW400BGN', 'nvme-nvme.8086-504846543632303530304d5734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.071324", "end": "2020-11-20 12:00:52.006542", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-FrVXG2-WhwF-lcFW-IiMe-BLzE-1mV9-eYsHRd", "nvme-INTEL_SSDPEDMD400G4_PHFT620500MW400BGN", "nvme-nvme.8086-504846543632303530304d5734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:50.935218", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-20T12:00:52.056 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:52.381 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRFPAXcbG29wAISIMfPlGCDsd6nwqUf7tg'], 'uuids': ['32c79644-208f-4f39-9ff3-e82133d5a59f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.370338", "end": "2020-11-20 12:00:52.334500", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRFPAXcbG29wAISIMfPlGCDsd6nwqUf7tg"], "labels": [], "masters": [], "uuids": ["32c79644-208f-4f39-9ff3-e82133d5a59f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:50.964162", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:53.494 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwDy97t4PHsw9KkmhQyGKY9ec4K5WyAZGd'], 'uuids': ['c7233954-7c8e-4b89-bf0e-cdcbd6035268'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.098333", "end": "2020-11-20 12:00:53.434233", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwDy97t4PHsw9KkmhQyGKY9ec4K5WyAZGd"], "labels": [], "masters": [], "uuids": ["c7233954-7c8e-4b89-bf0e-cdcbd6035268"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:52.335900", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:53.812 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-wufZ05-akSE-9AIT-Q2ZS-26cv-GOMe-yQoKRy', 'nvme-INTEL_SSDPED1D480GA_PHMB751300EW480DGN', 'nvme-nvme.8086-50484d42373531333030455734383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.085919", "end": "2020-11-20 12:00:53.760048", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-wufZ05-akSE-9AIT-Q2ZS-26cv-GOMe-yQoKRy", "nvme-INTEL_SSDPED1D480GA_PHMB751300EW480DGN", "nvme-nvme.8086-50484d42373531333030455734383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:52.674129", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2020-11-20T12:00:53.814 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:54.888 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXw6cmSzR0zosv4xL9N60BMJLn7Oo8ngtmh'], 'uuids': ['2719a773-35c7-4835-bf22-fb725967c5c5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.077832", "end": "2020-11-20 12:00:54.840013", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXw6cmSzR0zosv4xL9N60BMJLn7Oo8ngtmh"], "labels": [], "masters": [], "uuids": ["2719a773-35c7-4835-bf22-fb725967c5c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:53.762181", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:55.228 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRftB4Tqp5D3gzJ23PTjDXxXV9583ZPKLz'], 'uuids': ['358193a4-7f3e-4adb-af9e-0e73b6812952'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.078951", "end": "2020-11-20 12:00:55.187023", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRftB4Tqp5D3gzJ23PTjDXxXV9583ZPKLz"], "labels": [], "masters": [], "uuids": ["358193a4-7f3e-4adb-af9e-0e73b6812952"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:54.108072", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:56.312 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXw7tRPDL2Wo4ZdmCVgUkEXTMg3ZPh1hptf'], 'uuids': ['d5123bf7-71ea-40cc-ac80-bd3d74498fe0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.086489", "end": "2020-11-20 12:00:56.265843", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXw7tRPDL2Wo4ZdmCVgUkEXTMg3ZPh1hptf"], "labels": [], "masters": [], "uuids": ["d5123bf7-71ea-40cc-ac80-bd3d74498fe0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:55.179354", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:56.323 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFQE', 'wwn-0x5000c5009292232d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFQE', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009292232d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFQE-part1', 'wwn-0x5000c5009292232d-part1'], 'uuids': ['9bdf28e7-5b1a-4776-884c-191bd8ad39f5'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFQE", "wwn-0x5000c5009292232d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFQE-part1", "wwn-0x5000c5009292232d-part1"], "labels": [], "masters": [], "uuids": ["9bdf28e7-5b1a-4776-884c-191bd8ad39f5"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFQE", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292232d"}}, 2020-11-20T12:00:56.325 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-11-20T12:00:56.649 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRt7zxsdzdSzrlCqMONaJ2fY0coDHBzxes'], 'uuids': ['a8b0739a-bef3-4653-a475-32bd7719420e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.067195", "end": "2020-11-20 12:00:56.595662", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRt7zxsdzdSzrlCqMONaJ2fY0coDHBzxes"], "labels": [], "masters": [], "uuids": ["a8b0739a-bef3-4653-a475-32bd7719420e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:55.528467", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:57.713 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwZBhSvgWvn2NT32c85VzIoLYmIIaASjWn'], 'uuids': ['146f6d2e-9cfb-484a-897a-7c1ce3e8e470'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.075147", "end": "2020-11-20 12:00:57.674202", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwZBhSvgWvn2NT32c85VzIoLYmIIaASjWn"], "labels": [], "masters": [], "uuids": ["146f6d2e-9cfb-484a-897a-7c1ce3e8e470"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:56.599055", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:57.788 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:57 +0000 (0:00:07.173) 0:02:22.391 ******* 2020-11-20T12:00:58.056 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgR00nlIvSHLDDk3TB1J0PMem9yWLPZoIks'], 'uuids': ['d6971ea8-b8e5-406f-bb0c-313d7f0aa48b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.064424", "end": "2020-11-20 12:00:58.003749", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgR00nlIvSHLDDk3TB1J0PMem9yWLPZoIks"], "labels": [], "masters": [], "uuids": ["d6971ea8-b8e5-406f-bb0c-313d7f0aa48b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:56.939325", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:58.067 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ1L', 'wwn-0x5000c500871a93fb'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HQ1L', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a93fb', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ1L-part1', 'wwn-0x5000c500871a93fb-part1'], 'uuids': ['9bdf28e7-5b1a-4776-884c-191bd8ad39f5'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ1L", "wwn-0x5000c500871a93fb"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ1L-part1", "wwn-0x5000c500871a93fb-part1"], "labels": [], "masters": [], "uuids": ["9bdf28e7-5b1a-4776-884c-191bd8ad39f5"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQ1L", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a93fb"}}, 2020-11-20T12:00:58.069 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-11-20T12:00:59.483 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRbxdsSSJM8fZrV8cw64FzLbxuArUBzoXk'], 'uuids': ['0d95424f-19e4-42b5-884c-4443652de14a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.064554", "end": "2020-11-20 12:00:59.430705", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRbxdsSSJM8fZrV8cw64FzLbxuArUBzoXk"], "labels": [], "masters": [], "uuids": ["0d95424f-19e4-42b5-884c-4443652de14a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:58.366151", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:00:59.522 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:59 +0000 (0:00:01.734) 0:02:24.125 ******* 2020-11-20T12:00:59.958 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T12:01:01.168 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwG9l5QNtFR0YRCHz8CEKjZY693HIfJnAr'], 'uuids': ['6b6f1bbd-1d69-4b33-9fd0-17ab2a1bd58a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:01.778491", "end": "2020-11-20 12:00:59.900696", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwG9l5QNtFR0YRCHz8CEKjZY693HIfJnAr"], "labels": [], "masters": [], "uuids": ["6b6f1bbd-1d69-4b33-9fd0-17ab2a1bd58a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:58.122205", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 1.76712 s, 65.3 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 1.76712 s, 65.3 MB/s"], "stdout": "", "stdout_lines": []} changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-FrVXG2-WhwF-lcFW-IiMe-BLzE-1mV9-eYsHRd', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500MW400BGN', 'nvme-nvme.8086-504846543632303530304d5734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None 2020-11-20T12:01:01.170 INFO:teuthology.task.ansible.out:, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.421121", "end": "2020-11-20 12:01:00.675860", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-FrVXG2-WhwF-lcFW-IiMe-BLzE-1mV9-eYsHRd", "nvme-INTEL_SSDPEDMD400G4_PHFT620500MW400BGN", "nvme-nvme.8086-504846543632303530304d5734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:00.254739", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.410069 s, 281 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.410069 s, 281 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:01.885 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRFPAXcbG29wAISIMfPlGCDsd6nwqUf7tg'], 'uuids': ['32c79644-208f-4f39-9ff3-e82133d5a59f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:01.979999", "end": "2020-11-20 12:01:01.838506", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRFPAXcbG29wAISIMfPlGCDsd6nwqUf7tg"], "labels": [], "masters": [], "uuids": ["32c79644-208f-4f39-9ff3-e82133d5a59f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:00:59.858507", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 1.96824 s, 58.6 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 1.96824 s, 58.6 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:02.675 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-wufZ05-akSE-9AIT-Q2ZS-26cv-GOMe-yQoKRy', 'nvme-INTEL_SSDPED1D480GA_PHMB751300EW480DGN', 'nvme-nvme.8086-50484d42373531333030455734383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.438092", "end": "2020-11-20 12:01:02.625161", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-wufZ05-akSE-9AIT-Q2ZS-26cv-GOMe-yQoKRy", "nvme-INTEL_SSDPED1D480GA_PHMB751300EW480DGN", "nvme-nvme.8086-50484d42373531333030455734383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:02.187069", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.427174 s, 270 MB/s", "stderr_lines": ["1 2020-11-20T12:01:02.677 INFO:teuthology.task.ansible.out:10+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.427174 s, 270 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:02.858 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwDy97t4PHsw9KkmhQyGKY9ec4K5WyAZGd'], 'uuids': ['c7233954-7c8e-4b89-bf0e-cdcbd6035268'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:01.810143", "end": "2020-11-20 12:01:02.813982", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwDy97t4PHsw9KkmhQyGKY9ec4K5WyAZGd"], "labels": [], "masters": [], "uuids": ["c7233954-7c8e-4b89-bf0e-cdcbd6035268"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:01.003839", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 1.80046 s, 64.1 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 1.80046 s, 64.1 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:04.973 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXw6cmSzR0zosv4xL9N60BMJLn7Oo8ngtmh'], 'uuids': ['2719a773-35c7-4835-bf22-fb725967c5c5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:01.784688", "end": "2020-11-20 12:01:04.914654", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXw6cmSzR0zosv4xL9N60BMJLn7Oo8ngtmh"], "labels": [], "masters": [], "uuids": ["2719a773-35c7-4835-bf22-fb725967c5c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:03.129966", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 1.77387 s, 65.0 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 1.77387 s, 65.0 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:05.482 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRftB4Tqp5D3gzJ23PTjDXxXV9583ZPKLz'], 'uuids': ['358193a4-7f3e-4adb-af9e-0e73b6812952'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:02.499034", "end": "2020-11-20 12:01:05.434483", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRftB4Tqp5D3gzJ23PTjDXxXV9583ZPKLz"], "labels": [], "masters": [], "uuids": ["358193a4-7f3e-4adb-af9e-0e73b6812952"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:02.935449", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 2.48781 s, 46.4 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 2.48781 s, 46.4 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:07.389 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXw7tRPDL2Wo4ZdmCVgUkEXTMg3ZPh1hptf'], 'uuids': ['d5123bf7-71ea-40cc-ac80-bd3d74498fe0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:02.106425", "end": "2020-11-20 12:01:07.341215", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXw7tRPDL2Wo4ZdmCVgUkEXTMg3ZPh1hptf"], "labels": [], "masters": [], "uuids": ["d5123bf7-71ea-40cc-ac80-bd3d74498fe0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:05.234790", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 2.09521 s, 55.1 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 2.09521 s, 55.1 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:07.402 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFQE', 'wwn-0x5000c5009292232d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFQE', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009292232d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFQE-part1', 'wwn-0x5000c5009292232d-part1'], 'uuids': ['9bdf28e7-5b1a-4776-884c-191bd8ad39f5'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFQE", "wwn-0x5000c5009292232d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFQE-part1", "wwn-0x5000c5009292232d-part1"], "labels": [], "masters": [], "uuids": ["9bdf28e7-5b1a-4776-884c-191bd8ad39f5"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFQE", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292232d"}}, 2020-11-20T12:01:07.405 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.806 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRt7zxsdzdSzrlCqMONaJ2fY0coDHBzxes'], 'uuids': ['a8b0739a-bef3-4653-a475-32bd7719420e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:01.979452", "end": "2020-11-20 12:01:07.759706", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRt7zxsdzdSzrlCqMONaJ2fY0coDHBzxes"], "labels": [], "masters": [], "uuids": ["a8b0739a-bef3-4653-a475-32bd7719420e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:05.780254", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 1.96781 s, 58.6 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 1.96781 s, 58.6 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:09.530 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwZBhSvgWvn2NT32c85VzIoLYmIIaASjWn'], 'uuids': ['146f6d2e-9cfb-484a-897a-7c1ce3e8e470'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:01.793107", "end": "2020-11-20 12:01:09.489675", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-1fsLZgiunQgczd6BI8Cm36wIZ0jT4pXwZBhSvgWvn2NT32c85VzIoLYmIIaASjWn"], "labels": [], "masters": [], "uuids": ["146f6d2e-9cfb-484a-897a-7c1ce3e8e470"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:07.696568", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 1.78286 s, 64.7 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 1.78286 s, 64.7 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:09.585 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:09 +0000 (0:00:10.063) 0:02:34.189 ******* 2020-11-20T12:01:10.015 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi187.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.101203", "end": "2020-11-20 12:01:09.974199", "rc": 0, "start": "2020-11-20 12:01:09.872996", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:10.055 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:10 +0000 (0:00:00.469) 0:02:34.658 ******* 2020-11-20T12:01:10.117 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T12:01:10.119 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgR00nlIvSHLDDk3TB1J0PMem9yWLPZoIks'], 'uuids': ['d6971ea8-b8e5-406f-bb0c-313d7f0aa48b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:01.971453", "end": "2020-11-20 12:01:10.070201", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgR00nlIvSHLDDk3TB1J0PMem9yWLPZoIks"], "labels": [], "masters": [], "uuids": ["d6971ea8-b8e5-406f-bb0c-313d7f0aa48b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:08.098748", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 1.96003 s, 58.8 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 1.96003 s, 58.8 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:10.130 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ1L', 'wwn-0x5000c500871a93fb'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HQ1L', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a93fb', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ1L-part1', 'wwn-0x5000c500871a93fb-part1'], 'uuids': ['9bdf28e7-5b1a-4776-884c-191bd8ad39f5'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ1L", "wwn-0x5000c500871a93fb"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ1L-part1", "wwn-0x5000c500871a93fb-part1"], "labels": [], "masters": [], "uuids": ["9bdf28e7-5b1a-4776-884c-191bd8ad39f5"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQ1L", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a93fb"}}, 2020-11-20T12:01:10.131 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-11-20T12:01:10.468 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-20T12:01:10.470 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.072746", "end": "2020-11-20 12:01:10.437773", "rc": 0, "start": "2020-11-20 12:01:10.365027", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:10.502 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:10 +0000 (0:00:00.446) 0:02:35.105 ******* 2020-11-20T12:01:10.528 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-20T12:01:10.565 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:10 +0000 (0:00:00.062) 0:02:35.168 ******* 2020-11-20T12:01:10.591 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-20T12:01:10.593 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:10.631 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:10 +0000 (0:00:00.066) 0:02:35.235 ******* 2020-11-20T12:01:10.659 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-20T12:01:10.661 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:10.693 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:10 +0000 (0:00:00.061) 0:02:35.296 ******* 2020-11-20T12:01:10.720 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T12:01:10.721 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:10.770 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:10 +0000 (0:00:00.077) 0:02:35.373 ******* 2020-11-20T12:01:10.797 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-20T12:01:10.799 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:10.831 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:10 +0000 (0:00:00.060) 0:02:35.434 ******* 2020-11-20T12:01:11.187 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-20T12:01:11.189 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T12:01:11.230 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:11 +0000 (0:00:00.395) 0:02:35.829 ******* 2020-11-20T12:01:11.286 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T12:01:11.288 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 927554727936, 'block_size': 4096, 'block_total': 240095108, 'block_available': 226453791, 'block_used': 13641317, 'inode_total': 61054976, 'inode_available': 60988135, 'inode_used': 66841, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453791, "block_size": 4096, "block_total": 240095108, "block_used": 13641317, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60988135, "inode_total": 61054976, "inode_used": 66841, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 927554727936, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.322 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:11 +0000 (0:00:00.095) 0:02:35.925 ******* 2020-11-20T12:01:11.364 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-20T12:01:11.366 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.398 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:11 +0000 (0:00:00.076) 0:02:36.001 ******* 2020-11-20T12:01:11.425 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:11.427 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.460 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:11 +0000 (0:00:00.061) 0:02:36.063 ******* 2020-11-20T12:01:11.497 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:11.499 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.535 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:11 +0000 (0:00:00.075) 0:02:36.138 ******* 2020-11-20T12:01:11.563 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.602 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:11 +0000 (0:00:00.066) 0:02:36.205 ******* 2020-11-20T12:01:11.631 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-20T12:01:11.632 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.664 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:11 +0000 (0:00:00.062) 0:02:36.267 ******* 2020-11-20T12:01:11.691 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-20T12:01:11.692 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.725 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:11 +0000 (0:00:00.060) 0:02:36.328 ******* 2020-11-20T12:01:12.438 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T12:01:12.439 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRbxdsSSJM8fZrV8cw64FzLbxuArUBzoXk'], 'uuids': ['0d95424f-19e4-42b5-884c-4443652de14a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:01.974083", "end": "2020-11-20 12:01:12.394547", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ru2v5mh5CykAhBJgu24M0PhxdBPBjhgRbxdsSSJM8fZrV8cw64FzLbxuArUBzoXk"], "labels": [], "masters": [], "uuids": ["0d95424f-19e4-42b5-884c-4443652de14a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:10.420464", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 1.9631 s, 58.8 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 1.9631 s, 58.8 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:12.479 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:12 +0000 (0:00:00.753) 0:02:37.082 ******* 2020-11-20T12:01:12.721 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-20T12:01:12.723 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-20T12:01:12.757 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:12 +0000 (0:00:00.278) 0:02:37.360 ******* 2020-11-20T12:01:12.932 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-20T12:01:12.934 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.096603", "end": "2020-11-20 12:01:12.885240", "rc": 0, "start": "2020-11-20 12:01:12.788637", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:12.967 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:12 +0000 (0:00:00.209) 0:02:37.570 ******* 2020-11-20T12:01:13.393 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.078774", "end": "2020-11-20 12:01:13.361005", "rc": 0, "start": "2020-11-20 12:01:13.282231", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:13.428 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:13 +0000 (0:00:00.460) 0:02:38.031 ******* 2020-11-20T12:01:13.455 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-20T12:01:13.491 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:13 +0000 (0:00:00.063) 0:02:38.094 ******* 2020-11-20T12:01:13.518 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-20T12:01:13.520 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:13.554 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:13 +0000 (0:00:00.062) 0:02:38.157 ******* 2020-11-20T12:01:13.581 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:13.614 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:13 +0000 (0:00:00.060) 0:02:38.218 ******* 2020-11-20T12:01:13.642 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T12:01:13.644 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:13.678 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:13 +0000 (0:00:00.064) 0:02:38.282 ******* 2020-11-20T12:01:13.706 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-20T12:01:13.707 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:13.742 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:13 +0000 (0:00:00.062) 0:02:38.344 ******* 2020-11-20T12:01:14.097 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-20T12:01:14.099 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T12:01:14.140 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:14 +0000 (0:00:00.398) 0:02:38.743 ******* 2020-11-20T12:01:14.178 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi005.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 927554650112, 'block_size': 4096, 'block_total': 240095108, 'block_available': 226453772, 'block_used': 13641336, 'inode_total': 61054976, 'inode_available': 60988127, 'inode_used': 66849, 'uuid': '9bdf28e7-5b1a-4776-884c-191bd8ad39f5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453772, "block_size": 4096, "block_total": 240095108, "block_used": 13641336, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60988127, "inode_total": 61054976, "inode_used": 66849, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 927554650112, "size_total": 983429562368, "uuid": "9bdf28e7-5b1a-4776-884c-191bd8ad39f5"}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.213 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:14 +0000 (0:00:00.073) 0:02:38.816 ******* 2020-11-20T12:01:14.230 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:14.241 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.275 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:14 +0000 (0:00:00.062) 0:02:38.878 ******* 2020-11-20T12:01:14.302 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:14.304 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.336 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:14 +0000 (0:00:00.060) 0:02:38.939 ******* 2020-11-20T12:01:14.363 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.397 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:14 +0000 (0:00:00.060) 0:02:39.000 ******* 2020-11-20T12:01:14.426 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-20T12:01:14.428 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.459 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:14 +0000 (0:00:00.062) 0:02:39.062 ******* 2020-11-20T12:01:14.486 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-20T12:01:14.487 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.520 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:14 +0000 (0:00:00.060) 0:02:39.123 ******* 2020-11-20T12:01:14.546 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-20T12:01:14.548 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.580 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:14 +0000 (0:00:00.060) 0:02:39.183 ******* 2020-11-20T12:01:14.796 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T12:01:14.798 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:15.294 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-20T12:01:15.295 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-20T12:01:15.329 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:15 +0000 (0:00:00.748) 0:02:39.932 ******* 2020-11-20T12:01:15.398 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T12:01:15.400 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:15.984 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:15.984 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:16.043 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:16 +0000 (0:00:00.714) 0:02:40.647 ******* 2020-11-20T12:01:16.409 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T12:01:16.411 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-20T12:01:16.443 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:16 +0000 (0:00:00.399) 0:02:41.046 ******* 2020-11-20T12:01:16.610 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:16.801 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-20T12:01:16.803 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T12:01:17.128 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T12:01:17.213 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:17.456 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-20T12:01:17.459 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T12:01:17.784 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T12:01:17.795 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:17.833 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:17 +0000 (0:00:01.389) 0:02:42.436 ******* 2020-11-20T12:01:17.875 INFO:teuthology.task.ansible.out: 2020-11-20T12:01:17.878 INFO:teuthology.task.ansible.out:TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:18.197 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-20T12:01:18.199 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:01:18.231 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:18 +0000 (0:00:00.398) 0:02:42.835 ******* 2020-11-20T12:01:18.270 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-20T12:01:18.306 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:18 +0000 (0:00:00.074) 0:02:42.909 ******* 2020-11-20T12:01:18.514 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T12:01:18.516 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:18.561 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:18 +0000 (0:00:00.255) 0:02:43.164 ******* 2020-11-20T12:01:18.924 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-20T12:01:18.958 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:18 +0000 (0:00:00.397) 0:02:43.561 ******* 2020-11-20T12:01:19.033 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi187.front.sepia.ceph.com] => {"changed": true} 2020-11-20T12:01:19.069 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:19 +0000 (0:00:00.111) 0:02:43.672 ******* 2020-11-20T12:01:19.343 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-20T12:01:19.345 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T12:01:19.679 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T12:01:20.015 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T12:01:20.374 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T12:01:20.376 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.412 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:01.342) 0:02:45.015 ******* 2020-11-20T12:01:20.492 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-20T12:01:20.493 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-20T12:01:20.526 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.114) 0:02:45.129 ******* 2020-11-20T12:01:20.553 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-20T12:01:20.555 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.587 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.060) 0:02:45.190 ******* 2020-11-20T12:01:20.614 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-20T12:01:20.615 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.650 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.062) 0:02:45.253 ******* 2020-11-20T12:01:20.678 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-20T12:01:20.679 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.712 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.062) 0:02:45.315 ******* 2020-11-20T12:01:20.738 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-20T12:01:20.740 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.774 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.061) 0:02:45.377 ******* 2020-11-20T12:01:20.779 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-20T12:01:20.781 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:01:20.815 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.041) 0:02:45.418 ******* 2020-11-20T12:01:20.821 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-20T12:01:20.823 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.858 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.042) 0:02:45.461 ******* 2020-11-20T12:01:20.864 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-11-20T12:01:20.865 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-20T12:01:20.902 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.043) 0:02:45.505 ******* 2020-11-20T12:01:20.908 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-20T12:01:20.910 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.944 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.041) 0:02:45.547 ******* 2020-11-20T12:01:20.970 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-20T12:01:20.972 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.004 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.060) 0:02:45.608 ******* 2020-11-20T12:01:21.031 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-20T12:01:21.033 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.067 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.062) 0:02:45.670 ******* 2020-11-20T12:01:21.095 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.142 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.074) 0:02:45.745 ******* 2020-11-20T12:01:21.172 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.207 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.064) 0:02:45.810 ******* 2020-11-20T12:01:21.238 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.278 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.071) 0:02:45.881 ******* 2020-11-20T12:01:21.316 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-20T12:01:21.318 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true} 2020-11-20T12:01:21.351 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.073) 0:02:45.955 ******* 2020-11-20T12:01:22.199 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-20T12:01:22.201 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-11-20T12:01:22.235 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.883) 0:02:46.838 ******* 2020-11-20T12:01:22.540 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-20T12:01:22.541 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-20T12:01:22.575 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.339) 0:02:47.178 ******* 2020-11-20T12:01:22.602 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-20T12:01:22.603 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:22.636 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.060) 0:02:47.239 ******* 2020-11-20T12:01:22.663 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:22.697 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.061) 0:02:47.300 ******* 2020-11-20T12:01:22.724 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:22.760 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.063) 0:02:47.363 ******* 2020-11-20T12:01:22.767 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-20T12:01:22.768 INFO:teuthology.task.ansible.out:fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service chronyd: host"} ...ignoring 2020-11-20T12:01:22.803 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.043) 0:02:47.407 ******* 2020-11-20T12:01:22.809 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-20T12:01:22.811 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:22.849 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.044) 0:02:47.452 ******* 2020-11-20T12:01:22.855 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:22.857 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-20T12:01:22.898 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.048) 0:02:47.500 ******* 2020-11-20T12:01:22.904 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-20T12:01:22.906 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:22.940 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.043) 0:02:47.543 ******* 2020-11-20T12:01:22.947 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:22.949 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2020-11-20T12:01:22.986 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:22 +0000 (0:00:00.045) 0:02:47.589 ******* 2020-11-20T12:01:22.993 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-20T12:01:22.995 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.031 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.044) 0:02:47.633 ******* 2020-11-20T12:01:23.037 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.071 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.041) 0:02:47.674 ******* 2020-11-20T12:01:23.078 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-20T12:01:23.081 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.123 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.052) 0:02:47.727 ******* 2020-11-20T12:01:23.133 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-20T12:01:23.136 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.181 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.056) 0:02:47.784 ******* 2020-11-20T12:01:23.187 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-20T12:01:23.189 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.225 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.044) 0:02:47.828 ******* 2020-11-20T12:01:23.232 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.269 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.043) 0:02:47.872 ******* 2020-11-20T12:01:23.275 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-20T12:01:23.277 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.313 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.044) 0:02:47.916 ******* 2020-11-20T12:01:23.344 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-20T12:01:23.346 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.378 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.065) 0:02:47.981 ******* 2020-11-20T12:01:23.405 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-20T12:01:23.407 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:23.439 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.061) 0:02:48.043 ******* 2020-11-20T12:01:23.624 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-20T12:01:23.626 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-11-20T12:01:23.660 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:00.220) 0:02:48.263 ******* 2020-11-20T12:01:24.450 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-11-20T12:01:24.493 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:24 +0000 (0:00:00.832) 0:02:49.095 ******* 2020-11-20T12:01:24.637 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-20T12:01:24.639 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-11-20T12:01:24.679 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:24 +0000 (0:00:00.186) 0:02:49.282 ******* 2020-11-20T12:01:25.016 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-20T12:01:25.018 INFO:teuthology.task.ansible.out:fatal: [smithi005.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service chronyd: host"} ...ignoring 2020-11-20T12:01:25.054 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.375) 0:02:49.657 ******* 2020-11-20T12:01:25.082 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-20T12:01:25.117 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.062) 0:02:49.720 ******* 2020-11-20T12:01:25.143 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:25.145 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2020-11-20T12:01:25.179 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.062) 0:02:49.782 ******* 2020-11-20T12:01:25.206 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-20T12:01:25.208 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:25.240 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.060) 0:02:49.843 ******* 2020-11-20T12:01:25.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-20T12:01:25.268 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:25.300 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.060) 0:02:49.904 ******* 2020-11-20T12:01:25.327 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:25.364 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.063) 0:02:49.967 ******* 2020-11-20T12:01:25.718 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-20T12:01:25.719 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-11-20T12:01:25.751 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.387) 0:02:50.355 ******* 2020-11-20T12:01:26.520 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-20T12:01:26.521 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-20T12:01:26.554 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:26 +0000 (0:00:00.802) 0:02:51.157 ******* 2020-11-20T12:01:26.581 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-20T12:01:26.583 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:26.615 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:26 +0000 (0:00:00.061) 0:02:51.219 ******* 2020-11-20T12:01:26.642 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-20T12:01:26.644 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-20T12:01:26.676 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:26 +0000 (0:00:00.060) 0:02:51.280 ******* 2020-11-20T12:01:26.737 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-20T12:01:26.739 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-11-20T12:01:26.772 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:26 +0000 (0:00:00.095) 0:02:51.375 ******* 2020-11-20T12:01:27.008 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-20T12:01:27.010 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-11-20T12:01:27.043 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.271) 0:02:51.646 ******* 2020-11-20T12:01:27.070 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-20T12:01:27.072 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:27.105 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.061) 0:02:51.708 ******* 2020-11-20T12:01:27.132 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-20T12:01:27.134 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-20T12:01:27.167 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.061) 0:02:51.770 ******* 2020-11-20T12:01:27.196 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-20T12:01:27.198 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:27.230 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.063) 0:02:51.834 ******* 2020-11-20T12:01:27.564 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-20T12:01:27.599 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.368) 0:02:52.202 ******* 2020-11-20T12:01:27.603 INFO:teuthology.task.ansible.out: 2020-11-20T12:01:27.943 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-20T12:01:27.944 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-20T12:01:27.979 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.379) 0:02:52.582 ******* 2020-11-20T12:01:28.005 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-20T12:01:28.006 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.039 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.060) 0:02:52.642 ******* 2020-11-20T12:01:28.100 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi187.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.123 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.160 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.121) 0:02:52.764 ******* 2020-11-20T12:01:28.198 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-20T12:01:28.200 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.210 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.221 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.232 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.233 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.268 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.107) 0:02:52.871 ******* 2020-11-20T12:01:28.295 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:28.297 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.336 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.067) 0:02:52.939 ******* 2020-11-20T12:01:28.368 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.401 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.064) 0:02:53.004 ******* 2020-11-20T12:01:28.418 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-20T12:01:28.420 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-20T12:01:28.456 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.055) 0:02:53.059 ******* 2020-11-20T12:01:28.463 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-20T12:01:28.466 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.514 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.057) 0:02:53.117 ******* 2020-11-20T12:01:28.521 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-20T12:01:28.523 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.557 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.043) 0:02:53.160 ******* 2020-11-20T12:01:28.564 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-20T12:01:28.566 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.601 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.043) 0:02:53.204 ******* 2020-11-20T12:01:28.607 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-20T12:01:28.609 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-11-20T12:01:28.657 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.056) 0:02:53.260 ******* 2020-11-20T12:01:28.666 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-20T12:01:28.669 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.718 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.061) 0:02:53.322 ******* 2020-11-20T12:01:28.746 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-20T12:01:28.748 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.802 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.083) 0:02:53.405 ******* 2020-11-20T12:01:28.832 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.868 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.066) 0:02:53.472 ******* 2020-11-20T12:01:28.896 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.930 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.061) 0:02:53.533 ******* 2020-11-20T12:01:29.020 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-20T12:01:29.023 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-11-20T12:01:29.056 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.126) 0:02:53.659 ******* 2020-11-20T12:01:29.084 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-20T12:01:29.086 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:29.118 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.061) 0:02:53.721 ******* 2020-11-20T12:01:29.123 INFO:teuthology.task.ansible.out: 2020-11-20T12:01:29.179 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-20T12:01:29.181 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-20T12:01:29.219 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.100) 0:02:53.822 ******* 2020-11-20T12:01:29.250 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:29.287 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.067) 0:02:53.890 ******* 2020-11-20T12:01:29.304 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-20T12:01:29.306 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.017705", "end": "2020-11-20 12:01:29.257989", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-20 12:01:29.240284", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:29.644 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.012410", "end": "2020-11-20 12:01:29.604280", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-20 12:01:29.591870", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:29.646 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-20T12:01:29.681 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.394) 0:02:54.285 ******* 2020-11-20T12:01:29.971 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-20T12:01:29.973 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.012820", "end": "2020-11-20 12:01:29.929046", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 12:01:29.916226", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:30.037 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-20T12:01:30.039 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-20T12:01:30.074 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.392) 0:02:54.677 ******* 2020-11-20T12:01:30.102 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-20T12:01:30.103 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.137 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.062) 0:02:54.740 ******* 2020-11-20T12:01:30.176 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-20T12:01:30.178 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.189 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.200 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.212 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.247 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.110) 0:02:54.851 ******* 2020-11-20T12:01:30.284 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-20T12:01:30.286 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.296 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.298 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.308 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.344 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.095) 0:02:54.946 ******* 2020-11-20T12:01:30.350 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi187.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.032661", "end": "2020-11-20 12:01:30.275038", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-20 12:01:30.242377", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:30.388 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.044) 0:02:54.991 ******* 2020-11-20T12:01:30.394 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:30.396 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.433 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.045) 0:02:55.036 ******* 2020-11-20T12:01:30.441 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-20T12:01:30.492 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.059) 0:02:55.096 ******* 2020-11-20T12:01:30.501 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-20T12:01:30.502 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.552 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.059) 0:02:55.155 ******* 2020-11-20T12:01:30.581 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.634 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.082) 0:02:55.237 ******* 2020-11-20T12:01:30.666 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-20T12:01:30.667 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.700 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.066) 0:02:55.303 ******* 2020-11-20T12:01:30.729 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.767 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.067) 0:02:55.370 ******* 2020-11-20T12:01:30.794 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-20T12:01:30.796 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.831 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.064) 0:02:55.434 ******* 2020-11-20T12:01:30.847 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-20T12:01:30.849 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-20T12:01:30.884 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.935 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.103) 0:02:55.538 ******* 2020-11-20T12:01:30.974 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-20T12:01:30.976 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:31.009 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:31 +0000 (0:00:00.074) 0:02:55.612 ******* 2020-11-20T12:01:31.407 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-20T12:01:31.409 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.019520", "end": "2020-11-20 12:01:31.345731", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-20 12:01:31.326211", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:31.763 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.012691", "end": "2020-11-20 12:01:31.713846", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-20 12:01:31.701155", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:32.110 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.011592", "end": "2020-11-20 12:01:32.058427", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 12:01:32.046835", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:32.469 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.038873", "end": "2020-11-20 12:01:32.422312", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-20 12:01:32.383439", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:32.505 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:32 +0000 (0:00:01.495) 0:02:57.108 ******* 2020-11-20T12:01:32.532 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-20T12:01:32.569 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:32 +0000 (0:00:00.064) 0:02:57.173 ******* 2020-11-20T12:01:32.921 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-20T12:01:32.923 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-20T12:01:33.015 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-20T12:01:33.019 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-20T12:01:33.023 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-11-20T12:01:33.025 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi005.front.sepia.ceph.com : ok=115 changed=34 unreachable=0 failed=0 skipped=181 rescued=0 ignored=6 smithi187.front.sepia.ceph.com : ok=115 changed=34 unreachable=0 failed=0 skipped=181 rescued=0 ignored=6 Friday 20 November 2020 12:01:33 +0000 (0:00:00.453) 0:02:57.627 ******* =============================================================================== common : nrpe - Remove SELinux policy package -------------------------- 16.70s common : nrpe - Load SELinux policy package ---------------------------- 15.94s testnode : Blow away lingering OSD data and FSIDs ---------------------- 10.06s testnode : Zap all non-root disks --------------------------------------- 7.17s common : Make sure smartmontools is installed --------------------------- 4.87s common : Install Kerberos Packages (RedHat) ----------------------------- 4.34s Gathering Facts --------------------------------------------------------- 2.71s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 2.47s testnode : Install packages --------------------------------------------- 2.41s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 2.40s common : Configure epel repos in /etc/yum.repos.d/ ---------------------- 1.82s common : Configure local epel mirrorlists ------------------------------- 1.70s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.66s testnode : Configure local mirrorlists ---------------------------------- 1.63s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 1.57s testnode : Rebuild rpmdb ------------------------------------------------ 1.52s testnode : List any leftover Ceph artifacts from previous jobs ---------- 1.50s testnode : Ensure ceph packages are not present. ------------------------ 1.47s Gathering Facts 2020-11-20T12:01:33.026 INFO:teuthology.task.ansible.out: --------------------------------------------------------- 1.42s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.41s 2020-11-20T12:01:33.375 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi187.front.sepia.ceph.com'), Remote(name='ubuntu@smithi005.front.sepia.ceph.com')] 2020-11-20T12:01:33.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T12:01:33.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T12:01:33.508 INFO:teuthology.orchestra.run.smithi187:> true 2020-11-20T12:01:35.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T12:01:35.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T12:01:35.156 INFO:teuthology.orchestra.run.smithi005:> true 2020-11-20T12:01:35.322 INFO:teuthology.run_tasks:Running task clock... 2020-11-20T12:01:35.332 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-20T12:01:35.332 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-20T12:01:35.333 INFO:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-20T12:01:35.450 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-11-20T12:01:35.508 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-11-20T12:01:35.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2020-11-20T12:01:35.661 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2020-11-20T12:01:35.718 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-11-20T12:01:35.775 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-11-20T12:01:35.875 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2020-11-20T12:01:35.882 INFO:teuthology.orchestra.run.smithi187.stdout:210 Number of sources = 4 2020-11-20T12:01:35.882 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-11-20T12:01:35.882 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2020-11-20T12:01:35.883 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T12:01:35.883 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T12:01:35.883 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T12:01:35.883 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T12:01:35.884 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-20T12:01:35.884 INFO:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-20T12:01:36.007 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-11-20T12:01:36.071 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-11-20T12:01:36.184 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2020-11-20T12:01:36.234 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2020-11-20T12:01:36.295 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-11-20T12:01:36.359 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-11-20T12:01:36.478 INFO:teuthology.orchestra.run.smithi005.stderr:bash: ntpq: command not found 2020-11-20T12:01:36.484 INFO:teuthology.orchestra.run.smithi005.stdout:210 Number of sources = 4 2020-11-20T12:01:36.484 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-11-20T12:01:36.484 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2020-11-20T12:01:36.485 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T12:01:36.485 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T12:01:36.485 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T12:01:36.485 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T12:01:36.486 INFO:teuthology.run_tasks:Running task install... 2020-11-20T12:01:36.495 DEBUG:teuthology.task.install:project ceph 2020-11-20T12:01:36.496 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}} 2020-11-20T12:01:36.496 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'} 2020-11-20T12:01:36.496 INFO:teuthology.task.install:Using flavor: basic 2020-11-20T12:01:36.503 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-20T12:01:36.503 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2020-11-20T12:01:36.504 INFO:teuthology.orchestra.run.smithi187:> sudo lsb_release -is 2020-11-20T12:01:36.650 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS 2020-11-20T12:01:36.653 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T12:01:36.654 INFO:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2020-11-20T12:01:36.655 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'tag': None, 'wait_for_package': False} 2020-11-20T12:01:36.655 INFO:teuthology.orchestra.run.smithi187:> sudo lsb_release -is 2020-11-20T12:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS 2020-11-20T12:01:36.788 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T12:01:36.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:01:36.809 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS 2020-11-20T12:01:36.810 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T12:01:36.811 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'tag': None, 'wait_for_package': False} 2020-11-20T12:01:36.811 INFO:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2020-11-20T12:01:36.954 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/ 2020-11-20T12:01:36.954 INFO:teuthology.task.install.rpm:Package version is 16.0.0-7397.gb86e691b 2020-11-20T12:01:36.957 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS 2020-11-20T12:01:36.958 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T12:01:36.958 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:01:37.081 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-11-20T12:01:37.082 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:01:37.082 INFO:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-20T12:01:37.124 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/ 2020-11-20T12:01:37.124 INFO:teuthology.task.install.rpm:Package version is 16.0.0-7397.gb86e691b 2020-11-20T12:01:37.140 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-20T12:01:37.187 INFO:teuthology.orchestra.run.smithi187.stderr:1+1 records in 2020-11-20T12:01:37.188 INFO:teuthology.orchestra.run.smithi187.stderr:1+1 records out 2020-11-20T12:01:37.188 INFO:teuthology.orchestra.run.smithi187.stderr:676 bytes copied, 0.000128474 s, 5.3 MB/s 2020-11-20T12:01:37.193 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2020-11-20T12:01:37.193 INFO:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b86e691bf191566922a8f08e2a5abf64fde8cfcb/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-11-20T12:01:37.249 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-11-20T12:01:37.250 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:01:37.250 INFO:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-20T12:01:37.297 INFO:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-11-20T12:01:37.308 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-20T12:01:37.358 INFO:teuthology.orchestra.run.smithi005.stderr:1+1 records in 2020-11-20T12:01:37.359 INFO:teuthology.orchestra.run.smithi005.stderr:1+1 records out 2020-11-20T12:01:37.359 INFO:teuthology.orchestra.run.smithi005.stderr:676 bytes copied, 0.000136591 s, 4.9 MB/s 2020-11-20T12:01:37.362 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2020-11-20T12:01:37.362 INFO:teuthology.orchestra.run.smithi005:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b86e691bf191566922a8f08e2a5abf64fde8cfcb/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-11-20T12:01:37.454 INFO:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-11-20T12:01:37.471 INFO:teuthology.orchestra.run.smithi005:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-11-20T12:01:37.554 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2020-11-20T12:01:37.559 INFO:teuthology.orchestra.run.smithi187:> sudo yum clean all 2020-11-20T12:01:37.641 INFO:teuthology.orchestra.run.smithi005:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-11-20T12:01:37.749 INFO:teuthology.orchestra.run.smithi005.stdout:check_obsoletes = 1 2020-11-20T12:01:37.754 INFO:teuthology.orchestra.run.smithi005:> sudo yum clean all 2020-11-20T12:01:38.013 INFO:teuthology.orchestra.run.smithi187.stdout:42 files removed 2020-11-20T12:01:38.058 INFO:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2020-11-20T12:01:38.214 INFO:teuthology.orchestra.run.smithi005.stdout:42 files removed 2020-11-20T12:01:38.260 INFO:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2020-11-20T12:01:38.862 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 287 kB/s | 72 kB 00:00 2020-11-20T12:01:39.102 INFO:teuthology.orchestra.run.smithi005.stdout:ceph packages for x86_64 281 kB/s | 72 kB 00:00 2020-11-20T12:01:39.173 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 64 kB/s | 15 kB 00:00 2020-11-20T12:01:39.420 INFO:teuthology.orchestra.run.smithi005.stdout:ceph noarch packages 63 kB/s | 15 kB 00:00 2020-11-20T12:01:39.473 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 6.0 kB/s | 1.4 kB 00:00 2020-11-20T12:01:39.727 INFO:teuthology.orchestra.run.smithi005.stdout:ceph source packages 5.9 kB/s | 1.4 kB 00:00 2020-11-20T12:01:39.730 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - AppStream 35 MB/s | 7.0 MB 00:00 2020-11-20T12:01:39.978 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - AppStream 37 MB/s | 7.0 MB 00:00 2020-11-20T12:01:41.122 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - Base 23 MB/s | 2.2 MB 00:00 2020-11-20T12:01:41.412 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - Base 22 MB/s | 2.2 MB 00:00 2020-11-20T12:01:41.567 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - Extras 82 kB/s | 5.5 kB 00:00 2020-11-20T12:01:41.888 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - Extras 80 kB/s | 5.5 kB 00:00 2020-11-20T12:01:42.056 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 19 MB/s | 8.3 MB 00:00 2020-11-20T12:01:42.327 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 22 MB/s | 8.3 MB 00:00 2020-11-20T12:01:43.940 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 3.4 MB/s | 23 kB 00:00 2020-11-20T12:01:44.313 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 4.1 MB/s | 23 kB 00:00 2020-11-20T12:01:44.871 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2020-11-20T12:01:44.872 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2020-11-20T12:01:44.877 INFO:teuthology.orchestra.run.smithi187.stdout:Package NetworkManager-1:1.20.0-3.el8.x86_64 is already installed. 2020-11-20T12:01:44.877 INFO:teuthology.orchestra.run.smithi187.stdout:Package iproute-4.18.0-15.el8.x86_64 is already installed. 2020-11-20T12:01:44.878 INFO:teuthology.orchestra.run.smithi187.stdout:Package util-linux-2.32.1-17.el8.x86_64 is already installed. 2020-11-20T12:01:44.878 INFO:teuthology.orchestra.run.smithi187.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2020-11-20T12:01:44.879 INFO:teuthology.orchestra.run.smithi187.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2020-11-20T12:01:44.881 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2020-11-20T12:01:44.881 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsprogs-5.0.0-1.el8.x86_64 is already installed. 2020-11-20T12:01:45.032 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2020-11-20T12:01:45.074 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2020-11-20T12:01:45.074 INFO:teuthology.orchestra.run.smithi187.stdout: Package ArchVersion Repository Size 2020-11-20T12:01:45.074 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2020-11-20T12:01:45.074 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2020-11-20T12:01:45.075 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_642:16.0.0-7397.gb86e691b.el8 ceph 6.4 k 2020-11-20T12:01:45.075 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_642:16.0.0-7397.gb86e691b.el8 ceph 6.0 M 2020-11-20T12:01:45.075 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_642:16.0.0-7397.gb86e691b.el8 ceph 632 k 2020-11-20T12:01:45.075 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_642:16.0.0-7397.gb86e691b.el8 ceph 204 k 2020-11-20T12:01:45.075 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_642:16.0.0-7397.gb86e691b.el8 ceph 1.5 M 2020-11-20T12:01:45.076 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_642:16.0.0-7397.gb86e691b.el8 ceph 12 M 2020-11-20T12:01:45.076 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_642:16.0.0-7397.gb86e691b.el8 ceph 48 M 2020-11-20T12:01:45.076 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_642:16.0.0-7397.gb86e691b.el8 ceph 24 k 2020-11-20T12:01:45.076 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_642:16.0.0-7397.gb86e691b.el8 ceph 562 k 2020-11-20T12:01:45.077 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_642:16.0.0-7397.gb86e691b.el8 ceph 131 k 2020-11-20T12:01:45.077 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_642:16.0.0-7397.gb86e691b.el8 ceph 204 k 2020-11-20T12:01:45.077 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_642:16.0.0-7397.gb86e691b.el8 ceph 375 k 2020-11-20T12:01:45.077 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_642:16.0.0-7397.gb86e691b.el8 ceph 353 k 2020-11-20T12:01:45.077 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_642:16.0.0-7397.gb86e691b.el8 ceph 113 k 2020-11-20T12:01:45.078 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_642:16.0.0-7397.gb86e691b.el8 ceph 88 k 2020-11-20T12:01:45.078 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_642:16.0.0-7397.gb86e691b.el8 ceph 3.2 M 2020-11-20T12:01:45.078 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_642:16.0.0-7397.gb86e691b.el8 ceph 171 k 2020-11-20T12:01:45.078 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 66 k 2020-11-20T12:01:45.078 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 3.2 M 2020-11-20T12:01:45.079 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 8.6 M 2020-11-20T12:01:45.079 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 27 k 2020-11-20T12:01:45.079 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 57 k 2020-11-20T12:01:45.079 INFO:teuthology.orchestra.run.smithi187.stdout: bison x86_643.0.4-10.el8 CentOS-AppStream 688 k 2020-11-20T12:01:45.080 INFO:teuthology.orchestra.run.smithi187.stdout: flex x86_642.6.1-9.el8 CentOS-AppStream 320 k 2020-11-20T12:01:45.080 INFO:teuthology.orchestra.run.smithi187.stdout: libtool x86_642.4.6-25.el8 CentOS-AppStream 709 k 2020-11-20T12:01:45.080 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel x86_640.176-5.el8 CentOS-Base 54 k 2020-11-20T12:01:45.080 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel x86_640.3.112-1.el8 CentOS-Base 19 k 2020-11-20T12:01:45.080 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel x86_642.32.1-17.el8 CentOS-Base 95 k 2020-11-20T12:01:45.081 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel x86_641:1.1.1c-2.el8_1.1 CentOS-Base 2.3 M 2020-11-20T12:01:45.081 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel x86_645.0.0-1.el8 CentOS-Base 69 k 2020-11-20T12:01:45.081 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2020-11-20T12:01:45.081 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_642:16.0.0-7397.gb86e691b.el8 ceph 3.7 M 2020-11-20T12:01:45.081 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_642:16.0.0-7397.gb86e691b.el8 ceph 2.4 M 2020-11-20T12:01:45.082 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager x86_641:1.20.0-5.el8_1 CentOS-Base 2.1 M 2020-11-20T12:01:45.082 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-libnm x86_641:1.20.0-5.el8_1 CentOS-Base 1.7 M 2020-11-20T12:01:45.082 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-team x86_641:1.20.0-5.el8_1 CentOS-Base 133 k 2020-11-20T12:01:45.082 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-tui x86_641:1.20.0-5.el8_1 CentOS-Base 308 k 2020-11-20T12:01:45.083 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2020-11-20T12:01:45.083 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_642:16.0.0-7397.gb86e691b.el8 ceph 23 M 2020-11-20T12:01:45.083 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_642:16.0.0-7397.gb86e691b.el8 ceph 2.1 M 2020-11-20T12:01:45.083 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_642:16.0.0-7397.gb86e691b.el8 ceph 5.0 M 2020-11-20T12:01:45.083 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_642:16.0.0-7397.gb86e691b.el8 ceph 18 M 2020-11-20T12:01:45.084 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_642:16.0.0-7397.gb86e691b.el8 ceph 25 k 2020-11-20T12:01:45.084 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_642:16.0.0-7397.gb86e691b.el8 ceph 486 k 2020-11-20T12:01:45.084 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_642:16.0.0-7397.gb86e691b.el8 ceph 3.7 M 2020-11-20T12:01:45.084 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_642:16.0.0-7397.gb86e691b.el8 ceph 40 k 2020-11-20T12:01:45.085 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_642:16.0.0-7397.gb86e691b.el8 ceph 73 k 2020-11-20T12:01:45.085 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 24 k 2020-11-20T12:01:45.085 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 198 k 2020-11-20T12:01:45.085 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 8.8 k 2020-11-20T12:01:45.085 INFO:teuthology.orchestra.run.smithi187.stdout: automake noarch1.16.1-6.el8 CentOS-AppStream 713 k 2020-11-20T12:01:45.086 INFO:teuthology.orchestra.run.smithi187.stdout: jq x86_641.5-12.el8 CentOS-AppStream 161 k 2020-11-20T12:01:45.086 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_640.11.4-1.el8 CentOS-AppStream 353 k 2020-11-20T12:01:45.086 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_642.8.1-11.el8 CentOS-AppStream 259 k 2020-11-20T12:01:45.086 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma x86_646.8.2-1.el8 CentOS-AppStream 188 k 2020-11-20T12:01:45.086 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-11-20T12:01:45.087 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-11-20T12:01:45.087 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch2.5.1-5.el8 CentOS-AppStream 4.8 M 2020-11-20T12:01:45.087 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch2.10.1-2.el8_0 CentOS-AppStream 538 k 2020-11-20T12:01:45.087 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch1.21-2.el8 CentOS-AppStream 27 k 2020-11-20T12:01:45.087 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch1.10-11.el8 CentOS-AppStream 20 k 2020-11-20T12:01:45.088 INFO:teuthology.orchestra.run.smithi187.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-11-20T12:01:45.088 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-11-20T12:01:45.088 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-11-20T12:01:45.088 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-11-20T12:01:45.089 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-11-20T12:01:45.089 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch0.7.2-14.el8 CentOS-AppStream 44 k 2020-11-20T12:01:45.089 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch0.3.7-6.el8 CentOS-AppStream 126 k 2020-11-20T12:01:45.089 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch0.3.7-6.el8 CentOS-AppStream 110 k 2020-11-20T12:01:45.091 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-11-20T12:01:45.091 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-11-20T12:01:45.091 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-11-20T12:01:45.091 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-11-20T12:01:45.092 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_641.7.3.2-6.el8 CentOS-AppStream 298 k 2020-11-20T12:01:45.092 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_642.9.7-12.el8 CentOS-Base 82 k 2020-11-20T12:01:45.092 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-common x86_643.2.1-12.el8 CentOS-Base 21 k 2020-11-20T12:01:45.092 INFO:teuthology.orchestra.run.smithi187.stdout: keyutils-libs-devel x86_641.5.10-6.el8 CentOS-Base 48 k 2020-11-20T12:01:45.092 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-devel x86_641.17-9.el8 CentOS-Base 548 k 2020-11-20T12:01:45.093 INFO:teuthology.orchestra.run.smithi187.stdout: libcom_err-devel x86_641.44.6-3.el8 CentOS-Base 38 k 2020-11-20T12:01:45.093 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-11-20T12:01:45.093 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-11-20T12:01:45.093 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_640.9.0-1.el8 CentOS-Base 47 k 2020-11-20T12:01:45.093 INFO:teuthology.orchestra.run.smithi187.stdout: libselinux-devel x86_642.9-2.1.el8 CentOS-Base 199 k 2020-11-20T12:01:45.094 INFO:teuthology.orchestra.run.smithi187.stdout: libsepol-devel x86_642.9-1.el8 CentOS-Base 86 k 2020-11-20T12:01:45.094 INFO:teuthology.orchestra.run.smithi187.stdout: libverto-devel x86_640.3.0-5.el8 CentOS-Base 18 k 2020-11-20T12:01:45.094 INFO:teuthology.orchestra.run.smithi187.stdout: libxslt x86_641.1.32-3.el8 CentOS-Base 249 k 2020-11-20T12:01:45.094 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-devel x86_6410.32-1.el8 CentOS-Base 605 k 2020-11-20T12:01:45.095 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf16 x86_6410.32-1.el8 CentOS-Base 228 k 2020-11-20T12:01:45.095 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf32 x86_6410.32-1.el8 CentOS-Base 220 k 2020-11-20T12:01:45.095 INFO:teuthology.orchestra.run.smithi187.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-11-20T12:01:45.095 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-11-20T12:01:45.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch2.1.0-1.el8 CentOS-Base 155 k 2020-11-20T12:01:45.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-11-20T12:01:45.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-11-20T12:01:45.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch1.0.0-1.el8 CentOS-Base 43 k 2020-11-20T12:01:45.097 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-11-20T12:01:45.097 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel x86_641.2.11-10.el8 CentOS-Base 56 k 2020-11-20T12:01:45.097 INFO:teuthology.orchestra.run.smithi187.stdout: fmt x86_646.2.1-1.el8 epel 116 k 2020-11-20T12:01:45.097 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_642.7-6.el8 epel 315 k 2020-11-20T12:01:45.097 INFO:teuthology.orchestra.run.smithi187.stdout: leveldb x86_641.22-1.el8 epel 181 k 2020-11-20T12:01:45.098 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_642.6.2-3.el8 epel 59 k 2020-11-20T12:01:45.098 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_641.3.1-3.el8 epel 75 k 2020-11-20T12:01:45.098 INFO:teuthology.orchestra.run.smithi187.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-11-20T12:01:45.098 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-11-20T12:01:45.098 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-11-20T12:01:45.099 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch3.1.1-4.el8 epel 33 k 2020-11-20T12:01:45.099 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch2018.10.15-7.el8 epel 16 k 2020-11-20T12:01:45.099 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch8.2.1-1.el8 epel 145 k 2020-11-20T12:01:45.099 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch18.4.0-1.el8 epel 384 k 2020-11-20T12:01:45.099 INFO:teuthology.orchestra.run.smithi187.stdout: python3-execnet noarch1.7.1-1.el8 epel 355 k 2020-11-20T12:01:45.100 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch1:1.1.1-10.el8 epel 81 k 2020-11-20T12:01:45.100 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch6.2-6.el8 epel 11 k 2020-11-20T12:01:45.100 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch2.0-4.el8 epel 18 k 2020-11-20T12:01:45.100 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch11.0.0-2.el8 epel 1.2 M 2020-11-20T12:01:45.101 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch0.3.5-11.el8 epel 49 k 2020-11-20T12:01:45.101 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch7.2.0-3.el8 epel 59 k 2020-11-20T12:01:45.101 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch1.3.2-9.el8 epel 283 k 2020-11-20T12:01:45.101 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch2.6-1.el8 epel 16 k 2020-11-20T12:01:45.101 INFO:teuthology.orchestra.run.smithi187.stdout: python3-remoto noarch1.1.4-4.el8 epel 48 k 2020-11-20T12:01:45.102 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch0.7-6.el8 epel 34 k 2020-11-20T12:01:45.102 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch2.4.1-12.el8 epel 196 k 2020-11-20T12:01:45.102 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch3.4.2-15.el8 epel 73 k 2020-11-20T12:01:45.102 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric noarch0.8.1-17.el8 epel 19 k 2020-11-20T12:01:45.102 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch noarch3.4.0.3-18.el8 epel 24 k 2020-11-20T12:01:45.103 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch1.14.1-5.el8 epel 29 k 2020-11-20T12:01:45.103 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme noarch0.6.0-4.el8 epel 27 k 2020-11-20T12:01:45.103 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-11-20T12:01:45.103 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-11-20T12:01:45.103 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch0.56.0-5.el8 epel 61 k 2020-11-20T12:01:45.104 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-11-20T12:01:45.104 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch2.0-2.el8 epel 23 k 2020-11-20T12:01:45.104 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_641.6.1-11.el8 epel 69 k 2020-11-20T12:01:45.104 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2020-11-20T12:01:45.104 INFO:teuthology.orchestra.run.smithi187.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-11-20T12:01:45.105 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-11-20T12:01:45.105 INFO:teuthology.orchestra.run.smithi187.stdout:Downgrading: 2020-11-20T12:01:45.105 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-libs x86_641.17-9.el8 CentOS-Base 851 k 2020-11-20T12:01:45.105 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-workstation x86_641.17-9.el8 CentOS-Base 940 k 2020-11-20T12:01:45.106 INFO:teuthology.orchestra.run.smithi187.stdout: libkadm5 x86_641.17-9.el8 CentOS-Base 184 k 2020-11-20T12:01:45.106 INFO:teuthology.orchestra.run.smithi187.stdout: openssl x86_641:1.1.1c-2.el8_1.1 CentOS-Base 686 k 2020-11-20T12:01:45.106 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-libs x86_641:1.1.1c-2.el8_1.1 CentOS-Base 1.5 M 2020-11-20T12:01:45.106 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:01:45.106 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2020-11-20T12:01:45.107 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2020-11-20T12:01:45.107 INFO:teuthology.orchestra.run.smithi187.stdout:Install 126 Packages 2020-11-20T12:01:45.107 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 6 Packages 2020-11-20T12:01:45.108 INFO:teuthology.orchestra.run.smithi187.stdout:Downgrade 5 Packages 2020-11-20T12:01:45.108 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:01:45.108 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 202 M 2020-11-20T12:01:45.108 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2020-11-20T12:01:45.123 INFO:teuthology.orchestra.run.smithi187.stdout:(1/137): libkadm5-1.17-9.el8.x86_64.rpm 4.2 MB/s | 184 kB 00:00 2020-11-20T12:01:45.154 INFO:teuthology.orchestra.run.smithi187.stdout:(2/137): krb5-workstation-1.17-9.el8.x86_64.rpm 12 MB/s | 940 kB 00:00 2020-11-20T12:01:45.162 INFO:teuthology.orchestra.run.smithi187.stdout:(3/137): openssl-1.1.1c-2.el8_1.1.x86_64.rpm 17 MB/s | 686 kB 00:00 2020-11-20T12:01:45.169 INFO:teuthology.orchestra.run.smithi187.stdout:(4/137): krb5-libs-1.17-9.el8.x86_64.rpm 9.3 MB/s | 851 kB 00:00 2020-11-20T12:01:45.187 INFO:teuthology.orchestra.run.smithi187.stdout:(5/137): openssl-libs-1.1.1c-2.el8_1.1.x86_64.r 46 MB/s | 1.5 MB 00:00 2020-11-20T12:01:45.264 INFO:teuthology.orchestra.run.smithi187.stdout:(6/137): ceph-16.0.0-7397.gb86e691b.el8.x86_64. 64 kB/s | 6.4 kB 00:00 2020-11-20T12:01:45.323 INFO:teuthology.orchestra.run.smithi005.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2020-11-20T12:01:45.323 INFO:teuthology.orchestra.run.smithi005.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2020-11-20T12:01:45.328 INFO:teuthology.orchestra.run.smithi005.stdout:Package NetworkManager-1:1.20.0-3.el8.x86_64 is already installed. 2020-11-20T12:01:45.329 INFO:teuthology.orchestra.run.smithi005.stdout:Package iproute-4.18.0-15.el8.x86_64 is already installed. 2020-11-20T12:01:45.329 INFO:teuthology.orchestra.run.smithi005.stdout:Package util-linux-2.32.1-17.el8.x86_64 is already installed. 2020-11-20T12:01:45.330 INFO:teuthology.orchestra.run.smithi005.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2020-11-20T12:01:45.331 INFO:teuthology.orchestra.run.smithi005.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2020-11-20T12:01:45.333 INFO:teuthology.orchestra.run.smithi005.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2020-11-20T12:01:45.334 INFO:teuthology.orchestra.run.smithi005.stdout:Package xfsprogs-5.0.0-1.el8.x86_64 is already installed. 2020-11-20T12:01:45.435 INFO:teuthology.orchestra.run.smithi187.stdout:(7/137): ceph-fuse-16.0.0-7397.gb86e691b.el8.x8 3.6 MB/s | 632 kB 00:00 2020-11-20T12:01:45.469 INFO:teuthology.orchestra.run.smithi187.stdout:(8/137): ceph-immutable-object-cache-16.0.0-739 6.1 MB/s | 204 kB 00:00 2020-11-20T12:01:45.494 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2020-11-20T12:01:45.516 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2020-11-20T12:01:45.516 INFO:teuthology.orchestra.run.smithi005.stdout: Package ArchVersion Repository Size 2020-11-20T12:01:45.516 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2020-11-20T12:01:45.517 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2020-11-20T12:01:45.517 INFO:teuthology.orchestra.run.smithi005.stdout: ceph x86_642:16.0.0-7397.gb86e691b.el8 ceph 6.4 k 2020-11-20T12:01:45.517 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base x86_642:16.0.0-7397.gb86e691b.el8 ceph 6.0 M 2020-11-20T12:01:45.517 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse x86_642:16.0.0-7397.gb86e691b.el8 ceph 632 k 2020-11-20T12:01:45.517 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache x86_642:16.0.0-7397.gb86e691b.el8 ceph 204 k 2020-11-20T12:01:45.518 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr x86_642:16.0.0-7397.gb86e691b.el8 ceph 1.5 M 2020-11-20T12:01:45.518 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw x86_642:16.0.0-7397.gb86e691b.el8 ceph 12 M 2020-11-20T12:01:45.518 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test x86_642:16.0.0-7397.gb86e691b.el8 ceph 48 M 2020-11-20T12:01:45.518 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel x86_642:16.0.0-7397.gb86e691b.el8 ceph 24 k 2020-11-20T12:01:45.519 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2 x86_642:16.0.0-7397.gb86e691b.el8 ceph 562 k 2020-11-20T12:01:45.519 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel x86_642:16.0.0-7397.gb86e691b.el8 ceph 131 k 2020-11-20T12:01:45.519 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs x86_642:16.0.0-7397.gb86e691b.el8 ceph 204 k 2020-11-20T12:01:45.519 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados x86_642:16.0.0-7397.gb86e691b.el8 ceph 375 k 2020-11-20T12:01:45.519 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd x86_642:16.0.0-7397.gb86e691b.el8 ceph 353 k 2020-11-20T12:01:45.520 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw x86_642:16.0.0-7397.gb86e691b.el8 ceph 113 k 2020-11-20T12:01:45.520 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse x86_642:16.0.0-7397.gb86e691b.el8 ceph 88 k 2020-11-20T12:01:45.520 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror x86_642:16.0.0-7397.gb86e691b.el8 ceph 3.2 M 2020-11-20T12:01:45.520 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd x86_642:16.0.0-7397.gb86e691b.el8 ceph 171 k 2020-11-20T12:01:45.520 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 66 k 2020-11-20T12:01:45.521 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 3.2 M 2020-11-20T12:01:45.521 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 8.6 M 2020-11-20T12:01:45.521 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 27 k 2020-11-20T12:01:45.521 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 57 k 2020-11-20T12:01:45.522 INFO:teuthology.orchestra.run.smithi005.stdout: bison x86_643.0.4-10.el8 CentOS-AppStream 688 k 2020-11-20T12:01:45.522 INFO:teuthology.orchestra.run.smithi005.stdout: flex x86_642.6.1-9.el8 CentOS-AppStream 320 k 2020-11-20T12:01:45.522 INFO:teuthology.orchestra.run.smithi005.stdout: libtool x86_642.4.6-25.el8 CentOS-AppStream 709 k 2020-11-20T12:01:45.522 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-libelf-devel x86_640.176-5.el8 CentOS-Base 54 k 2020-11-20T12:01:45.522 INFO:teuthology.orchestra.run.smithi005.stdout: libaio-devel x86_640.3.112-1.el8 CentOS-Base 19 k 2020-11-20T12:01:45.523 INFO:teuthology.orchestra.run.smithi005.stdout: libuuid-devel x86_642.32.1-17.el8 CentOS-Base 95 k 2020-11-20T12:01:45.523 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-devel x86_641:1.1.1c-2.el8_1.1 CentOS-Base 2.3 M 2020-11-20T12:01:45.523 INFO:teuthology.orchestra.run.smithi005.stdout: xfsprogs-devel x86_645.0.0-1.el8 CentOS-Base 69 k 2020-11-20T12:01:45.523 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrading: 2020-11-20T12:01:45.523 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 x86_642:16.0.0-7397.gb86e691b.el8 ceph 3.7 M 2020-11-20T12:01:45.524 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1 x86_642:16.0.0-7397.gb86e691b.el8 ceph 2.4 M 2020-11-20T12:01:45.524 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager x86_641:1.20.0-5.el8_1 CentOS-Base 2.1 M 2020-11-20T12:01:45.524 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-libnm x86_641:1.20.0-5.el8_1 CentOS-Base 1.7 M 2020-11-20T12:01:45.524 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-team x86_641:1.20.0-5.el8_1 CentOS-Base 133 k 2020-11-20T12:01:45.525 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-tui x86_641:1.20.0-5.el8_1 CentOS-Base 308 k 2020-11-20T12:01:45.525 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2020-11-20T12:01:45.525 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common x86_642:16.0.0-7397.gb86e691b.el8 ceph 23 M 2020-11-20T12:01:45.525 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds x86_642:16.0.0-7397.gb86e691b.el8 ceph 2.1 M 2020-11-20T12:01:45.525 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon x86_642:16.0.0-7397.gb86e691b.el8 ceph 5.0 M 2020-11-20T12:01:45.526 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd x86_642:16.0.0-7397.gb86e691b.el8 ceph 18 M 2020-11-20T12:01:45.526 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux x86_642:16.0.0-7397.gb86e691b.el8 ceph 25 k 2020-11-20T12:01:45.526 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1 x86_642:16.0.0-7397.gb86e691b.el8 ceph 486 k 2020-11-20T12:01:45.526 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2 x86_642:16.0.0-7397.gb86e691b.el8 ceph 3.7 M 2020-11-20T12:01:45.526 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse x86_642:16.0.0-7397.gb86e691b.el8 ceph 40 k 2020-11-20T12:01:45.527 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common x86_642:16.0.0-7397.gb86e691b.el8 ceph 73 k 2020-11-20T12:01:45.527 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 24 k 2020-11-20T12:01:45.527 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 198 k 2020-11-20T12:01:45.527 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts noarch2:16.0.0-7397.gb86e691b.el8 ceph-noarch 8.8 k 2020-11-20T12:01:45.527 INFO:teuthology.orchestra.run.smithi005.stdout: automake noarch1.16.1-6.el8 CentOS-AppStream 713 k 2020-11-20T12:01:45.528 INFO:teuthology.orchestra.run.smithi005.stdout: jq x86_641.5-12.el8 CentOS-AppStream 161 k 2020-11-20T12:01:45.528 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka x86_640.11.4-1.el8 CentOS-AppStream 353 k 2020-11-20T12:01:45.528 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust x86_642.8.1-11.el8 CentOS-AppStream 259 k 2020-11-20T12:01:45.528 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma x86_646.8.2-1.el8 CentOS-AppStream 188 k 2020-11-20T12:01:45.528 INFO:teuthology.orchestra.run.smithi005.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-11-20T12:01:45.529 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-11-20T12:01:45.529 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel noarch2.5.1-5.el8 CentOS-AppStream 4.8 M 2020-11-20T12:01:45.529 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2 noarch2.10.1-2.el8_0 CentOS-AppStream 538 k 2020-11-20T12:01:45.529 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch noarch1.21-2.el8 CentOS-AppStream 27 k 2020-11-20T12:01:45.530 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer noarch1.10-11.el8 CentOS-AppStream 20 k 2020-11-20T12:01:45.530 INFO:teuthology.orchestra.run.smithi005.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-11-20T12:01:45.530 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-11-20T12:01:45.530 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-11-20T12:01:45.530 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-11-20T12:01:45.531 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-11-20T12:01:45.531 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable noarch0.7.2-14.el8 CentOS-AppStream 44 k 2020-11-20T12:01:45.531 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 noarch0.3.7-6.el8 CentOS-AppStream 126 k 2020-11-20T12:01:45.531 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules noarch0.3.7-6.el8 CentOS-AppStream 110 k 2020-11-20T12:01:45.531 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-11-20T12:01:45.532 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-11-20T12:01:45.532 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-11-20T12:01:45.532 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-11-20T12:01:45.532 INFO:teuthology.orchestra.run.smithi005.stdout: socat x86_641.7.3.2-6.el8 CentOS-AppStream 298 k 2020-11-20T12:01:45.532 INFO:teuthology.orchestra.run.smithi005.stdout: fuse x86_642.9.7-12.el8 CentOS-Base 82 k 2020-11-20T12:01:45.533 INFO:teuthology.orchestra.run.smithi005.stdout: fuse-common x86_643.2.1-12.el8 CentOS-Base 21 k 2020-11-20T12:01:45.533 INFO:teuthology.orchestra.run.smithi005.stdout: keyutils-libs-devel x86_641.5.10-6.el8 CentOS-Base 48 k 2020-11-20T12:01:45.533 INFO:teuthology.orchestra.run.smithi005.stdout: krb5-devel x86_641.17-9.el8 CentOS-Base 548 k 2020-11-20T12:01:45.533 INFO:teuthology.orchestra.run.smithi005.stdout: libcom_err-devel x86_641.44.6-3.el8 CentOS-Base 38 k 2020-11-20T12:01:45.534 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-11-20T12:01:45.534 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-11-20T12:01:45.534 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq x86_640.9.0-1.el8 CentOS-Base 47 k 2020-11-20T12:01:45.534 INFO:teuthology.orchestra.run.smithi005.stdout: libselinux-devel x86_642.9-2.1.el8 CentOS-Base 199 k 2020-11-20T12:01:45.534 INFO:teuthology.orchestra.run.smithi005.stdout: libsepol-devel x86_642.9-1.el8 CentOS-Base 86 k 2020-11-20T12:01:45.535 INFO:teuthology.orchestra.run.smithi005.stdout: libverto-devel x86_640.3.0-5.el8 CentOS-Base 18 k 2020-11-20T12:01:45.535 INFO:teuthology.orchestra.run.smithi005.stdout: libxslt x86_641.1.32-3.el8 CentOS-Base 249 k 2020-11-20T12:01:45.535 INFO:teuthology.orchestra.run.smithi005.stdout: pcre2-devel x86_6410.32-1.el8 CentOS-Base 605 k 2020-11-20T12:01:45.535 INFO:teuthology.orchestra.run.smithi005.stdout: pcre2-utf16 x86_6410.32-1.el8 CentOS-Base 228 k 2020-11-20T12:01:45.536 INFO:teuthology.orchestra.run.smithi005.stdout: pcre2-utf32 x86_6410.32-1.el8 CentOS-Base 220 k 2020-11-20T12:01:45.536 INFO:teuthology.orchestra.run.smithi005.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-11-20T12:01:45.536 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-11-20T12:01:45.536 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib noarch2.1.0-1.el8 CentOS-Base 155 k 2020-11-20T12:01:45.536 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-11-20T12:01:45.537 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-11-20T12:01:45.537 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib noarch1.0.0-1.el8 CentOS-Base 43 k 2020-11-20T12:01:45.537 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-11-20T12:01:45.537 INFO:teuthology.orchestra.run.smithi005.stdout: zlib-devel x86_641.2.11-10.el8 CentOS-Base 56 k 2020-11-20T12:01:45.537 INFO:teuthology.orchestra.run.smithi005.stdout: fmt x86_646.2.1-1.el8 epel 116 k 2020-11-20T12:01:45.538 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs x86_642.7-6.el8 epel 315 k 2020-11-20T12:01:45.538 INFO:teuthology.orchestra.run.smithi005.stdout: leveldb x86_641.22-1.el8 epel 181 k 2020-11-20T12:01:45.538 INFO:teuthology.orchestra.run.smithi005.stdout: liboath x86_642.6.2-3.el8 epel 59 k 2020-11-20T12:01:45.538 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind x86_641.3.1-3.el8 epel 75 k 2020-11-20T12:01:45.538 INFO:teuthology.orchestra.run.smithi005.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-11-20T12:01:45.539 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-11-20T12:01:45.539 INFO:teuthology.orchestra.run.smithi005.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-11-20T12:01:45.539 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools noarch3.1.1-4.el8 epel 33 k 2020-11-20T12:01:45.539 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi noarch2018.10.15-7.el8 epel 16 k 2020-11-20T12:01:45.540 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot noarch8.2.1-1.el8 epel 145 k 2020-11-20T12:01:45.540 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy noarch18.4.0-1.el8 epel 384 k 2020-11-20T12:01:45.540 INFO:teuthology.orchestra.run.smithi005.stdout: python3-execnet noarch1.7.1-1.el8 epel 355 k 2020-11-20T12:01:45.540 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth noarch1:1.1.1-10.el8 epel 81 k 2020-11-20T12:01:45.540 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco noarch6.2-6.el8 epel 11 k 2020-11-20T12:01:45.541 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools noarch2.0-4.el8 epel 18 k 2020-11-20T12:01:45.541 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes noarch11.0.0-2.el8 epel 1.2 M 2020-11-20T12:01:45.541 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils noarch0.3.5-11.el8 epel 49 k 2020-11-20T12:01:45.541 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools noarch7.2.0-3.el8 epel 59 k 2020-11-20T12:01:45.541 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan noarch1.3.2-9.el8 epel 283 k 2020-11-20T12:01:45.542 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend noarch2.6-1.el8 epel 16 k 2020-11-20T12:01:45.542 INFO:teuthology.orchestra.run.smithi005.stdout: python3-remoto noarch1.1.4-4.el8 epel 48 k 2020-11-20T12:01:45.542 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru noarch0.7-6.el8 epel 34 k 2020-11-20T12:01:45.542 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes noarch2.4.1-12.el8 epel 196 k 2020-11-20T12:01:45.542 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa noarch3.4.2-15.el8 epel 73 k 2020-11-20T12:01:45.543 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric noarch0.8.1-17.el8 epel 19 k 2020-11-20T12:01:45.543 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch noarch3.4.0.3-18.el8 epel 24 k 2020-11-20T12:01:45.543 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora noarch1.14.1-5.el8 epel 29 k 2020-11-20T12:01:45.543 INFO:teuthology.orchestra.run.smithi005.stdout: python3-trustme noarch0.6.0-4.el8 epel 27 k 2020-11-20T12:01:45.543 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-11-20T12:01:45.544 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-11-20T12:01:45.544 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client noarch0.56.0-5.el8 epel 61 k 2020-11-20T12:01:45.544 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-11-20T12:01:45.544 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile noarch2.0-2.el8 epel 23 k 2020-11-20T12:01:45.545 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet x86_641.6.1-11.el8 epel 69 k 2020-11-20T12:01:45.545 INFO:teuthology.orchestra.run.smithi005.stdout:Installing weak dependencies: 2020-11-20T12:01:45.545 INFO:teuthology.orchestra.run.smithi005.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-11-20T12:01:45.545 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-11-20T12:01:45.545 INFO:teuthology.orchestra.run.smithi005.stdout:Downgrading: 2020-11-20T12:01:45.546 INFO:teuthology.orchestra.run.smithi005.stdout: krb5-libs x86_641.17-9.el8 CentOS-Base 851 k 2020-11-20T12:01:45.546 INFO:teuthology.orchestra.run.smithi005.stdout: krb5-workstation x86_641.17-9.el8 CentOS-Base 940 k 2020-11-20T12:01:45.546 INFO:teuthology.orchestra.run.smithi005.stdout: libkadm5 x86_641.17-9.el8 CentOS-Base 184 k 2020-11-20T12:01:45.547 INFO:teuthology.orchestra.run.smithi005.stdout: openssl x86_641:1.1.1c-2.el8_1.1 CentOS-Base 686 k 2020-11-20T12:01:45.547 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-libs x86_641:1.1.1c-2.el8_1.1 CentOS-Base 1.5 M 2020-11-20T12:01:45.547 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:01:45.547 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2020-11-20T12:01:45.547 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2020-11-20T12:01:45.548 INFO:teuthology.orchestra.run.smithi005.stdout:Install 126 Packages 2020-11-20T12:01:45.548 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrade 6 Packages 2020-11-20T12:01:45.548 INFO:teuthology.orchestra.run.smithi005.stdout:Downgrade 5 Packages 2020-11-20T12:01:45.548 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:01:45.549 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 202 M 2020-11-20T12:01:45.549 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2020-11-20T12:01:45.581 INFO:teuthology.orchestra.run.smithi005.stdout:(1/137): libkadm5-1.17-9.el8.x86_64.rpm 4.7 MB/s | 184 kB 00:00 2020-11-20T12:01:45.596 INFO:teuthology.orchestra.run.smithi005.stdout:(2/137): krb5-workstation-1.17-9.el8.x86_64.rpm 17 MB/s | 940 kB 00:00 2020-11-20T12:01:45.610 INFO:teuthology.orchestra.run.smithi005.stdout:(3/137): openssl-1.1.1c-2.el8_1.1.x86_64.rpm 23 MB/s | 686 kB 00:00 2020-11-20T12:01:45.619 INFO:teuthology.orchestra.run.smithi005.stdout:(4/137): krb5-libs-1.17-9.el8.x86_64.rpm 11 MB/s | 851 kB 00:00 2020-11-20T12:01:45.629 INFO:teuthology.orchestra.run.smithi005.stdout:(5/137): openssl-libs-1.1.1c-2.el8_1.1.x86_64.r 45 MB/s | 1.5 MB 00:00 2020-11-20T12:01:45.734 INFO:teuthology.orchestra.run.smithi187.stdout:(9/137): ceph-mds-16.0.0-7397.gb86e691b.el8.x86 8.1 MB/s | 2.1 MB 00:00 2020-11-20T12:01:45.745 INFO:teuthology.orchestra.run.smithi005.stdout:(6/137): ceph-16.0.0-7397.gb86e691b.el8.x86_64. 48 kB/s | 6.4 kB 00:00 2020-11-20T12:01:45.966 INFO:teuthology.orchestra.run.smithi187.stdout:(10/137): ceph-base-16.0.0-7397.gb86e691b.el8.x 7.5 MB/s | 6.0 MB 00:00 2020-11-20T12:01:45.976 INFO:teuthology.orchestra.run.smithi187.stdout:(11/137): ceph-mgr-16.0.0-7397.gb86e691b.el8.x8 6.4 MB/s | 1.5 MB 00:00 2020-11-20T12:01:46.138 INFO:teuthology.orchestra.run.smithi005.stdout:(7/137): ceph-fuse-16.0.0-7397.gb86e691b.el8.x8 1.6 MB/s | 632 kB 00:00 2020-11-20T12:01:46.270 INFO:teuthology.orchestra.run.smithi005.stdout:(8/137): ceph-immutable-object-cache-16.0.0-739 1.5 MB/s | 204 kB 00:00 2020-11-20T12:01:46.566 INFO:teuthology.orchestra.run.smithi187.stdout:(12/137): ceph-mon-16.0.0-7397.gb86e691b.el8.x8 8.3 MB/s | 5.0 MB 00:00 2020-11-20T12:01:47.372 INFO:teuthology.orchestra.run.smithi005.stdout:(9/137): ceph-mds-16.0.0-7397.gb86e691b.el8.x86 2.0 MB/s | 2.1 MB 00:01 2020-11-20T12:01:47.985 INFO:teuthology.orchestra.run.smithi187.stdout:(13/137): ceph-common-16.0.0-7397.gb86e691b.el8 8.2 MB/s | 23 MB 00:02 2020-11-20T12:01:48.079 INFO:teuthology.orchestra.run.smithi005.stdout:(10/137): ceph-mgr-16.0.0-7397.gb86e691b.el8.x8 2.2 MB/s | 1.5 MB 00:00 2020-11-20T12:01:48.080 INFO:teuthology.orchestra.run.smithi187.stdout:(14/137): ceph-radosgw-16.0.0-7397.gb86e691b.el 7.8 MB/s | 12 MB 00:01 2020-11-20T12:01:48.081 INFO:teuthology.orchestra.run.smithi187.stdout:(15/137): ceph-selinux-16.0.0-7397.gb86e691b.el 261 kB/s | 25 kB 00:00 2020-11-20T12:01:48.138 INFO:teuthology.orchestra.run.smithi187.stdout:(16/137): libcephfs-devel-16.0.0-7397.gb86e691b 419 kB/s | 24 kB 00:00 2020-11-20T12:01:48.215 INFO:teuthology.orchestra.run.smithi187.stdout:(17/137): libcephfs2-16.0.0-7397.gb86e691b.el8. 7.2 MB/s | 562 kB 00:00 2020-11-20T12:01:48.274 INFO:teuthology.orchestra.run.smithi187.stdout:(18/137): librados-devel-16.0.0-7397.gb86e691b. 2.2 MB/s | 131 kB 00:00 2020-11-20T12:01:48.362 INFO:teuthology.orchestra.run.smithi187.stdout:(19/137): ceph-osd-16.0.0-7397.gb86e691b.el8.x8 7.5 MB/s | 18 MB 00:02 2020-11-20T12:01:48.371 INFO:teuthology.orchestra.run.smithi187.stdout:(20/137): libradosstriper1-16.0.0-7397.gb86e691 4.9 MB/s | 486 kB 00:00 2020-11-20T12:01:48.426 INFO:teuthology.orchestra.run.smithi187.stdout:(21/137): python3-ceph-argparse-16.0.0-7397.gb8 731 kB/s | 40 kB 00:00 2020-11-20T12:01:48.482 INFO:teuthology.orchestra.run.smithi187.stdout:(22/137): python3-ceph-common-16.0.0-7397.gb86e 1.3 MB/s | 73 kB 00:00 2020-11-20T12:01:48.545 INFO:teuthology.orchestra.run.smithi187.stdout:(23/137): python3-cephfs-16.0.0-7397.gb86e691b. 3.2 MB/s | 204 kB 00:00 2020-11-20T12:01:48.602 INFO:teuthology.orchestra.run.smithi005.stdout:(11/137): ceph-base-16.0.0-7397.gb86e691b.el8.x 2.0 MB/s | 6.0 MB 00:02 2020-11-20T12:01:48.610 INFO:teuthology.orchestra.run.smithi187.stdout:(24/137): python3-rados-16.0.0-7397.gb86e691b.e 5.6 MB/s | 375 kB 00:00 2020-11-20T12:01:48.682 INFO:teuthology.orchestra.run.smithi187.stdout:(25/137): python3-rbd-16.0.0-7397.gb86e691b.el8 4.9 MB/s | 353 kB 00:00 2020-11-20T12:01:48.742 INFO:teuthology.orchestra.run.smithi187.stdout:(26/137): python3-rgw-16.0.0-7397.gb86e691b.el8 1.9 MB/s | 113 kB 00:00 2020-11-20T12:01:48.792 INFO:teuthology.orchestra.run.smithi187.stdout:(27/137): librgw2-16.0.0-7397.gb86e691b.el8.x86 8.6 MB/s | 3.7 MB 00:00 2020-11-20T12:01:48.797 INFO:teuthology.orchestra.run.smithi187.stdout:(28/137): rbd-fuse-16.0.0-7397.gb86e691b.el8.x8 1.6 MB/s | 88 kB 00:00 2020-11-20T12:01:48.862 INFO:teuthology.orchestra.run.smithi187.stdout:(29/137): rbd-nbd-16.0.0-7397.gb86e691b.el8.x86 2.6 MB/s | 171 kB 00:00 2020-11-20T12:01:48.922 INFO:teuthology.orchestra.run.smithi187.stdout:(30/137): ceph-grafana-dashboards-16.0.0-7397.g 409 kB/s | 24 kB 00:00 2020-11-20T12:01:48.981 INFO:teuthology.orchestra.run.smithi187.stdout:(31/137): ceph-mgr-cephadm-16.0.0-7397.gb86e691 1.1 MB/s | 66 kB 00:00 2020-11-20T12:01:49.194 INFO:teuthology.orchestra.run.smithi187.stdout:(32/137): rbd-mirror-16.0.0-7397.gb86e691b.el8. 7.9 MB/s | 3.2 MB 00:00 2020-11-20T12:01:49.414 INFO:teuthology.orchestra.run.smithi187.stdout:(33/137): ceph-mgr-dashboard-16.0.0-7397.gb86e6 7.3 MB/s | 3.2 MB 00:00 2020-11-20T12:01:49.482 INFO:teuthology.orchestra.run.smithi187.stdout:(34/137): ceph-mgr-modules-core-16.0.0-7397.gb8 2.9 MB/s | 198 kB 00:00 2020-11-20T12:01:49.541 INFO:teuthology.orchestra.run.smithi187.stdout:(35/137): ceph-mgr-rook-16.0.0-7397.gb86e691b.e 469 kB/s | 27 kB 00:00 2020-11-20T12:01:49.599 INFO:teuthology.orchestra.run.smithi187.stdout:(36/137): ceph-prometheus-alerts-16.0.0-7397.gb 152 kB/s | 8.8 kB 00:00 2020-11-20T12:01:49.660 INFO:teuthology.orchestra.run.smithi187.stdout:(37/137): cephadm-16.0.0-7397.gb86e691b.el8.noa 950 kB/s | 57 kB 00:00 2020-11-20T12:01:49.685 INFO:teuthology.orchestra.run.smithi187.stdout:(38/137): automake-1.16.1-6.el8.noarch.rpm 29 MB/s | 713 kB 00:00 2020-11-20T12:01:49.704 INFO:teuthology.orchestra.run.smithi187.stdout:(39/137): bison-3.0.4-10.el8.x86_64.rpm 37 MB/s | 688 kB 00:00 2020-11-20T12:01:49.712 INFO:teuthology.orchestra.run.smithi187.stdout:(40/137): flex-2.6.1-9.el8.x86_64.rpm 40 MB/s | 320 kB 00:00 2020-11-20T12:01:49.716 INFO:teuthology.orchestra.run.smithi187.stdout:(41/137): jq-1.5-12.el8.x86_64.rpm 42 MB/s | 161 kB 00:00 2020-11-20T12:01:49.724 INFO:teuthology.orchestra.run.smithi187.stdout:(42/137): librdkafka-0.11.4-1.el8.x86_64.rpm 45 MB/s | 353 kB 00:00 2020-11-20T12:01:49.738 INFO:teuthology.orchestra.run.smithi187.stdout:(43/137): libtool-2.4.6-25.el8.x86_64.rpm 53 MB/s | 709 kB 00:00 2020-11-20T12:01:49.744 INFO:teuthology.orchestra.run.smithi187.stdout:(44/137): lttng-ust-2.8.1-11.el8.x86_64.rpm 44 MB/s | 259 kB 00:00 2020-11-20T12:01:49.749 INFO:teuthology.orchestra.run.smithi187.stdout:(45/137): oniguruma-6.8.2-1.el8.x86_64.rpm 41 MB/s | 188 kB 00:00 2020-11-20T12:01:49.813 INFO:teuthology.orchestra.run.smithi187.stdout:(46/137): openblas-0.3.3-2.el8.x86_64.rpm 67 MB/s | 4.3 MB 00:00 2020-11-20T12:01:49.875 INFO:teuthology.orchestra.run.smithi187.stdout:(47/137): openblas-threads-0.3.3-2.el8.x86_64.r 71 MB/s | 4.4 MB 00:00 2020-11-20T12:01:49.956 INFO:teuthology.orchestra.run.smithi187.stdout:(48/137): python3-babel-2.5.1-5.el8.noarch.rpm 62 MB/s | 4.8 MB 00:00 2020-11-20T12:01:49.961 INFO:teuthology.orchestra.run.smithi187.stdout:(49/137): python3-html5lib-0.999999999-6.el8.no 40 MB/s | 214 kB 00:00 2020-11-20T12:01:49.970 INFO:teuthology.orchestra.run.smithi187.stdout:(50/137): python3-jinja2-2.10.1-2.el8_0.noarch. 65 MB/s | 538 kB 00:00 2020-11-20T12:01:49.972 INFO:teuthology.orchestra.run.smithi187.stdout:(51/137): python3-jsonpatch-1.21-2.el8.noarch.r 15 MB/s | 27 kB 00:00 2020-11-20T12:01:49.973 INFO:teuthology.orchestra.run.smithi187.stdout:(52/137): python3-jsonpointer-1.10-11.el8.noarc 15 MB/s | 20 kB 00:00 2020-11-20T12:01:49.992 INFO:teuthology.orchestra.run.smithi005.stdout:(12/137): ceph-mon-16.0.0-7397.gb86e691b.el8.x8 2.6 MB/s | 5.0 MB 00:01 2020-11-20T12:01:49.996 INFO:teuthology.orchestra.run.smithi187.stdout:(53/137): python3-lxml-4.2.3-1.el8.x86_64.rpm 68 MB/s | 1.5 MB 00:00 2020-11-20T12:01:50.000 INFO:teuthology.orchestra.run.smithi187.stdout:(54/137): python3-mako-1.0.6-13.el8.noarch.rpm 44 MB/s | 157 kB 00:00 2020-11-20T12:01:50.001 INFO:teuthology.orchestra.run.smithi187.stdout:(55/137): python3-markupsafe-0.23-19.el8.x86_64 25 MB/s | 39 kB 00:00 2020-11-20T12:01:50.055 INFO:teuthology.orchestra.run.smithi187.stdout:(56/137): python3-numpy-1.14.3-9.el8.x86_64.rpm 71 MB/s | 3.7 MB 00:00 2020-11-20T12:01:50.060 INFO:teuthology.orchestra.run.smithi187.stdout:(57/137): python3-numpy-f2py-1.14.3-9.el8.x86_6 46 MB/s | 225 kB 00:00 2020-11-20T12:01:50.062 INFO:teuthology.orchestra.run.smithi187.stdout:(58/137): python3-prettytable-0.7.2-14.el8.noar 21 MB/s | 44 kB 00:00 2020-11-20T12:01:50.067 INFO:teuthology.orchestra.run.smithi187.stdout:(59/137): python3-pyasn1-0.3.7-6.el8.noarch.rpm 33 MB/s | 126 kB 00:00 2020-11-20T12:01:50.070 INFO:teuthology.orchestra.run.smithi187.stdout:(60/137): python3-pyasn1-modules-0.3.7-6.el8.no 34 MB/s | 110 kB 00:00 2020-11-20T12:01:50.072 INFO:teuthology.orchestra.run.smithi187.stdout:(61/137): python3-pytz-2017.2-9.el8.noarch.rpm 28 MB/s | 54 kB 00:00 2020-11-20T12:01:50.129 INFO:teuthology.orchestra.run.smithi187.stdout:(62/137): ceph-mgr-diskprediction-local-16.0.0- 9.2 MB/s | 8.6 MB 00:00 2020-11-20T12:01:50.142 INFO:teuthology.orchestra.run.smithi187.stdout:(63/137): python3-webencodings-0.5.1-6.el8.noar 2.1 MB/s | 27 kB 00:00 2020-11-20T12:01:50.177 INFO:teuthology.orchestra.run.smithi187.stdout:(64/137): python3-werkzeug-0.12.2-4.el8.noarch. 13 MB/s | 457 kB 00:00 2020-11-20T12:01:50.188 INFO:teuthology.orchestra.run.smithi187.stdout:(65/137): socat-1.7.3.2-6.el8.x86_64.rpm 26 MB/s | 298 kB 00:00 2020-11-20T12:01:50.191 INFO:teuthology.orchestra.run.smithi187.stdout:(66/137): elfutils-libelf-devel-0.176-5.el8.x86 20 MB/s | 54 kB 00:00 2020-11-20T12:01:50.195 INFO:teuthology.orchestra.run.smithi187.stdout:(67/137): fuse-2.9.7-12.el8.x86_64.rpm 25 MB/s | 82 kB 00:00 2020-11-20T12:01:50.198 INFO:teuthology.orchestra.run.smithi187.stdout:(68/137): fuse-common-3.2.1-12.el8.x86_64.rpm 9.4 MB/s | 21 kB 00:00 2020-11-20T12:01:50.201 INFO:teuthology.orchestra.run.smithi187.stdout:(69/137): keyutils-libs-devel-1.5.10-6.el8.x86_ 23 MB/s | 48 kB 00:00 2020-11-20T12:01:50.216 INFO:teuthology.orchestra.run.smithi187.stdout:(70/137): krb5-devel-1.17-9.el8.x86_64.rpm 36 MB/s | 548 kB 00:00 2020-11-20T12:01:50.218 INFO:teuthology.orchestra.run.smithi187.stdout:(71/137): libaio-devel-0.3.112-1.el8.x86_64.rpm 8.9 MB/s | 19 kB 00:00 2020-11-20T12:01:50.224 INFO:teuthology.orchestra.run.smithi187.stdout:(72/137): libcom_err-devel-1.44.6-3.el8.x86_64. 6.3 MB/s | 38 kB 00:00 2020-11-20T12:01:50.241 INFO:teuthology.orchestra.run.smithi187.stdout:(73/137): libgfortran-8.3.1-4.5.el8.x86_64.rpm 38 MB/s | 638 kB 00:00 2020-11-20T12:01:50.247 INFO:teuthology.orchestra.run.smithi187.stdout:(74/137): libquadmath-8.3.1-4.5.el8.x86_64.rpm 34 MB/s | 169 kB 00:00 2020-11-20T12:01:50.249 INFO:teuthology.orchestra.run.smithi187.stdout:(75/137): librabbitmq-0.9.0-1.el8.x86_64.rpm 26 MB/s | 47 kB 00:00 2020-11-20T12:01:50.254 INFO:teuthology.orchestra.run.smithi187.stdout:(76/137): libselinux-devel-2.9-2.1.el8.x86_64.r 35 MB/s | 199 kB 00:00 2020-11-20T12:01:50.258 INFO:teuthology.orchestra.run.smithi187.stdout:(77/137): libsepol-devel-2.9-1.el8.x86_64.rpm 27 MB/s | 86 kB 00:00 2020-11-20T12:01:50.262 INFO:teuthology.orchestra.run.smithi187.stdout:(78/137): libuuid-devel-2.32.1-17.el8.x86_64.rp 25 MB/s | 95 kB 00:00 2020-11-20T12:01:50.264 INFO:teuthology.orchestra.run.smithi187.stdout:(79/137): libverto-devel-0.3.0-5.el8.x86_64.rpm 11 MB/s | 18 kB 00:00 2020-11-20T12:01:50.272 INFO:teuthology.orchestra.run.smithi187.stdout:(80/137): libxslt-1.1.32-3.el8.x86_64.rpm 34 MB/s | 249 kB 00:00 2020-11-20T12:01:50.360 INFO:teuthology.orchestra.run.smithi187.stdout:(81/137): python3-scipy-1.0.0-20.module_el8.1.0 51 MB/s | 15 MB 00:00 2020-11-20T12:01:50.376 INFO:teuthology.orchestra.run.smithi187.stdout:(82/137): pcre2-devel-10.32-1.el8.x86_64.rpm 38 MB/s | 605 kB 00:00 2020-11-20T12:01:50.383 INFO:teuthology.orchestra.run.smithi187.stdout:(83/137): pcre2-utf16-10.32-1.el8.x86_64.rpm 35 MB/s | 228 kB 00:00 2020-11-20T12:01:50.392 INFO:teuthology.orchestra.run.smithi187.stdout:(84/137): openssl-devel-1.1.1c-2.el8_1.1.x86_64 19 MB/s | 2.3 MB 00:00 2020-11-20T12:01:50.397 INFO:teuthology.orchestra.run.smithi187.stdout:(85/137): pcre2-utf32-10.32-1.el8.x86_64.rpm 16 MB/s | 220 kB 00:00 2020-11-20T12:01:50.399 INFO:teuthology.orchestra.run.smithi187.stdout:(86/137): python3-chardet-3.0.4-7.el8.noarch.rp 29 MB/s | 195 kB 00:00 2020-11-20T12:01:50.400 INFO:teuthology.orchestra.run.smithi187.stdout:(87/137): python3-jwt-1.6.1-2.el8.noarch.rpm 16 MB/s | 43 kB 00:00 2020-11-20T12:01:50.402 INFO:teuthology.orchestra.run.smithi187.stdout:(88/137): python3-pysocks-1.6.8-3.el8.noarch.rp 17 MB/s | 34 kB 00:00 2020-11-20T12:01:50.404 INFO:teuthology.orchestra.run.smithi187.stdout:(89/137): python3-oauthlib-2.1.0-1.el8.noarch.r 32 MB/s | 155 kB 00:00 2020-11-20T12:01:50.406 INFO:teuthology.orchestra.run.smithi187.stdout:(90/137): python3-requests-oauthlib-1.0.0-1.el8 22 MB/s | 43 kB 00:00 2020-11-20T12:01:50.407 INFO:teuthology.orchestra.run.smithi187.stdout:(91/137): python3-requests-2.20.0-2.1.el8_1.noa 28 MB/s | 123 kB 00:00 2020-11-20T12:01:50.411 INFO:teuthology.orchestra.run.smithi187.stdout:(92/137): python3-urllib3-1.24.2-2.el8.noarch.r 40 MB/s | 176 kB 00:00 2020-11-20T12:01:50.412 INFO:teuthology.orchestra.run.smithi187.stdout:(93/137): xfsprogs-devel-5.0.0-1.el8.x86_64.rpm 16 MB/s | 69 kB 00:00 2020-11-20T12:01:50.413 INFO:teuthology.orchestra.run.smithi187.stdout:(94/137): zlib-devel-1.2.11-10.el8.x86_64.rpm 24 MB/s | 56 kB 00:00 2020-11-20T12:01:50.496 INFO:teuthology.orchestra.run.smithi187.stdout:(95/137): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2020-11-20T12:01:50.516 INFO:teuthology.orchestra.run.smithi187.stdout:(96/137): gperftools-libs-2.7-6.el8.x86_64.rpm 3.0 MB/s | 315 kB 00:00 2020-11-20T12:01:50.530 INFO:teuthology.orchestra.run.smithi187.stdout:(97/137): leveldb-1.22-1.el8.x86_64.rpm 5.2 MB/s | 181 kB 00:00 2020-11-20T12:01:50.534 INFO:teuthology.orchestra.run.smithi187.stdout:(98/137): liboath-2.6.2-3.el8.x86_64.rpm 3.2 MB/s | 59 kB 00:00 2020-11-20T12:01:50.549 INFO:teuthology.orchestra.run.smithi187.stdout:(99/137): libunwind-1.3.1-3.el8.x86_64.rpm 4.0 MB/s | 75 kB 00:00 2020-11-20T12:01:50.552 INFO:teuthology.orchestra.run.smithi187.stdout:(100/137): python3-apipkg-1.5-6.el8.noarch.rpm 1.2 MB/s | 21 kB 00:00 2020-11-20T12:01:50.568 INFO:teuthology.orchestra.run.smithi187.stdout:(101/137): python3-bcrypt-3.1.6-2.el8.1.x86_64. 2.4 MB/s | 44 kB 00:00 2020-11-20T12:01:50.574 INFO:teuthology.orchestra.run.smithi187.stdout:(102/137): python3-beautifulsoup4-4.6.3-2.el8.1 8.5 MB/s | 185 kB 00:00 2020-11-20T12:01:50.590 INFO:teuthology.orchestra.run.smithi187.stdout:(103/137): python3-cachetools-3.1.1-4.el8.noarc 1.5 MB/s | 33 kB 00:00 2020-11-20T12:01:50.591 INFO:teuthology.orchestra.run.smithi187.stdout:(104/137): python3-certifi-2018.10.15-7.el8.noa 948 kB/s | 16 kB 00:00 2020-11-20T12:01:50.612 INFO:teuthology.orchestra.run.smithi187.stdout:(105/137): python3-cheroot-8.2.1-1.el8.noarch.r 6.5 MB/s | 145 kB 00:00 2020-11-20T12:01:50.628 INFO:teuthology.orchestra.run.smithi187.stdout:(106/137): python3-cherrypy-18.4.0-1.el8.noarch 10 MB/s | 384 kB 00:00 2020-11-20T12:01:50.631 INFO:teuthology.orchestra.run.smithi187.stdout:(107/137): python3-cssselect-0.9.2-10.el8.noarc 2.2 MB/s | 40 kB 00:00 2020-11-20T12:01:50.651 INFO:teuthology.orchestra.run.smithi187.stdout:(108/137): python3-google-auth-1.1.1-10.el8.noa 4.1 MB/s | 81 kB 00:00 2020-11-20T12:01:50.656 INFO:teuthology.orchestra.run.smithi187.stdout:(109/137): python3-execnet-1.7.1-1.el8.noarch.r 13 MB/s | 355 kB 00:00 2020-11-20T12:01:50.668 INFO:teuthology.orchestra.run.smithi187.stdout:(110/137): python3-jaraco-6.2-6.el8.noarch.rpm 651 kB/s | 11 kB 00:00 2020-11-20T12:01:50.674 INFO:teuthology.orchestra.run.smithi187.stdout:(111/137): python3-jaraco-functools-2.0-4.el8.n 1.0 MB/s | 18 kB 00:00 2020-11-20T12:01:50.692 INFO:teuthology.orchestra.run.smithi187.stdout:(112/137): python3-logutils-0.3.5-11.el8.noarch 2.7 MB/s | 49 kB 00:00 2020-11-20T12:01:50.711 INFO:teuthology.orchestra.run.smithi187.stdout:(113/137): python3-more-itertools-7.2.0-3.el8.n 3.1 MB/s | 59 kB 00:00 2020-11-20T12:01:50.734 INFO:teuthology.orchestra.run.smithi187.stdout:(114/137): python3-pecan-1.3.2-9.el8.noarch.rpm 12 MB/s | 283 kB 00:00 2020-11-20T12:01:50.743 INFO:teuthology.orchestra.run.smithi187.stdout:(115/137): python3-kubernetes-11.0.0-2.el8.noar 16 MB/s | 1.2 MB 00:00 2020-11-20T12:01:50.752 INFO:teuthology.orchestra.run.smithi187.stdout:(116/137): python3-portend-2.6-1.el8.noarch.rpm 923 kB/s | 16 kB 00:00 2020-11-20T12:01:50.761 INFO:teuthology.orchestra.run.smithi187.stdout:(117/137): python3-remoto-1.1.4-4.el8.noarch.rp 2.6 MB/s | 48 kB 00:00 2020-11-20T12:01:50.769 INFO:teuthology.orchestra.run.smithi187.stdout:(118/137): python3-repoze-lru-0.7-6.el8.noarch. 1.9 MB/s | 34 kB 00:00 2020-11-20T12:01:50.783 INFO:teuthology.orchestra.run.smithi187.stdout:(119/137): python3-routes-2.4.1-12.el8.noarch.r 9.2 MB/s | 196 kB 00:00 2020-11-20T12:01:50.788 INFO:teuthology.orchestra.run.smithi187.stdout:(120/137): python3-rsa-3.4.2-15.el8.noarch.rpm 3.8 MB/s | 73 kB 00:00 2020-11-20T12:01:50.801 INFO:teuthology.orchestra.run.smithi187.stdout:(121/137): python3-simplegeneric-0.8.1-17.el8.n 1.1 MB/s | 19 kB 00:00 2020-11-20T12:01:50.805 INFO:teuthology.orchestra.run.smithi187.stdout:(122/137): python3-singledispatch-3.4.0.3-18.el 1.4 MB/s | 24 kB 00:00 2020-11-20T12:01:50.818 INFO:teuthology.orchestra.run.smithi187.stdout:(123/137): python3-tempora-1.14.1-5.el8.noarch. 1.7 MB/s | 29 kB 00:00 2020-11-20T12:01:50.823 INFO:teuthology.orchestra.run.smithi187.stdout:(124/137): python3-trustme-0.6.0-4.el8.noarch.r 1.5 MB/s | 27 kB 00:00 2020-11-20T12:01:50.845 INFO:teuthology.orchestra.run.smithi187.stdout:(125/137): python3-waitress-1.2.1-2.el8.1.noarc 9.6 MB/s | 241 kB 00:00 2020-11-20T12:01:50.849 INFO:teuthology.orchestra.run.smithi187.stdout:(126/137): python3-webob-1.8.5-1.el8.1.noarch.r 9.7 MB/s | 251 kB 00:00 2020-11-20T12:01:50.863 INFO:teuthology.orchestra.run.smithi187.stdout:(127/137): python3-websocket-client-0.56.0-5.el 3.3 MB/s | 61 kB 00:00 2020-11-20T12:01:50.869 INFO:teuthology.orchestra.run.smithi187.stdout:(128/137): python3-webtest-2.0.33-1.el8.noarch. 4.3 MB/s | 85 kB 00:00 2020-11-20T12:01:50.882 INFO:teuthology.orchestra.run.smithi187.stdout:(129/137): python3-zc-lockfile-2.0-2.el8.noarch 1.3 MB/s | 23 kB 00:00 2020-11-20T12:01:50.888 INFO:teuthology.orchestra.run.smithi187.stdout:(130/137): xmlstarlet-1.6.1-11.el8.x86_64.rpm 3.7 MB/s | 69 kB 00:00 2020-11-20T12:01:51.518 INFO:teuthology.orchestra.run.smithi187.stdout:(131/137): librados2-16.0.0-7397.gb86e691b.el8. 5.8 MB/s | 3.7 MB 00:00 2020-11-20T12:01:51.533 INFO:teuthology.orchestra.run.smithi187.stdout:(132/137): librbd1-16.0.0-7397.gb86e691b.el8.x8 3.7 MB/s | 2.4 MB 00:00 2020-11-20T12:01:51.573 INFO:teuthology.orchestra.run.smithi187.stdout:(133/137): NetworkManager-1.20.0-5.el8_1.x86_64 39 MB/s | 2.1 MB 00:00 2020-11-20T12:01:51.579 INFO:teuthology.orchestra.run.smithi187.stdout:(134/137): NetworkManager-team-1.20.0-5.el8_1.x 25 MB/s | 133 kB 00:00 2020-11-20T12:01:51.590 INFO:teuthology.orchestra.run.smithi187.stdout:(135/137): NetworkManager-tui-1.20.0-5.el8_1.x8 26 MB/s | 308 kB 00:00 2020-11-20T12:01:51.602 INFO:teuthology.orchestra.run.smithi187.stdout:(136/137): NetworkManager-libnm-1.20.0-5.el8_1. 25 MB/s | 1.7 MB 00:00 2020-11-20T12:01:52.013 INFO:teuthology.orchestra.run.smithi187.stdout:(137/137): ceph-test-16.0.0-7397.gb86e691b.el8. 12 MB/s | 48 MB 00:03 2020-11-20T12:01:52.017 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2020-11-20T12:01:52.017 INFO:teuthology.orchestra.run.smithi187.stdout:Total 29 MB/s | 202 MB 00:06 2020-11-20T12:01:52.124 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2020-11-20T12:01:52.336 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2020-11-20T12:01:52.336 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2020-11-20T12:01:52.506 INFO:teuthology.orchestra.run.smithi005.stdout:(13/137): ceph-radosgw-16.0.0-7397.gb86e691b.el 4.7 MB/s | 12 MB 00:02 2020-11-20T12:01:52.534 INFO:teuthology.orchestra.run.smithi005.stdout:(14/137): ceph-selinux-16.0.0-7397.gb86e691b.el 966 kB/s | 25 kB 00:00 2020-11-20T12:01:52.930 INFO:teuthology.orchestra.run.smithi005.stdout:(15/137): ceph-osd-16.0.0-7397.gb86e691b.el8.x8 4.1 MB/s | 18 MB 00:04 2020-11-20T12:01:52.959 INFO:teuthology.orchestra.run.smithi005.stdout:(16/137): libcephfs-devel-16.0.0-7397.gb86e691b 862 kB/s | 24 kB 00:00 2020-11-20T12:01:53.039 INFO:teuthology.orchestra.run.smithi005.stdout:(17/137): libcephfs2-16.0.0-7397.gb86e691b.el8. 6.8 MB/s | 562 kB 00:00 2020-11-20T12:01:53.069 INFO:teuthology.orchestra.run.smithi005.stdout:(18/137): librados-devel-16.0.0-7397.gb86e691b. 4.4 MB/s | 131 kB 00:00 2020-11-20T12:01:53.147 INFO:teuthology.orchestra.run.smithi005.stdout:(19/137): libradosstriper1-16.0.0-7397.gb86e691 6.1 MB/s | 486 kB 00:00 2020-11-20T12:01:53.600 INFO:teuthology.orchestra.run.smithi005.stdout:(20/137): librgw2-16.0.0-7397.gb86e691b.el8.x86 8.2 MB/s | 3.7 MB 00:00 2020-11-20T12:01:53.630 INFO:teuthology.orchestra.run.smithi005.stdout:(21/137): python3-ceph-argparse-16.0.0-7397.gb8 1.4 MB/s | 40 kB 00:00 2020-11-20T12:01:53.656 INFO:teuthology.orchestra.run.smithi005.stdout:(22/137): python3-ceph-common-16.0.0-7397.gb86e 2.8 MB/s | 73 kB 00:00 2020-11-20T12:01:53.686 INFO:teuthology.orchestra.run.smithi005.stdout:(23/137): python3-cephfs-16.0.0-7397.gb86e691b. 6.8 MB/s | 204 kB 00:00 2020-11-20T12:01:53.808 INFO:teuthology.orchestra.run.smithi005.stdout:(24/137): ceph-common-16.0.0-7397.gb86e691b.el8 2.8 MB/s | 23 MB 00:08 2020-11-20T12:01:53.815 INFO:teuthology.orchestra.run.smithi005.stdout:(25/137): python3-rados-16.0.0-7397.gb86e691b.e 2.8 MB/s | 375 kB 00:00 2020-11-20T12:01:53.841 INFO:teuthology.orchestra.run.smithi005.stdout:(26/137): python3-rgw-16.0.0-7397.gb86e691b.el8 4.4 MB/s | 113 kB 00:00 2020-11-20T12:01:53.861 INFO:teuthology.orchestra.run.smithi005.stdout:(27/137): python3-rbd-16.0.0-7397.gb86e691b.el8 6.6 MB/s | 353 kB 00:00 2020-11-20T12:01:53.866 INFO:teuthology.orchestra.run.smithi005.stdout:(28/137): rbd-fuse-16.0.0-7397.gb86e691b.el8.x8 3.4 MB/s | 88 kB 00:00 2020-11-20T12:01:53.894 INFO:teuthology.orchestra.run.smithi005.stdout:(29/137): rbd-nbd-16.0.0-7397.gb86e691b.el8.x86 6.0 MB/s | 171 kB 00:00 2020-11-20T12:01:53.919 INFO:teuthology.orchestra.run.smithi005.stdout:(30/137): ceph-grafana-dashboards-16.0.0-7397.g 985 kB/s | 24 kB 00:00 2020-11-20T12:01:53.946 INFO:teuthology.orchestra.run.smithi005.stdout:(31/137): ceph-mgr-cephadm-16.0.0-7397.gb86e691 2.5 MB/s | 66 kB 00:00 2020-11-20T12:01:54.284 INFO:teuthology.orchestra.run.smithi005.stdout:(32/137): rbd-mirror-16.0.0-7397.gb86e691b.el8. 7.5 MB/s | 3.2 MB 00:00 2020-11-20T12:01:54.328 INFO:teuthology.orchestra.run.smithi005.stdout:(33/137): ceph-mgr-dashboard-16.0.0-7397.gb86e6 8.3 MB/s | 3.2 MB 00:00 2020-11-20T12:01:54.340 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2020-11-20T12:01:54.341 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2020-11-20T12:01:54.361 INFO:teuthology.orchestra.run.smithi005.stdout:(34/137): ceph-mgr-modules-core-16.0.0-7397.gb8 6.1 MB/s | 198 kB 00:00 2020-11-20T12:01:54.386 INFO:teuthology.orchestra.run.smithi005.stdout:(35/137): ceph-mgr-rook-16.0.0-7397.gb86e691b.e 1.1 MB/s | 27 kB 00:00 2020-11-20T12:01:54.410 INFO:teuthology.orchestra.run.smithi005.stdout:(36/137): ceph-prometheus-alerts-16.0.0-7397.gb 374 kB/s | 8.8 kB 00:00 2020-11-20T12:01:54.437 INFO:teuthology.orchestra.run.smithi005.stdout:(37/137): cephadm-16.0.0-7397.gb86e691b.el8.noa 2.2 MB/s | 57 kB 00:00 2020-11-20T12:01:54.505 INFO:teuthology.orchestra.run.smithi005.stdout:(38/137): automake-1.16.1-6.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2020-11-20T12:01:54.526 INFO:teuthology.orchestra.run.smithi005.stdout:(39/137): bison-3.0.4-10.el8.x86_64.rpm 34 MB/s | 688 kB 00:00 2020-11-20T12:01:54.537 INFO:teuthology.orchestra.run.smithi005.stdout:(40/137): flex-2.6.1-9.el8.x86_64.rpm 29 MB/s | 320 kB 00:00 2020-11-20T12:01:54.542 INFO:teuthology.orchestra.run.smithi005.stdout:(41/137): jq-1.5-12.el8.x86_64.rpm 34 MB/s | 161 kB 00:00 2020-11-20T12:01:54.552 INFO:teuthology.orchestra.run.smithi005.stdout:(42/137): librdkafka-0.11.4-1.el8.x86_64.rpm 35 MB/s | 353 kB 00:00 2020-11-20T12:01:54.570 INFO:teuthology.orchestra.run.smithi005.stdout:(43/137): libtool-2.4.6-25.el8.x86_64.rpm 41 MB/s | 709 kB 00:00 2020-11-20T12:01:54.578 INFO:teuthology.orchestra.run.smithi005.stdout:(44/137): lttng-ust-2.8.1-11.el8.x86_64.rpm 33 MB/s | 259 kB 00:00 2020-11-20T12:01:54.584 INFO:teuthology.orchestra.run.smithi005.stdout:(45/137): oniguruma-6.8.2-1.el8.x86_64.rpm 31 MB/s | 188 kB 00:00 2020-11-20T12:01:54.668 INFO:teuthology.orchestra.run.smithi005.stdout:(46/137): openblas-0.3.3-2.el8.x86_64.rpm 51 MB/s | 4.3 MB 00:00 2020-11-20T12:01:54.748 INFO:teuthology.orchestra.run.smithi005.stdout:(47/137): openblas-threads-0.3.3-2.el8.x86_64.r 56 MB/s | 4.4 MB 00:00 2020-11-20T12:01:54.902 INFO:teuthology.orchestra.run.smithi005.stdout:(48/137): python3-babel-2.5.1-5.el8.noarch.rpm 31 MB/s | 4.8 MB 00:00 2020-11-20T12:01:54.910 INFO:teuthology.orchestra.run.smithi005.stdout:(49/137): python3-html5lib-0.999999999-6.el8.no 28 MB/s | 214 kB 00:00 2020-11-20T12:01:54.921 INFO:teuthology.orchestra.run.smithi005.stdout:(50/137): python3-jinja2-2.10.1-2.el8_0.noarch. 52 MB/s | 538 kB 00:00 2020-11-20T12:01:54.923 INFO:teuthology.orchestra.run.smithi005.stdout:(51/137): python3-jsonpatch-1.21-2.el8.noarch.r 14 MB/s | 27 kB 00:00 2020-11-20T12:01:54.925 INFO:teuthology.orchestra.run.smithi005.stdout:(52/137): python3-jsonpointer-1.10-11.el8.noarc 11 MB/s | 20 kB 00:00 2020-11-20T12:01:54.952 INFO:teuthology.orchestra.run.smithi005.stdout:(53/137): python3-lxml-4.2.3-1.el8.x86_64.rpm 56 MB/s | 1.5 MB 00:00 2020-11-20T12:01:54.958 INFO:teuthology.orchestra.run.smithi005.stdout:(54/137): python3-mako-1.0.6-13.el8.noarch.rpm 29 MB/s | 157 kB 00:00 2020-11-20T12:01:54.961 INFO:teuthology.orchestra.run.smithi005.stdout:(55/137): python3-markupsafe-0.23-19.el8.x86_64 16 MB/s | 39 kB 00:00 2020-11-20T12:01:55.033 INFO:teuthology.orchestra.run.smithi005.stdout:(56/137): python3-numpy-1.14.3-9.el8.x86_64.rpm 52 MB/s | 3.7 MB 00:00 2020-11-20T12:01:55.039 INFO:teuthology.orchestra.run.smithi005.stdout:(57/137): python3-numpy-f2py-1.14.3-9.el8.x86_6 37 MB/s | 225 kB 00:00 2020-11-20T12:01:55.041 INFO:teuthology.orchestra.run.smithi005.stdout:(58/137): python3-prettytable-0.7.2-14.el8.noar 24 MB/s | 44 kB 00:00 2020-11-20T12:01:55.044 INFO:teuthology.orchestra.run.smithi005.stdout:(59/137): python3-pyasn1-0.3.7-6.el8.noarch.rpm 41 MB/s | 126 kB 00:00 2020-11-20T12:01:55.047 INFO:teuthology.orchestra.run.smithi005.stdout:(60/137): python3-pyasn1-modules-0.3.7-6.el8.no 40 MB/s | 110 kB 00:00 2020-11-20T12:01:55.050 INFO:teuthology.orchestra.run.smithi005.stdout:(61/137): python3-pytz-2017.2-9.el8.noarch.rpm 24 MB/s | 54 kB 00:00 2020-11-20T12:01:55.316 INFO:teuthology.orchestra.run.smithi005.stdout:(62/137): python3-scipy-1.0.0-20.module_el8.1.0 55 MB/s | 15 MB 00:00 2020-11-20T12:01:55.318 INFO:teuthology.orchestra.run.smithi005.stdout:(63/137): python3-webencodings-0.5.1-6.el8.noar 12 MB/s | 27 kB 00:00 2020-11-20T12:01:55.439 INFO:teuthology.orchestra.run.smithi005.stdout:(64/137): ceph-mgr-diskprediction-local-16.0.0- 7.4 MB/s | 8.6 MB 00:01 2020-11-20T12:01:55.447 INFO:teuthology.orchestra.run.smithi005.stdout:(65/137): python3-werkzeug-0.12.2-4.el8.noarch. 3.5 MB/s | 457 kB 00:00 2020-11-20T12:01:55.450 INFO:teuthology.orchestra.run.smithi005.stdout:(66/137): elfutils-libelf-devel-0.176-5.el8.x86 22 MB/s | 54 kB 00:00 2020-11-20T12:01:55.453 INFO:teuthology.orchestra.run.smithi005.stdout:(67/137): fuse-2.9.7-12.el8.x86_64.rpm 26 MB/s | 82 kB 00:00 2020-11-20T12:01:55.455 INFO:teuthology.orchestra.run.smithi005.stdout:(68/137): fuse-common-3.2.1-12.el8.x86_64.rpm 13 MB/s | 21 kB 00:00 2020-11-20T12:01:55.457 INFO:teuthology.orchestra.run.smithi005.stdout:(69/137): keyutils-libs-devel-1.5.10-6.el8.x86_ 24 MB/s | 48 kB 00:00 2020-11-20T12:01:55.470 INFO:teuthology.orchestra.run.smithi005.stdout:(70/137): krb5-devel-1.17-9.el8.x86_64.rpm 42 MB/s | 548 kB 00:00 2020-11-20T12:01:55.472 INFO:teuthology.orchestra.run.smithi005.stdout:(71/137): libaio-devel-0.3.112-1.el8.x86_64.rpm 12 MB/s | 19 kB 00:00 2020-11-20T12:01:55.475 INFO:teuthology.orchestra.run.smithi005.stdout:(72/137): libcom_err-devel-1.44.6-3.el8.x86_64. 17 MB/s | 38 kB 00:00 2020-11-20T12:01:55.477 INFO:teuthology.orchestra.run.smithi005.stdout:(73/137): socat-1.7.3.2-6.el8.x86_64.rpm 7.7 MB/s | 298 kB 00:00 2020-11-20T12:01:55.485 INFO:teuthology.orchestra.run.smithi005.stdout:(74/137): libquadmath-8.3.1-4.5.el8.x86_64.rpm 22 MB/s | 169 kB 00:00 2020-11-20T12:01:55.488 INFO:teuthology.orchestra.run.smithi005.stdout:(75/137): librabbitmq-0.9.0-1.el8.x86_64.rpm 19 MB/s | 47 kB 00:00 2020-11-20T12:01:55.491 INFO:teuthology.orchestra.run.smithi005.stdout:(76/137): libgfortran-8.3.1-4.5.el8.x86_64.rpm 39 MB/s | 638 kB 00:00 2020-11-20T12:01:55.495 INFO:teuthology.orchestra.run.smithi005.stdout:(77/137): libsepol-devel-2.9-1.el8.x86_64.rpm 26 MB/s | 86 kB 00:00 2020-11-20T12:01:55.496 INFO:teuthology.orchestra.run.smithi005.stdout:(78/137): libselinux-devel-2.9-2.1.el8.x86_64.r 24 MB/s | 199 kB 00:00 2020-11-20T12:01:55.498 INFO:teuthology.orchestra.run.smithi005.stdout:(79/137): libuuid-devel-2.32.1-17.el8.x86_64.rp 28 MB/s | 95 kB 00:00 2020-11-20T12:01:55.499 INFO:teuthology.orchestra.run.smithi005.stdout:(80/137): libverto-devel-0.3.0-5.el8.x86_64.rpm 7.7 MB/s | 18 kB 00:00 2020-11-20T12:01:55.504 INFO:teuthology.orchestra.run.smithi005.stdout:(81/137): libxslt-1.1.32-3.el8.x86_64.rpm 42 MB/s | 249 kB 00:00 2020-11-20T12:01:55.522 INFO:teuthology.orchestra.run.smithi005.stdout:(82/137): pcre2-devel-10.32-1.el8.x86_64.rpm 35 MB/s | 605 kB 00:00 2020-11-20T12:01:55.529 INFO:teuthology.orchestra.run.smithi005.stdout:(83/137): pcre2-utf16-10.32-1.el8.x86_64.rpm 34 MB/s | 228 kB 00:00 2020-11-20T12:01:55.538 INFO:teuthology.orchestra.run.smithi005.stdout:(84/137): pcre2-utf32-10.32-1.el8.x86_64.rpm 24 MB/s | 220 kB 00:00 2020-11-20T12:01:55.544 INFO:teuthology.orchestra.run.smithi005.stdout:(85/137): python3-chardet-3.0.4-7.el8.noarch.rp 37 MB/s | 195 kB 00:00 2020-11-20T12:01:55.545 INFO:teuthology.orchestra.run.smithi005.stdout:(86/137): python3-jwt-1.6.1-2.el8.noarch.rpm 24 MB/s | 43 kB 00:00 2020-11-20T12:01:55.551 INFO:teuthology.orchestra.run.smithi005.stdout:(87/137): python3-oauthlib-2.1.0-1.el8.noarch.r 39 MB/s | 155 kB 00:00 2020-11-20T12:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout:(88/137): openssl-devel-1.1.1c-2.el8_1.1.x86_64 38 MB/s | 2.3 MB 00:00 2020-11-20T12:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout:(89/137): python3-pysocks-1.6.8-3.el8.noarch.rp 3.2 MB/s | 34 kB 00:00 2020-11-20T12:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout:(90/137): python3-requests-oauthlib-1.0.0-1.el8 18 MB/s | 43 kB 00:00 2020-11-20T12:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout:(91/137): python3-requests-2.20.0-2.1.el8_1.noa 22 MB/s | 123 kB 00:00 2020-11-20T12:01:55.569 INFO:teuthology.orchestra.run.smithi005.stdout:(92/137): python3-urllib3-1.24.2-2.el8.noarch.r 33 MB/s | 176 kB 00:00 2020-11-20T12:01:55.569 INFO:teuthology.orchestra.run.smithi005.stdout:(93/137): xfsprogs-devel-5.0.0-1.el8.x86_64.rpm 16 MB/s | 69 kB 00:00 2020-11-20T12:01:55.571 INFO:teuthology.orchestra.run.smithi005.stdout:(94/137): zlib-devel-1.2.11-10.el8.x86_64.rpm 30 MB/s | 56 kB 00:00 2020-11-20T12:01:55.654 INFO:teuthology.orchestra.run.smithi005.stdout:(95/137): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2020-11-20T12:01:55.689 INFO:teuthology.orchestra.run.smithi005.stdout:(96/137): leveldb-1.22-1.el8.x86_64.rpm 5.1 MB/s | 181 kB 00:00 2020-11-20T12:01:55.695 INFO:teuthology.orchestra.run.smithi005.stdout:(97/137): gperftools-libs-2.7-6.el8.x86_64.rpm 2.5 MB/s | 315 kB 00:00 2020-11-20T12:01:55.709 INFO:teuthology.orchestra.run.smithi005.stdout:(98/137): liboath-2.6.2-3.el8.x86_64.rpm 3.1 MB/s | 59 kB 00:00 2020-11-20T12:01:55.717 INFO:teuthology.orchestra.run.smithi005.stdout:(99/137): libunwind-1.3.1-3.el8.x86_64.rpm 3.3 MB/s | 75 kB 00:00 2020-11-20T12:01:55.728 INFO:teuthology.orchestra.run.smithi005.stdout:(100/137): python3-apipkg-1.5-6.el8.noarch.rpm 1.1 MB/s | 21 kB 00:00 2020-11-20T12:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout:(101/137): python3-bcrypt-3.1.6-2.el8.1.x86_64. 1.9 MB/s | 44 kB 00:00 2020-11-20T12:01:55.754 INFO:teuthology.orchestra.run.smithi005.stdout:(102/137): python3-beautifulsoup4-4.6.3-2.el8.1 7.2 MB/s | 185 kB 00:00 2020-11-20T12:01:55.762 INFO:teuthology.orchestra.run.smithi005.stdout:(103/137): python3-cachetools-3.1.1-4.el8.noarc 1.5 MB/s | 33 kB 00:00 2020-11-20T12:01:55.773 INFO:teuthology.orchestra.run.smithi005.stdout:(104/137): python3-certifi-2018.10.15-7.el8.noa 887 kB/s | 16 kB 00:00 2020-11-20T12:01:55.787 INFO:teuthology.orchestra.run.smithi005.stdout:(105/137): python3-cheroot-8.2.1-1.el8.noarch.r 5.7 MB/s | 145 kB 00:00 2020-11-20T12:01:55.811 INFO:teuthology.orchestra.run.smithi005.stdout:(106/137): python3-cherrypy-18.4.0-1.el8.noarch 10 MB/s | 384 kB 00:00 2020-11-20T12:01:55.813 INFO:teuthology.orchestra.run.smithi005.stdout:(107/137): python3-cssselect-0.9.2-10.el8.noarc 1.6 MB/s | 40 kB 00:00 2020-11-20T12:01:55.836 INFO:teuthology.orchestra.run.smithi005.stdout:(108/137): python3-google-auth-1.1.1-10.el8.noa 3.4 MB/s | 81 kB 00:00 2020-11-20T12:01:55.841 INFO:teuthology.orchestra.run.smithi005.stdout:(109/137): python3-execnet-1.7.1-1.el8.noarch.r 12 MB/s | 355 kB 00:00 2020-11-20T12:01:55.854 INFO:teuthology.orchestra.run.smithi005.stdout:(110/137): python3-jaraco-6.2-6.el8.noarch.rpm 653 kB/s | 11 kB 00:00 2020-11-20T12:01:55.862 INFO:teuthology.orchestra.run.smithi005.stdout:(111/137): python3-jaraco-functools-2.0-4.el8.n 850 kB/s | 18 kB 00:00 2020-11-20T12:01:55.884 INFO:teuthology.orchestra.run.smithi005.stdout:(112/137): python3-logutils-0.3.5-11.el8.noarch 2.2 MB/s | 49 kB 00:00 2020-11-20T12:01:55.907 INFO:teuthology.orchestra.run.smithi005.stdout:(113/137): python3-more-itertools-7.2.0-3.el8.n 2.6 MB/s | 59 kB 00:00 2020-11-20T12:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout:(114/137): python3-kubernetes-11.0.0-2.el8.noar 18 MB/s | 1.2 MB 00:00 2020-11-20T12:01:55.934 INFO:teuthology.orchestra.run.smithi005.stdout:(115/137): python3-pecan-1.3.2-9.el8.noarch.rpm 11 MB/s | 283 kB 00:00 2020-11-20T12:01:55.936 INFO:teuthology.orchestra.run.smithi005.stdout:(116/137): python3-portend-2.6-1.el8.noarch.rpm 923 kB/s | 16 kB 00:00 2020-11-20T12:01:55.955 INFO:teuthology.orchestra.run.smithi005.stdout:(117/137): python3-repoze-lru-0.7-6.el8.noarch. 1.8 MB/s | 34 kB 00:00 2020-11-20T12:01:55.958 INFO:teuthology.orchestra.run.smithi005.stdout:(118/137): python3-remoto-1.1.4-4.el8.noarch.rp 2.1 MB/s | 48 kB 00:00 2020-11-20T12:01:55.978 INFO:teuthology.orchestra.run.smithi005.stdout:(119/137): python3-routes-2.4.1-12.el8.noarch.r 8.7 MB/s | 196 kB 00:00 2020-11-20T12:01:55.981 INFO:teuthology.orchestra.run.smithi005.stdout:(120/137): python3-rsa-3.4.2-15.el8.noarch.rpm 3.1 MB/s | 73 kB 00:00 2020-11-20T12:01:55.996 INFO:teuthology.orchestra.run.smithi005.stdout:(121/137): python3-simplegeneric-0.8.1-17.el8.n 1.1 MB/s | 19 kB 00:00 2020-11-20T12:01:56.002 INFO:teuthology.orchestra.run.smithi005.stdout:(122/137): python3-singledispatch-3.4.0.3-18.el 1.1 MB/s | 24 kB 00:00 2020-11-20T12:01:56.015 INFO:teuthology.orchestra.run.smithi005.stdout:(123/137): python3-tempora-1.14.1-5.el8.noarch. 1.5 MB/s | 29 kB 00:00 2020-11-20T12:01:56.024 INFO:teuthology.orchestra.run.smithi005.stdout:(124/137): python3-trustme-0.6.0-4.el8.noarch.r 1.3 MB/s | 27 kB 00:00 2020-11-20T12:01:56.043 INFO:teuthology.orchestra.run.smithi005.stdout:(125/137): python3-waitress-1.2.1-2.el8.1.noarc 8.6 MB/s | 241 kB 00:00 2020-11-20T12:01:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:(126/137): python3-webob-1.8.5-1.el8.1.noarch.r 8.8 MB/s | 251 kB 00:00 2020-11-20T12:01:56.063 INFO:teuthology.orchestra.run.smithi005.stdout:(127/137): python3-websocket-client-0.56.0-5.el 3.1 MB/s | 61 kB 00:00 2020-11-20T12:01:56.075 INFO:teuthology.orchestra.run.smithi005.stdout:(128/137): python3-webtest-2.0.33-1.el8.noarch. 3.7 MB/s | 85 kB 00:00 2020-11-20T12:01:56.082 INFO:teuthology.orchestra.run.smithi005.stdout:(129/137): python3-zc-lockfile-2.0-2.el8.noarch 1.3 MB/s | 23 kB 00:00 2020-11-20T12:01:56.098 INFO:teuthology.orchestra.run.smithi005.stdout:(130/137): xmlstarlet-1.6.1-11.el8.x86_64.rpm 3.1 MB/s | 69 kB 00:00 2020-11-20T12:01:56.432 INFO:teuthology.orchestra.run.smithi005.stdout:(131/137): librbd1-16.0.0-7397.gb86e691b.el8.x8 7.2 MB/s | 2.4 MB 00:00 2020-11-20T12:01:56.479 INFO:teuthology.orchestra.run.smithi005.stdout:(132/137): NetworkManager-1.20.0-5.el8_1.x86_64 46 MB/s | 2.1 MB 00:00 2020-11-20T12:01:56.515 INFO:teuthology.orchestra.run.smithi005.stdout:(133/137): NetworkManager-libnm-1.20.0-5.el8_1. 51 MB/s | 1.7 MB 00:00 2020-11-20T12:01:56.522 INFO:teuthology.orchestra.run.smithi005.stdout:(134/137): NetworkManager-team-1.20.0-5.el8_1.x 31 MB/s | 133 kB 00:00 2020-11-20T12:01:56.524 INFO:teuthology.orchestra.run.smithi005.stdout:(135/137): NetworkManager-tui-1.20.0-5.el8_1.x8 53 MB/s | 308 kB 00:00 2020-11-20T12:01:56.594 INFO:teuthology.orchestra.run.smithi005.stdout:(136/137): librados2-16.0.0-7397.gb86e691b.el8. 7.3 MB/s | 3.7 MB 00:00 2020-11-20T12:01:57.368 INFO:teuthology.orchestra.run.smithi005.stdout:(137/137): ceph-test-16.0.0-7397.gb86e691b.el8. 9.9 MB/s | 48 MB 00:04 2020-11-20T12:01:57.373 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2020-11-20T12:01:57.373 INFO:teuthology.orchestra.run.smithi005.stdout:Total 17 MB/s | 202 MB 00:11 2020-11-20T12:01:57.486 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2020-11-20T12:01:57.708 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2020-11-20T12:01:57.709 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2020-11-20T12:01:59.637 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2020-11-20T12:01:59.769 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2020-11-20T12:01:59.770 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2020-11-20T12:01:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 1/1 2020-11-20T12:02:00.083 INFO:teuthology.orchestra.run.smithi187.stdout: Downgrading : openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 1/148 2020-11-20T12:02:00.206 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 1/148 2020-11-20T12:02:00.420 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fmt-6.2.1-1.el8.x86_64 2/148 2020-11-20T12:02:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/148 2020-11-20T12:02:00.894 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : leveldb-1.22-1.el8.x86_64 4/148 2020-11-20T12:02:01.089 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 5/148 2020-11-20T12:02:01.215 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 5/148 2020-11-20T12:02:01.465 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 6/148 2020-11-20T12:02:01.681 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 7/148 2020-11-20T12:02:02.249 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 7/148 2020-11-20T12:02:02.472 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 8/148 2020-11-20T12:02:02.847 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 8/148 2020-11-20T12:02:03.114 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 9/148 2020-11-20T12:02:03.309 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 9/148 2020-11-20T12:02:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 10/148 2020-11-20T12:02:03.643 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 10/148 2020-11-20T12:02:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 11/148 2020-11-20T12:02:04.455 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 12/148 2020-11-20T12:02:04.752 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 12/148 2020-11-20T12:02:05.027 INFO:teuthology.orchestra.run.smithi187.stdout: Downgrading : krb5-libs-1.17-9.el8.x86_64 13/148 2020-11-20T12:02:05.270 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2020-11-20T12:02:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 14/148 2020-11-20T12:02:05.511 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 1/1 2020-11-20T12:02:05.605 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 15/148 2020-11-20T12:02:05.717 INFO:teuthology.orchestra.run.smithi005.stdout: Downgrading : openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 1/148 2020-11-20T12:02:05.867 INFO:teuthology.orchestra.run.smithi187.stdout: Downgrading : libkadm5-1.17-9.el8.x86_64 16/148 2020-11-20T12:02:05.901 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 1/148 2020-11-20T12:02:06.038 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 17/148 2020-11-20T12:02:06.055 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fmt-6.2.1-1.el8.x86_64 2/148 2020-11-20T12:02:06.280 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/148 2020-11-20T12:02:06.537 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : leveldb-1.22-1.el8.x86_64 4/148 2020-11-20T12:02:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 17/148 2020-11-20T12:02:06.704 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 5/148 2020-11-20T12:02:06.818 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 5/148 2020-11-20T12:02:06.888 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64 18/148 2020-11-20T12:02:06.989 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64 18/148 2020-11-20T12:02:07.782 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 6/148 2020-11-20T12:02:07.782 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 7/148 2020-11-20T12:02:07.783 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-1:1.20.0-5.el8_1.x86_64 19/148 2020-11-20T12:02:07.783 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : NetworkManager-1:1.20.0-5.el8_1.x86_64 19/148 2020-11-20T12:02:07.883 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 7/148 2020-11-20T12:02:07.901 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-1:1.20.0-5.el8_1.x86_64 19/148 2020-11-20T12:02:08.029 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 8/148 2020-11-20T12:02:08.183 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 20/148 2020-11-20T12:02:08.421 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : zlib-devel-1.2.11-10.el8.x86_64 21/148 2020-11-20T12:02:08.499 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 8/148 2020-11-20T12:02:08.645 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 22/148 2020-11-20T12:02:08.745 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 9/148 2020-11-20T12:02:08.956 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 9/148 2020-11-20T12:02:09.095 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 10/148 2020-11-20T12:02:09.280 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 10/148 2020-11-20T12:02:09.343 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 23/148 2020-11-20T12:02:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 24/148 2020-11-20T12:02:09.775 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 24/148 2020-11-20T12:02:09.911 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 11/148 2020-11-20T12:02:09.929 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 25/148 2020-11-20T12:02:10.045 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 12/148 2020-11-20T12:02:10.350 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 12/148 2020-11-20T12:02:10.704 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 25/148 2020-11-20T12:02:10.761 INFO:teuthology.orchestra.run.smithi005.stdout: Downgrading : krb5-libs-1.17-9.el8.x86_64 13/148 2020-11-20T12:02:10.879 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.3-2.el8.x86_64 26/148 2020-11-20T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 14/148 2020-11-20T12:02:11.303 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 15/148 2020-11-20T12:02:11.553 INFO:teuthology.orchestra.run.smithi005.stdout: Downgrading : libkadm5-1.17-9.el8.x86_64 16/148 2020-11-20T12:02:11.641 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 26/148 2020-11-20T12:02:11.687 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 17/148 2020-11-20T12:02:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 27/148 2020-11-20T12:02:12.320 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 17/148 2020-11-20T12:02:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64 18/148 2020-11-20T12:02:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64 18/148 2020-11-20T12:02:12.749 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 27/148 2020-11-20T12:02:13.167 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-1:1.20.0-5.el8_1.x86_64 19/148 2020-11-20T12:02:13.191 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 28/148 2020-11-20T12:02:13.422 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : NetworkManager-1:1.20.0-5.el8_1.x86_64 19/148 2020-11-20T12:02:13.473 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/148 2020-11-20T12:02:13.656 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-1:1.20.0-5.el8_1.x86_64 19/148 2020-11-20T12:02:13.725 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 30/148 2020-11-20T12:02:13.941 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 20/148 2020-11-20T12:02:13.987 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 31/148 2020-11-20T12:02:14.164 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : zlib-devel-1.2.11-10.el8.x86_64 21/148 2020-11-20T12:02:14.221 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:16.0.0-7397.gb86e691b.el8. 32/148 2020-11-20T12:02:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 22/148 2020-11-20T12:02:14.540 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el 33/148 2020-11-20T12:02:14.676 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 23/148 2020-11-20T12:02:14.811 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 24/148 2020-11-20T12:02:14.981 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_6 34/148 2020-11-20T12:02:15.069 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 24/148 2020-11-20T12:02:15.253 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 25/148 2020-11-20T12:02:16.030 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 25/148 2020-11-20T12:02:16.214 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 35/148 2020-11-20T12:02:16.261 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-0.3.3-2.el8.x86_64 26/148 2020-11-20T12:02:16.646 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 36/148 2020-11-20T12:02:17.001 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 37/148 2020-11-20T12:02:17.053 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 26/148 2020-11-20T12:02:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 27/148 2020-11-20T12:02:17.211 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 38/148 2020-11-20T12:02:18.226 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 27/148 2020-11-20T12:02:18.642 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 28/148 2020-11-20T12:02:18.916 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/148 2020-11-20T12:02:19.126 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 30/148 2020-11-20T12:02:19.417 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 31/148 2020-11-20T12:02:19.629 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-common-2:16.0.0-7397.gb86e691b.el8. 32/148 2020-11-20T12:02:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 38/148 2020-11-20T12:02:19.804 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el 33/148 2020-11-20T12:02:20.103 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_6 34/148 2020-11-20T12:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af4 39/148 2020-11-20T12:02:20.434 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 40/148 2020-11-20T12:02:20.745 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 41/148 2020-11-20T12:02:21.078 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 42/148 2020-11-20T12:02:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-3.4.2-15.el8.noarch 43/148 2020-11-20T12:02:21.379 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 35/148 2020-11-20T12:02:21.599 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 44/148 2020-11-20T12:02:21.809 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 36/148 2020-11-20T12:02:21.889 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 45/148 2020-11-20T12:02:22.146 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_6 46/148 2020-11-20T12:02:22.183 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 37/148 2020-11-20T12:02:22.375 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 47/148 2020-11-20T12:02:22.406 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 38/148 2020-11-20T12:02:22.629 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/148 2020-11-20T12:02:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/148 2020-11-20T12:02:23.208 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 50/148 2020-11-20T12:02:23.410 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/148 2020-11-20T12:02:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/148 2020-11-20T12:02:23.799 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/148 2020-11-20T12:02:24.043 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/148 2020-11-20T12:02:24.316 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/148 2020-11-20T12:02:24.488 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/148 2020-11-20T12:02:24.706 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/148 2020-11-20T12:02:24.973 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/148 2020-11-20T12:02:25.108 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 38/148 2020-11-20T12:02:25.218 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 59/148 2020-11-20T12:02:25.422 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/148 2020-11-20T12:02:25.483 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af4 39/148 2020-11-20T12:02:25.742 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/148 2020-11-20T12:02:25.791 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 40/148 2020-11-20T12:02:26.009 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 62/148 2020-11-20T12:02:26.217 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 41/148 2020-11-20T12:02:26.254 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 63/148 2020-11-20T12:02:26.463 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 64/148 2020-11-20T12:02:26.533 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 42/148 2020-11-20T12:02:26.772 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 65/148 2020-11-20T12:02:26.777 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rsa-3.4.2-15.el8.noarch 43/148 2020-11-20T12:02:26.987 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 66/148 2020-11-20T12:02:27.068 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 44/148 2020-11-20T12:02:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/148 2020-11-20T12:02:27.322 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 45/148 2020-11-20T12:02:27.563 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 68/148 2020-11-20T12:02:27.567 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_6 46/148 2020-11-20T12:02:27.750 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 47/148 2020-11-20T12:02:27.932 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 69/148 2020-11-20T12:02:27.972 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/148 2020-11-20T12:02:28.143 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 70/148 2020-11-20T12:02:28.298 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/148 2020-11-20T12:02:28.430 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 71/148 2020-11-20T12:02:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 50/148 2020-11-20T12:02:28.670 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 72/148 2020-11-20T12:02:28.752 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/148 2020-11-20T12:02:28.901 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/148 2020-11-20T12:02:28.947 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 73/148 2020-11-20T12:02:29.116 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/148 2020-11-20T12:02:29.265 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 74/148 2020-11-20T12:02:29.388 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/148 2020-11-20T12:02:29.556 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 75/148 2020-11-20T12:02:29.650 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/148 2020-11-20T12:02:29.765 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 76/148 2020-11-20T12:02:29.862 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/148 2020-11-20T12:02:30.016 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/148 2020-11-20T12:02:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/148 2020-11-20T12:02:30.536 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 59/148 2020-11-20T12:02:30.721 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/148 2020-11-20T12:02:31.015 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/148 2020-11-20T12:02:31.242 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/148 2020-11-20T12:02:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 62/148 2020-11-20T12:02:31.504 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 63/148 2020-11-20T12:02:31.564 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-11.0.0-2.el8.noarch 78/148 2020-11-20T12:02:31.723 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 64/148 2020-11-20T12:02:31.806 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-utf32-10.32-1.el8.x86_64 79/148 2020-11-20T12:02:32.018 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 65/148 2020-11-20T12:02:32.216 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-utf16-10.32-1.el8.x86_64 80/148 2020-11-20T12:02:32.237 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 66/148 2020-11-20T12:02:32.425 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-devel-10.32-1.el8.x86_64 81/148 2020-11-20T12:02:32.476 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/148 2020-11-20T12:02:32.627 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 82/148 2020-11-20T12:02:32.815 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 68/148 2020-11-20T12:02:32.864 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libuuid-devel-2.32.1-17.el8.x86_64 83/148 2020-11-20T12:02:33.153 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 69/148 2020-11-20T12:02:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libsepol-devel-2.9-1.el8.x86_64 84/148 2020-11-20T12:02:33.384 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 70/148 2020-11-20T12:02:33.457 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libselinux-devel-2.9-2.1.el8.x86_64 85/148 2020-11-20T12:02:33.628 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 71/148 2020-11-20T12:02:33.656 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcom_err-devel-1.44.6-3.el8.x86_64 86/148 2020-11-20T12:02:33.875 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 72/148 2020-11-20T12:02:33.930 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 87/148 2020-11-20T12:02:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : krb5-devel-1.17-9.el8.x86_64 88/148 2020-11-20T12:02:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 89/148 2020-11-20T12:02:34.158 INFO:teuthology.orchestra.run.smithi187.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-11-20T12:02:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:02:34.169 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 73/148 2020-11-20T12:02:34.453 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 74/148 2020-11-20T12:02:34.584 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.7-12.el8.x86_64 90/148 2020-11-20T12:02:34.725 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 75/148 2020-11-20T12:02:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 91/148 2020-11-20T12:02:34.960 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 76/148 2020-11-20T12:02:35.443 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 92/148 2020-11-20T12:02:35.955 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 93/148 2020-11-20T12:02:36.298 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 94/148 2020-11-20T12:02:36.430 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/148 2020-11-20T12:02:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 95/148 2020-11-20T12:02:36.791 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-kubernetes-11.0.0-2.el8.noarch 78/148 2020-11-20T12:02:37.072 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 96/148 2020-11-20T12:02:37.074 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : pcre2-utf32-10.32-1.el8.x86_64 79/148 2020-11-20T12:02:37.250 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 97/148 2020-11-20T12:02:37.540 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : pcre2-utf16-10.32-1.el8.x86_64 80/148 2020-11-20T12:02:37.724 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : pcre2-devel-10.32-1.el8.x86_64 81/148 2020-11-20T12:02:37.935 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 82/148 2020-11-20T12:02:38.166 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libuuid-devel-2.32.1-17.el8.x86_64 83/148 2020-11-20T12:02:38.570 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libsepol-devel-2.9-1.el8.x86_64 84/148 2020-11-20T12:02:38.824 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libselinux-devel-2.9-2.1.el8.x86_64 85/148 2020-11-20T12:02:39.032 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcom_err-devel-1.44.6-3.el8.x86_64 86/148 2020-11-20T12:02:39.297 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 87/148 2020-11-20T12:02:39.549 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : krb5-devel-1.17-9.el8.x86_64 88/148 2020-11-20T12:02:39.550 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 89/148 2020-11-20T12:02:39.550 INFO:teuthology.orchestra.run.smithi005.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-11-20T12:02:39.550 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:02:40.134 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fuse-2.9.7-12.el8.x86_64 90/148 2020-11-20T12:02:40.443 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 91/148 2020-11-20T12:02:40.755 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 92/148 2020-11-20T12:02:41.264 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 93/148 2020-11-20T12:02:41.631 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 94/148 2020-11-20T12:02:42.372 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 95/148 2020-11-20T12:02:42.714 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 96/148 2020-11-20T12:02:42.900 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 97/148 2020-11-20T12:02:43.456 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 98/148 2020-11-20T12:02:43.692 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 98/148 2020-11-20T12:02:44.622 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 98/148 2020-11-20T12:02:44.918 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 99/148 2020-11-20T12:02:45.138 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 99/148 2020-11-20T12:02:45.281 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 100/148 2020-11-20T12:02:49.438 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 98/148 2020-11-20T12:02:49.658 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 98/148 2020-11-20T12:02:50.647 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 98/148 2020-11-20T12:02:51.086 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 99/148 2020-11-20T12:02:51.257 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 99/148 2020-11-20T12:02:51.414 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 100/148 2020-11-20T12:03:03.166 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 100/148 2020-11-20T12:03:03.166 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2020-11-20T12:03:03.166 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2020-11-20T12:03:03.166 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2020-11-20T12:03:03.167 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /run 2020-11-20T12:03:03.167 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2020-11-20T12:03:03.167 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2020-11-20T12:03:03.167 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2020-11-20T12:03:03.168 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2020-11-20T12:03:03.168 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2020-11-20T12:03:03.168 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:03.604 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 101/148 2020-11-20T12:03:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 101/148 2020-11-20T12:03:04.448 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 102/148 2020-11-20T12:03:06.265 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 102/148 2020-11-20T12:03:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 103/148 2020-11-20T12:03:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 103/148 2020-11-20T12:03:06.872 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 104/148 2020-11-20T12:03:07.134 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 105/148 2020-11-20T12:03:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 106/148 2020-11-20T12:03:07.470 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 106/148 2020-11-20T12:03:08.089 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : jq-1.5-12.el8.x86_64 107/148 2020-11-20T12:03:08.328 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : automake-1.16.1-6.el8.noarch 108/148 2020-11-20T12:03:10.091 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 100/148 2020-11-20T12:03:10.092 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /sys 2020-11-20T12:03:10.092 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /proc 2020-11-20T12:03:10.092 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /dev 2020-11-20T12:03:10.092 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /run 2020-11-20T12:03:10.093 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /mnt 2020-11-20T12:03:10.093 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /var/tmp 2020-11-20T12:03:10.093 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /home 2020-11-20T12:03:10.093 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /tmp 2020-11-20T12:03:10.094 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /dev 2020-11-20T12:03:10.094 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:10.563 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 101/148 2020-11-20T12:03:11.030 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 109/148 2020-11-20T12:03:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 101/148 2020-11-20T12:03:11.225 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 109/148 2020-11-20T12:03:11.379 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.e 110/148 2020-11-20T12:03:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 102/148 2020-11-20T12:03:11.632 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b. 111/148 2020-11-20T12:03:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 112/148 2020-11-20T12:03:12.991 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 112/148 2020-11-20T12:03:13.347 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 102/148 2020-11-20T12:03:13.575 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 103/148 2020-11-20T12:03:13.686 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 103/148 2020-11-20T12:03:13.882 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 104/148 2020-11-20T12:03:14.161 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 105/148 2020-11-20T12:03:14.336 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 106/148 2020-11-20T12:03:14.487 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 113/148 2020-11-20T12:03:14.699 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 106/148 2020-11-20T12:03:15.088 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : jq-1.5-12.el8.x86_64 107/148 2020-11-20T12:03:15.345 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : automake-1.16.1-6.el8.noarch 108/148 2020-11-20T12:03:15.833 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 113/148 2020-11-20T12:03:16.045 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 114/148 2020-11-20T12:03:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 114/148 2020-11-20T12:03:16.679 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el 115/148 2020-11-20T12:03:16.896 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 116/148 2020-11-20T12:03:16.995 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 116/148 2020-11-20T12:03:17.158 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 117/148 2020-11-20T12:03:17.208 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 117/148 2020-11-20T12:03:17.479 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 118/148 2020-11-20T12:03:17.633 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/148 2020-11-20T12:03:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 109/148 2020-11-20T12:03:18.247 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 109/148 2020-11-20T12:03:18.414 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.e 110/148 2020-11-20T12:03:18.634 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b. 111/148 2020-11-20T12:03:18.885 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 112/148 2020-11-20T12:03:20.138 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 112/148 2020-11-20T12:03:20.915 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 113/148 2020-11-20T12:03:22.337 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 113/148 2020-11-20T12:03:22.538 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 114/148 2020-11-20T12:03:22.690 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/148 2020-11-20T12:03:22.738 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 114/148 2020-11-20T12:03:23.215 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el 115/148 2020-11-20T12:03:23.408 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 116/148 2020-11-20T12:03:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 116/148 2020-11-20T12:03:23.668 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 117/148 2020-11-20T12:03:23.720 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 117/148 2020-11-20T12:03:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 118/148 2020-11-20T12:03:24.159 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/148 2020-11-20T12:03:24.202 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 120/148 2020-11-20T12:03:24.426 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 121/148 2020-11-20T12:03:24.945 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 121/148 2020-11-20T12:03:25.167 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 122/148 2020-11-20T12:03:25.350 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 122/148 2020-11-20T12:03:26.772 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 123/148 2020-11-20T12:03:27.066 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openssl-devel-1:1.1.1c-2.el8_1.1.x86_64 124/148 2020-11-20T12:03:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xfsprogs-devel-5.0.0-1.el8.x86_64 125/148 2020-11-20T12:03:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_ 126/148 2020-11-20T12:03:27.660 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : elfutils-libelf-devel-0.176-5.el8.x86_64 127/148 2020-11-20T12:03:27.933 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : NetworkManager-team-1:1.20.0-5.el8_1.x86_64 128/148 2020-11-20T12:03:28.539 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : NetworkManager-tui-1:1.20.0-5.el8_1.x86_64 129/148 2020-11-20T12:03:28.928 INFO:teuthology.orchestra.run.smithi187.stdout: Downgrading : krb5-workstation-1.17-9.el8.x86_64 130/148 2020-11-20T12:03:29.153 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 131/148 2020-11-20T12:03:29.435 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 132/148 2020-11-20T12:03:29.524 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/148 2020-11-20T12:03:29.666 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 133/148 2020-11-20T12:03:29.917 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 133/148 2020-11-20T12:03:30.169 INFO:teuthology.orchestra.run.smithi187.stdout: Downgrading : openssl-1:1.1.1c-2.el8_1.1.x86_64 134/148 2020-11-20T12:03:30.422 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 135/148 2020-11-20T12:03:30.673 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flex-2.6.1-9.el8.x86_64 136/148 2020-11-20T12:03:30.853 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 136/148 2020-11-20T12:03:31.056 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : bison-3.0.4-10.el8.x86_64 137/148 2020-11-20T12:03:31.110 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 120/148 2020-11-20T12:03:31.131 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 137/148 2020-11-20T12:03:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 121/148 2020-11-20T12:03:31.345 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : krb5-workstation-1.17-18.el8.x86_64 138/148 2020-11-20T12:03:31.471 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : NetworkManager-tui-1:1.20.0-3.el8.x86_64 139/148 2020-11-20T12:03:31.656 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libkadm5-1.17-18.el8.x86_64 140/148 2020-11-20T12:03:31.833 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 121/148 2020-11-20T12:03:31.836 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : openssl-1:1.1.1c-15.el8.x86_64 141/148 2020-11-20T12:03:32.068 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 122/148 2020-11-20T12:03:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : krb5-libs-1.17-18.el8.x86_64 142/148 2020-11-20T12:03:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 143/148 2020-11-20T12:03:32.246 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 122/148 2020-11-20T12:03:32.365 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 143/148 2020-11-20T12:03:32.539 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : NetworkManager-team-1:1.20.0-3.el8.x86_64 144/148 2020-11-20T12:03:32.586 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-1:1.20.0-3.el8.x86_64 145/148 2020-11-20T12:03:32.588 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : NetworkManager-1:1.20.0-3.el8.x86_64 145/148 2020-11-20T12:03:32.912 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-1:1.20.0-3.el8.x86_64 145/148 2020-11-20T12:03:32.913 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : NetworkManager-libnm-1:1.20.0-3.el8.x86_64 146/148 2020-11-20T12:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-libnm-1:1.20.0-3.el8.x86_64 146/148 2020-11-20T12:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 147/148 2020-11-20T12:03:33.525 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 147/148 2020-11-20T12:03:33.526 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : openssl-libs-1:1.1.1c-15.el8.x86_64 148/148 2020-11-20T12:03:33.670 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 123/148 2020-11-20T12:03:34.010 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openssl-devel-1:1.1.1c-2.el8_1.1.x86_64 124/148 2020-11-20T12:03:34.155 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xfsprogs-devel-5.0.0-1.el8.x86_64 125/148 2020-11-20T12:03:34.362 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_ 126/148 2020-11-20T12:03:34.545 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : elfutils-libelf-devel-0.176-5.el8.x86_64 127/148 2020-11-20T12:03:34.778 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : NetworkManager-team-1:1.20.0-5.el8_1.x86_64 128/148 2020-11-20T12:03:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openssl-libs-1:1.1.1c-15.el8.x86_64 148/148 2020-11-20T12:03:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : krb5-libs-1.17-9.el8.x86_64 1/148 2020-11-20T12:03:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : krb5-libs-1.17-18.el8.x86_64 2/148 2020-11-20T12:03:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : krb5-workstation-1.17-9.el8.x86_64 3/148 2020-11-20T12:03:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : krb5-workstation-1.17-18.el8.x86_64 4/148 2020-11-20T12:03:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libkadm5-1.17-9.el8.x86_64 5/148 2020-11-20T12:03:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libkadm5-1.17-18.el8.x86_64 6/148 2020-11-20T12:03:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-1:1.1.1c-2.el8_1.1.x86_64 7/148 2020-11-20T12:03:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-1:1.1.1c-15.el8.x86_64 8/148 2020-11-20T12:03:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 9/148 2020-11-20T12:03:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-libs-1:1.1.1c-15.el8.x86_64 10/148 2020-11-20T12:03:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 11/148 2020-11-20T12:03:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 12/148 2020-11-20T12:03:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 13/148 2020-11-20T12:03:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 14/148 2020-11-20T12:03:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 15/148 2020-11-20T12:03:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 16/148 2020-11-20T12:03:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 17/148 2020-11-20T12:03:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 18/148 2020-11-20T12:03:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 19/148 2020-11-20T12:03:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 20/148 2020-11-20T12:03:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 21/148 2020-11-20T12:03:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 22/148 2020-11-20T12:03:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_ 23/148 2020-11-20T12:03:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 24/148 2020-11-20T12:03:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_6 25/148 2020-11-20T12:03:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 26/148 2020-11-20T12:03:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 27/148 2020-11-20T12:03:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el 28/148 2020-11-20T12:03:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:16.0.0-7397.gb86e691b.el8. 29/148 2020-11-20T12:03:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_6 30/148 2020-11-20T12:03:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 31/148 2020-11-20T12:03:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 32/148 2020-11-20T12:03:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 33/148 2020-11-20T12:03:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 34/148 2020-11-20T12:03:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 35/148 2020-11-20T12:03:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 36/148 2020-11-20T12:03:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b. 37/148 2020-11-20T12:03:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 38/148 2020-11-20T12:03:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 39/148 2020-11-20T12:03:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 40/148 2020-11-20T12:03:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el 41/148 2020-11-20T12:03:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 42/148 2020-11-20T12:03:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.e 43/148 2020-11-20T12:03:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 44/148 2020-11-20T12:03:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : automake-1.16.1-6.el8.noarch 45/148 2020-11-20T12:03:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : bison-3.0.4-10.el8.x86_64 46/148 2020-11-20T12:03:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flex-2.6.1-9.el8.x86_64 47/148 2020-11-20T12:03:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : jq-1.5-12.el8.x86_64 48/148 2020-11-20T12:03:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 49/148 2020-11-20T12:03:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 50/148 2020-11-20T12:03:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 51/148 2020-11-20T12:03:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 52/148 2020-11-20T12:03:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 53/148 2020-11-20T12:03:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 54/148 2020-11-20T12:03:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 55/148 2020-11-20T12:03:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 56/148 2020-11-20T12:03:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 57/148 2020-11-20T12:03:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 58/148 2020-11-20T12:03:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 59/148 2020-11-20T12:03:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 60/148 2020-11-20T12:03:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 61/148 2020-11-20T12:03:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 62/148 2020-11-20T12:03:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 63/148 2020-11-20T12:03:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 64/148 2020-11-20T12:03:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 65/148 2020-11-20T12:03:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 66/148 2020-11-20T12:03:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 67/148 2020-11-20T12:03:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 68/148 2020-11-20T12:03:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af4 69/148 2020-11-20T12:03:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 70/148 2020-11-20T12:03:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 71/148 2020-11-20T12:03:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 72/148 2020-11-20T12:03:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-devel-0.176-5.el8.x86_64 73/148 2020-11-20T12:03:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 74/148 2020-11-20T12:03:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 75/148 2020-11-20T12:03:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/148 2020-11-20T12:03:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : krb5-devel-1.17-9.el8.x86_64 77/148 2020-11-20T12:03:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 78/148 2020-11-20T12:03:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcom_err-devel-1.44.6-3.el8.x86_64 79/148 2020-11-20T12:03:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 80/148 2020-11-20T12:03:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 81/148 2020-11-20T12:03:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 82/148 2020-11-20T12:03:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libselinux-devel-2.9-2.1.el8.x86_64 83/148 2020-11-20T12:03:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libsepol-devel-2.9-1.el8.x86_64 84/148 2020-11-20T12:03:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-devel-2.32.1-17.el8.x86_64 85/148 2020-11-20T12:03:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 86/148 2020-11-20T12:03:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 87/148 2020-11-20T12:03:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-devel-1:1.1.1c-2.el8_1.1.x86_64 88/148 2020-11-20T12:03:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-devel-10.32-1.el8.x86_64 89/148 2020-11-20T12:03:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-utf16-10.32-1.el8.x86_64 90/148 2020-11-20T12:03:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-utf32-10.32-1.el8.x86_64 91/148 2020-11-20T12:03:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 92/148 2020-11-20T12:03:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 93/148 2020-11-20T12:03:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 94/148 2020-11-20T12:03:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 95/148 2020-11-20T12:03:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 96/148 2020-11-20T12:03:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 97/148 2020-11-20T12:03:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 98/148 2020-11-20T12:03:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-devel-5.0.0-1.el8.x86_64 99/148 2020-11-20T12:03:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : zlib-devel-1.2.11-10.el8.x86_64 100/148 2020-11-20T12:03:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 101/148 2020-11-20T12:03:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 102/148 2020-11-20T12:03:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : leveldb-1.22-1.el8.x86_64 103/148 2020-11-20T12:03:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 104/148 2020-11-20T12:03:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 105/148 2020-11-20T12:03:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 106/148 2020-11-20T12:03:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 107/148 2020-11-20T12:03:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 108/148 2020-11-20T12:03:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 109/148 2020-11-20T12:03:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 110/148 2020-11-20T12:03:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 111/148 2020-11-20T12:03:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 112/148 2020-11-20T12:03:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 113/148 2020-11-20T12:03:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 114/148 2020-11-20T12:03:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 115/148 2020-11-20T12:03:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 116/148 2020-11-20T12:03:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 117/148 2020-11-20T12:03:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 118/148 2020-11-20T12:03:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 119/148 2020-11-20T12:03:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 120/148 2020-11-20T12:03:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 121/148 2020-11-20T12:03:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-2.6-1.el8.noarch 122/148 2020-11-20T12:03:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 123/148 2020-11-20T12:03:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 124/148 2020-11-20T12:03:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 125/148 2020-11-20T12:03:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-3.4.2-15.el8.noarch 126/148 2020-11-20T12:03:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 127/148 2020-11-20T12:03:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 128/148 2020-11-20T12:03:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 129/148 2020-11-20T12:03:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 130/148 2020-11-20T12:03:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 131/148 2020-11-20T12:03:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 132/148 2020-11-20T12:03:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 133/148 2020-11-20T12:03:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 134/148 2020-11-20T12:03:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 135/148 2020-11-20T12:03:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 136/148 2020-11-20T12:03:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 137/148 2020-11-20T12:03:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 138/148 2020-11-20T12:03:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 139/148 2020-11-20T12:03:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 140/148 2020-11-20T12:03:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-1:1.20.0-5.el8_1.x86_64 141/148 2020-11-20T12:03:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-1:1.20.0-3.el8.x86_64 142/148 2020-11-20T12:03:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64 143/148 2020-11-20T12:03:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-libnm-1:1.20.0-3.el8.x86_64 144/148 2020-11-20T12:03:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-team-1:1.20.0-5.el8_1.x86_64 145/148 2020-11-20T12:03:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-team-1:1.20.0-3.el8.x86_64 146/148 2020-11-20T12:03:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-tui-1:1.20.0-5.el8_1.x86_64 147/148 2020-11-20T12:03:35.377 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : NetworkManager-tui-1:1.20.0-5.el8_1.x86_64 129/148 2020-11-20T12:03:35.714 INFO:teuthology.orchestra.run.smithi005.stdout: Downgrading : krb5-workstation-1.17-9.el8.x86_64 130/148 2020-11-20T12:03:35.998 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 131/148 2020-11-20T12:03:36.313 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 132/148 2020-11-20T12:03:36.551 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 133/148 2020-11-20T12:03:36.834 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 133/148 2020-11-20T12:03:37.104 INFO:teuthology.orchestra.run.smithi005.stdout: Downgrading : openssl-1:1.1.1c-2.el8_1.1.x86_64 134/148 2020-11-20T12:03:37.146 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-tui-1:1.20.0-3.el8.x86_64 148/148 2020-11-20T12:03:37.146 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:37.147 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2020-11-20T12:03:37.147 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.147 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.148 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-1:1.20.0-5.el8_1.x86_64 2020-11-20T12:03:37.148 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64 2020-11-20T12:03:37.148 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-team-1:1.20.0-5.el8_1.x86_64 2020-11-20T12:03:37.149 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-tui-1:1.20.0-5.el8_1.x86_64 2020-11-20T12:03:37.149 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:37.149 INFO:teuthology.orchestra.run.smithi187.stdout:Downgraded: 2020-11-20T12:03:37.150 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-libs-1.17-9.el8.x86_64 krb5-workstation-1.17-9.el8.x86_64 2020-11-20T12:03:37.150 INFO:teuthology.orchestra.run.smithi187.stdout: libkadm5-1.17-9.el8.x86_64 openssl-1:1.1.1c-2.el8_1.1.x86_64 2020-11-20T12:03:37.150 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 2020-11-20T12:03:37.151 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:37.151 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2020-11-20T12:03:37.151 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.151 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.152 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.152 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.152 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.152 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.153 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.153 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.153 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.153 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.154 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.154 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.154 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.155 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.155 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.155 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.155 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.156 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:37.156 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:37.156 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:37.157 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:37.157 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:37.157 INFO:teuthology.orchestra.run.smithi187.stdout: bison-3.0.4-10.el8.x86_64 2020-11-20T12:03:37.157 INFO:teuthology.orchestra.run.smithi187.stdout: flex-2.6.1-9.el8.x86_64 2020-11-20T12:03:37.158 INFO:teuthology.orchestra.run.smithi187.stdout: libtool-2.4.6-25.el8.x86_64 2020-11-20T12:03:37.158 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel-0.176-5.el8.x86_64 2020-11-20T12:03:37.158 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel-0.3.112-1.el8.x86_64 2020-11-20T12:03:37.159 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel-2.32.1-17.el8.x86_64 2020-11-20T12:03:37.159 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel-1:1.1.1c-2.el8_1.1.x86_64 2020-11-20T12:03:37.159 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel-5.0.0-1.el8.x86_64 2020-11-20T12:03:37.159 INFO:teuthology.orchestra.run.smithi187.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-11-20T12:03:37.160 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-11-20T12:03:37.160 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.160 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.161 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.161 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.161 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.161 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.162 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.162 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.163 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:37.163 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:37.163 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:37.163 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:37.164 INFO:teuthology.orchestra.run.smithi187.stdout: automake-1.16.1-6.el8.noarch 2020-11-20T12:03:37.164 INFO:teuthology.orchestra.run.smithi187.stdout: jq-1.5-12.el8.x86_64 2020-11-20T12:03:37.164 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-11-20T12:03:37.165 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-11-20T12:03:37.165 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma-6.8.2-1.el8.x86_64 2020-11-20T12:03:37.165 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.3-2.el8.x86_64 2020-11-20T12:03:37.165 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-11-20T12:03:37.166 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.5.1-5.el8.noarch 2020-11-20T12:03:37.166 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2020-11-20T12:03:37.166 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-11-20T12:03:37.167 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-11-20T12:03:37.167 INFO:teuthology.orchestra.run.smithi187.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-11-20T12:03:37.167 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.0.6-13.el8.noarch 2020-11-20T12:03:37.167 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-11-20T12:03:37.168 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-11-20T12:03:37.168 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-11-20T12:03:37.168 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-11-20T12:03:37.169 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-11-20T12:03:37.169 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-11-20T12:03:37.169 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2017.2-9.el8.noarch 2020-11-20T12:03:37.169 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-11-20T12:03:37.170 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-11-20T12:03:37.170 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-11-20T12:03:37.170 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.3.2-6.el8.x86_64 2020-11-20T12:03:37.170 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-2.9.7-12.el8.x86_64 2020-11-20T12:03:37.171 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-11-20T12:03:37.171 INFO:teuthology.orchestra.run.smithi187.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2020-11-20T12:03:37.171 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-devel-1.17-9.el8.x86_64 2020-11-20T12:03:37.172 INFO:teuthology.orchestra.run.smithi187.stdout: libcom_err-devel-1.44.6-3.el8.x86_64 2020-11-20T12:03:37.172 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-11-20T12:03:37.172 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-11-20T12:03:37.172 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-11-20T12:03:37.173 INFO:teuthology.orchestra.run.smithi187.stdout: libselinux-devel-2.9-2.1.el8.x86_64 2020-11-20T12:03:37.173 INFO:teuthology.orchestra.run.smithi187.stdout: libsepol-devel-2.9-1.el8.x86_64 2020-11-20T12:03:37.173 INFO:teuthology.orchestra.run.smithi187.stdout: libverto-devel-0.3.0-5.el8.x86_64 2020-11-20T12:03:37.174 INFO:teuthology.orchestra.run.smithi187.stdout: libxslt-1.1.32-3.el8.x86_64 2020-11-20T12:03:37.174 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-devel-10.32-1.el8.x86_64 2020-11-20T12:03:37.174 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf16-10.32-1.el8.x86_64 2020-11-20T12:03:37.174 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf32-10.32-1.el8.x86_64 2020-11-20T12:03:37.175 INFO:teuthology.orchestra.run.smithi187.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-11-20T12:03:37.175 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-11-20T12:03:37.175 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-11-20T12:03:37.175 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-11-20T12:03:37.176 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-11-20T12:03:37.176 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-11-20T12:03:37.176 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-11-20T12:03:37.176 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel-1.2.11-10.el8.x86_64 2020-11-20T12:03:37.177 INFO:teuthology.orchestra.run.smithi187.stdout: fmt-6.2.1-1.el8.x86_64 2020-11-20T12:03:37.177 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-11-20T12:03:37.177 INFO:teuthology.orchestra.run.smithi187.stdout: leveldb-1.22-1.el8.x86_64 2020-11-20T12:03:37.177 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.2-3.el8.x86_64 2020-11-20T12:03:37.178 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.3.1-3.el8.x86_64 2020-11-20T12:03:37.178 INFO:teuthology.orchestra.run.smithi187.stdout: python3-apipkg-1.5-6.el8.noarch 2020-11-20T12:03:37.178 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-11-20T12:03:37.178 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-11-20T12:03:37.179 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-11-20T12:03:37.179 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-11-20T12:03:37.179 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-11-20T12:03:37.180 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-11-20T12:03:37.180 INFO:teuthology.orchestra.run.smithi187.stdout: python3-execnet-1.7.1-1.el8.noarch 2020-11-20T12:03:37.180 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-11-20T12:03:37.180 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-6.2-6.el8.noarch 2020-11-20T12:03:37.181 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-11-20T12:03:37.181 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-11.0.0-2.el8.noarch 2020-11-20T12:03:37.181 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-11-20T12:03:37.182 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-11-20T12:03:37.182 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-11-20T12:03:37.182 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-2.6-1.el8.noarch 2020-11-20T12:03:37.183 INFO:teuthology.orchestra.run.smithi187.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-11-20T12:03:37.183 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-11-20T12:03:37.183 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.4.1-12.el8.noarch 2020-11-20T12:03:37.183 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-3.4.2-15.el8.noarch 2020-11-20T12:03:37.184 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-11-20T12:03:37.184 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-11-20T12:03:37.184 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-11-20T12:03:37.184 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme-0.6.0-4.el8.noarch 2020-11-20T12:03:37.185 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-11-20T12:03:37.185 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-11-20T12:03:37.186 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-11-20T12:03:37.186 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-11-20T12:03:37.186 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-11-20T12:03:37.186 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2020-11-20T12:03:37.187 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:37.187 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2020-11-20T12:03:37.382 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 135/148 2020-11-20T12:03:37.427 DEBUG:teuthology.parallel:result is None 2020-11-20T12:03:37.699 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flex-2.6.1-9.el8.x86_64 136/148 2020-11-20T12:03:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 136/148 2020-11-20T12:03:38.116 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : bison-3.0.4-10.el8.x86_64 137/148 2020-11-20T12:03:38.192 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 137/148 2020-11-20T12:03:38.405 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : krb5-workstation-1.17-18.el8.x86_64 138/148 2020-11-20T12:03:38.572 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : NetworkManager-tui-1:1.20.0-3.el8.x86_64 139/148 2020-11-20T12:03:38.699 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : libkadm5-1.17-18.el8.x86_64 140/148 2020-11-20T12:03:38.930 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : openssl-1:1.1.1c-15.el8.x86_64 141/148 2020-11-20T12:03:39.162 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : krb5-libs-1.17-18.el8.x86_64 142/148 2020-11-20T12:03:39.163 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 143/148 2020-11-20T12:03:39.558 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 143/148 2020-11-20T12:03:39.715 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : NetworkManager-team-1:1.20.0-3.el8.x86_64 144/148 2020-11-20T12:03:39.761 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-1:1.20.0-3.el8.x86_64 145/148 2020-11-20T12:03:39.764 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : NetworkManager-1:1.20.0-3.el8.x86_64 145/148 2020-11-20T12:03:40.097 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-1:1.20.0-3.el8.x86_64 145/148 2020-11-20T12:03:40.098 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : NetworkManager-libnm-1:1.20.0-3.el8.x86_64 146/148 2020-11-20T12:03:40.397 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-libnm-1:1.20.0-3.el8.x86_64 146/148 2020-11-20T12:03:40.398 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 147/148 2020-11-20T12:03:40.830 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 147/148 2020-11-20T12:03:40.831 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : openssl-libs-1:1.1.1c-15.el8.x86_64 148/148 2020-11-20T12:03:42.378 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openssl-libs-1:1.1.1c-15.el8.x86_64 148/148 2020-11-20T12:03:42.378 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : krb5-libs-1.17-9.el8.x86_64 1/148 2020-11-20T12:03:42.379 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : krb5-libs-1.17-18.el8.x86_64 2/148 2020-11-20T12:03:42.379 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : krb5-workstation-1.17-9.el8.x86_64 3/148 2020-11-20T12:03:42.379 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : krb5-workstation-1.17-18.el8.x86_64 4/148 2020-11-20T12:03:42.379 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libkadm5-1.17-9.el8.x86_64 5/148 2020-11-20T12:03:42.380 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libkadm5-1.17-18.el8.x86_64 6/148 2020-11-20T12:03:42.380 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-1:1.1.1c-2.el8_1.1.x86_64 7/148 2020-11-20T12:03:42.380 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-1:1.1.1c-15.el8.x86_64 8/148 2020-11-20T12:03:42.380 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 9/148 2020-11-20T12:03:42.380 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-libs-1:1.1.1c-15.el8.x86_64 10/148 2020-11-20T12:03:42.381 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 11/148 2020-11-20T12:03:42.381 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 12/148 2020-11-20T12:03:42.381 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 13/148 2020-11-20T12:03:42.382 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 14/148 2020-11-20T12:03:42.382 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 15/148 2020-11-20T12:03:42.382 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 16/148 2020-11-20T12:03:42.382 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 17/148 2020-11-20T12:03:42.383 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 18/148 2020-11-20T12:03:42.383 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 19/148 2020-11-20T12:03:42.383 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 20/148 2020-11-20T12:03:42.383 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 21/148 2020-11-20T12:03:42.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 22/148 2020-11-20T12:03:42.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_ 23/148 2020-11-20T12:03:42.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 24/148 2020-11-20T12:03:42.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_6 25/148 2020-11-20T12:03:42.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 26/148 2020-11-20T12:03:42.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 27/148 2020-11-20T12:03:42.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el 28/148 2020-11-20T12:03:42.386 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-common-2:16.0.0-7397.gb86e691b.el8. 29/148 2020-11-20T12:03:42.386 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_6 30/148 2020-11-20T12:03:42.386 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 31/148 2020-11-20T12:03:42.386 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 32/148 2020-11-20T12:03:42.387 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 33/148 2020-11-20T12:03:42.387 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 34/148 2020-11-20T12:03:42.387 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 35/148 2020-11-20T12:03:42.388 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 36/148 2020-11-20T12:03:42.388 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b. 37/148 2020-11-20T12:03:42.388 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 38/148 2020-11-20T12:03:42.388 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 39/148 2020-11-20T12:03:42.389 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 40/148 2020-11-20T12:03:42.389 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el 41/148 2020-11-20T12:03:42.389 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 42/148 2020-11-20T12:03:42.389 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.e 43/148 2020-11-20T12:03:42.390 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 44/148 2020-11-20T12:03:42.390 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : automake-1.16.1-6.el8.noarch 45/148 2020-11-20T12:03:42.390 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : bison-3.0.4-10.el8.x86_64 46/148 2020-11-20T12:03:42.390 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flex-2.6.1-9.el8.x86_64 47/148 2020-11-20T12:03:42.391 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : jq-1.5-12.el8.x86_64 48/148 2020-11-20T12:03:42.391 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 49/148 2020-11-20T12:03:42.391 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 50/148 2020-11-20T12:03:42.391 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 51/148 2020-11-20T12:03:42.391 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 52/148 2020-11-20T12:03:42.392 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 53/148 2020-11-20T12:03:42.392 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 54/148 2020-11-20T12:03:42.392 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 55/148 2020-11-20T12:03:42.392 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 56/148 2020-11-20T12:03:42.393 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 57/148 2020-11-20T12:03:42.393 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 58/148 2020-11-20T12:03:42.393 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 59/148 2020-11-20T12:03:42.393 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 60/148 2020-11-20T12:03:42.394 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 61/148 2020-11-20T12:03:42.394 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 62/148 2020-11-20T12:03:42.394 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 63/148 2020-11-20T12:03:42.394 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 64/148 2020-11-20T12:03:42.395 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 65/148 2020-11-20T12:03:42.395 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 66/148 2020-11-20T12:03:42.395 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 67/148 2020-11-20T12:03:42.395 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 68/148 2020-11-20T12:03:42.396 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af4 69/148 2020-11-20T12:03:42.396 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 70/148 2020-11-20T12:03:42.396 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 71/148 2020-11-20T12:03:42.396 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 72/148 2020-11-20T12:03:42.397 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : elfutils-libelf-devel-0.176-5.el8.x86_64 73/148 2020-11-20T12:03:42.397 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 74/148 2020-11-20T12:03:42.397 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 75/148 2020-11-20T12:03:42.397 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/148 2020-11-20T12:03:42.398 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : krb5-devel-1.17-9.el8.x86_64 77/148 2020-11-20T12:03:42.398 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 78/148 2020-11-20T12:03:42.398 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcom_err-devel-1.44.6-3.el8.x86_64 79/148 2020-11-20T12:03:42.398 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 80/148 2020-11-20T12:03:42.399 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 81/148 2020-11-20T12:03:42.399 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 82/148 2020-11-20T12:03:42.399 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libselinux-devel-2.9-2.1.el8.x86_64 83/148 2020-11-20T12:03:42.399 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libsepol-devel-2.9-1.el8.x86_64 84/148 2020-11-20T12:03:42.400 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libuuid-devel-2.32.1-17.el8.x86_64 85/148 2020-11-20T12:03:42.400 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 86/148 2020-11-20T12:03:42.400 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 87/148 2020-11-20T12:03:42.400 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-devel-1:1.1.1c-2.el8_1.1.x86_64 88/148 2020-11-20T12:03:42.401 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : pcre2-devel-10.32-1.el8.x86_64 89/148 2020-11-20T12:03:42.401 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : pcre2-utf16-10.32-1.el8.x86_64 90/148 2020-11-20T12:03:42.401 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : pcre2-utf32-10.32-1.el8.x86_64 91/148 2020-11-20T12:03:42.401 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 92/148 2020-11-20T12:03:42.402 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 93/148 2020-11-20T12:03:42.402 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 94/148 2020-11-20T12:03:42.402 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 95/148 2020-11-20T12:03:42.402 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 96/148 2020-11-20T12:03:42.403 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 97/148 2020-11-20T12:03:42.403 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 98/148 2020-11-20T12:03:42.403 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xfsprogs-devel-5.0.0-1.el8.x86_64 99/148 2020-11-20T12:03:42.403 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : zlib-devel-1.2.11-10.el8.x86_64 100/148 2020-11-20T12:03:42.403 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 101/148 2020-11-20T12:03:42.404 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 102/148 2020-11-20T12:03:42.404 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : leveldb-1.22-1.el8.x86_64 103/148 2020-11-20T12:03:42.404 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 104/148 2020-11-20T12:03:42.404 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 105/148 2020-11-20T12:03:42.405 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 106/148 2020-11-20T12:03:42.405 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 107/148 2020-11-20T12:03:42.405 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 108/148 2020-11-20T12:03:42.405 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 109/148 2020-11-20T12:03:42.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 110/148 2020-11-20T12:03:42.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 111/148 2020-11-20T12:03:42.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 112/148 2020-11-20T12:03:42.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 113/148 2020-11-20T12:03:42.407 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 114/148 2020-11-20T12:03:42.407 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 115/148 2020-11-20T12:03:42.407 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 116/148 2020-11-20T12:03:42.407 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 117/148 2020-11-20T12:03:42.408 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 118/148 2020-11-20T12:03:42.408 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 119/148 2020-11-20T12:03:42.408 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 120/148 2020-11-20T12:03:42.409 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 121/148 2020-11-20T12:03:42.409 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-portend-2.6-1.el8.noarch 122/148 2020-11-20T12:03:42.409 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 123/148 2020-11-20T12:03:42.409 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 124/148 2020-11-20T12:03:42.410 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 125/148 2020-11-20T12:03:42.410 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rsa-3.4.2-15.el8.noarch 126/148 2020-11-20T12:03:42.410 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 127/148 2020-11-20T12:03:42.410 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 128/148 2020-11-20T12:03:42.411 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 129/148 2020-11-20T12:03:42.411 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 130/148 2020-11-20T12:03:42.411 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 131/148 2020-11-20T12:03:42.411 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 132/148 2020-11-20T12:03:42.412 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 133/148 2020-11-20T12:03:42.412 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 134/148 2020-11-20T12:03:42.412 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 135/148 2020-11-20T12:03:42.412 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 136/148 2020-11-20T12:03:42.413 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 137/148 2020-11-20T12:03:42.413 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 138/148 2020-11-20T12:03:42.413 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 139/148 2020-11-20T12:03:42.413 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 140/148 2020-11-20T12:03:42.414 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-1:1.20.0-5.el8_1.x86_64 141/148 2020-11-20T12:03:42.414 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-1:1.20.0-3.el8.x86_64 142/148 2020-11-20T12:03:42.414 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64 143/148 2020-11-20T12:03:42.414 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-libnm-1:1.20.0-3.el8.x86_64 144/148 2020-11-20T12:03:42.414 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-team-1:1.20.0-5.el8_1.x86_64 145/148 2020-11-20T12:03:42.415 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-team-1:1.20.0-3.el8.x86_64 146/148 2020-11-20T12:03:42.415 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-tui-1:1.20.0-5.el8_1.x86_64 147/148 2020-11-20T12:03:44.266 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-tui-1:1.20.0-3.el8.x86_64 148/148 2020-11-20T12:03:44.267 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:44.267 INFO:teuthology.orchestra.run.smithi005.stdout:Upgraded: 2020-11-20T12:03:44.268 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.268 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.268 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-1:1.20.0-5.el8_1.x86_64 2020-11-20T12:03:44.269 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64 2020-11-20T12:03:44.269 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-team-1:1.20.0-5.el8_1.x86_64 2020-11-20T12:03:44.269 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-tui-1:1.20.0-5.el8_1.x86_64 2020-11-20T12:03:44.270 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:44.270 INFO:teuthology.orchestra.run.smithi005.stdout:Downgraded: 2020-11-20T12:03:44.271 INFO:teuthology.orchestra.run.smithi005.stdout: krb5-libs-1.17-9.el8.x86_64 krb5-workstation-1.17-9.el8.x86_64 2020-11-20T12:03:44.271 INFO:teuthology.orchestra.run.smithi005.stdout: libkadm5-1.17-9.el8.x86_64 openssl-1:1.1.1c-2.el8_1.1.x86_64 2020-11-20T12:03:44.271 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-libs-1:1.1.1c-2.el8_1.1.x86_64 2020-11-20T12:03:44.272 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:44.273 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2020-11-20T12:03:44.273 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.273 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.273 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.274 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.274 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.274 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.274 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.275 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.275 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.275 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.275 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.275 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.276 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.276 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.276 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.276 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.276 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.277 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:44.277 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:44.277 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:44.277 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:44.278 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:44.278 INFO:teuthology.orchestra.run.smithi005.stdout: bison-3.0.4-10.el8.x86_64 2020-11-20T12:03:44.278 INFO:teuthology.orchestra.run.smithi005.stdout: flex-2.6.1-9.el8.x86_64 2020-11-20T12:03:44.278 INFO:teuthology.orchestra.run.smithi005.stdout: libtool-2.4.6-25.el8.x86_64 2020-11-20T12:03:44.278 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-libelf-devel-0.176-5.el8.x86_64 2020-11-20T12:03:44.279 INFO:teuthology.orchestra.run.smithi005.stdout: libaio-devel-0.3.112-1.el8.x86_64 2020-11-20T12:03:44.279 INFO:teuthology.orchestra.run.smithi005.stdout: libuuid-devel-2.32.1-17.el8.x86_64 2020-11-20T12:03:44.279 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-devel-1:1.1.1c-2.el8_1.1.x86_64 2020-11-20T12:03:44.279 INFO:teuthology.orchestra.run.smithi005.stdout: xfsprogs-devel-5.0.0-1.el8.x86_64 2020-11-20T12:03:44.279 INFO:teuthology.orchestra.run.smithi005.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-11-20T12:03:44.280 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-11-20T12:03:44.280 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.280 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.280 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.281 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.281 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.281 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.281 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.281 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.282 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T12:03:44.282 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:44.282 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:44.282 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T12:03:44.283 INFO:teuthology.orchestra.run.smithi005.stdout: automake-1.16.1-6.el8.noarch 2020-11-20T12:03:44.283 INFO:teuthology.orchestra.run.smithi005.stdout: jq-1.5-12.el8.x86_64 2020-11-20T12:03:44.283 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-11-20T12:03:44.283 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-11-20T12:03:44.283 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma-6.8.2-1.el8.x86_64 2020-11-20T12:03:44.284 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-0.3.3-2.el8.x86_64 2020-11-20T12:03:44.284 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-11-20T12:03:44.284 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel-2.5.1-5.el8.noarch 2020-11-20T12:03:44.284 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2020-11-20T12:03:44.285 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-11-20T12:03:44.285 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-11-20T12:03:44.285 INFO:teuthology.orchestra.run.smithi005.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-11-20T12:03:44.285 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako-1.0.6-13.el8.noarch 2020-11-20T12:03:44.285 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-11-20T12:03:44.286 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-11-20T12:03:44.286 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-11-20T12:03:44.286 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-11-20T12:03:44.286 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-11-20T12:03:44.286 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-11-20T12:03:44.287 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz-2017.2-9.el8.noarch 2020-11-20T12:03:44.287 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-11-20T12:03:44.287 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-11-20T12:03:44.287 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-11-20T12:03:44.288 INFO:teuthology.orchestra.run.smithi005.stdout: socat-1.7.3.2-6.el8.x86_64 2020-11-20T12:03:44.288 INFO:teuthology.orchestra.run.smithi005.stdout: fuse-2.9.7-12.el8.x86_64 2020-11-20T12:03:44.288 INFO:teuthology.orchestra.run.smithi005.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-11-20T12:03:44.288 INFO:teuthology.orchestra.run.smithi005.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2020-11-20T12:03:44.288 INFO:teuthology.orchestra.run.smithi005.stdout: krb5-devel-1.17-9.el8.x86_64 2020-11-20T12:03:44.289 INFO:teuthology.orchestra.run.smithi005.stdout: libcom_err-devel-1.44.6-3.el8.x86_64 2020-11-20T12:03:44.289 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-11-20T12:03:44.289 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-11-20T12:03:44.289 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-11-20T12:03:44.289 INFO:teuthology.orchestra.run.smithi005.stdout: libselinux-devel-2.9-2.1.el8.x86_64 2020-11-20T12:03:44.290 INFO:teuthology.orchestra.run.smithi005.stdout: libsepol-devel-2.9-1.el8.x86_64 2020-11-20T12:03:44.290 INFO:teuthology.orchestra.run.smithi005.stdout: libverto-devel-0.3.0-5.el8.x86_64 2020-11-20T12:03:44.290 INFO:teuthology.orchestra.run.smithi005.stdout: libxslt-1.1.32-3.el8.x86_64 2020-11-20T12:03:44.290 INFO:teuthology.orchestra.run.smithi005.stdout: pcre2-devel-10.32-1.el8.x86_64 2020-11-20T12:03:44.290 INFO:teuthology.orchestra.run.smithi005.stdout: pcre2-utf16-10.32-1.el8.x86_64 2020-11-20T12:03:44.291 INFO:teuthology.orchestra.run.smithi005.stdout: pcre2-utf32-10.32-1.el8.x86_64 2020-11-20T12:03:44.291 INFO:teuthology.orchestra.run.smithi005.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-11-20T12:03:44.291 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-11-20T12:03:44.291 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-11-20T12:03:44.291 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-11-20T12:03:44.292 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-11-20T12:03:44.292 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-11-20T12:03:44.292 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-11-20T12:03:44.292 INFO:teuthology.orchestra.run.smithi005.stdout: zlib-devel-1.2.11-10.el8.x86_64 2020-11-20T12:03:44.293 INFO:teuthology.orchestra.run.smithi005.stdout: fmt-6.2.1-1.el8.x86_64 2020-11-20T12:03:44.293 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-11-20T12:03:44.293 INFO:teuthology.orchestra.run.smithi005.stdout: leveldb-1.22-1.el8.x86_64 2020-11-20T12:03:44.293 INFO:teuthology.orchestra.run.smithi005.stdout: liboath-2.6.2-3.el8.x86_64 2020-11-20T12:03:44.293 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind-1.3.1-3.el8.x86_64 2020-11-20T12:03:44.294 INFO:teuthology.orchestra.run.smithi005.stdout: python3-apipkg-1.5-6.el8.noarch 2020-11-20T12:03:44.294 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-11-20T12:03:44.294 INFO:teuthology.orchestra.run.smithi005.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-11-20T12:03:44.294 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-11-20T12:03:44.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-11-20T12:03:44.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-11-20T12:03:44.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-11-20T12:03:44.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-execnet-1.7.1-1.el8.noarch 2020-11-20T12:03:44.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-11-20T12:03:44.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-6.2-6.el8.noarch 2020-11-20T12:03:44.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-11-20T12:03:44.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes-11.0.0-2.el8.noarch 2020-11-20T12:03:44.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-11-20T12:03:44.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-11-20T12:03:44.297 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-11-20T12:03:44.297 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend-2.6-1.el8.noarch 2020-11-20T12:03:44.297 INFO:teuthology.orchestra.run.smithi005.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-11-20T12:03:44.297 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-11-20T12:03:44.297 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes-2.4.1-12.el8.noarch 2020-11-20T12:03:44.298 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa-3.4.2-15.el8.noarch 2020-11-20T12:03:44.298 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-11-20T12:03:44.298 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-11-20T12:03:44.298 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-11-20T12:03:44.299 INFO:teuthology.orchestra.run.smithi005.stdout: python3-trustme-0.6.0-4.el8.noarch 2020-11-20T12:03:44.299 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-11-20T12:03:44.299 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-11-20T12:03:44.299 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-11-20T12:03:44.299 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-11-20T12:03:44.300 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-11-20T12:03:44.300 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2020-11-20T12:03:44.300 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:44.301 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2020-11-20T12:03:44.564 DEBUG:teuthology.parallel:result is None 2020-11-20T12:03:44.565 INFO:teuthology.orchestra.run.smithi187:> sudo lsb_release -is 2020-11-20T12:03:44.702 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS 2020-11-20T12:03:44.706 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T12:03:44.707 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:03:44.874 INFO:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-11-20T12:03:44.928 INFO:teuthology.orchestra.run.smithi187.stdout:16.0.0-7397.gb86e691b.el8 2020-11-20T12:03:44.929 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7397.gb86e691b.el8 2020-11-20T12:03:44.930 INFO:teuthology.task.install:The correct ceph version 16.0.0-7397.gb86e691b is installed. 2020-11-20T12:03:44.930 INFO:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2020-11-20T12:03:45.068 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS 2020-11-20T12:03:45.072 DEBUG:teuthology.misc:System to be installed: CentOS 2020-11-20T12:03:45.073 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:03:45.246 INFO:teuthology.orchestra.run.smithi005:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-11-20T12:03:45.304 INFO:teuthology.orchestra.run.smithi005.stdout:16.0.0-7397.gb86e691b.el8 2020-11-20T12:03:45.305 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7397.gb86e691b.el8 2020-11-20T12:03:45.306 INFO:teuthology.task.install:The correct ceph version 16.0.0-7397.gb86e691b is installed. 2020-11-20T12:03:45.306 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-20T12:03:45.307 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:45.307 INFO:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T12:03:45.358 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T12:03:45.409 INFO:teuthology.orchestra.run.smithi187.stderr:24+1 records in 2020-11-20T12:03:45.409 INFO:teuthology.orchestra.run.smithi187.stderr:24+1 records out 2020-11-20T12:03:45.410 INFO:teuthology.orchestra.run.smithi187.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000375092 s, 33.8 MB/s 2020-11-20T12:03:45.413 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:03:45.414 INFO:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T12:03:45.455 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T12:03:45.494 INFO:teuthology.orchestra.run.smithi005.stderr:24+1 records in 2020-11-20T12:03:45.495 INFO:teuthology.orchestra.run.smithi005.stderr:24+1 records out 2020-11-20T12:03:45.495 INFO:teuthology.orchestra.run.smithi005.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000348541 s, 36.3 MB/s 2020-11-20T12:03:45.498 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-20T12:03:45.498 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:45.498 INFO:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2020-11-20T12:03:45.537 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-20T12:03:45.614 INFO:teuthology.orchestra.run.smithi187.stderr:5+1 records in 2020-11-20T12:03:45.615 INFO:teuthology.orchestra.run.smithi187.stderr:5+1 records out 2020-11-20T12:03:45.615 INFO:teuthology.orchestra.run.smithi187.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 0.000333592 s, 9.2 MB/s 2020-11-20T12:03:45.619 INFO:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-20T12:03:45.712 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:03:45.713 INFO:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2020-11-20T12:03:45.757 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-20T12:03:45.793 INFO:teuthology.orchestra.run.smithi005.stderr:5+1 records in 2020-11-20T12:03:45.794 INFO:teuthology.orchestra.run.smithi005.stderr:5+1 records out 2020-11-20T12:03:45.794 INFO:teuthology.orchestra.run.smithi005.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 0.000129593 s, 23.6 MB/s 2020-11-20T12:03:45.797 INFO:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-20T12:03:45.917 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-20T12:03:45.918 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:45.918 INFO:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T12:03:45.973 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T12:03:46.020 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:03:46.020 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:03:46.020 INFO:teuthology.orchestra.run.smithi187.stderr:299 bytes copied, 0.000123138 s, 2.4 MB/s 2020-11-20T12:03:46.024 INFO:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-20T12:03:46.122 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:03:46.122 INFO:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T12:03:46.166 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T12:03:46.209 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:03:46.210 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:03:46.210 INFO:teuthology.orchestra.run.smithi005.stderr:299 bytes copied, 0.000103158 s, 2.9 MB/s 2020-11-20T12:03:46.212 INFO:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-20T12:03:46.293 INFO:teuthology.run_tasks:Running task ceph... 2020-11-20T12:03:46.368 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-20T12:03:46.369 INFO:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2020-11-20T12:03:46.378 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 777 /var/log/ceph 2020-11-20T12:03:46.478 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-20T12:03:46.479 INFO:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-20T12:03:46.483 INFO:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-20T12:03:46.584 INFO:tasks.ceph:Creating extra log directories... 2020-11-20T12:03:46.585 INFO:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-20T12:03:46.590 INFO:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-20T12:03:46.723 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2020-11-20T12:03:46.724 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2020-11-20T12:03:46.724 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2020-11-20T12:03:46.725 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:46.725 INFO:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T12:03:46.779 INFO:teuthology.orchestra.run.smithi187.stderr:+ dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T12:03:46.782 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:03:46.782 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:03:46.782 INFO:teuthology.orchestra.run.smithi187.stderr:408 bytes copied, 0.000125547 s, 3.2 MB/s 2020-11-20T12:03:46.783 INFO:teuthology.orchestra.run.smithi187:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2020-11-20T12:03:47.108 INFO:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2020-11-20T12:03:47.219 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:03:47.220 INFO:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T12:03:47.267 INFO:teuthology.orchestra.run.smithi005.stderr:+ dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T12:03:47.272 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:03:47.274 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:03:47.275 INFO:teuthology.orchestra.run.smithi005.stderr:408 bytes copied, 0.000110702 s, 3.7 MB/s 2020-11-20T12:03:47.276 INFO:teuthology.orchestra.run.smithi005:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2020-11-20T12:03:47.524 INFO:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2020-11-20T12:03:47.614 INFO:tasks.ceph:Creating ceph cluster ceph... 2020-11-20T12:03:47.614 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-20T12:03:47.615 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640308', 'branch': 'wip-pdonnell-testing-20201120.053228', 'description': 'fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '5640308', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.1', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20201120.053228'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20201120.053228', 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 95, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3']], 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20201120.053228', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-20_08:39:42', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.6030', 'targets': {'smithi187.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmEJkuo87iHMn7jODntPiOZNQeC8JxqLEsXf5Kfr7f6P7wo+BNVAX9coacFXpI/cVZm/PrSN3FI++JGh6KkubLfxAfS289nNRM653HD6JmGu9FbGy0pCH2Ekd4Ff3RT6ViYWUs/dl2GXe04muM0zTGoAcz82C5FmQzGQgzN12esoHqF7UXR9qjb361GDe1lrvTY+M/U8V8+LcRuDsYUne8Q4DwNHJ+uNnCJmMzTdWMQyo+jclbb1Ctg//XhWA19yaZRA9wbdSjncfcPPi3APfk42BG2ANSQ1u98wPtlmM/KuSKrUvB2lIhllnpoBuNjJH69cxAFQ4tLaRQgHdvw188AzfGgQIHoXFUMOCKXQa1QPIaJU5uWY6TdKMLLyoLGJ941mAvFwac+SeQYi1qyX0RLtWBHELqKcZpT/GiSndwTgp5195d3xzoOsQOhGM76nnWWPfX8M8T6rw1P+yRK0zTtrJ3mYVTeA6qjC3LGD4Sobef9Fb+tfGxqbelB8r1Mh8=', 'smithi005.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT6/H5StJqG+ZPdwMkhIuuh6om1MQcqs5XXVo+WKx5Cf9duUWjGUj8WWAfp/p0/iLH9FTfRZSYFDrzTevV6HLFNm3D57+RSHEnQSL7R3zWOsDvtG7teZ6O9d/kzF3ZdBNbqOsN020st1bIfgnPiz0awjJdC10s0LuTCKV9r4tRbHIbdxmekhA9GqjzUH9hByV2cdWkqbkqjnDlP5Tn8s13JRwA/+mVfnMUvoMslFOb+FMeZNQKKCfCnLLh70ZM6ZGiRtILlZAols5hqvCuDA7JYR7q2joUCIJL681c4nMx5CrHpB6peVw7b7r6OL4ZYP0j8YAL8ovIPoguvTBown7uL6XUgL+NkeQZV6l0fIT/coOMvIujx1P2N5kqYpZ5aYh+OYm68Ti3N6aklrjZnfmSxS8aBRNKdJUvRWwu8P0bztgWrX0b81QVT8FPBPQw1DmNBeGWY9ViKgb/RFY1kc81ENtZGkjPFF0dNvySM3SrooXJwkv1CroWvs72DhTRJks='}} 2020-11-20T12:03:47.616 INFO:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-11-20T12:03:47.621 INFO:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-11-20T12:03:47.684 INFO:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2020-11-20T12:03:47.704 INFO:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2020-11-20T12:03:47.799 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:47.799 INFO:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2020-11-20T12:03:47.857 INFO:teuthology.orchestra.run.smithi187.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-20T12:03:47.859 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/vg_nvme/lv_1 2020-11-20T12:03:47.859 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/vg_nvme/lv_2 2020-11-20T12:03:47.859 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/vg_nvme/lv_3 2020-11-20T12:03:47.860 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/vg_nvme/lv_4 2020-11-20T12:03:47.860 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:03:47.860 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:03:47.860 INFO:teuthology.orchestra.run.smithi187.stderr:72 bytes copied, 0.000151299 s, 476 kB/s 2020-11-20T12:03:47.861 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-20T12:03:47.862 INFO:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-20T12:03:47.979 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-20T12:03:47.980 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:03:47.980 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 442 Links: 1 2020-11-20T12:03:47.980 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:03:47.980 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T12:03:47.981 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2020-11-20 12:03:34.323230945 +0000 2020-11-20T12:03:47.981 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2020-11-20 12:01:13.594409544 +0000 2020-11-20T12:03:47.981 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2020-11-20 12:01:13.594409544 +0000 2020-11-20T12:03:47.981 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2020-11-20T12:03:48.028 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2020-11-20T12:03:48.029 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2020-11-20T12:03:48.029 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000434241 s, 1.2 MB/s 2020-11-20T12:03:48.039 INFO:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-20T12:03:48.110 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-20T12:03:48.111 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:03:48.111 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 448 Links: 1 2020-11-20T12:03:48.111 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:03:48.112 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T12:03:48.112 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2020-11-20 12:03:34.323230945 +0000 2020-11-20T12:03:48.112 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2020-11-20 12:01:14.172400042 +0000 2020-11-20T12:03:48.112 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2020-11-20 12:01:14.172400042 +0000 2020-11-20T12:03:48.113 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2020-11-20T12:03:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2020-11-20T12:03:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2020-11-20T12:03:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.00037615 s, 1.4 MB/s 2020-11-20T12:03:48.167 INFO:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-20T12:03:48.234 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-20T12:03:48.235 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:03:48.235 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 454 Links: 1 2020-11-20T12:03:48.235 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:03:48.235 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T12:03:48.236 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2020-11-20 12:03:34.323230945 +0000 2020-11-20T12:03:48.236 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2020-11-20 12:01:14.735390788 +0000 2020-11-20T12:03:48.236 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2020-11-20 12:01:14.735390788 +0000 2020-11-20T12:03:48.236 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2020-11-20T12:03:48.282 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2020-11-20T12:03:48.283 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2020-11-20T12:03:48.283 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000377384 s, 1.4 MB/s 2020-11-20T12:03:48.292 INFO:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-20T12:03:48.402 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-20T12:03:48.403 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:03:48.403 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 460 Links: 1 2020-11-20T12:03:48.403 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:03:48.404 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T12:03:48.404 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2020-11-20 12:03:34.323230945 +0000 2020-11-20T12:03:48.404 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2020-11-20 12:01:15.336380908 +0000 2020-11-20T12:03:48.404 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2020-11-20 12:01:15.336380908 +0000 2020-11-20T12:03:48.405 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2020-11-20T12:03:48.449 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2020-11-20T12:03:48.450 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2020-11-20T12:03:48.450 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000329274 s, 1.6 MB/s 2020-11-20T12:03:48.459 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:03:48.459 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:03:48.460 INFO:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2020-11-20T12:03:48.512 INFO:teuthology.orchestra.run.smithi005.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-20T12:03:48.513 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_1 2020-11-20T12:03:48.514 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_2 2020-11-20T12:03:48.514 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_3 2020-11-20T12:03:48.514 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_4 2020-11-20T12:03:48.514 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:03:48.515 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:03:48.515 INFO:teuthology.orchestra.run.smithi005.stderr:72 bytes copied, 0.000161265 s, 446 kB/s 2020-11-20T12:03:48.515 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-20T12:03:48.516 INFO:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-20T12:03:48.617 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-20T12:03:48.617 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:03:48.617 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 442 Links: 1 2020-11-20T12:03:48.618 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:03:48.618 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T12:03:48.618 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2020-11-20 12:03:41.729445921 +0000 2020-11-20T12:03:48.618 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2020-11-20 12:01:15.899817087 +0000 2020-11-20T12:03:48.619 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2020-11-20 12:01:15.899817087 +0000 2020-11-20T12:03:48.619 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2020-11-20T12:03:48.655 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2020-11-20T12:03:48.655 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2020-11-20T12:03:48.655 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00021638 s, 2.4 MB/s 2020-11-20T12:03:48.663 INFO:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-20T12:03:48.707 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-20T12:03:48.708 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:03:48.708 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 448 Links: 1 2020-11-20T12:03:48.708 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:03:48.708 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T12:03:48.708 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2020-11-20 12:03:41.729445921 +0000 2020-11-20T12:03:48.709 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2020-11-20 12:01:16.533803614 +0000 2020-11-20T12:03:48.709 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2020-11-20 12:01:16.533803614 +0000 2020-11-20T12:03:48.709 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2020-11-20T12:03:48.744 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2020-11-20T12:03:48.745 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2020-11-20T12:03:48.745 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000193348 s, 2.6 MB/s 2020-11-20T12:03:48.752 INFO:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-20T12:03:48.833 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-20T12:03:48.833 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:03:48.834 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 454 Links: 1 2020-11-20T12:03:48.834 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:03:48.834 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T12:03:48.834 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2020-11-20 12:03:41.729445921 +0000 2020-11-20T12:03:48.835 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2020-11-20 12:01:17.156790375 +0000 2020-11-20T12:03:48.835 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2020-11-20 12:01:17.156790375 +0000 2020-11-20T12:03:48.835 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2020-11-20T12:03:48.873 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2020-11-20T12:03:48.873 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2020-11-20T12:03:48.874 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000210186 s, 2.4 MB/s 2020-11-20T12:03:48.880 INFO:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-20T12:03:48.925 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-20T12:03:48.925 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:03:48.926 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 460 Links: 1 2020-11-20T12:03:48.926 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:03:48.926 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T12:03:48.926 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2020-11-20 12:03:41.730445900 +0000 2020-11-20T12:03:48.926 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2020-11-20 12:01:17.795776796 +0000 2020-11-20T12:03:48.927 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2020-11-20 12:01:17.795776796 +0000 2020-11-20T12:03:48.927 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2020-11-20T12:03:48.967 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2020-11-20T12:03:48.967 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2020-11-20T12:03:48.967 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000206041 s, 2.5 MB/s 2020-11-20T12:03:48.974 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:03:48.974 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2020-11-20T12:03:48.975 INFO:tasks.ceph:Generating config... 2020-11-20T12:03:48.976 INFO:tasks.ceph:[client] client mount timeout = 600 2020-11-20T12:03:48.977 INFO:tasks.ceph:[client] debug client = 20 2020-11-20T12:03:48.977 INFO:tasks.ceph:[client] debug ms = 1 2020-11-20T12:03:48.977 INFO:tasks.ceph:[client] pid file = 2020-11-20T12:03:48.978 INFO:tasks.ceph:[client] rados mon op timeout = 15m 2020-11-20T12:03:48.978 INFO:tasks.ceph:[client] rados osd op timeout = 15m 2020-11-20T12:03:48.979 INFO:tasks.ceph:[global] ms die on skipped message = False 2020-11-20T12:03:48.979 INFO:tasks.ceph:[mds] debug mds = 20 2020-11-20T12:03:48.979 INFO:tasks.ceph:[mds] debug ms = 1 2020-11-20T12:03:48.979 INFO:tasks.ceph:[mds] mds bal frag = True 2020-11-20T12:03:48.980 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2020-11-20T12:03:48.980 INFO:tasks.ceph:[mds] mds bal merge size = 5 2020-11-20T12:03:48.980 INFO:tasks.ceph:[mds] mds bal split bits = 3 2020-11-20T12:03:48.981 INFO:tasks.ceph:[mds] mds bal split size = 100 2020-11-20T12:03:48.981 INFO:tasks.ceph:[mds] mds debug frag = True 2020-11-20T12:03:48.981 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2020-11-20T12:03:48.981 INFO:tasks.ceph:[mds] mds op complaint time = 180 2020-11-20T12:03:48.982 INFO:tasks.ceph:[mds] mds verify scatter = True 2020-11-20T12:03:48.982 INFO:tasks.ceph:[mds] osd op complaint time = 180 2020-11-20T12:03:48.982 INFO:tasks.ceph:[mds] rados mon op timeout = 15m 2020-11-20T12:03:48.983 INFO:tasks.ceph:[mds] rados osd op timeout = 15m 2020-11-20T12:03:48.983 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-20T12:03:48.983 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-20T12:03:48.983 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-20T12:03:48.984 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-20T12:03:48.984 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-20T12:03:48.984 INFO:tasks.ceph:[mon] mon op complaint time = 120 2020-11-20T12:03:48.985 INFO:tasks.ceph:[osd] bdev async discard = True 2020-11-20T12:03:48.985 INFO:tasks.ceph:[osd] bdev enable discard = True 2020-11-20T12:03:48.985 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2020-11-20T12:03:48.985 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2020-11-20T12:03:48.986 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2020-11-20T12:03:48.986 INFO:tasks.ceph:[osd] debug bluefs = 20 2020-11-20T12:03:48.986 INFO:tasks.ceph:[osd] debug bluestore = 20 2020-11-20T12:03:48.987 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-20T12:03:48.987 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-20T12:03:48.987 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-20T12:03:48.987 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-20T12:03:48.988 INFO:tasks.ceph:[osd] debug rocksdb = 10 2020-11-20T12:03:48.988 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2020-11-20T12:03:48.988 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2020-11-20T12:03:48.989 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2020-11-20T12:03:48.989 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2020-11-20T12:03:48.989 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2020-11-20T12:03:48.989 INFO:tasks.ceph:[osd] osd op complaint time = 180 2020-11-20T12:03:48.990 INFO:tasks.ceph:Setting up mon.a... 2020-11-20T12:03:48.990 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2020-11-20T12:03:49.161 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.keyring 2020-11-20T12:03:49.171 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2020-11-20T12:03:49.305 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:03:49.403 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.187'), ('mon.b', '172.21.15.5'), ('mon.c', '[v2:172.21.15.5:3301,v1:172.21.15.5:6790]')] 2020-11-20T12:03:49.404 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.b': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}} 2020-11-20T12:03:49.405 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:49.406 INFO:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T12:03:49.457 INFO:teuthology.orchestra.run.smithi187.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T12:03:49.459 INFO:teuthology.orchestra.run.smithi187.stderr:6+1 records in 2020-11-20T12:03:49.460 INFO:teuthology.orchestra.run.smithi187.stderr:6+1 records out 2020-11-20T12:03:49.460 INFO:teuthology.orchestra.run.smithi187.stderr:3547 bytes (3.5 kB, 3.5 KiB) copied, 0.000173959 s, 20.4 MB/s 2020-11-20T12:03:49.461 INFO:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.187 --add b 172.21.15.5 --addv c '[v2:172.21.15.5:3301,v1:172.21.15.5:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:03:49.564 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:03:49.564 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: generated fsid 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:03:49.564 INFO:teuthology.orchestra.run.smithi187.stdout:epoch 0 2020-11-20T12:03:49.564 INFO:teuthology.orchestra.run.smithi187.stdout:fsid 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:03:49.565 INFO:teuthology.orchestra.run.smithi187.stdout:last_changed 2020-11-20T12:03:49.561294+0000 2020-11-20T12:03:49.565 INFO:teuthology.orchestra.run.smithi187.stdout:created 2020-11-20T12:03:49.561294+0000 2020-11-20T12:03:49.565 INFO:teuthology.orchestra.run.smithi187.stdout:min_mon_release 0 (unknown) 2020-11-20T12:03:49.565 INFO:teuthology.orchestra.run.smithi187.stdout:election_strategy: 1 2020-11-20T12:03:49.566 INFO:teuthology.orchestra.run.smithi187.stdout:0: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.a 2020-11-20T12:03:49.566 INFO:teuthology.orchestra.run.smithi187.stdout:1: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.b 2020-11-20T12:03:49.566 INFO:teuthology.orchestra.run.smithi187.stdout:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2020-11-20T12:03:49.566 INFO:teuthology.orchestra.run.smithi187.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2020-11-20T12:03:49.568 INFO:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T12:03:49.623 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 099efcab-1650-4909-9b03-145da903b26d... 2020-11-20T12:03:49.625 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:03:49.630 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:03:49.782 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:03:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:03:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:03:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:03:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:03:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:03:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:03:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:03:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:03:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:03:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:03:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:03:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:03:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:03:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:03:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:03:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:03:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:03:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:03:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:03:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:03:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:03:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:03:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:03:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:03:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:03:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:03:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:03:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:03:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:03:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:03:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:03:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:03:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:03:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.795 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:03:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:03:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:03:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:03:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:03:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:03:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:03:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:03:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:03:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:03:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:03:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.799 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:03:49.799 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.799 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:03:49.799 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.800 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:03:49.800 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.800 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:03:49.800 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:03:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:03:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:03:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:03:49.802 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:03:49.802 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:03:49.802 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:03:49.802 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:03:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:03:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:03:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:03:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:03:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:03:49.804 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:03:49.804 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:03:49.804 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:03:49.804 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:03:49.805 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:03:49.805 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:03:49.805 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:03:49.805 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:03:49.806 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.806 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:03:49.806 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:03:49.806 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:03:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:03:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:03:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:03:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:03:49.808 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:03:49.808 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.808 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:03:49.808 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:03:49.809 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:03:49.809 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:03:49.809 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:03:49.810 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:03:49.810 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:03:49.810 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:03:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:03:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:03:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:03:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:03:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:03:49.812 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:03:49.812 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:03:49.812 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:03:49.812 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:03:49.813 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:03:49.813 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:03:49.813 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:03:49.813 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:03:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:03:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:03:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:03:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:03:49.814 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:03:49.815 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:03:49.815 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:03:49.815 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:03:49.815 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:03:49.816 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:03:49.816 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:03:49.816 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:03:49.816 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:03:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:03:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:03:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:03:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:03:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:03:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:03:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:03:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:03:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:03:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:03:49.819 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:03:49.823 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:03:49.823 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:03:49.823 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:03:49.824 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:03:49.824 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.824 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:03:49.824 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.825 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:03:49.825 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.825 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:03:49.825 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:03:49.825 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.826 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:03:49.826 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:03:49.826 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.826 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:03:49.826 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:03:49.827 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.827 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:03:49.827 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:03:49.827 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.827 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:03:49.828 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.828 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:03:49.828 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:03:49.828 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:03:49.829 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:03:49.829 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:03:49.829 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:03:49.829 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:03:49.829 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:03:49.830 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:03:49.830 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:03:49.830 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.831 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:03:49.831 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.831 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:03:49.831 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.831 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:03:49.832 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.832 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:03:49.832 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:03:49.832 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:03:49.832 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:03:49.833 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:03:49.833 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:03:49.833 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.833 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:03:49.833 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:03:49.834 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.834 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:03:49.837 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:03:49.837 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.837 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:03:49.837 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:03:49.838 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.838 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:03:49.838 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:03:49.838 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:03:49.838 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.839 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:03:49.839 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:03:49.839 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.839 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:03:49.840 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.840 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:03:49.840 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.840 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:03:49.840 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:03:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:03:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:03:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:03:49.842 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:03:49.842 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:03:49.842 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:03:49.842 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:03:49.842 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:03:49.843 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:03:49.843 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:03:49.843 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:03:49.843 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:03:49.843 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:03:49.844 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:03:49.844 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:03:49.844 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:03:49.844 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:03:49.844 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:03:49.845 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:03:49.845 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:03:49.845 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:03:49.845 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.846 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:03:49.846 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:03:49.846 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:03:49.846 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:03:49.846 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:03:49.847 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:03:49.847 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:03:49.847 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:03:49.847 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.847 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:03:49.848 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:03:49.848 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:03:49.848 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:03:49.848 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:03:49.849 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:03:49.849 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:03:49.849 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:03:49.849 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:03:49.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:03:49.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:03:49.850 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:03:49.850 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:03:49.850 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:03:49.851 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:03:49.851 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:03:49.851 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:03:49.851 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:03:49.851 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:03:49.852 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:03:49.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:03:49.852 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:03:49.852 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:03:49.852 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:03:49.853 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:03:49.853 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:03:49.853 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:03:49.853 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:03:49.853 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:03:49.854 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:03:49.854 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:03:49.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:03:49.854 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:03:49.854 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:03:49.855 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:03:49.855 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:03:49.855 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:03:49.855 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:03:49.855 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:03:49.856 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:03:49.856 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:03:49.856 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:03:49.856 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:03:49.856 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:03:49.857 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:03:49.878 INFO:tasks.ceph:Creating admin key on mon.a... 2020-11-20T12:03:49.879 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2020-11-20T12:03:50.027 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-20T12:03:50.028 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:50.028 INFO:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2020-11-20T12:03:50.077 INFO:teuthology.orchestra.run.smithi187.stderr:+ dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2020-11-20T12:03:50.079 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.] 2020-11-20T12:03:50.080 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQClsLdfnIifERAAZNU9gkeeyc3HIS2rth/yRg== 2020-11-20T12:03:50.080 INFO:teuthology.orchestra.run.smithi187.stdout:[client.admin] 2020-11-20T12:03:50.080 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw== 2020-11-20T12:03:50.080 INFO:teuthology.orchestra.run.smithi187.stdout: caps mds = "allow *" 2020-11-20T12:03:50.081 INFO:teuthology.orchestra.run.smithi187.stdout: caps mgr = "allow *" 2020-11-20T12:03:50.081 INFO:teuthology.orchestra.run.smithi187.stdout: caps mon = "allow *" 2020-11-20T12:03:50.081 INFO:teuthology.orchestra.run.smithi187.stdout: caps osd = "allow *" 2020-11-20T12:03:50.081 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:03:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:03:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:206 bytes copied, 0.000126544 s, 1.6 MB/s 2020-11-20T12:03:50.082 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:50.083 INFO:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2020-11-20T12:03:50.131 INFO:teuthology.orchestra.run.smithi187.stderr:+ dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2020-11-20T12:03:50.133 INFO:teuthology.orchestra.run.smithi187.stdout: � ���PI �]��m���_�t!���_�t!�aXa �����bXb ���cXc ���abc 2020-11-20T12:03:50.134 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:03:50.134 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:03:50.134 INFO:teuthology.orchestra.run.smithi187.stderr:409 bytes copied, 0.00013386 s, 3.1 MB/s 2020-11-20T12:03:50.135 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:03:50.135 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:50.135 INFO:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T12:03:50.136 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:03:50.185 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T12:03:50.230 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:03:50.231 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:03:50.231 INFO:teuthology.orchestra.run.smithi187.stderr:206 bytes copied, 0.000168101 s, 1.2 MB/s 2020-11-20T12:03:50.234 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:03:50.299 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:03:50.300 INFO:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:03:50.350 INFO:teuthology.orchestra.run.smithi187.stderr:+ dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:03:50.352 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:03:50.352 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:03:50.352 INFO:teuthology.orchestra.run.smithi187.stderr:409 bytes copied, 0.000250706 s, 1.6 MB/s 2020-11-20T12:03:50.353 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T12:03:50.353 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:03:50.354 INFO:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T12:03:50.354 INFO:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:03:50.402 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T12:03:50.439 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:03:50.440 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:03:50.440 INFO:teuthology.orchestra.run.smithi005.stderr:206 bytes copied, 0.000117812 s, 1.7 MB/s 2020-11-20T12:03:50.442 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:03:50.482 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:03:50.482 INFO:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:03:50.519 INFO:teuthology.orchestra.run.smithi005.stderr:+ dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:03:50.521 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:03:50.521 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:03:50.521 INFO:teuthology.orchestra.run.smithi005.stderr:409 bytes copied, 9.8854e-05 s, 4.1 MB/s 2020-11-20T12:03:50.522 INFO:tasks.ceph:Setting up mon nodes... 2020-11-20T12:03:50.522 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-20T12:03:50.523 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2020-11-20T12:03:50.713 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2020-11-20T12:03:50.723 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2020-11-20T12:03:50.956 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2020-11-20T12:03:50.964 INFO:tasks.ceph:Setting up mds nodes... 2020-11-20T12:03:50.965 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2020-11-20T12:03:51.162 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2020-11-20T12:03:51.171 INFO:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2020-11-20T12:03:51.276 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2020-11-20T12:03:51.421 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2020-11-20T12:03:51.427 INFO:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2020-11-20T12:03:51.505 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-20T12:03:51.505 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2020-11-20T12:03:51.636 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.client.0.keyring 2020-11-20T12:03:51.691 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2020-11-20T12:03:51.812 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.client.1.keyring 2020-11-20T12:03:51.861 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2020-11-20T12:03:51.966 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.client.2.keyring 2020-11-20T12:03:52.015 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2020-11-20T12:03:52.149 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.client.3.keyring 2020-11-20T12:03:52.245 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-20T12:03:52.246 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2020-11-20T12:03:52.246 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2020-11-20T12:03:52.351 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:03:52.352 INFO:tasks.ceph:role: osd.0 2020-11-20T12:03:52.352 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:03:52.353 INFO:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-20T12:03:52.924 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:03:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:03:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T12:03:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 2020-11-20T12:03:52.925 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:03:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:03:52.925 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T12:03:52.926 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:03:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:03:52.926 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:03:52.933 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2020-11-20T12:03:52.933 INFO:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2020-11-20T12:03:53.135 INFO:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2020-11-20T12:03:53.252 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2020-11-20T12:03:53.349 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:03:53.349 INFO:tasks.ceph:role: osd.1 2020-11-20T12:03:53.350 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:03:53.350 INFO:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-20T12:03:53.892 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:03:53.892 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:03:53.893 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T12:03:53.893 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 2020-11-20T12:03:53.893 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:03:53.894 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:03:53.894 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T12:03:53.895 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:03:53.895 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:03:53.895 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:03:53.900 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2020-11-20T12:03:53.901 INFO:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2020-11-20T12:03:54.051 INFO:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2020-11-20T12:03:54.155 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2020-11-20T12:03:54.257 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:03:54.257 INFO:tasks.ceph:role: osd.2 2020-11-20T12:03:54.258 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:03:54.258 INFO:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-20T12:03:54.835 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:03:54.836 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:03:54.836 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T12:03:54.836 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 2020-11-20T12:03:54.836 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:03:54.836 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:03:54.837 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T12:03:54.837 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:03:54.837 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:03:54.837 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:03:54.843 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2020-11-20T12:03:54.843 INFO:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2020-11-20T12:03:55.045 INFO:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2020-11-20T12:03:55.157 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2020-11-20T12:03:55.259 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:03:55.259 INFO:tasks.ceph:role: osd.3 2020-11-20T12:03:55.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:03:55.260 INFO:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-11-20T12:03:55.816 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:03:55.816 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:03:55.817 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T12:03:55.817 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 2020-11-20T12:03:55.817 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:03:55.817 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:03:55.817 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T12:03:55.818 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:03:55.818 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:03:55.818 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:03:55.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2020-11-20T12:03:55.826 INFO:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2020-11-20T12:03:55.994 INFO:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2020-11-20T12:03:56.110 INFO:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:03:56.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:56.271+0000 7feb9bd62ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2020-11-20T12:03:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:56.272+0000 7feb9bd62ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2020-11-20T12:03:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:56.272+0000 7feb9bd62ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-20T12:03:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:56.272+0000 7feb9bd62ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-20T12:03:56.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:56.272+0000 7feb9bd62ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-20T12:03:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:56.283+0000 7feb9bd62ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2020-11-20T12:03:59.199 INFO:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2020-11-20T12:03:59.308 INFO:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:03:59.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:59.449+0000 7f9167f73ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2020-11-20T12:03:59.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:59.450+0000 7f9167f73ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2020-11-20T12:03:59.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:59.450+0000 7f9167f73ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-20T12:03:59.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:59.450+0000 7f9167f73ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-20T12:03:59.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:59.450+0000 7f9167f73ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-20T12:03:59.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:03:59.452+0000 7f9167f73ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2020-11-20T12:04:02.363 INFO:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2020-11-20T12:04:02.468 INFO:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:02.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:02.600+0000 7f4c5f010ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2020-11-20T12:04:02.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:02.600+0000 7f4c5f010ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2020-11-20T12:04:02.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:02.601+0000 7f4c5f010ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-20T12:04:02.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:02.601+0000 7f4c5f010ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-20T12:04:02.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:02.601+0000 7f4c5f010ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-20T12:04:02.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:02.603+0000 7f4c5f010ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2020-11-20T12:04:05.524 INFO:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2020-11-20T12:04:05.633 INFO:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:05.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:05.775+0000 7f0d74fa1ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2020-11-20T12:04:05.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:05.776+0000 7f0d74fa1ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2020-11-20T12:04:05.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:05.776+0000 7f0d74fa1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-11-20T12:04:05.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:05.776+0000 7f0d74fa1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-11-20T12:04:05.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:05.776+0000 7f0d74fa1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-11-20T12:04:05.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:05.778+0000 7f0d74fa1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2020-11-20T12:04:08.675 INFO:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2020-11-20T12:04:08.778 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2020-11-20T12:04:08.893 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:08.894 INFO:tasks.ceph:role: osd.4 2020-11-20T12:04:08.894 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T12:04:08.894 INFO:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-20T12:04:09.025 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:09.025 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:09.025 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T12:04:09.026 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2020-11-20T12:04:09.026 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:09.026 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:09.026 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T12:04:09.027 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:09.027 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:09.027 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:09.032 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2020-11-20T12:04:09.032 INFO:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2020-11-20T12:04:09.164 INFO:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2020-11-20T12:04:09.261 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2020-11-20T12:04:09.360 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:09.361 INFO:tasks.ceph:role: osd.5 2020-11-20T12:04:09.362 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T12:04:09.362 INFO:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-20T12:04:09.493 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:09.494 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:09.494 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T12:04:09.494 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2020-11-20T12:04:09.494 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:09.494 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:09.495 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T12:04:09.495 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:09.495 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:09.495 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:09.500 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2020-11-20T12:04:09.500 INFO:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2020-11-20T12:04:09.608 INFO:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2020-11-20T12:04:09.709 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2020-11-20T12:04:09.816 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:09.816 INFO:tasks.ceph:role: osd.6 2020-11-20T12:04:09.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T12:04:09.817 INFO:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-20T12:04:09.961 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:09.962 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:09.962 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T12:04:09.963 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2020-11-20T12:04:09.963 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:09.964 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:09.964 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T12:04:09.964 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:09.965 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:09.965 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:09.970 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2020-11-20T12:04:09.978 INFO:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2020-11-20T12:04:10.118 INFO:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2020-11-20T12:04:10.237 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2020-11-20T12:04:10.351 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:10.351 INFO:tasks.ceph:role: osd.7 2020-11-20T12:04:10.352 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T12:04:10.353 INFO:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-11-20T12:04:10.491 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T12:04:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2020-11-20T12:04:10.493 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:10.494 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T12:04:10.494 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:10.495 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:10.501 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2020-11-20T12:04:10.502 INFO:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2020-11-20T12:04:10.624 INFO:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2020-11-20T12:04:10.729 INFO:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:10.897 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:10.894+0000 7f2add0f6ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2020-11-20T12:04:10.898 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:10.895+0000 7f2add0f6ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2020-11-20T12:04:10.899 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:10.895+0000 7f2add0f6ec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-11-20T12:04:10.899 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:10.895+0000 7f2add0f6ec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-11-20T12:04:10.899 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:10.895+0000 7f2add0f6ec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-11-20T12:04:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:10.898+0000 7f2add0f6ec0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2020-11-20T12:04:13.804 INFO:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2020-11-20T12:04:13.915 INFO:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:14.060+0000 7fbda39cfec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2020-11-20T12:04:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:14.061+0000 7fbda39cfec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2020-11-20T12:04:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:14.061+0000 7fbda39cfec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-11-20T12:04:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:14.061+0000 7fbda39cfec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-11-20T12:04:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:14.061+0000 7fbda39cfec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-11-20T12:04:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:14.063+0000 7fbda39cfec0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2020-11-20T12:04:16.951 INFO:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2020-11-20T12:04:17.060 INFO:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:17.208 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:17.204+0000 7fbd7caa1ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2020-11-20T12:04:17.209 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:17.205+0000 7fbd7caa1ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2020-11-20T12:04:17.209 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:17.205+0000 7fbd7caa1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-11-20T12:04:17.209 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:17.205+0000 7fbd7caa1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-11-20T12:04:17.210 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:17.205+0000 7fbd7caa1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-11-20T12:04:17.210 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:17.207+0000 7fbd7caa1ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2020-11-20T12:04:17.618 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:04:17.624 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:04:20.100 INFO:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2020-11-20T12:04:20.226 INFO:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:20.377 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:20.374+0000 7fe9f4b8eec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2020-11-20T12:04:20.378 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:20.374+0000 7fe9f4b8eec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2020-11-20T12:04:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:20.374+0000 7fe9f4b8eec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-11-20T12:04:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:20.374+0000 7fe9f4b8eec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-11-20T12:04:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:20.374+0000 7fe9f4b8eec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-11-20T12:04:20.380 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:20.376+0000 7fe9f4b8eec0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2020-11-20T12:04:23.273 INFO:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2020-11-20T12:04:23.398 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-20T12:04:23.399 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:23.400 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2020-11-20T12:04:23.466 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2020-11-20T12:04:23.514 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.y] 2020-11-20T12:04:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ== 2020-11-20T12:04:23.515 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:23.516 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:23.516 INFO:teuthology.orchestra.run.smithi187.stderr:56 bytes copied, 0.000221697 s, 253 kB/s 2020-11-20T12:04:23.517 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:23.518 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2020-11-20T12:04:23.616 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2020-11-20T12:04:23.663 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:04:23.663 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w== 2020-11-20T12:04:23.664 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:23.664 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:23.664 INFO:teuthology.orchestra.run.smithi187.stderr:56 bytes copied, 0.000129956 s, 431 kB/s 2020-11-20T12:04:23.666 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:23.667 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2020-11-20T12:04:23.723 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2020-11-20T12:04:23.770 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2020-11-20T12:04:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA== 2020-11-20T12:04:23.771 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:23.771 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:23.771 INFO:teuthology.orchestra.run.smithi187.stderr:56 bytes copied, 0.000181586 s, 308 kB/s 2020-11-20T12:04:23.773 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:23.774 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2020-11-20T12:04:23.827 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2020-11-20T12:04:23.874 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2020-11-20T12:04:23.875 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ== 2020-11-20T12:04:23.875 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:23.875 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:23.876 INFO:teuthology.orchestra.run.smithi187.stderr:56 bytes copied, 0.00010796 s, 519 kB/s 2020-11-20T12:04:23.879 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:23.880 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2020-11-20T12:04:23.932 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2020-11-20T12:04:23.969 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2020-11-20T12:04:23.969 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A== 2020-11-20T12:04:23.970 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:23.970 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:23.970 INFO:teuthology.orchestra.run.smithi187.stderr:56 bytes copied, 0.000115016 s, 487 kB/s 2020-11-20T12:04:23.972 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:23.972 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2020-11-20T12:04:24.070 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2020-11-20T12:04:24.117 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.3] 2020-11-20T12:04:24.117 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg== 2020-11-20T12:04:24.118 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:24.118 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:24.118 INFO:teuthology.orchestra.run.smithi187.stderr:56 bytes copied, 0.000127072 s, 441 kB/s 2020-11-20T12:04:24.121 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:24.121 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2020-11-20T12:04:24.180 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2020-11-20T12:04:24.231 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2020-11-20T12:04:24.231 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg== 2020-11-20T12:04:24.232 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:24.232 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:24.232 INFO:teuthology.orchestra.run.smithi005.stderr:56 bytes copied, 0.000241789 s, 232 kB/s 2020-11-20T12:04:24.236 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:24.236 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2020-11-20T12:04:24.292 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2020-11-20T12:04:24.337 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:04:24.338 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ== 2020-11-20T12:04:24.338 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:24.338 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:24.338 INFO:teuthology.orchestra.run.smithi005.stderr:56 bytes copied, 0.000178704 s, 313 kB/s 2020-11-20T12:04:24.340 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:24.341 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2020-11-20T12:04:24.392 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2020-11-20T12:04:24.436 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.4] 2020-11-20T12:04:24.437 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ== 2020-11-20T12:04:24.437 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:24.437 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:24.437 INFO:teuthology.orchestra.run.smithi005.stderr:56 bytes copied, 0.000125815 s, 445 kB/s 2020-11-20T12:04:24.440 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:24.440 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2020-11-20T12:04:24.492 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2020-11-20T12:04:24.543 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.5] 2020-11-20T12:04:24.543 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ== 2020-11-20T12:04:24.544 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:24.544 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:24.544 INFO:teuthology.orchestra.run.smithi005.stderr:56 bytes copied, 0.000132484 s, 423 kB/s 2020-11-20T12:04:24.547 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:24.547 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2020-11-20T12:04:24.601 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2020-11-20T12:04:24.652 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.6] 2020-11-20T12:04:24.652 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg== 2020-11-20T12:04:24.653 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:24.653 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:24.654 INFO:teuthology.orchestra.run.smithi005.stderr:56 bytes copied, 0.000133928 s, 418 kB/s 2020-11-20T12:04:24.656 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:24.656 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2020-11-20T12:04:24.707 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2020-11-20T12:04:24.757 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.7] 2020-11-20T12:04:24.758 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ== 2020-11-20T12:04:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:56 bytes copied, 0.000130285 s, 430 kB/s 2020-11-20T12:04:24.760 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:24.760 INFO:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2020-11-20T12:04:24.817 INFO:teuthology.orchestra.run.smithi187.stderr:+ dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2020-11-20T12:04:24.820 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2020-11-20T12:04:24.820 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA== 2020-11-20T12:04:24.820 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:24.821 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:24.821 INFO:teuthology.orchestra.run.smithi187.stderr:59 bytes copied, 0.000143562 s, 411 kB/s 2020-11-20T12:04:24.821 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:24.822 INFO:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2020-11-20T12:04:24.871 INFO:teuthology.orchestra.run.smithi187.stderr:+ dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2020-11-20T12:04:24.873 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2020-11-20T12:04:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw== 2020-11-20T12:04:24.874 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:24.874 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:24.874 INFO:teuthology.orchestra.run.smithi187.stderr:59 bytes copied, 0.000152508 s, 387 kB/s 2020-11-20T12:04:24.875 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:24.876 INFO:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2020-11-20T12:04:24.934 INFO:teuthology.orchestra.run.smithi005.stderr:+ dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2020-11-20T12:04:24.938 INFO:teuthology.orchestra.run.smithi005.stdout:[client.2] 2020-11-20T12:04:24.939 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw== 2020-11-20T12:04:24.939 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:24.939 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:24.940 INFO:teuthology.orchestra.run.smithi005.stderr:59 bytes copied, 0.000147863 s, 399 kB/s 2020-11-20T12:04:24.940 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:24.941 INFO:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2020-11-20T12:04:24.996 INFO:teuthology.orchestra.run.smithi005.stderr:+ dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2020-11-20T12:04:24.998 INFO:teuthology.orchestra.run.smithi005.stdout:[client.3] 2020-11-20T12:04:24.999 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg== 2020-11-20T12:04:24.999 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:24.999 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:25.000 INFO:teuthology.orchestra.run.smithi005.stderr:59 bytes copied, 0.0001422 s, 415 kB/s 2020-11-20T12:04:25.000 INFO:tasks.ceph:Adding keys to all mons... 2020-11-20T12:04:25.001 INFO:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/ceph.keyring 2020-11-20T12:04:25.009 INFO:teuthology.orchestra.run.smithi187:> sudo tee -a /etc/ceph/ceph.keyring 2020-11-20T12:04:25.112 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.y] 2020-11-20T12:04:25.112 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ== 2020-11-20T12:04:25.112 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:04:25.113 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w== 2020-11-20T12:04:25.113 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2020-11-20T12:04:25.113 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA== 2020-11-20T12:04:25.113 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2020-11-20T12:04:25.114 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ== 2020-11-20T12:04:25.114 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2020-11-20T12:04:25.114 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A== 2020-11-20T12:04:25.114 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.3] 2020-11-20T12:04:25.115 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg== 2020-11-20T12:04:25.115 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2020-11-20T12:04:25.115 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg== 2020-11-20T12:04:25.115 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:04:25.115 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ== 2020-11-20T12:04:25.116 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.4] 2020-11-20T12:04:25.116 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ== 2020-11-20T12:04:25.116 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.5] 2020-11-20T12:04:25.116 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ== 2020-11-20T12:04:25.116 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.6] 2020-11-20T12:04:25.117 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg== 2020-11-20T12:04:25.117 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.7] 2020-11-20T12:04:25.117 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ== 2020-11-20T12:04:25.117 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2020-11-20T12:04:25.118 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA== 2020-11-20T12:04:25.118 INFO:teuthology.orchestra.run.smithi005.stdout:[client.1] 2020-11-20T12:04:25.118 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw== 2020-11-20T12:04:25.118 INFO:teuthology.orchestra.run.smithi005.stdout:[client.2] 2020-11-20T12:04:25.118 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw== 2020-11-20T12:04:25.119 INFO:teuthology.orchestra.run.smithi005.stdout:[client.3] 2020-11-20T12:04:25.119 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg== 2020-11-20T12:04:25.121 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.y] 2020-11-20T12:04:25.121 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ== 2020-11-20T12:04:25.122 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:04:25.122 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w== 2020-11-20T12:04:25.122 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2020-11-20T12:04:25.122 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA== 2020-11-20T12:04:25.123 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2020-11-20T12:04:25.123 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ== 2020-11-20T12:04:25.123 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2020-11-20T12:04:25.123 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A== 2020-11-20T12:04:25.123 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.3] 2020-11-20T12:04:25.124 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg== 2020-11-20T12:04:25.124 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.x] 2020-11-20T12:04:25.124 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg== 2020-11-20T12:04:25.124 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:04:25.124 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ== 2020-11-20T12:04:25.125 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.4] 2020-11-20T12:04:25.125 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ== 2020-11-20T12:04:25.125 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.5] 2020-11-20T12:04:25.125 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ== 2020-11-20T12:04:25.126 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.6] 2020-11-20T12:04:25.126 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg== 2020-11-20T12:04:25.126 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.7] 2020-11-20T12:04:25.126 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ== 2020-11-20T12:04:25.126 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2020-11-20T12:04:25.127 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA== 2020-11-20T12:04:25.127 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2020-11-20T12:04:25.127 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw== 2020-11-20T12:04:25.127 INFO:teuthology.orchestra.run.smithi187.stdout:[client.2] 2020-11-20T12:04:25.127 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw== 2020-11-20T12:04:25.128 INFO:teuthology.orchestra.run.smithi187.stdout:[client.3] 2020-11-20T12:04:25.128 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg== 2020-11-20T12:04:25.128 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T12:04:25.135 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T12:04:25.280 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T12:04:25.289 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T12:04:25.438 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:25.444 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:25.592 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:25.599 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:25.750 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:25.758 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:25.907 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:25.914 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.062 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T12:04:26.068 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T12:04:26.214 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T12:04:26.223 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T12:04:26.373 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.379 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.519 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.525 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.673 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.679 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.828 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.836 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:26.984 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:26.990 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:27.141 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:27.146 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:27.294 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:27.303 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:27.452 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:27.457 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:27.607 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-20T12:04:27.608 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2020-11-20T12:04:27.704 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T12:04:27.862 INFO:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2020-11-20T12:04:27.954 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2020-11-20T12:04:28.072 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T12:04:28.241 INFO:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2020-11-20T12:04:28.337 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2020-11-20T12:04:28.443 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T12:04:28.609 INFO:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2020-11-20T12:04:28.717 INFO:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:28.723 INFO:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:28.780 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2020-11-20T12:04:28.780 INFO:tasks.ceph.mon.a:Restarting daemon 2020-11-20T12:04:28.781 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-11-20T12:04:28.789 INFO:tasks.ceph.mon.a:Started 2020-11-20T12:04:28.789 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T12:04:28.790 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T12:04:28.796 INFO:tasks.ceph.mon.b:Started 2020-11-20T12:04:28.796 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T12:04:28.797 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T12:04:28.804 INFO:tasks.ceph.mon.c:Started 2020-11-20T12:04:28.804 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2020-11-20T12:04:28.805 INFO:tasks.ceph.mgr.y:Restarting daemon 2020-11-20T12:04:28.805 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2020-11-20T12:04:28.814 INFO:tasks.ceph.mgr.y:Started 2020-11-20T12:04:28.814 INFO:tasks.ceph.mgr.x:Restarting daemon 2020-11-20T12:04:28.814 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2020-11-20T12:04:28.819 INFO:tasks.ceph.mgr.x:Started 2020-11-20T12:04:28.819 INFO:tasks.ceph:Setting crush tunables to default 2020-11-20T12:04:28.819 INFO:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd crush tunables default 2020-11-20T12:04:29.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.033+0000 7f461d16b700 1 Processor -- start 2020-11-20T12:04:29.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.034+0000 7f461d16b700 1 -- start start 2020-11-20T12:04:29.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.035+0000 7f461d16b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:29.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.035+0000 7f461d16b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180b13e0 con 0x7f461809ca90 2020-11-20T12:04:29.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.035+0000 7f461d16b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4618062c10 con 0x7f4618065190 2020-11-20T12:04:29.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.035+0000 7f461d16b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461814ab30 con 0x7f4618065f90 2020-11-20T12:04:29.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.035+0000 7f4617fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4618065190 0x7f4618143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36312/0 (socket says 172.21.15.187:36312) 2020-11-20T12:04:29.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.035+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 learned_addr learned my addr 172.21.15.187:0/2179766438 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:04:29.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.035+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:29.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.036+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:29.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.036+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:29.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.036+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:29.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.036+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:29.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.238+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:29.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.238+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:29.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.238+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:29.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.238+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:29.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.238+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:29.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.639+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:29.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.640+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.640+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.640+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:29.640+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:30.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:30.442+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:30.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:30.442+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:30.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:30.442+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:30.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:30.442+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:30.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:30.442+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T12:04:32.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.035+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 msgr2=0x7f4618065570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:32.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.035+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f4618065f90 0x7f4618065570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:32.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.035+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 >> v1:172.21.15.187:6789/0 conn(0x7f4618065190 legacy=0x7f4618143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:32.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.035+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 >> v1:172.21.15.5:6789/0 conn(0x7f461809ca90 legacy=0x7f461809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:32.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.035+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:32.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.035+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:32.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.035+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461814ab30 con 0x7f45fc0033b0 2020-11-20T12:04:32.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.035+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0089d0 con 0x7f45fc005eb0 2020-11-20T12:04:32.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.036+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.036+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.036+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T12:04:32.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.036+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.036+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.036+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:32.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.037+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:32.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.037+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.037+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.037+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:32.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.237+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.238+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T12:04:32.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.238+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.238+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.238+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.238+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.239+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.239+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.239+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.239+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:32.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.640+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.640+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T12:04:32.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.640+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.640+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.640+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:32.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.641+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.641+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:32.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.641+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.641+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:32.641+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:33.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.442+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:33.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.442+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T12:04:33.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.442+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:33.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.442+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:33.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.443+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T12:04:33.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.443+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:33.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.443+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:33.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.443+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:33.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.443+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:33.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:33.443+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T12:04:35.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:35.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:35.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc005830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:35.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:35.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:35.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:35.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180b1640 con 0x7f4618065190 2020-11-20T12:04:35.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461814ab30 con 0x7f45fc0033b0 2020-11-20T12:04:35.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0089d0 con 0x7f45fc005eb0 2020-11-20T12:04:35.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.036+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 reap_dead start 2020-11-20T12:04:35.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:35.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:35.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:35.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T12:04:35.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:35.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:35.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:35.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:35.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:35.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.037+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:35.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.238+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:35.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:35.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T12:04:35.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:35.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:35.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:35.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:35.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:35.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:35.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.239+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:35.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:35.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:35.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:35.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:35.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:35.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T12:04:35.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:35.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:35.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.641+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:35.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:35.642+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:36.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.443+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:36.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.443+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:36.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.444+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:36.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.444+0000 7f460ffff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:36.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.444+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:36.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.444+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T12:04:36.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.444+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T12:04:36.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.444+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:36.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.444+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:36.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:36.444+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T12:04:38.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45fc008350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:38.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.036+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45fc008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 msgr2=0x7f45fc00c5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:38.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.036+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.187:3300/0 conn(0x7f45fc0033b0 0x7f45fc00c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 >> v1:172.21.15.187:6789/0 conn(0x7f4618065190 legacy=0x7f45fc00ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.036+0000 7f46167fc700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.036+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc0058e0 con 0x7f45fc00f400 2020-11-20T12:04:38.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.037+0000 7f46167fc700 1 -- 172.21.15.187:0/2179766438 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461814ab30 con 0x7f45fc00ca90 2020-11-20T12:04:38.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.038+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.038+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 msgr2=0x7f45fc00ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:38.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.038+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 msgr2=0x7f45fc00ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:38.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.038+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:38.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.039+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:38.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.241+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.241+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 msgr2=0x7f45fc00ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:38.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.241+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 msgr2=0x7f45fc00ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:38.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.241+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:38.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.241+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:38.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.642+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.643+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 msgr2=0x7f45fc00ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:38.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.643+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 msgr2=0x7f45fc00ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:38.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.643+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:38.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:38.643+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:39.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.099+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622619889 0 0) 0x7f45fc0058e0 con 0x7f45fc00f400 2020-11-20T12:04:39.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.099+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f8003240 con 0x7f45fc00f400 2020-11-20T12:04:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.101+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 903601980 0 0) 0x7f45f8003240 con 0x7f45fc00f400 2020-11-20T12:04:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.101+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 msgr2=0x7f45fc00ee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.101+0000 7f46177fe700 1 --2- 172.21.15.187:0/2179766438 >> v2:172.21.15.5:3300/0 conn(0x7f45fc00ca90 0x7f45fc00ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.101+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc014110 con 0x7f45fc00f400 2020-11-20T12:04:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.101+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4600017630 con 0x7f45fc00f400 2020-11-20T12:04:39.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.101+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 >> v1:172.21.15.5:6789/0 conn(0x7f45fc00f400 legacy=0x7f45fc011820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.102+0000 7f46177fe700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45f8003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.102+0000 7f46177fe700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f8003c20 0x7f45f80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.102+0000 7f46177fe700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f80049d0 0x7f45f8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.102+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461814ab30 con 0x7f45f80049d0 2020-11-20T12:04:39.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.102+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f8004210 con 0x7f45f8003c20 2020-11-20T12:04:39.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.102+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f800b730 con 0x7f45fc005eb0 2020-11-20T12:04:39.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.102+0000 7f461c96a700 1 -- 172.21.15.187:0/2179766438 reap_dead start 2020-11-20T12:04:39.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.103+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f8003c20 0x7f45f80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.103+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45f8003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.103+0000 7f461c96a700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f80049d0 0x7f45f8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.105+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f8003c20 msgr2=0x7f45f80044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:39.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.105+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f8003c20 0x7f45f80044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.105+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f80049d0 msgr2=0x7f45f8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:39.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.105+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f80049d0 0x7f45f8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.105+0000 7f4617fff700 1 -- 172.21.15.187:0/2179766438 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f8003240 con 0x7f45fc005eb0 2020-11-20T12:04:39.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.105+0000 7f460ffff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f8003c20 0x7f45f80044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:39.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.105+0000 7f4617fff700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45f8003730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4608005650 tx=0x7f4608009040).ready entity=mon.2 client_cookie=5403141bdc2f80de server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.106+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080070a0 con 0x7f45fc005eb0 2020-11-20T12:04:39.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.107+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460801a730 con 0x7f45fc005eb0 2020-11-20T12:04:39.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.107+0000 7f46177fe700 1 -- 172.21.15.187:0/2179766438 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460801aab0 con 0x7f45fc005eb0 2020-11-20T12:04:39.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 -- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f45f8003730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45f8003730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4608005650 tx=0x7f4608009040).stop 2020-11-20T12:04:39.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 -- 172.21.15.187:0/2179766438 shutdown_connections 2020-11-20T12:04:39.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f80049d0 0x7f45f8004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f45f8003730 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 --2- 172.21.15.187:0/2179766438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f8003c20 0x7f45f80044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 -- 172.21.15.187:0/2179766438 >> 172.21.15.187:0/2179766438 conn(0x7f46180bab30 msgr2=0x7f46180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 -- 172.21.15.187:0/2179766438 shutdown_connections 2020-11-20T12:04:39.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.109+0000 7f461d16b700 1 -- 172.21.15.187:0/2179766438 wait complete. 2020-11-20T12:04:39.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.113+0000 7f461d16b700 1 Processor -- start 2020-11-20T12:04:39.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.113+0000 7f461d16b700 1 -- start start 2020-11-20T12:04:39.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.114+0000 7f461d16b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f8003c20 0x7f4618148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.114+0000 7f461d16b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f80049d0 0x7f46180b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.114+0000 7f461d16b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f4618143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.114+0000 7f461d16b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4618147a60 con 0x7f45f8003c20 2020-11-20T12:04:39.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.114+0000 7f461d16b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4618147520 con 0x7f45f80049d0 2020-11-20T12:04:39.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.114+0000 7f461d16b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46181477c0 con 0x7f45fc005eb0 2020-11-20T12:04:39.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f460ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f80049d0 0x7f46180b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f460ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f80049d0 0x7f46180b78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:39834/0 (socket says 172.21.15.187:39834) 2020-11-20T12:04:39.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f460ffff700 1 -- 172.21.15.187:0/919324764 learned_addr learned my addr 172.21.15.187:0/919324764 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:39.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f4617fff700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f8003c20 0x7f4618148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f461c96a700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f4618143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f460ffff700 1 -- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 msgr2=0x7f4618143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f460ffff700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f4618143930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f460ffff700 1 -- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f8003c20 msgr2=0x7f4618148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f460ffff700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f8003c20 0x7f4618148340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f460ffff700 1 -- 172.21.15.187:0/919324764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4618147220 con 0x7f45f80049d0 2020-11-20T12:04:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.115+0000 7f4617fff700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f8003c20 0x7f4618148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.116+0000 7f460ffff700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f80049d0 0x7f46180b78b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f460001bb40 tx=0x7f4600008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.116+0000 7f4614ff9700 1 -- 172.21.15.187:0/919324764 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46000176c0 con 0x7f45f80049d0 2020-11-20T12:04:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.116+0000 7f4614ff9700 1 -- 172.21.15.187:0/919324764 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4600021040 con 0x7f45f80049d0 2020-11-20T12:04:39.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.116+0000 7f4614ff9700 1 -- 172.21.15.187:0/919324764 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4600018e60 con 0x7f45f80049d0 2020-11-20T12:04:39.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.116+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46180b7da0 con 0x7f45f80049d0 2020-11-20T12:04:39.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.116+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4618148070 con 0x7f45f80049d0 2020-11-20T12:04:39.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.118+0000 7f4614ff9700 1 -- 172.21.15.187:0/919324764 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f460002c030 con 0x7f45f80049d0 2020-11-20T12:04:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.118+0000 7f4614ff9700 1 -- 172.21.15.187:0/919324764 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f460002f7b0 con 0x7f45f80049d0 2020-11-20T12:04:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.118+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4618061a20 con 0x7f45f80049d0 2020-11-20T12:04:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.123+0000 7f4614ff9700 1 -- 172.21.15.187:0/919324764 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f4600034060 con 0x7f45f80049d0 2020-11-20T12:04:39.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.214+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f46180afc60 con 0x7f45f80049d0 2020-11-20T12:04:39.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.224+0000 7f4614ff9700 1 -- 172.21.15.187:0/919324764 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f460002a070 con 0x7f45f80049d0 2020-11-20T12:04:39.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.225+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f80049d0 msgr2=0x7f46180b78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.225+0000 7f461d16b700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f80049d0 0x7f46180b78b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f460001bb40 tx=0x7f4600008040).stop 2020-11-20T12:04:39.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.226+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 shutdown_connections 2020-11-20T12:04:39.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.226+0000 7f461d16b700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f45f8003c20 0x7f4618148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.226+0000 7f461d16b700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45fc005eb0 0x7f4618143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.226+0000 7f461d16b700 1 --2- 172.21.15.187:0/919324764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45f80049d0 0x7f46180b78b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.226+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 >> 172.21.15.187:0/919324764 conn(0x7f46180bab30 msgr2=0x7f4618145b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.226+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 shutdown_connections 2020-11-20T12:04:39.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:39.226+0000 7f461d16b700 1 -- 172.21.15.187:0/919324764 wait complete. 2020-11-20T12:04:39.232 INFO:teuthology.orchestra.run.smithi187.stderr:adjusted tunables profile to default 2020-11-20T12:04:39.251 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2020-11-20T12:04:39.251 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:39.252 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2020-11-20T12:04:39.297 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2020-11-20T12:04:39.341 INFO:teuthology.orchestra.run.smithi187.stdout:59e4fe7b-bbb5-405f-900c-7b2b145087c4 2020-11-20T12:04:39.342 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:39.342 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:39.343 INFO:teuthology.orchestra.run.smithi187.stderr:37 bytes copied, 0.000131885 s, 281 kB/s 2020-11-20T12:04:39.345 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:39.345 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2020-11-20T12:04:39.387 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2020-11-20T12:04:39.427 INFO:teuthology.orchestra.run.smithi187.stdout:3051fdba-a9b2-4e83-9bc7-acfd0e1036fc 2020-11-20T12:04:39.427 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:39.427 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:39.427 INFO:teuthology.orchestra.run.smithi187.stderr:37 bytes copied, 0.000120721 s, 306 kB/s 2020-11-20T12:04:39.429 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:39.429 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2020-11-20T12:04:39.517 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2020-11-20T12:04:39.557 INFO:teuthology.orchestra.run.smithi187.stdout:588f2d2b-cca4-4d3c-b599-ea8d32a04b26 2020-11-20T12:04:39.558 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:39.558 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:39.558 INFO:teuthology.orchestra.run.smithi187.stderr:37 bytes copied, 0.000133557 s, 277 kB/s 2020-11-20T12:04:39.561 INFO:teuthology.orchestra.run.smithi187:> set -ex 2020-11-20T12:04:39.561 INFO:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2020-11-20T12:04:39.605 INFO:teuthology.orchestra.run.smithi187.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2020-11-20T12:04:39.643 INFO:teuthology.orchestra.run.smithi187.stdout:e041ab46-c4c7-48ae-8144-caac199f94f7 2020-11-20T12:04:39.644 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2020-11-20T12:04:39.644 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2020-11-20T12:04:39.644 INFO:teuthology.orchestra.run.smithi187.stderr:37 bytes copied, 0.000146826 s, 252 kB/s 2020-11-20T12:04:39.646 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:39.646 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2020-11-20T12:04:39.700 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2020-11-20T12:04:39.750 INFO:teuthology.orchestra.run.smithi005.stdout:6adcc4eb-3632-47cf-a500-e134866924da 2020-11-20T12:04:39.751 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:39.751 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:39.751 INFO:teuthology.orchestra.run.smithi005.stderr:37 bytes copied, 0.00012926 s, 286 kB/s 2020-11-20T12:04:39.755 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:39.755 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2020-11-20T12:04:39.804 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2020-11-20T12:04:39.854 INFO:teuthology.orchestra.run.smithi005.stdout:77c9ee1b-3f40-4a15-84e0-8b89b96e5237 2020-11-20T12:04:39.855 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:39.855 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:39.855 INFO:teuthology.orchestra.run.smithi005.stderr:37 bytes copied, 0.000143827 s, 257 kB/s 2020-11-20T12:04:39.858 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:39.859 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2020-11-20T12:04:39.906 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2020-11-20T12:04:39.948 INFO:teuthology.orchestra.run.smithi005.stdout:d79cbb7e-2a05-4688-bd3d-1289b4b231bd 2020-11-20T12:04:39.949 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:39.949 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:39.949 INFO:teuthology.orchestra.run.smithi005.stderr:37 bytes copied, 0.000107675 s, 344 kB/s 2020-11-20T12:04:39.952 INFO:teuthology.orchestra.run.smithi005:> set -ex 2020-11-20T12:04:39.952 INFO:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2020-11-20T12:04:39.998 INFO:teuthology.orchestra.run.smithi005.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2020-11-20T12:04:40.043 INFO:teuthology.orchestra.run.smithi005.stdout:65c46353-4546-48a6-813e-3968184574c8 2020-11-20T12:04:40.044 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records in 2020-11-20T12:04:40.044 INFO:teuthology.orchestra.run.smithi005.stderr:0+1 records out 2020-11-20T12:04:40.044 INFO:teuthology.orchestra.run.smithi005.stderr:37 bytes copied, 0.000135018 s, 274 kB/s 2020-11-20T12:04:40.047 INFO:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 59e4fe7b-bbb5-405f-900c-7b2b145087c4 0 2020-11-20T12:04:40.273 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.270+0000 7f072a525700 1 Processor -- start 2020-11-20T12:04:40.274 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.270+0000 7f072a525700 1 -- start start 2020-11-20T12:04:40.275 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.271+0000 7f072a525700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724099da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.275 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.271+0000 7f072a525700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f07240b52f0 0x7f072409a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.275 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.271+0000 7f072a525700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0724148b60 con 0x7f0724066210 2020-11-20T12:04:40.276 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.271+0000 7f072a525700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0724146020 con 0x7f07240b52f0 2020-11-20T12:04:40.276 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.271+0000 7f072a525700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0724149540 con 0x7f0724065e30 2020-11-20T12:04:40.278 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0728ac2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f07240b52f0 0x7f072409a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.278 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0723fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.278 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0728ac2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f07240b52f0 0x7f072409a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44304/0 (socket says 172.21.15.5:44304) 2020-11-20T12:04:40.278 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0728ac2700 1 -- 172.21.15.5:0/1575214483 learned_addr learned my addr 172.21.15.5:0/1575214483 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:40.278 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0728ac2700 1 -- 172.21.15.5:0/1575214483 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 msgr2=0x7f0724099da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:40.279 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0728ac2700 1 --2- 172.21.15.5:0/1575214483 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724099da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.279 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0728ac2700 1 -- 172.21.15.5:0/1575214483 >> v1:172.21.15.187:6789/0 conn(0x7f0724066210 legacy=0x7f0724061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.279 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0728ac2700 1 -- 172.21.15.5:0/1575214483 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07240b72c0 con 0x7f07240b52f0 2020-11-20T12:04:40.279 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.272+0000 7f0723fff700 1 --2- 172.21.15.5:0/1575214483 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.273+0000 7f0728ac2700 1 --2- 172.21.15.5:0/1575214483 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07240b52f0 0x7f072409a290 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f071c009d50 tx=0x7f071c009290).ready entity=mon.0 client_cookie=fda00cc7b6d1e01b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.273+0000 7f0722ffd700 1 -- 172.21.15.5:0/1575214483 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c00bbc0 con 0x7f07240b52f0 2020-11-20T12:04:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.273+0000 7f0722ffd700 1 -- 172.21.15.5:0/1575214483 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071c00bd20 con 0x7f07240b52f0 2020-11-20T12:04:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.273+0000 7f0722ffd700 1 -- 172.21.15.5:0/1575214483 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c011610 con 0x7f07240b52f0 2020-11-20T12:04:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.274+0000 7f072a525700 1 -- 172.21.15.5:0/1575214483 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07240b52f0 msgr2=0x7f072409a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.281 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.274+0000 7f072a525700 1 --2- 172.21.15.5:0/1575214483 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07240b52f0 0x7f072409a290 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f071c009d50 tx=0x7f071c009290).stop 2020-11-20T12:04:40.281 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.274+0000 7f072a525700 1 -- 172.21.15.5:0/1575214483 shutdown_connections 2020-11-20T12:04:40.281 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.274+0000 7f072a525700 1 --2- 172.21.15.5:0/1575214483 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07240b52f0 0x7f072409a290 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.281 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.274+0000 7f072a525700 1 --2- 172.21.15.5:0/1575214483 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.282 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.274+0000 7f072a525700 1 -- 172.21.15.5:0/1575214483 >> 172.21.15.5:0/1575214483 conn(0x7f07240bacf0 msgr2=0x7f07240bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:40.282 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.274+0000 7f072a525700 1 -- 172.21.15.5:0/1575214483 shutdown_connections 2020-11-20T12:04:40.282 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.274+0000 7f072a525700 1 -- 172.21.15.5:0/1575214483 wait complete. 2020-11-20T12:04:40.284 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.276+0000 7f072a525700 1 Processor -- start 2020-11-20T12:04:40.285 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.277+0000 7f072a525700 1 -- start start 2020-11-20T12:04:40.285 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.278+0000 7f072a525700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.285 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.278+0000 7f072a525700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0724066210 0x7f07240b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.285 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.278+0000 7f072a525700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07240b52f0 0x7f07241436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.285 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.278+0000 7f072a525700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07240b2e40 con 0x7f0724066210 2020-11-20T12:04:40.286 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.278+0000 7f072a525700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07240b30e0 con 0x7f07240b52f0 2020-11-20T12:04:40.286 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.278+0000 7f072a525700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0724157130 con 0x7f0724065e30 2020-11-20T12:04:40.286 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.278+0000 7f0723fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.286 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f07237fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0724066210 0x7f07240b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.287 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0723fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40588/0 (socket says 172.21.15.5:40588) 2020-11-20T12:04:40.287 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0723fff700 1 -- 172.21.15.5:0/3622121430 learned_addr learned my addr 172.21.15.5:0/3622121430 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:40.287 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0728ac2700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07240b52f0 0x7f07241436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.287 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0723fff700 1 -- 172.21.15.5:0/3622121430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07240b52f0 msgr2=0x7f07241436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.288 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0723fff700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07240b52f0 0x7f07241436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.288 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0723fff700 1 -- 172.21.15.5:0/3622121430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0724066210 msgr2=0x7f07240b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.288 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0723fff700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0724066210 0x7f07240b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.288 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0723fff700 1 -- 172.21.15.5:0/3622121430 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07241458a0 con 0x7f0724065e30 2020-11-20T12:04:40.288 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f07237fe700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0724066210 0x7f07240b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:40.289 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.279+0000 7f0723fff700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724156b30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f071400ef50 tx=0x7f0714006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:40.289 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.280+0000 7f0703fff700 1 -- 172.21.15.5:0/3622121430 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071400cb30 con 0x7f0724065e30 2020-11-20T12:04:40.289 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.280+0000 7f0703fff700 1 -- 172.21.15.5:0/3622121430 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071400cc90 con 0x7f0724065e30 2020-11-20T12:04:40.289 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.280+0000 7f0703fff700 1 -- 172.21.15.5:0/3622121430 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714012600 con 0x7f0724065e30 2020-11-20T12:04:40.290 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.280+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07240b4bb0 con 0x7f0724065e30 2020-11-20T12:04:40.290 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.280+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0724143dd0 con 0x7f0724065e30 2020-11-20T12:04:40.290 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.280+0000 7f0703fff700 1 -- 172.21.15.5:0/3622121430 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0714012bd0 con 0x7f0724065e30 2020-11-20T12:04:40.290 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.280+0000 7f0703fff700 1 -- 172.21.15.5:0/3622121430 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f071401cb10 con 0x7f0724065e30 2020-11-20T12:04:40.291 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.281+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0724044f60 con 0x7f0724065e30 2020-11-20T12:04:40.291 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.286+0000 7f0703fff700 1 -- 172.21.15.5:0/3622121430 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f071400ce00 con 0x7f0724065e30 2020-11-20T12:04:40.393 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.390+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd new", "uuid": "59e4fe7b-bbb5-405f-900c-7b2b145087c4", "id": 0} v 0) v1 -- 0x7f0724068690 con 0x7f0724065e30 2020-11-20T12:04:40.400 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.396+0000 7f0703fff700 1 -- 172.21.15.5:0/3622121430 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "59e4fe7b-bbb5-405f-900c-7b2b145087c4", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f071401cd50 con 0x7f0724065e30 2020-11-20T12:04:40.400 INFO:teuthology.orchestra.run.smithi005.stdout:0 2020-11-20T12:04:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 msgr2=0x7f0724156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.402 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724156b30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f071400ef50 tx=0x7f0714006df0).stop 2020-11-20T12:04:40.402 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 shutdown_connections 2020-11-20T12:04:40.402 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07240b52f0 0x7f07241436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.402 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0724065e30 0x7f0724156b30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.403 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 --2- 172.21.15.5:0/3622121430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0724066210 0x7f07240b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.403 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 >> 172.21.15.5:0/3622121430 conn(0x7f07240bacf0 msgr2=0x7f07240b6270 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:40.403 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 shutdown_connections 2020-11-20T12:04:40.403 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.398+0000 7f072a525700 1 -- 172.21.15.5:0/3622121430 wait complete. 2020-11-20T12:04:40.421 INFO:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 3051fdba-a9b2-4e83-9bc7-acfd0e1036fc 1 2020-11-20T12:04:40.630 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.626+0000 7f9d232a3700 1 Processor -- start 2020-11-20T12:04:40.630 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.626+0000 7f9d232a3700 1 -- start start 2020-11-20T12:04:40.632 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d232a3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.632 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d232a3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9d1c1426f0 0x7f9d1c142b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.632 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d232a3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1c149930 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.633 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d232a3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d1c0a3560 con 0x7f9d1c1426f0 2020-11-20T12:04:40.633 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d232a3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c14a330 con 0x7f9d1c0bbe80 2020-11-20T12:04:40.633 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d21aa0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.633 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d21aa0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c0a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40594/0 (socket says 172.21.15.5:40594) 2020-11-20T12:04:40.634 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d21aa0700 1 -- 172.21.15.5:0/3992426688 learned_addr learned my addr 172.21.15.5:0/3992426688 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:40.634 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.628+0000 7f9d22aa2700 1 --2- 172.21.15.5:0/3992426688 >> v2:172.21.15.5:3300/0 conn(0x7f9d1c1426f0 0x7f9d1c142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.634 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.629+0000 7f9d21aa0700 1 -- 172.21.15.5:0/3992426688 >> v2:172.21.15.5:3300/0 conn(0x7f9d1c1426f0 msgr2=0x7f9d1c142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:40.634 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.629+0000 7f9d21aa0700 1 --2- 172.21.15.5:0/3992426688 >> v2:172.21.15.5:3300/0 conn(0x7f9d1c1426f0 0x7f9d1c142b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.634 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.629+0000 7f9d21aa0700 1 -- 172.21.15.5:0/3992426688 >> v1:172.21.15.187:6789/0 conn(0x7f9d1c0bbaa0 legacy=0x7f9d1c149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.635 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.629+0000 7f9d21aa0700 1 -- 172.21.15.5:0/3992426688 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c0a3250 con 0x7f9d1c0bbe80 2020-11-20T12:04:40.638 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.629+0000 7f9d21aa0700 1 --2- 172.21.15.5:0/3992426688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c0a4bf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9d0c006fd0 tx=0x7f9d0c006eb0).ready entity=mon.2 client_cookie=808d3abfede54942 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:40.639 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.629+0000 7f9d2129f700 1 -- 172.21.15.5:0/3992426688 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c009810 con 0x7f9d1c0bbe80 2020-11-20T12:04:40.639 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.629+0000 7f9d2129f700 1 -- 172.21.15.5:0/3992426688 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0c009970 con 0x7f9d1c0bbe80 2020-11-20T12:04:40.639 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.629+0000 7f9d2129f700 1 -- 172.21.15.5:0/3992426688 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c009cf0 con 0x7f9d1c0bbe80 2020-11-20T12:04:40.639 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.630+0000 7f9d232a3700 1 -- 172.21.15.5:0/3992426688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 msgr2=0x7f9d1c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.640 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.630+0000 7f9d232a3700 1 --2- 172.21.15.5:0/3992426688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c0a4bf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9d0c006fd0 tx=0x7f9d0c006eb0).stop 2020-11-20T12:04:40.640 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.630+0000 7f9d232a3700 1 -- 172.21.15.5:0/3992426688 shutdown_connections 2020-11-20T12:04:40.640 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.630+0000 7f9d232a3700 1 --2- 172.21.15.5:0/3992426688 >> v2:172.21.15.5:3300/0 conn(0x7f9d1c1426f0 0x7f9d1c142b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.640 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.630+0000 7f9d232a3700 1 --2- 172.21.15.5:0/3992426688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c0a4bf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.640 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.630+0000 7f9d232a3700 1 -- 172.21.15.5:0/3992426688 >> 172.21.15.5:0/3992426688 conn(0x7f9d1c061bd0 msgr2=0x7f9d1c0678a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:40.641 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.630+0000 7f9d232a3700 1 -- 172.21.15.5:0/3992426688 shutdown_connections 2020-11-20T12:04:40.641 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.630+0000 7f9d232a3700 1 -- 172.21.15.5:0/3992426688 wait complete. 2020-11-20T12:04:40.641 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.631+0000 7f9d232a3700 1 Processor -- start 2020-11-20T12:04:40.641 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.632+0000 7f9d232a3700 1 -- start start 2020-11-20T12:04:40.642 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.632+0000 7f9d232a3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9d1c0bbaa0 0x7f9d1c143160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.642 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.632+0000 7f9d232a3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.643 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.632+0000 7f9d232a3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d1c1426f0 0x7f9d1c0a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.643 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.632+0000 7f9d232a3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c1352b0 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.643 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.632+0000 7f9d232a3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c15c330 con 0x7f9d1c1426f0 2020-11-20T12:04:40.644 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.632+0000 7f9d232a3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c15c5d0 con 0x7f9d1c0bbe80 2020-11-20T12:04:40.644 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d222a1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9d1c0bbaa0 0x7f9d1c143160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.645 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d21aa0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.645 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d222a1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9d1c0bbaa0 0x7f9d1c143160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44318/0 (socket says 172.21.15.5:44318) 2020-11-20T12:04:40.646 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d222a1700 1 -- 172.21.15.5:0/1271760369 learned_addr learned my addr 172.21.15.5:0/1271760369 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:40.646 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d22aa2700 1 --2- 172.21.15.5:0/1271760369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d1c1426f0 0x7f9d1c0a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.646 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d222a1700 1 -- 172.21.15.5:0/1271760369 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 msgr2=0x7f9d1c145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.647 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d222a1700 1 --2- 172.21.15.5:0/1271760369 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c145600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.647 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d222a1700 1 -- 172.21.15.5:0/1271760369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d1c1426f0 msgr2=0x7f9d1c0a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.648 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d222a1700 1 --2- 172.21.15.5:0/1271760369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d1c1426f0 0x7f9d1c0a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.648 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.633+0000 7f9d222a1700 1 -- 172.21.15.5:0/1271760369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c0a1320 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.648 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.634+0000 7f9d222a1700 1 --2- 172.21.15.5:0/1271760369 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9d1c0bbaa0 0x7f9d1c143160 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f9d1800c710 tx=0x7f9d18007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:40.649 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.634+0000 7f9d0a7fc700 1 -- 172.21.15.5:0/1271760369 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800e560 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.649 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.634+0000 7f9d0a7fc700 1 -- 172.21.15.5:0/1271760369 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1800e6c0 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.650 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.634+0000 7f9d0a7fc700 1 -- 172.21.15.5:0/1271760369 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800ea60 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.650 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.634+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1c144450 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.650 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.634+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1c1448f0 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.651 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.634+0000 7f9d0a7fc700 1 -- 172.21.15.5:0/1271760369 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9d18019020 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.652 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.634+0000 7f9d0a7fc700 1 -- 172.21.15.5:0/1271760369 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7f9d1800edd0 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.652 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.635+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1c15c790 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.653 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.638+0000 7f9d0a7fc700 1 -- 172.21.15.5:0/1271760369 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f9d1801c7e0 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.743 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.739+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3051fdba-a9b2-4e83-9bc7-acfd0e1036fc", "id": 1} v 0) v1 -- 0x7f9d1c0a4790 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.748 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.744+0000 7f9d0a7fc700 1 -- 172.21.15.5:0/1271760369 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3051fdba-a9b2-4e83-9bc7-acfd0e1036fc", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9d1801c980 con 0x7f9d1c0bbaa0 2020-11-20T12:04:40.749 INFO:teuthology.orchestra.run.smithi005.stdout:1 2020-11-20T12:04:40.751 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9d1c0bbaa0 msgr2=0x7f9d1c143160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.751 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 --2- 172.21.15.5:0/1271760369 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9d1c0bbaa0 0x7f9d1c143160 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f9d1800c710 tx=0x7f9d18007c10).stop 2020-11-20T12:04:40.751 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 shutdown_connections 2020-11-20T12:04:40.751 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 --2- 172.21.15.5:0/1271760369 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9d1c0bbaa0 0x7f9d1c143160 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.752 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 --2- 172.21.15.5:0/1271760369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d1c1426f0 0x7f9d1c0a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.752 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 --2- 172.21.15.5:0/1271760369 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9d1c0bbe80 0x7f9d1c145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.752 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 >> 172.21.15.5:0/1271760369 conn(0x7f9d1c061bd0 msgr2=0x7f9d1c140d40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:40.753 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 shutdown_connections 2020-11-20T12:04:40.753 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:40.747+0000 7f9d232a3700 1 -- 172.21.15.5:0/1271760369 wait complete. 2020-11-20T12:04:40.771 INFO:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 588f2d2b-cca4-4d3c-b599-ea8d32a04b26 2 2020-11-20T12:04:41.019 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.015+0000 7f338b59e700 1 Processor -- start 2020-11-20T12:04:41.019 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.015+0000 7f338b59e700 1 -- start start 2020-11-20T12:04:41.024 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.016+0000 7f338b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f338c0bfbe0 0x7f338c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.024 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.016+0000 7f338b59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f338c09f560 0x7f338c1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.024 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.016+0000 7f338b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c14d110 con 0x7f338c0bf800 2020-11-20T12:04:41.024 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.016+0000 7f338b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338c14c9d0 con 0x7f338c0bfbe0 2020-11-20T12:04:41.025 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.016+0000 7f338b59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338c148aa0 con 0x7f338c09f560 2020-11-20T12:04:41.025 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f3389d9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f338c0bfbe0 0x7f338c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.025 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f3389d9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f338c0bfbe0 0x7f338c1480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44328/0 (socket says 172.21.15.5:44328) 2020-11-20T12:04:41.025 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f3389d9b700 1 -- 172.21.15.5:0/3763725718 learned_addr learned my addr 172.21.15.5:0/3763725718 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:41.025 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f338ad9d700 1 --2- 172.21.15.5:0/3763725718 >> v2:172.21.15.187:3300/0 conn(0x7f338c09f560 0x7f338c1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.026 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f3389d9b700 1 -- 172.21.15.5:0/3763725718 >> v2:172.21.15.187:3300/0 conn(0x7f338c09f560 msgr2=0x7f338c1485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:41.026 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f3389d9b700 1 --2- 172.21.15.5:0/3763725718 >> v2:172.21.15.187:3300/0 conn(0x7f338c09f560 0x7f338c1485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.026 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f3389d9b700 1 -- 172.21.15.5:0/3763725718 >> v1:172.21.15.187:6789/0 conn(0x7f338c0bf800 legacy=0x7f338c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:41.026 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f3389d9b700 1 -- 172.21.15.5:0/3763725718 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c14cbd0 con 0x7f338c0bfbe0 2020-11-20T12:04:41.027 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.017+0000 7f3389d9b700 1 --2- 172.21.15.5:0/3763725718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c0bfbe0 0x7f338c1480c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f338c09f2b0 tx=0x7f3380006eb0).ready entity=mon.0 client_cookie=4a6cfe99c9cb77cf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:41.027 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338959a700 1 -- 172.21.15.5:0/3763725718 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380009920 con 0x7f338c0bfbe0 2020-11-20T12:04:41.027 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338959a700 1 -- 172.21.15.5:0/3763725718 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3380009a80 con 0x7f338c0bfbe0 2020-11-20T12:04:41.027 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338959a700 1 -- 172.21.15.5:0/3763725718 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380009e00 con 0x7f338c0bfbe0 2020-11-20T12:04:41.028 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338b59e700 1 -- 172.21.15.5:0/3763725718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c0bfbe0 msgr2=0x7f338c1480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.028 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338b59e700 1 --2- 172.21.15.5:0/3763725718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c0bfbe0 0x7f338c1480c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f338c09f2b0 tx=0x7f3380006eb0).stop 2020-11-20T12:04:41.028 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338b59e700 1 -- 172.21.15.5:0/3763725718 shutdown_connections 2020-11-20T12:04:41.028 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338b59e700 1 --2- 172.21.15.5:0/3763725718 >> v2:172.21.15.187:3300/0 conn(0x7f338c09f560 0x7f338c1485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.029 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338b59e700 1 --2- 172.21.15.5:0/3763725718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c0bfbe0 0x7f338c1480c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.029 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.018+0000 7f338b59e700 1 -- 172.21.15.5:0/3763725718 >> 172.21.15.5:0/3763725718 conn(0x7f338c068370 msgr2=0x7f338c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:41.029 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.019+0000 7f338b59e700 1 -- 172.21.15.5:0/3763725718 shutdown_connections 2020-11-20T12:04:41.029 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.019+0000 7f338b59e700 1 -- 172.21.15.5:0/3763725718 wait complete. 2020-11-20T12:04:41.029 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.020+0000 7f338b59e700 1 Processor -- start 2020-11-20T12:04:41.030 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.020+0000 7f338b59e700 1 -- start start 2020-11-20T12:04:41.032 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f338b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f338c09f560 0x7f338c066b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.032 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f338b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f338c0bf800 0x7f338c147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.032 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f338b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c066300 0x7f338c147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.033 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f338b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c065920 con 0x7f338c066300 2020-11-20T12:04:41.033 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f338b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c065bc0 con 0x7f338c09f560 2020-11-20T12:04:41.033 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f338b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338c065e60 con 0x7f338c0bf800 2020-11-20T12:04:41.034 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f3389d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f338c0bf800 0x7f338c147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.034 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f3389d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f338c0bf800 0x7f338c147260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40614/0 (socket says 172.21.15.5:40614) 2020-11-20T12:04:41.035 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f3389d9b700 1 -- 172.21.15.5:0/3824027325 learned_addr learned my addr 172.21.15.5:0/3824027325 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:41.035 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.021+0000 7f338ad9d700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c066300 0x7f338c147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.035 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f338a59c700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f338c09f560 0x7f338c066b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.036 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f3389d9b700 1 -- 172.21.15.5:0/3824027325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f338c09f560 msgr2=0x7f338c066b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.036 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f3389d9b700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f338c09f560 0x7f338c066b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.037 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f3389d9b700 1 -- 172.21.15.5:0/3824027325 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c066300 msgr2=0x7f338c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.037 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f3389d9b700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c066300 0x7f338c147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.037 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f3389d9b700 1 -- 172.21.15.5:0/3824027325 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c065410 con 0x7f338c0bf800 2020-11-20T12:04:41.038 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f3389d9b700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f338c0bf800 0x7f338c147260 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f338c0bca50 tx=0x7f3380015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:41.038 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f337a7fc700 1 -- 172.21.15.5:0/3824027325 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380009920 con 0x7f338c0bf800 2020-11-20T12:04:41.039 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.022+0000 7f337a7fc700 1 -- 172.21.15.5:0/3824027325 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338001d5d0 con 0x7f338c0bf800 2020-11-20T12:04:41.039 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.023+0000 7f337a7fc700 1 -- 172.21.15.5:0/3824027325 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338001d970 con 0x7f338c0bf800 2020-11-20T12:04:41.039 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.023+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c065600 con 0x7f338c0bf800 2020-11-20T12:04:41.040 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.023+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c067270 con 0x7f338c0bf800 2020-11-20T12:04:41.040 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.023+0000 7f337a7fc700 1 -- 172.21.15.5:0/3824027325 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f3380025030 con 0x7f338c0bf800 2020-11-20T12:04:41.041 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.023+0000 7f337a7fc700 1 -- 172.21.15.5:0/3824027325 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f3380028890 con 0x7f338c0bf800 2020-11-20T12:04:41.041 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.024+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338c14c770 con 0x7f338c0bf800 2020-11-20T12:04:41.042 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.028+0000 7f337a7fc700 1 -- 172.21.15.5:0/3824027325 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f338002d060 con 0x7f338c0bf800 2020-11-20T12:04:41.134 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.130+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd new", "uuid": "588f2d2b-cca4-4d3c-b599-ea8d32a04b26", "id": 2} v 0) v1 -- 0x7f338c080050 con 0x7f338c0bf800 2020-11-20T12:04:41.140 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.136+0000 7f337a7fc700 1 -- 172.21.15.5:0/3824027325 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "588f2d2b-cca4-4d3c-b599-ea8d32a04b26", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3380023070 con 0x7f338c0bf800 2020-11-20T12:04:41.140 INFO:teuthology.orchestra.run.smithi005.stdout:2 2020-11-20T12:04:41.142 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f338c0bf800 msgr2=0x7f338c147260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.142 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f338c0bf800 0x7f338c147260 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f338c0bca50 tx=0x7f3380015000).stop 2020-11-20T12:04:41.142 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 shutdown_connections 2020-11-20T12:04:41.143 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f338c066300 0x7f338c147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.143 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f338c09f560 0x7f338c066b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.143 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 --2- 172.21.15.5:0/3824027325 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f338c0bf800 0x7f338c147260 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.143 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 >> 172.21.15.5:0/3824027325 conn(0x7f338c068370 msgr2=0x7f338c0be490 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:41.143 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 shutdown_connections 2020-11-20T12:04:41.144 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.138+0000 7f338b59e700 1 -- 172.21.15.5:0/3824027325 wait complete. 2020-11-20T12:04:41.161 INFO:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new e041ab46-c4c7-48ae-8144-caac199f94f7 3 2020-11-20T12:04:41.380 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.377+0000 7f26aefb7700 1 Processor -- start 2020-11-20T12:04:41.381 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.377+0000 7f26aefb7700 1 -- start start 2020-11-20T12:04:41.382 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.378+0000 7f26aefb7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f26a80b6650 0x7f26a80afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.383 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.379+0000 7f26aefb7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a80b1300 con 0x7f26a8066f70 2020-11-20T12:04:41.383 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.379+0000 7f26aefb7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a80b15a0 con 0x7f26a81484d0 2020-11-20T12:04:41.383 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.379+0000 7f26aefb7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a8067730 con 0x7f26a80b6650 2020-11-20T12:04:41.384 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.379+0000 7f26ad7b4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f26a80b6650 0x7f26a80afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.384 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.379+0000 7f26ae7b6700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f26a8066f70 0x7f26a80aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:39020/0 (socket says 172.21.15.5:39020) 2020-11-20T12:04:41.384 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.379+0000 7f26ae7b6700 1 -- 172.21.15.5:0/38821436 learned_addr learned my addr 172.21.15.5:0/38821436 (peer_addr_for_me v1:172.21.15.5:0/0) 2020-11-20T12:04:41.386 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.379+0000 7f26ad7b4700 1 -- 172.21.15.5:0/38821436 >> v1:172.21.15.187:6789/0 conn(0x7f26a81484d0 legacy=0x7f26a8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.386 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.379+0000 7f26ad7b4700 1 -- 172.21.15.5:0/38821436 >> v1:172.21.15.5:6789/0 conn(0x7f26a8066f70 legacy=0x7f26a80aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.386 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.380+0000 7f26ad7b4700 1 -- 172.21.15.5:0/38821436 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a80b1840 con 0x7f26a80b6650 2020-11-20T12:04:41.386 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.380+0000 7f26ad7b4700 1 --2- 172.21.15.5:0/38821436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a80b6650 0x7f26a80afa90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f269800b9a0 tx=0x7f2698006df0).ready entity=mon.0 client_cookie=1d9673f50ad48578 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:41.387 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.380+0000 7f26acfb3700 1 -- 172.21.15.5:0/38821436 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26980097b0 con 0x7f26a80b6650 2020-11-20T12:04:41.387 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.380+0000 7f26acfb3700 1 -- 172.21.15.5:0/38821436 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2698009910 con 0x7f26a80b6650 2020-11-20T12:04:41.387 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.380+0000 7f26acfb3700 1 -- 172.21.15.5:0/38821436 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698009c90 con 0x7f26a80b6650 2020-11-20T12:04:41.387 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.381+0000 7f26aefb7700 1 -- 172.21.15.5:0/38821436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a80b6650 msgr2=0x7f26a80afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.388 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.381+0000 7f26aefb7700 1 --2- 172.21.15.5:0/38821436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a80b6650 0x7f26a80afa90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f269800b9a0 tx=0x7f2698006df0).stop 2020-11-20T12:04:41.388 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.381+0000 7f26aefb7700 1 -- 172.21.15.5:0/38821436 shutdown_connections 2020-11-20T12:04:41.388 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.381+0000 7f26aefb7700 1 --2- 172.21.15.5:0/38821436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a80b6650 0x7f26a80afa90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.388 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.381+0000 7f26aefb7700 1 -- 172.21.15.5:0/38821436 >> 172.21.15.5:0/38821436 conn(0x7f26a80bab30 msgr2=0x7f26a80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:41.389 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.381+0000 7f26aefb7700 1 -- 172.21.15.5:0/38821436 shutdown_connections 2020-11-20T12:04:41.389 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.381+0000 7f26aefb7700 1 -- 172.21.15.5:0/38821436 wait complete. 2020-11-20T12:04:41.391 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.382+0000 7f26aefb7700 1 Processor -- start 2020-11-20T12:04:41.391 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.383+0000 7f26aefb7700 1 -- start start 2020-11-20T12:04:41.392 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.383+0000 7f26aefb7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26a8066f70 0x7f26a813ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.392 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.383+0000 7f26aefb7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26a80b6650 0x7f26a813bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.392 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26aefb7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a81484d0 0x7f26a8136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.392 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26aefb7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a813c010 con 0x7f26a81484d0 2020-11-20T12:04:41.392 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26aefb7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a813c230 con 0x7f26a8066f70 2020-11-20T12:04:41.393 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26aefb7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a813c4d0 con 0x7f26a80b6650 2020-11-20T12:04:41.393 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ad7b4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26a80b6650 0x7f26a813bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.393 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ad7b4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26a80b6650 0x7f26a813bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40626/0 (socket says 172.21.15.5:40626) 2020-11-20T12:04:41.393 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ad7b4700 1 -- 172.21.15.5:0/4034750808 learned_addr learned my addr 172.21.15.5:0/4034750808 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:41.393 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ae7b6700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a81484d0 0x7f26a8136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.394 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26adfb5700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26a8066f70 0x7f26a813ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.394 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ad7b4700 1 -- 172.21.15.5:0/4034750808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26a8066f70 msgr2=0x7f26a813ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.394 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ad7b4700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26a8066f70 0x7f26a813ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.394 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ad7b4700 1 -- 172.21.15.5:0/4034750808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a81484d0 msgr2=0x7f26a8136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.395 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ad7b4700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a81484d0 0x7f26a8136040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.395 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.384+0000 7f26ad7b4700 1 -- 172.21.15.5:0/4034750808 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a813c770 con 0x7f26a80b6650 2020-11-20T12:04:41.395 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.385+0000 7f26ad7b4700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26a80b6650 0x7f26a813bb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2698009fd0 tx=0x7f2698015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:41.395 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.385+0000 7f2695ffb700 1 -- 172.21.15.5:0/4034750808 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269801e460 con 0x7f26a80b6650 2020-11-20T12:04:41.395 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.385+0000 7f2695ffb700 1 -- 172.21.15.5:0/4034750808 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269801e5c0 con 0x7f26a80b6650 2020-11-20T12:04:41.396 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.385+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a813cc10 con 0x7f26a80b6650 2020-11-20T12:04:41.396 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.385+0000 7f2695ffb700 1 -- 172.21.15.5:0/4034750808 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269801e960 con 0x7f26a80b6650 2020-11-20T12:04:41.396 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.385+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a8134f40 con 0x7f26a80b6650 2020-11-20T12:04:41.396 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.385+0000 7f2695ffb700 1 -- 172.21.15.5:0/4034750808 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2698025030 con 0x7f26a80b6650 2020-11-20T12:04:41.396 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.386+0000 7f2695ffb700 1 -- 172.21.15.5:0/4034750808 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f269801d880 con 0x7f26a80b6650 2020-11-20T12:04:41.397 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.386+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a8061a20 con 0x7f26a80b6650 2020-11-20T12:04:41.397 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.390+0000 7f2695ffb700 1 -- 172.21.15.5:0/4034750808 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f2698023070 con 0x7f26a80b6650 2020-11-20T12:04:41.494 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.491+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e041ab46-c4c7-48ae-8144-caac199f94f7", "id": 3} v 0) v1 -- 0x7f26a80624e0 con 0x7f26a80b6650 2020-11-20T12:04:41.500 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.496+0000 7f2695ffb700 1 -- 172.21.15.5:0/4034750808 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e041ab46-c4c7-48ae-8144-caac199f94f7", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f269801d3c0 con 0x7f26a80b6650 2020-11-20T12:04:41.500 INFO:teuthology.orchestra.run.smithi005.stdout:3 2020-11-20T12:04:41.501 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26a80b6650 msgr2=0x7f26a813bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.501 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26a80b6650 0x7f26a813bb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2698009fd0 tx=0x7f2698015000).stop 2020-11-20T12:04:41.502 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 shutdown_connections 2020-11-20T12:04:41.502 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26a81484d0 0x7f26a8136040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.502 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26a8066f70 0x7f26a813ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 --2- 172.21.15.5:0/4034750808 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26a80b6650 0x7f26a813bb20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 >> 172.21.15.5:0/4034750808 conn(0x7f26a80bab30 msgr2=0x7f26a80afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 shutdown_connections 2020-11-20T12:04:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.498+0000 7f26aefb7700 1 -- 172.21.15.5:0/4034750808 wait complete. 2020-11-20T12:04:41.523 INFO:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 6adcc4eb-3632-47cf-a500-e134866924da 4 2020-11-20T12:04:41.739 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.736+0000 7f1a1d7aa700 1 Processor -- start 2020-11-20T12:04:41.740 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.736+0000 7f1a1d7aa700 1 -- start start 2020-11-20T12:04:41.741 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a1d7aa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1a180b1840 0x7f1a180b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.741 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a1d7aa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a180b8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.742 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a1d7aa700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a18149790 con 0x7f1a180b11d0 2020-11-20T12:04:41.742 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a1d7aa700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a18065ba0 con 0x7f1a180b1840 2020-11-20T12:04:41.742 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a1d7aa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a18149450 con 0x7f1a180b1c20 2020-11-20T12:04:41.744 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a1cfa9700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1a180b11d0 0x7f1a18066040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:39030/0 (socket says 172.21.15.5:39030) 2020-11-20T12:04:41.744 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a1cfa9700 1 -- 172.21.15.5:0/3064061984 learned_addr learned my addr 172.21.15.5:0/3064061984 (peer_addr_for_me v1:172.21.15.5:0/0) 2020-11-20T12:04:41.744 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a177fe700 1 --2- 172.21.15.5:0/3064061984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a180b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.744 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.738+0000 7f1a17fff700 1 --2- 172.21.15.5:0/3064061984 >> v2:172.21.15.187:3300/0 conn(0x7f1a180b1840 0x7f1a180b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.745 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.739+0000 7f1a16ffd700 1 -- 172.21.15.5:0/3064061984 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685252379 0 0) 0x7f1a18149790 con 0x7f1a180b11d0 2020-11-20T12:04:41.745 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.739+0000 7f1a16ffd700 1 -- 172.21.15.5:0/3064061984 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a04003240 con 0x7f1a180b11d0 2020-11-20T12:04:41.745 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.739+0000 7f1a177fe700 1 -- 172.21.15.5:0/3064061984 >> v2:172.21.15.187:3300/0 conn(0x7f1a180b1840 msgr2=0x7f1a180b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:41.745 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.739+0000 7f1a177fe700 1 --2- 172.21.15.5:0/3064061984 >> v2:172.21.15.187:3300/0 conn(0x7f1a180b1840 0x7f1a180b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.745 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.739+0000 7f1a177fe700 1 -- 172.21.15.5:0/3064061984 >> v1:172.21.15.5:6789/0 conn(0x7f1a180b11d0 legacy=0x7f1a18066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.746 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.739+0000 7f1a177fe700 1 -- 172.21.15.5:0/3064061984 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1805bd30 con 0x7f1a180b1c20 2020-11-20T12:04:41.746 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.739+0000 7f1a177fe700 1 --2- 172.21.15.5:0/3064061984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a180b8590 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f1a0c00b9a0 tx=0x7f1a0c006df0).ready entity=mon.2 client_cookie=d175ef78e5a58b52 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:41.746 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.740+0000 7f1a16ffd700 1 -- 172.21.15.5:0/3064061984 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c009710 con 0x7f1a180b1c20 2020-11-20T12:04:41.746 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.740+0000 7f1a16ffd700 1 -- 172.21.15.5:0/3064061984 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c009870 con 0x7f1a180b1c20 2020-11-20T12:04:41.747 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.740+0000 7f1a16ffd700 1 -- 172.21.15.5:0/3064061984 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c009bf0 con 0x7f1a180b1c20 2020-11-20T12:04:41.747 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.741+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/3064061984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 msgr2=0x7f1a180b8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.748 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.741+0000 7f1a1d7aa700 1 --2- 172.21.15.5:0/3064061984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a180b8590 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f1a0c00b9a0 tx=0x7f1a0c006df0).stop 2020-11-20T12:04:41.748 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.741+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/3064061984 shutdown_connections 2020-11-20T12:04:41.749 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.741+0000 7f1a1d7aa700 1 --2- 172.21.15.5:0/3064061984 >> v2:172.21.15.187:3300/0 conn(0x7f1a180b1840 0x7f1a180b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.749 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.741+0000 7f1a1d7aa700 1 --2- 172.21.15.5:0/3064061984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a180b8590 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.749 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.741+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/3064061984 >> 172.21.15.5:0/3064061984 conn(0x7f1a180bbeb0 msgr2=0x7f1a180b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:41.750 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.741+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/3064061984 shutdown_connections 2020-11-20T12:04:41.750 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.741+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/3064061984 wait complete. 2020-11-20T12:04:41.750 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.743+0000 7f1a1d7aa700 1 Processor -- start 2020-11-20T12:04:41.750 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.743+0000 7f1a1d7aa700 1 -- start start 2020-11-20T12:04:41.752 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a1d7aa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a180b11d0 0x7f1a180b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.752 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a1d7aa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a180b1840 0x7f1a180ba580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.753 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a1d7aa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a1809e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:41.753 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a1d7aa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1814ccc0 con 0x7f1a180b11d0 2020-11-20T12:04:41.753 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a1d7aa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1814cf60 con 0x7f1a180b1840 2020-11-20T12:04:41.753 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a1d7aa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1814d200 con 0x7f1a180b1c20 2020-11-20T12:04:41.754 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a17fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a180b11d0 0x7f1a180b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.754 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a17fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a180b11d0 0x7f1a180b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44354/0 (socket says 172.21.15.5:44354) 2020-11-20T12:04:41.754 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a17fff700 1 -- 172.21.15.5:0/289184669 learned_addr learned my addr 172.21.15.5:0/289184669 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:41.754 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a1cfa9700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a1809e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.754 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.745+0000 7f1a177fe700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a180b1840 0x7f1a180ba580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:41.755 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a17fff700 1 -- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 msgr2=0x7f1a1809e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.755 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a17fff700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a1809e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.755 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a17fff700 1 -- 172.21.15.5:0/289184669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a180b1840 msgr2=0x7f1a180ba580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.755 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a17fff700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a180b1840 0x7f1a180ba580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.755 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a17fff700 1 -- 172.21.15.5:0/289184669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180b4760 con 0x7f1a180b11d0 2020-11-20T12:04:41.756 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a1cfa9700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a1809e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:41.756 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a17fff700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a180b11d0 0x7f1a180b4270 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1a0800eba0 tx=0x7f1a08006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:41.756 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f19f7fff700 1 -- 172.21.15.5:0/289184669 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0800c9b0 con 0x7f1a180b11d0 2020-11-20T12:04:41.756 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f19f7fff700 1 -- 172.21.15.5:0/289184669 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0800cb10 con 0x7f1a180b11d0 2020-11-20T12:04:41.757 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f19f7fff700 1 -- 172.21.15.5:0/289184669 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0800ce90 con 0x7f1a180b11d0 2020-11-20T12:04:41.757 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a180b4bc0 con 0x7f1a180b11d0 2020-11-20T12:04:41.757 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.746+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a180bac30 con 0x7f1a180b11d0 2020-11-20T12:04:41.757 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.747+0000 7f19f7fff700 1 -- 172.21.15.5:0/289184669 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1a08012860 con 0x7f1a180b11d0 2020-11-20T12:04:41.757 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.747+0000 7f19f7fff700 1 -- 172.21.15.5:0/289184669 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f1a08012af0 con 0x7f1a180b11d0 2020-11-20T12:04:41.758 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.748+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1809e030 con 0x7f1a180b11d0 2020-11-20T12:04:41.758 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.750+0000 7f19f7fff700 1 -- 172.21.15.5:0/289184669 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f1a08012dc0 con 0x7f1a180b11d0 2020-11-20T12:04:41.856 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.853+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6adcc4eb-3632-47cf-a500-e134866924da", "id": 4} v 0) v1 -- 0x7f1a18061050 con 0x7f1a180b11d0 2020-11-20T12:04:41.861 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.857+0000 7f19f7fff700 1 -- 172.21.15.5:0/289184669 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6adcc4eb-3632-47cf-a500-e134866924da", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1a0801c460 con 0x7f1a180b11d0 2020-11-20T12:04:41.861 INFO:teuthology.orchestra.run.smithi005.stdout:4 2020-11-20T12:04:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a180b11d0 msgr2=0x7f1a180b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a180b11d0 0x7f1a180b4270 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1a0800eba0 tx=0x7f1a08006df0).stop 2020-11-20T12:04:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 shutdown_connections 2020-11-20T12:04:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a180b11d0 0x7f1a180b4270 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.864 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a180b1c20 0x7f1a1809e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.864 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 --2- 172.21.15.5:0/289184669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a180b1840 0x7f1a180ba580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.864 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 >> 172.21.15.5:0/289184669 conn(0x7f1a180bbeb0 msgr2=0x7f1a18140fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:41.864 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 shutdown_connections 2020-11-20T12:04:41.865 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:41.859+0000 7f1a1d7aa700 1 -- 172.21.15.5:0/289184669 wait complete. 2020-11-20T12:04:41.884 INFO:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 77c9ee1b-3f40-4a15-84e0-8b89b96e5237 5 2020-11-20T12:04:42.107 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.104+0000 7f057694f700 1 Processor -- start 2020-11-20T12:04:42.108 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.104+0000 7f057694f700 1 -- start start 2020-11-20T12:04:42.110 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.105+0000 7f057694f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05700bf800 0x7f057014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.110 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.105+0000 7f057694f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f057009f560 0x7f05700a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.111 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057694f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057014d110 con 0x7f05700bfbe0 2020-11-20T12:04:42.111 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057694f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057014cbd0 con 0x7f057009f560 2020-11-20T12:04:42.111 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057694f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057014ce70 con 0x7f05700bf800 2020-11-20T12:04:42.111 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057614e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f057009f560 0x7f05700a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.112 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057614e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f057009f560 0x7f05700a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44364/0 (socket says 172.21.15.5:44364) 2020-11-20T12:04:42.112 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057614e700 1 -- 172.21.15.5:0/1060672370 learned_addr learned my addr 172.21.15.5:0/1060672370 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:42.112 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057594d700 1 --2- 172.21.15.5:0/1060672370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05700bf800 0x7f057014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.112 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057614e700 1 -- 172.21.15.5:0/1060672370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05700bf800 msgr2=0x7f057014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:42.113 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057614e700 1 --2- 172.21.15.5:0/1060672370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05700bf800 0x7f057014c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.113 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.106+0000 7f057614e700 1 -- 172.21.15.5:0/1060672370 >> v1:172.21.15.187:6789/0 conn(0x7f05700bfbe0 legacy=0x7f05701480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.113 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.107+0000 7f057614e700 1 -- 172.21.15.5:0/1060672370 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05700a4cc0 con 0x7f057009f560 2020-11-20T12:04:42.114 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.107+0000 7f057614e700 1 --2- 172.21.15.5:0/1060672370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057009f560 0x7f05700a36e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0568009d50 tx=0x7f0568009290).ready entity=mon.0 client_cookie=6dab11cf3d91cb92 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:42.117 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.107+0000 7f057494b700 1 -- 172.21.15.5:0/1060672370 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056800bc50 con 0x7f057009f560 2020-11-20T12:04:42.117 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.107+0000 7f057494b700 1 -- 172.21.15.5:0/1060672370 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056800bdb0 con 0x7f057009f560 2020-11-20T12:04:42.117 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.107+0000 7f057494b700 1 -- 172.21.15.5:0/1060672370 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568011610 con 0x7f057009f560 2020-11-20T12:04:42.118 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.108+0000 7f057694f700 1 -- 172.21.15.5:0/1060672370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057009f560 msgr2=0x7f05700a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.118 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.108+0000 7f057694f700 1 --2- 172.21.15.5:0/1060672370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057009f560 0x7f05700a36e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0568009d50 tx=0x7f0568009290).stop 2020-11-20T12:04:42.118 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.108+0000 7f057694f700 1 -- 172.21.15.5:0/1060672370 shutdown_connections 2020-11-20T12:04:42.118 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.108+0000 7f057694f700 1 --2- 172.21.15.5:0/1060672370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057009f560 0x7f05700a36e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.118 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.108+0000 7f057694f700 1 --2- 172.21.15.5:0/1060672370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05700bf800 0x7f057014c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.119 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.108+0000 7f057694f700 1 -- 172.21.15.5:0/1060672370 >> 172.21.15.5:0/1060672370 conn(0x7f0570068370 msgr2=0x7f05700bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:42.119 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.108+0000 7f057694f700 1 -- 172.21.15.5:0/1060672370 shutdown_connections 2020-11-20T12:04:42.119 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.108+0000 7f057694f700 1 -- 172.21.15.5:0/1060672370 wait complete. 2020-11-20T12:04:42.119 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.109+0000 7f057694f700 1 Processor -- start 2020-11-20T12:04:42.120 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.110+0000 7f057694f700 1 -- start start 2020-11-20T12:04:42.120 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.110+0000 7f057694f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f057009f560 0x7f05700bac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.120 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.110+0000 7f057694f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05700bf800 0x7f05700b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.120 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.110+0000 7f057694f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057014ae90 0x7f0570158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.120 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.110+0000 7f057694f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0570159220 con 0x7f057014ae90 2020-11-20T12:04:42.121 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.110+0000 7f057694f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05701594c0 con 0x7f05700bf800 2020-11-20T12:04:42.121 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.110+0000 7f057694f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0570143770 con 0x7f057009f560 2020-11-20T12:04:42.121 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057594d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f057009f560 0x7f05700bac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.121 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057594d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f057009f560 0x7f05700bac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40648/0 (socket says 172.21.15.5:40648) 2020-11-20T12:04:42.122 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057594d700 1 -- 172.21.15.5:0/3229554669 learned_addr learned my addr 172.21.15.5:0/3229554669 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:42.122 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057614e700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057014ae90 0x7f0570158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.122 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057514c700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05700bf800 0x7f05700b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.122 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057594d700 1 -- 172.21.15.5:0/3229554669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05700bf800 msgr2=0x7f05700b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.122 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057594d700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05700bf800 0x7f05700b6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057594d700 1 -- 172.21.15.5:0/3229554669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057014ae90 msgr2=0x7f0570158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057594d700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057014ae90 0x7f0570158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.111+0000 7f057594d700 1 -- 172.21.15.5:0/3229554669 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057014b2d0 con 0x7f057009f560 2020-11-20T12:04:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.112+0000 7f057594d700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f057009f560 0x7f05700bac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f056c00eba0 tx=0x7f056c006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:42.124 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.112+0000 7f055dffb700 1 -- 172.21.15.5:0/3229554669 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c00c9b0 con 0x7f057009f560 2020-11-20T12:04:42.124 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.112+0000 7f055dffb700 1 -- 172.21.15.5:0/3229554669 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c00cb10 con 0x7f057009f560 2020-11-20T12:04:42.124 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.112+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05700ba0d0 con 0x7f057009f560 2020-11-20T12:04:42.124 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.112+0000 7f055dffb700 1 -- 172.21.15.5:0/3229554669 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c00ce90 con 0x7f057009f560 2020-11-20T12:04:42.124 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.112+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0570158dd0 con 0x7f057009f560 2020-11-20T12:04:42.125 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.112+0000 7f055dffb700 1 -- 172.21.15.5:0/3229554669 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f056c012860 con 0x7f057009f560 2020-11-20T12:04:42.125 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.112+0000 7f055dffb700 1 -- 172.21.15.5:0/3229554669 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f056c012b50 con 0x7f057009f560 2020-11-20T12:04:42.125 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.113+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0570159680 con 0x7f057009f560 2020-11-20T12:04:42.125 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.116+0000 7f055dffb700 1 -- 172.21.15.5:0/3229554669 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f056c026480 con 0x7f057009f560 2020-11-20T12:04:42.218 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.215+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd new", "uuid": "77c9ee1b-3f40-4a15-84e0-8b89b96e5237", "id": 5} v 0) v1 -- 0x7f05700bf460 con 0x7f057009f560 2020-11-20T12:04:42.225 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.221+0000 7f055dffb700 1 -- 172.21.15.5:0/3229554669 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "77c9ee1b-3f40-4a15-84e0-8b89b96e5237", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f056c026620 con 0x7f057009f560 2020-11-20T12:04:42.225 INFO:teuthology.orchestra.run.smithi005.stdout:5 2020-11-20T12:04:42.227 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f057009f560 msgr2=0x7f05700bac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.227 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f057009f560 0x7f05700bac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f056c00eba0 tx=0x7f056c006df0).stop 2020-11-20T12:04:42.227 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 shutdown_connections 2020-11-20T12:04:42.227 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f057014ae90 0x7f0570158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.228 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f057009f560 0x7f05700bac70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.228 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 --2- 172.21.15.5:0/3229554669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05700bf800 0x7f05700b6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.228 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 >> 172.21.15.5:0/3229554669 conn(0x7f0570068370 msgr2=0x7f05700a0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:42.228 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 shutdown_connections 2020-11-20T12:04:42.228 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.223+0000 7f057694f700 1 -- 172.21.15.5:0/3229554669 wait complete. 2020-11-20T12:04:42.250 INFO:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new d79cbb7e-2a05-4688-bd3d-1289b4b231bd 6 2020-11-20T12:04:42.478 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.475+0000 7f796202f700 1 Processor -- start 2020-11-20T12:04:42.479 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.475+0000 7f796202f700 1 -- start start 2020-11-20T12:04:42.480 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.476+0000 7f796202f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0bf800 0x7f795c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.481 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.476+0000 7f796202f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f795c0bfbe0 0x7f795c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.481 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.476+0000 7f796202f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f795c09f560 0x7f795c1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.481 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.476+0000 7f796202f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795c14d110 con 0x7f795c09f560 2020-11-20T12:04:42.481 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.476+0000 7f796202f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795c148aa0 con 0x7f795c0bfbe0 2020-11-20T12:04:42.482 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.476+0000 7f796202f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795c14cbd0 con 0x7f795c0bf800 2020-11-20T12:04:42.482 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.477+0000 7f796182e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f795c09f560 0x7f795c1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.482 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.477+0000 7f796182e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f795c09f560 0x7f795c1485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44374/0 (socket says 172.21.15.5:44374) 2020-11-20T12:04:42.482 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.477+0000 7f796182e700 1 -- 172.21.15.5:0/355871222 learned_addr learned my addr 172.21.15.5:0/355871222 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:42.482 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.477+0000 7f796082c700 1 --2- 172.21.15.5:0/355871222 >> v2:172.21.15.187:3300/0 conn(0x7f795c0bfbe0 0x7f795c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.483 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.477+0000 7f796102d700 1 --2- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0bf800 0x7f795c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.485 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.477+0000 7f796182e700 1 -- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0bf800 msgr2=0x7f795c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:42.485 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.477+0000 7f796182e700 1 --2- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0bf800 0x7f795c14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.485 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f796182e700 1 -- 172.21.15.5:0/355871222 >> v2:172.21.15.187:3300/0 conn(0x7f795c0bfbe0 msgr2=0x7f795c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:42.485 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f796182e700 1 --2- 172.21.15.5:0/355871222 >> v2:172.21.15.187:3300/0 conn(0x7f795c0bfbe0 0x7f795c1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f796182e700 1 -- 172.21.15.5:0/355871222 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c14ce70 con 0x7f795c09f560 2020-11-20T12:04:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f796102d700 1 --2- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0bf800 0x7f795c14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:42.487 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f796082c700 1 --2- 172.21.15.5:0/355871222 >> v2:172.21.15.187:3300/0 conn(0x7f795c0bfbe0 0x7f795c1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:42.488 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f796182e700 1 --2- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c09f560 0x7f795c1485b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f795c068830 tx=0x7f7954009290).ready entity=mon.0 client_cookie=4662935ac9e73674 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:42.488 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f794bfff700 1 -- 172.21.15.5:0/355871222 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795400bc50 con 0x7f795c09f560 2020-11-20T12:04:42.488 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f794bfff700 1 -- 172.21.15.5:0/355871222 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795400bdb0 con 0x7f795c09f560 2020-11-20T12:04:42.489 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.478+0000 7f794bfff700 1 -- 172.21.15.5:0/355871222 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954011610 con 0x7f795c09f560 2020-11-20T12:04:42.489 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 -- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c09f560 msgr2=0x7f795c1485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.489 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 --2- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c09f560 0x7f795c1485b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f795c068830 tx=0x7f7954009290).stop 2020-11-20T12:04:42.489 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 -- 172.21.15.5:0/355871222 shutdown_connections 2020-11-20T12:04:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 --2- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c09f560 0x7f795c1485b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 --2- 172.21.15.5:0/355871222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0bf800 0x7f795c14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 --2- 172.21.15.5:0/355871222 >> v2:172.21.15.187:3300/0 conn(0x7f795c0bfbe0 0x7f795c1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 -- 172.21.15.5:0/355871222 >> 172.21.15.5:0/355871222 conn(0x7f795c068370 msgr2=0x7f795c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:42.491 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 -- 172.21.15.5:0/355871222 shutdown_connections 2020-11-20T12:04:42.492 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.479+0000 7f796202f700 1 -- 172.21.15.5:0/355871222 wait complete. 2020-11-20T12:04:42.492 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.481+0000 7f796202f700 1 Processor -- start 2020-11-20T12:04:42.493 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.481+0000 7f796202f700 1 -- start start 2020-11-20T12:04:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796202f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c09f560 0x7f795c141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796202f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f795c0bf800 0x7f795c1427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.496 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796202f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c13fb00 0x7f795c1446f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.496 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796202f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c1492d0 con 0x7f795c13fb00 2020-11-20T12:04:42.496 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796202f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c149570 con 0x7f795c0bf800 2020-11-20T12:04:42.496 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796202f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795c140010 con 0x7f795c09f560 2020-11-20T12:04:42.496 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796102d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c09f560 0x7f795c141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.497 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796102d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c09f560 0x7f795c141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40660/0 (socket says 172.21.15.5:40660) 2020-11-20T12:04:42.497 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796102d700 1 -- 172.21.15.5:0/3443801152 learned_addr learned my addr 172.21.15.5:0/3443801152 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:42.497 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.482+0000 7f796182e700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c13fb00 0x7f795c1446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.497 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796082c700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f795c0bf800 0x7f795c1427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.498 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796102d700 1 -- 172.21.15.5:0/3443801152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f795c0bf800 msgr2=0x7f795c1427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.498 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796102d700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f795c0bf800 0x7f795c1427e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.498 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796102d700 1 -- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c13fb00 msgr2=0x7f795c1446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.498 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796102d700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c13fb00 0x7f795c1446f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.499 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796102d700 1 -- 172.21.15.5:0/3443801152 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c146510 con 0x7f795c09f560 2020-11-20T12:04:42.499 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796082c700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f795c0bf800 0x7f795c1427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:42.499 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796182e700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c13fb00 0x7f795c1446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:42.499 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.483+0000 7f796102d700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c09f560 0x7f795c141110 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f795800db60 tx=0x7f7958008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:42.500 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.484+0000 7f79497fa700 1 -- 172.21.15.5:0/3443801152 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958009850 con 0x7f795c09f560 2020-11-20T12:04:42.500 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.484+0000 7f79497fa700 1 -- 172.21.15.5:0/3443801152 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79580099b0 con 0x7f795c09f560 2020-11-20T12:04:42.500 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.484+0000 7f79497fa700 1 -- 172.21.15.5:0/3443801152 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958009d70 con 0x7f795c09f560 2020-11-20T12:04:42.501 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.484+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795c142cd0 con 0x7f795c09f560 2020-11-20T12:04:42.501 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.484+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c144e20 con 0x7f795c09f560 2020-11-20T12:04:42.501 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.484+0000 7f79497fa700 1 -- 172.21.15.5:0/3443801152 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f7958006280 con 0x7f795c09f560 2020-11-20T12:04:42.501 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.484+0000 7f79497fa700 1 -- 172.21.15.5:0/3443801152 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f7958005e30 con 0x7f795c09f560 2020-11-20T12:04:42.502 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.485+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795c080050 con 0x7f795c09f560 2020-11-20T12:04:42.502 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.487+0000 7f79497fa700 1 -- 172.21.15.5:0/3443801152 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f7958006510 con 0x7f795c09f560 2020-11-20T12:04:42.590 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.587+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d79cbb7e-2a05-4688-bd3d-1289b4b231bd", "id": 6} v 0) v1 -- 0x7f795c0a4d70 con 0x7f795c09f560 2020-11-20T12:04:42.597 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.593+0000 7f79497fa700 1 -- 172.21.15.5:0/3443801152 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d79cbb7e-2a05-4688-bd3d-1289b4b231bd", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7958005810 con 0x7f795c09f560 2020-11-20T12:04:42.597 INFO:teuthology.orchestra.run.smithi005.stdout:6 2020-11-20T12:04:42.598 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.594+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c09f560 msgr2=0x7f795c141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.599 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.594+0000 7f796202f700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c09f560 0x7f795c141110 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f795800db60 tx=0x7f7958008040).stop 2020-11-20T12:04:42.599 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.595+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 shutdown_connections 2020-11-20T12:04:42.599 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.595+0000 7f796202f700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c13fb00 0x7f795c1446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.599 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.595+0000 7f796202f700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c09f560 0x7f795c141110 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.599 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.595+0000 7f796202f700 1 --2- 172.21.15.5:0/3443801152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f795c0bf800 0x7f795c1427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.600 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.595+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 >> 172.21.15.5:0/3443801152 conn(0x7f795c068370 msgr2=0x7f795c141890 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:42.600 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.595+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 shutdown_connections 2020-11-20T12:04:42.600 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.595+0000 7f796202f700 1 -- 172.21.15.5:0/3443801152 wait complete. 2020-11-20T12:04:42.617 INFO:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 65c46353-4546-48a6-813e-3968184574c8 7 2020-11-20T12:04:42.836 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.833+0000 7f16f12af700 1 Processor -- start 2020-11-20T12:04:42.837 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.833+0000 7f16f12af700 1 -- start start 2020-11-20T12:04:42.838 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.834+0000 7f16f12af700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f16ec0679c0 0x7f16ec0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.839 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.834+0000 7f16f12af700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec145da0 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.839 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.834+0000 7f16f12af700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec067520 con 0x7f16ec0b73b0 2020-11-20T12:04:42.839 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.834+0000 7f16f12af700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec146010 con 0x7f16ec0679c0 2020-11-20T12:04:42.839 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.835+0000 7f16eaffd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f16ec0b6fd0 0x7f16ec066700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:39064/0 (socket says 172.21.15.5:39064) 2020-11-20T12:04:42.840 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.835+0000 7f16eaffd700 1 -- 172.21.15.5:0/2651302578 learned_addr learned my addr 172.21.15.5:0/2651302578 (peer_addr_for_me v1:172.21.15.5:0/0) 2020-11-20T12:04:42.840 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.835+0000 7f16eb7fe700 1 --2- 172.21.15.5:0/2651302578 >> v2:172.21.15.5:3300/0 conn(0x7f16ec0679c0 0x7f16ec0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.842 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.835+0000 7f16ea7fc700 1 -- 172.21.15.5:0/2651302578 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329753999 0 0) 0x7f16ec145da0 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.842 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.835+0000 7f16ea7fc700 1 -- 172.21.15.5:0/2651302578 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16cc003240 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.842 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.836+0000 7f16eb7fe700 1 -- 172.21.15.5:0/2651302578 >> v1:172.21.15.187:6789/0 conn(0x7f16ec0b73b0 legacy=0x7f16ec066e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.842 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.836+0000 7f16eb7fe700 1 -- 172.21.15.5:0/2651302578 >> v1:172.21.15.5:6789/0 conn(0x7f16ec0b6fd0 legacy=0x7f16ec066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.843 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.836+0000 7f16ea7fc700 1 -- 172.21.15.5:0/2651302578 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 981170595 0 0) 0x7f16cc003240 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.843 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.836+0000 7f16eb7fe700 1 -- 172.21.15.5:0/2651302578 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec1462b0 con 0x7f16ec0679c0 2020-11-20T12:04:42.843 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.836+0000 7f16eb7fe700 1 --2- 172.21.15.5:0/2651302578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0679c0 0x7f16ec0b5640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f16e400b9a0 tx=0x7f16e4006df0).ready entity=mon.0 client_cookie=f48d07195b69e6db server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:42.843 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.836+0000 7f16ea7fc700 1 -- 172.21.15.5:0/2651302578 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc001860 con 0x7f16ec0679c0 2020-11-20T12:04:42.844 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.836+0000 7f16ea7fc700 1 -- 172.21.15.5:0/2651302578 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e40094a0 con 0x7f16ec0679c0 2020-11-20T12:04:42.844 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.836+0000 7f16ea7fc700 1 -- 172.21.15.5:0/2651302578 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4009820 con 0x7f16ec0679c0 2020-11-20T12:04:42.845 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.837+0000 7f16f12af700 1 -- 172.21.15.5:0/2651302578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0679c0 msgr2=0x7f16ec0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.846 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.837+0000 7f16f12af700 1 --2- 172.21.15.5:0/2651302578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0679c0 0x7f16ec0b5640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f16e400b9a0 tx=0x7f16e4006df0).stop 2020-11-20T12:04:42.846 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.837+0000 7f16f12af700 1 -- 172.21.15.5:0/2651302578 shutdown_connections 2020-11-20T12:04:42.847 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.837+0000 7f16f12af700 1 --2- 172.21.15.5:0/2651302578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0679c0 0x7f16ec0b5640 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.848 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.837+0000 7f16f12af700 1 -- 172.21.15.5:0/2651302578 >> 172.21.15.5:0/2651302578 conn(0x7f16ec0bacf0 msgr2=0x7f16ec0bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:42.848 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.837+0000 7f16f12af700 1 -- 172.21.15.5:0/2651302578 shutdown_connections 2020-11-20T12:04:42.849 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.837+0000 7f16f12af700 1 -- 172.21.15.5:0/2651302578 wait complete. 2020-11-20T12:04:42.850 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.838+0000 7f16f12af700 1 Processor -- start 2020-11-20T12:04:42.850 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.838+0000 7f16f12af700 1 -- start start 2020-11-20T12:04:42.856 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16f12af700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16ec0679c0 0x7f16ec1575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.856 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16f12af700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0b6fd0 0x7f16ec09f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.856 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16f12af700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16ec0b73b0 0x7f16ec09f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:42.857 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16f12af700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec09fe20 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.857 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16f12af700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec090b60 con 0x7f16ec0679c0 2020-11-20T12:04:42.858 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16f12af700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec090e00 con 0x7f16ec0b73b0 2020-11-20T12:04:42.858 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16e3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0b6fd0 0x7f16ec09f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.858 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16e3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0b6fd0 0x7f16ec09f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44392/0 (socket says 172.21.15.5:44392) 2020-11-20T12:04:42.859 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.839+0000 7f16e3fff700 1 -- 172.21.15.5:0/3418499861 learned_addr learned my addr 172.21.15.5:0/3418499861 (peer_addr_for_me v2:172.21.15.5:0/0) 2020-11-20T12:04:42.859 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16eb7fe700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16ec0b73b0 0x7f16ec09f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.859 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16eaffd700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16ec0679c0 0x7f16ec1575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:42.860 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16e3fff700 1 -- 172.21.15.5:0/3418499861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16ec0b73b0 msgr2=0x7f16ec09f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.860 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16e3fff700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16ec0b73b0 0x7f16ec09f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.860 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16e3fff700 1 -- 172.21.15.5:0/3418499861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16ec0679c0 msgr2=0x7f16ec1575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.861 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16e3fff700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16ec0679c0 0x7f16ec1575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.861 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16e3fff700 1 -- 172.21.15.5:0/3418499861 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec0b2d20 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.862 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16eaffd700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16ec0679c0 0x7f16ec1575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:42.862 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16e3fff700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0b6fd0 0x7f16ec09f440 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f16d400c6e0 tx=0x7f16d4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:42.862 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.840+0000 7f16e2ffd700 1 -- 172.21.15.5:0/3418499861 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d4007020 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.863 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.841+0000 7f16e2ffd700 1 -- 172.21.15.5:0/3418499861 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d4007180 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.863 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.841+0000 7f16e2ffd700 1 -- 172.21.15.5:0/3418499861 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d40064f0 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.863 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.841+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec0a08f0 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.864 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.841+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec0906f0 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.864 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.841+0000 7f16e2ffd700 1 -- 172.21.15.5:0/3418499861 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f16d4010070 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.865 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.841+0000 7f16e2ffd700 1 -- 172.21.15.5:0/3418499861 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7f16d400eb70 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.865 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.842+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec0b2eb0 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.865 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.844+0000 7f16e2ffd700 1 -- 172.21.15.5:0/3418499861 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f16d4020060 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.949 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.945+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd new", "uuid": "65c46353-4546-48a6-813e-3968184574c8", "id": 7} v 0) v1 -- 0x7f16ec044f60 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.956 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.953+0000 7f16e2ffd700 1 -- 172.21.15.5:0/3418499861 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "65c46353-4546-48a6-813e-3968184574c8", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f16d400ee50 con 0x7f16ec0b6fd0 2020-11-20T12:04:42.957 INFO:teuthology.orchestra.run.smithi005.stdout:7 2020-11-20T12:04:42.959 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0b6fd0 msgr2=0x7f16ec09f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:42.959 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0b6fd0 0x7f16ec09f440 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f16d400c6e0 tx=0x7f16d4007c50).stop 2020-11-20T12:04:42.960 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 shutdown_connections 2020-11-20T12:04:42.960 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16ec0b73b0 0x7f16ec09f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.960 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16ec0679c0 0x7f16ec1575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.960 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 --2- 172.21.15.5:0/3418499861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16ec0b6fd0 0x7f16ec09f440 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:42.961 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 >> 172.21.15.5:0/3418499861 conn(0x7f16ec0bacf0 msgr2=0x7f16ec1451e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:42.961 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 shutdown_connections 2020-11-20T12:04:42.961 INFO:teuthology.orchestra.run.smithi005.stderr:2020-11-20T12:04:42.955+0000 7f16f12af700 1 -- 172.21.15.5:0/3418499861 wait complete. 2020-11-20T12:04:42.978 INFO:tasks.ceph.osd.0:Restarting daemon 2020-11-20T12:04:42.979 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-11-20T12:04:42.991 INFO:tasks.ceph.osd.0:Started 2020-11-20T12:04:42.992 INFO:tasks.ceph.osd.1:Restarting daemon 2020-11-20T12:04:42.992 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-11-20T12:04:43.002 INFO:tasks.ceph.osd.1:Started 2020-11-20T12:04:43.002 INFO:tasks.ceph.osd.2:Restarting daemon 2020-11-20T12:04:43.003 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2020-11-20T12:04:43.015 INFO:tasks.ceph.osd.2:Started 2020-11-20T12:04:43.016 INFO:tasks.ceph.osd.3:Restarting daemon 2020-11-20T12:04:43.017 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2020-11-20T12:04:43.026 INFO:tasks.ceph.osd.3:Started 2020-11-20T12:04:43.027 INFO:tasks.ceph.osd.4:Restarting daemon 2020-11-20T12:04:43.028 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2020-11-20T12:04:43.043 INFO:tasks.ceph.osd.4:Started 2020-11-20T12:04:43.044 INFO:tasks.ceph.osd.5:Restarting daemon 2020-11-20T12:04:43.044 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2020-11-20T12:04:43.054 INFO:tasks.ceph.osd.5:Started 2020-11-20T12:04:43.055 INFO:tasks.ceph.osd.6:Restarting daemon 2020-11-20T12:04:43.055 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2020-11-20T12:04:43.065 INFO:tasks.ceph.osd.6:Started 2020-11-20T12:04:43.066 INFO:tasks.ceph.osd.7:Restarting daemon 2020-11-20T12:04:43.067 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2020-11-20T12:04:43.073 INFO:tasks.ceph.osd.7:Started 2020-11-20T12:04:43.074 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-20T12:04:43.676 INFO:tasks.ceph.osd.0.smithi187.stderr:2020-11-20T12:04:43.673+0000 7f238ec4aec0 -1 Falling back to public interface 2020-11-20T12:04:43.698 INFO:tasks.ceph.osd.2.smithi187.stderr:2020-11-20T12:04:43.695+0000 7f7974ad7ec0 -1 Falling back to public interface 2020-11-20T12:04:43.714 INFO:tasks.ceph.osd.1.smithi187.stderr:2020-11-20T12:04:43.711+0000 7fa07376bec0 -1 Falling back to public interface 2020-11-20T12:04:43.744 INFO:tasks.ceph.osd.3.smithi187.stderr:2020-11-20T12:04:43.741+0000 7fd733cd1ec0 -1 Falling back to public interface 2020-11-20T12:04:43.765 INFO:tasks.ceph.osd.6.smithi005.stderr:2020-11-20T12:04:43.757+0000 7f40a467eec0 -1 Falling back to public interface 2020-11-20T12:04:43.766 INFO:tasks.ceph.osd.4.smithi005.stderr:2020-11-20T12:04:43.762+0000 7f2c0e6ecec0 -1 Falling back to public interface 2020-11-20T12:04:43.766 INFO:tasks.ceph.osd.5.smithi005.stderr:2020-11-20T12:04:43.762+0000 7ff48d4aeec0 -1 Falling back to public interface 2020-11-20T12:04:43.771 INFO:tasks.ceph.osd.7.smithi005.stderr:2020-11-20T12:04:43.767+0000 7fcc6dc2cec0 -1 Falling back to public interface 2020-11-20T12:04:43.881 INFO:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:44.082 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.079+0000 7fee54ee7700 1 Processor -- start 2020-11-20T12:04:44.082 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.079+0000 7fee54ee7700 1 -- start start 2020-11-20T12:04:44.083 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.080+0000 7fee54ee7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fee50067580 0x7fee50065be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:44.083 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.080+0000 7fee54ee7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee50149720 con 0x7fee50069060 2020-11-20T12:04:44.084 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.080+0000 7fee54ee7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500b7ae0 con 0x7fee500671a0 2020-11-20T12:04:44.084 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.080+0000 7fee54ee7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee500b7d80 con 0x7fee50067580 2020-11-20T12:04:44.084 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.080+0000 7fee4f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fee500671a0 0x7fee50131b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36496/0 (socket says 172.21.15.187:36496) 2020-11-20T12:04:44.084 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.080+0000 7fee4effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fee50067580 0x7fee50065be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:44.085 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.080+0000 7fee4f7fe700 1 -- 172.21.15.187:0/2038793481 learned_addr learned my addr 172.21.15.187:0/2038793481 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:04:44.086 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.081+0000 7fee4effd700 1 -- 172.21.15.187:0/2038793481 >> v1:172.21.15.187:6789/0 conn(0x7fee500671a0 legacy=0x7fee50131b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:44.086 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.081+0000 7fee4effd700 1 -- 172.21.15.187:0/2038793481 >> v1:172.21.15.5:6789/0 conn(0x7fee50069060 legacy=0x7fee500660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:44.087 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.081+0000 7fee4e7fc700 1 -- 172.21.15.187:0/2038793481 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588815027 0 0) 0x7fee500b7ae0 con 0x7fee500671a0 2020-11-20T12:04:44.087 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.081+0000 7fee4effd700 1 -- 172.21.15.187:0/2038793481 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee500b2470 con 0x7fee50067580 2020-11-20T12:04:44.087 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.081+0000 7fee4effd700 1 --2- 172.21.15.187:0/2038793481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50067580 0x7fee50065be0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fee4400b9a0 tx=0x7fee44006df0).ready entity=mon.1 client_cookie=4b9ce0a4224e2f90 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:44.087 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.081+0000 7fee4e7fc700 1 -- 172.21.15.187:0/2038793481 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440097b0 con 0x7fee50067580 2020-11-20T12:04:44.087 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.081+0000 7fee4e7fc700 1 -- 172.21.15.187:0/2038793481 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44009910 con 0x7fee50067580 2020-11-20T12:04:44.088 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.081+0000 7fee4e7fc700 1 -- 172.21.15.187:0/2038793481 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44009c90 con 0x7fee50067580 2020-11-20T12:04:44.088 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.082+0000 7fee54ee7700 1 -- 172.21.15.187:0/2038793481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50067580 msgr2=0x7fee50065be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:44.088 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.082+0000 7fee54ee7700 1 --2- 172.21.15.187:0/2038793481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50067580 0x7fee50065be0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fee4400b9a0 tx=0x7fee44006df0).stop 2020-11-20T12:04:44.088 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.082+0000 7fee54ee7700 1 -- 172.21.15.187:0/2038793481 shutdown_connections 2020-11-20T12:04:44.088 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.082+0000 7fee54ee7700 1 --2- 172.21.15.187:0/2038793481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50067580 0x7fee50065be0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:44.089 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.082+0000 7fee54ee7700 1 -- 172.21.15.187:0/2038793481 >> 172.21.15.187:0/2038793481 conn(0x7fee500bb5e0 msgr2=0x7fee500bb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:44.089 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.082+0000 7fee54ee7700 1 -- 172.21.15.187:0/2038793481 shutdown_connections 2020-11-20T12:04:44.089 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.082+0000 7fee54ee7700 1 -- 172.21.15.187:0/2038793481 wait complete. 2020-11-20T12:04:44.090 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.083+0000 7fee54ee7700 1 Processor -- start 2020-11-20T12:04:44.092 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.084+0000 7fee54ee7700 1 -- start start 2020-11-20T12:04:44.092 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.084+0000 7fee54ee7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee500671a0 0x7fee5014a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:44.092 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.084+0000 7fee54ee7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee50067580 0x7fee5012fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:44.093 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.084+0000 7fee54ee7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50069060 0x7fee5014b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:44.093 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.084+0000 7fee54ee7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee50133fd0 con 0x7fee50067580 2020-11-20T12:04:44.093 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.084+0000 7fee54ee7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee501341c0 con 0x7fee50069060 2020-11-20T12:04:44.093 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.084+0000 7fee54ee7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee50134460 con 0x7fee500671a0 2020-11-20T12:04:44.094 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50069060 0x7fee5014b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:44.094 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50069060 0x7fee5014b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:39900/0 (socket says 172.21.15.187:39900) 2020-11-20T12:04:44.094 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 -- 172.21.15.187:0/3858897680 learned_addr learned my addr 172.21.15.187:0/3858897680 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:44.094 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4f7fe700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee500671a0 0x7fee5014a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:44.094 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4effd700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee50067580 0x7fee5012fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:44.095 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 -- 172.21.15.187:0/3858897680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee500671a0 msgr2=0x7fee5014a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:44.095 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee500671a0 0x7fee5014a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:44.095 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 -- 172.21.15.187:0/3858897680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee50067580 msgr2=0x7fee5012fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:44.095 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee50067580 0x7fee5012fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:44.095 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 -- 172.21.15.187:0/3858897680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5012ff10 con 0x7fee50069060 2020-11-20T12:04:44.096 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.085+0000 7fee4ffff700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50069060 0x7fee5014b5d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fee4800ca90 tx=0x7fee48001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:44.096 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.086+0000 7fee2f7fe700 1 -- 172.21.15.187:0/3858897680 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4800ce60 con 0x7fee50069060 2020-11-20T12:04:44.096 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.086+0000 7fee2f7fe700 1 -- 172.21.15.187:0/3858897680 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4800a730 con 0x7fee50069060 2020-11-20T12:04:44.096 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.086+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee501324c0 con 0x7fee50069060 2020-11-20T12:04:44.096 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.086+0000 7fee2f7fe700 1 -- 172.21.15.187:0/3858897680 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4800aad0 con 0x7fee50069060 2020-11-20T12:04:44.097 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.086+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee50132810 con 0x7fee50069060 2020-11-20T12:04:44.097 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.086+0000 7fee2f7fe700 1 -- 172.21.15.187:0/3858897680 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 2) v1 ==== 33694+0+0 (secure 0 0 0) 0x7fee48019030 con 0x7fee50069060 2020-11-20T12:04:44.097 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.087+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee50060680 con 0x7fee50069060 2020-11-20T12:04:44.097 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.087+0000 7fee2f7fe700 1 -- 172.21.15.187:0/3858897680 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7fee4803c1a0 con 0x7fee50069060 2020-11-20T12:04:44.098 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.090+0000 7fee2f7fe700 1 -- 172.21.15.187:0/3858897680 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7fee4801b3b0 con 0x7fee50069060 2020-11-20T12:04:44.171 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.168+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee500446d0 con 0x7fee50069060 2020-11-20T12:04:44.172 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.169+0000 7fee2f7fe700 1 -- 172.21.15.187:0/3858897680 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7001 (secure 0 0 0) 0x7fee4803a090 con 0x7fee50069060 2020-11-20T12:04:44.173 INFO:teuthology.misc.health.smithi187.stdout: 2020-11-20T12:04:44.173 INFO:teuthology.misc.health.smithi187.stdout:{"epoch":10,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:42.947133+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:44.175 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.170+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50069060 msgr2=0x7fee5014b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:44.176 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.170+0000 7fee54ee7700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50069060 0x7fee5014b5d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fee4800ca90 tx=0x7fee48001360).stop 2020-11-20T12:04:44.176 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.170+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 shutdown_connections 2020-11-20T12:04:44.177 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.170+0000 7fee54ee7700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee50069060 0x7fee5014b5d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:44.177 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.170+0000 7fee54ee7700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee500671a0 0x7fee5014a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:44.177 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.170+0000 7fee54ee7700 1 --2- 172.21.15.187:0/3858897680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee50067580 0x7fee5012fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:44.178 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.170+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 >> 172.21.15.187:0/3858897680 conn(0x7fee500bb5e0 msgr2=0x7fee500b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:44.178 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.170+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 shutdown_connections 2020-11-20T12:04:44.179 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:44.171+0000 7fee54ee7700 1 -- 172.21.15.187:0/3858897680 wait complete. 2020-11-20T12:04:44.190 DEBUG:teuthology.misc:0 of 8 OSDs are up 2020-11-20T12:04:45.050 INFO:tasks.ceph.osd.0.smithi187.stderr:2020-11-20T12:04:45.044+0000 7f238ec4aec0 -1 osd.0 0 log_to_monitors {default=true} 2020-11-20T12:04:45.064 INFO:tasks.ceph.osd.2.smithi187.stderr:2020-11-20T12:04:45.060+0000 7f7974ad7ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-11-20T12:04:45.090 INFO:tasks.ceph.osd.1.smithi187.stderr:2020-11-20T12:04:45.087+0000 7fa07376bec0 -1 osd.1 0 log_to_monitors {default=true} 2020-11-20T12:04:45.131 INFO:tasks.ceph.osd.3.smithi187.stderr:2020-11-20T12:04:45.128+0000 7fd733cd1ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-11-20T12:04:45.162 INFO:tasks.ceph.osd.4.smithi005.stderr:2020-11-20T12:04:45.159+0000 7f2c0e6ecec0 -1 osd.4 0 log_to_monitors {default=true} 2020-11-20T12:04:45.164 INFO:tasks.ceph.osd.6.smithi005.stderr:2020-11-20T12:04:45.159+0000 7f40a467eec0 -1 osd.6 0 log_to_monitors {default=true} 2020-11-20T12:04:45.167 INFO:tasks.ceph.osd.5.smithi005.stderr:2020-11-20T12:04:45.163+0000 7ff48d4aeec0 -1 osd.5 0 log_to_monitors {default=true} 2020-11-20T12:04:45.175 INFO:tasks.ceph.osd.7.smithi005.stderr:2020-11-20T12:04:45.171+0000 7fcc6dc2cec0 -1 osd.7 0 log_to_monitors {default=true} 2020-11-20T12:04:46.570 INFO:tasks.ceph.osd.4.smithi005.stderr:2020-11-20T12:04:46.566+0000 7f2c04f02700 -1 osd.4 0 waiting for initial osdmap 2020-11-20T12:04:46.571 INFO:tasks.ceph.osd.6.smithi005.stderr:2020-11-20T12:04:46.566+0000 7f409ae94700 -1 osd.6 0 waiting for initial osdmap 2020-11-20T12:04:46.571 INFO:tasks.ceph.osd.2.smithi187.stderr:2020-11-20T12:04:46.566+0000 7f796b2ed700 -1 osd.2 0 waiting for initial osdmap 2020-11-20T12:04:46.572 INFO:tasks.ceph.osd.0.smithi187.stderr:2020-11-20T12:04:46.566+0000 7f2385460700 -1 osd.0 0 waiting for initial osdmap 2020-11-20T12:04:46.573 INFO:tasks.ceph.osd.1.smithi187.stderr:2020-11-20T12:04:46.569+0000 7fa069f81700 -1 osd.1 0 waiting for initial osdmap 2020-11-20T12:04:46.574 INFO:tasks.ceph.osd.7.smithi005.stderr:2020-11-20T12:04:46.569+0000 7fcc64442700 -1 osd.7 0 waiting for initial osdmap 2020-11-20T12:04:46.575 INFO:tasks.ceph.osd.3.smithi187.stderr:2020-11-20T12:04:46.569+0000 7fd72a4e7700 -1 osd.3 0 waiting for initial osdmap 2020-11-20T12:04:46.575 INFO:tasks.ceph.osd.5.smithi005.stderr:2020-11-20T12:04:46.570+0000 7ff483cc4700 -1 osd.5 0 waiting for initial osdmap 2020-11-20T12:04:46.591 INFO:tasks.ceph.osd.6.smithi005.stderr:2020-11-20T12:04:46.587+0000 7f4097614700 -1 osd.6 13 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2020-11-20T12:04:46.593 INFO:tasks.ceph.osd.2.smithi187.stderr:2020-11-20T12:04:46.590+0000 7f7967a6d700 -1 osd.2 13 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2020-11-20T12:04:46.594 INFO:tasks.ceph.osd.4.smithi005.stderr:2020-11-20T12:04:46.590+0000 7f2c01682700 -1 osd.4 13 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2020-11-20T12:04:46.595 INFO:tasks.ceph.osd.3.smithi187.stderr:2020-11-20T12:04:46.591+0000 7fd726c67700 -1 osd.3 13 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2020-11-20T12:04:46.596 INFO:tasks.ceph.osd.0.smithi187.stderr:2020-11-20T12:04:46.593+0000 7f2381be0700 -1 osd.0 13 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2020-11-20T12:04:46.596 INFO:tasks.ceph.osd.1.smithi187.stderr:2020-11-20T12:04:46.593+0000 7fa066701700 -1 osd.1 13 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2020-11-20T12:04:46.598 INFO:tasks.ceph.osd.5.smithi005.stderr:2020-11-20T12:04:46.593+0000 7ff480444700 -1 osd.5 13 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2020-11-20T12:04:46.598 INFO:tasks.ceph.osd.7.smithi005.stderr:2020-11-20T12:04:46.594+0000 7fcc60bc2700 -1 osd.7 13 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2020-11-20T12:04:47.809 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:04:47.825 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:04:51.000 INFO:teuthology.orchestra.run.smithi187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:51.209 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.205+0000 7f70b73e7700 1 Processor -- start 2020-11-20T12:04:51.209 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.205+0000 7f70b73e7700 1 -- start start 2020-11-20T12:04:51.210 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b73e7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f70b00bf820 0x7f70b0148f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.211 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b73e7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70b0148930 0x7f70b0149440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.211 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b73e7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b00b9830 con 0x7f70b00bfc00 2020-11-20T12:04:51.211 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b73e7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b01463b0 con 0x7f70b00bf820 2020-11-20T12:04:51.211 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b73e7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b009f0e0 con 0x7f70b0148930 2020-11-20T12:04:51.212 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b6be6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70b0148930 0x7f70b0149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.212 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b6be6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70b0148930 0x7f70b0149440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40140/0 (socket says 172.21.15.187:40140) 2020-11-20T12:04:51.212 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b6be6700 1 -- 172.21.15.187:0/971465070 learned_addr learned my addr 172.21.15.187:0/971465070 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:51.212 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.207+0000 7f70b5be4700 1 --1- 172.21.15.187:0/971465070 >> v1:172.21.15.187:6789/0 conn(0x7f70b00bfc00 0x7f70b0060ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36744/0 (socket says 172.21.15.187:36744) 2020-11-20T12:04:51.212 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.208+0000 7f70b63e5700 1 --2- 172.21.15.187:0/971465070 >> v2:172.21.15.5:3300/0 conn(0x7f70b00bf820 0x7f70b0148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.214 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.208+0000 7f70b6be6700 1 -- 172.21.15.187:0/971465070 >> v2:172.21.15.5:3300/0 conn(0x7f70b00bf820 msgr2=0x7f70b0148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:51.214 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.208+0000 7f70b6be6700 1 --2- 172.21.15.187:0/971465070 >> v2:172.21.15.5:3300/0 conn(0x7f70b00bf820 0x7f70b0148f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.215 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.208+0000 7f70b6be6700 1 -- 172.21.15.187:0/971465070 >> v1:172.21.15.187:6789/0 conn(0x7f70b00bfc00 legacy=0x7f70b0060ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.215 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.208+0000 7f70b6be6700 1 -- 172.21.15.187:0/971465070 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b009f380 con 0x7f70b0148930 2020-11-20T12:04:51.215 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.208+0000 7f70b63e5700 1 --2- 172.21.15.187:0/971465070 >> v2:172.21.15.5:3300/0 conn(0x7f70b00bf820 0x7f70b0148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:51.215 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.208+0000 7f70b6be6700 1 --2- 172.21.15.187:0/971465070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b0148930 0x7f70b0149440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f70b00bb660 tx=0x7f70a8006eb0).ready entity=mon.1 client_cookie=bf1fcd588548b7f7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.215 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.208+0000 7f70b53e3700 1 -- 172.21.15.187:0/971465070 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8009960 con 0x7f70b0148930 2020-11-20T12:04:51.216 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b53e3700 1 -- 172.21.15.187:0/971465070 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a8009ac0 con 0x7f70b0148930 2020-11-20T12:04:51.216 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b53e3700 1 -- 172.21.15.187:0/971465070 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8009e40 con 0x7f70b0148930 2020-11-20T12:04:51.216 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b73e7700 1 -- 172.21.15.187:0/971465070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b0148930 msgr2=0x7f70b0149440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.216 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b73e7700 1 --2- 172.21.15.187:0/971465070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b0148930 0x7f70b0149440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f70b00bb660 tx=0x7f70a8006eb0).stop 2020-11-20T12:04:51.217 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b73e7700 1 -- 172.21.15.187:0/971465070 shutdown_connections 2020-11-20T12:04:51.217 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b73e7700 1 --2- 172.21.15.187:0/971465070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b0148930 0x7f70b0149440 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.217 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b73e7700 1 --2- 172.21.15.187:0/971465070 >> v2:172.21.15.5:3300/0 conn(0x7f70b00bf820 0x7f70b0148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.217 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b73e7700 1 -- 172.21.15.187:0/971465070 >> 172.21.15.187:0/971465070 conn(0x7f70b00698f0 msgr2=0x7f70b00bef50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.217 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b73e7700 1 -- 172.21.15.187:0/971465070 shutdown_connections 2020-11-20T12:04:51.218 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.209+0000 7f70b73e7700 1 -- 172.21.15.187:0/971465070 wait complete. 2020-11-20T12:04:51.218 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.211+0000 7f70b73e7700 1 Processor -- start 2020-11-20T12:04:51.218 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.211+0000 7f70b73e7700 1 -- start start 2020-11-20T12:04:51.221 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b73e7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b00bf820 0x7f70b00b8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.221 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b73e7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70b014a7d0 0x7f70b00b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.221 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b73e7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70b00b71d0 0x7f70b00b8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.221 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b73e7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b00b9430 con 0x7f70b014a7d0 2020-11-20T12:04:51.222 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b73e7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b00a3150 con 0x7f70b00bf820 2020-11-20T12:04:51.222 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b73e7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b00a33f0 con 0x7f70b00b71d0 2020-11-20T12:04:51.222 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b63e5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b00bf820 0x7f70b00b8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.222 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b63e5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b00bf820 0x7f70b00b8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40142/0 (socket says 172.21.15.187:40142) 2020-11-20T12:04:51.223 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.212+0000 7f70b63e5700 1 -- 172.21.15.187:0/3522079337 learned_addr learned my addr 172.21.15.187:0/3522079337 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:51.223 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f70b5be4700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70b014a7d0 0x7f70b00b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.223 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f70b63e5700 1 -- 172.21.15.187:0/3522079337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70b00b71d0 msgr2=0x7f70b00b8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.223 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f70b63e5700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70b00b71d0 0x7f70b00b8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:51.224 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f70b63e5700 1 -- 172.21.15.187:0/3522079337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70b014a7d0 msgr2=0x7f70b00b8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.224 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f70b63e5700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70b014a7d0 0x7f70b00b8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.224 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f70b63e5700 1 -- 172.21.15.187:0/3522079337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b00a3690 con 0x7f70b00bf820 2020-11-20T12:04:51.224 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f70b63e5700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b00bf820 0x7f70b00b8560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f70ac00ef20 tx=0x7f70ac008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.224 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f709e7fc700 1 -- 172.21.15.187:0/3522079337 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac005a50 con 0x7f70b00bf820 2020-11-20T12:04:51.225 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f709e7fc700 1 -- 172.21.15.187:0/3522079337 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac005bb0 con 0x7f70b00bf820 2020-11-20T12:04:51.225 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b0138e70 con 0x7f70b00bf820 2020-11-20T12:04:51.225 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.213+0000 7f709e7fc700 1 -- 172.21.15.187:0/3522079337 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac00c940 con 0x7f70b00bf820 2020-11-20T12:04:51.225 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.214+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b00a3b30 con 0x7f70b00bf820 2020-11-20T12:04:51.225 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.215+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b00a3870 con 0x7f70b00bf820 2020-11-20T12:04:51.226 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.216+0000 7f709e7fc700 1 -- 172.21.15.187:0/3522079337 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f70ac00ccb0 con 0x7f70b00bf820 2020-11-20T12:04:51.226 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.216+0000 7f709e7fc700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7088056d20 0x7f70880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.226 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.216+0000 7f709e7fc700 1 -- 172.21.15.187:0/3522079337 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f70ac06fc10 con 0x7f70b00bf820 2020-11-20T12:04:51.226 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.216+0000 7f70b5be4700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7088056d20 0x7f70880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.226 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.217+0000 7f70b5be4700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7088056d20 0x7f70880591e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f70b00b9780 tx=0x7f70a0007180).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.227 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.221+0000 7f709e7fc700 1 -- 172.21.15.187:0/3522079337 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f70b00a3870 con 0x7f70b00bf820 2020-11-20T12:04:51.417 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.413+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70b00a3870 con 0x7f70b00bf820 2020-11-20T12:04:51.419 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.416+0000 7f709e7fc700 1 -- 172.21.15.187:0/3522079337 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12104 (secure 0 0 0) 0x7f70b00a3870 con 0x7f70b00bf820 2020-11-20T12:04:51.419 INFO:teuthology.misc.health.smithi187.stdout: 2020-11-20T12:04:51.419 INFO:teuthology.misc.health.smithi187.stdout:{"epoch":16,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:49.987644+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:51.421 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7088056d20 msgr2=0x7f70880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.421 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7088056d20 0x7f70880591e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f70b00b9780 tx=0x7f70a0007180).stop 2020-11-20T12:04:51.422 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b00bf820 msgr2=0x7f70b00b8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.422 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b00bf820 0x7f70b00b8560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f70ac00ef20 tx=0x7f70ac008040).stop 2020-11-20T12:04:51.422 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 shutdown_connections 2020-11-20T12:04:51.423 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7088056d20 0x7f70880591e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.423 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70b00b71d0 0x7f70b00b8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:51.423 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b00bf820 0x7f70b00b8560 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.423 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 --2- 172.21.15.187:0/3522079337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70b014a7d0 0x7f70b00b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.423 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.418+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 >> 172.21.15.187:0/3522079337 conn(0x7f70b00698f0 msgr2=0x7f70b00b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.424 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.419+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 shutdown_connections 2020-11-20T12:04:51.424 INFO:teuthology.misc.health.smithi187.stderr:2020-11-20T12:04:51.419+0000 7f70b73e7700 1 -- 172.21.15.187:0/3522079337 wait complete. 2020-11-20T12:04:51.440 DEBUG:teuthology.misc:8 of 8 OSDs are up 2020-11-20T12:04:51.440 INFO:tasks.ceph:Creating RBD pool 2020-11-20T12:04:51.441 INFO:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd pool create rbd 8 2020-11-20T12:04:51.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.667+0000 7f975dff5700 1 Processor -- start 2020-11-20T12:04:51.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.667+0000 7f975dff5700 1 -- start start 2020-11-20T12:04:51.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.668+0000 7f975dff5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9758065e30 0x7f9758099da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.668+0000 7f975dff5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97580b52f0 0x7f975809a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.669+0000 7f975dff5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9758148b60 con 0x7f9758066210 2020-11-20T12:04:51.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.669+0000 7f975dff5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9758146020 con 0x7f97580b52f0 2020-11-20T12:04:51.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.669+0000 7f975dff5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9758149540 con 0x7f9758065e30 2020-11-20T12:04:51.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.669+0000 7f9757fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97580b52f0 0x7f975809a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.669+0000 7f9756ffd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9758066210 0x7f9758061de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:58498/0 (socket says 172.21.15.187:58498) 2020-11-20T12:04:51.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.669+0000 7f9756ffd700 1 -- 172.21.15.187:0/2473406898 learned_addr learned my addr 172.21.15.187:0/2473406898 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:04:51.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.669+0000 7f97577fe700 1 --2- 172.21.15.187:0/2473406898 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9758065e30 0x7f9758099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.670+0000 7f9757fff700 1 -- 172.21.15.187:0/2473406898 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9758065e30 msgr2=0x7f9758099da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:51.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.670+0000 7f9757fff700 1 --2- 172.21.15.187:0/2473406898 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9758065e30 0x7f9758099da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.670+0000 7f9757fff700 1 -- 172.21.15.187:0/2473406898 >> v1:172.21.15.5:6789/0 conn(0x7f9758066210 legacy=0x7f9758061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.670+0000 7f9757fff700 1 -- 172.21.15.187:0/2473406898 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97580b72c0 con 0x7f97580b52f0 2020-11-20T12:04:51.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.670+0000 7f97567fc700 1 -- 172.21.15.187:0/2473406898 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657828715 0 0) 0x7f9758148b60 con 0x7f9758066210 2020-11-20T12:04:51.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.670+0000 7f9757fff700 1 --2- 172.21.15.187:0/2473406898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97580b52f0 0x7f975809a290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f9750009c90 tx=0x7f97500091d0).ready entity=mon.1 client_cookie=67ec52cbf8c96a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f97567fc700 1 -- 172.21.15.187:0/2473406898 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975000ba70 con 0x7f97580b52f0 2020-11-20T12:04:51.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f97567fc700 1 -- 172.21.15.187:0/2473406898 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975000bbd0 con 0x7f97580b52f0 2020-11-20T12:04:51.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f97567fc700 1 -- 172.21.15.187:0/2473406898 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750010470 con 0x7f97580b52f0 2020-11-20T12:04:51.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f975dff5700 1 -- 172.21.15.187:0/2473406898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97580b52f0 msgr2=0x7f975809a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f975dff5700 1 --2- 172.21.15.187:0/2473406898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97580b52f0 0x7f975809a290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f9750009c90 tx=0x7f97500091d0).stop 2020-11-20T12:04:51.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f975dff5700 1 -- 172.21.15.187:0/2473406898 shutdown_connections 2020-11-20T12:04:51.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f975dff5700 1 --2- 172.21.15.187:0/2473406898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97580b52f0 0x7f975809a290 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f975dff5700 1 --2- 172.21.15.187:0/2473406898 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9758065e30 0x7f9758099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.671+0000 7f975dff5700 1 -- 172.21.15.187:0/2473406898 >> 172.21.15.187:0/2473406898 conn(0x7f97580bacf0 msgr2=0x7f97580bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.672+0000 7f975dff5700 1 -- 172.21.15.187:0/2473406898 shutdown_connections 2020-11-20T12:04:51.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.672+0000 7f975dff5700 1 -- 172.21.15.187:0/2473406898 wait complete. 2020-11-20T12:04:51.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.673+0000 7f975dff5700 1 Processor -- start 2020-11-20T12:04:51.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.673+0000 7f975dff5700 1 -- start start 2020-11-20T12:04:51.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.674+0000 7f975dff5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9758065e30 0x7f9758157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.674+0000 7f975dff5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9758066210 0x7f9758157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.674+0000 7f975dff5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97580b52f0 0x7f97581583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.674+0000 7f975dff5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9758145390 con 0x7f9758065e30 2020-11-20T12:04:51.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.674+0000 7f975dff5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9758145630 con 0x7f9758066210 2020-11-20T12:04:51.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.674+0000 7f975dff5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97581458d0 con 0x7f97580b52f0 2020-11-20T12:04:51.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.674+0000 7f9756ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9758066210 0x7f9758157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9756ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9758066210 0x7f9758157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40158/0 (socket says 172.21.15.187:40158) 2020-11-20T12:04:51.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9756ffd700 1 -- 172.21.15.187:0/1607666074 learned_addr learned my addr 172.21.15.187:0/1607666074 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:51.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f97577fe700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9758065e30 0x7f9758157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9757fff700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97580b52f0 0x7f97581583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9756ffd700 1 -- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97580b52f0 msgr2=0x7f97581583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9756ffd700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97580b52f0 0x7f97581583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9756ffd700 1 -- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9758065e30 msgr2=0x7f9758157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9756ffd700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9758065e30 0x7f9758157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9756ffd700 1 -- 172.21.15.187:0/1607666074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975809fba0 con 0x7f9758066210 2020-11-20T12:04:51.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9757fff700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97580b52f0 0x7f97581583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:51.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f9756ffd700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9758066210 0x7f9758157c50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f974c00c760 tx=0x7f974c007a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f973f7fe700 1 -- 172.21.15.187:0/1607666074 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c007320 con 0x7f9758066210 2020-11-20T12:04:51.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.675+0000 7f973f7fe700 1 -- 172.21.15.187:0/1607666074 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c007480 con 0x7f9758066210 2020-11-20T12:04:51.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.676+0000 7f973f7fe700 1 -- 172.21.15.187:0/1607666074 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c005800 con 0x7f9758066210 2020-11-20T12:04:51.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.676+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97580a0170 con 0x7f9758066210 2020-11-20T12:04:51.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.676+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975809dee0 con 0x7f9758066210 2020-11-20T12:04:51.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.677+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9758044f60 con 0x7f9758066210 2020-11-20T12:04:51.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.677+0000 7f973f7fe700 1 -- 172.21.15.187:0/1607666074 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f974c010030 con 0x7f9758066210 2020-11-20T12:04:51.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.678+0000 7f973f7fe700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9734056d20 0x7f97340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.678+0000 7f973f7fe700 1 -- 172.21.15.187:0/1607666074 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f974c06f870 con 0x7f9758066210 2020-11-20T12:04:51.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.678+0000 7f97577fe700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9734056d20 0x7f97340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.679+0000 7f97577fe700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9734056d20 0x7f97340591e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f975815b6a0 tx=0x7f9748008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.683+0000 7f973f7fe700 1 -- 172.21.15.187:0/1607666074 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f974c045040 con 0x7f9758066210 2020-11-20T12:04:51.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:51.865+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f9758068690 con 0x7f9758066210 2020-11-20T12:04:52.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.272+0000 7f973f7fe700 1 -- 172.21.15.187:0/1607666074 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f974c048660 con 0x7f9758066210 2020-11-20T12:04:52.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9734056d20 msgr2=0x7f97340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9734056d20 0x7f97340591e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f975815b6a0 tx=0x7f9748008040).stop 2020-11-20T12:04:52.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9758066210 msgr2=0x7f9758157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9758066210 0x7f9758157c50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f974c00c760 tx=0x7f974c007a20).stop 2020-11-20T12:04:52.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 shutdown_connections 2020-11-20T12:04:52.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9734056d20 0x7f97340591e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9758065e30 0x7f9758157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97580b52f0 0x7f97581583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 --2- 172.21.15.187:0/1607666074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9758066210 0x7f9758157c50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 >> 172.21.15.187:0/1607666074 conn(0x7f97580bacf0 msgr2=0x7f9758148170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:52.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.276+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 shutdown_connections 2020-11-20T12:04:52.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.277+0000 7f975dff5700 1 -- 172.21.15.187:0/1607666074 wait complete. 2020-11-20T12:04:52.283 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'rbd' created 2020-11-20T12:04:52.310 INFO:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-20T12:04:52.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.542+0000 7f4c2cb8e700 1 Processor -- start 2020-11-20T12:04:52.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.542+0000 7f4c2cb8e700 1 -- start start 2020-11-20T12:04:52.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c2cb8e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 0x7f4c28065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c2cb8e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4c28065f90 0x7f4c28143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c2cb8e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4c28143c70 0x7f4c2809ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c2cb8e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2814ab30 con 0x7f4c28065f90 2020-11-20T12:04:52.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c2cb8e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c280b13e0 con 0x7f4c28143c70 2020-11-20T12:04:52.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c2cb8e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c28062c10 con 0x7f4c28065190 2020-11-20T12:04:52.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c27fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4c28143c70 0x7f4c2809ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c27fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4c28143c70 0x7f4c2809ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40166/0 (socket says 172.21.15.187:40166) 2020-11-20T12:04:52.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.544+0000 7f4c27fff700 1 -- 172.21.15.187:0/3939966696 learned_addr learned my addr 172.21.15.187:0/3939966696 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:52.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.545+0000 7f4c1effd700 1 --2- 172.21.15.187:0/3939966696 >> v2:172.21.15.5:3300/0 conn(0x7f4c28065f90 0x7f4c28143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.545+0000 7f4c277fe700 1 --2- 172.21.15.187:0/3939966696 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 0x7f4c28065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.545+0000 7f4c27fff700 1 -- 172.21.15.187:0/3939966696 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 msgr2=0x7f4c28065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:52.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.545+0000 7f4c27fff700 1 --2- 172.21.15.187:0/3939966696 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 0x7f4c28065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.545+0000 7f4c27fff700 1 -- 172.21.15.187:0/3939966696 >> v2:172.21.15.5:3300/0 conn(0x7f4c28065f90 msgr2=0x7f4c28143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:52.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.545+0000 7f4c27fff700 1 --2- 172.21.15.187:0/3939966696 >> v2:172.21.15.5:3300/0 conn(0x7f4c28065f90 0x7f4c28143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.545+0000 7f4c27fff700 1 -- 172.21.15.187:0/3939966696 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c280b1640 con 0x7f4c28143c70 2020-11-20T12:04:52.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.545+0000 7f4c27fff700 1 --2- 172.21.15.187:0/3939966696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c2809ca90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f4c280b6390 tx=0x7f4c20009290).ready entity=mon.1 client_cookie=dc7479eb54123332 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:52.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c26ffd700 1 -- 172.21.15.187:0/3939966696 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2000bc50 con 0x7f4c28143c70 2020-11-20T12:04:52.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c26ffd700 1 -- 172.21.15.187:0/3939966696 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2000bdb0 con 0x7f4c28143c70 2020-11-20T12:04:52.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c26ffd700 1 -- 172.21.15.187:0/3939966696 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20011610 con 0x7f4c28143c70 2020-11-20T12:04:52.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3939966696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 msgr2=0x7f4c2809ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3939966696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c2809ca90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f4c280b6390 tx=0x7f4c20009290).stop 2020-11-20T12:04:52.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3939966696 shutdown_connections 2020-11-20T12:04:52.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3939966696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c2809ca90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3939966696 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 0x7f4c28065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3939966696 >> v2:172.21.15.5:3300/0 conn(0x7f4c28065f90 0x7f4c28143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.546+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3939966696 >> 172.21.15.187:0/3939966696 conn(0x7f4c280bab30 msgr2=0x7f4c280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:52.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.547+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3939966696 shutdown_connections 2020-11-20T12:04:52.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.547+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3939966696 wait complete. 2020-11-20T12:04:52.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.548+0000 7f4c2cb8e700 1 Processor -- start 2020-11-20T12:04:52.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.548+0000 7f4c2cb8e700 1 -- start start 2020-11-20T12:04:52.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.549+0000 7f4c2cb8e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 0x7f4c2809aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.549+0000 7f4c2cb8e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4c28065f90 0x7f4c28143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.549+0000 7f4c2cb8e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c280a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.549+0000 7f4c2cb8e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2809b950 con 0x7f4c28065f90 2020-11-20T12:04:52.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.549+0000 7f4c2cb8e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2809bbc0 con 0x7f4c28143c70 2020-11-20T12:04:52.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.549+0000 7f4c2cb8e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2809be60 con 0x7f4c28065190 2020-11-20T12:04:52.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c280a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c280a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40174/0 (socket says 172.21.15.187:40174) 2020-11-20T12:04:52.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 -- 172.21.15.187:0/3225444352 learned_addr learned my addr 172.21.15.187:0/3225444352 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:52.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c1effd700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4c28065f90 0x7f4c28143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c277fe700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 0x7f4c2809aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 -- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 msgr2=0x7f4c2809aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 0x7f4c2809aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 -- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4c28065f90 msgr2=0x7f4c28143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4c28065f90 0x7f4c28143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 -- 172.21.15.187:0/3225444352 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c28143970 con 0x7f4c28143c70 2020-11-20T12:04:52.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c1effd700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4c28065f90 0x7f4c28143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:52.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.550+0000 7f4c27fff700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c280a0160 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f4c200092d0 tx=0x7f4c20016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:52.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.551+0000 7f4c1ffff700 1 -- 172.21.15.187:0/3225444352 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2000bc50 con 0x7f4c28143c70 2020-11-20T12:04:52.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.551+0000 7f4c1ffff700 1 -- 172.21.15.187:0/3225444352 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c20010c40 con 0x7f4c28143c70 2020-11-20T12:04:52.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.551+0000 7f4c1ffff700 1 -- 172.21.15.187:0/3225444352 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2001c470 con 0x7f4c28143c70 2020-11-20T12:04:52.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.551+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2809c9d0 con 0x7f4c28143c70 2020-11-20T12:04:52.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.551+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c280a0810 con 0x7f4c28143c70 2020-11-20T12:04:52.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.552+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c28061a20 con 0x7f4c28143c70 2020-11-20T12:04:52.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.552+0000 7f4c1ffff700 1 -- 172.21.15.187:0/3225444352 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4c20017070 con 0x7f4c28143c70 2020-11-20T12:04:52.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.553+0000 7f4c1ffff700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c04056d20 0x7f4c040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.553+0000 7f4c1ffff700 1 -- 172.21.15.187:0/3225444352 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4c20071a50 con 0x7f4c28143c70 2020-11-20T12:04:52.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.554+0000 7f4c277fe700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c04056d20 0x7f4c040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.554+0000 7f4c277fe700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c04056d20 0x7f4c040591e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f4c18009650 tx=0x7f4c18008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:52.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.557+0000 7f4c1ffff700 1 -- 172.21.15.187:0/3225444352 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4c200471e0 con 0x7f4c28143c70 2020-11-20T12:04:52.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:52.751+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4c280afc60 con 0x7f4c28143c70 2020-11-20T12:04:53.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.278+0000 7f4c1ffff700 1 -- 172.21.15.187:0/3225444352 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4c20015030 con 0x7f4c28143c70 2020-11-20T12:04:53.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.282+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c04056d20 msgr2=0x7f4c040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.282+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c04056d20 0x7f4c040591e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f4c18009650 tx=0x7f4c18008040).stop 2020-11-20T12:04:53.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.282+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 msgr2=0x7f4c280a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.282+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c280a0160 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f4c200092d0 tx=0x7f4c20016000).stop 2020-11-20T12:04:53.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.284+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 shutdown_connections 2020-11-20T12:04:53.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.284+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c04056d20 0x7f4c040591e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.284+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c28143c70 0x7f4c280a0160 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.284+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4c28065190 0x7f4c2809aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.284+0000 7f4c2cb8e700 1 --2- 172.21.15.187:0/3225444352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4c28065f90 0x7f4c28143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.284+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 >> 172.21.15.187:0/3225444352 conn(0x7f4c280bab30 msgr2=0x7f4c28066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:53.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.286+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 shutdown_connections 2020-11-20T12:04:53.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.287+0000 7f4c2cb8e700 1 -- 172.21.15.187:0/3225444352 wait complete. 2020-11-20T12:04:53.293 INFO:teuthology.orchestra.run.smithi187.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-20T12:04:53.319 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2020-11-20T12:04:53.319 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:04:53.320 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:04:53.325 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:04:53.326 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:04:53.326 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:04:53.334 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:04:53.335 INFO:tasks.ceph:Setting up CephFS filesystem... 2020-11-20T12:04:53.336 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:53.485 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:04:53.527 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:04:53.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.543+0000 7fea6cb8d700 1 Processor -- start 2020-11-20T12:04:53.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.543+0000 7fea6cb8d700 1 -- start start 2020-11-20T12:04:53.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.544+0000 7fea6cb8d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea680661f0 0x7fea680b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.544+0000 7fea6cb8d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fea68099ce0 0x7fea680b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.544+0000 7fea6cb8d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea680b5df0 con 0x7fea68065e10 2020-11-20T12:04:53.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.544+0000 7fea6cb8d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6809a440 con 0x7fea68099ce0 2020-11-20T12:04:53.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.544+0000 7fea6cb8d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6809a1a0 con 0x7fea680661f0 2020-11-20T12:04:53.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.544+0000 7fea6659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fea68065e10 0x7fea68144470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36802/0 (socket says 172.21.15.187:36802) 2020-11-20T12:04:53.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.544+0000 7fea6659c700 1 -- 172.21.15.187:0/707398651 learned_addr learned my addr 172.21.15.187:0/707398651 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:04:53.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.545+0000 7fea65d9b700 1 --2- 172.21.15.187:0/707398651 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea680661f0 0x7fea680b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.545+0000 7fea66d9d700 1 --2- 172.21.15.187:0/707398651 >> v2:172.21.15.5:3300/0 conn(0x7fea68099ce0 0x7fea680b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.545+0000 7fea66d9d700 1 -- 172.21.15.187:0/707398651 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea680661f0 msgr2=0x7fea680b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:53.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.545+0000 7fea66d9d700 1 --2- 172.21.15.187:0/707398651 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea680661f0 0x7fea680b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.545+0000 7fea66d9d700 1 -- 172.21.15.187:0/707398651 >> v1:172.21.15.187:6789/0 conn(0x7fea68065e10 legacy=0x7fea68144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.545+0000 7fea66d9d700 1 -- 172.21.15.187:0/707398651 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea680b0160 con 0x7fea68099ce0 2020-11-20T12:04:53.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.546+0000 7fea66d9d700 1 --2- 172.21.15.187:0/707398651 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68099ce0 0x7fea680b67d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fea60009c90 tx=0x7fea600091d0).ready entity=mon.0 client_cookie=d589c57ab9e47215 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:53.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.546+0000 7fea6559a700 1 -- 172.21.15.187:0/707398651 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6000bb00 con 0x7fea68099ce0 2020-11-20T12:04:53.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.546+0000 7fea6559a700 1 -- 172.21.15.187:0/707398651 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6000bc60 con 0x7fea68099ce0 2020-11-20T12:04:53.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6559a700 1 -- 172.21.15.187:0/707398651 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60010410 con 0x7fea68099ce0 2020-11-20T12:04:53.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6cb8d700 1 -- 172.21.15.187:0/707398651 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68099ce0 msgr2=0x7fea680b67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/707398651 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68099ce0 0x7fea680b67d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fea60009c90 tx=0x7fea600091d0).stop 2020-11-20T12:04:53.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6cb8d700 1 -- 172.21.15.187:0/707398651 shutdown_connections 2020-11-20T12:04:53.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/707398651 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68099ce0 0x7fea680b67d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/707398651 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea680661f0 0x7fea680b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6cb8d700 1 -- 172.21.15.187:0/707398651 >> 172.21.15.187:0/707398651 conn(0x7fea680bacd0 msgr2=0x7fea680bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:53.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6cb8d700 1 -- 172.21.15.187:0/707398651 shutdown_connections 2020-11-20T12:04:53.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.547+0000 7fea6cb8d700 1 -- 172.21.15.187:0/707398651 wait complete. 2020-11-20T12:04:53.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.549+0000 7fea6cb8d700 1 Processor -- start 2020-11-20T12:04:53.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.549+0000 7fea6cb8d700 1 -- start start 2020-11-20T12:04:53.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea6cb8d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68065e10 0x7fea681587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea6cb8d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea680661f0 0x7fea6809f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea6cb8d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea68099ce0 0x7fea680a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea6cb8d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6809ecb0 con 0x7fea68065e10 2020-11-20T12:04:53.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea6cb8d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6809ef50 con 0x7fea680661f0 2020-11-20T12:04:53.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea6cb8d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6809f1f0 con 0x7fea68099ce0 2020-11-20T12:04:53.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea65d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea680661f0 0x7fea6809f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea65d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea680661f0 0x7fea6809f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40204/0 (socket says 172.21.15.187:40204) 2020-11-20T12:04:53.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea65d9b700 1 -- 172.21.15.187:0/812884961 learned_addr learned my addr 172.21.15.187:0/812884961 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:53.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea6659c700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68065e10 0x7fea681587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.550+0000 7fea66d9d700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea68099ce0 0x7fea680a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea65d9b700 1 -- 172.21.15.187:0/812884961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea68099ce0 msgr2=0x7fea680a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea65d9b700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea68099ce0 0x7fea680a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea65d9b700 1 -- 172.21.15.187:0/812884961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68065e10 msgr2=0x7fea681587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea65d9b700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68065e10 0x7fea681587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea65d9b700 1 -- 172.21.15.187:0/812884961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6809fb80 con 0x7fea680661f0 2020-11-20T12:04:53.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea65d9b700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea680661f0 0x7fea6809f690 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fea5c00eba0 tx=0x7fea5c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:53.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea567fc700 1 -- 172.21.15.187:0/812884961 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00c8b0 con 0x7fea680661f0 2020-11-20T12:04:53.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea567fc700 1 -- 172.21.15.187:0/812884961 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c00ca10 con 0x7fea680661f0 2020-11-20T12:04:53.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea567fc700 1 -- 172.21.15.187:0/812884961 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00cd90 con 0x7fea680661f0 2020-11-20T12:04:53.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6809fdd0 con 0x7fea680661f0 2020-11-20T12:04:53.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.551+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea680a0b10 con 0x7fea680661f0 2020-11-20T12:04:53.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.553+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea68158ce0 con 0x7fea680661f0 2020-11-20T12:04:53.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.553+0000 7fea567fc700 1 -- 172.21.15.187:0/812884961 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fea5c022070 con 0x7fea680661f0 2020-11-20T12:04:53.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.554+0000 7fea567fc700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fea3c056d50 0x7fea3c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.554+0000 7fea567fc700 1 -- 172.21.15.187:0/812884961 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fea5c06fff0 con 0x7fea680661f0 2020-11-20T12:04:53.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.554+0000 7fea6659c700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fea3c056d50 0x7fea3c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.555+0000 7fea6659c700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fea3c056d50 0x7fea3c059210 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fea580045d0 tx=0x7fea58007b80).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:53.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.558+0000 7fea567fc700 1 -- 172.21.15.187:0/812884961 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fea5c0456f0 con 0x7fea680661f0 2020-11-20T12:04:53.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.741+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea68063600 con 0x7fea680661f0 2020-11-20T12:04:53.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.743+0000 7fea567fc700 1 -- 172.21.15.187:0/812884961 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+13637 (secure 0 0 0) 0x7fea5c016070 con 0x7fea680661f0 2020-11-20T12:04:53.748 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:04:53.748 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":18,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:53.266146+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:53.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fea3c056d50 msgr2=0x7fea3c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fea3c056d50 0x7fea3c059210 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fea580045d0 tx=0x7fea58007b80).stop 2020-11-20T12:04:53.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea680661f0 msgr2=0x7fea6809f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea680661f0 0x7fea6809f690 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fea5c00eba0 tx=0x7fea5c008040).stop 2020-11-20T12:04:53.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 shutdown_connections 2020-11-20T12:04:53.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fea3c056d50 0x7fea3c059210 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea68065e10 0x7fea681587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fea68099ce0 0x7fea680a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 --2- 172.21.15.187:0/812884961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea680661f0 0x7fea6809f690 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 >> 172.21.15.187:0/812884961 conn(0x7fea680bacd0 msgr2=0x7fea681438a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:53.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 shutdown_connections 2020-11-20T12:04:53.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:53.746+0000 7fea6cb8d700 1 -- 172.21.15.187:0/812884961 wait complete. 2020-11-20T12:04:53.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:04:53.773 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:04:54.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.010+0000 7f41ac8c9700 1 Processor -- start 2020-11-20T12:04:54.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.010+0000 7f41ac8c9700 1 -- start start 2020-11-20T12:04:54.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f41ac8c9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f41a81484d0 0x7f41a8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f41ac8c9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f41a80622c0 0x7f41a80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f41ac8c9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80b1300 con 0x7f41a8066f70 2020-11-20T12:04:54.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f41ac8c9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a8067730 con 0x7f41a81484d0 2020-11-20T12:04:54.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f41ac8c9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a80b15a0 con 0x7f41a80622c0 2020-11-20T12:04:54.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f419effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f41a80622c0 0x7f41a80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f419effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f41a80622c0 0x7f41a80b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40212/0 (socket says 172.21.15.187:40212) 2020-11-20T12:04:54.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f419effd700 1 -- 172.21.15.187:0/2716963333 learned_addr learned my addr 172.21.15.187:0/2716963333 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:54.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.012+0000 7f41a77fe700 1 --2- 172.21.15.187:0/2716963333 >> v2:172.21.15.5:3300/0 conn(0x7f41a81484d0 0x7f41a8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.013+0000 7f419effd700 1 -- 172.21.15.187:0/2716963333 >> v2:172.21.15.5:3300/0 conn(0x7f41a81484d0 msgr2=0x7f41a8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:54.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.013+0000 7f419effd700 1 --2- 172.21.15.187:0/2716963333 >> v2:172.21.15.5:3300/0 conn(0x7f41a81484d0 0x7f41a8061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.013+0000 7f419effd700 1 -- 172.21.15.187:0/2716963333 >> v1:172.21.15.5:6789/0 conn(0x7f41a8066f70 legacy=0x7f41a80b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:54.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.013+0000 7f419effd700 1 -- 172.21.15.187:0/2716963333 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80b1840 con 0x7f41a80622c0 2020-11-20T12:04:54.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.013+0000 7f419effd700 1 --2- 172.21.15.187:0/2716963333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a80622c0 0x7f41a80b5640 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f41a8068600 tx=0x7f4190006eb0).ready entity=mon.1 client_cookie=c817c36dffe5b61 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.013+0000 7f41a6ffd700 1 -- 172.21.15.187:0/2716963333 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190009890 con 0x7f41a80622c0 2020-11-20T12:04:54.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.013+0000 7f41a6ffd700 1 -- 172.21.15.187:0/2716963333 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41900099f0 con 0x7f41a80622c0 2020-11-20T12:04:54.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.013+0000 7f41a6ffd700 1 -- 172.21.15.187:0/2716963333 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190009d70 con 0x7f41a80622c0 2020-11-20T12:04:54.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.014+0000 7f41ac8c9700 1 -- 172.21.15.187:0/2716963333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a80622c0 msgr2=0x7f41a80b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.014+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/2716963333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a80622c0 0x7f41a80b5640 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f41a8068600 tx=0x7f4190006eb0).stop 2020-11-20T12:04:54.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.014+0000 7f41ac8c9700 1 -- 172.21.15.187:0/2716963333 shutdown_connections 2020-11-20T12:04:54.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.014+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/2716963333 >> v2:172.21.15.5:3300/0 conn(0x7f41a81484d0 0x7f41a8061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.014+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/2716963333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a80622c0 0x7f41a80b5640 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.014+0000 7f41ac8c9700 1 -- 172.21.15.187:0/2716963333 >> 172.21.15.187:0/2716963333 conn(0x7f41a80bab30 msgr2=0x7f41a80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.014+0000 7f41ac8c9700 1 -- 172.21.15.187:0/2716963333 shutdown_connections 2020-11-20T12:04:54.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.014+0000 7f41ac8c9700 1 -- 172.21.15.187:0/2716963333 wait complete. 2020-11-20T12:04:54.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.016+0000 7f41ac8c9700 1 Processor -- start 2020-11-20T12:04:54.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.016+0000 7f41ac8c9700 1 -- start start 2020-11-20T12:04:54.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.017+0000 7f41ac8c9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41a80622c0 0x7f41a80b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.017+0000 7f41ac8c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a8066f70 0x7f41a80ba210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.017+0000 7f41ac8c9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41a81484d0 0x7f41a8137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.017+0000 7f41ac8c9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80b49f0 con 0x7f41a81484d0 2020-11-20T12:04:54.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.017+0000 7f41ac8c9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a809b1f0 con 0x7f41a8066f70 2020-11-20T12:04:54.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.017+0000 7f41ac8c9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80b44b0 con 0x7f41a80622c0 2020-11-20T12:04:54.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.017+0000 7f419effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a8066f70 0x7f41a80ba210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f419effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a8066f70 0x7f41a80ba210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40218/0 (socket says 172.21.15.187:40218) 2020-11-20T12:04:54.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f419effd700 1 -- 172.21.15.187:0/669795454 learned_addr learned my addr 172.21.15.187:0/669795454 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:54.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f41a7fff700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41a81484d0 0x7f41a8137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f41a77fe700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41a80622c0 0x7f41a80b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f419effd700 1 -- 172.21.15.187:0/669795454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41a80622c0 msgr2=0x7f41a80b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f419effd700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41a80622c0 0x7f41a80b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f419effd700 1 -- 172.21.15.187:0/669795454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41a81484d0 msgr2=0x7f41a8137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f419effd700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41a81484d0 0x7f41a8137110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f419effd700 1 -- 172.21.15.187:0/669795454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80b4750 con 0x7f41a8066f70 2020-11-20T12:04:54.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.018+0000 7f419effd700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a8066f70 0x7f41a80ba210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4190010000 tx=0x7f4190015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.019+0000 7f419ffff700 1 -- 172.21.15.187:0/669795454 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190011dd0 con 0x7f41a8066f70 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.019+0000 7f419ffff700 1 -- 172.21.15.187:0/669795454 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419001e460 con 0x7f41a8066f70 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.019+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a80b41e0 con 0x7f41a8066f70 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.019+0000 7f419ffff700 1 -- 172.21.15.187:0/669795454 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419001e800 con 0x7f41a8066f70 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.019+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a80a05d0 con 0x7f41a8066f70 2020-11-20T12:04:54.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.020+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a809dd40 con 0x7f41a8066f70 2020-11-20T12:04:54.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.021+0000 7f419ffff700 1 -- 172.21.15.187:0/669795454 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4190025030 con 0x7f41a8066f70 2020-11-20T12:04:54.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.021+0000 7f419ffff700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4178056d20 0x7f41780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.021+0000 7f419ffff700 1 -- 172.21.15.187:0/669795454 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4190071420 con 0x7f41a8066f70 2020-11-20T12:04:54.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.022+0000 7f41a77fe700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4178056d20 0x7f41780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.022+0000 7f41a77fe700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4178056d20 0x7f41780591e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f41980095b0 tx=0x7f4198008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.025+0000 7f419ffff700 1 -- 172.21.15.187:0/669795454 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f41a809dd40 con 0x7f41a8066f70 2020-11-20T12:04:54.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.210+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f41a809dd40 con 0x7f41a8066f70 2020-11-20T12:04:54.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.210+0000 7f419ffff700 1 -- 172.21.15.187:0/669795454 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v18) v1 ==== 110+0+10 (secure 0 0 0) 0x7f41a809dd40 con 0x7f41a8066f70 2020-11-20T12:04:54.215 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:04:54.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.212+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4178056d20 msgr2=0x7f41780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4178056d20 0x7f41780591e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f41980095b0 tx=0x7f4198008040).stop 2020-11-20T12:04:54.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a8066f70 msgr2=0x7f41a80ba210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a8066f70 0x7f41a80ba210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4190010000 tx=0x7f4190015000).stop 2020-11-20T12:04:54.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 shutdown_connections 2020-11-20T12:04:54.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4178056d20 0x7f41780591e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41a81484d0 0x7f41a8137110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41a80622c0 0x7f41a80b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 --2- 172.21.15.187:0/669795454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a8066f70 0x7f41a80ba210 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 >> 172.21.15.187:0/669795454 conn(0x7f41a80bab30 msgr2=0x7f41a813f940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 shutdown_connections 2020-11-20T12:04:54.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.213+0000 7f41ac8c9700 1 -- 172.21.15.187:0/669795454 wait complete. 2020-11-20T12:04:54.241 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:04:54.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.480+0000 7f150ea5e700 1 Processor -- start 2020-11-20T12:04:54.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.481+0000 7f150ea5e700 1 -- start start 2020-11-20T12:04:54.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.482+0000 7f150ea5e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f15080b6650 0x7f15080afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.482+0000 7f150ea5e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15080b1300 con 0x7f15081484d0 2020-11-20T12:04:54.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.482+0000 7f150ea5e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15080b15a0 con 0x7f1508066f70 2020-11-20T12:04:54.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.482+0000 7f150ea5e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1508067730 con 0x7f15080b6650 2020-11-20T12:04:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.483+0000 7f150d25b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f15080b6650 0x7f15080afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.483+0000 7f150e25d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1508066f70 0x7f15080aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36834/0 (socket says 172.21.15.187:36834) 2020-11-20T12:04:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.483+0000 7f150e25d700 1 -- 172.21.15.187:0/2076565271 learned_addr learned my addr 172.21.15.187:0/2076565271 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:04:54.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.483+0000 7f150ca5a700 1 -- 172.21.15.187:0/2076565271 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544687219 0 0) 0x7f15080b15a0 con 0x7f1508066f70 2020-11-20T12:04:54.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.483+0000 7f150d25b700 1 -- 172.21.15.187:0/2076565271 >> v1:172.21.15.187:6789/0 conn(0x7f1508066f70 legacy=0x7f15080aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.483+0000 7f150d25b700 1 -- 172.21.15.187:0/2076565271 >> v1:172.21.15.5:6789/0 conn(0x7f15081484d0 legacy=0x7f1508061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.483+0000 7f150d25b700 1 -- 172.21.15.187:0/2076565271 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15080b1840 con 0x7f15080b6650 2020-11-20T12:04:54.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.484+0000 7f150d25b700 1 --2- 172.21.15.187:0/2076565271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15080b6650 0x7f15080afa90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f14f800b9a0 tx=0x7f14f8006df0).ready entity=mon.1 client_cookie=57aeb81440b513f7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.484+0000 7f150ca5a700 1 -- 172.21.15.187:0/2076565271 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f80097b0 con 0x7f15080b6650 2020-11-20T12:04:54.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.484+0000 7f150ca5a700 1 -- 172.21.15.187:0/2076565271 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8009910 con 0x7f15080b6650 2020-11-20T12:04:54.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.484+0000 7f150ca5a700 1 -- 172.21.15.187:0/2076565271 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8009c90 con 0x7f15080b6650 2020-11-20T12:04:54.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.485+0000 7f150ea5e700 1 -- 172.21.15.187:0/2076565271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15080b6650 msgr2=0x7f15080afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.485+0000 7f150ea5e700 1 --2- 172.21.15.187:0/2076565271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15080b6650 0x7f15080afa90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f14f800b9a0 tx=0x7f14f8006df0).stop 2020-11-20T12:04:54.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.485+0000 7f150ea5e700 1 -- 172.21.15.187:0/2076565271 shutdown_connections 2020-11-20T12:04:54.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.485+0000 7f150ea5e700 1 --2- 172.21.15.187:0/2076565271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15080b6650 0x7f15080afa90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.485+0000 7f150ea5e700 1 -- 172.21.15.187:0/2076565271 >> 172.21.15.187:0/2076565271 conn(0x7f15080bab30 msgr2=0x7f15080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.485+0000 7f150ea5e700 1 -- 172.21.15.187:0/2076565271 shutdown_connections 2020-11-20T12:04:54.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.485+0000 7f150ea5e700 1 -- 172.21.15.187:0/2076565271 wait complete. 2020-11-20T12:04:54.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.487+0000 7f150ea5e700 1 Processor -- start 2020-11-20T12:04:54.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.487+0000 7f150ea5e700 1 -- start start 2020-11-20T12:04:54.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.488+0000 7f150ea5e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1508066f70 0x7f150809b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.488+0000 7f150ea5e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15081484d0 0x7f1508144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.488+0000 7f150ea5e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f150809a670 0x7f1508142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.488+0000 7f150ea5e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15080a04d0 con 0x7f15081484d0 2020-11-20T12:04:54.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.488+0000 7f150ea5e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150809abf0 con 0x7f1508066f70 2020-11-20T12:04:54.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.488+0000 7f150ea5e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150809ae90 con 0x7f150809a670 2020-11-20T12:04:54.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.488+0000 7f150da5c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1508066f70 0x7f150809b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.488+0000 7f150d25b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15081484d0 0x7f1508144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150d25b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15081484d0 0x7f1508144440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:60726/0 (socket says 172.21.15.187:60726) 2020-11-20T12:04:54.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150d25b700 1 -- 172.21.15.187:0/159981047 learned_addr learned my addr 172.21.15.187:0/159981047 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:54.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150e25d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f150809a670 0x7f1508142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150da5c700 1 -- 172.21.15.187:0/159981047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f150809a670 msgr2=0x7f1508142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150da5c700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f150809a670 0x7f1508142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150da5c700 1 -- 172.21.15.187:0/159981047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15081484d0 msgr2=0x7f1508144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150da5c700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15081484d0 0x7f1508144440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150da5c700 1 -- 172.21.15.187:0/159981047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150809b130 con 0x7f1508066f70 2020-11-20T12:04:54.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.489+0000 7f150da5c700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1508066f70 0x7f150809b450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f150400c740 tx=0x7f1504007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.490+0000 7f14f5ffb700 1 -- 172.21.15.187:0/159981047 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504007020 con 0x7f1508066f70 2020-11-20T12:04:54.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.490+0000 7f14f5ffb700 1 -- 172.21.15.187:0/159981047 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1504007180 con 0x7f1508066f70 2020-11-20T12:04:54.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.490+0000 7f14f5ffb700 1 -- 172.21.15.187:0/159981047 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15040064f0 con 0x7f1508066f70 2020-11-20T12:04:54.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.490+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150809b940 con 0x7f1508066f70 2020-11-20T12:04:54.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.490+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15080b1f50 con 0x7f1508066f70 2020-11-20T12:04:54.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.491+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1508061a20 con 0x7f1508066f70 2020-11-20T12:04:54.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.492+0000 7f14f5ffb700 1 -- 172.21.15.187:0/159981047 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1504010030 con 0x7f1508066f70 2020-11-20T12:04:54.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.492+0000 7f14f5ffb700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f14e0056d20 0x7f14e00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.492+0000 7f14f5ffb700 1 -- 172.21.15.187:0/159981047 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f150406f3d0 con 0x7f1508066f70 2020-11-20T12:04:54.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.493+0000 7f150d25b700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f14e0056d20 0x7f14e00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.493+0000 7f150d25b700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f14e0056d20 0x7f14e00591e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f150809f100 tx=0x7f14f8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.497+0000 7f14f5ffb700 1 -- 172.21.15.187:0/159981047 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1504044ad0 con 0x7f1508066f70 2020-11-20T12:04:54.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.684+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f15080624e0 con 0x7f1508066f70 2020-11-20T12:04:54.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.684+0000 7f14f5ffb700 1 -- 172.21.15.187:0/159981047 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1504007cb0 con 0x7f1508066f70 2020-11-20T12:04:54.689 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:04:54.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.686+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f14e0056d20 msgr2=0x7f14e00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.686+0000 7f150ea5e700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f14e0056d20 0x7f14e00591e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f150809f100 tx=0x7f14f8015000).stop 2020-11-20T12:04:54.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.686+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1508066f70 msgr2=0x7f150809b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.686+0000 7f150ea5e700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1508066f70 0x7f150809b450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f150400c740 tx=0x7f1504007b90).stop 2020-11-20T12:04:54.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.687+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 shutdown_connections 2020-11-20T12:04:54.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.687+0000 7f150ea5e700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f14e0056d20 0x7f14e00591e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.687+0000 7f150ea5e700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f150809a670 0x7f1508142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.687+0000 7f150ea5e700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1508066f70 0x7f150809b450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.687+0000 7f150ea5e700 1 --2- 172.21.15.187:0/159981047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15081484d0 0x7f1508144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.687+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 >> 172.21.15.187:0/159981047 conn(0x7f15080bab30 msgr2=0x7f15080ba200 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.687+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 shutdown_connections 2020-11-20T12:04:54.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.687+0000 7f150ea5e700 1 -- 172.21.15.187:0/159981047 wait complete. 2020-11-20T12:04:54.715 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:04:54.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.951+0000 7fe46e8c7700 1 Processor -- start 2020-11-20T12:04:54.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.952+0000 7fe46e8c7700 1 -- start start 2020-11-20T12:04:54.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.953+0000 7fe46e8c7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1760 0x7fe468065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.953+0000 7fe46e8c7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe468065680 0x7fe46814c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.953+0000 7fe46e8c7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe468150390 con 0x7fe4680b1380 2020-11-20T12:04:54.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.953+0000 7fe46e8c7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4681511e0 con 0x7fe468065680 2020-11-20T12:04:54.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.953+0000 7fe46e8c7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46814d760 con 0x7fe4680b1760 2020-11-20T12:04:54.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.954+0000 7fe46e0c6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe468065680 0x7fe46814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.954+0000 7fe46d0c4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1760 0x7fe468065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.954+0000 7fe46d0c4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1760 0x7fe468065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:41126/0 (socket says 172.21.15.187:41126) 2020-11-20T12:04:54.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.954+0000 7fe46e0c6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe468065680 0x7fe46814c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:60736/0 (socket says 172.21.15.187:60736) 2020-11-20T12:04:54.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.954+0000 7fe46d0c4700 1 -- 172.21.15.187:0/1883662022 learned_addr learned my addr 172.21.15.187:0/1883662022 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:54.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.955+0000 7fe46d0c4700 1 -- 172.21.15.187:0/1883662022 >> v2:172.21.15.5:3300/0 conn(0x7fe468065680 msgr2=0x7fe46814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:54.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.955+0000 7fe46d0c4700 1 --2- 172.21.15.187:0/1883662022 >> v2:172.21.15.5:3300/0 conn(0x7fe468065680 0x7fe46814c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.955+0000 7fe46d0c4700 1 -- 172.21.15.187:0/1883662022 >> v1:172.21.15.5:6789/0 conn(0x7fe4680b1380 legacy=0x7fe468150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.955+0000 7fe46d0c4700 1 -- 172.21.15.187:0/1883662022 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468150630 con 0x7fe4680b1760 2020-11-20T12:04:54.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.955+0000 7fe46d0c4700 1 --2- 172.21.15.187:0/1883662022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1760 0x7fe468065190 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe458006fd0 tx=0x7fe458006eb0).ready entity=mon.2 client_cookie=7bf64892dc55fae4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46c8c3700 1 -- 172.21.15.187:0/1883662022 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458009810 con 0x7fe4680b1760 2020-11-20T12:04:54.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46c8c3700 1 -- 172.21.15.187:0/1883662022 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe458009970 con 0x7fe4680b1760 2020-11-20T12:04:54.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46c8c3700 1 -- 172.21.15.187:0/1883662022 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458009cf0 con 0x7fe4680b1760 2020-11-20T12:04:54.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1883662022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1760 msgr2=0x7fe468065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1883662022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1760 0x7fe468065190 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe458006fd0 tx=0x7fe458006eb0).stop 2020-11-20T12:04:54.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1883662022 shutdown_connections 2020-11-20T12:04:54.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1883662022 >> v2:172.21.15.5:3300/0 conn(0x7fe468065680 0x7fe46814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1883662022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1760 0x7fe468065190 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1883662022 >> 172.21.15.187:0/1883662022 conn(0x7fe4680bab30 msgr2=0x7fe4680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.956+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1883662022 shutdown_connections 2020-11-20T12:04:54.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.957+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1883662022 wait complete. 2020-11-20T12:04:54.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.958+0000 7fe46e8c7700 1 Processor -- start 2020-11-20T12:04:54.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.958+0000 7fe46e8c7700 1 -- start start 2020-11-20T12:04:54.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46e8c7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1380 0x7fe46814a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46e8c7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4680b1760 0x7fe4680b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46e8c7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4680b77d0 0x7fe468160740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46e8c7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe468160070 con 0x7fe4680b77d0 2020-11-20T12:04:54.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46e8c7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46814a620 con 0x7fe4680b1760 2020-11-20T12:04:54.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46e8c7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680b3dc0 con 0x7fe4680b1380 2020-11-20T12:04:54.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46d0c4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4680b1760 0x7fe4680b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46d0c4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4680b1760 0x7fe4680b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40246/0 (socket says 172.21.15.187:40246) 2020-11-20T12:04:54.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46d0c4700 1 -- 172.21.15.187:0/1515646583 learned_addr learned my addr 172.21.15.187:0/1515646583 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:54.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46d8c5700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1380 0x7fe46814a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46e0c6700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4680b77d0 0x7fe468160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46d0c4700 1 -- 172.21.15.187:0/1515646583 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1380 msgr2=0x7fe46814a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46d0c4700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1380 0x7fe46814a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46d0c4700 1 -- 172.21.15.187:0/1515646583 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4680b77d0 msgr2=0x7fe468160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.960+0000 7fe46d0c4700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4680b77d0 0x7fe468160740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.961+0000 7fe46d0c4700 1 -- 172.21.15.187:0/1515646583 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4681602a0 con 0x7fe4680b1760 2020-11-20T12:04:54.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.961+0000 7fe46d0c4700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4680b1760 0x7fe4680b72e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe458009fd0 tx=0x7fe458015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.961+0000 7fe455ffb700 1 -- 172.21.15.187:0/1515646583 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458011e50 con 0x7fe4680b1760 2020-11-20T12:04:54.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.961+0000 7fe455ffb700 1 -- 172.21.15.187:0/1515646583 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45801e460 con 0x7fe4680b1760 2020-11-20T12:04:54.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.961+0000 7fe455ffb700 1 -- 172.21.15.187:0/1515646583 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45801e7e0 con 0x7fe4680b1760 2020-11-20T12:04:54.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.961+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46809d470 con 0x7fe4680b1760 2020-11-20T12:04:54.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.961+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4680b45f0 con 0x7fe4680b1760 2020-11-20T12:04:54.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.962+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe468067d90 con 0x7fe4680b1760 2020-11-20T12:04:54.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.963+0000 7fe455ffb700 1 -- 172.21.15.187:0/1515646583 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe45801a040 con 0x7fe4680b1760 2020-11-20T12:04:54.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.964+0000 7fe455ffb700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe440056d20 0x7fe4400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.964+0000 7fe455ffb700 1 -- 172.21.15.187:0/1515646583 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe458071120 con 0x7fe4680b1760 2020-11-20T12:04:54.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.964+0000 7fe46d8c5700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe440056d20 0x7fe4400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.965+0000 7fe46d8c5700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe440056d20 0x7fe4400591e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe464003510 tx=0x7fe464009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:54.968+0000 7fe455ffb700 1 -- 172.21.15.187:0/1515646583 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe4580468d0 con 0x7fe4680b1760 2020-11-20T12:04:55.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.156+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe4680b3f00 con 0x7fe4680b1760 2020-11-20T12:04:55.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.156+0000 7fe455ffb700 1 -- 172.21.15.187:0/1515646583 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+143 (secure 0 0 0) 0x7fe458023070 con 0x7fe4680b1760 2020-11-20T12:04:55.161 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:04:55.161 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:04:55.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:04:55.162 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:04:55.162 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:04:55.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:04:55.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:04:55.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe440056d20 msgr2=0x7fe4400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe440056d20 0x7fe4400591e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe464003510 tx=0x7fe464009040).stop 2020-11-20T12:04:55.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4680b1760 msgr2=0x7fe4680b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4680b1760 0x7fe4680b72e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe458009fd0 tx=0x7fe458015000).stop 2020-11-20T12:04:55.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 shutdown_connections 2020-11-20T12:04:55.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe440056d20 0x7fe4400591e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4680b77d0 0x7fe468160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4680b1380 0x7fe46814a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 --2- 172.21.15.187:0/1515646583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4680b1760 0x7fe4680b72e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 >> 172.21.15.187:0/1515646583 conn(0x7fe4680bab30 msgr2=0x7fe4680ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 shutdown_connections 2020-11-20T12:04:55.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.159+0000 7fe46e8c7700 1 -- 172.21.15.187:0/1515646583 wait complete. 2020-11-20T12:04:55.184 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:04:55.185 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:04:55.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.423+0000 7fd3f9c84700 1 Processor -- start 2020-11-20T12:04:55.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.424+0000 7fd3f9c84700 1 -- start start 2020-11-20T12:04:55.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.425+0000 7fd3f9c84700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3f40b1380 0x7fd3f4150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.425+0000 7fd3f9c84700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd3f40b1760 0x7fd3f4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.425+0000 7fd3f9c84700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f4150390 con 0x7fd3f4065680 2020-11-20T12:04:55.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.425+0000 7fd3f9c84700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f4150630 con 0x7fd3f40b1760 2020-11-20T12:04:55.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.425+0000 7fd3f9c84700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f4068510 con 0x7fd3f40b1380 2020-11-20T12:04:55.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.425+0000 7fd3f8c82700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3f40b1380 0x7fd3f4150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.425+0000 7fd3f8c82700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3f40b1380 0x7fd3f4150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40252/0 (socket says 172.21.15.187:40252) 2020-11-20T12:04:55.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.425+0000 7fd3f8c82700 1 -- 172.21.15.187:0/2012476459 learned_addr learned my addr 172.21.15.187:0/2012476459 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:55.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.426+0000 7fd3ebfff700 1 --2- 172.21.15.187:0/2012476459 >> v2:172.21.15.5:3300/0 conn(0x7fd3f40b1760 0x7fd3f4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.426+0000 7fd3f8c82700 1 -- 172.21.15.187:0/2012476459 >> v2:172.21.15.5:3300/0 conn(0x7fd3f40b1760 msgr2=0x7fd3f4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:55.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.426+0000 7fd3f8c82700 1 --2- 172.21.15.187:0/2012476459 >> v2:172.21.15.5:3300/0 conn(0x7fd3f40b1760 0x7fd3f4065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.426+0000 7fd3f8c82700 1 -- 172.21.15.187:0/2012476459 >> v1:172.21.15.5:6789/0 conn(0x7fd3f4065680 legacy=0x7fd3f4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:55.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.426+0000 7fd3f8c82700 1 -- 172.21.15.187:0/2012476459 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f4150fc0 con 0x7fd3f40b1380 2020-11-20T12:04:55.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.427+0000 7fd3f8c82700 1 --2- 172.21.15.187:0/2012476459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f40b1380 0x7fd3f4150ad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd3f40b85e0 tx=0x7fd3f0009290).ready entity=mon.1 client_cookie=17f1b84a9d2dbf3f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:55.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.427+0000 7fd3eb7fe700 1 -- 172.21.15.187:0/2012476459 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f000bb80 con 0x7fd3f40b1380 2020-11-20T12:04:55.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.427+0000 7fd3eb7fe700 1 -- 172.21.15.187:0/2012476459 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f000bce0 con 0x7fd3f40b1380 2020-11-20T12:04:55.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.427+0000 7fd3eb7fe700 1 -- 172.21.15.187:0/2012476459 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0011610 con 0x7fd3f40b1380 2020-11-20T12:04:55.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.428+0000 7fd3f9c84700 1 -- 172.21.15.187:0/2012476459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f40b1380 msgr2=0x7fd3f4150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.428+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/2012476459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f40b1380 0x7fd3f4150ad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd3f40b85e0 tx=0x7fd3f0009290).stop 2020-11-20T12:04:55.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.428+0000 7fd3f9c84700 1 -- 172.21.15.187:0/2012476459 shutdown_connections 2020-11-20T12:04:55.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.428+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/2012476459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f40b1380 0x7fd3f4150ad0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.428+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/2012476459 >> v2:172.21.15.5:3300/0 conn(0x7fd3f40b1760 0x7fd3f4065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.428+0000 7fd3f9c84700 1 -- 172.21.15.187:0/2012476459 >> 172.21.15.187:0/2012476459 conn(0x7fd3f40bab30 msgr2=0x7fd3f40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:55.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.428+0000 7fd3f9c84700 1 -- 172.21.15.187:0/2012476459 shutdown_connections 2020-11-20T12:04:55.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.428+0000 7fd3f9c84700 1 -- 172.21.15.187:0/2012476459 wait complete. 2020-11-20T12:04:55.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.430+0000 7fd3f9c84700 1 Processor -- start 2020-11-20T12:04:55.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.430+0000 7fd3f9c84700 1 -- start start 2020-11-20T12:04:55.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.431+0000 7fd3f9c84700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3f40b1760 0x7fd3f40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.431+0000 7fd3f9c84700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3f409a670 0x7fd3f40b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.431+0000 7fd3f9c84700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f409dfc0 0x7fd3f40b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.431+0000 7fd3f9c84700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f40b7e70 con 0x7fd3f409a670 2020-11-20T12:04:55.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.431+0000 7fd3f9c84700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f409e540 con 0x7fd3f409dfc0 2020-11-20T12:04:55.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.431+0000 7fd3f9c84700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f409e7e0 con 0x7fd3f40b1760 2020-11-20T12:04:55.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f409dfc0 0x7fd3f40b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f409dfc0 0x7fd3f40b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40262/0 (socket says 172.21.15.187:40262) 2020-11-20T12:04:55.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 -- 172.21.15.187:0/4034122654 learned_addr learned my addr 172.21.15.187:0/4034122654 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:55.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3ebfff700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3f409a670 0x7fd3f40b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f8c82700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3f40b1760 0x7fd3f40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 -- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3f40b1760 msgr2=0x7fd3f40b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3f40b1760 0x7fd3f40b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 -- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3f409a670 msgr2=0x7fd3f40b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3f409a670 0x7fd3f40b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 -- 172.21.15.187:0/4034122654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f409ea80 con 0x7fd3f409dfc0 2020-11-20T12:04:55.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f8c82700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3f40b1760 0x7fd3f40b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:55.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.432+0000 7fd3f9483700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f409dfc0 0x7fd3f40b7230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd3ec0096e0 tx=0x7fd3ec006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:55.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.433+0000 7fd3e8ff9700 1 -- 172.21.15.187:0/4034122654 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec00d560 con 0x7fd3f409dfc0 2020-11-20T12:04:55.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.433+0000 7fd3e8ff9700 1 -- 172.21.15.187:0/4034122654 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec00d6e0 con 0x7fd3f409dfc0 2020-11-20T12:04:55.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.433+0000 7fd3e8ff9700 1 -- 172.21.15.187:0/4034122654 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec00daa0 con 0x7fd3f409dfc0 2020-11-20T12:04:55.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.433+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f40b4860 con 0x7fd3f409dfc0 2020-11-20T12:04:55.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.433+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f40b78e0 con 0x7fd3f409dfc0 2020-11-20T12:04:55.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.434+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f4061a20 con 0x7fd3f409dfc0 2020-11-20T12:04:55.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.434+0000 7fd3e8ff9700 1 -- 172.21.15.187:0/4034122654 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd3ec00f030 con 0x7fd3f409dfc0 2020-11-20T12:04:55.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.435+0000 7fd3e8ff9700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3cc056d20 0x7fd3cc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.435+0000 7fd3e8ff9700 1 -- 172.21.15.187:0/4034122654 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd3ec06fa70 con 0x7fd3f409dfc0 2020-11-20T12:04:55.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.435+0000 7fd3f8c82700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3cc056d20 0x7fd3cc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.436+0000 7fd3f8c82700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3cc056d20 0x7fd3cc0591e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd3f40632e0 tx=0x7fd3f0016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:55.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.439+0000 7fd3e8ff9700 1 -- 172.21.15.187:0/4034122654 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd3ec045170 con 0x7fd3f409dfc0 2020-11-20T12:04:55.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:55.625+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fd3f40b4a50 con 0x7fd3f409dfc0 2020-11-20T12:04:56.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.317+0000 7fd3e8ff9700 1 -- 172.21.15.187:0/4034122654 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v20) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd3ec048790 con 0x7fd3f409dfc0 2020-11-20T12:04:56.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.320+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3cc056d20 msgr2=0x7fd3cc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.320+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3cc056d20 0x7fd3cc0591e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd3f40632e0 tx=0x7fd3f0016000).stop 2020-11-20T12:04:56.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.320+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f409dfc0 msgr2=0x7fd3f40b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.320+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f409dfc0 0x7fd3f40b7230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd3ec0096e0 tx=0x7fd3ec006d70).stop 2020-11-20T12:04:56.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.321+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 shutdown_connections 2020-11-20T12:04:56.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.321+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3cc056d20 0x7fd3cc0591e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.321+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3f409dfc0 0x7fd3f40b7230 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.321+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3f40b1760 0x7fd3f40b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.321+0000 7fd3f9c84700 1 --2- 172.21.15.187:0/4034122654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3f409a670 0x7fd3f40b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.321+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 >> 172.21.15.187:0/4034122654 conn(0x7fd3f40bab30 msgr2=0x7fd3f414f300 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:56.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.321+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 shutdown_connections 2020-11-20T12:04:56.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.322+0000 7fd3f9c84700 1 -- 172.21.15.187:0/4034122654 wait complete. 2020-11-20T12:04:56.330 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:04:56.357 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:04:56.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.598+0000 7fb760868700 1 Processor -- start 2020-11-20T12:04:56.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.598+0000 7fb760868700 1 -- start start 2020-11-20T12:04:56.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.599+0000 7fb760868700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c063d40 0x7fb75c067e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.599+0000 7fb760868700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb75c1449c0 0x7fb75c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.599+0000 7fb760868700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c148670 con 0x7fb75c066920 2020-11-20T12:04:56.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.599+0000 7fb760868700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb75c0b15a0 con 0x7fb75c1449c0 2020-11-20T12:04:56.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.599+0000 7fb760868700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c148910 con 0x7fb75c063d40 2020-11-20T12:04:56.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.600+0000 7fb75b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb75c1449c0 0x7fb75c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.600+0000 7fb75259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c063d40 0x7fb75c067e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.600+0000 7fb75b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb75c1449c0 0x7fb75c144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:60766/0 (socket says 172.21.15.187:60766) 2020-11-20T12:04:56.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.600+0000 7fb75b59e700 1 -- 172.21.15.187:0/676132801 learned_addr learned my addr 172.21.15.187:0/676132801 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:56.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.601+0000 7fb75b59e700 1 -- 172.21.15.187:0/676132801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c063d40 msgr2=0x7fb75c067e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:56.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.601+0000 7fb75b59e700 1 --2- 172.21.15.187:0/676132801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c063d40 0x7fb75c067e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.601+0000 7fb75b59e700 1 -- 172.21.15.187:0/676132801 >> v1:172.21.15.5:6789/0 conn(0x7fb75c066920 legacy=0x7fb75c147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.601+0000 7fb75b59e700 1 -- 172.21.15.187:0/676132801 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c148bb0 con 0x7fb75c1449c0 2020-11-20T12:04:56.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.602+0000 7fb75b59e700 1 --2- 172.21.15.187:0/676132801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c1449c0 0x7fb75c144e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb754009d80 tx=0x7fb754009290).ready entity=mon.0 client_cookie=a4f6e6888759335c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.602+0000 7fb75a59c700 1 -- 172.21.15.187:0/676132801 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75400bc10 con 0x7fb75c1449c0 2020-11-20T12:04:56.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.602+0000 7fb75a59c700 1 -- 172.21.15.187:0/676132801 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75400bd70 con 0x7fb75c1449c0 2020-11-20T12:04:56.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.602+0000 7fb75a59c700 1 -- 172.21.15.187:0/676132801 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb754011610 con 0x7fb75c1449c0 2020-11-20T12:04:56.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.603+0000 7fb760868700 1 -- 172.21.15.187:0/676132801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c1449c0 msgr2=0x7fb75c144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.603+0000 7fb760868700 1 --2- 172.21.15.187:0/676132801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c1449c0 0x7fb75c144e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb754009d80 tx=0x7fb754009290).stop 2020-11-20T12:04:56.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.603+0000 7fb760868700 1 -- 172.21.15.187:0/676132801 shutdown_connections 2020-11-20T12:04:56.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.603+0000 7fb760868700 1 --2- 172.21.15.187:0/676132801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c1449c0 0x7fb75c144e30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.603+0000 7fb760868700 1 --2- 172.21.15.187:0/676132801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c063d40 0x7fb75c067e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.603+0000 7fb760868700 1 -- 172.21.15.187:0/676132801 >> 172.21.15.187:0/676132801 conn(0x7fb75c0baac0 msgr2=0x7fb75c0baed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:56.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.603+0000 7fb760868700 1 -- 172.21.15.187:0/676132801 shutdown_connections 2020-11-20T12:04:56.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.603+0000 7fb760868700 1 -- 172.21.15.187:0/676132801 wait complete. 2020-11-20T12:04:56.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.605+0000 7fb760868700 1 Processor -- start 2020-11-20T12:04:56.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.605+0000 7fb760868700 1 -- start start 2020-11-20T12:04:56.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.606+0000 7fb760868700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c063d40 0x7fb75c0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.606+0000 7fb760868700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb75c066920 0x7fb75c143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.606+0000 7fb760868700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c1449c0 0x7fb75c13afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.606+0000 7fb760868700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c0b1be0 con 0x7fb75c063d40 2020-11-20T12:04:56.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.606+0000 7fb760868700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c0b4400 con 0x7fb75c066920 2020-11-20T12:04:56.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.606+0000 7fb760868700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c0b46a0 con 0x7fb75c1449c0 2020-11-20T12:04:56.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb75c066920 0x7fb75c143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb75c066920 0x7fb75c143f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40274/0 (socket says 172.21.15.187:40274) 2020-11-20T12:04:56.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 -- 172.21.15.187:0/3246486127 learned_addr learned my addr 172.21.15.187:0/3246486127 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:56.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75ad9d700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c063d40 0x7fb75c0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75b59e700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c1449c0 0x7fb75c13afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 -- 172.21.15.187:0/3246486127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c1449c0 msgr2=0x7fb75c13afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c1449c0 0x7fb75c13afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 -- 172.21.15.187:0/3246486127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c063d40 msgr2=0x7fb75c0b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c063d40 0x7fb75c0b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 -- 172.21.15.187:0/3246486127 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c0b40c0 con 0x7fb75c066920 2020-11-20T12:04:56.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.607+0000 7fb75259c700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb75c066920 0x7fb75c143f60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fb74400ebd0 tx=0x7fb744006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.608+0000 7fb7537fe700 1 -- 172.21.15.187:0/3246486127 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74400ca40 con 0x7fb75c066920 2020-11-20T12:04:56.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.608+0000 7fb7537fe700 1 -- 172.21.15.187:0/3246486127 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74400cba0 con 0x7fb75c066920 2020-11-20T12:04:56.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.608+0000 7fb7537fe700 1 -- 172.21.15.187:0/3246486127 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440123f0 con 0x7fb75c066920 2020-11-20T12:04:56.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.608+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c144450 con 0x7fb75c066920 2020-11-20T12:04:56.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.608+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c09f3d0 con 0x7fb75c066920 2020-11-20T12:04:56.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.609+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75c0636b0 con 0x7fb75c066920 2020-11-20T12:04:56.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.609+0000 7fb7537fe700 1 -- 172.21.15.187:0/3246486127 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb744012760 con 0x7fb75c066920 2020-11-20T12:04:56.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.610+0000 7fb7537fe700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb738056d20 0x7fb7380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.610+0000 7fb7537fe700 1 -- 172.21.15.187:0/3246486127 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fb744070390 con 0x7fb75c066920 2020-11-20T12:04:56.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.610+0000 7fb75ad9d700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb738056d20 0x7fb7380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.611+0000 7fb75ad9d700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb738056d20 0x7fb7380591e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb74c000bc0 tx=0x7fb74c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.614+0000 7fb7537fe700 1 -- 172.21.15.187:0/3246486127 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb7440458a0 con 0x7fb75c066920 2020-11-20T12:04:56.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:56.796+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fb75c068310 con 0x7fb75c066920 2020-11-20T12:04:57.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.327+0000 7fb7537fe700 1 -- 172.21.15.187:0/3246486127 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v21) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb744048ec0 con 0x7fb75c066920 2020-11-20T12:04:57.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb738056d20 msgr2=0x7fb7380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb738056d20 0x7fb7380591e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb74c000bc0 tx=0x7fb74c009040).stop 2020-11-20T12:04:57.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb75c066920 msgr2=0x7fb75c143f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb75c066920 0x7fb75c143f60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fb74400ebd0 tx=0x7fb744006df0).stop 2020-11-20T12:04:57.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 shutdown_connections 2020-11-20T12:04:57.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb738056d20 0x7fb7380591e0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0x7fb74c000bc0 tx=0x7fb74c009040).stop 2020-11-20T12:04:57.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb75c063d40 0x7fb75c0b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb75c1449c0 0x7fb75c13afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 --2- 172.21.15.187:0/3246486127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb75c066920 0x7fb75c143f60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.332+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 >> 172.21.15.187:0/3246486127 conn(0x7fb75c0baac0 msgr2=0x7fb75c0b6170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.333+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 shutdown_connections 2020-11-20T12:04:57.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.333+0000 7fb760868700 1 -- 172.21.15.187:0/3246486127 wait complete. 2020-11-20T12:04:57.341 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:04:57.365 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:04:57.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.599+0000 7f33cc435700 1 Processor -- start 2020-11-20T12:04:57.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.599+0000 7f33cc435700 1 -- start start 2020-11-20T12:04:57.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33cc435700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f33c40b5e50 0x7f33c41464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33cc435700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33c40bc530 0x7f33c40bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33cc435700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c40b83a0 con 0x7f33c4146110 2020-11-20T12:04:57.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33cc435700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c414a680 con 0x7f33c40b5e50 2020-11-20T12:04:57.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33cc435700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c40b8640 con 0x7f33c40bc530 2020-11-20T12:04:57.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33ca9d2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33c40bc530 0x7f33c40bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33ca9d2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33c40bc530 0x7f33c40bc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40284/0 (socket says 172.21.15.187:40284) 2020-11-20T12:04:57.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33ca9d2700 1 -- 172.21.15.187:0/3799046649 learned_addr learned my addr 172.21.15.187:0/3799046649 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:57.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.601+0000 7f33c99d0700 1 --2- 172.21.15.187:0/3799046649 >> v2:172.21.15.5:3300/0 conn(0x7f33c40b5e50 0x7f33c41464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.602+0000 7f33ca9d2700 1 -- 172.21.15.187:0/3799046649 >> v2:172.21.15.5:3300/0 conn(0x7f33c40b5e50 msgr2=0x7f33c41464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:57.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.602+0000 7f33ca9d2700 1 --2- 172.21.15.187:0/3799046649 >> v2:172.21.15.5:3300/0 conn(0x7f33c40b5e50 0x7f33c41464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.602+0000 7f33ca9d2700 1 -- 172.21.15.187:0/3799046649 >> v1:172.21.15.5:6789/0 conn(0x7f33c4146110 legacy=0x7f33c4149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:57.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.602+0000 7f33ca9d2700 1 -- 172.21.15.187:0/3799046649 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c40b8090 con 0x7f33c40bc530 2020-11-20T12:04:57.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.602+0000 7f33ca9d2700 1 --2- 172.21.15.187:0/3799046649 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40bc530 0x7f33c40bc9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f33c4066100 tx=0x7f33bc009290).ready entity=mon.1 client_cookie=e2922d9f4b9d4780 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.602+0000 7f33c91cf700 1 -- 172.21.15.187:0/3799046649 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc00bb80 con 0x7f33c40bc530 2020-11-20T12:04:57.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33c91cf700 1 -- 172.21.15.187:0/3799046649 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc00bce0 con 0x7f33c40bc530 2020-11-20T12:04:57.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33c91cf700 1 -- 172.21.15.187:0/3799046649 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011610 con 0x7f33c40bc530 2020-11-20T12:04:57.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33cc435700 1 -- 172.21.15.187:0/3799046649 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40bc530 msgr2=0x7f33c40bc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33cc435700 1 --2- 172.21.15.187:0/3799046649 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40bc530 0x7f33c40bc9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f33c4066100 tx=0x7f33bc009290).stop 2020-11-20T12:04:57.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33cc435700 1 -- 172.21.15.187:0/3799046649 shutdown_connections 2020-11-20T12:04:57.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33cc435700 1 --2- 172.21.15.187:0/3799046649 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40bc530 0x7f33c40bc9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33cc435700 1 --2- 172.21.15.187:0/3799046649 >> v2:172.21.15.5:3300/0 conn(0x7f33c40b5e50 0x7f33c41464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33cc435700 1 -- 172.21.15.187:0/3799046649 >> 172.21.15.187:0/3799046649 conn(0x7f33c4061cf0 msgr2=0x7f33c40bbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33cc435700 1 -- 172.21.15.187:0/3799046649 shutdown_connections 2020-11-20T12:04:57.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.603+0000 7f33cc435700 1 -- 172.21.15.187:0/3799046649 wait complete. 2020-11-20T12:04:57.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.605+0000 7f33cc435700 1 Processor -- start 2020-11-20T12:04:57.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.605+0000 7f33cc435700 1 -- start start 2020-11-20T12:04:57.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33cc435700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40b5e50 0x7f33c409e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33cc435700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33c40bc530 0x7f33c4156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33cc435700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33c4146110 0x7f33c409d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33cc435700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c413f1b0 con 0x7f33c40bc530 2020-11-20T12:04:57.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33cc435700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c40b1fd0 con 0x7f33c40b5e50 2020-11-20T12:04:57.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33cc435700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c40b2200 con 0x7f33c4146110 2020-11-20T12:04:57.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca1d1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40b5e50 0x7f33c409e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca1d1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40b5e50 0x7f33c409e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40286/0 (socket says 172.21.15.187:40286) 2020-11-20T12:04:57.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca1d1700 1 -- 172.21.15.187:0/995172084 learned_addr learned my addr 172.21.15.187:0/995172084 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:57.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33c99d0700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33c40bc530 0x7f33c4156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca1d1700 1 -- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33c4146110 msgr2=0x7f33c409d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca9d2700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33c4146110 0x7f33c409d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca1d1700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33c4146110 0x7f33c409d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca1d1700 1 -- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33c40bc530 msgr2=0x7f33c4156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca1d1700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33c40bc530 0x7f33c4156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.606+0000 7f33ca1d1700 1 -- 172.21.15.187:0/995172084 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c40b24a0 con 0x7f33c40b5e50 2020-11-20T12:04:57.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.607+0000 7f33c99d0700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33c40bc530 0x7f33c4156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:57.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.607+0000 7f33ca1d1700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40b5e50 0x7f33c409e8d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f33c00096e0 tx=0x7f33c0006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.607+0000 7f33b27fc700 1 -- 172.21.15.187:0/995172084 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c000d560 con 0x7f33c40b5e50 2020-11-20T12:04:57.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.607+0000 7f33b27fc700 1 -- 172.21.15.187:0/995172084 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c0000e90 con 0x7f33c40b5e50 2020-11-20T12:04:57.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.607+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c4156970 con 0x7f33c40b5e50 2020-11-20T12:04:57.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.607+0000 7f33b27fc700 1 -- 172.21.15.187:0/995172084 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c000d930 con 0x7f33c40b5e50 2020-11-20T12:04:57.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.607+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c40b4a50 con 0x7f33c40b5e50 2020-11-20T12:04:57.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.608+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c40b4d30 con 0x7f33c40b5e50 2020-11-20T12:04:57.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.609+0000 7f33b27fc700 1 -- 172.21.15.187:0/995172084 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f33c000f030 con 0x7f33c40b5e50 2020-11-20T12:04:57.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.609+0000 7f33b27fc700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f339c056d50 0x7f339c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.610+0000 7f33b27fc700 1 -- 172.21.15.187:0/995172084 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f33c006fbe0 con 0x7f33c40b5e50 2020-11-20T12:04:57.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.610+0000 7f33c99d0700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f339c056d50 0x7f339c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.610+0000 7f33c99d0700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f339c056d50 0x7f339c059210 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f33b4006bd0 tx=0x7f33b4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.613+0000 7f33b27fc700 1 -- 172.21.15.187:0/995172084 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f33c40b4d30 con 0x7f33c40b5e50 2020-11-20T12:04:57.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.864+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f33c40b4d30 con 0x7f33c40b5e50 2020-11-20T12:04:57.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.879+0000 7f33b27fc700 1 -- 172.21.15.187:0/995172084 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 3 and data pool 4 v3) v1 ==== 187+0+0 (secure 0 0 0) 0x7f33c40b4d30 con 0x7f33c40b5e50 2020-11-20T12:04:57.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f339c056d50 msgr2=0x7f339c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f339c056d50 0x7f339c059210 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f33b4006bd0 tx=0x7f33b4008040).stop 2020-11-20T12:04:57.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40b5e50 msgr2=0x7f33c409e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40b5e50 0x7f33c409e8d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f33c00096e0 tx=0x7f33c0006d70).stop 2020-11-20T12:04:57.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 shutdown_connections 2020-11-20T12:04:57.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f339c056d50 0x7f339c059210 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0x7f33b4006bd0 tx=0x7f33b4008040).stop 2020-11-20T12:04:57.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33c4146110 0x7f33c409d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33c40b5e50 0x7f33c409e8d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 --2- 172.21.15.187:0/995172084 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33c40bc530 0x7f33c4156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.883+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 >> 172.21.15.187:0/995172084 conn(0x7f33c4061cf0 msgr2=0x7f33c4148ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.884+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 shutdown_connections 2020-11-20T12:04:57.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:57.884+0000 7f33cc435700 1 -- 172.21.15.187:0/995172084 wait complete. 2020-11-20T12:04:57.891 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 3 and data pool 4 2020-11-20T12:04:57.916 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:58.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.122+0000 7fdddf310700 1 Processor -- start 2020-11-20T12:04:58.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.123+0000 7fdddf310700 1 -- start start 2020-11-20T12:04:58.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.124+0000 7fdddf310700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddd8065190 0x7fddd8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.124+0000 7fdddf310700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fddd809ca90 0x7fddd809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.124+0000 7fdddf310700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd80b13e0 con 0x7fddd8065f90 2020-11-20T12:04:58.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.124+0000 7fdddf310700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd814ab30 con 0x7fddd809ca90 2020-11-20T12:04:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.124+0000 7fdddf310700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd8062c10 con 0x7fddd8065190 2020-11-20T12:04:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.124+0000 7fddddb0d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fddd8065f90 0x7fddd8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36914/0 (socket says 172.21.15.187:36914) 2020-11-20T12:04:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.124+0000 7fddddb0d700 1 -- 172.21.15.187:0/2519394633 learned_addr learned my addr 172.21.15.187:0/2519394633 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:04:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.124+0000 7fddde30e700 1 --2- 172.21.15.187:0/2519394633 >> v2:172.21.15.187:3300/0 conn(0x7fddd8065190 0x7fddd8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.125+0000 7fdddeb0f700 1 --2- 172.21.15.187:0/2519394633 >> v2:172.21.15.5:3300/0 conn(0x7fddd809ca90 0x7fddd809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.125+0000 7fdddd30c700 1 -- 172.21.15.187:0/2519394633 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149288693 0 0) 0x7fddd80b13e0 con 0x7fddd8065f90 2020-11-20T12:04:58.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.125+0000 7fdddd30c700 1 -- 172.21.15.187:0/2519394633 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc0003240 con 0x7fddd8065f90 2020-11-20T12:04:58.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.125+0000 7fddde30e700 1 -- 172.21.15.187:0/2519394633 >> v2:172.21.15.5:3300/0 conn(0x7fddd809ca90 msgr2=0x7fddd809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:58.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.125+0000 7fddde30e700 1 --2- 172.21.15.187:0/2519394633 >> v2:172.21.15.5:3300/0 conn(0x7fddd809ca90 0x7fddd809cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.125+0000 7fddde30e700 1 -- 172.21.15.187:0/2519394633 >> v1:172.21.15.187:6789/0 conn(0x7fddd8065f90 legacy=0x7fddd8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.125+0000 7fddde30e700 1 -- 172.21.15.187:0/2519394633 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd80b1640 con 0x7fddd8065190 2020-11-20T12:04:58.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.125+0000 7fddde30e700 1 --2- 172.21.15.187:0/2519394633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065190 0x7fddd8065570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fddd805c8b0 tx=0x7fddd40091d0).ready entity=mon.1 client_cookie=624dc25634eddbf0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddd30c700 1 -- 172.21.15.187:0/2519394633 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd400bad0 con 0x7fddd8065190 2020-11-20T12:04:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddd30c700 1 -- 172.21.15.187:0/2519394633 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd400bc30 con 0x7fddd8065190 2020-11-20T12:04:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddd30c700 1 -- 172.21.15.187:0/2519394633 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4011400 con 0x7fddd8065190 2020-11-20T12:04:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddf310700 1 -- 172.21.15.187:0/2519394633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065190 msgr2=0x7fddd8065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddf310700 1 --2- 172.21.15.187:0/2519394633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065190 0x7fddd8065570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fddd805c8b0 tx=0x7fddd40091d0).stop 2020-11-20T12:04:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddf310700 1 -- 172.21.15.187:0/2519394633 shutdown_connections 2020-11-20T12:04:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddf310700 1 --2- 172.21.15.187:0/2519394633 >> v2:172.21.15.5:3300/0 conn(0x7fddd809ca90 0x7fddd809cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddf310700 1 --2- 172.21.15.187:0/2519394633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065190 0x7fddd8065570 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddf310700 1 -- 172.21.15.187:0/2519394633 >> 172.21.15.187:0/2519394633 conn(0x7fddd80bab30 msgr2=0x7fddd80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddf310700 1 -- 172.21.15.187:0/2519394633 shutdown_connections 2020-11-20T12:04:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.126+0000 7fdddf310700 1 -- 172.21.15.187:0/2519394633 wait complete. 2020-11-20T12:04:58.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.128+0000 7fdddf310700 1 Processor -- start 2020-11-20T12:04:58.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.128+0000 7fdddf310700 1 -- start start 2020-11-20T12:04:58.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.129+0000 7fdddf310700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddd8065190 0x7fddd8149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.129+0000 7fdddf310700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065f90 0x7fddd809f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.129+0000 7fdddf310700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddd809ca90 0x7fddd80b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.129+0000 7fdddf310700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd8148780 con 0x7fddd8065190 2020-11-20T12:04:58.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.129+0000 7fdddf310700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd8148a20 con 0x7fddd8065f90 2020-11-20T12:04:58.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.129+0000 7fdddf310700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd8148cc0 con 0x7fddd809ca90 2020-11-20T12:04:58.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065f90 0x7fddd809f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065f90 0x7fddd809f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40314/0 (socket says 172.21.15.187:40314) 2020-11-20T12:04:58.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 -- 172.21.15.187:0/1821816191 learned_addr learned my addr 172.21.15.187:0/1821816191 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:58.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 -- 172.21.15.187:0/1821816191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddd809ca90 msgr2=0x7fddd80b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddd809ca90 0x7fddd80b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:58.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 -- 172.21.15.187:0/1821816191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddd8065190 msgr2=0x7fddd8149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddd8065190 0x7fddd8149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:58.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 -- 172.21.15.187:0/1821816191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8146c20 con 0x7fddd8065f90 2020-11-20T12:04:58.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddddb0d700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065f90 0x7fddd809f390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fddc800c710 tx=0x7fddc8007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.130+0000 7fddc67fc700 1 -- 172.21.15.187:0/1821816191 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8007320 con 0x7fddd8065f90 2020-11-20T12:04:58.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.131+0000 7fddc67fc700 1 -- 172.21.15.187:0/1821816191 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc8007480 con 0x7fddd8065f90 2020-11-20T12:04:58.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.131+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd809f880 con 0x7fddd8065f90 2020-11-20T12:04:58.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.131+0000 7fddc67fc700 1 -- 172.21.15.187:0/1821816191 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8005800 con 0x7fddd8065f90 2020-11-20T12:04:58.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.131+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd80b3b10 con 0x7fddd8065f90 2020-11-20T12:04:58.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.132+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd81485d0 con 0x7fddd8065f90 2020-11-20T12:04:58.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.132+0000 7fddc67fc700 1 -- 172.21.15.187:0/1821816191 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fddc8010030 con 0x7fddd8065f90 2020-11-20T12:04:58.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.133+0000 7fddc67fc700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddb0056d20 0x7fddb00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.133+0000 7fddc67fc700 1 -- 172.21.15.187:0/1821816191 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fddc806fd00 con 0x7fddd8065f90 2020-11-20T12:04:58.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.133+0000 7fddde30e700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddb0056d20 0x7fddb00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.134+0000 7fddde30e700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddb0056d20 0x7fddb00591e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fddd4009210 tx=0x7fddd4016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.137+0000 7fddc67fc700 1 -- 172.21.15.187:0/1821816191 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fddc8073030 con 0x7fddd8065f90 2020-11-20T12:04:58.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.322+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddd80b9d60 con 0x7fddd8065f90 2020-11-20T12:04:58.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.325+0000 7fddc67fc700 1 -- 172.21.15.187:0/1821816191 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16791 (secure 0 0 0) 0x7fddc8045150 con 0x7fddd8065f90 2020-11-20T12:04:58.330 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:04:58.331 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":22,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:57.867867+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:58.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddb0056d20 msgr2=0x7fddb00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddb0056d20 0x7fddb00591e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fddd4009210 tx=0x7fddd4016000).stop 2020-11-20T12:04:58.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065f90 msgr2=0x7fddd809f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065f90 0x7fddd809f390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fddc800c710 tx=0x7fddc8007aa0).stop 2020-11-20T12:04:58.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 shutdown_connections 2020-11-20T12:04:58.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddb0056d20 0x7fddb00591e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddd8065190 0x7fddd8149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddd809ca90 0x7fddd80b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 --2- 172.21.15.187:0/1821816191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd8065f90 0x7fddd809f390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 >> 172.21.15.187:0/1821816191 conn(0x7fddd80bab30 msgr2=0x7fddd80aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 shutdown_connections 2020-11-20T12:04:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.328+0000 7fdddf310700 1 -- 172.21.15.187:0/1821816191 wait complete. 2020-11-20T12:04:58.356 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:58.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.614+0000 7fd12490c700 1 Processor -- start 2020-11-20T12:04:58.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.614+0000 7fd12490c700 1 -- start start 2020-11-20T12:04:58.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.615+0000 7fd12490c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd120065f90 0x7fd120065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.615+0000 7fd12490c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd12009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.616+0000 7fd12490c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1200b13e0 con 0x7fd120065190 2020-11-20T12:04:58.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.616+0000 7fd12490c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12014ab30 con 0x7fd120065f90 2020-11-20T12:04:58.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.616+0000 7fd12490c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd120062c10 con 0x7fd12009ca90 2020-11-20T12:04:58.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.616+0000 7fd11f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd120065190 0x7fd120143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36926/0 (socket says 172.21.15.187:36926) 2020-11-20T12:04:58.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.616+0000 7fd11f7fe700 1 -- 172.21.15.187:0/2219048054 learned_addr learned my addr 172.21.15.187:0/2219048054 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:04:58.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.616+0000 7fd11ffff700 1 --2- 172.21.15.187:0/2219048054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd12009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.616+0000 7fd116ffd700 1 --2- 172.21.15.187:0/2219048054 >> v2:172.21.15.5:3300/0 conn(0x7fd120065f90 0x7fd120065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.617+0000 7fd11effd700 1 -- 172.21.15.187:0/2219048054 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336853127 0 0) 0x7fd1200b13e0 con 0x7fd120065190 2020-11-20T12:04:58.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.617+0000 7fd11effd700 1 -- 172.21.15.187:0/2219048054 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd100003240 con 0x7fd120065190 2020-11-20T12:04:58.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.617+0000 7fd11ffff700 1 -- 172.21.15.187:0/2219048054 >> v2:172.21.15.5:3300/0 conn(0x7fd120065f90 msgr2=0x7fd120065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:58.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.617+0000 7fd11ffff700 1 --2- 172.21.15.187:0/2219048054 >> v2:172.21.15.5:3300/0 conn(0x7fd120065f90 0x7fd120065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.617+0000 7fd11ffff700 1 -- 172.21.15.187:0/2219048054 >> v1:172.21.15.187:6789/0 conn(0x7fd120065190 legacy=0x7fd120143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.617+0000 7fd11ffff700 1 -- 172.21.15.187:0/2219048054 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1200b1640 con 0x7fd12009ca90 2020-11-20T12:04:58.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.618+0000 7fd11ffff700 1 --2- 172.21.15.187:0/2219048054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd12009cf40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fd118009c90 tx=0x7fd1180091d0).ready entity=mon.2 client_cookie=6fcc26a648cc1a8d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.618+0000 7fd11effd700 1 -- 172.21.15.187:0/2219048054 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11800ba50 con 0x7fd12009ca90 2020-11-20T12:04:58.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.618+0000 7fd11effd700 1 -- 172.21.15.187:0/2219048054 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11800bbb0 con 0x7fd12009ca90 2020-11-20T12:04:58.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.618+0000 7fd11effd700 1 -- 172.21.15.187:0/2219048054 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118011400 con 0x7fd12009ca90 2020-11-20T12:04:58.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.619+0000 7fd12490c700 1 -- 172.21.15.187:0/2219048054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 msgr2=0x7fd12009cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.619+0000 7fd12490c700 1 --2- 172.21.15.187:0/2219048054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd12009cf40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fd118009c90 tx=0x7fd1180091d0).stop 2020-11-20T12:04:58.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.619+0000 7fd12490c700 1 -- 172.21.15.187:0/2219048054 shutdown_connections 2020-11-20T12:04:58.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.619+0000 7fd12490c700 1 --2- 172.21.15.187:0/2219048054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd12009cf40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.619+0000 7fd12490c700 1 --2- 172.21.15.187:0/2219048054 >> v2:172.21.15.5:3300/0 conn(0x7fd120065f90 0x7fd120065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.619+0000 7fd12490c700 1 -- 172.21.15.187:0/2219048054 >> 172.21.15.187:0/2219048054 conn(0x7fd1200bab30 msgr2=0x7fd1200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.619+0000 7fd12490c700 1 -- 172.21.15.187:0/2219048054 shutdown_connections 2020-11-20T12:04:58.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.619+0000 7fd12490c700 1 -- 172.21.15.187:0/2219048054 wait complete. 2020-11-20T12:04:58.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.621+0000 7fd12490c700 1 Processor -- start 2020-11-20T12:04:58.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.621+0000 7fd12490c700 1 -- start start 2020-11-20T12:04:58.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd12490c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd120065190 0x7fd12009f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd12490c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd120065f90 0x7fd12009f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd12490c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd120145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd12490c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12014acc0 con 0x7fd120065f90 2020-11-20T12:04:58.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd12490c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12009fe60 con 0x7fd120065190 2020-11-20T12:04:58.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd12490c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1200a00b0 con 0x7fd12009ca90 2020-11-20T12:04:58.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd11f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd120065190 0x7fd12009f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd11f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd120065190 0x7fd12009f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40328/0 (socket says 172.21.15.187:40328) 2020-11-20T12:04:58.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd11f7fe700 1 -- 172.21.15.187:0/2690535605 learned_addr learned my addr 172.21.15.187:0/2690535605 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:58.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd11ffff700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd120145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.622+0000 7fd116ffd700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd120065f90 0x7fd12009f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd11f7fe700 1 -- 172.21.15.187:0/2690535605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 msgr2=0x7fd120145140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd11f7fe700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd120145140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd11f7fe700 1 -- 172.21.15.187:0/2690535605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd120065f90 msgr2=0x7fd12009f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd11f7fe700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd120065f90 0x7fd12009f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd11f7fe700 1 -- 172.21.15.187:0/2690535605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1200a0350 con 0x7fd120065190 2020-11-20T12:04:58.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd11f7fe700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd120065190 0x7fd12009f440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fd11000c6f0 tx=0x7fd110007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd117fff700 1 -- 172.21.15.187:0/2690535605 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110007320 con 0x7fd120065190 2020-11-20T12:04:58.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd117fff700 1 -- 172.21.15.187:0/2690535605 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd110007480 con 0x7fd120065190 2020-11-20T12:04:58.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd117fff700 1 -- 172.21.15.187:0/2690535605 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110005800 con 0x7fd120065190 2020-11-20T12:04:58.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1200a05f0 con 0x7fd120065190 2020-11-20T12:04:58.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.623+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd120142f70 con 0x7fd120065190 2020-11-20T12:04:58.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.625+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd120061a20 con 0x7fd120065190 2020-11-20T12:04:58.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.625+0000 7fd117fff700 1 -- 172.21.15.187:0/2690535605 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd110010030 con 0x7fd120065190 2020-11-20T12:04:58.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.626+0000 7fd117fff700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0fc056d50 0x7fd0fc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.626+0000 7fd117fff700 1 -- 172.21.15.187:0/2690535605 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd11006fc50 con 0x7fd120065190 2020-11-20T12:04:58.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.626+0000 7fd116ffd700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0fc056d50 0x7fd0fc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.627+0000 7fd116ffd700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0fc056d50 0x7fd0fc059210 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd120067630 tx=0x7fd108008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.630+0000 7fd117fff700 1 -- 172.21.15.187:0/2690535605 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd110045040 con 0x7fd120065190 2020-11-20T12:04:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.815+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1200afc60 con 0x7fd120065190 2020-11-20T12:04:58.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.818+0000 7fd117fff700 1 -- 172.21.15.187:0/2690535605 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16791 (secure 0 0 0) 0x7fd110048660 con 0x7fd120065190 2020-11-20T12:04:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:04:58.823 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":22,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:57.867867+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:58.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0fc056d50 msgr2=0x7fd0fc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0fc056d50 0x7fd0fc059210 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd120067630 tx=0x7fd108008040).stop 2020-11-20T12:04:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd120065190 msgr2=0x7fd12009f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd120065190 0x7fd12009f440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fd11000c6f0 tx=0x7fd110007ad0).stop 2020-11-20T12:04:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 shutdown_connections 2020-11-20T12:04:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0fc056d50 0x7fd0fc059210 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd12009ca90 0x7fd120145140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd120065190 0x7fd12009f440 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 --2- 172.21.15.187:0/2690535605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd120065f90 0x7fd12009f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 >> 172.21.15.187:0/2690535605 conn(0x7fd1200bab30 msgr2=0x7fd1200b9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 shutdown_connections 2020-11-20T12:04:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:58.820+0000 7fd12490c700 1 -- 172.21.15.187:0/2690535605 wait complete. 2020-11-20T12:04:58.847 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:04:59.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.084+0000 7f22708be700 1 Processor -- start 2020-11-20T12:04:59.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.085+0000 7f22708be700 1 -- start start 2020-11-20T12:04:59.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.086+0000 7f22708be700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f226c066f70 0x7f226c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.086+0000 7f22708be700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c0b1300 con 0x7f226c1484d0 2020-11-20T12:04:59.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.086+0000 7f22708be700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c0b15a0 con 0x7f226c0b6650 2020-11-20T12:04:59.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.086+0000 7f22708be700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226c067730 con 0x7f226c066f70 2020-11-20T12:04:59.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.086+0000 7f2263fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f226c0b6650 0x7f226c0afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36940/0 (socket says 172.21.15.187:36940) 2020-11-20T12:04:59.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.086+0000 7f226bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f226c066f70 0x7f226c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.086+0000 7f2263fff700 1 -- 172.21.15.187:0/3478224695 learned_addr learned my addr 172.21.15.187:0/3478224695 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:04:59.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.087+0000 7f226affd700 1 -- 172.21.15.187:0/3478224695 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093385130 0 0) 0x7f226c0b15a0 con 0x7f226c0b6650 2020-11-20T12:04:59.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.087+0000 7f226affd700 1 -- 172.21.15.187:0/3478224695 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f224c003240 con 0x7f226c0b6650 2020-11-20T12:04:59.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.087+0000 7f226bfff700 1 -- 172.21.15.187:0/3478224695 >> v1:172.21.15.187:6789/0 conn(0x7f226c0b6650 legacy=0x7f226c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.087+0000 7f226bfff700 1 -- 172.21.15.187:0/3478224695 >> v1:172.21.15.5:6789/0 conn(0x7f226c1484d0 legacy=0x7f226c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:59.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.087+0000 7f226bfff700 1 -- 172.21.15.187:0/3478224695 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c0b1840 con 0x7f226c066f70 2020-11-20T12:04:59.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.087+0000 7f226bfff700 1 --2- 172.21.15.187:0/3478224695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c066f70 0x7f226c0b01a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2264009c90 tx=0x7f22640091d0).ready entity=mon.1 client_cookie=363929b1ecdcde5f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.088+0000 7f226affd700 1 -- 172.21.15.187:0/3478224695 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226400ba30 con 0x7f226c066f70 2020-11-20T12:04:59.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.088+0000 7f226affd700 1 -- 172.21.15.187:0/3478224695 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226400bb90 con 0x7f226c066f70 2020-11-20T12:04:59.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.088+0000 7f226affd700 1 -- 172.21.15.187:0/3478224695 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264011400 con 0x7f226c066f70 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.089+0000 7f22708be700 1 -- 172.21.15.187:0/3478224695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c066f70 msgr2=0x7f226c0b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.089+0000 7f22708be700 1 --2- 172.21.15.187:0/3478224695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c066f70 0x7f226c0b01a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2264009c90 tx=0x7f22640091d0).stop 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.089+0000 7f22708be700 1 -- 172.21.15.187:0/3478224695 shutdown_connections 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.089+0000 7f22708be700 1 --2- 172.21.15.187:0/3478224695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c066f70 0x7f226c0b01a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.089+0000 7f22708be700 1 -- 172.21.15.187:0/3478224695 >> 172.21.15.187:0/3478224695 conn(0x7f226c0bab30 msgr2=0x7f226c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.089+0000 7f22708be700 1 -- 172.21.15.187:0/3478224695 shutdown_connections 2020-11-20T12:04:59.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.089+0000 7f22708be700 1 -- 172.21.15.187:0/3478224695 wait complete. 2020-11-20T12:04:59.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.091+0000 7f22708be700 1 Processor -- start 2020-11-20T12:04:59.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.092+0000 7f22708be700 1 -- start start 2020-11-20T12:04:59.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f22708be700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f226c066f70 0x7f226c144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f22708be700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c0b6650 0x7f226c0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f22708be700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f226c1484d0 0x7f226c0b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f22708be700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0a0b10 con 0x7f226c1484d0 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f22708be700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0a0870 con 0x7f226c0b6650 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f22708be700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0b3900 con 0x7f226c066f70 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f2263fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c0b6650 0x7f226c0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f2263fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c0b6650 0x7f226c0b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40342/0 (socket says 172.21.15.187:40342) 2020-11-20T12:04:59.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.093+0000 7f2263fff700 1 -- 172.21.15.187:0/2191587881 learned_addr learned my addr 172.21.15.187:0/2191587881 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:04:59.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f2263fff700 1 -- 172.21.15.187:0/2191587881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f226c066f70 msgr2=0x7f226c144c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f2263fff700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f226c066f70 0x7f226c144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f2263fff700 1 -- 172.21.15.187:0/2191587881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f226c1484d0 msgr2=0x7f226c0b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f2263fff700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f226c1484d0 0x7f226c0b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f2263fff700 1 -- 172.21.15.187:0/2191587881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c145100 con 0x7f226c0b6650 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f2263fff700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c0b6650 0x7f226c0b1a20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f225400c6f0 tx=0x7f2254007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f22637fe700 1 -- 172.21.15.187:0/2191587881 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254007320 con 0x7f226c0b6650 2020-11-20T12:04:59.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f22637fe700 1 -- 172.21.15.187:0/2191587881 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2254007480 con 0x7f226c0b6650 2020-11-20T12:04:59.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c0b3da0 con 0x7f226c0b6650 2020-11-20T12:04:59.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f22637fe700 1 -- 172.21.15.187:0/2191587881 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254005800 con 0x7f226c0b6650 2020-11-20T12:04:59.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.094+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c141de0 con 0x7f226c0b6650 2020-11-20T12:04:59.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.095+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c0a06c0 con 0x7f226c0b6650 2020-11-20T12:04:59.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.096+0000 7f22637fe700 1 -- 172.21.15.187:0/2191587881 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2254010030 con 0x7f226c0b6650 2020-11-20T12:04:59.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.096+0000 7f22637fe700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2248056d20 0x7f22480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.097+0000 7f22637fe700 1 -- 172.21.15.187:0/2191587881 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f225406fcf0 con 0x7f226c0b6650 2020-11-20T12:04:59.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.097+0000 7f226b7fe700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2248056d20 0x7f22480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.097+0000 7f226b7fe700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2248056d20 0x7f22480591e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f225c006fd0 tx=0x7f225c006930).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.100+0000 7f22637fe700 1 -- 172.21.15.187:0/2191587881 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f22540450c0 con 0x7f226c0b6650 2020-11-20T12:04:59.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.353+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f226c0b3ac0 con 0x7f226c0b6650 2020-11-20T12:05:00.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:04:59.997+0000 7f22637fe700 1 -- 172.21.15.187:0/2191587881 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f22540486e0 con 0x7f226c0b6650 2020-11-20T12:05:00.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.000+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2248056d20 msgr2=0x7f22480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2248056d20 0x7f22480591e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f225c006fd0 tx=0x7f225c006930).stop 2020-11-20T12:05:00.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c0b6650 msgr2=0x7f226c0b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c0b6650 0x7f226c0b1a20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f225400c6f0 tx=0x7f2254007ad0).stop 2020-11-20T12:05:00.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 shutdown_connections 2020-11-20T12:05:00.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2248056d20 0x7f22480591e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f226c1484d0 0x7f226c0b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:00.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f226c066f70 0x7f226c144c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:00.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 --2- 172.21.15.187:0/2191587881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f226c0b6650 0x7f226c0b1a20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 >> 172.21.15.187:0/2191587881 conn(0x7f226c0bab30 msgr2=0x7f226c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 shutdown_connections 2020-11-20T12:05:00.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.001+0000 7f22708be700 1 -- 172.21.15.187:0/2191587881 wait complete. 2020-11-20T12:05:00.034 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:00.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.316+0000 7faaeaab8700 1 Processor -- start 2020-11-20T12:05:00.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.316+0000 7faaeaab8700 1 -- start start 2020-11-20T12:05:00.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.317+0000 7faaeaab8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faae4145cc0 0x7faae4066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.317+0000 7faaeaab8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae41487e0 0x7faae4148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.317+0000 7faaeaab8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae414c170 con 0x7faae40683f0 2020-11-20T12:05:00.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.317+0000 7faaeaab8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae414c410 con 0x7faae4145cc0 2020-11-20T12:05:00.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.317+0000 7faaeaab8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae414c6b0 con 0x7faae41487e0 2020-11-20T12:05:00.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.318+0000 7faae92b5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7faae40683f0 0x7faae4065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36956/0 (socket says 172.21.15.187:36956) 2020-11-20T12:05:00.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.318+0000 7faae92b5700 1 -- 172.21.15.187:0/3537364957 learned_addr learned my addr 172.21.15.187:0/3537364957 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:00.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.318+0000 7faae9ab6700 1 --2- 172.21.15.187:0/3537364957 >> v2:172.21.15.187:3300/0 conn(0x7faae4145cc0 0x7faae4066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.318+0000 7faae8ab4700 1 -- 172.21.15.187:0/3537364957 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21897335 0 0) 0x7faae414c170 con 0x7faae40683f0 2020-11-20T12:05:00.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.318+0000 7faae8ab4700 1 -- 172.21.15.187:0/3537364957 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faacc003240 con 0x7faae40683f0 2020-11-20T12:05:00.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.318+0000 7faaea2b7700 1 --2- 172.21.15.187:0/3537364957 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae41487e0 0x7faae4148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae8ab4700 1 -- 172.21.15.187:0/3537364957 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3289662473 0 0) 0x7faacc003240 con 0x7faae40683f0 2020-11-20T12:05:00.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae9ab6700 1 -- 172.21.15.187:0/3537364957 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae41487e0 msgr2=0x7faae4148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae9ab6700 1 --2- 172.21.15.187:0/3537364957 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae41487e0 0x7faae4148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae9ab6700 1 -- 172.21.15.187:0/3537364957 >> v1:172.21.15.187:6789/0 conn(0x7faae40683f0 legacy=0x7faae4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae9ab6700 1 -- 172.21.15.187:0/3537364957 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae40658a0 con 0x7faae4145cc0 2020-11-20T12:05:00.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faaea2b7700 1 --2- 172.21.15.187:0/3537364957 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae41487e0 0x7faae4148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:00.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae9ab6700 1 --2- 172.21.15.187:0/3537364957 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae4066040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7faae0009c90 tx=0x7faae00091d0).ready entity=mon.1 client_cookie=d4f45536966213 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae8ab4700 1 -- 172.21.15.187:0/3537364957 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad40014e0 con 0x7faae4145cc0 2020-11-20T12:05:00.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae8ab4700 1 -- 172.21.15.187:0/3537364957 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae000b760 con 0x7faae4145cc0 2020-11-20T12:05:00.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.319+0000 7faae8ab4700 1 -- 172.21.15.187:0/3537364957 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae000bae0 con 0x7faae4145cc0 2020-11-20T12:05:00.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.320+0000 7faaeaab8700 1 -- 172.21.15.187:0/3537364957 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 msgr2=0x7faae4066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.320+0000 7faaeaab8700 1 --2- 172.21.15.187:0/3537364957 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae4066040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7faae0009c90 tx=0x7faae00091d0).stop 2020-11-20T12:05:00.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.320+0000 7faaeaab8700 1 -- 172.21.15.187:0/3537364957 shutdown_connections 2020-11-20T12:05:00.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.320+0000 7faaeaab8700 1 --2- 172.21.15.187:0/3537364957 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae41487e0 0x7faae4148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.320+0000 7faaeaab8700 1 --2- 172.21.15.187:0/3537364957 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae4066040 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.320+0000 7faaeaab8700 1 -- 172.21.15.187:0/3537364957 >> 172.21.15.187:0/3537364957 conn(0x7faae40bab30 msgr2=0x7faae40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.320+0000 7faaeaab8700 1 -- 172.21.15.187:0/3537364957 shutdown_connections 2020-11-20T12:05:00.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.320+0000 7faaeaab8700 1 -- 172.21.15.187:0/3537364957 wait complete. 2020-11-20T12:05:00.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.322+0000 7faaeaab8700 1 Processor -- start 2020-11-20T12:05:00.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.322+0000 7faaeaab8700 1 -- start start 2020-11-20T12:05:00.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.323+0000 7faaeaab8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae40683f0 0x7faae40b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.323+0000 7faaeaab8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae40a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.323+0000 7faaeaab8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faae41487e0 0x7faae4144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.323+0000 7faaeaab8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae409c660 con 0x7faae41487e0 2020-11-20T12:05:00.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.323+0000 7faaeaab8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae409cdd0 con 0x7faae4145cc0 2020-11-20T12:05:00.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.323+0000 7faaeaab8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae409c890 con 0x7faae40683f0 2020-11-20T12:05:00.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae40a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae40a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40356/0 (socket says 172.21.15.187:40356) 2020-11-20T12:05:00.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 -- 172.21.15.187:0/2122016625 learned_addr learned my addr 172.21.15.187:0/2122016625 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:00.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae9ab6700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae40683f0 0x7faae40b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faaea2b7700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faae41487e0 0x7faae4144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 -- 172.21.15.187:0/2122016625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae40683f0 msgr2=0x7faae40b1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae40683f0 0x7faae40b1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 -- 172.21.15.187:0/2122016625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faae41487e0 msgr2=0x7faae4144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faae41487e0 0x7faae4144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 -- 172.21.15.187:0/2122016625 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae409cb30 con 0x7faae4145cc0 2020-11-20T12:05:00.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.324+0000 7faae92b5700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae40a03a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7faad40013f0 tx=0x7faad4001af0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.325+0000 7faad1ffb700 1 -- 172.21.15.187:0/2122016625 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4007170 con 0x7faae4145cc0 2020-11-20T12:05:00.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.325+0000 7faad1ffb700 1 -- 172.21.15.187:0/2122016625 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad40072f0 con 0x7faae4145cc0 2020-11-20T12:05:00.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.325+0000 7faad1ffb700 1 -- 172.21.15.187:0/2122016625 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad400aa90 con 0x7faae4145cc0 2020-11-20T12:05:00.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.325+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae40a0890 con 0x7faae4145cc0 2020-11-20T12:05:00.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.325+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae409c1d0 con 0x7faae4145cc0 2020-11-20T12:05:00.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.326+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae4061a20 con 0x7faae4145cc0 2020-11-20T12:05:00.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.327+0000 7faad1ffb700 1 -- 172.21.15.187:0/2122016625 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faad4010070 con 0x7faae4145cc0 2020-11-20T12:05:00.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.327+0000 7faad1ffb700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faabc056d20 0x7faabc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.328+0000 7faad1ffb700 1 -- 172.21.15.187:0/2122016625 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faad406fd70 con 0x7faae4145cc0 2020-11-20T12:05:00.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.328+0000 7faae9ab6700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faabc056d20 0x7faabc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.328+0000 7faae9ab6700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faabc056d20 0x7faabc0591e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7faae000b670 tx=0x7faae0016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.331+0000 7faad1ffb700 1 -- 172.21.15.187:0/2122016625 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faad40450b0 con 0x7faae4145cc0 2020-11-20T12:05:00.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.577+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faae414bdf0 con 0x7faae4145cc0 2020-11-20T12:05:00.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.578+0000 7faad1ffb700 1 -- 172.21.15.187:0/2122016625 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2130 (secure 0 0 0) 0x7faad40486d0 con 0x7faae4145cc0 2020-11-20T12:05:00.582 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:00.583 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4211,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6833/2094645874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2094645874},{"type":"v1","addr":"172.21.15.5:6833","nonce":2094645874}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:57.868606+0000","modified":"2020-11-20T12:04:59.990344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3094884769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3094884769},{"type":"v1","addr":"172.21.15.187:6835","nonce":3094884769}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:00.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faabc056d20 msgr2=0x7faabc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faabc056d20 0x7faabc0591e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7faae000b670 tx=0x7faae0016000).stop 2020-11-20T12:05:00.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 msgr2=0x7faae40a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae40a03a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7faad40013f0 tx=0x7faad4001af0).stop 2020-11-20T12:05:00.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 shutdown_connections 2020-11-20T12:05:00.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faabc056d20 0x7faabc0591e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faae41487e0 0x7faae4144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faae40683f0 0x7faae40b1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 --2- 172.21.15.187:0/2122016625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faae4145cc0 0x7faae40a03a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.580+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 >> 172.21.15.187:0/2122016625 conn(0x7faae40bab30 msgr2=0x7faae41477f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.581+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 shutdown_connections 2020-11-20T12:05:00.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.581+0000 7faaeaab8700 1 -- 172.21.15.187:0/2122016625 wait complete. 2020-11-20T12:05:00.588 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 6 2020-11-20T12:05:00.609 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:00.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.850+0000 7f6e318de700 1 Processor -- start 2020-11-20T12:05:00.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.850+0000 7f6e318de700 1 -- start start 2020-11-20T12:05:00.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.851+0000 7f6e318de700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c1484d0 0x7f6e2c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.851+0000 7f6e318de700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e2c066f70 0x7f6e2c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.851+0000 7f6e318de700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c0b1300 con 0x7f6e2c0622c0 2020-11-20T12:05:00.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.851+0000 7f6e318de700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2c067730 con 0x7f6e2c066f70 2020-11-20T12:05:00.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.851+0000 7f6e318de700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c0b15a0 con 0x7f6e2c1484d0 2020-11-20T12:05:00.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.852+0000 7f6e23fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6e2c0622c0 0x7f6e2c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36972/0 (socket says 172.21.15.187:36972) 2020-11-20T12:05:00.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.852+0000 7f6e23fff700 1 -- 172.21.15.187:0/941957376 learned_addr learned my addr 172.21.15.187:0/941957376 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.852+0000 7f6e308dc700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c1484d0 0x7f6e2c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.852+0000 7f6e310dd700 1 --2- 172.21.15.187:0/941957376 >> v2:172.21.15.5:3300/0 conn(0x7f6e2c066f70 0x7f6e2c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.852+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661257827 0 0) 0x7f6e2c0b1300 con 0x7f6e2c0622c0 2020-11-20T12:05:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.852+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e18003240 con 0x7f6e2c0622c0 2020-11-20T12:05:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 365826594 0 0) 0x7f6e18003240 con 0x7f6e2c0622c0 2020-11-20T12:05:00.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c1484d0 msgr2=0x7f6e2c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c1484d0 0x7f6e2c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 >> v2:172.21.15.5:3300/0 conn(0x7f6e2c066f70 msgr2=0x7f6e2c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 --2- 172.21.15.187:0/941957376 >> v2:172.21.15.5:3300/0 conn(0x7f6e2c066f70 0x7f6e2c0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c0b1840 con 0x7f6e2c0622c0 2020-11-20T12:05:00.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6e1c0014e0 con 0x7f6e2c0622c0 2020-11-20T12:05:00.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 >> v1:172.21.15.187:6789/0 conn(0x7f6e2c0622c0 legacy=0x7f6e2c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e18003730 0x7f6e18003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c1484d0 0x7f6e180044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e180049d0 0x7f6e18004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c067730 con 0x7f6e180049d0 2020-11-20T12:05:00.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c0b15a0 con 0x7f6e18003730 2020-11-20T12:05:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180041d0 con 0x7f6e2c1484d0 2020-11-20T12:05:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.853+0000 7f6e308dc700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e18003730 0x7f6e18003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e310dd700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e180049d0 0x7f6e18004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e308dc700 1 -- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c1484d0 msgr2=0x7f6e180044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e308dc700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c1484d0 0x7f6e180044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e308dc700 1 -- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e180049d0 msgr2=0x7f6e18004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e308dc700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e180049d0 0x7f6e18004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e308dc700 1 -- 172.21.15.187:0/941957376 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e18003240 con 0x7f6e18003730 2020-11-20T12:05:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e310dd700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e180049d0 0x7f6e18004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e308dc700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e18003730 0x7f6e18003b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f6e28009ea0 tx=0x7f6e280091d0).ready entity=mon.1 client_cookie=16776af31a3f9bf0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e280095c0 con 0x7f6e18003730 2020-11-20T12:05:00.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2800eda0 con 0x7f6e18003730 2020-11-20T12:05:00.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.854+0000 7f6e237fe700 1 -- 172.21.15.187:0/941957376 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28012600 con 0x7f6e18003730 2020-11-20T12:05:00.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.855+0000 7f6e318de700 1 -- 172.21.15.187:0/941957376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e18003730 msgr2=0x7f6e18003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.855+0000 7f6e318de700 1 --2- 172.21.15.187:0/941957376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e18003730 0x7f6e18003b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f6e28009ea0 tx=0x7f6e280091d0).stop 2020-11-20T12:05:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.855+0000 7f6e310dd700 1 -- 172.21.15.187:0/941957376 reap_dead start 2020-11-20T12:05:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.855+0000 7f6e318de700 1 -- 172.21.15.187:0/941957376 shutdown_connections 2020-11-20T12:05:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.855+0000 7f6e318de700 1 -- 172.21.15.187:0/941957376 >> 172.21.15.187:0/941957376 conn(0x7f6e2c0bab30 msgr2=0x7f6e2c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.855+0000 7f6e318de700 1 -- 172.21.15.187:0/941957376 shutdown_connections 2020-11-20T12:05:00.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.855+0000 7f6e318de700 1 -- 172.21.15.187:0/941957376 wait complete. 2020-11-20T12:05:00.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.856+0000 7f6e318de700 1 Processor -- start 2020-11-20T12:05:00.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.857+0000 7f6e318de700 1 -- start start 2020-11-20T12:05:00.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.857+0000 7f6e318de700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e2c13b620 0x7f6e2c13ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.857+0000 7f6e318de700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c13ba00 0x7f6e2c13d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.857+0000 7f6e318de700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e2c135ea0 0x7f6e2c136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.857+0000 7f6e318de700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c13dc50 con 0x7f6e2c13b620 2020-11-20T12:05:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.857+0000 7f6e318de700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c136810 con 0x7f6e2c135ea0 2020-11-20T12:05:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.857+0000 7f6e318de700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c13de40 con 0x7f6e2c13ba00 2020-11-20T12:05:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e310dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e2c135ea0 0x7f6e2c136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e310dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e2c135ea0 0x7f6e2c136320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40378/0 (socket says 172.21.15.187:40378) 2020-11-20T12:05:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e310dd700 1 -- 172.21.15.187:0/2689483249 learned_addr learned my addr 172.21.15.187:0/2689483249 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e23fff700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c13ba00 0x7f6e2c13d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e308dc700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e2c13b620 0x7f6e2c13ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e310dd700 1 -- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c13ba00 msgr2=0x7f6e2c13d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e310dd700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c13ba00 0x7f6e2c13d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e310dd700 1 -- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e2c13b620 msgr2=0x7f6e2c13ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e310dd700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e2c13b620 0x7f6e2c13ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e310dd700 1 -- 172.21.15.187:0/2689483249 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c0b2d70 con 0x7f6e2c135ea0 2020-11-20T12:05:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.858+0000 7f6e308dc700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e2c13b620 0x7f6e2c13ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:00.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.859+0000 7f6e310dd700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e2c135ea0 0x7f6e2c136320 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f6e2c0b85e0 tx=0x7f6e24008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.859+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2689483249 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24017070 con 0x7f6e2c135ea0 2020-11-20T12:05:00.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.859+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2689483249 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e24012570 con 0x7f6e2c135ea0 2020-11-20T12:05:00.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.859+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2689483249 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e240128f0 con 0x7f6e2c135ea0 2020-11-20T12:05:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.859+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c0b3010 con 0x7f6e2c135ea0 2020-11-20T12:05:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.859+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c1593f0 con 0x7f6e2c135ea0 2020-11-20T12:05:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.860+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2c0b31a0 con 0x7f6e2c135ea0 2020-11-20T12:05:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.861+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2689483249 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6e24012ca0 con 0x7f6e2c135ea0 2020-11-20T12:05:00.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.861+0000 7f6e20ff9700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e04056d20 0x7f6e040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.861+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2689483249 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6e2406fb90 con 0x7f6e2c135ea0 2020-11-20T12:05:00.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.862+0000 7f6e308dc700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e04056d20 0x7f6e040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.862+0000 7f6e308dc700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e04056d20 0x7f6e040591e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f6e2c05ea30 tx=0x7f6e2801a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:00.865+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2689483249 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6e24044fe0 con 0x7f6e2c135ea0 2020-11-20T12:05:01.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.058+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e2c061a20 con 0x7f6e2c135ea0 2020-11-20T12:05:01.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.060+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2689483249 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f6e24048600 con 0x7f6e2c135ea0 2020-11-20T12:05:01.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:01.065 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:01.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e04056d20 msgr2=0x7f6e040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e04056d20 0x7f6e040591e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f6e2c05ea30 tx=0x7f6e2801a000).stop 2020-11-20T12:05:01.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e2c135ea0 msgr2=0x7f6e2c136320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e2c135ea0 0x7f6e2c136320 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f6e2c0b85e0 tx=0x7f6e24008040).stop 2020-11-20T12:05:01.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 shutdown_connections 2020-11-20T12:05:01.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e04056d20 0x7f6e040591e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e2c13b620 0x7f6e2c13ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e2c135ea0 0x7f6e2c136320 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 --2- 172.21.15.187:0/2689483249 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e2c13ba00 0x7f6e2c13d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.062+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 >> 172.21.15.187:0/2689483249 conn(0x7f6e2c0bab30 msgr2=0x7f6e2c065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:01.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.063+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 shutdown_connections 2020-11-20T12:05:01.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.063+0000 7f6e318de700 1 -- 172.21.15.187:0/2689483249 wait complete. 2020-11-20T12:05:01.090 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:01.091 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-20T12:05:01.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.329+0000 7fd85ad82700 1 Processor -- start 2020-11-20T12:05:01.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.329+0000 7fd85ad82700 1 -- start start 2020-11-20T12:05:01.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd85ad82700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd854145cc0 0x7fd854066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd85ad82700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8540683f0 0x7fd854065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd85ad82700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd85414c170 con 0x7fd854065680 2020-11-20T12:05:01.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd85ad82700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8541497f0 con 0x7fd854145cc0 2020-11-20T12:05:01.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd85ad82700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85414c410 con 0x7fd8540683f0 2020-11-20T12:05:01.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd85a581700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd854065680 0x7fd854061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:36990/0 (socket says 172.21.15.187:36990) 2020-11-20T12:05:01.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd85a581700 1 -- 172.21.15.187:0/739670059 learned_addr learned my addr 172.21.15.187:0/739670059 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:01.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd859d80700 1 --2- 172.21.15.187:0/739670059 >> v2:172.21.15.187:3300/0 conn(0x7fd854145cc0 0x7fd854066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.331+0000 7fd85957f700 1 --2- 172.21.15.187:0/739670059 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8540683f0 0x7fd854065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.332+0000 7fd858d7e700 1 -- 172.21.15.187:0/739670059 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827195145 0 0) 0x7fd85414c170 con 0x7fd854065680 2020-11-20T12:05:01.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.332+0000 7fd858d7e700 1 -- 172.21.15.187:0/739670059 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd83c003240 con 0x7fd854065680 2020-11-20T12:05:01.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.332+0000 7fd859d80700 1 -- 172.21.15.187:0/739670059 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8540683f0 msgr2=0x7fd854065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:01.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.332+0000 7fd859d80700 1 --2- 172.21.15.187:0/739670059 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8540683f0 0x7fd854065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.332+0000 7fd859d80700 1 -- 172.21.15.187:0/739670059 >> v1:172.21.15.187:6789/0 conn(0x7fd854065680 legacy=0x7fd854061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.332+0000 7fd859d80700 1 -- 172.21.15.187:0/739670059 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85414c6b0 con 0x7fd854145cc0 2020-11-20T12:05:01.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.332+0000 7fd859d80700 1 --2- 172.21.15.187:0/739670059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd854066040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd85000b9a0 tx=0x7fd850006df0).ready entity=mon.1 client_cookie=6e22b54018c9ddb7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:01.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd858d7e700 1 -- 172.21.15.187:0/739670059 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850009680 con 0x7fd854145cc0 2020-11-20T12:05:01.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd858d7e700 1 -- 172.21.15.187:0/739670059 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8500097e0 con 0x7fd854145cc0 2020-11-20T12:05:01.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd858d7e700 1 -- 172.21.15.187:0/739670059 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850009b60 con 0x7fd854145cc0 2020-11-20T12:05:01.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd85ad82700 1 -- 172.21.15.187:0/739670059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 msgr2=0x7fd854066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd85ad82700 1 --2- 172.21.15.187:0/739670059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd854066040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd85000b9a0 tx=0x7fd850006df0).stop 2020-11-20T12:05:01.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd85ad82700 1 -- 172.21.15.187:0/739670059 shutdown_connections 2020-11-20T12:05:01.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd85ad82700 1 --2- 172.21.15.187:0/739670059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd854066040 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd85ad82700 1 --2- 172.21.15.187:0/739670059 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8540683f0 0x7fd854065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd85ad82700 1 -- 172.21.15.187:0/739670059 >> 172.21.15.187:0/739670059 conn(0x7fd8540bab30 msgr2=0x7fd8540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:01.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd85ad82700 1 -- 172.21.15.187:0/739670059 shutdown_connections 2020-11-20T12:05:01.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.333+0000 7fd85ad82700 1 -- 172.21.15.187:0/739670059 wait complete. 2020-11-20T12:05:01.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.335+0000 7fd85ad82700 1 Processor -- start 2020-11-20T12:05:01.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.335+0000 7fd85ad82700 1 -- start start 2020-11-20T12:05:01.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85ad82700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd854065680 0x7fd854157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85ad82700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8540683f0 0x7fd85409d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85ad82700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd85414b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85ad82700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85409fa70 con 0x7fd8540683f0 2020-11-20T12:05:01.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85ad82700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85409fca0 con 0x7fd854145cc0 2020-11-20T12:05:01.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85ad82700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85409ff40 con 0x7fd854065680 2020-11-20T12:05:01.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85a581700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd85414b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85a581700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd85414b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40392/0 (socket says 172.21.15.187:40392) 2020-11-20T12:05:01.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85a581700 1 -- 172.21.15.187:0/289955563 learned_addr learned my addr 172.21.15.187:0/289955563 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:01.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd859d80700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd854065680 0x7fd854157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85a581700 1 -- 172.21.15.187:0/289955563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd854065680 msgr2=0x7fd854157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85a581700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd854065680 0x7fd854157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85a581700 1 -- 172.21.15.187:0/289955563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8540683f0 msgr2=0x7fd85409d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85a581700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8540683f0 0x7fd85409d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:01.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.336+0000 7fd85a581700 1 -- 172.21.15.187:0/289955563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd854158450 con 0x7fd854145cc0 2020-11-20T12:05:01.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.337+0000 7fd85a581700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd85414b000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd84c001420 tx=0x7fd84c007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:01.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.337+0000 7fd841ffb700 1 -- 172.21.15.187:0/289955563 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c005b80 con 0x7fd854145cc0 2020-11-20T12:05:01.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.337+0000 7fd841ffb700 1 -- 172.21.15.187:0/289955563 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c005ce0 con 0x7fd854145cc0 2020-11-20T12:05:01.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.337+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8540a07e0 con 0x7fd854145cc0 2020-11-20T12:05:01.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.337+0000 7fd841ffb700 1 -- 172.21.15.187:0/289955563 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c007320 con 0x7fd854145cc0 2020-11-20T12:05:01.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.337+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85414b4f0 con 0x7fd854145cc0 2020-11-20T12:05:01.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.338+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8540a0100 con 0x7fd854145cc0 2020-11-20T12:05:01.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.339+0000 7fd841ffb700 1 -- 172.21.15.187:0/289955563 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd84c010030 con 0x7fd854145cc0 2020-11-20T12:05:01.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.339+0000 7fd841ffb700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd82c056d50 0x7fd82c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.339+0000 7fd841ffb700 1 -- 172.21.15.187:0/289955563 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd84c06ffa0 con 0x7fd854145cc0 2020-11-20T12:05:01.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.340+0000 7fd859d80700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd82c056d50 0x7fd82c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.340+0000 7fd859d80700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd82c056d50 0x7fd82c059210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd85000c010 tx=0x7fd850015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:01.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.344+0000 7fd841ffb700 1 -- 172.21.15.187:0/289955563 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd84c0453f0 con 0x7fd854145cc0 2020-11-20T12:05:01.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.534+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd854061a20 con 0x7fd854145cc0 2020-11-20T12:05:01.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.537+0000 7fd841ffb700 1 -- 172.21.15.187:0/289955563 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7fd84c048a10 con 0x7fd854145cc0 2020-11-20T12:05:01.541 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:01.542 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:01.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd82c056d50 msgr2=0x7fd82c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd82c056d50 0x7fd82c059210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd85000c010 tx=0x7fd850015000).stop 2020-11-20T12:05:01.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 msgr2=0x7fd85414b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd85414b000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd84c001420 tx=0x7fd84c007e20).stop 2020-11-20T12:05:01.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 shutdown_connections 2020-11-20T12:05:01.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd82c056d50 0x7fd82c059210 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd854145cc0 0x7fd85414b000 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd854065680 0x7fd854157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 --2- 172.21.15.187:0/289955563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8540683f0 0x7fd85409d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:01.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 >> 172.21.15.187:0/289955563 conn(0x7fd8540bab30 msgr2=0x7fd854144de0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:01.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 shutdown_connections 2020-11-20T12:05:01.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.539+0000 7fd85ad82700 1 -- 172.21.15.187:0/289955563 wait complete. 2020-11-20T12:05:01.567 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:01.568 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:01.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.808+0000 7fa3c5fda700 1 Processor -- start 2020-11-20T12:05:01.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.808+0000 7fa3c5fda700 1 -- start start 2020-11-20T12:05:01.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.810+0000 7fa3c5fda700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa3c009f560 0x7fa3c0153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.810+0000 7fa3c5fda700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c014d110 con 0x7fa3c00bfbe0 2020-11-20T12:05:01.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.810+0000 7fa3c5fda700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c014cbd0 con 0x7fa3c00bf800 2020-11-20T12:05:01.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.810+0000 7fa3c5fda700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c014c9d0 con 0x7fa3c009f560 2020-11-20T12:05:01.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.810+0000 7fa3c4fd8700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa3c00bf800 0x7fa3c014c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37002/0 (socket says 172.21.15.187:37002) 2020-11-20T12:05:01.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.810+0000 7fa3c4fd8700 1 -- 172.21.15.187:0/1927589795 learned_addr learned my addr 172.21.15.187:0/1927589795 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:01.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.810+0000 7fa3c57d9700 1 --2- 172.21.15.187:0/1927589795 >> v2:172.21.15.5:3300/0 conn(0x7fa3c009f560 0x7fa3c0153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965959323 0 0) 0x7fa3c014cbd0 con 0x7fa3c00bf800 2020-11-20T12:05:01.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac003240 con 0x7fa3c00bf800 2020-11-20T12:05:01.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2540805621 0 0) 0x7fa3ac003240 con 0x7fa3c00bf800 2020-11-20T12:05:01.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 >> v2:172.21.15.5:3300/0 conn(0x7fa3c009f560 msgr2=0x7fa3c0153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:01.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1927589795 >> v2:172.21.15.5:3300/0 conn(0x7fa3c009f560 0x7fa3c0153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 >> v1:172.21.15.5:6789/0 conn(0x7fa3c00bfbe0 legacy=0x7fa3c01480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c014ce70 con 0x7fa3c00bf800 2020-11-20T12:05:01.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fa3bc0014e0 con 0x7fa3c00bf800 2020-11-20T12:05:01.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 >> v1:172.21.15.187:6789/0 conn(0x7fa3c00bf800 legacy=0x7fa3c014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3ac003780 0x7fa3ac003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac0044e0 0x7fa3ac004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3ac004e50 0x7fa3ac00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c014c9d0 con 0x7fa3ac0044e0 2020-11-20T12:05:01.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.811+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac005300 con 0x7fa3ac004e50 2020-11-20T12:05:01.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac00bc20 con 0x7fa3ac003780 2020-11-20T12:05:01.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3c57d9700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3ac004e50 0x7fa3ac00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3c4fd8700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3ac003780 0x7fa3ac003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3b7fff700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac0044e0 0x7fa3ac004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3c57d9700 1 -- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3ac003780 msgr2=0x7fa3ac003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:01.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3c57d9700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3ac003780 0x7fa3ac003ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3c57d9700 1 -- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac0044e0 msgr2=0x7fa3ac004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:01.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3c57d9700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac0044e0 0x7fa3ac004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3c57d9700 1 -- 172.21.15.187:0/1927589795 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac003240 con 0x7fa3ac004e50 2020-11-20T12:05:01.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.812+0000 7fa3b7fff700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac0044e0 0x7fa3ac004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:01.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.813+0000 7fa3c57d9700 1 -- 172.21.15.187:0/1927589795 reap_dead start 2020-11-20T12:05:01.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.813+0000 7fa3c57d9700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3ac004e50 0x7fa3ac00b730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa3b8009ea0 tx=0x7fa3b80091d0).ready entity=mon.1 client_cookie=7837e8b75c4402f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:01.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.813+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b80095c0 con 0x7fa3ac004e50 2020-11-20T12:05:01.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.813+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b800eda0 con 0x7fa3ac004e50 2020-11-20T12:05:01.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.813+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1927589795 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8012640 con 0x7fa3ac004e50 2020-11-20T12:05:01.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.813+0000 7fa3c5fda700 1 -- 172.21.15.187:0/1927589795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3ac004e50 msgr2=0x7fa3ac00b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.813+0000 7fa3c5fda700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3ac004e50 0x7fa3ac00b730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa3b8009ea0 tx=0x7fa3b80091d0).stop 2020-11-20T12:05:01.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.814+0000 7fa3c5fda700 1 -- 172.21.15.187:0/1927589795 shutdown_connections 2020-11-20T12:05:01.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.814+0000 7fa3c5fda700 1 --2- 172.21.15.187:0/1927589795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3ac004e50 0x7fa3ac00b730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.814+0000 7fa3c5fda700 1 -- 172.21.15.187:0/1927589795 >> 172.21.15.187:0/1927589795 conn(0x7fa3c0068370 msgr2=0x7fa3c00bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:01.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.814+0000 7fa3c5fda700 1 -- 172.21.15.187:0/1927589795 shutdown_connections 2020-11-20T12:05:01.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.814+0000 7fa3c5fda700 1 -- 172.21.15.187:0/1927589795 wait complete. 2020-11-20T12:05:01.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.815+0000 7fa3c5fda700 1 Processor -- start 2020-11-20T12:05:01.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.816+0000 7fa3c5fda700 1 -- start start 2020-11-20T12:05:01.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c5fda700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac004e50 0x7fa3c00b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c5fda700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3c00b6d50 0x7fa3c00ba7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c5fda700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c0145f40 0x7fa3c0144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c5fda700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c0143cb0 con 0x7fa3ac004e50 2020-11-20T12:05:01.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c5fda700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c0143f50 con 0x7fa3c0145f40 2020-11-20T12:05:01.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c5fda700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c01441f0 con 0x7fa3c00b6d50 2020-11-20T12:05:01.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c57d9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c0145f40 0x7fa3c0144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c57d9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c0145f40 0x7fa3c0144690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40412/0 (socket says 172.21.15.187:40412) 2020-11-20T12:05:01.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c57d9700 1 -- 172.21.15.187:0/3387013611 learned_addr learned my addr 172.21.15.187:0/3387013611 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:01.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c4fd8700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac004e50 0x7fa3c00b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3b7fff700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3c00b6d50 0x7fa3c00ba7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c57d9700 1 -- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3c00b6d50 msgr2=0x7fa3c00ba7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c57d9700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3c00b6d50 0x7fa3c00ba7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c57d9700 1 -- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac004e50 msgr2=0x7fa3c00b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:01.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.817+0000 7fa3c57d9700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac004e50 0x7fa3c00b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:01.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.818+0000 7fa3c57d9700 1 -- 172.21.15.187:0/3387013611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c00bace0 con 0x7fa3c0145f40 2020-11-20T12:05:01.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.818+0000 7fa3c4fd8700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac004e50 0x7fa3c00b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:01.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.818+0000 7fa3c57d9700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c0145f40 0x7fa3c0144690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa3c0062a00 tx=0x7fa3b8019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:01.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.818+0000 7fa3b4ff9700 1 -- 172.21.15.187:0/3387013611 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8006ca0 con 0x7fa3c0145f40 2020-11-20T12:05:01.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.818+0000 7fa3b4ff9700 1 -- 172.21.15.187:0/3387013611 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b80095c0 con 0x7fa3c0145f40 2020-11-20T12:05:01.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.818+0000 7fa3b4ff9700 1 -- 172.21.15.187:0/3387013611 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b801cea0 con 0x7fa3c0145f40 2020-11-20T12:05:01.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.818+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c0143880 con 0x7fa3c0145f40 2020-11-20T12:05:01.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.818+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c0146600 con 0x7fa3c0145f40 2020-11-20T12:05:01.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.819+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c0080050 con 0x7fa3c0145f40 2020-11-20T12:05:01.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.820+0000 7fa3b4ff9700 1 -- 172.21.15.187:0/3387013611 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa3b802c070 con 0x7fa3c0145f40 2020-11-20T12:05:01.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.821+0000 7fa3b4ff9700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa398056d20 0x7fa3980591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:01.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.821+0000 7fa3b4ff9700 1 -- 172.21.15.187:0/3387013611 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa3b8071d50 con 0x7fa3c0145f40 2020-11-20T12:05:01.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.821+0000 7fa3c4fd8700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa398056d20 0x7fa3980591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:01.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.822+0000 7fa3c4fd8700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa398056d20 0x7fa3980591e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa3bc007640 tx=0x7fa3bc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:01.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:01.825+0000 7fa3b4ff9700 1 -- 172.21.15.187:0/3387013611 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa3b80471a0 con 0x7fa3c0145f40 2020-11-20T12:05:02.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.013+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa3c01443b0 con 0x7fa3c0145f40 2020-11-20T12:05:02.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.014+0000 7fa3b4ff9700 1 -- 172.21.15.187:0/3387013611 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa3b8016070 con 0x7fa3c0145f40 2020-11-20T12:05:02.018 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:02.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.016+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa398056d20 msgr2=0x7fa3980591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.016+0000 7fa3c5fda700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa398056d20 0x7fa3980591e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa3bc007640 tx=0x7fa3bc009040).stop 2020-11-20T12:05:02.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.016+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c0145f40 msgr2=0x7fa3c0144690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.016+0000 7fa3c5fda700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c0145f40 0x7fa3c0144690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa3c0062a00 tx=0x7fa3b8019000).stop 2020-11-20T12:05:02.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.017+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 shutdown_connections 2020-11-20T12:05:02.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.017+0000 7fa3c5fda700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa398056d20 0x7fa3980591e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.017+0000 7fa3c5fda700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3ac004e50 0x7fa3c00b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.017+0000 7fa3c5fda700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c0145f40 0x7fa3c0144690 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.017+0000 7fa3c5fda700 1 --2- 172.21.15.187:0/3387013611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3c00b6d50 0x7fa3c00ba7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.017+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 >> 172.21.15.187:0/3387013611 conn(0x7fa3c0068370 msgr2=0x7fa3c00be730 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.017+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 shutdown_connections 2020-11-20T12:05:02.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.017+0000 7fa3c5fda700 1 -- 172.21.15.187:0/3387013611 wait complete. 2020-11-20T12:05:02.042 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:02.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.278+0000 7fd9cffff700 1 Processor -- start 2020-11-20T12:05:02.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.278+0000 7fd9cffff700 1 -- start start 2020-11-20T12:05:02.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.279+0000 7fd9cffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9d00bf800 0x7fd9d014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.279+0000 7fd9cffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d00bfbe0 0x7fd9d01480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.279+0000 7fd9cffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d014d110 con 0x7fd9d009f560 2020-11-20T12:05:02.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.279+0000 7fd9cffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d014cbd0 con 0x7fd9d00bf800 2020-11-20T12:05:02.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.279+0000 7fd9cffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d014ce70 con 0x7fd9d00bfbe0 2020-11-20T12:05:02.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.279+0000 7fd9ceffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9d00bf800 0x7fd9d014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.280+0000 7fd9ceffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9d00bf800 0x7fd9d014c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40416/0 (socket says 172.21.15.187:40416) 2020-11-20T12:05:02.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.280+0000 7fd9ceffd700 1 -- 172.21.15.187:0/1138049589 learned_addr learned my addr 172.21.15.187:0/1138049589 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:02.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.280+0000 7fd9c67fc700 1 --2- 172.21.15.187:0/1138049589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d00bfbe0 0x7fd9d01480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.280+0000 7fd9ceffd700 1 -- 172.21.15.187:0/1138049589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d00bfbe0 msgr2=0x7fd9d01480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.280+0000 7fd9ceffd700 1 --2- 172.21.15.187:0/1138049589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d00bfbe0 0x7fd9d01480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.280+0000 7fd9ceffd700 1 -- 172.21.15.187:0/1138049589 >> v1:172.21.15.187:6789/0 conn(0x7fd9d009f560 legacy=0x7fd9d0060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.280+0000 7fd9ceffd700 1 -- 172.21.15.187:0/1138049589 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d00a4cc0 con 0x7fd9d00bf800 2020-11-20T12:05:02.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.281+0000 7fd9ceffd700 1 --2- 172.21.15.187:0/1138049589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d014c2c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fd9c0009d50 tx=0x7fd9c0009290).ready entity=mon.1 client_cookie=c521f57f5c744e68 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.281+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/1138049589 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c000bd10 con 0x7fd9d00bf800 2020-11-20T12:05:02.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.281+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/1138049589 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c000be70 con 0x7fd9d00bf800 2020-11-20T12:05:02.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.281+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/1138049589 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0011610 con 0x7fd9d00bf800 2020-11-20T12:05:02.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.282+0000 7fd9cffff700 1 -- 172.21.15.187:0/1138049589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 msgr2=0x7fd9d014c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.282+0000 7fd9cffff700 1 --2- 172.21.15.187:0/1138049589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d014c2c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fd9c0009d50 tx=0x7fd9c0009290).stop 2020-11-20T12:05:02.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.282+0000 7fd9cffff700 1 -- 172.21.15.187:0/1138049589 shutdown_connections 2020-11-20T12:05:02.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.282+0000 7fd9cffff700 1 --2- 172.21.15.187:0/1138049589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d014c2c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.282+0000 7fd9cffff700 1 --2- 172.21.15.187:0/1138049589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d00bfbe0 0x7fd9d01480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.282+0000 7fd9cffff700 1 -- 172.21.15.187:0/1138049589 >> 172.21.15.187:0/1138049589 conn(0x7fd9d0068370 msgr2=0x7fd9d00bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.282+0000 7fd9cffff700 1 -- 172.21.15.187:0/1138049589 shutdown_connections 2020-11-20T12:05:02.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.282+0000 7fd9cffff700 1 -- 172.21.15.187:0/1138049589 wait complete. 2020-11-20T12:05:02.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.283+0000 7fd9cffff700 1 Processor -- start 2020-11-20T12:05:02.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.284+0000 7fd9cffff700 1 -- start start 2020-11-20T12:05:02.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.284+0000 7fd9cffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d009f560 0x7fd9d014a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.284+0000 7fd9cffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d0143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.284+0000 7fd9cffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9d01588f0 0x7fd9d0141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.284+0000 7fd9cffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d00b3e90 con 0x7fd9d01588f0 2020-11-20T12:05:02.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.284+0000 7fd9cffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d00b40c0 con 0x7fd9d00bf800 2020-11-20T12:05:02.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.284+0000 7fd9cffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d00b4360 con 0x7fd9d009f560 2020-11-20T12:05:02.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9c67fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d0143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9c67fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d0143240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40424/0 (socket says 172.21.15.187:40424) 2020-11-20T12:05:02.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9c67fc700 1 -- 172.21.15.187:0/2225201750 learned_addr learned my addr 172.21.15.187:0/2225201750 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:02.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9ceffd700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d009f560 0x7fd9d014a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9c67fc700 1 -- 172.21.15.187:0/2225201750 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d009f560 msgr2=0x7fd9d014a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9c67fc700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d009f560 0x7fd9d014a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9c67fc700 1 -- 172.21.15.187:0/2225201750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9d01588f0 msgr2=0x7fd9d0141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9cf7fe700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9d01588f0 0x7fd9d0141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9c67fc700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9d01588f0 0x7fd9d0141780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.285+0000 7fd9c67fc700 1 -- 172.21.15.187:0/2225201750 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d00b4600 con 0x7fd9d00bf800 2020-11-20T12:05:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.286+0000 7fd9c67fc700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d0143240 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fd9b800d870 tx=0x7fd9b8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.286+0000 7fd9c77fe700 1 -- 172.21.15.187:0/2225201750 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b80096a0 con 0x7fd9d00bf800 2020-11-20T12:05:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.286+0000 7fd9c77fe700 1 -- 172.21.15.187:0/2225201750 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b8009800 con 0x7fd9d00bf800 2020-11-20T12:05:02.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.286+0000 7fd9c77fe700 1 -- 172.21.15.187:0/2225201750 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8009ba0 con 0x7fd9d00bf800 2020-11-20T12:05:02.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.286+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d014a570 con 0x7fd9d00bf800 2020-11-20T12:05:02.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.286+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d014b790 con 0x7fd9d00bf800 2020-11-20T12:05:02.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.287+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d0080050 con 0x7fd9d00bf800 2020-11-20T12:05:02.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.288+0000 7fd9c77fe700 1 -- 172.21.15.187:0/2225201750 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd9b8013480 con 0x7fd9d00bf800 2020-11-20T12:05:02.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.288+0000 7fd9c77fe700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9ac056d20 0x7fd9ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.288+0000 7fd9c77fe700 1 -- 172.21.15.187:0/2225201750 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd9b80703e0 con 0x7fd9d00bf800 2020-11-20T12:05:02.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.289+0000 7fd9ceffd700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9ac056d20 0x7fd9ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.289+0000 7fd9ceffd700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9ac056d20 0x7fd9ac0591e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd9c0009fd0 tx=0x7fd9c0016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.292+0000 7fd9c77fe700 1 -- 172.21.15.187:0/2225201750 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd9b80456c0 con 0x7fd9d00bf800 2020-11-20T12:05:02.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.478+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd9d014c7b0 con 0x7fd9d00bf800 2020-11-20T12:05:02.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.479+0000 7fd9c77fe700 1 -- 172.21.15.187:0/2225201750 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd9b8048ce0 con 0x7fd9d00bf800 2020-11-20T12:05:02.483 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:02.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9ac056d20 msgr2=0x7fd9ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9ac056d20 0x7fd9ac0591e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd9c0009fd0 tx=0x7fd9c0016000).stop 2020-11-20T12:05:02.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 msgr2=0x7fd9d0143240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d0143240 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fd9b800d870 tx=0x7fd9b8008040).stop 2020-11-20T12:05:02.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 shutdown_connections 2020-11-20T12:05:02.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9ac056d20 0x7fd9ac0591e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9d01588f0 0x7fd9d0141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9d009f560 0x7fd9d014a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 --2- 172.21.15.187:0/2225201750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9d00bf800 0x7fd9d0143240 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 >> 172.21.15.187:0/2225201750 conn(0x7fd9d0068370 msgr2=0x7fd9d00be110 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.481+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 shutdown_connections 2020-11-20T12:05:02.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.482+0000 7fd9cffff700 1 -- 172.21.15.187:0/2225201750 wait complete. 2020-11-20T12:05:02.510 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:02.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.745+0000 7f9c7c644700 1 Processor -- start 2020-11-20T12:05:02.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.745+0000 7f9c7c644700 1 -- start start 2020-11-20T12:05:02.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.747+0000 7f9c7c644700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c740b73b0 0x7f9c74066e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.747+0000 7f9c7c644700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9c74067300 0x7f9c74144d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.747+0000 7f9c7c644700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c74146010 con 0x7f9c740b6fd0 2020-11-20T12:05:02.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.747+0000 7f9c7c644700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c741462b0 con 0x7f9c74067300 2020-11-20T12:05:02.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.747+0000 7f9c7c644700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74146550 con 0x7f9c740b73b0 2020-11-20T12:05:02.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.747+0000 7f9c7a3e0700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9c740b6fd0 0x7f9c74066700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37036/0 (socket says 172.21.15.187:37036) 2020-11-20T12:05:02.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.747+0000 7f9c7a3e0700 1 -- 172.21.15.187:0/2394837527 learned_addr learned my addr 172.21.15.187:0/2394837527 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:02.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.748+0000 7f9c79bdf700 1 --2- 172.21.15.187:0/2394837527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c740b73b0 0x7f9c74066e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.748+0000 7f9c7abe1700 1 --2- 172.21.15.187:0/2394837527 >> v2:172.21.15.5:3300/0 conn(0x7f9c74067300 0x7f9c74144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.748+0000 7f9c793de700 1 -- 172.21.15.187:0/2394837527 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367898724 0 0) 0x7f9c74146010 con 0x7f9c740b6fd0 2020-11-20T12:05:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.748+0000 7f9c793de700 1 -- 172.21.15.187:0/2394837527 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c003240 con 0x7f9c740b6fd0 2020-11-20T12:05:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.749+0000 7f9c7abe1700 1 -- 172.21.15.187:0/2394837527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c740b73b0 msgr2=0x7f9c74066e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.749+0000 7f9c7abe1700 1 --2- 172.21.15.187:0/2394837527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c740b73b0 0x7f9c74066e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.749+0000 7f9c7abe1700 1 -- 172.21.15.187:0/2394837527 >> v1:172.21.15.187:6789/0 conn(0x7f9c740b6fd0 legacy=0x7f9c74066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.749+0000 7f9c7abe1700 1 -- 172.21.15.187:0/2394837527 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c740b6650 con 0x7f9c74067300 2020-11-20T12:05:02.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.749+0000 7f9c7abe1700 1 --2- 172.21.15.187:0/2394837527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c74067300 0x7f9c74144d90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9c6c00b9a0 tx=0x7f9c6c006df0).ready entity=mon.0 client_cookie=71ffd3965d3aea2f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c793de700 1 -- 172.21.15.187:0/2394837527 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c00f770 con 0x7f9c74067300 2020-11-20T12:05:02.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c793de700 1 -- 172.21.15.187:0/2394837527 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c00f8d0 con 0x7f9c74067300 2020-11-20T12:05:02.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c793de700 1 -- 172.21.15.187:0/2394837527 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c00fc50 con 0x7f9c74067300 2020-11-20T12:05:02.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c7c644700 1 -- 172.21.15.187:0/2394837527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c74067300 msgr2=0x7f9c74144d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c7c644700 1 --2- 172.21.15.187:0/2394837527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c74067300 0x7f9c74144d90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9c6c00b9a0 tx=0x7f9c6c006df0).stop 2020-11-20T12:05:02.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c7c644700 1 -- 172.21.15.187:0/2394837527 shutdown_connections 2020-11-20T12:05:02.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c7c644700 1 --2- 172.21.15.187:0/2394837527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c74067300 0x7f9c74144d90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c7c644700 1 --2- 172.21.15.187:0/2394837527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c740b73b0 0x7f9c74066e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c7c644700 1 -- 172.21.15.187:0/2394837527 >> 172.21.15.187:0/2394837527 conn(0x7f9c740bacf0 msgr2=0x7f9c740bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c7c644700 1 -- 172.21.15.187:0/2394837527 shutdown_connections 2020-11-20T12:05:02.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.750+0000 7f9c7c644700 1 -- 172.21.15.187:0/2394837527 wait complete. 2020-11-20T12:05:02.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.752+0000 7f9c7c644700 1 Processor -- start 2020-11-20T12:05:02.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.752+0000 7f9c7c644700 1 -- start start 2020-11-20T12:05:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.753+0000 7f9c7c644700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c74067300 0x7f9c7415b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c7c644700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c740b6fd0 0x7f9c740a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c7c644700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c740b73b0 0x7f9c7409c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c7c644700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7409b690 con 0x7f9c740b73b0 2020-11-20T12:05:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c7c644700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7409b930 con 0x7f9c740b6fd0 2020-11-20T12:05:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c7c644700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7409bbd0 con 0x7f9c74067300 2020-11-20T12:05:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c79bdf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c740b6fd0 0x7f9c740a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c79bdf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c740b6fd0 0x7f9c740a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40438/0 (socket says 172.21.15.187:40438) 2020-11-20T12:05:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c79bdf700 1 -- 172.21.15.187:0/3396053315 learned_addr learned my addr 172.21.15.187:0/3396053315 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:02.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c7abe1700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c740b73b0 0x7f9c7409c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c79bdf700 1 -- 172.21.15.187:0/3396053315 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c74067300 msgr2=0x7f9c7415b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c79bdf700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c74067300 0x7f9c7415b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:02.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c79bdf700 1 -- 172.21.15.187:0/3396053315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c740b73b0 msgr2=0x7f9c7409c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c79bdf700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c740b73b0 0x7f9c7409c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.754+0000 7f9c79bdf700 1 -- 172.21.15.187:0/3396053315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7415bd70 con 0x7f9c740b6fd0 2020-11-20T12:05:02.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.755+0000 7f9c79bdf700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c740b6fd0 0x7f9c740a0510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f9c6400ebd0 tx=0x7f9c64006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.755+0000 7f9c627fc700 1 -- 172.21.15.187:0/3396053315 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6400b9a0 con 0x7f9c740b6fd0 2020-11-20T12:05:02.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.755+0000 7f9c627fc700 1 -- 172.21.15.187:0/3396053315 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6400bb00 con 0x7f9c740b6fd0 2020-11-20T12:05:02.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.755+0000 7f9c627fc700 1 -- 172.21.15.187:0/3396053315 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6400be80 con 0x7f9c740b6fd0 2020-11-20T12:05:02.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.755+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7409dca0 con 0x7f9c740b6fd0 2020-11-20T12:05:02.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.755+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7409d630 con 0x7f9c740b6fd0 2020-11-20T12:05:02.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.756+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c74044f60 con 0x7f9c740b6fd0 2020-11-20T12:05:02.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.757+0000 7f9c627fc700 1 -- 172.21.15.187:0/3396053315 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9c64012600 con 0x7f9c740b6fd0 2020-11-20T12:05:02.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.758+0000 7f9c627fc700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c4c056d20 0x7f9c4c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.758+0000 7f9c627fc700 1 -- 172.21.15.187:0/3396053315 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9c64070300 con 0x7f9c740b6fd0 2020-11-20T12:05:02.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.758+0000 7f9c7a3e0700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c4c056d20 0x7f9c4c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.758+0000 7f9c7a3e0700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c4c056d20 0x7f9c4c0591e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9c70005920 tx=0x7f9c70007a20).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.761+0000 7f9c627fc700 1 -- 172.21.15.187:0/3396053315 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9c64045750 con 0x7f9c740b6fd0 2020-11-20T12:05:02.895 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:05:02.891+0000 7fb372f3c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:05:02.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.946+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c74068690 con 0x7f9c740b6fd0 2020-11-20T12:05:03.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.947+0000 7f9c627fc700 1 -- 172.21.15.187:0/3396053315 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9c64048d70 con 0x7f9c740b6fd0 2020-11-20T12:05:03.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c4c056d20 msgr2=0x7f9c4c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c4c056d20 0x7f9c4c0591e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9c70005920 tx=0x7f9c70007a20).stop 2020-11-20T12:05:03.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c740b6fd0 msgr2=0x7f9c740a0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c740b6fd0 0x7f9c740a0510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f9c6400ebd0 tx=0x7f9c64006df0).stop 2020-11-20T12:05:03.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 shutdown_connections 2020-11-20T12:05:03.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c4c056d20 0x7f9c4c0591e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c740b73b0 0x7f9c7409c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c74067300 0x7f9c7415b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:03.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 --2- 172.21.15.187:0/3396053315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c740b6fd0 0x7f9c740a0510 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 >> 172.21.15.187:0/3396053315 conn(0x7f9c740bacf0 msgr2=0x7f9c740afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.949+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 shutdown_connections 2020-11-20T12:05:03.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:02.950+0000 7f9c7c644700 1 -- 172.21.15.187:0/3396053315 wait complete. 2020-11-20T12:05:03.335 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:03.335 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:03.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.587+0000 7f31a1f36700 1 Processor -- start 2020-11-20T12:05:03.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.587+0000 7f31a1f36700 1 -- start start 2020-11-20T12:05:03.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.589+0000 7f31a1f36700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c065190 0x7f319c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.589+0000 7f31a1f36700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f319c065f90 0x7f319c143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.589+0000 7f31a1f36700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f319c0b13e0 con 0x7f319c143c70 2020-11-20T12:05:03.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.589+0000 7f31a1f36700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319c14ab30 con 0x7f319c065f90 2020-11-20T12:05:03.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.589+0000 7f31a1f36700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c062c10 con 0x7f319c065190 2020-11-20T12:05:03.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.589+0000 7f319bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f319c065f90 0x7f319c143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.590+0000 7f319bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f319c065f90 0x7f319c143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40446/0 (socket says 172.21.15.187:40446) 2020-11-20T12:05:03.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.590+0000 7f319bfff700 1 -- 172.21.15.187:0/3877273679 learned_addr learned my addr 172.21.15.187:0/3877273679 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:03.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.590+0000 7f31a0f34700 1 --2- 172.21.15.187:0/3877273679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c065190 0x7f319c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.590+0000 7f319bfff700 1 -- 172.21.15.187:0/3877273679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c065190 msgr2=0x7f319c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:03.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.590+0000 7f319bfff700 1 --2- 172.21.15.187:0/3877273679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c065190 0x7f319c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.590+0000 7f319bfff700 1 -- 172.21.15.187:0/3877273679 >> v1:172.21.15.5:6789/0 conn(0x7f319c143c70 legacy=0x7f319c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.590+0000 7f319bfff700 1 -- 172.21.15.187:0/3877273679 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c0b1640 con 0x7f319c065f90 2020-11-20T12:05:03.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.591+0000 7f319bfff700 1 --2- 172.21.15.187:0/3877273679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065f90 0x7f319c143780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f319c14ac70 tx=0x7f3188006eb0).ready entity=mon.1 client_cookie=3bf2bee78bdc6635 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.591+0000 7f319b7fe700 1 -- 172.21.15.187:0/3877273679 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31880098a0 con 0x7f319c065f90 2020-11-20T12:05:03.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.591+0000 7f319b7fe700 1 -- 172.21.15.187:0/3877273679 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3188009a00 con 0x7f319c065f90 2020-11-20T12:05:03.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.591+0000 7f319b7fe700 1 -- 172.21.15.187:0/3877273679 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188009d80 con 0x7f319c065f90 2020-11-20T12:05:03.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.592+0000 7f31a1f36700 1 -- 172.21.15.187:0/3877273679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065f90 msgr2=0x7f319c143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.592+0000 7f31a1f36700 1 --2- 172.21.15.187:0/3877273679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065f90 0x7f319c143780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f319c14ac70 tx=0x7f3188006eb0).stop 2020-11-20T12:05:03.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.592+0000 7f31a1f36700 1 -- 172.21.15.187:0/3877273679 shutdown_connections 2020-11-20T12:05:03.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.592+0000 7f31a1f36700 1 --2- 172.21.15.187:0/3877273679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c065190 0x7f319c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.592+0000 7f31a1f36700 1 --2- 172.21.15.187:0/3877273679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065f90 0x7f319c143780 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.592+0000 7f31a1f36700 1 -- 172.21.15.187:0/3877273679 >> 172.21.15.187:0/3877273679 conn(0x7f319c0bab30 msgr2=0x7f319c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.592+0000 7f31a1f36700 1 -- 172.21.15.187:0/3877273679 shutdown_connections 2020-11-20T12:05:03.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.592+0000 7f31a1f36700 1 -- 172.21.15.187:0/3877273679 wait complete. 2020-11-20T12:05:03.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.594+0000 7f31a1f36700 1 Processor -- start 2020-11-20T12:05:03.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.594+0000 7f31a1f36700 1 -- start start 2020-11-20T12:05:03.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.595+0000 7f31a1f36700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065190 0x7f319c0990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a1f36700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f319c065f90 0x7f319c15c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a1f36700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c143c70 0x7f319c0ba140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a1f36700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c15cb00 con 0x7f319c065f90 2020-11-20T12:05:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a1f36700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c09f6e0 con 0x7f319c065190 2020-11-20T12:05:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a1f36700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c09f980 con 0x7f319c143c70 2020-11-20T12:05:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a0f34700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065190 0x7f319c0990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a0f34700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065190 0x7f319c0990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40450/0 (socket says 172.21.15.187:40450) 2020-11-20T12:05:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a0f34700 1 -- 172.21.15.187:0/2464147933 learned_addr learned my addr 172.21.15.187:0/2464147933 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a0f34700 1 -- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c143c70 msgr2=0x7f319c0ba140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f319bfff700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f319c065f90 0x7f319c15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a1735700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c143c70 0x7f319c0ba140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a0f34700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c143c70 0x7f319c0ba140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a0f34700 1 -- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f319c065f90 msgr2=0x7f319c15c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a0f34700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f319c065f90 0x7f319c15c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f31a0f34700 1 -- 172.21.15.187:0/2464147933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c09fc20 con 0x7f319c065190 2020-11-20T12:05:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.596+0000 7f319bfff700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f319c065f90 0x7f319c15c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:03.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.597+0000 7f31a0f34700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065190 0x7f319c0990a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f319000eba0 tx=0x7f3190006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.597+0000 7f3198ff9700 1 -- 172.21.15.187:0/2464147933 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319000ca60 con 0x7f319c065190 2020-11-20T12:05:03.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.597+0000 7f3198ff9700 1 -- 172.21.15.187:0/2464147933 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319000cbc0 con 0x7f319c065190 2020-11-20T12:05:03.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.597+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c0ba630 con 0x7f319c065190 2020-11-20T12:05:03.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.597+0000 7f3198ff9700 1 -- 172.21.15.187:0/2464147933 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31900123f0 con 0x7f319c065190 2020-11-20T12:05:03.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.597+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c0a0350 con 0x7f319c065190 2020-11-20T12:05:03.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.598+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c061a20 con 0x7f319c065190 2020-11-20T12:05:03.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.599+0000 7f3198ff9700 1 -- 172.21.15.187:0/2464147933 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f31900127a0 con 0x7f319c065190 2020-11-20T12:05:03.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.599+0000 7f3198ff9700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3174056d20 0x7f31740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.600+0000 7f3198ff9700 1 -- 172.21.15.187:0/2464147933 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3190070450 con 0x7f319c065190 2020-11-20T12:05:03.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.600+0000 7f319bfff700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3174056d20 0x7f31740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.600+0000 7f319bfff700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3174056d20 0x7f31740591e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f319c14ac70 tx=0x7f3188015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.603+0000 7f3198ff9700 1 -- 172.21.15.187:0/2464147933 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f31900458a0 con 0x7f319c065190 2020-11-20T12:05:03.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.793+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f319c0afc60 con 0x7f319c065190 2020-11-20T12:05:03.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.794+0000 7f3198ff9700 1 -- 172.21.15.187:0/2464147933 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3190048ec0 con 0x7f319c065190 2020-11-20T12:05:03.799 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:03.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3174056d20 msgr2=0x7f31740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3174056d20 0x7f31740591e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f319c14ac70 tx=0x7f3188015000).stop 2020-11-20T12:05:03.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065190 msgr2=0x7f319c0990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065190 0x7f319c0990a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f319000eba0 tx=0x7f3190006df0).stop 2020-11-20T12:05:03.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 shutdown_connections 2020-11-20T12:05:03.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3174056d20 0x7f31740591e0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0x7f319c14ac70 tx=0x7f3188015000).stop 2020-11-20T12:05:03.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f319c143c70 0x7f319c0ba140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f319c065190 0x7f319c0990a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 --2- 172.21.15.187:0/2464147933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f319c065f90 0x7f319c15c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 >> 172.21.15.187:0/2464147933 conn(0x7f319c0bab30 msgr2=0x7f319c0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 shutdown_connections 2020-11-20T12:05:03.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:03.797+0000 7f31a1f36700 1 -- 172.21.15.187:0/2464147933 wait complete. 2020-11-20T12:05:03.825 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2020-11-20T12:05:03.825 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2020-11-20T12:05:03.826 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2020-11-20T12:05:04.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.068+0000 7f4d67442700 1 Processor -- start 2020-11-20T12:05:04.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.068+0000 7f4d67442700 1 -- start start 2020-11-20T12:05:04.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.069+0000 7f4d67442700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d600bf800 0x7f4d6014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.069+0000 7f4d67442700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d600bfbe0 0x7f4d601480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.069+0000 7f4d67442700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6014d110 con 0x7f4d6009f560 2020-11-20T12:05:04.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.069+0000 7f4d67442700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6014cbd0 con 0x7f4d600bfbe0 2020-11-20T12:05:04.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.069+0000 7f4d67442700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014ce70 con 0x7f4d600bf800 2020-11-20T12:05:04.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.070+0000 7f4d66c41700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4d6009f560 0x7f4d60060c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37068/0 (socket says 172.21.15.187:37068) 2020-11-20T12:05:04.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.070+0000 7f4d66c41700 1 -- 172.21.15.187:0/3251246272 learned_addr learned my addr 172.21.15.187:0/3251246272 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:04.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.070+0000 7f4d65c3f700 1 --2- 172.21.15.187:0/3251246272 >> v2:172.21.15.5:3300/0 conn(0x7f4d600bfbe0 0x7f4d601480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.070+0000 7f4d66440700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d600bf800 0x7f4d6014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.070+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379888111 0 0) 0x7f4d6014d110 con 0x7f4d6009f560 2020-11-20T12:05:04.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.070+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d48003240 con 0x7f4d6009f560 2020-11-20T12:05:04.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1703548948 0 0) 0x7f4d48003240 con 0x7f4d6009f560 2020-11-20T12:05:04.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d600bf800 msgr2=0x7f4d6014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d600bf800 0x7f4d6014c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 >> v2:172.21.15.5:3300/0 conn(0x7f4d600bfbe0 msgr2=0x7f4d601480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 --2- 172.21.15.187:0/3251246272 >> v2:172.21.15.5:3300/0 conn(0x7f4d600bfbe0 0x7f4d601480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600a4cc0 con 0x7f4d6009f560 2020-11-20T12:05:04.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4d580014e0 con 0x7f4d6009f560 2020-11-20T12:05:04.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 >> v1:172.21.15.187:6789/0 conn(0x7f4d6009f560 legacy=0x7f4d60060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d600bf800 0x7f4d48003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d48003c40 0x7f4d480044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d480049d0 0x7f4d48004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014cbd0 con 0x7f4d480049d0 2020-11-20T12:05:04.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014ce70 con 0x7f4d48003c40 2020-11-20T12:05:04.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d48004200 con 0x7f4d600bf800 2020-11-20T12:05:04.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d65c3f700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d48003c40 0x7f4d480044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d66440700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d600bf800 0x7f4d48003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.071+0000 7f4d66c41700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d480049d0 0x7f4d48004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d65c3f700 1 -- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d600bf800 msgr2=0x7f4d48003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d65c3f700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d600bf800 0x7f4d48003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d65c3f700 1 -- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d480049d0 msgr2=0x7f4d48004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d65c3f700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d480049d0 0x7f4d48004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d65c3f700 1 -- 172.21.15.187:0/3251246272 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d48003240 con 0x7f4d48003c40 2020-11-20T12:05:04.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d65c3f700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d48003c40 0x7f4d480044e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4d600bcaa0 tx=0x7f4d50008040).ready entity=mon.1 client_cookie=47a478d6df620196 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5000d8f0 con 0x7f4d48003c40 2020-11-20T12:05:04.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5000da50 con 0x7f4d48003c40 2020-11-20T12:05:04.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.072+0000 7f4d6543e700 1 -- 172.21.15.187:0/3251246272 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5000ddd0 con 0x7f4d48003c40 2020-11-20T12:05:04.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.073+0000 7f4d67442700 1 -- 172.21.15.187:0/3251246272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d48003c40 msgr2=0x7f4d480044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.073+0000 7f4d67442700 1 --2- 172.21.15.187:0/3251246272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d48003c40 0x7f4d480044e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4d600bcaa0 tx=0x7f4d50008040).stop 2020-11-20T12:05:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.073+0000 7f4d66c41700 1 -- 172.21.15.187:0/3251246272 reap_dead start 2020-11-20T12:05:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.073+0000 7f4d67442700 1 -- 172.21.15.187:0/3251246272 shutdown_connections 2020-11-20T12:05:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.073+0000 7f4d67442700 1 -- 172.21.15.187:0/3251246272 >> 172.21.15.187:0/3251246272 conn(0x7f4d60068370 msgr2=0x7f4d600bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.073+0000 7f4d67442700 1 -- 172.21.15.187:0/3251246272 shutdown_connections 2020-11-20T12:05:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.073+0000 7f4d67442700 1 -- 172.21.15.187:0/3251246272 wait complete. 2020-11-20T12:05:04.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.074+0000 7f4d67442700 1 Processor -- start 2020-11-20T12:05:04.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.074+0000 7f4d67442700 1 -- start start 2020-11-20T12:05:04.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.075+0000 7f4d67442700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d60149b10 0x7f4d60145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.075+0000 7f4d67442700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d60146c80 0x7f4d60149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.075+0000 7f4d67442700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d6013ffd0 0x7f4d60140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.075+0000 7f4d67442700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60145750 con 0x7f4d6013ffd0 2020-11-20T12:05:04.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.075+0000 7f4d67442700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014a770 con 0x7f4d60149b10 2020-11-20T12:05:04.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.075+0000 7f4d67442700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014aa10 con 0x7f4d60146c80 2020-11-20T12:05:04.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.075+0000 7f4d66440700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d60149b10 0x7f4d60145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66440700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d60149b10 0x7f4d60145bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40470/0 (socket says 172.21.15.187:40470) 2020-11-20T12:05:04.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66440700 1 -- 172.21.15.187:0/3389377941 learned_addr learned my addr 172.21.15.187:0/3389377941 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:04.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d65c3f700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d60146c80 0x7f4d60149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66c41700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d6013ffd0 0x7f4d60140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66440700 1 -- 172.21.15.187:0/3389377941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d60146c80 msgr2=0x7f4d60149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66440700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d60146c80 0x7f4d60149ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66440700 1 -- 172.21.15.187:0/3389377941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d6013ffd0 msgr2=0x7f4d60140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66440700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d6013ffd0 0x7f4d60140410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66440700 1 -- 172.21.15.187:0/3389377941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d601460a0 con 0x7f4d60149b10 2020-11-20T12:05:04.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d66440700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d60149b10 0x7f4d60145bb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4d5c010d40 tx=0x7f4d5c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d4e7fc700 1 -- 172.21.15.187:0/3389377941 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c00ebd0 con 0x7f4d60149b10 2020-11-20T12:05:04.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d4e7fc700 1 -- 172.21.15.187:0/3389377941 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c00c030 con 0x7f4d60149b10 2020-11-20T12:05:04.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.076+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d60140900 con 0x7f4d60149b10 2020-11-20T12:05:04.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.077+0000 7f4d4e7fc700 1 -- 172.21.15.187:0/3389377941 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c014410 con 0x7f4d60149b10 2020-11-20T12:05:04.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.077+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6014a3e0 con 0x7f4d60149b10 2020-11-20T12:05:04.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.078+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d60080050 con 0x7f4d60149b10 2020-11-20T12:05:04.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.078+0000 7f4d4e7fc700 1 -- 172.21.15.187:0/3389377941 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4d5c012040 con 0x7f4d60149b10 2020-11-20T12:05:04.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.079+0000 7f4d4e7fc700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d38056d50 0x7f4d38059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.079+0000 7f4d4e7fc700 1 -- 172.21.15.187:0/3389377941 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4d5c0705f0 con 0x7f4d60149b10 2020-11-20T12:05:04.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.080+0000 7f4d65c3f700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d38056d50 0x7f4d38059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.080+0000 7f4d65c3f700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d38056d50 0x7f4d38059210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4d600bcaa0 tx=0x7f4d50008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.083+0000 7f4d4e7fc700 1 -- 172.21.15.187:0/3389377941 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4d5c045ac0 con 0x7f4d60149b10 2020-11-20T12:05:04.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.313+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4d601485b0 con 0x7f4d60149b10 2020-11-20T12:05:04.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.320+0000 7f4d4e7fc700 1 -- 172.21.15.187:0/3389377941 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+135029 (secure 0 0 0) 0x7f4d5c018020 con 0x7f4d60149b10 2020-11-20T12:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:04.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.324+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d38056d50 msgr2=0x7f4d38059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.324+0000 7f4d67442700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d38056d50 0x7f4d38059210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4d600bcaa0 tx=0x7f4d50008040).stop 2020-11-20T12:05:04.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.324+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d60149b10 msgr2=0x7f4d60145bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.324+0000 7f4d67442700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d60149b10 0x7f4d60145bb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4d5c010d40 tx=0x7f4d5c008040).stop 2020-11-20T12:05:04.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.325+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 shutdown_connections 2020-11-20T12:05:04.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.325+0000 7f4d67442700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d38056d50 0x7f4d38059210 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.325+0000 7f4d67442700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d6013ffd0 0x7f4d60140410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.325+0000 7f4d67442700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d60149b10 0x7f4d60145bb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.325+0000 7f4d67442700 1 --2- 172.21.15.187:0/3389377941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d60146c80 0x7f4d60149ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.325+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 >> 172.21.15.187:0/3389377941 conn(0x7f4d60068370 msgr2=0x7f4d60060c90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.325+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 shutdown_connections 2020-11-20T12:05:04.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.325+0000 7f4d67442700 1 -- 172.21.15.187:0/3389377941 wait complete. 2020-11-20T12:05:04.353 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":5,"active_gid":4102,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":19649},{"type":"v1","addr":"172.21.15.187:6801","nonce":19649}]},"active_addr":"172.21.15.187:6801/19649","active_change":"2020-11-20T12:04:43.506836+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":4110,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":2672140334}]},{"addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":3613460136}]},{"addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":3592923556}]}]}} 2020-11-20T12:05:04.358 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2020-11-20T12:05:04.358 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-11-20T12:05:04.359 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:04.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.588+0000 7f3b67fff700 1 Processor -- start 2020-11-20T12:05:04.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.588+0000 7f3b67fff700 1 -- start start 2020-11-20T12:05:04.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.589+0000 7f3b67fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68066f70 0x7f3b680b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.589+0000 7f3b67fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b680b1300 con 0x7f3b680b6650 2020-11-20T12:05:04.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.589+0000 7f3b67fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b680b15a0 con 0x7f3b681484d0 2020-11-20T12:05:04.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.589+0000 7f3b67fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68067730 con 0x7f3b68066f70 2020-11-20T12:05:04.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.590+0000 7f3b66ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3b681484d0 0x7f3b68061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37086/0 (socket says 172.21.15.187:37086) 2020-11-20T12:05:04.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.590+0000 7f3b66ffd700 1 -- 172.21.15.187:0/95297478 learned_addr learned my addr 172.21.15.187:0/95297478 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:04.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.590+0000 7f3b677fe700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68066f70 0x7f3b680b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441043656 0 0) 0x7f3b680b15a0 con 0x7f3b681484d0 2020-11-20T12:05:04.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b4c003240 con 0x7f3b681484d0 2020-11-20T12:05:04.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3218197603 0 0) 0x7f3b4c003240 con 0x7f3b681484d0 2020-11-20T12:05:04.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68066f70 msgr2=0x7f3b680b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68066f70 0x7f3b680b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 >> v1:172.21.15.5:6789/0 conn(0x7f3b680b6650 legacy=0x7f3b680afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b680b1840 con 0x7f3b681484d0 2020-11-20T12:05:04.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f3b58001520 con 0x7f3b681484d0 2020-11-20T12:05:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 >> v1:172.21.15.187:6789/0 conn(0x7f3b681484d0 legacy=0x7f3b68061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68066f70 0x7f3b4c003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b4c003c40 0x7f3b4c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b4c0049d0 0x7f3b4c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68067730 con 0x7f3b4c0049d0 2020-11-20T12:05:04.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c0041d0 con 0x7f3b4c003c40 2020-11-20T12:05:04.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.591+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c00fb40 con 0x7f3b68066f70 2020-11-20T12:05:04.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.592+0000 7f3b5e7fc700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b4c003c40 0x7f3b4c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.592+0000 7f3b677fe700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b4c0049d0 0x7f3b4c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.592+0000 7f3b66ffd700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68066f70 0x7f3b4c003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.592+0000 7f3b5e7fc700 1 -- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68066f70 msgr2=0x7f3b4c003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.592+0000 7f3b5e7fc700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68066f70 0x7f3b4c003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.592+0000 7f3b5e7fc700 1 -- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b4c0049d0 msgr2=0x7f3b4c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.593+0000 7f3b5e7fc700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b4c0049d0 0x7f3b4c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.593+0000 7f3b5e7fc700 1 -- 172.21.15.187:0/95297478 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c003240 con 0x7f3b4c003c40 2020-11-20T12:05:04.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.593+0000 7f3b677fe700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b4c0049d0 0x7f3b4c004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:04.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.593+0000 7f3b5e7fc700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b4c003c40 0x7f3b4c0044e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f3b50001920 tx=0x7f3b50007b90).ready entity=mon.1 client_cookie=52f1d3e9395919ad server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.593+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50007020 con 0x7f3b4c003c40 2020-11-20T12:05:04.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.593+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b50007180 con 0x7f3b4c003c40 2020-11-20T12:05:04.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.593+0000 7f3b667fc700 1 -- 172.21.15.187:0/95297478 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5000c5c0 con 0x7f3b4c003c40 2020-11-20T12:05:04.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.594+0000 7f3b67fff700 1 -- 172.21.15.187:0/95297478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b4c003c40 msgr2=0x7f3b4c0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.594+0000 7f3b67fff700 1 --2- 172.21.15.187:0/95297478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b4c003c40 0x7f3b4c0044e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f3b50001920 tx=0x7f3b50007b90).stop 2020-11-20T12:05:04.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.594+0000 7f3b677fe700 1 -- 172.21.15.187:0/95297478 reap_dead start 2020-11-20T12:05:04.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.594+0000 7f3b67fff700 1 -- 172.21.15.187:0/95297478 shutdown_connections 2020-11-20T12:05:04.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.594+0000 7f3b67fff700 1 -- 172.21.15.187:0/95297478 >> 172.21.15.187:0/95297478 conn(0x7f3b680bab30 msgr2=0x7f3b680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.594+0000 7f3b67fff700 1 -- 172.21.15.187:0/95297478 shutdown_connections 2020-11-20T12:05:04.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.594+0000 7f3b67fff700 1 -- 172.21.15.187:0/95297478 wait complete. 2020-11-20T12:05:04.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.595+0000 7f3b67fff700 1 Processor -- start 2020-11-20T12:05:04.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.595+0000 7f3b67fff700 1 -- start start 2020-11-20T12:05:04.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.596+0000 7f3b67fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68147210 0x7f3b680b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.596+0000 7f3b67fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b681475f0 0x7f3b681572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.596+0000 7f3b67fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b6809fb60 0x7f3b68136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.596+0000 7f3b67fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680a0000 con 0x7f3b6809fb60 2020-11-20T12:05:04.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.596+0000 7f3b67fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680a02a0 con 0x7f3b681475f0 2020-11-20T12:05:04.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.596+0000 7f3b67fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680a0540 con 0x7f3b68147210 2020-11-20T12:05:04.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b681475f0 0x7f3b681572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b681475f0 0x7f3b681572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40492/0 (socket says 172.21.15.187:40492) 2020-11-20T12:05:04.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 -- 172.21.15.187:0/3797000609 learned_addr learned my addr 172.21.15.187:0/3797000609 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:04.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b66ffd700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68147210 0x7f3b680b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b677fe700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b6809fb60 0x7f3b68136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 -- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68147210 msgr2=0x7f3b680b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68147210 0x7f3b680b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 -- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b6809fb60 msgr2=0x7f3b68136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b6809fb60 0x7f3b68136aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 -- 172.21.15.187:0/3797000609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6809b210 con 0x7f3b681475f0 2020-11-20T12:05:04.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b677fe700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b6809fb60 0x7f3b68136aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:04.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.597+0000 7f3b5e7fc700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b681475f0 0x7f3b681572a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3b50010000 tx=0x7f3b50015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.598+0000 7f3b5f7fe700 1 -- 172.21.15.187:0/3797000609 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50007540 con 0x7f3b681475f0 2020-11-20T12:05:04.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.598+0000 7f3b5f7fe700 1 -- 172.21.15.187:0/3797000609 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b50013a50 con 0x7f3b681475f0 2020-11-20T12:05:04.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.598+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b68136f90 con 0x7f3b681475f0 2020-11-20T12:05:04.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.598+0000 7f3b5f7fe700 1 -- 172.21.15.187:0/3797000609 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50013df0 con 0x7f3b681475f0 2020-11-20T12:05:04.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.598+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b681374d0 con 0x7f3b681475f0 2020-11-20T12:05:04.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.599+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b680a0700 con 0x7f3b681475f0 2020-11-20T12:05:04.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.600+0000 7f3b5f7fe700 1 -- 172.21.15.187:0/3797000609 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3b50026020 con 0x7f3b681475f0 2020-11-20T12:05:04.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.600+0000 7f3b5f7fe700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b38056d20 0x7f3b380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.600+0000 7f3b5f7fe700 1 -- 172.21.15.187:0/3797000609 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3b500713b0 con 0x7f3b681475f0 2020-11-20T12:05:04.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.601+0000 7f3b66ffd700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b38056d20 0x7f3b380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.601+0000 7f3b66ffd700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b38056d20 0x7f3b380591e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f3b580013f0 tx=0x7f3b58009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.605+0000 7f3b5f7fe700 1 -- 172.21.15.187:0/3797000609 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3b50046800 con 0x7f3b681475f0 2020-11-20T12:05:04.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.789+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b68061a20 con 0x7f3b681475f0 2020-11-20T12:05:04.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.791+0000 7f3b5f7fe700 1 -- 172.21.15.187:0/3797000609 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f3b50007720 con 0x7f3b681475f0 2020-11-20T12:05:04.796 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:04.796 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:04.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.793+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b38056d20 msgr2=0x7f3b380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.793+0000 7f3b67fff700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b38056d20 0x7f3b380591e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f3b580013f0 tx=0x7f3b58009040).stop 2020-11-20T12:05:04.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b681475f0 msgr2=0x7f3b681572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b681475f0 0x7f3b681572a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3b50010000 tx=0x7f3b50015000).stop 2020-11-20T12:05:04.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 shutdown_connections 2020-11-20T12:05:04.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b38056d20 0x7f3b380591e0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0x7f3b580013f0 tx=0x7f3b58009040).stop 2020-11-20T12:05:04.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b6809fb60 0x7f3b68136aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b68147210 0x7f3b680b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 --2- 172.21.15.187:0/3797000609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b681475f0 0x7f3b681572a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 >> 172.21.15.187:0/3797000609 conn(0x7f3b680bab30 msgr2=0x7f3b680b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 shutdown_connections 2020-11-20T12:05:04.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:04.794+0000 7f3b67fff700 1 -- 172.21.15.187:0/3797000609 wait complete. 2020-11-20T12:05:04.822 INFO:tasks.ceph.ceph_manager.ceph:all up! 2020-11-20T12:05:04.823 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:05.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.060+0000 7f9026821700 1 Processor -- start 2020-11-20T12:05:05.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.060+0000 7f9026821700 1 -- start start 2020-11-20T12:05:05.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.061+0000 7f9026821700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f90200bfbe0 0x7f90201480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.061+0000 7f9026821700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902009f560 0x7f90201485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.061+0000 7f9026821700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902014d110 con 0x7f90200bf800 2020-11-20T12:05:05.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.061+0000 7f9026821700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902014c9d0 con 0x7f90200bfbe0 2020-11-20T12:05:05.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.061+0000 7f9026821700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9020148aa0 con 0x7f902009f560 2020-11-20T12:05:05.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.062+0000 7f902581f700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f90200bf800 0x7f902014c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37104/0 (socket says 172.21.15.187:37104) 2020-11-20T12:05:05.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.062+0000 7f902581f700 1 -- 172.21.15.187:0/871841757 learned_addr learned my addr 172.21.15.187:0/871841757 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:05.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.062+0000 7f902501e700 1 --2- 172.21.15.187:0/871841757 >> v2:172.21.15.5:3300/0 conn(0x7f90200bfbe0 0x7f90201480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.062+0000 7f9026020700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902009f560 0x7f90201485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.062+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149874920 0 0) 0x7f902014d110 con 0x7f90200bf800 2020-11-20T12:05:05.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.062+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9008003240 con 0x7f90200bf800 2020-11-20T12:05:05.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 695754408 0 0) 0x7f9008003240 con 0x7f90200bf800 2020-11-20T12:05:05.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902009f560 msgr2=0x7f90201485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:05.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902009f560 0x7f90201485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 >> v2:172.21.15.5:3300/0 conn(0x7f90200bfbe0 msgr2=0x7f90201480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:05.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 --2- 172.21.15.187:0/871841757 >> v2:172.21.15.5:3300/0 conn(0x7f90200bfbe0 0x7f90201480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902014cbd0 con 0x7f90200bf800 2020-11-20T12:05:05.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f901c0014e0 con 0x7f90200bf800 2020-11-20T12:05:05.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 >> v1:172.21.15.187:6789/0 conn(0x7f90200bf800 legacy=0x7f902014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90080037b0 0x7f9008003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90080044e0 0x7f9008004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902009f560 0x7f9008004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902014c9d0 con 0x7f90080037b0 2020-11-20T12:05:05.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9020148aa0 con 0x7f90080044e0 2020-11-20T12:05:05.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9008005340 con 0x7f902009f560 2020-11-20T12:05:05.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902501e700 1 --2- 172.21.15.187:0/871841757 >> v2:172.21.15.5:3300/0 conn(0x7f90200bfbe0 0x7f90201480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:05.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f902501e700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90080044e0 0x7f9008004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.063+0000 7f9026020700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902009f560 0x7f9008004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902581f700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90080037b0 0x7f9008003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902501e700 1 -- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902009f560 msgr2=0x7f9008004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:05.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902501e700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902009f560 0x7f9008004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902501e700 1 -- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90080037b0 msgr2=0x7f9008003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:05.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902501e700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90080037b0 0x7f9008003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902501e700 1 -- 172.21.15.187:0/871841757 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9008003240 con 0x7f90080044e0 2020-11-20T12:05:05.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902501e700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90080044e0 0x7f9008004960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f901400bb30 tx=0x7f9014008040).ready entity=mon.1 client_cookie=b74397060fd482e8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901400b190 con 0x7f90080044e0 2020-11-20T12:05:05.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901400d9c0 con 0x7f90080044e0 2020-11-20T12:05:05.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.064+0000 7f902481d700 1 -- 172.21.15.187:0/871841757 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901400dd40 con 0x7f90080044e0 2020-11-20T12:05:05.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.065+0000 7f9026821700 1 -- 172.21.15.187:0/871841757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90080044e0 msgr2=0x7f9008004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.065+0000 7f9026821700 1 --2- 172.21.15.187:0/871841757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90080044e0 0x7f9008004960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f901400bb30 tx=0x7f9014008040).stop 2020-11-20T12:05:05.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.065+0000 7f9026020700 1 -- 172.21.15.187:0/871841757 reap_dead start 2020-11-20T12:05:05.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.065+0000 7f9026821700 1 -- 172.21.15.187:0/871841757 shutdown_connections 2020-11-20T12:05:05.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.065+0000 7f9026821700 1 -- 172.21.15.187:0/871841757 >> 172.21.15.187:0/871841757 conn(0x7f9020068370 msgr2=0x7f90200bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.065+0000 7f9026821700 1 -- 172.21.15.187:0/871841757 shutdown_connections 2020-11-20T12:05:05.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.065+0000 7f9026821700 1 -- 172.21.15.187:0/871841757 wait complete. 2020-11-20T12:05:05.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.066+0000 7f9026821700 1 Processor -- start 2020-11-20T12:05:05.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.066+0000 7f9026821700 1 -- start start 2020-11-20T12:05:05.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.067+0000 7f9026821700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90080044e0 0x7f90200b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.067+0000 7f9026821700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9020144680 0x7f902014aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.067+0000 7f9026821700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9020140ab0 0x7f9020140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.067+0000 7f9026821700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9020141420 con 0x7f9020140ab0 2020-11-20T12:05:05.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.067+0000 7f9026821700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90200b6840 con 0x7f9020144680 2020-11-20T12:05:05.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.067+0000 7f9026821700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90200b5bd0 con 0x7f90080044e0 2020-11-20T12:05:05.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9020144680 0x7f902014aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9020144680 0x7f902014aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40510/0 (socket says 172.21.15.187:40510) 2020-11-20T12:05:05.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 -- 172.21.15.187:0/1934068803 learned_addr learned my addr 172.21.15.187:0/1934068803 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:05.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f9026020700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9020140ab0 0x7f9020140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902581f700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90080044e0 0x7f90200b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 -- 172.21.15.187:0/1934068803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90080044e0 msgr2=0x7f90200b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90080044e0 0x7f90200b56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 -- 172.21.15.187:0/1934068803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9020140ab0 msgr2=0x7f9020140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9020140ab0 0x7f9020140f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 -- 172.21.15.187:0/1934068803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90200b5e60 con 0x7f9020144680 2020-11-20T12:05:05.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.068+0000 7f902501e700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9020144680 0x7f902014aef0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9008004410 tx=0x7f9014008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.069+0000 7f9011ffb700 1 -- 172.21.15.187:0/1934068803 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90140067c0 con 0x7f9020144680 2020-11-20T12:05:05.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.069+0000 7f9011ffb700 1 -- 172.21.15.187:0/1934068803 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014005c10 con 0x7f9020144680 2020-11-20T12:05:05.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.069+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90200b6100 con 0x7f9020144680 2020-11-20T12:05:05.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.069+0000 7f9011ffb700 1 -- 172.21.15.187:0/1934068803 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014012630 con 0x7f9020144680 2020-11-20T12:05:05.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.069+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902014b3e0 con 0x7f9020144680 2020-11-20T12:05:05.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.070+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9020080050 con 0x7f9020144680 2020-11-20T12:05:05.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.070+0000 7f9011ffb700 1 -- 172.21.15.187:0/1934068803 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f901400b700 con 0x7f9020144680 2020-11-20T12:05:05.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.071+0000 7f9011ffb700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ff8056d20 0x7f8ff80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.071+0000 7f9011ffb700 1 -- 172.21.15.187:0/1934068803 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9014072090 con 0x7f9020144680 2020-11-20T12:05:05.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.071+0000 7f902581f700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ff8056d20 0x7f8ff80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.072+0000 7f902581f700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ff8056d20 0x7f8ff80591e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f901c0056a0 tx=0x7f902009ec30).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.075+0000 7f9011ffb700 1 -- 172.21.15.187:0/1934068803 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9014047560 con 0x7f9020144680 2020-11-20T12:05:05.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.259+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90200a4cc0 con 0x7f9020144680 2020-11-20T12:05:05.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.261+0000 7f9011ffb700 1 -- 172.21.15.187:0/1934068803 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f901404ab80 con 0x7f9020144680 2020-11-20T12:05:05.269 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:05.269 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:05.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ff8056d20 msgr2=0x7f8ff80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ff8056d20 0x7f8ff80591e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f901c0056a0 tx=0x7f902009ec30).stop 2020-11-20T12:05:05.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9020144680 msgr2=0x7f902014aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9020144680 0x7f902014aef0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9008004410 tx=0x7f9014008040).stop 2020-11-20T12:05:05.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 shutdown_connections 2020-11-20T12:05:05.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ff8056d20 0x7f8ff80591e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9020140ab0 0x7f9020140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90080044e0 0x7f90200b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 --2- 172.21.15.187:0/1934068803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9020144680 0x7f902014aef0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 >> 172.21.15.187:0/1934068803 conn(0x7f9020068370 msgr2=0x7f902014bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 shutdown_connections 2020-11-20T12:05:05.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.264+0000 7f9026821700 1 -- 172.21.15.187:0/1934068803 wait complete. 2020-11-20T12:05:05.290 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2020-11-20T12:05:05.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.532+0000 7f126b1ca700 1 Processor -- start 2020-11-20T12:05:05.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.532+0000 7f126b1ca700 1 -- start start 2020-11-20T12:05:05.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.533+0000 7f126b1ca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f12641484d0 0x7f1264061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.534+0000 7f126b1ca700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b1300 con 0x7f1264066f70 2020-11-20T12:05:05.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.534+0000 7f126b1ca700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b15a0 con 0x7f12640622c0 2020-11-20T12:05:05.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.534+0000 7f126b1ca700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1264067730 con 0x7f12641484d0 2020-11-20T12:05:05.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.534+0000 7f126a1c8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f12641484d0 0x7f1264061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.534+0000 7f12699c7700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f12640622c0 0x7f12640b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37126/0 (socket says 172.21.15.187:37126) 2020-11-20T12:05:05.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.534+0000 7f12699c7700 1 -- 172.21.15.187:0/3491838862 learned_addr learned my addr 172.21.15.187:0/3491838862 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:05.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f12691c6700 1 -- 172.21.15.187:0/3491838862 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645995017 0 0) 0x7f12640b15a0 con 0x7f12640622c0 2020-11-20T12:05:05.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f12691c6700 1 -- 172.21.15.187:0/3491838862 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f124c003240 con 0x7f12640622c0 2020-11-20T12:05:05.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f126a1c8700 1 -- 172.21.15.187:0/3491838862 >> v1:172.21.15.187:6789/0 conn(0x7f12640622c0 legacy=0x7f12640b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f126a1c8700 1 -- 172.21.15.187:0/3491838862 >> v1:172.21.15.5:6789/0 conn(0x7f1264066f70 legacy=0x7f12640b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:05.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f126a1c8700 1 -- 172.21.15.187:0/3491838862 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640b1840 con 0x7f12641484d0 2020-11-20T12:05:05.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f126a1c8700 1 --2- 172.21.15.187:0/3491838862 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12641484d0 0x7f1264061dd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f126000b9a0 tx=0x7f1260006df0).ready entity=mon.1 client_cookie=80ff43c5e979f773 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f12691c6700 1 -- 172.21.15.187:0/3491838862 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12600097b0 con 0x7f12641484d0 2020-11-20T12:05:05.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f12691c6700 1 -- 172.21.15.187:0/3491838862 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260009910 con 0x7f12641484d0 2020-11-20T12:05:05.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.535+0000 7f12691c6700 1 -- 172.21.15.187:0/3491838862 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260009c90 con 0x7f12641484d0 2020-11-20T12:05:05.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.536+0000 7f126b1ca700 1 -- 172.21.15.187:0/3491838862 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12641484d0 msgr2=0x7f1264061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.536+0000 7f126b1ca700 1 --2- 172.21.15.187:0/3491838862 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12641484d0 0x7f1264061dd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f126000b9a0 tx=0x7f1260006df0).stop 2020-11-20T12:05:05.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.536+0000 7f126b1ca700 1 -- 172.21.15.187:0/3491838862 shutdown_connections 2020-11-20T12:05:05.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.536+0000 7f126b1ca700 1 --2- 172.21.15.187:0/3491838862 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12641484d0 0x7f1264061dd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.536+0000 7f126b1ca700 1 -- 172.21.15.187:0/3491838862 >> 172.21.15.187:0/3491838862 conn(0x7f12640bab30 msgr2=0x7f12640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.536+0000 7f126b1ca700 1 -- 172.21.15.187:0/3491838862 shutdown_connections 2020-11-20T12:05:05.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.536+0000 7f126b1ca700 1 -- 172.21.15.187:0/3491838862 wait complete. 2020-11-20T12:05:05.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.537+0000 7f126b1ca700 1 Processor -- start 2020-11-20T12:05:05.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.538+0000 7f126b1ca700 1 -- start start 2020-11-20T12:05:05.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.538+0000 7f126b1ca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12640622c0 0x7f126409fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.538+0000 7f126b1ca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1264066f70 0x7f12640b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.538+0000 7f126b1ca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12640b7bf0 0x7f1264151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126b1ca700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12641582d0 con 0x7f1264066f70 2020-11-20T12:05:05.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126b1ca700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126409a5f0 con 0x7f12640b7bf0 2020-11-20T12:05:05.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126b1ca700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126409a820 con 0x7f12640622c0 2020-11-20T12:05:05.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a9c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12640b7bf0 0x7f1264151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a9c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12640b7bf0 0x7f1264151ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40528/0 (socket says 172.21.15.187:40528) 2020-11-20T12:05:05.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a9c9700 1 -- 172.21.15.187:0/572996184 learned_addr learned my addr 172.21.15.187:0/572996184 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:05.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f12699c7700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1264066f70 0x7f12640b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a1c8700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12640622c0 0x7f126409fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a9c9700 1 -- 172.21.15.187:0/572996184 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12640622c0 msgr2=0x7f126409fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a9c9700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12640622c0 0x7f126409fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a9c9700 1 -- 172.21.15.187:0/572996184 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1264066f70 msgr2=0x7f12640b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a9c9700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1264066f70 0x7f12640b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a9c9700 1 -- 172.21.15.187:0/572996184 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126409aac0 con 0x7f12640b7bf0 2020-11-20T12:05:05.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f12699c7700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1264066f70 0x7f12640b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:05.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.539+0000 7f126a1c8700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12640622c0 0x7f126409fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:05.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.540+0000 7f126a9c9700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12640b7bf0 0x7f1264151ec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f125c00cac0 tx=0x7f125c007f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.540+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c00a730 con 0x7f12640b7bf0 2020-11-20T12:05:05.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.540+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c00a890 con 0x7f12640b7bf0 2020-11-20T12:05:05.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.540+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126409ad60 con 0x7f12640b7bf0 2020-11-20T12:05:05.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.540+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c00ac10 con 0x7f12640b7bf0 2020-11-20T12:05:05.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.540+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12640b80f0 con 0x7f12640b7bf0 2020-11-20T12:05:05.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.541+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1264061a20 con 0x7f12640b7bf0 2020-11-20T12:05:05.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.542+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f125c019030 con 0x7f12640b7bf0 2020-11-20T12:05:05.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.542+0000 7f12527fc700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.542+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f125c06f8b0 con 0x7f12640b7bf0 2020-11-20T12:05:05.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.543+0000 7f12527fc700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f123c05e270 0x7f123c060690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.543+0000 7f126a1c8700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.543+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f123c060cf0 con 0x7f123c05e270 2020-11-20T12:05:05.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.543+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f125c071db0 con 0x7f12640b7bf0 2020-11-20T12:05:05.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.543+0000 7f12699c7700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f123c05e270 0x7f123c060690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.543+0000 7f126a1c8700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1264066540 tx=0x7f1260015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.544+0000 7f12699c7700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f123c05e270 0x7f123c060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.545+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 <== osd.0 v2:172.21.15.187:6802/19903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f123c060cf0 con 0x7f123c05e270 2020-11-20T12:05:05.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.562+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1264152670 con 0x7f123c05e270 2020-11-20T12:05:05.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.563+0000 7f12527fc700 1 -- 172.21.15.187:0/572996184 <== osd.0 v2:172.21.15.187:6802/19903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1264152670 con 0x7f123c05e270 2020-11-20T12:05:05.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f123c05e270 msgr2=0x7f123c060690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f123c05e270 0x7f123c060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 msgr2=0x7f123c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1264066540 tx=0x7f1260015000).stop 2020-11-20T12:05:05.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12640b7bf0 msgr2=0x7f1264151ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 --2- 172.21.15.187:0/572996184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12640b7bf0 0x7f1264151ec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f125c00cac0 tx=0x7f125c007f70).stop 2020-11-20T12:05:05.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126a9c9700 1 -- 172.21.15.187:0/572996184 reap_dead start 2020-11-20T12:05:05.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 shutdown_connections 2020-11-20T12:05:05.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 >> 172.21.15.187:0/572996184 conn(0x7f12640bab30 msgr2=0x7f12640b6010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 shutdown_connections 2020-11-20T12:05:05.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.564+0000 7f126b1ca700 1 -- 172.21.15.187:0/572996184 wait complete. 2020-11-20T12:05:05.591 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:05.592 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-11-20T12:05:05.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.829+0000 7f8c0d4d2700 1 Processor -- start 2020-11-20T12:05:05.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.829+0000 7f8c0d4d2700 1 -- start start 2020-11-20T12:05:05.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.830+0000 7f8c0d4d2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8c080bf800 0x7f8c0814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.830+0000 7f8c0d4d2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0814d110 con 0x7f8c0809f560 2020-11-20T12:05:05.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.830+0000 7f8c0d4d2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0814cbd0 con 0x7f8c080bfbe0 2020-11-20T12:05:05.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.830+0000 7f8c0d4d2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0814ce70 con 0x7f8c080bf800 2020-11-20T12:05:05.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.831+0000 7f8c07fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8c080bf800 0x7f8c0814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.831+0000 7f8bff7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8c080bfbe0 0x7f8c081480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37142/0 (socket says 172.21.15.187:37142) 2020-11-20T12:05:05.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.831+0000 7f8bff7fe700 1 -- 172.21.15.187:0/2670349050 learned_addr learned my addr 172.21.15.187:0/2670349050 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:05.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.831+0000 7f8c077fe700 1 -- 172.21.15.187:0/2670349050 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194941959 0 0) 0x7f8c0814cbd0 con 0x7f8c080bfbe0 2020-11-20T12:05:05.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.831+0000 7f8c077fe700 1 -- 172.21.15.187:0/2670349050 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be8003240 con 0x7f8c080bfbe0 2020-11-20T12:05:05.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.831+0000 7f8c07fff700 1 -- 172.21.15.187:0/2670349050 >> v1:172.21.15.187:6789/0 conn(0x7f8c080bfbe0 legacy=0x7f8c081480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.831+0000 7f8c07fff700 1 -- 172.21.15.187:0/2670349050 >> v1:172.21.15.5:6789/0 conn(0x7f8c0809f560 legacy=0x7f8c080a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:05.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.832+0000 7f8c07fff700 1 -- 172.21.15.187:0/2670349050 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c080a4cc0 con 0x7f8c080bf800 2020-11-20T12:05:05.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.832+0000 7f8c07fff700 1 --2- 172.21.15.187:0/2670349050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c080bf800 0x7f8c0814c2c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8bf800b9a0 tx=0x7f8bf8006df0).ready entity=mon.1 client_cookie=a047da8c199d8211 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.832+0000 7f8c077fe700 1 -- 172.21.15.187:0/2670349050 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8009800 con 0x7f8c080bf800 2020-11-20T12:05:05.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.832+0000 7f8c077fe700 1 -- 172.21.15.187:0/2670349050 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf8009960 con 0x7f8c080bf800 2020-11-20T12:05:05.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.832+0000 7f8c077fe700 1 -- 172.21.15.187:0/2670349050 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8009ce0 con 0x7f8c080bf800 2020-11-20T12:05:05.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.833+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2670349050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c080bf800 msgr2=0x7f8c0814c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.833+0000 7f8c0d4d2700 1 --2- 172.21.15.187:0/2670349050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c080bf800 0x7f8c0814c2c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8bf800b9a0 tx=0x7f8bf8006df0).stop 2020-11-20T12:05:05.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.833+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2670349050 shutdown_connections 2020-11-20T12:05:05.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.833+0000 7f8c0d4d2700 1 --2- 172.21.15.187:0/2670349050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c080bf800 0x7f8c0814c2c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.833+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2670349050 >> 172.21.15.187:0/2670349050 conn(0x7f8c08068370 msgr2=0x7f8c080bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.833+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2670349050 shutdown_connections 2020-11-20T12:05:05.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.833+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2670349050 wait complete. 2020-11-20T12:05:05.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.834+0000 7f8c0d4d2700 1 Processor -- start 2020-11-20T12:05:05.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.835+0000 7f8c0d4d2700 1 -- start start 2020-11-20T12:05:05.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.835+0000 7f8c0d4d2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c0809f560 0x7f8c080ba690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.835+0000 7f8c0d4d2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c080bf800 0x7f8c081460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.835+0000 7f8c0d4d2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c08143bf0 0x7f8c08141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.835+0000 7f8c0d4d2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c08144050 con 0x7f8c0809f560 2020-11-20T12:05:05.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.835+0000 7f8c0d4d2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080bada0 con 0x7f8c08143bf0 2020-11-20T12:05:05.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.835+0000 7f8c0d4d2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080bb040 con 0x7f8c080bf800 2020-11-20T12:05:05.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c08143bf0 0x7f8c08141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c08143bf0 0x7f8c08141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40544/0 (socket says 172.21.15.187:40544) 2020-11-20T12:05:05.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 -- 172.21.15.187:0/2215419136 learned_addr learned my addr 172.21.15.187:0/2215419136 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:05.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 -- 172.21.15.187:0/2215419136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c080bf800 msgr2=0x7f8c081460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c080bf800 0x7f8c081460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:05.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 -- 172.21.15.187:0/2215419136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c0809f560 msgr2=0x7f8c080ba690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c07fff700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c0809f560 0x7f8c080ba690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c0809f560 0x7f8c080ba690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 -- 172.21.15.187:0/2215419136 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0815c620 con 0x7f8c08143bf0 2020-11-20T12:05:05.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.836+0000 7f8c0ccd1700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c08143bf0 0x7f8c08141000 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8c0000c860 tx=0x7f8c0000cc20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.837+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0000a840 con 0x7f8c08143bf0 2020-11-20T12:05:05.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.837+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0000a9c0 con 0x7f8c08143bf0 2020-11-20T12:05:05.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.837+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0815cac0 con 0x7f8c08143bf0 2020-11-20T12:05:05.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.837+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0000ad80 con 0x7f8c08143bf0 2020-11-20T12:05:05.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.837+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0815bc00 con 0x7f8c08143bf0 2020-11-20T12:05:05.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.838+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c080bac00 con 0x7f8c08143bf0 2020-11-20T12:05:05.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.839+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8c00010070 con 0x7f8c08143bf0 2020-11-20T12:05:05.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.839+0000 7f8c04ff9700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8be4056d20 0x7f8be40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.839+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8c0006fea0 con 0x7f8c08143bf0 2020-11-20T12:05:05.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.840+0000 7f8c07fff700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8be4056d20 0x7f8be40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.840+0000 7f8c04ff9700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f8be405e270 0x7f8be4060690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.840+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8be4060cf0 con 0x7f8be405e270 2020-11-20T12:05:05.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.840+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c00072510 con 0x7f8c08143bf0 2020-11-20T12:05:05.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.840+0000 7f8bff7fe700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f8be405e270 0x7f8be4060690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.840+0000 7f8c07fff700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8be4056d20 0x7f8be40591e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f8bf8006e30 tx=0x7f8bf8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.841+0000 7f8bff7fe700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f8be405e270 0x7f8be4060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.842+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f8be4060cf0 con 0x7f8be405e270 2020-11-20T12:05:05.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.859+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c08080050 con 0x7f8be405e270 2020-11-20T12:05:05.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.860+0000 7f8c04ff9700 1 -- 172.21.15.187:0/2215419136 <== osd.1 v2:172.21.15.187:6818/19920 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8c08080050 con 0x7f8be405e270 2020-11-20T12:05:05.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.860+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f8be405e270 msgr2=0x7f8be4060690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.860+0000 7f8c0d4d2700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f8be405e270 0x7f8be4060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8be4056d20 msgr2=0x7f8be40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0d4d2700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8be4056d20 0x7f8be40591e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f8bf8006e30 tx=0x7f8bf8015000).stop 2020-11-20T12:05:05.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c08143bf0 msgr2=0x7f8c08141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0d4d2700 1 --2- 172.21.15.187:0/2215419136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c08143bf0 0x7f8c08141000 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8c0000c860 tx=0x7f8c0000cc20).stop 2020-11-20T12:05:05.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0ccd1700 1 -- 172.21.15.187:0/2215419136 reap_dead start 2020-11-20T12:05:05.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 shutdown_connections 2020-11-20T12:05:05.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 >> 172.21.15.187:0/2215419136 conn(0x7f8c08068370 msgr2=0x7f8c080bbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 shutdown_connections 2020-11-20T12:05:05.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:05.861+0000 7f8c0d4d2700 1 -- 172.21.15.187:0/2215419136 wait complete. 2020-11-20T12:05:05.884 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:05.885 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-11-20T12:05:06.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.118+0000 7f0dba7d6700 1 Processor -- start 2020-11-20T12:05:06.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.118+0000 7f0dba7d6700 1 -- start start 2020-11-20T12:05:06.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.119+0000 7f0dba7d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db40bfbe0 0x7f0db41480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.120+0000 7f0dba7d6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0db409f560 0x7f0db41485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.120+0000 7f0dba7d6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db414d110 con 0x7f0db40bf800 2020-11-20T12:05:06.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.120+0000 7f0dba7d6700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db414c9d0 con 0x7f0db409f560 2020-11-20T12:05:06.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.120+0000 7f0dba7d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db4148aa0 con 0x7f0db40bfbe0 2020-11-20T12:05:06.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.120+0000 7f0db9fd5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0db409f560 0x7f0db41485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.120+0000 7f0db9fd5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0db409f560 0x7f0db41485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40554/0 (socket says 172.21.15.187:40554) 2020-11-20T12:05:06.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.120+0000 7f0db9fd5700 1 -- 172.21.15.187:0/1744226354 learned_addr learned my addr 172.21.15.187:0/1744226354 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:06.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.120+0000 7f0db8fd3700 1 --2- 172.21.15.187:0/1744226354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db40bfbe0 0x7f0db41480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.121+0000 7f0db9fd5700 1 -- 172.21.15.187:0/1744226354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db40bfbe0 msgr2=0x7f0db41480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.121+0000 7f0db9fd5700 1 --2- 172.21.15.187:0/1744226354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db40bfbe0 0x7f0db41480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.121+0000 7f0db9fd5700 1 -- 172.21.15.187:0/1744226354 >> v1:172.21.15.5:6789/0 conn(0x7f0db40bf800 legacy=0x7f0db414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.121+0000 7f0db9fd5700 1 -- 172.21.15.187:0/1744226354 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db414cbd0 con 0x7f0db409f560 2020-11-20T12:05:06.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.121+0000 7f0db9fd5700 1 --2- 172.21.15.187:0/1744226354 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db409f560 0x7f0db41485b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0dac009d80 tx=0x7f0dac009290).ready entity=mon.1 client_cookie=838f6d97ec8c86c8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.121+0000 7f0da3fff700 1 -- 172.21.15.187:0/1744226354 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00bcd0 con 0x7f0db409f560 2020-11-20T12:05:06.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.121+0000 7f0da3fff700 1 -- 172.21.15.187:0/1744226354 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac00be30 con 0x7f0db409f560 2020-11-20T12:05:06.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.121+0000 7f0da3fff700 1 -- 172.21.15.187:0/1744226354 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac011610 con 0x7f0db409f560 2020-11-20T12:05:06.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.122+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1744226354 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db409f560 msgr2=0x7f0db41485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.122+0000 7f0dba7d6700 1 --2- 172.21.15.187:0/1744226354 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db409f560 0x7f0db41485b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0dac009d80 tx=0x7f0dac009290).stop 2020-11-20T12:05:06.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.122+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1744226354 shutdown_connections 2020-11-20T12:05:06.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.122+0000 7f0dba7d6700 1 --2- 172.21.15.187:0/1744226354 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db409f560 0x7f0db41485b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.122+0000 7f0dba7d6700 1 --2- 172.21.15.187:0/1744226354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db40bfbe0 0x7f0db41480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.122+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1744226354 >> 172.21.15.187:0/1744226354 conn(0x7f0db4068370 msgr2=0x7f0db40bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.122+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1744226354 shutdown_connections 2020-11-20T12:05:06.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.122+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1744226354 wait complete. 2020-11-20T12:05:06.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.124+0000 7f0dba7d6700 1 Processor -- start 2020-11-20T12:05:06.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.124+0000 7f0dba7d6700 1 -- start start 2020-11-20T12:05:06.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0dba7d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db409f560 0x7f0db4146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0dba7d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db40bf800 0x7f0db40ba930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0dba7d6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0db40ba3c0 0x7f0db415bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0dba7d6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db415c930 con 0x7f0db40ba3c0 2020-11-20T12:05:06.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0dba7d6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db415cbd0 con 0x7f0db40bf800 2020-11-20T12:05:06.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0dba7d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db415c3c0 con 0x7f0db409f560 2020-11-20T12:05:06.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0db8fd3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db40bf800 0x7f0db40ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0db8fd3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db40bf800 0x7f0db40ba930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40560/0 (socket says 172.21.15.187:40560) 2020-11-20T12:05:06.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0db8fd3700 1 -- 172.21.15.187:0/1849323269 learned_addr learned my addr 172.21.15.187:0/1849323269 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:06.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0db97d4700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db409f560 0x7f0db4146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.125+0000 7f0db9fd5700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0db40ba3c0 0x7f0db415bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0db8fd3700 1 -- 172.21.15.187:0/1849323269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db409f560 msgr2=0x7f0db4146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0db8fd3700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0db409f560 0x7f0db4146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0db8fd3700 1 -- 172.21.15.187:0/1849323269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0db40ba3c0 msgr2=0x7f0db415bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0db8fd3700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0db40ba3c0 0x7f0db415bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0db8fd3700 1 -- 172.21.15.187:0/1849323269 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db415c580 con 0x7f0db40bf800 2020-11-20T12:05:06.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0db8fd3700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db40bf800 0x7f0db40ba930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0da400d8e0 tx=0x7f0da4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4009790 con 0x7f0db40bf800 2020-11-20T12:05:06.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da40098f0 con 0x7f0db40bf800 2020-11-20T12:05:06.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db4140b00 con 0x7f0db40bf800 2020-11-20T12:05:06.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4009c90 con 0x7f0db40bf800 2020-11-20T12:05:06.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.126+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db4141040 con 0x7f0db40bf800 2020-11-20T12:05:06.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.127+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0db40bae20 con 0x7f0db40bf800 2020-11-20T12:05:06.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.128+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0da4013480 con 0x7f0db40bf800 2020-11-20T12:05:06.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.129+0000 7f0da17fa700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d8c056d20 0x7f0d8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.129+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0da406feb0 con 0x7f0db40bf800 2020-11-20T12:05:06.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.129+0000 7f0db97d4700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d8c056d20 0x7f0d8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.129+0000 7f0da17fa700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f0d8c05e270 0x7f0d8c060690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.129+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d8c060cf0 con 0x7f0d8c05e270 2020-11-20T12:05:06.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.129+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0da4072520 con 0x7f0db40bf800 2020-11-20T12:05:06.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.129+0000 7f0db9fd5700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f0d8c05e270 0x7f0d8c060690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.130+0000 7f0db97d4700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d8c056d20 0x7f0d8c0591e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0db0003510 tx=0x7f0db0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.130+0000 7f0db9fd5700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f0d8c05e270 0x7f0d8c060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.131+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f0d8c060cf0 con 0x7f0d8c05e270 2020-11-20T12:05:06.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.149+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0db40a3400 con 0x7f0d8c05e270 2020-11-20T12:05:06.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.150+0000 7f0da17fa700 1 -- 172.21.15.187:0/1849323269 <== osd.2 v2:172.21.15.187:6810/19904 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0db40a3400 con 0x7f0d8c05e270 2020-11-20T12:05:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.150+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f0d8c05e270 msgr2=0x7f0d8c060690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.150+0000 7f0dba7d6700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f0d8c05e270 0x7f0d8c060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.150+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d8c056d20 msgr2=0x7f0d8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.150+0000 7f0dba7d6700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d8c056d20 0x7f0d8c0591e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0db0003510 tx=0x7f0db0009040).stop 2020-11-20T12:05:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.150+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db40bf800 msgr2=0x7f0db40ba930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.150+0000 7f0dba7d6700 1 --2- 172.21.15.187:0/1849323269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0db40bf800 0x7f0db40ba930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0da400d8e0 tx=0x7f0da4008040).stop 2020-11-20T12:05:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.150+0000 7f0db9fd5700 1 -- 172.21.15.187:0/1849323269 reap_dead start 2020-11-20T12:05:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.151+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 shutdown_connections 2020-11-20T12:05:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.151+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 >> 172.21.15.187:0/1849323269 conn(0x7f0db4068370 msgr2=0x7f0db4141c20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.151+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 shutdown_connections 2020-11-20T12:05:06.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.151+0000 7f0dba7d6700 1 -- 172.21.15.187:0/1849323269 wait complete. 2020-11-20T12:05:06.176 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:06.177 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2020-11-20T12:05:06.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.414+0000 7fbc263d8700 1 Processor -- start 2020-11-20T12:05:06.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.414+0000 7fbc263d8700 1 -- start start 2020-11-20T12:05:06.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.416+0000 7fbc263d8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbc2014d270 0x7fbc201573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.416+0000 7fbc263d8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc200658a0 con 0x7fbc200b1380 2020-11-20T12:05:06.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.416+0000 7fbc263d8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc20150390 con 0x7fbc200b1760 2020-11-20T12:05:06.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.416+0000 7fbc263d8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc201511e0 con 0x7fbc2014d270 2020-11-20T12:05:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.416+0000 7fbc24bd5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbc200b1760 0x7fbc20065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37174/0 (socket says 172.21.15.187:37174) 2020-11-20T12:05:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.416+0000 7fbc24bd5700 1 -- 172.21.15.187:0/2105089079 learned_addr learned my addr 172.21.15.187:0/2105089079 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.416+0000 7fbc25bd7700 1 --2- 172.21.15.187:0/2105089079 >> v2:172.21.15.5:3300/0 conn(0x7fbc2014d270 0x7fbc201573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237910880 0 0) 0x7fbc20150390 con 0x7fbc200b1760 2020-11-20T12:05:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c003240 con 0x7fbc200b1760 2020-11-20T12:05:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4197226120 0 0) 0x7fbc0c003240 con 0x7fbc200b1760 2020-11-20T12:05:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 >> v2:172.21.15.5:3300/0 conn(0x7fbc2014d270 msgr2=0x7fbc201573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2105089079 >> v2:172.21.15.5:3300/0 conn(0x7fbc2014d270 0x7fbc201573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 >> v1:172.21.15.5:6789/0 conn(0x7fbc200b1380 legacy=0x7fbc20150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc20150630 con 0x7fbc200b1760 2020-11-20T12:05:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fbc140014e0 con 0x7fbc200b1760 2020-11-20T12:05:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 >> v1:172.21.15.187:6789/0 conn(0x7fbc200b1760 legacy=0x7fbc20065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c003730 0x7fbc0c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c0044e0 0x7fbc0c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc0c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc201511e0 con 0x7fbc0c0044e0 2020-11-20T12:05:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c004260 con 0x7fbc0c004e50 2020-11-20T12:05:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.417+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c00bc20 con 0x7fbc0c003730 2020-11-20T12:05:06.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.418+0000 7fbc25bd7700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc0c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.418+0000 7fbc24bd5700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c0044e0 0x7fbc0c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.418+0000 7fbc253d6700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c003730 0x7fbc0c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.418+0000 7fbc25bd7700 1 -- 172.21.15.187:0/2105089079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c003730 msgr2=0x7fbc0c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.418+0000 7fbc25bd7700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c003730 0x7fbc0c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.418+0000 7fbc25bd7700 1 -- 172.21.15.187:0/2105089079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c0044e0 msgr2=0x7fbc0c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.418+0000 7fbc25bd7700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c0044e0 0x7fbc0c004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.418+0000 7fbc25bd7700 1 -- 172.21.15.187:0/2105089079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c003240 con 0x7fbc0c004e50 2020-11-20T12:05:06.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.419+0000 7fbc25bd7700 1 -- 172.21.15.187:0/2105089079 reap_dead start 2020-11-20T12:05:06.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.419+0000 7fbc25bd7700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc0c00b730 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fbc1800bc00 tx=0x7fbc18006df0).ready entity=mon.1 client_cookie=645cf43493d723de server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.419+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc18016070 con 0x7fbc0c004e50 2020-11-20T12:05:06.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.419+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1800b310 con 0x7fbc0c004e50 2020-11-20T12:05:06.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.419+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2105089079 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1800ed50 con 0x7fbc0c004e50 2020-11-20T12:05:06.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.420+0000 7fbc263d8700 1 -- 172.21.15.187:0/2105089079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 msgr2=0x7fbc0c00b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.420+0000 7fbc263d8700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc0c00b730 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fbc1800bc00 tx=0x7fbc18006df0).stop 2020-11-20T12:05:06.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.420+0000 7fbc263d8700 1 -- 172.21.15.187:0/2105089079 shutdown_connections 2020-11-20T12:05:06.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.420+0000 7fbc263d8700 1 --2- 172.21.15.187:0/2105089079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc0c00b730 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.420+0000 7fbc263d8700 1 -- 172.21.15.187:0/2105089079 >> 172.21.15.187:0/2105089079 conn(0x7fbc200bab30 msgr2=0x7fbc200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.421+0000 7fbc263d8700 1 -- 172.21.15.187:0/2105089079 shutdown_connections 2020-11-20T12:05:06.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.421+0000 7fbc263d8700 1 -- 172.21.15.187:0/2105089079 wait complete. 2020-11-20T12:05:06.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.423+0000 7fbc263d8700 1 Processor -- start 2020-11-20T12:05:06.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.423+0000 7fbc263d8700 1 -- start start 2020-11-20T12:05:06.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.424+0000 7fbc263d8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc2015c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.424+0000 7fbc263d8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc201604f0 0x7fbc20145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.424+0000 7fbc263d8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc200b2280 0x7fbc200b9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.424+0000 7fbc263d8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2015c8d0 con 0x7fbc200b2280 2020-11-20T12:05:06.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc263d8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2015cb70 con 0x7fbc0c004e50 2020-11-20T12:05:06.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc263d8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2015ce10 con 0x7fbc201604f0 2020-11-20T12:05:06.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc253d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc2015c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc253d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc2015c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40578/0 (socket says 172.21.15.187:40578) 2020-11-20T12:05:06.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc253d6700 1 -- 172.21.15.187:0/1131844018 learned_addr learned my addr 172.21.15.187:0/1131844018 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:06.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc24bd5700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc201604f0 0x7fbc20145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc25bd7700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc200b2280 0x7fbc200b9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc253d6700 1 -- 172.21.15.187:0/1131844018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc201604f0 msgr2=0x7fbc20145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc253d6700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc201604f0 0x7fbc20145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc253d6700 1 -- 172.21.15.187:0/1131844018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc200b2280 msgr2=0x7fbc200b9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc253d6700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc200b2280 0x7fbc200b9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.425+0000 7fbc253d6700 1 -- 172.21.15.187:0/1131844018 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc20146420 con 0x7fbc0c004e50 2020-11-20T12:05:06.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.426+0000 7fbc253d6700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc2015c290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fbc1000ea80 tx=0x7fbc10009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.426+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10003320 con 0x7fbc0c004e50 2020-11-20T12:05:06.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.426+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc10010040 con 0x7fbc0c004e50 2020-11-20T12:05:06.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.426+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2014a340 con 0x7fbc0c004e50 2020-11-20T12:05:06.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.426+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000a850 con 0x7fbc0c004e50 2020-11-20T12:05:06.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.426+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc200ba1f0 con 0x7fbc0c004e50 2020-11-20T12:05:06.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.427+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbc20061a20 con 0x7fbc0c004e50 2020-11-20T12:05:06.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.428+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbc1000abc0 con 0x7fbc0c004e50 2020-11-20T12:05:06.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.428+0000 7fbc1d7fa700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbf8056d70 0x7fbbf8059230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.429+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbc10070120 con 0x7fbc0c004e50 2020-11-20T12:05:06.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.429+0000 7fbc1d7fa700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fbbf805e340 0x7fbbf8060760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.429+0000 7fbc24bd5700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbf8056d70 0x7fbbf8059230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.429+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 --> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbf8060dc0 con 0x7fbbf805e340 2020-11-20T12:05:06.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.429+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc10072680 con 0x7fbc0c004e50 2020-11-20T12:05:06.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.429+0000 7fbc25bd7700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fbbf805e340 0x7fbbf8060760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.430+0000 7fbc24bd5700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbf8056d70 0x7fbbf8059230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbc140013b0 tx=0x7fbc14000f20).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.430+0000 7fbc25bd7700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fbbf805e340 0x7fbbf8060760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.431+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 <== osd.3 v2:172.21.15.187:6826/19935 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fbbf8060dc0 con 0x7fbbf805e340 2020-11-20T12:05:06.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.449+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 --> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbc2014f540 con 0x7fbbf805e340 2020-11-20T12:05:06.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.450+0000 7fbc1d7fa700 1 -- 172.21.15.187:0/1131844018 <== osd.3 v2:172.21.15.187:6826/19935 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbc2014f540 con 0x7fbbf805e340 2020-11-20T12:05:06.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.450+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fbbf805e340 msgr2=0x7fbbf8060760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.450+0000 7fbc263d8700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fbbf805e340 0x7fbbf8060760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.450+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbf8056d70 msgr2=0x7fbbf8059230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.450+0000 7fbc263d8700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbf8056d70 0x7fbbf8059230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbc140013b0 tx=0x7fbc14000f20).stop 2020-11-20T12:05:06.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.450+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 msgr2=0x7fbc2015c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.450+0000 7fbc263d8700 1 --2- 172.21.15.187:0/1131844018 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c004e50 0x7fbc2015c290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fbc1000ea80 tx=0x7fbc10009040).stop 2020-11-20T12:05:06.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.451+0000 7fbc25bd7700 1 -- 172.21.15.187:0/1131844018 reap_dead start 2020-11-20T12:05:06.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.451+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 shutdown_connections 2020-11-20T12:05:06.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.451+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 >> 172.21.15.187:0/1131844018 conn(0x7fbc200bab30 msgr2=0x7fbc20148da0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.451+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 shutdown_connections 2020-11-20T12:05:06.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.451+0000 7fbc263d8700 1 -- 172.21.15.187:0/1131844018 wait complete. 2020-11-20T12:05:06.476 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:06.476 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2020-11-20T12:05:06.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.716+0000 7f18acb55700 1 Processor -- start 2020-11-20T12:05:06.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.716+0000 7f18acb55700 1 -- start start 2020-11-20T12:05:06.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.718+0000 7f18acb55700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f18a8065190 0x7f18a8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.718+0000 7f18acb55700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f18a8065f90 0x7f18a8143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.718+0000 7f18acb55700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a80b13e0 con 0x7f18a8143c70 2020-11-20T12:05:06.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.718+0000 7f18acb55700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a814ab30 con 0x7f18a8065190 2020-11-20T12:05:06.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.718+0000 7f18acb55700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a8062c10 con 0x7f18a8065f90 2020-11-20T12:05:06.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.718+0000 7f189effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f18a8065f90 0x7f18a8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.718+0000 7f189effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f18a8065f90 0x7f18a8143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40592/0 (socket says 172.21.15.187:40592) 2020-11-20T12:05:06.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.718+0000 7f189effd700 1 -- 172.21.15.187:0/2083368134 learned_addr learned my addr 172.21.15.187:0/2083368134 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:06.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f18a77fe700 1 --2- 172.21.15.187:0/2083368134 >> v2:172.21.15.5:3300/0 conn(0x7f18a8065190 0x7f18a8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f189effd700 1 -- 172.21.15.187:0/2083368134 >> v2:172.21.15.5:3300/0 conn(0x7f18a8065190 msgr2=0x7f18a8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f189effd700 1 --2- 172.21.15.187:0/2083368134 >> v2:172.21.15.5:3300/0 conn(0x7f18a8065190 0x7f18a8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f189effd700 1 -- 172.21.15.187:0/2083368134 >> v1:172.21.15.5:6789/0 conn(0x7f18a8143c70 legacy=0x7f18a809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f189effd700 1 -- 172.21.15.187:0/2083368134 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a80b1640 con 0x7f18a8065f90 2020-11-20T12:05:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f18a77fe700 1 --2- 172.21.15.187:0/2083368134 >> v2:172.21.15.5:3300/0 conn(0x7f18a8065190 0x7f18a8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f189effd700 1 --2- 172.21.15.187:0/2083368134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065f90 0x7f18a8143780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f18a80b5fb0 tx=0x7f1890006eb0).ready entity=mon.1 client_cookie=445e7e8b9e13438d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f18a6ffd700 1 -- 172.21.15.187:0/2083368134 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890009920 con 0x7f18a8065f90 2020-11-20T12:05:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f18a6ffd700 1 -- 172.21.15.187:0/2083368134 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1890009a80 con 0x7f18a8065f90 2020-11-20T12:05:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.719+0000 7f18a6ffd700 1 -- 172.21.15.187:0/2083368134 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890009e00 con 0x7f18a8065f90 2020-11-20T12:05:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.720+0000 7f18acb55700 1 -- 172.21.15.187:0/2083368134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065f90 msgr2=0x7f18a8143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.720+0000 7f18acb55700 1 --2- 172.21.15.187:0/2083368134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065f90 0x7f18a8143780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f18a80b5fb0 tx=0x7f1890006eb0).stop 2020-11-20T12:05:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.720+0000 7f18acb55700 1 -- 172.21.15.187:0/2083368134 shutdown_connections 2020-11-20T12:05:06.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.720+0000 7f18acb55700 1 --2- 172.21.15.187:0/2083368134 >> v2:172.21.15.5:3300/0 conn(0x7f18a8065190 0x7f18a8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.720+0000 7f18acb55700 1 --2- 172.21.15.187:0/2083368134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065f90 0x7f18a8143780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.720+0000 7f18acb55700 1 -- 172.21.15.187:0/2083368134 >> 172.21.15.187:0/2083368134 conn(0x7f18a80bab30 msgr2=0x7f18a80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.720+0000 7f18acb55700 1 -- 172.21.15.187:0/2083368134 shutdown_connections 2020-11-20T12:05:06.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.720+0000 7f18acb55700 1 -- 172.21.15.187:0/2083368134 wait complete. 2020-11-20T12:05:06.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.722+0000 7f18acb55700 1 Processor -- start 2020-11-20T12:05:06.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.722+0000 7f18acb55700 1 -- start start 2020-11-20T12:05:06.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18acb55700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065190 0x7f18a8145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18acb55700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18a8065f90 0x7f18a815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18acb55700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18a8143c70 0x7f18a815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18acb55700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a80b1bb0 con 0x7f18a8065f90 2020-11-20T12:05:06.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18acb55700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a80b1e50 con 0x7f18a8065190 2020-11-20T12:05:06.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18acb55700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a80b20f0 con 0x7f18a8143c70 2020-11-20T12:05:06.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18a77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065190 0x7f18a8145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18a77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065190 0x7f18a8145320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40594/0 (socket says 172.21.15.187:40594) 2020-11-20T12:05:06.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18a77fe700 1 -- 172.21.15.187:0/3501954239 learned_addr learned my addr 172.21.15.187:0/3501954239 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:06.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18a7fff700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18a8143c70 0x7f18a815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f189effd700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18a8065f90 0x7f18a815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18a77fe700 1 -- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18a8143c70 msgr2=0x7f18a815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18a77fe700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18a8143c70 0x7f18a815bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.724+0000 7f18a77fe700 1 -- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18a8065f90 msgr2=0x7f18a815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f18a77fe700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18a8065f90 0x7f18a815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f18a77fe700 1 -- 172.21.15.187:0/3501954239 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a8145810 con 0x7f18a8065190 2020-11-20T12:05:06.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f189effd700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18a8065f90 0x7f18a815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:06.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f18a77fe700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065190 0x7f18a8145320 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f189800cb50 tx=0x7f1898008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898007860 con 0x7f18a8065190 2020-11-20T12:05:06.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18980079c0 con 0x7f18a8065190 2020-11-20T12:05:06.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a80a0420 con 0x7f18a8065190 2020-11-20T12:05:06.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898007d40 con 0x7f18a8065190 2020-11-20T12:05:06.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.725+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a80b47f0 con 0x7f18a8065190 2020-11-20T12:05:06.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.726+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18a8061a20 con 0x7f18a8065190 2020-11-20T12:05:06.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.727+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1898011620 con 0x7f18a8065190 2020-11-20T12:05:06.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.728+0000 7f189ffff700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1884056d20 0x7f18840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.728+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1898070290 con 0x7f18a8065190 2020-11-20T12:05:06.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.729+0000 7f189effd700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1884056d20 0x7f18840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.729+0000 7f189ffff700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f188405e270 0x7f1884060690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.729+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1884060cf0 con 0x7f188405e270 2020-11-20T12:05:06.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.729+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1898072900 con 0x7f18a8065190 2020-11-20T12:05:06.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.729+0000 7f189effd700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1884056d20 0x7f18840591e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f18a80b5fb0 tx=0x7f1890015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.730+0000 7f18a7fff700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f188405e270 0x7f1884060690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.731+0000 7f18a7fff700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f188405e270 0x7f1884060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.733+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f1884060cf0 con 0x7f188405e270 2020-11-20T12:05:06.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.752+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f18a809f750 con 0x7f188405e270 2020-11-20T12:05:06.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f189ffff700 1 -- 172.21.15.187:0/3501954239 <== osd.4 v2:172.21.15.5:6808/20156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f18a809f750 con 0x7f188405e270 2020-11-20T12:05:06.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f188405e270 msgr2=0x7f1884060690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18acb55700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f188405e270 0x7f1884060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1884056d20 msgr2=0x7f18840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18acb55700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1884056d20 0x7f18840591e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f18a80b5fb0 tx=0x7f1890015000).stop 2020-11-20T12:05:06.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065190 msgr2=0x7f18a8145320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18acb55700 1 --2- 172.21.15.187:0/3501954239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a8065190 0x7f18a8145320 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f189800cb50 tx=0x7f1898008040).stop 2020-11-20T12:05:06.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18a7fff700 1 -- 172.21.15.187:0/3501954239 reap_dead start 2020-11-20T12:05:06.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 shutdown_connections 2020-11-20T12:05:06.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.754+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 >> 172.21.15.187:0/3501954239 conn(0x7f18a80bab30 msgr2=0x7f18a80b8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.755+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 shutdown_connections 2020-11-20T12:05:06.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:06.755+0000 7f18acb55700 1 -- 172.21.15.187:0/3501954239 wait complete. 2020-11-20T12:05:06.780 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:06.781 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2020-11-20T12:05:07.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.018+0000 7f5ff286a700 1 Processor -- start 2020-11-20T12:05:07.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.019+0000 7f5ff286a700 1 -- start start 2020-11-20T12:05:07.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.020+0000 7f5ff286a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5fec065830 0x7f5fec14ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.020+0000 7f5ff286a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec149fc0 con 0x7f5fec066220 2020-11-20T12:05:07.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.020+0000 7f5ff286a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0b1710 con 0x7f5fec065450 2020-11-20T12:05:07.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.020+0000 7f5ff286a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fec0b19b0 con 0x7f5fec065830 2020-11-20T12:05:07.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.020+0000 7f5febfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5fec065450 0x7f5fec14a3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37210/0 (socket says 172.21.15.187:37210) 2020-11-20T12:05:07.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.020+0000 7f5febfff700 1 -- 172.21.15.187:0/734575871 learned_addr learned my addr 172.21.15.187:0/734575871 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:07.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5fe37fe700 1 --2- 172.21.15.187:0/734575871 >> v2:172.21.15.5:3300/0 conn(0x7f5fec065830 0x7f5fec14ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852289557 0 0) 0x7f5fec0b1710 con 0x7f5fec065450 2020-11-20T12:05:07.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fcc003240 con 0x7f5fec065450 2020-11-20T12:05:07.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 662586022 0 0) 0x7f5fcc003240 con 0x7f5fec065450 2020-11-20T12:05:07.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 >> v2:172.21.15.5:3300/0 conn(0x7f5fec065830 msgr2=0x7f5fec14ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5feb7fe700 1 --2- 172.21.15.187:0/734575871 >> v2:172.21.15.5:3300/0 conn(0x7f5fec065830 0x7f5fec14ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 >> v1:172.21.15.5:6789/0 conn(0x7f5fec066220 legacy=0x7f5fec145b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec0b74c0 con 0x7f5fec065450 2020-11-20T12:05:07.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.021+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5fdc0014e0 con 0x7f5fec065450 2020-11-20T12:05:07.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 >> v1:172.21.15.187:6789/0 conn(0x7f5fec065450 legacy=0x7f5fec14a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5feb7fe700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fcc003730 0x7f5fcc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5feb7fe700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc0044e0 0x7f5fcc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5feb7fe700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fcc004e50 0x7f5fcc00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0b19b0 con 0x7f5fcc0044e0 2020-11-20T12:05:07.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc004260 con 0x7f5fcc003730 2020-11-20T12:05:07.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc00bc20 con 0x7f5fcc004e50 2020-11-20T12:05:07.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5febfff700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fcc003730 0x7f5fcc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5fe37fe700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc0044e0 0x7f5fcc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5febfff700 1 -- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fcc004e50 msgr2=0x7f5fcc00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5ff0e07700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fcc004e50 0x7f5fcc00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.022+0000 7f5febfff700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fcc004e50 0x7f5fcc00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5febfff700 1 -- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc0044e0 msgr2=0x7f5fcc004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5febfff700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc0044e0 0x7f5fcc004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5febfff700 1 -- 172.21.15.187:0/734575871 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc003240 con 0x7f5fcc003730 2020-11-20T12:05:07.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5fe37fe700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc0044e0 0x7f5fcc004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:07.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5ff0e07700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fcc004e50 0x7f5fcc00b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:07.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5ff0e07700 1 -- 172.21.15.187:0/734575871 reap_dead start 2020-11-20T12:05:07.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5febfff700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fcc003730 0x7f5fcc003b50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5fdc00c410 tx=0x7f5fdc0077d0).ready entity=mon.1 client_cookie=db56082e6d119a6a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc00ce80 con 0x7f5fcc003730 2020-11-20T12:05:07.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc0072e0 con 0x7f5fcc003730 2020-11-20T12:05:07.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.023+0000 7f5feb7fe700 1 -- 172.21.15.187:0/734575871 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc005800 con 0x7f5fcc003730 2020-11-20T12:05:07.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.024+0000 7f5ff286a700 1 -- 172.21.15.187:0/734575871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fcc003730 msgr2=0x7f5fcc003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.024+0000 7f5ff286a700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fcc003730 0x7f5fcc003b50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5fdc00c410 tx=0x7f5fdc0077d0).stop 2020-11-20T12:05:07.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.024+0000 7f5ff286a700 1 -- 172.21.15.187:0/734575871 shutdown_connections 2020-11-20T12:05:07.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.024+0000 7f5ff286a700 1 --2- 172.21.15.187:0/734575871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fcc003730 0x7f5fcc003b50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.024+0000 7f5ff286a700 1 -- 172.21.15.187:0/734575871 >> 172.21.15.187:0/734575871 conn(0x7f5fec0baca0 msgr2=0x7f5fec0bb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.024+0000 7f5ff286a700 1 -- 172.21.15.187:0/734575871 shutdown_connections 2020-11-20T12:05:07.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.024+0000 7f5ff286a700 1 -- 172.21.15.187:0/734575871 wait complete. 2020-11-20T12:05:07.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.025+0000 7f5ff286a700 1 Processor -- start 2020-11-20T12:05:07.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.025+0000 7f5ff286a700 1 -- start start 2020-11-20T12:05:07.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.026+0000 7f5ff286a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc003730 0x7f5fec145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.026+0000 7f5ff286a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fec0b4550 0x7f5fec0b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.026+0000 7f5ff286a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fec09bcd0 0x7f5fec09c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.026+0000 7f5ff286a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec145b70 con 0x7f5fcc003730 2020-11-20T12:05:07.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.026+0000 7f5ff286a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0b3000 con 0x7f5fec0b4550 2020-11-20T12:05:07.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.026+0000 7f5ff286a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0b4100 con 0x7f5fec09bcd0 2020-11-20T12:05:07.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fec0b4550 0x7f5fec0b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fec0b4550 0x7f5fec0b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40618/0 (socket says 172.21.15.187:40618) 2020-11-20T12:05:07.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 -- 172.21.15.187:0/2946333866 learned_addr learned my addr 172.21.15.187:0/2946333866 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:07.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5ff0e07700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fec09bcd0 0x7f5fec09c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 -- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fec09bcd0 msgr2=0x7f5fec09c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fec09bcd0 0x7f5fec09c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 -- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc003730 msgr2=0x7f5fec145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5febfff700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc003730 0x7f5fec145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fcc003730 0x7f5fec145680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 -- 172.21.15.187:0/2946333866 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec139800 con 0x7f5fec0b4550 2020-11-20T12:05:07.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe37fe700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fec0b4550 0x7f5fec0b3c10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5fd400f860 tx=0x7f5fd4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.027+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4017070 con 0x7f5fec0b4550 2020-11-20T12:05:07.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.028+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd400d7d0 con 0x7f5fec0b4550 2020-11-20T12:05:07.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.028+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400db50 con 0x7f5fec0b4550 2020-11-20T12:05:07.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.028+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec0b3230 con 0x7f5fec0b4550 2020-11-20T12:05:07.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.028+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec0b3770 con 0x7f5fec0b4550 2020-11-20T12:05:07.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.029+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5fec044f60 con 0x7f5fec0b4550 2020-11-20T12:05:07.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.029+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5fd401c410 con 0x7f5fec0b4550 2020-11-20T12:05:07.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.030+0000 7f5fe8ff9700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fc8056d50 0x7f5fc8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.030+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5fd40701c0 con 0x7f5fec0b4550 2020-11-20T12:05:07.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.030+0000 7f5febfff700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fc8056d50 0x7f5fc8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.030+0000 7f5fe8ff9700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f5fc805e220 0x7f5fc8060640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.030+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fc8060ca0 con 0x7f5fc805e220 2020-11-20T12:05:07.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.030+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fd4072720 con 0x7f5fec0b4550 2020-11-20T12:05:07.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.031+0000 7f5febfff700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fc8056d50 0x7f5fc8059210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5fdc00c410 tx=0x7f5fdc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.031+0000 7f5ff0e07700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f5fc805e220 0x7f5fc8060640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.032+0000 7f5ff0e07700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f5fc805e220 0x7f5fc8060640 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.035+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f5fc8060ca0 con 0x7f5fc805e220 2020-11-20T12:05:07.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.053+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5fec0b3900 con 0x7f5fc805e220 2020-11-20T12:05:07.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5fe8ff9700 1 -- 172.21.15.187:0/2946333866 <== osd.5 v2:172.21.15.5:6809/20155 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5fec0b3900 con 0x7f5fc805e220 2020-11-20T12:05:07.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f5fc805e220 msgr2=0x7f5fc8060640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f5fc805e220 0x7f5fc8060640 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fc8056d50 msgr2=0x7f5fc8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fc8056d50 0x7f5fc8059210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5fdc00c410 tx=0x7f5fdc015000).stop 2020-11-20T12:05:07.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fec0b4550 msgr2=0x7f5fec0b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 --2- 172.21.15.187:0/2946333866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fec0b4550 0x7f5fec0b3c10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5fd400f860 tx=0x7f5fd4008040).stop 2020-11-20T12:05:07.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff0e07700 1 -- 172.21.15.187:0/2946333866 reap_dead start 2020-11-20T12:05:07.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 shutdown_connections 2020-11-20T12:05:07.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 >> 172.21.15.187:0/2946333866 conn(0x7f5fec0baca0 msgr2=0x7f5fec0b8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.055+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 shutdown_connections 2020-11-20T12:05:07.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.056+0000 7f5ff286a700 1 -- 172.21.15.187:0/2946333866 wait complete. 2020-11-20T12:05:07.079 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:07.080 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2020-11-20T12:05:07.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.316+0000 7fa8ebd47700 1 Processor -- start 2020-11-20T12:05:07.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.316+0000 7fa8ebd47700 1 -- start start 2020-11-20T12:05:07.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.317+0000 7fa8ebd47700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa8e40652e0 0x7fa8e40656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.317+0000 7fa8ebd47700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40b1590 0x7fa8e414a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.317+0000 7fa8ebd47700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa8e40b7300 0x7fa8e4143920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.317+0000 7fa8ebd47700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e4149dd0 con 0x7fa8e40652e0 2020-11-20T12:05:07.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ebd47700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e414a070 con 0x7fa8e40b7300 2020-11-20T12:05:07.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ebd47700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e414a310 con 0x7fa8e40b1590 2020-11-20T12:05:07.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ea2e4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa8e40b7300 0x7fa8e4143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ea2e4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa8e40b7300 0x7fa8e4143920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40628/0 (socket says 172.21.15.187:40628) 2020-11-20T12:05:07.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/2786273402 learned_addr learned my addr 172.21.15.187:0/2786273402 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:07.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8e9ae3700 1 --2- 172.21.15.187:0/2786273402 >> v2:172.21.15.5:3300/0 conn(0x7fa8e40652e0 0x7fa8e40656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/2786273402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40b1590 msgr2=0x7fa8e414a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8e92e2700 1 --2- 172.21.15.187:0/2786273402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40b1590 0x7fa8e414a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ea2e4700 1 --2- 172.21.15.187:0/2786273402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40b1590 0x7fa8e414a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/2786273402 >> v2:172.21.15.5:3300/0 conn(0x7fa8e40652e0 msgr2=0x7fa8e40656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ea2e4700 1 --2- 172.21.15.187:0/2786273402 >> v2:172.21.15.5:3300/0 conn(0x7fa8e40652e0 0x7fa8e40656c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.318+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/2786273402 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e414aca0 con 0x7fa8e40b7300 2020-11-20T12:05:07.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.319+0000 7fa8e9ae3700 1 --2- 172.21.15.187:0/2786273402 >> v2:172.21.15.5:3300/0 conn(0x7fa8e40652e0 0x7fa8e40656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:07.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.319+0000 7fa8e92e2700 1 --2- 172.21.15.187:0/2786273402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40b1590 0x7fa8e414a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:07.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.319+0000 7fa8ea2e4700 1 --2- 172.21.15.187:0/2786273402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 0x7fa8e4143920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa8e40b6880 tx=0x7fa8dc006eb0).ready entity=mon.1 client_cookie=8a81e9d68438b51f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.319+0000 7fa8e8ae1700 1 -- 172.21.15.187:0/2786273402 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0098d0 con 0x7fa8e40b7300 2020-11-20T12:05:07.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.319+0000 7fa8e8ae1700 1 -- 172.21.15.187:0/2786273402 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc009a30 con 0x7fa8e40b7300 2020-11-20T12:05:07.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.319+0000 7fa8e8ae1700 1 -- 172.21.15.187:0/2786273402 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc009db0 con 0x7fa8e40b7300 2020-11-20T12:05:07.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 -- 172.21.15.187:0/2786273402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 msgr2=0x7fa8e4143920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 --2- 172.21.15.187:0/2786273402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 0x7fa8e4143920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa8e40b6880 tx=0x7fa8dc006eb0).stop 2020-11-20T12:05:07.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 -- 172.21.15.187:0/2786273402 shutdown_connections 2020-11-20T12:05:07.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 --2- 172.21.15.187:0/2786273402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 0x7fa8e4143920 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 --2- 172.21.15.187:0/2786273402 >> v2:172.21.15.5:3300/0 conn(0x7fa8e40652e0 0x7fa8e40656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 --2- 172.21.15.187:0/2786273402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40b1590 0x7fa8e414a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 -- 172.21.15.187:0/2786273402 >> 172.21.15.187:0/2786273402 conn(0x7fa8e40bacd0 msgr2=0x7fa8e40bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 -- 172.21.15.187:0/2786273402 shutdown_connections 2020-11-20T12:05:07.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.320+0000 7fa8ebd47700 1 -- 172.21.15.187:0/2786273402 wait complete. 2020-11-20T12:05:07.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.321+0000 7fa8ebd47700 1 Processor -- start 2020-11-20T12:05:07.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.322+0000 7fa8ebd47700 1 -- start start 2020-11-20T12:05:07.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.322+0000 7fa8ebd47700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40652e0 0x7fa8e414d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.322+0000 7fa8ebd47700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8e40b1590 0x7fa8e4148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.322+0000 7fa8ebd47700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 0x7fa8e4142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.322+0000 7fa8ebd47700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40b2760 con 0x7fa8e40b1590 2020-11-20T12:05:07.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.322+0000 7fa8ebd47700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e414d9e0 con 0x7fa8e40b7300 2020-11-20T12:05:07.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.322+0000 7fa8ebd47700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e414dc80 con 0x7fa8e40652e0 2020-11-20T12:05:07.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8ea2e4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 0x7fa8e4142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8ea2e4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 0x7fa8e4142eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40636/0 (socket says 172.21.15.187:40636) 2020-11-20T12:05:07.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/804974318 learned_addr learned my addr 172.21.15.187:0/804974318 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:07.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8e92e2700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8e40b1590 0x7fa8e4148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8e9ae3700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40652e0 0x7fa8e414d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40652e0 msgr2=0x7fa8e414d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8ea2e4700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40652e0 0x7fa8e414d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8e40b1590 msgr2=0x7fa8e4148460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8ea2e4700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8e40b1590 0x7fa8e4148460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/804974318 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e414df20 con 0x7fa8e40b7300 2020-11-20T12:05:07.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.323+0000 7fa8e9ae3700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8e40652e0 0x7fa8e414d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:07.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.324+0000 7fa8ea2e4700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 0x7fa8e4142eb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa8e414a1b0 tx=0x7fa8dc015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.324+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc01d5d0 con 0x7fa8e40b7300 2020-11-20T12:05:07.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.324+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc01d730 con 0x7fa8e40b7300 2020-11-20T12:05:07.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.324+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e4148950 con 0x7fa8e40b7300 2020-11-20T12:05:07.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.324+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc01dad0 con 0x7fa8e40b7300 2020-11-20T12:05:07.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.324+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e415b1f0 con 0x7fa8e40b7300 2020-11-20T12:05:07.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.325+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa8e4063600 con 0x7fa8e40b7300 2020-11-20T12:05:07.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.326+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa8dc025030 con 0x7fa8e40b7300 2020-11-20T12:05:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.326+0000 7fa8d1ffb700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8bc056d20 0x7fa8bc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.326+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa8dc071ad0 con 0x7fa8e40b7300 2020-11-20T12:05:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.327+0000 7fa8e9ae3700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8bc056d20 0x7fa8bc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.327+0000 7fa8d1ffb700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fa8bc05e270 0x7fa8bc060690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.327+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8bc060cf0 con 0x7fa8bc05e270 2020-11-20T12:05:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.327+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8dc076220 con 0x7fa8e40b7300 2020-11-20T12:05:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.327+0000 7fa8e9ae3700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8bc056d20 0x7fa8bc0591e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa8e0008b60 tx=0x7fa8e0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.327+0000 7fa8e92e2700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fa8bc05e270 0x7fa8bc060690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.329+0000 7fa8e92e2700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fa8bc05e270 0x7fa8bc060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.331+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fa8bc060cf0 con 0x7fa8bc05e270 2020-11-20T12:05:07.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.349+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa8e414e0b0 con 0x7fa8bc05e270 2020-11-20T12:05:07.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.350+0000 7fa8d1ffb700 1 -- 172.21.15.187:0/804974318 <== osd.6 v2:172.21.15.5:6800/20157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa8e414e0b0 con 0x7fa8bc05e270 2020-11-20T12:05:07.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fa8bc05e270 msgr2=0x7fa8bc060690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fa8bc05e270 0x7fa8bc060690 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8bc056d20 msgr2=0x7fa8bc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8bc056d20 0x7fa8bc0591e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa8e0008b60 tx=0x7fa8e0008040).stop 2020-11-20T12:05:07.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 msgr2=0x7fa8e4142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 --2- 172.21.15.187:0/804974318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8e40b7300 0x7fa8e4142eb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa8e414a1b0 tx=0x7fa8dc015000).stop 2020-11-20T12:05:07.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ea2e4700 1 -- 172.21.15.187:0/804974318 reap_dead start 2020-11-20T12:05:07.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 shutdown_connections 2020-11-20T12:05:07.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 >> 172.21.15.187:0/804974318 conn(0x7fa8e40bacd0 msgr2=0x7fa8e409fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 shutdown_connections 2020-11-20T12:05:07.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.351+0000 7fa8ebd47700 1 -- 172.21.15.187:0/804974318 wait complete. 2020-11-20T12:05:07.379 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:07.379 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2020-11-20T12:05:07.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.615+0000 7fa2b8b35700 1 Processor -- start 2020-11-20T12:05:07.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.616+0000 7fa2b8b35700 1 -- start start 2020-11-20T12:05:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.617+0000 7fa2b8b35700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa2b40b1760 0x7fa2b4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.617+0000 7fa2b8b35700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b4150390 con 0x7fa2b40b1380 2020-11-20T12:05:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.617+0000 7fa2b8b35700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b4150630 con 0x7fa2b4065680 2020-11-20T12:05:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.617+0000 7fa2b8b35700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b41511e0 con 0x7fa2b40b1760 2020-11-20T12:05:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.618+0000 7fa2aaffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa2b40b1760 0x7fa2b4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.618+0000 7fa2b3fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa2b4065680 0x7fa2b414c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37252/0 (socket says 172.21.15.187:37252) 2020-11-20T12:05:07.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.618+0000 7fa2b3fff700 1 -- 172.21.15.187:0/2472948540 learned_addr learned my addr 172.21.15.187:0/2472948540 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:07.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.618+0000 7fa2aaffd700 1 -- 172.21.15.187:0/2472948540 >> v1:172.21.15.187:6789/0 conn(0x7fa2b4065680 legacy=0x7fa2b414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.618+0000 7fa2aaffd700 1 -- 172.21.15.187:0/2472948540 >> v1:172.21.15.5:6789/0 conn(0x7fa2b40b1380 legacy=0x7fa2b4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.618+0000 7fa2aaffd700 1 -- 172.21.15.187:0/2472948540 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b4068510 con 0x7fa2b40b1760 2020-11-20T12:05:07.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.618+0000 7fa2aaffd700 1 --2- 172.21.15.187:0/2472948540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 0x7fa2b4065190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fa29c00b9a0 tx=0x7fa29c006df0).ready entity=mon.1 client_cookie=2ef37110253a9215 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b2ffd700 1 -- 172.21.15.187:0/2472948540 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c0097b0 con 0x7fa2b40b1760 2020-11-20T12:05:07.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b2ffd700 1 -- 172.21.15.187:0/2472948540 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c009910 con 0x7fa2b40b1760 2020-11-20T12:05:07.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b2ffd700 1 -- 172.21.15.187:0/2472948540 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c009c90 con 0x7fa2b40b1760 2020-11-20T12:05:07.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b8b35700 1 -- 172.21.15.187:0/2472948540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 msgr2=0x7fa2b4065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b8b35700 1 --2- 172.21.15.187:0/2472948540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 0x7fa2b4065190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fa29c00b9a0 tx=0x7fa29c006df0).stop 2020-11-20T12:05:07.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b8b35700 1 -- 172.21.15.187:0/2472948540 shutdown_connections 2020-11-20T12:05:07.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b8b35700 1 --2- 172.21.15.187:0/2472948540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 0x7fa2b4065190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b8b35700 1 -- 172.21.15.187:0/2472948540 >> 172.21.15.187:0/2472948540 conn(0x7fa2b40bab30 msgr2=0x7fa2b40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b8b35700 1 -- 172.21.15.187:0/2472948540 shutdown_connections 2020-11-20T12:05:07.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.619+0000 7fa2b8b35700 1 -- 172.21.15.187:0/2472948540 wait complete. 2020-11-20T12:05:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.621+0000 7fa2b8b35700 1 Processor -- start 2020-11-20T12:05:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.622+0000 7fa2b8b35700 1 -- start start 2020-11-20T12:05:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2b8b35700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa2b40b1380 0x7fa2b413d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2b8b35700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 0x7fa2b413c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2b8b35700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2b40b3510 0x7fa2b413cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2b8b35700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b409aa00 con 0x7fa2b40b3510 2020-11-20T12:05:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2b8b35700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40b25c0 con 0x7fa2b40b1760 2020-11-20T12:05:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2b8b35700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40b2860 con 0x7fa2b40b1380 2020-11-20T12:05:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2aaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 0x7fa2b413c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2aaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 0x7fa2b413c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40650/0 (socket says 172.21.15.187:40650) 2020-11-20T12:05:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2aaffd700 1 -- 172.21.15.187:0/57506332 learned_addr learned my addr 172.21.15.187:0/57506332 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.623+0000 7fa2b37fe700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa2b40b1380 0x7fa2b413d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2aaffd700 1 -- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa2b40b1380 msgr2=0x7fa2b413d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2aaffd700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa2b40b1380 0x7fa2b413d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2aaffd700 1 -- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2b40b3510 msgr2=0x7fa2b413cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2aaffd700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2b40b3510 0x7fa2b413cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2aaffd700 1 -- 172.21.15.187:0/57506332 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b413d810 con 0x7fa2b40b1760 2020-11-20T12:05:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2aaffd700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 0x7fa2b413c720 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa29c006e30 tx=0x7fa29c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c01e460 con 0x7fa2b40b1760 2020-11-20T12:05:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c01e5c0 con 0x7fa2b40b1760 2020-11-20T12:05:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b409ca60 con 0x7fa2b40b1760 2020-11-20T12:05:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c01e940 con 0x7fa2b40b1760 2020-11-20T12:05:07.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.624+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b40b55c0 con 0x7fa2b40b1760 2020-11-20T12:05:07.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.625+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa2b4061a20 con 0x7fa2b40b1760 2020-11-20T12:05:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.626+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa29c025030 con 0x7fa2b40b1760 2020-11-20T12:05:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.627+0000 7fa2abfff700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa290056d20 0x7fa2900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.627+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa29c071920 con 0x7fa2b40b1760 2020-11-20T12:05:07.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.627+0000 7fa2abfff700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fa29005e270 0x7fa290060690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.627+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa290060cf0 con 0x7fa29005e270 2020-11-20T12:05:07.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.627+0000 7fa2b37fe700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa290056d20 0x7fa2900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.627+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa29c076220 con 0x7fa2b40b1760 2020-11-20T12:05:07.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.628+0000 7fa2b37fe700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa290056d20 0x7fa2900591e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa2a40034f0 tx=0x7fa2a4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.628+0000 7fa2b3fff700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fa29005e270 0x7fa290060690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.629+0000 7fa2b3fff700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fa29005e270 0x7fa290060690 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.632+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fa290060cf0 con 0x7fa29005e270 2020-11-20T12:05:07.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.650+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa2b409abc0 con 0x7fa29005e270 2020-11-20T12:05:07.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.652+0000 7fa2abfff700 1 -- 172.21.15.187:0/57506332 <== osd.7 v2:172.21.15.5:6814/20158 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa2ac001c80 con 0x7fa29005e270 2020-11-20T12:05:07.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fa29005e270 msgr2=0x7fa290060690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fa29005e270 0x7fa290060690 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa290056d20 msgr2=0x7fa2900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa290056d20 0x7fa2900591e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa2a40034f0 tx=0x7fa2a4009040).stop 2020-11-20T12:05:07.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 msgr2=0x7fa2b413c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 --2- 172.21.15.187:0/57506332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2b40b1760 0x7fa2b413c720 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa29c006e30 tx=0x7fa29c015000).stop 2020-11-20T12:05:07.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b3fff700 1 -- 172.21.15.187:0/57506332 reap_dead start 2020-11-20T12:05:07.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 shutdown_connections 2020-11-20T12:05:07.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 >> 172.21.15.187:0/57506332 conn(0x7fa2b40bab30 msgr2=0x7fa2b40b9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 shutdown_connections 2020-11-20T12:05:07.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.653+0000 7fa2b8b35700 1 -- 172.21.15.187:0/57506332 wait complete. 2020-11-20T12:05:07.678 INFO:teuthology.orchestra.run.smithi187.stdout:60129542150 2020-11-20T12:05:07.678 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-20T12:05:07.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.920+0000 7fef45764700 1 Processor -- start 2020-11-20T12:05:07.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.920+0000 7fef45764700 1 -- start start 2020-11-20T12:05:07.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.921+0000 7fef45764700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40065e10 0x7fef40144470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.921+0000 7fef45764700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fef400661f0 0x7fef40144960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.921+0000 7fef45764700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fef40099ce0 0x7fef40061de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.921+0000 7fef45764700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400b5df0 con 0x7fef40099ce0 2020-11-20T12:05:07.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.921+0000 7fef45764700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef4009a440 con 0x7fef400661f0 2020-11-20T12:05:07.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.921+0000 7fef45764700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4009a1a0 con 0x7fef40065e10 2020-11-20T12:05:07.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.922+0000 7fef367fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fef400661f0 0x7fef40144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.922+0000 7fef367fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fef400661f0 0x7fef40144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40660/0 (socket says 172.21.15.187:40660) 2020-11-20T12:05:07.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.922+0000 7fef367fc700 1 -- 172.21.15.187:0/2442105591 learned_addr learned my addr 172.21.15.187:0/2442105591 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:07.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.922+0000 7fef3effd700 1 --2- 172.21.15.187:0/2442105591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40065e10 0x7fef40144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.922+0000 7fef3f7fe700 1 --2- 172.21.15.187:0/2442105591 >> v2:172.21.15.5:3300/0 conn(0x7fef40099ce0 0x7fef40061de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.922+0000 7fef367fc700 1 -- 172.21.15.187:0/2442105591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40065e10 msgr2=0x7fef40144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.922+0000 7fef367fc700 1 --2- 172.21.15.187:0/2442105591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40065e10 0x7fef40144470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.923+0000 7fef367fc700 1 -- 172.21.15.187:0/2442105591 >> v2:172.21.15.5:3300/0 conn(0x7fef40099ce0 msgr2=0x7fef40061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.923+0000 7fef367fc700 1 --2- 172.21.15.187:0/2442105591 >> v2:172.21.15.5:3300/0 conn(0x7fef40099ce0 0x7fef40061de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.923+0000 7fef367fc700 1 -- 172.21.15.187:0/2442105591 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef400b0160 con 0x7fef400661f0 2020-11-20T12:05:07.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.923+0000 7fef367fc700 1 --2- 172.21.15.187:0/2442105591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40144960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fef400b8ab0 tx=0x7fef28006eb0).ready entity=mon.1 client_cookie=f1707bfdd9fa9b37 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.923+0000 7fef3e7fc700 1 -- 172.21.15.187:0/2442105591 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28009900 con 0x7fef400661f0 2020-11-20T12:05:07.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.923+0000 7fef3e7fc700 1 -- 172.21.15.187:0/2442105591 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef28009a60 con 0x7fef400661f0 2020-11-20T12:05:07.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.923+0000 7fef3e7fc700 1 -- 172.21.15.187:0/2442105591 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28009de0 con 0x7fef400661f0 2020-11-20T12:05:07.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 -- 172.21.15.187:0/2442105591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 msgr2=0x7fef40144960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 --2- 172.21.15.187:0/2442105591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40144960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fef400b8ab0 tx=0x7fef28006eb0).stop 2020-11-20T12:05:07.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 -- 172.21.15.187:0/2442105591 shutdown_connections 2020-11-20T12:05:07.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 --2- 172.21.15.187:0/2442105591 >> v2:172.21.15.5:3300/0 conn(0x7fef40099ce0 0x7fef40061de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 --2- 172.21.15.187:0/2442105591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40065e10 0x7fef40144470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 --2- 172.21.15.187:0/2442105591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40144960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 -- 172.21.15.187:0/2442105591 >> 172.21.15.187:0/2442105591 conn(0x7fef400bacd0 msgr2=0x7fef400bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 -- 172.21.15.187:0/2442105591 shutdown_connections 2020-11-20T12:05:07.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.924+0000 7fef45764700 1 -- 172.21.15.187:0/2442105591 wait complete. 2020-11-20T12:05:07.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.926+0000 7fef45764700 1 Processor -- start 2020-11-20T12:05:07.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.926+0000 7fef45764700 1 -- start start 2020-11-20T12:05:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.927+0000 7fef45764700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef40065e10 0x7fef40146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.927+0000 7fef45764700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.927+0000 7fef45764700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40099ce0 0x7fef40154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.927+0000 7fef45764700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4009fc80 con 0x7fef40065e10 2020-11-20T12:05:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.927+0000 7fef45764700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef40154990 con 0x7fef400661f0 2020-11-20T12:05:07.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.927+0000 7fef45764700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4009fef0 con 0x7fef40099ce0 2020-11-20T12:05:07.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef367fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef367fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40666/0 (socket says 172.21.15.187:40666) 2020-11-20T12:05:07.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef367fc700 1 -- 172.21.15.187:0/1310369824 learned_addr learned my addr 172.21.15.187:0/1310369824 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef3effd700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef40065e10 0x7fef40146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef3f7fe700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40099ce0 0x7fef40154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef367fc700 1 -- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40099ce0 msgr2=0x7fef40154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef367fc700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40099ce0 0x7fef40154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef367fc700 1 -- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef40065e10 msgr2=0x7fef40146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef367fc700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef40065e10 0x7fef40146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef367fc700 1 -- 172.21.15.187:0/1310369824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef401468f0 con 0x7fef400661f0 2020-11-20T12:05:07.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef3effd700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef40065e10 0x7fef40146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:07.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.928+0000 7fef3f7fe700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40099ce0 0x7fef40154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:07.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.929+0000 7fef367fc700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40153b70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fef28006ef0 tx=0x7fef28017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.929+0000 7fef377fe700 1 -- 172.21.15.187:0/1310369824 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28020070 con 0x7fef400661f0 2020-11-20T12:05:07.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.929+0000 7fef377fe700 1 -- 172.21.15.187:0/1310369824 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2801a480 con 0x7fef400661f0 2020-11-20T12:05:07.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.929+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef40146c50 con 0x7fef400661f0 2020-11-20T12:05:07.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.929+0000 7fef377fe700 1 -- 172.21.15.187:0/1310369824 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2801a820 con 0x7fef400661f0 2020-11-20T12:05:07.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.929+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef400b43b0 con 0x7fef400661f0 2020-11-20T12:05:07.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.930+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef40154790 con 0x7fef400661f0 2020-11-20T12:05:07.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.931+0000 7fef377fe700 1 -- 172.21.15.187:0/1310369824 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fef2801ac50 con 0x7fef400661f0 2020-11-20T12:05:07.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.932+0000 7fef377fe700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fef1c056d20 0x7fef1c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.932+0000 7fef377fe700 1 -- 172.21.15.187:0/1310369824 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fef280737c0 con 0x7fef400661f0 2020-11-20T12:05:07.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.932+0000 7fef3effd700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fef1c056d20 0x7fef1c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.933+0000 7fef3effd700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fef1c056d20 0x7fef1c0591e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fef30006fd0 tx=0x7fef30008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:07.937+0000 7fef377fe700 1 -- 172.21.15.187:0/1310369824 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fef28048c10 con 0x7fef400661f0 2020-11-20T12:05:08.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.126+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fef40063600 con 0x7fef400661f0 2020-11-20T12:05:08.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.127+0000 7fef377fe700 1 -- 172.21.15.187:0/1310369824 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fef2804c230 con 0x7fef400661f0 2020-11-20T12:05:08.131 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:08.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fef1c056d20 msgr2=0x7fef1c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fef1c056d20 0x7fef1c0591e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fef30006fd0 tx=0x7fef30008040).stop 2020-11-20T12:05:08.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 msgr2=0x7fef40153b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40153b70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fef28006ef0 tx=0x7fef28017000).stop 2020-11-20T12:05:08.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 shutdown_connections 2020-11-20T12:05:08.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fef1c056d20 0x7fef1c0591e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef40065e10 0x7fef40146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fef40099ce0 0x7fef40154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 --2- 172.21.15.187:0/1310369824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fef400661f0 0x7fef40153b70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 >> 172.21.15.187:0/1310369824 conn(0x7fef400bacd0 msgr2=0x7fef40141220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:08.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 shutdown_connections 2020-11-20T12:05:08.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.129+0000 7fef45764700 1 -- 172.21.15.187:0/1310369824 wait complete. 2020-11-20T12:05:08.157 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.0 2020-11-20T12:05:08.158 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-20T12:05:08.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.399+0000 7fc95ac9b700 1 Processor -- start 2020-11-20T12:05:08.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.400+0000 7fc95ac9b700 1 -- start start 2020-11-20T12:05:08.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.401+0000 7fc95ac9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc954065190 0x7fc954065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.401+0000 7fc95ac9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc954065f90 0x7fc954143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.401+0000 7fc95ac9b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9540b13e0 con 0x7fc954143c70 2020-11-20T12:05:08.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.401+0000 7fc95ac9b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc95414ab30 con 0x7fc954065f90 2020-11-20T12:05:08.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.401+0000 7fc95ac9b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc954062c10 con 0x7fc954065190 2020-11-20T12:05:08.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.401+0000 7fc95a49a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc954143c70 0x7fc95409ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37282/0 (socket says 172.21.15.187:37282) 2020-11-20T12:05:08.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.401+0000 7fc95a49a700 1 -- 172.21.15.187:0/2899706888 learned_addr learned my addr 172.21.15.187:0/2899706888 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:08.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc959c99700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc954065190 0x7fc954065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc959498700 1 --2- 172.21.15.187:0/2899706888 >> v2:172.21.15.5:3300/0 conn(0x7fc954065f90 0x7fc954143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204056990 0 0) 0x7fc9540b13e0 con 0x7fc954143c70 2020-11-20T12:05:08.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc93c003240 con 0x7fc954143c70 2020-11-20T12:05:08.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3345602689 0 0) 0x7fc93c003240 con 0x7fc954143c70 2020-11-20T12:05:08.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc954065190 msgr2=0x7fc954065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:08.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc954065190 0x7fc954065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 >> v2:172.21.15.5:3300/0 conn(0x7fc954065f90 msgr2=0x7fc954143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:08.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 --2- 172.21.15.187:0/2899706888 >> v2:172.21.15.5:3300/0 conn(0x7fc954065f90 0x7fc954143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9540b1640 con 0x7fc954143c70 2020-11-20T12:05:08.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.402+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fc94c001860 con 0x7fc954143c70 2020-11-20T12:05:08.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 >> v1:172.21.15.187:6789/0 conn(0x7fc954143c70 legacy=0x7fc95409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc958c97700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc93c003730 0x7fc93c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc958c97700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc93c0044e0 0x7fc93c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc958c97700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc93c004e50 0x7fc93c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95414ab30 con 0x7fc93c004e50 2020-11-20T12:05:08.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc954062c10 con 0x7fc93c003730 2020-11-20T12:05:08.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0041e0 con 0x7fc93c0044e0 2020-11-20T12:05:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc959c99700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc93c003730 0x7fc93c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.403+0000 7fc95a49a700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc93c004e50 0x7fc93c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.404+0000 7fc959498700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc93c0044e0 0x7fc93c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.404+0000 7fc959c99700 1 -- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc93c0044e0 msgr2=0x7fc93c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.404+0000 7fc959c99700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc93c0044e0 0x7fc93c004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.404+0000 7fc959c99700 1 -- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc93c004e50 msgr2=0x7fc93c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:08.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.404+0000 7fc959c99700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc93c004e50 0x7fc93c00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.404+0000 7fc959c99700 1 -- 172.21.15.187:0/2899706888 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c003240 con 0x7fc93c003730 2020-11-20T12:05:08.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.405+0000 7fc959c99700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc93c003730 0x7fc93c003b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fc9540b1b80 tx=0x7fc950006df0).ready entity=mon.1 client_cookie=beb304a1bb210da1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:08.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.405+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000b2d0 con 0x7fc93c003730 2020-11-20T12:05:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.405+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95000e960 con 0x7fc93c003730 2020-11-20T12:05:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.405+0000 7fc958c97700 1 -- 172.21.15.187:0/2899706888 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000ece0 con 0x7fc93c003730 2020-11-20T12:05:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.405+0000 7fc95ac9b700 1 -- 172.21.15.187:0/2899706888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc93c003730 msgr2=0x7fc93c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.405+0000 7fc95ac9b700 1 --2- 172.21.15.187:0/2899706888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc93c003730 0x7fc93c003b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fc9540b1b80 tx=0x7fc950006df0).stop 2020-11-20T12:05:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.405+0000 7fc95a49a700 1 -- 172.21.15.187:0/2899706888 reap_dead start 2020-11-20T12:05:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.406+0000 7fc95ac9b700 1 -- 172.21.15.187:0/2899706888 shutdown_connections 2020-11-20T12:05:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.406+0000 7fc95ac9b700 1 -- 172.21.15.187:0/2899706888 >> 172.21.15.187:0/2899706888 conn(0x7fc9540bab30 msgr2=0x7fc9540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.406+0000 7fc95ac9b700 1 -- 172.21.15.187:0/2899706888 shutdown_connections 2020-11-20T12:05:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.406+0000 7fc95ac9b700 1 -- 172.21.15.187:0/2899706888 wait complete. 2020-11-20T12:05:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.407+0000 7fc95ac9b700 1 Processor -- start 2020-11-20T12:05:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.407+0000 7fc95ac9b700 1 -- start start 2020-11-20T12:05:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.408+0000 7fc95ac9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc954141230 0x7fc9540a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.408+0000 7fc95ac9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9540678f0 0x7fc95409e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.408+0000 7fc95ac9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc95409eac0 0x7fc9540b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.408+0000 7fc95ac9b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95409dd80 con 0x7fc954141230 2020-11-20T12:05:08.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.408+0000 7fc95ac9b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95409e880 con 0x7fc9540678f0 2020-11-20T12:05:08.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.408+0000 7fc95ac9b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95409dae0 con 0x7fc95409eac0 2020-11-20T12:05:08.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.408+0000 7fc959498700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9540678f0 0x7fc95409e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959498700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9540678f0 0x7fc95409e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40686/0 (socket says 172.21.15.187:40686) 2020-11-20T12:05:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959498700 1 -- 172.21.15.187:0/4009752452 learned_addr learned my addr 172.21.15.187:0/4009752452 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959c99700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc954141230 0x7fc9540a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959498700 1 -- 172.21.15.187:0/4009752452 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc95409eac0 msgr2=0x7fc9540b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959498700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc95409eac0 0x7fc9540b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:08.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959498700 1 -- 172.21.15.187:0/4009752452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc954141230 msgr2=0x7fc9540a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959498700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc954141230 0x7fc9540a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959498700 1 -- 172.21.15.187:0/4009752452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9540682f0 con 0x7fc9540678f0 2020-11-20T12:05:08.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc959498700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9540678f0 0x7fc95409e390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc944010c80 tx=0x7fc944008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:08.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc941ffb700 1 -- 172.21.15.187:0/4009752452 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94400ea20 con 0x7fc9540678f0 2020-11-20T12:05:08.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc941ffb700 1 -- 172.21.15.187:0/4009752452 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94400eb80 con 0x7fc9540678f0 2020-11-20T12:05:08.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc954141610 con 0x7fc9540678f0 2020-11-20T12:05:08.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.409+0000 7fc941ffb700 1 -- 172.21.15.187:0/4009752452 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944014470 con 0x7fc9540678f0 2020-11-20T12:05:08.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.410+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9540b20a0 con 0x7fc9540678f0 2020-11-20T12:05:08.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.411+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc954068480 con 0x7fc9540678f0 2020-11-20T12:05:08.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.411+0000 7fc941ffb700 1 -- 172.21.15.187:0/4009752452 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc944024070 con 0x7fc9540678f0 2020-11-20T12:05:08.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.412+0000 7fc941ffb700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc92c056d20 0x7fc92c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.412+0000 7fc941ffb700 1 -- 172.21.15.187:0/4009752452 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc944070a80 con 0x7fc9540678f0 2020-11-20T12:05:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.412+0000 7fc959c99700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc92c056d20 0x7fc92c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.413+0000 7fc959c99700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc92c056d20 0x7fc92c0591e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc95405ea30 tx=0x7fc950019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.416+0000 7fc941ffb700 1 -- 172.21.15.187:0/4009752452 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc944045e50 con 0x7fc9540678f0 2020-11-20T12:05:08.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.598+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc9540b2320 con 0x7fc9540678f0 2020-11-20T12:05:08.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.599+0000 7fc941ffb700 1 -- 172.21.15.187:0/4009752452 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc944049470 con 0x7fc9540678f0 2020-11-20T12:05:08.603 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc92c056d20 msgr2=0x7fc92c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc92c056d20 0x7fc92c0591e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc95405ea30 tx=0x7fc950019000).stop 2020-11-20T12:05:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9540678f0 msgr2=0x7fc95409e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9540678f0 0x7fc95409e390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc944010c80 tx=0x7fc944008040).stop 2020-11-20T12:05:08.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 shutdown_connections 2020-11-20T12:05:08.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc92c056d20 0x7fc92c0591e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc954141230 0x7fc9540a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc95409eac0 0x7fc9540b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 --2- 172.21.15.187:0/4009752452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9540678f0 0x7fc95409e390 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 >> 172.21.15.187:0/4009752452 conn(0x7fc9540bab30 msgr2=0x7fc9541431d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 shutdown_connections 2020-11-20T12:05:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.601+0000 7fc95ac9b700 1 -- 172.21.15.187:0/4009752452 wait complete. 2020-11-20T12:05:08.626 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.1 2020-11-20T12:05:08.626 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-11-20T12:05:08.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.857+0000 7fc6e4e1a700 1 Processor -- start 2020-11-20T12:05:08.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.857+0000 7fc6e4e1a700 1 -- start start 2020-11-20T12:05:08.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6e4e1a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e01484d0 0x7fc6e0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6e4e1a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6e00622c0 0x7fc6e00b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6e4e1a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e00b1300 con 0x7fc6e0066f70 2020-11-20T12:05:08.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6e4e1a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e0067730 con 0x7fc6e00622c0 2020-11-20T12:05:08.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6e4e1a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e00b15a0 con 0x7fc6e01484d0 2020-11-20T12:05:08.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6d6ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6e00622c0 0x7fc6e00b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6d6ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6e00622c0 0x7fc6e00b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40696/0 (socket says 172.21.15.187:40696) 2020-11-20T12:05:08.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6d6ffd700 1 -- 172.21.15.187:0/2649798637 learned_addr learned my addr 172.21.15.187:0/2649798637 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:08.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.859+0000 7fc6df7fe700 1 --2- 172.21.15.187:0/2649798637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e01484d0 0x7fc6e0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.860+0000 7fc6d6ffd700 1 -- 172.21.15.187:0/2649798637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e01484d0 msgr2=0x7fc6e0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:08.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.860+0000 7fc6d6ffd700 1 --2- 172.21.15.187:0/2649798637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e01484d0 0x7fc6e0061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.860+0000 7fc6d6ffd700 1 -- 172.21.15.187:0/2649798637 >> v1:172.21.15.5:6789/0 conn(0x7fc6e0066f70 legacy=0x7fc6e00b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.860+0000 7fc6d6ffd700 1 -- 172.21.15.187:0/2649798637 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e00b1840 con 0x7fc6e00622c0 2020-11-20T12:05:08.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.860+0000 7fc6d6ffd700 1 --2- 172.21.15.187:0/2649798637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b5640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc6c8006fd0 tx=0x7fc6c8006eb0).ready entity=mon.1 client_cookie=ff85528515c80354 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:08.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.861+0000 7fc6deffd700 1 -- 172.21.15.187:0/2649798637 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80098d0 con 0x7fc6e00622c0 2020-11-20T12:05:08.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.861+0000 7fc6deffd700 1 -- 172.21.15.187:0/2649798637 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c8009a30 con 0x7fc6e00622c0 2020-11-20T12:05:08.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.861+0000 7fc6deffd700 1 -- 172.21.15.187:0/2649798637 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8009db0 con 0x7fc6e00622c0 2020-11-20T12:05:08.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.861+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/2649798637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 msgr2=0x7fc6e00b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.861+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/2649798637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b5640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc6c8006fd0 tx=0x7fc6c8006eb0).stop 2020-11-20T12:05:08.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.862+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/2649798637 shutdown_connections 2020-11-20T12:05:08.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.862+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/2649798637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e01484d0 0x7fc6e0061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.862+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/2649798637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b5640 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.862+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/2649798637 >> 172.21.15.187:0/2649798637 conn(0x7fc6e00bab30 msgr2=0x7fc6e00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:08.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.862+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/2649798637 shutdown_connections 2020-11-20T12:05:08.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.862+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/2649798637 wait complete. 2020-11-20T12:05:08.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.863+0000 7fc6e4e1a700 1 Processor -- start 2020-11-20T12:05:08.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.863+0000 7fc6e4e1a700 1 -- start start 2020-11-20T12:05:08.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.864+0000 7fc6e4e1a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.864+0000 7fc6e4e1a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e0066f70 0x7fc6e015a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.864+0000 7fc6e4e1a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6e01484d0 0x7fc6e00b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.864+0000 7fc6e4e1a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e009e5d0 con 0x7fc6e01484d0 2020-11-20T12:05:08.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.864+0000 7fc6e4e1a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e015a680 con 0x7fc6e00622c0 2020-11-20T12:05:08.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.864+0000 7fc6e4e1a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e00b3b10 con 0x7fc6e0066f70 2020-11-20T12:05:08.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40698/0 (socket says 172.21.15.187:40698) 2020-11-20T12:05:08.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 -- 172.21.15.187:0/3873310498 learned_addr learned my addr 172.21.15.187:0/3873310498 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:08.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6d6ffd700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e0066f70 0x7fc6e015a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6dffff700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6e01484d0 0x7fc6e00b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 -- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e0066f70 msgr2=0x7fc6e015a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e0066f70 0x7fc6e015a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 -- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6e01484d0 msgr2=0x7fc6e00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:08.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6e01484d0 0x7fc6e00b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:08.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 -- 172.21.15.187:0/3873310498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e00b3db0 con 0x7fc6e00622c0 2020-11-20T12:05:08.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6d6ffd700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e0066f70 0x7fc6e015a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:08.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6dffff700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6e01484d0 0x7fc6e00b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:08.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.865+0000 7fc6df7fe700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b2090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc6d000dcb0 tx=0x7fc6d0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:08.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.866+0000 7fc6d7fff700 1 -- 172.21.15.187:0/3873310498 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0009bd0 con 0x7fc6e00622c0 2020-11-20T12:05:08.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.866+0000 7fc6d7fff700 1 -- 172.21.15.187:0/3873310498 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d0009d30 con 0x7fc6e00622c0 2020-11-20T12:05:08.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.866+0000 7fc6d7fff700 1 -- 172.21.15.187:0/3873310498 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0009ea0 con 0x7fc6e00622c0 2020-11-20T12:05:08.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.866+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e00b3190 con 0x7fc6e00622c0 2020-11-20T12:05:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.866+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e00b3840 con 0x7fc6e00622c0 2020-11-20T12:05:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.867+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e009e740 con 0x7fc6e00622c0 2020-11-20T12:05:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.868+0000 7fc6d7fff700 1 -- 172.21.15.187:0/3873310498 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc6d0012840 con 0x7fc6e00622c0 2020-11-20T12:05:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.868+0000 7fc6d7fff700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6b4056d50 0x7fc6b4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:08.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.868+0000 7fc6d7fff700 1 -- 172.21.15.187:0/3873310498 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc6d00704e0 con 0x7fc6e00622c0 2020-11-20T12:05:08.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.869+0000 7fc6d6ffd700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6b4056d50 0x7fc6b4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:08.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.869+0000 7fc6d6ffd700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6b4056d50 0x7fc6b4059210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fc6c800bfb0 tx=0x7fc6c8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:08.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:08.872+0000 7fc6d7fff700 1 -- 172.21.15.187:0/3873310498 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc6e009e740 con 0x7fc6e00622c0 2020-11-20T12:05:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.064+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc6e00b3f40 con 0x7fc6e00622c0 2020-11-20T12:05:09.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.065+0000 7fc6d7fff700 1 -- 172.21.15.187:0/3873310498 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc6d00457c0 con 0x7fc6e00622c0 2020-11-20T12:05:09.069 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6b4056d50 msgr2=0x7fc6b4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6b4056d50 0x7fc6b4059210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fc6c800bfb0 tx=0x7fc6c8015000).stop 2020-11-20T12:05:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 msgr2=0x7fc6e00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b2090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc6d000dcb0 tx=0x7fc6d0008040).stop 2020-11-20T12:05:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 shutdown_connections 2020-11-20T12:05:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6b4056d50 0x7fc6b4059210 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6e01484d0 0x7fc6e00b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6e00622c0 0x7fc6e00b2090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 --2- 172.21.15.187:0/3873310498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6e0066f70 0x7fc6e015a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.067+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 >> 172.21.15.187:0/3873310498 conn(0x7fc6e00bab30 msgr2=0x7fc6e00b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.068+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 shutdown_connections 2020-11-20T12:05:09.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.068+0000 7fc6e4e1a700 1 -- 172.21.15.187:0/3873310498 wait complete. 2020-11-20T12:05:09.094 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.2 2020-11-20T12:05:09.095 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2020-11-20T12:05:09.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.334+0000 7fe7d719c700 1 Processor -- start 2020-11-20T12:05:09.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.334+0000 7fe7d719c700 1 -- start start 2020-11-20T12:05:09.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.335+0000 7fe7d719c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d0099ce0 0x7fe7d00b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.335+0000 7fe7d719c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d00b5df0 con 0x7fe7d0065e10 2020-11-20T12:05:09.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.335+0000 7fe7d719c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d009a440 con 0x7fe7d00661f0 2020-11-20T12:05:09.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.335+0000 7fe7d719c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d009a1a0 con 0x7fe7d0099ce0 2020-11-20T12:05:09.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.336+0000 7fe7c7fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe7d00661f0 0x7fe7d00b62e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37312/0 (socket says 172.21.15.187:37312) 2020-11-20T12:05:09.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.336+0000 7fe7c7fff700 1 -- 172.21.15.187:0/2086753129 learned_addr learned my addr 172.21.15.187:0/2086753129 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:09.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.336+0000 7fe7d5739700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d0099ce0 0x7fe7d00b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.336+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581243297 0 0) 0x7fe7d009a440 con 0x7fe7d00661f0 2020-11-20T12:05:09.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.336+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc003240 con 0x7fe7d00661f0 2020-11-20T12:05:09.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1577832854 0 0) 0x7fe7bc003240 con 0x7fe7d00661f0 2020-11-20T12:05:09.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d0099ce0 msgr2=0x7fe7d00b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d0099ce0 0x7fe7d00b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 >> v1:172.21.15.5:6789/0 conn(0x7fe7d0065e10 legacy=0x7fe7d0144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d00b0160 con 0x7fe7d00661f0 2020-11-20T12:05:09.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fe7c0001660 con 0x7fe7d00661f0 2020-11-20T12:05:09.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 >> v1:172.21.15.187:6789/0 conn(0x7fe7d00661f0 legacy=0x7fe7d00b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7bc003730 0x7fe7bc003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d0099ce0 0x7fe7bc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7bc0049d0 0x7fe7bc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d009a1a0 con 0x7fe7bc003730 2020-11-20T12:05:09.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc004140 con 0x7fe7bc0049d0 2020-11-20T12:05:09.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.337+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc004280 con 0x7fe7d0099ce0 2020-11-20T12:05:09.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7d5739700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7bc0049d0 0x7fe7bc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7d4f38700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7bc003730 0x7fe7bc003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7c7fff700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d0099ce0 0x7fe7bc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7d5739700 1 -- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d0099ce0 msgr2=0x7fe7bc0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7d5739700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d0099ce0 0x7fe7bc0044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7d5739700 1 -- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7bc003730 msgr2=0x7fe7bc003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7d5739700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7bc003730 0x7fe7bc003b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7d5739700 1 -- 172.21.15.187:0/2086753129 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc003240 con 0x7fe7bc0049d0 2020-11-20T12:05:09.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7d5739700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7bc0049d0 0x7fe7bc004e50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe7d00684a0 tx=0x7fe7c8006df0).ready entity=mon.1 client_cookie=6a779e15fc7aa9d0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c800b330 con 0x7fe7bc0049d0 2020-11-20T12:05:09.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c800eae0 con 0x7fe7bc0049d0 2020-11-20T12:05:09.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.338+0000 7fe7c77fe700 1 -- 172.21.15.187:0/2086753129 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c800ee60 con 0x7fe7bc0049d0 2020-11-20T12:05:09.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.339+0000 7fe7d719c700 1 -- 172.21.15.187:0/2086753129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7bc0049d0 msgr2=0x7fe7bc004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.339+0000 7fe7d719c700 1 --2- 172.21.15.187:0/2086753129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7bc0049d0 0x7fe7bc004e50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe7d00684a0 tx=0x7fe7c8006df0).stop 2020-11-20T12:05:09.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.339+0000 7fe7d5739700 1 -- 172.21.15.187:0/2086753129 reap_dead start 2020-11-20T12:05:09.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.339+0000 7fe7d719c700 1 -- 172.21.15.187:0/2086753129 shutdown_connections 2020-11-20T12:05:09.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.339+0000 7fe7d719c700 1 -- 172.21.15.187:0/2086753129 >> 172.21.15.187:0/2086753129 conn(0x7fe7d00bacd0 msgr2=0x7fe7d00bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.339+0000 7fe7d719c700 1 -- 172.21.15.187:0/2086753129 shutdown_connections 2020-11-20T12:05:09.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.339+0000 7fe7d719c700 1 -- 172.21.15.187:0/2086753129 wait complete. 2020-11-20T12:05:09.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.341+0000 7fe7d719c700 1 Processor -- start 2020-11-20T12:05:09.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.341+0000 7fe7d719c700 1 -- start start 2020-11-20T12:05:09.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d719c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7d0157190 0x7fe7d0157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d719c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7d013ee60 0x7fe7d013f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d719c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d009f370 0x7fe7d009f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d719c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d013f7d0 con 0x7fe7d013ee60 2020-11-20T12:05:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d719c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d009ed20 con 0x7fe7d0157190 2020-11-20T12:05:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d719c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d009efc0 con 0x7fe7d009f370 2020-11-20T12:05:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d4f38700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7d0157190 0x7fe7d0157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d4f38700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7d0157190 0x7fe7d0157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40718/0 (socket says 172.21.15.187:40718) 2020-11-20T12:05:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.342+0000 7fe7d4f38700 1 -- 172.21.15.187:0/1816571031 learned_addr learned my addr 172.21.15.187:0/1816571031 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:09.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7c7fff700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7d013ee60 0x7fe7d013f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7d5739700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d009f370 0x7fe7d009f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7d4f38700 1 -- 172.21.15.187:0/1816571031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d009f370 msgr2=0x7fe7d009f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7d4f38700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d009f370 0x7fe7d009f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7d4f38700 1 -- 172.21.15.187:0/1816571031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7d013ee60 msgr2=0x7fe7d013f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7d4f38700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7d013ee60 0x7fe7d013f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7d4f38700 1 -- 172.21.15.187:0/1816571031 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d009ea50 con 0x7fe7d0157190 2020-11-20T12:05:09.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7d4f38700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7d0157190 0x7fe7d0157570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe7cc00db30 tx=0x7fe7cc009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7c4ff9700 1 -- 172.21.15.187:0/1816571031 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc00b960 con 0x7fe7d0157190 2020-11-20T12:05:09.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7c4ff9700 1 -- 172.21.15.187:0/1816571031 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc00bac0 con 0x7fe7d0157190 2020-11-20T12:05:09.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.343+0000 7fe7c4ff9700 1 -- 172.21.15.187:0/1816571031 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc00be60 con 0x7fe7d0157190 2020-11-20T12:05:09.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.344+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d00b4500 con 0x7fe7d0157190 2020-11-20T12:05:09.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.344+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d0157c20 con 0x7fe7d0157190 2020-11-20T12:05:09.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.345+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d00b4720 con 0x7fe7d0157190 2020-11-20T12:05:09.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.345+0000 7fe7c4ff9700 1 -- 172.21.15.187:0/1816571031 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe7cc012620 con 0x7fe7d0157190 2020-11-20T12:05:09.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.346+0000 7fe7c4ff9700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7a8056d20 0x7fe7a80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.346+0000 7fe7c4ff9700 1 -- 172.21.15.187:0/1816571031 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe7cc070590 con 0x7fe7d0157190 2020-11-20T12:05:09.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.346+0000 7fe7c7fff700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7a8056d20 0x7fe7a80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.347+0000 7fe7c7fff700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7a8056d20 0x7fe7a80591e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fe7c0001530 tx=0x7fe7c0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.350+0000 7fe7c4ff9700 1 -- 172.21.15.187:0/1816571031 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe7d00b4720 con 0x7fe7d0157190 2020-11-20T12:05:09.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.535+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe7d009f180 con 0x7fe7d0157190 2020-11-20T12:05:09.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.536+0000 7fe7c4ff9700 1 -- 172.21.15.187:0/1816571031 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe7cc045980 con 0x7fe7d0157190 2020-11-20T12:05:09.541 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:09.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.538+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7a8056d20 msgr2=0x7fe7a80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.538+0000 7fe7d719c700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7a8056d20 0x7fe7a80591e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fe7c0001530 tx=0x7fe7c0009040).stop 2020-11-20T12:05:09.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.538+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7d0157190 msgr2=0x7fe7d0157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.538+0000 7fe7d719c700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7d0157190 0x7fe7d0157570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe7cc00db30 tx=0x7fe7cc009040).stop 2020-11-20T12:05:09.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.539+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 shutdown_connections 2020-11-20T12:05:09.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.539+0000 7fe7d719c700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7a8056d20 0x7fe7a80591e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.539+0000 7fe7d719c700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7d009f370 0x7fe7d009f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.539+0000 7fe7d719c700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7d0157190 0x7fe7d0157570 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.539+0000 7fe7d719c700 1 --2- 172.21.15.187:0/1816571031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7d013ee60 0x7fe7d013f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.539+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 >> 172.21.15.187:0/1816571031 conn(0x7fe7d00bacd0 msgr2=0x7fe7d0144240 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.539+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 shutdown_connections 2020-11-20T12:05:09.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.539+0000 7fe7d719c700 1 -- 172.21.15.187:0/1816571031 wait complete. 2020-11-20T12:05:09.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.3 2020-11-20T12:05:09.565 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2020-11-20T12:05:09.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.796+0000 7fcc42393700 1 Processor -- start 2020-11-20T12:05:09.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.796+0000 7fcc42393700 1 -- start start 2020-11-20T12:05:09.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.797+0000 7fcc42393700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c065f90 0x7fcc3c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.797+0000 7fcc42393700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3c0b13e0 con 0x7fcc3c09ca90 2020-11-20T12:05:09.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.797+0000 7fcc42393700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3c062c10 con 0x7fcc3c065190 2020-11-20T12:05:09.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.797+0000 7fcc42393700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c14ab30 con 0x7fcc3c065f90 2020-11-20T12:05:09.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.798+0000 7fcc41391700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcc3c065190 0x7fcc3c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37332/0 (socket says 172.21.15.187:37332) 2020-11-20T12:05:09.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.798+0000 7fcc41391700 1 -- 172.21.15.187:0/171952674 learned_addr learned my addr 172.21.15.187:0/171952674 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:09.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.798+0000 7fcc40b90700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c065f90 0x7fcc3c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432321538 0 0) 0x7fcc3c062c10 con 0x7fcc3c065190 2020-11-20T12:05:09.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc24003240 con 0x7fcc3c065190 2020-11-20T12:05:09.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3440039133 0 0) 0x7fcc24003240 con 0x7fcc3c065190 2020-11-20T12:05:09.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c065f90 msgr2=0x7fcc3c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c065f90 0x7fcc3c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 >> v1:172.21.15.5:6789/0 conn(0x7fcc3c09ca90 legacy=0x7fcc3c09cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c0b1640 con 0x7fcc3c065190 2020-11-20T12:05:09.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcc380014e0 con 0x7fcc3c065190 2020-11-20T12:05:09.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 >> v1:172.21.15.187:6789/0 conn(0x7fcc3c065190 legacy=0x7fcc3c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.799+0000 7fcc2bfff700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c065f90 0x7fcc24003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.800+0000 7fcc2bfff700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc24003c40 0x7fcc240044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.800+0000 7fcc2bfff700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc240049d0 0x7fcc24004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.800+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c14ab30 con 0x7fcc24003c40 2020-11-20T12:05:09.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.800+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc240041d0 con 0x7fcc240049d0 2020-11-20T12:05:09.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.800+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2400b730 con 0x7fcc3c065f90 2020-11-20T12:05:09.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.800+0000 7fcc41b92700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc240049d0 0x7fcc24004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.800+0000 7fcc41391700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c065f90 0x7fcc24003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.800+0000 7fcc40b90700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc24003c40 0x7fcc240044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc41b92700 1 -- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c065f90 msgr2=0x7fcc24003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc41b92700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c065f90 0x7fcc24003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc41b92700 1 -- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc24003c40 msgr2=0x7fcc240044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc41b92700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc24003c40 0x7fcc240044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc41b92700 1 -- 172.21.15.187:0/171952674 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc24003240 con 0x7fcc240049d0 2020-11-20T12:05:09.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc40b90700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc24003c40 0x7fcc240044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:09.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc41b92700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc240049d0 0x7fcc24004e50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fcc34001e70 tx=0x7fcc34007f10).ready entity=mon.1 client_cookie=dfca0b07940fbe5b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34001540 con 0x7fcc240049d0 2020-11-20T12:05:09.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3400ca10 con 0x7fcc240049d0 2020-11-20T12:05:09.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.801+0000 7fcc2bfff700 1 -- 172.21.15.187:0/171952674 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3400cdb0 con 0x7fcc240049d0 2020-11-20T12:05:09.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.802+0000 7fcc42393700 1 -- 172.21.15.187:0/171952674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc240049d0 msgr2=0x7fcc24004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.802+0000 7fcc42393700 1 --2- 172.21.15.187:0/171952674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc240049d0 0x7fcc24004e50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fcc34001e70 tx=0x7fcc34007f10).stop 2020-11-20T12:05:09.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.802+0000 7fcc41b92700 1 -- 172.21.15.187:0/171952674 reap_dead start 2020-11-20T12:05:09.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.802+0000 7fcc42393700 1 -- 172.21.15.187:0/171952674 shutdown_connections 2020-11-20T12:05:09.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.802+0000 7fcc42393700 1 -- 172.21.15.187:0/171952674 >> 172.21.15.187:0/171952674 conn(0x7fcc3c0bab30 msgr2=0x7fcc3c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.802+0000 7fcc42393700 1 -- 172.21.15.187:0/171952674 shutdown_connections 2020-11-20T12:05:09.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.802+0000 7fcc42393700 1 -- 172.21.15.187:0/171952674 wait complete. 2020-11-20T12:05:09.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.803+0000 7fcc42393700 1 Processor -- start 2020-11-20T12:05:09.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.804+0000 7fcc42393700 1 -- start start 2020-11-20T12:05:09.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.804+0000 7fcc42393700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c1496d0 0x7fcc3c147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.804+0000 7fcc42393700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc3c149ab0 0x7fcc3c0b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.804+0000 7fcc42393700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc3c0a06c0 0x7fcc3c0b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.804+0000 7fcc42393700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c147040 con 0x7fcc3c149ab0 2020-11-20T12:05:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.804+0000 7fcc42393700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c1472e0 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.804+0000 7fcc42393700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0b3e70 con 0x7fcc3c1496d0 2020-11-20T12:05:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.805+0000 7fcc41b92700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc3c0a06c0 0x7fcc3c0b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.805+0000 7fcc40b90700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc3c149ab0 0x7fcc3c0b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.805+0000 7fcc40b90700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc3c149ab0 0x7fcc3c0b2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:33004/0 (socket says 172.21.15.187:33004) 2020-11-20T12:05:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.805+0000 7fcc40b90700 1 -- 172.21.15.187:0/3138108013 learned_addr learned my addr 172.21.15.187:0/3138108013 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.805+0000 7fcc41391700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c1496d0 0x7fcc3c147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc41b92700 1 -- 172.21.15.187:0/3138108013 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c1496d0 msgr2=0x7fcc3c147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc41b92700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c1496d0 0x7fcc3c147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc41b92700 1 -- 172.21.15.187:0/3138108013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc3c149ab0 msgr2=0x7fcc3c0b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc41b92700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc3c149ab0 0x7fcc3c0b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc41b92700 1 -- 172.21.15.187:0/3138108013 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c146d30 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc41b92700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc3c0a06c0 0x7fcc3c0b2f70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fcc34001e70 tx=0x7fcc34015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc297fa700 1 -- 172.21.15.187:0/3138108013 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34014d80 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc297fa700 1 -- 172.21.15.187:0/3138108013 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3401f400 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.806+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c147c70 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.807+0000 7fcc297fa700 1 -- 172.21.15.187:0/3138108013 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3401f780 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.807+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c1482e0 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.808+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c1474a0 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.808+0000 7fcc297fa700 1 -- 172.21.15.187:0/3138108013 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcc34026030 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.809+0000 7fcc297fa700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc14056d20 0x7fcc140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.809+0000 7fcc297fa700 1 -- 172.21.15.187:0/3138108013 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcc34071370 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.809+0000 7fcc41391700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc14056d20 0x7fcc140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.810+0000 7fcc41391700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc14056d20 0x7fcc140591e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcc38001010 tx=0x7fcc3c0bb650).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.812+0000 7fcc297fa700 1 -- 172.21.15.187:0/3138108013 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc340467c0 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.989+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fcc3c147e60 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.990+0000 7fcc297fa700 1 -- 172.21.15.187:0/3138108013 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcc34024090 con 0x7fcc3c0a06c0 2020-11-20T12:05:09.995 INFO:teuthology.orchestra.run.smithi187.stdout:60129542149 2020-11-20T12:05:09.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc14056d20 msgr2=0x7fcc140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc14056d20 0x7fcc140591e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcc38001010 tx=0x7fcc3c0bb650).stop 2020-11-20T12:05:09.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc3c0a06c0 msgr2=0x7fcc3c0b2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc3c0a06c0 0x7fcc3c0b2f70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fcc34001e70 tx=0x7fcc34015000).stop 2020-11-20T12:05:09.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 shutdown_connections 2020-11-20T12:05:09.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc14056d20 0x7fcc140591e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc3c0a06c0 0x7fcc3c0b2f70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc3c1496d0 0x7fcc3c147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 --2- 172.21.15.187:0/3138108013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc3c149ab0 0x7fcc3c0b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.992+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 >> 172.21.15.187:0/3138108013 conn(0x7fcc3c0bab30 msgr2=0x7fcc3c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.993+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 shutdown_connections 2020-11-20T12:05:10.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:09.993+0000 7fcc42393700 1 -- 172.21.15.187:0/3138108013 wait complete. 2020-11-20T12:05:10.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.4 2020-11-20T12:05:10.020 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2020-11-20T12:05:10.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.255+0000 7f171d064700 1 Processor -- start 2020-11-20T12:05:10.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.256+0000 7f171d064700 1 -- start start 2020-11-20T12:05:10.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.257+0000 7f171d064700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 0x7f1718150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.257+0000 7f171d064700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f171814c260 0x7f171814c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.257+0000 7f171d064700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17180658a0 con 0x7f17180b1760 2020-11-20T12:05:10.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.257+0000 7f171d064700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1718150390 con 0x7f171814c260 2020-11-20T12:05:10.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.257+0000 7f171d064700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1718150630 con 0x7f17180b1380 2020-11-20T12:05:10.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.257+0000 7f17177fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f17180b1760 0x7f1718065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37352/0 (socket says 172.21.15.187:37352) 2020-11-20T12:05:10.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.257+0000 7f17177fe700 1 -- 172.21.15.187:0/3975098751 learned_addr learned my addr 172.21.15.187:0/3975098751 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:10.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.258+0000 7f171c863700 1 --2- 172.21.15.187:0/3975098751 >> v2:172.21.15.5:3300/0 conn(0x7f171814c260 0x7f171814c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.258+0000 7f1717fff700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 0x7f1718150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.258+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458943697 0 0) 0x7f17180658a0 con 0x7f17180b1760 2020-11-20T12:05:10.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.258+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1704003240 con 0x7f17180b1760 2020-11-20T12:05:10.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.258+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2341501977 0 0) 0x7f1704003240 con 0x7f17180b1760 2020-11-20T12:05:10.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.258+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 msgr2=0x7f1718150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.258+0000 7f1716ffd700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 0x7f1718150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 >> v2:172.21.15.5:3300/0 conn(0x7f171814c260 msgr2=0x7f171814c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 --2- 172.21.15.187:0/3975098751 >> v2:172.21.15.5:3300/0 conn(0x7f171814c260 0x7f171814c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718068510 con 0x7f17180b1760 2020-11-20T12:05:10.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f170c0014e0 con 0x7f17180b1760 2020-11-20T12:05:10.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 >> v1:172.21.15.187:6789/0 conn(0x7f17180b1760 legacy=0x7f1718065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1704003730 0x7f1704003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 0x7f17040044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17040049d0 0x7f1704004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1718150390 con 0x7f17040049d0 2020-11-20T12:05:10.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1718150630 con 0x7f1704003730 2020-11-20T12:05:10.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17040041d0 con 0x7f17180b1380 2020-11-20T12:05:10.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f1717fff700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1704003730 0x7f1704003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.259+0000 7f17177fe700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 0x7f17040044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f171c863700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17040049d0 0x7f1704004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1717fff700 1 -- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 msgr2=0x7f17040044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1717fff700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 0x7f17040044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1717fff700 1 -- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17040049d0 msgr2=0x7f1704004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1717fff700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17040049d0 0x7f1704004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1717fff700 1 -- 172.21.15.187:0/3975098751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704003240 con 0x7f1704003730 2020-11-20T12:05:10.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f171c863700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17040049d0 0x7f1704004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:10.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f17177fe700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17180b1380 0x7f17040044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:10.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1717fff700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1704003730 0x7f1704003b50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f170800bc00 tx=0x7f1708006df0).ready entity=mon.1 client_cookie=dbef71df1a7fbb32 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170800b310 con 0x7f1704003730 2020-11-20T12:05:10.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170800eb40 con 0x7f1704003730 2020-11-20T12:05:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.260+0000 7f1716ffd700 1 -- 172.21.15.187:0/3975098751 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170800eec0 con 0x7f1704003730 2020-11-20T12:05:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.261+0000 7f171d064700 1 -- 172.21.15.187:0/3975098751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1704003730 msgr2=0x7f1704003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.261+0000 7f171d064700 1 --2- 172.21.15.187:0/3975098751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1704003730 0x7f1704003b50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f170800bc00 tx=0x7f1708006df0).stop 2020-11-20T12:05:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.261+0000 7f171c863700 1 -- 172.21.15.187:0/3975098751 reap_dead start 2020-11-20T12:05:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.261+0000 7f171d064700 1 -- 172.21.15.187:0/3975098751 shutdown_connections 2020-11-20T12:05:10.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.261+0000 7f171d064700 1 -- 172.21.15.187:0/3975098751 >> 172.21.15.187:0/3975098751 conn(0x7f17180bab30 msgr2=0x7f17180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.261+0000 7f171d064700 1 -- 172.21.15.187:0/3975098751 shutdown_connections 2020-11-20T12:05:10.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.261+0000 7f171d064700 1 -- 172.21.15.187:0/3975098751 wait complete. 2020-11-20T12:05:10.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.263+0000 7f171d064700 1 Processor -- start 2020-11-20T12:05:10.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.263+0000 7f171d064700 1 -- start start 2020-11-20T12:05:10.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f171d064700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f171815c110 0x7f171815c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f171d064700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f171815c9e0 0x7f17180b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f171d064700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f171815ce60 0x7f17180b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f171d064700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17180b7490 con 0x7f171815ce60 2020-11-20T12:05:10.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f171d064700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171815d330 con 0x7f171815c110 2020-11-20T12:05:10.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f171d064700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17180b6660 con 0x7f171815c9e0 2020-11-20T12:05:10.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f1717fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f171815c110 0x7f171815c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f1717fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f171815c110 0x7f171815c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40760/0 (socket says 172.21.15.187:40760) 2020-11-20T12:05:10.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.264+0000 7f1717fff700 1 -- 172.21.15.187:0/2613899725 learned_addr learned my addr 172.21.15.187:0/2613899725 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:10.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f17177fe700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f171815c9e0 0x7f17180b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f171c863700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f171815ce60 0x7f17180b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f1717fff700 1 -- 172.21.15.187:0/2613899725 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f171815c9e0 msgr2=0x7f17180b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f1717fff700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f171815c9e0 0x7f17180b38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f1717fff700 1 -- 172.21.15.187:0/2613899725 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f171815ce60 msgr2=0x7f17180b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f1717fff700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f171815ce60 0x7f17180b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f1717fff700 1 -- 172.21.15.187:0/2613899725 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17180b7900 con 0x7f171815c110 2020-11-20T12:05:10.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f1717fff700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f171815c110 0x7f171815c4f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1708013000 tx=0x7f1708019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f16f7fff700 1 -- 172.21.15.187:0/2613899725 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170801d620 con 0x7f171815c110 2020-11-20T12:05:10.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f16f7fff700 1 -- 172.21.15.187:0/2613899725 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1708021070 con 0x7f171815c110 2020-11-20T12:05:10.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f16f7fff700 1 -- 172.21.15.187:0/2613899725 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170801d7a0 con 0x7f171815c110 2020-11-20T12:05:10.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.265+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17180b7da0 con 0x7f171815c110 2020-11-20T12:05:10.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.266+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17180b6420 con 0x7f171815c110 2020-11-20T12:05:10.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.267+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1718061a20 con 0x7f171815c110 2020-11-20T12:05:10.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.267+0000 7f16f7fff700 1 -- 172.21.15.187:0/2613899725 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f170801db50 con 0x7f171815c110 2020-11-20T12:05:10.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.268+0000 7f16f7fff700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16f0056d50 0x7f16f0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.268+0000 7f16f7fff700 1 -- 172.21.15.187:0/2613899725 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1708071fc0 con 0x7f171815c110 2020-11-20T12:05:10.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.268+0000 7f17177fe700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16f0056d50 0x7f16f0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.268+0000 7f17177fe700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16f0056d50 0x7f16f0059210 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f17180b33d0 tx=0x7f170c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.271+0000 7f16f7fff700 1 -- 172.21.15.187:0/2613899725 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1708047490 con 0x7f171815c110 2020-11-20T12:05:10.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.455+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1718067d30 con 0x7f171815c110 2020-11-20T12:05:10.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.456+0000 7f16f7fff700 1 -- 172.21.15.187:0/2613899725 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f170804aab0 con 0x7f171815c110 2020-11-20T12:05:10.461 INFO:teuthology.orchestra.run.smithi187.stdout:60129542150 2020-11-20T12:05:10.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16f0056d50 msgr2=0x7f16f0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16f0056d50 0x7f16f0059210 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f17180b33d0 tx=0x7f170c009040).stop 2020-11-20T12:05:10.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f171815c110 msgr2=0x7f171815c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f171815c110 0x7f171815c4f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1708013000 tx=0x7f1708019000).stop 2020-11-20T12:05:10.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 shutdown_connections 2020-11-20T12:05:10.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16f0056d50 0x7f16f0059210 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f171815ce60 0x7f17180b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f171815c110 0x7f171815c4f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 --2- 172.21.15.187:0/2613899725 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f171815c9e0 0x7f17180b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 >> 172.21.15.187:0/2613899725 conn(0x7f17180bab30 msgr2=0x7f171814b840 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 shutdown_connections 2020-11-20T12:05:10.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.458+0000 7f171d064700 1 -- 172.21.15.187:0/2613899725 wait complete. 2020-11-20T12:05:10.486 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.5 2020-11-20T12:05:10.487 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2020-11-20T12:05:10.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.721+0000 7efdfac21700 1 Processor -- start 2020-11-20T12:05:10.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.721+0000 7efdfac21700 1 -- start start 2020-11-20T12:05:10.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.723+0000 7efdfac21700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efdf40654a0 0x7efdf40b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.723+0000 7efdfac21700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efdf4065880 0x7efdf4144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.723+0000 7efdfac21700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf40b08e0 con 0x7efdf40b6c10 2020-11-20T12:05:10.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.723+0000 7efdfac21700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf40b0b80 con 0x7efdf4065880 2020-11-20T12:05:10.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.723+0000 7efdfac21700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf40b05f0 con 0x7efdf40654a0 2020-11-20T12:05:10.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.723+0000 7efdf89bd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efdf40654a0 0x7efdf40b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.723+0000 7efdf89bd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efdf40654a0 0x7efdf40b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40766/0 (socket says 172.21.15.187:40766) 2020-11-20T12:05:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.723+0000 7efdf89bd700 1 -- 172.21.15.187:0/1341218759 learned_addr learned my addr 172.21.15.187:0/1341218759 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.724+0000 7efdebfff700 1 --2- 172.21.15.187:0/1341218759 >> v2:172.21.15.5:3300/0 conn(0x7efdf4065880 0x7efdf4144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.724+0000 7efdf89bd700 1 -- 172.21.15.187:0/1341218759 >> v2:172.21.15.5:3300/0 conn(0x7efdf4065880 msgr2=0x7efdf4144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.724+0000 7efdf89bd700 1 --2- 172.21.15.187:0/1341218759 >> v2:172.21.15.5:3300/0 conn(0x7efdf4065880 0x7efdf4144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.724+0000 7efdf89bd700 1 -- 172.21.15.187:0/1341218759 >> v1:172.21.15.5:6789/0 conn(0x7efdf40b6c10 legacy=0x7efdf40b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.724+0000 7efdf89bd700 1 -- 172.21.15.187:0/1341218759 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf40676f0 con 0x7efdf40654a0 2020-11-20T12:05:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.724+0000 7efdf89bd700 1 --2- 172.21.15.187:0/1341218759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf40b1650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7efdf40b8ab0 tx=0x7efdf0006eb0).ready entity=mon.1 client_cookie=a991b5d4142cc668 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.724+0000 7efdeb7fe700 1 -- 172.21.15.187:0/1341218759 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0009960 con 0x7efdf40654a0 2020-11-20T12:05:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdeb7fe700 1 -- 172.21.15.187:0/1341218759 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf0009ac0 con 0x7efdf40654a0 2020-11-20T12:05:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdeb7fe700 1 -- 172.21.15.187:0/1341218759 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0009e40 con 0x7efdf40654a0 2020-11-20T12:05:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdfac21700 1 -- 172.21.15.187:0/1341218759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 msgr2=0x7efdf40b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdfac21700 1 --2- 172.21.15.187:0/1341218759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf40b1650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7efdf40b8ab0 tx=0x7efdf0006eb0).stop 2020-11-20T12:05:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdfac21700 1 -- 172.21.15.187:0/1341218759 shutdown_connections 2020-11-20T12:05:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdfac21700 1 --2- 172.21.15.187:0/1341218759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf40b1650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdfac21700 1 --2- 172.21.15.187:0/1341218759 >> v2:172.21.15.5:3300/0 conn(0x7efdf4065880 0x7efdf4144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdfac21700 1 -- 172.21.15.187:0/1341218759 >> 172.21.15.187:0/1341218759 conn(0x7efdf40bacd0 msgr2=0x7efdf40bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdfac21700 1 -- 172.21.15.187:0/1341218759 shutdown_connections 2020-11-20T12:05:10.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.725+0000 7efdfac21700 1 -- 172.21.15.187:0/1341218759 wait complete. 2020-11-20T12:05:10.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.727+0000 7efdfac21700 1 Processor -- start 2020-11-20T12:05:10.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.727+0000 7efdfac21700 1 -- start start 2020-11-20T12:05:10.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdfac21700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf41589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdfac21700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf4065880 0x7efdf4147990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdfac21700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf40b6c10 0x7efdf409dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdfac21700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4134830 con 0x7efdf4065880 2020-11-20T12:05:10.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdfac21700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf41582a0 con 0x7efdf40654a0 2020-11-20T12:05:10.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdfac21700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf409f870 con 0x7efdf40b6c10 2020-11-20T12:05:10.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdf89bd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf41589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdf89bd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf41589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40772/0 (socket says 172.21.15.187:40772) 2020-11-20T12:05:10.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdf89bd700 1 -- 172.21.15.187:0/416907739 learned_addr learned my addr 172.21.15.187:0/416907739 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:10.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdf91be700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf40b6c10 0x7efdf409dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdebfff700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf4065880 0x7efdf4147990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.728+0000 7efdf89bd700 1 -- 172.21.15.187:0/416907739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf40b6c10 msgr2=0x7efdf409dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efdf89bd700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf40b6c10 0x7efdf409dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efdf89bd700 1 -- 172.21.15.187:0/416907739 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf4065880 msgr2=0x7efdf4147990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efdf89bd700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf4065880 0x7efdf4147990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efdf89bd700 1 -- 172.21.15.187:0/416907739 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4158500 con 0x7efdf40654a0 2020-11-20T12:05:10.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efdf89bd700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf41589a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efdf000b1b0 tx=0x7efdf0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efde8ff9700 1 -- 172.21.15.187:0/416907739 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0011dd0 con 0x7efdf40654a0 2020-11-20T12:05:10.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efde8ff9700 1 -- 172.21.15.187:0/416907739 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf001e460 con 0x7efdf40654a0 2020-11-20T12:05:10.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf40b3c80 con 0x7efdf40654a0 2020-11-20T12:05:10.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efde8ff9700 1 -- 172.21.15.187:0/416907739 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf001e7e0 con 0x7efdf40654a0 2020-11-20T12:05:10.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.729+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf40b3970 con 0x7efdf40654a0 2020-11-20T12:05:10.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.730+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf4063600 con 0x7efdf40654a0 2020-11-20T12:05:10.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.731+0000 7efde8ff9700 1 -- 172.21.15.187:0/416907739 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efdf0025030 con 0x7efdf40654a0 2020-11-20T12:05:10.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.731+0000 7efde8ff9700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d50 0x7efdcc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.731+0000 7efde8ff9700 1 -- 172.21.15.187:0/416907739 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efdf00717b0 con 0x7efdf40654a0 2020-11-20T12:05:10.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.732+0000 7efdebfff700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d50 0x7efdcc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.732+0000 7efdebfff700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d50 0x7efdcc059210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7efdf4148380 tx=0x7efde4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.735+0000 7efde8ff9700 1 -- 172.21.15.187:0/416907739 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efdf0046cb0 con 0x7efdf40654a0 2020-11-20T12:05:10.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.921+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7efdf4067880 con 0x7efdf40654a0 2020-11-20T12:05:10.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.922+0000 7efde8ff9700 1 -- 172.21.15.187:0/416907739 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efdf0023020 con 0x7efdf40654a0 2020-11-20T12:05:10.926 INFO:teuthology.orchestra.run.smithi187.stdout:60129542150 2020-11-20T12:05:10.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.924+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d50 msgr2=0x7efdcc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.924+0000 7efdfac21700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d50 0x7efdcc059210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7efdf4148380 tx=0x7efde4008040).stop 2020-11-20T12:05:10.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.924+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 msgr2=0x7efdf41589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.924+0000 7efdfac21700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf41589a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efdf000b1b0 tx=0x7efdf0015000).stop 2020-11-20T12:05:10.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.925+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 shutdown_connections 2020-11-20T12:05:10.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.925+0000 7efdfac21700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d50 0x7efdcc059210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.925+0000 7efdfac21700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf40b6c10 0x7efdf409dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.925+0000 7efdfac21700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf40654a0 0x7efdf41589a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.925+0000 7efdfac21700 1 --2- 172.21.15.187:0/416907739 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf4065880 0x7efdf4147990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.925+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 >> 172.21.15.187:0/416907739 conn(0x7efdf40bacd0 msgr2=0x7efdf40afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.925+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 shutdown_connections 2020-11-20T12:05:10.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:10.925+0000 7efdfac21700 1 -- 172.21.15.187:0/416907739 wait complete. 2020-11-20T12:05:10.950 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.6 2020-11-20T12:05:11.018 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2020-11-20T12:05:11.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.266+0000 7fb5234ec700 1 Processor -- start 2020-11-20T12:05:11.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.266+0000 7fb5234ec700 1 -- start start 2020-11-20T12:05:11.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.268+0000 7fb5234ec700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c0b1380 0x7fb51c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.268+0000 7fb5234ec700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb51c14c260 0x7fb51c14c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.268+0000 7fb5234ec700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51c0658a0 con 0x7fb51c0b1760 2020-11-20T12:05:11.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.268+0000 7fb5234ec700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51c150390 con 0x7fb51c14c260 2020-11-20T12:05:11.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.268+0000 7fb5234ec700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c150630 con 0x7fb51c0b1380 2020-11-20T12:05:11.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.268+0000 7fb522ceb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb51c14c260 0x7fb51c14c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.268+0000 7fb5224ea700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c0b1380 0x7fb51c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.269+0000 7fb522ceb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb51c14c260 0x7fb51c14c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:33048/0 (socket says 172.21.15.187:33048) 2020-11-20T12:05:11.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.269+0000 7fb522ceb700 1 -- 172.21.15.187:0/133072954 learned_addr learned my addr 172.21.15.187:0/133072954 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:11.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.270+0000 7fb5224ea700 1 -- 172.21.15.187:0/133072954 >> v2:172.21.15.5:3300/0 conn(0x7fb51c14c260 msgr2=0x7fb51c14c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:11.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.270+0000 7fb5224ea700 1 --2- 172.21.15.187:0/133072954 >> v2:172.21.15.5:3300/0 conn(0x7fb51c14c260 0x7fb51c14c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.270+0000 7fb5224ea700 1 -- 172.21.15.187:0/133072954 >> v1:172.21.15.5:6789/0 conn(0x7fb51c0b1760 legacy=0x7fb51c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.270+0000 7fb5224ea700 1 -- 172.21.15.187:0/133072954 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c068510 con 0x7fb51c0b1380 2020-11-20T12:05:11.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.270+0000 7fb522ceb700 1 --2- 172.21.15.187:0/133072954 >> v2:172.21.15.5:3300/0 conn(0x7fb51c14c260 0x7fb51c14c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:05:11.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.270+0000 7fb5224ea700 1 --2- 172.21.15.187:0/133072954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c0b1380 0x7fb51c150ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fb51800b9a0 tx=0x7fb518006df0).ready entity=mon.2 client_cookie=1e3faa32527f6ac7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:11.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.270+0000 7fb5214e8700 1 -- 172.21.15.187:0/133072954 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5180097b0 con 0x7fb51c0b1380 2020-11-20T12:05:11.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5214e8700 1 -- 172.21.15.187:0/133072954 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb518009910 con 0x7fb51c0b1380 2020-11-20T12:05:11.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5214e8700 1 -- 172.21.15.187:0/133072954 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518009c90 con 0x7fb51c0b1380 2020-11-20T12:05:11.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5234ec700 1 -- 172.21.15.187:0/133072954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c0b1380 msgr2=0x7fb51c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5234ec700 1 --2- 172.21.15.187:0/133072954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c0b1380 0x7fb51c150ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fb51800b9a0 tx=0x7fb518006df0).stop 2020-11-20T12:05:11.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5234ec700 1 -- 172.21.15.187:0/133072954 shutdown_connections 2020-11-20T12:05:11.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5234ec700 1 --2- 172.21.15.187:0/133072954 >> v2:172.21.15.5:3300/0 conn(0x7fb51c14c260 0x7fb51c14c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5234ec700 1 --2- 172.21.15.187:0/133072954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c0b1380 0x7fb51c150ad0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5234ec700 1 -- 172.21.15.187:0/133072954 >> 172.21.15.187:0/133072954 conn(0x7fb51c0bab30 msgr2=0x7fb51c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:11.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5234ec700 1 -- 172.21.15.187:0/133072954 shutdown_connections 2020-11-20T12:05:11.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.271+0000 7fb5234ec700 1 -- 172.21.15.187:0/133072954 wait complete. 2020-11-20T12:05:11.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.273+0000 7fb5234ec700 1 Processor -- start 2020-11-20T12:05:11.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.274+0000 7fb5234ec700 1 -- start start 2020-11-20T12:05:11.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5234ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb51c0b1380 0x7fb51c09fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5234ec700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb51c0b1760 0x7fb51c149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5234ec700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c14c260 0x7fb51c0b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5234ec700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c09dfb0 con 0x7fb51c0b1760 2020-11-20T12:05:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5234ec700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0a0400 con 0x7fb51c0b1380 2020-11-20T12:05:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5234ec700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0a06a0 con 0x7fb51c14c260 2020-11-20T12:05:11.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5224ea700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb51c0b1380 0x7fb51c09fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5224ea700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb51c0b1380 0x7fb51c09fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40786/0 (socket says 172.21.15.187:40786) 2020-11-20T12:05:11.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb5224ea700 1 -- 172.21.15.187:0/1035445613 learned_addr learned my addr 172.21.15.187:0/1035445613 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:11.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb521ce9700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb51c0b1760 0x7fb51c149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.275+0000 7fb522ceb700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c14c260 0x7fb51c0b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb5224ea700 1 -- 172.21.15.187:0/1035445613 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c14c260 msgr2=0x7fb51c0b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb5224ea700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c14c260 0x7fb51c0b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb5224ea700 1 -- 172.21.15.187:0/1035445613 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb51c0b1760 msgr2=0x7fb51c149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb5224ea700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb51c0b1760 0x7fb51c149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb5224ea700 1 -- 172.21.15.187:0/1035445613 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c0a0940 con 0x7fb51c0b1380 2020-11-20T12:05:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb5224ea700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb51c0b1380 0x7fb51c09fe10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fb518006e30 tx=0x7fb518015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb50a7fc700 1 -- 172.21.15.187:0/1035445613 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518011dd0 con 0x7fb51c0b1380 2020-11-20T12:05:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb50a7fc700 1 -- 172.21.15.187:0/1035445613 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51801e460 con 0x7fb51c0b1380 2020-11-20T12:05:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb50a7fc700 1 -- 172.21.15.187:0/1035445613 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51801e800 con 0x7fb51c0b1380 2020-11-20T12:05:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.276+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51c14a000 con 0x7fb51c0b1380 2020-11-20T12:05:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.277+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51c09bf80 con 0x7fb51c0b1380 2020-11-20T12:05:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.278+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51c061a20 con 0x7fb51c0b1380 2020-11-20T12:05:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.278+0000 7fb50a7fc700 1 -- 172.21.15.187:0/1035445613 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb518025030 con 0x7fb51c0b1380 2020-11-20T12:05:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.279+0000 7fb50a7fc700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4f4056d20 0x7fb4f40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.279+0000 7fb50a7fc700 1 -- 172.21.15.187:0/1035445613 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb5180716a0 con 0x7fb51c0b1380 2020-11-20T12:05:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.279+0000 7fb521ce9700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4f4056d20 0x7fb4f40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.280+0000 7fb521ce9700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4f4056d20 0x7fb4f40591e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb51c0663b0 tx=0x7fb50c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:11.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.283+0000 7fb50a7fc700 1 -- 172.21.15.187:0/1035445613 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb518046ad0 con 0x7fb51c0b1380 2020-11-20T12:05:11.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.466+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb51c0a0ad0 con 0x7fb51c0b1380 2020-11-20T12:05:11.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.467+0000 7fb50a7fc700 1 -- 172.21.15.187:0/1035445613 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb518023070 con 0x7fb51c0b1380 2020-11-20T12:05:11.471 INFO:teuthology.orchestra.run.smithi187.stdout:60129542150 2020-11-20T12:05:11.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4f4056d20 msgr2=0x7fb4f40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4f4056d20 0x7fb4f40591e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb51c0663b0 tx=0x7fb50c009040).stop 2020-11-20T12:05:11.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb51c0b1380 msgr2=0x7fb51c09fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb51c0b1380 0x7fb51c09fe10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fb518006e30 tx=0x7fb518015000).stop 2020-11-20T12:05:11.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 shutdown_connections 2020-11-20T12:05:11.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4f4056d20 0x7fb4f40591e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb51c14c260 0x7fb51c0b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb51c0b1380 0x7fb51c09fe10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 --2- 172.21.15.187:0/1035445613 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb51c0b1760 0x7fb51c149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 >> 172.21.15.187:0/1035445613 conn(0x7fb51c0bab30 msgr2=0x7fb51c14e440 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:11.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.469+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 shutdown_connections 2020-11-20T12:05:11.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.470+0000 7fb5234ec700 1 -- 172.21.15.187:0/1035445613 wait complete. 2020-11-20T12:05:11.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2020-11-20T12:05:11.498 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-11-20T12:05:11.498 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:05:11.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.735+0000 7f4756384700 1 Processor -- start 2020-11-20T12:05:11.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.735+0000 7f4756384700 1 -- start start 2020-11-20T12:05:11.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.736+0000 7f4756384700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4750065190 0x7f4750065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.736+0000 7f4756384700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4750065f90 0x7f4750143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.736+0000 7f4756384700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500b13e0 con 0x7f4750143c70 2020-11-20T12:05:11.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.736+0000 7f4756384700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f475014ab30 con 0x7f4750065f90 2020-11-20T12:05:11.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.736+0000 7f4756384700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4750062c10 con 0x7f4750065190 2020-11-20T12:05:11.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.737+0000 7f4755b83700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4750143c70 0x7f475009ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37402/0 (socket says 172.21.15.187:37402) 2020-11-20T12:05:11.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.737+0000 7f4755b83700 1 -- 172.21.15.187:0/1812935871 learned_addr learned my addr 172.21.15.187:0/1812935871 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:11.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.737+0000 7f4755382700 1 --2- 172.21.15.187:0/1812935871 >> v2:172.21.15.187:3300/0 conn(0x7f4750065190 0x7f4750065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.737+0000 7f4754b81700 1 --2- 172.21.15.187:0/1812935871 >> v2:172.21.15.5:3300/0 conn(0x7f4750065f90 0x7f4750143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.738+0000 7f4755382700 1 -- 172.21.15.187:0/1812935871 >> v2:172.21.15.5:3300/0 conn(0x7f4750065f90 msgr2=0x7f4750143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:11.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.738+0000 7f4755382700 1 --2- 172.21.15.187:0/1812935871 >> v2:172.21.15.5:3300/0 conn(0x7f4750065f90 0x7f4750143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.738+0000 7f4755382700 1 -- 172.21.15.187:0/1812935871 >> v1:172.21.15.187:6789/0 conn(0x7f4750143c70 legacy=0x7f475009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.738+0000 7f4755382700 1 -- 172.21.15.187:0/1812935871 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47500b1640 con 0x7f4750065190 2020-11-20T12:05:11.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.738+0000 7f4755382700 1 --2- 172.21.15.187:0/1812935871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750065190 0x7f4750065570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f475005c8b0 tx=0x7f474c006df0).ready entity=mon.1 client_cookie=192987f3069300b4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:11.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.738+0000 7f4743fff700 1 -- 172.21.15.187:0/1812935871 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c009840 con 0x7f4750065190 2020-11-20T12:05:11.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.738+0000 7f4743fff700 1 -- 172.21.15.187:0/1812935871 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c0099a0 con 0x7f4750065190 2020-11-20T12:05:11.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.738+0000 7f4743fff700 1 -- 172.21.15.187:0/1812935871 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c009d20 con 0x7f4750065190 2020-11-20T12:05:11.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.739+0000 7f4756384700 1 -- 172.21.15.187:0/1812935871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750065190 msgr2=0x7f4750065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.739+0000 7f4756384700 1 --2- 172.21.15.187:0/1812935871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750065190 0x7f4750065570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f475005c8b0 tx=0x7f474c006df0).stop 2020-11-20T12:05:11.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.739+0000 7f4756384700 1 -- 172.21.15.187:0/1812935871 shutdown_connections 2020-11-20T12:05:11.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.739+0000 7f4756384700 1 --2- 172.21.15.187:0/1812935871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750065190 0x7f4750065570 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.739+0000 7f4756384700 1 --2- 172.21.15.187:0/1812935871 >> v2:172.21.15.5:3300/0 conn(0x7f4750065f90 0x7f4750143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.739+0000 7f4756384700 1 -- 172.21.15.187:0/1812935871 >> 172.21.15.187:0/1812935871 conn(0x7f47500bab30 msgr2=0x7f47500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:11.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.739+0000 7f4756384700 1 -- 172.21.15.187:0/1812935871 shutdown_connections 2020-11-20T12:05:11.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.739+0000 7f4756384700 1 -- 172.21.15.187:0/1812935871 wait complete. 2020-11-20T12:05:11.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.741+0000 7f4756384700 1 Processor -- start 2020-11-20T12:05:11.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.741+0000 7f4756384700 1 -- start start 2020-11-20T12:05:11.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4756384700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4750065190 0x7f47500990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4756384700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4750065f90 0x7f47500995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4756384700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750143c70 0x7f4750158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4756384700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47500b4650 con 0x7f4750065190 2020-11-20T12:05:11.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4756384700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47500b48f0 con 0x7f4750143c70 2020-11-20T12:05:11.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4756384700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47500b2630 con 0x7f4750065f90 2020-11-20T12:05:11.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4755b83700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750143c70 0x7f4750158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4755b83700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750143c70 0x7f4750158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40804/0 (socket says 172.21.15.187:40804) 2020-11-20T12:05:11.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4755b83700 1 -- 172.21.15.187:0/4013054513 learned_addr learned my addr 172.21.15.187:0/4013054513 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:11.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4755382700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4750065190 0x7f47500990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4754b81700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4750065f90 0x7f47500995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.742+0000 7f4755b83700 1 -- 172.21.15.187:0/4013054513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4750065f90 msgr2=0x7f47500995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f4755b83700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4750065f90 0x7f47500995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f4755b83700 1 -- 172.21.15.187:0/4013054513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4750065190 msgr2=0x7f47500990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f4755b83700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4750065190 0x7f47500990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f4755b83700 1 -- 172.21.15.187:0/4013054513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4750066a40 con 0x7f4750143c70 2020-11-20T12:05:11.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f4755b83700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750143c70 0x7f4750158770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f474800ca60 tx=0x7f4748007f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:11.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f47417fa700 1 -- 172.21.15.187:0/4013054513 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748005b80 con 0x7f4750143c70 2020-11-20T12:05:11.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f47417fa700 1 -- 172.21.15.187:0/4013054513 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748005d00 con 0x7f4750143c70 2020-11-20T12:05:11.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4750066c30 con 0x7f4750143c70 2020-11-20T12:05:11.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f47417fa700 1 -- 172.21.15.187:0/4013054513 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748007360 con 0x7f4750143c70 2020-11-20T12:05:11.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.743+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4750158ea0 con 0x7f4750143c70 2020-11-20T12:05:11.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.744+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47500b4ab0 con 0x7f4750143c70 2020-11-20T12:05:11.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.745+0000 7f47417fa700 1 -- 172.21.15.187:0/4013054513 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4748010030 con 0x7f4750143c70 2020-11-20T12:05:11.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.745+0000 7f47417fa700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4728056d20 0x7f47280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:11.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.745+0000 7f47417fa700 1 -- 172.21.15.187:0/4013054513 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f474806fec0 con 0x7f4750143c70 2020-11-20T12:05:11.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.746+0000 7f4755382700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4728056d20 0x7f47280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:11.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.746+0000 7f4755382700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4728056d20 0x7f47280591e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f474c00c010 tx=0x7f474c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:11.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.750+0000 7f47417fa700 1 -- 172.21.15.187:0/4013054513 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4748045290 con 0x7f4750143c70 2020-11-20T12:05:11.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.930+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4750099dd0 con 0x7f4728056d20 2020-11-20T12:05:11.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.933+0000 7f47417fa700 1 -- 172.21.15.187:0/4013054513 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82127 (secure 0 0 0) 0x7f4750099dd0 con 0x7f4728056d20 2020-11-20T12:05:11.939 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:11.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.936+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4728056d20 msgr2=0x7f47280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.936+0000 7f4756384700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4728056d20 0x7f47280591e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f474c00c010 tx=0x7f474c015000).stop 2020-11-20T12:05:11.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.936+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750143c70 msgr2=0x7f4750158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.936+0000 7f4756384700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750143c70 0x7f4750158770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f474800ca60 tx=0x7f4748007f60).stop 2020-11-20T12:05:11.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.937+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 shutdown_connections 2020-11-20T12:05:11.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.937+0000 7f4756384700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4728056d20 0x7f47280591e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.937+0000 7f4756384700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4750065190 0x7f47500990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.937+0000 7f4756384700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4750143c70 0x7f4750158770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.937+0000 7f4756384700 1 --2- 172.21.15.187:0/4013054513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4750065f90 0x7f47500995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.937+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 >> 172.21.15.187:0/4013054513 conn(0x7f47500bab30 msgr2=0x7f47500b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:11.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.937+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 shutdown_connections 2020-11-20T12:05:11.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:11.937+0000 7f4756384700 1 -- 172.21.15.187:0/4013054513 wait complete. 2020-11-20T12:05:11.944 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2020-11-20T12:05:11.961 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2020-11-20T12:05:11.527701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":40700,"kb_used_data":1916,"kb_used_omap":0,"kb_used_meta":38720,"kb_avail":754934020,"statfs":{"total":773094113280,"available":773052436480,"internally_reserved":0,"allocated":1961984,"data_stored":710013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39649280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004033"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.930064+0000","last_change":"2020-11-20T12:04:57.909926+0000","last_active":"2020-11-20T12:04:57.930064+0000","last_peered":"2020-11-20T12:04:57.930064+0000","last_clean":"2020-11-20T12:04:57.930064+0000","last_became_active":"2020-11-20T12:04:57.908811+0000","last_became_peered":"2020-11-20T12:04:57.908811+0000","last_unstale":"2020-11-20T12:04:57.930064+0000","last_undegraded":"2020-11-20T12:04:57.930064+0000","last_fullsized":"2020-11-20T12:04:57.930064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.888201+0000","last_change":"2020-11-20T12:04:53.296291+0000","last_active":"2020-11-20T12:04:57.888201+0000","last_peered":"2020-11-20T12:04:57.888201+0000","last_clean":"2020-11-20T12:04:57.888201+0000","last_became_active":"2020-11-20T12:04:53.294169+0000","last_became_peered":"2020-11-20T12:04:53.294169+0000","last_unstale":"2020-11-20T12:04:57.888201+0000","last_undegraded":"2020-11-20T12:04:57.888201+0000","last_fullsized":"2020-11-20T12:04:57.888201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"22'6","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.943940+0000","last_change":"2020-11-20T12:04:57.366733+0000","last_active":"2020-11-20T12:04:57.943940+0000","last_peered":"2020-11-20T12:04:57.943940+0000","last_clean":"2020-11-20T12:04:57.943940+0000","last_became_active":"2020-11-20T12:04:57.365105+0000","last_became_peered":"2020-11-20T12:04:57.365105+0000","last_unstale":"2020-11-20T12:04:57.943940+0000","last_undegraded":"2020-11-20T12:04:57.943940+0000","last_fullsized":"2020-11-20T12:04:57.943940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.930146+0000","last_change":"2020-11-20T12:04:57.911885+0000","last_active":"2020-11-20T12:04:57.930146+0000","last_peered":"2020-11-20T12:04:57.930146+0000","last_clean":"2020-11-20T12:04:57.930146+0000","last_became_active":"2020-11-20T12:04:57.910770+0000","last_became_peered":"2020-11-20T12:04:57.910770+0000","last_unstale":"2020-11-20T12:04:57.930146+0000","last_undegraded":"2020-11-20T12:04:57.930146+0000","last_fullsized":"2020-11-20T12:04:57.930146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.882829+0000","last_change":"2020-11-20T12:04:53.294740+0000","last_active":"2020-11-20T12:04:57.882829+0000","last_peered":"2020-11-20T12:04:57.882829+0000","last_clean":"2020-11-20T12:04:57.882829+0000","last_became_active":"2020-11-20T12:04:53.293882+0000","last_became_peered":"2020-11-20T12:04:53.293882+0000","last_unstale":"2020-11-20T12:04:57.882829+0000","last_undegraded":"2020-11-20T12:04:57.882829+0000","last_fullsized":"2020-11-20T12:04:57.882829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"22'5","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.906914+0000","last_change":"2020-11-20T12:04:57.345878+0000","last_active":"2020-11-20T12:04:57.906914+0000","last_peered":"2020-11-20T12:04:57.906914+0000","last_clean":"2020-11-20T12:04:57.906914+0000","last_became_active":"2020-11-20T12:04:57.344954+0000","last_became_peered":"2020-11-20T12:04:57.344954+0000","last_unstale":"2020-11-20T12:04:57.906914+0000","last_undegraded":"2020-11-20T12:04:57.906914+0000","last_fullsized":"2020-11-20T12:04:57.906914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.942423+0000","last_change":"2020-11-20T12:04:57.927464+0000","last_active":"2020-11-20T12:04:57.942423+0000","last_peered":"2020-11-20T12:04:57.942423+0000","last_clean":"2020-11-20T12:04:57.942423+0000","last_became_active":"2020-11-20T12:04:57.926845+0000","last_became_peered":"2020-11-20T12:04:57.926845+0000","last_unstale":"2020-11-20T12:04:57.942423+0000","last_undegraded":"2020-11-20T12:04:57.942423+0000","last_fullsized":"2020-11-20T12:04:57.942423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.883110+0000","last_change":"2020-11-20T12:04:53.299007+0000","last_active":"2020-11-20T12:04:57.883110+0000","last_peered":"2020-11-20T12:04:57.883110+0000","last_clean":"2020-11-20T12:04:57.883110+0000","last_became_active":"2020-11-20T12:04:53.297849+0000","last_became_peered":"2020-11-20T12:04:53.297849+0000","last_unstale":"2020-11-20T12:04:57.883110+0000","last_undegraded":"2020-11-20T12:04:57.883110+0000","last_fullsized":"2020-11-20T12:04:57.883110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"22'8","reported_seq":"18","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.933150+0000","last_change":"2020-11-20T12:04:57.359175+0000","last_active":"2020-11-20T12:04:57.933150+0000","last_peered":"2020-11-20T12:04:57.933150+0000","last_clean":"2020-11-20T12:04:57.933150+0000","last_became_active":"2020-11-20T12:04:57.352579+0000","last_became_peered":"2020-11-20T12:04:57.352579+0000","last_unstale":"2020-11-20T12:04:57.933150+0000","last_undegraded":"2020-11-20T12:04:57.933150+0000","last_fullsized":"2020-11-20T12:04:57.933150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.896690+0000","last_change":"2020-11-20T12:04:57.890119+0000","last_active":"2020-11-20T12:04:57.896690+0000","last_peered":"2020-11-20T12:04:57.896690+0000","last_clean":"2020-11-20T12:04:57.896690+0000","last_became_active":"2020-11-20T12:04:57.889641+0000","last_became_peered":"2020-11-20T12:04:57.889641+0000","last_unstale":"2020-11-20T12:04:57.896690+0000","last_undegraded":"2020-11-20T12:04:57.896690+0000","last_fullsized":"2020-11-20T12:04:57.896690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.882744+0000","last_change":"2020-11-20T12:04:54.004976+0000","last_active":"2020-11-20T12:04:57.882744+0000","last_peered":"2020-11-20T12:04:57.882744+0000","last_clean":"2020-11-20T12:04:57.882744+0000","last_became_active":"2020-11-20T12:04:54.004416+0000","last_became_peered":"2020-11-20T12:04:54.004416+0000","last_unstale":"2020-11-20T12:04:57.882744+0000","last_undegraded":"2020-11-20T12:04:57.882744+0000","last_fullsized":"2020-11-20T12:04:57.882744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'2","reported_seq":"18","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.880881+0000","last_change":"2020-11-20T12:04:49.157393+0000","last_active":"2020-11-20T12:04:57.880881+0000","last_peered":"2020-11-20T12:04:57.880881+0000","last_clean":"2020-11-20T12:04:57.880881+0000","last_became_active":"2020-11-20T12:04:49.156248+0000","last_became_peered":"2020-11-20T12:04:49.156248+0000","last_unstale":"2020-11-20T12:04:57.880881+0000","last_undegraded":"2020-11-20T12:04:57.880881+0000","last_fullsized":"2020-11-20T12:04:57.880881+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:44.517280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:44.517280+0000","last_clean_scrub_stamp":"2020-11-20T12:04:44.517280+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"22'1","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.929220+0000","last_change":"2020-11-20T12:04:57.346452+0000","last_active":"2020-11-20T12:04:57.929220+0000","last_peered":"2020-11-20T12:04:57.929220+0000","last_clean":"2020-11-20T12:04:57.929220+0000","last_became_active":"2020-11-20T12:04:57.345332+0000","last_became_peered":"2020-11-20T12:04:57.345332+0000","last_unstale":"2020-11-20T12:04:57.929220+0000","last_undegraded":"2020-11-20T12:04:57.929220+0000","last_fullsized":"2020-11-20T12:04:57.929220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.908970+0000","last_change":"2020-11-20T12:04:57.900065+0000","last_active":"2020-11-20T12:04:57.908970+0000","last_peered":"2020-11-20T12:04:57.908970+0000","last_clean":"2020-11-20T12:04:57.908970+0000","last_became_active":"2020-11-20T12:04:57.897190+0000","last_became_peered":"2020-11-20T12:04:57.897190+0000","last_unstale":"2020-11-20T12:04:57.908970+0000","last_undegraded":"2020-11-20T12:04:57.908970+0000","last_fullsized":"2020-11-20T12:04:57.908970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.882727+0000","last_change":"2020-11-20T12:04:53.297146+0000","last_active":"2020-11-20T12:04:57.882727+0000","last_peered":"2020-11-20T12:04:57.882727+0000","last_clean":"2020-11-20T12:04:57.882727+0000","last_became_active":"2020-11-20T12:04:53.296376+0000","last_became_peered":"2020-11-20T12:04:53.296376+0000","last_unstale":"2020-11-20T12:04:57.882727+0000","last_undegraded":"2020-11-20T12:04:57.882727+0000","last_fullsized":"2020-11-20T12:04:57.882727+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"10","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.884170+0000","last_change":"2020-11-20T12:04:57.358142+0000","last_active":"2020-11-20T12:04:57.884170+0000","last_peered":"2020-11-20T12:04:57.884170+0000","last_clean":"2020-11-20T12:04:57.884170+0000","last_became_active":"2020-11-20T12:04:57.356779+0000","last_became_peered":"2020-11-20T12:04:57.356779+0000","last_unstale":"2020-11-20T12:04:57.884170+0000","last_undegraded":"2020-11-20T12:04:57.884170+0000","last_fullsized":"2020-11-20T12:04:57.884170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.930770+0000","last_change":"2020-11-20T12:04:57.912459+0000","last_active":"2020-11-20T12:04:57.930770+0000","last_peered":"2020-11-20T12:04:57.930770+0000","last_clean":"2020-11-20T12:04:57.930770+0000","last_became_active":"2020-11-20T12:04:57.911192+0000","last_became_peered":"2020-11-20T12:04:57.911192+0000","last_unstale":"2020-11-20T12:04:57.930770+0000","last_undegraded":"2020-11-20T12:04:57.930770+0000","last_fullsized":"2020-11-20T12:04:57.930770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.883410+0000","last_change":"2020-11-20T12:04:53.303192+0000","last_active":"2020-11-20T12:04:57.883410+0000","last_peered":"2020-11-20T12:04:57.883410+0000","last_clean":"2020-11-20T12:04:57.883410+0000","last_became_active":"2020-11-20T12:04:53.302386+0000","last_became_peered":"2020-11-20T12:04:53.302386+0000","last_unstale":"2020-11-20T12:04:57.883410+0000","last_undegraded":"2020-11-20T12:04:57.883410+0000","last_fullsized":"2020-11-20T12:04:57.883410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"22'8","reported_seq":"18","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.950026+0000","last_change":"2020-11-20T12:04:57.362923+0000","last_active":"2020-11-20T12:04:57.950026+0000","last_peered":"2020-11-20T12:04:57.950026+0000","last_clean":"2020-11-20T12:04:57.950026+0000","last_became_active":"2020-11-20T12:04:57.361782+0000","last_became_peered":"2020-11-20T12:04:57.361782+0000","last_unstale":"2020-11-20T12:04:57.950026+0000","last_undegraded":"2020-11-20T12:04:57.950026+0000","last_fullsized":"2020-11-20T12:04:57.950026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.899315+0000","last_change":"2020-11-20T12:04:57.897209+0000","last_active":"2020-11-20T12:04:57.899315+0000","last_peered":"2020-11-20T12:04:57.899315+0000","last_clean":"2020-11-20T12:04:57.899315+0000","last_became_active":"2020-11-20T12:04:57.895996+0000","last_became_peered":"2020-11-20T12:04:57.895996+0000","last_unstale":"2020-11-20T12:04:57.899315+0000","last_undegraded":"2020-11-20T12:04:57.899315+0000","last_fullsized":"2020-11-20T12:04:57.899315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'6","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.945332+0000","last_change":"2020-11-20T12:04:57.341247+0000","last_active":"2020-11-20T12:04:57.945332+0000","last_peered":"2020-11-20T12:04:57.945332+0000","last_clean":"2020-11-20T12:04:57.945332+0000","last_became_active":"2020-11-20T12:04:57.340097+0000","last_became_peered":"2020-11-20T12:04:57.340097+0000","last_unstale":"2020-11-20T12:04:57.945332+0000","last_undegraded":"2020-11-20T12:04:57.945332+0000","last_fullsized":"2020-11-20T12:04:57.945332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.886598+0000","last_change":"2020-11-20T12:04:54.003353+0000","last_active":"2020-11-20T12:04:57.886598+0000","last_peered":"2020-11-20T12:04:57.886598+0000","last_clean":"2020-11-20T12:04:57.886598+0000","last_became_active":"2020-11-20T12:04:54.002137+0000","last_became_peered":"2020-11-20T12:04:54.002137+0000","last_unstale":"2020-11-20T12:04:57.886598+0000","last_undegraded":"2020-11-20T12:04:57.886598+0000","last_fullsized":"2020-11-20T12:04:57.886598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.892048+0000","last_change":"2020-11-20T12:04:57.888425+0000","last_active":"2020-11-20T12:04:57.892048+0000","last_peered":"2020-11-20T12:04:57.892048+0000","last_clean":"2020-11-20T12:04:57.892048+0000","last_became_active":"2020-11-20T12:04:57.887885+0000","last_became_peered":"2020-11-20T12:04:57.887885+0000","last_unstale":"2020-11-20T12:04:57.892048+0000","last_undegraded":"2020-11-20T12:04:57.892048+0000","last_fullsized":"2020-11-20T12:04:57.892048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.882787+0000","last_change":"2020-11-20T12:04:54.001880+0000","last_active":"2020-11-20T12:04:57.882787+0000","last_peered":"2020-11-20T12:04:57.882787+0000","last_clean":"2020-11-20T12:04:57.882787+0000","last_became_active":"2020-11-20T12:04:54.001419+0000","last_became_peered":"2020-11-20T12:04:54.001419+0000","last_unstale":"2020-11-20T12:04:57.882787+0000","last_undegraded":"2020-11-20T12:04:57.882787+0000","last_fullsized":"2020-11-20T12:04:57.882787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"23'4","reported_seq":"14","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-20T12:04:58.890623+0000","last_change":"2020-11-20T12:04:57.351275+0000","last_active":"2020-11-20T12:04:58.890623+0000","last_peered":"2020-11-20T12:04:58.890623+0000","last_clean":"2020-11-20T12:04:58.890623+0000","last_became_active":"2020-11-20T12:04:57.350194+0000","last_became_peered":"2020-11-20T12:04:57.350194+0000","last_unstale":"2020-11-20T12:04:58.890623+0000","last_undegraded":"2020-11-20T12:04:58.890623+0000","last_fullsized":"2020-11-20T12:04:58.890623+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366800,"statfs":{"total":96636764160,"available":96631603200,"internally_reserved":0,"allocated":237568,"data_stored":88251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.704}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9769999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4060000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.456}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.734}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.302}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0839999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5132,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366708,"statfs":{"total":96636764160,"available":96631508992,"internally_reserved":0,"allocated":266240,"data_stored":90521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.746}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.952}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8519999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6719999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9130000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6189999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0899999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366800,"statfs":{"total":96636764160,"available":96631603200,"internally_reserved":0,"allocated":237568,"data_stored":88251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.931}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2879999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7480000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5044,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366796,"statfs":{"total":96636764160,"available":96631599104,"internally_reserved":0,"allocated":241664,"data_stored":88285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4750000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6709999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5259999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7349999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.968}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.512}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5116,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366724,"statfs":{"total":96636764160,"available":96631525376,"internally_reserved":0,"allocated":249856,"data_stored":88445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5369999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9410000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5409999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6859999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5104,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366736,"statfs":{"total":96636764160,"available":96631537664,"internally_reserved":0,"allocated":237568,"data_stored":88251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.133}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5510000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.927}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0279999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4529999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0789999999999997}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":245760,"data_stored":88826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2650000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1840000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.48}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6459999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5439999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":245760,"data_stored":89183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.077}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1739999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6970000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5640000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0679999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3970000000000002}]}]}]}} 2020-11-20T12:05:11.963 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:05:12.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.194+0000 7fd8affff700 1 Processor -- start 2020-11-20T12:05:12.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.194+0000 7fd8affff700 1 -- start start 2020-11-20T12:05:12.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.195+0000 7fd8affff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd8b0145cc0 0x7fd8b0066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.195+0000 7fd8affff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8b00683f0 0x7fd8b0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.195+0000 7fd8affff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0065680 0x7fd8b0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.195+0000 7fd8affff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b014c170 con 0x7fd8b0145cc0 2020-11-20T12:05:12.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.195+0000 7fd8affff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b014c410 con 0x7fd8b00683f0 2020-11-20T12:05:12.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.195+0000 7fd8affff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b014c6b0 con 0x7fd8b0065680 2020-11-20T12:05:12.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8a67fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8b00683f0 0x7fd8b0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8a67fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8b00683f0 0x7fd8b0065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40810/0 (socket says 172.21.15.187:40810) 2020-11-20T12:05:12.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8a67fc700 1 -- 172.21.15.187:0/3155923470 learned_addr learned my addr 172.21.15.187:0/3155923470 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:12.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8af7fe700 1 --2- 172.21.15.187:0/3155923470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0065680 0x7fd8b0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8aeffd700 1 --2- 172.21.15.187:0/3155923470 >> v2:172.21.15.5:3300/0 conn(0x7fd8b0145cc0 0x7fd8b0066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8a67fc700 1 -- 172.21.15.187:0/3155923470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0065680 msgr2=0x7fd8b0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:12.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8a67fc700 1 --2- 172.21.15.187:0/3155923470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0065680 0x7fd8b0061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8a67fc700 1 -- 172.21.15.187:0/3155923470 >> v2:172.21.15.5:3300/0 conn(0x7fd8b0145cc0 msgr2=0x7fd8b0066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:12.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8a67fc700 1 --2- 172.21.15.187:0/3155923470 >> v2:172.21.15.5:3300/0 conn(0x7fd8b0145cc0 0x7fd8b0066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8a67fc700 1 -- 172.21.15.187:0/3155923470 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b014bdf0 con 0x7fd8b00683f0 2020-11-20T12:05:12.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.196+0000 7fd8aeffd700 1 --2- 172.21.15.187:0/3155923470 >> v2:172.21.15.5:3300/0 conn(0x7fd8b0145cc0 0x7fd8b0066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:12.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.197+0000 7fd8a67fc700 1 --2- 172.21.15.187:0/3155923470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0065190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fd8b00674b0 tx=0x7fd898006eb0).ready entity=mon.1 client_cookie=e8c4a87beb13e8be server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:12.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.197+0000 7fd8ae7fc700 1 -- 172.21.15.187:0/3155923470 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980098d0 con 0x7fd8b00683f0 2020-11-20T12:05:12.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.197+0000 7fd8ae7fc700 1 -- 172.21.15.187:0/3155923470 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898009a30 con 0x7fd8b00683f0 2020-11-20T12:05:12.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.197+0000 7fd8ae7fc700 1 -- 172.21.15.187:0/3155923470 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898009db0 con 0x7fd8b00683f0 2020-11-20T12:05:12.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.197+0000 7fd8affff700 1 -- 172.21.15.187:0/3155923470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 msgr2=0x7fd8b0065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.197+0000 7fd8affff700 1 --2- 172.21.15.187:0/3155923470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0065190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fd8b00674b0 tx=0x7fd898006eb0).stop 2020-11-20T12:05:12.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.198+0000 7fd8affff700 1 -- 172.21.15.187:0/3155923470 shutdown_connections 2020-11-20T12:05:12.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.198+0000 7fd8affff700 1 --2- 172.21.15.187:0/3155923470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0065680 0x7fd8b0061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.198+0000 7fd8affff700 1 --2- 172.21.15.187:0/3155923470 >> v2:172.21.15.5:3300/0 conn(0x7fd8b0145cc0 0x7fd8b0066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.198+0000 7fd8affff700 1 --2- 172.21.15.187:0/3155923470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0065190 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.198+0000 7fd8affff700 1 -- 172.21.15.187:0/3155923470 >> 172.21.15.187:0/3155923470 conn(0x7fd8b00bab30 msgr2=0x7fd8b00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:12.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.198+0000 7fd8affff700 1 -- 172.21.15.187:0/3155923470 shutdown_connections 2020-11-20T12:05:12.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.198+0000 7fd8affff700 1 -- 172.21.15.187:0/3155923470 wait complete. 2020-11-20T12:05:12.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.199+0000 7fd8affff700 1 Processor -- start 2020-11-20T12:05:12.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.199+0000 7fd8affff700 1 -- start start 2020-11-20T12:05:12.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.200+0000 7fd8affff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8b0065680 0x7fd8b014b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.200+0000 7fd8affff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.200+0000 7fd8affff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0145cc0 0x7fd8b00afa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.200+0000 7fd8affff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01490a0 con 0x7fd8b0065680 2020-11-20T12:05:12.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.200+0000 7fd8affff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b009d090 con 0x7fd8b00683f0 2020-11-20T12:05:12.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.200+0000 7fd8affff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01494f0 con 0x7fd8b0145cc0 2020-11-20T12:05:12.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.200+0000 7fd8a67fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8a67fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0148bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40816/0 (socket says 172.21.15.187:40816) 2020-11-20T12:05:12.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8a67fc700 1 -- 172.21.15.187:0/2802366117 learned_addr learned my addr 172.21.15.187:0/2802366117 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:12.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8a67fc700 1 -- 172.21.15.187:0/2802366117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0145cc0 msgr2=0x7fd8b00afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8aeffd700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8b0065680 0x7fd8b014b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8af7fe700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0145cc0 0x7fd8b00afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8a67fc700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0145cc0 0x7fd8b00afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8a67fc700 1 -- 172.21.15.187:0/2802366117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8b0065680 msgr2=0x7fd8b014b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8a67fc700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8b0065680 0x7fd8b014b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8a67fc700 1 -- 172.21.15.187:0/2802366117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b0149790 con 0x7fd8b00683f0 2020-11-20T12:05:12.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.201+0000 7fd8a67fc700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0148bb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd8b014c550 tx=0x7fd898015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:12.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.202+0000 7fd8a77fe700 1 -- 172.21.15.187:0/2802366117 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898011e50 con 0x7fd8b00683f0 2020-11-20T12:05:12.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.202+0000 7fd8a77fe700 1 -- 172.21.15.187:0/2802366117 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89801e460 con 0x7fd8b00683f0 2020-11-20T12:05:12.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.202+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b0149a30 con 0x7fd8b00683f0 2020-11-20T12:05:12.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.202+0000 7fd8a77fe700 1 -- 172.21.15.187:0/2802366117 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89801e7e0 con 0x7fd8b00683f0 2020-11-20T12:05:12.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.202+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b009c990 con 0x7fd8b00683f0 2020-11-20T12:05:12.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.203+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b0061a20 con 0x7fd8b00683f0 2020-11-20T12:05:12.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.203+0000 7fd8a77fe700 1 -- 172.21.15.187:0/2802366117 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd898025030 con 0x7fd8b00683f0 2020-11-20T12:05:12.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.204+0000 7fd8a77fe700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd88c056d20 0x7fd88c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.204+0000 7fd8a77fe700 1 -- 172.21.15.187:0/2802366117 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd898071970 con 0x7fd8b00683f0 2020-11-20T12:05:12.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.204+0000 7fd8aeffd700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd88c056d20 0x7fd88c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.205+0000 7fd8aeffd700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd88c056d20 0x7fd88c0591e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd8a0006fd0 tx=0x7fd8a0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:12.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.208+0000 7fd8a77fe700 1 -- 172.21.15.187:0/2802366117 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd898046d60 con 0x7fd8b00683f0 2020-11-20T12:05:12.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.387+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8b014cb50 con 0x7fd88c056d20 2020-11-20T12:05:12.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.396+0000 7fd8a77fe700 1 -- 172.21.15.187:0/2802366117 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82127 (secure 0 0 0) 0x7fd8b014cb50 con 0x7fd88c056d20 2020-11-20T12:05:12.401 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:12.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.398+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd88c056d20 msgr2=0x7fd88c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.398+0000 7fd8affff700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd88c056d20 0x7fd88c0591e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd8a0006fd0 tx=0x7fd8a0008040).stop 2020-11-20T12:05:12.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.398+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 msgr2=0x7fd8b0148bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.398+0000 7fd8affff700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0148bb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd8b014c550 tx=0x7fd898015000).stop 2020-11-20T12:05:12.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.399+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 shutdown_connections 2020-11-20T12:05:12.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.399+0000 7fd8affff700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd88c056d20 0x7fd88c0591e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.399+0000 7fd8affff700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8b0065680 0x7fd8b014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.399+0000 7fd8affff700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8b0145cc0 0x7fd8b00afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.399+0000 7fd8affff700 1 --2- 172.21.15.187:0/2802366117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8b00683f0 0x7fd8b0148bb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.399+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 >> 172.21.15.187:0/2802366117 conn(0x7fd8b00bab30 msgr2=0x7fd8b0067780 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:12.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.399+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 shutdown_connections 2020-11-20T12:05:12.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.399+0000 7fd8affff700 1 -- 172.21.15.187:0/2802366117 wait complete. 2020-11-20T12:05:12.406 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2020-11-20T12:05:12.426 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2020-11-20T12:05:11.527701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":40700,"kb_used_data":1916,"kb_used_omap":0,"kb_used_meta":38720,"kb_avail":754934020,"statfs":{"total":773094113280,"available":773052436480,"internally_reserved":0,"allocated":1961984,"data_stored":710013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39649280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004033"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.930064+0000","last_change":"2020-11-20T12:04:57.909926+0000","last_active":"2020-11-20T12:04:57.930064+0000","last_peered":"2020-11-20T12:04:57.930064+0000","last_clean":"2020-11-20T12:04:57.930064+0000","last_became_active":"2020-11-20T12:04:57.908811+0000","last_became_peered":"2020-11-20T12:04:57.908811+0000","last_unstale":"2020-11-20T12:04:57.930064+0000","last_undegraded":"2020-11-20T12:04:57.930064+0000","last_fullsized":"2020-11-20T12:04:57.930064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.888201+0000","last_change":"2020-11-20T12:04:53.296291+0000","last_active":"2020-11-20T12:04:57.888201+0000","last_peered":"2020-11-20T12:04:57.888201+0000","last_clean":"2020-11-20T12:04:57.888201+0000","last_became_active":"2020-11-20T12:04:53.294169+0000","last_became_peered":"2020-11-20T12:04:53.294169+0000","last_unstale":"2020-11-20T12:04:57.888201+0000","last_undegraded":"2020-11-20T12:04:57.888201+0000","last_fullsized":"2020-11-20T12:04:57.888201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"22'6","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.943940+0000","last_change":"2020-11-20T12:04:57.366733+0000","last_active":"2020-11-20T12:04:57.943940+0000","last_peered":"2020-11-20T12:04:57.943940+0000","last_clean":"2020-11-20T12:04:57.943940+0000","last_became_active":"2020-11-20T12:04:57.365105+0000","last_became_peered":"2020-11-20T12:04:57.365105+0000","last_unstale":"2020-11-20T12:04:57.943940+0000","last_undegraded":"2020-11-20T12:04:57.943940+0000","last_fullsized":"2020-11-20T12:04:57.943940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.930146+0000","last_change":"2020-11-20T12:04:57.911885+0000","last_active":"2020-11-20T12:04:57.930146+0000","last_peered":"2020-11-20T12:04:57.930146+0000","last_clean":"2020-11-20T12:04:57.930146+0000","last_became_active":"2020-11-20T12:04:57.910770+0000","last_became_peered":"2020-11-20T12:04:57.910770+0000","last_unstale":"2020-11-20T12:04:57.930146+0000","last_undegraded":"2020-11-20T12:04:57.930146+0000","last_fullsized":"2020-11-20T12:04:57.930146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.882829+0000","last_change":"2020-11-20T12:04:53.294740+0000","last_active":"2020-11-20T12:04:57.882829+0000","last_peered":"2020-11-20T12:04:57.882829+0000","last_clean":"2020-11-20T12:04:57.882829+0000","last_became_active":"2020-11-20T12:04:53.293882+0000","last_became_peered":"2020-11-20T12:04:53.293882+0000","last_unstale":"2020-11-20T12:04:57.882829+0000","last_undegraded":"2020-11-20T12:04:57.882829+0000","last_fullsized":"2020-11-20T12:04:57.882829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"22'5","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.906914+0000","last_change":"2020-11-20T12:04:57.345878+0000","last_active":"2020-11-20T12:04:57.906914+0000","last_peered":"2020-11-20T12:04:57.906914+0000","last_clean":"2020-11-20T12:04:57.906914+0000","last_became_active":"2020-11-20T12:04:57.344954+0000","last_became_peered":"2020-11-20T12:04:57.344954+0000","last_unstale":"2020-11-20T12:04:57.906914+0000","last_undegraded":"2020-11-20T12:04:57.906914+0000","last_fullsized":"2020-11-20T12:04:57.906914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.942423+0000","last_change":"2020-11-20T12:04:57.927464+0000","last_active":"2020-11-20T12:04:57.942423+0000","last_peered":"2020-11-20T12:04:57.942423+0000","last_clean":"2020-11-20T12:04:57.942423+0000","last_became_active":"2020-11-20T12:04:57.926845+0000","last_became_peered":"2020-11-20T12:04:57.926845+0000","last_unstale":"2020-11-20T12:04:57.942423+0000","last_undegraded":"2020-11-20T12:04:57.942423+0000","last_fullsized":"2020-11-20T12:04:57.942423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.883110+0000","last_change":"2020-11-20T12:04:53.299007+0000","last_active":"2020-11-20T12:04:57.883110+0000","last_peered":"2020-11-20T12:04:57.883110+0000","last_clean":"2020-11-20T12:04:57.883110+0000","last_became_active":"2020-11-20T12:04:53.297849+0000","last_became_peered":"2020-11-20T12:04:53.297849+0000","last_unstale":"2020-11-20T12:04:57.883110+0000","last_undegraded":"2020-11-20T12:04:57.883110+0000","last_fullsized":"2020-11-20T12:04:57.883110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"22'8","reported_seq":"18","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.933150+0000","last_change":"2020-11-20T12:04:57.359175+0000","last_active":"2020-11-20T12:04:57.933150+0000","last_peered":"2020-11-20T12:04:57.933150+0000","last_clean":"2020-11-20T12:04:57.933150+0000","last_became_active":"2020-11-20T12:04:57.352579+0000","last_became_peered":"2020-11-20T12:04:57.352579+0000","last_unstale":"2020-11-20T12:04:57.933150+0000","last_undegraded":"2020-11-20T12:04:57.933150+0000","last_fullsized":"2020-11-20T12:04:57.933150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.896690+0000","last_change":"2020-11-20T12:04:57.890119+0000","last_active":"2020-11-20T12:04:57.896690+0000","last_peered":"2020-11-20T12:04:57.896690+0000","last_clean":"2020-11-20T12:04:57.896690+0000","last_became_active":"2020-11-20T12:04:57.889641+0000","last_became_peered":"2020-11-20T12:04:57.889641+0000","last_unstale":"2020-11-20T12:04:57.896690+0000","last_undegraded":"2020-11-20T12:04:57.896690+0000","last_fullsized":"2020-11-20T12:04:57.896690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.882744+0000","last_change":"2020-11-20T12:04:54.004976+0000","last_active":"2020-11-20T12:04:57.882744+0000","last_peered":"2020-11-20T12:04:57.882744+0000","last_clean":"2020-11-20T12:04:57.882744+0000","last_became_active":"2020-11-20T12:04:54.004416+0000","last_became_peered":"2020-11-20T12:04:54.004416+0000","last_unstale":"2020-11-20T12:04:57.882744+0000","last_undegraded":"2020-11-20T12:04:57.882744+0000","last_fullsized":"2020-11-20T12:04:57.882744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'2","reported_seq":"18","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.880881+0000","last_change":"2020-11-20T12:04:49.157393+0000","last_active":"2020-11-20T12:04:57.880881+0000","last_peered":"2020-11-20T12:04:57.880881+0000","last_clean":"2020-11-20T12:04:57.880881+0000","last_became_active":"2020-11-20T12:04:49.156248+0000","last_became_peered":"2020-11-20T12:04:49.156248+0000","last_unstale":"2020-11-20T12:04:57.880881+0000","last_undegraded":"2020-11-20T12:04:57.880881+0000","last_fullsized":"2020-11-20T12:04:57.880881+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:44.517280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:44.517280+0000","last_clean_scrub_stamp":"2020-11-20T12:04:44.517280+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"22'1","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.929220+0000","last_change":"2020-11-20T12:04:57.346452+0000","last_active":"2020-11-20T12:04:57.929220+0000","last_peered":"2020-11-20T12:04:57.929220+0000","last_clean":"2020-11-20T12:04:57.929220+0000","last_became_active":"2020-11-20T12:04:57.345332+0000","last_became_peered":"2020-11-20T12:04:57.345332+0000","last_unstale":"2020-11-20T12:04:57.929220+0000","last_undegraded":"2020-11-20T12:04:57.929220+0000","last_fullsized":"2020-11-20T12:04:57.929220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.908970+0000","last_change":"2020-11-20T12:04:57.900065+0000","last_active":"2020-11-20T12:04:57.908970+0000","last_peered":"2020-11-20T12:04:57.908970+0000","last_clean":"2020-11-20T12:04:57.908970+0000","last_became_active":"2020-11-20T12:04:57.897190+0000","last_became_peered":"2020-11-20T12:04:57.897190+0000","last_unstale":"2020-11-20T12:04:57.908970+0000","last_undegraded":"2020-11-20T12:04:57.908970+0000","last_fullsized":"2020-11-20T12:04:57.908970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.882727+0000","last_change":"2020-11-20T12:04:53.297146+0000","last_active":"2020-11-20T12:04:57.882727+0000","last_peered":"2020-11-20T12:04:57.882727+0000","last_clean":"2020-11-20T12:04:57.882727+0000","last_became_active":"2020-11-20T12:04:53.296376+0000","last_became_peered":"2020-11-20T12:04:53.296376+0000","last_unstale":"2020-11-20T12:04:57.882727+0000","last_undegraded":"2020-11-20T12:04:57.882727+0000","last_fullsized":"2020-11-20T12:04:57.882727+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"10","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.884170+0000","last_change":"2020-11-20T12:04:57.358142+0000","last_active":"2020-11-20T12:04:57.884170+0000","last_peered":"2020-11-20T12:04:57.884170+0000","last_clean":"2020-11-20T12:04:57.884170+0000","last_became_active":"2020-11-20T12:04:57.356779+0000","last_became_peered":"2020-11-20T12:04:57.356779+0000","last_unstale":"2020-11-20T12:04:57.884170+0000","last_undegraded":"2020-11-20T12:04:57.884170+0000","last_fullsized":"2020-11-20T12:04:57.884170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.930770+0000","last_change":"2020-11-20T12:04:57.912459+0000","last_active":"2020-11-20T12:04:57.930770+0000","last_peered":"2020-11-20T12:04:57.930770+0000","last_clean":"2020-11-20T12:04:57.930770+0000","last_became_active":"2020-11-20T12:04:57.911192+0000","last_became_peered":"2020-11-20T12:04:57.911192+0000","last_unstale":"2020-11-20T12:04:57.930770+0000","last_undegraded":"2020-11-20T12:04:57.930770+0000","last_fullsized":"2020-11-20T12:04:57.930770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.883410+0000","last_change":"2020-11-20T12:04:53.303192+0000","last_active":"2020-11-20T12:04:57.883410+0000","last_peered":"2020-11-20T12:04:57.883410+0000","last_clean":"2020-11-20T12:04:57.883410+0000","last_became_active":"2020-11-20T12:04:53.302386+0000","last_became_peered":"2020-11-20T12:04:53.302386+0000","last_unstale":"2020-11-20T12:04:57.883410+0000","last_undegraded":"2020-11-20T12:04:57.883410+0000","last_fullsized":"2020-11-20T12:04:57.883410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"22'8","reported_seq":"18","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.950026+0000","last_change":"2020-11-20T12:04:57.362923+0000","last_active":"2020-11-20T12:04:57.950026+0000","last_peered":"2020-11-20T12:04:57.950026+0000","last_clean":"2020-11-20T12:04:57.950026+0000","last_became_active":"2020-11-20T12:04:57.361782+0000","last_became_peered":"2020-11-20T12:04:57.361782+0000","last_unstale":"2020-11-20T12:04:57.950026+0000","last_undegraded":"2020-11-20T12:04:57.950026+0000","last_fullsized":"2020-11-20T12:04:57.950026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.899315+0000","last_change":"2020-11-20T12:04:57.897209+0000","last_active":"2020-11-20T12:04:57.899315+0000","last_peered":"2020-11-20T12:04:57.899315+0000","last_clean":"2020-11-20T12:04:57.899315+0000","last_became_active":"2020-11-20T12:04:57.895996+0000","last_became_peered":"2020-11-20T12:04:57.895996+0000","last_unstale":"2020-11-20T12:04:57.899315+0000","last_undegraded":"2020-11-20T12:04:57.899315+0000","last_fullsized":"2020-11-20T12:04:57.899315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'6","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.945332+0000","last_change":"2020-11-20T12:04:57.341247+0000","last_active":"2020-11-20T12:04:57.945332+0000","last_peered":"2020-11-20T12:04:57.945332+0000","last_clean":"2020-11-20T12:04:57.945332+0000","last_became_active":"2020-11-20T12:04:57.340097+0000","last_became_peered":"2020-11-20T12:04:57.340097+0000","last_unstale":"2020-11-20T12:04:57.945332+0000","last_undegraded":"2020-11-20T12:04:57.945332+0000","last_fullsized":"2020-11-20T12:04:57.945332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.886598+0000","last_change":"2020-11-20T12:04:54.003353+0000","last_active":"2020-11-20T12:04:57.886598+0000","last_peered":"2020-11-20T12:04:57.886598+0000","last_clean":"2020-11-20T12:04:57.886598+0000","last_became_active":"2020-11-20T12:04:54.002137+0000","last_became_peered":"2020-11-20T12:04:54.002137+0000","last_unstale":"2020-11-20T12:04:57.886598+0000","last_undegraded":"2020-11-20T12:04:57.886598+0000","last_fullsized":"2020-11-20T12:04:57.886598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"9","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.892048+0000","last_change":"2020-11-20T12:04:57.888425+0000","last_active":"2020-11-20T12:04:57.892048+0000","last_peered":"2020-11-20T12:04:57.892048+0000","last_clean":"2020-11-20T12:04:57.892048+0000","last_became_active":"2020-11-20T12:04:57.887885+0000","last_became_peered":"2020-11-20T12:04:57.887885+0000","last_unstale":"2020-11-20T12:04:57.892048+0000","last_undegraded":"2020-11-20T12:04:57.892048+0000","last_fullsized":"2020-11-20T12:04:57.892048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:57.310400+0000","last_clean_scrub_stamp":"2020-11-20T12:04:57.310400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:57.882787+0000","last_change":"2020-11-20T12:04:54.001880+0000","last_active":"2020-11-20T12:04:57.882787+0000","last_peered":"2020-11-20T12:04:57.882787+0000","last_clean":"2020-11-20T12:04:57.882787+0000","last_became_active":"2020-11-20T12:04:54.001419+0000","last_became_peered":"2020-11-20T12:04:54.001419+0000","last_unstale":"2020-11-20T12:04:57.882787+0000","last_undegraded":"2020-11-20T12:04:57.882787+0000","last_fullsized":"2020-11-20T12:04:57.882787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"23'4","reported_seq":"14","reported_epoch":"23","state":"active+clean","last_fresh":"2020-11-20T12:04:58.890623+0000","last_change":"2020-11-20T12:04:57.351275+0000","last_active":"2020-11-20T12:04:58.890623+0000","last_peered":"2020-11-20T12:04:58.890623+0000","last_clean":"2020-11-20T12:04:58.890623+0000","last_became_active":"2020-11-20T12:04:57.350194+0000","last_became_peered":"2020-11-20T12:04:57.350194+0000","last_unstale":"2020-11-20T12:04:58.890623+0000","last_undegraded":"2020-11-20T12:04:58.890623+0000","last_fullsized":"2020-11-20T12:04:58.890623+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:56.298961+0000","last_clean_scrub_stamp":"2020-11-20T12:04:56.298961+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366800,"statfs":{"total":96636764160,"available":96631603200,"internally_reserved":0,"allocated":237568,"data_stored":88251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.704}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9769999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4060000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.456}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.734}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.302}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0839999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5132,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366708,"statfs":{"total":96636764160,"available":96631508992,"internally_reserved":0,"allocated":266240,"data_stored":90521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.746}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.952}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8519999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6719999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9130000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6189999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0899999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366800,"statfs":{"total":96636764160,"available":96631603200,"internally_reserved":0,"allocated":237568,"data_stored":88251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.931}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2879999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7480000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5044,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366796,"statfs":{"total":96636764160,"available":96631599104,"internally_reserved":0,"allocated":241664,"data_stored":88285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4750000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6709999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5259999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7349999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.968}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.512}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5116,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366724,"statfs":{"total":96636764160,"available":96631525376,"internally_reserved":0,"allocated":249856,"data_stored":88445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5369999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9410000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5409999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6859999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5104,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366736,"statfs":{"total":96636764160,"available":96631537664,"internally_reserved":0,"allocated":237568,"data_stored":88251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.133}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5510000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.927}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0279999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4529999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0789999999999997}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":245760,"data_stored":88826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2650000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1840000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.48}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6459999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5439999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5112,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366728,"statfs":{"total":96636764160,"available":96631529472,"internally_reserved":0,"allocated":245760,"data_stored":89183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.077}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1739999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6970000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5640000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0679999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3970000000000002}]}]}]}} 2020-11-20T12:05:12.429 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-11-20T12:05:12.429 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2020-11-20T12:05:12.429 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2020-11-20T12:05:12.430 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2020-11-20T12:05:12.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.642+0000 7f4f4bfff700 1 Processor -- start 2020-11-20T12:05:12.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.642+0000 7f4f4bfff700 1 -- start start 2020-11-20T12:05:12.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.644+0000 7f4f4bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4f4c062370 0x7f4f4c0b1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.644+0000 7f4f4bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 0x7f4f4c0b5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.644+0000 7f4f4bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c067040 con 0x7f4f4c061f90 2020-11-20T12:05:12.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.644+0000 7f4f4bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4c067270 con 0x7f4f4c062370 2020-11-20T12:05:12.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.644+0000 7f4f4bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c0b6c80 con 0x7f4f4c0b0c50 2020-11-20T12:05:12.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.645+0000 7f4f4a7fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4f4c062370 0x7f4f4c0b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.645+0000 7f4f4b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 0x7f4f4c0b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.645+0000 7f4f4a7fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4f4c062370 0x7f4f4c0b1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:33086/0 (socket says 172.21.15.187:33086) 2020-11-20T12:05:12.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.645+0000 7f4f4a7fc700 1 -- 172.21.15.187:0/3734337443 learned_addr learned my addr 172.21.15.187:0/3734337443 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:12.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.646+0000 7f4f4a7fc700 1 -- 172.21.15.187:0/3734337443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 msgr2=0x7f4f4c0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:12.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.646+0000 7f4f4a7fc700 1 --2- 172.21.15.187:0/3734337443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 0x7f4f4c0b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.646+0000 7f4f4a7fc700 1 -- 172.21.15.187:0/3734337443 >> v1:172.21.15.5:6789/0 conn(0x7f4f4c061f90 legacy=0x7f4f4c148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.646+0000 7f4f4a7fc700 1 -- 172.21.15.187:0/3734337443 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c062dd0 con 0x7f4f4c062370 2020-11-20T12:05:12.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.646+0000 7f4f4a7fc700 1 --2- 172.21.15.187:0/3734337443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c062370 0x7f4f4c0b1340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f4f40006fd0 tx=0x7f4f40006eb0).ready entity=mon.0 client_cookie=25a9241a705ea917 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:12.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f49ffb700 1 -- 172.21.15.187:0/3734337443 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40009810 con 0x7f4f4c062370 2020-11-20T12:05:12.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f49ffb700 1 -- 172.21.15.187:0/3734337443 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f40009970 con 0x7f4f4c062370 2020-11-20T12:05:12.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f49ffb700 1 -- 172.21.15.187:0/3734337443 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40009cf0 con 0x7f4f4c062370 2020-11-20T12:05:12.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f4bfff700 1 -- 172.21.15.187:0/3734337443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c062370 msgr2=0x7f4f4c0b1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/3734337443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c062370 0x7f4f4c0b1340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f4f40006fd0 tx=0x7f4f40006eb0).stop 2020-11-20T12:05:12.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f4bfff700 1 -- 172.21.15.187:0/3734337443 shutdown_connections 2020-11-20T12:05:12.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/3734337443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 0x7f4f4c0b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/3734337443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c062370 0x7f4f4c0b1340 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.647+0000 7f4f4bfff700 1 -- 172.21.15.187:0/3734337443 >> 172.21.15.187:0/3734337443 conn(0x7f4f4c0bab10 msgr2=0x7f4f4c0baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:12.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.648+0000 7f4f4bfff700 1 -- 172.21.15.187:0/3734337443 shutdown_connections 2020-11-20T12:05:12.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.648+0000 7f4f4bfff700 1 -- 172.21.15.187:0/3734337443 wait complete. 2020-11-20T12:05:12.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.649+0000 7f4f4bfff700 1 Processor -- start 2020-11-20T12:05:12.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.649+0000 7f4f4bfff700 1 -- start start 2020-11-20T12:05:12.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.650+0000 7f4f4bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c061f90 0x7f4f4c157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.650+0000 7f4f4bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f4c062370 0x7f4f4c157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.650+0000 7f4f4bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 0x7f4f4c131bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.650+0000 7f4f4bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c09d3a0 con 0x7f4f4c061f90 2020-11-20T12:05:12.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.650+0000 7f4f4bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c13b080 con 0x7f4f4c062370 2020-11-20T12:05:12.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.650+0000 7f4f4bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c13b520 con 0x7f4f4c0b0c50 2020-11-20T12:05:12.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.650+0000 7f4f4a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f4c062370 0x7f4f4c157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f4c062370 0x7f4f4c157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40830/0 (socket says 172.21.15.187:40830) 2020-11-20T12:05:12.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4a7fc700 1 -- 172.21.15.187:0/1151428266 learned_addr learned my addr 172.21.15.187:0/1151428266 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:12.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4affd700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c061f90 0x7f4f4c157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4b7fe700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 0x7f4f4c131bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4a7fc700 1 -- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 msgr2=0x7f4f4c131bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4a7fc700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 0x7f4f4c131bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4a7fc700 1 -- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c061f90 msgr2=0x7f4f4c157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4a7fc700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c061f90 0x7f4f4c157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4a7fc700 1 -- 172.21.15.187:0/1151428266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c157980 con 0x7f4f4c062370 2020-11-20T12:05:12.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4affd700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c061f90 0x7f4f4c157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:12.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4a7fc700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f4c062370 0x7f4f4c157490 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4f4c067980 tx=0x7f4f40015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:12.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f32ffd700 1 -- 172.21.15.187:0/1151428266 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40011e10 con 0x7f4f4c062370 2020-11-20T12:05:12.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f32ffd700 1 -- 172.21.15.187:0/1151428266 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4001e460 con 0x7f4f4c062370 2020-11-20T12:05:12.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.651+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c13c080 con 0x7f4f4c062370 2020-11-20T12:05:12.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.652+0000 7f4f32ffd700 1 -- 172.21.15.187:0/1151428266 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4001e7e0 con 0x7f4f4c062370 2020-11-20T12:05:12.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.652+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c09d140 con 0x7f4f4c062370 2020-11-20T12:05:12.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.653+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c13b240 con 0x7f4f4c062370 2020-11-20T12:05:12.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.653+0000 7f4f32ffd700 1 -- 172.21.15.187:0/1151428266 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4f40025030 con 0x7f4f4c062370 2020-11-20T12:05:12.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.654+0000 7f4f32ffd700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f28056d20 0x7f4f280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:12.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.654+0000 7f4f32ffd700 1 -- 172.21.15.187:0/1151428266 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4f40071910 con 0x7f4f4c062370 2020-11-20T12:05:12.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.654+0000 7f4f4affd700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f28056d20 0x7f4f280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:12.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.655+0000 7f4f4affd700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f28056d20 0x7f4f280591e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4f3c000bc0 tx=0x7f4f3c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:12.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.659+0000 7f4f32ffd700 1 -- 172.21.15.187:0/1151428266 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4f40046d00 con 0x7f4f4c062370 2020-11-20T12:05:12.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.911+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4f4c061a20 con 0x7f4f4c062370 2020-11-20T12:05:12.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.911+0000 7f4f32ffd700 1 -- 172.21.15.187:0/1151428266 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f4f40023070 con 0x7f4f4c062370 2020-11-20T12:05:12.916 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:12.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.913+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f28056d20 msgr2=0x7f4f280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f28056d20 0x7f4f280591e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4f3c000bc0 tx=0x7f4f3c009040).stop 2020-11-20T12:05:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f4c062370 msgr2=0x7f4f4c157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f4c062370 0x7f4f4c157490 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4f4c067980 tx=0x7f4f40015000).stop 2020-11-20T12:05:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 shutdown_connections 2020-11-20T12:05:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f28056d20 0x7f4f280591e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f4c061f90 0x7f4f4c157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f4c0b0c50 0x7f4f4c131bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 --2- 172.21.15.187:0/1151428266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f4c062370 0x7f4f4c157490 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 >> 172.21.15.187:0/1151428266 conn(0x7f4f4c0bab10 msgr2=0x7f4f4c09faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 shutdown_connections 2020-11-20T12:05:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:12.914+0000 7f4f4bfff700 1 -- 172.21.15.187:0/1151428266 wait complete. 2020-11-20T12:05:12.937 INFO:teuthology.orchestra.run.smithi187.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-20T12:05:12.937 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2020-11-20T12:05:12.938 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:13.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.174+0000 7ff5b9bca700 1 Processor -- start 2020-11-20T12:05:13.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.174+0000 7ff5b9bca700 1 -- start start 2020-11-20T12:05:13.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.175+0000 7ff5b9bca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff5b40661f0 0x7ff5b40b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.175+0000 7ff5b9bca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b4099ce0 0x7ff5b40b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.175+0000 7ff5b9bca700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b40b5df0 con 0x7ff5b4065e10 2020-11-20T12:05:13.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.175+0000 7ff5b9bca700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b409a440 con 0x7ff5b40661f0 2020-11-20T12:05:13.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.175+0000 7ff5b9bca700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b409a1a0 con 0x7ff5b4099ce0 2020-11-20T12:05:13.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.176+0000 7ff5b37fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff5b4065e10 0x7ff5b4144470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:59184/0 (socket says 172.21.15.187:59184) 2020-11-20T12:05:13.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.176+0000 7ff5b37fe700 1 -- 172.21.15.187:0/4281636173 learned_addr learned my addr 172.21.15.187:0/4281636173 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:13.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.176+0000 7ff5aaffd700 1 --2- 172.21.15.187:0/4281636173 >> v2:172.21.15.187:3300/0 conn(0x7ff5b40661f0 0x7ff5b40b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.176+0000 7ff5b3fff700 1 --2- 172.21.15.187:0/4281636173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b4099ce0 0x7ff5b40b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.177+0000 7ff5b2ffd700 1 -- 172.21.15.187:0/4281636173 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598876195 0 0) 0x7ff5b40b5df0 con 0x7ff5b4065e10 2020-11-20T12:05:13.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.177+0000 7ff5b2ffd700 1 -- 172.21.15.187:0/4281636173 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff594003240 con 0x7ff5b4065e10 2020-11-20T12:05:13.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.177+0000 7ff5aaffd700 1 -- 172.21.15.187:0/4281636173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b4099ce0 msgr2=0x7ff5b40b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:13.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.177+0000 7ff5aaffd700 1 --2- 172.21.15.187:0/4281636173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b4099ce0 0x7ff5b40b67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.177+0000 7ff5aaffd700 1 -- 172.21.15.187:0/4281636173 >> v1:172.21.15.5:6789/0 conn(0x7ff5b4065e10 legacy=0x7ff5b4144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.177+0000 7ff5aaffd700 1 -- 172.21.15.187:0/4281636173 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b40b0160 con 0x7ff5b40661f0 2020-11-20T12:05:13.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.177+0000 7ff5aaffd700 1 --2- 172.21.15.187:0/4281636173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b40661f0 0x7ff5b40b62e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff59c00b990 tx=0x7ff59c006df0).ready entity=mon.1 client_cookie=2b94640008119800 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:13.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b2ffd700 1 -- 172.21.15.187:0/4281636173 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c0096d0 con 0x7ff5b40661f0 2020-11-20T12:05:13.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b2ffd700 1 -- 172.21.15.187:0/4281636173 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c009830 con 0x7ff5b40661f0 2020-11-20T12:05:13.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b2ffd700 1 -- 172.21.15.187:0/4281636173 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c009bb0 con 0x7ff5b40661f0 2020-11-20T12:05:13.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b9bca700 1 -- 172.21.15.187:0/4281636173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b40661f0 msgr2=0x7ff5b40b62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/4281636173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b40661f0 0x7ff5b40b62e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff59c00b990 tx=0x7ff59c006df0).stop 2020-11-20T12:05:13.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b9bca700 1 -- 172.21.15.187:0/4281636173 shutdown_connections 2020-11-20T12:05:13.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/4281636173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b4099ce0 0x7ff5b40b67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/4281636173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b40661f0 0x7ff5b40b62e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b9bca700 1 -- 172.21.15.187:0/4281636173 >> 172.21.15.187:0/4281636173 conn(0x7ff5b40bacd0 msgr2=0x7ff5b40bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:13.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b9bca700 1 -- 172.21.15.187:0/4281636173 shutdown_connections 2020-11-20T12:05:13.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.178+0000 7ff5b9bca700 1 -- 172.21.15.187:0/4281636173 wait complete. 2020-11-20T12:05:13.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.180+0000 7ff5b9bca700 1 Processor -- start 2020-11-20T12:05:13.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.180+0000 7ff5b9bca700 1 -- start start 2020-11-20T12:05:13.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b9bca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b4065e10 0x7ff5b40b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b9bca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b40661f0 0x7ff5b413ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b9bca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b4099ce0 0x7ff5b413f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b9bca700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b413f7e0 con 0x7ff5b4099ce0 2020-11-20T12:05:13.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b9bca700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b409fda0 con 0x7ff5b4065e10 2020-11-20T12:05:13.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b9bca700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b40a0010 con 0x7ff5b40661f0 2020-11-20T12:05:13.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b4065e10 0x7ff5b40b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b4065e10 0x7ff5b40b1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40842/0 (socket says 172.21.15.187:40842) 2020-11-20T12:05:13.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.181+0000 7ff5b37fe700 1 -- 172.21.15.187:0/1620830594 learned_addr learned my addr 172.21.15.187:0/1620830594 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:13.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5b3fff700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b4099ce0 0x7ff5b413f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5aaffd700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b40661f0 0x7ff5b413ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5b37fe700 1 -- 172.21.15.187:0/1620830594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b40661f0 msgr2=0x7ff5b413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5b37fe700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b40661f0 0x7ff5b413ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5b37fe700 1 -- 172.21.15.187:0/1620830594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b4099ce0 msgr2=0x7ff5b413f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5b37fe700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b4099ce0 0x7ff5b413f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5b37fe700 1 -- 172.21.15.187:0/1620830594 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b40a02b0 con 0x7ff5b4065e10 2020-11-20T12:05:13.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5b37fe700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b4065e10 0x7ff5b40b1cb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff5a400c710 tx=0x7ff5a4007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:13.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5abfff700 1 -- 172.21.15.187:0/1620830594 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a40072e0 con 0x7ff5b4065e10 2020-11-20T12:05:13.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5abfff700 1 -- 172.21.15.187:0/1620830594 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a4007440 con 0x7ff5b4065e10 2020-11-20T12:05:13.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.182+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b409e040 con 0x7ff5b4065e10 2020-11-20T12:05:13.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.183+0000 7ff5abfff700 1 -- 172.21.15.187:0/1620830594 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4005800 con 0x7ff5b4065e10 2020-11-20T12:05:13.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.183+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b40a0db0 con 0x7ff5b4065e10 2020-11-20T12:05:13.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.184+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b4063600 con 0x7ff5b4065e10 2020-11-20T12:05:13.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.184+0000 7ff5abfff700 1 -- 172.21.15.187:0/1620830594 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff5a401a060 con 0x7ff5b4065e10 2020-11-20T12:05:13.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.184+0000 7ff5abfff700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.184+0000 7ff5abfff700 1 -- 172.21.15.187:0/1620830594 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff5a406f800 con 0x7ff5b4065e10 2020-11-20T12:05:13.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.185+0000 7ff5aaffd700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.185+0000 7ff5aaffd700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff5b40b50a0 tx=0x7ff59c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:13.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.188+0000 7ff5abfff700 1 -- 172.21.15.187:0/1620830594 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff5a4044cd0 con 0x7ff5b4065e10 2020-11-20T12:05:13.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.361+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b40666e0 con 0x7ff5b4065e10 2020-11-20T12:05:13.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.363+0000 7ff5abfff700 1 -- 172.21.15.187:0/1620830594 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7ff5a4018070 con 0x7ff5b4065e10 2020-11-20T12:05:13.367 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:13.368 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:13.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 msgr2=0x7ff590059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff5b40b50a0 tx=0x7ff59c015000).stop 2020-11-20T12:05:13.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b4065e10 msgr2=0x7ff5b40b1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b4065e10 0x7ff5b40b1cb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff5a400c710 tx=0x7ff5a4007ad0).stop 2020-11-20T12:05:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 shutdown_connections 2020-11-20T12:05:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b4099ce0 0x7ff5b413f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b4065e10 0x7ff5b40b1cb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 --2- 172.21.15.187:0/1620830594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b40661f0 0x7ff5b413ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 >> 172.21.15.187:0/1620830594 conn(0x7ff5b40bacd0 msgr2=0x7ff5b4065440 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 shutdown_connections 2020-11-20T12:05:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.365+0000 7ff5b9bca700 1 -- 172.21.15.187:0/1620830594 wait complete. 2020-11-20T12:05:13.392 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:13.394 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:13.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.628+0000 7f48e18f2700 1 Processor -- start 2020-11-20T12:05:13.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.628+0000 7f48e18f2700 1 -- start start 2020-11-20T12:05:13.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.629+0000 7f48e18f2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc09a8a0 0x7f48dc0656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.629+0000 7f48e18f2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48dc068030 0x7f48dc146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.629+0000 7f48e18f2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc066420 con 0x7f48dc065300 2020-11-20T12:05:13.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.629+0000 7f48e18f2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48dc0b7380 con 0x7f48dc068030 2020-11-20T12:05:13.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.629+0000 7f48e18f2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc061f60 con 0x7f48dc09a8a0 2020-11-20T12:05:13.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.630+0000 7f48daffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f48dc065300 0x7f48dc062400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37456/0 (socket says 172.21.15.187:37456) 2020-11-20T12:05:13.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.630+0000 7f48daffd700 1 -- 172.21.15.187:0/2305438093 learned_addr learned my addr 172.21.15.187:0/2305438093 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:13.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.630+0000 7f48db7fe700 1 --2- 172.21.15.187:0/2305438093 >> v2:172.21.15.187:3300/0 conn(0x7f48dc068030 0x7f48dc146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.630+0000 7f48da7fc700 1 --2- 172.21.15.187:0/2305438093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc09a8a0 0x7f48dc0656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.631+0000 7f48d9ffb700 1 -- 172.21.15.187:0/2305438093 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136198113 0 0) 0x7f48dc066420 con 0x7f48dc065300 2020-11-20T12:05:13.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.631+0000 7f48d9ffb700 1 -- 172.21.15.187:0/2305438093 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c8003240 con 0x7f48dc065300 2020-11-20T12:05:13.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.631+0000 7f48db7fe700 1 -- 172.21.15.187:0/2305438093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc09a8a0 msgr2=0x7f48dc0656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:13.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.631+0000 7f48db7fe700 1 --2- 172.21.15.187:0/2305438093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc09a8a0 0x7f48dc0656e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.631+0000 7f48db7fe700 1 -- 172.21.15.187:0/2305438093 >> v1:172.21.15.187:6789/0 conn(0x7f48dc065300 legacy=0x7f48dc062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.631+0000 7f48db7fe700 1 -- 172.21.15.187:0/2305438093 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc0b6fd0 con 0x7f48dc068030 2020-11-20T12:05:13.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.631+0000 7f48db7fe700 1 --2- 172.21.15.187:0/2305438093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc068030 0x7f48dc146950 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f48d4009c90 tx=0x7f48d40091d0).ready entity=mon.1 client_cookie=6a8dd3a5607bb7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:13.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48d9ffb700 1 -- 172.21.15.187:0/2305438093 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d400ba50 con 0x7f48dc068030 2020-11-20T12:05:13.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48d9ffb700 1 -- 172.21.15.187:0/2305438093 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d400bbb0 con 0x7f48dc068030 2020-11-20T12:05:13.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48d9ffb700 1 -- 172.21.15.187:0/2305438093 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4011400 con 0x7f48dc068030 2020-11-20T12:05:13.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48e18f2700 1 -- 172.21.15.187:0/2305438093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc068030 msgr2=0x7f48dc146950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48e18f2700 1 --2- 172.21.15.187:0/2305438093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc068030 0x7f48dc146950 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f48d4009c90 tx=0x7f48d40091d0).stop 2020-11-20T12:05:13.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48e18f2700 1 -- 172.21.15.187:0/2305438093 shutdown_connections 2020-11-20T12:05:13.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48e18f2700 1 --2- 172.21.15.187:0/2305438093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc068030 0x7f48dc146950 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48e18f2700 1 --2- 172.21.15.187:0/2305438093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc09a8a0 0x7f48dc0656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48e18f2700 1 -- 172.21.15.187:0/2305438093 >> 172.21.15.187:0/2305438093 conn(0x7f48dc0bacf0 msgr2=0x7f48dc0bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:13.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48e18f2700 1 -- 172.21.15.187:0/2305438093 shutdown_connections 2020-11-20T12:05:13.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.632+0000 7f48e18f2700 1 -- 172.21.15.187:0/2305438093 wait complete. 2020-11-20T12:05:13.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.634+0000 7f48e18f2700 1 Processor -- start 2020-11-20T12:05:13.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.634+0000 7f48e18f2700 1 -- start start 2020-11-20T12:05:13.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48e18f2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc065300 0x7f48dc157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48e18f2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc068030 0x7f48dc146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48e18f2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09a8a0 0x7f48dc145b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48e18f2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc099ab0 con 0x7f48dc068030 2020-11-20T12:05:13.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48e18f2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc158310 con 0x7f48dc09a8a0 2020-11-20T12:05:13.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48e18f2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc158510 con 0x7f48dc065300 2020-11-20T12:05:13.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48db7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09a8a0 0x7f48dc145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48db7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09a8a0 0x7f48dc145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40860/0 (socket says 172.21.15.187:40860) 2020-11-20T12:05:13.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.635+0000 7f48db7fe700 1 -- 172.21.15.187:0/3233927321 learned_addr learned my addr 172.21.15.187:0/3233927321 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:13.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48da7fc700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc068030 0x7f48dc146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48daffd700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc065300 0x7f48dc157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48db7fe700 1 -- 172.21.15.187:0/3233927321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc065300 msgr2=0x7f48dc157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48db7fe700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc065300 0x7f48dc157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48db7fe700 1 -- 172.21.15.187:0/3233927321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc068030 msgr2=0x7f48dc146e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48db7fe700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc068030 0x7f48dc146e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48db7fe700 1 -- 172.21.15.187:0/3233927321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc1587b0 con 0x7f48dc09a8a0 2020-11-20T12:05:13.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48db7fe700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09a8a0 0x7f48dc145b70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f48d4009210 tx=0x7f48d4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:13.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48c2ffd700 1 -- 172.21.15.187:0/3233927321 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4010780 con 0x7f48dc09a8a0 2020-11-20T12:05:13.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48c2ffd700 1 -- 172.21.15.187:0/3233927321 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d40108e0 con 0x7f48dc09a8a0 2020-11-20T12:05:13.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48c2ffd700 1 -- 172.21.15.187:0/3233927321 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4010c60 con 0x7f48dc09a8a0 2020-11-20T12:05:13.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.636+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48dc158a50 con 0x7f48dc09a8a0 2020-11-20T12:05:13.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.637+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc147560 con 0x7f48dc09a8a0 2020-11-20T12:05:13.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.638+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48dc044f60 con 0x7f48dc09a8a0 2020-11-20T12:05:13.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.638+0000 7f48c2ffd700 1 -- 172.21.15.187:0/3233927321 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f48d4024030 con 0x7f48dc09a8a0 2020-11-20T12:05:13.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.639+0000 7f48c2ffd700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b8056d20 0x7f48b80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:13.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.639+0000 7f48c2ffd700 1 -- 172.21.15.187:0/3233927321 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f48d4071a90 con 0x7f48dc09a8a0 2020-11-20T12:05:13.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.639+0000 7f48daffd700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b8056d20 0x7f48b80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:13.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.640+0000 7f48daffd700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b8056d20 0x7f48b80591e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f48cc0055a0 tx=0x7f48cc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:13.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.643+0000 7f48c2ffd700 1 -- 172.21.15.187:0/3233927321 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f48d4046ff0 con 0x7f48dc09a8a0 2020-11-20T12:05:13.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.813+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f48dc05cae0 con 0x7f48dc09a8a0 2020-11-20T12:05:13.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.814+0000 7f48c2ffd700 1 -- 172.21.15.187:0/3233927321 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f48d4022070 con 0x7f48dc09a8a0 2020-11-20T12:05:13.819 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:13.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b8056d20 msgr2=0x7f48b80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b8056d20 0x7f48b80591e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f48cc0055a0 tx=0x7f48cc009040).stop 2020-11-20T12:05:13.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09a8a0 msgr2=0x7f48dc145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:13.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09a8a0 0x7f48dc145b70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f48d4009210 tx=0x7f48d4016000).stop 2020-11-20T12:05:13.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 shutdown_connections 2020-11-20T12:05:13.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b8056d20 0x7f48b80591e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09a8a0 0x7f48dc145b70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc065300 0x7f48dc157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 --2- 172.21.15.187:0/3233927321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc068030 0x7f48dc146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:13.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 >> 172.21.15.187:0/3233927321 conn(0x7f48dc0bacf0 msgr2=0x7f48dc066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:13.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 shutdown_connections 2020-11-20T12:05:13.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:13.817+0000 7f48e18f2700 1 -- 172.21.15.187:0/3233927321 wait complete. 2020-11-20T12:05:13.845 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:14.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.086+0000 7fa6931d9700 1 Processor -- start 2020-11-20T12:05:14.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.086+0000 7fa6931d9700 1 -- start start 2020-11-20T12:05:14.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.088+0000 7fa6931d9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa68c1482d0 0x7fa68c0b6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.088+0000 7fa6931d9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa68c144b80 con 0x7fa68c068230 2020-11-20T12:05:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.088+0000 7fa6931d9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa68c1489e0 con 0x7fa68c0b57b0 2020-11-20T12:05:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.088+0000 7fa6931d9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68c148c80 con 0x7fa68c1482d0 2020-11-20T12:05:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.088+0000 7fa6921d7700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa68c0b57b0 0x7fa68c062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37470/0 (socket says 172.21.15.187:37470) 2020-11-20T12:05:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.088+0000 7fa6921d7700 1 -- 172.21.15.187:0/3138185715 learned_addr learned my addr 172.21.15.187:0/3138185715 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:14.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.088+0000 7fa6929d8700 1 --2- 172.21.15.187:0/3138185715 >> v2:172.21.15.187:3300/0 conn(0x7fa68c1482d0 0x7fa68c0b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.089+0000 7fa6929d8700 1 -- 172.21.15.187:0/3138185715 >> v1:172.21.15.187:6789/0 conn(0x7fa68c0b57b0 legacy=0x7fa68c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.089+0000 7fa6929d8700 1 -- 172.21.15.187:0/3138185715 >> v1:172.21.15.5:6789/0 conn(0x7fa68c068230 legacy=0x7fa68c0b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.089+0000 7fa6929d8700 1 -- 172.21.15.187:0/3138185715 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c148f20 con 0x7fa68c1482d0 2020-11-20T12:05:14.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.090+0000 7fa6929d8700 1 --2- 172.21.15.187:0/3138185715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c1482d0 0x7fa68c0b6660 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa684009c90 tx=0x7fa6840091d0).ready entity=mon.1 client_cookie=8384438b32cdf9a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:14.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.090+0000 7fa6911d5700 1 -- 172.21.15.187:0/3138185715 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68400bbc0 con 0x7fa68c1482d0 2020-11-20T12:05:14.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.090+0000 7fa6911d5700 1 -- 172.21.15.187:0/3138185715 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68400bd20 con 0x7fa68c1482d0 2020-11-20T12:05:14.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.090+0000 7fa6911d5700 1 -- 172.21.15.187:0/3138185715 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684011610 con 0x7fa68c1482d0 2020-11-20T12:05:14.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.091+0000 7fa6931d9700 1 -- 172.21.15.187:0/3138185715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c1482d0 msgr2=0x7fa68c0b6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.091+0000 7fa6931d9700 1 --2- 172.21.15.187:0/3138185715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c1482d0 0x7fa68c0b6660 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa684009c90 tx=0x7fa6840091d0).stop 2020-11-20T12:05:14.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.091+0000 7fa6931d9700 1 -- 172.21.15.187:0/3138185715 shutdown_connections 2020-11-20T12:05:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.091+0000 7fa6931d9700 1 --2- 172.21.15.187:0/3138185715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c1482d0 0x7fa68c0b6660 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.091+0000 7fa6931d9700 1 -- 172.21.15.187:0/3138185715 >> 172.21.15.187:0/3138185715 conn(0x7fa68c0bab30 msgr2=0x7fa68c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.091+0000 7fa6931d9700 1 -- 172.21.15.187:0/3138185715 shutdown_connections 2020-11-20T12:05:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.091+0000 7fa6931d9700 1 -- 172.21.15.187:0/3138185715 wait complete. 2020-11-20T12:05:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.092+0000 7fa6931d9700 1 Processor -- start 2020-11-20T12:05:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.093+0000 7fa6931d9700 1 -- start start 2020-11-20T12:05:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.093+0000 7fa6931d9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa68c068230 0x7fa68c1492e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.093+0000 7fa6931d9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c0b57b0 0x7fa68c15a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6931d9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa68c1482d0 0x7fa68c147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6931d9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c143b70 con 0x7fa68c1482d0 2020-11-20T12:05:14.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6931d9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c15a880 con 0x7fa68c0b57b0 2020-11-20T12:05:14.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6931d9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c14cb00 con 0x7fa68c068230 2020-11-20T12:05:14.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c0b57b0 0x7fa68c15a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c0b57b0 0x7fa68c15a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40872/0 (socket says 172.21.15.187:40872) 2020-11-20T12:05:14.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 -- 172.21.15.187:0/3305429684 learned_addr learned my addr 172.21.15.187:0/3305429684 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:14.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 -- 172.21.15.187:0/3305429684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa68c068230 msgr2=0x7fa68c1492e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa68c068230 0x7fa68c1492e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:14.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6929d8700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa68c1482d0 0x7fa68c147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 -- 172.21.15.187:0/3305429684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa68c1482d0 msgr2=0x7fa68c147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa68c1482d0 0x7fa68c147060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 -- 172.21.15.187:0/3305429684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c14cda0 con 0x7fa68c0b57b0 2020-11-20T12:05:14.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.094+0000 7fa6919d6700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c0b57b0 0x7fa68c15a390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa67c001e70 tx=0x7fa67c007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:14.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.095+0000 7fa67a7fc700 1 -- 172.21.15.187:0/3305429684 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c007020 con 0x7fa68c0b57b0 2020-11-20T12:05:14.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.095+0000 7fa67a7fc700 1 -- 172.21.15.187:0/3305429684 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c007180 con 0x7fa68c0b57b0 2020-11-20T12:05:14.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.095+0000 7fa67a7fc700 1 -- 172.21.15.187:0/3305429684 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c00e560 con 0x7fa68c0b57b0 2020-11-20T12:05:14.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.095+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68c14d040 con 0x7fa68c0b57b0 2020-11-20T12:05:14.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.095+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68c09d010 con 0x7fa68c0b57b0 2020-11-20T12:05:14.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.096+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68c09dcc0 con 0x7fa68c0b57b0 2020-11-20T12:05:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.096+0000 7fa67a7fc700 1 -- 172.21.15.187:0/3305429684 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa67c010070 con 0x7fa68c0b57b0 2020-11-20T12:05:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.097+0000 7fa67a7fc700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa664056d20 0x7fa6640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.097+0000 7fa67a7fc700 1 -- 172.21.15.187:0/3305429684 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa67c06f4b0 con 0x7fa68c0b57b0 2020-11-20T12:05:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.097+0000 7fa6921d7700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa664056d20 0x7fa6640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.098+0000 7fa6921d7700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa664056d20 0x7fa6640591e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa688001330 tx=0x7fa688007800).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:14.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.101+0000 7fa67a7fc700 1 -- 172.21.15.187:0/3305429684 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa67c0447f0 con 0x7fa68c0b57b0 2020-11-20T12:05:14.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.273+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa68c09d2f0 con 0x7fa68c0b57b0 2020-11-20T12:05:14.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.274+0000 7fa67a7fc700 1 -- 172.21.15.187:0/3305429684 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa67c047e10 con 0x7fa68c0b57b0 2020-11-20T12:05:14.279 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:14.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa664056d20 msgr2=0x7fa6640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa664056d20 0x7fa6640591e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa688001330 tx=0x7fa688007800).stop 2020-11-20T12:05:14.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c0b57b0 msgr2=0x7fa68c15a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c0b57b0 0x7fa68c15a390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa67c001e70 tx=0x7fa67c007c50).stop 2020-11-20T12:05:14.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 shutdown_connections 2020-11-20T12:05:14.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa664056d20 0x7fa6640591e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa68c1482d0 0x7fa68c147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa68c068230 0x7fa68c1492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:14.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 --2- 172.21.15.187:0/3305429684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa68c0b57b0 0x7fa68c15a390 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 >> 172.21.15.187:0/3305429684 conn(0x7fa68c0bab30 msgr2=0x7fa68c061dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:14.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 shutdown_connections 2020-11-20T12:05:14.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.277+0000 7fa6931d9700 1 -- 172.21.15.187:0/3305429684 wait complete. 2020-11-20T12:05:14.301 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:14.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.536+0000 7f6a9759e700 1 Processor -- start 2020-11-20T12:05:14.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.537+0000 7f6a9759e700 1 -- start start 2020-11-20T12:05:14.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.538+0000 7f6a9759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6a980634c0 0x7f6a9809da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.538+0000 7f6a9759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6a9809aff0 0x7f6a9809c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.538+0000 7f6a9759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9809be90 con 0x7f6a980630e0 2020-11-20T12:05:14.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.538+0000 7f6a9759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9809bca0 con 0x7f6a980634c0 2020-11-20T12:05:14.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.538+0000 7f6a9759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9809e070 con 0x7f6a9809aff0 2020-11-20T12:05:14.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.538+0000 7f6a9659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6a980630e0 0x7f6a980b65c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37484/0 (socket says 172.21.15.187:37484) 2020-11-20T12:05:14.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.538+0000 7f6a9659c700 1 -- 172.21.15.187:0/19919100 learned_addr learned my addr 172.21.15.187:0/19919100 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:14.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.538+0000 7f6a96d9d700 1 --2- 172.21.15.187:0/19919100 >> v2:172.21.15.187:3300/0 conn(0x7f6a9809aff0 0x7f6a9809c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.539+0000 7f6a8dd9b700 1 --2- 172.21.15.187:0/19919100 >> v2:172.21.15.5:3300/0 conn(0x7f6a980634c0 0x7f6a9809da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.539+0000 7f6a95d9b700 1 -- 172.21.15.187:0/19919100 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261995722 0 0) 0x7f6a9809be90 con 0x7f6a980630e0 2020-11-20T12:05:14.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.539+0000 7f6a95d9b700 1 -- 172.21.15.187:0/19919100 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a78003240 con 0x7f6a980630e0 2020-11-20T12:05:14.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.539+0000 7f6a96d9d700 1 -- 172.21.15.187:0/19919100 >> v2:172.21.15.5:3300/0 conn(0x7f6a980634c0 msgr2=0x7f6a9809da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:14.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.539+0000 7f6a96d9d700 1 --2- 172.21.15.187:0/19919100 >> v2:172.21.15.5:3300/0 conn(0x7f6a980634c0 0x7f6a9809da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.539+0000 7f6a96d9d700 1 -- 172.21.15.187:0/19919100 >> v1:172.21.15.187:6789/0 conn(0x7f6a980630e0 legacy=0x7f6a980b65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.539+0000 7f6a96d9d700 1 -- 172.21.15.187:0/19919100 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9809c0f0 con 0x7f6a9809aff0 2020-11-20T12:05:14.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.539+0000 7f6a96d9d700 1 --2- 172.21.15.187:0/19919100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a9809aff0 0x7f6a9809c590 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f6a9000b9a0 tx=0x7f6a90006df0).ready entity=mon.1 client_cookie=f02051bdf331f24 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:14.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a95d9b700 1 -- 172.21.15.187:0/19919100 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90009770 con 0x7f6a9809aff0 2020-11-20T12:05:14.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a95d9b700 1 -- 172.21.15.187:0/19919100 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a900098d0 con 0x7f6a9809aff0 2020-11-20T12:05:14.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a95d9b700 1 -- 172.21.15.187:0/19919100 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90009c50 con 0x7f6a9809aff0 2020-11-20T12:05:14.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a9759e700 1 -- 172.21.15.187:0/19919100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a9809aff0 msgr2=0x7f6a9809c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a9759e700 1 --2- 172.21.15.187:0/19919100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a9809aff0 0x7f6a9809c590 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f6a9000b9a0 tx=0x7f6a90006df0).stop 2020-11-20T12:05:14.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a9759e700 1 -- 172.21.15.187:0/19919100 shutdown_connections 2020-11-20T12:05:14.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a9759e700 1 --2- 172.21.15.187:0/19919100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a9809aff0 0x7f6a9809c590 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a9759e700 1 --2- 172.21.15.187:0/19919100 >> v2:172.21.15.5:3300/0 conn(0x7f6a980634c0 0x7f6a9809da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a9759e700 1 -- 172.21.15.187:0/19919100 >> 172.21.15.187:0/19919100 conn(0x7f6a980bbc20 msgr2=0x7f6a980bc030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:14.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a9759e700 1 -- 172.21.15.187:0/19919100 shutdown_connections 2020-11-20T12:05:14.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.540+0000 7f6a9759e700 1 -- 172.21.15.187:0/19919100 wait complete. 2020-11-20T12:05:14.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.542+0000 7f6a9759e700 1 Processor -- start 2020-11-20T12:05:14.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.542+0000 7f6a9759e700 1 -- start start 2020-11-20T12:05:14.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a9759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a980630e0 0x7f6a980b5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a9759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a980634c0 0x7f6a98146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a9759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a9809aff0 0x7f6a980a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a9759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a98142a10 con 0x7f6a980630e0 2020-11-20T12:05:14.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a9759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a980a0bb0 con 0x7f6a980634c0 2020-11-20T12:05:14.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a9759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a980a0dd0 con 0x7f6a9809aff0 2020-11-20T12:05:14.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a8dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a980634c0 0x7f6a98146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a8dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a980634c0 0x7f6a98146170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40886/0 (socket says 172.21.15.187:40886) 2020-11-20T12:05:14.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.543+0000 7f6a8dd9b700 1 -- 172.21.15.187:0/419421772 learned_addr learned my addr 172.21.15.187:0/419421772 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:14.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a96d9d700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a9809aff0 0x7f6a980a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a9659c700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a980630e0 0x7f6a980b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8dd9b700 1 -- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a9809aff0 msgr2=0x7f6a980a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8dd9b700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a9809aff0 0x7f6a980a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8dd9b700 1 -- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a980630e0 msgr2=0x7f6a980b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8dd9b700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a980630e0 0x7f6a980b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8dd9b700 1 -- 172.21.15.187:0/419421772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a98145a30 con 0x7f6a980634c0 2020-11-20T12:05:14.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a96d9d700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a9809aff0 0x7f6a980a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:14.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a9659c700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a980630e0 0x7f6a980b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:14.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8dd9b700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a980634c0 0x7f6a98146170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f6a8000db30 tx=0x7f6a80008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:14.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8effd700 1 -- 172.21.15.187:0/419421772 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80009730 con 0x7f6a980634c0 2020-11-20T12:05:14.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8effd700 1 -- 172.21.15.187:0/419421772 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a80009890 con 0x7f6a980634c0 2020-11-20T12:05:14.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a8effd700 1 -- 172.21.15.187:0/419421772 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80009c10 con 0x7f6a980634c0 2020-11-20T12:05:14.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.544+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a98145cd0 con 0x7f6a980634c0 2020-11-20T12:05:14.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.545+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a980a1900 con 0x7f6a980634c0 2020-11-20T12:05:14.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.546+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a98146660 con 0x7f6a980634c0 2020-11-20T12:05:14.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.546+0000 7f6a8effd700 1 -- 172.21.15.187:0/419421772 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6a80013480 con 0x7f6a980634c0 2020-11-20T12:05:14.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.547+0000 7f6a8effd700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a74056d20 0x7f6a740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.547+0000 7f6a8effd700 1 -- 172.21.15.187:0/419421772 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6a80070380 con 0x7f6a980634c0 2020-11-20T12:05:14.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.547+0000 7f6a9659c700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a74056d20 0x7f6a740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:14.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.548+0000 7f6a9659c700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a74056d20 0x7f6a740591e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f6a88005920 tx=0x7f6a88009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:14.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.552+0000 7f6a8effd700 1 -- 172.21.15.187:0/419421772 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6a800457f0 con 0x7f6a980634c0 2020-11-20T12:05:14.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.723+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a980b6cd0 con 0x7f6a980634c0 2020-11-20T12:05:14.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.724+0000 7f6a8effd700 1 -- 172.21.15.187:0/419421772 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6a80048e10 con 0x7f6a980634c0 2020-11-20T12:05:14.729 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:14.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.726+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a74056d20 msgr2=0x7f6a740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.726+0000 7f6a9759e700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a74056d20 0x7f6a740591e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f6a88005920 tx=0x7f6a88009040).stop 2020-11-20T12:05:14.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.726+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a980634c0 msgr2=0x7f6a98146170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:14.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.726+0000 7f6a9759e700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a980634c0 0x7f6a98146170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f6a8000db30 tx=0x7f6a80008040).stop 2020-11-20T12:05:14.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.727+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 shutdown_connections 2020-11-20T12:05:14.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.727+0000 7f6a9759e700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a74056d20 0x7f6a740591e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.727+0000 7f6a9759e700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a980630e0 0x7f6a980b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.727+0000 7f6a9759e700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a9809aff0 0x7f6a980a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.727+0000 7f6a9759e700 1 --2- 172.21.15.187:0/419421772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a980634c0 0x7f6a98146170 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:14.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.727+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 >> 172.21.15.187:0/419421772 conn(0x7f6a980bbc20 msgr2=0x7f6a98060170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:14.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.727+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 shutdown_connections 2020-11-20T12:05:14.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.727+0000 7f6a9759e700 1 -- 172.21.15.187:0/419421772 wait complete. 2020-11-20T12:05:14.754 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:14.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.991+0000 7f665d780700 1 Processor -- start 2020-11-20T12:05:14.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.992+0000 7f665d780700 1 -- start start 2020-11-20T12:05:14.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.993+0000 7f665d780700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f665809f560 0x7f6658153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:14.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.993+0000 7f665d780700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665814d110 con 0x7f66580bf800 2020-11-20T12:05:14.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.993+0000 7f665d780700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665814cbd0 con 0x7f66580bfbe0 2020-11-20T12:05:14.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.993+0000 7f665d780700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665814c9d0 con 0x7f665809f560 2020-11-20T12:05:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.993+0000 7f664ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f66580bfbe0 0x7f66581480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37500/0 (socket says 172.21.15.187:37500) 2020-11-20T12:05:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.993+0000 7f664ffff700 1 -- 172.21.15.187:0/2686957818 learned_addr learned my addr 172.21.15.187:0/2686957818 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f665cf7f700 1 --2- 172.21.15.187:0/2686957818 >> v2:172.21.15.187:3300/0 conn(0x7f665809f560 0x7f6658153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489322767 0 0) 0x7f665814cbd0 con 0x7f66580bfbe0 2020-11-20T12:05:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6638003240 con 0x7f66580bfbe0 2020-11-20T12:05:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1406106244 0 0) 0x7f6638003240 con 0x7f66580bfbe0 2020-11-20T12:05:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 >> v2:172.21.15.187:3300/0 conn(0x7f665809f560 msgr2=0x7f6658153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 --2- 172.21.15.187:0/2686957818 >> v2:172.21.15.187:3300/0 conn(0x7f665809f560 0x7f6658153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 >> v1:172.21.15.5:6789/0 conn(0x7f66580bf800 legacy=0x7f665814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665814ce70 con 0x7f66580bfbe0 2020-11-20T12:05:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f66400014e0 con 0x7f66580bfbe0 2020-11-20T12:05:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f665cf7f700 1 --2- 172.21.15.187:0/2686957818 >> v2:172.21.15.187:3300/0 conn(0x7f665809f560 0x7f6658153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:05:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.994+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 >> v1:172.21.15.187:6789/0 conn(0x7f66580bfbe0 legacy=0x7f66581480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f66577fe700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6638003780 0x7f6638003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f66577fe700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66380044e0 0x7f6638004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f66577fe700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6638004e50 0x7f663800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665814c9d0 con 0x7f6638004e50 2020-11-20T12:05:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6638005300 con 0x7f66380044e0 2020-11-20T12:05:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663800bc20 con 0x7f6638003780 2020-11-20T12:05:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f664ffff700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66380044e0 0x7f6638004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f664ffff700 1 -- 172.21.15.187:0/2686957818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6638003780 msgr2=0x7f6638003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f664ffff700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6638003780 0x7f6638003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f664ffff700 1 -- 172.21.15.187:0/2686957818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6638004e50 msgr2=0x7f663800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f665cf7f700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6638004e50 0x7f663800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f664ffff700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6638004e50 0x7f663800b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f664ffff700 1 -- 172.21.15.187:0/2686957818 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6638003240 con 0x7f66380044e0 2020-11-20T12:05:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.995+0000 7f665cf7f700 1 -- 172.21.15.187:0/2686957818 reap_dead start 2020-11-20T12:05:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f664ffff700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66380044e0 0x7f6638004960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f664000c230 tx=0x7f665814d000).ready entity=mon.1 client_cookie=c89a335be4de4ada server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400070d0 con 0x7f66380044e0 2020-11-20T12:05:15.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640007230 con 0x7f66380044e0 2020-11-20T12:05:15.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f66577fe700 1 -- 172.21.15.187:0/2686957818 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400055f0 con 0x7f66380044e0 2020-11-20T12:05:15.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f665d780700 1 -- 172.21.15.187:0/2686957818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66380044e0 msgr2=0x7f6638004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f665d780700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66380044e0 0x7f6638004960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f664000c230 tx=0x7f665814d000).stop 2020-11-20T12:05:15.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f665d780700 1 -- 172.21.15.187:0/2686957818 shutdown_connections 2020-11-20T12:05:15.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f665d780700 1 --2- 172.21.15.187:0/2686957818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66380044e0 0x7f6638004960 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.996+0000 7f665d780700 1 -- 172.21.15.187:0/2686957818 >> 172.21.15.187:0/2686957818 conn(0x7f6658068370 msgr2=0x7f66580bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.997+0000 7f665d780700 1 -- 172.21.15.187:0/2686957818 shutdown_connections 2020-11-20T12:05:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.997+0000 7f665d780700 1 -- 172.21.15.187:0/2686957818 wait complete. 2020-11-20T12:05:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.998+0000 7f665d780700 1 Processor -- start 2020-11-20T12:05:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.998+0000 7f665d780700 1 -- start start 2020-11-20T12:05:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f665d780700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66380044e0 0x7f6658142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f665d780700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66580ba2e0 0x7f6658147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f665d780700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66580ba760 0x7f665813ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f665d780700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66580babf0 con 0x7f66580ba760 2020-11-20T12:05:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f665d780700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6658145f00 con 0x7f66580ba2e0 2020-11-20T12:05:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f665d780700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66581433a0 con 0x7f66380044e0 2020-11-20T12:05:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f664ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66580ba2e0 0x7f6658147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f664ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66580ba2e0 0x7f6658147c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40906/0 (socket says 172.21.15.187:40906) 2020-11-20T12:05:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:14.999+0000 7f664ffff700 1 -- 172.21.15.187:0/463596495 learned_addr learned my addr 172.21.15.187:0/463596495 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f664ffff700 1 -- 172.21.15.187:0/463596495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66380044e0 msgr2=0x7f6658142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f664ffff700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66380044e0 0x7f6658142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f664ffff700 1 -- 172.21.15.187:0/463596495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66580ba760 msgr2=0x7f665813ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f664ffff700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66580ba760 0x7f665813ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f664ffff700 1 -- 172.21.15.187:0/463596495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66580b49f0 con 0x7f66580ba2e0 2020-11-20T12:05:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f664ffff700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66580ba2e0 0x7f6658147c40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f664000cce0 tx=0x7f6640013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f6654ff9700 1 -- 172.21.15.187:0/463596495 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640013450 con 0x7f66580ba2e0 2020-11-20T12:05:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f6654ff9700 1 -- 172.21.15.187:0/463596495 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66400135b0 con 0x7f66580ba2e0 2020-11-20T12:05:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6658148130 con 0x7f66580ba2e0 2020-11-20T12:05:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f6654ff9700 1 -- 172.21.15.187:0/463596495 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640013970 con 0x7f66580ba2e0 2020-11-20T12:05:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.000+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665813f0b0 con 0x7f66580ba2e0 2020-11-20T12:05:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.001+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6658080050 con 0x7f66580ba2e0 2020-11-20T12:05:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.002+0000 7f6654ff9700 1 -- 172.21.15.187:0/463596495 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6640026070 con 0x7f66580ba2e0 2020-11-20T12:05:15.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.002+0000 7f6654ff9700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6634056d50 0x7f6634059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.002+0000 7f6654ff9700 1 -- 172.21.15.187:0/463596495 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6640070f10 con 0x7f66580ba2e0 2020-11-20T12:05:15.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.003+0000 7f6657fff700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6634056d50 0x7f6634059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.003+0000 7f6657fff700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6634056d50 0x7f6634059210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6648007fd0 tx=0x7f6648007790).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.006+0000 7f6654ff9700 1 -- 172.21.15.187:0/463596495 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6640046250 con 0x7f66580ba2e0 2020-11-20T12:05:15.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.191+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6658146090 con 0x7f66580ba2e0 2020-11-20T12:05:15.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.192+0000 7f6654ff9700 1 -- 172.21.15.187:0/463596495 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6640049870 con 0x7f66580ba2e0 2020-11-20T12:05:15.196 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:15.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6634056d50 msgr2=0x7f6634059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6634056d50 0x7f6634059210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6648007fd0 tx=0x7f6648007790).stop 2020-11-20T12:05:15.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66580ba2e0 msgr2=0x7f6658147c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66580ba2e0 0x7f6658147c40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f664000cce0 tx=0x7f6640013000).stop 2020-11-20T12:05:15.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 shutdown_connections 2020-11-20T12:05:15.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6634056d50 0x7f6634059210 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66580ba760 0x7f665813ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:15.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66380044e0 0x7f6658142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:15.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 --2- 172.21.15.187:0/463596495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66580ba2e0 0x7f6658147c40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 >> 172.21.15.187:0/463596495 conn(0x7f6658068370 msgr2=0x7f66580bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 shutdown_connections 2020-11-20T12:05:15.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.194+0000 7f665d780700 1 -- 172.21.15.187:0/463596495 wait complete. 2020-11-20T12:05:15.221 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:15.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.455+0000 7f8e5f59e700 1 Processor -- start 2020-11-20T12:05:15.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.456+0000 7f8e5f59e700 1 -- start start 2020-11-20T12:05:15.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e5f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e601484d0 0x7f8e60061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e5f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e600622c0 0x7f8e600b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e5f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8e60066f70 0x7f8e600b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e5f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e60067730 con 0x7f8e60066f70 2020-11-20T12:05:15.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e5f59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e600b1300 con 0x7f8e600622c0 2020-11-20T12:05:15.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e5f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e600b15a0 con 0x7f8e601484d0 2020-11-20T12:05:15.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e55d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e600622c0 0x7f8e600b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e55d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e600622c0 0x7f8e600b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40912/0 (socket says 172.21.15.187:40912) 2020-11-20T12:05:15.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.457+0000 7f8e55d9b700 1 -- 172.21.15.187:0/412626757 learned_addr learned my addr 172.21.15.187:0/412626757 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:15.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e55d9b700 1 -- 172.21.15.187:0/412626757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e601484d0 msgr2=0x7f8e60061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:15.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e5ed9d700 1 --2- 172.21.15.187:0/412626757 >> v2:172.21.15.5:3300/0 conn(0x7f8e60066f70 0x7f8e600b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e5e59c700 1 --2- 172.21.15.187:0/412626757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e601484d0 0x7f8e60061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e55d9b700 1 --2- 172.21.15.187:0/412626757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e601484d0 0x7f8e60061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e55d9b700 1 -- 172.21.15.187:0/412626757 >> v2:172.21.15.5:3300/0 conn(0x7f8e60066f70 msgr2=0x7f8e600b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:15.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e55d9b700 1 --2- 172.21.15.187:0/412626757 >> v2:172.21.15.5:3300/0 conn(0x7f8e60066f70 0x7f8e600b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e55d9b700 1 -- 172.21.15.187:0/412626757 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e600b1840 con 0x7f8e600622c0 2020-11-20T12:05:15.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e5ed9d700 1 --2- 172.21.15.187:0/412626757 >> v2:172.21.15.5:3300/0 conn(0x7f8e60066f70 0x7f8e600b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:15.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e5e59c700 1 --2- 172.21.15.187:0/412626757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e601484d0 0x7f8e60061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:15.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.458+0000 7f8e55d9b700 1 --2- 172.21.15.187:0/412626757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600622c0 0x7f8e600b5640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8e60066380 tx=0x7f8e48006eb0).ready entity=mon.1 client_cookie=36c6839deabca128 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5dd9b700 1 -- 172.21.15.187:0/412626757 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e480098d0 con 0x7f8e600622c0 2020-11-20T12:05:15.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5dd9b700 1 -- 172.21.15.187:0/412626757 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e48009a30 con 0x7f8e600622c0 2020-11-20T12:05:15.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5dd9b700 1 -- 172.21.15.187:0/412626757 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48009db0 con 0x7f8e600622c0 2020-11-20T12:05:15.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5f59e700 1 -- 172.21.15.187:0/412626757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600622c0 msgr2=0x7f8e600b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/412626757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600622c0 0x7f8e600b5640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8e60066380 tx=0x7f8e48006eb0).stop 2020-11-20T12:05:15.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5f59e700 1 -- 172.21.15.187:0/412626757 shutdown_connections 2020-11-20T12:05:15.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/412626757 >> v2:172.21.15.5:3300/0 conn(0x7f8e60066f70 0x7f8e600b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/412626757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e601484d0 0x7f8e60061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/412626757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600622c0 0x7f8e600b5640 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.459+0000 7f8e5f59e700 1 -- 172.21.15.187:0/412626757 >> 172.21.15.187:0/412626757 conn(0x7f8e600bab30 msgr2=0x7f8e600baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.460+0000 7f8e5f59e700 1 -- 172.21.15.187:0/412626757 shutdown_connections 2020-11-20T12:05:15.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.460+0000 7f8e5f59e700 1 -- 172.21.15.187:0/412626757 wait complete. 2020-11-20T12:05:15.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.461+0000 7f8e5f59e700 1 Processor -- start 2020-11-20T12:05:15.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.461+0000 7f8e5f59e700 1 -- start start 2020-11-20T12:05:15.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e600622c0 0x7f8e6009b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e60066f70 0x7f8e60136df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e601484d0 0x7f8e601372e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e600a0b90 con 0x7f8e600622c0 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6009ff50 con 0x7f8e601484d0 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e600a0450 con 0x7f8e60066f70 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e601484d0 0x7f8e601372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e601484d0 0x7f8e601372e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40922/0 (socket says 172.21.15.187:40922) 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.462+0000 7f8e5ed9d700 1 -- 172.21.15.187:0/339444244 learned_addr learned my addr 172.21.15.187:0/339444244 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:15.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5e59c700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e600622c0 0x7f8e6009b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e55d9b700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e60066f70 0x7f8e60136df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5ed9d700 1 -- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e60066f70 msgr2=0x7f8e60136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5ed9d700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e60066f70 0x7f8e60136df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5ed9d700 1 -- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e600622c0 msgr2=0x7f8e6009b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5ed9d700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e600622c0 0x7f8e6009b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5ed9d700 1 -- 172.21.15.187:0/339444244 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e600a06f0 con 0x7f8e601484d0 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5e59c700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e600622c0 0x7f8e6009b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5ed9d700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e601484d0 0x7f8e601372e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f8e5800c8a0 tx=0x7f8e58008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e56ffd700 1 -- 172.21.15.187:0/339444244 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58007620 con 0x7f8e601484d0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e56ffd700 1 -- 172.21.15.187:0/339444244 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e58007780 con 0x7f8e601484d0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6009c550 con 0x7f8e601484d0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.463+0000 7f8e56ffd700 1 -- 172.21.15.187:0/339444244 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58007b20 con 0x7f8e601484d0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.464+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6009fab0 con 0x7f8e601484d0 2020-11-20T12:05:15.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.465+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e60061a20 con 0x7f8e601484d0 2020-11-20T12:05:15.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.465+0000 7f8e56ffd700 1 -- 172.21.15.187:0/339444244 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8e5801b460 con 0x7f8e601484d0 2020-11-20T12:05:15.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.466+0000 7f8e56ffd700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e3c056d20 0x7f8e3c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.466+0000 7f8e56ffd700 1 -- 172.21.15.187:0/339444244 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e5806ff30 con 0x7f8e601484d0 2020-11-20T12:05:15.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.466+0000 7f8e5e59c700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e3c056d20 0x7f8e3c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.467+0000 7f8e5e59c700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e3c056d20 0x7f8e3c0591e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f8e50008ac0 tx=0x7f8e50008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.470+0000 7f8e56ffd700 1 -- 172.21.15.187:0/339444244 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8e58045320 con 0x7f8e601484d0 2020-11-20T12:05:15.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.705+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e601488b0 con 0x7f8e601484d0 2020-11-20T12:05:15.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.706+0000 7f8e56ffd700 1 -- 172.21.15.187:0/339444244 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2130 (secure 0 0 0) 0x7f8e58048940 con 0x7f8e601484d0 2020-11-20T12:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:15.712 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4211,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6833/2094645874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2094645874},{"type":"v1","addr":"172.21.15.5:6833","nonce":2094645874}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:57.868606+0000","modified":"2020-11-20T12:04:59.990344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3094884769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3094884769},{"type":"v1","addr":"172.21.15.187:6835","nonce":3094884769}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:15.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e3c056d20 msgr2=0x7f8e3c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e3c056d20 0x7f8e3c0591e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f8e50008ac0 tx=0x7f8e50008040).stop 2020-11-20T12:05:15.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e601484d0 msgr2=0x7f8e601372e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e601484d0 0x7f8e601372e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f8e5800c8a0 tx=0x7f8e58008040).stop 2020-11-20T12:05:15.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 shutdown_connections 2020-11-20T12:05:15.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e3c056d20 0x7f8e3c0591e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e600622c0 0x7f8e6009b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e601484d0 0x7f8e601372e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 --2- 172.21.15.187:0/339444244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e60066f70 0x7f8e60136df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 >> 172.21.15.187:0/339444244 conn(0x7f8e600bab30 msgr2=0x7f8e600b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 shutdown_connections 2020-11-20T12:05:15.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:15.709+0000 7f8e5f59e700 1 -- 172.21.15.187:0/339444244 wait complete. 2020-11-20T12:05:15.716 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 6 2020-11-20T12:05:15.938 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:04:57.868606+0000', 'modified': '2020-11-20T12:04:59.990344+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4264': {'gid': 4264, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3094884769', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3094884769}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3094884769}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:05:15.938 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:05:15.939 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:05:16.212 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":4264,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":18.317710075000001,"mdsmap_epoch":6,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":22.70653111} 2020-11-20T12:05:16.213 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 4264, "mdsmap_epoch": 6, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 18.317710075, "state": "up:active", "uptime": 22.70653111, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:05:16.213 INFO:teuthology.run_tasks:Running task kclient... 2020-11-20T12:05:16.247 INFO:tasks.kclient:Mounting kernel clients... 2020-11-20T12:05:16.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.248 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:05:16.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.306 INFO:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2020-11-20T12:05:16.412 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2020-11-20T12:05:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:05:16.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.416 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:05:16.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.470 INFO:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2020-11-20T12:05:16.576 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2020-11-20T12:05:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:05:16.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.583 INFO:teuthology.orchestra.run.smithi005:> ip netns list 2020-11-20T12:05:16.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.646 INFO:teuthology.orchestra.run.smithi005:> sudo ip link delete ceph-brx 2020-11-20T12:05:16.764 INFO:teuthology.orchestra.run.smithi005.stderr:Cannot find device "ceph-brx" 2020-11-20T12:05:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:05:16.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.772 INFO:teuthology.orchestra.run.smithi005:> ip netns list 2020-11-20T12:05:16.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.831 INFO:teuthology.orchestra.run.smithi005:> sudo ip link delete ceph-brx 2020-11-20T12:05:16.942 INFO:teuthology.orchestra.run.smithi005.stderr:Cannot find device "ceph-brx" 2020-11-20T12:05:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:05:16.946 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:05:16.947 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:05:16.947 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:05:16.948 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:05:16.948 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:05:16.948 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2020-11-20T12:05:16.949 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:05:16.949 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:05:16.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:16.950 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:05:17.010 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:17.010 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:17.010 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:17.011 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:17.011 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:17.011 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:17.011 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:17.011 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:17.012 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:17.012 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:17.012 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:17.012 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:17.013 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:05:17.013 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:17.013 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:05:17.013 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42721sec preferred_lft 42721sec 2020-11-20T12:05:17.013 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:05:17.014 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:17.015 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2020-11-20T12:05:17.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:17.016 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:17.016 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:17.016 INFO:teuthology.orchestra.run.smithi187:> sudo ip link add name ceph-brx type bridge 2020-11-20T12:05:17.017 INFO:teuthology.orchestra.run.smithi187:> sudo ip addr flush dev ceph-brx 2020-11-20T12:05:17.017 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set ceph-brx up 2020-11-20T12:05:17.017 INFO:teuthology.orchestra.run.smithi187:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2020-11-20T12:05:17.018 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:17.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:17.615 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:05:17.698 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:05:17.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:17.702 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:05:17.778 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:05:17.779 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:05:17.779 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:05:17.780 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:05:17.780 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:05:17.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:17.783 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:17.783 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:17.784 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:05:17.784 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:05:17.785 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:05:17.785 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:17.980 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:05:17.989 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:05:18.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:18.715 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:05:18.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:18.771 INFO:teuthology.orchestra.run.smithi187:> ip netns list-id 2020-11-20T12:05:18.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:18.827 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:18.827 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:18.828 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:05:18.828 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2020-11-20T12:05:18.828 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:19.039 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2020-11-20T12:05:19.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:19.040 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:19.040 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:19.040 INFO:teuthology.orchestra.run.smithi187:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2020-11-20T12:05:19.041 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2020-11-20T12:05:19.041 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2020-11-20T12:05:19.041 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2020-11-20T12:05:19.042 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2020-11-20T12:05:19.042 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:19.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:19.436 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:19.436 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:19.437 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:05:19.437 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set dev brx.0 master ceph-brx 2020-11-20T12:05:19.437 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:19.673 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:19.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.905+0000 7f720a250700 1 Processor -- start 2020-11-20T12:05:19.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.905+0000 7f720a250700 1 -- start start 2020-11-20T12:05:19.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.906+0000 7f720a250700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040654a0 0x7f72040b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.906+0000 7f720a250700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7204065880 0x7f7204144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.906+0000 7f720a250700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f72040b6c10 0x7f7204145350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.906+0000 7f720a250700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72040b08e0 con 0x7f7204065880 2020-11-20T12:05:19.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.906+0000 7f720a250700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72040b0b80 con 0x7f72040b6c10 2020-11-20T12:05:19.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.906+0000 7f720a250700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72040b05f0 con 0x7f72040654a0 2020-11-20T12:05:19.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7203fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f72040b6c10 0x7f7204145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7203fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f72040b6c10 0x7f7204145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40930/0 (socket says 172.21.15.187:40930) 2020-11-20T12:05:19.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7203fff700 1 -- 172.21.15.187:0/4227963698 learned_addr learned my addr 172.21.15.187:0/4227963698 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:19.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f72037fe700 1 --2- 172.21.15.187:0/4227963698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040654a0 0x7f72040b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7203fff700 1 -- 172.21.15.187:0/4227963698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040654a0 msgr2=0x7f72040b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7202ffd700 1 --2- 172.21.15.187:0/4227963698 >> v2:172.21.15.5:3300/0 conn(0x7f7204065880 0x7f7204144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7203fff700 1 --2- 172.21.15.187:0/4227963698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040654a0 0x7f72040b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7203fff700 1 -- 172.21.15.187:0/4227963698 >> v2:172.21.15.5:3300/0 conn(0x7f7204065880 msgr2=0x7f7204144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7203fff700 1 --2- 172.21.15.187:0/4227963698 >> v2:172.21.15.5:3300/0 conn(0x7f7204065880 0x7f7204144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7203fff700 1 -- 172.21.15.187:0/4227963698 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040b5880 con 0x7f72040b6c10 2020-11-20T12:05:19.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f72037fe700 1 --2- 172.21.15.187:0/4227963698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040654a0 0x7f72040b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:19.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.907+0000 7f7202ffd700 1 --2- 172.21.15.187:0/4227963698 >> v2:172.21.15.5:3300/0 conn(0x7f7204065880 0x7f7204144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:19.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.908+0000 7f7203fff700 1 --2- 172.21.15.187:0/4227963698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72040b6c10 0x7f7204145350 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f7204148ca0 tx=0x7f71fc009290).ready entity=mon.1 client_cookie=8361ded68f547fd0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.908+0000 7f72027fc700 1 -- 172.21.15.187:0/4227963698 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc00bbc0 con 0x7f72040b6c10 2020-11-20T12:05:19.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.908+0000 7f72027fc700 1 -- 172.21.15.187:0/4227963698 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71fc00bd20 con 0x7f72040b6c10 2020-11-20T12:05:19.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.908+0000 7f72027fc700 1 -- 172.21.15.187:0/4227963698 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc011610 con 0x7f72040b6c10 2020-11-20T12:05:19.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 -- 172.21.15.187:0/4227963698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72040b6c10 msgr2=0x7f7204145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 --2- 172.21.15.187:0/4227963698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72040b6c10 0x7f7204145350 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f7204148ca0 tx=0x7f71fc009290).stop 2020-11-20T12:05:19.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 -- 172.21.15.187:0/4227963698 shutdown_connections 2020-11-20T12:05:19.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 --2- 172.21.15.187:0/4227963698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72040b6c10 0x7f7204145350 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 --2- 172.21.15.187:0/4227963698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040654a0 0x7f72040b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 --2- 172.21.15.187:0/4227963698 >> v2:172.21.15.5:3300/0 conn(0x7f7204065880 0x7f7204144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 -- 172.21.15.187:0/4227963698 >> 172.21.15.187:0/4227963698 conn(0x7f72040bacd0 msgr2=0x7f72040bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:19.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 -- 172.21.15.187:0/4227963698 shutdown_connections 2020-11-20T12:05:19.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.909+0000 7f720a250700 1 -- 172.21.15.187:0/4227963698 wait complete. 2020-11-20T12:05:19.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.911+0000 7f720a250700 1 Processor -- start 2020-11-20T12:05:19.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.911+0000 7f720a250700 1 -- start start 2020-11-20T12:05:19.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f720a250700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72040654a0 0x7f720409d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f720a250700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7204065880 0x7f7204157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f720a250700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040b6c10 0x7f720414c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f720a250700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72040b47e0 con 0x7f72040654a0 2020-11-20T12:05:19.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f720a250700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72040b42a0 con 0x7f7204065880 2020-11-20T12:05:19.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f720a250700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72040b4540 con 0x7f72040b6c10 2020-11-20T12:05:19.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f7202ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7204065880 0x7f7204157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f7202ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7204065880 0x7f7204157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40934/0 (socket says 172.21.15.187:40934) 2020-11-20T12:05:19.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.912+0000 7f7202ffd700 1 -- 172.21.15.187:0/2124673498 learned_addr learned my addr 172.21.15.187:0/2124673498 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:19.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f72037fe700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72040654a0 0x7f720409d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f7203fff700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040b6c10 0x7f720414c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f7202ffd700 1 -- 172.21.15.187:0/2124673498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040b6c10 msgr2=0x7f720414c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f7202ffd700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040b6c10 0x7f720414c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f7202ffd700 1 -- 172.21.15.187:0/2124673498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72040654a0 msgr2=0x7f720409d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f7202ffd700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72040654a0 0x7f720409d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f7202ffd700 1 -- 172.21.15.187:0/2124673498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72041580f0 con 0x7f7204065880 2020-11-20T12:05:19.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f7202ffd700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7204065880 0x7f7204157c00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f71f800c8f0 tx=0x7f71f8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f71e37fe700 1 -- 172.21.15.187:0/2124673498 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8015070 con 0x7f7204065880 2020-11-20T12:05:19.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f71e37fe700 1 -- 172.21.15.187:0/2124673498 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f8006020 con 0x7f7204065880 2020-11-20T12:05:19.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f71e37fe700 1 -- 172.21.15.187:0/2124673498 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80063a0 con 0x7f7204065880 2020-11-20T12:05:19.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.913+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720414c560 con 0x7f7204065880 2020-11-20T12:05:19.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.914+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720414aa90 con 0x7f7204065880 2020-11-20T12:05:19.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.915+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720415b220 con 0x7f7204065880 2020-11-20T12:05:19.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.915+0000 7f71e37fe700 1 -- 172.21.15.187:0/2124673498 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f71f8007920 con 0x7f7204065880 2020-11-20T12:05:19.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.916+0000 7f71e37fe700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71dc056d20 0x7f71dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.916+0000 7f71e37fe700 1 -- 172.21.15.187:0/2124673498 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f71f806fd20 con 0x7f7204065880 2020-11-20T12:05:19.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.916+0000 7f72037fe700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71dc056d20 0x7f71dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.917+0000 7f72037fe700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71dc056d20 0x7f71dc0591e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f71f400a8c0 tx=0x7f71f4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:19.920+0000 7f71e37fe700 1 -- 172.21.15.187:0/2124673498 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f720415b220 con 0x7f7204065880 2020-11-20T12:05:20.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.106+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f720415b220 con 0x7f7204065880 2020-11-20T12:05:20.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.109+0000 7f71e37fe700 1 -- 172.21.15.187:0/2124673498 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f720415b220 con 0x7f7204065880 2020-11-20T12:05:20.113 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:20.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71dc056d20 msgr2=0x7f71dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71dc056d20 0x7f71dc0591e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f71f400a8c0 tx=0x7f71f4008040).stop 2020-11-20T12:05:20.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7204065880 msgr2=0x7f7204157c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7204065880 0x7f7204157c00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f71f800c8f0 tx=0x7f71f8008040).stop 2020-11-20T12:05:20.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 shutdown_connections 2020-11-20T12:05:20.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71dc056d20 0x7f71dc0591e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72040654a0 0x7f720409d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72040b6c10 0x7f720414c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 --2- 172.21.15.187:0/2124673498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7204065880 0x7f7204157c00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.111+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 >> 172.21.15.187:0/2124673498 conn(0x7f72040bacd0 msgr2=0x7f720409f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.112+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 shutdown_connections 2020-11-20T12:05:20.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.112+0000 7f720a250700 1 -- 172.21.15.187:0/2124673498 wait complete. 2020-11-20T12:05:20.138 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:20.138 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:20.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.378+0000 7f59249c2700 1 Processor -- start 2020-11-20T12:05:20.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.378+0000 7f59249c2700 1 -- start start 2020-11-20T12:05:20.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.379+0000 7f59249c2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59201484d0 0x7f5920061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.379+0000 7f59249c2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5920066f70 0x7f59200b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.379+0000 7f59249c2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59200b1300 con 0x7f59200622c0 2020-11-20T12:05:20.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.379+0000 7f59249c2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5920067730 con 0x7f59201484d0 2020-11-20T12:05:20.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.379+0000 7f59249c2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59200b15a0 con 0x7f5920066f70 2020-11-20T12:05:20.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.380+0000 7f591f7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59201484d0 0x7f5920061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.380+0000 7f591f7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59201484d0 0x7f5920061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40940/0 (socket says 172.21.15.187:40940) 2020-11-20T12:05:20.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.380+0000 7f591f7fe700 1 -- 172.21.15.187:0/2269287242 learned_addr learned my addr 172.21.15.187:0/2269287242 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:20.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.380+0000 7f591f7fe700 1 -- 172.21.15.187:0/2269287242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5920066f70 msgr2=0x7f59200b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.380+0000 7f591f7fe700 1 --2- 172.21.15.187:0/2269287242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5920066f70 0x7f59200b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:20.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.380+0000 7f591f7fe700 1 -- 172.21.15.187:0/2269287242 >> v1:172.21.15.5:6789/0 conn(0x7f59200622c0 legacy=0x7f59200b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.380+0000 7f591f7fe700 1 -- 172.21.15.187:0/2269287242 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59200b1840 con 0x7f59201484d0 2020-11-20T12:05:20.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.380+0000 7f591f7fe700 1 --2- 172.21.15.187:0/2269287242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59201484d0 0x7f5920061dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f5910006fd0 tx=0x7f5910006eb0).ready entity=mon.1 client_cookie=9127591497f79ddc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f591effd700 1 -- 172.21.15.187:0/2269287242 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5910009890 con 0x7f59201484d0 2020-11-20T12:05:20.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f591effd700 1 -- 172.21.15.187:0/2269287242 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59100099f0 con 0x7f59201484d0 2020-11-20T12:05:20.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f591effd700 1 -- 172.21.15.187:0/2269287242 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5910009d70 con 0x7f59201484d0 2020-11-20T12:05:20.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f59249c2700 1 -- 172.21.15.187:0/2269287242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59201484d0 msgr2=0x7f5920061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f59249c2700 1 --2- 172.21.15.187:0/2269287242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59201484d0 0x7f5920061dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f5910006fd0 tx=0x7f5910006eb0).stop 2020-11-20T12:05:20.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f59249c2700 1 -- 172.21.15.187:0/2269287242 shutdown_connections 2020-11-20T12:05:20.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f59249c2700 1 --2- 172.21.15.187:0/2269287242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5920066f70 0x7f59200b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:20.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f59249c2700 1 --2- 172.21.15.187:0/2269287242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59201484d0 0x7f5920061dd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f59249c2700 1 -- 172.21.15.187:0/2269287242 >> 172.21.15.187:0/2269287242 conn(0x7f59200bab30 msgr2=0x7f59200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f59249c2700 1 -- 172.21.15.187:0/2269287242 shutdown_connections 2020-11-20T12:05:20.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.381+0000 7f59249c2700 1 -- 172.21.15.187:0/2269287242 wait complete. 2020-11-20T12:05:20.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.383+0000 7f59249c2700 1 Processor -- start 2020-11-20T12:05:20.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.383+0000 7f59249c2700 1 -- start start 2020-11-20T12:05:20.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.384+0000 7f59249c2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200622c0 0x7f5920142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.384+0000 7f59249c2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5920066f70 0x7f592015ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.384+0000 7f59249c2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201484d0 0x7f592015bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.384+0000 7f59249c2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592009fff0 con 0x7f5920066f70 2020-11-20T12:05:20.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.384+0000 7f59249c2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59200a0290 con 0x7f59200622c0 2020-11-20T12:05:20.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.384+0000 7f59249c2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59200a0530 con 0x7f59201484d0 2020-11-20T12:05:20.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.384+0000 7f591f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200622c0 0x7f5920142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200622c0 0x7f5920142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40946/0 (socket says 172.21.15.187:40946) 2020-11-20T12:05:20.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591f7fe700 1 -- 172.21.15.187:0/1046572592 learned_addr learned my addr 172.21.15.187:0/1046572592 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:20.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f5916ffd700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5920066f70 0x7f592015ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591ffff700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201484d0 0x7f592015bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591f7fe700 1 -- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201484d0 msgr2=0x7f592015bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591f7fe700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201484d0 0x7f592015bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591f7fe700 1 -- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5920066f70 msgr2=0x7f592015ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591f7fe700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5920066f70 0x7f592015ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591f7fe700 1 -- 172.21.15.187:0/1046572592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59200a09d0 con 0x7f59200622c0 2020-11-20T12:05:20.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591ffff700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201484d0 0x7f592015bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:20.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f591f7fe700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200622c0 0x7f5920142cd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f5910006ef0 tx=0x7f5910015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.385+0000 7f5917fff700 1 -- 172.21.15.187:0/1046572592 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591001e460 con 0x7f59200622c0 2020-11-20T12:05:20.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.386+0000 7f5917fff700 1 -- 172.21.15.187:0/1046572592 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591001e600 con 0x7f59200622c0 2020-11-20T12:05:20.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.386+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592009c700 con 0x7f59200622c0 2020-11-20T12:05:20.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.386+0000 7f5917fff700 1 -- 172.21.15.187:0/1046572592 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591001e9c0 con 0x7f59200622c0 2020-11-20T12:05:20.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.386+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592015c640 con 0x7f59200622c0 2020-11-20T12:05:20.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.387+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5920064080 con 0x7f59200622c0 2020-11-20T12:05:20.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.387+0000 7f5917fff700 1 -- 172.21.15.187:0/1046572592 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5910025030 con 0x7f59200622c0 2020-11-20T12:05:20.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.389+0000 7f5917fff700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58fc056d20 0x7f58fc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.389+0000 7f5917fff700 1 -- 172.21.15.187:0/1046572592 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5910071930 con 0x7f59200622c0 2020-11-20T12:05:20.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.389+0000 7f5916ffd700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58fc056d20 0x7f58fc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.390+0000 7f5916ffd700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58fc056d20 0x7f58fc0591e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f592009be80 tx=0x7f5908008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.394+0000 7f5917fff700 1 -- 172.21.15.187:0/1046572592 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5910046d20 con 0x7f59200622c0 2020-11-20T12:05:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.580+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5920061a20 con 0x7f59200622c0 2020-11-20T12:05:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.581+0000 7f5917fff700 1 -- 172.21.15.187:0/1046572592 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5910023020 con 0x7f59200622c0 2020-11-20T12:05:20.585 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:20.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.583+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58fc056d20 msgr2=0x7f58fc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.583+0000 7f59249c2700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58fc056d20 0x7f58fc0591e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f592009be80 tx=0x7f5908008040).stop 2020-11-20T12:05:20.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.583+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200622c0 msgr2=0x7f5920142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.583+0000 7f59249c2700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200622c0 0x7f5920142cd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f5910006ef0 tx=0x7f5910015000).stop 2020-11-20T12:05:20.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.584+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 shutdown_connections 2020-11-20T12:05:20.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.584+0000 7f59249c2700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58fc056d20 0x7f58fc0591e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.584+0000 7f59249c2700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201484d0 0x7f592015bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.584+0000 7f59249c2700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200622c0 0x7f5920142cd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.584+0000 7f59249c2700 1 --2- 172.21.15.187:0/1046572592 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5920066f70 0x7f592015ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.584+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 >> 172.21.15.187:0/1046572592 conn(0x7f59200bab30 msgr2=0x7f5920144250 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.585+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 shutdown_connections 2020-11-20T12:05:20.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.585+0000 7f59249c2700 1 -- 172.21.15.187:0/1046572592 wait complete. 2020-11-20T12:05:20.615 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:20.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.854+0000 7f484ecde700 1 Processor -- start 2020-11-20T12:05:20.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.854+0000 7f484ecde700 1 -- start start 2020-11-20T12:05:20.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.856+0000 7f484ecde700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48481484d0 0x7f4848061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.856+0000 7f484ecde700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480b1300 con 0x7f4848066f70 2020-11-20T12:05:20.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.856+0000 7f484ecde700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480b15a0 con 0x7f48480622c0 2020-11-20T12:05:20.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.856+0000 7f484ecde700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4848067730 con 0x7f48481484d0 2020-11-20T12:05:20.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.856+0000 7f484dcdc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48481484d0 0x7f4848061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.856+0000 7f484d4db700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f48480622c0 0x7f48480b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37562/0 (socket says 172.21.15.187:37562) 2020-11-20T12:05:20.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.856+0000 7f484d4db700 1 -- 172.21.15.187:0/1087372772 learned_addr learned my addr 172.21.15.187:0/1087372772 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:20.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.857+0000 7f484dcdc700 1 -- 172.21.15.187:0/1087372772 >> v1:172.21.15.187:6789/0 conn(0x7f48480622c0 legacy=0x7f48480b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.857+0000 7f484dcdc700 1 -- 172.21.15.187:0/1087372772 >> v1:172.21.15.5:6789/0 conn(0x7f4848066f70 legacy=0x7f48480b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.857+0000 7f484dcdc700 1 -- 172.21.15.187:0/1087372772 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48480b1840 con 0x7f48481484d0 2020-11-20T12:05:20.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.858+0000 7f484dcdc700 1 --2- 172.21.15.187:0/1087372772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48481484d0 0x7f4848061dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f484400b9a0 tx=0x7f4844006df0).ready entity=mon.1 client_cookie=a2cafedcef1b3569 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.858+0000 7f484ccda700 1 -- 172.21.15.187:0/1087372772 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48440097b0 con 0x7f48481484d0 2020-11-20T12:05:20.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.858+0000 7f484ccda700 1 -- 172.21.15.187:0/1087372772 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4844009910 con 0x7f48481484d0 2020-11-20T12:05:20.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.858+0000 7f484ccda700 1 -- 172.21.15.187:0/1087372772 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844009c90 con 0x7f48481484d0 2020-11-20T12:05:20.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.859+0000 7f484ecde700 1 -- 172.21.15.187:0/1087372772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48481484d0 msgr2=0x7f4848061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.859+0000 7f484ecde700 1 --2- 172.21.15.187:0/1087372772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48481484d0 0x7f4848061dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f484400b9a0 tx=0x7f4844006df0).stop 2020-11-20T12:05:20.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.859+0000 7f484ecde700 1 -- 172.21.15.187:0/1087372772 shutdown_connections 2020-11-20T12:05:20.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.859+0000 7f484ecde700 1 --2- 172.21.15.187:0/1087372772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48481484d0 0x7f4848061dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.859+0000 7f484ecde700 1 -- 172.21.15.187:0/1087372772 >> 172.21.15.187:0/1087372772 conn(0x7f48480bab30 msgr2=0x7f48480baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.859+0000 7f484ecde700 1 -- 172.21.15.187:0/1087372772 shutdown_connections 2020-11-20T12:05:20.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.859+0000 7f484ecde700 1 -- 172.21.15.187:0/1087372772 wait complete. 2020-11-20T12:05:20.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.860+0000 7f484ecde700 1 Processor -- start 2020-11-20T12:05:20.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.861+0000 7f484ecde700 1 -- start start 2020-11-20T12:05:20.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.861+0000 7f484ecde700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48480622c0 0x7f484809d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.861+0000 7f484ecde700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4848066f70 0x7f4848158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.861+0000 7f484ecde700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48480b9520 0x7f48480b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.861+0000 7f484ecde700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484809bd00 con 0x7f4848066f70 2020-11-20T12:05:20.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.861+0000 7f484ecde700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4848158c40 con 0x7f48480b9520 2020-11-20T12:05:20.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.861+0000 7f484ecde700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4848158ee0 con 0x7f48480622c0 2020-11-20T12:05:20.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48480b9520 0x7f48480b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48480b9520 0x7f48480b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40964/0 (socket says 172.21.15.187:40964) 2020-11-20T12:05:20.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 -- 172.21.15.187:0/1954848397 learned_addr learned my addr 172.21.15.187:0/1954848397 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:20.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484d4db700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4848066f70 0x7f4848158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484dcdc700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48480622c0 0x7f484809d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 -- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48480622c0 msgr2=0x7f484809d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48480622c0 0x7f484809d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 -- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4848066f70 msgr2=0x7f4848158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4848066f70 0x7f4848158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 -- 172.21.15.187:0/1954848397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848159180 con 0x7f48480b9520 2020-11-20T12:05:20.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484d4db700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4848066f70 0x7f4848158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:20.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.862+0000 7f484e4dd700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48480b9520 0x7f48480b4170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f484000ca90 tx=0x7f4840007f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.863+0000 7f4839ffb700 1 -- 172.21.15.187:0/1954848397 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000a7c0 con 0x7f48480b9520 2020-11-20T12:05:20.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.863+0000 7f4839ffb700 1 -- 172.21.15.187:0/1954848397 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484000a920 con 0x7f48480b9520 2020-11-20T12:05:20.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.863+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48480b9980 con 0x7f48480b9520 2020-11-20T12:05:20.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.863+0000 7f4839ffb700 1 -- 172.21.15.187:0/1954848397 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000aca0 con 0x7f48480b9520 2020-11-20T12:05:20.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.863+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48480b4820 con 0x7f48480b9520 2020-11-20T12:05:20.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.864+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4848159310 con 0x7f48480b9520 2020-11-20T12:05:20.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.865+0000 7f4839ffb700 1 -- 172.21.15.187:0/1954848397 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4840019030 con 0x7f48480b9520 2020-11-20T12:05:20.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.865+0000 7f4839ffb700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4820056d20 0x7f48200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.865+0000 7f4839ffb700 1 -- 172.21.15.187:0/1954848397 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f484006f870 con 0x7f48480b9520 2020-11-20T12:05:20.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.866+0000 7f484dcdc700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4820056d20 0x7f48200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.866+0000 7f484dcdc700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4820056d20 0x7f48200591e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f4848066540 tx=0x7f4844015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:20.869+0000 7f4839ffb700 1 -- 172.21.15.187:0/1954848397 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4840044d10 con 0x7f48480b9520 2020-11-20T12:05:21.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.054+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4848061a20 con 0x7f48480b9520 2020-11-20T12:05:21.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.055+0000 7f4839ffb700 1 -- 172.21.15.187:0/1954848397 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4840048330 con 0x7f48480b9520 2020-11-20T12:05:21.059 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:21.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4820056d20 msgr2=0x7f48200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4820056d20 0x7f48200591e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f4848066540 tx=0x7f4844015000).stop 2020-11-20T12:05:21.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48480b9520 msgr2=0x7f48480b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48480b9520 0x7f48480b4170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f484000ca90 tx=0x7f4840007f70).stop 2020-11-20T12:05:21.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 shutdown_connections 2020-11-20T12:05:21.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4820056d20 0x7f48200591e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48480b9520 0x7f48480b4170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48480622c0 0x7f484809d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 --2- 172.21.15.187:0/1954848397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4848066f70 0x7f4848158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 >> 172.21.15.187:0/1954848397 conn(0x7f48480bab30 msgr2=0x7f48480b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:21.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 shutdown_connections 2020-11-20T12:05:21.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.057+0000 7f484ecde700 1 -- 172.21.15.187:0/1954848397 wait complete. 2020-11-20T12:05:21.086 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:21.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.326+0000 7f52a3fff700 1 Processor -- start 2020-11-20T12:05:21.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.326+0000 7f52a3fff700 1 -- start start 2020-11-20T12:05:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.328+0000 7f52a3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40bf800 0x7f52a414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.328+0000 7f52a3fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f52a40bfbe0 0x7f52a41480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.328+0000 7f52a3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a414d110 con 0x7f52a409f560 2020-11-20T12:05:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.328+0000 7f52a3fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a414cbd0 con 0x7f52a40bfbe0 2020-11-20T12:05:21.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.328+0000 7f52a3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a414ce70 con 0x7f52a40bf800 2020-11-20T12:05:21.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.329+0000 7f52a2ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40bf800 0x7f52a414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.329+0000 7f52a27fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f52a40bfbe0 0x7f52a41480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.329+0000 7f52a2ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40bf800 0x7f52a414c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:41856/0 (socket says 172.21.15.187:41856) 2020-11-20T12:05:21.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.329+0000 7f52a2ffd700 1 -- 172.21.15.187:0/1089132895 learned_addr learned my addr 172.21.15.187:0/1089132895 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:21.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.329+0000 7f52a2ffd700 1 -- 172.21.15.187:0/1089132895 >> v2:172.21.15.5:3300/0 conn(0x7f52a40bfbe0 msgr2=0x7f52a41480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:21.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.329+0000 7f52a2ffd700 1 --2- 172.21.15.187:0/1089132895 >> v2:172.21.15.5:3300/0 conn(0x7f52a40bfbe0 0x7f52a41480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.329+0000 7f52a2ffd700 1 -- 172.21.15.187:0/1089132895 >> v1:172.21.15.5:6789/0 conn(0x7f52a409f560 legacy=0x7f52a4060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.330+0000 7f52a2ffd700 1 -- 172.21.15.187:0/1089132895 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a40a4cc0 con 0x7f52a40bf800 2020-11-20T12:05:21.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.330+0000 7f52a2ffd700 1 --2- 172.21.15.187:0/1089132895 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40bf800 0x7f52a414c2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5294006fd0 tx=0x7f5294006eb0).ready entity=mon.2 client_cookie=269e814e461618 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:21.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.330+0000 7f52a1ffb700 1 -- 172.21.15.187:0/1089132895 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52940098a0 con 0x7f52a40bf800 2020-11-20T12:05:21.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.330+0000 7f52a1ffb700 1 -- 172.21.15.187:0/1089132895 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5294009a00 con 0x7f52a40bf800 2020-11-20T12:05:21.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.330+0000 7f52a1ffb700 1 -- 172.21.15.187:0/1089132895 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5294009d80 con 0x7f52a40bf800 2020-11-20T12:05:21.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.331+0000 7f52a3fff700 1 -- 172.21.15.187:0/1089132895 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40bf800 msgr2=0x7f52a414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.331+0000 7f52a3fff700 1 --2- 172.21.15.187:0/1089132895 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40bf800 0x7f52a414c2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5294006fd0 tx=0x7f5294006eb0).stop 2020-11-20T12:05:21.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.331+0000 7f52a3fff700 1 -- 172.21.15.187:0/1089132895 shutdown_connections 2020-11-20T12:05:21.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.331+0000 7f52a3fff700 1 --2- 172.21.15.187:0/1089132895 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40bf800 0x7f52a414c2c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.331+0000 7f52a3fff700 1 --2- 172.21.15.187:0/1089132895 >> v2:172.21.15.5:3300/0 conn(0x7f52a40bfbe0 0x7f52a41480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.331+0000 7f52a3fff700 1 -- 172.21.15.187:0/1089132895 >> 172.21.15.187:0/1089132895 conn(0x7f52a4068370 msgr2=0x7f52a40bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:21.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.331+0000 7f52a3fff700 1 -- 172.21.15.187:0/1089132895 shutdown_connections 2020-11-20T12:05:21.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.331+0000 7f52a3fff700 1 -- 172.21.15.187:0/1089132895 wait complete. 2020-11-20T12:05:21.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.333+0000 7f52a3fff700 1 Processor -- start 2020-11-20T12:05:21.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.333+0000 7f52a3fff700 1 -- start start 2020-11-20T12:05:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.334+0000 7f52a3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52a409f560 0x7f52a4141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.334+0000 7f52a3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52a40bf800 0x7f52a40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.334+0000 7f52a3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40b4380 0x7f52a40b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.334+0000 7f52a3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40b4dc0 con 0x7f52a409f560 2020-11-20T12:05:21.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.334+0000 7f52a3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4149ca0 con 0x7f52a40bf800 2020-11-20T12:05:21.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.334+0000 7f52a3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40b6d30 con 0x7f52a40b4380 2020-11-20T12:05:21.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a27fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52a40bf800 0x7f52a40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a2ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52a409f560 0x7f52a4141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a37fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40b4380 0x7f52a40b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a2ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52a409f560 0x7f52a4141420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:33238/0 (socket says 172.21.15.187:33238) 2020-11-20T12:05:21.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a27fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52a40bf800 0x7f52a40b61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40978/0 (socket says 172.21.15.187:40978) 2020-11-20T12:05:21.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a2ffd700 1 -- 172.21.15.187:0/2104602662 learned_addr learned my addr 172.21.15.187:0/2104602662 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:21.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a27fc700 1 -- 172.21.15.187:0/2104602662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40b4380 msgr2=0x7f52a40b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a27fc700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40b4380 0x7f52a40b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a27fc700 1 -- 172.21.15.187:0/2104602662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52a409f560 msgr2=0x7f52a4141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a27fc700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52a409f560 0x7f52a4141420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.335+0000 7f52a27fc700 1 -- 172.21.15.187:0/2104602662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a4141910 con 0x7f52a40bf800 2020-11-20T12:05:21.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.336+0000 7f52a27fc700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52a40bf800 0x7f52a40b61c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f52a40bcaa0 tx=0x7f5298006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:21.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.336+0000 7f528affd700 1 -- 172.21.15.187:0/2104602662 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529800b8a0 con 0x7f52a40bf800 2020-11-20T12:05:21.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.336+0000 7f528affd700 1 -- 172.21.15.187:0/2104602662 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529800ba00 con 0x7f52a40bf800 2020-11-20T12:05:21.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.336+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a4158680 con 0x7f52a40bf800 2020-11-20T12:05:21.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.336+0000 7f528affd700 1 -- 172.21.15.187:0/2104602662 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529800bd80 con 0x7f52a40bf800 2020-11-20T12:05:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.336+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a414b7d0 con 0x7f52a40bf800 2020-11-20T12:05:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.338+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a40b6ba0 con 0x7f52a40bf800 2020-11-20T12:05:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.338+0000 7f528affd700 1 -- 172.21.15.187:0/2104602662 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5298012600 con 0x7f52a40bf800 2020-11-20T12:05:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.339+0000 7f528affd700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5280056d50 0x7f5280059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.339+0000 7f528affd700 1 -- 172.21.15.187:0/2104602662 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f52980700f0 con 0x7f52a40bf800 2020-11-20T12:05:21.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.339+0000 7f52a2ffd700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5280056d50 0x7f5280059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.340+0000 7f52a2ffd700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5280056d50 0x7f5280059210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5294002fc0 tx=0x7f5294015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:21.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.343+0000 7f528affd700 1 -- 172.21.15.187:0/2104602662 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f52980454e0 con 0x7f52a40bf800 2020-11-20T12:05:21.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.529+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f52a4080050 con 0x7f52a40bf800 2020-11-20T12:05:21.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.530+0000 7f528affd700 1 -- 172.21.15.187:0/2104602662 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5298048b00 con 0x7f52a40bf800 2020-11-20T12:05:21.534 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:21.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5280056d50 msgr2=0x7f5280059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5280056d50 0x7f5280059210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5294002fc0 tx=0x7f5294015000).stop 2020-11-20T12:05:21.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52a40bf800 msgr2=0x7f52a40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52a40bf800 0x7f52a40b61c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f52a40bcaa0 tx=0x7f5298006df0).stop 2020-11-20T12:05:21.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 shutdown_connections 2020-11-20T12:05:21.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5280056d50 0x7f5280059210 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52a409f560 0x7f52a4141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52a40b4380 0x7f52a40b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 --2- 172.21.15.187:0/2104602662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52a40bf800 0x7f52a40b61c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.532+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 >> 172.21.15.187:0/2104602662 conn(0x7f52a4068370 msgr2=0x7f52a40a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:21.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.533+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 shutdown_connections 2020-11-20T12:05:21.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.533+0000 7f52a3fff700 1 -- 172.21.15.187:0/2104602662 wait complete. 2020-11-20T12:05:21.559 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:21.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.808+0000 7fed09906700 1 Processor -- start 2020-11-20T12:05:21.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.808+0000 7fed09906700 1 -- start start 2020-11-20T12:05:21.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.809+0000 7fed09906700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fed04065190 0x7fed04065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.809+0000 7fed09906700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed040b13e0 con 0x7fed0409ca90 2020-11-20T12:05:21.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.809+0000 7fed09906700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed04062c10 con 0x7fed04065f90 2020-11-20T12:05:21.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.809+0000 7fed09906700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0414ab30 con 0x7fed04065190 2020-11-20T12:05:21.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.810+0000 7fed08904700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fed04065190 0x7fed04065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.810+0000 7fecfbfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fed04065f90 0x7fed04143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37588/0 (socket says 172.21.15.187:37588) 2020-11-20T12:05:21.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.810+0000 7fecfbfff700 1 -- 172.21.15.187:0/2335233810 learned_addr learned my addr 172.21.15.187:0/2335233810 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:21.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.810+0000 7fecfb7fe700 1 -- 172.21.15.187:0/2335233810 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213308241 0 0) 0x7fed04062c10 con 0x7fed04065f90 2020-11-20T12:05:21.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.810+0000 7fecfb7fe700 1 -- 172.21.15.187:0/2335233810 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf0003240 con 0x7fed04065f90 2020-11-20T12:05:21.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.811+0000 7fed08904700 1 -- 172.21.15.187:0/2335233810 >> v1:172.21.15.187:6789/0 conn(0x7fed04065f90 legacy=0x7fed04143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.811+0000 7fed08904700 1 -- 172.21.15.187:0/2335233810 >> v1:172.21.15.5:6789/0 conn(0x7fed0409ca90 legacy=0x7fed0409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.811+0000 7fed08904700 1 -- 172.21.15.187:0/2335233810 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed040b1640 con 0x7fed04065190 2020-11-20T12:05:21.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.811+0000 7fed08904700 1 --2- 172.21.15.187:0/2335233810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065190 0x7fed04065570 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fed0414ac70 tx=0x7fed000091d0).ready entity=mon.1 client_cookie=bc438543f8b91ff5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:21.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.811+0000 7fecfb7fe700 1 -- 172.21.15.187:0/2335233810 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000bad0 con 0x7fed04065190 2020-11-20T12:05:21.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.811+0000 7fecfb7fe700 1 -- 172.21.15.187:0/2335233810 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0000bc30 con 0x7fed04065190 2020-11-20T12:05:21.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.811+0000 7fecfb7fe700 1 -- 172.21.15.187:0/2335233810 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00011400 con 0x7fed04065190 2020-11-20T12:05:21.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.812+0000 7fed09906700 1 -- 172.21.15.187:0/2335233810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065190 msgr2=0x7fed04065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.812+0000 7fed09906700 1 --2- 172.21.15.187:0/2335233810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065190 0x7fed04065570 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fed0414ac70 tx=0x7fed000091d0).stop 2020-11-20T12:05:21.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.812+0000 7fed09906700 1 -- 172.21.15.187:0/2335233810 shutdown_connections 2020-11-20T12:05:21.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.812+0000 7fed09906700 1 --2- 172.21.15.187:0/2335233810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065190 0x7fed04065570 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.812+0000 7fed09906700 1 -- 172.21.15.187:0/2335233810 >> 172.21.15.187:0/2335233810 conn(0x7fed040bab30 msgr2=0x7fed040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:21.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.812+0000 7fed09906700 1 -- 172.21.15.187:0/2335233810 shutdown_connections 2020-11-20T12:05:21.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.812+0000 7fed09906700 1 -- 172.21.15.187:0/2335233810 wait complete. 2020-11-20T12:05:21.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.814+0000 7fed09906700 1 Processor -- start 2020-11-20T12:05:21.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.814+0000 7fed09906700 1 -- start start 2020-11-20T12:05:21.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fed09906700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed04065190 0x7fed0415c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fed09906700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065f90 0x7fed04158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fed09906700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0409ca90 0x7fed04144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fed09906700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed040a0420 con 0x7fed0409ca90 2020-11-20T12:05:21.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fed09906700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed041585f0 con 0x7fed04065f90 2020-11-20T12:05:21.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fed09906700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed04158890 con 0x7fed04065190 2020-11-20T12:05:21.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fecfbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065f90 0x7fed04158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fecfbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065f90 0x7fed04158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:40990/0 (socket says 172.21.15.187:40990) 2020-11-20T12:05:21.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.815+0000 7fecfbfff700 1 -- 172.21.15.187:0/1776635762 learned_addr learned my addr 172.21.15.187:0/1776635762 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:21.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fed08904700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed04065190 0x7fed0415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fed09105700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0409ca90 0x7fed04144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fecfbfff700 1 -- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed04065190 msgr2=0x7fed0415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fecfbfff700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed04065190 0x7fed0415c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fecfbfff700 1 -- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0409ca90 msgr2=0x7fed04144790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:21.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fecfbfff700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0409ca90 0x7fed04144790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:21.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fecfbfff700 1 -- 172.21.15.187:0/1776635762 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed04068370 con 0x7fed04065f90 2020-11-20T12:05:21.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fed09105700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0409ca90 0x7fed04144790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:21.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fecfbfff700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065f90 0x7fed04158d30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fecf400c710 tx=0x7fecf4007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:21.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fecf8ff9700 1 -- 172.21.15.187:0/1776635762 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4007320 con 0x7fed04065f90 2020-11-20T12:05:21.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.816+0000 7fecf8ff9700 1 -- 172.21.15.187:0/1776635762 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf4007480 con 0x7fed04065f90 2020-11-20T12:05:21.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.817+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed04159220 con 0x7fed04065f90 2020-11-20T12:05:21.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.817+0000 7fecf8ff9700 1 -- 172.21.15.187:0/1776635762 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4005800 con 0x7fed04065f90 2020-11-20T12:05:21.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.817+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed04144e40 con 0x7fed04065f90 2020-11-20T12:05:21.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.818+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed04158a50 con 0x7fed04065f90 2020-11-20T12:05:21.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.818+0000 7fecf8ff9700 1 -- 172.21.15.187:0/1776635762 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fecf4010030 con 0x7fed04065f90 2020-11-20T12:05:21.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.819+0000 7fecf8ff9700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecdc056d20 0x7fecdc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:21.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.819+0000 7fecf8ff9700 1 -- 172.21.15.187:0/1776635762 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fecf406fc20 con 0x7fed04065f90 2020-11-20T12:05:21.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.819+0000 7fed08904700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecdc056d20 0x7fecdc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:21.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.820+0000 7fed08904700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecdc056d20 0x7fecdc0591e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fed04146a40 tx=0x7fed00016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:21.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:21.823+0000 7fecf8ff9700 1 -- 172.21.15.187:0/1776635762 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fecf4045070 con 0x7fed04065f90 2020-11-20T12:05:22.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.011+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed04061a20 con 0x7fed04065f90 2020-11-20T12:05:22.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.012+0000 7fecf8ff9700 1 -- 172.21.15.187:0/1776635762 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fecf4048690 con 0x7fed04065f90 2020-11-20T12:05:22.016 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:22.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecdc056d20 msgr2=0x7fecdc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecdc056d20 0x7fecdc0591e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fed04146a40 tx=0x7fed00016000).stop 2020-11-20T12:05:22.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065f90 msgr2=0x7fed04158d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065f90 0x7fed04158d30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fecf400c710 tx=0x7fecf4007aa0).stop 2020-11-20T12:05:22.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 shutdown_connections 2020-11-20T12:05:22.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecdc056d20 0x7fecdc0591e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0409ca90 0x7fed04144790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed04065190 0x7fed0415c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 --2- 172.21.15.187:0/1776635762 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed04065f90 0x7fed04158d30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.014+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 >> 172.21.15.187:0/1776635762 conn(0x7fed040bab30 msgr2=0x7fed0409aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:22.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.015+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 shutdown_connections 2020-11-20T12:05:22.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.015+0000 7fed09906700 1 -- 172.21.15.187:0/1776635762 wait complete. 2020-11-20T12:05:22.043 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:05:22.043 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:22.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.282+0000 7f6e6ca75700 1 Processor -- start 2020-11-20T12:05:22.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.282+0000 7f6e6ca75700 1 -- start start 2020-11-20T12:05:22.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.283+0000 7f6e6ca75700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e6809f560 0x7f6e68153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:22.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.283+0000 7f6e6ca75700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6814d110 con 0x7f6e680bf800 2020-11-20T12:05:22.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.284+0000 7f6e6ca75700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6814cbd0 con 0x7f6e680bfbe0 2020-11-20T12:05:22.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.284+0000 7f6e6ca75700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6814c9d0 con 0x7f6e6809f560 2020-11-20T12:05:22.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.284+0000 7f6e5ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6e680bfbe0 0x7f6e681480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37604/0 (socket says 172.21.15.187:37604) 2020-11-20T12:05:22.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.284+0000 7f6e5ffff700 1 -- 172.21.15.187:0/1592071135 learned_addr learned my addr 172.21.15.187:0/1592071135 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:22.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.284+0000 7f6e67fff700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e6809f560 0x7f6e68153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:22.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228972576 0 0) 0x7f6e6814cbd0 con 0x7f6e680bfbe0 2020-11-20T12:05:22.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48003240 con 0x7f6e680bfbe0 2020-11-20T12:05:22.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3384862538 0 0) 0x7f6e48003240 con 0x7f6e680bfbe0 2020-11-20T12:05:22.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e6809f560 msgr2=0x7f6e68153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:22.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e6809f560 0x7f6e68153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 >> v1:172.21.15.5:6789/0 conn(0x7f6e680bf800 legacy=0x7f6e6814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6814ce70 con 0x7f6e680bfbe0 2020-11-20T12:05:22.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6e500014e0 con 0x7f6e680bfbe0 2020-11-20T12:05:22.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 >> v1:172.21.15.187:6789/0 conn(0x7f6e680bfbe0 legacy=0x7f6e681480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e48003730 0x7f6e48003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:22.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e6809f560 0x7f6e480044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:22.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e480049d0 0x7f6e48004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:22.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6814c9d0 con 0x7f6e480049d0 2020-11-20T12:05:22.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e480041d0 con 0x7f6e48003730 2020-11-20T12:05:22.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.285+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4800b730 con 0x7f6e6809f560 2020-11-20T12:05:22.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e677fe700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e48003730 0x7f6e48003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:22.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e5ffff700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e6809f560 0x7f6e480044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:22.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e67fff700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e480049d0 0x7f6e48004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:22.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e677fe700 1 -- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e6809f560 msgr2=0x7f6e480044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:22.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e677fe700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e6809f560 0x7f6e480044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e677fe700 1 -- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e480049d0 msgr2=0x7f6e48004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:22.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e677fe700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e480049d0 0x7f6e48004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e677fe700 1 -- 172.21.15.187:0/1592071135 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e48003240 con 0x7f6e48003730 2020-11-20T12:05:22.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.286+0000 7f6e677fe700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e48003730 0x7f6e48003b50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6e58001950 tx=0x7f6e58007b90).ready entity=mon.1 client_cookie=1f75e4d6ec2e8d12 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:22.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.287+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58007020 con 0x7f6e48003730 2020-11-20T12:05:22.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.287+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e58007180 con 0x7f6e48003730 2020-11-20T12:05:22.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.287+0000 7f6e66ffd700 1 -- 172.21.15.187:0/1592071135 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5800c7f0 con 0x7f6e48003730 2020-11-20T12:05:22.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.287+0000 7f6e6ca75700 1 -- 172.21.15.187:0/1592071135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e48003730 msgr2=0x7f6e48003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.287+0000 7f6e6ca75700 1 --2- 172.21.15.187:0/1592071135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e48003730 0x7f6e48003b50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6e58001950 tx=0x7f6e58007b90).stop 2020-11-20T12:05:22.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.287+0000 7f6e67fff700 1 -- 172.21.15.187:0/1592071135 reap_dead start 2020-11-20T12:05:22.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.287+0000 7f6e6ca75700 1 -- 172.21.15.187:0/1592071135 shutdown_connections 2020-11-20T12:05:22.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.287+0000 7f6e6ca75700 1 -- 172.21.15.187:0/1592071135 >> 172.21.15.187:0/1592071135 conn(0x7f6e68068370 msgr2=0x7f6e680bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:22.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.288+0000 7f6e6ca75700 1 -- 172.21.15.187:0/1592071135 shutdown_connections 2020-11-20T12:05:22.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.288+0000 7f6e6ca75700 1 -- 172.21.15.187:0/1592071135 wait complete. 2020-11-20T12:05:22.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.289+0000 7f6e6ca75700 1 Processor -- start 2020-11-20T12:05:22.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.289+0000 7f6e6ca75700 1 -- start start 2020-11-20T12:05:22.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.290+0000 7f6e6ca75700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e48003730 0x7f6e680bb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:22.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.290+0000 7f6e6ca75700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e680b61f0 0x7f6e681586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:22.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.290+0000 7f6e6ca75700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e68158ba0 0x7f6e681408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:22.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.290+0000 7f6e6ca75700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6814a7a0 con 0x7f6e68158ba0 2020-11-20T12:05:22.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.290+0000 7f6e6ca75700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6814aa40 con 0x7f6e680b61f0 2020-11-20T12:05:22.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.290+0000 7f6e6ca75700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6814ace0 con 0x7f6e48003730 2020-11-20T12:05:22.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.290+0000 7f6e5ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e680b61f0 0x7f6e681586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:22.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e5ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e680b61f0 0x7f6e681586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41010/0 (socket says 172.21.15.187:41010) 2020-11-20T12:05:22.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e5ffff700 1 -- 172.21.15.187:0/2878943923 learned_addr learned my addr 172.21.15.187:0/2878943923 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:22.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e677fe700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e48003730 0x7f6e680bb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:22.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e67fff700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e68158ba0 0x7f6e681408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:22.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e5ffff700 1 -- 172.21.15.187:0/2878943923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e48003730 msgr2=0x7f6e680bb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e5ffff700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e48003730 0x7f6e680bb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e5ffff700 1 -- 172.21.15.187:0/2878943923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e68158ba0 msgr2=0x7f6e681408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e5ffff700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e68158ba0 0x7f6e681408f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e5ffff700 1 -- 172.21.15.187:0/2878943923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e680bbdb0 con 0x7f6e680b61f0 2020-11-20T12:05:22.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.291+0000 7f6e5ffff700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e680b61f0 0x7f6e681586b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f6e5000eb40 tx=0x7f6e6814cd60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:22.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.292+0000 7f6e5f7fe700 1 -- 172.21.15.187:0/2878943923 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e500014b0 con 0x7f6e680b61f0 2020-11-20T12:05:22.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.292+0000 7f6e5f7fe700 1 -- 172.21.15.187:0/2878943923 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5000c940 con 0x7f6e680b61f0 2020-11-20T12:05:22.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.292+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e680b5910 con 0x7f6e680b61f0 2020-11-20T12:05:22.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.292+0000 7f6e5f7fe700 1 -- 172.21.15.187:0/2878943923 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5000ccc0 con 0x7f6e680b61f0 2020-11-20T12:05:22.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.292+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e680b6830 con 0x7f6e680b61f0 2020-11-20T12:05:22.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.293+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e680bc850 con 0x7f6e680b61f0 2020-11-20T12:05:22.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.293+0000 7f6e5f7fe700 1 -- 172.21.15.187:0/2878943923 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6e50014640 con 0x7f6e680b61f0 2020-11-20T12:05:22.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.294+0000 7f6e5f7fe700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e44056d50 0x7f6e44059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:22.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.294+0000 7f6e5f7fe700 1 -- 172.21.15.187:0/2878943923 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6e5006fbb0 con 0x7f6e680b61f0 2020-11-20T12:05:22.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.294+0000 7f6e677fe700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e44056d50 0x7f6e44059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:22.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.295+0000 7f6e677fe700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e44056d50 0x7f6e44059210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6e58001870 tx=0x7f6e58015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:22.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.298+0000 7f6e5f7fe700 1 -- 172.21.15.187:0/2878943923 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6e50076080 con 0x7f6e680b61f0 2020-11-20T12:05:22.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.542+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e68080050 con 0x7f6e680b61f0 2020-11-20T12:05:22.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.543+0000 7f6e5f7fe700 1 -- 172.21.15.187:0/2878943923 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2130 (secure 0 0 0) 0x7f6e50045000 con 0x7f6e680b61f0 2020-11-20T12:05:22.547 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:22.548 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4211,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6833/2094645874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2094645874},{"type":"v1","addr":"172.21.15.5:6833","nonce":2094645874}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:57.868606+0000","modified":"2020-11-20T12:04:59.990344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3094884769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3094884769},{"type":"v1","addr":"172.21.15.187:6835","nonce":3094884769}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:22.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e44056d50 msgr2=0x7f6e44059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e44056d50 0x7f6e44059210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6e58001870 tx=0x7f6e58015000).stop 2020-11-20T12:05:22.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e680b61f0 msgr2=0x7f6e681586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:22.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e680b61f0 0x7f6e681586b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f6e5000eb40 tx=0x7f6e6814cd60).stop 2020-11-20T12:05:22.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 shutdown_connections 2020-11-20T12:05:22.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e44056d50 0x7f6e44059210 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e68158ba0 0x7f6e681408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e48003730 0x7f6e680bb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 --2- 172.21.15.187:0/2878943923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e680b61f0 0x7f6e681586b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:22.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.545+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 >> 172.21.15.187:0/2878943923 conn(0x7f6e68068370 msgr2=0x7f6e68142ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:22.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.546+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 shutdown_connections 2020-11-20T12:05:22.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:22.546+0000 7f6e6ca75700 1 -- 172.21.15.187:0/2878943923 wait complete. 2020-11-20T12:05:22.553 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 6 2020-11-20T12:05:22.773 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:04:57.868606+0000', 'modified': '2020-11-20T12:04:59.990344+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4264': {'gid': 4264, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3094884769', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3094884769}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3094884769}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:05:22.773 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:05:22.774 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:05:23.053 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":4264,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":25.154061205000001,"mdsmap_epoch":6,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":29.542888741999999} 2020-11-20T12:05:23.054 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 4264, "mdsmap_epoch": 6, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 25.154061205, "state": "up:active", "uptime": 29.542888742, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:05:23.055 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:05:23.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:23.055 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:05:23.111 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:05:23.112 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:23.286 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:23.287 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:05:23.814 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:23.814 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:05:23.830 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:05:23.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:23.831 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:05:23.931 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:05:23.931 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:05:23.931 INFO:tasks.cephfs.mount:self.client.name = client.1 2020-11-20T12:05:23.932 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2020-11-20T12:05:23.932 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:05:23.932 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2020-11-20T12:05:23.932 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:05:23.933 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2020-11-20T12:05:23.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:23.934 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:05:23.985 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:23.985 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:23.985 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:23.985 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:23.986 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:23.986 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:23.986 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:23.986 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:23.987 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:23.987 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:23.987 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:23.987 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:23.987 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:05:23.988 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:23.988 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:05:23.988 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42714sec preferred_lft 42714sec 2020-11-20T12:05:23.988 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:05:23.988 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:23.989 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:05:23.989 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:23.989 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:05:23.989 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:23.990 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:05:23.990 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:23.990 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:05:23.990 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:05:23.990 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:05:23.991 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:23.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:23.992 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:05:24.092 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:05:24.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:24.096 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:05:24.148 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:05:24.149 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:05:24.149 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:05:24.149 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:05:24.149 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:05:24.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:24.151 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:24.151 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:24.151 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:05:24.152 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:05:24.152 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:05:24.152 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:24.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:24.407 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:05:24.467 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:05:24.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:24.468 INFO:teuthology.orchestra.run.smithi187:> ip netns list-id 2020-11-20T12:05:24.520 INFO:teuthology.orchestra.run.smithi187.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2020-11-20T12:05:24.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:24.522 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:24.522 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:24.523 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:05:24.523 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 1 2020-11-20T12:05:24.523 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:24.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:24.735 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2020-11-20T12:05:24.842 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:24.843 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:24.843 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:24.843 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:24.844 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:24.844 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:24.844 INFO:teuthology.orchestra.run.smithi187.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:05:24.844 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 32:f5:00:f5:8a:76 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2020-11-20T12:05:24.845 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2020-11-20T12:05:24.845 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:24.845 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::30f5:ff:fef5:8a76/64 scope link 2020-11-20T12:05:24.845 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:24.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:24.848 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2020-11-20T12:05:24.966 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:24.966 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:24.966 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:24.966 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:24.967 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:24.967 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:24.967 INFO:teuthology.orchestra.run.smithi187.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:05:24.967 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 32:f5:00:f5:8a:76 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2020-11-20T12:05:24.967 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2020-11-20T12:05:24.968 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:24.968 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::30f5:ff:fef5:8a76/64 scope link 2020-11-20T12:05:24.968 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:24.970 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.2/16 2020-11-20T12:05:24.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:24.971 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:24.972 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:24.972 INFO:teuthology.orchestra.run.smithi187:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.1 2020-11-20T12:05:24.972 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2020-11-20T12:05:24.973 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2020-11-20T12:05:24.973 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2020-11-20T12:05:24.973 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2020-11-20T12:05:24.974 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:25.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:25.357 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:25.358 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:05:25.358 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.1 up 2020-11-20T12:05:25.358 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set dev brx.1 master ceph-brx 2020-11-20T12:05:25.359 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:05:25.601 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:25.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.854+0000 7f5a621b0700 1 Processor -- start 2020-11-20T12:05:25.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.854+0000 7f5a621b0700 1 -- start start 2020-11-20T12:05:25.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.855+0000 7f5a621b0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 0x7f5a5c09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:25.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.855+0000 7f5a621b0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c0b13e0 con 0x7f5a5c065190 2020-11-20T12:05:25.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.856+0000 7f5a621b0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c09d690 con 0x7f5a5c065f90 2020-11-20T12:05:25.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.856+0000 7f5a621b0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c14ab30 con 0x7f5a5c065570 2020-11-20T12:05:25.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.856+0000 7f5a609ad700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5a5c065f90 0x7f5a5c09ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37628/0 (socket says 172.21.15.187:37628) 2020-11-20T12:05:25.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.856+0000 7f5a609ad700 1 -- 172.21.15.187:0/1993655935 learned_addr learned my addr 172.21.15.187:0/1993655935 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:25.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.856+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019755837 0 0) 0x7f5a5c09d690 con 0x7f5a5c065f90 2020-11-20T12:05:25.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a44003240 con 0x7f5a5c065f90 2020-11-20T12:05:25.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a619af700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 0x7f5a5c09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:25.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3747549426 0 0) 0x7f5a44003240 con 0x7f5a5c065f90 2020-11-20T12:05:25.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 msgr2=0x7f5a5c09d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:25.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 0x7f5a5c09d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:25.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 >> v1:172.21.15.5:6789/0 conn(0x7f5a5c065190 legacy=0x7f5a5c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:25.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c062c10 con 0x7f5a5c065f90 2020-11-20T12:05:25.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5a4c0014e0 con 0x7f5a5c065f90 2020-11-20T12:05:25.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a619af700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 0x7f5a5c09d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:25.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 >> v1:172.21.15.187:6789/0 conn(0x7f5a5c065f90 legacy=0x7f5a5c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:25.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a44003730 0x7f5a44003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:25.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 0x7f5a440044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:25.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a440049d0 0x7f5a44004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:25.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c14ab30 con 0x7f5a440049d0 2020-11-20T12:05:25.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a440041e0 con 0x7f5a44003730 2020-11-20T12:05:25.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.857+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4400b730 con 0x7f5a5c065570 2020-11-20T12:05:25.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.858+0000 7f5a611ae700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a44003730 0x7f5a44003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:25.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.858+0000 7f5a609ad700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 0x7f5a440044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:25.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.858+0000 7f5a611ae700 1 -- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 msgr2=0x7f5a440044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:25.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.858+0000 7f5a611ae700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c065570 0x7f5a440044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:25.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.858+0000 7f5a611ae700 1 -- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a440049d0 msgr2=0x7f5a44004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:25.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.858+0000 7f5a611ae700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a440049d0 0x7f5a44004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:25.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.858+0000 7f5a611ae700 1 -- 172.21.15.187:0/1993655935 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a44003240 con 0x7f5a44003730 2020-11-20T12:05:25.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.858+0000 7f5a611ae700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a44003730 0x7f5a44003b50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f5a58001e70 tx=0x7f5a58007f10).ready entity=mon.1 client_cookie=7599a7759f1d4956 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:25.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.859+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58001540 con 0x7f5a44003730 2020-11-20T12:05:25.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.859+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a58003030 con 0x7f5a44003730 2020-11-20T12:05:25.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.859+0000 7f5a4bfff700 1 -- 172.21.15.187:0/1993655935 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5800cdf0 con 0x7f5a44003730 2020-11-20T12:05:25.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.859+0000 7f5a621b0700 1 -- 172.21.15.187:0/1993655935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a44003730 msgr2=0x7f5a44003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:25.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.859+0000 7f5a621b0700 1 --2- 172.21.15.187:0/1993655935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a44003730 0x7f5a44003b50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f5a58001e70 tx=0x7f5a58007f10).stop 2020-11-20T12:05:25.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.859+0000 7f5a619af700 1 -- 172.21.15.187:0/1993655935 reap_dead start 2020-11-20T12:05:25.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.859+0000 7f5a621b0700 1 -- 172.21.15.187:0/1993655935 shutdown_connections 2020-11-20T12:05:25.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.859+0000 7f5a621b0700 1 -- 172.21.15.187:0/1993655935 >> 172.21.15.187:0/1993655935 conn(0x7f5a5c0bab30 msgr2=0x7f5a5c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:25.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.860+0000 7f5a621b0700 1 -- 172.21.15.187:0/1993655935 shutdown_connections 2020-11-20T12:05:25.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.860+0000 7f5a621b0700 1 -- 172.21.15.187:0/1993655935 wait complete. 2020-11-20T12:05:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.861+0000 7f5a621b0700 1 Processor -- start 2020-11-20T12:05:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.861+0000 7f5a621b0700 1 -- start start 2020-11-20T12:05:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.862+0000 7f5a621b0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c1542c0 0x7f5a5c09e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.862+0000 7f5a621b0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c1546a0 0x7f5a5c144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.862+0000 7f5a621b0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c147c90 0x7f5a5c1452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.862+0000 7f5a621b0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c155040 con 0x7f5a5c1542c0 2020-11-20T12:05:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.862+0000 7f5a621b0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c148210 con 0x7f5a5c147c90 2020-11-20T12:05:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.862+0000 7f5a621b0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c1487d0 con 0x7f5a5c1546a0 2020-11-20T12:05:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.862+0000 7f5a619af700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c147c90 0x7f5a5c1452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a619af700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c147c90 0x7f5a5c1452f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41036/0 (socket says 172.21.15.187:41036) 2020-11-20T12:05:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a619af700 1 -- 172.21.15.187:0/3388908069 learned_addr learned my addr 172.21.15.187:0/3388908069 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a611ae700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c1542c0 0x7f5a5c09e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a609ad700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c1546a0 0x7f5a5c144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a619af700 1 -- 172.21.15.187:0/3388908069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c1546a0 msgr2=0x7f5a5c144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a619af700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c1546a0 0x7f5a5c144e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a619af700 1 -- 172.21.15.187:0/3388908069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c1542c0 msgr2=0x7f5a5c09e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a619af700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c1542c0 0x7f5a5c09e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a619af700 1 -- 172.21.15.187:0/3388908069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c09e580 con 0x7f5a5c147c90 2020-11-20T12:05:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a619af700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c147c90 0x7f5a5c1452f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f5a5c0639b0 tx=0x7f5a54008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.863+0000 7f5a497fa700 1 -- 172.21.15.187:0/3388908069 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a54011590 con 0x7f5a5c147c90 2020-11-20T12:05:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.864+0000 7f5a497fa700 1 -- 172.21.15.187:0/3388908069 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a540116f0 con 0x7f5a5c147c90 2020-11-20T12:05:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.864+0000 7f5a497fa700 1 -- 172.21.15.187:0/3388908069 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a54011ab0 con 0x7f5a5c147c90 2020-11-20T12:05:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.864+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c1484b0 con 0x7f5a5c147c90 2020-11-20T12:05:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.864+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c0b4260 con 0x7f5a5c147c90 2020-11-20T12:05:25.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.865+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5c14a300 con 0x7f5a5c147c90 2020-11-20T12:05:25.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.865+0000 7f5a497fa700 1 -- 172.21.15.187:0/3388908069 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5a5401a030 con 0x7f5a5c147c90 2020-11-20T12:05:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.866+0000 7f5a497fa700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d20 0x7f5a340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.866+0000 7f5a497fa700 1 -- 172.21.15.187:0/3388908069 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5a5406f550 con 0x7f5a5c147c90 2020-11-20T12:05:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.866+0000 7f5a611ae700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d20 0x7f5a340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.867+0000 7f5a611ae700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d20 0x7f5a340591e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f5a5c0b6390 tx=0x7f5a58015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:25.870+0000 7f5a497fa700 1 -- 172.21.15.187:0/3388908069 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5a540449a0 con 0x7f5a5c147c90 2020-11-20T12:05:26.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.061+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a5c148990 con 0x7f5a5c147c90 2020-11-20T12:05:26.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.063+0000 7f5a497fa700 1 -- 172.21.15.187:0/3388908069 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f5a54047fc0 con 0x7f5a5c147c90 2020-11-20T12:05:26.068 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:26.068 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:26.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.065+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d20 msgr2=0x7f5a340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.065+0000 7f5a621b0700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d20 0x7f5a340591e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f5a5c0b6390 tx=0x7f5a58015000).stop 2020-11-20T12:05:26.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.065+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c147c90 msgr2=0x7f5a5c1452f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.065+0000 7f5a621b0700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c147c90 0x7f5a5c1452f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f5a5c0639b0 tx=0x7f5a54008040).stop 2020-11-20T12:05:26.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.066+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 shutdown_connections 2020-11-20T12:05:26.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.066+0000 7f5a621b0700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d20 0x7f5a340591e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.066+0000 7f5a621b0700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c1542c0 0x7f5a5c09e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.066+0000 7f5a621b0700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c147c90 0x7f5a5c1452f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.066+0000 7f5a621b0700 1 --2- 172.21.15.187:0/3388908069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c1546a0 0x7f5a5c144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.066+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 >> 172.21.15.187:0/3388908069 conn(0x7f5a5c0bab30 msgr2=0x7f5a5c09ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:26.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.066+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 shutdown_connections 2020-11-20T12:05:26.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.066+0000 7f5a621b0700 1 -- 172.21.15.187:0/3388908069 wait complete. 2020-11-20T12:05:26.095 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:26.095 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:26.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.336+0000 7efe2a871700 1 Processor -- start 2020-11-20T12:05:26.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.337+0000 7efe2a871700 1 -- start start 2020-11-20T12:05:26.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.338+0000 7efe2a871700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efe24065190 0x7efe24065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.338+0000 7efe2a871700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe240b13e0 con 0x7efe24065f90 2020-11-20T12:05:26.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.338+0000 7efe2a871700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe24062c10 con 0x7efe2409ca90 2020-11-20T12:05:26.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.338+0000 7efe2a871700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2414ab30 con 0x7efe24065190 2020-11-20T12:05:26.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.338+0000 7efe2a070700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7efe2409ca90 0x7efe2409cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37648/0 (socket says 172.21.15.187:37648) 2020-11-20T12:05:26.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.338+0000 7efe2a070700 1 -- 172.21.15.187:0/3576457401 learned_addr learned my addr 172.21.15.187:0/3576457401 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:26.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.339+0000 7efe2986f700 1 --2- 172.21.15.187:0/3576457401 >> v2:172.21.15.5:3300/0 conn(0x7efe24065190 0x7efe24065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:26.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.339+0000 7efe2886d700 1 -- 172.21.15.187:0/3576457401 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805438761 0 0) 0x7efe24062c10 con 0x7efe2409ca90 2020-11-20T12:05:26.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.339+0000 7efe2886d700 1 -- 172.21.15.187:0/3576457401 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c003240 con 0x7efe2409ca90 2020-11-20T12:05:26.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.340+0000 7efe2886d700 1 -- 172.21.15.187:0/3576457401 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1514106914 0 0) 0x7efe0c003240 con 0x7efe2409ca90 2020-11-20T12:05:26.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.340+0000 7efe2986f700 1 -- 172.21.15.187:0/3576457401 >> v1:172.21.15.187:6789/0 conn(0x7efe2409ca90 legacy=0x7efe2409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.340+0000 7efe2986f700 1 -- 172.21.15.187:0/3576457401 >> v1:172.21.15.5:6789/0 conn(0x7efe24065f90 legacy=0x7efe24143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.340+0000 7efe2986f700 1 -- 172.21.15.187:0/3576457401 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240b1640 con 0x7efe24065190 2020-11-20T12:05:26.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.340+0000 7efe2986f700 1 --2- 172.21.15.187:0/3576457401 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe24065190 0x7efe24065570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7efe2414ac70 tx=0x7efe20006df0).ready entity=mon.0 client_cookie=9119efcc37371559 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:26.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.341+0000 7efe2886d700 1 -- 172.21.15.187:0/3576457401 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c001860 con 0x7efe24065190 2020-11-20T12:05:26.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.341+0000 7efe2886d700 1 -- 172.21.15.187:0/3576457401 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe20012460 con 0x7efe24065190 2020-11-20T12:05:26.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.341+0000 7efe2886d700 1 -- 172.21.15.187:0/3576457401 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe200127e0 con 0x7efe24065190 2020-11-20T12:05:26.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.342+0000 7efe2a871700 1 -- 172.21.15.187:0/3576457401 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe24065190 msgr2=0x7efe24065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.342+0000 7efe2a871700 1 --2- 172.21.15.187:0/3576457401 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe24065190 0x7efe24065570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7efe2414ac70 tx=0x7efe20006df0).stop 2020-11-20T12:05:26.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.342+0000 7efe2a871700 1 -- 172.21.15.187:0/3576457401 shutdown_connections 2020-11-20T12:05:26.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.342+0000 7efe2a871700 1 --2- 172.21.15.187:0/3576457401 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe24065190 0x7efe24065570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.342+0000 7efe2a871700 1 -- 172.21.15.187:0/3576457401 >> 172.21.15.187:0/3576457401 conn(0x7efe240bab30 msgr2=0x7efe240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:26.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.342+0000 7efe2a871700 1 -- 172.21.15.187:0/3576457401 shutdown_connections 2020-11-20T12:05:26.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.342+0000 7efe2a871700 1 -- 172.21.15.187:0/3576457401 wait complete. 2020-11-20T12:05:26.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.344+0000 7efe2a871700 1 Processor -- start 2020-11-20T12:05:26.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.344+0000 7efe2a871700 1 -- start start 2020-11-20T12:05:26.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.345+0000 7efe2a871700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe24065190 0x7efe24144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.345+0000 7efe2a871700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24065f90 0x7efe2415a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2a871700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe2409ca90 0x7efe240b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2a871700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240a08c0 con 0x7efe24065190 2020-11-20T12:05:26.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2a871700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe24159990 con 0x7efe24065f90 2020-11-20T12:05:26.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2a871700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe24159c30 con 0x7efe2409ca90 2020-11-20T12:05:26.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2906e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24065f90 0x7efe2415a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:26.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2906e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24065f90 0x7efe2415a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41048/0 (socket says 172.21.15.187:41048) 2020-11-20T12:05:26.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2906e700 1 -- 172.21.15.187:0/262009652 learned_addr learned my addr 172.21.15.187:0/262009652 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:26.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2906e700 1 -- 172.21.15.187:0/262009652 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe2409ca90 msgr2=0x7efe240b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2906e700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe2409ca90 0x7efe240b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:26.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2906e700 1 -- 172.21.15.187:0/262009652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe24065190 msgr2=0x7efe24144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2906e700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe24065190 0x7efe24144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:26.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.346+0000 7efe2906e700 1 -- 172.21.15.187:0/262009652 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240b4a20 con 0x7efe24065f90 2020-11-20T12:05:26.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.347+0000 7efe2906e700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24065f90 0x7efe2415a0d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7efe1400c8b0 tx=0x7efe14007bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:26.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.347+0000 7efe11ffb700 1 -- 172.21.15.187:0/262009652 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1400a5c0 con 0x7efe24065f90 2020-11-20T12:05:26.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.347+0000 7efe11ffb700 1 -- 172.21.15.187:0/262009652 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1400a720 con 0x7efe24065f90 2020-11-20T12:05:26.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.347+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2415a5c0 con 0x7efe24065f90 2020-11-20T12:05:26.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.347+0000 7efe11ffb700 1 -- 172.21.15.187:0/262009652 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1400aac0 con 0x7efe24065f90 2020-11-20T12:05:26.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.347+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe240b2690 con 0x7efe24065f90 2020-11-20T12:05:26.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.348+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe24159df0 con 0x7efe24065f90 2020-11-20T12:05:26.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.349+0000 7efe11ffb700 1 -- 172.21.15.187:0/262009652 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efe14010070 con 0x7efe24065f90 2020-11-20T12:05:26.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.350+0000 7efe11ffb700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdfc056d20 0x7efdfc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.350+0000 7efe11ffb700 1 -- 172.21.15.187:0/262009652 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efe1406fa30 con 0x7efe24065f90 2020-11-20T12:05:26.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.350+0000 7efe2986f700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdfc056d20 0x7efdfc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:26.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.351+0000 7efe2986f700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdfc056d20 0x7efdfc0591e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7efe2000b150 tx=0x7efe20014000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:26.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.354+0000 7efe11ffb700 1 -- 172.21.15.187:0/262009652 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efe14044e80 con 0x7efe24065f90 2020-11-20T12:05:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.539+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7efe240b2940 con 0x7efe24065f90 2020-11-20T12:05:26.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.540+0000 7efe11ffb700 1 -- 172.21.15.187:0/262009652 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7efe140484a0 con 0x7efe24065f90 2020-11-20T12:05:26.544 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.542+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdfc056d20 msgr2=0x7efdfc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.542+0000 7efe2a871700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdfc056d20 0x7efdfc0591e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7efe2000b150 tx=0x7efe20014000).stop 2020-11-20T12:05:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.542+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24065f90 msgr2=0x7efe2415a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.542+0000 7efe2a871700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24065f90 0x7efe2415a0d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7efe1400c8b0 tx=0x7efe14007bd0).stop 2020-11-20T12:05:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.543+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 shutdown_connections 2020-11-20T12:05:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.543+0000 7efe2a871700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdfc056d20 0x7efdfc0591e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.543+0000 7efe2a871700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe24065190 0x7efe24144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.543+0000 7efe2a871700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe2409ca90 0x7efe240b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.543+0000 7efe2a871700 1 --2- 172.21.15.187:0/262009652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24065f90 0x7efe2415a0d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.543+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 >> 172.21.15.187:0/262009652 conn(0x7efe240bab30 msgr2=0x7efe240b6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.543+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 shutdown_connections 2020-11-20T12:05:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.543+0000 7efe2a871700 1 -- 172.21.15.187:0/262009652 wait complete. 2020-11-20T12:05:26.571 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:26.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.810+0000 7f4ca841b700 1 Processor -- start 2020-11-20T12:05:26.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.810+0000 7f4ca841b700 1 -- start start 2020-11-20T12:05:26.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.812+0000 7f4ca841b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4ca00b73b0 0x7f4ca0066e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.812+0000 7f4ca841b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca0146010 con 0x7f4ca0067300 2020-11-20T12:05:26.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.812+0000 7f4ca841b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca01462b0 con 0x7f4ca00b6fd0 2020-11-20T12:05:26.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.812+0000 7f4ca841b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca0146550 con 0x7f4ca00b73b0 2020-11-20T12:05:26.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.813+0000 7f4ca61b7700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4ca00b6fd0 0x7f4ca0066700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37664/0 (socket says 172.21.15.187:37664) 2020-11-20T12:05:26.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.813+0000 7f4ca61b7700 1 -- 172.21.15.187:0/3311986354 learned_addr learned my addr 172.21.15.187:0/3311986354 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:26.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.813+0000 7f4ca59b6700 1 --2- 172.21.15.187:0/3311986354 >> v2:172.21.15.5:3300/0 conn(0x7f4ca00b73b0 0x7f4ca0066e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:26.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.813+0000 7f4ca51b5700 1 -- 172.21.15.187:0/3311986354 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069093725 0 0) 0x7f4ca01462b0 con 0x7f4ca00b6fd0 2020-11-20T12:05:26.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.814+0000 7f4ca51b5700 1 -- 172.21.15.187:0/3311986354 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c88003240 con 0x7f4ca00b6fd0 2020-11-20T12:05:26.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.814+0000 7f4ca59b6700 1 -- 172.21.15.187:0/3311986354 >> v1:172.21.15.187:6789/0 conn(0x7f4ca00b6fd0 legacy=0x7f4ca0066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.814+0000 7f4ca51b5700 1 -- 172.21.15.187:0/3311986354 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642243683 0 0) 0x7f4ca0146010 con 0x7f4ca0067300 2020-11-20T12:05:26.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.814+0000 7f4ca59b6700 1 -- 172.21.15.187:0/3311986354 >> v1:172.21.15.5:6789/0 conn(0x7f4ca0067300 legacy=0x7f4ca0144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.814+0000 7f4ca59b6700 1 -- 172.21.15.187:0/3311986354 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca00b6650 con 0x7f4ca00b73b0 2020-11-20T12:05:26.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.814+0000 7f4ca59b6700 1 --2- 172.21.15.187:0/3311986354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 0x7f4ca0066e10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4c9000b9a0 tx=0x7f4c90006df0).ready entity=mon.0 client_cookie=73d976424461f5cd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:26.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.815+0000 7f4ca51b5700 1 -- 172.21.15.187:0/3311986354 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c900097b0 con 0x7f4ca00b73b0 2020-11-20T12:05:26.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.815+0000 7f4ca51b5700 1 -- 172.21.15.187:0/3311986354 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c90009910 con 0x7f4ca00b73b0 2020-11-20T12:05:26.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.815+0000 7f4ca51b5700 1 -- 172.21.15.187:0/3311986354 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90009c90 con 0x7f4ca00b73b0 2020-11-20T12:05:26.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.815+0000 7f4ca841b700 1 -- 172.21.15.187:0/3311986354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 msgr2=0x7f4ca0066e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.815+0000 7f4ca841b700 1 --2- 172.21.15.187:0/3311986354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 0x7f4ca0066e10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4c9000b9a0 tx=0x7f4c90006df0).stop 2020-11-20T12:05:26.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.816+0000 7f4ca841b700 1 -- 172.21.15.187:0/3311986354 shutdown_connections 2020-11-20T12:05:26.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.816+0000 7f4ca841b700 1 --2- 172.21.15.187:0/3311986354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 0x7f4ca0066e10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.816+0000 7f4ca841b700 1 -- 172.21.15.187:0/3311986354 >> 172.21.15.187:0/3311986354 conn(0x7f4ca00bacf0 msgr2=0x7f4ca00bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:26.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.816+0000 7f4ca841b700 1 -- 172.21.15.187:0/3311986354 shutdown_connections 2020-11-20T12:05:26.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.816+0000 7f4ca841b700 1 -- 172.21.15.187:0/3311986354 wait complete. 2020-11-20T12:05:26.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.817+0000 7f4ca841b700 1 Processor -- start 2020-11-20T12:05:26.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.817+0000 7f4ca841b700 1 -- start start 2020-11-20T12:05:26.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.818+0000 7f4ca841b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ca0067300 0x7f4ca01459b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.818+0000 7f4ca841b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ca00b6fd0 0x7f4ca015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.818+0000 7f4ca841b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 0x7f4ca008fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.818+0000 7f4ca841b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca015c9b0 con 0x7f4ca00b73b0 2020-11-20T12:05:26.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.818+0000 7f4ca841b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca015c470 con 0x7f4ca0067300 2020-11-20T12:05:26.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.818+0000 7f4ca841b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca015c710 con 0x7f4ca00b6fd0 2020-11-20T12:05:26.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.818+0000 7f4ca61b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ca0067300 0x7f4ca01459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:26.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca61b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ca0067300 0x7f4ca01459b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41062/0 (socket says 172.21.15.187:41062) 2020-11-20T12:05:26.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca61b7700 1 -- 172.21.15.187:0/3979088374 learned_addr learned my addr 172.21.15.187:0/3979088374 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:26.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca59b6700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ca00b6fd0 0x7f4ca015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:26.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca69b8700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 0x7f4ca008fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:26.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca61b7700 1 -- 172.21.15.187:0/3979088374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ca00b6fd0 msgr2=0x7f4ca015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca61b7700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ca00b6fd0 0x7f4ca015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca61b7700 1 -- 172.21.15.187:0/3979088374 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 msgr2=0x7f4ca008fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:26.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca61b7700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 0x7f4ca008fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:26.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca61b7700 1 -- 172.21.15.187:0/3979088374 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0090f00 con 0x7f4ca0067300 2020-11-20T12:05:26.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.819+0000 7f4ca61b7700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ca0067300 0x7f4ca01459b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f4c9c00c6d0 tx=0x7f4c9c007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:26.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.820+0000 7f4c8e7fc700 1 -- 172.21.15.187:0/3979088374 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0072e0 con 0x7f4ca0067300 2020-11-20T12:05:26.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.820+0000 7f4c8e7fc700 1 -- 172.21.15.187:0/3979088374 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c007440 con 0x7f4ca0067300 2020-11-20T12:05:26.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.820+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca015c100 con 0x7f4ca0067300 2020-11-20T12:05:26.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.820+0000 7f4c8e7fc700 1 -- 172.21.15.187:0/3979088374 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c005820 con 0x7f4ca0067300 2020-11-20T12:05:26.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.820+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca009a520 con 0x7f4ca0067300 2020-11-20T12:05:26.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.821+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca015c290 con 0x7f4ca0067300 2020-11-20T12:05:26.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.821+0000 7f4c8e7fc700 1 -- 172.21.15.187:0/3979088374 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4c9c010070 con 0x7f4ca0067300 2020-11-20T12:05:26.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.822+0000 7f4c8e7fc700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c78056d20 0x7f4c780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:26.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.822+0000 7f4c8e7fc700 1 -- 172.21.15.187:0/3979088374 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4c9c06f5e0 con 0x7f4ca0067300 2020-11-20T12:05:26.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.822+0000 7f4ca59b6700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c78056d20 0x7f4c780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:26.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.823+0000 7f4ca59b6700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c78056d20 0x7f4c780591e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4c90006e30 tx=0x7f4c90015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:26.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:26.826+0000 7f4c8e7fc700 1 -- 172.21.15.187:0/3979088374 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4c9c044a30 con 0x7f4ca0067300 2020-11-20T12:05:27.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.005+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4ca0044f60 con 0x7f4ca0067300 2020-11-20T12:05:27.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.006+0000 7f4c8e7fc700 1 -- 172.21.15.187:0/3979088374 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4c9c048050 con 0x7f4ca0067300 2020-11-20T12:05:27.009 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:27.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c78056d20 msgr2=0x7f4c780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c78056d20 0x7f4c780591e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4c90006e30 tx=0x7f4c90015000).stop 2020-11-20T12:05:27.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ca0067300 msgr2=0x7f4ca01459b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ca0067300 0x7f4ca01459b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f4c9c00c6d0 tx=0x7f4c9c007ac0).stop 2020-11-20T12:05:27.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 shutdown_connections 2020-11-20T12:05:27.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4c78056d20 0x7f4c780591e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca00b73b0 0x7f4ca008fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ca0067300 0x7f4ca01459b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 --2- 172.21.15.187:0/3979088374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ca00b6fd0 0x7f4ca015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 >> 172.21.15.187:0/3979088374 conn(0x7f4ca00bacf0 msgr2=0x7f4ca00b9600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:27.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 shutdown_connections 2020-11-20T12:05:27.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.008+0000 7f4ca841b700 1 -- 172.21.15.187:0/3979088374 wait complete. 2020-11-20T12:05:27.035 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:27.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.271+0000 7f998f59e700 1 Processor -- start 2020-11-20T12:05:27.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.272+0000 7f998f59e700 1 -- start start 2020-11-20T12:05:27.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.273+0000 7f998f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f99900b1380 0x7f9990150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.273+0000 7f998f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99900b1760 0x7f9990065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.273+0000 7f998f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9990150390 con 0x7f9990065680 2020-11-20T12:05:27.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.273+0000 7f998f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9990150630 con 0x7f99900b1380 2020-11-20T12:05:27.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.273+0000 7f998f59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9990068510 con 0x7f99900b1760 2020-11-20T12:05:27.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.273+0000 7f998dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99900b1760 0x7f9990065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.274+0000 7f998dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99900b1760 0x7f9990065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41070/0 (socket says 172.21.15.187:41070) 2020-11-20T12:05:27.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.274+0000 7f998dd9b700 1 -- 172.21.15.187:0/1769928798 learned_addr learned my addr 172.21.15.187:0/1769928798 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:27.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.274+0000 7f998e59c700 1 --2- 172.21.15.187:0/1769928798 >> v2:172.21.15.5:3300/0 conn(0x7f99900b1380 0x7f9990150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.274+0000 7f998dd9b700 1 -- 172.21.15.187:0/1769928798 >> v2:172.21.15.5:3300/0 conn(0x7f99900b1380 msgr2=0x7f9990150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:27.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.274+0000 7f998dd9b700 1 --2- 172.21.15.187:0/1769928798 >> v2:172.21.15.5:3300/0 conn(0x7f99900b1380 0x7f9990150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.274+0000 7f998dd9b700 1 -- 172.21.15.187:0/1769928798 >> v1:172.21.15.5:6789/0 conn(0x7f9990065680 legacy=0x7f9990061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.274+0000 7f998dd9b700 1 -- 172.21.15.187:0/1769928798 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990150fc0 con 0x7f99900b1760 2020-11-20T12:05:27.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.275+0000 7f998dd9b700 1 --2- 172.21.15.187:0/1769928798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990065190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9990063a20 tx=0x7f9984009290).ready entity=mon.1 client_cookie=f892b7a06e640701 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:27.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.275+0000 7f998d59a700 1 -- 172.21.15.187:0/1769928798 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998400bbc0 con 0x7f99900b1760 2020-11-20T12:05:27.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.275+0000 7f998d59a700 1 -- 172.21.15.187:0/1769928798 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998400bd20 con 0x7f99900b1760 2020-11-20T12:05:27.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.275+0000 7f998d59a700 1 -- 172.21.15.187:0/1769928798 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984011610 con 0x7f99900b1760 2020-11-20T12:05:27.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.276+0000 7f998f59e700 1 -- 172.21.15.187:0/1769928798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 msgr2=0x7f9990065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.276+0000 7f998f59e700 1 --2- 172.21.15.187:0/1769928798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990065190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9990063a20 tx=0x7f9984009290).stop 2020-11-20T12:05:27.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.276+0000 7f998f59e700 1 -- 172.21.15.187:0/1769928798 shutdown_connections 2020-11-20T12:05:27.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.276+0000 7f998f59e700 1 --2- 172.21.15.187:0/1769928798 >> v2:172.21.15.5:3300/0 conn(0x7f99900b1380 0x7f9990150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.276+0000 7f998f59e700 1 --2- 172.21.15.187:0/1769928798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990065190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.276+0000 7f998f59e700 1 -- 172.21.15.187:0/1769928798 >> 172.21.15.187:0/1769928798 conn(0x7f99900bab30 msgr2=0x7f99900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:27.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.276+0000 7f998f59e700 1 -- 172.21.15.187:0/1769928798 shutdown_connections 2020-11-20T12:05:27.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.276+0000 7f998f59e700 1 -- 172.21.15.187:0/1769928798 wait complete. 2020-11-20T12:05:27.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.277+0000 7f998f59e700 1 Processor -- start 2020-11-20T12:05:27.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.278+0000 7f998f59e700 1 -- start start 2020-11-20T12:05:27.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.278+0000 7f998f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99900b1380 0x7f99900a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.278+0000 7f998f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99900b7230 0x7f99900b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999009e7d0 con 0x7f99900b1380 2020-11-20T12:05:27.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b37a0 con 0x7f99900b1760 2020-11-20T12:05:27.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b3a40 con 0x7f99900b7230 2020-11-20T12:05:27.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990149b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41076/0 (socket says 172.21.15.187:41076) 2020-11-20T12:05:27.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998dd9b700 1 -- 172.21.15.187:0/1715901559 learned_addr learned my addr 172.21.15.187:0/1715901559 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:27.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998e59c700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99900b1380 0x7f99900a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998dd9b700 1 -- 172.21.15.187:0/1715901559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99900b7230 msgr2=0x7f99900b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998dd9b700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99900b7230 0x7f99900b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:27.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998dd9b700 1 -- 172.21.15.187:0/1715901559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99900b1380 msgr2=0x7f99900a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998dd9b700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99900b1380 0x7f99900a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998dd9b700 1 -- 172.21.15.187:0/1715901559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900b3ce0 con 0x7f99900b1760 2020-11-20T12:05:27.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.279+0000 7f998e59c700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99900b1380 0x7f99900a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:27.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.280+0000 7f998dd9b700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990149b80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9990068650 tx=0x7f9984016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:27.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.280+0000 7f997e7fc700 1 -- 172.21.15.187:0/1715901559 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998401b6a0 con 0x7f99900b1760 2020-11-20T12:05:27.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.280+0000 7f997e7fc700 1 -- 172.21.15.187:0/1715901559 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998401b800 con 0x7f99900b1760 2020-11-20T12:05:27.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.280+0000 7f997e7fc700 1 -- 172.21.15.187:0/1715901559 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998401bb80 con 0x7f99900b1760 2020-11-20T12:05:27.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.280+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999014a070 con 0x7f99900b1760 2020-11-20T12:05:27.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.280+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99900b3400 con 0x7f99900b1760 2020-11-20T12:05:27.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.282+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99900b3e70 con 0x7f99900b1760 2020-11-20T12:05:27.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.282+0000 7f997e7fc700 1 -- 172.21.15.187:0/1715901559 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9984023030 con 0x7f99900b1760 2020-11-20T12:05:27.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.282+0000 7f997e7fc700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9964056d20 0x7f99640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.283+0000 7f997e7fc700 1 -- 172.21.15.187:0/1715901559 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9984071450 con 0x7f99900b1760 2020-11-20T12:05:27.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.283+0000 7f998e59c700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9964056d20 0x7f99640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.283+0000 7f998e59c700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9964056d20 0x7f99640591e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9980009710 tx=0x7f9980008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:27.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.286+0000 7f997e7fc700 1 -- 172.21.15.187:0/1715901559 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f99840469d0 con 0x7f99900b1760 2020-11-20T12:05:27.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.474+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9990061a20 con 0x7f99900b1760 2020-11-20T12:05:27.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.475+0000 7f997e7fc700 1 -- 172.21.15.187:0/1715901559 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9984021020 con 0x7f99900b1760 2020-11-20T12:05:27.478 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:27.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.477+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9964056d20 msgr2=0x7f99640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.477+0000 7f998f59e700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9964056d20 0x7f99640591e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9980009710 tx=0x7f9980008040).stop 2020-11-20T12:05:27.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.477+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 msgr2=0x7f9990149b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.477+0000 7f998f59e700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990149b80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9990068650 tx=0x7f9984016000).stop 2020-11-20T12:05:27.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.478+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 shutdown_connections 2020-11-20T12:05:27.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.478+0000 7f998f59e700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9964056d20 0x7f99640591e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.478+0000 7f998f59e700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99900b1380 0x7f99900a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.478+0000 7f998f59e700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99900b7230 0x7f99900b7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:27.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.478+0000 7f998f59e700 1 --2- 172.21.15.187:0/1715901559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99900b1760 0x7f9990149b80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.478+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 >> 172.21.15.187:0/1715901559 conn(0x7f99900bab30 msgr2=0x7f999014f300 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:27.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.478+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 shutdown_connections 2020-11-20T12:05:27.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.478+0000 7f998f59e700 1 -- 172.21.15.187:0/1715901559 wait complete. 2020-11-20T12:05:27.502 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:27.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.746+0000 7f8e6cb54700 1 Processor -- start 2020-11-20T12:05:27.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.746+0000 7f8e6cb54700 1 -- start start 2020-11-20T12:05:27.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.748+0000 7f8e6cb54700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e680b1380 0x7f8e68150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.748+0000 7f8e6cb54700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8e680b1760 0x7f8e68065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.748+0000 7f8e6cb54700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e68150390 con 0x7f8e68065680 2020-11-20T12:05:27.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.748+0000 7f8e6cb54700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e68150630 con 0x7f8e680b1760 2020-11-20T12:05:27.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.748+0000 7f8e6cb54700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e68068510 con 0x7f8e680b1380 2020-11-20T12:05:27.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.748+0000 7f8e67fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8e68065680 0x7f8e68061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37692/0 (socket says 172.21.15.187:37692) 2020-11-20T12:05:27.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.748+0000 7f8e67fff700 1 -- 172.21.15.187:0/1194644529 learned_addr learned my addr 172.21.15.187:0/1194644529 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:27.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.748+0000 7f8e677fe700 1 --2- 172.21.15.187:0/1194644529 >> v2:172.21.15.187:3300/0 conn(0x7f8e680b1380 0x7f8e68150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.749+0000 7f8e66ffd700 1 --2- 172.21.15.187:0/1194644529 >> v2:172.21.15.5:3300/0 conn(0x7f8e680b1760 0x7f8e68065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.749+0000 7f8e667fc700 1 -- 172.21.15.187:0/1194644529 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820139206 0 0) 0x7f8e68150390 con 0x7f8e68065680 2020-11-20T12:05:27.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.749+0000 7f8e667fc700 1 -- 172.21.15.187:0/1194644529 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e54003240 con 0x7f8e68065680 2020-11-20T12:05:27.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.749+0000 7f8e677fe700 1 -- 172.21.15.187:0/1194644529 >> v2:172.21.15.5:3300/0 conn(0x7f8e680b1760 msgr2=0x7f8e68065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:27.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.749+0000 7f8e677fe700 1 --2- 172.21.15.187:0/1194644529 >> v2:172.21.15.5:3300/0 conn(0x7f8e680b1760 0x7f8e68065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.749+0000 7f8e677fe700 1 -- 172.21.15.187:0/1194644529 >> v1:172.21.15.187:6789/0 conn(0x7f8e68065680 legacy=0x7f8e68061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.749+0000 7f8e677fe700 1 -- 172.21.15.187:0/1194644529 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e68150fc0 con 0x7f8e680b1380 2020-11-20T12:05:27.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.749+0000 7f8e677fe700 1 --2- 172.21.15.187:0/1194644529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e68150ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8e680b85e0 tx=0x7f8e58006df0).ready entity=mon.1 client_cookie=3538e9a10201a56e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:27.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.750+0000 7f8e667fc700 1 -- 172.21.15.187:0/1194644529 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e580097b0 con 0x7f8e680b1380 2020-11-20T12:05:27.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.750+0000 7f8e667fc700 1 -- 172.21.15.187:0/1194644529 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e58009910 con 0x7f8e680b1380 2020-11-20T12:05:27.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.750+0000 7f8e667fc700 1 -- 172.21.15.187:0/1194644529 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58009c90 con 0x7f8e680b1380 2020-11-20T12:05:27.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.750+0000 7f8e6cb54700 1 -- 172.21.15.187:0/1194644529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 msgr2=0x7f8e68150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.750+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/1194644529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e68150ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8e680b85e0 tx=0x7f8e58006df0).stop 2020-11-20T12:05:27.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.751+0000 7f8e6cb54700 1 -- 172.21.15.187:0/1194644529 shutdown_connections 2020-11-20T12:05:27.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.751+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/1194644529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e68150ad0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.751+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/1194644529 >> v2:172.21.15.5:3300/0 conn(0x7f8e680b1760 0x7f8e68065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.751+0000 7f8e6cb54700 1 -- 172.21.15.187:0/1194644529 >> 172.21.15.187:0/1194644529 conn(0x7f8e680bab30 msgr2=0x7f8e680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:27.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.751+0000 7f8e6cb54700 1 -- 172.21.15.187:0/1194644529 shutdown_connections 2020-11-20T12:05:27.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.751+0000 7f8e6cb54700 1 -- 172.21.15.187:0/1194644529 wait complete. 2020-11-20T12:05:27.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.753+0000 7f8e6cb54700 1 Processor -- start 2020-11-20T12:05:27.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.753+0000 7f8e6cb54700 1 -- start start 2020-11-20T12:05:27.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e6cb54700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e680b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e6cb54700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e680b1760 0x7f8e680b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e6cb54700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e6814aa30 0x7f8e6814ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e6cb54700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6814b380 con 0x7f8e6814aa30 2020-11-20T12:05:27.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e6cb54700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6809de60 con 0x7f8e680b1380 2020-11-20T12:05:27.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e6cb54700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6809e100 con 0x7f8e680b1760 2020-11-20T12:05:27.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e677fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e680b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e677fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e680b66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41088/0 (socket says 172.21.15.187:41088) 2020-11-20T12:05:27.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e677fe700 1 -- 172.21.15.187:0/3099099834 learned_addr learned my addr 172.21.15.187:0/3099099834 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:27.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e67fff700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e6814aa30 0x7f8e6814ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e66ffd700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e680b1760 0x7f8e680b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e677fe700 1 -- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e680b1760 msgr2=0x7f8e680b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.755+0000 7f8e677fe700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e680b1760 0x7f8e680b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e677fe700 1 -- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e6814aa30 msgr2=0x7f8e6814ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e677fe700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e6814aa30 0x7f8e6814ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e677fe700 1 -- 172.21.15.187:0/3099099834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6809e3a0 con 0x7f8e680b1380 2020-11-20T12:05:27.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e67fff700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e6814aa30 0x7f8e6814ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:27.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e66ffd700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e680b1760 0x7f8e680b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:27.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e677fe700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e680b66e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f8e5800b150 tx=0x7f8e58015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:27.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e477fe700 1 -- 172.21.15.187:0/3099099834 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58011dd0 con 0x7f8e680b1380 2020-11-20T12:05:27.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e477fe700 1 -- 172.21.15.187:0/3099099834 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5801e460 con 0x7f8e680b1380 2020-11-20T12:05:27.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e477fe700 1 -- 172.21.15.187:0/3099099834 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5801e800 con 0x7f8e680b1380 2020-11-20T12:05:27.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6809e840 con 0x7f8e680b1380 2020-11-20T12:05:27.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.756+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6809dae0 con 0x7f8e680b1380 2020-11-20T12:05:27.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.757+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6809e530 con 0x7f8e680b1380 2020-11-20T12:05:27.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.758+0000 7f8e477fe700 1 -- 172.21.15.187:0/3099099834 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8e58025030 con 0x7f8e680b1380 2020-11-20T12:05:27.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.758+0000 7f8e477fe700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e40056d20 0x7f8e400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:27.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.759+0000 7f8e477fe700 1 -- 172.21.15.187:0/3099099834 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e580718b0 con 0x7f8e680b1380 2020-11-20T12:05:27.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.759+0000 7f8e66ffd700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e40056d20 0x7f8e400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:27.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.759+0000 7f8e66ffd700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e40056d20 0x7f8e400591e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8e680b7c50 tx=0x7f8e5c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:27.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.762+0000 7f8e477fe700 1 -- 172.21.15.187:0/3099099834 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8e58046c80 con 0x7f8e680b1380 2020-11-20T12:05:27.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.947+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8e68061a20 con 0x7f8e680b1380 2020-11-20T12:05:27.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.948+0000 7f8e477fe700 1 -- 172.21.15.187:0/3099099834 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8e5804a2a0 con 0x7f8e680b1380 2020-11-20T12:05:27.952 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:27.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.951+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e40056d20 msgr2=0x7f8e400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.951+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e40056d20 0x7f8e400591e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8e680b7c50 tx=0x7f8e5c008040).stop 2020-11-20T12:05:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.951+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 msgr2=0x7f8e680b66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.951+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e680b66e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f8e5800b150 tx=0x7f8e58015000).stop 2020-11-20T12:05:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.952+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 shutdown_connections 2020-11-20T12:05:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.952+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e40056d20 0x7f8e400591e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.952+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e6814aa30 0x7f8e6814ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.952+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e680b1380 0x7f8e680b66e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.952+0000 7f8e6cb54700 1 --2- 172.21.15.187:0/3099099834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e680b1760 0x7f8e680b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:27.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.952+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 >> 172.21.15.187:0/3099099834 conn(0x7f8e680bab30 msgr2=0x7f8e6814c850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:27.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.952+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 shutdown_connections 2020-11-20T12:05:27.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:27.952+0000 7f8e6cb54700 1 -- 172.21.15.187:0/3099099834 wait complete. 2020-11-20T12:05:27.978 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:05:27.979 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:28.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.212+0000 7fbf4e881700 1 Processor -- start 2020-11-20T12:05:28.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.212+0000 7fbf4e881700 1 -- start start 2020-11-20T12:05:28.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.213+0000 7fbf4e881700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbf481484d0 0x7fbf48061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:28.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.213+0000 7fbf4e881700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf480b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:28.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.213+0000 7fbf4e881700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf480b1300 con 0x7fbf48066f70 2020-11-20T12:05:28.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.213+0000 7fbf4e881700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf48067730 con 0x7fbf481484d0 2020-11-20T12:05:28.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.213+0000 7fbf4e881700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480b15a0 con 0x7fbf480622c0 2020-11-20T12:05:28.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.214+0000 7fbf4d07e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf480b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:28.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.214+0000 7fbf4d07e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf480b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:41986/0 (socket says 172.21.15.187:41986) 2020-11-20T12:05:28.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.214+0000 7fbf4d07e700 1 -- 172.21.15.187:0/2840893784 learned_addr learned my addr 172.21.15.187:0/2840893784 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:28.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.214+0000 7fbf4d87f700 1 --2- 172.21.15.187:0/2840893784 >> v2:172.21.15.5:3300/0 conn(0x7fbf481484d0 0x7fbf48061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:28.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.215+0000 7fbf4d07e700 1 -- 172.21.15.187:0/2840893784 >> v2:172.21.15.5:3300/0 conn(0x7fbf481484d0 msgr2=0x7fbf48061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:28.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.215+0000 7fbf4d07e700 1 --2- 172.21.15.187:0/2840893784 >> v2:172.21.15.5:3300/0 conn(0x7fbf481484d0 0x7fbf48061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.215+0000 7fbf4d07e700 1 -- 172.21.15.187:0/2840893784 >> v1:172.21.15.5:6789/0 conn(0x7fbf48066f70 legacy=0x7fbf480b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:28.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.215+0000 7fbf4d07e700 1 -- 172.21.15.187:0/2840893784 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480b1840 con 0x7fbf480622c0 2020-11-20T12:05:28.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.215+0000 7fbf4d07e700 1 --2- 172.21.15.187:0/2840893784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf480b5640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fbf38006fd0 tx=0x7fbf38006eb0).ready entity=mon.2 client_cookie=c9d94edc0c2ebf21 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:28.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.216+0000 7fbf4c87d700 1 -- 172.21.15.187:0/2840893784 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38009810 con 0x7fbf480622c0 2020-11-20T12:05:28.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.216+0000 7fbf4c87d700 1 -- 172.21.15.187:0/2840893784 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38009970 con 0x7fbf480622c0 2020-11-20T12:05:28.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.216+0000 7fbf4c87d700 1 -- 172.21.15.187:0/2840893784 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38009cf0 con 0x7fbf480622c0 2020-11-20T12:05:28.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.217+0000 7fbf4e881700 1 -- 172.21.15.187:0/2840893784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 msgr2=0x7fbf480b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:28.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.217+0000 7fbf4e881700 1 --2- 172.21.15.187:0/2840893784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf480b5640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fbf38006fd0 tx=0x7fbf38006eb0).stop 2020-11-20T12:05:28.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.217+0000 7fbf4e881700 1 -- 172.21.15.187:0/2840893784 shutdown_connections 2020-11-20T12:05:28.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.217+0000 7fbf4e881700 1 --2- 172.21.15.187:0/2840893784 >> v2:172.21.15.5:3300/0 conn(0x7fbf481484d0 0x7fbf48061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.217+0000 7fbf4e881700 1 --2- 172.21.15.187:0/2840893784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf480b5640 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.217+0000 7fbf4e881700 1 -- 172.21.15.187:0/2840893784 >> 172.21.15.187:0/2840893784 conn(0x7fbf480bab30 msgr2=0x7fbf480baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:28.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.217+0000 7fbf4e881700 1 -- 172.21.15.187:0/2840893784 shutdown_connections 2020-11-20T12:05:28.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.217+0000 7fbf4e881700 1 -- 172.21.15.187:0/2840893784 wait complete. 2020-11-20T12:05:28.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.219+0000 7fbf4e881700 1 Processor -- start 2020-11-20T12:05:28.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.219+0000 7fbf4e881700 1 -- start start 2020-11-20T12:05:28.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4e881700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf4809c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:28.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4e881700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf48066f70 0x7fbf48136b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:28.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4e881700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf481484d0 0x7fbf48137060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:28.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4e881700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf48137550 con 0x7fbf481484d0 2020-11-20T12:05:28.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4e881700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480b1a20 con 0x7fbf48066f70 2020-11-20T12:05:28.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4e881700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480b1cc0 con 0x7fbf480622c0 2020-11-20T12:05:28.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4d07e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf48066f70 0x7fbf48136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:28.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4d07e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf48066f70 0x7fbf48136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41104/0 (socket says 172.21.15.187:41104) 2020-11-20T12:05:28.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4d07e700 1 -- 172.21.15.187:0/3188545236 learned_addr learned my addr 172.21.15.187:0/3188545236 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:28.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4d87f700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf4809c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:28.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.220+0000 7fbf4e080700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf481484d0 0x7fbf48137060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:28.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf4d07e700 1 -- 172.21.15.187:0/3188545236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 msgr2=0x7fbf4809c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:28.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf4d07e700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf4809c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf4d07e700 1 -- 172.21.15.187:0/3188545236 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf481484d0 msgr2=0x7fbf48137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:28.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf4d07e700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf481484d0 0x7fbf48137060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf4d07e700 1 -- 172.21.15.187:0/3188545236 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480b1f60 con 0x7fbf48066f70 2020-11-20T12:05:28.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf4d07e700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf48066f70 0x7fbf48136b70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fbf38010000 tx=0x7fbf38015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:28.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf35ffb700 1 -- 172.21.15.187:0/3188545236 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38011d90 con 0x7fbf48066f70 2020-11-20T12:05:28.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf35ffb700 1 -- 172.21.15.187:0/3188545236 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3801e460 con 0x7fbf48066f70 2020-11-20T12:05:28.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf480b2200 con 0x7fbf48066f70 2020-11-20T12:05:28.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf35ffb700 1 -- 172.21.15.187:0/3188545236 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3801e7e0 con 0x7fbf48066f70 2020-11-20T12:05:28.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.221+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4809e4f0 con 0x7fbf48066f70 2020-11-20T12:05:28.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.223+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4809e770 con 0x7fbf48066f70 2020-11-20T12:05:28.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.223+0000 7fbf35ffb700 1 -- 172.21.15.187:0/3188545236 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbf38025030 con 0x7fbf48066f70 2020-11-20T12:05:28.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.223+0000 7fbf35ffb700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf20056d20 0x7fbf200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:28.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.224+0000 7fbf35ffb700 1 -- 172.21.15.187:0/3188545236 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbf38071750 con 0x7fbf48066f70 2020-11-20T12:05:28.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.224+0000 7fbf4d87f700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf20056d20 0x7fbf200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:28.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.224+0000 7fbf4d87f700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf20056d20 0x7fbf200591e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbf44006fd0 tx=0x7fbf44008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:28.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.227+0000 7fbf35ffb700 1 -- 172.21.15.187:0/3188545236 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbf38046c90 con 0x7fbf48066f70 2020-11-20T12:05:28.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.467+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf480b2390 con 0x7fbf48066f70 2020-11-20T12:05:28.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.468+0000 7fbf35ffb700 1 -- 172.21.15.187:0/3188545236 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2130 (secure 0 0 0) 0x7fbf38023070 con 0x7fbf48066f70 2020-11-20T12:05:28.472 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:28.472 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4211,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6833/2094645874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2094645874},{"type":"v1","addr":"172.21.15.5:6833","nonce":2094645874}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:57.868606+0000","modified":"2020-11-20T12:04:59.990344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3094884769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3094884769},{"type":"v1","addr":"172.21.15.187:6835","nonce":3094884769}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:28.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.470+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf20056d20 msgr2=0x7fbf200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:28.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.470+0000 7fbf4e881700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf20056d20 0x7fbf200591e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbf44006fd0 tx=0x7fbf44008040).stop 2020-11-20T12:05:28.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf48066f70 msgr2=0x7fbf48136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:28.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf48066f70 0x7fbf48136b70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fbf38010000 tx=0x7fbf38015000).stop 2020-11-20T12:05:28.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 shutdown_connections 2020-11-20T12:05:28.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf20056d20 0x7fbf200591e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf481484d0 0x7fbf48137060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf480622c0 0x7fbf4809c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 --2- 172.21.15.187:0/3188545236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf48066f70 0x7fbf48136b70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:28.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 >> 172.21.15.187:0/3188545236 conn(0x7fbf480bab30 msgr2=0x7fbf481442f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:28.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 shutdown_connections 2020-11-20T12:05:28.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:28.471+0000 7fbf4e881700 1 -- 172.21.15.187:0/3188545236 wait complete. 2020-11-20T12:05:28.477 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 6 2020-11-20T12:05:28.701 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:04:57.868606+0000', 'modified': '2020-11-20T12:04:59.990344+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4264': {'gid': 4264, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3094884769', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3094884769}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3094884769}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:05:28.701 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:05:28.702 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:05:28.980 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":4264,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":31.081813692000001,"mdsmap_epoch":6,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":35.470628152000003} 2020-11-20T12:05:28.981 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 4264, "mdsmap_epoch": 6, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 31.081813692, "state": "up:active", "uptime": 35.470628152, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:05:28.982 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:05:28.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:28.982 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.1 2020-11-20T12:05:29.035 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:05:29.036 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,name=1,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:29.163 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=1,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:29.164 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=1,norbytes" will pass to kernel. 2020-11-20T12:05:29.202 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=1,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:29.203 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=1,norbytes" will pass to kernel. 2020-11-20T12:05:29.209 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:05:29.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:29.210 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2020-11-20T12:05:29.301 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:05:29.301 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi005.front.sepia.ceph.com 2020-11-20T12:05:29.301 INFO:tasks.cephfs.mount:self.client.name = client.2 2020-11-20T12:05:29.302 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2020-11-20T12:05:29.302 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:05:29.302 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2020-11-20T12:05:29.303 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:05:29.303 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2020-11-20T12:05:29.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:29.304 INFO:teuthology.orchestra.run.smithi005:> ip addr 2020-11-20T12:05:29.368 INFO:teuthology.orchestra.run.smithi005.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:29.369 INFO:teuthology.orchestra.run.smithi005.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:29.369 INFO:teuthology.orchestra.run.smithi005.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:29.369 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:29.369 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:29.370 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:29.370 INFO:teuthology.orchestra.run.smithi005.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:29.370 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:6b:fd:de brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:29.370 INFO:teuthology.orchestra.run.smithi005.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:29.370 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:6b:fd:df brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:29.371 INFO:teuthology.orchestra.run.smithi005.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:05:29.371 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:bd:14:42 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:29.371 INFO:teuthology.orchestra.run.smithi005.stdout: inet 172.21.15.5/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2020-11-20T12:05:29.371 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft 42776sec preferred_lft 42776sec 2020-11-20T12:05:29.372 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 fe80::ec4:7aff:febd:1442/64 scope link 2020-11-20T12:05:29.372 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:29.372 INFO:teuthology.orchestra.run.smithi005.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:29.372 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:bd:14:43 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:29.373 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2020-11-20T12:05:29.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:29.374 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:29.374 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:29.375 INFO:teuthology.orchestra.run.smithi005:> sudo ip link add name ceph-brx type bridge 2020-11-20T12:05:29.375 INFO:teuthology.orchestra.run.smithi005:> sudo ip addr flush dev ceph-brx 2020-11-20T12:05:29.375 INFO:teuthology.orchestra.run.smithi005:> sudo ip link set ceph-brx up 2020-11-20T12:05:29.376 INFO:teuthology.orchestra.run.smithi005:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2020-11-20T12:05:29.376 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:30.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:30.037 INFO:teuthology.orchestra.run.smithi005:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:05:30.122 INFO:teuthology.orchestra.run.smithi005.stdout:1 2020-11-20T12:05:30.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:30.126 INFO:teuthology.orchestra.run.smithi005:> route 2020-11-20T12:05:30.184 INFO:teuthology.orchestra.run.smithi005.stdout:Kernel IP routing table 2020-11-20T12:05:30.185 INFO:teuthology.orchestra.run.smithi005.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:05:30.185 INFO:teuthology.orchestra.run.smithi005.stdout:default _gateway 0.0.0.0 UG 102 0 0 ens1f0 2020-11-20T12:05:30.185 INFO:teuthology.orchestra.run.smithi005.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 102 0 0 ens1f0 2020-11-20T12:05:30.186 INFO:teuthology.orchestra.run.smithi005.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:05:30.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:30.188 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:30.188 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:30.188 INFO:teuthology.orchestra.run.smithi005:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2020-11-20T12:05:30.189 INFO:teuthology.orchestra.run.smithi005:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2020-11-20T12:05:30.189 INFO:teuthology.orchestra.run.smithi005:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2020-11-20T12:05:30.189 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:31.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:31.220 INFO:teuthology.orchestra.run.smithi005:> ip netns list 2020-11-20T12:05:31.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:31.280 INFO:teuthology.orchestra.run.smithi005:> ip netns list-id 2020-11-20T12:05:31.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:31.337 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:31.337 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:31.337 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2020-11-20T12:05:31.338 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 0 2020-11-20T12:05:31.338 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:31.560 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.1/16 2020-11-20T12:05:31.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:31.561 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:31.561 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:31.562 INFO:teuthology.orchestra.run.smithi005:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.0 2020-11-20T12:05:31.562 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2020-11-20T12:05:31.562 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2020-11-20T12:05:31.563 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2020-11-20T12:05:31.563 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2020-11-20T12:05:31.563 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:31.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:31.997 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:31.997 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:31.998 INFO:teuthology.orchestra.run.smithi005:> sudo ip link set brx.0 up 2020-11-20T12:05:31.998 INFO:teuthology.orchestra.run.smithi005:> sudo ip link set dev brx.0 master ceph-brx 2020-11-20T12:05:31.998 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:32.251 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:32.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.536+0000 7fbc1ffff700 1 Processor -- start 2020-11-20T12:05:32.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.537+0000 7fbc1ffff700 1 -- start start 2020-11-20T12:05:32.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.538+0000 7fbc1ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc20066f70 0x7fbc200b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:32.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.538+0000 7fbc1ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc200b1300 con 0x7fbc200b6650 2020-11-20T12:05:32.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.538+0000 7fbc1ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc200b15a0 con 0x7fbc201484d0 2020-11-20T12:05:32.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.538+0000 7fbc1ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc20067730 con 0x7fbc20066f70 2020-11-20T12:05:32.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.538+0000 7fbc1effd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbc201484d0 0x7fbc20061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37720/0 (socket says 172.21.15.187:37720) 2020-11-20T12:05:32.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.538+0000 7fbc1effd700 1 -- 172.21.15.187:0/149785345 learned_addr learned my addr 172.21.15.187:0/149785345 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:32.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.539+0000 7fbc1f7fe700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc20066f70 0x7fbc200b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:32.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.539+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186919961 0 0) 0x7fbc200b15a0 con 0x7fbc201484d0 2020-11-20T12:05:32.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.539+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c003240 con 0x7fbc201484d0 2020-11-20T12:05:32.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.539+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3617522449 0 0) 0x7fbc0c003240 con 0x7fbc201484d0 2020-11-20T12:05:32.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.539+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc20066f70 msgr2=0x7fbc200b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:32.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.539+0000 7fbc1dffb700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc20066f70 0x7fbc200b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:32.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.539+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 >> v1:172.21.15.5:6789/0 conn(0x7fbc200b6650 legacy=0x7fbc200afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:32.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.539+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200b1840 con 0x7fbc201484d0 2020-11-20T12:05:32.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fbc10001520 con 0x7fbc201484d0 2020-11-20T12:05:32.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 >> v1:172.21.15.187:6789/0 conn(0x7fbc201484d0 legacy=0x7fbc20061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:32.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1dffb700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c003730 0x7fbc0c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:32.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1dffb700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c0044e0 0x7fbc0c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:32.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1dffb700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc20066f70 0x7fbc0c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:32.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc20067730 con 0x7fbc0c003730 2020-11-20T12:05:32.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c0042a0 con 0x7fbc0c0044e0 2020-11-20T12:05:32.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c00b730 con 0x7fbc20066f70 2020-11-20T12:05:32.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.540+0000 7fbc1e7fc700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c0044e0 0x7fbc0c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:32.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1effd700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c003730 0x7fbc0c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:32.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1e7fc700 1 -- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc20066f70 msgr2=0x7fbc0c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:32.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1e7fc700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc20066f70 0x7fbc0c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:32.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1e7fc700 1 -- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c003730 msgr2=0x7fbc0c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:32.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1e7fc700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c003730 0x7fbc0c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:32.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1e7fc700 1 -- 172.21.15.187:0/149785345 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c003240 con 0x7fbc0c0044e0 2020-11-20T12:05:32.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1effd700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c003730 0x7fbc0c003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:32.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1e7fc700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c0044e0 0x7fbc0c004960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbc14007e50 tx=0x7fbc14007b90).ready entity=mon.1 client_cookie=137bb40c6ee4c6c0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:32.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc14015070 con 0x7fbc0c0044e0 2020-11-20T12:05:32.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc14007020 con 0x7fbc0c0044e0 2020-11-20T12:05:32.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.541+0000 7fbc1dffb700 1 -- 172.21.15.187:0/149785345 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc140073c0 con 0x7fbc0c0044e0 2020-11-20T12:05:32.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.542+0000 7fbc1ffff700 1 -- 172.21.15.187:0/149785345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c0044e0 msgr2=0x7fbc0c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:32.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.542+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/149785345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c0044e0 0x7fbc0c004960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbc14007e50 tx=0x7fbc14007b90).stop 2020-11-20T12:05:32.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.542+0000 7fbc1f7fe700 1 -- 172.21.15.187:0/149785345 reap_dead start 2020-11-20T12:05:32.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.542+0000 7fbc1ffff700 1 -- 172.21.15.187:0/149785345 shutdown_connections 2020-11-20T12:05:32.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.542+0000 7fbc1ffff700 1 -- 172.21.15.187:0/149785345 >> 172.21.15.187:0/149785345 conn(0x7fbc200bab30 msgr2=0x7fbc200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:32.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.542+0000 7fbc1ffff700 1 -- 172.21.15.187:0/149785345 shutdown_connections 2020-11-20T12:05:32.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.542+0000 7fbc1ffff700 1 -- 172.21.15.187:0/149785345 wait complete. 2020-11-20T12:05:32.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.544+0000 7fbc1ffff700 1 Processor -- start 2020-11-20T12:05:32.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.544+0000 7fbc1ffff700 1 -- start start 2020-11-20T12:05:32.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2015b890 0x7fbc200991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:32.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc2015bc70 0x7fbc20142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:32.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc2015c0f0 0x7fbc20143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc200996c0 con 0x7fbc2015b890 2020-11-20T12:05:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc200998e0 con 0x7fbc2015bc70 2020-11-20T12:05:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2009fd80 con 0x7fbc2015c0f0 2020-11-20T12:05:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1e7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc2015bc70 0x7fbc20142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1e7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc2015bc70 0x7fbc20142e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41128/0 (socket says 172.21.15.187:41128) 2020-11-20T12:05:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1e7fc700 1 -- 172.21.15.187:0/2819143497 learned_addr learned my addr 172.21.15.187:0/2819143497 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1effd700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2015b890 0x7fbc200991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1e7fc700 1 -- 172.21.15.187:0/2819143497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc2015c0f0 msgr2=0x7fbc20143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1e7fc700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc2015c0f0 0x7fbc20143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1e7fc700 1 -- 172.21.15.187:0/2819143497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2015b890 msgr2=0x7fbc200991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:32.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1e7fc700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2015b890 0x7fbc200991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:32.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.546+0000 7fbc1e7fc700 1 -- 172.21.15.187:0/2819143497 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200a0020 con 0x7fbc2015bc70 2020-11-20T12:05:32.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.547+0000 7fbc1e7fc700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc2015bc70 0x7fbc20142e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fbc0c004430 tx=0x7fbc14017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:32.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.547+0000 7fbc06ffd700 1 -- 172.21.15.187:0/2819143497 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc14006d50 con 0x7fbc2015bc70 2020-11-20T12:05:32.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.547+0000 7fbc06ffd700 1 -- 172.21.15.187:0/2819143497 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc14011470 con 0x7fbc2015bc70 2020-11-20T12:05:32.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.547+0000 7fbc06ffd700 1 -- 172.21.15.187:0/2819143497 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc14011830 con 0x7fbc2015bc70 2020-11-20T12:05:32.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.547+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc20143810 con 0x7fbc2015bc70 2020-11-20T12:05:32.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.547+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2009ec60 con 0x7fbc2015bc70 2020-11-20T12:05:32.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.548+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc200a03e0 con 0x7fbc2015bc70 2020-11-20T12:05:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.549+0000 7fbc06ffd700 1 -- 172.21.15.187:0/2819143497 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbc1402c070 con 0x7fbc2015bc70 2020-11-20T12:05:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.550+0000 7fbc06ffd700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbfc056d50 0x7fbbfc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.550+0000 7fbc06ffd700 1 -- 172.21.15.187:0/2819143497 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbc14001e40 con 0x7fbc2015bc70 2020-11-20T12:05:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.550+0000 7fbc1effd700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbfc056d50 0x7fbbfc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.551+0000 7fbc1effd700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbfc056d50 0x7fbbfc059210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fbc10001e90 tx=0x7fbc10009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.554+0000 7fbc06ffd700 1 -- 172.21.15.187:0/2819143497 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbc14047440 con 0x7fbc2015bc70 2020-11-20T12:05:32.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.747+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc200a01b0 con 0x7fbc2015bc70 2020-11-20T12:05:32.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.749+0000 7fbc06ffd700 1 -- 172.21.15.187:0/2819143497 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7fbc14020090 con 0x7fbc2015bc70 2020-11-20T12:05:32.753 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:32.753 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:32.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.751+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbfc056d50 msgr2=0x7fbbfc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:32.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.751+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbfc056d50 0x7fbbfc059210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fbc10001e90 tx=0x7fbc10009040).stop 2020-11-20T12:05:32.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc2015bc70 msgr2=0x7fbc20142e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:32.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc2015bc70 0x7fbc20142e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fbc0c004430 tx=0x7fbc14017000).stop 2020-11-20T12:05:32.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 shutdown_connections 2020-11-20T12:05:32.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbbfc056d50 0x7fbbfc059210 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:32.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2015b890 0x7fbc200991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:32.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc2015c0f0 0x7fbc20143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:32.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 --2- 172.21.15.187:0/2819143497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc2015bc70 0x7fbc20142e30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:32.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 >> 172.21.15.187:0/2819143497 conn(0x7fbc200bab30 msgr2=0x7fbc200afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:32.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 shutdown_connections 2020-11-20T12:05:32.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:32.752+0000 7fbc1ffff700 1 -- 172.21.15.187:0/2819143497 wait complete. 2020-11-20T12:05:32.779 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:32.780 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:33.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.019+0000 7fa13ffff700 1 Processor -- start 2020-11-20T12:05:33.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.020+0000 7fa13ffff700 1 -- start start 2020-11-20T12:05:33.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.021+0000 7fa13ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa1400bbaa0 0x7fa1400a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.021+0000 7fa13ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa1401426f0 0x7fa140142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.021+0000 7fa13ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa140149930 con 0x7fa1400bbe80 2020-11-20T12:05:33.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.021+0000 7fa13ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1400a3560 con 0x7fa1401426f0 2020-11-20T12:05:33.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.021+0000 7fa13ffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa14014a330 con 0x7fa1400bbaa0 2020-11-20T12:05:33.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa1400bbaa0 0x7fa1400a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa1400bbaa0 0x7fa1400a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41136/0 (socket says 172.21.15.187:41136) 2020-11-20T12:05:33.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13effd700 1 -- 172.21.15.187:0/740786983 learned_addr learned my addr 172.21.15.187:0/740786983 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:33.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13f7fe700 1 --2- 172.21.15.187:0/740786983 >> v2:172.21.15.5:3300/0 conn(0x7fa1401426f0 0x7fa140142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13effd700 1 -- 172.21.15.187:0/740786983 >> v2:172.21.15.5:3300/0 conn(0x7fa1401426f0 msgr2=0x7fa140142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:33.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13effd700 1 --2- 172.21.15.187:0/740786983 >> v2:172.21.15.5:3300/0 conn(0x7fa1401426f0 0x7fa140142b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13effd700 1 -- 172.21.15.187:0/740786983 >> v1:172.21.15.5:6789/0 conn(0x7fa1400bbe80 legacy=0x7fa140149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:33.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13effd700 1 -- 172.21.15.187:0/740786983 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1400a3250 con 0x7fa1400bbaa0 2020-11-20T12:05:33.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.022+0000 7fa13effd700 1 --2- 172.21.15.187:0/740786983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1400bbaa0 0x7fa1400a4bf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa140060c00 tx=0x7fa130009290).ready entity=mon.1 client_cookie=6b5c28e7319713a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.023+0000 7fa13e7fc700 1 -- 172.21.15.187:0/740786983 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13000bb80 con 0x7fa1400bbaa0 2020-11-20T12:05:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.023+0000 7fa13e7fc700 1 -- 172.21.15.187:0/740786983 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13000bce0 con 0x7fa1400bbaa0 2020-11-20T12:05:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.023+0000 7fa13e7fc700 1 -- 172.21.15.187:0/740786983 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa130011610 con 0x7fa1400bbaa0 2020-11-20T12:05:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.023+0000 7fa13ffff700 1 -- 172.21.15.187:0/740786983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1400bbaa0 msgr2=0x7fa1400a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.023+0000 7fa13ffff700 1 --2- 172.21.15.187:0/740786983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1400bbaa0 0x7fa1400a4bf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa140060c00 tx=0x7fa130009290).stop 2020-11-20T12:05:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.024+0000 7fa13ffff700 1 -- 172.21.15.187:0/740786983 shutdown_connections 2020-11-20T12:05:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.024+0000 7fa13ffff700 1 --2- 172.21.15.187:0/740786983 >> v2:172.21.15.5:3300/0 conn(0x7fa1401426f0 0x7fa140142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.024+0000 7fa13ffff700 1 --2- 172.21.15.187:0/740786983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1400bbaa0 0x7fa1400a4bf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.024+0000 7fa13ffff700 1 -- 172.21.15.187:0/740786983 >> 172.21.15.187:0/740786983 conn(0x7fa140061bd0 msgr2=0x7fa1400678a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.024+0000 7fa13ffff700 1 -- 172.21.15.187:0/740786983 shutdown_connections 2020-11-20T12:05:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.024+0000 7fa13ffff700 1 -- 172.21.15.187:0/740786983 wait complete. 2020-11-20T12:05:33.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.026+0000 7fa13ffff700 1 Processor -- start 2020-11-20T12:05:33.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.026+0000 7fa13ffff700 1 -- start start 2020-11-20T12:05:33.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.027+0000 7fa13ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1400bbaa0 0x7fa140145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.027+0000 7fa13ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1400bbe80 0x7fa140134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.027+0000 7fa13ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1401426f0 0x7fa1401338d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.027+0000 7fa13ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa140144590 con 0x7fa1400bbe80 2020-11-20T12:05:33.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.027+0000 7fa13ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa140144830 con 0x7fa1401426f0 2020-11-20T12:05:33.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.027+0000 7fa13ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa140143340 con 0x7fa1400bbaa0 2020-11-20T12:05:33.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.027+0000 7fa13f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1401426f0 0x7fa1401338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1401426f0 0x7fa1401338d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41144/0 (socket says 172.21.15.187:41144) 2020-11-20T12:05:33.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13f7fe700 1 -- 172.21.15.187:0/1143100918 learned_addr learned my addr 172.21.15.187:0/1143100918 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:33.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa1367fc700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1400bbe80 0x7fa140134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13effd700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1400bbaa0 0x7fa140145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13f7fe700 1 -- 172.21.15.187:0/1143100918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1400bbaa0 msgr2=0x7fa140145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13f7fe700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1400bbaa0 0x7fa140145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13f7fe700 1 -- 172.21.15.187:0/1143100918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1400bbe80 msgr2=0x7fa140134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13f7fe700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1400bbe80 0x7fa140134560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13f7fe700 1 -- 172.21.15.187:0/1143100918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa140134a50 con 0x7fa1401426f0 2020-11-20T12:05:33.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.028+0000 7fa13f7fe700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1401426f0 0x7fa1401338d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fa13800a730 tx=0x7fa138008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.029+0000 7fa1377fe700 1 -- 172.21.15.187:0/1143100918 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13800e5d0 con 0x7fa1401426f0 2020-11-20T12:05:33.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.029+0000 7fa1377fe700 1 -- 172.21.15.187:0/1143100918 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13800c030 con 0x7fa1401426f0 2020-11-20T12:05:33.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.029+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1401435d0 con 0x7fa1401426f0 2020-11-20T12:05:33.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.029+0000 7fa1377fe700 1 -- 172.21.15.187:0/1143100918 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13800e980 con 0x7fa1401426f0 2020-11-20T12:05:33.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.029+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa140133f80 con 0x7fa1401426f0 2020-11-20T12:05:33.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.030+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa140143900 con 0x7fa1401426f0 2020-11-20T12:05:33.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.031+0000 7fa1377fe700 1 -- 172.21.15.187:0/1143100918 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa138019030 con 0x7fa1401426f0 2020-11-20T12:05:33.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.031+0000 7fa1377fe700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa11c056d20 0x7fa11c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.031+0000 7fa1377fe700 1 -- 172.21.15.187:0/1143100918 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa13806f860 con 0x7fa1401426f0 2020-11-20T12:05:33.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.032+0000 7fa13effd700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa11c056d20 0x7fa11c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.032+0000 7fa13effd700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa11c056d20 0x7fa11c0591e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa1300090f0 tx=0x7fa130016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.035+0000 7fa1377fe700 1 -- 172.21.15.187:0/1143100918 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa138044cb0 con 0x7fa1401426f0 2020-11-20T12:05:33.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.222+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa1400a4790 con 0x7fa1401426f0 2020-11-20T12:05:33.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.222+0000 7fa1377fe700 1 -- 172.21.15.187:0/1143100918 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa1380482d0 con 0x7fa1401426f0 2020-11-20T12:05:33.226 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:33.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.224+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa11c056d20 msgr2=0x7fa11c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.224+0000 7fa13ffff700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa11c056d20 0x7fa11c0591e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa1300090f0 tx=0x7fa130016000).stop 2020-11-20T12:05:33.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.224+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1401426f0 msgr2=0x7fa1401338d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.224+0000 7fa13ffff700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1401426f0 0x7fa1401338d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fa13800a730 tx=0x7fa138008040).stop 2020-11-20T12:05:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.225+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 shutdown_connections 2020-11-20T12:05:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.225+0000 7fa13ffff700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa11c056d20 0x7fa11c0591e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.225+0000 7fa13ffff700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1401426f0 0x7fa1401338d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.225+0000 7fa13ffff700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1400bbaa0 0x7fa140145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.225+0000 7fa13ffff700 1 --2- 172.21.15.187:0/1143100918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1400bbe80 0x7fa140134560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.225+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 >> 172.21.15.187:0/1143100918 conn(0x7fa140061bd0 msgr2=0x7fa140140c00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:33.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.225+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 shutdown_connections 2020-11-20T12:05:33.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.225+0000 7fa13ffff700 1 -- 172.21.15.187:0/1143100918 wait complete. 2020-11-20T12:05:33.250 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:33.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.483+0000 7f3e90b17700 1 Processor -- start 2020-11-20T12:05:33.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.484+0000 7f3e90b17700 1 -- start start 2020-11-20T12:05:33.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.485+0000 7f3e90b17700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e8c0b1380 0x7f3e8c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.485+0000 7f3e90b17700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.485+0000 7f3e90b17700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8c150390 con 0x7f3e8c065680 2020-11-20T12:05:33.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.485+0000 7f3e90b17700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8c150630 con 0x7f3e8c0b1380 2020-11-20T12:05:33.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.485+0000 7f3e90b17700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c068510 con 0x7f3e8c0b1760 2020-11-20T12:05:33.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e8b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e8c0b1380 0x7f3e8c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e8b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e8c0b1380 0x7f3e8c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41150/0 (socket says 172.21.15.187:41150) 2020-11-20T12:05:33.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e8b7fe700 1 -- 172.21.15.187:0/3843078127 learned_addr learned my addr 172.21.15.187:0/3843078127 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:33.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e82ffd700 1 --2- 172.21.15.187:0/3843078127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e8b7fe700 1 -- 172.21.15.187:0/3843078127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 msgr2=0x7f3e8c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:33.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e8b7fe700 1 --2- 172.21.15.187:0/3843078127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e8b7fe700 1 -- 172.21.15.187:0/3843078127 >> v1:172.21.15.5:6789/0 conn(0x7f3e8c065680 legacy=0x7f3e8c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:33.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e8b7fe700 1 -- 172.21.15.187:0/3843078127 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c150fc0 con 0x7f3e8c0b1380 2020-11-20T12:05:33.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.486+0000 7f3e82ffd700 1 --2- 172.21.15.187:0/3843078127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:33.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e8b7fe700 1 --2- 172.21.15.187:0/3843078127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c0b1380 0x7f3e8c150ad0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f3e7c006fd0 tx=0x7f3e7c006eb0).ready entity=mon.1 client_cookie=c69bfa73ba42dc19 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e8affd700 1 -- 172.21.15.187:0/3843078127 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c009890 con 0x7f3e8c0b1380 2020-11-20T12:05:33.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e8affd700 1 -- 172.21.15.187:0/3843078127 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7c0099f0 con 0x7f3e8c0b1380 2020-11-20T12:05:33.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e8affd700 1 -- 172.21.15.187:0/3843078127 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c009d70 con 0x7f3e8c0b1380 2020-11-20T12:05:33.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e90b17700 1 -- 172.21.15.187:0/3843078127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c0b1380 msgr2=0x7f3e8c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e90b17700 1 --2- 172.21.15.187:0/3843078127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c0b1380 0x7f3e8c150ad0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f3e7c006fd0 tx=0x7f3e7c006eb0).stop 2020-11-20T12:05:33.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e90b17700 1 -- 172.21.15.187:0/3843078127 shutdown_connections 2020-11-20T12:05:33.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e90b17700 1 --2- 172.21.15.187:0/3843078127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c0b1380 0x7f3e8c150ad0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.487+0000 7f3e90b17700 1 --2- 172.21.15.187:0/3843078127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.488+0000 7f3e90b17700 1 -- 172.21.15.187:0/3843078127 >> 172.21.15.187:0/3843078127 conn(0x7f3e8c0bab30 msgr2=0x7f3e8c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:33.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.488+0000 7f3e90b17700 1 -- 172.21.15.187:0/3843078127 shutdown_connections 2020-11-20T12:05:33.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.488+0000 7f3e90b17700 1 -- 172.21.15.187:0/3843078127 wait complete. 2020-11-20T12:05:33.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.489+0000 7f3e90b17700 1 Processor -- start 2020-11-20T12:05:33.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.490+0000 7f3e90b17700 1 -- start start 2020-11-20T12:05:33.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e90b17700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e8c0b1380 0x7f3e8c09ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e90b17700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c14a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e90b17700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c160420 0x7f3e8c0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e90b17700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c160910 con 0x7f3e8c0b1380 2020-11-20T12:05:33.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e90b17700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c09d100 con 0x7f3e8c160420 2020-11-20T12:05:33.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e90b17700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c09c0d0 con 0x7f3e8c0b1760 2020-11-20T12:05:33.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c160420 0x7f3e8c0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c160420 0x7f3e8c0b5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41156/0 (socket says 172.21.15.187:41156) 2020-11-20T12:05:33.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e8bfff700 1 -- 172.21.15.187:0/2754866482 learned_addr learned my addr 172.21.15.187:0/2754866482 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:33.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.491+0000 7f3e8b7fe700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e8c0b1380 0x7f3e8c09ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e8bfff700 1 -- 172.21.15.187:0/2754866482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 msgr2=0x7f3e8c14a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e82ffd700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c14a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e8bfff700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c14a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e8bfff700 1 -- 172.21.15.187:0/2754866482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e8c0b1380 msgr2=0x7f3e8c09ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e8bfff700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e8c0b1380 0x7f3e8c09ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e8bfff700 1 -- 172.21.15.187:0/2754866482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c09e3c0 con 0x7f3e8c160420 2020-11-20T12:05:33.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e8bfff700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c160420 0x7f3e8c0b5110 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f3e8400c860 tx=0x7f3e8400cc20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e83fff700 1 -- 172.21.15.187:0/2754866482 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8400a7b0 con 0x7f3e8c160420 2020-11-20T12:05:33.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e83fff700 1 -- 172.21.15.187:0/2754866482 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8400a930 con 0x7f3e8c160420 2020-11-20T12:05:33.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8c14ac60 con 0x7f3e8c160420 2020-11-20T12:05:33.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e83fff700 1 -- 172.21.15.187:0/2754866482 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8400acf0 con 0x7f3e8c160420 2020-11-20T12:05:33.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.492+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c09f3e0 con 0x7f3e8c160420 2020-11-20T12:05:33.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.494+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8c14af90 con 0x7f3e8c160420 2020-11-20T12:05:33.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.494+0000 7f3e83fff700 1 -- 172.21.15.187:0/2754866482 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3e84010070 con 0x7f3e8c160420 2020-11-20T12:05:33.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.495+0000 7f3e83fff700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e68056d20 0x7f3e680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.495+0000 7f3e83fff700 1 -- 172.21.15.187:0/2754866482 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e8406fd70 con 0x7f3e8c160420 2020-11-20T12:05:33.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.495+0000 7f3e8b7fe700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e68056d20 0x7f3e680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.496+0000 7f3e8b7fe700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e68056d20 0x7f3e680591e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f3e7c00b1b0 tx=0x7f3e7c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.499+0000 7f3e83fff700 1 -- 172.21.15.187:0/2754866482 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3e8c14af90 con 0x7f3e8c160420 2020-11-20T12:05:33.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.671+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3e8c14af90 con 0x7f3e8c160420 2020-11-20T12:05:33.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.672+0000 7f3e83fff700 1 -- 172.21.15.187:0/2754866482 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3e8c14af90 con 0x7f3e8c160420 2020-11-20T12:05:33.675 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:33.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e68056d20 msgr2=0x7f3e680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e68056d20 0x7f3e680591e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f3e7c00b1b0 tx=0x7f3e7c015000).stop 2020-11-20T12:05:33.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c160420 msgr2=0x7f3e8c0b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c160420 0x7f3e8c0b5110 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f3e8400c860 tx=0x7f3e8400cc20).stop 2020-11-20T12:05:33.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 shutdown_connections 2020-11-20T12:05:33.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e68056d20 0x7f3e680591e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e8c0b1380 0x7f3e8c09ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e8c160420 0x7f3e8c0b5110 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 --2- 172.21.15.187:0/2754866482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e8c0b1760 0x7f3e8c14a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 >> 172.21.15.187:0/2754866482 conn(0x7f3e8c0bab30 msgr2=0x7f3e8c0b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:33.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 shutdown_connections 2020-11-20T12:05:33.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.674+0000 7f3e90b17700 1 -- 172.21.15.187:0/2754866482 wait complete. 2020-11-20T12:05:33.700 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:33.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.934+0000 7fcf8e672700 1 Processor -- start 2020-11-20T12:05:33.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.934+0000 7fcf8e672700 1 -- start start 2020-11-20T12:05:33.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.936+0000 7fcf8e672700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 0x7fcf880b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.936+0000 7fcf8e672700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880685f0 con 0x7fcf880afca0 2020-11-20T12:05:33.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.936+0000 7fcf8e672700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880b0130 con 0x7fcf88061f90 2020-11-20T12:05:33.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.936+0000 7fcf8e672700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88067e10 con 0x7fcf88062370 2020-11-20T12:05:33.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.936+0000 7fcf8d670700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcf88061f90 0x7fcf880b04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37768/0 (socket says 172.21.15.187:37768) 2020-11-20T12:05:33.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.936+0000 7fcf8d670700 1 -- 172.21.15.187:0/3759266220 learned_addr learned my addr 172.21.15.187:0/3759266220 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:33.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf8ce6f700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 0x7fcf880b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055730543 0 0) 0x7fcf880b0130 con 0x7fcf88061f90 2020-11-20T12:05:33.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf70003240 con 0x7fcf88061f90 2020-11-20T12:05:33.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 949009634 0 0) 0x7fcf70003240 con 0x7fcf88061f90 2020-11-20T12:05:33.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 msgr2=0x7fcf880b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:33.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 0x7fcf880b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 >> v1:172.21.15.5:6789/0 conn(0x7fcf880afca0 legacy=0x7fcf88148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880680b0 con 0x7fcf88061f90 2020-11-20T12:05:33.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcf840014e0 con 0x7fcf88061f90 2020-11-20T12:05:33.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 >> v1:172.21.15.187:6789/0 conn(0x7fcf88061f90 legacy=0x7fcf880b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf70003730 0x7fcf70003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf700044e0 0x7fcf70004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.937+0000 7fcf77fff700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 0x7fcf70004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.938+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88067e10 con 0x7fcf700044e0 2020-11-20T12:05:33.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.938+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70005390 con 0x7fcf70003730 2020-11-20T12:05:33.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.938+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf700042f0 con 0x7fcf88062370 2020-11-20T12:05:33.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.938+0000 7fcf8ce6f700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf700044e0 0x7fcf70004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.938+0000 7fcf8de71700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 0x7fcf70004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.938+0000 7fcf8d670700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf70003730 0x7fcf70003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.939+0000 7fcf8de71700 1 -- 172.21.15.187:0/3759266220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf70003730 msgr2=0x7fcf70003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:33.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.939+0000 7fcf8de71700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf70003730 0x7fcf70003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.939+0000 7fcf8de71700 1 -- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf700044e0 msgr2=0x7fcf70004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:33.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.939+0000 7fcf8de71700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf700044e0 0x7fcf70004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.939+0000 7fcf8de71700 1 -- 172.21.15.187:0/3759266220 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf70003240 con 0x7fcf88062370 2020-11-20T12:05:33.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.939+0000 7fcf8ce6f700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf700044e0 0x7fcf70004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:05:33.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.939+0000 7fcf8d670700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf70003730 0x7fcf70003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:33.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.940+0000 7fcf8de71700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 0x7fcf70004e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fcf80001e70 tx=0x7fcf80007f10).ready entity=mon.2 client_cookie=3e1e03d550a41f3b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.940+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80001540 con 0x7fcf88062370 2020-11-20T12:05:33.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.940+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8000ca70 con 0x7fcf88062370 2020-11-20T12:05:33.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.940+0000 7fcf77fff700 1 -- 172.21.15.187:0/3759266220 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8000ce10 con 0x7fcf88062370 2020-11-20T12:05:33.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.941+0000 7fcf8e672700 1 -- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 msgr2=0x7fcf70004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.941+0000 7fcf8e672700 1 --2- 172.21.15.187:0/3759266220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf88062370 0x7fcf70004e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fcf80001e70 tx=0x7fcf80007f10).stop 2020-11-20T12:05:33.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.941+0000 7fcf8de71700 1 -- 172.21.15.187:0/3759266220 reap_dead start 2020-11-20T12:05:33.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.941+0000 7fcf8e672700 1 -- 172.21.15.187:0/3759266220 shutdown_connections 2020-11-20T12:05:33.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.941+0000 7fcf8e672700 1 -- 172.21.15.187:0/3759266220 >> 172.21.15.187:0/3759266220 conn(0x7fcf880bab30 msgr2=0x7fcf880baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:33.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.941+0000 7fcf8e672700 1 -- 172.21.15.187:0/3759266220 shutdown_connections 2020-11-20T12:05:33.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.941+0000 7fcf8e672700 1 -- 172.21.15.187:0/3759266220 wait complete. 2020-11-20T12:05:33.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.942+0000 7fcf8e672700 1 Processor -- start 2020-11-20T12:05:33.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.942+0000 7fcf8e672700 1 -- start start 2020-11-20T12:05:33.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8e672700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf880b2010 0x7fcf8809ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8e672700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf880b23f0 0x7fcf88136010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8e672700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf88136500 0x7fcf8809dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8e672700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88136980 con 0x7fcf880b2010 2020-11-20T12:05:33.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8e672700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880b2870 con 0x7fcf88136500 2020-11-20T12:05:33.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8e672700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880b2b10 con 0x7fcf880b23f0 2020-11-20T12:05:33.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8de71700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf88136500 0x7fcf8809dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8de71700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf88136500 0x7fcf8809dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41178/0 (socket says 172.21.15.187:41178) 2020-11-20T12:05:33.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.943+0000 7fcf8de71700 1 -- 172.21.15.187:0/2819089405 learned_addr learned my addr 172.21.15.187:0/2819089405 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:33.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8ce6f700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf880b23f0 0x7fcf88136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8d670700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf880b2010 0x7fcf8809ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8de71700 1 -- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf880b23f0 msgr2=0x7fcf88136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8de71700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf880b23f0 0x7fcf88136010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8de71700 1 -- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf880b2010 msgr2=0x7fcf8809ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:33.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8de71700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf880b2010 0x7fcf8809ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:33.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8de71700 1 -- 172.21.15.187:0/2819089405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880b2db0 con 0x7fcf88136500 2020-11-20T12:05:33.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8d670700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf880b2010 0x7fcf8809ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:33.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8de71700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf88136500 0x7fcf8809dae0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fcf80001e40 tx=0x7fcf80015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf757fa700 1 -- 172.21.15.187:0/2819089405 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8001f400 con 0x7fcf88136500 2020-11-20T12:05:33.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf757fa700 1 -- 172.21.15.187:0/2819089405 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8001f560 con 0x7fcf88136500 2020-11-20T12:05:33.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf757fa700 1 -- 172.21.15.187:0/2819089405 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8001f900 con 0x7fcf88136500 2020-11-20T12:05:33.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.944+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf880a0500 con 0x7fcf88136500 2020-11-20T12:05:33.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.945+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8809dfd0 con 0x7fcf88136500 2020-11-20T12:05:33.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.946+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf880b2f40 con 0x7fcf88136500 2020-11-20T12:05:33.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.946+0000 7fcf757fa700 1 -- 172.21.15.187:0/2819089405 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcf80026030 con 0x7fcf88136500 2020-11-20T12:05:33.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.947+0000 7fcf757fa700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf60056d20 0x7fcf600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:33.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.947+0000 7fcf757fa700 1 -- 172.21.15.187:0/2819089405 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcf80071740 con 0x7fcf88136500 2020-11-20T12:05:33.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.947+0000 7fcf8d670700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf60056d20 0x7fcf600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:33.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.948+0000 7fcf8d670700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf60056d20 0x7fcf600591e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fcf84001610 tx=0x7fcf88068240).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:33.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:33.951+0000 7fcf757fa700 1 -- 172.21.15.187:0/2819089405 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcf80046bd0 con 0x7fcf88136500 2020-11-20T12:05:34.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.139+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf88061a20 con 0x7fcf88136500 2020-11-20T12:05:34.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.140+0000 7fcf757fa700 1 -- 172.21.15.187:0/2819089405 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcf800240a0 con 0x7fcf88136500 2020-11-20T12:05:34.143 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:34.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf60056d20 msgr2=0x7fcf600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf60056d20 0x7fcf600591e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fcf84001610 tx=0x7fcf88068240).stop 2020-11-20T12:05:34.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf88136500 msgr2=0x7fcf8809dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf88136500 0x7fcf8809dae0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fcf80001e40 tx=0x7fcf80015000).stop 2020-11-20T12:05:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 shutdown_connections 2020-11-20T12:05:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf60056d20 0x7fcf600591e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf880b2010 0x7fcf8809ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf88136500 0x7fcf8809dae0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 --2- 172.21.15.187:0/2819089405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf880b23f0 0x7fcf88136010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.142+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 >> 172.21.15.187:0/2819089405 conn(0x7fcf880bab30 msgr2=0x7fcf880b7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:34.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.143+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 shutdown_connections 2020-11-20T12:05:34.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.143+0000 7fcf8e672700 1 -- 172.21.15.187:0/2819089405 wait complete. 2020-11-20T12:05:34.169 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:34.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.413+0000 7f9551df6700 1 Processor -- start 2020-11-20T12:05:34.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.413+0000 7f9551df6700 1 -- start start 2020-11-20T12:05:34.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.414+0000 7f9551df6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c1484d0 0x7f954c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.414+0000 7f9551df6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c0b1300 con 0x7f954c066f70 2020-11-20T12:05:34.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.414+0000 7f9551df6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c0b15a0 con 0x7f954c0622c0 2020-11-20T12:05:34.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.414+0000 7f9551df6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c067730 con 0x7f954c1484d0 2020-11-20T12:05:34.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.415+0000 7f9543fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f954c0622c0 0x7f954c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37792/0 (socket says 172.21.15.187:37792) 2020-11-20T12:05:34.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.415+0000 7f9543fff700 1 -- 172.21.15.187:0/4047900519 learned_addr learned my addr 172.21.15.187:0/4047900519 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:34.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.415+0000 7f95515f5700 1 --1- 172.21.15.187:0/4047900519 >> v1:172.21.15.5:6789/0 conn(0x7f954c066f70 0x7f954c0b5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:59530/0 (socket says 172.21.15.187:59530) 2020-11-20T12:05:34.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.415+0000 7f9550df4700 1 --2- 172.21.15.187:0/4047900519 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c1484d0 0x7f954c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.416+0000 7f95437fe700 1 -- 172.21.15.187:0/4047900519 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852928327 0 0) 0x7f954c0b15a0 con 0x7f954c0622c0 2020-11-20T12:05:34.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.416+0000 7f95437fe700 1 -- 172.21.15.187:0/4047900519 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9538003240 con 0x7f954c0622c0 2020-11-20T12:05:34.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.416+0000 7f95437fe700 1 -- 172.21.15.187:0/4047900519 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764822898 0 0) 0x7f954c0b1300 con 0x7f954c066f70 2020-11-20T12:05:34.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.416+0000 7f95437fe700 1 -- 172.21.15.187:0/4047900519 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f954c0b15a0 con 0x7f954c066f70 2020-11-20T12:05:34.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.416+0000 7f9550df4700 1 -- 172.21.15.187:0/4047900519 >> v1:172.21.15.187:6789/0 conn(0x7f954c0622c0 legacy=0x7f954c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.416+0000 7f9550df4700 1 -- 172.21.15.187:0/4047900519 >> v1:172.21.15.5:6789/0 conn(0x7f954c066f70 legacy=0x7f954c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.416+0000 7f9550df4700 1 -- 172.21.15.187:0/4047900519 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c0b1840 con 0x7f954c1484d0 2020-11-20T12:05:34.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.417+0000 7f9550df4700 1 --2- 172.21.15.187:0/4047900519 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c1484d0 0x7f954c061dd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9548009cc0 tx=0x7f95480091d0).ready entity=mon.2 client_cookie=86645f94b5eef3d2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:34.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.417+0000 7f95437fe700 1 -- 172.21.15.187:0/4047900519 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954800ba20 con 0x7f954c1484d0 2020-11-20T12:05:34.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.417+0000 7f95437fe700 1 -- 172.21.15.187:0/4047900519 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954800bb80 con 0x7f954c1484d0 2020-11-20T12:05:34.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.417+0000 7f95437fe700 1 -- 172.21.15.187:0/4047900519 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9548011400 con 0x7f954c1484d0 2020-11-20T12:05:34.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.418+0000 7f9551df6700 1 -- 172.21.15.187:0/4047900519 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c1484d0 msgr2=0x7f954c061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.418+0000 7f9551df6700 1 --2- 172.21.15.187:0/4047900519 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c1484d0 0x7f954c061dd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9548009cc0 tx=0x7f95480091d0).stop 2020-11-20T12:05:34.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.418+0000 7f9551df6700 1 -- 172.21.15.187:0/4047900519 shutdown_connections 2020-11-20T12:05:34.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.418+0000 7f9551df6700 1 --2- 172.21.15.187:0/4047900519 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c1484d0 0x7f954c061dd0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0x7f9548009cc0 tx=0x7f95480091d0).stop 2020-11-20T12:05:34.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.418+0000 7f9551df6700 1 -- 172.21.15.187:0/4047900519 >> 172.21.15.187:0/4047900519 conn(0x7f954c0bab30 msgr2=0x7f954c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:34.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.418+0000 7f9551df6700 1 -- 172.21.15.187:0/4047900519 shutdown_connections 2020-11-20T12:05:34.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.418+0000 7f9551df6700 1 -- 172.21.15.187:0/4047900519 wait complete. 2020-11-20T12:05:34.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.419+0000 7f9551df6700 1 Processor -- start 2020-11-20T12:05:34.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.420+0000 7f9551df6700 1 -- start start 2020-11-20T12:05:34.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.420+0000 7f9551df6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c0622c0 0x7f954c09ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.420+0000 7f9551df6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f954c066f70 0x7f954c157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.420+0000 7f9551df6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f954c157e20 0x7f954c154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.420+0000 7f9551df6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c142d20 con 0x7f954c157e20 2020-11-20T12:05:34.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.420+0000 7f9551df6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0a0730 con 0x7f954c066f70 2020-11-20T12:05:34.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.420+0000 7f9551df6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c09b6a0 con 0x7f954c0622c0 2020-11-20T12:05:34.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f954c066f70 0x7f954c157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f954c066f70 0x7f954c157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41188/0 (socket says 172.21.15.187:41188) 2020-11-20T12:05:34.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 -- 172.21.15.187:0/3303597334 learned_addr learned my addr 172.21.15.187:0/3303597334 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:34.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 -- 172.21.15.187:0/3303597334 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c0622c0 msgr2=0x7f954c09ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c0622c0 0x7f954c09ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:34.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 -- 172.21.15.187:0/3303597334 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f954c157e20 msgr2=0x7f954c154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f954c157e20 0x7f954c154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:34.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 -- 172.21.15.187:0/3303597334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c09bb40 con 0x7f954c066f70 2020-11-20T12:05:34.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.421+0000 7f9543fff700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f954c066f70 0x7f954c157930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f953c00c710 tx=0x7f953c007a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:34.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.422+0000 7f9540ff9700 1 -- 172.21.15.187:0/3303597334 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c007320 con 0x7f954c066f70 2020-11-20T12:05:34.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.422+0000 7f9540ff9700 1 -- 172.21.15.187:0/3303597334 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c007480 con 0x7f954c066f70 2020-11-20T12:05:34.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.422+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954c0a03b0 con 0x7f954c066f70 2020-11-20T12:05:34.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.422+0000 7f9540ff9700 1 -- 172.21.15.187:0/3303597334 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c005800 con 0x7f954c066f70 2020-11-20T12:05:34.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.422+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954c0b35b0 con 0x7f954c066f70 2020-11-20T12:05:34.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.423+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954c09b4e0 con 0x7f954c066f70 2020-11-20T12:05:34.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.423+0000 7f9540ff9700 1 -- 172.21.15.187:0/3303597334 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f953c010030 con 0x7f954c066f70 2020-11-20T12:05:34.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.424+0000 7f9540ff9700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d50 0x7f9524059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.424+0000 7f9540ff9700 1 -- 172.21.15.187:0/3303597334 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f953c06fd50 con 0x7f954c066f70 2020-11-20T12:05:34.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.424+0000 7f9550df4700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d50 0x7f9524059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.425+0000 7f9550df4700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d50 0x7f9524059210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f954c0b85e0 tx=0x7f9548016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:34.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.429+0000 7f9540ff9700 1 -- 172.21.15.187:0/3303597334 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f953c045090 con 0x7f954c066f70 2020-11-20T12:05:34.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.613+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f954c0a05a0 con 0x7f954c066f70 2020-11-20T12:05:34.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.614+0000 7f9540ff9700 1 -- 172.21.15.187:0/3303597334 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f953c0486b0 con 0x7f954c066f70 2020-11-20T12:05:34.618 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:34.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.616+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d50 msgr2=0x7f9524059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d50 0x7f9524059210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f954c0b85e0 tx=0x7f9548016000).stop 2020-11-20T12:05:34.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f954c066f70 msgr2=0x7f954c157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f954c066f70 0x7f954c157930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f953c00c710 tx=0x7f953c007a80).stop 2020-11-20T12:05:34.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 shutdown_connections 2020-11-20T12:05:34.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d50 0x7f9524059210 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f954c157e20 0x7f954c154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:34.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f954c0622c0 0x7f954c09ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:34.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 --2- 172.21.15.187:0/3303597334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f954c066f70 0x7f954c157930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 >> 172.21.15.187:0/3303597334 conn(0x7f954c0bab30 msgr2=0x7f954c143e30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:34.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 shutdown_connections 2020-11-20T12:05:34.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.617+0000 7f9551df6700 1 -- 172.21.15.187:0/3303597334 wait complete. 2020-11-20T12:05:34.644 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:05:34.645 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:34.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.880+0000 7f36a65e2700 1 Processor -- start 2020-11-20T12:05:34.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.880+0000 7f36a65e2700 1 -- start start 2020-11-20T12:05:34.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.882+0000 7f36a65e2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f36a0065190 0x7f36a0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.882+0000 7f36a65e2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 0x7f36a0143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.882+0000 7f36a65e2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a00b13e0 con 0x7f36a0143c70 2020-11-20T12:05:34.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.882+0000 7f36a65e2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a014ab30 con 0x7f36a0065190 2020-11-20T12:05:34.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.882+0000 7f36a65e2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a0062c10 con 0x7f36a0065f90 2020-11-20T12:05:34.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.882+0000 7f36a5de1700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f36a0143c70 0x7f36a009ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37806/0 (socket says 172.21.15.187:37806) 2020-11-20T12:05:34.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.882+0000 7f36a5de1700 1 -- 172.21.15.187:0/75650758 learned_addr learned my addr 172.21.15.187:0/75650758 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:34.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.882+0000 7f36a55e0700 1 --2- 172.21.15.187:0/75650758 >> v2:172.21.15.187:3300/0 conn(0x7f36a0065190 0x7f36a0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.883+0000 7f36a4ddf700 1 --2- 172.21.15.187:0/75650758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 0x7f36a0143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.883+0000 7f368ffff700 1 -- 172.21.15.187:0/75650758 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047119789 0 0) 0x7f36a00b13e0 con 0x7f36a0143c70 2020-11-20T12:05:34.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.883+0000 7f368ffff700 1 -- 172.21.15.187:0/75650758 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3688003240 con 0x7f36a0143c70 2020-11-20T12:05:34.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.883+0000 7f36a55e0700 1 -- 172.21.15.187:0/75650758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 msgr2=0x7f36a0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:34.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.883+0000 7f36a55e0700 1 --2- 172.21.15.187:0/75650758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 0x7f36a0143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.883+0000 7f36a55e0700 1 -- 172.21.15.187:0/75650758 >> v1:172.21.15.187:6789/0 conn(0x7f36a0143c70 legacy=0x7f36a009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.883+0000 7f36a55e0700 1 -- 172.21.15.187:0/75650758 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a00b1640 con 0x7f36a0065190 2020-11-20T12:05:34.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.884+0000 7f36a55e0700 1 --2- 172.21.15.187:0/75650758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0065190 0x7f36a0065570 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f36a014ac70 tx=0x7f369c0091d0).ready entity=mon.1 client_cookie=aaa0d4971f4974dc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:34.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.884+0000 7f368ffff700 1 -- 172.21.15.187:0/75650758 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c00ba20 con 0x7f36a0065190 2020-11-20T12:05:34.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.884+0000 7f368ffff700 1 -- 172.21.15.187:0/75650758 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369c00bb80 con 0x7f36a0065190 2020-11-20T12:05:34.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.884+0000 7f368ffff700 1 -- 172.21.15.187:0/75650758 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c011400 con 0x7f36a0065190 2020-11-20T12:05:34.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.884+0000 7f36a65e2700 1 -- 172.21.15.187:0/75650758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0065190 msgr2=0x7f36a0065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.884+0000 7f36a65e2700 1 --2- 172.21.15.187:0/75650758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0065190 0x7f36a0065570 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f36a014ac70 tx=0x7f369c0091d0).stop 2020-11-20T12:05:34.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.885+0000 7f36a65e2700 1 -- 172.21.15.187:0/75650758 shutdown_connections 2020-11-20T12:05:34.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.885+0000 7f36a65e2700 1 --2- 172.21.15.187:0/75650758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0065190 0x7f36a0065570 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.885+0000 7f36a65e2700 1 --2- 172.21.15.187:0/75650758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 0x7f36a0143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.885+0000 7f36a65e2700 1 -- 172.21.15.187:0/75650758 >> 172.21.15.187:0/75650758 conn(0x7f36a00bab30 msgr2=0x7f36a00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:34.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.885+0000 7f36a65e2700 1 -- 172.21.15.187:0/75650758 shutdown_connections 2020-11-20T12:05:34.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.885+0000 7f36a65e2700 1 -- 172.21.15.187:0/75650758 wait complete. 2020-11-20T12:05:34.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.886+0000 7f36a65e2700 1 Processor -- start 2020-11-20T12:05:34.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.886+0000 7f36a65e2700 1 -- start start 2020-11-20T12:05:34.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.887+0000 7f36a65e2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 0x7f36a015c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.887+0000 7f36a65e2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0143c70 0x7f36a0148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.887+0000 7f36a65e2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36a0148940 0x7f36a009cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.887+0000 7f36a65e2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a015b9d0 con 0x7f36a0148940 2020-11-20T12:05:34.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.887+0000 7f36a65e2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a015bc70 con 0x7f36a0143c70 2020-11-20T12:05:34.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.887+0000 7f36a65e2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a015bf10 con 0x7f36a0065f90 2020-11-20T12:05:34.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.887+0000 7f36a4ddf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0143c70 0x7f36a0148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a4ddf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0143c70 0x7f36a0148f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41204/0 (socket says 172.21.15.187:41204) 2020-11-20T12:05:34.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a4ddf700 1 -- 172.21.15.187:0/1850439087 learned_addr learned my addr 172.21.15.187:0/1850439087 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:34.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a55e0700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 0x7f36a015c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a5de1700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36a0148940 0x7f36a009cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a4ddf700 1 -- 172.21.15.187:0/1850439087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 msgr2=0x7f36a015c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a4ddf700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 0x7f36a015c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a4ddf700 1 -- 172.21.15.187:0/1850439087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36a0148940 msgr2=0x7f36a009cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:34.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a4ddf700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36a0148940 0x7f36a009cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:34.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a4ddf700 1 -- 172.21.15.187:0/1850439087 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0149400 con 0x7f36a0143c70 2020-11-20T12:05:34.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.888+0000 7f36a4ddf700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0143c70 0x7f36a0148f10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f369000eba0 tx=0x7f3690006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:34.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.889+0000 7f368d7fa700 1 -- 172.21.15.187:0/1850439087 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369000ca60 con 0x7f36a0143c70 2020-11-20T12:05:34.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.889+0000 7f368d7fa700 1 -- 172.21.15.187:0/1850439087 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369000cbc0 con 0x7f36a0143c70 2020-11-20T12:05:34.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.889+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a009d460 con 0x7f36a0143c70 2020-11-20T12:05:34.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.889+0000 7f368d7fa700 1 -- 172.21.15.187:0/1850439087 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36900123f0 con 0x7f36a0143c70 2020-11-20T12:05:34.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.889+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a009f940 con 0x7f36a0143c70 2020-11-20T12:05:34.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.890+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a015c0d0 con 0x7f36a0143c70 2020-11-20T12:05:34.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.891+0000 7f368d7fa700 1 -- 172.21.15.187:0/1850439087 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f36900127a0 con 0x7f36a0143c70 2020-11-20T12:05:34.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.892+0000 7f368d7fa700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3678056d20 0x7f36780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:34.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.892+0000 7f368d7fa700 1 -- 172.21.15.187:0/1850439087 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3690070450 con 0x7f36a0143c70 2020-11-20T12:05:34.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.892+0000 7f36a55e0700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3678056d20 0x7f36780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:34.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.893+0000 7f36a55e0700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3678056d20 0x7f36780591e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f36a00632e0 tx=0x7f369c016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:34.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:34.895+0000 7f368d7fa700 1 -- 172.21.15.187:0/1850439087 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f36900458a0 con 0x7f36a0143c70 2020-11-20T12:05:35.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.143+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36a0061a20 con 0x7f36a0143c70 2020-11-20T12:05:35.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.144+0000 7f368d7fa700 1 -- 172.21.15.187:0/1850439087 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2130 (secure 0 0 0) 0x7f3690048ec0 con 0x7f36a0143c70 2020-11-20T12:05:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:35.148 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4211,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6833/2094645874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2094645874},{"type":"v1","addr":"172.21.15.5:6833","nonce":2094645874}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:57.868606+0000","modified":"2020-11-20T12:04:59.990344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3094884769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3094884769},{"type":"v1","addr":"172.21.15.187:6835","nonce":3094884769}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:35.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3678056d20 msgr2=0x7f36780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:35.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3678056d20 0x7f36780591e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f36a00632e0 tx=0x7f369c016000).stop 2020-11-20T12:05:35.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0143c70 msgr2=0x7f36a0148f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:35.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0143c70 0x7f36a0148f10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f369000eba0 tx=0x7f3690006df0).stop 2020-11-20T12:05:35.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 shutdown_connections 2020-11-20T12:05:35.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3678056d20 0x7f36780591e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:35.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36a0148940 0x7f36a009cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:35.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36a0065f90 0x7f36a015c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:35.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 --2- 172.21.15.187:0/1850439087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36a0143c70 0x7f36a0148f10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:35.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.147+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 >> 172.21.15.187:0/1850439087 conn(0x7f36a00bab30 msgr2=0x7f36a00b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:35.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.148+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 shutdown_connections 2020-11-20T12:05:35.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:35.148+0000 7f36a65e2700 1 -- 172.21.15.187:0/1850439087 wait complete. 2020-11-20T12:05:35.153 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 6 2020-11-20T12:05:35.379 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:04:57.868606+0000', 'modified': '2020-11-20T12:04:59.990344+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4264': {'gid': 4264, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3094884769', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3094884769}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3094884769}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:05:35.379 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:05:35.380 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:05:35.680 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":4264,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":37.780611606000001,"mdsmap_epoch":6,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":42.169434787999997} 2020-11-20T12:05:35.680 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 4264, "mdsmap_epoch": 6, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 37.780611606, "state": "up:active", "uptime": 42.169434788, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:05:35.681 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:05:35.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:35.681 INFO:teuthology.orchestra.run.smithi005:> mkdir -p /home/ubuntu/cephtest/mnt.2 2020-11-20T12:05:35.750 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:05:35.751 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,name=2,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:35.924 INFO:teuthology.orchestra.run.smithi005.stdout:parsing options: rw,norequire_active_mds,name=2,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:35.925 INFO:teuthology.orchestra.run.smithi005.stdout:mount.ceph: options "norequire_active_mds,name=2,norbytes" will pass to kernel. 2020-11-20T12:05:36.519 INFO:teuthology.orchestra.run.smithi005.stdout:parsing options: rw,norequire_active_mds,name=2,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:36.520 INFO:teuthology.orchestra.run.smithi005.stdout:mount.ceph: options "norequire_active_mds,name=2,norbytes" will pass to kernel. 2020-11-20T12:05:36.528 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:05:36.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:36.529 INFO:teuthology.orchestra.run.smithi005:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2020-11-20T12:05:36.640 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:05:36.641 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi005.front.sepia.ceph.com 2020-11-20T12:05:36.641 INFO:tasks.cephfs.mount:self.client.name = client.3 2020-11-20T12:05:36.642 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2020-11-20T12:05:36.642 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:05:36.642 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2020-11-20T12:05:36.642 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:05:36.643 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2020-11-20T12:05:36.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:36.643 INFO:teuthology.orchestra.run.smithi005:> ip addr 2020-11-20T12:05:36.701 INFO:teuthology.orchestra.run.smithi005.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:36.701 INFO:teuthology.orchestra.run.smithi005.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:36.702 INFO:teuthology.orchestra.run.smithi005.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:36.702 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:36.702 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:36.703 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:36.703 INFO:teuthology.orchestra.run.smithi005.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:36.703 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:6b:fd:de brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:36.703 INFO:teuthology.orchestra.run.smithi005.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:36.704 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:6b:fd:df brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:36.704 INFO:teuthology.orchestra.run.smithi005.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:05:36.704 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:bd:14:42 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:36.704 INFO:teuthology.orchestra.run.smithi005.stdout: inet 172.21.15.5/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2020-11-20T12:05:36.705 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft 42769sec preferred_lft 42769sec 2020-11-20T12:05:36.705 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 fe80::ec4:7aff:febd:1442/64 scope link 2020-11-20T12:05:36.705 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:36.705 INFO:teuthology.orchestra.run.smithi005.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:05:36.705 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:bd:14:43 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:36.706 INFO:teuthology.orchestra.run.smithi005.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:05:36.706 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether aa:62:8a:07:d7:26 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:36.707 INFO:teuthology.orchestra.run.smithi005.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:05:36.707 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:36.707 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 fe80::e06c:a6ff:fe61:ad3e/64 scope link 2020-11-20T12:05:36.708 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:36.708 INFO:teuthology.orchestra.run.smithi005.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:05:36.708 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether aa:62:8a:07:d7:26 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2020-11-20T12:05:36.708 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 fe80::a862:8aff:fe07:d726/64 scope link 2020-11-20T12:05:36.709 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:36.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:36.713 INFO:teuthology.orchestra.run.smithi005:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:05:36.823 INFO:teuthology.orchestra.run.smithi005.stdout:1 2020-11-20T12:05:36.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:36.831 INFO:teuthology.orchestra.run.smithi005:> route 2020-11-20T12:05:36.896 INFO:teuthology.orchestra.run.smithi005.stdout:Kernel IP routing table 2020-11-20T12:05:36.897 INFO:teuthology.orchestra.run.smithi005.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:05:36.897 INFO:teuthology.orchestra.run.smithi005.stdout:default _gateway 0.0.0.0 UG 102 0 0 ens1f0 2020-11-20T12:05:36.897 INFO:teuthology.orchestra.run.smithi005.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 102 0 0 ens1f0 2020-11-20T12:05:36.898 INFO:teuthology.orchestra.run.smithi005.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:05:36.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:36.899 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:36.901 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:36.901 INFO:teuthology.orchestra.run.smithi005:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2020-11-20T12:05:36.902 INFO:teuthology.orchestra.run.smithi005:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2020-11-20T12:05:36.902 INFO:teuthology.orchestra.run.smithi005:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2020-11-20T12:05:36.903 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:37.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:37.182 INFO:teuthology.orchestra.run.smithi005:> ip netns list 2020-11-20T12:05:37.234 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 0) 2020-11-20T12:05:37.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:37.236 INFO:teuthology.orchestra.run.smithi005:> ip netns list-id 2020-11-20T12:05:37.291 INFO:teuthology.orchestra.run.smithi005.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.2) 2020-11-20T12:05:37.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:37.293 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:37.293 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:37.294 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2020-11-20T12:05:37.294 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2020-11-20T12:05:37.294 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:37.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:37.521 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr 2020-11-20T12:05:37.624 INFO:teuthology.orchestra.run.smithi005.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:37.624 INFO:teuthology.orchestra.run.smithi005.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:37.624 INFO:teuthology.orchestra.run.smithi005.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:37.625 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:37.625 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:37.625 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:37.625 INFO:teuthology.orchestra.run.smithi005.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:05:37.625 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether f2:7b:86:03:48:9c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2020-11-20T12:05:37.626 INFO:teuthology.orchestra.run.smithi005.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2020-11-20T12:05:37.626 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:37.626 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 fe80::f07b:86ff:fe03:489c/64 scope link 2020-11-20T12:05:37.626 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:37.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:37.628 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr 2020-11-20T12:05:37.733 INFO:teuthology.orchestra.run.smithi005.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:37.733 INFO:teuthology.orchestra.run.smithi005.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:37.733 INFO:teuthology.orchestra.run.smithi005.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:37.733 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:37.734 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:37.734 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:37.734 INFO:teuthology.orchestra.run.smithi005.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:05:37.734 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether f2:7b:86:03:48:9c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2020-11-20T12:05:37.734 INFO:teuthology.orchestra.run.smithi005.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2020-11-20T12:05:37.735 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:37.735 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 fe80::f07b:86ff:fe03:489c/64 scope link 2020-11-20T12:05:37.735 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:37.737 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2020-11-20T12:05:37.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:37.738 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:37.738 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:37.738 INFO:teuthology.orchestra.run.smithi005:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2020-11-20T12:05:37.739 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2020-11-20T12:05:37.739 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2020-11-20T12:05:37.739 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2020-11-20T12:05:37.739 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2020-11-20T12:05:37.740 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:38.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:38.136 INFO:teuthology.orchestra.run.smithi005:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:38.136 INFO:teuthology.orchestra.run.smithi005:> set -e 2020-11-20T12:05:38.137 INFO:teuthology.orchestra.run.smithi005:> sudo ip link set brx.1 up 2020-11-20T12:05:38.137 INFO:teuthology.orchestra.run.smithi005:> sudo ip link set dev brx.1 master ceph-brx 2020-11-20T12:05:38.137 INFO:teuthology.orchestra.run.smithi005:> ') 2020-11-20T12:05:38.386 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:38.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.632+0000 7fbeeffff700 1 Processor -- start 2020-11-20T12:05:38.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.632+0000 7fbeeffff700 1 -- start start 2020-11-20T12:05:38.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.633+0000 7fbeeffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef01484d0 0x7fbef0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:38.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.634+0000 7fbeeffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbef00622c0 0x7fbef00b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:38.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.634+0000 7fbeeffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef00b1300 con 0x7fbef0066f70 2020-11-20T12:05:38.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.634+0000 7fbeeffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef0067730 con 0x7fbef00622c0 2020-11-20T12:05:38.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.634+0000 7fbeeffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00b15a0 con 0x7fbef01484d0 2020-11-20T12:05:38.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.634+0000 7fbee67fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbef00622c0 0x7fbef00b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:38.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.634+0000 7fbee67fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbef00622c0 0x7fbef00b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41212/0 (socket says 172.21.15.187:41212) 2020-11-20T12:05:38.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.634+0000 7fbee67fc700 1 -- 172.21.15.187:0/1756824998 learned_addr learned my addr 172.21.15.187:0/1756824998 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:38.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.634+0000 7fbeef7fe700 1 --1- 172.21.15.187:0/1756824998 >> v1:172.21.15.187:6789/0 conn(0x7fbef0066f70 0x7fbef00b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37820/0 (socket says 172.21.15.187:37820) 2020-11-20T12:05:38.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbee67fc700 1 -- 172.21.15.187:0/1756824998 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef01484d0 msgr2=0x7fbef0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:38.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbeeeffd700 1 --2- 172.21.15.187:0/1756824998 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef01484d0 0x7fbef0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:38.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbee67fc700 1 --2- 172.21.15.187:0/1756824998 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef01484d0 0x7fbef0061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:38.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbee67fc700 1 -- 172.21.15.187:0/1756824998 >> v1:172.21.15.187:6789/0 conn(0x7fbef0066f70 legacy=0x7fbef00b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:38.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbee67fc700 1 -- 172.21.15.187:0/1756824998 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00b1840 con 0x7fbef00622c0 2020-11-20T12:05:38.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbeeeffd700 1 --2- 172.21.15.187:0/1756824998 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef01484d0 0x7fbef0061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:38.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbee67fc700 1 --2- 172.21.15.187:0/1756824998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef00622c0 0x7fbef00b5640 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbed8006fd0 tx=0x7fbed8006eb0).ready entity=mon.1 client_cookie=2d8561586972df server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:38.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbeee7fc700 1 -- 172.21.15.187:0/1756824998 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed80098d0 con 0x7fbef00622c0 2020-11-20T12:05:38.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbeee7fc700 1 -- 172.21.15.187:0/1756824998 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed8009a30 con 0x7fbef00622c0 2020-11-20T12:05:38.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.635+0000 7fbeee7fc700 1 -- 172.21.15.187:0/1756824998 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8009db0 con 0x7fbef00622c0 2020-11-20T12:05:38.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.636+0000 7fbeeffff700 1 -- 172.21.15.187:0/1756824998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef00622c0 msgr2=0x7fbef00b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:38.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.636+0000 7fbeeffff700 1 --2- 172.21.15.187:0/1756824998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef00622c0 0x7fbef00b5640 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbed8006fd0 tx=0x7fbed8006eb0).stop 2020-11-20T12:05:38.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.636+0000 7fbeeffff700 1 -- 172.21.15.187:0/1756824998 shutdown_connections 2020-11-20T12:05:38.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.636+0000 7fbeeffff700 1 --2- 172.21.15.187:0/1756824998 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef01484d0 0x7fbef0061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:38.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.636+0000 7fbeeffff700 1 --2- 172.21.15.187:0/1756824998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef00622c0 0x7fbef00b5640 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:38.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.636+0000 7fbeeffff700 1 -- 172.21.15.187:0/1756824998 >> 172.21.15.187:0/1756824998 conn(0x7fbef00bab30 msgr2=0x7fbef00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:38.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.636+0000 7fbeeffff700 1 -- 172.21.15.187:0/1756824998 shutdown_connections 2020-11-20T12:05:38.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.636+0000 7fbeeffff700 1 -- 172.21.15.187:0/1756824998 wait complete. 2020-11-20T12:05:38.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.638+0000 7fbeeffff700 1 Processor -- start 2020-11-20T12:05:38.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.638+0000 7fbeeffff700 1 -- start start 2020-11-20T12:05:38.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef0066f70 0x7fbef009d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:38.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbef01484d0 0x7fbef00b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:38.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef009c9b0 0x7fbef015bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:38.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef015cae0 con 0x7fbef01484d0 2020-11-20T12:05:38.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef015c5a0 con 0x7fbef0066f70 2020-11-20T12:05:38.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef015c840 con 0x7fbef009c9b0 2020-11-20T12:05:38.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeeffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef0066f70 0x7fbef009d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:38.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeeffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef0066f70 0x7fbef009d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41216/0 (socket says 172.21.15.187:41216) 2020-11-20T12:05:38.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeeffd700 1 -- 172.21.15.187:0/4167535854 learned_addr learned my addr 172.21.15.187:0/4167535854 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:38.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeeffd700 1 -- 172.21.15.187:0/4167535854 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef009c9b0 msgr2=0x7fbef015bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:38.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeeffd700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef009c9b0 0x7fbef015bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:38.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeeffd700 1 -- 172.21.15.187:0/4167535854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbef01484d0 msgr2=0x7fbef00b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:38.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeeffd700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbef01484d0 0x7fbef00b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:38.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.639+0000 7fbeeeffd700 1 -- 172.21.15.187:0/4167535854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef015c230 con 0x7fbef0066f70 2020-11-20T12:05:38.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.640+0000 7fbeeeffd700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef0066f70 0x7fbef009d240 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbee000c8f0 tx=0x7fbee0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:38.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.640+0000 7fbee77fe700 1 -- 172.21.15.187:0/4167535854 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee00077a0 con 0x7fbef0066f70 2020-11-20T12:05:38.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.640+0000 7fbee77fe700 1 -- 172.21.15.187:0/4167535854 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee0007900 con 0x7fbef0066f70 2020-11-20T12:05:38.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.640+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef00b3410 con 0x7fbef0066f70 2020-11-20T12:05:38.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.640+0000 7fbee77fe700 1 -- 172.21.15.187:0/4167535854 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee0007c80 con 0x7fbef0066f70 2020-11-20T12:05:38.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.640+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef00b3920 con 0x7fbef0066f70 2020-11-20T12:05:38.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.641+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef0061a20 con 0x7fbef0066f70 2020-11-20T12:05:38.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.642+0000 7fbee77fe700 1 -- 172.21.15.187:0/4167535854 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbee0011620 con 0x7fbef0066f70 2020-11-20T12:05:38.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.643+0000 7fbee77fe700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbecc056d20 0x7fbecc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:38.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.643+0000 7fbee77fe700 1 -- 172.21.15.187:0/4167535854 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbee00702d0 con 0x7fbef0066f70 2020-11-20T12:05:38.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.643+0000 7fbee67fc700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbecc056d20 0x7fbecc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:38.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.644+0000 7fbee67fc700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbecc056d20 0x7fbecc0591e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fbef00637d0 tx=0x7fbed8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:38.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.646+0000 7fbee77fe700 1 -- 172.21.15.187:0/4167535854 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbee0045640 con 0x7fbef0066f70 2020-11-20T12:05:38.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.831+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbef009cf10 con 0x7fbef0066f70 2020-11-20T12:05:38.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.833+0000 7fbee77fe700 1 -- 172.21.15.187:0/4167535854 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7fbee0048c60 con 0x7fbef0066f70 2020-11-20T12:05:38.837 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:38.838 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:38.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbecc056d20 msgr2=0x7fbecc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:38.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbecc056d20 0x7fbecc0591e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fbef00637d0 tx=0x7fbed8015000).stop 2020-11-20T12:05:38.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef0066f70 msgr2=0x7fbef009d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:38.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef0066f70 0x7fbef009d240 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbee000c8f0 tx=0x7fbee0008040).stop 2020-11-20T12:05:38.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 shutdown_connections 2020-11-20T12:05:38.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbecc056d20 0x7fbecc0591e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:38.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbef009c9b0 0x7fbef015bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:38.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbef0066f70 0x7fbef009d240 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:38.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 --2- 172.21.15.187:0/4167535854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbef01484d0 0x7fbef00b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:38.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 >> 172.21.15.187:0/4167535854 conn(0x7fbef00bab30 msgr2=0x7fbef00b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:38.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 shutdown_connections 2020-11-20T12:05:38.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:38.836+0000 7fbeeffff700 1 -- 172.21.15.187:0/4167535854 wait complete. 2020-11-20T12:05:38.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:38.864 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:39.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.109+0000 7f5a61ab9700 1 Processor -- start 2020-11-20T12:05:39.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.109+0000 7f5a61ab9700 1 -- start start 2020-11-20T12:05:39.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a61ab9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5a5c0bfbe0 0x7f5a5c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a61ab9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 0x7f5a5c1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a61ab9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c14d110 con 0x7f5a5c0bf800 2020-11-20T12:05:39.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a61ab9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5c14c9d0 con 0x7f5a5c0bfbe0 2020-11-20T12:05:39.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a61ab9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c148aa0 con 0x7f5a5c09f560 2020-11-20T12:05:39.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a5bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5a5c0bfbe0 0x7f5a5c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a5bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5a5c0bfbe0 0x7f5a5c1480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41226/0 (socket says 172.21.15.187:41226) 2020-11-20T12:05:39.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a5bfff700 1 -- 172.21.15.187:0/2101722428 learned_addr learned my addr 172.21.15.187:0/2101722428 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:39.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.111+0000 7f5a612b8700 1 --2- 172.21.15.187:0/2101722428 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 0x7f5a5c1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5bfff700 1 -- 172.21.15.187:0/2101722428 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 msgr2=0x7f5a5c1485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:39.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5bfff700 1 --2- 172.21.15.187:0/2101722428 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 0x7f5a5c1485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5b7fe700 1 -- 172.21.15.187:0/2101722428 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765251740 0 0) 0x7f5a5c14d110 con 0x7f5a5c0bf800 2020-11-20T12:05:39.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5bfff700 1 -- 172.21.15.187:0/2101722428 >> v1:172.21.15.187:6789/0 conn(0x7f5a5c0bf800 legacy=0x7f5a5c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5bfff700 1 -- 172.21.15.187:0/2101722428 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c14cbd0 con 0x7f5a5c0bfbe0 2020-11-20T12:05:39.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5bfff700 1 --2- 172.21.15.187:0/2101722428 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c0bfbe0 0x7f5a5c1480c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5a4c006fd0 tx=0x7f5a4c006eb0).ready entity=mon.1 client_cookie=142d05a95ee9b5d8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:39.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5b7fe700 1 -- 172.21.15.187:0/2101722428 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c009960 con 0x7f5a5c0bfbe0 2020-11-20T12:05:39.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5b7fe700 1 -- 172.21.15.187:0/2101722428 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4c009ac0 con 0x7f5a5c0bfbe0 2020-11-20T12:05:39.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.112+0000 7f5a5b7fe700 1 -- 172.21.15.187:0/2101722428 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c009e40 con 0x7f5a5c0bfbe0 2020-11-20T12:05:39.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.113+0000 7f5a61ab9700 1 -- 172.21.15.187:0/2101722428 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c0bfbe0 msgr2=0x7f5a5c1480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.113+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/2101722428 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c0bfbe0 0x7f5a5c1480c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5a4c006fd0 tx=0x7f5a4c006eb0).stop 2020-11-20T12:05:39.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.113+0000 7f5a61ab9700 1 -- 172.21.15.187:0/2101722428 shutdown_connections 2020-11-20T12:05:39.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.113+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/2101722428 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 0x7f5a5c1485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.113+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/2101722428 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c0bfbe0 0x7f5a5c1480c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.113+0000 7f5a61ab9700 1 -- 172.21.15.187:0/2101722428 >> 172.21.15.187:0/2101722428 conn(0x7f5a5c068370 msgr2=0x7f5a5c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.113+0000 7f5a61ab9700 1 -- 172.21.15.187:0/2101722428 shutdown_connections 2020-11-20T12:05:39.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.113+0000 7f5a61ab9700 1 -- 172.21.15.187:0/2101722428 wait complete. 2020-11-20T12:05:39.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.115+0000 7f5a61ab9700 1 Processor -- start 2020-11-20T12:05:39.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.115+0000 7f5a61ab9700 1 -- start start 2020-11-20T12:05:39.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a61ab9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 0x7f5a5c1431c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a61ab9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c0bf800 0x7f5a5c15bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a61ab9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c1436b0 0x7f5a5c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a61ab9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c0b6fe0 con 0x7f5a5c0bf800 2020-11-20T12:05:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a61ab9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c0b8db0 con 0x7f5a5c1436b0 2020-11-20T12:05:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a61ab9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c0b9050 con 0x7f5a5c09f560 2020-11-20T12:05:39.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a612b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c1436b0 0x7f5a5c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a612b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c1436b0 0x7f5a5c15c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41234/0 (socket says 172.21.15.187:41234) 2020-11-20T12:05:39.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a612b8700 1 -- 172.21.15.187:0/229032484 learned_addr learned my addr 172.21.15.187:0/229032484 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:39.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a60ab7700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 0x7f5a5c1431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.116+0000 7f5a5bfff700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c0bf800 0x7f5a5c15bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a612b8700 1 -- 172.21.15.187:0/229032484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 msgr2=0x7f5a5c1431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a612b8700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 0x7f5a5c1431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a612b8700 1 -- 172.21.15.187:0/229032484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c0bf800 msgr2=0x7f5a5c15bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a612b8700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c0bf800 0x7f5a5c15bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a612b8700 1 -- 172.21.15.187:0/229032484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c0b92f0 con 0x7f5a5c1436b0 2020-11-20T12:05:39.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a612b8700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c1436b0 0x7f5a5c15c2a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5a5400eb70 tx=0x7f5a54008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:39.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a58ff9700 1 -- 172.21.15.187:0/229032484 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5400c7a0 con 0x7f5a5c1436b0 2020-11-20T12:05:39.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a58ff9700 1 -- 172.21.15.187:0/229032484 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5400c900 con 0x7f5a5c1436b0 2020-11-20T12:05:39.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c0b65f0 con 0x7f5a5c1436b0 2020-11-20T12:05:39.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a58ff9700 1 -- 172.21.15.187:0/229032484 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5400cca0 con 0x7f5a5c1436b0 2020-11-20T12:05:39.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.117+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c15c810 con 0x7f5a5c1436b0 2020-11-20T12:05:39.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.119+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5c0a17d0 con 0x7f5a5c1436b0 2020-11-20T12:05:39.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.119+0000 7f5a58ff9700 1 -- 172.21.15.187:0/229032484 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5a54013640 con 0x7f5a5c1436b0 2020-11-20T12:05:39.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.120+0000 7f5a58ff9700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d50 0x7f5a34059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.120+0000 7f5a58ff9700 1 -- 172.21.15.187:0/229032484 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5a54070190 con 0x7f5a5c1436b0 2020-11-20T12:05:39.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.120+0000 7f5a60ab7700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d50 0x7f5a34059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.121+0000 7f5a60ab7700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d50 0x7f5a34059210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5a500034b0 tx=0x7f5a50009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:39.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.124+0000 7f5a58ff9700 1 -- 172.21.15.187:0/229032484 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5a540454d0 con 0x7f5a5c1436b0 2020-11-20T12:05:39.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.310+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5a5c080050 con 0x7f5a5c1436b0 2020-11-20T12:05:39.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.311+0000 7f5a58ff9700 1 -- 172.21.15.187:0/229032484 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5a54048af0 con 0x7f5a5c1436b0 2020-11-20T12:05:39.315 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:39.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d50 msgr2=0x7f5a34059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d50 0x7f5a34059210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5a500034b0 tx=0x7f5a50009040).stop 2020-11-20T12:05:39.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c1436b0 msgr2=0x7f5a5c15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c1436b0 0x7f5a5c15c2a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5a5400eb70 tx=0x7f5a54008040).stop 2020-11-20T12:05:39.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 shutdown_connections 2020-11-20T12:05:39.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a34056d50 0x7f5a34059210 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a5c1436b0 0x7f5a5c15c2a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a5c09f560 0x7f5a5c1431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 --2- 172.21.15.187:0/229032484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a5c0bf800 0x7f5a5c15bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 >> 172.21.15.187:0/229032484 conn(0x7f5a5c068370 msgr2=0x7f5a5c0bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:39.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 shutdown_connections 2020-11-20T12:05:39.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.314+0000 7f5a61ab9700 1 -- 172.21.15.187:0/229032484 wait complete. 2020-11-20T12:05:39.339 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:39.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.579+0000 7f06e5209700 1 Processor -- start 2020-11-20T12:05:39.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.579+0000 7f06e5209700 1 -- start start 2020-11-20T12:05:39.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.580+0000 7f06e5209700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e0065190 0x7f06e0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.580+0000 7f06e5209700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e00b13e0 con 0x7f06e0065f90 2020-11-20T12:05:39.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.580+0000 7f06e5209700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e0062c10 con 0x7f06e009ca90 2020-11-20T12:05:39.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.580+0000 7f06e5209700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e014ab30 con 0x7f06e0065190 2020-11-20T12:05:39.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.581+0000 7f06e4a08700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f06e009ca90 0x7f06e009cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37846/0 (socket says 172.21.15.187:37846) 2020-11-20T12:05:39.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.581+0000 7f06e4a08700 1 -- 172.21.15.187:0/709099177 learned_addr learned my addr 172.21.15.187:0/709099177 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:39.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.581+0000 7f06dffff700 1 --2- 172.21.15.187:0/709099177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e0065190 0x7f06e0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.582+0000 7f06df7fe700 1 -- 172.21.15.187:0/709099177 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813937549 0 0) 0x7f06e0062c10 con 0x7f06e009ca90 2020-11-20T12:05:39.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.582+0000 7f06df7fe700 1 -- 172.21.15.187:0/709099177 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06c0003240 con 0x7f06e009ca90 2020-11-20T12:05:39.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.582+0000 7f06df7fe700 1 -- 172.21.15.187:0/709099177 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1878869152 0 0) 0x7f06c0003240 con 0x7f06e009ca90 2020-11-20T12:05:39.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.582+0000 7f06dffff700 1 -- 172.21.15.187:0/709099177 >> v1:172.21.15.187:6789/0 conn(0x7f06e009ca90 legacy=0x7f06e009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.582+0000 7f06dffff700 1 -- 172.21.15.187:0/709099177 >> v1:172.21.15.5:6789/0 conn(0x7f06e0065f90 legacy=0x7f06e0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.582+0000 7f06dffff700 1 -- 172.21.15.187:0/709099177 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e00b1640 con 0x7f06e0065190 2020-11-20T12:05:39.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.583+0000 7f06dffff700 1 --2- 172.21.15.187:0/709099177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e0065190 0x7f06e0065570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f06d000b950 tx=0x7f06d0006df0).ready entity=mon.2 client_cookie=74429a9a305f7fa9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:39.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.583+0000 7f06df7fe700 1 -- 172.21.15.187:0/709099177 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8001860 con 0x7f06e0065190 2020-11-20T12:05:39.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06df7fe700 1 -- 172.21.15.187:0/709099177 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d00094c0 con 0x7f06e0065190 2020-11-20T12:05:39.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06df7fe700 1 -- 172.21.15.187:0/709099177 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d0009840 con 0x7f06e0065190 2020-11-20T12:05:39.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06e5209700 1 -- 172.21.15.187:0/709099177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e0065190 msgr2=0x7f06e0065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06e5209700 1 --2- 172.21.15.187:0/709099177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e0065190 0x7f06e0065570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f06d000b950 tx=0x7f06d0006df0).stop 2020-11-20T12:05:39.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06e5209700 1 -- 172.21.15.187:0/709099177 shutdown_connections 2020-11-20T12:05:39.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06e5209700 1 --2- 172.21.15.187:0/709099177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e0065190 0x7f06e0065570 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06e5209700 1 -- 172.21.15.187:0/709099177 >> 172.21.15.187:0/709099177 conn(0x7f06e00bab30 msgr2=0x7f06e00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:39.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06e5209700 1 -- 172.21.15.187:0/709099177 shutdown_connections 2020-11-20T12:05:39.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.584+0000 7f06e5209700 1 -- 172.21.15.187:0/709099177 wait complete. 2020-11-20T12:05:39.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.586+0000 7f06e5209700 1 Processor -- start 2020-11-20T12:05:39.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.586+0000 7f06e5209700 1 -- start start 2020-11-20T12:05:39.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.587+0000 7f06e5209700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e0065190 0x7f06e00991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.587+0000 7f06e5209700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0065f90 0x7f06e0147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.587+0000 7f06e5209700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e009ca90 0x7f06e0149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.587+0000 7f06e5209700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e00b1b40 con 0x7f06e0065190 2020-11-20T12:05:39.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.587+0000 7f06e5209700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e00b4110 con 0x7f06e0065f90 2020-11-20T12:05:39.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.587+0000 7f06e5209700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e0147e50 con 0x7f06e009ca90 2020-11-20T12:05:39.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.587+0000 7f06d77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0065f90 0x7f06e0147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.587+0000 7f06dffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e0065190 0x7f06e00991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06d77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0065f90 0x7f06e0147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41248/0 (socket says 172.21.15.187:41248) 2020-11-20T12:05:39.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06d77fe700 1 -- 172.21.15.187:0/751002045 learned_addr learned my addr 172.21.15.187:0/751002045 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:39.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06e4a08700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e009ca90 0x7f06e0149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06d77fe700 1 -- 172.21.15.187:0/751002045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e009ca90 msgr2=0x7f06e0149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06d77fe700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e009ca90 0x7f06e0149a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06d77fe700 1 -- 172.21.15.187:0/751002045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e0065190 msgr2=0x7f06e00991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06d77fe700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e0065190 0x7f06e00991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06d77fe700 1 -- 172.21.15.187:0/751002045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e00b1d70 con 0x7f06e0065f90 2020-11-20T12:05:39.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.588+0000 7f06d77fe700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0065f90 0x7f06e0147890 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f06c800c710 tx=0x7f06c8007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:39.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.589+0000 7f06dcff9700 1 -- 172.21.15.187:0/751002045 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8007020 con 0x7f06e0065f90 2020-11-20T12:05:39.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.589+0000 7f06dcff9700 1 -- 172.21.15.187:0/751002045 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c8007180 con 0x7f06e0065f90 2020-11-20T12:05:39.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.589+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e0149f30 con 0x7f06e0065f90 2020-11-20T12:05:39.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.589+0000 7f06dcff9700 1 -- 172.21.15.187:0/751002045 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c800e560 con 0x7f06e0065f90 2020-11-20T12:05:39.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.589+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e009acb0 con 0x7f06e0065f90 2020-11-20T12:05:39.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.591+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e0148010 con 0x7f06e0065f90 2020-11-20T12:05:39.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.591+0000 7f06dcff9700 1 -- 172.21.15.187:0/751002045 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f06c8010030 con 0x7f06e0065f90 2020-11-20T12:05:39.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.591+0000 7f06dcff9700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f06bc056d20 0x7f06bc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:39.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.592+0000 7f06dcff9700 1 -- 172.21.15.187:0/751002045 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f06c806f720 con 0x7f06e0065f90 2020-11-20T12:05:39.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.592+0000 7f06dffff700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f06bc056d20 0x7f06bc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:39.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.592+0000 7f06dffff700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f06bc056d20 0x7f06bc0591e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f06d000b150 tx=0x7f06d0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:39.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.595+0000 7f06dcff9700 1 -- 172.21.15.187:0/751002045 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f06c8044b10 con 0x7f06e0065f90 2020-11-20T12:05:39.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.780+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f06e00b1f00 con 0x7f06e0065f90 2020-11-20T12:05:39.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.781+0000 7f06dcff9700 1 -- 172.21.15.187:0/751002045 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f06c8048130 con 0x7f06e0065f90 2020-11-20T12:05:39.784 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:39.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f06bc056d20 msgr2=0x7f06bc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f06bc056d20 0x7f06bc0591e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f06d000b150 tx=0x7f06d0015000).stop 2020-11-20T12:05:39.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0065f90 msgr2=0x7f06e0147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0065f90 0x7f06e0147890 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f06c800c710 tx=0x7f06c8007c20).stop 2020-11-20T12:05:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 shutdown_connections 2020-11-20T12:05:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f06bc056d20 0x7f06bc0591e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e0065190 0x7f06e00991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06e009ca90 0x7f06e0149a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 --2- 172.21.15.187:0/751002045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0065f90 0x7f06e0147890 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.783+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 >> 172.21.15.187:0/751002045 conn(0x7f06e00bab30 msgr2=0x7f06e00aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.784+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 shutdown_connections 2020-11-20T12:05:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:39.784+0000 7f06e5209700 1 -- 172.21.15.187:0/751002045 wait complete. 2020-11-20T12:05:39.807 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:40.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.046+0000 7f8104812700 1 Processor -- start 2020-11-20T12:05:40.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.046+0000 7f8104812700 1 -- start start 2020-11-20T12:05:40.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.048+0000 7f8104812700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f81000a4bf0 0x7f81000a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.048+0000 7f8104812700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8100149930 con 0x7f81000bbaa0 2020-11-20T12:05:40.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.048+0000 7f8104812700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000a3560 con 0x7f81000bbe80 2020-11-20T12:05:40.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.048+0000 7f8104812700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810014a330 con 0x7f81000a4bf0 2020-11-20T12:05:40.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.048+0000 7f80f659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f81000bbe80 0x7f81001426f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37860/0 (socket says 172.21.15.187:37860) 2020-11-20T12:05:40.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.048+0000 7f80f659c700 1 -- 172.21.15.187:0/2994888818 learned_addr learned my addr 172.21.15.187:0/2994888818 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:40.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80ff59e700 1 --2- 172.21.15.187:0/2994888818 >> v2:172.21.15.5:3300/0 conn(0x7f81000a4bf0 0x7f81000a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50201662 0 0) 0x7f81000a3560 con 0x7f81000bbe80 2020-11-20T12:05:40.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e0003240 con 0x7f81000bbe80 2020-11-20T12:05:40.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4177684141 0 0) 0x7f80e0003240 con 0x7f81000bbe80 2020-11-20T12:05:40.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 >> v2:172.21.15.5:3300/0 conn(0x7f81000a4bf0 msgr2=0x7f81000a3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:40.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 --2- 172.21.15.187:0/2994888818 >> v2:172.21.15.5:3300/0 conn(0x7f81000a4bf0 0x7f81000a3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 >> v1:172.21.15.5:6789/0 conn(0x7f81000bbaa0 legacy=0x7f8100149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000a3250 con 0x7f81000bbe80 2020-11-20T12:05:40.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f80e80014e0 con 0x7f81000bbe80 2020-11-20T12:05:40.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.049+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 >> v1:172.21.15.187:6789/0 conn(0x7f81000bbe80 legacy=0x7f81001426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80fe59c700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80e0003730 0x7f80e0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80fe59c700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80e00044e0 0x7f80e0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80fe59c700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80e0004e50 0x7f80e000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810014a330 con 0x7f80e00044e0 2020-11-20T12:05:40.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e0004260 con 0x7f80e0003730 2020-11-20T12:05:40.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e000bc20 con 0x7f80e0004e50 2020-11-20T12:05:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80fed9d700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80e0003730 0x7f80e0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80f659c700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80e00044e0 0x7f80e0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.050+0000 7f80ff59e700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80e0004e50 0x7f80e000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fed9d700 1 -- 172.21.15.187:0/2994888818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80e0004e50 msgr2=0x7f80e000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:40.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fed9d700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80e0004e50 0x7f80e000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fed9d700 1 -- 172.21.15.187:0/2994888818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80e00044e0 msgr2=0x7f80e0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:40.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fed9d700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80e00044e0 0x7f80e0004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fed9d700 1 -- 172.21.15.187:0/2994888818 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e0003240 con 0x7f80e0003730 2020-11-20T12:05:40.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80ff59e700 1 -- 172.21.15.187:0/2994888818 reap_dead start 2020-11-20T12:05:40.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fed9d700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80e0003730 0x7f80e0003b50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f80f0001e70 tx=0x7f80f0007f10).ready entity=mon.1 client_cookie=e106649051527a94 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:40.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f0001540 con 0x7f80e0003730 2020-11-20T12:05:40.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f0003030 con 0x7f80e0003730 2020-11-20T12:05:40.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.051+0000 7f80fe59c700 1 -- 172.21.15.187:0/2994888818 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f000cdf0 con 0x7f80e0003730 2020-11-20T12:05:40.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.052+0000 7f8104812700 1 -- 172.21.15.187:0/2994888818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80e0003730 msgr2=0x7f80e0003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.052+0000 7f8104812700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80e0003730 0x7f80e0003b50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f80f0001e70 tx=0x7f80f0007f10).stop 2020-11-20T12:05:40.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.052+0000 7f8104812700 1 -- 172.21.15.187:0/2994888818 shutdown_connections 2020-11-20T12:05:40.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.052+0000 7f8104812700 1 --2- 172.21.15.187:0/2994888818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80e0003730 0x7f80e0003b50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.052+0000 7f8104812700 1 -- 172.21.15.187:0/2994888818 >> 172.21.15.187:0/2994888818 conn(0x7f8100061bd0 msgr2=0x7f81000678a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:40.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.052+0000 7f8104812700 1 -- 172.21.15.187:0/2994888818 shutdown_connections 2020-11-20T12:05:40.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.052+0000 7f8104812700 1 -- 172.21.15.187:0/2994888818 wait complete. 2020-11-20T12:05:40.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.054+0000 7f8104812700 1 Processor -- start 2020-11-20T12:05:40.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.054+0000 7f8104812700 1 -- start start 2020-11-20T12:05:40.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f8104812700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81000ba500 0x7f810015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f8104812700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81000ba8e0 0x7f810015c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f8104812700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8100142840 0x7f810015c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f8104812700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810009fa90 con 0x7f8100142840 2020-11-20T12:05:40.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f8104812700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810009fd30 con 0x7f81000ba8e0 2020-11-20T12:05:40.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f8104812700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810009ffd0 con 0x7f81000ba500 2020-11-20T12:05:40.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f80f659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81000ba8e0 0x7f810015c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f80f659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81000ba8e0 0x7f810015c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41266/0 (socket says 172.21.15.187:41266) 2020-11-20T12:05:40.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f80f659c700 1 -- 172.21.15.187:0/2616385955 learned_addr learned my addr 172.21.15.187:0/2616385955 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:40.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f80f659c700 1 -- 172.21.15.187:0/2616385955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81000ba500 msgr2=0x7f810015bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f80f659c700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81000ba500 0x7f810015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f80f659c700 1 -- 172.21.15.187:0/2616385955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8100142840 msgr2=0x7f810015c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f80f659c700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8100142840 0x7f810015c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.055+0000 7f80f659c700 1 -- 172.21.15.187:0/2616385955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000bad30 con 0x7f81000ba8e0 2020-11-20T12:05:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.056+0000 7f80f659c700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81000ba8e0 0x7f810015c0f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f80e8001e90 tx=0x7f8100149ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.056+0000 7f80f77fe700 1 -- 172.21.15.187:0/2616385955 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e800c870 con 0x7f81000ba8e0 2020-11-20T12:05:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.056+0000 7f80f77fe700 1 -- 172.21.15.187:0/2616385955 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e800c9d0 con 0x7f81000ba8e0 2020-11-20T12:05:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.056+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8100144c90 con 0x7f81000ba8e0 2020-11-20T12:05:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.056+0000 7f80f77fe700 1 -- 172.21.15.187:0/2616385955 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e800cd70 con 0x7f81000ba8e0 2020-11-20T12:05:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.056+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8100144f80 con 0x7f81000ba8e0 2020-11-20T12:05:40.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.057+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81001495a0 con 0x7f81000ba8e0 2020-11-20T12:05:40.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.058+0000 7f80f77fe700 1 -- 172.21.15.187:0/2616385955 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f80e8014640 con 0x7f81000ba8e0 2020-11-20T12:05:40.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.058+0000 7f80f77fe700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80dc056d20 0x7f80dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.059+0000 7f80f77fe700 1 -- 172.21.15.187:0/2616385955 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f80e806fb90 con 0x7f81000ba8e0 2020-11-20T12:05:40.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.059+0000 7f80fed9d700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80dc056d20 0x7f80dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.059+0000 7f80fed9d700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80dc056d20 0x7f80dc0591e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f810005e630 tx=0x7f80f0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:40.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.062+0000 7f80f77fe700 1 -- 172.21.15.187:0/2616385955 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f80e8044f80 con 0x7f81000ba8e0 2020-11-20T12:05:40.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.248+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f81000a0190 con 0x7f81000ba8e0 2020-11-20T12:05:40.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.249+0000 7f80f77fe700 1 -- 172.21.15.187:0/2616385955 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f80e80485a0 con 0x7f81000ba8e0 2020-11-20T12:05:40.252 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:40.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80dc056d20 msgr2=0x7f80dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80dc056d20 0x7f80dc0591e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f810005e630 tx=0x7f80f0015000).stop 2020-11-20T12:05:40.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81000ba8e0 msgr2=0x7f810015c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81000ba8e0 0x7f810015c0f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f80e8001e90 tx=0x7f8100149ac0).stop 2020-11-20T12:05:40.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 shutdown_connections 2020-11-20T12:05:40.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80dc056d20 0x7f80dc0591e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8100142840 0x7f810015c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:40.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81000ba500 0x7f810015bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:40.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 --2- 172.21.15.187:0/2616385955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81000ba8e0 0x7f810015c0f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.251+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 >> 172.21.15.187:0/2616385955 conn(0x7f8100061bd0 msgr2=0x7f8100148aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:40.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.252+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 shutdown_connections 2020-11-20T12:05:40.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.252+0000 7f8104812700 1 -- 172.21.15.187:0/2616385955 wait complete. 2020-11-20T12:05:40.279 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:40.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.527+0000 7f9878af5700 1 Processor -- start 2020-11-20T12:05:40.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.527+0000 7f9878af5700 1 -- start start 2020-11-20T12:05:40.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.528+0000 7f9878af5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f98740b73b0 0x7f9874066e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.528+0000 7f9878af5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9874067300 0x7f9874144d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.528+0000 7f9878af5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9874146010 con 0x7f98740b6fd0 2020-11-20T12:05:40.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.528+0000 7f9878af5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98741462b0 con 0x7f9874067300 2020-11-20T12:05:40.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.528+0000 7f9878af5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9874146550 con 0x7f98740b73b0 2020-11-20T12:05:40.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f987259c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f98740b6fd0 0x7f9874066700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37878/0 (socket says 172.21.15.187:37878) 2020-11-20T12:05:40.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f987259c700 1 -- 172.21.15.187:0/3035372614 learned_addr learned my addr 172.21.15.187:0/3035372614 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:40.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f9869d9b700 1 --2- 172.21.15.187:0/3035372614 >> v2:172.21.15.187:3300/0 conn(0x7f98740b73b0 0x7f9874066e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f9872d9d700 1 --2- 172.21.15.187:0/3035372614 >> v2:172.21.15.5:3300/0 conn(0x7f9874067300 0x7f9874144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f9871d9b700 1 -- 172.21.15.187:0/3035372614 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70199332 0 0) 0x7f9874146010 con 0x7f98740b6fd0 2020-11-20T12:05:40.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f9871d9b700 1 -- 172.21.15.187:0/3035372614 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9854003240 con 0x7f98740b6fd0 2020-11-20T12:05:40.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f9869d9b700 1 -- 172.21.15.187:0/3035372614 >> v2:172.21.15.5:3300/0 conn(0x7f9874067300 msgr2=0x7f9874144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:40.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f9869d9b700 1 --2- 172.21.15.187:0/3035372614 >> v2:172.21.15.5:3300/0 conn(0x7f9874067300 0x7f9874144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f9869d9b700 1 -- 172.21.15.187:0/3035372614 >> v1:172.21.15.187:6789/0 conn(0x7f98740b6fd0 legacy=0x7f9874066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.529+0000 7f9869d9b700 1 -- 172.21.15.187:0/3035372614 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98740b6650 con 0x7f98740b73b0 2020-11-20T12:05:40.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.530+0000 7f9869d9b700 1 --2- 172.21.15.187:0/3035372614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f9874066e10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f98740666b0 tx=0x7f985c006df0).ready entity=mon.1 client_cookie=4ea63cd9cd327c4a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:40.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.530+0000 7f9871d9b700 1 -- 172.21.15.187:0/3035372614 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c0097b0 con 0x7f98740b73b0 2020-11-20T12:05:40.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.530+0000 7f9871d9b700 1 -- 172.21.15.187:0/3035372614 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c009910 con 0x7f98740b73b0 2020-11-20T12:05:40.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.530+0000 7f9871d9b700 1 -- 172.21.15.187:0/3035372614 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c009c90 con 0x7f98740b73b0 2020-11-20T12:05:40.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.531+0000 7f9878af5700 1 -- 172.21.15.187:0/3035372614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 msgr2=0x7f9874066e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.531+0000 7f9878af5700 1 --2- 172.21.15.187:0/3035372614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f9874066e10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f98740666b0 tx=0x7f985c006df0).stop 2020-11-20T12:05:40.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.531+0000 7f9878af5700 1 -- 172.21.15.187:0/3035372614 shutdown_connections 2020-11-20T12:05:40.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.531+0000 7f9878af5700 1 --2- 172.21.15.187:0/3035372614 >> v2:172.21.15.5:3300/0 conn(0x7f9874067300 0x7f9874144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.531+0000 7f9878af5700 1 --2- 172.21.15.187:0/3035372614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f9874066e10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.531+0000 7f9878af5700 1 -- 172.21.15.187:0/3035372614 >> 172.21.15.187:0/3035372614 conn(0x7f98740bacf0 msgr2=0x7f98740bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:40.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.531+0000 7f9878af5700 1 -- 172.21.15.187:0/3035372614 shutdown_connections 2020-11-20T12:05:40.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.531+0000 7f9878af5700 1 -- 172.21.15.187:0/3035372614 wait complete. 2020-11-20T12:05:40.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.532+0000 7f9878af5700 1 Processor -- start 2020-11-20T12:05:40.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.532+0000 7f9878af5700 1 -- start start 2020-11-20T12:05:40.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.533+0000 7f9878af5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9874067300 0x7f9874158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.533+0000 7f9878af5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98740b6fd0 0x7f987409cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.533+0000 7f9878af5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f987409a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.533+0000 7f9878af5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987409d0c0 con 0x7f9874067300 2020-11-20T12:05:40.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.533+0000 7f9878af5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987409bf50 con 0x7f98740b73b0 2020-11-20T12:05:40.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.533+0000 7f9878af5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987409c1f0 con 0x7f98740b6fd0 2020-11-20T12:05:40.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f9872d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f987409a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f9872d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f987409a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41280/0 (socket says 172.21.15.187:41280) 2020-11-20T12:05:40.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f9872d9d700 1 -- 172.21.15.187:0/965977426 learned_addr learned my addr 172.21.15.187:0/965977426 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:40.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f9869d9b700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98740b6fd0 0x7f987409cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f987259c700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9874067300 0x7f9874158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f9872d9d700 1 -- 172.21.15.187:0/965977426 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98740b6fd0 msgr2=0x7f987409cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f9872d9d700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98740b6fd0 0x7f987409cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f9872d9d700 1 -- 172.21.15.187:0/965977426 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9874067300 msgr2=0x7f9874158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.534+0000 7f9872d9d700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9874067300 0x7f9874158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.535+0000 7f9872d9d700 1 -- 172.21.15.187:0/965977426 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987409c490 con 0x7f98740b73b0 2020-11-20T12:05:40.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.535+0000 7f9872d9d700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f987409a970 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f986c00eb70 tx=0x7f986c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:40.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.535+0000 7f986affd700 1 -- 172.21.15.187:0/965977426 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c00c7a0 con 0x7f98740b73b0 2020-11-20T12:05:40.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.535+0000 7f986affd700 1 -- 172.21.15.187:0/965977426 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c00c900 con 0x7f98740b73b0 2020-11-20T12:05:40.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.535+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987409c730 con 0x7f98740b73b0 2020-11-20T12:05:40.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.535+0000 7f986affd700 1 -- 172.21.15.187:0/965977426 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c00cc80 con 0x7f98740b73b0 2020-11-20T12:05:40.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.535+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987409a320 con 0x7f98740b73b0 2020-11-20T12:05:40.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.536+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987409c8c0 con 0x7f98740b73b0 2020-11-20T12:05:40.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.537+0000 7f986affd700 1 -- 172.21.15.187:0/965977426 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f986c013430 con 0x7f98740b73b0 2020-11-20T12:05:40.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.538+0000 7f986affd700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9850056d50 0x7f9850059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:40.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.538+0000 7f986affd700 1 -- 172.21.15.187:0/965977426 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f986c06fca0 con 0x7f98740b73b0 2020-11-20T12:05:40.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.538+0000 7f987259c700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9850056d50 0x7f9850059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:40.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.539+0000 7f987259c700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9850056d50 0x7f9850059210 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f98640055a0 tx=0x7f9864009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:40.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.542+0000 7f986affd700 1 -- 172.21.15.187:0/965977426 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f986c045090 con 0x7f98740b73b0 2020-11-20T12:05:40.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.727+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9874044f60 con 0x7f98740b73b0 2020-11-20T12:05:40.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.728+0000 7f986affd700 1 -- 172.21.15.187:0/965977426 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f986c0486b0 con 0x7f98740b73b0 2020-11-20T12:05:40.732 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:40.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9850056d50 msgr2=0x7f9850059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9850056d50 0x7f9850059210 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f98640055a0 tx=0x7f9864009040).stop 2020-11-20T12:05:40.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 msgr2=0x7f987409a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:40.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f987409a970 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f986c00eb70 tx=0x7f986c008040).stop 2020-11-20T12:05:40.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 shutdown_connections 2020-11-20T12:05:40.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9850056d50 0x7f9850059210 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9874067300 0x7f9874158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98740b73b0 0x7f987409a970 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 --2- 172.21.15.187:0/965977426 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98740b6fd0 0x7f987409cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:40.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 >> 172.21.15.187:0/965977426 conn(0x7f98740bacf0 msgr2=0x7f9874145660 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:40.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.731+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 shutdown_connections 2020-11-20T12:05:40.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.732+0000 7f9878af5700 1 -- 172.21.15.187:0/965977426 wait complete. 2020-11-20T12:05:40.756 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:05:40.756 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:41.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.997+0000 7fb570ded700 1 Processor -- start 2020-11-20T12:05:41.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.998+0000 7fb570ded700 1 -- start start 2020-11-20T12:05:41.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.999+0000 7fb570ded700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb56c065f90 0x7fb56c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:41.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.999+0000 7fb570ded700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb56c09ca90 0x7fb56c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:41.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.999+0000 7fb570ded700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0b13e0 con 0x7fb56c065190 2020-11-20T12:05:41.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.999+0000 7fb570ded700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56c14ab30 con 0x7fb56c065f90 2020-11-20T12:05:41.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:40.999+0000 7fb570ded700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56c062c10 con 0x7fb56c09ca90 2020-11-20T12:05:41.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.000+0000 7fb56bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb56c09ca90 0x7fb56c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:41.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.000+0000 7fb56b7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb56c065190 0x7fb56c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37894/0 (socket says 172.21.15.187:37894) 2020-11-20T12:05:41.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.000+0000 7fb56b7fe700 1 -- 172.21.15.187:0/2860034799 learned_addr learned my addr 172.21.15.187:0/2860034799 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:41.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.000+0000 7fb562ffd700 1 --2- 172.21.15.187:0/2860034799 >> v2:172.21.15.5:3300/0 conn(0x7fb56c065f90 0x7fb56c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:41.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.000+0000 7fb56affd700 1 -- 172.21.15.187:0/2860034799 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567575976 0 0) 0x7fb56c0b13e0 con 0x7fb56c065190 2020-11-20T12:05:41.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.000+0000 7fb56affd700 1 -- 172.21.15.187:0/2860034799 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb54c003240 con 0x7fb56c065190 2020-11-20T12:05:41.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.000+0000 7fb56bfff700 1 -- 172.21.15.187:0/2860034799 >> v2:172.21.15.5:3300/0 conn(0x7fb56c065f90 msgr2=0x7fb56c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:41.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.000+0000 7fb56bfff700 1 --2- 172.21.15.187:0/2860034799 >> v2:172.21.15.5:3300/0 conn(0x7fb56c065f90 0x7fb56c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:41.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.001+0000 7fb56bfff700 1 -- 172.21.15.187:0/2860034799 >> v1:172.21.15.187:6789/0 conn(0x7fb56c065190 legacy=0x7fb56c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:41.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.001+0000 7fb56bfff700 1 -- 172.21.15.187:0/2860034799 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0b1640 con 0x7fb56c09ca90 2020-11-20T12:05:41.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.001+0000 7fb562ffd700 1 --2- 172.21.15.187:0/2860034799 >> v2:172.21.15.5:3300/0 conn(0x7fb56c065f90 0x7fb56c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:41.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.001+0000 7fb56bfff700 1 --2- 172.21.15.187:0/2860034799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c09ca90 0x7fb56c09cf20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fb56c099f90 tx=0x7fb5640091d0).ready entity=mon.1 client_cookie=5c75e98110942960 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:41.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.001+0000 7fb56affd700 1 -- 172.21.15.187:0/2860034799 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56400bad0 con 0x7fb56c09ca90 2020-11-20T12:05:41.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.001+0000 7fb56affd700 1 -- 172.21.15.187:0/2860034799 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56400bc30 con 0x7fb56c09ca90 2020-11-20T12:05:41.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.001+0000 7fb56affd700 1 -- 172.21.15.187:0/2860034799 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564011400 con 0x7fb56c09ca90 2020-11-20T12:05:41.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.002+0000 7fb570ded700 1 -- 172.21.15.187:0/2860034799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c09ca90 msgr2=0x7fb56c09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:41.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.002+0000 7fb570ded700 1 --2- 172.21.15.187:0/2860034799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c09ca90 0x7fb56c09cf20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fb56c099f90 tx=0x7fb5640091d0).stop 2020-11-20T12:05:41.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.002+0000 7fb570ded700 1 -- 172.21.15.187:0/2860034799 shutdown_connections 2020-11-20T12:05:41.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.002+0000 7fb570ded700 1 --2- 172.21.15.187:0/2860034799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c09ca90 0x7fb56c09cf20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:41.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.002+0000 7fb570ded700 1 --2- 172.21.15.187:0/2860034799 >> v2:172.21.15.5:3300/0 conn(0x7fb56c065f90 0x7fb56c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:41.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.002+0000 7fb570ded700 1 -- 172.21.15.187:0/2860034799 >> 172.21.15.187:0/2860034799 conn(0x7fb56c0bab30 msgr2=0x7fb56c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:41.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.002+0000 7fb570ded700 1 -- 172.21.15.187:0/2860034799 shutdown_connections 2020-11-20T12:05:41.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.002+0000 7fb570ded700 1 -- 172.21.15.187:0/2860034799 wait complete. 2020-11-20T12:05:41.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.003+0000 7fb570ded700 1 Processor -- start 2020-11-20T12:05:41.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.003+0000 7fb570ded700 1 -- start start 2020-11-20T12:05:41.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.004+0000 7fb570ded700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb56c065190 0x7fb56c099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:41.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.004+0000 7fb570ded700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c065f90 0x7fb56c0a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:41.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.004+0000 7fb570ded700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb56c09ca90 0x7fb56c15b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:41.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.004+0000 7fb570ded700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c066710 con 0x7fb56c09ca90 2020-11-20T12:05:41.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.004+0000 7fb570ded700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c0669b0 con 0x7fb56c065f90 2020-11-20T12:05:41.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.004+0000 7fb570ded700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c066c50 con 0x7fb56c065190 2020-11-20T12:05:41.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c065f90 0x7fb56c0a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:41.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c065f90 0x7fb56c0a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41294/0 (socket says 172.21.15.187:41294) 2020-11-20T12:05:41.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 -- 172.21.15.187:0/1944458547 learned_addr learned my addr 172.21.15.187:0/1944458547 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:41.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 -- 172.21.15.187:0/1944458547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb56c065190 msgr2=0x7fb56c099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:41.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb56c065190 0x7fb56c099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:41.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 -- 172.21.15.187:0/1944458547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb56c09ca90 msgr2=0x7fb56c15b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:41.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb56c09ca90 0x7fb56c15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:41.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 -- 172.21.15.187:0/1944458547 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c141e50 con 0x7fb56c065f90 2020-11-20T12:05:41.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb562ffd700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c065f90 0x7fb56c0a0130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fb55400db60 tx=0x7fb554008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:41.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb563fff700 1 -- 172.21.15.187:0/1944458547 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5540098e0 con 0x7fb56c065f90 2020-11-20T12:05:41.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.005+0000 7fb563fff700 1 -- 172.21.15.187:0/1944458547 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb554009a40 con 0x7fb56c065f90 2020-11-20T12:05:41.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.006+0000 7fb563fff700 1 -- 172.21.15.187:0/1944458547 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554009dc0 con 0x7fb56c065f90 2020-11-20T12:05:41.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.006+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c0a0620 con 0x7fb56c065f90 2020-11-20T12:05:41.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.006+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c15baf0 con 0x7fb56c065f90 2020-11-20T12:05:41.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.007+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56c066e10 con 0x7fb56c065f90 2020-11-20T12:05:41.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.007+0000 7fb563fff700 1 -- 172.21.15.187:0/1944458547 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb554013690 con 0x7fb56c065f90 2020-11-20T12:05:41.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.008+0000 7fb563fff700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb548056d20 0x7fb5480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:41.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.008+0000 7fb563fff700 1 -- 172.21.15.187:0/1944458547 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb5540700c0 con 0x7fb56c065f90 2020-11-20T12:05:41.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.008+0000 7fb56b7fe700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb548056d20 0x7fb5480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:41.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.009+0000 7fb56b7fe700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb548056d20 0x7fb5480591e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb55c0055a0 tx=0x7fb55c0076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:41.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.012+0000 7fb563fff700 1 -- 172.21.15.187:0/1944458547 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb5540454b0 con 0x7fb56c065f90 2020-11-20T12:05:41.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.250+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb56c061a20 con 0x7fb56c065f90 2020-11-20T12:05:41.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.251+0000 7fb563fff700 1 -- 172.21.15.187:0/1944458547 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2130 (secure 0 0 0) 0x7fb554048ad0 con 0x7fb56c065f90 2020-11-20T12:05:41.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.253+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb548056d20 msgr2=0x7fb5480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.253+0000 7fb570ded700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb548056d20 0x7fb5480591e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb55c0055a0 tx=0x7fb55c0076a0).stop 2020-11-20T12:05:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.253+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c065f90 msgr2=0x7fb56c0a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.253+0000 7fb570ded700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c065f90 0x7fb56c0a0130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fb55400db60 tx=0x7fb554008040).stop 2020-11-20T12:05:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.254+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 shutdown_connections 2020-11-20T12:05:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.254+0000 7fb570ded700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb548056d20 0x7fb5480591e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.254+0000 7fb570ded700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb56c09ca90 0x7fb56c15b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.254+0000 7fb570ded700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb56c065190 0x7fb56c099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.254+0000 7fb570ded700 1 --2- 172.21.15.187:0/1944458547 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb56c065f90 0x7fb56c0a0130 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.254+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 >> 172.21.15.187:0/1944458547 conn(0x7fb56c0bab30 msgr2=0x7fb56c141070 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:41.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.254+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 shutdown_connections 2020-11-20T12:05:41.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:41.254+0000 7fb570ded700 1 -- 172.21.15.187:0/1944458547 wait complete. 2020-11-20T12:05:41.262 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:41.262 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4211,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6833/2094645874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2094645874},{"type":"v1","addr":"172.21.15.5:6833","nonce":2094645874}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:57.868606+0000","modified":"2020-11-20T12:04:59.990344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3094884769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3094884769},{"type":"v1","addr":"172.21.15.187:6835","nonce":3094884769}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:41.263 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 6 2020-11-20T12:05:41.482 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:04:57.868606+0000', 'modified': '2020-11-20T12:04:59.990344+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4264': {'gid': 4264, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3094884769', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3094884769}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3094884769}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:05:41.483 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:05:41.483 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:05:41.761 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":4264,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":43.862350091000003,"mdsmap_epoch":6,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":48.251173766999997} 2020-11-20T12:05:41.762 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 4264, "mdsmap_epoch": 6, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 43.862350091, "state": "up:active", "uptime": 48.251173767, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:05:41.763 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:05:41.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:41.763 INFO:teuthology.orchestra.run.smithi005:> mkdir -p /home/ubuntu/cephtest/mnt.3 2020-11-20T12:05:41.828 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:05:41.828 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,name=3,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:41.980 INFO:teuthology.orchestra.run.smithi005.stdout:parsing options: rw,norequire_active_mds,name=3,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:41.981 INFO:teuthology.orchestra.run.smithi005.stdout:mount.ceph: options "norequire_active_mds,name=3,norbytes" will pass to kernel. 2020-11-20T12:05:42.019 INFO:teuthology.orchestra.run.smithi005.stdout:parsing options: rw,norequire_active_mds,name=3,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:05:42.019 INFO:teuthology.orchestra.run.smithi005.stdout:mount.ceph: options "norequire_active_mds,name=3,norbytes" will pass to kernel. 2020-11-20T12:05:42.031 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:05:42.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:42.032 INFO:teuthology.orchestra.run.smithi005:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2020-11-20T12:05:42.128 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2020-11-20T12:05:42.159 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:42.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.406+0000 7ff7c7484700 1 Processor -- start 2020-11-20T12:05:42.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.407+0000 7ff7c7484700 1 -- start start 2020-11-20T12:05:42.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.408+0000 7ff7c7484700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7c00b57b0 0x7ff7c0068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.408+0000 7ff7c7484700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 0x7ff7c00b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.408+0000 7ff7c7484700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c0144b80 con 0x7ff7c01482d0 2020-11-20T12:05:42.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.408+0000 7ff7c7484700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c01489e0 con 0x7ff7c00b57b0 2020-11-20T12:05:42.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.408+0000 7ff7c7484700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c0148c80 con 0x7ff7c00b5f50 2020-11-20T12:05:42.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.408+0000 7ff7c5c81700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff7c01482d0 0x7ff7c0062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37910/0 (socket says 172.21.15.187:37910) 2020-11-20T12:05:42.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.408+0000 7ff7c6482700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7c00b57b0 0x7ff7c0068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.408+0000 7ff7c5c81700 1 -- 172.21.15.187:0/1718926946 learned_addr learned my addr 172.21.15.187:0/1718926946 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:42.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.409+0000 7ff7c6c83700 1 --2- 172.21.15.187:0/1718926946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 0x7ff7c00b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.409+0000 7ff7c5480700 1 -- 172.21.15.187:0/1718926946 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227659795 0 0) 0x7ff7c0144b80 con 0x7ff7c01482d0 2020-11-20T12:05:42.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.409+0000 7ff7c5480700 1 -- 172.21.15.187:0/1718926946 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a8003240 con 0x7ff7c01482d0 2020-11-20T12:05:42.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.409+0000 7ff7c6482700 1 -- 172.21.15.187:0/1718926946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 msgr2=0x7ff7c00b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:42.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.409+0000 7ff7c6482700 1 --2- 172.21.15.187:0/1718926946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 0x7ff7c00b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.409+0000 7ff7c6482700 1 -- 172.21.15.187:0/1718926946 >> v1:172.21.15.187:6789/0 conn(0x7ff7c01482d0 legacy=0x7ff7c0062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.409+0000 7ff7c6482700 1 -- 172.21.15.187:0/1718926946 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0148f20 con 0x7ff7c00b57b0 2020-11-20T12:05:42.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.410+0000 7ff7c6482700 1 --2- 172.21.15.187:0/1718926946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c00b57b0 0x7ff7c0068230 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7ff7bc00b9a0 tx=0x7ff7bc006df0).ready entity=mon.1 client_cookie=5cbd610fb76db83 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:42.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.410+0000 7ff7c5480700 1 -- 172.21.15.187:0/1718926946 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0097b0 con 0x7ff7c00b57b0 2020-11-20T12:05:42.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.410+0000 7ff7c5480700 1 -- 172.21.15.187:0/1718926946 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc009910 con 0x7ff7c00b57b0 2020-11-20T12:05:42.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.410+0000 7ff7c5480700 1 -- 172.21.15.187:0/1718926946 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc009c90 con 0x7ff7c00b57b0 2020-11-20T12:05:42.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.410+0000 7ff7c7484700 1 -- 172.21.15.187:0/1718926946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c00b57b0 msgr2=0x7ff7c0068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.410+0000 7ff7c7484700 1 --2- 172.21.15.187:0/1718926946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c00b57b0 0x7ff7c0068230 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7ff7bc00b9a0 tx=0x7ff7bc006df0).stop 2020-11-20T12:05:42.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.411+0000 7ff7c7484700 1 -- 172.21.15.187:0/1718926946 shutdown_connections 2020-11-20T12:05:42.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.411+0000 7ff7c7484700 1 --2- 172.21.15.187:0/1718926946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 0x7ff7c00b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.411+0000 7ff7c7484700 1 --2- 172.21.15.187:0/1718926946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c00b57b0 0x7ff7c0068230 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.411+0000 7ff7c7484700 1 -- 172.21.15.187:0/1718926946 >> 172.21.15.187:0/1718926946 conn(0x7ff7c00bab30 msgr2=0x7ff7c00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:42.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.411+0000 7ff7c7484700 1 -- 172.21.15.187:0/1718926946 shutdown_connections 2020-11-20T12:05:42.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.411+0000 7ff7c7484700 1 -- 172.21.15.187:0/1718926946 wait complete. 2020-11-20T12:05:42.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.412+0000 7ff7c7484700 1 Processor -- start 2020-11-20T12:05:42.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.412+0000 7ff7c7484700 1 -- start start 2020-11-20T12:05:42.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.413+0000 7ff7c7484700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 0x7ff7c00b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.413+0000 7ff7c7484700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c01482d0 0x7ff7c0142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.413+0000 7ff7c7484700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c0142d70 0x7ff7c009cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.413+0000 7ff7c7484700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00b22e0 con 0x7ff7c01482d0 2020-11-20T12:05:42.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.413+0000 7ff7c7484700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00b47f0 con 0x7ff7c0142d70 2020-11-20T12:05:42.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.413+0000 7ff7c7484700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c0099e10 con 0x7ff7c00b5f50 2020-11-20T12:05:42.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6c83700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c0142d70 0x7ff7c009cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6c83700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c0142d70 0x7ff7c009cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41310/0 (socket says 172.21.15.187:41310) 2020-11-20T12:05:42.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6c83700 1 -- 172.21.15.187:0/753431409 learned_addr learned my addr 172.21.15.187:0/753431409 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:42.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c5c81700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c01482d0 0x7ff7c0142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6c83700 1 -- 172.21.15.187:0/753431409 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 msgr2=0x7ff7c00b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6482700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 0x7ff7c00b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6c83700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 0x7ff7c00b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6c83700 1 -- 172.21.15.187:0/753431409 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c01482d0 msgr2=0x7ff7c0142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6c83700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c01482d0 0x7ff7c0142880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.414+0000 7ff7c6c83700 1 -- 172.21.15.187:0/753431409 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c009a0b0 con 0x7ff7c0142d70 2020-11-20T12:05:42.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.415+0000 7ff7c6c83700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c0142d70 0x7ff7c009cbe0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7ff7b800eb70 tx=0x7ff7b8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:42.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.415+0000 7ff7ae7fc700 1 -- 172.21.15.187:0/753431409 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b800c7a0 con 0x7ff7c0142d70 2020-11-20T12:05:42.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.415+0000 7ff7ae7fc700 1 -- 172.21.15.187:0/753431409 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b800c900 con 0x7ff7c0142d70 2020-11-20T12:05:42.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.415+0000 7ff7ae7fc700 1 -- 172.21.15.187:0/753431409 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b800cc80 con 0x7ff7c0142d70 2020-11-20T12:05:42.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.415+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c009a350 con 0x7ff7c0142d70 2020-11-20T12:05:42.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.415+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c01434a0 con 0x7ff7c0142d70 2020-11-20T12:05:42.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.416+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c00b6e50 con 0x7ff7c0142d70 2020-11-20T12:05:42.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.417+0000 7ff7ae7fc700 1 -- 172.21.15.187:0/753431409 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff7b801d460 con 0x7ff7c0142d70 2020-11-20T12:05:42.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.417+0000 7ff7ae7fc700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff798056d20 0x7ff7980591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.417+0000 7ff7ae7fc700 1 -- 172.21.15.187:0/753431409 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff7b8070050 con 0x7ff7c0142d70 2020-11-20T12:05:42.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.417+0000 7ff7c6482700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff798056d20 0x7ff7980591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.418+0000 7ff7c6482700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff798056d20 0x7ff7980591e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7ff7c0063890 tx=0x7ff7bc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:42.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.421+0000 7ff7ae7fc700 1 -- 172.21.15.187:0/753431409 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff7b8045440 con 0x7ff7c0142d70 2020-11-20T12:05:42.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.609+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7c0061a20 con 0x7ff7c0142d70 2020-11-20T12:05:42.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.611+0000 7ff7ae7fc700 1 -- 172.21.15.187:0/753431409 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7ff7b8048a60 con 0x7ff7c0142d70 2020-11-20T12:05:42.615 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:42.616 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:42.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.613+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff798056d20 msgr2=0x7ff7980591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.613+0000 7ff7c7484700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff798056d20 0x7ff7980591e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7ff7c0063890 tx=0x7ff7bc015000).stop 2020-11-20T12:05:42.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c0142d70 msgr2=0x7ff7c009cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c0142d70 0x7ff7c009cbe0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7ff7b800eb70 tx=0x7ff7b8008040).stop 2020-11-20T12:05:42.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 shutdown_connections 2020-11-20T12:05:42.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff798056d20 0x7ff7980591e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c0142d70 0x7ff7c009cbe0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c00b5f50 0x7ff7c00b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 --2- 172.21.15.187:0/753431409 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c01482d0 0x7ff7c0142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 >> 172.21.15.187:0/753431409 conn(0x7ff7c00bab30 msgr2=0x7ff7c009fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:42.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 shutdown_connections 2020-11-20T12:05:42.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.614+0000 7ff7c7484700 1 -- 172.21.15.187:0/753431409 wait complete. 2020-11-20T12:05:42.643 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:42.644 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:42.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.884+0000 7f7978859700 1 Processor -- start 2020-11-20T12:05:42.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.884+0000 7f7978859700 1 -- start start 2020-11-20T12:05:42.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.886+0000 7f7978859700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79740bfbe0 0x7f79741480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.886+0000 7f7978859700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f797409f560 0x7f79741485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.886+0000 7f7978859700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797414d110 con 0x7f79740bf800 2020-11-20T12:05:42.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.886+0000 7f7978859700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797414c9d0 con 0x7f797409f560 2020-11-20T12:05:42.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.886+0000 7f7978859700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7974148aa0 con 0x7f79740bfbe0 2020-11-20T12:05:42.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.887+0000 7f7972d9d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f79740bf800 0x7f797414c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:59662/0 (socket says 172.21.15.187:59662) 2020-11-20T12:05:42.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.887+0000 7f7972d9d700 1 -- 172.21.15.187:0/2313710399 learned_addr learned my addr 172.21.15.187:0/2313710399 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:42.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.887+0000 7f797259c700 1 --2- 172.21.15.187:0/2313710399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79740bfbe0 0x7f79741480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.887+0000 7f797359e700 1 --2- 172.21.15.187:0/2313710399 >> v2:172.21.15.5:3300/0 conn(0x7f797409f560 0x7f79741485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.888+0000 7f7971d9b700 1 -- 172.21.15.187:0/2313710399 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030442542 0 0) 0x7f797414d110 con 0x7f79740bf800 2020-11-20T12:05:42.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.888+0000 7f7971d9b700 1 -- 172.21.15.187:0/2313710399 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7960003240 con 0x7f79740bf800 2020-11-20T12:05:42.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.888+0000 7f797259c700 1 -- 172.21.15.187:0/2313710399 >> v2:172.21.15.5:3300/0 conn(0x7f797409f560 msgr2=0x7f79741485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:42.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.888+0000 7f797259c700 1 --2- 172.21.15.187:0/2313710399 >> v2:172.21.15.5:3300/0 conn(0x7f797409f560 0x7f79741485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.888+0000 7f797259c700 1 -- 172.21.15.187:0/2313710399 >> v1:172.21.15.5:6789/0 conn(0x7f79740bf800 legacy=0x7f797414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.889+0000 7f797259c700 1 -- 172.21.15.187:0/2313710399 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797414cbd0 con 0x7f79740bfbe0 2020-11-20T12:05:42.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.889+0000 7f797259c700 1 --2- 172.21.15.187:0/2313710399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79740bfbe0 0x7f79741480c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f796800b9d0 tx=0x7f7968006df0).ready entity=mon.2 client_cookie=acb509058ff52a6f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:42.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.889+0000 7f7971d9b700 1 -- 172.21.15.187:0/2313710399 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79680096d0 con 0x7f79740bfbe0 2020-11-20T12:05:42.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7971d9b700 1 -- 172.21.15.187:0/2313710399 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7968009830 con 0x7f79740bfbe0 2020-11-20T12:05:42.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7971d9b700 1 -- 172.21.15.187:0/2313710399 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968009bb0 con 0x7f79740bfbe0 2020-11-20T12:05:42.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7978859700 1 -- 172.21.15.187:0/2313710399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79740bfbe0 msgr2=0x7f79741480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7978859700 1 --2- 172.21.15.187:0/2313710399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79740bfbe0 0x7f79741480c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f796800b9d0 tx=0x7f7968006df0).stop 2020-11-20T12:05:42.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7978859700 1 -- 172.21.15.187:0/2313710399 shutdown_connections 2020-11-20T12:05:42.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7978859700 1 --2- 172.21.15.187:0/2313710399 >> v2:172.21.15.5:3300/0 conn(0x7f797409f560 0x7f79741485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7978859700 1 --2- 172.21.15.187:0/2313710399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79740bfbe0 0x7f79741480c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7978859700 1 -- 172.21.15.187:0/2313710399 >> 172.21.15.187:0/2313710399 conn(0x7f7974068370 msgr2=0x7f79740bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:42.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7978859700 1 -- 172.21.15.187:0/2313710399 shutdown_connections 2020-11-20T12:05:42.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.890+0000 7f7978859700 1 -- 172.21.15.187:0/2313710399 wait complete. 2020-11-20T12:05:42.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.892+0000 7f7978859700 1 Processor -- start 2020-11-20T12:05:42.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.892+0000 7f7978859700 1 -- start start 2020-11-20T12:05:42.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f7978859700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f797409f560 0x7f79740b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f7978859700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79740bf800 0x7f79741436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f7978859700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7974143b90 0x7f797415c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f7978859700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79740b9280 con 0x7f7974143b90 2020-11-20T12:05:42.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f7978859700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79740b9520 con 0x7f79740bf800 2020-11-20T12:05:42.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f7978859700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79740b97c0 con 0x7f797409f560 2020-11-20T12:05:42.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f797259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79740bf800 0x7f79741436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f797259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79740bf800 0x7f79741436a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41322/0 (socket says 172.21.15.187:41322) 2020-11-20T12:05:42.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.893+0000 7f797259c700 1 -- 172.21.15.187:0/2128549812 learned_addr learned my addr 172.21.15.187:0/2128549812 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:42.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f7972d9d700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f797409f560 0x7f79740b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f797359e700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7974143b90 0x7f797415c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f797259c700 1 -- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f797409f560 msgr2=0x7f79740b5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f797259c700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f797409f560 0x7f79740b5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f797259c700 1 -- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7974143b90 msgr2=0x7f797415c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:42.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f797259c700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7974143b90 0x7f797415c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:42.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f797259c700 1 -- 172.21.15.187:0/2128549812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7974143ff0 con 0x7f79740bf800 2020-11-20T12:05:42.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f797359e700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7974143b90 0x7f797415c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:42.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f797259c700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79740bf800 0x7f79741436a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f7968006e30 tx=0x7f7968015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:42.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f795affd700 1 -- 172.21.15.187:0/2128549812 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968011e50 con 0x7f79740bf800 2020-11-20T12:05:42.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f795affd700 1 -- 172.21.15.187:0/2128549812 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796801c5c0 con 0x7f79740bf800 2020-11-20T12:05:42.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79740b4d60 con 0x7f79740bf800 2020-11-20T12:05:42.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.894+0000 7f795affd700 1 -- 172.21.15.187:0/2128549812 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796801c980 con 0x7f79740bf800 2020-11-20T12:05:42.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.895+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797415c790 con 0x7f79740bf800 2020-11-20T12:05:42.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.896+0000 7f795affd700 1 -- 172.21.15.187:0/2128549812 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7968023030 con 0x7f79740bf800 2020-11-20T12:05:42.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.896+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974080050 con 0x7f79740bf800 2020-11-20T12:05:42.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.897+0000 7f795affd700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7950056d20 0x7f79500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:42.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.897+0000 7f795affd700 1 -- 172.21.15.187:0/2128549812 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7968071100 con 0x7f79740bf800 2020-11-20T12:05:42.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.897+0000 7f7972d9d700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7950056d20 0x7f79500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:42.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.898+0000 7f7972d9d700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7950056d20 0x7f79500591e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f79640055a0 tx=0x7f7964009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:42.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:42.901+0000 7f795affd700 1 -- 172.21.15.187:0/2128549812 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7968046550 con 0x7f79740bf800 2020-11-20T12:05:43.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.087+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f79740a46d0 con 0x7f79740bf800 2020-11-20T12:05:43.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.088+0000 7f795affd700 1 -- 172.21.15.187:0/2128549812 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f7968049b70 con 0x7f79740bf800 2020-11-20T12:05:43.092 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:43.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.090+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7950056d20 msgr2=0x7f79500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.090+0000 7f7978859700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7950056d20 0x7f79500591e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f79640055a0 tx=0x7f7964009040).stop 2020-11-20T12:05:43.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79740bf800 msgr2=0x7f79741436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79740bf800 0x7f79741436a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f7968006e30 tx=0x7f7968015000).stop 2020-11-20T12:05:43.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 shutdown_connections 2020-11-20T12:05:43.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7950056d20 0x7f79500591e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7974143b90 0x7f797415c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f797409f560 0x7f79740b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 --2- 172.21.15.187:0/2128549812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79740bf800 0x7f79741436a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 >> 172.21.15.187:0/2128549812 conn(0x7f7974068370 msgr2=0x7f7974147600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:43.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 shutdown_connections 2020-11-20T12:05:43.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.091+0000 7f7978859700 1 -- 172.21.15.187:0/2128549812 wait complete. 2020-11-20T12:05:43.118 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:43.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.366+0000 7f992ecf9700 1 Processor -- start 2020-11-20T12:05:43.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.366+0000 7f992ecf9700 1 -- start start 2020-11-20T12:05:43.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.367+0000 7f992ecf9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99280b1380 0x7f9928150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.367+0000 7f992ecf9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 0x7f992814c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.367+0000 7f992ecf9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99280658a0 con 0x7f99280b1760 2020-11-20T12:05:43.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.367+0000 7f992ecf9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9928150390 con 0x7f99280b1380 2020-11-20T12:05:43.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.367+0000 7f992ecf9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9928150630 con 0x7f992814c260 2020-11-20T12:05:43.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.368+0000 7f992d4f6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f99280b1760 0x7f9928065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37934/0 (socket says 172.21.15.187:37934) 2020-11-20T12:05:43.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.368+0000 7f992d4f6700 1 -- 172.21.15.187:0/2420929305 learned_addr learned my addr 172.21.15.187:0/2420929305 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:43.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.368+0000 7f992dcf7700 1 --2- 172.21.15.187:0/2420929305 >> v2:172.21.15.187:3300/0 conn(0x7f99280b1380 0x7f9928150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.368+0000 7f992ccf5700 1 -- 172.21.15.187:0/2420929305 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946978783 0 0) 0x7f99280658a0 con 0x7f99280b1760 2020-11-20T12:05:43.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.368+0000 7f992e4f8700 1 --2- 172.21.15.187:0/2420929305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 0x7f992814c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.368+0000 7f992ccf5700 1 -- 172.21.15.187:0/2420929305 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9910003240 con 0x7f99280b1760 2020-11-20T12:05:43.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.368+0000 7f992dcf7700 1 -- 172.21.15.187:0/2420929305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 msgr2=0x7f992814c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:43.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.368+0000 7f992dcf7700 1 --2- 172.21.15.187:0/2420929305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 0x7f992814c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.369+0000 7f992dcf7700 1 -- 172.21.15.187:0/2420929305 >> v1:172.21.15.187:6789/0 conn(0x7f99280b1760 legacy=0x7f9928065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.369+0000 7f992dcf7700 1 -- 172.21.15.187:0/2420929305 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9928068510 con 0x7f99280b1380 2020-11-20T12:05:43.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.369+0000 7f992dcf7700 1 --2- 172.21.15.187:0/2420929305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1380 0x7f9928150ad0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f992400b9a0 tx=0x7f9924006df0).ready entity=mon.1 client_cookie=cd64f60709181632 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:43.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.369+0000 7f992ccf5700 1 -- 172.21.15.187:0/2420929305 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99240097b0 con 0x7f99280b1380 2020-11-20T12:05:43.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.369+0000 7f992ccf5700 1 -- 172.21.15.187:0/2420929305 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9924009910 con 0x7f99280b1380 2020-11-20T12:05:43.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.369+0000 7f992ccf5700 1 -- 172.21.15.187:0/2420929305 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924009c90 con 0x7f99280b1380 2020-11-20T12:05:43.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.370+0000 7f992ecf9700 1 -- 172.21.15.187:0/2420929305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1380 msgr2=0x7f9928150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.370+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2420929305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1380 0x7f9928150ad0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f992400b9a0 tx=0x7f9924006df0).stop 2020-11-20T12:05:43.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.370+0000 7f992ecf9700 1 -- 172.21.15.187:0/2420929305 shutdown_connections 2020-11-20T12:05:43.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.370+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2420929305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 0x7f992814c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.370+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2420929305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1380 0x7f9928150ad0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.370+0000 7f992ecf9700 1 -- 172.21.15.187:0/2420929305 >> 172.21.15.187:0/2420929305 conn(0x7f99280bab30 msgr2=0x7f99280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:43.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.370+0000 7f992ecf9700 1 -- 172.21.15.187:0/2420929305 shutdown_connections 2020-11-20T12:05:43.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.370+0000 7f992ecf9700 1 -- 172.21.15.187:0/2420929305 wait complete. 2020-11-20T12:05:43.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.371+0000 7f992ecf9700 1 Processor -- start 2020-11-20T12:05:43.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.371+0000 7f992ecf9700 1 -- start start 2020-11-20T12:05:43.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.372+0000 7f992ecf9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99280b1380 0x7f992815cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.372+0000 7f992ecf9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1760 0x7f9928148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.372+0000 7f992ecf9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 0x7f99280b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.372+0000 7f992ecf9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9928148830 con 0x7f99280b1380 2020-11-20T12:05:43.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.372+0000 7f992ecf9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99280b7db0 con 0x7f99280b1760 2020-11-20T12:05:43.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.372+0000 7f992ecf9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992809a7f0 con 0x7f992814c260 2020-11-20T12:05:43.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1760 0x7f9928148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1760 0x7f9928148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41336/0 (socket says 172.21.15.187:41336) 2020-11-20T12:05:43.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 -- 172.21.15.187:0/2586137950 learned_addr learned my addr 172.21.15.187:0/2586137950 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:43.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992dcf7700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99280b1380 0x7f992815cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 -- 172.21.15.187:0/2586137950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 msgr2=0x7f99280b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 0x7f99280b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:43.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 -- 172.21.15.187:0/2586137950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99280b1380 msgr2=0x7f992815cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99280b1380 0x7f992815cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 -- 172.21.15.187:0/2586137950 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99280b5600 con 0x7f99280b1760 2020-11-20T12:05:43.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.373+0000 7f992d4f6700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1760 0x7f9928148340 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f991800c6f0 tx=0x7f9918007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:43.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.374+0000 7f9915ffb700 1 -- 172.21.15.187:0/2586137950 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918007320 con 0x7f99280b1760 2020-11-20T12:05:43.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.374+0000 7f9915ffb700 1 -- 172.21.15.187:0/2586137950 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9918007480 con 0x7f99280b1760 2020-11-20T12:05:43.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.374+0000 7f9915ffb700 1 -- 172.21.15.187:0/2586137950 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918005800 con 0x7f99280b1760 2020-11-20T12:05:43.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.374+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992809bea0 con 0x7f99280b1760 2020-11-20T12:05:43.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.374+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99280b7b40 con 0x7f99280b1760 2020-11-20T12:05:43.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.375+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992809a9b0 con 0x7f99280b1760 2020-11-20T12:05:43.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.375+0000 7f9915ffb700 1 -- 172.21.15.187:0/2586137950 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9918010030 con 0x7f99280b1760 2020-11-20T12:05:43.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.376+0000 7f9915ffb700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9900056d20 0x7f99000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.376+0000 7f9915ffb700 1 -- 172.21.15.187:0/2586137950 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f991806fca0 con 0x7f99280b1760 2020-11-20T12:05:43.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.376+0000 7f992dcf7700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9900056d20 0x7f99000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.377+0000 7f992dcf7700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9900056d20 0x7f99000591e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9924006e30 tx=0x7f9924015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:43.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.380+0000 7f9915ffb700 1 -- 172.21.15.187:0/2586137950 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9918074070 con 0x7f99280b1760 2020-11-20T12:05:43.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.566+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9928061a20 con 0x7f99280b1760 2020-11-20T12:05:43.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.567+0000 7f9915ffb700 1 -- 172.21.15.187:0/2586137950 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9918045090 con 0x7f99280b1760 2020-11-20T12:05:43.576 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:43.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9900056d20 msgr2=0x7f99000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9900056d20 0x7f99000591e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9924006e30 tx=0x7f9924015000).stop 2020-11-20T12:05:43.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1760 msgr2=0x7f9928148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1760 0x7f9928148340 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f991800c6f0 tx=0x7f9918007ad0).stop 2020-11-20T12:05:43.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 shutdown_connections 2020-11-20T12:05:43.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9900056d20 0x7f99000591e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99280b1380 0x7f992815cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f992814c260 0x7f99280b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:43.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 --2- 172.21.15.187:0/2586137950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99280b1760 0x7f9928148340 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 >> 172.21.15.187:0/2586137950 conn(0x7f99280bab30 msgr2=0x7f9928149080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:43.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 shutdown_connections 2020-11-20T12:05:43.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.570+0000 7f992ecf9700 1 -- 172.21.15.187:0/2586137950 wait complete. 2020-11-20T12:05:43.598 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:43.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.840+0000 7f9fe234d700 1 Processor -- start 2020-11-20T12:05:43.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.840+0000 7f9fe234d700 1 -- start start 2020-11-20T12:05:43.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.841+0000 7f9fe234d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 0x7f9fdc066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.841+0000 7f9fe234d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9fdc1487e0 0x7f9fdc148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.841+0000 7f9fe234d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc14c170 con 0x7f9fdc0683f0 2020-11-20T12:05:43.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.841+0000 7f9fe234d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fdc14c410 con 0x7f9fdc1487e0 2020-11-20T12:05:43.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.841+0000 7f9fe234d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc14c6b0 con 0x7f9fdc145cc0 2020-11-20T12:05:43.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.842+0000 7f9fe1b4c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9fdc1487e0 0x7f9fdc148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.842+0000 7f9fe134b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 0x7f9fdc066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.842+0000 7f9fe1b4c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9fdc1487e0 0x7f9fdc148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:33614/0 (socket says 172.21.15.187:33614) 2020-11-20T12:05:43.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.842+0000 7f9fe1b4c700 1 -- 172.21.15.187:0/2054148239 learned_addr learned my addr 172.21.15.187:0/2054148239 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:43.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.842+0000 7f9fe0b4a700 1 --1- 172.21.15.187:0/2054148239 >> v1:172.21.15.5:6789/0 conn(0x7f9fdc0683f0 0x7f9fdc065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:59694/0 (socket says 172.21.15.187:59694) 2020-11-20T12:05:43.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.843+0000 7f9fe1b4c700 1 -- 172.21.15.187:0/2054148239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 msgr2=0x7f9fdc066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:43.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.843+0000 7f9fe1b4c700 1 --2- 172.21.15.187:0/2054148239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 0x7f9fdc066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.843+0000 7f9fe1b4c700 1 -- 172.21.15.187:0/2054148239 >> v1:172.21.15.5:6789/0 conn(0x7f9fdc0683f0 legacy=0x7f9fdc065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.843+0000 7f9fe1b4c700 1 -- 172.21.15.187:0/2054148239 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc0658a0 con 0x7f9fdc1487e0 2020-11-20T12:05:43.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.844+0000 7f9fe1b4c700 1 --2- 172.21.15.187:0/2054148239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc148c90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f9fd4006fd0 tx=0x7f9fd4006eb0).ready entity=mon.0 client_cookie=2e370ae9867f56b3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:43.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.844+0000 7f9fcbfff700 1 -- 172.21.15.187:0/2054148239 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4009810 con 0x7f9fdc1487e0 2020-11-20T12:05:43.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.844+0000 7f9fcbfff700 1 -- 172.21.15.187:0/2054148239 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd4009970 con 0x7f9fdc1487e0 2020-11-20T12:05:43.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.844+0000 7f9fcbfff700 1 -- 172.21.15.187:0/2054148239 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4009cf0 con 0x7f9fdc1487e0 2020-11-20T12:05:43.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.845+0000 7f9fe234d700 1 -- 172.21.15.187:0/2054148239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc1487e0 msgr2=0x7f9fdc148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.845+0000 7f9fe234d700 1 --2- 172.21.15.187:0/2054148239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc148c90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f9fd4006fd0 tx=0x7f9fd4006eb0).stop 2020-11-20T12:05:43.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.845+0000 7f9fe234d700 1 -- 172.21.15.187:0/2054148239 shutdown_connections 2020-11-20T12:05:43.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.845+0000 7f9fe234d700 1 --2- 172.21.15.187:0/2054148239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc148c90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.845+0000 7f9fe234d700 1 --2- 172.21.15.187:0/2054148239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 0x7f9fdc066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.845+0000 7f9fe234d700 1 -- 172.21.15.187:0/2054148239 >> 172.21.15.187:0/2054148239 conn(0x7f9fdc0bab30 msgr2=0x7f9fdc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:43.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.845+0000 7f9fe234d700 1 -- 172.21.15.187:0/2054148239 shutdown_connections 2020-11-20T12:05:43.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.845+0000 7f9fe234d700 1 -- 172.21.15.187:0/2054148239 wait complete. 2020-11-20T12:05:43.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.847+0000 7f9fe234d700 1 Processor -- start 2020-11-20T12:05:43.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.847+0000 7f9fe234d700 1 -- start start 2020-11-20T12:05:43.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe234d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc0683f0 0x7f9fdc09c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe234d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 0x7f9fdc1576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe234d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc0b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe234d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc14b2b0 con 0x7f9fdc0683f0 2020-11-20T12:05:43.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe234d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc144950 con 0x7f9fdc1487e0 2020-11-20T12:05:43.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe234d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc09c9f0 con 0x7f9fdc145cc0 2020-11-20T12:05:43.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe1b4c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc0b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe1b4c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc0b7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41354/0 (socket says 172.21.15.187:41354) 2020-11-20T12:05:43.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.848+0000 7f9fe1b4c700 1 -- 172.21.15.187:0/1150178848 learned_addr learned my addr 172.21.15.187:0/1150178848 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:43.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe0b4a700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 0x7f9fdc1576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe134b700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc0683f0 0x7f9fdc09c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe1b4c700 1 -- 172.21.15.187:0/1150178848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 msgr2=0x7f9fdc1576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe1b4c700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 0x7f9fdc1576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe1b4c700 1 -- 172.21.15.187:0/1150178848 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc0683f0 msgr2=0x7f9fdc09c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:43.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe1b4c700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc0683f0 0x7f9fdc09c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:43.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe1b4c700 1 -- 172.21.15.187:0/1150178848 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc14b4a0 con 0x7f9fdc1487e0 2020-11-20T12:05:43.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe1b4c700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc0b7190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f9fd4006ef0 tx=0x7f9fd4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:43.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fc97fa700 1 -- 172.21.15.187:0/1150178848 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4006b20 con 0x7f9fdc1487e0 2020-11-20T12:05:43.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fc97fa700 1 -- 172.21.15.187:0/1150178848 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd401d5d0 con 0x7f9fdc1487e0 2020-11-20T12:05:43.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fdc14b740 con 0x7f9fdc1487e0 2020-11-20T12:05:43.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.849+0000 7f9fc97fa700 1 -- 172.21.15.187:0/1150178848 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd401d970 con 0x7f9fdc1487e0 2020-11-20T12:05:43.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.850+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fdc0b7680 con 0x7f9fdc1487e0 2020-11-20T12:05:43.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.851+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fdc14b8d0 con 0x7f9fdc1487e0 2020-11-20T12:05:43.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.851+0000 7f9fc97fa700 1 -- 172.21.15.187:0/1150178848 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9fd4025030 con 0x7f9fdc1487e0 2020-11-20T12:05:43.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.852+0000 7f9fc97fa700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9fb4056d20 0x7f9fb40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:43.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.852+0000 7f9fc97fa700 1 -- 172.21.15.187:0/1150178848 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9fd40719d0 con 0x7f9fdc1487e0 2020-11-20T12:05:43.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.852+0000 7f9fe134b700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9fb4056d20 0x7f9fb40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:43.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.853+0000 7f9fe134b700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9fb4056d20 0x7f9fb40591e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9fd8009d90 tx=0x7f9fd8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:43.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:43.856+0000 7f9fc97fa700 1 -- 172.21.15.187:0/1150178848 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9fd4046dc0 con 0x7f9fdc1487e0 2020-11-20T12:05:44.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.031+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9fdc061a20 con 0x7f9fdc1487e0 2020-11-20T12:05:44.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.032+0000 7f9fc97fa700 1 -- 172.21.15.187:0/1150178848 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9fd4023030 con 0x7f9fdc1487e0 2020-11-20T12:05:44.036 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:44.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9fb4056d20 msgr2=0x7f9fb40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9fb4056d20 0x7f9fb40591e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9fd8009d90 tx=0x7f9fd8008040).stop 2020-11-20T12:05:44.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fdc1487e0 msgr2=0x7f9fdc0b7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc0b7190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f9fd4006ef0 tx=0x7f9fd4015000).stop 2020-11-20T12:05:44.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 shutdown_connections 2020-11-20T12:05:44.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9fb4056d20 0x7f9fb40591e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9fdc0683f0 0x7f9fdc09c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fdc1487e0 0x7f9fdc0b7190 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 --2- 172.21.15.187:0/1150178848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9fdc145cc0 0x7f9fdc1576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 >> 172.21.15.187:0/1150178848 conn(0x7f9fdc0bab30 msgr2=0x7f9fdc145040 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:44.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 shutdown_connections 2020-11-20T12:05:44.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.034+0000 7f9fe234d700 1 -- 172.21.15.187:0/1150178848 wait complete. 2020-11-20T12:05:44.059 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:44.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.291+0000 7ff7ceed0700 1 Processor -- start 2020-11-20T12:05:44.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.291+0000 7ff7ceed0700 1 -- start start 2020-11-20T12:05:44.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.292+0000 7ff7ceed0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c80683f0 0x7ff7c8066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.292+0000 7ff7ceed0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7c81487e0 0x7ff7c8148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.292+0000 7ff7ceed0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c814c170 con 0x7ff7c8145cc0 2020-11-20T12:05:44.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.292+0000 7ff7ceed0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c814c410 con 0x7ff7c81487e0 2020-11-20T12:05:44.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.293+0000 7ff7ceed0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c814c6b0 con 0x7ff7c80683f0 2020-11-20T12:05:44.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.293+0000 7ff7ce6cf700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7c81487e0 0x7ff7c8148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:44.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.293+0000 7ff7ce6cf700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7c81487e0 0x7ff7c8148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41360/0 (socket says 172.21.15.187:41360) 2020-11-20T12:05:44.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.293+0000 7ff7ce6cf700 1 -- 172.21.15.187:0/1839435993 learned_addr learned my addr 172.21.15.187:0/1839435993 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:44.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.294+0000 7ff7ce6cf700 1 -- 172.21.15.187:0/1839435993 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c80683f0 msgr2=0x7ff7c8066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:44.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.294+0000 7ff7ce6cf700 1 --2- 172.21.15.187:0/1839435993 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c80683f0 0x7ff7c8066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:44.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.294+0000 7ff7ce6cf700 1 -- 172.21.15.187:0/1839435993 >> v1:172.21.15.5:6789/0 conn(0x7ff7c8145cc0 legacy=0x7ff7c8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:44.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.294+0000 7ff7ce6cf700 1 -- 172.21.15.187:0/1839435993 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c80658a0 con 0x7ff7c81487e0 2020-11-20T12:05:44.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.294+0000 7ff7ce6cf700 1 --2- 172.21.15.187:0/1839435993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c81487e0 0x7ff7c8148c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff7c0009d50 tx=0x7ff7c0009290).ready entity=mon.1 client_cookie=b6fe062662964386 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:44.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.294+0000 7ff7ccecc700 1 -- 172.21.15.187:0/1839435993 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c000bc40 con 0x7ff7c81487e0 2020-11-20T12:05:44.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.294+0000 7ff7ccecc700 1 -- 172.21.15.187:0/1839435993 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c000bda0 con 0x7ff7c81487e0 2020-11-20T12:05:44.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.294+0000 7ff7ccecc700 1 -- 172.21.15.187:0/1839435993 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0011610 con 0x7ff7c81487e0 2020-11-20T12:05:44.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.295+0000 7ff7ceed0700 1 -- 172.21.15.187:0/1839435993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c81487e0 msgr2=0x7ff7c8148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.295+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/1839435993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c81487e0 0x7ff7c8148c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff7c0009d50 tx=0x7ff7c0009290).stop 2020-11-20T12:05:44.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.295+0000 7ff7ceed0700 1 -- 172.21.15.187:0/1839435993 shutdown_connections 2020-11-20T12:05:44.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.295+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/1839435993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c81487e0 0x7ff7c8148c90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.295+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/1839435993 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c80683f0 0x7ff7c8066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:44.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.295+0000 7ff7ceed0700 1 -- 172.21.15.187:0/1839435993 >> 172.21.15.187:0/1839435993 conn(0x7ff7c80bab30 msgr2=0x7ff7c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:44.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.295+0000 7ff7ceed0700 1 -- 172.21.15.187:0/1839435993 shutdown_connections 2020-11-20T12:05:44.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.295+0000 7ff7ceed0700 1 -- 172.21.15.187:0/1839435993 wait complete. 2020-11-20T12:05:44.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.297+0000 7ff7ceed0700 1 Processor -- start 2020-11-20T12:05:44.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.297+0000 7ff7ceed0700 1 -- start start 2020-11-20T12:05:44.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.298+0000 7ff7ceed0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c80683f0 0x7ff7c809dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7ceed0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145cc0 0x7ff7c8155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7ceed0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c81487e0 0x7ff7c809ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7ceed0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8158a30 con 0x7ff7c81487e0 2020-11-20T12:05:44.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7ceed0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8158cd0 con 0x7ff7c8145cc0 2020-11-20T12:05:44.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7ceed0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8155810 con 0x7ff7c80683f0 2020-11-20T12:05:44.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7cd6cd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145cc0 0x7ff7c8155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:44.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7cd6cd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145cc0 0x7ff7c8155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41368/0 (socket says 172.21.15.187:41368) 2020-11-20T12:05:44.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7cd6cd700 1 -- 172.21.15.187:0/3370328212 learned_addr learned my addr 172.21.15.187:0/3370328212 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:44.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7cd6cd700 1 -- 172.21.15.187:0/3370328212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c80683f0 msgr2=0x7ff7c809dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7cd6cd700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c80683f0 0x7ff7c809dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:44.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7cd6cd700 1 -- 172.21.15.187:0/3370328212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c81487e0 msgr2=0x7ff7c809ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7cd6cd700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c81487e0 0x7ff7c809ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:44.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.299+0000 7ff7cd6cd700 1 -- 172.21.15.187:0/3370328212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c809b520 con 0x7ff7c8145cc0 2020-11-20T12:05:44.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.300+0000 7ff7cd6cd700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145cc0 0x7ff7c8155320 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7ff7b8009770 tx=0x7ff7b8006ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:44.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.300+0000 7ff7b5ffb700 1 -- 172.21.15.187:0/3370328212 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b800d640 con 0x7ff7c8145cc0 2020-11-20T12:05:44.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.300+0000 7ff7b5ffb700 1 -- 172.21.15.187:0/3370328212 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b8000e90 con 0x7ff7c8145cc0 2020-11-20T12:05:44.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.300+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c81586c0 con 0x7ff7c8145cc0 2020-11-20T12:05:44.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.300+0000 7ff7b5ffb700 1 -- 172.21.15.187:0/3370328212 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b800da10 con 0x7ff7c8145cc0 2020-11-20T12:05:44.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.300+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c80a0210 con 0x7ff7c8145cc0 2020-11-20T12:05:44.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.301+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c809b710 con 0x7ff7c8145cc0 2020-11-20T12:05:44.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.302+0000 7ff7b5ffb700 1 -- 172.21.15.187:0/3370328212 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff7b8017030 con 0x7ff7c8145cc0 2020-11-20T12:05:44.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.302+0000 7ff7b5ffb700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7a0056d20 0x7ff7a00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.302+0000 7ff7b5ffb700 1 -- 172.21.15.187:0/3370328212 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff7b806f100 con 0x7ff7c8145cc0 2020-11-20T12:05:44.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.303+0000 7ff7cdece700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7a0056d20 0x7ff7a00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:44.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.303+0000 7ff7cdece700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7a0056d20 0x7ff7a00591e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff7c815b2f0 tx=0x7ff7c40077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:44.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.307+0000 7ff7b5ffb700 1 -- 172.21.15.187:0/3370328212 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff7b8044600 con 0x7ff7c8145cc0 2020-11-20T12:05:44.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.494+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff7c8061a20 con 0x7ff7c8145cc0 2020-11-20T12:05:44.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.495+0000 7ff7b5ffb700 1 -- 172.21.15.187:0/3370328212 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff7b8047c20 con 0x7ff7c8145cc0 2020-11-20T12:05:44.500 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:44.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7a0056d20 msgr2=0x7ff7a00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7a0056d20 0x7ff7a00591e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff7c815b2f0 tx=0x7ff7c40077f0).stop 2020-11-20T12:05:44.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145cc0 msgr2=0x7ff7c8155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145cc0 0x7ff7c8155320 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7ff7b8009770 tx=0x7ff7b8006ec0).stop 2020-11-20T12:05:44.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 shutdown_connections 2020-11-20T12:05:44.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7a0056d20 0x7ff7a00591e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7c81487e0 0x7ff7c809ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:44.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7c80683f0 0x7ff7c809dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:44.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 --2- 172.21.15.187:0/3370328212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145cc0 0x7ff7c8155320 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 >> 172.21.15.187:0/3370328212 conn(0x7ff7c80bab30 msgr2=0x7ff7c8144e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:44.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 shutdown_connections 2020-11-20T12:05:44.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.498+0000 7ff7ceed0700 1 -- 172.21.15.187:0/3370328212 wait complete. 2020-11-20T12:05:44.526 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:44.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.767+0000 7f6754e8e700 1 Processor -- start 2020-11-20T12:05:44.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.767+0000 7f6754e8e700 1 -- start start 2020-11-20T12:05:44.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.769+0000 7f6754e8e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f67500b1760 0x7f6750065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.769+0000 7f6754e8e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6750150390 con 0x7f67500b1380 2020-11-20T12:05:44.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.769+0000 7f6754e8e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6750150630 con 0x7f6750065680 2020-11-20T12:05:44.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.769+0000 7f6754e8e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67501511e0 con 0x7f67500b1760 2020-11-20T12:05:44.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.769+0000 7f6746ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f67500b1760 0x7f6750065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:44.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.769+0000 7f674ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6750065680 0x7f675014c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:37984/0 (socket says 172.21.15.187:37984) 2020-11-20T12:05:44.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.769+0000 7f674ffff700 1 -- 172.21.15.187:0/3601145802 learned_addr learned my addr 172.21.15.187:0/3601145802 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:44.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.770+0000 7f6746ffd700 1 -- 172.21.15.187:0/3601145802 >> v1:172.21.15.187:6789/0 conn(0x7f6750065680 legacy=0x7f675014c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.770+0000 7f6746ffd700 1 -- 172.21.15.187:0/3601145802 >> v1:172.21.15.5:6789/0 conn(0x7f67500b1380 legacy=0x7f6750150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:44.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.770+0000 7f6746ffd700 1 -- 172.21.15.187:0/3601145802 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6750068510 con 0x7f67500b1760 2020-11-20T12:05:44.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.770+0000 7f6746ffd700 1 --2- 172.21.15.187:0/3601145802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b1760 0x7f6750065190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f673800b920 tx=0x7f6738006df0).ready entity=mon.1 client_cookie=d12b0fb9ff8220cf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:44.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.770+0000 7f674effd700 1 -- 172.21.15.187:0/3601145802 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738009730 con 0x7f67500b1760 2020-11-20T12:05:44.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.770+0000 7f674effd700 1 -- 172.21.15.187:0/3601145802 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738009890 con 0x7f67500b1760 2020-11-20T12:05:44.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.770+0000 7f674effd700 1 -- 172.21.15.187:0/3601145802 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738009c10 con 0x7f67500b1760 2020-11-20T12:05:44.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.771+0000 7f6754e8e700 1 -- 172.21.15.187:0/3601145802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b1760 msgr2=0x7f6750065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.771+0000 7f6754e8e700 1 --2- 172.21.15.187:0/3601145802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b1760 0x7f6750065190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f673800b920 tx=0x7f6738006df0).stop 2020-11-20T12:05:44.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.771+0000 7f6754e8e700 1 -- 172.21.15.187:0/3601145802 shutdown_connections 2020-11-20T12:05:44.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.771+0000 7f6754e8e700 1 --2- 172.21.15.187:0/3601145802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b1760 0x7f6750065190 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.771+0000 7f6754e8e700 1 -- 172.21.15.187:0/3601145802 >> 172.21.15.187:0/3601145802 conn(0x7f67500bab30 msgr2=0x7f67500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:44.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.771+0000 7f6754e8e700 1 -- 172.21.15.187:0/3601145802 shutdown_connections 2020-11-20T12:05:44.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.771+0000 7f6754e8e700 1 -- 172.21.15.187:0/3601145802 wait complete. 2020-11-20T12:05:44.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.772+0000 7f6754e8e700 1 Processor -- start 2020-11-20T12:05:44.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.773+0000 7f6754e8e700 1 -- start start 2020-11-20T12:05:44.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.773+0000 7f6754e8e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67500b1380 0x7f67500b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.773+0000 7f6754e8e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67500b1760 0x7f6750148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.773+0000 7f6754e8e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b2dc0 0x7f6750148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.773+0000 7f6754e8e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6750149250 con 0x7f67500b1380 2020-11-20T12:05:44.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.773+0000 7f6754e8e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675009b210 con 0x7f67500b2dc0 2020-11-20T12:05:44.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.773+0000 7f6754e8e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675009b440 con 0x7f67500b1760 2020-11-20T12:05:44.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b2dc0 0x7f6750148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:44.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b2dc0 0x7f6750148d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41384/0 (socket says 172.21.15.187:41384) 2020-11-20T12:05:44.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 -- 172.21.15.187:0/3607434918 learned_addr learned my addr 172.21.15.187:0/3607434918 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:44.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f6746ffd700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67500b1760 0x7f6750148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:44.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 -- 172.21.15.187:0/3607434918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67500b1760 msgr2=0x7f6750148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67500b1760 0x7f6750148870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 -- 172.21.15.187:0/3607434918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67500b1380 msgr2=0x7f67500b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674f7fe700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67500b1380 0x7f67500b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:44.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67500b1380 0x7f67500b28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 -- 172.21.15.187:0/3607434918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67500b7800 con 0x7f67500b2dc0 2020-11-20T12:05:44.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.774+0000 7f674ffff700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b2dc0 0x7f6750148d60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f67480012a0 tx=0x7f6748007ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:44.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.775+0000 7f6747fff700 1 -- 172.21.15.187:0/3607434918 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748005b80 con 0x7f67500b2dc0 2020-11-20T12:05:44.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.775+0000 7f6747fff700 1 -- 172.21.15.187:0/3607434918 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6748005ce0 con 0x7f67500b2dc0 2020-11-20T12:05:44.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.775+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675009b8e0 con 0x7f67500b2dc0 2020-11-20T12:05:44.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.775+0000 7f6747fff700 1 -- 172.21.15.187:0/3607434918 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748007360 con 0x7f67500b2dc0 2020-11-20T12:05:44.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.775+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675009be70 con 0x7f67500b2dc0 2020-11-20T12:05:44.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.776+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675009b600 con 0x7f67500b2dc0 2020-11-20T12:05:44.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.777+0000 7f6747fff700 1 -- 172.21.15.187:0/3607434918 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6748019020 con 0x7f67500b2dc0 2020-11-20T12:05:44.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.778+0000 7f6747fff700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f672c056d50 0x7f672c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:44.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.778+0000 7f6747fff700 1 -- 172.21.15.187:0/3607434918 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f674806fc20 con 0x7f67500b2dc0 2020-11-20T12:05:44.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.778+0000 7f674f7fe700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f672c056d50 0x7f672c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:44.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.779+0000 7f674f7fe700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f672c056d50 0x7f672c059210 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f6740002410 tx=0x7f6740009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:44.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.782+0000 7f6747fff700 1 -- 172.21.15.187:0/3607434918 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6748045070 con 0x7f67500b2dc0 2020-11-20T12:05:44.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.953+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6750061a20 con 0x7f67500b2dc0 2020-11-20T12:05:44.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.956+0000 7f6747fff700 1 -- 172.21.15.187:0/3607434918 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f6748048690 con 0x7f67500b2dc0 2020-11-20T12:05:44.960 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:44.960 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:44.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f672c056d50 msgr2=0x7f672c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f672c056d50 0x7f672c059210 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f6740002410 tx=0x7f6740009040).stop 2020-11-20T12:05:44.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b2dc0 msgr2=0x7f6750148d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:44.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b2dc0 0x7f6750148d60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f67480012a0 tx=0x7f6748007ef0).stop 2020-11-20T12:05:44.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 shutdown_connections 2020-11-20T12:05:44.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f672c056d50 0x7f672c059210 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67500b1380 0x7f67500b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67500b2dc0 0x7f6750148d60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 --2- 172.21.15.187:0/3607434918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67500b1760 0x7f6750148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:44.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 >> 172.21.15.187:0/3607434918 conn(0x7f67500bab30 msgr2=0x7f67500baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:44.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 shutdown_connections 2020-11-20T12:05:44.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:44.959+0000 7f6754e8e700 1 -- 172.21.15.187:0/3607434918 wait complete. 2020-11-20T12:05:44.983 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:44.984 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:45.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.219+0000 7f9af48bd700 1 Processor -- start 2020-11-20T12:05:45.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.219+0000 7f9af48bd700 1 -- start start 2020-11-20T12:05:45.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.221+0000 7f9af48bd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9af0145cc0 0x7f9af0066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.221+0000 7f9af48bd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9af00683f0 0x7f9af0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.221+0000 7f9af48bd700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af014c170 con 0x7f9af0065680 2020-11-20T12:05:45.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.221+0000 7f9af48bd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af01497f0 con 0x7f9af0145cc0 2020-11-20T12:05:45.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.221+0000 7f9af48bd700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af014c410 con 0x7f9af00683f0 2020-11-20T12:05:45.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.221+0000 7f9ae659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9af00683f0 0x7f9af0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.221+0000 7f9ae659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9af00683f0 0x7f9af0065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41390/0 (socket says 172.21.15.187:41390) 2020-11-20T12:05:45.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.221+0000 7f9ae659c700 1 -- 172.21.15.187:0/2320743119 learned_addr learned my addr 172.21.15.187:0/2320743119 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:45.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.222+0000 7f9aeed9d700 1 --2- 172.21.15.187:0/2320743119 >> v2:172.21.15.5:3300/0 conn(0x7f9af0145cc0 0x7f9af0066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.222+0000 7f9ae659c700 1 -- 172.21.15.187:0/2320743119 >> v2:172.21.15.5:3300/0 conn(0x7f9af0145cc0 msgr2=0x7f9af0066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:45.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.222+0000 7f9ae659c700 1 --2- 172.21.15.187:0/2320743119 >> v2:172.21.15.5:3300/0 conn(0x7f9af0145cc0 0x7f9af0066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.222+0000 7f9ae659c700 1 -- 172.21.15.187:0/2320743119 >> v1:172.21.15.187:6789/0 conn(0x7f9af0065680 legacy=0x7f9af0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.222+0000 7f9ae659c700 1 -- 172.21.15.187:0/2320743119 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af014c6b0 con 0x7f9af00683f0 2020-11-20T12:05:45.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.222+0000 7f9ae659c700 1 --2- 172.21.15.187:0/2320743119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af00683f0 0x7f9af0065190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f9af00674b0 tx=0x7f9ad8006eb0).ready entity=mon.1 client_cookie=3f6f36cb3f30bdb1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:45.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9aee59c700 1 -- 172.21.15.187:0/2320743119 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad80098d0 con 0x7f9af00683f0 2020-11-20T12:05:45.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9aee59c700 1 -- 172.21.15.187:0/2320743119 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad8009a30 con 0x7f9af00683f0 2020-11-20T12:05:45.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9aee59c700 1 -- 172.21.15.187:0/2320743119 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8009db0 con 0x7f9af00683f0 2020-11-20T12:05:45.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9af48bd700 1 -- 172.21.15.187:0/2320743119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af00683f0 msgr2=0x7f9af0065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9af48bd700 1 --2- 172.21.15.187:0/2320743119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af00683f0 0x7f9af0065190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f9af00674b0 tx=0x7f9ad8006eb0).stop 2020-11-20T12:05:45.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9af48bd700 1 -- 172.21.15.187:0/2320743119 shutdown_connections 2020-11-20T12:05:45.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9af48bd700 1 --2- 172.21.15.187:0/2320743119 >> v2:172.21.15.5:3300/0 conn(0x7f9af0145cc0 0x7f9af0066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9af48bd700 1 --2- 172.21.15.187:0/2320743119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af00683f0 0x7f9af0065190 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9af48bd700 1 -- 172.21.15.187:0/2320743119 >> 172.21.15.187:0/2320743119 conn(0x7f9af00bab30 msgr2=0x7f9af00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:45.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.223+0000 7f9af48bd700 1 -- 172.21.15.187:0/2320743119 shutdown_connections 2020-11-20T12:05:45.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.224+0000 7f9af48bd700 1 -- 172.21.15.187:0/2320743119 wait complete. 2020-11-20T12:05:45.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.225+0000 7f9af48bd700 1 Processor -- start 2020-11-20T12:05:45.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.226+0000 7f9af48bd700 1 -- start start 2020-11-20T12:05:45.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9af48bd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af0065680 0x7f9af01440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9af48bd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af00683f0 0x7f9af009af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9af48bd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9af0145cc0 0x7f9af0147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9af48bd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af009a7f0 con 0x7f9af0145cc0 2020-11-20T12:05:45.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9af48bd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af009aa90 con 0x7f9af0065680 2020-11-20T12:05:45.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9af48bd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af0144590 con 0x7f9af00683f0 2020-11-20T12:05:45.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9aeed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af0065680 0x7f9af01440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9aeed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af0065680 0x7f9af01440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41394/0 (socket says 172.21.15.187:41394) 2020-11-20T12:05:45.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9aeed9d700 1 -- 172.21.15.187:0/1782028123 learned_addr learned my addr 172.21.15.187:0/1782028123 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:45.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9aeed9d700 1 -- 172.21.15.187:0/1782028123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af00683f0 msgr2=0x7f9af009af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9aeed9d700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af00683f0 0x7f9af009af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:45.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9aeed9d700 1 -- 172.21.15.187:0/1782028123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9af0145cc0 msgr2=0x7f9af0147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9aeed9d700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9af0145cc0 0x7f9af0147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:45.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.227+0000 7f9aeed9d700 1 -- 172.21.15.187:0/1782028123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af009b420 con 0x7f9af0065680 2020-11-20T12:05:45.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.228+0000 7f9aeed9d700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af0065680 0x7f9af01440a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f9ae000d840 tx=0x7f9ae0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:45.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.228+0000 7f9ae77fe700 1 -- 172.21.15.187:0/1782028123 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae0009640 con 0x7f9af0065680 2020-11-20T12:05:45.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.228+0000 7f9ae77fe700 1 -- 172.21.15.187:0/1782028123 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae00097a0 con 0x7f9af0065680 2020-11-20T12:05:45.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.228+0000 7f9ae77fe700 1 -- 172.21.15.187:0/1782028123 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae0009b40 con 0x7f9af0065680 2020-11-20T12:05:45.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.228+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af00a0960 con 0x7f9af0065680 2020-11-20T12:05:45.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.228+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af00afcb0 con 0x7f9af0065680 2020-11-20T12:05:45.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.229+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af00a0b50 con 0x7f9af0065680 2020-11-20T12:05:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.230+0000 7f9ae77fe700 1 -- 172.21.15.187:0/1782028123 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9ae0013480 con 0x7f9af0065680 2020-11-20T12:05:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.230+0000 7f9ae77fe700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9acc056d20 0x7f9acc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.230+0000 7f9ae77fe700 1 -- 172.21.15.187:0/1782028123 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9ae0070370 con 0x7f9af0065680 2020-11-20T12:05:45.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.231+0000 7f9ae659c700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9acc056d20 0x7f9acc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.231+0000 7f9ae659c700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9acc056d20 0x7f9acc0591e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9af0147ad0 tx=0x7f9ad8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.236+0000 7f9ae77fe700 1 -- 172.21.15.187:0/1782028123 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9ae0045760 con 0x7f9af0065680 2020-11-20T12:05:45.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.421+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f9af00afe40 con 0x7f9af0065680 2020-11-20T12:05:45.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.422+0000 7f9ae77fe700 1 -- 172.21.15.187:0/1782028123 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f9ae0048d80 con 0x7f9af0065680 2020-11-20T12:05:45.426 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:45.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.424+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9acc056d20 msgr2=0x7f9acc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.424+0000 7f9af48bd700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9acc056d20 0x7f9acc0591e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9af0147ad0 tx=0x7f9ad8015000).stop 2020-11-20T12:05:45.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.424+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af0065680 msgr2=0x7f9af01440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.424+0000 7f9af48bd700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af0065680 0x7f9af01440a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f9ae000d840 tx=0x7f9ae0008040).stop 2020-11-20T12:05:45.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.424+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 shutdown_connections 2020-11-20T12:05:45.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.425+0000 7f9af48bd700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9acc056d20 0x7f9acc0591e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.425+0000 7f9af48bd700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9af0145cc0 0x7f9af0147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:45.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.425+0000 7f9af48bd700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af0065680 0x7f9af01440a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.425+0000 7f9af48bd700 1 --2- 172.21.15.187:0/1782028123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af00683f0 0x7f9af009af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:45.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.425+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 >> 172.21.15.187:0/1782028123 conn(0x7f9af00bab30 msgr2=0x7f9af00b6740 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:45.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.425+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 shutdown_connections 2020-11-20T12:05:45.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.425+0000 7f9af48bd700 1 -- 172.21.15.187:0/1782028123 wait complete. 2020-11-20T12:05:45.450 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:45.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.689+0000 7faefbfff700 1 Processor -- start 2020-11-20T12:05:45.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.689+0000 7faefbfff700 1 -- start start 2020-11-20T12:05:45.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.691+0000 7faefbfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faefc0bf800 0x7faefc14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.691+0000 7faefbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faefc09f560 0x7faefc0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.691+0000 7faefbfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc14d110 con 0x7faefc0bfbe0 2020-11-20T12:05:45.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.691+0000 7faefbfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faefc14cbd0 con 0x7faefc0bf800 2020-11-20T12:05:45.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.691+0000 7faefbfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faefc14ce70 con 0x7faefc09f560 2020-11-20T12:05:45.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.691+0000 7faefb7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faefc09f560 0x7faefc0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.691+0000 7faefb7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faefc09f560 0x7faefc0a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41406/0 (socket says 172.21.15.187:41406) 2020-11-20T12:05:45.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.691+0000 7faefb7fe700 1 -- 172.21.15.187:0/3142124381 learned_addr learned my addr 172.21.15.187:0/3142124381 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:45.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.692+0000 7faefb7fe700 1 -- 172.21.15.187:0/3142124381 >> v2:172.21.15.5:3300/0 conn(0x7faefc0bf800 msgr2=0x7faefc14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:45.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.692+0000 7faefb7fe700 1 --2- 172.21.15.187:0/3142124381 >> v2:172.21.15.5:3300/0 conn(0x7faefc0bf800 0x7faefc14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:45.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.692+0000 7faefb7fe700 1 -- 172.21.15.187:0/3142124381 >> v1:172.21.15.5:6789/0 conn(0x7faefc0bfbe0 legacy=0x7faefc1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:45.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.692+0000 7faefb7fe700 1 -- 172.21.15.187:0/3142124381 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc0a4cc0 con 0x7faefc09f560 2020-11-20T12:05:45.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.693+0000 7faefb7fe700 1 --2- 172.21.15.187:0/3142124381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc09f560 0x7faefc0a36e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7faefc09f2b0 tx=0x7faef4006eb0).ready entity=mon.1 client_cookie=95bef220ff1b45af server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:45.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.693+0000 7faef9ffb700 1 -- 172.21.15.187:0/3142124381 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4009920 con 0x7faefc09f560 2020-11-20T12:05:45.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.694+0000 7faef9ffb700 1 -- 172.21.15.187:0/3142124381 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef4009a80 con 0x7faefc09f560 2020-11-20T12:05:45.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.694+0000 7faef9ffb700 1 -- 172.21.15.187:0/3142124381 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4009e00 con 0x7faefc09f560 2020-11-20T12:05:45.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.694+0000 7faefbfff700 1 -- 172.21.15.187:0/3142124381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc09f560 msgr2=0x7faefc0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.694+0000 7faefbfff700 1 --2- 172.21.15.187:0/3142124381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc09f560 0x7faefc0a36e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7faefc09f2b0 tx=0x7faef4006eb0).stop 2020-11-20T12:05:45.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.695+0000 7faefbfff700 1 -- 172.21.15.187:0/3142124381 shutdown_connections 2020-11-20T12:05:45.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.695+0000 7faefbfff700 1 --2- 172.21.15.187:0/3142124381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc09f560 0x7faefc0a36e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.695+0000 7faefbfff700 1 --2- 172.21.15.187:0/3142124381 >> v2:172.21.15.5:3300/0 conn(0x7faefc0bf800 0x7faefc14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:45.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.695+0000 7faefbfff700 1 -- 172.21.15.187:0/3142124381 >> 172.21.15.187:0/3142124381 conn(0x7faefc068370 msgr2=0x7faefc0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:45.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.695+0000 7faefbfff700 1 -- 172.21.15.187:0/3142124381 shutdown_connections 2020-11-20T12:05:45.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.695+0000 7faefbfff700 1 -- 172.21.15.187:0/3142124381 wait complete. 2020-11-20T12:05:45.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.697+0000 7faefbfff700 1 Processor -- start 2020-11-20T12:05:45.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.697+0000 7faefbfff700 1 -- start start 2020-11-20T12:05:45.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faefc09f560 0x7faefc0b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc0bf800 0x7faefc140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faefc1449f0 0x7faefc14b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefbfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc144e90 con 0x7faefc1449f0 2020-11-20T12:05:45.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefbfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc145130 con 0x7faefc0bf800 2020-11-20T12:05:45.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefbfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faefc1453d0 con 0x7faefc09f560 2020-11-20T12:05:45.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefa7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc0bf800 0x7faefc140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefa7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc0bf800 0x7faefc140d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41410/0 (socket says 172.21.15.187:41410) 2020-11-20T12:05:45.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.698+0000 7faefa7fc700 1 -- 172.21.15.187:0/2132745664 learned_addr learned my addr 172.21.15.187:0/2132745664 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:45.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefaffd700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faefc09f560 0x7faefc0b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefb7fe700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faefc1449f0 0x7faefc14b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefa7fc700 1 -- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faefc09f560 msgr2=0x7faefc0b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefa7fc700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faefc09f560 0x7faefc0b55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefa7fc700 1 -- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faefc1449f0 msgr2=0x7faefc14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefa7fc700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faefc1449f0 0x7faefc14b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefa7fc700 1 -- 172.21.15.187:0/2132745664 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc1411f0 con 0x7faefc0bf800 2020-11-20T12:05:45.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefb7fe700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faefc1449f0 0x7faefc14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:45.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefa7fc700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc0bf800 0x7faefc140d00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7faef00096b0 tx=0x7faef0006d50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:45.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faee2ffd700 1 -- 172.21.15.187:0/2132745664 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef000d5c0 con 0x7faefc0bf800 2020-11-20T12:05:45.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faee2ffd700 1 -- 172.21.15.187:0/2132745664 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef0000e90 con 0x7faefc0bf800 2020-11-20T12:05:45.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc0b6620 con 0x7faefc0bf800 2020-11-20T12:05:45.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.699+0000 7faee2ffd700 1 -- 172.21.15.187:0/2132745664 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef000d970 con 0x7faefc0bf800 2020-11-20T12:05:45.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.700+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faefc14c0a0 con 0x7faefc0bf800 2020-11-20T12:05:45.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.701+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faefc080050 con 0x7faefc0bf800 2020-11-20T12:05:45.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.701+0000 7faee2ffd700 1 -- 172.21.15.187:0/2132745664 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faef000dce0 con 0x7faefc0bf800 2020-11-20T12:05:45.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.702+0000 7faee2ffd700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faed8056d20 0x7faed80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:45.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.702+0000 7faee2ffd700 1 -- 172.21.15.187:0/2132745664 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faef006f950 con 0x7faefc0bf800 2020-11-20T12:05:45.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.702+0000 7faefaffd700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faed8056d20 0x7faed80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:45.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.703+0000 7faefaffd700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faed8056d20 0x7faed80591e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7faefc0b50f0 tx=0x7faeec008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:45.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.706+0000 7faee2ffd700 1 -- 172.21.15.187:0/2132745664 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faef0044c30 con 0x7faefc0bf800 2020-11-20T12:05:45.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.891+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7faefc145590 con 0x7faefc0bf800 2020-11-20T12:05:45.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.892+0000 7faee2ffd700 1 -- 172.21.15.187:0/2132745664 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7faef0048250 con 0x7faefc0bf800 2020-11-20T12:05:45.896 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:45.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faed8056d20 msgr2=0x7faed80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faed8056d20 0x7faed80591e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7faefc0b50f0 tx=0x7faeec008040).stop 2020-11-20T12:05:45.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc0bf800 msgr2=0x7faefc140d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:45.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc0bf800 0x7faefc140d00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7faef00096b0 tx=0x7faef0006d50).stop 2020-11-20T12:05:45.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 shutdown_connections 2020-11-20T12:05:45.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faed8056d20 0x7faed80591e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faefc1449f0 0x7faefc14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faefc09f560 0x7faefc0b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 --2- 172.21.15.187:0/2132745664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faefc0bf800 0x7faefc140d00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:45.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.894+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 >> 172.21.15.187:0/2132745664 conn(0x7faefc068370 msgr2=0x7faefc0bb380 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:45.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.895+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 shutdown_connections 2020-11-20T12:05:45.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:45.895+0000 7faefbfff700 1 -- 172.21.15.187:0/2132745664 wait complete. 2020-11-20T12:05:45.923 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:46.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.149+0000 7f0e9575f700 1 Processor -- start 2020-11-20T12:05:46.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.149+0000 7f0e9575f700 1 -- start start 2020-11-20T12:05:46.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.150+0000 7f0e9575f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0e900bfbe0 0x7f0e901480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.150+0000 7f0e9575f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e9009f560 0x7f0e901485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.150+0000 7f0e9575f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9014d110 con 0x7f0e900bf800 2020-11-20T12:05:46.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e9575f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9014c9d0 con 0x7f0e900bfbe0 2020-11-20T12:05:46.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e9575f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e90148aa0 con 0x7f0e9009f560 2020-11-20T12:05:46.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e8f7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0e900bfbe0 0x7f0e901480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e8f7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0e900bfbe0 0x7f0e901480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41418/0 (socket says 172.21.15.187:41418) 2020-11-20T12:05:46.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e8f7fe700 1 -- 172.21.15.187:0/3237037292 learned_addr learned my addr 172.21.15.187:0/3237037292 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:46.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e8f7fe700 1 -- 172.21.15.187:0/3237037292 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e9009f560 msgr2=0x7f0e901485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e8f7fe700 1 --2- 172.21.15.187:0/3237037292 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e9009f560 0x7f0e901485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e8f7fe700 1 -- 172.21.15.187:0/3237037292 >> v1:172.21.15.5:6789/0 conn(0x7f0e900bf800 legacy=0x7f0e9014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.151+0000 7f0e8f7fe700 1 -- 172.21.15.187:0/3237037292 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9014cbd0 con 0x7f0e900bfbe0 2020-11-20T12:05:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e8f7fe700 1 --2- 172.21.15.187:0/3237037292 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e900bfbe0 0x7f0e901480c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0e78006fd0 tx=0x7f0e78006eb0).ready entity=mon.1 client_cookie=10b2f5a1e76d399c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e8effd700 1 -- 172.21.15.187:0/3237037292 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e78009920 con 0x7f0e900bfbe0 2020-11-20T12:05:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e8effd700 1 -- 172.21.15.187:0/3237037292 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e78009a80 con 0x7f0e900bfbe0 2020-11-20T12:05:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e8effd700 1 -- 172.21.15.187:0/3237037292 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e78009e00 con 0x7f0e900bfbe0 2020-11-20T12:05:46.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e9575f700 1 -- 172.21.15.187:0/3237037292 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e900bfbe0 msgr2=0x7f0e901480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e9575f700 1 --2- 172.21.15.187:0/3237037292 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e900bfbe0 0x7f0e901480c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0e78006fd0 tx=0x7f0e78006eb0).stop 2020-11-20T12:05:46.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e9575f700 1 -- 172.21.15.187:0/3237037292 shutdown_connections 2020-11-20T12:05:46.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e9575f700 1 --2- 172.21.15.187:0/3237037292 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e9009f560 0x7f0e901485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:46.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.152+0000 7f0e9575f700 1 --2- 172.21.15.187:0/3237037292 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e900bfbe0 0x7f0e901480c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.153+0000 7f0e9575f700 1 -- 172.21.15.187:0/3237037292 >> 172.21.15.187:0/3237037292 conn(0x7f0e90068370 msgr2=0x7f0e900bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:46.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.153+0000 7f0e9575f700 1 -- 172.21.15.187:0/3237037292 shutdown_connections 2020-11-20T12:05:46.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.153+0000 7f0e9575f700 1 -- 172.21.15.187:0/3237037292 wait complete. 2020-11-20T12:05:46.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.154+0000 7f0e9575f700 1 Processor -- start 2020-11-20T12:05:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.154+0000 7f0e9575f700 1 -- start start 2020-11-20T12:05:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e9575f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e9009f560 0x7f0e90145810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e9575f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900bf800 0x7f0e90060e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e9575f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e9015c260 0x7f0e90141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e9575f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e90146bd0 con 0x7f0e9015c260 2020-11-20T12:05:46.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e9575f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9015bb20 con 0x7f0e9009f560 2020-11-20T12:05:46.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e9575f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9015bdc0 con 0x7f0e900bf800 2020-11-20T12:05:46.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e8ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e9009f560 0x7f0e90145810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e8ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e9009f560 0x7f0e90145810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41424/0 (socket says 172.21.15.187:41424) 2020-11-20T12:05:46.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.155+0000 7f0e8ffff700 1 -- 172.21.15.187:0/2288793427 learned_addr learned my addr 172.21.15.187:0/2288793427 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:46.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e8f7fe700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900bf800 0x7f0e90060e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e8ffff700 1 -- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900bf800 msgr2=0x7f0e90060e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e94f5e700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e9015c260 0x7f0e90141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e8ffff700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900bf800 0x7f0e90060e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e8ffff700 1 -- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e9015c260 msgr2=0x7f0e90141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e8ffff700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e9015c260 0x7f0e90141660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e8ffff700 1 -- 172.21.15.187:0/2288793427 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e90147260 con 0x7f0e9009f560 2020-11-20T12:05:46.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e94f5e700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e9015c260 0x7f0e90141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:46.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e8ffff700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e9009f560 0x7f0e90145810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0e8000c730 tx=0x7f0e80007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:46.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e87fff700 1 -- 172.21.15.187:0/2288793427 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8000e560 con 0x7f0e9009f560 2020-11-20T12:05:46.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e87fff700 1 -- 172.21.15.187:0/2288793427 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8000e6e0 con 0x7f0e9009f560 2020-11-20T12:05:46.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.156+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e90061340 con 0x7f0e9009f560 2020-11-20T12:05:46.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.157+0000 7f0e87fff700 1 -- 172.21.15.187:0/2288793427 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8000eaa0 con 0x7f0e9009f560 2020-11-20T12:05:46.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.157+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9015c6a0 con 0x7f0e9009f560 2020-11-20T12:05:46.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.158+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e90080050 con 0x7f0e9009f560 2020-11-20T12:05:46.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.158+0000 7f0e87fff700 1 -- 172.21.15.187:0/2288793427 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0e80019030 con 0x7f0e9009f560 2020-11-20T12:05:46.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.159+0000 7f0e87fff700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d50 0x7f0e6c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.159+0000 7f0e87fff700 1 -- 172.21.15.187:0/2288793427 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0e8006f440 con 0x7f0e9009f560 2020-11-20T12:05:46.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.159+0000 7f0e8f7fe700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d50 0x7f0e6c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.160+0000 7f0e8f7fe700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d50 0x7f0e6c059210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0e90061c90 tx=0x7f0e78015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:46.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.163+0000 7f0e87fff700 1 -- 172.21.15.187:0/2288793427 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0e80043fd0 con 0x7f0e9009f560 2020-11-20T12:05:46.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.348+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0e9015bf80 con 0x7f0e9009f560 2020-11-20T12:05:46.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.349+0000 7f0e87fff700 1 -- 172.21.15.187:0/2288793427 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0e80044940 con 0x7f0e9009f560 2020-11-20T12:05:46.353 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:46.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d50 msgr2=0x7f0e6c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d50 0x7f0e6c059210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0e90061c90 tx=0x7f0e78015000).stop 2020-11-20T12:05:46.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e9009f560 msgr2=0x7f0e90145810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e9009f560 0x7f0e90145810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0e8000c730 tx=0x7f0e80007c30).stop 2020-11-20T12:05:46.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 shutdown_connections 2020-11-20T12:05:46.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d50 0x7f0e6c059210 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e9015c260 0x7f0e90141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e9009f560 0x7f0e90145810 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 --2- 172.21.15.187:0/2288793427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900bf800 0x7f0e90060e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 >> 172.21.15.187:0/2288793427 conn(0x7f0e90068370 msgr2=0x7f0e900a0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:46.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 shutdown_connections 2020-11-20T12:05:46.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.352+0000 7f0e9575f700 1 -- 172.21.15.187:0/2288793427 wait complete. 2020-11-20T12:05:46.377 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:46.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.618+0000 7ff80ffff700 1 Processor -- start 2020-11-20T12:05:46.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.619+0000 7ff80ffff700 1 -- start start 2020-11-20T12:05:46.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.620+0000 7ff80ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff810145cc0 0x7ff810066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.620+0000 7ff80ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff8100683f0 0x7ff810065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.620+0000 7ff80ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810065680 0x7ff810061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.620+0000 7ff80ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff81014c170 con 0x7ff8100683f0 2020-11-20T12:05:46.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.620+0000 7ff80ffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff81014c410 con 0x7ff810145cc0 2020-11-20T12:05:46.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.620+0000 7ff80ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81014c6b0 con 0x7ff810065680 2020-11-20T12:05:46.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff810145cc0 0x7ff810066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff810145cc0 0x7ff810066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41430/0 (socket says 172.21.15.187:41430) 2020-11-20T12:05:46.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80effd700 1 -- 172.21.15.187:0/2287073181 learned_addr learned my addr 172.21.15.187:0/2287073181 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:46.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff8067fc700 1 --2- 172.21.15.187:0/2287073181 >> v2:172.21.15.5:3300/0 conn(0x7ff8100683f0 0x7ff810065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80f7fe700 1 --2- 172.21.15.187:0/2287073181 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810065680 0x7ff810061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80effd700 1 -- 172.21.15.187:0/2287073181 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810065680 msgr2=0x7ff810061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:46.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80effd700 1 --2- 172.21.15.187:0/2287073181 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810065680 0x7ff810061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80effd700 1 -- 172.21.15.187:0/2287073181 >> v2:172.21.15.5:3300/0 conn(0x7ff8100683f0 msgr2=0x7ff810065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:46.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80effd700 1 --2- 172.21.15.187:0/2287073181 >> v2:172.21.15.5:3300/0 conn(0x7ff8100683f0 0x7ff810065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff80effd700 1 -- 172.21.15.187:0/2287073181 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81014bdf0 con 0x7ff810145cc0 2020-11-20T12:05:46.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.621+0000 7ff8067fc700 1 --2- 172.21.15.187:0/2287073181 >> v2:172.21.15.5:3300/0 conn(0x7ff8100683f0 0x7ff810065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:46.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.622+0000 7ff80effd700 1 --2- 172.21.15.187:0/2287073181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff810145cc0 0x7ff810066040 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ff8100b85e0 tx=0x7ff800006eb0).ready entity=mon.1 client_cookie=c175173356bb24ab server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:46.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.622+0000 7ff80e7fc700 1 -- 172.21.15.187:0/2287073181 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8000098d0 con 0x7ff810145cc0 2020-11-20T12:05:46.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.622+0000 7ff80e7fc700 1 -- 172.21.15.187:0/2287073181 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff800009a30 con 0x7ff810145cc0 2020-11-20T12:05:46.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.622+0000 7ff80e7fc700 1 -- 172.21.15.187:0/2287073181 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800009db0 con 0x7ff810145cc0 2020-11-20T12:05:46.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.622+0000 7ff80ffff700 1 -- 172.21.15.187:0/2287073181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff810145cc0 msgr2=0x7ff810066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.622+0000 7ff80ffff700 1 --2- 172.21.15.187:0/2287073181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff810145cc0 0x7ff810066040 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ff8100b85e0 tx=0x7ff800006eb0).stop 2020-11-20T12:05:46.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.623+0000 7ff80ffff700 1 -- 172.21.15.187:0/2287073181 shutdown_connections 2020-11-20T12:05:46.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.623+0000 7ff80ffff700 1 --2- 172.21.15.187:0/2287073181 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810065680 0x7ff810061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.623+0000 7ff80ffff700 1 --2- 172.21.15.187:0/2287073181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff810145cc0 0x7ff810066040 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.623+0000 7ff80ffff700 1 --2- 172.21.15.187:0/2287073181 >> v2:172.21.15.5:3300/0 conn(0x7ff8100683f0 0x7ff810065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.623+0000 7ff80ffff700 1 -- 172.21.15.187:0/2287073181 >> 172.21.15.187:0/2287073181 conn(0x7ff8100bab30 msgr2=0x7ff8100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:46.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.623+0000 7ff80ffff700 1 -- 172.21.15.187:0/2287073181 shutdown_connections 2020-11-20T12:05:46.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.623+0000 7ff80ffff700 1 -- 172.21.15.187:0/2287073181 wait complete. 2020-11-20T12:05:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.624+0000 7ff80ffff700 1 Processor -- start 2020-11-20T12:05:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.624+0000 7ff80ffff700 1 -- start start 2020-11-20T12:05:46.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.625+0000 7ff80ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff810065680 0x7ff8100a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.625+0000 7ff80ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8100683f0 0x7ff810144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.625+0000 7ff80ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810145cc0 0x7ff81009ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.625+0000 7ff80ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81014a450 con 0x7ff810065680 2020-11-20T12:05:46.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.625+0000 7ff80ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81009e6d0 con 0x7ff8100683f0 2020-11-20T12:05:46.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.625+0000 7ff80ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81009e970 con 0x7ff810145cc0 2020-11-20T12:05:46.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.625+0000 7ff8067fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8100683f0 0x7ff810144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff8067fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8100683f0 0x7ff810144f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41438/0 (socket says 172.21.15.187:41438) 2020-11-20T12:05:46.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff8067fc700 1 -- 172.21.15.187:0/720046104 learned_addr learned my addr 172.21.15.187:0/720046104 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:46.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff80f7fe700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810145cc0 0x7ff81009ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff80effd700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff810065680 0x7ff8100a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff8067fc700 1 -- 172.21.15.187:0/720046104 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810145cc0 msgr2=0x7ff81009ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff8067fc700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810145cc0 0x7ff81009ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff8067fc700 1 -- 172.21.15.187:0/720046104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff810065680 msgr2=0x7ff8100a0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff8067fc700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff810065680 0x7ff8100a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff8067fc700 1 -- 172.21.15.187:0/720046104 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8100a0510 con 0x7ff8100683f0 2020-11-20T12:05:46.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.626+0000 7ff8067fc700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8100683f0 0x7ff810144f10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff7f800db60 tx=0x7ff7f8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:46.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.627+0000 7ff8077fe700 1 -- 172.21.15.187:0/720046104 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80098e0 con 0x7ff8100683f0 2020-11-20T12:05:46.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.627+0000 7ff8077fe700 1 -- 172.21.15.187:0/720046104 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f8009a40 con 0x7ff8100683f0 2020-11-20T12:05:46.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.627+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81009d2b0 con 0x7ff8100683f0 2020-11-20T12:05:46.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.627+0000 7ff8077fe700 1 -- 172.21.15.187:0/720046104 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8009de0 con 0x7ff8100683f0 2020-11-20T12:05:46.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.627+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81014a880 con 0x7ff8100683f0 2020-11-20T12:05:46.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.628+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff810061a20 con 0x7ff8100683f0 2020-11-20T12:05:46.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.629+0000 7ff8077fe700 1 -- 172.21.15.187:0/720046104 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff7f8013690 con 0x7ff8100683f0 2020-11-20T12:05:46.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.629+0000 7ff8077fe700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7ec056d20 0x7ff7ec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:46.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.629+0000 7ff8077fe700 1 -- 172.21.15.187:0/720046104 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff7f8070080 con 0x7ff8100683f0 2020-11-20T12:05:46.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.630+0000 7ff80effd700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7ec056d20 0x7ff7ec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:46.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.630+0000 7ff80effd700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7ec056d20 0x7ff7ec0591e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff80000b1b0 tx=0x7ff800015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:46.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.633+0000 7ff8077fe700 1 -- 172.21.15.187:0/720046104 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff7f8045470 con 0x7ff8100683f0 2020-11-20T12:05:46.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.823+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff81014cb50 con 0x7ff8100683f0 2020-11-20T12:05:46.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.824+0000 7ff8077fe700 1 -- 172.21.15.187:0/720046104 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff7f8048a90 con 0x7ff8100683f0 2020-11-20T12:05:46.829 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:46.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7ec056d20 msgr2=0x7ff7ec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7ec056d20 0x7ff7ec0591e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff80000b1b0 tx=0x7ff800015000).stop 2020-11-20T12:05:46.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8100683f0 msgr2=0x7ff810144f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:46.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8100683f0 0x7ff810144f10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff7f800db60 tx=0x7ff7f8008040).stop 2020-11-20T12:05:46.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 shutdown_connections 2020-11-20T12:05:46.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7ec056d20 0x7ff7ec0591e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff810065680 0x7ff8100a0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff810145cc0 0x7ff81009ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 --2- 172.21.15.187:0/720046104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8100683f0 0x7ff810144f10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:46.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 >> 172.21.15.187:0/720046104 conn(0x7ff8100bab30 msgr2=0x7ff8100b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:46.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.827+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 shutdown_connections 2020-11-20T12:05:46.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:46.828+0000 7ff80ffff700 1 -- 172.21.15.187:0/720046104 wait complete. 2020-11-20T12:05:46.855 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.111+0000 7f30bd0a2700 1 Processor -- start 2020-11-20T12:05:47.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.111+0000 7f30bd0a2700 1 -- start start 2020-11-20T12:05:47.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.112+0000 7f30bd0a2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f30b8065190 0x7f30b8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.112+0000 7f30bd0a2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b8065f90 0x7f30b8143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.112+0000 7f30bd0a2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b80b13e0 con 0x7f30b8143c70 2020-11-20T12:05:47.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.112+0000 7f30bd0a2700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30b814ab30 con 0x7f30b8065190 2020-11-20T12:05:47.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.112+0000 7f30bd0a2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b8062c10 con 0x7f30b8065f90 2020-11-20T12:05:47.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.113+0000 7f30bc8a1700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f30b8143c70 0x7f30b809ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38052/0 (socket says 172.21.15.187:38052) 2020-11-20T12:05:47.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.113+0000 7f30bc8a1700 1 -- 172.21.15.187:0/2720914078 learned_addr learned my addr 172.21.15.187:0/2720914078 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.113+0000 7f30b7fff700 1 --2- 172.21.15.187:0/2720914078 >> v2:172.21.15.5:3300/0 conn(0x7f30b8065190 0x7f30b8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.113+0000 7f30af7fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b8065f90 0x7f30b8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.113+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174044113 0 0) 0x7f30b80b13e0 con 0x7f30b8143c70 2020-11-20T12:05:47.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.113+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3098003240 con 0x7f30b8143c70 2020-11-20T12:05:47.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1119458716 0 0) 0x7f3098003240 con 0x7f30b8143c70 2020-11-20T12:05:47.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b8065f90 msgr2=0x7f30b8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:47.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b8065f90 0x7f30b8143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 >> v2:172.21.15.5:3300/0 conn(0x7f30b8065190 msgr2=0x7f30b8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:47.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 --2- 172.21.15.187:0/2720914078 >> v2:172.21.15.5:3300/0 conn(0x7f30b8065190 0x7f30b8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b80b1640 con 0x7f30b8143c70 2020-11-20T12:05:47.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f30b0001860 con 0x7f30b8143c70 2020-11-20T12:05:47.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 >> v1:172.21.15.187:6789/0 conn(0x7f30b8143c70 legacy=0x7f30b809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b8065f90 0x7f30980037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3098003d30 0x7f30980044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30980049d0 0x7f3098004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b814ab30 con 0x7f30980049d0 2020-11-20T12:05:47.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b8062c10 con 0x7f3098003d30 2020-11-20T12:05:47.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30980042f0 con 0x7f30b8065f90 2020-11-20T12:05:47.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.114+0000 7f30af7fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3098003d30 0x7f30980044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.115+0000 7f30b7fff700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b8065f90 0x7f30980037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.115+0000 7f30af7fe700 1 -- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b8065f90 msgr2=0x7f30980037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:47.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.115+0000 7f30bc8a1700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30980049d0 0x7f3098004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.115+0000 7f30af7fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b8065f90 0x7f30980037d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.115+0000 7f30af7fe700 1 -- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30980049d0 msgr2=0x7f3098004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:47.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.115+0000 7f30af7fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30980049d0 0x7f3098004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.115+0000 7f30af7fe700 1 -- 172.21.15.187:0/2720914078 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098003240 con 0x7f3098003d30 2020-11-20T12:05:47.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.115+0000 7f30af7fe700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3098003d30 0x7f30980044e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f30a000bc00 tx=0x7f30a0006df0).ready entity=mon.1 client_cookie=3c6b309f50de6de1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:47.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a000b310 con 0x7f3098003d30 2020-11-20T12:05:47.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a000eae0 con 0x7f3098003d30 2020-11-20T12:05:47.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30b77fe700 1 -- 172.21.15.187:0/2720914078 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a000ee60 con 0x7f3098003d30 2020-11-20T12:05:47.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2720914078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3098003d30 msgr2=0x7f30980044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30bd0a2700 1 --2- 172.21.15.187:0/2720914078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3098003d30 0x7f30980044e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f30a000bc00 tx=0x7f30a0006df0).stop 2020-11-20T12:05:47.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30bc8a1700 1 -- 172.21.15.187:0/2720914078 reap_dead start 2020-11-20T12:05:47.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2720914078 shutdown_connections 2020-11-20T12:05:47.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2720914078 >> 172.21.15.187:0/2720914078 conn(0x7f30b80bab30 msgr2=0x7f30b80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:47.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2720914078 shutdown_connections 2020-11-20T12:05:47.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.116+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2720914078 wait complete. 2020-11-20T12:05:47.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.118+0000 7f30bd0a2700 1 Processor -- start 2020-11-20T12:05:47.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.118+0000 7f30bd0a2700 1 -- start start 2020-11-20T12:05:47.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.119+0000 7f30bd0a2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30b8149a10 0x7f30b80b9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.119+0000 7f30bd0a2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30b80b2640 0x7f30b8146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30bd0a2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b80b9e00 0x7f30b8146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30bd0a2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b809f690 con 0x7f30b8149a10 2020-11-20T12:05:47.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30bd0a2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b8138900 con 0x7f30b80b2640 2020-11-20T12:05:47.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30bd0a2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b80b2ae0 con 0x7f30b80b9e00 2020-11-20T12:05:47.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30af7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30b80b2640 0x7f30b8146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30af7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30b80b2640 0x7f30b8146670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41458/0 (socket says 172.21.15.187:41458) 2020-11-20T12:05:47.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30af7fe700 1 -- 172.21.15.187:0/2531309140 learned_addr learned my addr 172.21.15.187:0/2531309140 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:47.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30b7fff700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30b8149a10 0x7f30b80b9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30bc8a1700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b80b9e00 0x7f30b8146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30af7fe700 1 -- 172.21.15.187:0/2531309140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b80b9e00 msgr2=0x7f30b8146b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30af7fe700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b80b9e00 0x7f30b8146b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30af7fe700 1 -- 172.21.15.187:0/2531309140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30b8149a10 msgr2=0x7f30b80b9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30af7fe700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30b8149a10 0x7f30b80b9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.120+0000 7f30af7fe700 1 -- 172.21.15.187:0/2531309140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b809f8f0 con 0x7f30b80b2640 2020-11-20T12:05:47.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.121+0000 7f30af7fe700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30b80b2640 0x7f30b8146670 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f30a0013000 tx=0x7f30a0019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:47.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.121+0000 7f30b4ff9700 1 -- 172.21.15.187:0/2531309140 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a001d620 con 0x7f30b80b2640 2020-11-20T12:05:47.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.121+0000 7f30b4ff9700 1 -- 172.21.15.187:0/2531309140 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a0021070 con 0x7f30b80b2640 2020-11-20T12:05:47.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.121+0000 7f30b4ff9700 1 -- 172.21.15.187:0/2531309140 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a001d780 con 0x7f30b80b2640 2020-11-20T12:05:47.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.121+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b8147050 con 0x7f30b80b2640 2020-11-20T12:05:47.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.121+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b80b4990 con 0x7f30b80b2640 2020-11-20T12:05:47.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.122+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b8065700 con 0x7f30b80b2640 2020-11-20T12:05:47.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.123+0000 7f30b4ff9700 1 -- 172.21.15.187:0/2531309140 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f30a001db70 con 0x7f30b80b2640 2020-11-20T12:05:47.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.124+0000 7f30b4ff9700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3094056d20 0x7f30940591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.124+0000 7f30b4ff9700 1 -- 172.21.15.187:0/2531309140 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f30a0071da0 con 0x7f30b80b2640 2020-11-20T12:05:47.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.124+0000 7f30b7fff700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3094056d20 0x7f30940591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.125+0000 7f30b7fff700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3094056d20 0x7f30940591e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f30a8006bd0 tx=0x7f30a8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:47.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.128+0000 7f30b4ff9700 1 -- 172.21.15.187:0/2531309140 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f30a0047120 con 0x7f30b80b2640 2020-11-20T12:05:47.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.314+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30b809fa80 con 0x7f30b80b2640 2020-11-20T12:05:47.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.317+0000 7f30b4ff9700 1 -- 172.21.15.187:0/2531309140 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f30a004a740 con 0x7f30b80b2640 2020-11-20T12:05:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:47.325 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:47.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3094056d20 msgr2=0x7f30940591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3094056d20 0x7f30940591e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f30a8006bd0 tx=0x7f30a8008040).stop 2020-11-20T12:05:47.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30b80b2640 msgr2=0x7f30b8146670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30b80b2640 0x7f30b8146670 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f30a0013000 tx=0x7f30a0019000).stop 2020-11-20T12:05:47.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 shutdown_connections 2020-11-20T12:05:47.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3094056d20 0x7f30940591e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30b8149a10 0x7f30b80b9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f30b80b9e00 0x7f30b8146b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 --2- 172.21.15.187:0/2531309140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30b80b2640 0x7f30b8146670 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.319+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 >> 172.21.15.187:0/2531309140 conn(0x7f30b80bab30 msgr2=0x7f30b80b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:47.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.320+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 shutdown_connections 2020-11-20T12:05:47.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.320+0000 7f30bd0a2700 1 -- 172.21.15.187:0/2531309140 wait complete. 2020-11-20T12:05:47.346 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:47.347 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:47.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.588+0000 7f1b3ca6a700 1 Processor -- start 2020-11-20T12:05:47.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.588+0000 7f1b3ca6a700 1 -- start start 2020-11-20T12:05:47.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.590+0000 7f1b3ca6a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b3809f560 0x7f1b38153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.590+0000 7f1b3ca6a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3814d110 con 0x7f1b380bfbe0 2020-11-20T12:05:47.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.590+0000 7f1b3ca6a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3814cbd0 con 0x7f1b380bf800 2020-11-20T12:05:47.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.590+0000 7f1b3ca6a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3814c9d0 con 0x7f1b3809f560 2020-11-20T12:05:47.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.590+0000 7f1b377fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1b380bf800 0x7f1b3814c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38070/0 (socket says 172.21.15.187:38070) 2020-11-20T12:05:47.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.590+0000 7f1b377fe700 1 -- 172.21.15.187:0/5782176 learned_addr learned my addr 172.21.15.187:0/5782176 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:47.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b37fff700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b3809f560 0x7f1b38153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017094550 0 0) 0x7f1b3814cbd0 con 0x7f1b380bf800 2020-11-20T12:05:47.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b18003240 con 0x7f1b380bf800 2020-11-20T12:05:47.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 774318643 0 0) 0x7f1b18003240 con 0x7f1b380bf800 2020-11-20T12:05:47.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b3809f560 msgr2=0x7f1b38153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:47.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b3809f560 0x7f1b38153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 >> v1:172.21.15.5:6789/0 conn(0x7f1b380bfbe0 legacy=0x7f1b381480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3814ce70 con 0x7f1b380bf800 2020-11-20T12:05:47.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1b28001860 con 0x7f1b380bf800 2020-11-20T12:05:47.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b37fff700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b3809f560 0x7f1b38153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:47.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 >> v1:172.21.15.187:6789/0 conn(0x7f1b380bf800 legacy=0x7f1b3814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b18003730 0x7f1b18003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b180044e0 0x7f1b18004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b3809f560 0x7f1b18004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3814c9d0 con 0x7f1b180044e0 2020-11-20T12:05:47.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18004230 con 0x7f1b18003730 2020-11-20T12:05:47.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.591+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1800b730 con 0x7f1b3809f560 2020-11-20T12:05:47.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b377fe700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b18003730 0x7f1b18003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b2effd700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b180044e0 0x7f1b18004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b377fe700 1 -- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b3809f560 msgr2=0x7f1b18004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:47.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b377fe700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b3809f560 0x7f1b18004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:47.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b377fe700 1 -- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b180044e0 msgr2=0x7f1b18004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:47.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b377fe700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b180044e0 0x7f1b18004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b377fe700 1 -- 172.21.15.187:0/5782176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18003240 con 0x7f1b18003730 2020-11-20T12:05:47.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b2effd700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b180044e0 0x7f1b18004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:47.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.592+0000 7f1b377fe700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b18003730 0x7f1b18003b50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f1b2800c840 tx=0x7f1b3814cd60).ready entity=mon.1 client_cookie=79366512df73b512 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:47.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28007b50 con 0x7f1b18003730 2020-11-20T12:05:47.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b28005b80 con 0x7f1b18003730 2020-11-20T12:05:47.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b36ffd700 1 -- 172.21.15.187:0/5782176 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28007110 con 0x7f1b18003730 2020-11-20T12:05:47.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/5782176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b18003730 msgr2=0x7f1b18003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b3ca6a700 1 --2- 172.21.15.187:0/5782176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b18003730 0x7f1b18003b50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f1b2800c840 tx=0x7f1b3814cd60).stop 2020-11-20T12:05:47.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b37fff700 1 -- 172.21.15.187:0/5782176 reap_dead start 2020-11-20T12:05:47.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/5782176 shutdown_connections 2020-11-20T12:05:47.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/5782176 >> 172.21.15.187:0/5782176 conn(0x7f1b38068370 msgr2=0x7f1b380bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:47.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.593+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/5782176 shutdown_connections 2020-11-20T12:05:47.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.594+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/5782176 wait complete. 2020-11-20T12:05:47.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.595+0000 7f1b3ca6a700 1 Processor -- start 2020-11-20T12:05:47.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.595+0000 7f1b3ca6a700 1 -- start start 2020-11-20T12:05:47.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b3ca6a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b3815c210 0x7f1b3815c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b3ca6a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b3813c280 0x7f1b3815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b3ca6a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b38141020 0x7f1b381414a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b3ca6a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38147b30 con 0x7f1b3813c280 2020-11-20T12:05:47.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b3ca6a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b381473f0 con 0x7f1b3815c210 2020-11-20T12:05:47.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b3ca6a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38147690 con 0x7f1b38141020 2020-11-20T12:05:47.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b377fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b3815c210 0x7f1b3815c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b377fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b3815c210 0x7f1b3815c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41476/0 (socket says 172.21.15.187:41476) 2020-11-20T12:05:47.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.596+0000 7f1b377fe700 1 -- 172.21.15.187:0/3013811123 learned_addr learned my addr 172.21.15.187:0/3013811123 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:47.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b37fff700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b38141020 0x7f1b381414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b2effd700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b3813c280 0x7f1b3815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b377fe700 1 -- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b38141020 msgr2=0x7f1b381414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b377fe700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b38141020 0x7f1b381414a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b377fe700 1 -- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b3813c280 msgr2=0x7f1b3815cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b377fe700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b3813c280 0x7f1b3815cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b377fe700 1 -- 172.21.15.187:0/3013811123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38147110 con 0x7f1b3815c210 2020-11-20T12:05:47.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b2effd700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b3813c280 0x7f1b3815cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:47.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b377fe700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b3815c210 0x7f1b3815c5f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f1b2800d010 tx=0x7f1b28014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:47.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.597+0000 7f1b2ffff700 1 -- 172.21.15.187:0/3013811123 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28013620 con 0x7f1b3815c210 2020-11-20T12:05:47.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.598+0000 7f1b2ffff700 1 -- 172.21.15.187:0/3013811123 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b28013780 con 0x7f1b3815c210 2020-11-20T12:05:47.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.598+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b38141990 con 0x7f1b3815c210 2020-11-20T12:05:47.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.598+0000 7f1b2ffff700 1 -- 172.21.15.187:0/3013811123 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28013b00 con 0x7f1b3815c210 2020-11-20T12:05:47.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.598+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3813eb70 con 0x7f1b3815c210 2020-11-20T12:05:47.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.599+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38080050 con 0x7f1b3815c210 2020-11-20T12:05:47.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.600+0000 7f1b2ffff700 1 -- 172.21.15.187:0/3013811123 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1b2801b030 con 0x7f1b3815c210 2020-11-20T12:05:47.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.600+0000 7f1b2ffff700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b14056d50 0x7f1b14059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:47.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.600+0000 7f1b2ffff700 1 -- 172.21.15.187:0/3013811123 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1b28072260 con 0x7f1b3815c210 2020-11-20T12:05:47.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.601+0000 7f1b2effd700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b14056d50 0x7f1b14059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:47.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.601+0000 7f1b2effd700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b14056d50 0x7f1b14059210 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1b20000f20 tx=0x7f1b20009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:47.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.604+0000 7f1b2ffff700 1 -- 172.21.15.187:0/3013811123 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1b280476b0 con 0x7f1b3815c210 2020-11-20T12:05:47.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.788+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f1b38080050 con 0x7f1b3815c210 2020-11-20T12:05:47.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.790+0000 7f1b2ffff700 1 -- 172.21.15.187:0/3013811123 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f1b2804acd0 con 0x7f1b3815c210 2020-11-20T12:05:47.794 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:47.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b14056d50 msgr2=0x7f1b14059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b14056d50 0x7f1b14059210 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1b20000f20 tx=0x7f1b20009040).stop 2020-11-20T12:05:47.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b3815c210 msgr2=0x7f1b3815c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:47.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b3815c210 0x7f1b3815c5f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f1b2800d010 tx=0x7f1b28014000).stop 2020-11-20T12:05:47.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 shutdown_connections 2020-11-20T12:05:47.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b14056d50 0x7f1b14059210 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b38141020 0x7f1b381414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b3815c210 0x7f1b3815c5f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 --2- 172.21.15.187:0/3013811123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b3813c280 0x7f1b3815cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:47.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 >> 172.21.15.187:0/3013811123 conn(0x7f1b38068370 msgr2=0x7f1b380bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:47.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 shutdown_connections 2020-11-20T12:05:47.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:47.792+0000 7f1b3ca6a700 1 -- 172.21.15.187:0/3013811123 wait complete. 2020-11-20T12:05:47.820 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:48.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.078+0000 7fc69ace8700 1 Processor -- start 2020-11-20T12:05:48.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.078+0000 7fc69ace8700 1 -- start start 2020-11-20T12:05:48.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.079+0000 7fc69ace8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6941484d0 0x7fc694061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.079+0000 7fc69ace8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6940622c0 0x7fc6940b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.079+0000 7fc69ace8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc694066f70 0x7fc6940b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.079+0000 7fc69ace8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc694067730 con 0x7fc694066f70 2020-11-20T12:05:48.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.079+0000 7fc69ace8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6940b1300 con 0x7fc6941484d0 2020-11-20T12:05:48.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.079+0000 7fc69ace8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6940b15a0 con 0x7fc6940622c0 2020-11-20T12:05:48.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.080+0000 7fc699ce6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6941484d0 0x7fc694061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.080+0000 7fc699ce6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6941484d0 0x7fc694061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41484/0 (socket says 172.21.15.187:41484) 2020-11-20T12:05:48.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.080+0000 7fc699ce6700 1 -- 172.21.15.187:0/1841639645 learned_addr learned my addr 172.21.15.187:0/1841639645 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:48.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.080+0000 7fc69a4e7700 1 --2- 172.21.15.187:0/1841639645 >> v2:172.21.15.5:3300/0 conn(0x7fc694066f70 0x7fc6940b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.080+0000 7fc6994e5700 1 --2- 172.21.15.187:0/1841639645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6940622c0 0x7fc6940b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.080+0000 7fc699ce6700 1 -- 172.21.15.187:0/1841639645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6940622c0 msgr2=0x7fc6940b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:48.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.080+0000 7fc699ce6700 1 --2- 172.21.15.187:0/1841639645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6940622c0 0x7fc6940b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.081+0000 7fc699ce6700 1 -- 172.21.15.187:0/1841639645 >> v2:172.21.15.5:3300/0 conn(0x7fc694066f70 msgr2=0x7fc6940b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:48.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.081+0000 7fc699ce6700 1 --2- 172.21.15.187:0/1841639645 >> v2:172.21.15.5:3300/0 conn(0x7fc694066f70 0x7fc6940b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.081+0000 7fc699ce6700 1 -- 172.21.15.187:0/1841639645 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6940b1840 con 0x7fc6941484d0 2020-11-20T12:05:48.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.081+0000 7fc699ce6700 1 --2- 172.21.15.187:0/1841639645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6941484d0 0x7fc694061dd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fc6940b85e0 tx=0x7fc690006eb0).ready entity=mon.1 client_cookie=7e7be0e373e5363f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:48.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.081+0000 7fc698ce4700 1 -- 172.21.15.187:0/1841639645 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6900098d0 con 0x7fc6941484d0 2020-11-20T12:05:48.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.081+0000 7fc698ce4700 1 -- 172.21.15.187:0/1841639645 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc690009a30 con 0x7fc6941484d0 2020-11-20T12:05:48.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.081+0000 7fc698ce4700 1 -- 172.21.15.187:0/1841639645 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690009db0 con 0x7fc6941484d0 2020-11-20T12:05:48.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 -- 172.21.15.187:0/1841639645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6941484d0 msgr2=0x7fc694061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1841639645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6941484d0 0x7fc694061dd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fc6940b85e0 tx=0x7fc690006eb0).stop 2020-11-20T12:05:48.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 -- 172.21.15.187:0/1841639645 shutdown_connections 2020-11-20T12:05:48.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1841639645 >> v2:172.21.15.5:3300/0 conn(0x7fc694066f70 0x7fc6940b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1841639645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6941484d0 0x7fc694061dd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1841639645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6940622c0 0x7fc6940b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 -- 172.21.15.187:0/1841639645 >> 172.21.15.187:0/1841639645 conn(0x7fc6940bab30 msgr2=0x7fc6940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:48.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 -- 172.21.15.187:0/1841639645 shutdown_connections 2020-11-20T12:05:48.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.082+0000 7fc69ace8700 1 -- 172.21.15.187:0/1841639645 wait complete. 2020-11-20T12:05:48.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.084+0000 7fc69ace8700 1 Processor -- start 2020-11-20T12:05:48.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.084+0000 7fc69ace8700 1 -- start start 2020-11-20T12:05:48.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc69ace8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6940622c0 0x7fc69415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc69ace8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc694066f70 0x7fc6941436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc69ace8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6941484d0 0x7fc6941550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc69ace8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6940b1a40 con 0x7fc694066f70 2020-11-20T12:05:48.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc69ace8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69409f380 con 0x7fc6940622c0 2020-11-20T12:05:48.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc69ace8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6940b4530 con 0x7fc6941484d0 2020-11-20T12:05:48.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc699ce6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6940622c0 0x7fc69415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc699ce6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6940622c0 0x7fc69415bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41490/0 (socket says 172.21.15.187:41490) 2020-11-20T12:05:48.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc699ce6700 1 -- 172.21.15.187:0/1752692336 learned_addr learned my addr 172.21.15.187:0/1752692336 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:48.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc699ce6700 1 -- 172.21.15.187:0/1752692336 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6941484d0 msgr2=0x7fc6941550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc699ce6700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6941484d0 0x7fc6941550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:48.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc699ce6700 1 -- 172.21.15.187:0/1752692336 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc694066f70 msgr2=0x7fc6941436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc699ce6700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc694066f70 0x7fc6941436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:48.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.085+0000 7fc699ce6700 1 -- 172.21.15.187:0/1752692336 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6940b49d0 con 0x7fc6940622c0 2020-11-20T12:05:48.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.086+0000 7fc699ce6700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6940622c0 0x7fc69415bfe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fc690006ef0 tx=0x7fc690015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:48.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.086+0000 7fc681ffb700 1 -- 172.21.15.187:0/1752692336 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690011e50 con 0x7fc6940622c0 2020-11-20T12:05:48.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.086+0000 7fc681ffb700 1 -- 172.21.15.187:0/1752692336 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69001e460 con 0x7fc6940622c0 2020-11-20T12:05:48.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.086+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69409f620 con 0x7fc6940622c0 2020-11-20T12:05:48.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.086+0000 7fc681ffb700 1 -- 172.21.15.187:0/1752692336 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69001e7e0 con 0x7fc6940622c0 2020-11-20T12:05:48.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.086+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69409f000 con 0x7fc6940622c0 2020-11-20T12:05:48.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.087+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc694061a20 con 0x7fc6940622c0 2020-11-20T12:05:48.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.088+0000 7fc681ffb700 1 -- 172.21.15.187:0/1752692336 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc690025030 con 0x7fc6940622c0 2020-11-20T12:05:48.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.088+0000 7fc681ffb700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc66c056d20 0x7fc66c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.088+0000 7fc681ffb700 1 -- 172.21.15.187:0/1752692336 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc690071790 con 0x7fc6940622c0 2020-11-20T12:05:48.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.089+0000 7fc6994e5700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc66c056d20 0x7fc66c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.089+0000 7fc6994e5700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc66c056d20 0x7fc66c0591e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc694099e20 tx=0x7fc684008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:48.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.092+0000 7fc681ffb700 1 -- 172.21.15.187:0/1752692336 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc690046c90 con 0x7fc6940622c0 2020-11-20T12:05:48.185 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:05:48.196 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:05:48.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.283+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc6941488b0 con 0x7fc6940622c0 2020-11-20T12:05:48.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.283+0000 7fc681ffb700 1 -- 172.21.15.187:0/1752692336 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc690023020 con 0x7fc6940622c0 2020-11-20T12:05:48.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc66c056d20 msgr2=0x7fc66c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc66c056d20 0x7fc66c0591e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc694099e20 tx=0x7fc684008040).stop 2020-11-20T12:05:48.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6940622c0 msgr2=0x7fc69415bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6940622c0 0x7fc69415bfe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fc690006ef0 tx=0x7fc690015000).stop 2020-11-20T12:05:48.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 shutdown_connections 2020-11-20T12:05:48.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc66c056d20 0x7fc66c0591e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6941484d0 0x7fc6941550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:48.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6940622c0 0x7fc69415bfe0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 --2- 172.21.15.187:0/1752692336 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc694066f70 0x7fc6941436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:48.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 >> 172.21.15.187:0/1752692336 conn(0x7fc6940bab30 msgr2=0x7fc69414d850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:48.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 shutdown_connections 2020-11-20T12:05:48.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.286+0000 7fc69ace8700 1 -- 172.21.15.187:0/1752692336 wait complete. 2020-11-20T12:05:48.294 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:48.313 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:48.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.560+0000 7f6202108700 1 Processor -- start 2020-11-20T12:05:48.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.560+0000 7f6202108700 1 -- start start 2020-11-20T12:05:48.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.562+0000 7f6202108700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61fc0b1840 0x7f61fc0b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.562+0000 7f6202108700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc149790 con 0x7f61fc0b1c20 2020-11-20T12:05:48.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.562+0000 7f6202108700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc065ba0 con 0x7f61fc0b11d0 2020-11-20T12:05:48.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.562+0000 7f6202108700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61fc0b8ca0 con 0x7f61fc0b1840 2020-11-20T12:05:48.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.562+0000 7f6201907700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f61fc0b11d0 0x7f61fc066040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38108/0 (socket says 172.21.15.187:38108) 2020-11-20T12:05:48.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.562+0000 7f6201907700 1 -- 172.21.15.187:0/1536779223 learned_addr learned my addr 172.21.15.187:0/1536779223 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:48.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.562+0000 7f6201106700 1 --2- 172.21.15.187:0/1536779223 >> v2:172.21.15.5:3300/0 conn(0x7f61fc0b1840 0x7f61fc0b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859123430 0 0) 0x7f61fc065ba0 con 0x7f61fc0b11d0 2020-11-20T12:05:48.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e4003240 con 0x7f61fc0b11d0 2020-11-20T12:05:48.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3624665631 0 0) 0x7f61e4003240 con 0x7f61fc0b11d0 2020-11-20T12:05:48.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 >> v2:172.21.15.5:3300/0 conn(0x7f61fc0b1840 msgr2=0x7f61fc0b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:48.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 --2- 172.21.15.187:0/1536779223 >> v2:172.21.15.5:3300/0 conn(0x7f61fc0b1840 0x7f61fc0b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 >> v1:172.21.15.5:6789/0 conn(0x7f61fc0b1c20 legacy=0x7f61fc0b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc149450 con 0x7f61fc0b11d0 2020-11-20T12:05:48.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f61f4001860 con 0x7f61fc0b11d0 2020-11-20T12:05:48.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 >> v1:172.21.15.187:6789/0 conn(0x7f61fc0b11d0 legacy=0x7f61fc066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4003780 0x7f61e4003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.563+0000 7f61ebfff700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61e40044e0 0x7f61e4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.564+0000 7f61ebfff700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61e4004e50 0x7f61e400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.564+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc0b8ca0 con 0x7f61e4003780 2020-11-20T12:05:48.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.564+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4005300 con 0x7f61e4004e50 2020-11-20T12:05:48.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.564+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e400bc20 con 0x7f61e40044e0 2020-11-20T12:05:48.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.564+0000 7f6201907700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61e4004e50 0x7f61e400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.564+0000 7f6201106700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4003780 0x7f61e4003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.564+0000 7f6201907700 1 -- 172.21.15.187:0/1536779223 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61e40044e0 msgr2=0x7f61e4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:48.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f6201907700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61e40044e0 0x7f61e4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:48.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f6201907700 1 -- 172.21.15.187:0/1536779223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4003780 msgr2=0x7f61e4003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:48.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f6201907700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4003780 0x7f61e4003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f6201907700 1 -- 172.21.15.187:0/1536779223 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4003240 con 0x7f61e4004e50 2020-11-20T12:05:48.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f6201106700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4003780 0x7f61e4003ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:48.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f6201907700 1 -- 172.21.15.187:0/1536779223 reap_dead start 2020-11-20T12:05:48.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f6201907700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61e4004e50 0x7f61e400b730 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f61f4007fd0 tx=0x7f61f4007a50).ready entity=mon.1 client_cookie=7b94cf6edf2b8a18 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:48.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4005b80 con 0x7f61e4004e50 2020-11-20T12:05:48.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4005ce0 con 0x7f61e4004e50 2020-11-20T12:05:48.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.565+0000 7f61ebfff700 1 -- 172.21.15.187:0/1536779223 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4007320 con 0x7f61e4004e50 2020-11-20T12:05:48.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.566+0000 7f6202108700 1 -- 172.21.15.187:0/1536779223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61e4004e50 msgr2=0x7f61e400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.566+0000 7f6202108700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61e4004e50 0x7f61e400b730 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f61f4007fd0 tx=0x7f61f4007a50).stop 2020-11-20T12:05:48.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.566+0000 7f6202108700 1 -- 172.21.15.187:0/1536779223 shutdown_connections 2020-11-20T12:05:48.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.566+0000 7f6202108700 1 --2- 172.21.15.187:0/1536779223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61e4004e50 0x7f61e400b730 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.566+0000 7f6202108700 1 -- 172.21.15.187:0/1536779223 >> 172.21.15.187:0/1536779223 conn(0x7f61fc0bbeb0 msgr2=0x7f61fc0b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:48.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.566+0000 7f6202108700 1 -- 172.21.15.187:0/1536779223 shutdown_connections 2020-11-20T12:05:48.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.566+0000 7f6202108700 1 -- 172.21.15.187:0/1536779223 wait complete. 2020-11-20T12:05:48.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.568+0000 7f6202108700 1 Processor -- start 2020-11-20T12:05:48.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.568+0000 7f6202108700 1 -- start start 2020-11-20T12:05:48.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6202108700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4004e50 0x7f61fc09e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6202108700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61fc148210 0x7f61fc1439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6202108700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc1575a0 0x7f61fc1579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6202108700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc0b4b00 con 0x7f61e4004e50 2020-11-20T12:05:48.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6202108700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc157ed0 con 0x7f61fc1575a0 2020-11-20T12:05:48.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6202108700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc0b4cf0 con 0x7f61fc148210 2020-11-20T12:05:48.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6201907700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc1575a0 0x7f61fc1579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6201907700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc1575a0 0x7f61fc1579e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41512/0 (socket says 172.21.15.187:41512) 2020-11-20T12:05:48.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.569+0000 7f6201907700 1 -- 172.21.15.187:0/2918587708 learned_addr learned my addr 172.21.15.187:0/2918587708 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:48.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f6201907700 1 -- 172.21.15.187:0/2918587708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61fc148210 msgr2=0x7f61fc1439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f6200905700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61fc148210 0x7f61fc1439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f6201907700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61fc148210 0x7f61fc1439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f6201907700 1 -- 172.21.15.187:0/2918587708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4004e50 msgr2=0x7f61fc09e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f6201907700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4004e50 0x7f61fc09e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:48.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f6201907700 1 -- 172.21.15.187:0/2918587708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc0b4f90 con 0x7f61fc1575a0 2020-11-20T12:05:48.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f6201907700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc1575a0 0x7f61fc1579e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f61f4000f40 tx=0x7f61f4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:48.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f61e97fa700 1 -- 172.21.15.187:0/2918587708 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4013ac0 con 0x7f61fc1575a0 2020-11-20T12:05:48.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f61e97fa700 1 -- 172.21.15.187:0/2918587708 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4013c20 con 0x7f61fc1575a0 2020-11-20T12:05:48.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f61e97fa700 1 -- 172.21.15.187:0/2918587708 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4004880 con 0x7f61fc1575a0 2020-11-20T12:05:48.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.570+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc0b5230 con 0x7f61fc1575a0 2020-11-20T12:05:48.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.571+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc144c80 con 0x7f61fc1575a0 2020-11-20T12:05:48.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.572+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61fc061050 con 0x7f61fc1575a0 2020-11-20T12:05:48.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.572+0000 7f61e97fa700 1 -- 172.21.15.187:0/2918587708 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f61f4027070 con 0x7f61fc1575a0 2020-11-20T12:05:48.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.573+0000 7f61e97fa700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61d4056d20 0x7f61d40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:48.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.573+0000 7f61e97fa700 1 -- 172.21.15.187:0/2918587708 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f61f40711b0 con 0x7f61fc1575a0 2020-11-20T12:05:48.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.573+0000 7f6201106700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61d4056d20 0x7f61d40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:48.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.574+0000 7f6201106700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61d4056d20 0x7f61d40591e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f61f8006d00 tx=0x7f61f8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:48.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.577+0000 7f61e97fa700 1 -- 172.21.15.187:0/2918587708 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f61f4046600 con 0x7f61fc1575a0 2020-11-20T12:05:48.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.763+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f61fc0446d0 con 0x7f61fc1575a0 2020-11-20T12:05:48.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.764+0000 7f61e97fa700 1 -- 172.21.15.187:0/2918587708 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f61f4049c20 con 0x7f61fc1575a0 2020-11-20T12:05:48.768 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:48.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.767+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61d4056d20 msgr2=0x7f61d40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.767+0000 7f6202108700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61d4056d20 0x7f61d40591e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f61f8006d00 tx=0x7f61f8008040).stop 2020-11-20T12:05:48.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.767+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc1575a0 msgr2=0x7f61fc1579e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:48.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.767+0000 7f6202108700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc1575a0 0x7f61fc1579e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f61f4000f40 tx=0x7f61f4015000).stop 2020-11-20T12:05:48.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.768+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 shutdown_connections 2020-11-20T12:05:48.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.768+0000 7f6202108700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61d4056d20 0x7f61d40591e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.768+0000 7f6202108700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61e4004e50 0x7f61fc09e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:48.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.768+0000 7f6202108700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc1575a0 0x7f61fc1579e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.768+0000 7f6202108700 1 --2- 172.21.15.187:0/2918587708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61fc148210 0x7f61fc1439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:48.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.768+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 >> 172.21.15.187:0/2918587708 conn(0x7f61fc0bbeb0 msgr2=0x7f61fc0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:48.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.768+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 shutdown_connections 2020-11-20T12:05:48.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:48.768+0000 7f6202108700 1 -- 172.21.15.187:0/2918587708 wait complete. 2020-11-20T12:05:48.793 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:49.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.038+0000 7fe5849bc700 1 Processor -- start 2020-11-20T12:05:49.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.038+0000 7fe5849bc700 1 -- start start 2020-11-20T12:05:49.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.039+0000 7fe5849bc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065f90 0x7fe580065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.039+0000 7fe5849bc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe58009ca90 0x7fe58009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.039+0000 7fe5849bc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5800b13e0 con 0x7fe580065190 2020-11-20T12:05:49.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.039+0000 7fe5849bc700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58014ab30 con 0x7fe58009ca90 2020-11-20T12:05:49.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.039+0000 7fe5849bc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe580062c10 con 0x7fe580065f90 2020-11-20T12:05:49.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.040+0000 7fe57ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe58009ca90 0x7fe58009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.040+0000 7fe57ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe58009ca90 0x7fe58009cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41522/0 (socket says 172.21.15.187:41522) 2020-11-20T12:05:49.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.040+0000 7fe57ffff700 1 -- 172.21.15.187:0/741972132 learned_addr learned my addr 172.21.15.187:0/741972132 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:49.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.040+0000 7fe57effd700 1 --2- 172.21.15.187:0/741972132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065f90 0x7fe580065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.041+0000 7fe57ffff700 1 -- 172.21.15.187:0/741972132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065f90 msgr2=0x7fe580065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:49.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.041+0000 7fe57ffff700 1 --2- 172.21.15.187:0/741972132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065f90 0x7fe580065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.041+0000 7fe57ffff700 1 -- 172.21.15.187:0/741972132 >> v1:172.21.15.5:6789/0 conn(0x7fe580065190 legacy=0x7fe580143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:49.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.041+0000 7fe57ffff700 1 -- 172.21.15.187:0/741972132 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5800b1640 con 0x7fe58009ca90 2020-11-20T12:05:49.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.041+0000 7fe57ffff700 1 --2- 172.21.15.187:0/741972132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe58009cf20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fe58014ac70 tx=0x7fe578009290).ready entity=mon.1 client_cookie=28edbb047b6c0a15 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:49.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.041+0000 7fe57e7fc700 1 -- 172.21.15.187:0/741972132 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57800bc50 con 0x7fe58009ca90 2020-11-20T12:05:49.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.041+0000 7fe57e7fc700 1 -- 172.21.15.187:0/741972132 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57800bdb0 con 0x7fe58009ca90 2020-11-20T12:05:49.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.041+0000 7fe57e7fc700 1 -- 172.21.15.187:0/741972132 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578011610 con 0x7fe58009ca90 2020-11-20T12:05:49.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.042+0000 7fe5849bc700 1 -- 172.21.15.187:0/741972132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 msgr2=0x7fe58009cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.042+0000 7fe5849bc700 1 --2- 172.21.15.187:0/741972132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe58009cf20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fe58014ac70 tx=0x7fe578009290).stop 2020-11-20T12:05:49.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.042+0000 7fe5849bc700 1 -- 172.21.15.187:0/741972132 shutdown_connections 2020-11-20T12:05:49.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.042+0000 7fe5849bc700 1 --2- 172.21.15.187:0/741972132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe58009cf20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.042+0000 7fe5849bc700 1 --2- 172.21.15.187:0/741972132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065f90 0x7fe580065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.042+0000 7fe5849bc700 1 -- 172.21.15.187:0/741972132 >> 172.21.15.187:0/741972132 conn(0x7fe5800bab30 msgr2=0x7fe5800baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:49.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.042+0000 7fe5849bc700 1 -- 172.21.15.187:0/741972132 shutdown_connections 2020-11-20T12:05:49.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.042+0000 7fe5849bc700 1 -- 172.21.15.187:0/741972132 wait complete. 2020-11-20T12:05:49.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.044+0000 7fe5849bc700 1 Processor -- start 2020-11-20T12:05:49.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.044+0000 7fe5849bc700 1 -- start start 2020-11-20T12:05:49.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe5849bc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065190 0x7fe580146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe5849bc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe580065f90 0x7fe580155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe5849bc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe5800b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe5849bc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe580156fb0 con 0x7fe580065f90 2020-11-20T12:05:49.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe5849bc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800b49b0 con 0x7fe58009ca90 2020-11-20T12:05:49.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe5849bc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801565d0 con 0x7fe580065190 2020-11-20T12:05:49.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe57ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe5800b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe57ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe5800b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41528/0 (socket says 172.21.15.187:41528) 2020-11-20T12:05:49.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe57ffff700 1 -- 172.21.15.187:0/1449089731 learned_addr learned my addr 172.21.15.187:0/1449089731 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:49.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe57ffff700 1 -- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065190 msgr2=0x7fe580146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe57f7fe700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065190 0x7fe580146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.045+0000 7fe57effd700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe580065f90 0x7fe580155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe57ffff700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065190 0x7fe580146670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe57ffff700 1 -- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe580065f90 msgr2=0x7fe580155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe57ffff700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe580065f90 0x7fe580155ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe57ffff700 1 -- 172.21.15.187:0/1449089731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe580156870 con 0x7fe58009ca90 2020-11-20T12:05:49.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe57f7fe700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065190 0x7fe580146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:49.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe57ffff700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe5800b7410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fe5780092d0 tx=0x7fe578016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:49.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe55f7fe700 1 -- 172.21.15.187:0/1449089731 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57800bc50 con 0x7fe58009ca90 2020-11-20T12:05:49.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe55f7fe700 1 -- 172.21.15.187:0/1449089731 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe578010c00 con 0x7fe58009ca90 2020-11-20T12:05:49.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe580156b10 con 0x7fe58009ca90 2020-11-20T12:05:49.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe55f7fe700 1 -- 172.21.15.187:0/1449089731 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57801c470 con 0x7fe58009ca90 2020-11-20T12:05:49.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.046+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe580140d50 con 0x7fe58009ca90 2020-11-20T12:05:49.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.048+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe580156ca0 con 0x7fe58009ca90 2020-11-20T12:05:49.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.048+0000 7fe55f7fe700 1 -- 172.21.15.187:0/1449089731 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe578017070 con 0x7fe58009ca90 2020-11-20T12:05:49.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.049+0000 7fe55f7fe700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe558056d50 0x7fe558059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.049+0000 7fe55f7fe700 1 -- 172.21.15.187:0/1449089731 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe5780721e0 con 0x7fe58009ca90 2020-11-20T12:05:49.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.049+0000 7fe57f7fe700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe558056d50 0x7fe558059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.050+0000 7fe57f7fe700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe558056d50 0x7fe558059210 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe5700014d0 tx=0x7fe570009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:49.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.053+0000 7fe55f7fe700 1 -- 172.21.15.187:0/1449089731 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe5780476b0 con 0x7fe58009ca90 2020-11-20T12:05:49.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.242+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe580061a20 con 0x7fe58009ca90 2020-11-20T12:05:49.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.243+0000 7fe55f7fe700 1 -- 172.21.15.187:0/1449089731 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe578015020 con 0x7fe58009ca90 2020-11-20T12:05:49.246 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:49.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe558056d50 msgr2=0x7fe558059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe558056d50 0x7fe558059210 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe5700014d0 tx=0x7fe570009040).stop 2020-11-20T12:05:49.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 msgr2=0x7fe5800b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe5800b7410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fe5780092d0 tx=0x7fe578016000).stop 2020-11-20T12:05:49.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 shutdown_connections 2020-11-20T12:05:49.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe558056d50 0x7fe558059210 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58009ca90 0x7fe5800b7410 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe580065190 0x7fe580146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 --2- 172.21.15.187:0/1449089731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe580065f90 0x7fe580155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 >> 172.21.15.187:0/1449089731 conn(0x7fe5800bab30 msgr2=0x7fe5800b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:49.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.245+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 shutdown_connections 2020-11-20T12:05:49.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.246+0000 7fe5849bc700 1 -- 172.21.15.187:0/1449089731 wait complete. 2020-11-20T12:05:49.270 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:49.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.503+0000 7f8e57fff700 1 Processor -- start 2020-11-20T12:05:49.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.504+0000 7f8e57fff700 1 -- start start 2020-11-20T12:05:49.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.505+0000 7f8e57fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e581484d0 0x7f8e58061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.505+0000 7f8e57fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 0x7f8e580b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.505+0000 7f8e57fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580b1300 con 0x7f8e580622c0 2020-11-20T12:05:49.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.505+0000 7f8e57fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e58067730 con 0x7f8e581484d0 2020-11-20T12:05:49.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.505+0000 7f8e57fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580b15a0 con 0x7f8e58066f70 2020-11-20T12:05:49.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.506+0000 7f8e56ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e581484d0 0x7f8e58061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.506+0000 7f8e4e7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8e580622c0 0x7f8e580b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38140/0 (socket says 172.21.15.187:38140) 2020-11-20T12:05:49.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.506+0000 7f8e4e7fc700 1 -- 172.21.15.187:0/722641351 learned_addr learned my addr 172.21.15.187:0/722641351 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:49.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.506+0000 7f8e577fe700 1 --2- 172.21.15.187:0/722641351 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 0x7f8e580b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.506+0000 7f8e56ffd700 1 -- 172.21.15.187:0/722641351 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 msgr2=0x7f8e580b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:49.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.506+0000 7f8e56ffd700 1 --2- 172.21.15.187:0/722641351 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 0x7f8e580b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.506+0000 7f8e56ffd700 1 -- 172.21.15.187:0/722641351 >> v1:172.21.15.187:6789/0 conn(0x7f8e580622c0 legacy=0x7f8e580b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.506+0000 7f8e56ffd700 1 -- 172.21.15.187:0/722641351 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580b1840 con 0x7f8e581484d0 2020-11-20T12:05:49.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.507+0000 7f8e56ffd700 1 --2- 172.21.15.187:0/722641351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e58061dd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f8e4800b9a0 tx=0x7f8e48006df0).ready entity=mon.1 client_cookie=d13f266cccee49a2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:49.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.507+0000 7f8e567fc700 1 -- 172.21.15.187:0/722641351 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e480097b0 con 0x7f8e581484d0 2020-11-20T12:05:49.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.507+0000 7f8e567fc700 1 -- 172.21.15.187:0/722641351 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e48009910 con 0x7f8e581484d0 2020-11-20T12:05:49.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.507+0000 7f8e567fc700 1 -- 172.21.15.187:0/722641351 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48009c90 con 0x7f8e581484d0 2020-11-20T12:05:49.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.508+0000 7f8e57fff700 1 -- 172.21.15.187:0/722641351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 msgr2=0x7f8e58061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.508+0000 7f8e57fff700 1 --2- 172.21.15.187:0/722641351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e58061dd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f8e4800b9a0 tx=0x7f8e48006df0).stop 2020-11-20T12:05:49.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.508+0000 7f8e57fff700 1 -- 172.21.15.187:0/722641351 shutdown_connections 2020-11-20T12:05:49.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.508+0000 7f8e57fff700 1 --2- 172.21.15.187:0/722641351 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 0x7f8e580b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.508+0000 7f8e57fff700 1 --2- 172.21.15.187:0/722641351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e58061dd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.508+0000 7f8e57fff700 1 -- 172.21.15.187:0/722641351 >> 172.21.15.187:0/722641351 conn(0x7f8e580bab30 msgr2=0x7f8e580baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:49.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.508+0000 7f8e57fff700 1 -- 172.21.15.187:0/722641351 shutdown_connections 2020-11-20T12:05:49.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.508+0000 7f8e57fff700 1 -- 172.21.15.187:0/722641351 wait complete. 2020-11-20T12:05:49.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.510+0000 7f8e57fff700 1 Processor -- start 2020-11-20T12:05:49.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.510+0000 7f8e57fff700 1 -- start start 2020-11-20T12:05:49.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e57fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e580622c0 0x7f8e58158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e57fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 0x7f8e58144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e57fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e5815c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e57fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e581584f0 con 0x7f8e580622c0 2020-11-20T12:05:49.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e57fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58158790 con 0x7f8e581484d0 2020-11-20T12:05:49.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e57fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58158a30 con 0x7f8e58066f70 2020-11-20T12:05:49.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e577fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e5815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e577fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e5815c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41542/0 (socket says 172.21.15.187:41542) 2020-11-20T12:05:49.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.511+0000 7f8e577fe700 1 -- 172.21.15.187:0/2786909533 learned_addr learned my addr 172.21.15.187:0/2786909533 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:49.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e4e7fc700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 0x7f8e58144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e56ffd700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e580622c0 0x7f8e58158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e577fe700 1 -- 172.21.15.187:0/2786909533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 msgr2=0x7f8e58144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e577fe700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 0x7f8e58144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e577fe700 1 -- 172.21.15.187:0/2786909533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e580622c0 msgr2=0x7f8e58158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e577fe700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e580622c0 0x7f8e58158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e577fe700 1 -- 172.21.15.187:0/2786909533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58158210 con 0x7f8e581484d0 2020-11-20T12:05:49.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e577fe700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e5815c090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f8e5000eb40 tx=0x7f8e50008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:49.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e4f7fe700 1 -- 172.21.15.187:0/2786909533 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5000c7a0 con 0x7f8e581484d0 2020-11-20T12:05:49.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e4f7fe700 1 -- 172.21.15.187:0/2786909533 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5000c900 con 0x7f8e581484d0 2020-11-20T12:05:49.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.512+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e58144820 con 0x7f8e581484d0 2020-11-20T12:05:49.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.513+0000 7f8e4f7fe700 1 -- 172.21.15.187:0/2786909533 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5000cca0 con 0x7f8e581484d0 2020-11-20T12:05:49.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.513+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5815c580 con 0x7f8e581484d0 2020-11-20T12:05:49.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.514+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e58158bf0 con 0x7f8e581484d0 2020-11-20T12:05:49.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.514+0000 7f8e4f7fe700 1 -- 172.21.15.187:0/2786909533 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8e50013640 con 0x7f8e581484d0 2020-11-20T12:05:49.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.515+0000 7f8e4f7fe700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e34056d20 0x7f8e340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.515+0000 7f8e4f7fe700 1 -- 172.21.15.187:0/2786909533 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e5006fdb0 con 0x7f8e581484d0 2020-11-20T12:05:49.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.515+0000 7f8e56ffd700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e34056d20 0x7f8e340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:49.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.516+0000 7f8e56ffd700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e34056d20 0x7f8e340591e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f8e48006e30 tx=0x7f8e48015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:49.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.519+0000 7f8e4f7fe700 1 -- 172.21.15.187:0/2786909533 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8e50074070 con 0x7f8e581484d0 2020-11-20T12:05:49.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.703+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e58061a20 con 0x7f8e581484d0 2020-11-20T12:05:49.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.705+0000 7f8e4f7fe700 1 -- 172.21.15.187:0/2786909533 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f8e500450f0 con 0x7f8e581484d0 2020-11-20T12:05:49.709 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:49.710 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:49.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.707+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e34056d20 msgr2=0x7f8e340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.707+0000 7f8e57fff700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e34056d20 0x7f8e340591e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f8e48006e30 tx=0x7f8e48015000).stop 2020-11-20T12:05:49.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.707+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 msgr2=0x7f8e5815c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:49.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.707+0000 7f8e57fff700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e5815c090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f8e5000eb40 tx=0x7f8e50008040).stop 2020-11-20T12:05:49.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.708+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 shutdown_connections 2020-11-20T12:05:49.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.708+0000 7f8e57fff700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e34056d20 0x7f8e340591e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.708+0000 7f8e57fff700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e580622c0 0x7f8e58158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.708+0000 7f8e57fff700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e581484d0 0x7f8e5815c090 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.708+0000 7f8e57fff700 1 --2- 172.21.15.187:0/2786909533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e58066f70 0x7f8e58144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:49.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.708+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 >> 172.21.15.187:0/2786909533 conn(0x7f8e580bab30 msgr2=0x7f8e580b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:49.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.708+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 shutdown_connections 2020-11-20T12:05:49.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.708+0000 7f8e57fff700 1 -- 172.21.15.187:0/2786909533 wait complete. 2020-11-20T12:05:49.733 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:49.734 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:49.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.978+0000 7f8ced905700 1 Processor -- start 2020-11-20T12:05:49.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.978+0000 7f8ced905700 1 -- start start 2020-11-20T12:05:49.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.980+0000 7f8ced905700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 0x7f8ce8153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:49.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.980+0000 7f8ced905700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce814d110 con 0x7f8ce80bf800 2020-11-20T12:05:49.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.980+0000 7f8ced905700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce814cbd0 con 0x7f8ce80bfbe0 2020-11-20T12:05:49.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.980+0000 7f8ced905700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce814c9d0 con 0x7f8ce809f560 2020-11-20T12:05:49.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.980+0000 7f8ce7fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8ce80bfbe0 0x7f8ce81480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38154/0 (socket says 172.21.15.187:38154) 2020-11-20T12:05:50.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.980+0000 7f8ce7fff700 1 -- 172.21.15.187:0/1219972686 learned_addr learned my addr 172.21.15.187:0/1219972686 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:50.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.980+0000 7f8ced104700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 0x7f8ce8153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401426186 0 0) 0x7f8ce814cbd0 con 0x7f8ce80bfbe0 2020-11-20T12:05:50.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ccc003240 con 0x7f8ce80bfbe0 2020-11-20T12:05:50.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1293825023 0 0) 0x7f8ccc003240 con 0x7f8ce80bfbe0 2020-11-20T12:05:50.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 msgr2=0x7f8ce8153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:50.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 0x7f8ce8153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 >> v1:172.21.15.5:6789/0 conn(0x7f8ce80bf800 legacy=0x7f8ce814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce814ce70 con 0x7f8ce80bfbe0 2020-11-20T12:05:50.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8cd40014e0 con 0x7f8ce80bfbe0 2020-11-20T12:05:50.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 >> v1:172.21.15.187:6789/0 conn(0x7f8ce80bfbe0 legacy=0x7f8ce81480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 0x7f8ccc003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ccc003c40 0x7f8ccc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ccc0049d0 0x7f8ccc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce814c9d0 con 0x7f8ccc0049d0 2020-11-20T12:05:50.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc0041d0 con 0x7f8ccc003c40 2020-11-20T12:05:50.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.981+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc00b730 con 0x7f8ce809f560 2020-11-20T12:05:50.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.982+0000 7f8cec903700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 0x7f8ccc003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.982+0000 7f8ce7fff700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ccc003c40 0x7f8ccc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.982+0000 7f8ced104700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ccc0049d0 0x7f8ccc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.983+0000 7f8cec903700 1 -- 172.21.15.187:0/1219972686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ccc003c40 msgr2=0x7f8ccc0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:50.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.983+0000 7f8cec903700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ccc003c40 0x7f8ccc0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.983+0000 7f8cec903700 1 -- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ccc0049d0 msgr2=0x7f8ccc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:50.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.983+0000 7f8cec903700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ccc0049d0 0x7f8ccc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.983+0000 7f8cec903700 1 -- 172.21.15.187:0/1219972686 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc003240 con 0x7f8ce809f560 2020-11-20T12:05:50.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.984+0000 7f8cec903700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 0x7f8ccc003730 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8cdc001920 tx=0x7f8cdc007b90).ready entity=mon.2 client_cookie=d480d4c4da21ba69 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.984+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc007020 con 0x7f8ce809f560 2020-11-20T12:05:50.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.984+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc007180 con 0x7f8ce809f560 2020-11-20T12:05:50.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.984+0000 7f8ce77fe700 1 -- 172.21.15.187:0/1219972686 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc00c830 con 0x7f8ce809f560 2020-11-20T12:05:50.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.984+0000 7f8ced905700 1 -- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 msgr2=0x7f8ccc003730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.984+0000 7f8ced905700 1 --2- 172.21.15.187:0/1219972686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce809f560 0x7f8ccc003730 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8cdc001920 tx=0x7f8cdc007b90).stop 2020-11-20T12:05:50.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.985+0000 7f8ced104700 1 -- 172.21.15.187:0/1219972686 reap_dead start 2020-11-20T12:05:50.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.985+0000 7f8ced905700 1 -- 172.21.15.187:0/1219972686 shutdown_connections 2020-11-20T12:05:50.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.985+0000 7f8ced905700 1 -- 172.21.15.187:0/1219972686 >> 172.21.15.187:0/1219972686 conn(0x7f8ce8068370 msgr2=0x7f8ce80bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:50.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.985+0000 7f8ced905700 1 -- 172.21.15.187:0/1219972686 shutdown_connections 2020-11-20T12:05:50.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.985+0000 7f8ced905700 1 -- 172.21.15.187:0/1219972686 wait complete. 2020-11-20T12:05:50.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.986+0000 7f8ced905700 1 Processor -- start 2020-11-20T12:05:50.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.986+0000 7f8ced905700 1 -- start start 2020-11-20T12:05:50.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.987+0000 7f8ced905700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce81411a0 0x7f8ce8147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.987+0000 7f8ced905700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80b4ac0 0x7f8ce815be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.987+0000 7f8ced905700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce8141580 0x7f8ce8143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.987+0000 7f8ced905700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce815c830 con 0x7f8ce80b4ac0 2020-11-20T12:05:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.987+0000 7f8ced905700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce815cad0 con 0x7f8ce8141580 2020-11-20T12:05:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.987+0000 7f8ced905700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce80b47c0 con 0x7f8ce81411a0 2020-11-20T12:05:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce8141580 0x7f8ce8143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce8141580 0x7f8ce8143ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41560/0 (socket says 172.21.15.187:41560) 2020-11-20T12:05:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 -- 172.21.15.187:0/3536938241 learned_addr learned my addr 172.21.15.187:0/3536938241 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:50.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8cec903700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce81411a0 0x7f8ce8147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 -- 172.21.15.187:0/3536938241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce81411a0 msgr2=0x7f8ce8147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce81411a0 0x7f8ce8147320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 -- 172.21.15.187:0/3536938241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80b4ac0 msgr2=0x7f8ce815be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80b4ac0 0x7f8ce815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:50.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 -- 172.21.15.187:0/3536938241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce8147810 con 0x7f8ce8141580 2020-11-20T12:05:50.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.988+0000 7f8ced104700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce8141580 0x7f8ce8143ad0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f8ce80a46f0 tx=0x7f8ce0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.989+0000 7f8ce4ff9700 1 -- 172.21.15.187:0/3536938241 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce000c070 con 0x7f8ce8141580 2020-11-20T12:05:50.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.989+0000 7f8ce4ff9700 1 -- 172.21.15.187:0/3536938241 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce000f8b0 con 0x7f8ce8141580 2020-11-20T12:05:50.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.989+0000 7f8ce4ff9700 1 -- 172.21.15.187:0/3536938241 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce000fc70 con 0x7f8ce8141580 2020-11-20T12:05:50.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.989+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce80b4390 con 0x7f8ce8141580 2020-11-20T12:05:50.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.989+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce815c540 con 0x7f8ce8141580 2020-11-20T12:05:50.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.990+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce8080050 con 0x7f8ce8141580 2020-11-20T12:05:50.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.990+0000 7f8ce4ff9700 1 -- 172.21.15.187:0/3536938241 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8ce0012040 con 0x7f8ce8141580 2020-11-20T12:05:50.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.991+0000 7f8ce4ff9700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.991+0000 7f8ce4ff9700 1 -- 172.21.15.187:0/3536938241 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8ce0070140 con 0x7f8ce8141580 2020-11-20T12:05:50.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.992+0000 7f8cec903700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.992+0000 7f8cec903700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8ce80a1bc0 tx=0x7f8cdc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:49.995+0000 7f8ce4ff9700 1 -- 172.21.15.187:0/3536938241 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8ce0045510 con 0x7f8ce8141580 2020-11-20T12:05:50.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.189+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8ce815cc60 con 0x7f8ce8141580 2020-11-20T12:05:50.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.189+0000 7f8ce4ff9700 1 -- 172.21.15.187:0/3536938241 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8ce0048b30 con 0x7f8ce8141580 2020-11-20T12:05:50.194 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:50.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 msgr2=0x7f8cc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8ce80a1bc0 tx=0x7f8cdc015000).stop 2020-11-20T12:05:50.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce8141580 msgr2=0x7f8ce8143ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce8141580 0x7f8ce8143ad0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f8ce80a46f0 tx=0x7f8ce0008040).stop 2020-11-20T12:05:50.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 shutdown_connections 2020-11-20T12:05:50.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce8141580 0x7f8ce8143ad0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ce81411a0 0x7f8ce8147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 --2- 172.21.15.187:0/3536938241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80b4ac0 0x7f8ce815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:50.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.192+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 >> 172.21.15.187:0/3536938241 conn(0x7f8ce8068370 msgr2=0x7f8ce80be6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:50.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.193+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 shutdown_connections 2020-11-20T12:05:50.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.193+0000 7f8ced905700 1 -- 172.21.15.187:0/3536938241 wait complete. 2020-11-20T12:05:50.218 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:50.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.463+0000 7f8e0e1a6700 1 Processor -- start 2020-11-20T12:05:50.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.463+0000 7f8e0e1a6700 1 -- start start 2020-11-20T12:05:50.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.465+0000 7f8e0e1a6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e080bf800 0x7f8e0814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.465+0000 7f8e0e1a6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bfbe0 0x7f8e081480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.465+0000 7f8e0e1a6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0814d110 con 0x7f8e0809f560 2020-11-20T12:05:50.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.465+0000 7f8e0e1a6700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0814cbd0 con 0x7f8e080bf800 2020-11-20T12:05:50.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.465+0000 7f8e0e1a6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0814ce70 con 0x7f8e080bfbe0 2020-11-20T12:05:50.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.466+0000 7f8e0d1a4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e080bf800 0x7f8e0814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.466+0000 7f8e0c9a3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bfbe0 0x7f8e081480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.466+0000 7f8e0d9a5700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8e0809f560 0x7f8e08060c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:59918/0 (socket says 172.21.15.187:59918) 2020-11-20T12:05:50.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.466+0000 7f8e0d9a5700 1 -- 172.21.15.187:0/2317808574 learned_addr learned my addr 172.21.15.187:0/2317808574 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:50.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.467+0000 7f8e0d1a4700 1 -- 172.21.15.187:0/2317808574 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bfbe0 msgr2=0x7f8e081480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:50.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.467+0000 7f8df7fff700 1 -- 172.21.15.187:0/2317808574 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949040018 0 0) 0x7f8e0814d110 con 0x7f8e0809f560 2020-11-20T12:05:50.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.467+0000 7f8e0d1a4700 1 --2- 172.21.15.187:0/2317808574 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bfbe0 0x7f8e081480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.467+0000 7f8e0d1a4700 1 -- 172.21.15.187:0/2317808574 >> v1:172.21.15.5:6789/0 conn(0x7f8e0809f560 legacy=0x7f8e08060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.467+0000 7f8e0d1a4700 1 -- 172.21.15.187:0/2317808574 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e080a4cc0 con 0x7f8e080bf800 2020-11-20T12:05:50.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.467+0000 7f8e0d1a4700 1 --2- 172.21.15.187:0/2317808574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e080bf800 0x7f8e0814c2c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8e04009c90 tx=0x7f8e040091d0).ready entity=mon.1 client_cookie=90e2d6cdf57fe1b5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8df7fff700 1 -- 172.21.15.187:0/2317808574 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0400bb00 con 0x7f8e080bf800 2020-11-20T12:05:50.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8df7fff700 1 -- 172.21.15.187:0/2317808574 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0400bc60 con 0x7f8e080bf800 2020-11-20T12:05:50.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8df7fff700 1 -- 172.21.15.187:0/2317808574 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04011400 con 0x7f8e080bf800 2020-11-20T12:05:50.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/2317808574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e080bf800 msgr2=0x7f8e0814c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/2317808574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e080bf800 0x7f8e0814c2c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8e04009c90 tx=0x7f8e040091d0).stop 2020-11-20T12:05:50.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/2317808574 shutdown_connections 2020-11-20T12:05:50.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/2317808574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e080bf800 0x7f8e0814c2c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/2317808574 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bfbe0 0x7f8e081480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.468+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/2317808574 >> 172.21.15.187:0/2317808574 conn(0x7f8e08068370 msgr2=0x7f8e080bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:50.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.469+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/2317808574 shutdown_connections 2020-11-20T12:05:50.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.469+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/2317808574 wait complete. 2020-11-20T12:05:50.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.470+0000 7f8e0e1a6700 1 Processor -- start 2020-11-20T12:05:50.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.470+0000 7f8e0e1a6700 1 -- start start 2020-11-20T12:05:50.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.471+0000 7f8e0e1a6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e0809f560 0x7f8e080b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.471+0000 7f8e0e1a6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bf800 0x7f8e080b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.471+0000 7f8e0e1a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e081458a0 0x7f8e08143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.471+0000 7f8e0e1a6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e081466c0 con 0x7f8e0809f560 2020-11-20T12:05:50.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.471+0000 7f8e0e1a6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0814a360 con 0x7f8e081458a0 2020-11-20T12:05:50.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.471+0000 7f8e0e1a6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0814a600 con 0x7f8e080bf800 2020-11-20T12:05:50.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.471+0000 7f8e0d9a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e081458a0 0x7f8e08143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d9a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e081458a0 0x7f8e08143d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41576/0 (socket says 172.21.15.187:41576) 2020-11-20T12:05:50.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d9a5700 1 -- 172.21.15.187:0/3790751032 learned_addr learned my addr 172.21.15.187:0/3790751032 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:50.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0c9a3700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bf800 0x7f8e080b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d1a4700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e0809f560 0x7f8e080b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d9a5700 1 -- 172.21.15.187:0/3790751032 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bf800 msgr2=0x7f8e080b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d9a5700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bf800 0x7f8e080b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d9a5700 1 -- 172.21.15.187:0/3790751032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e0809f560 msgr2=0x7f8e080b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d9a5700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e0809f560 0x7f8e080b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d9a5700 1 -- 172.21.15.187:0/3790751032 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0814a8a0 con 0x7f8e081458a0 2020-11-20T12:05:50.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.472+0000 7f8e0d9a5700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e081458a0 0x7f8e08143d90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8e0000c730 tx=0x7f8e00007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.473+0000 7f8df57fa700 1 -- 172.21.15.187:0/3790751032 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00007320 con 0x7f8e081458a0 2020-11-20T12:05:50.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.473+0000 7f8df57fa700 1 -- 172.21.15.187:0/3790751032 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00007480 con 0x7f8e081458a0 2020-11-20T12:05:50.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.473+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e080b4a30 con 0x7f8e081458a0 2020-11-20T12:05:50.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.473+0000 7f8df57fa700 1 -- 172.21.15.187:0/3790751032 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00005800 con 0x7f8e081458a0 2020-11-20T12:05:50.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.473+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0814afe0 con 0x7f8e081458a0 2020-11-20T12:05:50.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.474+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e08080050 con 0x7f8e081458a0 2020-11-20T12:05:50.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.474+0000 7f8df57fa700 1 -- 172.21.15.187:0/3790751032 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8e00010030 con 0x7f8e081458a0 2020-11-20T12:05:50.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.475+0000 7f8df57fa700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8de0056d50 0x7f8de0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.475+0000 7f8df57fa700 1 -- 172.21.15.187:0/3790751032 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e0006fcc0 con 0x7f8e081458a0 2020-11-20T12:05:50.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.475+0000 7f8e0d1a4700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8de0056d50 0x7f8de0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.476+0000 7f8e0d1a4700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8de0056d50 0x7f8de0059210 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f8e04009440 tx=0x7f8e04016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.479+0000 7f8df57fa700 1 -- 172.21.15.187:0/3790751032 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8e000450b0 con 0x7f8e081458a0 2020-11-20T12:05:50.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.666+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8e080446d0 con 0x7f8e081458a0 2020-11-20T12:05:50.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.666+0000 7f8df57fa700 1 -- 172.21.15.187:0/3790751032 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8e000486d0 con 0x7f8e081458a0 2020-11-20T12:05:50.670 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:50.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8de0056d50 msgr2=0x7f8de0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8de0056d50 0x7f8de0059210 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f8e04009440 tx=0x7f8e04016000).stop 2020-11-20T12:05:50.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e081458a0 msgr2=0x7f8e08143d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e081458a0 0x7f8e08143d90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8e0000c730 tx=0x7f8e00007a60).stop 2020-11-20T12:05:50.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 shutdown_connections 2020-11-20T12:05:50.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8de0056d50 0x7f8de0059210 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e0809f560 0x7f8e080b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e081458a0 0x7f8e08143d90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 --2- 172.21.15.187:0/3790751032 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e080bf800 0x7f8e080b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 >> 172.21.15.187:0/3790751032 conn(0x7f8e08068370 msgr2=0x7f8e080a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:50.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 shutdown_connections 2020-11-20T12:05:50.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.669+0000 7f8e0e1a6700 1 -- 172.21.15.187:0/3790751032 wait complete. 2020-11-20T12:05:50.694 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:50.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.932+0000 7f3e2cce8700 1 Processor -- start 2020-11-20T12:05:50.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.932+0000 7f3e2cce8700 1 -- start start 2020-11-20T12:05:50.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.933+0000 7f3e2cce8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e2809a8a0 0x7f3e280656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.933+0000 7f3e2cce8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 0x7f3e28146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.933+0000 7f3e2cce8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e28066420 con 0x7f3e28065300 2020-11-20T12:05:50.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.933+0000 7f3e2cce8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e280b7380 con 0x7f3e2809a8a0 2020-11-20T12:05:50.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.933+0000 7f3e2cce8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e28061f60 con 0x7f3e28068030 2020-11-20T12:05:50.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.934+0000 7f3e1dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e2809a8a0 0x7f3e280656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.934+0000 7f3e1dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e2809a8a0 0x7f3e280656e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41582/0 (socket says 172.21.15.187:41582) 2020-11-20T12:05:50.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.934+0000 7f3e1dd9b700 1 -- 172.21.15.187:0/1157571215 learned_addr learned my addr 172.21.15.187:0/1157571215 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:50.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.934+0000 7f3e1dd9b700 1 -- 172.21.15.187:0/1157571215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 msgr2=0x7f3e28146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:50.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.934+0000 7f3e1dd9b700 1 --2- 172.21.15.187:0/1157571215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 0x7f3e28146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:50.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.934+0000 7f3e1dd9b700 1 -- 172.21.15.187:0/1157571215 >> v1:172.21.15.5:6789/0 conn(0x7f3e28065300 legacy=0x7f3e28062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:50.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.934+0000 7f3e1dd9b700 1 -- 172.21.15.187:0/1157571215 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e280b6fd0 con 0x7f3e2809a8a0 2020-11-20T12:05:50.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.935+0000 7f3e1dd9b700 1 --2- 172.21.15.187:0/1157571215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e280656e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f3e10006fd0 tx=0x7f3e10006eb0).ready entity=mon.1 client_cookie=914077a479de2396 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.935+0000 7f3e25d9b700 1 -- 172.21.15.187:0/1157571215 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10009920 con 0x7f3e2809a8a0 2020-11-20T12:05:50.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.935+0000 7f3e25d9b700 1 -- 172.21.15.187:0/1157571215 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e10009a80 con 0x7f3e2809a8a0 2020-11-20T12:05:50.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.935+0000 7f3e25d9b700 1 -- 172.21.15.187:0/1157571215 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10009e00 con 0x7f3e2809a8a0 2020-11-20T12:05:50.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.936+0000 7f3e2cce8700 1 -- 172.21.15.187:0/1157571215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 msgr2=0x7f3e280656e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.936+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/1157571215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e280656e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f3e10006fd0 tx=0x7f3e10006eb0).stop 2020-11-20T12:05:50.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.936+0000 7f3e2cce8700 1 -- 172.21.15.187:0/1157571215 shutdown_connections 2020-11-20T12:05:50.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.936+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/1157571215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 0x7f3e28146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:50.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.936+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/1157571215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e280656e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.936+0000 7f3e2cce8700 1 -- 172.21.15.187:0/1157571215 >> 172.21.15.187:0/1157571215 conn(0x7f3e280bacf0 msgr2=0x7f3e280bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:50.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.936+0000 7f3e2cce8700 1 -- 172.21.15.187:0/1157571215 shutdown_connections 2020-11-20T12:05:50.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.936+0000 7f3e2cce8700 1 -- 172.21.15.187:0/1157571215 wait complete. 2020-11-20T12:05:50.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.937+0000 7f3e2cce8700 1 Processor -- start 2020-11-20T12:05:50.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.938+0000 7f3e2cce8700 1 -- start start 2020-11-20T12:05:50.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.938+0000 7f3e2cce8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e28065300 0x7f3e28138340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.938+0000 7f3e2cce8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 0x7f3e280b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e2cce8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e2809de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e2cce8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2809fa50 con 0x7f3e28065300 2020-11-20T12:05:50.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e2cce8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2809fcf0 con 0x7f3e2809a8a0 2020-11-20T12:05:50.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e2cce8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2814ad50 con 0x7f3e28068030 2020-11-20T12:05:50.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e26d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e2809de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e26d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e2809de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41590/0 (socket says 172.21.15.187:41590) 2020-11-20T12:05:50.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e26d9d700 1 -- 172.21.15.187:0/253759033 learned_addr learned my addr 172.21.15.187:0/253759033 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:50.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e1dd9b700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 0x7f3e280b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e26d9d700 1 -- 172.21.15.187:0/253759033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 msgr2=0x7f3e280b8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e26d9d700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 0x7f3e280b8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:50.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e26d9d700 1 -- 172.21.15.187:0/253759033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e28065300 msgr2=0x7f3e28138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:50.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e26d9d700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e28065300 0x7f3e28138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:50.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.939+0000 7f3e26d9d700 1 -- 172.21.15.187:0/253759033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e28137ed0 con 0x7f3e2809a8a0 2020-11-20T12:05:50.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.940+0000 7f3e26d9d700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e2809de30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f3e2000bae0 tx=0x7f3e20009220).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.940+0000 7f3e1effd700 1 -- 172.21.15.187:0/253759033 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000c930 con 0x7f3e2809a8a0 2020-11-20T12:05:50.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.940+0000 7f3e1effd700 1 -- 172.21.15.187:0/253759033 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2000ca90 con 0x7f3e2809a8a0 2020-11-20T12:05:50.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.940+0000 7f3e1effd700 1 -- 172.21.15.187:0/253759033 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000cc40 con 0x7f3e2809a8a0 2020-11-20T12:05:50.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.940+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2809ec50 con 0x7f3e2809a8a0 2020-11-20T12:05:50.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.940+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e280b9fd0 con 0x7f3e2809a8a0 2020-11-20T12:05:50.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.941+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2809ede0 con 0x7f3e2809a8a0 2020-11-20T12:05:50.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.942+0000 7f3e1effd700 1 -- 172.21.15.187:0/253759033 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3e2001c070 con 0x7f3e2809a8a0 2020-11-20T12:05:50.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.943+0000 7f3e1effd700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3df8056d20 0x7f3df80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:50.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.943+0000 7f3e1effd700 1 -- 172.21.15.187:0/253759033 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e2006f870 con 0x7f3e2809a8a0 2020-11-20T12:05:50.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.943+0000 7f3e2659c700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3df8056d20 0x7f3df80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:50.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.944+0000 7f3e2659c700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3df8056d20 0x7f3df80591e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3e18006fd0 tx=0x7f3e18006910).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:50.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:50.947+0000 7f3e1effd700 1 -- 172.21.15.187:0/253759033 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3e20044bb0 con 0x7f3e2809a8a0 2020-11-20T12:05:51.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.127+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e28044f60 con 0x7f3e2809a8a0 2020-11-20T12:05:51.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.128+0000 7f3e1effd700 1 -- 172.21.15.187:0/253759033 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3e200481d0 con 0x7f3e2809a8a0 2020-11-20T12:05:51.132 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:51.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3df8056d20 msgr2=0x7f3df80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3df8056d20 0x7f3df80591e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3e18006fd0 tx=0x7f3e18006910).stop 2020-11-20T12:05:51.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 msgr2=0x7f3e2809de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e2809de30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f3e2000bae0 tx=0x7f3e20009220).stop 2020-11-20T12:05:51.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 shutdown_connections 2020-11-20T12:05:51.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3df8056d20 0x7f3df80591e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e28065300 0x7f3e28138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e2809a8a0 0x7f3e2809de30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 --2- 172.21.15.187:0/253759033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e28068030 0x7f3e280b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 >> 172.21.15.187:0/253759033 conn(0x7f3e280bacf0 msgr2=0x7f3e28099c00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:51.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 shutdown_connections 2020-11-20T12:05:51.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.131+0000 7f3e2cce8700 1 -- 172.21.15.187:0/253759033 wait complete. 2020-11-20T12:05:51.156 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:51.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.393+0000 7f5ed1f6a700 1 Processor -- start 2020-11-20T12:05:51.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.393+0000 7f5ed1f6a700 1 -- start start 2020-11-20T12:05:51.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.394+0000 7f5ed1f6a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0bf800 0x7f5ecc14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.394+0000 7f5ed1f6a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc14d110 con 0x7f5ecc0bfbe0 2020-11-20T12:05:51.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.395+0000 7f5ed1f6a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc14cbd0 con 0x7f5ecc09f560 2020-11-20T12:05:51.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.395+0000 7f5ed1f6a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc14ce70 con 0x7f5ecc0bf800 2020-11-20T12:05:51.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.395+0000 7f5ed1769700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5ecc09f560 0x7f5ecc0a36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38204/0 (socket says 172.21.15.187:38204) 2020-11-20T12:05:51.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.395+0000 7f5ed1769700 1 -- 172.21.15.187:0/4144829052 learned_addr learned my addr 172.21.15.187:0/4144829052 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:51.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.395+0000 7f5ed0f68700 1 --2- 172.21.15.187:0/4144829052 >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0bf800 0x7f5ecc14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705975477 0 0) 0x7f5ecc14cbd0 con 0x7f5ecc09f560 2020-11-20T12:05:51.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8003240 con 0x7f5ecc09f560 2020-11-20T12:05:51.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2851857199 0 0) 0x7f5eb8003240 con 0x7f5ecc09f560 2020-11-20T12:05:51.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0bf800 msgr2=0x7f5ecc14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:51.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 --2- 172.21.15.187:0/4144829052 >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0bf800 0x7f5ecc14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 >> v1:172.21.15.5:6789/0 conn(0x7f5ecc0bfbe0 legacy=0x7f5ecc1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0a4cc0 con 0x7f5ecc09f560 2020-11-20T12:05:51.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5ec40018a0 con 0x7f5ecc09f560 2020-11-20T12:05:51.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 >> v1:172.21.15.187:6789/0 conn(0x7f5ecc09f560 legacy=0x7f5ecc0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5eb8003730 0x7f5eb8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5eb80044e0 0x7f5eb8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5eb8004e50 0x7f5eb800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc14ce70 con 0x7f5eb80044e0 2020-11-20T12:05:51.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8004260 con 0x7f5eb8003730 2020-11-20T12:05:51.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.396+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb800bc20 con 0x7f5eb8004e50 2020-11-20T12:05:51.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ed0f68700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5eb8003730 0x7f5eb8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ec3fff700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5eb80044e0 0x7f5eb8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ed0f68700 1 -- 172.21.15.187:0/4144829052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5eb8004e50 msgr2=0x7f5eb800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:51.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ed0f68700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5eb8004e50 0x7f5eb800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ed0f68700 1 -- 172.21.15.187:0/4144829052 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5eb80044e0 msgr2=0x7f5eb8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:51.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ed0f68700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5eb80044e0 0x7f5eb8004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ed0f68700 1 -- 172.21.15.187:0/4144829052 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8003240 con 0x7f5eb8003730 2020-11-20T12:05:51.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ed1769700 1 -- 172.21.15.187:0/4144829052 reap_dead start 2020-11-20T12:05:51.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ec3fff700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5eb80044e0 0x7f5eb8004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:51.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.397+0000 7f5ed0f68700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5eb8003730 0x7f5eb8003b50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f5ec800bc00 tx=0x7f5ec8006df0).ready entity=mon.1 client_cookie=43c7074d7045cf60 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:51.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec800b310 con 0x7f5eb8003730 2020-11-20T12:05:51.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec800eb40 con 0x7f5eb8003730 2020-11-20T12:05:51.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ec37fe700 1 -- 172.21.15.187:0/4144829052 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec800eec0 con 0x7f5eb8003730 2020-11-20T12:05:51.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/4144829052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5eb8003730 msgr2=0x7f5eb8003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ed1f6a700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5eb8003730 0x7f5eb8003b50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f5ec800bc00 tx=0x7f5ec8006df0).stop 2020-11-20T12:05:51.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/4144829052 shutdown_connections 2020-11-20T12:05:51.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ed1f6a700 1 --2- 172.21.15.187:0/4144829052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5eb8003730 0x7f5eb8003b50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/4144829052 >> 172.21.15.187:0/4144829052 conn(0x7f5ecc068370 msgr2=0x7f5ecc0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:51.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/4144829052 shutdown_connections 2020-11-20T12:05:51.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.398+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/4144829052 wait complete. 2020-11-20T12:05:51.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.400+0000 7f5ed1f6a700 1 Processor -- start 2020-11-20T12:05:51.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.400+0000 7f5ed1f6a700 1 -- start start 2020-11-20T12:05:51.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed1f6a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b6580 0x7f5ecc0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed1f6a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b6960 0x7f5ecc141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed1f6a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc14bbe0 0x7f5ecc1488a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed1f6a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0bae30 con 0x7f5ecc0b6960 2020-11-20T12:05:51.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed1f6a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0b3fa0 con 0x7f5ecc0b6580 2020-11-20T12:05:51.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed1f6a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0b4240 con 0x7f5ecc14bbe0 2020-11-20T12:05:51.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed0f68700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b6580 0x7f5ecc0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed0f68700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b6580 0x7f5ecc0b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41606/0 (socket says 172.21.15.187:41606) 2020-11-20T12:05:51.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed0f68700 1 -- 172.21.15.187:0/2652038275 learned_addr learned my addr 172.21.15.187:0/2652038275 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:51.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.401+0000 7f5ed0f68700 1 -- 172.21.15.187:0/2652038275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc14bbe0 msgr2=0x7f5ecc1488a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ed0f68700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc14bbe0 0x7f5ecc1488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ed0f68700 1 -- 172.21.15.187:0/2652038275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b6960 msgr2=0x7f5ecc141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ed0f68700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b6960 0x7f5ecc141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ed0f68700 1 -- 172.21.15.187:0/2652038275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0baab0 con 0x7f5ecc0b6580 2020-11-20T12:05:51.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ed0f68700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b6580 0x7f5ecc0b6e20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f5ec8006fd0 tx=0x7f5ec8019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:51.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/2652038275 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec801d620 con 0x7f5ecc0b6580 2020-11-20T12:05:51.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/2652038275 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec801d780 con 0x7f5ecc0b6580 2020-11-20T12:05:51.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/2652038275 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec801d970 con 0x7f5ecc0b6580 2020-11-20T12:05:51.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc0b7310 con 0x7f5ecc0b6580 2020-11-20T12:05:51.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.402+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc0bb1c0 con 0x7f5ecc0b6580 2020-11-20T12:05:51.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.403+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc080050 con 0x7f5ecc0b6580 2020-11-20T12:05:51.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.404+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/2652038275 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5ec802d070 con 0x7f5ecc0b6580 2020-11-20T12:05:51.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.404+0000 7f5ec0ff9700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.404+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/2652038275 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5ec8072050 con 0x7f5ecc0b6580 2020-11-20T12:05:51.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.405+0000 7f5ec3fff700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.405+0000 7f5ec3fff700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f5ecc14a000 tx=0x7f5ebc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:51.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.408+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/2652038275 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5ec8047520 con 0x7f5ecc0b6580 2020-11-20T12:05:51.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.595+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ecc0b4400 con 0x7f5ecc0b6580 2020-11-20T12:05:51.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.596+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/2652038275 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5ec804ab40 con 0x7f5ecc0b6580 2020-11-20T12:05:51.601 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:51.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.598+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 msgr2=0x7f5ea40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.598+0000 7f5ed1f6a700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f5ecc14a000 tx=0x7f5ebc009040).stop 2020-11-20T12:05:51.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.598+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b6580 msgr2=0x7f5ecc0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.598+0000 7f5ed1f6a700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b6580 0x7f5ecc0b6e20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f5ec8006fd0 tx=0x7f5ec8019000).stop 2020-11-20T12:05:51.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.599+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 shutdown_connections 2020-11-20T12:05:51.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.599+0000 7f5ed1f6a700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.599+0000 7f5ed1f6a700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc14bbe0 0x7f5ecc1488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.599+0000 7f5ed1f6a700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b6580 0x7f5ecc0b6e20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.599+0000 7f5ed1f6a700 1 --2- 172.21.15.187:0/2652038275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b6960 0x7f5ecc141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.599+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 >> 172.21.15.187:0/2652038275 conn(0x7f5ecc068370 msgr2=0x7f5ecc0bb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:51.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.599+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 shutdown_connections 2020-11-20T12:05:51.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.599+0000 7f5ed1f6a700 1 -- 172.21.15.187:0/2652038275 wait complete. 2020-11-20T12:05:51.643 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:05:51.644 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2020-11-20T12:05:51.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.881+0000 7f945bfff700 1 Processor -- start 2020-11-20T12:05:51.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.881+0000 7f945bfff700 1 -- start start 2020-11-20T12:05:51.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f945c0b6650 0x7f945c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f945c066f70 0x7f945c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c0b1300 con 0x7f945c1484d0 2020-11-20T12:05:51.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c067730 con 0x7f945c066f70 2020-11-20T12:05:51.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0b15a0 con 0x7f945c0b6650 2020-11-20T12:05:51.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f945c066f70 0x7f945c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f945c066f70 0x7f945c0aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41616/0 (socket says 172.21.15.187:41616) 2020-11-20T12:05:51.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945b7fe700 1 -- 172.21.15.187:0/3196961290 learned_addr learned my addr 172.21.15.187:0/3196961290 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:51.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.883+0000 7f945b7fe700 1 -- 172.21.15.187:0/3196961290 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f945c0b6650 msgr2=0x7f945c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:51.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.884+0000 7f945b7fe700 1 --2- 172.21.15.187:0/3196961290 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f945c0b6650 0x7f945c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.884+0000 7f945b7fe700 1 -- 172.21.15.187:0/3196961290 >> v1:172.21.15.5:6789/0 conn(0x7f945c1484d0 legacy=0x7f945c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:51.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.884+0000 7f945b7fe700 1 -- 172.21.15.187:0/3196961290 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0b1840 con 0x7f945c066f70 2020-11-20T12:05:51.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.884+0000 7f945b7fe700 1 --2- 172.21.15.187:0/3196961290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c0aff80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9454009d50 tx=0x7f9454009290).ready entity=mon.1 client_cookie=577eb18e9f7c9f09 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:51.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.884+0000 7f945a7fc700 1 -- 172.21.15.187:0/3196961290 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945400bc40 con 0x7f945c066f70 2020-11-20T12:05:51.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.884+0000 7f945a7fc700 1 -- 172.21.15.187:0/3196961290 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945400bda0 con 0x7f945c066f70 2020-11-20T12:05:51.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.884+0000 7f945a7fc700 1 -- 172.21.15.187:0/3196961290 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9454011610 con 0x7f945c066f70 2020-11-20T12:05:51.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.885+0000 7f945bfff700 1 -- 172.21.15.187:0/3196961290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 msgr2=0x7f945c0aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.885+0000 7f945bfff700 1 --2- 172.21.15.187:0/3196961290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c0aff80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9454009d50 tx=0x7f9454009290).stop 2020-11-20T12:05:51.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.885+0000 7f945bfff700 1 -- 172.21.15.187:0/3196961290 shutdown_connections 2020-11-20T12:05:51.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.885+0000 7f945bfff700 1 --2- 172.21.15.187:0/3196961290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c0aff80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.885+0000 7f945bfff700 1 --2- 172.21.15.187:0/3196961290 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f945c0b6650 0x7f945c0afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.885+0000 7f945bfff700 1 -- 172.21.15.187:0/3196961290 >> 172.21.15.187:0/3196961290 conn(0x7f945c0bab30 msgr2=0x7f945c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:51.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.885+0000 7f945bfff700 1 -- 172.21.15.187:0/3196961290 shutdown_connections 2020-11-20T12:05:51.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.885+0000 7f945bfff700 1 -- 172.21.15.187:0/3196961290 wait complete. 2020-11-20T12:05:51.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.886+0000 7f945bfff700 1 Processor -- start 2020-11-20T12:05:51.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.886+0000 7f945bfff700 1 -- start start 2020-11-20T12:05:51.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c1479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f945c0b6650 0x7f945c138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f945c1484d0 0x7f945c09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0b47f0 con 0x7f945c0b6650 2020-11-20T12:05:51.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c09e670 con 0x7f945c066f70 2020-11-20T12:05:51.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0b44e0 con 0x7f945c1484d0 2020-11-20T12:05:51.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c1479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c1479d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41620/0 (socket says 172.21.15.187:41620) 2020-11-20T12:05:51.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.887+0000 7f945affd700 1 -- 172.21.15.187:0/138058321 learned_addr learned my addr 172.21.15.187:0/138058321 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:51.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f94527fc700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f945c0b6650 0x7f945c138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f945affd700 1 -- 172.21.15.187:0/138058321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f945c1484d0 msgr2=0x7f945c09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f945affd700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f945c1484d0 0x7f945c09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:51.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f945affd700 1 -- 172.21.15.187:0/138058321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f945c0b6650 msgr2=0x7f945c138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:51.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f945affd700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f945c0b6650 0x7f945c138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:51.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f945affd700 1 -- 172.21.15.187:0/138058321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c138850 con 0x7f945c066f70 2020-11-20T12:05:51.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f945affd700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c1479d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f944c0096e0 tx=0x7f944c006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:51.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f94537fe700 1 -- 172.21.15.187:0/138058321 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c00d5c0 con 0x7f945c066f70 2020-11-20T12:05:51.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f94537fe700 1 -- 172.21.15.187:0/138058321 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c000e90 con 0x7f945c066f70 2020-11-20T12:05:51.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c09fb00 con 0x7f945c066f70 2020-11-20T12:05:51.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.888+0000 7f94537fe700 1 -- 172.21.15.187:0/138058321 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c00d990 con 0x7f945c066f70 2020-11-20T12:05:51.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.889+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c09ecd0 con 0x7f945c066f70 2020-11-20T12:05:51.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.890+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c061a20 con 0x7f945c066f70 2020-11-20T12:05:51.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.890+0000 7f94537fe700 1 -- 172.21.15.187:0/138058321 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f944c00dd40 con 0x7f945c066f70 2020-11-20T12:05:51.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.891+0000 7f94537fe700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9438056d20 0x7f94380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:51.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.891+0000 7f94537fe700 1 -- 172.21.15.187:0/138058321 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f944c06f840 con 0x7f945c066f70 2020-11-20T12:05:51.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.891+0000 7f94527fc700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9438056d20 0x7f94380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:51.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.892+0000 7f94527fc700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9438056d20 0x7f94380591e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f945c0b3cb0 tx=0x7f9444008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:51.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:51.895+0000 7f94537fe700 1 -- 172.21.15.187:0/138058321 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f944c044c90 con 0x7f945c066f70 2020-11-20T12:05:52.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.152+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f945c0624e0 con 0x7f945c066f70 2020-11-20T12:05:52.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.181+0000 7f94537fe700 1 -- 172.21.15.187:0/138058321 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v66) v1 ==== 135+0+0 (secure 0 0 0) 0x7f944c0482b0 con 0x7f945c066f70 2020-11-20T12:05:52.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.183+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9438056d20 msgr2=0x7f94380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:52.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.183+0000 7f945bfff700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9438056d20 0x7f94380591e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f945c0b3cb0 tx=0x7f9444008040).stop 2020-11-20T12:05:52.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 msgr2=0x7f945c1479d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:52.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c1479d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f944c0096e0 tx=0x7f944c006d70).stop 2020-11-20T12:05:52.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 shutdown_connections 2020-11-20T12:05:52.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9438056d20 0x7f94380591e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f945c1484d0 0x7f945c09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:52.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f945c066f70 0x7f945c1479d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 --2- 172.21.15.187:0/138058321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f945c0b6650 0x7f945c138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 >> 172.21.15.187:0/138058321 conn(0x7f945c0bab30 msgr2=0x7f945c13cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:52.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 shutdown_connections 2020-11-20T12:05:52.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.184+0000 7f945bfff700 1 -- 172.21.15.187:0/138058321 wait complete. 2020-11-20T12:05:52.212 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:05:52.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.450+0000 7f3421915700 1 Processor -- start 2020-11-20T12:05:52.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.451+0000 7f3421915700 1 -- start start 2020-11-20T12:05:52.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.452+0000 7f3421915700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f341c065300 0x7f341c0656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:52.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.452+0000 7f3421915700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 0x7f341c146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:52.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.452+0000 7f3421915700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c066420 con 0x7f341c09a8a0 2020-11-20T12:05:52.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.452+0000 7f3421915700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341c0b7380 con 0x7f341c065300 2020-11-20T12:05:52.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.452+0000 7f3421915700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341c061f60 con 0x7f341c068030 2020-11-20T12:05:52.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.452+0000 7f341affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f341c065300 0x7f341c0656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:52.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.452+0000 7f34127fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f341c09a8a0 0x7f341c062400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38236/0 (socket says 172.21.15.187:38236) 2020-11-20T12:05:52.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.452+0000 7f34127fc700 1 -- 172.21.15.187:0/913940694 learned_addr learned my addr 172.21.15.187:0/913940694 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:52.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.453+0000 7f341b7fe700 1 --2- 172.21.15.187:0/913940694 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 0x7f341c146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:52.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.453+0000 7f341a7fc700 1 -- 172.21.15.187:0/913940694 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84587530 0 0) 0x7f341c066420 con 0x7f341c09a8a0 2020-11-20T12:05:52.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.453+0000 7f341a7fc700 1 -- 172.21.15.187:0/913940694 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc003240 con 0x7f341c09a8a0 2020-11-20T12:05:52.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.453+0000 7f341affd700 1 -- 172.21.15.187:0/913940694 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 msgr2=0x7f341c146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:52.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.453+0000 7f341affd700 1 --2- 172.21.15.187:0/913940694 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 0x7f341c146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.453+0000 7f341affd700 1 -- 172.21.15.187:0/913940694 >> v1:172.21.15.187:6789/0 conn(0x7f341c09a8a0 legacy=0x7f341c062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:52.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.453+0000 7f341affd700 1 -- 172.21.15.187:0/913940694 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c0b6fd0 con 0x7f341c065300 2020-11-20T12:05:52.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.454+0000 7f341affd700 1 --2- 172.21.15.187:0/913940694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c0656e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f340c00b9a0 tx=0x7f340c006df0).ready entity=mon.1 client_cookie=b1cf866297c4038d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:52.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.454+0000 7f341a7fc700 1 -- 172.21.15.187:0/913940694 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c009840 con 0x7f341c065300 2020-11-20T12:05:52.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.454+0000 7f341a7fc700 1 -- 172.21.15.187:0/913940694 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c0099a0 con 0x7f341c065300 2020-11-20T12:05:52.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.454+0000 7f341a7fc700 1 -- 172.21.15.187:0/913940694 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c009d20 con 0x7f341c065300 2020-11-20T12:05:52.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.454+0000 7f3421915700 1 -- 172.21.15.187:0/913940694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 msgr2=0x7f341c0656e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:52.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.454+0000 7f3421915700 1 --2- 172.21.15.187:0/913940694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c0656e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f340c00b9a0 tx=0x7f340c006df0).stop 2020-11-20T12:05:52.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.455+0000 7f3421915700 1 -- 172.21.15.187:0/913940694 shutdown_connections 2020-11-20T12:05:52.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.455+0000 7f3421915700 1 --2- 172.21.15.187:0/913940694 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 0x7f341c146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.455+0000 7f3421915700 1 --2- 172.21.15.187:0/913940694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c0656e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.455+0000 7f3421915700 1 -- 172.21.15.187:0/913940694 >> 172.21.15.187:0/913940694 conn(0x7f341c0bacf0 msgr2=0x7f341c0bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:52.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.455+0000 7f3421915700 1 -- 172.21.15.187:0/913940694 shutdown_connections 2020-11-20T12:05:52.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.455+0000 7f3421915700 1 -- 172.21.15.187:0/913940694 wait complete. 2020-11-20T12:05:52.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.456+0000 7f3421915700 1 Processor -- start 2020-11-20T12:05:52.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.456+0000 7f3421915700 1 -- start start 2020-11-20T12:05:52.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.457+0000 7f3421915700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:52.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.457+0000 7f3421915700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 0x7f341c0b8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:52.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.457+0000 7f3421915700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f341c09a8a0 0x7f341c158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:52.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.457+0000 7f3421915700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c15c620 con 0x7f341c09a8a0 2020-11-20T12:05:52.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.457+0000 7f3421915700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c15c0e0 con 0x7f341c065300 2020-11-20T12:05:52.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.457+0000 7f3421915700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341c15c380 con 0x7f341c068030 2020-11-20T12:05:52.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.457+0000 7f341affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:52.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f341affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c149cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41634/0 (socket says 172.21.15.187:41634) 2020-11-20T12:05:52.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f341affd700 1 -- 172.21.15.187:0/4248898975 learned_addr learned my addr 172.21.15.187:0/4248898975 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:52.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f341affd700 1 -- 172.21.15.187:0/4248898975 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 msgr2=0x7f341c0b8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:52.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f34127fc700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 0x7f341c0b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:52.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f341affd700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 0x7f341c0b8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f341affd700 1 -- 172.21.15.187:0/4248898975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f341c09a8a0 msgr2=0x7f341c158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:52.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f341affd700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f341c09a8a0 0x7f341c158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:52.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f341affd700 1 -- 172.21.15.187:0/4248898975 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c158280 con 0x7f341c065300 2020-11-20T12:05:52.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.458+0000 7f341affd700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c149cf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f340c006e30 tx=0x7f340c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:52.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.459+0000 7f34137fe700 1 -- 172.21.15.187:0/4248898975 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c01e460 con 0x7f341c065300 2020-11-20T12:05:52.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.459+0000 7f34137fe700 1 -- 172.21.15.187:0/4248898975 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c01e5c0 con 0x7f341c065300 2020-11-20T12:05:52.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.459+0000 7f34137fe700 1 -- 172.21.15.187:0/4248898975 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c01e940 con 0x7f341c065300 2020-11-20T12:05:52.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.459+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c14a1e0 con 0x7f341c065300 2020-11-20T12:05:52.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.459+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c0b8d40 con 0x7f341c065300 2020-11-20T12:05:52.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.460+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c158410 con 0x7f341c065300 2020-11-20T12:05:52.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.460+0000 7f34137fe700 1 -- 172.21.15.187:0/4248898975 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f340c025030 con 0x7f341c065300 2020-11-20T12:05:52.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.461+0000 7f34137fe700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33f8056d50 0x7f33f8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:52.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.461+0000 7f34137fe700 1 -- 172.21.15.187:0/4248898975 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f340c071930 con 0x7f341c065300 2020-11-20T12:05:52.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.461+0000 7f34127fc700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33f8056d50 0x7f33f8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:52.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.462+0000 7f34127fc700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33f8056d50 0x7f33f8059210 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f34040055a0 tx=0x7f3404009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:52.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.465+0000 7f34137fe700 1 -- 172.21.15.187:0/4248898975 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f340c046d20 con 0x7f341c065300 2020-11-20T12:05:52.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.651+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f341c044f60 con 0x7f341c065300 2020-11-20T12:05:52.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.663+0000 7f34137fe700 1 -- 172.21.15.187:0/4248898975 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f340c023030 con 0x7f341c065300 2020-11-20T12:05:52.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33f8056d50 msgr2=0x7f33f8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:52.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33f8056d50 0x7f33f8059210 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f34040055a0 tx=0x7f3404009040).stop 2020-11-20T12:05:52.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 msgr2=0x7f341c149cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:52.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c149cf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f340c006e30 tx=0x7f340c015000).stop 2020-11-20T12:05:52.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 shutdown_connections 2020-11-20T12:05:52.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33f8056d50 0x7f33f8059210 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f341c09a8a0 0x7f341c158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:52.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c065300 0x7f341c149cf0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 --2- 172.21.15.187:0/4248898975 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f341c068030 0x7f341c0b8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:52.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 >> 172.21.15.187:0/4248898975 conn(0x7f341c0bacf0 msgr2=0x7f341c0afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:52.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 shutdown_connections 2020-11-20T12:05:52.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:52.666+0000 7f3421915700 1 -- 172.21.15.187:0/4248898975 wait complete. 2020-11-20T12:05:52.691 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:05:52.692 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:05:52.809 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:05:52.823 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:05:52.934 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:05:52.953 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:05:52.953 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:05:52.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:52.954 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2020-11-20T12:05:53.093 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:05:53.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:53.094 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:05:53.151 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2020-11-20T12:05:53.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:53.152 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2020-11-20T12:05:53.273 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:05:53.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:53.274 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2020-11-20T12:05:53.324 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2020-11-20T12:05:53.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:53.325 INFO:teuthology.orchestra.run.smithi005:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2020-11-20T12:05:53.478 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T12:05:53.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:53.479 INFO:teuthology.orchestra.run.smithi005:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2020-11-20T12:05:53.545 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2020-11-20T12:05:53.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:53.546 INFO:teuthology.orchestra.run.smithi005:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2020-11-20T12:05:53.667 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi005.front.sepia.ceph.com 2020-11-20T12:05:53.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:53.668 INFO:teuthology.orchestra.run.smithi005:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2020-11-20T12:05:53.725 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:05:53.726 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:53.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.975+0000 7f1ce3fff700 1 Processor -- start 2020-11-20T12:05:53.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.975+0000 7f1ce3fff700 1 -- start start 2020-11-20T12:05:53.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.977+0000 7f1ce3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce409eee0 0x7f1ce4067410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:53.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.977+0000 7f1ce3fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1ce409f2c0 0x7f1ce4146d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:53.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.977+0000 7f1ce3fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce41440a0 con 0x7f1ce40ba9b0 2020-11-20T12:05:53.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.977+0000 7f1ce3fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce4146690 con 0x7f1ce409f2c0 2020-11-20T12:05:53.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.977+0000 7f1ce3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce409c750 con 0x7f1ce409eee0 2020-11-20T12:05:53.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.977+0000 7f1cda7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1ce40ba9b0 0x7f1ce4063f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38248/0 (socket says 172.21.15.187:38248) 2020-11-20T12:05:53.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.977+0000 7f1cda7fc700 1 -- 172.21.15.187:0/1603116285 learned_addr learned my addr 172.21.15.187:0/1603116285 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:53.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.978+0000 7f1ce2ffd700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce409eee0 0x7f1ce4067410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:53.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.978+0000 7f1ce37fe700 1 --2- 172.21.15.187:0/1603116285 >> v2:172.21.15.5:3300/0 conn(0x7f1ce409f2c0 0x7f1ce4146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:53.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.978+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861443994 0 0) 0x7f1ce41440a0 con 0x7f1ce40ba9b0 2020-11-20T12:05:53.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.978+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc4003240 con 0x7f1ce40ba9b0 2020-11-20T12:05:53.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.978+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2660709340 0 0) 0x7f1cc4003240 con 0x7f1ce40ba9b0 2020-11-20T12:05:53.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.978+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce409eee0 msgr2=0x7f1ce4067410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:53.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.978+0000 7f1ce27fc700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce409eee0 0x7f1ce4067410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:53.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 >> v2:172.21.15.5:3300/0 conn(0x7f1ce409f2c0 msgr2=0x7f1ce4146d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:53.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 --2- 172.21.15.187:0/1603116285 >> v2:172.21.15.5:3300/0 conn(0x7f1ce409f2c0 0x7f1ce4146d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:53.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce40b9440 con 0x7f1ce40ba9b0 2020-11-20T12:05:53.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1ccc0014e0 con 0x7f1ce40ba9b0 2020-11-20T12:05:53.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 >> v1:172.21.15.187:6789/0 conn(0x7f1ce40ba9b0 legacy=0x7f1ce4063f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:53.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cc40037b0 0x7f1cc4003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:53.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cc40044e0 0x7f1cc4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:53.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cc4004e50 0x7f1cc400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:53.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce409c750 con 0x7f1cc40044e0 2020-11-20T12:05:53.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40052d0 con 0x7f1cc4004e50 2020-11-20T12:05:53.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc400bc20 con 0x7f1cc40037b0 2020-11-20T12:05:53.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.979+0000 7f1ce37fe700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cc4004e50 0x7f1cc400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:53.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1cda7fc700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cc40044e0 0x7f1cc4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:53.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce2ffd700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cc40037b0 0x7f1cc4003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:53.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce37fe700 1 -- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cc40037b0 msgr2=0x7f1cc4003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:53.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce37fe700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cc40037b0 0x7f1cc4003bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:53.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce37fe700 1 -- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cc40044e0 msgr2=0x7f1cc4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:53.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce37fe700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cc40044e0 0x7f1cc4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:53.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce37fe700 1 -- 172.21.15.187:0/1603116285 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc4003240 con 0x7f1cc4004e50 2020-11-20T12:05:53.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce37fe700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cc4004e50 0x7f1cc400b730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f1cdc009ea0 tx=0x7f1cdc0091d0).ready entity=mon.1 client_cookie=25c4d02f216ce4a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:53.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc0095c0 con 0x7f1cc4004e50 2020-11-20T12:05:53.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc00ed40 con 0x7f1cc4004e50 2020-11-20T12:05:53.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.980+0000 7f1ce27fc700 1 -- 172.21.15.187:0/1603116285 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc012600 con 0x7f1cc4004e50 2020-11-20T12:05:53.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.981+0000 7f1ce3fff700 1 -- 172.21.15.187:0/1603116285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cc4004e50 msgr2=0x7f1cc400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:53.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.981+0000 7f1ce3fff700 1 --2- 172.21.15.187:0/1603116285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cc4004e50 0x7f1cc400b730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f1cdc009ea0 tx=0x7f1cdc0091d0).stop 2020-11-20T12:05:53.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.981+0000 7f1ce37fe700 1 -- 172.21.15.187:0/1603116285 reap_dead start 2020-11-20T12:05:53.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.981+0000 7f1ce3fff700 1 -- 172.21.15.187:0/1603116285 shutdown_connections 2020-11-20T12:05:53.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.981+0000 7f1ce3fff700 1 -- 172.21.15.187:0/1603116285 >> 172.21.15.187:0/1603116285 conn(0x7f1ce40bce50 msgr2=0x7f1ce40bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:53.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.981+0000 7f1ce3fff700 1 -- 172.21.15.187:0/1603116285 shutdown_connections 2020-11-20T12:05:53.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.981+0000 7f1ce3fff700 1 -- 172.21.15.187:0/1603116285 wait complete. 2020-11-20T12:05:53.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.983+0000 7f1ce3fff700 1 Processor -- start 2020-11-20T12:05:53.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.983+0000 7f1ce3fff700 1 -- start start 2020-11-20T12:05:54.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1ce3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce40b64f0 0x7f1ce415bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1ce3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ce409fe00 0x7f1ce40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1ce3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ce41587b0 0x7f1ce4158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1ce3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce415c3d0 con 0x7f1ce41587b0 2020-11-20T12:05:54.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1ce3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce4158010 con 0x7f1ce409fe00 2020-11-20T12:05:54.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1ce3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce40b4ee0 con 0x7f1ce40b64f0 2020-11-20T12:05:54.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1cda7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ce409fe00 0x7f1ce40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1cda7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ce409fe00 0x7f1ce40b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41656/0 (socket says 172.21.15.187:41656) 2020-11-20T12:05:54.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1cda7fc700 1 -- 172.21.15.187:0/3194331454 learned_addr learned my addr 172.21.15.187:0/3194331454 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:54.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1ce2ffd700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce40b64f0 0x7f1ce415bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1cda7fc700 1 -- 172.21.15.187:0/3194331454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce40b64f0 msgr2=0x7f1ce415bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1cda7fc700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce40b64f0 0x7f1ce415bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1cda7fc700 1 -- 172.21.15.187:0/3194331454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ce41587b0 msgr2=0x7f1ce4158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1cda7fc700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ce41587b0 0x7f1ce4158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:54.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1cda7fc700 1 -- 172.21.15.187:0/3194331454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce40b50a0 con 0x7f1ce409fe00 2020-11-20T12:05:54.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.984+0000 7f1ce2ffd700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce40b64f0 0x7f1ce415bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:54.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.985+0000 7f1cda7fc700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ce409fe00 0x7f1ce40b68d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1ccc0011c0 tx=0x7f1ce40b95d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:54.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.985+0000 7f1cdb7fe700 1 -- 172.21.15.187:0/3194331454 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00a8c0 con 0x7f1ce409fe00 2020-11-20T12:05:54.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.985+0000 7f1cdb7fe700 1 -- 172.21.15.187:0/3194331454 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc010040 con 0x7f1ce409fe00 2020-11-20T12:05:54.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.985+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce4151c50 con 0x7f1ce409fe00 2020-11-20T12:05:54.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.985+0000 7f1cdb7fe700 1 -- 172.21.15.187:0/3194331454 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00ac30 con 0x7f1ce409fe00 2020-11-20T12:05:54.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.985+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce4152110 con 0x7f1ce409fe00 2020-11-20T12:05:54.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.986+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce409c0a0 con 0x7f1ce409fe00 2020-11-20T12:05:54.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.987+0000 7f1cdb7fe700 1 -- 172.21.15.187:0/3194331454 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1ccc01b020 con 0x7f1ce409fe00 2020-11-20T12:05:54.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.987+0000 7f1cdb7fe700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cc0056d20 0x7f1cc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.987+0000 7f1cdb7fe700 1 -- 172.21.15.187:0/3194331454 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ccc06f920 con 0x7f1ce409fe00 2020-11-20T12:05:54.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.988+0000 7f1ce2ffd700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cc0056d20 0x7f1cc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.988+0000 7f1ce2ffd700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cc0056d20 0x7f1cc00591e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1cd4006bd0 tx=0x7f1cd4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:54.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:53.992+0000 7f1cdb7fe700 1 -- 172.21.15.187:0/3194331454 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1ccc044c60 con 0x7f1ce409fe00 2020-11-20T12:05:54.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.239+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ce415c540 con 0x7f1ce409fe00 2020-11-20T12:05:54.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.240+0000 7f1cdb7fe700 1 -- 172.21.15.187:0/3194331454 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2130 (secure 0 0 0) 0x7f1ccc048280 con 0x7f1ce409fe00 2020-11-20T12:05:54.244 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:54.244 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4211,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6833/2094645874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2094645874},{"type":"v1","addr":"172.21.15.5:6833","nonce":2094645874}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:57.868606+0000","modified":"2020-11-20T12:04:59.990344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3094884769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3094884769},{"type":"v1","addr":"172.21.15.187:6835","nonce":3094884769}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:54.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cc0056d20 msgr2=0x7f1cc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cc0056d20 0x7f1cc00591e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1cd4006bd0 tx=0x7f1cd4008040).stop 2020-11-20T12:05:54.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ce409fe00 msgr2=0x7f1ce40b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ce409fe00 0x7f1ce40b68d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1ccc0011c0 tx=0x7f1ce40b95d0).stop 2020-11-20T12:05:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 shutdown_connections 2020-11-20T12:05:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cc0056d20 0x7f1cc00591e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ce41587b0 0x7f1ce4158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ce40b64f0 0x7f1ce415bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 --2- 172.21.15.187:0/3194331454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ce409fe00 0x7f1ce40b68d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 >> 172.21.15.187:0/3194331454 conn(0x7f1ce40bce50 msgr2=0x7f1ce4067410 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:54.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 shutdown_connections 2020-11-20T12:05:54.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.243+0000 7f1ce3fff700 1 -- 172.21.15.187:0/3194331454 wait complete. 2020-11-20T12:05:54.249 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 6 2020-11-20T12:05:54.271 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:54.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.509+0000 7fed2684a700 1 Processor -- start 2020-11-20T12:05:54.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.509+0000 7fed2684a700 1 -- start start 2020-11-20T12:05:54.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.510+0000 7fed2684a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed200b57b0 0x7fed20068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.510+0000 7fed2684a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed20144b80 con 0x7fed200b5f50 2020-11-20T12:05:54.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.510+0000 7fed2684a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed201489e0 con 0x7fed201482d0 2020-11-20T12:05:54.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.510+0000 7fed2684a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed20148c80 con 0x7fed200b57b0 2020-11-20T12:05:54.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.511+0000 7fed25047700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fed201482d0 0x7fed20062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38270/0 (socket says 172.21.15.187:38270) 2020-11-20T12:05:54.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.511+0000 7fed25047700 1 -- 172.21.15.187:0/857905835 learned_addr learned my addr 172.21.15.187:0/857905835 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:54.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.511+0000 7fed25848700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed200b57b0 0x7fed20068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.511+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307220950 0 0) 0x7fed201489e0 con 0x7fed201482d0 2020-11-20T12:05:54.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.511+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed0c003240 con 0x7fed201482d0 2020-11-20T12:05:54.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1127205162 0 0) 0x7fed0c003240 con 0x7fed201482d0 2020-11-20T12:05:54.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed200b57b0 msgr2=0x7fed20068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:54.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed200b57b0 0x7fed20068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 >> v1:172.21.15.5:6789/0 conn(0x7fed200b5f50 legacy=0x7fed200b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20148f20 con 0x7fed201482d0 2020-11-20T12:05:54.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fed14001860 con 0x7fed201482d0 2020-11-20T12:05:54.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 >> v1:172.21.15.187:6789/0 conn(0x7fed201482d0 legacy=0x7fed20062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0c003730 0x7fed0c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed200b57b0 0x7fed0c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c0049d0 0x7fed0c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed20148c80 con 0x7fed0c003730 2020-11-20T12:05:54.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c004220 con 0x7fed0c0049d0 2020-11-20T12:05:54.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.512+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c00b730 con 0x7fed200b57b0 2020-11-20T12:05:54.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.513+0000 7fed26049700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c0049d0 0x7fed0c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.513+0000 7fed25848700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0c003730 0x7fed0c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.513+0000 7fed25047700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed200b57b0 0x7fed0c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.513+0000 7fed26049700 1 -- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed200b57b0 msgr2=0x7fed0c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.513+0000 7fed26049700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed200b57b0 0x7fed0c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.513+0000 7fed26049700 1 -- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0c003730 msgr2=0x7fed0c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.513+0000 7fed26049700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed0c003730 0x7fed0c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.513+0000 7fed26049700 1 -- 172.21.15.187:0/857905835 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c003240 con 0x7fed0c0049d0 2020-11-20T12:05:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.514+0000 7fed26049700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c0049d0 0x7fed0c004e50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fed10007e80 tx=0x7fed10007b90).ready entity=mon.1 client_cookie=86abb2feca3c4ebf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.514+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10015070 con 0x7fed0c0049d0 2020-11-20T12:05:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.514+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed10007020 con 0x7fed0c0049d0 2020-11-20T12:05:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.514+0000 7fed24846700 1 -- 172.21.15.187:0/857905835 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100073c0 con 0x7fed0c0049d0 2020-11-20T12:05:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.515+0000 7fed2684a700 1 -- 172.21.15.187:0/857905835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c0049d0 msgr2=0x7fed0c004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.515+0000 7fed2684a700 1 --2- 172.21.15.187:0/857905835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c0049d0 0x7fed0c004e50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fed10007e80 tx=0x7fed10007b90).stop 2020-11-20T12:05:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.515+0000 7fed26049700 1 -- 172.21.15.187:0/857905835 reap_dead start 2020-11-20T12:05:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.515+0000 7fed2684a700 1 -- 172.21.15.187:0/857905835 shutdown_connections 2020-11-20T12:05:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.515+0000 7fed2684a700 1 -- 172.21.15.187:0/857905835 >> 172.21.15.187:0/857905835 conn(0x7fed200bab30 msgr2=0x7fed200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.515+0000 7fed2684a700 1 -- 172.21.15.187:0/857905835 shutdown_connections 2020-11-20T12:05:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.515+0000 7fed2684a700 1 -- 172.21.15.187:0/857905835 wait complete. 2020-11-20T12:05:54.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.516+0000 7fed2684a700 1 Processor -- start 2020-11-20T12:05:54.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.516+0000 7fed2684a700 1 -- start start 2020-11-20T12:05:54.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.517+0000 7fed2684a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed2015b670 0x7fed20145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.517+0000 7fed2684a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed2015ba50 0x7fed200a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.517+0000 7fed2684a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed200a0560 0x7fed20157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.517+0000 7fed2684a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed200a09a0 con 0x7fed2015b670 2020-11-20T12:05:54.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.517+0000 7fed2684a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed201479b0 con 0x7fed200a0560 2020-11-20T12:05:54.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.517+0000 7fed2684a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed200b2df0 con 0x7fed2015ba50 2020-11-20T12:05:54.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed200a0560 0x7fed20157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed200a0560 0x7fed20157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41676/0 (socket says 172.21.15.187:41676) 2020-11-20T12:05:54.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 -- 172.21.15.187:0/3169958274 learned_addr learned my addr 172.21.15.187:0/3169958274 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:54.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 -- 172.21.15.187:0/3169958274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed2015ba50 msgr2=0x7fed200a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed2015ba50 0x7fed200a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:54.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 -- 172.21.15.187:0/3169958274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed2015b670 msgr2=0x7fed20145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed2015b670 0x7fed20145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:54.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 -- 172.21.15.187:0/3169958274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed200b3290 con 0x7fed200a0560 2020-11-20T12:05:54.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.518+0000 7fed26049700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed200a0560 0x7fed20157ea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fed0c0043b0 tx=0x7fed10017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:54.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.519+0000 7fed1dffb700 1 -- 172.21.15.187:0/3169958274 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000cea0 con 0x7fed200a0560 2020-11-20T12:05:54.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.519+0000 7fed1dffb700 1 -- 172.21.15.187:0/3169958274 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed10000e00 con 0x7fed200a0560 2020-11-20T12:05:54.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.519+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2015c210 con 0x7fed200a0560 2020-11-20T12:05:54.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.519+0000 7fed1dffb700 1 -- 172.21.15.187:0/3169958274 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100116a0 con 0x7fed200a0560 2020-11-20T12:05:54.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.519+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed20158390 con 0x7fed200a0560 2020-11-20T12:05:54.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.520+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed200b2fb0 con 0x7fed200a0560 2020-11-20T12:05:54.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.521+0000 7fed1dffb700 1 -- 172.21.15.187:0/3169958274 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fed10011a10 con 0x7fed200a0560 2020-11-20T12:05:54.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.521+0000 7fed1dffb700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecf8056d50 0x7fecf8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:54.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.521+0000 7fed1dffb700 1 -- 172.21.15.187:0/3169958274 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fed10071f00 con 0x7fed200a0560 2020-11-20T12:05:54.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.522+0000 7fed25848700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecf8056d50 0x7fecf8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:54.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.522+0000 7fed25848700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecf8056d50 0x7fecf8059210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fed18006fd0 tx=0x7fed18008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:54.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.525+0000 7fed1dffb700 1 -- 172.21.15.187:0/3169958274 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fed10047240 con 0x7fed200a0560 2020-11-20T12:05:54.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.711+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed20061a20 con 0x7fed200a0560 2020-11-20T12:05:54.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.713+0000 7fed1dffb700 1 -- 172.21.15.187:0/3169958274 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7fed1004a860 con 0x7fed200a0560 2020-11-20T12:05:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:54.719 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:54.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.715+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecf8056d50 msgr2=0x7fecf8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.715+0000 7fed2684a700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecf8056d50 0x7fecf8059210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fed18006fd0 tx=0x7fed18008040).stop 2020-11-20T12:05:54.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.715+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed200a0560 msgr2=0x7fed20157ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:54.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.715+0000 7fed2684a700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed200a0560 0x7fed20157ea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fed0c0043b0 tx=0x7fed10017000).stop 2020-11-20T12:05:54.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.716+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 shutdown_connections 2020-11-20T12:05:54.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.716+0000 7fed2684a700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecf8056d50 0x7fecf8059210 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.716+0000 7fed2684a700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed2015b670 0x7fed20145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:54.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.716+0000 7fed2684a700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed200a0560 0x7fed20157ea0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:54.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.716+0000 7fed2684a700 1 --2- 172.21.15.187:0/3169958274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed2015ba50 0x7fed200a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:54.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.716+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 >> 172.21.15.187:0/3169958274 conn(0x7fed200bab30 msgr2=0x7fed20065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:54.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.716+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 shutdown_connections 2020-11-20T12:05:54.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.716+0000 7fed2684a700 1 -- 172.21.15.187:0/3169958274 wait complete. 2020-11-20T12:05:54.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:55.702880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:56.876271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:54.743 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:55.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.999+0000 7f90310d1700 1 Processor -- start 2020-11-20T12:05:55.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:54.999+0000 7f90310d1700 1 -- start start 2020-11-20T12:05:55.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.000+0000 7f90310d1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f902c0b57b0 0x7f902c068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.000+0000 7f90310d1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f902c0b5f50 0x7f902c0b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.000+0000 7f90310d1700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c144b80 con 0x7f902c1482d0 2020-11-20T12:05:55.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.000+0000 7f90310d1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c1489e0 con 0x7f902c0b5f50 2020-11-20T12:05:55.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.000+0000 7f90310d1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c148c80 con 0x7f902c0b57b0 2020-11-20T12:05:55.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.001+0000 7f902bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f902c0b57b0 0x7f902c068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.001+0000 7f90237fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f902c1482d0 0x7f902c062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38292/0 (socket says 172.21.15.187:38292) 2020-11-20T12:05:55.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.001+0000 7f90237fe700 1 -- 172.21.15.187:0/2674245168 learned_addr learned my addr 172.21.15.187:0/2674245168 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:55.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.001+0000 7f902bfff700 1 -- 172.21.15.187:0/2674245168 >> v2:172.21.15.5:3300/0 conn(0x7f902c0b5f50 msgr2=0x7f902c0b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:55.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.001+0000 7f902bfff700 1 --2- 172.21.15.187:0/2674245168 >> v2:172.21.15.5:3300/0 conn(0x7f902c0b5f50 0x7f902c0b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:55.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.001+0000 7f902bfff700 1 -- 172.21.15.187:0/2674245168 >> v1:172.21.15.187:6789/0 conn(0x7f902c1482d0 legacy=0x7f902c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.001+0000 7f902b7fe700 1 -- 172.21.15.187:0/2674245168 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881487006 0 0) 0x7f902c144b80 con 0x7f902c1482d0 2020-11-20T12:05:55.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.001+0000 7f902bfff700 1 -- 172.21.15.187:0/2674245168 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c148f20 con 0x7f902c0b57b0 2020-11-20T12:05:55.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.002+0000 7f902bfff700 1 --2- 172.21.15.187:0/2674245168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b57b0 0x7f902c068230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f902c0b85e0 tx=0x7f901c006df0).ready entity=mon.1 client_cookie=123eb8345e64710b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.002+0000 7f902b7fe700 1 -- 172.21.15.187:0/2674245168 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c0097b0 con 0x7f902c0b57b0 2020-11-20T12:05:55.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.002+0000 7f902b7fe700 1 -- 172.21.15.187:0/2674245168 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c009910 con 0x7f902c0b57b0 2020-11-20T12:05:55.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.002+0000 7f902b7fe700 1 -- 172.21.15.187:0/2674245168 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c009c90 con 0x7f902c0b57b0 2020-11-20T12:05:55.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.003+0000 7f90310d1700 1 -- 172.21.15.187:0/2674245168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b57b0 msgr2=0x7f902c068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.003+0000 7f90310d1700 1 --2- 172.21.15.187:0/2674245168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b57b0 0x7f902c068230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f902c0b85e0 tx=0x7f901c006df0).stop 2020-11-20T12:05:55.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.003+0000 7f90310d1700 1 -- 172.21.15.187:0/2674245168 shutdown_connections 2020-11-20T12:05:55.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.003+0000 7f90310d1700 1 --2- 172.21.15.187:0/2674245168 >> v2:172.21.15.5:3300/0 conn(0x7f902c0b5f50 0x7f902c0b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:55.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.003+0000 7f90310d1700 1 --2- 172.21.15.187:0/2674245168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b57b0 0x7f902c068230 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.003+0000 7f90310d1700 1 -- 172.21.15.187:0/2674245168 >> 172.21.15.187:0/2674245168 conn(0x7f902c0bab30 msgr2=0x7f902c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:55.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.003+0000 7f90310d1700 1 -- 172.21.15.187:0/2674245168 shutdown_connections 2020-11-20T12:05:55.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.003+0000 7f90310d1700 1 -- 172.21.15.187:0/2674245168 wait complete. 2020-11-20T12:05:55.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.004+0000 7f90310d1700 1 Processor -- start 2020-11-20T12:05:55.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.004+0000 7f90310d1700 1 -- start start 2020-11-20T12:05:55.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.005+0000 7f90310d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902c0b57b0 0x7f902c0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.005+0000 7f90310d1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b5f50 0x7f902c1433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.005+0000 7f90310d1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c1482d0 0x7f902c0b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.005+0000 7f90310d1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c09e5d0 con 0x7f902c1482d0 2020-11-20T12:05:55.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.005+0000 7f90310d1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0b4a70 con 0x7f902c0b5f50 2020-11-20T12:05:55.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.005+0000 7f90310d1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0b4530 con 0x7f902c0b57b0 2020-11-20T12:05:55.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b5f50 0x7f902c1433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b5f50 0x7f902c1433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41690/0 (socket says 172.21.15.187:41690) 2020-11-20T12:05:55.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 -- 172.21.15.187:0/3282073986 learned_addr learned my addr 172.21.15.187:0/3282073986 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:55.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 -- 172.21.15.187:0/3282073986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902c0b57b0 msgr2=0x7f902c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902c0b57b0 0x7f902c0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:55.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 -- 172.21.15.187:0/3282073986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c1482d0 msgr2=0x7f902c0b24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90308d0700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c1482d0 0x7f902c0b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c1482d0 0x7f902c0b24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 -- 172.21.15.187:0/3282073986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c0b47d0 con 0x7f902c0b5f50 2020-11-20T12:05:55.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.006+0000 7f90237fe700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b5f50 0x7f902c1433e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f901400c790 tx=0x7f9014007c90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.007+0000 7f9028ff9700 1 -- 172.21.15.187:0/3282073986 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90140072e0 con 0x7f902c0b5f50 2020-11-20T12:05:55.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.007+0000 7f9028ff9700 1 -- 172.21.15.187:0/3282073986 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014007440 con 0x7f902c0b5f50 2020-11-20T12:05:55.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.007+0000 7f9028ff9700 1 -- 172.21.15.187:0/3282073986 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014005840 con 0x7f902c0b5f50 2020-11-20T12:05:55.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.007+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c0b20a0 con 0x7f902c0b5f50 2020-11-20T12:05:55.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.007+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c147720 con 0x7f902c0b5f50 2020-11-20T12:05:55.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.008+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c061a20 con 0x7f902c0b5f50 2020-11-20T12:05:55.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.009+0000 7f9028ff9700 1 -- 172.21.15.187:0/3282073986 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9014010070 con 0x7f902c0b5f50 2020-11-20T12:05:55.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.009+0000 7f9028ff9700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9008056d20 0x7f90080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.009+0000 7f9028ff9700 1 -- 172.21.15.187:0/3282073986 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f901406f640 con 0x7f902c0b5f50 2020-11-20T12:05:55.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.010+0000 7f902bfff700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9008056d20 0x7f90080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.010+0000 7f902bfff700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9008056d20 0x7f90080591e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f902c144140 tx=0x7f901c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.013+0000 7f9028ff9700 1 -- 172.21.15.187:0/3282073986 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9014044a90 con 0x7f902c0b5f50 2020-11-20T12:05:55.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.200+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f902c15b930 con 0x7f902c0b5f50 2020-11-20T12:05:55.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.201+0000 7f9028ff9700 1 -- 172.21.15.187:0/3282073986 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f90140480b0 con 0x7f902c0b5f50 2020-11-20T12:05:55.206 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:05:55.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.204+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9008056d20 msgr2=0x7f90080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.204+0000 7f90310d1700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9008056d20 0x7f90080591e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f902c144140 tx=0x7f901c015000).stop 2020-11-20T12:05:55.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.204+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b5f50 msgr2=0x7f902c1433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.204+0000 7f90310d1700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b5f50 0x7f902c1433e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f901400c790 tx=0x7f9014007c90).stop 2020-11-20T12:05:55.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.205+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 shutdown_connections 2020-11-20T12:05:55.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.205+0000 7f90310d1700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9008056d20 0x7f90080591e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.205+0000 7f90310d1700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c1482d0 0x7f902c0b24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.205+0000 7f90310d1700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f902c0b57b0 0x7f902c0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:55.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.205+0000 7f90310d1700 1 --2- 172.21.15.187:0/3282073986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902c0b5f50 0x7f902c1433e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.205+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 >> 172.21.15.187:0/3282073986 conn(0x7f902c0bab30 msgr2=0x7f902c0b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:55.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.205+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 shutdown_connections 2020-11-20T12:05:55.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.205+0000 7f90310d1700 1 -- 172.21.15.187:0/3282073986 wait complete. 2020-11-20T12:05:55.230 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:55.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.477+0000 7f09d533e700 1 Processor -- start 2020-11-20T12:05:55.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.477+0000 7f09d533e700 1 -- start start 2020-11-20T12:05:55.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.479+0000 7f09d533e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d00b1760 0x7f09d0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.479+0000 7f09d533e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f09d0065680 0x7f09d014c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.479+0000 7f09d533e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d0150390 con 0x7f09d00b1380 2020-11-20T12:05:55.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.479+0000 7f09d533e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d01511e0 con 0x7f09d0065680 2020-11-20T12:05:55.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.479+0000 7f09d533e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d014d760 con 0x7f09d00b1760 2020-11-20T12:05:55.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.480+0000 7f09d4b3d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f09d0065680 0x7f09d014c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.480+0000 7f09d4b3d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f09d0065680 0x7f09d014c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41698/0 (socket says 172.21.15.187:41698) 2020-11-20T12:05:55.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.480+0000 7f09d4b3d700 1 -- 172.21.15.187:0/54288532 learned_addr learned my addr 172.21.15.187:0/54288532 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:55.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.480+0000 7f09c77fe700 1 --2- 172.21.15.187:0/54288532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d00b1760 0x7f09d0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.481+0000 7f09d4b3d700 1 -- 172.21.15.187:0/54288532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d00b1760 msgr2=0x7f09d0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:55.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.481+0000 7f09d4b3d700 1 --2- 172.21.15.187:0/54288532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d00b1760 0x7f09d0065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.481+0000 7f09d4b3d700 1 -- 172.21.15.187:0/54288532 >> v1:172.21.15.5:6789/0 conn(0x7f09d00b1380 legacy=0x7f09d0150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.481+0000 7f09d4b3d700 1 -- 172.21.15.187:0/54288532 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d0150630 con 0x7f09d0065680 2020-11-20T12:05:55.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.481+0000 7f09d4b3d700 1 --2- 172.21.15.187:0/54288532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d0065680 0x7f09d014c260 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f09c8009d50 tx=0x7f09c8009290).ready entity=mon.1 client_cookie=f359fe286515fcf6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.481+0000 7f09cf7fe700 1 -- 172.21.15.187:0/54288532 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c800bbc0 con 0x7f09d0065680 2020-11-20T12:05:55.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.481+0000 7f09cf7fe700 1 -- 172.21.15.187:0/54288532 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c800bd20 con 0x7f09d0065680 2020-11-20T12:05:55.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09cf7fe700 1 -- 172.21.15.187:0/54288532 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8011610 con 0x7f09d0065680 2020-11-20T12:05:55.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09d533e700 1 -- 172.21.15.187:0/54288532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d0065680 msgr2=0x7f09d014c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09d533e700 1 --2- 172.21.15.187:0/54288532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d0065680 0x7f09d014c260 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f09c8009d50 tx=0x7f09c8009290).stop 2020-11-20T12:05:55.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09d533e700 1 -- 172.21.15.187:0/54288532 shutdown_connections 2020-11-20T12:05:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09d533e700 1 --2- 172.21.15.187:0/54288532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d0065680 0x7f09d014c260 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09d533e700 1 --2- 172.21.15.187:0/54288532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d00b1760 0x7f09d0065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09d533e700 1 -- 172.21.15.187:0/54288532 >> 172.21.15.187:0/54288532 conn(0x7f09d00bab30 msgr2=0x7f09d00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09d533e700 1 -- 172.21.15.187:0/54288532 shutdown_connections 2020-11-20T12:05:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.482+0000 7f09d533e700 1 -- 172.21.15.187:0/54288532 wait complete. 2020-11-20T12:05:55.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.484+0000 7f09d533e700 1 Processor -- start 2020-11-20T12:05:55.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.484+0000 7f09d533e700 1 -- start start 2020-11-20T12:05:55.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.485+0000 7f09d533e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09d00b1380 0x7f09d015c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.485+0000 7f09d533e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d00b1760 0x7f09d009cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.485+0000 7f09d533e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d009d000 0x7f09d00b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.485+0000 7f09d533e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00b3f10 con 0x7f09d00b1380 2020-11-20T12:05:55.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.485+0000 7f09d533e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d015d020 con 0x7f09d00b1760 2020-11-20T12:05:55.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.485+0000 7f09d533e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00b4640 con 0x7f09d009d000 2020-11-20T12:05:55.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09c77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d00b1760 0x7f09d009cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09c77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d00b1760 0x7f09d009cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41706/0 (socket says 172.21.15.187:41706) 2020-11-20T12:05:55.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09c77fe700 1 -- 172.21.15.187:0/2581586153 learned_addr learned my addr 172.21.15.187:0/2581586153 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:55.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09d4b3d700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d009d000 0x7f09d00b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09cffff700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09d00b1380 0x7f09d015c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09c77fe700 1 -- 172.21.15.187:0/2581586153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d009d000 msgr2=0x7f09d00b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09c77fe700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d009d000 0x7f09d00b2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09c77fe700 1 -- 172.21.15.187:0/2581586153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09d00b1380 msgr2=0x7f09d015c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09c77fe700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09d00b1380 0x7f09d015c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.486+0000 7f09c77fe700 1 -- 172.21.15.187:0/2581586153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d009ada0 con 0x7f09d00b1760 2020-11-20T12:05:55.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.487+0000 7f09c77fe700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d00b1760 0x7f09d009cb10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f09b800eba0 tx=0x7f09b8006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.487+0000 7f09ccff9700 1 -- 172.21.15.187:0/2581586153 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b800ca60 con 0x7f09d00b1760 2020-11-20T12:05:55.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.487+0000 7f09ccff9700 1 -- 172.21.15.187:0/2581586153 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b800cbc0 con 0x7f09d00b1760 2020-11-20T12:05:55.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.487+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d015cbe0 con 0x7f09d00b1760 2020-11-20T12:05:55.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.487+0000 7f09ccff9700 1 -- 172.21.15.187:0/2581586153 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b80123f0 con 0x7f09d00b1760 2020-11-20T12:05:55.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.487+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d00b3340 con 0x7f09d00b1760 2020-11-20T12:05:55.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.489+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d015ce50 con 0x7f09d00b1760 2020-11-20T12:05:55.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.489+0000 7f09ccff9700 1 -- 172.21.15.187:0/2581586153 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f09b80127a0 con 0x7f09d00b1760 2020-11-20T12:05:55.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.490+0000 7f09ccff9700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f09ac056d20 0x7f09ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.490+0000 7f09ccff9700 1 -- 172.21.15.187:0/2581586153 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f09b80704b0 con 0x7f09d00b1760 2020-11-20T12:05:55.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.490+0000 7f09cffff700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f09ac056d20 0x7f09ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.491+0000 7f09cffff700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f09ac056d20 0x7f09ac0591e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f09d0159eb0 tx=0x7f09c0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.494+0000 7f09ccff9700 1 -- 172.21.15.187:0/2581586153 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f09b8045900 con 0x7f09d00b1760 2020-11-20T12:05:55.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.679+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f09d0061a20 con 0x7f09d00b1760 2020-11-20T12:05:55.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.679+0000 7f09ccff9700 1 -- 172.21.15.187:0/2581586153 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f09b8048f20 con 0x7f09d00b1760 2020-11-20T12:05:55.684 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:55.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f09ac056d20 msgr2=0x7f09ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f09ac056d20 0x7f09ac0591e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f09d0159eb0 tx=0x7f09c0009040).stop 2020-11-20T12:05:55.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d00b1760 msgr2=0x7f09d009cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d00b1760 0x7f09d009cb10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f09b800eba0 tx=0x7f09b8006df0).stop 2020-11-20T12:05:55.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 shutdown_connections 2020-11-20T12:05:55.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f09ac056d20 0x7f09ac0591e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09d00b1380 0x7f09d015c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09d009d000 0x7f09d00b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 --2- 172.21.15.187:0/2581586153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09d00b1760 0x7f09d009cb10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 >> 172.21.15.187:0/2581586153 conn(0x7f09d00bab30 msgr2=0x7f09d014c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:55.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 shutdown_connections 2020-11-20T12:05:55.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.682+0000 7f09d533e700 1 -- 172.21.15.187:0/2581586153 wait complete. 2020-11-20T12:05:55.708 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:55.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.953+0000 7f75a759e700 1 Processor -- start 2020-11-20T12:05:55.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.953+0000 7f75a759e700 1 -- start start 2020-11-20T12:05:55.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.954+0000 7f75a759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f75a80b6650 0x7f75a80afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.954+0000 7f75a759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a80b1300 con 0x7f75a8066f70 2020-11-20T12:05:55.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.954+0000 7f75a759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a80b15a0 con 0x7f75a81484d0 2020-11-20T12:05:55.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.954+0000 7f75a759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a8067730 con 0x7f75a80b6650 2020-11-20T12:05:55.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.955+0000 7f75a659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f75a81484d0 0x7f75a8061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38316/0 (socket says 172.21.15.187:38316) 2020-11-20T12:05:55.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.955+0000 7f75a659c700 1 -- 172.21.15.187:0/2814555036 learned_addr learned my addr 172.21.15.187:0/2814555036 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:55.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.955+0000 7f759dd9b700 1 --2- 172.21.15.187:0/2814555036 >> v2:172.21.15.187:3300/0 conn(0x7f75a80b6650 0x7f75a80afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.955+0000 7f75a5d9b700 1 -- 172.21.15.187:0/2814555036 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084307266 0 0) 0x7f75a80b15a0 con 0x7f75a81484d0 2020-11-20T12:05:55.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.956+0000 7f75a5d9b700 1 -- 172.21.15.187:0/2814555036 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7588003240 con 0x7f75a81484d0 2020-11-20T12:05:55.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.956+0000 7f75a5d9b700 1 -- 172.21.15.187:0/2814555036 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4286988804 0 0) 0x7f7588003240 con 0x7f75a81484d0 2020-11-20T12:05:55.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.956+0000 7f759dd9b700 1 -- 172.21.15.187:0/2814555036 >> v1:172.21.15.187:6789/0 conn(0x7f75a81484d0 legacy=0x7f75a8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.956+0000 7f759dd9b700 1 -- 172.21.15.187:0/2814555036 >> v1:172.21.15.5:6789/0 conn(0x7f75a8066f70 legacy=0x7f75a80aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.956+0000 7f759dd9b700 1 -- 172.21.15.187:0/2814555036 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a80b1840 con 0x7f75a80b6650 2020-11-20T12:05:55.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.956+0000 7f759dd9b700 1 --2- 172.21.15.187:0/2814555036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a80afa90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f759000b920 tx=0x7f7590006df0).ready entity=mon.1 client_cookie=b7dc588ee705901f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a5d9b700 1 -- 172.21.15.187:0/2814555036 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598001860 con 0x7f75a80b6650 2020-11-20T12:05:55.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a5d9b700 1 -- 172.21.15.187:0/2814555036 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590011400 con 0x7f75a80b6650 2020-11-20T12:05:55.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a5d9b700 1 -- 172.21.15.187:0/2814555036 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590011780 con 0x7f75a80b6650 2020-11-20T12:05:55.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a759e700 1 -- 172.21.15.187:0/2814555036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 msgr2=0x7f75a80afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a759e700 1 --2- 172.21.15.187:0/2814555036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a80afa90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f759000b920 tx=0x7f7590006df0).stop 2020-11-20T12:05:55.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a759e700 1 -- 172.21.15.187:0/2814555036 shutdown_connections 2020-11-20T12:05:55.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a759e700 1 --2- 172.21.15.187:0/2814555036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a80afa90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a759e700 1 -- 172.21.15.187:0/2814555036 >> 172.21.15.187:0/2814555036 conn(0x7f75a80bab30 msgr2=0x7f75a80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:55.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a759e700 1 -- 172.21.15.187:0/2814555036 shutdown_connections 2020-11-20T12:05:55.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.957+0000 7f75a759e700 1 -- 172.21.15.187:0/2814555036 wait complete. 2020-11-20T12:05:55.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.959+0000 7f75a759e700 1 Processor -- start 2020-11-20T12:05:55.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.959+0000 7f75a759e700 1 -- start start 2020-11-20T12:05:55.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f75a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75a8066f70 0x7f75a809cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f75a759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a809d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f75a759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75a81484d0 0x7f75a815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f75a759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a809d650 con 0x7f75a81484d0 2020-11-20T12:05:55.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f75a759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a80b3050 con 0x7f75a80b6650 2020-11-20T12:05:55.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f75a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a80b32f0 con 0x7f75a8066f70 2020-11-20T12:05:55.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f759dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a809d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f759dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a809d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41716/0 (socket says 172.21.15.187:41716) 2020-11-20T12:05:55.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.960+0000 7f759dd9b700 1 -- 172.21.15.187:0/1075405696 learned_addr learned my addr 172.21.15.187:0/1075405696 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:55.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f75a659c700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75a8066f70 0x7f75a809cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759dd9b700 1 -- 172.21.15.187:0/1075405696 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75a8066f70 msgr2=0x7f75a809cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759dd9b700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75a8066f70 0x7f75a809cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:55.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759dd9b700 1 -- 172.21.15.187:0/1075405696 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75a81484d0 msgr2=0x7f75a815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:55.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759dd9b700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75a81484d0 0x7f75a815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:55.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759dd9b700 1 -- 172.21.15.187:0/1075405696 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a80b3590 con 0x7f75a80b6650 2020-11-20T12:05:55.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759dd9b700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a809d160 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f75a8066430 tx=0x7f7590015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759effd700 1 -- 172.21.15.187:0/1075405696 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590009cd0 con 0x7f75a80b6650 2020-11-20T12:05:55.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759effd700 1 -- 172.21.15.187:0/1075405696 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590009e30 con 0x7f75a80b6650 2020-11-20T12:05:55.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a80b3a30 con 0x7f75a80b6650 2020-11-20T12:05:55.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.961+0000 7f759effd700 1 -- 172.21.15.187:0/1075405696 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759001e670 con 0x7f75a80b6650 2020-11-20T12:05:55.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.962+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a814d3d0 con 0x7f75a80b6650 2020-11-20T12:05:55.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.963+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a80b3720 con 0x7f75a80b6650 2020-11-20T12:05:55.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.963+0000 7f759effd700 1 -- 172.21.15.187:0/1075405696 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7590025030 con 0x7f75a80b6650 2020-11-20T12:05:55.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.964+0000 7f759effd700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7584056d20 0x7f75840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:55.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.964+0000 7f759effd700 1 -- 172.21.15.187:0/1075405696 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7590071530 con 0x7f75a80b6650 2020-11-20T12:05:55.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.964+0000 7f75a659c700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7584056d20 0x7f75840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:55.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.965+0000 7f75a659c700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7584056d20 0x7f75840591e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f7598001520 tx=0x7f7598009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:55.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:55.967+0000 7f759effd700 1 -- 172.21.15.187:0/1075405696 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f75900469d0 con 0x7f75a80b6650 2020-11-20T12:05:56.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.159+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f75a8061a20 con 0x7f75a80b6650 2020-11-20T12:05:56.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.160+0000 7f759effd700 1 -- 172.21.15.187:0/1075405696 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7590023030 con 0x7f75a80b6650 2020-11-20T12:05:56.164 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:56.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.162+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7584056d20 msgr2=0x7f75840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.162+0000 7f75a759e700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7584056d20 0x7f75840591e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f7598001520 tx=0x7f7598009040).stop 2020-11-20T12:05:56.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.162+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 msgr2=0x7f75a809d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.162+0000 7f75a759e700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a809d160 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f75a8066430 tx=0x7f7590015000).stop 2020-11-20T12:05:56.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.163+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 shutdown_connections 2020-11-20T12:05:56.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.163+0000 7f75a759e700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7584056d20 0x7f75840591e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.163+0000 7f75a759e700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75a81484d0 0x7f75a815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:56.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.163+0000 7f75a759e700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75a8066f70 0x7f75a809cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.163+0000 7f75a759e700 1 --2- 172.21.15.187:0/1075405696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75a80b6650 0x7f75a809d160 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.163+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 >> 172.21.15.187:0/1075405696 conn(0x7f75a80bab30 msgr2=0x7f75a8147140 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:56.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.163+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 shutdown_connections 2020-11-20T12:05:56.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.163+0000 7f75a759e700 1 -- 172.21.15.187:0/1075405696 wait complete. 2020-11-20T12:05:56.187 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:56.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.426+0000 7f20affff700 1 Processor -- start 2020-11-20T12:05:56.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.426+0000 7f20affff700 1 -- start start 2020-11-20T12:05:56.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.427+0000 7f20affff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20b01484d0 0x7f20b0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.427+0000 7f20affff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20b00622c0 0x7f20b00b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.427+0000 7f20affff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b00b1300 con 0x7f20b0066f70 2020-11-20T12:05:56.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.427+0000 7f20affff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b0067730 con 0x7f20b00622c0 2020-11-20T12:05:56.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.427+0000 7f20affff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b00b15a0 con 0x7f20b01484d0 2020-11-20T12:05:56.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20aeffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20b01484d0 0x7f20b0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20aeffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20b01484d0 0x7f20b0061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41728/0 (socket says 172.21.15.187:41728) 2020-11-20T12:05:56.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20aeffd700 1 -- 172.21.15.187:0/3199391949 learned_addr learned my addr 172.21.15.187:0/3199391949 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:56.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20a67fc700 1 --2- 172.21.15.187:0/3199391949 >> v2:172.21.15.5:3300/0 conn(0x7f20b00622c0 0x7f20b00b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20aeffd700 1 -- 172.21.15.187:0/3199391949 >> v2:172.21.15.5:3300/0 conn(0x7f20b00622c0 msgr2=0x7f20b00b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:56.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20aeffd700 1 --2- 172.21.15.187:0/3199391949 >> v2:172.21.15.5:3300/0 conn(0x7f20b00622c0 0x7f20b00b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20aeffd700 1 -- 172.21.15.187:0/3199391949 >> v1:172.21.15.5:6789/0 conn(0x7f20b0066f70 legacy=0x7f20b00b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:56.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20aeffd700 1 -- 172.21.15.187:0/3199391949 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b00b1840 con 0x7f20b01484d0 2020-11-20T12:05:56.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.428+0000 7f20a67fc700 1 --2- 172.21.15.187:0/3199391949 >> v2:172.21.15.5:3300/0 conn(0x7f20b00622c0 0x7f20b00b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:56.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.429+0000 7f20aeffd700 1 --2- 172.21.15.187:0/3199391949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b01484d0 0x7f20b0061dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f20b00b85e0 tx=0x7f20a0006eb0).ready entity=mon.1 client_cookie=449de1539ff2299e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:56.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.429+0000 7f20ae7fc700 1 -- 172.21.15.187:0/3199391949 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0009890 con 0x7f20b01484d0 2020-11-20T12:05:56.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.429+0000 7f20ae7fc700 1 -- 172.21.15.187:0/3199391949 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a00099f0 con 0x7f20b01484d0 2020-11-20T12:05:56.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.429+0000 7f20ae7fc700 1 -- 172.21.15.187:0/3199391949 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0009d70 con 0x7f20b01484d0 2020-11-20T12:05:56.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.430+0000 7f20affff700 1 -- 172.21.15.187:0/3199391949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b01484d0 msgr2=0x7f20b0061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.430+0000 7f20affff700 1 --2- 172.21.15.187:0/3199391949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b01484d0 0x7f20b0061dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f20b00b85e0 tx=0x7f20a0006eb0).stop 2020-11-20T12:05:56.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.430+0000 7f20affff700 1 -- 172.21.15.187:0/3199391949 shutdown_connections 2020-11-20T12:05:56.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.430+0000 7f20affff700 1 --2- 172.21.15.187:0/3199391949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b01484d0 0x7f20b0061dd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.430+0000 7f20affff700 1 --2- 172.21.15.187:0/3199391949 >> v2:172.21.15.5:3300/0 conn(0x7f20b00622c0 0x7f20b00b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.430+0000 7f20affff700 1 -- 172.21.15.187:0/3199391949 >> 172.21.15.187:0/3199391949 conn(0x7f20b00bab30 msgr2=0x7f20b00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:56.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.430+0000 7f20affff700 1 -- 172.21.15.187:0/3199391949 shutdown_connections 2020-11-20T12:05:56.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.430+0000 7f20affff700 1 -- 172.21.15.187:0/3199391949 wait complete. 2020-11-20T12:05:56.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.432+0000 7f20affff700 1 Processor -- start 2020-11-20T12:05:56.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.432+0000 7f20affff700 1 -- start start 2020-11-20T12:05:56.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.433+0000 7f20affff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20b00622c0 0x7f20b009d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.433+0000 7f20affff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b0066f70 0x7f20b0147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.433+0000 7f20affff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20b009b530 0x7f20b00b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.433+0000 7f20affff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00b3f70 con 0x7f20b00622c0 2020-11-20T12:05:56.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.433+0000 7f20affff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00b4210 con 0x7f20b0066f70 2020-11-20T12:05:56.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.433+0000 7f20affff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b0147cf0 con 0x7f20b009b530 2020-11-20T12:05:56.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20a67fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b0066f70 0x7f20b0147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20a67fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b0066f70 0x7f20b0147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41732/0 (socket says 172.21.15.187:41732) 2020-11-20T12:05:56.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20a67fc700 1 -- 172.21.15.187:0/350261166 learned_addr learned my addr 172.21.15.187:0/350261166 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:56.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20aeffd700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20b00622c0 0x7f20b009d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20af7fe700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20b009b530 0x7f20b00b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20a67fc700 1 -- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20b009b530 msgr2=0x7f20b00b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20a67fc700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20b009b530 0x7f20b00b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20a67fc700 1 -- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20b00622c0 msgr2=0x7f20b009d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20a67fc700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20b00622c0 0x7f20b009d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20a67fc700 1 -- 172.21.15.187:0/350261166 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b009bd80 con 0x7f20b0066f70 2020-11-20T12:05:56.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.434+0000 7f20af7fe700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20b009b530 0x7f20b00b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:56.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.435+0000 7f20a67fc700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b0066f70 0x7f20b0147800 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f209800cb50 tx=0x7f2098008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:56.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.435+0000 7f20a77fe700 1 -- 172.21.15.187:0/350261166 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098007860 con 0x7f20b0066f70 2020-11-20T12:05:56.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.435+0000 7f20a77fe700 1 -- 172.21.15.187:0/350261166 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20980079c0 con 0x7f20b0066f70 2020-11-20T12:05:56.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.435+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b00b7dc0 con 0x7f20b0066f70 2020-11-20T12:05:56.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.435+0000 7f20a77fe700 1 -- 172.21.15.187:0/350261166 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098007d40 con 0x7f20b0066f70 2020-11-20T12:05:56.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.435+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b0144ee0 con 0x7f20b0066f70 2020-11-20T12:05:56.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.436+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b00b43d0 con 0x7f20b0066f70 2020-11-20T12:05:56.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.437+0000 7f20a77fe700 1 -- 172.21.15.187:0/350261166 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2098011620 con 0x7f20b0066f70 2020-11-20T12:05:56.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.437+0000 7f20a77fe700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f208c056d20 0x7f208c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.437+0000 7f20a77fe700 1 -- 172.21.15.187:0/350261166 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2098070050 con 0x7f20b0066f70 2020-11-20T12:05:56.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.438+0000 7f20aeffd700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f208c056d20 0x7f208c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.438+0000 7f20aeffd700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f208c056d20 0x7f208c0591e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f20b0066540 tx=0x7f20a0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:56.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.441+0000 7f20a77fe700 1 -- 172.21.15.187:0/350261166 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2098045440 con 0x7f20b0066f70 2020-11-20T12:05:56.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.629+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f20b0061a20 con 0x7f20b0066f70 2020-11-20T12:05:56.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.629+0000 7f20a77fe700 1 -- 172.21.15.187:0/350261166 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2098048a60 con 0x7f20b0066f70 2020-11-20T12:05:56.634 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:05:56.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f208c056d20 msgr2=0x7f208c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f208c056d20 0x7f208c0591e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f20b0066540 tx=0x7f20a0015000).stop 2020-11-20T12:05:56.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b0066f70 msgr2=0x7f20b0147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b0066f70 0x7f20b0147800 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f209800cb50 tx=0x7f2098008040).stop 2020-11-20T12:05:56.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 shutdown_connections 2020-11-20T12:05:56.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f208c056d20 0x7f208c0591e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20b00622c0 0x7f20b009d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20b009b530 0x7f20b00b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 --2- 172.21.15.187:0/350261166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20b0066f70 0x7f20b0147800 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 >> 172.21.15.187:0/350261166 conn(0x7f20b00bab30 msgr2=0x7f20b0142fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:56.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 shutdown_connections 2020-11-20T12:05:56.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.633+0000 7f20affff700 1 -- 172.21.15.187:0/350261166 wait complete. 2020-11-20T12:05:56.663 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:56.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.906+0000 7f80b3fff700 1 Processor -- start 2020-11-20T12:05:56.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.906+0000 7f80b3fff700 1 -- start start 2020-11-20T12:05:56.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.907+0000 7f80b3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b40b1760 0x7f80b4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.907+0000 7f80b3fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f80b4065680 0x7f80b414c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.907+0000 7f80b3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b4150390 con 0x7f80b40b1380 2020-11-20T12:05:56.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.907+0000 7f80b3fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b41511e0 con 0x7f80b4065680 2020-11-20T12:05:56.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.907+0000 7f80b3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b414d760 con 0x7f80b40b1760 2020-11-20T12:05:56.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.908+0000 7f80aa7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b40b1760 0x7f80b4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.908+0000 7f80b37fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f80b4065680 0x7f80b414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.908+0000 7f80aa7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b40b1760 0x7f80b4065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:42628/0 (socket says 172.21.15.187:42628) 2020-11-20T12:05:56.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.908+0000 7f80aa7fc700 1 -- 172.21.15.187:0/3712278596 learned_addr learned my addr 172.21.15.187:0/3712278596 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:56.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.908+0000 7f80b2ffd700 1 --1- 172.21.15.187:0/3712278596 >> v1:172.21.15.5:6789/0 conn(0x7f80b40b1380 0x7f80b4150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:60086/0 (socket says 172.21.15.187:60086) 2020-11-20T12:05:56.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.909+0000 7f80b27fc700 1 -- 172.21.15.187:0/3712278596 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348100189 0 0) 0x7f80b4150390 con 0x7f80b40b1380 2020-11-20T12:05:56.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.909+0000 7f80aa7fc700 1 -- 172.21.15.187:0/3712278596 >> v2:172.21.15.5:3300/0 conn(0x7f80b4065680 msgr2=0x7f80b414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:56.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.910+0000 7f80aa7fc700 1 --2- 172.21.15.187:0/3712278596 >> v2:172.21.15.5:3300/0 conn(0x7f80b4065680 0x7f80b414c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.910+0000 7f80aa7fc700 1 -- 172.21.15.187:0/3712278596 >> v1:172.21.15.5:6789/0 conn(0x7f80b40b1380 legacy=0x7f80b4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.910+0000 7f80aa7fc700 1 -- 172.21.15.187:0/3712278596 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b4150630 con 0x7f80b40b1760 2020-11-20T12:05:56.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.910+0000 7f80aa7fc700 1 --2- 172.21.15.187:0/3712278596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b40b1760 0x7f80b4065190 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f809c006fd0 tx=0x7f809c006eb0).ready entity=mon.2 client_cookie=43519d89b93631e5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:56.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.911+0000 7f80b27fc700 1 -- 172.21.15.187:0/3712278596 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c009810 con 0x7f80b40b1760 2020-11-20T12:05:56.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.911+0000 7f80b27fc700 1 -- 172.21.15.187:0/3712278596 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c009970 con 0x7f80b40b1760 2020-11-20T12:05:56.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.911+0000 7f80b27fc700 1 -- 172.21.15.187:0/3712278596 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c009cf0 con 0x7f80b40b1760 2020-11-20T12:05:56.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.912+0000 7f80b3fff700 1 -- 172.21.15.187:0/3712278596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b40b1760 msgr2=0x7f80b4065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.912+0000 7f80b3fff700 1 --2- 172.21.15.187:0/3712278596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b40b1760 0x7f80b4065190 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f809c006fd0 tx=0x7f809c006eb0).stop 2020-11-20T12:05:56.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.912+0000 7f80b3fff700 1 -- 172.21.15.187:0/3712278596 shutdown_connections 2020-11-20T12:05:56.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.912+0000 7f80b3fff700 1 --2- 172.21.15.187:0/3712278596 >> v2:172.21.15.5:3300/0 conn(0x7f80b4065680 0x7f80b414c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.912+0000 7f80b3fff700 1 --2- 172.21.15.187:0/3712278596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b40b1760 0x7f80b4065190 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.912+0000 7f80b3fff700 1 -- 172.21.15.187:0/3712278596 >> 172.21.15.187:0/3712278596 conn(0x7f80b40bab30 msgr2=0x7f80b40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:56.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.912+0000 7f80b3fff700 1 -- 172.21.15.187:0/3712278596 shutdown_connections 2020-11-20T12:05:56.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.912+0000 7f80b3fff700 1 -- 172.21.15.187:0/3712278596 wait complete. 2020-11-20T12:05:56.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.914+0000 7f80b3fff700 1 Processor -- start 2020-11-20T12:05:56.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.914+0000 7f80b3fff700 1 -- start start 2020-11-20T12:05:56.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.915+0000 7f80b3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80b40b1380 0x7f80b40b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.915+0000 7f80b3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80b40b5950 0x7f80b414ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.915+0000 7f80b3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b414a960 0x7f80b414adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.915+0000 7f80b3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b409a670 con 0x7f80b40b1380 2020-11-20T12:05:56.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.915+0000 7f80b3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40b65f0 con 0x7f80b40b5950 2020-11-20T12:05:56.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.915+0000 7f80b3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40b6890 con 0x7f80b414a960 2020-11-20T12:05:56.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.915+0000 7f80aa7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80b40b5950 0x7f80b414ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80aa7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80b40b5950 0x7f80b414ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41748/0 (socket says 172.21.15.187:41748) 2020-11-20T12:05:56.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80aa7fc700 1 -- 172.21.15.187:0/588895324 learned_addr learned my addr 172.21.15.187:0/588895324 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:56.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80b37fe700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b414a960 0x7f80b414adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80b2ffd700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80b40b1380 0x7f80b40b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80aa7fc700 1 -- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b414a960 msgr2=0x7f80b414adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80aa7fc700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b414a960 0x7f80b414adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80aa7fc700 1 -- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80b40b1380 msgr2=0x7f80b40b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:56.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80aa7fc700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80b40b1380 0x7f80b40b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:56.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80aa7fc700 1 -- 172.21.15.187:0/588895324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b40b6340 con 0x7f80b40b5950 2020-11-20T12:05:56.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80b37fe700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b414a960 0x7f80b414adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:56.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.916+0000 7f80aa7fc700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80b40b5950 0x7f80b414ea60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f80b40b85e0 tx=0x7f809c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:56.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.917+0000 7f80ab7fe700 1 -- 172.21.15.187:0/588895324 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c011e00 con 0x7f80b40b5950 2020-11-20T12:05:56.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.917+0000 7f80ab7fe700 1 -- 172.21.15.187:0/588895324 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c01e460 con 0x7f80b40b5950 2020-11-20T12:05:56.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.917+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b414ef50 con 0x7f80b40b5950 2020-11-20T12:05:56.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.917+0000 7f80ab7fe700 1 -- 172.21.15.187:0/588895324 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c01e7e0 con 0x7f80b40b5950 2020-11-20T12:05:56.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.917+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b40b1d00 con 0x7f80b40b5950 2020-11-20T12:05:56.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.918+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b40b9320 con 0x7f80b40b5950 2020-11-20T12:05:56.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.918+0000 7f80ab7fe700 1 -- 172.21.15.187:0/588895324 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f809c025030 con 0x7f80b40b5950 2020-11-20T12:05:56.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.919+0000 7f80ab7fe700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8088056d20 0x7f80880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:56.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.919+0000 7f80ab7fe700 1 -- 172.21.15.187:0/588895324 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f809c0716a0 con 0x7f80b40b5950 2020-11-20T12:05:56.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.919+0000 7f80b2ffd700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8088056d20 0x7f80880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:56.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.920+0000 7f80b2ffd700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8088056d20 0x7f80880591e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f80b40b4a10 tx=0x7f80a4007180).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:56.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:56.923+0000 7f80ab7fe700 1 -- 172.21.15.187:0/588895324 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f809c046ba0 con 0x7f80b40b5950 2020-11-20T12:05:57.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.169+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80b4061a20 con 0x7f80b40b5950 2020-11-20T12:05:57.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.171+0000 7f80ab7fe700 1 -- 172.21.15.187:0/588895324 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2130 (secure 0 0 0) 0x7f809c023070 con 0x7f80b40b5950 2020-11-20T12:05:57.175 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:57.175 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4211,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6833/2094645874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2094645874},{"type":"v1","addr":"172.21.15.5:6833","nonce":2094645874}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:57.868606+0000","modified":"2020-11-20T12:04:59.990344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3094884769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3094884769},{"type":"v1","addr":"172.21.15.187:6835","nonce":3094884769}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:57.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8088056d20 msgr2=0x7f80880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8088056d20 0x7f80880591e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f80b40b4a10 tx=0x7f80a4007180).stop 2020-11-20T12:05:57.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80b40b5950 msgr2=0x7f80b414ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80b40b5950 0x7f80b414ea60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f80b40b85e0 tx=0x7f809c015000).stop 2020-11-20T12:05:57.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 shutdown_connections 2020-11-20T12:05:57.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8088056d20 0x7f80880591e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80b40b1380 0x7f80b40b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80b414a960 0x7f80b414adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 --2- 172.21.15.187:0/588895324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80b40b5950 0x7f80b414ea60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.173+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 >> 172.21.15.187:0/588895324 conn(0x7f80b40bab30 msgr2=0x7f80b40ba2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:57.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.174+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 shutdown_connections 2020-11-20T12:05:57.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.174+0000 7f80b3fff700 1 -- 172.21.15.187:0/588895324 wait complete. 2020-11-20T12:05:57.181 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 6 2020-11-20T12:05:57.199 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:57.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.435+0000 7f0d9a4a9700 1 Processor -- start 2020-11-20T12:05:57.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.435+0000 7f0d9a4a9700 1 -- start start 2020-11-20T12:05:57.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.437+0000 7f0d9a4a9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0d94067eb0 0x7f0d940b5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.437+0000 7f0d9a4a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d94068290 0x7f0d940653c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.437+0000 7f0d9a4a9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9414c510 con 0x7f0d940658b0 2020-11-20T12:05:57.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.437+0000 7f0d9a4a9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d94065d40 con 0x7f0d94067eb0 2020-11-20T12:05:57.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.437+0000 7f0d9a4a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9414c7b0 con 0x7f0d94068290 2020-11-20T12:05:57.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.438+0000 7f0d98ca6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d94068290 0x7f0d940653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.438+0000 7f0d994a7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0d94067eb0 0x7f0d940b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.438+0000 7f0d98ca6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d94068290 0x7f0d940653c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:42640/0 (socket says 172.21.15.187:42640) 2020-11-20T12:05:57.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.438+0000 7f0d98ca6700 1 -- 172.21.15.187:0/2655656803 learned_addr learned my addr 172.21.15.187:0/2655656803 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:57.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.438+0000 7f0d99ca8700 1 --1- 172.21.15.187:0/2655656803 >> v1:172.21.15.5:6789/0 conn(0x7f0d940658b0 0x7f0d94061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:60104/0 (socket says 172.21.15.187:60104) 2020-11-20T12:05:57.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.439+0000 7f0d83fff700 1 -- 172.21.15.187:0/2655656803 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193943827 0 0) 0x7f0d9414c510 con 0x7f0d940658b0 2020-11-20T12:05:57.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.439+0000 7f0d83fff700 1 -- 172.21.15.187:0/2655656803 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d7c003240 con 0x7f0d940658b0 2020-11-20T12:05:57.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.439+0000 7f0d98ca6700 1 -- 172.21.15.187:0/2655656803 >> v2:172.21.15.5:3300/0 conn(0x7f0d94067eb0 msgr2=0x7f0d940b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:57.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.439+0000 7f0d98ca6700 1 --2- 172.21.15.187:0/2655656803 >> v2:172.21.15.5:3300/0 conn(0x7f0d94067eb0 0x7f0d940b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.439+0000 7f0d98ca6700 1 -- 172.21.15.187:0/2655656803 >> v1:172.21.15.5:6789/0 conn(0x7f0d940658b0 legacy=0x7f0d94061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.439+0000 7f0d98ca6700 1 -- 172.21.15.187:0/2655656803 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9414ca50 con 0x7f0d94068290 2020-11-20T12:05:57.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.440+0000 7f0d98ca6700 1 --2- 172.21.15.187:0/2655656803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d94068290 0x7f0d940653c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0d84006fd0 tx=0x7f0d84006eb0).ready entity=mon.2 client_cookie=a69466e3178e9ead server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:57.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d83fff700 1 -- 172.21.15.187:0/2655656803 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84009720 con 0x7f0d94068290 2020-11-20T12:05:57.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d83fff700 1 -- 172.21.15.187:0/2655656803 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d84009880 con 0x7f0d94068290 2020-11-20T12:05:57.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d83fff700 1 -- 172.21.15.187:0/2655656803 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84009c00 con 0x7f0d94068290 2020-11-20T12:05:57.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/2655656803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d94068290 msgr2=0x7f0d940653c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/2655656803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d94068290 0x7f0d940653c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0d84006fd0 tx=0x7f0d84006eb0).stop 2020-11-20T12:05:57.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/2655656803 shutdown_connections 2020-11-20T12:05:57.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/2655656803 >> v2:172.21.15.5:3300/0 conn(0x7f0d94067eb0 0x7f0d940b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/2655656803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d94068290 0x7f0d940653c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.441+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/2655656803 >> 172.21.15.187:0/2655656803 conn(0x7f0d940bab30 msgr2=0x7f0d940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:57.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.442+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/2655656803 shutdown_connections 2020-11-20T12:05:57.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.442+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/2655656803 wait complete. 2020-11-20T12:05:57.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.443+0000 7f0d9a4a9700 1 Processor -- start 2020-11-20T12:05:57.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.443+0000 7f0d9a4a9700 1 -- start start 2020-11-20T12:05:57.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.444+0000 7f0d9a4a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d940658b0 0x7f0d9413bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.444+0000 7f0d9a4a9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d94067eb0 0x7f0d941499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.444+0000 7f0d9a4a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d94068290 0x7f0d94149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.444+0000 7f0d9a4a9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9409ca00 con 0x7f0d94067eb0 2020-11-20T12:05:57.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.444+0000 7f0d9a4a9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9409cca0 con 0x7f0d94068290 2020-11-20T12:05:57.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.444+0000 7f0d9a4a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9409cf40 con 0x7f0d940658b0 2020-11-20T12:05:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d99ca8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d94068290 0x7f0d94149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d99ca8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d94068290 0x7f0d94149eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41762/0 (socket says 172.21.15.187:41762) 2020-11-20T12:05:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d99ca8700 1 -- 172.21.15.187:0/542325274 learned_addr learned my addr 172.21.15.187:0/542325274 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d98ca6700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d94067eb0 0x7f0d941499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d994a7700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d940658b0 0x7f0d9413bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d99ca8700 1 -- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d940658b0 msgr2=0x7f0d9413bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d99ca8700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d940658b0 0x7f0d9413bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d99ca8700 1 -- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d94067eb0 msgr2=0x7f0d941499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d99ca8700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d94067eb0 0x7f0d941499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d99ca8700 1 -- 172.21.15.187:0/542325274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9413c4a0 con 0x7f0d94068290 2020-11-20T12:05:57.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.445+0000 7f0d994a7700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d940658b0 0x7f0d9413bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:57.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.446+0000 7f0d99ca8700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d94068290 0x7f0d94149eb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f0d8c00ca90 tx=0x7f0d8c007de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:57.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.446+0000 7f0d817fa700 1 -- 172.21.15.187:0/542325274 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c005ac0 con 0x7f0d94068290 2020-11-20T12:05:57.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.446+0000 7f0d817fa700 1 -- 172.21.15.187:0/542325274 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8c005c20 con 0x7f0d94068290 2020-11-20T12:05:57.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.446+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d940a0890 con 0x7f0d94068290 2020-11-20T12:05:57.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.446+0000 7f0d817fa700 1 -- 172.21.15.187:0/542325274 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c007110 con 0x7f0d94068290 2020-11-20T12:05:57.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.446+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9409c770 con 0x7f0d94068290 2020-11-20T12:05:57.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.447+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d94061a20 con 0x7f0d94068290 2020-11-20T12:05:57.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.448+0000 7f0d817fa700 1 -- 172.21.15.187:0/542325274 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0d8c010030 con 0x7f0d94068290 2020-11-20T12:05:57.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.448+0000 7f0d817fa700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d6c056d20 0x7f0d6c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.448+0000 7f0d817fa700 1 -- 172.21.15.187:0/542325274 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0d8c06fee0 con 0x7f0d94068290 2020-11-20T12:05:57.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.449+0000 7f0d994a7700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d6c056d20 0x7f0d6c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.449+0000 7f0d994a7700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d6c056d20 0x7f0d6c0591e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0d90006bd0 tx=0x7f0d90008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:57.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.452+0000 7f0d817fa700 1 -- 172.21.15.187:0/542325274 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0d8c045330 con 0x7f0d94068290 2020-11-20T12:05:57.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.636+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d9414c190 con 0x7f0d94068290 2020-11-20T12:05:57.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.638+0000 7f0d817fa700 1 -- 172.21.15.187:0/542325274 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f0d8c048950 con 0x7f0d94068290 2020-11-20T12:05:57.643 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:05:57.643 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":23,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:04:58.873621+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:55.702880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:56.876271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:57.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d6c056d20 msgr2=0x7f0d6c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d6c056d20 0x7f0d6c0591e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0d90006bd0 tx=0x7f0d90008040).stop 2020-11-20T12:05:57.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d94068290 msgr2=0x7f0d94149eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d94068290 0x7f0d94149eb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f0d8c00ca90 tx=0x7f0d8c007de0).stop 2020-11-20T12:05:57.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 shutdown_connections 2020-11-20T12:05:57.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d6c056d20 0x7f0d6c0591e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d94068290 0x7f0d94149eb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d940658b0 0x7f0d9413bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 --2- 172.21.15.187:0/542325274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d94067eb0 0x7f0d941499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 >> 172.21.15.187:0/542325274 conn(0x7f0d940bab30 msgr2=0x7f0d940b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:57.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 shutdown_connections 2020-11-20T12:05:57.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.641+0000 7f0d9a4a9700 1 -- 172.21.15.187:0/542325274 wait complete. 2020-11-20T12:05:57.669 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:05:57.670 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:05:57.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.910+0000 7f07fed87700 1 Processor -- start 2020-11-20T12:05:57.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.910+0000 7f07fed87700 1 -- start start 2020-11-20T12:05:57.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.911+0000 7f07fed87700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f80b6650 0x7f07f80afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.911+0000 7f07fed87700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f07f8066f70 0x7f07f80aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.911+0000 7f07fed87700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f80b1300 con 0x7f07f81484d0 2020-11-20T12:05:57.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.911+0000 7f07fed87700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f8067730 con 0x7f07f8066f70 2020-11-20T12:05:57.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.911+0000 7f07fed87700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80b15a0 con 0x7f07f80b6650 2020-11-20T12:05:57.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.912+0000 7f07fd584700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f80b6650 0x7f07f80afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.912+0000 7f07fe586700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f07f8066f70 0x7f07f80aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.913+0000 7f07fdd85700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f07f81484d0 0x7f07f8061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:60118/0 (socket says 172.21.15.187:60118) 2020-11-20T12:05:57.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.913+0000 7f07fe586700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f07f8066f70 0x7f07f80aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:34030/0 (socket says 172.21.15.187:34030) 2020-11-20T12:05:57.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.913+0000 7f07fdd85700 1 -- 172.21.15.187:0/3768327358 learned_addr learned my addr 172.21.15.187:0/3768327358 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:57.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.913+0000 7f07fd584700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f80b6650 0x7f07f80afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:42654/0 (socket says 172.21.15.187:42654) 2020-11-20T12:05:57.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.914+0000 7f07fe586700 1 -- 172.21.15.187:0/3768327358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f80b6650 msgr2=0x7f07f80afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:57.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.914+0000 7f07fe586700 1 --2- 172.21.15.187:0/3768327358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f80b6650 0x7f07f80afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.914+0000 7f07fe586700 1 -- 172.21.15.187:0/3768327358 >> v1:172.21.15.5:6789/0 conn(0x7f07f81484d0 legacy=0x7f07f8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.914+0000 7f07fe586700 1 -- 172.21.15.187:0/3768327358 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f80b1840 con 0x7f07f8066f70 2020-11-20T12:05:57.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.914+0000 7f07fe586700 1 --2- 172.21.15.187:0/3768327358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f8066f70 0x7f07f80aff80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f07f000b9a0 tx=0x7f07f0006df0).ready entity=mon.0 client_cookie=e9288908a3dab741 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:57.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fcd83700 1 -- 172.21.15.187:0/3768327358 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f00097b0 con 0x7f07f8066f70 2020-11-20T12:05:57.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fcd83700 1 -- 172.21.15.187:0/3768327358 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f0009910 con 0x7f07f8066f70 2020-11-20T12:05:57.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fcd83700 1 -- 172.21.15.187:0/3768327358 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f0009c90 con 0x7f07f8066f70 2020-11-20T12:05:57.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fed87700 1 -- 172.21.15.187:0/3768327358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f8066f70 msgr2=0x7f07f80aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fed87700 1 --2- 172.21.15.187:0/3768327358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f8066f70 0x7f07f80aff80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f07f000b9a0 tx=0x7f07f0006df0).stop 2020-11-20T12:05:57.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fed87700 1 -- 172.21.15.187:0/3768327358 shutdown_connections 2020-11-20T12:05:57.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fed87700 1 --2- 172.21.15.187:0/3768327358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f8066f70 0x7f07f80aff80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fed87700 1 --2- 172.21.15.187:0/3768327358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f80b6650 0x7f07f80afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.915+0000 7f07fed87700 1 -- 172.21.15.187:0/3768327358 >> 172.21.15.187:0/3768327358 conn(0x7f07f80bab30 msgr2=0x7f07f80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:57.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.916+0000 7f07fed87700 1 -- 172.21.15.187:0/3768327358 shutdown_connections 2020-11-20T12:05:57.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.916+0000 7f07fed87700 1 -- 172.21.15.187:0/3768327358 wait complete. 2020-11-20T12:05:57.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.918+0000 7f07fed87700 1 Processor -- start 2020-11-20T12:05:57.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.918+0000 7f07fed87700 1 -- start start 2020-11-20T12:05:57.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.919+0000 7f07fed87700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f8066f70 0x7f07f809c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.919+0000 7f07fed87700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f80b6650 0x7f07f8143800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.919+0000 7f07fed87700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07f81484d0 0x7f07f809dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.919+0000 7f07fed87700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80b3580 con 0x7f07f80b6650 2020-11-20T12:05:57.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.919+0000 7f07fed87700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80b3820 con 0x7f07f81484d0 2020-11-20T12:05:57.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.919+0000 7f07fed87700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80b3ac0 con 0x7f07f8066f70 2020-11-20T12:05:57.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.919+0000 7f07fe586700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07f81484d0 0x7f07f809dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fe586700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07f81484d0 0x7f07f809dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41776/0 (socket says 172.21.15.187:41776) 2020-11-20T12:05:57.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fe586700 1 -- 172.21.15.187:0/881938154 learned_addr learned my addr 172.21.15.187:0/881938154 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:57.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fd584700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f80b6650 0x7f07f8143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fe586700 1 -- 172.21.15.187:0/881938154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f8066f70 msgr2=0x7f07f809c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fe586700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f8066f70 0x7f07f809c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:57.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fe586700 1 -- 172.21.15.187:0/881938154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f80b6650 msgr2=0x7f07f8143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:57.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fe586700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f80b6650 0x7f07f8143800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:57.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fe586700 1 -- 172.21.15.187:0/881938154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f809d580 con 0x7f07f81484d0 2020-11-20T12:05:57.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07fe586700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07f81484d0 0x7f07f809dae0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f07f8066430 tx=0x7f07f0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:57.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.920+0000 7f07e5ffb700 1 -- 172.21.15.187:0/881938154 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f001e460 con 0x7f07f81484d0 2020-11-20T12:05:57.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.921+0000 7f07e5ffb700 1 -- 172.21.15.187:0/881938154 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f001e5c0 con 0x7f07f81484d0 2020-11-20T12:05:57.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.921+0000 7f07e5ffb700 1 -- 172.21.15.187:0/881938154 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f001e940 con 0x7f07f81484d0 2020-11-20T12:05:57.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.921+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f8143cf0 con 0x7f07f81484d0 2020-11-20T12:05:57.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.921+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f809e210 con 0x7f07f81484d0 2020-11-20T12:05:57.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.922+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f80b3c80 con 0x7f07f81484d0 2020-11-20T12:05:57.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.922+0000 7f07e5ffb700 1 -- 172.21.15.187:0/881938154 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f07f0025030 con 0x7f07f81484d0 2020-11-20T12:05:57.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.923+0000 7f07e5ffb700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07d0056d20 0x7f07d00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:57.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.923+0000 7f07e5ffb700 1 -- 172.21.15.187:0/881938154 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f07f0071990 con 0x7f07f81484d0 2020-11-20T12:05:57.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.923+0000 7f07fdd85700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07d0056d20 0x7f07d00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:57.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.924+0000 7f07fdd85700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07d0056d20 0x7f07d00591e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f07f4001990 tx=0x7f07f4007be0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:57.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:57.927+0000 7f07e5ffb700 1 -- 172.21.15.187:0/881938154 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f07f0046d80 con 0x7f07f81484d0 2020-11-20T12:05:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:58.173+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f07f8061a20 con 0x7f07f81484d0 2020-11-20T12:05:59.047 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:05:59.044+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:05:59.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.063+0000 7f07e5ffb700 1 -- 172.21.15.187:0/881938154 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7f07f0023070 con 0x7f07f81484d0 2020-11-20T12:05:59.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07d0056d20 msgr2=0x7f07d00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:59.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07d0056d20 0x7f07d00591e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f07f4001990 tx=0x7f07f4007be0).stop 2020-11-20T12:05:59.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07f81484d0 msgr2=0x7f07f809dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:59.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07f81484d0 0x7f07f809dae0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f07f8066430 tx=0x7f07f0015000).stop 2020-11-20T12:05:59.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 shutdown_connections 2020-11-20T12:05:59.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07d0056d20 0x7f07d00591e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:59.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07f81484d0 0x7f07f809dae0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:59.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07f8066f70 0x7f07f809c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:59.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 --2- 172.21.15.187:0/881938154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07f80b6650 0x7f07f8143800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:59.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 >> 172.21.15.187:0/881938154 conn(0x7f07f80bab30 msgr2=0x7f07f81450b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:59.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 shutdown_connections 2020-11-20T12:05:59.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.067+0000 7f07fed87700 1 -- 172.21.15.187:0/881938154 wait complete. 2020-11-20T12:05:59.075 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:05:59.085 INFO:tasks.ceph.mds.a.smithi187.stderr: -2410> 2020-11-20T12:05:02.891+0000 7fb372f3c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:05:59.100 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:05:59.172 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:05:59.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.327+0000 7f2d38bf3700 1 Processor -- start 2020-11-20T12:05:59.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.328+0000 7f2d38bf3700 1 -- start start 2020-11-20T12:05:59.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.329+0000 7f2d38bf3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d34065300 0x7f2d340656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:59.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.329+0000 7f2d38bf3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d34066420 con 0x7f2d34068030 2020-11-20T12:05:59.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.329+0000 7f2d38bf3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d340b7380 con 0x7f2d3409a8a0 2020-11-20T12:05:59.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.329+0000 7f2d38bf3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d34061f60 con 0x7f2d34065300 2020-11-20T12:05:59.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.329+0000 7f2d3259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d34065300 0x7f2d340656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:59.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.329+0000 7f2d31d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2d3409a8a0 0x7f2d34062400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38400/0 (socket says 172.21.15.187:38400) 2020-11-20T12:05:59.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.329+0000 7f2d31d9b700 1 -- 172.21.15.187:0/4271850027 learned_addr learned my addr 172.21.15.187:0/4271850027 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:05:59.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.330+0000 7f2d3259c700 1 -- 172.21.15.187:0/4271850027 >> v1:172.21.15.187:6789/0 conn(0x7f2d3409a8a0 legacy=0x7f2d34062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:59.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.330+0000 7f2d3259c700 1 -- 172.21.15.187:0/4271850027 >> v1:172.21.15.5:6789/0 conn(0x7f2d34068030 legacy=0x7f2d34146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:59.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.330+0000 7f2d3159a700 1 -- 172.21.15.187:0/4271850027 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069813382 0 0) 0x7f2d340b7380 con 0x7f2d3409a8a0 2020-11-20T12:05:59.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.330+0000 7f2d3259c700 1 -- 172.21.15.187:0/4271850027 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d340b6fd0 con 0x7f2d34065300 2020-11-20T12:05:59.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.330+0000 7f2d3259c700 1 --2- 172.21.15.187:0/4271850027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34065300 0x7f2d340656e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f2d2400b9d0 tx=0x7f2d24006df0).ready entity=mon.1 client_cookie=82e14c021f07505f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:59.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.331+0000 7f2d3159a700 1 -- 172.21.15.187:0/4271850027 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24009880 con 0x7f2d34065300 2020-11-20T12:05:59.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.331+0000 7f2d3159a700 1 -- 172.21.15.187:0/4271850027 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d240099e0 con 0x7f2d34065300 2020-11-20T12:05:59.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.331+0000 7f2d3159a700 1 -- 172.21.15.187:0/4271850027 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24009d60 con 0x7f2d34065300 2020-11-20T12:05:59.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.331+0000 7f2d38bf3700 1 -- 172.21.15.187:0/4271850027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34065300 msgr2=0x7f2d340656e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:59.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.331+0000 7f2d38bf3700 1 --2- 172.21.15.187:0/4271850027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34065300 0x7f2d340656e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f2d2400b9d0 tx=0x7f2d24006df0).stop 2020-11-20T12:05:59.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.332+0000 7f2d38bf3700 1 -- 172.21.15.187:0/4271850027 shutdown_connections 2020-11-20T12:05:59.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.332+0000 7f2d38bf3700 1 --2- 172.21.15.187:0/4271850027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34065300 0x7f2d340656e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:59.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.332+0000 7f2d38bf3700 1 -- 172.21.15.187:0/4271850027 >> 172.21.15.187:0/4271850027 conn(0x7f2d340bacf0 msgr2=0x7f2d340bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:59.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.332+0000 7f2d38bf3700 1 -- 172.21.15.187:0/4271850027 shutdown_connections 2020-11-20T12:05:59.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.332+0000 7f2d38bf3700 1 -- 172.21.15.187:0/4271850027 wait complete. 2020-11-20T12:05:59.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.333+0000 7f2d38bf3700 1 Processor -- start 2020-11-20T12:05:59.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.333+0000 7f2d38bf3700 1 -- start start 2020-11-20T12:05:59.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d38bf3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d34065300 0x7f2d34159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:59.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d38bf3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34068030 0x7f2d340a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:59.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d38bf3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d3409a8a0 0x7f2d3414a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:59.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d38bf3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3415a3a0 con 0x7f2d3409a8a0 2020-11-20T12:05:59.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d38bf3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341594d0 con 0x7f2d34068030 2020-11-20T12:05:59.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d38bf3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34159770 con 0x7f2d34065300 2020-11-20T12:05:59.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d31d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34068030 0x7f2d340a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:59.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d31d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34068030 0x7f2d340a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41800/0 (socket says 172.21.15.187:41800) 2020-11-20T12:05:59.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.334+0000 7f2d31d9b700 1 -- 172.21.15.187:0/1473424094 learned_addr learned my addr 172.21.15.187:0/1473424094 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:05:59.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d31d9b700 1 -- 172.21.15.187:0/1473424094 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d34065300 msgr2=0x7f2d34159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:59.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d31d9b700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d34065300 0x7f2d34159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:59.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d31d9b700 1 -- 172.21.15.187:0/1473424094 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d3409a8a0 msgr2=0x7f2d3414a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:59.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d31d9b700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d3409a8a0 0x7f2d3414a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:05:59.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d31d9b700 1 -- 172.21.15.187:0/1473424094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d34159a10 con 0x7f2d34068030 2020-11-20T12:05:59.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d31d9b700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34068030 0x7f2d340a0460 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f2d2800c720 tx=0x7f2d28007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d227fc700 1 -- 172.21.15.187:0/1473424094 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28007320 con 0x7f2d34068030 2020-11-20T12:05:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d227fc700 1 -- 172.21.15.187:0/1473424094 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d28007480 con 0x7f2d34068030 2020-11-20T12:05:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.335+0000 7f2d227fc700 1 -- 172.21.15.187:0/1473424094 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28005800 con 0x7f2d34068030 2020-11-20T12:05:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.336+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d340a0950 con 0x7f2d34068030 2020-11-20T12:05:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.336+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d340b12e0 con 0x7f2d34068030 2020-11-20T12:05:59.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.337+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d340b1530 con 0x7f2d34068030 2020-11-20T12:05:59.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.338+0000 7f2d227fc700 1 -- 172.21.15.187:0/1473424094 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2d28010030 con 0x7f2d34068030 2020-11-20T12:05:59.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.338+0000 7f2d227fc700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d08056d20 0x7f2d080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:59.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.338+0000 7f2d227fc700 1 -- 172.21.15.187:0/1473424094 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2d2806fda0 con 0x7f2d34068030 2020-11-20T12:05:59.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.339+0000 7f2d3259c700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d08056d20 0x7f2d080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:59.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.339+0000 7f2d3259c700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d08056d20 0x7f2d080591e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2d24009fd0 tx=0x7f2d24015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:59.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.343+0000 7f2d227fc700 1 -- 172.21.15.187:0/1473424094 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2d280451a0 con 0x7f2d34068030 2020-11-20T12:05:59.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:05:59.587+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2d34159ba0 con 0x7f2d34068030 2020-11-20T12:06:00.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.071+0000 7f2d227fc700 1 -- 172.21.15.187:0/1473424094 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2d28007c00 con 0x7f2d34068030 2020-11-20T12:06:00.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d08056d20 msgr2=0x7f2d080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:00.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d08056d20 0x7f2d080591e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2d24009fd0 tx=0x7f2d24015000).stop 2020-11-20T12:06:00.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34068030 msgr2=0x7f2d340a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:00.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34068030 0x7f2d340a0460 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f2d2800c720 tx=0x7f2d28007ac0).stop 2020-11-20T12:06:00.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 shutdown_connections 2020-11-20T12:06:00.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d08056d20 0x7f2d080591e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:00.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d3409a8a0 0x7f2d3414a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:00.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d34065300 0x7f2d34159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:00.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 --2- 172.21.15.187:0/1473424094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d34068030 0x7f2d340a0460 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:00.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 >> 172.21.15.187:0/1473424094 conn(0x7f2d340bacf0 msgr2=0x7f2d340666e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:00.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 shutdown_connections 2020-11-20T12:06:00.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.075+0000 7f2d38bf3700 1 -- 172.21.15.187:0/1473424094 wait complete. 2020-11-20T12:06:00.105 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:06:00.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.351+0000 7f547ee3c700 1 Processor -- start 2020-11-20T12:06:00.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.351+0000 7f547ee3c700 1 -- start start 2020-11-20T12:06:00.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.353+0000 7f547ee3c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f54780b6650 0x7f54780afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:00.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.353+0000 7f547ee3c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5478066f70 0x7f54780aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:00.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.353+0000 7f547ee3c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54780b1300 con 0x7f54781484d0 2020-11-20T12:06:00.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.353+0000 7f547ee3c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5478067730 con 0x7f54780b6650 2020-11-20T12:06:00.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.353+0000 7f547ee3c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54780b15a0 con 0x7f5478066f70 2020-11-20T12:06:00.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.353+0000 7f547d639700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f54780b6650 0x7f54780afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:00.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.353+0000 7f547d639700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f54780b6650 0x7f54780afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41810/0 (socket says 172.21.15.187:41810) 2020-11-20T12:06:00.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.353+0000 7f547d639700 1 -- 172.21.15.187:0/1817083192 learned_addr learned my addr 172.21.15.187:0/1817083192 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:00.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.354+0000 7f547d639700 1 -- 172.21.15.187:0/1817083192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5478066f70 msgr2=0x7f54780aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:00.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.354+0000 7f547d639700 1 --2- 172.21.15.187:0/1817083192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5478066f70 0x7f54780aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:00.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.354+0000 7f547d639700 1 -- 172.21.15.187:0/1817083192 >> v1:172.21.15.5:6789/0 conn(0x7f54781484d0 legacy=0x7f5478061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:00.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.354+0000 7f547d639700 1 -- 172.21.15.187:0/1817083192 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54780b1840 con 0x7f54780b6650 2020-11-20T12:06:00.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.354+0000 7f547d639700 1 --2- 172.21.15.187:0/1817083192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780b6650 0x7f54780afa90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f5468006fd0 tx=0x7f5468006eb0).ready entity=mon.1 client_cookie=a7dd580c4a0c50d8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:00.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.354+0000 7f547ce38700 1 -- 172.21.15.187:0/1817083192 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5468009890 con 0x7f54780b6650 2020-11-20T12:06:00.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.354+0000 7f547ce38700 1 -- 172.21.15.187:0/1817083192 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54680099f0 con 0x7f54780b6650 2020-11-20T12:06:00.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.354+0000 7f547ce38700 1 -- 172.21.15.187:0/1817083192 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5468009d70 con 0x7f54780b6650 2020-11-20T12:06:00.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.355+0000 7f547ee3c700 1 -- 172.21.15.187:0/1817083192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780b6650 msgr2=0x7f54780afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:00.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.355+0000 7f547ee3c700 1 --2- 172.21.15.187:0/1817083192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780b6650 0x7f54780afa90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f5468006fd0 tx=0x7f5468006eb0).stop 2020-11-20T12:06:00.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.355+0000 7f547ee3c700 1 -- 172.21.15.187:0/1817083192 shutdown_connections 2020-11-20T12:06:00.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.355+0000 7f547ee3c700 1 --2- 172.21.15.187:0/1817083192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5478066f70 0x7f54780aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:00.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.355+0000 7f547ee3c700 1 --2- 172.21.15.187:0/1817083192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780b6650 0x7f54780afa90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:00.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.355+0000 7f547ee3c700 1 -- 172.21.15.187:0/1817083192 >> 172.21.15.187:0/1817083192 conn(0x7f54780bab30 msgr2=0x7f54780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:00.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.355+0000 7f547ee3c700 1 -- 172.21.15.187:0/1817083192 shutdown_connections 2020-11-20T12:06:00.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.355+0000 7f547ee3c700 1 -- 172.21.15.187:0/1817083192 wait complete. 2020-11-20T12:06:00.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.357+0000 7f547ee3c700 1 Processor -- start 2020-11-20T12:06:00.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.357+0000 7f547ee3c700 1 -- start start 2020-11-20T12:06:00.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.358+0000 7f547ee3c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5478066f70 0x7f54780b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:00.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.358+0000 7f547ee3c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54780b6650 0x7f54781331c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:00.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.358+0000 7f547ee3c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54781484d0 0x7f547813cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:00.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.358+0000 7f547ee3c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5478139ae0 con 0x7f54781484d0 2020-11-20T12:06:00.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.358+0000 7f547ee3c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54780a0340 con 0x7f5478066f70 2020-11-20T12:06:00.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.358+0000 7f547ee3c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54780a05e0 con 0x7f54780b6650 2020-11-20T12:06:00.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547de3a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5478066f70 0x7f54780b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:00.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547de3a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5478066f70 0x7f54780b23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41812/0 (socket says 172.21.15.187:41812) 2020-11-20T12:06:00.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547de3a700 1 -- 172.21.15.187:0/2604269517 learned_addr learned my addr 172.21.15.187:0/2604269517 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:00.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547de3a700 1 -- 172.21.15.187:0/2604269517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54780b6650 msgr2=0x7f54781331c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:00.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547de3a700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54780b6650 0x7f54781331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:00.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547e63b700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54781484d0 0x7f547813cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:00.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547de3a700 1 -- 172.21.15.187:0/2604269517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54781484d0 msgr2=0x7f547813cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:00.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547de3a700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54781484d0 0x7f547813cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:00.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.359+0000 7f547de3a700 1 -- 172.21.15.187:0/2604269517 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54780a0880 con 0x7f5478066f70 2020-11-20T12:06:00.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.360+0000 7f547de3a700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5478066f70 0x7f54780b23f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f5474009710 tx=0x7f5474006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:00.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.360+0000 7f5465ffb700 1 -- 172.21.15.187:0/2604269517 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547400d560 con 0x7f5478066f70 2020-11-20T12:06:00.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.360+0000 7f5465ffb700 1 -- 172.21.15.187:0/2604269517 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5474000e90 con 0x7f5478066f70 2020-11-20T12:06:00.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.360+0000 7f5465ffb700 1 -- 172.21.15.187:0/2604269517 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547400d950 con 0x7f5478066f70 2020-11-20T12:06:00.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.360+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54781336b0 con 0x7f5478066f70 2020-11-20T12:06:00.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.360+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547813d470 con 0x7f5478066f70 2020-11-20T12:06:00.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.361+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5478061a20 con 0x7f5478066f70 2020-11-20T12:06:00.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.362+0000 7f5465ffb700 1 -- 172.21.15.187:0/2604269517 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f547400f030 con 0x7f5478066f70 2020-11-20T12:06:00.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.362+0000 7f5465ffb700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5450056d50 0x7f5450059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:00.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.362+0000 7f5465ffb700 1 -- 172.21.15.187:0/2604269517 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f547406fd00 con 0x7f5478066f70 2020-11-20T12:06:00.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.363+0000 7f547d639700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5450056d50 0x7f5450059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:00.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.363+0000 7f547d639700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5450056d50 0x7f5450059210 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f547815b740 tx=0x7f5468015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:00.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.366+0000 7f5465ffb700 1 -- 172.21.15.187:0/2604269517 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5474045020 con 0x7f5478066f70 2020-11-20T12:06:00.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:00.536+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f54780624e0 con 0x7f5478066f70 2020-11-20T12:06:01.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.076+0000 7f5465ffb700 1 -- 172.21.15.187:0/2604269517 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5474048640 con 0x7f5478066f70 2020-11-20T12:06:01.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.080+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5450056d50 msgr2=0x7f5450059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:01.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.080+0000 7f547ee3c700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5450056d50 0x7f5450059210 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f547815b740 tx=0x7f5468015000).stop 2020-11-20T12:06:01.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.080+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5478066f70 msgr2=0x7f54780b23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:01.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.080+0000 7f547ee3c700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5478066f70 0x7f54780b23f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f5474009710 tx=0x7f5474006d70).stop 2020-11-20T12:06:01.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.081+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 shutdown_connections 2020-11-20T12:06:01.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.081+0000 7f547ee3c700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5450056d50 0x7f5450059210 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:01.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.081+0000 7f547ee3c700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54781484d0 0x7f547813cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:01.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.081+0000 7f547ee3c700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5478066f70 0x7f54780b23f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:01.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.081+0000 7f547ee3c700 1 --2- 172.21.15.187:0/2604269517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54780b6650 0x7f54781331c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:01.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.081+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 >> 172.21.15.187:0/2604269517 conn(0x7f54780bab30 msgr2=0x7f54781471f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:01.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.081+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 shutdown_connections 2020-11-20T12:06:01.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.081+0000 7f547ee3c700 1 -- 172.21.15.187:0/2604269517 wait complete. 2020-11-20T12:06:01.087 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:06:01.108 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:06:01.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.350+0000 7f532359e700 1 Processor -- start 2020-11-20T12:06:01.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.351+0000 7f532359e700 1 -- start start 2020-11-20T12:06:01.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.352+0000 7f532359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53241484d0 0x7f5324061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:01.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.352+0000 7f532359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5324066f70 0x7f53240b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:01.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.352+0000 7f532359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53240b1300 con 0x7f53240622c0 2020-11-20T12:06:01.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.352+0000 7f532359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5324067730 con 0x7f53241484d0 2020-11-20T12:06:01.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.352+0000 7f532359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53240b15a0 con 0x7f5324066f70 2020-11-20T12:06:01.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.352+0000 7f532259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53241484d0 0x7f5324061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:01.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.352+0000 7f5319d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f53240622c0 0x7f53240b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38428/0 (socket says 172.21.15.187:38428) 2020-11-20T12:06:01.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.352+0000 7f5319d9b700 1 -- 172.21.15.187:0/644935410 learned_addr learned my addr 172.21.15.187:0/644935410 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:01.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.353+0000 7f5322d9d700 1 --2- 172.21.15.187:0/644935410 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5324066f70 0x7f53240b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:01.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.353+0000 7f532259c700 1 -- 172.21.15.187:0/644935410 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5324066f70 msgr2=0x7f53240b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:01.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.353+0000 7f532259c700 1 --2- 172.21.15.187:0/644935410 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5324066f70 0x7f53240b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:01.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.353+0000 7f532259c700 1 -- 172.21.15.187:0/644935410 >> v1:172.21.15.187:6789/0 conn(0x7f53240622c0 legacy=0x7f53240b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:01.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.353+0000 7f532259c700 1 -- 172.21.15.187:0/644935410 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53240b1840 con 0x7f53241484d0 2020-11-20T12:06:01.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.353+0000 7f532259c700 1 --2- 172.21.15.187:0/644935410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53241484d0 0x7f5324061dd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f5314009c90 tx=0x7f53140091d0).ready entity=mon.1 client_cookie=1aac1ce44889a490 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:01.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.354+0000 7f5321d9b700 1 -- 172.21.15.187:0/644935410 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531400bbc0 con 0x7f53241484d0 2020-11-20T12:06:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.354+0000 7f5321d9b700 1 -- 172.21.15.187:0/644935410 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531400bd20 con 0x7f53241484d0 2020-11-20T12:06:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.354+0000 7f5321d9b700 1 -- 172.21.15.187:0/644935410 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314010680 con 0x7f53241484d0 2020-11-20T12:06:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.354+0000 7f532359e700 1 -- 172.21.15.187:0/644935410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53241484d0 msgr2=0x7f5324061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.354+0000 7f532359e700 1 --2- 172.21.15.187:0/644935410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53241484d0 0x7f5324061dd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f5314009c90 tx=0x7f53140091d0).stop 2020-11-20T12:06:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.355+0000 7f532359e700 1 -- 172.21.15.187:0/644935410 shutdown_connections 2020-11-20T12:06:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.355+0000 7f532359e700 1 --2- 172.21.15.187:0/644935410 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5324066f70 0x7f53240b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.355+0000 7f532359e700 1 --2- 172.21.15.187:0/644935410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53241484d0 0x7f5324061dd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.355+0000 7f532359e700 1 -- 172.21.15.187:0/644935410 >> 172.21.15.187:0/644935410 conn(0x7f53240bab30 msgr2=0x7f53240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.355+0000 7f532359e700 1 -- 172.21.15.187:0/644935410 shutdown_connections 2020-11-20T12:06:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.355+0000 7f532359e700 1 -- 172.21.15.187:0/644935410 wait complete. 2020-11-20T12:06:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.357+0000 7f532359e700 1 Processor -- start 2020-11-20T12:06:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.357+0000 7f532359e700 1 -- start start 2020-11-20T12:06:01.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.358+0000 7f532359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53240622c0 0x7f5324065930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:01.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.358+0000 7f532359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5324066f70 0x7f5324142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:01.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.358+0000 7f532359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53241484d0 0x7f5324142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:01.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.358+0000 7f532359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532415c500 con 0x7f53240622c0 2020-11-20T12:06:01.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.358+0000 7f532359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532415c7a0 con 0x7f5324066f70 2020-11-20T12:06:01.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.358+0000 7f532359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53241595c0 con 0x7f53241484d0 2020-11-20T12:06:01.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.358+0000 7f5319d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5324066f70 0x7f5324142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:01.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5319d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5324066f70 0x7f5324142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41828/0 (socket says 172.21.15.187:41828) 2020-11-20T12:06:01.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5319d9b700 1 -- 172.21.15.187:0/955988140 learned_addr learned my addr 172.21.15.187:0/955988140 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:01.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5322d9d700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53241484d0 0x7f5324142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:01.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f532259c700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53240622c0 0x7f5324065930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:01.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5319d9b700 1 -- 172.21.15.187:0/955988140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53241484d0 msgr2=0x7f5324142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:01.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5319d9b700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53241484d0 0x7f5324142c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:01.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5319d9b700 1 -- 172.21.15.187:0/955988140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53240622c0 msgr2=0x7f5324065930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:01.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5319d9b700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53240622c0 0x7f5324065930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:01.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5319d9b700 1 -- 172.21.15.187:0/955988140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532415bbc0 con 0x7f5324066f70 2020-11-20T12:06:01.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.359+0000 7f5319d9b700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5324066f70 0x7f5324142770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f530c00c710 tx=0x7f530c007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:01.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.360+0000 7f531affd700 1 -- 172.21.15.187:0/955988140 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c007320 con 0x7f5324066f70 2020-11-20T12:06:01.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.360+0000 7f531affd700 1 -- 172.21.15.187:0/955988140 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c007480 con 0x7f5324066f70 2020-11-20T12:06:01.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.360+0000 7f531affd700 1 -- 172.21.15.187:0/955988140 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c005800 con 0x7f5324066f70 2020-11-20T12:06:01.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.360+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5324065e20 con 0x7f5324066f70 2020-11-20T12:06:01.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.360+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5324143150 con 0x7f5324066f70 2020-11-20T12:06:01.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.361+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532415c360 con 0x7f5324066f70 2020-11-20T12:06:01.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.362+0000 7f531affd700 1 -- 172.21.15.187:0/955988140 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f530c010030 con 0x7f5324066f70 2020-11-20T12:06:01.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.363+0000 7f531affd700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5300056d20 0x7f53000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:01.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.363+0000 7f531affd700 1 -- 172.21.15.187:0/955988140 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f530c06fb50 con 0x7f5324066f70 2020-11-20T12:06:01.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.363+0000 7f532259c700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5300056d20 0x7f53000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:01.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.364+0000 7f532259c700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5300056d20 0x7f53000591e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5314009210 tx=0x7f5314018000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:01.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.367+0000 7f531affd700 1 -- 172.21.15.187:0/955988140 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f532415c360 con 0x7f5324066f70 2020-11-20T12:06:01.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:01.559+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f532415c360 con 0x7f5324066f70 2020-11-20T12:06:02.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.089+0000 7f531affd700 1 -- 172.21.15.187:0/955988140 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f532415c360 con 0x7f5324066f70 2020-11-20T12:06:02.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.092+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5300056d20 msgr2=0x7f53000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:02.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.092+0000 7f532359e700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5300056d20 0x7f53000591e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5314009210 tx=0x7f5314018000).stop 2020-11-20T12:06:02.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.092+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5324066f70 msgr2=0x7f5324142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:02.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.092+0000 7f532359e700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5324066f70 0x7f5324142770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f530c00c710 tx=0x7f530c007c10).stop 2020-11-20T12:06:02.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.093+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 shutdown_connections 2020-11-20T12:06:02.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.093+0000 7f532359e700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5300056d20 0x7f53000591e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.093+0000 7f532359e700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53240622c0 0x7f5324065930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.093+0000 7f532359e700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53241484d0 0x7f5324142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.093+0000 7f532359e700 1 --2- 172.21.15.187:0/955988140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5324066f70 0x7f5324142770 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.093+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 >> 172.21.15.187:0/955988140 conn(0x7f53240bab30 msgr2=0x7f53240b9120 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:02.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.093+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 shutdown_connections 2020-11-20T12:06:02.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.094+0000 7f532359e700 1 -- 172.21.15.187:0/955988140 wait complete. 2020-11-20T12:06:02.100 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:06:02.123 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:06:02.124 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:06:02.124 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:06:02.124 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:06:02.125 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:06:02.125 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:06:02.125 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:06:02.125 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:06:02.141 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:06:02.142 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:06:02.146 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:06:02.146 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:06:02.147 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:06:02.147 DEBUG:teuthology.parallel:result is None 2020-11-20T12:06:02.195 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:06:02.195 DEBUG:teuthology.parallel:result is None 2020-11-20T12:06:02.196 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:06:02.319 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:06:02.395 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:06:02.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.415+0000 7fab991d6700 1 Processor -- start 2020-11-20T12:06:02.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.416+0000 7fab991d6700 1 -- start start 2020-11-20T12:06:02.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.417+0000 7fab991d6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fab940b6650 0x7fab940afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.417+0000 7fab991d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab94066f70 0x7fab940aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.417+0000 7fab991d6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab940b1300 con 0x7fab941484d0 2020-11-20T12:06:02.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.417+0000 7fab991d6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab94067730 con 0x7fab940b6650 2020-11-20T12:06:02.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.417+0000 7fab991d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab940b15a0 con 0x7fab94066f70 2020-11-20T12:06:02.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.417+0000 7fab93fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fab941484d0 0x7fab94061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38458/0 (socket says 172.21.15.187:38458) 2020-11-20T12:06:02.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.417+0000 7fab93fff700 1 -- 172.21.15.187:0/1777953005 learned_addr learned my addr 172.21.15.187:0/1777953005 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:02.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab989d5700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab94066f70 0x7fab940aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab8b7fe700 1 --2- 172.21.15.187:0/1777953005 >> v2:172.21.15.5:3300/0 conn(0x7fab940b6650 0x7fab940afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648038058 0 0) 0x7fab940b1300 con 0x7fab941484d0 2020-11-20T12:06:02.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab74003240 con 0x7fab941484d0 2020-11-20T12:06:02.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1024695003 0 0) 0x7fab74003240 con 0x7fab941484d0 2020-11-20T12:06:02.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab94066f70 msgr2=0x7fab940aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:02.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab937fe700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab94066f70 0x7fab940aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 >> v2:172.21.15.5:3300/0 conn(0x7fab940b6650 msgr2=0x7fab940afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:02.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.418+0000 7fab937fe700 1 --2- 172.21.15.187:0/1777953005 >> v2:172.21.15.5:3300/0 conn(0x7fab940b6650 0x7fab940afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab940b1840 con 0x7fab941484d0 2020-11-20T12:06:02.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fab840014e0 con 0x7fab941484d0 2020-11-20T12:06:02.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 >> v1:172.21.15.187:6789/0 conn(0x7fab941484d0 legacy=0x7fab94061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:02.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab74003730 0x7fab74003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab740044e0 0x7fab74004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab94066f70 0x7fab74004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94067730 con 0x7fab740044e0 2020-11-20T12:06:02.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab940b15a0 con 0x7fab74003730 2020-11-20T12:06:02.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab74004270 con 0x7fab94066f70 2020-11-20T12:06:02.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab93fff700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab74003730 0x7fab74003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab989d5700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab94066f70 0x7fab74004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab8b7fe700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab740044e0 0x7fab74004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab93fff700 1 -- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab94066f70 msgr2=0x7fab74004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:02.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab93fff700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab94066f70 0x7fab74004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab93fff700 1 -- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab740044e0 msgr2=0x7fab74004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:02.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.419+0000 7fab93fff700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab740044e0 0x7fab74004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.420+0000 7fab93fff700 1 -- 172.21.15.187:0/1777953005 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab74003240 con 0x7fab74003730 2020-11-20T12:06:02.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.420+0000 7fab93fff700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab74003730 0x7fab74003b50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fab8400c440 tx=0x7fab84007800).ready entity=mon.1 client_cookie=40d003c02e0d9e94 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:02.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.420+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab840072e0 con 0x7fab74003730 2020-11-20T12:06:02.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.420+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab84007440 con 0x7fab74003730 2020-11-20T12:06:02.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.420+0000 7fab937fe700 1 -- 172.21.15.187:0/1777953005 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84005820 con 0x7fab74003730 2020-11-20T12:06:02.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.421+0000 7fab991d6700 1 -- 172.21.15.187:0/1777953005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab74003730 msgr2=0x7fab74003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:02.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.421+0000 7fab991d6700 1 --2- 172.21.15.187:0/1777953005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab74003730 0x7fab74003b50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fab8400c440 tx=0x7fab84007800).stop 2020-11-20T12:06:02.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.421+0000 7fab989d5700 1 -- 172.21.15.187:0/1777953005 reap_dead start 2020-11-20T12:06:02.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.421+0000 7fab991d6700 1 -- 172.21.15.187:0/1777953005 shutdown_connections 2020-11-20T12:06:02.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.421+0000 7fab991d6700 1 -- 172.21.15.187:0/1777953005 >> 172.21.15.187:0/1777953005 conn(0x7fab940bab30 msgr2=0x7fab940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:02.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.421+0000 7fab991d6700 1 -- 172.21.15.187:0/1777953005 shutdown_connections 2020-11-20T12:06:02.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.421+0000 7fab991d6700 1 -- 172.21.15.187:0/1777953005 wait complete. 2020-11-20T12:06:02.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.422+0000 7fab991d6700 1 Processor -- start 2020-11-20T12:06:02.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.422+0000 7fab991d6700 1 -- start start 2020-11-20T12:06:02.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab991d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab94145d20 0x7fab940b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab991d6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab94146100 0x7fab9409ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab991d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab9409f0c0 0x7fab941440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab991d6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9409f560 con 0x7fab94146100 2020-11-20T12:06:02.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab991d6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9409f800 con 0x7fab94145d20 2020-11-20T12:06:02.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab991d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9409faa0 con 0x7fab9409f0c0 2020-11-20T12:06:02.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab93fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab94145d20 0x7fab940b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab93fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab94145d20 0x7fab940b3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41864/0 (socket says 172.21.15.187:41864) 2020-11-20T12:06:02.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.423+0000 7fab93fff700 1 -- 172.21.15.187:0/4131407549 learned_addr learned my addr 172.21.15.187:0/4131407549 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:02.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab8b7fe700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab94146100 0x7fab9409ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab93fff700 1 -- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab9409f0c0 msgr2=0x7fab941440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:02.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab989d5700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab9409f0c0 0x7fab941440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab93fff700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab9409f0c0 0x7fab941440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab93fff700 1 -- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab94146100 msgr2=0x7fab9409ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:02.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab93fff700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab94146100 0x7fab9409ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:02.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab93fff700 1 -- 172.21.15.187:0/4131407549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab941466f0 con 0x7fab94145d20 2020-11-20T12:06:02.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab989d5700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab9409f0c0 0x7fab941440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:02.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab93fff700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab94145d20 0x7fab940b3870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fab8400c410 tx=0x7fab84015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:02.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab90ff9700 1 -- 172.21.15.187:0/4131407549 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab840072e0 con 0x7fab94145d20 2020-11-20T12:06:02.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab90ff9700 1 -- 172.21.15.187:0/4131407549 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8400ceb0 con 0x7fab94145d20 2020-11-20T12:06:02.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab940a0430 con 0x7fab94145d20 2020-11-20T12:06:02.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.424+0000 7fab90ff9700 1 -- 172.21.15.187:0/4131407549 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84013870 con 0x7fab94145d20 2020-11-20T12:06:02.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.425+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab94144820 con 0x7fab94145d20 2020-11-20T12:06:02.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.426+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab94061a20 con 0x7fab94145d20 2020-11-20T12:06:02.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.426+0000 7fab90ff9700 1 -- 172.21.15.187:0/4131407549 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fab84010070 con 0x7fab94145d20 2020-11-20T12:06:02.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.427+0000 7fab90ff9700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab70056d20 0x7fab700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:02.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.427+0000 7fab90ff9700 1 -- 172.21.15.187:0/4131407549 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fab84071100 con 0x7fab94145d20 2020-11-20T12:06:02.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.427+0000 7fab8b7fe700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab70056d20 0x7fab700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:02.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.428+0000 7fab8b7fe700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab70056d20 0x7fab700591e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fab7c006a90 tx=0x7fab7c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:02.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.431+0000 7fab90ff9700 1 -- 172.21.15.187:0/4131407549 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fab84046830 con 0x7fab94145d20 2020-11-20T12:06:02.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:02.639+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fab940624e0 con 0x7fab94145d20 2020-11-20T12:06:03.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.092+0000 7fab90ff9700 1 -- 172.21.15.187:0/4131407549 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7fab84049e50 con 0x7fab94145d20 2020-11-20T12:06:03.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab70056d20 msgr2=0x7fab700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab70056d20 0x7fab700591e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fab7c006a90 tx=0x7fab7c008040).stop 2020-11-20T12:06:03.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab94145d20 msgr2=0x7fab940b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab94145d20 0x7fab940b3870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fab8400c410 tx=0x7fab84015000).stop 2020-11-20T12:06:03.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 shutdown_connections 2020-11-20T12:06:03.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab70056d20 0x7fab700591e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab9409f0c0 0x7fab941440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab94145d20 0x7fab940b3870 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 --2- 172.21.15.187:0/4131407549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab94146100 0x7fab9409ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.095+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 >> 172.21.15.187:0/4131407549 conn(0x7fab940bab30 msgr2=0x7fab94147230 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:03.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.096+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 shutdown_connections 2020-11-20T12:06:03.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.096+0000 7fab991d6700 1 -- 172.21.15.187:0/4131407549 wait complete. 2020-11-20T12:06:03.103 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:06:03.125 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:06:03.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.359+0000 7f4ae9582700 1 Processor -- start 2020-11-20T12:06:03.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.359+0000 7f4ae9582700 1 -- start start 2020-11-20T12:06:03.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.361+0000 7f4ae9582700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b1760 0x7f4ae4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.361+0000 7f4ae9582700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae4150390 con 0x7f4ae40b1380 2020-11-20T12:06:03.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.361+0000 7f4ae9582700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae4150630 con 0x7f4ae4065680 2020-11-20T12:06:03.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.361+0000 7f4ae9582700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae41511e0 con 0x7f4ae40b1760 2020-11-20T12:06:03.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.362+0000 7f4ae8d81700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4ae4065680 0x7f4ae414c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38482/0 (socket says 172.21.15.187:38482) 2020-11-20T12:06:03.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.362+0000 7f4ae37fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b1760 0x7f4ae4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.362+0000 7f4ae8d81700 1 -- 172.21.15.187:0/43870627 learned_addr learned my addr 172.21.15.187:0/43870627 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:03.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.362+0000 7f4ae2ffd700 1 -- 172.21.15.187:0/43870627 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499322504 0 0) 0x7f4ae4150630 con 0x7f4ae4065680 2020-11-20T12:06:03.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.362+0000 7f4ae2ffd700 1 -- 172.21.15.187:0/43870627 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad0003240 con 0x7f4ae4065680 2020-11-20T12:06:03.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.363+0000 7f4ae37fe700 1 -- 172.21.15.187:0/43870627 >> v1:172.21.15.187:6789/0 conn(0x7f4ae4065680 legacy=0x7f4ae414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.363+0000 7f4ae37fe700 1 -- 172.21.15.187:0/43870627 >> v1:172.21.15.5:6789/0 conn(0x7f4ae40b1380 legacy=0x7f4ae4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.363+0000 7f4ae37fe700 1 -- 172.21.15.187:0/43870627 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae4068510 con 0x7f4ae40b1760 2020-11-20T12:06:03.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.363+0000 7f4ae37fe700 1 --2- 172.21.15.187:0/43870627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b1760 0x7f4ae4065190 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4ad800b9d0 tx=0x7f4ad8006df0).ready entity=mon.2 client_cookie=900959ea6bb95c93 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:03.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.364+0000 7f4ae2ffd700 1 -- 172.21.15.187:0/43870627 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad80097b0 con 0x7f4ae40b1760 2020-11-20T12:06:03.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.364+0000 7f4ae2ffd700 1 -- 172.21.15.187:0/43870627 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad8009910 con 0x7f4ae40b1760 2020-11-20T12:06:03.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.364+0000 7f4ae2ffd700 1 -- 172.21.15.187:0/43870627 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8009c90 con 0x7f4ae40b1760 2020-11-20T12:06:03.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.364+0000 7f4ae9582700 1 -- 172.21.15.187:0/43870627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b1760 msgr2=0x7f4ae4065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.364+0000 7f4ae9582700 1 --2- 172.21.15.187:0/43870627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b1760 0x7f4ae4065190 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4ad800b9d0 tx=0x7f4ad8006df0).stop 2020-11-20T12:06:03.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.365+0000 7f4ae9582700 1 -- 172.21.15.187:0/43870627 shutdown_connections 2020-11-20T12:06:03.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.365+0000 7f4ae9582700 1 --2- 172.21.15.187:0/43870627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b1760 0x7f4ae4065190 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.365+0000 7f4ae9582700 1 -- 172.21.15.187:0/43870627 >> 172.21.15.187:0/43870627 conn(0x7f4ae40bab30 msgr2=0x7f4ae40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:03.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.365+0000 7f4ae9582700 1 -- 172.21.15.187:0/43870627 shutdown_connections 2020-11-20T12:06:03.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.365+0000 7f4ae9582700 1 -- 172.21.15.187:0/43870627 wait complete. 2020-11-20T12:06:03.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.366+0000 7f4ae9582700 1 Processor -- start 2020-11-20T12:06:03.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.366+0000 7f4ae9582700 1 -- start start 2020-11-20T12:06:03.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.367+0000 7f4ae9582700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae40b1380 0x7f4ae409ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.367+0000 7f4ae9582700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b45e0 0x7f4ae40b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.367+0000 7f4ae9582700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae40b29e0 0x7f4ae40b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.367+0000 7f4ae9582700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae4160c50 con 0x7f4ae40b29e0 2020-11-20T12:06:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.367+0000 7f4ae9582700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae409b8d0 con 0x7f4ae40b1380 2020-11-20T12:06:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.367+0000 7f4ae9582700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae409b5c0 con 0x7f4ae40b45e0 2020-11-20T12:06:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.367+0000 7f4ae3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae40b1380 0x7f4ae409ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae40b1380 0x7f4ae409ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41878/0 (socket says 172.21.15.187:41878) 2020-11-20T12:06:03.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae3fff700 1 -- 172.21.15.187:0/915609593 learned_addr learned my addr 172.21.15.187:0/915609593 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:03.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae37fe700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b45e0 0x7f4ae40b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae3fff700 1 -- 172.21.15.187:0/915609593 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b45e0 msgr2=0x7f4ae40b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae3fff700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b45e0 0x7f4ae40b24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae3fff700 1 -- 172.21.15.187:0/915609593 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae40b29e0 msgr2=0x7f4ae40b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae3fff700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae40b29e0 0x7f4ae40b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:03.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae3fff700 1 -- 172.21.15.187:0/915609593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae409d3a0 con 0x7f4ae40b1380 2020-11-20T12:06:03.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ae3fff700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae40b1380 0x7f4ae409ceb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f4ad400c710 tx=0x7f4ad4007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:03.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ac3fff700 1 -- 172.21.15.187:0/915609593 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400e560 con 0x7f4ae40b1380 2020-11-20T12:06:03.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ac3fff700 1 -- 172.21.15.187:0/915609593 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad400e6c0 con 0x7f4ae40b1380 2020-11-20T12:06:03.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.368+0000 7f4ac3fff700 1 -- 172.21.15.187:0/915609593 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400ea40 con 0x7f4ae40b1380 2020-11-20T12:06:03.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.369+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae4160820 con 0x7f4ae40b1380 2020-11-20T12:06:03.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.369+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae40b2110 con 0x7f4ae40b1380 2020-11-20T12:06:03.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.370+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae4061a20 con 0x7f4ae40b1380 2020-11-20T12:06:03.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.370+0000 7f4ac3fff700 1 -- 172.21.15.187:0/915609593 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4ad4010030 con 0x7f4ae40b1380 2020-11-20T12:06:03.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.371+0000 7f4ac3fff700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4abc056d50 0x7f4abc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.371+0000 7f4ac3fff700 1 -- 172.21.15.187:0/915609593 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4ad406fa00 con 0x7f4ae40b1380 2020-11-20T12:06:03.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.372+0000 7f4ae37fe700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4abc056d50 0x7f4abc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.372+0000 7f4ae37fe700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4abc056d50 0x7f4abc059210 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4ae4065dc0 tx=0x7f4ad8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:03.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.375+0000 7f4ac3fff700 1 -- 172.21.15.187:0/915609593 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4ad4045120 con 0x7f4ae40b1380 2020-11-20T12:06:03.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.621+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4ae40bb730 con 0x7f4ae40b1380 2020-11-20T12:06:03.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.622+0000 7f4ac3fff700 1 -- 172.21.15.187:0/915609593 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+5917 (secure 0 0 0) 0x7f4ad400edb0 con 0x7f4ae40b1380 2020-11-20T12:06:03.627 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:03.627 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:06:03.627 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:06:03.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.628 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:06:03.628 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:06:03.628 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.629 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:03.629 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.629 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.629 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.629 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.630 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:06:03.630 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:06:03.630 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.631 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:03.631 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.631 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.631 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.632 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.632 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:06:03.633 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:06:03.633 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.633 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.633 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.634 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.634 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.635 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:06:03.635 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:06:03.635 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.636 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.636 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.636 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.636 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.637 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:06:03.637 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:06:03.637 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.638 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.638 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.638 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.638 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.639 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:06:03.639 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:06:03.639 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.640 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.640 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.640 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.640 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.641 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:06:03.641 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:06:03.641 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.642 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.642 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.642 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.643 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.643 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:06:03.644 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:06:03.644 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.644 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.645 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.645 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.645 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.647 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:06:03.647 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:06:03.647 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.648 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.648 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.649 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.649 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.650 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:06:03.650 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:06:03.651 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.651 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.651 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.652 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.652 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.653 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:06:03.653 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:06:03.653 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.654 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:03.654 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:06:03.654 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:06:03.654 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:06:03.655 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.655 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:06:03.655 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:06:03.656 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.656 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:03.656 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:06:03.656 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:06:03.656 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:06:03.657 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.657 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:06:03.658 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:06:03.658 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.658 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:03.658 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:06:03.659 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:06:03.659 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:06:03.659 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.660 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:06:03.660 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:06:03.661 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.661 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:03.661 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:06:03.661 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:06:03.661 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:06:03.662 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.663 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:06:03.663 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:06:03.663 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.663 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:06:03.663 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:03.664 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:03.664 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.664 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.665 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.bootstrap-mds", 2020-11-20T12:06:03.665 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDXsLdfZkEKBBAAMUo2S+EO1j8e7onRQJ83Og==", 2020-11-20T12:06:03.665 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.665 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile bootstrap-mds" 2020-11-20T12:06:03.666 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.667 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.bootstrap-mgr", 2020-11-20T12:06:03.667 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDXsLdfFWkKBBAAJUZSuhTKjqdk5bzZKdNyYw==", 2020-11-20T12:06:03.668 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.668 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile bootstrap-mgr" 2020-11-20T12:06:03.668 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.669 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.bootstrap-osd", 2020-11-20T12:06:03.669 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDXsLdftogKBBAAB8UNK27ym/rsQBlk90Ns7w==", 2020-11-20T12:06:03.669 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.669 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile bootstrap-osd" 2020-11-20T12:06:03.670 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.670 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.bootstrap-rbd", 2020-11-20T12:06:03.671 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDXsLdftasKBBAAFvDch4+shfD52CoVUlUVRw==", 2020-11-20T12:06:03.671 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.671 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile bootstrap-rbd" 2020-11-20T12:06:03.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.672 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.bootstrap-rbd-mirror", 2020-11-20T12:06:03.673 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDXsLdfqswKBBAAuctnB+umQLSt3Yid27jN2Q==", 2020-11-20T12:06:03.673 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.673 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2020-11-20T12:06:03.674 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.678 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.bootstrap-rgw", 2020-11-20T12:06:03.678 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDXsLdfZu0KBBAAzcww7J/uLoSDp+enaHgZhQ==", 2020-11-20T12:06:03.678 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.678 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile bootstrap-rgw" 2020-11-20T12:06:03.679 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.679 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:06:03.680 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:06:03.680 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.680 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:06:03.680 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:06:03.681 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.681 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:03.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:03.682 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:06:03.682 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:06:03.682 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:03.682 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:06:03.682 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:06:03.683 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:03.683 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.683 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:03.683 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:06:03.684 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:06:03.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.624+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4abc056d50 msgr2=0x7f4abc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.624+0000 7f4ae9582700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4abc056d50 0x7f4abc059210 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4ae4065dc0 tx=0x7f4ad8015000).stop 2020-11-20T12:06:03.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae40b1380 msgr2=0x7f4ae409ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae40b1380 0x7f4ae409ceb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f4ad400c710 tx=0x7f4ad4007c00).stop 2020-11-20T12:06:03.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 shutdown_connections 2020-11-20T12:06:03.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4abc056d50 0x7f4abc059210 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae40b29e0 0x7f4ae40b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae40b1380 0x7f4ae409ceb0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 --2- 172.21.15.187:0/915609593 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ae40b45e0 0x7f4ae40b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 >> 172.21.15.187:0/915609593 conn(0x7f4ae40bab30 msgr2=0x7f4ae414f300 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 shutdown_connections 2020-11-20T12:06:03.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.625+0000 7f4ae9582700 1 -- 172.21.15.187:0/915609593 wait complete. 2020-11-20T12:06:03.688 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2020-11-20T12:06:03.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.927+0000 7fd2f28d6700 1 Processor -- start 2020-11-20T12:06:03.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.927+0000 7fd2f28d6700 1 -- start start 2020-11-20T12:06:03.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.929+0000 7fd2f28d6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd2ec0b1380 0x7fd2ec150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.929+0000 7fd2f28d6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2ec14c260 0x7fd2ec14c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.929+0000 7fd2f28d6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0658a0 con 0x7fd2ec0b1760 2020-11-20T12:06:03.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.929+0000 7fd2f28d6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec150390 con 0x7fd2ec0b1380 2020-11-20T12:06:03.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.929+0000 7fd2f28d6700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec150630 con 0x7fd2ec14c260 2020-11-20T12:06:03.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.929+0000 7fd2f20d5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2ec14c260 0x7fd2ec14c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.929+0000 7fd2f20d5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2ec14c260 0x7fd2ec14c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41890/0 (socket says 172.21.15.187:41890) 2020-11-20T12:06:03.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.929+0000 7fd2f20d5700 1 -- 172.21.15.187:0/3976090174 learned_addr learned my addr 172.21.15.187:0/3976090174 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:03.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f18d4700 1 --2- 172.21.15.187:0/3976090174 >> v2:172.21.15.5:3300/0 conn(0x7fd2ec0b1380 0x7fd2ec150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f20d5700 1 -- 172.21.15.187:0/3976090174 >> v2:172.21.15.5:3300/0 conn(0x7fd2ec0b1380 msgr2=0x7fd2ec150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:03.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f20d5700 1 --2- 172.21.15.187:0/3976090174 >> v2:172.21.15.5:3300/0 conn(0x7fd2ec0b1380 0x7fd2ec150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f20d5700 1 -- 172.21.15.187:0/3976090174 >> v1:172.21.15.5:6789/0 conn(0x7fd2ec0b1760 legacy=0x7fd2ec065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:03.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f20d5700 1 -- 172.21.15.187:0/3976090174 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec068510 con 0x7fd2ec14c260 2020-11-20T12:06:03.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f20d5700 1 --2- 172.21.15.187:0/3976090174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec14c260 0x7fd2ec14c6c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fd2ec061740 tx=0x7fd2e4009290).ready entity=mon.1 client_cookie=81bb837d747f49bf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:03.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f08d2700 1 -- 172.21.15.187:0/3976090174 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e400bb80 con 0x7fd2ec14c260 2020-11-20T12:06:03.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f08d2700 1 -- 172.21.15.187:0/3976090174 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e400bce0 con 0x7fd2ec14c260 2020-11-20T12:06:03.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.930+0000 7fd2f08d2700 1 -- 172.21.15.187:0/3976090174 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4011610 con 0x7fd2ec14c260 2020-11-20T12:06:03.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.931+0000 7fd2f28d6700 1 -- 172.21.15.187:0/3976090174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec14c260 msgr2=0x7fd2ec14c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.931+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/3976090174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec14c260 0x7fd2ec14c6c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fd2ec061740 tx=0x7fd2e4009290).stop 2020-11-20T12:06:03.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.931+0000 7fd2f28d6700 1 -- 172.21.15.187:0/3976090174 shutdown_connections 2020-11-20T12:06:03.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.931+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/3976090174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec14c260 0x7fd2ec14c6c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.931+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/3976090174 >> v2:172.21.15.5:3300/0 conn(0x7fd2ec0b1380 0x7fd2ec150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.931+0000 7fd2f28d6700 1 -- 172.21.15.187:0/3976090174 >> 172.21.15.187:0/3976090174 conn(0x7fd2ec0bab30 msgr2=0x7fd2ec0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:03.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.931+0000 7fd2f28d6700 1 -- 172.21.15.187:0/3976090174 shutdown_connections 2020-11-20T12:06:03.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.931+0000 7fd2f28d6700 1 -- 172.21.15.187:0/3976090174 wait complete. 2020-11-20T12:06:03.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.933+0000 7fd2f28d6700 1 Processor -- start 2020-11-20T12:06:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.933+0000 7fd2f28d6700 1 -- start start 2020-11-20T12:06:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f28d6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2ec0b1380 0x7fd2ec09d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f28d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec0b1760 0x7fd2ec0b9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f28d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2ec14c260 0x7fd2ec09e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f28d6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0b3af0 con 0x7fd2ec0b1380 2020-11-20T12:06:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f28d6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec09a670 con 0x7fd2ec0b1760 2020-11-20T12:06:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f28d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0b3d20 con 0x7fd2ec14c260 2020-11-20T12:06:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f10d3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec0b1760 0x7fd2ec0b9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f10d3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec0b1760 0x7fd2ec0b9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41894/0 (socket says 172.21.15.187:41894) 2020-11-20T12:06:03.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f10d3700 1 -- 172.21.15.187:0/957181792 learned_addr learned my addr 172.21.15.187:0/957181792 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:03.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.935+0000 7fd2f18d4700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2ec0b1380 0x7fd2ec09d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2f10d3700 1 -- 172.21.15.187:0/957181792 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2ec14c260 msgr2=0x7fd2ec09e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2f10d3700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2ec14c260 0x7fd2ec09e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:03.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2f10d3700 1 -- 172.21.15.187:0/957181792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2ec0b1380 msgr2=0x7fd2ec09d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:03.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2f10d3700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2ec0b1380 0x7fd2ec09d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:03.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2f10d3700 1 -- 172.21.15.187:0/957181792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec09a8d0 con 0x7fd2ec0b1760 2020-11-20T12:06:03.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2f10d3700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec0b1760 0x7fd2ec0b9ad0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fd2e000c760 tx=0x7fd2e0007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:03.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2ddffb700 1 -- 172.21.15.187:0/957181792 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e000e560 con 0x7fd2ec0b1760 2020-11-20T12:06:03.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2ddffb700 1 -- 172.21.15.187:0/957181792 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e000e6e0 con 0x7fd2ec0b1760 2020-11-20T12:06:03.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec0b9fc0 con 0x7fd2ec0b1760 2020-11-20T12:06:03.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2ddffb700 1 -- 172.21.15.187:0/957181792 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e000eaa0 con 0x7fd2ec0b1760 2020-11-20T12:06:03.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.936+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec09ef70 con 0x7fd2ec0b1760 2020-11-20T12:06:03.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.937+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec061a20 con 0x7fd2ec0b1760 2020-11-20T12:06:03.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.938+0000 7fd2ddffb700 1 -- 172.21.15.187:0/957181792 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd2e0019030 con 0x7fd2ec0b1760 2020-11-20T12:06:03.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.939+0000 7fd2ddffb700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd2c8056d50 0x7fd2c8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:03.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.939+0000 7fd2ddffb700 1 -- 172.21.15.187:0/957181792 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd2e006f240 con 0x7fd2ec0b1760 2020-11-20T12:06:03.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.939+0000 7fd2f18d4700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd2c8056d50 0x7fd2c8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:03.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.940+0000 7fd2f18d4700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd2c8056d50 0x7fd2c8059210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd2e800d990 tx=0x7fd2e8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:03.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:03.942+0000 7fd2ddffb700 1 -- 172.21.15.187:0/957181792 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd2e0043fd0 con 0x7fd2ec0b1760 2020-11-20T12:06:04.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.180+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fd2ec09aa60 con 0x7fd2ec0b1760 2020-11-20T12:06:04.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.190+0000 7fd2ddffb700 1 -- 172.21.15.187:0/957181792 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v5) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd2e0044960 con 0x7fd2ec0b1760 2020-11-20T12:06:04.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.192+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd2c8056d50 msgr2=0x7fd2c8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.192+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd2c8056d50 0x7fd2c8059210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd2e800d990 tx=0x7fd2e8008040).stop 2020-11-20T12:06:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec0b1760 msgr2=0x7fd2ec0b9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec0b1760 0x7fd2ec0b9ad0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fd2e000c760 tx=0x7fd2e0007c30).stop 2020-11-20T12:06:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 shutdown_connections 2020-11-20T12:06:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd2c8056d50 0x7fd2c8059210 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2ec0b1380 0x7fd2ec09d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2ec14c260 0x7fd2ec09e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:04.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 --2- 172.21.15.187:0/957181792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ec0b1760 0x7fd2ec0b9ad0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 >> 172.21.15.187:0/957181792 conn(0x7fd2ec0bab30 msgr2=0x7fd2ec14b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:04.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 shutdown_connections 2020-11-20T12:06:04.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.193+0000 7fd2f28d6700 1 -- 172.21.15.187:0/957181792 wait complete. 2020-11-20T12:06:04.200 INFO:teuthology.orchestra.run.smithi187.stderr:updated 2020-11-20T12:06:04.219 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2020-11-20T12:06:04.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.436+0000 7efe0d504700 1 Processor -- start 2020-11-20T12:06:04.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.436+0000 7efe0d504700 1 -- start start 2020-11-20T12:06:04.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.437+0000 7efe0d504700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efe080bd620 0x7efe081465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.437+0000 7efe0d504700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efe080b9290 0x7efe0814fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.438+0000 7efe0d504700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe08146d00 con 0x7efe080bda00 2020-11-20T12:06:04.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.438+0000 7efe0d504700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe08146ae0 con 0x7efe080b9290 2020-11-20T12:06:04.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.438+0000 7efe0d504700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe08146fa0 con 0x7efe080bd620 2020-11-20T12:06:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.438+0000 7efe07fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efe080bd620 0x7efe081465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.438+0000 7efe07fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efe080bd620 0x7efe081465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41900/0 (socket says 172.21.15.187:41900) 2020-11-20T12:06:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.438+0000 7efe07fff700 1 -- 172.21.15.187:0/4026936849 learned_addr learned my addr 172.21.15.187:0/4026936849 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.438+0000 7efe07fff700 1 -- 172.21.15.187:0/4026936849 >> v2:172.21.15.5:3300/0 conn(0x7efe080b9290 msgr2=0x7efe0814fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.438+0000 7efe07fff700 1 --2- 172.21.15.187:0/4026936849 >> v2:172.21.15.5:3300/0 conn(0x7efe080b9290 0x7efe0814fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.439+0000 7efe07fff700 1 -- 172.21.15.187:0/4026936849 >> v1:172.21.15.187:6789/0 conn(0x7efe080bda00 legacy=0x7efe08140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.439+0000 7efe07fff700 1 -- 172.21.15.187:0/4026936849 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe08147240 con 0x7efe080bd620 2020-11-20T12:06:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.439+0000 7efe07fff700 1 --2- 172.21.15.187:0/4026936849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080bd620 0x7efe081465f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7efe080ba820 tx=0x7efdf8009290).ready entity=mon.1 client_cookie=9c3d66df156686ea server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:04.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.439+0000 7efe077fe700 1 -- 172.21.15.187:0/4026936849 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf800bc50 con 0x7efe080bd620 2020-11-20T12:06:04.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.439+0000 7efe077fe700 1 -- 172.21.15.187:0/4026936849 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf800bdb0 con 0x7efe080bd620 2020-11-20T12:06:04.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.439+0000 7efe077fe700 1 -- 172.21.15.187:0/4026936849 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8011610 con 0x7efe080bd620 2020-11-20T12:06:04.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.440+0000 7efe0d504700 1 -- 172.21.15.187:0/4026936849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080bd620 msgr2=0x7efe081465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.440+0000 7efe0d504700 1 --2- 172.21.15.187:0/4026936849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080bd620 0x7efe081465f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7efe080ba820 tx=0x7efdf8009290).stop 2020-11-20T12:06:04.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.440+0000 7efe0d504700 1 -- 172.21.15.187:0/4026936849 shutdown_connections 2020-11-20T12:06:04.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.440+0000 7efe0d504700 1 --2- 172.21.15.187:0/4026936849 >> v2:172.21.15.5:3300/0 conn(0x7efe080b9290 0x7efe0814fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:04.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.440+0000 7efe0d504700 1 --2- 172.21.15.187:0/4026936849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080bd620 0x7efe081465f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.440+0000 7efe0d504700 1 -- 172.21.15.187:0/4026936849 >> 172.21.15.187:0/4026936849 conn(0x7efe08060c10 msgr2=0x7efe080bcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:04.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.440+0000 7efe0d504700 1 -- 172.21.15.187:0/4026936849 shutdown_connections 2020-11-20T12:06:04.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.440+0000 7efe0d504700 1 -- 172.21.15.187:0/4026936849 wait complete. 2020-11-20T12:06:04.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.441+0000 7efe0d504700 1 Processor -- start 2020-11-20T12:06:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.442+0000 7efe0d504700 1 -- start start 2020-11-20T12:06:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.442+0000 7efe0d504700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b9290 0x7efe0813e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.442+0000 7efe0d504700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe080bd620 0x7efe0815b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.442+0000 7efe0d504700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b4ee0 0x7efe0815bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.442+0000 7efe0d504700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe080b85d0 con 0x7efe080bd620 2020-11-20T12:06:04.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.442+0000 7efe0d504700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe080b8870 con 0x7efe080b9290 2020-11-20T12:06:04.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.442+0000 7efe0d504700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe080b8b10 con 0x7efe080b4ee0 2020-11-20T12:06:04.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe07fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b9290 0x7efe0813e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe07fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b9290 0x7efe0813e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41906/0 (socket says 172.21.15.187:41906) 2020-11-20T12:06:04.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe07fff700 1 -- 172.21.15.187:0/2486811222 learned_addr learned my addr 172.21.15.187:0/2486811222 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:04.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe0cd03700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b4ee0 0x7efe0815bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe07fff700 1 -- 172.21.15.187:0/2486811222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b4ee0 msgr2=0x7efe0815bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe07fff700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b4ee0 0x7efe0815bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe07fff700 1 -- 172.21.15.187:0/2486811222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe080bd620 msgr2=0x7efe0815b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efdff7fe700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe080bd620 0x7efe0815b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe07fff700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe080bd620 0x7efe0815b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.443+0000 7efe07fff700 1 -- 172.21.15.187:0/2486811222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe080b8280 con 0x7efe080b9290 2020-11-20T12:06:04.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.444+0000 7efe07fff700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b9290 0x7efe0813e1e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7efe081470e0 tx=0x7efdf8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:04.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.444+0000 7efe04ff9700 1 -- 172.21.15.187:0/2486811222 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf800bc50 con 0x7efe080b9290 2020-11-20T12:06:04.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.444+0000 7efe04ff9700 1 -- 172.21.15.187:0/2486811222 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf8010c40 con 0x7efe080b9290 2020-11-20T12:06:04.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.444+0000 7efe04ff9700 1 -- 172.21.15.187:0/2486811222 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf801c470 con 0x7efe080b9290 2020-11-20T12:06:04.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.444+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0815c0e0 con 0x7efe080b9290 2020-11-20T12:06:04.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.444+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0813e6d0 con 0x7efe080b9290 2020-11-20T12:06:04.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.445+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe080b8cd0 con 0x7efe080b9290 2020-11-20T12:06:04.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.446+0000 7efe04ff9700 1 -- 172.21.15.187:0/2486811222 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efdf8017070 con 0x7efe080b9290 2020-11-20T12:06:04.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.446+0000 7efe04ff9700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.446+0000 7efe04ff9700 1 -- 172.21.15.187:0/2486811222 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efdf8071fd0 con 0x7efe080b9290 2020-11-20T12:06:04.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.447+0000 7efdff7fe700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.447+0000 7efdff7fe700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7efe080b6020 tx=0x7efdf0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:04.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.450+0000 7efe04ff9700 1 -- 172.21.15.187:0/2486811222 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efdf8047750 con 0x7efe080b9290 2020-11-20T12:06:04.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.683+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7efe0809d720 con 0x7efe080b9290 2020-11-20T12:06:04.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.693+0000 7efe04ff9700 1 -- 172.21.15.187:0/2486811222 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v6) v1 ==== 97+0+0 (secure 0 0 0) 0x7efdf8015070 con 0x7efe080b9290 2020-11-20T12:06:04.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.695+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 msgr2=0x7efde4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.695+0000 7efe0d504700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7efe080b6020 tx=0x7efdf0009040).stop 2020-11-20T12:06:04.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.695+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b9290 msgr2=0x7efe0813e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.695+0000 7efe0d504700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b9290 0x7efe0813e1e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7efe081470e0 tx=0x7efdf8016000).stop 2020-11-20T12:06:04.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.695+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 shutdown_connections 2020-11-20T12:06:04.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.695+0000 7efe0d504700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.695+0000 7efe0d504700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b4ee0 0x7efe0815bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.696+0000 7efe0d504700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b9290 0x7efe0813e1e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.696+0000 7efe0d504700 1 --2- 172.21.15.187:0/2486811222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe080bd620 0x7efe0815b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.696+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 >> 172.21.15.187:0/2486811222 conn(0x7efe08060c10 msgr2=0x7efe08144ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:04.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.696+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 shutdown_connections 2020-11-20T12:06:04.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.696+0000 7efe0d504700 1 -- 172.21.15.187:0/2486811222 wait complete. 2020-11-20T12:06:04.703 INFO:teuthology.orchestra.run.smithi187.stderr:updated 2020-11-20T12:06:04.721 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2020-11-20T12:06:04.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.957+0000 7f074bfff700 1 Processor -- start 2020-11-20T12:06:04.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.957+0000 7f074bfff700 1 -- start start 2020-11-20T12:06:04.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f074c0b57b0 0x7f074c068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f074c1482d0 0x7f074c062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 0x7f074c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f074c144b80 con 0x7f074c1482d0 2020-11-20T12:06:04.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f074c1489e0 con 0x7f074c0b57b0 2020-11-20T12:06:04.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c148c80 con 0x7f074c062b50 2020-11-20T12:06:04.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f074c0b57b0 0x7f074c068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f074c0b57b0 0x7f074c068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41914/0 (socket says 172.21.15.187:41914) 2020-11-20T12:06:04.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.959+0000 7f074affd700 1 -- 172.21.15.187:0/1184874647 learned_addr learned my addr 172.21.15.187:0/1184874647 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:04.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.960+0000 7f074b7fe700 1 --2- 172.21.15.187:0/1184874647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 0x7f074c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.960+0000 7f074a7fc700 1 --2- 172.21.15.187:0/1184874647 >> v2:172.21.15.5:3300/0 conn(0x7f074c1482d0 0x7f074c062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.960+0000 7f074affd700 1 -- 172.21.15.187:0/1184874647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 msgr2=0x7f074c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:04.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.960+0000 7f074affd700 1 --2- 172.21.15.187:0/1184874647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 0x7f074c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.960+0000 7f074affd700 1 -- 172.21.15.187:0/1184874647 >> v2:172.21.15.5:3300/0 conn(0x7f074c1482d0 msgr2=0x7f074c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:04.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.960+0000 7f074affd700 1 --2- 172.21.15.187:0/1184874647 >> v2:172.21.15.5:3300/0 conn(0x7f074c1482d0 0x7f074c062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.960+0000 7f074affd700 1 -- 172.21.15.187:0/1184874647 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c148f20 con 0x7f074c0b57b0 2020-11-20T12:06:04.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.960+0000 7f074a7fc700 1 --2- 172.21.15.187:0/1184874647 >> v2:172.21.15.5:3300/0 conn(0x7f074c1482d0 0x7f074c062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:06:04.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.961+0000 7f074affd700 1 --2- 172.21.15.187:0/1184874647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c0b57b0 0x7f074c068230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f074c0b85e0 tx=0x7f073c006eb0).ready entity=mon.1 client_cookie=c7547d69da9d5ef server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:04.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.961+0000 7f0749ffb700 1 -- 172.21.15.187:0/1184874647 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c0098d0 con 0x7f074c0b57b0 2020-11-20T12:06:04.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.961+0000 7f0749ffb700 1 -- 172.21.15.187:0/1184874647 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c009a30 con 0x7f074c0b57b0 2020-11-20T12:06:04.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.961+0000 7f0749ffb700 1 -- 172.21.15.187:0/1184874647 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c009db0 con 0x7f074c0b57b0 2020-11-20T12:06:04.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 -- 172.21.15.187:0/1184874647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c0b57b0 msgr2=0x7f074c068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 --2- 172.21.15.187:0/1184874647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c0b57b0 0x7f074c068230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f074c0b85e0 tx=0x7f073c006eb0).stop 2020-11-20T12:06:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 -- 172.21.15.187:0/1184874647 shutdown_connections 2020-11-20T12:06:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 --2- 172.21.15.187:0/1184874647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 0x7f074c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 --2- 172.21.15.187:0/1184874647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c0b57b0 0x7f074c068230 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 --2- 172.21.15.187:0/1184874647 >> v2:172.21.15.5:3300/0 conn(0x7f074c1482d0 0x7f074c062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 -- 172.21.15.187:0/1184874647 >> 172.21.15.187:0/1184874647 conn(0x7f074c0bab30 msgr2=0x7f074c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 -- 172.21.15.187:0/1184874647 shutdown_connections 2020-11-20T12:06:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.962+0000 7f074bfff700 1 -- 172.21.15.187:0/1184874647 wait complete. 2020-11-20T12:06:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.964+0000 7f074bfff700 1 Processor -- start 2020-11-20T12:06:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.964+0000 7f074bfff700 1 -- start start 2020-11-20T12:06:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.965+0000 7f074bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 0x7f074c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.965+0000 7f074bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c1482d0 0x7f074c0b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.965+0000 7f074bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f074c0b3060 0x7f074c145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.965+0000 7f074bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c159000 con 0x7f074c0b3060 2020-11-20T12:06:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.965+0000 7f074bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0b2650 con 0x7f074c1482d0 2020-11-20T12:06:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.965+0000 7f074bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0b34e0 con 0x7f074c062b50 2020-11-20T12:06:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c1482d0 0x7f074c0b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c1482d0 0x7f074c0b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41922/0 (socket says 172.21.15.187:41922) 2020-11-20T12:06:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074a7fc700 1 -- 172.21.15.187:0/4023373596 learned_addr learned my addr 172.21.15.187:0/4023373596 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074a7fc700 1 -- 172.21.15.187:0/4023373596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 msgr2=0x7f074c0b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074a7fc700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 0x7f074c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074b7fe700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f074c0b3060 0x7f074c145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074a7fc700 1 -- 172.21.15.187:0/4023373596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f074c0b3060 msgr2=0x7f074c145fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074a7fc700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f074c0b3060 0x7f074c145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.966+0000 7f074a7fc700 1 -- 172.21.15.187:0/4023373596 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c159260 con 0x7f074c1482d0 2020-11-20T12:06:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.967+0000 7f074a7fc700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c1482d0 0x7f074c0b2b70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f074000db60 tx=0x7f0740008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.967+0000 7f0732ffd700 1 -- 172.21.15.187:0/4023373596 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740009790 con 0x7f074c1482d0 2020-11-20T12:06:04.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.967+0000 7f0732ffd700 1 -- 172.21.15.187:0/4023373596 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07400098f0 con 0x7f074c1482d0 2020-11-20T12:06:04.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.967+0000 7f0732ffd700 1 -- 172.21.15.187:0/4023373596 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740009cb0 con 0x7f074c1482d0 2020-11-20T12:06:04.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.967+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074c159500 con 0x7f074c1482d0 2020-11-20T12:06:04.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.967+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074c0b1d40 con 0x7f074c1482d0 2020-11-20T12:06:04.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.968+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c0682e0 con 0x7f074c1482d0 2020-11-20T12:06:04.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.969+0000 7f0732ffd700 1 -- 172.21.15.187:0/4023373596 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0740013690 con 0x7f074c1482d0 2020-11-20T12:06:04.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.969+0000 7f0732ffd700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0728056d20 0x7f07280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:04.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.970+0000 7f0732ffd700 1 -- 172.21.15.187:0/4023373596 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f074006ffd0 con 0x7f074c1482d0 2020-11-20T12:06:04.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.970+0000 7f074affd700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0728056d20 0x7f07280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:04.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.970+0000 7f074affd700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0728056d20 0x7f07280591e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f074c063890 tx=0x7f073c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:04.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:04.973+0000 7f0732ffd700 1 -- 172.21.15.187:0/4023373596 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f07400456f0 con 0x7f074c1482d0 2020-11-20T12:06:05.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.224+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f074c145cd0 con 0x7f074c1482d0 2020-11-20T12:06:05.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.234+0000 7f0732ffd700 1 -- 172.21.15.187:0/4023373596 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0740048d10 con 0x7f074c1482d0 2020-11-20T12:06:05.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0728056d20 msgr2=0x7f07280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:05.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0728056d20 0x7f07280591e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f074c063890 tx=0x7f073c015000).stop 2020-11-20T12:06:05.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c1482d0 msgr2=0x7f074c0b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:05.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c1482d0 0x7f074c0b2b70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f074000db60 tx=0x7f0740008040).stop 2020-11-20T12:06:05.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 shutdown_connections 2020-11-20T12:06:05.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0728056d20 0x7f07280591e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f074c0b3060 0x7f074c145fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f074c062b50 0x7f074c0b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:05.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 --2- 172.21.15.187:0/4023373596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f074c1482d0 0x7f074c0b2b70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 >> 172.21.15.187:0/4023373596 conn(0x7f074c0bab30 msgr2=0x7f074c0b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:05.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 shutdown_connections 2020-11-20T12:06:05.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.237+0000 7f074bfff700 1 -- 172.21.15.187:0/4023373596 wait complete. 2020-11-20T12:06:05.244 INFO:teuthology.orchestra.run.smithi187.stderr:updated 2020-11-20T12:06:05.263 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2020-11-20T12:06:05.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.497+0000 7fc45f59e700 1 Processor -- start 2020-11-20T12:06:05.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.497+0000 7fc45f59e700 1 -- start start 2020-11-20T12:06:05.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.498+0000 7fc45f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bda00 0x7fc4601465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:05.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.499+0000 7fc45f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc4600b9290 0x7fc46014fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:05.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.499+0000 7fc45f59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc460146d00 con 0x7fc4600bd620 2020-11-20T12:06:05.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.499+0000 7fc45f59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc460146ae0 con 0x7fc4600b9290 2020-11-20T12:06:05.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.499+0000 7fc45f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc460146fa0 con 0x7fc4600bda00 2020-11-20T12:06:05.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.499+0000 7fc45e59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc4600bd620 0x7fc460140330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38534/0 (socket says 172.21.15.187:38534) 2020-11-20T12:06:05.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.499+0000 7fc45e59c700 1 -- 172.21.15.187:0/2627713918 learned_addr learned my addr 172.21.15.187:0/2627713918 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:05.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.499+0000 7fc45ed9d700 1 --2- 172.21.15.187:0/2627713918 >> v2:172.21.15.187:3300/0 conn(0x7fc4600b9290 0x7fc46014fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:05.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.499+0000 7fc457fff700 1 --2- 172.21.15.187:0/2627713918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bda00 0x7fc4601465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:05.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.500+0000 7fc45dd9b700 1 -- 172.21.15.187:0/2627713918 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008782099 0 0) 0x7fc460146d00 con 0x7fc4600bd620 2020-11-20T12:06:05.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.500+0000 7fc45dd9b700 1 -- 172.21.15.187:0/2627713918 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc440003240 con 0x7fc4600bd620 2020-11-20T12:06:05.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.501+0000 7fc45ed9d700 1 -- 172.21.15.187:0/2627713918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bda00 msgr2=0x7fc4601465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:05.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.501+0000 7fc45ed9d700 1 --2- 172.21.15.187:0/2627713918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bda00 0x7fc4601465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.501+0000 7fc45ed9d700 1 -- 172.21.15.187:0/2627713918 >> v1:172.21.15.187:6789/0 conn(0x7fc4600bd620 legacy=0x7fc460140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:05.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.501+0000 7fc45ed9d700 1 -- 172.21.15.187:0/2627713918 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460147240 con 0x7fc4600b9290 2020-11-20T12:06:05.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.501+0000 7fc45ed9d700 1 --2- 172.21.15.187:0/2627713918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc46014fbf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc458009c90 tx=0x7fc4580091d0).ready entity=mon.1 client_cookie=b3f0b797318a5e0f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:05.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45dd9b700 1 -- 172.21.15.187:0/2627713918 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45800ba50 con 0x7fc4600b9290 2020-11-20T12:06:05.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45dd9b700 1 -- 172.21.15.187:0/2627713918 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45800bbb0 con 0x7fc4600b9290 2020-11-20T12:06:05.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45dd9b700 1 -- 172.21.15.187:0/2627713918 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458011400 con 0x7fc4600b9290 2020-11-20T12:06:05.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45f59e700 1 -- 172.21.15.187:0/2627713918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 msgr2=0x7fc46014fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:05.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45f59e700 1 --2- 172.21.15.187:0/2627713918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc46014fbf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc458009c90 tx=0x7fc4580091d0).stop 2020-11-20T12:06:05.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45f59e700 1 -- 172.21.15.187:0/2627713918 shutdown_connections 2020-11-20T12:06:05.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45f59e700 1 --2- 172.21.15.187:0/2627713918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc46014fbf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45f59e700 1 --2- 172.21.15.187:0/2627713918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bda00 0x7fc4601465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.502+0000 7fc45f59e700 1 -- 172.21.15.187:0/2627713918 >> 172.21.15.187:0/2627713918 conn(0x7fc460060c10 msgr2=0x7fc4600bcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:05.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.503+0000 7fc45f59e700 1 -- 172.21.15.187:0/2627713918 shutdown_connections 2020-11-20T12:06:05.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.503+0000 7fc45f59e700 1 -- 172.21.15.187:0/2627713918 wait complete. 2020-11-20T12:06:05.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.505+0000 7fc45f59e700 1 Processor -- start 2020-11-20T12:06:05.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.505+0000 7fc45f59e700 1 -- start start 2020-11-20T12:06:05.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.506+0000 7fc45f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc4601583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:05.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.506+0000 7fc45f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bd620 0x7fc460145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:05.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.506+0000 7fc45f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4601588a0 0x7fc4600b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:05.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.506+0000 7fc45f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc460157ce0 con 0x7fc4601588a0 2020-11-20T12:06:05.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.506+0000 7fc45f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc460157f10 con 0x7fc4600b9290 2020-11-20T12:06:05.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.506+0000 7fc45f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600b2800 con 0x7fc4600bd620 2020-11-20T12:06:05.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.506+0000 7fc45e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc4601583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:05.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc4601583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41934/0 (socket says 172.21.15.187:41934) 2020-11-20T12:06:05.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45e59c700 1 -- 172.21.15.187:0/291096730 learned_addr learned my addr 172.21.15.187:0/291096730 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:05.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc457fff700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bd620 0x7fc460145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:05.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45ed9d700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4601588a0 0x7fc4600b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:05.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45e59c700 1 -- 172.21.15.187:0/291096730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bd620 msgr2=0x7fc460145140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:05.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45e59c700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bd620 0x7fc460145140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45e59c700 1 -- 172.21.15.187:0/291096730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4601588a0 msgr2=0x7fc4600b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:05.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45e59c700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4601588a0 0x7fc4600b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45e59c700 1 -- 172.21.15.187:0/291096730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460061ba0 con 0x7fc4600b9290 2020-11-20T12:06:05.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.507+0000 7fc45e59c700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc4601583b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc45000c6f0 tx=0x7fc450007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:05.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.508+0000 7fc4567fc700 1 -- 172.21.15.187:0/291096730 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450007320 con 0x7fc4600b9290 2020-11-20T12:06:05.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.508+0000 7fc4567fc700 1 -- 172.21.15.187:0/291096730 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450007480 con 0x7fc4600b9290 2020-11-20T12:06:05.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.508+0000 7fc4567fc700 1 -- 172.21.15.187:0/291096730 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450005800 con 0x7fc4600b9290 2020-11-20T12:06:05.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.508+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4600b23d0 con 0x7fc4600b9290 2020-11-20T12:06:05.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.508+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc460145870 con 0x7fc4600b9290 2020-11-20T12:06:05.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.509+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4600b25f0 con 0x7fc4600b9290 2020-11-20T12:06:05.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.510+0000 7fc4567fc700 1 -- 172.21.15.187:0/291096730 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc450010030 con 0x7fc4600b9290 2020-11-20T12:06:05.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.510+0000 7fc4567fc700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc43c056d20 0x7fc43c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:05.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.510+0000 7fc4567fc700 1 -- 172.21.15.187:0/291096730 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc45006fa90 con 0x7fc4600b9290 2020-11-20T12:06:05.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.511+0000 7fc457fff700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc43c056d20 0x7fc43c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:05.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.511+0000 7fc457fff700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc43c056d20 0x7fc43c0591e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc4600bd430 tx=0x7fc448008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:05.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.514+0000 7fc4567fc700 1 -- 172.21.15.187:0/291096730 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc450045130 con 0x7fc4600b9290 2020-11-20T12:06:05.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.766+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fc46009d720 con 0x7fc4600b9290 2020-11-20T12:06:05.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.778+0000 7fc4567fc700 1 -- 172.21.15.187:0/291096730 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc450048750 con 0x7fc4600b9290 2020-11-20T12:06:05.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc43c056d20 msgr2=0x7fc43c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:05.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc43c056d20 0x7fc43c0591e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc4600bd430 tx=0x7fc448008040).stop 2020-11-20T12:06:05.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 msgr2=0x7fc4601583b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:05.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc4601583b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc45000c6f0 tx=0x7fc450007ad0).stop 2020-11-20T12:06:05.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 shutdown_connections 2020-11-20T12:06:05.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc43c056d20 0x7fc43c0591e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4601588a0 0x7fc4600b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4600b9290 0x7fc4601583b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 --2- 172.21.15.187:0/291096730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4600bd620 0x7fc460145140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:05.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.780+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 >> 172.21.15.187:0/291096730 conn(0x7fc460060c10 msgr2=0x7fc4600b9770 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:05.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.781+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 shutdown_connections 2020-11-20T12:06:05.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:05.781+0000 7fc45f59e700 1 -- 172.21.15.187:0/291096730 wait complete. 2020-11-20T12:06:05.788 INFO:teuthology.orchestra.run.smithi187.stderr:updated 2020-11-20T12:06:05.808 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2020-11-20T12:06:06.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.048+0000 7f533b485700 1 Processor -- start 2020-11-20T12:06:06.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.048+0000 7f533b485700 1 -- start start 2020-11-20T12:06:06.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.050+0000 7f533b485700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53341484d0 0x7f5334061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.050+0000 7f533b485700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 0x7f53340b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.050+0000 7f533b485700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53340b1300 con 0x7f5334066f70 2020-11-20T12:06:06.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.050+0000 7f533b485700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5334067730 con 0x7f53341484d0 2020-11-20T12:06:06.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.050+0000 7f533b485700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53340b15a0 con 0x7f53340622c0 2020-11-20T12:06:06.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.050+0000 7f533ac84700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5334066f70 0x7f53340b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38552/0 (socket says 172.21.15.187:38552) 2020-11-20T12:06:06.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.050+0000 7f533ac84700 1 -- 172.21.15.187:0/537765944 learned_addr learned my addr 172.21.15.187:0/537765944 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:06.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.050+0000 7f533a483700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53341484d0 0x7f5334061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.051+0000 7f5339c82700 1 --2- 172.21.15.187:0/537765944 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 0x7f53340b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.051+0000 7f5339481700 1 -- 172.21.15.187:0/537765944 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148220583 0 0) 0x7f53340b1300 con 0x7f5334066f70 2020-11-20T12:06:06.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.051+0000 7f5339481700 1 -- 172.21.15.187:0/537765944 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531c003240 con 0x7f5334066f70 2020-11-20T12:06:06.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.051+0000 7f533a483700 1 -- 172.21.15.187:0/537765944 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 msgr2=0x7f53340b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:06.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.051+0000 7f533a483700 1 --2- 172.21.15.187:0/537765944 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 0x7f53340b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.051+0000 7f533a483700 1 -- 172.21.15.187:0/537765944 >> v1:172.21.15.187:6789/0 conn(0x7f5334066f70 legacy=0x7f53340b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.051+0000 7f533a483700 1 -- 172.21.15.187:0/537765944 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53340b1840 con 0x7f53341484d0 2020-11-20T12:06:06.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f533a483700 1 --2- 172.21.15.187:0/537765944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53341484d0 0x7f5334061dd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f5330009c90 tx=0x7f53300091d0).ready entity=mon.1 client_cookie=8d80fc88688535a6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:06.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f5339481700 1 -- 172.21.15.187:0/537765944 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533000ba70 con 0x7f53341484d0 2020-11-20T12:06:06.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f5339481700 1 -- 172.21.15.187:0/537765944 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533000bbd0 con 0x7f53341484d0 2020-11-20T12:06:06.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f5339481700 1 -- 172.21.15.187:0/537765944 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330011400 con 0x7f53341484d0 2020-11-20T12:06:06.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f533b485700 1 -- 172.21.15.187:0/537765944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53341484d0 msgr2=0x7f5334061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f533b485700 1 --2- 172.21.15.187:0/537765944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53341484d0 0x7f5334061dd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f5330009c90 tx=0x7f53300091d0).stop 2020-11-20T12:06:06.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f533b485700 1 -- 172.21.15.187:0/537765944 shutdown_connections 2020-11-20T12:06:06.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f533b485700 1 --2- 172.21.15.187:0/537765944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53341484d0 0x7f5334061dd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f533b485700 1 --2- 172.21.15.187:0/537765944 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 0x7f53340b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.052+0000 7f533b485700 1 -- 172.21.15.187:0/537765944 >> 172.21.15.187:0/537765944 conn(0x7f53340bab30 msgr2=0x7f53340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:06.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.053+0000 7f533b485700 1 -- 172.21.15.187:0/537765944 shutdown_connections 2020-11-20T12:06:06.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.053+0000 7f533b485700 1 -- 172.21.15.187:0/537765944 wait complete. 2020-11-20T12:06:06.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.054+0000 7f533b485700 1 Processor -- start 2020-11-20T12:06:06.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.054+0000 7f533b485700 1 -- start start 2020-11-20T12:06:06.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f533b485700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 0x7f533415c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f533b485700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5334066f70 0x7f53340b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f533b485700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53341484d0 0x7f533409bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f533b485700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533409b770 con 0x7f53341484d0 2020-11-20T12:06:06.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f533b485700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533409d170 con 0x7f5334066f70 2020-11-20T12:06:06.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f533b485700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533409d410 con 0x7f53340622c0 2020-11-20T12:06:06.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f5339c82700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5334066f70 0x7f53340b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f5339c82700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5334066f70 0x7f53340b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41950/0 (socket says 172.21.15.187:41950) 2020-11-20T12:06:06.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.055+0000 7f5339c82700 1 -- 172.21.15.187:0/1772925941 learned_addr learned my addr 172.21.15.187:0/1772925941 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:06.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f533a483700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 0x7f533415c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f5339c82700 1 -- 172.21.15.187:0/1772925941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 msgr2=0x7f533415c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f533ac84700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53341484d0 0x7f533409bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f5339c82700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 0x7f533415c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f5339c82700 1 -- 172.21.15.187:0/1772925941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53341484d0 msgr2=0x7f533409bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f5339c82700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53341484d0 0x7f533409bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f5339c82700 1 -- 172.21.15.187:0/1772925941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533409ce60 con 0x7f5334066f70 2020-11-20T12:06:06.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f5339c82700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5334066f70 0x7f53340b3c10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f532400db90 tx=0x7f5324008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:06.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f53227fc700 1 -- 172.21.15.187:0/1772925941 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324009820 con 0x7f5334066f70 2020-11-20T12:06:06.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.056+0000 7f53227fc700 1 -- 172.21.15.187:0/1772925941 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5324009980 con 0x7f5334066f70 2020-11-20T12:06:06.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.057+0000 7f53227fc700 1 -- 172.21.15.187:0/1772925941 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324009d40 con 0x7f5334066f70 2020-11-20T12:06:06.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.057+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53340b4100 con 0x7f5334066f70 2020-11-20T12:06:06.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.057+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533409f580 con 0x7f5334066f70 2020-11-20T12:06:06.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.058+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5334061a20 con 0x7f5334066f70 2020-11-20T12:06:06.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.058+0000 7f53227fc700 1 -- 172.21.15.187:0/1772925941 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5324012630 con 0x7f5334066f70 2020-11-20T12:06:06.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.059+0000 7f53227fc700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f530c056d20 0x7f530c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.059+0000 7f53227fc700 1 -- 172.21.15.187:0/1772925941 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5324070020 con 0x7f5334066f70 2020-11-20T12:06:06.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.060+0000 7f533a483700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f530c056d20 0x7f530c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.060+0000 7f533a483700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f530c056d20 0x7f530c0591e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f5330009440 tx=0x7f5330016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:06.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.063+0000 7f53227fc700 1 -- 172.21.15.187:0/1772925941 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f53240457a0 con 0x7f5334066f70 2020-11-20T12:06:06.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.316+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f53341488b0 con 0x7f5334066f70 2020-11-20T12:06:06.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.326+0000 7f53227fc700 1 -- 172.21.15.187:0/1772925941 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5324048dc0 con 0x7f5334066f70 2020-11-20T12:06:06.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f530c056d20 msgr2=0x7f530c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f530c056d20 0x7f530c0591e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f5330009440 tx=0x7f5330016000).stop 2020-11-20T12:06:06.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5334066f70 msgr2=0x7f53340b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5334066f70 0x7f53340b3c10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f532400db90 tx=0x7f5324008040).stop 2020-11-20T12:06:06.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 shutdown_connections 2020-11-20T12:06:06.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f530c056d20 0x7f530c0591e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53341484d0 0x7f533409bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53340622c0 0x7f533415c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 --2- 172.21.15.187:0/1772925941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5334066f70 0x7f53340b3c10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 >> 172.21.15.187:0/1772925941 conn(0x7f53340bab30 msgr2=0x7f53340b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:06.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.329+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 shutdown_connections 2020-11-20T12:06:06.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.330+0000 7f533b485700 1 -- 172.21.15.187:0/1772925941 wait complete. 2020-11-20T12:06:06.338 INFO:teuthology.orchestra.run.smithi187.stderr:updated 2020-11-20T12:06:06.358 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2020-11-20T12:06:06.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.601+0000 7ff0765d6700 1 Processor -- start 2020-11-20T12:06:06.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.601+0000 7ff0765d6700 1 -- start start 2020-11-20T12:06:06.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.603+0000 7ff0765d6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff070065190 0x7ff070065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.603+0000 7ff0765d6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0700b13e0 con 0x7ff070065f90 2020-11-20T12:06:06.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.603+0000 7ff0765d6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff070062c10 con 0x7ff07009ca90 2020-11-20T12:06:06.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.603+0000 7ff0765d6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07014ab30 con 0x7ff070065190 2020-11-20T12:06:06.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.603+0000 7ff075dd5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff07009ca90 0x7ff07009cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38564/0 (socket says 172.21.15.187:38564) 2020-11-20T12:06:06.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.603+0000 7ff075dd5700 1 -- 172.21.15.187:0/1195586682 learned_addr learned my addr 172.21.15.187:0/1195586682 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:06.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff0755d4700 1 --2- 172.21.15.187:0/1195586682 >> v2:172.21.15.5:3300/0 conn(0x7ff070065190 0x7ff070065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945806022 0 0) 0x7ff070062c10 con 0x7ff07009ca90 2020-11-20T12:06:06.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff058003240 con 0x7ff07009ca90 2020-11-20T12:06:06.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1362170788 0 0) 0x7ff058003240 con 0x7ff07009ca90 2020-11-20T12:06:06.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 >> v2:172.21.15.5:3300/0 conn(0x7ff070065190 msgr2=0x7ff070065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:06.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 --2- 172.21.15.187:0/1195586682 >> v2:172.21.15.5:3300/0 conn(0x7ff070065190 0x7ff070065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 >> v1:172.21.15.5:6789/0 conn(0x7ff070065f90 legacy=0x7ff070143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:06.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700b1640 con 0x7ff07009ca90 2020-11-20T12:06:06.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff068001860 con 0x7ff07009ca90 2020-11-20T12:06:06.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 >> v1:172.21.15.187:6789/0 conn(0x7ff07009ca90 legacy=0x7ff07009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff058003800 0x7ff058003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.604+0000 7ff05ffff700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0580044e0 0x7ff058004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff05ffff700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff05800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07014ab30 con 0x7ff0580044e0 2020-11-20T12:06:06.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058005300 con 0x7ff058004e50 2020-11-20T12:06:06.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05800bc20 con 0x7ff058003800 2020-11-20T12:06:06.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff075dd5700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff05800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff0755d4700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff058003800 0x7ff058003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff075dd5700 1 -- 172.21.15.187:0/1195586682 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff058003800 msgr2=0x7ff058003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:06.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff075dd5700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff058003800 0x7ff058003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff075dd5700 1 -- 172.21.15.187:0/1195586682 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0580044e0 msgr2=0x7ff058004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:06.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff075dd5700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0580044e0 0x7ff058004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:06.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.605+0000 7ff075dd5700 1 -- 172.21.15.187:0/1195586682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058003240 con 0x7ff058004e50 2020-11-20T12:06:06.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.606+0000 7ff075dd5700 1 -- 172.21.15.187:0/1195586682 reap_dead start 2020-11-20T12:06:06.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.606+0000 7ff075dd5700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff05800b730 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7ff06800c590 tx=0x7ff0700b0fb0).ready entity=mon.1 client_cookie=91b4a0028c19e939 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:06.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.606+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068005b80 con 0x7ff058004e50 2020-11-20T12:06:06.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.606+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068005ce0 con 0x7ff058004e50 2020-11-20T12:06:06.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.606+0000 7ff05ffff700 1 -- 172.21.15.187:0/1195586682 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068007320 con 0x7ff058004e50 2020-11-20T12:06:06.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.606+0000 7ff0765d6700 1 -- 172.21.15.187:0/1195586682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 msgr2=0x7ff05800b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.606+0000 7ff0765d6700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff05800b730 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7ff06800c590 tx=0x7ff0700b0fb0).stop 2020-11-20T12:06:06.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.607+0000 7ff0765d6700 1 -- 172.21.15.187:0/1195586682 shutdown_connections 2020-11-20T12:06:06.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.607+0000 7ff0765d6700 1 --2- 172.21.15.187:0/1195586682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff05800b730 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.607+0000 7ff0765d6700 1 -- 172.21.15.187:0/1195586682 >> 172.21.15.187:0/1195586682 conn(0x7ff0700bab30 msgr2=0x7ff0700baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:06.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.607+0000 7ff0765d6700 1 -- 172.21.15.187:0/1195586682 shutdown_connections 2020-11-20T12:06:06.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.607+0000 7ff0765d6700 1 -- 172.21.15.187:0/1195586682 wait complete. 2020-11-20T12:06:06.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.608+0000 7ff0765d6700 1 Processor -- start 2020-11-20T12:06:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.608+0000 7ff0765d6700 1 -- start start 2020-11-20T12:06:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.609+0000 7ff0765d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff07015b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.609+0000 7ff0765d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff070067da0 0x7ff0700b7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.609+0000 7ff0765d6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0701449c0 0x7ff070144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.609+0000 7ff0765d6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07015bc60 con 0x7ff0701449c0 2020-11-20T12:06:06.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.609+0000 7ff0765d6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07015bf00 con 0x7ff058004e50 2020-11-20T12:06:06.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.609+0000 7ff0765d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07015c1a0 con 0x7ff070067da0 2020-11-20T12:06:06.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff07015b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff07015b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41968/0 (socket says 172.21.15.187:41968) 2020-11-20T12:06:06.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 -- 172.21.15.187:0/986378415 learned_addr learned my addr 172.21.15.187:0/986378415 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:06.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff074dd3700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff070067da0 0x7ff0700b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 -- 172.21.15.187:0/986378415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff070067da0 msgr2=0x7ff0700b7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff070067da0 0x7ff0700b7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 -- 172.21.15.187:0/986378415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0701449c0 msgr2=0x7ff070144e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T12:06:06.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0701449c0 0x7ff070144e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:06.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 -- 172.21.15.187:0/986378415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07015b990 con 0x7ff058004e50 2020-11-20T12:06:06.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.610+0000 7ff0755d4700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff07015b4a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7ff06c0109f0 tx=0x7ff06c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.611+0000 7ff05d7fa700 1 -- 172.21.15.187:0/986378415 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c00e9f0 con 0x7ff058004e50 2020-11-20T12:06:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.611+0000 7ff05d7fa700 1 -- 172.21.15.187:0/986378415 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c00eb50 con 0x7ff058004e50 2020-11-20T12:06:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.611+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0700b7880 con 0x7ff058004e50 2020-11-20T12:06:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.611+0000 7ff05d7fa700 1 -- 172.21.15.187:0/986378415 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c00eed0 con 0x7ff058004e50 2020-11-20T12:06:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.611+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff070066590 con 0x7ff058004e50 2020-11-20T12:06:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.612+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff070062b50 con 0x7ff058004e50 2020-11-20T12:06:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.612+0000 7ff05d7fa700 1 -- 172.21.15.187:0/986378415 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff06c014680 con 0x7ff058004e50 2020-11-20T12:06:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.613+0000 7ff05d7fa700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff048056d20 0x7ff0480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.613+0000 7ff05d7fa700 1 -- 172.21.15.187:0/986378415 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff06c0700e0 con 0x7ff058004e50 2020-11-20T12:06:06.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.613+0000 7ff074dd3700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff048056d20 0x7ff0480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:06.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.614+0000 7ff074dd3700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff048056d20 0x7ff0480591e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7ff0700681e0 tx=0x7ff060009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:06.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.618+0000 7ff05d7fa700 1 -- 172.21.15.187:0/986378415 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff06c045860 con 0x7ff058004e50 2020-11-20T12:06:06.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.861+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7ff07015c360 con 0x7ff058004e50 2020-11-20T12:06:06.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.871+0000 7ff05d7fa700 1 -- 172.21.15.187:0/986378415 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff06c048e80 con 0x7ff058004e50 2020-11-20T12:06:06.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.873+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff048056d20 msgr2=0x7ff0480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.873+0000 7ff0765d6700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff048056d20 0x7ff0480591e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7ff0700681e0 tx=0x7ff060009040).stop 2020-11-20T12:06:06.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.873+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 msgr2=0x7ff07015b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:06.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.873+0000 7ff0765d6700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff07015b4a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7ff06c0109f0 tx=0x7ff06c008040).stop 2020-11-20T12:06:06.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.874+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 shutdown_connections 2020-11-20T12:06:06.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.874+0000 7ff0765d6700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff048056d20 0x7ff0480591e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.874+0000 7ff0765d6700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0701449c0 0x7ff070144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:06.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.874+0000 7ff0765d6700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff058004e50 0x7ff07015b4a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.874+0000 7ff0765d6700 1 --2- 172.21.15.187:0/986378415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff070067da0 0x7ff0700b7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:06.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.874+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 >> 172.21.15.187:0/986378415 conn(0x7ff0700bab30 msgr2=0x7ff070145c50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:06.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.874+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 shutdown_connections 2020-11-20T12:06:06.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:06.874+0000 7ff0765d6700 1 -- 172.21.15.187:0/986378415 wait complete. 2020-11-20T12:06:06.881 INFO:teuthology.orchestra.run.smithi187.stderr:updated 2020-11-20T12:06:06.899 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:06:07.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.123+0000 7f7d0004a700 1 Processor -- start 2020-11-20T12:06:07.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.123+0000 7f7d0004a700 1 -- start start 2020-11-20T12:06:07.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7d0004a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7cf814c750 0x7f7cf8062580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7d0004a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7cf8062a70 0x7f7cf80b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7d0004a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8061e80 con 0x7f7cf80666e0 2020-11-20T12:06:07.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7d0004a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf80620e0 con 0x7f7cf814c750 2020-11-20T12:06:07.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7d0004a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf80b1500 con 0x7f7cf8062a70 2020-11-20T12:06:07.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7cfe5e7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7cf8062a70 0x7f7cf80b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7cfd5e5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7cf80666e0 0x7f7cf8067d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38584/0 (socket says 172.21.15.187:38584) 2020-11-20T12:06:07.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7cfd5e5700 1 -- 172.21.15.187:0/4275899480 learned_addr learned my addr 172.21.15.187:0/4275899480 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:07.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.125+0000 7f7cfdde6700 1 --2- 172.21.15.187:0/4275899480 >> v2:172.21.15.5:3300/0 conn(0x7f7cf814c750 0x7f7cf8062580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.126+0000 7f7cfe5e7700 1 -- 172.21.15.187:0/4275899480 >> v2:172.21.15.5:3300/0 conn(0x7f7cf814c750 msgr2=0x7f7cf8062580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:07.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.126+0000 7f7cfe5e7700 1 --2- 172.21.15.187:0/4275899480 >> v2:172.21.15.5:3300/0 conn(0x7f7cf814c750 0x7f7cf8062580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.126+0000 7f7cfe5e7700 1 -- 172.21.15.187:0/4275899480 >> v1:172.21.15.187:6789/0 conn(0x7f7cf80666e0 legacy=0x7f7cf8067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.126+0000 7f7cfe5e7700 1 -- 172.21.15.187:0/4275899480 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf80b17d0 con 0x7f7cf8062a70 2020-11-20T12:06:07.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.126+0000 7f7cfcde4700 1 -- 172.21.15.187:0/4275899480 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115008678 0 0) 0x7f7cf8061e80 con 0x7f7cf80666e0 2020-11-20T12:06:07.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.126+0000 7f7cfe5e7700 1 --2- 172.21.15.187:0/4275899480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf8062a70 0x7f7cf80b0c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7cf8067040 tx=0x7f7cf00091d0).ready entity=mon.1 client_cookie=7f9662a5d5f5411a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:07.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.126+0000 7f7cfcde4700 1 -- 172.21.15.187:0/4275899480 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf000bbc0 con 0x7f7cf8062a70 2020-11-20T12:06:07.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.126+0000 7f7cfcde4700 1 -- 172.21.15.187:0/4275899480 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf000bd20 con 0x7f7cf8062a70 2020-11-20T12:06:07.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7cfcde4700 1 -- 172.21.15.187:0/4275899480 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0011610 con 0x7f7cf8062a70 2020-11-20T12:06:07.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7d0004a700 1 -- 172.21.15.187:0/4275899480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf8062a70 msgr2=0x7f7cf80b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4275899480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf8062a70 0x7f7cf80b0c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7cf8067040 tx=0x7f7cf00091d0).stop 2020-11-20T12:06:07.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7d0004a700 1 -- 172.21.15.187:0/4275899480 shutdown_connections 2020-11-20T12:06:07.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4275899480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf8062a70 0x7f7cf80b0c40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4275899480 >> v2:172.21.15.5:3300/0 conn(0x7f7cf814c750 0x7f7cf8062580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7d0004a700 1 -- 172.21.15.187:0/4275899480 >> 172.21.15.187:0/4275899480 conn(0x7f7cf80bacd0 msgr2=0x7f7cf80bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:07.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7d0004a700 1 -- 172.21.15.187:0/4275899480 shutdown_connections 2020-11-20T12:06:07.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.127+0000 7f7d0004a700 1 -- 172.21.15.187:0/4275899480 wait complete. 2020-11-20T12:06:07.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.129+0000 7f7d0004a700 1 Processor -- start 2020-11-20T12:06:07.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.129+0000 7f7d0004a700 1 -- start start 2020-11-20T12:06:07.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7d0004a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cf8062a70 0x7f7cf814dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7d0004a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf80666e0 0x7f7cf81511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7d0004a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cf814c750 0x7f7cf813d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7d0004a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf814e480 con 0x7f7cf8062a70 2020-11-20T12:06:07.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7d0004a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf814e720 con 0x7f7cf80666e0 2020-11-20T12:06:07.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7d0004a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf814e9c0 con 0x7f7cf814c750 2020-11-20T12:06:07.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7cfd5e5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf80666e0 0x7f7cf81511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7cfd5e5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf80666e0 0x7f7cf81511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:41986/0 (socket says 172.21.15.187:41986) 2020-11-20T12:06:07.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.130+0000 7f7cfd5e5700 1 -- 172.21.15.187:0/4278129028 learned_addr learned my addr 172.21.15.187:0/4278129028 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:07.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfe5e7700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cf814c750 0x7f7cf813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfdde6700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cf8062a70 0x7f7cf814dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfd5e5700 1 -- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cf814c750 msgr2=0x7f7cf813d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfd5e5700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cf814c750 0x7f7cf813d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfd5e5700 1 -- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cf8062a70 msgr2=0x7f7cf814dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfd5e5700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cf8062a70 0x7f7cf814dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfd5e5700 1 -- 172.21.15.187:0/4278129028 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf814e1b0 con 0x7f7cf80666e0 2020-11-20T12:06:07.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfdde6700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cf8062a70 0x7f7cf814dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:07.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7cfd5e5700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf80666e0 0x7f7cf81511e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f7ce800c710 tx=0x7f7ce8001800).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:07.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7ce5ffb700 1 -- 172.21.15.187:0/4278129028 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8007320 con 0x7f7cf80666e0 2020-11-20T12:06:07.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7ce5ffb700 1 -- 172.21.15.187:0/4278129028 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce80054c0 con 0x7f7cf80666e0 2020-11-20T12:06:07.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf81516d0 con 0x7f7cf80666e0 2020-11-20T12:06:07.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7ce5ffb700 1 -- 172.21.15.187:0/4278129028 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8005840 con 0x7f7cf80666e0 2020-11-20T12:06:07.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.131+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf814fb00 con 0x7f7cf80666e0 2020-11-20T12:06:07.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.132+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf814eb80 con 0x7f7cf80666e0 2020-11-20T12:06:07.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.133+0000 7f7ce5ffb700 1 -- 172.21.15.187:0/4278129028 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7ce8010030 con 0x7f7cf80666e0 2020-11-20T12:06:07.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.133+0000 7f7ce5ffb700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cd0056d20 0x7f7cd00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.134+0000 7f7ce5ffb700 1 -- 172.21.15.187:0/4278129028 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f7ce806fa60 con 0x7f7cf80666e0 2020-11-20T12:06:07.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.134+0000 7f7cfdde6700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cd0056d20 0x7f7cd00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.134+0000 7f7cfdde6700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cd0056d20 0x7f7cd00591e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7cf4009670 tx=0x7f7cf4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:07.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.137+0000 7f7ce5ffb700 1 -- 172.21.15.187:0/4278129028 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7ce80451e0 con 0x7f7cf80666e0 2020-11-20T12:06:07.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.326+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cf8063600 con 0x7f7cf80666e0 2020-11-20T12:06:07.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.331+0000 7f7ce5ffb700 1 -- 172.21.15.187:0/4278129028 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+13627 (secure 0 0 0) 0x7f7ce8048800 con 0x7f7cf80666e0 2020-11-20T12:06:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:07.336 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":28,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:06:06.426540+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:06:07.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cd0056d20 msgr2=0x7f7cd00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cd0056d20 0x7f7cd00591e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7cf4009670 tx=0x7f7cf4008040).stop 2020-11-20T12:06:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf80666e0 msgr2=0x7f7cf81511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf80666e0 0x7f7cf81511e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f7ce800c710 tx=0x7f7ce8001800).stop 2020-11-20T12:06:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 shutdown_connections 2020-11-20T12:06:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cd0056d20 0x7f7cd00591e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cf8062a70 0x7f7cf814dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cf814c750 0x7f7cf813d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 --2- 172.21.15.187:0/4278129028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cf80666e0 0x7f7cf81511e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.334+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 >> 172.21.15.187:0/4278129028 conn(0x7f7cf80bacd0 msgr2=0x7f7cf814b510 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.335+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 shutdown_connections 2020-11-20T12:06:07.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.335+0000 7f7d0004a700 1 -- 172.21.15.187:0/4278129028 wait complete. 2020-11-20T12:06:07.362 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:06:07.363 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:06:07.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.597+0000 7fa8cdd4b700 1 Processor -- start 2020-11-20T12:06:07.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.597+0000 7fa8cdd4b700 1 -- start start 2020-11-20T12:06:07.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.598+0000 7fa8cdd4b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c8065190 0x7fa8c8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.599+0000 7fa8cdd4b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c80b13e0 con 0x7fa8c8065f90 2020-11-20T12:06:07.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.599+0000 7fa8cdd4b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c8062c10 con 0x7fa8c809ca90 2020-11-20T12:06:07.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.599+0000 7fa8cdd4b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c814ab30 con 0x7fa8c8065190 2020-11-20T12:06:07.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.599+0000 7fa8cd54a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa8c809ca90 0x7fa8c809cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38598/0 (socket says 172.21.15.187:38598) 2020-11-20T12:06:07.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.599+0000 7fa8cd54a700 1 -- 172.21.15.187:0/3440991386 learned_addr learned my addr 172.21.15.187:0/3440991386 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:07.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.599+0000 7fa8ccd49700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c8065190 0x7fa8c8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862542763 0 0) 0x7fa8c8062c10 con 0x7fa8c809ca90 2020-11-20T12:06:07.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b4003240 con 0x7fa8c809ca90 2020-11-20T12:06:07.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2291003027 0 0) 0x7fa8b4003240 con 0x7fa8c809ca90 2020-11-20T12:06:07.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c8065190 msgr2=0x7fa8c8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:07.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c8065190 0x7fa8c8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 >> v1:172.21.15.5:6789/0 conn(0x7fa8c8065f90 legacy=0x7fa8c8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c80b1640 con 0x7fa8c809ca90 2020-11-20T12:06:07.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fa8c00014e0 con 0x7fa8c809ca90 2020-11-20T12:06:07.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 >> v1:172.21.15.187:6789/0 conn(0x7fa8c809ca90 legacy=0x7fa8c809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8b4003730 0x7fa8b4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.600+0000 7fa8bf7fe700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c8065190 0x7fa8b40044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.601+0000 7fa8bf7fe700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b40049d0 0x7fa8b4004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.601+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c814ab30 con 0x7fa8b4003730 2020-11-20T12:06:07.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.601+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40041d0 con 0x7fa8b40049d0 2020-11-20T12:06:07.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.601+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b400b730 con 0x7fa8c8065190 2020-11-20T12:06:07.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.601+0000 7fa8cd54a700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b40049d0 0x7fa8b4004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.601+0000 7fa8bffff700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c8065190 0x7fa8b40044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.601+0000 7fa8ccd49700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8b4003730 0x7fa8b4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.602+0000 7fa8cd54a700 1 -- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c8065190 msgr2=0x7fa8b40044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:07.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.602+0000 7fa8cd54a700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c8065190 0x7fa8b40044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.602+0000 7fa8cd54a700 1 -- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8b4003730 msgr2=0x7fa8b4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:07.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.602+0000 7fa8cd54a700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8b4003730 0x7fa8b4003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.602+0000 7fa8cd54a700 1 -- 172.21.15.187:0/3440991386 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b4003240 con 0x7fa8b40049d0 2020-11-20T12:06:07.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.602+0000 7fa8ccd49700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8b4003730 0x7fa8b4003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:06:07.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.602+0000 7fa8cd54a700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b40049d0 0x7fa8b4004e50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fa8c000c470 tx=0x7fa8c80bb650).ready entity=mon.1 client_cookie=3d51960af95dc260 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:07.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.602+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00070d0 con 0x7fa8b40049d0 2020-11-20T12:06:07.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c0007230 con 0x7fa8b40049d0 2020-11-20T12:06:07.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8bf7fe700 1 -- 172.21.15.187:0/3440991386 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00055f0 con 0x7fa8b40049d0 2020-11-20T12:06:07.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/3440991386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b40049d0 msgr2=0x7fa8b4004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8cdd4b700 1 --2- 172.21.15.187:0/3440991386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b40049d0 0x7fa8b4004e50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fa8c000c470 tx=0x7fa8c80bb650).stop 2020-11-20T12:06:07.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8cd54a700 1 -- 172.21.15.187:0/3440991386 reap_dead start 2020-11-20T12:06:07.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/3440991386 shutdown_connections 2020-11-20T12:06:07.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/3440991386 >> 172.21.15.187:0/3440991386 conn(0x7fa8c80bab30 msgr2=0x7fa8c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:07.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/3440991386 shutdown_connections 2020-11-20T12:06:07.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.603+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/3440991386 wait complete. 2020-11-20T12:06:07.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.605+0000 7fa8cdd4b700 1 Processor -- start 2020-11-20T12:06:07.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.605+0000 7fa8cdd4b700 1 -- start start 2020-11-20T12:06:07.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8cdd4b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8c809c0e0 0x7fa8c809f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8cdd4b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8c809c4c0 0x7fa8c8147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8cdd4b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c81472a0 0x7fa8c8148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8cdd4b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8145960 con 0x7fa8c809c4c0 2020-11-20T12:06:07.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8cdd4b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c809fa40 con 0x7fa8c809c0e0 2020-11-20T12:06:07.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8cdd4b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80b4400 con 0x7fa8c81472a0 2020-11-20T12:06:07.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8ccd49700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8c809c0e0 0x7fa8c809f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8ccd49700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8c809c0e0 0x7fa8c809f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42004/0 (socket says 172.21.15.187:42004) 2020-11-20T12:06:07.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.606+0000 7fa8ccd49700 1 -- 172.21.15.187:0/330745611 learned_addr learned my addr 172.21.15.187:0/330745611 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:07.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8bffff700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8c809c4c0 0x7fa8c8147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8ccd49700 1 -- 172.21.15.187:0/330745611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c81472a0 msgr2=0x7fa8c8148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8ccd49700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c81472a0 0x7fa8c8148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8ccd49700 1 -- 172.21.15.187:0/330745611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8c809c4c0 msgr2=0x7fa8c8147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8ccd49700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8c809c4c0 0x7fa8c8147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8ccd49700 1 -- 172.21.15.187:0/330745611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8148d50 con 0x7fa8c809c0e0 2020-11-20T12:06:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8bffff700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8c809c4c0 0x7fa8c8147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8ccd49700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8c809c0e0 0x7fa8c809f550 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fa8c4010fa0 tx=0x7fa8c4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8bcff9700 1 -- 172.21.15.187:0/330745611 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c400ee50 con 0x7fa8c809c0e0 2020-11-20T12:06:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8bcff9700 1 -- 172.21.15.187:0/330745611 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c4014430 con 0x7fa8c809c0e0 2020-11-20T12:06:07.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c8149050 con 0x7fa8c809c0e0 2020-11-20T12:06:07.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.607+0000 7fa8bcff9700 1 -- 172.21.15.187:0/330745611 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40147b0 con 0x7fa8c809c0e0 2020-11-20T12:06:07.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.608+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c815a6f0 con 0x7fa8c809c0e0 2020-11-20T12:06:07.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.609+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c809a2c0 con 0x7fa8c809c0e0 2020-11-20T12:06:07.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.609+0000 7fa8bcff9700 1 -- 172.21.15.187:0/330745611 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa8c4023070 con 0x7fa8c809c0e0 2020-11-20T12:06:07.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.610+0000 7fa8bcff9700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8a0056d20 0x7fa8a00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:07.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.610+0000 7fa8bcff9700 1 -- 172.21.15.187:0/330745611 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa8c4070700 con 0x7fa8c809c0e0 2020-11-20T12:06:07.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.610+0000 7fa8bffff700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8a0056d20 0x7fa8a00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:07.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.611+0000 7fa8bffff700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8a0056d20 0x7fa8a00591e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa8b8007970 tx=0x7fa8b8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:07.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.614+0000 7fa8bcff9700 1 -- 172.21.15.187:0/330745611 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa8c4045d70 con 0x7fa8c809c0e0 2020-11-20T12:06:07.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.800+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa8c8061a20 con 0x7fa8c809c0e0 2020-11-20T12:06:07.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.801+0000 7fa8bcff9700 1 -- 172.21.15.187:0/330745611 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v28) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa8c4021070 con 0x7fa8c809c0e0 2020-11-20T12:06:07.806 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:06:07.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8a0056d20 msgr2=0x7fa8a00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8a0056d20 0x7fa8a00591e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa8b8007970 tx=0x7fa8b8009040).stop 2020-11-20T12:06:07.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8c809c0e0 msgr2=0x7fa8c809f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:07.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8c809c0e0 0x7fa8c809f550 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fa8c4010fa0 tx=0x7fa8c4008040).stop 2020-11-20T12:06:07.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 shutdown_connections 2020-11-20T12:06:07.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa8a0056d20 0x7fa8a00591e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa8c81472a0 0x7fa8c8148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:07.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8c809c0e0 0x7fa8c809f550 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 --2- 172.21.15.187:0/330745611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8c809c4c0 0x7fa8c8147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:07.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.803+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 >> 172.21.15.187:0/330745611 conn(0x7fa8c80bab30 msgr2=0x7fa8c8145c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:07.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.804+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 shutdown_connections 2020-11-20T12:06:07.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:07.804+0000 7fa8cdd4b700 1 -- 172.21.15.187:0/330745611 wait complete. 2020-11-20T12:06:07.832 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:06:08.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.081+0000 7facff59e700 1 Processor -- start 2020-11-20T12:06:08.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.081+0000 7facff59e700 1 -- start start 2020-11-20T12:06:08.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.082+0000 7facff59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001484d0 0x7fad00061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.082+0000 7facff59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fad00066f70 0x7fad000b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.082+0000 7facff59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad000b1300 con 0x7fad000622c0 2020-11-20T12:06:08.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.082+0000 7facff59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad00067730 con 0x7fad00066f70 2020-11-20T12:06:08.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.082+0000 7facff59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad000b15a0 con 0x7fad001484d0 2020-11-20T12:06:08.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.083+0000 7facfdd9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fad000622c0 0x7fad000b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38618/0 (socket says 172.21.15.187:38618) 2020-11-20T12:06:08.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.083+0000 7facfdd9b700 1 -- 172.21.15.187:0/1639155289 learned_addr learned my addr 172.21.15.187:0/1639155289 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:08.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.083+0000 7facfe59c700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001484d0 0x7fad00061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.083+0000 7facfed9d700 1 --2- 172.21.15.187:0/1639155289 >> v2:172.21.15.5:3300/0 conn(0x7fad00066f70 0x7fad000b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.083+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943874715 0 0) 0x7fad000b1300 con 0x7fad000622c0 2020-11-20T12:06:08.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face4003240 con 0x7fad000622c0 2020-11-20T12:06:08.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1135251970 0 0) 0x7face4003240 con 0x7fad000622c0 2020-11-20T12:06:08.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001484d0 msgr2=0x7fad00061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:08.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001484d0 0x7fad00061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 >> v2:172.21.15.5:3300/0 conn(0x7fad00066f70 msgr2=0x7fad000b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:08.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 --2- 172.21.15.187:0/1639155289 >> v2:172.21.15.5:3300/0 conn(0x7fad00066f70 0x7fad000b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000b1840 con 0x7fad000622c0 2020-11-20T12:06:08.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7facf40014e0 con 0x7fad000622c0 2020-11-20T12:06:08.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 >> v1:172.21.15.187:6789/0 conn(0x7fad000622c0 legacy=0x7fad000b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7face40037b0 0x7face4003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7face40044e0 0x7face4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7face4004e50 0x7face400d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad00067730 con 0x7face40044e0 2020-11-20T12:06:08.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000b15a0 con 0x7face4004e50 2020-11-20T12:06:08.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.084+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face40052d0 con 0x7face40037b0 2020-11-20T12:06:08.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfe59c700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001484d0 0x7fad00061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:06:08.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfed9d700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7face4004e50 0x7face400d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfdd9b700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7face40044e0 0x7face4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfe59c700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7face40037b0 0x7face4003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfed9d700 1 -- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7face40037b0 msgr2=0x7face4003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:08.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfed9d700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7face40037b0 0x7face4003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfed9d700 1 -- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7face40044e0 msgr2=0x7face4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:08.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfed9d700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7face40044e0 0x7face4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.085+0000 7facfed9d700 1 -- 172.21.15.187:0/1639155289 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face4003240 con 0x7face4004e50 2020-11-20T12:06:08.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.086+0000 7facfe59c700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7face40037b0 0x7face4003bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:08.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.086+0000 7facfed9d700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7face4004e50 0x7face400d940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7facf8009ea0 tx=0x7facf80091d0).ready entity=mon.1 client_cookie=cf914f9d5bc2016e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:08.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.086+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf80095c0 con 0x7face4004e50 2020-11-20T12:06:08.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.086+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf800eda0 con 0x7face4004e50 2020-11-20T12:06:08.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.086+0000 7facfd59a700 1 -- 172.21.15.187:0/1639155289 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8012600 con 0x7face4004e50 2020-11-20T12:06:08.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.087+0000 7facff59e700 1 -- 172.21.15.187:0/1639155289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7face4004e50 msgr2=0x7face400d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.087+0000 7facff59e700 1 --2- 172.21.15.187:0/1639155289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7face4004e50 0x7face400d940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7facf8009ea0 tx=0x7facf80091d0).stop 2020-11-20T12:06:08.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.087+0000 7facfed9d700 1 -- 172.21.15.187:0/1639155289 reap_dead start 2020-11-20T12:06:08.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.087+0000 7facff59e700 1 -- 172.21.15.187:0/1639155289 shutdown_connections 2020-11-20T12:06:08.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.087+0000 7facff59e700 1 -- 172.21.15.187:0/1639155289 >> 172.21.15.187:0/1639155289 conn(0x7fad000bab30 msgr2=0x7fad000baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:08.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.087+0000 7facff59e700 1 -- 172.21.15.187:0/1639155289 shutdown_connections 2020-11-20T12:06:08.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.087+0000 7facff59e700 1 -- 172.21.15.187:0/1639155289 wait complete. 2020-11-20T12:06:08.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.089+0000 7facff59e700 1 Processor -- start 2020-11-20T12:06:08.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.089+0000 7facff59e700 1 -- start start 2020-11-20T12:06:08.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.090+0000 7facff59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001465d0 0x7fad00143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.090+0000 7facff59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001469b0 0x7fad0009cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.090+0000 7facff59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0009ca30 0x7fad00141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.090+0000 7facff59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad00143240 con 0x7fad001469b0 2020-11-20T12:06:08.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.090+0000 7facff59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad00143430 con 0x7fad0009ca30 2020-11-20T12:06:08.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.090+0000 7facff59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad001436d0 con 0x7fad001465d0 2020-11-20T12:06:08.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.090+0000 7facfed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0009ca30 0x7fad00141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0009ca30 0x7fad00141400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42028/0 (socket says 172.21.15.187:42028) 2020-11-20T12:06:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfed9d700 1 -- 172.21.15.187:0/3734271924 learned_addr learned my addr 172.21.15.187:0/3734271924 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfe59c700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001465d0 0x7fad00143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfdd9b700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001469b0 0x7fad0009cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfed9d700 1 -- 172.21.15.187:0/3734271924 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001465d0 msgr2=0x7fad00143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfed9d700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001465d0 0x7fad00143e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfed9d700 1 -- 172.21.15.187:0/3734271924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001469b0 msgr2=0x7fad0009cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfed9d700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001469b0 0x7fad0009cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfed9d700 1 -- 172.21.15.187:0/3734271924 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad00143970 con 0x7fad0009ca30 2020-11-20T12:06:08.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.091+0000 7facfed9d700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0009ca30 0x7fad00141400 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7facf8014000 tx=0x7facf801a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.092+0000 7facee7fc700 1 -- 172.21.15.187:0/3734271924 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf80095c0 con 0x7fad0009ca30 2020-11-20T12:06:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.092+0000 7facee7fc700 1 -- 172.21.15.187:0/3734271924 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf8012e10 con 0x7fad0009ca30 2020-11-20T12:06:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.092+0000 7facee7fc700 1 -- 172.21.15.187:0/3734271924 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf801a7f0 con 0x7fad0009ca30 2020-11-20T12:06:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.092+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad001418f0 con 0x7fad0009ca30 2020-11-20T12:06:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.092+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad00141d90 con 0x7fad0009ca30 2020-11-20T12:06:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.093+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad00061a20 con 0x7fad0009ca30 2020-11-20T12:06:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.094+0000 7facee7fc700 1 -- 172.21.15.187:0/3734271924 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7facf802c070 con 0x7fad0009ca30 2020-11-20T12:06:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.094+0000 7facee7fc700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facd4056d20 0x7facd40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.094+0000 7facee7fc700 1 -- 172.21.15.187:0/3734271924 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7facf8002030 con 0x7fad0009ca30 2020-11-20T12:06:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.095+0000 7facfe59c700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facd4056d20 0x7facd40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.095+0000 7facfe59c700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facd4056d20 0x7facd40591e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7facf0006dc0 tx=0x7facf0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.098+0000 7facee7fc700 1 -- 172.21.15.187:0/3734271924 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7facf80475b0 con 0x7fad0009ca30 2020-11-20T12:06:08.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.284+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fad001488b0 con 0x7fad0009ca30 2020-11-20T12:06:08.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.285+0000 7facee7fc700 1 -- 172.21.15.187:0/3734271924 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7facf8016070 con 0x7fad0009ca30 2020-11-20T12:06:08.289 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:06:08.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facd4056d20 msgr2=0x7facd40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facd4056d20 0x7facd40591e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7facf0006dc0 tx=0x7facf0008040).stop 2020-11-20T12:06:08.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0009ca30 msgr2=0x7fad00141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0009ca30 0x7fad00141400 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7facf8014000 tx=0x7facf801a000).stop 2020-11-20T12:06:08.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 shutdown_connections 2020-11-20T12:06:08.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facd4056d20 0x7facd40591e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0009ca30 0x7fad00141400 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad001465d0 0x7fad00143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.287+0000 7facff59e700 1 --2- 172.21.15.187:0/3734271924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001469b0 0x7fad0009cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.288+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 >> 172.21.15.187:0/3734271924 conn(0x7fad000bab30 msgr2=0x7fad000b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:08.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.288+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 shutdown_connections 2020-11-20T12:06:08.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.288+0000 7facff59e700 1 -- 172.21.15.187:0/3734271924 wait complete. 2020-11-20T12:06:08.315 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:06:08.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.548+0000 7faf76d40700 1 Processor -- start 2020-11-20T12:06:08.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.548+0000 7faf76d40700 1 -- start start 2020-11-20T12:06:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.549+0000 7faf76d40700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 0x7faf70065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.549+0000 7faf76d40700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf7009ca90 0x7faf7009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.549+0000 7faf76d40700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700b13e0 con 0x7faf70065190 2020-11-20T12:06:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.549+0000 7faf76d40700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7014ab30 con 0x7faf7009ca90 2020-11-20T12:06:08.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.549+0000 7faf76d40700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf70062c10 con 0x7faf70065f90 2020-11-20T12:06:08.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.551+0000 7faf7653f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf7009ca90 0x7faf7009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.551+0000 7faf75d3e700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7faf70065190 0x7faf70143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:60380/0 (socket says 172.21.15.187:60380) 2020-11-20T12:06:08.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.551+0000 7faf7553d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 0x7faf70065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.551+0000 7faf75d3e700 1 -- 172.21.15.187:0/217966534 learned_addr learned my addr 172.21.15.187:0/217966534 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:08.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.552+0000 7faf7653f700 1 -- 172.21.15.187:0/217966534 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 msgr2=0x7faf70065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:08.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.552+0000 7faf7653f700 1 --2- 172.21.15.187:0/217966534 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 0x7faf70065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.552+0000 7faf7653f700 1 -- 172.21.15.187:0/217966534 >> v1:172.21.15.5:6789/0 conn(0x7faf70065190 legacy=0x7faf70143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.552+0000 7faf7653f700 1 -- 172.21.15.187:0/217966534 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700b1640 con 0x7faf7009ca90 2020-11-20T12:06:08.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.552+0000 7faf7653f700 1 --2- 172.21.15.187:0/217966534 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf7009ca90 0x7faf7009cf20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7faf7014ac70 tx=0x7faf68006df0).ready entity=mon.0 client_cookie=e4f78d865cfe37bb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:08.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.553+0000 7faf74d3c700 1 -- 172.21.15.187:0/217966534 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68009840 con 0x7faf7009ca90 2020-11-20T12:06:08.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.553+0000 7faf74d3c700 1 -- 172.21.15.187:0/217966534 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf680099a0 con 0x7faf7009ca90 2020-11-20T12:06:08.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.553+0000 7faf74d3c700 1 -- 172.21.15.187:0/217966534 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68009d20 con 0x7faf7009ca90 2020-11-20T12:06:08.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.554+0000 7faf76d40700 1 -- 172.21.15.187:0/217966534 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf7009ca90 msgr2=0x7faf7009cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.554+0000 7faf76d40700 1 --2- 172.21.15.187:0/217966534 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf7009ca90 0x7faf7009cf20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7faf7014ac70 tx=0x7faf68006df0).stop 2020-11-20T12:06:08.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.554+0000 7faf76d40700 1 -- 172.21.15.187:0/217966534 shutdown_connections 2020-11-20T12:06:08.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.554+0000 7faf76d40700 1 --2- 172.21.15.187:0/217966534 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf7009ca90 0x7faf7009cf20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.554+0000 7faf76d40700 1 --2- 172.21.15.187:0/217966534 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 0x7faf70065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.554+0000 7faf76d40700 1 -- 172.21.15.187:0/217966534 >> 172.21.15.187:0/217966534 conn(0x7faf700bab30 msgr2=0x7faf700baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:08.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.554+0000 7faf76d40700 1 -- 172.21.15.187:0/217966534 shutdown_connections 2020-11-20T12:06:08.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.554+0000 7faf76d40700 1 -- 172.21.15.187:0/217966534 wait complete. 2020-11-20T12:06:08.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.556+0000 7faf76d40700 1 Processor -- start 2020-11-20T12:06:08.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.556+0000 7faf76d40700 1 -- start start 2020-11-20T12:06:08.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.557+0000 7faf76d40700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf70065190 0x7faf700b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.557+0000 7faf76d40700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 0x7faf70067560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.557+0000 7faf76d40700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf7009ca90 0x7faf700b7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.557+0000 7faf76d40700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700a0520 con 0x7faf70065190 2020-11-20T12:06:08.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.557+0000 7faf76d40700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700a07c0 con 0x7faf7009ca90 2020-11-20T12:06:08.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.557+0000 7faf76d40700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf70099880 con 0x7faf70065f90 2020-11-20T12:06:08.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.557+0000 7faf7653f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf7009ca90 0x7faf700b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf7653f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf7009ca90 0x7faf700b7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42042/0 (socket says 172.21.15.187:42042) 2020-11-20T12:06:08.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf7653f700 1 -- 172.21.15.187:0/518642153 learned_addr learned my addr 172.21.15.187:0/518642153 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:08.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf75d3e700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf70065190 0x7faf700b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf7553d700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 0x7faf70067560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf7653f700 1 -- 172.21.15.187:0/518642153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 msgr2=0x7faf70067560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf7653f700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 0x7faf70067560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf7653f700 1 -- 172.21.15.187:0/518642153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf70065190 msgr2=0x7faf700b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf7653f700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf70065190 0x7faf700b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.558+0000 7faf7653f700 1 -- 172.21.15.187:0/518642153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700a0210 con 0x7faf7009ca90 2020-11-20T12:06:08.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.559+0000 7faf7653f700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf7009ca90 0x7faf700b7250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7faf68006e30 tx=0x7faf68015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:08.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.559+0000 7faf5dffb700 1 -- 172.21.15.187:0/518642153 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68006b20 con 0x7faf7009ca90 2020-11-20T12:06:08.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.559+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf700b21f0 con 0x7faf7009ca90 2020-11-20T12:06:08.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.559+0000 7faf5dffb700 1 -- 172.21.15.187:0/518642153 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6801d5d0 con 0x7faf7009ca90 2020-11-20T12:06:08.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.559+0000 7faf5dffb700 1 -- 172.21.15.187:0/518642153 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6801d950 con 0x7faf7009ca90 2020-11-20T12:06:08.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.559+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf70067c10 con 0x7faf7009ca90 2020-11-20T12:06:08.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.560+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf70061a20 con 0x7faf7009ca90 2020-11-20T12:06:08.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.561+0000 7faf5dffb700 1 -- 172.21.15.187:0/518642153 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faf68025030 con 0x7faf7009ca90 2020-11-20T12:06:08.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.562+0000 7faf5dffb700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf48056d20 0x7faf480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:08.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.562+0000 7faf5dffb700 1 -- 172.21.15.187:0/518642153 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7faf68071510 con 0x7faf7009ca90 2020-11-20T12:06:08.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.562+0000 7faf75d3e700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf48056d20 0x7faf480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:08.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.563+0000 7faf75d3e700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf48056d20 0x7faf480591e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7faf6c007fd0 tx=0x7faf6c007f00).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:08.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.566+0000 7faf5dffb700 1 -- 172.21.15.187:0/518642153 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faf68046dc0 con 0x7faf7009ca90 2020-11-20T12:06:08.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.755+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7faf700b92c0 con 0x7faf7009ca90 2020-11-20T12:06:08.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.756+0000 7faf5dffb700 1 -- 172.21.15.187:0/518642153 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v28) v1 ==== 84+0+143 (secure 0 0 0) 0x7faf68023020 con 0x7faf7009ca90 2020-11-20T12:06:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:08.762 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:06:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:06:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:06:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:06:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:06:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:08.763 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:06:08.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf48056d20 msgr2=0x7faf480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf48056d20 0x7faf480591e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7faf6c007fd0 tx=0x7faf6c007f00).stop 2020-11-20T12:06:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf7009ca90 msgr2=0x7faf700b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf7009ca90 0x7faf700b7250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7faf68006e30 tx=0x7faf68015000).stop 2020-11-20T12:06:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 shutdown_connections 2020-11-20T12:06:08.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf48056d20 0x7faf480591e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf70065190 0x7faf700b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf7009ca90 0x7faf700b7250 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 --2- 172.21.15.187:0/518642153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf70065f90 0x7faf70067560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:08.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 >> 172.21.15.187:0/518642153 conn(0x7faf700bab30 msgr2=0x7faf700b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:08.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 shutdown_connections 2020-11-20T12:06:08.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:08.759+0000 7faf76d40700 1 -- 172.21.15.187:0/518642153 wait complete. 2020-11-20T12:06:08.784 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:06:08.785 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:06:09.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.030+0000 7fc70a74a700 1 Processor -- start 2020-11-20T12:06:09.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.030+0000 7fc70a74a700 1 -- start start 2020-11-20T12:06:09.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.031+0000 7fc70a74a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc7040b6650 0x7fc7040afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:09.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.031+0000 7fc70a74a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc704066f70 0x7fc7040aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:09.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.031+0000 7fc70a74a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040b1300 con 0x7fc7041484d0 2020-11-20T12:06:09.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.031+0000 7fc70a74a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc704067730 con 0x7fc704066f70 2020-11-20T12:06:09.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.031+0000 7fc70a74a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040b15a0 con 0x7fc7040b6650 2020-11-20T12:06:09.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.032+0000 7fc708f47700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc7040b6650 0x7fc7040afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:09.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.032+0000 7fc708f47700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc7040b6650 0x7fc7040afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42050/0 (socket says 172.21.15.187:42050) 2020-11-20T12:06:09.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.032+0000 7fc708f47700 1 -- 172.21.15.187:0/4037863525 learned_addr learned my addr 172.21.15.187:0/4037863525 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:09.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.032+0000 7fc709f49700 1 --2- 172.21.15.187:0/4037863525 >> v2:172.21.15.5:3300/0 conn(0x7fc704066f70 0x7fc7040aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:09.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.032+0000 7fc708f47700 1 -- 172.21.15.187:0/4037863525 >> v2:172.21.15.5:3300/0 conn(0x7fc704066f70 msgr2=0x7fc7040aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:09.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.033+0000 7fc708f47700 1 --2- 172.21.15.187:0/4037863525 >> v2:172.21.15.5:3300/0 conn(0x7fc704066f70 0x7fc7040aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:09.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.033+0000 7fc708f47700 1 -- 172.21.15.187:0/4037863525 >> v1:172.21.15.5:6789/0 conn(0x7fc7041484d0 legacy=0x7fc704061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:09.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.033+0000 7fc708f47700 1 -- 172.21.15.187:0/4037863525 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040b1840 con 0x7fc7040b6650 2020-11-20T12:06:09.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.033+0000 7fc708f47700 1 --2- 172.21.15.187:0/4037863525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040b6650 0x7fc7040afa90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fc704068600 tx=0x7fc700006eb0).ready entity=mon.1 client_cookie=3164de7fe50486f7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:09.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.033+0000 7fc6fbfff700 1 -- 172.21.15.187:0/4037863525 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700009890 con 0x7fc7040b6650 2020-11-20T12:06:09.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.033+0000 7fc6fbfff700 1 -- 172.21.15.187:0/4037863525 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7000099f0 con 0x7fc7040b6650 2020-11-20T12:06:09.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.033+0000 7fc6fbfff700 1 -- 172.21.15.187:0/4037863525 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700009d70 con 0x7fc7040b6650 2020-11-20T12:06:09.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.034+0000 7fc70a74a700 1 -- 172.21.15.187:0/4037863525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040b6650 msgr2=0x7fc7040afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:09.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.034+0000 7fc70a74a700 1 --2- 172.21.15.187:0/4037863525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040b6650 0x7fc7040afa90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fc704068600 tx=0x7fc700006eb0).stop 2020-11-20T12:06:09.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.034+0000 7fc70a74a700 1 -- 172.21.15.187:0/4037863525 shutdown_connections 2020-11-20T12:06:09.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.034+0000 7fc70a74a700 1 --2- 172.21.15.187:0/4037863525 >> v2:172.21.15.5:3300/0 conn(0x7fc704066f70 0x7fc7040aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:09.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.034+0000 7fc70a74a700 1 --2- 172.21.15.187:0/4037863525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040b6650 0x7fc7040afa90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:09.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.034+0000 7fc70a74a700 1 -- 172.21.15.187:0/4037863525 >> 172.21.15.187:0/4037863525 conn(0x7fc7040bab30 msgr2=0x7fc7040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:09.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.034+0000 7fc70a74a700 1 -- 172.21.15.187:0/4037863525 shutdown_connections 2020-11-20T12:06:09.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.034+0000 7fc70a74a700 1 -- 172.21.15.187:0/4037863525 wait complete. 2020-11-20T12:06:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.036+0000 7fc70a74a700 1 Processor -- start 2020-11-20T12:06:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.036+0000 7fc70a74a700 1 -- start start 2020-11-20T12:06:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc70a74a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc704066f70 0x7fc70409f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc70a74a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7040b6650 0x7fc7040b9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc70a74a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7041484d0 0x7fc7040a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc70a74a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70409fb20 con 0x7fc7040b6650 2020-11-20T12:06:09.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc70a74a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70409fdc0 con 0x7fc704066f70 2020-11-20T12:06:09.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc70a74a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040a0060 con 0x7fc7041484d0 2020-11-20T12:06:09.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc709748700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc704066f70 0x7fc70409f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:09.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc709748700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc704066f70 0x7fc70409f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42052/0 (socket says 172.21.15.187:42052) 2020-11-20T12:06:09.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc709748700 1 -- 172.21.15.187:0/496378796 learned_addr learned my addr 172.21.15.187:0/496378796 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc709748700 1 -- 172.21.15.187:0/496378796 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7041484d0 msgr2=0x7fc7040a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc709748700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7041484d0 0x7fc7040a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc709748700 1 -- 172.21.15.187:0/496378796 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7040b6650 msgr2=0x7fc7040b9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc709748700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7040b6650 0x7fc7040b9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.038+0000 7fc709748700 1 -- 172.21.15.187:0/496378796 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70409cbd0 con 0x7fc704066f70 2020-11-20T12:06:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.039+0000 7fc709748700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc704066f70 0x7fc70409f140 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fc6f40096e0 tx=0x7fc6f4006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.039+0000 7fc6f97fa700 1 -- 172.21.15.187:0/496378796 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f400d5c0 con 0x7fc704066f70 2020-11-20T12:06:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.039+0000 7fc6f97fa700 1 -- 172.21.15.187:0/496378796 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4000e90 con 0x7fc704066f70 2020-11-20T12:06:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.039+0000 7fc6f97fa700 1 -- 172.21.15.187:0/496378796 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f400d990 con 0x7fc704066f70 2020-11-20T12:06:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.039+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7040b9b00 con 0x7fc704066f70 2020-11-20T12:06:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.039+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7040a09f0 con 0x7fc704066f70 2020-11-20T12:06:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.040+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7040a0220 con 0x7fc704066f70 2020-11-20T12:06:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.041+0000 7fc6f97fa700 1 -- 172.21.15.187:0/496378796 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc6f400dd40 con 0x7fc704066f70 2020-11-20T12:06:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.041+0000 7fc6f97fa700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.041+0000 7fc6f97fa700 1 -- 172.21.15.187:0/496378796 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc6f406f620 con 0x7fc704066f70 2020-11-20T12:06:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.042+0000 7fc708f47700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:09.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.042+0000 7fc708f47700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc70409df10 tx=0x7fc700015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:09.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.047+0000 7fc6f97fa700 1 -- 172.21.15.187:0/496378796 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc6f4044c90 con 0x7fc704066f70 2020-11-20T12:06:09.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:09.234+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fc704061a20 con 0x7fc704066f70 2020-11-20T12:06:10.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.027+0000 7fc6f97fa700 1 -- 172.21.15.187:0/496378796 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc6f40482b0 con 0x7fc704066f70 2020-11-20T12:06:10.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 msgr2=0x7fc6dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:10.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc70409df10 tx=0x7fc700015000).stop 2020-11-20T12:06:10.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc704066f70 msgr2=0x7fc70409f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:10.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc704066f70 0x7fc70409f140 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fc6f40096e0 tx=0x7fc6f4006d70).stop 2020-11-20T12:06:10.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 shutdown_connections 2020-11-20T12:06:10.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0x7fc70409df10 tx=0x7fc700015000).stop 2020-11-20T12:06:10.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7041484d0 0x7fc7040a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:10.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc704066f70 0x7fc70409f140 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:10.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 --2- 172.21.15.187:0/496378796 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7040b6650 0x7fc7040b9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:10.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.032+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 >> 172.21.15.187:0/496378796 conn(0x7fc7040bab30 msgr2=0x7fc7040ba2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:10.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.033+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 shutdown_connections 2020-11-20T12:06:10.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.033+0000 7fc70a74a700 1 -- 172.21.15.187:0/496378796 wait complete. 2020-11-20T12:06:10.040 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:06:10.065 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:06:10.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.303+0000 7f147759e700 1 Processor -- start 2020-11-20T12:06:10.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.303+0000 7f147759e700 1 -- start start 2020-11-20T12:06:10.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.304+0000 7f147759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f14780b6650 0x7f14780afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:10.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.304+0000 7f147759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1478066f70 0x7f14780aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:10.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.304+0000 7f147759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14780b1300 con 0x7f14781484d0 2020-11-20T12:06:10.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.304+0000 7f147759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1478067730 con 0x7f14780b6650 2020-11-20T12:06:10.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.304+0000 7f147759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b15a0 con 0x7f1478066f70 2020-11-20T12:06:10.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.305+0000 7f147659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f14781484d0 0x7f1478061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38666/0 (socket says 172.21.15.187:38666) 2020-11-20T12:06:10.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.305+0000 7f147659c700 1 -- 172.21.15.187:0/3025736484 learned_addr learned my addr 172.21.15.187:0/3025736484 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:10.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.305+0000 7f146dd9b700 1 --2- 172.21.15.187:0/3025736484 >> v2:172.21.15.187:3300/0 conn(0x7f14780b6650 0x7f14780afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:10.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.305+0000 7f1475d9b700 1 -- 172.21.15.187:0/3025736484 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454704158 0 0) 0x7f14780b1300 con 0x7f14781484d0 2020-11-20T12:06:10.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.305+0000 7f1475d9b700 1 -- 172.21.15.187:0/3025736484 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1458003240 con 0x7f14781484d0 2020-11-20T12:06:10.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.305+0000 7f1476d9d700 1 --2- 172.21.15.187:0/3025736484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1478066f70 0x7f14780aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:10.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f146dd9b700 1 -- 172.21.15.187:0/3025736484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1478066f70 msgr2=0x7f14780aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:10.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f146dd9b700 1 --2- 172.21.15.187:0/3025736484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1478066f70 0x7f14780aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:10.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f146dd9b700 1 -- 172.21.15.187:0/3025736484 >> v1:172.21.15.187:6789/0 conn(0x7f14781484d0 legacy=0x7f1478061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:10.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f146dd9b700 1 -- 172.21.15.187:0/3025736484 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14780b1840 con 0x7f14780b6650 2020-11-20T12:06:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f1476d9d700 1 --2- 172.21.15.187:0/3025736484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1478066f70 0x7f14780aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f146dd9b700 1 --2- 172.21.15.187:0/3025736484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b6650 0x7f14780afa90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f146000b9a0 tx=0x7f1460006df0).ready entity=mon.1 client_cookie=ee85a5e6568aaeee server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f1475d9b700 1 -- 172.21.15.187:0/3025736484 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14600097b0 con 0x7f14780b6650 2020-11-20T12:06:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f1475d9b700 1 -- 172.21.15.187:0/3025736484 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1460009910 con 0x7f14780b6650 2020-11-20T12:06:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.306+0000 7f1475d9b700 1 -- 172.21.15.187:0/3025736484 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460009c90 con 0x7f14780b6650 2020-11-20T12:06:10.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.307+0000 7f147759e700 1 -- 172.21.15.187:0/3025736484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b6650 msgr2=0x7f14780afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:10.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.307+0000 7f147759e700 1 --2- 172.21.15.187:0/3025736484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b6650 0x7f14780afa90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f146000b9a0 tx=0x7f1460006df0).stop 2020-11-20T12:06:10.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.307+0000 7f147759e700 1 -- 172.21.15.187:0/3025736484 shutdown_connections 2020-11-20T12:06:10.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.307+0000 7f147759e700 1 --2- 172.21.15.187:0/3025736484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1478066f70 0x7f14780aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:10.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.307+0000 7f147759e700 1 --2- 172.21.15.187:0/3025736484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b6650 0x7f14780afa90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:10.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.307+0000 7f147759e700 1 -- 172.21.15.187:0/3025736484 >> 172.21.15.187:0/3025736484 conn(0x7f14780bab30 msgr2=0x7f14780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:10.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.307+0000 7f147759e700 1 -- 172.21.15.187:0/3025736484 shutdown_connections 2020-11-20T12:06:10.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.307+0000 7f147759e700 1 -- 172.21.15.187:0/3025736484 wait complete. 2020-11-20T12:06:10.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.309+0000 7f147759e700 1 Processor -- start 2020-11-20T12:06:10.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.309+0000 7f147759e700 1 -- start start 2020-11-20T12:06:10.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.309+0000 7f147759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1478066f70 0x7f14780b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:10.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.309+0000 7f147759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14780b6650 0x7f147809b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:10.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14781484d0 0x7f14781421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:10.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147809e6e0 con 0x7f14780b6650 2020-11-20T12:06:10.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b25d0 con 0x7f1478066f70 2020-11-20T12:06:10.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b4580 con 0x7f14781484d0 2020-11-20T12:06:10.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1478066f70 0x7f14780b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:10.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1478066f70 0x7f14780b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42066/0 (socket says 172.21.15.187:42066) 2020-11-20T12:06:10.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147659c700 1 -- 172.21.15.187:0/2918164761 learned_addr learned my addr 172.21.15.187:0/2918164761 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:10.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147659c700 1 -- 172.21.15.187:0/2918164761 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14781484d0 msgr2=0x7f14781421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:10.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147659c700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14781484d0 0x7f14781421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:10.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147659c700 1 -- 172.21.15.187:0/2918164761 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14780b6650 msgr2=0x7f147809b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:10.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147659c700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14780b6650 0x7f147809b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:10.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.310+0000 7f147659c700 1 -- 172.21.15.187:0/2918164761 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147809f4b0 con 0x7f1478066f70 2020-11-20T12:06:10.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.311+0000 7f147659c700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1478066f70 0x7f14780b4090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f146800c710 tx=0x7f1468007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:10.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.311+0000 7f146effd700 1 -- 172.21.15.187:0/2918164761 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14680072e0 con 0x7f1478066f70 2020-11-20T12:06:10.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.311+0000 7f146effd700 1 -- 172.21.15.187:0/2918164761 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468007440 con 0x7f1478066f70 2020-11-20T12:06:10.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.311+0000 7f146effd700 1 -- 172.21.15.187:0/2918164761 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468005820 con 0x7f1478066f70 2020-11-20T12:06:10.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.311+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147809e240 con 0x7f1478066f70 2020-11-20T12:06:10.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.311+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147809bda0 con 0x7f1478066f70 2020-11-20T12:06:10.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.312+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147809e540 con 0x7f1478066f70 2020-11-20T12:06:10.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.313+0000 7f146effd700 1 -- 172.21.15.187:0/2918164761 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1468010070 con 0x7f1478066f70 2020-11-20T12:06:10.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.313+0000 7f146effd700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1454056d50 0x7f1454059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:10.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.314+0000 7f146effd700 1 -- 172.21.15.187:0/2918164761 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f146806f510 con 0x7f1478066f70 2020-11-20T12:06:10.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.314+0000 7f146dd9b700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1454056d50 0x7f1454059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:10.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.314+0000 7f146dd9b700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1454056d50 0x7f1454059210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1478065200 tx=0x7f1460015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:10.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.318+0000 7f146effd700 1 -- 172.21.15.187:0/2918164761 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f147809e540 con 0x7f1478066f70 2020-11-20T12:06:10.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:10.502+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f147809e540 con 0x7f1478066f70 2020-11-20T12:06:11.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.034+0000 7f146effd700 1 -- 172.21.15.187:0/2918164761 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v30) v1 ==== 125+0+0 (secure 0 0 0) 0x7f147809e540 con 0x7f1478066f70 2020-11-20T12:06:11.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.039+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1454056d50 msgr2=0x7f1454059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.039+0000 7f147759e700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1454056d50 0x7f1454059210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1478065200 tx=0x7f1460015000).stop 2020-11-20T12:06:11.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.039+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1478066f70 msgr2=0x7f14780b4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.039+0000 7f147759e700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1478066f70 0x7f14780b4090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f146800c710 tx=0x7f1468007ad0).stop 2020-11-20T12:06:11.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.040+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 shutdown_connections 2020-11-20T12:06:11.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.040+0000 7f147759e700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1454056d50 0x7f1454059210 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0x7f1478065200 tx=0x7f1460015000).stop 2020-11-20T12:06:11.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.040+0000 7f147759e700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14781484d0 0x7f14781421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:11.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.040+0000 7f147759e700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1478066f70 0x7f14780b4090 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.040+0000 7f147759e700 1 --2- 172.21.15.187:0/2918164761 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14780b6650 0x7f147809b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:11.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.040+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 >> 172.21.15.187:0/2918164761 conn(0x7f14780bab30 msgr2=0x7f1478142cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:11.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.041+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 shutdown_connections 2020-11-20T12:06:11.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.041+0000 7f147759e700 1 -- 172.21.15.187:0/2918164761 wait complete. 2020-11-20T12:06:11.047 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:06:11.075 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:06:11.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.310+0000 7ff987fff700 1 Processor -- start 2020-11-20T12:06:11.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.310+0000 7ff987fff700 1 -- start start 2020-11-20T12:06:11.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.312+0000 7ff987fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff9880bf800 0x7ff98814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.312+0000 7ff987fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff98809f560 0x7ff9880a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.312+0000 7ff987fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98814d110 con 0x7ff9880bfbe0 2020-11-20T12:06:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.312+0000 7ff987fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98814cbd0 con 0x7ff98809f560 2020-11-20T12:06:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.312+0000 7ff987fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98814ce70 con 0x7ff9880bf800 2020-11-20T12:06:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.312+0000 7ff97ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff9880bfbe0 0x7ff9881480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38682/0 (socket says 172.21.15.187:38682) 2020-11-20T12:06:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.312+0000 7ff97ffff700 1 -- 172.21.15.187:0/1669454849 learned_addr learned my addr 172.21.15.187:0/1669454849 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.312+0000 7ff986ffd700 1 --2- 172.21.15.187:0/1669454849 >> v2:172.21.15.187:3300/0 conn(0x7ff9880bf800 0x7ff98814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.313+0000 7ff9877fe700 1 --2- 172.21.15.187:0/1669454849 >> v2:172.21.15.5:3300/0 conn(0x7ff98809f560 0x7ff9880a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.313+0000 7ff986ffd700 1 -- 172.21.15.187:0/1669454849 >> v2:172.21.15.5:3300/0 conn(0x7ff98809f560 msgr2=0x7ff9880a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:11.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.313+0000 7ff986ffd700 1 --2- 172.21.15.187:0/1669454849 >> v2:172.21.15.5:3300/0 conn(0x7ff98809f560 0x7ff9880a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.313+0000 7ff986ffd700 1 -- 172.21.15.187:0/1669454849 >> v1:172.21.15.187:6789/0 conn(0x7ff9880bfbe0 legacy=0x7ff9881480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.313+0000 7ff986ffd700 1 -- 172.21.15.187:0/1669454849 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880a4cc0 con 0x7ff9880bf800 2020-11-20T12:06:11.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.313+0000 7ff9867fc700 1 -- 172.21.15.187:0/1669454849 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508993404 0 0) 0x7ff98814d110 con 0x7ff9880bfbe0 2020-11-20T12:06:11.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.313+0000 7ff986ffd700 1 --2- 172.21.15.187:0/1669454849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9880bf800 0x7ff98814c2c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7ff988068780 tx=0x7ff978006df0).ready entity=mon.1 client_cookie=6751d7203ebf4179 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:11.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff9867fc700 1 -- 172.21.15.187:0/1669454849 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978009840 con 0x7ff9880bf800 2020-11-20T12:06:11.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff9867fc700 1 -- 172.21.15.187:0/1669454849 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9780099a0 con 0x7ff9880bf800 2020-11-20T12:06:11.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff9867fc700 1 -- 172.21.15.187:0/1669454849 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978009d20 con 0x7ff9880bf800 2020-11-20T12:06:11.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff987fff700 1 -- 172.21.15.187:0/1669454849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9880bf800 msgr2=0x7ff98814c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff987fff700 1 --2- 172.21.15.187:0/1669454849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9880bf800 0x7ff98814c2c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7ff988068780 tx=0x7ff978006df0).stop 2020-11-20T12:06:11.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff987fff700 1 -- 172.21.15.187:0/1669454849 shutdown_connections 2020-11-20T12:06:11.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff987fff700 1 --2- 172.21.15.187:0/1669454849 >> v2:172.21.15.5:3300/0 conn(0x7ff98809f560 0x7ff9880a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff987fff700 1 --2- 172.21.15.187:0/1669454849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9880bf800 0x7ff98814c2c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff987fff700 1 -- 172.21.15.187:0/1669454849 >> 172.21.15.187:0/1669454849 conn(0x7ff988068370 msgr2=0x7ff9880bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:11.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.314+0000 7ff987fff700 1 -- 172.21.15.187:0/1669454849 shutdown_connections 2020-11-20T12:06:11.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.315+0000 7ff987fff700 1 -- 172.21.15.187:0/1669454849 wait complete. 2020-11-20T12:06:11.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.316+0000 7ff987fff700 1 Processor -- start 2020-11-20T12:06:11.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.317+0000 7ff987fff700 1 -- start start 2020-11-20T12:06:11.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff987fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff98809f560 0x7ff98815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff987fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff9880bf800 0x7ff988158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff987fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988158a00 0x7ff988135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff987fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98815c580 con 0x7ff98809f560 2020-11-20T12:06:11.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff987fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98815c820 con 0x7ff988158a00 2020-11-20T12:06:11.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff987fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff988144610 con 0x7ff9880bf800 2020-11-20T12:06:11.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff9877fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988158a00 0x7ff988135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff9877fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988158a00 0x7ff988135610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42084/0 (socket says 172.21.15.187:42084) 2020-11-20T12:06:11.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff9877fe700 1 -- 172.21.15.187:0/3296545489 learned_addr learned my addr 172.21.15.187:0/3296545489 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:11.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.318+0000 7ff986ffd700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff98809f560 0x7ff98815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff97ffff700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff9880bf800 0x7ff988158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff9877fe700 1 -- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff9880bf800 msgr2=0x7ff988158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff9877fe700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff9880bf800 0x7ff988158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff9877fe700 1 -- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff98809f560 msgr2=0x7ff98815bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff9877fe700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff98809f560 0x7ff98815bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff9877fe700 1 -- 172.21.15.187:0/3296545489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98815c130 con 0x7ff988158a00 2020-11-20T12:06:11.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff97ffff700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff9880bf800 0x7ff988158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:11.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff9877fe700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988158a00 0x7ff988135610 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7ff98000eb40 tx=0x7ff980008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:11.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff97effd700 1 -- 172.21.15.187:0/3296545489 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98000c830 con 0x7ff988158a00 2020-11-20T12:06:11.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff97effd700 1 -- 172.21.15.187:0/3296545489 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98000c990 con 0x7ff988158a00 2020-11-20T12:06:11.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff988146250 con 0x7ff988158a00 2020-11-20T12:06:11.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff97effd700 1 -- 172.21.15.187:0/3296545489 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98000cd30 con 0x7ff988158a00 2020-11-20T12:06:11.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.319+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff988135d40 con 0x7ff988158a00 2020-11-20T12:06:11.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.320+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff988135f60 con 0x7ff988158a00 2020-11-20T12:06:11.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.321+0000 7ff97effd700 1 -- 172.21.15.187:0/3296545489 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff980013640 con 0x7ff988158a00 2020-11-20T12:06:11.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.322+0000 7ff97effd700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff964056d50 0x7ff964059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.322+0000 7ff97effd700 1 -- 172.21.15.187:0/3296545489 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ff98006fc50 con 0x7ff988158a00 2020-11-20T12:06:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.322+0000 7ff986ffd700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff964056d50 0x7ff964059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.323+0000 7ff986ffd700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff964056d50 0x7ff964059210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff97800c010 tx=0x7ff978015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:11.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.325+0000 7ff97effd700 1 -- 172.21.15.187:0/3296545489 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff980045110 con 0x7ff988158a00 2020-11-20T12:06:11.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.571+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ff988080050 con 0x7ff988158a00 2020-11-20T12:06:11.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.586+0000 7ff97effd700 1 -- 172.21.15.187:0/3296545489 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 5 and data pool 6 v10) v1 ==== 187+0+0 (secure 0 0 0) 0x7ff980048730 con 0x7ff988158a00 2020-11-20T12:06:11.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.590+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff964056d50 msgr2=0x7ff964059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.591+0000 7ff987fff700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff964056d50 0x7ff964059210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff97800c010 tx=0x7ff978015000).stop 2020-11-20T12:06:11.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.591+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988158a00 msgr2=0x7ff988135610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.591+0000 7ff987fff700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988158a00 0x7ff988135610 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7ff98000eb40 tx=0x7ff980008040).stop 2020-11-20T12:06:11.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.593+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 shutdown_connections 2020-11-20T12:06:11.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.593+0000 7ff987fff700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff964056d50 0x7ff964059210 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.593+0000 7ff987fff700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff98809f560 0x7ff98815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.593+0000 7ff987fff700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988158a00 0x7ff988135610 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.593+0000 7ff987fff700 1 --2- 172.21.15.187:0/3296545489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff9880bf800 0x7ff988158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.593+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 >> 172.21.15.187:0/3296545489 conn(0x7ff988068370 msgr2=0x7ff988141d30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:11.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.594+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 shutdown_connections 2020-11-20T12:06:11.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.594+0000 7ff987fff700 1 -- 172.21.15.187:0/3296545489 wait complete. 2020-11-20T12:06:11.600 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 5 and data pool 6 2020-11-20T12:06:11.622 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:06:11.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.830+0000 7f6ca759e700 1 Processor -- start 2020-11-20T12:06:11.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.831+0000 7f6ca759e700 1 -- start start 2020-11-20T12:06:11.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.832+0000 7f6ca759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6ca81484d0 0x7f6ca8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.832+0000 7f6ca759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6ca80622c0 0x7f6ca80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.832+0000 7f6ca759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca8066f70 0x7f6ca80b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.832+0000 7f6ca759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca8067730 con 0x7f6ca81484d0 2020-11-20T12:06:11.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.832+0000 7f6ca759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca80b1300 con 0x7f6ca80622c0 2020-11-20T12:06:11.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.832+0000 7f6ca759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80b15a0 con 0x7f6ca8066f70 2020-11-20T12:06:11.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6ca80622c0 0x7f6ca80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6ca80622c0 0x7f6ca80b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42100/0 (socket says 172.21.15.187:42100) 2020-11-20T12:06:11.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 -- 172.21.15.187:0/3731725377 learned_addr learned my addr 172.21.15.187:0/3731725377 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:11.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6ca6d9d700 1 --2- 172.21.15.187:0/3731725377 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca8066f70 0x7f6ca80b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6ca659c700 1 --2- 172.21.15.187:0/3731725377 >> v2:172.21.15.5:3300/0 conn(0x7f6ca81484d0 0x7f6ca8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 -- 172.21.15.187:0/3731725377 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca8066f70 msgr2=0x7f6ca80b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:11.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 --2- 172.21.15.187:0/3731725377 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca8066f70 0x7f6ca80b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 -- 172.21.15.187:0/3731725377 >> v2:172.21.15.5:3300/0 conn(0x7f6ca81484d0 msgr2=0x7f6ca8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:11.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 --2- 172.21.15.187:0/3731725377 >> v2:172.21.15.5:3300/0 conn(0x7f6ca81484d0 0x7f6ca8061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 -- 172.21.15.187:0/3731725377 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca80b1840 con 0x7f6ca80622c0 2020-11-20T12:06:11.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.833+0000 7f6c9dd9b700 1 --2- 172.21.15.187:0/3731725377 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca80b5640 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f6ca8068600 tx=0x7f6c90006eb0).ready entity=mon.1 client_cookie=c693e75216232fe2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:11.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca5d9b700 1 -- 172.21.15.187:0/3731725377 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900098d0 con 0x7f6ca80622c0 2020-11-20T12:06:11.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca5d9b700 1 -- 172.21.15.187:0/3731725377 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c90009a30 con 0x7f6ca80622c0 2020-11-20T12:06:11.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca5d9b700 1 -- 172.21.15.187:0/3731725377 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90009db0 con 0x7f6ca80622c0 2020-11-20T12:06:11.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca759e700 1 -- 172.21.15.187:0/3731725377 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 msgr2=0x7f6ca80b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3731725377 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca80b5640 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f6ca8068600 tx=0x7f6c90006eb0).stop 2020-11-20T12:06:11.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca759e700 1 -- 172.21.15.187:0/3731725377 shutdown_connections 2020-11-20T12:06:11.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3731725377 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca8066f70 0x7f6ca80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3731725377 >> v2:172.21.15.5:3300/0 conn(0x7f6ca81484d0 0x7f6ca8061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3731725377 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca80b5640 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.834+0000 7f6ca759e700 1 -- 172.21.15.187:0/3731725377 >> 172.21.15.187:0/3731725377 conn(0x7f6ca80bab30 msgr2=0x7f6ca80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:11.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.835+0000 7f6ca759e700 1 -- 172.21.15.187:0/3731725377 shutdown_connections 2020-11-20T12:06:11.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.835+0000 7f6ca759e700 1 -- 172.21.15.187:0/3731725377 wait complete. 2020-11-20T12:06:11.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.836+0000 7f6ca759e700 1 Processor -- start 2020-11-20T12:06:11.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.836+0000 7f6ca759e700 1 -- start start 2020-11-20T12:06:11.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca809df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ca8066f70 0x7f6ca815c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca81484d0 0x7f6ca809c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca809e570 con 0x7f6ca8066f70 2020-11-20T12:06:11.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca809e810 con 0x7f6ca80622c0 2020-11-20T12:06:11.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca809a670 con 0x7f6ca81484d0 2020-11-20T12:06:11.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca809df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca809df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42104/0 (socket says 172.21.15.187:42104) 2020-11-20T12:06:11.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.837+0000 7f6ca659c700 1 -- 172.21.15.187:0/3215614117 learned_addr learned my addr 172.21.15.187:0/3215614117 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:11.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6ca659c700 1 -- 172.21.15.187:0/3215614117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca81484d0 msgr2=0x7f6ca809c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6ca659c700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca81484d0 0x7f6ca809c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:11.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6ca659c700 1 -- 172.21.15.187:0/3215614117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ca8066f70 msgr2=0x7f6ca815c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:11.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6c9dd9b700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ca8066f70 0x7f6ca815c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6ca659c700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ca8066f70 0x7f6ca815c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:11.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6ca659c700 1 -- 172.21.15.187:0/3215614117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca809c760 con 0x7f6ca80622c0 2020-11-20T12:06:11.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6ca659c700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca809df40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f6c9800d8e0 tx=0x7f6c98008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:11.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6c9effd700 1 -- 172.21.15.187:0/3215614117 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98009790 con 0x7f6ca80622c0 2020-11-20T12:06:11.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6c9effd700 1 -- 172.21.15.187:0/3215614117 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c980098f0 con 0x7f6ca80622c0 2020-11-20T12:06:11.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca809cfd0 con 0x7f6ca80622c0 2020-11-20T12:06:11.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.838+0000 7f6c9effd700 1 -- 172.21.15.187:0/3215614117 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98009c90 con 0x7f6ca80622c0 2020-11-20T12:06:11.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.839+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca809d4e0 con 0x7f6ca80622c0 2020-11-20T12:06:11.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.840+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca809a830 con 0x7f6ca80622c0 2020-11-20T12:06:11.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.840+0000 7f6c9effd700 1 -- 172.21.15.187:0/3215614117 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6c98013480 con 0x7f6ca80622c0 2020-11-20T12:06:11.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.841+0000 7f6c9effd700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c84056d50 0x7f6c84059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:11.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.841+0000 7f6c9effd700 1 -- 172.21.15.187:0/3215614117 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6c9806fea0 con 0x7f6ca80622c0 2020-11-20T12:06:11.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.841+0000 7f6c9dd9b700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c84056d50 0x7f6c84059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:11.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.842+0000 7f6c9dd9b700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c84056d50 0x7f6c84059210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f6ca80bb680 tx=0x7f6c90015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:11.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:11.845+0000 7f6c9effd700 1 -- 172.21.15.187:0/3215614117 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6c98045290 con 0x7f6ca80622c0 2020-11-20T12:06:12.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.052+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ca8061a20 con 0x7f6ca80622c0 2020-11-20T12:06:12.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.059+0000 7f6c9effd700 1 -- 172.21.15.187:0/3215614117 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+16791 (secure 0 0 0) 0x7f6c980488b0 con 0x7f6ca80622c0 2020-11-20T12:06:12.063 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:12.064 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":31,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:06:11.574192+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:06:09.312569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2020-11-20T12:06:10.581583+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:06:12.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c84056d50 msgr2=0x7f6c84059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c84056d50 0x7f6c84059210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f6ca80bb680 tx=0x7f6c90015000).stop 2020-11-20T12:06:12.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 msgr2=0x7f6ca809df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca809df40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f6c9800d8e0 tx=0x7f6c98008040).stop 2020-11-20T12:06:12.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 shutdown_connections 2020-11-20T12:06:12.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c84056d50 0x7f6c84059210 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ca81484d0 0x7f6ca809c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:12.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca80622c0 0x7f6ca809df40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 --2- 172.21.15.187:0/3215614117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ca8066f70 0x7f6ca815c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.061+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 >> 172.21.15.187:0/3215614117 conn(0x7f6ca80bab30 msgr2=0x7f6ca80b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:12.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.062+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 shutdown_connections 2020-11-20T12:06:12.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.062+0000 7f6ca759e700 1 -- 172.21.15.187:0/3215614117 wait complete. 2020-11-20T12:06:12.088 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:06:12.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.323+0000 7fd9229ab700 1 Processor -- start 2020-11-20T12:06:12.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.323+0000 7fd9229ab700 1 -- start start 2020-11-20T12:06:12.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.324+0000 7fd9229ab700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd91c14d270 0x7fd91c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.324+0000 7fd9229ab700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0658a0 con 0x7fd91c0b1760 2020-11-20T12:06:12.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.324+0000 7fd9229ab700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c150390 con 0x7fd91c0b1380 2020-11-20T12:06:12.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.324+0000 7fd9229ab700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91c1511e0 con 0x7fd91c14d270 2020-11-20T12:06:12.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.325+0000 7fd9221aa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd91c14d270 0x7fd91c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.325+0000 7fd9219a9700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd91c0b1380 0x7fd91c150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38716/0 (socket says 172.21.15.187:38716) 2020-11-20T12:06:12.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.325+0000 7fd9219a9700 1 -- 172.21.15.187:0/2893089394 learned_addr learned my addr 172.21.15.187:0/2893089394 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:12.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.325+0000 7fd9221aa700 1 -- 172.21.15.187:0/2893089394 >> v1:172.21.15.187:6789/0 conn(0x7fd91c0b1380 legacy=0x7fd91c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.325+0000 7fd9221aa700 1 -- 172.21.15.187:0/2893089394 >> v1:172.21.15.5:6789/0 conn(0x7fd91c0b1760 legacy=0x7fd91c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:12.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.326+0000 7fd9221aa700 1 -- 172.21.15.187:0/2893089394 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c150630 con 0x7fd91c14d270 2020-11-20T12:06:12.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.326+0000 7fd9221aa700 1 --2- 172.21.15.187:0/2893089394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c14d270 0x7fd91c1573a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fd914009c90 tx=0x7fd9140091d0).ready entity=mon.1 client_cookie=47b2cde2b90bbb1d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:12.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.326+0000 7fd9209a7700 1 -- 172.21.15.187:0/2893089394 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91400bb80 con 0x7fd91c14d270 2020-11-20T12:06:12.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.326+0000 7fd9209a7700 1 -- 172.21.15.187:0/2893089394 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91400bce0 con 0x7fd91c14d270 2020-11-20T12:06:12.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.326+0000 7fd9209a7700 1 -- 172.21.15.187:0/2893089394 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914011610 con 0x7fd91c14d270 2020-11-20T12:06:12.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.327+0000 7fd9229ab700 1 -- 172.21.15.187:0/2893089394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c14d270 msgr2=0x7fd91c1573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.327+0000 7fd9229ab700 1 --2- 172.21.15.187:0/2893089394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c14d270 0x7fd91c1573a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fd914009c90 tx=0x7fd9140091d0).stop 2020-11-20T12:06:12.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.327+0000 7fd9229ab700 1 -- 172.21.15.187:0/2893089394 shutdown_connections 2020-11-20T12:06:12.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.327+0000 7fd9229ab700 1 --2- 172.21.15.187:0/2893089394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c14d270 0x7fd91c1573a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.327+0000 7fd9229ab700 1 -- 172.21.15.187:0/2893089394 >> 172.21.15.187:0/2893089394 conn(0x7fd91c0bab30 msgr2=0x7fd91c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:12.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.327+0000 7fd9229ab700 1 -- 172.21.15.187:0/2893089394 shutdown_connections 2020-11-20T12:06:12.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.327+0000 7fd9229ab700 1 -- 172.21.15.187:0/2893089394 wait complete. 2020-11-20T12:06:12.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.328+0000 7fd9229ab700 1 Processor -- start 2020-11-20T12:06:12.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.328+0000 7fd9229ab700 1 -- start start 2020-11-20T12:06:12.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.329+0000 7fd9229ab700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd91c0b1380 0x7fd91c0a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.329+0000 7fd9229ab700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c0b1760 0x7fd91c1484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.329+0000 7fd9229ab700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd91c14d270 0x7fd91c1489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.329+0000 7fd9229ab700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c148ec0 con 0x7fd91c0b1380 2020-11-20T12:06:12.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.329+0000 7fd9229ab700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c09f8e0 con 0x7fd91c0b1760 2020-11-20T12:06:12.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.329+0000 7fd9229ab700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c09fb80 con 0x7fd91c14d270 2020-11-20T12:06:12.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c0b1760 0x7fd91c1484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c0b1760 0x7fd91c1484e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42118/0 (socket says 172.21.15.187:42118) 2020-11-20T12:06:12.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 -- 172.21.15.187:0/2788413287 learned_addr learned my addr 172.21.15.187:0/2788413287 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:12.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9219a9700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd91c0b1380 0x7fd91c0a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 -- 172.21.15.187:0/2788413287 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd91c14d270 msgr2=0x7fd91c1489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd91c14d270 0x7fd91c1489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:12.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 -- 172.21.15.187:0/2788413287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd91c0b1380 msgr2=0x7fd91c0a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd91c0b1380 0x7fd91c0a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 -- 172.21.15.187:0/2788413287 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c09af90 con 0x7fd91c0b1760 2020-11-20T12:06:12.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.330+0000 7fd9211a8700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c0b1760 0x7fd91c1484e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fd90c00c760 tx=0x7fd90c007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:12.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.331+0000 7fd909ffb700 1 -- 172.21.15.187:0/2788413287 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c00e560 con 0x7fd91c0b1760 2020-11-20T12:06:12.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.331+0000 7fd909ffb700 1 -- 172.21.15.187:0/2788413287 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c00e6e0 con 0x7fd91c0b1760 2020-11-20T12:06:12.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.331+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c0a0ac0 con 0x7fd91c0b1760 2020-11-20T12:06:12.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.331+0000 7fd909ffb700 1 -- 172.21.15.187:0/2788413287 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c00eaa0 con 0x7fd91c0b1760 2020-11-20T12:06:12.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.331+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c09f480 con 0x7fd91c0b1760 2020-11-20T12:06:12.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.332+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c09b120 con 0x7fd91c0b1760 2020-11-20T12:06:12.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.333+0000 7fd909ffb700 1 -- 172.21.15.187:0/2788413287 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd90c010070 con 0x7fd91c0b1760 2020-11-20T12:06:12.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.333+0000 7fd909ffb700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8f4056d20 0x7fd8f40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.333+0000 7fd909ffb700 1 -- 172.21.15.187:0/2788413287 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd90c06fbd0 con 0x7fd91c0b1760 2020-11-20T12:06:12.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.333+0000 7fd9219a9700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8f4056d20 0x7fd8f40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.334+0000 7fd9219a9700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8f4056d20 0x7fd8f40591e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd918007fd0 tx=0x7fd918009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:12.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.337+0000 7fd909ffb700 1 -- 172.21.15.187:0/2788413287 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd90c045020 con 0x7fd91c0b1760 2020-11-20T12:06:12.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.523+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd91c061a20 con 0x7fd91c0b1760 2020-11-20T12:06:12.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.524+0000 7fd909ffb700 1 -- 172.21.15.187:0/2788413287 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+16791 (secure 0 0 0) 0x7fd90c048640 con 0x7fd91c0b1760 2020-11-20T12:06:12.528 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:12.529 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":31,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:06:11.574192+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:06:09.312569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2020-11-20T12:06:10.581583+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:06:12.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8f4056d20 msgr2=0x7fd8f40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8f4056d20 0x7fd8f40591e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd918007fd0 tx=0x7fd918009040).stop 2020-11-20T12:06:12.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c0b1760 msgr2=0x7fd91c1484e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c0b1760 0x7fd91c1484e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fd90c00c760 tx=0x7fd90c007c30).stop 2020-11-20T12:06:12.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 shutdown_connections 2020-11-20T12:06:12.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8f4056d20 0x7fd8f40591e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd91c0b1380 0x7fd91c0a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd91c14d270 0x7fd91c1489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:12.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 --2- 172.21.15.187:0/2788413287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd91c0b1760 0x7fd91c1484e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 >> 172.21.15.187:0/2788413287 conn(0x7fd91c0bab30 msgr2=0x7fd91c14f290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:12.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 shutdown_connections 2020-11-20T12:06:12.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.526+0000 7fd9229ab700 1 -- 172.21.15.187:0/2788413287 wait complete. 2020-11-20T12:06:12.551 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:06:12.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.785+0000 7f812759e700 1 Processor -- start 2020-11-20T12:06:12.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.786+0000 7f812759e700 1 -- start start 2020-11-20T12:06:12.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.787+0000 7f812759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1760 0x7f8128065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.787+0000 7f812759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8128150390 con 0x7f81280b1380 2020-11-20T12:06:12.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.787+0000 7f812759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8128150630 con 0x7f8128065680 2020-11-20T12:06:12.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.787+0000 7f812759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81281511e0 con 0x7f81280b1760 2020-11-20T12:06:12.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.788+0000 7f8126d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8128065680 0x7f812814c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38734/0 (socket says 172.21.15.187:38734) 2020-11-20T12:06:12.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.788+0000 7f8126d9d700 1 -- 172.21.15.187:0/1279460028 learned_addr learned my addr 172.21.15.187:0/1279460028 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:12.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.788+0000 7f8125d9b700 1 --2- 172.21.15.187:0/1279460028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1760 0x7f8128065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.789+0000 7f812559a700 1 -- 172.21.15.187:0/1279460028 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104511191 0 0) 0x7f8128150630 con 0x7f8128065680 2020-11-20T12:06:12.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.789+0000 7f812559a700 1 -- 172.21.15.187:0/1279460028 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f810c003240 con 0x7f8128065680 2020-11-20T12:06:12.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.789+0000 7f8125d9b700 1 -- 172.21.15.187:0/1279460028 >> v1:172.21.15.187:6789/0 conn(0x7f8128065680 legacy=0x7f812814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.789+0000 7f8125d9b700 1 -- 172.21.15.187:0/1279460028 >> v1:172.21.15.5:6789/0 conn(0x7f81280b1380 legacy=0x7f8128150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.789+0000 7f812559a700 1 -- 172.21.15.187:0/1279460028 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1460583117 0 0) 0x7f810c003240 con 0x7f8128065680 2020-11-20T12:06:12.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.789+0000 7f8125d9b700 1 -- 172.21.15.187:0/1279460028 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8128068510 con 0x7f81280b1760 2020-11-20T12:06:12.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.790+0000 7f8125d9b700 1 --2- 172.21.15.187:0/1279460028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1760 0x7f8128065190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f811c00b9d0 tx=0x7f811c006df0).ready entity=mon.2 client_cookie=6f276bb34d83b6a6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:12.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.791+0000 7f812559a700 1 -- 172.21.15.187:0/1279460028 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c0097b0 con 0x7f81280b1760 2020-11-20T12:06:12.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.791+0000 7f812559a700 1 -- 172.21.15.187:0/1279460028 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c009910 con 0x7f81280b1760 2020-11-20T12:06:12.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.791+0000 7f812559a700 1 -- 172.21.15.187:0/1279460028 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c009c90 con 0x7f81280b1760 2020-11-20T12:06:12.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.792+0000 7f812759e700 1 -- 172.21.15.187:0/1279460028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1760 msgr2=0x7f8128065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.792+0000 7f812759e700 1 --2- 172.21.15.187:0/1279460028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1760 0x7f8128065190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f811c00b9d0 tx=0x7f811c006df0).stop 2020-11-20T12:06:12.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.792+0000 7f812759e700 1 -- 172.21.15.187:0/1279460028 shutdown_connections 2020-11-20T12:06:12.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.792+0000 7f812759e700 1 --2- 172.21.15.187:0/1279460028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1760 0x7f8128065190 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.792+0000 7f812759e700 1 -- 172.21.15.187:0/1279460028 >> 172.21.15.187:0/1279460028 conn(0x7f81280bab30 msgr2=0x7f81280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:12.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.792+0000 7f812759e700 1 -- 172.21.15.187:0/1279460028 shutdown_connections 2020-11-20T12:06:12.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.792+0000 7f812759e700 1 -- 172.21.15.187:0/1279460028 wait complete. 2020-11-20T12:06:12.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.794+0000 7f812759e700 1 Processor -- start 2020-11-20T12:06:12.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.794+0000 7f812759e700 1 -- start start 2020-11-20T12:06:12.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.795+0000 7f812759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1380 0x7f8128062350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.795+0000 7f812759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81280b1760 0x7f81280b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.795+0000 7f812759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f812809e4f0 0x7f81280b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.795+0000 7f812759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81280b1cc0 con 0x7f812809e4f0 2020-11-20T12:06:12.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.795+0000 7f812759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81280b1f60 con 0x7f81280b1760 2020-11-20T12:06:12.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.795+0000 7f812759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8128062840 con 0x7f81280b1380 2020-11-20T12:06:12.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.795+0000 7f8125d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81280b1760 0x7f81280b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8125d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81280b1760 0x7f81280b2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42132/0 (socket says 172.21.15.187:42132) 2020-11-20T12:06:12.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8125d9b700 1 -- 172.21.15.187:0/3061861295 learned_addr learned my addr 172.21.15.187:0/3061861295 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:12.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f812659c700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1380 0x7f8128062350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8126d9d700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f812809e4f0 0x7f81280b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8125d9b700 1 -- 172.21.15.187:0/3061861295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1380 msgr2=0x7f8128062350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8125d9b700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1380 0x7f8128062350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8125d9b700 1 -- 172.21.15.187:0/3061861295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f812809e4f0 msgr2=0x7f81280b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:12.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8125d9b700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f812809e4f0 0x7f81280b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:12.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8125d9b700 1 -- 172.21.15.187:0/3061861295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81280b4810 con 0x7f81280b1760 2020-11-20T12:06:12.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.796+0000 7f8125d9b700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81280b1760 0x7f81280b2400 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f811c0098e0 tx=0x7f811c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:12.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.797+0000 7f81167fc700 1 -- 172.21.15.187:0/3061861295 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c011d50 con 0x7f81280b1760 2020-11-20T12:06:12.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.797+0000 7f81167fc700 1 -- 172.21.15.187:0/3061861295 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c01e460 con 0x7f81280b1760 2020-11-20T12:06:12.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.797+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81280b70c0 con 0x7f81280b1760 2020-11-20T12:06:12.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.797+0000 7f81167fc700 1 -- 172.21.15.187:0/3061861295 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c01e820 con 0x7f81280b1760 2020-11-20T12:06:12.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.797+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812809eb60 con 0x7f81280b1760 2020-11-20T12:06:12.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.798+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81280b2120 con 0x7f81280b1760 2020-11-20T12:06:12.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.798+0000 7f81167fc700 1 -- 172.21.15.187:0/3061861295 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f811c01a040 con 0x7f81280b1760 2020-11-20T12:06:12.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.799+0000 7f81167fc700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80fc056d50 0x7f80fc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:12.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.799+0000 7f81167fc700 1 -- 172.21.15.187:0/3061861295 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f811c071510 con 0x7f81280b1760 2020-11-20T12:06:12.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.799+0000 7f812659c700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80fc056d50 0x7f80fc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:12.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.800+0000 7f812659c700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80fc056d50 0x7f80fc059210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8118003c00 tx=0x7f8118009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:12.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:12.804+0000 7f81167fc700 1 -- 172.21.15.187:0/3061861295 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f811c046990 con 0x7f81280b1760 2020-11-20T12:06:13.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:13.056+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f81280b49a0 con 0x7f81280b1760 2020-11-20T12:06:14.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.073+0000 7f81167fc700 1 -- 172.21.15.187:0/3061861295 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7f811c023020 con 0x7f81280b1760 2020-11-20T12:06:14.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.076+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80fc056d50 msgr2=0x7f80fc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.076+0000 7f812759e700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80fc056d50 0x7f80fc059210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8118003c00 tx=0x7f8118009040).stop 2020-11-20T12:06:14.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.076+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81280b1760 msgr2=0x7f81280b2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.076+0000 7f812759e700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81280b1760 0x7f81280b2400 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f811c0098e0 tx=0x7f811c015000).stop 2020-11-20T12:06:14.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.077+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 shutdown_connections 2020-11-20T12:06:14.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.077+0000 7f812759e700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80fc056d50 0x7f80fc059210 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.077+0000 7f812759e700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f812809e4f0 0x7f81280b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.077+0000 7f812759e700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81280b1380 0x7f8128062350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.077+0000 7f812759e700 1 --2- 172.21.15.187:0/3061861295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81280b1760 0x7f81280b2400 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.077+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 >> 172.21.15.187:0/3061861295 conn(0x7f81280bab30 msgr2=0x7f81281491d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:14.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.078+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 shutdown_connections 2020-11-20T12:06:14.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.078+0000 7f812759e700 1 -- 172.21.15.187:0/3061861295 wait complete. 2020-11-20T12:06:14.112 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:14.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.361+0000 7fb536f40700 1 Processor -- start 2020-11-20T12:06:14.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.361+0000 7fb536f40700 1 -- start start 2020-11-20T12:06:14.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.362+0000 7fb536f40700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb530065f90 0x7fb530065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.362+0000 7fb536f40700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 0x7fb53009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.362+0000 7fb536f40700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5300b13e0 con 0x7fb530065190 2020-11-20T12:06:14.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.362+0000 7fb536f40700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53014ab30 con 0x7fb530065f90 2020-11-20T12:06:14.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.362+0000 7fb536f40700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb530062c10 con 0x7fb53009ca90 2020-11-20T12:06:14.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.363+0000 7fb53573d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb530065f90 0x7fb530065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.363+0000 7fb53573d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb530065f90 0x7fb530065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:34404/0 (socket says 172.21.15.187:34404) 2020-11-20T12:06:14.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.363+0000 7fb53573d700 1 -- 172.21.15.187:0/3248243437 learned_addr learned my addr 172.21.15.187:0/3248243437 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:14.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.363+0000 7fb53673f700 1 --2- 172.21.15.187:0/3248243437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 0x7fb53009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.364+0000 7fb53573d700 1 -- 172.21.15.187:0/3248243437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 msgr2=0x7fb53009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:14.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.364+0000 7fb53573d700 1 --2- 172.21.15.187:0/3248243437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 0x7fb53009cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.364+0000 7fb53573d700 1 -- 172.21.15.187:0/3248243437 >> v1:172.21.15.5:6789/0 conn(0x7fb530065190 legacy=0x7fb530143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.364+0000 7fb53573d700 1 -- 172.21.15.187:0/3248243437 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5300b1640 con 0x7fb530065f90 2020-11-20T12:06:14.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.365+0000 7fb53573d700 1 --2- 172.21.15.187:0/3248243437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065f90 0x7fb530065570 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb520006fd0 tx=0x7fb520006eb0).ready entity=mon.0 client_cookie=615af1992deccd55 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:14.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.365+0000 7fb534f3c700 1 -- 172.21.15.187:0/3248243437 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200098a0 con 0x7fb530065f90 2020-11-20T12:06:14.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.365+0000 7fb534f3c700 1 -- 172.21.15.187:0/3248243437 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520009a00 con 0x7fb530065f90 2020-11-20T12:06:14.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.365+0000 7fb534f3c700 1 -- 172.21.15.187:0/3248243437 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520009d80 con 0x7fb530065f90 2020-11-20T12:06:14.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.366+0000 7fb536f40700 1 -- 172.21.15.187:0/3248243437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065f90 msgr2=0x7fb530065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.366+0000 7fb536f40700 1 --2- 172.21.15.187:0/3248243437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065f90 0x7fb530065570 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb520006fd0 tx=0x7fb520006eb0).stop 2020-11-20T12:06:14.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.366+0000 7fb536f40700 1 -- 172.21.15.187:0/3248243437 shutdown_connections 2020-11-20T12:06:14.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.366+0000 7fb536f40700 1 --2- 172.21.15.187:0/3248243437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 0x7fb53009cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.366+0000 7fb536f40700 1 --2- 172.21.15.187:0/3248243437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065f90 0x7fb530065570 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.366+0000 7fb536f40700 1 -- 172.21.15.187:0/3248243437 >> 172.21.15.187:0/3248243437 conn(0x7fb5300bab30 msgr2=0x7fb5300baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:14.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.366+0000 7fb536f40700 1 -- 172.21.15.187:0/3248243437 shutdown_connections 2020-11-20T12:06:14.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.366+0000 7fb536f40700 1 -- 172.21.15.187:0/3248243437 wait complete. 2020-11-20T12:06:14.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.368+0000 7fb536f40700 1 Processor -- start 2020-11-20T12:06:14.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.368+0000 7fb536f40700 1 -- start start 2020-11-20T12:06:14.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb536f40700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065190 0x7fb530067400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb536f40700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb530065f90 0x7fb530146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb536f40700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 0x7fb5300b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb536f40700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb530067ed0 con 0x7fb530065190 2020-11-20T12:06:14.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb536f40700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb530067c30 con 0x7fb530065f90 2020-11-20T12:06:14.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb536f40700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53015bc30 con 0x7fb53009ca90 2020-11-20T12:06:14.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb53573d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb530065f90 0x7fb530146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb53573d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb530065f90 0x7fb530146830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42146/0 (socket says 172.21.15.187:42146) 2020-11-20T12:06:14.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb53573d700 1 -- 172.21.15.187:0/3206300164 learned_addr learned my addr 172.21.15.187:0/3206300164 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:14.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb535f3e700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065190 0x7fb530067400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.370+0000 7fb53673f700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 0x7fb5300b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb53573d700 1 -- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 msgr2=0x7fb5300b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb53573d700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 0x7fb5300b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb53573d700 1 -- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065190 msgr2=0x7fb530067400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb53573d700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065190 0x7fb530067400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb53573d700 1 -- 172.21.15.187:0/3206300164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5300678f0 con 0x7fb530065f90 2020-11-20T12:06:14.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb535f3e700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065190 0x7fb530067400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:06:14.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb53573d700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb530065f90 0x7fb530146830 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fb52000b1b0 tx=0x7fb520015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:14.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb51dffb700 1 -- 172.21.15.187:0/3206300164 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520011e50 con 0x7fb530065f90 2020-11-20T12:06:14.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb51dffb700 1 -- 172.21.15.187:0/3206300164 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52001e460 con 0x7fb530065f90 2020-11-20T12:06:14.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.371+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb530146d20 con 0x7fb530065f90 2020-11-20T12:06:14.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.372+0000 7fb51dffb700 1 -- 172.21.15.187:0/3206300164 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52001e800 con 0x7fb530065f90 2020-11-20T12:06:14.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.372+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5300b2030 con 0x7fb530065f90 2020-11-20T12:06:14.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.373+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb530061a20 con 0x7fb530065f90 2020-11-20T12:06:14.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.373+0000 7fb51dffb700 1 -- 172.21.15.187:0/3206300164 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb52001a040 con 0x7fb530065f90 2020-11-20T12:06:14.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.374+0000 7fb51dffb700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb508056d20 0x7fb5080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.374+0000 7fb51dffb700 1 -- 172.21.15.187:0/3206300164 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb5200715f0 con 0x7fb530065f90 2020-11-20T12:06:14.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.374+0000 7fb535f3e700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb508056d20 0x7fb5080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.375+0000 7fb535f3e700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb508056d20 0x7fb5080591e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb52c001570 tx=0x7fb52c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:14.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.378+0000 7fb51dffb700 1 -- 172.21.15.187:0/3206300164 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb5200469e0 con 0x7fb530065f90 2020-11-20T12:06:14.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.626+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb53015bda0 con 0x7fb530065f90 2020-11-20T12:06:14.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.627+0000 7fb51dffb700 1 -- 172.21.15.187:0/3206300164 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7fb520023070 con 0x7fb530065f90 2020-11-20T12:06:14.632 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:14.632 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:14.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb508056d20 msgr2=0x7fb5080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb508056d20 0x7fb5080591e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb52c001570 tx=0x7fb52c009040).stop 2020-11-20T12:06:14.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb530065f90 msgr2=0x7fb530146830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb530065f90 0x7fb530146830 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fb52000b1b0 tx=0x7fb520015000).stop 2020-11-20T12:06:14.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 shutdown_connections 2020-11-20T12:06:14.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb508056d20 0x7fb5080591e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb530065190 0x7fb530067400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53009ca90 0x7fb5300b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 --2- 172.21.15.187:0/3206300164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb530065f90 0x7fb530146830 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 >> 172.21.15.187:0/3206300164 conn(0x7fb5300bab30 msgr2=0x7fb5300ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:14.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 shutdown_connections 2020-11-20T12:06:14.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.630+0000 7fb536f40700 1 -- 172.21.15.187:0/3206300164 wait complete. 2020-11-20T12:06:14.637 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:14.658 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:06:14.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.892+0000 7f70ebfff700 1 Processor -- start 2020-11-20T12:06:14.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.892+0000 7f70ebfff700 1 -- start start 2020-11-20T12:06:14.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.893+0000 7f70ebfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70ec0b6650 0x7f70ec0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.893+0000 7f70ebfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f70ec066f70 0x7f70ec0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.893+0000 7f70ebfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0b1300 con 0x7f70ec1484d0 2020-11-20T12:06:14.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.893+0000 7f70ebfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec067730 con 0x7f70ec066f70 2020-11-20T12:06:14.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.893+0000 7f70ebfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec0b15a0 con 0x7f70ec0b6650 2020-11-20T12:06:14.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.894+0000 7f70e27fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70ec0b6650 0x7f70ec0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.894+0000 7f70e27fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70ec0b6650 0x7f70ec0afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42154/0 (socket says 172.21.15.187:42154) 2020-11-20T12:06:14.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.894+0000 7f70e27fc700 1 -- 172.21.15.187:0/3128031078 learned_addr learned my addr 172.21.15.187:0/3128031078 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:14.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.894+0000 7f70eb7fe700 1 --2- 172.21.15.187:0/3128031078 >> v2:172.21.15.5:3300/0 conn(0x7f70ec066f70 0x7f70ec0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.894+0000 7f70e27fc700 1 -- 172.21.15.187:0/3128031078 >> v2:172.21.15.5:3300/0 conn(0x7f70ec066f70 msgr2=0x7f70ec0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:14.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.894+0000 7f70e27fc700 1 --2- 172.21.15.187:0/3128031078 >> v2:172.21.15.5:3300/0 conn(0x7f70ec066f70 0x7f70ec0aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.894+0000 7f70e27fc700 1 -- 172.21.15.187:0/3128031078 >> v1:172.21.15.5:6789/0 conn(0x7f70ec1484d0 legacy=0x7f70ec061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:14.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.894+0000 7f70e27fc700 1 -- 172.21.15.187:0/3128031078 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec0b1840 con 0x7f70ec0b6650 2020-11-20T12:06:14.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.895+0000 7f70e27fc700 1 --2- 172.21.15.187:0/3128031078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0afa90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f70ec068600 tx=0x7f70d4006eb0).ready entity=mon.1 client_cookie=97926b443d397652 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:14.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.895+0000 7f70ea7fc700 1 -- 172.21.15.187:0/3128031078 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4009890 con 0x7f70ec0b6650 2020-11-20T12:06:14.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.895+0000 7f70ea7fc700 1 -- 172.21.15.187:0/3128031078 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d40099f0 con 0x7f70ec0b6650 2020-11-20T12:06:14.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.895+0000 7f70ea7fc700 1 -- 172.21.15.187:0/3128031078 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4009d70 con 0x7f70ec0b6650 2020-11-20T12:06:14.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.895+0000 7f70ebfff700 1 -- 172.21.15.187:0/3128031078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 msgr2=0x7f70ec0afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.895+0000 7f70ebfff700 1 --2- 172.21.15.187:0/3128031078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0afa90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f70ec068600 tx=0x7f70d4006eb0).stop 2020-11-20T12:06:14.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.896+0000 7f70ebfff700 1 -- 172.21.15.187:0/3128031078 shutdown_connections 2020-11-20T12:06:14.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.896+0000 7f70ebfff700 1 --2- 172.21.15.187:0/3128031078 >> v2:172.21.15.5:3300/0 conn(0x7f70ec066f70 0x7f70ec0aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.896+0000 7f70ebfff700 1 --2- 172.21.15.187:0/3128031078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0afa90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.896+0000 7f70ebfff700 1 -- 172.21.15.187:0/3128031078 >> 172.21.15.187:0/3128031078 conn(0x7f70ec0bab30 msgr2=0x7f70ec0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:14.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.896+0000 7f70ebfff700 1 -- 172.21.15.187:0/3128031078 shutdown_connections 2020-11-20T12:06:14.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.896+0000 7f70ebfff700 1 -- 172.21.15.187:0/3128031078 wait complete. 2020-11-20T12:06:14.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.897+0000 7f70ebfff700 1 Processor -- start 2020-11-20T12:06:14.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.897+0000 7f70ebfff700 1 -- start start 2020-11-20T12:06:14.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.898+0000 7f70ebfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec066f70 0x7f70ec15c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.898+0000 7f70ebfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.898+0000 7f70ebfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec1484d0 0x7f70ec0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.898+0000 7f70ebfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec09e4d0 con 0x7f70ec066f70 2020-11-20T12:06:14.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.898+0000 7f70ebfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0b22e0 con 0x7f70ec0b6650 2020-11-20T12:06:14.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.898+0000 7f70ebfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec09e230 con 0x7f70ec1484d0 2020-11-20T12:06:14.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42160/0 (socket says 172.21.15.187:42160) 2020-11-20T12:06:14.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 -- 172.21.15.187:0/2564497655 learned_addr learned my addr 172.21.15.187:0/2564497655 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:14.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 -- 172.21.15.187:0/2564497655 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec1484d0 msgr2=0x7f70ec0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70eb7fe700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec1484d0 0x7f70ec0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70eaffd700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec066f70 0x7f70ec15c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec1484d0 0x7f70ec0b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 -- 172.21.15.187:0/2564497655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec066f70 msgr2=0x7f70ec15c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:14.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec066f70 0x7f70ec15c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:14.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 -- 172.21.15.187:0/2564497655 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec09c510 con 0x7f70ec0b6650 2020-11-20T12:06:14.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.899+0000 7f70e27fc700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0b1df0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f70ec0b16e0 tx=0x7f70d4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:14.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.900+0000 7f70e37fe700 1 -- 172.21.15.187:0/2564497655 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d401e460 con 0x7f70ec0b6650 2020-11-20T12:06:14.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.900+0000 7f70e37fe700 1 -- 172.21.15.187:0/2564497655 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d401e5c0 con 0x7f70ec0b6650 2020-11-20T12:06:14.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.900+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec0b4680 con 0x7f70ec0b6650 2020-11-20T12:06:14.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.900+0000 7f70e37fe700 1 -- 172.21.15.187:0/2564497655 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d401e940 con 0x7f70ec0b6650 2020-11-20T12:06:14.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.900+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec09de00 con 0x7f70ec0b6650 2020-11-20T12:06:14.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.901+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ec152bf0 con 0x7f70ec0b6650 2020-11-20T12:06:14.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.901+0000 7f70e37fe700 1 -- 172.21.15.187:0/2564497655 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f70d4025030 con 0x7f70ec0b6650 2020-11-20T12:06:14.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.902+0000 7f70e37fe700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c8056d20 0x7f70c80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:14.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.902+0000 7f70e37fe700 1 -- 172.21.15.187:0/2564497655 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f70d40717e0 con 0x7f70ec0b6650 2020-11-20T12:06:14.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.902+0000 7f70eaffd700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c8056d20 0x7f70c80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:14.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.903+0000 7f70eaffd700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c8056d20 0x7f70c80591e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f70dc00a550 tx=0x7f70dc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:14.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:14.906+0000 7f70e37fe700 1 -- 172.21.15.187:0/2564497655 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f70d4046ce0 con 0x7f70ec0b6650 2020-11-20T12:06:15.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.089+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70ec09c6a0 con 0x7f70ec0b6650 2020-11-20T12:06:15.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.090+0000 7f70e37fe700 1 -- 172.21.15.187:0/2564497655 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f70d40311a0 con 0x7f70ec0b6650 2020-11-20T12:06:15.095 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:15.095 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":32,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:06:12.578719+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:06:09.312569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2020-11-20T12:06:10.581583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:06:15.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.092+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c8056d20 msgr2=0x7f70c80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.092+0000 7f70ebfff700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c8056d20 0x7f70c80591e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f70dc00a550 tx=0x7f70dc008040).stop 2020-11-20T12:06:15.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.092+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 msgr2=0x7f70ec0b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.092+0000 7f70ebfff700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0b1df0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f70ec0b16e0 tx=0x7f70d4015000).stop 2020-11-20T12:06:15.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.093+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 shutdown_connections 2020-11-20T12:06:15.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.093+0000 7f70ebfff700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c8056d20 0x7f70c80591e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.093+0000 7f70ebfff700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec066f70 0x7f70ec15c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.093+0000 7f70ebfff700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec1484d0 0x7f70ec0b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.093+0000 7f70ebfff700 1 --2- 172.21.15.187:0/2564497655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec0b6650 0x7f70ec0b1df0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.093+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 >> 172.21.15.187:0/2564497655 conn(0x7f70ec0bab30 msgr2=0x7f70ec066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:15.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.093+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 shutdown_connections 2020-11-20T12:06:15.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.093+0000 7f70ebfff700 1 -- 172.21.15.187:0/2564497655 wait complete. 2020-11-20T12:06:15.119 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:06:15.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.349+0000 7fbb8b4a4700 1 Processor -- start 2020-11-20T12:06:15.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.349+0000 7fbb8b4a4700 1 -- start start 2020-11-20T12:06:15.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.351+0000 7fbb8b4a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb840b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.351+0000 7fbb8b4a4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb840b11d0 0x7fbb84066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.351+0000 7fbb8b4a4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb84149790 con 0x7fbb840b1c20 2020-11-20T12:06:15.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.351+0000 7fbb8b4a4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb84065ba0 con 0x7fbb840b11d0 2020-11-20T12:06:15.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.351+0000 7fbb8b4a4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb840b8ca0 con 0x7fbb840b1840 2020-11-20T12:06:15.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.351+0000 7fbb8aca3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb840b11d0 0x7fbb84066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.351+0000 7fbb8aca3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb840b11d0 0x7fbb84066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42168/0 (socket says 172.21.15.187:42168) 2020-11-20T12:06:15.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.351+0000 7fbb8aca3700 1 -- 172.21.15.187:0/272984542 learned_addr learned my addr 172.21.15.187:0/272984542 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:15.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.352+0000 7fbb8aca3700 1 -- 172.21.15.187:0/272984542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 msgr2=0x7fbb840b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:15.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.352+0000 7fbb8a4a2700 1 --2- 172.21.15.187:0/272984542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb840b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.352+0000 7fbb8aca3700 1 --2- 172.21.15.187:0/272984542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb840b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.352+0000 7fbb8aca3700 1 -- 172.21.15.187:0/272984542 >> v1:172.21.15.5:6789/0 conn(0x7fbb840b1c20 legacy=0x7fbb840b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.352+0000 7fbb8aca3700 1 -- 172.21.15.187:0/272984542 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb84149450 con 0x7fbb840b11d0 2020-11-20T12:06:15.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.352+0000 7fbb8a4a2700 1 --2- 172.21.15.187:0/272984542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb840b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:15.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.352+0000 7fbb8aca3700 1 --2- 172.21.15.187:0/272984542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b11d0 0x7fbb84066040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fbb7c009d50 tx=0x7fbb7c009290).ready entity=mon.1 client_cookie=e8d4021e4a61b7a1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:15.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.352+0000 7fbb894a0700 1 -- 172.21.15.187:0/272984542 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c00bd10 con 0x7fbb840b11d0 2020-11-20T12:06:15.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb894a0700 1 -- 172.21.15.187:0/272984542 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7c00be70 con 0x7fbb840b11d0 2020-11-20T12:06:15.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb894a0700 1 -- 172.21.15.187:0/272984542 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c011610 con 0x7fbb840b11d0 2020-11-20T12:06:15.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/272984542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b11d0 msgr2=0x7fbb84066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/272984542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b11d0 0x7fbb84066040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fbb7c009d50 tx=0x7fbb7c009290).stop 2020-11-20T12:06:15.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/272984542 shutdown_connections 2020-11-20T12:06:15.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/272984542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b11d0 0x7fbb84066040 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/272984542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb840b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/272984542 >> 172.21.15.187:0/272984542 conn(0x7fbb840bbeb0 msgr2=0x7fbb840b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:15.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/272984542 shutdown_connections 2020-11-20T12:06:15.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.353+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/272984542 wait complete. 2020-11-20T12:06:15.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.355+0000 7fbb8b4a4700 1 Processor -- start 2020-11-20T12:06:15.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.355+0000 7fbb8b4a4700 1 -- start start 2020-11-20T12:06:15.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8b4a4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb840b11d0 0x7fbb84066120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8b4a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb84136030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8b4a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b1c20 0x7fbb840b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8b4a4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb84143df0 con 0x7fbb840b11d0 2020-11-20T12:06:15.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8b4a4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb84144ea0 con 0x7fbb840b1c20 2020-11-20T12:06:15.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8b4a4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb84145140 con 0x7fbb840b1840 2020-11-20T12:06:15.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8aca3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b1c20 0x7fbb840b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8aca3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b1c20 0x7fbb840b25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42176/0 (socket says 172.21.15.187:42176) 2020-11-20T12:06:15.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8aca3700 1 -- 172.21.15.187:0/501864941 learned_addr learned my addr 172.21.15.187:0/501864941 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:15.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb89ca1700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb84136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.356+0000 7fbb8a4a2700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb840b11d0 0x7fbb84066120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8aca3700 1 -- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 msgr2=0x7fbb84136030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8aca3700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb84136030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8aca3700 1 -- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb840b11d0 msgr2=0x7fbb84066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8aca3700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb840b11d0 0x7fbb84066120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8aca3700 1 -- 172.21.15.187:0/501864941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb84066610 con 0x7fbb840b1c20 2020-11-20T12:06:15.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb89ca1700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb84136030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:15.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8a4a2700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb840b11d0 0x7fbb84066120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:15.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8aca3700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b1c20 0x7fbb840b25a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fbb7c0092d0 tx=0x7fbb7c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:15.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb727fc700 1 -- 172.21.15.187:0/501864941 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c020070 con 0x7fbb840b1c20 2020-11-20T12:06:15.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb727fc700 1 -- 172.21.15.187:0/501864941 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7c009e90 con 0x7fbb840b1c20 2020-11-20T12:06:15.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb84143a00 con 0x7fbb840b1c20 2020-11-20T12:06:15.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb727fc700 1 -- 172.21.15.187:0/501864941 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c00bd10 con 0x7fbb840b1c20 2020-11-20T12:06:15.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.357+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb84144b50 con 0x7fbb840b1c20 2020-11-20T12:06:15.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.358+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb84145300 con 0x7fbb840b1c20 2020-11-20T12:06:15.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.359+0000 7fbb727fc700 1 -- 172.21.15.187:0/501864941 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbb7c017070 con 0x7fbb840b1c20 2020-11-20T12:06:15.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.360+0000 7fbb727fc700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbb5c056d20 0x7fbb5c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.360+0000 7fbb727fc700 1 -- 172.21.15.187:0/501864941 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbb7c071de0 con 0x7fbb840b1c20 2020-11-20T12:06:15.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.360+0000 7fbb8a4a2700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbb5c056d20 0x7fbb5c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.361+0000 7fbb8a4a2700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbb5c056d20 0x7fbb5c0591e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fbb8000b870 tx=0x7fbb80008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:15.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.364+0000 7fbb727fc700 1 -- 172.21.15.187:0/501864941 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbb7c047230 con 0x7fbb840b1c20 2020-11-20T12:06:15.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.614+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fbb84061050 con 0x7fbb840b1c20 2020-11-20T12:06:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.624+0000 7fbb727fc700 1 -- 172.21.15.187:0/501864941 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbb7c015070 con 0x7fbb840b1c20 2020-11-20T12:06:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbb5c056d20 msgr2=0x7fbb5c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbb5c056d20 0x7fbb5c0591e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fbb8000b870 tx=0x7fbb80008040).stop 2020-11-20T12:06:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b1c20 msgr2=0x7fbb840b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b1c20 0x7fbb840b25a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fbb7c0092d0 tx=0x7fbb7c016000).stop 2020-11-20T12:06:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 shutdown_connections 2020-11-20T12:06:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbb5c056d20 0x7fbb5c0591e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb840b11d0 0x7fbb84066120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840b1c20 0x7fbb840b25a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 --2- 172.21.15.187:0/501864941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb840b1840 0x7fbb84136030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.626+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 >> 172.21.15.187:0/501864941 conn(0x7fbb840bbeb0 msgr2=0x7fbb8414c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.627+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 shutdown_connections 2020-11-20T12:06:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.627+0000 7fbb8b4a4700 1 -- 172.21.15.187:0/501864941 wait complete. 2020-11-20T12:06:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:06:15.655 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:15.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.882+0000 7fc475ad7700 1 Processor -- start 2020-11-20T12:06:15.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.882+0000 7fc475ad7700 1 -- start start 2020-11-20T12:06:15.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.883+0000 7fc475ad7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc4700bbaa0 0x7fc4700a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.883+0000 7fc475ad7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc470149c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.883+0000 7fc475ad7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc470149930 con 0x7fc47014a110 2020-11-20T12:06:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.883+0000 7fc475ad7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4700a3560 con 0x7fc4700bbaa0 2020-11-20T12:06:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.883+0000 7fc475ad7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700a3250 con 0x7fc4700bbe80 2020-11-20T12:06:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.884+0000 7fc474ad5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc4700bbaa0 0x7fc4700a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.884+0000 7fc474ad5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc4700bbaa0 0x7fc4700a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42182/0 (socket says 172.21.15.187:42182) 2020-11-20T12:06:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.884+0000 7fc474ad5700 1 -- 172.21.15.187:0/3789506036 learned_addr learned my addr 172.21.15.187:0/3789506036 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.884+0000 7fc46ffff700 1 --2- 172.21.15.187:0/3789506036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc470149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.884+0000 7fc474ad5700 1 -- 172.21.15.187:0/3789506036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 msgr2=0x7fc470149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.884+0000 7fc474ad5700 1 --2- 172.21.15.187:0/3789506036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc470149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.884+0000 7fc474ad5700 1 -- 172.21.15.187:0/3789506036 >> v1:172.21.15.5:6789/0 conn(0x7fc47014a110 legacy=0x7fc4701426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:15.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.884+0000 7fc474ad5700 1 -- 172.21.15.187:0/3789506036 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4700a4790 con 0x7fc4700bbaa0 2020-11-20T12:06:15.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.885+0000 7fc474ad5700 1 --2- 172.21.15.187:0/3789506036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4700bbaa0 0x7fc4700a4bf0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fc464006fd0 tx=0x7fc464006eb0).ready entity=mon.1 client_cookie=a15b2e1dccf71fba server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:15.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.885+0000 7fc46f7fe700 1 -- 172.21.15.187:0/3789506036 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464009890 con 0x7fc4700bbaa0 2020-11-20T12:06:15.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.885+0000 7fc46f7fe700 1 -- 172.21.15.187:0/3789506036 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4640099f0 con 0x7fc4700bbaa0 2020-11-20T12:06:15.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.885+0000 7fc46f7fe700 1 -- 172.21.15.187:0/3789506036 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464009d70 con 0x7fc4700bbaa0 2020-11-20T12:06:15.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.886+0000 7fc475ad7700 1 -- 172.21.15.187:0/3789506036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4700bbaa0 msgr2=0x7fc4700a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.886+0000 7fc475ad7700 1 --2- 172.21.15.187:0/3789506036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4700bbaa0 0x7fc4700a4bf0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fc464006fd0 tx=0x7fc464006eb0).stop 2020-11-20T12:06:15.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.886+0000 7fc475ad7700 1 -- 172.21.15.187:0/3789506036 shutdown_connections 2020-11-20T12:06:15.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.886+0000 7fc475ad7700 1 --2- 172.21.15.187:0/3789506036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4700bbaa0 0x7fc4700a4bf0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.886+0000 7fc475ad7700 1 --2- 172.21.15.187:0/3789506036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc470149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.886+0000 7fc475ad7700 1 -- 172.21.15.187:0/3789506036 >> 172.21.15.187:0/3789506036 conn(0x7fc470061bd0 msgr2=0x7fc4700678a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:15.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.886+0000 7fc475ad7700 1 -- 172.21.15.187:0/3789506036 shutdown_connections 2020-11-20T12:06:15.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.886+0000 7fc475ad7700 1 -- 172.21.15.187:0/3789506036 wait complete. 2020-11-20T12:06:15.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.887+0000 7fc475ad7700 1 Processor -- start 2020-11-20T12:06:15.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.888+0000 7fc475ad7700 1 -- start start 2020-11-20T12:06:15.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.888+0000 7fc475ad7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc47013f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.888+0000 7fc475ad7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47014a110 0x7fc47009f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc475ad7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc47009f560 0x7fc4700b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc475ad7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc470145a90 con 0x7fc47009f560 2020-11-20T12:06:15.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc475ad7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47015c850 con 0x7fc47014a110 2020-11-20T12:06:15.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc475ad7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc470145ef0 con 0x7fc4700bbe80 2020-11-20T12:06:15.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc46ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47014a110 0x7fc47009f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc46ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47014a110 0x7fc47009f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42188/0 (socket says 172.21.15.187:42188) 2020-11-20T12:06:15.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc46ffff700 1 -- 172.21.15.187:0/2152993951 learned_addr learned my addr 172.21.15.187:0/2152993951 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:15.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc474ad5700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc47013f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc46ffff700 1 -- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 msgr2=0x7fc47013f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc4752d6700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc47009f560 0x7fc4700b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc46ffff700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc47013f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.889+0000 7fc46ffff700 1 -- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc47009f560 msgr2=0x7fc4700b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:15.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc46ffff700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc47009f560 0x7fc4700b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:15.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc46ffff700 1 -- 172.21.15.187:0/2152993951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47015c5b0 con 0x7fc47014a110 2020-11-20T12:06:15.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc474ad5700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc47013f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:15.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc4752d6700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc47009f560 0x7fc4700b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:15.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc46ffff700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47014a110 0x7fc47009f070 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fc45c00db60 tx=0x7fc45c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:15.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc46cff9700 1 -- 172.21.15.187:0/2152993951 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c009790 con 0x7fc47014a110 2020-11-20T12:06:15.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc46cff9700 1 -- 172.21.15.187:0/2152993951 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45c0098f0 con 0x7fc47014a110 2020-11-20T12:06:15.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc46cff9700 1 -- 172.21.15.187:0/2152993951 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c009c90 con 0x7fc47014a110 2020-11-20T12:06:15.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.890+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47015c2e0 con 0x7fc47014a110 2020-11-20T12:06:15.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.891+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47013f680 con 0x7fc47014a110 2020-11-20T12:06:15.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.892+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc470145c00 con 0x7fc47014a110 2020-11-20T12:06:15.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.892+0000 7fc46cff9700 1 -- 172.21.15.187:0/2152993951 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc45c013480 con 0x7fc47014a110 2020-11-20T12:06:15.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.893+0000 7fc46cff9700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc448056d20 0x7fc4480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:15.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.893+0000 7fc46cff9700 1 -- 172.21.15.187:0/2152993951 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc45c0705d0 con 0x7fc47014a110 2020-11-20T12:06:15.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.893+0000 7fc474ad5700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc448056d20 0x7fc4480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:15.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.894+0000 7fc474ad5700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc448056d20 0x7fc4480591e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc470060c60 tx=0x7fc464015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:15.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:15.897+0000 7fc46cff9700 1 -- 172.21.15.187:0/2152993951 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc45c0459c0 con 0x7fc47014a110 2020-11-20T12:06:16.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.142+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc470142e00 con 0x7fc47014a110 2020-11-20T12:06:16.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.143+0000 7fc46cff9700 1 -- 172.21.15.187:0/2152993951 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7fc45c048fe0 con 0x7fc47014a110 2020-11-20T12:06:16.148 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:16.148 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:16.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.145+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc448056d20 msgr2=0x7fc4480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:16.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.145+0000 7fc475ad7700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc448056d20 0x7fc4480591e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc470060c60 tx=0x7fc464015000).stop 2020-11-20T12:06:16.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.145+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47014a110 msgr2=0x7fc47009f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:16.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.145+0000 7fc475ad7700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47014a110 0x7fc47009f070 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fc45c00db60 tx=0x7fc45c008040).stop 2020-11-20T12:06:16.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.146+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 shutdown_connections 2020-11-20T12:06:16.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.146+0000 7fc475ad7700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc448056d20 0x7fc4480591e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:16.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.146+0000 7fc475ad7700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc47009f560 0x7fc4700b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.146+0000 7fc475ad7700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4700bbe80 0x7fc47013f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.146+0000 7fc475ad7700 1 --2- 172.21.15.187:0/2152993951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47014a110 0x7fc47009f070 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.146+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 >> 172.21.15.187:0/2152993951 conn(0x7fc470061bd0 msgr2=0x7fc470141c40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.146+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 shutdown_connections 2020-11-20T12:06:16.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:16.146+0000 7fc475ad7700 1 -- 172.21.15.187:0/2152993951 wait complete. 2020-11-20T12:06:16.153 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:16.372 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:06:11.574607+0000', 'modified': '2020-11-20T12:06:14.058447+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4966}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4966': {'gid': 4966, 'name': 'a', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/2376356363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2376356363}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2376356363}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:06:16.372 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:06:16.373 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:06:16.649 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":4966,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":5.0456456110000003,"mdsmap_epoch":13,"osdmap_epoch":32,"osdmap_epoch_barrier":32,"uptime":14.306508057} 2020-11-20T12:06:16.650 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 4966, "mdsmap_epoch": 13, "osdmap_epoch": 32, "osdmap_epoch_barrier": 32, "rank_uptime": 5.045645611, "state": "up:active", "uptime": 14.306508057, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:06:16.651 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:06:16.652 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:06:16.652 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:06:16.653 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:06:16.653 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:06:16.654 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:06:16.655 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:06:16.655 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:06:16.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:16.657 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:06:16.709 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:06:16.709 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:06:16.709 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:06:16.710 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:06:16.710 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:06:16.710 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:06:16.710 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:06:16.710 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:06:16.711 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:06:16.711 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:06:16.711 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:06:16.711 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:06:16.711 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:06:16.712 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:06:16.712 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:06:16.712 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42662sec preferred_lft 42662sec 2020-11-20T12:06:16.712 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:06:16.712 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:06:16.713 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:06:16.713 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:06:16.713 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:06:16.713 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:06:16.713 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:06:16.714 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:06:16.714 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:06:16.714 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:06:16.714 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:06:16.714 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:06:16.715 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:06:16.715 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:06:16.715 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:06:16.715 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:06:16.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:16.717 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:06:16.806 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:06:16.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:16.811 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:06:16.864 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:06:16.864 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:06:16.865 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:06:16.865 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:06:16.865 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:06:16.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:16.867 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:06:16.867 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:06:16.868 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:06:16.868 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:06:16.868 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:06:16.869 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:06:17.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:17.125 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:06:17.178 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:06:17.179 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:06:17.180 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:06:17.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:17.181 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:06:17.284 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:06:17.457 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:06:17.337+0000 7f6adc244700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:06:17.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.697+0000 7f7952c2f700 1 Processor -- start 2020-11-20T12:06:17.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.697+0000 7f7952c2f700 1 -- start start 2020-11-20T12:06:17.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.698+0000 7f7952c2f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f794c1484d0 0x7f794c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:17.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.698+0000 7f7952c2f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f794c066f70 0x7f794c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:17.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.699+0000 7f7952c2f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794c0b1300 con 0x7f794c0622c0 2020-11-20T12:06:17.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.699+0000 7f7952c2f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794c067730 con 0x7f794c1484d0 2020-11-20T12:06:17.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.699+0000 7f7952c2f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794c0b15a0 con 0x7f794c066f70 2020-11-20T12:06:17.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.699+0000 7f795242e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f794c066f70 0x7f794c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:17.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.699+0000 7f795242e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f794c066f70 0x7f794c0b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42198/0 (socket says 172.21.15.187:42198) 2020-11-20T12:06:17.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.699+0000 7f795242e700 1 -- 172.21.15.187:0/1134750934 learned_addr learned my addr 172.21.15.187:0/1134750934 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:17.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.699+0000 7f7951c2d700 1 --2- 172.21.15.187:0/1134750934 >> v2:172.21.15.5:3300/0 conn(0x7f794c1484d0 0x7f794c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:17.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.700+0000 7f795242e700 1 -- 172.21.15.187:0/1134750934 >> v2:172.21.15.5:3300/0 conn(0x7f794c1484d0 msgr2=0x7f794c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:17.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.700+0000 7f795242e700 1 --2- 172.21.15.187:0/1134750934 >> v2:172.21.15.5:3300/0 conn(0x7f794c1484d0 0x7f794c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.700+0000 7f795242e700 1 -- 172.21.15.187:0/1134750934 >> v1:172.21.15.5:6789/0 conn(0x7f794c0622c0 legacy=0x7f794c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:17.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.700+0000 7f795242e700 1 -- 172.21.15.187:0/1134750934 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c0b1840 con 0x7f794c066f70 2020-11-20T12:06:17.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.700+0000 7f795242e700 1 --2- 172.21.15.187:0/1134750934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c066f70 0x7f794c0b5d50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f794c045000 tx=0x7f7944009290).ready entity=mon.1 client_cookie=7b62fe32d2b35a8d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:17.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.700+0000 7f7950c2b700 1 -- 172.21.15.187:0/1134750934 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794400bb80 con 0x7f794c066f70 2020-11-20T12:06:17.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.700+0000 7f7950c2b700 1 -- 172.21.15.187:0/1134750934 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794400bce0 con 0x7f794c066f70 2020-11-20T12:06:17.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.700+0000 7f7950c2b700 1 -- 172.21.15.187:0/1134750934 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944011610 con 0x7f794c066f70 2020-11-20T12:06:17.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.701+0000 7f7952c2f700 1 -- 172.21.15.187:0/1134750934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c066f70 msgr2=0x7f794c0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:17.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.701+0000 7f7952c2f700 1 --2- 172.21.15.187:0/1134750934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c066f70 0x7f794c0b5d50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f794c045000 tx=0x7f7944009290).stop 2020-11-20T12:06:17.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.701+0000 7f7952c2f700 1 -- 172.21.15.187:0/1134750934 shutdown_connections 2020-11-20T12:06:17.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.701+0000 7f7952c2f700 1 --2- 172.21.15.187:0/1134750934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c066f70 0x7f794c0b5d50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.701+0000 7f7952c2f700 1 --2- 172.21.15.187:0/1134750934 >> v2:172.21.15.5:3300/0 conn(0x7f794c1484d0 0x7f794c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.701+0000 7f7952c2f700 1 -- 172.21.15.187:0/1134750934 >> 172.21.15.187:0/1134750934 conn(0x7f794c0bab30 msgr2=0x7f794c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:17.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.701+0000 7f7952c2f700 1 -- 172.21.15.187:0/1134750934 shutdown_connections 2020-11-20T12:06:17.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.701+0000 7f7952c2f700 1 -- 172.21.15.187:0/1134750934 wait complete. 2020-11-20T12:06:17.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.703+0000 7f7952c2f700 1 Processor -- start 2020-11-20T12:06:17.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.703+0000 7f7952c2f700 1 -- start start 2020-11-20T12:06:17.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.704+0000 7f7952c2f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f794c0622c0 0x7f794c09f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:17.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.704+0000 7f7952c2f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f794c066f70 0x7f794c09ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:17.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.704+0000 7f7952c2f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c1484d0 0x7f794c141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:17.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.704+0000 7f7952c2f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c143680 con 0x7f794c0622c0 2020-11-20T12:06:17.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.704+0000 7f7952c2f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c143920 con 0x7f794c1484d0 2020-11-20T12:06:17.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.704+0000 7f7952c2f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c143bc0 con 0x7f794c066f70 2020-11-20T12:06:17.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795242e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c1484d0 0x7f794c141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:17.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795242e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c1484d0 0x7f794c141c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42204/0 (socket says 172.21.15.187:42204) 2020-11-20T12:06:17.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795242e700 1 -- 172.21.15.187:0/2560926425 learned_addr learned my addr 172.21.15.187:0/2560926425 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:17.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795242e700 1 -- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f794c066f70 msgr2=0x7f794c09ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:17.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f7951c2d700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f794c0622c0 0x7f794c09f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:17.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795142c700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f794c066f70 0x7f794c09ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:17.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795242e700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f794c066f70 0x7f794c09ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795242e700 1 -- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f794c0622c0 msgr2=0x7f794c09f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:17.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795242e700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f794c0622c0 0x7f794c09f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.705+0000 7f795242e700 1 -- 172.21.15.187:0/2560926425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c144060 con 0x7f794c1484d0 2020-11-20T12:06:17.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.706+0000 7f7951c2d700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f794c0622c0 0x7f794c09f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:17.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.706+0000 7f795242e700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c1484d0 0x7f794c141c90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f79440092d0 tx=0x7f7944016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:17.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.706+0000 7f7939ffb700 1 -- 172.21.15.187:0/2560926425 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794400bb80 con 0x7f794c1484d0 2020-11-20T12:06:17.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.706+0000 7f7939ffb700 1 -- 172.21.15.187:0/2560926425 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794400be50 con 0x7f794c1484d0 2020-11-20T12:06:17.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.706+0000 7f7939ffb700 1 -- 172.21.15.187:0/2560926425 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944010e70 con 0x7f794c1484d0 2020-11-20T12:06:17.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.706+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794c09cf20 con 0x7f794c1484d0 2020-11-20T12:06:17.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.706+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c09edb0 con 0x7f794c1484d0 2020-11-20T12:06:17.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.707+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794c1434c0 con 0x7f794c1484d0 2020-11-20T12:06:17.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.708+0000 7f7939ffb700 1 -- 172.21.15.187:0/2560926425 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7944017070 con 0x7f794c1484d0 2020-11-20T12:06:17.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.709+0000 7f7939ffb700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7924056d20 0x7f79240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:17.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.709+0000 7f7939ffb700 1 -- 172.21.15.187:0/2560926425 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7944072260 con 0x7f794c1484d0 2020-11-20T12:06:17.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.709+0000 7f7951c2d700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7924056d20 0x7f79240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:17.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.710+0000 7f7951c2d700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7924056d20 0x7f79240591e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f7948009670 tx=0x7f7948008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:17.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.712+0000 7f7939ffb700 1 -- 172.21.15.187:0/2560926425 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f79440475a0 con 0x7f794c1484d0 2020-11-20T12:06:17.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.896+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f794c143d80 con 0x7f794c1484d0 2020-11-20T12:06:17.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.898+0000 7f7939ffb700 1 -- 172.21.15.187:0/2560926425 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f7944015020 con 0x7f794c1484d0 2020-11-20T12:06:17.903 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:17.903 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":32,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:06:12.578719+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:06:09.312569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2020-11-20T12:06:10.581583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:06:17.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.900+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7924056d20 msgr2=0x7f79240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.900+0000 7f7952c2f700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7924056d20 0x7f79240591e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f7948009670 tx=0x7f7948008040).stop 2020-11-20T12:06:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.900+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c1484d0 msgr2=0x7f794c141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.900+0000 7f7952c2f700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c1484d0 0x7f794c141c90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f79440092d0 tx=0x7f7944016000).stop 2020-11-20T12:06:17.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.901+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 shutdown_connections 2020-11-20T12:06:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.901+0000 7f7952c2f700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7924056d20 0x7f79240591e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.901+0000 7f7952c2f700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f794c0622c0 0x7f794c09f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.901+0000 7f7952c2f700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f794c1484d0 0x7f794c141c90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.901+0000 7f7952c2f700 1 --2- 172.21.15.187:0/2560926425 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f794c066f70 0x7f794c09ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.901+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 >> 172.21.15.187:0/2560926425 conn(0x7f794c0bab30 msgr2=0x7f794c0b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:17.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.901+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 shutdown_connections 2020-11-20T12:06:17.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:17.901+0000 7f7952c2f700 1 -- 172.21.15.187:0/2560926425 wait complete. 2020-11-20T12:06:17.928 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:06:09.312569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:06:10.581583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:06:17.929 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:06:18.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.162+0000 7fda2bfff700 1 Processor -- start 2020-11-20T12:06:18.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.162+0000 7fda2bfff700 1 -- start start 2020-11-20T12:06:18.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.163+0000 7fda2bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fda2c066f70 0x7fda2c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.163+0000 7fda2bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c0b1300 con 0x7fda2c1484d0 2020-11-20T12:06:18.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.163+0000 7fda2bfff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c0b15a0 con 0x7fda2c0b6650 2020-11-20T12:06:18.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.163+0000 7fda2bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2c067730 con 0x7fda2c066f70 2020-11-20T12:06:18.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.164+0000 7fda2b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fda2c066f70 0x7fda2c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.164+0000 7fda2b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fda2c066f70 0x7fda2c0b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42210/0 (socket says 172.21.15.187:42210) 2020-11-20T12:06:18.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.164+0000 7fda2b7fe700 1 -- 172.21.15.187:0/3036532756 learned_addr learned my addr 172.21.15.187:0/3036532756 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:18.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.164+0000 7fda2b7fe700 1 -- 172.21.15.187:0/3036532756 >> v1:172.21.15.187:6789/0 conn(0x7fda2c0b6650 legacy=0x7fda2c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:18.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.164+0000 7fda2b7fe700 1 -- 172.21.15.187:0/3036532756 >> v1:172.21.15.5:6789/0 conn(0x7fda2c1484d0 legacy=0x7fda2c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:18.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.164+0000 7fda2b7fe700 1 -- 172.21.15.187:0/3036532756 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0b1840 con 0x7fda2c066f70 2020-11-20T12:06:18.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.165+0000 7fda2b7fe700 1 --2- 172.21.15.187:0/3036532756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c0b01a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fda24009d50 tx=0x7fda24009290).ready entity=mon.1 client_cookie=b28f7dad0acc9f5b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:18.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.165+0000 7fda2a7fc700 1 -- 172.21.15.187:0/3036532756 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2400bc00 con 0x7fda2c066f70 2020-11-20T12:06:18.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.165+0000 7fda2a7fc700 1 -- 172.21.15.187:0/3036532756 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2400bd60 con 0x7fda2c066f70 2020-11-20T12:06:18.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.165+0000 7fda2a7fc700 1 -- 172.21.15.187:0/3036532756 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24011610 con 0x7fda2c066f70 2020-11-20T12:06:18.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.166+0000 7fda2bfff700 1 -- 172.21.15.187:0/3036532756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 msgr2=0x7fda2c0b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.166+0000 7fda2bfff700 1 --2- 172.21.15.187:0/3036532756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c0b01a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fda24009d50 tx=0x7fda24009290).stop 2020-11-20T12:06:18.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.166+0000 7fda2bfff700 1 -- 172.21.15.187:0/3036532756 shutdown_connections 2020-11-20T12:06:18.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.166+0000 7fda2bfff700 1 --2- 172.21.15.187:0/3036532756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c0b01a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.166+0000 7fda2bfff700 1 -- 172.21.15.187:0/3036532756 >> 172.21.15.187:0/3036532756 conn(0x7fda2c0bab30 msgr2=0x7fda2c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:18.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.166+0000 7fda2bfff700 1 -- 172.21.15.187:0/3036532756 shutdown_connections 2020-11-20T12:06:18.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.166+0000 7fda2bfff700 1 -- 172.21.15.187:0/3036532756 wait complete. 2020-11-20T12:06:18.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.168+0000 7fda2bfff700 1 Processor -- start 2020-11-20T12:06:18.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.168+0000 7fda2bfff700 1 -- start start 2020-11-20T12:06:18.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.169+0000 7fda2bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c09ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.169+0000 7fda2bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda2c0b6650 0x7fda2c130e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.169+0000 7fda2bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda2c1484d0 0x7fda2c0b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.169+0000 7fda2bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c09f3e0 con 0x7fda2c0b6650 2020-11-20T12:06:18.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.169+0000 7fda2bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c09f610 con 0x7fda2c066f70 2020-11-20T12:06:18.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.169+0000 7fda2bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c09b8f0 con 0x7fda2c1484d0 2020-11-20T12:06:18.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c09ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c09ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42216/0 (socket says 172.21.15.187:42216) 2020-11-20T12:06:18.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2affd700 1 -- 172.21.15.187:0/2547637704 learned_addr learned my addr 172.21.15.187:0/2547637704 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:18.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda227fc700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda2c0b6650 0x7fda2c130e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2b7fe700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda2c1484d0 0x7fda2c0b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2affd700 1 -- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda2c1484d0 msgr2=0x7fda2c0b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2affd700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda2c1484d0 0x7fda2c0b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2affd700 1 -- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda2c0b6650 msgr2=0x7fda2c130e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2affd700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda2c0b6650 0x7fda2c130e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2affd700 1 -- 172.21.15.187:0/2547637704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0b40e0 con 0x7fda2c066f70 2020-11-20T12:06:18.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda227fc700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda2c0b6650 0x7fda2c130e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:18.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.170+0000 7fda2b7fe700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda2c1484d0 0x7fda2c0b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:18.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.171+0000 7fda2affd700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c09ed60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fda1c0096e0 tx=0x7fda1c006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:18.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.171+0000 7fda237fe700 1 -- 172.21.15.187:0/2547637704 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00d5c0 con 0x7fda2c066f70 2020-11-20T12:06:18.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.171+0000 7fda237fe700 1 -- 172.21.15.187:0/2547637704 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c000e90 con 0x7fda2c066f70 2020-11-20T12:06:18.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.171+0000 7fda237fe700 1 -- 172.21.15.187:0/2547637704 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00d990 con 0x7fda2c066f70 2020-11-20T12:06:18.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.171+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c09bd90 con 0x7fda2c066f70 2020-11-20T12:06:18.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.171+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c09c230 con 0x7fda2c066f70 2020-11-20T12:06:18.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.172+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c09bab0 con 0x7fda2c066f70 2020-11-20T12:06:18.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.173+0000 7fda237fe700 1 -- 172.21.15.187:0/2547637704 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fda1c00dd40 con 0x7fda2c066f70 2020-11-20T12:06:18.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.173+0000 7fda237fe700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda08056d20 0x7fda080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.174+0000 7fda237fe700 1 -- 172.21.15.187:0/2547637704 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fda1c06f950 con 0x7fda2c066f70 2020-11-20T12:06:18.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.174+0000 7fda227fc700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda08056d20 0x7fda080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.174+0000 7fda227fc700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda08056d20 0x7fda080591e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fda2c09cbc0 tx=0x7fda14009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:18.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.177+0000 7fda237fe700 1 -- 172.21.15.187:0/2547637704 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fda1c044c90 con 0x7fda2c066f70 2020-11-20T12:06:18.336 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:18.347 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:18.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.363+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fda2c061a20 con 0x7fda2c066f70 2020-11-20T12:06:18.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.364+0000 7fda237fe700 1 -- 172.21.15.187:0/2547637704 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v32) v1 ==== 110+0+10 (secure 0 0 0) 0x7fda1c0482b0 con 0x7fda2c066f70 2020-11-20T12:06:18.369 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:06:18.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda08056d20 msgr2=0x7fda080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda08056d20 0x7fda080591e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fda2c09cbc0 tx=0x7fda14009040).stop 2020-11-20T12:06:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 msgr2=0x7fda2c09ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c09ed60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fda1c0096e0 tx=0x7fda1c006d70).stop 2020-11-20T12:06:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 shutdown_connections 2020-11-20T12:06:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda08056d20 0x7fda080591e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda2c1484d0 0x7fda2c0b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda2c066f70 0x7fda2c09ed60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 --2- 172.21.15.187:0/2547637704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda2c0b6650 0x7fda2c130e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 >> 172.21.15.187:0/2547637704 conn(0x7fda2c0bab30 msgr2=0x7fda2c0ba320 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:18.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.366+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 shutdown_connections 2020-11-20T12:06:18.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.367+0000 7fda2bfff700 1 -- 172.21.15.187:0/2547637704 wait complete. 2020-11-20T12:06:18.392 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:06:18.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.659+0000 7f6fa9240700 1 Processor -- start 2020-11-20T12:06:18.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.660+0000 7f6fa9240700 1 -- start start 2020-11-20T12:06:18.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.661+0000 7f6fa9240700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6fa41484d0 0x7f6fa4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.661+0000 7f6fa9240700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa40b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.661+0000 7f6fa9240700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa40b1300 con 0x7f6fa40622c0 2020-11-20T12:06:18.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.661+0000 7f6fa9240700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa4067730 con 0x7f6fa41484d0 2020-11-20T12:06:18.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.661+0000 7f6fa9240700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40b15a0 con 0x7f6fa4066f70 2020-11-20T12:06:18.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.661+0000 7f6fa3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6fa41484d0 0x7f6fa4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.661+0000 7f6fa3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6fa41484d0 0x7f6fa4061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42222/0 (socket says 172.21.15.187:42222) 2020-11-20T12:06:18.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.661+0000 7f6fa3fff700 1 -- 172.21.15.187:0/3717688647 learned_addr learned my addr 172.21.15.187:0/3717688647 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:18.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.662+0000 7f6fa8a3f700 1 --2- 172.21.15.187:0/3717688647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa40b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.662+0000 7f6fa3fff700 1 -- 172.21.15.187:0/3717688647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 msgr2=0x7f6fa40b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:18.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.662+0000 7f6fa3fff700 1 --2- 172.21.15.187:0/3717688647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa40b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.662+0000 7f6fa3fff700 1 -- 172.21.15.187:0/3717688647 >> v1:172.21.15.5:6789/0 conn(0x7f6fa40622c0 legacy=0x7f6fa40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:18.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.662+0000 7f6fa3fff700 1 -- 172.21.15.187:0/3717688647 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40b1840 con 0x7f6fa41484d0 2020-11-20T12:06:18.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.662+0000 7f6fa3fff700 1 --2- 172.21.15.187:0/3717688647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa41484d0 0x7f6fa4061dd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6f94006fd0 tx=0x7f6f94006eb0).ready entity=mon.1 client_cookie=a65d32e5c7fedead server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:18.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.663+0000 7f6fa37fe700 1 -- 172.21.15.187:0/3717688647 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94009890 con 0x7f6fa41484d0 2020-11-20T12:06:18.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.663+0000 7f6fa37fe700 1 -- 172.21.15.187:0/3717688647 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f940099f0 con 0x7f6fa41484d0 2020-11-20T12:06:18.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.663+0000 7f6fa37fe700 1 -- 172.21.15.187:0/3717688647 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94009d70 con 0x7f6fa41484d0 2020-11-20T12:06:18.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.663+0000 7f6fa9240700 1 -- 172.21.15.187:0/3717688647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa41484d0 msgr2=0x7f6fa4061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.664+0000 7f6fa9240700 1 --2- 172.21.15.187:0/3717688647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa41484d0 0x7f6fa4061dd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6f94006fd0 tx=0x7f6f94006eb0).stop 2020-11-20T12:06:18.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.664+0000 7f6fa9240700 1 -- 172.21.15.187:0/3717688647 shutdown_connections 2020-11-20T12:06:18.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.664+0000 7f6fa9240700 1 --2- 172.21.15.187:0/3717688647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa40b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.664+0000 7f6fa9240700 1 --2- 172.21.15.187:0/3717688647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa41484d0 0x7f6fa4061dd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.664+0000 7f6fa9240700 1 -- 172.21.15.187:0/3717688647 >> 172.21.15.187:0/3717688647 conn(0x7f6fa40bab30 msgr2=0x7f6fa40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:18.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.664+0000 7f6fa9240700 1 -- 172.21.15.187:0/3717688647 shutdown_connections 2020-11-20T12:06:18.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.664+0000 7f6fa9240700 1 -- 172.21.15.187:0/3717688647 wait complete. 2020-11-20T12:06:18.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.665+0000 7f6fa9240700 1 Processor -- start 2020-11-20T12:06:18.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.665+0000 7f6fa9240700 1 -- start start 2020-11-20T12:06:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.666+0000 7f6fa9240700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa40622c0 0x7f6fa40b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.666+0000 7f6fa9240700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa4136da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.666+0000 7f6fa9240700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fa41484d0 0x7f6fa4137290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.666+0000 7f6fa9240700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa4137780 con 0x7f6fa41484d0 2020-11-20T12:06:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.666+0000 7f6fa9240700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa409b910 con 0x7f6fa40622c0 2020-11-20T12:06:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.666+0000 7f6fa9240700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa409e440 con 0x7f6fa4066f70 2020-11-20T12:06:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa40622c0 0x7f6fa40b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa40622c0 0x7f6fa40b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42228/0 (socket says 172.21.15.187:42228) 2020-11-20T12:06:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa3fff700 1 -- 172.21.15.187:0/2210333216 learned_addr learned my addr 172.21.15.187:0/2210333216 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6f9b7fe700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa4136da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa8a3f700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fa41484d0 0x7f6fa4137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa3fff700 1 -- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 msgr2=0x7f6fa4136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa3fff700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa4136da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa3fff700 1 -- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fa41484d0 msgr2=0x7f6fa4137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa3fff700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fa41484d0 0x7f6fa4137290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa3fff700 1 -- 172.21.15.187:0/2210333216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa409e6e0 con 0x7f6fa40622c0 2020-11-20T12:06:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6f9b7fe700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa4136da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.667+0000 7f6fa8a3f700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fa41484d0 0x7f6fa4137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:18.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.668+0000 7f6fa3fff700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa40622c0 0x7f6fa40b3c10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f6f94009e90 tx=0x7f6f94015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:18.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.668+0000 7f6fa0ff9700 1 -- 172.21.15.187:0/2210333216 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9401e460 con 0x7f6fa40622c0 2020-11-20T12:06:18.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.668+0000 7f6fa0ff9700 1 -- 172.21.15.187:0/2210333216 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9401e5c0 con 0x7f6fa40622c0 2020-11-20T12:06:18.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.668+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa40b4160 con 0x7f6fa40622c0 2020-11-20T12:06:18.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.668+0000 7f6fa0ff9700 1 -- 172.21.15.187:0/2210333216 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9401e960 con 0x7f6fa40622c0 2020-11-20T12:06:18.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.668+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa409c960 con 0x7f6fa40622c0 2020-11-20T12:06:18.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.669+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa409e870 con 0x7f6fa40622c0 2020-11-20T12:06:18.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.670+0000 7f6fa0ff9700 1 -- 172.21.15.187:0/2210333216 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6f94025030 con 0x7f6fa40622c0 2020-11-20T12:06:18.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.670+0000 7f6fa0ff9700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f80056d50 0x7f6f80059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:18.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.671+0000 7f6fa0ff9700 1 -- 172.21.15.187:0/2210333216 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6f94071cc0 con 0x7f6fa40622c0 2020-11-20T12:06:18.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.671+0000 7f6f9b7fe700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f80056d50 0x7f6f80059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:18.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.671+0000 7f6f9b7fe700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f80056d50 0x7f6f80059210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f6fa4065de0 tx=0x7f6f8c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:18.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.674+0000 7f6fa0ff9700 1 -- 172.21.15.187:0/2210333216 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6f94047110 con 0x7f6fa40622c0 2020-11-20T12:06:18.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.863+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6fa4061a20 con 0x7f6fa40622c0 2020-11-20T12:06:18.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.864+0000 7f6fa0ff9700 1 -- 172.21.15.187:0/2210333216 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v32) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6f9404a730 con 0x7f6fa40622c0 2020-11-20T12:06:18.868 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:06:18.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f80056d50 msgr2=0x7f6f80059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f80056d50 0x7f6f80059210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f6fa4065de0 tx=0x7f6f8c009040).stop 2020-11-20T12:06:18.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa40622c0 msgr2=0x7f6fa40b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:18.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa40622c0 0x7f6fa40b3c10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f6f94009e90 tx=0x7f6f94015000).stop 2020-11-20T12:06:18.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 shutdown_connections 2020-11-20T12:06:18.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f80056d50 0x7f6f80059210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fa41484d0 0x7f6fa4137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa40622c0 0x7f6fa40b3c10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 --2- 172.21.15.187:0/2210333216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fa4066f70 0x7f6fa4136da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:18.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 >> 172.21.15.187:0/2210333216 conn(0x7f6fa40bab30 msgr2=0x7f6fa40b6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:18.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.866+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 shutdown_connections 2020-11-20T12:06:18.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:18.867+0000 7f6fa9240700 1 -- 172.21.15.187:0/2210333216 wait complete. 2020-11-20T12:06:18.892 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:06:19.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.134+0000 7efc566b4700 1 Processor -- start 2020-11-20T12:06:19.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.134+0000 7efc566b4700 1 -- start start 2020-11-20T12:06:19.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.135+0000 7efc566b4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc500b57b0 0x7efc50068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.135+0000 7efc566b4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efc501482d0 0x7efc50062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.135+0000 7efc566b4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc50144b80 con 0x7efc50062b50 2020-11-20T12:06:19.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.135+0000 7efc566b4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc501489e0 con 0x7efc501482d0 2020-11-20T12:06:19.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.135+0000 7efc566b4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc50148c80 con 0x7efc500b57b0 2020-11-20T12:06:19.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.136+0000 7efc55eb3700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7efc50062b50 0x7efc50065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38846/0 (socket says 172.21.15.187:38846) 2020-11-20T12:06:19.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.136+0000 7efc55eb3700 1 -- 172.21.15.187:0/3645222513 learned_addr learned my addr 172.21.15.187:0/3645222513 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:19.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.136+0000 7efc556b2700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc500b57b0 0x7efc50068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.136+0000 7efc54eb1700 1 --2- 172.21.15.187:0/3645222513 >> v2:172.21.15.5:3300/0 conn(0x7efc501482d0 0x7efc50062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.136+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615064974 0 0) 0x7efc50144b80 con 0x7efc50062b50 2020-11-20T12:06:19.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3c003240 con 0x7efc50062b50 2020-11-20T12:06:19.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4143581175 0 0) 0x7efc3c003240 con 0x7efc50062b50 2020-11-20T12:06:19.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc500b57b0 msgr2=0x7efc50068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:19.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc500b57b0 0x7efc50068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 >> v2:172.21.15.5:3300/0 conn(0x7efc501482d0 msgr2=0x7efc50062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:19.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 --2- 172.21.15.187:0/3645222513 >> v2:172.21.15.5:3300/0 conn(0x7efc501482d0 0x7efc50062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50148f20 con 0x7efc50062b50 2020-11-20T12:06:19.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7efc48001860 con 0x7efc50062b50 2020-11-20T12:06:19.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 >> v1:172.21.15.187:6789/0 conn(0x7efc50062b50 legacy=0x7efc50065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc3c0037b0 0x7efc3c003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c0044e0 0x7efc3c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc500b57b0 0x7efc3c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc501489e0 con 0x7efc3c0037b0 2020-11-20T12:06:19.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc50148c80 con 0x7efc3c0044e0 2020-11-20T12:06:19.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.137+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c005340 con 0x7efc500b57b0 2020-11-20T12:06:19.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.138+0000 7efc54eb1700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c0044e0 0x7efc3c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.138+0000 7efc556b2700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc3c0037b0 0x7efc3c003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.138+0000 7efc54eb1700 1 -- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc500b57b0 msgr2=0x7efc3c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:19.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.138+0000 7efc54eb1700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc500b57b0 0x7efc3c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:19.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.138+0000 7efc54eb1700 1 -- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc3c0037b0 msgr2=0x7efc3c003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:19.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.138+0000 7efc54eb1700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc3c0037b0 0x7efc3c003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.138+0000 7efc54eb1700 1 -- 172.21.15.187:0/3645222513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c003240 con 0x7efc3c0044e0 2020-11-20T12:06:19.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.138+0000 7efc54eb1700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c0044e0 0x7efc3c004960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7efc50063440 tx=0x7efc44006df0).ready entity=mon.1 client_cookie=483b16d31dc56ca server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:19.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.139+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4400b330 con 0x7efc3c0044e0 2020-11-20T12:06:19.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.139+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4400eae0 con 0x7efc3c0044e0 2020-11-20T12:06:19.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.139+0000 7efc4ffff700 1 -- 172.21.15.187:0/3645222513 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4400ee60 con 0x7efc3c0044e0 2020-11-20T12:06:19.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.139+0000 7efc566b4700 1 -- 172.21.15.187:0/3645222513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c0044e0 msgr2=0x7efc3c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.139+0000 7efc566b4700 1 --2- 172.21.15.187:0/3645222513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c0044e0 0x7efc3c004960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7efc50063440 tx=0x7efc44006df0).stop 2020-11-20T12:06:19.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.139+0000 7efc55eb3700 1 -- 172.21.15.187:0/3645222513 reap_dead start 2020-11-20T12:06:19.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.140+0000 7efc566b4700 1 -- 172.21.15.187:0/3645222513 shutdown_connections 2020-11-20T12:06:19.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.140+0000 7efc566b4700 1 -- 172.21.15.187:0/3645222513 >> 172.21.15.187:0/3645222513 conn(0x7efc500bab30 msgr2=0x7efc500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:19.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.140+0000 7efc566b4700 1 -- 172.21.15.187:0/3645222513 shutdown_connections 2020-11-20T12:06:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.140+0000 7efc566b4700 1 -- 172.21.15.187:0/3645222513 wait complete. 2020-11-20T12:06:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.141+0000 7efc566b4700 1 Processor -- start 2020-11-20T12:06:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.141+0000 7efc566b4700 1 -- start start 2020-11-20T12:06:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.142+0000 7efc566b4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc500a0120 0x7efc500a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.142+0000 7efc566b4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5009f710 0x7efc50157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.142+0000 7efc566b4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc50142cd0 0x7efc50143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.142+0000 7efc566b4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5009ced0 con 0x7efc500a0120 2020-11-20T12:06:19.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.142+0000 7efc566b4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5009d170 con 0x7efc5009f710 2020-11-20T12:06:19.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.142+0000 7efc566b4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5009d410 con 0x7efc50142cd0 2020-11-20T12:06:19.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5009f710 0x7efc50157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5009f710 0x7efc50157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42252/0 (socket says 172.21.15.187:42252) 2020-11-20T12:06:19.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 -- 172.21.15.187:0/779775192 learned_addr learned my addr 172.21.15.187:0/779775192 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:19.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc55eb3700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc50142cd0 0x7efc50143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc556b2700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc500a0120 0x7efc500a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 -- 172.21.15.187:0/779775192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc50142cd0 msgr2=0x7efc50143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc50142cd0 0x7efc50143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 -- 172.21.15.187:0/779775192 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc500a0120 msgr2=0x7efc500a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc500a0120 0x7efc500a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 -- 172.21.15.187:0/779775192 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50158290 con 0x7efc5009f710 2020-11-20T12:06:19.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.143+0000 7efc54eb1700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5009f710 0x7efc50157da0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7efc44006fd0 tx=0x7efc44016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:19.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.144+0000 7efc4d7fa700 1 -- 172.21.15.187:0/779775192 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44023070 con 0x7efc5009f710 2020-11-20T12:06:19.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.144+0000 7efc4d7fa700 1 -- 172.21.15.187:0/779775192 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc440125c0 con 0x7efc5009f710 2020-11-20T12:06:19.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.144+0000 7efc4d7fa700 1 -- 172.21.15.187:0/779775192 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44012960 con 0x7efc5009f710 2020-11-20T12:06:19.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.144+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5009dce0 con 0x7efc5009f710 2020-11-20T12:06:19.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.144+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc50145330 con 0x7efc5009f710 2020-11-20T12:06:19.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.145+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5009d5d0 con 0x7efc5009f710 2020-11-20T12:06:19.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.145+0000 7efc4d7fa700 1 -- 172.21.15.187:0/779775192 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efc44012d10 con 0x7efc5009f710 2020-11-20T12:06:19.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.146+0000 7efc4d7fa700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc28056d20 0x7efc280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.146+0000 7efc4d7fa700 1 -- 172.21.15.187:0/779775192 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efc44072150 con 0x7efc5009f710 2020-11-20T12:06:19.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.147+0000 7efc556b2700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc28056d20 0x7efc280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.147+0000 7efc556b2700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc28056d20 0x7efc280591e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7efc40006a90 tx=0x7efc40008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:19.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.150+0000 7efc4d7fa700 1 -- 172.21.15.187:0/779775192 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efc44047490 con 0x7efc5009f710 2020-11-20T12:06:19.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.337+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc50061a20 con 0x7efc5009f710 2020-11-20T12:06:19.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.338+0000 7efc4d7fa700 1 -- 172.21.15.187:0/779775192 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v32) v1 ==== 104+0+10 (secure 0 0 0) 0x7efc44006e30 con 0x7efc5009f710 2020-11-20T12:06:19.343 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:06:19.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc28056d20 msgr2=0x7efc280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc28056d20 0x7efc280591e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7efc40006a90 tx=0x7efc40008040).stop 2020-11-20T12:06:19.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5009f710 msgr2=0x7efc50157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5009f710 0x7efc50157da0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7efc44006fd0 tx=0x7efc44016000).stop 2020-11-20T12:06:19.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 shutdown_connections 2020-11-20T12:06:19.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc28056d20 0x7efc280591e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc500a0120 0x7efc500a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc50142cd0 0x7efc50143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 --2- 172.21.15.187:0/779775192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5009f710 0x7efc50157da0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 >> 172.21.15.187:0/779775192 conn(0x7efc500bab30 msgr2=0x7efc500afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:19.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.341+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 shutdown_connections 2020-11-20T12:06:19.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.342+0000 7efc566b4700 1 -- 172.21.15.187:0/779775192 wait complete. 2020-11-20T12:06:19.369 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:06:19.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.611+0000 7fd8cc7fa700 1 Processor -- start 2020-11-20T12:06:19.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.611+0000 7fd8cc7fa700 1 -- start start 2020-11-20T12:06:19.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.612+0000 7fd8cc7fa700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd8c409a8a0 0x7fd8c40656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.612+0000 7fd8cc7fa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4068030 0x7fd8c4146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.612+0000 7fd8cc7fa700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c4066420 con 0x7fd8c4065300 2020-11-20T12:06:19.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.612+0000 7fd8cc7fa700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c40b7380 con 0x7fd8c409a8a0 2020-11-20T12:06:19.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.612+0000 7fd8cc7fa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4061f60 con 0x7fd8c4068030 2020-11-20T12:06:19.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.613+0000 7fd8ca596700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd8c4065300 0x7fd8c4062400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:60604/0 (socket says 172.21.15.187:60604) 2020-11-20T12:06:19.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.613+0000 7fd8ca596700 1 -- 172.21.15.187:0/3591360863 learned_addr learned my addr 172.21.15.187:0/3591360863 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:19.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.613+0000 7fd8c9d95700 1 --2- 172.21.15.187:0/3591360863 >> v2:172.21.15.5:3300/0 conn(0x7fd8c409a8a0 0x7fd8c40656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.613+0000 7fd8cad97700 1 --2- 172.21.15.187:0/3591360863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4068030 0x7fd8c4146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.614+0000 7fd8c9594700 1 -- 172.21.15.187:0/3591360863 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233030184 0 0) 0x7fd8c4066420 con 0x7fd8c4065300 2020-11-20T12:06:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.614+0000 7fd8c9594700 1 -- 172.21.15.187:0/3591360863 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ac003240 con 0x7fd8c4065300 2020-11-20T12:06:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.614+0000 7fd8c9d95700 1 -- 172.21.15.187:0/3591360863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4068030 msgr2=0x7fd8c4146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.614+0000 7fd8c9d95700 1 --2- 172.21.15.187:0/3591360863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4068030 0x7fd8c4146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.614+0000 7fd8c9d95700 1 -- 172.21.15.187:0/3591360863 >> v1:172.21.15.5:6789/0 conn(0x7fd8c4065300 legacy=0x7fd8c4062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.614+0000 7fd8c9d95700 1 -- 172.21.15.187:0/3591360863 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c40b6fd0 con 0x7fd8c409a8a0 2020-11-20T12:06:19.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.615+0000 7fd8c9d95700 1 --2- 172.21.15.187:0/3591360863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 0x7fd8c40656e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fd8b400b9a0 tx=0x7fd8b4006df0).ready entity=mon.0 client_cookie=e68df11ec76e0cf1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:19.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8c9594700 1 -- 172.21.15.187:0/3591360863 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4009710 con 0x7fd8c409a8a0 2020-11-20T12:06:19.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8c9594700 1 -- 172.21.15.187:0/3591360863 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b4009870 con 0x7fd8c409a8a0 2020-11-20T12:06:19.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8c9594700 1 -- 172.21.15.187:0/3591360863 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4009bf0 con 0x7fd8c409a8a0 2020-11-20T12:06:19.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3591360863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 msgr2=0x7fd8c40656e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3591360863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 0x7fd8c40656e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fd8b400b9a0 tx=0x7fd8b4006df0).stop 2020-11-20T12:06:19.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3591360863 shutdown_connections 2020-11-20T12:06:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3591360863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4068030 0x7fd8c4146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3591360863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 0x7fd8c40656e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.616+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3591360863 >> 172.21.15.187:0/3591360863 conn(0x7fd8c40bacf0 msgr2=0x7fd8c40bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.617+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3591360863 shutdown_connections 2020-11-20T12:06:19.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.617+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3591360863 wait complete. 2020-11-20T12:06:19.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.618+0000 7fd8cc7fa700 1 Processor -- start 2020-11-20T12:06:19.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.619+0000 7fd8cc7fa700 1 -- start start 2020-11-20T12:06:19.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8cc7fa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4065300 0x7fd8c40a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8cc7fa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c4068030 0x7fd8c415b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8cc7fa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 0x7fd8c40ba0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8cc7fa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40b8c50 con 0x7fd8c409a8a0 2020-11-20T12:06:19.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8cc7fa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40b2c80 con 0x7fd8c4068030 2020-11-20T12:06:19.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8cc7fa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40b2eb0 con 0x7fd8c4065300 2020-11-20T12:06:19.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8c9d95700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c4068030 0x7fd8c415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8c9d95700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c4068030 0x7fd8c415b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42264/0 (socket says 172.21.15.187:42264) 2020-11-20T12:06:19.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8c9d95700 1 -- 172.21.15.187:0/3734327226 learned_addr learned my addr 172.21.15.187:0/3734327226 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:19.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.620+0000 7fd8ca596700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4065300 0x7fd8c40a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8c9d95700 1 -- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4065300 msgr2=0x7fd8c40a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8c9d95700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4065300 0x7fd8c40a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8cad97700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 0x7fd8c40ba0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8c9d95700 1 -- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 msgr2=0x7fd8c40ba0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8c9d95700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 0x7fd8c40ba0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8c9d95700 1 -- 172.21.15.187:0/3734327226 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c40b3150 con 0x7fd8c4068030 2020-11-20T12:06:19.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8ca596700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4065300 0x7fd8c40a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:19.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8cad97700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 0x7fd8c40ba0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:19.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8c9d95700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c4068030 0x7fd8c415b940 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd8b4006e30 tx=0x7fd8b4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:19.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8b27fc700 1 -- 172.21.15.187:0/3734327226 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4011ea0 con 0x7fd8c4068030 2020-11-20T12:06:19.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8b27fc700 1 -- 172.21.15.187:0/3734327226 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b401d450 con 0x7fd8c4068030 2020-11-20T12:06:19.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8b27fc700 1 -- 172.21.15.187:0/3734327226 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b401d7f0 con 0x7fd8c4068030 2020-11-20T12:06:19.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.621+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c40a0a40 con 0x7fd8c4068030 2020-11-20T12:06:19.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.622+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c415c1e0 con 0x7fd8c4068030 2020-11-20T12:06:19.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.623+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c40a0c30 con 0x7fd8c4068030 2020-11-20T12:06:19.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.623+0000 7fd8b27fc700 1 -- 172.21.15.187:0/3734327226 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd8b400f070 con 0x7fd8c4068030 2020-11-20T12:06:19.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.624+0000 7fd8b27fc700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd89c056d20 0x7fd89c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:19.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.624+0000 7fd8b27fc700 1 -- 172.21.15.187:0/3734327226 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd8b4071260 con 0x7fd8c4068030 2020-11-20T12:06:19.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.624+0000 7fd8ca596700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd89c056d20 0x7fd89c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:19.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.625+0000 7fd8ca596700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd89c056d20 0x7fd89c0591e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd8c0005920 tx=0x7fd8c0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:19.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.628+0000 7fd8b27fc700 1 -- 172.21.15.187:0/3734327226 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd8b40466e0 con 0x7fd8c4068030 2020-11-20T12:06:19.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.808+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8c40b32e0 con 0x7fd8c4068030 2020-11-20T12:06:19.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.809+0000 7fd8b27fc700 1 -- 172.21.15.187:0/3734327226 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v32) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd8b4022020 con 0x7fd8c4068030 2020-11-20T12:06:19.814 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:06:19.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd89c056d20 msgr2=0x7fd89c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd89c056d20 0x7fd89c0591e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd8c0005920 tx=0x7fd8c0009040).stop 2020-11-20T12:06:19.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c4068030 msgr2=0x7fd8c415b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:19.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c4068030 0x7fd8c415b940 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd8b4006e30 tx=0x7fd8b4015000).stop 2020-11-20T12:06:19.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 shutdown_connections 2020-11-20T12:06:19.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd89c056d20 0x7fd89c0591e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c409a8a0 0x7fd8c40ba0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8c4065300 0x7fd8c40a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 --2- 172.21.15.187:0/3734327226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c4068030 0x7fd8c415b940 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 >> 172.21.15.187:0/3734327226 conn(0x7fd8c40bacf0 msgr2=0x7fd8c4146e10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.811+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 shutdown_connections 2020-11-20T12:06:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:19.812+0000 7fd8cc7fa700 1 -- 172.21.15.187:0/3734327226 wait complete. 2020-11-20T12:06:19.836 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:06:19.837 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:20.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.071+0000 7ff22abe8700 1 Processor -- start 2020-11-20T12:06:20.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.071+0000 7ff22abe8700 1 -- start start 2020-11-20T12:06:20.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.072+0000 7ff22abe8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff2240683f0 0x7ff224066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:20.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.072+0000 7ff22abe8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22414c170 con 0x7ff224145cc0 2020-11-20T12:06:20.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.072+0000 7ff22abe8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22414c410 con 0x7ff2241487e0 2020-11-20T12:06:20.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.072+0000 7ff22abe8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22414c6b0 con 0x7ff2240683f0 2020-11-20T12:06:20.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.073+0000 7ff2293e5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff2240683f0 0x7ff224066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:20.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.073+0000 7ff2293e5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff2240683f0 0x7ff224066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42270/0 (socket says 172.21.15.187:42270) 2020-11-20T12:06:20.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.073+0000 7ff2293e5700 1 -- 172.21.15.187:0/3290664598 learned_addr learned my addr 172.21.15.187:0/3290664598 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:20.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.073+0000 7ff2293e5700 1 -- 172.21.15.187:0/3290664598 >> v1:172.21.15.187:6789/0 conn(0x7ff2241487e0 legacy=0x7ff224148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:20.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.074+0000 7ff2293e5700 1 -- 172.21.15.187:0/3290664598 >> v1:172.21.15.5:6789/0 conn(0x7ff224145cc0 legacy=0x7ff224065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:20.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.074+0000 7ff2293e5700 1 -- 172.21.15.187:0/3290664598 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240658a0 con 0x7ff2240683f0 2020-11-20T12:06:20.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.074+0000 7ff2293e5700 1 --2- 172.21.15.187:0/3290664598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff224066040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7ff214006fd0 tx=0x7ff214006eb0).ready entity=mon.1 client_cookie=8e05d3bb890a83ec server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:20.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff228be4700 1 -- 172.21.15.187:0/3290664598 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214009890 con 0x7ff2240683f0 2020-11-20T12:06:20.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff228be4700 1 -- 172.21.15.187:0/3290664598 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2140099f0 con 0x7ff2240683f0 2020-11-20T12:06:20.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff228be4700 1 -- 172.21.15.187:0/3290664598 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214009d70 con 0x7ff2240683f0 2020-11-20T12:06:20.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff22abe8700 1 -- 172.21.15.187:0/3290664598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 msgr2=0x7ff224066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:20.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff22abe8700 1 --2- 172.21.15.187:0/3290664598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff224066040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7ff214006fd0 tx=0x7ff214006eb0).stop 2020-11-20T12:06:20.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff22abe8700 1 -- 172.21.15.187:0/3290664598 shutdown_connections 2020-11-20T12:06:20.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff22abe8700 1 --2- 172.21.15.187:0/3290664598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff224066040 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:20.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff22abe8700 1 -- 172.21.15.187:0/3290664598 >> 172.21.15.187:0/3290664598 conn(0x7ff2240bab30 msgr2=0x7ff2240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:20.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff22abe8700 1 -- 172.21.15.187:0/3290664598 shutdown_connections 2020-11-20T12:06:20.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.075+0000 7ff22abe8700 1 -- 172.21.15.187:0/3290664598 wait complete. 2020-11-20T12:06:20.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.077+0000 7ff22abe8700 1 Processor -- start 2020-11-20T12:06:20.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.077+0000 7ff22abe8700 1 -- start start 2020-11-20T12:06:20.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.078+0000 7ff22abe8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff22409fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:20.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.078+0000 7ff22abe8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff224145cc0 0x7ff22415d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:20.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.078+0000 7ff22abe8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2241487e0 0x7ff224144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:20.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.078+0000 7ff22abe8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22415c950 con 0x7ff2241487e0 2020-11-20T12:06:20.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.078+0000 7ff22abe8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22409ff00 con 0x7ff2240683f0 2020-11-20T12:06:20.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff22abe8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22415cbb0 con 0x7ff224145cc0 2020-11-20T12:06:20.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff229be6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff22409fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:20.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff229be6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff22409fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42276/0 (socket says 172.21.15.187:42276) 2020-11-20T12:06:20.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff229be6700 1 -- 172.21.15.187:0/852799540 learned_addr learned my addr 172.21.15.187:0/852799540 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:20.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff229be6700 1 -- 172.21.15.187:0/852799540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff224145cc0 msgr2=0x7ff22415d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T12:06:20.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff229be6700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff224145cc0 0x7ff22415d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:20.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff229be6700 1 -- 172.21.15.187:0/852799540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2241487e0 msgr2=0x7ff224144f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T12:06:20.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff229be6700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2241487e0 0x7ff224144f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:20.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.079+0000 7ff229be6700 1 -- 172.21.15.187:0/852799540 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22415ce50 con 0x7ff2240683f0 2020-11-20T12:06:20.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.080+0000 7ff229be6700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff22409fa10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ff22000c730 tx=0x7ff220007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:20.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.080+0000 7ff211ffb700 1 -- 172.21.15.187:0/852799540 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220007020 con 0x7ff2240683f0 2020-11-20T12:06:20.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.080+0000 7ff211ffb700 1 -- 172.21.15.187:0/852799540 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2200071a0 con 0x7ff2240683f0 2020-11-20T12:06:20.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.080+0000 7ff211ffb700 1 -- 172.21.15.187:0/852799540 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22000e560 con 0x7ff2240683f0 2020-11-20T12:06:20.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.080+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22415d7e0 con 0x7ff2240683f0 2020-11-20T12:06:20.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.080+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2240b1380 con 0x7ff2240683f0 2020-11-20T12:06:20.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.081+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff224147a90 con 0x7ff2240683f0 2020-11-20T12:06:20.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.082+0000 7ff211ffb700 1 -- 172.21.15.187:0/852799540 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff220010070 con 0x7ff2240683f0 2020-11-20T12:06:20.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.082+0000 7ff211ffb700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1fc056d20 0x7ff1fc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:20.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.082+0000 7ff211ffb700 1 -- 172.21.15.187:0/852799540 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff22006f3e0 con 0x7ff2240683f0 2020-11-20T12:06:20.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.083+0000 7ff2293e5700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1fc056d20 0x7ff1fc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:20.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.083+0000 7ff2293e5700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1fc056d20 0x7ff1fc0591e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff214006ef0 tx=0x7ff214015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:20.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.086+0000 7ff211ffb700 1 -- 172.21.15.187:0/852799540 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff220044830 con 0x7ff2240683f0 2020-11-20T12:06:20.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.329+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff224061a20 con 0x7ff2240683f0 2020-11-20T12:06:20.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.331+0000 7ff211ffb700 1 -- 172.21.15.187:0/852799540 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7ff220007d70 con 0x7ff2240683f0 2020-11-20T12:06:20.335 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:20.335 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:20.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1fc056d20 msgr2=0x7ff1fc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:20.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1fc056d20 0x7ff1fc0591e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff214006ef0 tx=0x7ff214015000).stop 2020-11-20T12:06:20.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 msgr2=0x7ff22409fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:20.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff22409fa10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ff22000c730 tx=0x7ff220007c50).stop 2020-11-20T12:06:20.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 shutdown_connections 2020-11-20T12:06:20.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1fc056d20 0x7ff1fc0591e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:20.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2241487e0 0x7ff224144f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:20.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2240683f0 0x7ff22409fa10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:20.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 --2- 172.21.15.187:0/852799540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff224145cc0 0x7ff22415d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:20.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 >> 172.21.15.187:0/852799540 conn(0x7ff2240bab30 msgr2=0x7ff2240b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:20.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.333+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 shutdown_connections 2020-11-20T12:06:20.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:20.334+0000 7ff22abe8700 1 -- 172.21.15.187:0/852799540 wait complete. 2020-11-20T12:06:20.341 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:20.563 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:06:11.574607+0000', 'modified': '2020-11-20T12:06:14.058447+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4966}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4966': {'gid': 4966, 'name': 'a', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/2376356363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2376356363}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2376356363}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:06:20.564 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:06:20.564 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:06:20.840 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":4966,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":9.2369250750000003,"mdsmap_epoch":13,"osdmap_epoch":32,"osdmap_epoch_barrier":32,"uptime":18.497790195} 2020-11-20T12:06:20.841 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 4966, "mdsmap_epoch": 13, "osdmap_epoch": 32, "osdmap_epoch_barrier": 32, "rank_uptime": 9.236925075, "state": "up:active", "uptime": 18.497790195, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:06:20.842 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:06:20.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:20.843 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:06:20.899 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:06:20.900 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:06:21.039 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:06:21.040 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:06:21.078 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:06:21.078 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:06:21.086 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:06:21.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:21.087 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:06:21.192 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:06:21.196 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:06:21.197 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:06:21.197 INFO:teuthology.orchestra.run.smithi187:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2020-11-20T12:06:21.198 INFO:teuthology.orchestra.run.smithi187:> subdir = "delete_me" 2020-11-20T12:06:21.198 INFO:teuthology.orchestra.run.smithi187:> size = 1024 2020-11-20T12:06:21.199 INFO:teuthology.orchestra.run.smithi187:> file_count = 1000 2020-11-20T12:06:21.199 INFO:teuthology.orchestra.run.smithi187:> os.mkdir(os.path.join(mountpoint, subdir)) 2020-11-20T12:06:21.202 INFO:teuthology.orchestra.run.smithi187:> for i in range(0, file_count): 2020-11-20T12:06:21.202 INFO:teuthology.orchestra.run.smithi187:> filename = "{0}_{1}.bin".format(i, size) 2020-11-20T12:06:21.204 INFO:teuthology.orchestra.run.smithi187:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2020-11-20T12:06:21.205 INFO:teuthology.orchestra.run.smithi187:> f.write(size * '"'"'x'"'"') 2020-11-20T12:06:21.205 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:06:23.999 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:24.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.289+0000 7f6d69f4a700 1 Processor -- start 2020-11-20T12:06:24.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.290+0000 7f6d69f4a700 1 -- start start 2020-11-20T12:06:24.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.291+0000 7f6d69f4a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b1760 0x7f6d64065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.291+0000 7f6d69f4a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6d64065680 0x7f6d6414c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.291+0000 7f6d69f4a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d64150390 con 0x7f6d640b1380 2020-11-20T12:06:24.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.291+0000 7f6d69f4a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d641511e0 con 0x7f6d64065680 2020-11-20T12:06:24.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.291+0000 7f6d69f4a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6414d760 con 0x7f6d640b1760 2020-11-20T12:06:24.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.292+0000 7f6d68f48700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6d640b1380 0x7f6d64150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38896/0 (socket says 172.21.15.187:38896) 2020-11-20T12:06:24.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.292+0000 7f6d68f48700 1 -- 172.21.15.187:0/2860844367 learned_addr learned my addr 172.21.15.187:0/2860844367 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:24.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.292+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235371947 0 0) 0x7f6d64150390 con 0x7f6d640b1380 2020-11-20T12:06:24.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.292+0000 7f6d5bfff700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b1760 0x7f6d64065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.292+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d50003240 con 0x7f6d640b1380 2020-11-20T12:06:24.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.292+0000 7f6d69749700 1 --2- 172.21.15.187:0/2860844367 >> v2:172.21.15.5:3300/0 conn(0x7f6d64065680 0x7f6d6414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 895617026 0 0) 0x7f6d50003240 con 0x7f6d640b1380 2020-11-20T12:06:24.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b1760 msgr2=0x7f6d64065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:24.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b1760 0x7f6d64065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 >> v2:172.21.15.5:3300/0 conn(0x7f6d64065680 msgr2=0x7f6d6414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:24.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 --2- 172.21.15.187:0/2860844367 >> v2:172.21.15.5:3300/0 conn(0x7f6d64065680 0x7f6d6414c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d64150630 con 0x7f6d640b1380 2020-11-20T12:06:24.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6d540014e0 con 0x7f6d640b1380 2020-11-20T12:06:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d69749700 1 --2- 172.21.15.187:0/2860844367 >> v2:172.21.15.5:3300/0 conn(0x7f6d64065680 0x7f6d6414c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:06:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 >> v1:172.21.15.187:6789/0 conn(0x7f6d640b1380 legacy=0x7f6d64150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d500037b0 0x7f6d50003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d500044e0 0x7f6d50004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.293+0000 7f6d5b7fe700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b1760 0x7f6d50004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641511e0 con 0x7f6d500044e0 2020-11-20T12:06:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6414d760 con 0x7f6d500037b0 2020-11-20T12:06:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500042b0 con 0x7f6d640b1760 2020-11-20T12:06:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d68f48700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d500037b0 0x7f6d50003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d69749700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b1760 0x7f6d50004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d5bfff700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d500044e0 0x7f6d50004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d68f48700 1 -- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b1760 msgr2=0x7f6d50004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:24.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d68f48700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b1760 0x7f6d50004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d68f48700 1 -- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d500044e0 msgr2=0x7f6d50004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:24.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d68f48700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d500044e0 0x7f6d50004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.294+0000 7f6d68f48700 1 -- 172.21.15.187:0/2860844367 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d50003240 con 0x7f6d500037b0 2020-11-20T12:06:24.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.295+0000 7f6d68f48700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d500037b0 0x7f6d50003b90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f6d54005020 tx=0x7f6d54008af0).ready entity=mon.1 client_cookie=b64a7fc4d0ed861f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:24.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.295+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54008e70 con 0x7f6d500037b0 2020-11-20T12:06:24.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.295+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d54008170 con 0x7f6d500037b0 2020-11-20T12:06:24.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.295+0000 7f6d5b7fe700 1 -- 172.21.15.187:0/2860844367 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54005800 con 0x7f6d500037b0 2020-11-20T12:06:24.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.295+0000 7f6d69f4a700 1 -- 172.21.15.187:0/2860844367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d500037b0 msgr2=0x7f6d50003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:24.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.295+0000 7f6d69f4a700 1 --2- 172.21.15.187:0/2860844367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d500037b0 0x7f6d50003b90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f6d54005020 tx=0x7f6d54008af0).stop 2020-11-20T12:06:24.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.295+0000 7f6d69749700 1 -- 172.21.15.187:0/2860844367 reap_dead start 2020-11-20T12:06:24.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.296+0000 7f6d69f4a700 1 -- 172.21.15.187:0/2860844367 shutdown_connections 2020-11-20T12:06:24.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.296+0000 7f6d69f4a700 1 -- 172.21.15.187:0/2860844367 >> 172.21.15.187:0/2860844367 conn(0x7f6d640bab30 msgr2=0x7f6d640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:24.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.296+0000 7f6d69f4a700 1 -- 172.21.15.187:0/2860844367 shutdown_connections 2020-11-20T12:06:24.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.296+0000 7f6d69f4a700 1 -- 172.21.15.187:0/2860844367 wait complete. 2020-11-20T12:06:24.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.297+0000 7f6d69f4a700 1 Processor -- start 2020-11-20T12:06:24.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.297+0000 7f6d69f4a700 1 -- start start 2020-11-20T12:06:24.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.298+0000 7f6d69f4a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b4320 0x7f6d6409bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.298+0000 7f6d69f4a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d640b4700 0x7f6d6415c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.298+0000 7f6d69f4a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d6415c110 0x7f6d6415f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.298+0000 7f6d69f4a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6409c3a0 con 0x7f6d640b4700 2020-11-20T12:06:24.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.298+0000 7f6d69f4a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6409c640 con 0x7f6d6415c110 2020-11-20T12:06:24.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.298+0000 7f6d69f4a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6409bff0 con 0x7f6d640b4320 2020-11-20T12:06:24.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.298+0000 7f6d69749700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d6415c110 0x7f6d6415f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d69749700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d6415c110 0x7f6d6415f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42306/0 (socket says 172.21.15.187:42306) 2020-11-20T12:06:24.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d69749700 1 -- 172.21.15.187:0/211326090 learned_addr learned my addr 172.21.15.187:0/211326090 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:24.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d68f48700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b4320 0x7f6d6409bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d5bfff700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d640b4700 0x7f6d6415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d69749700 1 -- 172.21.15.187:0/211326090 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b4320 msgr2=0x7f6d6409bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d69749700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b4320 0x7f6d6409bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d69749700 1 -- 172.21.15.187:0/211326090 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d640b4700 msgr2=0x7f6d6415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d69749700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d640b4700 0x7f6d6415c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d69749700 1 -- 172.21.15.187:0/211326090 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6415cc30 con 0x7f6d6415c110 2020-11-20T12:06:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.299+0000 7f6d69749700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d6415c110 0x7f6d6415f590 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6d640639b0 tx=0x7f6d5c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.300+0000 7f6d58ff9700 1 -- 172.21.15.187:0/211326090 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c011660 con 0x7f6d6415c110 2020-11-20T12:06:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.300+0000 7f6d58ff9700 1 -- 172.21.15.187:0/211326090 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c00c030 con 0x7f6d6415c110 2020-11-20T12:06:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.300+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6415ce80 con 0x7f6d6415c110 2020-11-20T12:06:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.300+0000 7f6d58ff9700 1 -- 172.21.15.187:0/211326090 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c011a10 con 0x7f6d6415c110 2020-11-20T12:06:24.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.300+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6415fc40 con 0x7f6d6415c110 2020-11-20T12:06:24.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.301+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6409c1b0 con 0x7f6d6415c110 2020-11-20T12:06:24.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.302+0000 7f6d58ff9700 1 -- 172.21.15.187:0/211326090 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6d5c013400 con 0x7f6d6415c110 2020-11-20T12:06:24.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.302+0000 7f6d58ff9700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d3c056d20 0x7f6d3c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:24.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.302+0000 7f6d58ff9700 1 -- 172.21.15.187:0/211326090 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6d5c06fc40 con 0x7f6d6415c110 2020-11-20T12:06:24.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.302+0000 7f6d68f48700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d3c056d20 0x7f6d3c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:24.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.303+0000 7f6d68f48700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d3c056d20 0x7f6d3c0591e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6d6405ea30 tx=0x7f6d54016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:24.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.306+0000 7f6d58ff9700 1 -- 172.21.15.187:0/211326090 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6d5c044f20 con 0x7f6d6415c110 2020-11-20T12:06:24.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.549+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d64061a20 con 0x7f6d6415c110 2020-11-20T12:06:24.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.550+0000 7f6d58ff9700 1 -- 172.21.15.187:0/211326090 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7f6d5c017020 con 0x7f6d6415c110 2020-11-20T12:06:24.554 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:24.555 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:24.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.552+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d3c056d20 msgr2=0x7f6d3c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:24.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.552+0000 7f6d69f4a700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d3c056d20 0x7f6d3c0591e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6d6405ea30 tx=0x7f6d54016000).stop 2020-11-20T12:06:24.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.552+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d6415c110 msgr2=0x7f6d6415f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:24.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.552+0000 7f6d69f4a700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d6415c110 0x7f6d6415f590 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6d640639b0 tx=0x7f6d5c008040).stop 2020-11-20T12:06:24.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.553+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 shutdown_connections 2020-11-20T12:06:24.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.553+0000 7f6d69f4a700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d3c056d20 0x7f6d3c0591e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.553+0000 7f6d69f4a700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d6415c110 0x7f6d6415f590 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.553+0000 7f6d69f4a700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d640b4320 0x7f6d6409bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.553+0000 7f6d69f4a700 1 --2- 172.21.15.187:0/211326090 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d640b4700 0x7f6d6415c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:24.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.553+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 >> 172.21.15.187:0/211326090 conn(0x7f6d640bab30 msgr2=0x7f6d6414cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:24.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.553+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 shutdown_connections 2020-11-20T12:06:24.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:24.553+0000 7f6d69f4a700 1 -- 172.21.15.187:0/211326090 wait complete. 2020-11-20T12:06:24.560 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:24.583 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:06:25.419 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:06:25.420 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:06:25.420 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:06:25.421 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:06:25.421 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:06:25.422 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:06:25.422 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2020-11-20T12:06:25.422 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:06:25.584 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627776 2020-11-20T12:06:25.772 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_metadata stat 10000000000.00000000 2020-11-20T12:06:25.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.861+0000 7f7165ab1d00 1 Processor -- start 2020-11-20T12:06:25.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.861+0000 7f7165ab1d00 1 -- start start 2020-11-20T12:06:25.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.862+0000 7f7165ab1d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d1f38d3740 0x55d1f3959440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:25.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.862+0000 7f7165ab1d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f3891010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:25.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f7165ab1d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d1f3960f00 0x55d1f38b6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:25.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f7165ab1d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f380c810 con 0x55d1f3960f00 2020-11-20T12:06:25.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f7165ab1d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f3959930 con 0x55d1f395ae30 2020-11-20T12:06:25.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f7165ab1d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1f395e300 con 0x55d1f38d3740 2020-11-20T12:06:25.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f714ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f3891010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:25.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f714ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f3891010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42322/0 (socket says 172.21.15.187:42322) 2020-11-20T12:06:25.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f714ffff700 1 -- 172.21.15.187:0/1816269158 learned_addr learned my addr 172.21.15.187:0/1816269158 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:25.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f714ffff700 1 -- 172.21.15.187:0/1816269158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d1f38d3740 msgr2=0x55d1f3959440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:25.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f714ffff700 1 --2- 172.21.15.187:0/1816269158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d1f38d3740 0x55d1f3959440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:25.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f714ffff700 1 -- 172.21.15.187:0/1816269158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d1f3960f00 msgr2=0x55d1f38b6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:25.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f714ffff700 1 --2- 172.21.15.187:0/1816269158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d1f3960f00 0x55d1f38b6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.863+0000 7f714ffff700 1 -- 172.21.15.187:0/1816269158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f38364e0 con 0x55d1f395ae30 2020-11-20T12:06:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f714ffff700 1 --2- 172.21.15.187:0/1816269158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f3891010 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f714000ab30 tx=0x7f7140005ba0).ready entity=mon.1 client_cookie=21c7b32df46ab999 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7155354700 1 -- 172.21.15.187:0/1816269158 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714000f070 con 0x55d1f395ae30 2020-11-20T12:06:25.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7155354700 1 -- 172.21.15.187:0/1816269158 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7140008b10 con 0x55d1f395ae30 2020-11-20T12:06:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7155354700 1 -- 172.21.15.187:0/1816269158 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140008e90 con 0x55d1f395ae30 2020-11-20T12:06:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7165ab1d00 1 -- 172.21.15.187:0/1816269158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 msgr2=0x55d1f3891010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7165ab1d00 1 --2- 172.21.15.187:0/1816269158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f3891010 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f714000ab30 tx=0x7f7140005ba0).stop 2020-11-20T12:06:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7165ab1d00 1 -- 172.21.15.187:0/1816269158 shutdown_connections 2020-11-20T12:06:25.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7165ab1d00 1 --2- 172.21.15.187:0/1816269158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d1f3960f00 0x55d1f38b6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7165ab1d00 1 --2- 172.21.15.187:0/1816269158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d1f38d3740 0x55d1f3959440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7165ab1d00 1 --2- 172.21.15.187:0/1816269158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f3891010 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.864+0000 7f7165ab1d00 1 -- 172.21.15.187:0/1816269158 >> 172.21.15.187:0/1816269158 conn(0x55d1f388ff00 msgr2=0x55d1f3960010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.865+0000 7f7165ab1d00 1 -- 172.21.15.187:0/1816269158 shutdown_connections 2020-11-20T12:06:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.865+0000 7f7165ab1d00 1 -- 172.21.15.187:0/1816269158 wait complete. 2020-11-20T12:06:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.865+0000 7f7165ab1d00 1 Processor -- start 2020-11-20T12:06:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.866+0000 7f7165ab1d00 1 -- start start 2020-11-20T12:06:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.866+0000 7f7165ab1d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d1f38d3740 0x55d1f3970260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.866+0000 7f7165ab1d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f38b8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.866+0000 7f7165ab1d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d1f3960f00 0x55d1f39588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.866+0000 7f7165ab1d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f392ce70 con 0x55d1f38d3740 2020-11-20T12:06:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.866+0000 7f7165ab1d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f3950410 con 0x55d1f395ae30 2020-11-20T12:06:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.866+0000 7f7165ab1d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1f394fa30 con 0x55d1f3960f00 2020-11-20T12:06:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f714ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f38b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:25.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f714ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f38b8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42328/0 (socket says 172.21.15.187:42328) 2020-11-20T12:06:25.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f714ffff700 1 -- 172.21.15.187:0/2094260477 learned_addr learned my addr 172.21.15.187:0/2094260477 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:25.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f7156356700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d1f38d3740 0x55d1f3970260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:25.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f714ffff700 1 -- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d1f3960f00 msgr2=0x55d1f39588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f714ffff700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d1f3960f00 0x55d1f39588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f714ffff700 1 -- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d1f38d3740 msgr2=0x55d1f3970260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f714ffff700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d1f38d3740 0x55d1f3970260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.867+0000 7f714ffff700 1 -- 172.21.15.187:0/2094260477 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f3831ec0 con 0x55d1f395ae30 2020-11-20T12:06:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.868+0000 7f714ffff700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f38b8330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f7140005970 tx=0x7f7140018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.868+0000 7f714effd700 1 -- 172.21.15.187:0/2094260477 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714001c650 con 0x55d1f395ae30 2020-11-20T12:06:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.868+0000 7f714effd700 1 -- 172.21.15.187:0/2094260477 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714001c7b0 con 0x55d1f395ae30 2020-11-20T12:06:25.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.868+0000 7f714effd700 1 -- 172.21.15.187:0/2094260477 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714001cb50 con 0x55d1f395ae30 2020-11-20T12:06:25.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.868+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d1f3810250 con 0x55d1f395ae30 2020-11-20T12:06:25.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.868+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1f394ff70 con 0x55d1f395ae30 2020-11-20T12:06:25.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.870+0000 7f714effd700 1 -- 172.21.15.187:0/2094260477 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f714002c070 con 0x55d1f395ae30 2020-11-20T12:06:25.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.870+0000 7f714effd700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71380559a0 0x7f7138057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:25.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.871+0000 7f714effd700 1 -- 172.21.15.187:0/2094260477 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7140070fc0 con 0x55d1f395ae30 2020-11-20T12:06:25.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.871+0000 7f7156356700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71380559a0 0x7f7138057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:25.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.871+0000 7f7165ab1d00 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55d1f395cbd0 0x55d1f39737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:25.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.871+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:1 5.0 5:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected e32) v8 -- 0x55d1f3958da0 con 0x55d1f395cbd0 2020-11-20T12:06:25.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.871+0000 7f7156356700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71380559a0 0x7f7138057e60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x55d1f38b6320 tx=0x7f7148000b80).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:25.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.872+0000 7f7156b57700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55d1f395cbd0 0x55d1f39737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:25.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.873+0000 7f7156b57700 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55d1f395cbd0 0x55d1f39737b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:25.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7156b57700 1 -- 172.21.15.187:0/2094260477 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv13 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7f7150010570 con 0x55d1f395cbd0 2020-11-20T12:06:25.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55d1f395cbd0 msgr2=0x55d1f39737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:25.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7165ab1d00 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55d1f395cbd0 0x55d1f39737b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:25.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71380559a0 msgr2=0x7f7138057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:25.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7165ab1d00 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71380559a0 0x7f7138057e60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x55d1f38b6320 tx=0x7f7148000b80).stop 2020-11-20T12:06:25.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 msgr2=0x55d1f38b8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:25.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7165ab1d00 1 --2- 172.21.15.187:0/2094260477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d1f395ae30 0x55d1f38b8330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f7140005970 tx=0x7f7140018000).stop 2020-11-20T12:06:25.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7156b57700 1 -- 172.21.15.187:0/2094260477 reap_dead start 2020-11-20T12:06:25.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 shutdown_connections 2020-11-20T12:06:25.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.874+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 >> 172.21.15.187:0/2094260477 conn(0x55d1f388ff00 msgr2=0x55d1f38b6550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:25.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.875+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 shutdown_connections 2020-11-20T12:06:25.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:25.875+0000 7f7165ab1d00 1 -- 172.21.15.187:0/2094260477 wait complete. 2020-11-20T12:06:25.890 INFO:teuthology.orchestra.run.smithi187.stdout:cephfs_metadata/10000000000.00000000 mtime 2020-11-20T12:06:25.000000+0000, size 0 2020-11-20T12:06:25.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:06:25.891 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -rf delete_me) 2020-11-20T12:06:29.536 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:29.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.782+0000 7fd9e4c4f700 1 Processor -- start 2020-11-20T12:06:29.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.783+0000 7fd9e4c4f700 1 -- start start 2020-11-20T12:06:29.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.785+0000 7fd9e4c4f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9e0065e10 0x7fd9e0144470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:29.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.785+0000 7fd9e4c4f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0099ce0 0x7fd9e0144960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:29.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.785+0000 7fd9e4c4f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e00b5df0 con 0x7fd9e00661f0 2020-11-20T12:06:29.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.785+0000 7fd9e4c4f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e00b72f0 con 0x7fd9e0065e10 2020-11-20T12:06:29.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.785+0000 7fd9e4c4f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e009a440 con 0x7fd9e0099ce0 2020-11-20T12:06:29.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.785+0000 7fd9de59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9e0065e10 0x7fd9e0144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:29.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.786+0000 7fd9de59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9e0065e10 0x7fd9e0144470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42344/0 (socket says 172.21.15.187:42344) 2020-11-20T12:06:29.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.786+0000 7fd9de59c700 1 -- 172.21.15.187:0/490386729 learned_addr learned my addr 172.21.15.187:0/490386729 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:29.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.786+0000 7fd9de59c700 1 -- 172.21.15.187:0/490386729 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0099ce0 msgr2=0x7fd9e0144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:29.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.786+0000 7fd9de59c700 1 --2- 172.21.15.187:0/490386729 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0099ce0 0x7fd9e0144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:29.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.786+0000 7fd9de59c700 1 -- 172.21.15.187:0/490386729 >> v1:172.21.15.5:6789/0 conn(0x7fd9e00661f0 legacy=0x7fd9e0061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:29.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.786+0000 7fd9de59c700 1 -- 172.21.15.187:0/490386729 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e009a1a0 con 0x7fd9e0065e10 2020-11-20T12:06:29.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.787+0000 7fd9de59c700 1 --2- 172.21.15.187:0/490386729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0065e10 0x7fd9e0144470 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd9e00b7430 tx=0x7fd9d0006eb0).ready entity=mon.1 client_cookie=f6fef170db426d28 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:29.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.787+0000 7fd9dd59a700 1 -- 172.21.15.187:0/490386729 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0009960 con 0x7fd9e0065e10 2020-11-20T12:06:29.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.787+0000 7fd9dd59a700 1 -- 172.21.15.187:0/490386729 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d0009ac0 con 0x7fd9e0065e10 2020-11-20T12:06:29.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.787+0000 7fd9dd59a700 1 -- 172.21.15.187:0/490386729 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0009e40 con 0x7fd9e0065e10 2020-11-20T12:06:29.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.788+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/490386729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0065e10 msgr2=0x7fd9e0144470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:29.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.788+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/490386729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0065e10 0x7fd9e0144470 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd9e00b7430 tx=0x7fd9d0006eb0).stop 2020-11-20T12:06:29.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.788+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/490386729 shutdown_connections 2020-11-20T12:06:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.788+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/490386729 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0099ce0 0x7fd9e0144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.788+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/490386729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0065e10 0x7fd9e0144470 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.788+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/490386729 >> 172.21.15.187:0/490386729 conn(0x7fd9e00bacd0 msgr2=0x7fd9e00bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.788+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/490386729 shutdown_connections 2020-11-20T12:06:29.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.788+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/490386729 wait complete. 2020-11-20T12:06:29.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.790+0000 7fd9e4c4f700 1 Processor -- start 2020-11-20T12:06:29.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.791+0000 7fd9e4c4f700 1 -- start start 2020-11-20T12:06:29.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9e4c4f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0065e10 0x7fd9e0140e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:29.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9e4c4f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9e00661f0 0x7fd9e0141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:29.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9e4c4f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0099ce0 0x7fd9e01431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:29.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9e4c4f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0142ad0 con 0x7fd9e00661f0 2020-11-20T12:06:29.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9e4c4f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0141e50 con 0x7fd9e0099ce0 2020-11-20T12:06:29.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9e4c4f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0141bb0 con 0x7fd9e0065e10 2020-11-20T12:06:29.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9ded9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0099ce0 0x7fd9e01431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:29.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9ded9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0099ce0 0x7fd9e01431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42354/0 (socket says 172.21.15.187:42354) 2020-11-20T12:06:29.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.792+0000 7fd9ded9d700 1 -- 172.21.15.187:0/3318394214 learned_addr learned my addr 172.21.15.187:0/3318394214 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:29.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9ddd9b700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9e00661f0 0x7fd9e0141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:29.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9de59c700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0065e10 0x7fd9e0140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:29.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9ded9d700 1 -- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0065e10 msgr2=0x7fd9e0140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9ded9d700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0065e10 0x7fd9e0140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9ded9d700 1 -- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9e00661f0 msgr2=0x7fd9e0141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9ded9d700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9e00661f0 0x7fd9e0141370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9ded9d700 1 -- 172.21.15.187:0/3318394214 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e0097d80 con 0x7fd9e0099ce0 2020-11-20T12:06:29.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9de59c700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0065e10 0x7fd9e0140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:29.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9ded9d700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0099ce0 0x7fd9e01431a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd9d800bab0 tx=0x7fd9d80091a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:29.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.793+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/3318394214 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d800c890 con 0x7fd9e0099ce0 2020-11-20T12:06:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.794+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/3318394214 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d800c9f0 con 0x7fd9e0099ce0 2020-11-20T12:06:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.794+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/3318394214 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d800cba0 con 0x7fd9e0099ce0 2020-11-20T12:06:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.794+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e0098180 con 0x7fd9e0099ce0 2020-11-20T12:06:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.794+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e0098460 con 0x7fd9e0099ce0 2020-11-20T12:06:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.795+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e0097f70 con 0x7fd9e0099ce0 2020-11-20T12:06:29.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.796+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/3318394214 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd9d801c070 con 0x7fd9e0099ce0 2020-11-20T12:06:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.797+0000 7fd9ce7fc700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9b4056d20 0x7fd9b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.797+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/3318394214 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd9d806f670 con 0x7fd9e0099ce0 2020-11-20T12:06:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.798+0000 7fd9de59c700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9b4056d20 0x7fd9b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.798+0000 7fd9de59c700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9b4056d20 0x7fd9b40591e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd9d0006ef0 tx=0x7fd9d0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:29.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:29.802+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/3318394214 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd9e0097f70 con 0x7fd9e0099ce0 2020-11-20T12:06:30.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.135+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9e0097f70 con 0x7fd9e0099ce0 2020-11-20T12:06:30.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.135+0000 7fd9ce7fc700 1 -- 172.21.15.187:0/3318394214 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7fd9e0097f70 con 0x7fd9e0099ce0 2020-11-20T12:06:30.139 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:30.139 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:30.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9b4056d20 msgr2=0x7fd9b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:30.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9b4056d20 0x7fd9b40591e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd9d0006ef0 tx=0x7fd9d0015000).stop 2020-11-20T12:06:30.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0099ce0 msgr2=0x7fd9e01431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:30.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0099ce0 0x7fd9e01431a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd9d800bab0 tx=0x7fd9d80091a0).stop 2020-11-20T12:06:30.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 shutdown_connections 2020-11-20T12:06:30.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9b4056d20 0x7fd9b40591e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:30.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e0099ce0 0x7fd9e01431a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:30.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9e0065e10 0x7fd9e0140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:30.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 --2- 172.21.15.187:0/3318394214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9e00661f0 0x7fd9e0141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:30.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 >> 172.21.15.187:0/3318394214 conn(0x7fd9e00bacd0 msgr2=0x7fd9e01442d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:30.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 shutdown_connections 2020-11-20T12:06:30.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.138+0000 7fd9e4c4f700 1 -- 172.21.15.187:0/3318394214 wait complete. 2020-11-20T12:06:30.145 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:30.164 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:06:30.556 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:06:30.557 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:06:30.557 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:30.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.791+0000 7f5348a61700 1 Processor -- start 2020-11-20T12:06:30.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.792+0000 7f5348a61700 1 -- start start 2020-11-20T12:06:30.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.793+0000 7f5348a61700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53441484d0 0x7f5344061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:30.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.793+0000 7f5348a61700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53440b1300 con 0x7f53440622c0 2020-11-20T12:06:30.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.793+0000 7f5348a61700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53440b15a0 con 0x7f5344066f70 2020-11-20T12:06:30.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.793+0000 7f5348a61700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5344067730 con 0x7f53441484d0 2020-11-20T12:06:30.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.793+0000 7f53437fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53441484d0 0x7f5344061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:30.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.793+0000 7f53437fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53441484d0 0x7f5344061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42360/0 (socket says 172.21.15.187:42360) 2020-11-20T12:06:30.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.793+0000 7f53437fe700 1 -- 172.21.15.187:0/4095113009 learned_addr learned my addr 172.21.15.187:0/4095113009 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:30.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.794+0000 7f53437fe700 1 -- 172.21.15.187:0/4095113009 >> v1:172.21.15.187:6789/0 conn(0x7f5344066f70 legacy=0x7f53440b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:30.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.794+0000 7f53437fe700 1 -- 172.21.15.187:0/4095113009 >> v1:172.21.15.5:6789/0 conn(0x7f53440622c0 legacy=0x7f53440b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:30.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.794+0000 7f53437fe700 1 -- 172.21.15.187:0/4095113009 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53440b1840 con 0x7f53441484d0 2020-11-20T12:06:30.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.794+0000 7f53437fe700 1 --2- 172.21.15.187:0/4095113009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53441484d0 0x7f5344061dd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5334006fd0 tx=0x7f5334006eb0).ready entity=mon.1 client_cookie=496070480f569ead server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:30.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f53427fc700 1 -- 172.21.15.187:0/4095113009 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334009890 con 0x7f53441484d0 2020-11-20T12:06:30.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f53427fc700 1 -- 172.21.15.187:0/4095113009 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53340099f0 con 0x7f53441484d0 2020-11-20T12:06:30.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f53427fc700 1 -- 172.21.15.187:0/4095113009 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334009d70 con 0x7f53441484d0 2020-11-20T12:06:30.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f5348a61700 1 -- 172.21.15.187:0/4095113009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53441484d0 msgr2=0x7f5344061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:30.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f5348a61700 1 --2- 172.21.15.187:0/4095113009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53441484d0 0x7f5344061dd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5334006fd0 tx=0x7f5334006eb0).stop 2020-11-20T12:06:30.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f5348a61700 1 -- 172.21.15.187:0/4095113009 shutdown_connections 2020-11-20T12:06:30.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f5348a61700 1 --2- 172.21.15.187:0/4095113009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53441484d0 0x7f5344061dd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:30.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f5348a61700 1 -- 172.21.15.187:0/4095113009 >> 172.21.15.187:0/4095113009 conn(0x7f53440bab30 msgr2=0x7f53440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:30.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f5348a61700 1 -- 172.21.15.187:0/4095113009 shutdown_connections 2020-11-20T12:06:30.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.795+0000 7f5348a61700 1 -- 172.21.15.187:0/4095113009 wait complete. 2020-11-20T12:06:30.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.797+0000 7f5348a61700 1 Processor -- start 2020-11-20T12:06:30.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.797+0000 7f5348a61700 1 -- start start 2020-11-20T12:06:30.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5348a61700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53440622c0 0x7f5344099420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:30.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5348a61700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5344066f70 0x7f5344144430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:30.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5348a61700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53441484d0 0x7f5344144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:30.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5348a61700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53440b3f10 con 0x7f53440622c0 2020-11-20T12:06:30.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5348a61700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53440b41b0 con 0x7f5344066f70 2020-11-20T12:06:30.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5348a61700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5344099910 con 0x7f53441484d0 2020-11-20T12:06:30.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5342ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5344066f70 0x7f5344144430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:30.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5342ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5344066f70 0x7f5344144430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42366/0 (socket says 172.21.15.187:42366) 2020-11-20T12:06:30.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.798+0000 7f5342ffd700 1 -- 172.21.15.187:0/3367459907 learned_addr learned my addr 172.21.15.187:0/3367459907 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:30.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f5342ffd700 1 -- 172.21.15.187:0/3367459907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53441484d0 msgr2=0x7f5344144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:30.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f5342ffd700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53441484d0 0x7f5344144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:30.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f5342ffd700 1 -- 172.21.15.187:0/3367459907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53440622c0 msgr2=0x7f5344099420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f5342ffd700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53440622c0 0x7f5344099420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f5342ffd700 1 -- 172.21.15.187:0/3367459907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534409a740 con 0x7f5344066f70 2020-11-20T12:06:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f5342ffd700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5344066f70 0x7f5344144430 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f533800c710 tx=0x7f5338007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f53237fe700 1 -- 172.21.15.187:0/3367459907 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338007020 con 0x7f5344066f70 2020-11-20T12:06:30.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f53237fe700 1 -- 172.21.15.187:0/3367459907 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5338007180 con 0x7f5344066f70 2020-11-20T12:06:30.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f53237fe700 1 -- 172.21.15.187:0/3367459907 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800e560 con 0x7f5344066f70 2020-11-20T12:06:30.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53440b3ae0 con 0x7f5344066f70 2020-11-20T12:06:30.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.799+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53440b2e00 con 0x7f5344066f70 2020-11-20T12:06:30.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.801+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53440b4370 con 0x7f5344066f70 2020-11-20T12:06:30.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.801+0000 7f53237fe700 1 -- 172.21.15.187:0/3367459907 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5338010070 con 0x7f5344066f70 2020-11-20T12:06:30.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.802+0000 7f53237fe700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f531c056d20 0x7f531c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.802+0000 7f53237fe700 1 -- 172.21.15.187:0/3367459907 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f533806f4c0 con 0x7f5344066f70 2020-11-20T12:06:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.802+0000 7f53437fe700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f531c056d20 0x7f531c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.803+0000 7f53437fe700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f531c056d20 0x7f531c0591e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5334006ef0 tx=0x7f5334015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:30.806+0000 7f53237fe700 1 -- 172.21.15.187:0/3367459907 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5338044910 con 0x7f5344066f70 2020-11-20T12:06:31.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.052+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5344062000 con 0x7f5344066f70 2020-11-20T12:06:31.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.053+0000 7f53237fe700 1 -- 172.21.15.187:0/3367459907 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7f5338047f30 con 0x7f5344066f70 2020-11-20T12:06:31.056 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:31.057 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:31.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.055+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f531c056d20 msgr2=0x7f531c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:31.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.055+0000 7f5348a61700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f531c056d20 0x7f531c0591e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5334006ef0 tx=0x7f5334015000).stop 2020-11-20T12:06:31.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.055+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5344066f70 msgr2=0x7f5344144430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:31.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.055+0000 7f5348a61700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5344066f70 0x7f5344144430 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f533800c710 tx=0x7f5338007c20).stop 2020-11-20T12:06:31.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.056+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 shutdown_connections 2020-11-20T12:06:31.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.056+0000 7f5348a61700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f531c056d20 0x7f531c0591e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.056+0000 7f5348a61700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53440622c0 0x7f5344099420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:31.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.056+0000 7f5348a61700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53441484d0 0x7f5344144920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:31.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.056+0000 7f5348a61700 1 --2- 172.21.15.187:0/3367459907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5344066f70 0x7f5344144430 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.056+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 >> 172.21.15.187:0/3367459907 conn(0x7f53440bab30 msgr2=0x7f5344145020 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:31.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.056+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 shutdown_connections 2020-11-20T12:06:31.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.056+0000 7f5348a61700 1 -- 172.21.15.187:0/3367459907 wait complete. 2020-11-20T12:06:31.062 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:31.082 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:06:31.351 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_created":1001}} 2020-11-20T12:06:31.351 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2020-11-20T12:06:31.352 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:31.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.592+0000 7fba73fff700 1 Processor -- start 2020-11-20T12:06:31.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.593+0000 7fba73fff700 1 -- start start 2020-11-20T12:06:31.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.594+0000 7fba73fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fba740b1380 0x7fba74150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:31.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.594+0000 7fba73fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba7414c260 0x7fba7414c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:31.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.594+0000 7fba73fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba740658a0 con 0x7fba740b1760 2020-11-20T12:06:31.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.594+0000 7fba73fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba74150390 con 0x7fba740b1380 2020-11-20T12:06:31.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.594+0000 7fba73fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba74150630 con 0x7fba7414c260 2020-11-20T12:06:31.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.594+0000 7fba737fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba7414c260 0x7fba7414c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:31.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.594+0000 7fba737fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba7414c260 0x7fba7414c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42376/0 (socket says 172.21.15.187:42376) 2020-11-20T12:06:31.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.594+0000 7fba737fe700 1 -- 172.21.15.187:0/2917330797 learned_addr learned my addr 172.21.15.187:0/2917330797 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:31.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.595+0000 7fba72ffd700 1 --2- 172.21.15.187:0/2917330797 >> v2:172.21.15.5:3300/0 conn(0x7fba740b1380 0x7fba74150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:31.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.595+0000 7fba737fe700 1 -- 172.21.15.187:0/2917330797 >> v2:172.21.15.5:3300/0 conn(0x7fba740b1380 msgr2=0x7fba74150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:31.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.595+0000 7fba737fe700 1 --2- 172.21.15.187:0/2917330797 >> v2:172.21.15.5:3300/0 conn(0x7fba740b1380 0x7fba74150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.595+0000 7fba737fe700 1 -- 172.21.15.187:0/2917330797 >> v1:172.21.15.5:6789/0 conn(0x7fba740b1760 legacy=0x7fba74065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:31.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.595+0000 7fba737fe700 1 -- 172.21.15.187:0/2917330797 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba74068510 con 0x7fba7414c260 2020-11-20T12:06:31.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.595+0000 7fba737fe700 1 --2- 172.21.15.187:0/2917330797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7414c260 0x7fba7414c6c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fba74061740 tx=0x7fba6c009290).ready entity=mon.1 client_cookie=e6a4569ac3d5d6e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:31.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.596+0000 7fba727fc700 1 -- 172.21.15.187:0/2917330797 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c00bb80 con 0x7fba7414c260 2020-11-20T12:06:31.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.596+0000 7fba727fc700 1 -- 172.21.15.187:0/2917330797 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6c00bce0 con 0x7fba7414c260 2020-11-20T12:06:31.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.596+0000 7fba727fc700 1 -- 172.21.15.187:0/2917330797 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c011610 con 0x7fba7414c260 2020-11-20T12:06:31.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.596+0000 7fba73fff700 1 -- 172.21.15.187:0/2917330797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7414c260 msgr2=0x7fba7414c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:31.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.596+0000 7fba73fff700 1 --2- 172.21.15.187:0/2917330797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7414c260 0x7fba7414c6c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fba74061740 tx=0x7fba6c009290).stop 2020-11-20T12:06:31.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.596+0000 7fba73fff700 1 -- 172.21.15.187:0/2917330797 shutdown_connections 2020-11-20T12:06:31.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.597+0000 7fba73fff700 1 --2- 172.21.15.187:0/2917330797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7414c260 0x7fba7414c6c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.597+0000 7fba73fff700 1 --2- 172.21.15.187:0/2917330797 >> v2:172.21.15.5:3300/0 conn(0x7fba740b1380 0x7fba74150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.597+0000 7fba73fff700 1 -- 172.21.15.187:0/2917330797 >> 172.21.15.187:0/2917330797 conn(0x7fba740bab30 msgr2=0x7fba740baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:31.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.597+0000 7fba73fff700 1 -- 172.21.15.187:0/2917330797 shutdown_connections 2020-11-20T12:06:31.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.597+0000 7fba73fff700 1 -- 172.21.15.187:0/2917330797 wait complete. 2020-11-20T12:06:31.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.599+0000 7fba73fff700 1 Processor -- start 2020-11-20T12:06:31.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.599+0000 7fba73fff700 1 -- start start 2020-11-20T12:06:31.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba73fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba740b1380 0x7fba7409bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:31.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba73fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba740b1760 0x7fba7409e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:31.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba73fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba7414c260 0x7fba740b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:31.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba73fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7415fbe0 con 0x7fba7414c260 2020-11-20T12:06:31.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba73fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7415fe80 con 0x7fba740b1760 2020-11-20T12:06:31.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba73fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba74160120 con 0x7fba740b1380 2020-11-20T12:06:31.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba6a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba740b1760 0x7fba7409e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:31.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba6a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba740b1760 0x7fba7409e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42380/0 (socket says 172.21.15.187:42380) 2020-11-20T12:06:31.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba6a7fc700 1 -- 172.21.15.187:0/559687614 learned_addr learned my addr 172.21.15.187:0/559687614 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:31.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba72ffd700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba740b1380 0x7fba7409bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:31.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.601+0000 7fba6a7fc700 1 -- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba740b1380 msgr2=0x7fba7409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:31.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba6a7fc700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba740b1380 0x7fba7409bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba6a7fc700 1 -- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba7414c260 msgr2=0x7fba740b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:31.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba737fe700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba7414c260 0x7fba740b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:31.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba6a7fc700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba7414c260 0x7fba740b3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba6a7fc700 1 -- 172.21.15.187:0/559687614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba741605c0 con 0x7fba740b1760 2020-11-20T12:06:31.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba72ffd700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba740b1380 0x7fba7409bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:31.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba6a7fc700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba740b1760 0x7fba7409e240 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fba5c00c730 tx=0x7fba5c007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:31.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba6b7fe700 1 -- 172.21.15.187:0/559687614 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c00e560 con 0x7fba740b1760 2020-11-20T12:06:31.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba6b7fe700 1 -- 172.21.15.187:0/559687614 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5c00e6e0 con 0x7fba740b1760 2020-11-20T12:06:31.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7409e730 con 0x7fba740b1760 2020-11-20T12:06:31.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba6b7fe700 1 -- 172.21.15.187:0/559687614 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c00eaa0 con 0x7fba740b1760 2020-11-20T12:06:31.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.602+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7415f980 con 0x7fba740b1760 2020-11-20T12:06:31.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.604+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba741602e0 con 0x7fba740b1760 2020-11-20T12:06:31.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.604+0000 7fba6b7fe700 1 -- 172.21.15.187:0/559687614 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fba5c019030 con 0x7fba740b1760 2020-11-20T12:06:31.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.605+0000 7fba6b7fe700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba44056d20 0x7fba440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:31.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.605+0000 7fba6b7fe700 1 -- 172.21.15.187:0/559687614 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fba5c06f480 con 0x7fba740b1760 2020-11-20T12:06:31.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.605+0000 7fba72ffd700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba44056d20 0x7fba440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:31.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.606+0000 7fba72ffd700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba44056d20 0x7fba440591e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fba64009710 tx=0x7fba64008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:31.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.609+0000 7fba6b7fe700 1 -- 172.21.15.187:0/559687614 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fba5c043fd0 con 0x7fba740b1760 2020-11-20T12:06:31.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.855+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba74061a20 con 0x7fba740b1760 2020-11-20T12:06:31.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.856+0000 7fba6b7fe700 1 -- 172.21.15.187:0/559687614 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7fba5c044980 con 0x7fba740b1760 2020-11-20T12:06:31.859 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:31.860 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:31.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba44056d20 msgr2=0x7fba440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:31.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba44056d20 0x7fba440591e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fba64009710 tx=0x7fba64008040).stop 2020-11-20T12:06:31.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba740b1760 msgr2=0x7fba7409e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:31.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba740b1760 0x7fba7409e240 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fba5c00c730 tx=0x7fba5c007c30).stop 2020-11-20T12:06:31.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 shutdown_connections 2020-11-20T12:06:31.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba44056d20 0x7fba440591e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba7414c260 0x7fba740b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba740b1380 0x7fba7409bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 --2- 172.21.15.187:0/559687614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba740b1760 0x7fba7409e240 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:31.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 >> 172.21.15.187:0/559687614 conn(0x7fba740bab30 msgr2=0x7fba740afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:31.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 shutdown_connections 2020-11-20T12:06:31.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:31.859+0000 7fba73fff700 1 -- 172.21.15.187:0/559687614 wait complete. 2020-11-20T12:06:31.865 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:31.887 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:06:32.159 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":945}} 2020-11-20T12:06:32.160 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 945 } } 2020-11-20T12:06:32.160 DEBUG:tasks.ceph_test_case:wait_until_equal: 945 != 1001, waiting (timeout=600)... 2020-11-20T12:06:37.162 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:37.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.415+0000 7fd8ccedf700 1 Processor -- start 2020-11-20T12:06:37.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.415+0000 7fd8ccedf700 1 -- start start 2020-11-20T12:06:37.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.416+0000 7fd8ccedf700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd8c8065e10 0x7fd8c8144470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.416+0000 7fd8ccedf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c80b5df0 con 0x7fd8c8099ce0 2020-11-20T12:06:37.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.416+0000 7fd8ccedf700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c809a440 con 0x7fd8c80661f0 2020-11-20T12:06:37.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.416+0000 7fd8ccedf700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c80b72f0 con 0x7fd8c8065e10 2020-11-20T12:06:37.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.417+0000 7fd8bdd9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd8c80661f0 0x7fd8c8061de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:38994/0 (socket says 172.21.15.187:38994) 2020-11-20T12:06:37.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.417+0000 7fd8bdd9b700 1 -- 172.21.15.187:0/534356177 learned_addr learned my addr 172.21.15.187:0/534356177 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:37.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.417+0000 7fd8c659c700 1 --2- 172.21.15.187:0/534356177 >> v2:172.21.15.5:3300/0 conn(0x7fd8c8065e10 0x7fd8c8144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.417+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741196851 0 0) 0x7fd8c809a440 con 0x7fd8c80661f0 2020-11-20T12:06:37.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.417+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a8003240 con 0x7fd8c80661f0 2020-11-20T12:06:37.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1249297998 0 0) 0x7fd8a8003240 con 0x7fd8c80661f0 2020-11-20T12:06:37.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 >> v2:172.21.15.5:3300/0 conn(0x7fd8c8065e10 msgr2=0x7fd8c8144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:37.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 --2- 172.21.15.187:0/534356177 >> v2:172.21.15.5:3300/0 conn(0x7fd8c8065e10 0x7fd8c8144470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 >> v1:172.21.15.5:6789/0 conn(0x7fd8c8099ce0 legacy=0x7fd8c80624f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c8062c00 con 0x7fd8c80661f0 2020-11-20T12:06:37.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd8b0001660 con 0x7fd8c80661f0 2020-11-20T12:06:37.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 >> v1:172.21.15.187:6789/0 conn(0x7fd8c80661f0 legacy=0x7fd8c8061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a8003730 0x7fd8a8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a80044e0 0x7fd8a8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8a8004e50 0x7fd8a800c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c80b72f0 con 0x7fd8a8004e50 2020-11-20T12:06:37.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a8004280 con 0x7fd8a8003730 2020-11-20T12:06:37.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.418+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a800cd30 con 0x7fd8a80044e0 2020-11-20T12:06:37.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.419+0000 7fd8c659c700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a8003730 0x7fd8a8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.419+0000 7fd8c659c700 1 -- 172.21.15.187:0/534356177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a80044e0 msgr2=0x7fd8a8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:37.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.419+0000 7fd8c659c700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a80044e0 0x7fd8a8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:37.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.419+0000 7fd8c659c700 1 -- 172.21.15.187:0/534356177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8a8004e50 msgr2=0x7fd8a800c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:37.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.419+0000 7fd8c659c700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8a8004e50 0x7fd8a800c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:37.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.419+0000 7fd8c659c700 1 -- 172.21.15.187:0/534356177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a8003240 con 0x7fd8a8003730 2020-11-20T12:06:37.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.419+0000 7fd8c6d9d700 1 -- 172.21.15.187:0/534356177 reap_dead start 2020-11-20T12:06:37.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.419+0000 7fd8c659c700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a8003730 0x7fd8a8003b70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd8b800bc00 tx=0x7fd8b8006df0).ready entity=mon.1 client_cookie=eb09bf7b6dd14f6d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.420+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800b310 con 0x7fd8a8003730 2020-11-20T12:06:37.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.420+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b800eb40 con 0x7fd8a8003730 2020-11-20T12:06:37.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.420+0000 7fd8c5d9b700 1 -- 172.21.15.187:0/534356177 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800eec0 con 0x7fd8a8003730 2020-11-20T12:06:37.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.420+0000 7fd8ccedf700 1 -- 172.21.15.187:0/534356177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a8003730 msgr2=0x7fd8a8003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.420+0000 7fd8ccedf700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a8003730 0x7fd8a8003b70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd8b800bc00 tx=0x7fd8b8006df0).stop 2020-11-20T12:06:37.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.420+0000 7fd8ccedf700 1 -- 172.21.15.187:0/534356177 shutdown_connections 2020-11-20T12:06:37.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.420+0000 7fd8ccedf700 1 --2- 172.21.15.187:0/534356177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a8003730 0x7fd8a8003b70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.420+0000 7fd8ccedf700 1 -- 172.21.15.187:0/534356177 >> 172.21.15.187:0/534356177 conn(0x7fd8c80bacd0 msgr2=0x7fd8c80bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:37.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.421+0000 7fd8ccedf700 1 -- 172.21.15.187:0/534356177 shutdown_connections 2020-11-20T12:06:37.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.421+0000 7fd8ccedf700 1 -- 172.21.15.187:0/534356177 wait complete. 2020-11-20T12:06:37.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.422+0000 7fd8ccedf700 1 Processor -- start 2020-11-20T12:06:37.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.422+0000 7fd8ccedf700 1 -- start start 2020-11-20T12:06:37.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.423+0000 7fd8ccedf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a8003730 0x7fd8c814cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.423+0000 7fd8ccedf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c809dc80 0x7fd8c815b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.423+0000 7fd8ccedf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c815ba60 0x7fd8c8099360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.423+0000 7fd8ccedf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c809e550 con 0x7fd8c815ba60 2020-11-20T12:06:37.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.423+0000 7fd8ccedf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c809ac00 con 0x7fd8c809dc80 2020-11-20T12:06:37.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.423+0000 7fd8ccedf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c809e7f0 con 0x7fd8a8003730 2020-11-20T12:06:37.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.423+0000 7fd8bdd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c809dc80 0x7fd8c815b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8bdd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c809dc80 0x7fd8c815b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42400/0 (socket says 172.21.15.187:42400) 2020-11-20T12:06:37.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8bdd9b700 1 -- 172.21.15.187:0/1097288121 learned_addr learned my addr 172.21.15.187:0/1097288121 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:37.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8c6d9d700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c815ba60 0x7fd8c8099360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8c659c700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a8003730 0x7fd8c814cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8bdd9b700 1 -- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a8003730 msgr2=0x7fd8c814cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8bdd9b700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a8003730 0x7fd8c814cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8bdd9b700 1 -- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c815ba60 msgr2=0x7fd8c8099360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8bdd9b700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c815ba60 0x7fd8c8099360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8bdd9b700 1 -- 172.21.15.187:0/1097288121 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c814d2c0 con 0x7fd8c809dc80 2020-11-20T12:06:37.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8c659c700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a8003730 0x7fd8c814cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:37.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.424+0000 7fd8bdd9b700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c809dc80 0x7fd8c815b570 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fd8b000ca60 tx=0x7fd8b0002c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.425+0000 7fd8beffd700 1 -- 172.21.15.187:0/1097288121 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0007320 con 0x7fd8c809dc80 2020-11-20T12:06:37.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.425+0000 7fd8beffd700 1 -- 172.21.15.187:0/1097288121 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b000a5e0 con 0x7fd8c809dc80 2020-11-20T12:06:37.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.425+0000 7fd8beffd700 1 -- 172.21.15.187:0/1097288121 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b000a9a0 con 0x7fd8c809dc80 2020-11-20T12:06:37.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.425+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c809d4f0 con 0x7fd8c809dc80 2020-11-20T12:06:37.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.425+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c809e2c0 con 0x7fd8c809dc80 2020-11-20T12:06:37.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.426+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c80bb0f0 con 0x7fd8c809dc80 2020-11-20T12:06:37.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.426+0000 7fd8beffd700 1 -- 172.21.15.187:0/1097288121 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd8b0010030 con 0x7fd8c809dc80 2020-11-20T12:06:37.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.427+0000 7fd8beffd700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8a4056d20 0x7fd8a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.427+0000 7fd8beffd700 1 -- 172.21.15.187:0/1097288121 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd8b006fe70 con 0x7fd8c809dc80 2020-11-20T12:06:37.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.427+0000 7fd8c659c700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8a4056d20 0x7fd8a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.428+0000 7fd8c659c700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8a4056d20 0x7fd8a40591e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd8b8006e30 tx=0x7fd8b8019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.431+0000 7fd8beffd700 1 -- 172.21.15.187:0/1097288121 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd8b0045330 con 0x7fd8c809dc80 2020-11-20T12:06:37.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.675+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8c809ad90 con 0x7fd8c809dc80 2020-11-20T12:06:37.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.676+0000 7fd8beffd700 1 -- 172.21.15.187:0/1097288121 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7fd8b0001310 con 0x7fd8c809dc80 2020-11-20T12:06:37.680 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:37.680 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:37.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8a4056d20 msgr2=0x7fd8a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8a4056d20 0x7fd8a40591e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd8b8006e30 tx=0x7fd8b8019000).stop 2020-11-20T12:06:37.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c809dc80 msgr2=0x7fd8c815b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c809dc80 0x7fd8c815b570 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fd8b000ca60 tx=0x7fd8b0002c10).stop 2020-11-20T12:06:37.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 shutdown_connections 2020-11-20T12:06:37.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8a4056d20 0x7fd8a40591e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8c815ba60 0x7fd8c8099360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8a8003730 0x7fd8c814cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 --2- 172.21.15.187:0/1097288121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8c809dc80 0x7fd8c815b570 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 >> 172.21.15.187:0/1097288121 conn(0x7fd8c80bacd0 msgr2=0x7fd8c80b8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:37.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 shutdown_connections 2020-11-20T12:06:37.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:37.679+0000 7fd8ccedf700 1 -- 172.21.15.187:0/1097288121 wait complete. 2020-11-20T12:06:37.685 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:37.705 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:06:37.978 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":1001}} 2020-11-20T12:06:37.979 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2020-11-20T12:06:37.979 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:38.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.221+0000 7f7fe8a1f700 1 Processor -- start 2020-11-20T12:06:38.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.221+0000 7f7fe8a1f700 1 -- start start 2020-11-20T12:06:38.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.222+0000 7f7fe8a1f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7fe41484d0 0x7f7fe4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.222+0000 7f7fe8a1f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe40622c0 0x7f7fe40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.222+0000 7f7fe8a1f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe40b1300 con 0x7f7fe4066f70 2020-11-20T12:06:38.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.222+0000 7f7fe8a1f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe4067730 con 0x7f7fe41484d0 2020-11-20T12:06:38.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.222+0000 7f7fe8a1f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe40b15a0 con 0x7f7fe40622c0 2020-11-20T12:06:38.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.223+0000 7f7fe2d9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7fe41484d0 0x7f7fe4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.223+0000 7f7fe259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe40622c0 0x7f7fe40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.223+0000 7f7fe359e700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7fe4066f70 0x7f7fe40b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:60758/0 (socket says 172.21.15.187:60758) 2020-11-20T12:06:38.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.223+0000 7f7fe359e700 1 -- 172.21.15.187:0/2482873872 learned_addr learned my addr 172.21.15.187:0/2482873872 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.224+0000 7f7fe2d9d700 1 -- 172.21.15.187:0/2482873872 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe40622c0 msgr2=0x7f7fe40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.224+0000 7f7fe2d9d700 1 --2- 172.21.15.187:0/2482873872 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe40622c0 0x7f7fe40b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.224+0000 7f7fe2d9d700 1 -- 172.21.15.187:0/2482873872 >> v1:172.21.15.5:6789/0 conn(0x7f7fe4066f70 legacy=0x7f7fe40b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.224+0000 7f7fe2d9d700 1 -- 172.21.15.187:0/2482873872 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe40b1840 con 0x7f7fe41484d0 2020-11-20T12:06:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.225+0000 7f7fe2d9d700 1 --2- 172.21.15.187:0/2482873872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 0x7f7fe4061dd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7fd400b9a0 tx=0x7f7fd4006df0).ready entity=mon.0 client_cookie=3d3623f4660d4443 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:38.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.225+0000 7f7fe1d9b700 1 -- 172.21.15.187:0/2482873872 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40097b0 con 0x7f7fe41484d0 2020-11-20T12:06:38.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe1d9b700 1 -- 172.21.15.187:0/2482873872 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd4009910 con 0x7f7fe41484d0 2020-11-20T12:06:38.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe1d9b700 1 -- 172.21.15.187:0/2482873872 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4009c90 con 0x7f7fe41484d0 2020-11-20T12:06:38.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/2482873872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 msgr2=0x7f7fe4061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/2482873872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 0x7f7fe4061dd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7fd400b9a0 tx=0x7f7fd4006df0).stop 2020-11-20T12:06:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/2482873872 shutdown_connections 2020-11-20T12:06:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/2482873872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 0x7f7fe4061dd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/2482873872 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe40622c0 0x7f7fe40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/2482873872 >> 172.21.15.187:0/2482873872 conn(0x7f7fe40bab30 msgr2=0x7f7fe40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:38.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/2482873872 shutdown_connections 2020-11-20T12:06:38.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.226+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/2482873872 wait complete. 2020-11-20T12:06:38.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.228+0000 7f7fe8a1f700 1 Processor -- start 2020-11-20T12:06:38.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.228+0000 7f7fe8a1f700 1 -- start start 2020-11-20T12:06:38.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.229+0000 7f7fe8a1f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fe40622c0 0x7f7fe40a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.229+0000 7f7fe8a1f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe4066f70 0x7f7fe409f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.229+0000 7f7fe8a1f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 0x7f7fe41421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.229+0000 7f7fe8a1f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe409ce30 con 0x7f7fe41484d0 2020-11-20T12:06:38.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.229+0000 7f7fe8a1f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe409d0d0 con 0x7f7fe40622c0 2020-11-20T12:06:38.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.229+0000 7f7fe8a1f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe409d370 con 0x7f7fe4066f70 2020-11-20T12:06:38.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe2d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fe40622c0 0x7f7fe40a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe2d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fe40622c0 0x7f7fe40a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42412/0 (socket says 172.21.15.187:42412) 2020-11-20T12:06:38.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe2d9d700 1 -- 172.21.15.187:0/3721487026 learned_addr learned my addr 172.21.15.187:0/3721487026 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:38.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe259c700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe4066f70 0x7f7fe409f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe2d9d700 1 -- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe4066f70 msgr2=0x7f7fe409f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe359e700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 0x7f7fe41421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe2d9d700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe4066f70 0x7f7fe409f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe2d9d700 1 -- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 msgr2=0x7f7fe41421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe2d9d700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 0x7f7fe41421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe2d9d700 1 -- 172.21.15.187:0/3721487026 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe409a670 con 0x7f7fe40622c0 2020-11-20T12:06:38.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe259c700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe4066f70 0x7f7fe409f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:38.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.230+0000 7f7fe359e700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 0x7f7fe41421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:38.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.231+0000 7f7fe2d9d700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fe40622c0 0x7f7fe40a0320 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f7fe4066430 tx=0x7f7fd4017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:38.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.231+0000 7f7fcaffd700 1 -- 172.21.15.187:0/3721487026 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4020070 con 0x7f7fe40622c0 2020-11-20T12:06:38.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.231+0000 7f7fcaffd700 1 -- 172.21.15.187:0/3721487026 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd4010e40 con 0x7f7fe40622c0 2020-11-20T12:06:38.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.231+0000 7f7fcaffd700 1 -- 172.21.15.187:0/3721487026 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd401a690 con 0x7f7fe40622c0 2020-11-20T12:06:38.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.231+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe40a0810 con 0x7f7fe40622c0 2020-11-20T12:06:38.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.231+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe409c990 con 0x7f7fe40622c0 2020-11-20T12:06:38.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.232+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe409cc70 con 0x7f7fe40622c0 2020-11-20T12:06:38.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.232+0000 7f7fcaffd700 1 -- 172.21.15.187:0/3721487026 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7fd402c070 con 0x7f7fe40622c0 2020-11-20T12:06:38.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.233+0000 7f7fcaffd700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc0056d20 0x7f7fc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.233+0000 7f7fcaffd700 1 -- 172.21.15.187:0/3721487026 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7fd4073a50 con 0x7f7fe40622c0 2020-11-20T12:06:38.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.233+0000 7f7fe259c700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc0056d20 0x7f7fc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.234+0000 7f7fe259c700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc0056d20 0x7f7fc00591e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f7fd8006bd0 tx=0x7f7fd8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:38.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.238+0000 7f7fcaffd700 1 -- 172.21.15.187:0/3721487026 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7fd4048d90 con 0x7f7fe40622c0 2020-11-20T12:06:38.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.480+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fe409d530 con 0x7f7fe40622c0 2020-11-20T12:06:38.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.481+0000 7f7fcaffd700 1 -- 172.21.15.187:0/3721487026 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7f7fd404c3b0 con 0x7f7fe40622c0 2020-11-20T12:06:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:38.485 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:38.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.483+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc0056d20 msgr2=0x7f7fc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.483+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc0056d20 0x7f7fc00591e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f7fd8006bd0 tx=0x7f7fd8008040).stop 2020-11-20T12:06:38.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.483+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fe40622c0 msgr2=0x7f7fe40a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.483+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fe40622c0 0x7f7fe40a0320 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f7fe4066430 tx=0x7f7fd4017000).stop 2020-11-20T12:06:38.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.484+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 shutdown_connections 2020-11-20T12:06:38.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.484+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc0056d20 0x7f7fc00591e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.484+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fe41484d0 0x7f7fe41421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.484+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fe40622c0 0x7f7fe40a0320 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.484+0000 7f7fe8a1f700 1 --2- 172.21.15.187:0/3721487026 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fe4066f70 0x7f7fe409f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.484+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 >> 172.21.15.187:0/3721487026 conn(0x7f7fe40bab30 msgr2=0x7f7fe4142d10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:38.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.484+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 shutdown_connections 2020-11-20T12:06:38.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.484+0000 7f7fe8a1f700 1 -- 172.21.15.187:0/3721487026 wait complete. 2020-11-20T12:06:38.490 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:38.510 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2020-11-20T12:06:38.776 INFO:teuthology.orchestra.run.smithi187.stdout:{"purge_queue":{"pq_executed":1001}} 2020-11-20T12:06:38.777 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2020-11-20T12:06:38.778 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_metadata stat 10000000000.00000000 2020-11-20T12:06:38.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.848+0000 7f94f5c44d00 1 Processor -- start 2020-11-20T12:06:38.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.848+0000 7f94f5c44d00 1 -- start start 2020-11-20T12:06:38.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94f5c44d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a163a0 0x55a739a5aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94f5c44d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a3dd40 0x55a739a59a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94f5c44d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a739a57ad0 0x55a739add6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94f5c44d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7399c2e30 con 0x55a739a3dd40 2020-11-20T12:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94f5c44d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a739991810 con 0x55a739a163a0 2020-11-20T12:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94f5c44d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a739a5a9e0 con 0x55a739a57ad0 2020-11-20T12:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94e64e9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a163a0 0x55a739a5aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94e64e9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a163a0 0x55a739a5aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42426/0 (socket says 172.21.15.187:42426) 2020-11-20T12:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.849+0000 7f94e64e9700 1 -- 172.21.15.187:0/55188227 learned_addr learned my addr 172.21.15.187:0/55188227 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e64e9700 1 -- 172.21.15.187:0/55188227 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a739a57ad0 msgr2=0x55a739add6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:38.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e64e9700 1 --2- 172.21.15.187:0/55188227 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a739a57ad0 0x55a739add6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:38.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e64e9700 1 -- 172.21.15.187:0/55188227 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a3dd40 msgr2=0x55a739a59a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:38.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e64e9700 1 --2- 172.21.15.187:0/55188227 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a3dd40 0x55a739a59a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:38.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e64e9700 1 -- 172.21.15.187:0/55188227 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7399b6ec0 con 0x55a739a163a0 2020-11-20T12:06:38.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e64e9700 1 --2- 172.21.15.187:0/55188227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a163a0 0x55a739a5aba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f94d8008490 tx=0x7f94d8015000).ready entity=mon.1 client_cookie=b4c046f123e49b42 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:38.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e4ce6700 1 -- 172.21.15.187:0/55188227 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d8020070 con 0x55a739a163a0 2020-11-20T12:06:38.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e4ce6700 1 -- 172.21.15.187:0/55188227 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d8008dc0 con 0x55a739a163a0 2020-11-20T12:06:38.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.850+0000 7f94e4ce6700 1 -- 172.21.15.187:0/55188227 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d8012810 con 0x55a739a163a0 2020-11-20T12:06:38.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.851+0000 7f94f5c44d00 1 -- 172.21.15.187:0/55188227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a163a0 msgr2=0x55a739a5aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.851+0000 7f94f5c44d00 1 --2- 172.21.15.187:0/55188227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a163a0 0x55a739a5aba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f94d8008490 tx=0x7f94d8015000).stop 2020-11-20T12:06:38.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.851+0000 7f94f5c44d00 1 -- 172.21.15.187:0/55188227 shutdown_connections 2020-11-20T12:06:38.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.851+0000 7f94f5c44d00 1 --2- 172.21.15.187:0/55188227 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a739a57ad0 0x55a739add6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:38.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.851+0000 7f94f5c44d00 1 --2- 172.21.15.187:0/55188227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a163a0 0x55a739a5aba0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.851+0000 7f94f5c44d00 1 --2- 172.21.15.187:0/55188227 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a3dd40 0x55a739a59a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:38.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.851+0000 7f94f5c44d00 1 -- 172.21.15.187:0/55188227 >> 172.21.15.187:0/55188227 conn(0x55a739a150e0 msgr2=0x55a739ae52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:38.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.852+0000 7f94f5c44d00 1 -- 172.21.15.187:0/55188227 shutdown_connections 2020-11-20T12:06:38.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.852+0000 7f94f5c44d00 1 -- 172.21.15.187:0/55188227 wait complete. 2020-11-20T12:06:38.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.853+0000 7f94f5c44d00 1 Processor -- start 2020-11-20T12:06:38.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.853+0000 7f94f5c44d00 1 -- start start 2020-11-20T12:06:38.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.854+0000 7f94f5c44d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a163a0 0x55a739a1bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.854+0000 7f94f5c44d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a3dd40 0x55a739a1c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.854+0000 7f94f5c44d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a739a57ad0 0x55a739af8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.854+0000 7f94f5c44d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a739afa330 con 0x55a739a163a0 2020-11-20T12:06:38.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94f5c44d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a739afa5c0 con 0x55a739a3dd40 2020-11-20T12:06:38.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94f5c44d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a739afa860 con 0x55a739a57ad0 2020-11-20T12:06:38.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e5ce8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a3dd40 0x55a739a1c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e5ce8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a3dd40 0x55a739a1c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42436/0 (socket says 172.21.15.187:42436) 2020-11-20T12:06:38.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e5ce8700 1 -- 172.21.15.187:0/3194713528 learned_addr learned my addr 172.21.15.187:0/3194713528 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:38.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e64e9700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a163a0 0x55a739a1bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e6cea700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a739a57ad0 0x55a739af8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e5ce8700 1 -- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a739a57ad0 msgr2=0x55a739af8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e5ce8700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a739a57ad0 0x55a739af8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e5ce8700 1 -- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a163a0 msgr2=0x55a739a1bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e5ce8700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a163a0 0x55a739a1bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e5ce8700 1 -- 172.21.15.187:0/3194713528 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a739995250 con 0x55a739a3dd40 2020-11-20T12:06:38.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.855+0000 7f94e64e9700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a739a163a0 0x55a739a1bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:38.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.856+0000 7f94e5ce8700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a3dd40 0x55a739a1c0b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x55a739ae2620 tx=0x7f94dc001700).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:38.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.856+0000 7f94d6ffd700 1 -- 172.21.15.187:0/3194713528 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc009ea0 con 0x55a739a3dd40 2020-11-20T12:06:38.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.856+0000 7f94d6ffd700 1 -- 172.21.15.187:0/3194713528 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc011420 con 0x55a739a3dd40 2020-11-20T12:06:38.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.856+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a739ae45b0 con 0x55a739a3dd40 2020-11-20T12:06:38.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.856+0000 7f94d6ffd700 1 -- 172.21.15.187:0/3194713528 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0117c0 con 0x55a739a3dd40 2020-11-20T12:06:38.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.856+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a739ae4210 con 0x55a739a3dd40 2020-11-20T12:06:38.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.858+0000 7f94d6ffd700 1 -- 172.21.15.187:0/3194713528 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f94dc011b70 con 0x55a739a3dd40 2020-11-20T12:06:38.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.858+0000 7f94d6ffd700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f94c80559a0 0x7f94c8057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.859+0000 7f94d6ffd700 1 -- 172.21.15.187:0/3194713528 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f94dc06e330 con 0x55a739a3dd40 2020-11-20T12:06:38.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.859+0000 7f94e64e9700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f94c80559a0 0x7f94c8057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.859+0000 7f94f5c44d00 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a739adef60 0x55a739afb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:38.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.859+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:1 5.0 5:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected e32) v8 -- 0x55a739afb700 con 0x55a739adef60 2020-11-20T12:06:38.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.859+0000 7f94e64e9700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f94c80559a0 0x7f94c8057e60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x55a739aed0c0 tx=0x7f94d801b000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:38.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.860+0000 7f94e6cea700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a739adef60 0x55a739afb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:38.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.861+0000 7f94e6cea700 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a739adef60 0x55a739afb210 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:38.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94e6cea700 1 -- 172.21.15.187:0/3194713528 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f94e00017f0 con 0x55a739adef60 2020-11-20T12:06:38.918 INFO:teuthology.orchestra.run.smithi187.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2020-11-20T12:06:38.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a739adef60 msgr2=0x55a739afb210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a739adef60 0x55a739afb210 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:38.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f94c80559a0 msgr2=0x7f94c8057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f94c80559a0 0x7f94c8057e60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x55a739aed0c0 tx=0x7f94d801b000).stop 2020-11-20T12:06:38.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a3dd40 msgr2=0x55a739a1c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:38.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 --2- 172.21.15.187:0/3194713528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a739a3dd40 0x55a739a1c0b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x55a739ae2620 tx=0x7f94dc001700).stop 2020-11-20T12:06:38.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94e6cea700 1 -- 172.21.15.187:0/3194713528 reap_dead start 2020-11-20T12:06:38.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 shutdown_connections 2020-11-20T12:06:38.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 >> 172.21.15.187:0/3194713528 conn(0x55a739a150e0 msgr2=0x55a739a59a60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:38.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 shutdown_connections 2020-11-20T12:06:38.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:38.862+0000 7f94f5c44d00 1 -- 172.21.15.187:0/3194713528 wait complete. 2020-11-20T12:06:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:06:38.923 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2020-11-20T12:06:39.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.169+0000 7f157d7d0700 1 Processor -- start 2020-11-20T12:06:39.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.169+0000 7f157d7d0700 1 -- start start 2020-11-20T12:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.170+0000 7f157d7d0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 0x7f15780b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.170+0000 7f157d7d0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1578066040 0x7f1578065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.170+0000 7f157d7d0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15781502d0 con 0x7f1578065680 2020-11-20T12:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.170+0000 7f157d7d0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1578150500 con 0x7f1578066040 2020-11-20T12:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.170+0000 7f157d7d0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15781507a0 con 0x7f157814a110 2020-11-20T12:06:39.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.171+0000 7f15777fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1578066040 0x7f1578065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:39.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.171+0000 7f157cfcf700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1578065680 0x7f157814cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39052/0 (socket says 172.21.15.187:39052) 2020-11-20T12:06:39.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.171+0000 7f157cfcf700 1 -- 172.21.15.187:0/4276940959 learned_addr learned my addr 172.21.15.187:0/4276940959 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:39.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.171+0000 7f1577fff700 1 --2- 172.21.15.187:0/4276940959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 0x7f15780b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:39.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f15777fe700 1 -- 172.21.15.187:0/4276940959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 msgr2=0x7f15780b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:39.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f15777fe700 1 --2- 172.21.15.187:0/4276940959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 0x7f15780b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:39.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f15777fe700 1 -- 172.21.15.187:0/4276940959 >> v1:172.21.15.187:6789/0 conn(0x7f1578065680 legacy=0x7f157814cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:39.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f15777fe700 1 -- 172.21.15.187:0/4276940959 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15780b1960 con 0x7f1578066040 2020-11-20T12:06:39.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f1576ffd700 1 -- 172.21.15.187:0/4276940959 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952971970 0 0) 0x7f15781502d0 con 0x7f1578065680 2020-11-20T12:06:39.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f1577fff700 1 --2- 172.21.15.187:0/4276940959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 0x7f15780b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:39.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f15777fe700 1 --2- 172.21.15.187:0/4276940959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578065190 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f156c00b9a0 tx=0x7f156c006df0).ready entity=mon.1 client_cookie=61aef95ac62f101a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:39.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f1576ffd700 1 -- 172.21.15.187:0/4276940959 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c0097b0 con 0x7f1578066040 2020-11-20T12:06:39.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f1576ffd700 1 -- 172.21.15.187:0/4276940959 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c009910 con 0x7f1578066040 2020-11-20T12:06:39.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.172+0000 7f1576ffd700 1 -- 172.21.15.187:0/4276940959 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c009c90 con 0x7f1578066040 2020-11-20T12:06:39.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.173+0000 7f157d7d0700 1 -- 172.21.15.187:0/4276940959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 msgr2=0x7f1578065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:39.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.173+0000 7f157d7d0700 1 --2- 172.21.15.187:0/4276940959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578065190 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f156c00b9a0 tx=0x7f156c006df0).stop 2020-11-20T12:06:39.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.173+0000 7f157d7d0700 1 -- 172.21.15.187:0/4276940959 shutdown_connections 2020-11-20T12:06:39.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.173+0000 7f157d7d0700 1 --2- 172.21.15.187:0/4276940959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 0x7f15780b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:39.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.173+0000 7f157d7d0700 1 --2- 172.21.15.187:0/4276940959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578065190 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:39.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.173+0000 7f157d7d0700 1 -- 172.21.15.187:0/4276940959 >> 172.21.15.187:0/4276940959 conn(0x7f15780bab10 msgr2=0x7f15780baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:39.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.173+0000 7f157d7d0700 1 -- 172.21.15.187:0/4276940959 shutdown_connections 2020-11-20T12:06:39.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.173+0000 7f157d7d0700 1 -- 172.21.15.187:0/4276940959 wait complete. 2020-11-20T12:06:39.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.174+0000 7f157d7d0700 1 Processor -- start 2020-11-20T12:06:39.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.175+0000 7f157d7d0700 1 -- start start 2020-11-20T12:06:39.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.175+0000 7f157d7d0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1578065680 0x7f157814b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:39.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.175+0000 7f157d7d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:39.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f157d7d0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 0x7f15780a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:39.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f157d7d0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157814fa80 con 0x7f1578065680 2020-11-20T12:06:39.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f157d7d0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157814f310 con 0x7f1578066040 2020-11-20T12:06:39.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f157d7d0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157814b8e0 con 0x7f157814a110 2020-11-20T12:06:39.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f15777fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:39.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f15777fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578148cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42450/0 (socket says 172.21.15.187:42450) 2020-11-20T12:06:39.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f15777fe700 1 -- 172.21.15.187:0/144983992 learned_addr learned my addr 172.21.15.187:0/144983992 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:39.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f15777fe700 1 -- 172.21.15.187:0/144983992 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 msgr2=0x7f15780a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:39.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f15777fe700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 0x7f15780a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:39.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f15777fe700 1 -- 172.21.15.187:0/144983992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1578065680 msgr2=0x7f157814b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:39.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f1577fff700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1578065680 0x7f157814b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:39.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f15777fe700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1578065680 0x7f157814b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:39.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.176+0000 7f15777fe700 1 -- 172.21.15.187:0/144983992 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157814f540 con 0x7f1578066040 2020-11-20T12:06:39.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.177+0000 7f15777fe700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578148cb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f156c010000 tx=0x7f156c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:39.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.177+0000 7f1557fff700 1 -- 172.21.15.187:0/144983992 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c011e50 con 0x7f1578066040 2020-11-20T12:06:39.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.177+0000 7f1557fff700 1 -- 172.21.15.187:0/144983992 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c01e460 con 0x7f1578066040 2020-11-20T12:06:39.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.177+0000 7f1557fff700 1 -- 172.21.15.187:0/144983992 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c01e7e0 con 0x7f1578066040 2020-11-20T12:06:39.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.177+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157814f7e0 con 0x7f1578066040 2020-11-20T12:06:39.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.177+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157809dae0 con 0x7f1578066040 2020-11-20T12:06:39.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.178+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1578061a20 con 0x7f1578066040 2020-11-20T12:06:39.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.179+0000 7f1557fff700 1 -- 172.21.15.187:0/144983992 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f156c025030 con 0x7f1578066040 2020-11-20T12:06:39.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.179+0000 7f1557fff700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1550056d20 0x7f15500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:39.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.179+0000 7f1557fff700 1 -- 172.21.15.187:0/144983992 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f156c071b10 con 0x7f1578066040 2020-11-20T12:06:39.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.180+0000 7f1577fff700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1550056d20 0x7f15500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:39.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.180+0000 7f1577fff700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1550056d20 0x7f15500591e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f156800b6f0 tx=0x7f1568008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:39.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.184+0000 7f1557fff700 1 -- 172.21.15.187:0/144983992 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f156c046f10 con 0x7f1578066040 2020-11-20T12:06:39.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.419+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f157805cae0 con 0x7f1578066040 2020-11-20T12:06:39.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.420+0000 7f1557fff700 1 -- 172.21.15.187:0/144983992 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1972 (secure 0 0 0) 0x7f156c023070 con 0x7f1578066040 2020-11-20T12:06:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:39.424 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:06:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:06:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 773016186880, 2020-11-20T12:06:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 77926400, 2020-11-20T12:06:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 77926400, 2020-11-20T12:06:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 0.00010079807543661445, 2020-11-20T12:06:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "num_osds": 8, 2020-11-20T12:06:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_osds": 8, 2020-11-20T12:06:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_omap_osds": 8 2020-11-20T12:06:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "stats_by_class": { 2020-11-20T12:06:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "ssd": { 2020-11-20T12:06:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:06:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 773016186880, 2020-11-20T12:06:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 77926400, 2020-11-20T12:06:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 77926400, 2020-11-20T12:06:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 0.00010079807543661445 2020-11-20T12:06:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "pools": [ 2020-11-20T12:06:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "device_health_metrics", 2020-11-20T12:06:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 1, 2020-11-20T12:06:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:06:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 2, 2020-11-20T12:06:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:06:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:06:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:06:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367173533696 2020-11-20T12:06:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "rbd", 2020-11-20T12:06:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 2, 2020-11-20T12:06:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:06:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:06:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:06:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:06:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:06:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367173533696 2020-11-20T12:06:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_metadata", 2020-11-20T12:06:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 5, 2020-11-20T12:06:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 4087419, 2020-11-20T12:06:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 88, 2020-11-20T12:06:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 8048, 2020-11-20T12:06:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 8241152, 2020-11-20T12:06:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 1.122229514294304e-05, 2020-11-20T12:06:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367173533696 2020-11-20T12:06:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_data", 2020-11-20T12:06:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 6, 2020-11-20T12:06:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 56320, 2020-11-20T12:06:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 55, 2020-11-20T12:06:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 440, 2020-11-20T12:06:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 450560, 2020-11-20T12:06:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 6.135514922789298e-07, 2020-11-20T12:06:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367173533696 2020-11-20T12:06:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:06:39.439 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:06:39.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.422+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1550056d20 msgr2=0x7f15500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:39.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.422+0000 7f157d7d0700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1550056d20 0x7f15500591e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f156800b6f0 tx=0x7f1568008040).stop 2020-11-20T12:06:39.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.422+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 msgr2=0x7f1578148cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:39.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.422+0000 7f157d7d0700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578148cb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f156c010000 tx=0x7f156c015000).stop 2020-11-20T12:06:39.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.423+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 shutdown_connections 2020-11-20T12:06:39.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.423+0000 7f157d7d0700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1550056d20 0x7f15500591e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:39.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.423+0000 7f157d7d0700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1578065680 0x7f157814b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:39.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.423+0000 7f157d7d0700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f157814a110 0x7f15780a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:39.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.423+0000 7f157d7d0700 1 --2- 172.21.15.187:0/144983992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1578066040 0x7f1578148cb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:39.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.423+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 >> 172.21.15.187:0/144983992 conn(0x7f15780bab10 msgr2=0x7f15780b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:39.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.423+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 shutdown_connections 2020-11-20T12:06:39.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:39.423+0000 7f157d7d0700 1 -- 172.21.15.187:0/144983992 wait complete. 2020-11-20T12:06:39.446 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60)... 2020-11-20T12:06:44.447 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2020-11-20T12:06:44.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.695+0000 7f632359e700 1 Processor -- start 2020-11-20T12:06:44.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.695+0000 7f632359e700 1 -- start start 2020-11-20T12:06:44.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.696+0000 7f632359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6324065350 0x7f6324061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.697+0000 7f632359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065730 0x7f63240622c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.697+0000 7f632359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632414c110 con 0x7f63240b1790 2020-11-20T12:06:44.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.697+0000 7f632359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632414c3b0 con 0x7f6324065350 2020-11-20T12:06:44.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.697+0000 7f632359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632414c650 con 0x7f6324065730 2020-11-20T12:06:44.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.697+0000 7f632259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6324065350 0x7f6324061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.697+0000 7f632259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6324065350 0x7f6324061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42456/0 (socket says 172.21.15.187:42456) 2020-11-20T12:06:44.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.697+0000 7f632259c700 1 -- 172.21.15.187:0/3642172060 learned_addr learned my addr 172.21.15.187:0/3642172060 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:44.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.697+0000 7f6319d9b700 1 --2- 172.21.15.187:0/3642172060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065730 0x7f63240622c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.698+0000 7f632259c700 1 -- 172.21.15.187:0/3642172060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065730 msgr2=0x7f63240622c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:44.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.698+0000 7f632259c700 1 --2- 172.21.15.187:0/3642172060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065730 0x7f63240622c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.698+0000 7f632259c700 1 -- 172.21.15.187:0/3642172060 >> v1:172.21.15.187:6789/0 conn(0x7f63240b1790 legacy=0x7f6324147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.698+0000 7f632259c700 1 -- 172.21.15.187:0/3642172060 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632414be00 con 0x7f6324065350 2020-11-20T12:06:44.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.698+0000 7f632259c700 1 --2- 172.21.15.187:0/3642172060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065350 0x7f6324061dd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f6314006fd0 tx=0x7f6314006e50).ready entity=mon.1 client_cookie=1fd08d7ceab03f6b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:44.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f6321d9b700 1 -- 172.21.15.187:0/3642172060 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314009900 con 0x7f6324065350 2020-11-20T12:06:44.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f6321d9b700 1 -- 172.21.15.187:0/3642172060 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314009a60 con 0x7f6324065350 2020-11-20T12:06:44.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f6321d9b700 1 -- 172.21.15.187:0/3642172060 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314009de0 con 0x7f6324065350 2020-11-20T12:06:44.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f632359e700 1 -- 172.21.15.187:0/3642172060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065350 msgr2=0x7f6324061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f632359e700 1 --2- 172.21.15.187:0/3642172060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065350 0x7f6324061dd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f6314006fd0 tx=0x7f6314006e50).stop 2020-11-20T12:06:44.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f632359e700 1 -- 172.21.15.187:0/3642172060 shutdown_connections 2020-11-20T12:06:44.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f632359e700 1 --2- 172.21.15.187:0/3642172060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065350 0x7f6324061dd0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f632359e700 1 --2- 172.21.15.187:0/3642172060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065730 0x7f63240622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f632359e700 1 -- 172.21.15.187:0/3642172060 >> 172.21.15.187:0/3642172060 conn(0x7f63240bab10 msgr2=0x7f63240baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:44.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f632359e700 1 -- 172.21.15.187:0/3642172060 shutdown_connections 2020-11-20T12:06:44.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.699+0000 7f632359e700 1 -- 172.21.15.187:0/3642172060 wait complete. 2020-11-20T12:06:44.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.701+0000 7f632359e700 1 Processor -- start 2020-11-20T12:06:44.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.701+0000 7f632359e700 1 -- start start 2020-11-20T12:06:44.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.702+0000 7f632359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065350 0x7f63241603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.702+0000 7f632359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065730 0x7f632415f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.702+0000 7f632359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63240b1790 0x7f63240b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.702+0000 7f632359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63241608b0 con 0x7f63240b1790 2020-11-20T12:06:44.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.702+0000 7f632359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6324149900 con 0x7f6324065730 2020-11-20T12:06:44.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.702+0000 7f632359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6324149ba0 con 0x7f6324065350 2020-11-20T12:06:44.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6319d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065730 0x7f632415f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6319d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065730 0x7f632415f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42462/0 (socket says 172.21.15.187:42462) 2020-11-20T12:06:44.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6319d9b700 1 -- 172.21.15.187:0/3124417961 learned_addr learned my addr 172.21.15.187:0/3124417961 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:44.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6319d9b700 1 -- 172.21.15.187:0/3124417961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065350 msgr2=0x7f63241603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6319d9b700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065350 0x7f63241603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:44.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6322d9d700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63240b1790 0x7f63240b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6319d9b700 1 -- 172.21.15.187:0/3124417961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63240b1790 msgr2=0x7f63240b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6319d9b700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63240b1790 0x7f63240b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.703+0000 7f6319d9b700 1 -- 172.21.15.187:0/3124417961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324149e40 con 0x7f6324065730 2020-11-20T12:06:44.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.704+0000 7f6322d9d700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63240b1790 0x7f63240b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:44.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.704+0000 7f6319d9b700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065730 0x7f632415f960 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f630c00d900 tx=0x7f630c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:44.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.704+0000 7f631affd700 1 -- 172.21.15.187:0/3124417961 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c009790 con 0x7f6324065730 2020-11-20T12:06:44.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.704+0000 7f631affd700 1 -- 172.21.15.187:0/3124417961 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c0098f0 con 0x7f6324065730 2020-11-20T12:06:44.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.704+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6324149620 con 0x7f6324065730 2020-11-20T12:06:44.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.704+0000 7f631affd700 1 -- 172.21.15.187:0/3124417961 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c009c90 con 0x7f6324065730 2020-11-20T12:06:44.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.704+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63240b5980 con 0x7f6324065730 2020-11-20T12:06:44.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.705+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6324149fd0 con 0x7f6324065730 2020-11-20T12:06:44.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.706+0000 7f631affd700 1 -- 172.21.15.187:0/3124417961 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f630c013480 con 0x7f6324065730 2020-11-20T12:06:44.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.706+0000 7f631affd700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6300056d20 0x7f63000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.707+0000 7f631affd700 1 -- 172.21.15.187:0/3124417961 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f630c06ff70 con 0x7f6324065730 2020-11-20T12:06:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.707+0000 7f632259c700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6300056d20 0x7f63000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.708+0000 7f632259c700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6300056d20 0x7f63000591e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f6314006e90 tx=0x7f6314015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.711+0000 7f631affd700 1 -- 172.21.15.187:0/3124417961 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f630c045360 con 0x7f6324065730 2020-11-20T12:06:44.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.963+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f6324061a20 con 0x7f6324065730 2020-11-20T12:06:44.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.964+0000 7f631affd700 1 -- 172.21.15.187:0/3124417961 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1941 (secure 0 0 0) 0x7f630c048980 con 0x7f6324065730 2020-11-20T12:06:44.967 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:44.968 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:06:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:06:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 773020364800, 2020-11-20T12:06:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 73748480, 2020-11-20T12:06:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 73748480, 2020-11-20T12:06:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 9.5393923402298242e-05, 2020-11-20T12:06:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: "num_osds": 8, 2020-11-20T12:06:44.970 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_osds": 8, 2020-11-20T12:06:44.970 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_omap_osds": 8 2020-11-20T12:06:44.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:44.970 INFO:teuthology.orchestra.run.smithi187.stdout: "stats_by_class": { 2020-11-20T12:06:44.971 INFO:teuthology.orchestra.run.smithi187.stdout: "ssd": { 2020-11-20T12:06:44.971 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:06:44.971 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 773020364800, 2020-11-20T12:06:44.971 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 73748480, 2020-11-20T12:06:44.971 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 73748480, 2020-11-20T12:06:44.972 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 9.5393923402298242e-05 2020-11-20T12:06:44.972 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:44.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:44.973 INFO:teuthology.orchestra.run.smithi187.stdout: "pools": [ 2020-11-20T12:06:44.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:44.973 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "device_health_metrics", 2020-11-20T12:06:44.973 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 1, 2020-11-20T12:06:44.974 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:44.974 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:06:44.975 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 2, 2020-11-20T12:06:44.975 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:06:44.975 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:06:44.975 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:06:44.975 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367173763072 2020-11-20T12:06:44.976 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:44.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:44.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:44.976 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "rbd", 2020-11-20T12:06:44.977 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 2, 2020-11-20T12:06:44.977 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:44.977 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:06:44.977 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:06:44.977 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:06:44.978 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:06:44.978 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:06:44.978 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367173763072 2020-11-20T12:06:44.978 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:44.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:44.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:44.979 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_metadata", 2020-11-20T12:06:44.979 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 5, 2020-11-20T12:06:44.980 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:44.980 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 4126627, 2020-11-20T12:06:44.980 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 23, 2020-11-20T12:06:44.980 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 8120, 2020-11-20T12:06:44.981 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 8314880, 2020-11-20T12:06:44.981 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 1.1322685168124735e-05, 2020-11-20T12:06:44.981 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367173763072 2020-11-20T12:06:44.981 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:44.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:44.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:44.982 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_data", 2020-11-20T12:06:44.982 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 6, 2020-11-20T12:06:44.982 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:06:44.983 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:06:44.984 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:06:44.984 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:06:44.985 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:06:44.985 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:06:44.985 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367173763072 2020-11-20T12:06:44.985 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:44.986 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:44.986 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:06:44.986 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:06:44.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6300056d20 msgr2=0x7f63000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6300056d20 0x7f63000591e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f6314006e90 tx=0x7f6314015000).stop 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065730 msgr2=0x7f632415f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065730 0x7f632415f960 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f630c00d900 tx=0x7f630c008040).stop 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 shutdown_connections 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6300056d20 0x7f63000591e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63240b1790 0x7f63240b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6324065350 0x7f63241603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:44.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 --2- 172.21.15.187:0/3124417961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6324065730 0x7f632415f960 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 >> 172.21.15.187:0/3124417961 conn(0x7f63240bab10 msgr2=0x7f632409cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:44.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 shutdown_connections 2020-11-20T12:06:44.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:44.966+0000 7f632359e700 1 -- 172.21.15.187:0/3124417961 wait complete. 2020-11-20T12:06:44.993 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2020-11-20T12:06:44.993 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:06:44.994 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:06:45.112 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:06:45.130 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:06:45.235 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:06:45.254 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:06:45.255 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:06:45.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:06:45.255 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:06:45.380 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:06:45.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:06:45.381 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:06:45.435 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:06:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.670+0000 7f7f03fff700 1 Processor -- start 2020-11-20T12:06:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.671+0000 7f7f03fff700 1 -- start start 2020-11-20T12:06:45.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.672+0000 7f7f03fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7f04065190 0x7f7f04065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:45.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.672+0000 7f7f03fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f0409ca90 0x7f7f0409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.672+0000 7f7f03fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f040b13e0 con 0x7f7f04065f90 2020-11-20T12:06:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.672+0000 7f7f03fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0414ab30 con 0x7f7f04065190 2020-11-20T12:06:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.672+0000 7f7f03fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f04062c10 con 0x7f7f0409ca90 2020-11-20T12:06:45.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.673+0000 7f7f037fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f0409ca90 0x7f7f0409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:45.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.673+0000 7f7f037fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f0409ca90 0x7f7f0409cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42474/0 (socket says 172.21.15.187:42474) 2020-11-20T12:06:45.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.673+0000 7f7f037fe700 1 -- 172.21.15.187:0/4088628479 learned_addr learned my addr 172.21.15.187:0/4088628479 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:45.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.673+0000 7f7f02ffd700 1 --2- 172.21.15.187:0/4088628479 >> v2:172.21.15.5:3300/0 conn(0x7f7f04065190 0x7f7f04065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.673+0000 7f7f037fe700 1 -- 172.21.15.187:0/4088628479 >> v2:172.21.15.5:3300/0 conn(0x7f7f04065190 msgr2=0x7f7f04065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.673+0000 7f7f037fe700 1 --2- 172.21.15.187:0/4088628479 >> v2:172.21.15.5:3300/0 conn(0x7f7f04065190 0x7f7f04065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.673+0000 7f7f037fe700 1 -- 172.21.15.187:0/4088628479 >> v1:172.21.15.5:6789/0 conn(0x7f7f04065f90 legacy=0x7f7f04143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.673+0000 7f7f037fe700 1 -- 172.21.15.187:0/4088628479 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040b1640 con 0x7f7f0409ca90 2020-11-20T12:06:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.674+0000 7f7f037fe700 1 --2- 172.21.15.187:0/4088628479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f0409ca90 0x7f7f0409cf20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f7f04099f90 tx=0x7f7efc009290).ready entity=mon.1 client_cookie=d4f52a7e03a02d5a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.674+0000 7f7f01ffb700 1 -- 172.21.15.187:0/4088628479 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc00bc10 con 0x7f7f0409ca90 2020-11-20T12:06:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.674+0000 7f7f01ffb700 1 -- 172.21.15.187:0/4088628479 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc00bd70 con 0x7f7f0409ca90 2020-11-20T12:06:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.674+0000 7f7f01ffb700 1 -- 172.21.15.187:0/4088628479 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc011610 con 0x7f7f0409ca90 2020-11-20T12:06:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.675+0000 7f7f03fff700 1 -- 172.21.15.187:0/4088628479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f0409ca90 msgr2=0x7f7f0409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.675+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4088628479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f0409ca90 0x7f7f0409cf20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f7f04099f90 tx=0x7f7efc009290).stop 2020-11-20T12:06:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.675+0000 7f7f03fff700 1 -- 172.21.15.187:0/4088628479 shutdown_connections 2020-11-20T12:06:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.675+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4088628479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f0409ca90 0x7f7f0409cf20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.675+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4088628479 >> v2:172.21.15.5:3300/0 conn(0x7f7f04065190 0x7f7f04065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.675+0000 7f7f03fff700 1 -- 172.21.15.187:0/4088628479 >> 172.21.15.187:0/4088628479 conn(0x7f7f040bab30 msgr2=0x7f7f040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.675+0000 7f7f03fff700 1 -- 172.21.15.187:0/4088628479 shutdown_connections 2020-11-20T12:06:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.675+0000 7f7f03fff700 1 -- 172.21.15.187:0/4088628479 wait complete. 2020-11-20T12:06:45.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.677+0000 7f7f03fff700 1 Processor -- start 2020-11-20T12:06:45.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.677+0000 7f7f03fff700 1 -- start start 2020-11-20T12:06:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f03fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f04065190 0x7f7f0415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f03fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f04065f90 0x7f7f04099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f03fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f0409ca90 0x7f7f040b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f03fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040b2e10 con 0x7f7f0409ca90 2020-11-20T12:06:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f03fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040b3070 con 0x7f7f04065190 2020-11-20T12:06:45.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f03fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040b3310 con 0x7f7f04065f90 2020-11-20T12:06:45.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f02ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f04065190 0x7f7f0415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:45.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f02ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f04065190 0x7f7f0415bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42476/0 (socket says 172.21.15.187:42476) 2020-11-20T12:06:45.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.678+0000 7f7f02ffd700 1 -- 172.21.15.187:0/4013064415 learned_addr learned my addr 172.21.15.187:0/4013064415 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.679+0000 7f7f037fe700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f0409ca90 0x7f7f040b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.679+0000 7f7f02ffd700 1 -- 172.21.15.187:0/4013064415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f04065f90 msgr2=0x7f7f04099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.679+0000 7f7f02ffd700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f04065f90 0x7f7f04099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.679+0000 7f7f02ffd700 1 -- 172.21.15.187:0/4013064415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f0409ca90 msgr2=0x7f7f040b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.679+0000 7f7f02ffd700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f0409ca90 0x7f7f040b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.679+0000 7f7f02ffd700 1 -- 172.21.15.187:0/4013064415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0415bfb0 con 0x7f7f04065190 2020-11-20T12:06:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.679+0000 7f7f02ffd700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f04065190 0x7f7f0415bac0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f7ef400db60 tx=0x7f7ef4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.680+0000 7f7efaffd700 1 -- 172.21.15.187:0/4013064415 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4009820 con 0x7f7f04065190 2020-11-20T12:06:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.680+0000 7f7efaffd700 1 -- 172.21.15.187:0/4013064415 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef4009980 con 0x7f7f04065190 2020-11-20T12:06:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.680+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f04099a00 con 0x7f7f04065190 2020-11-20T12:06:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.680+0000 7f7efaffd700 1 -- 172.21.15.187:0/4013064415 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4009d20 con 0x7f7f04065190 2020-11-20T12:06:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.680+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0415c520 con 0x7f7f04065190 2020-11-20T12:06:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.681+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f040bb490 con 0x7f7f04065190 2020-11-20T12:06:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.681+0000 7f7efaffd700 1 -- 172.21.15.187:0/4013064415 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7ef4013690 con 0x7f7f04065190 2020-11-20T12:06:45.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.682+0000 7f7efaffd700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ed8056d20 0x7f7ed80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:45.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.682+0000 7f7efaffd700 1 -- 172.21.15.187:0/4013064415 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7ef40706f0 con 0x7f7f04065190 2020-11-20T12:06:45.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.682+0000 7f7f027fc700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ed8056d20 0x7f7ed80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:45.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.683+0000 7f7f027fc700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ed8056d20 0x7f7ed80591e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7f0409f5a0 tx=0x7f7ef00077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:45.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.686+0000 7f7efaffd700 1 -- 172.21.15.187:0/4013064415 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7ef4045a30 con 0x7f7f04065190 2020-11-20T12:06:45.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.874+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7f04061a20 con 0x7f7f04065190 2020-11-20T12:06:45.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.884+0000 7f7efaffd700 1 -- 172.21.15.187:0/4013064415 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7ef4049050 con 0x7f7f04065190 2020-11-20T12:06:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ed8056d20 msgr2=0x7f7ed80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ed8056d20 0x7f7ed80591e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7f0409f5a0 tx=0x7f7ef00077f0).stop 2020-11-20T12:06:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f04065190 msgr2=0x7f7f0415bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f04065190 0x7f7f0415bac0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f7ef400db60 tx=0x7f7ef4008040).stop 2020-11-20T12:06:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 shutdown_connections 2020-11-20T12:06:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ed8056d20 0x7f7ed80591e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f0409ca90 0x7f7f040b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f04065190 0x7f7f0415bac0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 --2- 172.21.15.187:0/4013064415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f04065f90 0x7f7f04099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 >> 172.21.15.187:0/4013064415 conn(0x7f7f040bab30 msgr2=0x7f7f04149d30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:45.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 shutdown_connections 2020-11-20T12:06:45.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:45.887+0000 7f7f03fff700 1 -- 172.21.15.187:0/4013064415 wait complete. 2020-11-20T12:06:45.912 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2020-11-20T12:06:46.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.152+0000 7f49f2c0a700 1 Processor -- start 2020-11-20T12:06:46.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.152+0000 7f49f2c0a700 1 -- start start 2020-11-20T12:06:46.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.154+0000 7f49f2c0a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec0bfbe0 0x7f49ec1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:46.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.154+0000 7f49f2c0a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f49ec09f560 0x7f49ec1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:46.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.154+0000 7f49f2c0a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec14d110 con 0x7f49ec0bf800 2020-11-20T12:06:46.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.154+0000 7f49f2c0a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ec14c9d0 con 0x7f49ec09f560 2020-11-20T12:06:46.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.154+0000 7f49f2c0a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec148aa0 con 0x7f49ec0bfbe0 2020-11-20T12:06:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.155+0000 7f49f2409700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f49ec09f560 0x7f49ec1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.155+0000 7f49f2409700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f49ec09f560 0x7f49ec1485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:34752/0 (socket says 172.21.15.187:34752) 2020-11-20T12:06:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.155+0000 7f49f2409700 1 -- 172.21.15.187:0/3395288555 learned_addr learned my addr 172.21.15.187:0/3395288555 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.155+0000 7f49f1407700 1 --2- 172.21.15.187:0/3395288555 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec0bfbe0 0x7f49ec1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:46.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.156+0000 7f49f2409700 1 -- 172.21.15.187:0/3395288555 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec0bfbe0 msgr2=0x7f49ec1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.156+0000 7f49f2409700 1 --2- 172.21.15.187:0/3395288555 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec0bfbe0 0x7f49ec1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.156+0000 7f49f2409700 1 -- 172.21.15.187:0/3395288555 >> v1:172.21.15.5:6789/0 conn(0x7f49ec0bf800 legacy=0x7f49ec14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.156+0000 7f49f2409700 1 -- 172.21.15.187:0/3395288555 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec14cbd0 con 0x7f49ec09f560 2020-11-20T12:06:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.156+0000 7f49f2409700 1 --2- 172.21.15.187:0/3395288555 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 0x7f49ec1485b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f49e400ba20 tx=0x7f49e4006eb0).ready entity=mon.0 client_cookie=26a91169b0c8aff3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:46.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f0c06700 1 -- 172.21.15.187:0/3395288555 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4009860 con 0x7f49ec09f560 2020-11-20T12:06:46.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f0c06700 1 -- 172.21.15.187:0/3395288555 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e40099c0 con 0x7f49ec09f560 2020-11-20T12:06:46.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f0c06700 1 -- 172.21.15.187:0/3395288555 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4009d40 con 0x7f49ec09f560 2020-11-20T12:06:46.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f2c0a700 1 -- 172.21.15.187:0/3395288555 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 msgr2=0x7f49ec1485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:46.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/3395288555 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 0x7f49ec1485b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f49e400ba20 tx=0x7f49e4006eb0).stop 2020-11-20T12:06:46.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f2c0a700 1 -- 172.21.15.187:0/3395288555 shutdown_connections 2020-11-20T12:06:46.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/3395288555 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 0x7f49ec1485b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:46.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/3395288555 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec0bfbe0 0x7f49ec1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:46.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f2c0a700 1 -- 172.21.15.187:0/3395288555 >> 172.21.15.187:0/3395288555 conn(0x7f49ec068370 msgr2=0x7f49ec0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:46.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f2c0a700 1 -- 172.21.15.187:0/3395288555 shutdown_connections 2020-11-20T12:06:46.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.157+0000 7f49f2c0a700 1 -- 172.21.15.187:0/3395288555 wait complete. 2020-11-20T12:06:46.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.159+0000 7f49f2c0a700 1 Processor -- start 2020-11-20T12:06:46.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.159+0000 7f49f2c0a700 1 -- start start 2020-11-20T12:06:46.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f2c0a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 0x7f49ec145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:46.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f2c0a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0bf800 0x7f49ec135a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:46.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f2c0a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec15bc20 0x7f49ec15c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:46.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f2c0a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec145fd0 con 0x7f49ec09f560 2020-11-20T12:06:46.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f2c0a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0b6770 con 0x7f49ec0bf800 2020-11-20T12:06:46.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f2c0a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0b6a10 con 0x7f49ec15bc20 2020-11-20T12:06:46.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f1407700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0bf800 0x7f49ec135a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:46.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f1407700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0bf800 0x7f49ec135a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42492/0 (socket says 172.21.15.187:42492) 2020-11-20T12:06:46.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f1407700 1 -- 172.21.15.187:0/142172575 learned_addr learned my addr 172.21.15.187:0/142172575 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:46.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f2409700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec15bc20 0x7f49ec15c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:46.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.160+0000 7f49f1c08700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 0x7f49ec145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:46.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f1407700 1 -- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec15bc20 msgr2=0x7f49ec15c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:46.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f1407700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec15bc20 0x7f49ec15c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:46.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f1407700 1 -- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 msgr2=0x7f49ec145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:46.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f1407700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 0x7f49ec145ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:46.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f1407700 1 -- 172.21.15.187:0/142172575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec135ef0 con 0x7f49ec0bf800 2020-11-20T12:06:46.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f1c08700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 0x7f49ec145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:46.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f1407700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0bf800 0x7f49ec135a00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f49dc00ec00 tx=0x7f49dc006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:46.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49d9ffb700 1 -- 172.21.15.187:0/142172575 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc00c2e0 con 0x7f49ec0bf800 2020-11-20T12:06:46.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49d9ffb700 1 -- 172.21.15.187:0/142172575 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49dc00b9a0 con 0x7f49ec0bf800 2020-11-20T12:06:46.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49d9ffb700 1 -- 172.21.15.187:0/142172575 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc00bd40 con 0x7f49ec0bf800 2020-11-20T12:06:46.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ec15c550 con 0x7f49ec0bf800 2020-11-20T12:06:46.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.161+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ec158600 con 0x7f49ec0bf800 2020-11-20T12:06:46.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.163+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ec146150 con 0x7f49ec0bf800 2020-11-20T12:06:46.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.163+0000 7f49d9ffb700 1 -- 172.21.15.187:0/142172575 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f49dc012600 con 0x7f49ec0bf800 2020-11-20T12:06:46.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.164+0000 7f49d9ffb700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f49c8056d20 0x7f49c80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:46.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.164+0000 7f49d9ffb700 1 -- 172.21.15.187:0/142172575 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f49dc0702b0 con 0x7f49ec0bf800 2020-11-20T12:06:46.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.164+0000 7f49f1c08700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f49c8056d20 0x7f49c80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:46.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.165+0000 7f49f1c08700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f49c8056d20 0x7f49c80591e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f49e8001bc0 tx=0x7f49e8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:46.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.168+0000 7f49d9ffb700 1 -- 172.21.15.187:0/142172575 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f49dc045700 con 0x7f49ec0bf800 2020-11-20T12:06:46.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:46.422+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f49ec0615e0 con 0x7f49ec0bf800 2020-11-20T12:06:47.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.141+0000 7f49d9ffb700 1 -- 172.21.15.187:0/142172575 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v111) v1 ==== 132+0+0 (secure 0 0 0) 0x7f49dc048d20 con 0x7f49ec0bf800 2020-11-20T12:06:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.144+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f49c8056d20 msgr2=0x7f49c80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.144+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f49c8056d20 0x7f49c80591e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f49e8001bc0 tx=0x7f49e8009040).stop 2020-11-20T12:06:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.144+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0bf800 msgr2=0x7f49ec135a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.144+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0bf800 0x7f49ec135a00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f49dc00ec00 tx=0x7f49dc006df0).stop 2020-11-20T12:06:47.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.145+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 shutdown_connections 2020-11-20T12:06:47.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.145+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f49c8056d20 0x7f49c80591e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.145+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49ec09f560 0x7f49ec145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.145+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49ec15bc20 0x7f49ec15c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.145+0000 7f49f2c0a700 1 --2- 172.21.15.187:0/142172575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0bf800 0x7f49ec135a00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.145+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 >> 172.21.15.187:0/142172575 conn(0x7f49ec068370 msgr2=0x7f49ec0bbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:47.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.145+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 shutdown_connections 2020-11-20T12:06:47.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.145+0000 7f49f2c0a700 1 -- 172.21.15.187:0/142172575 wait complete. 2020-11-20T12:06:47.173 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:06:47.173 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:06:47.174 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2020-11-20T12:06:47.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.412+0000 7f141a6dd700 1 Processor -- start 2020-11-20T12:06:47.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.412+0000 7f141a6dd700 1 -- start start 2020-11-20T12:06:47.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.413+0000 7f141a6dd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 0x7f1414065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.413+0000 7f141a6dd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1414065f90 0x7f1414143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.413+0000 7f141a6dd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14140b13e0 con 0x7f1414143c70 2020-11-20T12:06:47.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.413+0000 7f141a6dd700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141414ab30 con 0x7f1414065f90 2020-11-20T12:06:47.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.413+0000 7f141a6dd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1414062c10 con 0x7f1414065190 2020-11-20T12:06:47.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.414+0000 7f1418eda700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1414065f90 0x7f1414143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.414+0000 7f1418eda700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1414065f90 0x7f1414143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42502/0 (socket says 172.21.15.187:42502) 2020-11-20T12:06:47.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.414+0000 7f1418eda700 1 -- 172.21.15.187:0/637675921 learned_addr learned my addr 172.21.15.187:0/637675921 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:47.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.415+0000 7f1418eda700 1 -- 172.21.15.187:0/637675921 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 msgr2=0x7f1414065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:47.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.415+0000 7f1418eda700 1 --2- 172.21.15.187:0/637675921 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 0x7f1414065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:47.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.415+0000 7f1418eda700 1 -- 172.21.15.187:0/637675921 >> v1:172.21.15.5:6789/0 conn(0x7f1414143c70 legacy=0x7f141409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:47.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.415+0000 7f1418eda700 1 -- 172.21.15.187:0/637675921 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14140b1640 con 0x7f1414065f90 2020-11-20T12:06:47.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.415+0000 7f1418eda700 1 --2- 172.21.15.187:0/637675921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414143780 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f141414ac70 tx=0x7f1404006eb0).ready entity=mon.1 client_cookie=bfa5d30d40f1bc2d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:47.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f1403fff700 1 -- 172.21.15.187:0/637675921 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1404009960 con 0x7f1414065f90 2020-11-20T12:06:47.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f1403fff700 1 -- 172.21.15.187:0/637675921 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1404009ac0 con 0x7f1414065f90 2020-11-20T12:06:47.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f1403fff700 1 -- 172.21.15.187:0/637675921 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1404009e40 con 0x7f1414065f90 2020-11-20T12:06:47.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f141a6dd700 1 -- 172.21.15.187:0/637675921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 msgr2=0x7f1414143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f141a6dd700 1 --2- 172.21.15.187:0/637675921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414143780 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f141414ac70 tx=0x7f1404006eb0).stop 2020-11-20T12:06:47.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f141a6dd700 1 -- 172.21.15.187:0/637675921 shutdown_connections 2020-11-20T12:06:47.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f141a6dd700 1 --2- 172.21.15.187:0/637675921 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 0x7f1414065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:47.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f141a6dd700 1 --2- 172.21.15.187:0/637675921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414143780 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f141a6dd700 1 -- 172.21.15.187:0/637675921 >> 172.21.15.187:0/637675921 conn(0x7f14140bab30 msgr2=0x7f14140baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:47.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f141a6dd700 1 -- 172.21.15.187:0/637675921 shutdown_connections 2020-11-20T12:06:47.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.416+0000 7f141a6dd700 1 -- 172.21.15.187:0/637675921 wait complete. 2020-11-20T12:06:47.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.418+0000 7f141a6dd700 1 Processor -- start 2020-11-20T12:06:47.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.418+0000 7f141a6dd700 1 -- start start 2020-11-20T12:06:47.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f141a6dd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 0x7f141415a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f141a6dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f141a6dd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1414143c70 0x7f14141598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f141a6dd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14140b3430 con 0x7f1414143c70 2020-11-20T12:06:47.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f141a6dd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14140b36d0 con 0x7f1414065f90 2020-11-20T12:06:47.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f141a6dd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1414146ab0 con 0x7f1414065190 2020-11-20T12:06:47.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f1418eda700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f1418eda700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414142fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42506/0 (socket says 172.21.15.187:42506) 2020-11-20T12:06:47.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.419+0000 7f1418eda700 1 -- 172.21.15.187:0/4123921019 learned_addr learned my addr 172.21.15.187:0/4123921019 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:47.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f1419edc700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1414143c70 0x7f14141598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f1418eda700 1 -- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 msgr2=0x7f141415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f14196db700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 0x7f141415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f1418eda700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 0x7f141415a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f1418eda700 1 -- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1414143c70 msgr2=0x7f14141598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f1418eda700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1414143c70 0x7f14141598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f1418eda700 1 -- 172.21.15.187:0/4123921019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14141434a0 con 0x7f1414065f90 2020-11-20T12:06:47.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f1419edc700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1414143c70 0x7f14141598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f1418eda700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414142fb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f1404010000 tx=0x7f1404015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f14017fa700 1 -- 172.21.15.187:0/4123921019 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1404011ea0 con 0x7f1414065f90 2020-11-20T12:06:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f14017fa700 1 -- 172.21.15.187:0/4123921019 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140401d5d0 con 0x7f1414065f90 2020-11-20T12:06:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14140a0070 con 0x7f1414065f90 2020-11-20T12:06:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.420+0000 7f14017fa700 1 -- 172.21.15.187:0/4123921019 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140401d990 con 0x7f1414065f90 2020-11-20T12:06:47.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.421+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1414159da0 con 0x7f1414065f90 2020-11-20T12:06:47.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.422+0000 7f14017fa700 1 -- 172.21.15.187:0/4123921019 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1404025030 con 0x7f1414065f90 2020-11-20T12:06:47.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.422+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1414146c70 con 0x7f1414065f90 2020-11-20T12:06:47.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.423+0000 7f14017fa700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13ec056d20 0x7f13ec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.423+0000 7f14017fa700 1 -- 172.21.15.187:0/4123921019 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1404071840 con 0x7f1414065f90 2020-11-20T12:06:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.423+0000 7f14196db700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13ec056d20 0x7f13ec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.424+0000 7f14196db700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13ec056d20 0x7f13ec0591e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f14140b1e90 tx=0x7f1410008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.428+0000 7f14017fa700 1 -- 172.21.15.187:0/4123921019 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1404046d40 con 0x7f1414065f90 2020-11-20T12:06:47.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:47.681+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f141415a050 con 0x7f1414065f90 2020-11-20T12:06:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.151+0000 7f14017fa700 1 -- 172.21.15.187:0/4123921019 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v112) v1 ==== 136+0+0 (secure 0 0 0) 0x7f1404023070 con 0x7f1414065f90 2020-11-20T12:06:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13ec056d20 msgr2=0x7f13ec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13ec056d20 0x7f13ec0591e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f14140b1e90 tx=0x7f1410008040).stop 2020-11-20T12:06:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 msgr2=0x7f1414142fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414142fb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f1404010000 tx=0x7f1404015000).stop 2020-11-20T12:06:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 shutdown_connections 2020-11-20T12:06:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13ec056d20 0x7f13ec0591e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1414143c70 0x7f14141598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1414065190 0x7f141415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 --2- 172.21.15.187:0/4123921019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1414065f90 0x7f1414142fb0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 >> 172.21.15.187:0/4123921019 conn(0x7f14140bab30 msgr2=0x7f14141498b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.155+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 shutdown_connections 2020-11-20T12:06:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.156+0000 7f141a6dd700 1 -- 172.21.15.187:0/4123921019 wait complete. 2020-11-20T12:06:48.188 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:06:48.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.435+0000 7fe335af3700 1 Processor -- start 2020-11-20T12:06:48.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.435+0000 7fe335af3700 1 -- start start 2020-11-20T12:06:48.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.436+0000 7fe335af3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe330061de0 0x7fe330066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:48.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.436+0000 7fe335af3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3301487c0 0x7fe330066bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:48.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.436+0000 7fe335af3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe330062280 con 0x7fe3301483e0 2020-11-20T12:06:48.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.436+0000 7fe335af3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe330062520 con 0x7fe330061de0 2020-11-20T12:06:48.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.436+0000 7fe335af3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300627c0 con 0x7fe3301487c0 2020-11-20T12:06:48.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.437+0000 7fe32effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe330061de0 0x7fe330066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:48.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.437+0000 7fe32effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe330061de0 0x7fe330066700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42512/0 (socket says 172.21.15.187:42512) 2020-11-20T12:06:48.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.437+0000 7fe32effd700 1 -- 172.21.15.187:0/3703644743 learned_addr learned my addr 172.21.15.187:0/3703644743 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:48.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.437+0000 7fe32effd700 1 -- 172.21.15.187:0/3703644743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3301487c0 msgr2=0x7fe330066bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:48.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.437+0000 7fe32e7fc700 1 -- 172.21.15.187:0/3703644743 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251578063 0 0) 0x7fe330062280 con 0x7fe3301483e0 2020-11-20T12:06:48.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.437+0000 7fe32ffff700 1 --2- 172.21.15.187:0/3703644743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3301487c0 0x7fe330066bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:48.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.438+0000 7fe32effd700 1 --2- 172.21.15.187:0/3703644743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3301487c0 0x7fe330066bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.438+0000 7fe32effd700 1 -- 172.21.15.187:0/3703644743 >> v1:172.21.15.187:6789/0 conn(0x7fe3301483e0 legacy=0x7fe330065300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:48.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.438+0000 7fe32effd700 1 -- 172.21.15.187:0/3703644743 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe330065a10 con 0x7fe330061de0 2020-11-20T12:06:48.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.438+0000 7fe32ffff700 1 --2- 172.21.15.187:0/3703644743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3301487c0 0x7fe330066bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:48.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.438+0000 7fe32effd700 1 --2- 172.21.15.187:0/3703644743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330061de0 0x7fe330066700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fe324006fd0 tx=0x7fe324006eb0).ready entity=mon.1 client_cookie=cefac3fd26ca82ee server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:48.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.438+0000 7fe32e7fc700 1 -- 172.21.15.187:0/3703644743 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3240098d0 con 0x7fe330061de0 2020-11-20T12:06:48.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.438+0000 7fe32e7fc700 1 -- 172.21.15.187:0/3703644743 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe324009a30 con 0x7fe330061de0 2020-11-20T12:06:48.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.438+0000 7fe32e7fc700 1 -- 172.21.15.187:0/3703644743 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324009db0 con 0x7fe330061de0 2020-11-20T12:06:48.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.439+0000 7fe335af3700 1 -- 172.21.15.187:0/3703644743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330061de0 msgr2=0x7fe330066700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:48.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.439+0000 7fe335af3700 1 --2- 172.21.15.187:0/3703644743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330061de0 0x7fe330066700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fe324006fd0 tx=0x7fe324006eb0).stop 2020-11-20T12:06:48.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.439+0000 7fe335af3700 1 -- 172.21.15.187:0/3703644743 shutdown_connections 2020-11-20T12:06:48.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.439+0000 7fe335af3700 1 --2- 172.21.15.187:0/3703644743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3301487c0 0x7fe330066bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.439+0000 7fe335af3700 1 --2- 172.21.15.187:0/3703644743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330061de0 0x7fe330066700 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.439+0000 7fe335af3700 1 -- 172.21.15.187:0/3703644743 >> 172.21.15.187:0/3703644743 conn(0x7fe3300bacf0 msgr2=0x7fe3300bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:48.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.439+0000 7fe335af3700 1 -- 172.21.15.187:0/3703644743 shutdown_connections 2020-11-20T12:06:48.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.439+0000 7fe335af3700 1 -- 172.21.15.187:0/3703644743 wait complete. 2020-11-20T12:06:48.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.441+0000 7fe335af3700 1 Processor -- start 2020-11-20T12:06:48.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.441+0000 7fe335af3700 1 -- start start 2020-11-20T12:06:48.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.441+0000 7fe335af3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3301483e0 0x7fe3300b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:48.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.441+0000 7fe335af3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3301487c0 0x7fe33015c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:48.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe335af3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe330148bf0 0x7fe33015c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:48.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe335af3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330149090 con 0x7fe3301487c0 2020-11-20T12:06:48.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe335af3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330149330 con 0x7fe3301483e0 2020-11-20T12:06:48.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe335af3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3301495d0 con 0x7fe330148bf0 2020-11-20T12:06:48.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe32f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3301483e0 0x7fe3300b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:48.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe32f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3301483e0 0x7fe3300b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42518/0 (socket says 172.21.15.187:42518) 2020-11-20T12:06:48.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe32f7fe700 1 -- 172.21.15.187:0/2284149662 learned_addr learned my addr 172.21.15.187:0/2284149662 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:48.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe32ffff700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe330148bf0 0x7fe33015c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:48.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.442+0000 7fe32effd700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3301487c0 0x7fe33015c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:48.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe32f7fe700 1 -- 172.21.15.187:0/2284149662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe330148bf0 msgr2=0x7fe33015c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:48.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe32f7fe700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe330148bf0 0x7fe33015c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe32f7fe700 1 -- 172.21.15.187:0/2284149662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3301487c0 msgr2=0x7fe33015c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:48.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe32f7fe700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3301487c0 0x7fe33015c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe32f7fe700 1 -- 172.21.15.187:0/2284149662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3300b3b50 con 0x7fe3301483e0 2020-11-20T12:06:48.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe32f7fe700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3301483e0 0x7fe3300b3660 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fe320007fd0 tx=0x7fe320007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:48.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe30f7fe700 1 -- 172.21.15.187:0/2284149662 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32000e560 con 0x7fe3301483e0 2020-11-20T12:06:48.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe30f7fe700 1 -- 172.21.15.187:0/2284149662 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32000e6e0 con 0x7fe3301483e0 2020-11-20T12:06:48.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe330149a70 con 0x7fe3301483e0 2020-11-20T12:06:48.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe30f7fe700 1 -- 172.21.15.187:0/2284149662 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32000eaa0 con 0x7fe3301483e0 2020-11-20T12:06:48.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.443+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33015cc90 con 0x7fe3301483e0 2020-11-20T12:06:48.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.444+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3300bb5e0 con 0x7fe3301483e0 2020-11-20T12:06:48.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.445+0000 7fe30f7fe700 1 -- 172.21.15.187:0/2284149662 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe320010070 con 0x7fe3301483e0 2020-11-20T12:06:48.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.446+0000 7fe30f7fe700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe308056d20 0x7fe3080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:48.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.446+0000 7fe30f7fe700 1 -- 172.21.15.187:0/2284149662 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe32006f800 con 0x7fe3301483e0 2020-11-20T12:06:48.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.446+0000 7fe32effd700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe308056d20 0x7fe3080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:48.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.447+0000 7fe32effd700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe308056d20 0x7fe3080591e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe33009a740 tx=0x7fe324015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:48.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.450+0000 7fe30f7fe700 1 -- 172.21.15.187:0/2284149662 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe320044c50 con 0x7fe3301483e0 2020-11-20T12:06:48.492 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:48.501 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:48.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.639+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7fe330044f60 con 0x7fe3301483e0 2020-11-20T12:06:48.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.648+0000 7fe30f7fe700 1 -- 172.21.15.187:0/2284149662 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe320048270 con 0x7fe3301483e0 2020-11-20T12:06:48.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe308056d20 msgr2=0x7fe3080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:48.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe308056d20 0x7fe3080591e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe33009a740 tx=0x7fe324015000).stop 2020-11-20T12:06:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3301483e0 msgr2=0x7fe3300b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3301483e0 0x7fe3300b3660 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fe320007fd0 tx=0x7fe320007ac0).stop 2020-11-20T12:06:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 shutdown_connections 2020-11-20T12:06:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe308056d20 0x7fe3080591e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe330148bf0 0x7fe33015c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3301483e0 0x7fe3300b3660 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 --2- 172.21.15.187:0/2284149662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3301487c0 0x7fe33015c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:48.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 >> 172.21.15.187:0/2284149662 conn(0x7fe3300bacf0 msgr2=0x7fe3300b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:48.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 shutdown_connections 2020-11-20T12:06:48.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:48.651+0000 7fe335af3700 1 -- 172.21.15.187:0/2284149662 wait complete. 2020-11-20T12:06:48.676 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:06:48.677 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:06:48.796 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:06:48.816 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:06:48.926 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:06:48.946 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:06:48.947 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:06:48.947 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:49.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.183+0000 7fd3d1c24700 1 Processor -- start 2020-11-20T12:06:49.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.184+0000 7fd3d1c24700 1 -- start start 2020-11-20T12:06:49.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d1c24700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3cc09dba0 0x7fd3cc0676a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d1c24700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 0x7fd3cc145f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d1c24700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd3cc0b8190 0x7fd3cc146400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d1c24700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3cc1468f0 con 0x7fd3cc0b8190 2020-11-20T12:06:49.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d1c24700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3cc063520 con 0x7fd3cc09dba0 2020-11-20T12:06:49.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d1c24700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0637c0 con 0x7fd3cc09df80 2020-11-20T12:06:49.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d0c22700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3cc09dba0 0x7fd3cc0676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d0c22700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3cc09dba0 0x7fd3cc0676a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42526/0 (socket says 172.21.15.187:42526) 2020-11-20T12:06:49.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.185+0000 7fd3d0c22700 1 -- 172.21.15.187:0/1551095061 learned_addr learned my addr 172.21.15.187:0/1551095061 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:49.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3d0c22700 1 -- 172.21.15.187:0/1551095061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 msgr2=0x7fd3cc145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:49.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3d0c22700 1 --2- 172.21.15.187:0/1551095061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 0x7fd3cc145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:49.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3d0c22700 1 -- 172.21.15.187:0/1551095061 >> v2:172.21.15.5:3300/0 conn(0x7fd3cc0b8190 msgr2=0x7fd3cc146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:49.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3d0c22700 1 --2- 172.21.15.187:0/1551095061 >> v2:172.21.15.5:3300/0 conn(0x7fd3cc0b8190 0x7fd3cc146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:49.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3d0c22700 1 -- 172.21.15.187:0/1551095061 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc063a60 con 0x7fd3cc09dba0 2020-11-20T12:06:49.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3d0c22700 1 --2- 172.21.15.187:0/1551095061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc0676a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fd3cc0b96f0 tx=0x7fd3c8006eb0).ready entity=mon.1 client_cookie=204abb27135fb2b2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3c37fe700 1 -- 172.21.15.187:0/1551095061 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c80098d0 con 0x7fd3cc09dba0 2020-11-20T12:06:49.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3c37fe700 1 -- 172.21.15.187:0/1551095061 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c8009a30 con 0x7fd3cc09dba0 2020-11-20T12:06:49.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.186+0000 7fd3c37fe700 1 -- 172.21.15.187:0/1551095061 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8009db0 con 0x7fd3cc09dba0 2020-11-20T12:06:49.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1551095061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 msgr2=0x7fd3cc0676a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1551095061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc0676a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fd3cc0b96f0 tx=0x7fd3c8006eb0).stop 2020-11-20T12:06:49.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1551095061 shutdown_connections 2020-11-20T12:06:49.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1551095061 >> v2:172.21.15.5:3300/0 conn(0x7fd3cc0b8190 0x7fd3cc146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:49.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1551095061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc0676a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1551095061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 0x7fd3cc145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:49.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1551095061 >> 172.21.15.187:0/1551095061 conn(0x7fd3cc0bbc20 msgr2=0x7fd3cc0bc030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:49.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1551095061 shutdown_connections 2020-11-20T12:06:49.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.187+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1551095061 wait complete. 2020-11-20T12:06:49.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.189+0000 7fd3d1c24700 1 Processor -- start 2020-11-20T12:06:49.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.189+0000 7fd3d1c24700 1 -- start start 2020-11-20T12:06:49.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.189+0000 7fd3d1c24700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc14c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d1c24700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 0x7fd3cc0b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d1c24700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3cc0b8190 0x7fd3cc143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d1c24700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0b27d0 con 0x7fd3cc0b8190 2020-11-20T12:06:49.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d1c24700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0b2a70 con 0x7fd3cc09dba0 2020-11-20T12:06:49.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d1c24700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0b2d10 con 0x7fd3cc09df80 2020-11-20T12:06:49.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d0c22700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc14c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d0c22700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc14c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42532/0 (socket says 172.21.15.187:42532) 2020-11-20T12:06:49.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d0c22700 1 -- 172.21.15.187:0/1693071355 learned_addr learned my addr 172.21.15.187:0/1693071355 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:49.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3c3fff700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 0x7fd3cc0b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d0c22700 1 -- 172.21.15.187:0/1693071355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 msgr2=0x7fd3cc0b1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d0c22700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 0x7fd3cc0b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d1423700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3cc0b8190 0x7fd3cc143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d0c22700 1 -- 172.21.15.187:0/1693071355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3cc0b8190 msgr2=0x7fd3cc143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.190+0000 7fd3d0c22700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3cc0b8190 0x7fd3cc143220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.191+0000 7fd3d0c22700 1 -- 172.21.15.187:0/1693071355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc14c5f0 con 0x7fd3cc09dba0 2020-11-20T12:06:49.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.191+0000 7fd3d0c22700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc14c100 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd3c8006ef0 tx=0x7fd3c8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.191+0000 7fd3c0ff9700 1 -- 172.21.15.187:0/1693071355 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c801d5d0 con 0x7fd3cc09dba0 2020-11-20T12:06:49.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.191+0000 7fd3c0ff9700 1 -- 172.21.15.187:0/1693071355 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c801d730 con 0x7fd3cc09dba0 2020-11-20T12:06:49.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.191+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3cc0b5a90 con 0x7fd3cc09dba0 2020-11-20T12:06:49.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.191+0000 7fd3c0ff9700 1 -- 172.21.15.187:0/1693071355 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c801dad0 con 0x7fd3cc09dba0 2020-11-20T12:06:49.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.191+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3cc0b2570 con 0x7fd3cc09dba0 2020-11-20T12:06:49.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.193+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3cc0b2ed0 con 0x7fd3cc09dba0 2020-11-20T12:06:49.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.193+0000 7fd3c0ff9700 1 -- 172.21.15.187:0/1693071355 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd3c8025030 con 0x7fd3cc09dba0 2020-11-20T12:06:49.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.194+0000 7fd3c0ff9700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3a4056d20 0x7fd3a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.194+0000 7fd3c0ff9700 1 -- 172.21.15.187:0/1693071355 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd3c8071810 con 0x7fd3cc09dba0 2020-11-20T12:06:49.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.194+0000 7fd3c3fff700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3a4056d20 0x7fd3a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.195+0000 7fd3c3fff700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3a4056d20 0x7fd3a40591e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd3cc0b15d0 tx=0x7fd3bc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.198+0000 7fd3c0ff9700 1 -- 172.21.15.187:0/1693071355 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd3c8046d10 con 0x7fd3cc09dba0 2020-11-20T12:06:49.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.442+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3cc0446d0 con 0x7fd3cc09dba0 2020-11-20T12:06:49.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.443+0000 7fd3c0ff9700 1 -- 172.21.15.187:0/1693071355 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7fd3c8023070 con 0x7fd3cc09dba0 2020-11-20T12:06:49.447 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:49.447 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:49.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3a4056d20 msgr2=0x7fd3a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3a4056d20 0x7fd3a40591e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd3cc0b15d0 tx=0x7fd3bc008040).stop 2020-11-20T12:06:49.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 msgr2=0x7fd3cc14c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc14c100 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd3c8006ef0 tx=0x7fd3c8015000).stop 2020-11-20T12:06:49.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 shutdown_connections 2020-11-20T12:06:49.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd3a4056d20 0x7fd3a40591e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3cc0b8190 0x7fd3cc143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc09dba0 0x7fd3cc14c100 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 --2- 172.21.15.187:0/1693071355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3cc09df80 0x7fd3cc0b1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 >> 172.21.15.187:0/1693071355 conn(0x7fd3cc0bbc20 msgr2=0x7fd3cc143d20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:49.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 shutdown_connections 2020-11-20T12:06:49.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.446+0000 7fd3d1c24700 1 -- 172.21.15.187:0/1693071355 wait complete. 2020-11-20T12:06:49.632 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:49.633 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:06:49.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.871+0000 7f4702ed9700 1 Processor -- start 2020-11-20T12:06:49.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.872+0000 7f4702ed9700 1 -- start start 2020-11-20T12:06:49.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.873+0000 7f4702ed9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f46fc1484d0 0x7f46fc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.873+0000 7f4702ed9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc066f70 0x7f46fc0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.873+0000 7f4702ed9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc0b1300 con 0x7f46fc0622c0 2020-11-20T12:06:49.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.873+0000 7f4702ed9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc067730 con 0x7f46fc1484d0 2020-11-20T12:06:49.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.873+0000 7f4702ed9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0b15a0 con 0x7f46fc066f70 2020-11-20T12:06:49.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.874+0000 7f4701ed7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f46fc1484d0 0x7f46fc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.874+0000 7f4701ed7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f46fc1484d0 0x7f46fc061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:34804/0 (socket says 172.21.15.187:34804) 2020-11-20T12:06:49.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.874+0000 7f4701ed7700 1 -- 172.21.15.187:0/3672226043 learned_addr learned my addr 172.21.15.187:0/3672226043 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:49.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.874+0000 7f47026d8700 1 --2- 172.21.15.187:0/3672226043 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc066f70 0x7f46fc0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.875+0000 7f4701ed7700 1 -- 172.21.15.187:0/3672226043 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc066f70 msgr2=0x7f46fc0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:49.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.875+0000 7f4701ed7700 1 --2- 172.21.15.187:0/3672226043 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc066f70 0x7f46fc0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.875+0000 7f4701ed7700 1 -- 172.21.15.187:0/3672226043 >> v1:172.21.15.5:6789/0 conn(0x7f46fc0622c0 legacy=0x7f46fc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.875+0000 7f4701ed7700 1 -- 172.21.15.187:0/3672226043 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0b1840 con 0x7f46fc1484d0 2020-11-20T12:06:49.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.875+0000 7f4701ed7700 1 --2- 172.21.15.187:0/3672226043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46fc1484d0 0x7f46fc061dd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f46f8006fd0 tx=0x7f46f8006eb0).ready entity=mon.0 client_cookie=269774f928a3e5dc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.876+0000 7f4700ed5700 1 -- 172.21.15.187:0/3672226043 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8009810 con 0x7f46fc1484d0 2020-11-20T12:06:49.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.876+0000 7f4700ed5700 1 -- 172.21.15.187:0/3672226043 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f8009970 con 0x7f46fc1484d0 2020-11-20T12:06:49.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.876+0000 7f4700ed5700 1 -- 172.21.15.187:0/3672226043 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8009cf0 con 0x7f46fc1484d0 2020-11-20T12:06:49.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.876+0000 7f4702ed9700 1 -- 172.21.15.187:0/3672226043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46fc1484d0 msgr2=0x7f46fc061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.876+0000 7f4702ed9700 1 --2- 172.21.15.187:0/3672226043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46fc1484d0 0x7f46fc061dd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f46f8006fd0 tx=0x7f46f8006eb0).stop 2020-11-20T12:06:49.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.877+0000 7f4702ed9700 1 -- 172.21.15.187:0/3672226043 shutdown_connections 2020-11-20T12:06:49.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.877+0000 7f4702ed9700 1 --2- 172.21.15.187:0/3672226043 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc066f70 0x7f46fc0b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.877+0000 7f4702ed9700 1 --2- 172.21.15.187:0/3672226043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46fc1484d0 0x7f46fc061dd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.877+0000 7f4702ed9700 1 -- 172.21.15.187:0/3672226043 >> 172.21.15.187:0/3672226043 conn(0x7f46fc0bab30 msgr2=0x7f46fc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:49.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.877+0000 7f4702ed9700 1 -- 172.21.15.187:0/3672226043 shutdown_connections 2020-11-20T12:06:49.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.877+0000 7f4702ed9700 1 -- 172.21.15.187:0/3672226043 wait complete. 2020-11-20T12:06:49.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.879+0000 7f4702ed9700 1 Processor -- start 2020-11-20T12:06:49.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.879+0000 7f4702ed9700 1 -- start start 2020-11-20T12:06:49.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.880+0000 7f4702ed9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46fc0622c0 0x7f46fc14b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.880+0000 7f4702ed9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46fc066f70 0x7f46fc143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.880+0000 7f4702ed9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc1484d0 0x7f46fc14cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.880+0000 7f4702ed9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc14a160 con 0x7f46fc066f70 2020-11-20T12:06:49.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.880+0000 7f4702ed9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc143bb0 con 0x7f46fc0622c0 2020-11-20T12:06:49.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.880+0000 7f4702ed9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc143e50 con 0x7f46fc1484d0 2020-11-20T12:06:49.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.880+0000 7f4701ed7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46fc0622c0 0x7f46fc14b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.881+0000 7f4701ed7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46fc0622c0 0x7f46fc14b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42546/0 (socket says 172.21.15.187:42546) 2020-11-20T12:06:49.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.881+0000 7f4701ed7700 1 -- 172.21.15.187:0/2849159063 learned_addr learned my addr 172.21.15.187:0/2849159063 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:49.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.881+0000 7f4701ed7700 1 -- 172.21.15.187:0/2849159063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc1484d0 msgr2=0x7f46fc14cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.881+0000 7f4701ed7700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc1484d0 0x7f46fc14cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:49.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.881+0000 7f4701ed7700 1 -- 172.21.15.187:0/2849159063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46fc066f70 msgr2=0x7f46fc143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.881+0000 7f4701ed7700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46fc066f70 0x7f46fc143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:49.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.881+0000 7f4701ed7700 1 -- 172.21.15.187:0/2849159063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc1440f0 con 0x7f46fc0622c0 2020-11-20T12:06:49.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.881+0000 7f4701ed7700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46fc0622c0 0x7f46fc14b580 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f46f800b1b0 tx=0x7f46f8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.882+0000 7f46e9ffb700 1 -- 172.21.15.187:0/2849159063 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8011d50 con 0x7f46fc0622c0 2020-11-20T12:06:49.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.882+0000 7f46e9ffb700 1 -- 172.21.15.187:0/2849159063 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f801e460 con 0x7f46fc0622c0 2020-11-20T12:06:49.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.882+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc143620 con 0x7f46fc0622c0 2020-11-20T12:06:49.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.882+0000 7f46e9ffb700 1 -- 172.21.15.187:0/2849159063 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f801e820 con 0x7f46fc0622c0 2020-11-20T12:06:49.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.882+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc1447b0 con 0x7f46fc0622c0 2020-11-20T12:06:49.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.883+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc061a20 con 0x7f46fc0622c0 2020-11-20T12:06:49.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.883+0000 7f46e9ffb700 1 -- 172.21.15.187:0/2849159063 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f46f8025030 con 0x7f46fc0622c0 2020-11-20T12:06:49.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.884+0000 7f46e9ffb700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f46d4056d20 0x7f46d40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.884+0000 7f46e9ffb700 1 -- 172.21.15.187:0/2849159063 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f46f8071940 con 0x7f46fc0622c0 2020-11-20T12:06:49.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.885+0000 7f47016d6700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f46d4056d20 0x7f46d40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.885+0000 7f47016d6700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f46d4056d20 0x7f46d40591e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f46fc144d50 tx=0x7f46ec007810).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:49.888+0000 7f46e9ffb700 1 -- 172.21.15.187:0/2849159063 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f46f8046cc0 con 0x7f46fc0622c0 2020-11-20T12:06:50.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.074+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46fc1488b0 con 0x7f46fc0622c0 2020-11-20T12:06:50.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.076+0000 7f46e9ffb700 1 -- 172.21.15.187:0/2849159063 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f46f80069e0 con 0x7f46fc0622c0 2020-11-20T12:06:50.079 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:50.080 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":32,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:06:12.578719+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:06:09.312569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2020-11-20T12:06:10.581583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:06:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.078+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f46d4056d20 msgr2=0x7f46d40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.078+0000 7f4702ed9700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f46d4056d20 0x7f46d40591e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f46fc144d50 tx=0x7f46ec007810).stop 2020-11-20T12:06:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.078+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46fc0622c0 msgr2=0x7f46fc14b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46fc0622c0 0x7f46fc14b580 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f46f800b1b0 tx=0x7f46f8015000).stop 2020-11-20T12:06:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 shutdown_connections 2020-11-20T12:06:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f46d4056d20 0x7f46d40591e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46fc1484d0 0x7f46fc14cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:50.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46fc0622c0 0x7f46fc14b580 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 --2- 172.21.15.187:0/2849159063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46fc066f70 0x7f46fc143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:50.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 >> 172.21.15.187:0/2849159063 conn(0x7f46fc0bab30 msgr2=0x7f46fc0b9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:50.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 shutdown_connections 2020-11-20T12:06:50.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.079+0000 7f4702ed9700 1 -- 172.21.15.187:0/2849159063 wait complete. 2020-11-20T12:06:50.103 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:06:09.312569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:06:10.581583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:06:50.104 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:06:50.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.350+0000 7fe0aee1d700 1 Processor -- start 2020-11-20T12:06:50.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.350+0000 7fe0aee1d700 1 -- start start 2020-11-20T12:06:50.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.351+0000 7fe0aee1d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe0a8065f90 0x7fe0a8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.351+0000 7fe0aee1d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe0a809ca90 0x7fe0a809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.351+0000 7fe0aee1d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a80b13e0 con 0x7fe0a8065190 2020-11-20T12:06:50.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.351+0000 7fe0aee1d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a814ab30 con 0x7fe0a809ca90 2020-11-20T12:06:50.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.351+0000 7fe0aee1d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a8062c10 con 0x7fe0a8065f90 2020-11-20T12:06:50.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.352+0000 7fe0ad61a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe0a8065f90 0x7fe0a8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.352+0000 7fe0ade1b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe0a8065190 0x7fe0a8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39160/0 (socket says 172.21.15.187:39160) 2020-11-20T12:06:50.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.352+0000 7fe0ade1b700 1 -- 172.21.15.187:0/2705530842 learned_addr learned my addr 172.21.15.187:0/2705530842 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:50.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.352+0000 7fe0ae61c700 1 --2- 172.21.15.187:0/2705530842 >> v2:172.21.15.5:3300/0 conn(0x7fe0a809ca90 0x7fe0a809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.353+0000 7fe0ad61a700 1 -- 172.21.15.187:0/2705530842 >> v2:172.21.15.5:3300/0 conn(0x7fe0a809ca90 msgr2=0x7fe0a809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:50.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.353+0000 7fe0ad61a700 1 --2- 172.21.15.187:0/2705530842 >> v2:172.21.15.5:3300/0 conn(0x7fe0a809ca90 0x7fe0a809cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.353+0000 7fe0ad61a700 1 -- 172.21.15.187:0/2705530842 >> v1:172.21.15.187:6789/0 conn(0x7fe0a8065190 legacy=0x7fe0a8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.353+0000 7fe0ad61a700 1 -- 172.21.15.187:0/2705530842 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a80b1640 con 0x7fe0a8065f90 2020-11-20T12:06:50.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.353+0000 7fe0ad61a700 1 --2- 172.21.15.187:0/2705530842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a8065f90 0x7fe0a8065570 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fe0a80b5fb0 tx=0x7fe0a4006df0).ready entity=mon.1 client_cookie=1a98e0a36b24f6b2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:50.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.353+0000 7fe0ace19700 1 -- 172.21.15.187:0/2705530842 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4009840 con 0x7fe0a8065f90 2020-11-20T12:06:50.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.353+0000 7fe0ace19700 1 -- 172.21.15.187:0/2705530842 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a40099a0 con 0x7fe0a8065f90 2020-11-20T12:06:50.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.353+0000 7fe0ace19700 1 -- 172.21.15.187:0/2705530842 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4009d20 con 0x7fe0a8065f90 2020-11-20T12:06:50.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.354+0000 7fe0aee1d700 1 -- 172.21.15.187:0/2705530842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a8065f90 msgr2=0x7fe0a8065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.354+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/2705530842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a8065f90 0x7fe0a8065570 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fe0a80b5fb0 tx=0x7fe0a4006df0).stop 2020-11-20T12:06:50.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.354+0000 7fe0aee1d700 1 -- 172.21.15.187:0/2705530842 shutdown_connections 2020-11-20T12:06:50.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.354+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/2705530842 >> v2:172.21.15.5:3300/0 conn(0x7fe0a809ca90 0x7fe0a809cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.354+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/2705530842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a8065f90 0x7fe0a8065570 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.354+0000 7fe0aee1d700 1 -- 172.21.15.187:0/2705530842 >> 172.21.15.187:0/2705530842 conn(0x7fe0a80bab30 msgr2=0x7fe0a80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:50.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.354+0000 7fe0aee1d700 1 -- 172.21.15.187:0/2705530842 shutdown_connections 2020-11-20T12:06:50.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.354+0000 7fe0aee1d700 1 -- 172.21.15.187:0/2705530842 wait complete. 2020-11-20T12:06:50.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.355+0000 7fe0aee1d700 1 Processor -- start 2020-11-20T12:06:50.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.356+0000 7fe0aee1d700 1 -- start start 2020-11-20T12:06:50.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.356+0000 7fe0aee1d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0a8065190 0x7fe0a80b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.356+0000 7fe0aee1d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a809ca90 0x7fe0a809e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.356+0000 7fe0aee1d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0a8143310 0x7fe0a8144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.356+0000 7fe0aee1d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a8143c70 con 0x7fe0a8065190 2020-11-20T12:06:50.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.356+0000 7fe0aee1d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a809ea70 con 0x7fe0a809ca90 2020-11-20T12:06:50.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.356+0000 7fe0aee1d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a80b1df0 con 0x7fe0a8143310 2020-11-20T12:06:50.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a809ca90 0x7fe0a809e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a809ca90 0x7fe0a809e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42564/0 (socket says 172.21.15.187:42564) 2020-11-20T12:06:50.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 -- 172.21.15.187:0/571984407 learned_addr learned my addr 172.21.15.187:0/571984407 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:50.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 -- 172.21.15.187:0/571984407 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0a8143310 msgr2=0x7fe0a8144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0a8143310 0x7fe0a8144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:50.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 -- 172.21.15.187:0/571984407 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0a8065190 msgr2=0x7fe0a80b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0a8065190 0x7fe0a80b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:50.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 -- 172.21.15.187:0/571984407 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a80b2090 con 0x7fe0a809ca90 2020-11-20T12:06:50.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.357+0000 7fe0ad61a700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a809ca90 0x7fe0a809e4a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fe0a4010000 tx=0x7fe0a4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:50.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.358+0000 7fe099ffb700 1 -- 172.21.15.187:0/571984407 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a401e460 con 0x7fe0a809ca90 2020-11-20T12:06:50.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.358+0000 7fe099ffb700 1 -- 172.21.15.187:0/571984407 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a401e5c0 con 0x7fe0a809ca90 2020-11-20T12:06:50.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.358+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a81453e0 con 0x7fe0a809ca90 2020-11-20T12:06:50.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.358+0000 7fe099ffb700 1 -- 172.21.15.187:0/571984407 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a401e940 con 0x7fe0a809ca90 2020-11-20T12:06:50.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.358+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a81456e0 con 0x7fe0a809ca90 2020-11-20T12:06:50.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.359+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a80b2220 con 0x7fe0a809ca90 2020-11-20T12:06:50.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.360+0000 7fe099ffb700 1 -- 172.21.15.187:0/571984407 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe0a4025030 con 0x7fe0a809ca90 2020-11-20T12:06:50.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.360+0000 7fe099ffb700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe080056d50 0x7fe080059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.360+0000 7fe099ffb700 1 -- 172.21.15.187:0/571984407 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe0a40719e0 con 0x7fe0a809ca90 2020-11-20T12:06:50.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.360+0000 7fe0ade1b700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe080056d50 0x7fe080059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.361+0000 7fe0ade1b700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe080056d50 0x7fe080059210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe09c0055a0 tx=0x7fe09c0077d0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:50.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.364+0000 7fe099ffb700 1 -- 172.21.15.187:0/571984407 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe0a4046d90 con 0x7fe0a809ca90 2020-11-20T12:06:50.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.551+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe0a8061a20 con 0x7fe0a809ca90 2020-11-20T12:06:50.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.552+0000 7fe099ffb700 1 -- 172.21.15.187:0/571984407 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v32) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe0a4023070 con 0x7fe0a809ca90 2020-11-20T12:06:50.556 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:06:50.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.554+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe080056d50 msgr2=0x7fe080059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.554+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe080056d50 0x7fe080059210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe09c0055a0 tx=0x7fe09c0077d0).stop 2020-11-20T12:06:50.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a809ca90 msgr2=0x7fe0a809e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a809ca90 0x7fe0a809e4a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fe0a4010000 tx=0x7fe0a4015000).stop 2020-11-20T12:06:50.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 shutdown_connections 2020-11-20T12:06:50.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe080056d50 0x7fe080059210 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0a8065190 0x7fe0a80b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:50.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0a8143310 0x7fe0a8144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:50.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 --2- 172.21.15.187:0/571984407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a809ca90 0x7fe0a809e4a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 >> 172.21.15.187:0/571984407 conn(0x7fe0a80bab30 msgr2=0x7fe0a80668d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:50.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 shutdown_connections 2020-11-20T12:06:50.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.555+0000 7fe0aee1d700 1 -- 172.21.15.187:0/571984407 wait complete. 2020-11-20T12:06:50.582 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:06:50.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.818+0000 7fd25c8a0700 1 Processor -- start 2020-11-20T12:06:50.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.818+0000 7fd25c8a0700 1 -- start start 2020-11-20T12:06:50.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.820+0000 7fd25c8a0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2580bf800 0x7fd25814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.820+0000 7fd25c8a0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2580bfbe0 0x7fd2581480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.820+0000 7fd25c8a0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25814d110 con 0x7fd25809f560 2020-11-20T12:06:50.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.820+0000 7fd25c8a0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25814cbd0 con 0x7fd2580bf800 2020-11-20T12:06:50.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.820+0000 7fd25c8a0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25814ce70 con 0x7fd2580bfbe0 2020-11-20T12:06:50.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.820+0000 7fd256d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2580bf800 0x7fd25814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd256d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2580bf800 0x7fd25814c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42568/0 (socket says 172.21.15.187:42568) 2020-11-20T12:06:50.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd256d9d700 1 -- 172.21.15.187:0/2968629739 learned_addr learned my addr 172.21.15.187:0/2968629739 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:50.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd24ffff700 1 --2- 172.21.15.187:0/2968629739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2580bfbe0 0x7fd2581480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd256d9d700 1 -- 172.21.15.187:0/2968629739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2580bfbe0 msgr2=0x7fd2581480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:50.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd256d9d700 1 --2- 172.21.15.187:0/2968629739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2580bfbe0 0x7fd2581480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd256d9d700 1 -- 172.21.15.187:0/2968629739 >> v1:172.21.15.5:6789/0 conn(0x7fd25809f560 legacy=0x7fd258060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:50.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd256d9d700 1 -- 172.21.15.187:0/2968629739 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2580a4cc0 con 0x7fd2580bf800 2020-11-20T12:06:50.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd24ffff700 1 --2- 172.21.15.187:0/2968629739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2580bfbe0 0x7fd2581480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:50.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.821+0000 7fd256d9d700 1 --2- 172.21.15.187:0/2968629739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd25814c2c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fd248006fd0 tx=0x7fd248006eb0).ready entity=mon.1 client_cookie=8cfe826dfe5259f7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:50.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25659c700 1 -- 172.21.15.187:0/2968629739 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248009920 con 0x7fd2580bf800 2020-11-20T12:06:50.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25659c700 1 -- 172.21.15.187:0/2968629739 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd248009a80 con 0x7fd2580bf800 2020-11-20T12:06:50.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25659c700 1 -- 172.21.15.187:0/2968629739 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248009e00 con 0x7fd2580bf800 2020-11-20T12:06:50.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25c8a0700 1 -- 172.21.15.187:0/2968629739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 msgr2=0x7fd25814c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/2968629739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd25814c2c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fd248006fd0 tx=0x7fd248006eb0).stop 2020-11-20T12:06:50.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25c8a0700 1 -- 172.21.15.187:0/2968629739 shutdown_connections 2020-11-20T12:06:50.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/2968629739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd25814c2c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/2968629739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2580bfbe0 0x7fd2581480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.822+0000 7fd25c8a0700 1 -- 172.21.15.187:0/2968629739 >> 172.21.15.187:0/2968629739 conn(0x7fd258068370 msgr2=0x7fd2580bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:50.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.823+0000 7fd25c8a0700 1 -- 172.21.15.187:0/2968629739 shutdown_connections 2020-11-20T12:06:50.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.823+0000 7fd25c8a0700 1 -- 172.21.15.187:0/2968629739 wait complete. 2020-11-20T12:06:50.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.824+0000 7fd25c8a0700 1 Processor -- start 2020-11-20T12:06:50.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.824+0000 7fd25c8a0700 1 -- start start 2020-11-20T12:06:50.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.825+0000 7fd25c8a0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd25809f560 0x7fd2580a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.825+0000 7fd25c8a0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd258145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.825+0000 7fd25c8a0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd25815c5b0 0x7fd258146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.825+0000 7fd25c8a0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25815ca50 con 0x7fd25809f560 2020-11-20T12:06:50.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.825+0000 7fd25c8a0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2580b5cb0 con 0x7fd2580bf800 2020-11-20T12:06:50.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.825+0000 7fd25c8a0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2580b5f50 con 0x7fd25815c5b0 2020-11-20T12:06:50.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd258145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd258145d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42578/0 (socket says 172.21.15.187:42578) 2020-11-20T12:06:50.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 -- 172.21.15.187:0/4039890189 learned_addr learned my addr 172.21.15.187:0/4039890189 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:50.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd25759e700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd25815c5b0 0x7fd258146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 -- 172.21.15.187:0/4039890189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd25815c5b0 msgr2=0x7fd258146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd25815c5b0 0x7fd258146260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:50.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 -- 172.21.15.187:0/4039890189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd25809f560 msgr2=0x7fd2580a3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:50.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd25809f560 0x7fd2580a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:50.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 -- 172.21.15.187:0/4039890189 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2580bac20 con 0x7fd2580bf800 2020-11-20T12:06:50.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.826+0000 7fd24ffff700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd258145d70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fd24000c920 tx=0x7fd240008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:50.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.827+0000 7fd24effd700 1 -- 172.21.15.187:0/4039890189 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240007710 con 0x7fd2580bf800 2020-11-20T12:06:50.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.827+0000 7fd24effd700 1 -- 172.21.15.187:0/4039890189 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240007870 con 0x7fd2580bf800 2020-11-20T12:06:50.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.827+0000 7fd24effd700 1 -- 172.21.15.187:0/4039890189 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240007bf0 con 0x7fd2580bf800 2020-11-20T12:06:50.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.827+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2580a4250 con 0x7fd2580bf800 2020-11-20T12:06:50.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.827+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2580ba600 con 0x7fd2580bf800 2020-11-20T12:06:50.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.828+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd258080050 con 0x7fd2580bf800 2020-11-20T12:06:50.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.828+0000 7fd24effd700 1 -- 172.21.15.187:0/4039890189 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd24001b460 con 0x7fd2580bf800 2020-11-20T12:06:50.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.829+0000 7fd24effd700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd234056d20 0x7fd2340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:50.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.830+0000 7fd24effd700 1 -- 172.21.15.187:0/4039890189 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd240070290 con 0x7fd2580bf800 2020-11-20T12:06:50.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.830+0000 7fd256d9d700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd234056d20 0x7fd2340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:50.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.830+0000 7fd256d9d700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd234056d20 0x7fd2340591e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fd248006ef0 tx=0x7fd248015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:50.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:50.833+0000 7fd24effd700 1 -- 172.21.15.187:0/4039890189 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd240045570 con 0x7fd2580bf800 2020-11-20T12:06:51.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.019+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd2580b6110 con 0x7fd2580bf800 2020-11-20T12:06:51.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.021+0000 7fd24effd700 1 -- 172.21.15.187:0/4039890189 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v32) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd240048b90 con 0x7fd2580bf800 2020-11-20T12:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:06:51.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd234056d20 msgr2=0x7fd2340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd234056d20 0x7fd2340591e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fd248006ef0 tx=0x7fd248015000).stop 2020-11-20T12:06:51.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 msgr2=0x7fd258145d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd258145d70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fd24000c920 tx=0x7fd240008040).stop 2020-11-20T12:06:51.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 shutdown_connections 2020-11-20T12:06:51.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd234056d20 0x7fd2340591e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd25809f560 0x7fd2580a3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:51.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd25815c5b0 0x7fd258146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 --2- 172.21.15.187:0/4039890189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2580bf800 0x7fd258145d70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 >> 172.21.15.187:0/4039890189 conn(0x7fd258068370 msgr2=0x7fd2580bcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:51.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.023+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 shutdown_connections 2020-11-20T12:06:51.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.024+0000 7fd25c8a0700 1 -- 172.21.15.187:0/4039890189 wait complete. 2020-11-20T12:06:51.050 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:06:51.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.287+0000 7fcf0359e700 1 Processor -- start 2020-11-20T12:06:51.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.287+0000 7fcf0359e700 1 -- start start 2020-11-20T12:06:51.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.289+0000 7fcf0359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcf040bf800 0x7fcf0414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.289+0000 7fcf0359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0414d110 con 0x7fcf040bfbe0 2020-11-20T12:06:51.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.289+0000 7fcf0359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0414cbd0 con 0x7fcf0409f560 2020-11-20T12:06:51.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.289+0000 7fcf0359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0414ce70 con 0x7fcf040bf800 2020-11-20T12:06:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.289+0000 7fcf02d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcf0409f560 0x7fcf040a36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39190/0 (socket says 172.21.15.187:39190) 2020-11-20T12:06:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.289+0000 7fcf02d9d700 1 -- 172.21.15.187:0/3821034961 learned_addr learned my addr 172.21.15.187:0/3821034961 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.289+0000 7fcf0259c700 1 --2- 172.21.15.187:0/3821034961 >> v2:172.21.15.187:3300/0 conn(0x7fcf040bf800 0x7fcf0414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291721082 0 0) 0x7fcf0414cbd0 con 0x7fcf0409f560 2020-11-20T12:06:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee4003240 con 0x7fcf0409f560 2020-11-20T12:06:51.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3698028583 0 0) 0x7fcee4003240 con 0x7fcf0409f560 2020-11-20T12:06:51.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 >> v2:172.21.15.187:3300/0 conn(0x7fcf040bf800 msgr2=0x7fcf0414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:51.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 --2- 172.21.15.187:0/3821034961 >> v2:172.21.15.187:3300/0 conn(0x7fcf040bf800 0x7fcf0414c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 >> v1:172.21.15.5:6789/0 conn(0x7fcf040bfbe0 legacy=0x7fcf041480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf040a4cc0 con 0x7fcf0409f560 2020-11-20T12:06:51.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcefc0018a0 con 0x7fcf0409f560 2020-11-20T12:06:51.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 >> v1:172.21.15.187:6789/0 conn(0x7fcf0409f560 legacy=0x7fcf040a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcee4003730 0x7fcee4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcee40044e0 0x7fcee4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcee4004e50 0x7fcee400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0414ce70 con 0x7fcee4004e50 2020-11-20T12:06:51.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4004260 con 0x7fcee40044e0 2020-11-20T12:06:51.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.290+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee400bc20 con 0x7fcee4003730 2020-11-20T12:06:51.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.291+0000 7fcefbfff700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcee40044e0 0x7fcee4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.291+0000 7fcefbfff700 1 -- 172.21.15.187:0/3821034961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcee4003730 msgr2=0x7fcee4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:51.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.291+0000 7fcefbfff700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcee4003730 0x7fcee4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:51.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.291+0000 7fcefbfff700 1 -- 172.21.15.187:0/3821034961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcee4004e50 msgr2=0x7fcee400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:51.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.291+0000 7fcefbfff700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcee4004e50 0x7fcee400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:51.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.291+0000 7fcefbfff700 1 -- 172.21.15.187:0/3821034961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee4003240 con 0x7fcee40044e0 2020-11-20T12:06:51.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.291+0000 7fcf02d9d700 1 -- 172.21.15.187:0/3821034961 reap_dead start 2020-11-20T12:06:51.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.291+0000 7fcefbfff700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcee40044e0 0x7fcee4004960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fceec001920 tx=0x7fceec007b90).ready entity=mon.1 client_cookie=1c668490ab7c393 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:51.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.292+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec007020 con 0x7fcee40044e0 2020-11-20T12:06:51.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.292+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec007180 con 0x7fcee40044e0 2020-11-20T12:06:51.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.292+0000 7fcf01d9b700 1 -- 172.21.15.187:0/3821034961 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec00c620 con 0x7fcee40044e0 2020-11-20T12:06:51.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.292+0000 7fcf0359e700 1 -- 172.21.15.187:0/3821034961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcee40044e0 msgr2=0x7fcee4004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.292+0000 7fcf0359e700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcee40044e0 0x7fcee4004960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fceec001920 tx=0x7fceec007b90).stop 2020-11-20T12:06:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.292+0000 7fcf0359e700 1 -- 172.21.15.187:0/3821034961 shutdown_connections 2020-11-20T12:06:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.293+0000 7fcf0359e700 1 --2- 172.21.15.187:0/3821034961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcee40044e0 0x7fcee4004960 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.293+0000 7fcf0359e700 1 -- 172.21.15.187:0/3821034961 >> 172.21.15.187:0/3821034961 conn(0x7fcf04068370 msgr2=0x7fcf040bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:51.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.293+0000 7fcf0359e700 1 -- 172.21.15.187:0/3821034961 shutdown_connections 2020-11-20T12:06:51.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.293+0000 7fcf0359e700 1 -- 172.21.15.187:0/3821034961 wait complete. 2020-11-20T12:06:51.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.294+0000 7fcf0359e700 1 Processor -- start 2020-11-20T12:06:51.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.294+0000 7fcf0359e700 1 -- start start 2020-11-20T12:06:51.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcf0359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcee40044e0 0x7fcf0415c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcf0359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04145870 0x7fcf0414b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcf0359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf04140b60 0x7fcf04140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcf0359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040b5b30 con 0x7fcf04140b60 2020-11-20T12:06:51.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcf0359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040bab10 con 0x7fcf04145870 2020-11-20T12:06:51.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcf0359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf04145cf0 con 0x7fcee40044e0 2020-11-20T12:06:51.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcefbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04145870 0x7fcf0414b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcefbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04145870 0x7fcf0414b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42596/0 (socket says 172.21.15.187:42596) 2020-11-20T12:06:51.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.295+0000 7fcefbfff700 1 -- 172.21.15.187:0/1675141721 learned_addr learned my addr 172.21.15.187:0/1675141721 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:51.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefbfff700 1 -- 172.21.15.187:0/1675141721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcee40044e0 msgr2=0x7fcf0415c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcf02d9d700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf04140b60 0x7fcf04140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefbfff700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcee40044e0 0x7fcf0415c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:51.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefbfff700 1 -- 172.21.15.187:0/1675141721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf04140b60 msgr2=0x7fcf04140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefbfff700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf04140b60 0x7fcf04140f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefbfff700 1 -- 172.21.15.187:0/1675141721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf040badb0 con 0x7fcf04145870 2020-11-20T12:06:51.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefbfff700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04145870 0x7fcf0414b2f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fceec000dd0 tx=0x7fceec015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:51.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefa7fc700 1 -- 172.21.15.187:0/1675141721 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec013aa0 con 0x7fcf04145870 2020-11-20T12:06:51.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefa7fc700 1 -- 172.21.15.187:0/1675141721 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec013c00 con 0x7fcf04145870 2020-11-20T12:06:51.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcefa7fc700 1 -- 172.21.15.187:0/1675141721 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec020440 con 0x7fcf04145870 2020-11-20T12:06:51.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.296+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf040ba420 con 0x7fcf04145870 2020-11-20T12:06:51.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.297+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0414b7e0 con 0x7fcf04145870 2020-11-20T12:06:51.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.297+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf04080050 con 0x7fcf04145870 2020-11-20T12:06:51.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.298+0000 7fcefa7fc700 1 -- 172.21.15.187:0/1675141721 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fceec026020 con 0x7fcf04145870 2020-11-20T12:06:51.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.299+0000 7fcefa7fc700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcee0056d20 0x7fcee00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.299+0000 7fcefa7fc700 1 -- 172.21.15.187:0/1675141721 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fceec071580 con 0x7fcf04145870 2020-11-20T12:06:51.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.299+0000 7fcf0259c700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcee0056d20 0x7fcee00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.300+0000 7fcf0259c700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcee0056d20 0x7fcee00591e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fcef4006a30 tx=0x7fcef4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:51.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.303+0000 7fcefa7fc700 1 -- 172.21.15.187:0/1675141721 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fceec0468c0 con 0x7fcf04145870 2020-11-20T12:06:51.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.490+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf040b5ca0 con 0x7fcf04145870 2020-11-20T12:06:51.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.491+0000 7fcefa7fc700 1 -- 172.21.15.187:0/1675141721 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v32) v1 ==== 104+0+10 (secure 0 0 0) 0x7fceec049ee0 con 0x7fcf04145870 2020-11-20T12:06:51.494 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:06:51.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcee0056d20 msgr2=0x7fcee00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcee0056d20 0x7fcee00591e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fcef4006a30 tx=0x7fcef4008040).stop 2020-11-20T12:06:51.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04145870 msgr2=0x7fcf0414b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04145870 0x7fcf0414b2f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fceec000dd0 tx=0x7fceec015000).stop 2020-11-20T12:06:51.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 shutdown_connections 2020-11-20T12:06:51.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcee0056d20 0x7fcee00591e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf04140b60 0x7fcf04140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcee40044e0 0x7fcf0415c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:51.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 --2- 172.21.15.187:0/1675141721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04145870 0x7fcf0414b2f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 >> 172.21.15.187:0/1675141721 conn(0x7fcf04068370 msgr2=0x7fcf040bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:51.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 shutdown_connections 2020-11-20T12:06:51.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.494+0000 7fcf0359e700 1 -- 172.21.15.187:0/1675141721 wait complete. 2020-11-20T12:06:51.519 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:06:51.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.763+0000 7f7e95bc6700 1 Processor -- start 2020-11-20T12:06:51.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.764+0000 7f7e95bc6700 1 -- start start 2020-11-20T12:06:51.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.765+0000 7f7e95bc6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7e900b1380 0x7f7e90150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.765+0000 7f7e95bc6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7e900b1760 0x7f7e90065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.765+0000 7f7e95bc6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e90150390 con 0x7f7e90065680 2020-11-20T12:06:51.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.765+0000 7f7e95bc6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e90150630 con 0x7f7e900b1760 2020-11-20T12:06:51.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.765+0000 7f7e95bc6700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e90068510 con 0x7f7e900b1380 2020-11-20T12:06:51.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.765+0000 7f7e953c5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7e90065680 0x7f7e90061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39212/0 (socket says 172.21.15.187:39212) 2020-11-20T12:06:51.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.765+0000 7f7e94bc4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7e900b1380 0x7f7e90150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.765+0000 7f7e953c5700 1 -- 172.21.15.187:0/1906697164 learned_addr learned my addr 172.21.15.187:0/1906697164 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:51.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.766+0000 7f7e877fe700 1 -- 172.21.15.187:0/1906697164 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643678325 0 0) 0x7f7e90150390 con 0x7f7e90065680 2020-11-20T12:06:51.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.766+0000 7f7e877fe700 1 -- 172.21.15.187:0/1906697164 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e7c003240 con 0x7f7e90065680 2020-11-20T12:06:51.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.766+0000 7f7e87fff700 1 --2- 172.21.15.187:0/1906697164 >> v2:172.21.15.5:3300/0 conn(0x7f7e900b1760 0x7f7e90065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.766+0000 7f7e94bc4700 1 -- 172.21.15.187:0/1906697164 >> v2:172.21.15.5:3300/0 conn(0x7f7e900b1760 msgr2=0x7f7e90065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:51.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.766+0000 7f7e94bc4700 1 --2- 172.21.15.187:0/1906697164 >> v2:172.21.15.5:3300/0 conn(0x7f7e900b1760 0x7f7e90065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.766+0000 7f7e877fe700 1 -- 172.21.15.187:0/1906697164 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2958222444 0 0) 0x7f7e7c003240 con 0x7f7e90065680 2020-11-20T12:06:51.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.766+0000 7f7e94bc4700 1 -- 172.21.15.187:0/1906697164 >> v1:172.21.15.187:6789/0 conn(0x7f7e90065680 legacy=0x7f7e90061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.767+0000 7f7e94bc4700 1 -- 172.21.15.187:0/1906697164 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e90150fc0 con 0x7f7e900b1380 2020-11-20T12:06:51.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.767+0000 7f7e94bc4700 1 --2- 172.21.15.187:0/1906697164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e900b1380 0x7f7e90150ad0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f7e900b85e0 tx=0x7f7e8c0091d0).ready entity=mon.1 client_cookie=97b5624fabbfc249 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:51.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.767+0000 7f7e877fe700 1 -- 172.21.15.187:0/1906697164 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e880014e0 con 0x7f7e900b1380 2020-11-20T12:06:51.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.767+0000 7f7e877fe700 1 -- 172.21.15.187:0/1906697164 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00b760 con 0x7f7e900b1380 2020-11-20T12:06:51.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.767+0000 7f7e877fe700 1 -- 172.21.15.187:0/1906697164 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00bae0 con 0x7f7e900b1380 2020-11-20T12:06:51.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.768+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1906697164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e900b1380 msgr2=0x7f7e90150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.768+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1906697164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e900b1380 0x7f7e90150ad0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f7e900b85e0 tx=0x7f7e8c0091d0).stop 2020-11-20T12:06:51.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.768+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1906697164 shutdown_connections 2020-11-20T12:06:51.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.768+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1906697164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e900b1380 0x7f7e90150ad0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.768+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1906697164 >> v2:172.21.15.5:3300/0 conn(0x7f7e900b1760 0x7f7e90065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.768+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1906697164 >> 172.21.15.187:0/1906697164 conn(0x7f7e900bab30 msgr2=0x7f7e900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:51.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.768+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1906697164 shutdown_connections 2020-11-20T12:06:51.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.768+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1906697164 wait complete. 2020-11-20T12:06:51.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.770+0000 7f7e95bc6700 1 Processor -- start 2020-11-20T12:06:51.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.770+0000 7f7e95bc6700 1 -- start start 2020-11-20T12:06:51.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e95bc6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e900b1380 0x7f7e9009a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e95bc6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e900b1760 0x7f7e9014a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e95bc6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e9014abe0 0x7f7e9015f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e95bc6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9009f010 con 0x7f7e900b1380 2020-11-20T12:06:51.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e95bc6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900a0590 con 0x7f7e9014abe0 2020-11-20T12:06:51.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e95bc6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9009ac70 con 0x7f7e900b1760 2020-11-20T12:06:51.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e953c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e9014abe0 0x7f7e9015f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e953c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e9014abe0 0x7f7e9015f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42612/0 (socket says 172.21.15.187:42612) 2020-11-20T12:06:51.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.771+0000 7f7e953c5700 1 -- 172.21.15.187:0/1429709021 learned_addr learned my addr 172.21.15.187:0/1429709021 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:51.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e94bc4700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e900b1380 0x7f7e9009a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e953c5700 1 -- 172.21.15.187:0/1429709021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e900b1760 msgr2=0x7f7e9014a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e953c5700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e900b1760 0x7f7e9014a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:51.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e953c5700 1 -- 172.21.15.187:0/1429709021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e900b1380 msgr2=0x7f7e9009a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e953c5700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e900b1380 0x7f7e9009a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e953c5700 1 -- 172.21.15.187:0/1429709021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e900b3e60 con 0x7f7e9014abe0 2020-11-20T12:06:51.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e953c5700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e9014abe0 0x7f7e9015f9f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f7e8800cae0 tx=0x7f7e88001a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:51.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e84ff9700 1 -- 172.21.15.187:0/1429709021 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88007170 con 0x7f7e9014abe0 2020-11-20T12:06:51.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e84ff9700 1 -- 172.21.15.187:0/1429709021 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e880072d0 con 0x7f7e9014abe0 2020-11-20T12:06:51.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9009eba0 con 0x7f7e9014abe0 2020-11-20T12:06:51.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e84ff9700 1 -- 172.21.15.187:0/1429709021 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88004920 con 0x7f7e9014abe0 2020-11-20T12:06:51.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.772+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e900b47a0 con 0x7f7e9014abe0 2020-11-20T12:06:51.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.774+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e90061a20 con 0x7f7e9014abe0 2020-11-20T12:06:51.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.774+0000 7f7e84ff9700 1 -- 172.21.15.187:0/1429709021 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7e88010070 con 0x7f7e9014abe0 2020-11-20T12:06:51.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.775+0000 7f7e84ff9700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e68056d20 0x7f7e680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:51.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.775+0000 7f7e84ff9700 1 -- 172.21.15.187:0/1429709021 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7e8806fb40 con 0x7f7e9014abe0 2020-11-20T12:06:51.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.775+0000 7f7e94bc4700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e68056d20 0x7f7e680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:51.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.776+0000 7f7e94bc4700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e68056d20 0x7f7e680591e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7e8c009210 tx=0x7f7e8c016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:51.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.779+0000 7f7e84ff9700 1 -- 172.21.15.187:0/1429709021 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7e88044e80 con 0x7f7e9014abe0 2020-11-20T12:06:51.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.963+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e900b83c0 con 0x7f7e9014abe0 2020-11-20T12:06:51.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.964+0000 7f7e84ff9700 1 -- 172.21.15.187:0/1429709021 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v32) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7e880484a0 con 0x7f7e9014abe0 2020-11-20T12:06:51.968 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:06:51.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e68056d20 msgr2=0x7f7e680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e68056d20 0x7f7e680591e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7e8c009210 tx=0x7f7e8c016000).stop 2020-11-20T12:06:51.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e9014abe0 msgr2=0x7f7e9015f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:51.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e9014abe0 0x7f7e9015f9f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f7e8800cae0 tx=0x7f7e88001a40).stop 2020-11-20T12:06:51.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 shutdown_connections 2020-11-20T12:06:51.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e68056d20 0x7f7e680591e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e900b1380 0x7f7e9009a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e9014abe0 0x7f7e9015f9f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:51.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 --2- 172.21.15.187:0/1429709021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e900b1760 0x7f7e9014a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:51.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 >> 172.21.15.187:0/1429709021 conn(0x7f7e900bab30 msgr2=0x7f7e9014b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:51.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 shutdown_connections 2020-11-20T12:06:51.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:51.967+0000 7f7e95bc6700 1 -- 172.21.15.187:0/1429709021 wait complete. 2020-11-20T12:06:51.994 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:52.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.233+0000 7f07e7fff700 1 Processor -- start 2020-11-20T12:06:52.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.233+0000 7f07e7fff700 1 -- start start 2020-11-20T12:06:52.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.234+0000 7f07e7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e80b6650 0x7f07e80afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.234+0000 7f07e7fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e80b1300 con 0x7f07e8066f70 2020-11-20T12:06:52.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.234+0000 7f07e7fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e80b15a0 con 0x7f07e81484d0 2020-11-20T12:06:52.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.234+0000 7f07e7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8067730 con 0x7f07e80b6650 2020-11-20T12:06:52.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.235+0000 7f07e6ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f07e81484d0 0x7f07e8061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39222/0 (socket says 172.21.15.187:39222) 2020-11-20T12:06:52.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.235+0000 7f07e6ffd700 1 -- 172.21.15.187:0/2586919551 learned_addr learned my addr 172.21.15.187:0/2586919551 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:52.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.235+0000 7f07de7fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e80b6650 0x7f07e80afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.236+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539451272 0 0) 0x7f07e80b15a0 con 0x7f07e81484d0 2020-11-20T12:06:52.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.236+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07c8003240 con 0x7f07e81484d0 2020-11-20T12:06:52.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.236+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 693300151 0 0) 0x7f07c8003240 con 0x7f07e81484d0 2020-11-20T12:06:52.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.236+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e80b6650 msgr2=0x7f07e80afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:52.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.236+0000 7f07e67fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e80b6650 0x7f07e80afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 >> v1:172.21.15.5:6789/0 conn(0x7f07e8066f70 legacy=0x7f07e80aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e80b1840 con 0x7f07e81484d0 2020-11-20T12:06:52.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f07d8001520 con 0x7f07e81484d0 2020-11-20T12:06:52.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 >> v1:172.21.15.187:6789/0 conn(0x7f07e81484d0 legacy=0x7f07e8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e80b6650 0x7f07c8003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07c8003c40 0x7f07c80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07c80049d0 0x7f07c8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8067730 con 0x7f07c80049d0 2020-11-20T12:06:52.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80041d0 con 0x7f07c8003c40 2020-11-20T12:06:52.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c800d940 con 0x7f07e80b6650 2020-11-20T12:06:52.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07de7fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07c8003c40 0x7f07c80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.237+0000 7f07e6ffd700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e80b6650 0x7f07c8003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07de7fc700 1 -- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e80b6650 msgr2=0x7f07c8003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:52.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07de7fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e80b6650 0x7f07c8003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07de7fc700 1 -- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07c80049d0 msgr2=0x7f07c8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:52.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07de7fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07c80049d0 0x7f07c8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:52.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07de7fc700 1 -- 172.21.15.187:0/2586919551 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c8003240 con 0x7f07c8003c40 2020-11-20T12:06:52.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07de7fc700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07c8003c40 0x7f07c80044e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f07d000bc00 tx=0x7f07d0006df0).ready entity=mon.1 client_cookie=c35f450f39e3fc3a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d000b310 con 0x7f07c8003c40 2020-11-20T12:06:52.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d000eae0 con 0x7f07c8003c40 2020-11-20T12:06:52.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.238+0000 7f07e67fc700 1 -- 172.21.15.187:0/2586919551 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d000ee60 con 0x7f07c8003c40 2020-11-20T12:06:52.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.239+0000 7f07e7fff700 1 -- 172.21.15.187:0/2586919551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07c8003c40 msgr2=0x7f07c80044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.239+0000 7f07e7fff700 1 --2- 172.21.15.187:0/2586919551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07c8003c40 0x7f07c80044e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f07d000bc00 tx=0x7f07d0006df0).stop 2020-11-20T12:06:52.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.239+0000 7f07e77fe700 1 -- 172.21.15.187:0/2586919551 reap_dead start 2020-11-20T12:06:52.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.239+0000 7f07e7fff700 1 -- 172.21.15.187:0/2586919551 shutdown_connections 2020-11-20T12:06:52.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.239+0000 7f07e7fff700 1 -- 172.21.15.187:0/2586919551 >> 172.21.15.187:0/2586919551 conn(0x7f07e80bab30 msgr2=0x7f07e80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:52.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.239+0000 7f07e7fff700 1 -- 172.21.15.187:0/2586919551 shutdown_connections 2020-11-20T12:06:52.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.239+0000 7f07e7fff700 1 -- 172.21.15.187:0/2586919551 wait complete. 2020-11-20T12:06:52.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.241+0000 7f07e7fff700 1 Processor -- start 2020-11-20T12:06:52.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.242+0000 7f07e7fff700 1 -- start start 2020-11-20T12:06:52.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07e7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07e809b0c0 0x7f07e809b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07e7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8147710 0x7f07e809a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07e7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e809ba40 0x7f07e809bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07e7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80b1e50 con 0x7f07e809b0c0 2020-11-20T12:06:52.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07e7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80b3f70 con 0x7f07e8147710 2020-11-20T12:06:52.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07e7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80b2040 con 0x7f07e809ba40 2020-11-20T12:06:52.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07de7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8147710 0x7f07e809a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07de7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8147710 0x7f07e809a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42630/0 (socket says 172.21.15.187:42630) 2020-11-20T12:06:52.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.243+0000 7f07de7fc700 1 -- 172.21.15.187:0/2113337815 learned_addr learned my addr 172.21.15.187:0/2113337815 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:52.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07e77fe700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e809ba40 0x7f07e809bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07e6ffd700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07e809b0c0 0x7f07e809b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07de7fc700 1 -- 172.21.15.187:0/2113337815 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e809ba40 msgr2=0x7f07e809bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07de7fc700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e809ba40 0x7f07e809bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07de7fc700 1 -- 172.21.15.187:0/2113337815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07e809b0c0 msgr2=0x7f07e809b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07de7fc700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07e809b0c0 0x7f07e809b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07de7fc700 1 -- 172.21.15.187:0/2113337815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e809ab60 con 0x7f07e8147710 2020-11-20T12:06:52.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07de7fc700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8147710 0x7f07e809a670 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f07c8004360 tx=0x7f07d0019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07df7fe700 1 -- 172.21.15.187:0/2113337815 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d000b390 con 0x7f07e8147710 2020-11-20T12:06:52.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07df7fe700 1 -- 172.21.15.187:0/2113337815 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0021070 con 0x7f07e8147710 2020-11-20T12:06:52.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07df7fe700 1 -- 172.21.15.187:0/2113337815 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d001d620 con 0x7f07e8147710 2020-11-20T12:06:52.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.244+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e80b3bf0 con 0x7f07e8147710 2020-11-20T12:06:52.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.245+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e80b24e0 con 0x7f07e8147710 2020-11-20T12:06:52.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.246+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e80b3de0 con 0x7f07e8147710 2020-11-20T12:06:52.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.246+0000 7f07df7fe700 1 -- 172.21.15.187:0/2113337815 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f07d001d9d0 con 0x7f07e8147710 2020-11-20T12:06:52.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.247+0000 7f07df7fe700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07c4056d20 0x7f07c40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.247+0000 7f07df7fe700 1 -- 172.21.15.187:0/2113337815 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f07d0071e50 con 0x7f07e8147710 2020-11-20T12:06:52.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.247+0000 7f07e6ffd700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07c4056d20 0x7f07c40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.248+0000 7f07e6ffd700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07c4056d20 0x7f07c40591e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f07d80013f0 tx=0x7f07d8001db0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.252+0000 7f07df7fe700 1 -- 172.21.15.187:0/2113337815 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f07d0047360 con 0x7f07e8147710 2020-11-20T12:06:52.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.497+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07e8061a20 con 0x7f07e8147710 2020-11-20T12:06:52.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.498+0000 7f07df7fe700 1 -- 172.21.15.187:0/2113337815 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2130 (secure 0 0 0) 0x7f07d004a940 con 0x7f07e8147710 2020-11-20T12:06:52.502 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:52.502 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4505,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/234452751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":234452751},{"type":"v1","addr":"172.21.15.5:6833","nonce":234452751}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:06:11.574607+0000","modified":"2020-11-20T12:06:14.058447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4966":{"gid":4966,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/2376356363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2376356363},{"type":"v1","addr":"172.21.15.187:6835","nonce":2376356363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2020-11-20T12:06:52.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07c4056d20 msgr2=0x7f07c40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07c4056d20 0x7f07c40591e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f07d80013f0 tx=0x7f07d8001db0).stop 2020-11-20T12:06:52.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8147710 msgr2=0x7f07e809a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8147710 0x7f07e809a670 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f07c8004360 tx=0x7f07d0019000).stop 2020-11-20T12:06:52.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 shutdown_connections 2020-11-20T12:06:52.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f07c4056d20 0x7f07c40591e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07e809b0c0 0x7f07e809b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07e809ba40 0x7f07e809bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 --2- 172.21.15.187:0/2113337815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8147710 0x7f07e809a670 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.500+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 >> 172.21.15.187:0/2113337815 conn(0x7f07e80bab30 msgr2=0x7f07e80afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:52.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.501+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 shutdown_connections 2020-11-20T12:06:52.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.501+0000 7f07e7fff700 1 -- 172.21.15.187:0/2113337815 wait complete. 2020-11-20T12:06:52.507 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 13 2020-11-20T12:06:52.525 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:06:52.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.754+0000 7f6651bde700 1 Processor -- start 2020-11-20T12:06:52.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.754+0000 7f6651bde700 1 -- start start 2020-11-20T12:06:52.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.755+0000 7f6651bde700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f664c065190 0x7f664c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.755+0000 7f6651bde700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 0x7f664c09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.755+0000 7f6651bde700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0b13e0 con 0x7f664c065f90 2020-11-20T12:06:52.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.755+0000 7f6651bde700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c14ab30 con 0x7f664c065190 2020-11-20T12:06:52.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.755+0000 7f6651bde700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664c062c10 con 0x7f664c09ca90 2020-11-20T12:06:52.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.756+0000 7f6643fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f664c065f90 0x7f664c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39246/0 (socket says 172.21.15.187:39246) 2020-11-20T12:06:52.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.756+0000 7f6643fff700 1 -- 172.21.15.187:0/3165662819 learned_addr learned my addr 172.21.15.187:0/3165662819 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:52.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.756+0000 7f6650bdc700 1 --2- 172.21.15.187:0/3165662819 >> v2:172.21.15.5:3300/0 conn(0x7f664c065190 0x7f664c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.756+0000 7f66513dd700 1 --2- 172.21.15.187:0/3165662819 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 0x7f664c09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.757+0000 7f66437fe700 1 -- 172.21.15.187:0/3165662819 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900305175 0 0) 0x7f664c0b13e0 con 0x7f664c065f90 2020-11-20T12:06:52.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.757+0000 7f66437fe700 1 -- 172.21.15.187:0/3165662819 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6638003240 con 0x7f664c065f90 2020-11-20T12:06:52.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.758+0000 7f6650bdc700 1 -- 172.21.15.187:0/3165662819 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 msgr2=0x7f664c09cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:52.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.758+0000 7f6650bdc700 1 --2- 172.21.15.187:0/3165662819 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 0x7f664c09cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.758+0000 7f6650bdc700 1 -- 172.21.15.187:0/3165662819 >> v1:172.21.15.187:6789/0 conn(0x7f664c065f90 legacy=0x7f664c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.758+0000 7f66437fe700 1 -- 172.21.15.187:0/3165662819 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3484545583 0 0) 0x7f6638003240 con 0x7f664c065f90 2020-11-20T12:06:52.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.758+0000 7f6650bdc700 1 -- 172.21.15.187:0/3165662819 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0b1640 con 0x7f664c065190 2020-11-20T12:06:52.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.758+0000 7f6650bdc700 1 --2- 172.21.15.187:0/3165662819 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065190 0x7f664c065570 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f664800b9a0 tx=0x7f6648006df0).ready entity=mon.0 client_cookie=202e07875b28e654 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.759+0000 7f66437fe700 1 -- 172.21.15.187:0/3165662819 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648009710 con 0x7f664c065190 2020-11-20T12:06:52.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.759+0000 7f66437fe700 1 -- 172.21.15.187:0/3165662819 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648009870 con 0x7f664c065190 2020-11-20T12:06:52.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.759+0000 7f66437fe700 1 -- 172.21.15.187:0/3165662819 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648009bf0 con 0x7f664c065190 2020-11-20T12:06:52.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.760+0000 7f6651bde700 1 -- 172.21.15.187:0/3165662819 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065190 msgr2=0x7f664c065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.760+0000 7f6651bde700 1 --2- 172.21.15.187:0/3165662819 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065190 0x7f664c065570 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f664800b9a0 tx=0x7f6648006df0).stop 2020-11-20T12:06:52.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.760+0000 7f6651bde700 1 -- 172.21.15.187:0/3165662819 shutdown_connections 2020-11-20T12:06:52.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.760+0000 7f6651bde700 1 --2- 172.21.15.187:0/3165662819 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 0x7f664c09cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.760+0000 7f6651bde700 1 --2- 172.21.15.187:0/3165662819 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065190 0x7f664c065570 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.760+0000 7f6651bde700 1 -- 172.21.15.187:0/3165662819 >> 172.21.15.187:0/3165662819 conn(0x7f664c0bab30 msgr2=0x7f664c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:52.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.760+0000 7f6651bde700 1 -- 172.21.15.187:0/3165662819 shutdown_connections 2020-11-20T12:06:52.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.760+0000 7f6651bde700 1 -- 172.21.15.187:0/3165662819 wait complete. 2020-11-20T12:06:52.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.762+0000 7f6651bde700 1 Processor -- start 2020-11-20T12:06:52.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.762+0000 7f6651bde700 1 -- start start 2020-11-20T12:06:52.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f6651bde700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065f90 0x7f664c142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f6651bde700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 0x7f664c1432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f6651bde700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f664c149800 0x7f664c159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f6651bde700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c148b00 con 0x7f664c065f90 2020-11-20T12:06:52.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f6651bde700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c148d60 con 0x7f664c149800 2020-11-20T12:06:52.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f6651bde700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664c149000 con 0x7f664c09ca90 2020-11-20T12:06:52.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f66513dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f664c149800 0x7f664c159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f66513dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f664c149800 0x7f664c159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42644/0 (socket says 172.21.15.187:42644) 2020-11-20T12:06:52.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f66513dd700 1 -- 172.21.15.187:0/2098633385 learned_addr learned my addr 172.21.15.187:0/2098633385 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:52.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f66513dd700 1 -- 172.21.15.187:0/2098633385 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 msgr2=0x7f664c1432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f66513dd700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 0x7f664c1432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:52.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.763+0000 7f6650bdc700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065f90 0x7f664c142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f66513dd700 1 -- 172.21.15.187:0/2098633385 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065f90 msgr2=0x7f664c142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f66513dd700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065f90 0x7f664c142e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f66513dd700 1 -- 172.21.15.187:0/2098633385 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c15a470 con 0x7f664c149800 2020-11-20T12:06:52.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f66513dd700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f664c149800 0x7f664c159f80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f664400ed20 tx=0x7f66440091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f6640ff9700 1 -- 172.21.15.187:0/2098633385 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664400cbb0 con 0x7f664c149800 2020-11-20T12:06:52.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f6640ff9700 1 -- 172.21.15.187:0/2098633385 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664400cd10 con 0x7f664c149800 2020-11-20T12:06:52.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c15a660 con 0x7f664c149800 2020-11-20T12:06:52.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f6640ff9700 1 -- 172.21.15.187:0/2098633385 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6644012620 con 0x7f664c149800 2020-11-20T12:06:52.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.764+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c15aad0 con 0x7f664c149800 2020-11-20T12:06:52.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.765+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c14a290 con 0x7f664c149800 2020-11-20T12:06:52.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.766+0000 7f6640ff9700 1 -- 172.21.15.187:0/2098633385 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6644022070 con 0x7f664c149800 2020-11-20T12:06:52.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.766+0000 7f6640ff9700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6624056d20 0x7f66240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.767+0000 7f6640ff9700 1 -- 172.21.15.187:0/2098633385 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f66440707c0 con 0x7f664c149800 2020-11-20T12:06:52.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.767+0000 7f6650bdc700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6624056d20 0x7f66240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.767+0000 7f6650bdc700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6624056d20 0x7f66240591e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f664c063480 tx=0x7f6648015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.770+0000 7f6640ff9700 1 -- 172.21.15.187:0/2098633385 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6644045b90 con 0x7f664c149800 2020-11-20T12:06:52.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.956+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f664c0b97b0 con 0x7f664c149800 2020-11-20T12:06:52.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.958+0000 7f6640ff9700 1 -- 172.21.15.187:0/2098633385 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f6644016020 con 0x7f664c149800 2020-11-20T12:06:52.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6624056d20 msgr2=0x7f66240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6624056d20 0x7f66240591e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f664c063480 tx=0x7f6648015000).stop 2020-11-20T12:06:52.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f664c149800 msgr2=0x7f664c159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f664c149800 0x7f664c159f80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f664400ed20 tx=0x7f66440091d0).stop 2020-11-20T12:06:52.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 shutdown_connections 2020-11-20T12:06:52.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6624056d20 0x7f66240591e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f664c065f90 0x7f664c142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f664c149800 0x7f664c159f80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 --2- 172.21.15.187:0/2098633385 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f664c09ca90 0x7f664c1432f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:52.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.960+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 >> 172.21.15.187:0/2098633385 conn(0x7f664c0bab30 msgr2=0x7f664c09bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:52.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.961+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 shutdown_connections 2020-11-20T12:06:52.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:52.961+0000 7f6651bde700 1 -- 172.21.15.187:0/2098633385 wait complete. 2020-11-20T12:06:52.972 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:52.972 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":32,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:06:12.578719+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:06:09.312569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2020-11-20T12:06:10.581583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:06:52.986 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:06:52.986 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:06:53.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.201+0000 7f858e0b8700 1 Processor -- start 2020-11-20T12:06:53.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.201+0000 7f858e0b8700 1 -- start start 2020-11-20T12:06:53.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.202+0000 7f858e0b8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8588066f70 0x7f85880b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:53.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.202+0000 7f858e0b8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85880b1300 con 0x7f85880b6650 2020-11-20T12:06:53.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.202+0000 7f858e0b8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85880b15a0 con 0x7f85881484d0 2020-11-20T12:06:53.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.202+0000 7f858e0b8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8588067730 con 0x7f8588066f70 2020-11-20T12:06:53.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.202+0000 7f858d0b6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f85881484d0 0x7f8588061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39256/0 (socket says 172.21.15.187:39256) 2020-11-20T12:06:53.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.202+0000 7f858d0b6700 1 -- 172.21.15.187:0/40040559 learned_addr learned my addr 172.21.15.187:0/40040559 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:53.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.203+0000 7f858d8b7700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8588066f70 0x7f85880b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:53.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.203+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928905195 0 0) 0x7f85880b15a0 con 0x7f85881484d0 2020-11-20T12:06:53.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.203+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8570003240 con 0x7f85881484d0 2020-11-20T12:06:53.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.203+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 259356922 0 0) 0x7f8570003240 con 0x7f85881484d0 2020-11-20T12:06:53.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.203+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8588066f70 msgr2=0x7f85880b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:53.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8588066f70 0x7f85880b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:53.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 >> v1:172.21.15.5:6789/0 conn(0x7f85880b6650 legacy=0x7f85880afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:53.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85880b1840 con 0x7f85881484d0 2020-11-20T12:06:53.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8584001520 con 0x7f85881484d0 2020-11-20T12:06:53.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 >> v1:172.21.15.187:6789/0 conn(0x7f85881484d0 legacy=0x7f8588061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:53.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8588066f70 0x7f8570003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:53.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8570003c40 0x7f85700044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:53.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85700049d0 0x7f8570004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:53.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8588067730 con 0x7f85700049d0 2020-11-20T12:06:53.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85700041d0 con 0x7f8570003c40 2020-11-20T12:06:53.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857000b730 con 0x7f8588066f70 2020-11-20T12:06:53.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.204+0000 7f858c8b5700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8570003c40 0x7f85700044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:53.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.205+0000 7f858d0b6700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8588066f70 0x7f8570003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:53.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.205+0000 7f858d8b7700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85700049d0 0x7f8570004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:53.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.205+0000 7f858c8b5700 1 -- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8588066f70 msgr2=0x7f8570003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:53.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.205+0000 7f858c8b5700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8588066f70 0x7f8570003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:53.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.205+0000 7f858c8b5700 1 -- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85700049d0 msgr2=0x7f8570004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:53.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.205+0000 7f858c8b5700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85700049d0 0x7f8570004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:53.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.205+0000 7f858c8b5700 1 -- 172.21.15.187:0/40040559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8570003240 con 0x7f8570003c40 2020-11-20T12:06:53.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.205+0000 7f858c8b5700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8570003c40 0x7f85700044e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f8578001920 tx=0x7f8578007b90).ready entity=mon.1 client_cookie=e679cfe8749dd141 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:53.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8578007020 con 0x7f8570003c40 2020-11-20T12:06:53.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8578007180 con 0x7f8570003c40 2020-11-20T12:06:53.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f8577fff700 1 -- 172.21.15.187:0/40040559 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857800c5c0 con 0x7f8570003c40 2020-11-20T12:06:53.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f858e0b8700 1 -- 172.21.15.187:0/40040559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8570003c40 msgr2=0x7f85700044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:53.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f858e0b8700 1 --2- 172.21.15.187:0/40040559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8570003c40 0x7f85700044e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f8578001920 tx=0x7f8578007b90).stop 2020-11-20T12:06:53.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f858d8b7700 1 -- 172.21.15.187:0/40040559 reap_dead start 2020-11-20T12:06:53.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f858e0b8700 1 -- 172.21.15.187:0/40040559 shutdown_connections 2020-11-20T12:06:53.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f858e0b8700 1 -- 172.21.15.187:0/40040559 >> 172.21.15.187:0/40040559 conn(0x7f85880bab30 msgr2=0x7f85880baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:53.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f858e0b8700 1 -- 172.21.15.187:0/40040559 shutdown_connections 2020-11-20T12:06:53.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.206+0000 7f858e0b8700 1 -- 172.21.15.187:0/40040559 wait complete. 2020-11-20T12:06:53.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.208+0000 7f858e0b8700 1 Processor -- start 2020-11-20T12:06:53.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.208+0000 7f858e0b8700 1 -- start start 2020-11-20T12:06:53.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858e0b8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8588147200 0x7f858809a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:53.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858e0b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85881475e0 0x7f858809fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:53.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858e0b8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858815c3a0 0x7f858809ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:53.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858e0b8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85880a0010 con 0x7f8588147200 2020-11-20T12:06:53.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858e0b8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85880a02a0 con 0x7f85881475e0 2020-11-20T12:06:53.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858e0b8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85880a0540 con 0x7f858815c3a0 2020-11-20T12:06:53.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858c8b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85881475e0 0x7f858809fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:53.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858c8b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85881475e0 0x7f858809fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42664/0 (socket says 172.21.15.187:42664) 2020-11-20T12:06:53.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858c8b5700 1 -- 172.21.15.187:0/1516204266 learned_addr learned my addr 172.21.15.187:0/1516204266 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:53.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858c8b5700 1 -- 172.21.15.187:0/1516204266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858815c3a0 msgr2=0x7f858809ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:53.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858c8b5700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858815c3a0 0x7f858809ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:53.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858c8b5700 1 -- 172.21.15.187:0/1516204266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8588147200 msgr2=0x7f858809a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:53.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858c8b5700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8588147200 0x7f858809a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:53.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.209+0000 7f858c8b5700 1 -- 172.21.15.187:0/1516204266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858809d2f0 con 0x7f85881475e0 2020-11-20T12:06:53.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.210+0000 7f858c8b5700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85881475e0 0x7f858809fb20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8578000fd0 tx=0x7f8578015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:53.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.210+0000 7f85757fa700 1 -- 172.21.15.187:0/1516204266 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8578013a10 con 0x7f85881475e0 2020-11-20T12:06:53.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.210+0000 7f85757fa700 1 -- 172.21.15.187:0/1516204266 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8578013b90 con 0x7f85881475e0 2020-11-20T12:06:53.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.210+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858809d540 con 0x7f85881475e0 2020-11-20T12:06:53.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.210+0000 7f85757fa700 1 -- 172.21.15.187:0/1516204266 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8578020440 con 0x7f85881475e0 2020-11-20T12:06:53.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.210+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85880b2740 con 0x7f85881475e0 2020-11-20T12:06:53.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.211+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85880a0700 con 0x7f85881475e0 2020-11-20T12:06:53.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.212+0000 7f85757fa700 1 -- 172.21.15.187:0/1516204266 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8578026020 con 0x7f85881475e0 2020-11-20T12:06:53.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.212+0000 7f85757fa700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8560056d20 0x7f85600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:53.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.212+0000 7f85757fa700 1 -- 172.21.15.187:0/1516204266 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8578071430 con 0x7f85881475e0 2020-11-20T12:06:53.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.213+0000 7f858d0b6700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8560056d20 0x7f85600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:53.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.213+0000 7f858d0b6700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8560056d20 0x7f85600591e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8584001650 tx=0x7f8584000e90).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:53.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.216+0000 7f85757fa700 1 -- 172.21.15.187:0/1516204266 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8578046930 con 0x7f85881475e0 2020-11-20T12:06:53.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:53.462+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8588061a20 con 0x7f85881475e0 2020-11-20T12:06:54.184 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:06:54.180+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:06:54.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.197+0000 7f85757fa700 1 -- 172.21.15.187:0/1516204266 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v14) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8578049f50 con 0x7f85881475e0 2020-11-20T12:06:54.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.200+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8560056d20 msgr2=0x7f85600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.200+0000 7f858e0b8700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8560056d20 0x7f85600591e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8584001650 tx=0x7f8584000e90).stop 2020-11-20T12:06:54.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85881475e0 msgr2=0x7f858809fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85881475e0 0x7f858809fb20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8578000fd0 tx=0x7f8578015000).stop 2020-11-20T12:06:54.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 shutdown_connections 2020-11-20T12:06:54.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8560056d20 0x7f85600591e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8588147200 0x7f858809a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:54.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858815c3a0 0x7f858809ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:54.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 --2- 172.21.15.187:0/1516204266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85881475e0 0x7f858809fb20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 >> 172.21.15.187:0/1516204266 conn(0x7f85880bab30 msgr2=0x7f85880b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:54.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 shutdown_connections 2020-11-20T12:06:54.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.201+0000 7f858e0b8700 1 -- 172.21.15.187:0/1516204266 wait complete. 2020-11-20T12:06:54.209 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:06:54.231 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:06:54.319 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:06:54.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.465+0000 7faae1ace700 1 Processor -- start 2020-11-20T12:06:54.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.465+0000 7faae1ace700 1 -- start start 2020-11-20T12:06:54.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.467+0000 7faae1ace700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0bf800 0x7faadc14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.467+0000 7faae1ace700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faadc09f560 0x7faadc0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.467+0000 7faae1ace700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc14d110 con 0x7faadc0bfbe0 2020-11-20T12:06:54.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.467+0000 7faae1ace700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faadc14cbd0 con 0x7faadc09f560 2020-11-20T12:06:54.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.467+0000 7faae1ace700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faadc14ce70 con 0x7faadc0bf800 2020-11-20T12:06:54.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.467+0000 7faae0acc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0bf800 0x7faadc14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.468+0000 7faae0acc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0bf800 0x7faadc14c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:43570/0 (socket says 172.21.15.187:43570) 2020-11-20T12:06:54.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.468+0000 7faae0acc700 1 -- 172.21.15.187:0/3368409573 learned_addr learned my addr 172.21.15.187:0/3368409573 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:54.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.467+0000 7faae12cd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faadc09f560 0x7faadc0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.468+0000 7faae0acc700 1 -- 172.21.15.187:0/3368409573 >> v2:172.21.15.187:3300/0 conn(0x7faadc09f560 msgr2=0x7faadc0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:54.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.468+0000 7faae0acc700 1 --2- 172.21.15.187:0/3368409573 >> v2:172.21.15.187:3300/0 conn(0x7faadc09f560 0x7faadc0a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.468+0000 7faae0acc700 1 -- 172.21.15.187:0/3368409573 >> v1:172.21.15.5:6789/0 conn(0x7faadc0bfbe0 legacy=0x7faadc1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.468+0000 7faae0acc700 1 -- 172.21.15.187:0/3368409573 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc0a4cc0 con 0x7faadc0bf800 2020-11-20T12:06:54.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.469+0000 7faae0acc700 1 --2- 172.21.15.187:0/3368409573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0bf800 0x7faadc14c2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7faad8006fd0 tx=0x7faad8006eb0).ready entity=mon.2 client_cookie=12d44dbf9527ccd2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.469+0000 7faad37fe700 1 -- 172.21.15.187:0/3368409573 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8009960 con 0x7faadc0bf800 2020-11-20T12:06:54.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faad37fe700 1 -- 172.21.15.187:0/3368409573 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad8009ac0 con 0x7faadc0bf800 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faad37fe700 1 -- 172.21.15.187:0/3368409573 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8009e40 con 0x7faadc0bf800 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faae1ace700 1 -- 172.21.15.187:0/3368409573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0bf800 msgr2=0x7faadc14c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faae1ace700 1 --2- 172.21.15.187:0/3368409573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0bf800 0x7faadc14c2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7faad8006fd0 tx=0x7faad8006eb0).stop 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faae1ace700 1 -- 172.21.15.187:0/3368409573 shutdown_connections 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faae1ace700 1 --2- 172.21.15.187:0/3368409573 >> v2:172.21.15.187:3300/0 conn(0x7faadc09f560 0x7faadc0a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faae1ace700 1 --2- 172.21.15.187:0/3368409573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0bf800 0x7faadc14c2c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faae1ace700 1 -- 172.21.15.187:0/3368409573 >> 172.21.15.187:0/3368409573 conn(0x7faadc068370 msgr2=0x7faadc0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:54.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faae1ace700 1 -- 172.21.15.187:0/3368409573 shutdown_connections 2020-11-20T12:06:54.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.470+0000 7faae1ace700 1 -- 172.21.15.187:0/3368409573 wait complete. 2020-11-20T12:06:54.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.472+0000 7faae1ace700 1 Processor -- start 2020-11-20T12:06:54.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.472+0000 7faae1ace700 1 -- start start 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.473+0000 7faae1ace700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faadc09f560 0x7faadc159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.473+0000 7faae1ace700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faadc0b3e90 0x7faadc0ba880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.473+0000 7faae1ace700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0b46e0 0x7faadc0b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.473+0000 7faae1ace700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0b4310 con 0x7faadc09f560 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.473+0000 7faae1ace700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0b5050 con 0x7faadc0b3e90 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.473+0000 7faae1ace700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faadc14c210 con 0x7faadc0b46e0 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.473+0000 7faad3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faadc0b3e90 0x7faadc0ba880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faad3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faadc0b3e90 0x7faadc0ba880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42690/0 (socket says 172.21.15.187:42690) 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faad3fff700 1 -- 172.21.15.187:0/815701816 learned_addr learned my addr 172.21.15.187:0/815701816 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faae0acc700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faadc09f560 0x7faadc159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faae12cd700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0b46e0 0x7faadc0b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faad3fff700 1 -- 172.21.15.187:0/815701816 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0b46e0 msgr2=0x7faadc0b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faad3fff700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0b46e0 0x7faadc0b4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faad3fff700 1 -- 172.21.15.187:0/815701816 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faadc09f560 msgr2=0x7faadc159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faad3fff700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faadc09f560 0x7faadc159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faad3fff700 1 -- 172.21.15.187:0/815701816 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc14c4b0 con 0x7faadc0b3e90 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.474+0000 7faad3fff700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faadc0b3e90 0x7faadc0ba880 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7faacc00c710 tx=0x7faacc007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.475+0000 7faad0ff9700 1 -- 172.21.15.187:0/815701816 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc007020 con 0x7faadc0b3e90 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.475+0000 7faad0ff9700 1 -- 172.21.15.187:0/815701816 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faacc007180 con 0x7faadc0b3e90 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.475+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc0b6ef0 con 0x7faadc0b3e90 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.475+0000 7faad0ff9700 1 -- 172.21.15.187:0/815701816 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc0064f0 con 0x7faadc0b3e90 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.475+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc14be70 con 0x7faadc0b3e90 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.476+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc080050 con 0x7faadc0b3e90 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.476+0000 7faad0ff9700 1 -- 172.21.15.187:0/815701816 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faacc010030 con 0x7faadc0b3e90 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.477+0000 7faad0ff9700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faab4056d20 0x7faab40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.477+0000 7faad0ff9700 1 -- 172.21.15.187:0/815701816 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6159+0+0 (secure 0 0 0) 0x7faacc06f750 con 0x7faadc0b3e90 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.477+0000 7faae0acc700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faab4056d20 0x7faab40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.478+0000 7faae0acc700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faab4056d20 0x7faab40591e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7faadc068780 tx=0x7faad8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.481+0000 7faad0ff9700 1 -- 172.21.15.187:0/815701816 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faacc044b50 con 0x7faadc0b3e90 2020-11-20T12:06:54.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:54.727+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7faadc14c7b0 con 0x7faadc0b3e90 2020-11-20T12:06:55.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.480+0000 7faad0ff9700 1 -- 172.21.15.187:0/815701816 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7faacc048170 con 0x7faadc0b3e90 2020-11-20T12:06:55.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.484+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faab4056d20 msgr2=0x7faab40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.484+0000 7faae1ace700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faab4056d20 0x7faab40591e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7faadc068780 tx=0x7faad8015000).stop 2020-11-20T12:06:55.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.484+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faadc0b3e90 msgr2=0x7faadc0ba880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.484+0000 7faae1ace700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faadc0b3e90 0x7faadc0ba880 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7faacc00c710 tx=0x7faacc007b90).stop 2020-11-20T12:06:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.485+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 shutdown_connections 2020-11-20T12:06:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.485+0000 7faae1ace700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faab4056d20 0x7faab40591e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.485+0000 7faae1ace700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faadc09f560 0x7faadc159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.485+0000 7faae1ace700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faadc0b46e0 0x7faadc0b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.485+0000 7faae1ace700 1 --2- 172.21.15.187:0/815701816 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faadc0b3e90 0x7faadc0ba880 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.485+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 >> 172.21.15.187:0/815701816 conn(0x7faadc068370 msgr2=0x7faadc0bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:55.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.485+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 shutdown_connections 2020-11-20T12:06:55.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.485+0000 7faae1ace700 1 -- 172.21.15.187:0/815701816 wait complete. 2020-11-20T12:06:55.514 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:06:55.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.758+0000 7f63d759e700 1 Processor -- start 2020-11-20T12:06:55.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.758+0000 7f63d759e700 1 -- start start 2020-11-20T12:06:55.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.759+0000 7f63d759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f63d80b1380 0x7f63d8150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:55.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.760+0000 7f63d759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80658a0 con 0x7f63d80b1760 2020-11-20T12:06:55.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.760+0000 7f63d759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8150390 con 0x7f63d814c260 2020-11-20T12:06:55.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.760+0000 7f63d759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d8150630 con 0x7f63d80b1380 2020-11-20T12:06:55.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.760+0000 7f63d6d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f63d814c260 0x7f63d814c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39306/0 (socket says 172.21.15.187:39306) 2020-11-20T12:06:55.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.760+0000 7f63d6d9d700 1 -- 172.21.15.187:0/2166266684 learned_addr learned my addr 172.21.15.187:0/2166266684 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:55.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.760+0000 7f63d659c700 1 --2- 172.21.15.187:0/2166266684 >> v2:172.21.15.5:3300/0 conn(0x7f63d80b1380 0x7f63d8150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:55.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.761+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126804036 0 0) 0x7f63d8150390 con 0x7f63d814c260 2020-11-20T12:06:55.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.761+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b8003240 con 0x7f63d814c260 2020-11-20T12:06:55.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.761+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3519483461 0 0) 0x7f63b8003240 con 0x7f63d814c260 2020-11-20T12:06:55.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.761+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 >> v2:172.21.15.5:3300/0 conn(0x7f63d80b1380 msgr2=0x7f63d8150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:55.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.761+0000 7f63d5d9b700 1 --2- 172.21.15.187:0/2166266684 >> v2:172.21.15.5:3300/0 conn(0x7f63d80b1380 0x7f63d8150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.761+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 >> v1:172.21.15.5:6789/0 conn(0x7f63d80b1760 legacy=0x7f63d8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.761+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d8068510 con 0x7f63d814c260 2020-11-20T12:06:55.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.761+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f63d0001860 con 0x7f63d814c260 2020-11-20T12:06:55.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 >> v1:172.21.15.187:6789/0 conn(0x7f63d814c260 legacy=0x7f63d814c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63d5d9b700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b8003780 0x7f63b8003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:55.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63d5d9b700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b80044e0 0x7f63b8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:55.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63d5d9b700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b8004e50 0x7f63b800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:55.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d8150630 con 0x7f63b80044e0 2020-11-20T12:06:55.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b8005300 con 0x7f63b8003780 2020-11-20T12:06:55.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b800bc20 con 0x7f63b8004e50 2020-11-20T12:06:55.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63d659c700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b8003780 0x7f63b8003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:55.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.762+0000 7f63cffff700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b80044e0 0x7f63b8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:55.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d6d9d700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b8004e50 0x7f63b800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:55.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d659c700 1 -- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b8004e50 msgr2=0x7f63b800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:55.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d659c700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b8004e50 0x7f63b800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d659c700 1 -- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b80044e0 msgr2=0x7f63b8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:55.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d659c700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b80044e0 0x7f63b8004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d659c700 1 -- 172.21.15.187:0/2166266684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8003240 con 0x7f63b8003780 2020-11-20T12:06:55.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63cffff700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b80044e0 0x7f63b8004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:06:55.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d6d9d700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b8004e50 0x7f63b800b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:06:55.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d6d9d700 1 -- 172.21.15.187:0/2166266684 reap_dead start 2020-11-20T12:06:55.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.763+0000 7f63d659c700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b8003780 0x7f63b8003ba0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f63c800bc00 tx=0x7f63c8006df0).ready entity=mon.1 client_cookie=5f33f0f819543996 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:55.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c800b310 con 0x7f63b8003780 2020-11-20T12:06:55.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c800eb40 con 0x7f63b8003780 2020-11-20T12:06:55.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d5d9b700 1 -- 172.21.15.187:0/2166266684 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c800eec0 con 0x7f63b8003780 2020-11-20T12:06:55.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d759e700 1 -- 172.21.15.187:0/2166266684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b8003780 msgr2=0x7f63b8003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d759e700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b8003780 0x7f63b8003ba0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f63c800bc00 tx=0x7f63c8006df0).stop 2020-11-20T12:06:55.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d759e700 1 -- 172.21.15.187:0/2166266684 shutdown_connections 2020-11-20T12:06:55.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d759e700 1 --2- 172.21.15.187:0/2166266684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b8003780 0x7f63b8003ba0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d759e700 1 -- 172.21.15.187:0/2166266684 >> 172.21.15.187:0/2166266684 conn(0x7f63d80bab30 msgr2=0x7f63d80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:55.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.764+0000 7f63d759e700 1 -- 172.21.15.187:0/2166266684 shutdown_connections 2020-11-20T12:06:55.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.765+0000 7f63d759e700 1 -- 172.21.15.187:0/2166266684 wait complete. 2020-11-20T12:06:55.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.766+0000 7f63d759e700 1 Processor -- start 2020-11-20T12:06:55.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.766+0000 7f63d759e700 1 -- start start 2020-11-20T12:06:55.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63d80b94e0 0x7f63d815c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:55.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63d80b98c0 0x7f63d80b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:55.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63d80b3000 0x7f63d80b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:55.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d80b3950 con 0x7f63d80b98c0 2020-11-20T12:06:55.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d815c6f0 con 0x7f63d80b3000 2020-11-20T12:06:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d815c980 con 0x7f63d80b94e0 2020-11-20T12:06:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63d80b3000 0x7f63d80b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63d80b3000 0x7f63d80b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42712/0 (socket says 172.21.15.187:42712) 2020-11-20T12:06:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.767+0000 7f63d6d9d700 1 -- 172.21.15.187:0/457361601 learned_addr learned my addr 172.21.15.187:0/457361601 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.768+0000 7f63cffff700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63d80b98c0 0x7f63d80b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.768+0000 7f63d6d9d700 1 -- 172.21.15.187:0/457361601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63d80b94e0 msgr2=0x7f63d815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.768+0000 7f63d6d9d700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63d80b94e0 0x7f63d815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.768+0000 7f63d6d9d700 1 -- 172.21.15.187:0/457361601 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63d80b98c0 msgr2=0x7f63d80b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.768+0000 7f63d6d9d700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63d80b98c0 0x7f63d80b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.768+0000 7f63d6d9d700 1 -- 172.21.15.187:0/457361601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d815cc20 con 0x7f63d80b3000 2020-11-20T12:06:55.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.768+0000 7f63d6d9d700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63d80b3000 0x7f63d80b3460 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f63d80b50b0 tx=0x7f63d0002080).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:55.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.768+0000 7f63ce7fc700 1 -- 172.21.15.187:0/457361601 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d000cbd0 con 0x7f63d80b3000 2020-11-20T12:06:55.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.769+0000 7f63ce7fc700 1 -- 172.21.15.187:0/457361601 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d000cd30 con 0x7f63d80b3000 2020-11-20T12:06:55.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.769+0000 7f63ce7fc700 1 -- 172.21.15.187:0/457361601 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0004a90 con 0x7f63d80b3000 2020-11-20T12:06:55.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.769+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d815cec0 con 0x7f63d80b3000 2020-11-20T12:06:55.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.769+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d809c620 con 0x7f63d80b3000 2020-11-20T12:06:55.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.770+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d815d050 con 0x7f63d80b3000 2020-11-20T12:06:55.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.770+0000 7f63ce7fc700 1 -- 172.21.15.187:0/457361601 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f63d001e070 con 0x7f63d80b3000 2020-11-20T12:06:55.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.771+0000 7f63ce7fc700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f63b4056d20 0x7f63b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:55.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.771+0000 7f63ce7fc700 1 -- 172.21.15.187:0/457361601 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f63d006fb40 con 0x7f63d80b3000 2020-11-20T12:06:55.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.771+0000 7f63d659c700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f63b4056d20 0x7f63b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:55.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.772+0000 7f63d659c700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f63b4056d20 0x7f63b40591e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f63d805ea30 tx=0x7f63c8019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:55.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.775+0000 7f63ce7fc700 1 -- 172.21.15.187:0/457361601 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f63d0044f40 con 0x7f63d80b3000 2020-11-20T12:06:55.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:55.954+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f63d8061a20 con 0x7f63d80b3000 2020-11-20T12:06:56.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.493+0000 7f63ce7fc700 1 -- 172.21.15.187:0/457361601 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v34) v1 ==== 181+0+0 (secure 0 0 0) 0x7f63d000cea0 con 0x7f63d80b3000 2020-11-20T12:06:56.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f63b4056d20 msgr2=0x7f63b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:56.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f63b4056d20 0x7f63b40591e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f63d805ea30 tx=0x7f63c8019000).stop 2020-11-20T12:06:56.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63d80b3000 msgr2=0x7f63d80b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:56.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63d80b3000 0x7f63d80b3460 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f63d80b50b0 tx=0x7f63d0002080).stop 2020-11-20T12:06:56.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 shutdown_connections 2020-11-20T12:06:56.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f63b4056d20 0x7f63b40591e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:56.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63d80b3000 0x7f63d80b3460 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:56.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63d80b94e0 0x7f63d815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:56.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 --2- 172.21.15.187:0/457361601 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63d80b98c0 0x7f63d80b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:56.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.497+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 >> 172.21.15.187:0/457361601 conn(0x7f63d80bab30 msgr2=0x7f63d80b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:56.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.498+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 shutdown_connections 2020-11-20T12:06:56.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.498+0000 7f63d759e700 1 -- 172.21.15.187:0/457361601 wait complete. 2020-11-20T12:06:56.504 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:06:56.532 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:06:56.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.779+0000 7fedf359e700 1 Processor -- start 2020-11-20T12:06:56.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.779+0000 7fedf359e700 1 -- start start 2020-11-20T12:06:56.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.780+0000 7fedf359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fedf41484d0 0x7fedf4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:56.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.781+0000 7fedf359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:56.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.781+0000 7fedf359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf40b1300 con 0x7fedf4066f70 2020-11-20T12:06:56.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.781+0000 7fedf359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf4067730 con 0x7fedf41484d0 2020-11-20T12:06:56.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.781+0000 7fedf359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40b15a0 con 0x7fedf40622c0 2020-11-20T12:06:56.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.782+0000 7fedf259c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fedf41484d0 0x7fedf4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:56.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.782+0000 7fedf1d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:56.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.782+0000 7fedf1d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf40b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:43606/0 (socket says 172.21.15.187:43606) 2020-11-20T12:06:56.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.782+0000 7fedf1d9b700 1 -- 172.21.15.187:0/686770874 learned_addr learned my addr 172.21.15.187:0/686770874 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:56.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.782+0000 7fedf2d9d700 1 --1- 172.21.15.187:0/686770874 >> v1:172.21.15.5:6789/0 conn(0x7fedf4066f70 0x7fedf40b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:32836/0 (socket says 172.21.15.187:32836) 2020-11-20T12:06:56.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.783+0000 7fedf259c700 1 -- 172.21.15.187:0/686770874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 msgr2=0x7fedf40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:56.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.783+0000 7fedf259c700 1 --2- 172.21.15.187:0/686770874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf40b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:56.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.783+0000 7fedf259c700 1 -- 172.21.15.187:0/686770874 >> v1:172.21.15.5:6789/0 conn(0x7fedf4066f70 legacy=0x7fedf40b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:56.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.783+0000 7fedf159a700 1 -- 172.21.15.187:0/686770874 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196043018 0 0) 0x7fedf40b1300 con 0x7fedf4066f70 2020-11-20T12:06:56.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.783+0000 7fedf259c700 1 -- 172.21.15.187:0/686770874 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf40b1840 con 0x7fedf41484d0 2020-11-20T12:06:56.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.783+0000 7fedf259c700 1 --2- 172.21.15.187:0/686770874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf41484d0 0x7fedf4061dd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fede400b9a0 tx=0x7fede4006df0).ready entity=mon.0 client_cookie=f49cf37285f5c3a5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:56.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.784+0000 7fedf159a700 1 -- 172.21.15.187:0/686770874 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede40097b0 con 0x7fedf41484d0 2020-11-20T12:06:56.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.784+0000 7fedf159a700 1 -- 172.21.15.187:0/686770874 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede4009910 con 0x7fedf41484d0 2020-11-20T12:06:56.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.784+0000 7fedf159a700 1 -- 172.21.15.187:0/686770874 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede4009c90 con 0x7fedf41484d0 2020-11-20T12:06:56.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.785+0000 7fedf359e700 1 -- 172.21.15.187:0/686770874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf41484d0 msgr2=0x7fedf4061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:56.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.785+0000 7fedf359e700 1 --2- 172.21.15.187:0/686770874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf41484d0 0x7fedf4061dd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fede400b9a0 tx=0x7fede4006df0).stop 2020-11-20T12:06:56.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.785+0000 7fedf359e700 1 -- 172.21.15.187:0/686770874 shutdown_connections 2020-11-20T12:06:56.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.785+0000 7fedf359e700 1 --2- 172.21.15.187:0/686770874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf41484d0 0x7fedf4061dd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:56.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.785+0000 7fedf359e700 1 --2- 172.21.15.187:0/686770874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:56.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.785+0000 7fedf359e700 1 -- 172.21.15.187:0/686770874 >> 172.21.15.187:0/686770874 conn(0x7fedf40bab30 msgr2=0x7fedf40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:56.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.785+0000 7fedf359e700 1 -- 172.21.15.187:0/686770874 shutdown_connections 2020-11-20T12:06:56.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.785+0000 7fedf359e700 1 -- 172.21.15.187:0/686770874 wait complete. 2020-11-20T12:06:56.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.787+0000 7fedf359e700 1 Processor -- start 2020-11-20T12:06:56.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.787+0000 7fedf359e700 1 -- start start 2020-11-20T12:06:56.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:56.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf4066f70 0x7fedf415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:56.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedf41484d0 0x7fedf40990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:56.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40a0520 con 0x7fedf4066f70 2020-11-20T12:06:56.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf409fd80 con 0x7fedf41484d0 2020-11-20T12:06:56.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf415c360 con 0x7fedf40622c0 2020-11-20T12:06:56.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf2d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedf41484d0 0x7fedf40990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:56.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf2d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedf41484d0 0x7fedf40990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42726/0 (socket says 172.21.15.187:42726) 2020-11-20T12:06:56.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.788+0000 7fedf2d9d700 1 -- 172.21.15.187:0/2172660584 learned_addr learned my addr 172.21.15.187:0/2172660584 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:56.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf1d9b700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf4066f70 0x7fedf415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:56.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf259c700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:56.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf2d9d700 1 -- 172.21.15.187:0/2172660584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 msgr2=0x7fedf415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:56.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf2d9d700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf415be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:56.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf2d9d700 1 -- 172.21.15.187:0/2172660584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf4066f70 msgr2=0x7fedf415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:56.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf2d9d700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf4066f70 0x7fedf415b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:56.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf2d9d700 1 -- 172.21.15.187:0/2172660584 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf409ffe0 con 0x7fedf41484d0 2020-11-20T12:06:56.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf2d9d700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedf41484d0 0x7fedf40990c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fedec00cae0 tx=0x7fedec007db0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:56.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fede27fc700 1 -- 172.21.15.187:0/2172660584 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec005ac0 con 0x7fedf41484d0 2020-11-20T12:06:56.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fede27fc700 1 -- 172.21.15.187:0/2172660584 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec005c20 con 0x7fedf41484d0 2020-11-20T12:06:56.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fede27fc700 1 -- 172.21.15.187:0/2172660584 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec007110 con 0x7fedf41484d0 2020-11-20T12:06:56.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf40a0280 con 0x7fedf41484d0 2020-11-20T12:06:56.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.789+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf40a08c0 con 0x7fedf41484d0 2020-11-20T12:06:56.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.790+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf4061a20 con 0x7fedf41484d0 2020-11-20T12:06:56.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.791+0000 7fede27fc700 1 -- 172.21.15.187:0/2172660584 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fedec010030 con 0x7fedf41484d0 2020-11-20T12:06:56.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.792+0000 7fede27fc700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedc8056d20 0x7fedc80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:56.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.792+0000 7fede27fc700 1 -- 172.21.15.187:0/2172660584 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fedec06ffb0 con 0x7fedf41484d0 2020-11-20T12:06:56.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.792+0000 7fedf259c700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedc8056d20 0x7fedc80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:56.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.793+0000 7fedf259c700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedc8056d20 0x7fedc80591e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fedf409eb90 tx=0x7fede4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:56.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.796+0000 7fede27fc700 1 -- 172.21.15.187:0/2172660584 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fedec045450 con 0x7fedf41484d0 2020-11-20T12:06:56.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:56.979+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fedf41488b0 con 0x7fedf41484d0 2020-11-20T12:06:57.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.495+0000 7fede27fc700 1 -- 172.21.15.187:0/2172660584 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v35) v1 ==== 169+0+0 (secure 0 0 0) 0x7fedec048a70 con 0x7fedf41484d0 2020-11-20T12:06:57.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.499+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedc8056d20 msgr2=0x7fedc80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.499+0000 7fedf359e700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedc8056d20 0x7fedc80591e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fedf409eb90 tx=0x7fede4015000).stop 2020-11-20T12:06:57.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.499+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedf41484d0 msgr2=0x7fedf40990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.499+0000 7fedf359e700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedf41484d0 0x7fedf40990c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fedec00cae0 tx=0x7fedec007db0).stop 2020-11-20T12:06:57.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.500+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 shutdown_connections 2020-11-20T12:06:57.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.500+0000 7fedf359e700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedc8056d20 0x7fedc80591e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.500+0000 7fedf359e700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedf41484d0 0x7fedf40990c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.500+0000 7fedf359e700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedf40622c0 0x7fedf415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.500+0000 7fedf359e700 1 --2- 172.21.15.187:0/2172660584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedf4066f70 0x7fedf415b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.500+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 >> 172.21.15.187:0/2172660584 conn(0x7fedf40bab30 msgr2=0x7fedf4144fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:57.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.501+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 shutdown_connections 2020-11-20T12:06:57.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.501+0000 7fedf359e700 1 -- 172.21.15.187:0/2172660584 wait complete. 2020-11-20T12:06:57.508 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:06:57.544 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:06:57.544 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:06:57.544 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:06:57.545 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:06:57.545 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:06:57.545 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:06:57.546 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:06:57.546 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:06:57.570 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:06:57.570 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:06:57.576 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:06:57.577 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:06:57.583 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:06:57.584 DEBUG:teuthology.parallel:result is None 2020-11-20T12:06:57.618 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:06:57.619 DEBUG:teuthology.parallel:result is None 2020-11-20T12:06:57.620 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:06:57.770 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:06:57.778 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:06:57.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.838+0000 7f5c7c867700 1 Processor -- start 2020-11-20T12:06:57.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.838+0000 7f5c7c867700 1 -- start start 2020-11-20T12:06:57.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.839+0000 7f5c7c867700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5c780b6650 0x7f5c780afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.839+0000 7f5c7c867700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c780b1300 con 0x7f5c781484d0 2020-11-20T12:06:57.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.839+0000 7f5c7c867700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c780b15a0 con 0x7f5c78066f70 2020-11-20T12:06:57.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.839+0000 7f5c7c867700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c78067730 con 0x7f5c780b6650 2020-11-20T12:06:57.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.840+0000 7f5c7759e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5c78066f70 0x7f5c780aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39352/0 (socket says 172.21.15.187:39352) 2020-11-20T12:06:57.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.840+0000 7f5c7759e700 1 -- 172.21.15.187:0/1003472014 learned_addr learned my addr 172.21.15.187:0/1003472014 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:57.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.840+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106384647 0 0) 0x7f5c780b15a0 con 0x7f5c78066f70 2020-11-20T12:06:57.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.840+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c58003240 con 0x7f5c78066f70 2020-11-20T12:06:57.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.840+0000 7f5c6ffff700 1 --2- 172.21.15.187:0/1003472014 >> v2:172.21.15.5:3300/0 conn(0x7f5c780b6650 0x7f5c780afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1571106102 0 0) 0x7f5c58003240 con 0x7f5c78066f70 2020-11-20T12:06:57.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 >> v2:172.21.15.5:3300/0 conn(0x7f5c780b6650 msgr2=0x7f5c780afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:57.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 --2- 172.21.15.187:0/1003472014 >> v2:172.21.15.5:3300/0 conn(0x7f5c780b6650 0x7f5c780afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 >> v1:172.21.15.5:6789/0 conn(0x7f5c781484d0 legacy=0x7f5c78061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c780b1840 con 0x7f5c78066f70 2020-11-20T12:06:57.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5c70001860 con 0x7f5c78066f70 2020-11-20T12:06:57.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 >> v1:172.21.15.187:6789/0 conn(0x7f5c78066f70 legacy=0x7f5c780aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58003780 0x7f5c58003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c580044e0 0x7f5c58004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c58004e50 0x7f5c5800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c78067730 con 0x7f5c58003780 2020-11-20T12:06:57.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58005300 con 0x7f5c58004e50 2020-11-20T12:06:57.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5800bc20 con 0x7f5c580044e0 2020-11-20T12:06:57.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c7759e700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c58004e50 0x7f5c5800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.841+0000 7f5c76d9d700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58003780 0x7f5c58003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c6ffff700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c580044e0 0x7f5c58004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7759e700 1 -- 172.21.15.187:0/1003472014 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c580044e0 msgr2=0x7f5c58004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:57.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7759e700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c580044e0 0x7f5c58004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7759e700 1 -- 172.21.15.187:0/1003472014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58003780 msgr2=0x7f5c58003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:57.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7759e700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58003780 0x7f5c58003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7759e700 1 -- 172.21.15.187:0/1003472014 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c58003240 con 0x7f5c58004e50 2020-11-20T12:06:57.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7759e700 1 -- 172.21.15.187:0/1003472014 reap_dead start 2020-11-20T12:06:57.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7759e700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c58004e50 0x7f5c5800b730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f5c70007fd0 tx=0x7f5c70007a50).ready entity=mon.1 client_cookie=db4afec9c41275d2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:57.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70005b80 con 0x7f5c58004e50 2020-11-20T12:06:57.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c70005ce0 con 0x7f5c58004e50 2020-11-20T12:06:57.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.842+0000 7f5c7659c700 1 -- 172.21.15.187:0/1003472014 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70007320 con 0x7f5c58004e50 2020-11-20T12:06:57.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.843+0000 7f5c7c867700 1 -- 172.21.15.187:0/1003472014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c58004e50 msgr2=0x7f5c5800b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.843+0000 7f5c7c867700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c58004e50 0x7f5c5800b730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f5c70007fd0 tx=0x7f5c70007a50).stop 2020-11-20T12:06:57.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.843+0000 7f5c7c867700 1 -- 172.21.15.187:0/1003472014 shutdown_connections 2020-11-20T12:06:57.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.843+0000 7f5c7c867700 1 --2- 172.21.15.187:0/1003472014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c58004e50 0x7f5c5800b730 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.843+0000 7f5c7c867700 1 -- 172.21.15.187:0/1003472014 >> 172.21.15.187:0/1003472014 conn(0x7f5c780bab30 msgr2=0x7f5c780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:57.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.843+0000 7f5c7c867700 1 -- 172.21.15.187:0/1003472014 shutdown_connections 2020-11-20T12:06:57.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.843+0000 7f5c7c867700 1 -- 172.21.15.187:0/1003472014 wait complete. 2020-11-20T12:06:57.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.845+0000 7f5c7c867700 1 Processor -- start 2020-11-20T12:06:57.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.845+0000 7f5c7c867700 1 -- start start 2020-11-20T12:06:57.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c7c867700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58004e50 0x7f5c780b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c7c867700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c78146f30 0x7f5c78136da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c7c867700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c78137290 0x7f5c7813c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c7c867700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7813d3f0 con 0x7f5c58004e50 2020-11-20T12:06:57.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c7c867700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7813d690 con 0x7f5c78146f30 2020-11-20T12:06:57.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c7c867700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c78147720 con 0x7f5c78137290 2020-11-20T12:06:57.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c78146f30 0x7f5c78136da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c78146f30 0x7f5c78136da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42756/0 (socket says 172.21.15.187:42756) 2020-11-20T12:06:57.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c6ffff700 1 -- 172.21.15.187:0/2020692466 learned_addr learned my addr 172.21.15.187:0/2020692466 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:57.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.846+0000 7f5c76d9d700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58004e50 0x7f5c780b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c7759e700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c78137290 0x7f5c7813c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6ffff700 1 -- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c78137290 msgr2=0x7f5c7813c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6ffff700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c78137290 0x7f5c7813c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6ffff700 1 -- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58004e50 msgr2=0x7f5c780b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6ffff700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58004e50 0x7f5c780b2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6ffff700 1 -- 172.21.15.187:0/2020692466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c781478b0 con 0x7f5c78146f30 2020-11-20T12:06:57.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c76d9d700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58004e50 0x7f5c780b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:57.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6ffff700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c78146f30 0x7f5c78136da0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f5c60010970 tx=0x7f5c60008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:57.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6effd700 1 -- 172.21.15.187:0/2020692466 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6000e930 con 0x7f5c78146f30 2020-11-20T12:06:57.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6effd700 1 -- 172.21.15.187:0/2020692466 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6000ea90 con 0x7f5c78146f30 2020-11-20T12:06:57.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7813cda0 con 0x7f5c78146f30 2020-11-20T12:06:57.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c6effd700 1 -- 172.21.15.187:0/2020692466 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6000ee50 con 0x7f5c78146f30 2020-11-20T12:06:57.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.847+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7813e420 con 0x7f5c78146f30 2020-11-20T12:06:57.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.848+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7813d7d0 con 0x7f5c78146f30 2020-11-20T12:06:57.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.849+0000 7f5c6effd700 1 -- 172.21.15.187:0/2020692466 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5c60024070 con 0x7f5c78146f30 2020-11-20T12:06:57.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.849+0000 7f5c6effd700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5c54056d20 0x7f5c540591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.850+0000 7f5c6effd700 1 -- 172.21.15.187:0/2020692466 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5c60070570 con 0x7f5c78146f30 2020-11-20T12:06:57.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.850+0000 7f5c76d9d700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5c54056d20 0x7f5c540591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.850+0000 7f5c76d9d700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5c54056d20 0x7f5c540591e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5c68000f20 tx=0x7f5c68009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:57.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:57.853+0000 7f5c6effd700 1 -- 172.21.15.187:0/2020692466 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5c60045ca0 con 0x7f5c78146f30 2020-11-20T12:06:58.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.057+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5c78061a20 con 0x7f5c78146f30 2020-11-20T12:06:58.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.523+0000 7f5c6effd700 1 -- 172.21.15.187:0/2020692466 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v36) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5c60018070 con 0x7f5c78146f30 2020-11-20T12:06:58.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5c54056d20 msgr2=0x7f5c540591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:58.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5c54056d20 0x7f5c540591e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5c68000f20 tx=0x7f5c68009040).stop 2020-11-20T12:06:58.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c78146f30 msgr2=0x7f5c78136da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:58.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c78146f30 0x7f5c78136da0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f5c60010970 tx=0x7f5c60008040).stop 2020-11-20T12:06:58.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 shutdown_connections 2020-11-20T12:06:58.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5c54056d20 0x7f5c540591e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:58.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c58004e50 0x7f5c780b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:58.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c78137290 0x7f5c7813c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:58.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.527+0000 7f5c7c867700 1 --2- 172.21.15.187:0/2020692466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c78146f30 0x7f5c78136da0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:58.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.528+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 >> 172.21.15.187:0/2020692466 conn(0x7f5c780bab30 msgr2=0x7f5c78061e60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:58.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.528+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 shutdown_connections 2020-11-20T12:06:58.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.528+0000 7f5c7c867700 1 -- 172.21.15.187:0/2020692466 wait complete. 2020-11-20T12:06:58.535 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:06:58.560 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:06:58.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.805+0000 7f3adc814700 1 Processor -- start 2020-11-20T12:06:58.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.805+0000 7f3adc814700 1 -- start start 2020-11-20T12:06:58.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.806+0000 7f3adc814700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ad80b57b0 0x7f3ad8068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:58.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.806+0000 7f3adc814700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 0x7f3ad8062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:58.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3adc814700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3ad8062b50 0x7f3ad8065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:58.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3adc814700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad8144b80 con 0x7f3ad8062b50 2020-11-20T12:06:58.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3adc814700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad81489e0 con 0x7f3ad80b57b0 2020-11-20T12:06:58.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3adc814700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad8148c80 con 0x7f3ad81482d0 2020-11-20T12:06:58.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3ad6d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ad80b57b0 0x7f3ad8068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:58.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3ad6d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ad80b57b0 0x7f3ad8068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42762/0 (socket says 172.21.15.187:42762) 2020-11-20T12:06:58.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3ad6d9d700 1 -- 172.21.15.187:0/1024552017 learned_addr learned my addr 172.21.15.187:0/1024552017 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:58.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3ad6d9d700 1 -- 172.21.15.187:0/1024552017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 msgr2=0x7f3ad8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3ad6d9d700 1 --2- 172.21.15.187:0/1024552017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 0x7f3ad8062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3ad6d9d700 1 -- 172.21.15.187:0/1024552017 >> v2:172.21.15.5:3300/0 conn(0x7f3ad8062b50 msgr2=0x7f3ad8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3ad6d9d700 1 --2- 172.21.15.187:0/1024552017 >> v2:172.21.15.5:3300/0 conn(0x7f3ad8062b50 0x7f3ad8065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:58.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.807+0000 7f3ad6d9d700 1 -- 172.21.15.187:0/1024552017 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad8148f20 con 0x7f3ad80b57b0 2020-11-20T12:06:58.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.808+0000 7f3ad6d9d700 1 --2- 172.21.15.187:0/1024552017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad80b57b0 0x7f3ad8068230 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f3ad80b85e0 tx=0x7f3ac8006eb0).ready entity=mon.1 client_cookie=9b8af4571135913d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:58.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.808+0000 7f3ad659c700 1 -- 172.21.15.187:0/1024552017 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac80098d0 con 0x7f3ad80b57b0 2020-11-20T12:06:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.808+0000 7f3ad659c700 1 -- 172.21.15.187:0/1024552017 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac8009a30 con 0x7f3ad80b57b0 2020-11-20T12:06:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.808+0000 7f3ad659c700 1 -- 172.21.15.187:0/1024552017 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8009db0 con 0x7f3ad80b57b0 2020-11-20T12:06:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 -- 172.21.15.187:0/1024552017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad80b57b0 msgr2=0x7f3ad8068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 --2- 172.21.15.187:0/1024552017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad80b57b0 0x7f3ad8068230 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f3ad80b85e0 tx=0x7f3ac8006eb0).stop 2020-11-20T12:06:58.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 -- 172.21.15.187:0/1024552017 shutdown_connections 2020-11-20T12:06:58.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 --2- 172.21.15.187:0/1024552017 >> v2:172.21.15.5:3300/0 conn(0x7f3ad8062b50 0x7f3ad8065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:58.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 --2- 172.21.15.187:0/1024552017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad80b57b0 0x7f3ad8068230 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:58.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 --2- 172.21.15.187:0/1024552017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 0x7f3ad8062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:06:58.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 -- 172.21.15.187:0/1024552017 >> 172.21.15.187:0/1024552017 conn(0x7f3ad80bab30 msgr2=0x7f3ad80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:58.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 -- 172.21.15.187:0/1024552017 shutdown_connections 2020-11-20T12:06:58.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.809+0000 7f3adc814700 1 -- 172.21.15.187:0/1024552017 wait complete. 2020-11-20T12:06:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.811+0000 7f3adc814700 1 Processor -- start 2020-11-20T12:06:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.811+0000 7f3adc814700 1 -- start start 2020-11-20T12:06:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3adc814700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad8062b50 0x7f3ad8139060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3adc814700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ad80b57b0 0x7f3ad809f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3adc814700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 0x7f3ad80b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3adc814700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad809cd30 con 0x7f3ad80b57b0 2020-11-20T12:06:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3adc814700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad809ca70 con 0x7f3ad8062b50 2020-11-20T12:06:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3adc814700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad809e870 con 0x7f3ad81482d0 2020-11-20T12:06:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3ad6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad8062b50 0x7f3ad8139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3ad6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad8062b50 0x7f3ad8139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42768/0 (socket says 172.21.15.187:42768) 2020-11-20T12:06:58.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.812+0000 7f3ad6d9d700 1 -- 172.21.15.187:0/479779017 learned_addr learned my addr 172.21.15.187:0/479779017 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:58.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ad759e700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 0x7f3ad80b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:58.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ace59c700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ad80b57b0 0x7f3ad809f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:58.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ad6d9d700 1 -- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 msgr2=0x7f3ad80b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:58.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ad6d9d700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 0x7f3ad80b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:58.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ad6d9d700 1 -- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ad80b57b0 msgr2=0x7f3ad809f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ad6d9d700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ad80b57b0 0x7f3ad809f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ad6d9d700 1 -- 172.21.15.187:0/479779017 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad809d220 con 0x7f3ad8062b50 2020-11-20T12:06:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ad759e700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 0x7f3ad80b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3ad6d9d700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad8062b50 0x7f3ad8139060 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f3ac8006ef0 tx=0x7f3ac8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3acf7fe700 1 -- 172.21.15.187:0/479779017 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac801e460 con 0x7f3ad8062b50 2020-11-20T12:06:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.813+0000 7f3acf7fe700 1 -- 172.21.15.187:0/479779017 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac801e5c0 con 0x7f3ad8062b50 2020-11-20T12:06:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.814+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad809d470 con 0x7f3ad8062b50 2020-11-20T12:06:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.814+0000 7f3acf7fe700 1 -- 172.21.15.187:0/479779017 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac801e980 con 0x7f3ad8062b50 2020-11-20T12:06:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.814+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad815c0e0 con 0x7f3ad8062b50 2020-11-20T12:06:58.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.815+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad80b0190 con 0x7f3ad8062b50 2020-11-20T12:06:58.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.815+0000 7f3acf7fe700 1 -- 172.21.15.187:0/479779017 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3ac8025030 con 0x7f3ad8062b50 2020-11-20T12:06:58.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.816+0000 7f3acf7fe700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ab4056d20 0x7f3ab40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:58.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.816+0000 7f3acf7fe700 1 -- 172.21.15.187:0/479779017 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3ac8071670 con 0x7f3ad8062b50 2020-11-20T12:06:58.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.816+0000 7f3ace59c700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ab4056d20 0x7f3ab40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:58.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.817+0000 7f3ace59c700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ab4056d20 0x7f3ab40591e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3ac000a6a0 tx=0x7f3ac0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:58.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:58.820+0000 7f3acf7fe700 1 -- 172.21.15.187:0/479779017 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3ac8046ec0 con 0x7f3ad8062b50 2020-11-20T12:06:59.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.070+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3ad8061a20 con 0x7f3ad8062b50 2020-11-20T12:06:59.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.071+0000 7f3acf7fe700 1 -- 172.21.15.187:0/479779017 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f3ac8023030 con 0x7f3ad8062b50 2020-11-20T12:06:59.076 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:59.076 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:06:59.076 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:06:59.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.077 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:06:59.077 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:06:59.077 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.077 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:59.078 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.078 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.078 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.078 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.079 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:06:59.079 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:06:59.079 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.080 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:59.080 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.080 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.080 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.081 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.081 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:06:59.081 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:06:59.082 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.082 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.082 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.082 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.083 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.083 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:06:59.084 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:06:59.084 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.084 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.084 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.085 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.085 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.086 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:06:59.086 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:06:59.086 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.086 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.086 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.087 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.087 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.088 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:06:59.088 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:06:59.088 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.088 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.088 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.089 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.089 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.090 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:06:59.090 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:06:59.090 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.090 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.091 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.091 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.091 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.092 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:06:59.092 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:06:59.092 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.093 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.093 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.093 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.093 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.094 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:06:59.094 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:06:59.094 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.095 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.095 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.095 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.095 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.096 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:06:59.096 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:06:59.096 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.097 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.097 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.097 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.097 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.098 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:06:59.098 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:06:59.098 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.099 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:59.099 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:06:59.099 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:06:59.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.100 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:06:59.100 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:06:59.101 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.101 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:59.101 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:06:59.101 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:06:59.102 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:06:59.102 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.102 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:06:59.103 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:06:59.103 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.103 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:59.103 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:06:59.103 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:06:59.104 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:06:59.104 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.105 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:06:59.105 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:06:59.105 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.105 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:06:59.105 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:06:59.106 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:06:59.106 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:06:59.106 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.107 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:06:59.107 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:06:59.107 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.108 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:06:59.108 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:06:59.108 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:06:59.108 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.108 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.109 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:06:59.109 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:06:59.109 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.110 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:06:59.110 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:06:59.110 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.111 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:06:59.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:06:59.111 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:06:59.111 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:06:59.112 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:06:59.112 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:06:59.112 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:06:59.112 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:06:59.113 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.113 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:06:59.113 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:06:59.113 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:06:59.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.073+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ab4056d20 msgr2=0x7f3ab40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.073+0000 7f3adc814700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ab4056d20 0x7f3ab40591e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3ac000a6a0 tx=0x7f3ac0008040).stop 2020-11-20T12:06:59.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.073+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad8062b50 msgr2=0x7f3ad8139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.073+0000 7f3adc814700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad8062b50 0x7f3ad8139060 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f3ac8006ef0 tx=0x7f3ac8015000).stop 2020-11-20T12:06:59.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.074+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 shutdown_connections 2020-11-20T12:06:59.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.074+0000 7f3adc814700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ab4056d20 0x7f3ab40591e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.074+0000 7f3adc814700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ad81482d0 0x7f3ad80b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.074+0000 7f3adc814700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ad8062b50 0x7f3ad8139060 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.074+0000 7f3adc814700 1 --2- 172.21.15.187:0/479779017 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ad80b57b0 0x7f3ad809f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.074+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 >> 172.21.15.187:0/479779017 conn(0x7f3ad80bab30 msgr2=0x7f3ad80b9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:59.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.074+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 shutdown_connections 2020-11-20T12:06:59.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.074+0000 7f3adc814700 1 -- 172.21.15.187:0/479779017 wait complete. 2020-11-20T12:06:59.118 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:06:59.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.340+0000 7fbb1aaca700 1 Processor -- start 2020-11-20T12:06:59.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.341+0000 7fbb1aaca700 1 -- start start 2020-11-20T12:06:59.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.342+0000 7fbb1aaca700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb14065f90 0x7fbb14065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.342+0000 7fbb1aaca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb1409ca90 0x7fbb1409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.342+0000 7fbb1aaca700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb140b13e0 con 0x7fbb14065190 2020-11-20T12:06:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.342+0000 7fbb1aaca700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1414ab30 con 0x7fbb14065f90 2020-11-20T12:06:59.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.342+0000 7fbb1aaca700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb14062c10 con 0x7fbb1409ca90 2020-11-20T12:06:59.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.343+0000 7fbb1a2c9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb1409ca90 0x7fbb1409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.343+0000 7fbb1a2c9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb1409ca90 0x7fbb1409cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42778/0 (socket says 172.21.15.187:42778) 2020-11-20T12:06:59.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.343+0000 7fbb1a2c9700 1 -- 172.21.15.187:0/3712886067 learned_addr learned my addr 172.21.15.187:0/3712886067 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:59.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.343+0000 7fbb192c7700 1 --2- 172.21.15.187:0/3712886067 >> v2:172.21.15.5:3300/0 conn(0x7fbb14065f90 0x7fbb14065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.343+0000 7fbb1a2c9700 1 -- 172.21.15.187:0/3712886067 >> v2:172.21.15.5:3300/0 conn(0x7fbb14065f90 msgr2=0x7fbb14065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:59.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.343+0000 7fbb1a2c9700 1 --2- 172.21.15.187:0/3712886067 >> v2:172.21.15.5:3300/0 conn(0x7fbb14065f90 0x7fbb14065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.343+0000 7fbb1a2c9700 1 -- 172.21.15.187:0/3712886067 >> v1:172.21.15.5:6789/0 conn(0x7fbb14065190 legacy=0x7fbb14143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.343+0000 7fbb1a2c9700 1 -- 172.21.15.187:0/3712886067 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb140b1640 con 0x7fbb1409ca90 2020-11-20T12:06:59.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb1a2c9700 1 --2- 172.21.15.187:0/3712886067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb1409cf20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fbb14099f90 tx=0x7fbb0c009290).ready entity=mon.1 client_cookie=3d6bbb966222ff4e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:59.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb18ac6700 1 -- 172.21.15.187:0/3712886067 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00bc50 con 0x7fbb1409ca90 2020-11-20T12:06:59.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb18ac6700 1 -- 172.21.15.187:0/3712886067 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c00bdb0 con 0x7fbb1409ca90 2020-11-20T12:06:59.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb18ac6700 1 -- 172.21.15.187:0/3712886067 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c011610 con 0x7fbb1409ca90 2020-11-20T12:06:59.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb1aaca700 1 -- 172.21.15.187:0/3712886067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 msgr2=0x7fbb1409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/3712886067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb1409cf20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fbb14099f90 tx=0x7fbb0c009290).stop 2020-11-20T12:06:59.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb1aaca700 1 -- 172.21.15.187:0/3712886067 shutdown_connections 2020-11-20T12:06:59.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/3712886067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb1409cf20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.344+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/3712886067 >> v2:172.21.15.5:3300/0 conn(0x7fbb14065f90 0x7fbb14065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.345+0000 7fbb1aaca700 1 -- 172.21.15.187:0/3712886067 >> 172.21.15.187:0/3712886067 conn(0x7fbb140bab30 msgr2=0x7fbb140baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:59.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.345+0000 7fbb1aaca700 1 -- 172.21.15.187:0/3712886067 shutdown_connections 2020-11-20T12:06:59.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.345+0000 7fbb1aaca700 1 -- 172.21.15.187:0/3712886067 wait complete. 2020-11-20T12:06:59.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.346+0000 7fbb1aaca700 1 Processor -- start 2020-11-20T12:06:59.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.347+0000 7fbb1aaca700 1 -- start start 2020-11-20T12:06:59.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1aaca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb14065190 0x7fbb140b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1aaca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb14065f90 0x7fbb141579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1aaca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb140676a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1aaca700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb141561b0 con 0x7fbb14065f90 2020-11-20T12:06:59.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1aaca700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1409f920 con 0x7fbb1409ca90 2020-11-20T12:06:59.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1aaca700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1409fbc0 con 0x7fbb14065190 2020-11-20T12:06:59.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1a2c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb140676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1a2c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb140676a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42786/0 (socket says 172.21.15.187:42786) 2020-11-20T12:06:59.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb1a2c9700 1 -- 172.21.15.187:0/1373847237 learned_addr learned my addr 172.21.15.187:0/1373847237 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:59.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.348+0000 7fbb19ac8700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb14065190 0x7fbb140b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb1a2c9700 1 -- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb14065190 msgr2=0x7fbb140b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb192c7700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb14065f90 0x7fbb141579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb1a2c9700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb14065190 0x7fbb140b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb1a2c9700 1 -- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb14065f90 msgr2=0x7fbb141579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb1a2c9700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb14065f90 0x7fbb141579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb1a2c9700 1 -- 172.21.15.187:0/1373847237 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1409fe60 con 0x7fbb1409ca90 2020-11-20T12:06:59.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb19ac8700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb14065190 0x7fbb140b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:59.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb1a2c9700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb140676a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fbb0c0092d0 tx=0x7fbb0c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:59.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb01ffb700 1 -- 172.21.15.187:0/1373847237 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00bc50 con 0x7fbb1409ca90 2020-11-20T12:06:59.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb01ffb700 1 -- 172.21.15.187:0/1373847237 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c010c40 con 0x7fbb1409ca90 2020-11-20T12:06:59.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb01ffb700 1 -- 172.21.15.187:0/1373847237 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c01c470 con 0x7fbb1409ca90 2020-11-20T12:06:59.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb140b48b0 con 0x7fbb1409ca90 2020-11-20T12:06:59.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.349+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb14157ea0 con 0x7fbb1409ca90 2020-11-20T12:06:59.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.350+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb140a0220 con 0x7fbb1409ca90 2020-11-20T12:06:59.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.351+0000 7fbb01ffb700 1 -- 172.21.15.187:0/1373847237 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbb0c017070 con 0x7fbb1409ca90 2020-11-20T12:06:59.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.352+0000 7fbb01ffb700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbaec056d20 0x7fbaec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.352+0000 7fbb01ffb700 1 -- 172.21.15.187:0/1373847237 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fbb0c071ec0 con 0x7fbb1409ca90 2020-11-20T12:06:59.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.352+0000 7fbb19ac8700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbaec056d20 0x7fbaec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.352+0000 7fbb19ac8700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbaec056d20 0x7fbaec0591e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fbb100014d0 tx=0x7fbb10009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:59.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.355+0000 7fbb01ffb700 1 -- 172.21.15.187:0/1373847237 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbb0c047640 con 0x7fbb1409ca90 2020-11-20T12:06:59.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.545+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb140b4a40 con 0x7fbb1409ca90 2020-11-20T12:06:59.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.547+0000 7fbb01ffb700 1 -- 172.21.15.187:0/1373847237 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13627 (secure 0 0 0) 0x7fbb0c015070 con 0x7fbb1409ca90 2020-11-20T12:06:59.551 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:06:59.552 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":36,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:06:58.503065+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:06:59.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.549+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbaec056d20 msgr2=0x7fbaec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.549+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbaec056d20 0x7fbaec0591e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fbb100014d0 tx=0x7fbb10009040).stop 2020-11-20T12:06:59.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.549+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 msgr2=0x7fbb140676a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.549+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb140676a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fbb0c0092d0 tx=0x7fbb0c016000).stop 2020-11-20T12:06:59.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.550+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 shutdown_connections 2020-11-20T12:06:59.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.550+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbaec056d20 0x7fbaec0591e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.550+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb1409ca90 0x7fbb140676a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.550+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb14065190 0x7fbb140b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.550+0000 7fbb1aaca700 1 --2- 172.21.15.187:0/1373847237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb14065f90 0x7fbb141579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.550+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 >> 172.21.15.187:0/1373847237 conn(0x7fbb140bab30 msgr2=0x7fbb140aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:59.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.550+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 shutdown_connections 2020-11-20T12:06:59.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.550+0000 7fbb1aaca700 1 -- 172.21.15.187:0/1373847237 wait complete. 2020-11-20T12:06:59.577 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:06:59.578 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:06:59.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.813+0000 7f666f59e700 1 Processor -- start 2020-11-20T12:06:59.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.813+0000 7f666f59e700 1 -- start start 2020-11-20T12:06:59.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.814+0000 7f666f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f66700bd620 0x7f66701465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.814+0000 7f666f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66700bda00 0x7f6670140330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.814+0000 7f666f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6670146d00 con 0x7f6670140820 2020-11-20T12:06:59.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.814+0000 7f666f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6670146ae0 con 0x7f66700bd620 2020-11-20T12:06:59.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.814+0000 7f666f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6670146fa0 con 0x7f66700bda00 2020-11-20T12:06:59.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.816+0000 7f666dd9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66700bda00 0x7f6670140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.816+0000 7f666ed9d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6670140820 0x7f6670067340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:32908/0 (socket says 172.21.15.187:32908) 2020-11-20T12:06:59.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.816+0000 7f666e59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f66700bd620 0x7f66701465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.816+0000 7f666ed9d700 1 -- 172.21.15.187:0/89387543 learned_addr learned my addr 172.21.15.187:0/89387543 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:06:59.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.816+0000 7f666dd9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66700bda00 0x7f6670140330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:43682/0 (socket says 172.21.15.187:43682) 2020-11-20T12:06:59.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.817+0000 7f666e59c700 1 -- 172.21.15.187:0/89387543 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66700bda00 msgr2=0x7f6670140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:59.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.817+0000 7f666e59c700 1 --2- 172.21.15.187:0/89387543 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66700bda00 0x7f6670140330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.817+0000 7f666e59c700 1 -- 172.21.15.187:0/89387543 >> v1:172.21.15.5:6789/0 conn(0x7f6670140820 legacy=0x7f6670067340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.817+0000 7f666e59c700 1 -- 172.21.15.187:0/89387543 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6670147240 con 0x7f66700bd620 2020-11-20T12:06:59.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.817+0000 7f666e59c700 1 --2- 172.21.15.187:0/89387543 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 0x7f66701465f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f666000b9d0 tx=0x7f6660006df0).ready entity=mon.0 client_cookie=a9495ad178b02fe5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:59.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.818+0000 7f666d59a700 1 -- 172.21.15.187:0/89387543 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660009800 con 0x7f66700bd620 2020-11-20T12:06:59.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.818+0000 7f666d59a700 1 -- 172.21.15.187:0/89387543 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6660009960 con 0x7f66700bd620 2020-11-20T12:06:59.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.818+0000 7f666d59a700 1 -- 172.21.15.187:0/89387543 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660009ce0 con 0x7f66700bd620 2020-11-20T12:06:59.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.819+0000 7f666f59e700 1 -- 172.21.15.187:0/89387543 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 msgr2=0x7f66701465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.819+0000 7f666f59e700 1 --2- 172.21.15.187:0/89387543 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 0x7f66701465f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f666000b9d0 tx=0x7f6660006df0).stop 2020-11-20T12:06:59.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.819+0000 7f666f59e700 1 -- 172.21.15.187:0/89387543 shutdown_connections 2020-11-20T12:06:59.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.819+0000 7f666f59e700 1 --2- 172.21.15.187:0/89387543 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 0x7f66701465f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.819+0000 7f666f59e700 1 --2- 172.21.15.187:0/89387543 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66700bda00 0x7f6670140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.819+0000 7f666f59e700 1 -- 172.21.15.187:0/89387543 >> 172.21.15.187:0/89387543 conn(0x7f6670060c10 msgr2=0x7f66700bcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:59.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.819+0000 7f666f59e700 1 -- 172.21.15.187:0/89387543 shutdown_connections 2020-11-20T12:06:59.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.819+0000 7f666f59e700 1 -- 172.21.15.187:0/89387543 wait complete. 2020-11-20T12:06:59.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.821+0000 7f666f59e700 1 Processor -- start 2020-11-20T12:06:59.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.821+0000 7f666f59e700 1 -- start start 2020-11-20T12:06:59.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 0x7f6670134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6670140820 0x7f66700a2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66701373d0 0x7f6670135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6670134bd0 con 0x7f66700bd620 2020-11-20T12:06:59.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66701441b0 con 0x7f66701373d0 2020-11-20T12:06:59.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6670144450 con 0x7f6670140820 2020-11-20T12:06:59.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66701373d0 0x7f6670135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66701373d0 0x7f6670135e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42798/0 (socket says 172.21.15.187:42798) 2020-11-20T12:06:59.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666ed9d700 1 -- 172.21.15.187:0/3163527035 learned_addr learned my addr 172.21.15.187:0/3163527035 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:06:59.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.822+0000 7f666e59c700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 0x7f6670134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666dd9b700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6670140820 0x7f66700a2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666ed9d700 1 -- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6670140820 msgr2=0x7f66700a2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666ed9d700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6670140820 0x7f66700a2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666ed9d700 1 -- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 msgr2=0x7f6670134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:59.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666ed9d700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 0x7f6670134300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:59.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666ed9d700 1 -- 172.21.15.187:0/3163527035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66701446f0 con 0x7f66701373d0 2020-11-20T12:06:59.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666dd9b700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6670140820 0x7f66700a2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:06:59.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666ed9d700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66701373d0 0x7f6670135e50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f666800c860 tx=0x7f666800cc20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:59.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f665e7fc700 1 -- 172.21.15.187:0/3163527035 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6668005b80 con 0x7f66701373d0 2020-11-20T12:06:59.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f665e7fc700 1 -- 172.21.15.187:0/3163527035 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6668005ce0 con 0x7f66701373d0 2020-11-20T12:06:59.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f665e7fc700 1 -- 172.21.15.187:0/3163527035 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6668007320 con 0x7f66701373d0 2020-11-20T12:06:59.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.823+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66700a2510 con 0x7f66701373d0 2020-11-20T12:06:59.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.824+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6670136340 con 0x7f66701373d0 2020-11-20T12:06:59.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.825+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6670146700 con 0x7f66701373d0 2020-11-20T12:06:59.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.825+0000 7f665e7fc700 1 -- 172.21.15.187:0/3163527035 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6668010030 con 0x7f66701373d0 2020-11-20T12:06:59.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.826+0000 7f665e7fc700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6644056d20 0x7f66440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:59.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.826+0000 7f665e7fc700 1 -- 172.21.15.187:0/3163527035 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f666806fbf0 con 0x7f66701373d0 2020-11-20T12:06:59.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.826+0000 7f666e59c700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6644056d20 0x7f66440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:59.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.827+0000 7f666e59c700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6644056d20 0x7f66440591e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f666000b920 tx=0x7f6660015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:59.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:06:59.829+0000 7f665e7fc700 1 -- 172.21.15.187:0/3163527035 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6668045370 con 0x7f66701373d0 2020-11-20T12:07:00.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.015+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f667009d720 con 0x7f66701373d0 2020-11-20T12:07:00.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.016+0000 7f665e7fc700 1 -- 172.21.15.187:0/3163527035 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v36) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6668048990 con 0x7f66701373d0 2020-11-20T12:07:00.020 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:07:00.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.018+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6644056d20 msgr2=0x7f66440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.018+0000 7f666f59e700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6644056d20 0x7f66440591e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f666000b920 tx=0x7f6660015000).stop 2020-11-20T12:07:00.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.018+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66701373d0 msgr2=0x7f6670135e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.018+0000 7f666f59e700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66701373d0 0x7f6670135e50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f666800c860 tx=0x7f666800cc20).stop 2020-11-20T12:07:00.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.019+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 shutdown_connections 2020-11-20T12:07:00.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.019+0000 7f666f59e700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6644056d20 0x7f66440591e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.019+0000 7f666f59e700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66700bd620 0x7f6670134300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.019+0000 7f666f59e700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66701373d0 0x7f6670135e50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.019+0000 7f666f59e700 1 --2- 172.21.15.187:0/3163527035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6670140820 0x7f66700a2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.019+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 >> 172.21.15.187:0/3163527035 conn(0x7f6670060c10 msgr2=0x7f66700bcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:00.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.019+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 shutdown_connections 2020-11-20T12:07:00.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.019+0000 7f666f59e700 1 -- 172.21.15.187:0/3163527035 wait complete. 2020-11-20T12:07:00.047 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:07:00.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.290+0000 7ff52f59e700 1 Processor -- start 2020-11-20T12:07:00.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.290+0000 7ff52f59e700 1 -- start start 2020-11-20T12:07:00.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bfbe0 0x7ff5301480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff53009f560 0x7ff5301485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52f59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53014d110 con 0x7ff5300bf800 2020-11-20T12:07:00.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52f59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53014c9d0 con 0x7ff53009f560 2020-11-20T12:07:00.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff530148aa0 con 0x7ff5300bfbe0 2020-11-20T12:07:00.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52e59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff5300bf800 0x7ff53014c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39410/0 (socket says 172.21.15.187:39410) 2020-11-20T12:07:00.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52e59c700 1 -- 172.21.15.187:0/3338837771 learned_addr learned my addr 172.21.15.187:0/3338837771 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:00.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52ed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff53009f560 0x7ff5301485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.292+0000 7ff52dd9b700 1 --2- 172.21.15.187:0/3338837771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bfbe0 0x7ff5301480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52ed9d700 1 -- 172.21.15.187:0/3338837771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bfbe0 msgr2=0x7ff5301480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:00.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52ed9d700 1 --2- 172.21.15.187:0/3338837771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bfbe0 0x7ff5301480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52ed9d700 1 -- 172.21.15.187:0/3338837771 >> v1:172.21.15.187:6789/0 conn(0x7ff5300bf800 legacy=0x7ff53014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52ed9d700 1 -- 172.21.15.187:0/3338837771 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53014cbd0 con 0x7ff53009f560 2020-11-20T12:07:00.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52d59a700 1 -- 172.21.15.187:0/3338837771 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486995000 0 0) 0x7ff53014d110 con 0x7ff5300bf800 2020-11-20T12:07:00.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52dd9b700 1 --2- 172.21.15.187:0/3338837771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bfbe0 0x7ff5301480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:00.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52ed9d700 1 --2- 172.21.15.187:0/3338837771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53009f560 0x7ff5301485b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7ff528009cc0 tx=0x7ff5280091d0).ready entity=mon.1 client_cookie=711f3301aa203f36 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:00.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52d59a700 1 -- 172.21.15.187:0/3338837771 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52800bc10 con 0x7ff53009f560 2020-11-20T12:07:00.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52d59a700 1 -- 172.21.15.187:0/3338837771 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52800bd70 con 0x7ff53009f560 2020-11-20T12:07:00.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.293+0000 7ff52d59a700 1 -- 172.21.15.187:0/3338837771 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528011610 con 0x7ff53009f560 2020-11-20T12:07:00.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.294+0000 7ff52f59e700 1 -- 172.21.15.187:0/3338837771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53009f560 msgr2=0x7ff5301485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.294+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3338837771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53009f560 0x7ff5301485b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7ff528009cc0 tx=0x7ff5280091d0).stop 2020-11-20T12:07:00.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.294+0000 7ff52f59e700 1 -- 172.21.15.187:0/3338837771 shutdown_connections 2020-11-20T12:07:00.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.294+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3338837771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53009f560 0x7ff5301485b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.294+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3338837771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bfbe0 0x7ff5301480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.294+0000 7ff52f59e700 1 -- 172.21.15.187:0/3338837771 >> 172.21.15.187:0/3338837771 conn(0x7ff530068370 msgr2=0x7ff5300bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:00.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.294+0000 7ff52f59e700 1 -- 172.21.15.187:0/3338837771 shutdown_connections 2020-11-20T12:07:00.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.294+0000 7ff52f59e700 1 -- 172.21.15.187:0/3338837771 wait complete. 2020-11-20T12:07:00.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.296+0000 7ff52f59e700 1 Processor -- start 2020-11-20T12:07:00.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.296+0000 7ff52f59e700 1 -- start start 2020-11-20T12:07:00.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.297+0000 7ff52f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff53009f560 0x7ff530142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bf800 0x7ff5300b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5300b59a0 0x7ff53014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff530143160 con 0x7ff53009f560 2020-11-20T12:07:00.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300b5e30 con 0x7ff5300b59a0 2020-11-20T12:07:00.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300b60d0 con 0x7ff5300bf800 2020-11-20T12:07:00.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5300b59a0 0x7ff53014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5300b59a0 0x7ff53014d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42814/0 (socket says 172.21.15.187:42814) 2020-11-20T12:07:00.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52ed9d700 1 -- 172.21.15.187:0/3355733711 learned_addr learned my addr 172.21.15.187:0/3355733711 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:00.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52e59c700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff53009f560 0x7ff530142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52dd9b700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bf800 0x7ff5300b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52ed9d700 1 -- 172.21.15.187:0/3355733711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bf800 msgr2=0x7ff5300b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52ed9d700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bf800 0x7ff5300b54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52ed9d700 1 -- 172.21.15.187:0/3355733711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff53009f560 msgr2=0x7ff530142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.298+0000 7ff52ed9d700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff53009f560 0x7ff530142c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.299+0000 7ff52ed9d700 1 -- 172.21.15.187:0/3355733711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53015bb90 con 0x7ff5300b59a0 2020-11-20T12:07:00.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.299+0000 7ff52ed9d700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5300b59a0 0x7ff53014d320 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7ff528009c90 tx=0x7ff528016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:00.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.299+0000 7ff51e7fc700 1 -- 172.21.15.187:0/3355733711 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528021070 con 0x7ff5300b59a0 2020-11-20T12:07:00.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.299+0000 7ff51e7fc700 1 -- 172.21.15.187:0/3355733711 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff528010e90 con 0x7ff5300b59a0 2020-11-20T12:07:00.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.299+0000 7ff51e7fc700 1 -- 172.21.15.187:0/3355733711 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528006ea0 con 0x7ff5300b59a0 2020-11-20T12:07:00.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.299+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53014d810 con 0x7ff5300b59a0 2020-11-20T12:07:00.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.299+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff530148dd0 con 0x7ff5300b59a0 2020-11-20T12:07:00.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.300+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff530080050 con 0x7ff5300b59a0 2020-11-20T12:07:00.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.301+0000 7ff51e7fc700 1 -- 172.21.15.187:0/3355733711 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff528017070 con 0x7ff5300b59a0 2020-11-20T12:07:00.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.302+0000 7ff51e7fc700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff504056d20 0x7ff5040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.302+0000 7ff51e7fc700 1 -- 172.21.15.187:0/3355733711 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff5280720e0 con 0x7ff5300b59a0 2020-11-20T12:07:00.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.302+0000 7ff52e59c700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff504056d20 0x7ff5040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.303+0000 7ff52e59c700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff504056d20 0x7ff5040591e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ff5200011a0 tx=0x7ff520009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:00.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.306+0000 7ff51e7fc700 1 -- 172.21.15.187:0/3355733711 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff528047860 con 0x7ff5300b59a0 2020-11-20T12:07:00.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.490+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff53014c690 con 0x7ff5300b59a0 2020-11-20T12:07:00.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.491+0000 7ff51e7fc700 1 -- 172.21.15.187:0/3355733711 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v36) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff528015070 con 0x7ff5300b59a0 2020-11-20T12:07:00.496 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:07:00.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.493+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff504056d20 msgr2=0x7ff5040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.493+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff504056d20 0x7ff5040591e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ff5200011a0 tx=0x7ff520009040).stop 2020-11-20T12:07:00.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.493+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5300b59a0 msgr2=0x7ff53014d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.493+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5300b59a0 0x7ff53014d320 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7ff528009c90 tx=0x7ff528016000).stop 2020-11-20T12:07:00.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.493+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 shutdown_connections 2020-11-20T12:07:00.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.494+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff504056d20 0x7ff5040591e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.494+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff53009f560 0x7ff530142c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.494+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5300b59a0 0x7ff53014d320 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.494+0000 7ff52f59e700 1 --2- 172.21.15.187:0/3355733711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5300bf800 0x7ff5300b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.494+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 >> 172.21.15.187:0/3355733711 conn(0x7ff530068370 msgr2=0x7ff5300bbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:00.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.494+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 shutdown_connections 2020-11-20T12:07:00.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.494+0000 7ff52f59e700 1 -- 172.21.15.187:0/3355733711 wait complete. 2020-11-20T12:07:00.521 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:07:00.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.747+0000 7f3e017d5700 1 Processor -- start 2020-11-20T12:07:00.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.747+0000 7f3e017d5700 1 -- start start 2020-11-20T12:07:00.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.748+0000 7f3e017d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 0x7f3dfc0b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.748+0000 7f3e017d5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3dfc065880 0x7f3dfc144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.748+0000 7f3e017d5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3dfc0b6c10 0x7f3dfc145350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.748+0000 7f3e017d5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc0b08e0 con 0x7f3dfc0b6c10 2020-11-20T12:07:00.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.748+0000 7f3e017d5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc0b0b80 con 0x7f3dfc065880 2020-11-20T12:07:00.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.748+0000 7f3e017d5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0b05f0 con 0x7f3dfc0654a0 2020-11-20T12:07:00.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfa7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3dfc065880 0x7f3dfc144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfa7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3dfc065880 0x7f3dfc144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42820/0 (socket says 172.21.15.187:42820) 2020-11-20T12:07:00.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfa7fc700 1 -- 172.21.15.187:0/819519572 learned_addr learned my addr 172.21.15.187:0/819519572 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:00.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfaffd700 1 --2- 172.21.15.187:0/819519572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 0x7f3dfc0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfa7fc700 1 -- 172.21.15.187:0/819519572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 msgr2=0x7f3dfc0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:00.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfa7fc700 1 --2- 172.21.15.187:0/819519572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 0x7f3dfc0b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfb7fe700 1 --2- 172.21.15.187:0/819519572 >> v2:172.21.15.5:3300/0 conn(0x7f3dfc0b6c10 0x7f3dfc145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfa7fc700 1 -- 172.21.15.187:0/819519572 >> v2:172.21.15.5:3300/0 conn(0x7f3dfc0b6c10 msgr2=0x7f3dfc145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:00.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfa7fc700 1 --2- 172.21.15.187:0/819519572 >> v2:172.21.15.5:3300/0 conn(0x7f3dfc0b6c10 0x7f3dfc145350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfa7fc700 1 -- 172.21.15.187:0/819519572 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0b5880 con 0x7f3dfc065880 2020-11-20T12:07:00.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.749+0000 7f3dfb7fe700 1 --2- 172.21.15.187:0/819519572 >> v2:172.21.15.5:3300/0 conn(0x7f3dfc0b6c10 0x7f3dfc145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:00.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.750+0000 7f3dfa7fc700 1 --2- 172.21.15.187:0/819519572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc065880 0x7f3dfc144e60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f3dfc064860 tx=0x7f3df0006e50).ready entity=mon.1 client_cookie=4fd89becec68b202 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:00.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.750+0000 7f3df9ffb700 1 -- 172.21.15.187:0/819519572 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0006030 con 0x7f3dfc065880 2020-11-20T12:07:00.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.750+0000 7f3df9ffb700 1 -- 172.21.15.187:0/819519572 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df0006190 con 0x7f3dfc065880 2020-11-20T12:07:00.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.750+0000 7f3df9ffb700 1 -- 172.21.15.187:0/819519572 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0006510 con 0x7f3dfc065880 2020-11-20T12:07:00.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 -- 172.21.15.187:0/819519572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc065880 msgr2=0x7f3dfc144e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 --2- 172.21.15.187:0/819519572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc065880 0x7f3dfc144e60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f3dfc064860 tx=0x7f3df0006e50).stop 2020-11-20T12:07:00.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 -- 172.21.15.187:0/819519572 shutdown_connections 2020-11-20T12:07:00.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 --2- 172.21.15.187:0/819519572 >> v2:172.21.15.5:3300/0 conn(0x7f3dfc0b6c10 0x7f3dfc145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 --2- 172.21.15.187:0/819519572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 0x7f3dfc0b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 --2- 172.21.15.187:0/819519572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc065880 0x7f3dfc144e60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 -- 172.21.15.187:0/819519572 >> 172.21.15.187:0/819519572 conn(0x7f3dfc0bacd0 msgr2=0x7f3dfc0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:00.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 -- 172.21.15.187:0/819519572 shutdown_connections 2020-11-20T12:07:00.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.751+0000 7f3e017d5700 1 -- 172.21.15.187:0/819519572 wait complete. 2020-11-20T12:07:00.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.753+0000 7f3e017d5700 1 Processor -- start 2020-11-20T12:07:00.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.753+0000 7f3e017d5700 1 -- start start 2020-11-20T12:07:00.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3e017d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 0x7f3dfc09e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3e017d5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3dfc0b6c10 0x7f3dfc158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3e017d5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc0b9970 0x7f3dfc09c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3e017d5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc1591d0 con 0x7f3dfc0b6c10 2020-11-20T12:07:00.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3e017d5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc09ddf0 con 0x7f3dfc0b9970 2020-11-20T12:07:00.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3e017d5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc09e090 con 0x7f3dfc0654a0 2020-11-20T12:07:00.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3dfb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc0b9970 0x7f3dfc09c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3dfb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc0b9970 0x7f3dfc09c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42828/0 (socket says 172.21.15.187:42828) 2020-11-20T12:07:00.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3dfb7fe700 1 -- 172.21.15.187:0/3104719831 learned_addr learned my addr 172.21.15.187:0/3104719831 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:00.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3dfaffd700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 0x7f3dfc09e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.754+0000 7f3dfa7fc700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3dfc0b6c10 0x7f3dfc158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3dfb7fe700 1 -- 172.21.15.187:0/3104719831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 msgr2=0x7f3dfc09e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3dfb7fe700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 0x7f3dfc09e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3dfb7fe700 1 -- 172.21.15.187:0/3104719831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3dfc0b6c10 msgr2=0x7f3dfc158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3dfb7fe700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3dfc0b6c10 0x7f3dfc158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3dfb7fe700 1 -- 172.21.15.187:0/3104719831 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc09e330 con 0x7f3dfc0b9970 2020-11-20T12:07:00.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3dfb7fe700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc0b9970 0x7f3dfc09c150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f3df400cb50 tx=0x7f3df4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:00.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3de2ffd700 1 -- 172.21.15.187:0/3104719831 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4007900 con 0x7f3dfc0b9970 2020-11-20T12:07:00.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3de2ffd700 1 -- 172.21.15.187:0/3104719831 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df4007a60 con 0x7f3dfc0b9970 2020-11-20T12:07:00.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.755+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc09c640 con 0x7f3dfc0b9970 2020-11-20T12:07:00.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.756+0000 7f3de2ffd700 1 -- 172.21.15.187:0/3104719831 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4007de0 con 0x7f3dfc0b9970 2020-11-20T12:07:00.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.756+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc148340 con 0x7f3dfc0b9970 2020-11-20T12:07:00.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.757+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dfc09e4c0 con 0x7f3dfc0b9970 2020-11-20T12:07:00.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.757+0000 7f3de2ffd700 1 -- 172.21.15.187:0/3104719831 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3df4011620 con 0x7f3dfc0b9970 2020-11-20T12:07:00.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.758+0000 7f3de2ffd700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dd8056d50 0x7f3dd8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:00.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.758+0000 7f3de2ffd700 1 -- 172.21.15.187:0/3104719831 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3df406ff50 con 0x7f3dfc0b9970 2020-11-20T12:07:00.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.758+0000 7f3dfaffd700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dd8056d50 0x7f3dd8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:00.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.759+0000 7f3dfaffd700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dd8056d50 0x7f3dd8059210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3dec00d990 tx=0x7f3dec008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:00.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.761+0000 7f3de2ffd700 1 -- 172.21.15.187:0/3104719831 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3df40456d0 con 0x7f3dfc0b9970 2020-11-20T12:07:00.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.946+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3dfc063600 con 0x7f3dfc0b9970 2020-11-20T12:07:00.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.947+0000 7f3de2ffd700 1 -- 172.21.15.187:0/3104719831 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v36) v1 ==== 84+0+143 (secure 0 0 0) 0x7f3df4048cf0 con 0x7f3dfc0b9970 2020-11-20T12:07:00.952 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:00.952 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:07:00.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:07:00.953 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:07:00.953 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:07:00.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:07:00.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:07:00.954 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:07:00.954 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:07:00.954 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:07:00.954 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:07:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.949+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dd8056d50 msgr2=0x7f3dd8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.949+0000 7f3e017d5700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dd8056d50 0x7f3dd8059210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3dec00d990 tx=0x7f3dec008040).stop 2020-11-20T12:07:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.949+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc0b9970 msgr2=0x7f3dfc09c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.949+0000 7f3e017d5700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc0b9970 0x7f3dfc09c150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f3df400cb50 tx=0x7f3df4008040).stop 2020-11-20T12:07:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.950+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 shutdown_connections 2020-11-20T12:07:00.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.950+0000 7f3e017d5700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dd8056d50 0x7f3dd8059210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.950+0000 7f3e017d5700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dfc0b9970 0x7f3dfc09c150 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.950+0000 7f3e017d5700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3dfc0654a0 0x7f3dfc09e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.950+0000 7f3e017d5700 1 --2- 172.21.15.187:0/3104719831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3dfc0b6c10 0x7f3dfc158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:00.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.950+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 >> 172.21.15.187:0/3104719831 conn(0x7f3dfc0bacd0 msgr2=0x7f3dfc144d60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:00.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.950+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 shutdown_connections 2020-11-20T12:07:00.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:00.950+0000 7f3e017d5700 1 -- 172.21.15.187:0/3104719831 wait complete. 2020-11-20T12:07:00.978 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:07:00.979 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:07:01.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.223+0000 7f18dae9e700 1 Processor -- start 2020-11-20T12:07:01.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.223+0000 7f18dae9e700 1 -- start start 2020-11-20T12:07:01.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.224+0000 7f18dae9e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d40683f0 0x7f18d4066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.224+0000 7f18dae9e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f18d41487e0 0x7f18d4148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.224+0000 7f18dae9e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d414c170 con 0x7f18d4145cc0 2020-11-20T12:07:01.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.224+0000 7f18dae9e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d414c410 con 0x7f18d41487e0 2020-11-20T12:07:01.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.224+0000 7f18dae9e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d414c6b0 con 0x7f18d40683f0 2020-11-20T12:07:01.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.225+0000 7f18d9e9c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f18d4145cc0 0x7f18d4065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39438/0 (socket says 172.21.15.187:39438) 2020-11-20T12:07:01.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.225+0000 7f18d9e9c700 1 -- 172.21.15.187:0/3023482882 learned_addr learned my addr 172.21.15.187:0/3023482882 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:01.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.225+0000 7f18da69d700 1 --2- 172.21.15.187:0/3023482882 >> v2:172.21.15.187:3300/0 conn(0x7f18d41487e0 0x7f18d4148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.225+0000 7f18d969b700 1 --2- 172.21.15.187:0/3023482882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d40683f0 0x7f18d4066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18d8e9a700 1 -- 172.21.15.187:0/3023482882 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811117428 0 0) 0x7f18d414c170 con 0x7f18d4145cc0 2020-11-20T12:07:01.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18da69d700 1 -- 172.21.15.187:0/3023482882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d40683f0 msgr2=0x7f18d4066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:01.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18da69d700 1 --2- 172.21.15.187:0/3023482882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d40683f0 0x7f18d4066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18da69d700 1 -- 172.21.15.187:0/3023482882 >> v1:172.21.15.187:6789/0 conn(0x7f18d4145cc0 legacy=0x7f18d4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18da69d700 1 -- 172.21.15.187:0/3023482882 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d40658a0 con 0x7f18d41487e0 2020-11-20T12:07:01.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18da69d700 1 --2- 172.21.15.187:0/3023482882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d41487e0 0x7f18d4148c90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f18cc009c90 tx=0x7f18cc0091d0).ready entity=mon.1 client_cookie=101fed99ffd0ceb5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:01.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18d8e9a700 1 -- 172.21.15.187:0/3023482882 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc00bbc0 con 0x7f18d41487e0 2020-11-20T12:07:01.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18d8e9a700 1 -- 172.21.15.187:0/3023482882 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc00bd20 con 0x7f18d41487e0 2020-11-20T12:07:01.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.226+0000 7f18d8e9a700 1 -- 172.21.15.187:0/3023482882 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc011610 con 0x7f18d41487e0 2020-11-20T12:07:01.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.227+0000 7f18dae9e700 1 -- 172.21.15.187:0/3023482882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d41487e0 msgr2=0x7f18d4148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.227+0000 7f18dae9e700 1 --2- 172.21.15.187:0/3023482882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d41487e0 0x7f18d4148c90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f18cc009c90 tx=0x7f18cc0091d0).stop 2020-11-20T12:07:01.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.227+0000 7f18dae9e700 1 -- 172.21.15.187:0/3023482882 shutdown_connections 2020-11-20T12:07:01.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.227+0000 7f18dae9e700 1 --2- 172.21.15.187:0/3023482882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d41487e0 0x7f18d4148c90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.227+0000 7f18dae9e700 1 --2- 172.21.15.187:0/3023482882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d40683f0 0x7f18d4066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.227+0000 7f18dae9e700 1 -- 172.21.15.187:0/3023482882 >> 172.21.15.187:0/3023482882 conn(0x7f18d40bab30 msgr2=0x7f18d40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:01.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.227+0000 7f18dae9e700 1 -- 172.21.15.187:0/3023482882 shutdown_connections 2020-11-20T12:07:01.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.227+0000 7f18dae9e700 1 -- 172.21.15.187:0/3023482882 wait complete. 2020-11-20T12:07:01.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.229+0000 7f18dae9e700 1 Processor -- start 2020-11-20T12:07:01.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.229+0000 7f18dae9e700 1 -- start start 2020-11-20T12:07:01.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.230+0000 7f18dae9e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d40683f0 0x7f18d4158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.230+0000 7f18dae9e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4145cc0 0x7f18d409e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.230+0000 7f18dae9e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d41487e0 0x7f18d409ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.230+0000 7f18dae9e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4157ea0 con 0x7f18d4145cc0 2020-11-20T12:07:01.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.230+0000 7f18dae9e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4158140 con 0x7f18d40683f0 2020-11-20T12:07:01.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.230+0000 7f18dae9e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41583e0 con 0x7f18d41487e0 2020-11-20T12:07:01.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.231+0000 7f18d9e9c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d40683f0 0x7f18d4158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.231+0000 7f18d9e9c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d40683f0 0x7f18d4158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42838/0 (socket says 172.21.15.187:42838) 2020-11-20T12:07:01.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.231+0000 7f18d9e9c700 1 -- 172.21.15.187:0/1393125063 learned_addr learned my addr 172.21.15.187:0/1393125063 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:01.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.231+0000 7f18da69d700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d41487e0 0x7f18d409ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.231+0000 7f18d969b700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4145cc0 0x7f18d409e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18d9e9c700 1 -- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d41487e0 msgr2=0x7f18d409ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18d9e9c700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d41487e0 0x7f18d409ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18d9e9c700 1 -- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4145cc0 msgr2=0x7f18d409e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18d9e9c700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4145cc0 0x7f18d409e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18d9e9c700 1 -- 172.21.15.187:0/1393125063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d409e880 con 0x7f18d40683f0 2020-11-20T12:07:01.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18d969b700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4145cc0 0x7f18d409e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:01.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18d9e9c700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d40683f0 0x7f18d4158880 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f18d000c720 tx=0x7f18d0007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:01.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18c1ffb700 1 -- 172.21.15.187:0/1393125063 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0007320 con 0x7f18d40683f0 2020-11-20T12:07:01.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18c1ffb700 1 -- 172.21.15.187:0/1393125063 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d0007480 con 0x7f18d40683f0 2020-11-20T12:07:01.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d409f320 con 0x7f18d40683f0 2020-11-20T12:07:01.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18c1ffb700 1 -- 172.21.15.187:0/1393125063 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0005800 con 0x7f18d40683f0 2020-11-20T12:07:01.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.232+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d414b520 con 0x7f18d40683f0 2020-11-20T12:07:01.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.233+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d41585a0 con 0x7f18d40683f0 2020-11-20T12:07:01.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.234+0000 7f18c1ffb700 1 -- 172.21.15.187:0/1393125063 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f18d0010030 con 0x7f18d40683f0 2020-11-20T12:07:01.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.235+0000 7f18c1ffb700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18ac056d20 0x7f18ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.235+0000 7f18c1ffb700 1 -- 172.21.15.187:0/1393125063 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f18d006faf0 con 0x7f18d40683f0 2020-11-20T12:07:01.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.235+0000 7f18d969b700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18ac056d20 0x7f18ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.235+0000 7f18d969b700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18ac056d20 0x7f18ac0591e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f18d40a0cb0 tx=0x7f18c4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:01.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.238+0000 7f18c1ffb700 1 -- 172.21.15.187:0/1393125063 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f18d00451a0 con 0x7f18d40683f0 2020-11-20T12:07:01.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.410+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f18d4061a20 con 0x7f18d40683f0 2020-11-20T12:07:01.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.550+0000 7f18c1ffb700 1 -- 172.21.15.187:0/1393125063 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v37) v1 ==== 133+0+0 (secure 0 0 0) 0x7f18d0007c00 con 0x7f18d40683f0 2020-11-20T12:07:01.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.554+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18ac056d20 msgr2=0x7f18ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.554+0000 7f18dae9e700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18ac056d20 0x7f18ac0591e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f18d40a0cb0 tx=0x7f18c4008040).stop 2020-11-20T12:07:01.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.554+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d40683f0 msgr2=0x7f18d4158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.554+0000 7f18dae9e700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d40683f0 0x7f18d4158880 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f18d000c720 tx=0x7f18d0007ac0).stop 2020-11-20T12:07:01.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.555+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 shutdown_connections 2020-11-20T12:07:01.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.555+0000 7f18dae9e700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18ac056d20 0x7f18ac0591e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.555+0000 7f18dae9e700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d41487e0 0x7f18d409ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.555+0000 7f18dae9e700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d40683f0 0x7f18d4158880 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.555+0000 7f18dae9e700 1 --2- 172.21.15.187:0/1393125063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4145cc0 0x7f18d409e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.555+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 >> 172.21.15.187:0/1393125063 conn(0x7f18d40bab30 msgr2=0x7f18d4144550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:01.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.555+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 shutdown_connections 2020-11-20T12:07:01.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.555+0000 7f18dae9e700 1 -- 172.21.15.187:0/1393125063 wait complete. 2020-11-20T12:07:01.562 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:07:01.586 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:07:01.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.844+0000 7f231ffff700 1 Processor -- start 2020-11-20T12:07:01.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.845+0000 7f231ffff700 1 -- start start 2020-11-20T12:07:01.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.846+0000 7f231ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23201484d0 0x7f2320061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.846+0000 7f231ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f23200622c0 0x7f23200b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.846+0000 7f231ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23200b1300 con 0x7f2320066f70 2020-11-20T12:07:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.846+0000 7f231ffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2320067730 con 0x7f23200622c0 2020-11-20T12:07:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.846+0000 7f231ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23200b15a0 con 0x7f23201484d0 2020-11-20T12:07:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.846+0000 7f231e7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f23200622c0 0x7f23200b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.846+0000 7f231f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2320066f70 0x7f23200b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39456/0 (socket says 172.21.15.187:39456) 2020-11-20T12:07:01.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.846+0000 7f231f7fe700 1 -- 172.21.15.187:0/132281959 learned_addr learned my addr 172.21.15.187:0/132281959 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:01.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231effd700 1 --2- 172.21.15.187:0/132281959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23201484d0 0x7f2320061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231e7fc700 1 -- 172.21.15.187:0/132281959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23201484d0 msgr2=0x7f2320061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231e7fc700 1 --2- 172.21.15.187:0/132281959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23201484d0 0x7f2320061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231e7fc700 1 -- 172.21.15.187:0/132281959 >> v1:172.21.15.187:6789/0 conn(0x7f2320066f70 legacy=0x7f23200b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231dffb700 1 -- 172.21.15.187:0/132281959 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626260236 0 0) 0x7f23200b1300 con 0x7f2320066f70 2020-11-20T12:07:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231e7fc700 1 -- 172.21.15.187:0/132281959 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200b1840 con 0x7f23200622c0 2020-11-20T12:07:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231e7fc700 1 --2- 172.21.15.187:0/132281959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23200622c0 0x7f23200b5640 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f231400b9a0 tx=0x7f2314006df0).ready entity=mon.1 client_cookie=dfce92a0e6c23b99 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231dffb700 1 -- 172.21.15.187:0/132281959 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23140097b0 con 0x7f23200622c0 2020-11-20T12:07:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231dffb700 1 -- 172.21.15.187:0/132281959 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2314009910 con 0x7f23200622c0 2020-11-20T12:07:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.847+0000 7f231dffb700 1 -- 172.21.15.187:0/132281959 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314009c90 con 0x7f23200622c0 2020-11-20T12:07:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.848+0000 7f231ffff700 1 -- 172.21.15.187:0/132281959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23200622c0 msgr2=0x7f23200b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.848+0000 7f231ffff700 1 --2- 172.21.15.187:0/132281959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23200622c0 0x7f23200b5640 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f231400b9a0 tx=0x7f2314006df0).stop 2020-11-20T12:07:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.848+0000 7f231ffff700 1 -- 172.21.15.187:0/132281959 shutdown_connections 2020-11-20T12:07:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.848+0000 7f231ffff700 1 --2- 172.21.15.187:0/132281959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23201484d0 0x7f2320061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.848+0000 7f231ffff700 1 --2- 172.21.15.187:0/132281959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23200622c0 0x7f23200b5640 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.848+0000 7f231ffff700 1 -- 172.21.15.187:0/132281959 >> 172.21.15.187:0/132281959 conn(0x7f23200bab30 msgr2=0x7f23200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:01.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.848+0000 7f231ffff700 1 -- 172.21.15.187:0/132281959 shutdown_connections 2020-11-20T12:07:01.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.848+0000 7f231ffff700 1 -- 172.21.15.187:0/132281959 wait complete. 2020-11-20T12:07:01.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.850+0000 7f231ffff700 1 Processor -- start 2020-11-20T12:07:01.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.850+0000 7f231ffff700 1 -- start start 2020-11-20T12:07:01.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23200622c0 0x7f232009b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2320066f70 0x7f232009bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23201484d0 0x7f232009c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232015c1a0 con 0x7f23200622c0 2020-11-20T12:07:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23200b4660 con 0x7f23201484d0 2020-11-20T12:07:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23200b4320 con 0x7f2320066f70 2020-11-20T12:07:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23201484d0 0x7f232009c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23201484d0 0x7f232009c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42856/0 (socket says 172.21.15.187:42856) 2020-11-20T12:07:01.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.851+0000 7f231f7fe700 1 -- 172.21.15.187:0/918942586 learned_addr learned my addr 172.21.15.187:0/918942586 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:01.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231effd700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23200622c0 0x7f232009b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231e7fc700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2320066f70 0x7f232009bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231f7fe700 1 -- 172.21.15.187:0/918942586 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2320066f70 msgr2=0x7f232009bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231f7fe700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2320066f70 0x7f232009bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231f7fe700 1 -- 172.21.15.187:0/918942586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23200622c0 msgr2=0x7f232009b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231f7fe700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23200622c0 0x7f232009b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231f7fe700 1 -- 172.21.15.187:0/918942586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200b3d60 con 0x7f23201484d0 2020-11-20T12:07:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231f7fe700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23201484d0 0x7f232009c290 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f231800cab0 tx=0x7f2318007de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f2306ffd700 1 -- 172.21.15.187:0/918942586 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318005c50 con 0x7f23201484d0 2020-11-20T12:07:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f2306ffd700 1 -- 172.21.15.187:0/918942586 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2318005db0 con 0x7f23201484d0 2020-11-20T12:07:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23200b3f50 con 0x7f23201484d0 2020-11-20T12:07:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f2306ffd700 1 -- 172.21.15.187:0/918942586 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318007340 con 0x7f23201484d0 2020-11-20T12:07:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.852+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232009cd60 con 0x7f23201484d0 2020-11-20T12:07:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.854+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232015c310 con 0x7f23201484d0 2020-11-20T12:07:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.854+0000 7f2306ffd700 1 -- 172.21.15.187:0/918942586 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2318010030 con 0x7f23201484d0 2020-11-20T12:07:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.855+0000 7f2306ffd700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22fc056d20 0x7f22fc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.855+0000 7f2306ffd700 1 -- 172.21.15.187:0/918942586 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f23180700a0 con 0x7f23201484d0 2020-11-20T12:07:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.855+0000 7f231effd700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22fc056d20 0x7f22fc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.856+0000 7f231effd700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22fc056d20 0x7f22fc0591e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f23200b3530 tx=0x7f2310008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:01.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:01.858+0000 7f2306ffd700 1 -- 172.21.15.187:0/918942586 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f23180455b0 con 0x7f23201484d0 2020-11-20T12:07:02.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.051+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f2320061a20 con 0x7f23201484d0 2020-11-20T12:07:02.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.795+0000 7f2306ffd700 1 -- 172.21.15.187:0/918942586 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v39) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2318048bd0 con 0x7f23201484d0 2020-11-20T12:07:02.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.799+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22fc056d20 msgr2=0x7f22fc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:02.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.799+0000 7f231ffff700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22fc056d20 0x7f22fc0591e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f23200b3530 tx=0x7f2310008040).stop 2020-11-20T12:07:02.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.799+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23201484d0 msgr2=0x7f232009c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:02.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.799+0000 7f231ffff700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23201484d0 0x7f232009c290 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f231800cab0 tx=0x7f2318007de0).stop 2020-11-20T12:07:02.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.800+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 shutdown_connections 2020-11-20T12:07:02.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.800+0000 7f231ffff700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22fc056d20 0x7f22fc0591e0 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0x7f23200b3530 tx=0x7f2310008040).stop 2020-11-20T12:07:02.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.800+0000 7f231ffff700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23200622c0 0x7f232009b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:02.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.800+0000 7f231ffff700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23201484d0 0x7f232009c290 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:02.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.800+0000 7f231ffff700 1 --2- 172.21.15.187:0/918942586 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2320066f70 0x7f232009bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:02.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.801+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 >> 172.21.15.187:0/918942586 conn(0x7f23200bab30 msgr2=0x7f232013b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:02.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.801+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 shutdown_connections 2020-11-20T12:07:02.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:02.801+0000 7f231ffff700 1 -- 172.21.15.187:0/918942586 wait complete. 2020-11-20T12:07:02.808 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:07:02.836 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:07:03.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.077+0000 7f25188e6700 1 Processor -- start 2020-11-20T12:07:03.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.078+0000 7f25188e6700 1 -- start start 2020-11-20T12:07:03.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.079+0000 7f25188e6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f25140675d0 0x7f251414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.079+0000 7f25188e6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25140ba9b0 0x7f2514148400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.079+0000 7f25188e6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251414be70 con 0x7f25140679b0 2020-11-20T12:07:03.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.079+0000 7f25188e6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251414c110 con 0x7f25140675d0 2020-11-20T12:07:03.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.079+0000 7f25188e6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251414c3b0 con 0x7f25140ba9b0 2020-11-20T12:07:03.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.080+0000 7f250affd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f25140679b0 0x7f2514147cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39470/0 (socket says 172.21.15.187:39470) 2020-11-20T12:07:03.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.080+0000 7f250affd700 1 -- 172.21.15.187:0/1873148515 learned_addr learned my addr 172.21.15.187:0/1873148515 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:03.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.080+0000 7f25137fe700 1 --2- 172.21.15.187:0/1873148515 >> v2:172.21.15.5:3300/0 conn(0x7f25140675d0 0x7f251414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.080+0000 7f2513fff700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25140ba9b0 0x7f2514148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.080+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265529416 0 0) 0x7f251414be70 con 0x7f25140679b0 2020-11-20T12:07:03.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.080+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f4003240 con 0x7f25140679b0 2020-11-20T12:07:03.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4021486239 0 0) 0x7f24f4003240 con 0x7f25140679b0 2020-11-20T12:07:03.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25140ba9b0 msgr2=0x7f2514148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:03.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25140ba9b0 0x7f2514148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 >> v2:172.21.15.5:3300/0 conn(0x7f25140675d0 msgr2=0x7f251414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:03.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 --2- 172.21.15.187:0/1873148515 >> v2:172.21.15.5:3300/0 conn(0x7f25140675d0 0x7f251414c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251414cd40 con 0x7f25140679b0 2020-11-20T12:07:03.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f24fc0014e0 con 0x7f25140679b0 2020-11-20T12:07:03.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 >> v1:172.21.15.187:6789/0 conn(0x7f25140679b0 legacy=0x7f2514147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25140ba9b0 0x7f24f40037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f4003ca0 0x7f24f40044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f40049d0 0x7f24f4004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251414c110 con 0x7f24f4003ca0 2020-11-20T12:07:03.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251414c3b0 con 0x7f24f40049d0 2020-11-20T12:07:03.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.081+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4004240 con 0x7f25140ba9b0 2020-11-20T12:07:03.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.082+0000 7f2513fff700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f40049d0 0x7f24f4004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.082+0000 7f25137fe700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25140ba9b0 0x7f24f40037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.082+0000 7f250affd700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f4003ca0 0x7f24f40044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.082+0000 7f2513fff700 1 -- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25140ba9b0 msgr2=0x7f24f40037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:03.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.082+0000 7f2513fff700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25140ba9b0 0x7f24f40037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.083+0000 7f2513fff700 1 -- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f4003ca0 msgr2=0x7f24f40044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:03.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.083+0000 7f2513fff700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f4003ca0 0x7f24f40044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.083+0000 7f2513fff700 1 -- 172.21.15.187:0/1873148515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f4003240 con 0x7f24f40049d0 2020-11-20T12:07:03.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.083+0000 7f2513fff700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f40049d0 0x7f24f4004e50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f25140badb0 tx=0x7f250c0091d0).ready entity=mon.1 client_cookie=d6af6a1c5204ae71 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.083+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0095c0 con 0x7f24f40049d0 2020-11-20T12:07:03.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.083+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c00ecf0 con 0x7f24f40049d0 2020-11-20T12:07:03.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.083+0000 7f2512ffd700 1 -- 172.21.15.187:0/1873148515 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c012600 con 0x7f24f40049d0 2020-11-20T12:07:03.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.084+0000 7f25188e6700 1 -- 172.21.15.187:0/1873148515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f40049d0 msgr2=0x7f24f4004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.084+0000 7f25188e6700 1 --2- 172.21.15.187:0/1873148515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f40049d0 0x7f24f4004e50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f25140badb0 tx=0x7f250c0091d0).stop 2020-11-20T12:07:03.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.084+0000 7f2513fff700 1 -- 172.21.15.187:0/1873148515 reap_dead start 2020-11-20T12:07:03.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.084+0000 7f25188e6700 1 -- 172.21.15.187:0/1873148515 shutdown_connections 2020-11-20T12:07:03.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.084+0000 7f25188e6700 1 -- 172.21.15.187:0/1873148515 >> 172.21.15.187:0/1873148515 conn(0x7f25140bce50 msgr2=0x7f25140bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:03.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.084+0000 7f25188e6700 1 -- 172.21.15.187:0/1873148515 shutdown_connections 2020-11-20T12:07:03.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.084+0000 7f25188e6700 1 -- 172.21.15.187:0/1873148515 wait complete. 2020-11-20T12:07:03.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.085+0000 7f25188e6700 1 Processor -- start 2020-11-20T12:07:03.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.086+0000 7f25188e6700 1 -- start start 2020-11-20T12:07:03.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.086+0000 7f25188e6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f251413efe0 0x7f251413e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f25188e6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f251413f3c0 0x7f251413e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f25188e6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f251409d790 0x7f2514093920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f25188e6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251413eb60 con 0x7f251409d790 2020-11-20T12:07:03.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f25188e6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251409cb80 con 0x7f251413f3c0 2020-11-20T12:07:03.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f25188e6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251409dce0 con 0x7f251413efe0 2020-11-20T12:07:03.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f250affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f251413f3c0 0x7f251413e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f250affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f251413f3c0 0x7f251413e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42876/0 (socket says 172.21.15.187:42876) 2020-11-20T12:07:03.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f250affd700 1 -- 172.21.15.187:0/4142296383 learned_addr learned my addr 172.21.15.187:0/4142296383 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:03.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f25137fe700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f251413efe0 0x7f251413e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.087+0000 7f2513fff700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f251409d790 0x7f2514093920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250affd700 1 -- 172.21.15.187:0/4142296383 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f251413efe0 msgr2=0x7f251413e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250affd700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f251413efe0 0x7f251413e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250affd700 1 -- 172.21.15.187:0/4142296383 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f251409d790 msgr2=0x7f2514093920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250affd700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f251409d790 0x7f2514093920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250affd700 1 -- 172.21.15.187:0/4142296383 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251409df80 con 0x7f251413f3c0 2020-11-20T12:07:03.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250affd700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f251413f3c0 0x7f251413e670 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f24fc005c20 tx=0x7f24fc0019a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250bfff700 1 -- 172.21.15.187:0/4142296383 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00e8d0 con 0x7f251413f3c0 2020-11-20T12:07:03.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250bfff700 1 -- 172.21.15.187:0/4142296383 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc010040 con 0x7f251413f3c0 2020-11-20T12:07:03.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251409e220 con 0x7f251413f3c0 2020-11-20T12:07:03.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f250bfff700 1 -- 172.21.15.187:0/4142296383 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00ec60 con 0x7f251413f3c0 2020-11-20T12:07:03.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.088+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251409f100 con 0x7f251413f3c0 2020-11-20T12:07:03.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.089+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25140632a0 con 0x7f251413f3c0 2020-11-20T12:07:03.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.090+0000 7f250bfff700 1 -- 172.21.15.187:0/4142296383 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f24fc015400 con 0x7f251413f3c0 2020-11-20T12:07:03.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.091+0000 7f250bfff700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24f0056d50 0x7f24f0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.091+0000 7f250bfff700 1 -- 172.21.15.187:0/4142296383 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f24fc06ffd0 con 0x7f251413f3c0 2020-11-20T12:07:03.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.091+0000 7f25137fe700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24f0056d50 0x7f24f0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.092+0000 7f25137fe700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24f0056d50 0x7f24f0059210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f2504006d00 tx=0x7f2504008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.095+0000 7f250bfff700 1 -- 172.21.15.187:0/4142296383 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f24fc045430 con 0x7f251413f3c0 2020-11-20T12:07:03.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.344+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f251409e3b0 con 0x7f251413f3c0 2020-11-20T12:07:03.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.362+0000 7f250bfff700 1 -- 172.21.15.187:0/4142296383 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 7 and data pool 8 v17) v1 ==== 187+0+0 (secure 0 0 0) 0x7f24fc048a50 con 0x7f251413f3c0 2020-11-20T12:07:03.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.365+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24f0056d50 msgr2=0x7f24f0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.365+0000 7f25188e6700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24f0056d50 0x7f24f0059210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f2504006d00 tx=0x7f2504008040).stop 2020-11-20T12:07:03.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.365+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f251413f3c0 msgr2=0x7f251413e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.365+0000 7f25188e6700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f251413f3c0 0x7f251413e670 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f24fc005c20 tx=0x7f24fc0019a0).stop 2020-11-20T12:07:03.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.367+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 shutdown_connections 2020-11-20T12:07:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.367+0000 7f25188e6700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24f0056d50 0x7f24f0059210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.367+0000 7f25188e6700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f251409d790 0x7f2514093920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.367+0000 7f25188e6700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f251413efe0 0x7f251413e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.367+0000 7f25188e6700 1 --2- 172.21.15.187:0/4142296383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f251413f3c0 0x7f251413e670 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.367+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 >> 172.21.15.187:0/4142296383 conn(0x7f25140bce50 msgr2=0x7f251414b550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:03.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.368+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 shutdown_connections 2020-11-20T12:07:03.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.370+0000 7f25188e6700 1 -- 172.21.15.187:0/4142296383 wait complete. 2020-11-20T12:07:03.379 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 7 and data pool 8 2020-11-20T12:07:03.401 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:07:03.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.620+0000 7ff4923e1700 1 Processor -- start 2020-11-20T12:07:03.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.621+0000 7ff4923e1700 1 -- start start 2020-11-20T12:07:03.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.622+0000 7ff4923e1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff48c0b6650 0x7ff48c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.622+0000 7ff4923e1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff48c066f70 0x7ff48c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.622+0000 7ff4923e1700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48c0b1300 con 0x7ff48c1484d0 2020-11-20T12:07:03.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.622+0000 7ff4923e1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff48c067730 con 0x7ff48c0b6650 2020-11-20T12:07:03.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.622+0000 7ff4923e1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff48c0b15a0 con 0x7ff48c066f70 2020-11-20T12:07:03.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.623+0000 7ff4913df700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff48c1484d0 0x7ff48c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39496/0 (socket says 172.21.15.187:39496) 2020-11-20T12:07:03.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.623+0000 7ff491be0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff48c066f70 0x7ff48c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.623+0000 7ff4913df700 1 -- 172.21.15.187:0/20975938 learned_addr learned my addr 172.21.15.187:0/20975938 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:03.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.623+0000 7ff490bde700 1 --2- 172.21.15.187:0/20975938 >> v2:172.21.15.5:3300/0 conn(0x7ff48c0b6650 0x7ff48c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.623+0000 7ff47bfff700 1 -- 172.21.15.187:0/20975938 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199633318 0 0) 0x7ff48c0b1300 con 0x7ff48c1484d0 2020-11-20T12:07:03.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.623+0000 7ff47bfff700 1 -- 172.21.15.187:0/20975938 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff474003240 con 0x7ff48c1484d0 2020-11-20T12:07:03.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.623+0000 7ff491be0700 1 -- 172.21.15.187:0/20975938 >> v2:172.21.15.5:3300/0 conn(0x7ff48c0b6650 msgr2=0x7ff48c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:03.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.623+0000 7ff491be0700 1 --2- 172.21.15.187:0/20975938 >> v2:172.21.15.5:3300/0 conn(0x7ff48c0b6650 0x7ff48c0afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.624+0000 7ff491be0700 1 -- 172.21.15.187:0/20975938 >> v1:172.21.15.187:6789/0 conn(0x7ff48c1484d0 legacy=0x7ff48c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.624+0000 7ff491be0700 1 -- 172.21.15.187:0/20975938 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c0b1840 con 0x7ff48c066f70 2020-11-20T12:07:03.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.624+0000 7ff491be0700 1 --2- 172.21.15.187:0/20975938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0aff80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7ff48c045000 tx=0x7ff4840091d0).ready entity=mon.1 client_cookie=d524400529407b90 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.624+0000 7ff47bfff700 1 -- 172.21.15.187:0/20975938 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48400b9c0 con 0x7ff48c066f70 2020-11-20T12:07:03.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.624+0000 7ff47bfff700 1 -- 172.21.15.187:0/20975938 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48400bb20 con 0x7ff48c066f70 2020-11-20T12:07:03.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.624+0000 7ff47bfff700 1 -- 172.21.15.187:0/20975938 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48400bea0 con 0x7ff48c066f70 2020-11-20T12:07:03.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.625+0000 7ff4923e1700 1 -- 172.21.15.187:0/20975938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 msgr2=0x7ff48c0aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.625+0000 7ff4923e1700 1 --2- 172.21.15.187:0/20975938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0aff80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7ff48c045000 tx=0x7ff4840091d0).stop 2020-11-20T12:07:03.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.625+0000 7ff4923e1700 1 -- 172.21.15.187:0/20975938 shutdown_connections 2020-11-20T12:07:03.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.625+0000 7ff4923e1700 1 --2- 172.21.15.187:0/20975938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0aff80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.625+0000 7ff4923e1700 1 --2- 172.21.15.187:0/20975938 >> v2:172.21.15.5:3300/0 conn(0x7ff48c0b6650 0x7ff48c0afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.625+0000 7ff4923e1700 1 -- 172.21.15.187:0/20975938 >> 172.21.15.187:0/20975938 conn(0x7ff48c0bab30 msgr2=0x7ff48c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:03.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.625+0000 7ff4923e1700 1 -- 172.21.15.187:0/20975938 shutdown_connections 2020-11-20T12:07:03.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.625+0000 7ff4923e1700 1 -- 172.21.15.187:0/20975938 wait complete. 2020-11-20T12:07:03.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.626+0000 7ff4923e1700 1 Processor -- start 2020-11-20T12:07:03.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.627+0000 7ff4923e1700 1 -- start start 2020-11-20T12:07:03.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.627+0000 7ff4923e1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.627+0000 7ff4923e1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff48c0b6650 0x7ff48c0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.627+0000 7ff4923e1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff48c1484d0 0x7ff48c143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.627+0000 7ff4923e1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c155bc0 con 0x7ff48c1484d0 2020-11-20T12:07:03.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.627+0000 7ff4923e1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c155e60 con 0x7ff48c066f70 2020-11-20T12:07:03.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.627+0000 7ff4923e1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c156100 con 0x7ff48c0b6650 2020-11-20T12:07:03.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff4913df700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff4913df700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42896/0 (socket says 172.21.15.187:42896) 2020-11-20T12:07:03.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff4913df700 1 -- 172.21.15.187:0/3415326091 learned_addr learned my addr 172.21.15.187:0/3415326091 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:03.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff490bde700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff48c0b6650 0x7ff48c0afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff4913df700 1 -- 172.21.15.187:0/3415326091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff48c0b6650 msgr2=0x7ff48c0afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff491be0700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff48c1484d0 0x7ff48c143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff4913df700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff48c0b6650 0x7ff48c0afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff4913df700 1 -- 172.21.15.187:0/3415326091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff48c1484d0 msgr2=0x7ff48c143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff4913df700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff48c1484d0 0x7ff48c143230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.628+0000 7ff4913df700 1 -- 172.21.15.187:0/3415326091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c1563a0 con 0x7ff48c066f70 2020-11-20T12:07:03.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.629+0000 7ff4913df700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0a05b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7ff48800c6e0 tx=0x7ff488007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.629+0000 7ff4797fa700 1 -- 172.21.15.187:0/3415326091 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4880072e0 con 0x7ff48c066f70 2020-11-20T12:07:03.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.629+0000 7ff4797fa700 1 -- 172.21.15.187:0/3415326091 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff488007440 con 0x7ff48c066f70 2020-11-20T12:07:03.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.629+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c0b0140 con 0x7ff48c066f70 2020-11-20T12:07:03.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.629+0000 7ff4797fa700 1 -- 172.21.15.187:0/3415326091 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488005840 con 0x7ff48c066f70 2020-11-20T12:07:03.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.629+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48c156a80 con 0x7ff48c066f70 2020-11-20T12:07:03.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.631+0000 7ff4797fa700 1 -- 172.21.15.187:0/3415326091 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff488010070 con 0x7ff48c066f70 2020-11-20T12:07:03.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.632+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48c156530 con 0x7ff48c066f70 2020-11-20T12:07:03.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.632+0000 7ff4797fa700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff464056d20 0x7ff4640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.632+0000 7ff4797fa700 1 -- 172.21.15.187:0/3415326091 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff48806f510 con 0x7ff48c066f70 2020-11-20T12:07:03.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.633+0000 7ff490bde700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff464056d20 0x7ff4640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.633+0000 7ff490bde700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff464056d20 0x7ff4640591e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff47c006bd0 tx=0x7ff47c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.637+0000 7ff4797fa700 1 -- 172.21.15.187:0/3415326091 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff488044900 con 0x7ff48c066f70 2020-11-20T12:07:03.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.822+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff48c061a20 con 0x7ff48c066f70 2020-11-20T12:07:03.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.823+0000 7ff4797fa700 1 -- 172.21.15.187:0/3415326091 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16791 (secure 0 0 0) 0x7ff488047f20 con 0x7ff48c066f70 2020-11-20T12:07:03.828 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:03.828 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":40,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:07:03.348004+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:07:01.490118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2020-11-20T12:07:02.131720+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:07:03.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.825+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff464056d20 msgr2=0x7ff4640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.825+0000 7ff4923e1700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff464056d20 0x7ff4640591e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff47c006bd0 tx=0x7ff47c008040).stop 2020-11-20T12:07:03.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.825+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 msgr2=0x7ff48c0a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.825+0000 7ff4923e1700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0a05b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7ff48800c6e0 tx=0x7ff488007ac0).stop 2020-11-20T12:07:03.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.826+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 shutdown_connections 2020-11-20T12:07:03.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.826+0000 7ff4923e1700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff464056d20 0x7ff4640591e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.826+0000 7ff4923e1700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff48c1484d0 0x7ff48c143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.826+0000 7ff4923e1700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff48c066f70 0x7ff48c0a05b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.826+0000 7ff4923e1700 1 --2- 172.21.15.187:0/3415326091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff48c0b6650 0x7ff48c0afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.826+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 >> 172.21.15.187:0/3415326091 conn(0x7ff48c0bab30 msgr2=0x7ff48c0b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:03.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.826+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 shutdown_connections 2020-11-20T12:07:03.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:03.826+0000 7ff4923e1700 1 -- 172.21.15.187:0/3415326091 wait complete. 2020-11-20T12:07:03.852 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:07:04.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.093+0000 7fc790e11700 1 Processor -- start 2020-11-20T12:07:04.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.093+0000 7fc790e11700 1 -- start start 2020-11-20T12:07:04.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.094+0000 7fc790e11700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc78c0bf800 0x7fc78c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.094+0000 7fc790e11700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c14d110 con 0x7fc78c09f560 2020-11-20T12:07:04.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.094+0000 7fc790e11700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c14cbd0 con 0x7fc78c0bfbe0 2020-11-20T12:07:04.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.094+0000 7fc790e11700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78c14ce70 con 0x7fc78c0bf800 2020-11-20T12:07:04.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.095+0000 7fc782ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc78c0bfbe0 0x7fc78c1480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39512/0 (socket says 172.21.15.187:39512) 2020-11-20T12:07:04.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.095+0000 7fc782ffd700 1 -- 172.21.15.187:0/265913108 learned_addr learned my addr 172.21.15.187:0/265913108 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:04.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.095+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533503094 0 0) 0x7fc78c14cbd0 con 0x7fc78c0bfbe0 2020-11-20T12:07:04.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.095+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76c003240 con 0x7fc78c0bfbe0 2020-11-20T12:07:04.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.095+0000 7fc78b7fe700 1 --2- 172.21.15.187:0/265913108 >> v2:172.21.15.5:3300/0 conn(0x7fc78c0bf800 0x7fc78c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 987862527 0 0) 0x7fc76c003240 con 0x7fc78c0bfbe0 2020-11-20T12:07:04.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 >> v2:172.21.15.5:3300/0 conn(0x7fc78c0bf800 msgr2=0x7fc78c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:04.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 --2- 172.21.15.187:0/265913108 >> v2:172.21.15.5:3300/0 conn(0x7fc78c0bf800 0x7fc78c14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:04.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 >> v1:172.21.15.5:6789/0 conn(0x7fc78c09f560 legacy=0x7fc78c0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0a4cc0 con 0x7fc78c0bfbe0 2020-11-20T12:07:04.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fc774001520 con 0x7fc78c0bfbe0 2020-11-20T12:07:04.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 >> v1:172.21.15.187:6789/0 conn(0x7fc78c0bfbe0 legacy=0x7fc78c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c003800 0x7fc76c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c0044e0 0x7fc76c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc76c004e50 0x7fc76c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c14ce70 con 0x7fc76c003800 2020-11-20T12:07:04.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c005300 con 0x7fc76c0044e0 2020-11-20T12:07:04.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c00bc20 con 0x7fc76c004e50 2020-11-20T12:07:04.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.096+0000 7fc782ffd700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c0044e0 0x7fc76c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc782ffd700 1 -- 172.21.15.187:0/265913108 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc76c004e50 msgr2=0x7fc76c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:04.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc782ffd700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc76c004e50 0x7fc76c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:04.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc782ffd700 1 -- 172.21.15.187:0/265913108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c003800 msgr2=0x7fc76c003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:04.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc78b7fe700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c003800 0x7fc76c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc782ffd700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c003800 0x7fc76c003c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:04.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc782ffd700 1 -- 172.21.15.187:0/265913108 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c003240 con 0x7fc76c0044e0 2020-11-20T12:07:04.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc78bfff700 1 -- 172.21.15.187:0/265913108 reap_dead start 2020-11-20T12:07:04.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc782ffd700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c0044e0 0x7fc76c004960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fc77400c240 tx=0x7fc774007750).ready entity=mon.1 client_cookie=34d68cb09003b604 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:04.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7740070d0 con 0x7fc76c0044e0 2020-11-20T12:07:04.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc774007230 con 0x7fc76c0044e0 2020-11-20T12:07:04.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.097+0000 7fc78affd700 1 -- 172.21.15.187:0/265913108 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7740055f0 con 0x7fc76c0044e0 2020-11-20T12:07:04.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.098+0000 7fc790e11700 1 -- 172.21.15.187:0/265913108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c0044e0 msgr2=0x7fc76c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.098+0000 7fc790e11700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c0044e0 0x7fc76c004960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fc77400c240 tx=0x7fc774007750).stop 2020-11-20T12:07:04.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.098+0000 7fc790e11700 1 -- 172.21.15.187:0/265913108 shutdown_connections 2020-11-20T12:07:04.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.098+0000 7fc790e11700 1 --2- 172.21.15.187:0/265913108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c0044e0 0x7fc76c004960 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:04.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.098+0000 7fc790e11700 1 -- 172.21.15.187:0/265913108 >> 172.21.15.187:0/265913108 conn(0x7fc78c068370 msgr2=0x7fc78c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:04.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.098+0000 7fc790e11700 1 -- 172.21.15.187:0/265913108 shutdown_connections 2020-11-20T12:07:04.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.098+0000 7fc790e11700 1 -- 172.21.15.187:0/265913108 wait complete. 2020-11-20T12:07:04.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.099+0000 7fc790e11700 1 Processor -- start 2020-11-20T12:07:04.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.100+0000 7fc790e11700 1 -- start start 2020-11-20T12:07:04.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.100+0000 7fc790e11700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c0044e0 0x7fc78c0ba810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.100+0000 7fc790e11700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc78c14b5e0 0x7fc78c140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.100+0000 7fc790e11700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc78c140f10 0x7fc78c15bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.100+0000 7fc790e11700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0bad00 con 0x7fc76c0044e0 2020-11-20T12:07:04.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.100+0000 7fc790e11700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0b6700 con 0x7fc78c140f10 2020-11-20T12:07:04.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.100+0000 7fc790e11700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0b69a0 con 0x7fc78c14b5e0 2020-11-20T12:07:04.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc78c140f10 0x7fc78c15bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc78c140f10 0x7fc78c15bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42920/0 (socket says 172.21.15.187:42920) 2020-11-20T12:07:04.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 -- 172.21.15.187:0/2179416117 learned_addr learned my addr 172.21.15.187:0/2179416117 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:04.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78b7fe700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c0044e0 0x7fc78c0ba810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 -- 172.21.15.187:0/2179416117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc78c14b5e0 msgr2=0x7fc78c140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc78c14b5e0 0x7fc78c140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:04.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 -- 172.21.15.187:0/2179416117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c0044e0 msgr2=0x7fc78c0ba810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c0044e0 0x7fc78c0ba810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:04.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 -- 172.21.15.187:0/2179416117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0b63f0 con 0x7fc78c140f10 2020-11-20T12:07:04.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.101+0000 7fc78bfff700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc78c140f10 0x7fc78c15bc20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fc78c062a00 tx=0x7fc784007d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:04.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.102+0000 7fc783fff700 1 -- 172.21.15.187:0/2179416117 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784005a70 con 0x7fc78c140f10 2020-11-20T12:07:04.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.102+0000 7fc783fff700 1 -- 172.21.15.187:0/2179416117 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc784005bd0 con 0x7fc78c140f10 2020-11-20T12:07:04.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.102+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c141370 con 0x7fc78c140f10 2020-11-20T12:07:04.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.102+0000 7fc783fff700 1 -- 172.21.15.187:0/2179416117 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784007070 con 0x7fc78c140f10 2020-11-20T12:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.102+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c14ba60 con 0x7fc78c140f10 2020-11-20T12:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.103+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78c080050 con 0x7fc78c140f10 2020-11-20T12:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.103+0000 7fc783fff700 1 -- 172.21.15.187:0/2179416117 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc78401b060 con 0x7fc78c140f10 2020-11-20T12:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.104+0000 7fc783fff700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc768056d20 0x7fc7680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.104+0000 7fc783fff700 1 -- 172.21.15.187:0/2179416117 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc78406f7c0 con 0x7fc78c140f10 2020-11-20T12:07:04.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.104+0000 7fc78b7fe700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc768056d20 0x7fc7680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.105+0000 7fc78b7fe700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc768056d20 0x7fc7680591e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fc78c05ea30 tx=0x7fc77c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:04.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.108+0000 7fc783fff700 1 -- 172.21.15.187:0/2179416117 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc784044b00 con 0x7fc78c140f10 2020-11-20T12:07:04.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.279+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc78c0b6b60 con 0x7fc78c140f10 2020-11-20T12:07:04.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.281+0000 7fc783fff700 1 -- 172.21.15.187:0/2179416117 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16791 (secure 0 0 0) 0x7fc784019070 con 0x7fc78c140f10 2020-11-20T12:07:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:04.287 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":40,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:07:03.348004+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:07:01.490118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2020-11-20T12:07:02.131720+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:07:04.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.283+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc768056d20 msgr2=0x7fc7680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.283+0000 7fc790e11700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc768056d20 0x7fc7680591e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fc78c05ea30 tx=0x7fc77c008040).stop 2020-11-20T12:07:04.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.283+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc78c140f10 msgr2=0x7fc78c15bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.283+0000 7fc790e11700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc78c140f10 0x7fc78c15bc20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fc78c062a00 tx=0x7fc784007d80).stop 2020-11-20T12:07:04.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.284+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 shutdown_connections 2020-11-20T12:07:04.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.284+0000 7fc790e11700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc768056d20 0x7fc7680591e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:04.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.284+0000 7fc790e11700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc76c0044e0 0x7fc78c0ba810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:04.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.284+0000 7fc790e11700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc78c140f10 0x7fc78c15bc20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:04.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.284+0000 7fc790e11700 1 --2- 172.21.15.187:0/2179416117 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc78c14b5e0 0x7fc78c140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:04.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.284+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 >> 172.21.15.187:0/2179416117 conn(0x7fc78c068370 msgr2=0x7fc78c141af0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:04.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.284+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 shutdown_connections 2020-11-20T12:07:04.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.284+0000 7fc790e11700 1 -- 172.21.15.187:0/2179416117 wait complete. 2020-11-20T12:07:04.308 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:07:04.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.541+0000 7f806ae43700 1 Processor -- start 2020-11-20T12:07:04.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.542+0000 7f806ae43700 1 -- start start 2020-11-20T12:07:04.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.543+0000 7f806ae43700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8064065190 0x7f8064065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.543+0000 7f806ae43700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8064065f90 0x7f8064143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.543+0000 7f806ae43700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80640b13e0 con 0x7f8064143c70 2020-11-20T12:07:04.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.543+0000 7f806ae43700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806414ab30 con 0x7f8064065190 2020-11-20T12:07:04.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.543+0000 7f806ae43700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8064062c10 con 0x7f8064065f90 2020-11-20T12:07:04.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.544+0000 7f8069e41700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8064065190 0x7f8064065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.544+0000 7f8069e41700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8064065190 0x7f8064065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42926/0 (socket says 172.21.15.187:42926) 2020-11-20T12:07:04.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.544+0000 7f8069e41700 1 -- 172.21.15.187:0/934441877 learned_addr learned my addr 172.21.15.187:0/934441877 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:04.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.544+0000 7f8069e41700 1 -- 172.21.15.187:0/934441877 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8064065f90 msgr2=0x7f8064143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:04.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.545+0000 7f8069e41700 1 --2- 172.21.15.187:0/934441877 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8064065f90 0x7f8064143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:04.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.545+0000 7f8069e41700 1 -- 172.21.15.187:0/934441877 >> v1:172.21.15.5:6789/0 conn(0x7f8064143c70 legacy=0x7f806409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.545+0000 7f8069e41700 1 -- 172.21.15.187:0/934441877 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80640b1640 con 0x7f8064065190 2020-11-20T12:07:04.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.545+0000 7f8069e41700 1 --2- 172.21.15.187:0/934441877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065190 0x7f8064065570 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f806414ac70 tx=0x7f8060006eb0).ready entity=mon.1 client_cookie=cc7f8a7760bc8f4c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:04.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.545+0000 7f8068e3f700 1 -- 172.21.15.187:0/934441877 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8060009960 con 0x7f8064065190 2020-11-20T12:07:04.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.545+0000 7f8068e3f700 1 -- 172.21.15.187:0/934441877 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8060009ac0 con 0x7f8064065190 2020-11-20T12:07:04.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.545+0000 7f8068e3f700 1 -- 172.21.15.187:0/934441877 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8060009e40 con 0x7f8064065190 2020-11-20T12:07:04.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.546+0000 7f806ae43700 1 -- 172.21.15.187:0/934441877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065190 msgr2=0x7f8064065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.546+0000 7f806ae43700 1 --2- 172.21.15.187:0/934441877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065190 0x7f8064065570 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f806414ac70 tx=0x7f8060006eb0).stop 2020-11-20T12:07:04.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.546+0000 7f806ae43700 1 -- 172.21.15.187:0/934441877 shutdown_connections 2020-11-20T12:07:04.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.546+0000 7f806ae43700 1 --2- 172.21.15.187:0/934441877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065190 0x7f8064065570 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:04.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.546+0000 7f806ae43700 1 --2- 172.21.15.187:0/934441877 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8064065f90 0x7f8064143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:04.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.546+0000 7f806ae43700 1 -- 172.21.15.187:0/934441877 >> 172.21.15.187:0/934441877 conn(0x7f80640bab30 msgr2=0x7f80640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:04.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.546+0000 7f806ae43700 1 -- 172.21.15.187:0/934441877 shutdown_connections 2020-11-20T12:07:04.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.546+0000 7f806ae43700 1 -- 172.21.15.187:0/934441877 wait complete. 2020-11-20T12:07:04.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.548+0000 7f806ae43700 1 Processor -- start 2020-11-20T12:07:04.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.548+0000 7f806ae43700 1 -- start start 2020-11-20T12:07:04.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.549+0000 7f806ae43700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065f90 0x7f8064148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f806ae43700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8064143c70 0x7f806415c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f806ae43700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80640990a0 0x7f806409f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f806ae43700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806415ca40 con 0x7f8064143c70 2020-11-20T12:07:04.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f806ae43700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806415cce0 con 0x7f8064065f90 2020-11-20T12:07:04.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f806ae43700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8064099730 con 0x7f80640990a0 2020-11-20T12:07:04.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f8069e41700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065f90 0x7f8064148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f8069e41700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065f90 0x7f8064148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42930/0 (socket says 172.21.15.187:42930) 2020-11-20T12:07:04.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f8069e41700 1 -- 172.21.15.187:0/3581349551 learned_addr learned my addr 172.21.15.187:0/3581349551 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:04.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f8069e41700 1 -- 172.21.15.187:0/3581349551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80640990a0 msgr2=0x7f806409f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f8069e41700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80640990a0 0x7f806409f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:04.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f8069e41700 1 -- 172.21.15.187:0/3581349551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8064143c70 msgr2=0x7f806415c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:04.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f8069e41700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8064143c70 0x7f806415c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:04.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.550+0000 7f8069e41700 1 -- 172.21.15.187:0/3581349551 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80640a05d0 con 0x7f8064065f90 2020-11-20T12:07:04.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.551+0000 7f8069e41700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065f90 0x7f8064148dd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f80600094f0 tx=0x7f8060015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:04.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.551+0000 7f8051ffb700 1 -- 172.21.15.187:0/3581349551 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806001e460 con 0x7f8064065f90 2020-11-20T12:07:04.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.551+0000 7f8051ffb700 1 -- 172.21.15.187:0/3581349551 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806001e5c0 con 0x7f8064065f90 2020-11-20T12:07:04.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.551+0000 7f8051ffb700 1 -- 172.21.15.187:0/3581349551 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806001e940 con 0x7f8064065f90 2020-11-20T12:07:04.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.551+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80640a07c0 con 0x7f8064065f90 2020-11-20T12:07:04.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.551+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806415c700 con 0x7f8064065f90 2020-11-20T12:07:04.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.552+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8064061a20 con 0x7f8064065f90 2020-11-20T12:07:04.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.553+0000 7f8051ffb700 1 -- 172.21.15.187:0/3581349551 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f806001a040 con 0x7f8064065f90 2020-11-20T12:07:04.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.553+0000 7f8051ffb700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f803c056d20 0x7f803c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:04.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.553+0000 7f8051ffb700 1 -- 172.21.15.187:0/3581349551 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f80600716f0 con 0x7f8064065f90 2020-11-20T12:07:04.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.553+0000 7f8069640700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f803c056d20 0x7f803c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:04.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.554+0000 7f8069640700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f803c056d20 0x7f803c0591e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8064147e50 tx=0x7f8054006a80).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:04.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.557+0000 7f8051ffb700 1 -- 172.21.15.187:0/3581349551 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8060046ae0 con 0x7f8064065f90 2020-11-20T12:07:04.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:04.805+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f80640afc60 con 0x7f8064065f90 2020-11-20T12:07:05.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.589+0000 7f8051ffb700 1 -- 172.21.15.187:0/3581349551 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v19) v1 ==== 118+0+0 (secure 0 0 0) 0x7f80600230b0 con 0x7f8064065f90 2020-11-20T12:07:05.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f803c056d20 msgr2=0x7f803c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:05.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f803c056d20 0x7f803c0591e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8064147e50 tx=0x7f8054006a80).stop 2020-11-20T12:07:05.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065f90 msgr2=0x7f8064148dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:05.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065f90 0x7f8064148dd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f80600094f0 tx=0x7f8060015000).stop 2020-11-20T12:07:05.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 shutdown_connections 2020-11-20T12:07:05.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f803c056d20 0x7f803c0591e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:05.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80640990a0 0x7f806409f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:05.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8064065f90 0x7f8064148dd0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:05.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 --2- 172.21.15.187:0/3581349551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8064143c70 0x7f806415c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:05.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 >> 172.21.15.187:0/3581349551 conn(0x7f80640bab30 msgr2=0x7f806409d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:05.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 shutdown_connections 2020-11-20T12:07:05.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.593+0000 7f806ae43700 1 -- 172.21.15.187:0/3581349551 wait complete. 2020-11-20T12:07:05.629 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:05.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.867+0000 7feb7c8d5700 1 Processor -- start 2020-11-20T12:07:05.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.867+0000 7feb7c8d5700 1 -- start start 2020-11-20T12:07:05.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.868+0000 7feb7c8d5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7feb780b57b0 0x7feb78068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:05.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.868+0000 7feb7c8d5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feb781482d0 0x7feb78062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:05.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.868+0000 7feb7c8d5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb78144b80 con 0x7feb78062b50 2020-11-20T12:07:05.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.868+0000 7feb7c8d5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb781489e0 con 0x7feb780b57b0 2020-11-20T12:07:05.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.868+0000 7feb7c8d5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb78148c80 con 0x7feb781482d0 2020-11-20T12:07:05.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.869+0000 7feb6e59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feb781482d0 0x7feb78062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:05.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.869+0000 7feb7759e700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7feb78062b50 0x7feb78065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33056/0 (socket says 172.21.15.187:33056) 2020-11-20T12:07:05.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.869+0000 7feb7759e700 1 -- 172.21.15.187:0/379242277 learned_addr learned my addr 172.21.15.187:0/379242277 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:05.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.869+0000 7feb76d9d700 1 --2- 172.21.15.187:0/379242277 >> v2:172.21.15.5:3300/0 conn(0x7feb780b57b0 0x7feb78068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:05.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.870+0000 7feb6e59c700 1 -- 172.21.15.187:0/379242277 >> v2:172.21.15.5:3300/0 conn(0x7feb780b57b0 msgr2=0x7feb78068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:05.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.870+0000 7feb6e59c700 1 --2- 172.21.15.187:0/379242277 >> v2:172.21.15.5:3300/0 conn(0x7feb780b57b0 0x7feb78068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:05.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.870+0000 7feb6e59c700 1 -- 172.21.15.187:0/379242277 >> v1:172.21.15.5:6789/0 conn(0x7feb78062b50 legacy=0x7feb78065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:05.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.870+0000 7feb6e59c700 1 -- 172.21.15.187:0/379242277 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78148f20 con 0x7feb781482d0 2020-11-20T12:07:05.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.871+0000 7feb6e59c700 1 --2- 172.21.15.187:0/379242277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb781482d0 0x7feb78062660 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7feb7809d820 tx=0x7feb60006e80).ready entity=mon.1 client_cookie=c8f24bee92e85d0d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:05.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.871+0000 7feb7659c700 1 -- 172.21.15.187:0/379242277 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60014070 con 0x7feb781482d0 2020-11-20T12:07:05.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.871+0000 7feb7659c700 1 -- 172.21.15.187:0/379242277 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6000a770 con 0x7feb781482d0 2020-11-20T12:07:05.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.871+0000 7feb7659c700 1 -- 172.21.15.187:0/379242277 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6000aaf0 con 0x7feb781482d0 2020-11-20T12:07:05.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.872+0000 7feb7c8d5700 1 -- 172.21.15.187:0/379242277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb781482d0 msgr2=0x7feb78062660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:05.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.872+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/379242277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb781482d0 0x7feb78062660 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7feb7809d820 tx=0x7feb60006e80).stop 2020-11-20T12:07:05.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.872+0000 7feb7c8d5700 1 -- 172.21.15.187:0/379242277 shutdown_connections 2020-11-20T12:07:05.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.872+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/379242277 >> v2:172.21.15.5:3300/0 conn(0x7feb780b57b0 0x7feb78068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:05.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.872+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/379242277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb781482d0 0x7feb78062660 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:05.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.872+0000 7feb7c8d5700 1 -- 172.21.15.187:0/379242277 >> 172.21.15.187:0/379242277 conn(0x7feb780bab30 msgr2=0x7feb780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:05.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.872+0000 7feb7c8d5700 1 -- 172.21.15.187:0/379242277 shutdown_connections 2020-11-20T12:07:05.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.872+0000 7feb7c8d5700 1 -- 172.21.15.187:0/379242277 wait complete. 2020-11-20T12:07:05.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.874+0000 7feb7c8d5700 1 Processor -- start 2020-11-20T12:07:05.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.874+0000 7feb7c8d5700 1 -- start start 2020-11-20T12:07:05.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb7c8d5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb78062b50 0x7feb78140c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:05.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb7c8d5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb780b57b0 0x7feb78146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:05.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb7c8d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb7815bcf0 0x7feb7815c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:05.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb7c8d5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb781478c0 con 0x7feb780b57b0 2020-11-20T12:07:05.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb7c8d5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78147380 con 0x7feb78062b50 2020-11-20T12:07:05.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb7c8d5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb78147620 con 0x7feb7815bcf0 2020-11-20T12:07:05.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb76d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb78062b50 0x7feb78140c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:05.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb76d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb78062b50 0x7feb78140c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42944/0 (socket says 172.21.15.187:42944) 2020-11-20T12:07:05.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.875+0000 7feb76d9d700 1 -- 172.21.15.187:0/1939879268 learned_addr learned my addr 172.21.15.187:0/1939879268 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:05.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb76d9d700 1 -- 172.21.15.187:0/1939879268 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb7815bcf0 msgr2=0x7feb7815c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:05.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb76d9d700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb7815bcf0 0x7feb7815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:05.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb76d9d700 1 -- 172.21.15.187:0/1939879268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb780b57b0 msgr2=0x7feb78146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:05.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb6e59c700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb780b57b0 0x7feb78146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:05.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb76d9d700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb780b57b0 0x7feb78146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:05.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb76d9d700 1 -- 172.21.15.187:0/1939879268 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78141160 con 0x7feb78062b50 2020-11-20T12:07:05.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb6e59c700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb780b57b0 0x7feb78146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:05.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb76d9d700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb78062b50 0x7feb78140c70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7feb6800ec00 tx=0x7feb68006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:05.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb6f7fe700 1 -- 172.21.15.187:0/1939879268 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6800b9a0 con 0x7feb78062b50 2020-11-20T12:07:05.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb6f7fe700 1 -- 172.21.15.187:0/1939879268 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6800bb00 con 0x7feb78062b50 2020-11-20T12:07:05.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb78143f10 con 0x7feb78062b50 2020-11-20T12:07:05.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.876+0000 7feb6f7fe700 1 -- 172.21.15.187:0/1939879268 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6800be80 con 0x7feb78062b50 2020-11-20T12:07:05.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.877+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7815c640 con 0x7feb78062b50 2020-11-20T12:07:05.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.878+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb780b2960 con 0x7feb78062b50 2020-11-20T12:07:05.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.878+0000 7feb6f7fe700 1 -- 172.21.15.187:0/1939879268 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feb68012600 con 0x7feb78062b50 2020-11-20T12:07:05.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.879+0000 7feb6f7fe700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb54056d20 0x7feb540591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:05.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.879+0000 7feb6f7fe700 1 -- 172.21.15.187:0/1939879268 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feb68070400 con 0x7feb78062b50 2020-11-20T12:07:05.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.879+0000 7feb6e59c700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb54056d20 0x7feb540591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:05.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.880+0000 7feb6e59c700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb54056d20 0x7feb540591e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7feb780633a0 tx=0x7feb60006e80).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:05.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:05.883+0000 7feb6f7fe700 1 -- 172.21.15.187:0/1939879268 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feb780b2960 con 0x7feb78062b50 2020-11-20T12:07:06.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.137+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb780b2960 con 0x7feb78062b50 2020-11-20T12:07:06.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.139+0000 7feb6f7fe700 1 -- 172.21.15.187:0/1939879268 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7feb780b2960 con 0x7feb78062b50 2020-11-20T12:07:06.144 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:06.144 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:06.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb54056d20 msgr2=0x7feb540591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb54056d20 0x7feb540591e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7feb780633a0 tx=0x7feb60006e80).stop 2020-11-20T12:07:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb78062b50 msgr2=0x7feb78140c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb78062b50 0x7feb78140c70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7feb6800ec00 tx=0x7feb68006df0).stop 2020-11-20T12:07:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 shutdown_connections 2020-11-20T12:07:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb54056d20 0x7feb540591e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb7815bcf0 0x7feb7815c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:06.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb78062b50 0x7feb78140c70 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 --2- 172.21.15.187:0/1939879268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb780b57b0 0x7feb78146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 >> 172.21.15.187:0/1939879268 conn(0x7feb780bab30 msgr2=0x7feb780a0290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:06.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.141+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 shutdown_connections 2020-11-20T12:07:06.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.142+0000 7feb7c8d5700 1 -- 172.21.15.187:0/1939879268 wait complete. 2020-11-20T12:07:06.148 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:06.170 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:07:06.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.406+0000 7fb8e4faf700 1 Processor -- start 2020-11-20T12:07:06.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.406+0000 7fb8e4faf700 1 -- start start 2020-11-20T12:07:06.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.407+0000 7fb8e4faf700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8e00654a0 0x7fb8e00b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.407+0000 7fb8e4faf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e00b6c10 0x7fb8e00666e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.407+0000 7fb8e4faf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e00b08e0 con 0x7fb8e0065880 2020-11-20T12:07:06.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.407+0000 7fb8e4faf700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e00b0b80 con 0x7fb8e00654a0 2020-11-20T12:07:06.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.407+0000 7fb8e4faf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e00b05f0 con 0x7fb8e00b6c10 2020-11-20T12:07:06.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.408+0000 7fb8d5d9b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb8e0065880 0x7fb8e0144e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33070/0 (socket says 172.21.15.187:33070) 2020-11-20T12:07:06.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.408+0000 7fb8ded9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e00b6c10 0x7fb8e00666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.408+0000 7fb8d5d9b700 1 -- 172.21.15.187:0/2528634453 learned_addr learned my addr 172.21.15.187:0/2528634453 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:06.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.409+0000 7fb8de59c700 1 --2- 172.21.15.187:0/2528634453 >> v2:172.21.15.5:3300/0 conn(0x7fb8e00654a0 0x7fb8e00b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.409+0000 7fb8ddd9b700 1 -- 172.21.15.187:0/2528634453 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124215803 0 0) 0x7fb8e00b08e0 con 0x7fb8e0065880 2020-11-20T12:07:06.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.409+0000 7fb8ddd9b700 1 -- 172.21.15.187:0/2528634453 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c0003240 con 0x7fb8e0065880 2020-11-20T12:07:06.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.410+0000 7fb8de59c700 1 -- 172.21.15.187:0/2528634453 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e00b6c10 msgr2=0x7fb8e00666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:06.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.410+0000 7fb8de59c700 1 --2- 172.21.15.187:0/2528634453 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e00b6c10 0x7fb8e00666e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.410+0000 7fb8de59c700 1 -- 172.21.15.187:0/2528634453 >> v1:172.21.15.5:6789/0 conn(0x7fb8e0065880 legacy=0x7fb8e0144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.410+0000 7fb8de59c700 1 -- 172.21.15.187:0/2528634453 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e0145570 con 0x7fb8e00654a0 2020-11-20T12:07:06.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.410+0000 7fb8ded9d700 1 --2- 172.21.15.187:0/2528634453 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e00b6c10 0x7fb8e00666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:07:06.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.410+0000 7fb8de59c700 1 --2- 172.21.15.187:0/2528634453 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8e00654a0 0x7fb8e00b1650 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fb8d000b9a0 tx=0x7fb8d0006df0).ready entity=mon.0 client_cookie=8128d642bb7446c0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.411+0000 7fb8ddd9b700 1 -- 172.21.15.187:0/2528634453 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0009710 con 0x7fb8e00654a0 2020-11-20T12:07:06.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.411+0000 7fb8ddd9b700 1 -- 172.21.15.187:0/2528634453 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d0009870 con 0x7fb8e00654a0 2020-11-20T12:07:06.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.411+0000 7fb8ddd9b700 1 -- 172.21.15.187:0/2528634453 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0009bf0 con 0x7fb8e00654a0 2020-11-20T12:07:06.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.411+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2528634453 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8e00654a0 msgr2=0x7fb8e00b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.411+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2528634453 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8e00654a0 0x7fb8e00b1650 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fb8d000b9a0 tx=0x7fb8d0006df0).stop 2020-11-20T12:07:06.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.412+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2528634453 shutdown_connections 2020-11-20T12:07:06.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.412+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2528634453 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e00b6c10 0x7fb8e00666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.412+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2528634453 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8e00654a0 0x7fb8e00b1650 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.412+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2528634453 >> 172.21.15.187:0/2528634453 conn(0x7fb8e00bacd0 msgr2=0x7fb8e00bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:06.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.412+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2528634453 shutdown_connections 2020-11-20T12:07:06.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.412+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2528634453 wait complete. 2020-11-20T12:07:06.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.413+0000 7fb8e4faf700 1 Processor -- start 2020-11-20T12:07:06.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.414+0000 7fb8e4faf700 1 -- start start 2020-11-20T12:07:06.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.414+0000 7fb8e4faf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8e00654a0 0x7fb8e009bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.414+0000 7fb8e4faf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e0065880 0x7fb8e00b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.414+0000 7fb8e4faf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8e00b6c10 0x7fb8e01432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8e4faf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e009d190 con 0x7fb8e00b6c10 2020-11-20T12:07:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8e4faf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e009d430 con 0x7fb8e00654a0 2020-11-20T12:07:06.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8e4faf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e0146990 con 0x7fb8e0065880 2020-11-20T12:07:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8de59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8e00654a0 0x7fb8e009bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8de59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8e00654a0 0x7fb8e009bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42958/0 (socket says 172.21.15.187:42958) 2020-11-20T12:07:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8de59c700 1 -- 172.21.15.187:0/2058434154 learned_addr learned my addr 172.21.15.187:0/2058434154 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:06.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8de59c700 1 -- 172.21.15.187:0/2058434154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e0065880 msgr2=0x7fb8e00b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8de59c700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e0065880 0x7fb8e00b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8de59c700 1 -- 172.21.15.187:0/2058434154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8e00b6c10 msgr2=0x7fb8e01432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8de59c700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8e00b6c10 0x7fb8e01432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.415+0000 7fb8de59c700 1 -- 172.21.15.187:0/2058434154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e009a8d0 con 0x7fb8e00654a0 2020-11-20T12:07:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.416+0000 7fb8de59c700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8e00654a0 0x7fb8e009bd60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fb8d000c010 tx=0x7fb8d0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.416+0000 7fb8d6ffd700 1 -- 172.21.15.187:0/2058434154 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0011e50 con 0x7fb8e00654a0 2020-11-20T12:07:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.416+0000 7fb8d6ffd700 1 -- 172.21.15.187:0/2058434154 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d001e460 con 0x7fb8e00654a0 2020-11-20T12:07:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.416+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e00b4840 con 0x7fb8e00654a0 2020-11-20T12:07:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.416+0000 7fb8d6ffd700 1 -- 172.21.15.187:0/2058434154 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d001e7e0 con 0x7fb8e00654a0 2020-11-20T12:07:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.416+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e009ce70 con 0x7fb8e00654a0 2020-11-20T12:07:06.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.417+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e0146b50 con 0x7fb8e00654a0 2020-11-20T12:07:06.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.418+0000 7fb8d6ffd700 1 -- 172.21.15.187:0/2058434154 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb8d0025030 con 0x7fb8e00654a0 2020-11-20T12:07:06.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.418+0000 7fb8d6ffd700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8bc056d20 0x7fb8bc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.419+0000 7fb8d6ffd700 1 -- 172.21.15.187:0/2058434154 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb8d0071700 con 0x7fb8e00654a0 2020-11-20T12:07:06.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.419+0000 7fb8d5d9b700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8bc056d20 0x7fb8bc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.419+0000 7fb8d5d9b700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8bc056d20 0x7fb8bc0591e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb8c80055f0 tx=0x7fb8c80076f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.422+0000 7fb8d6ffd700 1 -- 172.21.15.187:0/2058434154 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb8d0046b30 con 0x7fb8e00654a0 2020-11-20T12:07:06.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.611+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8e0063600 con 0x7fb8e00654a0 2020-11-20T12:07:06.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.612+0000 7fb8d6ffd700 1 -- 172.21.15.187:0/2058434154 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16778 (secure 0 0 0) 0x7fb8d00311a0 con 0x7fb8e00654a0 2020-11-20T12:07:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:06.617 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":42,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:07:05.571797+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:07:01.490118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2020-11-20T12:07:02.131720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:07:06.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8bc056d20 msgr2=0x7fb8bc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8bc056d20 0x7fb8bc0591e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb8c80055f0 tx=0x7fb8c80076f0).stop 2020-11-20T12:07:06.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8e00654a0 msgr2=0x7fb8e009bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8e00654a0 0x7fb8e009bd60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fb8d000c010 tx=0x7fb8d0015000).stop 2020-11-20T12:07:06.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 shutdown_connections 2020-11-20T12:07:06.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8bc056d20 0x7fb8bc0591e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8e00b6c10 0x7fb8e01432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:06.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8e00654a0 0x7fb8e009bd60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 --2- 172.21.15.187:0/2058434154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8e0065880 0x7fb8e00b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 >> 172.21.15.187:0/2058434154 conn(0x7fb8e00bacd0 msgr2=0x7fb8e00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 shutdown_connections 2020-11-20T12:07:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.615+0000 7fb8e4faf700 1 -- 172.21.15.187:0/2058434154 wait complete. 2020-11-20T12:07:06.642 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:07:06.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.887+0000 7fe4cc99f700 1 Processor -- start 2020-11-20T12:07:06.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.887+0000 7fe4cc99f700 1 -- start start 2020-11-20T12:07:06.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.889+0000 7fe4cc99f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c80b57b0 0x7fe4c8068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.889+0000 7fe4cc99f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c8144b80 con 0x7fe4c81482d0 2020-11-20T12:07:06.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.889+0000 7fe4cc99f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c81489e0 con 0x7fe4c80b5f50 2020-11-20T12:07:06.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.889+0000 7fe4cc99f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8148c80 con 0x7fe4c80b57b0 2020-11-20T12:07:06.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.889+0000 7fe4c759e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe4c80b5f50 0x7fe4c80b6400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39574/0 (socket says 172.21.15.187:39574) 2020-11-20T12:07:06.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.889+0000 7fe4c759e700 1 -- 172.21.15.187:0/506504790 learned_addr learned my addr 172.21.15.187:0/506504790 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:06.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.890+0000 7fe4c6d9d700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c80b57b0 0x7fe4c8068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.890+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919718274 0 0) 0x7fe4c81489e0 con 0x7fe4c80b5f50 2020-11-20T12:07:06.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.890+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a8003240 con 0x7fe4c80b5f50 2020-11-20T12:07:06.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.890+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377378153 0 0) 0x7fe4c8144b80 con 0x7fe4c81482d0 2020-11-20T12:07:06.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.890+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c81489e0 con 0x7fe4c81482d0 2020-11-20T12:07:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3008130610 0 0) 0x7fe4a8003240 con 0x7fe4c80b5f50 2020-11-20T12:07:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c80b57b0 msgr2=0x7fe4c8068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c80b57b0 0x7fe4c8068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 >> v1:172.21.15.5:6789/0 conn(0x7fe4c81482d0 legacy=0x7fe4c8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8148f20 con 0x7fe4c80b5f50 2020-11-20T12:07:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fe4c0001860 con 0x7fe4c80b5f50 2020-11-20T12:07:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 >> v1:172.21.15.187:6789/0 conn(0x7fe4c80b5f50 legacy=0x7fe4c80b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c80b57b0 0x7fe4a80038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4a8003db0 0x7fe4a80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4a80049d0 0x7fe4a8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8148c80 con 0x7fe4a80049d0 2020-11-20T12:07:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a8005340 con 0x7fe4a8003db0 2020-11-20T12:07:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a800b760 con 0x7fe4c80b57b0 2020-11-20T12:07:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4be59c700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4a8003db0 0x7fe4a80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.891+0000 7fe4c6d9d700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c80b57b0 0x7fe4a80038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4c759e700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4a80049d0 0x7fe4a8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4be59c700 1 -- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c80b57b0 msgr2=0x7fe4a80038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:06.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4be59c700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c80b57b0 0x7fe4a80038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4be59c700 1 -- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4a80049d0 msgr2=0x7fe4a8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:06.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4be59c700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4a80049d0 0x7fe4a8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4be59c700 1 -- 172.21.15.187:0/506504790 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a8003240 con 0x7fe4a8003db0 2020-11-20T12:07:06.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4c759e700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4a80049d0 0x7fe4a8004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:06.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4be59c700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4a8003db0 0x7fe4a80044e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fe4b0001a50 tx=0x7fe4b0007b90).ready entity=mon.1 client_cookie=3b0c34b0507e4c63 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.892+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0007020 con 0x7fe4a8003db0 2020-11-20T12:07:06.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b0007180 con 0x7fe4a8003db0 2020-11-20T12:07:06.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4c659c700 1 -- 172.21.15.187:0/506504790 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b000c8f0 con 0x7fe4a8003db0 2020-11-20T12:07:06.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4cc99f700 1 -- 172.21.15.187:0/506504790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4a8003db0 msgr2=0x7fe4a80044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4cc99f700 1 --2- 172.21.15.187:0/506504790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4a8003db0 0x7fe4a80044e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fe4b0001a50 tx=0x7fe4b0007b90).stop 2020-11-20T12:07:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4c759e700 1 -- 172.21.15.187:0/506504790 reap_dead start 2020-11-20T12:07:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4cc99f700 1 -- 172.21.15.187:0/506504790 shutdown_connections 2020-11-20T12:07:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4cc99f700 1 -- 172.21.15.187:0/506504790 >> 172.21.15.187:0/506504790 conn(0x7fe4c80bab30 msgr2=0x7fe4c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4cc99f700 1 -- 172.21.15.187:0/506504790 shutdown_connections 2020-11-20T12:07:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.893+0000 7fe4cc99f700 1 -- 172.21.15.187:0/506504790 wait complete. 2020-11-20T12:07:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.895+0000 7fe4cc99f700 1 Processor -- start 2020-11-20T12:07:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.895+0000 7fe4cc99f700 1 -- start start 2020-11-20T12:07:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4cc99f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4c808ebe0 0x7fe4c81593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4cc99f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4c808efc0 0x7fe4c81598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4cc99f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c809c770 0x7fe4c815f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4cc99f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8159dc0 con 0x7fe4c808ebe0 2020-11-20T12:07:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4cc99f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c809c5d0 con 0x7fe4c808efc0 2020-11-20T12:07:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4cc99f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8147560 con 0x7fe4c809c770 2020-11-20T12:07:06.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4be59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4c808efc0 0x7fe4c81598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4be59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4c808efc0 0x7fe4c81598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42980/0 (socket says 172.21.15.187:42980) 2020-11-20T12:07:06.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4be59c700 1 -- 172.21.15.187:0/2674883288 learned_addr learned my addr 172.21.15.187:0/2674883288 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:06.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.896+0000 7fe4c6d9d700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4c808ebe0 0x7fe4c81593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.897+0000 7fe4c759e700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c809c770 0x7fe4c815f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.897+0000 7fe4be59c700 1 -- 172.21.15.187:0/2674883288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c809c770 msgr2=0x7fe4c815f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.897+0000 7fe4be59c700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c809c770 0x7fe4c815f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.897+0000 7fe4be59c700 1 -- 172.21.15.187:0/2674883288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4c808ebe0 msgr2=0x7fe4c81593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.897+0000 7fe4be59c700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4c808ebe0 0x7fe4c81593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.897+0000 7fe4be59c700 1 -- 172.21.15.187:0/2674883288 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8147800 con 0x7fe4c808efc0 2020-11-20T12:07:06.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.897+0000 7fe4be59c700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4c808efc0 0x7fe4c81598d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fe4b0011000 tx=0x7fe4b0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.898+0000 7fe4bf7fe700 1 -- 172.21.15.187:0/2674883288 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0001ea0 con 0x7fe4c808efc0 2020-11-20T12:07:06.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.898+0000 7fe4bf7fe700 1 -- 172.21.15.187:0/2674883288 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b000c3c0 con 0x7fe4c808efc0 2020-11-20T12:07:06.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.898+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c815ccf0 con 0x7fe4c808efc0 2020-11-20T12:07:06.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.898+0000 7fe4bf7fe700 1 -- 172.21.15.187:0/2674883288 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b000c8f0 con 0x7fe4c808efc0 2020-11-20T12:07:06.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.898+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c809d5c0 con 0x7fe4c808efc0 2020-11-20T12:07:06.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.899+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c815ce80 con 0x7fe4c808efc0 2020-11-20T12:07:06.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.899+0000 7fe4bf7fe700 1 -- 172.21.15.187:0/2674883288 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe4b0024030 con 0x7fe4c808efc0 2020-11-20T12:07:06.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.900+0000 7fe4bf7fe700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a4056d20 0x7fe4a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.901+0000 7fe4bf7fe700 1 -- 172.21.15.187:0/2674883288 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe4b00712b0 con 0x7fe4c808efc0 2020-11-20T12:07:06.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.901+0000 7fe4c6d9d700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a4056d20 0x7fe4a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.901+0000 7fe4c6d9d700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a4056d20 0x7fe4a40591e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fe4b8006fd0 tx=0x7fe4b8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:06.904+0000 7fe4bf7fe700 1 -- 172.21.15.187:0/2674883288 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe4b00467b0 con 0x7fe4c808efc0 2020-11-20T12:07:07.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.163+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fe4c8061a20 con 0x7fe4c808efc0 2020-11-20T12:07:07.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.174+0000 7fe4bf7fe700 1 -- 172.21.15.187:0/2674883288 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe4b0022070 con 0x7fe4c808efc0 2020-11-20T12:07:07.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a4056d20 msgr2=0x7fe4a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:07.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a4056d20 0x7fe4a40591e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fe4b8006fd0 tx=0x7fe4b8008040).stop 2020-11-20T12:07:07.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4c808efc0 msgr2=0x7fe4c81598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:07.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4c808efc0 0x7fe4c81598d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fe4b0011000 tx=0x7fe4b0016000).stop 2020-11-20T12:07:07.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 shutdown_connections 2020-11-20T12:07:07.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a4056d20 0x7fe4a40591e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4c808ebe0 0x7fe4c81593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4c809c770 0x7fe4c815f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 --2- 172.21.15.187:0/2674883288 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4c808efc0 0x7fe4c81598d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 >> 172.21.15.187:0/2674883288 conn(0x7fe4c80bab30 msgr2=0x7fe4c809fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:07.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.177+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 shutdown_connections 2020-11-20T12:07:07.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.178+0000 7fe4cc99f700 1 -- 172.21.15.187:0/2674883288 wait complete. 2020-11-20T12:07:07.185 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:07:07.206 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:07.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.436+0000 7f5e2a11a700 1 Processor -- start 2020-11-20T12:07:07.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.436+0000 7f5e2a11a700 1 -- start start 2020-11-20T12:07:07.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.437+0000 7f5e2a11a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5e24065190 0x7f5e24065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:07.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.437+0000 7f5e2a11a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e240b13e0 con 0x7f5e2409ca90 2020-11-20T12:07:07.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.437+0000 7f5e2a11a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e24062c10 con 0x7f5e24065f90 2020-11-20T12:07:07.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.437+0000 7f5e2a11a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2414ab30 con 0x7f5e24065190 2020-11-20T12:07:07.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e28917700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5e24065f90 0x7f5e24143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39594/0 (socket says 172.21.15.187:39594) 2020-11-20T12:07:07.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e28917700 1 -- 172.21.15.187:0/3859058995 learned_addr learned my addr 172.21.15.187:0/3859058995 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:07.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573192807 0 0) 0x7f5e24062c10 con 0x7f5e24065f90 2020-11-20T12:07:07.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c003240 con 0x7f5e24065f90 2020-11-20T12:07:07.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e29118700 1 --2- 172.21.15.187:0/3859058995 >> v2:172.21.15.5:3300/0 conn(0x7f5e24065190 0x7f5e24065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:07.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2195049001 0 0) 0x7f5e0c003240 con 0x7f5e24065f90 2020-11-20T12:07:07.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 >> v2:172.21.15.5:3300/0 conn(0x7f5e24065190 msgr2=0x7f5e24065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e13fff700 1 --2- 172.21.15.187:0/3859058995 >> v2:172.21.15.5:3300/0 conn(0x7f5e24065190 0x7f5e24065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 >> v1:172.21.15.5:6789/0 conn(0x7f5e2409ca90 legacy=0x7f5e2409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.438+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e240b1640 con 0x7f5e24065f90 2020-11-20T12:07:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5e140014e0 con 0x7f5e24065f90 2020-11-20T12:07:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 >> v1:172.21.15.187:6789/0 conn(0x7f5e24065f90 legacy=0x7f5e24143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e13fff700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c003800 0x7f5e0c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e13fff700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e0c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e13fff700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c004e50 0x7f5e0c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2414ab30 con 0x7f5e0c003800 2020-11-20T12:07:07.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c005300 con 0x7f5e0c0044e0 2020-11-20T12:07:07.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c00bc20 con 0x7f5e0c004e50 2020-11-20T12:07:07.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e28917700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e0c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:07.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e29919700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c004e50 0x7f5e0c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:07.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.439+0000 7f5e29118700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c003800 0x7f5e0c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:07.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e28917700 1 -- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c004e50 msgr2=0x7f5e0c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:07.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e28917700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c004e50 0x7f5e0c00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e28917700 1 -- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c003800 msgr2=0x7f5e0c003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:07.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e28917700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c003800 0x7f5e0c003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e28917700 1 -- 172.21.15.187:0/3859058995 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c003240 con 0x7f5e0c0044e0 2020-11-20T12:07:07.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e29118700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c003800 0x7f5e0c003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:07.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e29919700 1 -- 172.21.15.187:0/3859058995 reap_dead start 2020-11-20T12:07:07.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e28917700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e0c004960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f5e1400c1e0 tx=0x7f5e240b0fb0).ready entity=mon.1 client_cookie=e47571cb23318228 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:07.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e140070d0 con 0x7f5e0c0044e0 2020-11-20T12:07:07.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e14007230 con 0x7f5e0c0044e0 2020-11-20T12:07:07.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.440+0000 7f5e13fff700 1 -- 172.21.15.187:0/3859058995 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e140055f0 con 0x7f5e0c0044e0 2020-11-20T12:07:07.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.441+0000 7f5e2a11a700 1 -- 172.21.15.187:0/3859058995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 msgr2=0x7f5e0c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:07.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.441+0000 7f5e2a11a700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e0c004960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f5e1400c1e0 tx=0x7f5e240b0fb0).stop 2020-11-20T12:07:07.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.441+0000 7f5e2a11a700 1 -- 172.21.15.187:0/3859058995 shutdown_connections 2020-11-20T12:07:07.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.441+0000 7f5e2a11a700 1 --2- 172.21.15.187:0/3859058995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e0c004960 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.441+0000 7f5e2a11a700 1 -- 172.21.15.187:0/3859058995 >> 172.21.15.187:0/3859058995 conn(0x7f5e240bab30 msgr2=0x7f5e240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:07.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.441+0000 7f5e2a11a700 1 -- 172.21.15.187:0/3859058995 shutdown_connections 2020-11-20T12:07:07.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.441+0000 7f5e2a11a700 1 -- 172.21.15.187:0/3859058995 wait complete. 2020-11-20T12:07:07.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.443+0000 7f5e2a11a700 1 Processor -- start 2020-11-20T12:07:07.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.443+0000 7f5e2a11a700 1 -- start start 2020-11-20T12:07:07.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.444+0000 7f5e2a11a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e240a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:07.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.444+0000 7f5e2a11a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e240b7850 0x7f5e24130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:07.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.444+0000 7f5e2a11a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e241311d0 0x7f5e240b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:07.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.444+0000 7f5e2a11a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e240a0980 con 0x7f5e240b7850 2020-11-20T12:07:07.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.444+0000 7f5e2a11a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2409e790 con 0x7f5e0c0044e0 2020-11-20T12:07:07.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.444+0000 7f5e2a11a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2409ea30 con 0x7f5e241311d0 2020-11-20T12:07:07.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e240a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:07.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e240a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:42998/0 (socket says 172.21.15.187:42998) 2020-11-20T12:07:07.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 -- 172.21.15.187:0/2141854807 learned_addr learned my addr 172.21.15.187:0/2141854807 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:07.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 -- 172.21.15.187:0/2141854807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e241311d0 msgr2=0x7f5e240b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:07.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e241311d0 0x7f5e240b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:07.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 -- 172.21.15.187:0/2141854807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e240b7850 msgr2=0x7f5e24130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:07.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e240b7850 0x7f5e24130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:07.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 -- 172.21.15.187:0/2141854807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2409ecd0 con 0x7f5e0c0044e0 2020-11-20T12:07:07.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.445+0000 7f5e29118700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e240a0490 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f5e2000b740 tx=0x7f5e20008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:07.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.446+0000 7f5e117fa700 1 -- 172.21.15.187:0/2141854807 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2000f640 con 0x7f5e0c0044e0 2020-11-20T12:07:07.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.446+0000 7f5e117fa700 1 -- 172.21.15.187:0/2141854807 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2000c040 con 0x7f5e0c0044e0 2020-11-20T12:07:07.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.446+0000 7f5e117fa700 1 -- 172.21.15.187:0/2141854807 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2000f9b0 con 0x7f5e0c0044e0 2020-11-20T12:07:07.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.446+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2409e4c0 con 0x7f5e0c0044e0 2020-11-20T12:07:07.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.446+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e240b7c90 con 0x7f5e0c0044e0 2020-11-20T12:07:07.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.447+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e241498b0 con 0x7f5e0c0044e0 2020-11-20T12:07:07.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.447+0000 7f5e117fa700 1 -- 172.21.15.187:0/2141854807 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5e20019030 con 0x7f5e0c0044e0 2020-11-20T12:07:07.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.448+0000 7f5e117fa700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5dfc056d20 0x7f5dfc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:07.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.448+0000 7f5e117fa700 1 -- 172.21.15.187:0/2141854807 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5e2006f920 con 0x7f5e0c0044e0 2020-11-20T12:07:07.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.448+0000 7f5e28917700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5dfc056d20 0x7f5dfc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:07.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.449+0000 7f5e28917700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5dfc056d20 0x7f5dfc0591e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f5e1400cc50 tx=0x7f5e14015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:07.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.452+0000 7f5e117fa700 1 -- 172.21.15.187:0/2141854807 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5e20043fd0 con 0x7f5e0c0044e0 2020-11-20T12:07:07.579 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:07:07.574+0000 7f8132093700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:07:07.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.683+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e24061a20 con 0x7f5e0c0044e0 2020-11-20T12:07:07.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.684+0000 7f5e117fa700 1 -- 172.21.15.187:0/2141854807 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f5e20044d10 con 0x7f5e0c0044e0 2020-11-20T12:07:07.688 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:07.689 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:07.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5dfc056d20 msgr2=0x7f5dfc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:07.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5dfc056d20 0x7f5dfc0591e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f5e1400cc50 tx=0x7f5e14015000).stop 2020-11-20T12:07:07.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 msgr2=0x7f5e240a0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:07.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e240a0490 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f5e2000b740 tx=0x7f5e20008040).stop 2020-11-20T12:07:07.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 shutdown_connections 2020-11-20T12:07:07.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5dfc056d20 0x7f5dfc0591e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e241311d0 0x7f5e240b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:07.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0044e0 0x7f5e240a0490 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:07.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 --2- 172.21.15.187:0/2141854807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e240b7850 0x7f5e24130ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:07.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 >> 172.21.15.187:0/2141854807 conn(0x7f5e240bab30 msgr2=0x7f5e2409aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:07.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.686+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 shutdown_connections 2020-11-20T12:07:07.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:07.687+0000 7f5e2a11a700 1 -- 172.21.15.187:0/2141854807 wait complete. 2020-11-20T12:07:07.693 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:07.915 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:07:03.348416+0000', 'modified': '2020-11-20T12:07:05.571685+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5335}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5335': {'gid': 5335, 'name': 'a', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/2880645780', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2880645780}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2880645780}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:07:07.916 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:07:07.916 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:07:08.193 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":5335,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.8148346850000001,"mdsmap_epoch":19,"osdmap_epoch":42,"osdmap_epoch_barrier":41,"uptime":10.390114096} 2020-11-20T12:07:08.193 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5335, "mdsmap_epoch": 19, "osdmap_epoch": 42, "osdmap_epoch_barrier": 41, "rank_uptime": 4.814834685, "state": "up:active", "uptime": 10.390114096, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:07:08.194 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:07:08.194 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:07:08.195 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:07:08.195 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:07:08.195 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:07:08.196 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:07:08.196 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:07:08.196 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:07:08.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:08.197 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:07:08.250 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:07:08.250 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:07:08.251 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:07:08.251 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:08.251 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:07:08.251 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:08.252 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:07:08.252 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:08.252 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:07:08.252 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:08.253 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:07:08.253 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:08.253 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:07:08.253 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:08.253 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:07:08.254 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42610sec preferred_lft 42610sec 2020-11-20T12:07:08.254 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:07:08.254 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:08.254 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:07:08.255 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:08.255 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:07:08.255 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:08.255 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:07:08.256 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:08.256 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:07:08.256 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:07:08.257 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:07:08.257 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:08.257 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:07:08.257 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:07:08.258 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:07:08.258 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:08.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:08.259 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:07:08.354 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:07:08.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:08.358 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:07:08.407 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:07:08.408 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:07:08.408 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:07:08.408 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:07:08.409 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:07:08.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:08.410 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:07:08.411 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:07:08.411 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:07:08.411 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:07:08.412 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:07:08.412 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:07:08.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:08.649 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:07:08.697 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:07:08.697 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:07:08.698 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:07:08.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:08.699 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:07:08.799 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:07:09.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.041+0000 7f82bf59e700 1 Processor -- start 2020-11-20T12:07:09.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.041+0000 7f82bf59e700 1 -- start start 2020-11-20T12:07:09.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.043+0000 7f82bf59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f82c0068230 0x7f82c01482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.043+0000 7f82bf59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c0144b80 con 0x7f82c00b5f50 2020-11-20T12:07:09.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.043+0000 7f82bf59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c01489e0 con 0x7f82c00b57b0 2020-11-20T12:07:09.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.043+0000 7f82bf59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c01487c0 con 0x7f82c0068230 2020-11-20T12:07:09.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.043+0000 7f82be59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f82c00b57b0 0x7f82c0062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39612/0 (socket says 172.21.15.187:39612) 2020-11-20T12:07:09.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.043+0000 7f82bdd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f82c0068230 0x7f82c01482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.043+0000 7f82be59c700 1 -- 172.21.15.187:0/4136433982 learned_addr learned my addr 172.21.15.187:0/4136433982 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:09.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.044+0000 7f82bd59a700 1 -- 172.21.15.187:0/4136433982 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806574292 0 0) 0x7f82c01489e0 con 0x7f82c00b57b0 2020-11-20T12:07:09.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.044+0000 7f82bdd9b700 1 -- 172.21.15.187:0/4136433982 >> v1:172.21.15.187:6789/0 conn(0x7f82c00b57b0 legacy=0x7f82c0062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.044+0000 7f82bdd9b700 1 -- 172.21.15.187:0/4136433982 >> v1:172.21.15.5:6789/0 conn(0x7f82c00b5f50 legacy=0x7f82c00b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:09.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.044+0000 7f82bdd9b700 1 -- 172.21.15.187:0/4136433982 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0148c80 con 0x7f82c0068230 2020-11-20T12:07:09.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.044+0000 7f82bdd9b700 1 --2- 172.21.15.187:0/4136433982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c0068230 0x7f82c01482d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f82b400b9a0 tx=0x7f82b4006df0).ready entity=mon.1 client_cookie=4dbab26681435b75 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.044+0000 7f82bd59a700 1 -- 172.21.15.187:0/4136433982 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4009770 con 0x7f82c0068230 2020-11-20T12:07:09.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.044+0000 7f82bd59a700 1 -- 172.21.15.187:0/4136433982 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b40098d0 con 0x7f82c0068230 2020-11-20T12:07:09.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.044+0000 7f82bd59a700 1 -- 172.21.15.187:0/4136433982 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4009c50 con 0x7f82c0068230 2020-11-20T12:07:09.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.045+0000 7f82bf59e700 1 -- 172.21.15.187:0/4136433982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c0068230 msgr2=0x7f82c01482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.045+0000 7f82bf59e700 1 --2- 172.21.15.187:0/4136433982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c0068230 0x7f82c01482d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f82b400b9a0 tx=0x7f82b4006df0).stop 2020-11-20T12:07:09.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.045+0000 7f82bf59e700 1 -- 172.21.15.187:0/4136433982 shutdown_connections 2020-11-20T12:07:09.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.045+0000 7f82bf59e700 1 --2- 172.21.15.187:0/4136433982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c0068230 0x7f82c01482d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.045+0000 7f82bf59e700 1 -- 172.21.15.187:0/4136433982 >> 172.21.15.187:0/4136433982 conn(0x7f82c00bab30 msgr2=0x7f82c00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:09.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.045+0000 7f82bf59e700 1 -- 172.21.15.187:0/4136433982 shutdown_connections 2020-11-20T12:07:09.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.045+0000 7f82bf59e700 1 -- 172.21.15.187:0/4136433982 wait complete. 2020-11-20T12:07:09.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.047+0000 7f82bf59e700 1 Processor -- start 2020-11-20T12:07:09.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.047+0000 7f82bf59e700 1 -- start start 2020-11-20T12:07:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bf59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82c0068230 0x7f82c01545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82c00b57b0 0x7f82c009b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bf59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c00b5f50 0x7f82c009cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bf59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c0153c90 con 0x7f82c0068230 2020-11-20T12:07:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bf59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c0153e80 con 0x7f82c00b5f50 2020-11-20T12:07:09.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c0154120 con 0x7f82c00b57b0 2020-11-20T12:07:09.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c00b5f50 0x7f82c009cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c00b5f50 0x7f82c009cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43016/0 (socket says 172.21.15.187:43016) 2020-11-20T12:07:09.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bed9d700 1 -- 172.21.15.187:0/3570732086 learned_addr learned my addr 172.21.15.187:0/3570732086 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:09.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bdd9b700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82c00b57b0 0x7f82c009b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82be59c700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82c0068230 0x7f82c01545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.048+0000 7f82bed9d700 1 -- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82c00b57b0 msgr2=0x7f82c009b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82bed9d700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82c00b57b0 0x7f82c009b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82bed9d700 1 -- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82c0068230 msgr2=0x7f82c01545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82bed9d700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82c0068230 0x7f82c01545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82bed9d700 1 -- 172.21.15.187:0/3570732086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0154ab0 con 0x7f82c00b5f50 2020-11-20T12:07:09.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82bdd9b700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82c00b57b0 0x7f82c009b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:09.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82bed9d700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c00b5f50 0x7f82c009cbd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f82b800bae0 tx=0x7f82b80090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82ae7fc700 1 -- 172.21.15.187:0/3570732086 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b800c7d0 con 0x7f82c00b5f50 2020-11-20T12:07:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82ae7fc700 1 -- 172.21.15.187:0/3570732086 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b800c950 con 0x7f82c00b5f50 2020-11-20T12:07:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82ae7fc700 1 -- 172.21.15.187:0/3570732086 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b800cb00 con 0x7f82c00b5f50 2020-11-20T12:07:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c009b5f0 con 0x7f82c00b5f50 2020-11-20T12:07:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.049+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c009a620 con 0x7f82c00b5f50 2020-11-20T12:07:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.051+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c01542e0 con 0x7f82c00b5f50 2020-11-20T12:07:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.051+0000 7f82ae7fc700 1 -- 172.21.15.187:0/3570732086 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f82b801c070 con 0x7f82c00b5f50 2020-11-20T12:07:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.052+0000 7f82ae7fc700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8294056d20 0x7f82940591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.052+0000 7f82ae7fc700 1 -- 172.21.15.187:0/3570732086 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f82b806f640 con 0x7f82c00b5f50 2020-11-20T12:07:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.052+0000 7f82be59c700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8294056d20 0x7f82940591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.053+0000 7f82be59c700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8294056d20 0x7f82940591e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f82b0000bc0 tx=0x7f82b0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.056+0000 7f82ae7fc700 1 -- 172.21.15.187:0/3570732086 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f82b8044a90 con 0x7f82c00b5f50 2020-11-20T12:07:09.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.242+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82c009b780 con 0x7f82c00b5f50 2020-11-20T12:07:09.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.243+0000 7f82ae7fc700 1 -- 172.21.15.187:0/3570732086 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f82b80480b0 con 0x7f82c00b5f50 2020-11-20T12:07:09.248 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:09.248 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":42,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:07:05.571797+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:07:01.490118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2020-11-20T12:07:02.131720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:07:09.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8294056d20 msgr2=0x7f82940591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8294056d20 0x7f82940591e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f82b0000bc0 tx=0x7f82b0009040).stop 2020-11-20T12:07:09.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c00b5f50 msgr2=0x7f82c009cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c00b5f50 0x7f82c009cbd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f82b800bae0 tx=0x7f82b80090f0).stop 2020-11-20T12:07:09.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 shutdown_connections 2020-11-20T12:07:09.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8294056d20 0x7f82940591e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82c0068230 0x7f82c01545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82c00b5f50 0x7f82c009cbd0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 --2- 172.21.15.187:0/3570732086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82c00b57b0 0x7f82c009b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.245+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 >> 172.21.15.187:0/3570732086 conn(0x7f82c00bab30 msgr2=0x7f82c0144090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:09.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.246+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 shutdown_connections 2020-11-20T12:07:09.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.246+0000 7f82bf59e700 1 -- 172.21.15.187:0/3570732086 wait complete. 2020-11-20T12:07:09.271 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:07:01.490118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:07:02.131720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:07:09.272 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:07:09.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.514+0000 7fa533fff700 1 Processor -- start 2020-11-20T12:07:09.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.515+0000 7fa533fff700 1 -- start start 2020-11-20T12:07:09.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.516+0000 7fa533fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa5340b6650 0x7fa5340afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.516+0000 7fa533fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa534066f70 0x7fa5340aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.516+0000 7fa533fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5340b1300 con 0x7fa5341484d0 2020-11-20T12:07:09.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.516+0000 7fa533fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa534067730 con 0x7fa5340b6650 2020-11-20T12:07:09.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.516+0000 7fa533fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5340b15a0 con 0x7fa534066f70 2020-11-20T12:07:09.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.517+0000 7fa5337fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa534066f70 0x7fa5340aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.517+0000 7fa5337fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa534066f70 0x7fa5340aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43024/0 (socket says 172.21.15.187:43024) 2020-11-20T12:07:09.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.517+0000 7fa52a7fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa5340b6650 0x7fa5340afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.517+0000 7fa5337fe700 1 -- 172.21.15.187:0/4122752437 learned_addr learned my addr 172.21.15.187:0/4122752437 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:09.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.517+0000 7fa5337fe700 1 -- 172.21.15.187:0/4122752437 >> v2:172.21.15.5:3300/0 conn(0x7fa5340b6650 msgr2=0x7fa5340afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:09.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.517+0000 7fa5337fe700 1 --2- 172.21.15.187:0/4122752437 >> v2:172.21.15.5:3300/0 conn(0x7fa5340b6650 0x7fa5340afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.517+0000 7fa5337fe700 1 -- 172.21.15.187:0/4122752437 >> v1:172.21.15.5:6789/0 conn(0x7fa5341484d0 legacy=0x7fa534061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.517+0000 7fa5337fe700 1 -- 172.21.15.187:0/4122752437 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5340b1840 con 0x7fa534066f70 2020-11-20T12:07:09.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.518+0000 7fa5337fe700 1 --2- 172.21.15.187:0/4122752437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa534066f70 0x7fa5340aff80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fa534045000 tx=0x7fa52c006eb0).ready entity=mon.1 client_cookie=22bb3972d884d985 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.518+0000 7fa5327fc700 1 -- 172.21.15.187:0/4122752437 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c009810 con 0x7fa534066f70 2020-11-20T12:07:09.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.518+0000 7fa5327fc700 1 -- 172.21.15.187:0/4122752437 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c009970 con 0x7fa534066f70 2020-11-20T12:07:09.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.518+0000 7fa5327fc700 1 -- 172.21.15.187:0/4122752437 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c009cf0 con 0x7fa534066f70 2020-11-20T12:07:09.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.519+0000 7fa533fff700 1 -- 172.21.15.187:0/4122752437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa534066f70 msgr2=0x7fa5340aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.519+0000 7fa533fff700 1 --2- 172.21.15.187:0/4122752437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa534066f70 0x7fa5340aff80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fa534045000 tx=0x7fa52c006eb0).stop 2020-11-20T12:07:09.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.519+0000 7fa533fff700 1 -- 172.21.15.187:0/4122752437 shutdown_connections 2020-11-20T12:07:09.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.519+0000 7fa533fff700 1 --2- 172.21.15.187:0/4122752437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa534066f70 0x7fa5340aff80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.519+0000 7fa533fff700 1 --2- 172.21.15.187:0/4122752437 >> v2:172.21.15.5:3300/0 conn(0x7fa5340b6650 0x7fa5340afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.519+0000 7fa533fff700 1 -- 172.21.15.187:0/4122752437 >> 172.21.15.187:0/4122752437 conn(0x7fa5340bab30 msgr2=0x7fa5340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:09.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.519+0000 7fa533fff700 1 -- 172.21.15.187:0/4122752437 shutdown_connections 2020-11-20T12:07:09.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.519+0000 7fa533fff700 1 -- 172.21.15.187:0/4122752437 wait complete. 2020-11-20T12:07:09.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.521+0000 7fa533fff700 1 Processor -- start 2020-11-20T12:07:09.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.521+0000 7fa533fff700 1 -- start start 2020-11-20T12:07:09.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.522+0000 7fa533fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa534066f70 0x7fa5341381a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.522+0000 7fa533fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa5340b6650 0x7fa534138690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.522+0000 7fa533fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5341484d0 0x7fa53415b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.522+0000 7fa533fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5340b4010 con 0x7fa5340b6650 2020-11-20T12:07:09.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa533fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5340a0410 con 0x7fa5341484d0 2020-11-20T12:07:09.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa533fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5340a06b0 con 0x7fa534066f70 2020-11-20T12:07:09.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa5337fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5341484d0 0x7fa53415b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa5337fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5341484d0 0x7fa53415b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43030/0 (socket says 172.21.15.187:43030) 2020-11-20T12:07:09.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa5337fe700 1 -- 172.21.15.187:0/3576435525 learned_addr learned my addr 172.21.15.187:0/3576435525 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:09.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa5337fe700 1 -- 172.21.15.187:0/3576435525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa534066f70 msgr2=0x7fa5341381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa5337fe700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa534066f70 0x7fa5341381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:09.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa5337fe700 1 -- 172.21.15.187:0/3576435525 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa5340b6650 msgr2=0x7fa534138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa5337fe700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa5340b6650 0x7fa534138690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:09.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.523+0000 7fa5337fe700 1 -- 172.21.15.187:0/3576435525 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53409a5f0 con 0x7fa5341484d0 2020-11-20T12:07:09.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.524+0000 7fa5337fe700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5341484d0 0x7fa53415b700 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fa52c006ef0 tx=0x7fa52c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.524+0000 7fa52b7fe700 1 -- 172.21.15.187:0/3576435525 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c006b20 con 0x7fa5341484d0 2020-11-20T12:07:09.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.524+0000 7fa52b7fe700 1 -- 172.21.15.187:0/3576435525 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c01d5d0 con 0x7fa5341484d0 2020-11-20T12:07:09.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.524+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53409a7e0 con 0x7fa5341484d0 2020-11-20T12:07:09.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.524+0000 7fa52b7fe700 1 -- 172.21.15.187:0/3576435525 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c01d970 con 0x7fa5341484d0 2020-11-20T12:07:09.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.524+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53415bdb0 con 0x7fa5341484d0 2020-11-20T12:07:09.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.525+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa534061a20 con 0x7fa5341484d0 2020-11-20T12:07:09.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.526+0000 7fa52b7fe700 1 -- 172.21.15.187:0/3576435525 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa52c025030 con 0x7fa5341484d0 2020-11-20T12:07:09.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.526+0000 7fa52b7fe700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa510056d20 0x7fa5100591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.526+0000 7fa52b7fe700 1 -- 172.21.15.187:0/3576435525 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa52c071950 con 0x7fa5341484d0 2020-11-20T12:07:09.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.527+0000 7fa532ffd700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa510056d20 0x7fa5100591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.527+0000 7fa532ffd700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa510056d20 0x7fa5100591e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fa524001990 tx=0x7fa524007b90).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.530+0000 7fa52b7fe700 1 -- 172.21.15.187:0/3576435525 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa52c046da0 con 0x7fa5341484d0 2020-11-20T12:07:09.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.717+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa5340624e0 con 0x7fa5341484d0 2020-11-20T12:07:09.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.718+0000 7fa52b7fe700 1 -- 172.21.15.187:0/3576435525 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa52c023020 con 0x7fa5341484d0 2020-11-20T12:07:09.722 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:07:09.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.720+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa510056d20 msgr2=0x7fa5100591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.720+0000 7fa533fff700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa510056d20 0x7fa5100591e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fa524001990 tx=0x7fa524007b90).stop 2020-11-20T12:07:09.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.720+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5341484d0 msgr2=0x7fa53415b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.720+0000 7fa533fff700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5341484d0 0x7fa53415b700 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fa52c006ef0 tx=0x7fa52c015000).stop 2020-11-20T12:07:09.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.721+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 shutdown_connections 2020-11-20T12:07:09.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.721+0000 7fa533fff700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa510056d20 0x7fa5100591e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.721+0000 7fa533fff700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5341484d0 0x7fa53415b700 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.721+0000 7fa533fff700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa534066f70 0x7fa5341381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:09.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.721+0000 7fa533fff700 1 --2- 172.21.15.187:0/3576435525 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa5340b6650 0x7fa534138690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:09.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.721+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 >> 172.21.15.187:0/3576435525 conn(0x7fa5340bab30 msgr2=0x7fa53413cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:09.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.721+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 shutdown_connections 2020-11-20T12:07:09.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.721+0000 7fa533fff700 1 -- 172.21.15.187:0/3576435525 wait complete. 2020-11-20T12:07:09.747 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:07:09.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.971+0000 7f03e2029700 1 Processor -- start 2020-11-20T12:07:09.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.972+0000 7f03e2029700 1 -- start start 2020-11-20T12:07:09.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.973+0000 7f03e2029700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f03dc0bda00 0x7f03dc0b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.973+0000 7f03e2029700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc0a2770 0x7f03dc0a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.973+0000 7f03e2029700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc146c40 con 0x7f03dc0bd620 2020-11-20T12:07:09.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.973+0000 7f03e2029700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03dc0b4250 con 0x7f03dc0bda00 2020-11-20T12:07:09.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.973+0000 7f03e2029700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc0b44f0 con 0x7f03dc0a2770 2020-11-20T12:07:09.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.973+0000 7f03e0826700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f03dc0bda00 0x7f03dc0b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.973+0000 7f03e1027700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f03dc0bd620 0x7f03dc146530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39640/0 (socket says 172.21.15.187:39640) 2020-11-20T12:07:09.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.974+0000 7f03e1027700 1 -- 172.21.15.187:0/1055369052 learned_addr learned my addr 172.21.15.187:0/1055369052 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:09.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.974+0000 7f03e1828700 1 --2- 172.21.15.187:0/1055369052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc0a2770 0x7f03dc0a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.974+0000 7f03e0826700 1 -- 172.21.15.187:0/1055369052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc0a2770 msgr2=0x7f03dc0a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:09.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.974+0000 7f03cbfff700 1 -- 172.21.15.187:0/1055369052 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182932203 0 0) 0x7f03dc146c40 con 0x7f03dc0bd620 2020-11-20T12:07:09.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.974+0000 7f03e0826700 1 --2- 172.21.15.187:0/1055369052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc0a2770 0x7f03dc0a2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.974+0000 7f03e0826700 1 -- 172.21.15.187:0/1055369052 >> v1:172.21.15.187:6789/0 conn(0x7f03dc0bd620 legacy=0x7f03dc146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.974+0000 7f03e0826700 1 -- 172.21.15.187:0/1055369052 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc149b40 con 0x7f03dc0bda00 2020-11-20T12:07:09.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.974+0000 7f03e1828700 1 --2- 172.21.15.187:0/1055369052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc0a2770 0x7f03dc0a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:09.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.975+0000 7f03e0826700 1 --2- 172.21.15.187:0/1055369052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0bda00 0x7f03dc0b6dd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f03cc00b9a0 tx=0x7f03cc006df0).ready entity=mon.1 client_cookie=64c0941e2b77da43 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.975+0000 7f03cbfff700 1 -- 172.21.15.187:0/1055369052 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc009840 con 0x7f03dc0bda00 2020-11-20T12:07:09.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.975+0000 7f03cbfff700 1 -- 172.21.15.187:0/1055369052 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc0099a0 con 0x7f03dc0bda00 2020-11-20T12:07:09.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.975+0000 7f03cbfff700 1 -- 172.21.15.187:0/1055369052 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc009d20 con 0x7f03dc0bda00 2020-11-20T12:07:09.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.976+0000 7f03e2029700 1 -- 172.21.15.187:0/1055369052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0bda00 msgr2=0x7f03dc0b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.976+0000 7f03e2029700 1 --2- 172.21.15.187:0/1055369052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0bda00 0x7f03dc0b6dd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f03cc00b9a0 tx=0x7f03cc006df0).stop 2020-11-20T12:07:09.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.976+0000 7f03e2029700 1 -- 172.21.15.187:0/1055369052 shutdown_connections 2020-11-20T12:07:09.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.976+0000 7f03e2029700 1 --2- 172.21.15.187:0/1055369052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc0a2770 0x7f03dc0a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.976+0000 7f03e2029700 1 --2- 172.21.15.187:0/1055369052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0bda00 0x7f03dc0b6dd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:09.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.976+0000 7f03e2029700 1 -- 172.21.15.187:0/1055369052 >> 172.21.15.187:0/1055369052 conn(0x7f03dc060c40 msgr2=0x7f03dc0bcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:09.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.976+0000 7f03e2029700 1 -- 172.21.15.187:0/1055369052 shutdown_connections 2020-11-20T12:07:09.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.976+0000 7f03e2029700 1 -- 172.21.15.187:0/1055369052 wait complete. 2020-11-20T12:07:09.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.978+0000 7f03e2029700 1 Processor -- start 2020-11-20T12:07:09.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.978+0000 7f03e2029700 1 -- start start 2020-11-20T12:07:09.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e2029700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0a2770 0x7f03dc0b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e2029700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03dc0bd620 0x7f03dc1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e2029700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc115d30 0x7f03dc145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e2029700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc0b9db0 con 0x7f03dc0bd620 2020-11-20T12:07:09.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e2029700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc0ba050 con 0x7f03dc0a2770 2020-11-20T12:07:09.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e2029700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc0ba2f0 con 0x7f03dc115d30 2020-11-20T12:07:09.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e1027700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0a2770 0x7f03dc0b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e1027700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0a2770 0x7f03dc0b97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43040/0 (socket says 172.21.15.187:43040) 2020-11-20T12:07:09.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.979+0000 7f03e1027700 1 -- 172.21.15.187:0/2228003224 learned_addr learned my addr 172.21.15.187:0/2228003224 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:09.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03e1027700 1 -- 172.21.15.187:0/2228003224 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc115d30 msgr2=0x7f03dc145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03e1027700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc115d30 0x7f03dc145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:09.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03e1027700 1 -- 172.21.15.187:0/2228003224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03dc0bd620 msgr2=0x7f03dc1585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:09.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03e1027700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03dc0bd620 0x7f03dc1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:09.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03e1027700 1 -- 172.21.15.187:0/2228003224 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0ba790 con 0x7f03dc0a2770 2020-11-20T12:07:09.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03e1027700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0a2770 0x7f03dc0b97f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f03d800cae0 tx=0x7f03d8007e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03c97fa700 1 -- 172.21.15.187:0/2228003224 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d8005b80 con 0x7f03dc0a2770 2020-11-20T12:07:09.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03c97fa700 1 -- 172.21.15.187:0/2228003224 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d8005ce0 con 0x7f03dc0a2770 2020-11-20T12:07:09.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03c97fa700 1 -- 172.21.15.187:0/2228003224 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d8007320 con 0x7f03dc0a2770 2020-11-20T12:07:09.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03dc158ac0 con 0x7f03dc0a2770 2020-11-20T12:07:09.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.980+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03dc1458c0 con 0x7f03dc0a2770 2020-11-20T12:07:09.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.981+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03dc1464c0 con 0x7f03dc0a2770 2020-11-20T12:07:09.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.982+0000 7f03c97fa700 1 -- 172.21.15.187:0/2228003224 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f03d8010030 con 0x7f03dc0a2770 2020-11-20T12:07:09.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.983+0000 7f03c97fa700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d20 0x7f03b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:09.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.983+0000 7f03c97fa700 1 -- 172.21.15.187:0/2228003224 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f03d8070240 con 0x7f03dc0a2770 2020-11-20T12:07:09.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.983+0000 7f03e0826700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d20 0x7f03b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:09.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.983+0000 7f03e0826700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d20 0x7f03b40591e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f03dc0b9ce0 tx=0x7f03cc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:09.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:09.986+0000 7f03c97fa700 1 -- 172.21.15.187:0/2228003224 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f03d8045580 con 0x7f03dc0a2770 2020-11-20T12:07:10.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.176+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f03dc0ba4b0 con 0x7f03dc0a2770 2020-11-20T12:07:10.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.177+0000 7f03c97fa700 1 -- 172.21.15.187:0/2228003224 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v42) v1 ==== 92+0+10 (secure 0 0 0) 0x7f03d8048ba0 con 0x7f03dc0a2770 2020-11-20T12:07:10.181 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:07:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d20 msgr2=0x7f03b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d20 0x7f03b40591e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f03dc0b9ce0 tx=0x7f03cc015000).stop 2020-11-20T12:07:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0a2770 msgr2=0x7f03dc0b97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0a2770 0x7f03dc0b97f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f03d800cae0 tx=0x7f03d8007e40).stop 2020-11-20T12:07:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 shutdown_connections 2020-11-20T12:07:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d20 0x7f03b40591e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc115d30 0x7f03dc145190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:10.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03dc0a2770 0x7f03dc0b97f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 --2- 172.21.15.187:0/2228003224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03dc0bd620 0x7f03dc1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:10.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 >> 172.21.15.187:0/2228003224 conn(0x7f03dc060c40 msgr2=0x7f03dc0bcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:10.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 shutdown_connections 2020-11-20T12:07:10.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.179+0000 7f03e2029700 1 -- 172.21.15.187:0/2228003224 wait complete. 2020-11-20T12:07:10.205 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:07:10.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.459+0000 7fcdf738f700 1 Processor -- start 2020-11-20T12:07:10.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.459+0000 7fcdf738f700 1 -- start start 2020-11-20T12:07:10.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.460+0000 7fcdf738f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdf00683f0 0x7fcdf0066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.460+0000 7fcdf738f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf014c170 con 0x7fcdf0145cc0 2020-11-20T12:07:10.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.461+0000 7fcdf738f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf014c410 con 0x7fcdf01487e0 2020-11-20T12:07:10.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.461+0000 7fcdf738f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf014c6b0 con 0x7fcdf00683f0 2020-11-20T12:07:10.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.461+0000 7fcdf5b8c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdf00683f0 0x7fcdf0066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.461+0000 7fcdf6b8e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcdf01487e0 0x7fcdf0148c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39656/0 (socket says 172.21.15.187:39656) 2020-11-20T12:07:10.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.461+0000 7fcdf5b8c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdf00683f0 0x7fcdf0066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43052/0 (socket says 172.21.15.187:43052) 2020-11-20T12:07:10.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.461+0000 7fcdf6b8e700 1 -- 172.21.15.187:0/729890493 learned_addr learned my addr 172.21.15.187:0/729890493 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:10.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.462+0000 7fcdf5b8c700 1 -- 172.21.15.187:0/729890493 >> v1:172.21.15.187:6789/0 conn(0x7fcdf01487e0 legacy=0x7fcdf0148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.462+0000 7fcdf5b8c700 1 -- 172.21.15.187:0/729890493 >> v1:172.21.15.5:6789/0 conn(0x7fcdf0145cc0 legacy=0x7fcdf0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.462+0000 7fcdf5b8c700 1 -- 172.21.15.187:0/729890493 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf00658a0 con 0x7fcdf00683f0 2020-11-20T12:07:10.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.462+0000 7fcdf5b8c700 1 --2- 172.21.15.187:0/729890493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf00683f0 0x7fcdf0066040 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fcde000b9a0 tx=0x7fcde0006df0).ready entity=mon.1 client_cookie=8fec930829a1f77c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:10.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.462+0000 7fcdf538b700 1 -- 172.21.15.187:0/729890493 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde00097b0 con 0x7fcdf00683f0 2020-11-20T12:07:10.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.462+0000 7fcdf538b700 1 -- 172.21.15.187:0/729890493 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde0009910 con 0x7fcdf00683f0 2020-11-20T12:07:10.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.462+0000 7fcdf538b700 1 -- 172.21.15.187:0/729890493 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0009c90 con 0x7fcdf00683f0 2020-11-20T12:07:10.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.463+0000 7fcdf738f700 1 -- 172.21.15.187:0/729890493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf00683f0 msgr2=0x7fcdf0066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.463+0000 7fcdf738f700 1 --2- 172.21.15.187:0/729890493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf00683f0 0x7fcdf0066040 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fcde000b9a0 tx=0x7fcde0006df0).stop 2020-11-20T12:07:10.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.463+0000 7fcdf738f700 1 -- 172.21.15.187:0/729890493 shutdown_connections 2020-11-20T12:07:10.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.463+0000 7fcdf738f700 1 --2- 172.21.15.187:0/729890493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf00683f0 0x7fcdf0066040 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.463+0000 7fcdf738f700 1 -- 172.21.15.187:0/729890493 >> 172.21.15.187:0/729890493 conn(0x7fcdf00bab30 msgr2=0x7fcdf00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:10.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.463+0000 7fcdf738f700 1 -- 172.21.15.187:0/729890493 shutdown_connections 2020-11-20T12:07:10.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.463+0000 7fcdf738f700 1 -- 172.21.15.187:0/729890493 wait complete. 2020-11-20T12:07:10.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.465+0000 7fcdf738f700 1 Processor -- start 2020-11-20T12:07:10.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.466+0000 7fcdf738f700 1 -- start start 2020-11-20T12:07:10.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf738f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdf00683f0 0x7fcdf00b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf738f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdf0145cc0 0x7fcdf0145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf738f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf01487e0 0x7fcdf009dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf738f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf01398a0 con 0x7fcdf00683f0 2020-11-20T12:07:10.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf738f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf01455f0 con 0x7fcdf01487e0 2020-11-20T12:07:10.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf738f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf009b920 con 0x7fcdf0145cc0 2020-11-20T12:07:10.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf6b8e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf01487e0 0x7fcdf009dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf6b8e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf01487e0 0x7fcdf009dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43058/0 (socket says 172.21.15.187:43058) 2020-11-20T12:07:10.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.467+0000 7fcdf6b8e700 1 -- 172.21.15.187:0/3477089559 learned_addr learned my addr 172.21.15.187:0/3477089559 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:10.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf638d700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdf00683f0 0x7fcdf00b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf5b8c700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdf0145cc0 0x7fcdf0145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf6b8e700 1 -- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdf0145cc0 msgr2=0x7fcdf0145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf6b8e700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdf0145cc0 0x7fcdf0145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf6b8e700 1 -- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdf00683f0 msgr2=0x7fcdf00b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf6b8e700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdf00683f0 0x7fcdf00b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf6b8e700 1 -- 172.21.15.187:0/3477089559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf009bbc0 con 0x7fcdf01487e0 2020-11-20T12:07:10.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf638d700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdf00683f0 0x7fcdf00b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:10.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf6b8e700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf01487e0 0x7fcdf009dae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fcde800c6e0 tx=0x7fcde8007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:10.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdde7fc700 1 -- 172.21.15.187:0/3477089559 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde80072e0 con 0x7fcdf01487e0 2020-11-20T12:07:10.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdde7fc700 1 -- 172.21.15.187:0/3477089559 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde8007440 con 0x7fcdf01487e0 2020-11-20T12:07:10.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf009be60 con 0x7fcdf01487e0 2020-11-20T12:07:10.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdde7fc700 1 -- 172.21.15.187:0/3477089559 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8005800 con 0x7fcdf01487e0 2020-11-20T12:07:10.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.468+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf009e050 con 0x7fcdf01487e0 2020-11-20T12:07:10.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.470+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf0061a20 con 0x7fcdf01487e0 2020-11-20T12:07:10.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.470+0000 7fcdde7fc700 1 -- 172.21.15.187:0/3477089559 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcde801a060 con 0x7fcdf01487e0 2020-11-20T12:07:10.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.471+0000 7fcdde7fc700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdc8056d50 0x7fcdc8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.471+0000 7fcdde7fc700 1 -- 172.21.15.187:0/3477089559 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcde806f790 con 0x7fcdf01487e0 2020-11-20T12:07:10.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.471+0000 7fcdf638d700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdc8056d50 0x7fcdc8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.472+0000 7fcdf638d700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdc8056d50 0x7fcdc8059210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fcdec001bc0 tx=0x7fcdec009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:10.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.475+0000 7fcdde7fc700 1 -- 172.21.15.187:0/3477089559 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcde8044be0 con 0x7fcdf01487e0 2020-11-20T12:07:10.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.653+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcdf014bdf0 con 0x7fcdf01487e0 2020-11-20T12:07:10.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.654+0000 7fcdde7fc700 1 -- 172.21.15.187:0/3477089559 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcde80180a0 con 0x7fcdf01487e0 2020-11-20T12:07:10.658 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:07:10.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdc8056d50 msgr2=0x7fcdc8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdc8056d50 0x7fcdc8059210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fcdec001bc0 tx=0x7fcdec009040).stop 2020-11-20T12:07:10.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf01487e0 msgr2=0x7fcdf009dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf01487e0 0x7fcdf009dae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fcde800c6e0 tx=0x7fcde8007c30).stop 2020-11-20T12:07:10.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 shutdown_connections 2020-11-20T12:07:10.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdc8056d50 0x7fcdc8059210 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdf00683f0 0x7fcdf00b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdf01487e0 0x7fcdf009dae0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 --2- 172.21.15.187:0/3477089559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdf0145cc0 0x7fcdf0145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 >> 172.21.15.187:0/3477089559 conn(0x7fcdf00bab30 msgr2=0x7fcdf0147710 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:10.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.656+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 shutdown_connections 2020-11-20T12:07:10.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.657+0000 7fcdf738f700 1 -- 172.21.15.187:0/3477089559 wait complete. 2020-11-20T12:07:10.681 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:07:10.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.916+0000 7f2855851700 1 Processor -- start 2020-11-20T12:07:10.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.916+0000 7f2855851700 1 -- start start 2020-11-20T12:07:10.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.917+0000 7f2855851700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2850066040 0x7f2850148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.917+0000 7f2855851700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285014c170 con 0x7f28500683f0 2020-11-20T12:07:10.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.917+0000 7f2855851700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285014c410 con 0x7f2850145cc0 2020-11-20T12:07:10.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.917+0000 7f2855851700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285014c6b0 con 0x7f2850066040 2020-11-20T12:07:10.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.918+0000 7f2847fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f28500683f0 0x7f28501487e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33178/0 (socket says 172.21.15.187:33178) 2020-11-20T12:07:10.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.918+0000 7f2847fff700 1 -- 172.21.15.187:0/1209672134 learned_addr learned my addr 172.21.15.187:0/1209672134 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:10.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.918+0000 7f285484f700 1 --1- 172.21.15.187:0/1209672134 >> v1:172.21.15.187:6789/0 conn(0x7f2850145cc0 0x7f2850065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39670/0 (socket says 172.21.15.187:39670) 2020-11-20T12:07:10.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.919+0000 7f2855050700 1 --2- 172.21.15.187:0/1209672134 >> v2:172.21.15.5:3300/0 conn(0x7f2850066040 0x7f2850148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.919+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222664853 0 0) 0x7f285014c410 con 0x7f2850145cc0 2020-11-20T12:07:10.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.919+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c003240 con 0x7f2850145cc0 2020-11-20T12:07:10.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.919+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635390985 0 0) 0x7f285014c170 con 0x7f28500683f0 2020-11-20T12:07:10.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.919+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f285014c410 con 0x7f28500683f0 2020-11-20T12:07:10.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 835204204 0 0) 0x7f283c003240 con 0x7f2850145cc0 2020-11-20T12:07:10.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 >> v2:172.21.15.5:3300/0 conn(0x7f2850066040 msgr2=0x7f2850148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:10.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 --2- 172.21.15.187:0/1209672134 >> v2:172.21.15.5:3300/0 conn(0x7f2850066040 0x7f2850148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 >> v1:172.21.15.5:6789/0 conn(0x7f28500683f0 legacy=0x7f28501487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500658a0 con 0x7f2850145cc0 2020-11-20T12:07:10.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f284c0014e0 con 0x7f2850145cc0 2020-11-20T12:07:10.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 >> v1:172.21.15.187:6789/0 conn(0x7f2850145cc0 legacy=0x7f2850065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f283c004920 0x7f283c004d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f283c003900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f283c003df0 0x7f283c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285014c6b0 con 0x7f283c003df0 2020-11-20T12:07:10.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c004350 con 0x7f283c003480 2020-11-20T12:07:10.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283c00bca0 con 0x7f283c004920 2020-11-20T12:07:10.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.920+0000 7f2847fff700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f283c003900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f285484f700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f283c004920 0x7f283c004d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f2855050700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f283c003df0 0x7f283c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f2847fff700 1 -- 172.21.15.187:0/1209672134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f283c004920 msgr2=0x7f283c004d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:10.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f2847fff700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f283c004920 0x7f283c004d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f2847fff700 1 -- 172.21.15.187:0/1209672134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f283c003df0 msgr2=0x7f283c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:10.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f2847fff700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f283c003df0 0x7f283c00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f2847fff700 1 -- 172.21.15.187:0/1209672134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c003240 con 0x7f283c003480 2020-11-20T12:07:10.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f2855050700 1 -- 172.21.15.187:0/1209672134 reap_dead start 2020-11-20T12:07:10.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.921+0000 7f2847fff700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f283c003900 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f2840001a50 tx=0x7f2840007c80).ready entity=mon.1 client_cookie=edf55865d52b85b5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:10.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28400072e0 con 0x7f283c003480 2020-11-20T12:07:10.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2840007440 con 0x7f283c003480 2020-11-20T12:07:10.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f28477fe700 1 -- 172.21.15.187:0/1209672134 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284000c9d0 con 0x7f283c003480 2020-11-20T12:07:10.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f2855851700 1 -- 172.21.15.187:0/1209672134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 msgr2=0x7f283c003900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f2855851700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f283c003900 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f2840001a50 tx=0x7f2840007c80).stop 2020-11-20T12:07:10.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f2855851700 1 -- 172.21.15.187:0/1209672134 shutdown_connections 2020-11-20T12:07:10.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f2855851700 1 --2- 172.21.15.187:0/1209672134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f283c003900 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f2855851700 1 -- 172.21.15.187:0/1209672134 >> 172.21.15.187:0/1209672134 conn(0x7f28500bab30 msgr2=0x7f28500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:10.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f2855851700 1 -- 172.21.15.187:0/1209672134 shutdown_connections 2020-11-20T12:07:10.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.922+0000 7f2855851700 1 -- 172.21.15.187:0/1209672134 wait complete. 2020-11-20T12:07:10.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.924+0000 7f2855851700 1 Processor -- start 2020-11-20T12:07:10.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.924+0000 7f2855851700 1 -- start start 2020-11-20T12:07:10.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f2855851700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f285009cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f2855851700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f285009b3d0 0x7f285013b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f2855851700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f285009b850 0x7f28500b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f2855851700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285009bd50 con 0x7f285009b850 2020-11-20T12:07:10.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f2855851700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500b2f70 con 0x7f283c003480 2020-11-20T12:07:10.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f2855851700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28500b3210 con 0x7f285009b3d0 2020-11-20T12:07:10.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f285484f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f285009cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f285484f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f285009cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43074/0 (socket says 172.21.15.187:43074) 2020-11-20T12:07:10.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.925+0000 7f285484f700 1 -- 172.21.15.187:0/107831091 learned_addr learned my addr 172.21.15.187:0/107831091 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:10.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f2855050700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f285009b850 0x7f28500b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f2847fff700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f285009b3d0 0x7f285013b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f285484f700 1 -- 172.21.15.187:0/107831091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f285009b3d0 msgr2=0x7f285013b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f285484f700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f285009b3d0 0x7f285013b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f285484f700 1 -- 172.21.15.187:0/107831091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f285009b850 msgr2=0x7f28500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:10.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f285484f700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f285009b850 0x7f28500b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:10.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f285484f700 1 -- 172.21.15.187:0/107831091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850141060 con 0x7f283c003480 2020-11-20T12:07:10.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f285484f700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f285009cf90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f284c00eae0 tx=0x7f284c000f50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:10.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f2844ff9700 1 -- 172.21.15.187:0/107831091 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c00c870 con 0x7f283c003480 2020-11-20T12:07:10.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f2844ff9700 1 -- 172.21.15.187:0/107831091 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c00c9f0 con 0x7f283c003480 2020-11-20T12:07:10.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f2844ff9700 1 -- 172.21.15.187:0/107831091 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c00cdb0 con 0x7f283c003480 2020-11-20T12:07:10.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.926+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285009d480 con 0x7f283c003480 2020-11-20T12:07:10.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.927+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285013b7b0 con 0x7f283c003480 2020-11-20T12:07:10.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.928+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2850061a20 con 0x7f283c003480 2020-11-20T12:07:10.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.929+0000 7f2844ff9700 1 -- 172.21.15.187:0/107831091 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f284c001130 con 0x7f283c003480 2020-11-20T12:07:10.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.930+0000 7f2844ff9700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2828056d20 0x7f28280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:10.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.930+0000 7f2844ff9700 1 -- 172.21.15.187:0/107831091 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f284c06fd30 con 0x7f283c003480 2020-11-20T12:07:10.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.930+0000 7f2847fff700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2828056d20 0x7f28280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:10.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.931+0000 7f2847fff700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2828056d20 0x7f28280591e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f2840001e90 tx=0x7f2840015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:10.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:10.939+0000 7f2844ff9700 1 -- 172.21.15.187:0/107831091 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f284c045180 con 0x7f283c003480 2020-11-20T12:07:11.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.129+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f285014bdf0 con 0x7f283c003480 2020-11-20T12:07:11.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.129+0000 7f2844ff9700 1 -- 172.21.15.187:0/107831091 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+10 (secure 0 0 0) 0x7f284c0487a0 con 0x7f283c003480 2020-11-20T12:07:11.134 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:07:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.131+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2828056d20 msgr2=0x7f28280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.131+0000 7f2855851700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2828056d20 0x7f28280591e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f2840001e90 tx=0x7f2840015000).stop 2020-11-20T12:07:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.131+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 msgr2=0x7f285009cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.131+0000 7f2855851700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f285009cf90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f284c00eae0 tx=0x7f284c000f50).stop 2020-11-20T12:07:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.132+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 shutdown_connections 2020-11-20T12:07:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.132+0000 7f2855851700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2828056d20 0x7f28280591e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.132+0000 7f2855851700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f285009b850 0x7f28500b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.132+0000 7f2855851700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f283c003480 0x7f285009cf90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.132+0000 7f2855851700 1 --2- 172.21.15.187:0/107831091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f285009b3d0 0x7f285013b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.132+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 >> 172.21.15.187:0/107831091 conn(0x7f28500bab30 msgr2=0x7f2850142a70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.132+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 shutdown_connections 2020-11-20T12:07:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.132+0000 7f2855851700 1 -- 172.21.15.187:0/107831091 wait complete. 2020-11-20T12:07:11.160 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:07:11.161 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:11.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.399+0000 7f7bdee85700 1 Processor -- start 2020-11-20T12:07:11.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.399+0000 7f7bdee85700 1 -- start start 2020-11-20T12:07:11.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.400+0000 7f7bdee85700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7bd8066f70 0x7f7bd80b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:11.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.400+0000 7f7bdee85700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd80b1300 con 0x7f7bd81484d0 2020-11-20T12:07:11.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.400+0000 7f7bdee85700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd80b15a0 con 0x7f7bd80b6650 2020-11-20T12:07:11.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.400+0000 7f7bdee85700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd8067730 con 0x7f7bd8066f70 2020-11-20T12:07:11.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.400+0000 7f7bdd682700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7bd80b6650 0x7f7bd80afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39690/0 (socket says 172.21.15.187:39690) 2020-11-20T12:07:11.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.400+0000 7f7bdd682700 1 -- 172.21.15.187:0/1771384022 learned_addr learned my addr 172.21.15.187:0/1771384022 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:11.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215284551 0 0) 0x7f7bd80b15a0 con 0x7f7bd80b6650 2020-11-20T12:07:11.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc0003240 con 0x7f7bd80b6650 2020-11-20T12:07:11.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bde684700 1 --2- 172.21.15.187:0/1771384022 >> v2:172.21.15.5:3300/0 conn(0x7f7bd8066f70 0x7f7bd80b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:11.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2656983603 0 0) 0x7f7bc0003240 con 0x7f7bd80b6650 2020-11-20T12:07:11.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 >> v2:172.21.15.5:3300/0 conn(0x7f7bd8066f70 msgr2=0x7f7bd80b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:11.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 --2- 172.21.15.187:0/1771384022 >> v2:172.21.15.5:3300/0 conn(0x7f7bd8066f70 0x7f7bd80b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 >> v1:172.21.15.5:6789/0 conn(0x7f7bd81484d0 legacy=0x7f7bd8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd80b1840 con 0x7f7bd80b6650 2020-11-20T12:07:11.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f7bc80014e0 con 0x7f7bd80b6650 2020-11-20T12:07:11.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.401+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 >> v1:172.21.15.187:6789/0 conn(0x7f7bd80b6650 legacy=0x7f7bd80afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bdce81700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bc0003730 0x7f7bc0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:11.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bdce81700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bc00044e0 0x7f7bc0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:11.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bdce81700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bc0004e50 0x7f7bc000fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:11.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd8067730 con 0x7f7bc00044e0 2020-11-20T12:07:11.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc0004260 con 0x7f7bc0003730 2020-11-20T12:07:11.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc0010030 con 0x7f7bc0004e50 2020-11-20T12:07:11.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bdd682700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bc00044e0 0x7f7bc0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:11.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bde684700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bc0004e50 0x7f7bc000fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:11.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.402+0000 7f7bdde83700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bc0003730 0x7f7bc0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:11.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.403+0000 7f7bdd682700 1 -- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bc0004e50 msgr2=0x7f7bc000fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:11.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.403+0000 7f7bdd682700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bc0004e50 0x7f7bc000fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.403+0000 7f7bdd682700 1 -- 172.21.15.187:0/1771384022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bc0003730 msgr2=0x7f7bc0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:11.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.403+0000 7f7bdd682700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bc0003730 0x7f7bc0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.403+0000 7f7bdd682700 1 -- 172.21.15.187:0/1771384022 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc0003240 con 0x7f7bc00044e0 2020-11-20T12:07:11.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.403+0000 7f7bde684700 1 -- 172.21.15.187:0/1771384022 reap_dead start 2020-11-20T12:07:11.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.404+0000 7f7bdd682700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bc00044e0 0x7f7bc0004960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7bc800c410 tx=0x7f7bc80077d0).ready entity=mon.0 client_cookie=fd335630f5f5d305 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:11.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.404+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc80070d0 con 0x7f7bc00044e0 2020-11-20T12:07:11.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.404+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc8007230 con 0x7f7bc00044e0 2020-11-20T12:07:11.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.404+0000 7f7bdce81700 1 -- 172.21.15.187:0/1771384022 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc80055f0 con 0x7f7bc00044e0 2020-11-20T12:07:11.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.405+0000 7f7bdee85700 1 -- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bc00044e0 msgr2=0x7f7bc0004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.405+0000 7f7bdee85700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bc00044e0 0x7f7bc0004960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7bc800c410 tx=0x7f7bc80077d0).stop 2020-11-20T12:07:11.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.405+0000 7f7bdee85700 1 -- 172.21.15.187:0/1771384022 shutdown_connections 2020-11-20T12:07:11.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.405+0000 7f7bdee85700 1 --2- 172.21.15.187:0/1771384022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bc00044e0 0x7f7bc0004960 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.405+0000 7f7bdee85700 1 -- 172.21.15.187:0/1771384022 >> 172.21.15.187:0/1771384022 conn(0x7f7bd80bab30 msgr2=0x7f7bd80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:11.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.405+0000 7f7bdee85700 1 -- 172.21.15.187:0/1771384022 shutdown_connections 2020-11-20T12:07:11.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.405+0000 7f7bdee85700 1 -- 172.21.15.187:0/1771384022 wait complete. 2020-11-20T12:07:11.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.407+0000 7f7bdee85700 1 Processor -- start 2020-11-20T12:07:11.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.407+0000 7f7bdee85700 1 -- start start 2020-11-20T12:07:11.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.408+0000 7f7bdee85700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bd809cfe0 0x7f7bd809b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:11.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdee85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bd809d3c0 0x7f7bd8141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:11.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdee85700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bd8142430 0x7f7bd80b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:11.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdee85700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd8142890 con 0x7f7bd809cfe0 2020-11-20T12:07:11.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdee85700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd809ba00 con 0x7f7bd809d3c0 2020-11-20T12:07:11.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdee85700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd809bca0 con 0x7f7bd8142430 2020-11-20T12:07:11.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdd682700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bd809d3c0 0x7f7bd8141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:11.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdd682700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bd809d3c0 0x7f7bd8141f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43096/0 (socket says 172.21.15.187:43096) 2020-11-20T12:07:11.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdd682700 1 -- 172.21.15.187:0/206092541 learned_addr learned my addr 172.21.15.187:0/206092541 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:11.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdde83700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bd809cfe0 0x7f7bd809b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:11.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdd682700 1 -- 172.21.15.187:0/206092541 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bd8142430 msgr2=0x7f7bd80b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdd682700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bd8142430 0x7f7bd80b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:11.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdd682700 1 -- 172.21.15.187:0/206092541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bd809cfe0 msgr2=0x7f7bd809b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdd682700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bd809cfe0 0x7f7bd809b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.409+0000 7f7bdd682700 1 -- 172.21.15.187:0/206092541 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8154000 con 0x7f7bd809d3c0 2020-11-20T12:07:11.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.410+0000 7f7bdd682700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bd809d3c0 0x7f7bd8141f40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f7bc800a000 tx=0x7f7bc8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:11.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.410+0000 7f7bc5ffb700 1 -- 172.21.15.187:0/206092541 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8001a80 con 0x7f7bd809d3c0 2020-11-20T12:07:11.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.410+0000 7f7bc5ffb700 1 -- 172.21.15.187:0/206092541 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc8001460 con 0x7f7bd809d3c0 2020-11-20T12:07:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.410+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd80b2bc0 con 0x7f7bd809d3c0 2020-11-20T12:07:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.410+0000 7f7bc5ffb700 1 -- 172.21.15.187:0/206092541 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc80136a0 con 0x7f7bd809d3c0 2020-11-20T12:07:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.410+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd809b760 con 0x7f7bd809d3c0 2020-11-20T12:07:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.412+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd8061a20 con 0x7f7bd809d3c0 2020-11-20T12:07:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.412+0000 7f7bc5ffb700 1 -- 172.21.15.187:0/206092541 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7bc8026020 con 0x7f7bd809d3c0 2020-11-20T12:07:11.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.413+0000 7f7bc5ffb700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb0056d50 0x7f7bb0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:11.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.413+0000 7f7bc5ffb700 1 -- 172.21.15.187:0/206092541 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7bc8071510 con 0x7f7bd809d3c0 2020-11-20T12:07:11.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.414+0000 7f7bdde83700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb0056d50 0x7f7bb0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:11.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.414+0000 7f7bdde83700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb0056d50 0x7f7bb0059210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f7bd4007fd0 tx=0x7f7bd4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:11.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.418+0000 7f7bc5ffb700 1 -- 172.21.15.187:0/206092541 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7bc8046850 con 0x7f7bd809d3c0 2020-11-20T12:07:11.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.662+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bd81488b0 con 0x7f7bd809d3c0 2020-11-20T12:07:11.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.663+0000 7f7bc5ffb700 1 -- 172.21.15.187:0/206092541 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f7bc8049e70 con 0x7f7bd809d3c0 2020-11-20T12:07:11.669 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:11.670 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:11.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.665+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb0056d50 msgr2=0x7f7bb0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.665+0000 7f7bdee85700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb0056d50 0x7f7bb0059210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f7bd4007fd0 tx=0x7f7bd4009040).stop 2020-11-20T12:07:11.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.665+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bd809d3c0 msgr2=0x7f7bd8141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:11.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.665+0000 7f7bdee85700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bd809d3c0 0x7f7bd8141f40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f7bc800a000 tx=0x7f7bc8015000).stop 2020-11-20T12:07:11.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.666+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 shutdown_connections 2020-11-20T12:07:11.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.666+0000 7f7bdee85700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb0056d50 0x7f7bb0059210 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.666+0000 7f7bdee85700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bd809cfe0 0x7f7bd809b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.666+0000 7f7bdee85700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bd8142430 0x7f7bd80b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:11.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.666+0000 7f7bdee85700 1 --2- 172.21.15.187:0/206092541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bd809d3c0 0x7f7bd8141f40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:11.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.666+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 >> 172.21.15.187:0/206092541 conn(0x7f7bd80bab30 msgr2=0x7f7bd80b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:11.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.666+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 shutdown_connections 2020-11-20T12:07:11.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:11.666+0000 7f7bdee85700 1 -- 172.21.15.187:0/206092541 wait complete. 2020-11-20T12:07:11.674 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:11.893 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:07:03.348416+0000', 'modified': '2020-11-20T12:07:05.571685+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5335}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5335': {'gid': 5335, 'name': 'a', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/2880645780', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2880645780}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2880645780}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:07:11.894 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:07:11.894 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:07:12.175 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":5335,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":8.7977025619999996,"mdsmap_epoch":19,"osdmap_epoch":42,"osdmap_epoch_barrier":41,"uptime":14.372982447} 2020-11-20T12:07:12.176 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5335, "mdsmap_epoch": 19, "osdmap_epoch": 42, "osdmap_epoch_barrier": 41, "rank_uptime": 8.797702562, "state": "up:active", "uptime": 14.372982447, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:07:12.176 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:07:12.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:12.177 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:07:12.230 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:07:12.231 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:07:12.373 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:07:12.374 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:07:12.412 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:07:12.412 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:07:12.419 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:07:12.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:12.420 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:07:12.505 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:12.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.727+0000 7fb84a5ee700 1 Processor -- start 2020-11-20T12:07:12.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.727+0000 7fb84a5ee700 1 -- start start 2020-11-20T12:07:12.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.728+0000 7fb84a5ee700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb844066540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:12.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.729+0000 7fb84a5ee700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8440b8000 0x7fb844066a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:12.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.729+0000 7fb84a5ee700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8440b65d0 con 0x7fb844060330 2020-11-20T12:07:12.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.729+0000 7fb84a5ee700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8440677d0 con 0x7fb8440b8000 2020-11-20T12:07:12.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.729+0000 7fb84a5ee700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb844068380 con 0x7fb844060710 2020-11-20T12:07:12.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.729+0000 7fb8495ec700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb844060330 0x7fb844067c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33222/0 (socket says 172.21.15.187:33222) 2020-11-20T12:07:12.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.730+0000 7fb8495ec700 1 -- 172.21.15.187:0/3651710640 learned_addr learned my addr 172.21.15.187:0/3651710640 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:12.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.730+0000 7fb848deb700 1 --2- 172.21.15.187:0/3651710640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb844066540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:12.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.730+0000 7fb849ded700 1 --2- 172.21.15.187:0/3651710640 >> v2:172.21.15.5:3300/0 conn(0x7fb8440b8000 0x7fb844066a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:12.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.731+0000 7fb833fff700 1 -- 172.21.15.187:0/3651710640 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908563108 0 0) 0x7fb8440b65d0 con 0x7fb844060330 2020-11-20T12:07:12.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.731+0000 7fb833fff700 1 -- 172.21.15.187:0/3651710640 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c003240 con 0x7fb844060330 2020-11-20T12:07:12.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.731+0000 7fb848deb700 1 -- 172.21.15.187:0/3651710640 >> v2:172.21.15.5:3300/0 conn(0x7fb8440b8000 msgr2=0x7fb844066a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:12.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.731+0000 7fb848deb700 1 --2- 172.21.15.187:0/3651710640 >> v2:172.21.15.5:3300/0 conn(0x7fb8440b8000 0x7fb844066a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:12.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.731+0000 7fb848deb700 1 -- 172.21.15.187:0/3651710640 >> v1:172.21.15.5:6789/0 conn(0x7fb844060330 legacy=0x7fb844067c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:12.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.731+0000 7fb848deb700 1 -- 172.21.15.187:0/3651710640 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8440b5f30 con 0x7fb844060710 2020-11-20T12:07:12.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.731+0000 7fb848deb700 1 --2- 172.21.15.187:0/3651710640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb844066540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb83400b9a0 tx=0x7fb834006df0).ready entity=mon.2 client_cookie=4eab5004e77907bd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:12.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb833fff700 1 -- 172.21.15.187:0/3651710640 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834009710 con 0x7fb844060710 2020-11-20T12:07:12.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb833fff700 1 -- 172.21.15.187:0/3651710640 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb834009870 con 0x7fb844060710 2020-11-20T12:07:12.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb833fff700 1 -- 172.21.15.187:0/3651710640 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834009bf0 con 0x7fb844060710 2020-11-20T12:07:12.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3651710640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 msgr2=0x7fb844066540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:12.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3651710640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb844066540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb83400b9a0 tx=0x7fb834006df0).stop 2020-11-20T12:07:12.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3651710640 shutdown_connections 2020-11-20T12:07:12.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3651710640 >> v2:172.21.15.5:3300/0 conn(0x7fb8440b8000 0x7fb844066a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:12.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3651710640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb844066540 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:12.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.732+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3651710640 >> 172.21.15.187:0/3651710640 conn(0x7fb8440bbc50 msgr2=0x7fb8440bc060 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:12.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.733+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3651710640 shutdown_connections 2020-11-20T12:07:12.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.733+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3651710640 wait complete. 2020-11-20T12:07:12.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.734+0000 7fb84a5ee700 1 Processor -- start 2020-11-20T12:07:12.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.734+0000 7fb84a5ee700 1 -- start start 2020-11-20T12:07:12.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.735+0000 7fb84a5ee700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb844060330 0x7fb84409dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:12.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.735+0000 7fb84a5ee700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb84409e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:12.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.735+0000 7fb84a5ee700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8440b8000 0x7fb844148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:12.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.735+0000 7fb84a5ee700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84414b720 con 0x7fb8440b8000 2020-11-20T12:07:12.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.735+0000 7fb84a5ee700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84415c280 con 0x7fb844060330 2020-11-20T12:07:12.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.735+0000 7fb84a5ee700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84415c520 con 0x7fb844060710 2020-11-20T12:07:12.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb8495ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb844060330 0x7fb84409dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:12.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb8495ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb844060330 0x7fb84409dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43112/0 (socket says 172.21.15.187:43112) 2020-11-20T12:07:12.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb8495ec700 1 -- 172.21.15.187:0/3030085987 learned_addr learned my addr 172.21.15.187:0/3030085987 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:12.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb848deb700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb84409e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:12.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb849ded700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8440b8000 0x7fb844148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:12.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb8495ec700 1 -- 172.21.15.187:0/3030085987 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 msgr2=0x7fb84409e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:12.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb8495ec700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb84409e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:12.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb8495ec700 1 -- 172.21.15.187:0/3030085987 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8440b8000 msgr2=0x7fb844148cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:12.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb8495ec700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8440b8000 0x7fb844148cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:12.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.736+0000 7fb8495ec700 1 -- 172.21.15.187:0/3030085987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84415b940 con 0x7fb844060330 2020-11-20T12:07:12.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.737+0000 7fb8495ec700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb844060330 0x7fb84409dce0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fb84000c720 tx=0x7fb840007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:12.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.737+0000 7fb8317fa700 1 -- 172.21.15.187:0/3030085987 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840007320 con 0x7fb844060330 2020-11-20T12:07:12.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.737+0000 7fb8317fa700 1 -- 172.21.15.187:0/3030085987 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb840007480 con 0x7fb844060330 2020-11-20T12:07:12.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.737+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84415bbf0 con 0x7fb844060330 2020-11-20T12:07:12.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.737+0000 7fb8317fa700 1 -- 172.21.15.187:0/3030085987 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840005800 con 0x7fb844060330 2020-11-20T12:07:12.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.737+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8441474e0 con 0x7fb844060330 2020-11-20T12:07:12.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.738+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84415c6e0 con 0x7fb844060330 2020-11-20T12:07:12.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.739+0000 7fb8317fa700 1 -- 172.21.15.187:0/3030085987 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb840010030 con 0x7fb844060330 2020-11-20T12:07:12.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.739+0000 7fb8317fa700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb81c056d50 0x7fb81c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:12.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.739+0000 7fb8317fa700 1 -- 172.21.15.187:0/3030085987 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb84006fd50 con 0x7fb844060330 2020-11-20T12:07:12.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.740+0000 7fb848deb700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb81c056d50 0x7fb81c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:12.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.740+0000 7fb848deb700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb81c056d50 0x7fb81c059210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fb83400b9a0 tx=0x7fb834015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:12.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.743+0000 7fb8317fa700 1 -- 172.21.15.187:0/3030085987 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb8400451c0 con 0x7fb844060330 2020-11-20T12:07:12.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.989+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb84415c0b0 con 0x7fb844060330 2020-11-20T12:07:12.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.989+0000 7fb8317fa700 1 -- 172.21.15.187:0/3030085987 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fb8400487e0 con 0x7fb844060330 2020-11-20T12:07:12.994 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:12.995 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:12.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.991+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb81c056d50 msgr2=0x7fb81c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:12.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.991+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb81c056d50 0x7fb81c059210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fb83400b9a0 tx=0x7fb834015000).stop 2020-11-20T12:07:12.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.991+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb844060330 msgr2=0x7fb84409dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:12.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.991+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb844060330 0x7fb84409dce0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fb84000c720 tx=0x7fb840007ad0).stop 2020-11-20T12:07:12.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.992+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 shutdown_connections 2020-11-20T12:07:12.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.992+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb81c056d50 0x7fb81c059210 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:12.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.992+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8440b8000 0x7fb844148cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:13.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.992+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb844060330 0x7fb84409dce0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:13.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.992+0000 7fb84a5ee700 1 --2- 172.21.15.187:0/3030085987 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb844060710 0x7fb84409e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:13.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.992+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 >> 172.21.15.187:0/3030085987 conn(0x7fb8440bbc50 msgr2=0x7fb844148160 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:13.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.992+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 shutdown_connections 2020-11-20T12:07:13.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:12.992+0000 7fb84a5ee700 1 -- 172.21.15.187:0/3030085987 wait complete. 2020-11-20T12:07:13.002 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:13.021 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_bal_fragment_size_max 50 2020-11-20T12:07:13.287 INFO:teuthology.orchestra.run.smithi187.stdout:{"success":"mds_bal_fragment_size_max = '50' (not observed, change may require restart) "} 2020-11-20T12:07:13.288 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_bal_fragment_size_max = '50' (not observed, change may require restart) " } 2020-11-20T12:07:13.288 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:07:13.289 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:07:13.289 INFO:teuthology.orchestra.run.smithi187:> path = os.path.join("/home/ubuntu/cephtest/mnt.0", "subdir") 2020-11-20T12:07:13.289 INFO:teuthology.orchestra.run.smithi187:> os.mkdir(path) 2020-11-20T12:07:13.290 INFO:teuthology.orchestra.run.smithi187:> for n in range(0, 51): 2020-11-20T12:07:13.291 INFO:teuthology.orchestra.run.smithi187:> with open(os.path.join(path, "%s" % n), '"'"'w'"'"') as f: 2020-11-20T12:07:13.291 INFO:teuthology.orchestra.run.smithi187:> f.write(str(n)) 2020-11-20T12:07:13.292 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:07:13.584 INFO:teuthology.orchestra.run.smithi187.stderr:Traceback (most recent call last): 2020-11-20T12:07:13.585 INFO:teuthology.orchestra.run.smithi187.stderr: File "", line 6, in 2020-11-20T12:07:13.585 INFO:teuthology.orchestra.run.smithi187.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/50' 2020-11-20T12:07:13.643 INFO:teuthology.orchestra.run.smithi187.stderr:daemon-helper: command failed with exit status 1 2020-11-20T12:07:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:07:13.658 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:07:13.658 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:07:13.659 INFO:teuthology.orchestra.run.smithi187:> path = os.path.join("/home/ubuntu/cephtest/mnt.0", "subdir2") 2020-11-20T12:07:13.659 INFO:teuthology.orchestra.run.smithi187:> os.mkdir(path) 2020-11-20T12:07:13.659 INFO:teuthology.orchestra.run.smithi187:> for n in range(0, 50): 2020-11-20T12:07:13.660 INFO:teuthology.orchestra.run.smithi187:> with open(os.path.join(path, "%s" % n), '"'"'w'"'"') as f: 2020-11-20T12:07:13.660 INFO:teuthology.orchestra.run.smithi187:> f.write(str(n)) 2020-11-20T12:07:13.660 INFO:teuthology.orchestra.run.smithi187:> dfd = os.open(path, os.O_DIRECTORY) 2020-11-20T12:07:13.661 INFO:teuthology.orchestra.run.smithi187:> os.fsync(dfd) 2020-11-20T12:07:13.661 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:07:18.094 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:18.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.339+0000 7f28ed608700 1 Processor -- start 2020-11-20T12:07:18.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.339+0000 7f28ed608700 1 -- start start 2020-11-20T12:07:18.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.341+0000 7f28ed608700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e8065190 0x7f28e8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:18.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.341+0000 7f28ed608700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e80b13e0 con 0x7f28e809ca90 2020-11-20T12:07:18.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.341+0000 7f28ed608700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e8062c10 con 0x7f28e8065f90 2020-11-20T12:07:18.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.341+0000 7f28ed608700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e814ab30 con 0x7f28e8065190 2020-11-20T12:07:18.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.341+0000 7f28df7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f28e8065f90 0x7f28e8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39728/0 (socket says 172.21.15.187:39728) 2020-11-20T12:07:18.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.341+0000 7f28df7fe700 1 -- 172.21.15.187:0/3001105709 learned_addr learned my addr 172.21.15.187:0/3001105709 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:18.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e7fff700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e8065190 0x7f28e8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:18.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655141583 0 0) 0x7f28e8062c10 con 0x7f28e8065f90 2020-11-20T12:07:18.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c8003240 con 0x7f28e8065f90 2020-11-20T12:07:18.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3549087859 0 0) 0x7f28c8003240 con 0x7f28e8065f90 2020-11-20T12:07:18.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e8065190 msgr2=0x7f28e8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:18.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e8065190 0x7f28e8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:18.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 >> v1:172.21.15.5:6789/0 conn(0x7f28e809ca90 legacy=0x7f28e809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:18.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e80b1640 con 0x7f28e8065f90 2020-11-20T12:07:18.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f28d00014e0 con 0x7f28e8065f90 2020-11-20T12:07:18.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 >> v1:172.21.15.187:6789/0 conn(0x7f28e8065f90 legacy=0x7f28e8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:18.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e8065190 0x7f28c8003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:18.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28c8003c40 0x7f28c80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:18.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28c80049d0 0x7f28c8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:18.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e814ab30 con 0x7f28c8003c40 2020-11-20T12:07:18.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80041d0 con 0x7f28c80049d0 2020-11-20T12:07:18.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.342+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c800b730 con 0x7f28e8065190 2020-11-20T12:07:18.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.343+0000 7f28ece07700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28c80049d0 0x7f28c8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:18.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.343+0000 7f28df7fe700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28c8003c40 0x7f28c80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:18.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.343+0000 7f28ece07700 1 -- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e8065190 msgr2=0x7f28c8003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:18.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.343+0000 7f28ece07700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e8065190 0x7f28c8003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:18.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.343+0000 7f28ece07700 1 -- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28c8003c40 msgr2=0x7f28c80044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:18.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.343+0000 7f28ece07700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28c8003c40 0x7f28c80044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:18.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.343+0000 7f28ece07700 1 -- 172.21.15.187:0/3001105709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c8003240 con 0x7f28c80049d0 2020-11-20T12:07:18.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.344+0000 7f28ece07700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28c80049d0 0x7f28c8004e50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f28e0001e70 tx=0x7f28e0007f10).ready entity=mon.1 client_cookie=fd4e215733a93fc2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:18.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.344+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0001540 con 0x7f28c80049d0 2020-11-20T12:07:18.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.344+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e0003030 con 0x7f28c80049d0 2020-11-20T12:07:18.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.344+0000 7f28e77fe700 1 -- 172.21.15.187:0/3001105709 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e000cd90 con 0x7f28c80049d0 2020-11-20T12:07:18.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.344+0000 7f28ed608700 1 -- 172.21.15.187:0/3001105709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28c80049d0 msgr2=0x7f28c8004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:18.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.344+0000 7f28ed608700 1 --2- 172.21.15.187:0/3001105709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28c80049d0 0x7f28c8004e50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f28e0001e70 tx=0x7f28e0007f10).stop 2020-11-20T12:07:18.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.345+0000 7f28ece07700 1 -- 172.21.15.187:0/3001105709 reap_dead start 2020-11-20T12:07:18.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.345+0000 7f28ed608700 1 -- 172.21.15.187:0/3001105709 shutdown_connections 2020-11-20T12:07:18.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.345+0000 7f28ed608700 1 -- 172.21.15.187:0/3001105709 >> 172.21.15.187:0/3001105709 conn(0x7f28e80bab30 msgr2=0x7f28e80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:18.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.345+0000 7f28ed608700 1 -- 172.21.15.187:0/3001105709 shutdown_connections 2020-11-20T12:07:18.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.345+0000 7f28ed608700 1 -- 172.21.15.187:0/3001105709 wait complete. 2020-11-20T12:07:18.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.346+0000 7f28ed608700 1 Processor -- start 2020-11-20T12:07:18.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.347+0000 7f28ed608700 1 -- start start 2020-11-20T12:07:18.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.347+0000 7f28ed608700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28e814bf00 0x7f28e814d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:18.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.347+0000 7f28ed608700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e814c2e0 0x7f28e8144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:18.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.347+0000 7f28ed608700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28e8145420 0x7f28e80b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:18.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.347+0000 7f28ed608700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e8145880 con 0x7f28e8145420 2020-11-20T12:07:18.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.347+0000 7f28ed608700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e814c840 con 0x7f28e814bf00 2020-11-20T12:07:18.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.347+0000 7f28ed608700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e814cae0 con 0x7f28e814c2e0 2020-11-20T12:07:18.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28e814bf00 0x7f28e814d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:18.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28e814bf00 0x7f28e814d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43134/0 (socket says 172.21.15.187:43134) 2020-11-20T12:07:18.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 -- 172.21.15.187:0/312364494 learned_addr learned my addr 172.21.15.187:0/312364494 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:18.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 -- 172.21.15.187:0/312364494 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e814c2e0 msgr2=0x7f28e8144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:18.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e814c2e0 0x7f28e8144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:18.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 -- 172.21.15.187:0/312364494 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28e8145420 msgr2=0x7f28e80b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:18.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28e8145420 0x7f28e80b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:18.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 -- 172.21.15.187:0/312364494 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e814cd80 con 0x7f28e814bf00 2020-11-20T12:07:18.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.348+0000 7f28e7fff700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28e814bf00 0x7f28e814d730 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f28d800bfd0 tx=0x7f28d8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:18.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.349+0000 7f28e4ff9700 1 -- 172.21.15.187:0/312364494 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d8017070 con 0x7f28e814bf00 2020-11-20T12:07:18.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.349+0000 7f28e4ff9700 1 -- 172.21.15.187:0/312364494 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d800ded0 con 0x7f28e814bf00 2020-11-20T12:07:18.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.349+0000 7f28e4ff9700 1 -- 172.21.15.187:0/312364494 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d80099c0 con 0x7f28e814bf00 2020-11-20T12:07:18.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.349+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e814d260 con 0x7f28e814bf00 2020-11-20T12:07:18.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.349+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e80b2890 con 0x7f28e814bf00 2020-11-20T12:07:18.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.350+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e8061a20 con 0x7f28e814bf00 2020-11-20T12:07:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.350+0000 7f28e4ff9700 1 -- 172.21.15.187:0/312364494 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f28d8013480 con 0x7f28e814bf00 2020-11-20T12:07:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.351+0000 7f28e4ff9700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f28c4056d20 0x7f28c40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.351+0000 7f28e4ff9700 1 -- 172.21.15.187:0/312364494 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f28d806fff0 con 0x7f28e814bf00 2020-11-20T12:07:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.353+0000 7f28df7fe700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f28c4056d20 0x7f28c40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:18.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.353+0000 7f28df7fe700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f28c4056d20 0x7f28c40591e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f28d0001010 tx=0x7f28e80bb650).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:18.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.356+0000 7f28e4ff9700 1 -- 172.21.15.187:0/312364494 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f28d80452d0 con 0x7f28e814bf00 2020-11-20T12:07:18.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.605+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28e80629f0 con 0x7f28e814bf00 2020-11-20T12:07:18.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.606+0000 7f28e4ff9700 1 -- 172.21.15.187:0/312364494 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f28d8009db0 con 0x7f28e814bf00 2020-11-20T12:07:18.611 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:18.611 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:18.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f28c4056d20 msgr2=0x7f28c40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:18.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f28c4056d20 0x7f28c40591e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f28d0001010 tx=0x7f28e80bb650).stop 2020-11-20T12:07:18.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28e814bf00 msgr2=0x7f28e814d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:18.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28e814bf00 0x7f28e814d730 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f28d800bfd0 tx=0x7f28d8008040).stop 2020-11-20T12:07:18.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 shutdown_connections 2020-11-20T12:07:18.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f28c4056d20 0x7f28c40591e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:18.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28e8145420 0x7f28e80b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:18.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28e814bf00 0x7f28e814d730 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:18.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 --2- 172.21.15.187:0/312364494 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28e814c2e0 0x7f28e8144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:18.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 >> 172.21.15.187:0/312364494 conn(0x7f28e80bab30 msgr2=0x7f28e8145b90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:18.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 shutdown_connections 2020-11-20T12:07:18.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:18.609+0000 7f28ed608700 1 -- 172.21.15.187:0/312364494 wait complete. 2020-11-20T12:07:18.616 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:18.635 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir2 0/0 1 2020-11-20T12:07:18.649 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:07:18.657 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:07:18.885 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2020-11-20T12:07:18.885 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:07:18.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:07:18.886 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:07:19.172 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:07:19.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:19.173 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:07:19.224 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:07:19.593 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:07:19.594 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:07:19.594 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:07:19.595 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:07:19.595 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:07:19.595 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:07:19.596 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:07:19.596 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:07:19.596 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:07:19.597 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:07:19.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:19.598 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:07:19.648 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:07:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:07:19.649 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:07:19.649 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:19.649 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:07:19.649 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:19.650 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:07:19.650 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:19.650 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:07:19.650 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:19.650 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:07:19.651 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:19.651 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:07:19.651 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:19.651 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:07:19.651 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42599sec preferred_lft 42599sec 2020-11-20T12:07:19.652 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:07:19.652 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:19.652 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:07:19.652 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:07:19.653 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:07:19.653 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:19.653 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:07:19.653 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:19.653 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:07:19.654 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:07:19.654 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:07:19.654 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:19.654 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:07:19.655 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:07:19.655 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:07:19.655 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:07:19.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:19.657 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:07:19.750 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:07:19.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:19.756 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:07:19.811 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:07:19.811 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:07:19.811 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:07:19.812 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:07:19.812 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:07:19.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:19.813 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:07:19.814 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:07:19.814 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:07:19.815 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:07:19.815 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:07:19.815 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:07:20.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:20.073 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:07:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:07:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:07:20.127 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:07:20.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:20.128 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:07:20.227 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:07:20.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.467+0000 7f81429a1700 1 Processor -- start 2020-11-20T12:07:20.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.467+0000 7f81429a1700 1 -- start start 2020-11-20T12:07:20.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.469+0000 7f81429a1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f813c0bf800 0x7f813c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.469+0000 7f81429a1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f813c0bfbe0 0x7f813c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.469+0000 7f81429a1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 0x7f813c1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.469+0000 7f81429a1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c14d110 con 0x7f813c0bf800 2020-11-20T12:07:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.469+0000 7f81429a1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c148aa0 con 0x7f813c0bfbe0 2020-11-20T12:07:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.469+0000 7f81429a1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c14cbd0 con 0x7f813c09f560 2020-11-20T12:07:20.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.469+0000 7f814119e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f813c0bfbe0 0x7f813c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814119e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f813c0bfbe0 0x7f813c1480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43158/0 (socket says 172.21.15.187:43158) 2020-11-20T12:07:20.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814119e700 1 -- 172.21.15.187:0/2109715622 learned_addr learned my addr 172.21.15.187:0/2109715622 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:20.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f81421a0700 1 --2- 172.21.15.187:0/2109715622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 0x7f813c1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814199f700 1 --2- 172.21.15.187:0/2109715622 >> v2:172.21.15.5:3300/0 conn(0x7f813c0bf800 0x7f813c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814119e700 1 -- 172.21.15.187:0/2109715622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 msgr2=0x7f813c1485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:20.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814119e700 1 --2- 172.21.15.187:0/2109715622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 0x7f813c1485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814119e700 1 -- 172.21.15.187:0/2109715622 >> v2:172.21.15.5:3300/0 conn(0x7f813c0bf800 msgr2=0x7f813c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:20.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814119e700 1 --2- 172.21.15.187:0/2109715622 >> v2:172.21.15.5:3300/0 conn(0x7f813c0bf800 0x7f813c14c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814119e700 1 -- 172.21.15.187:0/2109715622 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c14ce70 con 0x7f813c0bfbe0 2020-11-20T12:07:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f81421a0700 1 --2- 172.21.15.187:0/2109715622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 0x7f813c1485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.470+0000 7f814119e700 1 --2- 172.21.15.187:0/2109715622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bfbe0 0x7f813c1480c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f813c0687b0 tx=0x7f812c006eb0).ready entity=mon.1 client_cookie=52ed0a1244adbc82 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f814099d700 1 -- 172.21.15.187:0/2109715622 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c009960 con 0x7f813c0bfbe0 2020-11-20T12:07:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f814099d700 1 -- 172.21.15.187:0/2109715622 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c009ac0 con 0x7f813c0bfbe0 2020-11-20T12:07:20.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f814099d700 1 -- 172.21.15.187:0/2109715622 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c009e40 con 0x7f813c0bfbe0 2020-11-20T12:07:20.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f81429a1700 1 -- 172.21.15.187:0/2109715622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bfbe0 msgr2=0x7f813c1480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f81429a1700 1 --2- 172.21.15.187:0/2109715622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bfbe0 0x7f813c1480c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f813c0687b0 tx=0x7f812c006eb0).stop 2020-11-20T12:07:20.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f81429a1700 1 -- 172.21.15.187:0/2109715622 shutdown_connections 2020-11-20T12:07:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f81429a1700 1 --2- 172.21.15.187:0/2109715622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 0x7f813c1485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f81429a1700 1 --2- 172.21.15.187:0/2109715622 >> v2:172.21.15.5:3300/0 conn(0x7f813c0bf800 0x7f813c14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f81429a1700 1 --2- 172.21.15.187:0/2109715622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bfbe0 0x7f813c1480c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.471+0000 7f81429a1700 1 -- 172.21.15.187:0/2109715622 >> 172.21.15.187:0/2109715622 conn(0x7f813c068370 msgr2=0x7f813c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.472+0000 7f81429a1700 1 -- 172.21.15.187:0/2109715622 shutdown_connections 2020-11-20T12:07:20.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.472+0000 7f81429a1700 1 -- 172.21.15.187:0/2109715622 wait complete. 2020-11-20T12:07:20.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.473+0000 7f81429a1700 1 Processor -- start 2020-11-20T12:07:20.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.473+0000 7f81429a1700 1 -- start start 2020-11-20T12:07:20.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.474+0000 7f81429a1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 0x7f813c141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.474+0000 7f81429a1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bf800 0x7f813c0bb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.474+0000 7f81429a1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f813c0b4980 0x7f813c1584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.474+0000 7f81429a1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1460a0 con 0x7f813c0b4980 2020-11-20T12:07:20.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.474+0000 7f81429a1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0b65b0 con 0x7f813c0bf800 2020-11-20T12:07:20.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.474+0000 7f81429a1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0b6850 con 0x7f813c09f560 2020-11-20T12:07:20.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bf800 0x7f813c0bb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bf800 0x7f813c0bb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43166/0 (socket says 172.21.15.187:43166) 2020-11-20T12:07:20.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 -- 172.21.15.187:0/3795834118 learned_addr learned my addr 172.21.15.187:0/3795834118 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:20.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 -- 172.21.15.187:0/3795834118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 msgr2=0x7f813c141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 0x7f813c141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:20.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 -- 172.21.15.187:0/3795834118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f813c0b4980 msgr2=0x7f813c1584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f813c0b4980 0x7f813c1584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:20.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 -- 172.21.15.187:0/3795834118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c145d20 con 0x7f813c0bf800 2020-11-20T12:07:20.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.475+0000 7f814119e700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bf800 0x7f813c0bb9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f813c148be0 tx=0x7f812c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:20.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.476+0000 7f8129ffb700 1 -- 172.21.15.187:0/3795834118 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c01d6a0 con 0x7f813c0bf800 2020-11-20T12:07:20.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.476+0000 7f8129ffb700 1 -- 172.21.15.187:0/3795834118 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c01d820 con 0x7f813c0bf800 2020-11-20T12:07:20.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.476+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c0bbeb0 con 0x7f813c0bf800 2020-11-20T12:07:20.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.476+0000 7f8129ffb700 1 -- 172.21.15.187:0/3795834118 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c01dbe0 con 0x7f813c0bf800 2020-11-20T12:07:20.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.476+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c0bc1b0 con 0x7f813c0bf800 2020-11-20T12:07:20.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.477+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c080050 con 0x7f813c0bf800 2020-11-20T12:07:20.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.477+0000 7f8129ffb700 1 -- 172.21.15.187:0/3795834118 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f812c025030 con 0x7f813c0bf800 2020-11-20T12:07:20.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.478+0000 7f8129ffb700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8114056d20 0x7f81140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.478+0000 7f8129ffb700 1 -- 172.21.15.187:0/3795834118 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f812c071850 con 0x7f813c0bf800 2020-11-20T12:07:20.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.478+0000 7f814199f700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8114056d20 0x7f81140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.479+0000 7f814199f700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8114056d20 0x7f81140591e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f8138006fd0 tx=0x7f8138008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:20.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.483+0000 7f8129ffb700 1 -- 172.21.15.187:0/3795834118 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f812c046d50 con 0x7f813c0bf800 2020-11-20T12:07:20.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.669+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f813c145f10 con 0x7f813c0bf800 2020-11-20T12:07:20.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.672+0000 7f8129ffb700 1 -- 172.21.15.187:0/3795834118 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f812c0311a0 con 0x7f813c0bf800 2020-11-20T12:07:20.677 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:20.677 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":42,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:07:05.571797+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:07:01.490118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2020-11-20T12:07:02.131720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:07:20.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.674+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8114056d20 msgr2=0x7f81140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.674+0000 7f81429a1700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8114056d20 0x7f81140591e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f8138006fd0 tx=0x7f8138008040).stop 2020-11-20T12:07:20.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.674+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bf800 msgr2=0x7f813c0bb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.674+0000 7f81429a1700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bf800 0x7f813c0bb9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f813c148be0 tx=0x7f812c015000).stop 2020-11-20T12:07:20.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.675+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 shutdown_connections 2020-11-20T12:07:20.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.675+0000 7f81429a1700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8114056d20 0x7f81140591e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.675+0000 7f81429a1700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f813c0b4980 0x7f813c1584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:20.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.675+0000 7f81429a1700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f813c09f560 0x7f813c141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:20.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.675+0000 7f81429a1700 1 --2- 172.21.15.187:0/3795834118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f813c0bf800 0x7f813c0bb9c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.675+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 >> 172.21.15.187:0/3795834118 conn(0x7f813c068370 msgr2=0x7f813c1471e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:20.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.675+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 shutdown_connections 2020-11-20T12:07:20.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.675+0000 7f81429a1700 1 -- 172.21.15.187:0/3795834118 wait complete. 2020-11-20T12:07:20.701 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:07:01.490118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:07:02.131720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:07:20.702 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:07:20.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.960+0000 7f65c7fff700 1 Processor -- start 2020-11-20T12:07:20.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.960+0000 7f65c7fff700 1 -- start start 2020-11-20T12:07:20.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.961+0000 7f65c7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f65c80b1380 0x7f65c8150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.961+0000 7f65c7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c814c260 0x7f65c814c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.962+0000 7f65c7fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c80658a0 con 0x7f65c80b1760 2020-11-20T12:07:20.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.962+0000 7f65c7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c8150390 con 0x7f65c80b1380 2020-11-20T12:07:20.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.962+0000 7f65c7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c8150630 con 0x7f65c814c260 2020-11-20T12:07:20.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.962+0000 7f65c77fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c814c260 0x7f65c814c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.963+0000 7f65c77fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c814c260 0x7f65c814c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:44062/0 (socket says 172.21.15.187:44062) 2020-11-20T12:07:20.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.963+0000 7f65c77fe700 1 -- 172.21.15.187:0/463556889 learned_addr learned my addr 172.21.15.187:0/463556889 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:20.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.963+0000 7f65be7fc700 1 --1- 172.21.15.187:0/463556889 >> v1:172.21.15.5:6789/0 conn(0x7f65c80b1760 0x7f65c8065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33286/0 (socket says 172.21.15.187:33286) 2020-11-20T12:07:20.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.963+0000 7f65c6ffd700 1 --2- 172.21.15.187:0/463556889 >> v2:172.21.15.5:3300/0 conn(0x7f65c80b1380 0x7f65c8150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.964+0000 7f65c77fe700 1 -- 172.21.15.187:0/463556889 >> v2:172.21.15.5:3300/0 conn(0x7f65c80b1380 msgr2=0x7f65c8150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:20.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.964+0000 7f65c77fe700 1 --2- 172.21.15.187:0/463556889 >> v2:172.21.15.5:3300/0 conn(0x7f65c80b1380 0x7f65c8150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.964+0000 7f65c77fe700 1 -- 172.21.15.187:0/463556889 >> v1:172.21.15.5:6789/0 conn(0x7f65c80b1760 legacy=0x7f65c8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.964+0000 7f65c77fe700 1 -- 172.21.15.187:0/463556889 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c8068510 con 0x7f65c814c260 2020-11-20T12:07:20.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.964+0000 7f65c77fe700 1 --2- 172.21.15.187:0/463556889 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c814c260 0x7f65c814c6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f65c0006fd0 tx=0x7f65c0006eb0).ready entity=mon.2 client_cookie=a1d15ffbeaa6245f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:20.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.964+0000 7f65c67fc700 1 -- 172.21.15.187:0/463556889 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0009810 con 0x7f65c814c260 2020-11-20T12:07:20.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c67fc700 1 -- 172.21.15.187:0/463556889 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c0009970 con 0x7f65c814c260 2020-11-20T12:07:20.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c67fc700 1 -- 172.21.15.187:0/463556889 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0009cf0 con 0x7f65c814c260 2020-11-20T12:07:20.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c7fff700 1 -- 172.21.15.187:0/463556889 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c814c260 msgr2=0x7f65c814c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c7fff700 1 --2- 172.21.15.187:0/463556889 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c814c260 0x7f65c814c6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f65c0006fd0 tx=0x7f65c0006eb0).stop 2020-11-20T12:07:20.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c7fff700 1 -- 172.21.15.187:0/463556889 shutdown_connections 2020-11-20T12:07:20.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c7fff700 1 --2- 172.21.15.187:0/463556889 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c814c260 0x7f65c814c6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c7fff700 1 --2- 172.21.15.187:0/463556889 >> v2:172.21.15.5:3300/0 conn(0x7f65c80b1380 0x7f65c8150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c7fff700 1 -- 172.21.15.187:0/463556889 >> 172.21.15.187:0/463556889 conn(0x7f65c80bab30 msgr2=0x7f65c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:20.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c7fff700 1 -- 172.21.15.187:0/463556889 shutdown_connections 2020-11-20T12:07:20.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.965+0000 7f65c7fff700 1 -- 172.21.15.187:0/463556889 wait complete. 2020-11-20T12:07:20.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.967+0000 7f65c7fff700 1 Processor -- start 2020-11-20T12:07:20.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.968+0000 7f65c7fff700 1 -- start start 2020-11-20T12:07:20.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65c7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c80b1380 0x7f65c80b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65c7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65c80b1760 0x7f65c809dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65c7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65c814c260 0x7f65c80b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65c7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b30b0 con 0x7f65c814c260 2020-11-20T12:07:20.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65c7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b26d0 con 0x7f65c80b1760 2020-11-20T12:07:20.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65c7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b2970 con 0x7f65c80b1380 2020-11-20T12:07:20.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65be7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65c80b1760 0x7f65c809dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65be7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65c80b1760 0x7f65c809dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43178/0 (socket says 172.21.15.187:43178) 2020-11-20T12:07:20.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65be7fc700 1 -- 172.21.15.187:0/1959619709 learned_addr learned my addr 172.21.15.187:0/1959619709 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:20.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65c77fe700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65c814c260 0x7f65c80b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65c6ffd700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c80b1380 0x7f65c80b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.969+0000 7f65be7fc700 1 -- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c80b1380 msgr2=0x7f65c80b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65be7fc700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c80b1380 0x7f65c80b6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65be7fc700 1 -- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65c814c260 msgr2=0x7f65c80b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:20.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65be7fc700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65c814c260 0x7f65c80b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:20.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65be7fc700 1 -- 172.21.15.187:0/1959619709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80b2c10 con 0x7f65c80b1760 2020-11-20T12:07:20.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65c77fe700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65c814c260 0x7f65c80b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:20.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65be7fc700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65c80b1760 0x7f65c809dae0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f65b000c740 tx=0x7f65b0007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:20.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65bf7fe700 1 -- 172.21.15.187:0/1959619709 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0007320 con 0x7f65c80b1760 2020-11-20T12:07:20.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65bf7fe700 1 -- 172.21.15.187:0/1959619709 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b0005530 con 0x7f65c80b1760 2020-11-20T12:07:20.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65bf7fe700 1 -- 172.21.15.187:0/1959619709 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00058b0 con 0x7f65c80b1760 2020-11-20T12:07:20.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c80b4690 con 0x7f65c80b1760 2020-11-20T12:07:20.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.970+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c815bd30 con 0x7f65c80b1760 2020-11-20T12:07:20.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.972+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c8061a20 con 0x7f65c80b1760 2020-11-20T12:07:20.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.972+0000 7f65bf7fe700 1 -- 172.21.15.187:0/1959619709 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f65b0010030 con 0x7f65c80b1760 2020-11-20T12:07:20.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.973+0000 7f65bf7fe700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65a4056da0 0x7f65a4059260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:20.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.973+0000 7f65bf7fe700 1 -- 172.21.15.187:0/1959619709 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f65b006fb40 con 0x7f65c80b1760 2020-11-20T12:07:20.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.973+0000 7f65c6ffd700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65a4056da0 0x7f65a4059260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:20.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.974+0000 7f65c6ffd700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65a4056da0 0x7f65a4059260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f65b8009440 tx=0x7f65b8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:20.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:20.977+0000 7f65bf7fe700 1 -- 172.21.15.187:0/1959619709 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f65b0044fb0 con 0x7f65c80b1760 2020-11-20T12:07:21.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.168+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f65c80a09c0 con 0x7f65c80b1760 2020-11-20T12:07:21.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.169+0000 7f65bf7fe700 1 -- 172.21.15.187:0/1959619709 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f65b00485d0 con 0x7f65c80b1760 2020-11-20T12:07:21.173 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:07:21.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65a4056da0 msgr2=0x7f65a4059260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65a4056da0 0x7f65a4059260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f65b8009440 tx=0x7f65b8008040).stop 2020-11-20T12:07:21.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65c80b1760 msgr2=0x7f65c809dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65c80b1760 0x7f65c809dae0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f65b000c740 tx=0x7f65b0007c00).stop 2020-11-20T12:07:21.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 shutdown_connections 2020-11-20T12:07:21.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65a4056da0 0x7f65a4059260 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65c814c260 0x7f65c80b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65c80b1380 0x7f65c80b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.171+0000 7f65c7fff700 1 --2- 172.21.15.187:0/1959619709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65c80b1760 0x7f65c809dae0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.172+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 >> 172.21.15.187:0/1959619709 conn(0x7f65c80bab30 msgr2=0x7f65c814b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:21.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.172+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 shutdown_connections 2020-11-20T12:07:21.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.172+0000 7f65c7fff700 1 -- 172.21.15.187:0/1959619709 wait complete. 2020-11-20T12:07:21.200 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:07:21.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.441+0000 7fd7a759e700 1 Processor -- start 2020-11-20T12:07:21.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.442+0000 7fd7a759e700 1 -- start start 2020-11-20T12:07:21.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.443+0000 7fd7a759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd7a8065190 0x7fd7a8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.443+0000 7fd7a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 0x7fd7a8143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.443+0000 7fd7a759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a80b13e0 con 0x7fd7a8143c70 2020-11-20T12:07:21.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.443+0000 7fd7a759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a814ab30 con 0x7fd7a8065190 2020-11-20T12:07:21.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.443+0000 7fd7a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8062c10 con 0x7fd7a8065f90 2020-11-20T12:07:21.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.443+0000 7fd7a6d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd7a8143c70 0x7fd7a809ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39790/0 (socket says 172.21.15.187:39790) 2020-11-20T12:07:21.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.443+0000 7fd7a6d9d700 1 -- 172.21.15.187:0/1636808876 learned_addr learned my addr 172.21.15.187:0/1636808876 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:21.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a659c700 1 --2- 172.21.15.187:0/1636808876 >> v2:172.21.15.5:3300/0 conn(0x7fd7a8065190 0x7fd7a8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 0x7fd7a8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158262391 0 0) 0x7fd7a80b13e0 con 0x7fd7a8143c70 2020-11-20T12:07:21.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788003240 con 0x7fd7a8143c70 2020-11-20T12:07:21.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 917368526 0 0) 0x7fd788003240 con 0x7fd7a8143c70 2020-11-20T12:07:21.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 msgr2=0x7fd7a8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:21.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 0x7fd7a8143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 >> v2:172.21.15.5:3300/0 conn(0x7fd7a8065190 msgr2=0x7fd7a8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:21.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 --2- 172.21.15.187:0/1636808876 >> v2:172.21.15.5:3300/0 conn(0x7fd7a8065190 0x7fd7a8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a80b1640 con 0x7fd7a8143c70 2020-11-20T12:07:21.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd7a0001860 con 0x7fd7a8143c70 2020-11-20T12:07:21.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 0x7fd7a8143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:07:21.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.444+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 >> v1:172.21.15.187:6789/0 conn(0x7fd7a8143c70 legacy=0x7fd7a809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd7a5d9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd788003730 0x7fd788003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd7a5d9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7880044e0 0x7fd788004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd7a5d9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 0x7fd788004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a814ab30 con 0x7fd788003730 2020-11-20T12:07:21.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8062c10 con 0x7fd7880044e0 2020-11-20T12:07:21.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd788004250 con 0x7fd7a8065f90 2020-11-20T12:07:21.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd7a659c700 1 --2- 172.21.15.187:0/1636808876 >> v2:172.21.15.5:3300/0 conn(0x7fd7a8065190 0x7fd7a8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:07:21.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7880044e0 0x7fd788004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd79dd9b700 1 -- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 msgr2=0x7fd788004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:21.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.445+0000 7fd7a6d9d700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 0x7fd788004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 0x7fd788004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd79dd9b700 1 -- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd788003730 msgr2=0x7fd788003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:21.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd788003730 0x7fd788003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:21.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd79dd9b700 1 -- 172.21.15.187:0/1636808876 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788003240 con 0x7fd7880044e0 2020-11-20T12:07:21.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd7a6d9d700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8065f90 0x7fd788004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:21.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7880044e0 0x7fd788004960 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7fd79000bb40 tx=0x7fd790008040).ready entity=mon.1 client_cookie=6c4ea5380d492187 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:21.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79000da20 con 0x7fd7880044e0 2020-11-20T12:07:21.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79000db80 con 0x7fd7880044e0 2020-11-20T12:07:21.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd7a5d9b700 1 -- 172.21.15.187:0/1636808876 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790012420 con 0x7fd7880044e0 2020-11-20T12:07:21.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd7a759e700 1 -- 172.21.15.187:0/1636808876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7880044e0 msgr2=0x7fd788004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.446+0000 7fd7a759e700 1 --2- 172.21.15.187:0/1636808876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7880044e0 0x7fd788004960 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7fd79000bb40 tx=0x7fd790008040).stop 2020-11-20T12:07:21.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.447+0000 7fd7a6d9d700 1 -- 172.21.15.187:0/1636808876 reap_dead start 2020-11-20T12:07:21.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.447+0000 7fd7a759e700 1 -- 172.21.15.187:0/1636808876 shutdown_connections 2020-11-20T12:07:21.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.447+0000 7fd7a759e700 1 -- 172.21.15.187:0/1636808876 >> 172.21.15.187:0/1636808876 conn(0x7fd7a80bab30 msgr2=0x7fd7a80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:21.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.447+0000 7fd7a759e700 1 -- 172.21.15.187:0/1636808876 shutdown_connections 2020-11-20T12:07:21.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.447+0000 7fd7a759e700 1 -- 172.21.15.187:0/1636808876 wait complete. 2020-11-20T12:07:21.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.448+0000 7fd7a759e700 1 Processor -- start 2020-11-20T12:07:21.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.448+0000 7fd7a759e700 1 -- start start 2020-11-20T12:07:21.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.449+0000 7fd7a759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7a809cc20 0x7fd7a80b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.449+0000 7fd7a759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7a809d000 0x7fd7a815a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.449+0000 7fd7a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8142c50 0x7fd7a8143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.449+0000 7fd7a759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a815a570 con 0x7fd7a809d000 2020-11-20T12:07:21.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.449+0000 7fd7a759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80b27c0 con 0x7fd7a809cc20 2020-11-20T12:07:21.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.449+0000 7fd7a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80b2a60 con 0x7fd7a8142c50 2020-11-20T12:07:21.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7a809cc20 0x7fd7a80b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7a809cc20 0x7fd7a80b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43198/0 (socket says 172.21.15.187:43198) 2020-11-20T12:07:21.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 -- 172.21.15.187:0/3391942977 learned_addr learned my addr 172.21.15.187:0/3391942977 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:21.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7a809d000 0x7fd7a815a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a6d9d700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8142c50 0x7fd7a8143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 -- 172.21.15.187:0/3391942977 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8142c50 msgr2=0x7fd7a8143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8142c50 0x7fd7a8143070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 -- 172.21.15.187:0/3391942977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7a809d000 msgr2=0x7fd7a815a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7a809d000 0x7fd7a815a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 -- 172.21.15.187:0/3391942977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a80b2d00 con 0x7fd7a809cc20 2020-11-20T12:07:21.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.450+0000 7fd7a659c700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7a809cc20 0x7fd7a80b31a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7fd79800d8e0 tx=0x7fd798008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:21.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.451+0000 7fd79effd700 1 -- 172.21.15.187:0/3391942977 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798009790 con 0x7fd7a809cc20 2020-11-20T12:07:21.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.451+0000 7fd79effd700 1 -- 172.21.15.187:0/3391942977 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7980098f0 con 0x7fd7a809cc20 2020-11-20T12:07:21.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.451+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a80b24b0 con 0x7fd7a809cc20 2020-11-20T12:07:21.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.451+0000 7fd79effd700 1 -- 172.21.15.187:0/3391942977 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798009cb0 con 0x7fd7a809cc20 2020-11-20T12:07:21.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.451+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a8148a30 con 0x7fd7a809cc20 2020-11-20T12:07:21.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.452+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a8061a20 con 0x7fd7a809cc20 2020-11-20T12:07:21.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.453+0000 7fd79effd700 1 -- 172.21.15.187:0/3391942977 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd798019020 con 0x7fd7a809cc20 2020-11-20T12:07:21.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.453+0000 7fd79effd700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd784056d20 0x7fd7840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.453+0000 7fd79effd700 1 -- 172.21.15.187:0/3391942977 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd79806fc60 con 0x7fd7a809cc20 2020-11-20T12:07:21.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.454+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd784056d20 0x7fd7840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.454+0000 7fd79dd9b700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd784056d20 0x7fd7840591e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fd7a80b6390 tx=0x7fd790008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:21.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.457+0000 7fd79effd700 1 -- 172.21.15.187:0/3391942977 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd7980450b0 con 0x7fd7a809cc20 2020-11-20T12:07:21.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.642+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd7a80afc60 con 0x7fd7a809cc20 2020-11-20T12:07:21.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.642+0000 7fd79effd700 1 -- 172.21.15.187:0/3391942977 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v42) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd7980486d0 con 0x7fd7a809cc20 2020-11-20T12:07:21.647 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:07:21.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd784056d20 msgr2=0x7fd7840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd784056d20 0x7fd7840591e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fd7a80b6390 tx=0x7fd790008040).stop 2020-11-20T12:07:21.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7a809cc20 msgr2=0x7fd7a80b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7a809cc20 0x7fd7a80b31a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7fd79800d8e0 tx=0x7fd798008040).stop 2020-11-20T12:07:21.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 shutdown_connections 2020-11-20T12:07:21.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd784056d20 0x7fd7840591e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7a8142c50 0x7fd7a8143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7a809cc20 0x7fd7a80b31a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 --2- 172.21.15.187:0/3391942977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7a809d000 0x7fd7a815a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.646+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 >> 172.21.15.187:0/3391942977 conn(0x7fd7a80bab30 msgr2=0x7fd7a80b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:21.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.647+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 shutdown_connections 2020-11-20T12:07:21.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.647+0000 7fd7a759e700 1 -- 172.21.15.187:0/3391942977 wait complete. 2020-11-20T12:07:21.672 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:07:21.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.922+0000 7f191f2a4700 1 Processor -- start 2020-11-20T12:07:21.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.922+0000 7f191f2a4700 1 -- start start 2020-11-20T12:07:21.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.923+0000 7f191f2a4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f19180b6650 0x7f19180afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.923+0000 7f191f2a4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1918066f70 0x7f19180aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.923+0000 7f191f2a4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19180b1300 con 0x7f19181484d0 2020-11-20T12:07:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.923+0000 7f191f2a4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1918067730 con 0x7f1918066f70 2020-11-20T12:07:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.923+0000 7f191f2a4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19180b15a0 con 0x7f19180b6650 2020-11-20T12:07:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.924+0000 7f191daa1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f19180b6650 0x7f19180afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.924+0000 7f191e2a2700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f19181484d0 0x7f1918061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39810/0 (socket says 172.21.15.187:39810) 2020-11-20T12:07:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.924+0000 7f191e2a2700 1 -- 172.21.15.187:0/3023147584 learned_addr learned my addr 172.21.15.187:0/3023147584 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.924+0000 7f191eaa3700 1 --2- 172.21.15.187:0/3023147584 >> v2:172.21.15.5:3300/0 conn(0x7f1918066f70 0x7f19180aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191daa1700 1 -- 172.21.15.187:0/3023147584 >> v2:172.21.15.5:3300/0 conn(0x7f1918066f70 msgr2=0x7f19180aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191daa1700 1 --2- 172.21.15.187:0/3023147584 >> v2:172.21.15.5:3300/0 conn(0x7f1918066f70 0x7f19180aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191daa1700 1 -- 172.21.15.187:0/3023147584 >> v1:172.21.15.187:6789/0 conn(0x7f19181484d0 legacy=0x7f1918061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191d2a0700 1 -- 172.21.15.187:0/3023147584 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154875263 0 0) 0x7f19180b1300 con 0x7f19181484d0 2020-11-20T12:07:21.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191daa1700 1 -- 172.21.15.187:0/3023147584 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19180b1840 con 0x7f19180b6650 2020-11-20T12:07:21.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191daa1700 1 --2- 172.21.15.187:0/3023147584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19180b6650 0x7f19180afa90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f1918068600 tx=0x7f19140091d0).ready entity=mon.1 client_cookie=de1d6887c10143a3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:21.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191d2a0700 1 -- 172.21.15.187:0/3023147584 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191400bbc0 con 0x7f19180b6650 2020-11-20T12:07:21.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191d2a0700 1 -- 172.21.15.187:0/3023147584 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191400bd20 con 0x7f19180b6650 2020-11-20T12:07:21.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.925+0000 7f191d2a0700 1 -- 172.21.15.187:0/3023147584 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1914011610 con 0x7f19180b6650 2020-11-20T12:07:21.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.926+0000 7f191f2a4700 1 -- 172.21.15.187:0/3023147584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19180b6650 msgr2=0x7f19180afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.926+0000 7f191f2a4700 1 --2- 172.21.15.187:0/3023147584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19180b6650 0x7f19180afa90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f1918068600 tx=0x7f19140091d0).stop 2020-11-20T12:07:21.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.926+0000 7f191f2a4700 1 -- 172.21.15.187:0/3023147584 shutdown_connections 2020-11-20T12:07:21.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.926+0000 7f191f2a4700 1 --2- 172.21.15.187:0/3023147584 >> v2:172.21.15.5:3300/0 conn(0x7f1918066f70 0x7f19180aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.926+0000 7f191f2a4700 1 --2- 172.21.15.187:0/3023147584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19180b6650 0x7f19180afa90 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.926+0000 7f191f2a4700 1 -- 172.21.15.187:0/3023147584 >> 172.21.15.187:0/3023147584 conn(0x7f19180bab30 msgr2=0x7f19180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:21.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.926+0000 7f191f2a4700 1 -- 172.21.15.187:0/3023147584 shutdown_connections 2020-11-20T12:07:21.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.926+0000 7f191f2a4700 1 -- 172.21.15.187:0/3023147584 wait complete. 2020-11-20T12:07:21.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.928+0000 7f191f2a4700 1 Processor -- start 2020-11-20T12:07:21.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.928+0000 7f191f2a4700 1 -- start start 2020-11-20T12:07:21.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191f2a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1918066f70 0x7f191809a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191f2a4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19180b6650 0x7f191809aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191f2a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19181484d0 0x7f19180b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191f2a4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191809b0f0 con 0x7f19180b6650 2020-11-20T12:07:21.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191f2a4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19180b2bd0 con 0x7f19181484d0 2020-11-20T12:07:21.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191f2a4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19180b48e0 con 0x7f1918066f70 2020-11-20T12:07:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191eaa3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19181484d0 0x7f19180b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191eaa3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19181484d0 0x7f19180b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43214/0 (socket says 172.21.15.187:43214) 2020-11-20T12:07:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.929+0000 7f191eaa3700 1 -- 172.21.15.187:0/2018200270 learned_addr learned my addr 172.21.15.187:0/2018200270 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191e2a2700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1918066f70 0x7f191809a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191eaa3700 1 -- 172.21.15.187:0/2018200270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1918066f70 msgr2=0x7f191809a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191eaa3700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1918066f70 0x7f191809a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:21.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191eaa3700 1 -- 172.21.15.187:0/2018200270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19180b6650 msgr2=0x7f191809aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:21.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191eaa3700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19180b6650 0x7f191809aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:21.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191eaa3700 1 -- 172.21.15.187:0/2018200270 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19180b3070 con 0x7f19181484d0 2020-11-20T12:07:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191e2a2700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1918066f70 0x7f191809a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191eaa3700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19181484d0 0x7f19180b1a00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f191000d830 tx=0x7f1910008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f190a7fc700 1 -- 172.21.15.187:0/2018200270 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910009620 con 0x7f19181484d0 2020-11-20T12:07:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f190a7fc700 1 -- 172.21.15.187:0/2018200270 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1910009780 con 0x7f19181484d0 2020-11-20T12:07:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19180b3620 con 0x7f19181484d0 2020-11-20T12:07:21.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f190a7fc700 1 -- 172.21.15.187:0/2018200270 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910009b20 con 0x7f19181484d0 2020-11-20T12:07:21.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.930+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191809c3b0 con 0x7f19181484d0 2020-11-20T12:07:21.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.931+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1918062f10 con 0x7f19181484d0 2020-11-20T12:07:21.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.932+0000 7f190a7fc700 1 -- 172.21.15.187:0/2018200270 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1910013480 con 0x7f19181484d0 2020-11-20T12:07:21.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.933+0000 7f190a7fc700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18f0056d20 0x7f18f00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:21.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.933+0000 7f190a7fc700 1 -- 172.21.15.187:0/2018200270 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f191006ff00 con 0x7f19181484d0 2020-11-20T12:07:21.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.933+0000 7f191e2a2700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18f0056d20 0x7f18f00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:21.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.934+0000 7f191e2a2700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18f0056d20 0x7f18f00591e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f190c001500 tx=0x7f190c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:21.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:21.937+0000 7f190a7fc700 1 -- 172.21.15.187:0/2018200270 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f19100452f0 con 0x7f19181484d0 2020-11-20T12:07:22.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.127+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f19180b37b0 con 0x7f19181484d0 2020-11-20T12:07:22.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.128+0000 7f190a7fc700 1 -- 172.21.15.187:0/2018200270 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1910048910 con 0x7f19181484d0 2020-11-20T12:07:22.133 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:07:22.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.130+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18f0056d20 msgr2=0x7f18f00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.130+0000 7f191f2a4700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18f0056d20 0x7f18f00591e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f190c001500 tx=0x7f190c009040).stop 2020-11-20T12:07:22.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19181484d0 msgr2=0x7f19180b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19181484d0 0x7f19180b1a00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f191000d830 tx=0x7f1910008040).stop 2020-11-20T12:07:22.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 shutdown_connections 2020-11-20T12:07:22.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18f0056d20 0x7f18f00591e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19181484d0 0x7f19180b1a00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1918066f70 0x7f191809a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 --2- 172.21.15.187:0/2018200270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19180b6650 0x7f191809aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:22.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 >> 172.21.15.187:0/2018200270 conn(0x7f19180bab30 msgr2=0x7f191814ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:22.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 shutdown_connections 2020-11-20T12:07:22.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.131+0000 7f191f2a4700 1 -- 172.21.15.187:0/2018200270 wait complete. 2020-11-20T12:07:22.159 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:07:22.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.403+0000 7f7071a5c700 1 Processor -- start 2020-11-20T12:07:22.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.403+0000 7f7071a5c700 1 -- start start 2020-11-20T12:07:22.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.405+0000 7f7071a5c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f706c0bfbe0 0x7f706c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.405+0000 7f7071a5c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c14d110 con 0x7f706c09f560 2020-11-20T12:07:22.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.405+0000 7f7071a5c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c14cbd0 con 0x7f706c0bf800 2020-11-20T12:07:22.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.405+0000 7f7071a5c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706c14c9d0 con 0x7f706c0bfbe0 2020-11-20T12:07:22.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.405+0000 7f7070a5a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f706c0bf800 0x7f706c14c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39824/0 (socket says 172.21.15.187:39824) 2020-11-20T12:07:22.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.405+0000 7f7070a5a700 1 -- 172.21.15.187:0/4259823797 learned_addr learned my addr 172.21.15.187:0/4259823797 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:22.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706bfff700 1 --2- 172.21.15.187:0/4259823797 >> v2:172.21.15.5:3300/0 conn(0x7f706c0bfbe0 0x7f706c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286459899 0 0) 0x7f706c14cbd0 con 0x7f706c0bf800 2020-11-20T12:07:22.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7050003240 con 0x7f706c0bf800 2020-11-20T12:07:22.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1842867951 0 0) 0x7f7050003240 con 0x7f706c0bf800 2020-11-20T12:07:22.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 >> v2:172.21.15.5:3300/0 conn(0x7f706c0bfbe0 msgr2=0x7f706c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:22.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 --2- 172.21.15.187:0/4259823797 >> v2:172.21.15.5:3300/0 conn(0x7f706c0bfbe0 0x7f706c1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 >> v1:172.21.15.5:6789/0 conn(0x7f706c09f560 legacy=0x7f706c0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:22.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c14ce70 con 0x7f706c0bf800 2020-11-20T12:07:22.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f7060001520 con 0x7f706c0bf800 2020-11-20T12:07:22.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 >> v1:172.21.15.187:6789/0 conn(0x7f706c0bf800 legacy=0x7f706c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7050003730 0x7f7050003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.406+0000 7f706b7fe700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70500044e0 0x7f7050004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.407+0000 7f706b7fe700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7050004e50 0x7f705000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.407+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c14c9d0 con 0x7f70500044e0 2020-11-20T12:07:22.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.407+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7050004240 con 0x7f7050004e50 2020-11-20T12:07:22.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.407+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705000bc20 con 0x7f7050003730 2020-11-20T12:07:22.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.407+0000 7f707125b700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7050004e50 0x7f705000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.407+0000 7f706bfff700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70500044e0 0x7f7050004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.407+0000 7f7070a5a700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7050003730 0x7f7050003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.408+0000 7f707125b700 1 -- 172.21.15.187:0/4259823797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7050003730 msgr2=0x7f7050003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:22.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.408+0000 7f707125b700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7050003730 0x7f7050003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.408+0000 7f707125b700 1 -- 172.21.15.187:0/4259823797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70500044e0 msgr2=0x7f7050004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:22.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.408+0000 7f707125b700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70500044e0 0x7f7050004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.408+0000 7f707125b700 1 -- 172.21.15.187:0/4259823797 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7050003240 con 0x7f7050004e50 2020-11-20T12:07:22.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.408+0000 7f707125b700 1 -- 172.21.15.187:0/4259823797 reap_dead start 2020-11-20T12:07:22.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.408+0000 7f707125b700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7050004e50 0x7f705000b730 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f7064002db0 tx=0x7f7064007f10).ready entity=mon.1 client_cookie=44713bd23a590d81 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:22.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.409+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064002430 con 0x7f7050004e50 2020-11-20T12:07:22.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.409+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706400c8b0 con 0x7f7050004e50 2020-11-20T12:07:22.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.409+0000 7f706b7fe700 1 -- 172.21.15.187:0/4259823797 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706400cc70 con 0x7f7050004e50 2020-11-20T12:07:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.409+0000 7f7071a5c700 1 -- 172.21.15.187:0/4259823797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7050004e50 msgr2=0x7f705000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.409+0000 7f7071a5c700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7050004e50 0x7f705000b730 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f7064002db0 tx=0x7f7064007f10).stop 2020-11-20T12:07:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.409+0000 7f7071a5c700 1 -- 172.21.15.187:0/4259823797 shutdown_connections 2020-11-20T12:07:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.409+0000 7f7071a5c700 1 --2- 172.21.15.187:0/4259823797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7050004e50 0x7f705000b730 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.409+0000 7f7071a5c700 1 -- 172.21.15.187:0/4259823797 >> 172.21.15.187:0/4259823797 conn(0x7f706c068370 msgr2=0x7f706c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.410+0000 7f7071a5c700 1 -- 172.21.15.187:0/4259823797 shutdown_connections 2020-11-20T12:07:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.410+0000 7f7071a5c700 1 -- 172.21.15.187:0/4259823797 wait complete. 2020-11-20T12:07:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.412+0000 7f7071a5c700 1 Processor -- start 2020-11-20T12:07:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.412+0000 7f7071a5c700 1 -- start start 2020-11-20T12:07:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f7071a5c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7050004e50 0x7f706c137580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f7071a5c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f706c0b8e80 0x7f706c136870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f7071a5c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f706c136d60 0x7f706c148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f7071a5c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c1371c0 con 0x7f7050004e50 2020-11-20T12:07:22.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f7071a5c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0ba160 con 0x7f706c0b8e80 2020-11-20T12:07:22.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f7071a5c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0ba400 con 0x7f706c136d60 2020-11-20T12:07:22.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f706bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f706c0b8e80 0x7f706c136870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f706bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f706c0b8e80 0x7f706c136870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43232/0 (socket says 172.21.15.187:43232) 2020-11-20T12:07:22.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.413+0000 7f706bfff700 1 -- 172.21.15.187:0/1200423934 learned_addr learned my addr 172.21.15.187:0/1200423934 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:22.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.414+0000 7f7070a5a700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7050004e50 0x7f706c137580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.414+0000 7f706bfff700 1 -- 172.21.15.187:0/1200423934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f706c136d60 msgr2=0x7f706c148b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.414+0000 7f706bfff700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f706c136d60 0x7f706c148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.414+0000 7f706bfff700 1 -- 172.21.15.187:0/1200423934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7050004e50 msgr2=0x7f706c137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.414+0000 7f706bfff700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7050004e50 0x7f706c137580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.414+0000 7f706bfff700 1 -- 172.21.15.187:0/1200423934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c137a70 con 0x7f706c0b8e80 2020-11-20T12:07:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.414+0000 7f706bfff700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f706c0b8e80 0x7f706c136870 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f7058006f70 tx=0x7f705800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.414+0000 7f7068ff9700 1 -- 172.21.15.187:0/1200423934 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800e960 con 0x7f706c0b8e80 2020-11-20T12:07:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.415+0000 7f7068ff9700 1 -- 172.21.15.187:0/1200423934 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705800eac0 con 0x7f706c0b8e80 2020-11-20T12:07:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.415+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c0b9da0 con 0x7f706c0b8e80 2020-11-20T12:07:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.415+0000 7f7068ff9700 1 -- 172.21.15.187:0/1200423934 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800ee40 con 0x7f706c0b8e80 2020-11-20T12:07:22.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.415+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c147760 con 0x7f706c0b8e80 2020-11-20T12:07:22.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.416+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c080050 con 0x7f706c0b8e80 2020-11-20T12:07:22.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.416+0000 7f7068ff9700 1 -- 172.21.15.187:0/1200423934 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7058014680 con 0x7f706c0b8e80 2020-11-20T12:07:22.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.417+0000 7f7068ff9700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7044056d20 0x7f70440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.417+0000 7f7068ff9700 1 -- 172.21.15.187:0/1200423934 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f70580701a0 con 0x7f706c0b8e80 2020-11-20T12:07:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.417+0000 7f7070a5a700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7044056d20 0x7f70440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.418+0000 7f7070a5a700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7044056d20 0x7f70440591e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7060007870 tx=0x7f7060009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.421+0000 7f7068ff9700 1 -- 172.21.15.187:0/1200423934 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f70580455f0 con 0x7f706c0b8e80 2020-11-20T12:07:22.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.606+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f706c0ba5c0 con 0x7f706c0b8e80 2020-11-20T12:07:22.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.607+0000 7f7068ff9700 1 -- 172.21.15.187:0/1200423934 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7058048c10 con 0x7f706c0b8e80 2020-11-20T12:07:22.611 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:07:22.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.609+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7044056d20 msgr2=0x7f70440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.609+0000 7f7071a5c700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7044056d20 0x7f70440591e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7060007870 tx=0x7f7060009040).stop 2020-11-20T12:07:22.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.609+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f706c0b8e80 msgr2=0x7f706c136870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.609+0000 7f7071a5c700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f706c0b8e80 0x7f706c136870 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f7058006f70 tx=0x7f705800b040).stop 2020-11-20T12:07:22.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.610+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 shutdown_connections 2020-11-20T12:07:22.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.610+0000 7f7071a5c700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7044056d20 0x7f70440591e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.610+0000 7f7071a5c700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7050004e50 0x7f706c137580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.610+0000 7f7071a5c700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f706c136d60 0x7f706c148b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:22.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.610+0000 7f7071a5c700 1 --2- 172.21.15.187:0/1200423934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f706c0b8e80 0x7f706c136870 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.610+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 >> 172.21.15.187:0/1200423934 conn(0x7f706c068370 msgr2=0x7f706c143f20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:22.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.610+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 shutdown_connections 2020-11-20T12:07:22.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.610+0000 7f7071a5c700 1 -- 172.21.15.187:0/1200423934 wait complete. 2020-11-20T12:07:22.637 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:07:22.638 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:22.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.868+0000 7f7c1d687700 1 Processor -- start 2020-11-20T12:07:22.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.868+0000 7f7c1d687700 1 -- start start 2020-11-20T12:07:22.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.870+0000 7f7c1d687700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7c18066040 0x7f7c18148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.870+0000 7f7c1d687700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1814c170 con 0x7f7c180683f0 2020-11-20T12:07:22.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.870+0000 7f7c1d687700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1814c410 con 0x7f7c18145cc0 2020-11-20T12:07:22.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.870+0000 7f7c1d687700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1814c6b0 con 0x7f7c18066040 2020-11-20T12:07:22.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.870+0000 7f7c17fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7c18145cc0 0x7f7c18065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39844/0 (socket says 172.21.15.187:39844) 2020-11-20T12:07:22.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.870+0000 7f7c17fff700 1 -- 172.21.15.187:0/1781627665 learned_addr learned my addr 172.21.15.187:0/1781627665 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:22.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281422713 0 0) 0x7f7c1814c410 con 0x7f7c18145cc0 2020-11-20T12:07:22.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf8003240 con 0x7f7c18145cc0 2020-11-20T12:07:22.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c1ce86700 1 --2- 172.21.15.187:0/1781627665 >> v2:172.21.15.5:3300/0 conn(0x7f7c18066040 0x7f7c18148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 244504356 0 0) 0x7f7bf8003240 con 0x7f7c18145cc0 2020-11-20T12:07:22.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 >> v2:172.21.15.5:3300/0 conn(0x7f7c18066040 msgr2=0x7f7c18148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:22.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 --2- 172.21.15.187:0/1781627665 >> v2:172.21.15.5:3300/0 conn(0x7f7c18066040 0x7f7c18148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 >> v1:172.21.15.5:6789/0 conn(0x7f7c180683f0 legacy=0x7f7c181487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180658a0 con 0x7f7c18145cc0 2020-11-20T12:07:22.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f7c080014e0 con 0x7f7c18145cc0 2020-11-20T12:07:22.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 >> v1:172.21.15.187:6789/0 conn(0x7f7c18145cc0 legacy=0x7f7c18065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bf8003730 0x7f7bf8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bf80044e0 0x7f7bf8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bf8004e50 0x7f7bf800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1814c6b0 con 0x7f7bf8004e50 2020-11-20T12:07:22.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf8004260 con 0x7f7bf8003730 2020-11-20T12:07:22.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.871+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf800bc20 con 0x7f7bf80044e0 2020-11-20T12:07:22.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.872+0000 7f7c17fff700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bf8003730 0x7f7bf8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.872+0000 7f7c17fff700 1 -- 172.21.15.187:0/1781627665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bf80044e0 msgr2=0x7f7bf8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:22.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.872+0000 7f7c17fff700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bf80044e0 0x7f7bf8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:22.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.872+0000 7f7c17fff700 1 -- 172.21.15.187:0/1781627665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bf8004e50 msgr2=0x7f7bf800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:22.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.872+0000 7f7c17fff700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bf8004e50 0x7f7bf800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:22.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.872+0000 7f7c17fff700 1 -- 172.21.15.187:0/1781627665 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf8003240 con 0x7f7bf8003730 2020-11-20T12:07:22.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.872+0000 7f7c1ce86700 1 -- 172.21.15.187:0/1781627665 reap_dead start 2020-11-20T12:07:22.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.872+0000 7f7c17fff700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bf8003730 0x7f7bf8003b50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f7c0800c410 tx=0x7f7c080077d0).ready entity=mon.1 client_cookie=4a83896dd95a56d4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:22.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0800ce80 con 0x7f7bf8003730 2020-11-20T12:07:22.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c080072e0 con 0x7f7bf8003730 2020-11-20T12:07:22.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c177fe700 1 -- 172.21.15.187:0/1781627665 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08005800 con 0x7f7bf8003730 2020-11-20T12:07:22.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c1d687700 1 -- 172.21.15.187:0/1781627665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bf8003730 msgr2=0x7f7bf8003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c1d687700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bf8003730 0x7f7bf8003b50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f7c0800c410 tx=0x7f7c080077d0).stop 2020-11-20T12:07:22.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c1d687700 1 -- 172.21.15.187:0/1781627665 shutdown_connections 2020-11-20T12:07:22.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c1d687700 1 --2- 172.21.15.187:0/1781627665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bf8003730 0x7f7bf8003b50 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:22.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c1d687700 1 -- 172.21.15.187:0/1781627665 >> 172.21.15.187:0/1781627665 conn(0x7f7c180bab30 msgr2=0x7f7c180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.873+0000 7f7c1d687700 1 -- 172.21.15.187:0/1781627665 shutdown_connections 2020-11-20T12:07:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.874+0000 7f7c1d687700 1 -- 172.21.15.187:0/1781627665 wait complete. 2020-11-20T12:07:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.875+0000 7f7c1d687700 1 Processor -- start 2020-11-20T12:07:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.875+0000 7f7c1d687700 1 -- start start 2020-11-20T12:07:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.876+0000 7f7c1d687700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bf8003730 0x7f7c18144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.876+0000 7f7c1d687700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c180b6460 0x7f7c180b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.876+0000 7f7c1d687700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c180b6dd0 0x7f7c1809bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.876+0000 7f7c1d687700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180b7670 con 0x7f7c180b6dd0 2020-11-20T12:07:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.876+0000 7f7c1d687700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1809bb20 con 0x7f7c180b6460 2020-11-20T12:07:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.876+0000 7f7c1d687700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180b73d0 con 0x7f7bf8003730 2020-11-20T12:07:22.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.877+0000 7f7c0f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c180b6460 0x7f7c180b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.877+0000 7f7c0f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c180b6460 0x7f7c180b68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43250/0 (socket says 172.21.15.187:43250) 2020-11-20T12:07:22.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.877+0000 7f7c0f7fe700 1 -- 172.21.15.187:0/1247348660 learned_addr learned my addr 172.21.15.187:0/1247348660 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:22.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.877+0000 7f7c0f7fe700 1 -- 172.21.15.187:0/1247348660 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bf8003730 msgr2=0x7f7c18144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.877+0000 7f7c0f7fe700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bf8003730 0x7f7c18144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:22.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.877+0000 7f7c0f7fe700 1 -- 172.21.15.187:0/1247348660 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c180b6dd0 msgr2=0x7f7c1809bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:22.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.877+0000 7f7c0f7fe700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c180b6dd0 0x7f7c1809bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:22.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.877+0000 7f7c0f7fe700 1 -- 172.21.15.187:0/1247348660 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180b4290 con 0x7f7c180b6460 2020-11-20T12:07:22.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.878+0000 7f7c0f7fe700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c180b6460 0x7f7c180b68e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f7c0000c740 tx=0x7f7c00007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:22.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.878+0000 7f7c14ff9700 1 -- 172.21.15.187:0/1247348660 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0000e490 con 0x7f7c180b6460 2020-11-20T12:07:22.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.878+0000 7f7c14ff9700 1 -- 172.21.15.187:0/1247348660 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0000e5f0 con 0x7f7c180b6460 2020-11-20T12:07:22.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.878+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c18144520 con 0x7f7c180b6460 2020-11-20T12:07:22.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.878+0000 7f7c14ff9700 1 -- 172.21.15.187:0/1247348660 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0000e9b0 con 0x7f7c180b6460 2020-11-20T12:07:22.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.878+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1809c6b0 con 0x7f7c180b6460 2020-11-20T12:07:22.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.879+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c180b7210 con 0x7f7c180b6460 2020-11-20T12:07:22.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.880+0000 7f7c14ff9700 1 -- 172.21.15.187:0/1247348660 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7c00010070 con 0x7f7c180b6460 2020-11-20T12:07:22.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.881+0000 7f7c14ff9700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bf4056d20 0x7f7bf40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:22.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.881+0000 7f7c14ff9700 1 -- 172.21.15.187:0/1247348660 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7c0006f5b0 con 0x7f7c180b6460 2020-11-20T12:07:22.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.881+0000 7f7c17fff700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bf4056d20 0x7f7bf40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:22.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.882+0000 7f7c17fff700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bf4056d20 0x7f7bf40591e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f7c0800c410 tx=0x7f7c08015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:22.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:22.885+0000 7f7c14ff9700 1 -- 172.21.15.187:0/1247348660 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7c00044920 con 0x7f7c180b6460 2020-11-20T12:07:23.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.129+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c18061a20 con 0x7f7c180b6460 2020-11-20T12:07:23.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.130+0000 7f7c14ff9700 1 -- 172.21.15.187:0/1247348660 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f7c00047f40 con 0x7f7c180b6460 2020-11-20T12:07:23.135 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:23.135 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:23.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.132+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bf4056d20 msgr2=0x7f7bf40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:23.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.132+0000 7f7c1d687700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bf4056d20 0x7f7bf40591e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f7c0800c410 tx=0x7f7c08015000).stop 2020-11-20T12:07:23.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.132+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c180b6460 msgr2=0x7f7c180b68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:23.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c180b6460 0x7f7c180b68e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f7c0000c740 tx=0x7f7c00007b90).stop 2020-11-20T12:07:23.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 shutdown_connections 2020-11-20T12:07:23.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bf4056d20 0x7f7bf40591e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:23.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c180b6dd0 0x7f7c1809bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:23.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bf8003730 0x7f7c18144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:23.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 --2- 172.21.15.187:0/1247348660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c180b6460 0x7f7c180b68e0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:23.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 >> 172.21.15.187:0/1247348660 conn(0x7f7c180bab30 msgr2=0x7f7c1809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:23.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 shutdown_connections 2020-11-20T12:07:23.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:23.133+0000 7f7c1d687700 1 -- 172.21.15.187:0/1247348660 wait complete. 2020-11-20T12:07:23.140 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:23.360 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:07:03.348416+0000', 'modified': '2020-11-20T12:07:05.571685+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5335}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5335': {'gid': 5335, 'name': 'a', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/2880645780', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2880645780}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2880645780}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:07:23.361 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:07:23.361 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:07:23.637 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":5335,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":20.261974666,"mdsmap_epoch":19,"osdmap_epoch":42,"osdmap_epoch_barrier":41,"uptime":25.837247565999998} 2020-11-20T12:07:23.638 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5335, "mdsmap_epoch": 19, "osdmap_epoch": 42, "osdmap_epoch_barrier": 41, "rank_uptime": 20.261974666, "state": "up:active", "uptime": 25.837247566, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:07:23.638 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:07:23.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:23.639 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:07:23.690 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:07:23.691 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:07:23.828 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:07:23.829 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:07:23.868 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:07:23.868 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:07:23.877 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:07:23.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:07:23.878 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:07:23.972 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:07:23.972 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:07:23.973 INFO:teuthology.orchestra.run.smithi187:> path = os.path.join("/home/ubuntu/cephtest/mnt.0", "subdir2") 2020-11-20T12:07:23.973 INFO:teuthology.orchestra.run.smithi187:> for n in range(50, (50*3)//2): 2020-11-20T12:07:23.974 INFO:teuthology.orchestra.run.smithi187:> with open(os.path.join(path, "%s" % n), '"'"'w'"'"') as f: 2020-11-20T12:07:23.974 INFO:teuthology.orchestra.run.smithi187:> f.write(str(n)) 2020-11-20T12:07:23.974 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:07:24.349 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:24.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.637+0000 7f48d6195700 1 Processor -- start 2020-11-20T12:07:24.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.637+0000 7f48d6195700 1 -- start start 2020-11-20T12:07:24.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.638+0000 7f48d6195700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d00b57b0 0x7f48d0068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.638+0000 7f48d6195700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f48d01482d0 0x7f48d0062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.638+0000 7f48d6195700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d0144b80 con 0x7f48d0062b50 2020-11-20T12:07:24.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.638+0000 7f48d6195700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d01489e0 con 0x7f48d01482d0 2020-11-20T12:07:24.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.638+0000 7f48d6195700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0148c80 con 0x7f48d00b57b0 2020-11-20T12:07:24.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.639+0000 7f48d5994700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f48d0062b50 0x7f48d0065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39874/0 (socket says 172.21.15.187:39874) 2020-11-20T12:07:24.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.639+0000 7f48d5994700 1 -- 172.21.15.187:0/74079602 learned_addr learned my addr 172.21.15.187:0/74079602 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:24.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.639+0000 7f48d4992700 1 --2- 172.21.15.187:0/74079602 >> v2:172.21.15.5:3300/0 conn(0x7f48d01482d0 0x7f48d0062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:24.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.639+0000 7f48d5193700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d00b57b0 0x7f48d0068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:24.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.639+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772182472 0 0) 0x7f48d0144b80 con 0x7f48d0062b50 2020-11-20T12:07:24.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.639+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b8003240 con 0x7f48d0062b50 2020-11-20T12:07:24.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4056611181 0 0) 0x7f48b8003240 con 0x7f48d0062b50 2020-11-20T12:07:24.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d00b57b0 msgr2=0x7f48d0068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:24.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d00b57b0 0x7f48d0068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:24.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 >> v2:172.21.15.5:3300/0 conn(0x7f48d01482d0 msgr2=0x7f48d0062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:24.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 --2- 172.21.15.187:0/74079602 >> v2:172.21.15.5:3300/0 conn(0x7f48d01482d0 0x7f48d0062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:24.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d0148f20 con 0x7f48d0062b50 2020-11-20T12:07:24.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f48c80014e0 con 0x7f48d0062b50 2020-11-20T12:07:24.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 >> v1:172.21.15.187:6789/0 conn(0x7f48d0062b50 legacy=0x7f48d0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:24.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d00b57b0 0x7f48b80037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48b8003d10 0x7f48b80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48b80049d0 0x7f48b8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01489e0 con 0x7f48b80049d0 2020-11-20T12:07:24.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0148c80 con 0x7f48b8003d10 2020-11-20T12:07:24.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80042d0 con 0x7f48d00b57b0 2020-11-20T12:07:24.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.640+0000 7f48d4992700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48b8003d10 0x7f48b80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:24.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48d4992700 1 -- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d00b57b0 msgr2=0x7f48b80037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:24.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48d4992700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d00b57b0 0x7f48b80037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:24.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48d4992700 1 -- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48b80049d0 msgr2=0x7f48b8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:24.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48d4992700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48b80049d0 0x7f48b8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:24.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48d4992700 1 -- 172.21.15.187:0/74079602 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b8003240 con 0x7f48b8003d10 2020-11-20T12:07:24.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48d4992700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48b8003d10 0x7f48b80044e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f48d0063440 tx=0x7f48c0006df0).ready entity=mon.1 client_cookie=778c89ea7934a028 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:24.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c000b330 con 0x7f48b8003d10 2020-11-20T12:07:24.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c000eae0 con 0x7f48b8003d10 2020-11-20T12:07:24.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.641+0000 7f48bffff700 1 -- 172.21.15.187:0/74079602 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c000ee60 con 0x7f48b8003d10 2020-11-20T12:07:24.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.642+0000 7f48d6195700 1 -- 172.21.15.187:0/74079602 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48b8003d10 msgr2=0x7f48b80044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:24.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.642+0000 7f48d6195700 1 --2- 172.21.15.187:0/74079602 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48b8003d10 0x7f48b80044e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f48d0063440 tx=0x7f48c0006df0).stop 2020-11-20T12:07:24.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.642+0000 7f48d5994700 1 -- 172.21.15.187:0/74079602 reap_dead start 2020-11-20T12:07:24.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.642+0000 7f48d6195700 1 -- 172.21.15.187:0/74079602 shutdown_connections 2020-11-20T12:07:24.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.642+0000 7f48d6195700 1 -- 172.21.15.187:0/74079602 >> 172.21.15.187:0/74079602 conn(0x7f48d00bab30 msgr2=0x7f48d00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:24.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.642+0000 7f48d6195700 1 -- 172.21.15.187:0/74079602 shutdown_connections 2020-11-20T12:07:24.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.642+0000 7f48d6195700 1 -- 172.21.15.187:0/74079602 wait complete. 2020-11-20T12:07:24.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.643+0000 7f48d6195700 1 Processor -- start 2020-11-20T12:07:24.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.643+0000 7f48d6195700 1 -- start start 2020-11-20T12:07:24.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.644+0000 7f48d6195700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48d0157330 0x7f48d009f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.644+0000 7f48d6195700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d0145cf0 0x7f48d0140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.644+0000 7f48d6195700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d0157710 0x7f48d0141440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.644+0000 7f48d6195700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0141930 con 0x7f48d0157330 2020-11-20T12:07:24.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.644+0000 7f48d6195700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0157c70 con 0x7f48d0145cf0 2020-11-20T12:07:24.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.644+0000 7f48d6195700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d009ce90 con 0x7f48d0157710 2020-11-20T12:07:24.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d0145cf0 0x7f48d0140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:24.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d0145cf0 0x7f48d0140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43278/0 (socket says 172.21.15.187:43278) 2020-11-20T12:07:24.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 -- 172.21.15.187:0/2203488278 learned_addr learned my addr 172.21.15.187:0/2203488278 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:24.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d5193700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48d0157330 0x7f48d009f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:24.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 -- 172.21.15.187:0/2203488278 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d0157710 msgr2=0x7f48d0141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:24.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d0157710 0x7f48d0141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:24.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 -- 172.21.15.187:0/2203488278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48d0157330 msgr2=0x7f48d009f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:24.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48d0157330 0x7f48d009f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:24.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 -- 172.21.15.187:0/2203488278 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d009d330 con 0x7f48d0145cf0 2020-11-20T12:07:24.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d5193700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48d0157330 0x7f48d009f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:24.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48d4992700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d0145cf0 0x7f48d0140f50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f48c0012000 tx=0x7f48c0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:24.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.645+0000 7f48bd7fa700 1 -- 172.21.15.187:0/2203488278 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c0023070 con 0x7f48d0145cf0 2020-11-20T12:07:24.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.646+0000 7f48bd7fa700 1 -- 172.21.15.187:0/2203488278 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c00125c0 con 0x7f48d0145cf0 2020-11-20T12:07:24.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.646+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d01478a0 con 0x7f48d0145cf0 2020-11-20T12:07:24.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.646+0000 7f48bd7fa700 1 -- 172.21.15.187:0/2203488278 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c0012960 con 0x7f48d0145cf0 2020-11-20T12:07:24.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.646+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d0146ac0 con 0x7f48d0145cf0 2020-11-20T12:07:24.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.647+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d009d050 con 0x7f48d0145cf0 2020-11-20T12:07:24.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.647+0000 7f48bd7fa700 1 -- 172.21.15.187:0/2203488278 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f48c001f430 con 0x7f48d0145cf0 2020-11-20T12:07:24.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.648+0000 7f48bd7fa700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48a8056d20 0x7f48a80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:24.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.648+0000 7f48bd7fa700 1 -- 172.21.15.187:0/2203488278 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f48c00720d0 con 0x7f48d0145cf0 2020-11-20T12:07:24.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.648+0000 7f48d5193700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48a8056d20 0x7f48a80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:24.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.649+0000 7f48d5193700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48a8056d20 0x7f48a80591e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f48cc006f50 tx=0x7f48cc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:24.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.652+0000 7f48bd7fa700 1 -- 172.21.15.187:0/2203488278 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f48c00474c0 con 0x7f48d0145cf0 2020-11-20T12:07:24.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.894+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48d0061a20 con 0x7f48d0145cf0 2020-11-20T12:07:24.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.895+0000 7f48bd7fa700 1 -- 172.21.15.187:0/2203488278 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f48c004aae0 con 0x7f48d0145cf0 2020-11-20T12:07:24.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.897+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48a8056d20 msgr2=0x7f48a80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:24.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48a8056d20 0x7f48a80591e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f48cc006f50 tx=0x7f48cc008040).stop 2020-11-20T12:07:24.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d0145cf0 msgr2=0x7f48d0140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:24.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d0145cf0 0x7f48d0140f50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f48c0012000 tx=0x7f48c0016000).stop 2020-11-20T12:07:24.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 shutdown_connections 2020-11-20T12:07:24.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48a8056d20 0x7f48a80591e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:24.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48d0157330 0x7f48d009f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:24.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48d0157710 0x7f48d0141440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:24.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 --2- 172.21.15.187:0/2203488278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d0145cf0 0x7f48d0140f50 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:24.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 >> 172.21.15.187:0/2203488278 conn(0x7f48d00bab30 msgr2=0x7f48d01440b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:24.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 shutdown_connections 2020-11-20T12:07:24.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:24.898+0000 7f48d6195700 1 -- 172.21.15.187:0/2203488278 wait complete. 2020-11-20T12:07:24.940 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:24.940 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:24.940 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:24.942 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:07:25.210 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_created":0}} 2020-11-20T12:07:25.211 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2020-11-20T12:07:25.212 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:07:25.213 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:07:25.213 INFO:teuthology.orchestra.run.smithi187:> path = os.path.join("/home/ubuntu/cephtest/mnt.0", "subdir3") 2020-11-20T12:07:25.213 INFO:teuthology.orchestra.run.smithi187:> os.mkdir(path) 2020-11-20T12:07:25.214 INFO:teuthology.orchestra.run.smithi187:> for n in range(500): 2020-11-20T12:07:25.214 INFO:teuthology.orchestra.run.smithi187:> fpath = os.path.join(path, "%s" % n) 2020-11-20T12:07:25.215 INFO:teuthology.orchestra.run.smithi187:> with open(fpath, '"'"'w'"'"') as f: 2020-11-20T12:07:25.215 INFO:teuthology.orchestra.run.smithi187:> f.write(str(n)) 2020-11-20T12:07:25.215 INFO:teuthology.orchestra.run.smithi187:> os.unlink(fpath) 2020-11-20T12:07:25.216 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:07:25.610 INFO:teuthology.orchestra.run.smithi187.stderr:Traceback (most recent call last): 2020-11-20T12:07:25.610 INFO:teuthology.orchestra.run.smithi187.stderr: File "", line 9, in 2020-11-20T12:07:25.611 INFO:teuthology.orchestra.run.smithi187.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir3/50' 2020-11-20T12:07:25.776 INFO:teuthology.orchestra.run.smithi187.stderr:daemon-helper: command failed with exit status 1 2020-11-20T12:07:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:07:25.794 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:26.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.042+0000 7f6331cfa700 1 Processor -- start 2020-11-20T12:07:26.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.042+0000 7f6331cfa700 1 -- start start 2020-11-20T12:07:26.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.044+0000 7f6331cfa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c145cc0 0x7f632c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:26.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.044+0000 7f6331cfa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f632c1487e0 0x7f632c148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:26.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.044+0000 7f6331cfa700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c14c170 con 0x7f632c0683f0 2020-11-20T12:07:26.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.044+0000 7f6331cfa700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c14c410 con 0x7f632c1487e0 2020-11-20T12:07:26.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.044+0000 7f6331cfa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c14c6b0 con 0x7f632c145cc0 2020-11-20T12:07:26.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.044+0000 7f63314f9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f632c1487e0 0x7f632c148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:26.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.045+0000 7f63314f9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f632c1487e0 0x7f632c148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43288/0 (socket says 172.21.15.187:43288) 2020-11-20T12:07:26.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.045+0000 7f63314f9700 1 -- 172.21.15.187:0/1690350717 learned_addr learned my addr 172.21.15.187:0/1690350717 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:26.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.045+0000 7f6330cf8700 1 --2- 172.21.15.187:0/1690350717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c145cc0 0x7f632c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:26.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.045+0000 7f63314f9700 1 -- 172.21.15.187:0/1690350717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c145cc0 msgr2=0x7f632c066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:26.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.045+0000 7f63314f9700 1 --2- 172.21.15.187:0/1690350717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c145cc0 0x7f632c066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:26.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.045+0000 7f63314f9700 1 -- 172.21.15.187:0/1690350717 >> v1:172.21.15.5:6789/0 conn(0x7f632c0683f0 legacy=0x7f632c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:26.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.045+0000 7f63314f9700 1 -- 172.21.15.187:0/1690350717 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c0658a0 con 0x7f632c1487e0 2020-11-20T12:07:26.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.046+0000 7f63314f9700 1 --2- 172.21.15.187:0/1690350717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c1487e0 0x7f632c148c90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f6324009d50 tx=0x7f6324009290).ready entity=mon.1 client_cookie=2b6f3b367974b11b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:26.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.046+0000 7f632b7fe700 1 -- 172.21.15.187:0/1690350717 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632400bc80 con 0x7f632c1487e0 2020-11-20T12:07:26.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.046+0000 7f632b7fe700 1 -- 172.21.15.187:0/1690350717 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632400bde0 con 0x7f632c1487e0 2020-11-20T12:07:26.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.046+0000 7f632b7fe700 1 -- 172.21.15.187:0/1690350717 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324011610 con 0x7f632c1487e0 2020-11-20T12:07:26.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.047+0000 7f6331cfa700 1 -- 172.21.15.187:0/1690350717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c1487e0 msgr2=0x7f632c148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:26.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.047+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1690350717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c1487e0 0x7f632c148c90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f6324009d50 tx=0x7f6324009290).stop 2020-11-20T12:07:26.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.047+0000 7f6331cfa700 1 -- 172.21.15.187:0/1690350717 shutdown_connections 2020-11-20T12:07:26.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.047+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1690350717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c1487e0 0x7f632c148c90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:26.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.047+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1690350717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c145cc0 0x7f632c066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:26.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.047+0000 7f6331cfa700 1 -- 172.21.15.187:0/1690350717 >> 172.21.15.187:0/1690350717 conn(0x7f632c0bab30 msgr2=0x7f632c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:26.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.047+0000 7f6331cfa700 1 -- 172.21.15.187:0/1690350717 shutdown_connections 2020-11-20T12:07:26.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.047+0000 7f6331cfa700 1 -- 172.21.15.187:0/1690350717 wait complete. 2020-11-20T12:07:26.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.048+0000 7f6331cfa700 1 Processor -- start 2020-11-20T12:07:26.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.049+0000 7f6331cfa700 1 -- start start 2020-11-20T12:07:26.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.049+0000 7f6331cfa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c0683f0 0x7f632c14b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:26.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.049+0000 7f6331cfa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c145cc0 0x7f632c144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:26.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.049+0000 7f6331cfa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c1487e0 0x7f632c09a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:26.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.049+0000 7f6331cfa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c09b660 con 0x7f632c0683f0 2020-11-20T12:07:26.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.049+0000 7f6331cfa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c09b940 con 0x7f632c145cc0 2020-11-20T12:07:26.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.049+0000 7f6331cfa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c09bbe0 con 0x7f632c1487e0 2020-11-20T12:07:26.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c145cc0 0x7f632c144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:26.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c145cc0 0x7f632c144e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43292/0 (socket says 172.21.15.187:43292) 2020-11-20T12:07:26.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 -- 172.21.15.187:0/1196504969 learned_addr learned my addr 172.21.15.187:0/1196504969 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:26.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 -- 172.21.15.187:0/1196504969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c1487e0 msgr2=0x7f632c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:26.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c1487e0 0x7f632c09a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:26.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 -- 172.21.15.187:0/1196504969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c0683f0 msgr2=0x7f632c14b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:26.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c0683f0 0x7f632c14b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:26.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 -- 172.21.15.187:0/1196504969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c09be80 con 0x7f632c145cc0 2020-11-20T12:07:26.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.050+0000 7f632bfff700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c145cc0 0x7f632c144e20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f631800c740 tx=0x7f6318007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:26.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.051+0000 7f6328ff9700 1 -- 172.21.15.187:0/1196504969 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318007020 con 0x7f632c145cc0 2020-11-20T12:07:26.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.051+0000 7f6328ff9700 1 -- 172.21.15.187:0/1196504969 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6318007180 con 0x7f632c145cc0 2020-11-20T12:07:26.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.051+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c14ba60 con 0x7f632c145cc0 2020-11-20T12:07:26.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.051+0000 7f6328ff9700 1 -- 172.21.15.187:0/1196504969 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63180064f0 con 0x7f632c145cc0 2020-11-20T12:07:26.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.051+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c14adc0 con 0x7f632c145cc0 2020-11-20T12:07:26.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.052+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c14b3e0 con 0x7f632c145cc0 2020-11-20T12:07:26.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.053+0000 7f6328ff9700 1 -- 172.21.15.187:0/1196504969 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6318010070 con 0x7f632c145cc0 2020-11-20T12:07:26.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.053+0000 7f6328ff9700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6304056d20 0x7f63040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:26.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.053+0000 7f6328ff9700 1 -- 172.21.15.187:0/1196504969 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f631806f420 con 0x7f632c145cc0 2020-11-20T12:07:26.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.054+0000 7f6330cf8700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6304056d20 0x7f63040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:26.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.054+0000 7f6330cf8700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6304056d20 0x7f63040591e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f632000b740 tx=0x7f6320008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:26.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.058+0000 7f6328ff9700 1 -- 172.21.15.187:0/1196504969 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6318044870 con 0x7f632c145cc0 2020-11-20T12:07:26.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.303+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f632c0bb990 con 0x7f632c145cc0 2020-11-20T12:07:26.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.304+0000 7f6328ff9700 1 -- 172.21.15.187:0/1196504969 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f6318047e90 con 0x7f632c145cc0 2020-11-20T12:07:26.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6304056d20 msgr2=0x7f63040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:26.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6304056d20 0x7f63040591e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f632000b740 tx=0x7f6320008040).stop 2020-11-20T12:07:26.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c145cc0 msgr2=0x7f632c144e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:26.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c145cc0 0x7f632c144e20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f631800c740 tx=0x7f6318007b90).stop 2020-11-20T12:07:26.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 shutdown_connections 2020-11-20T12:07:26.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6304056d20 0x7f63040591e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:26.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c0683f0 0x7f632c14b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:26.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c1487e0 0x7f632c09a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:26.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 --2- 172.21.15.187:0/1196504969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f632c145cc0 0x7f632c144e20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:26.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.306+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 >> 172.21.15.187:0/1196504969 conn(0x7f632c0bab30 msgr2=0x7f632c0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:26.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.307+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 shutdown_connections 2020-11-20T12:07:26.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:26.307+0000 7f6331cfa700 1 -- 172.21.15.187:0/1196504969 wait complete. 2020-11-20T12:07:26.854 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:26.855 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:26.855 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:26.855 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:07:27.134 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_created":50}} 2020-11-20T12:07:27.134 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 50 } } 2020-11-20T12:07:27.135 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:27.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.388+0000 7f3e5e1f9700 1 Processor -- start 2020-11-20T12:07:27.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.388+0000 7f3e5e1f9700 1 -- start start 2020-11-20T12:07:27.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.389+0000 7f3e5e1f9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e58068230 0x7f3e581482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.389+0000 7f3e5e1f9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3e580b5f50 0x7f3e580b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.389+0000 7f3e5e1f9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e58144b80 con 0x7f3e580b57b0 2020-11-20T12:07:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.389+0000 7f3e5e1f9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e581487c0 con 0x7f3e580b5f50 2020-11-20T12:07:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.389+0000 7f3e5e1f9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e581489e0 con 0x7f3e58068230 2020-11-20T12:07:27.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.390+0000 7f3e5c9f6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e58068230 0x7f3e581482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:27.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.390+0000 7f3e5c9f6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e58068230 0x7f3e581482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43300/0 (socket says 172.21.15.187:43300) 2020-11-20T12:07:27.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.390+0000 7f3e5c9f6700 1 -- 172.21.15.187:0/1721432491 learned_addr learned my addr 172.21.15.187:0/1721432491 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.390+0000 7f3e5d9f8700 1 --2- 172.21.15.187:0/1721432491 >> v2:172.21.15.5:3300/0 conn(0x7f3e580b5f50 0x7f3e580b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.390+0000 7f3e5c9f6700 1 -- 172.21.15.187:0/1721432491 >> v2:172.21.15.5:3300/0 conn(0x7f3e580b5f50 msgr2=0x7f3e580b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.391+0000 7f3e5c9f6700 1 --2- 172.21.15.187:0/1721432491 >> v2:172.21.15.5:3300/0 conn(0x7f3e580b5f50 0x7f3e580b6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.391+0000 7f3e5c9f6700 1 -- 172.21.15.187:0/1721432491 >> v1:172.21.15.5:6789/0 conn(0x7f3e580b57b0 legacy=0x7f3e58062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.391+0000 7f3e5c9f6700 1 -- 172.21.15.187:0/1721432491 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e58148c80 con 0x7f3e58068230 2020-11-20T12:07:27.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.391+0000 7f3e5d9f8700 1 --2- 172.21.15.187:0/1721432491 >> v2:172.21.15.5:3300/0 conn(0x7f3e580b5f50 0x7f3e580b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:27.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.391+0000 7f3e5c9f6700 1 --2- 172.21.15.187:0/1721432491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e58068230 0x7f3e581482d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f3e5809d820 tx=0x7f3e48006eb0).ready entity=mon.1 client_cookie=758f5036517ad416 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:27.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.391+0000 7f3e47fff700 1 -- 172.21.15.187:0/1721432491 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e480098d0 con 0x7f3e58068230 2020-11-20T12:07:27.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.391+0000 7f3e47fff700 1 -- 172.21.15.187:0/1721432491 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48009a30 con 0x7f3e58068230 2020-11-20T12:07:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.391+0000 7f3e47fff700 1 -- 172.21.15.187:0/1721432491 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48009db0 con 0x7f3e58068230 2020-11-20T12:07:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.392+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/1721432491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e58068230 msgr2=0x7f3e581482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.392+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/1721432491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e58068230 0x7f3e581482d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f3e5809d820 tx=0x7f3e48006eb0).stop 2020-11-20T12:07:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.392+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/1721432491 shutdown_connections 2020-11-20T12:07:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.392+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/1721432491 >> v2:172.21.15.5:3300/0 conn(0x7f3e580b5f50 0x7f3e580b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.392+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/1721432491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e58068230 0x7f3e581482d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:27.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.392+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/1721432491 >> 172.21.15.187:0/1721432491 conn(0x7f3e580bab30 msgr2=0x7f3e580baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:27.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.392+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/1721432491 shutdown_connections 2020-11-20T12:07:27.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.392+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/1721432491 wait complete. 2020-11-20T12:07:27.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.393+0000 7f3e5e1f9700 1 Processor -- start 2020-11-20T12:07:27.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.394+0000 7f3e5e1f9700 1 -- start start 2020-11-20T12:07:27.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.394+0000 7f3e5e1f9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e58068230 0x7f3e5809f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:27.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.394+0000 7f3e5e1f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e580b57b0 0x7f3e581388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:27.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.394+0000 7f3e5e1f9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e580b5f50 0x7f3e58138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:27.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.394+0000 7f3e5e1f9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580a03c0 con 0x7f3e58068230 2020-11-20T12:07:27.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.394+0000 7f3e5e1f9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580a0660 con 0x7f3e580b5f50 2020-11-20T12:07:27.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.394+0000 7f3e5e1f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5809f830 con 0x7f3e580b57b0 2020-11-20T12:07:27.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d9f8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e580b5f50 0x7f3e58138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:27.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d9f8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e580b5f50 0x7f3e58138d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43308/0 (socket says 172.21.15.187:43308) 2020-11-20T12:07:27.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d9f8700 1 -- 172.21.15.187:0/3098819794 learned_addr learned my addr 172.21.15.187:0/3098819794 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:27.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d9f8700 1 -- 172.21.15.187:0/3098819794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e580b57b0 msgr2=0x7f3e581388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:27.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d9f8700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e580b57b0 0x7f3e581388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:27.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d1f7700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e58068230 0x7f3e5809f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:27.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d9f8700 1 -- 172.21.15.187:0/3098819794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e58068230 msgr2=0x7f3e5809f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:27.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d9f8700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e58068230 0x7f3e5809f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:27.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d9f8700 1 -- 172.21.15.187:0/3098819794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5809df90 con 0x7f3e580b5f50 2020-11-20T12:07:27.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.395+0000 7f3e5d1f7700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e58068230 0x7f3e5809f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:27.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.396+0000 7f3e5d9f8700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e580b5f50 0x7f3e58138d90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f3e5000c870 tx=0x7f3e50008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:27.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.396+0000 7f3e457fa700 1 -- 172.21.15.187:0/3098819794 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50007620 con 0x7f3e580b5f50 2020-11-20T12:07:27.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.396+0000 7f3e457fa700 1 -- 172.21.15.187:0/3098819794 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e50007780 con 0x7f3e580b5f50 2020-11-20T12:07:27.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.396+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5809e240 con 0x7f3e580b5f50 2020-11-20T12:07:27.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.396+0000 7f3e457fa700 1 -- 172.21.15.187:0/3098819794 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50007b20 con 0x7f3e580b5f50 2020-11-20T12:07:27.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.396+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5813c100 con 0x7f3e580b5f50 2020-11-20T12:07:27.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.397+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e580b0570 con 0x7f3e580b5f50 2020-11-20T12:07:27.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.398+0000 7f3e457fa700 1 -- 172.21.15.187:0/3098819794 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3e5001b460 con 0x7f3e580b5f50 2020-11-20T12:07:27.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.398+0000 7f3e457fa700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e30056d20 0x7f3e300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:27.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.398+0000 7f3e457fa700 1 -- 172.21.15.187:0/3098819794 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e5006fff0 con 0x7f3e580b5f50 2020-11-20T12:07:27.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.399+0000 7f3e5d1f7700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e30056d20 0x7f3e300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:27.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.400+0000 7f3e5d1f7700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e30056d20 0x7f3e300591e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f3e54001bc0 tx=0x7f3e54009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:27.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.403+0000 7f3e457fa700 1 -- 172.21.15.187:0/3098819794 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3e50045310 con 0x7f3e580b5f50 2020-11-20T12:07:27.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.651+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e580a0820 con 0x7f3e580b5f50 2020-11-20T12:07:27.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.652+0000 7f3e457fa700 1 -- 172.21.15.187:0/3098819794 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f3e50048930 con 0x7f3e580b5f50 2020-11-20T12:07:27.657 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:27.657 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:27.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e30056d20 msgr2=0x7f3e300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:27.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e30056d20 0x7f3e300591e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f3e54001bc0 tx=0x7f3e54009040).stop 2020-11-20T12:07:27.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e580b5f50 msgr2=0x7f3e58138d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:27.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e580b5f50 0x7f3e58138d90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f3e5000c870 tx=0x7f3e50008040).stop 2020-11-20T12:07:27.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 shutdown_connections 2020-11-20T12:07:27.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e30056d20 0x7f3e300591e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:27.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e58068230 0x7f3e5809f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:27.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e580b5f50 0x7f3e58138d90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:27.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 --2- 172.21.15.187:0/3098819794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e580b57b0 0x7f3e581388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:27.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.654+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 >> 172.21.15.187:0/3098819794 conn(0x7f3e580bab30 msgr2=0x7f3e580b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:27.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.655+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 shutdown_connections 2020-11-20T12:07:27.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:27.655+0000 7f3e5e1f9700 1 -- 172.21.15.187:0/3098819794 wait complete. 2020-11-20T12:07:27.662 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:27.681 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:07:27.960 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:07:27.962 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:07:27.963 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 50, waiting (timeout=60)... 2020-11-20T12:07:32.964 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:33.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.211+0000 7fae6d97c700 1 Processor -- start 2020-11-20T12:07:33.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.211+0000 7fae6d97c700 1 -- start start 2020-11-20T12:07:33.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.213+0000 7fae6d97c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 0x7fae68150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:33.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.213+0000 7fae6d97c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fae680b1760 0x7fae68065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:33.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.213+0000 7fae6d97c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae68150390 con 0x7fae68065680 2020-11-20T12:07:33.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.213+0000 7fae6d97c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae68150630 con 0x7fae680b1760 2020-11-20T12:07:33.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.213+0000 7fae6d97c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae68068510 con 0x7fae680b1380 2020-11-20T12:07:33.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.213+0000 7fae67fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fae680b1760 0x7fae68065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:33.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.213+0000 7fae6d17b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fae68065680 0x7fae68061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39922/0 (socket says 172.21.15.187:39922) 2020-11-20T12:07:33.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.213+0000 7fae6d17b700 1 -- 172.21.15.187:0/3507624710 learned_addr learned my addr 172.21.15.187:0/3507624710 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:33.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.214+0000 7fae6c97a700 1 --2- 172.21.15.187:0/3507624710 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 0x7fae68150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:33.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.214+0000 7fae677fe700 1 -- 172.21.15.187:0/3507624710 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633566333 0 0) 0x7fae68150390 con 0x7fae68065680 2020-11-20T12:07:33.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.214+0000 7fae677fe700 1 -- 172.21.15.187:0/3507624710 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae4c003240 con 0x7fae68065680 2020-11-20T12:07:33.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.214+0000 7fae67fff700 1 -- 172.21.15.187:0/3507624710 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 msgr2=0x7fae68150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:33.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.214+0000 7fae67fff700 1 --2- 172.21.15.187:0/3507624710 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 0x7fae68150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.214+0000 7fae67fff700 1 -- 172.21.15.187:0/3507624710 >> v1:172.21.15.187:6789/0 conn(0x7fae68065680 legacy=0x7fae68061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:33.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.214+0000 7fae67fff700 1 -- 172.21.15.187:0/3507624710 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae68150fc0 con 0x7fae680b1760 2020-11-20T12:07:33.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.214+0000 7fae67fff700 1 --2- 172.21.15.187:0/3507624710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae68065190 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fae5400b9a0 tx=0x7fae54006df0).ready entity=mon.1 client_cookie=dbde0c6a2e8d884e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:33.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae677fe700 1 -- 172.21.15.187:0/3507624710 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae540097b0 con 0x7fae680b1760 2020-11-20T12:07:33.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae677fe700 1 -- 172.21.15.187:0/3507624710 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae54009910 con 0x7fae680b1760 2020-11-20T12:07:33.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae677fe700 1 -- 172.21.15.187:0/3507624710 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54009c90 con 0x7fae680b1760 2020-11-20T12:07:33.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae6d97c700 1 -- 172.21.15.187:0/3507624710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 msgr2=0x7fae68065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:33.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3507624710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae68065190 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fae5400b9a0 tx=0x7fae54006df0).stop 2020-11-20T12:07:33.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae6d97c700 1 -- 172.21.15.187:0/3507624710 shutdown_connections 2020-11-20T12:07:33.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3507624710 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 0x7fae68150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3507624710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae68065190 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae6d97c700 1 -- 172.21.15.187:0/3507624710 >> 172.21.15.187:0/3507624710 conn(0x7fae680bab30 msgr2=0x7fae680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:33.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae6d97c700 1 -- 172.21.15.187:0/3507624710 shutdown_connections 2020-11-20T12:07:33.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.215+0000 7fae6d97c700 1 -- 172.21.15.187:0/3507624710 wait complete. 2020-11-20T12:07:33.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.217+0000 7fae6d97c700 1 Processor -- start 2020-11-20T12:07:33.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.217+0000 7fae6d97c700 1 -- start start 2020-11-20T12:07:33.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae6d97c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 0x7fae68148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:33.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae6d97c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae6815f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:33.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae6d97c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae680b7ad0 0x7fae6815fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:33.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae6d97c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae68148600 con 0x7fae680b7ad0 2020-11-20T12:07:33.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae6d97c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae680b3740 con 0x7fae680b1760 2020-11-20T12:07:33.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae6d97c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae680b39e0 con 0x7fae680b1380 2020-11-20T12:07:33.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae67fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae6815f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:33.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae67fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae6815f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43320/0 (socket says 172.21.15.187:43320) 2020-11-20T12:07:33.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.218+0000 7fae67fff700 1 -- 172.21.15.187:0/3398944277 learned_addr learned my addr 172.21.15.187:0/3398944277 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:33.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae6c97a700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 0x7fae68148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:33.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae67fff700 1 -- 172.21.15.187:0/3398944277 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 msgr2=0x7fae68148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:33.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae67fff700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 0x7fae68148110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae6d17b700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae680b7ad0 0x7fae6815fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae67fff700 1 -- 172.21.15.187:0/3398944277 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae680b7ad0 msgr2=0x7fae6815fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae67fff700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae680b7ad0 0x7fae6815fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae67fff700 1 -- 172.21.15.187:0/3398944277 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae680b3490 con 0x7fae680b1760 2020-11-20T12:07:33.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae67fff700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae6815f860 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fae5400c010 tx=0x7fae54015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:33.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae64ff9700 1 -- 172.21.15.187:0/3398944277 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5401e460 con 0x7fae680b1760 2020-11-20T12:07:33.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae64ff9700 1 -- 172.21.15.187:0/3398944277 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5401e5c0 con 0x7fae680b1760 2020-11-20T12:07:33.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae680b53c0 con 0x7fae680b1760 2020-11-20T12:07:33.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.219+0000 7fae64ff9700 1 -- 172.21.15.187:0/3398944277 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5401e960 con 0x7fae680b1760 2020-11-20T12:07:33.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.220+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae68160400 con 0x7fae680b1760 2020-11-20T12:07:33.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.221+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae68061a20 con 0x7fae680b1760 2020-11-20T12:07:33.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.221+0000 7fae64ff9700 1 -- 172.21.15.187:0/3398944277 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fae54025030 con 0x7fae680b1760 2020-11-20T12:07:33.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.222+0000 7fae64ff9700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae40056d20 0x7fae400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:33.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.222+0000 7fae64ff9700 1 -- 172.21.15.187:0/3398944277 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fae540718c0 con 0x7fae680b1760 2020-11-20T12:07:33.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.222+0000 7fae6c97a700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae40056d20 0x7fae400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:33.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.223+0000 7fae6c97a700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae40056d20 0x7fae400591e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fae5c009650 tx=0x7fae5c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:33.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.226+0000 7fae64ff9700 1 -- 172.21.15.187:0/3398944277 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fae54046d10 con 0x7fae680b1760 2020-11-20T12:07:33.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.474+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae680b57a0 con 0x7fae680b1760 2020-11-20T12:07:33.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.475+0000 7fae64ff9700 1 -- 172.21.15.187:0/3398944277 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fae54023020 con 0x7fae680b1760 2020-11-20T12:07:33.480 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:33.480 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:33.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.477+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae40056d20 msgr2=0x7fae400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:33.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.477+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae40056d20 0x7fae400591e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fae5c009650 tx=0x7fae5c008040).stop 2020-11-20T12:07:33.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.477+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 msgr2=0x7fae6815f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:33.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.477+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae6815f860 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fae5400c010 tx=0x7fae54015000).stop 2020-11-20T12:07:33.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.478+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 shutdown_connections 2020-11-20T12:07:33.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.478+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae40056d20 0x7fae400591e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.478+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae680b7ad0 0x7fae6815fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.478+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae680b1380 0x7fae68148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.478+0000 7fae6d97c700 1 --2- 172.21.15.187:0/3398944277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae680b1760 0x7fae6815f860 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:33.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.478+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 >> 172.21.15.187:0/3398944277 conn(0x7fae680bab30 msgr2=0x7fae680afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:33.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.478+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 shutdown_connections 2020-11-20T12:07:33.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:33.478+0000 7fae6d97c700 1 -- 172.21.15.187:0/3398944277 wait complete. 2020-11-20T12:07:33.490 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:33.506 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:07:33.777 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:07:33.778 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:07:33.779 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 50, waiting (timeout=60)... 2020-11-20T12:07:38.779 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:39.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.039+0000 7f326759e700 1 Processor -- start 2020-11-20T12:07:39.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.039+0000 7f326759e700 1 -- start start 2020-11-20T12:07:39.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.040+0000 7f326759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32681484d0 0x7f3268061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:39.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.040+0000 7f326759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3268066f70 0x7f32680b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:39.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.040+0000 7f326759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32680b1300 con 0x7f32680622c0 2020-11-20T12:07:39.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.040+0000 7f326759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3268067730 con 0x7f32681484d0 2020-11-20T12:07:39.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.040+0000 7f326759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32680b15a0 con 0x7f3268066f70 2020-11-20T12:07:39.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.041+0000 7f326659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32681484d0 0x7f3268061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:39.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.041+0000 7f326659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32681484d0 0x7f3268061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43326/0 (socket says 172.21.15.187:43326) 2020-11-20T12:07:39.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.041+0000 7f326659c700 1 -- 172.21.15.187:0/1380087438 learned_addr learned my addr 172.21.15.187:0/1380087438 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:39.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.041+0000 7f3266d9d700 1 --2- 172.21.15.187:0/1380087438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3268066f70 0x7f32680b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:39.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.042+0000 7f326659c700 1 -- 172.21.15.187:0/1380087438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3268066f70 msgr2=0x7f32680b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:39.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.042+0000 7f326659c700 1 --2- 172.21.15.187:0/1380087438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3268066f70 0x7f32680b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:39.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.042+0000 7f326659c700 1 -- 172.21.15.187:0/1380087438 >> v1:172.21.15.187:6789/0 conn(0x7f32680622c0 legacy=0x7f32680b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:39.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.042+0000 7f326659c700 1 -- 172.21.15.187:0/1380087438 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32680b1840 con 0x7f32681484d0 2020-11-20T12:07:39.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.042+0000 7f326659c700 1 --2- 172.21.15.187:0/1380087438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32681484d0 0x7f3268061dd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f3258006fd0 tx=0x7f3258006eb0).ready entity=mon.1 client_cookie=887a9f4c32f84781 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:39.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.042+0000 7f3265d9b700 1 -- 172.21.15.187:0/1380087438 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32580098d0 con 0x7f32681484d0 2020-11-20T12:07:39.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.042+0000 7f3265d9b700 1 -- 172.21.15.187:0/1380087438 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3258009a30 con 0x7f32681484d0 2020-11-20T12:07:39.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.042+0000 7f3265d9b700 1 -- 172.21.15.187:0/1380087438 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258009db0 con 0x7f32681484d0 2020-11-20T12:07:39.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.043+0000 7f326759e700 1 -- 172.21.15.187:0/1380087438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32681484d0 msgr2=0x7f3268061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:39.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.043+0000 7f326759e700 1 --2- 172.21.15.187:0/1380087438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32681484d0 0x7f3268061dd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f3258006fd0 tx=0x7f3258006eb0).stop 2020-11-20T12:07:39.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.043+0000 7f326759e700 1 -- 172.21.15.187:0/1380087438 shutdown_connections 2020-11-20T12:07:39.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.043+0000 7f326759e700 1 --2- 172.21.15.187:0/1380087438 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3268066f70 0x7f32680b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:39.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.043+0000 7f326759e700 1 --2- 172.21.15.187:0/1380087438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32681484d0 0x7f3268061dd0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:39.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.043+0000 7f326759e700 1 -- 172.21.15.187:0/1380087438 >> 172.21.15.187:0/1380087438 conn(0x7f32680bab30 msgr2=0x7f32680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:39.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.043+0000 7f326759e700 1 -- 172.21.15.187:0/1380087438 shutdown_connections 2020-11-20T12:07:39.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.043+0000 7f326759e700 1 -- 172.21.15.187:0/1380087438 wait complete. 2020-11-20T12:07:39.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.045+0000 7f326759e700 1 Processor -- start 2020-11-20T12:07:39.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.045+0000 7f326759e700 1 -- start start 2020-11-20T12:07:39.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32680622c0 0x7f32680b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:39.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3268066f70 0x7f3268147530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:39.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32681484d0 0x7f32680b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:39.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268146410 con 0x7f3268066f70 2020-11-20T12:07:39.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268146640 con 0x7f32680622c0 2020-11-20T12:07:39.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32681468e0 con 0x7f32681484d0 2020-11-20T12:07:39.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32680622c0 0x7f32680b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:39.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32680622c0 0x7f32680b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43332/0 (socket says 172.21.15.187:43332) 2020-11-20T12:07:39.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.046+0000 7f326659c700 1 -- 172.21.15.187:0/4156775437 learned_addr learned my addr 172.21.15.187:0/4156775437 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:39.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f326659c700 1 -- 172.21.15.187:0/4156775437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32681484d0 msgr2=0x7f32680b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:39.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f3266d9d700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32681484d0 0x7f32680b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:39.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f326659c700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32681484d0 0x7f32680b6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:39.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f326659c700 1 -- 172.21.15.187:0/4156775437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3268066f70 msgr2=0x7f3268147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:39.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f326659c700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3268066f70 0x7f3268147530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:39.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f326659c700 1 -- 172.21.15.187:0/4156775437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3268147a20 con 0x7f32680622c0 2020-11-20T12:07:39.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f326659c700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32680622c0 0x7f32680b1f40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f3258006ef0 tx=0x7f3258015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:39.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f325e7fc700 1 -- 172.21.15.187:0/4156775437 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325801e460 con 0x7f32680622c0 2020-11-20T12:07:39.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f325e7fc700 1 -- 172.21.15.187:0/4156775437 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325801e5c0 con 0x7f32680622c0 2020-11-20T12:07:39.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32680b6650 con 0x7f32680622c0 2020-11-20T12:07:39.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.047+0000 7f325e7fc700 1 -- 172.21.15.187:0/4156775437 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325801e940 con 0x7f32680622c0 2020-11-20T12:07:39.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.048+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3268142d00 con 0x7f32680622c0 2020-11-20T12:07:39.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.049+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3268149c80 con 0x7f32680622c0 2020-11-20T12:07:39.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.050+0000 7f325e7fc700 1 -- 172.21.15.187:0/4156775437 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3258025030 con 0x7f32680622c0 2020-11-20T12:07:39.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.050+0000 7f325e7fc700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3244056d20 0x7f32440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:39.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.050+0000 7f325e7fc700 1 -- 172.21.15.187:0/4156775437 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3258071710 con 0x7f32680622c0 2020-11-20T12:07:39.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.051+0000 7f325ffff700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3244056d20 0x7f32440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:39.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.051+0000 7f325ffff700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3244056d20 0x7f32440591e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f32680b97c0 tx=0x7f3250007810).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:39.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.055+0000 7f325e7fc700 1 -- 172.21.15.187:0/4156775437 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3258046c10 con 0x7f32680622c0 2020-11-20T12:07:39.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.302+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3268146aa0 con 0x7f32680622c0 2020-11-20T12:07:39.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.303+0000 7f325e7fc700 1 -- 172.21.15.187:0/4156775437 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f3258023070 con 0x7f32680622c0 2020-11-20T12:07:39.307 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:39.307 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:39.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3244056d20 msgr2=0x7f32440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:39.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3244056d20 0x7f32440591e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f32680b97c0 tx=0x7f3250007810).stop 2020-11-20T12:07:39.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32680622c0 msgr2=0x7f32680b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:39.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32680622c0 0x7f32680b1f40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f3258006ef0 tx=0x7f3258015000).stop 2020-11-20T12:07:39.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 shutdown_connections 2020-11-20T12:07:39.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3244056d20 0x7f32440591e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:39.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32681484d0 0x7f32680b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:39.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32680622c0 0x7f32680b1f40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:39.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 --2- 172.21.15.187:0/4156775437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3268066f70 0x7f3268147530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:39.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 >> 172.21.15.187:0/4156775437 conn(0x7f32680bab30 msgr2=0x7f3268144e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:39.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 shutdown_connections 2020-11-20T12:07:39.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:39.306+0000 7f326759e700 1 -- 172.21.15.187:0/4156775437 wait complete. 2020-11-20T12:07:39.312 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:39.332 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:07:39.611 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:07:39.612 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:07:39.612 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 50, waiting (timeout=60)... 2020-11-20T12:07:44.614 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:44.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.858+0000 7f4d14d76700 1 Processor -- start 2020-11-20T12:07:44.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.859+0000 7f4d14d76700 1 -- start start 2020-11-20T12:07:44.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.860+0000 7f4d14d76700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d100b57b0 0x7f4d10068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:44.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.860+0000 7f4d14d76700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b5f50 0x7f4d100b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:44.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.860+0000 7f4d14d76700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d10144b80 con 0x7f4d101482d0 2020-11-20T12:07:44.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.860+0000 7f4d14d76700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d101489e0 con 0x7f4d100b57b0 2020-11-20T12:07:44.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.860+0000 7f4d14d76700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d10148c80 con 0x7f4d100b5f50 2020-11-20T12:07:44.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.861+0000 7f4d0ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b5f50 0x7f4d100b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:44.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.861+0000 7f4d0ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b5f50 0x7f4d100b63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:44230/0 (socket says 172.21.15.187:44230) 2020-11-20T12:07:44.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.861+0000 7f4d0ffff700 1 -- 172.21.15.187:0/3907880127 learned_addr learned my addr 172.21.15.187:0/3907880127 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:44.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.861+0000 7f4d0f7fe700 1 --2- 172.21.15.187:0/3907880127 >> v2:172.21.15.5:3300/0 conn(0x7f4d100b57b0 0x7f4d10068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:44.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.862+0000 7f4d0effd700 1 -- 172.21.15.187:0/3907880127 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033109167 0 0) 0x7f4d10144b80 con 0x7f4d101482d0 2020-11-20T12:07:44.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.862+0000 7f4d0effd700 1 -- 172.21.15.187:0/3907880127 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cf0003240 con 0x7f4d101482d0 2020-11-20T12:07:44.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.862+0000 7f4d0ffff700 1 -- 172.21.15.187:0/3907880127 >> v2:172.21.15.5:3300/0 conn(0x7f4d100b57b0 msgr2=0x7f4d10068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:44.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.862+0000 7f4d0ffff700 1 --2- 172.21.15.187:0/3907880127 >> v2:172.21.15.5:3300/0 conn(0x7f4d100b57b0 0x7f4d10068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:44.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.862+0000 7f4d0ffff700 1 -- 172.21.15.187:0/3907880127 >> v1:172.21.15.5:6789/0 conn(0x7f4d101482d0 legacy=0x7f4d10062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:44.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.862+0000 7f4d0ffff700 1 -- 172.21.15.187:0/3907880127 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d10148f20 con 0x7f4d100b5f50 2020-11-20T12:07:44.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.863+0000 7f4d0ffff700 1 --2- 172.21.15.187:0/3907880127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b5f50 0x7f4d100b63c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4d0800aef0 tx=0x7f4d0800a430).ready entity=mon.2 client_cookie=ebd5db939b31b72e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:44.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.863+0000 7f4d0effd700 1 -- 172.21.15.187:0/3907880127 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800cdb0 con 0x7f4d100b5f50 2020-11-20T12:07:44.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.863+0000 7f4d0effd700 1 -- 172.21.15.187:0/3907880127 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d08010410 con 0x7f4d100b5f50 2020-11-20T12:07:44.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.863+0000 7f4d0effd700 1 -- 172.21.15.187:0/3907880127 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08010790 con 0x7f4d100b5f50 2020-11-20T12:07:44.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.864+0000 7f4d14d76700 1 -- 172.21.15.187:0/3907880127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b5f50 msgr2=0x7f4d100b63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:44.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.864+0000 7f4d14d76700 1 --2- 172.21.15.187:0/3907880127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b5f50 0x7f4d100b63c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4d0800aef0 tx=0x7f4d0800a430).stop 2020-11-20T12:07:44.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.864+0000 7f4d14d76700 1 -- 172.21.15.187:0/3907880127 shutdown_connections 2020-11-20T12:07:44.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.864+0000 7f4d14d76700 1 --2- 172.21.15.187:0/3907880127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b5f50 0x7f4d100b63c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:44.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.864+0000 7f4d14d76700 1 --2- 172.21.15.187:0/3907880127 >> v2:172.21.15.5:3300/0 conn(0x7f4d100b57b0 0x7f4d10068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:44.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.864+0000 7f4d14d76700 1 -- 172.21.15.187:0/3907880127 >> 172.21.15.187:0/3907880127 conn(0x7f4d100bab30 msgr2=0x7f4d100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:44.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.864+0000 7f4d14d76700 1 -- 172.21.15.187:0/3907880127 shutdown_connections 2020-11-20T12:07:44.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.864+0000 7f4d14d76700 1 -- 172.21.15.187:0/3907880127 wait complete. 2020-11-20T12:07:44.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.865+0000 7f4d14d76700 1 Processor -- start 2020-11-20T12:07:44.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.866+0000 7f4d14d76700 1 -- start start 2020-11-20T12:07:44.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.866+0000 7f4d14d76700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b57b0 0x7f4d10090730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:44.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.866+0000 7f4d14d76700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d100b5f50 0x7f4d10092590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:44.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.866+0000 7f4d14d76700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d101482d0 0x7f4d10156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:44.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.866+0000 7f4d14d76700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d10092e30 con 0x7f4d100b5f50 2020-11-20T12:07:44.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.866+0000 7f4d14d76700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100930d0 con 0x7f4d101482d0 2020-11-20T12:07:44.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.866+0000 7f4d14d76700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d10093370 con 0x7f4d100b57b0 2020-11-20T12:07:44.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d0ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d101482d0 0x7f4d10156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:44.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d0ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d101482d0 0x7f4d10156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43348/0 (socket says 172.21.15.187:43348) 2020-11-20T12:07:44.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d0ffff700 1 -- 172.21.15.187:0/417871115 learned_addr learned my addr 172.21.15.187:0/417871115 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:44.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d0f7fe700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b57b0 0x7f4d10090730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:44.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d07fff700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d100b5f50 0x7f4d10092590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:44.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d0ffff700 1 -- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b57b0 msgr2=0x7f4d10090730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:44.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d0ffff700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b57b0 0x7f4d10090730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:44.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d0ffff700 1 -- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d100b5f50 msgr2=0x7f4d10092590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:44.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.867+0000 7f4d0ffff700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d100b5f50 0x7f4d10092590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:44.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.868+0000 7f4d0ffff700 1 -- 172.21.15.187:0/417871115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d10090c20 con 0x7f4d101482d0 2020-11-20T12:07:44.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.868+0000 7f4d07fff700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d100b5f50 0x7f4d10092590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:44.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.868+0000 7f4d0ffff700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d101482d0 0x7f4d10156460 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f4d0800c900 tx=0x7f4d0800a430).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:44.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.868+0000 7f4d077fe700 1 -- 172.21.15.187:0/417871115 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0801aa70 con 0x7f4d101482d0 2020-11-20T12:07:44.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.868+0000 7f4d077fe700 1 -- 172.21.15.187:0/417871115 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0801abd0 con 0x7f4d101482d0 2020-11-20T12:07:44.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.868+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d10092a80 con 0x7f4d101482d0 2020-11-20T12:07:44.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.868+0000 7f4d077fe700 1 -- 172.21.15.187:0/417871115 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08023430 con 0x7f4d101482d0 2020-11-20T12:07:44.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.868+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d10156070 con 0x7f4d101482d0 2020-11-20T12:07:44.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.869+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d10061a20 con 0x7f4d101482d0 2020-11-20T12:07:44.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.870+0000 7f4d077fe700 1 -- 172.21.15.187:0/417871115 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4d0802a070 con 0x7f4d101482d0 2020-11-20T12:07:44.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.870+0000 7f4d077fe700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4cec056d20 0x7f4cec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:44.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.870+0000 7f4d077fe700 1 -- 172.21.15.187:0/417871115 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4d08071c80 con 0x7f4d101482d0 2020-11-20T12:07:44.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.871+0000 7f4d0f7fe700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4cec056d20 0x7f4cec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:44.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.871+0000 7f4d0f7fe700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4cec056d20 0x7f4cec0591e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4d00006bd0 tx=0x7f4d00008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:44.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:44.874+0000 7f4d077fe700 1 -- 172.21.15.187:0/417871115 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4d08046fc0 con 0x7f4d101482d0 2020-11-20T12:07:45.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.118+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d100446d0 con 0x7f4d101482d0 2020-11-20T12:07:45.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.119+0000 7f4d077fe700 1 -- 172.21.15.187:0/417871115 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f4d0804a5e0 con 0x7f4d101482d0 2020-11-20T12:07:45.123 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:45.123 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:45.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4cec056d20 msgr2=0x7f4cec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:45.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4cec056d20 0x7f4cec0591e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4d00006bd0 tx=0x7f4d00008040).stop 2020-11-20T12:07:45.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d101482d0 msgr2=0x7f4d10156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:45.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d101482d0 0x7f4d10156460 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f4d0800c900 tx=0x7f4d0800a430).stop 2020-11-20T12:07:45.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 shutdown_connections 2020-11-20T12:07:45.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4cec056d20 0x7f4cec0591e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:45.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d101482d0 0x7f4d10156460 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:45.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d100b57b0 0x7f4d10090730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:45.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 --2- 172.21.15.187:0/417871115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d100b5f50 0x7f4d10092590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:45.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 >> 172.21.15.187:0/417871115 conn(0x7f4d100bab30 msgr2=0x7f4d100b9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:45.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 shutdown_connections 2020-11-20T12:07:45.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:45.122+0000 7f4d14d76700 1 -- 172.21.15.187:0/417871115 wait complete. 2020-11-20T12:07:45.128 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:45.150 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:07:45.424 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:07:45.425 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:07:45.425 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 50, waiting (timeout=60)... 2020-11-20T12:07:48.795 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:07:48.803 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:07:50.427 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:50.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.676+0000 7f6a68802700 1 Processor -- start 2020-11-20T12:07:50.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.676+0000 7f6a68802700 1 -- start start 2020-11-20T12:07:50.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.678+0000 7f6a68802700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6a64068230 0x7f6a641482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:50.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.678+0000 7f6a68802700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6a640b5f50 0x7f6a640b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:50.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.678+0000 7f6a68802700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a64144b80 con 0x7f6a640b57b0 2020-11-20T12:07:50.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.678+0000 7f6a68802700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a641487c0 con 0x7f6a640b5f50 2020-11-20T12:07:50.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.678+0000 7f6a68802700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a641489e0 con 0x7f6a64068230 2020-11-20T12:07:50.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.678+0000 7f6a62d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6a640b57b0 0x7f6a64062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39960/0 (socket says 172.21.15.187:39960) 2020-11-20T12:07:50.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.678+0000 7f6a62d9d700 1 -- 172.21.15.187:0/1252055617 learned_addr learned my addr 172.21.15.187:0/1252055617 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:07:50.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.678+0000 7f6a6259c700 1 --2- 172.21.15.187:0/1252055617 >> v2:172.21.15.187:3300/0 conn(0x7f6a64068230 0x7f6a641482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:50.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.679+0000 7f6a6359e700 1 --2- 172.21.15.187:0/1252055617 >> v2:172.21.15.5:3300/0 conn(0x7f6a640b5f50 0x7f6a640b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:50.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.679+0000 7f6a6259c700 1 -- 172.21.15.187:0/1252055617 >> v2:172.21.15.5:3300/0 conn(0x7f6a640b5f50 msgr2=0x7f6a640b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:50.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.679+0000 7f6a6259c700 1 --2- 172.21.15.187:0/1252055617 >> v2:172.21.15.5:3300/0 conn(0x7f6a640b5f50 0x7f6a640b6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.679+0000 7f6a6259c700 1 -- 172.21.15.187:0/1252055617 >> v1:172.21.15.187:6789/0 conn(0x7f6a640b57b0 legacy=0x7f6a64062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:50.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.679+0000 7f6a6259c700 1 -- 172.21.15.187:0/1252055617 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a64148c80 con 0x7f6a64068230 2020-11-20T12:07:50.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.679+0000 7f6a6259c700 1 --2- 172.21.15.187:0/1252055617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a64068230 0x7f6a641482d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f6a6409d820 tx=0x7f6a58006df0).ready entity=mon.1 client_cookie=8329dcfc563126ed server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:50.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.680+0000 7f6a61d9b700 1 -- 172.21.15.187:0/1252055617 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a580097b0 con 0x7f6a64068230 2020-11-20T12:07:50.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.680+0000 7f6a61d9b700 1 -- 172.21.15.187:0/1252055617 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a58009910 con 0x7f6a64068230 2020-11-20T12:07:50.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.680+0000 7f6a61d9b700 1 -- 172.21.15.187:0/1252055617 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58009c90 con 0x7f6a64068230 2020-11-20T12:07:50.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.680+0000 7f6a68802700 1 -- 172.21.15.187:0/1252055617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a64068230 msgr2=0x7f6a641482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:50.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.680+0000 7f6a68802700 1 --2- 172.21.15.187:0/1252055617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a64068230 0x7f6a641482d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f6a6409d820 tx=0x7f6a58006df0).stop 2020-11-20T12:07:50.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.680+0000 7f6a68802700 1 -- 172.21.15.187:0/1252055617 shutdown_connections 2020-11-20T12:07:50.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.680+0000 7f6a68802700 1 --2- 172.21.15.187:0/1252055617 >> v2:172.21.15.5:3300/0 conn(0x7f6a640b5f50 0x7f6a640b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.681+0000 7f6a68802700 1 --2- 172.21.15.187:0/1252055617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a64068230 0x7f6a641482d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.681+0000 7f6a68802700 1 -- 172.21.15.187:0/1252055617 >> 172.21.15.187:0/1252055617 conn(0x7f6a640bab30 msgr2=0x7f6a640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:50.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.681+0000 7f6a68802700 1 -- 172.21.15.187:0/1252055617 shutdown_connections 2020-11-20T12:07:50.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.681+0000 7f6a68802700 1 -- 172.21.15.187:0/1252055617 wait complete. 2020-11-20T12:07:50.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.682+0000 7f6a68802700 1 Processor -- start 2020-11-20T12:07:50.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.682+0000 7f6a68802700 1 -- start start 2020-11-20T12:07:50.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.683+0000 7f6a68802700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a64068230 0x7f6a640b7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:50.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.683+0000 7f6a68802700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a640b57b0 0x7f6a64093270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:50.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.683+0000 7f6a68802700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a640b5f50 0x7f6a64147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:50.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.683+0000 7f6a68802700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64093760 con 0x7f6a640b57b0 2020-11-20T12:07:50.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.683+0000 7f6a68802700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a640b6d50 con 0x7f6a640b5f50 2020-11-20T12:07:50.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.683+0000 7f6a68802700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a640b6fc0 con 0x7f6a64068230 2020-11-20T12:07:50.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.683+0000 7f6a6359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a640b5f50 0x7f6a64147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:50.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a640b5f50 0x7f6a64147cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43364/0 (socket says 172.21.15.187:43364) 2020-11-20T12:07:50.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6359e700 1 -- 172.21.15.187:0/1336959229 learned_addr learned my addr 172.21.15.187:0/1336959229 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:50.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6259c700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a640b57b0 0x7f6a64093270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:50.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a62d9d700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a64068230 0x7f6a640b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:50.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6359e700 1 -- 172.21.15.187:0/1336959229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a64068230 msgr2=0x7f6a640b7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:50.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6359e700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a64068230 0x7f6a640b7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6359e700 1 -- 172.21.15.187:0/1336959229 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a640b57b0 msgr2=0x7f6a64093270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:50.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6359e700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a640b57b0 0x7f6a64093270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6359e700 1 -- 172.21.15.187:0/1336959229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a640b7260 con 0x7f6a640b5f50 2020-11-20T12:07:50.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.684+0000 7f6a6359e700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a640b5f50 0x7f6a64147cc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f6a5c00eba0 tx=0x7f6a5c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:50.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.685+0000 7f6a4affd700 1 -- 172.21.15.187:0/1336959229 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c00c7f0 con 0x7f6a640b5f50 2020-11-20T12:07:50.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.685+0000 7f6a4affd700 1 -- 172.21.15.187:0/1336959229 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c00c950 con 0x7f6a640b5f50 2020-11-20T12:07:50.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.685+0000 7f6a4affd700 1 -- 172.21.15.187:0/1336959229 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c00ccf0 con 0x7f6a640b5f50 2020-11-20T12:07:50.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.685+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a641481b0 con 0x7f6a640b5f50 2020-11-20T12:07:50.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.685+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6415b820 con 0x7f6a640b5f50 2020-11-20T12:07:50.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.686+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a64061a20 con 0x7f6a640b5f50 2020-11-20T12:07:50.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.687+0000 7f6a4affd700 1 -- 172.21.15.187:0/1336959229 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6a5c013640 con 0x7f6a640b5f50 2020-11-20T12:07:50.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.687+0000 7f6a4affd700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a40056d20 0x7f6a400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:50.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.687+0000 7f6a4affd700 1 -- 172.21.15.187:0/1336959229 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6a5c070010 con 0x7f6a640b5f50 2020-11-20T12:07:50.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.687+0000 7f6a62d9d700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a40056d20 0x7f6a400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:50.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.688+0000 7f6a62d9d700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a40056d20 0x7f6a400591e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f6a540055a0 tx=0x7f6a54009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:50.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.691+0000 7f6a4affd700 1 -- 172.21.15.187:0/1336959229 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6a5c045400 con 0x7f6a640b5f50 2020-11-20T12:07:50.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.937+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a640446d0 con 0x7f6a640b5f50 2020-11-20T12:07:50.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.938+0000 7f6a4affd700 1 -- 172.21.15.187:0/1336959229 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f6a5c048a20 con 0x7f6a640b5f50 2020-11-20T12:07:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:50.943 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:50.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a40056d20 msgr2=0x7f6a400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:50.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a40056d20 0x7f6a400591e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f6a540055a0 tx=0x7f6a54009040).stop 2020-11-20T12:07:50.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a640b5f50 msgr2=0x7f6a64147cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:50.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a640b5f50 0x7f6a64147cc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f6a5c00eba0 tx=0x7f6a5c008040).stop 2020-11-20T12:07:50.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 shutdown_connections 2020-11-20T12:07:50.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a40056d20 0x7f6a400591e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a640b5f50 0x7f6a64147cc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a64068230 0x7f6a640b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 --2- 172.21.15.187:0/1336959229 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a640b57b0 0x7f6a64093270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:50.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 >> 172.21.15.187:0/1336959229 conn(0x7f6a640bab30 msgr2=0x7f6a640afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:50.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 shutdown_connections 2020-11-20T12:07:50.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:50.941+0000 7f6a68802700 1 -- 172.21.15.187:0/1336959229 wait complete. 2020-11-20T12:07:50.948 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:50.970 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:07:51.248 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:07:51.248 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:07:51.249 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 50, waiting (timeout=60)... 2020-11-20T12:07:56.250 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:56.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.498+0000 7fbbcdff6700 1 Processor -- start 2020-11-20T12:07:56.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.498+0000 7fbbcdff6700 1 -- start start 2020-11-20T12:07:56.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.499+0000 7fbbcdff6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc8062580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:56.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.499+0000 7fbbcdff6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbbc80666e0 0x7fbbc8067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:56.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.499+0000 7fbbcdff6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbbc8062a70 0x7fbbc8068220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:56.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.499+0000 7fbbcdff6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc8061e80 con 0x7fbbc80666e0 2020-11-20T12:07:56.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.499+0000 7fbbcdff6700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc80620e0 con 0x7fbbc8062a70 2020-11-20T12:07:56.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.499+0000 7fbbcdff6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc80b1500 con 0x7fbbc814c750 2020-11-20T12:07:56.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbbc8062a70 0x7fbbc8068220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:56.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbbc8062a70 0x7fbbc8068220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43378/0 (socket says 172.21.15.187:43378) 2020-11-20T12:07:56.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc7fff700 1 -- 172.21.15.187:0/1027078848 learned_addr learned my addr 172.21.15.187:0/1027078848 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:56.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc77fe700 1 --2- 172.21.15.187:0/1027078848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc8062580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:56.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc6ffd700 1 --2- 172.21.15.187:0/1027078848 >> v2:172.21.15.5:3300/0 conn(0x7fbbc80666e0 0x7fbbc8067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:56.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc7fff700 1 -- 172.21.15.187:0/1027078848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 msgr2=0x7fbbc8062580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:56.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc7fff700 1 --2- 172.21.15.187:0/1027078848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc8062580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc7fff700 1 -- 172.21.15.187:0/1027078848 >> v2:172.21.15.5:3300/0 conn(0x7fbbc80666e0 msgr2=0x7fbbc8067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:56.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc7fff700 1 --2- 172.21.15.187:0/1027078848 >> v2:172.21.15.5:3300/0 conn(0x7fbbc80666e0 0x7fbbc8067d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc7fff700 1 -- 172.21.15.187:0/1027078848 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc80b17d0 con 0x7fbbc8062a70 2020-11-20T12:07:56.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.500+0000 7fbbc77fe700 1 --2- 172.21.15.187:0/1027078848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc8062580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:56.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.501+0000 7fbbc7fff700 1 --2- 172.21.15.187:0/1027078848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc8062a70 0x7fbbc8068220 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fbbc8067c10 tx=0x7fbbc0009290).ready entity=mon.1 client_cookie=739dc61727fac67c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:56.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.501+0000 7fbbc67fc700 1 -- 172.21.15.187:0/1027078848 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc000bbc0 con 0x7fbbc8062a70 2020-11-20T12:07:56.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.501+0000 7fbbc67fc700 1 -- 172.21.15.187:0/1027078848 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc000bd20 con 0x7fbbc8062a70 2020-11-20T12:07:56.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.501+0000 7fbbc67fc700 1 -- 172.21.15.187:0/1027078848 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0011610 con 0x7fbbc8062a70 2020-11-20T12:07:56.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 -- 172.21.15.187:0/1027078848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc8062a70 msgr2=0x7fbbc8068220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:56.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/1027078848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc8062a70 0x7fbbc8068220 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fbbc8067c10 tx=0x7fbbc0009290).stop 2020-11-20T12:07:56.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 -- 172.21.15.187:0/1027078848 shutdown_connections 2020-11-20T12:07:56.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/1027078848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc8062a70 0x7fbbc8068220 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/1027078848 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc8062580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/1027078848 >> v2:172.21.15.5:3300/0 conn(0x7fbbc80666e0 0x7fbbc8067d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 -- 172.21.15.187:0/1027078848 >> 172.21.15.187:0/1027078848 conn(0x7fbbc80bacd0 msgr2=0x7fbbc80bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:56.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 -- 172.21.15.187:0/1027078848 shutdown_connections 2020-11-20T12:07:56.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.502+0000 7fbbcdff6700 1 -- 172.21.15.187:0/1027078848 wait complete. 2020-11-20T12:07:56.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.503+0000 7fbbcdff6700 1 Processor -- start 2020-11-20T12:07:56.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.503+0000 7fbbcdff6700 1 -- start start 2020-11-20T12:07:56.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.504+0000 7fbbcdff6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbc8062a70 0x7fbbc80ba420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:56.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.504+0000 7fbbcdff6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc80666e0 0x7fbbc814d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:56.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.504+0000 7fbbcdff6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc80b3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:56.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.504+0000 7fbbcdff6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc814cf00 con 0x7fbbc8062a70 2020-11-20T12:07:56.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.504+0000 7fbbcdff6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc814d1a0 con 0x7fbbc80666e0 2020-11-20T12:07:56.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.504+0000 7fbbcdff6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc809a3e0 con 0x7fbbc814c750 2020-11-20T12:07:56.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc80666e0 0x7fbbc814d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:56.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc80666e0 0x7fbbc814d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43382/0 (socket says 172.21.15.187:43382) 2020-11-20T12:07:56.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc6ffd700 1 -- 172.21.15.187:0/2377561271 learned_addr learned my addr 172.21.15.187:0/2377561271 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:07:56.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc7fff700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc80b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:56.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc77fe700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbc8062a70 0x7fbbc80ba420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:56.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc6ffd700 1 -- 172.21.15.187:0/2377561271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 msgr2=0x7fbbc80b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:56.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc6ffd700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc80b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc6ffd700 1 -- 172.21.15.187:0/2377561271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbc8062a70 msgr2=0x7fbbc80ba420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:56.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc6ffd700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbc8062a70 0x7fbbc80ba420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.505+0000 7fbbc6ffd700 1 -- 172.21.15.187:0/2377561271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc80ba910 con 0x7fbbc80666e0 2020-11-20T12:07:56.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.506+0000 7fbbc6ffd700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc80666e0 0x7fbbc814d640 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7fbbbc00a7e0 tx=0x7fbbbc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:56.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.506+0000 7fbbaf7fe700 1 -- 172.21.15.187:0/2377561271 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc006020 con 0x7fbbc80666e0 2020-11-20T12:07:56.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.506+0000 7fbbaf7fe700 1 -- 172.21.15.187:0/2377561271 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc00c030 con 0x7fbbc80666e0 2020-11-20T12:07:56.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.506+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc815ffb0 con 0x7fbbc80666e0 2020-11-20T12:07:56.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.506+0000 7fbbaf7fe700 1 -- 172.21.15.187:0/2377561271 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0063d0 con 0x7fbbc80666e0 2020-11-20T12:07:56.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.506+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc814dcf0 con 0x7fbbc80666e0 2020-11-20T12:07:56.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.507+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc8063600 con 0x7fbbc80666e0 2020-11-20T12:07:56.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.508+0000 7fbbaf7fe700 1 -- 172.21.15.187:0/2377561271 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbbbc0058e0 con 0x7fbbc80666e0 2020-11-20T12:07:56.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.508+0000 7fbbaf7fe700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbba4056d20 0x7fbba40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:56.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.508+0000 7fbbaf7fe700 1 -- 172.21.15.187:0/2377561271 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbbbc06fb10 con 0x7fbbc80666e0 2020-11-20T12:07:56.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.509+0000 7fbbc77fe700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbba4056d20 0x7fbba40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:56.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.509+0000 7fbbc77fe700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbba4056d20 0x7fbba40591e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fbbb800c840 tx=0x7fbbb8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:56.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.512+0000 7fbbaf7fe700 1 -- 172.21.15.187:0/2377561271 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbbbc044e90 con 0x7fbbc80666e0 2020-11-20T12:07:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.754+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbc814cb30 con 0x7fbbc80666e0 2020-11-20T12:07:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.755+0000 7fbbaf7fe700 1 -- 172.21.15.187:0/2377561271 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fbbbc0484b0 con 0x7fbbc80666e0 2020-11-20T12:07:56.759 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:07:56.759 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:07:56.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbba4056d20 msgr2=0x7fbba40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbba4056d20 0x7fbba40591e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fbbb800c840 tx=0x7fbbb8008040).stop 2020-11-20T12:07:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc80666e0 msgr2=0x7fbbc814d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc80666e0 0x7fbbc814d640 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7fbbbc00a7e0 tx=0x7fbbbc008040).stop 2020-11-20T12:07:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 shutdown_connections 2020-11-20T12:07:56.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbba4056d20 0x7fbba40591e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbc8062a70 0x7fbbc80ba420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbc814c750 0x7fbbc80b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 --2- 172.21.15.187:0/2377561271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbc80666e0 0x7fbbc814d640 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:56.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.757+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 >> 172.21.15.187:0/2377561271 conn(0x7fbbc80bacd0 msgr2=0x7fbbc80a05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:56.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.758+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 shutdown_connections 2020-11-20T12:07:56.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:07:56.758+0000 7fbbcdff6700 1 -- 172.21.15.187:0/2377561271 wait complete. 2020-11-20T12:07:56.765 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:07:56.782 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:07:57.054 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:07:57.054 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:07:57.055 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 50, waiting (timeout=60)... 2020-11-20T12:08:02.056 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:02.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.302+0000 7fb0153c4700 1 Processor -- start 2020-11-20T12:08:02.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.302+0000 7fb0153c4700 1 -- start start 2020-11-20T12:08:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.303+0000 7fb0153c4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb010065f90 0x7fb010065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.303+0000 7fb0153c4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb01009ca90 0x7fb01009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.303+0000 7fb0153c4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0100b13e0 con 0x7fb010065190 2020-11-20T12:08:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.303+0000 7fb0153c4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb01014ab30 con 0x7fb010065f90 2020-11-20T12:08:02.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.303+0000 7fb0153c4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb010062c10 con 0x7fb01009ca90 2020-11-20T12:08:02.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.304+0000 7fb00ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb010065190 0x7fb010143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:39996/0 (socket says 172.21.15.187:39996) 2020-11-20T12:08:02.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.304+0000 7fb00ffff700 1 -- 172.21.15.187:0/1202718830 learned_addr learned my addr 172.21.15.187:0/1202718830 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:02.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.304+0000 7fb014bc3700 1 --2- 172.21.15.187:0/1202718830 >> v2:172.21.15.187:3300/0 conn(0x7fb01009ca90 0x7fb01009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:02.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.305+0000 7fb007fff700 1 --2- 172.21.15.187:0/1202718830 >> v2:172.21.15.5:3300/0 conn(0x7fb010065f90 0x7fb010065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:02.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.305+0000 7fb00f7fe700 1 -- 172.21.15.187:0/1202718830 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110179745 0 0) 0x7fb0100b13e0 con 0x7fb010065190 2020-11-20T12:08:02.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.305+0000 7fb00f7fe700 1 -- 172.21.15.187:0/1202718830 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff0003240 con 0x7fb010065190 2020-11-20T12:08:02.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.305+0000 7fb014bc3700 1 -- 172.21.15.187:0/1202718830 >> v2:172.21.15.5:3300/0 conn(0x7fb010065f90 msgr2=0x7fb010065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:02.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.305+0000 7fb014bc3700 1 --2- 172.21.15.187:0/1202718830 >> v2:172.21.15.5:3300/0 conn(0x7fb010065f90 0x7fb010065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:02.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.305+0000 7fb014bc3700 1 -- 172.21.15.187:0/1202718830 >> v1:172.21.15.187:6789/0 conn(0x7fb010065190 legacy=0x7fb010143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:02.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.305+0000 7fb014bc3700 1 -- 172.21.15.187:0/1202718830 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0100b1640 con 0x7fb01009ca90 2020-11-20T12:08:02.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.305+0000 7fb014bc3700 1 --2- 172.21.15.187:0/1202718830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb01009ca90 0x7fb01009cf20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fb010099f90 tx=0x7fb0080091d0).ready entity=mon.1 client_cookie=a3e089b0455f9229 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:02.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.306+0000 7fb00f7fe700 1 -- 172.21.15.187:0/1202718830 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800ba20 con 0x7fb01009ca90 2020-11-20T12:08:02.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.306+0000 7fb00f7fe700 1 -- 172.21.15.187:0/1202718830 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00800bb80 con 0x7fb01009ca90 2020-11-20T12:08:02.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.306+0000 7fb00f7fe700 1 -- 172.21.15.187:0/1202718830 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008011400 con 0x7fb01009ca90 2020-11-20T12:08:02.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.306+0000 7fb0153c4700 1 -- 172.21.15.187:0/1202718830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb01009ca90 msgr2=0x7fb01009cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:02.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.306+0000 7fb0153c4700 1 --2- 172.21.15.187:0/1202718830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb01009ca90 0x7fb01009cf20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fb010099f90 tx=0x7fb0080091d0).stop 2020-11-20T12:08:02.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.306+0000 7fb0153c4700 1 -- 172.21.15.187:0/1202718830 shutdown_connections 2020-11-20T12:08:02.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.307+0000 7fb0153c4700 1 --2- 172.21.15.187:0/1202718830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb01009ca90 0x7fb01009cf20 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:02.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.307+0000 7fb0153c4700 1 --2- 172.21.15.187:0/1202718830 >> v2:172.21.15.5:3300/0 conn(0x7fb010065f90 0x7fb010065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:02.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.307+0000 7fb0153c4700 1 -- 172.21.15.187:0/1202718830 >> 172.21.15.187:0/1202718830 conn(0x7fb0100bab30 msgr2=0x7fb0100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:02.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.307+0000 7fb0153c4700 1 -- 172.21.15.187:0/1202718830 shutdown_connections 2020-11-20T12:08:02.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.307+0000 7fb0153c4700 1 -- 172.21.15.187:0/1202718830 wait complete. 2020-11-20T12:08:02.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.308+0000 7fb0153c4700 1 Processor -- start 2020-11-20T12:08:02.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.308+0000 7fb0153c4700 1 -- start start 2020-11-20T12:08:02.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb0153c4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb010065190 0x7fb0101450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:02.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb0153c4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb010065f90 0x7fb0101596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:02.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb0153c4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb01009ca90 0x7fb010159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:02.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb0153c4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb010140ab0 con 0x7fb01009ca90 2020-11-20T12:08:02.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb0153c4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0101560a0 con 0x7fb010065190 2020-11-20T12:08:02.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb0153c4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb010156340 con 0x7fb010065f90 2020-11-20T12:08:02.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb00ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb010065190 0x7fb0101450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:02.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb00ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb010065190 0x7fb0101450a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43396/0 (socket says 172.21.15.187:43396) 2020-11-20T12:08:02.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.309+0000 7fb00ffff700 1 -- 172.21.15.187:0/2639932157 learned_addr learned my addr 172.21.15.187:0/2639932157 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:02.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00ffff700 1 -- 172.21.15.187:0/2639932157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb010065f90 msgr2=0x7fb0101596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:02.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00ffff700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb010065f90 0x7fb0101596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:02.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00ffff700 1 -- 172.21.15.187:0/2639932157 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb01009ca90 msgr2=0x7fb010159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:02.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00ffff700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb01009ca90 0x7fb010159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:02.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00ffff700 1 -- 172.21.15.187:0/2639932157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0101565e0 con 0x7fb010065190 2020-11-20T12:08:02.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00ffff700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb010065190 0x7fb0101450a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7fb00000c6a0 tx=0x7fb000007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:02.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00cff9700 1 -- 172.21.15.187:0/2639932157 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0000072e0 con 0x7fb010065190 2020-11-20T12:08:02.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00cff9700 1 -- 172.21.15.187:0/2639932157 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb000007440 con 0x7fb010065190 2020-11-20T12:08:02.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0100a0570 con 0x7fb010065190 2020-11-20T12:08:02.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.310+0000 7fb00cff9700 1 -- 172.21.15.187:0/2639932157 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000005820 con 0x7fb010065190 2020-11-20T12:08:02.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.311+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb010155da0 con 0x7fb010065190 2020-11-20T12:08:02.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.312+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb010156770 con 0x7fb010065190 2020-11-20T12:08:02.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.312+0000 7fb00cff9700 1 -- 172.21.15.187:0/2639932157 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb000010070 con 0x7fb010065190 2020-11-20T12:08:02.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.313+0000 7fb00cff9700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fafec056d20 0x7fafec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:02.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.313+0000 7fb00cff9700 1 -- 172.21.15.187:0/2639932157 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb00006f770 con 0x7fb010065190 2020-11-20T12:08:02.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.313+0000 7fb007fff700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fafec056d20 0x7fafec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:02.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.314+0000 7fb007fff700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fafec056d20 0x7fafec0591e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fb01009e840 tx=0x7faff8006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:02.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.317+0000 7fb00cff9700 1 -- 172.21.15.187:0/2639932157 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb000044ab0 con 0x7fb010065190 2020-11-20T12:08:02.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.566+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb010061a20 con 0x7fb010065190 2020-11-20T12:08:02.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.566+0000 7fb00cff9700 1 -- 172.21.15.187:0/2639932157 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fb0000480d0 con 0x7fb010065190 2020-11-20T12:08:02.572 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:02.572 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:08:02.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fafec056d20 msgr2=0x7fafec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:02.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fafec056d20 0x7fafec0591e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fb01009e840 tx=0x7faff8006df0).stop 2020-11-20T12:08:02.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb010065190 msgr2=0x7fb0101450a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:02.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb010065190 0x7fb0101450a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7fb00000c6a0 tx=0x7fb000007ac0).stop 2020-11-20T12:08:02.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 shutdown_connections 2020-11-20T12:08:02.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fafec056d20 0x7fafec0591e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:02.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb01009ca90 0x7fb010159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:02.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb010065190 0x7fb0101450a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:02.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 --2- 172.21.15.187:0/2639932157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb010065f90 0x7fb0101596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:02.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 >> 172.21.15.187:0/2639932157 conn(0x7fb0100bab30 msgr2=0x7fb0101412e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:02.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 shutdown_connections 2020-11-20T12:08:02.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:02.569+0000 7fb0153c4700 1 -- 172.21.15.187:0/2639932157 wait complete. 2020-11-20T12:08:02.577 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:08:02.595 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:08:02.858 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:08:02.859 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:08:02.859 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 50, waiting (timeout=60)... 2020-11-20T12:08:07.860 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:08.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.107+0000 7f3f01b89700 1 Processor -- start 2020-11-20T12:08:08.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.107+0000 7f3f01b89700 1 -- start start 2020-11-20T12:08:08.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.109+0000 7f3f01b89700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3efc1484d0 0x7f3efc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.109+0000 7f3f01b89700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc066f70 0x7f3efc0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.109+0000 7f3f01b89700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3efc0b1300 con 0x7f3efc0622c0 2020-11-20T12:08:08.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.109+0000 7f3f01b89700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3efc067730 con 0x7f3efc1484d0 2020-11-20T12:08:08.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.109+0000 7f3f01b89700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc0b15a0 con 0x7f3efc066f70 2020-11-20T12:08:08.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.109+0000 7f3f00b87700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3efc1484d0 0x7f3efc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.109+0000 7f3f00b87700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3efc1484d0 0x7f3efc061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43404/0 (socket says 172.21.15.187:43404) 2020-11-20T12:08:08.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.109+0000 7f3f00b87700 1 -- 172.21.15.187:0/373062497 learned_addr learned my addr 172.21.15.187:0/373062497 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:08.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.110+0000 7f3f01388700 1 --2- 172.21.15.187:0/373062497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc066f70 0x7f3efc0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.110+0000 7f3f00b87700 1 -- 172.21.15.187:0/373062497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc066f70 msgr2=0x7f3efc0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.110+0000 7f3f00b87700 1 --2- 172.21.15.187:0/373062497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc066f70 0x7f3efc0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.110+0000 7f3f00b87700 1 -- 172.21.15.187:0/373062497 >> v1:172.21.15.5:6789/0 conn(0x7f3efc0622c0 legacy=0x7f3efc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.110+0000 7f3f00b87700 1 -- 172.21.15.187:0/373062497 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc0b1840 con 0x7f3efc1484d0 2020-11-20T12:08:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.110+0000 7f3f00b87700 1 --2- 172.21.15.187:0/373062497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc061dd0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f3ef8009d50 tx=0x7f3ef8009290).ready entity=mon.1 client_cookie=27957574ec4f7eb9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3ef37fe700 1 -- 172.21.15.187:0/373062497 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef800bc40 con 0x7f3efc1484d0 2020-11-20T12:08:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3ef37fe700 1 -- 172.21.15.187:0/373062497 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef800bda0 con 0x7f3efc1484d0 2020-11-20T12:08:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3ef37fe700 1 -- 172.21.15.187:0/373062497 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8011610 con 0x7f3efc1484d0 2020-11-20T12:08:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3f01b89700 1 -- 172.21.15.187:0/373062497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 msgr2=0x7f3efc061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3f01b89700 1 --2- 172.21.15.187:0/373062497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc061dd0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f3ef8009d50 tx=0x7f3ef8009290).stop 2020-11-20T12:08:08.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3f01b89700 1 -- 172.21.15.187:0/373062497 shutdown_connections 2020-11-20T12:08:08.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3f01b89700 1 --2- 172.21.15.187:0/373062497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc066f70 0x7f3efc0b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3f01b89700 1 --2- 172.21.15.187:0/373062497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc061dd0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3f01b89700 1 -- 172.21.15.187:0/373062497 >> 172.21.15.187:0/373062497 conn(0x7f3efc0bab30 msgr2=0x7f3efc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:08.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3f01b89700 1 -- 172.21.15.187:0/373062497 shutdown_connections 2020-11-20T12:08:08.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.111+0000 7f3f01b89700 1 -- 172.21.15.187:0/373062497 wait complete. 2020-11-20T12:08:08.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.113+0000 7f3f01b89700 1 Processor -- start 2020-11-20T12:08:08.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.113+0000 7f3f01b89700 1 -- start start 2020-11-20T12:08:08.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.114+0000 7f3f01b89700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc0622c0 0x7f3efc1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.114+0000 7f3f01b89700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3efc066f70 0x7f3efc0b9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.114+0000 7f3f01b89700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc13d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.114+0000 7f3f01b89700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc158900 con 0x7f3efc066f70 2020-11-20T12:08:08.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.114+0000 7f3f01b89700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc158ba0 con 0x7f3efc1484d0 2020-11-20T12:08:08.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.114+0000 7f3f01b89700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc1585b0 con 0x7f3efc0622c0 2020-11-20T12:08:08.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.114+0000 7f3f01388700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc13d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.114+0000 7f3f01388700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc13d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43414/0 (socket says 172.21.15.187:43414) 2020-11-20T12:08:08.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3f01388700 1 -- 172.21.15.187:0/3149374982 learned_addr learned my addr 172.21.15.187:0/3149374982 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:08.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3f00b87700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc0622c0 0x7f3efc1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3f01388700 1 -- 172.21.15.187:0/3149374982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc0622c0 msgr2=0x7f3efc1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3f01388700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc0622c0 0x7f3efc1580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3ef3fff700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3efc066f70 0x7f3efc0b9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3f01388700 1 -- 172.21.15.187:0/3149374982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3efc066f70 msgr2=0x7f3efc0b9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3f01388700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3efc066f70 0x7f3efc0b9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3f01388700 1 -- 172.21.15.187:0/3149374982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc0ba190 con 0x7f3efc1484d0 2020-11-20T12:08:08.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3f01388700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc13d2f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f3ef400d830 tx=0x7f3ef4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.115+0000 7f3ef0ff9700 1 -- 172.21.15.187:0/3149374982 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40096b0 con 0x7f3efc1484d0 2020-11-20T12:08:08.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.116+0000 7f3ef0ff9700 1 -- 172.21.15.187:0/3149374982 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef4009810 con 0x7f3efc1484d0 2020-11-20T12:08:08.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.116+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3efc13cb00 con 0x7f3efc1484d0 2020-11-20T12:08:08.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.116+0000 7f3ef0ff9700 1 -- 172.21.15.187:0/3149374982 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4009bb0 con 0x7f3efc1484d0 2020-11-20T12:08:08.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.116+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3efc09fb70 con 0x7f3efc1484d0 2020-11-20T12:08:08.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.117+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3efc158770 con 0x7f3efc1484d0 2020-11-20T12:08:08.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.117+0000 7f3ef0ff9700 1 -- 172.21.15.187:0/3149374982 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3ef4013480 con 0x7f3efc1484d0 2020-11-20T12:08:08.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.118+0000 7f3ef0ff9700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ed4056d20 0x7f3ed40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.118+0000 7f3ef0ff9700 1 -- 172.21.15.187:0/3149374982 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ef406ff50 con 0x7f3efc1484d0 2020-11-20T12:08:08.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.118+0000 7f3f00b87700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ed4056d20 0x7f3ed40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.119+0000 7f3f00b87700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ed4056d20 0x7f3ed40591e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3ef8009fd0 tx=0x7f3ef8016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.123+0000 7f3ef0ff9700 1 -- 172.21.15.187:0/3149374982 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3ef4045340 con 0x7f3efc1484d0 2020-11-20T12:08:08.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.371+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3efc061a20 con 0x7f3efc1484d0 2020-11-20T12:08:08.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.372+0000 7f3ef0ff9700 1 -- 172.21.15.187:0/3149374982 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f3ef4048960 con 0x7f3efc1484d0 2020-11-20T12:08:08.376 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:08.377 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:08:08.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ed4056d20 msgr2=0x7f3ed40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ed4056d20 0x7f3ed40591e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3ef8009fd0 tx=0x7f3ef8016000).stop 2020-11-20T12:08:08.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 msgr2=0x7f3efc13d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc13d2f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f3ef400d830 tx=0x7f3ef4008040).stop 2020-11-20T12:08:08.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 shutdown_connections 2020-11-20T12:08:08.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ed4056d20 0x7f3ed40591e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3efc1484d0 0x7f3efc13d2f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3efc0622c0 0x7f3efc1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 --2- 172.21.15.187:0/3149374982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3efc066f70 0x7f3efc0b9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 >> 172.21.15.187:0/3149374982 conn(0x7f3efc0bab30 msgr2=0x7f3efc0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:08.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 shutdown_connections 2020-11-20T12:08:08.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.375+0000 7f3f01b89700 1 -- 172.21.15.187:0/3149374982 wait complete. 2020-11-20T12:08:08.382 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:08:08.403 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:08:08.681 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":50}} 2020-11-20T12:08:08.681 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 50 } } 2020-11-20T12:08:08.682 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:08.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.924+0000 7f95d71a7700 1 Processor -- start 2020-11-20T12:08:08.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.924+0000 7f95d71a7700 1 -- start start 2020-11-20T12:08:08.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.925+0000 7f95d71a7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f95d01484d0 0x7f95d0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.925+0000 7f95d71a7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95d00622c0 0x7f95d00b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.925+0000 7f95d71a7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d0066f70 0x7f95d00b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.925+0000 7f95d71a7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d0067730 con 0x7f95d01484d0 2020-11-20T12:08:08.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.925+0000 7f95d71a7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d00b1300 con 0x7f95d00622c0 2020-11-20T12:08:08.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.925+0000 7f95d71a7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d00b15a0 con 0x7f95d0066f70 2020-11-20T12:08:08.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.926+0000 7f95d59a4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95d00622c0 0x7f95d00b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.926+0000 7f95d59a4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95d00622c0 0x7f95d00b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43420/0 (socket says 172.21.15.187:43420) 2020-11-20T12:08:08.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.926+0000 7f95d59a4700 1 -- 172.21.15.187:0/122668675 learned_addr learned my addr 172.21.15.187:0/122668675 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:08.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.926+0000 7f95d69a6700 1 --2- 172.21.15.187:0/122668675 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d0066f70 0x7f95d00b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.926+0000 7f95d61a5700 1 --2- 172.21.15.187:0/122668675 >> v2:172.21.15.5:3300/0 conn(0x7f95d01484d0 0x7f95d0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.927+0000 7f95d59a4700 1 -- 172.21.15.187:0/122668675 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d0066f70 msgr2=0x7f95d00b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:08.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.927+0000 7f95d59a4700 1 --2- 172.21.15.187:0/122668675 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d0066f70 0x7f95d00b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.927+0000 7f95d59a4700 1 -- 172.21.15.187:0/122668675 >> v2:172.21.15.5:3300/0 conn(0x7f95d01484d0 msgr2=0x7f95d0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:08.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.927+0000 7f95d59a4700 1 --2- 172.21.15.187:0/122668675 >> v2:172.21.15.5:3300/0 conn(0x7f95d01484d0 0x7f95d0061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.927+0000 7f95d59a4700 1 -- 172.21.15.187:0/122668675 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00b1840 con 0x7f95d00622c0 2020-11-20T12:08:08.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.927+0000 7f95d59a4700 1 --2- 172.21.15.187:0/122668675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d00622c0 0x7f95d00b5640 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f95d0068600 tx=0x7f95c0006eb0).ready entity=mon.1 client_cookie=cb8f0f740c4634d2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.927+0000 7f95d51a3700 1 -- 172.21.15.187:0/122668675 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c00098d0 con 0x7f95d00622c0 2020-11-20T12:08:08.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.927+0000 7f95d51a3700 1 -- 172.21.15.187:0/122668675 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c0009a30 con 0x7f95d00622c0 2020-11-20T12:08:08.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.928+0000 7f95d51a3700 1 -- 172.21.15.187:0/122668675 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c0009db0 con 0x7f95d00622c0 2020-11-20T12:08:08.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.928+0000 7f95d71a7700 1 -- 172.21.15.187:0/122668675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d00622c0 msgr2=0x7f95d00b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.928+0000 7f95d71a7700 1 --2- 172.21.15.187:0/122668675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d00622c0 0x7f95d00b5640 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f95d0068600 tx=0x7f95c0006eb0).stop 2020-11-20T12:08:08.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.928+0000 7f95d71a7700 1 -- 172.21.15.187:0/122668675 shutdown_connections 2020-11-20T12:08:08.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.928+0000 7f95d71a7700 1 --2- 172.21.15.187:0/122668675 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d0066f70 0x7f95d00b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.928+0000 7f95d71a7700 1 --2- 172.21.15.187:0/122668675 >> v2:172.21.15.5:3300/0 conn(0x7f95d01484d0 0x7f95d0061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.928+0000 7f95d71a7700 1 --2- 172.21.15.187:0/122668675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d00622c0 0x7f95d00b5640 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.928+0000 7f95d71a7700 1 -- 172.21.15.187:0/122668675 >> 172.21.15.187:0/122668675 conn(0x7f95d00bab30 msgr2=0x7f95d00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:08.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.929+0000 7f95d71a7700 1 -- 172.21.15.187:0/122668675 shutdown_connections 2020-11-20T12:08:08.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.929+0000 7f95d71a7700 1 -- 172.21.15.187:0/122668675 wait complete. 2020-11-20T12:08:08.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.930+0000 7f95d71a7700 1 Processor -- start 2020-11-20T12:08:08.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.930+0000 7f95d71a7700 1 -- start start 2020-11-20T12:08:08.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.931+0000 7f95d71a7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d00622c0 0x7f95d00a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.931+0000 7f95d71a7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95d0066f70 0x7f95d00b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.931+0000 7f95d71a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d01484d0 0x7f95d0137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.931+0000 7f95d71a7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d009fa60 con 0x7f95d0066f70 2020-11-20T12:08:08.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.931+0000 7f95d71a7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d009fc50 con 0x7f95d01484d0 2020-11-20T12:08:08.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.931+0000 7f95d71a7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d009fef0 con 0x7f95d00622c0 2020-11-20T12:08:08.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d69a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d01484d0 0x7f95d0137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d69a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d01484d0 0x7f95d0137ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43428/0 (socket says 172.21.15.187:43428) 2020-11-20T12:08:08.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d69a6700 1 -- 172.21.15.187:0/253700092 learned_addr learned my addr 172.21.15.187:0/253700092 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:08.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d61a5700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d00622c0 0x7f95d00a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d59a4700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95d0066f70 0x7f95d00b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d69a6700 1 -- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d00622c0 msgr2=0x7f95d00a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d69a6700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d00622c0 0x7f95d00a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d69a6700 1 -- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95d0066f70 msgr2=0x7f95d00b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d69a6700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95d0066f70 0x7f95d00b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.932+0000 7f95d69a6700 1 -- 172.21.15.187:0/253700092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00a0190 con 0x7f95d01484d0 2020-11-20T12:08:08.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.933+0000 7f95d59a4700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95d0066f70 0x7f95d00b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:08:08.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.933+0000 7f95d69a6700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d01484d0 0x7f95d0137ce0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f95c800eb40 tx=0x7f95c8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.933+0000 7f95be7fc700 1 -- 172.21.15.187:0/253700092 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c800c740 con 0x7f95d01484d0 2020-11-20T12:08:08.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.933+0000 7f95be7fc700 1 -- 172.21.15.187:0/253700092 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c800c8a0 con 0x7f95d01484d0 2020-11-20T12:08:08.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.933+0000 7f95be7fc700 1 -- 172.21.15.187:0/253700092 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c800cc40 con 0x7f95d01484d0 2020-11-20T12:08:08.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.933+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d00a0b20 con 0x7f95d01484d0 2020-11-20T12:08:08.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.933+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d009a5f0 con 0x7f95d01484d0 2020-11-20T12:08:08.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.934+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d00a0320 con 0x7f95d01484d0 2020-11-20T12:08:08.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.935+0000 7f95be7fc700 1 -- 172.21.15.187:0/253700092 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f95c8013640 con 0x7f95d01484d0 2020-11-20T12:08:08.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.936+0000 7f95be7fc700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95a8056d20 0x7f95a80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.936+0000 7f95be7fc700 1 -- 172.21.15.187:0/253700092 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f95c806fc60 con 0x7f95d01484d0 2020-11-20T12:08:08.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.936+0000 7f95d61a5700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95a8056d20 0x7f95a80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.937+0000 7f95d61a5700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95a8056d20 0x7f95a80591e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f95cc00ba00 tx=0x7f95cc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:08.940+0000 7f95be7fc700 1 -- 172.21.15.187:0/253700092 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f95c8045030 con 0x7f95d01484d0 2020-11-20T12:08:09.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.183+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95d0061a20 con 0x7f95d01484d0 2020-11-20T12:08:09.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.184+0000 7f95be7fc700 1 -- 172.21.15.187:0/253700092 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f95c8048650 con 0x7f95d01484d0 2020-11-20T12:08:09.189 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:09.189 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:08:09.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.186+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95a8056d20 msgr2=0x7f95a80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:09.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.186+0000 7f95d71a7700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95a8056d20 0x7f95a80591e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f95cc00ba00 tx=0x7f95cc008040).stop 2020-11-20T12:08:09.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.186+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d01484d0 msgr2=0x7f95d0137ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:09.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.186+0000 7f95d71a7700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d01484d0 0x7f95d0137ce0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f95c800eb40 tx=0x7f95c8008040).stop 2020-11-20T12:08:09.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.187+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 shutdown_connections 2020-11-20T12:08:09.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.187+0000 7f95d71a7700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95a8056d20 0x7f95a80591e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:09.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.187+0000 7f95d71a7700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95d01484d0 0x7f95d0137ce0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:09.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.187+0000 7f95d71a7700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95d00622c0 0x7f95d00a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:09.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.187+0000 7f95d71a7700 1 --2- 172.21.15.187:0/253700092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95d0066f70 0x7f95d00b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:09.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.187+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 >> 172.21.15.187:0/253700092 conn(0x7f95d00bab30 msgr2=0x7f95d00b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:09.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.187+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 shutdown_connections 2020-11-20T12:08:09.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:09.187+0000 7f95d71a7700 1 -- 172.21.15.187:0/253700092 wait complete. 2020-11-20T12:08:09.194 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:08:09.213 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2020-11-20T12:08:09.481 INFO:teuthology.orchestra.run.smithi187.stdout:{"purge_queue":{"pq_executed":50}} 2020-11-20T12:08:09.481 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 50 } } 2020-11-20T12:08:09.482 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:08:09.482 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:08:09.483 INFO:teuthology.orchestra.run.smithi187:> path = os.path.join("/home/ubuntu/cephtest/mnt.0", "subdir4") 2020-11-20T12:08:09.483 INFO:teuthology.orchestra.run.smithi187:> os.mkdir(path) 2020-11-20T12:08:09.483 INFO:teuthology.orchestra.run.smithi187:> for n in range(50): 2020-11-20T12:08:09.484 INFO:teuthology.orchestra.run.smithi187:> fpath = os.path.join(path, "%s" % n) 2020-11-20T12:08:09.484 INFO:teuthology.orchestra.run.smithi187:> with open(fpath, '"'"'w'"'"') as f: 2020-11-20T12:08:09.485 INFO:teuthology.orchestra.run.smithi187:> f.write(str(n)) 2020-11-20T12:08:09.485 INFO:teuthology.orchestra.run.smithi187:> os.unlink(fpath) 2020-11-20T12:08:09.486 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:08:09.861 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:08:09.861 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:08:09.980 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:08:09.998 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:08:10.111 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:08:10.125 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:08:10.126 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:08:10.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:08:10.127 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:08:10.424 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:08:10.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:10.425 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:08:10.473 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:08:10.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.714+0000 7f77eb59e700 1 Processor -- start 2020-11-20T12:08:10.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.714+0000 7f77eb59e700 1 -- start start 2020-11-20T12:08:10.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.716+0000 7f77eb59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f77ec1484d0 0x7f77ec061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.716+0000 7f77eb59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ec0b1300 con 0x7f77ec0622c0 2020-11-20T12:08:10.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.716+0000 7f77eb59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ec0b15a0 con 0x7f77ec066f70 2020-11-20T12:08:10.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.716+0000 7f77eb59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ec067730 con 0x7f77ec1484d0 2020-11-20T12:08:10.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.716+0000 7f77ea59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f77ec1484d0 0x7f77ec061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.716+0000 7f77ead9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f77ec066f70 0x7f77ec0b5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40042/0 (socket says 172.21.15.187:40042) 2020-11-20T12:08:10.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.716+0000 7f77ead9d700 1 -- 172.21.15.187:0/278997342 learned_addr learned my addr 172.21.15.187:0/278997342 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:10.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.717+0000 7f77ea59c700 1 -- 172.21.15.187:0/278997342 >> v1:172.21.15.187:6789/0 conn(0x7f77ec066f70 legacy=0x7f77ec0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.717+0000 7f77ea59c700 1 -- 172.21.15.187:0/278997342 >> v1:172.21.15.5:6789/0 conn(0x7f77ec0622c0 legacy=0x7f77ec0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:10.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.717+0000 7f77ea59c700 1 -- 172.21.15.187:0/278997342 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec0b1840 con 0x7f77ec1484d0 2020-11-20T12:08:10.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.717+0000 7f77ea59c700 1 --2- 172.21.15.187:0/278997342 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec1484d0 0x7f77ec061dd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f77dc009c90 tx=0x7f77dc0091d0).ready entity=mon.1 client_cookie=3272d661d8e6d560 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:10.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.717+0000 7f77e9d9b700 1 -- 172.21.15.187:0/278997342 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00bb80 con 0x7f77ec1484d0 2020-11-20T12:08:10.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.717+0000 7f77e9d9b700 1 -- 172.21.15.187:0/278997342 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc00bce0 con 0x7f77ec1484d0 2020-11-20T12:08:10.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.717+0000 7f77e9d9b700 1 -- 172.21.15.187:0/278997342 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc011610 con 0x7f77ec1484d0 2020-11-20T12:08:10.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.718+0000 7f77eb59e700 1 -- 172.21.15.187:0/278997342 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec1484d0 msgr2=0x7f77ec061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.718+0000 7f77eb59e700 1 --2- 172.21.15.187:0/278997342 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec1484d0 0x7f77ec061dd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f77dc009c90 tx=0x7f77dc0091d0).stop 2020-11-20T12:08:10.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.718+0000 7f77eb59e700 1 -- 172.21.15.187:0/278997342 shutdown_connections 2020-11-20T12:08:10.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.718+0000 7f77eb59e700 1 --2- 172.21.15.187:0/278997342 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec1484d0 0x7f77ec061dd0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.718+0000 7f77eb59e700 1 -- 172.21.15.187:0/278997342 >> 172.21.15.187:0/278997342 conn(0x7f77ec0bab30 msgr2=0x7f77ec0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:10.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.718+0000 7f77eb59e700 1 -- 172.21.15.187:0/278997342 shutdown_connections 2020-11-20T12:08:10.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.718+0000 7f77eb59e700 1 -- 172.21.15.187:0/278997342 wait complete. 2020-11-20T12:08:10.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.720+0000 7f77eb59e700 1 Processor -- start 2020-11-20T12:08:10.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.720+0000 7f77eb59e700 1 -- start start 2020-11-20T12:08:10.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77eb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f77ec0622c0 0x7f77ec14af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77eb59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec066f70 0x7f77ec142ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77eb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77ec1484d0 0x7f77ec1434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77eb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec09a670 con 0x7f77ec1484d0 2020-11-20T12:08:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77eb59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec149f60 con 0x7f77ec066f70 2020-11-20T12:08:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77eb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec14a200 con 0x7f77ec0622c0 2020-11-20T12:08:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77e1d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec066f70 0x7f77ec142ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77e1d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec066f70 0x7f77ec142ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43440/0 (socket says 172.21.15.187:43440) 2020-11-20T12:08:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77e1d9b700 1 -- 172.21.15.187:0/3587583366 learned_addr learned my addr 172.21.15.187:0/3587583366 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77e1d9b700 1 -- 172.21.15.187:0/3587583366 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f77ec0622c0 msgr2=0x7f77ec14af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77e1d9b700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f77ec0622c0 0x7f77ec14af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77e1d9b700 1 -- 172.21.15.187:0/3587583366 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77ec1484d0 msgr2=0x7f77ec1434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77e1d9b700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77ec1484d0 0x7f77ec1434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.721+0000 7f77e1d9b700 1 -- 172.21.15.187:0/3587583366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec0b61d0 con 0x7f77ec066f70 2020-11-20T12:08:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.722+0000 7f77e1d9b700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec066f70 0x7f77ec142ff0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f77d400c790 tx=0x7f77d4007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.722+0000 7f77e2ffd700 1 -- 172.21.15.187:0/3587583366 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d400e560 con 0x7f77ec066f70 2020-11-20T12:08:10.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.722+0000 7f77e2ffd700 1 -- 172.21.15.187:0/3587583366 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d400e700 con 0x7f77ec066f70 2020-11-20T12:08:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.722+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec14b420 con 0x7f77ec066f70 2020-11-20T12:08:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.722+0000 7f77e2ffd700 1 -- 172.21.15.187:0/3587583366 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d400eac0 con 0x7f77ec066f70 2020-11-20T12:08:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.722+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec149c10 con 0x7f77ec066f70 2020-11-20T12:08:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.723+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec061a20 con 0x7f77ec066f70 2020-11-20T12:08:10.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.724+0000 7f77e2ffd700 1 -- 172.21.15.187:0/3587583366 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f77d4019030 con 0x7f77ec066f70 2020-11-20T12:08:10.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.724+0000 7f77e2ffd700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f77c8056d20 0x7f77c80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.724+0000 7f77e2ffd700 1 -- 172.21.15.187:0/3587583366 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f77d406f430 con 0x7f77ec066f70 2020-11-20T12:08:10.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.725+0000 7f77ea59c700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f77c8056d20 0x7f77c80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.725+0000 7f77ea59c700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f77c8056d20 0x7f77c80591e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f77dc009210 tx=0x7f77dc016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:10.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.728+0000 7f77e2ffd700 1 -- 172.21.15.187:0/3587583366 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f77d4043fd0 con 0x7f77ec066f70 2020-11-20T12:08:10.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.917+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f77ec09a7b0 con 0x7f77ec066f70 2020-11-20T12:08:10.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.928+0000 7f77e2ffd700 1 -- 172.21.15.187:0/3587583366 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7f77d4044820 con 0x7f77ec066f70 2020-11-20T12:08:10.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.930+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f77c8056d20 msgr2=0x7f77c80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.930+0000 7f77eb59e700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f77c8056d20 0x7f77c80591e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f77dc009210 tx=0x7f77dc016000).stop 2020-11-20T12:08:10.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec066f70 msgr2=0x7f77ec142ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec066f70 0x7f77ec142ff0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f77d400c790 tx=0x7f77d4007c00).stop 2020-11-20T12:08:10.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 shutdown_connections 2020-11-20T12:08:10.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f77c8056d20 0x7f77c80591e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77ec1484d0 0x7f77ec1434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:10.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f77ec0622c0 0x7f77ec14af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:10.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 --2- 172.21.15.187:0/3587583366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77ec066f70 0x7f77ec142ff0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 >> 172.21.15.187:0/3587583366 conn(0x7f77ec0bab30 msgr2=0x7f77ec1450b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:10.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 shutdown_connections 2020-11-20T12:08:10.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:10.931+0000 7f77eb59e700 1 -- 172.21.15.187:0/3587583366 wait complete. 2020-11-20T12:08:10.958 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2020-11-20T12:08:11.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.215+0000 7f33b5eaf700 1 Processor -- start 2020-11-20T12:08:11.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.215+0000 7f33b5eaf700 1 -- start start 2020-11-20T12:08:11.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.217+0000 7f33b5eaf700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33b009f560 0x7f33b0153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:11.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.217+0000 7f33b5eaf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b014d110 con 0x7f33b00bf800 2020-11-20T12:08:11.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.217+0000 7f33b5eaf700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b014cbd0 con 0x7f33b00bfbe0 2020-11-20T12:08:11.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.217+0000 7f33b5eaf700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b014c9d0 con 0x7f33b009f560 2020-11-20T12:08:11.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.217+0000 7f33b4ead700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f33b00bf800 0x7f33b014c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33562/0 (socket says 172.21.15.187:33562) 2020-11-20T12:08:11.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.218+0000 7f33b4ead700 1 -- 172.21.15.187:0/3908583132 learned_addr learned my addr 172.21.15.187:0/3908583132 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:11.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.217+0000 7f33affff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f33b00bfbe0 0x7f33b01480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40054/0 (socket says 172.21.15.187:40054) 2020-11-20T12:08:11.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.218+0000 7f33b56ae700 1 --2- 172.21.15.187:0/3908583132 >> v2:172.21.15.187:3300/0 conn(0x7f33b009f560 0x7f33b0153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:11.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.218+0000 7f33af7fe700 1 -- 172.21.15.187:0/3908583132 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600489134 0 0) 0x7f33b014d110 con 0x7f33b00bf800 2020-11-20T12:08:11.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.218+0000 7f33af7fe700 1 -- 172.21.15.187:0/3908583132 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3394003240 con 0x7f33b00bf800 2020-11-20T12:08:11.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.218+0000 7f33af7fe700 1 -- 172.21.15.187:0/3908583132 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388658307 0 0) 0x7f33b014cbd0 con 0x7f33b00bfbe0 2020-11-20T12:08:11.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.219+0000 7f33af7fe700 1 -- 172.21.15.187:0/3908583132 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b014d110 con 0x7f33b00bfbe0 2020-11-20T12:08:11.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.219+0000 7f33b56ae700 1 -- 172.21.15.187:0/3908583132 >> v1:172.21.15.187:6789/0 conn(0x7f33b00bfbe0 legacy=0x7f33b01480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:11.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.219+0000 7f33b56ae700 1 -- 172.21.15.187:0/3908583132 >> v1:172.21.15.5:6789/0 conn(0x7f33b00bf800 legacy=0x7f33b014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:11.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.219+0000 7f33b56ae700 1 -- 172.21.15.187:0/3908583132 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b014ce70 con 0x7f33b009f560 2020-11-20T12:08:11.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.219+0000 7f33b56ae700 1 --2- 172.21.15.187:0/3908583132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b009f560 0x7f33b0153580 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f33a8009c90 tx=0x7f33a80091d0).ready entity=mon.1 client_cookie=7b887b328c81c50 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:11.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.219+0000 7f33af7fe700 1 -- 172.21.15.187:0/3908583132 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a800ba50 con 0x7f33b009f560 2020-11-20T12:08:11.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.219+0000 7f33af7fe700 1 -- 172.21.15.187:0/3908583132 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a800bbb0 con 0x7f33b009f560 2020-11-20T12:08:11.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.219+0000 7f33af7fe700 1 -- 172.21.15.187:0/3908583132 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8011400 con 0x7f33b009f560 2020-11-20T12:08:11.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.220+0000 7f33b5eaf700 1 -- 172.21.15.187:0/3908583132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b009f560 msgr2=0x7f33b0153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:11.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.220+0000 7f33b5eaf700 1 --2- 172.21.15.187:0/3908583132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b009f560 0x7f33b0153580 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f33a8009c90 tx=0x7f33a80091d0).stop 2020-11-20T12:08:11.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.220+0000 7f33b5eaf700 1 -- 172.21.15.187:0/3908583132 shutdown_connections 2020-11-20T12:08:11.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.220+0000 7f33b5eaf700 1 --2- 172.21.15.187:0/3908583132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b009f560 0x7f33b0153580 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:11.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.220+0000 7f33b5eaf700 1 -- 172.21.15.187:0/3908583132 >> 172.21.15.187:0/3908583132 conn(0x7f33b0068370 msgr2=0x7f33b00bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:11.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.220+0000 7f33b5eaf700 1 -- 172.21.15.187:0/3908583132 shutdown_connections 2020-11-20T12:08:11.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.220+0000 7f33b5eaf700 1 -- 172.21.15.187:0/3908583132 wait complete. 2020-11-20T12:08:11.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.222+0000 7f33b5eaf700 1 Processor -- start 2020-11-20T12:08:11.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.222+0000 7f33b5eaf700 1 -- start start 2020-11-20T12:08:11.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.223+0000 7f33b5eaf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33b009f560 0x7f33b01577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:11.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.223+0000 7f33b5eaf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33b00bf800 0x7f33b0157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:11.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.223+0000 7f33b5eaf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b00b4780 0x7f33b0158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:11.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.223+0000 7f33b5eaf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b00b6cb0 con 0x7f33b00bf800 2020-11-20T12:08:11.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.223+0000 7f33b5eaf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b014af20 con 0x7f33b00b4780 2020-11-20T12:08:11.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.223+0000 7f33b5eaf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b00b4d00 con 0x7f33b009f560 2020-11-20T12:08:11.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b00b4780 0x7f33b0158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:11.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b00b4780 0x7f33b0158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43454/0 (socket says 172.21.15.187:43454) 2020-11-20T12:08:11.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 -- 172.21.15.187:0/1532590836 learned_addr learned my addr 172.21.15.187:0/1532590836 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:11.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33affff700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33b00bf800 0x7f33b0157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:11.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b4ead700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33b009f560 0x7f33b01577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:11.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 -- 172.21.15.187:0/1532590836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33b009f560 msgr2=0x7f33b01577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:11.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33b009f560 0x7f33b01577b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:11.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 -- 172.21.15.187:0/1532590836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33b00bf800 msgr2=0x7f33b0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:11.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33b00bf800 0x7f33b0157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:11.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 -- 172.21.15.187:0/1532590836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b00b6ee0 con 0x7f33b00b4780 2020-11-20T12:08:11.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.224+0000 7f33b56ae700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b00b4780 0x7f33b0158470 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f33a8006f30 tx=0x7f33a8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:11.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.225+0000 7f33acff9700 1 -- 172.21.15.187:0/1532590836 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8011400 con 0x7f33b00b4780 2020-11-20T12:08:11.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.225+0000 7f33acff9700 1 -- 172.21.15.187:0/1532590836 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a8010c00 con 0x7f33b00b4780 2020-11-20T12:08:11.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.225+0000 7f33acff9700 1 -- 172.21.15.187:0/1532590836 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a800ba50 con 0x7f33b00b4780 2020-11-20T12:08:11.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.225+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b00b7380 con 0x7f33b00b4780 2020-11-20T12:08:11.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.225+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b0158ba0 con 0x7f33b00b4780 2020-11-20T12:08:11.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.226+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b0080050 con 0x7f33b00b4780 2020-11-20T12:08:11.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.227+0000 7f33acff9700 1 -- 172.21.15.187:0/1532590836 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f33a8024030 con 0x7f33b00b4780 2020-11-20T12:08:11.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.227+0000 7f33acff9700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3388056d20 0x7f33880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:11.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.227+0000 7f33acff9700 1 -- 172.21.15.187:0/1532590836 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f33a8071d00 con 0x7f33b00b4780 2020-11-20T12:08:11.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.228+0000 7f33b4ead700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3388056d20 0x7f33880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:11.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.228+0000 7f33b4ead700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3388056d20 0x7f33880591e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f33a40055a0 tx=0x7f33a4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:11.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.231+0000 7f33acff9700 1 -- 172.21.15.187:0/1532590836 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f33a8075030 con 0x7f33b00b4780 2020-11-20T12:08:11.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:11.511+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f33b00446d0 con 0x7f33b00b4780 2020-11-20T12:08:12.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.129+0000 7f33acff9700 1 -- 172.21.15.187:0/1532590836 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v174) v1 ==== 133+0+0 (secure 0 0 0) 0x7f33a8022030 con 0x7f33b00b4780 2020-11-20T12:08:12.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3388056d20 msgr2=0x7f33880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:12.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3388056d20 0x7f33880591e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f33a40055a0 tx=0x7f33a4009040).stop 2020-11-20T12:08:12.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b00b4780 msgr2=0x7f33b0158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:12.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b00b4780 0x7f33b0158470 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f33a8006f30 tx=0x7f33a8016000).stop 2020-11-20T12:08:12.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 shutdown_connections 2020-11-20T12:08:12.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3388056d20 0x7f33880591e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33b00b4780 0x7f33b0158470 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33b009f560 0x7f33b01577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 --2- 172.21.15.187:0/1532590836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33b00bf800 0x7f33b0157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 >> 172.21.15.187:0/1532590836 conn(0x7f33b0068370 msgr2=0x7f33b00bb600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:12.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 shutdown_connections 2020-11-20T12:08:12.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.133+0000 7f33b5eaf700 1 -- 172.21.15.187:0/1532590836 wait complete. 2020-11-20T12:08:12.165 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:08:12.166 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:08:12.167 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2020-11-20T12:08:12.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.403+0000 7f664359e700 1 Processor -- start 2020-11-20T12:08:12.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.404+0000 7f664359e700 1 -- start start 2020-11-20T12:08:12.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.405+0000 7f664359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:12.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.405+0000 7f664359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f66440b5f50 0x7f66440b6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:12.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.405+0000 7f664359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6644144b80 con 0x7f66441482d0 2020-11-20T12:08:12.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.405+0000 7f664359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66441489e0 con 0x7f66440b5f50 2020-11-20T12:08:12.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.405+0000 7f664359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6644148c80 con 0x7f66440b57b0 2020-11-20T12:08:12.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.405+0000 7f6642d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f66440b5f50 0x7f66440b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:12.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.405+0000 7f6641d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f66441482d0 0x7f6644062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40066/0 (socket says 172.21.15.187:40066) 2020-11-20T12:08:12.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.405+0000 7f6641d9b700 1 -- 172.21.15.187:0/4152555344 learned_addr learned my addr 172.21.15.187:0/4152555344 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:12.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.406+0000 7f6642d9d700 1 -- 172.21.15.187:0/4152555344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 msgr2=0x7f6644068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:12.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.406+0000 7f664259c700 1 --2- 172.21.15.187:0/4152555344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:12.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.406+0000 7f6642d9d700 1 --2- 172.21.15.187:0/4152555344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.406+0000 7f6642d9d700 1 -- 172.21.15.187:0/4152555344 >> v1:172.21.15.187:6789/0 conn(0x7f66441482d0 legacy=0x7f6644062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:12.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.406+0000 7f6642d9d700 1 -- 172.21.15.187:0/4152555344 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6644148f20 con 0x7f66440b5f50 2020-11-20T12:08:12.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.406+0000 7f664259c700 1 --2- 172.21.15.187:0/4152555344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:12.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.406+0000 7f6642d9d700 1 --2- 172.21.15.187:0/4152555344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f66440b6400 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f663c009c90 tx=0x7f663c0091d0).ready entity=mon.1 client_cookie=4e506a5b49ace7e1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:12.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.407+0000 7f664159a700 1 -- 172.21.15.187:0/4152555344 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c00bbc0 con 0x7f66440b5f50 2020-11-20T12:08:12.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.407+0000 7f664159a700 1 -- 172.21.15.187:0/4152555344 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c00bd20 con 0x7f66440b5f50 2020-11-20T12:08:12.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.407+0000 7f664159a700 1 -- 172.21.15.187:0/4152555344 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c011610 con 0x7f66440b5f50 2020-11-20T12:08:12.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.408+0000 7f664359e700 1 -- 172.21.15.187:0/4152555344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 msgr2=0x7f66440b6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:12.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.408+0000 7f664359e700 1 --2- 172.21.15.187:0/4152555344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f66440b6400 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f663c009c90 tx=0x7f663c0091d0).stop 2020-11-20T12:08:12.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.408+0000 7f664359e700 1 -- 172.21.15.187:0/4152555344 shutdown_connections 2020-11-20T12:08:12.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.408+0000 7f664359e700 1 --2- 172.21.15.187:0/4152555344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f66440b6400 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.408+0000 7f664359e700 1 --2- 172.21.15.187:0/4152555344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.408+0000 7f664359e700 1 -- 172.21.15.187:0/4152555344 >> 172.21.15.187:0/4152555344 conn(0x7f66440bab30 msgr2=0x7f66440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:12.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.408+0000 7f664359e700 1 -- 172.21.15.187:0/4152555344 shutdown_connections 2020-11-20T12:08:12.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.408+0000 7f664359e700 1 -- 172.21.15.187:0/4152555344 wait complete. 2020-11-20T12:08:12.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.409+0000 7f664359e700 1 Processor -- start 2020-11-20T12:08:12.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.409+0000 7f664359e700 1 -- start start 2020-11-20T12:08:12.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.410+0000 7f664359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:12.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.410+0000 7f664359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f6644149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:12.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.410+0000 7f664359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66441482d0 0x7f664414a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:12.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.410+0000 7f664359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644158130 con 0x7f66441482d0 2020-11-20T12:08:12.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.410+0000 7f664359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664415c490 con 0x7f66440b5f50 2020-11-20T12:08:12.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.410+0000 7f664359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6644158320 con 0x7f66440b57b0 2020-11-20T12:08:12.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f6644149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:12.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f6644149e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43466/0 (socket says 172.21.15.187:43466) 2020-11-20T12:08:12.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 -- 172.21.15.187:0/3464762473 learned_addr learned my addr 172.21.15.187:0/3464762473 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:12.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 -- 172.21.15.187:0/3464762473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 msgr2=0x7f6644147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:12.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6642d9d700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66441482d0 0x7f664414a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:12.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f664259c700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:12.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644147490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 -- 172.21.15.187:0/3464762473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66441482d0 msgr2=0x7f664414a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:12.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66441482d0 0x7f664414a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:12.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 -- 172.21.15.187:0/3464762473 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6644147980 con 0x7f66440b5f50 2020-11-20T12:08:12.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.411+0000 7f6641d9b700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f6644149e40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f663800c710 tx=0x7f6638007c70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:12.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.412+0000 7f66327fc700 1 -- 172.21.15.187:0/3464762473 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380072e0 con 0x7f66440b5f50 2020-11-20T12:08:12.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.412+0000 7f66327fc700 1 -- 172.21.15.187:0/3464762473 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6638007440 con 0x7f66440b5f50 2020-11-20T12:08:12.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.412+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664415c060 con 0x7f66440b5f50 2020-11-20T12:08:12.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.412+0000 7f66327fc700 1 -- 172.21.15.187:0/3464762473 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638005800 con 0x7f66440b5f50 2020-11-20T12:08:12.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.412+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66440b1ea0 con 0x7f66440b5f50 2020-11-20T12:08:12.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.413+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66441584e0 con 0x7f66440b5f50 2020-11-20T12:08:12.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.413+0000 7f66327fc700 1 -- 172.21.15.187:0/3464762473 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f663801a060 con 0x7f66440b5f50 2020-11-20T12:08:12.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.414+0000 7f66327fc700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6618056d20 0x7f66180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:12.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.414+0000 7f66327fc700 1 -- 172.21.15.187:0/3464762473 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f663806f460 con 0x7f66440b5f50 2020-11-20T12:08:12.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.415+0000 7f664259c700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6618056d20 0x7f66180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:12.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.415+0000 7f664259c700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6618056d20 0x7f66180591e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f663400b7b0 tx=0x7f6634008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:12.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.418+0000 7f66327fc700 1 -- 172.21.15.187:0/3464762473 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f66380447a0 con 0x7f66440b5f50 2020-11-20T12:08:12.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:12.676+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f6644061a20 con 0x7f66440b5f50 2020-11-20T12:08:13.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.135+0000 7f66327fc700 1 -- 172.21.15.187:0/3464762473 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v175) v1 ==== 135+0+0 (secure 0 0 0) 0x7f6638018020 con 0x7f66440b5f50 2020-11-20T12:08:13.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6618056d20 msgr2=0x7f66180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6618056d20 0x7f66180591e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f663400b7b0 tx=0x7f6634008040).stop 2020-11-20T12:08:13.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 msgr2=0x7f6644149e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f6644149e40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f663800c710 tx=0x7f6638007c70).stop 2020-11-20T12:08:13.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 shutdown_connections 2020-11-20T12:08:13.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6618056d20 0x7f66180591e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66441482d0 0x7f664414a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66440b57b0 0x7f6644147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 --2- 172.21.15.187:0/3464762473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66440b5f50 0x7f6644149e40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 >> 172.21.15.187:0/3464762473 conn(0x7f66440bab30 msgr2=0x7f66440b9620 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:13.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.138+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 shutdown_connections 2020-11-20T12:08:13.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.139+0000 7f664359e700 1 -- 172.21.15.187:0/3464762473 wait complete. 2020-11-20T12:08:13.624 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:08:13.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.877+0000 7f6097fff700 1 Processor -- start 2020-11-20T12:08:13.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.877+0000 7f6097fff700 1 -- start start 2020-11-20T12:08:13.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.878+0000 7f6097fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098066040 0x7f6098148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.878+0000 7f6097fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609814c170 con 0x7f60980683f0 2020-11-20T12:08:13.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.879+0000 7f6097fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609814c410 con 0x7f6098145cc0 2020-11-20T12:08:13.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.879+0000 7f6097fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609814c6b0 con 0x7f6098066040 2020-11-20T12:08:13.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.879+0000 7f6096ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6098145cc0 0x7f6098065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40082/0 (socket says 172.21.15.187:40082) 2020-11-20T12:08:13.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.879+0000 7f6096ffd700 1 -- 172.21.15.187:0/3297224106 learned_addr learned my addr 172.21.15.187:0/3297224106 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:13.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.879+0000 7f60977fe700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098066040 0x7f6098148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139670006 0 0) 0x7f609814c410 con 0x7f6098145cc0 2020-11-20T12:08:13.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6078003240 con 0x7f6098145cc0 2020-11-20T12:08:13.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2387756318 0 0) 0x7f6078003240 con 0x7f6098145cc0 2020-11-20T12:08:13.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098066040 msgr2=0x7f6098148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:13.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098066040 0x7f6098148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 >> v1:172.21.15.5:6789/0 conn(0x7f60980683f0 legacy=0x7f60981487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60980658a0 con 0x7f6098145cc0 2020-11-20T12:08:13.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f60880014e0 con 0x7f6098145cc0 2020-11-20T12:08:13.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 >> v1:172.21.15.187:6789/0 conn(0x7f6098145cc0 legacy=0x7f6098065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.880+0000 7f60967fc700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6078003730 0x7f6078003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.881+0000 7f60967fc700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60780044e0 0x7f6078004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.881+0000 7f60967fc700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098066040 0x7f6078004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.881+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609814c6b0 con 0x7f60780044e0 2020-11-20T12:08:13.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.881+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6078005390 con 0x7f6078003730 2020-11-20T12:08:13.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.881+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60780042f0 con 0x7f6098066040 2020-11-20T12:08:13.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.881+0000 7f6096ffd700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6078003730 0x7f6078003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.881+0000 7f608e7fc700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60780044e0 0x7f6078004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f6096ffd700 1 -- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098066040 msgr2=0x7f6078004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:13.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f6096ffd700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098066040 0x7f6078004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:13.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f6096ffd700 1 -- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60780044e0 msgr2=0x7f6078004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:13.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f6096ffd700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60780044e0 0x7f6078004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f6096ffd700 1 -- 172.21.15.187:0/3297224106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078003240 con 0x7f6078003730 2020-11-20T12:08:13.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f6096ffd700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6078003730 0x7f6078003b50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f608800c410 tx=0x7f60880077d0).ready entity=mon.1 client_cookie=16b58932f5af1ab0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:13.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608800ce20 con 0x7f6078003730 2020-11-20T12:08:13.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60880072e0 con 0x7f6078003730 2020-11-20T12:08:13.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.882+0000 7f60967fc700 1 -- 172.21.15.187:0/3297224106 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6088005800 con 0x7f6078003730 2020-11-20T12:08:13.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.883+0000 7f6097fff700 1 -- 172.21.15.187:0/3297224106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6078003730 msgr2=0x7f6078003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.883+0000 7f6097fff700 1 --2- 172.21.15.187:0/3297224106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6078003730 0x7f6078003b50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f608800c410 tx=0x7f60880077d0).stop 2020-11-20T12:08:13.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.883+0000 7f60977fe700 1 -- 172.21.15.187:0/3297224106 reap_dead start 2020-11-20T12:08:13.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.883+0000 7f6097fff700 1 -- 172.21.15.187:0/3297224106 shutdown_connections 2020-11-20T12:08:13.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.883+0000 7f6097fff700 1 -- 172.21.15.187:0/3297224106 >> 172.21.15.187:0/3297224106 conn(0x7f60980bab30 msgr2=0x7f60980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:13.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.883+0000 7f6097fff700 1 -- 172.21.15.187:0/3297224106 shutdown_connections 2020-11-20T12:08:13.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.883+0000 7f6097fff700 1 -- 172.21.15.187:0/3297224106 wait complete. 2020-11-20T12:08:13.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.885+0000 7f6097fff700 1 Processor -- start 2020-11-20T12:08:13.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.885+0000 7f6097fff700 1 -- start start 2020-11-20T12:08:13.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.886+0000 7f6097fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098158200 0x7f60980b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.886+0000 7f6097fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f609809b590 0x7f60981585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.886+0000 7f6097fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60980b4570 0x7f609809cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.886+0000 7f6097fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60980b3450 con 0x7f609809b590 2020-11-20T12:08:13.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.886+0000 7f6097fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6098158c60 con 0x7f60980b4570 2020-11-20T12:08:13.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.886+0000 7f6097fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609809b230 con 0x7f6098158200 2020-11-20T12:08:13.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.886+0000 7f60977fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60980b4570 0x7f609809cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.886+0000 7f60977fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60980b4570 0x7f609809cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43488/0 (socket says 172.21.15.187:43488) 2020-11-20T12:08:13.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f60977fe700 1 -- 172.21.15.187:0/1447668417 learned_addr learned my addr 172.21.15.187:0/1447668417 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:13.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f608e7fc700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f609809b590 0x7f60981585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f6096ffd700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098158200 0x7f60980b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f60977fe700 1 -- 172.21.15.187:0/1447668417 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098158200 msgr2=0x7f60980b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f60977fe700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098158200 0x7f60980b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f60977fe700 1 -- 172.21.15.187:0/1447668417 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f609809b590 msgr2=0x7f60981585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f60977fe700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f609809b590 0x7f60981585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f60977fe700 1 -- 172.21.15.187:0/1447668417 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609809d230 con 0x7f60980b4570 2020-11-20T12:08:13.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.887+0000 7f60977fe700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60980b4570 0x7f609809cd40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f60980639b0 tx=0x7f60900091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:13.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.888+0000 7f608f7fe700 1 -- 172.21.15.187:0/1447668417 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090006cc0 con 0x7f60980b4570 2020-11-20T12:08:13.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.888+0000 7f608f7fe700 1 -- 172.21.15.187:0/1447668417 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609000c820 con 0x7f60980b4570 2020-11-20T12:08:13.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.888+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609809d4e0 con 0x7f60980b4570 2020-11-20T12:08:13.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.888+0000 7f608f7fe700 1 -- 172.21.15.187:0/1447668417 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609000cbc0 con 0x7f60980b4570 2020-11-20T12:08:13.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.888+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609809ecf0 con 0x7f60980b4570 2020-11-20T12:08:13.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.889+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6098158ad0 con 0x7f60980b4570 2020-11-20T12:08:13.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.890+0000 7f608f7fe700 1 -- 172.21.15.187:0/1447668417 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f609001c460 con 0x7f60980b4570 2020-11-20T12:08:13.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.890+0000 7f608f7fe700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6074056d20 0x7f60740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.891+0000 7f608f7fe700 1 -- 172.21.15.187:0/1447668417 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6090070360 con 0x7f60980b4570 2020-11-20T12:08:13.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.891+0000 7f6096ffd700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6074056d20 0x7f60740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.891+0000 7f6096ffd700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6074056d20 0x7f60740591e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f609814bd50 tx=0x7f6088015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:13.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:13.894+0000 7f608f7fe700 1 -- 172.21.15.187:0/1447668417 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6090045750 con 0x7f60980b4570 2020-11-20T12:08:14.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.081+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f60980b3590 con 0x7f60980b4570 2020-11-20T12:08:14.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.093+0000 7f608f7fe700 1 -- 172.21.15.187:0/1447668417 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6090048d70 con 0x7f60980b4570 2020-11-20T12:08:14.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.095+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6074056d20 msgr2=0x7f60740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:14.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.095+0000 7f6097fff700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6074056d20 0x7f60740591e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f609814bd50 tx=0x7f6088015000).stop 2020-11-20T12:08:14.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60980b4570 msgr2=0x7f609809cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:14.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60980b4570 0x7f609809cd40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f60980639b0 tx=0x7f60900091d0).stop 2020-11-20T12:08:14.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 shutdown_connections 2020-11-20T12:08:14.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6074056d20 0x7f60740591e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60980b4570 0x7f609809cd40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6098158200 0x7f60980b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 --2- 172.21.15.187:0/1447668417 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f609809b590 0x7f60981585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 >> 172.21.15.187:0/1447668417 conn(0x7f60980bab30 msgr2=0x7f60980afb70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 shutdown_connections 2020-11-20T12:08:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.096+0000 7f6097fff700 1 -- 172.21.15.187:0/1447668417 wait complete. 2020-11-20T12:08:14.125 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:08:14.126 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:08:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:08:14.265 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:08:14.379 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:08:14.398 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:08:14.398 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:08:14.399 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:14.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.633+0000 7f40b9fb1700 1 Processor -- start 2020-11-20T12:08:14.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.634+0000 7f40b9fb1700 1 -- start start 2020-11-20T12:08:14.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.635+0000 7f40b9fb1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b40666e0 0x7f40b40b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:14.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.635+0000 7f40b9fb1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f40b4067d30 0x7f40b40681c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:14.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.635+0000 7f40b9fb1700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b4061e80 con 0x7f40b414c750 2020-11-20T12:08:14.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.635+0000 7f40b9fb1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b40620e0 con 0x7f40b4067d30 2020-11-20T12:08:14.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.635+0000 7f40b9fb1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b4062c90 con 0x7f40b40666e0 2020-11-20T12:08:14.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.635+0000 7f40b37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f40b414c750 0x7f40b4062580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40098/0 (socket says 172.21.15.187:40098) 2020-11-20T12:08:14.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.635+0000 7f40b37fe700 1 -- 172.21.15.187:0/364527989 learned_addr learned my addr 172.21.15.187:0/364527989 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:14.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.635+0000 7f40b3fff700 1 --2- 172.21.15.187:0/364527989 >> v2:172.21.15.187:3300/0 conn(0x7f40b4067d30 0x7f40b40681c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:14.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40aaffd700 1 --2- 172.21.15.187:0/364527989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b40666e0 0x7f40b40b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:14.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40b2ffd700 1 -- 172.21.15.187:0/364527989 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975735071 0 0) 0x7f40b4061e80 con 0x7f40b414c750 2020-11-20T12:08:14.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40b2ffd700 1 -- 172.21.15.187:0/364527989 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4094003240 con 0x7f40b414c750 2020-11-20T12:08:14.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40b3fff700 1 -- 172.21.15.187:0/364527989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b40666e0 msgr2=0x7f40b40b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:14.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40b3fff700 1 --2- 172.21.15.187:0/364527989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b40666e0 0x7f40b40b0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40b3fff700 1 -- 172.21.15.187:0/364527989 >> v1:172.21.15.187:6789/0 conn(0x7f40b414c750 legacy=0x7f40b4062580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:14.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40b3fff700 1 -- 172.21.15.187:0/364527989 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40b1500 con 0x7f40b4067d30 2020-11-20T12:08:14.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40aaffd700 1 --2- 172.21.15.187:0/364527989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b40666e0 0x7f40b40b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:08:14.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.636+0000 7f40b3fff700 1 --2- 172.21.15.187:0/364527989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b40681c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f40ac009c90 tx=0x7f40ac0091d0).ready entity=mon.1 client_cookie=cf9f9532ad7d9f4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:14.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.637+0000 7f40b2ffd700 1 -- 172.21.15.187:0/364527989 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac00ba70 con 0x7f40b4067d30 2020-11-20T12:08:14.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.637+0000 7f40b2ffd700 1 -- 172.21.15.187:0/364527989 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac00bbd0 con 0x7f40b4067d30 2020-11-20T12:08:14.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.637+0000 7f40b2ffd700 1 -- 172.21.15.187:0/364527989 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac011400 con 0x7f40b4067d30 2020-11-20T12:08:14.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.637+0000 7f40b9fb1700 1 -- 172.21.15.187:0/364527989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 msgr2=0x7f40b40681c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:14.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.637+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/364527989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b40681c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f40ac009c90 tx=0x7f40ac0091d0).stop 2020-11-20T12:08:14.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.638+0000 7f40b9fb1700 1 -- 172.21.15.187:0/364527989 shutdown_connections 2020-11-20T12:08:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.638+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/364527989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b40681c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.638+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/364527989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b40666e0 0x7f40b40b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.638+0000 7f40b9fb1700 1 -- 172.21.15.187:0/364527989 >> 172.21.15.187:0/364527989 conn(0x7f40b40bacd0 msgr2=0x7f40b40bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.638+0000 7f40b9fb1700 1 -- 172.21.15.187:0/364527989 shutdown_connections 2020-11-20T12:08:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.638+0000 7f40b9fb1700 1 -- 172.21.15.187:0/364527989 wait complete. 2020-11-20T12:08:14.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.640+0000 7f40b9fb1700 1 Processor -- start 2020-11-20T12:08:14.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.640+0000 7f40b9fb1700 1 -- start start 2020-11-20T12:08:14.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.641+0000 7f40b9fb1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40b40666e0 0x7f40b414e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:14.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.641+0000 7f40b9fb1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b415f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:14.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.641+0000 7f40b9fb1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b414c750 0x7f40b409cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:14.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.641+0000 7f40b9fb1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b409ee00 con 0x7f40b40666e0 2020-11-20T12:08:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.641+0000 7f40b9fb1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b4160400 con 0x7f40b4067d30 2020-11-20T12:08:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.641+0000 7f40b9fb1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b414ef10 con 0x7f40b414c750 2020-11-20T12:08:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b415f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b415f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43500/0 (socket says 172.21.15.187:43500) 2020-11-20T12:08:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 -- 172.21.15.187:0/2670122533 learned_addr learned my addr 172.21.15.187:0/2670122533 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40b3fff700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b414c750 0x7f40b409cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40b37fe700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40b40666e0 0x7f40b414e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 -- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b414c750 msgr2=0x7f40b409cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b414c750 0x7f40b409cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 -- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40b40666e0 msgr2=0x7f40b414e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40b40666e0 0x7f40b414e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 -- 172.21.15.187:0/2670122533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b409f030 con 0x7f40b4067d30 2020-11-20T12:08:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40b3fff700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b414c750 0x7f40b409cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.642+0000 7f40aaffd700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b415f6f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f409c00db60 tx=0x7f409c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.643+0000 7f40abfff700 1 -- 172.21.15.187:0/2670122533 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c006030 con 0x7f40b4067d30 2020-11-20T12:08:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.643+0000 7f40abfff700 1 -- 172.21.15.187:0/2670122533 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c006190 con 0x7f40b4067d30 2020-11-20T12:08:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.643+0000 7f40abfff700 1 -- 172.21.15.187:0/2670122533 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c006510 con 0x7f40b4067d30 2020-11-20T12:08:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.643+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b414f280 con 0x7f40b4067d30 2020-11-20T12:08:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.643+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b4151960 con 0x7f40b4067d30 2020-11-20T12:08:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.644+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b409f1c0 con 0x7f40b4067d30 2020-11-20T12:08:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.645+0000 7f40abfff700 1 -- 172.21.15.187:0/2670122533 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f409c0099a0 con 0x7f40b4067d30 2020-11-20T12:08:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.645+0000 7f40abfff700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4090056d20 0x7f40900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.645+0000 7f40abfff700 1 -- 172.21.15.187:0/2670122533 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f409c0702c0 con 0x7f40b4067d30 2020-11-20T12:08:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.646+0000 7f40b37fe700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4090056d20 0x7f40900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.646+0000 7f40b37fe700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4090056d20 0x7f40900591e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f40a40055a0 tx=0x7f40a4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.650+0000 7f40abfff700 1 -- 172.21.15.187:0/2670122533 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f409c045710 con 0x7f40b4067d30 2020-11-20T12:08:14.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.885+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40b4063600 con 0x7f40b4067d30 2020-11-20T12:08:14.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.886+0000 7f40abfff700 1 -- 172.21.15.187:0/2670122533 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f409c048d30 con 0x7f40b4067d30 2020-11-20T12:08:14.890 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:14.891 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:08:14.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4090056d20 msgr2=0x7f40900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:14.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4090056d20 0x7f40900591e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f40a40055a0 tx=0x7f40a4009040).stop 2020-11-20T12:08:14.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 msgr2=0x7f40b415f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:14.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b415f6f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f409c00db60 tx=0x7f409c008040).stop 2020-11-20T12:08:14.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 shutdown_connections 2020-11-20T12:08:14.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4090056d20 0x7f40900591e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40b40666e0 0x7f40b414e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40b414c750 0x7f40b409cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 --2- 172.21.15.187:0/2670122533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b4067d30 0x7f40b415f6f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:14.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 >> 172.21.15.187:0/2670122533 conn(0x7f40b40bacd0 msgr2=0x7f40b414b910 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:14.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 shutdown_connections 2020-11-20T12:08:14.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:14.888+0000 7f40b9fb1700 1 -- 172.21.15.187:0/2670122533 wait complete. 2020-11-20T12:08:14.896 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:08:14.916 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:15.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.152+0000 7fad4b59e700 1 Processor -- start 2020-11-20T12:08:15.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.153+0000 7fad4b59e700 1 -- start start 2020-11-20T12:08:15.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.154+0000 7fad4b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.154+0000 7fad4b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c0b1300 con 0x7fad4c066f70 2020-11-20T12:08:15.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.154+0000 7fad4b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c0b15a0 con 0x7fad4c1484d0 2020-11-20T12:08:15.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.154+0000 7fad4b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c067730 con 0x7fad4c0b6650 2020-11-20T12:08:15.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.155+0000 7fad49d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.155+0000 7fad4ad9d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fad4c066f70 0x7fad4c0aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33626/0 (socket says 172.21.15.187:33626) 2020-11-20T12:08:15.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.155+0000 7fad4ad9d700 1 -- 172.21.15.187:0/2181797246 learned_addr learned my addr 172.21.15.187:0/2181797246 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:15.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.155+0000 7fad4a59c700 1 --1- 172.21.15.187:0/2181797246 >> v1:172.21.15.187:6789/0 conn(0x7fad4c1484d0 0x7fad4c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40112/0 (socket says 172.21.15.187:40112) 2020-11-20T12:08:15.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.156+0000 7fad4959a700 1 -- 172.21.15.187:0/2181797246 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341448576 0 0) 0x7fad4c0b15a0 con 0x7fad4c1484d0 2020-11-20T12:08:15.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.156+0000 7fad4959a700 1 -- 172.21.15.187:0/2181797246 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad30003240 con 0x7fad4c1484d0 2020-11-20T12:08:15.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.156+0000 7fad4959a700 1 -- 172.21.15.187:0/2181797246 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823784827 0 0) 0x7fad4c0b1300 con 0x7fad4c066f70 2020-11-20T12:08:15.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.156+0000 7fad4959a700 1 -- 172.21.15.187:0/2181797246 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad4c0b15a0 con 0x7fad4c066f70 2020-11-20T12:08:15.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.156+0000 7fad49d9b700 1 -- 172.21.15.187:0/2181797246 >> v1:172.21.15.187:6789/0 conn(0x7fad4c1484d0 legacy=0x7fad4c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.156+0000 7fad4959a700 1 -- 172.21.15.187:0/2181797246 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3754670321 0 0) 0x7fad30003240 con 0x7fad4c1484d0 2020-11-20T12:08:15.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.156+0000 7fad49d9b700 1 -- 172.21.15.187:0/2181797246 >> v1:172.21.15.5:6789/0 conn(0x7fad4c066f70 legacy=0x7fad4c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.156+0000 7fad49d9b700 1 -- 172.21.15.187:0/2181797246 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c0b1840 con 0x7fad4c0b6650 2020-11-20T12:08:15.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.157+0000 7fad49d9b700 1 --2- 172.21.15.187:0/2181797246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c0afa90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fad4000b950 tx=0x7fad40006df0).ready entity=mon.2 client_cookie=dfd054ce4da1a542 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.157+0000 7fad4959a700 1 -- 172.21.15.187:0/2181797246 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c001520 con 0x7fad4c0b6650 2020-11-20T12:08:15.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.157+0000 7fad4959a700 1 -- 172.21.15.187:0/2181797246 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40011400 con 0x7fad4c0b6650 2020-11-20T12:08:15.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.158+0000 7fad4959a700 1 -- 172.21.15.187:0/2181797246 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40011780 con 0x7fad4c0b6650 2020-11-20T12:08:15.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.158+0000 7fad4b59e700 1 -- 172.21.15.187:0/2181797246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 msgr2=0x7fad4c0afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.158+0000 7fad4b59e700 1 --2- 172.21.15.187:0/2181797246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c0afa90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fad4000b950 tx=0x7fad40006df0).stop 2020-11-20T12:08:15.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.158+0000 7fad4b59e700 1 -- 172.21.15.187:0/2181797246 shutdown_connections 2020-11-20T12:08:15.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.158+0000 7fad4b59e700 1 --2- 172.21.15.187:0/2181797246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c0afa90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.158+0000 7fad4b59e700 1 -- 172.21.15.187:0/2181797246 >> 172.21.15.187:0/2181797246 conn(0x7fad4c0bab30 msgr2=0x7fad4c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:15.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.158+0000 7fad4b59e700 1 -- 172.21.15.187:0/2181797246 shutdown_connections 2020-11-20T12:08:15.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.158+0000 7fad4b59e700 1 -- 172.21.15.187:0/2181797246 wait complete. 2020-11-20T12:08:15.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.160+0000 7fad4b59e700 1 Processor -- start 2020-11-20T12:08:15.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.160+0000 7fad4b59e700 1 -- start start 2020-11-20T12:08:15.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c066f70 0x7fad4c09d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c1484d0 0x7fad4c1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c09c230 con 0x7fad4c1484d0 2020-11-20T12:08:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c09c4d0 con 0x7fad4c066f70 2020-11-20T12:08:15.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c09c770 con 0x7fad4c0b6650 2020-11-20T12:08:15.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c066f70 0x7fad4c09d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c066f70 0x7fad4c09d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43512/0 (socket says 172.21.15.187:43512) 2020-11-20T12:08:15.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4a59c700 1 -- 172.21.15.187:0/3685957324 learned_addr learned my addr 172.21.15.187:0/3685957324 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:15.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad49d9b700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4a59c700 1 -- 172.21.15.187:0/3685957324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 msgr2=0x7fad4c15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4a59c700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c15bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4a59c700 1 -- 172.21.15.187:0/3685957324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c1484d0 msgr2=0x7fad4c1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.161+0000 7fad4a59c700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c1484d0 0x7fad4c1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:15.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.162+0000 7fad4a59c700 1 -- 172.21.15.187:0/3685957324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c09d5b0 con 0x7fad4c066f70 2020-11-20T12:08:15.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.162+0000 7fad4a59c700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c066f70 0x7fad4c09d0c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fad3c00cae0 tx=0x7fad3c0019e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.162+0000 7fad3a7fc700 1 -- 172.21.15.187:0/3685957324 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c007170 con 0x7fad4c066f70 2020-11-20T12:08:15.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.162+0000 7fad3a7fc700 1 -- 172.21.15.187:0/3685957324 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c0072d0 con 0x7fad4c066f70 2020-11-20T12:08:15.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.162+0000 7fad3a7fc700 1 -- 172.21.15.187:0/3685957324 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c004920 con 0x7fad4c066f70 2020-11-20T12:08:15.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.162+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c0b4610 con 0x7fad4c066f70 2020-11-20T12:08:15.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.162+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c15c210 con 0x7fad4c066f70 2020-11-20T12:08:15.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.163+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c061a20 con 0x7fad4c066f70 2020-11-20T12:08:15.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.164+0000 7fad3a7fc700 1 -- 172.21.15.187:0/3685957324 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fad3c010070 con 0x7fad4c066f70 2020-11-20T12:08:15.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.164+0000 7fad3a7fc700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad20056d20 0x7fad200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.164+0000 7fad3a7fc700 1 -- 172.21.15.187:0/3685957324 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fad3c06fa40 con 0x7fad4c066f70 2020-11-20T12:08:15.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.165+0000 7fad49d9b700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad20056d20 0x7fad200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.165+0000 7fad49d9b700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad20056d20 0x7fad200591e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fad4000b920 tx=0x7fad40015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.168+0000 7fad3a7fc700 1 -- 172.21.15.187:0/3685957324 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fad3c044e30 con 0x7fad4c066f70 2020-11-20T12:08:15.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.340+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad4c0624e0 con 0x7fad4c066f70 2020-11-20T12:08:15.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.343+0000 7fad3a7fc700 1 -- 172.21.15.187:0/3685957324 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16778 (secure 0 0 0) 0x7fad3c0271a0 con 0x7fad4c066f70 2020-11-20T12:08:15.348 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:15.348 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":42,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:07:05.571797+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:07:01.490118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2020-11-20T12:07:02.131720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:15.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.345+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad20056d20 msgr2=0x7fad200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.345+0000 7fad4b59e700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad20056d20 0x7fad200591e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fad4000b920 tx=0x7fad40015000).stop 2020-11-20T12:08:15.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.345+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c066f70 msgr2=0x7fad4c09d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.345+0000 7fad4b59e700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c066f70 0x7fad4c09d0c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fad3c00cae0 tx=0x7fad3c0019e0).stop 2020-11-20T12:08:15.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.345+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 shutdown_connections 2020-11-20T12:08:15.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.346+0000 7fad4b59e700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad20056d20 0x7fad200591e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.346+0000 7fad4b59e700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c1484d0 0x7fad4c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:15.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.346+0000 7fad4b59e700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c066f70 0x7fad4c09d0c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.346+0000 7fad4b59e700 1 --2- 172.21.15.187:0/3685957324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c0b6650 0x7fad4c15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.346+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 >> 172.21.15.187:0/3685957324 conn(0x7fad4c0bab30 msgr2=0x7fad4c0b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:15.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.346+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 shutdown_connections 2020-11-20T12:08:15.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.346+0000 7fad4b59e700 1 -- 172.21.15.187:0/3685957324 wait complete. 2020-11-20T12:08:15.373 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:07:01.490118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:07:02.131720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:08:15.374 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:08:15.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.612+0000 7f3bb8aa9700 1 Processor -- start 2020-11-20T12:08:15.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.612+0000 7f3bb8aa9700 1 -- start start 2020-11-20T12:08:15.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.614+0000 7f3bb8aa9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4065f90 0x7f3bb4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.614+0000 7f3bb8aa9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3bb409ca90 0x7f3bb409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.614+0000 7f3bb8aa9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb40b13e0 con 0x7f3bb4065190 2020-11-20T12:08:15.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.614+0000 7f3bb8aa9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb414ab30 con 0x7f3bb409ca90 2020-11-20T12:08:15.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.614+0000 7f3bb8aa9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4062c10 con 0x7f3bb4065f90 2020-11-20T12:08:15.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.614+0000 7f3bb3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3bb409ca90 0x7f3bb409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.614+0000 7f3bb37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3bb4065190 0x7f3bb4143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40126/0 (socket says 172.21.15.187:40126) 2020-11-20T12:08:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.614+0000 7f3bb37fe700 1 -- 172.21.15.187:0/1767463617 learned_addr learned my addr 172.21.15.187:0/1767463617 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3babfff700 1 --2- 172.21.15.187:0/1767463617 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4065f90 0x7f3bb4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb2ffd700 1 -- 172.21.15.187:0/1767463617 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838370402 0 0) 0x7f3bb40b13e0 con 0x7f3bb4065190 2020-11-20T12:08:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb2ffd700 1 -- 172.21.15.187:0/1767463617 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b94003240 con 0x7f3bb4065190 2020-11-20T12:08:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb3fff700 1 -- 172.21.15.187:0/1767463617 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4065f90 msgr2=0x7f3bb4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb3fff700 1 --2- 172.21.15.187:0/1767463617 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4065f90 0x7f3bb4065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb3fff700 1 -- 172.21.15.187:0/1767463617 >> v1:172.21.15.187:6789/0 conn(0x7f3bb4065190 legacy=0x7f3bb4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb3fff700 1 -- 172.21.15.187:0/1767463617 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb40b1640 con 0x7f3bb409ca90 2020-11-20T12:08:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb3fff700 1 --2- 172.21.15.187:0/1767463617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb409ca90 0x7f3bb409cf20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f3bb414ac70 tx=0x7f3bac0091d0).ready entity=mon.1 client_cookie=d0d0a13b1ed332bc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb2ffd700 1 -- 172.21.15.187:0/1767463617 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac00bad0 con 0x7f3bb409ca90 2020-11-20T12:08:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb2ffd700 1 -- 172.21.15.187:0/1767463617 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac00bc30 con 0x7f3bb409ca90 2020-11-20T12:08:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.615+0000 7f3bb2ffd700 1 -- 172.21.15.187:0/1767463617 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac011400 con 0x7f3bb409ca90 2020-11-20T12:08:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.616+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/1767463617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb409ca90 msgr2=0x7f3bb409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.616+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/1767463617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb409ca90 0x7f3bb409cf20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f3bb414ac70 tx=0x7f3bac0091d0).stop 2020-11-20T12:08:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.616+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/1767463617 shutdown_connections 2020-11-20T12:08:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.616+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/1767463617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb409ca90 0x7f3bb409cf20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.616+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/1767463617 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4065f90 0x7f3bb4065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.616+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/1767463617 >> 172.21.15.187:0/1767463617 conn(0x7f3bb40bab30 msgr2=0x7f3bb40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:15.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.616+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/1767463617 shutdown_connections 2020-11-20T12:08:15.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.616+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/1767463617 wait complete. 2020-11-20T12:08:15.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.618+0000 7f3bb8aa9700 1 Processor -- start 2020-11-20T12:08:15.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.618+0000 7f3bb8aa9700 1 -- start start 2020-11-20T12:08:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.618+0000 7f3bb8aa9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb4065190 0x7f3bb4066590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3bb8aa9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4065f90 0x7f3bb4143160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3bb8aa9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb409ca90 0x7f3bb4143650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3bb8aa9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb409ddc0 con 0x7f3bb4065190 2020-11-20T12:08:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3bb8aa9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40b4a80 con 0x7f3bb4065f90 2020-11-20T12:08:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3bb8aa9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40992a0 con 0x7f3bb409ca90 2020-11-20T12:08:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3babfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4065f90 0x7f3bb4143160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3babfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4065f90 0x7f3bb4143160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43528/0 (socket says 172.21.15.187:43528) 2020-11-20T12:08:15.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3babfff700 1 -- 172.21.15.187:0/2079654571 learned_addr learned my addr 172.21.15.187:0/2079654571 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3babfff700 1 -- 172.21.15.187:0/2079654571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb409ca90 msgr2=0x7f3bb4143650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3bb3fff700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb409ca90 0x7f3bb4143650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3bb37fe700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb4065190 0x7f3bb4066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3babfff700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb409ca90 0x7f3bb4143650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3babfff700 1 -- 172.21.15.187:0/2079654571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb4065190 msgr2=0x7f3bb4066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3babfff700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb4065190 0x7f3bb4066590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.619+0000 7f3babfff700 1 -- 172.21.15.187:0/2079654571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb4099540 con 0x7f3bb4065f90 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.620+0000 7f3babfff700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4065f90 0x7f3bb4143160 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f3b9c00db60 tx=0x7f3b9c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.620+0000 7f3bab7fe700 1 -- 172.21.15.187:0/2079654571 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c009820 con 0x7f3bb4065f90 2020-11-20T12:08:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.620+0000 7f3bab7fe700 1 -- 172.21.15.187:0/2079654571 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c009980 con 0x7f3bb4065f90 2020-11-20T12:08:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.620+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb4066a80 con 0x7f3bb4065f90 2020-11-20T12:08:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.620+0000 7f3bab7fe700 1 -- 172.21.15.187:0/2079654571 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c009d20 con 0x7f3bb4065f90 2020-11-20T12:08:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.620+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb40a0170 con 0x7f3bb4065f90 2020-11-20T12:08:15.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.621+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb4061a20 con 0x7f3bb4065f90 2020-11-20T12:08:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.622+0000 7f3bab7fe700 1 -- 172.21.15.187:0/2079654571 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3b9c013690 con 0x7f3bb4065f90 2020-11-20T12:08:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.622+0000 7f3bab7fe700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b90056d20 0x7f3b900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.623+0000 7f3bab7fe700 1 -- 172.21.15.187:0/2079654571 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3b9c0706e0 con 0x7f3bb4065f90 2020-11-20T12:08:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.623+0000 7f3bb37fe700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b90056d20 0x7f3b900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.623+0000 7f3bb37fe700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b90056d20 0x7f3b900591e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f3ba40055a0 tx=0x7f3ba4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.626+0000 7f3bab7fe700 1 -- 172.21.15.187:0/2079654571 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3b9c045ad0 con 0x7f3bb4065f90 2020-11-20T12:08:15.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.810+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3bb40afc60 con 0x7f3bb4065f90 2020-11-20T12:08:15.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.811+0000 7f3bab7fe700 1 -- 172.21.15.187:0/2079654571 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3b9c0490f0 con 0x7f3bb4065f90 2020-11-20T12:08:15.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b90056d20 msgr2=0x7f3b900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b90056d20 0x7f3b900591e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f3ba40055a0 tx=0x7f3ba4009040).stop 2020-11-20T12:08:15.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4065f90 msgr2=0x7f3bb4143160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4065f90 0x7f3bb4143160 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f3b9c00db60 tx=0x7f3b9c008040).stop 2020-11-20T12:08:15.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 shutdown_connections 2020-11-20T12:08:15.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b90056d20 0x7f3b900591e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb4065190 0x7f3bb4066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb409ca90 0x7f3bb4143650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 --2- 172.21.15.187:0/2079654571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4065f90 0x7f3bb4143160 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 >> 172.21.15.187:0/2079654571 conn(0x7f3bb40bab30 msgr2=0x7f3bb409b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:15.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.813+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 shutdown_connections 2020-11-20T12:08:15.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:15.814+0000 7f3bb8aa9700 1 -- 172.21.15.187:0/2079654571 wait complete. 2020-11-20T12:08:15.820 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:08:15.837 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:08:16.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.069+0000 7ff967f5b700 1 Processor -- start 2020-11-20T12:08:16.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.069+0000 7ff967f5b700 1 -- start start 2020-11-20T12:08:16.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.070+0000 7ff967f5b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff9600b6fd0 0x7ff960066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.070+0000 7ff967f5b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff960146010 con 0x7ff960067300 2020-11-20T12:08:16.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.070+0000 7ff967f5b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9601462b0 con 0x7ff9600b73b0 2020-11-20T12:08:16.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.070+0000 7ff967f5b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff960146550 con 0x7ff9600b6fd0 2020-11-20T12:08:16.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.071+0000 7ff9654f6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff9600b73b0 0x7ff960066bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40140/0 (socket says 172.21.15.187:40140) 2020-11-20T12:08:16.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.071+0000 7ff9654f6700 1 -- 172.21.15.187:0/3789220517 learned_addr learned my addr 172.21.15.187:0/3789220517 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:16.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.071+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808476812 0 0) 0x7ff9601462b0 con 0x7ff9600b73b0 2020-11-20T12:08:16.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.071+0000 7ff965cf7700 1 --2- 172.21.15.187:0/3789220517 >> v2:172.21.15.5:3300/0 conn(0x7ff9600b6fd0 0x7ff960066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.071+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff948003240 con 0x7ff9600b73b0 2020-11-20T12:08:16.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4099620718 0 0) 0x7ff948003240 con 0x7ff9600b73b0 2020-11-20T12:08:16.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 >> v2:172.21.15.5:3300/0 conn(0x7ff9600b6fd0 msgr2=0x7ff960066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:16.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 --2- 172.21.15.187:0/3789220517 >> v2:172.21.15.5:3300/0 conn(0x7ff9600b6fd0 0x7ff960066700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 >> v1:172.21.15.5:6789/0 conn(0x7ff960067300 legacy=0x7ff960144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600b6650 con 0x7ff9600b73b0 2020-11-20T12:08:16.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff9500014e0 con 0x7ff9600b73b0 2020-11-20T12:08:16.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 >> v1:172.21.15.187:6789/0 conn(0x7ff9600b73b0 legacy=0x7ff960066bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff948003780 0x7ff948003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.072+0000 7ff964cf5700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480044e0 0x7ff948004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.073+0000 7ff964cf5700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff948004e50 0x7ff94800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.073+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff960146550 con 0x7ff9480044e0 2020-11-20T12:08:16.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.073+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff948005330 con 0x7ff948003780 2020-11-20T12:08:16.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.073+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94800bc20 con 0x7ff948004e50 2020-11-20T12:08:16.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.073+0000 7ff965cf7700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff948003780 0x7ff948003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.073+0000 7ff9654f6700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480044e0 0x7ff948004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.073+0000 7ff9664f8700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff948004e50 0x7ff94800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.074+0000 7ff965cf7700 1 -- 172.21.15.187:0/3789220517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff948004e50 msgr2=0x7ff94800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:16.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.074+0000 7ff965cf7700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff948004e50 0x7ff94800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.074+0000 7ff965cf7700 1 -- 172.21.15.187:0/3789220517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480044e0 msgr2=0x7ff948004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:16.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.074+0000 7ff965cf7700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480044e0 0x7ff948004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.074+0000 7ff965cf7700 1 -- 172.21.15.187:0/3789220517 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff948003240 con 0x7ff948003780 2020-11-20T12:08:16.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.074+0000 7ff9664f8700 1 -- 172.21.15.187:0/3789220517 reap_dead start 2020-11-20T12:08:16.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.074+0000 7ff965cf7700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff948003780 0x7ff948003ba0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7ff95c00bc00 tx=0x7ff95c006df0).ready entity=mon.1 client_cookie=de8057e6fe18127e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:16.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.075+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c00b310 con 0x7ff948003780 2020-11-20T12:08:16.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.075+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c00ea30 con 0x7ff948003780 2020-11-20T12:08:16.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.075+0000 7ff964cf5700 1 -- 172.21.15.187:0/3789220517 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c00edb0 con 0x7ff948003780 2020-11-20T12:08:16.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.075+0000 7ff967f5b700 1 -- 172.21.15.187:0/3789220517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff948003780 msgr2=0x7ff948003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.075+0000 7ff967f5b700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff948003780 0x7ff948003ba0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7ff95c00bc00 tx=0x7ff95c006df0).stop 2020-11-20T12:08:16.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.076+0000 7ff967f5b700 1 -- 172.21.15.187:0/3789220517 shutdown_connections 2020-11-20T12:08:16.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.076+0000 7ff967f5b700 1 --2- 172.21.15.187:0/3789220517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff948003780 0x7ff948003ba0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.076+0000 7ff967f5b700 1 -- 172.21.15.187:0/3789220517 >> 172.21.15.187:0/3789220517 conn(0x7ff9600bacf0 msgr2=0x7ff9600bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:16.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.076+0000 7ff967f5b700 1 -- 172.21.15.187:0/3789220517 shutdown_connections 2020-11-20T12:08:16.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.076+0000 7ff967f5b700 1 -- 172.21.15.187:0/3789220517 wait complete. 2020-11-20T12:08:16.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.078+0000 7ff967f5b700 1 Processor -- start 2020-11-20T12:08:16.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.078+0000 7ff967f5b700 1 -- start start 2020-11-20T12:08:16.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff967f5b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9600b4220 0x7ff96015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff967f5b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9600b4600 0x7ff960090a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff967f5b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff960090f80 0x7ff96013aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff967f5b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96015c950 con 0x7ff9600b4600 2020-11-20T12:08:16.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff967f5b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600900b0 con 0x7ff9600b4220 2020-11-20T12:08:16.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff967f5b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff960090350 con 0x7ff960090f80 2020-11-20T12:08:16.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff965cf7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9600b4220 0x7ff96015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff965cf7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9600b4220 0x7ff96015c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43546/0 (socket says 172.21.15.187:43546) 2020-11-20T12:08:16.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff965cf7700 1 -- 172.21.15.187:0/2800655480 learned_addr learned my addr 172.21.15.187:0/2800655480 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:16.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff9654f6700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9600b4600 0x7ff960090a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff965cf7700 1 -- 172.21.15.187:0/2800655480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff960090f80 msgr2=0x7ff96013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff965cf7700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff960090f80 0x7ff96013aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:16.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff965cf7700 1 -- 172.21.15.187:0/2800655480 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9600b4600 msgr2=0x7ff960090a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff965cf7700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9600b4600 0x7ff960090a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.079+0000 7ff965cf7700 1 -- 172.21.15.187:0/2800655480 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600905f0 con 0x7ff9600b4220 2020-11-20T12:08:16.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.080+0000 7ff965cf7700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9600b4220 0x7ff96015c460 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7ff95c013000 tx=0x7ff95c019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:16.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.080+0000 7ff94dffb700 1 -- 172.21.15.187:0/2800655480 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c00b310 con 0x7ff9600b4220 2020-11-20T12:08:16.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.080+0000 7ff94dffb700 1 -- 172.21.15.187:0/2800655480 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c01d620 con 0x7ff9600b4220 2020-11-20T12:08:16.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.080+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff960143400 con 0x7ff9600b4220 2020-11-20T12:08:16.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.080+0000 7ff94dffb700 1 -- 172.21.15.187:0/2800655480 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c01d7d0 con 0x7ff9600b4220 2020-11-20T12:08:16.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.080+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96013b720 con 0x7ff9600b4220 2020-11-20T12:08:16.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.081+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff960143730 con 0x7ff9600b4220 2020-11-20T12:08:16.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.082+0000 7ff94dffb700 1 -- 172.21.15.187:0/2800655480 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff95c02d070 con 0x7ff9600b4220 2020-11-20T12:08:16.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.082+0000 7ff94dffb700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff938056d20 0x7ff9380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.082+0000 7ff94dffb700 1 -- 172.21.15.187:0/2800655480 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff95c071e90 con 0x7ff9600b4220 2020-11-20T12:08:16.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.083+0000 7ff9654f6700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff938056d20 0x7ff9380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.083+0000 7ff9654f6700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff938056d20 0x7ff9380591e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff950000dd0 tx=0x7ff950009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:16.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.087+0000 7ff94dffb700 1 -- 172.21.15.187:0/2800655480 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff95c06efd0 con 0x7ff9600b4220 2020-11-20T12:08:16.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.277+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff960090780 con 0x7ff9600b4220 2020-11-20T12:08:16.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.278+0000 7ff94dffb700 1 -- 172.21.15.187:0/2800655480 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v42) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff95c02b070 con 0x7ff9600b4220 2020-11-20T12:08:16.282 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:16.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff938056d20 msgr2=0x7ff9380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff938056d20 0x7ff9380591e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff950000dd0 tx=0x7ff950009040).stop 2020-11-20T12:08:16.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9600b4220 msgr2=0x7ff96015c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9600b4220 0x7ff96015c460 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7ff95c013000 tx=0x7ff95c019000).stop 2020-11-20T12:08:16.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 shutdown_connections 2020-11-20T12:08:16.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff938056d20 0x7ff9380591e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff960090f80 0x7ff96013aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:16.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9600b4220 0x7ff96015c460 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 --2- 172.21.15.187:0/2800655480 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9600b4600 0x7ff960090a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 >> 172.21.15.187:0/2800655480 conn(0x7ff9600bacf0 msgr2=0x7ff9600bb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:16.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 shutdown_connections 2020-11-20T12:08:16.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.280+0000 7ff967f5b700 1 -- 172.21.15.187:0/2800655480 wait complete. 2020-11-20T12:08:16.309 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:08:16.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.544+0000 7f2feb59e700 1 Processor -- start 2020-11-20T12:08:16.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.544+0000 7f2feb59e700 1 -- start start 2020-11-20T12:08:16.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.546+0000 7f2feb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec068230 0x7f2fec1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.546+0000 7f2feb59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2fec0b5f50 0x7f2fec0b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.546+0000 7f2feb59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec144b80 con 0x7f2fec0b57b0 2020-11-20T12:08:16.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.546+0000 7f2feb59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec1487c0 con 0x7f2fec0b5f50 2020-11-20T12:08:16.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.546+0000 7f2feb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1489e0 con 0x7f2fec068230 2020-11-20T12:08:16.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.547+0000 7f2fea59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2fec0b57b0 0x7f2fec062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40160/0 (socket says 172.21.15.187:40160) 2020-11-20T12:08:16.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.547+0000 7f2fea59c700 1 -- 172.21.15.187:0/148438817 learned_addr learned my addr 172.21.15.187:0/148438817 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:16.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.547+0000 7f2fe1d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec068230 0x7f2fec1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.547+0000 7f2fead9d700 1 --2- 172.21.15.187:0/148438817 >> v2:172.21.15.5:3300/0 conn(0x7f2fec0b5f50 0x7f2fec0b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.547+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545414185 0 0) 0x7f2fec144b80 con 0x7f2fec0b57b0 2020-11-20T12:08:16.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.547+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc003240 con 0x7f2fec0b57b0 2020-11-20T12:08:16.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4078827781 0 0) 0x7f2fcc003240 con 0x7f2fec0b57b0 2020-11-20T12:08:16.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec068230 msgr2=0x7f2fec1482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:16.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec068230 0x7f2fec1482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 >> v2:172.21.15.5:3300/0 conn(0x7f2fec0b5f50 msgr2=0x7f2fec0b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:16.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 --2- 172.21.15.187:0/148438817 >> v2:172.21.15.5:3300/0 conn(0x7f2fec0b5f50 0x7f2fec0b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec148c80 con 0x7f2fec0b57b0 2020-11-20T12:08:16.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f2fdc0014e0 con 0x7f2fec0b57b0 2020-11-20T12:08:16.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 >> v1:172.21.15.187:6789/0 conn(0x7f2fec0b57b0 legacy=0x7f2fec062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fcc003730 0x7f2fcc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fcc0044e0 0x7f2fcc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec068230 0x7f2fcc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1487c0 con 0x7f2fcc003730 2020-11-20T12:08:16.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1489e0 con 0x7f2fcc0044e0 2020-11-20T12:08:16.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc004270 con 0x7f2fec068230 2020-11-20T12:08:16.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.548+0000 7f2fe1d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fcc0044e0 0x7f2fcc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.549+0000 7f2fead9d700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec068230 0x7f2fcc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.549+0000 7f2fea59c700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fcc003730 0x7f2fcc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.549+0000 7f2fe1d9b700 1 -- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec068230 msgr2=0x7f2fcc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:16.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.549+0000 7f2fe1d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec068230 0x7f2fcc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.549+0000 7f2fe1d9b700 1 -- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fcc003730 msgr2=0x7f2fcc003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:16.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.549+0000 7f2fe1d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fcc003730 0x7f2fcc003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.549+0000 7f2fe1d9b700 1 -- 172.21.15.187:0/148438817 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc003240 con 0x7f2fcc0044e0 2020-11-20T12:08:16.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.549+0000 7f2fe1d9b700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fcc0044e0 0x7f2fcc004960 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f2fd400ba80 tx=0x7f2fd4008040).ready entity=mon.1 client_cookie=cf5534b6889e056a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:16.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd400d8f0 con 0x7f2fcc0044e0 2020-11-20T12:08:16.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd400da50 con 0x7f2fcc0044e0 2020-11-20T12:08:16.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2fe9d9b700 1 -- 172.21.15.187:0/148438817 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd400ddd0 con 0x7f2fcc0044e0 2020-11-20T12:08:16.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2feb59e700 1 -- 172.21.15.187:0/148438817 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fcc0044e0 msgr2=0x7f2fcc004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2feb59e700 1 --2- 172.21.15.187:0/148438817 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fcc0044e0 0x7f2fcc004960 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f2fd400ba80 tx=0x7f2fd4008040).stop 2020-11-20T12:08:16.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2fead9d700 1 -- 172.21.15.187:0/148438817 reap_dead start 2020-11-20T12:08:16.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2feb59e700 1 -- 172.21.15.187:0/148438817 shutdown_connections 2020-11-20T12:08:16.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2feb59e700 1 -- 172.21.15.187:0/148438817 >> 172.21.15.187:0/148438817 conn(0x7f2fec0bab30 msgr2=0x7f2fec0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:16.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2feb59e700 1 -- 172.21.15.187:0/148438817 shutdown_connections 2020-11-20T12:08:16.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.550+0000 7f2feb59e700 1 -- 172.21.15.187:0/148438817 wait complete. 2020-11-20T12:08:16.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.552+0000 7f2feb59e700 1 Processor -- start 2020-11-20T12:08:16.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.552+0000 7f2feb59e700 1 -- start start 2020-11-20T12:08:16.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2feb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fec09fec0 0x7f2fec145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2feb59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fec0a02a0 0x7f2fec0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2feb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec08ea20 0x7f2fec147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2feb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0b3ec0 con 0x7f2fec09fec0 2020-11-20T12:08:16.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2feb59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1460a0 con 0x7f2fec0a02a0 2020-11-20T12:08:16.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2feb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0b3780 con 0x7f2fec08ea20 2020-11-20T12:08:16.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2fe1d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fec0a02a0 0x7f2fec0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2fe1d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fec0a02a0 0x7f2fec0b1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43568/0 (socket says 172.21.15.187:43568) 2020-11-20T12:08:16.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.553+0000 7f2fe1d9b700 1 -- 172.21.15.187:0/1113411684 learned_addr learned my addr 172.21.15.187:0/1113411684 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:16.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fea59c700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fec09fec0 0x7f2fec145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe1d9b700 1 -- 172.21.15.187:0/1113411684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec08ea20 msgr2=0x7f2fec147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe1d9b700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec08ea20 0x7f2fec147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:16.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe1d9b700 1 -- 172.21.15.187:0/1113411684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fec09fec0 msgr2=0x7f2fec145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe1d9b700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fec09fec0 0x7f2fec145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe1d9b700 1 -- 172.21.15.187:0/1113411684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0b3a20 con 0x7f2fec0a02a0 2020-11-20T12:08:16.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe1d9b700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fec0a02a0 0x7f2fec0b1b80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f2fd4012000 tx=0x7f2fd4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:16.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe2ffd700 1 -- 172.21.15.187:0/1113411684 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4006920 con 0x7f2fec0a02a0 2020-11-20T12:08:16.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe2ffd700 1 -- 172.21.15.187:0/1113411684 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd401d620 con 0x7f2fec0a02a0 2020-11-20T12:08:16.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2fe2ffd700 1 -- 172.21.15.187:0/1113411684 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd401d9c0 con 0x7f2fec0a02a0 2020-11-20T12:08:16.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.554+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec0b2070 con 0x7f2fec0a02a0 2020-11-20T12:08:16.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.555+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec147bc0 con 0x7f2fec0a02a0 2020-11-20T12:08:16.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.556+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec145ef0 con 0x7f2fec0a02a0 2020-11-20T12:08:16.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.556+0000 7f2fe2ffd700 1 -- 172.21.15.187:0/1113411684 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2fd401dd30 con 0x7f2fec0a02a0 2020-11-20T12:08:16.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.557+0000 7f2fe2ffd700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fc8056d50 0x7f2fc8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:16.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.557+0000 7f2fe2ffd700 1 -- 172.21.15.187:0/1113411684 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2fd40715a0 con 0x7f2fec0a02a0 2020-11-20T12:08:16.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.557+0000 7f2fea59c700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fc8056d50 0x7f2fc8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:16.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.558+0000 7f2fea59c700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fc8056d50 0x7f2fc8059210 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2fdc001e90 tx=0x7f2fdc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:16.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.561+0000 7f2fe2ffd700 1 -- 172.21.15.187:0/1113411684 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2fd40469f0 con 0x7f2fec0a02a0 2020-11-20T12:08:16.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.744+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2fec0b5cf0 con 0x7f2fec0a02a0 2020-11-20T12:08:16.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.745+0000 7f2fe2ffd700 1 -- 172.21.15.187:0/1113411684 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2fd404a010 con 0x7f2fec0a02a0 2020-11-20T12:08:16.750 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:16.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.747+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fc8056d50 msgr2=0x7f2fc8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.747+0000 7f2feb59e700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fc8056d50 0x7f2fc8059210 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2fdc001e90 tx=0x7f2fdc009040).stop 2020-11-20T12:08:16.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fec0a02a0 msgr2=0x7f2fec0b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:16.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fec0a02a0 0x7f2fec0b1b80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f2fd4012000 tx=0x7f2fd4008040).stop 2020-11-20T12:08:16.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 shutdown_connections 2020-11-20T12:08:16.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fc8056d50 0x7f2fc8059210 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fec09fec0 0x7f2fec145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2fec08ea20 0x7f2fec147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:16.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 --2- 172.21.15.187:0/1113411684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fec0a02a0 0x7f2fec0b1b80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:16.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 >> 172.21.15.187:0/1113411684 conn(0x7f2fec0bab30 msgr2=0x7f2fec065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:16.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 shutdown_connections 2020-11-20T12:08:16.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:16.748+0000 7f2feb59e700 1 -- 172.21.15.187:0/1113411684 wait complete. 2020-11-20T12:08:16.774 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:08:17.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.015+0000 7f197b59e700 1 Processor -- start 2020-11-20T12:08:17.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.015+0000 7f197b59e700 1 -- start start 2020-11-20T12:08:17.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.016+0000 7f197b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b1380 0x7f197c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.017+0000 7f197b59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f197c0b1760 0x7f197c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.017+0000 7f197b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197c150390 con 0x7f197c065680 2020-11-20T12:08:17.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.017+0000 7f197b59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197c150630 con 0x7f197c0b1760 2020-11-20T12:08:17.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.017+0000 7f197b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c068510 con 0x7f197c0b1380 2020-11-20T12:08:17.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.017+0000 7f1979d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f197c0b1760 0x7f197c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.017+0000 7f197ad9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f197c065680 0x7f197c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40182/0 (socket says 172.21.15.187:40182) 2020-11-20T12:08:17.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.017+0000 7f197ad9d700 1 -- 172.21.15.187:0/4024982601 learned_addr learned my addr 172.21.15.187:0/4024982601 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:17.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.017+0000 7f197a59c700 1 --2- 172.21.15.187:0/4024982601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b1380 0x7f197c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.018+0000 7f197959a700 1 -- 172.21.15.187:0/4024982601 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917639696 0 0) 0x7f197c150390 con 0x7f197c065680 2020-11-20T12:08:17.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.018+0000 7f197959a700 1 -- 172.21.15.187:0/4024982601 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1960003240 con 0x7f197c065680 2020-11-20T12:08:17.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.018+0000 7f1979d9b700 1 -- 172.21.15.187:0/4024982601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b1380 msgr2=0x7f197c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:17.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.018+0000 7f1979d9b700 1 --2- 172.21.15.187:0/4024982601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b1380 0x7f197c150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.018+0000 7f1979d9b700 1 -- 172.21.15.187:0/4024982601 >> v1:172.21.15.187:6789/0 conn(0x7f197c065680 legacy=0x7f197c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.018+0000 7f1979d9b700 1 -- 172.21.15.187:0/4024982601 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c150fc0 con 0x7f197c0b1760 2020-11-20T12:08:17.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.019+0000 7f197a59c700 1 --2- 172.21.15.187:0/4024982601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b1380 0x7f197c150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:08:17.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.019+0000 7f1979d9b700 1 --2- 172.21.15.187:0/4024982601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1760 0x7f197c065190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f197000b9a0 tx=0x7f1970006df0).ready entity=mon.1 client_cookie=378484865c1fdea0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.019+0000 7f197959a700 1 -- 172.21.15.187:0/4024982601 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970009680 con 0x7f197c0b1760 2020-11-20T12:08:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.019+0000 7f197959a700 1 -- 172.21.15.187:0/4024982601 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19700097e0 con 0x7f197c0b1760 2020-11-20T12:08:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.019+0000 7f197959a700 1 -- 172.21.15.187:0/4024982601 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970009b60 con 0x7f197c0b1760 2020-11-20T12:08:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.020+0000 7f197b59e700 1 -- 172.21.15.187:0/4024982601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1760 msgr2=0x7f197c065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.020+0000 7f197b59e700 1 --2- 172.21.15.187:0/4024982601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1760 0x7f197c065190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f197000b9a0 tx=0x7f1970006df0).stop 2020-11-20T12:08:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.020+0000 7f197b59e700 1 -- 172.21.15.187:0/4024982601 shutdown_connections 2020-11-20T12:08:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.020+0000 7f197b59e700 1 --2- 172.21.15.187:0/4024982601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b1380 0x7f197c150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.020+0000 7f197b59e700 1 --2- 172.21.15.187:0/4024982601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1760 0x7f197c065190 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.020+0000 7f197b59e700 1 -- 172.21.15.187:0/4024982601 >> 172.21.15.187:0/4024982601 conn(0x7f197c0bab30 msgr2=0x7f197c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.020+0000 7f197b59e700 1 -- 172.21.15.187:0/4024982601 shutdown_connections 2020-11-20T12:08:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.020+0000 7f197b59e700 1 -- 172.21.15.187:0/4024982601 wait complete. 2020-11-20T12:08:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.021+0000 7f197b59e700 1 Processor -- start 2020-11-20T12:08:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.022+0000 7f197b59e700 1 -- start start 2020-11-20T12:08:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.022+0000 7f197b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1380 0x7f197c0b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.022+0000 7f197b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f197c0b73e0 0x7f197c1488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.022+0000 7f197b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b53c0 0x7f197c0b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.022+0000 7f197b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c0b3640 con 0x7f197c0b73e0 2020-11-20T12:08:17.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.022+0000 7f197b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c0b3950 con 0x7f197c0b1380 2020-11-20T12:08:17.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.022+0000 7f197b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c0b3bf0 con 0x7f197c0b53c0 2020-11-20T12:08:17.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f197a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1380 0x7f197c0b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f197a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1380 0x7f197c0b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43578/0 (socket says 172.21.15.187:43578) 2020-11-20T12:08:17.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f197a59c700 1 -- 172.21.15.187:0/76421611 learned_addr learned my addr 172.21.15.187:0/76421611 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f1979d9b700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f197c0b73e0 0x7f197c1488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f197a59c700 1 -- 172.21.15.187:0/76421611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b53c0 msgr2=0x7f197c0b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f197a59c700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b53c0 0x7f197c0b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f197a59c700 1 -- 172.21.15.187:0/76421611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f197c0b73e0 msgr2=0x7f197c1488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f197a59c700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f197c0b73e0 0x7f197c1488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f197a59c700 1 -- 172.21.15.187:0/76421611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c0b3370 con 0x7f197c0b1380 2020-11-20T12:08:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.023+0000 7f1979d9b700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f197c0b73e0 0x7f197c1488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.024+0000 7f197a59c700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1380 0x7f197c0b2e80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f197400efd0 tx=0x7f19740091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.024+0000 7f196a7fc700 1 -- 172.21.15.187:0/76421611 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197400ce80 con 0x7f197c0b1380 2020-11-20T12:08:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.024+0000 7f196a7fc700 1 -- 172.21.15.187:0/76421611 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19740123f0 con 0x7f197c0b1380 2020-11-20T12:08:17.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.024+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c148dd0 con 0x7f197c0b1380 2020-11-20T12:08:17.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.024+0000 7f196a7fc700 1 -- 172.21.15.187:0/76421611 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974012770 con 0x7f197c0b1380 2020-11-20T12:08:17.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.024+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c14b5e0 con 0x7f197c0b1380 2020-11-20T12:08:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.025+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c061a20 con 0x7f197c0b1380 2020-11-20T12:08:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.026+0000 7f196a7fc700 1 -- 172.21.15.187:0/76421611 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1974022070 con 0x7f197c0b1380 2020-11-20T12:08:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.026+0000 7f196a7fc700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1950056d20 0x7f19500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.027+0000 7f196a7fc700 1 -- 172.21.15.187:0/76421611 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1974070a20 con 0x7f197c0b1380 2020-11-20T12:08:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.027+0000 7f1979d9b700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1950056d20 0x7f19500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.027+0000 7f1979d9b700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1950056d20 0x7f19500591e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f197c065dc0 tx=0x7f1970015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.030+0000 7f196a7fc700 1 -- 172.21.15.187:0/76421611 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1974045df0 con 0x7f197c0b1380 2020-11-20T12:08:17.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.218+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f197c0446d0 con 0x7f197c0b1380 2020-11-20T12:08:17.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.219+0000 7f196a7fc700 1 -- 172.21.15.187:0/76421611 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1974070070 con 0x7f197c0b1380 2020-11-20T12:08:17.224 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:17.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1950056d20 msgr2=0x7f19500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1950056d20 0x7f19500591e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f197c065dc0 tx=0x7f1970015000).stop 2020-11-20T12:08:17.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1380 msgr2=0x7f197c0b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1380 0x7f197c0b2e80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f197400efd0 tx=0x7f19740091d0).stop 2020-11-20T12:08:17.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 shutdown_connections 2020-11-20T12:08:17.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1950056d20 0x7f19500591e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f197c0b53c0 0x7f197c0b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:17.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f197c0b1380 0x7f197c0b2e80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 --2- 172.21.15.187:0/76421611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f197c0b73e0 0x7f197c1488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 >> 172.21.15.187:0/76421611 conn(0x7f197c0bab30 msgr2=0x7f197c14e220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:17.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 shutdown_connections 2020-11-20T12:08:17.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.222+0000 7f197b59e700 1 -- 172.21.15.187:0/76421611 wait complete. 2020-11-20T12:08:17.248 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:17.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.484+0000 7fd061a93700 1 Processor -- start 2020-11-20T12:08:17.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.484+0000 7fd061a93700 1 -- start start 2020-11-20T12:08:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.485+0000 7fd061a93700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd05c068230 0x7fd05c1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.485+0000 7fd061a93700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c144b80 con 0x7fd05c0b5f50 2020-11-20T12:08:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.485+0000 7fd061a93700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c1489e0 con 0x7fd05c0b57b0 2020-11-20T12:08:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.485+0000 7fd061a93700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c1487c0 con 0x7fd05c068230 2020-11-20T12:08:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.486+0000 7fd060a91700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd05c0b57b0 0x7fd05c062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40192/0 (socket says 172.21.15.187:40192) 2020-11-20T12:08:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.486+0000 7fd060a91700 1 -- 172.21.15.187:0/459777363 learned_addr learned my addr 172.21.15.187:0/459777363 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.486+0000 7fd053fff700 1 --2- 172.21.15.187:0/459777363 >> v2:172.21.15.187:3300/0 conn(0x7fd05c068230 0x7fd05c1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.487+0000 7fd0537fe700 1 -- 172.21.15.187:0/459777363 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111621601 0 0) 0x7fd05c1489e0 con 0x7fd05c0b57b0 2020-11-20T12:08:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.487+0000 7fd0537fe700 1 -- 172.21.15.187:0/459777363 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd048003240 con 0x7fd05c0b57b0 2020-11-20T12:08:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.487+0000 7fd053fff700 1 -- 172.21.15.187:0/459777363 >> v1:172.21.15.187:6789/0 conn(0x7fd05c0b57b0 legacy=0x7fd05c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.487+0000 7fd053fff700 1 -- 172.21.15.187:0/459777363 >> v1:172.21.15.5:6789/0 conn(0x7fd05c0b5f50 legacy=0x7fd05c0b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.487+0000 7fd053fff700 1 -- 172.21.15.187:0/459777363 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c148c80 con 0x7fd05c068230 2020-11-20T12:08:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.487+0000 7fd053fff700 1 --2- 172.21.15.187:0/459777363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c068230 0x7fd05c1482d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fd04c00b9a0 tx=0x7fd04c006df0).ready entity=mon.1 client_cookie=8d3b148c8e9a3286 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd0537fe700 1 -- 172.21.15.187:0/459777363 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c0097b0 con 0x7fd05c068230 2020-11-20T12:08:17.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd0537fe700 1 -- 172.21.15.187:0/459777363 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04c009910 con 0x7fd05c068230 2020-11-20T12:08:17.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd0537fe700 1 -- 172.21.15.187:0/459777363 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c009c90 con 0x7fd05c068230 2020-11-20T12:08:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd061a93700 1 -- 172.21.15.187:0/459777363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c068230 msgr2=0x7fd05c1482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd061a93700 1 --2- 172.21.15.187:0/459777363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c068230 0x7fd05c1482d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fd04c00b9a0 tx=0x7fd04c006df0).stop 2020-11-20T12:08:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd061a93700 1 -- 172.21.15.187:0/459777363 shutdown_connections 2020-11-20T12:08:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd061a93700 1 --2- 172.21.15.187:0/459777363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c068230 0x7fd05c1482d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd061a93700 1 -- 172.21.15.187:0/459777363 >> 172.21.15.187:0/459777363 conn(0x7fd05c0bab30 msgr2=0x7fd05c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.488+0000 7fd061a93700 1 -- 172.21.15.187:0/459777363 shutdown_connections 2020-11-20T12:08:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.489+0000 7fd061a93700 1 -- 172.21.15.187:0/459777363 wait complete. 2020-11-20T12:08:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.490+0000 7fd061a93700 1 Processor -- start 2020-11-20T12:08:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.490+0000 7fd061a93700 1 -- start start 2020-11-20T12:08:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.491+0000 7fd061a93700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c0b57b0 0x7fd05c0b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.491+0000 7fd061a93700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd05c0b5f50 0x7fd05c157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.491+0000 7fd061a93700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd05c148130 0x7fd05c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.491+0000 7fd061a93700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0b6e30 con 0x7fd05c148130 2020-11-20T12:08:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.491+0000 7fd061a93700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0b70d0 con 0x7fd05c0b57b0 2020-11-20T12:08:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.491+0000 7fd061a93700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0b7370 con 0x7fd05c0b5f50 2020-11-20T12:08:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.491+0000 7fd060a91700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c0b57b0 0x7fd05c0b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd060a91700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c0b57b0 0x7fd05c0b7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43592/0 (socket says 172.21.15.187:43592) 2020-11-20T12:08:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd060a91700 1 -- 172.21.15.187:0/4239059324 learned_addr learned my addr 172.21.15.187:0/4239059324 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:17.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd053fff700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd05c0b5f50 0x7fd05c157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd061292700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd05c148130 0x7fd05c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd060a91700 1 -- 172.21.15.187:0/4239059324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd05c0b5f50 msgr2=0x7fd05c157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd060a91700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd05c0b5f50 0x7fd05c157820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd060a91700 1 -- 172.21.15.187:0/4239059324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd05c148130 msgr2=0x7fd05c157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd060a91700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd05c148130 0x7fd05c157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd060a91700 1 -- 172.21.15.187:0/4239059324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c1438e0 con 0x7fd05c0b57b0 2020-11-20T12:08:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.492+0000 7fd060a91700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c0b57b0 0x7fd05c0b7810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7fd05800c6f0 tx=0x7fd058007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.493+0000 7fd050ff9700 1 -- 172.21.15.187:0/4239059324 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd058007320 con 0x7fd05c0b57b0 2020-11-20T12:08:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.493+0000 7fd050ff9700 1 -- 172.21.15.187:0/4239059324 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd058007480 con 0x7fd05c0b57b0 2020-11-20T12:08:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.493+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c143b30 con 0x7fd05c0b57b0 2020-11-20T12:08:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.493+0000 7fd050ff9700 1 -- 172.21.15.187:0/4239059324 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd058005800 con 0x7fd05c0b57b0 2020-11-20T12:08:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.493+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c144070 con 0x7fd05c0b57b0 2020-11-20T12:08:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.494+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c0b7530 con 0x7fd05c0b57b0 2020-11-20T12:08:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.494+0000 7fd050ff9700 1 -- 172.21.15.187:0/4239059324 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd058010030 con 0x7fd05c0b57b0 2020-11-20T12:08:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.495+0000 7fd050ff9700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd034056d20 0x7fd0340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.495+0000 7fd050ff9700 1 -- 172.21.15.187:0/4239059324 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd05806fc00 con 0x7fd05c0b57b0 2020-11-20T12:08:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.495+0000 7fd053fff700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd034056d20 0x7fd0340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.496+0000 7fd053fff700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd034056d20 0x7fd0340591e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fd05c0b1b80 tx=0x7fd04c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.499+0000 7fd050ff9700 1 -- 172.21.15.187:0/4239059324 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd058075020 con 0x7fd05c0b57b0 2020-11-20T12:08:17.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.743+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd05c061a20 con 0x7fd05c0b57b0 2020-11-20T12:08:19.267 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:08:19.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.744+0000 7fd050ff9700 1 -- 172.21.15.187:0/4239059324 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fd058075870 con 0x7fd05c0b57b0 2020-11-20T12:08:19.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd034056d20 msgr2=0x7fd0340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd034056d20 0x7fd0340591e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fd05c0b1b80 tx=0x7fd04c015000).stop 2020-11-20T12:08:19.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c0b57b0 msgr2=0x7fd05c0b7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c0b57b0 0x7fd05c0b7810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7fd05800c6f0 tx=0x7fd058007ad0).stop 2020-11-20T12:08:19.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 shutdown_connections 2020-11-20T12:08:19.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd034056d20 0x7fd0340591e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd05c148130 0x7fd05c157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd05c0b57b0 0x7fd05c0b7810 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 --2- 172.21.15.187:0/4239059324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd05c0b5f50 0x7fd05c157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 >> 172.21.15.187:0/4239059324 conn(0x7fd05c0bab30 msgr2=0x7fd05c0b9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:19.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.746+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 shutdown_connections 2020-11-20T12:08:19.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:17.747+0000 7fd061a93700 1 -- 172.21.15.187:0/4239059324 wait complete. 2020-11-20T12:08:19.271 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 19 2020-11-20T12:08:19.271 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:19.271 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2489532388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2489532388},{"type":"v1","addr":"172.21.15.5:6833","nonce":2489532388}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:07:03.348416+0000","modified":"2020-11-20T12:07:05.571685+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5335":{"gid":5335,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2880645780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2880645780},{"type":"v1","addr":"172.21.15.187:6835","nonce":2880645780}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2020-11-20T12:08:19.272 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:19.281 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:08:19.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.498+0000 7f5215299700 1 Processor -- start 2020-11-20T12:08:19.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.498+0000 7f5215299700 1 -- start start 2020-11-20T12:08:19.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.499+0000 7f5215299700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5210065190 0x7f5210065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.499+0000 7f5215299700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5210065f90 0x7f5210143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.499+0000 7f5215299700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100b13e0 con 0x7f5210143c70 2020-11-20T12:08:19.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.499+0000 7f5215299700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521014ab30 con 0x7f5210065f90 2020-11-20T12:08:19.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.500+0000 7f5215299700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5210062c10 con 0x7f5210065190 2020-11-20T12:08:19.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.500+0000 7f520ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5210065190 0x7f5210065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.500+0000 7f5214a98700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5210143c70 0x7f521009ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40208/0 (socket says 172.21.15.187:40208) 2020-11-20T12:08:19.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.500+0000 7f5214a98700 1 -- 172.21.15.187:0/3416104940 learned_addr learned my addr 172.21.15.187:0/3416104940 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:19.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.501+0000 7f520f7fe700 1 --2- 172.21.15.187:0/3416104940 >> v2:172.21.15.5:3300/0 conn(0x7f5210065f90 0x7f5210143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.501+0000 7f520ffff700 1 -- 172.21.15.187:0/3416104940 >> v2:172.21.15.5:3300/0 conn(0x7f5210065f90 msgr2=0x7f5210143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:19.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.501+0000 7f520ffff700 1 --2- 172.21.15.187:0/3416104940 >> v2:172.21.15.5:3300/0 conn(0x7f5210065f90 0x7f5210143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.501+0000 7f520ffff700 1 -- 172.21.15.187:0/3416104940 >> v1:172.21.15.187:6789/0 conn(0x7f5210143c70 legacy=0x7f521009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.501+0000 7f520ffff700 1 -- 172.21.15.187:0/3416104940 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100b1640 con 0x7f5210065190 2020-11-20T12:08:19.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.501+0000 7f520ffff700 1 --2- 172.21.15.187:0/3416104940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210065190 0x7f5210065570 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f521005c8b0 tx=0x7f5200006df0).ready entity=mon.1 client_cookie=542e9124a9bcfeff server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:19.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.501+0000 7f520effd700 1 -- 172.21.15.187:0/3416104940 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200009840 con 0x7f5210065190 2020-11-20T12:08:19.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.501+0000 7f520effd700 1 -- 172.21.15.187:0/3416104940 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52000099a0 con 0x7f5210065190 2020-11-20T12:08:19.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f520effd700 1 -- 172.21.15.187:0/3416104940 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200009d20 con 0x7f5210065190 2020-11-20T12:08:19.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f5215299700 1 -- 172.21.15.187:0/3416104940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210065190 msgr2=0x7f5210065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f5215299700 1 --2- 172.21.15.187:0/3416104940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210065190 0x7f5210065570 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f521005c8b0 tx=0x7f5200006df0).stop 2020-11-20T12:08:19.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f5215299700 1 -- 172.21.15.187:0/3416104940 shutdown_connections 2020-11-20T12:08:19.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f5215299700 1 --2- 172.21.15.187:0/3416104940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210065190 0x7f5210065570 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f5215299700 1 --2- 172.21.15.187:0/3416104940 >> v2:172.21.15.5:3300/0 conn(0x7f5210065f90 0x7f5210143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f5215299700 1 -- 172.21.15.187:0/3416104940 >> 172.21.15.187:0/3416104940 conn(0x7f52100bab30 msgr2=0x7f52100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:19.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f5215299700 1 -- 172.21.15.187:0/3416104940 shutdown_connections 2020-11-20T12:08:19.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.502+0000 7f5215299700 1 -- 172.21.15.187:0/3416104940 wait complete. 2020-11-20T12:08:19.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.504+0000 7f5215299700 1 Processor -- start 2020-11-20T12:08:19.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.505+0000 7f5215299700 1 -- start start 2020-11-20T12:08:19.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f5215299700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5210065f90 0x7f5210147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f5215299700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210143c70 0x7f5210149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f5215299700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52100b2790 0x7f521009fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f5215299700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521009f4a0 con 0x7f52100b2790 2020-11-20T12:08:19.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f5215299700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521009f740 con 0x7f5210143c70 2020-11-20T12:08:19.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f5215299700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5210149bf0 con 0x7f5210065f90 2020-11-20T12:08:19.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f520f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210143c70 0x7f5210149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f520f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210143c70 0x7f5210149700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43606/0 (socket says 172.21.15.187:43606) 2020-11-20T12:08:19.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f520f7fe700 1 -- 172.21.15.187:0/2578391794 learned_addr learned my addr 172.21.15.187:0/2578391794 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:19.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.506+0000 7f5214a98700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52100b2790 0x7f521009fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f520ffff700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5210065f90 0x7f5210147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f520f7fe700 1 -- 172.21.15.187:0/2578391794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5210065f90 msgr2=0x7f5210147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f520f7fe700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5210065f90 0x7f5210147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f520f7fe700 1 -- 172.21.15.187:0/2578391794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52100b2790 msgr2=0x7f521009fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f520f7fe700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52100b2790 0x7f521009fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f520f7fe700 1 -- 172.21.15.187:0/2578391794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521009f120 con 0x7f5210143c70 2020-11-20T12:08:19.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f520f7fe700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210143c70 0x7f5210149700 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f520400db60 tx=0x7f5204008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:19.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f51effff700 1 -- 172.21.15.187:0/2578391794 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204009790 con 0x7f5210143c70 2020-11-20T12:08:19.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f51effff700 1 -- 172.21.15.187:0/2578391794 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52040098f0 con 0x7f5210143c70 2020-11-20T12:08:19.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5210148da0 con 0x7f5210143c70 2020-11-20T12:08:19.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f51effff700 1 -- 172.21.15.187:0/2578391794 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204009cb0 con 0x7f5210143c70 2020-11-20T12:08:19.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.507+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52100b2d70 con 0x7f5210143c70 2020-11-20T12:08:19.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.508+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5210061a20 con 0x7f5210143c70 2020-11-20T12:08:19.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.509+0000 7f51effff700 1 -- 172.21.15.187:0/2578391794 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5204012630 con 0x7f5210143c70 2020-11-20T12:08:19.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.510+0000 7f51effff700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e8056d20 0x7f51e80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.510+0000 7f51effff700 1 -- 172.21.15.187:0/2578391794 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f52040702f0 con 0x7f5210143c70 2020-11-20T12:08:19.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.510+0000 7f520ffff700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e8056d20 0x7f51e80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.511+0000 7f520ffff700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e8056d20 0x7f51e80591e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5210063480 tx=0x7f5200015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:19.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.514+0000 7f51effff700 1 -- 172.21.15.187:0/2578391794 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f52040456c0 con 0x7f5210143c70 2020-11-20T12:08:19.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.702+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52100afc60 con 0x7f5210143c70 2020-11-20T12:08:19.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.705+0000 7f51effff700 1 -- 172.21.15.187:0/2578391794 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f5204048ce0 con 0x7f5210143c70 2020-11-20T12:08:19.709 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:19.710 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":42,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:07:05.571797+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:07:01.490118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2020-11-20T12:07:02.131720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:19.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e8056d20 msgr2=0x7f51e80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e8056d20 0x7f51e80591e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5210063480 tx=0x7f5200015000).stop 2020-11-20T12:08:19.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210143c70 msgr2=0x7f5210149700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210143c70 0x7f5210149700 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f520400db60 tx=0x7f5204008040).stop 2020-11-20T12:08:19.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 shutdown_connections 2020-11-20T12:08:19.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e8056d20 0x7f51e80591e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52100b2790 0x7f521009fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5210065f90 0x7f5210147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 --2- 172.21.15.187:0/2578391794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5210143c70 0x7f5210149700 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.707+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 >> 172.21.15.187:0/2578391794 conn(0x7f52100bab30 msgr2=0x7f52100b6400 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:19.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.708+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 shutdown_connections 2020-11-20T12:08:19.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.708+0000 7f5215299700 1 -- 172.21.15.187:0/2578391794 wait complete. 2020-11-20T12:08:19.733 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:08:19.734 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:08:19.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.971+0000 7fedab59e700 1 Processor -- start 2020-11-20T12:08:19.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.971+0000 7fedab59e700 1 -- start start 2020-11-20T12:08:19.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.972+0000 7fedab59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fedac065190 0x7fedac065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.972+0000 7fedab59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fedac09ca90 0x7fedac09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.973+0000 7fedab59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac0b13e0 con 0x7fedac065f90 2020-11-20T12:08:19.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.973+0000 7fedab59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedac14ab30 con 0x7fedac065190 2020-11-20T12:08:19.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.973+0000 7fedab59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedac062c10 con 0x7fedac09ca90 2020-11-20T12:08:19.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.973+0000 7feda1d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fedac065f90 0x7fedac143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40222/0 (socket says 172.21.15.187:40222) 2020-11-20T12:08:19.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.973+0000 7feda1d9b700 1 -- 172.21.15.187:0/282475592 learned_addr learned my addr 172.21.15.187:0/282475592 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:19.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.973+0000 7fedaad9d700 1 --2- 172.21.15.187:0/282475592 >> v2:172.21.15.187:3300/0 conn(0x7fedac09ca90 0x7fedac09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.973+0000 7fedaa59c700 1 --2- 172.21.15.187:0/282475592 >> v2:172.21.15.5:3300/0 conn(0x7fedac065190 0x7fedac065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.974+0000 7feda9d9b700 1 -- 172.21.15.187:0/282475592 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34468554 0 0) 0x7fedac0b13e0 con 0x7fedac065f90 2020-11-20T12:08:19.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.974+0000 7feda9d9b700 1 -- 172.21.15.187:0/282475592 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8c003240 con 0x7fedac065f90 2020-11-20T12:08:19.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.974+0000 7feda9d9b700 1 -- 172.21.15.187:0/282475592 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2756668807 0 0) 0x7fed8c003240 con 0x7fedac065f90 2020-11-20T12:08:19.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.974+0000 7fedaad9d700 1 -- 172.21.15.187:0/282475592 >> v2:172.21.15.5:3300/0 conn(0x7fedac065190 msgr2=0x7fedac065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:19.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.974+0000 7fedaad9d700 1 --2- 172.21.15.187:0/282475592 >> v2:172.21.15.5:3300/0 conn(0x7fedac065190 0x7fedac065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.974+0000 7fedaad9d700 1 -- 172.21.15.187:0/282475592 >> v1:172.21.15.187:6789/0 conn(0x7fedac065f90 legacy=0x7fedac143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.974+0000 7fedaad9d700 1 -- 172.21.15.187:0/282475592 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac0b1640 con 0x7fedac09ca90 2020-11-20T12:08:19.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.974+0000 7fedaad9d700 1 --2- 172.21.15.187:0/282475592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac09cf20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fedac099f90 tx=0x7feda40091d0).ready entity=mon.1 client_cookie=57254befc6a5235d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:19.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7feda9d9b700 1 -- 172.21.15.187:0/282475592 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed940014e0 con 0x7fedac09ca90 2020-11-20T12:08:19.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7feda9d9b700 1 -- 172.21.15.187:0/282475592 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda400b710 con 0x7fedac09ca90 2020-11-20T12:08:19.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7feda9d9b700 1 -- 172.21.15.187:0/282475592 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda400ba90 con 0x7fedac09ca90 2020-11-20T12:08:19.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7fedab59e700 1 -- 172.21.15.187:0/282475592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 msgr2=0x7fedac09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7fedab59e700 1 --2- 172.21.15.187:0/282475592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac09cf20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fedac099f90 tx=0x7feda40091d0).stop 2020-11-20T12:08:19.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7fedab59e700 1 -- 172.21.15.187:0/282475592 shutdown_connections 2020-11-20T12:08:19.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7fedab59e700 1 --2- 172.21.15.187:0/282475592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac09cf20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7fedab59e700 1 --2- 172.21.15.187:0/282475592 >> v2:172.21.15.5:3300/0 conn(0x7fedac065190 0x7fedac065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7fedab59e700 1 -- 172.21.15.187:0/282475592 >> 172.21.15.187:0/282475592 conn(0x7fedac0bab30 msgr2=0x7fedac0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:19.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7fedab59e700 1 -- 172.21.15.187:0/282475592 shutdown_connections 2020-11-20T12:08:19.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.975+0000 7fedab59e700 1 -- 172.21.15.187:0/282475592 wait complete. 2020-11-20T12:08:19.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.977+0000 7fedab59e700 1 Processor -- start 2020-11-20T12:08:19.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.977+0000 7fedab59e700 1 -- start start 2020-11-20T12:08:19.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedab59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedac065190 0x7fedac0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedab59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedac065f90 0x7fedac09bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedab59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac066f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedab59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac09de60 con 0x7fedac065190 2020-11-20T12:08:19.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedab59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac0b26e0 con 0x7fedac09ca90 2020-11-20T12:08:19.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedab59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac0b4560 con 0x7fedac065f90 2020-11-20T12:08:19.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedaad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac066f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedaad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac066f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43624/0 (socket says 172.21.15.187:43624) 2020-11-20T12:08:19.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.978+0000 7fedaad9d700 1 -- 172.21.15.187:0/403523646 learned_addr learned my addr 172.21.15.187:0/403523646 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:19.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7fedaa59c700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedac065190 0x7fedac0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7feda1d9b700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedac065f90 0x7fedac09bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7fedaad9d700 1 -- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedac065f90 msgr2=0x7fedac09bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7fedaad9d700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedac065f90 0x7fedac09bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7fedaad9d700 1 -- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedac065190 msgr2=0x7fedac0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:19.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7fedaad9d700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedac065190 0x7fedac0b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:19.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7fedaad9d700 1 -- 172.21.15.187:0/403523646 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac0b2b80 con 0x7fedac09ca90 2020-11-20T12:08:19.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7feda1d9b700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedac065f90 0x7fedac09bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:19.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7fedaad9d700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac066f50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7feda400c010 tx=0x7feda4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:19.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7feda2ffd700 1 -- 172.21.15.187:0/403523646 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda40099e0 con 0x7fedac09ca90 2020-11-20T12:08:19.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7feda2ffd700 1 -- 172.21.15.187:0/403523646 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda401d660 con 0x7fedac09ca90 2020-11-20T12:08:19.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.979+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac09dae0 con 0x7fedac09ca90 2020-11-20T12:08:19.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.980+0000 7feda2ffd700 1 -- 172.21.15.187:0/403523646 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda401d9e0 con 0x7fedac09ca90 2020-11-20T12:08:19.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.980+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac067da0 con 0x7fedac09ca90 2020-11-20T12:08:19.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.981+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac0b28a0 con 0x7fedac09ca90 2020-11-20T12:08:19.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.981+0000 7feda2ffd700 1 -- 172.21.15.187:0/403523646 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feda4017040 con 0x7fedac09ca90 2020-11-20T12:08:19.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.982+0000 7feda2ffd700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed88056d20 0x7fed880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:19.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.982+0000 7feda2ffd700 1 -- 172.21.15.187:0/403523646 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feda40723a0 con 0x7fedac09ca90 2020-11-20T12:08:19.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.982+0000 7fedaa59c700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed88056d20 0x7fed880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:19.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.983+0000 7fedaa59c700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed88056d20 0x7fed880591e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fed9c006fd0 tx=0x7fed9c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:19.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:19.986+0000 7feda2ffd700 1 -- 172.21.15.187:0/403523646 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feda40476e0 con 0x7fedac09ca90 2020-11-20T12:08:20.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:20.232+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fedac09dca0 con 0x7fedac09ca90 2020-11-20T12:08:21.190 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:08:21.186+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:08:21.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.195+0000 7feda2ffd700 1 -- 172.21.15.187:0/403523646 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) v1 ==== 161+0+0 (secure 0 0 0) 0x7feda4015070 con 0x7fedac09ca90 2020-11-20T12:08:21.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.198+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed88056d20 msgr2=0x7fed880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.198+0000 7fedab59e700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed88056d20 0x7fed880591e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fed9c006fd0 tx=0x7fed9c008040).stop 2020-11-20T12:08:21.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.198+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 msgr2=0x7fedac066f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.198+0000 7fedab59e700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac066f50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7feda400c010 tx=0x7feda4016000).stop 2020-11-20T12:08:21.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.199+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 shutdown_connections 2020-11-20T12:08:21.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.199+0000 7fedab59e700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed88056d20 0x7fed880591e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.199+0000 7fedab59e700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fedac065190 0x7fedac0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.199+0000 7fedab59e700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedac09ca90 0x7fedac066f50 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.199+0000 7fedab59e700 1 --2- 172.21.15.187:0/403523646 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fedac065f90 0x7fedac09bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.199+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 >> 172.21.15.187:0/403523646 conn(0x7fedac0bab30 msgr2=0x7fedac1412e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:21.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.199+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 shutdown_connections 2020-11-20T12:08:21.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.200+0000 7fedab59e700 1 -- 172.21.15.187:0/403523646 wait complete. 2020-11-20T12:08:21.206 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:08:21.231 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:08:21.316 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:08:21.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.461+0000 7f0261282700 1 Processor -- start 2020-11-20T12:08:21.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.461+0000 7f0261282700 1 -- start start 2020-11-20T12:08:21.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.462+0000 7f0261282700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c09f560 0x7f025c153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:21.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.462+0000 7f0261282700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025c14d110 con 0x7f025c0bfbe0 2020-11-20T12:08:21.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.462+0000 7f0261282700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025c14cbd0 con 0x7f025c0bf800 2020-11-20T12:08:21.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.462+0000 7f0261282700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c14c9d0 con 0x7f025c09f560 2020-11-20T12:08:21.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.463+0000 7f025bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f025c0bf800 0x7f025c14c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40246/0 (socket says 172.21.15.187:40246) 2020-11-20T12:08:21.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.463+0000 7f025bfff700 1 -- 172.21.15.187:0/2651922616 learned_addr learned my addr 172.21.15.187:0/2651922616 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:21.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.463+0000 7f0260a81700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c09f560 0x7f025c153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:21.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.463+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351708853 0 0) 0x7f025c14cbd0 con 0x7f025c0bf800 2020-11-20T12:08:21.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0248003240 con 0x7f025c0bf800 2020-11-20T12:08:21.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 299156321 0 0) 0x7f0248003240 con 0x7f025c0bf800 2020-11-20T12:08:21.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c09f560 msgr2=0x7f025c153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:21.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c09f560 0x7f025c153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 >> v1:172.21.15.5:6789/0 conn(0x7f025c0bfbe0 legacy=0x7f025c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c14ce70 con 0x7f025c0bf800 2020-11-20T12:08:21.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f024c0014e0 con 0x7f025c0bf800 2020-11-20T12:08:21.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 >> v1:172.21.15.187:6789/0 conn(0x7f025c0bf800 legacy=0x7f025c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c09f560 0x7f0248003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:21.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0248003c90 0x7f02480044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:21.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02480049d0 0x7f0248004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:21.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c14c9d0 con 0x7f0248003c90 2020-11-20T12:08:21.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248004220 con 0x7f02480049d0 2020-11-20T12:08:21.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.464+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024800b730 con 0x7f025c09f560 2020-11-20T12:08:21.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.465+0000 7f0260a81700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02480049d0 0x7f0248004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:21.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.465+0000 7f025bfff700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c09f560 0x7f0248003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:21.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.465+0000 7f025b7fe700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0248003c90 0x7f02480044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:21.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.465+0000 7f0260a81700 1 -- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c09f560 msgr2=0x7f0248003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:21.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.465+0000 7f0260a81700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c09f560 0x7f0248003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.465+0000 7f0260a81700 1 -- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0248003c90 msgr2=0x7f02480044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:21.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.465+0000 7f0260a81700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0248003c90 0x7f02480044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.465+0000 7f0260a81700 1 -- 172.21.15.187:0/2651922616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0248003240 con 0x7f02480049d0 2020-11-20T12:08:21.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.466+0000 7f0260a81700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02480049d0 0x7f0248004e50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f0254009d80 tx=0x7f0254008040).ready entity=mon.1 client_cookie=65d483c12edb897c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:21.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.466+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025400c070 con 0x7f02480049d0 2020-11-20T12:08:21.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.466+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025400ea20 con 0x7f02480049d0 2020-11-20T12:08:21.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.466+0000 7f025affd700 1 -- 172.21.15.187:0/2651922616 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025400ede0 con 0x7f02480049d0 2020-11-20T12:08:21.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.467+0000 7f0261282700 1 -- 172.21.15.187:0/2651922616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02480049d0 msgr2=0x7f0248004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.467+0000 7f0261282700 1 --2- 172.21.15.187:0/2651922616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02480049d0 0x7f0248004e50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f0254009d80 tx=0x7f0254008040).stop 2020-11-20T12:08:21.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.467+0000 7f0260a81700 1 -- 172.21.15.187:0/2651922616 reap_dead start 2020-11-20T12:08:21.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.467+0000 7f0261282700 1 -- 172.21.15.187:0/2651922616 shutdown_connections 2020-11-20T12:08:21.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.467+0000 7f0261282700 1 -- 172.21.15.187:0/2651922616 >> 172.21.15.187:0/2651922616 conn(0x7f025c068370 msgr2=0x7f025c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:21.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.467+0000 7f0261282700 1 -- 172.21.15.187:0/2651922616 shutdown_connections 2020-11-20T12:08:21.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.467+0000 7f0261282700 1 -- 172.21.15.187:0/2651922616 wait complete. 2020-11-20T12:08:21.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.468+0000 7f0261282700 1 Processor -- start 2020-11-20T12:08:21.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.469+0000 7f0261282700 1 -- start start 2020-11-20T12:08:21.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.469+0000 7f0261282700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c1346e0 0x7f025c1471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:21.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.469+0000 7f0261282700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f025c141810 0x7f025c141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:21.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.469+0000 7f0261282700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f025c0b4a30 0x7f025c0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:21.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.469+0000 7f0261282700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c14aa30 con 0x7f025c0b4a30 2020-11-20T12:08:21.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.469+0000 7f0261282700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c134300 con 0x7f025c141810 2020-11-20T12:08:21.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.469+0000 7f0261282700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c14ac60 con 0x7f025c1346e0 2020-11-20T12:08:21.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f025c141810 0x7f025c141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:21.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f025c141810 0x7f025c141c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43654/0 (socket says 172.21.15.187:43654) 2020-11-20T12:08:21.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 -- 172.21.15.187:0/403221709 learned_addr learned my addr 172.21.15.187:0/403221709 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:21.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 -- 172.21.15.187:0/403221709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c1346e0 msgr2=0x7f025c1471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c1346e0 0x7f025c1471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:21.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 -- 172.21.15.187:0/403221709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f025c0b4a30 msgr2=0x7f025c0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f025c0b4a30 0x7f025c0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:21.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 -- 172.21.15.187:0/403221709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c14af00 con 0x7f025c141810 2020-11-20T12:08:21.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.470+0000 7f025b7fe700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f025c141810 0x7f025c141c90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f025000e730 tx=0x7f0250009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:21.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.471+0000 7f023bfff700 1 -- 172.21.15.187:0/403221709 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250010550 con 0x7f025c141810 2020-11-20T12:08:21.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.471+0000 7f023bfff700 1 -- 172.21.15.187:0/403221709 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0250010040 con 0x7f025c141810 2020-11-20T12:08:21.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.471+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025c146780 con 0x7f025c141810 2020-11-20T12:08:21.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.471+0000 7f023bfff700 1 -- 172.21.15.187:0/403221709 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02500108c0 con 0x7f025c141810 2020-11-20T12:08:21.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.471+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f025c134c80 con 0x7f025c141810 2020-11-20T12:08:21.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.472+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025c061dc0 con 0x7f025c141810 2020-11-20T12:08:21.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.473+0000 7f023bfff700 1 -- 172.21.15.187:0/403221709 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0250010c30 con 0x7f025c141810 2020-11-20T12:08:21.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.473+0000 7f023bfff700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0234056d20 0x7f02340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:21.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.473+0000 7f023bfff700 1 -- 172.21.15.187:0/403221709 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0250070030 con 0x7f025c141810 2020-11-20T12:08:21.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.474+0000 7f025bfff700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0234056d20 0x7f02340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:21.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.474+0000 7f025bfff700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0234056d20 0x7f02340591e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f024c000dd0 tx=0x7f025c14cd60).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:21.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.478+0000 7f023bfff700 1 -- 172.21.15.187:0/403221709 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f02500453d0 con 0x7f025c141810 2020-11-20T12:08:21.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:21.727+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f025c080050 con 0x7f025c141810 2020-11-20T12:08:22.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.215+0000 7f023bfff700 1 -- 172.21.15.187:0/403221709 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f02500489f0 con 0x7f025c141810 2020-11-20T12:08:22.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0234056d20 msgr2=0x7f02340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:22.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0234056d20 0x7f02340591e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f024c000dd0 tx=0x7f025c14cd60).stop 2020-11-20T12:08:22.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f025c141810 msgr2=0x7f025c141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:22.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f025c141810 0x7f025c141c90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f025000e730 tx=0x7f0250009040).stop 2020-11-20T12:08:22.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 shutdown_connections 2020-11-20T12:08:22.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0234056d20 0x7f02340591e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:22.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f025c0b4a30 0x7f025c0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:22.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f025c1346e0 0x7f025c1471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:22.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 --2- 172.21.15.187:0/403221709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f025c141810 0x7f025c141c90 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:22.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.218+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 >> 172.21.15.187:0/403221709 conn(0x7f025c068370 msgr2=0x7f025c14c240 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:22.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.219+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 shutdown_connections 2020-11-20T12:08:22.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.219+0000 7f0261282700 1 -- 172.21.15.187:0/403221709 wait complete. 2020-11-20T12:08:22.250 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:08:22.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.506+0000 7f0c889b5700 1 Processor -- start 2020-11-20T12:08:22.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.506+0000 7f0c889b5700 1 -- start start 2020-11-20T12:08:22.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.507+0000 7f0c889b5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0c840b6650 0x7f0c840afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:22.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.507+0000 7f0c889b5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840b1300 con 0x7f0c841484d0 2020-11-20T12:08:22.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.507+0000 7f0c889b5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840b15a0 con 0x7f0c84066f70 2020-11-20T12:08:22.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.507+0000 7f0c889b5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c84067730 con 0x7f0c840b6650 2020-11-20T12:08:22.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.508+0000 7f0c8259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0c840b6650 0x7f0c840afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:22.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.508+0000 7f0c8359e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0c84066f70 0x7f0c840aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40270/0 (socket says 172.21.15.187:40270) 2020-11-20T12:08:22.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.508+0000 7f0c8359e700 1 -- 172.21.15.187:0/3628293633 learned_addr learned my addr 172.21.15.187:0/3628293633 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:22.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.509+0000 7f0c81d9b700 1 -- 172.21.15.187:0/3628293633 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377885949 0 0) 0x7f0c840b15a0 con 0x7f0c84066f70 2020-11-20T12:08:22.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.509+0000 7f0c8259c700 1 -- 172.21.15.187:0/3628293633 >> v1:172.21.15.187:6789/0 conn(0x7f0c84066f70 legacy=0x7f0c840aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:22.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.509+0000 7f0c8259c700 1 -- 172.21.15.187:0/3628293633 >> v1:172.21.15.5:6789/0 conn(0x7f0c841484d0 legacy=0x7f0c84061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:22.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.509+0000 7f0c8259c700 1 -- 172.21.15.187:0/3628293633 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c840b1840 con 0x7f0c840b6650 2020-11-20T12:08:22.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.509+0000 7f0c8259c700 1 --2- 172.21.15.187:0/3628293633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c840afa90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f0c7800b9a0 tx=0x7f0c78006df0).ready entity=mon.1 client_cookie=1770c2547a09c39f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:22.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.509+0000 7f0c81d9b700 1 -- 172.21.15.187:0/3628293633 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c78009770 con 0x7f0c840b6650 2020-11-20T12:08:22.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.509+0000 7f0c81d9b700 1 -- 172.21.15.187:0/3628293633 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c780098d0 con 0x7f0c840b6650 2020-11-20T12:08:22.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.509+0000 7f0c81d9b700 1 -- 172.21.15.187:0/3628293633 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c78009c50 con 0x7f0c840b6650 2020-11-20T12:08:22.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.510+0000 7f0c889b5700 1 -- 172.21.15.187:0/3628293633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 msgr2=0x7f0c840afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:22.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.510+0000 7f0c889b5700 1 --2- 172.21.15.187:0/3628293633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c840afa90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f0c7800b9a0 tx=0x7f0c78006df0).stop 2020-11-20T12:08:22.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.510+0000 7f0c889b5700 1 -- 172.21.15.187:0/3628293633 shutdown_connections 2020-11-20T12:08:22.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.510+0000 7f0c889b5700 1 --2- 172.21.15.187:0/3628293633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c840afa90 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:22.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.510+0000 7f0c889b5700 1 -- 172.21.15.187:0/3628293633 >> 172.21.15.187:0/3628293633 conn(0x7f0c840bab30 msgr2=0x7f0c840baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:22.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.510+0000 7f0c889b5700 1 -- 172.21.15.187:0/3628293633 shutdown_connections 2020-11-20T12:08:22.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.510+0000 7f0c889b5700 1 -- 172.21.15.187:0/3628293633 wait complete. 2020-11-20T12:08:22.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.512+0000 7f0c889b5700 1 Processor -- start 2020-11-20T12:08:22.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.512+0000 7f0c889b5700 1 -- start start 2020-11-20T12:08:22.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c889b5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c84066f70 0x7f0c8409cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:22.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c889b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c84142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:22.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c889b5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c841484d0 0x7f0c841431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:22.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c889b5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8414b190 con 0x7f0c841484d0 2020-11-20T12:08:22.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c889b5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c841436e0 con 0x7f0c840b6650 2020-11-20T12:08:22.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c889b5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8409d590 con 0x7f0c84066f70 2020-11-20T12:08:22.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c8259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c84142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:22.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c8259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c84142d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43668/0 (socket says 172.21.15.187:43668) 2020-11-20T12:08:22.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.513+0000 7f0c8259c700 1 -- 172.21.15.187:0/2297682946 learned_addr learned my addr 172.21.15.187:0/2297682946 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:22.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c82d9d700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c84066f70 0x7f0c8409cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:22.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c8259c700 1 -- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c84066f70 msgr2=0x7f0c8409cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:22.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c8359e700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c841484d0 0x7f0c841431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:22.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c8259c700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c84066f70 0x7f0c8409cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:22.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c8259c700 1 -- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c841484d0 msgr2=0x7f0c841431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:22.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c8259c700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c841484d0 0x7f0c841431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:22.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c8259c700 1 -- 172.21.15.187:0/2297682946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8414aa50 con 0x7f0c840b6650 2020-11-20T12:08:22.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c82d9d700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c84066f70 0x7f0c8409cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:22.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c8259c700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c84142d00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f0c78000dd0 tx=0x7f0c78015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:22.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c6affd700 1 -- 172.21.15.187:0/2297682946 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c78011e60 con 0x7f0c840b6650 2020-11-20T12:08:22.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c6affd700 1 -- 172.21.15.187:0/2297682946 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7801d5d0 con 0x7f0c840b6650 2020-11-20T12:08:22.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8414acf0 con 0x7f0c840b6650 2020-11-20T12:08:22.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c6affd700 1 -- 172.21.15.187:0/2297682946 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7801d950 con 0x7f0c840b6650 2020-11-20T12:08:22.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.514+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8415b900 con 0x7f0c840b6650 2020-11-20T12:08:22.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.516+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c840b6e10 con 0x7f0c840b6650 2020-11-20T12:08:22.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.516+0000 7f0c6affd700 1 -- 172.21.15.187:0/2297682946 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0c78025030 con 0x7f0c840b6650 2020-11-20T12:08:22.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.517+0000 7f0c6affd700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c60056d20 0x7f0c600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:22.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.517+0000 7f0c6affd700 1 -- 172.21.15.187:0/2297682946 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0c780719e0 con 0x7f0c840b6650 2020-11-20T12:08:22.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.517+0000 7f0c82d9d700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c60056d20 0x7f0c600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:22.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.518+0000 7f0c82d9d700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c60056d20 0x7f0c600591e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0c8414b380 tx=0x7f0c74008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:22.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.520+0000 7f0c6affd700 1 -- 172.21.15.187:0/2297682946 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0c78046de0 con 0x7f0c840b6650 2020-11-20T12:08:22.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:22.708+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0c8414ae80 con 0x7f0c840b6650 2020-11-20T12:08:23.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.229+0000 7f0c6affd700 1 -- 172.21.15.187:0/2297682946 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0c78023070 con 0x7f0c840b6650 2020-11-20T12:08:23.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.233+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c60056d20 msgr2=0x7f0c600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.233+0000 7f0c889b5700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c60056d20 0x7f0c600591e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0c8414b380 tx=0x7f0c74008040).stop 2020-11-20T12:08:23.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.233+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 msgr2=0x7f0c84142d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.233+0000 7f0c889b5700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c84142d00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f0c78000dd0 tx=0x7f0c78015000).stop 2020-11-20T12:08:23.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.234+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 shutdown_connections 2020-11-20T12:08:23.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.234+0000 7f0c889b5700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c60056d20 0x7f0c600591e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.234+0000 7f0c889b5700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c841484d0 0x7f0c841431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.234+0000 7f0c889b5700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c84066f70 0x7f0c8409cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.234+0000 7f0c889b5700 1 --2- 172.21.15.187:0/2297682946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c840b6650 0x7f0c84142d00 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.234+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 >> 172.21.15.187:0/2297682946 conn(0x7f0c840bab30 msgr2=0x7f0c840afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.234+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 shutdown_connections 2020-11-20T12:08:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.234+0000 7f0c889b5700 1 -- 172.21.15.187:0/2297682946 wait complete. 2020-11-20T12:08:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:08:23.264 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:08:23.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.504+0000 7fbf6f4bb700 1 Processor -- start 2020-11-20T12:08:23.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.504+0000 7fbf6f4bb700 1 -- start start 2020-11-20T12:08:23.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.506+0000 7fbf6f4bb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.506+0000 7fbf6f4bb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbf6809ca90 0x7fbf6809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.506+0000 7fbf6f4bb700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680b13e0 con 0x7fbf68065190 2020-11-20T12:08:23.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.506+0000 7fbf6f4bb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6814ab30 con 0x7fbf6809ca90 2020-11-20T12:08:23.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.506+0000 7fbf6f4bb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68062c10 con 0x7fbf68065f90 2020-11-20T12:08:23.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.506+0000 7fbf6dcb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.507+0000 7fbf6dcb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:44564/0 (socket says 172.21.15.187:44564) 2020-11-20T12:08:23.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.507+0000 7fbf6dcb8700 1 -- 172.21.15.187:0/2868386329 learned_addr learned my addr 172.21.15.187:0/2868386329 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:23.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.507+0000 7fbf6ecba700 1 --2- 172.21.15.187:0/2868386329 >> v2:172.21.15.5:3300/0 conn(0x7fbf6809ca90 0x7fbf6809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.508+0000 7fbf6ecba700 1 -- 172.21.15.187:0/2868386329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 msgr2=0x7fbf68065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:23.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.508+0000 7fbf6ecba700 1 --2- 172.21.15.187:0/2868386329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.508+0000 7fbf6ecba700 1 -- 172.21.15.187:0/2868386329 >> v1:172.21.15.5:6789/0 conn(0x7fbf68065190 legacy=0x7fbf68143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.508+0000 7fbf6ecba700 1 -- 172.21.15.187:0/2868386329 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680b1640 con 0x7fbf6809ca90 2020-11-20T12:08:23.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.508+0000 7fbf6ecba700 1 --2- 172.21.15.187:0/2868386329 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 0x7fbf6809cf20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fbf6814ac70 tx=0x7fbf600091d0).ready entity=mon.0 client_cookie=4c8cb6ab2227a38c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:23.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6d4b7700 1 -- 172.21.15.187:0/2868386329 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000bad0 con 0x7fbf6809ca90 2020-11-20T12:08:23.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6d4b7700 1 -- 172.21.15.187:0/2868386329 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6000bc30 con 0x7fbf6809ca90 2020-11-20T12:08:23.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6d4b7700 1 -- 172.21.15.187:0/2868386329 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60011400 con 0x7fbf6809ca90 2020-11-20T12:08:23.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/2868386329 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 msgr2=0x7fbf6809cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/2868386329 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 0x7fbf6809cf20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fbf6814ac70 tx=0x7fbf600091d0).stop 2020-11-20T12:08:23.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/2868386329 shutdown_connections 2020-11-20T12:08:23.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/2868386329 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 0x7fbf6809cf20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/2868386329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/2868386329 >> 172.21.15.187:0/2868386329 conn(0x7fbf680bab30 msgr2=0x7fbf680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:23.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.509+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/2868386329 shutdown_connections 2020-11-20T12:08:23.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.510+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/2868386329 wait complete. 2020-11-20T12:08:23.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.511+0000 7fbf6f4bb700 1 Processor -- start 2020-11-20T12:08:23.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.511+0000 7fbf6f4bb700 1 -- start start 2020-11-20T12:08:23.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6f4bb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf68065190 0x7fbf6809e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6f4bb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68144230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6f4bb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 0x7fbf6809f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6f4bb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6809e8a0 con 0x7fbf6809ca90 2020-11-20T12:08:23.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6f4bb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680a0200 con 0x7fbf68065190 2020-11-20T12:08:23.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6f4bb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680a04a0 con 0x7fbf68065f90 2020-11-20T12:08:23.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6e4b9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf68065190 0x7fbf6809e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6e4b9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf68065190 0x7fbf6809e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43680/0 (socket says 172.21.15.187:43680) 2020-11-20T12:08:23.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.512+0000 7fbf6e4b9700 1 -- 172.21.15.187:0/1702230699 learned_addr learned my addr 172.21.15.187:0/1702230699 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:23.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.513+0000 7fbf6dcb8700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68144230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.513+0000 7fbf6ecba700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 0x7fbf6809f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.513+0000 7fbf6e4b9700 1 -- 172.21.15.187:0/1702230699 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 msgr2=0x7fbf68144230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.513+0000 7fbf6e4b9700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68144230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.513+0000 7fbf6e4b9700 1 -- 172.21.15.187:0/1702230699 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 msgr2=0x7fbf6809f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.513+0000 7fbf6e4b9700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 0x7fbf6809f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.513+0000 7fbf6e4b9700 1 -- 172.21.15.187:0/1702230699 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf68144720 con 0x7fbf68065190 2020-11-20T12:08:23.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.513+0000 7fbf6e4b9700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf68065190 0x7fbf6809e270 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fbf64001e70 tx=0x7fbf64007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:23.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.514+0000 7fbf567fc700 1 -- 172.21.15.187:0/1702230699 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64007020 con 0x7fbf68065190 2020-11-20T12:08:23.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.514+0000 7fbf567fc700 1 -- 172.21.15.187:0/1702230699 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf64007180 con 0x7fbf68065190 2020-11-20T12:08:23.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.514+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6809fa50 con 0x7fbf68065190 2020-11-20T12:08:23.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.514+0000 7fbf567fc700 1 -- 172.21.15.187:0/1702230699 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6400e560 con 0x7fbf68065190 2020-11-20T12:08:23.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.514+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6809ff40 con 0x7fbf68065190 2020-11-20T12:08:23.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.515+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf680a0660 con 0x7fbf68065190 2020-11-20T12:08:23.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.515+0000 7fbf567fc700 1 -- 172.21.15.187:0/1702230699 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbf64018020 con 0x7fbf68065190 2020-11-20T12:08:23.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.516+0000 7fbf567fc700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf40056d20 0x7fbf400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.516+0000 7fbf567fc700 1 -- 172.21.15.187:0/1702230699 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fbf6406f820 con 0x7fbf68065190 2020-11-20T12:08:23.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.516+0000 7fbf6dcb8700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf40056d20 0x7fbf400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.517+0000 7fbf6dcb8700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf40056d20 0x7fbf400591e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fbf58006a10 tx=0x7fbf58008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:23.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.520+0000 7fbf567fc700 1 -- 172.21.15.187:0/1702230699 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbf64044cf0 con 0x7fbf68065190 2020-11-20T12:08:23.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:23.711+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf68061a20 con 0x7fbf68065190 2020-11-20T12:08:24.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.243+0000 7fbf567fc700 1 -- 172.21.15.187:0/1702230699 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbf64016030 con 0x7fbf68065190 2020-11-20T12:08:24.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.246+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf40056d20 msgr2=0x7fbf400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:24.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.246+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf40056d20 0x7fbf400591e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fbf58006a10 tx=0x7fbf58008040).stop 2020-11-20T12:08:24.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.246+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf68065190 msgr2=0x7fbf6809e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:24.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.246+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf68065190 0x7fbf6809e270 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fbf64001e70 tx=0x7fbf64007c50).stop 2020-11-20T12:08:24.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.247+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 shutdown_connections 2020-11-20T12:08:24.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.247+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbf40056d20 0x7fbf400591e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.247+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf6809ca90 0x7fbf6809f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.247+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf68065190 0x7fbf6809e270 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.247+0000 7fbf6f4bb700 1 --2- 172.21.15.187:0/1702230699 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf68065f90 0x7fbf68144230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.247+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 >> 172.21.15.187:0/1702230699 conn(0x7fbf680bab30 msgr2=0x7fbf680666a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:24.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.247+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 shutdown_connections 2020-11-20T12:08:24.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.247+0000 7fbf6f4bb700 1 -- 172.21.15.187:0/1702230699 wait complete. 2020-11-20T12:08:24.254 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:08:24.280 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:08:24.280 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:08:24.281 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:08:24.281 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:08:24.281 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:08:24.282 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:08:24.282 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:08:24.282 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:08:24.308 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:08:24.308 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:08:24.309 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:08:24.310 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:08:24.367 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:08:24.368 DEBUG:teuthology.parallel:result is None 2020-11-20T12:08:24.370 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:08:24.371 DEBUG:teuthology.parallel:result is None 2020-11-20T12:08:24.371 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:08:24.538 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:08:24.566 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:08:24.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.598+0000 7f3b1359e700 1 Processor -- start 2020-11-20T12:08:24.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.598+0000 7f3b1359e700 1 -- start start 2020-11-20T12:08:24.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b141484d0 0x7f3b14061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:24.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3b140622c0 0x7f3b140b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:24.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b140b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:24.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b14067730 con 0x7f3b140622c0 2020-11-20T12:08:24.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b140b1300 con 0x7f3b141484d0 2020-11-20T12:08:24.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140b15a0 con 0x7f3b14066f70 2020-11-20T12:08:24.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b141484d0 0x7f3b14061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:24.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b141484d0 0x7f3b14061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43700/0 (socket says 172.21.15.187:43700) 2020-11-20T12:08:24.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.599+0000 7f3b1259c700 1 -- 172.21.15.187:0/824841885 learned_addr learned my addr 172.21.15.187:0/824841885 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:24.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b09d9b700 1 --2- 172.21.15.187:0/824841885 >> v2:172.21.15.5:3300/0 conn(0x7f3b140622c0 0x7f3b140b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:24.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b12d9d700 1 --2- 172.21.15.187:0/824841885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b140b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:24.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b1259c700 1 -- 172.21.15.187:0/824841885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 msgr2=0x7f3b140b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:24.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b1259c700 1 --2- 172.21.15.187:0/824841885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b140b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b1259c700 1 -- 172.21.15.187:0/824841885 >> v2:172.21.15.5:3300/0 conn(0x7f3b140622c0 msgr2=0x7f3b140b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:24.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b1259c700 1 --2- 172.21.15.187:0/824841885 >> v2:172.21.15.5:3300/0 conn(0x7f3b140622c0 0x7f3b140b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b1259c700 1 -- 172.21.15.187:0/824841885 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b140b1840 con 0x7f3b141484d0 2020-11-20T12:08:24.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b1259c700 1 --2- 172.21.15.187:0/824841885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b14061dd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f3b140b85e0 tx=0x7f3b04006eb0).ready entity=mon.1 client_cookie=185eab0cfce57368 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:24.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.600+0000 7f3b11d9b700 1 -- 172.21.15.187:0/824841885 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04009810 con 0x7f3b141484d0 2020-11-20T12:08:24.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b11d9b700 1 -- 172.21.15.187:0/824841885 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b04009970 con 0x7f3b141484d0 2020-11-20T12:08:24.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b11d9b700 1 -- 172.21.15.187:0/824841885 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04009cf0 con 0x7f3b141484d0 2020-11-20T12:08:24.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 -- 172.21.15.187:0/824841885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 msgr2=0x7f3b14061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:24.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 --2- 172.21.15.187:0/824841885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b14061dd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f3b140b85e0 tx=0x7f3b04006eb0).stop 2020-11-20T12:08:24.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 -- 172.21.15.187:0/824841885 shutdown_connections 2020-11-20T12:08:24.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 --2- 172.21.15.187:0/824841885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b140b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 --2- 172.21.15.187:0/824841885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b14061dd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 --2- 172.21.15.187:0/824841885 >> v2:172.21.15.5:3300/0 conn(0x7f3b140622c0 0x7f3b140b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 -- 172.21.15.187:0/824841885 >> 172.21.15.187:0/824841885 conn(0x7f3b140bab30 msgr2=0x7f3b140baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:24.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 -- 172.21.15.187:0/824841885 shutdown_connections 2020-11-20T12:08:24.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.601+0000 7f3b1359e700 1 -- 172.21.15.187:0/824841885 wait complete. 2020-11-20T12:08:24.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.603+0000 7f3b1359e700 1 Processor -- start 2020-11-20T12:08:24.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.603+0000 7f3b1359e700 1 -- start start 2020-11-20T12:08:24.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b1359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b140622c0 0x7f3b1409bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:24.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b1359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b1413e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:24.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b1359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b1413d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:24.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b1359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1413cc50 con 0x7f3b140622c0 2020-11-20T12:08:24.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b1359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1413cef0 con 0x7f3b141484d0 2020-11-20T12:08:24.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b1359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1413d190 con 0x7f3b14066f70 2020-11-20T12:08:24.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b12d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b1413d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:24.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b12d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b1413d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43710/0 (socket says 172.21.15.187:43710) 2020-11-20T12:08:24.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b12d9d700 1 -- 172.21.15.187:0/3208462460 learned_addr learned my addr 172.21.15.187:0/3208462460 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:24.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b1259c700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b140622c0 0x7f3b1409bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:24.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.604+0000 7f3b09d9b700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b1413e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:24.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b12d9d700 1 -- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 msgr2=0x7f3b1413e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:24.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b12d9d700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b1413e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b12d9d700 1 -- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b140622c0 msgr2=0x7f3b1409bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:24.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b12d9d700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b140622c0 0x7f3b1409bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:24.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b12d9d700 1 -- 172.21.15.187:0/3208462460 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1409c2b0 con 0x7f3b141484d0 2020-11-20T12:08:24.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b1259c700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b140622c0 0x7f3b1409bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:24.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b09d9b700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b1413e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:24.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b12d9d700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b1413d630 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f3b0c00eb40 tx=0x7f3b0c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:24.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b0affd700 1 -- 172.21.15.187:0/3208462460 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c00c7a0 con 0x7f3b141484d0 2020-11-20T12:08:24.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b0affd700 1 -- 172.21.15.187:0/3208462460 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0c00c900 con 0x7f3b141484d0 2020-11-20T12:08:24.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b0affd700 1 -- 172.21.15.187:0/3208462460 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c00cca0 con 0x7f3b141484d0 2020-11-20T12:08:24.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1413ed90 con 0x7f3b141484d0 2020-11-20T12:08:24.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.605+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1413dce0 con 0x7f3b141484d0 2020-11-20T12:08:24.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.606+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b14061ce0 con 0x7f3b141484d0 2020-11-20T12:08:24.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.607+0000 7f3b0affd700 1 -- 172.21.15.187:0/3208462460 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3b0c013640 con 0x7f3b141484d0 2020-11-20T12:08:24.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.608+0000 7f3b0affd700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3af0056d20 0x7f3af00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:24.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.608+0000 7f3b0affd700 1 -- 172.21.15.187:0/3208462460 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f3b0c06f810 con 0x7f3b141484d0 2020-11-20T12:08:24.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.608+0000 7f3b1259c700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3af0056d20 0x7f3af00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:24.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.609+0000 7f3b1259c700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3af0056d20 0x7f3af00591e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3b0400b1b0 tx=0x7f3b04015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:24.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.612+0000 7f3b0affd700 1 -- 172.21.15.187:0/3208462460 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3b0c044f40 con 0x7f3b141484d0 2020-11-20T12:08:24.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:24.813+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3b1413dce0 con 0x7f3b141484d0 2020-11-20T12:08:25.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.245+0000 7f3b0affd700 1 -- 172.21.15.187:0/3208462460 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v46) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3b1413dce0 con 0x7f3b141484d0 2020-11-20T12:08:25.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3af0056d20 msgr2=0x7f3af00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3af0056d20 0x7f3af00591e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3b0400b1b0 tx=0x7f3b04015000).stop 2020-11-20T12:08:25.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 msgr2=0x7f3b1413d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b1413d630 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f3b0c00eb40 tx=0x7f3b0c008040).stop 2020-11-20T12:08:25.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 shutdown_connections 2020-11-20T12:08:25.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3af0056d20 0x7f3af00591e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b140622c0 0x7f3b1409bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b141484d0 0x7f3b1413d630 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 --2- 172.21.15.187:0/3208462460 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b14066f70 0x7f3b1413e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.249+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 >> 172.21.15.187:0/3208462460 conn(0x7f3b140bab30 msgr2=0x7f3b140b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:25.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.250+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 shutdown_connections 2020-11-20T12:08:25.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.250+0000 7f3b1359e700 1 -- 172.21.15.187:0/3208462460 wait complete. 2020-11-20T12:08:25.257 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:08:25.282 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:08:25.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.520+0000 7fd5132c0700 1 Processor -- start 2020-11-20T12:08:25.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.520+0000 7fd5132c0700 1 -- start start 2020-11-20T12:08:25.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.521+0000 7fd5132c0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065190 0x7fd50c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.522+0000 7fd5132c0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd50c09ca90 0x7fd50c09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.522+0000 7fd5132c0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c0b13e0 con 0x7fd50c065f90 2020-11-20T12:08:25.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.522+0000 7fd5132c0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd50c14ab30 con 0x7fd50c09ca90 2020-11-20T12:08:25.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.522+0000 7fd5132c0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c062c10 con 0x7fd50c065190 2020-11-20T12:08:25.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.523+0000 7fd512abf700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd50c09ca90 0x7fd50c09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.523+0000 7fd511abd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd50c065f90 0x7fd50c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33832/0 (socket says 172.21.15.187:33832) 2020-11-20T12:08:25.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.523+0000 7fd511abd700 1 -- 172.21.15.187:0/2873537259 learned_addr learned my addr 172.21.15.187:0/2873537259 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:25.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.523+0000 7fd5122be700 1 --2- 172.21.15.187:0/2873537259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065190 0x7fd50c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.524+0000 7fd512abf700 1 -- 172.21.15.187:0/2873537259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065190 msgr2=0x7fd50c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:25.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.524+0000 7fd512abf700 1 --2- 172.21.15.187:0/2873537259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065190 0x7fd50c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.524+0000 7fd512abf700 1 -- 172.21.15.187:0/2873537259 >> v1:172.21.15.5:6789/0 conn(0x7fd50c065f90 legacy=0x7fd50c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.524+0000 7fd512abf700 1 -- 172.21.15.187:0/2873537259 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0b1640 con 0x7fd50c09ca90 2020-11-20T12:08:25.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.524+0000 7fd5112bc700 1 -- 172.21.15.187:0/2873537259 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419382700 0 0) 0x7fd50c0b13e0 con 0x7fd50c065f90 2020-11-20T12:08:25.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.524+0000 7fd512abf700 1 --2- 172.21.15.187:0/2873537259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c09ca90 0x7fd50c09cf40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd50c14ac70 tx=0x7fd5040091d0).ready entity=mon.0 client_cookie=e598655059e8f6c4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:25.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.525+0000 7fd5112bc700 1 -- 172.21.15.187:0/2873537259 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50400bad0 con 0x7fd50c09ca90 2020-11-20T12:08:25.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.525+0000 7fd5112bc700 1 -- 172.21.15.187:0/2873537259 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50400bc30 con 0x7fd50c09ca90 2020-11-20T12:08:25.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.525+0000 7fd5112bc700 1 -- 172.21.15.187:0/2873537259 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504011400 con 0x7fd50c09ca90 2020-11-20T12:08:25.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.525+0000 7fd5132c0700 1 -- 172.21.15.187:0/2873537259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c09ca90 msgr2=0x7fd50c09cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.525+0000 7fd5132c0700 1 --2- 172.21.15.187:0/2873537259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c09ca90 0x7fd50c09cf40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd50c14ac70 tx=0x7fd5040091d0).stop 2020-11-20T12:08:25.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.526+0000 7fd5132c0700 1 -- 172.21.15.187:0/2873537259 shutdown_connections 2020-11-20T12:08:25.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.526+0000 7fd5132c0700 1 --2- 172.21.15.187:0/2873537259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c09ca90 0x7fd50c09cf40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.526+0000 7fd5132c0700 1 --2- 172.21.15.187:0/2873537259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065190 0x7fd50c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.526+0000 7fd5132c0700 1 -- 172.21.15.187:0/2873537259 >> 172.21.15.187:0/2873537259 conn(0x7fd50c0bab30 msgr2=0x7fd50c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:25.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.526+0000 7fd5132c0700 1 -- 172.21.15.187:0/2873537259 shutdown_connections 2020-11-20T12:08:25.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.526+0000 7fd5132c0700 1 -- 172.21.15.187:0/2873537259 wait complete. 2020-11-20T12:08:25.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.528+0000 7fd5132c0700 1 Processor -- start 2020-11-20T12:08:25.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.528+0000 7fd5132c0700 1 -- start start 2020-11-20T12:08:25.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.529+0000 7fd5132c0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c065190 0x7fd50c145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.529+0000 7fd5132c0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065f90 0x7fd50c141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.529+0000 7fd5132c0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd50c09ca90 0x7fd50c141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.529+0000 7fd5132c0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c09e760 con 0x7fd50c065190 2020-11-20T12:08:25.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.529+0000 7fd5132c0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c09ea00 con 0x7fd50c09ca90 2020-11-20T12:08:25.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.529+0000 7fd5132c0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c09eca0 con 0x7fd50c065f90 2020-11-20T12:08:25.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd512abf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd50c09ca90 0x7fd50c141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd512abf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd50c09ca90 0x7fd50c141a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43724/0 (socket says 172.21.15.187:43724) 2020-11-20T12:08:25.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd512abf700 1 -- 172.21.15.187:0/4064103605 learned_addr learned my addr 172.21.15.187:0/4064103605 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:25.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd5122be700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c065190 0x7fd50c145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd511abd700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065f90 0x7fd50c141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd512abf700 1 -- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065f90 msgr2=0x7fd50c141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd512abf700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065f90 0x7fd50c141510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd512abf700 1 -- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c065190 msgr2=0x7fd50c145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd512abf700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c065190 0x7fd50c145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd512abf700 1 -- 172.21.15.187:0/4064103605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c068320 con 0x7fd50c09ca90 2020-11-20T12:08:25.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.530+0000 7fd5122be700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c065190 0x7fd50c145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:08:25.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.531+0000 7fd512abf700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd50c09ca90 0x7fd50c141a00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7fd504009210 tx=0x7fd504016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:25.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.531+0000 7fd4fa7fc700 1 -- 172.21.15.187:0/4064103605 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504010800 con 0x7fd50c09ca90 2020-11-20T12:08:25.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.531+0000 7fd4fa7fc700 1 -- 172.21.15.187:0/4064103605 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd504010960 con 0x7fd50c09ca90 2020-11-20T12:08:25.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.531+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c1455f0 con 0x7fd50c09ca90 2020-11-20T12:08:25.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.531+0000 7fd4fa7fc700 1 -- 172.21.15.187:0/4064103605 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504010d40 con 0x7fd50c09ca90 2020-11-20T12:08:25.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.531+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50c09fce0 con 0x7fd50c09ca90 2020-11-20T12:08:25.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.532+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50c146420 con 0x7fd50c09ca90 2020-11-20T12:08:25.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.533+0000 7fd4fa7fc700 1 -- 172.21.15.187:0/4064103605 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd504017070 con 0x7fd50c09ca90 2020-11-20T12:08:25.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.534+0000 7fd4fa7fc700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e4056d20 0x7fd4e40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.534+0000 7fd4fa7fc700 1 -- 172.21.15.187:0/4064103605 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd504072410 con 0x7fd50c09ca90 2020-11-20T12:08:25.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.534+0000 7fd5122be700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e4056d20 0x7fd4e40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.535+0000 7fd5122be700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e4056d20 0x7fd4e40591e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fd508006fd0 tx=0x7fd508008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:25.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.538+0000 7fd4fa7fc700 1 -- 172.21.15.187:0/4064103605 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd504047a80 con 0x7fd50c09ca90 2020-11-20T12:08:25.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.791+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd50c14a2c0 con 0x7fd50c09ca90 2020-11-20T12:08:25.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.792+0000 7fd4fa7fc700 1 -- 172.21.15.187:0/4064103605 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+4570 (secure 0 0 0) 0x7fd504015020 con 0x7fd50c09ca90 2020-11-20T12:08:25.796 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:25.797 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:08:25.797 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:08:25.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.797 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:08:25.798 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:08:25.798 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.798 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:08:25.798 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.798 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.800 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:08:25.800 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:08:25.800 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.800 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:08:25.800 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.801 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.801 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.801 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.802 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:08:25.802 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:08:25.802 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.802 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.803 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.803 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.803 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.804 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:08:25.804 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:08:25.804 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.804 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.804 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.805 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.805 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.805 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:08:25.806 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:08:25.806 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.806 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.806 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.806 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.807 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.807 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:08:25.807 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:08:25.808 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.808 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.808 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.808 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.809 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.809 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:08:25.809 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:08:25.810 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.810 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.810 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.810 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.810 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.811 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:08:25.811 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:08:25.811 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.812 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.812 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.812 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.812 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.813 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:08:25.813 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:08:25.813 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.813 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.814 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.814 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.814 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.815 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:08:25.815 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:08:25.815 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.815 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.815 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.816 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.816 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.817 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:08:25.817 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:08:25.817 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.817 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:08:25.817 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:08:25.818 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:08:25.818 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.818 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:08:25.819 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:08:25.819 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.819 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:08:25.819 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:08:25.819 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:08:25.820 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:08:25.820 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.820 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:08:25.821 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:08:25.821 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.821 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:08:25.821 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:08:25.821 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:08:25.822 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:08:25.822 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.822 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:08:25.823 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:08:25.823 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.823 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:08:25.823 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:08:25.823 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:08:25.824 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:08:25.824 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.825 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:08:25.825 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:08:25.825 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.825 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:08:25.825 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:08:25.826 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:08:25.826 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.826 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.827 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:08:25.827 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:08:25.827 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.827 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:08:25.828 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:08:25.828 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.828 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:25.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:25.829 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:08:25.829 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:08:25.829 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:08:25.829 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:08:25.829 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:08:25.830 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:08:25.830 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.830 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:25.830 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:08:25.830 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:08:25.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e4056d20 msgr2=0x7fd4e40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e4056d20 0x7fd4e40591e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fd508006fd0 tx=0x7fd508008040).stop 2020-11-20T12:08:25.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd50c09ca90 msgr2=0x7fd50c141a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd50c09ca90 0x7fd50c141a00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7fd504009210 tx=0x7fd504016000).stop 2020-11-20T12:08:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 shutdown_connections 2020-11-20T12:08:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e4056d20 0x7fd4e40591e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd50c065190 0x7fd50c145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd50c09ca90 0x7fd50c141a00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 --2- 172.21.15.187:0/4064103605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd50c065f90 0x7fd50c141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 >> 172.21.15.187:0/4064103605 conn(0x7fd50c0bab30 msgr2=0x7fd50c1497d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:25.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 shutdown_connections 2020-11-20T12:08:25.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:25.794+0000 7fd5132c0700 1 -- 172.21.15.187:0/4064103605 wait complete. 2020-11-20T12:08:25.835 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:26.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.073+0000 7f7439f73700 1 Processor -- start 2020-11-20T12:08:26.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.073+0000 7f7439f73700 1 -- start start 2020-11-20T12:08:26.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.075+0000 7f7439f73700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 0x7f7434066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.075+0000 7f7439f73700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74340683f0 0x7f7434065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.075+0000 7f7439f73700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743414c170 con 0x7f7434065680 2020-11-20T12:08:26.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.075+0000 7f7439f73700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74341497f0 con 0x7f74340683f0 2020-11-20T12:08:26.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.075+0000 7f7439f73700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743414c410 con 0x7f7434145cc0 2020-11-20T12:08:26.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.075+0000 7f7433fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74340683f0 0x7f7434065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.076+0000 7f7439772700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7434065680 0x7f7434061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:33846/0 (socket says 172.21.15.187:33846) 2020-11-20T12:08:26.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.076+0000 7f7433fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74340683f0 0x7f7434065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43732/0 (socket says 172.21.15.187:43732) 2020-11-20T12:08:26.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.076+0000 7f7438f71700 1 --2- 172.21.15.187:0/747629364 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 0x7f7434066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.076+0000 7f7439772700 1 -- 172.21.15.187:0/747629364 learned_addr learned my addr 172.21.15.187:0/747629364 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:26.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.076+0000 7f7433fff700 1 -- 172.21.15.187:0/747629364 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 msgr2=0x7f7434066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:26.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.076+0000 7f7433fff700 1 --2- 172.21.15.187:0/747629364 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 0x7f7434066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.076+0000 7f7433fff700 1 -- 172.21.15.187:0/747629364 >> v1:172.21.15.5:6789/0 conn(0x7f7434065680 legacy=0x7f7434061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.076+0000 7f7433fff700 1 -- 172.21.15.187:0/747629364 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743414c6b0 con 0x7f74340683f0 2020-11-20T12:08:26.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.077+0000 7f7433fff700 1 --2- 172.21.15.187:0/747629364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74340683f0 0x7f7434065190 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f742000b9a0 tx=0x7f7420006df0).ready entity=mon.1 client_cookie=42c36b1dcc4e5a1b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:26.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.077+0000 7f74337fe700 1 -- 172.21.15.187:0/747629364 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74200097b0 con 0x7f74340683f0 2020-11-20T12:08:26.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.077+0000 7f74337fe700 1 -- 172.21.15.187:0/747629364 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7420009910 con 0x7f74340683f0 2020-11-20T12:08:26.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.077+0000 7f74337fe700 1 -- 172.21.15.187:0/747629364 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420009c90 con 0x7f74340683f0 2020-11-20T12:08:26.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.078+0000 7f7439f73700 1 -- 172.21.15.187:0/747629364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74340683f0 msgr2=0x7f7434065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.078+0000 7f7439f73700 1 --2- 172.21.15.187:0/747629364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74340683f0 0x7f7434065190 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f742000b9a0 tx=0x7f7420006df0).stop 2020-11-20T12:08:26.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.078+0000 7f7439f73700 1 -- 172.21.15.187:0/747629364 shutdown_connections 2020-11-20T12:08:26.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.078+0000 7f7439f73700 1 --2- 172.21.15.187:0/747629364 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 0x7f7434066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.078+0000 7f7439f73700 1 --2- 172.21.15.187:0/747629364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74340683f0 0x7f7434065190 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.078+0000 7f7439f73700 1 -- 172.21.15.187:0/747629364 >> 172.21.15.187:0/747629364 conn(0x7f74340bab30 msgr2=0x7f74340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:26.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.078+0000 7f7439f73700 1 -- 172.21.15.187:0/747629364 shutdown_connections 2020-11-20T12:08:26.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.078+0000 7f7439f73700 1 -- 172.21.15.187:0/747629364 wait complete. 2020-11-20T12:08:26.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.080+0000 7f7439f73700 1 Processor -- start 2020-11-20T12:08:26.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.080+0000 7f7439f73700 1 -- start start 2020-11-20T12:08:26.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.081+0000 7f7439f73700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7434065680 0x7f743415c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.081+0000 7f7439f73700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74340683f0 0x7f743415fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.081+0000 7f7439f73700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 0x7f7434144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.081+0000 7f7439f73700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74341605e0 con 0x7f74340683f0 2020-11-20T12:08:26.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.081+0000 7f7439f73700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7434160880 con 0x7f7434065680 2020-11-20T12:08:26.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.081+0000 7f7439f73700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743415c940 con 0x7f7434145cc0 2020-11-20T12:08:26.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7434065680 0x7f743415c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7434065680 0x7f743415c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43734/0 (socket says 172.21.15.187:43734) 2020-11-20T12:08:26.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 -- 172.21.15.187:0/1529185430 learned_addr learned my addr 172.21.15.187:0/1529185430 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:26.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 -- 172.21.15.187:0/1529185430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 msgr2=0x7f7434144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 0x7f7434144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:26.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 -- 172.21.15.187:0/1529185430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74340683f0 msgr2=0x7f743415fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74340683f0 0x7f743415fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:26.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 -- 172.21.15.187:0/1529185430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743409ce80 con 0x7f7434065680 2020-11-20T12:08:26.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.082+0000 7f7438f71700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7434065680 0x7f743415c450 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f742800ebd0 tx=0x7f7428006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:26.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.083+0000 7f7430ff9700 1 -- 172.21.15.187:0/1529185430 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742800b9a0 con 0x7f7434065680 2020-11-20T12:08:26.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.083+0000 7f7430ff9700 1 -- 172.21.15.187:0/1529185430 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742800bb00 con 0x7f7434065680 2020-11-20T12:08:26.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.083+0000 7f7430ff9700 1 -- 172.21.15.187:0/1529185430 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742800be80 con 0x7f7434065680 2020-11-20T12:08:26.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.083+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74340b6bf0 con 0x7f7434065680 2020-11-20T12:08:26.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.083+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743409d3d0 con 0x7f7434065680 2020-11-20T12:08:26.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.084+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7434061a20 con 0x7f7434065680 2020-11-20T12:08:26.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.085+0000 7f7430ff9700 1 -- 172.21.15.187:0/1529185430 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7428012600 con 0x7f7434065680 2020-11-20T12:08:26.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.085+0000 7f7430ff9700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d50 0x7f740c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.085+0000 7f7430ff9700 1 -- 172.21.15.187:0/1529185430 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f7428070000 con 0x7f7434065680 2020-11-20T12:08:26.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.086+0000 7f7433fff700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d50 0x7f740c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.086+0000 7f7433fff700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d50 0x7f740c059210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7434067210 tx=0x7f7420015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:26.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.091+0000 7f7430ff9700 1 -- 172.21.15.187:0/1529185430 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7428045780 con 0x7f7434065680 2020-11-20T12:08:26.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.277+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74340b6de0 con 0x7f7434065680 2020-11-20T12:08:26.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.279+0000 7f7430ff9700 1 -- 172.21.15.187:0/1529185430 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13627 (secure 0 0 0) 0x7f7428048da0 con 0x7f7434065680 2020-11-20T12:08:26.283 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:26.284 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":46,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:08:25.235888+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:26.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d50 msgr2=0x7f740c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d50 0x7f740c059210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7434067210 tx=0x7f7420015000).stop 2020-11-20T12:08:26.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7434065680 msgr2=0x7f743415c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7434065680 0x7f743415c450 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f742800ebd0 tx=0x7f7428006df0).stop 2020-11-20T12:08:26.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 shutdown_connections 2020-11-20T12:08:26.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d50 0x7f740c059210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7434145cc0 0x7f7434144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:26.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7434065680 0x7f743415c450 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 --2- 172.21.15.187:0/1529185430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74340683f0 0x7f743415fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:26.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.281+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 >> 172.21.15.187:0/1529185430 conn(0x7f74340bab30 msgr2=0x7f7434145000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:26.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.282+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 shutdown_connections 2020-11-20T12:08:26.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.282+0000 7f7439f73700 1 -- 172.21.15.187:0/1529185430 wait complete. 2020-11-20T12:08:26.307 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:08:26.308 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:08:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.542+0000 7f899d468700 1 Processor -- start 2020-11-20T12:08:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.542+0000 7f899d468700 1 -- start start 2020-11-20T12:08:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.543+0000 7f899d468700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89981484d0 0x7f8998061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.543+0000 7f899d468700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980b1300 con 0x7f8998066f70 2020-11-20T12:08:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.543+0000 7f899d468700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980b15a0 con 0x7f89980622c0 2020-11-20T12:08:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.543+0000 7f899d468700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8998067730 con 0x7f89981484d0 2020-11-20T12:08:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.544+0000 7f8997fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89981484d0 0x7f8998061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.544+0000 7f898f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f89980622c0 0x7f89980b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40352/0 (socket says 172.21.15.187:40352) 2020-11-20T12:08:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.544+0000 7f8997fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89981484d0 0x7f8998061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43742/0 (socket says 172.21.15.187:43742) 2020-11-20T12:08:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.544+0000 7f898f7fe700 1 -- 172.21.15.187:0/1339185876 learned_addr learned my addr 172.21.15.187:0/1339185876 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:26.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.545+0000 7f8997fff700 1 -- 172.21.15.187:0/1339185876 >> v1:172.21.15.187:6789/0 conn(0x7f89980622c0 legacy=0x7f89980b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.545+0000 7f8997fff700 1 -- 172.21.15.187:0/1339185876 >> v1:172.21.15.5:6789/0 conn(0x7f8998066f70 legacy=0x7f89980b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.545+0000 7f8997fff700 1 -- 172.21.15.187:0/1339185876 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89980b1840 con 0x7f89981484d0 2020-11-20T12:08:26.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.545+0000 7f8997fff700 1 --2- 172.21.15.187:0/1339185876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89981484d0 0x7f8998061dd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f8988009c90 tx=0x7f89880091d0).ready entity=mon.1 client_cookie=610c3240cb3f33ef server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:26.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.546+0000 7f89977fe700 1 -- 172.21.15.187:0/1339185876 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898800bbc0 con 0x7f89981484d0 2020-11-20T12:08:26.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.546+0000 7f89977fe700 1 -- 172.21.15.187:0/1339185876 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898800bd20 con 0x7f89981484d0 2020-11-20T12:08:26.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.546+0000 7f89977fe700 1 -- 172.21.15.187:0/1339185876 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988011610 con 0x7f89981484d0 2020-11-20T12:08:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.546+0000 7f899d468700 1 -- 172.21.15.187:0/1339185876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89981484d0 msgr2=0x7f8998061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.546+0000 7f899d468700 1 --2- 172.21.15.187:0/1339185876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89981484d0 0x7f8998061dd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f8988009c90 tx=0x7f89880091d0).stop 2020-11-20T12:08:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.546+0000 7f899d468700 1 -- 172.21.15.187:0/1339185876 shutdown_connections 2020-11-20T12:08:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.546+0000 7f899d468700 1 --2- 172.21.15.187:0/1339185876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89981484d0 0x7f8998061dd0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.546+0000 7f899d468700 1 -- 172.21.15.187:0/1339185876 >> 172.21.15.187:0/1339185876 conn(0x7f89980bab30 msgr2=0x7f89980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:26.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.547+0000 7f899d468700 1 -- 172.21.15.187:0/1339185876 shutdown_connections 2020-11-20T12:08:26.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.547+0000 7f899d468700 1 -- 172.21.15.187:0/1339185876 wait complete. 2020-11-20T12:08:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.548+0000 7f899d468700 1 Processor -- start 2020-11-20T12:08:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.549+0000 7f899d468700 1 -- start start 2020-11-20T12:08:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f899d468700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89980622c0 0x7f8998138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f899d468700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998066f70 0x7f899815b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f899d468700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89981484d0 0x7f89980b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f899d468700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8998139050 con 0x7f8998066f70 2020-11-20T12:08:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f899d468700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980b4950 con 0x7f89980622c0 2020-11-20T12:08:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f899d468700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89980a07c0 con 0x7f89981484d0 2020-11-20T12:08:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f8997fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89980622c0 0x7f8998138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f8997fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89980622c0 0x7f8998138b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43750/0 (socket says 172.21.15.187:43750) 2020-11-20T12:08:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.550+0000 7f8997fff700 1 -- 172.21.15.187:0/4251884442 learned_addr learned my addr 172.21.15.187:0/4251884442 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.551+0000 7f898f7fe700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998066f70 0x7f899815b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.551+0000 7f899cc67700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89981484d0 0x7f89980b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.551+0000 7f8997fff700 1 -- 172.21.15.187:0/4251884442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89981484d0 msgr2=0x7f89980b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.551+0000 7f8997fff700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89981484d0 0x7f89980b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.551+0000 7f8997fff700 1 -- 172.21.15.187:0/4251884442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998066f70 msgr2=0x7f899815b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.551+0000 7f8997fff700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998066f70 0x7f899815b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.551+0000 7f8997fff700 1 -- 172.21.15.187:0/4251884442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89980b4640 con 0x7f89980622c0 2020-11-20T12:08:26.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.552+0000 7f8997fff700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89980622c0 0x7f8998138b60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f8988009fd0 tx=0x7f8988016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:26.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.552+0000 7f8994ff9700 1 -- 172.21.15.187:0/4251884442 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898801c6c0 con 0x7f89980622c0 2020-11-20T12:08:26.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.552+0000 7f8994ff9700 1 -- 172.21.15.187:0/4251884442 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898801c820 con 0x7f89980622c0 2020-11-20T12:08:26.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.552+0000 7f8994ff9700 1 -- 172.21.15.187:0/4251884442 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898801cba0 con 0x7f89980622c0 2020-11-20T12:08:26.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.552+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89980a04b0 con 0x7f89980622c0 2020-11-20T12:08:26.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.552+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899815be90 con 0x7f89980622c0 2020-11-20T12:08:26.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.553+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89980a0980 con 0x7f89980622c0 2020-11-20T12:08:26.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.554+0000 7f8994ff9700 1 -- 172.21.15.187:0/4251884442 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8988017070 con 0x7f89980622c0 2020-11-20T12:08:26.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.555+0000 7f8994ff9700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8974056d20 0x7f89740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:26.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.555+0000 7f8994ff9700 1 -- 172.21.15.187:0/4251884442 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8988072110 con 0x7f89980622c0 2020-11-20T12:08:26.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.555+0000 7f898f7fe700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8974056d20 0x7f89740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:26.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.555+0000 7f898f7fe700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8974056d20 0x7f89740591e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f89980b2b60 tx=0x7f8980007b80).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:26.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.558+0000 7f8994ff9700 1 -- 172.21.15.187:0/4251884442 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f89880477a0 con 0x7f89980622c0 2020-11-20T12:08:26.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.744+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8998061a20 con 0x7f89980622c0 2020-11-20T12:08:26.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.745+0000 7f8994ff9700 1 -- 172.21.15.187:0/4251884442 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v46) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8988015070 con 0x7f89980622c0 2020-11-20T12:08:26.749 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:08:26.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.747+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8974056d20 msgr2=0x7f89740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.747+0000 7f899d468700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8974056d20 0x7f89740591e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f89980b2b60 tx=0x7f8980007b80).stop 2020-11-20T12:08:26.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.747+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89980622c0 msgr2=0x7f8998138b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:26.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.747+0000 7f899d468700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89980622c0 0x7f8998138b60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f8988009fd0 tx=0x7f8988016000).stop 2020-11-20T12:08:26.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.748+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 shutdown_connections 2020-11-20T12:08:26.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.748+0000 7f899d468700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8974056d20 0x7f89740591e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.748+0000 7f899d468700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89981484d0 0x7f89980b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.748+0000 7f899d468700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89980622c0 0x7f8998138b60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.748+0000 7f899d468700 1 --2- 172.21.15.187:0/4251884442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998066f70 0x7f899815b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:26.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.748+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 >> 172.21.15.187:0/4251884442 conn(0x7f89980bab30 msgr2=0x7f89981472a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:26.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.748+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 shutdown_connections 2020-11-20T12:08:26.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:26.748+0000 7f899d468700 1 -- 172.21.15.187:0/4251884442 wait complete. 2020-11-20T12:08:26.773 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:08:27.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.005+0000 7f29bd365700 1 Processor -- start 2020-11-20T12:08:27.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.005+0000 7f29bd365700 1 -- start start 2020-11-20T12:08:27.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.007+0000 7f29bd365700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80b8750 0x7f29b8147420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.007+0000 7f29bd365700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f29b80b8b30 0x7f29b8066510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.007+0000 7f29bd365700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f29b805bda0 0x7f29b8140680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.007+0000 7f29bd365700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b8065b30 con 0x7f29b805bda0 2020-11-20T12:08:27.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.007+0000 7f29bd365700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b8065dd0 con 0x7f29b80b8b30 2020-11-20T12:08:27.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.007+0000 7f29bd365700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b8066070 con 0x7f29b80b8750 2020-11-20T12:08:27.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.007+0000 7f29af7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f29b80b8b30 0x7f29b8066510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29af7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f29b80b8b30 0x7f29b8066510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43758/0 (socket says 172.21.15.187:43758) 2020-11-20T12:08:27.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29af7fe700 1 -- 172.21.15.187:0/3242001836 learned_addr learned my addr 172.21.15.187:0/3242001836 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:27.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29b7fff700 1 --2- 172.21.15.187:0/3242001836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80b8750 0x7f29b8147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29bcb64700 1 --2- 172.21.15.187:0/3242001836 >> v2:172.21.15.5:3300/0 conn(0x7f29b805bda0 0x7f29b8140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29af7fe700 1 -- 172.21.15.187:0/3242001836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80b8750 msgr2=0x7f29b8147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:27.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29af7fe700 1 --2- 172.21.15.187:0/3242001836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80b8750 0x7f29b8147420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29af7fe700 1 -- 172.21.15.187:0/3242001836 >> v2:172.21.15.5:3300/0 conn(0x7f29b805bda0 msgr2=0x7f29b8140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:27.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29af7fe700 1 --2- 172.21.15.187:0/3242001836 >> v2:172.21.15.5:3300/0 conn(0x7f29b805bda0 0x7f29b8140680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29af7fe700 1 -- 172.21.15.187:0/3242001836 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b8147910 con 0x7f29b80b8b30 2020-11-20T12:08:27.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.008+0000 7f29bcb64700 1 --2- 172.21.15.187:0/3242001836 >> v2:172.21.15.5:3300/0 conn(0x7f29b805bda0 0x7f29b8140680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:08:27.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29af7fe700 1 --2- 172.21.15.187:0/3242001836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8b30 0x7f29b8066510 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f29b8068950 tx=0x7f29a0006eb0).ready entity=mon.1 client_cookie=f87cf87a474a53ef server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29b77fe700 1 -- 172.21.15.187:0/3242001836 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0009810 con 0x7f29b80b8b30 2020-11-20T12:08:27.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29b77fe700 1 -- 172.21.15.187:0/3242001836 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0009970 con 0x7f29b80b8b30 2020-11-20T12:08:27.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29b77fe700 1 -- 172.21.15.187:0/3242001836 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0009cf0 con 0x7f29b80b8b30 2020-11-20T12:08:27.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29bd365700 1 -- 172.21.15.187:0/3242001836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8b30 msgr2=0x7f29b8066510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29bd365700 1 --2- 172.21.15.187:0/3242001836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8b30 0x7f29b8066510 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f29b8068950 tx=0x7f29a0006eb0).stop 2020-11-20T12:08:27.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29bd365700 1 -- 172.21.15.187:0/3242001836 shutdown_connections 2020-11-20T12:08:27.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29bd365700 1 --2- 172.21.15.187:0/3242001836 >> v2:172.21.15.5:3300/0 conn(0x7f29b805bda0 0x7f29b8140680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29bd365700 1 --2- 172.21.15.187:0/3242001836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80b8750 0x7f29b8147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29bd365700 1 --2- 172.21.15.187:0/3242001836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8b30 0x7f29b8066510 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.009+0000 7f29bd365700 1 -- 172.21.15.187:0/3242001836 >> 172.21.15.187:0/3242001836 conn(0x7f29b80bbeb0 msgr2=0x7f29b80b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:27.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.010+0000 7f29bd365700 1 -- 172.21.15.187:0/3242001836 shutdown_connections 2020-11-20T12:08:27.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.010+0000 7f29bd365700 1 -- 172.21.15.187:0/3242001836 wait complete. 2020-11-20T12:08:27.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.011+0000 7f29bd365700 1 Processor -- start 2020-11-20T12:08:27.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.011+0000 7f29bd365700 1 -- start start 2020-11-20T12:08:27.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29bd365700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29b805bda0 0x7f29b80a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29bd365700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8750 0x7f29b80b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29bd365700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80a0c10 0x7f29b80b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29bd365700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b80a09e0 con 0x7f29b805bda0 2020-11-20T12:08:27.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29bd365700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b80b5090 con 0x7f29b80b8750 2020-11-20T12:08:27.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29bd365700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b813fbe0 con 0x7f29b80a0c10 2020-11-20T12:08:27.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29af7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8750 0x7f29b80b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29af7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8750 0x7f29b80b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43766/0 (socket says 172.21.15.187:43766) 2020-11-20T12:08:27.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.012+0000 7f29af7fe700 1 -- 172.21.15.187:0/1036493225 learned_addr learned my addr 172.21.15.187:0/1036493225 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:27.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29b7fff700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29b805bda0 0x7f29b80a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29bcb64700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80a0c10 0x7f29b80b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29af7fe700 1 -- 172.21.15.187:0/1036493225 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80a0c10 msgr2=0x7f29b80b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29af7fe700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80a0c10 0x7f29b80b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29af7fe700 1 -- 172.21.15.187:0/1036493225 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29b805bda0 msgr2=0x7f29b80a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29af7fe700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29b805bda0 0x7f29b80a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29af7fe700 1 -- 172.21.15.187:0/1036493225 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b815c630 con 0x7f29b80b8750 2020-11-20T12:08:27.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29af7fe700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8750 0x7f29b80b4ba0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f29b8065f10 tx=0x7f29a0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29b4ff9700 1 -- 172.21.15.187:0/1036493225 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0006b20 con 0x7f29b80b8750 2020-11-20T12:08:27.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29b4ff9700 1 -- 172.21.15.187:0/1036493225 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a001d5d0 con 0x7f29b80b8750 2020-11-20T12:08:27.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b815c8d0 con 0x7f29b80b8750 2020-11-20T12:08:27.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.013+0000 7f29b4ff9700 1 -- 172.21.15.187:0/1036493225 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a001d970 con 0x7f29b80b8750 2020-11-20T12:08:27.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.014+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b80b3090 con 0x7f29b80b8750 2020-11-20T12:08:27.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.015+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b8066430 con 0x7f29b80b8750 2020-11-20T12:08:27.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.015+0000 7f29b4ff9700 1 -- 172.21.15.187:0/1036493225 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f29a0025030 con 0x7f29b80b8750 2020-11-20T12:08:27.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.016+0000 7f29b4ff9700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2994056d20 0x7f29940591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.016+0000 7f29b4ff9700 1 -- 172.21.15.187:0/1036493225 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f29a0071200 con 0x7f29b80b8750 2020-11-20T12:08:27.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.016+0000 7f29b7fff700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2994056d20 0x7f29940591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.017+0000 7f29b7fff700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2994056d20 0x7f29940591e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f29b809e540 tx=0x7f29a8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.020+0000 7f29b4ff9700 1 -- 172.21.15.187:0/1036493225 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f29a0046a30 con 0x7f29b80b8750 2020-11-20T12:08:27.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.199+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f29b8066a00 con 0x7f29b80b8750 2020-11-20T12:08:27.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.200+0000 7f29b4ff9700 1 -- 172.21.15.187:0/1036493225 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v46) v1 ==== 92+0+10 (secure 0 0 0) 0x7f29a0023070 con 0x7f29b80b8750 2020-11-20T12:08:27.204 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:27.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2994056d20 msgr2=0x7f29940591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2994056d20 0x7f29940591e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f29b809e540 tx=0x7f29a8008040).stop 2020-11-20T12:08:27.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8750 msgr2=0x7f29b80b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8750 0x7f29b80b4ba0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f29b8065f10 tx=0x7f29a0015000).stop 2020-11-20T12:08:27.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 shutdown_connections 2020-11-20T12:08:27.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2994056d20 0x7f29940591e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29b805bda0 0x7f29b80a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f29b80a0c10 0x7f29b80b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 --2- 172.21.15.187:0/1036493225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29b80b8750 0x7f29b80b4ba0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.202+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 >> 172.21.15.187:0/1036493225 conn(0x7f29b80bbeb0 msgr2=0x7f29b8145d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:27.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.203+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 shutdown_connections 2020-11-20T12:08:27.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.203+0000 7f29bd365700 1 -- 172.21.15.187:0/1036493225 wait complete. 2020-11-20T12:08:27.228 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:08:27.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.458+0000 7fa72ae29700 1 Processor -- start 2020-11-20T12:08:27.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.459+0000 7fa72ae29700 1 -- start start 2020-11-20T12:08:27.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.460+0000 7fa72ae29700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa7240661f0 0x7fa7240b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.460+0000 7fa72ae29700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa724099ce0 0x7fa7240b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.460+0000 7fa72ae29700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7240b5df0 con 0x7fa724065e10 2020-11-20T12:08:27.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.460+0000 7fa72ae29700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72409a440 con 0x7fa724099ce0 2020-11-20T12:08:27.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.460+0000 7fa72ae29700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72409a1a0 con 0x7fa7240661f0 2020-11-20T12:08:27.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa723fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa7240661f0 0x7fa7240b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa723fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa7240661f0 0x7fa7240b62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43772/0 (socket says 172.21.15.187:43772) 2020-11-20T12:08:27.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa723fff700 1 -- 172.21.15.187:0/3873620322 learned_addr learned my addr 172.21.15.187:0/3873620322 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:27.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa723fff700 1 -- 172.21.15.187:0/3873620322 >> v2:172.21.15.5:3300/0 conn(0x7fa724099ce0 msgr2=0x7fa7240b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:27.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa7293c6700 1 --2- 172.21.15.187:0/3873620322 >> v2:172.21.15.5:3300/0 conn(0x7fa724099ce0 0x7fa7240b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa723fff700 1 --2- 172.21.15.187:0/3873620322 >> v2:172.21.15.5:3300/0 conn(0x7fa724099ce0 0x7fa7240b67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa723fff700 1 -- 172.21.15.187:0/3873620322 >> v1:172.21.15.5:6789/0 conn(0x7fa724065e10 legacy=0x7fa724144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:27.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa723fff700 1 -- 172.21.15.187:0/3873620322 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7240b0160 con 0x7fa7240661f0 2020-11-20T12:08:27.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.461+0000 7fa723fff700 1 --2- 172.21.15.187:0/3873620322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7240661f0 0x7fa7240b62e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fa724064380 tx=0x7fa710006e50).ready entity=mon.1 client_cookie=aa8726047387c55 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa7237fe700 1 -- 172.21.15.187:0/3873620322 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710006030 con 0x7fa7240661f0 2020-11-20T12:08:27.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa7237fe700 1 -- 172.21.15.187:0/3873620322 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa710006190 con 0x7fa7240661f0 2020-11-20T12:08:27.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa7237fe700 1 -- 172.21.15.187:0/3873620322 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710006510 con 0x7fa7240661f0 2020-11-20T12:08:27.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa72ae29700 1 -- 172.21.15.187:0/3873620322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7240661f0 msgr2=0x7fa7240b62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa72ae29700 1 --2- 172.21.15.187:0/3873620322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7240661f0 0x7fa7240b62e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fa724064380 tx=0x7fa710006e50).stop 2020-11-20T12:08:27.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa72ae29700 1 -- 172.21.15.187:0/3873620322 shutdown_connections 2020-11-20T12:08:27.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa72ae29700 1 --2- 172.21.15.187:0/3873620322 >> v2:172.21.15.5:3300/0 conn(0x7fa724099ce0 0x7fa7240b67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa72ae29700 1 --2- 172.21.15.187:0/3873620322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7240661f0 0x7fa7240b62e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.462+0000 7fa72ae29700 1 -- 172.21.15.187:0/3873620322 >> 172.21.15.187:0/3873620322 conn(0x7fa7240bacd0 msgr2=0x7fa7240bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:27.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.463+0000 7fa72ae29700 1 -- 172.21.15.187:0/3873620322 shutdown_connections 2020-11-20T12:08:27.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.463+0000 7fa72ae29700 1 -- 172.21.15.187:0/3873620322 wait complete. 2020-11-20T12:08:27.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.465+0000 7fa72ae29700 1 Processor -- start 2020-11-20T12:08:27.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.465+0000 7fa72ae29700 1 -- start start 2020-11-20T12:08:27.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa72ae29700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa724065e10 0x7fa724158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa72ae29700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7240661f0 0x7fa72415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa72ae29700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa724099ce0 0x7fa72414c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa72ae29700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7241580c0 con 0x7fa724099ce0 2020-11-20T12:08:27.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa72ae29700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa724158360 con 0x7fa724065e10 2020-11-20T12:08:27.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa72ae29700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa724158600 con 0x7fa7240661f0 2020-11-20T12:08:27.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa728bc5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa724065e10 0x7fa724158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa728bc5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa724065e10 0x7fa724158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43776/0 (socket says 172.21.15.187:43776) 2020-11-20T12:08:27.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.466+0000 7fa728bc5700 1 -- 172.21.15.187:0/331528123 learned_addr learned my addr 172.21.15.187:0/331528123 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:27.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa723fff700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7240661f0 0x7fa72415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa7293c6700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa724099ce0 0x7fa72414c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa728bc5700 1 -- 172.21.15.187:0/331528123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7240661f0 msgr2=0x7fa72415be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa728bc5700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7240661f0 0x7fa72415be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa728bc5700 1 -- 172.21.15.187:0/331528123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa724099ce0 msgr2=0x7fa72414c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa728bc5700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa724099ce0 0x7fa72414c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa728bc5700 1 -- 172.21.15.187:0/331528123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72415c370 con 0x7fa724065e10 2020-11-20T12:08:27.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa728bc5700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa724065e10 0x7fa724158aa0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fa7180096e0 tx=0x7fa718006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa720ff9700 1 -- 172.21.15.187:0/331528123 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71800d560 con 0x7fa724065e10 2020-11-20T12:08:27.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa720ff9700 1 -- 172.21.15.187:0/331528123 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa718000e90 con 0x7fa724065e10 2020-11-20T12:08:27.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72414ce70 con 0x7fa724065e10 2020-11-20T12:08:27.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.467+0000 7fa720ff9700 1 -- 172.21.15.187:0/331528123 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71800d930 con 0x7fa724065e10 2020-11-20T12:08:27.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.468+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72409c820 con 0x7fa724065e10 2020-11-20T12:08:27.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.469+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa724063600 con 0x7fa724065e10 2020-11-20T12:08:27.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.469+0000 7fa720ff9700 1 -- 172.21.15.187:0/331528123 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa71800f030 con 0x7fa724065e10 2020-11-20T12:08:27.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.470+0000 7fa720ff9700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa6fc056d20 0x7fa6fc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.470+0000 7fa720ff9700 1 -- 172.21.15.187:0/331528123 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa71806f920 con 0x7fa724065e10 2020-11-20T12:08:27.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.470+0000 7fa723fff700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa6fc056d20 0x7fa6fc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.471+0000 7fa723fff700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa6fc056d20 0x7fa6fc0591e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fa7240b50a0 tx=0x7fa710015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.474+0000 7fa720ff9700 1 -- 172.21.15.187:0/331528123 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa7180450a0 con 0x7fa724065e10 2020-11-20T12:08:27.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.647+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa7240666e0 con 0x7fa724065e10 2020-11-20T12:08:27.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.648+0000 7fa720ff9700 1 -- 172.21.15.187:0/331528123 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v46) v1 ==== 84+0+143 (secure 0 0 0) 0x7fa7180486c0 con 0x7fa724065e10 2020-11-20T12:08:27.653 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:27.653 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:08:27.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:27.653 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:08:27.654 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:08:27.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:27.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:27.654 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:08:27.654 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:08:27.655 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:27.655 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:08:27.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.650+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa6fc056d20 msgr2=0x7fa6fc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.650+0000 7fa72ae29700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa6fc056d20 0x7fa6fc0591e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fa7240b50a0 tx=0x7fa710015000).stop 2020-11-20T12:08:27.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.650+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa724065e10 msgr2=0x7fa724158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.650+0000 7fa72ae29700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa724065e10 0x7fa724158aa0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fa7180096e0 tx=0x7fa718006d70).stop 2020-11-20T12:08:27.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.651+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 shutdown_connections 2020-11-20T12:08:27.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.651+0000 7fa72ae29700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa6fc056d20 0x7fa6fc0591e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.651+0000 7fa72ae29700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa724099ce0 0x7fa72414c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.651+0000 7fa72ae29700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa724065e10 0x7fa724158aa0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.651+0000 7fa72ae29700 1 --2- 172.21.15.187:0/331528123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7240661f0 0x7fa72415be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.651+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 >> 172.21.15.187:0/331528123 conn(0x7fa7240bacd0 msgr2=0x7fa7240b5160 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:27.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.651+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 shutdown_connections 2020-11-20T12:08:27.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.651+0000 7fa72ae29700 1 -- 172.21.15.187:0/331528123 wait complete. 2020-11-20T12:08:27.678 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:08:27.678 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:08:27.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.932+0000 7f69b7fff700 1 Processor -- start 2020-11-20T12:08:27.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.932+0000 7f69b7fff700 1 -- start start 2020-11-20T12:08:27.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.933+0000 7f69b7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f69b8068230 0x7f69b81482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.933+0000 7f69b7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f69b80b5f50 0x7f69b80b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.933+0000 7f69b7fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b8144b80 con 0x7f69b80b57b0 2020-11-20T12:08:27.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.933+0000 7f69b7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b81487c0 con 0x7f69b8068230 2020-11-20T12:08:27.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.933+0000 7f69b7fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b81489e0 con 0x7f69b80b5f50 2020-11-20T12:08:27.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.934+0000 7f69b77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f69b80b5f50 0x7f69b80b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.934+0000 7f69b6ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f69b80b57b0 0x7f69b8062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40390/0 (socket says 172.21.15.187:40390) 2020-11-20T12:08:27.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.934+0000 7f69b6ffd700 1 -- 172.21.15.187:0/1850325077 learned_addr learned my addr 172.21.15.187:0/1850325077 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:27.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.934+0000 7f69ae7fc700 1 --2- 172.21.15.187:0/1850325077 >> v2:172.21.15.5:3300/0 conn(0x7f69b8068230 0x7f69b81482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.935+0000 7f69b77fe700 1 -- 172.21.15.187:0/1850325077 >> v2:172.21.15.5:3300/0 conn(0x7f69b8068230 msgr2=0x7f69b81482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:27.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.935+0000 7f69b67fc700 1 -- 172.21.15.187:0/1850325077 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598115660 0 0) 0x7f69b8144b80 con 0x7f69b80b57b0 2020-11-20T12:08:27.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.935+0000 7f69b77fe700 1 --2- 172.21.15.187:0/1850325077 >> v2:172.21.15.5:3300/0 conn(0x7f69b8068230 0x7f69b81482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.935+0000 7f69b77fe700 1 -- 172.21.15.187:0/1850325077 >> v1:172.21.15.187:6789/0 conn(0x7f69b80b57b0 legacy=0x7f69b8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.935+0000 7f69b77fe700 1 -- 172.21.15.187:0/1850325077 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b8148c80 con 0x7f69b80b5f50 2020-11-20T12:08:27.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.935+0000 7f69b77fe700 1 --2- 172.21.15.187:0/1850325077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b5f50 0x7f69b80b6380 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f69b8066230 tx=0x7f69b00091d0).ready entity=mon.1 client_cookie=1df8fb49928fe1ca server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.936+0000 7f69b67fc700 1 -- 172.21.15.187:0/1850325077 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000ba70 con 0x7f69b80b5f50 2020-11-20T12:08:27.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.936+0000 7f69b67fc700 1 -- 172.21.15.187:0/1850325077 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b000bbd0 con 0x7f69b80b5f50 2020-11-20T12:08:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.936+0000 7f69b67fc700 1 -- 172.21.15.187:0/1850325077 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0011400 con 0x7f69b80b5f50 2020-11-20T12:08:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.936+0000 7f69b7fff700 1 -- 172.21.15.187:0/1850325077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b5f50 msgr2=0x7f69b80b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.936+0000 7f69b7fff700 1 --2- 172.21.15.187:0/1850325077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b5f50 0x7f69b80b6380 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f69b8066230 tx=0x7f69b00091d0).stop 2020-11-20T12:08:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.937+0000 7f69b7fff700 1 -- 172.21.15.187:0/1850325077 shutdown_connections 2020-11-20T12:08:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.937+0000 7f69b7fff700 1 --2- 172.21.15.187:0/1850325077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b5f50 0x7f69b80b6380 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.937+0000 7f69b7fff700 1 --2- 172.21.15.187:0/1850325077 >> v2:172.21.15.5:3300/0 conn(0x7f69b8068230 0x7f69b81482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:27.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.937+0000 7f69b7fff700 1 -- 172.21.15.187:0/1850325077 >> 172.21.15.187:0/1850325077 conn(0x7f69b80bab30 msgr2=0x7f69b80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:27.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.937+0000 7f69b7fff700 1 -- 172.21.15.187:0/1850325077 shutdown_connections 2020-11-20T12:08:27.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.937+0000 7f69b7fff700 1 -- 172.21.15.187:0/1850325077 wait complete. 2020-11-20T12:08:27.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.938+0000 7f69b7fff700 1 Processor -- start 2020-11-20T12:08:27.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.938+0000 7f69b7fff700 1 -- start start 2020-11-20T12:08:27.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.939+0000 7f69b7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f69b8068230 0x7f69b8158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.939+0000 7f69b7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b57b0 0x7f69b8146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.939+0000 7f69b7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69b80b5f50 0x7f69b8147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.939+0000 7f69b7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b81474c0 con 0x7f69b80b5f50 2020-11-20T12:08:27.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.939+0000 7f69b7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b81465f0 con 0x7f69b80b57b0 2020-11-20T12:08:27.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.939+0000 7f69b7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8146890 con 0x7f69b8068230 2020-11-20T12:08:27.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b57b0 0x7f69b8146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b57b0 0x7f69b8146fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43790/0 (socket says 172.21.15.187:43790) 2020-11-20T12:08:27.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 -- 172.21.15.187:0/989731970 learned_addr learned my addr 172.21.15.187:0/989731970 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:27.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 -- 172.21.15.187:0/989731970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f69b8068230 msgr2=0x7f69b8158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f69b8068230 0x7f69b8158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:27.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 -- 172.21.15.187:0/989731970 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69b80b5f50 msgr2=0x7f69b8147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:27.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69b80b5f50 0x7f69b8147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:27.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 -- 172.21.15.187:0/989731970 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b8146b30 con 0x7f69b80b57b0 2020-11-20T12:08:27.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.940+0000 7f69ae7fc700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b57b0 0x7f69b8146fd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f69a000ebd0 tx=0x7f69a0006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.941+0000 7f69af7fe700 1 -- 172.21.15.187:0/989731970 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a000b9a0 con 0x7f69b80b57b0 2020-11-20T12:08:27.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.941+0000 7f69af7fe700 1 -- 172.21.15.187:0/989731970 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a000bb00 con 0x7f69b80b57b0 2020-11-20T12:08:27.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.941+0000 7f69af7fe700 1 -- 172.21.15.187:0/989731970 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a000be80 con 0x7f69b80b57b0 2020-11-20T12:08:27.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.941+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b81462e0 con 0x7f69b80b57b0 2020-11-20T12:08:27.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.941+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b8148300 con 0x7f69b80b57b0 2020-11-20T12:08:27.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.942+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b81485e0 con 0x7f69b80b57b0 2020-11-20T12:08:27.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.942+0000 7f69af7fe700 1 -- 172.21.15.187:0/989731970 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f69a0012600 con 0x7f69b80b57b0 2020-11-20T12:08:27.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.943+0000 7f69af7fe700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6994056d20 0x7f69940591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:27.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.943+0000 7f69af7fe700 1 -- 172.21.15.187:0/989731970 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f69a0070040 con 0x7f69b80b57b0 2020-11-20T12:08:27.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.943+0000 7f69b6ffd700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6994056d20 0x7f69940591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:27.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.944+0000 7f69b6ffd700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6994056d20 0x7f69940591e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f69a80011a0 tx=0x7f69a80076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:27.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:27.947+0000 7f69af7fe700 1 -- 172.21.15.187:0/989731970 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f69a00457c0 con 0x7f69b80b57b0 2020-11-20T12:08:28.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.133+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f69b8144ad0 con 0x7f69b80b57b0 2020-11-20T12:08:28.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.285+0000 7f69af7fe700 1 -- 172.21.15.187:0/989731970 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v47) v1 ==== 133+0+0 (secure 0 0 0) 0x7f69a0048de0 con 0x7f69b80b57b0 2020-11-20T12:08:28.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.288+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6994056d20 msgr2=0x7f69940591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:28.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.288+0000 7f69b7fff700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6994056d20 0x7f69940591e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f69a80011a0 tx=0x7f69a80076a0).stop 2020-11-20T12:08:28.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.288+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b57b0 msgr2=0x7f69b8146fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:28.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.288+0000 7f69b7fff700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b57b0 0x7f69b8146fd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f69a000ebd0 tx=0x7f69a0006df0).stop 2020-11-20T12:08:28.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.289+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 shutdown_connections 2020-11-20T12:08:28.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.289+0000 7f69b7fff700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6994056d20 0x7f69940591e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:28.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.289+0000 7f69b7fff700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69b80b5f50 0x7f69b8147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:28.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.289+0000 7f69b7fff700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f69b8068230 0x7f69b8158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:28.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.289+0000 7f69b7fff700 1 --2- 172.21.15.187:0/989731970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b80b57b0 0x7f69b8146fd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:28.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.289+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 >> 172.21.15.187:0/989731970 conn(0x7f69b80bab30 msgr2=0x7f69b809fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:28.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.289+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 shutdown_connections 2020-11-20T12:08:28.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.289+0000 7f69b7fff700 1 -- 172.21.15.187:0/989731970 wait complete. 2020-11-20T12:08:28.297 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:08:28.325 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:08:28.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.564+0000 7fdb63fff700 1 Processor -- start 2020-11-20T12:08:28.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.564+0000 7fdb63fff700 1 -- start start 2020-11-20T12:08:28.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.565+0000 7fdb63fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdb640b6650 0x7fdb640afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:28.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.565+0000 7fdb63fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb64066f70 0x7fdb640aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:28.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.565+0000 7fdb63fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb640b1300 con 0x7fdb641484d0 2020-11-20T12:08:28.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.565+0000 7fdb63fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb64067730 con 0x7fdb640b6650 2020-11-20T12:08:28.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.565+0000 7fdb63fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640b15a0 con 0x7fdb64066f70 2020-11-20T12:08:28.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.566+0000 7fdb637fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb64066f70 0x7fdb640aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:28.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.566+0000 7fdb637fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb64066f70 0x7fdb640aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:44690/0 (socket says 172.21.15.187:44690) 2020-11-20T12:08:28.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.566+0000 7fdb637fe700 1 -- 172.21.15.187:0/4066996620 learned_addr learned my addr 172.21.15.187:0/4066996620 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:28.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.566+0000 7fdb5a7fc700 1 --2- 172.21.15.187:0/4066996620 >> v2:172.21.15.5:3300/0 conn(0x7fdb640b6650 0x7fdb640afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:28.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.567+0000 7fdb637fe700 1 -- 172.21.15.187:0/4066996620 >> v2:172.21.15.5:3300/0 conn(0x7fdb640b6650 msgr2=0x7fdb640afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:28.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.567+0000 7fdb637fe700 1 --2- 172.21.15.187:0/4066996620 >> v2:172.21.15.5:3300/0 conn(0x7fdb640b6650 0x7fdb640afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:28.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.567+0000 7fdb637fe700 1 -- 172.21.15.187:0/4066996620 >> v1:172.21.15.5:6789/0 conn(0x7fdb641484d0 legacy=0x7fdb64061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:28.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.567+0000 7fdb637fe700 1 -- 172.21.15.187:0/4066996620 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb640b1840 con 0x7fdb64066f70 2020-11-20T12:08:28.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.568+0000 7fdb637fe700 1 --2- 172.21.15.187:0/4066996620 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb64066f70 0x7fdb640aff80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fdb5c009d50 tx=0x7fdb5c009290).ready entity=mon.2 client_cookie=e739aeff5b3cd206 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:28.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.569+0000 7fdb627fc700 1 -- 172.21.15.187:0/4066996620 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c00bbc0 con 0x7fdb64066f70 2020-11-20T12:08:28.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.569+0000 7fdb627fc700 1 -- 172.21.15.187:0/4066996620 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c00bd20 con 0x7fdb64066f70 2020-11-20T12:08:28.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.569+0000 7fdb627fc700 1 -- 172.21.15.187:0/4066996620 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c011610 con 0x7fdb64066f70 2020-11-20T12:08:28.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.570+0000 7fdb63fff700 1 -- 172.21.15.187:0/4066996620 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb64066f70 msgr2=0x7fdb640aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:28.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.570+0000 7fdb63fff700 1 --2- 172.21.15.187:0/4066996620 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb64066f70 0x7fdb640aff80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fdb5c009d50 tx=0x7fdb5c009290).stop 2020-11-20T12:08:28.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.570+0000 7fdb63fff700 1 -- 172.21.15.187:0/4066996620 shutdown_connections 2020-11-20T12:08:28.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.570+0000 7fdb63fff700 1 --2- 172.21.15.187:0/4066996620 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb64066f70 0x7fdb640aff80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:28.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.570+0000 7fdb63fff700 1 --2- 172.21.15.187:0/4066996620 >> v2:172.21.15.5:3300/0 conn(0x7fdb640b6650 0x7fdb640afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:28.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.570+0000 7fdb63fff700 1 -- 172.21.15.187:0/4066996620 >> 172.21.15.187:0/4066996620 conn(0x7fdb640bab30 msgr2=0x7fdb640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:28.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.570+0000 7fdb63fff700 1 -- 172.21.15.187:0/4066996620 shutdown_connections 2020-11-20T12:08:28.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.570+0000 7fdb63fff700 1 -- 172.21.15.187:0/4066996620 wait complete. 2020-11-20T12:08:28.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.573+0000 7fdb63fff700 1 Processor -- start 2020-11-20T12:08:28.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.574+0000 7fdb63fff700 1 -- start start 2020-11-20T12:08:28.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb63fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb64066f70 0x7fdb6415b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:28.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb63fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb640b6650 0x7fdb64139690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:28.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb63fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb641484d0 0x7fdb64138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:28.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb63fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6409e250 con 0x7fdb64066f70 2020-11-20T12:08:28.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb63fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6415bb70 con 0x7fdb640b6650 2020-11-20T12:08:28.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb63fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6415be00 con 0x7fdb641484d0 2020-11-20T12:08:28.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb5a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb640b6650 0x7fdb64139690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:28.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb5a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb640b6650 0x7fdb64139690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43808/0 (socket says 172.21.15.187:43808) 2020-11-20T12:08:28.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.575+0000 7fdb5a7fc700 1 -- 172.21.15.187:0/283411713 learned_addr learned my addr 172.21.15.187:0/283411713 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:28.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb637fe700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb641484d0 0x7fdb64138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:28.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb62ffd700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb64066f70 0x7fdb6415b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:28.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5a7fc700 1 -- 172.21.15.187:0/283411713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb641484d0 msgr2=0x7fdb64138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:28.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5a7fc700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb641484d0 0x7fdb64138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:28.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5a7fc700 1 -- 172.21.15.187:0/283411713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb64066f70 msgr2=0x7fdb6415b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:28.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5a7fc700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb64066f70 0x7fdb6415b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:28.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5a7fc700 1 -- 172.21.15.187:0/283411713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6415c0a0 con 0x7fdb640b6650 2020-11-20T12:08:28.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5a7fc700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb640b6650 0x7fdb64139690 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fdb4c00ebd0 tx=0x7fdb4c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:28.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5b7fe700 1 -- 172.21.15.187:0/283411713 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00ca60 con 0x7fdb640b6650 2020-11-20T12:08:28.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5b7fe700 1 -- 172.21.15.187:0/283411713 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c00cbc0 con 0x7fdb640b6650 2020-11-20T12:08:28.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6415c340 con 0x7fdb640b6650 2020-11-20T12:08:28.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.576+0000 7fdb5b7fe700 1 -- 172.21.15.187:0/283411713 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0123f0 con 0x7fdb640b6650 2020-11-20T12:08:28.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.577+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6409bd80 con 0x7fdb640b6650 2020-11-20T12:08:28.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.578+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6409c060 con 0x7fdb640b6650 2020-11-20T12:08:28.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.578+0000 7fdb5b7fe700 1 -- 172.21.15.187:0/283411713 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdb4c012760 con 0x7fdb640b6650 2020-11-20T12:08:28.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.579+0000 7fdb5b7fe700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb40056d20 0x7fdb400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:28.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.579+0000 7fdb5b7fe700 1 -- 172.21.15.187:0/283411713 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fdb4c070420 con 0x7fdb640b6650 2020-11-20T12:08:28.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.579+0000 7fdb62ffd700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb40056d20 0x7fdb400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:28.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.580+0000 7fdb62ffd700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb40056d20 0x7fdb400591e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fdb54003450 tx=0x7fdb54009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:28.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.583+0000 7fdb5b7fe700 1 -- 172.21.15.187:0/283411713 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdb4c045930 con 0x7fdb640b6650 2020-11-20T12:08:28.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:28.778+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fdb6415c4d0 con 0x7fdb640b6650 2020-11-20T12:08:29.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.626+0000 7fdb5b7fe700 1 -- 172.21.15.187:0/283411713 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v49) v1 ==== 125+0+0 (secure 0 0 0) 0x7fdb4c048f50 con 0x7fdb640b6650 2020-11-20T12:08:29.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.629+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb40056d20 msgr2=0x7fdb400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:29.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.629+0000 7fdb63fff700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb40056d20 0x7fdb400591e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fdb54003450 tx=0x7fdb54009040).stop 2020-11-20T12:08:29.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.629+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb640b6650 msgr2=0x7fdb64139690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:29.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.629+0000 7fdb63fff700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb640b6650 0x7fdb64139690 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fdb4c00ebd0 tx=0x7fdb4c006df0).stop 2020-11-20T12:08:29.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.629+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 shutdown_connections 2020-11-20T12:08:29.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.629+0000 7fdb63fff700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb40056d20 0x7fdb400591e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:29.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.630+0000 7fdb63fff700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb64066f70 0x7fdb6415b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:29.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.630+0000 7fdb63fff700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb641484d0 0x7fdb64138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.630+0000 7fdb63fff700 1 --2- 172.21.15.187:0/283411713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb640b6650 0x7fdb64139690 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.630+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 >> 172.21.15.187:0/283411713 conn(0x7fdb640bab30 msgr2=0x7fdb640afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.630+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 shutdown_connections 2020-11-20T12:08:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.630+0000 7fdb63fff700 1 -- 172.21.15.187:0/283411713 wait complete. 2020-11-20T12:08:29.645 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:08:29.668 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:08:29.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.907+0000 7f4a7fde9700 1 Processor -- start 2020-11-20T12:08:29.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.907+0000 7f4a7fde9700 1 -- start start 2020-11-20T12:08:29.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.908+0000 7f4a7fde9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4a78068030 0x7f4a780656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:29.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.908+0000 7f4a7fde9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a78066420 con 0x7f4a7809a8a0 2020-11-20T12:08:29.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.909+0000 7f4a7fde9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a780b7380 con 0x7f4a78065300 2020-11-20T12:08:29.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.909+0000 7f4a7fde9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a78061f60 con 0x7f4a78068030 2020-11-20T12:08:29.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.909+0000 7f4a7db85700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4a78065300 0x7f4a78062400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40422/0 (socket says 172.21.15.187:40422) 2020-11-20T12:08:29.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.909+0000 7f4a7db85700 1 -- 172.21.15.187:0/2758809055 learned_addr learned my addr 172.21.15.187:0/2758809055 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:29.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7e386700 1 --2- 172.21.15.187:0/2758809055 >> v2:172.21.15.5:3300/0 conn(0x7f4a78068030 0x7f4a780656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:29.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787452509 0 0) 0x7f4a780b7380 con 0x7f4a78065300 2020-11-20T12:08:29.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a64003240 con 0x7f4a78065300 2020-11-20T12:08:29.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2058637 0 0) 0x7f4a64003240 con 0x7f4a78065300 2020-11-20T12:08:29.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 >> v2:172.21.15.5:3300/0 conn(0x7f4a78068030 msgr2=0x7f4a780656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:29.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 --2- 172.21.15.187:0/2758809055 >> v2:172.21.15.5:3300/0 conn(0x7f4a78068030 0x7f4a780656e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:29.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 >> v1:172.21.15.5:6789/0 conn(0x7f4a7809a8a0 legacy=0x7f4a78146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:29.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780b6fd0 con 0x7f4a78065300 2020-11-20T12:08:29.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4a680014e0 con 0x7f4a78065300 2020-11-20T12:08:29.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 >> v1:172.21.15.187:6789/0 conn(0x7f4a78065300 legacy=0x7f4a78062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:29.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64003780 0x7f4a64003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:29.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a640044e0 0x7f4a64004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:29.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a64004e50 0x7f4a6400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:29.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a78061f60 con 0x7f4a640044e0 2020-11-20T12:08:29.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64005300 con 0x7f4a64003780 2020-11-20T12:08:29.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.910+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6400bc20 con 0x7f4a64004e50 2020-11-20T12:08:29.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.911+0000 7f4a7db85700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64003780 0x7f4a64003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:29.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.911+0000 7f4a7e386700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a64004e50 0x7f4a6400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:29.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.911+0000 7f4a7d384700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a640044e0 0x7f4a64004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:29.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.911+0000 7f4a7db85700 1 -- 172.21.15.187:0/2758809055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a64004e50 msgr2=0x7f4a6400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:29.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.911+0000 7f4a7db85700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a64004e50 0x7f4a6400b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:29.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.911+0000 7f4a7db85700 1 -- 172.21.15.187:0/2758809055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a640044e0 msgr2=0x7f4a64004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:29.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.912+0000 7f4a7db85700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a640044e0 0x7f4a64004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:29.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.912+0000 7f4a7db85700 1 -- 172.21.15.187:0/2758809055 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64003240 con 0x7f4a64003780 2020-11-20T12:08:29.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.912+0000 7f4a7e386700 1 -- 172.21.15.187:0/2758809055 reap_dead start 2020-11-20T12:08:29.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.912+0000 7f4a7db85700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64003780 0x7f4a64003ba0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f4a6800c210 tx=0x7f4a680076d0).ready entity=mon.1 client_cookie=3e4bdc32d39c82f1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:29.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.912+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a680070d0 con 0x7f4a64003780 2020-11-20T12:08:29.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.912+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68007230 con 0x7f4a64003780 2020-11-20T12:08:29.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.912+0000 7f4a7cb83700 1 -- 172.21.15.187:0/2758809055 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a680055f0 con 0x7f4a64003780 2020-11-20T12:08:29.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.913+0000 7f4a7fde9700 1 -- 172.21.15.187:0/2758809055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64003780 msgr2=0x7f4a64003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:29.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.913+0000 7f4a7fde9700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64003780 0x7f4a64003ba0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f4a6800c210 tx=0x7f4a680076d0).stop 2020-11-20T12:08:29.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.913+0000 7f4a7fde9700 1 -- 172.21.15.187:0/2758809055 shutdown_connections 2020-11-20T12:08:29.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.913+0000 7f4a7fde9700 1 --2- 172.21.15.187:0/2758809055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64003780 0x7f4a64003ba0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:29.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.913+0000 7f4a7fde9700 1 -- 172.21.15.187:0/2758809055 >> 172.21.15.187:0/2758809055 conn(0x7f4a780bacf0 msgr2=0x7f4a780bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:29.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.913+0000 7f4a7fde9700 1 -- 172.21.15.187:0/2758809055 shutdown_connections 2020-11-20T12:08:29.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.913+0000 7f4a7fde9700 1 -- 172.21.15.187:0/2758809055 wait complete. 2020-11-20T12:08:29.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.914+0000 7f4a7fde9700 1 Processor -- start 2020-11-20T12:08:29.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.915+0000 7f4a7fde9700 1 -- start start 2020-11-20T12:08:29.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.915+0000 7f4a7fde9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a78145250 0x7f4a78145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:29.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.915+0000 7f4a7fde9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a780a0690 0x7f4a780b7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:29.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.915+0000 7f4a7fde9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a780b7c30 0x7f4a78154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:29.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.915+0000 7f4a7fde9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780a0b30 con 0x7f4a780b7c30 2020-11-20T12:08:29.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.915+0000 7f4a7fde9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780a0dd0 con 0x7f4a78145250 2020-11-20T12:08:29.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.915+0000 7f4a7fde9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a78153d60 con 0x7f4a780a0690 2020-11-20T12:08:29.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.916+0000 7f4a7db85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a78145250 0x7f4a78145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:29.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.916+0000 7f4a7db85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a78145250 0x7f4a78145630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43826/0 (socket says 172.21.15.187:43826) 2020-11-20T12:08:29.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.916+0000 7f4a7db85700 1 -- 172.21.15.187:0/787352334 learned_addr learned my addr 172.21.15.187:0/787352334 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:29.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.916+0000 7f4a7db85700 1 -- 172.21.15.187:0/787352334 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a780a0690 msgr2=0x7f4a780b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:29.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.916+0000 7f4a7db85700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a780a0690 0x7f4a780b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:29.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.916+0000 7f4a7db85700 1 -- 172.21.15.187:0/787352334 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a780b7c30 msgr2=0x7f4a78154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:29.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.916+0000 7f4a7db85700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a780b7c30 0x7f4a78154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:29.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.916+0000 7f4a7db85700 1 -- 172.21.15.187:0/787352334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780b4a00 con 0x7f4a78145250 2020-11-20T12:08:29.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.917+0000 7f4a7db85700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a78145250 0x7f4a78145630 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f4a68001010 tx=0x7f4a68015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:29.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.917+0000 7f4a75ffb700 1 -- 172.21.15.187:0/787352334 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68000c00 con 0x7f4a78145250 2020-11-20T12:08:29.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.917+0000 7f4a75ffb700 1 -- 172.21.15.187:0/787352334 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68013660 con 0x7f4a78145250 2020-11-20T12:08:29.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.917+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a780b1cb0 con 0x7f4a78145250 2020-11-20T12:08:29.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.917+0000 7f4a75ffb700 1 -- 172.21.15.187:0/787352334 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a680139e0 con 0x7f4a78145250 2020-11-20T12:08:29.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.917+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a78154930 con 0x7f4a78145250 2020-11-20T12:08:29.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.918+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a780b1fe0 con 0x7f4a78145250 2020-11-20T12:08:29.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.919+0000 7f4a75ffb700 1 -- 172.21.15.187:0/787352334 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4a68027070 con 0x7f4a78145250 2020-11-20T12:08:29.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.919+0000 7f4a75ffb700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a50056d20 0x7f4a500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:29.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.919+0000 7f4a75ffb700 1 -- 172.21.15.187:0/787352334 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f4a68071010 con 0x7f4a78145250 2020-11-20T12:08:29.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.920+0000 7f4a7d384700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a50056d20 0x7f4a500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:29.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.920+0000 7f4a7d384700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a50056d20 0x7f4a500591e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4a6c000f20 tx=0x7f4a6c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:29.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:29.923+0000 7f4a75ffb700 1 -- 172.21.15.187:0/787352334 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4a680464d0 con 0x7f4a78145250 2020-11-20T12:08:30.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.173+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f4a78153f20 con 0x7f4a78145250 2020-11-20T12:08:30.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.187+0000 7f4a75ffb700 1 -- 172.21.15.187:0/787352334 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 9 and data pool 10 v23) v1 ==== 188+0+0 (secure 0 0 0) 0x7f4a68049af0 con 0x7f4a78145250 2020-11-20T12:08:30.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.192+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a50056d20 msgr2=0x7f4a500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.192+0000 7f4a7fde9700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a50056d20 0x7f4a500591e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4a6c000f20 tx=0x7f4a6c009040).stop 2020-11-20T12:08:30.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.192+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a78145250 msgr2=0x7f4a78145630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.192+0000 7f4a7fde9700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a78145250 0x7f4a78145630 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f4a68001010 tx=0x7f4a68015000).stop 2020-11-20T12:08:30.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.193+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 shutdown_connections 2020-11-20T12:08:30.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.193+0000 7f4a7fde9700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a50056d20 0x7f4a500591e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.193+0000 7f4a7fde9700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a780b7c30 0x7f4a78154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:30.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.193+0000 7f4a7fde9700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a78145250 0x7f4a78145630 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.193+0000 7f4a7fde9700 1 --2- 172.21.15.187:0/787352334 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a780a0690 0x7f4a780b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:30.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.193+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 >> 172.21.15.187:0/787352334 conn(0x7f4a780bacf0 msgr2=0x7f4a78062320 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:30.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.196+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 shutdown_connections 2020-11-20T12:08:30.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.196+0000 7f4a7fde9700 1 -- 172.21.15.187:0/787352334 wait complete. 2020-11-20T12:08:30.202 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 9 and data pool 10 2020-11-20T12:08:30.227 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:30.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.443+0000 7ff6ea0d0700 1 Processor -- start 2020-11-20T12:08:30.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.443+0000 7ff6ea0d0700 1 -- start start 2020-11-20T12:08:30.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.445+0000 7ff6ea0d0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6e4065570 0x7ff6e409d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.445+0000 7ff6ea0d0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e40b13e0 con 0x7ff6e4065190 2020-11-20T12:08:30.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.445+0000 7ff6ea0d0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e409d690 con 0x7ff6e4065f90 2020-11-20T12:08:30.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.445+0000 7ff6ea0d0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e414ab30 con 0x7ff6e4065570 2020-11-20T12:08:30.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.445+0000 7ff6e88cd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff6e4065f90 0x7ff6e409ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40452/0 (socket says 172.21.15.187:40452) 2020-11-20T12:08:30.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.445+0000 7ff6e88cd700 1 -- 172.21.15.187:0/147939733 learned_addr learned my addr 172.21.15.187:0/147939733 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.445+0000 7ff6e98cf700 1 --2- 172.21.15.187:0/147939733 >> v2:172.21.15.187:3300/0 conn(0x7ff6e4065570 0x7ff6e409d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.446+0000 7ff6d3fff700 1 -- 172.21.15.187:0/147939733 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498525831 0 0) 0x7ff6e409d690 con 0x7ff6e4065f90 2020-11-20T12:08:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.446+0000 7ff6d3fff700 1 -- 172.21.15.187:0/147939733 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6cc003240 con 0x7ff6e4065f90 2020-11-20T12:08:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.446+0000 7ff6e98cf700 1 -- 172.21.15.187:0/147939733 >> v1:172.21.15.187:6789/0 conn(0x7ff6e4065f90 legacy=0x7ff6e409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.446+0000 7ff6e98cf700 1 -- 172.21.15.187:0/147939733 >> v1:172.21.15.5:6789/0 conn(0x7ff6e4065190 legacy=0x7ff6e4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.446+0000 7ff6e98cf700 1 -- 172.21.15.187:0/147939733 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4062c10 con 0x7ff6e4065570 2020-11-20T12:08:30.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.446+0000 7ff6e98cf700 1 --2- 172.21.15.187:0/147939733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e409d1a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff6e409d820 tx=0x7ff6dc0091d0).ready entity=mon.1 client_cookie=b18f5f60b142b6b7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:30.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6d3fff700 1 -- 172.21.15.187:0/147939733 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00bad0 con 0x7ff6e4065570 2020-11-20T12:08:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6d3fff700 1 -- 172.21.15.187:0/147939733 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc00bc30 con 0x7ff6e4065570 2020-11-20T12:08:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6d3fff700 1 -- 172.21.15.187:0/147939733 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc011400 con 0x7ff6e4065570 2020-11-20T12:08:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/147939733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 msgr2=0x7ff6e409d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6ea0d0700 1 --2- 172.21.15.187:0/147939733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e409d1a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff6e409d820 tx=0x7ff6dc0091d0).stop 2020-11-20T12:08:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/147939733 shutdown_connections 2020-11-20T12:08:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6ea0d0700 1 --2- 172.21.15.187:0/147939733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e409d1a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/147939733 >> 172.21.15.187:0/147939733 conn(0x7ff6e40bab30 msgr2=0x7ff6e40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.447+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/147939733 shutdown_connections 2020-11-20T12:08:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.448+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/147939733 wait complete. 2020-11-20T12:08:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.449+0000 7ff6ea0d0700 1 Processor -- start 2020-11-20T12:08:30.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.450+0000 7ff6ea0d0700 1 -- start start 2020-11-20T12:08:30.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.451+0000 7ff6ea0d0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6e4065190 0x7ff6e415b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.451+0000 7ff6ea0d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e40b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.451+0000 7ff6ea0d0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6e4065f90 0x7ff6e41480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.451+0000 7ff6ea0d0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4147520 con 0x7ff6e4065f90 2020-11-20T12:08:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.451+0000 7ff6ea0d0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4147710 con 0x7ff6e4065570 2020-11-20T12:08:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.451+0000 7ff6ea0d0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41479b0 con 0x7ff6e4065190 2020-11-20T12:08:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e40b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e40b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43854/0 (socket says 172.21.15.187:43854) 2020-11-20T12:08:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 -- 172.21.15.187:0/2282472259 learned_addr learned my addr 172.21.15.187:0/2282472259 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 -- 172.21.15.187:0/2282472259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6e4065190 msgr2=0x7ff6e415b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6e4065190 0x7ff6e415b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 -- 172.21.15.187:0/2282472259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6e4065f90 msgr2=0x7ff6e41480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6e4065f90 0x7ff6e41480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 -- 172.21.15.187:0/2282472259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4147c50 con 0x7ff6e4065570 2020-11-20T12:08:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.452+0000 7ff6e88cd700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e40b4440 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7ff6d400c740 tx=0x7ff6d4007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.453+0000 7ff6d17fa700 1 -- 172.21.15.187:0/2282472259 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40072e0 con 0x7ff6e4065570 2020-11-20T12:08:30.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.453+0000 7ff6d17fa700 1 -- 172.21.15.187:0/2282472259 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d4007440 con 0x7ff6e4065570 2020-11-20T12:08:30.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.453+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e41485e0 con 0x7ff6e4065570 2020-11-20T12:08:30.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.453+0000 7ff6d17fa700 1 -- 172.21.15.187:0/2282472259 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4005800 con 0x7ff6e4065570 2020-11-20T12:08:30.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.453+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e4149180 con 0x7ff6e4065570 2020-11-20T12:08:30.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.454+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e4147de0 con 0x7ff6e4065570 2020-11-20T12:08:30.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.454+0000 7ff6d17fa700 1 -- 172.21.15.187:0/2282472259 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff6d4010070 con 0x7ff6e4065570 2020-11-20T12:08:30.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.455+0000 7ff6d17fa700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6bc056d20 0x7ff6bc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.455+0000 7ff6d17fa700 1 -- 172.21.15.187:0/2282472259 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff6d406f600 con 0x7ff6e4065570 2020-11-20T12:08:30.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.455+0000 7ff6e90ce700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6bc056d20 0x7ff6bc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.456+0000 7ff6e90ce700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6bc056d20 0x7ff6bc0591e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7ff6e00012f0 tx=0x7ff6e00077d0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:30.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.459+0000 7ff6d17fa700 1 -- 172.21.15.187:0/2282472259 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff6d4044a50 con 0x7ff6e4065570 2020-11-20T12:08:30.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.646+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6e4061a20 con 0x7ff6e4065570 2020-11-20T12:08:30.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.647+0000 7ff6d17fa700 1 -- 172.21.15.187:0/2282472259 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16793 (secure 0 0 0) 0x7ff6d4048070 con 0x7ff6e4065570 2020-11-20T12:08:30.652 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:30.653 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":50,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:08:30.176995+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:08:28.213673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2020-11-20T12:08:28.856619+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:30.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6bc056d20 msgr2=0x7ff6bc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6bc056d20 0x7ff6bc0591e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7ff6e00012f0 tx=0x7ff6e00077d0).stop 2020-11-20T12:08:30.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 msgr2=0x7ff6e40b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e40b4440 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7ff6d400c740 tx=0x7ff6d4007aa0).stop 2020-11-20T12:08:30.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 shutdown_connections 2020-11-20T12:08:30.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6bc056d20 0x7ff6bc0591e0 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0x7ff6e00012f0 tx=0x7ff6e00077d0).stop 2020-11-20T12:08:30.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6e4065f90 0x7ff6e41480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:30.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6e4065190 0x7ff6e415b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:30.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 --2- 172.21.15.187:0/2282472259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6e4065570 0x7ff6e40b4440 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.649+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 >> 172.21.15.187:0/2282472259 conn(0x7ff6e40bab30 msgr2=0x7ff6e40baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:30.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.650+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 shutdown_connections 2020-11-20T12:08:30.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.650+0000 7ff6ea0d0700 1 -- 172.21.15.187:0/2282472259 wait complete. 2020-11-20T12:08:30.676 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:30.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.908+0000 7f976e495700 1 Processor -- start 2020-11-20T12:08:30.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.908+0000 7f976e495700 1 -- start start 2020-11-20T12:08:30.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.910+0000 7f976e495700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f97681484d0 0x7f9768061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.910+0000 7f976e495700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97680622c0 0x7f97680b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.910+0000 7f976e495700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9768066f70 0x7f97680b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.910+0000 7f976e495700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9768067730 con 0x7f97681484d0 2020-11-20T12:08:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.910+0000 7f976e495700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97680b1300 con 0x7f97680622c0 2020-11-20T12:08:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.910+0000 7f976e495700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97680b15a0 con 0x7f9768066f70 2020-11-20T12:08:30.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.910+0000 7f976cc92700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97680622c0 0x7f97680b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976d493700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f97681484d0 0x7f9768061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976d493700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f97681484d0 0x7f9768061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:36122/0 (socket says 172.21.15.187:36122) 2020-11-20T12:08:30.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976d493700 1 -- 172.21.15.187:0/410622476 learned_addr learned my addr 172.21.15.187:0/410622476 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:30.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976dc94700 1 --2- 172.21.15.187:0/410622476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9768066f70 0x7f97680b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976cc92700 1 -- 172.21.15.187:0/410622476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9768066f70 msgr2=0x7f97680b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976cc92700 1 --2- 172.21.15.187:0/410622476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9768066f70 0x7f97680b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976cc92700 1 -- 172.21.15.187:0/410622476 >> v2:172.21.15.5:3300/0 conn(0x7f97681484d0 msgr2=0x7f9768061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976cc92700 1 --2- 172.21.15.187:0/410622476 >> v2:172.21.15.5:3300/0 conn(0x7f97681484d0 0x7f9768061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.911+0000 7f976cc92700 1 -- 172.21.15.187:0/410622476 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97680b1840 con 0x7f97680622c0 2020-11-20T12:08:30.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.912+0000 7f976cc92700 1 --2- 172.21.15.187:0/410622476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f97680b5640 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f9768068600 tx=0x7f9758006df0).ready entity=mon.1 client_cookie=69677400e845bfa7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:30.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.912+0000 7f9757fff700 1 -- 172.21.15.187:0/410622476 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97580097b0 con 0x7f97680622c0 2020-11-20T12:08:30.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.912+0000 7f9757fff700 1 -- 172.21.15.187:0/410622476 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9758009910 con 0x7f97680622c0 2020-11-20T12:08:30.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.912+0000 7f9757fff700 1 -- 172.21.15.187:0/410622476 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758009c90 con 0x7f97680622c0 2020-11-20T12:08:30.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 -- 172.21.15.187:0/410622476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 msgr2=0x7f97680b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 --2- 172.21.15.187:0/410622476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f97680b5640 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f9768068600 tx=0x7f9758006df0).stop 2020-11-20T12:08:30.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 -- 172.21.15.187:0/410622476 shutdown_connections 2020-11-20T12:08:30.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 --2- 172.21.15.187:0/410622476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9768066f70 0x7f97680b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 --2- 172.21.15.187:0/410622476 >> v2:172.21.15.5:3300/0 conn(0x7f97681484d0 0x7f9768061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 --2- 172.21.15.187:0/410622476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f97680b5640 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 -- 172.21.15.187:0/410622476 >> 172.21.15.187:0/410622476 conn(0x7f97680bab30 msgr2=0x7f97680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:30.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 -- 172.21.15.187:0/410622476 shutdown_connections 2020-11-20T12:08:30.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.913+0000 7f976e495700 1 -- 172.21.15.187:0/410622476 wait complete. 2020-11-20T12:08:30.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.915+0000 7f976e495700 1 Processor -- start 2020-11-20T12:08:30.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.915+0000 7f976e495700 1 -- start start 2020-11-20T12:08:30.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.916+0000 7f976e495700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f976814a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.916+0000 7f976e495700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768066f70 0x7f9768098e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.916+0000 7f976e495700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97681484d0 0x7f9768099380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.916+0000 7f976e495700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768158ed0 con 0x7f9768066f70 2020-11-20T12:08:30.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.916+0000 7f976e495700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681584f0 con 0x7f97680622c0 2020-11-20T12:08:30.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.916+0000 7f976e495700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9768158790 con 0x7f97681484d0 2020-11-20T12:08:30.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976d493700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f976814a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976d493700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f976814a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43866/0 (socket says 172.21.15.187:43866) 2020-11-20T12:08:30.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976d493700 1 -- 172.21.15.187:0/3174097663 learned_addr learned my addr 172.21.15.187:0/3174097663 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:30.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976cc92700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768066f70 0x7f9768098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976dc94700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97681484d0 0x7f9768099380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976d493700 1 -- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97681484d0 msgr2=0x7f9768099380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976d493700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97681484d0 0x7f9768099380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976d493700 1 -- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768066f70 msgr2=0x7f9768098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:30.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976d493700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768066f70 0x7f9768098e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:30.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976d493700 1 -- 172.21.15.187:0/3174097663 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9768158a30 con 0x7f97680622c0 2020-11-20T12:08:30.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.917+0000 7f976dc94700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97681484d0 0x7f9768099380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:30.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.918+0000 7f976d493700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f976814a1c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f976400ef20 tx=0x7f9764009290).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:30.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.918+0000 7f97557fa700 1 -- 172.21.15.187:0/3174097663 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976400cc70 con 0x7f97680622c0 2020-11-20T12:08:30.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.918+0000 7f97557fa700 1 -- 172.21.15.187:0/3174097663 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976400cdd0 con 0x7f97680622c0 2020-11-20T12:08:30.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.918+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9768099870 con 0x7f97680622c0 2020-11-20T12:08:30.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.918+0000 7f97557fa700 1 -- 172.21.15.187:0/3174097663 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764012620 con 0x7f97680622c0 2020-11-20T12:08:30.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.918+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976814a6b0 con 0x7f97680622c0 2020-11-20T12:08:30.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.919+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9768061a20 con 0x7f97680622c0 2020-11-20T12:08:30.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.920+0000 7f97557fa700 1 -- 172.21.15.187:0/3174097663 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9764022070 con 0x7f97680622c0 2020-11-20T12:08:30.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.921+0000 7f97557fa700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:30.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.921+0000 7f97557fa700 1 -- 172.21.15.187:0/3174097663 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9764070950 con 0x7f97680622c0 2020-11-20T12:08:30.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.921+0000 7f976cc92700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:30.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.922+0000 7f976cc92700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f9768068600 tx=0x7f9758015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:30.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:30.924+0000 7f97557fa700 1 -- 172.21.15.187:0/3174097663 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9764045da0 con 0x7f97680622c0 2020-11-20T12:08:31.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.111+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97681488b0 con 0x7f97680622c0 2020-11-20T12:08:31.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.114+0000 7f97557fa700 1 -- 172.21.15.187:0/3174097663 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16793 (secure 0 0 0) 0x7f9764016070 con 0x7f97680622c0 2020-11-20T12:08:31.119 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:31.119 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":50,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:08:30.176995+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:08:28.213673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2020-11-20T12:08:28.856619+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:31.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 msgr2=0x7f97400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:31.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f9768068600 tx=0x7f9758015000).stop 2020-11-20T12:08:31.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 msgr2=0x7f976814a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:31.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f976814a1c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f976400ef20 tx=0x7f9764009290).stop 2020-11-20T12:08:31.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 shutdown_connections 2020-11-20T12:08:31.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:31.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97681484d0 0x7f9768099380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:31.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97680622c0 0x7f976814a1c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:31.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 --2- 172.21.15.187:0/3174097663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768066f70 0x7f9768098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:31.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.116+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 >> 172.21.15.187:0/3174097663 conn(0x7f97680bab30 msgr2=0x7f97680b9000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:31.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.117+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 shutdown_connections 2020-11-20T12:08:31.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.117+0000 7f976e495700 1 -- 172.21.15.187:0/3174097663 wait complete. 2020-11-20T12:08:31.144 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:08:31.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.372+0000 7fbe0b59e700 1 Processor -- start 2020-11-20T12:08:31.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.373+0000 7fbe0b59e700 1 -- start start 2020-11-20T12:08:31.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.374+0000 7fbe0b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065570 0x7fbe0c09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:31.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.374+0000 7fbe0b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c0b13e0 con 0x7fbe0c065f90 2020-11-20T12:08:31.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.374+0000 7fbe0b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c09d690 con 0x7fbe0c065190 2020-11-20T12:08:31.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.374+0000 7fbe0b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c14ab30 con 0x7fbe0c065570 2020-11-20T12:08:31.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.374+0000 7fbe0a59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbe0c065190 0x7fbe0c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40478/0 (socket says 172.21.15.187:40478) 2020-11-20T12:08:31.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.374+0000 7fbe0a59c700 1 -- 172.21.15.187:0/3761981229 learned_addr learned my addr 172.21.15.187:0/3761981229 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:31.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.375+0000 7fbe0ad9d700 1 --2- 172.21.15.187:0/3761981229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065570 0x7fbe0c09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:31.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.375+0000 7fbe09d9b700 1 -- 172.21.15.187:0/3761981229 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575807457 0 0) 0x7fbe0c09d690 con 0x7fbe0c065190 2020-11-20T12:08:31.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.376+0000 7fbe09d9b700 1 -- 172.21.15.187:0/3761981229 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec003240 con 0x7fbe0c065190 2020-11-20T12:08:31.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.376+0000 7fbe09d9b700 1 -- 172.21.15.187:0/3761981229 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784779760 0 0) 0x7fbe0c0b13e0 con 0x7fbe0c065f90 2020-11-20T12:08:31.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.376+0000 7fbe09d9b700 1 -- 172.21.15.187:0/3761981229 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe0c09d690 con 0x7fbe0c065f90 2020-11-20T12:08:31.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.376+0000 7fbe0ad9d700 1 -- 172.21.15.187:0/3761981229 >> v1:172.21.15.187:6789/0 conn(0x7fbe0c065190 legacy=0x7fbe0c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:31.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.376+0000 7fbe09d9b700 1 -- 172.21.15.187:0/3761981229 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 843459973 0 0) 0x7fbdec003240 con 0x7fbe0c065190 2020-11-20T12:08:31.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.376+0000 7fbe0ad9d700 1 -- 172.21.15.187:0/3761981229 >> v1:172.21.15.5:6789/0 conn(0x7fbe0c065f90 legacy=0x7fbe0c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:31.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.376+0000 7fbe0ad9d700 1 -- 172.21.15.187:0/3761981229 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c062c10 con 0x7fbe0c065570 2020-11-20T12:08:31.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.377+0000 7fbe0ad9d700 1 --2- 172.21.15.187:0/3761981229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065570 0x7fbe0c09d1a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fbe04009c40 tx=0x7fbe040091d0).ready entity=mon.2 client_cookie=f761edc89f002558 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:31.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.377+0000 7fbe09d9b700 1 -- 172.21.15.187:0/3761981229 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc0014e0 con 0x7fbe0c065570 2020-11-20T12:08:31.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.377+0000 7fbe09d9b700 1 -- 172.21.15.187:0/3761981229 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0400b7c0 con 0x7fbe0c065570 2020-11-20T12:08:31.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.377+0000 7fbe09d9b700 1 -- 172.21.15.187:0/3761981229 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0400bb40 con 0x7fbe0c065570 2020-11-20T12:08:31.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.378+0000 7fbe0b59e700 1 -- 172.21.15.187:0/3761981229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065570 msgr2=0x7fbe0c09d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:31.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.378+0000 7fbe0b59e700 1 --2- 172.21.15.187:0/3761981229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065570 0x7fbe0c09d1a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fbe04009c40 tx=0x7fbe040091d0).stop 2020-11-20T12:08:31.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.378+0000 7fbe0b59e700 1 -- 172.21.15.187:0/3761981229 shutdown_connections 2020-11-20T12:08:31.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.378+0000 7fbe0b59e700 1 --2- 172.21.15.187:0/3761981229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065570 0x7fbe0c09d1a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:31.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.378+0000 7fbe0b59e700 1 -- 172.21.15.187:0/3761981229 >> 172.21.15.187:0/3761981229 conn(0x7fbe0c0bab30 msgr2=0x7fbe0c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:31.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.378+0000 7fbe0b59e700 1 -- 172.21.15.187:0/3761981229 shutdown_connections 2020-11-20T12:08:31.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.378+0000 7fbe0b59e700 1 -- 172.21.15.187:0/3761981229 wait complete. 2020-11-20T12:08:31.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.379+0000 7fbe0b59e700 1 Processor -- start 2020-11-20T12:08:31.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.380+0000 7fbe0b59e700 1 -- start start 2020-11-20T12:08:31.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.380+0000 7fbe0b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0c065190 0x7fbe0c158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:31.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.380+0000 7fbe0b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c065570 0x7fbe0c148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:31.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.380+0000 7fbe0b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065f90 0x7fbe0c147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:31.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c1474c0 con 0x7fbe0c065570 2020-11-20T12:08:31.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c147760 con 0x7fbe0c065190 2020-11-20T12:08:31.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c147a00 con 0x7fbe0c065f90 2020-11-20T12:08:31.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0c065190 0x7fbe0c158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:31.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0c065190 0x7fbe0c158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43878/0 (socket says 172.21.15.187:43878) 2020-11-20T12:08:31.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0a59c700 1 -- 172.21.15.187:0/684088506 learned_addr learned my addr 172.21.15.187:0/684088506 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:31.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe01d9b700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c065570 0x7fbe0c148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:31.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0a59c700 1 -- 172.21.15.187:0/684088506 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065f90 msgr2=0x7fbe0c147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:31.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0a59c700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065f90 0x7fbe0c147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:31.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0a59c700 1 -- 172.21.15.187:0/684088506 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c065570 msgr2=0x7fbe0c148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:31.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0a59c700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c065570 0x7fbe0c148c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:31.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe0a59c700 1 -- 172.21.15.187:0/684088506 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c159000 con 0x7fbe0c065190 2020-11-20T12:08:31.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.381+0000 7fbe01d9b700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c065570 0x7fbe0c148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:31.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.382+0000 7fbe0a59c700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0c065190 0x7fbe0c158b10 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fbdfc00cae0 tx=0x7fbdfc001b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:31.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.382+0000 7fbe02ffd700 1 -- 172.21.15.187:0/684088506 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc007170 con 0x7fbe0c065190 2020-11-20T12:08:31.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.382+0000 7fbe02ffd700 1 -- 172.21.15.187:0/684088506 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdfc0072f0 con 0x7fbe0c065190 2020-11-20T12:08:31.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.382+0000 7fbe02ffd700 1 -- 172.21.15.187:0/684088506 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc00aad0 con 0x7fbe0c065190 2020-11-20T12:08:31.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.382+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c149120 con 0x7fbe0c065190 2020-11-20T12:08:31.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.382+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c0a0290 con 0x7fbe0c065190 2020-11-20T12:08:31.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.383+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0c061a20 con 0x7fbe0c065190 2020-11-20T12:08:31.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.384+0000 7fbe02ffd700 1 -- 172.21.15.187:0/684088506 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbdfc010070 con 0x7fbe0c065190 2020-11-20T12:08:31.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.385+0000 7fbe02ffd700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde8056d20 0x7fbde80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:31.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.385+0000 7fbe02ffd700 1 -- 172.21.15.187:0/684088506 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbdfc06fc70 con 0x7fbe0c065190 2020-11-20T12:08:31.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.385+0000 7fbe01d9b700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde8056d20 0x7fbde80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:31.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.385+0000 7fbe01d9b700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde8056d20 0x7fbde80591e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fbe0c09cf20 tx=0x7fbdf4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:31.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.388+0000 7fbe02ffd700 1 -- 172.21.15.187:0/684088506 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbdfc0450c0 con 0x7fbe0c065190 2020-11-20T12:08:31.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:31.639+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbe0c0bb8d0 con 0x7fbe0c065190 2020-11-20T12:08:32.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.323+0000 7fbe02ffd700 1 -- 172.21.15.187:0/684088506 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbdfc0486e0 con 0x7fbe0c065190 2020-11-20T12:08:32.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde8056d20 msgr2=0x7fbde80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:32.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde8056d20 0x7fbde80591e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fbe0c09cf20 tx=0x7fbdf4009040).stop 2020-11-20T12:08:32.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0c065190 msgr2=0x7fbe0c158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:32.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0c065190 0x7fbe0c158b10 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fbdfc00cae0 tx=0x7fbdfc001b40).stop 2020-11-20T12:08:32.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 shutdown_connections 2020-11-20T12:08:32.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde8056d20 0x7fbde80591e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c065f90 0x7fbe0c147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0c065190 0x7fbe0c158b10 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 --2- 172.21.15.187:0/684088506 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c065570 0x7fbe0c148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.326+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 >> 172.21.15.187:0/684088506 conn(0x7fbe0c0bab30 msgr2=0x7fbe0c0aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.327+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 shutdown_connections 2020-11-20T12:08:32.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.327+0000 7fbe0b59e700 1 -- 172.21.15.187:0/684088506 wait complete. 2020-11-20T12:08:32.360 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:32.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.603+0000 7fa93cc8d700 1 Processor -- start 2020-11-20T12:08:32.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.603+0000 7fa93cc8d700 1 -- start start 2020-11-20T12:08:32.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.604+0000 7fa93cc8d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa9381484d0 0x7fa938061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:32.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.604+0000 7fa93cc8d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9380622c0 0x7fa9380b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:32.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.604+0000 7fa93cc8d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9380b1300 con 0x7fa938066f70 2020-11-20T12:08:32.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.604+0000 7fa93cc8d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa938067730 con 0x7fa9381484d0 2020-11-20T12:08:32.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.604+0000 7fa93cc8d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380b15a0 con 0x7fa9380622c0 2020-11-20T12:08:32.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.605+0000 7fa92effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9380622c0 0x7fa9380b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:32.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.606+0000 7fa92effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9380622c0 0x7fa9380b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:44778/0 (socket says 172.21.15.187:44778) 2020-11-20T12:08:32.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.606+0000 7fa92effd700 1 -- 172.21.15.187:0/2905021165 learned_addr learned my addr 172.21.15.187:0/2905021165 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:32.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.606+0000 7fa9377fe700 1 --2- 172.21.15.187:0/2905021165 >> v2:172.21.15.5:3300/0 conn(0x7fa9381484d0 0x7fa938061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.607+0000 7fa936ffd700 1 -- 172.21.15.187:0/2905021165 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477795910 0 0) 0x7fa9380b1300 con 0x7fa938066f70 2020-11-20T12:08:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.607+0000 7fa92effd700 1 -- 172.21.15.187:0/2905021165 >> v2:172.21.15.5:3300/0 conn(0x7fa9381484d0 msgr2=0x7fa938061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.607+0000 7fa92effd700 1 --2- 172.21.15.187:0/2905021165 >> v2:172.21.15.5:3300/0 conn(0x7fa9381484d0 0x7fa938061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.607+0000 7fa92effd700 1 -- 172.21.15.187:0/2905021165 >> v1:172.21.15.5:6789/0 conn(0x7fa938066f70 legacy=0x7fa9380b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.607+0000 7fa92effd700 1 -- 172.21.15.187:0/2905021165 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9380b1840 con 0x7fa9380622c0 2020-11-20T12:08:32.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.607+0000 7fa92effd700 1 --2- 172.21.15.187:0/2905021165 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9380622c0 0x7fa9380b5640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fa920006fd0 tx=0x7fa920006eb0).ready entity=mon.2 client_cookie=4f2f2d5687d7b653 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:32.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.608+0000 7fa936ffd700 1 -- 172.21.15.187:0/2905021165 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa920009810 con 0x7fa9380622c0 2020-11-20T12:08:32.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.608+0000 7fa936ffd700 1 -- 172.21.15.187:0/2905021165 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa920009970 con 0x7fa9380622c0 2020-11-20T12:08:32.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.608+0000 7fa936ffd700 1 -- 172.21.15.187:0/2905021165 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa920009cf0 con 0x7fa9380622c0 2020-11-20T12:08:32.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.609+0000 7fa93cc8d700 1 -- 172.21.15.187:0/2905021165 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9380622c0 msgr2=0x7fa9380b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.609+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/2905021165 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9380622c0 0x7fa9380b5640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fa920006fd0 tx=0x7fa920006eb0).stop 2020-11-20T12:08:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.609+0000 7fa93cc8d700 1 -- 172.21.15.187:0/2905021165 shutdown_connections 2020-11-20T12:08:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.609+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/2905021165 >> v2:172.21.15.5:3300/0 conn(0x7fa9381484d0 0x7fa938061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.609+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/2905021165 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9380622c0 0x7fa9380b5640 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.609+0000 7fa93cc8d700 1 -- 172.21.15.187:0/2905021165 >> 172.21.15.187:0/2905021165 conn(0x7fa9380bab30 msgr2=0x7fa9380baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:32.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.609+0000 7fa93cc8d700 1 -- 172.21.15.187:0/2905021165 shutdown_connections 2020-11-20T12:08:32.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.609+0000 7fa93cc8d700 1 -- 172.21.15.187:0/2905021165 wait complete. 2020-11-20T12:08:32.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.611+0000 7fa93cc8d700 1 Processor -- start 2020-11-20T12:08:32.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.611+0000 7fa93cc8d700 1 -- start start 2020-11-20T12:08:32.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.612+0000 7fa93cc8d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa938066f70 0x7fa938065a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.612+0000 7fa93cc8d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9381484d0 0x7fa9380b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.612+0000 7fa93cc8d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa93809b450 0x7fa93809b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.612+0000 7fa93cc8d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380b4450 con 0x7fa93809b450 2020-11-20T12:08:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.612+0000 7fa93cc8d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa938145360 con 0x7fa9381484d0 2020-11-20T12:08:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.612+0000 7fa93cc8d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa938145600 con 0x7fa938066f70 2020-11-20T12:08:32.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa92effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9381484d0 0x7fa9380b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:32.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa92effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9381484d0 0x7fa9380b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43894/0 (socket says 172.21.15.187:43894) 2020-11-20T12:08:32.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa92effd700 1 -- 172.21.15.187:0/3062370624 learned_addr learned my addr 172.21.15.187:0/3062370624 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:32.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa9377fe700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa938066f70 0x7fa938065a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:32.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa937fff700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa93809b450 0x7fa93809b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:32.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa92effd700 1 -- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa938066f70 msgr2=0x7fa938065a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa92effd700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa938066f70 0x7fa938065a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa92effd700 1 -- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa93809b450 msgr2=0x7fa93809b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa92effd700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa93809b450 0x7fa93809b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa92effd700 1 -- 172.21.15.187:0/3062370624 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938065f50 con 0x7fa9381484d0 2020-11-20T12:08:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.613+0000 7fa937fff700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa93809b450 0x7fa93809b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:32.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.614+0000 7fa92effd700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9381484d0 0x7fa9380b3b40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fa938064da0 tx=0x7fa920015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:32.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.614+0000 7fa92ffff700 1 -- 172.21.15.187:0/3062370624 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa920011e00 con 0x7fa9381484d0 2020-11-20T12:08:32.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.614+0000 7fa92ffff700 1 -- 172.21.15.187:0/3062370624 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92001e460 con 0x7fa9381484d0 2020-11-20T12:08:32.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.614+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93809b0f0 con 0x7fa9381484d0 2020-11-20T12:08:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.614+0000 7fa92ffff700 1 -- 172.21.15.187:0/3062370624 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92001e7e0 con 0x7fa9381484d0 2020-11-20T12:08:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.614+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9380b41f0 con 0x7fa9381484d0 2020-11-20T12:08:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.615+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9381457c0 con 0x7fa9381484d0 2020-11-20T12:08:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.616+0000 7fa92ffff700 1 -- 172.21.15.187:0/3062370624 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa920025030 con 0x7fa9381484d0 2020-11-20T12:08:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.617+0000 7fa92ffff700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa914056d50 0x7fa914059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:32.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.617+0000 7fa92ffff700 1 -- 172.21.15.187:0/3062370624 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa920071580 con 0x7fa9381484d0 2020-11-20T12:08:32.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.617+0000 7fa9377fe700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa914056d50 0x7fa914059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:32.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.618+0000 7fa9377fe700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa914056d50 0x7fa914059210 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa928006fd0 tx=0x7fa928008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:32.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.621+0000 7fa92ffff700 1 -- 172.21.15.187:0/3062370624 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa920046a80 con 0x7fa9381484d0 2020-11-20T12:08:32.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.864+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa938061a20 con 0x7fa9381484d0 2020-11-20T12:08:32.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.864+0000 7fa92ffff700 1 -- 172.21.15.187:0/3062370624 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2135 (secure 0 0 0) 0x7fa920023070 con 0x7fa9381484d0 2020-11-20T12:08:32.869 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:32.869 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5019,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/4036313482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4036313482},{"type":"v1","addr":"172.21.15.5:6833","nonce":4036313482}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:08:30.177391+0000","modified":"2020-11-20T12:08:32.308400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5743},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5743":{"gid":5743,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1756209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1756209233},{"type":"v1","addr":"172.21.15.187:6835","nonce":1756209233}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2020-11-20T12:08:32.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa914056d50 msgr2=0x7fa914059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:32.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa914056d50 0x7fa914059210 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa928006fd0 tx=0x7fa928008040).stop 2020-11-20T12:08:32.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9381484d0 msgr2=0x7fa9380b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:32.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9381484d0 0x7fa9380b3b40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fa938064da0 tx=0x7fa920015000).stop 2020-11-20T12:08:32.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 shutdown_connections 2020-11-20T12:08:32.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa914056d50 0x7fa914059210 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa93809b450 0x7fa93809b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa938066f70 0x7fa938065a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 --2- 172.21.15.187:0/3062370624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9381484d0 0x7fa9380b3b40 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:32.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 >> 172.21.15.187:0/3062370624 conn(0x7fa9380bab30 msgr2=0x7fa9380b6050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:32.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 shutdown_connections 2020-11-20T12:08:32.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:32.867+0000 7fa93cc8d700 1 -- 172.21.15.187:0/3062370624 wait complete. 2020-11-20T12:08:32.874 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 26 2020-11-20T12:08:32.894 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:33.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.131+0000 7ffbbad6e700 1 Processor -- start 2020-11-20T12:08:33.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.131+0000 7ffbbad6e700 1 -- start start 2020-11-20T12:08:33.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.132+0000 7ffbbad6e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40bfbe0 0x7ffbb41480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:33.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.132+0000 7ffbbad6e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb414d110 con 0x7ffbb40bf800 2020-11-20T12:08:33.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.132+0000 7ffbbad6e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb414cbd0 con 0x7ffbb409f560 2020-11-20T12:08:33.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.132+0000 7ffbbad6e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb414c9d0 con 0x7ffbb40bfbe0 2020-11-20T12:08:33.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.133+0000 7ffbba56d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ffbb409f560 0x7ffbb40a36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40508/0 (socket says 172.21.15.187:40508) 2020-11-20T12:08:33.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.133+0000 7ffbba56d700 1 -- 172.21.15.187:0/2498801443 learned_addr learned my addr 172.21.15.187:0/2498801443 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:33.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.133+0000 7ffbb956b700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40bfbe0 0x7ffbb41480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:33.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.133+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018607447 0 0) 0x7ffbb414cbd0 con 0x7ffbb409f560 2020-11-20T12:08:33.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.133+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb9c003240 con 0x7ffbb409f560 2020-11-20T12:08:33.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2320756092 0 0) 0x7ffb9c003240 con 0x7ffbb409f560 2020-11-20T12:08:33.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40bfbe0 msgr2=0x7ffbb41480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:33.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40bfbe0 0x7ffbb41480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:33.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 >> v1:172.21.15.5:6789/0 conn(0x7ffbb40bf800 legacy=0x7ffbb414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:33.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb414ce70 con 0x7ffbb409f560 2020-11-20T12:08:33.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ffbac001860 con 0x7ffbb409f560 2020-11-20T12:08:33.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 >> v1:172.21.15.187:6789/0 conn(0x7ffbb409f560 legacy=0x7ffbb40a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:33.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0037e0 0x7ffb9c003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:33.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb9c0044e0 0x7ffb9c004920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:33.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c004e10 0x7ffb9c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:33.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb414c9d0 con 0x7ffb9c0037e0 2020-11-20T12:08:33.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c004340 con 0x7ffb9c004e10 2020-11-20T12:08:33.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c005310 con 0x7ffb9c0044e0 2020-11-20T12:08:33.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.134+0000 7ffbba56d700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c004e10 0x7ffb9c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:33.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbb9d6c700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0037e0 0x7ffb9c003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:33.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbba56d700 1 -- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb9c0044e0 msgr2=0x7ffb9c004920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:33.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbba56d700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb9c0044e0 0x7ffb9c004920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:33.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbba56d700 1 -- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0037e0 msgr2=0x7ffb9c003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:33.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbba56d700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0037e0 0x7ffb9c003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:33.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbba56d700 1 -- 172.21.15.187:0/2498801443 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c003240 con 0x7ffb9c004e10 2020-11-20T12:08:33.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbba56d700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c004e10 0x7ffb9c00b730 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7ffbac00c600 tx=0x7ffbb414cd60).ready entity=mon.1 client_cookie=6637ba1f905b9cb3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:33.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac005b80 con 0x7ffb9c004e10 2020-11-20T12:08:33.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac005ce0 con 0x7ffb9c004e10 2020-11-20T12:08:33.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.135+0000 7ffbb8d6a700 1 -- 172.21.15.187:0/2498801443 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac007320 con 0x7ffb9c004e10 2020-11-20T12:08:33.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.136+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2498801443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c004e10 msgr2=0x7ffb9c00b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:33.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.136+0000 7ffbbad6e700 1 --2- 172.21.15.187:0/2498801443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c004e10 0x7ffb9c00b730 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7ffbac00c600 tx=0x7ffbb414cd60).stop 2020-11-20T12:08:33.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.136+0000 7ffbba56d700 1 -- 172.21.15.187:0/2498801443 reap_dead start 2020-11-20T12:08:33.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.136+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2498801443 shutdown_connections 2020-11-20T12:08:33.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.136+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2498801443 >> 172.21.15.187:0/2498801443 conn(0x7ffbb4068370 msgr2=0x7ffbb40bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:33.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.137+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2498801443 shutdown_connections 2020-11-20T12:08:33.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.137+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2498801443 wait complete. 2020-11-20T12:08:33.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.138+0000 7ffbbad6e700 1 Processor -- start 2020-11-20T12:08:33.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.138+0000 7ffbbad6e700 1 -- start start 2020-11-20T12:08:33.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.139+0000 7ffbbad6e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb415b6e0 0x7ffbb40b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:33.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.139+0000 7ffbbad6e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb415bac0 0x7ffbb414b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:33.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.139+0000 7ffbbad6e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb415bf10 0x7ffbb414bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:33.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.139+0000 7ffbbad6e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb414bfd0 con 0x7ffbb415bf10 2020-11-20T12:08:33.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.139+0000 7ffbbad6e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb4147ee0 con 0x7ffbb415bac0 2020-11-20T12:08:33.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.139+0000 7ffbbad6e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb4148180 con 0x7ffbb415b6e0 2020-11-20T12:08:33.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.139+0000 7ffbb956b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb415bac0 0x7ffbb414b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:33.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.139+0000 7ffbb956b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb415bac0 0x7ffbb414b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43912/0 (socket says 172.21.15.187:43912) 2020-11-20T12:08:33.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb956b700 1 -- 172.21.15.187:0/2946702105 learned_addr learned my addr 172.21.15.187:0/2946702105 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:33.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb9d6c700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb415b6e0 0x7ffbb40b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:33.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbba56d700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb415bf10 0x7ffbb414bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:33.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb956b700 1 -- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb415b6e0 msgr2=0x7ffbb40b66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:33.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb956b700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb415b6e0 0x7ffbb40b66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:33.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb956b700 1 -- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb415bf10 msgr2=0x7ffbb414bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:33.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb956b700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb415bf10 0x7ffbb414bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:33.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb956b700 1 -- 172.21.15.187:0/2946702105 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb40b6bc0 con 0x7ffbb415bac0 2020-11-20T12:08:33.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb9d6c700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb415b6e0 0x7ffbb40b66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:33.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffbb956b700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb415bac0 0x7ffbb414b5f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7ffba400bfd0 tx=0x7ffba4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:33.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.140+0000 7ffba1ffb700 1 -- 172.21.15.187:0/2946702105 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba4017070 con 0x7ffbb415bac0 2020-11-20T12:08:33.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.141+0000 7ffba1ffb700 1 -- 172.21.15.187:0/2946702105 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba400ded0 con 0x7ffbb415bac0 2020-11-20T12:08:33.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.141+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb4157f50 con 0x7ffbb415bac0 2020-11-20T12:08:33.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.141+0000 7ffba1ffb700 1 -- 172.21.15.187:0/2946702105 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba40099c0 con 0x7ffbb415bac0 2020-11-20T12:08:33.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.141+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb41584e0 con 0x7ffbb415bac0 2020-11-20T12:08:33.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.142+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4080050 con 0x7ffbb415bac0 2020-11-20T12:08:33.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.142+0000 7ffba1ffb700 1 -- 172.21.15.187:0/2946702105 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffba4013480 con 0x7ffbb415bac0 2020-11-20T12:08:33.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.143+0000 7ffba1ffb700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:33.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.143+0000 7ffba1ffb700 1 -- 172.21.15.187:0/2946702105 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffba406ff70 con 0x7ffbb415bac0 2020-11-20T12:08:33.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.143+0000 7ffbb9d6c700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:33.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.144+0000 7ffbb9d6c700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7ffbb00034c0 tx=0x7ffbb0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:33.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.147+0000 7ffba1ffb700 1 -- 172.21.15.187:0/2946702105 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffba4045360 con 0x7ffbb415bac0 2020-11-20T12:08:33.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.332+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbb4148340 con 0x7ffbb415bac0 2020-11-20T12:08:33.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.335+0000 7ffba1ffb700 1 -- 172.21.15.187:0/2946702105 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16780 (secure 0 0 0) 0x7ffba4048980 con 0x7ffbb415bac0 2020-11-20T12:08:33.340 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:33.341 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":51,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:08:31.182203+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:08:28.213673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2020-11-20T12:08:28.856619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:33.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 msgr2=0x7ffb8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:33.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7ffbb00034c0 tx=0x7ffbb0009040).stop 2020-11-20T12:08:33.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb415bac0 msgr2=0x7ffbb414b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:33.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb415bac0 0x7ffbb414b5f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7ffba400bfd0 tx=0x7ffba4008040).stop 2020-11-20T12:08:33.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 shutdown_connections 2020-11-20T12:08:33.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:33.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb415bf10 0x7ffbb414bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:33.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb415b6e0 0x7ffbb40b66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:33.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 --2- 172.21.15.187:0/2946702105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb415bac0 0x7ffbb414b5f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:33.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 >> 172.21.15.187:0/2946702105 conn(0x7ffbb4068370 msgr2=0x7ffbb40bccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:33.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 shutdown_connections 2020-11-20T12:08:33.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:33.339+0000 7ffbbad6e700 1 -- 172.21.15.187:0/2946702105 wait complete. 2020-11-20T12:08:33.368 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:08:34.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.206+0000 7fd50e757700 1 Processor -- start 2020-11-20T12:08:34.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.206+0000 7fd50e757700 1 -- start start 2020-11-20T12:08:34.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.207+0000 7fd50e757700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd5080b57b0 0x7fd508068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.207+0000 7fd50e757700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b5f50 0x7fd5080b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.207+0000 7fd50e757700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd508144b80 con 0x7fd5081482d0 2020-11-20T12:08:34.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.207+0000 7fd50e757700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5081489e0 con 0x7fd5080b57b0 2020-11-20T12:08:34.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.207+0000 7fd50e757700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd508148c80 con 0x7fd5080b5f50 2020-11-20T12:08:34.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.207+0000 7fd50d755700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd5080b57b0 0x7fd508068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.208+0000 7fd50cf54700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd5081482d0 0x7fd508062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:34036/0 (socket says 172.21.15.187:34036) 2020-11-20T12:08:34.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.208+0000 7fd50cf54700 1 -- 172.21.15.187:0/4027345113 learned_addr learned my addr 172.21.15.187:0/4027345113 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:34.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.208+0000 7fd50df56700 1 --2- 172.21.15.187:0/4027345113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b5f50 0x7fd5080b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.208+0000 7fd50d755700 1 -- 172.21.15.187:0/4027345113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b5f50 msgr2=0x7fd5080b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:34.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.208+0000 7fd50d755700 1 --2- 172.21.15.187:0/4027345113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b5f50 0x7fd5080b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.208+0000 7fd50d755700 1 -- 172.21.15.187:0/4027345113 >> v1:172.21.15.5:6789/0 conn(0x7fd5081482d0 legacy=0x7fd508062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.208+0000 7fd50d755700 1 -- 172.21.15.187:0/4027345113 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd508148f20 con 0x7fd5080b57b0 2020-11-20T12:08:34.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.209+0000 7fd50d755700 1 --2- 172.21.15.187:0/4027345113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5080b57b0 0x7fd508068230 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7fd50400b9a0 tx=0x7fd504006df0).ready entity=mon.1 client_cookie=bb7eaae80414ec93 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:34.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.209+0000 7fd4f7fff700 1 -- 172.21.15.187:0/4027345113 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5040097b0 con 0x7fd5080b57b0 2020-11-20T12:08:34.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.209+0000 7fd4f7fff700 1 -- 172.21.15.187:0/4027345113 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd504009910 con 0x7fd5080b57b0 2020-11-20T12:08:34.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.209+0000 7fd4f7fff700 1 -- 172.21.15.187:0/4027345113 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504009c90 con 0x7fd5080b57b0 2020-11-20T12:08:34.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.210+0000 7fd50e757700 1 -- 172.21.15.187:0/4027345113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5080b57b0 msgr2=0x7fd508068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.210+0000 7fd50e757700 1 --2- 172.21.15.187:0/4027345113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5080b57b0 0x7fd508068230 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7fd50400b9a0 tx=0x7fd504006df0).stop 2020-11-20T12:08:34.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.210+0000 7fd50e757700 1 -- 172.21.15.187:0/4027345113 shutdown_connections 2020-11-20T12:08:34.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.210+0000 7fd50e757700 1 --2- 172.21.15.187:0/4027345113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b5f50 0x7fd5080b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.210+0000 7fd50e757700 1 --2- 172.21.15.187:0/4027345113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5080b57b0 0x7fd508068230 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.210+0000 7fd50e757700 1 -- 172.21.15.187:0/4027345113 >> 172.21.15.187:0/4027345113 conn(0x7fd5080bab30 msgr2=0x7fd5080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:34.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.210+0000 7fd50e757700 1 -- 172.21.15.187:0/4027345113 shutdown_connections 2020-11-20T12:08:34.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.210+0000 7fd50e757700 1 -- 172.21.15.187:0/4027345113 wait complete. 2020-11-20T12:08:34.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.211+0000 7fd50e757700 1 Processor -- start 2020-11-20T12:08:34.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.211+0000 7fd50e757700 1 -- start start 2020-11-20T12:08:34.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.212+0000 7fd50e757700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b57b0 0x7fd50809f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.212+0000 7fd50e757700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5080b5f50 0x7fd5080aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.212+0000 7fd50e757700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5081482d0 0x7fd5081399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.212+0000 7fd50e757700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b49b0 con 0x7fd5080b5f50 2020-11-20T12:08:34.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.212+0000 7fd50e757700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50813a5d0 con 0x7fd5081482d0 2020-11-20T12:08:34.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.212+0000 7fd50e757700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b4be0 con 0x7fd5080b57b0 2020-11-20T12:08:34.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.212+0000 7fd50df56700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5081482d0 0x7fd5081399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50df56700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5081482d0 0x7fd5081399b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43930/0 (socket says 172.21.15.187:43930) 2020-11-20T12:08:34.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50df56700 1 -- 172.21.15.187:0/2550666961 learned_addr learned my addr 172.21.15.187:0/2550666961 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:34.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50cf54700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5080b5f50 0x7fd5080aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50d755700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b57b0 0x7fd50809f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50df56700 1 -- 172.21.15.187:0/2550666961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b57b0 msgr2=0x7fd50809f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50df56700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b57b0 0x7fd50809f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50df56700 1 -- 172.21.15.187:0/2550666961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5080b5f50 msgr2=0x7fd5080aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50df56700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5080b5f50 0x7fd5080aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50df56700 1 -- 172.21.15.187:0/2550666961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50809d540 con 0x7fd5081482d0 2020-11-20T12:08:34.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.213+0000 7fd50df56700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5081482d0 0x7fd5081399b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fd50000eba0 tx=0x7fd500008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:34.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.214+0000 7fd4f57fa700 1 -- 172.21.15.187:0/2550666961 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50000c7f0 con 0x7fd5081482d0 2020-11-20T12:08:34.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.214+0000 7fd4f57fa700 1 -- 172.21.15.187:0/2550666961 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50000c950 con 0x7fd5081482d0 2020-11-20T12:08:34.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.214+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5080b0400 con 0x7fd5081482d0 2020-11-20T12:08:34.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.214+0000 7fd4f57fa700 1 -- 172.21.15.187:0/2550666961 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50000ccf0 con 0x7fd5081482d0 2020-11-20T12:08:34.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.214+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5080b1df0 con 0x7fd5081482d0 2020-11-20T12:08:34.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.215+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd508061a20 con 0x7fd5081482d0 2020-11-20T12:08:34.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.216+0000 7fd4f57fa700 1 -- 172.21.15.187:0/2550666961 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd500013640 con 0x7fd5081482d0 2020-11-20T12:08:34.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.216+0000 7fd4f57fa700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e0056d50 0x7fd4e0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.216+0000 7fd4f57fa700 1 -- 172.21.15.187:0/2550666961 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd50006fc50 con 0x7fd5081482d0 2020-11-20T12:08:34.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.217+0000 7fd50d755700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e0056d50 0x7fd4e0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.217+0000 7fd50d755700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e0056d50 0x7fd4e0059210 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fd50400b150 tx=0x7fd504015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:34.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.220+0000 7fd4f57fa700 1 -- 172.21.15.187:0/2550666961 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd5000450a0 con 0x7fd5081482d0 2020-11-20T12:08:34.301 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:08:34.297+0000 7f23f8f7d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:08:34.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.479+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fd50813a710 con 0x7fd5081482d0 2020-11-20T12:08:34.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.491+0000 7fd4f57fa700 1 -- 172.21.15.187:0/2550666961 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd5000486c0 con 0x7fd5081482d0 2020-11-20T12:08:34.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e0056d50 msgr2=0x7fd4e0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e0056d50 0x7fd4e0059210 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fd50400b150 tx=0x7fd504015000).stop 2020-11-20T12:08:34.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5081482d0 msgr2=0x7fd5081399b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5081482d0 0x7fd5081399b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fd50000eba0 tx=0x7fd500008040).stop 2020-11-20T12:08:34.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 shutdown_connections 2020-11-20T12:08:34.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd4e0056d50 0x7fd4e0059210 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5081482d0 0x7fd5081399b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5080b57b0 0x7fd50809f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.492+0000 7fd50e757700 1 --2- 172.21.15.187:0/2550666961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5080b5f50 0x7fd5080aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.493+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 >> 172.21.15.187:0/2550666961 conn(0x7fd5080bab30 msgr2=0x7fd50813aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:34.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.493+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 shutdown_connections 2020-11-20T12:08:34.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.493+0000 7fd50e757700 1 -- 172.21.15.187:0/2550666961 wait complete. 2020-11-20T12:08:34.503 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:08:34.521 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:34.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.761+0000 7fbeeae3a700 1 Processor -- start 2020-11-20T12:08:34.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.761+0000 7fbeeae3a700 1 -- start start 2020-11-20T12:08:34.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.763+0000 7fbeeae3a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbee4065190 0x7fbee4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.763+0000 7fbeeae3a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee40b13e0 con 0x7fbee4065f90 2020-11-20T12:08:34.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.763+0000 7fbeeae3a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee4062c10 con 0x7fbee409ca90 2020-11-20T12:08:34.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.763+0000 7fbeeae3a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee414ab30 con 0x7fbee4065190 2020-11-20T12:08:34.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.763+0000 7fbee9e38700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbee4065190 0x7fbee4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.763+0000 7fbeea639700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbee409ca90 0x7fbee409cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40544/0 (socket says 172.21.15.187:40544) 2020-11-20T12:08:34.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.763+0000 7fbeea639700 1 -- 172.21.15.187:0/478170835 learned_addr learned my addr 172.21.15.187:0/478170835 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:34.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.764+0000 7fbee9e38700 1 -- 172.21.15.187:0/478170835 >> v1:172.21.15.187:6789/0 conn(0x7fbee409ca90 legacy=0x7fbee409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.764+0000 7fbee9e38700 1 -- 172.21.15.187:0/478170835 >> v1:172.21.15.5:6789/0 conn(0x7fbee4065f90 legacy=0x7fbee4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:34.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.764+0000 7fbee8e36700 1 -- 172.21.15.187:0/478170835 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493237078 0 0) 0x7fbee4062c10 con 0x7fbee409ca90 2020-11-20T12:08:34.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.764+0000 7fbee9e38700 1 -- 172.21.15.187:0/478170835 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee40b1640 con 0x7fbee4065190 2020-11-20T12:08:34.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.764+0000 7fbee9e38700 1 --2- 172.21.15.187:0/478170835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065190 0x7fbee4065570 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fbee414ac70 tx=0x7fbee00091d0).ready entity=mon.1 client_cookie=429a81a05102b70 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:34.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.764+0000 7fbee8e36700 1 -- 172.21.15.187:0/478170835 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee000bc10 con 0x7fbee4065190 2020-11-20T12:08:34.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.764+0000 7fbee8e36700 1 -- 172.21.15.187:0/478170835 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee000bd70 con 0x7fbee4065190 2020-11-20T12:08:34.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.764+0000 7fbee8e36700 1 -- 172.21.15.187:0/478170835 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee0011610 con 0x7fbee4065190 2020-11-20T12:08:34.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.765+0000 7fbeeae3a700 1 -- 172.21.15.187:0/478170835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065190 msgr2=0x7fbee4065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.765+0000 7fbeeae3a700 1 --2- 172.21.15.187:0/478170835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065190 0x7fbee4065570 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fbee414ac70 tx=0x7fbee00091d0).stop 2020-11-20T12:08:34.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.765+0000 7fbeeae3a700 1 -- 172.21.15.187:0/478170835 shutdown_connections 2020-11-20T12:08:34.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.765+0000 7fbeeae3a700 1 --2- 172.21.15.187:0/478170835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065190 0x7fbee4065570 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.765+0000 7fbeeae3a700 1 -- 172.21.15.187:0/478170835 >> 172.21.15.187:0/478170835 conn(0x7fbee40bab30 msgr2=0x7fbee40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.765+0000 7fbeeae3a700 1 -- 172.21.15.187:0/478170835 shutdown_connections 2020-11-20T12:08:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.766+0000 7fbeeae3a700 1 -- 172.21.15.187:0/478170835 wait complete. 2020-11-20T12:08:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.767+0000 7fbeeae3a700 1 Processor -- start 2020-11-20T12:08:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.767+0000 7fbeeae3a700 1 -- start start 2020-11-20T12:08:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.768+0000 7fbeeae3a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbee4065190 0x7fbee41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.768+0000 7fbeeae3a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065f90 0x7fbee4158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.768+0000 7fbeeae3a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbee409ca90 0x7fbee40b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.768+0000 7fbeeae3a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee4147100 con 0x7fbee4065190 2020-11-20T12:08:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.768+0000 7fbeeae3a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee41473a0 con 0x7fbee4065f90 2020-11-20T12:08:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.768+0000 7fbeeae3a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee40a0720 con 0x7fbee409ca90 2020-11-20T12:08:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065f90 0x7fbee4158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065f90 0x7fbee4158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43942/0 (socket says 172.21.15.187:43942) 2020-11-20T12:08:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 -- 172.21.15.187:0/715611188 learned_addr learned my addr 172.21.15.187:0/715611188 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9e38700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbee4065190 0x7fbee41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbeea639700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbee409ca90 0x7fbee40b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 -- 172.21.15.187:0/715611188 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbee409ca90 msgr2=0x7fbee40b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbee409ca90 0x7fbee40b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 -- 172.21.15.187:0/715611188 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbee4065190 msgr2=0x7fbee41584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbee4065190 0x7fbee41584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:34.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 -- 172.21.15.187:0/715611188 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee4159380 con 0x7fbee4065f90 2020-11-20T12:08:34.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.769+0000 7fbee9637700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065f90 0x7fbee4158e90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fbed400c760 tx=0x7fbed4007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:34.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.770+0000 7fbed1ffb700 1 -- 172.21.15.187:0/715611188 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed400e560 con 0x7fbee4065f90 2020-11-20T12:08:34.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.770+0000 7fbed1ffb700 1 -- 172.21.15.187:0/715611188 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed400e6c0 con 0x7fbee4065f90 2020-11-20T12:08:34.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.770+0000 7fbed1ffb700 1 -- 172.21.15.187:0/715611188 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed400ea80 con 0x7fbee4065f90 2020-11-20T12:08:34.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.770+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee4147ae0 con 0x7fbee4065f90 2020-11-20T12:08:34.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.770+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee4146db0 con 0x7fbee4065f90 2020-11-20T12:08:34.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.771+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee4147d00 con 0x7fbee4065f90 2020-11-20T12:08:34.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.772+0000 7fbed1ffb700 1 -- 172.21.15.187:0/715611188 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbed4019030 con 0x7fbee4065f90 2020-11-20T12:08:34.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.772+0000 7fbed1ffb700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbebc056d20 0x7fbebc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:34.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.772+0000 7fbed1ffb700 1 -- 172.21.15.187:0/715611188 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbed406f3f0 con 0x7fbee4065f90 2020-11-20T12:08:34.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.773+0000 7fbee9e38700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbebc056d20 0x7fbebc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:34.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.773+0000 7fbee9e38700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbebc056d20 0x7fbebc0591e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fbee409fdb0 tx=0x7fbee0016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:34.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:34.777+0000 7fbed1ffb700 1 -- 172.21.15.187:0/715611188 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbed4044890 con 0x7fbee4065f90 2020-11-20T12:08:35.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.023+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbee4061a20 con 0x7fbee4065f90 2020-11-20T12:08:35.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.025+0000 7fbed1ffb700 1 -- 172.21.15.187:0/715611188 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2135 (secure 0 0 0) 0x7fbed4047eb0 con 0x7fbee4065f90 2020-11-20T12:08:35.029 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:35.029 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5019,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/4036313482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4036313482},{"type":"v1","addr":"172.21.15.5:6833","nonce":4036313482}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:08:30.177391+0000","modified":"2020-11-20T12:08:32.308400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5743},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5743":{"gid":5743,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1756209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1756209233},{"type":"v1","addr":"172.21.15.187:6835","nonce":1756209233}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2020-11-20T12:08:35.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.026+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbebc056d20 msgr2=0x7fbebc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:35.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.026+0000 7fbeeae3a700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbebc056d20 0x7fbebc0591e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fbee409fdb0 tx=0x7fbee0016000).stop 2020-11-20T12:08:35.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.026+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065f90 msgr2=0x7fbee4158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:35.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.026+0000 7fbeeae3a700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065f90 0x7fbee4158e90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fbed400c760 tx=0x7fbed4007c00).stop 2020-11-20T12:08:35.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.028+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 shutdown_connections 2020-11-20T12:08:35.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.028+0000 7fbeeae3a700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbebc056d20 0x7fbebc0591e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:35.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.028+0000 7fbeeae3a700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbee4065190 0x7fbee41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:35.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.028+0000 7fbeeae3a700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbee409ca90 0x7fbee40b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:35.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.028+0000 7fbeeae3a700 1 --2- 172.21.15.187:0/715611188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee4065f90 0x7fbee4158e90 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:35.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.028+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 >> 172.21.15.187:0/715611188 conn(0x7fbee40bab30 msgr2=0x7fbee409aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:35.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.028+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 shutdown_connections 2020-11-20T12:08:35.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:35.028+0000 7fbeeae3a700 1 -- 172.21.15.187:0/715611188 wait complete. 2020-11-20T12:08:35.034 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 26 2020-11-20T12:08:35.255 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:08:30.177391+0000', 'modified': '2020-11-20T12:08:32.308400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5743}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5743': {'gid': 5743, 'name': 'a', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1756209233', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1756209233}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1756209233}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:08:35.255 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:08:35.256 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:08:35.542 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":5743,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":5.3345833760000003,"mdsmap_epoch":26,"osdmap_epoch":51,"osdmap_epoch_barrier":51,"uptime":10.979293477000001} 2020-11-20T12:08:35.543 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5743, "mdsmap_epoch": 26, "osdmap_epoch": 51, "osdmap_epoch_barrier": 51, "rank_uptime": 5.334583376, "state": "up:active", "uptime": 10.979293477, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:08:35.543 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:08:35.544 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:08:35.544 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:08:35.544 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:08:35.545 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:08:35.545 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:08:35.546 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:08:35.546 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:08:35.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:35.547 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:08:35.607 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:08:35.608 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:08:35.608 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:08:35.609 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:08:35.609 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:08:35.609 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:08:35.610 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:08:35.610 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:08:35.610 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:08:35.610 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:08:35.611 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:08:35.611 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:08:35.611 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:08:35.611 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:08:35.611 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:08:35.612 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42523sec preferred_lft 42523sec 2020-11-20T12:08:35.612 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:08:35.612 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:08:35.612 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:08:35.612 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:08:35.613 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:08:35.613 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:08:35.613 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:08:35.613 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:08:35.614 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:08:35.614 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:08:35.614 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:08:35.614 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:08:35.614 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:08:35.615 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:08:35.615 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:08:35.615 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:08:35.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:35.616 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:08:35.714 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:08:35.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:35.720 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:08:35.779 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:08:35.779 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:08:35.780 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:08:35.780 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:08:35.780 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:08:35.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:35.782 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:08:35.782 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:08:35.783 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:08:35.783 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:08:35.784 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:08:35.784 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:08:36.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:36.040 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:08:36.098 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:08:36.098 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:08:36.099 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:08:36.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:36.100 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:08:36.208 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:36.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.452+0000 7fb527fff700 1 Processor -- start 2020-11-20T12:08:36.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.452+0000 7fb527fff700 1 -- start start 2020-11-20T12:08:36.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.454+0000 7fb527fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb5280b1760 0x7fb528065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.454+0000 7fb527fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb528150390 con 0x7fb528065680 2020-11-20T12:08:36.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.454+0000 7fb527fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb528150630 con 0x7fb5280b1380 2020-11-20T12:08:36.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.454+0000 7fb527fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5281511e0 con 0x7fb5280b1760 2020-11-20T12:08:36.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.454+0000 7fb526ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb5280b1380 0x7fb528150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40554/0 (socket says 172.21.15.187:40554) 2020-11-20T12:08:36.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.454+0000 7fb526ffd700 1 -- 172.21.15.187:0/581257739 learned_addr learned my addr 172.21.15.187:0/581257739 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:36.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.455+0000 7fb51e7fc700 1 --2- 172.21.15.187:0/581257739 >> v2:172.21.15.187:3300/0 conn(0x7fb5280b1760 0x7fb528065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.455+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320469400 0 0) 0x7fb528150630 con 0x7fb5280b1380 2020-11-20T12:08:36.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.455+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb508003240 con 0x7fb5280b1380 2020-11-20T12:08:36.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662039811 0 0) 0x7fb528150390 con 0x7fb528065680 2020-11-20T12:08:36.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528150630 con 0x7fb528065680 2020-11-20T12:08:36.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1897532806 0 0) 0x7fb508003240 con 0x7fb5280b1380 2020-11-20T12:08:36.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 >> v2:172.21.15.187:3300/0 conn(0x7fb5280b1760 msgr2=0x7fb528065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:36.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 --2- 172.21.15.187:0/581257739 >> v2:172.21.15.187:3300/0 conn(0x7fb5280b1760 0x7fb528065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 >> v1:172.21.15.5:6789/0 conn(0x7fb528065680 legacy=0x7fb52814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb528068510 con 0x7fb5280b1380 2020-11-20T12:08:36.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fb5180014e0 con 0x7fb5280b1380 2020-11-20T12:08:36.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 >> v1:172.21.15.187:6789/0 conn(0x7fb5280b1380 legacy=0x7fb528150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080038c0 0x7fb508003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5080044e0 0x7fb508004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb508004e50 0x7fb50800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5281511e0 con 0x7fb508004e50 2020-11-20T12:08:36.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5080053b0 con 0x7fb5080038c0 2020-11-20T12:08:36.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50800bca0 con 0x7fb5080044e0 2020-11-20T12:08:36.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.456+0000 7fb526ffd700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080038c0 0x7fb508003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb526ffd700 1 -- 172.21.15.187:0/581257739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5080044e0 msgr2=0x7fb508004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:36.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb526ffd700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5080044e0 0x7fb508004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:36.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb526ffd700 1 -- 172.21.15.187:0/581257739 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb508004e50 msgr2=0x7fb50800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:36.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb526ffd700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb508004e50 0x7fb50800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:36.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb526ffd700 1 -- 172.21.15.187:0/581257739 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508003240 con 0x7fb5080038c0 2020-11-20T12:08:36.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb5277fe700 1 -- 172.21.15.187:0/581257739 reap_dead start 2020-11-20T12:08:36.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb526ffd700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080038c0 0x7fb508003cc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fb51800c440 tx=0x7fb518007780).ready entity=mon.1 client_cookie=1a5575d257ff2303 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:36.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5180072e0 con 0x7fb5080038c0 2020-11-20T12:08:36.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb518007440 con 0x7fb5080038c0 2020-11-20T12:08:36.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.457+0000 7fb5267fc700 1 -- 172.21.15.187:0/581257739 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518005800 con 0x7fb5080038c0 2020-11-20T12:08:36.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.458+0000 7fb527fff700 1 -- 172.21.15.187:0/581257739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080038c0 msgr2=0x7fb508003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.458+0000 7fb527fff700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080038c0 0x7fb508003cc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fb51800c440 tx=0x7fb518007780).stop 2020-11-20T12:08:36.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.458+0000 7fb527fff700 1 -- 172.21.15.187:0/581257739 shutdown_connections 2020-11-20T12:08:36.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.458+0000 7fb527fff700 1 --2- 172.21.15.187:0/581257739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080038c0 0x7fb508003cc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.458+0000 7fb527fff700 1 -- 172.21.15.187:0/581257739 >> 172.21.15.187:0/581257739 conn(0x7fb5280bab30 msgr2=0x7fb5280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:36.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.458+0000 7fb527fff700 1 -- 172.21.15.187:0/581257739 shutdown_connections 2020-11-20T12:08:36.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.458+0000 7fb527fff700 1 -- 172.21.15.187:0/581257739 wait complete. 2020-11-20T12:08:36.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.459+0000 7fb527fff700 1 Processor -- start 2020-11-20T12:08:36.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.460+0000 7fb527fff700 1 -- start start 2020-11-20T12:08:36.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.460+0000 7fb527fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080038c0 0x7fb5280b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.460+0000 7fb527fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5280b2fd0 0x7fb528148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.460+0000 7fb527fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb528148b30 0x7fb52809efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.460+0000 7fb527fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52809e920 con 0x7fb5080038c0 2020-11-20T12:08:36.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.460+0000 7fb527fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5280b3490 con 0x7fb528148b30 2020-11-20T12:08:36.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.460+0000 7fb527fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52809eb10 con 0x7fb5280b2fd0 2020-11-20T12:08:36.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb5277fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb528148b30 0x7fb52809efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb5277fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb528148b30 0x7fb52809efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43964/0 (socket says 172.21.15.187:43964) 2020-11-20T12:08:36.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb5277fe700 1 -- 172.21.15.187:0/641467233 learned_addr learned my addr 172.21.15.187:0/641467233 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:36.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb526ffd700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080038c0 0x7fb5280b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb51e7fc700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5280b2fd0 0x7fb528148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb5277fe700 1 -- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5280b2fd0 msgr2=0x7fb528148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb5277fe700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5280b2fd0 0x7fb528148640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb5277fe700 1 -- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080038c0 msgr2=0x7fb5280b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb5277fe700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080038c0 0x7fb5280b2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb5277fe700 1 -- 172.21.15.187:0/641467233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5280b4280 con 0x7fb528148b30 2020-11-20T12:08:36.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.461+0000 7fb526ffd700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080038c0 0x7fb5280b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:36.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.462+0000 7fb5277fe700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb528148b30 0x7fb52809efb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7fb5280b50b0 tx=0x7fb520007d00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:36.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.462+0000 7fb51f7fe700 1 -- 172.21.15.187:0/641467233 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520005a70 con 0x7fb528148b30 2020-11-20T12:08:36.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.462+0000 7fb51f7fe700 1 -- 172.21.15.187:0/641467233 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520005bd0 con 0x7fb528148b30 2020-11-20T12:08:36.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.462+0000 7fb51f7fe700 1 -- 172.21.15.187:0/641467233 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520007070 con 0x7fb528148b30 2020-11-20T12:08:36.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.462+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52809f4a0 con 0x7fb528148b30 2020-11-20T12:08:36.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.462+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52809f9c0 con 0x7fb528148b30 2020-11-20T12:08:36.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.463+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52809ecd0 con 0x7fb528148b30 2020-11-20T12:08:36.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.464+0000 7fb51f7fe700 1 -- 172.21.15.187:0/641467233 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb520010030 con 0x7fb528148b30 2020-11-20T12:08:36.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.464+0000 7fb51f7fe700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb504056d50 0x7fb504059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.465+0000 7fb51f7fe700 1 -- 172.21.15.187:0/641467233 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb52006f000 con 0x7fb528148b30 2020-11-20T12:08:36.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.465+0000 7fb526ffd700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb504056d50 0x7fb504059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.466+0000 7fb526ffd700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb504056d50 0x7fb504059210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fb51800c440 tx=0x7fb518015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:36.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.469+0000 7fb51f7fe700 1 -- 172.21.15.187:0/641467233 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb520044450 con 0x7fb528148b30 2020-11-20T12:08:36.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.658+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb528061a20 con 0x7fb528148b30 2020-11-20T12:08:36.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.660+0000 7fb51f7fe700 1 -- 172.21.15.187:0/641467233 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16780 (secure 0 0 0) 0x7fb520047a70 con 0x7fb528148b30 2020-11-20T12:08:36.664 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:36.665 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":51,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:08:31.182203+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:08:28.213673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2020-11-20T12:08:28.856619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:36.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb504056d50 msgr2=0x7fb504059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb504056d50 0x7fb504059210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fb51800c440 tx=0x7fb518015000).stop 2020-11-20T12:08:36.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb528148b30 msgr2=0x7fb52809efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb528148b30 0x7fb52809efb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7fb5280b50b0 tx=0x7fb520007d00).stop 2020-11-20T12:08:36.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 shutdown_connections 2020-11-20T12:08:36.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb504056d50 0x7fb504059210 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080038c0 0x7fb5280b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb528148b30 0x7fb52809efb0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 --2- 172.21.15.187:0/641467233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5280b2fd0 0x7fb528148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.663+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 >> 172.21.15.187:0/641467233 conn(0x7fb5280bab30 msgr2=0x7fb528149090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:36.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.664+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 shutdown_connections 2020-11-20T12:08:36.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.664+0000 7fb527fff700 1 -- 172.21.15.187:0/641467233 wait complete. 2020-11-20T12:08:36.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:08:28.213673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:08:28.856619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:08:36.689 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:08:36.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.926+0000 7fed593a7700 1 Processor -- start 2020-11-20T12:08:36.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.927+0000 7fed593a7700 1 -- start start 2020-11-20T12:08:36.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.928+0000 7fed593a7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fed54065f90 0x7fed54065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.928+0000 7fed593a7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed540b13e0 con 0x7fed5409ca90 2020-11-20T12:08:36.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.928+0000 7fed593a7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed54062c10 con 0x7fed54065190 2020-11-20T12:08:36.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.928+0000 7fed593a7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5414ab30 con 0x7fed54065f90 2020-11-20T12:08:36.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.928+0000 7fed53fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fed54065190 0x7fed54143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40576/0 (socket says 172.21.15.187:40576) 2020-11-20T12:08:36.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.928+0000 7fed53fff700 1 -- 172.21.15.187:0/1631641278 learned_addr learned my addr 172.21.15.187:0/1631641278 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:36.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.929+0000 7fed4b7fe700 1 --2- 172.21.15.187:0/1631641278 >> v2:172.21.15.5:3300/0 conn(0x7fed54065f90 0x7fed54065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.929+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973010334 0 0) 0x7fed54062c10 con 0x7fed54065190 2020-11-20T12:08:36.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.929+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed34003240 con 0x7fed54065190 2020-11-20T12:08:36.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.929+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1079009511 0 0) 0x7fed34003240 con 0x7fed54065190 2020-11-20T12:08:36.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.929+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 >> v2:172.21.15.5:3300/0 conn(0x7fed54065f90 msgr2=0x7fed54065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:36.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.929+0000 7fed537fe700 1 --2- 172.21.15.187:0/1631641278 >> v2:172.21.15.5:3300/0 conn(0x7fed54065f90 0x7fed54065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.929+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 >> v1:172.21.15.5:6789/0 conn(0x7fed5409ca90 legacy=0x7fed5409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed540b1640 con 0x7fed54065190 2020-11-20T12:08:36.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fed440014e0 con 0x7fed54065190 2020-11-20T12:08:36.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 >> v1:172.21.15.187:6789/0 conn(0x7fed54065190 legacy=0x7fed54143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed34003730 0x7fed34003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed340044e0 0x7fed34004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed3400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5414ab30 con 0x7fed34003730 2020-11-20T12:08:36.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed34004280 con 0x7fed34004e50 2020-11-20T12:08:36.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3400bc20 con 0x7fed340044e0 2020-11-20T12:08:36.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.930+0000 7fed58ba6700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed3400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed4b7fe700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed340044e0 0x7fed34004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed58ba6700 1 -- 172.21.15.187:0/1631641278 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed340044e0 msgr2=0x7fed34004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:36.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed58ba6700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed340044e0 0x7fed34004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed58ba6700 1 -- 172.21.15.187:0/1631641278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed34003730 msgr2=0x7fed34003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:36.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed58ba6700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed34003730 0x7fed34003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:36.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed58ba6700 1 -- 172.21.15.187:0/1631641278 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed34003240 con 0x7fed34004e50 2020-11-20T12:08:36.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed58ba6700 1 -- 172.21.15.187:0/1631641278 reap_dead start 2020-11-20T12:08:36.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed58ba6700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed3400b730 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fed4c001e70 tx=0x7fed4c007f10).ready entity=mon.1 client_cookie=e553852c074cfb3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:36.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c001540 con 0x7fed34004e50 2020-11-20T12:08:36.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c003030 con 0x7fed34004e50 2020-11-20T12:08:36.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.931+0000 7fed537fe700 1 -- 172.21.15.187:0/1631641278 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c00cdd0 con 0x7fed34004e50 2020-11-20T12:08:36.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.932+0000 7fed593a7700 1 -- 172.21.15.187:0/1631641278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 msgr2=0x7fed3400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.932+0000 7fed593a7700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed3400b730 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fed4c001e70 tx=0x7fed4c007f10).stop 2020-11-20T12:08:36.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.932+0000 7fed593a7700 1 -- 172.21.15.187:0/1631641278 shutdown_connections 2020-11-20T12:08:36.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.932+0000 7fed593a7700 1 --2- 172.21.15.187:0/1631641278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed3400b730 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:36.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.932+0000 7fed593a7700 1 -- 172.21.15.187:0/1631641278 >> 172.21.15.187:0/1631641278 conn(0x7fed540bab30 msgr2=0x7fed540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:36.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.932+0000 7fed593a7700 1 -- 172.21.15.187:0/1631641278 shutdown_connections 2020-11-20T12:08:36.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.932+0000 7fed593a7700 1 -- 172.21.15.187:0/1631641278 wait complete. 2020-11-20T12:08:36.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.933+0000 7fed593a7700 1 Processor -- start 2020-11-20T12:08:36.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.934+0000 7fed593a7700 1 -- start start 2020-11-20T12:08:36.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.934+0000 7fed593a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed5409e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.934+0000 7fed593a7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed5409ff40 0x7fed5409eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.934+0000 7fed593a7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed540b9e50 0x7fed54148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.934+0000 7fed593a7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed540a0a00 con 0x7fed5409ff40 2020-11-20T12:08:36.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.934+0000 7fed593a7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed540a04c0 con 0x7fed34004e50 2020-11-20T12:08:36.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.934+0000 7fed593a7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed540a0760 con 0x7fed540b9e50 2020-11-20T12:08:36.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed5409e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed5409e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43980/0 (socket says 172.21.15.187:43980) 2020-11-20T12:08:36.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 -- 172.21.15.187:0/554581721 learned_addr learned my addr 172.21.15.187:0/554581721 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:36.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 -- 172.21.15.187:0/554581721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed540b9e50 msgr2=0x7fed54148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed540b9e50 0x7fed54148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:36.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 -- 172.21.15.187:0/554581721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed5409ff40 msgr2=0x7fed5409eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:36.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed5409ff40 0x7fed5409eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:36.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 -- 172.21.15.187:0/554581721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed540ba2d0 con 0x7fed34004e50 2020-11-20T12:08:36.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.935+0000 7fed53fff700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed5409e4b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7fed4400ca90 tx=0x7fed540b0fb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:36.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.936+0000 7fed50ff9700 1 -- 172.21.15.187:0/554581721 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44007170 con 0x7fed34004e50 2020-11-20T12:08:36.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.936+0000 7fed50ff9700 1 -- 172.21.15.187:0/554581721 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed440072f0 con 0x7fed34004e50 2020-11-20T12:08:36.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.936+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed54149220 con 0x7fed34004e50 2020-11-20T12:08:36.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.936+0000 7fed50ff9700 1 -- 172.21.15.187:0/554581721 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4400a9c0 con 0x7fed34004e50 2020-11-20T12:08:36.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.936+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5415b970 con 0x7fed34004e50 2020-11-20T12:08:36.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.937+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed540ba4c0 con 0x7fed34004e50 2020-11-20T12:08:36.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.937+0000 7fed50ff9700 1 -- 172.21.15.187:0/554581721 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fed44010070 con 0x7fed34004e50 2020-11-20T12:08:36.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.938+0000 7fed50ff9700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed30056d20 0x7fed300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:36.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.938+0000 7fed50ff9700 1 -- 172.21.15.187:0/554581721 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fed4406f820 con 0x7fed34004e50 2020-11-20T12:08:36.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.938+0000 7fed4b7fe700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed30056d20 0x7fed300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:36.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.939+0000 7fed4b7fe700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed30056d20 0x7fed300591e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fed3c006ee0 tx=0x7fed3c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:36.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:36.942+0000 7fed50ff9700 1 -- 172.21.15.187:0/554581721 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fed44044c70 con 0x7fed34004e50 2020-11-20T12:08:37.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.125+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fed54061a20 con 0x7fed34004e50 2020-11-20T12:08:37.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.126+0000 7fed50ff9700 1 -- 172.21.15.187:0/554581721 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v51) v1 ==== 110+0+10 (secure 0 0 0) 0x7fed44048290 con 0x7fed34004e50 2020-11-20T12:08:37.129 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:08:37.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed30056d20 msgr2=0x7fed300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed30056d20 0x7fed300591e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fed3c006ee0 tx=0x7fed3c008040).stop 2020-11-20T12:08:37.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 msgr2=0x7fed5409e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed5409e4b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7fed4400ca90 tx=0x7fed540b0fb0).stop 2020-11-20T12:08:37.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 shutdown_connections 2020-11-20T12:08:37.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fed30056d20 0x7fed300591e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fed540b9e50 0x7fed54148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:37.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed34004e50 0x7fed5409e4b0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 --2- 172.21.15.187:0/554581721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fed5409ff40 0x7fed5409eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:37.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 >> 172.21.15.187:0/554581721 conn(0x7fed540bab30 msgr2=0x7fed54144ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:37.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.128+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 shutdown_connections 2020-11-20T12:08:37.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.129+0000 7fed593a7700 1 -- 172.21.15.187:0/554581721 wait complete. 2020-11-20T12:08:37.154 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:08:37.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.393+0000 7f758b340700 1 Processor -- start 2020-11-20T12:08:37.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.393+0000 7f758b340700 1 -- start start 2020-11-20T12:08:37.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.395+0000 7f758b340700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7584065190 0x7f7584065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.395+0000 7f758b340700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7584065f90 0x7f7584143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.395+0000 7f758b340700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840b13e0 con 0x7f7584143c70 2020-11-20T12:08:37.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.395+0000 7f758b340700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f758414ab30 con 0x7f7584065190 2020-11-20T12:08:37.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.395+0000 7f758b340700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7584062c10 con 0x7f7584065f90 2020-11-20T12:08:37.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.395+0000 7f758ab3f700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7584143c70 0x7f758409ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40598/0 (socket says 172.21.15.187:40598) 2020-11-20T12:08:37.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.395+0000 7f758ab3f700 1 -- 172.21.15.187:0/4025576127 learned_addr learned my addr 172.21.15.187:0/4025576127 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:37.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758a33e700 1 --2- 172.21.15.187:0/4025576127 >> v2:172.21.15.5:3300/0 conn(0x7f7584065190 0x7f7584065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f7589b3d700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7584065f90 0x7f7584143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281074918 0 0) 0x7f75840b13e0 con 0x7f7584143c70 2020-11-20T12:08:37.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c003240 con 0x7f7584143c70 2020-11-20T12:08:37.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1163816653 0 0) 0x7f756c003240 con 0x7f7584143c70 2020-11-20T12:08:37.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7584065f90 msgr2=0x7f7584143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:37.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758933c700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7584065f90 0x7f7584143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 >> v2:172.21.15.5:3300/0 conn(0x7f7584065190 msgr2=0x7f7584065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:37.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758933c700 1 --2- 172.21.15.187:0/4025576127 >> v2:172.21.15.5:3300/0 conn(0x7f7584065190 0x7f7584065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.396+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840b1640 con 0x7f7584143c70 2020-11-20T12:08:37.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f757c001860 con 0x7f7584143c70 2020-11-20T12:08:37.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 >> v1:172.21.15.187:6789/0 conn(0x7f7584143c70 legacy=0x7f758409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758933c700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756c003730 0x7f756c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758933c700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7584065f90 0x7f756c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758933c700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f756c0049d0 0x7f756c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758414ab30 con 0x7f756c0049d0 2020-11-20T12:08:37.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7584062c10 con 0x7f756c003730 2020-11-20T12:08:37.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0041f0 con 0x7f7584065f90 2020-11-20T12:08:37.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.397+0000 7f758a33e700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756c003730 0x7f756c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.398+0000 7f7589b3d700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7584065f90 0x7f756c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.398+0000 7f758a33e700 1 -- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7584065f90 msgr2=0x7f756c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:37.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.398+0000 7f758a33e700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7584065f90 0x7f756c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.398+0000 7f758a33e700 1 -- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f756c0049d0 msgr2=0x7f756c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:37.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.398+0000 7f758ab3f700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f756c0049d0 0x7f756c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.398+0000 7f758a33e700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f756c0049d0 0x7f756c004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.398+0000 7f758a33e700 1 -- 172.21.15.187:0/4025576127 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c003240 con 0x7f756c003730 2020-11-20T12:08:37.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.398+0000 7f758a33e700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756c003730 0x7f756c003b70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f758000bc00 tx=0x7f7580006df0).ready entity=mon.1 client_cookie=a658185029eaaef1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:37.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758000b310 con 0x7f756c003730 2020-11-20T12:08:37.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758000eb40 con 0x7f756c003730 2020-11-20T12:08:37.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758933c700 1 -- 172.21.15.187:0/4025576127 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758000eec0 con 0x7f756c003730 2020-11-20T12:08:37.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758b340700 1 -- 172.21.15.187:0/4025576127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756c003730 msgr2=0x7f756c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758b340700 1 --2- 172.21.15.187:0/4025576127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756c003730 0x7f756c003b70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f758000bc00 tx=0x7f7580006df0).stop 2020-11-20T12:08:37.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758ab3f700 1 -- 172.21.15.187:0/4025576127 reap_dead start 2020-11-20T12:08:37.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758b340700 1 -- 172.21.15.187:0/4025576127 shutdown_connections 2020-11-20T12:08:37.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758b340700 1 -- 172.21.15.187:0/4025576127 >> 172.21.15.187:0/4025576127 conn(0x7f75840bab30 msgr2=0x7f75840baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:37.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758b340700 1 -- 172.21.15.187:0/4025576127 shutdown_connections 2020-11-20T12:08:37.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.399+0000 7f758b340700 1 -- 172.21.15.187:0/4025576127 wait complete. 2020-11-20T12:08:37.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.401+0000 7f758b340700 1 Processor -- start 2020-11-20T12:08:37.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.401+0000 7f758b340700 1 -- start start 2020-11-20T12:08:37.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758b340700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f758409bbe0 0x7f75840b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758b340700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f758409bfc0 0x7f7584146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758b340700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7584147220 0x7f7584142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758b340700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758409f8d0 con 0x7f758409bfc0 2020-11-20T12:08:37.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758b340700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758409fb70 con 0x7f7584147220 2020-11-20T12:08:37.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758b340700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7584146670 con 0x7f758409bbe0 2020-11-20T12:08:37.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758ab3f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7584147220 0x7f7584142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758ab3f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7584147220 0x7f7584142ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44004/0 (socket says 172.21.15.187:44004) 2020-11-20T12:08:37.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.402+0000 7f758ab3f700 1 -- 172.21.15.187:0/848696563 learned_addr learned my addr 172.21.15.187:0/848696563 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:37.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f7589b3d700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f758409bfc0 0x7f7584146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f758a33e700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f758409bbe0 0x7f75840b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f758ab3f700 1 -- 172.21.15.187:0/848696563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f758409bbe0 msgr2=0x7f75840b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f758ab3f700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f758409bbe0 0x7f75840b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f758ab3f700 1 -- 172.21.15.187:0/848696563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f758409bfc0 msgr2=0x7f7584146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f758ab3f700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f758409bfc0 0x7f7584146d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f758ab3f700 1 -- 172.21.15.187:0/848696563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7584146830 con 0x7f7584147220 2020-11-20T12:08:37.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f758ab3f700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7584147220 0x7f7584142ca0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f7584144050 tx=0x7f757c009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:37.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f75727fc700 1 -- 172.21.15.187:0/848696563 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c004b50 con 0x7f7584147220 2020-11-20T12:08:37.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f75727fc700 1 -- 172.21.15.187:0/848696563 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c004cd0 con 0x7f7584147220 2020-11-20T12:08:37.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758409f350 con 0x7f7584147220 2020-11-20T12:08:37.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.403+0000 7f75727fc700 1 -- 172.21.15.187:0/848696563 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c0073c0 con 0x7f7584147220 2020-11-20T12:08:37.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.404+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75841433d0 con 0x7f7584147220 2020-11-20T12:08:37.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.405+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758409f730 con 0x7f7584147220 2020-11-20T12:08:37.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.405+0000 7f75727fc700 1 -- 172.21.15.187:0/848696563 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f757c01e070 con 0x7f7584147220 2020-11-20T12:08:37.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.406+0000 7f75727fc700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755c056d20 0x7f755c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.406+0000 7f75727fc700 1 -- 172.21.15.187:0/848696563 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f757c06f5c0 con 0x7f7584147220 2020-11-20T12:08:37.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.406+0000 7f758a33e700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755c056d20 0x7f755c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.407+0000 7f758a33e700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755c056d20 0x7f755c0591e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f758405f200 tx=0x7f7580019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:37.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.410+0000 7f75727fc700 1 -- 172.21.15.187:0/848696563 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f758409f730 con 0x7f7584147220 2020-11-20T12:08:37.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.582+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f758409f730 con 0x7f7584147220 2020-11-20T12:08:37.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.583+0000 7f75727fc700 1 -- 172.21.15.187:0/848696563 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v51) v1 ==== 92+0+10 (secure 0 0 0) 0x7f758409f730 con 0x7f7584147220 2020-11-20T12:08:37.586 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:37.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755c056d20 msgr2=0x7f755c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755c056d20 0x7f755c0591e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f758405f200 tx=0x7f7580019000).stop 2020-11-20T12:08:37.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7584147220 msgr2=0x7f7584142ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7584147220 0x7f7584142ca0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f7584144050 tx=0x7f757c009040).stop 2020-11-20T12:08:37.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 shutdown_connections 2020-11-20T12:08:37.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755c056d20 0x7f755c0591e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7584147220 0x7f7584142ca0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f758409bbe0 0x7f75840b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 --2- 172.21.15.187:0/848696563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f758409bfc0 0x7f7584146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.585+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 >> 172.21.15.187:0/848696563 conn(0x7f75840bab30 msgr2=0x7f758409d140 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.586+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 shutdown_connections 2020-11-20T12:08:37.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.586+0000 7f758b340700 1 -- 172.21.15.187:0/848696563 wait complete. 2020-11-20T12:08:37.610 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:08:37.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.843+0000 7fcdb0a53700 1 Processor -- start 2020-11-20T12:08:37.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.843+0000 7fcdb0a53700 1 -- start start 2020-11-20T12:08:37.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.844+0000 7fcdb0a53700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac1484d0 0x7fcdac061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.844+0000 7fcdb0a53700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdac0622c0 0x7fcdac0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.844+0000 7fcdb0a53700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac0b1300 con 0x7fcdac066f70 2020-11-20T12:08:37.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.844+0000 7fcdb0a53700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdac067730 con 0x7fcdac0622c0 2020-11-20T12:08:37.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.844+0000 7fcdb0a53700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0b15a0 con 0x7fcdac1484d0 2020-11-20T12:08:37.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.845+0000 7fcdab59e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcdac066f70 0x7fcdac0b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40618/0 (socket says 172.21.15.187:40618) 2020-11-20T12:08:37.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.845+0000 7fcdab59e700 1 -- 172.21.15.187:0/2661197258 learned_addr learned my addr 172.21.15.187:0/2661197258 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:37.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.845+0000 7fcda259c700 1 --2- 172.21.15.187:0/2661197258 >> v2:172.21.15.187:3300/0 conn(0x7fcdac0622c0 0x7fcdac0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.845+0000 7fcdaad9d700 1 --2- 172.21.15.187:0/2661197258 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac1484d0 0x7fcdac061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.845+0000 7fcdaa59c700 1 -- 172.21.15.187:0/2661197258 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534818649 0 0) 0x7fcdac0b1300 con 0x7fcdac066f70 2020-11-20T12:08:37.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.845+0000 7fcdaa59c700 1 -- 172.21.15.187:0/2661197258 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd8c003240 con 0x7fcdac066f70 2020-11-20T12:08:37.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.846+0000 7fcda259c700 1 -- 172.21.15.187:0/2661197258 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac1484d0 msgr2=0x7fcdac061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:37.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.846+0000 7fcda259c700 1 --2- 172.21.15.187:0/2661197258 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac1484d0 0x7fcdac061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.846+0000 7fcda259c700 1 -- 172.21.15.187:0/2661197258 >> v1:172.21.15.187:6789/0 conn(0x7fcdac066f70 legacy=0x7fcdac0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.846+0000 7fcda259c700 1 -- 172.21.15.187:0/2661197258 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac0b1840 con 0x7fcdac0622c0 2020-11-20T12:08:37.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.846+0000 7fcda259c700 1 --2- 172.21.15.187:0/2661197258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac0622c0 0x7fcdac0b5640 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fcd9400b9a0 tx=0x7fcd94006df0).ready entity=mon.1 client_cookie=5f3d29d7d41c1c9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:37.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.846+0000 7fcdaa59c700 1 -- 172.21.15.187:0/2661197258 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940097b0 con 0x7fcdac0622c0 2020-11-20T12:08:37.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.846+0000 7fcdaa59c700 1 -- 172.21.15.187:0/2661197258 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94009910 con 0x7fcdac0622c0 2020-11-20T12:08:37.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.846+0000 7fcdaa59c700 1 -- 172.21.15.187:0/2661197258 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94009c90 con 0x7fcdac0622c0 2020-11-20T12:08:37.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.847+0000 7fcdb0a53700 1 -- 172.21.15.187:0/2661197258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac0622c0 msgr2=0x7fcdac0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.847+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/2661197258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac0622c0 0x7fcdac0b5640 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fcd9400b9a0 tx=0x7fcd94006df0).stop 2020-11-20T12:08:37.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.847+0000 7fcdb0a53700 1 -- 172.21.15.187:0/2661197258 shutdown_connections 2020-11-20T12:08:37.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.847+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/2661197258 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac1484d0 0x7fcdac061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.847+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/2661197258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac0622c0 0x7fcdac0b5640 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:37.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.847+0000 7fcdb0a53700 1 -- 172.21.15.187:0/2661197258 >> 172.21.15.187:0/2661197258 conn(0x7fcdac0bab30 msgr2=0x7fcdac0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:37.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.847+0000 7fcdb0a53700 1 -- 172.21.15.187:0/2661197258 shutdown_connections 2020-11-20T12:08:37.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.847+0000 7fcdb0a53700 1 -- 172.21.15.187:0/2661197258 wait complete. 2020-11-20T12:08:37.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.849+0000 7fcdb0a53700 1 Processor -- start 2020-11-20T12:08:37.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.849+0000 7fcdb0a53700 1 -- start start 2020-11-20T12:08:37.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.850+0000 7fcdb0a53700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac0622c0 0x7fcdac09c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.850+0000 7fcdb0a53700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdac066f70 0x7fcdac0b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.850+0000 7fcdb0a53700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac1484d0 0x7fcdac132670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.850+0000 7fcdb0a53700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac09c670 con 0x7fcdac066f70 2020-11-20T12:08:37.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.850+0000 7fcdb0a53700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac09d040 con 0x7fcdac1484d0 2020-11-20T12:08:37.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.850+0000 7fcdb0a53700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac09d2e0 con 0x7fcdac0622c0 2020-11-20T12:08:37.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac1484d0 0x7fcdac132670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac1484d0 0x7fcdac132670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44018/0 (socket says 172.21.15.187:44018) 2020-11-20T12:08:37.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 -- 172.21.15.187:0/196198077 learned_addr learned my addr 172.21.15.187:0/196198077 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:37.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 -- 172.21.15.187:0/196198077 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac0622c0 msgr2=0x7fcdac09c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac0622c0 0x7fcdac09c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:37.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 -- 172.21.15.187:0/196198077 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdac066f70 msgr2=0x7fcdac0b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:37.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdac066f70 0x7fcdac0b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:37.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 -- 172.21.15.187:0/196198077 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac152430 con 0x7fcdac1484d0 2020-11-20T12:08:37.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.851+0000 7fcdab59e700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac1484d0 0x7fcdac132670 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fcda40010a0 tx=0x7fcda4007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:37.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.852+0000 7fcda37fe700 1 -- 172.21.15.187:0/196198077 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda4007320 con 0x7fcdac1484d0 2020-11-20T12:08:37.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.852+0000 7fcda37fe700 1 -- 172.21.15.187:0/196198077 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda4007480 con 0x7fcdac1484d0 2020-11-20T12:08:37.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.852+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac09d580 con 0x7fcdac1484d0 2020-11-20T12:08:37.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.852+0000 7fcda37fe700 1 -- 172.21.15.187:0/196198077 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda4005800 con 0x7fcdac1484d0 2020-11-20T12:08:37.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.852+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac0b4160 con 0x7fcdac1484d0 2020-11-20T12:08:37.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.853+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac061a20 con 0x7fcdac1484d0 2020-11-20T12:08:37.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.854+0000 7fcda37fe700 1 -- 172.21.15.187:0/196198077 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcda4010030 con 0x7fcdac1484d0 2020-11-20T12:08:37.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.854+0000 7fcda37fe700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcd88056d20 0x7fcd880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:37.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.854+0000 7fcda37fe700 1 -- 172.21.15.187:0/196198077 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcda406fc70 con 0x7fcdac1484d0 2020-11-20T12:08:37.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.855+0000 7fcdaad9d700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcd88056d20 0x7fcd880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:37.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.855+0000 7fcdaad9d700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcd88056d20 0x7fcd880591e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fcd9c006fd0 tx=0x7fcd9c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:37.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:37.858+0000 7fcda37fe700 1 -- 172.21.15.187:0/196198077 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcda40450c0 con 0x7fcdac1484d0 2020-11-20T12:08:38.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.045+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcdac1488b0 con 0x7fcdac1484d0 2020-11-20T12:08:38.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.045+0000 7fcda37fe700 1 -- 172.21.15.187:0/196198077 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcda4007be0 con 0x7fcdac1484d0 2020-11-20T12:08:38.049 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:38.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.047+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcd88056d20 msgr2=0x7fcd880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.047+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcd88056d20 0x7fcd880591e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fcd9c006fd0 tx=0x7fcd9c008040).stop 2020-11-20T12:08:38.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.047+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac1484d0 msgr2=0x7fcdac132670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.047+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac1484d0 0x7fcdac132670 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fcda40010a0 tx=0x7fcda4007aa0).stop 2020-11-20T12:08:38.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.048+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 shutdown_connections 2020-11-20T12:08:38.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.048+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcd88056d20 0x7fcd880591e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.048+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdac1484d0 0x7fcdac132670 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.048+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdac0622c0 0x7fcdac09c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:38.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.048+0000 7fcdb0a53700 1 --2- 172.21.15.187:0/196198077 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdac066f70 0x7fcdac0b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:38.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.048+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 >> 172.21.15.187:0/196198077 conn(0x7fcdac0bab30 msgr2=0x7fcdac0b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:38.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.048+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 shutdown_connections 2020-11-20T12:08:38.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.048+0000 7fcdb0a53700 1 -- 172.21.15.187:0/196198077 wait complete. 2020-11-20T12:08:38.077 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:08:38.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.308+0000 7fa7397bd700 1 Processor -- start 2020-11-20T12:08:38.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.309+0000 7fa7397bd700 1 -- start start 2020-11-20T12:08:38.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.310+0000 7fa7397bd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa734068230 0x7fa7341482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.310+0000 7fa7397bd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa7340b5f50 0x7fa7340b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.310+0000 7fa7397bd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa734144b80 con 0x7fa7340b57b0 2020-11-20T12:08:38.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.310+0000 7fa7397bd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7341487c0 con 0x7fa7340b5f50 2020-11-20T12:08:38.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.310+0000 7fa7397bd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7341489e0 con 0x7fa734068230 2020-11-20T12:08:38.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.311+0000 7fa733fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa7340b57b0 0x7fa734062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:34138/0 (socket says 172.21.15.187:34138) 2020-11-20T12:08:38.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.311+0000 7fa733fff700 1 -- 172.21.15.187:0/3908811995 learned_addr learned my addr 172.21.15.187:0/3908811995 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:38.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.311+0000 7fa72b7fe700 1 --2- 172.21.15.187:0/3908811995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa734068230 0x7fa7341482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.311+0000 7fa738fbc700 1 --2- 172.21.15.187:0/3908811995 >> v2:172.21.15.5:3300/0 conn(0x7fa7340b5f50 0x7fa7340b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.312+0000 7fa7337fe700 1 -- 172.21.15.187:0/3908811995 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651478069 0 0) 0x7fa734144b80 con 0x7fa7340b57b0 2020-11-20T12:08:38.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.312+0000 7fa7337fe700 1 -- 172.21.15.187:0/3908811995 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa714003240 con 0x7fa7340b57b0 2020-11-20T12:08:38.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.312+0000 7fa72b7fe700 1 -- 172.21.15.187:0/3908811995 >> v2:172.21.15.5:3300/0 conn(0x7fa7340b5f50 msgr2=0x7fa7340b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:38.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.312+0000 7fa72b7fe700 1 --2- 172.21.15.187:0/3908811995 >> v2:172.21.15.5:3300/0 conn(0x7fa7340b5f50 0x7fa7340b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.312+0000 7fa72b7fe700 1 -- 172.21.15.187:0/3908811995 >> v1:172.21.15.5:6789/0 conn(0x7fa7340b57b0 legacy=0x7fa734062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.312+0000 7fa72b7fe700 1 -- 172.21.15.187:0/3908811995 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734148c80 con 0x7fa734068230 2020-11-20T12:08:38.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.313+0000 7fa72b7fe700 1 --2- 172.21.15.187:0/3908811995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa734068230 0x7fa7341482d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa71c00b9a0 tx=0x7fa71c006df0).ready entity=mon.2 client_cookie=e62ec74a79847948 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:38.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.313+0000 7fa7337fe700 1 -- 172.21.15.187:0/3908811995 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c009680 con 0x7fa734068230 2020-11-20T12:08:38.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.313+0000 7fa7337fe700 1 -- 172.21.15.187:0/3908811995 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71c0097e0 con 0x7fa734068230 2020-11-20T12:08:38.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.313+0000 7fa7337fe700 1 -- 172.21.15.187:0/3908811995 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c009b60 con 0x7fa734068230 2020-11-20T12:08:38.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.314+0000 7fa7397bd700 1 -- 172.21.15.187:0/3908811995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa734068230 msgr2=0x7fa7341482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.314+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3908811995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa734068230 0x7fa7341482d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa71c00b9a0 tx=0x7fa71c006df0).stop 2020-11-20T12:08:38.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.314+0000 7fa7397bd700 1 -- 172.21.15.187:0/3908811995 shutdown_connections 2020-11-20T12:08:38.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.314+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3908811995 >> v2:172.21.15.5:3300/0 conn(0x7fa7340b5f50 0x7fa7340b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.314+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3908811995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa734068230 0x7fa7341482d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.314+0000 7fa7397bd700 1 -- 172.21.15.187:0/3908811995 >> 172.21.15.187:0/3908811995 conn(0x7fa7340bab30 msgr2=0x7fa7340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:38.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.314+0000 7fa7397bd700 1 -- 172.21.15.187:0/3908811995 shutdown_connections 2020-11-20T12:08:38.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.314+0000 7fa7397bd700 1 -- 172.21.15.187:0/3908811995 wait complete. 2020-11-20T12:08:38.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.316+0000 7fa7397bd700 1 Processor -- start 2020-11-20T12:08:38.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.316+0000 7fa7397bd700 1 -- start start 2020-11-20T12:08:38.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.317+0000 7fa7397bd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa734068230 0x7fa7340b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.317+0000 7fa7397bd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7340b57b0 0x7fa73409d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.317+0000 7fa7397bd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7340b5f50 0x7fa7340b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.317+0000 7fa7397bd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73409a7f0 con 0x7fa7340b57b0 2020-11-20T12:08:38.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.317+0000 7fa7397bd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa734099ce0 con 0x7fa734068230 2020-11-20T12:08:38.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.317+0000 7fa7397bd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa734099f80 con 0x7fa7340b5f50 2020-11-20T12:08:38.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa733fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa734068230 0x7fa7340b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa733fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa734068230 0x7fa7340b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44028/0 (socket says 172.21.15.187:44028) 2020-11-20T12:08:38.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa733fff700 1 -- 172.21.15.187:0/3944805133 learned_addr learned my addr 172.21.15.187:0/3944805133 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:38.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa738fbc700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7340b5f50 0x7fa7340b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa733fff700 1 -- 172.21.15.187:0/3944805133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7340b5f50 msgr2=0x7fa7340b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa72b7fe700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7340b57b0 0x7fa73409d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa733fff700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7340b5f50 0x7fa7340b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa733fff700 1 -- 172.21.15.187:0/3944805133 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7340b57b0 msgr2=0x7fa73409d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa733fff700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7340b57b0 0x7fa73409d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.318+0000 7fa733fff700 1 -- 172.21.15.187:0/3944805133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7340b48b0 con 0x7fa734068230 2020-11-20T12:08:38.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.319+0000 7fa733fff700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa734068230 0x7fa7340b43c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7fa72400c6f0 tx=0x7fa724007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:38.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.319+0000 7fa730ff9700 1 -- 172.21.15.187:0/3944805133 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724007320 con 0x7fa734068230 2020-11-20T12:08:38.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.319+0000 7fa730ff9700 1 -- 172.21.15.187:0/3944805133 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa724007480 con 0x7fa734068230 2020-11-20T12:08:38.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.319+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73409d610 con 0x7fa734068230 2020-11-20T12:08:38.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.319+0000 7fa730ff9700 1 -- 172.21.15.187:0/3944805133 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724005800 con 0x7fa734068230 2020-11-20T12:08:38.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.319+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa734144d70 con 0x7fa734068230 2020-11-20T12:08:38.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.320+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa734061a20 con 0x7fa734068230 2020-11-20T12:08:38.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.321+0000 7fa730ff9700 1 -- 172.21.15.187:0/3944805133 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa724010030 con 0x7fa734068230 2020-11-20T12:08:38.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.322+0000 7fa730ff9700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa710056d20 0x7fa7100591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.322+0000 7fa730ff9700 1 -- 172.21.15.187:0/3944805133 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa72406fce0 con 0x7fa734068230 2020-11-20T12:08:38.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.322+0000 7fa72b7fe700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa710056d20 0x7fa7100591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.323+0000 7fa72b7fe700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa710056d20 0x7fa7100591e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fa73409a330 tx=0x7fa71c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:38.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.325+0000 7fa730ff9700 1 -- 172.21.15.187:0/3944805133 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa7240450d0 con 0x7fa734068230 2020-11-20T12:08:38.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.518+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa734151570 con 0x7fa734068230 2020-11-20T12:08:38.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.519+0000 7fa730ff9700 1 -- 172.21.15.187:0/3944805133 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa7240486f0 con 0x7fa734068230 2020-11-20T12:08:38.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa710056d20 msgr2=0x7fa7100591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa710056d20 0x7fa7100591e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fa73409a330 tx=0x7fa71c015000).stop 2020-11-20T12:08:38.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa734068230 msgr2=0x7fa7340b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa734068230 0x7fa7340b43c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7fa72400c6f0 tx=0x7fa724007ad0).stop 2020-11-20T12:08:38.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 shutdown_connections 2020-11-20T12:08:38.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa710056d20 0x7fa7100591e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7340b5f50 0x7fa7340b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa734068230 0x7fa7340b43c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 --2- 172.21.15.187:0/3944805133 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7340b57b0 0x7fa73409d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.521+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 >> 172.21.15.187:0/3944805133 conn(0x7fa7340bab30 msgr2=0x7fa7340b95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:38.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.522+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 shutdown_connections 2020-11-20T12:08:38.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.522+0000 7fa7397bd700 1 -- 172.21.15.187:0/3944805133 wait complete. 2020-11-20T12:08:38.557 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:38.557 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:08:38.558 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:38.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.796+0000 7f741db3c700 1 Processor -- start 2020-11-20T12:08:38.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.796+0000 7f741db3c700 1 -- start start 2020-11-20T12:08:38.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.798+0000 7f741db3c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74180b57b0 0x7f7418068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.798+0000 7f741db3c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f74180b5f50 0x7f74180b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.798+0000 7f741db3c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7418144b80 con 0x7f74181482d0 2020-11-20T12:08:38.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.798+0000 7f741db3c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74181489e0 con 0x7f74180b5f50 2020-11-20T12:08:38.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.798+0000 7f741db3c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7418148c80 con 0x7f74180b57b0 2020-11-20T12:08:38.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.798+0000 7f741cb3a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74180b57b0 0x7f7418068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.798+0000 7f740ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f74181482d0 0x7f7418062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40646/0 (socket says 172.21.15.187:40646) 2020-11-20T12:08:38.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.798+0000 7f740ffff700 1 -- 172.21.15.187:0/3637788425 learned_addr learned my addr 172.21.15.187:0/3637788425 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:38.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.799+0000 7f741d33b700 1 --2- 172.21.15.187:0/3637788425 >> v2:172.21.15.5:3300/0 conn(0x7f74180b5f50 0x7f74180b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.799+0000 7f740f7fe700 1 -- 172.21.15.187:0/3637788425 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064293039 0 0) 0x7f7418144b80 con 0x7f74181482d0 2020-11-20T12:08:38.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.799+0000 7f741cb3a700 1 -- 172.21.15.187:0/3637788425 >> v2:172.21.15.5:3300/0 conn(0x7f74180b5f50 msgr2=0x7f74180b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:38.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.799+0000 7f741cb3a700 1 --2- 172.21.15.187:0/3637788425 >> v2:172.21.15.5:3300/0 conn(0x7f74180b5f50 0x7f74180b6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.799+0000 7f741cb3a700 1 -- 172.21.15.187:0/3637788425 >> v1:172.21.15.187:6789/0 conn(0x7f74181482d0 legacy=0x7f7418062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.799+0000 7f741cb3a700 1 -- 172.21.15.187:0/3637788425 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7418148f20 con 0x7f74180b57b0 2020-11-20T12:08:38.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.800+0000 7f741cb3a700 1 --2- 172.21.15.187:0/3637788425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b57b0 0x7f7418068230 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f74180b85e0 tx=0x7f7414006df0).ready entity=mon.1 client_cookie=5b58609da0fe2bf8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:38.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.800+0000 7f740f7fe700 1 -- 172.21.15.187:0/3637788425 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74140097b0 con 0x7f74180b57b0 2020-11-20T12:08:38.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.800+0000 7f740f7fe700 1 -- 172.21.15.187:0/3637788425 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414009910 con 0x7f74180b57b0 2020-11-20T12:08:38.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.800+0000 7f740f7fe700 1 -- 172.21.15.187:0/3637788425 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414009c90 con 0x7f74180b57b0 2020-11-20T12:08:38.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.801+0000 7f741db3c700 1 -- 172.21.15.187:0/3637788425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b57b0 msgr2=0x7f7418068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.801+0000 7f741db3c700 1 --2- 172.21.15.187:0/3637788425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b57b0 0x7f7418068230 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f74180b85e0 tx=0x7f7414006df0).stop 2020-11-20T12:08:38.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.801+0000 7f741db3c700 1 -- 172.21.15.187:0/3637788425 shutdown_connections 2020-11-20T12:08:38.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.801+0000 7f741db3c700 1 --2- 172.21.15.187:0/3637788425 >> v2:172.21.15.5:3300/0 conn(0x7f74180b5f50 0x7f74180b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.801+0000 7f741db3c700 1 --2- 172.21.15.187:0/3637788425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b57b0 0x7f7418068230 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:38.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.801+0000 7f741db3c700 1 -- 172.21.15.187:0/3637788425 >> 172.21.15.187:0/3637788425 conn(0x7f74180bab30 msgr2=0x7f74180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:38.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.801+0000 7f741db3c700 1 -- 172.21.15.187:0/3637788425 shutdown_connections 2020-11-20T12:08:38.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.801+0000 7f741db3c700 1 -- 172.21.15.187:0/3637788425 wait complete. 2020-11-20T12:08:38.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.802+0000 7f741db3c700 1 Processor -- start 2020-11-20T12:08:38.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.803+0000 7f741db3c700 1 -- start start 2020-11-20T12:08:38.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.803+0000 7f741db3c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b5f50 0x7f74180b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.803+0000 7f741db3c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74181482d0 0x7f74180b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.803+0000 7f741db3c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f741813a760 0x7f741809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.803+0000 7f741db3c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418139f10 con 0x7f741813a760 2020-11-20T12:08:38.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.803+0000 7f741db3c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418146df0 con 0x7f74180b5f50 2020-11-20T12:08:38.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.803+0000 7f741db3c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7418147090 con 0x7f74181482d0 2020-11-20T12:08:38.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b5f50 0x7f74180b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b5f50 0x7f74180b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44042/0 (socket says 172.21.15.187:44042) 2020-11-20T12:08:38.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 -- 172.21.15.187:0/3517210833 learned_addr learned my addr 172.21.15.187:0/3517210833 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:38.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 -- 172.21.15.187:0/3517210833 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74181482d0 msgr2=0x7f74180b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74181482d0 0x7f74180b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:38.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 -- 172.21.15.187:0/3517210833 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f741813a760 msgr2=0x7f741809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:38.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f741813a760 0x7f741809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:38.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 -- 172.21.15.187:0/3517210833 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7418147330 con 0x7f74180b5f50 2020-11-20T12:08:38.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.804+0000 7f741cb3a700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b5f50 0x7f74180b3200 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f7414006e30 tx=0x7f7414015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:38.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.805+0000 7f740cff9700 1 -- 172.21.15.187:0/3517210833 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414011e00 con 0x7f74180b5f50 2020-11-20T12:08:38.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.805+0000 7f740cff9700 1 -- 172.21.15.187:0/3517210833 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741401e460 con 0x7f74180b5f50 2020-11-20T12:08:38.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.805+0000 7f740cff9700 1 -- 172.21.15.187:0/3517210833 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741401e800 con 0x7f74180b5f50 2020-11-20T12:08:38.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.805+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7418147990 con 0x7f74180b5f50 2020-11-20T12:08:38.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.805+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74180b3f90 con 0x7f74180b5f50 2020-11-20T12:08:38.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.806+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74180b38b0 con 0x7f74180b5f50 2020-11-20T12:08:38.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.807+0000 7f740cff9700 1 -- 172.21.15.187:0/3517210833 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7414025030 con 0x7f74180b5f50 2020-11-20T12:08:38.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.807+0000 7f740cff9700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f73f0056d50 0x7f73f0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:38.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.807+0000 7f740cff9700 1 -- 172.21.15.187:0/3517210833 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f74140717a0 con 0x7f74180b5f50 2020-11-20T12:08:38.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.807+0000 7f740ffff700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f73f0056d50 0x7f73f0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:38.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.808+0000 7f740ffff700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f73f0056d50 0x7f73f0059210 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f741813a1e0 tx=0x7f74080078a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:38.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:38.811+0000 7f740cff9700 1 -- 172.21.15.187:0/3517210833 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7414046ca0 con 0x7f74180b5f50 2020-11-20T12:08:39.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.057+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74181474c0 con 0x7f74180b5f50 2020-11-20T12:08:39.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.058+0000 7f740cff9700 1 -- 172.21.15.187:0/3517210833 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f7414023070 con 0x7f74180b5f50 2020-11-20T12:08:39.062 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:39.062 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5019,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/4036313482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4036313482},{"type":"v1","addr":"172.21.15.5:6833","nonce":4036313482}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:08:30.177391+0000","modified":"2020-11-20T12:08:32.308400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5743},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5743":{"gid":5743,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1756209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1756209233},{"type":"v1","addr":"172.21.15.187:6835","nonce":1756209233}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2020-11-20T12:08:39.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f73f0056d50 msgr2=0x7f73f0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:39.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f73f0056d50 0x7f73f0059210 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f741813a1e0 tx=0x7f74080078a0).stop 2020-11-20T12:08:39.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b5f50 msgr2=0x7f74180b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:39.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b5f50 0x7f74180b3200 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f7414006e30 tx=0x7f7414015000).stop 2020-11-20T12:08:39.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 shutdown_connections 2020-11-20T12:08:39.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f73f0056d50 0x7f73f0059210 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:39.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f741813a760 0x7f741809ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:39.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74180b5f50 0x7f74180b3200 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:39.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 --2- 172.21.15.187:0/3517210833 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74181482d0 0x7f74180b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:39.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 >> 172.21.15.187:0/3517210833 conn(0x7f74180bab30 msgr2=0x7f7418065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:39.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 shutdown_connections 2020-11-20T12:08:39.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:39.061+0000 7f741db3c700 1 -- 172.21.15.187:0/3517210833 wait complete. 2020-11-20T12:08:39.069 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 26 2020-11-20T12:08:39.288 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:08:30.177391+0000', 'modified': '2020-11-20T12:08:32.308400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5743}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5743': {'gid': 5743, 'name': 'a', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1756209233', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1756209233}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1756209233}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:08:39.288 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:08:39.289 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:08:39.560 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":5743,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":9.3529459700000004,"mdsmap_epoch":26,"osdmap_epoch":51,"osdmap_epoch_barrier":51,"uptime":14.997648720999999} 2020-11-20T12:08:39.560 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5743, "mdsmap_epoch": 26, "osdmap_epoch": 51, "osdmap_epoch_barrier": 51, "rank_uptime": 9.35294597, "state": "up:active", "uptime": 14.997648721, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:08:39.561 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:08:39.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:39.562 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:08:39.619 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:08:39.620 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:08:39.820 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:08:39.820 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:08:39.857 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:08:39.857 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:08:39.864 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:08:39.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:39.865 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:08:39.971 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:08:39.972 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo touch fancy_layout_file) 2020-11-20T12:08:40.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:08:40.091 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2020-11-20T12:08:40.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:08:40.246 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2020-11-20T12:08:40.900 INFO:teuthology.orchestra.run.smithi187.stderr:35+0 records in 2020-11-20T12:08:40.901 INFO:teuthology.orchestra.run.smithi187.stderr:35+0 records out 2020-11-20T12:08:40.901 INFO:teuthology.orchestra.run.smithi187.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.54843 s, 66.9 MB/s 2020-11-20T12:08:40.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:08:40.905 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -f fancy_layout_file) 2020-11-20T12:08:40.990 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:41.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.218+0000 7fa7c2df5700 1 Processor -- start 2020-11-20T12:08:41.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.218+0000 7fa7c2df5700 1 -- start start 2020-11-20T12:08:41.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.220+0000 7fa7c2df5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa7bc0b57b0 0x7fa7bc068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:41.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.220+0000 7fa7c2df5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa7bc0b5f50 0x7fa7bc0b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:41.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.220+0000 7fa7c2df5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc144b80 con 0x7fa7bc1482d0 2020-11-20T12:08:41.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.220+0000 7fa7c2df5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc1489e0 con 0x7fa7bc0b57b0 2020-11-20T12:08:41.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.220+0000 7fa7c2df5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc148c80 con 0x7fa7bc0b5f50 2020-11-20T12:08:41.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.220+0000 7fa7c25f4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa7bc0b5f50 0x7fa7bc0b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:41.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.220+0000 7fa7c25f4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa7bc0b5f50 0x7fa7bc0b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44064/0 (socket says 172.21.15.187:44064) 2020-11-20T12:08:41.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.220+0000 7fa7c25f4700 1 -- 172.21.15.187:0/1596708890 learned_addr learned my addr 172.21.15.187:0/1596708890 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:41.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c1df3700 1 --2- 172.21.15.187:0/1596708890 >> v2:172.21.15.5:3300/0 conn(0x7fa7bc0b57b0 0x7fa7bc068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:41.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c25f4700 1 -- 172.21.15.187:0/1596708890 >> v2:172.21.15.5:3300/0 conn(0x7fa7bc0b57b0 msgr2=0x7fa7bc068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:41.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c25f4700 1 --2- 172.21.15.187:0/1596708890 >> v2:172.21.15.5:3300/0 conn(0x7fa7bc0b57b0 0x7fa7bc068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c25f4700 1 -- 172.21.15.187:0/1596708890 >> v1:172.21.15.5:6789/0 conn(0x7fa7bc1482d0 legacy=0x7fa7bc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:41.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c25f4700 1 -- 172.21.15.187:0/1596708890 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc148f20 con 0x7fa7bc0b5f50 2020-11-20T12:08:41.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c1df3700 1 --2- 172.21.15.187:0/1596708890 >> v2:172.21.15.5:3300/0 conn(0x7fa7bc0b57b0 0x7fa7bc068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:41.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c25f4700 1 --2- 172.21.15.187:0/1596708890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc0b5f50 0x7fa7bc0b6380 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fa7bc066230 tx=0x7fa7b4009290).ready entity=mon.1 client_cookie=91f473ae47de1d33 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:41.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c0df1700 1 -- 172.21.15.187:0/1596708890 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400bb80 con 0x7fa7bc0b5f50 2020-11-20T12:08:41.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c0df1700 1 -- 172.21.15.187:0/1596708890 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b400bce0 con 0x7fa7bc0b5f50 2020-11-20T12:08:41.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.221+0000 7fa7c0df1700 1 -- 172.21.15.187:0/1596708890 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4011610 con 0x7fa7bc0b5f50 2020-11-20T12:08:41.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.222+0000 7fa7c2df5700 1 -- 172.21.15.187:0/1596708890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc0b5f50 msgr2=0x7fa7bc0b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:41.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.222+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/1596708890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc0b5f50 0x7fa7bc0b6380 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fa7bc066230 tx=0x7fa7b4009290).stop 2020-11-20T12:08:41.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.222+0000 7fa7c2df5700 1 -- 172.21.15.187:0/1596708890 shutdown_connections 2020-11-20T12:08:41.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.222+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/1596708890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc0b5f50 0x7fa7bc0b6380 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.222+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/1596708890 >> v2:172.21.15.5:3300/0 conn(0x7fa7bc0b57b0 0x7fa7bc068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.222+0000 7fa7c2df5700 1 -- 172.21.15.187:0/1596708890 >> 172.21.15.187:0/1596708890 conn(0x7fa7bc0bab30 msgr2=0x7fa7bc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:41.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.222+0000 7fa7c2df5700 1 -- 172.21.15.187:0/1596708890 shutdown_connections 2020-11-20T12:08:41.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.222+0000 7fa7c2df5700 1 -- 172.21.15.187:0/1596708890 wait complete. 2020-11-20T12:08:41.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.224+0000 7fa7c2df5700 1 Processor -- start 2020-11-20T12:08:41.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.224+0000 7fa7c2df5700 1 -- start start 2020-11-20T12:08:41.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c2df5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7bc0b57b0 0x7fa7bc140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:41.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c2df5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7bc0b5f50 0x7fa7bc144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:41.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c2df5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc1482d0 0x7fa7bc13c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:41.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c2df5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b3080 con 0x7fa7bc0b57b0 2020-11-20T12:08:41.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c2df5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b2de0 con 0x7fa7bc1482d0 2020-11-20T12:08:41.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c2df5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b3360 con 0x7fa7bc0b5f50 2020-11-20T12:08:41.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c25f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc1482d0 0x7fa7bc13c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:41.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c25f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc1482d0 0x7fa7bc13c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44070/0 (socket says 172.21.15.187:44070) 2020-11-20T12:08:41.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.226+0000 7fa7c25f4700 1 -- 172.21.15.187:0/4275396458 learned_addr learned my addr 172.21.15.187:0/4275396458 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:41.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c1df3700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7bc0b57b0 0x7fa7bc140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:41.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c15f2700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7bc0b5f50 0x7fa7bc144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:41.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c25f4700 1 -- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7bc0b5f50 msgr2=0x7fa7bc144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:41.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c25f4700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7bc0b5f50 0x7fa7bc144f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c25f4700 1 -- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7bc0b57b0 msgr2=0x7fa7bc140fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:41.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c25f4700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7bc0b57b0 0x7fa7bc140fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c25f4700 1 -- 172.21.15.187:0/4275396458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc145420 con 0x7fa7bc1482d0 2020-11-20T12:08:41.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c1df3700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7bc0b57b0 0x7fa7bc140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:41.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c15f2700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7bc0b5f50 0x7fa7bc144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:41.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7c25f4700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc1482d0 0x7fa7bc13c090 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7fa7b40090f0 tx=0x7fa7b4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:41.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.227+0000 7fa7a9ffb700 1 -- 172.21.15.187:0/4275396458 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4011e70 con 0x7fa7bc1482d0 2020-11-20T12:08:41.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.228+0000 7fa7a9ffb700 1 -- 172.21.15.187:0/4275396458 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b400be50 con 0x7fa7bc1482d0 2020-11-20T12:08:41.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.228+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc145b90 con 0x7fa7bc1482d0 2020-11-20T12:08:41.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.228+0000 7fa7a9ffb700 1 -- 172.21.15.187:0/4275396458 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b401c470 con 0x7fa7bc1482d0 2020-11-20T12:08:41.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.228+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc0b2ac0 con 0x7fa7bc1482d0 2020-11-20T12:08:41.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.229+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7bc0b3520 con 0x7fa7bc1482d0 2020-11-20T12:08:41.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.229+0000 7fa7a9ffb700 1 -- 172.21.15.187:0/4275396458 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa7b4017070 con 0x7fa7bc1482d0 2020-11-20T12:08:41.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.230+0000 7fa7a9ffb700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa794056d20 0x7fa7940591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:41.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.230+0000 7fa7a9ffb700 1 -- 172.21.15.187:0/4275396458 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa7b40721a0 con 0x7fa7bc1482d0 2020-11-20T12:08:41.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.230+0000 7fa7c1df3700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa794056d20 0x7fa7940591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:41.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.231+0000 7fa7c1df3700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa794056d20 0x7fa7940591e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa7b800b7d0 tx=0x7fa7b8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:41.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.234+0000 7fa7a9ffb700 1 -- 172.21.15.187:0/4275396458 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa7b40475f0 con 0x7fa7bc1482d0 2020-11-20T12:08:41.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.466+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7bc0b4c00 con 0x7fa7bc1482d0 2020-11-20T12:08:41.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.467+0000 7fa7a9ffb700 1 -- 172.21.15.187:0/4275396458 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2135 (secure 0 0 0) 0x7fa7b4015070 con 0x7fa7bc1482d0 2020-11-20T12:08:41.471 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:41.472 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5019,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/4036313482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4036313482},{"type":"v1","addr":"172.21.15.5:6833","nonce":4036313482}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:08:30.177391+0000","modified":"2020-11-20T12:08:32.308400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5743},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5743":{"gid":5743,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1756209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1756209233},{"type":"v1","addr":"172.21.15.187:6835","nonce":1756209233}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2020-11-20T12:08:41.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.469+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa794056d20 msgr2=0x7fa7940591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:41.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.469+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa794056d20 0x7fa7940591e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa7b800b7d0 tx=0x7fa7b8008040).stop 2020-11-20T12:08:41.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.469+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc1482d0 msgr2=0x7fa7bc13c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:41.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.469+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc1482d0 0x7fa7bc13c090 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7fa7b40090f0 tx=0x7fa7b4016000).stop 2020-11-20T12:08:41.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.470+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 shutdown_connections 2020-11-20T12:08:41.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.470+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa794056d20 0x7fa7940591e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.470+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7bc0b57b0 0x7fa7bc140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.470+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc1482d0 0x7fa7bc13c090 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.470+0000 7fa7c2df5700 1 --2- 172.21.15.187:0/4275396458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa7bc0b5f50 0x7fa7bc144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:41.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.470+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 >> 172.21.15.187:0/4275396458 conn(0x7fa7bc0bab30 msgr2=0x7fa7bc065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:41.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.470+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 shutdown_connections 2020-11-20T12:08:41.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:41.470+0000 7fa7c2df5700 1 -- 172.21.15.187:0/4275396458 wait complete. 2020-11-20T12:08:41.477 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 26 2020-11-20T12:08:41.493 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:08:41.786 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:08:41.787 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:08:41.788 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2020-11-20T12:08:42.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.036+0000 7fa345f5c700 1 Processor -- start 2020-11-20T12:08:42.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.036+0000 7fa345f5c700 1 -- start start 2020-11-20T12:08:42.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.037+0000 7fa345f5c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa340061f90 0x7fa3400b1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:42.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.037+0000 7fa345f5c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa340067040 con 0x7fa3400b0c50 2020-11-20T12:08:42.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.037+0000 7fa345f5c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa340067270 con 0x7fa340062370 2020-11-20T12:08:42.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.037+0000 7fa345f5c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3400b6c80 con 0x7fa340061f90 2020-11-20T12:08:42.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.038+0000 7fa337fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa340062370 0x7fa340148130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40684/0 (socket says 172.21.15.187:40684) 2020-11-20T12:08:42.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.038+0000 7fa337fff700 1 -- 172.21.15.187:0/1739468201 learned_addr learned my addr 172.21.15.187:0/1739468201 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:42.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.038+0000 7fa344f5a700 1 --2- 172.21.15.187:0/1739468201 >> v2:172.21.15.5:3300/0 conn(0x7fa340061f90 0x7fa3400b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:42.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.038+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705158103 0 0) 0x7fa340067270 con 0x7fa340062370 2020-11-20T12:08:42.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.038+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa32c003240 con 0x7fa340062370 2020-11-20T12:08:42.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3626957258 0 0) 0x7fa32c003240 con 0x7fa340062370 2020-11-20T12:08:42.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 >> v2:172.21.15.5:3300/0 conn(0x7fa340061f90 msgr2=0x7fa3400b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:42.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 --2- 172.21.15.187:0/1739468201 >> v2:172.21.15.5:3300/0 conn(0x7fa340061f90 0x7fa3400b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 >> v1:172.21.15.5:6789/0 conn(0x7fa3400b0c50 legacy=0x7fa3400b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:42.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa340062dd0 con 0x7fa340062370 2020-11-20T12:08:42.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fa3300014e0 con 0x7fa340062370 2020-11-20T12:08:42.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 >> v1:172.21.15.187:6789/0 conn(0x7fa340062370 legacy=0x7fa340148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:42.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa32c003800 0x7fa32c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:42.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c0044e0 0x7fa32c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:42.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa32c004e50 0x7fa32c00d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:42.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400b6c80 con 0x7fa32c004e50 2020-11-20T12:08:42.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c005300 con 0x7fa32c003800 2020-11-20T12:08:42.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c00de30 con 0x7fa32c0044e0 2020-11-20T12:08:42.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.039+0000 7fa344f5a700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa32c003800 0x7fa32c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:42.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa337fff700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c0044e0 0x7fa32c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:42.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa344f5a700 1 -- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c0044e0 msgr2=0x7fa32c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:42.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa34575b700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa32c004e50 0x7fa32c00d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:42.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa344f5a700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c0044e0 0x7fa32c004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa344f5a700 1 -- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa32c004e50 msgr2=0x7fa32c00d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:42.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa344f5a700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa32c004e50 0x7fa32c00d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa344f5a700 1 -- 172.21.15.187:0/1739468201 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c003240 con 0x7fa32c003800 2020-11-20T12:08:42.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa34575b700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa32c004e50 0x7fa32c00d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:42.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa34575b700 1 -- 172.21.15.187:0/1739468201 reap_dead start 2020-11-20T12:08:42.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.040+0000 7fa344f5a700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa32c003800 0x7fa32c003c20 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fa33c00bbc0 tx=0x7fa33c008040).ready entity=mon.1 client_cookie=35a7339c11c0036 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:42.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c00b2c0 con 0x7fa32c003800 2020-11-20T12:08:42.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c00eac0 con 0x7fa32c003800 2020-11-20T12:08:42.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa3377fe700 1 -- 172.21.15.187:0/1739468201 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c00ee40 con 0x7fa32c003800 2020-11-20T12:08:42.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa345f5c700 1 -- 172.21.15.187:0/1739468201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa32c003800 msgr2=0x7fa32c003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:42.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa345f5c700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa32c003800 0x7fa32c003c20 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fa33c00bbc0 tx=0x7fa33c008040).stop 2020-11-20T12:08:42.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa345f5c700 1 -- 172.21.15.187:0/1739468201 shutdown_connections 2020-11-20T12:08:42.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa345f5c700 1 --2- 172.21.15.187:0/1739468201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa32c003800 0x7fa32c003c20 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa345f5c700 1 -- 172.21.15.187:0/1739468201 >> 172.21.15.187:0/1739468201 conn(0x7fa3400bab10 msgr2=0x7fa3400baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:42.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa345f5c700 1 -- 172.21.15.187:0/1739468201 shutdown_connections 2020-11-20T12:08:42.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.041+0000 7fa345f5c700 1 -- 172.21.15.187:0/1739468201 wait complete. 2020-11-20T12:08:42.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.043+0000 7fa345f5c700 1 Processor -- start 2020-11-20T12:08:42.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.043+0000 7fa345f5c700 1 -- start start 2020-11-20T12:08:42.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.044+0000 7fa345f5c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c003800 0x7fa340142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:42.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.044+0000 7fa345f5c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3400b3720 0x7fa3401433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:42.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.044+0000 7fa345f5c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340157740 0x7fa340157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:42.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.044+0000 7fa345f5c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34009d3d0 con 0x7fa3400b3720 2020-11-20T12:08:42.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.044+0000 7fa345f5c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3401438d0 con 0x7fa340157740 2020-11-20T12:08:42.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.044+0000 7fa345f5c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400b3ca0 con 0x7fa32c003800 2020-11-20T12:08:42.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa34575b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340157740 0x7fa340157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:42.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa34575b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340157740 0x7fa340157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44090/0 (socket says 172.21.15.187:44090) 2020-11-20T12:08:42.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa34575b700 1 -- 172.21.15.187:0/1951131916 learned_addr learned my addr 172.21.15.187:0/1951131916 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:42.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa344f5a700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c003800 0x7fa340142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:42.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa337fff700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3400b3720 0x7fa3401433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:42.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa34575b700 1 -- 172.21.15.187:0/1951131916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c003800 msgr2=0x7fa340142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:42.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa34575b700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c003800 0x7fa340142ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa34575b700 1 -- 172.21.15.187:0/1951131916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3400b3720 msgr2=0x7fa3401433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:42.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa34575b700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3400b3720 0x7fa3401433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.045+0000 7fa34575b700 1 -- 172.21.15.187:0/1951131916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400b3f40 con 0x7fa340157740 2020-11-20T12:08:42.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.046+0000 7fa34575b700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340157740 0x7fa340157ba0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7fa340062a70 tx=0x7fa338007cd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:42.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.046+0000 7fa334ff9700 1 -- 172.21.15.187:0/1951131916 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338011870 con 0x7fa340157740 2020-11-20T12:08:42.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.046+0000 7fa334ff9700 1 -- 172.21.15.187:0/1951131916 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3380119d0 con 0x7fa340157740 2020-11-20T12:08:42.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.046+0000 7fa334ff9700 1 -- 172.21.15.187:0/1951131916 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338011d70 con 0x7fa340157740 2020-11-20T12:08:42.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.046+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3400b41e0 con 0x7fa340157740 2020-11-20T12:08:42.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.046+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34009bef0 con 0x7fa340157740 2020-11-20T12:08:42.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.047+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3400b4370 con 0x7fa340157740 2020-11-20T12:08:42.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.048+0000 7fa334ff9700 1 -- 172.21.15.187:0/1951131916 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa338021070 con 0x7fa340157740 2020-11-20T12:08:42.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.048+0000 7fa334ff9700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa318056d20 0x7fa3180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:42.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.048+0000 7fa334ff9700 1 -- 172.21.15.187:0/1951131916 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa338073840 con 0x7fa340157740 2020-11-20T12:08:42.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.049+0000 7fa344f5a700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa318056d20 0x7fa3180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:42.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.049+0000 7fa344f5a700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa318056d20 0x7fa3180591e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fa33c006fd0 tx=0x7fa33c019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:42.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.053+0000 7fa334ff9700 1 -- 172.21.15.187:0/1951131916 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa338048c10 con 0x7fa340157740 2020-11-20T12:08:42.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.306+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fa340061a20 con 0x7fa340157740 2020-11-20T12:08:42.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.307+0000 7fa334ff9700 1 -- 172.21.15.187:0/1951131916 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1935 (secure 0 0 0) 0x7fa33801f070 con 0x7fa340157740 2020-11-20T12:08:42.310 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:42.311 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:08:42.311 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:08:42.311 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:08:42.311 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 773018685440, 2020-11-20T12:08:42.312 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 75427840, 2020-11-20T12:08:42.312 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 75427840, 2020-11-20T12:08:42.312 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 9.7566182375885546e-05, 2020-11-20T12:08:42.312 INFO:teuthology.orchestra.run.smithi187.stdout: "num_osds": 8, 2020-11-20T12:08:42.312 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_osds": 8, 2020-11-20T12:08:42.313 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_omap_osds": 8 2020-11-20T12:08:42.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:42.313 INFO:teuthology.orchestra.run.smithi187.stdout: "stats_by_class": { 2020-11-20T12:08:42.313 INFO:teuthology.orchestra.run.smithi187.stdout: "ssd": { 2020-11-20T12:08:42.314 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:08:42.314 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 773018685440, 2020-11-20T12:08:42.314 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 75427840, 2020-11-20T12:08:42.314 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 75427840, 2020-11-20T12:08:42.314 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 9.7566182375885546e-05 2020-11-20T12:08:42.315 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:42.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:42.315 INFO:teuthology.orchestra.run.smithi187.stdout: "pools": [ 2020-11-20T12:08:42.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:42.315 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "device_health_metrics", 2020-11-20T12:08:42.316 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 1, 2020-11-20T12:08:42.316 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:08:42.316 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:08:42.316 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 2, 2020-11-20T12:08:42.317 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:08:42.317 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:08:42.317 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:08:42.317 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367170486272 2020-11-20T12:08:42.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:42.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:42.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:42.318 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "rbd", 2020-11-20T12:08:42.318 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 2, 2020-11-20T12:08:42.319 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:08:42.319 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:08:42.319 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:08:42.319 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:08:42.320 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:08:42.320 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:08:42.320 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367170486272 2020-11-20T12:08:42.320 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:42.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:42.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:42.321 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_metadata", 2020-11-20T12:08:42.321 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 9, 2020-11-20T12:08:42.321 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:08:42.321 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 2304, 2020-11-20T12:08:42.322 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 22, 2020-11-20T12:08:42.322 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 64, 2020-11-20T12:08:42.322 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 65536, 2020-11-20T12:08:42.322 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 8.9244636569674185e-08, 2020-11-20T12:08:42.322 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367170486272 2020-11-20T12:08:42.323 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:42.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:08:42.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:08:42.323 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_data", 2020-11-20T12:08:42.324 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 10, 2020-11-20T12:08:42.324 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:08:42.324 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:08:42.324 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:08:42.324 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:08:42.325 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:08:42.325 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:08:42.325 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367170486272 2020-11-20T12:08:42.325 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:42.325 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:08:42.326 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:08:42.326 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:08:42.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.309+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa318056d20 msgr2=0x7fa3180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:42.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.309+0000 7fa345f5c700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa318056d20 0x7fa3180591e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fa33c006fd0 tx=0x7fa33c019000).stop 2020-11-20T12:08:42.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.309+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340157740 msgr2=0x7fa340157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:42.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.309+0000 7fa345f5c700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340157740 0x7fa340157ba0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7fa340062a70 tx=0x7fa338007cd0).stop 2020-11-20T12:08:42.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.310+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 shutdown_connections 2020-11-20T12:08:42.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.310+0000 7fa345f5c700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa318056d20 0x7fa3180591e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.310+0000 7fa345f5c700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340157740 0x7fa340157ba0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.310+0000 7fa345f5c700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa32c003800 0x7fa340142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.310+0000 7fa345f5c700 1 --2- 172.21.15.187:0/1951131916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3400b3720 0x7fa3401433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:42.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.310+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 >> 172.21.15.187:0/1951131916 conn(0x7fa3400bab10 msgr2=0x7fa340147420 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:42.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.310+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 shutdown_connections 2020-11-20T12:08:42.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:42.310+0000 7fa345f5c700 1 -- 172.21.15.187:0/1951131916 wait complete. 2020-11-20T12:08:42.337 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2020-11-20T12:08:42.338 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:08:42.338 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:08:42.456 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:08:42.470 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:08:42.579 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:08:42.600 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:08:42.600 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:08:42.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:08:42.601 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:08:42.768 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:08:42.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:08:42.769 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:08:42.826 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:08:43.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.067+0000 7f1cbb50f700 1 Processor -- start 2020-11-20T12:08:43.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.067+0000 7f1cbb50f700 1 -- start start 2020-11-20T12:08:43.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.068+0000 7f1cbb50f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1cb4065f90 0x7f1cb4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.068+0000 7f1cbb50f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1cb409ca90 0x7f1cb409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.068+0000 7f1cbb50f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb40b13e0 con 0x7f1cb4065190 2020-11-20T12:08:43.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.068+0000 7f1cbb50f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb414ab30 con 0x7f1cb4065f90 2020-11-20T12:08:43.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.068+0000 7f1cbb50f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb4062c10 con 0x7f1cb409ca90 2020-11-20T12:08:43.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.069+0000 7f1cbad0e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1cb409ca90 0x7f1cb409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.069+0000 7f1cbad0e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1cb409ca90 0x7f1cb409cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44100/0 (socket says 172.21.15.187:44100) 2020-11-20T12:08:43.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.069+0000 7f1cbad0e700 1 -- 172.21.15.187:0/3000759345 learned_addr learned my addr 172.21.15.187:0/3000759345 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:43.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.069+0000 7f1cb9d0c700 1 --2- 172.21.15.187:0/3000759345 >> v2:172.21.15.5:3300/0 conn(0x7f1cb4065f90 0x7f1cb4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cbad0e700 1 -- 172.21.15.187:0/3000759345 >> v2:172.21.15.5:3300/0 conn(0x7f1cb4065f90 msgr2=0x7f1cb4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:43.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cbad0e700 1 --2- 172.21.15.187:0/3000759345 >> v2:172.21.15.5:3300/0 conn(0x7f1cb4065f90 0x7f1cb4065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cbad0e700 1 -- 172.21.15.187:0/3000759345 >> v1:172.21.15.5:6789/0 conn(0x7f1cb4065190 legacy=0x7f1cb4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:43.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cbad0e700 1 -- 172.21.15.187:0/3000759345 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb40b1640 con 0x7f1cb409ca90 2020-11-20T12:08:43.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cb9d0c700 1 --2- 172.21.15.187:0/3000759345 >> v2:172.21.15.5:3300/0 conn(0x7f1cb4065f90 0x7f1cb4065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:08:43.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cbad0e700 1 --2- 172.21.15.187:0/3000759345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb409cf20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f1cb4099f90 tx=0x7f1cac009290).ready entity=mon.1 client_cookie=6dca26a9ae39a25b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:43.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cb950b700 1 -- 172.21.15.187:0/3000759345 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac00bc50 con 0x7f1cb409ca90 2020-11-20T12:08:43.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cb950b700 1 -- 172.21.15.187:0/3000759345 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac00bdb0 con 0x7f1cb409ca90 2020-11-20T12:08:43.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.070+0000 7f1cb950b700 1 -- 172.21.15.187:0/3000759345 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac011610 con 0x7f1cb409ca90 2020-11-20T12:08:43.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.071+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3000759345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 msgr2=0x7f1cb409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:43.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.071+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3000759345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb409cf20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f1cb4099f90 tx=0x7f1cac009290).stop 2020-11-20T12:08:43.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.071+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3000759345 shutdown_connections 2020-11-20T12:08:43.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.071+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3000759345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb409cf20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.071+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3000759345 >> v2:172.21.15.5:3300/0 conn(0x7f1cb4065f90 0x7f1cb4065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.071+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3000759345 >> 172.21.15.187:0/3000759345 conn(0x7f1cb40bab30 msgr2=0x7f1cb40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:43.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.071+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3000759345 shutdown_connections 2020-11-20T12:08:43.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.071+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3000759345 wait complete. 2020-11-20T12:08:43.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.073+0000 7f1cbb50f700 1 Processor -- start 2020-11-20T12:08:43.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.073+0000 7f1cbb50f700 1 -- start start 2020-11-20T12:08:43.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbb50f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cb4065190 0x7f1cb409dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbb50f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cb4065f90 0x7f1cb4067720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbb50f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb4143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbb50f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb409f590 con 0x7f1cb4065f90 2020-11-20T12:08:43.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbb50f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb409dfd0 con 0x7f1cb409ca90 2020-11-20T12:08:43.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbb50f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb409e260 con 0x7f1cb4065190 2020-11-20T12:08:43.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbad0e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb4143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbad0e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb4143090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44106/0 (socket says 172.21.15.187:44106) 2020-11-20T12:08:43.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.074+0000 7f1cbad0e700 1 -- 172.21.15.187:0/3047804614 learned_addr learned my addr 172.21.15.187:0/3047804614 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:43.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cb9d0c700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cb4065f90 0x7f1cb4067720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cba50d700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cb4065190 0x7f1cb409dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cbad0e700 1 -- 172.21.15.187:0/3047804614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cb4065190 msgr2=0x7f1cb409dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:43.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cbad0e700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cb4065190 0x7f1cb409dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cbad0e700 1 -- 172.21.15.187:0/3047804614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cb4065f90 msgr2=0x7f1cb4067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:43.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cbad0e700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cb4065f90 0x7f1cb4067720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cbad0e700 1 -- 172.21.15.187:0/3047804614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb409e500 con 0x7f1cb409ca90 2020-11-20T12:08:43.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cbad0e700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb4143090 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f1cac0106d0 tx=0x7f1cac016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:43.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1ca67fc700 1 -- 172.21.15.187:0/3047804614 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac011e70 con 0x7f1cb409ca90 2020-11-20T12:08:43.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1ca67fc700 1 -- 172.21.15.187:0/3047804614 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac010b40 con 0x7f1cb409ca90 2020-11-20T12:08:43.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1ca67fc700 1 -- 172.21.15.187:0/3047804614 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac01c470 con 0x7f1cb409ca90 2020-11-20T12:08:43.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.075+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb409e7a0 con 0x7f1cb409ca90 2020-11-20T12:08:43.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.076+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb409ecf0 con 0x7f1cb409ca90 2020-11-20T12:08:43.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.077+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb409efd0 con 0x7f1cb409ca90 2020-11-20T12:08:43.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.077+0000 7f1ca67fc700 1 -- 172.21.15.187:0/3047804614 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1cac023030 con 0x7f1cb409ca90 2020-11-20T12:08:43.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.078+0000 7f1ca67fc700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c8c056d20 0x7f1c8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.078+0000 7f1ca67fc700 1 -- 172.21.15.187:0/3047804614 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1cac071850 con 0x7f1cb409ca90 2020-11-20T12:08:43.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.078+0000 7f1cba50d700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c8c056d20 0x7f1c8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.079+0000 7f1cba50d700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c8c056d20 0x7f1c8c0591e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f1ca8001410 tx=0x7f1ca8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:43.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.082+0000 7f1ca67fc700 1 -- 172.21.15.187:0/3047804614 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1cac046be0 con 0x7f1cb409ca90 2020-11-20T12:08:43.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.277+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1cb409e930 con 0x7f1cb409ca90 2020-11-20T12:08:43.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.288+0000 7f1ca67fc700 1 -- 172.21.15.187:0/3047804614 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1cac04a200 con 0x7f1cb409ca90 2020-11-20T12:08:43.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.290+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c8c056d20 msgr2=0x7f1c8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:43.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.290+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c8c056d20 0x7f1c8c0591e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f1ca8001410 tx=0x7f1ca8009040).stop 2020-11-20T12:08:43.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.290+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 msgr2=0x7f1cb4143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:43.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.290+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb4143090 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f1cac0106d0 tx=0x7f1cac016000).stop 2020-11-20T12:08:43.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.291+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 shutdown_connections 2020-11-20T12:08:43.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.291+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c8c056d20 0x7f1c8c0591e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.291+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb409ca90 0x7f1cb4143090 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.291+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cb4065190 0x7f1cb409dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.291+0000 7f1cbb50f700 1 --2- 172.21.15.187:0/3047804614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cb4065f90 0x7f1cb4067720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.291+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 >> 172.21.15.187:0/3047804614 conn(0x7f1cb40bab30 msgr2=0x7f1cb40b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:43.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.291+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 shutdown_connections 2020-11-20T12:08:43.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.291+0000 7f1cbb50f700 1 -- 172.21.15.187:0/3047804614 wait complete. 2020-11-20T12:08:43.318 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2020-11-20T12:08:43.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.558+0000 7f59d3254700 1 Processor -- start 2020-11-20T12:08:43.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.558+0000 7f59d3254700 1 -- start start 2020-11-20T12:08:43.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.560+0000 7f59d3254700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59cc0652e0 0x7f59cc0656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.560+0000 7f59d3254700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b1590 0x7f59cc14a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.560+0000 7f59d3254700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc149dd0 con 0x7f59cc0b7300 2020-11-20T12:08:43.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.560+0000 7f59d3254700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59cc14a070 con 0x7f59cc0652e0 2020-11-20T12:08:43.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.560+0000 7f59d3254700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc14a310 con 0x7f59cc0b1590 2020-11-20T12:08:43.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.560+0000 7f59d0ff0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59cc0652e0 0x7f59cc0656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.560+0000 7f59d0ff0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59cc0652e0 0x7f59cc0656c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44114/0 (socket says 172.21.15.187:44114) 2020-11-20T12:08:43.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.560+0000 7f59d0ff0700 1 -- 172.21.15.187:0/3448038679 learned_addr learned my addr 172.21.15.187:0/3448038679 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:43.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.561+0000 7f59cbfff700 1 --2- 172.21.15.187:0/3448038679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b1590 0x7f59cc14a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.561+0000 7f59d0ff0700 1 -- 172.21.15.187:0/3448038679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b1590 msgr2=0x7f59cc14a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:43.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.561+0000 7f59d0ff0700 1 --2- 172.21.15.187:0/3448038679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b1590 0x7f59cc14a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.561+0000 7f59d0ff0700 1 -- 172.21.15.187:0/3448038679 >> v1:172.21.15.5:6789/0 conn(0x7f59cc0b7300 legacy=0x7f59cc143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:43.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.561+0000 7f59d0ff0700 1 -- 172.21.15.187:0/3448038679 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc14aca0 con 0x7f59cc0652e0 2020-11-20T12:08:43.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.561+0000 7f59cbfff700 1 --2- 172.21.15.187:0/3448038679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b1590 0x7f59cc14a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:43.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.561+0000 7f59d0ff0700 1 --2- 172.21.15.187:0/3448038679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0652e0 0x7f59cc0656c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f59c0006fd0 tx=0x7f59c0006eb0).ready entity=mon.1 client_cookie=ea7eb2459ddcdb42 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:43.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59cb7fe700 1 -- 172.21.15.187:0/3448038679 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0009890 con 0x7f59cc0652e0 2020-11-20T12:08:43.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59cb7fe700 1 -- 172.21.15.187:0/3448038679 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c00099f0 con 0x7f59cc0652e0 2020-11-20T12:08:43.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59cb7fe700 1 -- 172.21.15.187:0/3448038679 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0009d70 con 0x7f59cc0652e0 2020-11-20T12:08:43.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59d3254700 1 -- 172.21.15.187:0/3448038679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0652e0 msgr2=0x7f59cc0656c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:43.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59d3254700 1 --2- 172.21.15.187:0/3448038679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0652e0 0x7f59cc0656c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f59c0006fd0 tx=0x7f59c0006eb0).stop 2020-11-20T12:08:43.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59d3254700 1 -- 172.21.15.187:0/3448038679 shutdown_connections 2020-11-20T12:08:43.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59d3254700 1 --2- 172.21.15.187:0/3448038679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0652e0 0x7f59cc0656c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59d3254700 1 --2- 172.21.15.187:0/3448038679 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b1590 0x7f59cc14a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59d3254700 1 -- 172.21.15.187:0/3448038679 >> 172.21.15.187:0/3448038679 conn(0x7f59cc0bacd0 msgr2=0x7f59cc0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:43.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.562+0000 7f59d3254700 1 -- 172.21.15.187:0/3448038679 shutdown_connections 2020-11-20T12:08:43.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.563+0000 7f59d3254700 1 -- 172.21.15.187:0/3448038679 wait complete. 2020-11-20T12:08:43.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.564+0000 7f59d3254700 1 Processor -- start 2020-11-20T12:08:43.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.564+0000 7f59d3254700 1 -- start start 2020-11-20T12:08:43.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.565+0000 7f59d3254700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59cc0652e0 0x7f59cc147030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.565+0000 7f59d3254700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0b1590 0x7f59cc1370f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.565+0000 7f59d3254700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b7300 0x7f59cc148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.565+0000 7f59d3254700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc149920 con 0x7f59cc0652e0 2020-11-20T12:08:43.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.565+0000 7f59d3254700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc1491e0 con 0x7f59cc0b1590 2020-11-20T12:08:43.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.565+0000 7f59d3254700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc149480 con 0x7f59cc0b7300 2020-11-20T12:08:43.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.565+0000 7f59cbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0b1590 0x7f59cc1370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59cbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0b1590 0x7f59cc1370f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44122/0 (socket says 172.21.15.187:44122) 2020-11-20T12:08:43.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59cbfff700 1 -- 172.21.15.187:0/3804835683 learned_addr learned my addr 172.21.15.187:0/3804835683 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:43.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59cbfff700 1 -- 172.21.15.187:0/3804835683 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b7300 msgr2=0x7f59cc148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:43.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59d0ff0700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59cc0652e0 0x7f59cc147030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59d17f1700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b7300 0x7f59cc148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59cbfff700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b7300 0x7f59cc148750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59cbfff700 1 -- 172.21.15.187:0/3804835683 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59cc0652e0 msgr2=0x7f59cc147030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:43.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59cbfff700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59cc0652e0 0x7f59cc147030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:43.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59cbfff700 1 -- 172.21.15.187:0/3804835683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc1462f0 con 0x7f59cc0b1590 2020-11-20T12:08:43.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.566+0000 7f59cbfff700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0b1590 0x7f59cc1370f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f59b800cb80 tx=0x7f59b8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:43.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.567+0000 7f59c8ff9700 1 -- 172.21.15.187:0/3804835683 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8015070 con 0x7f59cc0b1590 2020-11-20T12:08:43.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.567+0000 7f59c8ff9700 1 -- 172.21.15.187:0/3804835683 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8006020 con 0x7f59cc0b1590 2020-11-20T12:08:43.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.567+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc1375e0 con 0x7f59cc0b1590 2020-11-20T12:08:43.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.567+0000 7f59c8ff9700 1 -- 172.21.15.187:0/3804835683 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b80063a0 con 0x7f59cc0b1590 2020-11-20T12:08:43.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.567+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc156150 con 0x7f59cc0b1590 2020-11-20T12:08:43.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.568+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc1563a0 con 0x7f59cc0b1590 2020-11-20T12:08:43.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.568+0000 7f59c8ff9700 1 -- 172.21.15.187:0/3804835683 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f59b8006500 con 0x7f59cc0b1590 2020-11-20T12:08:43.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.569+0000 7f59c8ff9700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59a4056d20 0x7f59a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:43.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.569+0000 7f59c8ff9700 1 -- 172.21.15.187:0/3804835683 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f59b806fe00 con 0x7f59cc0b1590 2020-11-20T12:08:43.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.569+0000 7f59d0ff0700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59a4056d20 0x7f59a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:43.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.570+0000 7f59d0ff0700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59a4056d20 0x7f59a40591e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f59c0006ef0 tx=0x7f59c0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:43.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.573+0000 7f59c8ff9700 1 -- 172.21.15.187:0/3804835683 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f59cc1563a0 con 0x7f59cc0b1590 2020-11-20T12:08:43.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:43.828+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f59cc1563a0 con 0x7f59cc0b1590 2020-11-20T12:08:44.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.557+0000 7f59c8ff9700 1 -- 172.21.15.187:0/3804835683 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v206) v1 ==== 132+0+0 (secure 0 0 0) 0x7f59cc1563a0 con 0x7f59cc0b1590 2020-11-20T12:08:44.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59a4056d20 msgr2=0x7f59a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:44.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59a4056d20 0x7f59a40591e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f59c0006ef0 tx=0x7f59c0015000).stop 2020-11-20T12:08:44.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0b1590 msgr2=0x7f59cc1370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:44.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0b1590 0x7f59cc1370f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f59b800cb80 tx=0x7f59b8008040).stop 2020-11-20T12:08:44.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 shutdown_connections 2020-11-20T12:08:44.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59a4056d20 0x7f59a40591e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59cc0652e0 0x7f59cc147030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59cc0b7300 0x7f59cc148750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 --2- 172.21.15.187:0/3804835683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59cc0b1590 0x7f59cc1370f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 >> 172.21.15.187:0/3804835683 conn(0x7f59cc0bacd0 msgr2=0x7f59cc1433a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:44.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.561+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 shutdown_connections 2020-11-20T12:08:44.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.562+0000 7f59d3254700 1 -- 172.21.15.187:0/3804835683 wait complete. 2020-11-20T12:08:44.595 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:08:44.596 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:08:44.597 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2020-11-20T12:08:44.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.838+0000 7f194098e700 1 Processor -- start 2020-11-20T12:08:44.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.838+0000 7f194098e700 1 -- start start 2020-11-20T12:08:44.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.840+0000 7f194098e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f193c0b1380 0x7f193c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:44.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.840+0000 7f194098e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f193c14c260 0x7f193c14c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:44.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.840+0000 7f194098e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193c0658a0 con 0x7f193c0b1760 2020-11-20T12:08:44.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.840+0000 7f194098e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c150390 con 0x7f193c14c260 2020-11-20T12:08:44.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.840+0000 7f194098e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c150630 con 0x7f193c0b1380 2020-11-20T12:08:44.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.840+0000 7f193b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f193c0b1380 0x7f193c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:44.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.840+0000 7f193b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f193c0b1380 0x7f193c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44128/0 (socket says 172.21.15.187:44128) 2020-11-20T12:08:44.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.840+0000 7f193b7fe700 1 -- 172.21.15.187:0/149354423 learned_addr learned my addr 172.21.15.187:0/149354423 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:44.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.841+0000 7f193bfff700 1 --2- 172.21.15.187:0/149354423 >> v2:172.21.15.5:3300/0 conn(0x7f193c14c260 0x7f193c14c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:44.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.841+0000 7f193b7fe700 1 -- 172.21.15.187:0/149354423 >> v2:172.21.15.5:3300/0 conn(0x7f193c14c260 msgr2=0x7f193c14c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:44.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.841+0000 7f193b7fe700 1 --2- 172.21.15.187:0/149354423 >> v2:172.21.15.5:3300/0 conn(0x7f193c14c260 0x7f193c14c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.841+0000 7f193b7fe700 1 -- 172.21.15.187:0/149354423 >> v1:172.21.15.5:6789/0 conn(0x7f193c0b1760 legacy=0x7f193c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:44.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.841+0000 7f193b7fe700 1 -- 172.21.15.187:0/149354423 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c068510 con 0x7f193c0b1380 2020-11-20T12:08:44.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.841+0000 7f193bfff700 1 --2- 172.21.15.187:0/149354423 >> v2:172.21.15.5:3300/0 conn(0x7f193c14c260 0x7f193c14c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:44.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.841+0000 7f193b7fe700 1 --2- 172.21.15.187:0/149354423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c150ad0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f193c0b85e0 tx=0x7f192c006eb0).ready entity=mon.1 client_cookie=c7bef22a73211d2a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:44.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f193affd700 1 -- 172.21.15.187:0/149354423 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c009890 con 0x7f193c0b1380 2020-11-20T12:08:44.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f193affd700 1 -- 172.21.15.187:0/149354423 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c0099f0 con 0x7f193c0b1380 2020-11-20T12:08:44.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f193affd700 1 -- 172.21.15.187:0/149354423 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c009d70 con 0x7f193c0b1380 2020-11-20T12:08:44.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f194098e700 1 -- 172.21.15.187:0/149354423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 msgr2=0x7f193c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:44.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f194098e700 1 --2- 172.21.15.187:0/149354423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c150ad0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f193c0b85e0 tx=0x7f192c006eb0).stop 2020-11-20T12:08:44.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f194098e700 1 -- 172.21.15.187:0/149354423 shutdown_connections 2020-11-20T12:08:44.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f194098e700 1 --2- 172.21.15.187:0/149354423 >> v2:172.21.15.5:3300/0 conn(0x7f193c14c260 0x7f193c14c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f194098e700 1 --2- 172.21.15.187:0/149354423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c150ad0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.842+0000 7f194098e700 1 -- 172.21.15.187:0/149354423 >> 172.21.15.187:0/149354423 conn(0x7f193c0bab30 msgr2=0x7f193c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:44.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.843+0000 7f194098e700 1 -- 172.21.15.187:0/149354423 shutdown_connections 2020-11-20T12:08:44.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.843+0000 7f194098e700 1 -- 172.21.15.187:0/149354423 wait complete. 2020-11-20T12:08:44.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.844+0000 7f194098e700 1 Processor -- start 2020-11-20T12:08:44.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.845+0000 7f194098e700 1 -- start start 2020-11-20T12:08:44.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f194098e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c13e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:44.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f194098e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f193c0b1760 0x7f193c0b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:44.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f194098e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f193c14c260 0x7f193c13d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:44.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f194098e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c0b7e30 con 0x7f193c0b1760 2020-11-20T12:08:44.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f194098e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c13e2f0 con 0x7f193c0b1380 2020-11-20T12:08:44.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f194098e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c13ec80 con 0x7f193c14c260 2020-11-20T12:08:44.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f193b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c13e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:44.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f193b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c13e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44134/0 (socket says 172.21.15.187:44134) 2020-11-20T12:08:44.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f193b7fe700 1 -- 172.21.15.187:0/2864085565 learned_addr learned my addr 172.21.15.187:0/2864085565 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:44.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f193b7fe700 1 -- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f193c14c260 msgr2=0x7f193c13d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:44.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f193bfff700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f193c14c260 0x7f193c13d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:44.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f1933fff700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f193c0b1760 0x7f193c0b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:44.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f193b7fe700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f193c14c260 0x7f193c13d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.846+0000 7f193b7fe700 1 -- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f193c0b1760 msgr2=0x7f193c0b9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:44.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f193b7fe700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f193c0b1760 0x7f193c0b9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:44.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f193b7fe700 1 -- 172.21.15.187:0/2864085565 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c0ba170 con 0x7f193c0b1380 2020-11-20T12:08:44.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f1933fff700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f193c0b1760 0x7f193c0b9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:44.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f193b7fe700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c13e790 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f193c150770 tx=0x7f192c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:44.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f19337fe700 1 -- 172.21.15.187:0/2864085565 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c011e50 con 0x7f193c0b1380 2020-11-20T12:08:44.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f19337fe700 1 -- 172.21.15.187:0/2864085565 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c01e460 con 0x7f193c0b1380 2020-11-20T12:08:44.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f19337fe700 1 -- 172.21.15.187:0/2864085565 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c01e7e0 con 0x7f193c0b1380 2020-11-20T12:08:44.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c09ed30 con 0x7f193c0b1380 2020-11-20T12:08:44.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.847+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c09f240 con 0x7f193c0b1380 2020-11-20T12:08:44.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.849+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c061a20 con 0x7f193c0b1380 2020-11-20T12:08:44.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.849+0000 7f19337fe700 1 -- 172.21.15.187:0/2864085565 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f192c025030 con 0x7f193c0b1380 2020-11-20T12:08:44.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.850+0000 7f19337fe700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1918056d20 0x7f19180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:44.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.850+0000 7f19337fe700 1 -- 172.21.15.187:0/2864085565 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f192c071850 con 0x7f193c0b1380 2020-11-20T12:08:44.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.850+0000 7f1933fff700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1918056d20 0x7f19180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:44.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.850+0000 7f1933fff700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1918056d20 0x7f19180591e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f193c0b6c40 tx=0x7f1924009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:44.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:44.854+0000 7f19337fe700 1 -- 172.21.15.187:0/2864085565 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f192c046d50 con 0x7f193c0b1380 2020-11-20T12:08:45.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.110+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f193c0446d0 con 0x7f193c0b1380 2020-11-20T12:08:45.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.563+0000 7f19337fe700 1 -- 172.21.15.187:0/2864085565 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v207) v1 ==== 137+0+0 (secure 0 0 0) 0x7f192c023070 con 0x7f193c0b1380 2020-11-20T12:08:45.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1918056d20 msgr2=0x7f19180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:45.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1918056d20 0x7f19180591e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f193c0b6c40 tx=0x7f1924009040).stop 2020-11-20T12:08:45.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 msgr2=0x7f193c13e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:45.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c13e790 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f193c150770 tx=0x7f192c015000).stop 2020-11-20T12:08:45.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 shutdown_connections 2020-11-20T12:08:45.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1918056d20 0x7f19180591e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f193c14c260 0x7f193c13d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f193c0b1380 0x7f193c13e790 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 --2- 172.21.15.187:0/2864085565 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f193c0b1760 0x7f193c0b9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 >> 172.21.15.187:0/2864085565 conn(0x7f193c0bab30 msgr2=0x7f193c14c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:45.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 shutdown_connections 2020-11-20T12:08:45.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.566+0000 7f194098e700 1 -- 172.21.15.187:0/2864085565 wait complete. 2020-11-20T12:08:45.595 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:08:45.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.883+0000 7f286b59e700 1 Processor -- start 2020-11-20T12:08:45.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.884+0000 7f286b59e700 1 -- start start 2020-11-20T12:08:45.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.885+0000 7f286b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065f90 0x7f286c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.885+0000 7f286b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f286c09ca90 0x7f286c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.885+0000 7f286b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c0b13e0 con 0x7f286c065190 2020-11-20T12:08:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.885+0000 7f286b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c14ab30 con 0x7f286c09ca90 2020-11-20T12:08:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.885+0000 7f286b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c062c10 con 0x7f286c065f90 2020-11-20T12:08:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.886+0000 7f286a59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f286c065190 0x7f286c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40752/0 (socket says 172.21.15.187:40752) 2020-11-20T12:08:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.886+0000 7f286a59c700 1 -- 172.21.15.187:0/131073349 learned_addr learned my addr 172.21.15.187:0/131073349 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.886+0000 7f286ad9d700 1 --2- 172.21.15.187:0/131073349 >> v2:172.21.15.5:3300/0 conn(0x7f286c09ca90 0x7f286c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.886+0000 7f2861d9b700 1 --2- 172.21.15.187:0/131073349 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065f90 0x7f286c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:45.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.887+0000 7f2869d9b700 1 -- 172.21.15.187:0/131073349 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676923452 0 0) 0x7f286c0b13e0 con 0x7f286c065190 2020-11-20T12:08:45.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.887+0000 7f2869d9b700 1 -- 172.21.15.187:0/131073349 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284c003240 con 0x7f286c065190 2020-11-20T12:08:45.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.887+0000 7f2861d9b700 1 -- 172.21.15.187:0/131073349 >> v2:172.21.15.5:3300/0 conn(0x7f286c09ca90 msgr2=0x7f286c09cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:45.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.887+0000 7f2861d9b700 1 --2- 172.21.15.187:0/131073349 >> v2:172.21.15.5:3300/0 conn(0x7f286c09ca90 0x7f286c09cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.887+0000 7f2861d9b700 1 -- 172.21.15.187:0/131073349 >> v1:172.21.15.187:6789/0 conn(0x7f286c065190 legacy=0x7f286c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:45.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.887+0000 7f2861d9b700 1 -- 172.21.15.187:0/131073349 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c0b1640 con 0x7f286c065f90 2020-11-20T12:08:45.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.888+0000 7f2861d9b700 1 --2- 172.21.15.187:0/131073349 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065f90 0x7f286c065570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f286c14ac70 tx=0x7f2854006df0).ready entity=mon.2 client_cookie=30bc54f875ec6e9f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:45.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.888+0000 7f2869d9b700 1 -- 172.21.15.187:0/131073349 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28540096d0 con 0x7f286c065f90 2020-11-20T12:08:45.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f2869d9b700 1 -- 172.21.15.187:0/131073349 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2854009830 con 0x7f286c065f90 2020-11-20T12:08:45.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f2869d9b700 1 -- 172.21.15.187:0/131073349 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854009bb0 con 0x7f286c065f90 2020-11-20T12:08:45.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f286b59e700 1 -- 172.21.15.187:0/131073349 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065f90 msgr2=0x7f286c065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:45.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f286b59e700 1 --2- 172.21.15.187:0/131073349 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065f90 0x7f286c065570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f286c14ac70 tx=0x7f2854006df0).stop 2020-11-20T12:08:45.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f286b59e700 1 -- 172.21.15.187:0/131073349 shutdown_connections 2020-11-20T12:08:45.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f286b59e700 1 --2- 172.21.15.187:0/131073349 >> v2:172.21.15.5:3300/0 conn(0x7f286c09ca90 0x7f286c09cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f286b59e700 1 --2- 172.21.15.187:0/131073349 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065f90 0x7f286c065570 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f286b59e700 1 -- 172.21.15.187:0/131073349 >> 172.21.15.187:0/131073349 conn(0x7f286c0bab30 msgr2=0x7f286c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:45.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f286b59e700 1 -- 172.21.15.187:0/131073349 shutdown_connections 2020-11-20T12:08:45.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.889+0000 7f286b59e700 1 -- 172.21.15.187:0/131073349 wait complete. 2020-11-20T12:08:45.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.891+0000 7f286b59e700 1 Processor -- start 2020-11-20T12:08:45.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.891+0000 7f286b59e700 1 -- start start 2020-11-20T12:08:45.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f286b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065190 0x7f286c0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:45.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f286b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f286c065f90 0x7f286c09eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:45.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f286b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f286c09ca90 0x7f286c142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:45.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f286b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0b2de0 con 0x7f286c09ca90 2020-11-20T12:08:45.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f286b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c159050 con 0x7f286c065f90 2020-11-20T12:08:45.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f286b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c067710 con 0x7f286c065190 2020-11-20T12:08:45.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f2861d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f286c065f90 0x7f286c09eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:45.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f2861d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f286c065f90 0x7f286c09eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44150/0 (socket says 172.21.15.187:44150) 2020-11-20T12:08:45.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.892+0000 7f2861d9b700 1 -- 172.21.15.187:0/3416896442 learned_addr learned my addr 172.21.15.187:0/3416896442 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:45.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f286a59c700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065190 0x7f286c0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:45.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f286ad9d700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f286c09ca90 0x7f286c142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:45.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f2861d9b700 1 -- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065190 msgr2=0x7f286c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:45.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f2861d9b700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065190 0x7f286c0b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f2861d9b700 1 -- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f286c09ca90 msgr2=0x7f286c142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:45.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f2861d9b700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f286c09ca90 0x7f286c142d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:45.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f2861d9b700 1 -- 172.21.15.187:0/3416896442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c158d80 con 0x7f286c065f90 2020-11-20T12:08:45.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f286ad9d700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f286c09ca90 0x7f286c142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:45.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f2861d9b700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f286c065f90 0x7f286c09eb00 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f286c063390 tx=0x7f2854015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:45.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.893+0000 7f2862ffd700 1 -- 172.21.15.187:0/3416896442 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285401d5d0 con 0x7f286c065f90 2020-11-20T12:08:45.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.894+0000 7f2862ffd700 1 -- 172.21.15.187:0/3416896442 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285401d730 con 0x7f286c065f90 2020-11-20T12:08:45.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.894+0000 7f2862ffd700 1 -- 172.21.15.187:0/3416896442 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285401dad0 con 0x7f286c065f90 2020-11-20T12:08:45.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.894+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c0b47c0 con 0x7f286c065f90 2020-11-20T12:08:45.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.894+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c09f230 con 0x7f286c065f90 2020-11-20T12:08:45.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.895+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c061a20 con 0x7f286c065f90 2020-11-20T12:08:45.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.895+0000 7f2862ffd700 1 -- 172.21.15.187:0/3416896442 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2854025030 con 0x7f286c065f90 2020-11-20T12:08:45.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.896+0000 7f2862ffd700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2848056d20 0x7f28480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:45.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.896+0000 7f2862ffd700 1 -- 172.21.15.187:0/3416896442 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f28540719c0 con 0x7f286c065f90 2020-11-20T12:08:45.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.896+0000 7f286a59c700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2848056d20 0x7f28480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:45.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.897+0000 7f286a59c700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2848056d20 0x7f28480591e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f285c0055a0 tx=0x7f285c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:45.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:45.900+0000 7f2862ffd700 1 -- 172.21.15.187:0/3416896442 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2854046d90 con 0x7f286c065f90 2020-11-20T12:08:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.088+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f286c0afc60 con 0x7f286c065f90 2020-11-20T12:08:46.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.098+0000 7f2862ffd700 1 -- 172.21.15.187:0/3416896442 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2854023070 con 0x7f286c065f90 2020-11-20T12:08:46.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.100+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2848056d20 msgr2=0x7f28480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2848056d20 0x7f28480591e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f285c0055a0 tx=0x7f285c009040).stop 2020-11-20T12:08:46.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f286c065f90 msgr2=0x7f286c09eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f286c065f90 0x7f286c09eb00 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f286c063390 tx=0x7f2854015000).stop 2020-11-20T12:08:46.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 shutdown_connections 2020-11-20T12:08:46.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2848056d20 0x7f28480591e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f286c09ca90 0x7f286c142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f286c065190 0x7f286c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 --2- 172.21.15.187:0/3416896442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f286c065f90 0x7f286c09eb00 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 >> 172.21.15.187:0/3416896442 conn(0x7f286c0bab30 msgr2=0x7f286c09aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:46.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 shutdown_connections 2020-11-20T12:08:46.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.101+0000 7f286b59e700 1 -- 172.21.15.187:0/3416896442 wait complete. 2020-11-20T12:08:46.131 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:08:46.133 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:08:46.278 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:08:46.302 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:08:46.413 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:08:46.430 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:08:46.430 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:08:46.431 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:46.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.664+0000 7fae4924b700 1 Processor -- start 2020-11-20T12:08:46.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.664+0000 7fae4924b700 1 -- start start 2020-11-20T12:08:46.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.665+0000 7fae4924b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fae44068230 0x7fae441482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:46.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.665+0000 7fae4924b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae44144b80 con 0x7fae440b57b0 2020-11-20T12:08:46.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.665+0000 7fae4924b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae441489e0 con 0x7fae440b5f50 2020-11-20T12:08:46.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.665+0000 7fae4924b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae441487c0 con 0x7fae44068230 2020-11-20T12:08:46.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.666+0000 7fae48a4a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fae440b5f50 0x7fae440b6380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40766/0 (socket says 172.21.15.187:40766) 2020-11-20T12:08:46.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.666+0000 7fae48a4a700 1 -- 172.21.15.187:0/434644684 learned_addr learned my addr 172.21.15.187:0/434644684 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:46.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.666+0000 7fae3b7fe700 1 --2- 172.21.15.187:0/434644684 >> v2:172.21.15.5:3300/0 conn(0x7fae44068230 0x7fae441482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:46.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.666+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371472646 0 0) 0x7fae441489e0 con 0x7fae440b5f50 2020-11-20T12:08:46.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.666+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae24003240 con 0x7fae440b5f50 2020-11-20T12:08:46.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1116630207 0 0) 0x7fae24003240 con 0x7fae440b5f50 2020-11-20T12:08:46.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 >> v2:172.21.15.5:3300/0 conn(0x7fae44068230 msgr2=0x7fae441482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:46.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 --2- 172.21.15.187:0/434644684 >> v2:172.21.15.5:3300/0 conn(0x7fae44068230 0x7fae441482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 >> v1:172.21.15.5:6789/0 conn(0x7fae440b57b0 legacy=0x7fae44062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae44148c80 con 0x7fae440b5f50 2020-11-20T12:08:46.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fae3c001860 con 0x7fae440b5f50 2020-11-20T12:08:46.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 >> v1:172.21.15.187:6789/0 conn(0x7fae440b5f50 legacy=0x7fae440b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae24003730 0x7fae24003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:46.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae240044e0 0x7fae24004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:46.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae24004e50 0x7fae2400d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:46.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae441487c0 con 0x7fae24003730 2020-11-20T12:08:46.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae24004260 con 0x7fae24004e50 2020-11-20T12:08:46.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2400de30 con 0x7fae240044e0 2020-11-20T12:08:46.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.667+0000 7fae48a4a700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae24004e50 0x7fae2400d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:46.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae48a4a700 1 -- 172.21.15.187:0/434644684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae240044e0 msgr2=0x7fae24004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:46.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae3b7fe700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae240044e0 0x7fae24004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:46.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae43fff700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae24003730 0x7fae24003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:46.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae48a4a700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae240044e0 0x7fae24004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae48a4a700 1 -- 172.21.15.187:0/434644684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae24003730 msgr2=0x7fae24003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:46.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae48a4a700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae24003730 0x7fae24003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae48a4a700 1 -- 172.21.15.187:0/434644684 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae24003240 con 0x7fae24004e50 2020-11-20T12:08:46.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae48a4a700 1 -- 172.21.15.187:0/434644684 reap_dead start 2020-11-20T12:08:46.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.668+0000 7fae48a4a700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae24004e50 0x7fae2400d940 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fae3c00c840 tx=0x7fae44066490).ready entity=mon.1 client_cookie=f58060b4a89c1fa2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:46.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c007b50 con 0x7fae24004e50 2020-11-20T12:08:46.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c005b80 con 0x7fae24004e50 2020-11-20T12:08:46.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae437fe700 1 -- 172.21.15.187:0/434644684 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c007110 con 0x7fae24004e50 2020-11-20T12:08:46.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae4924b700 1 -- 172.21.15.187:0/434644684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae24004e50 msgr2=0x7fae2400d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae4924b700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae24004e50 0x7fae2400d940 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fae3c00c840 tx=0x7fae44066490).stop 2020-11-20T12:08:46.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae4924b700 1 -- 172.21.15.187:0/434644684 shutdown_connections 2020-11-20T12:08:46.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae4924b700 1 --2- 172.21.15.187:0/434644684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae24004e50 0x7fae2400d940 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae4924b700 1 -- 172.21.15.187:0/434644684 >> 172.21.15.187:0/434644684 conn(0x7fae440bab30 msgr2=0x7fae440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:46.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae4924b700 1 -- 172.21.15.187:0/434644684 shutdown_connections 2020-11-20T12:08:46.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.669+0000 7fae4924b700 1 -- 172.21.15.187:0/434644684 wait complete. 2020-11-20T12:08:46.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.671+0000 7fae4924b700 1 Processor -- start 2020-11-20T12:08:46.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.671+0000 7fae4924b700 1 -- start start 2020-11-20T12:08:46.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae4924b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae24004e50 0x7fae4409c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:46.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae4924b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae44157040 0x7fae4409a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:46.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae4924b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae440a0050 0x7fae440a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae4924b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4409d720 con 0x7fae440a0050 2020-11-20T12:08:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae4924b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae440a09c0 con 0x7fae44157040 2020-11-20T12:08:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae4924b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4409c7e0 con 0x7fae24004e50 2020-11-20T12:08:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae3b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae44157040 0x7fae4409a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae3b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae44157040 0x7fae4409a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44170/0 (socket says 172.21.15.187:44170) 2020-11-20T12:08:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae3b7fe700 1 -- 172.21.15.187:0/1904236212 learned_addr learned my addr 172.21.15.187:0/1904236212 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.672+0000 7fae43fff700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae24004e50 0x7fae4409c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae3b7fe700 1 -- 172.21.15.187:0/1904236212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae24004e50 msgr2=0x7fae4409c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae3b7fe700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae24004e50 0x7fae4409c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae3b7fe700 1 -- 172.21.15.187:0/1904236212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae440a0050 msgr2=0x7fae440a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae3b7fe700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae440a0050 0x7fae440a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae3b7fe700 1 -- 172.21.15.187:0/1904236212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4409ca80 con 0x7fae44157040 2020-11-20T12:08:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae3b7fe700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae44157040 0x7fae4409a810 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fae2c00f830 tx=0x7fae2c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae40ff9700 1 -- 172.21.15.187:0/1904236212 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c017070 con 0x7fae44157040 2020-11-20T12:08:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae40ff9700 1 -- 172.21.15.187:0/1904236212 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c011660 con 0x7fae44157040 2020-11-20T12:08:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4409d1c0 con 0x7fae44157040 2020-11-20T12:08:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae40ff9700 1 -- 172.21.15.187:0/1904236212 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0119e0 con 0x7fae44157040 2020-11-20T12:08:46.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.673+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae44153a70 con 0x7fae44157040 2020-11-20T12:08:46.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.675+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4409c630 con 0x7fae44157040 2020-11-20T12:08:46.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.675+0000 7fae40ff9700 1 -- 172.21.15.187:0/1904236212 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fae2c01c410 con 0x7fae44157040 2020-11-20T12:08:46.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.675+0000 7fae40ff9700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae20056d20 0x7fae200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:46.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.676+0000 7fae40ff9700 1 -- 172.21.15.187:0/1904236212 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fae2c06ff60 con 0x7fae44157040 2020-11-20T12:08:46.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.676+0000 7fae43fff700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae20056d20 0x7fae200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:46.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.676+0000 7fae43fff700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae20056d20 0x7fae200591e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fae34000bc0 tx=0x7fae34009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:46.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.679+0000 7fae40ff9700 1 -- 172.21.15.187:0/1904236212 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fae2c045240 con 0x7fae44157040 2020-11-20T12:08:46.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.922+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae44061a20 con 0x7fae44157040 2020-11-20T12:08:46.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.923+0000 7fae40ff9700 1 -- 172.21.15.187:0/1904236212 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2135 (secure 0 0 0) 0x7fae2c048860 con 0x7fae44157040 2020-11-20T12:08:46.927 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:46.927 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5019,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/4036313482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4036313482},{"type":"v1","addr":"172.21.15.5:6833","nonce":4036313482}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:08:30.177391+0000","modified":"2020-11-20T12:08:32.308400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5743},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5743":{"gid":5743,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1756209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1756209233},{"type":"v1","addr":"172.21.15.187:6835","nonce":1756209233}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2020-11-20T12:08:46.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae20056d20 msgr2=0x7fae200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae20056d20 0x7fae200591e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fae34000bc0 tx=0x7fae34009040).stop 2020-11-20T12:08:46.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae44157040 msgr2=0x7fae4409a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae44157040 0x7fae4409a810 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fae2c00f830 tx=0x7fae2c008040).stop 2020-11-20T12:08:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 shutdown_connections 2020-11-20T12:08:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fae20056d20 0x7fae200591e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae440a0050 0x7fae440a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:46.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae24004e50 0x7fae4409c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 --2- 172.21.15.187:0/1904236212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae44157040 0x7fae4409a810 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:46.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 >> 172.21.15.187:0/1904236212 conn(0x7fae440bab30 msgr2=0x7fae440b93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:46.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 shutdown_connections 2020-11-20T12:08:46.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:46.927+0000 7fae4924b700 1 -- 172.21.15.187:0/1904236212 wait complete. 2020-11-20T12:08:46.933 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 26 2020-11-20T12:08:46.952 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:47.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.193+0000 7fe7fb59e700 1 Processor -- start 2020-11-20T12:08:47.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.193+0000 7fe7fb59e700 1 -- start start 2020-11-20T12:08:47.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.194+0000 7fe7fb59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe7fc0b1380 0x7fe7fc150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.194+0000 7fe7fb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc14c260 0x7fe7fc14c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.194+0000 7fe7fb59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc0658a0 con 0x7fe7fc0b1760 2020-11-20T12:08:47.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.195+0000 7fe7fb59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc150390 con 0x7fe7fc0b1380 2020-11-20T12:08:47.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.195+0000 7fe7fb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc150630 con 0x7fe7fc14c260 2020-11-20T12:08:47.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.195+0000 7fe7fad9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc14c260 0x7fe7fc14c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.195+0000 7fe7f1d9b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe7fc0b1760 0x7fe7fc065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:34294/0 (socket says 172.21.15.187:34294) 2020-11-20T12:08:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.196+0000 7fe7f1d9b700 1 -- 172.21.15.187:0/578784708 learned_addr learned my addr 172.21.15.187:0/578784708 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.196+0000 7fe7fa59c700 1 --2- 172.21.15.187:0/578784708 >> v2:172.21.15.5:3300/0 conn(0x7fe7fc0b1380 0x7fe7fc150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.197+0000 7fe7f9d9b700 1 -- 172.21.15.187:0/578784708 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807853399 0 0) 0x7fe7fc0658a0 con 0x7fe7fc0b1760 2020-11-20T12:08:47.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.197+0000 7fe7f9d9b700 1 -- 172.21.15.187:0/578784708 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7dc003240 con 0x7fe7fc0b1760 2020-11-20T12:08:47.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.197+0000 7fe7fa59c700 1 -- 172.21.15.187:0/578784708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc14c260 msgr2=0x7fe7fc14c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:47.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.197+0000 7fe7fa59c700 1 --2- 172.21.15.187:0/578784708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc14c260 0x7fe7fc14c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.197+0000 7fe7fa59c700 1 -- 172.21.15.187:0/578784708 >> v1:172.21.15.5:6789/0 conn(0x7fe7fc0b1760 legacy=0x7fe7fc065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.197+0000 7fe7fa59c700 1 -- 172.21.15.187:0/578784708 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc068510 con 0x7fe7fc0b1380 2020-11-20T12:08:47.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.197+0000 7fe7fa59c700 1 --2- 172.21.15.187:0/578784708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 0x7fe7fc150ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fe7ec00b9a0 tx=0x7fe7ec006df0).ready entity=mon.0 client_cookie=49f2ab698ddddb72 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:47.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.198+0000 7fe7f9d9b700 1 -- 172.21.15.187:0/578784708 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec009680 con 0x7fe7fc0b1380 2020-11-20T12:08:47.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.198+0000 7fe7f9d9b700 1 -- 172.21.15.187:0/578784708 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec0097e0 con 0x7fe7fc0b1380 2020-11-20T12:08:47.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.198+0000 7fe7f9d9b700 1 -- 172.21.15.187:0/578784708 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec009b60 con 0x7fe7fc0b1380 2020-11-20T12:08:47.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.199+0000 7fe7fb59e700 1 -- 172.21.15.187:0/578784708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 msgr2=0x7fe7fc150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.199+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/578784708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 0x7fe7fc150ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fe7ec00b9a0 tx=0x7fe7ec006df0).stop 2020-11-20T12:08:47.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.199+0000 7fe7fb59e700 1 -- 172.21.15.187:0/578784708 shutdown_connections 2020-11-20T12:08:47.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.199+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/578784708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc14c260 0x7fe7fc14c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.199+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/578784708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 0x7fe7fc150ad0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.199+0000 7fe7fb59e700 1 -- 172.21.15.187:0/578784708 >> 172.21.15.187:0/578784708 conn(0x7fe7fc0bab30 msgr2=0x7fe7fc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:47.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.199+0000 7fe7fb59e700 1 -- 172.21.15.187:0/578784708 shutdown_connections 2020-11-20T12:08:47.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.199+0000 7fe7fb59e700 1 -- 172.21.15.187:0/578784708 wait complete. 2020-11-20T12:08:47.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.201+0000 7fe7fb59e700 1 Processor -- start 2020-11-20T12:08:47.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.201+0000 7fe7fb59e700 1 -- start start 2020-11-20T12:08:47.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.202+0000 7fe7fb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 0x7fe7fc09c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.202+0000 7fe7fb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc0b1760 0x7fe7fc15cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.202+0000 7fe7fb59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7fc14c260 0x7fe7fc0b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.202+0000 7fe7fb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc15d190 con 0x7fe7fc0b1380 2020-11-20T12:08:47.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.202+0000 7fe7fb59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc09bb70 con 0x7fe7fc14c260 2020-11-20T12:08:47.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.202+0000 7fe7fb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc09be10 con 0x7fe7fc0b1760 2020-11-20T12:08:47.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.203+0000 7fe7fad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7fc14c260 0x7fe7fc0b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.203+0000 7fe7fad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7fc14c260 0x7fe7fc0b74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44186/0 (socket says 172.21.15.187:44186) 2020-11-20T12:08:47.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.203+0000 7fe7fad9d700 1 -- 172.21.15.187:0/2081878723 learned_addr learned my addr 172.21.15.187:0/2081878723 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:47.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.203+0000 7fe7f1d9b700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc0b1760 0x7fe7fc15cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.203+0000 7fe7fa59c700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 0x7fe7fc09c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7fad9d700 1 -- 172.21.15.187:0/2081878723 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc0b1760 msgr2=0x7fe7fc15cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7fad9d700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc0b1760 0x7fe7fc15cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7fad9d700 1 -- 172.21.15.187:0/2081878723 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 msgr2=0x7fe7fc09c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7fad9d700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 0x7fe7fc09c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7fad9d700 1 -- 172.21.15.187:0/2081878723 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc09c0b0 con 0x7fe7fc14c260 2020-11-20T12:08:47.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7fad9d700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7fc14c260 0x7fe7fc0b74d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fe7f400ed20 tx=0x7fe7f40091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:47.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7f2ffd700 1 -- 172.21.15.187:0/2081878723 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400cbb0 con 0x7fe7fc14c260 2020-11-20T12:08:47.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7f2ffd700 1 -- 172.21.15.187:0/2081878723 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f400cd10 con 0x7fe7fc14c260 2020-11-20T12:08:47.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.204+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc09cd80 con 0x7fe7fc14c260 2020-11-20T12:08:47.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.205+0000 7fe7f2ffd700 1 -- 172.21.15.187:0/2081878723 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4012620 con 0x7fe7fc14c260 2020-11-20T12:08:47.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.205+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc159b20 con 0x7fe7fc14c260 2020-11-20T12:08:47.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.206+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc09c240 con 0x7fe7fc14c260 2020-11-20T12:08:47.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.206+0000 7fe7f2ffd700 1 -- 172.21.15.187:0/2081878723 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe7f4022070 con 0x7fe7fc14c260 2020-11-20T12:08:47.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.207+0000 7fe7f2ffd700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7d8056d20 0x7fe7d80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.207+0000 7fe7f2ffd700 1 -- 172.21.15.187:0/2081878723 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe7f4070740 con 0x7fe7fc14c260 2020-11-20T12:08:47.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.207+0000 7fe7fa59c700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7d8056d20 0x7fe7d80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.208+0000 7fe7fa59c700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7d8056d20 0x7fe7d80591e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fe7ec006e30 tx=0x7fe7ec015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:47.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.211+0000 7fe7f2ffd700 1 -- 172.21.15.187:0/2081878723 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe7f4045b90 con 0x7fe7fc14c260 2020-11-20T12:08:47.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.396+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7fc061a20 con 0x7fe7fc14c260 2020-11-20T12:08:47.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.398+0000 7fe7f2ffd700 1 -- 172.21.15.187:0/2081878723 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16780 (secure 0 0 0) 0x7fe7f4016020 con 0x7fe7fc14c260 2020-11-20T12:08:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:47.403 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":51,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:08:31.182203+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:08:28.213673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2020-11-20T12:08:28.856619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:47.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.400+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7d8056d20 msgr2=0x7fe7d80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.400+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7d8056d20 0x7fe7d80591e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fe7ec006e30 tx=0x7fe7ec015000).stop 2020-11-20T12:08:47.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.400+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7fc14c260 msgr2=0x7fe7fc0b74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.400+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7fc14c260 0x7fe7fc0b74d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fe7f400ed20 tx=0x7fe7f40091d0).stop 2020-11-20T12:08:47.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.401+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 shutdown_connections 2020-11-20T12:08:47.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.401+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe7d8056d20 0x7fe7d80591e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.401+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7fc0b1380 0x7fe7fc09c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.401+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7fc14c260 0x7fe7fc0b74d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.401+0000 7fe7fb59e700 1 --2- 172.21.15.187:0/2081878723 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7fc0b1760 0x7fe7fc15cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.401+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 >> 172.21.15.187:0/2081878723 conn(0x7fe7fc0bab30 msgr2=0x7fe7fc14f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:47.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.401+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 shutdown_connections 2020-11-20T12:08:47.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.401+0000 7fe7fb59e700 1 -- 172.21.15.187:0/2081878723 wait complete. 2020-11-20T12:08:47.429 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:08:28.213673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:08:28.856619+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:08:47.430 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:08:47.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.676+0000 7f023183b700 1 Processor -- start 2020-11-20T12:08:47.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.676+0000 7f023183b700 1 -- start start 2020-11-20T12:08:47.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.677+0000 7f023183b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f022c0b1380 0x7f022c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.677+0000 7f023183b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f022c0b1760 0x7f022c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.677+0000 7f023183b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065680 0x7f022c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.677+0000 7f023183b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022c150390 con 0x7f022c0b1760 2020-11-20T12:08:47.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.677+0000 7f023183b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022c150630 con 0x7f022c0b1380 2020-11-20T12:08:47.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.677+0000 7f023183b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022c068510 con 0x7f022c065680 2020-11-20T12:08:47.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.678+0000 7f0230839700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f022c0b1380 0x7f022c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.678+0000 7f0230839700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f022c0b1380 0x7f022c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44190/0 (socket says 172.21.15.187:44190) 2020-11-20T12:08:47.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.678+0000 7f0230839700 1 -- 172.21.15.187:0/884616733 learned_addr learned my addr 172.21.15.187:0/884616733 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:47.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.678+0000 7f022bfff700 1 --2- 172.21.15.187:0/884616733 >> v2:172.21.15.5:3300/0 conn(0x7f022c0b1760 0x7f022c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.678+0000 7f023103a700 1 --2- 172.21.15.187:0/884616733 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065680 0x7f022c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.678+0000 7f0230839700 1 -- 172.21.15.187:0/884616733 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065680 msgr2=0x7f022c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:47.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.678+0000 7f0230839700 1 --2- 172.21.15.187:0/884616733 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065680 0x7f022c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.679+0000 7f0230839700 1 -- 172.21.15.187:0/884616733 >> v2:172.21.15.5:3300/0 conn(0x7f022c0b1760 msgr2=0x7f022c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:47.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.679+0000 7f0230839700 1 --2- 172.21.15.187:0/884616733 >> v2:172.21.15.5:3300/0 conn(0x7f022c0b1760 0x7f022c065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.679+0000 7f0230839700 1 -- 172.21.15.187:0/884616733 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c150fc0 con 0x7f022c0b1380 2020-11-20T12:08:47.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.679+0000 7f0230839700 1 --2- 172.21.15.187:0/884616733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b1380 0x7f022c150ad0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f022c0b85e0 tx=0x7f0220006eb0).ready entity=mon.1 client_cookie=74c7947e62af6819 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:47.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.679+0000 7f022b7fe700 1 -- 172.21.15.187:0/884616733 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02200098d0 con 0x7f022c0b1380 2020-11-20T12:08:47.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.679+0000 7f022b7fe700 1 -- 172.21.15.187:0/884616733 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0220009a30 con 0x7f022c0b1380 2020-11-20T12:08:47.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.679+0000 7f022b7fe700 1 -- 172.21.15.187:0/884616733 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0220009db0 con 0x7f022c0b1380 2020-11-20T12:08:47.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 -- 172.21.15.187:0/884616733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b1380 msgr2=0x7f022c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 --2- 172.21.15.187:0/884616733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b1380 0x7f022c150ad0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f022c0b85e0 tx=0x7f0220006eb0).stop 2020-11-20T12:08:47.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 -- 172.21.15.187:0/884616733 shutdown_connections 2020-11-20T12:08:47.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 --2- 172.21.15.187:0/884616733 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065680 0x7f022c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 --2- 172.21.15.187:0/884616733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b1380 0x7f022c150ad0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 --2- 172.21.15.187:0/884616733 >> v2:172.21.15.5:3300/0 conn(0x7f022c0b1760 0x7f022c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 -- 172.21.15.187:0/884616733 >> 172.21.15.187:0/884616733 conn(0x7f022c0bab30 msgr2=0x7f022c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:47.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 -- 172.21.15.187:0/884616733 shutdown_connections 2020-11-20T12:08:47.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.680+0000 7f023183b700 1 -- 172.21.15.187:0/884616733 wait complete. 2020-11-20T12:08:47.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.682+0000 7f023183b700 1 Processor -- start 2020-11-20T12:08:47.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.682+0000 7f023183b700 1 -- start start 2020-11-20T12:08:47.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.683+0000 7f023183b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c0b1380 0x7f022c09ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023183b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c0b1760 0x7f022c1496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023183b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b4290 0x7f022c149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023183b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c14a0c0 con 0x7f022c0b1380 2020-11-20T12:08:47.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023183b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0b2df0 con 0x7f022c0b4290 2020-11-20T12:08:47.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023183b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0b3090 con 0x7f022c0b1760 2020-11-20T12:08:47.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023103a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b4290 0x7f022c149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023103a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b4290 0x7f022c149bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44200/0 (socket says 172.21.15.187:44200) 2020-11-20T12:08:47.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023103a700 1 -- 172.21.15.187:0/2198917500 learned_addr learned my addr 172.21.15.187:0/2198917500 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:47.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f0230839700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c0b1380 0x7f022c09ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f022bfff700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c0b1760 0x7f022c1496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023103a700 1 -- 172.21.15.187:0/2198917500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c0b1760 msgr2=0x7f022c1496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023103a700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c0b1760 0x7f022c1496e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023103a700 1 -- 172.21.15.187:0/2198917500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c0b1380 msgr2=0x7f022c09ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023103a700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c0b1380 0x7f022c09ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.684+0000 7f023103a700 1 -- 172.21.15.187:0/2198917500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c0b3330 con 0x7f022c0b4290 2020-11-20T12:08:47.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.685+0000 7f023103a700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b4290 0x7f022c149bd0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f022400eb70 tx=0x7f0224008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:47.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.685+0000 7f0228ff9700 1 -- 172.21.15.187:0/2198917500 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022400c7a0 con 0x7f022c0b4290 2020-11-20T12:08:47.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.685+0000 7f0228ff9700 1 -- 172.21.15.187:0/2198917500 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022400c900 con 0x7f022c0b4290 2020-11-20T12:08:47.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.685+0000 7f0228ff9700 1 -- 172.21.15.187:0/2198917500 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022400cc80 con 0x7f022c0b4290 2020-11-20T12:08:47.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.685+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c0b2b10 con 0x7f022c0b4290 2020-11-20T12:08:47.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.685+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c09d1c0 con 0x7f022c0b4290 2020-11-20T12:08:47.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.686+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c061a20 con 0x7f022c0b4290 2020-11-20T12:08:47.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.687+0000 7f0228ff9700 1 -- 172.21.15.187:0/2198917500 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f022401d460 con 0x7f022c0b4290 2020-11-20T12:08:47.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.688+0000 7f0228ff9700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:47.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.688+0000 7f0228ff9700 1 -- 172.21.15.187:0/2198917500 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f022406ffd0 con 0x7f022c0b4290 2020-11-20T12:08:47.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.688+0000 7f0230839700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:47.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.689+0000 7f0230839700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f022000b1b0 tx=0x7f0220015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:47.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.691+0000 7f0228ff9700 1 -- 172.21.15.187:0/2198917500 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f02240453c0 con 0x7f022c0b4290 2020-11-20T12:08:47.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.876+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f022c099ad0 con 0x7f022c0b4290 2020-11-20T12:08:47.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.877+0000 7f0228ff9700 1 -- 172.21.15.187:0/2198917500 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v51) v1 ==== 110+0+10 (secure 0 0 0) 0x7f02240489e0 con 0x7f022c0b4290 2020-11-20T12:08:47.880 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:08:47.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.880+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 msgr2=0x7f02040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.880+0000 7f023183b700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f022000b1b0 tx=0x7f0220015000).stop 2020-11-20T12:08:47.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b4290 msgr2=0x7f022c149bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:47.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b4290 0x7f022c149bd0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f022400eb70 tx=0x7f0224008040).stop 2020-11-20T12:08:47.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 shutdown_connections 2020-11-20T12:08:47.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c0b1380 0x7f022c09ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c0b4290 0x7f022c149bd0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 --2- 172.21.15.187:0/2198917500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c0b1760 0x7f022c1496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:47.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 >> 172.21.15.187:0/2198917500 conn(0x7f022c0bab30 msgr2=0x7f022c14b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:47.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 shutdown_connections 2020-11-20T12:08:47.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:47.881+0000 7f023183b700 1 -- 172.21.15.187:0/2198917500 wait complete. 2020-11-20T12:08:47.908 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:08:48.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.144+0000 7fd4a99cf700 1 Processor -- start 2020-11-20T12:08:48.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.144+0000 7fd4a99cf700 1 -- start start 2020-11-20T12:08:48.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.145+0000 7fd4a99cf700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4a4065f90 0x7fd4a4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:48.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.146+0000 7fd4a99cf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a40b13e0 con 0x7fd4a409ca90 2020-11-20T12:08:48.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.146+0000 7fd4a99cf700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a4062c10 con 0x7fd4a4065190 2020-11-20T12:08:48.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.146+0000 7fd4a99cf700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a414ab30 con 0x7fd4a4065f90 2020-11-20T12:08:48.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.146+0000 7fd4a89cd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd4a4065190 0x7fd4a4143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40810/0 (socket says 172.21.15.187:40810) 2020-11-20T12:08:48.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.146+0000 7fd4a89cd700 1 -- 172.21.15.187:0/3782000378 learned_addr learned my addr 172.21.15.187:0/3782000378 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a3fff700 1 --2- 172.21.15.187:0/3782000378 >> v2:172.21.15.5:3300/0 conn(0x7fd4a4065f90 0x7fd4a4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541657552 0 0) 0x7fd4a4062c10 con 0x7fd4a4065190 2020-11-20T12:08:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd484003240 con 0x7fd4a4065190 2020-11-20T12:08:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4194316202 0 0) 0x7fd484003240 con 0x7fd4a4065190 2020-11-20T12:08:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 >> v2:172.21.15.5:3300/0 conn(0x7fd4a4065f90 msgr2=0x7fd4a4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 --2- 172.21.15.187:0/3782000378 >> v2:172.21.15.5:3300/0 conn(0x7fd4a4065f90 0x7fd4a4065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 >> v1:172.21.15.5:6789/0 conn(0x7fd4a409ca90 legacy=0x7fd4a409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a40b1640 con 0x7fd4a4065190 2020-11-20T12:08:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd494001860 con 0x7fd4a4065190 2020-11-20T12:08:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 >> v1:172.21.15.187:6789/0 conn(0x7fd4a4065190 legacy=0x7fd4a4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.147+0000 7fd4a37fe700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd484003780 0x7fd484003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a37fe700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd484004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a37fe700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd484004e50 0x7fd48400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a414ab30 con 0x7fd484004e50 2020-11-20T12:08:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd484005300 con 0x7fd4840044e0 2020-11-20T12:08:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48400bc20 con 0x7fd484003780 2020-11-20T12:08:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a3fff700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd484004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a89cd700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd484003780 0x7fd484003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a91ce700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd484004e50 0x7fd48400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a3fff700 1 -- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd484003780 msgr2=0x7fd484003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a3fff700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd484003780 0x7fd484003bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.148+0000 7fd4a3fff700 1 -- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd484004e50 msgr2=0x7fd48400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.149+0000 7fd4a3fff700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd484004e50 0x7fd48400b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.149+0000 7fd4a3fff700 1 -- 172.21.15.187:0/3782000378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd484003240 con 0x7fd4840044e0 2020-11-20T12:08:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.149+0000 7fd4a91ce700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd484004e50 0x7fd48400b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.149+0000 7fd4a91ce700 1 -- 172.21.15.187:0/3782000378 reap_dead start 2020-11-20T12:08:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.149+0000 7fd4a3fff700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd484004960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fd4a4066440 tx=0x7fd48c006df0).ready entity=mon.1 client_cookie=27837800b2699d79 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.149+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00b330 con 0x7fd4840044e0 2020-11-20T12:08:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.149+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c00eae0 con 0x7fd4840044e0 2020-11-20T12:08:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.149+0000 7fd4a37fe700 1 -- 172.21.15.187:0/3782000378 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00ee60 con 0x7fd4840044e0 2020-11-20T12:08:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.150+0000 7fd4a99cf700 1 -- 172.21.15.187:0/3782000378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 msgr2=0x7fd484004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.150+0000 7fd4a99cf700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd484004960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fd4a4066440 tx=0x7fd48c006df0).stop 2020-11-20T12:08:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.150+0000 7fd4a99cf700 1 -- 172.21.15.187:0/3782000378 shutdown_connections 2020-11-20T12:08:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.150+0000 7fd4a99cf700 1 --2- 172.21.15.187:0/3782000378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd484004960 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.150+0000 7fd4a99cf700 1 -- 172.21.15.187:0/3782000378 >> 172.21.15.187:0/3782000378 conn(0x7fd4a40bab30 msgr2=0x7fd4a40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.150+0000 7fd4a99cf700 1 -- 172.21.15.187:0/3782000378 shutdown_connections 2020-11-20T12:08:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.150+0000 7fd4a99cf700 1 -- 172.21.15.187:0/3782000378 wait complete. 2020-11-20T12:08:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.151+0000 7fd4a99cf700 1 Processor -- start 2020-11-20T12:08:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.151+0000 7fd4a99cf700 1 -- start start 2020-11-20T12:08:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.152+0000 7fd4a99cf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd4a41476d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.152+0000 7fd4a99cf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40b4450 0x7fd4a40b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.152+0000 7fd4a99cf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b7b80 0x7fd4a40b7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.152+0000 7fd4a99cf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40b3820 con 0x7fd4a40b7b80 2020-11-20T12:08:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.152+0000 7fd4a99cf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40b3a70 con 0x7fd4840044e0 2020-11-20T12:08:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.152+0000 7fd4a99cf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40b3d10 con 0x7fd4a40b4450 2020-11-20T12:08:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd4a41476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd4a41476d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44216/0 (socket says 172.21.15.187:44216) 2020-11-20T12:08:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 -- 172.21.15.187:0/1774568011 learned_addr learned my addr 172.21.15.187:0/1774568011 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a91ce700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b7b80 0x7fd4a40b7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a3fff700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40b4450 0x7fd4a40b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 -- 172.21.15.187:0/1774568011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40b4450 msgr2=0x7fd4a40b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40b4450 0x7fd4a40b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 -- 172.21.15.187:0/1774568011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b7b80 msgr2=0x7fd4a40b7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b7b80 0x7fd4a40b7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 -- 172.21.15.187:0/1774568011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a40b3fb0 con 0x7fd4840044e0 2020-11-20T12:08:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.153+0000 7fd4a89cd700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd4a41476d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7fd49400de80 tx=0x7fd494009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.154+0000 7fd4a0ff9700 1 -- 172.21.15.187:0/1774568011 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49400bc60 con 0x7fd4840044e0 2020-11-20T12:08:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.154+0000 7fd4a0ff9700 1 -- 172.21.15.187:0/1774568011 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49400bdc0 con 0x7fd4840044e0 2020-11-20T12:08:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.154+0000 7fd4a0ff9700 1 -- 172.21.15.187:0/1774568011 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4940073a0 con 0x7fd4840044e0 2020-11-20T12:08:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.154+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a4147bc0 con 0x7fd4840044e0 2020-11-20T12:08:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.154+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a40b48d0 con 0x7fd4840044e0 2020-11-20T12:08:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.155+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a40b4140 con 0x7fd4840044e0 2020-11-20T12:08:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.156+0000 7fd4a0ff9700 1 -- 172.21.15.187:0/1774568011 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd494004880 con 0x7fd4840044e0 2020-11-20T12:08:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.156+0000 7fd4a0ff9700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd480056d20 0x7fd4800591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.156+0000 7fd4a0ff9700 1 -- 172.21.15.187:0/1774568011 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd494070620 con 0x7fd4840044e0 2020-11-20T12:08:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.157+0000 7fd4a3fff700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd480056d20 0x7fd4800591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.157+0000 7fd4a3fff700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd480056d20 0x7fd4800591e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fd48c006e30 tx=0x7fd48c016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:48.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.160+0000 7fd4a0ff9700 1 -- 172.21.15.187:0/1774568011 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd494045a70 con 0x7fd4840044e0 2020-11-20T12:08:48.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.344+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd4a4061a20 con 0x7fd4840044e0 2020-11-20T12:08:48.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.345+0000 7fd4a0ff9700 1 -- 172.21.15.187:0/1774568011 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v51) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd494007710 con 0x7fd4840044e0 2020-11-20T12:08:48.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.347+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd480056d20 msgr2=0x7fd4800591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:48.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.347+0000 7fd4a99cf700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd480056d20 0x7fd4800591e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fd48c006e30 tx=0x7fd48c016000).stop 2020-11-20T12:08:48.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.347+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 msgr2=0x7fd4a41476d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:48.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.347+0000 7fd4a99cf700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd4a41476d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7fd49400de80 tx=0x7fd494009040).stop 2020-11-20T12:08:48.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.348+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 shutdown_connections 2020-11-20T12:08:48.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.348+0000 7fd4a99cf700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd480056d20 0x7fd4800591e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.348+0000 7fd4a99cf700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b7b80 0x7fd4a40b7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.348+0000 7fd4a99cf700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4840044e0 0x7fd4a41476d0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.348+0000 7fd4a99cf700 1 --2- 172.21.15.187:0/1774568011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40b4450 0x7fd4a40b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:48.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.348+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 >> 172.21.15.187:0/1774568011 conn(0x7fd4a40bab30 msgr2=0x7fd4a4141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:48.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.348+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 shutdown_connections 2020-11-20T12:08:48.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:48.348+0000 7fd4a99cf700 1 -- 172.21.15.187:0/1774568011 wait complete. 2020-11-20T12:08:48.876 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:48.877 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:08:49.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.129+0000 7fdf6eec6700 1 Processor -- start 2020-11-20T12:08:49.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.129+0000 7fdf6eec6700 1 -- start start 2020-11-20T12:08:49.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.131+0000 7fdf6eec6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf680656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.131+0000 7fdf6eec6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdf68068030 0x7fdf68146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.131+0000 7fdf6eec6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf68066420 con 0x7fdf6809a8a0 2020-11-20T12:08:49.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.131+0000 7fdf6eec6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf680b7380 con 0x7fdf68068030 2020-11-20T12:08:49.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.131+0000 7fdf6eec6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf68061f60 con 0x7fdf68065300 2020-11-20T12:08:49.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.132+0000 7fdf6cc62700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf680656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.132+0000 7fdf6d463700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdf68068030 0x7fdf68146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.132+0000 7fdf6d463700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdf68068030 0x7fdf68146950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:36490/0 (socket says 172.21.15.187:36490) 2020-11-20T12:08:49.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.132+0000 7fdf6cc62700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf680656e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:45116/0 (socket says 172.21.15.187:45116) 2020-11-20T12:08:49.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.132+0000 7fdf6d463700 1 -- 172.21.15.187:0/1502617647 learned_addr learned my addr 172.21.15.187:0/1502617647 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:49.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.133+0000 7fdf6d463700 1 -- 172.21.15.187:0/1502617647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 msgr2=0x7fdf680656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:49.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.133+0000 7fdf6d463700 1 --2- 172.21.15.187:0/1502617647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf680656e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.133+0000 7fdf6d463700 1 -- 172.21.15.187:0/1502617647 >> v1:172.21.15.5:6789/0 conn(0x7fdf6809a8a0 legacy=0x7fdf68062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.133+0000 7fdf6d463700 1 -- 172.21.15.187:0/1502617647 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680b6fd0 con 0x7fdf68068030 2020-11-20T12:08:49.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.133+0000 7fdf6d463700 1 --2- 172.21.15.187:0/1502617647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf68068030 0x7fdf68146950 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fdf60009d80 tx=0x7fdf60009290).ready entity=mon.0 client_cookie=21b7af3c3dc5013a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:49.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.134+0000 7fdf5f7fe700 1 -- 172.21.15.187:0/1502617647 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6000bc10 con 0x7fdf68068030 2020-11-20T12:08:49.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.134+0000 7fdf5f7fe700 1 -- 172.21.15.187:0/1502617647 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6000bd70 con 0x7fdf68068030 2020-11-20T12:08:49.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.134+0000 7fdf5f7fe700 1 -- 172.21.15.187:0/1502617647 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60011610 con 0x7fdf68068030 2020-11-20T12:08:49.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.134+0000 7fdf6eec6700 1 -- 172.21.15.187:0/1502617647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf68068030 msgr2=0x7fdf68146950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.134+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/1502617647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf68068030 0x7fdf68146950 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fdf60009d80 tx=0x7fdf60009290).stop 2020-11-20T12:08:49.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.135+0000 7fdf6eec6700 1 -- 172.21.15.187:0/1502617647 shutdown_connections 2020-11-20T12:08:49.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.135+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/1502617647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf68068030 0x7fdf68146950 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.135+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/1502617647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf680656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.135+0000 7fdf6eec6700 1 -- 172.21.15.187:0/1502617647 >> 172.21.15.187:0/1502617647 conn(0x7fdf680bacf0 msgr2=0x7fdf680bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:49.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.135+0000 7fdf6eec6700 1 -- 172.21.15.187:0/1502617647 shutdown_connections 2020-11-20T12:08:49.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.135+0000 7fdf6eec6700 1 -- 172.21.15.187:0/1502617647 wait complete. 2020-11-20T12:08:49.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.136+0000 7fdf6eec6700 1 Processor -- start 2020-11-20T12:08:49.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.137+0000 7fdf6eec6700 1 -- start start 2020-11-20T12:08:49.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.137+0000 7fdf6eec6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf6813fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.137+0000 7fdf6eec6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf68068030 0x7fdf681402d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.137+0000 7fdf6eec6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6809a8a0 0x7fdf680b1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.137+0000 7fdf6eec6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680b4c20 con 0x7fdf68068030 2020-11-20T12:08:49.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.137+0000 7fdf6eec6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680b01c0 con 0x7fdf6809a8a0 2020-11-20T12:08:49.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.137+0000 7fdf6eec6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680b0e70 con 0x7fdf68065300 2020-11-20T12:08:49.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6d463700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6809a8a0 0x7fdf680b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6d463700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6809a8a0 0x7fdf680b1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44234/0 (socket says 172.21.15.187:44234) 2020-11-20T12:08:49.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6d463700 1 -- 172.21.15.187:0/3739465863 learned_addr learned my addr 172.21.15.187:0/3739465863 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:49.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6cc62700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf6813fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6d463700 1 -- 172.21.15.187:0/3739465863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 msgr2=0x7fdf6813fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6d463700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf6813fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6d463700 1 -- 172.21.15.187:0/3739465863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf68068030 msgr2=0x7fdf681402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6d463700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf68068030 0x7fdf681402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:49.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6d463700 1 -- 172.21.15.187:0/3739465863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680b1110 con 0x7fdf6809a8a0 2020-11-20T12:08:49.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.138+0000 7fdf6cc62700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf6813fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:49.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.139+0000 7fdf6d463700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6809a8a0 0x7fdf680b1910 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fdf600092d0 tx=0x7fdf60016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:49.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.139+0000 7fdf5cff9700 1 -- 172.21.15.187:0/3739465863 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6000bee0 con 0x7fdf6809a8a0 2020-11-20T12:08:49.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.139+0000 7fdf5cff9700 1 -- 172.21.15.187:0/3739465863 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf60011650 con 0x7fdf6809a8a0 2020-11-20T12:08:49.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.139+0000 7fdf5cff9700 1 -- 172.21.15.187:0/3739465863 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60010e10 con 0x7fdf6809a8a0 2020-11-20T12:08:49.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.139+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf68140820 con 0x7fdf6809a8a0 2020-11-20T12:08:49.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.139+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf680aff90 con 0x7fdf6809a8a0 2020-11-20T12:08:49.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.140+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf680b12a0 con 0x7fdf6809a8a0 2020-11-20T12:08:49.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.141+0000 7fdf5cff9700 1 -- 172.21.15.187:0/3739465863 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdf60023030 con 0x7fdf6809a8a0 2020-11-20T12:08:49.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.141+0000 7fdf5cff9700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf40056d20 0x7fdf400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.141+0000 7fdf5cff9700 1 -- 172.21.15.187:0/3739465863 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdf600716c0 con 0x7fdf6809a8a0 2020-11-20T12:08:49.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.142+0000 7fdf6cc62700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf40056d20 0x7fdf400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.142+0000 7fdf6cc62700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf40056d20 0x7fdf400591e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fdf64006bd0 tx=0x7fdf64008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:49.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.146+0000 7fdf5cff9700 1 -- 172.21.15.187:0/3739465863 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdf60046c40 con 0x7fdf6809a8a0 2020-11-20T12:08:49.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.331+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf68044f60 con 0x7fdf6809a8a0 2020-11-20T12:08:49.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.332+0000 7fdf5cff9700 1 -- 172.21.15.187:0/3739465863 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+10 (secure 0 0 0) 0x7fdf60021020 con 0x7fdf6809a8a0 2020-11-20T12:08:49.336 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.334+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf40056d20 msgr2=0x7fdf400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.334+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf40056d20 0x7fdf400591e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fdf64006bd0 tx=0x7fdf64008040).stop 2020-11-20T12:08:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.334+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6809a8a0 msgr2=0x7fdf680b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.334+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6809a8a0 0x7fdf680b1910 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fdf600092d0 tx=0x7fdf60016000).stop 2020-11-20T12:08:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.334+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 shutdown_connections 2020-11-20T12:08:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.334+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf40056d20 0x7fdf400591e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.335+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6809a8a0 0x7fdf680b1910 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.335+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf68065300 0x7fdf6813fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.335+0000 7fdf6eec6700 1 --2- 172.21.15.187:0/3739465863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf68068030 0x7fdf681402d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:49.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.335+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 >> 172.21.15.187:0/3739465863 conn(0x7fdf680bacf0 msgr2=0x7fdf6809caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:49.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.335+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 shutdown_connections 2020-11-20T12:08:49.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.335+0000 7fdf6eec6700 1 -- 172.21.15.187:0/3739465863 wait complete. 2020-11-20T12:08:49.362 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:08:49.413 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:08:49.425 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:08:49.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.594+0000 7f12ceb8f700 1 Processor -- start 2020-11-20T12:08:49.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.594+0000 7f12ceb8f700 1 -- start start 2020-11-20T12:08:49.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.595+0000 7f12ceb8f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c80bf800 0x7f12c814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.595+0000 7f12ceb8f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12c80bfbe0 0x7f12c81480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.595+0000 7f12ceb8f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c814d110 con 0x7f12c809f560 2020-11-20T12:08:49.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.595+0000 7f12ceb8f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c814cbd0 con 0x7f12c80bfbe0 2020-11-20T12:08:49.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.595+0000 7f12ceb8f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c814ce70 con 0x7f12c80bf800 2020-11-20T12:08:49.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.596+0000 7f12cdb8d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c80bf800 0x7f12c814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.596+0000 7f12cd38c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12c80bfbe0 0x7f12c81480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.596+0000 7f12cdb8d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c80bf800 0x7f12c814c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:45126/0 (socket says 172.21.15.187:45126) 2020-11-20T12:08:49.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.596+0000 7f12cdb8d700 1 -- 172.21.15.187:0/987647131 learned_addr learned my addr 172.21.15.187:0/987647131 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:49.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.596+0000 7f12ce38e700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f12c809f560 0x7f12c8060c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:34358/0 (socket says 172.21.15.187:34358) 2020-11-20T12:08:49.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.597+0000 7f12cdb8d700 1 -- 172.21.15.187:0/987647131 >> v2:172.21.15.5:3300/0 conn(0x7f12c80bfbe0 msgr2=0x7f12c81480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:49.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.597+0000 7f12cdb8d700 1 --2- 172.21.15.187:0/987647131 >> v2:172.21.15.5:3300/0 conn(0x7f12c80bfbe0 0x7f12c81480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.597+0000 7f12cdb8d700 1 -- 172.21.15.187:0/987647131 >> v1:172.21.15.5:6789/0 conn(0x7f12c809f560 legacy=0x7f12c8060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.598+0000 7f12cdb8d700 1 -- 172.21.15.187:0/987647131 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80a4cc0 con 0x7f12c80bf800 2020-11-20T12:08:49.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.598+0000 7f12cdb8d700 1 --2- 172.21.15.187:0/987647131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c80bf800 0x7f12c814c2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f12c4009d50 tx=0x7f12c4009290).ready entity=mon.2 client_cookie=3d83e70c4797ac server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:49.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.598+0000 7f12ccb8b700 1 -- 172.21.15.187:0/987647131 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c400bc50 con 0x7f12c80bf800 2020-11-20T12:08:49.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.598+0000 7f12ccb8b700 1 -- 172.21.15.187:0/987647131 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c400bdb0 con 0x7f12c80bf800 2020-11-20T12:08:49.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.598+0000 7f12ccb8b700 1 -- 172.21.15.187:0/987647131 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4011610 con 0x7f12c80bf800 2020-11-20T12:08:49.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.599+0000 7f12ceb8f700 1 -- 172.21.15.187:0/987647131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c80bf800 msgr2=0x7f12c814c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.599+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/987647131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c80bf800 0x7f12c814c2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f12c4009d50 tx=0x7f12c4009290).stop 2020-11-20T12:08:49.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.599+0000 7f12ceb8f700 1 -- 172.21.15.187:0/987647131 shutdown_connections 2020-11-20T12:08:49.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.599+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/987647131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c80bf800 0x7f12c814c2c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.599+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/987647131 >> v2:172.21.15.5:3300/0 conn(0x7f12c80bfbe0 0x7f12c81480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.599+0000 7f12ceb8f700 1 -- 172.21.15.187:0/987647131 >> 172.21.15.187:0/987647131 conn(0x7f12c8068370 msgr2=0x7f12c80bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:49.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.599+0000 7f12ceb8f700 1 -- 172.21.15.187:0/987647131 shutdown_connections 2020-11-20T12:08:49.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.599+0000 7f12ceb8f700 1 -- 172.21.15.187:0/987647131 wait complete. 2020-11-20T12:08:49.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.601+0000 7f12ceb8f700 1 Processor -- start 2020-11-20T12:08:49.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.601+0000 7f12ceb8f700 1 -- start start 2020-11-20T12:08:49.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.602+0000 7f12ceb8f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c809f560 0x7f12c81586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.602+0000 7f12ceb8f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12c80bf800 0x7f12c8141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.602+0000 7f12ceb8f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c814bec0 0x7f12c80b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.602+0000 7f12ceb8f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c815ba50 con 0x7f12c814bec0 2020-11-20T12:08:49.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.602+0000 7f12ceb8f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c81487e0 con 0x7f12c80bf800 2020-11-20T12:08:49.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.602+0000 7f12ceb8f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c8141a80 con 0x7f12c809f560 2020-11-20T12:08:49.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.602+0000 7f12cd38c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12c80bf800 0x7f12c8141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cd38c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12c80bf800 0x7f12c8141590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44246/0 (socket says 172.21.15.187:44246) 2020-11-20T12:08:49.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cd38c700 1 -- 172.21.15.187:0/2363570451 learned_addr learned my addr 172.21.15.187:0/2363570451 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:49.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cdb8d700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c809f560 0x7f12c81586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12ce38e700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c814bec0 0x7f12c80b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cd38c700 1 -- 172.21.15.187:0/2363570451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c809f560 msgr2=0x7f12c81586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cd38c700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c809f560 0x7f12c81586d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cd38c700 1 -- 172.21.15.187:0/2363570451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c814bec0 msgr2=0x7f12c80b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cd38c700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c814bec0 0x7f12c80b4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cd38c700 1 -- 172.21.15.187:0/2363570451 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8148a40 con 0x7f12c80bf800 2020-11-20T12:08:49.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.603+0000 7f12cd38c700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12c80bf800 0x7f12c8141590 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f12b800eba0 tx=0x7f12b8006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:49.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.604+0000 7f12b5ffb700 1 -- 172.21.15.187:0/2363570451 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b800b8d0 con 0x7f12c80bf800 2020-11-20T12:08:49.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.604+0000 7f12b5ffb700 1 -- 172.21.15.187:0/2363570451 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b800ba30 con 0x7f12c80bf800 2020-11-20T12:08:49.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.604+0000 7f12b5ffb700 1 -- 172.21.15.187:0/2363570451 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b800bdb0 con 0x7f12c80bf800 2020-11-20T12:08:49.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.604+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c8148ce0 con 0x7f12c80bf800 2020-11-20T12:08:49.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.604+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c80b49f0 con 0x7f12c80bf800 2020-11-20T12:08:49.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.605+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c80b4cd0 con 0x7f12c80bf800 2020-11-20T12:08:49.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.605+0000 7f12b5ffb700 1 -- 172.21.15.187:0/2363570451 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f12b8012600 con 0x7f12c80bf800 2020-11-20T12:08:49.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.606+0000 7f12b5ffb700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f12a0056d50 0x7f12a0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:49.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.606+0000 7f12b5ffb700 1 -- 172.21.15.187:0/2363570451 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f12b8070210 con 0x7f12c80bf800 2020-11-20T12:08:49.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.606+0000 7f12cdb8d700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f12a0056d50 0x7f12a0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:49.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.607+0000 7f12cdb8d700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f12a0056d50 0x7f12a0059210 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f12c4009fd0 tx=0x7f12c4016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:49.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.610+0000 7f12b5ffb700 1 -- 172.21.15.187:0/2363570451 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f12b8045600 con 0x7f12c80bf800 2020-11-20T12:08:49.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.793+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12c8080050 con 0x7f12c80bf800 2020-11-20T12:08:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.794+0000 7f12b5ffb700 1 -- 172.21.15.187:0/2363570451 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+10 (secure 0 0 0) 0x7f12b8048c20 con 0x7f12c80bf800 2020-11-20T12:08:49.798 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:08:49.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f12a0056d50 msgr2=0x7f12a0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f12a0056d50 0x7f12a0059210 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f12c4009fd0 tx=0x7f12c4016000).stop 2020-11-20T12:08:49.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12c80bf800 msgr2=0x7f12c8141590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:49.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12c80bf800 0x7f12c8141590 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f12b800eba0 tx=0x7f12b8006df0).stop 2020-11-20T12:08:49.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 shutdown_connections 2020-11-20T12:08:49.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f12a0056d50 0x7f12a0059210 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c814bec0 0x7f12c80b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c809f560 0x7f12c81586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 --2- 172.21.15.187:0/2363570451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12c80bf800 0x7f12c8141590 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:49.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.797+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 >> 172.21.15.187:0/2363570451 conn(0x7f12c8068370 msgr2=0x7f12c80a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:49.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.798+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 shutdown_connections 2020-11-20T12:08:49.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:49.798+0000 7f12ceb8f700 1 -- 172.21.15.187:0/2363570451 wait complete. 2020-11-20T12:08:49.824 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:50.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.061+0000 7f8d460d1700 1 Processor -- start 2020-11-20T12:08:50.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.061+0000 7f8d460d1700 1 -- start start 2020-11-20T12:08:50.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.063+0000 7f8d460d1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8d400b1380 0x7f8d40150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.063+0000 7f8d460d1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8d400b1760 0x7f8d40065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.063+0000 7f8d460d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d40065680 0x7f8d40061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.063+0000 7f8d460d1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d40150390 con 0x7f8d400b1380 2020-11-20T12:08:50.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.063+0000 7f8d460d1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d40150630 con 0x7f8d400b1760 2020-11-20T12:08:50.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.063+0000 7f8d460d1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40068510 con 0x7f8d40065680 2020-11-20T12:08:50.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d448ce700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8d400b1760 0x7f8d40065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d448ce700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8d400b1760 0x7f8d40065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44256/0 (socket says 172.21.15.187:44256) 2020-11-20T12:08:50.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d448ce700 1 -- 172.21.15.187:0/1551597871 learned_addr learned my addr 172.21.15.187:0/1551597871 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:50.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d450cf700 1 --2- 172.21.15.187:0/1551597871 >> v2:172.21.15.5:3300/0 conn(0x7f8d400b1380 0x7f8d40150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d458d0700 1 --2- 172.21.15.187:0/1551597871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d40065680 0x7f8d40061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d448ce700 1 -- 172.21.15.187:0/1551597871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d40065680 msgr2=0x7f8d40061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:50.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d448ce700 1 --2- 172.21.15.187:0/1551597871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d40065680 0x7f8d40061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d448ce700 1 -- 172.21.15.187:0/1551597871 >> v2:172.21.15.5:3300/0 conn(0x7f8d400b1380 msgr2=0x7f8d40150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:50.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d448ce700 1 --2- 172.21.15.187:0/1551597871 >> v2:172.21.15.5:3300/0 conn(0x7f8d400b1380 0x7f8d40150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.064+0000 7f8d448ce700 1 -- 172.21.15.187:0/1551597871 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d40150fc0 con 0x7f8d400b1760 2020-11-20T12:08:50.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.065+0000 7f8d448ce700 1 --2- 172.21.15.187:0/1551597871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d40065190 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f8d40063a20 tx=0x7f8d34006eb0).ready entity=mon.1 client_cookie=1ef342476d3c2091 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:50.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.065+0000 7f8d33fff700 1 -- 172.21.15.187:0/1551597871 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34009720 con 0x7f8d400b1760 2020-11-20T12:08:50.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.065+0000 7f8d33fff700 1 -- 172.21.15.187:0/1551597871 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d34009880 con 0x7f8d400b1760 2020-11-20T12:08:50.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.065+0000 7f8d33fff700 1 -- 172.21.15.187:0/1551597871 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34009c00 con 0x7f8d400b1760 2020-11-20T12:08:50.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 -- 172.21.15.187:0/1551597871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 msgr2=0x7f8d40065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1551597871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d40065190 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f8d40063a20 tx=0x7f8d34006eb0).stop 2020-11-20T12:08:50.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 -- 172.21.15.187:0/1551597871 shutdown_connections 2020-11-20T12:08:50.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1551597871 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d40065680 0x7f8d40061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1551597871 >> v2:172.21.15.5:3300/0 conn(0x7f8d400b1380 0x7f8d40150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1551597871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d40065190 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 -- 172.21.15.187:0/1551597871 >> 172.21.15.187:0/1551597871 conn(0x7f8d400bab30 msgr2=0x7f8d400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:50.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 -- 172.21.15.187:0/1551597871 shutdown_connections 2020-11-20T12:08:50.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.066+0000 7f8d460d1700 1 -- 172.21.15.187:0/1551597871 wait complete. 2020-11-20T12:08:50.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.068+0000 7f8d460d1700 1 Processor -- start 2020-11-20T12:08:50.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.068+0000 7f8d460d1700 1 -- start start 2020-11-20T12:08:50.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d460d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d400b1380 0x7f8d4015d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d460d1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d4015c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d460d1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d400b5ab0 0x7f8d4014a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d460d1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400b4250 con 0x7f8d400b5ab0 2020-11-20T12:08:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d460d1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4015caf0 con 0x7f8d400b1760 2020-11-20T12:08:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d460d1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400b44a0 con 0x7f8d400b1380 2020-11-20T12:08:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d448ce700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d4015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d448ce700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d4015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44260/0 (socket says 172.21.15.187:44260) 2020-11-20T12:08:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d448ce700 1 -- 172.21.15.187:0/1932741235 learned_addr learned my addr 172.21.15.187:0/1932741235 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.069+0000 7f8d450cf700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d400b1380 0x7f8d4015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d458d0700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d400b5ab0 0x7f8d4014a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d448ce700 1 -- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d400b1380 msgr2=0x7f8d4015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d448ce700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d400b1380 0x7f8d4015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d448ce700 1 -- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d400b5ab0 msgr2=0x7f8d4014a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d448ce700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d400b5ab0 0x7f8d4014a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d448ce700 1 -- 172.21.15.187:0/1932741235 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400b4740 con 0x7f8d400b1760 2020-11-20T12:08:50.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d458d0700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d400b5ab0 0x7f8d4014a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:50.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d448ce700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d4015c600 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f8d34006ef0 tx=0x7f8d34015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:50.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d317fa700 1 -- 172.21.15.187:0/1932741235 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34011d70 con 0x7f8d400b1760 2020-11-20T12:08:50.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d317fa700 1 -- 172.21.15.187:0/1932741235 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3401e460 con 0x7f8d400b1760 2020-11-20T12:08:50.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4009f350 con 0x7f8d400b1760 2020-11-20T12:08:50.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d317fa700 1 -- 172.21.15.187:0/1932741235 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3401e7e0 con 0x7f8d400b1760 2020-11-20T12:08:50.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.070+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d400b3e90 con 0x7f8d400b1760 2020-11-20T12:08:50.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.072+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d40061a20 con 0x7f8d400b1760 2020-11-20T12:08:50.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.072+0000 7f8d317fa700 1 -- 172.21.15.187:0/1932741235 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8d34025030 con 0x7f8d400b1760 2020-11-20T12:08:50.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.073+0000 7f8d317fa700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d18056d20 0x7f8d180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.073+0000 7f8d317fa700 1 -- 172.21.15.187:0/1932741235 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8d340715a0 con 0x7f8d400b1760 2020-11-20T12:08:50.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.073+0000 7f8d450cf700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d18056d20 0x7f8d180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.074+0000 7f8d450cf700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d18056d20 0x7f8d180591e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f8d400a02e0 tx=0x7f8d3c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:50.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.076+0000 7f8d317fa700 1 -- 172.21.15.187:0/1932741235 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8d34046aa0 con 0x7f8d400b1760 2020-11-20T12:08:50.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.327+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d400b49c0 con 0x7f8d400b1760 2020-11-20T12:08:50.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.328+0000 7f8d317fa700 1 -- 172.21.15.187:0/1932741235 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f8d34023020 con 0x7f8d400b1760 2020-11-20T12:08:50.332 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:50.332 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5019,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/4036313482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4036313482},{"type":"v1","addr":"172.21.15.5:6833","nonce":4036313482}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:08:30.177391+0000","modified":"2020-11-20T12:08:32.308400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5743},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5743":{"gid":5743,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1756209233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1756209233},{"type":"v1","addr":"172.21.15.187:6835","nonce":1756209233}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2020-11-20T12:08:50.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d18056d20 msgr2=0x7f8d180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d18056d20 0x7f8d180591e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f8d400a02e0 tx=0x7f8d3c008040).stop 2020-11-20T12:08:50.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 msgr2=0x7f8d4015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d4015c600 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f8d34006ef0 tx=0x7f8d34015000).stop 2020-11-20T12:08:50.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 shutdown_connections 2020-11-20T12:08:50.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d18056d20 0x7f8d180591e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d400b5ab0 0x7f8d4014a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d400b1380 0x7f8d4015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 --2- 172.21.15.187:0/1932741235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d400b1760 0x7f8d4015c600 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 >> 172.21.15.187:0/1932741235 conn(0x7f8d400bab30 msgr2=0x7f8d400b0920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:50.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 shutdown_connections 2020-11-20T12:08:50.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.331+0000 7f8d460d1700 1 -- 172.21.15.187:0/1932741235 wait complete. 2020-11-20T12:08:50.339 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 26 2020-11-20T12:08:50.357 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:08:50.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.587+0000 7f60dbfff700 1 Processor -- start 2020-11-20T12:08:50.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.587+0000 7f60dbfff700 1 -- start start 2020-11-20T12:08:50.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.589+0000 7f60dbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 0x7f60dc068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.589+0000 7f60dbfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60dc144b80 con 0x7f60dc0b5f50 2020-11-20T12:08:50.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.589+0000 7f60dbfff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60dc1489e0 con 0x7f60dc1482d0 2020-11-20T12:08:50.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.589+0000 7f60dbfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc148c80 con 0x7f60dc0b57b0 2020-11-20T12:08:50.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.589+0000 7f60da7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f60dc1482d0 0x7f60dc062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40872/0 (socket says 172.21.15.187:40872) 2020-11-20T12:08:50.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.589+0000 7f60da7fc700 1 -- 172.21.15.187:0/1022846884 learned_addr learned my addr 172.21.15.187:0/1022846884 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:50.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301169152 0 0) 0x7f60dc1489e0 con 0x7f60dc1482d0 2020-11-20T12:08:50.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c8003240 con 0x7f60dc1482d0 2020-11-20T12:08:50.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60daffd700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 0x7f60dc068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 728408240 0 0) 0x7f60c8003240 con 0x7f60dc1482d0 2020-11-20T12:08:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 msgr2=0x7f60dc068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 0x7f60dc068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 >> v1:172.21.15.5:6789/0 conn(0x7f60dc0b5f50 legacy=0x7f60dc0b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc148f20 con 0x7f60dc1482d0 2020-11-20T12:08:50.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f60d00014e0 con 0x7f60dc1482d0 2020-11-20T12:08:50.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 >> v1:172.21.15.187:6789/0 conn(0x7f60dc1482d0 legacy=0x7f60dc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60c8003730 0x7f60c8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60c80044e0 0x7f60c8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 0x7f60c8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc148c80 con 0x7f60c80044e0 2020-11-20T12:08:50.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c8005390 con 0x7f60c8003730 2020-11-20T12:08:50.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.590+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c80042f0 con 0x7f60dc0b57b0 2020-11-20T12:08:50.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.591+0000 7f60daffd700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60c8003730 0x7f60c8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.591+0000 7f60da7fc700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60c80044e0 0x7f60c8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.591+0000 7f60db7fe700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 0x7f60c8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.591+0000 7f60daffd700 1 -- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 msgr2=0x7f60c8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.591+0000 7f60daffd700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 0x7f60c8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.591+0000 7f60daffd700 1 -- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60c80044e0 msgr2=0x7f60c8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.591+0000 7f60daffd700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60c80044e0 0x7f60c8004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.592+0000 7f60daffd700 1 -- 172.21.15.187:0/1022846884 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8003240 con 0x7f60c8003730 2020-11-20T12:08:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.592+0000 7f60db7fe700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b57b0 0x7f60c8004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.592+0000 7f60daffd700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60c8003730 0x7f60c8003b50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f60cc00ba00 tx=0x7f60cc008040).ready entity=mon.1 client_cookie=ab13c4781142ef5c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.592+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00b270 con 0x7f60c8003730 2020-11-20T12:08:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.592+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc00d990 con 0x7f60c8003730 2020-11-20T12:08:50.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.592+0000 7f60d9ffb700 1 -- 172.21.15.187:0/1022846884 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00dd10 con 0x7f60c8003730 2020-11-20T12:08:50.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.593+0000 7f60dbfff700 1 -- 172.21.15.187:0/1022846884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60c8003730 msgr2=0x7f60c8003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.593+0000 7f60dbfff700 1 --2- 172.21.15.187:0/1022846884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60c8003730 0x7f60c8003b50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f60cc00ba00 tx=0x7f60cc008040).stop 2020-11-20T12:08:50.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.593+0000 7f60db7fe700 1 -- 172.21.15.187:0/1022846884 reap_dead start 2020-11-20T12:08:50.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.593+0000 7f60dbfff700 1 -- 172.21.15.187:0/1022846884 shutdown_connections 2020-11-20T12:08:50.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.593+0000 7f60dbfff700 1 -- 172.21.15.187:0/1022846884 >> 172.21.15.187:0/1022846884 conn(0x7f60dc0bab30 msgr2=0x7f60dc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:50.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.593+0000 7f60dbfff700 1 -- 172.21.15.187:0/1022846884 shutdown_connections 2020-11-20T12:08:50.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.593+0000 7f60dbfff700 1 -- 172.21.15.187:0/1022846884 wait complete. 2020-11-20T12:08:50.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.595+0000 7f60dbfff700 1 Processor -- start 2020-11-20T12:08:50.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.595+0000 7f60dbfff700 1 -- start start 2020-11-20T12:08:50.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60dbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60dc0b6d00 0x7f60dc09f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60dbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b70e0 0x7f60dc1470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60dbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60dc09e5d0 0x7f60dc140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60dbfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc147750 con 0x7f60dc0b6d00 2020-11-20T12:08:50.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60dbfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc1479f0 con 0x7f60dc09e5d0 2020-11-20T12:08:50.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60dbfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc147c90 con 0x7f60dc0b70e0 2020-11-20T12:08:50.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60db7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60dc09e5d0 0x7f60dc140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60db7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60dc09e5d0 0x7f60dc140d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44278/0 (socket says 172.21.15.187:44278) 2020-11-20T12:08:50.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.596+0000 7f60db7fe700 1 -- 172.21.15.187:0/3785007081 learned_addr learned my addr 172.21.15.187:0/3785007081 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:50.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60db7fe700 1 -- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b70e0 msgr2=0x7f60dc1470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60da7fc700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b70e0 0x7f60dc1470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60daffd700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60dc0b6d00 0x7f60dc09f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60db7fe700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b70e0 0x7f60dc1470b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60db7fe700 1 -- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60dc0b6d00 msgr2=0x7f60dc09f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60db7fe700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60dc0b6d00 0x7f60dc09f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60db7fe700 1 -- 172.21.15.187:0/3785007081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc09ea10 con 0x7f60dc09e5d0 2020-11-20T12:08:50.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60daffd700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60dc0b6d00 0x7f60dc09f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:50.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60da7fc700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b70e0 0x7f60dc1470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:50.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60db7fe700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60dc09e5d0 0x7f60dc140d50 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f60d4000f40 tx=0x7f60d4009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:50.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60c2ffd700 1 -- 172.21.15.187:0/3785007081 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d400b660 con 0x7f60dc09e5d0 2020-11-20T12:08:50.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60c2ffd700 1 -- 172.21.15.187:0/3785007081 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d400b7c0 con 0x7f60dc09e5d0 2020-11-20T12:08:50.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60dc141240 con 0x7f60dc09e5d0 2020-11-20T12:08:50.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60c2ffd700 1 -- 172.21.15.187:0/3785007081 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d400bb60 con 0x7f60dc09e5d0 2020-11-20T12:08:50.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.597+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60dc141730 con 0x7f60dc09e5d0 2020-11-20T12:08:50.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.598+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc1475a0 con 0x7f60dc09e5d0 2020-11-20T12:08:50.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.599+0000 7f60c2ffd700 1 -- 172.21.15.187:0/3785007081 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f60d401a030 con 0x7f60dc09e5d0 2020-11-20T12:08:50.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.600+0000 7f60c2ffd700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f60b8056d20 0x7f60b80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:50.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.600+0000 7f60c2ffd700 1 -- 172.21.15.187:0/3785007081 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f60d406fd70 con 0x7f60dc09e5d0 2020-11-20T12:08:50.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.600+0000 7f60daffd700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f60b8056d20 0x7f60b80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:50.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.601+0000 7f60daffd700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f60b8056d20 0x7f60b80591e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f60dc063890 tx=0x7f60cc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:50.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.604+0000 7f60c2ffd700 1 -- 172.21.15.187:0/3785007081 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f60d40450b0 con 0x7f60dc09e5d0 2020-11-20T12:08:50.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.788+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60dc061a20 con 0x7f60dc09e5d0 2020-11-20T12:08:50.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.790+0000 7f60c2ffd700 1 -- 172.21.15.187:0/3785007081 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16780 (secure 0 0 0) 0x7f60d40486d0 con 0x7f60dc09e5d0 2020-11-20T12:08:50.794 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:08:50.794 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":51,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:08:31.182203+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:08:28.213673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2020-11-20T12:08:28.856619+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:08:50.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f60b8056d20 msgr2=0x7f60b80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f60b8056d20 0x7f60b80591e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f60dc063890 tx=0x7f60cc008040).stop 2020-11-20T12:08:50.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60dc09e5d0 msgr2=0x7f60dc140d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:50.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60dc09e5d0 0x7f60dc140d50 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f60d4000f40 tx=0x7f60d4009040).stop 2020-11-20T12:08:50.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 shutdown_connections 2020-11-20T12:08:50.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f60b8056d20 0x7f60b80591e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60dc0b6d00 0x7f60dc09f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60dc09e5d0 0x7f60dc140d50 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 --2- 172.21.15.187:0/3785007081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60dc0b70e0 0x7f60dc1470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:50.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 >> 172.21.15.187:0/3785007081 conn(0x7f60dc0bab30 msgr2=0x7f60dc141cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:50.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 shutdown_connections 2020-11-20T12:08:50.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:50.792+0000 7f60dbfff700 1 -- 172.21.15.187:0/3785007081 wait complete. 2020-11-20T12:08:50.818 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:08:50.818 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:08:51.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.065+0000 7f8c5759e700 1 Processor -- start 2020-11-20T12:08:51.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.066+0000 7f8c5759e700 1 -- start start 2020-11-20T12:08:51.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.067+0000 7f8c5759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8c58065190 0x7f8c58065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:51.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.067+0000 7f8c5759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c580b13e0 con 0x7f8c5809ca90 2020-11-20T12:08:51.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.067+0000 7f8c5759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c58062c10 con 0x7f8c58065f90 2020-11-20T12:08:51.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.067+0000 7f8c5759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c5814ab30 con 0x7f8c58065190 2020-11-20T12:08:51.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.067+0000 7f8c5659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8c58065190 0x7f8c58065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:51.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.067+0000 7f8c5659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8c58065190 0x7f8c58065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44286/0 (socket says 172.21.15.187:44286) 2020-11-20T12:08:51.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.067+0000 7f8c5659c700 1 -- 172.21.15.187:0/2121038781 learned_addr learned my addr 172.21.15.187:0/2121038781 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:51.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.068+0000 7f8c5659c700 1 -- 172.21.15.187:0/2121038781 >> v1:172.21.15.187:6789/0 conn(0x7f8c58065f90 legacy=0x7f8c58143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:51.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.068+0000 7f8c5659c700 1 -- 172.21.15.187:0/2121038781 >> v1:172.21.15.5:6789/0 conn(0x7f8c5809ca90 legacy=0x7f8c5809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:51.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.068+0000 7f8c5659c700 1 -- 172.21.15.187:0/2121038781 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c580b1640 con 0x7f8c58065190 2020-11-20T12:08:51.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.068+0000 7f8c5659c700 1 --2- 172.21.15.187:0/2121038781 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c58065190 0x7f8c58065570 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f8c5814ac70 tx=0x7f8c48009290).ready entity=mon.1 client_cookie=c6a7f4865eb09cfe server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:51.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.068+0000 7f8c55d9b700 1 -- 172.21.15.187:0/2121038781 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4800bc10 con 0x7f8c58065190 2020-11-20T12:08:51.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c55d9b700 1 -- 172.21.15.187:0/2121038781 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4800bd70 con 0x7f8c58065190 2020-11-20T12:08:51.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c55d9b700 1 -- 172.21.15.187:0/2121038781 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c48011610 con 0x7f8c58065190 2020-11-20T12:08:51.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c5759e700 1 -- 172.21.15.187:0/2121038781 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c58065190 msgr2=0x7f8c58065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:51.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c5759e700 1 --2- 172.21.15.187:0/2121038781 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c58065190 0x7f8c58065570 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f8c5814ac70 tx=0x7f8c48009290).stop 2020-11-20T12:08:51.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c5759e700 1 -- 172.21.15.187:0/2121038781 shutdown_connections 2020-11-20T12:08:51.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c5759e700 1 --2- 172.21.15.187:0/2121038781 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c58065190 0x7f8c58065570 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:51.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c5759e700 1 -- 172.21.15.187:0/2121038781 >> 172.21.15.187:0/2121038781 conn(0x7f8c580bab30 msgr2=0x7f8c580baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:51.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c5759e700 1 -- 172.21.15.187:0/2121038781 shutdown_connections 2020-11-20T12:08:51.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.069+0000 7f8c5759e700 1 -- 172.21.15.187:0/2121038781 wait complete. 2020-11-20T12:08:51.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.071+0000 7f8c5759e700 1 Processor -- start 2020-11-20T12:08:51.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.071+0000 7f8c5759e700 1 -- start start 2020-11-20T12:08:51.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.071+0000 7f8c5759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c58065190 0x7f8c58068140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:51.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c5759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c58065f90 0x7f8c5809f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:51.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c5759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c5809ca90 0x7f8c58066590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:51.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c5759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580999e0 con 0x7f8c58065f90 2020-11-20T12:08:51.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c5759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58146b50 con 0x7f8c5809ca90 2020-11-20T12:08:51.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c5759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58146df0 con 0x7f8c58065190 2020-11-20T12:08:51.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c56d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c5809ca90 0x7f8c58066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:51.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c56d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c5809ca90 0x7f8c58066590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44296/0 (socket says 172.21.15.187:44296) 2020-11-20T12:08:51.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c56d9d700 1 -- 172.21.15.187:0/1991127542 learned_addr learned my addr 172.21.15.187:0/1991127542 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:51.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c5659c700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c58065190 0x7f8c58068140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:51.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c4dd9b700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c58065f90 0x7f8c5809f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:51.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c56d9d700 1 -- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c58065190 msgr2=0x7f8c58068140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:51.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c56d9d700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c58065190 0x7f8c58068140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:51.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c56d9d700 1 -- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c58065f90 msgr2=0x7f8c5809f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:51.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.072+0000 7f8c56d9d700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c58065f90 0x7f8c5809f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:51.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.073+0000 7f8c56d9d700 1 -- 172.21.15.187:0/1991127542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c58147090 con 0x7f8c5809ca90 2020-11-20T12:08:51.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.073+0000 7f8c5659c700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c58065190 0x7f8c58068140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:51.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.073+0000 7f8c56d9d700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c5809ca90 0x7f8c58066590 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f8c50009690 tx=0x7f8c50006d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:51.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.073+0000 7f8c4effd700 1 -- 172.21.15.187:0/1991127542 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5000d560 con 0x7f8c5809ca90 2020-11-20T12:08:51.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.073+0000 7f8c4effd700 1 -- 172.21.15.187:0/1991127542 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5000d6e0 con 0x7f8c5809ca90 2020-11-20T12:08:51.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.073+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5809f900 con 0x7f8c5809ca90 2020-11-20T12:08:51.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.073+0000 7f8c4effd700 1 -- 172.21.15.187:0/1991127542 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5000daa0 con 0x7f8c5809ca90 2020-11-20T12:08:51.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.073+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c58146640 con 0x7f8c5809ca90 2020-11-20T12:08:51.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.074+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c58061a20 con 0x7f8c5809ca90 2020-11-20T12:08:51.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.075+0000 7f8c4effd700 1 -- 172.21.15.187:0/1991127542 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8c5000f070 con 0x7f8c5809ca90 2020-11-20T12:08:51.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.076+0000 7f8c4effd700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c34056d20 0x7f8c340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:51.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.076+0000 7f8c4effd700 1 -- 172.21.15.187:0/1991127542 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8c5006fa00 con 0x7f8c5809ca90 2020-11-20T12:08:51.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.076+0000 7f8c5659c700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c34056d20 0x7f8c340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:51.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.076+0000 7f8c5659c700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c34056d20 0x7f8c340591e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f8c48006e70 tx=0x7f8c48016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:51.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.079+0000 7f8c4effd700 1 -- 172.21.15.187:0/1991127542 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8c50044e50 con 0x7f8c5809ca90 2020-11-20T12:08:51.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:51.327+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8c5809d630 con 0x7f8c5809ca90 2020-11-20T12:08:52.095 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:08:52.090+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:08:52.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.109+0000 7f8c4effd700 1 -- 172.21.15.187:0/1991127542 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8c50048470 con 0x7f8c5809ca90 2020-11-20T12:08:52.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c34056d20 msgr2=0x7f8c340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:52.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c34056d20 0x7f8c340591e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f8c48006e70 tx=0x7f8c48016000).stop 2020-11-20T12:08:52.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c5809ca90 msgr2=0x7f8c58066590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:52.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c5809ca90 0x7f8c58066590 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f8c50009690 tx=0x7f8c50006d10).stop 2020-11-20T12:08:52.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 shutdown_connections 2020-11-20T12:08:52.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c34056d20 0x7f8c340591e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c5809ca90 0x7f8c58066590 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c58065190 0x7f8c58068140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 --2- 172.21.15.187:0/1991127542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c58065f90 0x7f8c5809f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 >> 172.21.15.187:0/1991127542 conn(0x7f8c580bab30 msgr2=0x7f8c581445e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:52.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.113+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 shutdown_connections 2020-11-20T12:08:52.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.114+0000 7f8c5759e700 1 -- 172.21.15.187:0/1991127542 wait complete. 2020-11-20T12:08:52.121 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:08:52.133 INFO:tasks.ceph.mds.a.smithi187.stderr: -2822> 2020-11-20T12:08:34.297+0000 7f23f8f7d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:08:52.143 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:08:52.203 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:08:52.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.366+0000 7f58c7fff700 1 Processor -- start 2020-11-20T12:08:52.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.366+0000 7f58c7fff700 1 -- start start 2020-11-20T12:08:52.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.367+0000 7f58c7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f58c81484d0 0x7f58c8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:52.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.367+0000 7f58c7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c80622c0 0x7f58c80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:52.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.367+0000 7f58c7fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c80b1300 con 0x7f58c8066f70 2020-11-20T12:08:52.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.367+0000 7f58c7fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c8067730 con 0x7f58c81484d0 2020-11-20T12:08:52.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.367+0000 7f58c7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c80b15a0 con 0x7f58c80622c0 2020-11-20T12:08:52.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.368+0000 7f58c6ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f58c81484d0 0x7f58c8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:52.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.368+0000 7f58c6ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f58c81484d0 0x7f58c8061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44314/0 (socket says 172.21.15.187:44314) 2020-11-20T12:08:52.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.368+0000 7f58c6ffd700 1 -- 172.21.15.187:0/1200913376 learned_addr learned my addr 172.21.15.187:0/1200913376 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:52.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.368+0000 7f58c67fc700 1 --2- 172.21.15.187:0/1200913376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c80622c0 0x7f58c80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:52.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.369+0000 7f58c6ffd700 1 -- 172.21.15.187:0/1200913376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c80622c0 msgr2=0x7f58c80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:52.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.369+0000 7f58c6ffd700 1 --2- 172.21.15.187:0/1200913376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c80622c0 0x7f58c80b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.369+0000 7f58c6ffd700 1 -- 172.21.15.187:0/1200913376 >> v1:172.21.15.5:6789/0 conn(0x7f58c8066f70 legacy=0x7f58c80b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:52.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.369+0000 7f58c6ffd700 1 -- 172.21.15.187:0/1200913376 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c80b1840 con 0x7f58c81484d0 2020-11-20T12:08:52.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.369+0000 7f58c6ffd700 1 --2- 172.21.15.187:0/1200913376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81484d0 0x7f58c8061dd0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f58b8006fd0 tx=0x7f58b8006eb0).ready entity=mon.1 client_cookie=2d92ae733babf849 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:52.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.369+0000 7f58c5ffb700 1 -- 172.21.15.187:0/1200913376 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b80098d0 con 0x7f58c81484d0 2020-11-20T12:08:52.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.369+0000 7f58c5ffb700 1 -- 172.21.15.187:0/1200913376 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b8009a30 con 0x7f58c81484d0 2020-11-20T12:08:52.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c5ffb700 1 -- 172.21.15.187:0/1200913376 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b8009db0 con 0x7f58c81484d0 2020-11-20T12:08:52.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c7fff700 1 -- 172.21.15.187:0/1200913376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81484d0 msgr2=0x7f58c8061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:52.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1200913376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81484d0 0x7f58c8061dd0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f58b8006fd0 tx=0x7f58b8006eb0).stop 2020-11-20T12:08:52.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c7fff700 1 -- 172.21.15.187:0/1200913376 shutdown_connections 2020-11-20T12:08:52.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1200913376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81484d0 0x7f58c8061dd0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1200913376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c80622c0 0x7f58c80b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c7fff700 1 -- 172.21.15.187:0/1200913376 >> 172.21.15.187:0/1200913376 conn(0x7f58c80bab30 msgr2=0x7f58c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:52.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c7fff700 1 -- 172.21.15.187:0/1200913376 shutdown_connections 2020-11-20T12:08:52.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.370+0000 7f58c7fff700 1 -- 172.21.15.187:0/1200913376 wait complete. 2020-11-20T12:08:52.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.372+0000 7f58c7fff700 1 Processor -- start 2020-11-20T12:08:52.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.372+0000 7f58c7fff700 1 -- start start 2020-11-20T12:08:52.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.373+0000 7f58c7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f58c80622c0 0x7f58c80b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:52.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.373+0000 7f58c7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c8066f70 0x7f58c809a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:52.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81476c0 0x7f58c809ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:52.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c80990a0 con 0x7f58c80622c0 2020-11-20T12:08:52.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c8099340 con 0x7f58c81476c0 2020-11-20T12:08:52.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c809bfc0 con 0x7f58c8066f70 2020-11-20T12:08:52.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81476c0 0x7f58c809ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:52.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81476c0 0x7f58c809ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44322/0 (socket says 172.21.15.187:44322) 2020-11-20T12:08:52.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c77fe700 1 -- 172.21.15.187:0/1087138304 learned_addr learned my addr 172.21.15.187:0/1087138304 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:52.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c67fc700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c8066f70 0x7f58c809a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:52.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.374+0000 7f58c6ffd700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f58c80622c0 0x7f58c80b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:52.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58c77fe700 1 -- 172.21.15.187:0/1087138304 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c8066f70 msgr2=0x7f58c809a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:52.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58c77fe700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c8066f70 0x7f58c809a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58c77fe700 1 -- 172.21.15.187:0/1087138304 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f58c80622c0 msgr2=0x7f58c80b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:52.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58c77fe700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f58c80622c0 0x7f58c80b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:52.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58c77fe700 1 -- 172.21.15.187:0/1087138304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c80b4560 con 0x7f58c81476c0 2020-11-20T12:08:52.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58c77fe700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81476c0 0x7f58c809ab60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f58c000ca60 tx=0x7f58c0007f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:52.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58aeffd700 1 -- 172.21.15.187:0/1087138304 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c000a8c0 con 0x7f58c81476c0 2020-11-20T12:08:52.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58aeffd700 1 -- 172.21.15.187:0/1087138304 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c000aa20 con 0x7f58c81476c0 2020-11-20T12:08:52.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58aeffd700 1 -- 172.21.15.187:0/1087138304 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c000ada0 con 0x7f58c81476c0 2020-11-20T12:08:52.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c8142910 con 0x7f58c81476c0 2020-11-20T12:08:52.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.375+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c8144e90 con 0x7f58c81476c0 2020-11-20T12:08:52.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.377+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c809c180 con 0x7f58c81476c0 2020-11-20T12:08:52.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.377+0000 7f58aeffd700 1 -- 172.21.15.187:0/1087138304 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f58c0019030 con 0x7f58c81476c0 2020-11-20T12:08:52.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.378+0000 7f58aeffd700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58a4056d20 0x7f58a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:52.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.378+0000 7f58aeffd700 1 -- 172.21.15.187:0/1087138304 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f58c006f830 con 0x7f58c81476c0 2020-11-20T12:08:52.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.378+0000 7f58c6ffd700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58a4056d20 0x7f58a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:52.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.379+0000 7f58c6ffd700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58a4056d20 0x7f58a40591e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f58c8066380 tx=0x7f58b8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:52.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.382+0000 7f58aeffd700 1 -- 172.21.15.187:0/1087138304 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f58c0043fd0 con 0x7f58c81476c0 2020-11-20T12:08:52.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:52.624+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f58c8061a20 con 0x7f58c81476c0 2020-11-20T12:08:53.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.344+0000 7f58aeffd700 1 -- 172.21.15.187:0/1087138304 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f58c0044ce0 con 0x7f58c81476c0 2020-11-20T12:08:53.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58a4056d20 msgr2=0x7f58a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:53.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58a4056d20 0x7f58a40591e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f58c8066380 tx=0x7f58b8015000).stop 2020-11-20T12:08:53.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81476c0 msgr2=0x7f58c809ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:53.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81476c0 0x7f58c809ab60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f58c000ca60 tx=0x7f58c0007f40).stop 2020-11-20T12:08:53.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 shutdown_connections 2020-11-20T12:08:53.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58a4056d20 0x7f58a40591e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:53.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f58c80622c0 0x7f58c80b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:53.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c81476c0 0x7f58c809ab60 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:53.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 --2- 172.21.15.187:0/1087138304 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f58c8066f70 0x7f58c809a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:53.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.348+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 >> 172.21.15.187:0/1087138304 conn(0x7f58c80bab30 msgr2=0x7f58c80b9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:53.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.349+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 shutdown_connections 2020-11-20T12:08:53.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.349+0000 7f58c7fff700 1 -- 172.21.15.187:0/1087138304 wait complete. 2020-11-20T12:08:53.381 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:08:53.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.631+0000 7f520f0b8700 1 Processor -- start 2020-11-20T12:08:53.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.631+0000 7f520f0b8700 1 -- start start 2020-11-20T12:08:53.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.633+0000 7f520f0b8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f52080bfbe0 0x7f52081480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:53.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.633+0000 7f520f0b8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520814d110 con 0x7f520809f560 2020-11-20T12:08:53.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.633+0000 7f520f0b8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520814cbd0 con 0x7f52080bf800 2020-11-20T12:08:53.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.633+0000 7f520f0b8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520814c9d0 con 0x7f52080bfbe0 2020-11-20T12:08:53.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.634+0000 7f520e0b6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f52080bf800 0x7f520814c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40932/0 (socket says 172.21.15.187:40932) 2020-11-20T12:08:53.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.634+0000 7f520d8b5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f52080bfbe0 0x7f52081480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:53.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.634+0000 7f520e0b6700 1 -- 172.21.15.187:0/635629033 learned_addr learned my addr 172.21.15.187:0/635629033 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:53.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.634+0000 7f520d8b5700 1 -- 172.21.15.187:0/635629033 >> v1:172.21.15.187:6789/0 conn(0x7f52080bf800 legacy=0x7f520814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:53.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.634+0000 7f520d8b5700 1 -- 172.21.15.187:0/635629033 >> v1:172.21.15.5:6789/0 conn(0x7f520809f560 legacy=0x7f52080a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:53.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.634+0000 7f520d0b4700 1 -- 172.21.15.187:0/635629033 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436583153 0 0) 0x7f520814cbd0 con 0x7f52080bf800 2020-11-20T12:08:53.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.634+0000 7f520d8b5700 1 -- 172.21.15.187:0/635629033 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520814ce70 con 0x7f52080bfbe0 2020-11-20T12:08:53.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.635+0000 7f520d8b5700 1 --2- 172.21.15.187:0/635629033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bfbe0 0x7f52081480c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f51f800b9a0 tx=0x7f51f8006df0).ready entity=mon.1 client_cookie=9ae2c8e7d25f357b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.635+0000 7f520d0b4700 1 -- 172.21.15.187:0/635629033 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8009800 con 0x7f52080bfbe0 2020-11-20T12:08:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.635+0000 7f520d0b4700 1 -- 172.21.15.187:0/635629033 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8009960 con 0x7f52080bfbe0 2020-11-20T12:08:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.635+0000 7f520d0b4700 1 -- 172.21.15.187:0/635629033 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8009ce0 con 0x7f52080bfbe0 2020-11-20T12:08:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.635+0000 7f520f0b8700 1 -- 172.21.15.187:0/635629033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bfbe0 msgr2=0x7f52081480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:53.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.635+0000 7f520f0b8700 1 --2- 172.21.15.187:0/635629033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bfbe0 0x7f52081480c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f51f800b9a0 tx=0x7f51f8006df0).stop 2020-11-20T12:08:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.636+0000 7f520f0b8700 1 -- 172.21.15.187:0/635629033 shutdown_connections 2020-11-20T12:08:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.636+0000 7f520f0b8700 1 --2- 172.21.15.187:0/635629033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bfbe0 0x7f52081480c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.636+0000 7f520f0b8700 1 -- 172.21.15.187:0/635629033 >> 172.21.15.187:0/635629033 conn(0x7f5208068370 msgr2=0x7f52080bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.636+0000 7f520f0b8700 1 -- 172.21.15.187:0/635629033 shutdown_connections 2020-11-20T12:08:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.636+0000 7f520f0b8700 1 -- 172.21.15.187:0/635629033 wait complete. 2020-11-20T12:08:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.637+0000 7f520f0b8700 1 Processor -- start 2020-11-20T12:08:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.637+0000 7f520f0b8700 1 -- start start 2020-11-20T12:08:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.638+0000 7f520f0b8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520809f560 0x7f5208145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.638+0000 7f520f0b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bf800 0x7f5208134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:53.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.638+0000 7f520f0b8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5208145d80 0x7f52081353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:53.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.638+0000 7f520f0b8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5208144630 con 0x7f520809f560 2020-11-20T12:08:53.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.638+0000 7f520f0b8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52081448d0 con 0x7f52080bf800 2020-11-20T12:08:53.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.638+0000 7f520f0b8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5208144b70 con 0x7f5208145d80 2020-11-20T12:08:53.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bf800 0x7f5208134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:53.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bf800 0x7f5208134c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44334/0 (socket says 172.21.15.187:44334) 2020-11-20T12:08:53.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 -- 172.21.15.187:0/2071943472 learned_addr learned my addr 172.21.15.187:0/2071943472 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:53.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520e0b6700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520809f560 0x7f5208145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:53.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 -- 172.21.15.187:0/2071943472 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5208145d80 msgr2=0x7f52081353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5208145d80 0x7f52081353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 -- 172.21.15.187:0/2071943472 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520809f560 msgr2=0x7f5208145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520809f560 0x7f5208145010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:53.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 -- 172.21.15.187:0/2071943472 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5208135170 con 0x7f52080bf800 2020-11-20T12:08:53.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520e0b6700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520809f560 0x7f5208145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:53.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.639+0000 7f520d8b5700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bf800 0x7f5208134c80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f51f800b150 tx=0x7f51f8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:53.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.640+0000 7f51f67fc700 1 -- 172.21.15.187:0/2071943472 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f801d6a0 con 0x7f52080bf800 2020-11-20T12:08:53.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.640+0000 7f51f67fc700 1 -- 172.21.15.187:0/2071943472 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f801d800 con 0x7f52080bf800 2020-11-20T12:08:53.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.640+0000 7f51f67fc700 1 -- 172.21.15.187:0/2071943472 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f801db80 con 0x7f52080bf800 2020-11-20T12:08:53.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.640+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5208135e40 con 0x7f52080bf800 2020-11-20T12:08:53.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.640+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5208147a80 con 0x7f52080bf800 2020-11-20T12:08:53.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.641+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5208080050 con 0x7f52080bf800 2020-11-20T12:08:53.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.641+0000 7f51f67fc700 1 -- 172.21.15.187:0/2071943472 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f51f8025030 con 0x7f52080bf800 2020-11-20T12:08:53.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.642+0000 7f51f67fc700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e0056d20 0x7f51e00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:53.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.642+0000 7f51f67fc700 1 -- 172.21.15.187:0/2071943472 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f51f8071b40 con 0x7f52080bf800 2020-11-20T12:08:53.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.642+0000 7f520e0b6700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e0056d20 0x7f51e00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:53.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.643+0000 7f520e0b6700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e0056d20 0x7f51e00591e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f5204001500 tx=0x7f5204009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:53.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.646+0000 7f51f67fc700 1 -- 172.21.15.187:0/2071943472 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f51f8046ee0 con 0x7f52080bf800 2020-11-20T12:08:53.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:53.834+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5208144d30 con 0x7f52080bf800 2020-11-20T12:08:54.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.356+0000 7f51f67fc700 1 -- 172.21.15.187:0/2071943472 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v53) v1 ==== 181+0+0 (secure 0 0 0) 0x7f51f8023030 con 0x7f52080bf800 2020-11-20T12:08:54.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e0056d20 msgr2=0x7f51e00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:54.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e0056d20 0x7f51e00591e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f5204001500 tx=0x7f5204009040).stop 2020-11-20T12:08:54.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bf800 msgr2=0x7f5208134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:54.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bf800 0x7f5208134c80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f51f800b150 tx=0x7f51f8015000).stop 2020-11-20T12:08:54.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 shutdown_connections 2020-11-20T12:08:54.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e0056d20 0x7f51e00591e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:54.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520809f560 0x7f5208145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:54.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5208145d80 0x7f52081353a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:08:54.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 --2- 172.21.15.187:0/2071943472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52080bf800 0x7f5208134c80 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:54.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.360+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 >> 172.21.15.187:0/2071943472 conn(0x7f5208068370 msgr2=0x7f52080bbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:54.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.361+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 shutdown_connections 2020-11-20T12:08:54.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.361+0000 7f520f0b8700 1 -- 172.21.15.187:0/2071943472 wait complete. 2020-11-20T12:08:54.368 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:08:54.392 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:08:54.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.634+0000 7f35115d0700 1 Processor -- start 2020-11-20T12:08:54.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.634+0000 7f35115d0700 1 -- start start 2020-11-20T12:08:54.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.635+0000 7f35115d0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f350c0683f0 0x7f350c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:54.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.635+0000 7f35115d0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c14c170 con 0x7f350c145cc0 2020-11-20T12:08:54.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.635+0000 7f35115d0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c14c410 con 0x7f350c1487e0 2020-11-20T12:08:54.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.635+0000 7f35115d0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c14c6b0 con 0x7f350c0683f0 2020-11-20T12:08:54.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.636+0000 7f3510dcf700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f350c1487e0 0x7f350c148c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40950/0 (socket says 172.21.15.187:40950) 2020-11-20T12:08:54.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.636+0000 7f3510dcf700 1 -- 172.21.15.187:0/3788116671 learned_addr learned my addr 172.21.15.187:0/3788116671 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:08:54.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.636+0000 7f35037fe700 1 --2- 172.21.15.187:0/3788116671 >> v2:172.21.15.5:3300/0 conn(0x7f350c0683f0 0x7f350c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:54.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.637+0000 7f350b7fe700 1 -- 172.21.15.187:0/3788116671 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36989702 0 0) 0x7f350c14c410 con 0x7f350c1487e0 2020-11-20T12:08:54.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.637+0000 7f35037fe700 1 -- 172.21.15.187:0/3788116671 >> v1:172.21.15.187:6789/0 conn(0x7f350c1487e0 legacy=0x7f350c148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:54.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.637+0000 7f35037fe700 1 -- 172.21.15.187:0/3788116671 >> v1:172.21.15.5:6789/0 conn(0x7f350c145cc0 legacy=0x7f350c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:54.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.637+0000 7f35037fe700 1 -- 172.21.15.187:0/3788116671 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c0658a0 con 0x7f350c0683f0 2020-11-20T12:08:54.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.638+0000 7f35037fe700 1 --2- 172.21.15.187:0/3788116671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c0683f0 0x7f350c066040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f34f400b9a0 tx=0x7f34f4006df0).ready entity=mon.0 client_cookie=874767c224be9a62 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:54.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.638+0000 7f350b7fe700 1 -- 172.21.15.187:0/3788116671 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f40097b0 con 0x7f350c0683f0 2020-11-20T12:08:54.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f350b7fe700 1 -- 172.21.15.187:0/3788116671 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f4009910 con 0x7f350c0683f0 2020-11-20T12:08:54.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f350b7fe700 1 -- 172.21.15.187:0/3788116671 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f4009c90 con 0x7f350c0683f0 2020-11-20T12:08:54.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f35115d0700 1 -- 172.21.15.187:0/3788116671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c0683f0 msgr2=0x7f350c066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:54.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f35115d0700 1 --2- 172.21.15.187:0/3788116671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c0683f0 0x7f350c066040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f34f400b9a0 tx=0x7f34f4006df0).stop 2020-11-20T12:08:54.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f35115d0700 1 -- 172.21.15.187:0/3788116671 shutdown_connections 2020-11-20T12:08:54.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f35115d0700 1 --2- 172.21.15.187:0/3788116671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c0683f0 0x7f350c066040 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:54.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f35115d0700 1 -- 172.21.15.187:0/3788116671 >> 172.21.15.187:0/3788116671 conn(0x7f350c0bab30 msgr2=0x7f350c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:54.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f35115d0700 1 -- 172.21.15.187:0/3788116671 shutdown_connections 2020-11-20T12:08:54.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.639+0000 7f35115d0700 1 -- 172.21.15.187:0/3788116671 wait complete. 2020-11-20T12:08:54.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.641+0000 7f35115d0700 1 Processor -- start 2020-11-20T12:08:54.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.641+0000 7f35115d0700 1 -- start start 2020-11-20T12:08:54.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f35115d0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f350c0683f0 0x7f350c0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:54.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f35115d0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c145cc0 0x7f350c15c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:54.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f35115d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f350c1487e0 0x7f350c14a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:54.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f35115d0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0b0920 con 0x7f350c145cc0 2020-11-20T12:08:54.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f35115d0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0b0bc0 con 0x7f350c1487e0 2020-11-20T12:08:54.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f35115d0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0b0e60 con 0x7f350c0683f0 2020-11-20T12:08:54.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f3510dcf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f350c1487e0 0x7f350c14a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:54.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f3510dcf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f350c1487e0 0x7f350c14a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44350/0 (socket says 172.21.15.187:44350) 2020-11-20T12:08:54.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.642+0000 7f3510dcf700 1 -- 172.21.15.187:0/3513428894 learned_addr learned my addr 172.21.15.187:0/3513428894 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:08:54.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f35037fe700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c145cc0 0x7f350c15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:54.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f350bfff700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f350c0683f0 0x7f350c0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:54.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f3510dcf700 1 -- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f350c0683f0 msgr2=0x7f350c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:54.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f3510dcf700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f350c0683f0 0x7f350c0b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:54.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f3510dcf700 1 -- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c145cc0 msgr2=0x7f350c15c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:54.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f3510dcf700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c145cc0 0x7f350c15c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:54.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f3510dcf700 1 -- 172.21.15.187:0/3513428894 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c14aec0 con 0x7f350c1487e0 2020-11-20T12:08:54.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f350bfff700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f350c0683f0 0x7f350c0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:54.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f3510dcf700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f350c1487e0 0x7f350c14a9d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f350400bae0 tx=0x7f35040090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:54.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.643+0000 7f3508ff9700 1 -- 172.21.15.187:0/3513428894 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350400c830 con 0x7f350c1487e0 2020-11-20T12:08:54.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.644+0000 7f3508ff9700 1 -- 172.21.15.187:0/3513428894 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350400c990 con 0x7f350c1487e0 2020-11-20T12:08:54.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.644+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c0b05a0 con 0x7f350c1487e0 2020-11-20T12:08:54.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.644+0000 7f3508ff9700 1 -- 172.21.15.187:0/3513428894 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350400cb20 con 0x7f350c1487e0 2020-11-20T12:08:54.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.644+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c144c30 con 0x7f350c1487e0 2020-11-20T12:08:54.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.645+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c0b0790 con 0x7f350c1487e0 2020-11-20T12:08:54.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.645+0000 7f3508ff9700 1 -- 172.21.15.187:0/3513428894 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f350401d3a0 con 0x7f350c1487e0 2020-11-20T12:08:54.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.646+0000 7f3508ff9700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f34e8056d20 0x7f34e80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:54.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.646+0000 7f3508ff9700 1 -- 172.21.15.187:0/3513428894 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f350406f770 con 0x7f350c1487e0 2020-11-20T12:08:54.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.646+0000 7f350bfff700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f34e8056d20 0x7f34e80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:54.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.647+0000 7f350bfff700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f34e8056d20 0x7f34e80591e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f34fc0014d0 tx=0x7f34fc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:54.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.650+0000 7f3508ff9700 1 -- 172.21.15.187:0/3513428894 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3504044cc0 con 0x7f350c1487e0 2020-11-20T12:08:54.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:54.835+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f350c0b1020 con 0x7f350c1487e0 2020-11-20T12:08:55.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.360+0000 7f3508ff9700 1 -- 172.21.15.187:0/3513428894 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v54) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3504019030 con 0x7f350c1487e0 2020-11-20T12:08:55.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.363+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f34e8056d20 msgr2=0x7f34e80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:55.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.363+0000 7f35115d0700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f34e8056d20 0x7f34e80591e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f34fc0014d0 tx=0x7f34fc009040).stop 2020-11-20T12:08:55.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.364+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f350c1487e0 msgr2=0x7f350c14a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:55.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.364+0000 7f35115d0700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f350c1487e0 0x7f350c14a9d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f350400bae0 tx=0x7f35040090f0).stop 2020-11-20T12:08:55.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.364+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 shutdown_connections 2020-11-20T12:08:55.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.364+0000 7f35115d0700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f34e8056d20 0x7f34e80591e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:55.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.364+0000 7f35115d0700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f350c1487e0 0x7f350c14a9d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:55.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.364+0000 7f35115d0700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f350c0683f0 0x7f350c0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:55.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.364+0000 7f35115d0700 1 --2- 172.21.15.187:0/3513428894 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f350c145cc0 0x7f350c15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:55.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.364+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 >> 172.21.15.187:0/3513428894 conn(0x7f350c0bab30 msgr2=0x7f350c14b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:55.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.365+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 shutdown_connections 2020-11-20T12:08:55.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:08:55.365+0000 7f35115d0700 1 -- 172.21.15.187:0/3513428894 wait complete. 2020-11-20T12:08:55.371 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:08:55.399 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:08:55.400 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:08:55.400 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:08:55.400 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:08:55.400 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:08:55.401 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:08:55.401 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:08:55.401 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:08:55.432 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:08:55.432 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:08:55.485 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:08:55.486 DEBUG:teuthology.parallel:result is None 2020-11-20T12:08:55.673 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:09:00.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2020-11-20T12:09:01.427 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:09:01.427 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:09:01.439 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:09:01.440 DEBUG:teuthology.parallel:result is None 2020-11-20T12:09:01.441 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:09:01.640 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:09:01.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.700+0000 7f954d0f9700 1 Processor -- start 2020-11-20T12:09:01.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.700+0000 7f954d0f9700 1 -- start start 2020-11-20T12:09:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.702+0000 7f954d0f9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f95480675d0 0x7f954814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.702+0000 7f954d0f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f9548147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.702+0000 7f954d0f9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954814be70 con 0x7f95480ba9b0 2020-11-20T12:09:01.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.702+0000 7f954d0f9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954814c110 con 0x7f95480675d0 2020-11-20T12:09:01.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.702+0000 7f954d0f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954814c3b0 con 0x7f95480679b0 2020-11-20T12:09:01.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.703+0000 7f9547fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f95480675d0 0x7f954814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:01.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.703+0000 7f9547fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f95480675d0 0x7f954814c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:36638/0 (socket says 172.21.15.187:36638) 2020-11-20T12:09:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.703+0000 7f954c8f8700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f95480ba9b0 0x7f95481481e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40984/0 (socket says 172.21.15.187:40984) 2020-11-20T12:09:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.703+0000 7f9547fff700 1 -- 172.21.15.187:0/3126212758 learned_addr learned my addr 172.21.15.187:0/3126212758 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.703+0000 7f953f7fe700 1 --2- 172.21.15.187:0/3126212758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f9548147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.703+0000 7f95477fe700 1 -- 172.21.15.187:0/3126212758 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010430953 0 0) 0x7f954814be70 con 0x7f95480ba9b0 2020-11-20T12:09:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.703+0000 7f95477fe700 1 -- 172.21.15.187:0/3126212758 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9528003240 con 0x7f95480ba9b0 2020-11-20T12:09:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.703+0000 7f953f7fe700 1 -- 172.21.15.187:0/3126212758 >> v2:172.21.15.5:3300/0 conn(0x7f95480675d0 msgr2=0x7f954814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.704+0000 7f953f7fe700 1 --2- 172.21.15.187:0/3126212758 >> v2:172.21.15.5:3300/0 conn(0x7f95480675d0 0x7f954814c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.704+0000 7f953f7fe700 1 -- 172.21.15.187:0/3126212758 >> v1:172.21.15.187:6789/0 conn(0x7f95480ba9b0 legacy=0x7f95481481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.704+0000 7f953f7fe700 1 -- 172.21.15.187:0/3126212758 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954814cd40 con 0x7f95480679b0 2020-11-20T12:09:01.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.704+0000 7f953f7fe700 1 --2- 172.21.15.187:0/3126212758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f9548147cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f953000b9a0 tx=0x7f9530006df0).ready entity=mon.2 client_cookie=12033eb8de09209b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:01.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.704+0000 7f95477fe700 1 -- 172.21.15.187:0/3126212758 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530009680 con 0x7f95480679b0 2020-11-20T12:09:01.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f95477fe700 1 -- 172.21.15.187:0/3126212758 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95300097e0 con 0x7f95480679b0 2020-11-20T12:09:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f95477fe700 1 -- 172.21.15.187:0/3126212758 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530009b60 con 0x7f95480679b0 2020-11-20T12:09:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f954d0f9700 1 -- 172.21.15.187:0/3126212758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 msgr2=0x7f9548147cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f954d0f9700 1 --2- 172.21.15.187:0/3126212758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f9548147cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f953000b9a0 tx=0x7f9530006df0).stop 2020-11-20T12:09:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f954d0f9700 1 -- 172.21.15.187:0/3126212758 shutdown_connections 2020-11-20T12:09:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f954d0f9700 1 --2- 172.21.15.187:0/3126212758 >> v2:172.21.15.5:3300/0 conn(0x7f95480675d0 0x7f954814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:01.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f954d0f9700 1 --2- 172.21.15.187:0/3126212758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f9548147cf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:01.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f954d0f9700 1 -- 172.21.15.187:0/3126212758 >> 172.21.15.187:0/3126212758 conn(0x7f95480bce50 msgr2=0x7f95480bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:01.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f954d0f9700 1 -- 172.21.15.187:0/3126212758 shutdown_connections 2020-11-20T12:09:01.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.705+0000 7f954d0f9700 1 -- 172.21.15.187:0/3126212758 wait complete. 2020-11-20T12:09:01.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.707+0000 7f954d0f9700 1 Processor -- start 2020-11-20T12:09:01.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.708+0000 7f954d0f9700 1 -- start start 2020-11-20T12:09:01.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f954d0f9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95480675d0 0x7f9548157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:01.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f954d0f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f954809dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:01.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f954d0f9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95480ba9b0 0x7f954815ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:01.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f954d0f9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954809ff80 con 0x7f95480ba9b0 2020-11-20T12:09:01.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f954d0f9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95480a0220 con 0x7f95480675d0 2020-11-20T12:09:01.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f954d0f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95480a04c0 con 0x7f95480679b0 2020-11-20T12:09:01.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f9547fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95480675d0 0x7f9548157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:01.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f9547fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95480675d0 0x7f9548157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44380/0 (socket says 172.21.15.187:44380) 2020-11-20T12:09:01.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.709+0000 7f9547fff700 1 -- 172.21.15.187:0/2823223627 learned_addr learned my addr 172.21.15.187:0/2823223627 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:01.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f953f7fe700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f954809dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:01.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f954c8f8700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95480ba9b0 0x7f954815ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:01.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9547fff700 1 -- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 msgr2=0x7f954809dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:01.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9547fff700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f954809dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:01.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9547fff700 1 -- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95480ba9b0 msgr2=0x7f954815ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:01.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9547fff700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95480ba9b0 0x7f954815ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:01.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9547fff700 1 -- 172.21.15.187:0/2823223627 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954809f770 con 0x7f95480675d0 2020-11-20T12:09:01.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f953f7fe700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f954809dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:01.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9547fff700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95480675d0 0x7f9548157e50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f953800f840 tx=0x7f9538008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:01.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9544ff9700 1 -- 172.21.15.187:0/2823223627 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953800d6e0 con 0x7f95480675d0 2020-11-20T12:09:01.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9544ff9700 1 -- 172.21.15.187:0/2823223627 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953800d840 con 0x7f95480675d0 2020-11-20T12:09:01.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954809dfb0 con 0x7f95480675d0 2020-11-20T12:09:01.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f9544ff9700 1 -- 172.21.15.187:0/2823223627 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953800dbc0 con 0x7f95480675d0 2020-11-20T12:09:01.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.710+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95480a1e40 con 0x7f95480675d0 2020-11-20T12:09:01.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.712+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95480bcb60 con 0x7f95480675d0 2020-11-20T12:09:01.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.712+0000 7f9544ff9700 1 -- 172.21.15.187:0/2823223627 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9538024070 con 0x7f95480675d0 2020-11-20T12:09:01.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.713+0000 7f9544ff9700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d20 0x7f95240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:01.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.713+0000 7f9544ff9700 1 -- 172.21.15.187:0/2823223627 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9538070350 con 0x7f95480675d0 2020-11-20T12:09:01.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.713+0000 7f953f7fe700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d20 0x7f95240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:01.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.714+0000 7f953f7fe700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d20 0x7f95240591e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f954815c530 tx=0x7f9530015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:01.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.717+0000 7f9544ff9700 1 -- 172.21.15.187:0/2823223627 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9538045970 con 0x7f95480675d0 2020-11-20T12:09:01.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:01.921+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f95480446d0 con 0x7f95480675d0 2020-11-20T12:09:02.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.113+0000 7f9544ff9700 1 -- 172.21.15.187:0/2823223627 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v55) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9538018020 con 0x7f95480675d0 2020-11-20T12:09:02.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.116+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d20 msgr2=0x7f95240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.116+0000 7f954d0f9700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d20 0x7f95240591e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f954815c530 tx=0x7f9530015000).stop 2020-11-20T12:09:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.116+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95480675d0 msgr2=0x7f9548157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.116+0000 7f954d0f9700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95480675d0 0x7f9548157e50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f953800f840 tx=0x7f9538008040).stop 2020-11-20T12:09:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.116+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 shutdown_connections 2020-11-20T12:09:02.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.117+0000 7f954d0f9700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9524056d20 0x7f95240591e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.117+0000 7f954d0f9700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95480ba9b0 0x7f954815ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.117+0000 7f954d0f9700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95480675d0 0x7f9548157e50 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.117+0000 7f954d0f9700 1 --2- 172.21.15.187:0/2823223627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95480679b0 0x7f954809dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.117+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 >> 172.21.15.187:0/2823223627 conn(0x7f95480bce50 msgr2=0x7f954814a840 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:02.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.117+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 shutdown_connections 2020-11-20T12:09:02.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.117+0000 7f954d0f9700 1 -- 172.21.15.187:0/2823223627 wait complete. 2020-11-20T12:09:02.125 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:09:02.147 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:09:02.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.389+0000 7f8ae88f1700 1 Processor -- start 2020-11-20T12:09:02.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.390+0000 7f8ae88f1700 1 -- start start 2020-11-20T12:09:02.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.391+0000 7f8ae88f1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8ae4065f90 0x7f8ae4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:02.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.391+0000 7f8ae88f1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8ae409ca90 0x7f8ae409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:02.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.391+0000 7f8ae88f1700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40b13e0 con 0x7f8ae4065190 2020-11-20T12:09:02.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.391+0000 7f8ae88f1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae414ab30 con 0x7f8ae409ca90 2020-11-20T12:09:02.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.391+0000 7f8ae88f1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae4062c10 con 0x7f8ae4065f90 2020-11-20T12:09:02.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ae2d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8ae4065190 0x7f8ae4143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:40994/0 (socket says 172.21.15.187:40994) 2020-11-20T12:09:02.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ae2d9d700 1 -- 172.21.15.187:0/2459013228 learned_addr learned my addr 172.21.15.187:0/2459013228 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:02.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ada59c700 1 --2- 172.21.15.187:0/2459013228 >> v2:172.21.15.187:3300/0 conn(0x7f8ae4065f90 0x7f8ae4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:02.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ae259c700 1 -- 172.21.15.187:0/2459013228 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386672046 0 0) 0x7f8ae40b13e0 con 0x7f8ae4065190 2020-11-20T12:09:02.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ae359e700 1 --2- 172.21.15.187:0/2459013228 >> v2:172.21.15.5:3300/0 conn(0x7f8ae409ca90 0x7f8ae409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:02.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ae259c700 1 -- 172.21.15.187:0/2459013228 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac4003240 con 0x7f8ae4065190 2020-11-20T12:09:02.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ada59c700 1 -- 172.21.15.187:0/2459013228 >> v2:172.21.15.5:3300/0 conn(0x7f8ae409ca90 msgr2=0x7f8ae409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:02.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ada59c700 1 --2- 172.21.15.187:0/2459013228 >> v2:172.21.15.5:3300/0 conn(0x7f8ae409ca90 0x7f8ae409cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ada59c700 1 -- 172.21.15.187:0/2459013228 >> v1:172.21.15.187:6789/0 conn(0x7f8ae4065190 legacy=0x7f8ae4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ada59c700 1 -- 172.21.15.187:0/2459013228 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae40b1640 con 0x7f8ae4065f90 2020-11-20T12:09:02.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.392+0000 7f8ae359e700 1 --2- 172.21.15.187:0/2459013228 >> v2:172.21.15.5:3300/0 conn(0x7f8ae409ca90 0x7f8ae409cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:02.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.393+0000 7f8ada59c700 1 --2- 172.21.15.187:0/2459013228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae4065f90 0x7f8ae4065570 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f8ae40b5fb0 tx=0x7f8acc006df0).ready entity=mon.1 client_cookie=d16962b6f5fd9531 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:02.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.393+0000 7f8ae259c700 1 -- 172.21.15.187:0/2459013228 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc009840 con 0x7f8ae4065f90 2020-11-20T12:09:02.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.393+0000 7f8ae259c700 1 -- 172.21.15.187:0/2459013228 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc0099a0 con 0x7f8ae4065f90 2020-11-20T12:09:02.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.393+0000 7f8ae259c700 1 -- 172.21.15.187:0/2459013228 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc009d20 con 0x7f8ae4065f90 2020-11-20T12:09:02.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.394+0000 7f8ae88f1700 1 -- 172.21.15.187:0/2459013228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae4065f90 msgr2=0x7f8ae4065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.394+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/2459013228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae4065f90 0x7f8ae4065570 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f8ae40b5fb0 tx=0x7f8acc006df0).stop 2020-11-20T12:09:02.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.394+0000 7f8ae88f1700 1 -- 172.21.15.187:0/2459013228 shutdown_connections 2020-11-20T12:09:02.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.394+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/2459013228 >> v2:172.21.15.5:3300/0 conn(0x7f8ae409ca90 0x7f8ae409cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.394+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/2459013228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae4065f90 0x7f8ae4065570 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.394+0000 7f8ae88f1700 1 -- 172.21.15.187:0/2459013228 >> 172.21.15.187:0/2459013228 conn(0x7f8ae40bab30 msgr2=0x7f8ae40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:02.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.395+0000 7f8ae88f1700 1 -- 172.21.15.187:0/2459013228 shutdown_connections 2020-11-20T12:09:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.395+0000 7f8ae88f1700 1 -- 172.21.15.187:0/2459013228 wait complete. 2020-11-20T12:09:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.396+0000 7f8ae88f1700 1 Processor -- start 2020-11-20T12:09:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.396+0000 7f8ae88f1700 1 -- start start 2020-11-20T12:09:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.397+0000 7f8ae88f1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae4065190 0x7f8ae40992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.397+0000 7f8ae88f1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae4065f90 0x7f8ae409dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:02.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.397+0000 7f8ae88f1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae409ca90 0x7f8ae4141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:02.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.397+0000 7f8ae88f1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40b7a20 con 0x7f8ae4065190 2020-11-20T12:09:02.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.397+0000 7f8ae88f1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40b7cc0 con 0x7f8ae409ca90 2020-11-20T12:09:02.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.397+0000 7f8ae88f1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40b1c20 con 0x7f8ae4065f90 2020-11-20T12:09:02.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae409ca90 0x7f8ae4141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:02.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae409ca90 0x7f8ae4141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44398/0 (socket says 172.21.15.187:44398) 2020-11-20T12:09:02.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae359e700 1 -- 172.21.15.187:0/339243878 learned_addr learned my addr 172.21.15.187:0/339243878 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:02.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae2d9d700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae4065190 0x7f8ae40992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:02.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ada59c700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae4065f90 0x7f8ae409dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:02.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae359e700 1 -- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae4065f90 msgr2=0x7f8ae409dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae359e700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae4065f90 0x7f8ae409dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae359e700 1 -- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae4065190 msgr2=0x7f8ae40992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae359e700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae4065190 0x7f8ae40992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ae359e700 1 -- 172.21.15.187:0/339243878 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae40997e0 con 0x7f8ae409ca90 2020-11-20T12:09:02.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.398+0000 7f8ada59c700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae4065f90 0x7f8ae409dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:02.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.399+0000 7f8ae359e700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae409ca90 0x7f8ae4141a80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f8adc00cb50 tx=0x7f8adc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:02.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.399+0000 7f8adb7fe700 1 -- 172.21.15.187:0/339243878 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc007990 con 0x7f8ae409ca90 2020-11-20T12:09:02.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.399+0000 7f8adb7fe700 1 -- 172.21.15.187:0/339243878 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc007af0 con 0x7f8ae409ca90 2020-11-20T12:09:02.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.399+0000 7f8adb7fe700 1 -- 172.21.15.187:0/339243878 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc007e70 con 0x7f8ae409ca90 2020-11-20T12:09:02.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.399+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae40b7580 con 0x7f8ae409ca90 2020-11-20T12:09:02.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.399+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae409e440 con 0x7f8ae409ca90 2020-11-20T12:09:02.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.400+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae4061a20 con 0x7f8ae409ca90 2020-11-20T12:09:02.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.401+0000 7f8adb7fe700 1 -- 172.21.15.187:0/339243878 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8adc011620 con 0x7f8ae409ca90 2020-11-20T12:09:02.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.401+0000 7f8adb7fe700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ac0056d20 0x7f8ac00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:02.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.401+0000 7f8adb7fe700 1 -- 172.21.15.187:0/339243878 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8adc0701f0 con 0x7f8ae409ca90 2020-11-20T12:09:02.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.402+0000 7f8ae2d9d700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ac0056d20 0x7f8ac00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:02.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.402+0000 7f8ae2d9d700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ac0056d20 0x7f8ac00591e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8ad40055a0 tx=0x7f8ad4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:02.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.405+0000 7f8adb7fe700 1 -- 172.21.15.187:0/339243878 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8adc045860 con 0x7f8ae409ca90 2020-11-20T12:09:02.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.656+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8ae40afc60 con 0x7f8ae409ca90 2020-11-20T12:09:02.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.657+0000 7f8adb7fe700 1 -- 172.21.15.187:0/339243878 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f8adc048e80 con 0x7f8ae409ca90 2020-11-20T12:09:02.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ac0056d20 msgr2=0x7f8ac00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ac0056d20 0x7f8ac00591e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8ad40055a0 tx=0x7f8ad4009040).stop 2020-11-20T12:09:02.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae409ca90 msgr2=0x7f8ae4141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae409ca90 0x7f8ae4141a80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f8adc00cb50 tx=0x7f8adc008040).stop 2020-11-20T12:09:02.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 shutdown_connections 2020-11-20T12:09:02.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ac0056d20 0x7f8ac00591e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae4065190 0x7f8ae40992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae409ca90 0x7f8ae4141a80 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 --2- 172.21.15.187:0/339243878 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae4065f90 0x7f8ae409dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 >> 172.21.15.187:0/339243878 conn(0x7f8ae40bab30 msgr2=0x7f8ae409bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:02.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.659+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 shutdown_connections 2020-11-20T12:09:02.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.660+0000 7f8ae88f1700 1 -- 172.21.15.187:0/339243878 wait complete. 2020-11-20T12:09:02.668 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:02.669 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:09:02.669 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:09:02.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.669 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:09:02.669 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:09:02.670 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.670 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:09:02.670 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.670 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.671 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.671 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:09:02.672 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:09:02.672 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.672 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:09:02.672 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.672 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.673 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.673 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.674 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:09:02.674 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:09:02.674 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.674 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.675 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.675 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.675 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.676 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:09:02.676 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:09:02.676 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.676 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.677 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.677 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.677 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.678 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:09:02.678 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:09:02.678 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.678 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.679 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.679 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.679 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.680 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:09:02.680 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:09:02.680 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.680 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.680 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.681 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.681 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.682 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:09:02.682 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:09:02.682 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.682 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.682 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.683 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.683 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.683 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:09:02.684 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:09:02.684 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.684 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.684 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.684 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.685 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.688 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:09:02.693 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:09:02.693 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.693 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.694 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.694 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.694 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.695 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:09:02.695 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:09:02.695 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.695 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.696 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.696 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.696 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.697 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:09:02.697 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:09:02.697 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.697 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:09:02.697 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:09:02.698 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:09:02.698 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.698 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:09:02.699 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:09:02.699 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.699 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:09:02.699 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:09:02.699 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:09:02.700 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:09:02.700 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.700 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:09:02.701 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:09:02.701 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.701 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:09:02.701 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:09:02.702 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:09:02.702 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:09:02.702 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.705 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:09:02.705 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:09:02.705 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.706 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:09:02.706 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:09:02.706 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:09:02.706 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:09:02.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.707 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:09:02.707 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:09:02.708 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.708 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:09:02.708 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:09:02.708 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:09:02.709 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.709 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.709 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:09:02.710 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:09:02.710 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.710 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:09:02.710 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:09:02.710 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.711 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:02.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:02.711 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:09:02.712 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:09:02.712 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:09:02.712 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:09:02.712 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:09:02.712 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:09:02.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:02.713 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:09:02.713 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:09:02.715 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:09:02.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.963+0000 7ff161bba700 1 Processor -- start 2020-11-20T12:09:02.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.963+0000 7ff161bba700 1 -- start start 2020-11-20T12:09:02.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.965+0000 7ff161bba700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c0683f0 0x7ff15c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:02.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.965+0000 7ff161bba700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff15c1487e0 0x7ff15c148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:02.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.965+0000 7ff161bba700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c14c170 con 0x7ff15c145cc0 2020-11-20T12:09:02.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.965+0000 7ff161bba700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15c14c410 con 0x7ff15c1487e0 2020-11-20T12:09:02.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.965+0000 7ff161bba700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c14c6b0 con 0x7ff15c0683f0 2020-11-20T12:09:02.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.965+0000 7ff160bb8700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff15c145cc0 0x7ff15c065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41012/0 (socket says 172.21.15.187:41012) 2020-11-20T12:09:02.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.965+0000 7ff160bb8700 1 -- 172.21.15.187:0/3190755701 learned_addr learned my addr 172.21.15.187:0/3190755701 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:02.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.966+0000 7ff1613b9700 1 --2- 172.21.15.187:0/3190755701 >> v2:172.21.15.5:3300/0 conn(0x7ff15c1487e0 0x7ff15c148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:02.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.966+0000 7ff153fff700 1 --2- 172.21.15.187:0/3190755701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c0683f0 0x7ff15c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:02.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.966+0000 7ff1537fe700 1 -- 172.21.15.187:0/3190755701 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114224231 0 0) 0x7ff15c14c170 con 0x7ff15c145cc0 2020-11-20T12:09:02.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.966+0000 7ff1537fe700 1 -- 172.21.15.187:0/3190755701 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff148003240 con 0x7ff15c145cc0 2020-11-20T12:09:02.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.967+0000 7ff153fff700 1 -- 172.21.15.187:0/3190755701 >> v2:172.21.15.5:3300/0 conn(0x7ff15c1487e0 msgr2=0x7ff15c148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:02.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.967+0000 7ff153fff700 1 --2- 172.21.15.187:0/3190755701 >> v2:172.21.15.5:3300/0 conn(0x7ff15c1487e0 0x7ff15c148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:02.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.967+0000 7ff1537fe700 1 -- 172.21.15.187:0/3190755701 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2558685431 0 0) 0x7ff148003240 con 0x7ff15c145cc0 2020-11-20T12:09:02.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.967+0000 7ff153fff700 1 -- 172.21.15.187:0/3190755701 >> v1:172.21.15.187:6789/0 conn(0x7ff15c145cc0 legacy=0x7ff15c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.967+0000 7ff153fff700 1 -- 172.21.15.187:0/3190755701 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c0658a0 con 0x7ff15c0683f0 2020-11-20T12:09:02.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.967+0000 7ff153fff700 1 --2- 172.21.15.187:0/3190755701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c0683f0 0x7ff15c066040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7ff14c00b9a0 tx=0x7ff14c006df0).ready entity=mon.2 client_cookie=eb46c38f388a77ce server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:02.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.968+0000 7ff1537fe700 1 -- 172.21.15.187:0/3190755701 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580014e0 con 0x7ff15c0683f0 2020-11-20T12:09:02.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.968+0000 7ff1537fe700 1 -- 172.21.15.187:0/3190755701 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c011400 con 0x7ff15c0683f0 2020-11-20T12:09:02.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.968+0000 7ff1537fe700 1 -- 172.21.15.187:0/3190755701 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c011780 con 0x7ff15c0683f0 2020-11-20T12:09:02.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.969+0000 7ff161bba700 1 -- 172.21.15.187:0/3190755701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c0683f0 msgr2=0x7ff15c066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:02.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.969+0000 7ff161bba700 1 --2- 172.21.15.187:0/3190755701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c0683f0 0x7ff15c066040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7ff14c00b9a0 tx=0x7ff14c006df0).stop 2020-11-20T12:09:03.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.969+0000 7ff161bba700 1 -- 172.21.15.187:0/3190755701 shutdown_connections 2020-11-20T12:09:03.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.969+0000 7ff161bba700 1 --2- 172.21.15.187:0/3190755701 >> v2:172.21.15.5:3300/0 conn(0x7ff15c1487e0 0x7ff15c148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.969+0000 7ff161bba700 1 --2- 172.21.15.187:0/3190755701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c0683f0 0x7ff15c066040 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.969+0000 7ff161bba700 1 -- 172.21.15.187:0/3190755701 >> 172.21.15.187:0/3190755701 conn(0x7ff15c0bab30 msgr2=0x7ff15c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:03.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.970+0000 7ff161bba700 1 -- 172.21.15.187:0/3190755701 shutdown_connections 2020-11-20T12:09:03.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.970+0000 7ff161bba700 1 -- 172.21.15.187:0/3190755701 wait complete. 2020-11-20T12:09:03.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.972+0000 7ff161bba700 1 Processor -- start 2020-11-20T12:09:03.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.972+0000 7ff161bba700 1 -- start start 2020-11-20T12:09:03.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.973+0000 7ff161bba700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff15c0683f0 0x7ff15c09ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.973+0000 7ff161bba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff15c145cc0 0x7ff15c09ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.973+0000 7ff161bba700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c1487e0 0x7ff15c09f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.973+0000 7ff161bba700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c0a0560 con 0x7ff15c0683f0 2020-11-20T12:09:03.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.973+0000 7ff161bba700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c0a0800 con 0x7ff15c145cc0 2020-11-20T12:09:03.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.973+0000 7ff161bba700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c09fa90 con 0x7ff15c1487e0 2020-11-20T12:09:03.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff15c145cc0 0x7ff15c09ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff15c145cc0 0x7ff15c09ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44410/0 (socket says 172.21.15.187:44410) 2020-11-20T12:09:03.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 -- 172.21.15.187:0/452060019 learned_addr learned my addr 172.21.15.187:0/452060019 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:03.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff160bb8700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff15c0683f0 0x7ff15c09ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff1613b9700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c1487e0 0x7ff15c09f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 -- 172.21.15.187:0/452060019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c1487e0 msgr2=0x7ff15c09f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c1487e0 0x7ff15c09f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 -- 172.21.15.187:0/452060019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff15c0683f0 msgr2=0x7ff15c09ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff15c0683f0 0x7ff15c09ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 -- 172.21.15.187:0/452060019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c09d1c0 con 0x7ff15c145cc0 2020-11-20T12:09:03.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.974+0000 7ff153fff700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff15c145cc0 0x7ff15c09ef70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7ff14c006e30 tx=0x7ff14c013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:03.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.975+0000 7ff150ff9700 1 -- 172.21.15.187:0/452060019 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c010c00 con 0x7ff15c145cc0 2020-11-20T12:09:03.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.975+0000 7ff150ff9700 1 -- 172.21.15.187:0/452060019 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c010d60 con 0x7ff15c145cc0 2020-11-20T12:09:03.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.975+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c15b6a0 con 0x7ff15c145cc0 2020-11-20T12:09:03.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.975+0000 7ff150ff9700 1 -- 172.21.15.187:0/452060019 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c01e690 con 0x7ff15c145cc0 2020-11-20T12:09:03.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.975+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c15bb30 con 0x7ff15c145cc0 2020-11-20T12:09:03.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.976+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c09fc50 con 0x7ff15c145cc0 2020-11-20T12:09:03.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.977+0000 7ff150ff9700 1 -- 172.21.15.187:0/452060019 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff14c025030 con 0x7ff15c145cc0 2020-11-20T12:09:03.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.977+0000 7ff150ff9700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff134056d20 0x7ff1340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.978+0000 7ff150ff9700 1 -- 172.21.15.187:0/452060019 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff14c071420 con 0x7ff15c145cc0 2020-11-20T12:09:03.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.978+0000 7ff160bb8700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff134056d20 0x7ff1340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.978+0000 7ff160bb8700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff134056d20 0x7ff1340591e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7ff158001e90 tx=0x7ff158009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:03.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:02.982+0000 7ff150ff9700 1 -- 172.21.15.187:0/452060019 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff14c046b10 con 0x7ff15c145cc0 2020-11-20T12:09:03.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.172+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff15c061a20 con 0x7ff15c145cc0 2020-11-20T12:09:03.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.174+0000 7ff150ff9700 1 -- 172.21.15.187:0/452060019 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+13628 (secure 0 0 0) 0x7ff14c023030 con 0x7ff15c145cc0 2020-11-20T12:09:03.179 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:03.179 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":55,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:09:02.094698+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:09:03.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff134056d20 msgr2=0x7ff1340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff134056d20 0x7ff1340591e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7ff158001e90 tx=0x7ff158009040).stop 2020-11-20T12:09:03.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff15c145cc0 msgr2=0x7ff15c09ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff15c145cc0 0x7ff15c09ef70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7ff14c006e30 tx=0x7ff14c013000).stop 2020-11-20T12:09:03.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 shutdown_connections 2020-11-20T12:09:03.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff134056d20 0x7ff1340591e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff15c0683f0 0x7ff15c09ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff15c1487e0 0x7ff15c09f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 --2- 172.21.15.187:0/452060019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff15c145cc0 0x7ff15c09ef70 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 >> 172.21.15.187:0/452060019 conn(0x7ff15c0bab30 msgr2=0x7ff15c062500 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:03.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.177+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 shutdown_connections 2020-11-20T12:09:03.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.178+0000 7ff161bba700 1 -- 172.21.15.187:0/452060019 wait complete. 2020-11-20T12:09:03.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:09:03.208 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:09:03.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.470+0000 7f124359e700 1 Processor -- start 2020-11-20T12:09:03.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.470+0000 7f124359e700 1 -- start start 2020-11-20T12:09:03.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f124359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12441484d0 0x7f1244061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f124359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12440622c0 0x7f12440b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f124359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12440b1300 con 0x7f1244066f70 2020-11-20T12:09:03.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f124359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1244067730 con 0x7f12441484d0 2020-11-20T12:09:03.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f124359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12440b15a0 con 0x7f12440622c0 2020-11-20T12:09:03.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f1242d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1244066f70 0x7f12440b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41026/0 (socket says 172.21.15.187:41026) 2020-11-20T12:09:03.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f1242d9d700 1 -- 172.21.15.187:0/4087344211 learned_addr learned my addr 172.21.15.187:0/4087344211 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:03.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f1239d9b700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12440622c0 0x7f12440b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.472+0000 7f124259c700 1 --2- 172.21.15.187:0/4087344211 >> v2:172.21.15.5:3300/0 conn(0x7f12441484d0 0x7f1244061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161575550 0 0) 0x7f12440b1300 con 0x7f1244066f70 2020-11-20T12:09:03.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1224003240 con 0x7f1244066f70 2020-11-20T12:09:03.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 204666775 0 0) 0x7f1224003240 con 0x7f1244066f70 2020-11-20T12:09:03.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12440622c0 msgr2=0x7f12440b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:03.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12440622c0 0x7f12440b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 >> v2:172.21.15.5:3300/0 conn(0x7f12441484d0 msgr2=0x7f1244061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:03.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 --2- 172.21.15.187:0/4087344211 >> v2:172.21.15.5:3300/0 conn(0x7f12441484d0 0x7f1244061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12440b1840 con 0x7f1244066f70 2020-11-20T12:09:03.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f123c0014e0 con 0x7f1244066f70 2020-11-20T12:09:03.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 >> v1:172.21.15.187:6789/0 conn(0x7f1244066f70 legacy=0x7f12440b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12440622c0 0x7f1224003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1224003c40 0x7f12240044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12240049d0 0x7f1224004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244067730 con 0x7f1224003c40 2020-11-20T12:09:03.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12440b15a0 con 0x7f12240049d0 2020-11-20T12:09:03.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.473+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1224004200 con 0x7f12440622c0 2020-11-20T12:09:03.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.474+0000 7f1242d9d700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12240049d0 0x7f1224004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.474+0000 7f124259c700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12440622c0 0x7f1224003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.474+0000 7f1239d9b700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1224003c40 0x7f12240044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.474+0000 7f1242d9d700 1 -- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12440622c0 msgr2=0x7f1224003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:03.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.474+0000 7f1242d9d700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12440622c0 0x7f1224003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.474+0000 7f1242d9d700 1 -- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1224003c40 msgr2=0x7f12240044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:03.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.474+0000 7f1242d9d700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1224003c40 0x7f12240044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.474+0000 7f1242d9d700 1 -- 172.21.15.187:0/4087344211 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1224003240 con 0x7f12240049d0 2020-11-20T12:09:03.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.475+0000 7f1242d9d700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12240049d0 0x7f1224004e50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f123c00c440 tx=0x7f123c007800).ready entity=mon.1 client_cookie=78029190013c86f8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:03.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.475+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c0072e0 con 0x7f12240049d0 2020-11-20T12:09:03.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.475+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c007440 con 0x7f12240049d0 2020-11-20T12:09:03.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.475+0000 7f1241d9b700 1 -- 172.21.15.187:0/4087344211 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c005840 con 0x7f12240049d0 2020-11-20T12:09:03.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.476+0000 7f124359e700 1 -- 172.21.15.187:0/4087344211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12240049d0 msgr2=0x7f1224004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.476+0000 7f124359e700 1 --2- 172.21.15.187:0/4087344211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12240049d0 0x7f1224004e50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f123c00c440 tx=0x7f123c007800).stop 2020-11-20T12:09:03.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.476+0000 7f1242d9d700 1 -- 172.21.15.187:0/4087344211 reap_dead start 2020-11-20T12:09:03.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.476+0000 7f124359e700 1 -- 172.21.15.187:0/4087344211 shutdown_connections 2020-11-20T12:09:03.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.476+0000 7f124359e700 1 -- 172.21.15.187:0/4087344211 >> 172.21.15.187:0/4087344211 conn(0x7f12440bab30 msgr2=0x7f12440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:03.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.476+0000 7f124359e700 1 -- 172.21.15.187:0/4087344211 shutdown_connections 2020-11-20T12:09:03.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.476+0000 7f124359e700 1 -- 172.21.15.187:0/4087344211 wait complete. 2020-11-20T12:09:03.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.478+0000 7f124359e700 1 Processor -- start 2020-11-20T12:09:03.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.478+0000 7f124359e700 1 -- start start 2020-11-20T12:09:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12440a0000 0x7f12440b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12440a03e0 0x7f12441589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f124415b570 0x7f124415b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124409cbb0 con 0x7f12440a03e0 2020-11-20T12:09:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244158ee0 con 0x7f12440a0000 2020-11-20T12:09:03.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12441582b0 con 0x7f124415b570 2020-11-20T12:09:03.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12440a0000 0x7f12440b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12440a0000 0x7f12440b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44428/0 (socket says 172.21.15.187:44428) 2020-11-20T12:09:03.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.479+0000 7f124259c700 1 -- 172.21.15.187:0/2593822907 learned_addr learned my addr 172.21.15.187:0/2593822907 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:03.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f1239d9b700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12440a03e0 0x7f12441589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f1242d9d700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f124415b570 0x7f124415b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f124259c700 1 -- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f124415b570 msgr2=0x7f124415b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f124259c700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f124415b570 0x7f124415b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f124259c700 1 -- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12440a03e0 msgr2=0x7f12441589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f124259c700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12440a03e0 0x7f12441589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f124259c700 1 -- 172.21.15.187:0/2593822907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244158550 con 0x7f12440a0000 2020-11-20T12:09:03.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f1239d9b700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12440a03e0 0x7f12441589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:03.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f124259c700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12440a0000 0x7f12440b4740 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f12340090e0 tx=0x7f1234008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:03.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f123affd700 1 -- 172.21.15.187:0/2593822907 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123400ec30 con 0x7f12440a0000 2020-11-20T12:09:03.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f123affd700 1 -- 172.21.15.187:0/2593822907 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123400ed90 con 0x7f12440a0000 2020-11-20T12:09:03.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1244157fa0 con 0x7f12440a0000 2020-11-20T12:09:03.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f123affd700 1 -- 172.21.15.187:0/2593822907 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234014640 con 0x7f12440a0000 2020-11-20T12:09:03.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.480+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124415c120 con 0x7f12440a0000 2020-11-20T12:09:03.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.481+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12440a07c0 con 0x7f12440a0000 2020-11-20T12:09:03.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.482+0000 7f123affd700 1 -- 172.21.15.187:0/2593822907 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1234024070 con 0x7f12440a0000 2020-11-20T12:09:03.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.483+0000 7f123affd700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1220056d20 0x7f12200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.483+0000 7f123affd700 1 -- 172.21.15.187:0/2593822907 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f1234070460 con 0x7f12440a0000 2020-11-20T12:09:03.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.483+0000 7f1239d9b700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1220056d20 0x7f12200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.484+0000 7f1239d9b700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1220056d20 0x7f12200591e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f12440b44e0 tx=0x7f122c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:03.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.486+0000 7f123affd700 1 -- 172.21.15.187:0/2593822907 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1234045be0 con 0x7f12440a0000 2020-11-20T12:09:03.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.671+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f124415c3d0 con 0x7f12440a0000 2020-11-20T12:09:03.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.672+0000 7f123affd700 1 -- 172.21.15.187:0/2593822907 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v55) v1 ==== 110+0+10 (secure 0 0 0) 0x7f1234049200 con 0x7f12440a0000 2020-11-20T12:09:03.677 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:09:03.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.674+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1220056d20 msgr2=0x7f12200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.674+0000 7f124359e700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1220056d20 0x7f12200591e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f12440b44e0 tx=0x7f122c008040).stop 2020-11-20T12:09:03.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.674+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12440a0000 msgr2=0x7f12440b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.674+0000 7f124359e700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12440a0000 0x7f12440b4740 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f12340090e0 tx=0x7f1234008040).stop 2020-11-20T12:09:03.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.675+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 shutdown_connections 2020-11-20T12:09:03.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.675+0000 7f124359e700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1220056d20 0x7f12200591e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.675+0000 7f124359e700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f124415b570 0x7f124415b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.675+0000 7f124359e700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12440a0000 0x7f12440b4740 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.675+0000 7f124359e700 1 --2- 172.21.15.187:0/2593822907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12440a03e0 0x7f12441589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.675+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 >> 172.21.15.187:0/2593822907 conn(0x7f12440bab30 msgr2=0x7f1244065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:03.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.675+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 shutdown_connections 2020-11-20T12:09:03.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.675+0000 7f124359e700 1 -- 172.21.15.187:0/2593822907 wait complete. 2020-11-20T12:09:03.704 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:09:03.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.955+0000 7fd4965d2700 1 Processor -- start 2020-11-20T12:09:03.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.955+0000 7fd4965d2700 1 -- start start 2020-11-20T12:09:03.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.957+0000 7fd4965d2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd490145cc0 0x7fd490066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.957+0000 7fd4965d2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4901487e0 0x7fd490148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.957+0000 7fd4965d2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49014c170 con 0x7fd4900683f0 2020-11-20T12:09:03.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.957+0000 7fd4965d2700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49014c410 con 0x7fd4901487e0 2020-11-20T12:09:03.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.957+0000 7fd4965d2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49014c6b0 con 0x7fd490145cc0 2020-11-20T12:09:03.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.957+0000 7fd494dcf700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd4900683f0 0x7fd490065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41042/0 (socket says 172.21.15.187:41042) 2020-11-20T12:09:03.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.957+0000 7fd494dcf700 1 -- 172.21.15.187:0/1380358297 learned_addr learned my addr 172.21.15.187:0/1380358297 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:03.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.957+0000 7fd4955d0700 1 --2- 172.21.15.187:0/1380358297 >> v2:172.21.15.187:3300/0 conn(0x7fd490145cc0 0x7fd490066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd495dd1700 1 --2- 172.21.15.187:0/1380358297 >> v2:172.21.15.5:3300/0 conn(0x7fd4901487e0 0x7fd490148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd4955d0700 1 -- 172.21.15.187:0/1380358297 >> v2:172.21.15.5:3300/0 conn(0x7fd4901487e0 msgr2=0x7fd490148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:03.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd4955d0700 1 --2- 172.21.15.187:0/1380358297 >> v2:172.21.15.5:3300/0 conn(0x7fd4901487e0 0x7fd490148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd4955d0700 1 -- 172.21.15.187:0/1380358297 >> v1:172.21.15.187:6789/0 conn(0x7fd4900683f0 legacy=0x7fd490065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd47ffff700 1 -- 172.21.15.187:0/1380358297 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067072878 0 0) 0x7fd49014c170 con 0x7fd4900683f0 2020-11-20T12:09:03.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd4955d0700 1 -- 172.21.15.187:0/1380358297 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900658a0 con 0x7fd490145cc0 2020-11-20T12:09:03.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd4955d0700 1 --2- 172.21.15.187:0/1380358297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490145cc0 0x7fd490066040 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7fd4900b85e0 tx=0x7fd48c006df0).ready entity=mon.1 client_cookie=8f36fe1689ed5473 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:03.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd47ffff700 1 -- 172.21.15.187:0/1380358297 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0097b0 con 0x7fd490145cc0 2020-11-20T12:09:03.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd47ffff700 1 -- 172.21.15.187:0/1380358297 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c009910 con 0x7fd490145cc0 2020-11-20T12:09:03.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.958+0000 7fd47ffff700 1 -- 172.21.15.187:0/1380358297 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c009c90 con 0x7fd490145cc0 2020-11-20T12:09:03.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.959+0000 7fd4965d2700 1 -- 172.21.15.187:0/1380358297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490145cc0 msgr2=0x7fd490066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.959+0000 7fd4965d2700 1 --2- 172.21.15.187:0/1380358297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490145cc0 0x7fd490066040 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7fd4900b85e0 tx=0x7fd48c006df0).stop 2020-11-20T12:09:03.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.959+0000 7fd4965d2700 1 -- 172.21.15.187:0/1380358297 shutdown_connections 2020-11-20T12:09:03.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.959+0000 7fd4965d2700 1 --2- 172.21.15.187:0/1380358297 >> v2:172.21.15.5:3300/0 conn(0x7fd4901487e0 0x7fd490148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.959+0000 7fd4965d2700 1 --2- 172.21.15.187:0/1380358297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490145cc0 0x7fd490066040 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:03.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.959+0000 7fd4965d2700 1 -- 172.21.15.187:0/1380358297 >> 172.21.15.187:0/1380358297 conn(0x7fd4900bab30 msgr2=0x7fd4900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:03.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.960+0000 7fd4965d2700 1 -- 172.21.15.187:0/1380358297 shutdown_connections 2020-11-20T12:09:03.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.960+0000 7fd4965d2700 1 -- 172.21.15.187:0/1380358297 wait complete. 2020-11-20T12:09:03.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.961+0000 7fd4965d2700 1 Processor -- start 2020-11-20T12:09:03.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.962+0000 7fd4965d2700 1 -- start start 2020-11-20T12:09:03.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.962+0000 7fd4965d2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900683f0 0x7fd49009c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4965d2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd490145cc0 0x7fd49009e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4965d2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4901487e0 0x7fd49015bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4965d2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49009efd0 con 0x7fd490145cc0 2020-11-20T12:09:03.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4965d2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49015b400 con 0x7fd4900683f0 2020-11-20T12:09:03.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4965d2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49015b6a0 con 0x7fd4901487e0 2020-11-20T12:09:03.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4955d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900683f0 0x7fd49009c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4955d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900683f0 0x7fd49009c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44442/0 (socket says 172.21.15.187:44442) 2020-11-20T12:09:03.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4955d0700 1 -- 172.21.15.187:0/366785609 learned_addr learned my addr 172.21.15.187:0/366785609 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:03.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4955d0700 1 -- 172.21.15.187:0/366785609 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4901487e0 msgr2=0x7fd49015bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4955d0700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4901487e0 0x7fd49015bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:03.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4955d0700 1 -- 172.21.15.187:0/366785609 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd490145cc0 msgr2=0x7fd49009e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:03.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4955d0700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd490145cc0 0x7fd49009e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:03.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.963+0000 7fd4955d0700 1 -- 172.21.15.187:0/366785609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900a0640 con 0x7fd4900683f0 2020-11-20T12:09:03.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.964+0000 7fd4955d0700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900683f0 0x7fd49009c900 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fd48c00bee0 tx=0x7fd48c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:03.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.964+0000 7fd47d7fa700 1 -- 172.21.15.187:0/366785609 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c011e50 con 0x7fd4900683f0 2020-11-20T12:09:03.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.964+0000 7fd47d7fa700 1 -- 172.21.15.187:0/366785609 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c01e460 con 0x7fd4900683f0 2020-11-20T12:09:03.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.964+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49009cdf0 con 0x7fd4900683f0 2020-11-20T12:09:03.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.964+0000 7fd47d7fa700 1 -- 172.21.15.187:0/366785609 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c01e820 con 0x7fd4900683f0 2020-11-20T12:09:03.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.964+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49009ed00 con 0x7fd4900683f0 2020-11-20T12:09:03.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.965+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd490061a20 con 0x7fd4900683f0 2020-11-20T12:09:03.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.966+0000 7fd47d7fa700 1 -- 172.21.15.187:0/366785609 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd48c025030 con 0x7fd4900683f0 2020-11-20T12:09:03.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.966+0000 7fd47d7fa700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd468056d20 0x7fd4680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:03.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.966+0000 7fd47d7fa700 1 -- 172.21.15.187:0/366785609 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd48c071660 con 0x7fd4900683f0 2020-11-20T12:09:03.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.967+0000 7fd494dcf700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd468056d20 0x7fd4680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:03.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.967+0000 7fd494dcf700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd468056d20 0x7fd4680591e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fd4800011a0 tx=0x7fd4800076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:03.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:03.970+0000 7fd47d7fa700 1 -- 172.21.15.187:0/366785609 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd48c046d60 con 0x7fd4900683f0 2020-11-20T12:09:04.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.157+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd49014bdf0 con 0x7fd4900683f0 2020-11-20T12:09:04.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.157+0000 7fd47d7fa700 1 -- 172.21.15.187:0/366785609 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd48c023070 con 0x7fd4900683f0 2020-11-20T12:09:04.162 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:09:04.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd468056d20 msgr2=0x7fd4680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd468056d20 0x7fd4680591e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fd4800011a0 tx=0x7fd4800076a0).stop 2020-11-20T12:09:04.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900683f0 msgr2=0x7fd49009c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900683f0 0x7fd49009c900 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fd48c00bee0 tx=0x7fd48c015000).stop 2020-11-20T12:09:04.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 shutdown_connections 2020-11-20T12:09:04.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd468056d20 0x7fd4680591e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4901487e0 0x7fd49015bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:04.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900683f0 0x7fd49009c900 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 --2- 172.21.15.187:0/366785609 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd490145cc0 0x7fd49009e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:04.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 >> 172.21.15.187:0/366785609 conn(0x7fd4900bab30 msgr2=0x7fd490062570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:04.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 shutdown_connections 2020-11-20T12:09:04.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.160+0000 7fd4965d2700 1 -- 172.21.15.187:0/366785609 wait complete. 2020-11-20T12:09:04.190 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:09:04.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.436+0000 7f0451f73700 1 Processor -- start 2020-11-20T12:09:04.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.436+0000 7f0451f73700 1 -- start start 2020-11-20T12:09:04.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f0451f73700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f044c065190 0x7f044c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f0451f73700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f044c065f90 0x7f044c143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f0451f73700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c143c70 0x7f044c09ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f0451f73700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044c14ab30 con 0x7f044c065190 2020-11-20T12:09:04.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f0451f73700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044c0b13e0 con 0x7f044c065f90 2020-11-20T12:09:04.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f0451f73700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c062c10 con 0x7f044c143c70 2020-11-20T12:09:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f044bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f044c065f90 0x7f044c143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f044bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f044c065f90 0x7f044c143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44450/0 (socket says 172.21.15.187:44450) 2020-11-20T12:09:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.438+0000 7f044bfff700 1 -- 172.21.15.187:0/388452504 learned_addr learned my addr 172.21.15.187:0/388452504 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f0450f71700 1 --2- 172.21.15.187:0/388452504 >> v2:172.21.15.5:3300/0 conn(0x7f044c065190 0x7f044c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044bfff700 1 -- 172.21.15.187:0/388452504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c143c70 msgr2=0x7f044c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f0451772700 1 --2- 172.21.15.187:0/388452504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c143c70 0x7f044c09ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044bfff700 1 --2- 172.21.15.187:0/388452504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c143c70 0x7f044c09ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044bfff700 1 -- 172.21.15.187:0/388452504 >> v2:172.21.15.5:3300/0 conn(0x7f044c065190 msgr2=0x7f044c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044bfff700 1 --2- 172.21.15.187:0/388452504 >> v2:172.21.15.5:3300/0 conn(0x7f044c065190 0x7f044c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044bfff700 1 -- 172.21.15.187:0/388452504 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c0b1640 con 0x7f044c065f90 2020-11-20T12:09:04.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f0450f71700 1 --2- 172.21.15.187:0/388452504 >> v2:172.21.15.5:3300/0 conn(0x7f044c065190 0x7f044c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:04.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f0451772700 1 --2- 172.21.15.187:0/388452504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c143c70 0x7f044c09ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:04.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044bfff700 1 --2- 172.21.15.187:0/388452504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065f90 0x7f044c143780 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f044c0b5fb0 tx=0x7f0438006eb0).ready entity=mon.1 client_cookie=3feac3439bc90285 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:04.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044b7fe700 1 -- 172.21.15.187:0/388452504 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438009960 con 0x7f044c065f90 2020-11-20T12:09:04.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044b7fe700 1 -- 172.21.15.187:0/388452504 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438009ac0 con 0x7f044c065f90 2020-11-20T12:09:04.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.439+0000 7f044b7fe700 1 -- 172.21.15.187:0/388452504 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438009e40 con 0x7f044c065f90 2020-11-20T12:09:04.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 -- 172.21.15.187:0/388452504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065f90 msgr2=0x7f044c143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 --2- 172.21.15.187:0/388452504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065f90 0x7f044c143780 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f044c0b5fb0 tx=0x7f0438006eb0).stop 2020-11-20T12:09:04.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 -- 172.21.15.187:0/388452504 shutdown_connections 2020-11-20T12:09:04.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 --2- 172.21.15.187:0/388452504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c143c70 0x7f044c09ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 --2- 172.21.15.187:0/388452504 >> v2:172.21.15.5:3300/0 conn(0x7f044c065190 0x7f044c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 --2- 172.21.15.187:0/388452504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065f90 0x7f044c143780 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 -- 172.21.15.187:0/388452504 >> 172.21.15.187:0/388452504 conn(0x7f044c0bab30 msgr2=0x7f044c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 -- 172.21.15.187:0/388452504 shutdown_connections 2020-11-20T12:09:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.440+0000 7f0451f73700 1 -- 172.21.15.187:0/388452504 wait complete. 2020-11-20T12:09:04.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.442+0000 7f0451f73700 1 Processor -- start 2020-11-20T12:09:04.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.442+0000 7f0451f73700 1 -- start start 2020-11-20T12:09:04.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.443+0000 7f0451f73700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065190 0x7f044c1393e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.443+0000 7f0451f73700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c065f90 0x7f044c09bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.443+0000 7f0451f73700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f044c143c70 0x7f044c09c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.444+0000 7f0451f73700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c1398d0 con 0x7f044c143c70 2020-11-20T12:09:04.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.444+0000 7f0451f73700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c09c250 con 0x7f044c065190 2020-11-20T12:09:04.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.444+0000 7f0451f73700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c0b1bb0 con 0x7f044c065f90 2020-11-20T12:09:04.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.444+0000 7f0450f71700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065190 0x7f044c1393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.444+0000 7f0450f71700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065190 0x7f044c1393e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44456/0 (socket says 172.21.15.187:44456) 2020-11-20T12:09:04.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.444+0000 7f0450f71700 1 -- 172.21.15.187:0/4147170578 learned_addr learned my addr 172.21.15.187:0/4147170578 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:04.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.444+0000 7f044bfff700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c065f90 0x7f044c09bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.444+0000 7f0451772700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f044c143c70 0x7f044c09c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0450f71700 1 -- 172.21.15.187:0/4147170578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c065f90 msgr2=0x7f044c09bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0450f71700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c065f90 0x7f044c09bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0450f71700 1 -- 172.21.15.187:0/4147170578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f044c143c70 msgr2=0x7f044c09c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0450f71700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f044c143c70 0x7f044c09c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0450f71700 1 -- 172.21.15.187:0/4147170578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c0b1de0 con 0x7f044c065190 2020-11-20T12:09:04.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0450f71700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065190 0x7f044c1393e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f044000cb50 tx=0x7f0440008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:04.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0448ff9700 1 -- 172.21.15.187:0/4147170578 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04400077d0 con 0x7f044c065190 2020-11-20T12:09:04.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0448ff9700 1 -- 172.21.15.187:0/4147170578 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0440007930 con 0x7f044c065190 2020-11-20T12:09:04.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044c0b2080 con 0x7f044c065190 2020-11-20T12:09:04.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0448ff9700 1 -- 172.21.15.187:0/4147170578 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440007cb0 con 0x7f044c065190 2020-11-20T12:09:04.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.445+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044c09ce80 con 0x7f044c065190 2020-11-20T12:09:04.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.446+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c061a20 con 0x7f044c065190 2020-11-20T12:09:04.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.447+0000 7f0448ff9700 1 -- 172.21.15.187:0/4147170578 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0440011620 con 0x7f044c065190 2020-11-20T12:09:04.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.448+0000 7f0448ff9700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0424056d20 0x7f04240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.448+0000 7f0448ff9700 1 -- 172.21.15.187:0/4147170578 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f044006ff80 con 0x7f044c065190 2020-11-20T12:09:04.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.448+0000 7f044bfff700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0424056d20 0x7f04240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.449+0000 7f044bfff700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0424056d20 0x7f04240591e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f044c0b5fb0 tx=0x7f0438015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:04.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.452+0000 7f0448ff9700 1 -- 172.21.15.187:0/4147170578 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0440045680 con 0x7f044c065190 2020-11-20T12:09:04.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.638+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f044c0afc60 con 0x7f044c065190 2020-11-20T12:09:04.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.639+0000 7f0448ff9700 1 -- 172.21.15.187:0/4147170578 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v55) v1 ==== 84+0+143 (secure 0 0 0) 0x7f0440048ca0 con 0x7f044c065190 2020-11-20T12:09:04.643 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:04.644 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:09:04.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:04.644 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:09:04.644 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:09:04.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:09:04.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:09:04.645 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:09:04.645 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:09:04.645 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:09:04.646 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:09:04.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0424056d20 msgr2=0x7f04240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0424056d20 0x7f04240591e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f044c0b5fb0 tx=0x7f0438015000).stop 2020-11-20T12:09:04.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065190 msgr2=0x7f044c1393e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065190 0x7f044c1393e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f044000cb50 tx=0x7f0440008040).stop 2020-11-20T12:09:04.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 shutdown_connections 2020-11-20T12:09:04.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0424056d20 0x7f04240591e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f044c143c70 0x7f044c09c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f044c065190 0x7f044c1393e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 --2- 172.21.15.187:0/4147170578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f044c065f90 0x7f044c09bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 >> 172.21.15.187:0/4147170578 conn(0x7f044c0bab30 msgr2=0x7f044c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:04.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 shutdown_connections 2020-11-20T12:09:04.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.641+0000 7f0451f73700 1 -- 172.21.15.187:0/4147170578 wait complete. 2020-11-20T12:09:04.670 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:09:04.671 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:09:04.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.915+0000 7f8058caa700 1 Processor -- start 2020-11-20T12:09:04.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.916+0000 7f8058caa700 1 -- start start 2020-11-20T12:09:04.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.917+0000 7f8058caa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.917+0000 7f8058caa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8054065f90 0x7f8054143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.917+0000 7f8058caa700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8054143c70 0x7f805409ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.917+0000 7f8058caa700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805414ab30 con 0x7f8054143c70 2020-11-20T12:09:04.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.917+0000 7f8058caa700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80540b13e0 con 0x7f8054065f90 2020-11-20T12:09:04.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.917+0000 7f8058caa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8054062c10 con 0x7f8054065190 2020-11-20T12:09:04.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.917+0000 7f804affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8054065f90 0x7f8054143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f804affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8054065f90 0x7f8054143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44466/0 (socket says 172.21.15.187:44466) 2020-11-20T12:09:04.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f804affd700 1 -- 172.21.15.187:0/2902401251 learned_addr learned my addr 172.21.15.187:0/2902401251 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:04.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f8053fff700 1 --2- 172.21.15.187:0/2902401251 >> v2:172.21.15.5:3300/0 conn(0x7f8054143c70 0x7f805409ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f80537fe700 1 --2- 172.21.15.187:0/2902401251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f804affd700 1 -- 172.21.15.187:0/2902401251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 msgr2=0x7f8054065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:04.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f804affd700 1 --2- 172.21.15.187:0/2902401251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f804affd700 1 -- 172.21.15.187:0/2902401251 >> v2:172.21.15.5:3300/0 conn(0x7f8054143c70 msgr2=0x7f805409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:04.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f804affd700 1 --2- 172.21.15.187:0/2902401251 >> v2:172.21.15.5:3300/0 conn(0x7f8054143c70 0x7f805409ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f804affd700 1 -- 172.21.15.187:0/2902401251 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80540b1640 con 0x7f8054065f90 2020-11-20T12:09:04.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f8053fff700 1 --2- 172.21.15.187:0/2902401251 >> v2:172.21.15.5:3300/0 conn(0x7f8054143c70 0x7f805409ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:04.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f80537fe700 1 --2- 172.21.15.187:0/2902401251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:04.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.918+0000 7f804affd700 1 --2- 172.21.15.187:0/2902401251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054065f90 0x7f8054143780 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f80540632e0 tx=0x7f803c006eb0).ready entity=mon.1 client_cookie=744059f688a4e43e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:04.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.919+0000 7f8052ffd700 1 -- 172.21.15.187:0/2902401251 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c009960 con 0x7f8054065f90 2020-11-20T12:09:04.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.919+0000 7f8052ffd700 1 -- 172.21.15.187:0/2902401251 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803c009ac0 con 0x7f8054065f90 2020-11-20T12:09:04.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.919+0000 7f8052ffd700 1 -- 172.21.15.187:0/2902401251 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c009e40 con 0x7f8054065f90 2020-11-20T12:09:04.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 -- 172.21.15.187:0/2902401251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054065f90 msgr2=0x7f8054143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 --2- 172.21.15.187:0/2902401251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054065f90 0x7f8054143780 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f80540632e0 tx=0x7f803c006eb0).stop 2020-11-20T12:09:04.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 -- 172.21.15.187:0/2902401251 shutdown_connections 2020-11-20T12:09:04.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 --2- 172.21.15.187:0/2902401251 >> v2:172.21.15.5:3300/0 conn(0x7f8054143c70 0x7f805409ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 --2- 172.21.15.187:0/2902401251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 --2- 172.21.15.187:0/2902401251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054065f90 0x7f8054143780 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 -- 172.21.15.187:0/2902401251 >> 172.21.15.187:0/2902401251 conn(0x7f80540bab30 msgr2=0x7f80540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:04.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 -- 172.21.15.187:0/2902401251 shutdown_connections 2020-11-20T12:09:04.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.920+0000 7f8058caa700 1 -- 172.21.15.187:0/2902401251 wait complete. 2020-11-20T12:09:04.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.922+0000 7f8058caa700 1 Processor -- start 2020-11-20T12:09:04.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.922+0000 7f8058caa700 1 -- start start 2020-11-20T12:09:04.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.923+0000 7f8058caa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054067280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.923+0000 7f8058caa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8054065f90 0x7f80541473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.923+0000 7f8058caa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054143c70 0x7f80540992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.923+0000 7f8058caa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8054067ab0 con 0x7f8054065f90 2020-11-20T12:09:04.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.923+0000 7f8058caa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8054067d50 con 0x7f8054143c70 2020-11-20T12:09:04.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.923+0000 7f8058caa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8054067ff0 con 0x7f8054065190 2020-11-20T12:09:04.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f8053fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054143c70 0x7f80540992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f8053fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054143c70 0x7f80540992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44474/0 (socket says 172.21.15.187:44474) 2020-11-20T12:09:04.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f8053fff700 1 -- 172.21.15.187:0/3962256116 learned_addr learned my addr 172.21.15.187:0/3962256116 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:04.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f804affd700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8054065f90 0x7f80541473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f80537fe700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054067280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f8053fff700 1 -- 172.21.15.187:0/3962256116 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 msgr2=0x7f8054067280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f8053fff700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054067280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f8053fff700 1 -- 172.21.15.187:0/3962256116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8054065f90 msgr2=0x7f80541473d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:04.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f8053fff700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8054065f90 0x7f80541473d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:04.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.924+0000 7f8053fff700 1 -- 172.21.15.187:0/3962256116 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054067770 con 0x7f8054143c70 2020-11-20T12:09:04.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.925+0000 7f8053fff700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054143c70 0x7f80540992c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f804c00cb50 tx=0x7f804c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:04.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.925+0000 7f804bfff700 1 -- 172.21.15.187:0/3962256116 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c007900 con 0x7f8054143c70 2020-11-20T12:09:04.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.925+0000 7f804bfff700 1 -- 172.21.15.187:0/3962256116 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c007a60 con 0x7f8054143c70 2020-11-20T12:09:04.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.925+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80541478c0 con 0x7f8054143c70 2020-11-20T12:09:04.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.925+0000 7f804bfff700 1 -- 172.21.15.187:0/3962256116 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c007de0 con 0x7f8054143c70 2020-11-20T12:09:04.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.925+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8054148fd0 con 0x7f8054143c70 2020-11-20T12:09:04.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.926+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8054149220 con 0x7f8054143c70 2020-11-20T12:09:04.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.927+0000 7f804bfff700 1 -- 172.21.15.187:0/3962256116 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f804c011620 con 0x7f8054143c70 2020-11-20T12:09:04.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.928+0000 7f804bfff700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8030056d20 0x7f80300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:04.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.928+0000 7f804bfff700 1 -- 172.21.15.187:0/3962256116 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f804c06ff60 con 0x7f8054143c70 2020-11-20T12:09:04.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.929+0000 7f80537fe700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8030056d20 0x7f80300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:04.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.929+0000 7f80537fe700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8030056d20 0x7f80300591e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f804400b7b0 tx=0x7f8044008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:04.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:04.932+0000 7f804bfff700 1 -- 172.21.15.187:0/3962256116 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f804c0456e0 con 0x7f8054143c70 2020-11-20T12:09:05.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.120+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f80540681b0 con 0x7f8054143c70 2020-11-20T12:09:05.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.684+0000 7f804bfff700 1 -- 172.21.15.187:0/3962256116 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v56) v1 ==== 133+0+0 (secure 0 0 0) 0x7f804c048d00 con 0x7f8054143c70 2020-11-20T12:09:05.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.687+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8030056d20 msgr2=0x7f80300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:05.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.687+0000 7f8058caa700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8030056d20 0x7f80300591e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f804400b7b0 tx=0x7f8044008040).stop 2020-11-20T12:09:05.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.687+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054143c70 msgr2=0x7f80540992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:05.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.687+0000 7f8058caa700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054143c70 0x7f80540992c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f804c00cb50 tx=0x7f804c008040).stop 2020-11-20T12:09:05.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.688+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 shutdown_connections 2020-11-20T12:09:05.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.688+0000 7f8058caa700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8030056d20 0x7f80300591e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.688+0000 7f8058caa700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8054143c70 0x7f80540992c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.688+0000 7f8058caa700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8054065190 0x7f8054067280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.688+0000 7f8058caa700 1 --2- 172.21.15.187:0/3962256116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8054065f90 0x7f80541473d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.688+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 >> 172.21.15.187:0/3962256116 conn(0x7f80540bab30 msgr2=0x7f80540b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:05.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.688+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 shutdown_connections 2020-11-20T12:09:05.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.688+0000 7f8058caa700 1 -- 172.21.15.187:0/3962256116 wait complete. 2020-11-20T12:09:05.698 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:09:05.724 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:09:05.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.961+0000 7f7af759e700 1 Processor -- start 2020-11-20T12:09:05.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.962+0000 7f7af759e700 1 -- start start 2020-11-20T12:09:05.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.963+0000 7f7af759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af8068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:05.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.963+0000 7f7af759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7af80b5f50 0x7f7af80b6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:05.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.963+0000 7f7af759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af8144b80 con 0x7f7af81482d0 2020-11-20T12:09:05.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.963+0000 7f7af759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af81489e0 con 0x7f7af80b5f50 2020-11-20T12:09:05.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.963+0000 7f7af759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8148c80 con 0x7f7af80b57b0 2020-11-20T12:09:05.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.964+0000 7f7af659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af8068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:05.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.964+0000 7f7af659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af8068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:45368/0 (socket says 172.21.15.187:45368) 2020-11-20T12:09:05.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.964+0000 7f7af659c700 1 -- 172.21.15.187:0/78417093 learned_addr learned my addr 172.21.15.187:0/78417093 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:05.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.964+0000 7f7af6d9d700 1 --2- 172.21.15.187:0/78417093 >> v2:172.21.15.5:3300/0 conn(0x7f7af80b5f50 0x7f7af80b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:05.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.965+0000 7f7af659c700 1 -- 172.21.15.187:0/78417093 >> v2:172.21.15.5:3300/0 conn(0x7f7af80b5f50 msgr2=0x7f7af80b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:05.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.965+0000 7f7af659c700 1 --2- 172.21.15.187:0/78417093 >> v2:172.21.15.5:3300/0 conn(0x7f7af80b5f50 0x7f7af80b6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.965+0000 7f7af659c700 1 -- 172.21.15.187:0/78417093 >> v1:172.21.15.5:6789/0 conn(0x7f7af81482d0 legacy=0x7f7af8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:05.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.965+0000 7f7af659c700 1 -- 172.21.15.187:0/78417093 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8148f20 con 0x7f7af80b57b0 2020-11-20T12:09:05.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.966+0000 7f7af659c700 1 --2- 172.21.15.187:0/78417093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af8068230 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7ae8006fd0 tx=0x7f7ae8006eb0).ready entity=mon.2 client_cookie=ca0797c548df0daa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:05.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.966+0000 7f7af559a700 1 -- 172.21.15.187:0/78417093 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8009810 con 0x7f7af80b57b0 2020-11-20T12:09:05.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.967+0000 7f7af559a700 1 -- 172.21.15.187:0/78417093 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae8009970 con 0x7f7af80b57b0 2020-11-20T12:09:05.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.967+0000 7f7af559a700 1 -- 172.21.15.187:0/78417093 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8009cf0 con 0x7f7af80b57b0 2020-11-20T12:09:05.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.967+0000 7f7af759e700 1 -- 172.21.15.187:0/78417093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 msgr2=0x7f7af8068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:05.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.967+0000 7f7af759e700 1 --2- 172.21.15.187:0/78417093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af8068230 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7ae8006fd0 tx=0x7f7ae8006eb0).stop 2020-11-20T12:09:05.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.967+0000 7f7af759e700 1 -- 172.21.15.187:0/78417093 shutdown_connections 2020-11-20T12:09:05.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.967+0000 7f7af759e700 1 --2- 172.21.15.187:0/78417093 >> v2:172.21.15.5:3300/0 conn(0x7f7af80b5f50 0x7f7af80b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.967+0000 7f7af759e700 1 --2- 172.21.15.187:0/78417093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af8068230 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.967+0000 7f7af759e700 1 -- 172.21.15.187:0/78417093 >> 172.21.15.187:0/78417093 conn(0x7f7af80bab30 msgr2=0x7f7af80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:05.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.968+0000 7f7af759e700 1 -- 172.21.15.187:0/78417093 shutdown_connections 2020-11-20T12:09:05.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.968+0000 7f7af759e700 1 -- 172.21.15.187:0/78417093 wait complete. 2020-11-20T12:09:05.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.969+0000 7f7af759e700 1 Processor -- start 2020-11-20T12:09:05.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.970+0000 7f7af759e700 1 -- start start 2020-11-20T12:09:05.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.970+0000 7f7af759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af81585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:05.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.970+0000 7f7af759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7af80b5f50 0x7f7af80b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:05.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7af81482d0 0x7f7af80b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:05.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80b4a70 con 0x7f7af80b5f50 2020-11-20T12:09:05.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80b47d0 con 0x7f7af81482d0 2020-11-20T12:09:05.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8158ac0 con 0x7f7af80b57b0 2020-11-20T12:09:05.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7af81482d0 0x7f7af80b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:05.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7af81482d0 0x7f7af80b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44488/0 (socket says 172.21.15.187:44488) 2020-11-20T12:09:05.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af6d9d700 1 -- 172.21.15.187:0/667120616 learned_addr learned my addr 172.21.15.187:0/667120616 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:05.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af659c700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af81585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:05.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.971+0000 7f7af5d9b700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7af80b5f50 0x7f7af80b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:05.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7af6d9d700 1 -- 172.21.15.187:0/667120616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 msgr2=0x7f7af81585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:05.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7af6d9d700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af81585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7af6d9d700 1 -- 172.21.15.187:0/667120616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7af80b5f50 msgr2=0x7f7af80b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:05.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7af6d9d700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7af80b5f50 0x7f7af80b76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:05.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7af6d9d700 1 -- 172.21.15.187:0/667120616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8158c80 con 0x7f7af81482d0 2020-11-20T12:09:05.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7af6d9d700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7af81482d0 0x7f7af80b3d30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f7af000ef50 tx=0x7f7af00091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:05.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7ae67fc700 1 -- 172.21.15.187:0/667120616 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af000cd20 con 0x7f7af81482d0 2020-11-20T12:09:05.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7ae67fc700 1 -- 172.21.15.187:0/667120616 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af000ce80 con 0x7f7af81482d0 2020-11-20T12:09:05.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af80b7bc0 con 0x7f7af81482d0 2020-11-20T12:09:05.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7ae67fc700 1 -- 172.21.15.187:0/667120616 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0012620 con 0x7f7af81482d0 2020-11-20T12:09:05.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.972+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af80b7ec0 con 0x7f7af81482d0 2020-11-20T12:09:05.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.974+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af80ba330 con 0x7f7af81482d0 2020-11-20T12:09:05.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.974+0000 7f7ae67fc700 1 -- 172.21.15.187:0/667120616 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7af0022070 con 0x7f7af81482d0 2020-11-20T12:09:05.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.975+0000 7f7ae67fc700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7acc056d20 0x7f7acc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:05.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.975+0000 7f7ae67fc700 1 -- 172.21.15.187:0/667120616 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f7af00706b0 con 0x7f7af81482d0 2020-11-20T12:09:05.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.975+0000 7f7af659c700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7acc056d20 0x7f7acc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:05.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.976+0000 7f7af659c700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7acc056d20 0x7f7acc0591e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7af81440d0 tx=0x7f7ae8017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:05.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:05.979+0000 7f7ae67fc700 1 -- 172.21.15.187:0/667120616 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7af0045cd0 con 0x7f7af81482d0 2020-11-20T12:09:06.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.166+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f7af8061a20 con 0x7f7af81482d0 2020-11-20T12:09:06.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.708+0000 7f7ae67fc700 1 -- 172.21.15.187:0/667120616 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v57) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7af0016070 con 0x7f7af81482d0 2020-11-20T12:09:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7acc056d20 msgr2=0x7f7acc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7acc056d20 0x7f7acc0591e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7af81440d0 tx=0x7f7ae8017000).stop 2020-11-20T12:09:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7af81482d0 msgr2=0x7f7af80b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7af81482d0 0x7f7af80b3d30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f7af000ef50 tx=0x7f7af00091d0).stop 2020-11-20T12:09:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 shutdown_connections 2020-11-20T12:09:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7acc056d20 0x7f7acc0591e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7af81482d0 0x7f7af80b3d30 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7af80b57b0 0x7f7af81585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:06.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 --2- 172.21.15.187:0/667120616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7af80b5f50 0x7f7af80b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:06.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.722+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 >> 172.21.15.187:0/667120616 conn(0x7f7af80bab30 msgr2=0x7f7af80b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:06.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.724+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 shutdown_connections 2020-11-20T12:09:06.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.724+0000 7f7af759e700 1 -- 172.21.15.187:0/667120616 wait complete. 2020-11-20T12:09:06.731 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:09:06.756 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:09:06.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2020-11-20T12:09:06.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.989+0000 7fd1fe292700 1 Processor -- start 2020-11-20T12:09:06.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.989+0000 7fd1fe292700 1 -- start start 2020-11-20T12:09:06.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.991+0000 7fd1fe292700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1f81484d0 0x7fd1f8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:06.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.991+0000 7fd1fe292700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 0x7fd1f80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:06.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.991+0000 7fd1fe292700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f80b1300 con 0x7fd1f8066f70 2020-11-20T12:09:06.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.991+0000 7fd1fe292700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f8067730 con 0x7fd1f81484d0 2020-11-20T12:09:06.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.991+0000 7fd1fe292700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f80b15a0 con 0x7fd1f80622c0 2020-11-20T12:09:06.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.991+0000 7fd1fd290700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1f81484d0 0x7fd1f8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:06.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.991+0000 7fd1fd290700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1f81484d0 0x7fd1f8061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44492/0 (socket says 172.21.15.187:44492) 2020-11-20T12:09:06.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.991+0000 7fd1fd290700 1 -- 172.21.15.187:0/4280026958 learned_addr learned my addr 172.21.15.187:0/4280026958 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:06.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.992+0000 7fd1fca8f700 1 --2- 172.21.15.187:0/4280026958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 0x7fd1f80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:06.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.992+0000 7fd1fd290700 1 -- 172.21.15.187:0/4280026958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 msgr2=0x7fd1f80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:06.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.992+0000 7fd1fd290700 1 --2- 172.21.15.187:0/4280026958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 0x7fd1f80b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:06.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.992+0000 7fd1fd290700 1 -- 172.21.15.187:0/4280026958 >> v1:172.21.15.187:6789/0 conn(0x7fd1f8066f70 legacy=0x7fd1f80b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:06.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.992+0000 7fd1fd290700 1 -- 172.21.15.187:0/4280026958 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f80b1840 con 0x7fd1f81484d0 2020-11-20T12:09:06.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.992+0000 7fd1fd290700 1 --2- 172.21.15.187:0/4280026958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f81484d0 0x7fd1f8061dd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fd1f4006fd0 tx=0x7fd1f4006eb0).ready entity=mon.1 client_cookie=fb702d8295bb04eb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:06.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.993+0000 7fd1e7fff700 1 -- 172.21.15.187:0/4280026958 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40098d0 con 0x7fd1f81484d0 2020-11-20T12:09:07.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.993+0000 7fd1e7fff700 1 -- 172.21.15.187:0/4280026958 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f4009a30 con 0x7fd1f81484d0 2020-11-20T12:09:07.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.993+0000 7fd1e7fff700 1 -- 172.21.15.187:0/4280026958 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4009db0 con 0x7fd1f81484d0 2020-11-20T12:09:07.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.993+0000 7fd1fe292700 1 -- 172.21.15.187:0/4280026958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f81484d0 msgr2=0x7fd1f8061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.993+0000 7fd1fe292700 1 --2- 172.21.15.187:0/4280026958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f81484d0 0x7fd1f8061dd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fd1f4006fd0 tx=0x7fd1f4006eb0).stop 2020-11-20T12:09:07.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.994+0000 7fd1fe292700 1 -- 172.21.15.187:0/4280026958 shutdown_connections 2020-11-20T12:09:07.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.994+0000 7fd1fe292700 1 --2- 172.21.15.187:0/4280026958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f81484d0 0x7fd1f8061dd0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:07.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.994+0000 7fd1fe292700 1 --2- 172.21.15.187:0/4280026958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 0x7fd1f80b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:07.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.994+0000 7fd1fe292700 1 -- 172.21.15.187:0/4280026958 >> 172.21.15.187:0/4280026958 conn(0x7fd1f80bab30 msgr2=0x7fd1f80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:07.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.994+0000 7fd1fe292700 1 -- 172.21.15.187:0/4280026958 shutdown_connections 2020-11-20T12:09:07.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.994+0000 7fd1fe292700 1 -- 172.21.15.187:0/4280026958 wait complete. 2020-11-20T12:09:07.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.995+0000 7fd1fe292700 1 Processor -- start 2020-11-20T12:09:07.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.995+0000 7fd1fe292700 1 -- start start 2020-11-20T12:09:07.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.996+0000 7fd1fe292700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 0x7fd1f809ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.996+0000 7fd1fe292700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1f8066f70 0x7fd1f80a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.996+0000 7fd1fe292700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f815ba20 0x7fd1f81431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.996+0000 7fd1fe292700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f809f1b0 con 0x7fd1f8066f70 2020-11-20T12:09:07.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.996+0000 7fd1fe292700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f809f450 con 0x7fd1f815ba20 2020-11-20T12:09:07.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.996+0000 7fd1fe292700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f809f6f0 con 0x7fd1f80622c0 2020-11-20T12:09:07.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f815ba20 0x7fd1f81431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:07.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f815ba20 0x7fd1f81431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44502/0 (socket says 172.21.15.187:44502) 2020-11-20T12:09:07.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 -- 172.21.15.187:0/1294569312 learned_addr learned my addr 172.21.15.187:0/1294569312 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:07.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 -- 172.21.15.187:0/1294569312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 msgr2=0x7fd1f809ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 0x7fd1f809ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:07.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 -- 172.21.15.187:0/1294569312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1f8066f70 msgr2=0x7fd1f80a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1f8066f70 0x7fd1f80a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:07.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 -- 172.21.15.187:0/1294569312 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f809d520 con 0x7fd1f815ba20 2020-11-20T12:09:07.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1fda91700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f815ba20 0x7fd1f81431a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7fd1f000ca90 tx=0x7fd1f0001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:07.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.997+0000 7fd1e57fa700 1 -- 172.21.15.187:0/1294569312 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f000a730 con 0x7fd1f815ba20 2020-11-20T12:09:07.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.998+0000 7fd1e57fa700 1 -- 172.21.15.187:0/1294569312 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f000a890 con 0x7fd1f815ba20 2020-11-20T12:09:07.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.998+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f80a09a0 con 0x7fd1f815ba20 2020-11-20T12:09:07.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.998+0000 7fd1e57fa700 1 -- 172.21.15.187:0/1294569312 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f000ac10 con 0x7fd1f815ba20 2020-11-20T12:09:07.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.998+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f8143690 con 0x7fd1f815ba20 2020-11-20T12:09:07.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.999+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f809f8b0 con 0x7fd1f815ba20 2020-11-20T12:09:07.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:06.999+0000 7fd1e57fa700 1 -- 172.21.15.187:0/1294569312 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd1f0010070 con 0x7fd1f815ba20 2020-11-20T12:09:07.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.000+0000 7fd1e57fa700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1d0056d50 0x7fd1d0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.000+0000 7fd1e57fa700 1 -- 172.21.15.187:0/1294569312 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fd1f006fd70 con 0x7fd1f815ba20 2020-11-20T12:09:07.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.000+0000 7fd1fd290700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1d0056d50 0x7fd1d0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:07.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.001+0000 7fd1fd290700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1d0056d50 0x7fd1d0059210 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fd1f8066380 tx=0x7fd1f4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:07.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.004+0000 7fd1e57fa700 1 -- 172.21.15.187:0/1294569312 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd1f0045120 con 0x7fd1f815ba20 2020-11-20T12:09:07.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.253+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fd1f8061a20 con 0x7fd1f815ba20 2020-11-20T12:09:07.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.267+0000 7fd1e57fa700 1 -- 172.21.15.187:0/1294569312 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 11 and data pool 12 v31) v1 ==== 189+0+0 (secure 0 0 0) 0x7fd1f0048740 con 0x7fd1f815ba20 2020-11-20T12:09:07.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.278+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1d0056d50 msgr2=0x7fd1d0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.278+0000 7fd1fe292700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1d0056d50 0x7fd1d0059210 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fd1f8066380 tx=0x7fd1f4015000).stop 2020-11-20T12:09:07.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.278+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f815ba20 msgr2=0x7fd1f81431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.278+0000 7fd1fe292700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f815ba20 0x7fd1f81431a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7fd1f000ca90 tx=0x7fd1f0001360).stop 2020-11-20T12:09:07.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.279+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 shutdown_connections 2020-11-20T12:09:07.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.279+0000 7fd1fe292700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1d0056d50 0x7fd1d0059210 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0x7fd1f8066380 tx=0x7fd1f4015000).stop 2020-11-20T12:09:07.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.279+0000 7fd1fe292700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1f815ba20 0x7fd1f81431a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:07.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.279+0000 7fd1fe292700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd1f80622c0 0x7fd1f809ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:07.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.279+0000 7fd1fe292700 1 --2- 172.21.15.187:0/1294569312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1f8066f70 0x7fd1f80a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:07.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.279+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 >> 172.21.15.187:0/1294569312 conn(0x7fd1f80bab30 msgr2=0x7fd1f80b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:07.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.280+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 shutdown_connections 2020-11-20T12:09:07.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.282+0000 7fd1fe292700 1 -- 172.21.15.187:0/1294569312 wait complete. 2020-11-20T12:09:07.289 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 11 and data pool 12 2020-11-20T12:09:07.312 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:09:07.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.519+0000 7f5e5dc2b700 1 Processor -- start 2020-11-20T12:09:07.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.519+0000 7f5e5dc2b700 1 -- start start 2020-11-20T12:09:07.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.520+0000 7f5e5dc2b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5e58065190 0x7f5e58065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.520+0000 7f5e5dc2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 0x7f5e58143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.520+0000 7f5e5dc2b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e580b13e0 con 0x7f5e58143c70 2020-11-20T12:09:07.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.520+0000 7f5e5dc2b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5814ab30 con 0x7f5e58065190 2020-11-20T12:09:07.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.520+0000 7f5e5dc2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58062c10 con 0x7f5e58065f90 2020-11-20T12:09:07.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.521+0000 7f5e5cc29700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5e58065190 0x7f5e58065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:07.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.521+0000 7f5e5cc29700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5e58065190 0x7f5e58065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44516/0 (socket says 172.21.15.187:44516) 2020-11-20T12:09:07.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.521+0000 7f5e5cc29700 1 -- 172.21.15.187:0/744814011 learned_addr learned my addr 172.21.15.187:0/744814011 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:07.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.521+0000 7f5e57fff700 1 --2- 172.21.15.187:0/744814011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 0x7f5e58143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:07.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.522+0000 7f5e5cc29700 1 -- 172.21.15.187:0/744814011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 msgr2=0x7f5e58143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.522+0000 7f5e5cc29700 1 --2- 172.21.15.187:0/744814011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 0x7f5e58143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.522+0000 7f5e5cc29700 1 -- 172.21.15.187:0/744814011 >> v1:172.21.15.5:6789/0 conn(0x7f5e58143c70 legacy=0x7f5e5809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.522+0000 7f5e5cc29700 1 -- 172.21.15.187:0/744814011 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e580b1640 con 0x7f5e58065190 2020-11-20T12:09:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.522+0000 7f5e5cc29700 1 --2- 172.21.15.187:0/744814011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58065570 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f5e5814ac70 tx=0x7f5e4c006eb0).ready entity=mon.1 client_cookie=887c2e847fb59758 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.522+0000 7f5e577fe700 1 -- 172.21.15.187:0/744814011 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c009920 con 0x7f5e58065190 2020-11-20T12:09:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.522+0000 7f5e577fe700 1 -- 172.21.15.187:0/744814011 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c009a80 con 0x7f5e58065190 2020-11-20T12:09:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.522+0000 7f5e577fe700 1 -- 172.21.15.187:0/744814011 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c009e00 con 0x7f5e58065190 2020-11-20T12:09:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.523+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/744814011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 msgr2=0x7f5e58065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.523+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/744814011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58065570 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f5e5814ac70 tx=0x7f5e4c006eb0).stop 2020-11-20T12:09:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.523+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/744814011 shutdown_connections 2020-11-20T12:09:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.523+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/744814011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58065570 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.523+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/744814011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 0x7f5e58143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.523+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/744814011 >> 172.21.15.187:0/744814011 conn(0x7f5e580bab30 msgr2=0x7f5e580baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.523+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/744814011 shutdown_connections 2020-11-20T12:09:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.523+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/744814011 wait complete. 2020-11-20T12:09:07.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.525+0000 7f5e5dc2b700 1 Processor -- start 2020-11-20T12:09:07.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.525+0000 7f5e5dc2b700 1 -- start start 2020-11-20T12:09:07.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.526+0000 7f5e5dc2b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.526+0000 7f5e5dc2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 0x7f5e5815bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.526+0000 7f5e5dc2b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e58143c70 0x7f5e58148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.526+0000 7f5e5dc2b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580a0990 con 0x7f5e58143c70 2020-11-20T12:09:07.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.526+0000 7f5e5dc2b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5815c4a0 con 0x7f5e58065190 2020-11-20T12:09:07.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.526+0000 7f5e5dc2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5815c740 con 0x7f5e58065f90 2020-11-20T12:09:07.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:07.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44522/0 (socket says 172.21.15.187:44522) 2020-11-20T12:09:07.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 -- 172.21.15.187:0/479743749 learned_addr learned my addr 172.21.15.187:0/479743749 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:07.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 -- 172.21.15.187:0/479743749 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 msgr2=0x7f5e5815bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 0x7f5e5815bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:07.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 -- 172.21.15.187:0/479743749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e58143c70 msgr2=0x7f5e58148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e58143c70 0x7f5e58148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:07.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 -- 172.21.15.187:0/479743749 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5815c050 con 0x7f5e58065190 2020-11-20T12:09:07.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.527+0000 7f5e5cc29700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58148dd0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f5e4c006ef0 tx=0x7f5e4c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:07.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.528+0000 7f5e54ff9700 1 -- 172.21.15.187:0/479743749 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c01e460 con 0x7f5e58065190 2020-11-20T12:09:07.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.528+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5815c240 con 0x7f5e58065190 2020-11-20T12:09:07.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.528+0000 7f5e54ff9700 1 -- 172.21.15.187:0/479743749 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c01e5c0 con 0x7f5e58065190 2020-11-20T12:09:07.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.528+0000 7f5e54ff9700 1 -- 172.21.15.187:0/479743749 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c01e960 con 0x7f5e58065190 2020-11-20T12:09:07.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.528+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e58148b70 con 0x7f5e58065190 2020-11-20T12:09:07.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.529+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e58061a20 con 0x7f5e58065190 2020-11-20T12:09:07.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.530+0000 7f5e54ff9700 1 -- 172.21.15.187:0/479743749 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5e4c01a040 con 0x7f5e58065190 2020-11-20T12:09:07.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.530+0000 7f5e54ff9700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e30056d20 0x7f5e300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:07.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.530+0000 7f5e54ff9700 1 -- 172.21.15.187:0/479743749 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5e4c0715f0 con 0x7f5e58065190 2020-11-20T12:09:07.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.531+0000 7f5e57fff700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e30056d20 0x7f5e300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:07.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.531+0000 7f5e57fff700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e30056d20 0x7f5e300591e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f5e5815b900 tx=0x7f5e44008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:07.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.534+0000 7f5e54ff9700 1 -- 172.21.15.187:0/479743749 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5e4c046af0 con 0x7f5e58065190 2020-11-20T12:09:07.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.719+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e580afc60 con 0x7f5e58065190 2020-11-20T12:09:07.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.722+0000 7f5e54ff9700 1 -- 172.21.15.187:0/479743749 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16794 (secure 0 0 0) 0x7f5e4c023020 con 0x7f5e58065190 2020-11-20T12:09:07.727 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:07.727 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":58,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:09:07.256472+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:09:05.207488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2020-11-20T12:09:06.244532+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:09:07.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.724+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e30056d20 msgr2=0x7f5e300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.724+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e30056d20 0x7f5e300591e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f5e5815b900 tx=0x7f5e44008040).stop 2020-11-20T12:09:07.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 msgr2=0x7f5e58148dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:07.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58148dd0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f5e4c006ef0 tx=0x7f5e4c015000).stop 2020-11-20T12:09:07.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 shutdown_connections 2020-11-20T12:09:07.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e30056d20 0x7f5e300591e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:07.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e58143c70 0x7f5e58148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:07.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e58065190 0x7f5e58148dd0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:07.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 --2- 172.21.15.187:0/479743749 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e58065f90 0x7f5e5815bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:07.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 >> 172.21.15.187:0/479743749 conn(0x7f5e580bab30 msgr2=0x7f5e580b9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:07.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 shutdown_connections 2020-11-20T12:09:07.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:07.725+0000 7f5e5dc2b700 1 -- 172.21.15.187:0/479743749 wait complete. 2020-11-20T12:09:07.750 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:09:08.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.002+0000 7f7495d3b700 1 Processor -- start 2020-11-20T12:09:08.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.002+0000 7f7495d3b700 1 -- start start 2020-11-20T12:09:08.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.004+0000 7f7495d3b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74900b6650 0x7f74900afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.004+0000 7f7495d3b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900b1300 con 0x7f7490066f70 2020-11-20T12:09:08.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.004+0000 7f7495d3b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900b15a0 con 0x7f74901484d0 2020-11-20T12:09:08.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.004+0000 7f7495d3b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7490067730 con 0x7f74900b6650 2020-11-20T12:09:08.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.004+0000 7f7487fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74900b6650 0x7f74900afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.004+0000 7f7487fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74900b6650 0x7f74900afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44532/0 (socket says 172.21.15.187:44532) 2020-11-20T12:09:08.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.004+0000 7f7487fff700 1 -- 172.21.15.187:0/3999519808 learned_addr learned my addr 172.21.15.187:0/3999519808 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:08.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.004+0000 7f7494d39700 1 --1- 172.21.15.187:0/3999519808 >> v1:172.21.15.187:6789/0 conn(0x7f74901484d0 0x7f7490061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41136/0 (socket says 172.21.15.187:41136) 2020-11-20T12:09:08.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.005+0000 7f7487fff700 1 -- 172.21.15.187:0/3999519808 >> v1:172.21.15.187:6789/0 conn(0x7f74901484d0 legacy=0x7f7490061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:08.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.005+0000 7f7487fff700 1 -- 172.21.15.187:0/3999519808 >> v1:172.21.15.5:6789/0 conn(0x7f7490066f70 legacy=0x7f74900aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.005+0000 7f7487fff700 1 -- 172.21.15.187:0/3999519808 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900b1840 con 0x7f74900b6650 2020-11-20T12:09:08.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.005+0000 7f7487fff700 1 --2- 172.21.15.187:0/3999519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74900b6650 0x7f74900afa90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f7480006fd0 tx=0x7f7480006eb0).ready entity=mon.1 client_cookie=44a30d3546e53d52 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:08.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.006+0000 7f74877fe700 1 -- 172.21.15.187:0/3999519808 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800098d0 con 0x7f74900b6650 2020-11-20T12:09:08.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.006+0000 7f74877fe700 1 -- 172.21.15.187:0/3999519808 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7480009a30 con 0x7f74900b6650 2020-11-20T12:09:08.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.006+0000 7f74877fe700 1 -- 172.21.15.187:0/3999519808 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480009db0 con 0x7f74900b6650 2020-11-20T12:09:08.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.006+0000 7f7495d3b700 1 -- 172.21.15.187:0/3999519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74900b6650 msgr2=0x7f74900afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.006+0000 7f7495d3b700 1 --2- 172.21.15.187:0/3999519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74900b6650 0x7f74900afa90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f7480006fd0 tx=0x7f7480006eb0).stop 2020-11-20T12:09:08.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.007+0000 7f7495d3b700 1 -- 172.21.15.187:0/3999519808 shutdown_connections 2020-11-20T12:09:08.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.007+0000 7f7495d3b700 1 --2- 172.21.15.187:0/3999519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74900b6650 0x7f74900afa90 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.007+0000 7f7495d3b700 1 -- 172.21.15.187:0/3999519808 >> 172.21.15.187:0/3999519808 conn(0x7f74900bab30 msgr2=0x7f74900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:08.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.007+0000 7f7495d3b700 1 -- 172.21.15.187:0/3999519808 shutdown_connections 2020-11-20T12:09:08.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.007+0000 7f7495d3b700 1 -- 172.21.15.187:0/3999519808 wait complete. 2020-11-20T12:09:08.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.008+0000 7f7495d3b700 1 Processor -- start 2020-11-20T12:09:08.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.008+0000 7f7495d3b700 1 -- start start 2020-11-20T12:09:08.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.009+0000 7f7495d3b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7490066f70 0x7f749013e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.009+0000 7f7495d3b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74900b6650 0x7f749009c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.009+0000 7f7495d3b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74901484d0 0x7f749009fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.009+0000 7f7495d3b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749009cbf0 con 0x7f74900b6650 2020-11-20T12:09:08.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.009+0000 7f7495d3b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749009ce90 con 0x7f74901484d0 2020-11-20T12:09:08.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.009+0000 7f7495d3b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749013c100 con 0x7f7490066f70 2020-11-20T12:09:08.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f749553a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74901484d0 0x7f749009fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f749553a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74901484d0 0x7f749009fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44540/0 (socket says 172.21.15.187:44540) 2020-11-20T12:09:08.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f749553a700 1 -- 172.21.15.187:0/3189244606 learned_addr learned my addr 172.21.15.187:0/3189244606 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:08.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f7487fff700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74900b6650 0x7f749009c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f749553a700 1 -- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7490066f70 msgr2=0x7f749013e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f7494d39700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7490066f70 0x7f749013e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f749553a700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7490066f70 0x7f749013e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f749553a700 1 -- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74900b6650 msgr2=0x7f749009c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f749553a700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74900b6650 0x7f749009c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f749553a700 1 -- 172.21.15.187:0/3189244606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749009f7b0 con 0x7f74901484d0 2020-11-20T12:09:08.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.010+0000 7f7487fff700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74900b6650 0x7f749009c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:08.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.011+0000 7f749553a700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74901484d0 0x7f749009fc10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f748800ca90 tx=0x7f7488007f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:08.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.011+0000 7f7484ff9700 1 -- 172.21.15.187:0/3189244606 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748800a7f0 con 0x7f74901484d0 2020-11-20T12:09:08.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.011+0000 7f7484ff9700 1 -- 172.21.15.187:0/3189244606 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748800a950 con 0x7f74901484d0 2020-11-20T12:09:08.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.011+0000 7f7484ff9700 1 -- 172.21.15.187:0/3189244606 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748800acd0 con 0x7f74901484d0 2020-11-20T12:09:08.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.011+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749013e5f0 con 0x7f74901484d0 2020-11-20T12:09:08.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.011+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749009c920 con 0x7f74901484d0 2020-11-20T12:09:08.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.012+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749013c2c0 con 0x7f74901484d0 2020-11-20T12:09:08.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.013+0000 7f7484ff9700 1 -- 172.21.15.187:0/3189244606 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7488019030 con 0x7f74901484d0 2020-11-20T12:09:08.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.013+0000 7f7484ff9700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7468056d50 0x7f7468059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.013+0000 7f7484ff9700 1 -- 172.21.15.187:0/3189244606 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f748806f780 con 0x7f74901484d0 2020-11-20T12:09:08.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.014+0000 7f7494d39700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7468056d50 0x7f7468059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.014+0000 7f7494d39700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7468056d50 0x7f7468059210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f748c007fd0 tx=0x7f748c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:08.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.017+0000 7f7484ff9700 1 -- 172.21.15.187:0/3189244606 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7488043fd0 con 0x7f74901484d0 2020-11-20T12:09:08.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.206+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7490061a20 con 0x7f74901484d0 2020-11-20T12:09:08.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.209+0000 7f7484ff9700 1 -- 172.21.15.187:0/3189244606 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16794 (secure 0 0 0) 0x7f74880251b0 con 0x7f74901484d0 2020-11-20T12:09:08.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.211+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7468056d50 msgr2=0x7f7468059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.211+0000 7f7495d3b700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7468056d50 0x7f7468059210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f748c007fd0 tx=0x7f748c009040).stop 2020-11-20T12:09:08.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.211+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74901484d0 msgr2=0x7f749009fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.211+0000 7f7495d3b700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74901484d0 0x7f749009fc10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f748800ca90 tx=0x7f7488007f10).stop 2020-11-20T12:09:08.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.211+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 shutdown_connections 2020-11-20T12:09:08.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.211+0000 7f7495d3b700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7468056d50 0x7f7468059210 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.212+0000 7f7495d3b700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74901484d0 0x7f749009fc10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.212+0000 7f7495d3b700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7490066f70 0x7f749013e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.212+0000 7f7495d3b700 1 --2- 172.21.15.187:0/3189244606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74900b6650 0x7f749009c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.212+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 >> 172.21.15.187:0/3189244606 conn(0x7f74900bab30 msgr2=0x7f74900afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:08.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.212+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 shutdown_connections 2020-11-20T12:09:08.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.212+0000 7f7495d3b700 1 -- 172.21.15.187:0/3189244606 wait complete. 2020-11-20T12:09:08.353 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:08.354 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":58,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:09:07.256472+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:09:05.207488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2020-11-20T12:09:06.244532+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:09:08.355 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:09:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.606+0000 7f314b59e700 1 Processor -- start 2020-11-20T12:09:08.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.606+0000 7f314b59e700 1 -- start start 2020-11-20T12:09:08.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.607+0000 7f314b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f314c0b1380 0x7f314c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.607+0000 7f314b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b1760 0x7f314c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.607+0000 7f314b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314c150390 con 0x7f314c065680 2020-11-20T12:09:08.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.607+0000 7f314b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f314c150630 con 0x7f314c0b1380 2020-11-20T12:09:08.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.607+0000 7f314b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c068510 con 0x7f314c0b1760 2020-11-20T12:09:08.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.608+0000 7f314ad9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f314c065680 0x7f314c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41152/0 (socket says 172.21.15.187:41152) 2020-11-20T12:09:08.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.608+0000 7f314ad9d700 1 -- 172.21.15.187:0/2100866189 learned_addr learned my addr 172.21.15.187:0/2100866189 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:08.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.608+0000 7f3149d9b700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b1760 0x7f314c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.608+0000 7f314a59c700 1 --2- 172.21.15.187:0/2100866189 >> v2:172.21.15.5:3300/0 conn(0x7f314c0b1380 0x7f314c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.608+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335321028 0 0) 0x7f314c150390 con 0x7f314c065680 2020-11-20T12:09:08.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.608+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3130003240 con 0x7f314c065680 2020-11-20T12:09:08.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 161276063 0 0) 0x7f3130003240 con 0x7f314c065680 2020-11-20T12:09:08.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b1760 msgr2=0x7f314c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:08.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b1760 0x7f314c065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 >> v2:172.21.15.5:3300/0 conn(0x7f314c0b1380 msgr2=0x7f314c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:08.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 --2- 172.21.15.187:0/2100866189 >> v2:172.21.15.5:3300/0 conn(0x7f314c0b1380 0x7f314c150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c150fc0 con 0x7f314c065680 2020-11-20T12:09:08.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f3144001860 con 0x7f314c065680 2020-11-20T12:09:08.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 >> v1:172.21.15.187:6789/0 conn(0x7f314c065680 legacy=0x7f314c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f31300037b0 0x7f3130003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b1760 0x7f31300044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31300049d0 0x7f3130004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c150630 con 0x7f31300037b0 2020-11-20T12:09:08.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c068510 con 0x7f31300049d0 2020-11-20T12:09:08.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31300042a0 con 0x7f314c0b1760 2020-11-20T12:09:08.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.609+0000 7f314ad9d700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31300049d0 0x7f3130004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.610+0000 7f314a59c700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f31300037b0 0x7f3130003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.610+0000 7f3149d9b700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b1760 0x7f31300044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.610+0000 7f314ad9d700 1 -- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b1760 msgr2=0x7f31300044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:08.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.610+0000 7f314ad9d700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b1760 0x7f31300044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.610+0000 7f314ad9d700 1 -- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f31300037b0 msgr2=0x7f3130003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:08.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.610+0000 7f314ad9d700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f31300037b0 0x7f3130003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.610+0000 7f314ad9d700 1 -- 172.21.15.187:0/2100866189 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130003240 con 0x7f31300049d0 2020-11-20T12:09:08.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.610+0000 7f314ad9d700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31300049d0 0x7f3130004e50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f314400c5d0 tx=0x7f3144007ad0).ready entity=mon.1 client_cookie=91159a3cfe2e2cff server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:08.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144005b80 con 0x7f31300049d0 2020-11-20T12:09:08.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3144005ce0 con 0x7f31300049d0 2020-11-20T12:09:08.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314959a700 1 -- 172.21.15.187:0/2100866189 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144007320 con 0x7f31300049d0 2020-11-20T12:09:08.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314b59e700 1 -- 172.21.15.187:0/2100866189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31300049d0 msgr2=0x7f3130004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314b59e700 1 --2- 172.21.15.187:0/2100866189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31300049d0 0x7f3130004e50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f314400c5d0 tx=0x7f3144007ad0).stop 2020-11-20T12:09:08.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314ad9d700 1 -- 172.21.15.187:0/2100866189 reap_dead start 2020-11-20T12:09:08.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314b59e700 1 -- 172.21.15.187:0/2100866189 shutdown_connections 2020-11-20T12:09:08.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314b59e700 1 -- 172.21.15.187:0/2100866189 >> 172.21.15.187:0/2100866189 conn(0x7f314c0bab30 msgr2=0x7f314c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:08.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314b59e700 1 -- 172.21.15.187:0/2100866189 shutdown_connections 2020-11-20T12:09:08.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.611+0000 7f314b59e700 1 -- 172.21.15.187:0/2100866189 wait complete. 2020-11-20T12:09:08.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.613+0000 7f314b59e700 1 Processor -- start 2020-11-20T12:09:08.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.613+0000 7f314b59e700 1 -- start start 2020-11-20T12:09:08.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f314b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b2ed0 0x7f314c14cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f314b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f314c0b32b0 0x7f314c0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f314b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f314c0b1b40 0x7f314c09dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f314b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c14c1b0 con 0x7f314c0b1b40 2020-11-20T12:09:08.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f314b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c14c450 con 0x7f314c0b32b0 2020-11-20T12:09:08.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f314b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c14c6f0 con 0x7f314c0b2ed0 2020-11-20T12:09:08.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f3149d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f314c0b32b0 0x7f314c0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f3149d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f314c0b32b0 0x7f314c0b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44558/0 (socket says 172.21.15.187:44558) 2020-11-20T12:09:08.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.614+0000 7f3149d9b700 1 -- 172.21.15.187:0/106347815 learned_addr learned my addr 172.21.15.187:0/106347815 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:08.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f314a59c700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b2ed0 0x7f314c14cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f314ad9d700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f314c0b1b40 0x7f314c09dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f3149d9b700 1 -- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b2ed0 msgr2=0x7f314c14cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f3149d9b700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b2ed0 0x7f314c14cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f3149d9b700 1 -- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f314c0b1b40 msgr2=0x7f314c09dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:08.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f3149d9b700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f314c0b1b40 0x7f314c09dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:08.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f3149d9b700 1 -- 172.21.15.187:0/106347815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c09dfd0 con 0x7f314c0b32b0 2020-11-20T12:09:08.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f314ad9d700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f314c0b1b40 0x7f314c09dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:08.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f3149d9b700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f314c0b32b0 0x7f314c0b2250 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f3140009720 tx=0x7f3140008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:08.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.615+0000 7f313a7fc700 1 -- 172.21.15.187:0/106347815 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31400133f0 con 0x7f314c0b32b0 2020-11-20T12:09:08.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.616+0000 7f313a7fc700 1 -- 172.21.15.187:0/106347815 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3140013550 con 0x7f314c0b32b0 2020-11-20T12:09:08.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.616+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314c09e220 con 0x7f314c0b32b0 2020-11-20T12:09:08.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.616+0000 7f313a7fc700 1 -- 172.21.15.187:0/106347815 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31400138d0 con 0x7f314c0b32b0 2020-11-20T12:09:08.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.616+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c09e7e0 con 0x7f314c0b32b0 2020-11-20T12:09:08.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.617+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314c14c8b0 con 0x7f314c0b32b0 2020-11-20T12:09:08.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.617+0000 7f313a7fc700 1 -- 172.21.15.187:0/106347815 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3140019020 con 0x7f314c0b32b0 2020-11-20T12:09:08.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.618+0000 7f313a7fc700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3120056d20 0x7f31200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:08.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.618+0000 7f313a7fc700 1 -- 172.21.15.187:0/106347815 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f314006fa70 con 0x7f314c0b32b0 2020-11-20T12:09:08.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.618+0000 7f314a59c700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3120056d20 0x7f31200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:08.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.619+0000 7f314a59c700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3120056d20 0x7f31200591e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f313c006a90 tx=0x7f313c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:08.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.622+0000 7f313a7fc700 1 -- 172.21.15.187:0/106347815 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3140044d50 con 0x7f314c0b32b0 2020-11-20T12:09:08.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:08.868+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f314c09e3b0 con 0x7f314c0b32b0 2020-11-20T12:09:09.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.288+0000 7f313a7fc700 1 -- 172.21.15.187:0/106347815 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) v1 ==== 118+0+0 (secure 0 0 0) 0x7f314c09e3b0 con 0x7f314c0b32b0 2020-11-20T12:09:09.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.291+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3120056d20 msgr2=0x7f31200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:09.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.291+0000 7f314b59e700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3120056d20 0x7f31200591e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f313c006a90 tx=0x7f313c008040).stop 2020-11-20T12:09:09.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.291+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f314c0b32b0 msgr2=0x7f314c0b2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:09.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.291+0000 7f314b59e700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f314c0b32b0 0x7f314c0b2250 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f3140009720 tx=0x7f3140008040).stop 2020-11-20T12:09:09.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.292+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 shutdown_connections 2020-11-20T12:09:09.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.292+0000 7f314b59e700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3120056d20 0x7f31200591e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.292+0000 7f314b59e700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f314c0b1b40 0x7f314c09dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.292+0000 7f314b59e700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f314c0b2ed0 0x7f314c14cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.292+0000 7f314b59e700 1 --2- 172.21.15.187:0/106347815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f314c0b32b0 0x7f314c0b2250 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.292+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 >> 172.21.15.187:0/106347815 conn(0x7f314c0bab30 msgr2=0x7f314c14b670 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:09.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.292+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 shutdown_connections 2020-11-20T12:09:09.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.292+0000 7f314b59e700 1 -- 172.21.15.187:0/106347815 wait complete. 2020-11-20T12:09:09.325 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:09.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.558+0000 7f20098af700 1 Processor -- start 2020-11-20T12:09:09.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.558+0000 7f20098af700 1 -- start start 2020-11-20T12:09:09.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.560+0000 7f20098af700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040bda00 0x7f20040b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:09.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.560+0000 7f20098af700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20040a2770 0x7f20040a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:09.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.560+0000 7f20098af700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2004146c40 con 0x7f20040bd620 2020-11-20T12:09:09.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.560+0000 7f20098af700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20040b4250 con 0x7f20040a2770 2020-11-20T12:09:09.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.560+0000 7f20098af700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20040b44f0 con 0x7f20040bda00 2020-11-20T12:09:09.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.561+0000 7f20090ae700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20040a2770 0x7f20040a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:09.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.561+0000 7f1ffbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040bda00 0x7f20040b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:09.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.561+0000 7f1ffbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040bda00 0x7f20040b6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:45454/0 (socket says 172.21.15.187:45454) 2020-11-20T12:09:09.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.561+0000 7f1ffbfff700 1 -- 172.21.15.187:0/2906348544 learned_addr learned my addr 172.21.15.187:0/2906348544 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:09.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.562+0000 7f1ffbfff700 1 -- 172.21.15.187:0/2906348544 >> v2:172.21.15.5:3300/0 conn(0x7f20040a2770 msgr2=0x7f20040a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:09.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.562+0000 7f1ffbfff700 1 --2- 172.21.15.187:0/2906348544 >> v2:172.21.15.5:3300/0 conn(0x7f20040a2770 0x7f20040a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.562+0000 7f1ffbfff700 1 -- 172.21.15.187:0/2906348544 >> v1:172.21.15.5:6789/0 conn(0x7f20040bd620 legacy=0x7f2004146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:09.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.562+0000 7f1ffbfff700 1 -- 172.21.15.187:0/2906348544 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004149b40 con 0x7f20040bda00 2020-11-20T12:09:09.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.562+0000 7f1ffbfff700 1 --2- 172.21.15.187:0/2906348544 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040bda00 0x7f20040b6dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1ff4006fd0 tx=0x7f1ff4006eb0).ready entity=mon.2 client_cookie=12a99d2a3077132 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:09.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.563+0000 7f1ffb7fe700 1 -- 172.21.15.187:0/2906348544 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40098a0 con 0x7f20040bda00 2020-11-20T12:09:09.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.563+0000 7f1ffb7fe700 1 -- 172.21.15.187:0/2906348544 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4009a00 con 0x7f20040bda00 2020-11-20T12:09:09.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.563+0000 7f1ffb7fe700 1 -- 172.21.15.187:0/2906348544 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4009d80 con 0x7f20040bda00 2020-11-20T12:09:09.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.564+0000 7f20098af700 1 -- 172.21.15.187:0/2906348544 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040bda00 msgr2=0x7f20040b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:09.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.564+0000 7f20098af700 1 --2- 172.21.15.187:0/2906348544 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040bda00 0x7f20040b6dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1ff4006fd0 tx=0x7f1ff4006eb0).stop 2020-11-20T12:09:09.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.564+0000 7f20098af700 1 -- 172.21.15.187:0/2906348544 shutdown_connections 2020-11-20T12:09:09.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.564+0000 7f20098af700 1 --2- 172.21.15.187:0/2906348544 >> v2:172.21.15.5:3300/0 conn(0x7f20040a2770 0x7f20040a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.564+0000 7f20098af700 1 --2- 172.21.15.187:0/2906348544 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040bda00 0x7f20040b6dd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.564+0000 7f20098af700 1 -- 172.21.15.187:0/2906348544 >> 172.21.15.187:0/2906348544 conn(0x7f2004060c40 msgr2=0x7f20040bcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:09.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.564+0000 7f20098af700 1 -- 172.21.15.187:0/2906348544 shutdown_connections 2020-11-20T12:09:09.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.564+0000 7f20098af700 1 -- 172.21.15.187:0/2906348544 wait complete. 2020-11-20T12:09:09.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.565+0000 7f20098af700 1 Processor -- start 2020-11-20T12:09:09.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.566+0000 7f20098af700 1 -- start start 2020-11-20T12:09:09.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.566+0000 7f20098af700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040a2770 0x7f2004155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:09.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.566+0000 7f20098af700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20040bd620 0x7f2004159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:09.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.566+0000 7f20098af700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2004159e70 0x7f2004145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:09.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.567+0000 7f20098af700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2004156bb0 con 0x7f2004159e70 2020-11-20T12:09:09.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.567+0000 7f20098af700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20041561d0 con 0x7f20040bd620 2020-11-20T12:09:09.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.567+0000 7f20098af700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2004156470 con 0x7f20040a2770 2020-11-20T12:09:09.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.567+0000 7f1ffbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20040bd620 0x7f2004159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:09.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.567+0000 7f1ffbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20040bd620 0x7f2004159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44570/0 (socket says 172.21.15.187:44570) 2020-11-20T12:09:09.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.567+0000 7f1ffbfff700 1 -- 172.21.15.187:0/3197549595 learned_addr learned my addr 172.21.15.187:0/3197549595 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:09.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.567+0000 7f1ffbfff700 1 -- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040a2770 msgr2=0x7f2004155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:09.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f20090ae700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2004159e70 0x7f2004145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:09.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.567+0000 7f20088ad700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040a2770 0x7f2004155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:09.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f1ffbfff700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040a2770 0x7f2004155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f1ffbfff700 1 -- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2004159e70 msgr2=0x7f2004145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:09.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f1ffbfff700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2004159e70 0x7f2004145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f1ffbfff700 1 -- 172.21.15.187:0/3197549595 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004156710 con 0x7f20040bd620 2020-11-20T12:09:09.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f20090ae700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2004159e70 0x7f2004145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:09.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f20088ad700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040a2770 0x7f2004155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:09.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f1ffbfff700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20040bd620 0x7f2004159980 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f1ff400b1b0 tx=0x7f1ff4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:09.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f1ff8ff9700 1 -- 172.21.15.187:0/3197549595 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4011e50 con 0x7f20040bd620 2020-11-20T12:09:09.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2004146400 con 0x7f20040bd620 2020-11-20T12:09:09.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f1ff8ff9700 1 -- 172.21.15.187:0/3197549595 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff401e460 con 0x7f20040bd620 2020-11-20T12:09:09.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f1ff8ff9700 1 -- 172.21.15.187:0/3197549595 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff401e7e0 con 0x7f20040bd620 2020-11-20T12:09:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.568+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200414c040 con 0x7f20040bd620 2020-11-20T12:09:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.569+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20041568a0 con 0x7f20040bd620 2020-11-20T12:09:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.570+0000 7f1ff8ff9700 1 -- 172.21.15.187:0/3197549595 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1ff4025030 con 0x7f20040bd620 2020-11-20T12:09:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.571+0000 7f1ff8ff9700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fdc056d20 0x7f1fdc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.571+0000 7f1ff8ff9700 1 -- 172.21.15.187:0/3197549595 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ff40718c0 con 0x7f20040bd620 2020-11-20T12:09:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.571+0000 7f20088ad700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fdc056d20 0x7f1fdc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.571+0000 7f20088ad700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fdc056d20 0x7f1fdc0591e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2000001be0 tx=0x7f2000009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.575+0000 7f1ff8ff9700 1 -- 172.21.15.187:0/3197549595 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1ff4046cb0 con 0x7f20040bd620 2020-11-20T12:09:09.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.818+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20040b3830 con 0x7f20040bd620 2020-11-20T12:09:09.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.819+0000 7f1ff8ff9700 1 -- 172.21.15.187:0/3197549595 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+2133 (secure 0 0 0) 0x7f1ff4023070 con 0x7f20040bd620 2020-11-20T12:09:09.824 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:09.824 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":34,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/170052250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":170052250},{"type":"v1","addr":"172.21.15.5:6833","nonce":170052250}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:09.281700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3570428099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3570428099},{"type":"v1","addr":"172.21.15.187:6835","nonce":3570428099}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:09.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.821+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fdc056d20 msgr2=0x7f1fdc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:09.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.821+0000 7f20098af700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fdc056d20 0x7f1fdc0591e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2000001be0 tx=0x7f2000009040).stop 2020-11-20T12:09:09.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.821+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20040bd620 msgr2=0x7f2004159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:09.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.821+0000 7f20098af700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20040bd620 0x7f2004159980 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f1ff400b1b0 tx=0x7f1ff4015000).stop 2020-11-20T12:09:09.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.822+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 shutdown_connections 2020-11-20T12:09:09.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.822+0000 7f20098af700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fdc056d20 0x7f1fdc0591e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.822+0000 7f20098af700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2004159e70 0x7f2004145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.822+0000 7f20098af700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20040a2770 0x7f2004155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.822+0000 7f20098af700 1 --2- 172.21.15.187:0/3197549595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20040bd620 0x7f2004159980 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:09.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.822+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 >> 172.21.15.187:0/3197549595 conn(0x7f2004060c40 msgr2=0x7f20040b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:09.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.822+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 shutdown_connections 2020-11-20T12:09:09.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:09.822+0000 7f20098af700 1 -- 172.21.15.187:0/3197549595 wait complete. 2020-11-20T12:09:09.829 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 34 2020-11-20T12:09:09.847 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:09:10.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.079+0000 7f88e759e700 1 Processor -- start 2020-11-20T12:09:10.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.080+0000 7f88e759e700 1 -- start start 2020-11-20T12:09:10.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.081+0000 7f88e759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f88e80b57b0 0x7f88e8068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.081+0000 7f88e759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e8144b80 con 0x7f88e80b5f50 2020-11-20T12:09:10.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.081+0000 7f88e759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e81489e0 con 0x7f88e81482d0 2020-11-20T12:09:10.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.081+0000 7f88e759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e8148c80 con 0x7f88e80b57b0 2020-11-20T12:09:10.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.081+0000 7f88e5d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f88e81482d0 0x7f88e8062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41186/0 (socket says 172.21.15.187:41186) 2020-11-20T12:09:10.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.081+0000 7f88e5d9b700 1 -- 172.21.15.187:0/2141588370 learned_addr learned my addr 172.21.15.187:0/2141588370 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:10.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.082+0000 7f88e659c700 1 --2- 172.21.15.187:0/2141588370 >> v2:172.21.15.5:3300/0 conn(0x7f88e80b57b0 0x7f88e8068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.082+0000 7f88e559a700 1 -- 172.21.15.187:0/2141588370 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796838456 0 0) 0x7f88e81489e0 con 0x7f88e81482d0 2020-11-20T12:09:10.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.082+0000 7f88e559a700 1 -- 172.21.15.187:0/2141588370 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88cc003240 con 0x7f88e81482d0 2020-11-20T12:09:10.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.083+0000 7f88e659c700 1 -- 172.21.15.187:0/2141588370 >> v1:172.21.15.187:6789/0 conn(0x7f88e81482d0 legacy=0x7f88e8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.083+0000 7f88e659c700 1 -- 172.21.15.187:0/2141588370 >> v1:172.21.15.5:6789/0 conn(0x7f88e80b5f50 legacy=0x7f88e80b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.083+0000 7f88e659c700 1 -- 172.21.15.187:0/2141588370 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e8148f20 con 0x7f88e80b57b0 2020-11-20T12:09:10.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.083+0000 7f88e659c700 1 --2- 172.21.15.187:0/2141588370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b57b0 0x7f88e8068230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f88d800b9a0 tx=0x7f88d8006df0).ready entity=mon.0 client_cookie=794b60c2a24f8608 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:10.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.083+0000 7f88e559a700 1 -- 172.21.15.187:0/2141588370 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d80097b0 con 0x7f88e80b57b0 2020-11-20T12:09:10.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.083+0000 7f88e559a700 1 -- 172.21.15.187:0/2141588370 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d8009910 con 0x7f88e80b57b0 2020-11-20T12:09:10.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.083+0000 7f88e559a700 1 -- 172.21.15.187:0/2141588370 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8009c90 con 0x7f88e80b57b0 2020-11-20T12:09:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.084+0000 7f88e759e700 1 -- 172.21.15.187:0/2141588370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b57b0 msgr2=0x7f88e8068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.084+0000 7f88e759e700 1 --2- 172.21.15.187:0/2141588370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b57b0 0x7f88e8068230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f88d800b9a0 tx=0x7f88d8006df0).stop 2020-11-20T12:09:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.084+0000 7f88e759e700 1 -- 172.21.15.187:0/2141588370 shutdown_connections 2020-11-20T12:09:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.084+0000 7f88e759e700 1 --2- 172.21.15.187:0/2141588370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b57b0 0x7f88e8068230 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.084+0000 7f88e759e700 1 -- 172.21.15.187:0/2141588370 >> 172.21.15.187:0/2141588370 conn(0x7f88e80bab30 msgr2=0x7f88e80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.084+0000 7f88e759e700 1 -- 172.21.15.187:0/2141588370 shutdown_connections 2020-11-20T12:09:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.084+0000 7f88e759e700 1 -- 172.21.15.187:0/2141588370 wait complete. 2020-11-20T12:09:10.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.086+0000 7f88e759e700 1 Processor -- start 2020-11-20T12:09:10.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.086+0000 7f88e759e700 1 -- start start 2020-11-20T12:09:10.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.086+0000 7f88e759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88e80b57b0 0x7f88e80a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.086+0000 7f88e759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b5f50 0x7f88e8146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.086+0000 7f88e759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88e81482d0 0x7f88e814f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.086+0000 7f88e759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80a0a30 con 0x7f88e80b5f50 2020-11-20T12:09:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.086+0000 7f88e759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e8140e00 con 0x7f88e81482d0 2020-11-20T12:09:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.086+0000 7f88e759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e81410a0 con 0x7f88e80b57b0 2020-11-20T12:09:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88e81482d0 0x7f88e814f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88e81482d0 0x7f88e814f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44586/0 (socket says 172.21.15.187:44586) 2020-11-20T12:09:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e6d9d700 1 -- 172.21.15.187:0/2171809251 learned_addr learned my addr 172.21.15.187:0/2171809251 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e5d9b700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b5f50 0x7f88e8146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e659c700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88e80b57b0 0x7f88e80a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e6d9d700 1 -- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88e80b57b0 msgr2=0x7f88e80a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e6d9d700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88e80b57b0 0x7f88e80a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e6d9d700 1 -- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b5f50 msgr2=0x7f88e8146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.087+0000 7f88e6d9d700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b5f50 0x7f88e8146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88e6d9d700 1 -- 172.21.15.187:0/2171809251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e8141340 con 0x7f88e81482d0 2020-11-20T12:09:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88e5d9b700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b5f50 0x7f88e8146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88e659c700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88e80b57b0 0x7f88e80a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88e6d9d700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88e81482d0 0x7f88e814f230 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f88e000ca60 tx=0x7f88e0007f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88d67fc700 1 -- 172.21.15.187:0/2171809251 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e000a830 con 0x7f88e81482d0 2020-11-20T12:09:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88d67fc700 1 -- 172.21.15.187:0/2171809251 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e000a990 con 0x7f88e81482d0 2020-11-20T12:09:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e814f720 con 0x7f88e81482d0 2020-11-20T12:09:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88d67fc700 1 -- 172.21.15.187:0/2171809251 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e000ad10 con 0x7f88e81482d0 2020-11-20T12:09:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.088+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e8140a40 con 0x7f88e81482d0 2020-11-20T12:09:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.090+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e8061a20 con 0x7f88e81482d0 2020-11-20T12:09:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.090+0000 7f88d67fc700 1 -- 172.21.15.187:0/2171809251 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f88e0019030 con 0x7f88e81482d0 2020-11-20T12:09:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.091+0000 7f88d67fc700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f88bc056d50 0x7f88bc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.091+0000 7f88d67fc700 1 -- 172.21.15.187:0/2171809251 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f88e006f880 con 0x7f88e81482d0 2020-11-20T12:09:10.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.091+0000 7f88e659c700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f88bc056d50 0x7f88bc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.092+0000 7f88e659c700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f88bc056d50 0x7f88bc059210 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f88d8006e30 tx=0x7f88d8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:10.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.095+0000 7f88d67fc700 1 -- 172.21.15.187:0/2171809251 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f88e0043fd0 con 0x7f88e81482d0 2020-11-20T12:09:10.104 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:09:10.100+0000 7f679aa21700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:09:10.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.280+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88e8062d70 con 0x7f88e81482d0 2020-11-20T12:09:10.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.282+0000 7f88d67fc700 1 -- 172.21.15.187:0/2171809251 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f88e00251a0 con 0x7f88e81482d0 2020-11-20T12:09:10.287 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:10.287 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":59,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:09:08.261747+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:09:05.207488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2020-11-20T12:09:06.244532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:09:10.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f88bc056d50 msgr2=0x7f88bc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f88bc056d50 0x7f88bc059210 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f88d8006e30 tx=0x7f88d8015000).stop 2020-11-20T12:09:10.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88e81482d0 msgr2=0x7f88e814f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88e81482d0 0x7f88e814f230 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f88e000ca60 tx=0x7f88e0007f40).stop 2020-11-20T12:09:10.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 shutdown_connections 2020-11-20T12:09:10.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f88bc056d50 0x7f88bc059210 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88e81482d0 0x7f88e814f230 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88e80b57b0 0x7f88e80a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 --2- 172.21.15.187:0/2171809251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80b5f50 0x7f88e8146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 >> 172.21.15.187:0/2171809251 conn(0x7f88e80bab30 msgr2=0x7f88e81440e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:10.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.284+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 shutdown_connections 2020-11-20T12:09:10.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.285+0000 7f88e759e700 1 -- 172.21.15.187:0/2171809251 wait complete. 2020-11-20T12:09:10.314 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:09:10.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.551+0000 7f751edc6700 1 Processor -- start 2020-11-20T12:09:10.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.552+0000 7f751edc6700 1 -- start start 2020-11-20T12:09:10.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.553+0000 7f751edc6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7518065190 0x7f7518065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.553+0000 7f751edc6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f751809ca90 0x7f751809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.553+0000 7f751edc6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180b13e0 con 0x7f7518065f90 2020-11-20T12:09:10.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.553+0000 7f751edc6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751814ab30 con 0x7f7518065190 2020-11-20T12:09:10.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.553+0000 7f751edc6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7518062c10 con 0x7f751809ca90 2020-11-20T12:09:10.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.554+0000 7f751d5c3700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7518065f90 0x7f7518143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41202/0 (socket says 172.21.15.187:41202) 2020-11-20T12:09:10.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.554+0000 7f751d5c3700 1 -- 172.21.15.187:0/752168636 learned_addr learned my addr 172.21.15.187:0/752168636 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:10.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.554+0000 7f751ddc4700 1 --2- 172.21.15.187:0/752168636 >> v2:172.21.15.5:3300/0 conn(0x7f7518065190 0x7f7518065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.554+0000 7f751e5c5700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f751809ca90 0x7f751809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.554+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972048078 0 0) 0x7f75180b13e0 con 0x7f7518065f90 2020-11-20T12:09:10.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.554+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7500003240 con 0x7f7518065f90 2020-11-20T12:09:10.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1941905122 0 0) 0x7f7500003240 con 0x7f7518065f90 2020-11-20T12:09:10.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f751809ca90 msgr2=0x7f751809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:10.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f751809ca90 0x7f751809cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 >> v2:172.21.15.5:3300/0 conn(0x7f7518065190 msgr2=0x7f7518065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:10.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 --2- 172.21.15.187:0/752168636 >> v2:172.21.15.5:3300/0 conn(0x7f7518065190 0x7f7518065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180b1640 con 0x7f7518065f90 2020-11-20T12:09:10.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f75080014e0 con 0x7f7518065f90 2020-11-20T12:09:10.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 >> v1:172.21.15.187:6789/0 conn(0x7f7518065f90 legacy=0x7f7518143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75000037b0 0x7f7500003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f751809ca90 0x7f75000044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75000049d0 0x7f7500004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751814ab30 con 0x7f75000049d0 2020-11-20T12:09:10.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518062c10 con 0x7f75000037b0 2020-11-20T12:09:10.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75000042a0 con 0x7f751809ca90 2020-11-20T12:09:10.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.555+0000 7f751ddc4700 1 --2- 172.21.15.187:0/752168636 >> v2:172.21.15.5:3300/0 conn(0x7f7518065190 0x7f7518065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:09:10.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751ddc4700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75000037b0 0x7f7500003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751d5c3700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f751809ca90 0x7f75000044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751e5c5700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75000049d0 0x7f7500004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751ddc4700 1 -- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f751809ca90 msgr2=0x7f75000044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:10.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751ddc4700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f751809ca90 0x7f75000044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751ddc4700 1 -- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75000049d0 msgr2=0x7f7500004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:10.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751ddc4700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75000049d0 0x7f7500004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751ddc4700 1 -- 172.21.15.187:0/752168636 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500003240 con 0x7f75000037b0 2020-11-20T12:09:10.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.556+0000 7f751e5c5700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75000049d0 0x7f7500004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:10.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751ddc4700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75000037b0 0x7f7500003bd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f751400be60 tx=0x7f7514006df0).ready entity=mon.1 client_cookie=5d21ab18a64aa65b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:10.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751400b590 con 0x7f75000037b0 2020-11-20T12:09:10.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751400edc0 con 0x7f75000037b0 2020-11-20T12:09:10.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751cdc2700 1 -- 172.21.15.187:0/752168636 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514012600 con 0x7f75000037b0 2020-11-20T12:09:10.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751edc6700 1 -- 172.21.15.187:0/752168636 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75000037b0 msgr2=0x7f7500003bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751edc6700 1 --2- 172.21.15.187:0/752168636 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75000037b0 0x7f7500003bd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f751400be60 tx=0x7f7514006df0).stop 2020-11-20T12:09:10.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751e5c5700 1 -- 172.21.15.187:0/752168636 reap_dead start 2020-11-20T12:09:10.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751edc6700 1 -- 172.21.15.187:0/752168636 shutdown_connections 2020-11-20T12:09:10.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.557+0000 7f751edc6700 1 -- 172.21.15.187:0/752168636 >> 172.21.15.187:0/752168636 conn(0x7f75180bab30 msgr2=0x7f75180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:10.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.558+0000 7f751edc6700 1 -- 172.21.15.187:0/752168636 shutdown_connections 2020-11-20T12:09:10.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.558+0000 7f751edc6700 1 -- 172.21.15.187:0/752168636 wait complete. 2020-11-20T12:09:10.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.559+0000 7f751edc6700 1 Processor -- start 2020-11-20T12:09:10.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.559+0000 7f751edc6700 1 -- start start 2020-11-20T12:09:10.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751edc6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7518142a30 0x7f7518158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751edc6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7518142e10 0x7f751815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751edc6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75181436c0 0x7f7518151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751edc6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518159100 con 0x7f75181436c0 2020-11-20T12:09:10.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751edc6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518158230 con 0x7f7518142e10 2020-11-20T12:09:10.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751edc6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75181584d0 con 0x7f7518142a30 2020-11-20T12:09:10.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751d5c3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7518142e10 0x7f751815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751d5c3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7518142e10 0x7f751815bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44604/0 (socket says 172.21.15.187:44604) 2020-11-20T12:09:10.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.560+0000 7f751d5c3700 1 -- 172.21.15.187:0/1657324933 learned_addr learned my addr 172.21.15.187:0/1657324933 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:10.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751ddc4700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7518142a30 0x7f7518158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751e5c5700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75181436c0 0x7f7518151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751d5c3700 1 -- 172.21.15.187:0/1657324933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7518142a30 msgr2=0x7f7518158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751d5c3700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7518142a30 0x7f7518158c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751d5c3700 1 -- 172.21.15.187:0/1657324933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75181436c0 msgr2=0x7f7518151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751d5c3700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75181436c0 0x7f7518151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751d5c3700 1 -- 172.21.15.187:0/1657324933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7518158770 con 0x7f7518142e10 2020-11-20T12:09:10.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751d5c3700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7518142e10 0x7f751815bf30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f7508001e90 tx=0x7f75180bb650).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:10.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f7505ffb700 1 -- 172.21.15.187:0/1657324933 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750800cbd0 con 0x7f7518142e10 2020-11-20T12:09:10.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f7505ffb700 1 -- 172.21.15.187:0/1657324933 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750800cd30 con 0x7f7518142e10 2020-11-20T12:09:10.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f7505ffb700 1 -- 172.21.15.187:0/1657324933 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750800cea0 con 0x7f7518142e10 2020-11-20T12:09:10.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.561+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751815c420 con 0x7f7518142e10 2020-11-20T12:09:10.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.562+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7518152390 con 0x7f7518142e10 2020-11-20T12:09:10.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.563+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75180b1800 con 0x7f7518142e10 2020-11-20T12:09:10.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.563+0000 7f7505ffb700 1 -- 172.21.15.187:0/1657324933 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f75080148d0 con 0x7f7518142e10 2020-11-20T12:09:10.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.564+0000 7f7505ffb700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f74f0056d20 0x7f74f00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:10.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.564+0000 7f7505ffb700 1 -- 172.21.15.187:0/1657324933 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f750806ffe0 con 0x7f7518142e10 2020-11-20T12:09:10.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.564+0000 7f751ddc4700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f74f0056d20 0x7f74f00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:10.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.565+0000 7f751ddc4700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f74f0056d20 0x7f74f00591e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f751805f730 tx=0x7f7514019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:10.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.567+0000 7f7505ffb700 1 -- 172.21.15.187:0/1657324933 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7508045340 con 0x7f7518142e10 2020-11-20T12:09:10.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.817+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f7518158900 con 0x7f7518142e10 2020-11-20T12:09:10.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.829+0000 7f7505ffb700 1 -- 172.21.15.187:0/1657324933 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7508048960 con 0x7f7518142e10 2020-11-20T12:09:10.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.831+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f74f0056d20 msgr2=0x7f74f00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.831+0000 7f751edc6700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f74f0056d20 0x7f74f00591e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f751805f730 tx=0x7f7514019000).stop 2020-11-20T12:09:10.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.831+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7518142e10 msgr2=0x7f751815bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:10.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.831+0000 7f751edc6700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7518142e10 0x7f751815bf30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f7508001e90 tx=0x7f75180bb650).stop 2020-11-20T12:09:10.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.831+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 shutdown_connections 2020-11-20T12:09:10.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.831+0000 7f751edc6700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f74f0056d20 0x7f74f00591e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.832+0000 7f751edc6700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75181436c0 0x7f7518151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.832+0000 7f751edc6700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7518142a30 0x7f7518158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.832+0000 7f751edc6700 1 --2- 172.21.15.187:0/1657324933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7518142e10 0x7f751815bf30 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.832+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 >> 172.21.15.187:0/1657324933 conn(0x7f75180bab30 msgr2=0x7f75180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.832+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 shutdown_connections 2020-11-20T12:09:10.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:10.832+0000 7f751edc6700 1 -- 172.21.15.187:0/1657324933 wait complete. 2020-11-20T12:09:10.839 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:09:10.856 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:11.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.080+0000 7fcad1faa700 1 Processor -- start 2020-11-20T12:09:11.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.080+0000 7fcad1faa700 1 -- start start 2020-11-20T12:09:11.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.081+0000 7fcad1faa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065190 0x7fcacc065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:11.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.081+0000 7fcad1faa700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcacc09ca90 0x7fcacc09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:11.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.081+0000 7fcad1faa700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcacc0b13e0 con 0x7fcacc065f90 2020-11-20T12:09:11.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.081+0000 7fcad1faa700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcacc14ab30 con 0x7fcacc09ca90 2020-11-20T12:09:11.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.081+0000 7fcad1faa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc062c10 con 0x7fcacc065190 2020-11-20T12:09:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.082+0000 7fcad0fa8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065190 0x7fcacc065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.082+0000 7fcad0fa8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065190 0x7fcacc065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:45502/0 (socket says 172.21.15.187:45502) 2020-11-20T12:09:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.082+0000 7fcad0fa8700 1 -- 172.21.15.187:0/3030340958 learned_addr learned my addr 172.21.15.187:0/3030340958 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.082+0000 7fcad17a9700 1 --2- 172.21.15.187:0/3030340958 >> v2:172.21.15.5:3300/0 conn(0x7fcacc09ca90 0x7fcacc09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:11.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.083+0000 7fcad0fa8700 1 -- 172.21.15.187:0/3030340958 >> v2:172.21.15.5:3300/0 conn(0x7fcacc09ca90 msgr2=0x7fcacc09cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:11.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.083+0000 7fcad0fa8700 1 --2- 172.21.15.187:0/3030340958 >> v2:172.21.15.5:3300/0 conn(0x7fcacc09ca90 0x7fcacc09cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.083+0000 7fcad0fa8700 1 -- 172.21.15.187:0/3030340958 >> v1:172.21.15.5:6789/0 conn(0x7fcacc065f90 legacy=0x7fcacc143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:11.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.084+0000 7fcad0fa8700 1 -- 172.21.15.187:0/3030340958 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc0b1640 con 0x7fcacc065190 2020-11-20T12:09:11.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.084+0000 7fcad0fa8700 1 --2- 172.21.15.187:0/3030340958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065190 0x7fcacc065570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fcacc14ac70 tx=0x7fcac8009290).ready entity=mon.2 client_cookie=f1610d206fc936a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:11.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.085+0000 7fcac37fe700 1 -- 172.21.15.187:0/3030340958 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac800bc10 con 0x7fcacc065190 2020-11-20T12:09:11.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.085+0000 7fcac37fe700 1 -- 172.21.15.187:0/3030340958 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac800bd70 con 0x7fcacc065190 2020-11-20T12:09:11.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.085+0000 7fcac37fe700 1 -- 172.21.15.187:0/3030340958 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8011610 con 0x7fcacc065190 2020-11-20T12:09:11.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.085+0000 7fcad1faa700 1 -- 172.21.15.187:0/3030340958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065190 msgr2=0x7fcacc065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:11.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.085+0000 7fcad1faa700 1 --2- 172.21.15.187:0/3030340958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065190 0x7fcacc065570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fcacc14ac70 tx=0x7fcac8009290).stop 2020-11-20T12:09:11.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.086+0000 7fcad1faa700 1 -- 172.21.15.187:0/3030340958 shutdown_connections 2020-11-20T12:09:11.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.086+0000 7fcad1faa700 1 --2- 172.21.15.187:0/3030340958 >> v2:172.21.15.5:3300/0 conn(0x7fcacc09ca90 0x7fcacc09cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.086+0000 7fcad1faa700 1 --2- 172.21.15.187:0/3030340958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065190 0x7fcacc065570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.086+0000 7fcad1faa700 1 -- 172.21.15.187:0/3030340958 >> 172.21.15.187:0/3030340958 conn(0x7fcacc0bab30 msgr2=0x7fcacc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:11.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.086+0000 7fcad1faa700 1 -- 172.21.15.187:0/3030340958 shutdown_connections 2020-11-20T12:09:11.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.086+0000 7fcad1faa700 1 -- 172.21.15.187:0/3030340958 wait complete. 2020-11-20T12:09:11.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.087+0000 7fcad1faa700 1 Processor -- start 2020-11-20T12:09:11.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.087+0000 7fcad1faa700 1 -- start start 2020-11-20T12:09:11.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.088+0000 7fcad1faa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc065190 0x7fcacc142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:11.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.088+0000 7fcad1faa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065f90 0x7fcacc0668f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:11.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.088+0000 7fcad1faa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc09ca90 0x7fcacc0b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:11.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.088+0000 7fcad1faa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc099a30 con 0x7fcacc09ca90 2020-11-20T12:09:11.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.088+0000 7fcad1faa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc0b3700 con 0x7fcacc065190 2020-11-20T12:09:11.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.088+0000 7fcad1faa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc15c820 con 0x7fcacc065f90 2020-11-20T12:09:11.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad0fa8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc065190 0x7fcacc142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:11.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad0fa8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc065190 0x7fcacc142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44618/0 (socket says 172.21.15.187:44618) 2020-11-20T12:09:11.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad0fa8700 1 -- 172.21.15.187:0/852043601 learned_addr learned my addr 172.21.15.187:0/852043601 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:11.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcac3fff700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065f90 0x7fcacc0668f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:11.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad17a9700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc09ca90 0x7fcacc0b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:11.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad0fa8700 1 -- 172.21.15.187:0/852043601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065f90 msgr2=0x7fcacc0668f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:11.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad0fa8700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065f90 0x7fcacc0668f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad0fa8700 1 -- 172.21.15.187:0/852043601 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc09ca90 msgr2=0x7fcacc0b7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:11.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad0fa8700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc09ca90 0x7fcacc0b7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.089+0000 7fcad0fa8700 1 -- 172.21.15.187:0/852043601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc142e20 con 0x7fcacc065190 2020-11-20T12:09:11.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.090+0000 7fcad0fa8700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc065190 0x7fcacc142930 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fcac80090f0 tx=0x7fcac8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:11.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.090+0000 7fcac0ff9700 1 -- 172.21.15.187:0/852043601 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8020070 con 0x7fcacc065190 2020-11-20T12:09:11.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.090+0000 7fcac0ff9700 1 -- 172.21.15.187:0/852043601 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac801b6b0 con 0x7fcacc065190 2020-11-20T12:09:11.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.090+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcacc15c2e0 con 0x7fcacc065190 2020-11-20T12:09:11.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.090+0000 7fcac0ff9700 1 -- 172.21.15.187:0/852043601 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac801ba30 con 0x7fcacc065190 2020-11-20T12:09:11.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.090+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcacc066de0 con 0x7fcacc065190 2020-11-20T12:09:11.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.091+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcacc061a20 con 0x7fcacc065190 2020-11-20T12:09:11.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.092+0000 7fcac0ff9700 1 -- 172.21.15.187:0/852043601 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcac8017070 con 0x7fcacc065190 2020-11-20T12:09:11.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.092+0000 7fcac0ff9700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaa4056d20 0x7fcaa40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:11.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.093+0000 7fcac0ff9700 1 -- 172.21.15.187:0/852043601 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcac8071b10 con 0x7fcacc065190 2020-11-20T12:09:11.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.093+0000 7fcac3fff700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaa4056d20 0x7fcaa40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:11.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.094+0000 7fcac3fff700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaa4056d20 0x7fcaa40591e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fcacc1583c0 tx=0x7fcabc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:11.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.096+0000 7fcac0ff9700 1 -- 172.21.15.187:0/852043601 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcac8046f40 con 0x7fcacc065190 2020-11-20T12:09:11.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.325+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcacc0afc60 con 0x7fcacc065190 2020-11-20T12:09:11.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.326+0000 7fcac0ff9700 1 -- 172.21.15.187:0/852043601 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+2133 (secure 0 0 0) 0x7fcac8015070 con 0x7fcacc065190 2020-11-20T12:09:11.331 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:11.331 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":34,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/170052250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":170052250},{"type":"v1","addr":"172.21.15.5:6833","nonce":170052250}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:09.281700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3570428099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3570428099},{"type":"v1","addr":"172.21.15.187:6835","nonce":3570428099}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.328+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaa4056d20 msgr2=0x7fcaa40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:11.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.328+0000 7fcad1faa700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaa4056d20 0x7fcaa40591e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fcacc1583c0 tx=0x7fcabc009040).stop 2020-11-20T12:09:11.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.328+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc065190 msgr2=0x7fcacc142930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:11.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.328+0000 7fcad1faa700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc065190 0x7fcacc142930 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fcac80090f0 tx=0x7fcac8016000).stop 2020-11-20T12:09:11.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.329+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 shutdown_connections 2020-11-20T12:09:11.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.329+0000 7fcad1faa700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaa4056d20 0x7fcaa40591e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.329+0000 7fcad1faa700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc09ca90 0x7fcacc0b7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.329+0000 7fcad1faa700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc065190 0x7fcacc142930 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.329+0000 7fcad1faa700 1 --2- 172.21.15.187:0/852043601 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcacc065f90 0x7fcacc0668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:11.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.329+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 >> 172.21.15.187:0/852043601 conn(0x7fcacc0bab30 msgr2=0x7fcacc09b720 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:11.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.329+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 shutdown_connections 2020-11-20T12:09:11.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:11.329+0000 7fcad1faa700 1 -- 172.21.15.187:0/852043601 wait complete. 2020-11-20T12:09:11.336 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 34 2020-11-20T12:09:11.556 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:09:07.256865+0000', 'modified': '2020-11-20T12:09:09.281700+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6010}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6010': {'gid': 6010, 'name': 'a', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3570428099', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3570428099}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3570428099}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:09:11.557 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:09:11.557 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:09:11.837 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6010,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.5500030709999999,"mdsmap_epoch":34,"osdmap_epoch":59,"osdmap_epoch_barrier":59,"uptime":16.138545123} 2020-11-20T12:09:11.837 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6010, "mdsmap_epoch": 34, "osdmap_epoch": 59, "osdmap_epoch_barrier": 59, "rank_uptime": 4.550003071, "state": "up:active", "uptime": 16.138545123, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:09:11.838 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:09:11.838 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:09:11.839 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:09:11.839 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:09:11.840 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:09:11.840 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:09:11.840 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:09:11.841 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:09:11.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:11.842 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:09:11.894 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:09:11.894 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:09:11.894 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:09:11.895 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:09:11.895 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:09:11.895 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:09:11.895 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:09:11.895 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:09:11.896 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:09:11.896 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:09:11.896 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:09:11.896 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:09:11.896 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:09:11.897 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:09:11.897 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:09:11.897 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42487sec preferred_lft 42487sec 2020-11-20T12:09:11.897 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:09:11.898 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:09:11.898 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:09:11.898 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:09:11.898 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:09:11.898 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:09:11.899 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:09:11.899 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:09:11.899 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:09:11.899 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:09:11.899 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:09:11.900 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:09:11.900 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:09:11.900 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:09:11.900 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:09:11.900 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:09:11.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:11.902 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:09:11.999 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:09:12.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:12.004 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:09:12.059 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:09:12.060 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:09:12.060 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:09:12.060 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:09:12.061 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:09:12.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:12.062 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:09:12.063 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:09:12.063 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:09:12.063 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:09:12.064 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:09:12.064 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:09:12.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:12.314 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:09:12.362 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:09:12.362 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:09:12.363 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:09:12.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:12.364 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:09:12.465 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:09:12.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.707+0000 7f8ae7567700 1 Processor -- start 2020-11-20T12:09:12.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.707+0000 7f8ae7567700 1 -- start start 2020-11-20T12:09:12.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.709+0000 7f8ae7567700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 0x7f8ae0146d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:12.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.709+0000 7f8ae7567700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae01440a0 con 0x7f8ae00ba9b0 2020-11-20T12:09:12.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.709+0000 7f8ae7567700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae009c750 con 0x7f8ae009eee0 2020-11-20T12:09:12.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.709+0000 7f8ae7567700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae0146690 con 0x7f8ae009f2c0 2020-11-20T12:09:12.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.710+0000 7f8ae6565700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8ae009eee0 0x7f8ae0067410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41232/0 (socket says 172.21.15.187:41232) 2020-11-20T12:09:12.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.710+0000 7f8ae6565700 1 -- 172.21.15.187:0/4159036106 learned_addr learned my addr 172.21.15.187:0/4159036106 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:12.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.710+0000 7f8ae6d66700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 0x7f8ae0146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:12.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.710+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909806892 0 0) 0x7f8ae009c750 con 0x7f8ae009eee0 2020-11-20T12:09:12.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.710+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac8003240 con 0x7f8ae009eee0 2020-11-20T12:09:12.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2162771827 0 0) 0x7f8ac8003240 con 0x7f8ae009eee0 2020-11-20T12:09:12.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 msgr2=0x7f8ae0146d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:12.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 0x7f8ae0146d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 >> v1:172.21.15.5:6789/0 conn(0x7f8ae00ba9b0 legacy=0x7f8ae0063f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:12.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae00b9440 con 0x7f8ae009eee0 2020-11-20T12:09:12.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8adc001520 con 0x7f8ae009eee0 2020-11-20T12:09:12.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 >> v1:172.21.15.187:6789/0 conn(0x7f8ae009eee0 legacy=0x7f8ae0067410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:12.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ac8003730 0x7f8ac8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:12.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 0x7f8ac80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:12.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac80049d0 0x7f8ac8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:12.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80041d0 con 0x7f8ac8003730 2020-11-20T12:09:12.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac8004340 con 0x7f8ac80049d0 2020-11-20T12:09:12.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac800fb40 con 0x7f8ae009f2c0 2020-11-20T12:09:12.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.711+0000 7f8ae6d66700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac80049d0 0x7f8ac8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:12.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae5d64700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 0x7f8ac80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:12.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae6565700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ac8003730 0x7f8ac8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:12.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae6d66700 1 -- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 msgr2=0x7f8ac80044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:12.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae6d66700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 0x7f8ac80044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae6d66700 1 -- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ac8003730 msgr2=0x7f8ac8003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:12.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae6d66700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ac8003730 0x7f8ac8003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae6d66700 1 -- 172.21.15.187:0/4159036106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8003240 con 0x7f8ac80049d0 2020-11-20T12:09:12.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae5d64700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae009f2c0 0x7f8ac80044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:12.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae6d66700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac80049d0 0x7f8ac8004e50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f8ad8009e70 tx=0x7f8ad80091d0).ready entity=mon.1 client_cookie=fa757bd58457df42 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:12.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80095c0 con 0x7f8ac80049d0 2020-11-20T12:09:12.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad800ed70 con 0x7f8ac80049d0 2020-11-20T12:09:12.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.712+0000 7f8ae5563700 1 -- 172.21.15.187:0/4159036106 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800eee0 con 0x7f8ac80049d0 2020-11-20T12:09:12.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.713+0000 7f8ae7567700 1 -- 172.21.15.187:0/4159036106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac80049d0 msgr2=0x7f8ac8004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:12.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.713+0000 7f8ae7567700 1 --2- 172.21.15.187:0/4159036106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac80049d0 0x7f8ac8004e50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f8ad8009e70 tx=0x7f8ad80091d0).stop 2020-11-20T12:09:12.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.713+0000 7f8ae6d66700 1 -- 172.21.15.187:0/4159036106 reap_dead start 2020-11-20T12:09:12.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.713+0000 7f8ae7567700 1 -- 172.21.15.187:0/4159036106 shutdown_connections 2020-11-20T12:09:12.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.713+0000 7f8ae7567700 1 -- 172.21.15.187:0/4159036106 >> 172.21.15.187:0/4159036106 conn(0x7f8ae00bce50 msgr2=0x7f8ae00bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:12.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.713+0000 7f8ae7567700 1 -- 172.21.15.187:0/4159036106 shutdown_connections 2020-11-20T12:09:12.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.713+0000 7f8ae7567700 1 -- 172.21.15.187:0/4159036106 wait complete. 2020-11-20T12:09:12.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.715+0000 7f8ae7567700 1 Processor -- start 2020-11-20T12:09:12.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.715+0000 7f8ae7567700 1 -- start start 2020-11-20T12:09:12.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae7567700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae0147dd0 0x7f8ae0062e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:12.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae7567700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae01481b0 0x7f8ae009db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:12.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae7567700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae0061bf0 0x7f8ae0140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:12.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae7567700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae009e140 con 0x7f8ae01481b0 2020-11-20T12:09:12.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae7567700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae009e370 con 0x7f8ae0061bf0 2020-11-20T12:09:12.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae7567700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae009e610 con 0x7f8ae0147dd0 2020-11-20T12:09:12.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae6d66700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae0061bf0 0x7f8ae0140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:12.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae6d66700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae0061bf0 0x7f8ae0140240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44642/0 (socket says 172.21.15.187:44642) 2020-11-20T12:09:12.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae6d66700 1 -- 172.21.15.187:0/1300664512 learned_addr learned my addr 172.21.15.187:0/1300664512 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:12.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.716+0000 7f8ae6565700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae0147dd0 0x7f8ae0062e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:12.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ae5d64700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae01481b0 0x7f8ae009db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:12.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ae6d66700 1 -- 172.21.15.187:0/1300664512 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae0147dd0 msgr2=0x7f8ae0062e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:12.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ae6d66700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae0147dd0 0x7f8ae0062e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ae6d66700 1 -- 172.21.15.187:0/1300664512 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae01481b0 msgr2=0x7f8ae009db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:12.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ae6d66700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae01481b0 0x7f8ae009db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ae6d66700 1 -- 172.21.15.187:0/1300664512 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0061ff0 con 0x7f8ae0061bf0 2020-11-20T12:09:12.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ae6d66700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae0061bf0 0x7f8ae0140240 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f8ad8014000 tx=0x7f8ad801a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:12.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ace7fc700 1 -- 172.21.15.187:0/1300664512 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80095c0 con 0x7f8ae0061bf0 2020-11-20T12:09:12.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ace7fc700 1 -- 172.21.15.187:0/1300664512 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad8025430 con 0x7f8ae0061bf0 2020-11-20T12:09:12.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae0140730 con 0x7f8ae0061bf0 2020-11-20T12:09:12.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.717+0000 7f8ace7fc700 1 -- 172.21.15.187:0/1300664512 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80257f0 con 0x7f8ae0061bf0 2020-11-20T12:09:12.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.718+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae0140c50 con 0x7f8ae0061bf0 2020-11-20T12:09:12.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.718+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae0062b80 con 0x7f8ae0061bf0 2020-11-20T12:09:12.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.719+0000 7f8ace7fc700 1 -- 172.21.15.187:0/1300664512 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8ad802c070 con 0x7f8ae0061bf0 2020-11-20T12:09:12.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.720+0000 7f8ace7fc700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ab8056d50 0x7f8ab8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:12.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.720+0000 7f8ace7fc700 1 -- 172.21.15.187:0/1300664512 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8ad8002030 con 0x7f8ae0061bf0 2020-11-20T12:09:12.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.720+0000 7f8ae6565700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ab8056d50 0x7f8ab8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:12.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.721+0000 7f8ae6565700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ab8056d50 0x7f8ab8059210 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f8adc001200 tx=0x7f8adc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:12.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.724+0000 7f8ace7fc700 1 -- 172.21.15.187:0/1300664512 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8ae0062b80 con 0x7f8ae0061bf0 2020-11-20T12:09:12.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.911+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ae0062b80 con 0x7f8ae0061bf0 2020-11-20T12:09:12.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.914+0000 7f8ace7fc700 1 -- 172.21.15.187:0/1300664512 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f8ae0062b80 con 0x7f8ae0061bf0 2020-11-20T12:09:12.918 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:12.919 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":59,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:09:08.261747+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:09:05.207488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2020-11-20T12:09:06.244532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:09:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ab8056d50 msgr2=0x7f8ab8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ab8056d50 0x7f8ab8059210 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f8adc001200 tx=0x7f8adc009040).stop 2020-11-20T12:09:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae0061bf0 msgr2=0x7f8ae0140240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae0061bf0 0x7f8ae0140240 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f8ad8014000 tx=0x7f8ad801a000).stop 2020-11-20T12:09:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 shutdown_connections 2020-11-20T12:09:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8ab8056d50 0x7f8ab8059210 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae0061bf0 0x7f8ae0140240 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ae0147dd0 0x7f8ae0062e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 --2- 172.21.15.187:0/1300664512 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ae01481b0 0x7f8ae009db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:12.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 >> 172.21.15.187:0/1300664512 conn(0x7f8ae00bce50 msgr2=0x7f8ae00b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:12.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 shutdown_connections 2020-11-20T12:09:12.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:12.916+0000 7f8ae7567700 1 -- 172.21.15.187:0/1300664512 wait complete. 2020-11-20T12:09:12.946 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:09:05.207488+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:09:06.244532+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:09:12.947 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:09:13.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.191+0000 7f7f49bd8700 1 Processor -- start 2020-11-20T12:09:13.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.191+0000 7f7f49bd8700 1 -- start start 2020-11-20T12:09:13.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.193+0000 7f7f49bd8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441483e0 0x7f7f44065300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.193+0000 7f7f49bd8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f44061de0 0x7f7f4414c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.193+0000 7f7f49bd8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f44067710 con 0x7f7f441487c0 2020-11-20T12:09:13.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.193+0000 7f7f49bd8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f44062280 con 0x7f7f44061de0 2020-11-20T12:09:13.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.193+0000 7f7f49bd8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f44062520 con 0x7f7f441483e0 2020-11-20T12:09:13.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.193+0000 7f7f3affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f44061de0 0x7f7f4414c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.194+0000 7f7f3affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f44061de0 0x7f7f4414c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44650/0 (socket says 172.21.15.187:44650) 2020-11-20T12:09:13.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.194+0000 7f7f3affd700 1 -- 172.21.15.187:0/2625909329 learned_addr learned my addr 172.21.15.187:0/2625909329 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:13.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.194+0000 7f7f437fe700 1 --2- 172.21.15.187:0/2625909329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441483e0 0x7f7f44065300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.194+0000 7f7f3affd700 1 -- 172.21.15.187:0/2625909329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441483e0 msgr2=0x7f7f44065300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:13.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.194+0000 7f7f3affd700 1 --2- 172.21.15.187:0/2625909329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441483e0 0x7f7f44065300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.194+0000 7f7f3affd700 1 -- 172.21.15.187:0/2625909329 >> v1:172.21.15.5:6789/0 conn(0x7f7f441487c0 legacy=0x7f7f4414c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.195+0000 7f7f3affd700 1 -- 172.21.15.187:0/2625909329 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f440627c0 con 0x7f7f44061de0 2020-11-20T12:09:13.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.195+0000 7f7f3affd700 1 --2- 172.21.15.187:0/2625909329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44061de0 0x7f7f4414c150 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f7f2c006fd0 tx=0x7f7f2c006eb0).ready entity=mon.1 client_cookie=c50d32dc36daa235 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:13.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.195+0000 7f7f42ffd700 1 -- 172.21.15.187:0/2625909329 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0098d0 con 0x7f7f44061de0 2020-11-20T12:09:13.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f42ffd700 1 -- 172.21.15.187:0/2625909329 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c009a30 con 0x7f7f44061de0 2020-11-20T12:09:13.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f42ffd700 1 -- 172.21.15.187:0/2625909329 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c009db0 con 0x7f7f44061de0 2020-11-20T12:09:13.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f49bd8700 1 -- 172.21.15.187:0/2625909329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44061de0 msgr2=0x7f7f4414c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/2625909329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44061de0 0x7f7f4414c150 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f7f2c006fd0 tx=0x7f7f2c006eb0).stop 2020-11-20T12:09:13.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f49bd8700 1 -- 172.21.15.187:0/2625909329 shutdown_connections 2020-11-20T12:09:13.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/2625909329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441483e0 0x7f7f44065300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/2625909329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44061de0 0x7f7f4414c150 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f49bd8700 1 -- 172.21.15.187:0/2625909329 >> 172.21.15.187:0/2625909329 conn(0x7f7f440bacf0 msgr2=0x7f7f440bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:13.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f49bd8700 1 -- 172.21.15.187:0/2625909329 shutdown_connections 2020-11-20T12:09:13.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.196+0000 7f7f49bd8700 1 -- 172.21.15.187:0/2625909329 wait complete. 2020-11-20T12:09:13.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.198+0000 7f7f49bd8700 1 Processor -- start 2020-11-20T12:09:13.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.199+0000 7f7f49bd8700 1 -- start start 2020-11-20T12:09:13.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f49bd8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f441483e0 0x7f7f4409c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f49bd8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441487c0 0x7f7f440b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f49bd8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44158040 0x7f7f44158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f49bd8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4414a0e0 con 0x7f7f441483e0 2020-11-20T12:09:13.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f49bd8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4409cd50 con 0x7f7f44158040 2020-11-20T12:09:13.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f49bd8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4409cfe0 con 0x7f7f441487c0 2020-11-20T12:09:13.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f43fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44158040 0x7f7f44158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f43fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44158040 0x7f7f44158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44656/0 (socket says 172.21.15.187:44656) 2020-11-20T12:09:13.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f43fff700 1 -- 172.21.15.187:0/379680076 learned_addr learned my addr 172.21.15.187:0/379680076 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:13.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f3affd700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441487c0 0x7f7f440b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.200+0000 7f7f437fe700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f441483e0 0x7f7f4409c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f43fff700 1 -- 172.21.15.187:0/379680076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441487c0 msgr2=0x7f7f440b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f43fff700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441487c0 0x7f7f440b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f43fff700 1 -- 172.21.15.187:0/379680076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f441483e0 msgr2=0x7f7f4409c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f43fff700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f441483e0 0x7f7f4409c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f43fff700 1 -- 172.21.15.187:0/379680076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4409d280 con 0x7f7f44158040 2020-11-20T12:09:13.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f43fff700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44158040 0x7f7f44158480 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f7f3c00ca90 tx=0x7f7f3c007f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:13.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f3bfff700 1 -- 172.21.15.187:0/379680076 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c00a730 con 0x7f7f44158040 2020-11-20T12:09:13.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f3bfff700 1 -- 172.21.15.187:0/379680076 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3c00a890 con 0x7f7f44158040 2020-11-20T12:09:13.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f3bfff700 1 -- 172.21.15.187:0/379680076 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c00ac30 con 0x7f7f44158040 2020-11-20T12:09:13.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4409d520 con 0x7f7f44158040 2020-11-20T12:09:13.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.201+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f44158970 con 0x7f7f44158040 2020-11-20T12:09:13.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.202+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4409d6b0 con 0x7f7f44158040 2020-11-20T12:09:13.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.203+0000 7f7f3bfff700 1 -- 172.21.15.187:0/379680076 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7f3c019030 con 0x7f7f44158040 2020-11-20T12:09:13.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.204+0000 7f7f3bfff700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f20056d20 0x7f7f200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.204+0000 7f7f3bfff700 1 -- 172.21.15.187:0/379680076 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7f3c06f890 con 0x7f7f44158040 2020-11-20T12:09:13.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.205+0000 7f7f437fe700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f20056d20 0x7f7f200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.205+0000 7f7f437fe700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f20056d20 0x7f7f200591e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f7f34009650 tx=0x7f7f34008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:13.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.208+0000 7f7f3bfff700 1 -- 172.21.15.187:0/379680076 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7f3c044ce0 con 0x7f7f44158040 2020-11-20T12:09:13.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.398+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7f44044f60 con 0x7f7f44158040 2020-11-20T12:09:13.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.399+0000 7f7f3bfff700 1 -- 172.21.15.187:0/379680076 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v59) v1 ==== 110+0+10 (secure 0 0 0) 0x7f7f3c048300 con 0x7f7f44158040 2020-11-20T12:09:13.404 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:09:13.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.401+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f20056d20 msgr2=0x7f7f200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.401+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f20056d20 0x7f7f200591e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f7f34009650 tx=0x7f7f34008040).stop 2020-11-20T12:09:13.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.401+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44158040 msgr2=0x7f7f44158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.401+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44158040 0x7f7f44158480 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f7f3c00ca90 tx=0x7f7f3c007f70).stop 2020-11-20T12:09:13.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.402+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 shutdown_connections 2020-11-20T12:09:13.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.402+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f20056d20 0x7f7f200591e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.402+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f441483e0 0x7f7f4409c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.402+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f44158040 0x7f7f44158480 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.402+0000 7f7f49bd8700 1 --2- 172.21.15.187:0/379680076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f441487c0 0x7f7f440b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.402+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 >> 172.21.15.187:0/379680076 conn(0x7f7f440bacf0 msgr2=0x7f7f440b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:13.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.402+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 shutdown_connections 2020-11-20T12:09:13.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.402+0000 7f7f49bd8700 1 -- 172.21.15.187:0/379680076 wait complete. 2020-11-20T12:09:13.427 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:09:13.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.668+0000 7fd7cd568700 1 Processor -- start 2020-11-20T12:09:13.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.668+0000 7fd7cd568700 1 -- start start 2020-11-20T12:09:13.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.670+0000 7fd7cd568700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd7c8065190 0x7fd7c8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.670+0000 7fd7cd568700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c80b13e0 con 0x7fd7c8065f90 2020-11-20T12:09:13.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.670+0000 7fd7cd568700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c8062c10 con 0x7fd7c809ca90 2020-11-20T12:09:13.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.670+0000 7fd7cd568700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c814ab30 con 0x7fd7c8065190 2020-11-20T12:09:13.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.670+0000 7fd7c7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd7c8065190 0x7fd7c8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.670+0000 7fd7c7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd7c8065190 0x7fd7c8065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44660/0 (socket says 172.21.15.187:44660) 2020-11-20T12:09:13.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.670+0000 7fd7c7fff700 1 -- 172.21.15.187:0/1907764537 learned_addr learned my addr 172.21.15.187:0/1907764537 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:13.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.671+0000 7fd7c7fff700 1 -- 172.21.15.187:0/1907764537 >> v1:172.21.15.187:6789/0 conn(0x7fd7c809ca90 legacy=0x7fd7c809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:13.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.671+0000 7fd7c7fff700 1 -- 172.21.15.187:0/1907764537 >> v1:172.21.15.5:6789/0 conn(0x7fd7c8065f90 legacy=0x7fd7c8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.671+0000 7fd7c7fff700 1 -- 172.21.15.187:0/1907764537 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c80b1640 con 0x7fd7c8065190 2020-11-20T12:09:13.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.671+0000 7fd7c7fff700 1 --2- 172.21.15.187:0/1907764537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065190 0x7fd7c8065570 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7fd7c814ac70 tx=0x7fd7b8009290).ready entity=mon.1 client_cookie=a7605632adf77af9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:13.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.672+0000 7fd7c77fe700 1 -- 172.21.15.187:0/1907764537 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b800bc10 con 0x7fd7c8065190 2020-11-20T12:09:13.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.672+0000 7fd7c77fe700 1 -- 172.21.15.187:0/1907764537 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b800bd70 con 0x7fd7c8065190 2020-11-20T12:09:13.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.672+0000 7fd7c77fe700 1 -- 172.21.15.187:0/1907764537 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8011610 con 0x7fd7c8065190 2020-11-20T12:09:13.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.672+0000 7fd7cd568700 1 -- 172.21.15.187:0/1907764537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065190 msgr2=0x7fd7c8065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.672+0000 7fd7cd568700 1 --2- 172.21.15.187:0/1907764537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065190 0x7fd7c8065570 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7fd7c814ac70 tx=0x7fd7b8009290).stop 2020-11-20T12:09:13.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.672+0000 7fd7cd568700 1 -- 172.21.15.187:0/1907764537 shutdown_connections 2020-11-20T12:09:13.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.672+0000 7fd7cd568700 1 --2- 172.21.15.187:0/1907764537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065190 0x7fd7c8065570 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.672+0000 7fd7cd568700 1 -- 172.21.15.187:0/1907764537 >> 172.21.15.187:0/1907764537 conn(0x7fd7c80bab30 msgr2=0x7fd7c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:13.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.673+0000 7fd7cd568700 1 -- 172.21.15.187:0/1907764537 shutdown_connections 2020-11-20T12:09:13.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.673+0000 7fd7cd568700 1 -- 172.21.15.187:0/1907764537 wait complete. 2020-11-20T12:09:13.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.674+0000 7fd7cd568700 1 Processor -- start 2020-11-20T12:09:13.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.674+0000 7fd7cd568700 1 -- start start 2020-11-20T12:09:13.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7cd568700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065f90 0x7fd7c81585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7cd568700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809ca90 0x7fd7c8067a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7cd568700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c815bc30 0x7fd7c815c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7cd568700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80b3950 con 0x7fd7c809ca90 2020-11-20T12:09:13.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7cd568700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8158e00 con 0x7fd7c8065f90 2020-11-20T12:09:13.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7cd568700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c81590a0 con 0x7fd7c815bc30 2020-11-20T12:09:13.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7c7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065f90 0x7fd7c81585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7c7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065f90 0x7fd7c81585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44666/0 (socket says 172.21.15.187:44666) 2020-11-20T12:09:13.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.675+0000 7fd7c7fff700 1 -- 172.21.15.187:0/2401985062 learned_addr learned my addr 172.21.15.187:0/2401985062 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:13.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7ccd67700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c815bc30 0x7fd7c815c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7bf7fe700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809ca90 0x7fd7c8067a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7c7fff700 1 -- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c815bc30 msgr2=0x7fd7c815c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7c7fff700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c815bc30 0x7fd7c815c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7c7fff700 1 -- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809ca90 msgr2=0x7fd7c8067a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7c7fff700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809ca90 0x7fd7c8067a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7c7fff700 1 -- 172.21.15.187:0/2401985062 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c8159340 con 0x7fd7c8065f90 2020-11-20T12:09:13.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7ccd67700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c815bc30 0x7fd7c815c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:13.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7bf7fe700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809ca90 0x7fd7c8067a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:13.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.676+0000 7fd7c7fff700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065f90 0x7fd7c81585e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7fd7c80b5fb0 tx=0x7fd7b8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:13.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.677+0000 7fd7c4ff9700 1 -- 172.21.15.187:0/2401985062 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8010770 con 0x7fd7c8065f90 2020-11-20T12:09:13.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.677+0000 7fd7c4ff9700 1 -- 172.21.15.187:0/2401985062 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b80108d0 con 0x7fd7c8065f90 2020-11-20T12:09:13.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.677+0000 7fd7c4ff9700 1 -- 172.21.15.187:0/2401985062 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8010c70 con 0x7fd7c8065f90 2020-11-20T12:09:13.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.677+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c8158ad0 con 0x7fd7c8065f90 2020-11-20T12:09:13.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.677+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c815c660 con 0x7fd7c8065f90 2020-11-20T12:09:13.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.678+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c81594d0 con 0x7fd7c8065f90 2020-11-20T12:09:13.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.678+0000 7fd7c4ff9700 1 -- 172.21.15.187:0/2401985062 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd7b8017070 con 0x7fd7c8065f90 2020-11-20T12:09:13.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.679+0000 7fd7c4ff9700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7a4056d20 0x7fd7a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:13.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.679+0000 7fd7c4ff9700 1 -- 172.21.15.187:0/2401985062 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd7b8071e90 con 0x7fd7c8065f90 2020-11-20T12:09:13.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.679+0000 7fd7bf7fe700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7a4056d20 0x7fd7a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:13.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.680+0000 7fd7bf7fe700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7a4056d20 0x7fd7a40591e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd7c809fbb0 tx=0x7fd7b0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:13.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.683+0000 7fd7c4ff9700 1 -- 172.21.15.187:0/2401985062 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd7b80472e0 con 0x7fd7c8065f90 2020-11-20T12:09:13.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.867+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd7c8061a20 con 0x7fd7c8065f90 2020-11-20T12:09:13.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.868+0000 7fd7c4ff9700 1 -- 172.21.15.187:0/2401985062 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v59) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd7b804a900 con 0x7fd7c8065f90 2020-11-20T12:09:13.873 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:09:13.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.870+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7a4056d20 msgr2=0x7fd7a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.870+0000 7fd7cd568700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7a4056d20 0x7fd7a40591e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd7c809fbb0 tx=0x7fd7b0009040).stop 2020-11-20T12:09:13.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.870+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065f90 msgr2=0x7fd7c81585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:13.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.870+0000 7fd7cd568700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065f90 0x7fd7c81585e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7fd7c80b5fb0 tx=0x7fd7b8016000).stop 2020-11-20T12:09:13.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.871+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 shutdown_connections 2020-11-20T12:09:13.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.871+0000 7fd7cd568700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7a4056d20 0x7fd7a40591e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.871+0000 7fd7cd568700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c815bc30 0x7fd7c815c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.871+0000 7fd7cd568700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7c8065f90 0x7fd7c81585e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.871+0000 7fd7cd568700 1 --2- 172.21.15.187:0/2401985062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809ca90 0x7fd7c8067a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:13.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.871+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 >> 172.21.15.187:0/2401985062 conn(0x7fd7c80bab30 msgr2=0x7fd7c80b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:13.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.871+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 shutdown_connections 2020-11-20T12:09:13.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:13.871+0000 7fd7cd568700 1 -- 172.21.15.187:0/2401985062 wait complete. 2020-11-20T12:09:13.899 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:09:14.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.166+0000 7f8b5e878700 1 Processor -- start 2020-11-20T12:09:14.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.167+0000 7f8b5e878700 1 -- start start 2020-11-20T12:09:14.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.168+0000 7f8b5e878700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8b580ba210 0x7f8b5814da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.168+0000 7f8b5e878700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b580baaa0 con 0x7f8b580ba5f0 2020-11-20T12:09:14.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.168+0000 7f8b5e878700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5814d090 con 0x7f8b580b6cf0 2020-11-20T12:09:14.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.168+0000 7f8b5e878700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5814d330 con 0x7f8b580ba210 2020-11-20T12:09:14.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.169+0000 7f8b5ce15700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8b580b6cf0 0x7f8b58099ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41284/0 (socket says 172.21.15.187:41284) 2020-11-20T12:09:14.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.169+0000 7f8b5ce15700 1 -- 172.21.15.187:0/618947339 learned_addr learned my addr 172.21.15.187:0/618947339 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:14.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.169+0000 7f8b57fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8b580ba210 0x7f8b5814da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.169+0000 7f8b577fe700 1 -- 172.21.15.187:0/618947339 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077162061 0 0) 0x7f8b5814d090 con 0x7f8b580b6cf0 2020-11-20T12:09:14.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.169+0000 7f8b577fe700 1 -- 172.21.15.187:0/618947339 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b38003240 con 0x7f8b580b6cf0 2020-11-20T12:09:14.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.169+0000 7f8b57fff700 1 -- 172.21.15.187:0/618947339 >> v1:172.21.15.187:6789/0 conn(0x7f8b580b6cf0 legacy=0x7f8b58099ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.169+0000 7f8b57fff700 1 -- 172.21.15.187:0/618947339 >> v1:172.21.15.5:6789/0 conn(0x7f8b580ba5f0 legacy=0x7f8b58065300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:14.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.169+0000 7f8b57fff700 1 -- 172.21.15.187:0/618947339 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5814d5d0 con 0x7f8b580ba210 2020-11-20T12:09:14.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.170+0000 7f8b57fff700 1 --2- 172.21.15.187:0/618947339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba210 0x7f8b5814da70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f8b4800b9a0 tx=0x7f8b48006df0).ready entity=mon.1 client_cookie=baf2309078dec838 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:14.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.170+0000 7f8b577fe700 1 -- 172.21.15.187:0/618947339 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48009770 con 0x7f8b580ba210 2020-11-20T12:09:14.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.170+0000 7f8b577fe700 1 -- 172.21.15.187:0/618947339 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b480098d0 con 0x7f8b580ba210 2020-11-20T12:09:14.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.170+0000 7f8b577fe700 1 -- 172.21.15.187:0/618947339 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48009c50 con 0x7f8b580ba210 2020-11-20T12:09:14.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.170+0000 7f8b5e878700 1 -- 172.21.15.187:0/618947339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba210 msgr2=0x7f8b5814da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.170+0000 7f8b5e878700 1 --2- 172.21.15.187:0/618947339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba210 0x7f8b5814da70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f8b4800b9a0 tx=0x7f8b48006df0).stop 2020-11-20T12:09:14.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.171+0000 7f8b5e878700 1 -- 172.21.15.187:0/618947339 shutdown_connections 2020-11-20T12:09:14.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.171+0000 7f8b5e878700 1 --2- 172.21.15.187:0/618947339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba210 0x7f8b5814da70 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.171+0000 7f8b5e878700 1 -- 172.21.15.187:0/618947339 >> 172.21.15.187:0/618947339 conn(0x7f8b580bacf0 msgr2=0x7f8b580bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:14.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.171+0000 7f8b5e878700 1 -- 172.21.15.187:0/618947339 shutdown_connections 2020-11-20T12:09:14.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.171+0000 7f8b5e878700 1 -- 172.21.15.187:0/618947339 wait complete. 2020-11-20T12:09:14.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.172+0000 7f8b5e878700 1 Processor -- start 2020-11-20T12:09:14.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.172+0000 7f8b5e878700 1 -- start start 2020-11-20T12:09:14.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.173+0000 7f8b5e878700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b580b6cf0 0x7f8b58149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.173+0000 7f8b5e878700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b580ba210 0x7f8b58090890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.173+0000 7f8b5e878700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba5f0 0x7f8b5815fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.173+0000 7f8b5e878700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580910b0 con 0x7f8b580b6cf0 2020-11-20T12:09:14.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.173+0000 7f8b5e878700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580b2e30 con 0x7f8b580ba5f0 2020-11-20T12:09:14.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.173+0000 7f8b5e878700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580b30d0 con 0x7f8b580ba210 2020-11-20T12:09:14.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba5f0 0x7f8b5815fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba5f0 0x7f8b5815fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44684/0 (socket says 172.21.15.187:44684) 2020-11-20T12:09:14.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 -- 172.21.15.187:0/4244865009 learned_addr learned my addr 172.21.15.187:0/4244865009 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:14.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b4f7fe700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b580ba210 0x7f8b58090890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 -- 172.21.15.187:0/4244865009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b580ba210 msgr2=0x7f8b58090890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b580ba210 0x7f8b58090890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 -- 172.21.15.187:0/4244865009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b580b6cf0 msgr2=0x7f8b58149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b580b6cf0 0x7f8b58149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:14.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 -- 172.21.15.187:0/4244865009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b58090d80 con 0x7f8b580ba5f0 2020-11-20T12:09:14.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.174+0000 7f8b5ce15700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba5f0 0x7f8b5815fe90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f8b50001420 tx=0x7f8b50007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:14.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.175+0000 7f8b54ff9700 1 -- 172.21.15.187:0/4244865009 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50005b80 con 0x7f8b580ba5f0 2020-11-20T12:09:14.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.175+0000 7f8b54ff9700 1 -- 172.21.15.187:0/4244865009 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b50005ce0 con 0x7f8b580ba5f0 2020-11-20T12:09:14.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.175+0000 7f8b54ff9700 1 -- 172.21.15.187:0/4244865009 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50007320 con 0x7f8b580ba5f0 2020-11-20T12:09:14.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.175+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5809e630 con 0x7f8b580ba5f0 2020-11-20T12:09:14.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.175+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b581605c0 con 0x7f8b580ba5f0 2020-11-20T12:09:14.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.176+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b580b3290 con 0x7f8b580ba5f0 2020-11-20T12:09:14.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.177+0000 7f8b54ff9700 1 -- 172.21.15.187:0/4244865009 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8b50010030 con 0x7f8b580ba5f0 2020-11-20T12:09:14.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.177+0000 7f8b54ff9700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d50 0x7f8b34059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.177+0000 7f8b54ff9700 1 -- 172.21.15.187:0/4244865009 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8b5006fef0 con 0x7f8b580ba5f0 2020-11-20T12:09:14.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.177+0000 7f8b57fff700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d50 0x7f8b34059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.178+0000 7f8b57fff700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d50 0x7f8b34059210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8b48006e30 tx=0x7f8b48015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:14.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.181+0000 7f8b54ff9700 1 -- 172.21.15.187:0/4244865009 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8b50045340 con 0x7f8b580ba5f0 2020-11-20T12:09:14.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.367+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8b5809e820 con 0x7f8b580ba5f0 2020-11-20T12:09:14.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.368+0000 7f8b54ff9700 1 -- 172.21.15.187:0/4244865009 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8b50048960 con 0x7f8b580ba5f0 2020-11-20T12:09:14.375 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:09:14.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d50 msgr2=0x7f8b34059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d50 0x7f8b34059210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8b48006e30 tx=0x7f8b48015000).stop 2020-11-20T12:09:14.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba5f0 msgr2=0x7f8b5815fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba5f0 0x7f8b5815fe90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f8b50001420 tx=0x7f8b50007e20).stop 2020-11-20T12:09:14.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 shutdown_connections 2020-11-20T12:09:14.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d50 0x7f8b34059210 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b580b6cf0 0x7f8b58149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:14.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b580ba5f0 0x7f8b5815fe90 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 --2- 172.21.15.187:0/4244865009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b580ba210 0x7f8b58090890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 >> 172.21.15.187:0/4244865009 conn(0x7f8b580bacf0 msgr2=0x7f8b580b70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:14.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 shutdown_connections 2020-11-20T12:09:14.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.371+0000 7f8b5e878700 1 -- 172.21.15.187:0/4244865009 wait complete. 2020-11-20T12:09:14.400 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:09:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.650+0000 7fb201825700 1 Processor -- start 2020-11-20T12:09:14.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.650+0000 7fb201825700 1 -- start start 2020-11-20T12:09:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.651+0000 7fb201825700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb1fc065190 0x7fb1fc065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.652+0000 7fb201825700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb1fc065f90 0x7fb1fc143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.652+0000 7fb201825700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc0b13e0 con 0x7fb1fc143c70 2020-11-20T12:09:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.652+0000 7fb201825700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1fc14ab30 con 0x7fb1fc065f90 2020-11-20T12:09:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.652+0000 7fb201825700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1fc062c10 con 0x7fb1fc065190 2020-11-20T12:09:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.652+0000 7fb200823700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb1fc065190 0x7fb1fc065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.652+0000 7fb200823700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb1fc065190 0x7fb1fc065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44690/0 (socket says 172.21.15.187:44690) 2020-11-20T12:09:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.652+0000 7fb200823700 1 -- 172.21.15.187:0/1440178374 learned_addr learned my addr 172.21.15.187:0/1440178374 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.652+0000 7fb1fbfff700 1 --2- 172.21.15.187:0/1440178374 >> v2:172.21.15.5:3300/0 conn(0x7fb1fc065f90 0x7fb1fc143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.653+0000 7fb200823700 1 -- 172.21.15.187:0/1440178374 >> v2:172.21.15.5:3300/0 conn(0x7fb1fc065f90 msgr2=0x7fb1fc143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.653+0000 7fb200823700 1 --2- 172.21.15.187:0/1440178374 >> v2:172.21.15.5:3300/0 conn(0x7fb1fc065f90 0x7fb1fc143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.653+0000 7fb200823700 1 -- 172.21.15.187:0/1440178374 >> v1:172.21.15.5:6789/0 conn(0x7fb1fc143c70 legacy=0x7fb1fc09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.653+0000 7fb200823700 1 -- 172.21.15.187:0/1440178374 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0b1640 con 0x7fb1fc065190 2020-11-20T12:09:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.653+0000 7fb1fbfff700 1 --2- 172.21.15.187:0/1440178374 >> v2:172.21.15.5:3300/0 conn(0x7fb1fc065f90 0x7fb1fc143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.653+0000 7fb200823700 1 --2- 172.21.15.187:0/1440178374 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc065190 0x7fb1fc065570 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fb1fc05c8b0 tx=0x7fb1f0006eb0).ready entity=mon.1 client_cookie=bc43583e61a24bde server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.653+0000 7fb1fb7fe700 1 -- 172.21.15.187:0/1440178374 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0009920 con 0x7fb1fc065190 2020-11-20T12:09:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb1fb7fe700 1 -- 172.21.15.187:0/1440178374 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f0009a80 con 0x7fb1fc065190 2020-11-20T12:09:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb1fb7fe700 1 -- 172.21.15.187:0/1440178374 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0009e00 con 0x7fb1fc065190 2020-11-20T12:09:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb201825700 1 -- 172.21.15.187:0/1440178374 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc065190 msgr2=0x7fb1fc065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb201825700 1 --2- 172.21.15.187:0/1440178374 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc065190 0x7fb1fc065570 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fb1fc05c8b0 tx=0x7fb1f0006eb0).stop 2020-11-20T12:09:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb201825700 1 -- 172.21.15.187:0/1440178374 shutdown_connections 2020-11-20T12:09:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb201825700 1 --2- 172.21.15.187:0/1440178374 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc065190 0x7fb1fc065570 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb201825700 1 --2- 172.21.15.187:0/1440178374 >> v2:172.21.15.5:3300/0 conn(0x7fb1fc065f90 0x7fb1fc143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb201825700 1 -- 172.21.15.187:0/1440178374 >> 172.21.15.187:0/1440178374 conn(0x7fb1fc0bab30 msgr2=0x7fb1fc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:14.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.654+0000 7fb201825700 1 -- 172.21.15.187:0/1440178374 shutdown_connections 2020-11-20T12:09:14.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.655+0000 7fb201825700 1 -- 172.21.15.187:0/1440178374 wait complete. 2020-11-20T12:09:14.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.656+0000 7fb201825700 1 Processor -- start 2020-11-20T12:09:14.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.657+0000 7fb201825700 1 -- start start 2020-11-20T12:09:14.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201825700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1fc065190 0x7fb1fc0b4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201825700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1fc065f90 0x7fb1fc15bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201825700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc143c70 0x7fb1fc15c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201825700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc158970 con 0x7fb1fc065f90 2020-11-20T12:09:14.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201825700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc158c10 con 0x7fb1fc143c70 2020-11-20T12:09:14.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201825700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc158eb0 con 0x7fb1fc065190 2020-11-20T12:09:14.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201024700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc143c70 0x7fb1fc15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201024700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc143c70 0x7fb1fc15c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44698/0 (socket says 172.21.15.187:44698) 2020-11-20T12:09:14.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.658+0000 7fb201024700 1 -- 172.21.15.187:0/1355481731 learned_addr learned my addr 172.21.15.187:0/1355481731 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:14.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb1fbfff700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1fc065f90 0x7fb1fc15bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb200823700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1fc065190 0x7fb1fc0b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb201024700 1 -- 172.21.15.187:0/1355481731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1fc065190 msgr2=0x7fb1fc0b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb201024700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1fc065190 0x7fb1fc0b4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb201024700 1 -- 172.21.15.187:0/1355481731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1fc065f90 msgr2=0x7fb1fc15bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb201024700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1fc065f90 0x7fb1fc15bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb201024700 1 -- 172.21.15.187:0/1355481731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc146870 con 0x7fb1fc143c70 2020-11-20T12:09:14.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb201024700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc143c70 0x7fb1fc15c040 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7fb1f400bae0 tx=0x7fb1f40090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:14.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb1f8ff9700 1 -- 172.21.15.187:0/1355481731 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400c890 con 0x7fb1fc143c70 2020-11-20T12:09:14.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb1f8ff9700 1 -- 172.21.15.187:0/1355481731 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f400ca10 con 0x7fb1fc143c70 2020-11-20T12:09:14.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc0b4700 con 0x7fb1fc143c70 2020-11-20T12:09:14.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.659+0000 7fb1f8ff9700 1 -- 172.21.15.187:0/1355481731 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400cbc0 con 0x7fb1fc143c70 2020-11-20T12:09:14.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.660+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc158580 con 0x7fb1fc143c70 2020-11-20T12:09:14.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.661+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc159070 con 0x7fb1fc143c70 2020-11-20T12:09:14.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.661+0000 7fb1f8ff9700 1 -- 172.21.15.187:0/1355481731 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb1f401c070 con 0x7fb1fc143c70 2020-11-20T12:09:14.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.662+0000 7fb1f8ff9700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb1d4056d20 0x7fb1d40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:14.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.662+0000 7fb1f8ff9700 1 -- 172.21.15.187:0/1355481731 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb1f406f720 con 0x7fb1fc143c70 2020-11-20T12:09:14.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.662+0000 7fb200823700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb1d4056d20 0x7fb1d40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:14.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.663+0000 7fb200823700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb1d4056d20 0x7fb1d40591e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb1f000b1b0 tx=0x7fb1f0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:14.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.666+0000 7fb1f8ff9700 1 -- 172.21.15.187:0/1355481731 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb1f4044b70 con 0x7fb1fc143c70 2020-11-20T12:09:14.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.854+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb1fc061a20 con 0x7fb1fc143c70 2020-11-20T12:09:14.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.855+0000 7fb1f8ff9700 1 -- 172.21.15.187:0/1355481731 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb1f40091f0 con 0x7fb1fc143c70 2020-11-20T12:09:14.861 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:09:14.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.857+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb1d4056d20 msgr2=0x7fb1d40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.857+0000 7fb201825700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb1d4056d20 0x7fb1d40591e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb1f000b1b0 tx=0x7fb1f0015000).stop 2020-11-20T12:09:14.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.857+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc143c70 msgr2=0x7fb1fc15c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:14.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.857+0000 7fb201825700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc143c70 0x7fb1fc15c040 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7fb1f400bae0 tx=0x7fb1f40090f0).stop 2020-11-20T12:09:14.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.858+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 shutdown_connections 2020-11-20T12:09:14.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.858+0000 7fb201825700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb1d4056d20 0x7fb1d40591e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.858+0000 7fb201825700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1fc143c70 0x7fb1fc15c040 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.858+0000 7fb201825700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1fc065190 0x7fb1fc0b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.858+0000 7fb201825700 1 --2- 172.21.15.187:0/1355481731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1fc065f90 0x7fb1fc15bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:14.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.858+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 >> 172.21.15.187:0/1355481731 conn(0x7fb1fc0bab30 msgr2=0x7fb1fc144190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:14.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.858+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 shutdown_connections 2020-11-20T12:09:14.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:14.858+0000 7fb201825700 1 -- 172.21.15.187:0/1355481731 wait complete. 2020-11-20T12:09:14.884 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:09:14.884 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:15.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.134+0000 7ffa78bcc700 1 Processor -- start 2020-11-20T12:09:15.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.134+0000 7ffa78bcc700 1 -- start start 2020-11-20T12:09:15.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.135+0000 7ffa78bcc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa740b1760 0x7ffa74065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:15.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.136+0000 7ffa78bcc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa74065680 0x7ffa7414c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:15.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.136+0000 7ffa78bcc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa74150390 con 0x7ffa740b1380 2020-11-20T12:09:15.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.136+0000 7ffa78bcc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa741511e0 con 0x7ffa740b1760 2020-11-20T12:09:15.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.136+0000 7ffa78bcc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7414d760 con 0x7ffa74065680 2020-11-20T12:09:15.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.136+0000 7ffa6bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa740b1760 0x7ffa74065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:15.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.136+0000 7ffa6bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa740b1760 0x7ffa74065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:36968/0 (socket says 172.21.15.187:36968) 2020-11-20T12:09:15.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.136+0000 7ffa6bfff700 1 -- 172.21.15.187:0/1144196239 learned_addr learned my addr 172.21.15.187:0/1144196239 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:15.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.136+0000 7ffa737fe700 1 --1- 172.21.15.187:0/1144196239 >> v1:172.21.15.5:6789/0 conn(0x7ffa740b1380 0x7ffa74150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:34818/0 (socket says 172.21.15.187:34818) 2020-11-20T12:09:15.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.137+0000 7ffa73fff700 1 --2- 172.21.15.187:0/1144196239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa74065680 0x7ffa7414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:15.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.137+0000 7ffa72ffd700 1 -- 172.21.15.187:0/1144196239 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362417105 0 0) 0x7ffa74150390 con 0x7ffa740b1380 2020-11-20T12:09:15.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.137+0000 7ffa72ffd700 1 -- 172.21.15.187:0/1144196239 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa54003240 con 0x7ffa740b1380 2020-11-20T12:09:15.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.138+0000 7ffa6bfff700 1 -- 172.21.15.187:0/1144196239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa74065680 msgr2=0x7ffa7414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:15.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.138+0000 7ffa6bfff700 1 --2- 172.21.15.187:0/1144196239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa74065680 0x7ffa7414c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.138+0000 7ffa6bfff700 1 -- 172.21.15.187:0/1144196239 >> v1:172.21.15.5:6789/0 conn(0x7ffa740b1380 legacy=0x7ffa74150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:15.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.138+0000 7ffa6bfff700 1 -- 172.21.15.187:0/1144196239 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74150630 con 0x7ffa740b1760 2020-11-20T12:09:15.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.138+0000 7ffa6bfff700 1 --2- 172.21.15.187:0/1144196239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1760 0x7ffa74065190 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ffa5c00ba20 tx=0x7ffa5c006eb0).ready entity=mon.0 client_cookie=ee1b30d7bdde2b2a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:15.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa72ffd700 1 -- 172.21.15.187:0/1144196239 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c0097d0 con 0x7ffa740b1760 2020-11-20T12:09:15.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa72ffd700 1 -- 172.21.15.187:0/1144196239 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5c009930 con 0x7ffa740b1760 2020-11-20T12:09:15.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa72ffd700 1 -- 172.21.15.187:0/1144196239 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c009cb0 con 0x7ffa740b1760 2020-11-20T12:09:15.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa78bcc700 1 -- 172.21.15.187:0/1144196239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1760 msgr2=0x7ffa74065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:15.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/1144196239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1760 0x7ffa74065190 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ffa5c00ba20 tx=0x7ffa5c006eb0).stop 2020-11-20T12:09:15.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa78bcc700 1 -- 172.21.15.187:0/1144196239 shutdown_connections 2020-11-20T12:09:15.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/1144196239 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa74065680 0x7ffa7414c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/1144196239 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1760 0x7ffa74065190 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa78bcc700 1 -- 172.21.15.187:0/1144196239 >> 172.21.15.187:0/1144196239 conn(0x7ffa740bab30 msgr2=0x7ffa740baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:15.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa78bcc700 1 -- 172.21.15.187:0/1144196239 shutdown_connections 2020-11-20T12:09:15.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.139+0000 7ffa78bcc700 1 -- 172.21.15.187:0/1144196239 wait complete. 2020-11-20T12:09:15.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.141+0000 7ffa78bcc700 1 Processor -- start 2020-11-20T12:09:15.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.141+0000 7ffa78bcc700 1 -- start start 2020-11-20T12:09:15.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa78bcc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1380 0x7ffa740b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:15.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa78bcc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa740b1760 0x7ffa74149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:15.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa78bcc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa740b7710 0x7ffa7414a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:15.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa78bcc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740b9b00 con 0x7ffa740b1380 2020-11-20T12:09:15.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa78bcc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740b9da0 con 0x7ffa740b7710 2020-11-20T12:09:15.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa78bcc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740ba040 con 0x7ffa740b1760 2020-11-20T12:09:15.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa73fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa740b7710 0x7ffa7414a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:15.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa73fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa740b7710 0x7ffa7414a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44712/0 (socket says 172.21.15.187:44712) 2020-11-20T12:09:15.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.142+0000 7ffa73fff700 1 -- 172.21.15.187:0/146748130 learned_addr learned my addr 172.21.15.187:0/146748130 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:15.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa737fe700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1380 0x7ffa740b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:15.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa6bfff700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa740b1760 0x7ffa74149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:15.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa73fff700 1 -- 172.21.15.187:0/146748130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa740b1760 msgr2=0x7ffa74149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:15.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa73fff700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa740b1760 0x7ffa74149b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa73fff700 1 -- 172.21.15.187:0/146748130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1380 msgr2=0x7ffa740b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:15.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa73fff700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1380 0x7ffa740b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa73fff700 1 -- 172.21.15.187:0/146748130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa740b9810 con 0x7ffa740b7710 2020-11-20T12:09:15.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa73fff700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa740b7710 0x7ffa7414a010 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7ffa6c00ed20 tx=0x7ffa6c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:15.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.143+0000 7ffa6b7fe700 1 -- 172.21.15.187:0/146748130 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c00cbb0 con 0x7ffa740b7710 2020-11-20T12:09:15.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.144+0000 7ffa6b7fe700 1 -- 172.21.15.187:0/146748130 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6c00cd10 con 0x7ffa740b7710 2020-11-20T12:09:15.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.144+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa740b4590 con 0x7ffa740b7710 2020-11-20T12:09:15.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.144+0000 7ffa6b7fe700 1 -- 172.21.15.187:0/146748130 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c012620 con 0x7ffa740b7710 2020-11-20T12:09:15.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.144+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa740b8b00 con 0x7ffa740b7710 2020-11-20T12:09:15.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.145+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa74061a20 con 0x7ffa740b7710 2020-11-20T12:09:15.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.145+0000 7ffa6b7fe700 1 -- 172.21.15.187:0/146748130 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffa6c022070 con 0x7ffa740b7710 2020-11-20T12:09:15.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.146+0000 7ffa6b7fe700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa50056d20 0x7ffa500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:15.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.146+0000 7ffa6b7fe700 1 -- 172.21.15.187:0/146748130 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffa6c0706d0 con 0x7ffa740b7710 2020-11-20T12:09:15.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.146+0000 7ffa737fe700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa50056d20 0x7ffa500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:15.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.147+0000 7ffa737fe700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa50056d20 0x7ffa500591e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7ffa64007de0 tx=0x7ffa64007140).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:15.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.150+0000 7ffa6b7fe700 1 -- 172.21.15.187:0/146748130 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffa6c045b20 con 0x7ffa740b7710 2020-11-20T12:09:15.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.397+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa74068510 con 0x7ffa740b7710 2020-11-20T12:09:15.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.398+0000 7ffa6b7fe700 1 -- 172.21.15.187:0/146748130 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+2133 (secure 0 0 0) 0x7ffa6c016020 con 0x7ffa740b7710 2020-11-20T12:09:15.402 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:15.402 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":34,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/170052250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":170052250},{"type":"v1","addr":"172.21.15.5:6833","nonce":170052250}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:09.281700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3570428099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3570428099},{"type":"v1","addr":"172.21.15.187:6835","nonce":3570428099}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa50056d20 msgr2=0x7ffa500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa50056d20 0x7ffa500591e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7ffa64007de0 tx=0x7ffa64007140).stop 2020-11-20T12:09:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa740b7710 msgr2=0x7ffa7414a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa740b7710 0x7ffa7414a010 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7ffa6c00ed20 tx=0x7ffa6c0091d0).stop 2020-11-20T12:09:15.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 shutdown_connections 2020-11-20T12:09:15.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa50056d20 0x7ffa500591e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740b1380 0x7ffa740b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa740b7710 0x7ffa7414a010 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 --2- 172.21.15.187:0/146748130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa740b1760 0x7ffa74149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:15.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.400+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 >> 172.21.15.187:0/146748130 conn(0x7ffa740bab30 msgr2=0x7ffa740baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:15.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.401+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 shutdown_connections 2020-11-20T12:09:15.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:15.401+0000 7ffa78bcc700 1 -- 172.21.15.187:0/146748130 wait complete. 2020-11-20T12:09:15.408 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 34 2020-11-20T12:09:15.628 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:09:07.256865+0000', 'modified': '2020-11-20T12:09:09.281700+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6010}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6010': {'gid': 6010, 'name': 'a', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3570428099', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3570428099}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3570428099}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:09:15.629 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:09:15.629 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:09:15.901 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6010,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":8.6163697940000006,"mdsmap_epoch":34,"osdmap_epoch":59,"osdmap_epoch_barrier":59,"uptime":20.204916921999999} 2020-11-20T12:09:15.902 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6010, "mdsmap_epoch": 34, "osdmap_epoch": 59, "osdmap_epoch_barrier": 59, "rank_uptime": 8.616369794, "state": "up:active", "uptime": 20.204916922, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:09:15.902 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:09:15.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:15.903 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:09:15.957 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:09:15.957 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:09:16.092 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:09:16.092 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:09:16.128 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:09:16.128 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:09:16.137 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:09:16.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:16.138 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:09:16.225 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:16.235 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:16.418 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:09:16.419 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:09:16.419 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:16.419 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:09:16.420 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.420 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:09:16.420 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.420 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:16.420 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.421 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:16.421 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:09:16.421 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.421 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:09:16.422 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:09:16.422 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.422 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:16.422 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:16.422 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.423 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:16.423 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:16.423 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.424 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:09:16.424 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.424 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:16.424 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:09:16.424 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:16.425 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:16.425 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:16.425 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:16.425 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:09:16.425 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:16.426 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:16.426 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:16.426 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.426 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:16.426 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.427 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:09:16.427 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.427 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:09:16.427 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.427 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:09:16.428 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:09:16.428 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:09:16.428 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:16.428 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:09:16.428 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:16.429 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.429 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:09:16.429 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:09:16.429 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.429 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:16.430 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:09:16.430 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.430 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:09:16.430 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:16.431 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.431 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:09:16.431 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:09:16.431 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:16.431 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.432 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:09:16.432 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:09:16.432 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.432 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:16.432 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.432 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:09:16.433 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.433 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:09:16.433 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.433 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:16.434 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:16.434 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.434 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:09:16.434 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:09:16.434 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:09:16.435 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:09:16.435 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:09:16.435 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:09:16.435 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:09:16.435 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:09:16.435 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:09:16.436 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:09:16.436 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:09:16.436 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:09:16.436 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:09:16.436 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:09:16.437 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:16.437 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:16.437 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:16.437 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:16.438 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:09:16.438 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:16.438 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.438 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:09:16.438 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:16.439 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:09:16.439 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:16.439 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:16.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:16.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:16.440 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:16.440 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.440 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:09:16.440 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:16.440 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:16.441 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:09:16.441 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:16.441 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:09:16.441 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:16.441 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:16.442 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:16.442 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:09:16.442 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:16.442 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:09:16.442 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.443 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:09:16.443 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:09:16.443 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:09:16.443 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:09:16.443 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:16.444 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:16.444 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:09:16.444 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:09:16.444 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:16.444 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:09:16.445 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:16.445 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:16.445 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:09:16.445 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:09:16.445 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:09:16.446 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:09:16.446 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:09:16.446 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:09:16.446 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:09:16.446 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:16.447 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:09:16.447 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:16.447 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:09:16.447 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:09:16.447 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:09:16.448 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:09:16.448 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:09:16.448 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:09:16.448 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:09:16.448 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:16.449 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:16.449 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:16.449 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 6 2020-11-20T12:09:16.450 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:09:16.450 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:09:16.450 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:16.450 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:09:16.451 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.451 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:09:16.451 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.451 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:16.451 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.452 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:16.452 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:09:16.452 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.452 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:09:16.452 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:09:16.453 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.453 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:16.453 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:16.453 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.453 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:16.454 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:16.454 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.454 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:09:16.454 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.455 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:16.455 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:09:16.455 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:16.455 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:16.455 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:16.456 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:16.456 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:09:16.456 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:16.456 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:16.456 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:16.457 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.457 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:16.457 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.457 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:09:16.457 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.458 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:09:16.458 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.458 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:09:16.458 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:09:16.458 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:09:16.459 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:16.459 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:09:16.459 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:16.459 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.459 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:09:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:09:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:09:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.461 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:09:16.461 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:16.461 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.461 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:09:16.461 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:09:16.462 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:16.462 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.462 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:09:16.462 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:09:16.462 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.463 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:16.463 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.463 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:09:16.463 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.463 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:09:16.464 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.464 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:16.464 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:16.464 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.464 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:09:16.465 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:09:16.465 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:09:16.465 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:09:16.465 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:09:16.465 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:09:16.466 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:09:16.466 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:09:16.466 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:09:16.466 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:09:16.466 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:09:16.467 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:09:16.467 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:09:16.467 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:09:16.467 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:16.468 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:16.468 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:16.468 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:16.468 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:09:16.468 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:16.469 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.469 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:09:16.469 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:16.469 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:09:16.469 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:16.470 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:16.470 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:16.470 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:16.470 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:16.470 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.471 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:09:16.471 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:16.471 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:16.471 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:09:16.472 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:16.472 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:09:16.472 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:16.472 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:16.472 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:16.473 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:09:16.473 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:16.473 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:09:16.473 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.473 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:09:16.474 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:09:16.474 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:09:16.474 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:09:16.474 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:16.474 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:16.475 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:09:16.475 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:09:16.475 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:16.475 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:09:16.475 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:16.476 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:16.476 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:09:16.476 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:09:16.476 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:09:16.476 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:09:16.477 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:09:16.477 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:09:16.477 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:09:16.477 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:16.478 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:09:16.478 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:16.478 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:09:16.478 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:09:16.478 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:09:16.478 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:09:16.479 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:09:16.479 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:09:16.479 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:09:16.479 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:16.479 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:16.480 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:16.480 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 6 2020-11-20T12:09:16.512 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:16.517 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:16.717 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:09:16.718 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:09:16.718 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:16.718 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:09:16.718 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.719 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:09:16.719 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.719 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:16.719 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.720 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:16.720 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:09:16.720 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.721 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:09:16.721 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:09:16.722 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.722 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:16.722 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:16.722 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.722 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:16.723 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:16.723 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.723 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:09:16.723 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.723 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:16.724 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:09:16.724 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:16.724 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:16.724 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:16.725 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:16.725 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:09:16.725 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:16.725 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:16.725 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:16.726 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.726 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:16.726 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.726 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:09:16.726 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.727 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:09:16.727 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.727 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:09:16.727 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:09:16.728 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:09:16.728 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:16.728 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:09:16.728 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:16.728 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.729 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:09:16.729 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:09:16.729 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.729 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:16.730 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:09:16.730 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.730 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:09:16.730 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:16.730 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.731 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:09:16.731 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:09:16.731 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:16.731 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.731 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:09:16.732 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:09:16.732 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.732 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:16.732 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.733 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:09:16.733 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.733 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:09:16.733 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.733 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:16.734 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:16.734 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.734 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:09:16.734 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:09:16.734 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:09:16.735 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:09:16.735 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:09:16.735 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:09:16.735 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:09:16.735 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:09:16.736 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:09:16.736 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:09:16.736 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:09:16.736 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:09:16.736 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:09:16.737 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:09:16.737 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:16.737 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:16.737 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:16.738 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:16.738 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:09:16.738 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:16.738 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.738 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:09:16.739 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:16.739 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:09:16.739 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:16.739 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:16.739 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:16.740 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:16.740 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:16.740 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.740 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:09:16.740 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:16.741 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:16.741 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:09:16.741 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:16.741 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:09:16.742 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:16.742 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:16.742 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:16.742 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:09:16.742 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:16.743 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:09:16.743 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:16.743 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:09:16.743 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:09:16.743 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:09:16.744 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:09:16.744 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:16.744 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:16.744 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:09:16.744 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:09:16.745 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:16.745 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:09:16.745 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:16.745 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:16.745 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:09:16.746 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:09:16.746 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:09:16.746 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:09:16.746 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:09:16.746 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:09:16.747 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:09:16.747 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:16.747 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:09:16.747 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:16.747 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:09:16.748 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:09:16.748 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:09:16.748 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:09:16.748 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:09:16.748 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:09:16.749 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:09:16.749 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:16.749 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:16.749 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:16.750 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 6 2020-11-20T12:09:16.750 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops = 100000000 2020-11-20T12:09:16.750 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:09:16.751 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:09:16.751 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:16.751 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:09:16.751 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.752 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:09:16.752 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.752 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:16.752 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.752 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:16.753 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:09:16.753 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.753 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:09:16.753 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:09:16.754 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.754 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:16.754 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:16.754 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.754 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:16.755 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:16.755 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.755 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:09:16.755 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.756 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:16.756 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:09:16.756 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:16.756 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:16.756 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:16.757 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:16.757 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:09:16.757 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:16.757 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:16.757 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:16.758 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.758 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:16.758 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.758 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:09:16.758 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.759 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:09:16.759 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.759 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:09:16.759 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:09:16.760 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:09:16.760 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:16.760 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:09:16.760 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:16.760 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.761 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:09:16.761 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:09:16.761 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.761 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:16.761 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:09:16.762 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.762 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:09:16.762 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:16.762 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.762 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:09:16.763 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:09:16.763 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:16.763 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.763 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:09:16.764 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:09:16.764 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.764 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:16.764 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.764 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:09:16.765 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.765 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:09:16.765 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.765 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:16.766 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:16.766 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.766 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:09:16.766 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:09:16.767 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:09:16.767 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:09:16.767 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:09:16.767 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:09:16.767 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:09:16.768 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:09:16.768 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:09:16.768 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:09:16.768 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:09:16.768 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:09:16.769 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:09:16.769 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:09:16.769 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:16.769 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:16.769 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:16.770 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:16.770 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:09:16.770 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:16.770 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.770 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:09:16.771 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:16.771 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:09:16.771 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:16.771 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:16.771 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:16.772 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:16.772 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:16.772 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.772 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:09:16.773 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:16.773 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:16.773 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:09:16.773 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:16.774 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:09:16.774 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:16.774 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:16.774 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:16.774 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:09:16.775 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:16.775 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:09:16.775 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:16.775 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:09:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:09:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:09:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:09:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:09:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:09:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:09:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:16.778 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:16.778 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:09:16.778 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:09:16.778 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:09:16.778 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:09:16.779 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:09:16.779 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:09:16.779 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:09:16.779 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:16.779 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:09:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:09:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:09:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:09:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:09:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:09:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:09:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:09:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:16.782 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:16.782 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 6 2020-11-20T12:09:16.782 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops = 100000000 2020-11-20T12:09:16.784 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:09:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.025+0000 7f7de8c48700 1 Processor -- start 2020-11-20T12:09:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.025+0000 7f7de8c48700 1 -- start start 2020-11-20T12:09:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.026+0000 7f7de8c48700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7de409cba0 0x7f7de4099e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.026+0000 7f7de8c48700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 0x7f7de4061f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.026+0000 7f7de8c48700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de4062470 con 0x7f7de409cf80 2020-11-20T12:09:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.026+0000 7f7de8c48700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de4067500 con 0x7f7de409cba0 2020-11-20T12:09:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.026+0000 7f7de8c48700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de40677a0 con 0x7f7de41464d0 2020-11-20T12:09:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de37fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7de409cba0 0x7f7de4099e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de37fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7de409cba0 0x7f7de4099e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44720/0 (socket says 172.21.15.187:44720) 2020-11-20T12:09:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de37fe700 1 -- 172.21.15.187:0/2367977583 learned_addr learned my addr 172.21.15.187:0/2367977583 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de3fff700 1 --2- 172.21.15.187:0/2367977583 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 0x7f7de4061f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de37fe700 1 -- 172.21.15.187:0/2367977583 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 msgr2=0x7f7de4061f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de37fe700 1 --2- 172.21.15.187:0/2367977583 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 0x7f7de4061f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de37fe700 1 -- 172.21.15.187:0/2367977583 >> v1:172.21.15.5:6789/0 conn(0x7f7de409cf80 legacy=0x7f7de4145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de37fe700 1 -- 172.21.15.187:0/2367977583 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de409e010 con 0x7f7de409cba0 2020-11-20T12:09:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.027+0000 7f7de37fe700 1 --2- 172.21.15.187:0/2367977583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de4099e10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f7dd4009d50 tx=0x7f7dd4009290).ready entity=mon.1 client_cookie=a5847a78a2f9c74d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de2ffd700 1 -- 172.21.15.187:0/2367977583 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd400bc40 con 0x7f7de409cba0 2020-11-20T12:09:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de2ffd700 1 -- 172.21.15.187:0/2367977583 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd400bda0 con 0x7f7de409cba0 2020-11-20T12:09:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de2ffd700 1 -- 172.21.15.187:0/2367977583 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4011610 con 0x7f7de409cba0 2020-11-20T12:09:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de8c48700 1 -- 172.21.15.187:0/2367977583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 msgr2=0x7f7de4099e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2367977583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de4099e10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f7dd4009d50 tx=0x7f7dd4009290).stop 2020-11-20T12:09:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de8c48700 1 -- 172.21.15.187:0/2367977583 shutdown_connections 2020-11-20T12:09:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2367977583 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 0x7f7de4061f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2367977583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de4099e10 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.028+0000 7f7de8c48700 1 -- 172.21.15.187:0/2367977583 >> 172.21.15.187:0/2367977583 conn(0x7f7de40b76c0 msgr2=0x7f7de40b7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:17.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.029+0000 7f7de8c48700 1 -- 172.21.15.187:0/2367977583 shutdown_connections 2020-11-20T12:09:17.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.029+0000 7f7de8c48700 1 -- 172.21.15.187:0/2367977583 wait complete. 2020-11-20T12:09:17.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.030+0000 7f7de8c48700 1 Processor -- start 2020-11-20T12:09:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.030+0000 7f7de8c48700 1 -- start start 2020-11-20T12:09:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.031+0000 7f7de8c48700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de40b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.031+0000 7f7de8c48700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de409cf80 0x7f7de414b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.031+0000 7f7de8c48700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 0x7f7de4142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.031+0000 7f7de8c48700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de40afe90 con 0x7f7de409cf80 2020-11-20T12:09:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.031+0000 7f7de8c48700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de40b0130 con 0x7f7de409cba0 2020-11-20T12:09:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.031+0000 7f7de8c48700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de40b03d0 con 0x7f7de41464d0 2020-11-20T12:09:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de40b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de40b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44726/0 (socket says 172.21.15.187:44726) 2020-11-20T12:09:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 -- 172.21.15.187:0/2666471379 learned_addr learned my addr 172.21.15.187:0/2666471379 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 -- 172.21.15.187:0/2666471379 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 msgr2=0x7f7de4142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 0x7f7de4142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 -- 172.21.15.187:0/2666471379 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de409cf80 msgr2=0x7f7de414b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de409cf80 0x7f7de414b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:17.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 -- 172.21.15.187:0/2666471379 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de40afb80 con 0x7f7de409cba0 2020-11-20T12:09:17.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.032+0000 7f7de37fe700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de40b0870 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f7de409d8b0 tx=0x7f7dd4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.033+0000 7f7ddbfff700 1 -- 172.21.15.187:0/2666471379 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4010780 con 0x7f7de409cba0 2020-11-20T12:09:17.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.033+0000 7f7ddbfff700 1 -- 172.21.15.187:0/2666471379 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd40108e0 con 0x7f7de409cba0 2020-11-20T12:09:17.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.033+0000 7f7ddbfff700 1 -- 172.21.15.187:0/2666471379 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4010c60 con 0x7f7de409cba0 2020-11-20T12:09:17.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.033+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de40b12f0 con 0x7f7de409cba0 2020-11-20T12:09:17.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.033+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de4142d50 con 0x7f7de409cba0 2020-11-20T12:09:17.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.034+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de40b0590 con 0x7f7de409cba0 2020-11-20T12:09:17.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.034+0000 7f7ddbfff700 1 -- 172.21.15.187:0/2666471379 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7dd4023030 con 0x7f7de409cba0 2020-11-20T12:09:17.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.035+0000 7f7ddbfff700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dc0056d20 0x7f7dc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.035+0000 7f7ddbfff700 1 -- 172.21.15.187:0/2666471379 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7dd4071690 con 0x7f7de409cba0 2020-11-20T12:09:17.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.035+0000 7f7ddaffd700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dc0056d20 0x7f7dc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.036+0000 7f7ddaffd700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dc0056d20 0x7f7dc00591e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f7de40b2930 tx=0x7f7dcc0077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.039+0000 7f7ddbfff700 1 -- 172.21.15.187:0/2666471379 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7dd4046b00 con 0x7f7de409cba0 2020-11-20T12:09:17.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.226+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7de4069ca0 con 0x7f7de409cba0 2020-11-20T12:09:17.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.228+0000 7f7ddbfff700 1 -- 172.21.15.187:0/2666471379 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7dd40210a0 con 0x7f7de409cba0 2020-11-20T12:09:17.234 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:09:17.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.230+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dc0056d20 msgr2=0x7f7dc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.230+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dc0056d20 0x7f7dc00591e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f7de40b2930 tx=0x7f7dcc0077f0).stop 2020-11-20T12:09:17.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.230+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 msgr2=0x7f7de40b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.230+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de40b0870 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f7de409d8b0 tx=0x7f7dd4016000).stop 2020-11-20T12:09:17.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.231+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 shutdown_connections 2020-11-20T12:09:17.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.231+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dc0056d20 0x7f7dc00591e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.231+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de41464d0 0x7f7de4142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:17.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.231+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de409cba0 0x7f7de40b0870 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.231+0000 7f7de8c48700 1 --2- 172.21.15.187:0/2666471379 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de409cf80 0x7f7de414b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:17.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.231+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 >> 172.21.15.187:0/2666471379 conn(0x7f7de40b76c0 msgr2=0x7f7de4144cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:17.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.231+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 shutdown_connections 2020-11-20T12:09:17.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.231+0000 7f7de8c48700 1 -- 172.21.15.187:0/2666471379 wait complete. 2020-11-20T12:09:17.260 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:17.268 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:17.461 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:09:17.461 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:09:17.461 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:17.462 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:09:17.462 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.462 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:09:17.462 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.463 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:17.463 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.463 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:17.463 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:09:17.463 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.464 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:09:17.464 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:09:17.464 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.464 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:17.464 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:17.465 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.465 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:17.465 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:17.465 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.466 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:09:17.466 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.466 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:17.466 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:09:17.466 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:17.467 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:17.467 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:17.467 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:17.467 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:09:17.467 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:17.468 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:17.468 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:17.468 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.468 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:17.468 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.469 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:09:17.469 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.469 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:09:17.469 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.470 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:09:17.470 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:09:17.470 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:09:17.470 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:17.470 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:09:17.471 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:17.471 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.471 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:09:17.471 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:09:17.471 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.472 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:17.472 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:09:17.472 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.472 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:09:17.473 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:17.473 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.473 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:09:17.473 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:09:17.473 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:17.474 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.474 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:09:17.474 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:09:17.474 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.474 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:17.475 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.475 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:09:17.475 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.475 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:09:17.475 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.476 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:17.476 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:17.476 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.476 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:09:17.476 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:09:17.477 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:09:17.477 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:09:17.477 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:09:17.477 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:09:17.478 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:09:17.478 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:09:17.478 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:09:17.478 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:09:17.478 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:09:17.479 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:09:17.479 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:09:17.479 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:09:17.479 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:17.479 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:17.480 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:17.480 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:17.480 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:09:17.480 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:17.480 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.481 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:09:17.481 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:17.481 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:09:17.481 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:17.481 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:17.482 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:17.482 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:17.482 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:17.482 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.483 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:09:17.483 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:17.483 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:17.483 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:09:17.483 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:17.484 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:09:17.484 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:17.484 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:17.484 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:17.484 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:09:17.485 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:17.485 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:09:17.485 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:17.485 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:09:17.485 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:09:17.486 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:09:17.486 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:09:17.486 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:17.486 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:17.487 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:09:17.487 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:09:17.487 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:17.487 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:09:17.487 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:17.488 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:17.488 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:09:17.488 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:09:17.488 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:09:17.488 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:09:17.489 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:09:17.489 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:09:17.489 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:09:17.489 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:17.489 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:09:17.490 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:17.490 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:09:17.490 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:09:17.490 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:09:17.490 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:09:17.491 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:09:17.491 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:09:17.491 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:09:17.491 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:17.491 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:17.492 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:17.492 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 6 2020-11-20T12:09:17.492 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops = 100000000 2020-11-20T12:09:17.492 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops_per_pg = 12500000.0 2020-11-20T12:09:17.493 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:09:17.493 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:09:17.493 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:17.494 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:09:17.494 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.494 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:09:17.494 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.494 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:17.495 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.495 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:17.495 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:09:17.495 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.495 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:09:17.496 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:09:17.496 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.496 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:17.496 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:17.497 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.497 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:17.497 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:17.497 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.497 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:09:17.498 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.498 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:17.498 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:09:17.498 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:17.498 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:17.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:17.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:17.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:09:17.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:17.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:17.500 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:17.500 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.500 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:17.500 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.500 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:09:17.501 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.501 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:09:17.501 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.501 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:09:17.502 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:09:17.502 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:09:17.502 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:17.502 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:09:17.502 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:17.503 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.503 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:09:17.503 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:09:17.503 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.503 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:17.504 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:09:17.504 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.504 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:09:17.504 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:17.504 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.505 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:09:17.505 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:09:17.505 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:17.505 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.505 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:09:17.506 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:09:17.506 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.506 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:17.506 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.506 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:09:17.507 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.507 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:09:17.507 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.507 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:17.507 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:17.508 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.508 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:09:17.508 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:09:17.508 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:09:17.508 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:09:17.509 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:09:17.509 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:09:17.509 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:09:17.509 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:09:17.510 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:09:17.510 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:09:17.510 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:09:17.510 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:09:17.510 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:09:17.511 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:09:17.511 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:17.511 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:17.511 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:17.511 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:17.512 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:09:17.512 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:17.512 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.512 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:09:17.512 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:17.513 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:09:17.513 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:17.513 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:17.513 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:17.513 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:17.514 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:17.514 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.514 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:09:17.514 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:17.514 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:17.515 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:09:17.515 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:17.515 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:09:17.515 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:17.515 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:17.516 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:17.516 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:09:17.516 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:17.516 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:09:17.516 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:17.517 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:09:17.517 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:09:17.517 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:09:17.517 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:09:17.518 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:17.518 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:17.518 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:09:17.518 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:09:17.518 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:17.519 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:09:17.519 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:17.519 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:17.519 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:09:17.519 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:09:17.520 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:09:17.520 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:09:17.520 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:09:17.520 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:09:17.520 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:09:17.521 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:17.521 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:09:17.521 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:17.521 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:09:17.521 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:09:17.522 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:09:17.522 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:09:17.522 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:09:17.522 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:09:17.522 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:09:17.523 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:17.523 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:17.523 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:17.523 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 6 2020-11-20T12:09:17.523 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops = 100000000 2020-11-20T12:09:17.524 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops_per_pg = 12500000.0 2020-11-20T12:09:17.527 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:09:17.527 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:09:17.528 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:09:17.528 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:09:17.546 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:09:17.547 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2020-11-20T12:09:17.548 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:09:17.548 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2020-11-20T12:09:17.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.806+0000 7f1710416700 1 Processor -- start 2020-11-20T12:09:17.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.807+0000 7f1710416700 1 -- start start 2020-11-20T12:09:17.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.808+0000 7f1710416700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170809f690 0x7f170812e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.808+0000 7f1710416700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170812eb20 con 0x7f17081469d0 2020-11-20T12:09:17.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.808+0000 7f1710416700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170812edc0 con 0x7f170809fa70 2020-11-20T12:09:17.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.808+0000 7f1710416700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1708145800 con 0x7f170809f690 2020-11-20T12:09:17.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.808+0000 7f170e9b3700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f170809fa70 0x7f1708145ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41346/0 (socket says 172.21.15.187:41346) 2020-11-20T12:09:17.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.808+0000 7f170e9b3700 1 -- 172.21.15.187:0/2185785328 learned_addr learned my addr 172.21.15.187:0/2185785328 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:17.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.809+0000 7f170e1b2700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170809f690 0x7f170812e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.809+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473127040 0 0) 0x7f170812edc0 con 0x7f170809fa70 2020-11-20T12:09:17.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.809+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f0003240 con 0x7f170809fa70 2020-11-20T12:09:17.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.809+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3780625420 0 0) 0x7f16f0003240 con 0x7f170809fa70 2020-11-20T12:09:17.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.809+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170809f690 msgr2=0x7f170812e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:17.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.809+0000 7f170d1b0700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170809f690 0x7f170812e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 >> v1:172.21.15.5:6789/0 conn(0x7f17081469d0 legacy=0x7f1708061200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170812e870 con 0x7f170809fa70 2020-11-20T12:09:17.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1700001860 con 0x7f170809fa70 2020-11-20T12:09:17.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 >> v1:172.21.15.187:6789/0 conn(0x7f170809fa70 legacy=0x7f1708145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f0003730 0x7f16f0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170809f690 0x7f16f00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f00049d0 0x7f16f0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1708145800 con 0x7f16f0003730 2020-11-20T12:09:17.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00041d0 con 0x7f16f00049d0 2020-11-20T12:09:17.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0004310 con 0x7f170809f690 2020-11-20T12:09:17.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.810+0000 7f170e9b3700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f00049d0 0x7f16f0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170d9b1700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170809f690 0x7f16f00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170e1b2700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f0003730 0x7f16f0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170e9b3700 1 -- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170809f690 msgr2=0x7f16f00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:17.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170e9b3700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170809f690 0x7f16f00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170e9b3700 1 -- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f0003730 msgr2=0x7f16f0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:17.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170e9b3700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f0003730 0x7f16f0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170e9b3700 1 -- 172.21.15.187:0/2185785328 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0003240 con 0x7f16f00049d0 2020-11-20T12:09:17.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170e9b3700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f00049d0 0x7f16f0004e50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f170000c840 tx=0x7f170809f190).ready entity=mon.1 client_cookie=8f270c095c74b5a1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.811+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700007b50 con 0x7f16f00049d0 2020-11-20T12:09:17.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1700005b80 con 0x7f16f00049d0 2020-11-20T12:09:17.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f170d1b0700 1 -- 172.21.15.187:0/2185785328 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700007110 con 0x7f16f00049d0 2020-11-20T12:09:17.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f1710416700 1 -- 172.21.15.187:0/2185785328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f00049d0 msgr2=0x7f16f0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f1710416700 1 --2- 172.21.15.187:0/2185785328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f00049d0 0x7f16f0004e50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f170000c840 tx=0x7f170809f190).stop 2020-11-20T12:09:17.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f170e9b3700 1 -- 172.21.15.187:0/2185785328 reap_dead start 2020-11-20T12:09:17.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f1710416700 1 -- 172.21.15.187:0/2185785328 shutdown_connections 2020-11-20T12:09:17.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f1710416700 1 -- 172.21.15.187:0/2185785328 >> 172.21.15.187:0/2185785328 conn(0x7f17080bb950 msgr2=0x7f17080b7920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:17.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f1710416700 1 -- 172.21.15.187:0/2185785328 shutdown_connections 2020-11-20T12:09:17.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.812+0000 7f1710416700 1 -- 172.21.15.187:0/2185785328 wait complete. 2020-11-20T12:09:17.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f1710416700 1 Processor -- start 2020-11-20T12:09:17.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f1710416700 1 -- start start 2020-11-20T12:09:17.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f1710416700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17080602b0 0x7f17080b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f1710416700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1708060690 0x7f1708155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f1710416700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170815c1f0 0x7f17081562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f1710416700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17080b52a0 con 0x7f17080602b0 2020-11-20T12:09:17.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f1710416700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170815c6c0 con 0x7f1708060690 2020-11-20T12:09:17.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f1710416700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170815c960 con 0x7f170815c1f0 2020-11-20T12:09:17.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f170d9b1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1708060690 0x7f1708155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f170d9b1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1708060690 0x7f1708155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44756/0 (socket says 172.21.15.187:44756) 2020-11-20T12:09:17.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f170d9b1700 1 -- 172.21.15.187:0/678327484 learned_addr learned my addr 172.21.15.187:0/678327484 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:17.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f170e9b3700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170815c1f0 0x7f17081562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f170e1b2700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17080602b0 0x7f17080b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f170d9b1700 1 -- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170815c1f0 msgr2=0x7f17081562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f170d9b1700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170815c1f0 0x7f17081562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f170d9b1700 1 -- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17080602b0 msgr2=0x7f17080b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f170d9b1700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17080602b0 0x7f17080b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f170d9b1700 1 -- 172.21.15.187:0/678327484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170815cc00 con 0x7f1708060690 2020-11-20T12:09:17.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f170e9b3700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170815c1f0 0x7f17081562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:17.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f170e1b2700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17080602b0 0x7f17080b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:17.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f170d9b1700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1708060690 0x7f1708155de0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f16f800e650 tx=0x7f16f8009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f16f67fc700 1 -- 172.21.15.187:0/678327484 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8010550 con 0x7f1708060690 2020-11-20T12:09:17.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f16f67fc700 1 -- 172.21.15.187:0/678327484 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f8010040 con 0x7f1708060690 2020-11-20T12:09:17.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f16f67fc700 1 -- 172.21.15.187:0/678327484 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f80108c0 con 0x7f1708060690 2020-11-20T12:09:17.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170815d0a0 con 0x7f1708060690 2020-11-20T12:09:17.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170815f7c0 con 0x7f1708060690 2020-11-20T12:09:17.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.818+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170815cd90 con 0x7f1708060690 2020-11-20T12:09:17.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.819+0000 7f16f67fc700 1 -- 172.21.15.187:0/678327484 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f16f8010cb0 con 0x7f1708060690 2020-11-20T12:09:17.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f16f67fc700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f16f67fc700 1 -- 172.21.15.187:0/678327484 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f16f806ffd0 con 0x7f1708060690 2020-11-20T12:09:17.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.821+0000 7f170e1b2700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.821+0000 7f170e1b2700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f1704006a70 tx=0x7f1704008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.813+0000 7f13bb59e700 1 Processor -- start 2020-11-20T12:09:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.813+0000 7f13bb59e700 1 -- start start 2020-11-20T12:09:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13bb59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f13bc148510 0x7f13bc067530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13bb59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13bc066d00 0x7f13bc065b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13bb59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b50d0 con 0x7f13bc066000 2020-11-20T12:09:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13bb59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc0b63a0 con 0x7f13bc148510 2020-11-20T12:09:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13bb59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc0b6640 con 0x7f13bc066d00 2020-11-20T12:09:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13b9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13bc066d00 0x7f13bc065b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13b9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13bc066d00 0x7f13bc065b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44750/0 (socket says 172.21.15.187:44750) 2020-11-20T12:09:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13b9d9b700 1 -- 172.21.15.187:0/536469050 learned_addr learned my addr 172.21.15.187:0/536469050 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.814+0000 7f13bad9d700 1 --1- 172.21.15.187:0/536469050 >> v1:172.21.15.187:6789/0 conn(0x7f13bc066000 0x7f13bc0afc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41358/0 (socket says 172.21.15.187:41358) 2020-11-20T12:09:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13ba59c700 1 --2- 172.21.15.187:0/536469050 >> v2:172.21.15.5:3300/0 conn(0x7f13bc148510 0x7f13bc067530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13b9d9b700 1 -- 172.21.15.187:0/536469050 >> v2:172.21.15.5:3300/0 conn(0x7f13bc148510 msgr2=0x7f13bc067530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13b9d9b700 1 --2- 172.21.15.187:0/536469050 >> v2:172.21.15.5:3300/0 conn(0x7f13bc148510 0x7f13bc067530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13b9d9b700 1 -- 172.21.15.187:0/536469050 >> v1:172.21.15.187:6789/0 conn(0x7f13bc066000 legacy=0x7f13bc0afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13b9d9b700 1 -- 172.21.15.187:0/536469050 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0b68e0 con 0x7f13bc066d00 2020-11-20T12:09:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13b9d9b700 1 --2- 172.21.15.187:0/536469050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc065b10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f13bc0b6e40 tx=0x7f13b0006eb0).ready entity=mon.1 client_cookie=3df029a44cbbf9e2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13b959a700 1 -- 172.21.15.187:0/536469050 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0009960 con 0x7f13bc066d00 2020-11-20T12:09:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13b959a700 1 -- 172.21.15.187:0/536469050 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b0009ac0 con 0x7f13bc066d00 2020-11-20T12:09:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.815+0000 7f13b959a700 1 -- 172.21.15.187:0/536469050 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0009e40 con 0x7f13bc066d00 2020-11-20T12:09:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f13bb59e700 1 -- 172.21.15.187:0/536469050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 msgr2=0x7f13bc065b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f13bb59e700 1 --2- 172.21.15.187:0/536469050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc065b10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f13bc0b6e40 tx=0x7f13b0006eb0).stop 2020-11-20T12:09:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f13bb59e700 1 -- 172.21.15.187:0/536469050 shutdown_connections 2020-11-20T12:09:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f13bb59e700 1 --2- 172.21.15.187:0/536469050 >> v2:172.21.15.5:3300/0 conn(0x7f13bc148510 0x7f13bc067530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f13bb59e700 1 --2- 172.21.15.187:0/536469050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc065b10 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.816+0000 7f13bb59e700 1 -- 172.21.15.187:0/536469050 >> 172.21.15.187:0/536469050 conn(0x7f13bc0b8240 msgr2=0x7f13bc065400 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f13bb59e700 1 -- 172.21.15.187:0/536469050 shutdown_connections 2020-11-20T12:09:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.817+0000 7f13bb59e700 1 -- 172.21.15.187:0/536469050 wait complete. 2020-11-20T12:09:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.818+0000 7f13bb59e700 1 Processor -- start 2020-11-20T12:09:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.819+0000 7f13bb59e700 1 -- start start 2020-11-20T12:09:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.819+0000 7f13bb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13bc066000 0x7f13bc0b9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13bb59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc14a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13bb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13bc148510 0x7f13bc146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13bb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0a0380 con 0x7f13bc066000 2020-11-20T12:09:17.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13bb59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0a0570 con 0x7f13bc066d00 2020-11-20T12:09:17.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13bb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc149fb0 con 0x7f13bc148510 2020-11-20T12:09:17.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13b9d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc14a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13b9d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc14a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44762/0 (socket says 172.21.15.187:44762) 2020-11-20T12:09:17.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13b9d9b700 1 -- 172.21.15.187:0/3997976217 learned_addr learned my addr 172.21.15.187:0/3997976217 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13bad9d700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13bc148510 0x7f13bc146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.820+0000 7f13ba59c700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13bc066000 0x7f13bc0b9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.821+0000 7f13b9d9b700 1 -- 172.21.15.187:0/3997976217 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13bc148510 msgr2=0x7f13bc146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.821+0000 7f13b9d9b700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13bc148510 0x7f13bc146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.821+0000 7f13b9d9b700 1 -- 172.21.15.187:0/3997976217 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13bc066000 msgr2=0x7f13bc0b9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:17.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.821+0000 7f13b9d9b700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13bc066000 0x7f13bc0b9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:17.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.821+0000 7f13b9d9b700 1 -- 172.21.15.187:0/3997976217 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0ba240 con 0x7f13bc066d00 2020-11-20T12:09:17.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.821+0000 7f13b9d9b700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc14a450 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f13bc149e00 tx=0x7f13b0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.824+0000 7f13aa7fc700 1 -- 172.21.15.187:0/3997976217 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b001e460 con 0x7f13bc066d00 2020-11-20T12:09:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.824+0000 7f13aa7fc700 1 -- 172.21.15.187:0/3997976217 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b001e5c0 con 0x7f13bc066d00 2020-11-20T12:09:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.824+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc0b3cb0 con 0x7f13bc066d00 2020-11-20T12:09:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.824+0000 7f13aa7fc700 1 -- 172.21.15.187:0/3997976217 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b001e960 con 0x7f13bc066d00 2020-11-20T12:09:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.824+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc14ab00 con 0x7f13bc066d00 2020-11-20T12:09:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.825+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc0b3f80 con 0x7f13bc066d00 2020-11-20T12:09:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.825+0000 7f13aa7fc700 1 -- 172.21.15.187:0/3997976217 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f13b0025030 con 0x7f13bc066d00 2020-11-20T12:09:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.826+0000 7f13aa7fc700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1390056d20 0x7f13900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.826+0000 7f13aa7fc700 1 -- 172.21.15.187:0/3997976217 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13b00717a0 con 0x7f13bc066d00 2020-11-20T12:09:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.826+0000 7f13ba59c700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1390056d20 0x7f13900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.827+0000 7f13ba59c700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1390056d20 0x7f13900591e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f13ac00d8f0 tx=0x7f13ac008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.830+0000 7f13aa7fc700 1 -- 172.21.15.187:0/3997976217 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f13b0046c40 con 0x7f13bc066d00 2020-11-20T12:09:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:17.824+0000 7f16f67fc700 1 -- 172.21.15.187:0/678327484 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f16f8045420 con 0x7f1708060690 2020-11-20T12:09:18.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.083+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f1708069660 con 0x7f1708060690 2020-11-20T12:09:18.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.089+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f13bc14a170 con 0x7f13bc066d00 2020-11-20T12:09:18.350 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:09:18.346+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:09:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.361+0000 7f16f67fc700 1 -- 172.21.15.187:0/678327484 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 4847 v35) v1 ==== 95+0+0 (secure 0 0 0) 0x7f16f8048a40 con 0x7f1708060690 2020-11-20T12:09:18.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.361+0000 7f13aa7fc700 1 -- 172.21.15.187:0/3997976217 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6010 v35) v1 ==== 95+0+0 (secure 0 0 0) 0x7f13b0023020 con 0x7f13bc066d00 2020-11-20T12:09:18.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.365+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1390056d20 msgr2=0x7f13900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.365+0000 7f13bb59e700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1390056d20 0x7f13900591e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f13ac00d8f0 tx=0x7f13ac008040).stop 2020-11-20T12:09:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.365+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 msgr2=0x7f13bc14a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.365+0000 7f13bb59e700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc14a450 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f13bc149e00 tx=0x7f13b0015000).stop 2020-11-20T12:09:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 shutdown_connections 2020-11-20T12:09:18.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f13bb59e700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1390056d20 0x7f13900591e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f13bb59e700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13bc066000 0x7f13bc0b9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f13bb59e700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13bc148510 0x7f13bc146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f13bb59e700 1 --2- 172.21.15.187:0/3997976217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13bc066d00 0x7f13bc14a450 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 >> 172.21.15.187:0/3997976217 conn(0x7f13bc0b8240 msgr2=0x7f13bc1506c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:18.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 shutdown_connections 2020-11-20T12:09:18.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f13bb59e700 1 -- 172.21.15.187:0/3997976217 wait complete. 2020-11-20T12:09:18.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.365+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 msgr2=0x7f16e00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.365+0000 7f1710416700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f1704006a70 tx=0x7f1704008040).stop 2020-11-20T12:09:18.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.365+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1708060690 msgr2=0x7f1708155de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.365+0000 7f1710416700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1708060690 0x7f1708155de0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f16f800e650 tx=0x7f16f8009040).stop 2020-11-20T12:09:18.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 shutdown_connections 2020-11-20T12:09:18.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f1710416700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f1710416700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17080602b0 0x7f17080b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f1710416700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f170815c1f0 0x7f17081562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f1710416700 1 --2- 172.21.15.187:0/678327484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1708060690 0x7f1708155de0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 >> 172.21.15.187:0/678327484 conn(0x7f17080bb950 msgr2=0x7f17080b7090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:18.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 shutdown_connections 2020-11-20T12:09:18.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.366+0000 7f1710416700 1 -- 172.21.15.187:0/678327484 wait complete. 2020-11-20T12:09:18.376 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 4847 2020-11-20T12:09:18.376 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 6010 2020-11-20T12:09:18.404 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:09:18.405 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:09:18.406 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:09:18.406 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:09:18.412 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:09:18.413 DEBUG:teuthology.parallel:result is None 2020-11-20T12:09:18.415 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:09:18.416 DEBUG:teuthology.parallel:result is None 2020-11-20T12:09:18.417 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:18.569 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:09:18.619 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:09:18.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.620+0000 7f86d7fff700 1 Processor -- start 2020-11-20T12:09:18.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.621+0000 7f86d7fff700 1 -- start start 2020-11-20T12:09:18.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.622+0000 7f86d7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f86d8065c90 0x7f86d80b5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:18.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.622+0000 7f86d7fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d8149f60 con 0x7f86d8144070 2020-11-20T12:09:18.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.622+0000 7f86d7fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d814a200 con 0x7f86d8066070 2020-11-20T12:09:18.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.622+0000 7f86d7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d814a4a0 con 0x7f86d8065c90 2020-11-20T12:09:18.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.622+0000 7f86cffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f86d8066070 0x7f86d8143960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41390/0 (socket says 172.21.15.187:41390) 2020-11-20T12:09:18.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.622+0000 7f86cffff700 1 -- 172.21.15.187:0/4197216838 learned_addr learned my addr 172.21.15.187:0/4197216838 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:18.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.622+0000 7f86d6ffd700 1 --2- 172.21.15.187:0/4197216838 >> v2:172.21.15.5:3300/0 conn(0x7f86d8065c90 0x7f86d80b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:18.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638357599 0 0) 0x7f86d814a200 con 0x7f86d8066070 2020-11-20T12:09:18.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b8003240 con 0x7f86d8066070 2020-11-20T12:09:18.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650156077 0 0) 0x7f86d8149f60 con 0x7f86d8144070 2020-11-20T12:09:18.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d814a200 con 0x7f86d8144070 2020-11-20T12:09:18.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 58222154 0 0) 0x7f86b8003240 con 0x7f86d8066070 2020-11-20T12:09:18.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 >> v2:172.21.15.5:3300/0 conn(0x7f86d8065c90 msgr2=0x7f86d80b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:18.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 --2- 172.21.15.187:0/4197216838 >> v2:172.21.15.5:3300/0 conn(0x7f86d8065c90 0x7f86d80b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 >> v1:172.21.15.5:6789/0 conn(0x7f86d8144070 legacy=0x7f86d8099ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d809a8e0 con 0x7f86d8066070 2020-11-20T12:09:18.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f86c00014e0 con 0x7f86d8066070 2020-11-20T12:09:18.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d6ffd700 1 --2- 172.21.15.187:0/4197216838 >> v2:172.21.15.5:3300/0 conn(0x7f86d8065c90 0x7f86d80b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:09:18.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 >> v1:172.21.15.187:6789/0 conn(0x7f86d8066070 legacy=0x7f86d8143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80038c0 0x7f86b8003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:18.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86b80044e0 0x7f86b8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:18.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86b8004e50 0x7f86b800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:18.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d814a4a0 con 0x7f86b8004e50 2020-11-20T12:09:18.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80053b0 con 0x7f86b80044e0 2020-11-20T12:09:18.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.623+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b800bca0 con 0x7f86b80038c0 2020-11-20T12:09:18.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86cffff700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86b80044e0 0x7f86b8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:18.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86d6ffd700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80038c0 0x7f86b8003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:18.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86d77fe700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86b8004e50 0x7f86b800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:18.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86cffff700 1 -- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80038c0 msgr2=0x7f86b8003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:18.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86cffff700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80038c0 0x7f86b8003cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86cffff700 1 -- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86b8004e50 msgr2=0x7f86b800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:18.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86cffff700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86b8004e50 0x7f86b800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86cffff700 1 -- 172.21.15.187:0/4197216838 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b8003240 con 0x7f86b80044e0 2020-11-20T12:09:18.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86d77fe700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86b8004e50 0x7f86b800b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86d77fe700 1 -- 172.21.15.187:0/4197216838 reap_dead start 2020-11-20T12:09:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86cffff700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86b80044e0 0x7f86b8004960 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f86c000c420 tx=0x7f86c0007780).ready entity=mon.1 client_cookie=da70743f30018969 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.624+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c00070d0 con 0x7f86b80044e0 2020-11-20T12:09:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c0007230 con 0x7f86b80044e0 2020-11-20T12:09:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d67fc700 1 -- 172.21.15.187:0/4197216838 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c00055f0 con 0x7f86b80044e0 2020-11-20T12:09:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d7fff700 1 -- 172.21.15.187:0/4197216838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86b80044e0 msgr2=0x7f86b8004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d7fff700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86b80044e0 0x7f86b8004960 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f86c000c420 tx=0x7f86c0007780).stop 2020-11-20T12:09:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d7fff700 1 -- 172.21.15.187:0/4197216838 shutdown_connections 2020-11-20T12:09:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d7fff700 1 --2- 172.21.15.187:0/4197216838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86b80044e0 0x7f86b8004960 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d7fff700 1 -- 172.21.15.187:0/4197216838 >> 172.21.15.187:0/4197216838 conn(0x7f86d80b8280 msgr2=0x7f86d80653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d7fff700 1 -- 172.21.15.187:0/4197216838 shutdown_connections 2020-11-20T12:09:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.625+0000 7f86d7fff700 1 -- 172.21.15.187:0/4197216838 wait complete. 2020-11-20T12:09:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.627+0000 7f86d7fff700 1 Processor -- start 2020-11-20T12:09:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.627+0000 7f86d7fff700 1 -- start start 2020-11-20T12:09:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86d7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80044e0 0x7f86d80b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86d7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d8099890 0x7f86d81580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86d7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86d809ded0 0x7f86d809e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86d7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d808ea20 con 0x7f86d809ded0 2020-11-20T12:09:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86d7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d8158a00 con 0x7f86d8099890 2020-11-20T12:09:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86d7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d8158ca0 con 0x7f86b80044e0 2020-11-20T12:09:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86cffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d8099890 0x7f86d81580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86cffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d8099890 0x7f86d81580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44796/0 (socket says 172.21.15.187:44796) 2020-11-20T12:09:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86cffff700 1 -- 172.21.15.187:0/1954578048 learned_addr learned my addr 172.21.15.187:0/1954578048 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86d6ffd700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80044e0 0x7f86d80b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:18.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.628+0000 7f86d77fe700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86d809ded0 0x7f86d809e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:18.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86cffff700 1 -- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80044e0 msgr2=0x7f86d80b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86cffff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80044e0 0x7f86d80b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86cffff700 1 -- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86d809ded0 msgr2=0x7f86d809e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86cffff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86d809ded0 0x7f86d809e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86cffff700 1 -- 172.21.15.187:0/1954578048 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d809f790 con 0x7f86d8099890 2020-11-20T12:09:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86d77fe700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86d809ded0 0x7f86d809e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86cffff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d8099890 0x7f86d81580f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f86b800bc70 tx=0x7f86c0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86ceffd700 1 -- 172.21.15.187:0/1954578048 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0020480 con 0x7f86d8099890 2020-11-20T12:09:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86ceffd700 1 -- 172.21.15.187:0/1954578048 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c00205e0 con 0x7f86d8099890 2020-11-20T12:09:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86ceffd700 1 -- 172.21.15.187:0/1954578048 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c00209a0 con 0x7f86d8099890 2020-11-20T12:09:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d81585e0 con 0x7f86d8099890 2020-11-20T12:09:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.629+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d808f050 con 0x7f86d8099890 2020-11-20T12:09:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.630+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d8068ef0 con 0x7f86d8099890 2020-11-20T12:09:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.631+0000 7f86ceffd700 1 -- 172.21.15.187:0/1954578048 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f86c0010070 con 0x7f86d8099890 2020-11-20T12:09:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.632+0000 7f86ceffd700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f86b4056d20 0x7f86b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.632+0000 7f86ceffd700 1 -- 172.21.15.187:0/1954578048 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f86c0071640 con 0x7f86d8099890 2020-11-20T12:09:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.632+0000 7f86d6ffd700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f86b4056d20 0x7f86b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.632+0000 7f86d6ffd700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f86b4056d20 0x7f86b40591e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f86c8006a90 tx=0x7f86c8006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:18.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.635+0000 7f86ceffd700 1 -- 172.21.15.187:0/1954578048 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f86c0046a40 con 0x7f86d8099890 2020-11-20T12:09:18.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.865+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86d809f920 con 0x7f86d8099890 2020-11-20T12:09:18.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.866+0000 7f86ceffd700 1 -- 172.21.15.187:0/1954578048 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+1423 (secure 0 0 0) 0x7f86c004a060 con 0x7f86d8099890 2020-11-20T12:09:18.870 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:18.871 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:18.346613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:18.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f86b4056d20 msgr2=0x7f86b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f86b4056d20 0x7f86b40591e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f86c8006a90 tx=0x7f86c8006df0).stop 2020-11-20T12:09:18.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d8099890 msgr2=0x7f86d81580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:18.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d8099890 0x7f86d81580f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f86b800bc70 tx=0x7f86c0015000).stop 2020-11-20T12:09:18.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 shutdown_connections 2020-11-20T12:09:18.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f86b4056d20 0x7f86b40591e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86d809ded0 0x7f86d809e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86b80044e0 0x7f86d80b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 --2- 172.21.15.187:0/1954578048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d8099890 0x7f86d81580f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:18.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 >> 172.21.15.187:0/1954578048 conn(0x7f86d80b8280 msgr2=0x7f86d80b8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:18.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.868+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 shutdown_connections 2020-11-20T12:09:18.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:18.869+0000 7f86d7fff700 1 -- 172.21.15.187:0/1954578048 wait complete. 2020-11-20T12:09:18.876 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 35 2020-11-20T12:09:19.096 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:09:07.256865+0000', 'modified': '2020-11-20T12:09:18.346613+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 61, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:09:19.097 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2020-11-20T12:09:19.572 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:19.584 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:20.098 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:20.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.335+0000 7f6792fc5700 1 Processor -- start 2020-11-20T12:09:20.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.335+0000 7f6792fc5700 1 -- start start 2020-11-20T12:09:20.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.336+0000 7f6792fc5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c066070 0x7f678c144070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:20.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.336+0000 7f6792fc5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f678c0b5750 0x7f678c09a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:20.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.336+0000 7f6792fc5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c149f60 con 0x7f678c065c90 2020-11-20T12:09:20.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.336+0000 7f6792fc5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678c14a200 con 0x7f678c0b5750 2020-11-20T12:09:20.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.336+0000 7f6792fc5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c14a4a0 con 0x7f678c066070 2020-11-20T12:09:20.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.337+0000 7f67927c4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f678c0b5750 0x7f678c09a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:20.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.337+0000 7f6791fc3700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f678c065c90 0x7f678c143960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41408/0 (socket says 172.21.15.187:41408) 2020-11-20T12:09:20.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.337+0000 7f6791fc3700 1 -- 172.21.15.187:0/2243608087 learned_addr learned my addr 172.21.15.187:0/2243608087 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:20.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.337+0000 7f67917c2700 1 --2- 172.21.15.187:0/2243608087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c066070 0x7f678c144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:20.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.337+0000 7f67927c4700 1 -- 172.21.15.187:0/2243608087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c066070 msgr2=0x7f678c144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:20.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.337+0000 7f67927c4700 1 --2- 172.21.15.187:0/2243608087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c066070 0x7f678c144070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:20.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.338+0000 7f67927c4700 1 -- 172.21.15.187:0/2243608087 >> v1:172.21.15.187:6789/0 conn(0x7f678c065c90 legacy=0x7f678c143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:20.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.338+0000 7f67927c4700 1 -- 172.21.15.187:0/2243608087 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c14a940 con 0x7f678c0b5750 2020-11-20T12:09:20.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.338+0000 7f67917c2700 1 --2- 172.21.15.187:0/2243608087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c066070 0x7f678c144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:20.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.338+0000 7f67927c4700 1 --2- 172.21.15.187:0/2243608087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c0b5750 0x7f678c09a8e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f678400b9a0 tx=0x7f6784006df0).ready entity=mon.1 client_cookie=ed8790ee885db639 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:20.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.338+0000 7f6790fc1700 1 -- 172.21.15.187:0/2243608087 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784009840 con 0x7f678c0b5750 2020-11-20T12:09:20.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.338+0000 7f6790fc1700 1 -- 172.21.15.187:0/2243608087 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67840099a0 con 0x7f678c0b5750 2020-11-20T12:09:20.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.338+0000 7f6790fc1700 1 -- 172.21.15.187:0/2243608087 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784009d20 con 0x7f678c0b5750 2020-11-20T12:09:20.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.339+0000 7f6792fc5700 1 -- 172.21.15.187:0/2243608087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c0b5750 msgr2=0x7f678c09a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:20.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.339+0000 7f6792fc5700 1 --2- 172.21.15.187:0/2243608087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c0b5750 0x7f678c09a8e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f678400b9a0 tx=0x7f6784006df0).stop 2020-11-20T12:09:20.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.339+0000 7f6792fc5700 1 -- 172.21.15.187:0/2243608087 shutdown_connections 2020-11-20T12:09:20.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.339+0000 7f6792fc5700 1 --2- 172.21.15.187:0/2243608087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c0b5750 0x7f678c09a8e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:20.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.339+0000 7f6792fc5700 1 --2- 172.21.15.187:0/2243608087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c066070 0x7f678c144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:20.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.339+0000 7f6792fc5700 1 -- 172.21.15.187:0/2243608087 >> 172.21.15.187:0/2243608087 conn(0x7f678c0b8280 msgr2=0x7f678c0653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:20.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.339+0000 7f6792fc5700 1 -- 172.21.15.187:0/2243608087 shutdown_connections 2020-11-20T12:09:20.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.339+0000 7f6792fc5700 1 -- 172.21.15.187:0/2243608087 wait complete. 2020-11-20T12:09:20.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.341+0000 7f6792fc5700 1 Processor -- start 2020-11-20T12:09:20.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.341+0000 7f6792fc5700 1 -- start start 2020-11-20T12:09:20.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f6792fc5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f678c065c90 0x7f678c095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:20.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f6792fc5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c066070 0x7f678c15b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:20.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f6792fc5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c0b5750 0x7f678c15bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:20.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f6792fc5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0666e0 con 0x7f678c065c90 2020-11-20T12:09:20.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f6792fc5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c096030 con 0x7f678c066070 2020-11-20T12:09:20.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f6792fc5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c066940 con 0x7f678c0b5750 2020-11-20T12:09:20.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f67917c2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c066070 0x7f678c15b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:20.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f67917c2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c066070 0x7f678c15b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44810/0 (socket says 172.21.15.187:44810) 2020-11-20T12:09:20.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f67917c2700 1 -- 172.21.15.187:0/1906644423 learned_addr learned my addr 172.21.15.187:0/1906644423 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:20.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f67917c2700 1 -- 172.21.15.187:0/1906644423 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c0b5750 msgr2=0x7f678c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:20.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f67917c2700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c0b5750 0x7f678c15bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:20.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f6791fc3700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f678c065c90 0x7f678c095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:20.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f67917c2700 1 -- 172.21.15.187:0/1906644423 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f678c065c90 msgr2=0x7f678c095a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:20.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f67917c2700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f678c065c90 0x7f678c095a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:20.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.342+0000 7f67917c2700 1 -- 172.21.15.187:0/1906644423 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c093c70 con 0x7f678c066070 2020-11-20T12:09:20.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.343+0000 7f67917c2700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c066070 0x7f678c15b760 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f677c00cb50 tx=0x7f677c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:20.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.343+0000 7f6779ffb700 1 -- 172.21.15.187:0/1906644423 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c0077d0 con 0x7f678c066070 2020-11-20T12:09:20.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.343+0000 7f6779ffb700 1 -- 172.21.15.187:0/1906644423 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677c007930 con 0x7f678c066070 2020-11-20T12:09:20.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.343+0000 7f6779ffb700 1 -- 172.21.15.187:0/1906644423 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c007cb0 con 0x7f678c066070 2020-11-20T12:09:20.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.343+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c15c140 con 0x7f678c066070 2020-11-20T12:09:20.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.343+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c067020 con 0x7f678c066070 2020-11-20T12:09:20.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.344+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c068ef0 con 0x7f678c066070 2020-11-20T12:09:20.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.345+0000 7f6779ffb700 1 -- 172.21.15.187:0/1906644423 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f677c011620 con 0x7f678c066070 2020-11-20T12:09:20.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.345+0000 7f6779ffb700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6764056d50 0x7f6764059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:20.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.345+0000 7f6779ffb700 1 -- 172.21.15.187:0/1906644423 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f677c070220 con 0x7f678c066070 2020-11-20T12:09:20.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.346+0000 7f6791fc3700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6764056d50 0x7f6764059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:20.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.346+0000 7f6791fc3700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6764056d50 0x7f6764059210 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6788001a60 tx=0x7f6788009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:20.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.349+0000 7f6779ffb700 1 -- 172.21.15.187:0/1906644423 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f677c0455c0 con 0x7f678c066070 2020-11-20T12:09:20.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.575+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f678c066b00 con 0x7f678c066070 2020-11-20T12:09:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.576+0000 7f6779ffb700 1 -- 172.21.15.187:0/1906644423 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+1423 (secure 0 0 0) 0x7f677c048be0 con 0x7f678c066070 2020-11-20T12:09:20.581 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:20.581 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:18.346613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.578+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6764056d50 msgr2=0x7f6764059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:20.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.578+0000 7f6792fc5700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6764056d50 0x7f6764059210 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6788001a60 tx=0x7f6788009040).stop 2020-11-20T12:09:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.578+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c066070 msgr2=0x7f678c15b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.578+0000 7f6792fc5700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c066070 0x7f678c15b760 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f677c00cb50 tx=0x7f677c008040).stop 2020-11-20T12:09:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.579+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 shutdown_connections 2020-11-20T12:09:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.579+0000 7f6792fc5700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6764056d50 0x7f6764059210 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.579+0000 7f6792fc5700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f678c065c90 0x7f678c095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.579+0000 7f6792fc5700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f678c0b5750 0x7f678c15bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.579+0000 7f6792fc5700 1 --2- 172.21.15.187:0/1906644423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f678c066070 0x7f678c15b760 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.579+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 >> 172.21.15.187:0/1906644423 conn(0x7f678c0b8280 msgr2=0x7f678c09c580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:20.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.579+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 shutdown_connections 2020-11-20T12:09:20.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:20.579+0000 7f6792fc5700 1 -- 172.21.15.187:0/1906644423 wait complete. 2020-11-20T12:09:20.586 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 35 2020-11-20T12:09:20.806 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:09:07.256865+0000', 'modified': '2020-11-20T12:09:18.346613+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 61, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:09:20.807 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2020-11-20T12:09:21.809 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:22.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.071+0000 7fc096b3d700 1 Processor -- start 2020-11-20T12:09:22.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.071+0000 7fc096b3d700 1 -- start start 2020-11-20T12:09:22.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.073+0000 7fc096b3d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc09014cc80 0x7fc090145980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:22.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.073+0000 7fc096b3d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09014d0b0 con 0x7fc090065c90 2020-11-20T12:09:22.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.073+0000 7fc096b3d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09014c2a0 con 0x7fc090066070 2020-11-20T12:09:22.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.073+0000 7fc096b3d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09014c540 con 0x7fc09014cc80 2020-11-20T12:09:22.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.073+0000 7fc09533a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc090066070 0x7fc0900b6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41424/0 (socket says 172.21.15.187:41424) 2020-11-20T12:09:22.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.073+0000 7fc09533a700 1 -- 172.21.15.187:0/1930980162 learned_addr learned my addr 172.21.15.187:0/1930980162 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:22.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.074+0000 7fc09633c700 1 --2- 172.21.15.187:0/1930980162 >> v2:172.21.15.5:3300/0 conn(0x7fc09014cc80 0x7fc090145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:22.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.074+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700057292 0 0) 0x7fc09014c2a0 con 0x7fc090066070 2020-11-20T12:09:22.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.074+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc078003240 con 0x7fc090066070 2020-11-20T12:09:22.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.074+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2011215356 0 0) 0x7fc078003240 con 0x7fc090066070 2020-11-20T12:09:22.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.074+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 >> v2:172.21.15.5:3300/0 conn(0x7fc09014cc80 msgr2=0x7fc090145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:22.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.074+0000 7fc094b39700 1 --2- 172.21.15.187:0/1930980162 >> v2:172.21.15.5:3300/0 conn(0x7fc09014cc80 0x7fc090145980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.074+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 >> v1:172.21.15.5:6789/0 conn(0x7fc090065c90 legacy=0x7fc0900b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:22.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09014c7e0 con 0x7fc090066070 2020-11-20T12:09:22.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fc0800014e0 con 0x7fc090066070 2020-11-20T12:09:22.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 >> v1:172.21.15.187:6789/0 conn(0x7fc090066070 legacy=0x7fc0900b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:22.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc078003730 0x7fc078003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:22.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0780044e0 0x7fc078004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:22.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc078004e50 0x7fc07800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:22.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09014c540 con 0x7fc078004e50 2020-11-20T12:09:22.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc078004260 con 0x7fc078003730 2020-11-20T12:09:22.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07800bc20 con 0x7fc0780044e0 2020-11-20T12:09:22.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.075+0000 7fc095b3b700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc078003730 0x7fc078003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:22.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc09633c700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc078004e50 0x7fc07800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:22.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc09533a700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0780044e0 0x7fc078004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:22.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc095b3b700 1 -- 172.21.15.187:0/1930980162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0780044e0 msgr2=0x7fc078004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:22.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc095b3b700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0780044e0 0x7fc078004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc095b3b700 1 -- 172.21.15.187:0/1930980162 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc078004e50 msgr2=0x7fc07800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:22.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc095b3b700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc078004e50 0x7fc07800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc095b3b700 1 -- 172.21.15.187:0/1930980162 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc078003240 con 0x7fc078003730 2020-11-20T12:09:22.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc09633c700 1 -- 172.21.15.187:0/1930980162 reap_dead start 2020-11-20T12:09:22.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc095b3b700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc078003730 0x7fc078003b50 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7fc08c001e70 tx=0x7fc08c007f10).ready entity=mon.1 client_cookie=99adb9e1d258ca82 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:22.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c001540 con 0x7fc078003730 2020-11-20T12:09:22.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c003030 con 0x7fc078003730 2020-11-20T12:09:22.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.076+0000 7fc094b39700 1 -- 172.21.15.187:0/1930980162 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c00cdf0 con 0x7fc078003730 2020-11-20T12:09:22.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.077+0000 7fc096b3d700 1 -- 172.21.15.187:0/1930980162 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc078003730 msgr2=0x7fc078003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:22.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.077+0000 7fc096b3d700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc078003730 0x7fc078003b50 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7fc08c001e70 tx=0x7fc08c007f10).stop 2020-11-20T12:09:22.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.077+0000 7fc096b3d700 1 -- 172.21.15.187:0/1930980162 shutdown_connections 2020-11-20T12:09:22.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.077+0000 7fc096b3d700 1 --2- 172.21.15.187:0/1930980162 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc078003730 0x7fc078003b50 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.077+0000 7fc096b3d700 1 -- 172.21.15.187:0/1930980162 >> 172.21.15.187:0/1930980162 conn(0x7fc0900b8280 msgr2=0x7fc0900653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:22.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.077+0000 7fc096b3d700 1 -- 172.21.15.187:0/1930980162 shutdown_connections 2020-11-20T12:09:22.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.077+0000 7fc096b3d700 1 -- 172.21.15.187:0/1930980162 wait complete. 2020-11-20T12:09:22.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.079+0000 7fc096b3d700 1 Processor -- start 2020-11-20T12:09:22.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.079+0000 7fc096b3d700 1 -- start start 2020-11-20T12:09:22.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc096b3d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc09009f4e0 0x7fc09015be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:22.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc096b3d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc09009c790 0x7fc09015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:22.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc096b3d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc09015b920 0x7fc09009e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:22.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc096b3d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09014ab90 con 0x7fc09009f4e0 2020-11-20T12:09:22.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc096b3d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09015c380 con 0x7fc09009c790 2020-11-20T12:09:22.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc096b3d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09009d120 con 0x7fc09015b920 2020-11-20T12:09:22.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc09533a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc09009c790 0x7fc09015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:22.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc09533a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc09009c790 0x7fc09015b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44830/0 (socket says 172.21.15.187:44830) 2020-11-20T12:09:22.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc09533a700 1 -- 172.21.15.187:0/1735470177 learned_addr learned my addr 172.21.15.187:0/1735470177 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:22.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.080+0000 7fc095b3b700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc09009f4e0 0x7fc09015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:22.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc09633c700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc09015b920 0x7fc09009e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:22.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc09533a700 1 -- 172.21.15.187:0/1735470177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc09015b920 msgr2=0x7fc09009e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:22.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc09533a700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc09015b920 0x7fc09009e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc09533a700 1 -- 172.21.15.187:0/1735470177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc09009f4e0 msgr2=0x7fc09015be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:22.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc09533a700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc09009f4e0 0x7fc09015be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc09533a700 1 -- 172.21.15.187:0/1735470177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09009d3c0 con 0x7fc09009c790 2020-11-20T12:09:22.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc09533a700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc09009c790 0x7fc09015b430 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7fc08000eae0 tx=0x7fc090067dd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:22.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc07dffb700 1 -- 172.21.15.187:0/1735470177 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08000c900 con 0x7fc09009c790 2020-11-20T12:09:22.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc07dffb700 1 -- 172.21.15.187:0/1735470177 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08000ca60 con 0x7fc09009c790 2020-11-20T12:09:22.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09009d660 con 0x7fc09009c790 2020-11-20T12:09:22.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.081+0000 7fc07dffb700 1 -- 172.21.15.187:0/1735470177 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08000ce00 con 0x7fc09009c790 2020-11-20T12:09:22.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.082+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09009fdf0 con 0x7fc09009c790 2020-11-20T12:09:22.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.083+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc090068ef0 con 0x7fc09009c790 2020-11-20T12:09:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.083+0000 7fc07dffb700 1 -- 172.21.15.187:0/1735470177 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc080014640 con 0x7fc09009c790 2020-11-20T12:09:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.084+0000 7fc07dffb700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc068056d50 0x7fc068059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.084+0000 7fc07dffb700 1 -- 172.21.15.187:0/1735470177 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc08006fc30 con 0x7fc09009c790 2020-11-20T12:09:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.084+0000 7fc095b3b700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc068056d50 0x7fc068059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:22.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.085+0000 7fc095b3b700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc068056d50 0x7fc068059210 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fc09005d960 tx=0x7fc08c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:22.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.088+0000 7fc07dffb700 1 -- 172.21.15.187:0/1735470177 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc080044fd0 con 0x7fc09009c790 2020-11-20T12:09:22.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.333+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc090044f60 con 0x7fc09009c790 2020-11-20T12:09:22.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.334+0000 7fc07dffb700 1 -- 172.21.15.187:0/1735470177 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+1423 (secure 0 0 0) 0x7fc0800485f0 con 0x7fc09009c790 2020-11-20T12:09:22.338 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:22.339 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:18.346613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:22.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc068056d50 msgr2=0x7fc068059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:22.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc068056d50 0x7fc068059210 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fc09005d960 tx=0x7fc08c015000).stop 2020-11-20T12:09:22.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc09009c790 msgr2=0x7fc09015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:22.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc09009c790 0x7fc09015b430 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7fc08000eae0 tx=0x7fc090067dd0).stop 2020-11-20T12:09:22.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 shutdown_connections 2020-11-20T12:09:22.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc068056d50 0x7fc068059210 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc09009f4e0 0x7fc09015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc09015b920 0x7fc09009e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 --2- 172.21.15.187:0/1735470177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc09009c790 0x7fc09015b430 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:22.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 >> 172.21.15.187:0/1735470177 conn(0x7fc0900b8280 msgr2=0x7fc0900afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:22.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 shutdown_connections 2020-11-20T12:09:22.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:22.337+0000 7fc096b3d700 1 -- 172.21.15.187:0/1735470177 wait complete. 2020-11-20T12:09:22.344 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 35 2020-11-20T12:09:22.568 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:09:07.256865+0000', 'modified': '2020-11-20T12:09:18.346613+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 61, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:09:22.569 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2020-11-20T12:09:23.570 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:23.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.816+0000 7f531da27700 1 Processor -- start 2020-11-20T12:09:23.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.816+0000 7f531da27700 1 -- start start 2020-11-20T12:09:23.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.818+0000 7f531da27700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5318065c90 0x7f531814cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:23.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.818+0000 7f531da27700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5318066070 0x7f53180b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:23.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.818+0000 7f531da27700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531814c2a0 con 0x7f53180b6dc0 2020-11-20T12:09:23.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.818+0000 7f531da27700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531814c540 con 0x7f5318065c90 2020-11-20T12:09:23.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.818+0000 7f531da27700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531814c7e0 con 0x7f5318066070 2020-11-20T12:09:23.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.818+0000 7f5317fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5318066070 0x7f53180b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:23.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.818+0000 7f5317fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5318066070 0x7f53180b5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44850/0 (socket says 172.21.15.187:44850) 2020-11-20T12:09:23.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.818+0000 7f5317fff700 1 -- 172.21.15.187:0/3765698625 learned_addr learned my addr 172.21.15.187:0/3765698625 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:23.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.819+0000 7f5317fff700 1 -- 172.21.15.187:0/3765698625 >> v2:172.21.15.5:3300/0 conn(0x7f5318065c90 msgr2=0x7f531814cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:23.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.819+0000 7f5317fff700 1 --2- 172.21.15.187:0/3765698625 >> v2:172.21.15.5:3300/0 conn(0x7f5318065c90 0x7f531814cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:23.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.819+0000 7f5317fff700 1 -- 172.21.15.187:0/3765698625 >> v1:172.21.15.5:6789/0 conn(0x7f53180b6dc0 legacy=0x7f5318145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:23.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.819+0000 7f5317fff700 1 -- 172.21.15.187:0/3765698625 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53180b5800 con 0x7f5318066070 2020-11-20T12:09:23.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.819+0000 7f5317fff700 1 --2- 172.21.15.187:0/3765698625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f53180b5310 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f5318064cd0 tx=0x7f530c006eb0).ready entity=mon.1 client_cookie=cd210cb1350fa110 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:23.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.819+0000 7f53177fe700 1 -- 172.21.15.187:0/3765698625 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c009890 con 0x7f5318066070 2020-11-20T12:09:23.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.819+0000 7f53177fe700 1 -- 172.21.15.187:0/3765698625 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c0099f0 con 0x7f5318066070 2020-11-20T12:09:23.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.819+0000 7f53177fe700 1 -- 172.21.15.187:0/3765698625 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c009d70 con 0x7f5318066070 2020-11-20T12:09:23.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.820+0000 7f531da27700 1 -- 172.21.15.187:0/3765698625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 msgr2=0x7f53180b5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:23.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.820+0000 7f531da27700 1 --2- 172.21.15.187:0/3765698625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f53180b5310 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f5318064cd0 tx=0x7f530c006eb0).stop 2020-11-20T12:09:23.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.820+0000 7f531da27700 1 -- 172.21.15.187:0/3765698625 shutdown_connections 2020-11-20T12:09:23.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.820+0000 7f531da27700 1 --2- 172.21.15.187:0/3765698625 >> v2:172.21.15.5:3300/0 conn(0x7f5318065c90 0x7f531814cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:23.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.820+0000 7f531da27700 1 --2- 172.21.15.187:0/3765698625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f53180b5310 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:23.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.820+0000 7f531da27700 1 -- 172.21.15.187:0/3765698625 >> 172.21.15.187:0/3765698625 conn(0x7f53180b8280 msgr2=0x7f53180653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:23.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.820+0000 7f531da27700 1 -- 172.21.15.187:0/3765698625 shutdown_connections 2020-11-20T12:09:23.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.820+0000 7f531da27700 1 -- 172.21.15.187:0/3765698625 wait complete. 2020-11-20T12:09:23.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.821+0000 7f531da27700 1 Processor -- start 2020-11-20T12:09:23.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.822+0000 7f531da27700 1 -- start start 2020-11-20T12:09:23.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.822+0000 7f531da27700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5318065c90 0x7f53180b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:23.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.822+0000 7f531da27700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f5318158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:23.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.822+0000 7f531da27700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53180b6dc0 0x7f531809f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:23.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f531da27700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180b3b90 con 0x7f53180b6dc0 2020-11-20T12:09:23.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f531da27700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180b3e30 con 0x7f5318066070 2020-11-20T12:09:23.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f531da27700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53180b40d0 con 0x7f5318065c90 2020-11-20T12:09:23.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f5317fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f5318158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:23.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f5317fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f5318158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44852/0 (socket says 172.21.15.187:44852) 2020-11-20T12:09:23.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f5317fff700 1 -- 172.21.15.187:0/3887154395 learned_addr learned my addr 172.21.15.187:0/3887154395 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:23.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f5317fff700 1 -- 172.21.15.187:0/3887154395 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5318065c90 msgr2=0x7f53180b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:23.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f5317fff700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5318065c90 0x7f53180b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:23.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f5317fff700 1 -- 172.21.15.187:0/3887154395 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53180b6dc0 msgr2=0x7f531809f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:23.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f5317fff700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53180b6dc0 0x7f531809f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:23.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.823+0000 7f5317fff700 1 -- 172.21.15.187:0/3887154395 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531809fc80 con 0x7f5318066070 2020-11-20T12:09:23.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.824+0000 7f5317fff700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f5318158b80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f530c006fd0 tx=0x7f530c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:23.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.824+0000 7f5314ff9700 1 -- 172.21.15.187:0/3887154395 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c01e460 con 0x7f5318066070 2020-11-20T12:09:23.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.824+0000 7f5314ff9700 1 -- 172.21.15.187:0/3887154395 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c01e5c0 con 0x7f5318066070 2020-11-20T12:09:23.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.824+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53180b3810 con 0x7f5318066070 2020-11-20T12:09:23.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.824+0000 7f5314ff9700 1 -- 172.21.15.187:0/3887154395 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c01e960 con 0x7f5318066070 2020-11-20T12:09:23.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.824+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531814a090 con 0x7f5318066070 2020-11-20T12:09:23.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.825+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53180b8b90 con 0x7f5318066070 2020-11-20T12:09:23.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.826+0000 7f5314ff9700 1 -- 172.21.15.187:0/3887154395 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f530c025030 con 0x7f5318066070 2020-11-20T12:09:23.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.826+0000 7f5314ff9700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f52f0056d50 0x7f52f0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:23.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.826+0000 7f5314ff9700 1 -- 172.21.15.187:0/3887154395 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f530c0717d0 con 0x7f5318066070 2020-11-20T12:09:23.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.827+0000 7f531ca25700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f52f0056d50 0x7f52f0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:23.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.827+0000 7f531ca25700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f52f0056d50 0x7f52f0059210 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f5308009540 tx=0x7f5308006a80).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:23.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:23.830+0000 7f5314ff9700 1 -- 172.21.15.187:0/3887154395 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f530c046b70 con 0x7f5318066070 2020-11-20T12:09:24.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.077+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5318068ef0 con 0x7f5318066070 2020-11-20T12:09:24.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.078+0000 7f5314ff9700 1 -- 172.21.15.187:0/3887154395 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f530c023070 con 0x7f5318066070 2020-11-20T12:09:24.083 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:24.084 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:23.388961+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:replay","state_seq":2,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:24.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f52f0056d50 msgr2=0x7f52f0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:24.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f52f0056d50 0x7f52f0059210 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f5308009540 tx=0x7f5308006a80).stop 2020-11-20T12:09:24.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 msgr2=0x7f5318158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:24.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f5318158b80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f530c006fd0 tx=0x7f530c015000).stop 2020-11-20T12:09:24.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 shutdown_connections 2020-11-20T12:09:24.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f52f0056d50 0x7f52f0059210 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53180b6dc0 0x7f531809f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5318065c90 0x7f53180b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 --2- 172.21.15.187:0/3887154395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5318066070 0x7f5318158b80 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 >> 172.21.15.187:0/3887154395 conn(0x7f53180b8280 msgr2=0x7f531809dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:24.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.081+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 shutdown_connections 2020-11-20T12:09:24.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:24.082+0000 7f531da27700 1 -- 172.21.15.187:0/3887154395 wait complete. 2020-11-20T12:09:24.089 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 37 2020-11-20T12:09:24.312 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:09:07.256865+0000', 'modified': '2020-11-20T12:09:23.388961+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 61, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6178': {'gid': 6178, 'name': 'a', 'rank': 0, 'incarnation': 37, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.187:6835/609118159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 609118159}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 609118159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:09:24.313 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6178:up:replay 2020-11-20T12:09:25.316 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:25.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.902+0000 7f26b88cc700 1 Processor -- start 2020-11-20T12:09:25.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.902+0000 7f26b88cc700 1 -- start start 2020-11-20T12:09:25.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.904+0000 7f26b88cc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b40b00b0 0x7f26b413e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:25.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.904+0000 7f26b88cc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f26b413ee00 0x7f26b40b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:25.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.904+0000 7f26b88cc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b4148770 con 0x7f26b4148f90 2020-11-20T12:09:25.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.904+0000 7f26b88cc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b40b5490 con 0x7f26b413ee00 2020-11-20T12:09:25.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.904+0000 7f26b88cc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b409a3f0 con 0x7f26b40b00b0 2020-11-20T12:09:25.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.904+0000 7f26b37fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f26b4148f90 0x7f26b4099ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:34978/0 (socket says 172.21.15.187:34978) 2020-11-20T12:09:25.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.904+0000 7f26b37fe700 1 -- 172.21.15.187:0/2694939823 learned_addr learned my addr 172.21.15.187:0/2694939823 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:25.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.904+0000 7f26aaffd700 1 --2- 172.21.15.187:0/2694939823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b40b00b0 0x7f26b413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:25.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.905+0000 7f26b3fff700 1 --2- 172.21.15.187:0/2694939823 >> v2:172.21.15.5:3300/0 conn(0x7f26b413ee00 0x7f26b40b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:25.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.905+0000 7f26b2ffd700 1 -- 172.21.15.187:0/2694939823 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98780699 0 0) 0x7f26b4148770 con 0x7f26b4148f90 2020-11-20T12:09:25.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.906+0000 7f26b2ffd700 1 -- 172.21.15.187:0/2694939823 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2698003240 con 0x7f26b4148f90 2020-11-20T12:09:25.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.906+0000 7f26aaffd700 1 -- 172.21.15.187:0/2694939823 >> v2:172.21.15.5:3300/0 conn(0x7f26b413ee00 msgr2=0x7f26b40b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:25.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.906+0000 7f26aaffd700 1 --2- 172.21.15.187:0/2694939823 >> v2:172.21.15.5:3300/0 conn(0x7f26b413ee00 0x7f26b40b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:25.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.906+0000 7f26aaffd700 1 -- 172.21.15.187:0/2694939823 >> v1:172.21.15.5:6789/0 conn(0x7f26b4148f90 legacy=0x7f26b4099ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:25.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.906+0000 7f26aaffd700 1 -- 172.21.15.187:0/2694939823 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b4066750 con 0x7f26b40b00b0 2020-11-20T12:09:25.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.906+0000 7f26aaffd700 1 --2- 172.21.15.187:0/2694939823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b40b00b0 0x7f26b413e910 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f269c00b9a0 tx=0x7f269c006df0).ready entity=mon.2 client_cookie=bd1c0b6d03f6e3ad server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:25.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.907+0000 7f26b2ffd700 1 -- 172.21.15.187:0/2694939823 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c009680 con 0x7f26b40b00b0 2020-11-20T12:09:25.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.907+0000 7f26b2ffd700 1 -- 172.21.15.187:0/2694939823 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c0097e0 con 0x7f26b40b00b0 2020-11-20T12:09:25.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.907+0000 7f26b2ffd700 1 -- 172.21.15.187:0/2694939823 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c009b60 con 0x7f26b40b00b0 2020-11-20T12:09:25.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.908+0000 7f26b88cc700 1 -- 172.21.15.187:0/2694939823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b40b00b0 msgr2=0x7f26b413e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:25.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.908+0000 7f26b88cc700 1 --2- 172.21.15.187:0/2694939823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b40b00b0 0x7f26b413e910 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f269c00b9a0 tx=0x7f269c006df0).stop 2020-11-20T12:09:25.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.908+0000 7f26b88cc700 1 -- 172.21.15.187:0/2694939823 shutdown_connections 2020-11-20T12:09:25.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.908+0000 7f26b88cc700 1 --2- 172.21.15.187:0/2694939823 >> v2:172.21.15.5:3300/0 conn(0x7f26b413ee00 0x7f26b40b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:25.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.908+0000 7f26b88cc700 1 --2- 172.21.15.187:0/2694939823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b40b00b0 0x7f26b413e910 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:25.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.908+0000 7f26b88cc700 1 -- 172.21.15.187:0/2694939823 >> 172.21.15.187:0/2694939823 conn(0x7f26b40b8280 msgr2=0x7f26b40653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:25.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.908+0000 7f26b88cc700 1 -- 172.21.15.187:0/2694939823 shutdown_connections 2020-11-20T12:09:25.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.908+0000 7f26b88cc700 1 -- 172.21.15.187:0/2694939823 wait complete. 2020-11-20T12:09:25.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.910+0000 7f26b88cc700 1 Processor -- start 2020-11-20T12:09:25.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.910+0000 7f26b88cc700 1 -- start start 2020-11-20T12:09:25.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.911+0000 7f26b88cc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26b40b00b0 0x7f26b409e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:25.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.911+0000 7f26b88cc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26b413ee00 0x7f26b4143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:25.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.911+0000 7f26b88cc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b4148f90 0x7f26b409f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:25.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.911+0000 7f26b88cc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4145aa0 con 0x7f26b40b00b0 2020-11-20T12:09:25.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.911+0000 7f26b88cc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415c150 con 0x7f26b413ee00 2020-11-20T12:09:25.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.911+0000 7f26b88cc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415c380 con 0x7f26b4148f90 2020-11-20T12:09:25.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26aaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26b413ee00 0x7f26b4143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:25.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26aaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26b413ee00 0x7f26b4143bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44870/0 (socket says 172.21.15.187:44870) 2020-11-20T12:09:25.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26aaffd700 1 -- 172.21.15.187:0/211620450 learned_addr learned my addr 172.21.15.187:0/211620450 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:25.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26b37fe700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26b40b00b0 0x7f26b409e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:25.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26b3fff700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b4148f90 0x7f26b409f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:25.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26aaffd700 1 -- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b4148f90 msgr2=0x7f26b409f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:25.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26aaffd700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b4148f90 0x7f26b409f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:25.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26aaffd700 1 -- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26b40b00b0 msgr2=0x7f26b409e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:25.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26aaffd700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26b40b00b0 0x7f26b409e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:25.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26aaffd700 1 -- 172.21.15.187:0/211620450 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b409fcc0 con 0x7f26b413ee00 2020-11-20T12:09:25.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.912+0000 7f26b3fff700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b4148f90 0x7f26b409f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:25.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.913+0000 7f26aaffd700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26b413ee00 0x7f26b4143bb0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f269c010000 tx=0x7f269c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:25.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.913+0000 7f26abfff700 1 -- 172.21.15.187:0/211620450 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c011d70 con 0x7f26b413ee00 2020-11-20T12:09:25.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.913+0000 7f26abfff700 1 -- 172.21.15.187:0/211620450 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c01e460 con 0x7f26b413ee00 2020-11-20T12:09:25.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.913+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b415b8f0 con 0x7f26b413ee00 2020-11-20T12:09:25.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.913+0000 7f26abfff700 1 -- 172.21.15.187:0/211620450 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c01e7e0 con 0x7f26b413ee00 2020-11-20T12:09:25.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.913+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b415bd30 con 0x7f26b413ee00 2020-11-20T12:09:25.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.914+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b4068ef0 con 0x7f26b413ee00 2020-11-20T12:09:25.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.915+0000 7f26abfff700 1 -- 172.21.15.187:0/211620450 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f269c025030 con 0x7f26b413ee00 2020-11-20T12:09:25.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.915+0000 7f26abfff700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2684056d20 0x7f26840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:25.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.915+0000 7f26abfff700 1 -- 172.21.15.187:0/211620450 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f269c071740 con 0x7f26b413ee00 2020-11-20T12:09:25.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.916+0000 7f26b37fe700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2684056d20 0x7f26840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:25.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.916+0000 7f26b37fe700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2684056d20 0x7f26840591e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f26a40055a0 tx=0x7f26a4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:25.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:25.921+0000 7f26abfff700 1 -- 172.21.15.187:0/211620450 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f269c046bf0 con 0x7f26b413ee00 2020-11-20T12:09:26.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.174+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26b4044f60 con 0x7f26b413ee00 2020-11-20T12:09:26.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.175+0000 7f26abfff700 1 -- 172.21.15.187:0/211620450 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f269c023070 con 0x7f26b413ee00 2020-11-20T12:09:26.180 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:26.180 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:25.404219+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:26.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2684056d20 msgr2=0x7f26840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:26.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2684056d20 0x7f26840591e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f26a40055a0 tx=0x7f26a4009040).stop 2020-11-20T12:09:26.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26b413ee00 msgr2=0x7f26b4143bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:26.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26b413ee00 0x7f26b4143bb0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f269c010000 tx=0x7f269c015000).stop 2020-11-20T12:09:26.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 shutdown_connections 2020-11-20T12:09:26.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2684056d20 0x7f26840591e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:26.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26b40b00b0 0x7f26b409e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:26.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f26b4148f90 0x7f26b409f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:26.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 --2- 172.21.15.187:0/211620450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26b413ee00 0x7f26b4143bb0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:26.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 >> 172.21.15.187:0/211620450 conn(0x7f26b40b8280 msgr2=0x7f26b40b8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:26.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 shutdown_connections 2020-11-20T12:09:26.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:26.178+0000 7f26b88cc700 1 -- 172.21.15.187:0/211620450 wait complete. 2020-11-20T12:09:26.185 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 39 2020-11-20T12:09:26.406 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:09:07.256865+0000', 'modified': '2020-11-20T12:09:25.404219+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 61, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6178': {'gid': 6178, 'name': 'a', 'rank': 0, 'incarnation': 37, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.187:6835/609118159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 609118159}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 609118159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:09:26.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6178:up:rejoin 2020-11-20T12:09:27.409 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:27.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.654+0000 7f1e0cb72700 1 Processor -- start 2020-11-20T12:09:27.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.654+0000 7f1e0cb72700 1 -- start start 2020-11-20T12:09:27.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.656+0000 7f1e0cb72700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e080bc460 0x7f1e0814c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.656+0000 7f1e0cb72700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1e080b96f0 0x7f1e0809ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.656+0000 7f1e0cb72700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0814cc10 con 0x7f1e080bc840 2020-11-20T12:09:27.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.656+0000 7f1e0cb72700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0814c9f0 con 0x7f1e080b96f0 2020-11-20T12:09:27.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.656+0000 7f1e0cb72700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0814ceb0 con 0x7f1e080bc460 2020-11-20T12:09:27.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.657+0000 7f1dfffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1e080bc840 0x7f1e081485e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41486/0 (socket says 172.21.15.187:41486) 2020-11-20T12:09:27.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.657+0000 7f1dfffff700 1 -- 172.21.15.187:0/2064753435 learned_addr learned my addr 172.21.15.187:0/2064753435 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:27.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.657+0000 7f1e077fe700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e080bc460 0x7f1e0814c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:27.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.657+0000 7f1e07fff700 1 --2- 172.21.15.187:0/2064753435 >> v2:172.21.15.5:3300/0 conn(0x7f1e080b96f0 0x7f1e0809ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:27.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.657+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236549645 0 0) 0x7f1e0814cc10 con 0x7f1e080bc840 2020-11-20T12:09:27.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.657+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1de8003240 con 0x7f1e080bc840 2020-11-20T12:09:27.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3035366080 0 0) 0x7f1de8003240 con 0x7f1e080bc840 2020-11-20T12:09:27.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e080bc460 msgr2=0x7f1e0814c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:27.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e080bc460 0x7f1e0814c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:27.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 >> v2:172.21.15.5:3300/0 conn(0x7f1e080b96f0 msgr2=0x7f1e0809ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:27.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 --2- 172.21.15.187:0/2064753435 >> v2:172.21.15.5:3300/0 conn(0x7f1e080b96f0 0x7f1e0809ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:27.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0814d150 con 0x7f1e080bc840 2020-11-20T12:09:27.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1df00014e0 con 0x7f1e080bc840 2020-11-20T12:09:27.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 >> v1:172.21.15.187:6789/0 conn(0x7f1e080bc840 legacy=0x7f1e081485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:27.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e080bc460 0x7f1de8003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1de8003c40 0x7f1de80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1de80049d0 0x7f1de8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0814c9f0 con 0x7f1de80049d0 2020-11-20T12:09:27.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0814ceb0 con 0x7f1de8003c40 2020-11-20T12:09:27.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8004200 con 0x7f1e080bc460 2020-11-20T12:09:27.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.658+0000 7f1dfffff700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1de8003c40 0x7f1de80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:27.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1e077fe700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e080bc460 0x7f1de8003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:27.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1dfffff700 1 -- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e080bc460 msgr2=0x7f1de8003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:27.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1dfffff700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e080bc460 0x7f1de8003730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:27.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1dfffff700 1 -- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1de80049d0 msgr2=0x7f1de8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:27.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1e07fff700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1de80049d0 0x7f1de8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:27.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1dfffff700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1de80049d0 0x7f1de8004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:27.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1dfffff700 1 -- 172.21.15.187:0/2064753435 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8003240 con 0x7f1de8003c40 2020-11-20T12:09:27.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1dfffff700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1de8003c40 0x7f1de80044e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f1e080bb4c0 tx=0x7f1e0814d2e0).ready entity=mon.1 client_cookie=579c8723ec67dcdf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:27.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df00072e0 con 0x7f1de8003c40 2020-11-20T12:09:27.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df0007440 con 0x7f1de8003c40 2020-11-20T12:09:27.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.659+0000 7f1e06ffd700 1 -- 172.21.15.187:0/2064753435 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0005820 con 0x7f1de8003c40 2020-11-20T12:09:27.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.660+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2064753435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1de8003c40 msgr2=0x7f1de80044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:27.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.660+0000 7f1e0cb72700 1 --2- 172.21.15.187:0/2064753435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1de8003c40 0x7f1de80044e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f1e080bb4c0 tx=0x7f1e0814d2e0).stop 2020-11-20T12:09:27.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.660+0000 7f1e07fff700 1 -- 172.21.15.187:0/2064753435 reap_dead start 2020-11-20T12:09:27.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.660+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2064753435 shutdown_connections 2020-11-20T12:09:27.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.660+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2064753435 >> 172.21.15.187:0/2064753435 conn(0x7f1e080bfb90 msgr2=0x7f1e080bbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:27.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.660+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2064753435 shutdown_connections 2020-11-20T12:09:27.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.660+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2064753435 wait complete. 2020-11-20T12:09:27.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.662+0000 7f1e0cb72700 1 Processor -- start 2020-11-20T12:09:27.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.662+0000 7f1e0cb72700 1 -- start start 2020-11-20T12:09:27.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1e0cb72700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e0815bd50 0x7f1e0815c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1e0cb72700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0813b5f0 0x7f1e0813ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1e0cb72700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e08118ac0 0x7f1e08118f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1e0cb72700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080a30b0 con 0x7f1e0815bd50 2020-11-20T12:09:27.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1e0cb72700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080a3350 con 0x7f1e0813b5f0 2020-11-20T12:09:27.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1e0cb72700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080a35f0 con 0x7f1e08118ac0 2020-11-20T12:09:27.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1dfffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0813b5f0 0x7f1e0813ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:27.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1dfffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0813b5f0 0x7f1e0813ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44892/0 (socket says 172.21.15.187:44892) 2020-11-20T12:09:27.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.663+0000 7f1dfffff700 1 -- 172.21.15.187:0/2089168019 learned_addr learned my addr 172.21.15.187:0/2089168019 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:27.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dfffff700 1 -- 172.21.15.187:0/2089168019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e08118ac0 msgr2=0x7f1e08118f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:27.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dfffff700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e08118ac0 0x7f1e08118f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:27.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dfffff700 1 -- 172.21.15.187:0/2089168019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e0815bd50 msgr2=0x7f1e0815c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:27.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dfffff700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e0815bd50 0x7f1e0815c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:27.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dfffff700 1 -- 172.21.15.187:0/2089168019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0813bf60 con 0x7f1e0813b5f0 2020-11-20T12:09:27.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dfffff700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0813b5f0 0x7f1e0813ba70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f1e0814c460 tx=0x7f1df0013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:27.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dff7fe700 1 -- 172.21.15.187:0/2089168019 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df001a070 con 0x7f1e0813b5f0 2020-11-20T12:09:27.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dff7fe700 1 -- 172.21.15.187:0/2089168019 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df0013660 con 0x7f1e0813b5f0 2020-11-20T12:09:27.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1dff7fe700 1 -- 172.21.15.187:0/2089168019 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df00072e0 con 0x7f1e0813b5f0 2020-11-20T12:09:27.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e080a2d30 con 0x7f1e0813b5f0 2020-11-20T12:09:27.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.664+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0815c620 con 0x7f1e0813b5f0 2020-11-20T12:09:27.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.665+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e08068630 con 0x7f1e0813b5f0 2020-11-20T12:09:27.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.666+0000 7f1dff7fe700 1 -- 172.21.15.187:0/2089168019 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1df0026070 con 0x7f1e0813b5f0 2020-11-20T12:09:27.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.666+0000 7f1dff7fe700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1de4056d20 0x7f1de40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:27.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.667+0000 7f1dff7fe700 1 -- 172.21.15.187:0/2089168019 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1df0070be0 con 0x7f1e0813b5f0 2020-11-20T12:09:27.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.667+0000 7f1e077fe700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1de4056d20 0x7f1de40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:27.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.667+0000 7f1e077fe700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1de4056d20 0x7f1de40591e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f1df8006fd0 tx=0x7f1df8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:27.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.670+0000 7f1dff7fe700 1 -- 172.21.15.187:0/2089168019 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1df0045fe0 con 0x7f1e0813b5f0 2020-11-20T12:09:27.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.913+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e08080210 con 0x7f1e0813b5f0 2020-11-20T12:09:27.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.914+0000 7f1dff7fe700 1 -- 172.21.15.187:0/2089168019 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f1df0049600 con 0x7f1e0813b5f0 2020-11-20T12:09:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:27.919 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:27.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.916+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1de4056d20 msgr2=0x7f1de40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:27.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.916+0000 7f1e0cb72700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1de4056d20 0x7f1de40591e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f1df8006fd0 tx=0x7f1df8008040).stop 2020-11-20T12:09:27.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.916+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0813b5f0 msgr2=0x7f1e0813ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:27.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.916+0000 7f1e0cb72700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0813b5f0 0x7f1e0813ba70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f1e0814c460 tx=0x7f1df0013000).stop 2020-11-20T12:09:27.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.917+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 shutdown_connections 2020-11-20T12:09:27.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.917+0000 7f1e0cb72700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1de4056d20 0x7f1de40591e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:27.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.917+0000 7f1e0cb72700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e0815bd50 0x7f1e0815c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:27.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.917+0000 7f1e0cb72700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e08118ac0 0x7f1e08118f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:27.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.917+0000 7f1e0cb72700 1 --2- 172.21.15.187:0/2089168019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0813b5f0 0x7f1e0813ba70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:27.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.917+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 >> 172.21.15.187:0/2089168019 conn(0x7f1e080bfb90 msgr2=0x7f1e08148260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:27.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.917+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 shutdown_connections 2020-11-20T12:09:27.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:27.917+0000 7f1e0cb72700 1 -- 172.21.15.187:0/2089168019 wait complete. 2020-11-20T12:09:27.925 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:28.144 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:09:07.256865+0000', 'modified': '2020-11-20T12:09:27.418850+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 61, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6178': {'gid': 6178, 'name': 'a', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.187:6835/609118159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 609118159}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 609118159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:09:28.145 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:09:28.145 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:09:28.406 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:09:28.402+0000 7f4c3512f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:09:28.430 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6178,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":5.004228511,"mdsmap_epoch":41,"osdmap_epoch":61,"osdmap_epoch_barrier":61,"uptime":9.8357324140000006} 2020-11-20T12:09:28.430 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6178, "mdsmap_epoch": 41, "osdmap_epoch": 61, "osdmap_epoch_barrier": 61, "rank_uptime": 5.004228511, "state": "up:active", "uptime": 9.835732414, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:09:28.431 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:09:28.431 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:09:28.432 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:09:28.432 INFO:teuthology.orchestra.run.smithi187:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2020-11-20T12:09:28.433 INFO:teuthology.orchestra.run.smithi187:> subdir = "delete_me" 2020-11-20T12:09:28.433 INFO:teuthology.orchestra.run.smithi187:> size_unit = 1024 2020-11-20T12:09:28.434 INFO:teuthology.orchestra.run.smithi187:> file_multiplier = 200 2020-11-20T12:09:28.434 INFO:teuthology.orchestra.run.smithi187:> os.mkdir(os.path.join(mountpoint, subdir)) 2020-11-20T12:09:28.434 INFO:teuthology.orchestra.run.smithi187:> for i in range(0, file_multiplier): 2020-11-20T12:09:28.435 INFO:teuthology.orchestra.run.smithi187:> for size in range(0, 16*size_unit, size_unit): 2020-11-20T12:09:28.435 INFO:teuthology.orchestra.run.smithi187:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2020-11-20T12:09:28.435 INFO:teuthology.orchestra.run.smithi187:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2020-11-20T12:09:28.436 INFO:teuthology.orchestra.run.smithi187:> f.write(size * '"'"'x'"'"') 2020-11-20T12:09:28.436 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:09:38.229 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_ops 2020-11-20T12:09:38.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:09:38.231 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -rf delete_me) 2020-11-20T12:09:38.524 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_max_purge_ops":"100000000"} 2020-11-20T12:09:38.525 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2020-11-20T12:09:38.526 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_max_purge_files 2020-11-20T12:09:38.788 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_max_purge_files":"6"} 2020-11-20T12:09:38.788 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2020-11-20T12:09:38.789 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:39.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.014+0000 7eff23fff700 1 Processor -- start 2020-11-20T12:09:39.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.014+0000 7eff23fff700 1 -- start start 2020-11-20T12:09:39.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.015+0000 7eff23fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7eff240bc840 0x7eff2414c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:39.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.015+0000 7eff23fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7eff240b96f0 0x7eff2409ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:39.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.015+0000 7eff23fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2414cc10 con 0x7eff240bc460 2020-11-20T12:09:39.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.015+0000 7eff23fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2414c9f0 con 0x7eff240bc840 2020-11-20T12:09:39.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.015+0000 7eff23fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2414ceb0 con 0x7eff240b96f0 2020-11-20T12:09:39.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.016+0000 7eff227fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7eff240bc840 0x7eff2414c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:39.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.016+0000 7eff237fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7eff240b96f0 0x7eff2409ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:39.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.016+0000 7eff22ffd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7eff240bc460 0x7eff241485e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:35026/0 (socket says 172.21.15.187:35026) 2020-11-20T12:09:39.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.016+0000 7eff22ffd700 1 -- 172.21.15.187:0/917158936 learned_addr learned my addr 172.21.15.187:0/917158936 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:39.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.017+0000 7eff237fe700 1 -- 172.21.15.187:0/917158936 >> v2:172.21.15.5:3300/0 conn(0x7eff240bc840 msgr2=0x7eff2414c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:39.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.017+0000 7eff237fe700 1 --2- 172.21.15.187:0/917158936 >> v2:172.21.15.5:3300/0 conn(0x7eff240bc840 0x7eff2414c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.017+0000 7eff237fe700 1 -- 172.21.15.187:0/917158936 >> v1:172.21.15.5:6789/0 conn(0x7eff240bc460 legacy=0x7eff241485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:39.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.017+0000 7eff237fe700 1 -- 172.21.15.187:0/917158936 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2414d150 con 0x7eff240b96f0 2020-11-20T12:09:39.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.017+0000 7eff237fe700 1 --2- 172.21.15.187:0/917158936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff240b96f0 0x7eff2409ff90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7eff240bb9c0 tx=0x7eff1c0091d0).ready entity=mon.1 client_cookie=5d5277a4ad116b05 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:39.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.018+0000 7eff21ffb700 1 -- 172.21.15.187:0/917158936 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c00bad0 con 0x7eff240b96f0 2020-11-20T12:09:39.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.018+0000 7eff21ffb700 1 -- 172.21.15.187:0/917158936 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c00bc30 con 0x7eff240b96f0 2020-11-20T12:09:39.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.018+0000 7eff21ffb700 1 -- 172.21.15.187:0/917158936 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c011400 con 0x7eff240b96f0 2020-11-20T12:09:39.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.019+0000 7eff23fff700 1 -- 172.21.15.187:0/917158936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff240b96f0 msgr2=0x7eff2409ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:39.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.019+0000 7eff23fff700 1 --2- 172.21.15.187:0/917158936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff240b96f0 0x7eff2409ff90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7eff240bb9c0 tx=0x7eff1c0091d0).stop 2020-11-20T12:09:39.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.019+0000 7eff23fff700 1 -- 172.21.15.187:0/917158936 shutdown_connections 2020-11-20T12:09:39.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.019+0000 7eff23fff700 1 --2- 172.21.15.187:0/917158936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff240b96f0 0x7eff2409ff90 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.019+0000 7eff23fff700 1 --2- 172.21.15.187:0/917158936 >> v2:172.21.15.5:3300/0 conn(0x7eff240bc840 0x7eff2414c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.019+0000 7eff23fff700 1 -- 172.21.15.187:0/917158936 >> 172.21.15.187:0/917158936 conn(0x7eff240bfb90 msgr2=0x7eff240bbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:39.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.019+0000 7eff23fff700 1 -- 172.21.15.187:0/917158936 shutdown_connections 2020-11-20T12:09:39.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.019+0000 7eff23fff700 1 -- 172.21.15.187:0/917158936 wait complete. 2020-11-20T12:09:39.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.020+0000 7eff23fff700 1 Processor -- start 2020-11-20T12:09:39.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.021+0000 7eff23fff700 1 -- start start 2020-11-20T12:09:39.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff23fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff240b96f0 0x7eff240b8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:39.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff23fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff240bc460 0x7eff240605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:39.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff23fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff24158f60 0x7eff2414a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:39.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff23fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240b8d30 con 0x7eff240b96f0 2020-11-20T12:09:39.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff23fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2415b9a0 con 0x7eff24158f60 2020-11-20T12:09:39.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff23fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff240b8a60 con 0x7eff240bc460 2020-11-20T12:09:39.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff237fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff24158f60 0x7eff2414a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:39.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff237fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff24158f60 0x7eff2414a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44920/0 (socket says 172.21.15.187:44920) 2020-11-20T12:09:39.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff237fe700 1 -- 172.21.15.187:0/3877709892 learned_addr learned my addr 172.21.15.187:0/3877709892 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:39.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff22ffd700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff240b96f0 0x7eff240b8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:39.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.022+0000 7eff227fc700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff240bc460 0x7eff240605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:39.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff237fe700 1 -- 172.21.15.187:0/3877709892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff240bc460 msgr2=0x7eff240605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:39.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff237fe700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff240bc460 0x7eff240605b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff237fe700 1 -- 172.21.15.187:0/3877709892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff240b96f0 msgr2=0x7eff240b8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:39.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff237fe700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff240b96f0 0x7eff240b8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff237fe700 1 -- 172.21.15.187:0/3877709892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff24060aa0 con 0x7eff24158f60 2020-11-20T12:09:39.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff237fe700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff24158f60 0x7eff2414a240 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7eff1c009210 tx=0x7eff1c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:39.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff0affd700 1 -- 172.21.15.187:0/3877709892 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c021070 con 0x7eff24158f60 2020-11-20T12:09:39.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff0affd700 1 -- 172.21.15.187:0/3877709892 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c010700 con 0x7eff24158f60 2020-11-20T12:09:39.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff0affd700 1 -- 172.21.15.187:0/3877709892 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c010aa0 con 0x7eff24158f60 2020-11-20T12:09:39.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2414a730 con 0x7eff24158f60 2020-11-20T12:09:39.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.023+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2415be40 con 0x7eff24158f60 2020-11-20T12:09:39.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.025+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff24068630 con 0x7eff24158f60 2020-11-20T12:09:39.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.025+0000 7eff0affd700 1 -- 172.21.15.187:0/3877709892 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7eff1c017070 con 0x7eff24158f60 2020-11-20T12:09:39.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.026+0000 7eff0affd700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff00056d70 0x7eff00059230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:39.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.026+0000 7eff0affd700 1 -- 172.21.15.187:0/3877709892 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7eff1c0723c0 con 0x7eff24158f60 2020-11-20T12:09:39.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.026+0000 7eff22ffd700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff00056d70 0x7eff00059230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:39.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.027+0000 7eff22ffd700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff00056d70 0x7eff00059230 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7eff14000bc0 tx=0x7eff14009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:39.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.030+0000 7eff0affd700 1 -- 172.21.15.187:0/3877709892 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7eff1c047730 con 0x7eff24158f60 2020-11-20T12:09:39.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.289+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff24080210 con 0x7eff24158f60 2020-11-20T12:09:39.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.289+0000 7eff0affd700 1 -- 172.21.15.187:0/3877709892 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7eff1c015070 con 0x7eff24158f60 2020-11-20T12:09:39.293 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:39.293 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:39.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff00056d70 msgr2=0x7eff00059230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:39.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff00056d70 0x7eff00059230 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7eff14000bc0 tx=0x7eff14009040).stop 2020-11-20T12:09:39.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff24158f60 msgr2=0x7eff2414a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:39.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff24158f60 0x7eff2414a240 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7eff1c009210 tx=0x7eff1c016000).stop 2020-11-20T12:09:39.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 shutdown_connections 2020-11-20T12:09:39.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff00056d70 0x7eff00059230 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff240b96f0 0x7eff240b8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff24158f60 0x7eff2414a240 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 --2- 172.21.15.187:0/3877709892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff240bc460 0x7eff240605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 >> 172.21.15.187:0/3877709892 conn(0x7eff240bfb90 msgr2=0x7eff241480d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:39.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 shutdown_connections 2020-11-20T12:09:39.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:39.292+0000 7eff23fff700 1 -- 172.21.15.187:0/3877709892 wait complete. 2020-11-20T12:09:39.299 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:39.318 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:39.564 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":82,"msgr_send_messages":82,"msgr_recv_bytes":30227,"msgr_send_bytes":12182530,"msgr_created_connections":6,"msgr_active_connections":0,"msgr_running_total_time":0.109447616,"msgr_running_send_time":0.084991753,"msgr_running_recv_time":0.007768256,"msgr_running_fast_dispatch_time":0.011843816,"msgr_send_messages_queue_lat":{"avgcount":81,"sum":0.025451869,"avgtime":0.000314220},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6010,"msgr_send_messages":13115,"msgr_recv_bytes":1690301,"msgr_send_bytes":7658534,"msgr_created_connections":5,"msgr_active_connections":2,"msgr_running_total_time":3.198828074,"msgr_running_send_time":2.722630391,"msgr_running_recv_time":0.768893347,"msgr_running_fast_dispatch_time":0.002868309,"msgr_send_messages_queue_lat":{"avgcount":13113,"sum":0.912895688,"avgtime":0.000069617},"msgr_handle_ack_lat":{"avgcount":11841,"sum":0.006704559,"avgtime":0.000000566}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":61,"msgr_send_messages":57,"msgr_recv_bytes":96478,"msgr_send_bytes":456260,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.028497081,"msgr_running_send_time":0.011093144,"msgr_running_recv_time":0.021729695,"msgr_running_fast_dispatch_time":0.004567460,"msgr_send_messages_queue_lat":{"avgcount":56,"sum":0.012928567,"avgtime":0.000230867},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":552,"complete_latency":{"avgcount":165,"sum":1.551519277,"avgtime":0.009403147}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000190808,"avgtime":0.000047702}},"mds":{"request":3815,"reply":3813,"reply_latency":{"avgcount":3813,"sum":7.789338867,"avgtime":0.002042837},"forward":0,"dir_fetch":12,"dir_commit":73,"dir_split":9,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3827,"traverse_hit":3816,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":1157056,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2963,"root_rbytes":15762432,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":69,"ceph_cap_op_grant":3610,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2142,"handle_client_caps_dirty":2073,"handle_client_cap_release":0,"process_request_cap_release":612},"mds_cache":{"num_strays":505,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":505,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":5874,"evex":0,"evtrm":0,"ev":5876,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":7,"segexg":0,"segexd":0,"expos":4194304,"wrpos":16823734,"rdpos":4195196,"jlat":{"avgcount":30,"sum":0.359752656,"avgtime":0.011991755},"replayed":2},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":76,"dir+":85,"dir-":9,"dn":3716,"dn+":3716,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":98220,"heap":356572},"mds_server":{"dispatch_client_request":3894,"dispatch_server_request":0,"handle_client_request":3815,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":505,"sum":0.510999641,"avgtime":0.001011880},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3585,"average_load":3585,"avg_session_uptime":762},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":275285,"buffer_anon_items":3311,"buffer_meta_bytes":1584,"buffer_meta_items":18,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":10601561,"mds_co_items":122678,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":151,"op_send_bytes":13622163,"op_resend":0,"op_reply":151,"oplen_avg":{"avgcount":151,"sum":253},"op":151,"op_r":26,"op_w":125,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":30,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":16,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":5,"osd_session_open":5,"osd_session_close":0,"osd_laggy":0,"omap_wr":150,"omap_rd":28,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":4264,"max":104857600,"get_started":0,"get":6149,"get_sum":1356522,"get_or_fail_fail":0,"get_or_fail_success":6149,"take":0,"take_sum":0,"put":6147,"put_sum":1352258,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":151,"take_sum":13623095,"put":122,"put_sum":13623095,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":151,"take_sum":151,"put":151,"put_sum":151,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":46518,"max":3758096384,"get_started":0,"get":5874,"get_sum":12628274,"get_or_fail_fail":0,"get_or_fail_success":5874,"take":0,"take_sum":0,"put":30,"put_sum":12581756,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:39.566 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 0, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 30227, "msgr_recv_messages": 82, "msgr_running_fast_dispatch_time": 0.011843816, "msgr_running_recv_time": 0.007768256, "msgr_running_send_time": 0.084991753, "msgr_running_total_time": 0.109447616, "msgr_send_bytes": 12182530, "msgr_send_messages": 82, "msgr_send_messages_queue_lat": { "avgcount": 81, "avgtime": 0.00031422, "sum": 0.025451869 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 2, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 11841, "avgtime": 5.66e-07, "sum": 0.006704559 }, "msgr_recv_bytes": 1690301, "msgr_recv_messages": 6010, "msgr_running_fast_dispatch_time": 0.002868309, "msgr_running_recv_time": 0.768893347, "msgr_running_send_time": 2.722630391, "msgr_running_total_time": 3.198828074, "msgr_send_bytes": 7658534, "msgr_send_messages": 13115, "msgr_send_messages_queue_lat": { "avgcount": 13113, "avgtime": 6.9617e-05, "sum": 0.912895688 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 96478, "msgr_recv_messages": 61, "msgr_running_fast_dispatch_time": 0.00456746, "msgr_running_recv_time": 0.021729695, "msgr_running_send_time": 0.011093144, "msgr_running_total_time": 0.028497081, "msgr_send_bytes": 456260, "msgr_send_messages": 57, "msgr_send_messages_queue_lat": { "avgcount": 56, "avgtime": 0.000230867, "sum": 0.012928567 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 165, "avgtime": 0.009403147, "sum": 1.551519277 }, "queue_len": 552 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 4.7702e-05, "sum": 0.000190808 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3610, "ceph_cap_op_revoke": 69, "ceph_cap_op_trunc": 0, "dir_commit": 73, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2142, "handle_client_caps_dirty": 2073, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 1157056, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 612, "q": 0, "reply": 3813, "reply_latency": { "avgcount": 3813, "avgtime": 0.002042837, "sum": 7.789338867 }, "request": 3815, "root_rbytes": 15762432, "root_rfiles": 2963, "root_rsnaps": 0, "subtrees": 2, "traverse": 3827, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3816, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 505, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 505, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 5876, "evadd": 5874, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 30, "avgtime": 0.011991755, "sum": 0.359752656 }, "rdpos": 4195196, "replayed": 2, "seg": 7, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 16823734 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 76, "dir+": 85, "dir-": 9, "dn": 3716, "dn+": 3716, "dn-": 0, "heap": 356572, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 98220 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3894, "dispatch_server_request": 0, "handle_client_request": 3815, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 505, "avgtime": 0.00101188, "sum": 0.510999641 } }, "mds_sessions": { "average_load": 3585, "avg_session_uptime": 762, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3585 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 275285, "buffer_anon_items": 3311, "buffer_meta_bytes": 1584, "buffer_meta_items": 18, "mds_co_bytes": 10601561, "mds_co_items": 122678, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 28, "omap_wr": 150, "op": 151, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 26, "op_reply": 151, "op_resend": 0, "op_rmw": 0, "op_send": 151, "op_send_bytes": 13622163, "op_w": 125, "oplen_avg": { "avgcount": 151, "sum": 253 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 5, "osd_sessions": 5, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 16, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 30, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6149, "get_or_fail_fail": 0, "get_or_fail_success": 6149, "get_started": 0, "get_sum": 1356522, "max": 104857600, "put": 6147, "put_sum": 1352258, "take": 0, "take_sum": 0, "val": 4264, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 122, "put_sum": 13623095, "take": 151, "take_sum": 13623095, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 151, "put_sum": 151, "take": 151, "take_sum": 151, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 5874, "get_or_fail_fail": 0, "get_or_fail_success": 5874, "get_started": 0, "get_sum": 12628274, "max": 3758096384, "put": 30, "put_sum": 12581756, "take": 0, "take_sum": 0, "val": 46518, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:39.567 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/505, 0/505 2020-11-20T12:09:40.568 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:40.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.860+0000 7fe57336f700 1 Processor -- start 2020-11-20T12:09:40.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.860+0000 7fe57336f700 1 -- start start 2020-11-20T12:09:40.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.862+0000 7fe57336f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe56c14cc80 0x7fe56c145980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:40.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.862+0000 7fe57336f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c14d0b0 con 0x7fe56c066070 2020-11-20T12:09:40.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.862+0000 7fe57336f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c14c2a0 con 0x7fe56c065c90 2020-11-20T12:09:40.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.862+0000 7fe57336f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c14c540 con 0x7fe56c14cc80 2020-11-20T12:09:40.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.862+0000 7fe57236d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe56c065c90 0x7fe56c0b5310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41536/0 (socket says 172.21.15.187:41536) 2020-11-20T12:09:40.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.863+0000 7fe57236d700 1 -- 172.21.15.187:0/1573953091 learned_addr learned my addr 172.21.15.187:0/1573953091 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:40.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.863+0000 7fe572b6e700 1 --2- 172.21.15.187:0/1573953091 >> v2:172.21.15.5:3300/0 conn(0x7fe56c14cc80 0x7fe56c145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:40.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.864+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930242651 0 0) 0x7fe56c14c2a0 con 0x7fe56c065c90 2020-11-20T12:09:40.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.864+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe554003240 con 0x7fe56c065c90 2020-11-20T12:09:40.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.864+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1346794419 0 0) 0x7fe554003240 con 0x7fe56c065c90 2020-11-20T12:09:40.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.864+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 >> v2:172.21.15.5:3300/0 conn(0x7fe56c14cc80 msgr2=0x7fe56c145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:40.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.864+0000 7fe57136b700 1 --2- 172.21.15.187:0/1573953091 >> v2:172.21.15.5:3300/0 conn(0x7fe56c14cc80 0x7fe56c145980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:40.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.864+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 >> v1:172.21.15.5:6789/0 conn(0x7fe56c066070 legacy=0x7fe56c0b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:40.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c14c7e0 con 0x7fe56c065c90 2020-11-20T12:09:40.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fe5680014e0 con 0x7fe56c065c90 2020-11-20T12:09:40.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 >> v1:172.21.15.187:6789/0 conn(0x7fe56c065c90 legacy=0x7fe56c0b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:40.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe554003730 0x7fe554003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:40.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe5540044e0 0x7fe554004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:40.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe554004e50 0x7fe55400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:40.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c14c540 con 0x7fe5540044e0 2020-11-20T12:09:40.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554004260 con 0x7fe554003730 2020-11-20T12:09:40.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55400bc20 con 0x7fe554004e50 2020-11-20T12:09:40.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.865+0000 7fe57236d700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe554003730 0x7fe554003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:40.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe572b6e700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe554004e50 0x7fe55400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:40.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57236d700 1 -- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe554004e50 msgr2=0x7fe55400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:40.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57236d700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe554004e50 0x7fe55400b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:40.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe571b6c700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe5540044e0 0x7fe554004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:40.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57236d700 1 -- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe5540044e0 msgr2=0x7fe554004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:40.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57236d700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe5540044e0 0x7fe554004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:40.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57236d700 1 -- 172.21.15.187:0/1573953091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe554003240 con 0x7fe554003730 2020-11-20T12:09:40.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe572b6e700 1 -- 172.21.15.187:0/1573953091 reap_dead start 2020-11-20T12:09:40.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe571b6c700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe5540044e0 0x7fe554004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:40.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57236d700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe554003730 0x7fe554003b50 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fe56800c440 tx=0x7fe56c067dd0).ready entity=mon.1 client_cookie=ce4fa1cde2d493a9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:40.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56800ce30 con 0x7fe554003730 2020-11-20T12:09:40.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5680072e0 con 0x7fe554003730 2020-11-20T12:09:40.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.866+0000 7fe57136b700 1 -- 172.21.15.187:0/1573953091 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568005820 con 0x7fe554003730 2020-11-20T12:09:40.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.867+0000 7fe57336f700 1 -- 172.21.15.187:0/1573953091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe554003730 msgr2=0x7fe554003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:40.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.867+0000 7fe57336f700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe554003730 0x7fe554003b50 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fe56800c440 tx=0x7fe56c067dd0).stop 2020-11-20T12:09:40.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.867+0000 7fe57336f700 1 -- 172.21.15.187:0/1573953091 shutdown_connections 2020-11-20T12:09:40.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.867+0000 7fe57336f700 1 --2- 172.21.15.187:0/1573953091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe554003730 0x7fe554003b50 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:40.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.867+0000 7fe57336f700 1 -- 172.21.15.187:0/1573953091 >> 172.21.15.187:0/1573953091 conn(0x7fe56c0b8280 msgr2=0x7fe56c0653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:40.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.868+0000 7fe57336f700 1 -- 172.21.15.187:0/1573953091 shutdown_connections 2020-11-20T12:09:40.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.868+0000 7fe57336f700 1 -- 172.21.15.187:0/1573953091 wait complete. 2020-11-20T12:09:40.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.871+0000 7fe57336f700 1 Processor -- start 2020-11-20T12:09:40.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.871+0000 7fe57336f700 1 -- start start 2020-11-20T12:09:40.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe57336f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe56c14f5b0 0x7fe56c09b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:40.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe57336f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe56c14f990 0x7fe56c09f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:40.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe57336f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe56c09a810 0x7fe56c09f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:40.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe571b6c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe56c14f990 0x7fe56c09f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:40.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe57236d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe56c14f5b0 0x7fe56c09b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:40.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe57336f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c143120 con 0x7fe56c09a810 2020-11-20T12:09:40.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe57336f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c09afa0 con 0x7fe56c14f5b0 2020-11-20T12:09:40.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe572b6e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe56c09a810 0x7fe56c09f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:40.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe571b6c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe56c14f990 0x7fe56c09f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:45832/0 (socket says 172.21.15.187:45832) 2020-11-20T12:09:40.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe571b6c700 1 -- 172.21.15.187:0/334333573 learned_addr learned my addr 172.21.15.187:0/334333573 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:40.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.873+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c09d690 con 0x7fe56c14f990 2020-11-20T12:09:40.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.874+0000 7fe572b6e700 1 -- 172.21.15.187:0/334333573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe56c14f990 msgr2=0x7fe56c09f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:40.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.874+0000 7fe572b6e700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe56c14f990 0x7fe56c09f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:40.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.874+0000 7fe572b6e700 1 -- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe56c14f5b0 msgr2=0x7fe56c09b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:40.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.875+0000 7fe572b6e700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe56c14f5b0 0x7fe56c09b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:40.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.875+0000 7fe572b6e700 1 -- 172.21.15.187:0/334333573 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c09d850 con 0x7fe56c09a810 2020-11-20T12:09:40.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.875+0000 7fe571b6c700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe56c14f990 0x7fe56c09f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:09:40.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.875+0000 7fe57236d700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe56c14f5b0 0x7fe56c09b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:09:40.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.875+0000 7fe572b6e700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe56c09a810 0x7fe56c09f950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fe56c066660 tx=0x7fe564008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:40.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.876+0000 7fe55a7fc700 1 -- 172.21.15.187:0/334333573 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56400e960 con 0x7fe56c09a810 2020-11-20T12:09:40.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.876+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c0b1d20 con 0x7fe56c09a810 2020-11-20T12:09:40.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.876+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c0a0080 con 0x7fe56c09a810 2020-11-20T12:09:40.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.877+0000 7fe55a7fc700 1 -- 172.21.15.187:0/334333573 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56400eb00 con 0x7fe56c09a810 2020-11-20T12:09:40.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.877+0000 7fe55a7fc700 1 -- 172.21.15.187:0/334333573 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56400eec0 con 0x7fe56c09a810 2020-11-20T12:09:40.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.879+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56c068ef0 con 0x7fe56c09a810 2020-11-20T12:09:40.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.879+0000 7fe55a7fc700 1 -- 172.21.15.187:0/334333573 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe564014640 con 0x7fe56c09a810 2020-11-20T12:09:40.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.880+0000 7fe55a7fc700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe544056d50 0x7fe544059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:40.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.880+0000 7fe55a7fc700 1 -- 172.21.15.187:0/334333573 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe564070570 con 0x7fe56c09a810 2020-11-20T12:09:40.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.881+0000 7fe57236d700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe544056d50 0x7fe544059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:40.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.882+0000 7fe57236d700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe544056d50 0x7fe544059210 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fe56c05d960 tx=0x7fe568015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:40.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:40.891+0000 7fe55a7fc700 1 -- 172.21.15.187:0/334333573 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe5640458f0 con 0x7fe56c09a810 2020-11-20T12:09:41.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.199+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe56c044f60 con 0x7fe56c09a810 2020-11-20T12:09:41.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.201+0000 7fe55a7fc700 1 -- 172.21.15.187:0/334333573 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fe564048f10 con 0x7fe56c09a810 2020-11-20T12:09:41.204 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:41.204 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:41.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe544056d50 msgr2=0x7fe544059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:41.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe544056d50 0x7fe544059210 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fe56c05d960 tx=0x7fe568015000).stop 2020-11-20T12:09:41.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe56c09a810 msgr2=0x7fe56c09f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:41.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe56c09a810 0x7fe56c09f950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fe56c066660 tx=0x7fe564008040).stop 2020-11-20T12:09:41.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 shutdown_connections 2020-11-20T12:09:41.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe544056d50 0x7fe544059210 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe56c09a810 0x7fe56c09f950 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe56c14f5b0 0x7fe56c09b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 --2- 172.21.15.187:0/334333573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe56c14f990 0x7fe56c09f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 >> 172.21.15.187:0/334333573 conn(0x7fe56c0b8280 msgr2=0x7fe56c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:41.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 shutdown_connections 2020-11-20T12:09:41.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:41.204+0000 7fe57336f700 1 -- 172.21.15.187:0/334333573 wait complete. 2020-11-20T12:09:41.210 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:41.230 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:41.498 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":271,"msgr_send_messages":271,"msgr_recv_bytes":112501,"msgr_send_bytes":12277089,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.140079291,"msgr_running_send_time":0.101463454,"msgr_running_recv_time":0.014438426,"msgr_running_fast_dispatch_time":0.017043924,"msgr_send_messages_queue_lat":{"avgcount":270,"sum":0.055355289,"avgtime":0.000205019},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7127,"msgr_send_messages":14598,"msgr_recv_bytes":1972449,"msgr_send_bytes":8302094,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.626938119,"msgr_running_send_time":3.074667146,"msgr_running_recv_time":0.845462731,"msgr_running_fast_dispatch_time":0.005850299,"msgr_send_messages_queue_lat":{"avgcount":14596,"sum":1.026936341,"avgtime":0.000070357},"msgr_handle_ack_lat":{"avgcount":13125,"sum":0.007187827,"avgtime":0.000000547}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":143,"msgr_send_messages":139,"msgr_recv_bytes":117298,"msgr_send_bytes":482410,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.037758296,"msgr_running_send_time":0.014078837,"msgr_running_recv_time":0.024496908,"msgr_running_fast_dispatch_time":0.007009839,"msgr_send_messages_queue_lat":{"avgcount":138,"sum":0.027063150,"avgtime":0.000196109},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":166,"sum":1.795377129,"avgtime":0.010815524}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":311,"sum":0.080522325,"avgtime":0.000258914}},"mds":{"request":4847,"reply":4845,"reply_latency":{"avgcount":4845,"sum":9.491323313,"avgtime":0.001958993},"forward":0,"dir_fetch":12,"dir_commit":73,"dir_split":9,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":4859,"traverse_hit":4848,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":1157056,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2963,"root_rbytes":15762432,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":69,"ceph_cap_op_grant":3794,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2142,"handle_client_caps_dirty":2073,"handle_client_cap_release":2,"process_request_cap_release":1644},"mds_cache":{"num_strays":1197,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1537,"strays_enqueued":340,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":7362,"evex":0,"evtrm":0,"ev":7364,"evexg":0,"evexd":0,"segadd":7,"segex":0,"segtrm":0,"seg":8,"segexg":0,"segexd":0,"expos":4194304,"wrpos":20550805,"rdpos":4195196,"jlat":{"avgcount":30,"sum":0.359752656,"avgtime":0.011991755},"replayed":2},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":76,"dir+":85,"dir-":9,"dn":4748,"dn+":4748,"dn-":0,"cap":2862,"cap+":3202,"cap-":340,"rss":98220,"heap":356572},"mds_server":{"dispatch_client_request":4926,"dispatch_server_request":0,"handle_client_request":4847,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1537,"sum":2.212984087,"avgtime":0.001439807},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4522,"average_load":4522,"avg_session_uptime":762},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":275285,"buffer_anon_items":3311,"buffer_meta_bytes":1584,"buffer_meta_items":18,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":10601561,"mds_co_items":122678,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":505,"op_send_bytes":13656683,"op_resend":0,"op_reply":505,"oplen_avg":{"avgcount":505,"sum":607},"op":505,"op_r":29,"op_w":476,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":11,"osdop_write":33,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":362,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":150,"omap_rd":28,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":340,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":176,"max":104857600,"get_started":0,"get":7538,"get_sum":1633564,"get_or_fail_fail":0,"get_or_fail_success":7538,"take":0,"take_sum":0,"put":7537,"put_sum":1633388,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":505,"take_sum":13691955,"put":130,"put_sum":13691955,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":505,"take_sum":505,"put":505,"put_sum":505,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":340,"get_sum":34340,"get_or_fail_fail":0,"get_or_fail_success":340,"take":0,"take_sum":0,"put":3,"put_sum":34340,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":3773589,"max":3758096384,"get_started":0,"get":7362,"get_sum":16355345,"get_or_fail_fail":0,"get_or_fail_success":7362,"take":0,"take_sum":0,"put":30,"put_sum":12581756,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:41.500 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 112501, "msgr_recv_messages": 271, "msgr_running_fast_dispatch_time": 0.017043924, "msgr_running_recv_time": 0.014438426, "msgr_running_send_time": 0.101463454, "msgr_running_total_time": 0.140079291, "msgr_send_bytes": 12277089, "msgr_send_messages": 271, "msgr_send_messages_queue_lat": { "avgcount": 270, "avgtime": 0.000205019, "sum": 0.055355289 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 13125, "avgtime": 5.47e-07, "sum": 0.007187827 }, "msgr_recv_bytes": 1972449, "msgr_recv_messages": 7127, "msgr_running_fast_dispatch_time": 0.005850299, "msgr_running_recv_time": 0.845462731, "msgr_running_send_time": 3.074667146, "msgr_running_total_time": 3.626938119, "msgr_send_bytes": 8302094, "msgr_send_messages": 14598, "msgr_send_messages_queue_lat": { "avgcount": 14596, "avgtime": 7.0357e-05, "sum": 1.026936341 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 117298, "msgr_recv_messages": 143, "msgr_running_fast_dispatch_time": 0.007009839, "msgr_running_recv_time": 0.024496908, "msgr_running_send_time": 0.014078837, "msgr_running_total_time": 0.037758296, "msgr_send_bytes": 482410, "msgr_send_messages": 139, "msgr_send_messages_queue_lat": { "avgcount": 138, "avgtime": 0.000196109, "sum": 0.02706315 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 166, "avgtime": 0.010815524, "sum": 1.795377129 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 311, "avgtime": 0.000258914, "sum": 0.080522325 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3794, "ceph_cap_op_revoke": 69, "ceph_cap_op_trunc": 0, "dir_commit": 73, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 2, "handle_client_caps": 2142, "handle_client_caps_dirty": 2073, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 1157056, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1644, "q": 0, "reply": 4845, "reply_latency": { "avgcount": 4845, "avgtime": 0.001958993, "sum": 9.491323313 }, "request": 4847, "root_rbytes": 15762432, "root_rfiles": 2963, "root_rsnaps": 0, "subtrees": 2, "traverse": 4859, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4848, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1197, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1537, "strays_enqueued": 340, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7364, "evadd": 7362, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 30, "avgtime": 0.011991755, "sum": 0.359752656 }, "rdpos": 4195196, "replayed": 2, "seg": 8, "segadd": 7, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 20550805 }, "mds_mem": { "cap": 2862, "cap+": 3202, "cap-": 340, "dir": 76, "dir+": 85, "dir-": 9, "dn": 4748, "dn+": 4748, "dn-": 0, "heap": 356572, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 98220 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4926, "dispatch_server_request": 0, "handle_client_request": 4847, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1537, "avgtime": 0.001439807, "sum": 2.212984087 } }, "mds_sessions": { "average_load": 4522, "avg_session_uptime": 762, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4522 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 275285, "buffer_anon_items": 3311, "buffer_meta_bytes": 1584, "buffer_meta_items": 18, "mds_co_bytes": 10601561, "mds_co_items": 122678, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 28, "omap_wr": 150, "op": 505, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 29, "op_reply": 505, "op_resend": 0, "op_rmw": 0, "op_send": 505, "op_send_bytes": 13656683, "op_w": 476, "oplen_avg": { "avgcount": 505, "sum": 607 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 362, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 11, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 33, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 340, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7538, "get_or_fail_fail": 0, "get_or_fail_success": 7538, "get_started": 0, "get_sum": 1633564, "max": 104857600, "put": 7537, "put_sum": 1633388, "take": 0, "take_sum": 0, "val": 176, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 130, "put_sum": 13691955, "take": 505, "take_sum": 13691955, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 505, "put_sum": 505, "take": 505, "take_sum": 505, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 340, "get_or_fail_fail": 0, "get_or_fail_success": 340, "get_started": 0, "get_sum": 34340, "max": 3758096384, "put": 3, "put_sum": 34340, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 7362, "get_or_fail_fail": 0, "get_or_fail_success": 7362, "get_started": 0, "get_sum": 16355345, "max": 3758096384, "put": 30, "put_sum": 12581756, "take": 0, "take_sum": 0, "val": 3773589, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:41.500 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1197, 340/1537 2020-11-20T12:09:42.502 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:43.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.843+0000 7f6c4982e700 1 Processor -- start 2020-11-20T12:09:43.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.843+0000 7f6c4982e700 1 -- start start 2020-11-20T12:09:43.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.844+0000 7f6c4982e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6c440aff90 0x7f6c440668f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.844+0000 7f6c4982e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c440b1880 con 0x7f6c4409a080 2020-11-20T12:09:43.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.844+0000 7f6c4982e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c440b0510 con 0x7f6c4407f0a0 2020-11-20T12:09:43.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.844+0000 7f6c4982e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c440b07b0 con 0x7f6c440aff90 2020-11-20T12:09:43.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.845+0000 7f6c4902d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6c4407f0a0 0x7f6c4407f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41560/0 (socket says 172.21.15.187:41560) 2020-11-20T12:09:43.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.845+0000 7f6c4902d700 1 -- 172.21.15.187:0/3819171092 learned_addr learned my addr 172.21.15.187:0/3819171092 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:43.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.845+0000 7f6c3bfff700 1 --2- 172.21.15.187:0/3819171092 >> v2:172.21.15.5:3300/0 conn(0x7f6c440aff90 0x7f6c440668f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.845+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662687923 0 0) 0x7f6c440b0510 con 0x7f6c4407f0a0 2020-11-20T12:09:43.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.845+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c2c003240 con 0x7f6c4407f0a0 2020-11-20T12:09:43.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.846+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2397707840 0 0) 0x7f6c2c003240 con 0x7f6c4407f0a0 2020-11-20T12:09:43.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 >> v2:172.21.15.5:3300/0 conn(0x7f6c440aff90 msgr2=0x7f6c440668f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:43.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 --2- 172.21.15.187:0/3819171092 >> v2:172.21.15.5:3300/0 conn(0x7f6c440aff90 0x7f6c440668f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 >> v1:172.21.15.5:6789/0 conn(0x7f6c4409a080 legacy=0x7f6c44065ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c440b0a50 con 0x7f6c4407f0a0 2020-11-20T12:09:43.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6c3c001860 con 0x7f6c4407f0a0 2020-11-20T12:09:43.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 >> v1:172.21.15.187:6789/0 conn(0x7f6c4407f0a0 legacy=0x7f6c4407f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c2c003730 0x7f6c2c003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c2c0044e0 0x7f6c2c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c004e50 0x7f6c2c00fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440b07b0 con 0x7f6c2c004e50 2020-11-20T12:09:43.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c007710 con 0x7f6c2c0044e0 2020-11-20T12:09:43.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.857+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c004270 con 0x7f6c2c003730 2020-11-20T12:09:43.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.858+0000 7f6c3bfff700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c2c0044e0 0x7f6c2c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.858+0000 7f6c4882c700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c2c003730 0x7f6c2c003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3bfff700 1 -- 172.21.15.187:0/3819171092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c2c003730 msgr2=0x7f6c2c003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:43.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3bfff700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c2c003730 0x7f6c2c003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3bfff700 1 -- 172.21.15.187:0/3819171092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c004e50 msgr2=0x7f6c2c00fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:43.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3bfff700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c004e50 0x7f6c2c00fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:43.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3bfff700 1 -- 172.21.15.187:0/3819171092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c003240 con 0x7f6c2c0044e0 2020-11-20T12:09:43.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c4902d700 1 -- 172.21.15.187:0/3819171092 reap_dead start 2020-11-20T12:09:43.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3bfff700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c2c0044e0 0x7f6c2c004960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f6c4000cba0 tx=0x7f6c40006df0).ready entity=mon.1 client_cookie=751c64abe1bade17 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:43.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4000c330 con 0x7f6c2c0044e0 2020-11-20T12:09:43.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4000c020 con 0x7f6c2c0044e0 2020-11-20T12:09:43.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.859+0000 7f6c3b7fe700 1 -- 172.21.15.187:0/3819171092 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4000ed90 con 0x7f6c2c0044e0 2020-11-20T12:09:43.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.860+0000 7f6c4982e700 1 -- 172.21.15.187:0/3819171092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c2c0044e0 msgr2=0x7f6c2c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.860+0000 7f6c4982e700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c2c0044e0 0x7f6c2c004960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f6c4000cba0 tx=0x7f6c40006df0).stop 2020-11-20T12:09:43.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.861+0000 7f6c4982e700 1 -- 172.21.15.187:0/3819171092 shutdown_connections 2020-11-20T12:09:43.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.861+0000 7f6c4982e700 1 --2- 172.21.15.187:0/3819171092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c2c0044e0 0x7f6c2c004960 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.861+0000 7f6c4982e700 1 -- 172.21.15.187:0/3819171092 >> 172.21.15.187:0/3819171092 conn(0x7f6c440b8280 msgr2=0x7f6c440653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:43.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.861+0000 7f6c4982e700 1 -- 172.21.15.187:0/3819171092 shutdown_connections 2020-11-20T12:09:43.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.861+0000 7f6c4982e700 1 -- 172.21.15.187:0/3819171092 wait complete. 2020-11-20T12:09:43.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.862+0000 7f6c4982e700 1 Processor -- start 2020-11-20T12:09:43.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.863+0000 7f6c4982e700 1 -- start start 2020-11-20T12:09:43.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.863+0000 7f6c4982e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c0044e0 0x7f6c4409cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.863+0000 7f6c4982e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c4409c0d0 0x7f6c44159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.863+0000 7f6c4982e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c4409fce0 0x7f6c4407e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.864+0000 7f6c4982e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4409d4d0 con 0x7f6c2c0044e0 2020-11-20T12:09:43.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.864+0000 7f6c4982e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4409de40 con 0x7f6c4409c0d0 2020-11-20T12:09:43.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.864+0000 7f6c4982e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4409e0e0 con 0x7f6c4409fce0 2020-11-20T12:09:43.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.864+0000 7f6c3bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c4409c0d0 0x7f6c44159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.864+0000 7f6c4882c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c0044e0 0x7f6c4409cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.864+0000 7f6c4882c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c0044e0 0x7f6c4409cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:37226/0 (socket says 172.21.15.187:37226) 2020-11-20T12:09:43.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.865+0000 7f6c4882c700 1 -- 172.21.15.187:0/1954515872 learned_addr learned my addr 172.21.15.187:0/1954515872 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:43.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.865+0000 7f6c4902d700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c4409fce0 0x7f6c4407e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.865+0000 7f6c3bfff700 1 -- 172.21.15.187:0/1954515872 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c4409fce0 msgr2=0x7f6c4407e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.865+0000 7f6c3bfff700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c4409fce0 0x7f6c4407e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.865+0000 7f6c3bfff700 1 -- 172.21.15.187:0/1954515872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c0044e0 msgr2=0x7f6c4409cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.865+0000 7f6c3bfff700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c0044e0 0x7f6c4409cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.865+0000 7f6c3bfff700 1 -- 172.21.15.187:0/1954515872 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c44159520 con 0x7f6c4409c0d0 2020-11-20T12:09:43.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.866+0000 7f6c3bfff700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c4409c0d0 0x7f6c44159030 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f6c40016000 tx=0x7f6c40019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:43.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.867+0000 7f6c38ff9700 1 -- 172.21.15.187:0/1954515872 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4001e070 con 0x7f6c4409c0d0 2020-11-20T12:09:43.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.867+0000 7f6c38ff9700 1 -- 172.21.15.187:0/1954515872 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c40011e50 con 0x7f6c4409c0d0 2020-11-20T12:09:43.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.867+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c440a0160 con 0x7f6c4409c0d0 2020-11-20T12:09:43.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.867+0000 7f6c38ff9700 1 -- 172.21.15.187:0/1954515872 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40019860 con 0x7f6c4409c0d0 2020-11-20T12:09:43.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.867+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4407ecd0 con 0x7f6c4409c0d0 2020-11-20T12:09:43.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.868+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c440a02f0 con 0x7f6c4409c0d0 2020-11-20T12:09:43.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.869+0000 7f6c38ff9700 1 -- 172.21.15.187:0/1954515872 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6c40019c50 con 0x7f6c4409c0d0 2020-11-20T12:09:43.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.869+0000 7f6c38ff9700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c1c056d20 0x7f6c1c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.869+0000 7f6c38ff9700 1 -- 172.21.15.187:0/1954515872 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6c400715a0 con 0x7f6c4409c0d0 2020-11-20T12:09:43.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.870+0000 7f6c4882c700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c1c056d20 0x7f6c1c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.870+0000 7f6c4882c700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c1c056d20 0x7f6c1c0591e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f6c440b60b0 tx=0x7f6c34009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:43.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:43.877+0000 7f6c38ff9700 1 -- 172.21.15.187:0/1954515872 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6c400469a0 con 0x7f6c4409c0d0 2020-11-20T12:09:44.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.164+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c44068ef0 con 0x7f6c4409c0d0 2020-11-20T12:09:44.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.170+0000 7f6c38ff9700 1 -- 172.21.15.187:0/1954515872 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f6c40049fc0 con 0x7f6c4409c0d0 2020-11-20T12:09:44.174 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:44.174 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:44.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c1c056d20 msgr2=0x7f6c1c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:44.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c1c056d20 0x7f6c1c0591e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f6c440b60b0 tx=0x7f6c34009040).stop 2020-11-20T12:09:44.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c4409c0d0 msgr2=0x7f6c44159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:44.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c4409c0d0 0x7f6c44159030 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f6c40016000 tx=0x7f6c40019000).stop 2020-11-20T12:09:44.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 shutdown_connections 2020-11-20T12:09:44.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6c1c056d20 0x7f6c1c0591e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:44.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c2c0044e0 0x7f6c4409cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:44.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c4409fce0 0x7f6c4407e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:44.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 --2- 172.21.15.187:0/1954515872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c4409c0d0 0x7f6c44159030 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:44.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.173+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 >> 172.21.15.187:0/1954515872 conn(0x7f6c440b8280 msgr2=0x7f6c44157490 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:44.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.174+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 shutdown_connections 2020-11-20T12:09:44.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:44.174+0000 7f6c4982e700 1 -- 172.21.15.187:0/1954515872 wait complete. 2020-11-20T12:09:44.179 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:44.208 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:44.498 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":615,"msgr_send_messages":615,"msgr_recv_bytes":268493,"msgr_send_bytes":12455531,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.197446654,"msgr_running_send_time":0.132730512,"msgr_running_recv_time":0.025848650,"msgr_running_fast_dispatch_time":0.027826869,"msgr_send_messages_queue_lat":{"avgcount":614,"sum":0.087585015,"avgtime":0.000142646},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8833,"msgr_send_messages":19445,"msgr_recv_bytes":2467158,"msgr_send_bytes":13739125,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":4.394596588,"msgr_running_send_time":3.731763609,"msgr_running_recv_time":2.385620051,"msgr_running_fast_dispatch_time":0.014175783,"msgr_send_messages_queue_lat":{"avgcount":19443,"sum":1.336713415,"avgtime":0.000068750},"msgr_handle_ack_lat":{"avgcount":16240,"sum":0.008250722,"avgtime":0.000000508}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":323,"msgr_send_messages":319,"msgr_recv_bytes":163076,"msgr_send_bytes":1304505,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.148916890,"msgr_running_send_time":0.049756383,"msgr_running_recv_time":0.029471198,"msgr_running_fast_dispatch_time":0.075229016,"msgr_send_messages_queue_lat":{"avgcount":318,"sum":0.050945533,"avgtime":0.000160206},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":182,"sum":2.971864108,"avgtime":0.016328923}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":921,"sum":0.257686747,"avgtime":0.000279790}},"mds":{"request":5675,"reply":5673,"reply_latency":{"avgcount":5673,"sum":11.531628735,"avgtime":0.002032721},"forward":0,"dir_fetch":12,"dir_commit":74,"dir_split":9,"dir_merge":0,"inodes":2191,"inodes_top":0,"inodes_bottom":2191,"inodes_pin_tail":0,"inodes_pinned":2191,"inodes_expired":0,"inodes_with_caps":2025,"caps":2025,"subtrees":2,"traverse":5687,"traverse_hit":5676,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":1157056,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":835,"root_rbytes":4599808,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":77,"ceph_cap_op_grant":5668,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2328,"handle_client_caps_dirty":2259,"handle_client_cap_release":7,"process_request_cap_release":2472},"mds_cache":{"num_strays":1345,"num_strays_delayed":157,"num_strays_enqueuing":0,"strays_created":2365,"strays_enqueued":1020,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":10,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9149,"evex":0,"evtrm":0,"ev":9151,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":10,"segexg":0,"segexd":0,"expos":4194304,"wrpos":24170471,"rdpos":4195196,"jlat":{"avgcount":36,"sum":0.649874707,"avgtime":0.018052075},"replayed":2},"mds_mem":{"ino":2194,"ino+":3214,"ino-":1020,"dir":45,"dir+":86,"dir-":41,"dn":4556,"dn+":5576,"dn-":1020,"cap":2025,"cap+":3202,"cap-":1177,"rss":99548,"heap":356572},"mds_server":{"dispatch_client_request":5754,"dispatch_server_request":0,"handle_client_request":5675,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2365,"sum":4.253289509,"avgtime":0.001798431},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5218,"average_load":5218,"avg_session_uptime":767},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":17592994,"buffer_anon_items":7825,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12850536,"mds_co_items":196586,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1202,"op_send_bytes":21005447,"op_resend":0,"op_reply":1202,"oplen_avg":{"avgcount":1202,"sum":1306},"op":1202,"op_r":32,"op_w":1170,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":14,"osdop_write":42,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":1043,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":153,"omap_rd":28,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":1020,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":128239,"max":104857600,"get_started":0,"get":9767,"get_sum":2153627,"get_or_fail_fail":0,"get_or_fail_success":9767,"take":0,"take_sum":0,"put":9256,"put_sum":2025388,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1202,"take_sum":21109399,"put":146,"put_sum":21109399,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1202,"take_sum":1202,"put":1202,"put_sum":1202,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1020,"get_sum":103020,"get_or_fail_fail":0,"get_or_fail_success":1020,"take":0,"take_sum":0,"put":6,"put_sum":103020,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":197600,"max":3758096384,"get_started":0,"get":9150,"get_sum":19977025,"get_or_fail_fail":0,"get_or_fail_success":9150,"take":0,"take_sum":0,"put":36,"put_sum":19779425,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:44.500 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 268493, "msgr_recv_messages": 615, "msgr_running_fast_dispatch_time": 0.027826869, "msgr_running_recv_time": 0.02584865, "msgr_running_send_time": 0.132730512, "msgr_running_total_time": 0.197446654, "msgr_send_bytes": 12455531, "msgr_send_messages": 615, "msgr_send_messages_queue_lat": { "avgcount": 614, "avgtime": 0.000142646, "sum": 0.087585015 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 16240, "avgtime": 5.08e-07, "sum": 0.008250722 }, "msgr_recv_bytes": 2467158, "msgr_recv_messages": 8833, "msgr_running_fast_dispatch_time": 0.014175783, "msgr_running_recv_time": 2.385620051, "msgr_running_send_time": 3.731763609, "msgr_running_total_time": 4.394596588, "msgr_send_bytes": 13739125, "msgr_send_messages": 19445, "msgr_send_messages_queue_lat": { "avgcount": 19443, "avgtime": 6.875e-05, "sum": 1.336713415 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 163076, "msgr_recv_messages": 323, "msgr_running_fast_dispatch_time": 0.075229016, "msgr_running_recv_time": 0.029471198, "msgr_running_send_time": 0.049756383, "msgr_running_total_time": 0.14891689, "msgr_send_bytes": 1304505, "msgr_send_messages": 319, "msgr_send_messages_queue_lat": { "avgcount": 318, "avgtime": 0.000160206, "sum": 0.050945533 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 182, "avgtime": 0.016328923, "sum": 2.971864108 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 921, "avgtime": 0.00027979, "sum": 0.257686747 }, "queue_len": 0 }, "mds": { "caps": 2025, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5668, "ceph_cap_op_revoke": 77, "ceph_cap_op_trunc": 0, "dir_commit": 74, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 7, "handle_client_caps": 2328, "handle_client_caps_dirty": 2259, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 2191, "inodes_bottom": 2191, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 2191, "inodes_top": 0, "inodes_with_caps": 2025, "load_cent": 1157056, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2472, "q": 0, "reply": 5673, "reply_latency": { "avgcount": 5673, "avgtime": 0.002032721, "sum": 11.531628735 }, "request": 5675, "root_rbytes": 4599808, "root_rfiles": 835, "root_rsnaps": 0, "subtrees": 2, "traverse": 5687, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5676, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 10, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1345, "num_strays_delayed": 157, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2365, "strays_enqueued": 1020, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9151, "evadd": 9149, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 36, "avgtime": 0.018052075, "sum": 0.649874707 }, "rdpos": 4195196, "replayed": 2, "seg": 10, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 24170471 }, "mds_mem": { "cap": 2025, "cap+": 3202, "cap-": 1177, "dir": 45, "dir+": 86, "dir-": 41, "dn": 4556, "dn+": 5576, "dn-": 1020, "heap": 356572, "ino": 2194, "ino+": 3214, "ino-": 1020, "rss": 99548 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5754, "dispatch_server_request": 0, "handle_client_request": 5675, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2365, "avgtime": 0.001798431, "sum": 4.253289509 } }, "mds_sessions": { "average_load": 5218, "avg_session_uptime": 767, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5218 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 17592994, "buffer_anon_items": 7825, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 12850536, "mds_co_items": 196586, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 28, "omap_wr": 153, "op": 1202, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 32, "op_reply": 1202, "op_resend": 0, "op_rmw": 0, "op_send": 1202, "op_send_bytes": 21005447, "op_w": 1170, "oplen_avg": { "avgcount": 1202, "sum": 1306 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1043, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 14, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 42, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1020, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9767, "get_or_fail_fail": 0, "get_or_fail_success": 9767, "get_started": 0, "get_sum": 2153627, "max": 104857600, "put": 9256, "put_sum": 2025388, "take": 0, "take_sum": 0, "val": 128239, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 146, "put_sum": 21109399, "take": 1202, "take_sum": 21109399, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1202, "put_sum": 1202, "take": 1202, "take_sum": 1202, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1020, "get_or_fail_fail": 0, "get_or_fail_success": 1020, "get_started": 0, "get_sum": 103020, "max": 3758096384, "put": 6, "put_sum": 103020, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 9150, "get_or_fail_fail": 0, "get_or_fail_success": 9150, "get_started": 0, "get_sum": 19977025, "max": 3758096384, "put": 36, "put_sum": 19779425, "take": 0, "take_sum": 0, "val": 197600, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:44.501 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1345, 1020/2365 2020-11-20T12:09:45.502 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:45.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.736+0000 7f53f927e700 1 Processor -- start 2020-11-20T12:09:45.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.736+0000 7f53f927e700 1 -- start start 2020-11-20T12:09:45.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.737+0000 7f53f927e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f53f4065c90 0x7f53f40b5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:45.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.737+0000 7f53f927e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53f4066070 0x7f53f409a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:45.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.737+0000 7f53f927e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f414a940 0x7f53f4143960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:45.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.737+0000 7f53f927e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f4149f60 con 0x7f53f4065c90 2020-11-20T12:09:45.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.737+0000 7f53f927e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f414a200 con 0x7f53f4066070 2020-11-20T12:09:45.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.737+0000 7f53f927e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f414a4a0 con 0x7f53f414a940 2020-11-20T12:09:45.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f37fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53f4066070 0x7f53f409a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:45.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f37fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53f4066070 0x7f53f409a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44972/0 (socket says 172.21.15.187:44972) 2020-11-20T12:09:45.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f37fe700 1 -- 172.21.15.187:0/1424721286 learned_addr learned my addr 172.21.15.187:0/1424721286 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:45.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f3fff700 1 --2- 172.21.15.187:0/1424721286 >> v2:172.21.15.5:3300/0 conn(0x7f53f4065c90 0x7f53f40b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:45.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f8a7d700 1 --2- 172.21.15.187:0/1424721286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f414a940 0x7f53f4143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:45.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f37fe700 1 -- 172.21.15.187:0/1424721286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f414a940 msgr2=0x7f53f4143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:45.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f37fe700 1 --2- 172.21.15.187:0/1424721286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f414a940 0x7f53f4143960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:45.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f37fe700 1 -- 172.21.15.187:0/1424721286 >> v2:172.21.15.5:3300/0 conn(0x7f53f4065c90 msgr2=0x7f53f40b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:45.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f37fe700 1 --2- 172.21.15.187:0/1424721286 >> v2:172.21.15.5:3300/0 conn(0x7f53f4065c90 0x7f53f40b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:45.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f37fe700 1 -- 172.21.15.187:0/1424721286 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f414add0 con 0x7f53f4066070 2020-11-20T12:09:45.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.738+0000 7f53f3fff700 1 --2- 172.21.15.187:0/1424721286 >> v2:172.21.15.5:3300/0 conn(0x7f53f4065c90 0x7f53f40b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:45.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.739+0000 7f53f37fe700 1 --2- 172.21.15.187:0/1424721286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409a8e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f53f40b67a0 tx=0x7f53e8006eb0).ready entity=mon.1 client_cookie=f2c8fe2b882d863f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:45.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.739+0000 7f53f2ffd700 1 -- 172.21.15.187:0/1424721286 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8009960 con 0x7f53f4066070 2020-11-20T12:09:45.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.739+0000 7f53f2ffd700 1 -- 172.21.15.187:0/1424721286 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e8009ac0 con 0x7f53f4066070 2020-11-20T12:09:45.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.739+0000 7f53f2ffd700 1 -- 172.21.15.187:0/1424721286 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8009e40 con 0x7f53f4066070 2020-11-20T12:09:45.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.739+0000 7f53f927e700 1 -- 172.21.15.187:0/1424721286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 msgr2=0x7f53f409a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:45.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.739+0000 7f53f927e700 1 --2- 172.21.15.187:0/1424721286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409a8e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f53f40b67a0 tx=0x7f53e8006eb0).stop 2020-11-20T12:09:45.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.740+0000 7f53f927e700 1 -- 172.21.15.187:0/1424721286 shutdown_connections 2020-11-20T12:09:45.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.740+0000 7f53f927e700 1 --2- 172.21.15.187:0/1424721286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f414a940 0x7f53f4143960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:45.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.740+0000 7f53f927e700 1 --2- 172.21.15.187:0/1424721286 >> v2:172.21.15.5:3300/0 conn(0x7f53f4065c90 0x7f53f40b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:45.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.740+0000 7f53f927e700 1 --2- 172.21.15.187:0/1424721286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409a8e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:45.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.740+0000 7f53f927e700 1 -- 172.21.15.187:0/1424721286 >> 172.21.15.187:0/1424721286 conn(0x7f53f40b8280 msgr2=0x7f53f40653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:45.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.740+0000 7f53f927e700 1 -- 172.21.15.187:0/1424721286 shutdown_connections 2020-11-20T12:09:45.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.740+0000 7f53f927e700 1 -- 172.21.15.187:0/1424721286 wait complete. 2020-11-20T12:09:45.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.741+0000 7f53f927e700 1 Processor -- start 2020-11-20T12:09:45.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.741+0000 7f53f927e700 1 -- start start 2020-11-20T12:09:45.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f927e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f4065c90 0x7f53f415c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:45.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f927e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:45.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f927e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53f414a940 0x7f53f413ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:45.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f927e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f415cbb0 con 0x7f53f414a940 2020-11-20T12:09:45.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f927e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f40b39a0 con 0x7f53f4066070 2020-11-20T12:09:45.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f927e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f4126dd0 con 0x7f53f4065c90 2020-11-20T12:09:45.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:45.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44978/0 (socket says 172.21.15.187:44978) 2020-11-20T12:09:45.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f37fe700 1 -- 172.21.15.187:0/1696724392 learned_addr learned my addr 172.21.15.187:0/1696724392 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:45.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f3fff700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f4065c90 0x7f53f415c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:45.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.743+0000 7f53f8a7d700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53f414a940 0x7f53f413ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:45.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53f37fe700 1 -- 172.21.15.187:0/1696724392 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f4065c90 msgr2=0x7f53f415c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:45.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53f37fe700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f4065c90 0x7f53f415c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:45.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53f37fe700 1 -- 172.21.15.187:0/1696724392 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53f414a940 msgr2=0x7f53f413ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:45.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53f37fe700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53f414a940 0x7f53f413ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:45.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53f37fe700 1 -- 172.21.15.187:0/1696724392 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f4127070 con 0x7f53f4066070 2020-11-20T12:09:45.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53f37fe700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409c330 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f53e8006fd0 tx=0x7f53e8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:45.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53d3fff700 1 -- 172.21.15.187:0/1696724392 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8011e60 con 0x7f53f4066070 2020-11-20T12:09:45.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53d3fff700 1 -- 172.21.15.187:0/1696724392 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e801d5d0 con 0x7f53f4066070 2020-11-20T12:09:45.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f4127310 con 0x7f53f4066070 2020-11-20T12:09:45.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53d3fff700 1 -- 172.21.15.187:0/1696724392 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e801d950 con 0x7f53f4066070 2020-11-20T12:09:45.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.744+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f409c8a0 con 0x7f53f4066070 2020-11-20T12:09:45.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.746+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f40687c0 con 0x7f53f4066070 2020-11-20T12:09:45.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.746+0000 7f53d3fff700 1 -- 172.21.15.187:0/1696724392 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f53e8025030 con 0x7f53f4066070 2020-11-20T12:09:45.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.747+0000 7f53d3fff700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53cc056d20 0x7f53cc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:45.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.747+0000 7f53d3fff700 1 -- 172.21.15.187:0/1696724392 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f53e8071a80 con 0x7f53f4066070 2020-11-20T12:09:45.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.747+0000 7f53f3fff700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53cc056d20 0x7f53cc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:45.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.748+0000 7f53f3fff700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53cc056d20 0x7f53cc0591e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f53f40b0880 tx=0x7f53e4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:45.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:45.751+0000 7f53d3fff700 1 -- 172.21.15.187:0/1696724392 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f53e8046ea0 con 0x7f53f4066070 2020-11-20T12:09:46.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.044+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53f40641e0 con 0x7f53f4066070 2020-11-20T12:09:46.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.044+0000 7f53d3fff700 1 -- 172.21.15.187:0/1696724392 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f53e8023030 con 0x7f53f4066070 2020-11-20T12:09:46.048 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:46.048 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:46.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.046+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53cc056d20 msgr2=0x7f53cc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.046+0000 7f53f927e700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53cc056d20 0x7f53cc0591e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f53f40b0880 tx=0x7f53e4008040).stop 2020-11-20T12:09:46.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.046+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 msgr2=0x7f53f409c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.046+0000 7f53f927e700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409c330 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f53e8006fd0 tx=0x7f53e8015000).stop 2020-11-20T12:09:46.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.047+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 shutdown_connections 2020-11-20T12:09:46.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.047+0000 7f53f927e700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53cc056d20 0x7f53cc0591e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.047+0000 7f53f927e700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53f414a940 0x7f53f413ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.047+0000 7f53f927e700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53f4065c90 0x7f53f415c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.047+0000 7f53f927e700 1 --2- 172.21.15.187:0/1696724392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53f4066070 0x7f53f409c330 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.047+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 >> 172.21.15.187:0/1696724392 conn(0x7f53f40b8280 msgr2=0x7f53f40b8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:46.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.048+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 shutdown_connections 2020-11-20T12:09:46.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:46.048+0000 7f53f927e700 1 -- 172.21.15.187:0/1696724392 wait complete. 2020-11-20T12:09:46.054 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:46.080 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:46.362 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":878,"msgr_send_messages":878,"msgr_recv_bytes":385478,"msgr_send_bytes":12597218,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.244296239,"msgr_running_send_time":0.159639903,"msgr_running_recv_time":0.034665144,"msgr_running_fast_dispatch_time":0.035759152,"msgr_send_messages_queue_lat":{"avgcount":877,"sum":0.144326278,"avgtime":0.000164568},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9653,"msgr_send_messages":22310,"msgr_recv_bytes":2683940,"msgr_send_bytes":14714475,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":4.862234393,"msgr_running_send_time":4.131895498,"msgr_running_recv_time":2.436217234,"msgr_running_fast_dispatch_time":0.018674004,"msgr_send_messages_queue_lat":{"avgcount":22308,"sum":1.540169460,"avgtime":0.000069041},"msgr_handle_ack_lat":{"avgcount":18423,"sum":0.008912822,"avgtime":0.000000483}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":470,"msgr_send_messages":466,"msgr_recv_bytes":200586,"msgr_send_bytes":3042740,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.177865500,"msgr_running_send_time":0.067126372,"msgr_running_recv_time":0.033254389,"msgr_running_fast_dispatch_time":0.081355211,"msgr_send_messages_queue_lat":{"avgcount":465,"sum":0.112509449,"avgtime":0.000241955},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":198,"sum":3.505130446,"avgtime":0.017702679}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1352,"sum":0.720105927,"avgtime":0.000532622}},"mds":{"request":6348,"reply":6345,"reply_latency":{"avgcount":6345,"sum":13.737966000,"avgtime":0.002165164},"forward":0,"dir_fetch":12,"dir_commit":82,"dir_split":9,"dir_merge":1,"inodes":1694,"inodes_top":0,"inodes_bottom":1694,"inodes_pin_tail":0,"inodes_pinned":1694,"inodes_expired":1594,"inodes_with_caps":1175,"caps":1175,"subtrees":2,"traverse":6359,"traverse_hit":6348,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":1157056,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":589,"root_rbytes":3209216,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":83,"ceph_cap_op_grant":6626,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2841,"handle_client_caps_dirty":2758,"handle_client_cap_release":12,"process_request_cap_release":3145},"mds_cache":{"num_strays":1520,"num_strays_delayed":654,"num_strays_enqueuing":0,"strays_created":3037,"strays_enqueued":1517,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":11,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":10907,"evex":0,"evtrm":0,"ev":10909,"evexg":0,"evexd":0,"segadd":12,"segex":0,"segtrm":0,"seg":13,"segexg":0,"segexd":0,"expos":4194304,"wrpos":27792197,"rdpos":4195196,"jlat":{"avgcount":39,"sum":1.000481992,"avgtime":0.025653384},"replayed":2},"mds_mem":{"ino":1697,"ino+":3214,"ino-":1517,"dir":52,"dir+":94,"dir-":42,"dn":3137,"dn+":6248,"dn-":3111,"cap":1175,"cap+":3202,"cap-":2027,"rss":100340,"heap":356572},"mds_server":{"dispatch_client_request":6427,"dispatch_server_request":0,"handle_client_request":6348,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3037,"sum":6.459626774,"avgtime":0.002126976},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5738,"average_load":5738,"avg_session_uptime":767},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":17592994,"buffer_anon_items":7825,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12850536,"mds_co_items":196586,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1750,"op_send_bytes":24353269,"op_resend":0,"op_reply":1750,"oplen_avg":{"avgcount":1750,"sum":1862},"op":1750,"op_r":35,"op_w":1715,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":17,"osdop_write":48,"osdop_writefull":6,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":1573,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":169,"omap_rd":28,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":1517,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":10998,"get_sum":2425291,"get_or_fail_fail":0,"get_or_fail_success":10998,"take":0,"take_sum":0,"put":10998,"put_sum":2425291,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1750,"take_sum":24507418,"put":164,"put_sum":24507418,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1750,"take_sum":1750,"put":1750,"put_sum":1750,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1517,"get_sum":153217,"get_or_fail_fail":0,"get_or_fail_success":1517,"take":0,"take_sum":0,"put":9,"put_sum":153217,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":613344,"max":3758096384,"get_started":0,"get":10908,"get_sum":23600042,"get_or_fail_fail":0,"get_or_fail_success":10908,"take":0,"take_sum":0,"put":39,"put_sum":22986698,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:46.365 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 385478, "msgr_recv_messages": 878, "msgr_running_fast_dispatch_time": 0.035759152, "msgr_running_recv_time": 0.034665144, "msgr_running_send_time": 0.159639903, "msgr_running_total_time": 0.244296239, "msgr_send_bytes": 12597218, "msgr_send_messages": 878, "msgr_send_messages_queue_lat": { "avgcount": 877, "avgtime": 0.000164568, "sum": 0.144326278 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 18423, "avgtime": 4.83e-07, "sum": 0.008912822 }, "msgr_recv_bytes": 2683940, "msgr_recv_messages": 9653, "msgr_running_fast_dispatch_time": 0.018674004, "msgr_running_recv_time": 2.436217234, "msgr_running_send_time": 4.131895498, "msgr_running_total_time": 4.862234393, "msgr_send_bytes": 14714475, "msgr_send_messages": 22310, "msgr_send_messages_queue_lat": { "avgcount": 22308, "avgtime": 6.9041e-05, "sum": 1.54016946 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 200586, "msgr_recv_messages": 470, "msgr_running_fast_dispatch_time": 0.081355211, "msgr_running_recv_time": 0.033254389, "msgr_running_send_time": 0.067126372, "msgr_running_total_time": 0.1778655, "msgr_send_bytes": 3042740, "msgr_send_messages": 466, "msgr_send_messages_queue_lat": { "avgcount": 465, "avgtime": 0.000241955, "sum": 0.112509449 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 198, "avgtime": 0.017702679, "sum": 3.505130446 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1352, "avgtime": 0.000532622, "sum": 0.720105927 }, "queue_len": 0 }, "mds": { "caps": 1175, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6626, "ceph_cap_op_revoke": 83, "ceph_cap_op_trunc": 0, "dir_commit": 82, "dir_fetch": 12, "dir_merge": 1, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 12, "handle_client_caps": 2841, "handle_client_caps_dirty": 2758, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1694, "inodes_bottom": 1694, "inodes_expired": 1594, "inodes_pin_tail": 0, "inodes_pinned": 1694, "inodes_top": 0, "inodes_with_caps": 1175, "load_cent": 1157056, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3145, "q": 0, "reply": 6345, "reply_latency": { "avgcount": 6345, "avgtime": 0.002165164, "sum": 13.737966 }, "request": 6348, "root_rbytes": 3209216, "root_rfiles": 589, "root_rsnaps": 0, "subtrees": 2, "traverse": 6359, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6348, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 11, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1520, "num_strays_delayed": 654, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3037, "strays_enqueued": 1517, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10909, "evadd": 10907, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 39, "avgtime": 0.025653384, "sum": 1.000481992 }, "rdpos": 4195196, "replayed": 2, "seg": 13, "segadd": 12, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 27792197 }, "mds_mem": { "cap": 1175, "cap+": 3202, "cap-": 2027, "dir": 52, "dir+": 94, "dir-": 42, "dn": 3137, "dn+": 6248, "dn-": 3111, "heap": 356572, "ino": 1697, "ino+": 3214, "ino-": 1517, "rss": 100340 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6427, "dispatch_server_request": 0, "handle_client_request": 6348, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3037, "avgtime": 0.002126976, "sum": 6.459626774 } }, "mds_sessions": { "average_load": 5738, "avg_session_uptime": 767, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5738 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 17592994, "buffer_anon_items": 7825, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 12850536, "mds_co_items": 196586, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 28, "omap_wr": 169, "op": 1750, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 35, "op_reply": 1750, "op_resend": 0, "op_rmw": 0, "op_send": 1750, "op_send_bytes": 24353269, "op_w": 1715, "oplen_avg": { "avgcount": 1750, "sum": 1862 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1573, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 6, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1517, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10998, "get_or_fail_fail": 0, "get_or_fail_success": 10998, "get_started": 0, "get_sum": 2425291, "max": 104857600, "put": 10998, "put_sum": 2425291, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 164, "put_sum": 24507418, "take": 1750, "take_sum": 24507418, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1750, "put_sum": 1750, "take": 1750, "take_sum": 1750, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1517, "get_or_fail_fail": 0, "get_or_fail_success": 1517, "get_started": 0, "get_sum": 153217, "max": 3758096384, "put": 9, "put_sum": 153217, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 10908, "get_or_fail_fail": 0, "get_or_fail_success": 10908, "get_started": 0, "get_sum": 23600042, "max": 3758096384, "put": 39, "put_sum": 22986698, "take": 0, "take_sum": 0, "val": 613344, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:46.366 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1520, 1517/3037 2020-11-20T12:09:46.761 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:47.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.043+0000 7f5226651700 1 Processor -- start 2020-11-20T12:09:47.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.043+0000 7f5226651700 1 -- start start 2020-11-20T12:09:47.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.045+0000 7f5226651700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f52201485e0 0x7f52200bc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:47.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.045+0000 7f5226651700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52200b9a00 con 0x7f522014c4a0 2020-11-20T12:09:47.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.045+0000 7f5226651700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52200b5e80 con 0x7f52200bc790 2020-11-20T12:09:47.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.045+0000 7f5226651700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522014ca20 con 0x7f52201485e0 2020-11-20T12:09:47.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.046+0000 7f522564f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f52201485e0 0x7f52200bc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:47.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.046+0000 7f522564f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f52201485e0 0x7f52200bc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44984/0 (socket says 172.21.15.187:44984) 2020-11-20T12:09:47.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.046+0000 7f522564f700 1 -- 172.21.15.187:0/1030421423 learned_addr learned my addr 172.21.15.187:0/1030421423 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:47.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.047+0000 7f522564f700 1 -- 172.21.15.187:0/1030421423 >> v1:172.21.15.187:6789/0 conn(0x7f52200bc790 legacy=0x7f52200b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:47.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.047+0000 7f522564f700 1 -- 172.21.15.187:0/1030421423 >> v1:172.21.15.5:6789/0 conn(0x7f522014c4a0 legacy=0x7f52200b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:47.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.047+0000 7f522564f700 1 -- 172.21.15.187:0/1030421423 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522014ccc0 con 0x7f52201485e0 2020-11-20T12:09:47.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.047+0000 7f522564f700 1 --2- 172.21.15.187:0/1030421423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52201485e0 0x7f52200bc2a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f521c006fd0 tx=0x7f521c006eb0).ready entity=mon.1 client_cookie=30c53c5206e21b07 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:47.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.048+0000 7f520ffff700 1 -- 172.21.15.187:0/1030421423 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c009890 con 0x7f52201485e0 2020-11-20T12:09:47.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.048+0000 7f520ffff700 1 -- 172.21.15.187:0/1030421423 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c0099f0 con 0x7f52201485e0 2020-11-20T12:09:47.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.048+0000 7f520ffff700 1 -- 172.21.15.187:0/1030421423 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c009d70 con 0x7f52201485e0 2020-11-20T12:09:47.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.049+0000 7f5226651700 1 -- 172.21.15.187:0/1030421423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52201485e0 msgr2=0x7f52200bc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:47.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.049+0000 7f5226651700 1 --2- 172.21.15.187:0/1030421423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52201485e0 0x7f52200bc2a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f521c006fd0 tx=0x7f521c006eb0).stop 2020-11-20T12:09:47.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.049+0000 7f5226651700 1 -- 172.21.15.187:0/1030421423 shutdown_connections 2020-11-20T12:09:47.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.049+0000 7f5226651700 1 --2- 172.21.15.187:0/1030421423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52201485e0 0x7f52200bc2a0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:47.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.049+0000 7f5226651700 1 -- 172.21.15.187:0/1030421423 >> 172.21.15.187:0/1030421423 conn(0x7f52200bfbb0 msgr2=0x7f52200bbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:47.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.050+0000 7f5226651700 1 -- 172.21.15.187:0/1030421423 shutdown_connections 2020-11-20T12:09:47.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.051+0000 7f5226651700 1 -- 172.21.15.187:0/1030421423 wait complete. 2020-11-20T12:09:47.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.053+0000 7f5226651700 1 Processor -- start 2020-11-20T12:09:47.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.054+0000 7f5226651700 1 -- start start 2020-11-20T12:09:47.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.055+0000 7f5226651700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52200bc790 0x7f522013ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:47.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.055+0000 7f5226651700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f522014c4a0 0x7f52200b8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:47.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.055+0000 7f5226651700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52200a4270 0x7f5220160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:47.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.055+0000 7f5226651700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52200b6770 con 0x7f522014c4a0 2020-11-20T12:09:47.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.055+0000 7f5226651700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5220140400 con 0x7f52200a4270 2020-11-20T12:09:47.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.055+0000 7f5226651700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52200a3330 con 0x7f52200bc790 2020-11-20T12:09:47.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.055+0000 7f5225e50700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52200a4270 0x7f5220160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:47.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f5225e50700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52200a4270 0x7f5220160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44994/0 (socket says 172.21.15.187:44994) 2020-11-20T12:09:47.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f5225e50700 1 -- 172.21.15.187:0/1646102923 learned_addr learned my addr 172.21.15.187:0/1646102923 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:47.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f522564f700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52200bc790 0x7f522013ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:47.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f5224e4e700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f522014c4a0 0x7f52200b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:47.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f5225e50700 1 -- 172.21.15.187:0/1646102923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52200bc790 msgr2=0x7f522013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:47.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f5225e50700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52200bc790 0x7f522013ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:47.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f5225e50700 1 -- 172.21.15.187:0/1646102923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f522014c4a0 msgr2=0x7f52200b8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:47.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f5225e50700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f522014c4a0 0x7f52200b8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:47.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.056+0000 7f5225e50700 1 -- 172.21.15.187:0/1646102923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52200a35d0 con 0x7f52200a4270 2020-11-20T12:09:47.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.057+0000 7f5225e50700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52200a4270 0x7f5220160340 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f521800c810 tx=0x7f5218007f20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:47.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.057+0000 7f520d7fa700 1 -- 172.21.15.187:0/1646102923 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521800a650 con 0x7f52200a4270 2020-11-20T12:09:47.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.057+0000 7f520d7fa700 1 -- 172.21.15.187:0/1646102923 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521800a7b0 con 0x7f52200a4270 2020-11-20T12:09:47.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.057+0000 7f520d7fa700 1 -- 172.21.15.187:0/1646102923 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521800ab30 con 0x7f52200a4270 2020-11-20T12:09:47.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.058+0000 7f5226651700 1 -- 172.21.15.187:0/1646102923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52200a4a60 con 0x7f52200a4270 2020-11-20T12:09:47.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.058+0000 7f5226651700 1 -- 172.21.15.187:0/1646102923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52201472a0 con 0x7f52200a4270 2020-11-20T12:09:47.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.060+0000 7f51feffd700 1 -- 172.21.15.187:0/1646102923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f0004f90 con 0x7f52200a4270 2020-11-20T12:09:47.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.061+0000 7f520d7fa700 1 -- 172.21.15.187:0/1646102923 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5218010070 con 0x7f52200a4270 2020-11-20T12:09:47.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.062+0000 7f520d7fa700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51f8056d70 0x7f51f8059230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:47.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.062+0000 7f520d7fa700 1 -- 172.21.15.187:0/1646102923 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f521806fc40 con 0x7f52200a4270 2020-11-20T12:09:47.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.062+0000 7f522564f700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51f8056d70 0x7f51f8059230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:47.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.063+0000 7f522564f700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51f8056d70 0x7f51f8059230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f5220068fe0 tx=0x7f521c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:47.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.068+0000 7f520d7fa700 1 -- 172.21.15.187:0/1646102923 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f52180450c0 con 0x7f52200a4270 2020-11-20T12:09:47.367 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:47.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.436+0000 7f51feffd700 1 -- 172.21.15.187:0/1646102923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f0001660 con 0x7f52200a4270 2020-11-20T12:09:47.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.437+0000 7f520d7fa700 1 -- 172.21.15.187:0/1646102923 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f52180486e0 con 0x7f52200a4270 2020-11-20T12:09:47.441 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:47.441 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:47.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 -- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51f8056d70 msgr2=0x7f51f8059230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:47.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51f8056d70 0x7f51f8059230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f5220068fe0 tx=0x7f521c015000).stop 2020-11-20T12:09:47.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 -- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52200a4270 msgr2=0x7f5220160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:47.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52200a4270 0x7f5220160340 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f521800c810 tx=0x7f5218007f20).stop 2020-11-20T12:09:47.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 -- 172.21.15.187:0/1646102923 shutdown_connections 2020-11-20T12:09:47.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51f8056d70 0x7f51f8059230 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:47.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52200a4270 0x7f5220160340 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:47.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52200bc790 0x7f522013ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:47.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 --2- 172.21.15.187:0/1646102923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f522014c4a0 0x7f52200b8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:47.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 -- 172.21.15.187:0/1646102923 >> 172.21.15.187:0/1646102923 conn(0x7f52200bfbb0 msgr2=0x7f522009ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:47.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.439+0000 7f51feffd700 1 -- 172.21.15.187:0/1646102923 shutdown_connections 2020-11-20T12:09:47.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.440+0000 7f51feffd700 1 -- 172.21.15.187:0/1646102923 wait complete. 2020-11-20T12:09:47.449 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:47.462 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:09:47.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.560+0000 7f955cca7700 1 Processor -- start 2020-11-20T12:09:48.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.560+0000 7f955cca7700 1 -- start start 2020-11-20T12:09:48.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.561+0000 7f955cca7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95580bc460 0x7f955814c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.561+0000 7f955cca7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95580b96f0 0x7f955809ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.561+0000 7f955cca7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955814cc10 con 0x7f95580bc840 2020-11-20T12:09:48.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.561+0000 7f955cca7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955814c9f0 con 0x7f95580b96f0 2020-11-20T12:09:48.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.561+0000 7f955cca7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955814ceb0 con 0x7f95580bc460 2020-11-20T12:09:48.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.561+0000 7f9556ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f95580bc840 0x7f95581485e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41606/0 (socket says 172.21.15.187:41606) 2020-11-20T12:09:48.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.561+0000 7f9556ffd700 1 -- 172.21.15.187:0/433795068 learned_addr learned my addr 172.21.15.187:0/433795068 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:48.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f9557fff700 1 --2- 172.21.15.187:0/433795068 >> v2:172.21.15.187:3300/0 conn(0x7f95580b96f0 0x7f955809ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f95577fe700 1 --2- 172.21.15.187:0/433795068 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95580bc460 0x7f955814c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f95567fc700 1 -- 172.21.15.187:0/433795068 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920429293 0 0) 0x7f955814cc10 con 0x7f95580bc840 2020-11-20T12:09:48.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f95567fc700 1 -- 172.21.15.187:0/433795068 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9544003240 con 0x7f95580bc840 2020-11-20T12:09:48.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f9557fff700 1 -- 172.21.15.187:0/433795068 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95580bc460 msgr2=0x7f955814c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:48.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f9557fff700 1 --2- 172.21.15.187:0/433795068 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95580bc460 0x7f955814c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f9557fff700 1 -- 172.21.15.187:0/433795068 >> v1:172.21.15.187:6789/0 conn(0x7f95580bc840 legacy=0x7f95581485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f9557fff700 1 -- 172.21.15.187:0/433795068 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955814d150 con 0x7f95580b96f0 2020-11-20T12:09:48.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.562+0000 7f9557fff700 1 --2- 172.21.15.187:0/433795068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580b96f0 0x7f955809ff90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7f9550009c90 tx=0x7f95500091d0).ready entity=mon.1 client_cookie=781234c6064656a4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:48.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f95567fc700 1 -- 172.21.15.187:0/433795068 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955000ba70 con 0x7f95580b96f0 2020-11-20T12:09:48.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f95567fc700 1 -- 172.21.15.187:0/433795068 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955000bbd0 con 0x7f95580b96f0 2020-11-20T12:09:48.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f95567fc700 1 -- 172.21.15.187:0/433795068 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9550011400 con 0x7f95580b96f0 2020-11-20T12:09:48.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f955cca7700 1 -- 172.21.15.187:0/433795068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580b96f0 msgr2=0x7f955809ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f955cca7700 1 --2- 172.21.15.187:0/433795068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580b96f0 0x7f955809ff90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7f9550009c90 tx=0x7f95500091d0).stop 2020-11-20T12:09:48.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f955cca7700 1 -- 172.21.15.187:0/433795068 shutdown_connections 2020-11-20T12:09:48.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f955cca7700 1 --2- 172.21.15.187:0/433795068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580b96f0 0x7f955809ff90 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f955cca7700 1 --2- 172.21.15.187:0/433795068 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95580bc460 0x7f955814c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f955cca7700 1 -- 172.21.15.187:0/433795068 >> 172.21.15.187:0/433795068 conn(0x7f95580bfb90 msgr2=0x7f95580bbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:48.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.563+0000 7f955cca7700 1 -- 172.21.15.187:0/433795068 shutdown_connections 2020-11-20T12:09:48.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.564+0000 7f955cca7700 1 -- 172.21.15.187:0/433795068 wait complete. 2020-11-20T12:09:48.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.565+0000 7f955cca7700 1 Processor -- start 2020-11-20T12:09:48.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.565+0000 7f955cca7700 1 -- start start 2020-11-20T12:09:48.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f955cca7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95580b96f0 0x7f955814a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f955cca7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580bc460 0x7f955814d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f955cca7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f955814b1e0 0x7f95580b87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f955cca7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95580b6c60 con 0x7f95580b96f0 2020-11-20T12:09:48.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f955cca7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95580b6f00 con 0x7f95580bc460 2020-11-20T12:09:48.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f955cca7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95580b71a0 con 0x7f955814b1e0 2020-11-20T12:09:48.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f9556ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580bc460 0x7f955814d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f9556ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580bc460 0x7f955814d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45006/0 (socket says 172.21.15.187:45006) 2020-11-20T12:09:48.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.566+0000 7f9556ffd700 1 -- 172.21.15.187:0/740301479 learned_addr learned my addr 172.21.15.187:0/740301479 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:48.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f95577fe700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95580b96f0 0x7f955814a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f9556ffd700 1 -- 172.21.15.187:0/740301479 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f955814b1e0 msgr2=0x7f95580b87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f9556ffd700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f955814b1e0 0x7f95580b87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:48.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f9556ffd700 1 -- 172.21.15.187:0/740301479 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95580b96f0 msgr2=0x7f955814a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f9556ffd700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95580b96f0 0x7f955814a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f9556ffd700 1 -- 172.21.15.187:0/740301479 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955814da90 con 0x7f95580bc460 2020-11-20T12:09:48.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f9556ffd700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580bc460 0x7f955814d5a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f954c00c6e0 tx=0x7f954c0079e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:48.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f95377fe700 1 -- 172.21.15.187:0/740301479 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c0072e0 con 0x7f95580bc460 2020-11-20T12:09:48.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f95377fe700 1 -- 172.21.15.187:0/740301479 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954c007460 con 0x7f95580bc460 2020-11-20T12:09:48.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f95377fe700 1 -- 172.21.15.187:0/740301479 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c005840 con 0x7f95580bc460 2020-11-20T12:09:48.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95580b8ca0 con 0x7f95580bc460 2020-11-20T12:09:48.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.567+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9558137c50 con 0x7f95580bc460 2020-11-20T12:09:48.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.569+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9558068630 con 0x7f95580bc460 2020-11-20T12:09:48.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.569+0000 7f95377fe700 1 -- 172.21.15.187:0/740301479 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f954c010070 con 0x7f95580bc460 2020-11-20T12:09:48.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.570+0000 7f95377fe700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9530056d50 0x7f9530059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.570+0000 7f95377fe700 1 -- 172.21.15.187:0/740301479 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f954c06f550 con 0x7f95580bc460 2020-11-20T12:09:48.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.570+0000 7f95577fe700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9530056d50 0x7f9530059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.571+0000 7f95577fe700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9530056d50 0x7f9530059210 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9548009650 tx=0x7f9548008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:48.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.573+0000 7f95377fe700 1 -- 172.21.15.187:0/740301479 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f954c044950 con 0x7f95580bc460 2020-11-20T12:09:48.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.884+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9558080210 con 0x7f95580bc460 2020-11-20T12:09:48.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.886+0000 7f95377fe700 1 -- 172.21.15.187:0/740301479 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f954c018070 con 0x7f95580bc460 2020-11-20T12:09:48.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.891+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9530056d50 msgr2=0x7f9530059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.891+0000 7f955cca7700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9530056d50 0x7f9530059210 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9548009650 tx=0x7f9548008040).stop 2020-11-20T12:09:48.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.891+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580bc460 msgr2=0x7f955814d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.891+0000 7f955cca7700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580bc460 0x7f955814d5a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f954c00c6e0 tx=0x7f954c0079e0).stop 2020-11-20T12:09:48.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.892+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 shutdown_connections 2020-11-20T12:09:48.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.892+0000 7f955cca7700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9530056d50 0x7f9530059210 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.892+0000 7f955cca7700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95580b96f0 0x7f955814a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.892+0000 7f955cca7700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f955814b1e0 0x7f95580b87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:48.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.892+0000 7f955cca7700 1 --2- 172.21.15.187:0/740301479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95580bc460 0x7f955814d5a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.892+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 >> 172.21.15.187:0/740301479 conn(0x7f95580bfb90 msgr2=0x7f95580a1160 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:48.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.892+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 shutdown_connections 2020-11-20T12:09:48.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:47.892+0000 7f955cca7700 1 -- 172.21.15.187:0/740301479 wait complete. 2020-11-20T12:09:48.227 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:48.227 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:48.227 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:48.228 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:48.229 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:09:48.229 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:09:48.492 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1728,"msgr_send_messages":1728,"msgr_recv_bytes":711772,"msgr_send_bytes":13196111,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.310277420,"msgr_running_send_time":0.186421481,"msgr_running_recv_time":0.051583180,"msgr_running_fast_dispatch_time":0.053883200,"msgr_send_messages_queue_lat":{"avgcount":1727,"sum":1.198759830,"avgtime":0.000694128},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10270,"msgr_send_messages":23610,"msgr_recv_bytes":2865595,"msgr_send_bytes":15241632,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":5.055876704,"msgr_running_send_time":4.285038953,"msgr_running_recv_time":2.459161042,"msgr_running_fast_dispatch_time":0.029502082,"msgr_send_messages_queue_lat":{"avgcount":23608,"sum":1.790395786,"avgtime":0.000075838},"msgr_handle_ack_lat":{"avgcount":18937,"sum":0.009153746,"avgtime":0.000000483}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":909,"msgr_send_messages":905,"msgr_recv_bytes":334474,"msgr_send_bytes":4701625,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.223302374,"msgr_running_send_time":0.088535210,"msgr_running_recv_time":0.042214058,"msgr_running_fast_dispatch_time":0.094041448,"msgr_send_messages_queue_lat":{"avgcount":904,"sum":0.536593344,"avgtime":0.000593576},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":245,"sum":3.899690486,"avgtime":0.015917104}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1986,"sum":0.826503969,"avgtime":0.000416165}},"mds":{"request":6511,"reply":6509,"reply_latency":{"avgcount":6509,"sum":14.094907452,"avgtime":0.002165448},"forward":0,"dir_fetch":12,"dir_commit":100,"dir_split":11,"dir_merge":1,"inodes":1014,"inodes_top":0,"inodes_bottom":1014,"inodes_pin_tail":0,"inodes_pinned":1014,"inodes_expired":1594,"inodes_with_caps":1005,"caps":1005,"subtrees":2,"traverse":6523,"traverse_hit":6512,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":1157056,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":85,"ceph_cap_op_grant":6969,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2843,"handle_client_caps_dirty":2758,"handle_client_cap_release":13,"process_request_cap_release":3309},"mds_cache":{"num_strays":1004,"num_strays_delayed":694,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2197,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":12,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":11774,"evex":11773,"evtrm":11773,"ev":11776,"evexg":0,"evexd":11773,"segadd":14,"segex":14,"segtrm":14,"seg":15,"segexg":0,"segexd":14,"expos":28766476,"wrpos":28772150,"rdpos":4195196,"jlat":{"avgcount":44,"sum":1.260527568,"avgtime":0.028648353},"replayed":2},"mds_mem":{"ino":1017,"ino+":3214,"ino-":2197,"dir":59,"dir+":102,"dir-":43,"dn":1015,"dn+":6412,"dn-":5397,"cap":1005,"cap+":3202,"cap-":2197,"rss":100776,"heap":356572},"mds_server":{"dispatch_client_request":6593,"dispatch_server_request":0,"handle_client_request":6511,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.205110729,"avgtime":0.205110729},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.611457497,"avgtime":0.002066080},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":767},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":17592994,"buffer_anon_items":7825,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12850536,"mds_co_items":196586,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3485,"op_send_bytes":26756110,"op_resend":0,"op_reply":3485,"oplen_avg":{"avgcount":3485,"sum":5649},"op":3485,"op_r":38,"op_w":3447,"op_rmw":0,"op_pg":0,"osdop_stat":10,"osdop_create":1012,"osdop_read":20,"osdop_write":56,"osdop_writefull":11,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":3,"osdop_delete":2258,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":2024,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":205,"omap_rd":28,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":2197,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12903,"get_sum":2898081,"get_or_fail_fail":0,"get_or_fail_success":12903,"take":0,"take_sum":0,"put":12903,"put_sum":2898081,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3485,"take_sum":26978939,"put":1212,"put_sum":26978939,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3485,"take_sum":3485,"put":3485,"put_sum":3485,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2197,"get_sum":221897,"get_or_fail_fail":0,"get_or_fail_success":2197,"take":0,"take_sum":0,"put":12,"put_sum":221897,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":189,"max":3758096384,"get_started":0,"get":11774,"get_sum":24576690,"get_or_fail_fail":0,"get_or_fail_success":11774,"take":0,"take_sum":0,"put":44,"put_sum":24576501,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:48.494 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 711772, "msgr_recv_messages": 1728, "msgr_running_fast_dispatch_time": 0.0538832, "msgr_running_recv_time": 0.05158318, "msgr_running_send_time": 0.186421481, "msgr_running_total_time": 0.31027742, "msgr_send_bytes": 13196111, "msgr_send_messages": 1728, "msgr_send_messages_queue_lat": { "avgcount": 1727, "avgtime": 0.000694128, "sum": 1.19875983 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 18937, "avgtime": 4.83e-07, "sum": 0.009153746 }, "msgr_recv_bytes": 2865595, "msgr_recv_messages": 10270, "msgr_running_fast_dispatch_time": 0.029502082, "msgr_running_recv_time": 2.459161042, "msgr_running_send_time": 4.285038953, "msgr_running_total_time": 5.055876704, "msgr_send_bytes": 15241632, "msgr_send_messages": 23610, "msgr_send_messages_queue_lat": { "avgcount": 23608, "avgtime": 7.5838e-05, "sum": 1.790395786 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 334474, "msgr_recv_messages": 909, "msgr_running_fast_dispatch_time": 0.094041448, "msgr_running_recv_time": 0.042214058, "msgr_running_send_time": 0.08853521, "msgr_running_total_time": 0.223302374, "msgr_send_bytes": 4701625, "msgr_send_messages": 905, "msgr_send_messages_queue_lat": { "avgcount": 904, "avgtime": 0.000593576, "sum": 0.536593344 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 245, "avgtime": 0.015917104, "sum": 3.899690486 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1986, "avgtime": 0.000416165, "sum": 0.826503969 }, "queue_len": 0 }, "mds": { "caps": 1005, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6969, "ceph_cap_op_revoke": 85, "ceph_cap_op_trunc": 0, "dir_commit": 100, "dir_fetch": 12, "dir_merge": 1, "dir_split": 11, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 13, "handle_client_caps": 2843, "handle_client_caps_dirty": 2758, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1014, "inodes_bottom": 1014, "inodes_expired": 1594, "inodes_pin_tail": 0, "inodes_pinned": 1014, "inodes_top": 0, "inodes_with_caps": 1005, "load_cent": 1157056, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3309, "q": 0, "reply": 6509, "reply_latency": { "avgcount": 6509, "avgtime": 0.002165448, "sum": 14.094907452 }, "request": 6511, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 6523, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6512, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 12, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1004, "num_strays_delayed": 694, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2197, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11776, "evadd": 11774, "evex": 11773, "evexd": 11773, "evexg": 0, "evtrm": 11773, "expos": 28766476, "jlat": { "avgcount": 44, "avgtime": 0.028648353, "sum": 1.260527568 }, "rdpos": 4195196, "replayed": 2, "seg": 15, "segadd": 14, "segex": 14, "segexd": 14, "segexg": 0, "segtrm": 14, "wrpos": 28772150 }, "mds_mem": { "cap": 1005, "cap+": 3202, "cap-": 2197, "dir": 59, "dir+": 102, "dir-": 43, "dn": 1015, "dn+": 6412, "dn-": 5397, "heap": 356572, "ino": 1017, "ino+": 3214, "ino-": 2197, "rss": 100776 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6593, "dispatch_server_request": 0, "handle_client_request": 6511, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.205110729, "sum": 0.205110729 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.00206608, "sum": 6.611457497 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 767, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 17592994, "buffer_anon_items": 7825, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 12850536, "mds_co_items": 196586, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 28, "omap_wr": 205, "op": 3485, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 38, "op_reply": 3485, "op_resend": 0, "op_rmw": 0, "op_send": 3485, "op_send_bytes": 26756110, "op_w": 3447, "oplen_avg": { "avgcount": 3485, "sum": 5649 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1012, "osdop_delete": 2258, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 20, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2024, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 10, "osdop_truncate": 3, "osdop_watch": 0, "osdop_write": 56, "osdop_writefull": 11, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 2197, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12903, "get_or_fail_fail": 0, "get_or_fail_success": 12903, "get_started": 0, "get_sum": 2898081, "max": 104857600, "put": 12903, "put_sum": 2898081, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1212, "put_sum": 26978939, "take": 3485, "take_sum": 26978939, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3485, "put_sum": 3485, "take": 3485, "take_sum": 3485, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2197, "get_or_fail_fail": 0, "get_or_fail_success": 2197, "get_started": 0, "get_sum": 221897, "max": 3758096384, "put": 12, "put_sum": 221897, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 11774, "get_or_fail_fail": 0, "get_or_fail_success": 11774, "get_started": 0, "get_sum": 24576690, "max": 3758096384, "put": 44, "put_sum": 24576501, "take": 0, "take_sum": 0, "val": 189, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:48.494 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1004, 2197/3201 2020-11-20T12:09:49.496 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:49.744 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:49.753 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:49.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.751+0000 7f24f7fff700 1 Processor -- start 2020-11-20T12:09:49.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.751+0000 7f24f7fff700 1 -- start start 2020-11-20T12:09:49.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24f7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f24f8065ad0 0x7f24f8065eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:49.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24f7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f24f80b16a0 0x7f24f80afc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:49.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24f7fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f80b0fa0 con 0x7f24f8142950 2020-11-20T12:09:49.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24f7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f8149ac0 con 0x7f24f8065ad0 2020-11-20T12:09:49.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24f7fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f8149d60 con 0x7f24f80b16a0 2020-11-20T12:09:49.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24f77fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f24f8142950 0x7f24f8142db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41622/0 (socket says 172.21.15.187:41622) 2020-11-20T12:09:49.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24f77fe700 1 -- 172.21.15.187:0/4287364131 learned_addr learned my addr 172.21.15.187:0/4287364131 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:49.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24ee7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f24f80b16a0 0x7f24f80afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:49.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.753+0000 7f24f6ffd700 1 --2- 172.21.15.187:0/4287364131 >> v2:172.21.15.5:3300/0 conn(0x7f24f8065ad0 0x7f24f8065eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:49.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24f67fc700 1 -- 172.21.15.187:0/4287364131 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640096733 0 0) 0x7f24f80b0fa0 con 0x7f24f8142950 2020-11-20T12:09:49.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24f67fc700 1 -- 172.21.15.187:0/4287364131 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d8003240 con 0x7f24f8142950 2020-11-20T12:09:49.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24ee7fc700 1 -- 172.21.15.187:0/4287364131 >> v2:172.21.15.5:3300/0 conn(0x7f24f8065ad0 msgr2=0x7f24f8065eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:49.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24ee7fc700 1 --2- 172.21.15.187:0/4287364131 >> v2:172.21.15.5:3300/0 conn(0x7f24f8065ad0 0x7f24f8065eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:49.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24ee7fc700 1 -- 172.21.15.187:0/4287364131 >> v1:172.21.15.187:6789/0 conn(0x7f24f8142950 legacy=0x7f24f8142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:49.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24ee7fc700 1 -- 172.21.15.187:0/4287364131 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f81459c0 con 0x7f24f80b16a0 2020-11-20T12:09:49.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24f6ffd700 1 --2- 172.21.15.187:0/4287364131 >> v2:172.21.15.5:3300/0 conn(0x7f24f8065ad0 0x7f24f8065eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:49.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24ee7fc700 1 --2- 172.21.15.187:0/4287364131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f80b16a0 0x7f24f80afc70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f24f80b53c0 tx=0x7f24e0006df0).ready entity=mon.1 client_cookie=7f906fdec9013e8e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:49.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24f67fc700 1 -- 172.21.15.187:0/4287364131 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0009840 con 0x7f24f80b16a0 2020-11-20T12:09:49.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24f67fc700 1 -- 172.21.15.187:0/4287364131 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e00099a0 con 0x7f24f80b16a0 2020-11-20T12:09:49.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.754+0000 7f24f67fc700 1 -- 172.21.15.187:0/4287364131 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0009d20 con 0x7f24f80b16a0 2020-11-20T12:09:49.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.755+0000 7f24f7fff700 1 -- 172.21.15.187:0/4287364131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f80b16a0 msgr2=0x7f24f80afc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:49.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.755+0000 7f24f7fff700 1 --2- 172.21.15.187:0/4287364131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f80b16a0 0x7f24f80afc70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f24f80b53c0 tx=0x7f24e0006df0).stop 2020-11-20T12:09:49.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.755+0000 7f24f7fff700 1 -- 172.21.15.187:0/4287364131 shutdown_connections 2020-11-20T12:09:49.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.755+0000 7f24f7fff700 1 --2- 172.21.15.187:0/4287364131 >> v2:172.21.15.5:3300/0 conn(0x7f24f8065ad0 0x7f24f8065eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:49.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.755+0000 7f24f7fff700 1 --2- 172.21.15.187:0/4287364131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f80b16a0 0x7f24f80afc70 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:49.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.755+0000 7f24f7fff700 1 -- 172.21.15.187:0/4287364131 >> 172.21.15.187:0/4287364131 conn(0x7f24f80b8280 msgr2=0x7f24f80653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:49.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.755+0000 7f24f7fff700 1 -- 172.21.15.187:0/4287364131 shutdown_connections 2020-11-20T12:09:49.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.755+0000 7f24f7fff700 1 -- 172.21.15.187:0/4287364131 wait complete. 2020-11-20T12:09:49.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.757+0000 7f24f7fff700 1 Processor -- start 2020-11-20T12:09:49.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.757+0000 7f24f7fff700 1 -- start start 2020-11-20T12:09:49.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.758+0000 7f24f7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f24f8065ad0 0x7f24f80b4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:49.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f80b16a0 0x7f24f8158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:49.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f8142950 0x7f24f80b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:49.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80b2fd0 con 0x7f24f80b16a0 2020-11-20T12:09:49.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80b4840 con 0x7f24f8142950 2020-11-20T12:09:49.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80b4ae0 con 0x7f24f8065ad0 2020-11-20T12:09:49.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f8142950 0x7f24f80b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:49.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f8142950 0x7f24f80b2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45022/0 (socket says 172.21.15.187:45022) 2020-11-20T12:09:49.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f77fe700 1 -- 172.21.15.187:0/3230716887 learned_addr learned my addr 172.21.15.187:0/3230716887 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:49.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24ee7fc700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f80b16a0 0x7f24f8158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:49.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.759+0000 7f24f6ffd700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f24f8065ad0 0x7f24f80b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:49.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24f77fe700 1 -- 172.21.15.187:0/3230716887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f24f8065ad0 msgr2=0x7f24f80b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:49.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24f77fe700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f24f8065ad0 0x7f24f80b4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:49.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24f77fe700 1 -- 172.21.15.187:0/3230716887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f80b16a0 msgr2=0x7f24f8158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:49.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24f77fe700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f80b16a0 0x7f24f8158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:49.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24f77fe700 1 -- 172.21.15.187:0/3230716887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f80b4d80 con 0x7f24f8142950 2020-11-20T12:09:49.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24f77fe700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f8142950 0x7f24f80b2ae0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f24f0001420 tx=0x7f24f0007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:49.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24ef7fe700 1 -- 172.21.15.187:0/3230716887 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0005b80 con 0x7f24f8142950 2020-11-20T12:09:49.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24ef7fe700 1 -- 172.21.15.187:0/3230716887 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0005ce0 con 0x7f24f8142950 2020-11-20T12:09:49.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24ef7fe700 1 -- 172.21.15.187:0/3230716887 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0007320 con 0x7f24f8142950 2020-11-20T12:09:49.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f8062010 con 0x7f24f8142950 2020-11-20T12:09:49.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.760+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f8062480 con 0x7f24f8142950 2020-11-20T12:09:49.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.762+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f8068ef0 con 0x7f24f8142950 2020-11-20T12:09:49.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.762+0000 7f24ef7fe700 1 -- 172.21.15.187:0/3230716887 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f24f0010030 con 0x7f24f8142950 2020-11-20T12:09:49.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.763+0000 7f24ef7fe700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24d4056d50 0x7f24d4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:49.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.763+0000 7f24ef7fe700 1 -- 172.21.15.187:0/3230716887 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f24f006ffc0 con 0x7f24f8142950 2020-11-20T12:09:49.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.763+0000 7f24f6ffd700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24d4056d50 0x7f24d4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:49.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.764+0000 7f24f6ffd700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24d4056d50 0x7f24d4059210 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f24e8006bd0 tx=0x7f24e8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:49.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:49.768+0000 7f24ef7fe700 1 -- 172.21.15.187:0/3230716887 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f24f00452b0 con 0x7f24f8142950 2020-11-20T12:09:50.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.021+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24f8044f60 con 0x7f24f8142950 2020-11-20T12:09:50.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.022+0000 7f24ef7fe700 1 -- 172.21.15.187:0/3230716887 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f24f00488d0 con 0x7f24f8142950 2020-11-20T12:09:50.026 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:50.026 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:50.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.027+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24d4056d50 msgr2=0x7f24d4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:50.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.027+0000 7f24f7fff700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24d4056d50 0x7f24d4059210 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f24e8006bd0 tx=0x7f24e8008040).stop 2020-11-20T12:09:50.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.027+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f8142950 msgr2=0x7f24f80b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:50.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.027+0000 7f24f7fff700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f8142950 0x7f24f80b2ae0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f24f0001420 tx=0x7f24f0007e20).stop 2020-11-20T12:09:50.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.028+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 shutdown_connections 2020-11-20T12:09:50.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.028+0000 7f24f7fff700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f24d4056d50 0x7f24d4059210 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.028+0000 7f24f7fff700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24f8142950 0x7f24f80b2ae0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.028+0000 7f24f7fff700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f24f8065ad0 0x7f24f80b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.028+0000 7f24f7fff700 1 --2- 172.21.15.187:0/3230716887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24f80b16a0 0x7f24f8158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.028+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 >> 172.21.15.187:0/3230716887 conn(0x7f24f80b8280 msgr2=0x7f24f8141f10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:50.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.030+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 shutdown_connections 2020-11-20T12:09:50.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:50.030+0000 7f24f7fff700 1 -- 172.21.15.187:0/3230716887 wait complete. 2020-11-20T12:09:50.036 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:50.064 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:50.467 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1728,"msgr_send_messages":1728,"msgr_recv_bytes":711772,"msgr_send_bytes":13196111,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.310277420,"msgr_running_send_time":0.186421481,"msgr_running_recv_time":0.051583180,"msgr_running_fast_dispatch_time":0.053883200,"msgr_send_messages_queue_lat":{"avgcount":1727,"sum":1.198759830,"avgtime":0.000694128},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11234,"msgr_send_messages":23905,"msgr_recv_bytes":3176127,"msgr_send_bytes":15337873,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":5.127790835,"msgr_running_send_time":4.315032508,"msgr_running_recv_time":3.807383221,"msgr_running_fast_dispatch_time":0.029639685,"msgr_send_messages_queue_lat":{"avgcount":23903,"sum":2.759939750,"avgtime":0.000115464},"msgr_handle_ack_lat":{"avgcount":19073,"sum":0.009199536,"avgtime":0.000000482}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":913,"msgr_send_messages":909,"msgr_recv_bytes":335458,"msgr_send_bytes":5291036,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.225723245,"msgr_running_send_time":0.090100996,"msgr_running_recv_time":0.042378267,"msgr_running_fast_dispatch_time":0.094644242,"msgr_send_messages_queue_lat":{"avgcount":908,"sum":0.537602953,"avgtime":0.000592073},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":251,"sum":4.053697700,"avgtime":0.016150190}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1986,"sum":0.826503969,"avgtime":0.000416165}},"mds":{"request":6511,"reply":6509,"reply_latency":{"avgcount":6509,"sum":14.094907452,"avgtime":0.002165448},"forward":0,"dir_fetch":12,"dir_commit":100,"dir_split":11,"dir_merge":1,"inodes":1014,"inodes_top":0,"inodes_bottom":1014,"inodes_pin_tail":0,"inodes_pinned":1014,"inodes_expired":1594,"inodes_with_caps":1005,"caps":1005,"subtrees":2,"traverse":6523,"traverse_hit":6512,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":1157056,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":85,"ceph_cap_op_grant":7021,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3246,"handle_client_caps_dirty":3000,"handle_client_cap_release":15,"process_request_cap_release":3309},"mds_cache":{"num_strays":1004,"num_strays_delayed":107,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2197,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":12,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":12178,"evex":11773,"evtrm":11773,"ev":407,"evexg":0,"evexd":11773,"segadd":15,"segex":14,"segtrm":14,"seg":2,"segexg":0,"segexd":14,"expos":28766476,"wrpos":29575555,"rdpos":4195196,"jlat":{"avgcount":47,"sum":1.373638783,"avgtime":0.029226357},"replayed":2},"mds_mem":{"ino":1017,"ino+":3214,"ino-":2197,"dir":59,"dir+":102,"dir-":43,"dn":1015,"dn+":6412,"dn-":5397,"cap":931,"cap+":3202,"cap-":2271,"rss":100776,"heap":356572},"mds_server":{"dispatch_client_request":6593,"dispatch_server_request":0,"handle_client_request":6511,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.205110729,"avgtime":0.205110729},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.611457497,"avgtime":0.002066080},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":772},"mempool":{"bloom_filter_bytes":120,"bloom_filter_items":120,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":212138,"buffer_anon_items":1104,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":3438678,"mds_co_items":64520,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3489,"op_send_bytes":27344277,"op_resend":0,"op_reply":3489,"oplen_avg":{"avgcount":3489,"sum":5653},"op":3489,"op_r":38,"op_w":3451,"op_rmw":0,"op_pg":0,"osdop_stat":10,"osdop_create":1012,"osdop_read":20,"osdop_write":59,"osdop_writefull":11,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":3,"osdop_delete":2259,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":2024,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":205,"omap_rd":28,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":2197,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":139988,"max":104857600,"get_started":0,"get":13874,"get_sum":3138609,"get_or_fail_fail":0,"get_or_fail_success":13874,"take":0,"take_sum":0,"put":13315,"put_sum":2998621,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3489,"take_sum":27567106,"put":1215,"put_sum":27567106,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3489,"take_sum":3489,"put":3489,"put_sum":3489,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2197,"get_sum":221897,"get_or_fail_fail":0,"get_or_fail_success":2197,"take":0,"take_sum":0,"put":12,"put_sum":221897,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":215427,"max":3758096384,"get_started":0,"get":12178,"get_sum":25380095,"get_or_fail_fail":0,"get_or_fail_success":12178,"take":0,"take_sum":0,"put":47,"put_sum":25164668,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:50.469 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 711772, "msgr_recv_messages": 1728, "msgr_running_fast_dispatch_time": 0.0538832, "msgr_running_recv_time": 0.05158318, "msgr_running_send_time": 0.186421481, "msgr_running_total_time": 0.31027742, "msgr_send_bytes": 13196111, "msgr_send_messages": 1728, "msgr_send_messages_queue_lat": { "avgcount": 1727, "avgtime": 0.000694128, "sum": 1.19875983 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19073, "avgtime": 4.82e-07, "sum": 0.009199536 }, "msgr_recv_bytes": 3176127, "msgr_recv_messages": 11234, "msgr_running_fast_dispatch_time": 0.029639685, "msgr_running_recv_time": 3.807383221, "msgr_running_send_time": 4.315032508, "msgr_running_total_time": 5.127790835, "msgr_send_bytes": 15337873, "msgr_send_messages": 23905, "msgr_send_messages_queue_lat": { "avgcount": 23903, "avgtime": 0.000115464, "sum": 2.75993975 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 335458, "msgr_recv_messages": 913, "msgr_running_fast_dispatch_time": 0.094644242, "msgr_running_recv_time": 0.042378267, "msgr_running_send_time": 0.090100996, "msgr_running_total_time": 0.225723245, "msgr_send_bytes": 5291036, "msgr_send_messages": 909, "msgr_send_messages_queue_lat": { "avgcount": 908, "avgtime": 0.000592073, "sum": 0.537602953 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 251, "avgtime": 0.01615019, "sum": 4.0536977 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1986, "avgtime": 0.000416165, "sum": 0.826503969 }, "queue_len": 0 }, "mds": { "caps": 1005, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7021, "ceph_cap_op_revoke": 85, "ceph_cap_op_trunc": 0, "dir_commit": 100, "dir_fetch": 12, "dir_merge": 1, "dir_split": 11, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 15, "handle_client_caps": 3246, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1014, "inodes_bottom": 1014, "inodes_expired": 1594, "inodes_pin_tail": 0, "inodes_pinned": 1014, "inodes_top": 0, "inodes_with_caps": 1005, "load_cent": 1157056, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3309, "q": 0, "reply": 6509, "reply_latency": { "avgcount": 6509, "avgtime": 0.002165448, "sum": 14.094907452 }, "request": 6511, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 6523, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6512, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 12, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1004, "num_strays_delayed": 107, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2197, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 407, "evadd": 12178, "evex": 11773, "evexd": 11773, "evexg": 0, "evtrm": 11773, "expos": 28766476, "jlat": { "avgcount": 47, "avgtime": 0.029226357, "sum": 1.373638783 }, "rdpos": 4195196, "replayed": 2, "seg": 2, "segadd": 15, "segex": 14, "segexd": 14, "segexg": 0, "segtrm": 14, "wrpos": 29575555 }, "mds_mem": { "cap": 931, "cap+": 3202, "cap-": 2271, "dir": 59, "dir+": 102, "dir-": 43, "dn": 1015, "dn+": 6412, "dn-": 5397, "heap": 356572, "ino": 1017, "ino+": 3214, "ino-": 2197, "rss": 100776 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6593, "dispatch_server_request": 0, "handle_client_request": 6511, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.205110729, "sum": 0.205110729 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.00206608, "sum": 6.611457497 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 772, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 120, "bloom_filter_items": 120, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 212138, "buffer_anon_items": 1104, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 3438678, "mds_co_items": 64520, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 28, "omap_wr": 205, "op": 3489, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 38, "op_reply": 3489, "op_resend": 0, "op_rmw": 0, "op_send": 3489, "op_send_bytes": 27344277, "op_w": 3451, "oplen_avg": { "avgcount": 3489, "sum": 5653 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1012, "osdop_delete": 2259, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 20, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2024, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 10, "osdop_truncate": 3, "osdop_watch": 0, "osdop_write": 59, "osdop_writefull": 11, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 2197, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13874, "get_or_fail_fail": 0, "get_or_fail_success": 13874, "get_started": 0, "get_sum": 3138609, "max": 104857600, "put": 13315, "put_sum": 2998621, "take": 0, "take_sum": 0, "val": 139988, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1215, "put_sum": 27567106, "take": 3489, "take_sum": 27567106, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3489, "put_sum": 3489, "take": 3489, "take_sum": 3489, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2197, "get_or_fail_fail": 0, "get_or_fail_success": 2197, "get_started": 0, "get_sum": 221897, "max": 3758096384, "put": 12, "put_sum": 221897, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 12178, "get_or_fail_fail": 0, "get_or_fail_success": 12178, "get_started": 0, "get_sum": 25380095, "max": 3758096384, "put": 47, "put_sum": 25164668, "take": 0, "take_sum": 0, "val": 215427, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:50.470 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1004, 2197/3201 2020-11-20T12:09:51.471 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:51.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.720+0000 7feec1a75700 1 Processor -- start 2020-11-20T12:09:51.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.721+0000 7feec1a75700 1 -- start start 2020-11-20T12:09:51.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.722+0000 7feec1a75700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7feebc149950 0x7feebc0b9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:51.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.722+0000 7feec1a75700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feebc14d140 con 0x7feebc0a1490 2020-11-20T12:09:51.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.722+0000 7feec1a75700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feebc14cea0 con 0x7feebc0a1870 2020-11-20T12:09:51.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.722+0000 7feec1a75700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feebc148da0 con 0x7feebc149950 2020-11-20T12:09:51.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.723+0000 7feeb3fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7feebc0a1870 0x7feebc149240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41634/0 (socket says 172.21.15.187:41634) 2020-11-20T12:09:51.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.723+0000 7feeb3fff700 1 -- 172.21.15.187:0/1961754540 learned_addr learned my addr 172.21.15.187:0/1961754540 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:51.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.723+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842623516 0 0) 0x7feebc14cea0 con 0x7feebc0a1870 2020-11-20T12:09:51.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.723+0000 7feec1274700 1 --2- 172.21.15.187:0/1961754540 >> v2:172.21.15.5:3300/0 conn(0x7feebc149950 0x7feebc0b9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:51.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.723+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea8003240 con 0x7feebc0a1870 2020-11-20T12:09:51.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 828379565 0 0) 0x7feea8003240 con 0x7feebc0a1870 2020-11-20T12:09:51.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 >> v2:172.21.15.5:3300/0 conn(0x7feebc149950 msgr2=0x7feebc0b9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:51.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 --2- 172.21.15.187:0/1961754540 >> v2:172.21.15.5:3300/0 conn(0x7feebc149950 0x7feebc0b9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 >> v1:172.21.15.5:6789/0 conn(0x7feebc0a1490 legacy=0x7feebc09fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc0a04e0 con 0x7feebc0a1870 2020-11-20T12:09:51.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7feeac0014e0 con 0x7feebc0a1870 2020-11-20T12:09:51.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 >> v1:172.21.15.187:6789/0 conn(0x7feebc0a1870 legacy=0x7feebc149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feea8003780 0x7feea8003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:51.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea80044e0 0x7feea8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:51.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8004e50 0x7feea800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:51.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc148da0 con 0x7feea80044e0 2020-11-20T12:09:51.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea8005330 con 0x7feea8003780 2020-11-20T12:09:51.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea800bc20 con 0x7feea8004e50 2020-11-20T12:09:51.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.724+0000 7feec0a73700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feea8003780 0x7feea8003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:51.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feec0a73700 1 -- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8004e50 msgr2=0x7feea800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:51.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feec1274700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8004e50 0x7feea800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:51.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feeb3fff700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea80044e0 0x7feea8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:51.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feec0a73700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8004e50 0x7feea800b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feec0a73700 1 -- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea80044e0 msgr2=0x7feea8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:51.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feec0a73700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea80044e0 0x7feea8004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feec0a73700 1 -- 172.21.15.187:0/1961754540 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea8003240 con 0x7feea8003780 2020-11-20T12:09:51.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feeb3fff700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea80044e0 0x7feea8004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:51.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feec1274700 1 -- 172.21.15.187:0/1961754540 reap_dead start 2020-11-20T12:09:51.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feec0a73700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feea8003780 0x7feea8003ba0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7feeb8001920 tx=0x7feeb8007b90).ready entity=mon.1 client_cookie=5935a378bc298e15 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:51.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.725+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb8007020 con 0x7feea8003780 2020-11-20T12:09:51.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb8007180 con 0x7feea8003780 2020-11-20T12:09:51.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feeb37fe700 1 -- 172.21.15.187:0/1961754540 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb800c620 con 0x7feea8003780 2020-11-20T12:09:51.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feec1a75700 1 -- 172.21.15.187:0/1961754540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feea8003780 msgr2=0x7feea8003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feec1a75700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feea8003780 0x7feea8003ba0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7feeb8001920 tx=0x7feeb8007b90).stop 2020-11-20T12:09:51.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feec1a75700 1 -- 172.21.15.187:0/1961754540 shutdown_connections 2020-11-20T12:09:51.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feec1a75700 1 --2- 172.21.15.187:0/1961754540 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feea8003780 0x7feea8003ba0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feec1a75700 1 -- 172.21.15.187:0/1961754540 >> 172.21.15.187:0/1961754540 conn(0x7feebc0bf110 msgr2=0x7feebc0bf520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:51.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feec1a75700 1 -- 172.21.15.187:0/1961754540 shutdown_connections 2020-11-20T12:09:51.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.726+0000 7feec1a75700 1 -- 172.21.15.187:0/1961754540 wait complete. 2020-11-20T12:09:51.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.728+0000 7feec1a75700 1 Processor -- start 2020-11-20T12:09:51.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.728+0000 7feec1a75700 1 -- start start 2020-11-20T12:09:51.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feec1a75700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8003780 0x7feebc158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:51.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feec1a75700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc14b370 0x7feebc1189f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:51.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feec1a75700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc118ee0 0x7feebc13b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:51.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feec1a75700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc119370 con 0x7feebc118ee0 2020-11-20T12:09:51.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feec1a75700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc119610 con 0x7feebc14b370 2020-11-20T12:09:51.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feec1a75700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc1198b0 con 0x7feea8003780 2020-11-20T12:09:51.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feeb3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc14b370 0x7feebc1189f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:51.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feeb3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc14b370 0x7feebc1189f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45040/0 (socket says 172.21.15.187:45040) 2020-11-20T12:09:51.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feeb3fff700 1 -- 172.21.15.187:0/1617180942 learned_addr learned my addr 172.21.15.187:0/1617180942 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:51.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feec0a73700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8003780 0x7feebc158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:51.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feec1274700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc118ee0 0x7feebc13b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:51.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feeb3fff700 1 -- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8003780 msgr2=0x7feebc158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.729+0000 7feeb3fff700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8003780 0x7feebc158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feeb3fff700 1 -- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc118ee0 msgr2=0x7feebc13b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feeb3fff700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc118ee0 0x7feebc13b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feeb3fff700 1 -- 172.21.15.187:0/1617180942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc158870 con 0x7feebc14b370 2020-11-20T12:09:51.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feec0a73700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8003780 0x7feebc158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:51.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feec1274700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc118ee0 0x7feebc13b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:51.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feeb3fff700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc14b370 0x7feebc1189f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7feeac00db00 tx=0x7feeac009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:51.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feeb0ff9700 1 -- 172.21.15.187:0/1617180942 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac007170 con 0x7feebc14b370 2020-11-20T12:09:51.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feeb0ff9700 1 -- 172.21.15.187:0/1617180942 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeac0072d0 con 0x7feebc14b370 2020-11-20T12:09:51.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feebc13bb00 con 0x7feebc14b370 2020-11-20T12:09:51.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feeb0ff9700 1 -- 172.21.15.187:0/1617180942 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac00ba70 con 0x7feebc14b370 2020-11-20T12:09:51.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.730+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feebc0a2d70 con 0x7feebc14b370 2020-11-20T12:09:51.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.731+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feebc0a4fc0 con 0x7feebc14b370 2020-11-20T12:09:51.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.732+0000 7feeb0ff9700 1 -- 172.21.15.187:0/1617180942 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feeac004500 con 0x7feebc14b370 2020-11-20T12:09:51.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.732+0000 7feeb0ff9700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee94056d20 0x7fee940591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:51.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.732+0000 7feeb0ff9700 1 -- 172.21.15.187:0/1617180942 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7feeac070440 con 0x7feebc14b370 2020-11-20T12:09:51.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.733+0000 7feec0a73700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee94056d20 0x7fee940591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:51.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.733+0000 7feec0a73700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee94056d20 0x7fee940591e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7feeb8000f90 tx=0x7feeb8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:51.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.736+0000 7feeb0ff9700 1 -- 172.21.15.187:0/1617180942 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feeac045840 con 0x7feebc14b370 2020-11-20T12:09:51.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.980+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feebc044f60 con 0x7feebc14b370 2020-11-20T12:09:51.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.982+0000 7feeb0ff9700 1 -- 172.21.15.187:0/1617180942 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7feeac048e60 con 0x7feebc14b370 2020-11-20T12:09:51.985 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:51.985 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:51.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee94056d20 msgr2=0x7fee940591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee94056d20 0x7fee940591e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7feeb8000f90 tx=0x7feeb8015000).stop 2020-11-20T12:09:51.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc14b370 msgr2=0x7feebc1189f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc14b370 0x7feebc1189f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7feeac00db00 tx=0x7feeac009040).stop 2020-11-20T12:09:51.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 shutdown_connections 2020-11-20T12:09:51.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee94056d20 0x7fee940591e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc118ee0 0x7feebc13b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feea8003780 0x7feebc158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 --2- 172.21.15.187:0/1617180942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc14b370 0x7feebc1189f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 >> 172.21.15.187:0/1617180942 conn(0x7feebc0bf110 msgr2=0x7feebc09ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:51.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 shutdown_connections 2020-11-20T12:09:51.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:51.984+0000 7feec1a75700 1 -- 172.21.15.187:0/1617180942 wait complete. 2020-11-20T12:09:51.990 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:52.009 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:52.276 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1776,"msgr_send_messages":1776,"msgr_recv_bytes":736557,"msgr_send_bytes":13224016,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.319506417,"msgr_running_send_time":0.190858988,"msgr_running_recv_time":0.053602671,"msgr_running_fast_dispatch_time":0.055904989,"msgr_send_messages_queue_lat":{"avgcount":1775,"sum":1.206933456,"avgtime":0.000679962},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11554,"msgr_send_messages":23989,"msgr_recv_bytes":3276783,"msgr_send_bytes":15364767,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":5.151446236,"msgr_running_send_time":4.326146401,"msgr_running_recv_time":3.825547970,"msgr_running_fast_dispatch_time":0.032689296,"msgr_send_messages_queue_lat":{"avgcount":23987,"sum":2.777854271,"avgtime":0.000115806},"msgr_handle_ack_lat":{"avgcount":19075,"sum":0.009200822,"avgtime":0.000000482}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":949,"msgr_send_messages":945,"msgr_recv_bytes":344620,"msgr_send_bytes":5302761,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.231609058,"msgr_running_send_time":0.092691393,"msgr_running_recv_time":0.043686840,"msgr_running_fast_dispatch_time":0.096005962,"msgr_send_messages_queue_lat":{"avgcount":944,"sum":0.540205308,"avgtime":0.000572251},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":251,"sum":4.053697700,"avgtime":0.016150190}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2115,"sum":0.852027035,"avgtime":0.000402849}},"mds":{"request":6511,"reply":6509,"reply_latency":{"avgcount":6509,"sum":14.094907452,"avgtime":0.002165448},"forward":0,"dir_fetch":12,"dir_commit":100,"dir_split":11,"dir_merge":1,"inodes":1014,"inodes_top":0,"inodes_bottom":1014,"inodes_pin_tail":0,"inodes_pinned":1014,"inodes_expired":1594,"inodes_with_caps":1005,"caps":1005,"subtrees":2,"traverse":6523,"traverse_hit":6512,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":1157056,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":85,"ceph_cap_op_grant":7023,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4015,"handle_client_caps_dirty":3000,"handle_client_cap_release":38,"process_request_cap_release":3309},"mds_cache":{"num_strays":879,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2322,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":12,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":13072,"evex":11773,"evtrm":11773,"ev":1301,"evexg":0,"evexd":11773,"segadd":15,"segex":14,"segtrm":14,"seg":2,"segexg":0,"segexd":14,"expos":28766476,"wrpos":31153234,"rdpos":4195196,"jlat":{"avgcount":47,"sum":1.373638783,"avgtime":0.029226357},"replayed":2},"mds_mem":{"ino":1017,"ino+":3214,"ino-":2197,"dir":26,"dir+":102,"dir-":76,"dn":1015,"dn+":6412,"dn-":5397,"cap":1,"cap+":3202,"cap-":3201,"rss":100776,"heap":356572},"mds_server":{"dispatch_client_request":6593,"dispatch_server_request":0,"handle_client_request":6511,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.205110729,"avgtime":0.205110729},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.611457497,"avgtime":0.002066080},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":772},"mempool":{"bloom_filter_bytes":120,"bloom_filter_items":120,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":212138,"buffer_anon_items":1104,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":3438678,"mds_co_items":64520,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3654,"op_send_bytes":27357008,"op_resend":0,"op_reply":3654,"oplen_avg":{"avgcount":3654,"sum":5818},"op":3654,"op_r":41,"op_w":3613,"op_rmw":0,"op_pg":0,"osdop_stat":10,"osdop_create":1012,"osdop_read":23,"osdop_write":62,"osdop_writefull":12,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":3,"osdop_delete":2417,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":2024,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":205,"omap_rd":28,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":39,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":2322,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14275,"get_sum":3239900,"get_or_fail_fail":0,"get_or_fail_success":14275,"take":0,"take_sum":0,"put":14275,"put_sum":3239900,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3654,"take_sum":27592478,"put":1222,"put_sum":27592478,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3654,"take_sum":3654,"put":3654,"put_sum":3654,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2322,"get_sum":234538,"get_or_fail_fail":0,"get_or_fail_success":2322,"take":0,"take_sum":0,"put":15,"put_sum":234538,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":1793106,"max":3758096384,"get_started":0,"get":13072,"get_sum":26957774,"get_or_fail_fail":0,"get_or_fail_success":13072,"take":0,"take_sum":0,"put":47,"put_sum":25164668,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:52.277 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 736557, "msgr_recv_messages": 1776, "msgr_running_fast_dispatch_time": 0.055904989, "msgr_running_recv_time": 0.053602671, "msgr_running_send_time": 0.190858988, "msgr_running_total_time": 0.319506417, "msgr_send_bytes": 13224016, "msgr_send_messages": 1776, "msgr_send_messages_queue_lat": { "avgcount": 1775, "avgtime": 0.000679962, "sum": 1.206933456 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19075, "avgtime": 4.82e-07, "sum": 0.009200822 }, "msgr_recv_bytes": 3276783, "msgr_recv_messages": 11554, "msgr_running_fast_dispatch_time": 0.032689296, "msgr_running_recv_time": 3.82554797, "msgr_running_send_time": 4.326146401, "msgr_running_total_time": 5.151446236, "msgr_send_bytes": 15364767, "msgr_send_messages": 23989, "msgr_send_messages_queue_lat": { "avgcount": 23987, "avgtime": 0.000115806, "sum": 2.777854271 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 344620, "msgr_recv_messages": 949, "msgr_running_fast_dispatch_time": 0.096005962, "msgr_running_recv_time": 0.04368684, "msgr_running_send_time": 0.092691393, "msgr_running_total_time": 0.231609058, "msgr_send_bytes": 5302761, "msgr_send_messages": 945, "msgr_send_messages_queue_lat": { "avgcount": 944, "avgtime": 0.000572251, "sum": 0.540205308 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 251, "avgtime": 0.01615019, "sum": 4.0536977 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2115, "avgtime": 0.000402849, "sum": 0.852027035 }, "queue_len": 0 }, "mds": { "caps": 1005, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7023, "ceph_cap_op_revoke": 85, "ceph_cap_op_trunc": 0, "dir_commit": 100, "dir_fetch": 12, "dir_merge": 1, "dir_split": 11, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 38, "handle_client_caps": 4015, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1014, "inodes_bottom": 1014, "inodes_expired": 1594, "inodes_pin_tail": 0, "inodes_pinned": 1014, "inodes_top": 0, "inodes_with_caps": 1005, "load_cent": 1157056, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3309, "q": 0, "reply": 6509, "reply_latency": { "avgcount": 6509, "avgtime": 0.002165448, "sum": 14.094907452 }, "request": 6511, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 6523, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6512, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 12, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 879, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2322, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1301, "evadd": 13072, "evex": 11773, "evexd": 11773, "evexg": 0, "evtrm": 11773, "expos": 28766476, "jlat": { "avgcount": 47, "avgtime": 0.029226357, "sum": 1.373638783 }, "rdpos": 4195196, "replayed": 2, "seg": 2, "segadd": 15, "segex": 14, "segexd": 14, "segexg": 0, "segtrm": 14, "wrpos": 31153234 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 26, "dir+": 102, "dir-": 76, "dn": 1015, "dn+": 6412, "dn-": 5397, "heap": 356572, "ino": 1017, "ino+": 3214, "ino-": 2197, "rss": 100776 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6593, "dispatch_server_request": 0, "handle_client_request": 6511, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.205110729, "sum": 0.205110729 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.00206608, "sum": 6.611457497 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 772, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 120, "bloom_filter_items": 120, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 212138, "buffer_anon_items": 1104, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 3438678, "mds_co_items": 64520, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 28, "omap_wr": 205, "op": 3654, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 41, "op_reply": 3654, "op_resend": 0, "op_rmw": 0, "op_send": 3654, "op_send_bytes": 27357008, "op_w": 3613, "oplen_avg": { "avgcount": 3654, "sum": 5818 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1012, "osdop_delete": 2417, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 23, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2024, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 10, "osdop_truncate": 3, "osdop_watch": 0, "osdop_write": 62, "osdop_writefull": 12, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 2322, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 39, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14275, "get_or_fail_fail": 0, "get_or_fail_success": 14275, "get_started": 0, "get_sum": 3239900, "max": 104857600, "put": 14275, "put_sum": 3239900, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1222, "put_sum": 27592478, "take": 3654, "take_sum": 27592478, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3654, "put_sum": 3654, "take": 3654, "take_sum": 3654, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2322, "get_or_fail_fail": 0, "get_or_fail_success": 2322, "get_started": 0, "get_sum": 234538, "max": 3758096384, "put": 15, "put_sum": 234538, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 13072, "get_or_fail_fail": 0, "get_or_fail_success": 13072, "get_started": 0, "get_sum": 26957774, "max": 3758096384, "put": 47, "put_sum": 25164668, "take": 0, "take_sum": 0, "val": 1793106, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:52.278 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/879, 2322/3201 2020-11-20T12:09:53.279 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:54.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.132+0000 7fe6c13bb700 1 Processor -- start 2020-11-20T12:09:54.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.132+0000 7fe6c13bb700 1 -- start start 2020-11-20T12:09:54.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6c13bb700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe6bc063590 0x7fe6bc0b9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:54.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6c13bb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe6bc148590 0x7fe6bc060280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:54.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6c13bb700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc0614a0 con 0x7fe6bc1481b0 2020-11-20T12:09:54.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6c13bb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6bc061740 con 0x7fe6bc148590 2020-11-20T12:09:54.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6c13bb700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6bc067260 con 0x7fe6bc063590 2020-11-20T12:09:54.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6bb7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe6bc063590 0x7fe6bc0b9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:54.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6bb7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe6bc063590 0x7fe6bc0b9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45048/0 (socket says 172.21.15.187:45048) 2020-11-20T12:09:54.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6bb7fe700 1 -- 172.21.15.187:0/504401310 learned_addr learned my addr 172.21.15.187:0/504401310 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:54.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6bb7fe700 1 -- 172.21.15.187:0/504401310 >> v2:172.21.15.5:3300/0 conn(0x7fe6bc148590 msgr2=0x7fe6bc060280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:54.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.134+0000 7fe6c0bba700 1 --2- 172.21.15.187:0/504401310 >> v2:172.21.15.5:3300/0 conn(0x7fe6bc148590 0x7fe6bc060280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:54.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.135+0000 7fe6bb7fe700 1 --2- 172.21.15.187:0/504401310 >> v2:172.21.15.5:3300/0 conn(0x7fe6bc148590 0x7fe6bc060280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:54.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.135+0000 7fe6bb7fe700 1 -- 172.21.15.187:0/504401310 >> v1:172.21.15.5:6789/0 conn(0x7fe6bc1481b0 legacy=0x7fe6bc066b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:54.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.135+0000 7fe6bb7fe700 1 -- 172.21.15.187:0/504401310 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc067900 con 0x7fe6bc063590 2020-11-20T12:09:54.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.135+0000 7fe6bb7fe700 1 --2- 172.21.15.187:0/504401310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc063590 0x7fe6bc0b9a00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7fe6bc0bb0e0 tx=0x7fe6a8006eb0).ready entity=mon.1 client_cookie=d60870983012019e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:54.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.135+0000 7fe6baffd700 1 -- 172.21.15.187:0/504401310 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8009890 con 0x7fe6bc063590 2020-11-20T12:09:54.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.135+0000 7fe6baffd700 1 -- 172.21.15.187:0/504401310 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a80099f0 con 0x7fe6bc063590 2020-11-20T12:09:54.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.135+0000 7fe6baffd700 1 -- 172.21.15.187:0/504401310 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8009d70 con 0x7fe6bc063590 2020-11-20T12:09:54.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.136+0000 7fe6c13bb700 1 -- 172.21.15.187:0/504401310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc063590 msgr2=0x7fe6bc0b9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:54.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.136+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/504401310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc063590 0x7fe6bc0b9a00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7fe6bc0bb0e0 tx=0x7fe6a8006eb0).stop 2020-11-20T12:09:54.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.136+0000 7fe6c13bb700 1 -- 172.21.15.187:0/504401310 shutdown_connections 2020-11-20T12:09:54.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.136+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/504401310 >> v2:172.21.15.5:3300/0 conn(0x7fe6bc148590 0x7fe6bc060280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:54.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.136+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/504401310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc063590 0x7fe6bc0b9a00 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:54.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.136+0000 7fe6c13bb700 1 -- 172.21.15.187:0/504401310 >> 172.21.15.187:0/504401310 conn(0x7fe6bc0b7bb0 msgr2=0x7fe6bc0b7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:54.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.136+0000 7fe6c13bb700 1 -- 172.21.15.187:0/504401310 shutdown_connections 2020-11-20T12:09:54.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.136+0000 7fe6c13bb700 1 -- 172.21.15.187:0/504401310 wait complete. 2020-11-20T12:09:54.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.138+0000 7fe6c13bb700 1 Processor -- start 2020-11-20T12:09:54.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.138+0000 7fe6c13bb700 1 -- start start 2020-11-20T12:09:54.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c13bb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6bc063590 0x7fe6bc13b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:54.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c13bb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6bc1481b0 0x7fe6bc0b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:54.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c13bb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc148590 0x7fe6bc1193d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:54.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c13bb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc1459e0 con 0x7fe6bc063590 2020-11-20T12:09:54.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c13bb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc145c80 con 0x7fe6bc148590 2020-11-20T12:09:54.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c13bb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc145f20 con 0x7fe6bc1481b0 2020-11-20T12:09:54.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c0bba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc148590 0x7fe6bc1193d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:54.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c0bba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc148590 0x7fe6bc1193d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45056/0 (socket says 172.21.15.187:45056) 2020-11-20T12:09:54.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c0bba700 1 -- 172.21.15.187:0/1818781622 learned_addr learned my addr 172.21.15.187:0/1818781622 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:54.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6bb7fe700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6bc1481b0 0x7fe6bc0b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:54.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6bbfff700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6bc063590 0x7fe6bc13b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:54.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c0bba700 1 -- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6bc1481b0 msgr2=0x7fe6bc0b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:54.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c0bba700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6bc1481b0 0x7fe6bc0b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:54.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c0bba700 1 -- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6bc063590 msgr2=0x7fe6bc13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:54.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c0bba700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6bc063590 0x7fe6bc13b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:54.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6c0bba700 1 -- 172.21.15.187:0/1818781622 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc13be80 con 0x7fe6bc148590 2020-11-20T12:09:54.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.139+0000 7fe6bbfff700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6bc063590 0x7fe6bc13b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:54.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.140+0000 7fe6c0bba700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc148590 0x7fe6bc1193d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fe6b400caa0 tx=0x7fe6b4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:54.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.140+0000 7fe6b3fff700 1 -- 172.21.15.187:0/1818781622 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4007970 con 0x7fe6bc148590 2020-11-20T12:09:54.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.140+0000 7fe6b3fff700 1 -- 172.21.15.187:0/1818781622 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b4007ad0 con 0x7fe6bc148590 2020-11-20T12:09:54.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.140+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6bc0b4f00 con 0x7fe6bc148590 2020-11-20T12:09:54.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.140+0000 7fe6b3fff700 1 -- 172.21.15.187:0/1818781622 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4007e70 con 0x7fe6bc148590 2020-11-20T12:09:54.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.140+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6bc0a0000 con 0x7fe6bc148590 2020-11-20T12:09:54.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.142+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6bc1460e0 con 0x7fe6bc148590 2020-11-20T12:09:54.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.142+0000 7fe6b3fff700 1 -- 172.21.15.187:0/1818781622 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe6b4011620 con 0x7fe6bc148590 2020-11-20T12:09:54.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.143+0000 7fe6b3fff700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe690056d20 0x7fe6900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:54.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.143+0000 7fe6b3fff700 1 -- 172.21.15.187:0/1818781622 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe6b4070490 con 0x7fe6bc148590 2020-11-20T12:09:54.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.143+0000 7fe6bbfff700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe690056d20 0x7fe6900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:54.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.144+0000 7fe6bbfff700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe690056d20 0x7fe6900591e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fe6ac002510 tx=0x7fe6ac009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:54.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.147+0000 7fe6b3fff700 1 -- 172.21.15.187:0/1818781622 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe6b4045890 con 0x7fe6bc148590 2020-11-20T12:09:54.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.470+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6bc044f60 con 0x7fe6bc148590 2020-11-20T12:09:54.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.471+0000 7fe6b3fff700 1 -- 172.21.15.187:0/1818781622 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fe6b4048eb0 con 0x7fe6bc148590 2020-11-20T12:09:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:54.476 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:54.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.475+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe690056d20 msgr2=0x7fe6900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:54.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.475+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe690056d20 0x7fe6900591e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fe6ac002510 tx=0x7fe6ac009040).stop 2020-11-20T12:09:54.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.475+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc148590 msgr2=0x7fe6bc1193d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:54.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.475+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc148590 0x7fe6bc1193d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fe6b400caa0 tx=0x7fe6b4008040).stop 2020-11-20T12:09:54.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.476+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 shutdown_connections 2020-11-20T12:09:54.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.476+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe690056d20 0x7fe6900591e0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0x7fe6ac002510 tx=0x7fe6ac009040).stop 2020-11-20T12:09:54.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.476+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6bc063590 0x7fe6bc13b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:54.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.476+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6bc148590 0x7fe6bc1193d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:54.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.476+0000 7fe6c13bb700 1 --2- 172.21.15.187:0/1818781622 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6bc1481b0 0x7fe6bc0b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:54.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.476+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 >> 172.21.15.187:0/1818781622 conn(0x7fe6bc0b7bb0 msgr2=0x7fe6bc0b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:54.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.478+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 shutdown_connections 2020-11-20T12:09:54.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:54.478+0000 7fe6c13bb700 1 -- 172.21.15.187:0/1818781622 wait complete. 2020-11-20T12:09:54.483 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:54.510 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:54.805 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2092,"msgr_send_messages":2092,"msgr_recv_bytes":879898,"msgr_send_bytes":13387897,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.358618578,"msgr_running_send_time":0.207462666,"msgr_running_recv_time":0.062736902,"msgr_running_fast_dispatch_time":0.066138882,"msgr_send_messages_queue_lat":{"avgcount":2091,"sum":1.222108977,"avgtime":0.000584461},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11709,"msgr_send_messages":24142,"msgr_recv_bytes":3315975,"msgr_send_bytes":17208131,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":5.183988949,"msgr_running_send_time":4.344035477,"msgr_running_recv_time":3.831083591,"msgr_running_fast_dispatch_time":0.039928826,"msgr_send_messages_queue_lat":{"avgcount":24140,"sum":2.785872464,"avgtime":0.000115404},"msgr_handle_ack_lat":{"avgcount":19075,"sum":0.009200822,"avgtime":0.000000482}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1116,"msgr_send_messages":1112,"msgr_recv_bytes":387136,"msgr_send_bytes":5370456,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.255197090,"msgr_running_send_time":0.101079190,"msgr_running_recv_time":0.049908667,"msgr_running_fast_dispatch_time":0.102770303,"msgr_send_messages_queue_lat":{"avgcount":1111,"sum":0.549175089,"avgtime":0.000494307},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":254,"sum":4.287930645,"avgtime":0.016881616}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2722,"sum":0.994644525,"avgtime":0.000365409}},"mds":{"request":6511,"reply":6509,"reply_latency":{"avgcount":6509,"sum":14.094907452,"avgtime":0.002165448},"forward":0,"dir_fetch":12,"dir_commit":100,"dir_split":11,"dir_merge":1,"inodes":1014,"inodes_top":0,"inodes_bottom":1014,"inodes_pin_tail":0,"inodes_pinned":1014,"inodes_expired":1594,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6523,"traverse_hit":6512,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":157834,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":85,"ceph_cap_op_grant":7023,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4015,"handle_client_caps_dirty":3000,"handle_client_cap_release":38,"process_request_cap_release":3309},"mds_cache":{"num_strays":254,"num_strays_delayed":254,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2947,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":12,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":13698,"evex":11773,"evtrm":11773,"ev":1927,"evexg":0,"evexd":11773,"segadd":16,"segex":14,"segtrm":14,"seg":3,"segexg":0,"segexd":14,"expos":28766476,"wrpos":31534341,"rdpos":4195196,"jlat":{"avgcount":48,"sum":1.406295715,"avgtime":0.029297827},"replayed":2},"mds_mem":{"ino":1017,"ino+":3214,"ino-":2197,"dir":26,"dir+":102,"dir-":76,"dn":1015,"dn+":6412,"dn-":5397,"cap":1,"cap+":3202,"cap-":3201,"rss":88124,"heap":356572},"mds_server":{"dispatch_client_request":6593,"dispatch_server_request":0,"handle_client_request":6511,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.205110729,"avgtime":0.205110729},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.611457497,"avgtime":0.002066080},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":777},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":11788138,"buffer_anon_items":3998,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":4736790,"mds_co_items":87898,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4288,"op_send_bytes":29227438,"op_resend":0,"op_reply":4288,"oplen_avg":{"avgcount":4288,"sum":6454},"op":4288,"op_r":44,"op_w":4244,"op_rmw":0,"op_pg":0,"osdop_stat":10,"osdop_create":1012,"osdop_read":26,"osdop_write":66,"osdop_writefull":13,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":3,"osdop_delete":3042,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":2024,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":207,"omap_rd":28,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":39,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":2947,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14913,"get_sum":3407525,"get_or_fail_fail":0,"get_or_fail_success":14913,"take":0,"take_sum":0,"put":14913,"put_sum":3407525,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4288,"take_sum":29526033,"put":1231,"put_sum":29526033,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4288,"take_sum":4288,"put":4288,"put_sum":4288,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2947,"get_sum":297663,"get_or_fail_fail":0,"get_or_fail_success":2947,"take":0,"take_sum":0,"put":18,"put_sum":297663,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":381107,"max":3758096384,"get_started":0,"get":13698,"get_sum":27338881,"get_or_fail_fail":0,"get_or_fail_success":13698,"take":0,"take_sum":0,"put":48,"put_sum":26957774,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:54.807 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 879898, "msgr_recv_messages": 2092, "msgr_running_fast_dispatch_time": 0.066138882, "msgr_running_recv_time": 0.062736902, "msgr_running_send_time": 0.207462666, "msgr_running_total_time": 0.358618578, "msgr_send_bytes": 13387897, "msgr_send_messages": 2092, "msgr_send_messages_queue_lat": { "avgcount": 2091, "avgtime": 0.000584461, "sum": 1.222108977 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19075, "avgtime": 4.82e-07, "sum": 0.009200822 }, "msgr_recv_bytes": 3315975, "msgr_recv_messages": 11709, "msgr_running_fast_dispatch_time": 0.039928826, "msgr_running_recv_time": 3.831083591, "msgr_running_send_time": 4.344035477, "msgr_running_total_time": 5.183988949, "msgr_send_bytes": 17208131, "msgr_send_messages": 24142, "msgr_send_messages_queue_lat": { "avgcount": 24140, "avgtime": 0.000115404, "sum": 2.785872464 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 387136, "msgr_recv_messages": 1116, "msgr_running_fast_dispatch_time": 0.102770303, "msgr_running_recv_time": 0.049908667, "msgr_running_send_time": 0.10107919, "msgr_running_total_time": 0.25519709, "msgr_send_bytes": 5370456, "msgr_send_messages": 1112, "msgr_send_messages_queue_lat": { "avgcount": 1111, "avgtime": 0.000494307, "sum": 0.549175089 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 254, "avgtime": 0.016881616, "sum": 4.287930645 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2722, "avgtime": 0.000365409, "sum": 0.994644525 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7023, "ceph_cap_op_revoke": 85, "ceph_cap_op_trunc": 0, "dir_commit": 100, "dir_fetch": 12, "dir_merge": 1, "dir_split": 11, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 38, "handle_client_caps": 4015, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1014, "inodes_bottom": 1014, "inodes_expired": 1594, "inodes_pin_tail": 0, "inodes_pinned": 1014, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 157834, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3309, "q": 0, "reply": 6509, "reply_latency": { "avgcount": 6509, "avgtime": 0.002165448, "sum": 14.094907452 }, "request": 6511, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 6523, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6512, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 12, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 254, "num_strays_delayed": 254, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2947, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1927, "evadd": 13698, "evex": 11773, "evexd": 11773, "evexg": 0, "evtrm": 11773, "expos": 28766476, "jlat": { "avgcount": 48, "avgtime": 0.029297827, "sum": 1.406295715 }, "rdpos": 4195196, "replayed": 2, "seg": 3, "segadd": 16, "segex": 14, "segexd": 14, "segexg": 0, "segtrm": 14, "wrpos": 31534341 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 26, "dir+": 102, "dir-": 76, "dn": 1015, "dn+": 6412, "dn-": 5397, "heap": 356572, "ino": 1017, "ino+": 3214, "ino-": 2197, "rss": 88124 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6593, "dispatch_server_request": 0, "handle_client_request": 6511, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.205110729, "sum": 0.205110729 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.00206608, "sum": 6.611457497 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 777, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 11788138, "buffer_anon_items": 3998, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 4736790, "mds_co_items": 87898, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 28, "omap_wr": 207, "op": 4288, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 44, "op_reply": 4288, "op_resend": 0, "op_rmw": 0, "op_send": 4288, "op_send_bytes": 29227438, "op_w": 4244, "oplen_avg": { "avgcount": 4288, "sum": 6454 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1012, "osdop_delete": 3042, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 26, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2024, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 10, "osdop_truncate": 3, "osdop_watch": 0, "osdop_write": 66, "osdop_writefull": 13, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 2947, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 39, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14913, "get_or_fail_fail": 0, "get_or_fail_success": 14913, "get_started": 0, "get_sum": 3407525, "max": 104857600, "put": 14913, "put_sum": 3407525, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1231, "put_sum": 29526033, "take": 4288, "take_sum": 29526033, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4288, "put_sum": 4288, "take": 4288, "take_sum": 4288, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2947, "get_or_fail_fail": 0, "get_or_fail_success": 2947, "get_started": 0, "get_sum": 297663, "max": 3758096384, "put": 18, "put_sum": 297663, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 13698, "get_or_fail_fail": 0, "get_or_fail_success": 13698, "get_started": 0, "get_sum": 27338881, "max": 3758096384, "put": 48, "put_sum": 26957774, "take": 0, "take_sum": 0, "val": 381107, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:54.808 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/254, 2947/3201 2020-11-20T12:09:55.809 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:56.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.056+0000 7fe615088700 1 Processor -- start 2020-11-20T12:09:56.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.057+0000 7fe615088700 1 -- start start 2020-11-20T12:09:56.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.058+0000 7fe615088700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe610065c90 0x7fe61014cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.058+0000 7fe615088700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe610066070 0x7fe6100b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.058+0000 7fe615088700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61014c2a0 con 0x7fe6100b6dc0 2020-11-20T12:09:56.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.058+0000 7fe615088700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61014c540 con 0x7fe610065c90 2020-11-20T12:09:56.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.058+0000 7fe615088700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61014c7e0 con 0x7fe610066070 2020-11-20T12:09:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.059+0000 7fe6077fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe610066070 0x7fe6100b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.059+0000 7fe6077fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe610066070 0x7fe6100b5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45062/0 (socket says 172.21.15.187:45062) 2020-11-20T12:09:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.059+0000 7fe6077fe700 1 -- 172.21.15.187:0/202845827 learned_addr learned my addr 172.21.15.187:0/202845827 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.059+0000 7fe60ffff700 1 --2- 172.21.15.187:0/202845827 >> v2:172.21.15.5:3300/0 conn(0x7fe610065c90 0x7fe61014cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.059+0000 7fe6077fe700 1 -- 172.21.15.187:0/202845827 >> v2:172.21.15.5:3300/0 conn(0x7fe610065c90 msgr2=0x7fe61014cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:56.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.059+0000 7fe6077fe700 1 --2- 172.21.15.187:0/202845827 >> v2:172.21.15.5:3300/0 conn(0x7fe610065c90 0x7fe61014cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.059+0000 7fe6077fe700 1 -- 172.21.15.187:0/202845827 >> v1:172.21.15.5:6789/0 conn(0x7fe6100b6dc0 legacy=0x7fe610145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:56.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.059+0000 7fe6077fe700 1 -- 172.21.15.187:0/202845827 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6100b5800 con 0x7fe610066070 2020-11-20T12:09:56.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.060+0000 7fe6077fe700 1 --2- 172.21.15.187:0/202845827 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe6100b5310 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7fe610064cd0 tx=0x7fe5f8006eb0).ready entity=mon.1 client_cookie=5043127585d6cf64 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:56.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.060+0000 7fe60f7fe700 1 -- 172.21.15.187:0/202845827 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8009890 con 0x7fe610066070 2020-11-20T12:09:56.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.060+0000 7fe60f7fe700 1 -- 172.21.15.187:0/202845827 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f80099f0 con 0x7fe610066070 2020-11-20T12:09:56.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.060+0000 7fe60f7fe700 1 -- 172.21.15.187:0/202845827 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8009d70 con 0x7fe610066070 2020-11-20T12:09:56.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.060+0000 7fe615088700 1 -- 172.21.15.187:0/202845827 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 msgr2=0x7fe6100b5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.060+0000 7fe615088700 1 --2- 172.21.15.187:0/202845827 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe6100b5310 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7fe610064cd0 tx=0x7fe5f8006eb0).stop 2020-11-20T12:09:56.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.061+0000 7fe615088700 1 -- 172.21.15.187:0/202845827 shutdown_connections 2020-11-20T12:09:56.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.061+0000 7fe615088700 1 --2- 172.21.15.187:0/202845827 >> v2:172.21.15.5:3300/0 conn(0x7fe610065c90 0x7fe61014cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.061+0000 7fe615088700 1 --2- 172.21.15.187:0/202845827 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe6100b5310 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.061+0000 7fe615088700 1 -- 172.21.15.187:0/202845827 >> 172.21.15.187:0/202845827 conn(0x7fe6100b8280 msgr2=0x7fe6100653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:56.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.061+0000 7fe615088700 1 -- 172.21.15.187:0/202845827 shutdown_connections 2020-11-20T12:09:56.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.061+0000 7fe615088700 1 -- 172.21.15.187:0/202845827 wait complete. 2020-11-20T12:09:56.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.062+0000 7fe615088700 1 Processor -- start 2020-11-20T12:09:56.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.062+0000 7fe615088700 1 -- start start 2020-11-20T12:09:56.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.063+0000 7fe615088700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe610065c90 0x7fe6100ba2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.063+0000 7fe615088700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe610094360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.063+0000 7fe615088700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6100b6dc0 0x7fe61013e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.063+0000 7fe615088700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61013ec60 con 0x7fe6100b6dc0 2020-11-20T12:09:56.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.063+0000 7fe615088700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61013ef00 con 0x7fe610066070 2020-11-20T12:09:56.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.063+0000 7fe615088700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61013f1a0 con 0x7fe610065c90 2020-11-20T12:09:56.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe610094360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe610094360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45070/0 (socket says 172.21.15.187:45070) 2020-11-20T12:09:56.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 -- 172.21.15.187:0/2002788432 learned_addr learned my addr 172.21.15.187:0/2002788432 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:56.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe60ffff700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe610065c90 0x7fe6100ba2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe614887700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6100b6dc0 0x7fe61013e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 -- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe610065c90 msgr2=0x7fe6100ba2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe610065c90 0x7fe6100ba2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 -- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6100b6dc0 msgr2=0x7fe61013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6100b6dc0 0x7fe61013e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 -- 172.21.15.187:0/2002788432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61013f440 con 0x7fe610066070 2020-11-20T12:09:56.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe60ffff700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe610065c90 0x7fe6100ba2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:56.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe614887700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6100b6dc0 0x7fe61013e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:56.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.064+0000 7fe6077fe700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe610094360 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7fe6100b1b90 tx=0x7fe5f8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:56.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.065+0000 7fe60cff9700 1 -- 172.21.15.187:0/2002788432 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f801e460 con 0x7fe610066070 2020-11-20T12:09:56.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.065+0000 7fe60cff9700 1 -- 172.21.15.187:0/2002788432 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f801e5c0 con 0x7fe610066070 2020-11-20T12:09:56.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.065+0000 7fe60cff9700 1 -- 172.21.15.187:0/2002788432 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f801e980 con 0x7fe610066070 2020-11-20T12:09:56.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.065+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6100ba7b0 con 0x7fe610066070 2020-11-20T12:09:56.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.065+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe610094a10 con 0x7fe610066070 2020-11-20T12:09:56.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.066+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe610066850 con 0x7fe610066070 2020-11-20T12:09:56.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.067+0000 7fe60cff9700 1 -- 172.21.15.187:0/2002788432 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe5f8025030 con 0x7fe610066070 2020-11-20T12:09:56.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.067+0000 7fe60cff9700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe5ec056d20 0x7fe5ec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.067+0000 7fe60cff9700 1 -- 172.21.15.187:0/2002788432 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe5f8071cb0 con 0x7fe610066070 2020-11-20T12:09:56.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.067+0000 7fe60ffff700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe5ec056d20 0x7fe5ec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.068+0000 7fe60ffff700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe5ec056d20 0x7fe5ec0591e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fe60000b7f0 tx=0x7fe600008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:56.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.071+0000 7fe60cff9700 1 -- 172.21.15.187:0/2002788432 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe5f80470b0 con 0x7fe610066070 2020-11-20T12:09:56.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.315+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6100b8c00 con 0x7fe610066070 2020-11-20T12:09:56.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.316+0000 7fe60cff9700 1 -- 172.21.15.187:0/2002788432 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fe5f8023020 con 0x7fe610066070 2020-11-20T12:09:56.320 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:56.320 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:56.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe5ec056d20 msgr2=0x7fe5ec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe5ec056d20 0x7fe5ec0591e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fe60000b7f0 tx=0x7fe600008040).stop 2020-11-20T12:09:56.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 msgr2=0x7fe610094360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe610094360 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7fe6100b1b90 tx=0x7fe5f8015000).stop 2020-11-20T12:09:56.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 shutdown_connections 2020-11-20T12:09:56.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe5ec056d20 0x7fe5ec0591e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6100b6dc0 0x7fe61013e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe610065c90 0x7fe6100ba2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 --2- 172.21.15.187:0/2002788432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe610066070 0x7fe610094360 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.318+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 >> 172.21.15.187:0/2002788432 conn(0x7fe6100b8280 msgr2=0x7fe6100b9060 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:56.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.319+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 shutdown_connections 2020-11-20T12:09:56.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.319+0000 7fe615088700 1 -- 172.21.15.187:0/2002788432 wait complete. 2020-11-20T12:09:56.326 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:56.344 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:56.614 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2249,"msgr_send_messages":2249,"msgr_recv_bytes":945382,"msgr_send_bytes":13463586,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.398396053,"msgr_running_send_time":0.224495389,"msgr_running_recv_time":0.072516442,"msgr_running_fast_dispatch_time":0.075852693,"msgr_send_messages_queue_lat":{"avgcount":2248,"sum":1.236317026,"avgtime":0.000549963},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11745,"msgr_send_messages":24177,"msgr_recv_bytes":3325057,"msgr_send_bytes":17219378,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":5.191368670,"msgr_running_send_time":4.346780468,"msgr_running_recv_time":3.832938007,"msgr_running_fast_dispatch_time":0.041951876,"msgr_send_messages_queue_lat":{"avgcount":24175,"sum":2.788146379,"avgtime":0.000115331},"msgr_handle_ack_lat":{"avgcount":19075,"sum":0.009200822,"avgtime":0.000000482}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1185,"msgr_send_messages":1181,"msgr_recv_bytes":404662,"msgr_send_bytes":5392467,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.270091317,"msgr_running_send_time":0.105974932,"msgr_running_recv_time":0.053826370,"msgr_running_fast_dispatch_time":0.107403071,"msgr_send_messages_queue_lat":{"avgcount":1180,"sum":0.555746542,"avgtime":0.000470971},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":254,"sum":4.287930645,"avgtime":0.016881616}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2976,"sum":1.070157956,"avgtime":0.000359596}},"mds":{"request":6511,"reply":6509,"reply_latency":{"avgcount":6509,"sum":14.094907452,"avgtime":0.002165448},"forward":0,"dir_fetch":12,"dir_commit":100,"dir_split":11,"dir_merge":1,"inodes":1014,"inodes_top":0,"inodes_bottom":1014,"inodes_pin_tail":0,"inodes_pinned":1014,"inodes_expired":1594,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6523,"traverse_hit":6512,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":157834,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":85,"ceph_cap_op_grant":7023,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4015,"handle_client_caps_dirty":3000,"handle_client_cap_release":38,"process_request_cap_release":3309},"mds_cache":{"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":12,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":13952,"evex":11773,"evtrm":11773,"ev":2181,"evexg":0,"evexd":11773,"segadd":16,"segex":14,"segtrm":14,"seg":3,"segexg":0,"segexd":14,"expos":28766476,"wrpos":31686487,"rdpos":4195196,"jlat":{"avgcount":48,"sum":1.406295715,"avgtime":0.029297827},"replayed":2},"mds_mem":{"ino":892,"ino+":3214,"ino-":2322,"dir":26,"dir+":102,"dir-":76,"dn":1015,"dn+":6412,"dn-":5397,"cap":1,"cap+":3202,"cap-":3201,"rss":88124,"heap":356572},"mds_server":{"dispatch_client_request":6593,"dispatch_server_request":0,"handle_client_request":6511,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.205110729,"avgtime":0.205110729},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.611457497,"avgtime":0.002066080},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":777},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":11788138,"buffer_anon_items":3998,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":4736790,"mds_co_items":87898,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4549,"op_send_bytes":29253182,"op_resend":0,"op_reply":4549,"oplen_avg":{"avgcount":4549,"sum":6715},"op":4549,"op_r":47,"op_w":4502,"op_rmw":0,"op_pg":0,"osdop_stat":10,"osdop_create":1012,"osdop_read":29,"osdop_write":69,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":3,"osdop_delete":3296,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":2024,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":207,"omap_rd":28,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":39,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15175,"get_sum":3476053,"get_or_fail_fail":0,"get_or_fail_success":15175,"take":0,"take_sum":0,"put":15175,"put_sum":3476053,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4549,"take_sum":29577431,"put":1238,"put_sum":29577431,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4549,"take_sum":4549,"put":4549,"put_sum":4549,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323317,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":21,"put_sum":323317,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":533253,"max":3758096384,"get_started":0,"get":13952,"get_sum":27491027,"get_or_fail_fail":0,"get_or_fail_success":13952,"take":0,"take_sum":0,"put":48,"put_sum":26957774,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:56.616 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 945382, "msgr_recv_messages": 2249, "msgr_running_fast_dispatch_time": 0.075852693, "msgr_running_recv_time": 0.072516442, "msgr_running_send_time": 0.224495389, "msgr_running_total_time": 0.398396053, "msgr_send_bytes": 13463586, "msgr_send_messages": 2249, "msgr_send_messages_queue_lat": { "avgcount": 2248, "avgtime": 0.000549963, "sum": 1.236317026 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19075, "avgtime": 4.82e-07, "sum": 0.009200822 }, "msgr_recv_bytes": 3325057, "msgr_recv_messages": 11745, "msgr_running_fast_dispatch_time": 0.041951876, "msgr_running_recv_time": 3.832938007, "msgr_running_send_time": 4.346780468, "msgr_running_total_time": 5.19136867, "msgr_send_bytes": 17219378, "msgr_send_messages": 24177, "msgr_send_messages_queue_lat": { "avgcount": 24175, "avgtime": 0.000115331, "sum": 2.788146379 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 404662, "msgr_recv_messages": 1185, "msgr_running_fast_dispatch_time": 0.107403071, "msgr_running_recv_time": 0.05382637, "msgr_running_send_time": 0.105974932, "msgr_running_total_time": 0.270091317, "msgr_send_bytes": 5392467, "msgr_send_messages": 1181, "msgr_send_messages_queue_lat": { "avgcount": 1180, "avgtime": 0.000470971, "sum": 0.555746542 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 254, "avgtime": 0.016881616, "sum": 4.287930645 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2976, "avgtime": 0.000359596, "sum": 1.070157956 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7023, "ceph_cap_op_revoke": 85, "ceph_cap_op_trunc": 0, "dir_commit": 100, "dir_fetch": 12, "dir_merge": 1, "dir_split": 11, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 38, "handle_client_caps": 4015, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1014, "inodes_bottom": 1014, "inodes_expired": 1594, "inodes_pin_tail": 0, "inodes_pinned": 1014, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 157834, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3309, "q": 0, "reply": 6509, "reply_latency": { "avgcount": 6509, "avgtime": 0.002165448, "sum": 14.094907452 }, "request": 6511, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 6523, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6512, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 12, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 2181, "evadd": 13952, "evex": 11773, "evexd": 11773, "evexg": 0, "evtrm": 11773, "expos": 28766476, "jlat": { "avgcount": 48, "avgtime": 0.029297827, "sum": 1.406295715 }, "rdpos": 4195196, "replayed": 2, "seg": 3, "segadd": 16, "segex": 14, "segexd": 14, "segexg": 0, "segtrm": 14, "wrpos": 31686487 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 26, "dir+": 102, "dir-": 76, "dn": 1015, "dn+": 6412, "dn-": 5397, "heap": 356572, "ino": 892, "ino+": 3214, "ino-": 2322, "rss": 88124 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6593, "dispatch_server_request": 0, "handle_client_request": 6511, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.205110729, "sum": 0.205110729 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.00206608, "sum": 6.611457497 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 777, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 11788138, "buffer_anon_items": 3998, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 4736790, "mds_co_items": 87898, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 28, "omap_wr": 207, "op": 4549, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 47, "op_reply": 4549, "op_resend": 0, "op_rmw": 0, "op_send": 4549, "op_send_bytes": 29253182, "op_w": 4502, "oplen_avg": { "avgcount": 4549, "sum": 6715 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1012, "osdop_delete": 3296, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 29, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2024, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 10, "osdop_truncate": 3, "osdop_watch": 0, "osdop_write": 69, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 39, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15175, "get_or_fail_fail": 0, "get_or_fail_success": 15175, "get_started": 0, "get_sum": 3476053, "max": 104857600, "put": 15175, "put_sum": 3476053, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1238, "put_sum": 29577431, "take": 4549, "take_sum": 29577431, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4549, "put_sum": 4549, "take": 4549, "take_sum": 4549, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323317, "max": 3758096384, "put": 21, "put_sum": 323317, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 13952, "get_or_fail_fail": 0, "get_or_fail_success": 13952, "get_started": 0, "get_sum": 27491027, "max": 3758096384, "put": 48, "put_sum": 26957774, "take": 0, "take_sum": 0, "val": 533253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:56.618 INFO:tasks.cephfs.test_strays:Complete purge in 8 seconds 2020-11-20T12:09:56.618 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:56.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.851+0000 7fcddc878700 1 Processor -- start 2020-11-20T12:09:56.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.851+0000 7fcddc878700 1 -- start start 2020-11-20T12:09:56.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.852+0000 7fcddc878700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd8066070 0x7fcdd8144070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.852+0000 7fcddc878700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdd80b5750 0x7fcdd809a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.852+0000 7fcddc878700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd8149f60 con 0x7fcdd8065c90 2020-11-20T12:09:56.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.852+0000 7fcddc878700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd814a200 con 0x7fcdd80b5750 2020-11-20T12:09:56.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.852+0000 7fcddc878700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd814a4a0 con 0x7fcdd8066070 2020-11-20T12:09:56.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdd7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdd80b5750 0x7fcdd809a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdd7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdd80b5750 0x7fcdd809a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45078/0 (socket says 172.21.15.187:45078) 2020-11-20T12:09:56.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdd7fff700 1 -- 172.21.15.187:0/479839203 learned_addr learned my addr 172.21.15.187:0/479839203 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:56.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdceffd700 1 --2- 172.21.15.187:0/479839203 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd8066070 0x7fcdd8144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdd7fff700 1 -- 172.21.15.187:0/479839203 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd8066070 msgr2=0x7fcdd8144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:56.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdd7fff700 1 --2- 172.21.15.187:0/479839203 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd8066070 0x7fcdd8144070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdd7fff700 1 -- 172.21.15.187:0/479839203 >> v1:172.21.15.5:6789/0 conn(0x7fcdd8065c90 legacy=0x7fcdd8143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:56.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdd7fff700 1 -- 172.21.15.187:0/479839203 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd814a940 con 0x7fcdd80b5750 2020-11-20T12:09:56.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.853+0000 7fcdd7fff700 1 --2- 172.21.15.187:0/479839203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd80b5750 0x7fcdd809a8e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fcdd0009d50 tx=0x7fcdd0009290).ready entity=mon.1 client_cookie=f406efd616c1f13 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:56.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.854+0000 7fcdd6ffd700 1 -- 172.21.15.187:0/479839203 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd000bcd0 con 0x7fcdd80b5750 2020-11-20T12:09:56.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.854+0000 7fcdd6ffd700 1 -- 172.21.15.187:0/479839203 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd000be30 con 0x7fcdd80b5750 2020-11-20T12:09:56.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.854+0000 7fcdd6ffd700 1 -- 172.21.15.187:0/479839203 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0011610 con 0x7fcdd80b5750 2020-11-20T12:09:56.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.854+0000 7fcddc878700 1 -- 172.21.15.187:0/479839203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd80b5750 msgr2=0x7fcdd809a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.854+0000 7fcddc878700 1 --2- 172.21.15.187:0/479839203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd80b5750 0x7fcdd809a8e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fcdd0009d50 tx=0x7fcdd0009290).stop 2020-11-20T12:09:56.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.854+0000 7fcddc878700 1 -- 172.21.15.187:0/479839203 shutdown_connections 2020-11-20T12:09:56.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.855+0000 7fcddc878700 1 --2- 172.21.15.187:0/479839203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd80b5750 0x7fcdd809a8e0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.855+0000 7fcddc878700 1 --2- 172.21.15.187:0/479839203 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd8066070 0x7fcdd8144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.855+0000 7fcddc878700 1 -- 172.21.15.187:0/479839203 >> 172.21.15.187:0/479839203 conn(0x7fcdd80b8280 msgr2=0x7fcdd80653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:56.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.855+0000 7fcddc878700 1 -- 172.21.15.187:0/479839203 shutdown_connections 2020-11-20T12:09:56.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.855+0000 7fcddc878700 1 -- 172.21.15.187:0/479839203 wait complete. 2020-11-20T12:09:56.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.856+0000 7fcddc878700 1 Processor -- start 2020-11-20T12:09:56.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.856+0000 7fcddc878700 1 -- start start 2020-11-20T12:09:56.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.857+0000 7fcddc878700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd8065c90 0x7fcdd809ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.857+0000 7fcddc878700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd8066070 0x7fcdd815b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.857+0000 7fcddc878700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd80b5750 0x7fcdd8094f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.857+0000 7fcddc878700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b3030 con 0x7fcdd8066070 2020-11-20T12:09:56.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.857+0000 7fcddc878700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd809ef60 con 0x7fcdd8065c90 2020-11-20T12:09:56.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.857+0000 7fcddc878700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b3260 con 0x7fcdd80b5750 2020-11-20T12:09:56.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.857+0000 7fcdd77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd8065c90 0x7fcdd809ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.857+0000 7fcdd77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd8065c90 0x7fcdd809ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45080/0 (socket says 172.21.15.187:45080) 2020-11-20T12:09:56.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdd77fe700 1 -- 172.21.15.187:0/109502397 learned_addr learned my addr 172.21.15.187:0/109502397 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:56.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdd77fe700 1 -- 172.21.15.187:0/109502397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd80b5750 msgr2=0x7fcdd8094f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdd77fe700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd80b5750 0x7fcdd8094f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:56.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdd77fe700 1 -- 172.21.15.187:0/109502397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd8066070 msgr2=0x7fcdd815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdd77fe700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd8066070 0x7fcdd815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:56.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdd77fe700 1 -- 172.21.15.187:0/109502397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80b3500 con 0x7fcdd8065c90 2020-11-20T12:09:56.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdd77fe700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd8065c90 0x7fcdd809ea70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7fcdc80096e0 tx=0x7fcdc8006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:56.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdcffff700 1 -- 172.21.15.187:0/109502397 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc800d560 con 0x7fcdd8065c90 2020-11-20T12:09:56.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdcffff700 1 -- 172.21.15.187:0/109502397 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc8000e90 con 0x7fcdd8065c90 2020-11-20T12:09:56.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcdcffff700 1 -- 172.21.15.187:0/109502397 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc800d930 con 0x7fcdd8065c90 2020-11-20T12:09:56.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.858+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd80b37a0 con 0x7fcdd8065c90 2020-11-20T12:09:56.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.859+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd815beb0 con 0x7fcdd8065c90 2020-11-20T12:09:56.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.860+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd80b75b0 con 0x7fcdd8065c90 2020-11-20T12:09:56.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.860+0000 7fcdcffff700 1 -- 172.21.15.187:0/109502397 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcdc800f030 con 0x7fcdd8065c90 2020-11-20T12:09:56.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.861+0000 7fcdcffff700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcda8056d20 0x7fcda80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:56.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.861+0000 7fcdcffff700 1 -- 172.21.15.187:0/109502397 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcdc806fc00 con 0x7fcdd8065c90 2020-11-20T12:09:56.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.861+0000 7fcdceffd700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcda8056d20 0x7fcda80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:56.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.862+0000 7fcdceffd700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcda8056d20 0x7fcda80591e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fcdd815b760 tx=0x7fcdc0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:56.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:56.865+0000 7fcdcffff700 1 -- 172.21.15.187:0/109502397 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcdc8075020 con 0x7fcdd8065c90 2020-11-20T12:09:57.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.111+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd8064170 con 0x7fcdd8065c90 2020-11-20T12:09:57.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.113+0000 7fcdcffff700 1 -- 172.21.15.187:0/109502397 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fcdc8075870 con 0x7fcdd8065c90 2020-11-20T12:09:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:57.116 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:09:57.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcda8056d20 msgr2=0x7fcda80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:57.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcda8056d20 0x7fcda80591e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fcdd815b760 tx=0x7fcdc0008040).stop 2020-11-20T12:09:57.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd8065c90 msgr2=0x7fcdd809ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:57.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd8065c90 0x7fcdd809ea70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7fcdc80096e0 tx=0x7fcdc8006d70).stop 2020-11-20T12:09:57.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 shutdown_connections 2020-11-20T12:09:57.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcda8056d20 0x7fcda80591e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:57.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd80b5750 0x7fcdd8094f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:57.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd8065c90 0x7fcdd809ea70 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:57.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 --2- 172.21.15.187:0/109502397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd8066070 0x7fcdd815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:57.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.115+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 >> 172.21.15.187:0/109502397 conn(0x7fcdd80b8280 msgr2=0x7fcdd809a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:57.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.116+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 shutdown_connections 2020-11-20T12:09:57.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:57.116+0000 7fcddc878700 1 -- 172.21.15.187:0/109502397 wait complete. 2020-11-20T12:09:57.122 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:09:57.141 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:09:57.416 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2249,"msgr_send_messages":2249,"msgr_recv_bytes":945382,"msgr_send_bytes":13463586,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.398396053,"msgr_running_send_time":0.224495389,"msgr_running_recv_time":0.072516442,"msgr_running_fast_dispatch_time":0.075852693,"msgr_send_messages_queue_lat":{"avgcount":2248,"sum":1.236317026,"avgtime":0.000549963},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11746,"msgr_send_messages":24177,"msgr_recv_bytes":3325257,"msgr_send_bytes":17219378,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":5.191580048,"msgr_running_send_time":4.346885135,"msgr_running_recv_time":3.832978916,"msgr_running_fast_dispatch_time":0.042013340,"msgr_send_messages_queue_lat":{"avgcount":24175,"sum":2.788146379,"avgtime":0.000115331},"msgr_handle_ack_lat":{"avgcount":19075,"sum":0.009200822,"avgtime":0.000000482}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1185,"msgr_send_messages":1181,"msgr_recv_bytes":404662,"msgr_send_bytes":5392467,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.270091317,"msgr_running_send_time":0.105974932,"msgr_running_recv_time":0.053826370,"msgr_running_fast_dispatch_time":0.107403071,"msgr_send_messages_queue_lat":{"avgcount":1180,"sum":0.555746542,"avgtime":0.000470971},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":254,"sum":4.287930645,"avgtime":0.016881616}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2976,"sum":1.070157956,"avgtime":0.000359596}},"mds":{"request":6511,"reply":6509,"reply_latency":{"avgcount":6509,"sum":14.094907452,"avgtime":0.002165448},"forward":0,"dir_fetch":12,"dir_commit":100,"dir_split":11,"dir_merge":1,"inodes":1014,"inodes_top":0,"inodes_bottom":1014,"inodes_pin_tail":0,"inodes_pinned":1014,"inodes_expired":1594,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6523,"traverse_hit":6512,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":157834,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":85,"ceph_cap_op_grant":7023,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4015,"handle_client_caps_dirty":3000,"handle_client_cap_release":38,"process_request_cap_release":3309},"mds_cache":{"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":12,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":13952,"evex":11773,"evtrm":11773,"ev":2181,"evexg":0,"evexd":11773,"segadd":16,"segex":14,"segtrm":14,"seg":3,"segexg":0,"segexd":14,"expos":28766476,"wrpos":31686487,"rdpos":4195196,"jlat":{"avgcount":48,"sum":1.406295715,"avgtime":0.029297827},"replayed":2},"mds_mem":{"ino":892,"ino+":3214,"ino-":2322,"dir":26,"dir+":102,"dir-":76,"dn":1015,"dn+":6412,"dn-":5397,"cap":1,"cap+":3202,"cap-":3201,"rss":88124,"heap":356572},"mds_server":{"dispatch_client_request":6593,"dispatch_server_request":0,"handle_client_request":6511,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":7.000802562,"avgtime":0.002187750},"req_getattr_latency":{"avgcount":2,"sum":0.001534461,"avgtime":0.000767230},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.006277551,"avgtime":0.006277551},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":105,"sum":0.269724652,"avgtime":0.002568806},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.205110729,"avgtime":0.205110729},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.611457497,"avgtime":0.002066080},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5874,"average_load":5874,"avg_session_uptime":777},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":11788138,"buffer_anon_items":3998,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":4736790,"mds_co_items":87898,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4549,"op_send_bytes":29253182,"op_resend":0,"op_reply":4549,"oplen_avg":{"avgcount":4549,"sum":6715},"op":4549,"op_r":47,"op_w":4502,"op_rmw":0,"op_pg":0,"osdop_stat":10,"osdop_create":1012,"osdop_read":29,"osdop_write":69,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":3,"osdop_delete":3296,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":2024,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":207,"omap_rd":28,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":39,"pq_executing":0,"pq_executing_high_water":6,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15176,"get_sum":3476179,"get_or_fail_fail":0,"get_or_fail_success":15176,"take":0,"take_sum":0,"put":15176,"put_sum":3476179,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4549,"take_sum":29577431,"put":1238,"put_sum":29577431,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4549,"take_sum":4549,"put":4549,"put_sum":4549,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323317,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":21,"put_sum":323317,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x557a1e5560a0":{"val":533253,"max":3758096384,"get_started":0,"get":13952,"get_sum":27491027,"get_or_fail_fail":0,"get_or_fail_success":13952,"take":0,"take_sum":0,"put":48,"put_sum":26957774,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:09:57.418 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 945382, "msgr_recv_messages": 2249, "msgr_running_fast_dispatch_time": 0.075852693, "msgr_running_recv_time": 0.072516442, "msgr_running_send_time": 0.224495389, "msgr_running_total_time": 0.398396053, "msgr_send_bytes": 13463586, "msgr_send_messages": 2249, "msgr_send_messages_queue_lat": { "avgcount": 2248, "avgtime": 0.000549963, "sum": 1.236317026 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19075, "avgtime": 4.82e-07, "sum": 0.009200822 }, "msgr_recv_bytes": 3325257, "msgr_recv_messages": 11746, "msgr_running_fast_dispatch_time": 0.04201334, "msgr_running_recv_time": 3.832978916, "msgr_running_send_time": 4.346885135, "msgr_running_total_time": 5.191580048, "msgr_send_bytes": 17219378, "msgr_send_messages": 24177, "msgr_send_messages_queue_lat": { "avgcount": 24175, "avgtime": 0.000115331, "sum": 2.788146379 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 404662, "msgr_recv_messages": 1185, "msgr_running_fast_dispatch_time": 0.107403071, "msgr_running_recv_time": 0.05382637, "msgr_running_send_time": 0.105974932, "msgr_running_total_time": 0.270091317, "msgr_send_bytes": 5392467, "msgr_send_messages": 1181, "msgr_send_messages_queue_lat": { "avgcount": 1180, "avgtime": 0.000470971, "sum": 0.555746542 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 254, "avgtime": 0.016881616, "sum": 4.287930645 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2976, "avgtime": 0.000359596, "sum": 1.070157956 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7023, "ceph_cap_op_revoke": 85, "ceph_cap_op_trunc": 0, "dir_commit": 100, "dir_fetch": 12, "dir_merge": 1, "dir_split": 11, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 38, "handle_client_caps": 4015, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1014, "inodes_bottom": 1014, "inodes_expired": 1594, "inodes_pin_tail": 0, "inodes_pinned": 1014, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 157834, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3309, "q": 0, "reply": 6509, "reply_latency": { "avgcount": 6509, "avgtime": 0.002165448, "sum": 14.094907452 }, "request": 6511, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 6523, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6512, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 12, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 2181, "evadd": 13952, "evex": 11773, "evexd": 11773, "evexg": 0, "evtrm": 11773, "expos": 28766476, "jlat": { "avgcount": 48, "avgtime": 0.029297827, "sum": 1.406295715 }, "rdpos": 4195196, "replayed": 2, "seg": 3, "segadd": 16, "segex": 14, "segexd": 14, "segexg": 0, "segtrm": 14, "wrpos": 31686487 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 26, "dir+": 102, "dir-": 76, "dn": 1015, "dn+": 6412, "dn-": 5397, "heap": 356572, "ino": 892, "ino+": 3214, "ino-": 2322, "rss": 88124 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6593, "dispatch_server_request": 0, "handle_client_request": 6511, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.00218775, "sum": 7.000802562 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.00076723, "sum": 0.001534461 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.006277551, "sum": 0.006277551 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 105, "avgtime": 0.002568806, "sum": 0.269724652 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.205110729, "sum": 0.205110729 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.00206608, "sum": 6.611457497 } }, "mds_sessions": { "average_load": 5874, "avg_session_uptime": 777, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5874 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 11788138, "buffer_anon_items": 3998, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 4736790, "mds_co_items": 87898, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 28, "omap_wr": 207, "op": 4549, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 47, "op_reply": 4549, "op_resend": 0, "op_rmw": 0, "op_send": 4549, "op_send_bytes": 29253182, "op_w": 4502, "oplen_avg": { "avgcount": 4549, "sum": 6715 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1012, "osdop_delete": 3296, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 29, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2024, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 10, "osdop_truncate": 3, "osdop_watch": 0, "osdop_write": 69, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 39, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15176, "get_or_fail_fail": 0, "get_or_fail_success": 15176, "get_started": 0, "get_sum": 3476179, "max": 104857600, "put": 15176, "put_sum": 3476179, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1238, "put_sum": 29577431, "take": 4549, "take_sum": 29577431, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4549, "put_sum": 4549, "take": 4549, "take_sum": 4549, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323317, "max": 3758096384, "put": 21, "put_sum": 323317, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x557a1e5560a0": { "get": 13952, "get_or_fail_fail": 0, "get_or_fail_success": 13952, "get_started": 0, "get_sum": 27491027, "max": 3758096384, "put": 48, "put_sum": 26957774, "take": 0, "take_sum": 0, "val": 533253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:09:57.419 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:09:57.419 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:09:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:09:57.554 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:09:57.683 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:09:57.707 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:09:57.708 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:09:57.709 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:09:57.709 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:09:57.903 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:09:57.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:09:57.904 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:09:57.961 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:57.970 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:58.155 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:09:58.155 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:09:58.156 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:58.156 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:09:58.156 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.156 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:09:58.156 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.157 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:58.157 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.157 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:58.157 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:09:58.158 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.158 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:09:58.158 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:09:58.158 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.159 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:58.159 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:58.159 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.159 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:58.159 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:58.160 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.160 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:09:58.160 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.161 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:58.161 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:09:58.161 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:58.161 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:58.161 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:58.162 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:58.162 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:09:58.162 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:58.162 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:58.162 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:58.163 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.163 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:58.163 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.163 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:09:58.163 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.164 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:09:58.164 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.164 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:09:58.164 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:09:58.165 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:09:58.165 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:58.165 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:09:58.165 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:58.165 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.166 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:09:58.166 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:09:58.166 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.166 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:58.166 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:09:58.167 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.167 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:09:58.167 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:58.167 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.168 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:09:58.168 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:09:58.168 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:58.168 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.168 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:09:58.169 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:09:58.169 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.169 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:58.169 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.169 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:09:58.170 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.170 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:09:58.170 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.170 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:58.170 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:58.171 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.171 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:09:58.171 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:09:58.171 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:09:58.171 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:09:58.172 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:09:58.172 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:09:58.172 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:09:58.172 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:09:58.173 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:09:58.173 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:09:58.173 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:09:58.173 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:09:58.173 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:09:58.174 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:09:58.174 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:58.174 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:58.174 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:58.174 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:58.175 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:09:58.175 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:58.175 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.175 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:09:58.175 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.176 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:09:58.176 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:58.176 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:58.176 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.176 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.177 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:58.177 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.177 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:09:58.177 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.178 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:58.178 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:09:58.178 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:58.178 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:09:58.179 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:58.179 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.179 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:58.179 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:09:58.180 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.180 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:09:58.180 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.180 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:09:58.181 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:09:58.181 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:09:58.181 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:09:58.181 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:58.182 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:58.182 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:09:58.182 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:09:58.182 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.182 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:09:58.183 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.183 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.183 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:09:58.184 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:09:58.184 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:09:58.184 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:09:58.184 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:09:58.185 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:09:58.186 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:09:58.186 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.186 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:09:58.186 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:58.187 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:09:58.187 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:09:58.187 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:09:58.187 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:09:58.187 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:09:58.188 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:09:58.188 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:09:58.188 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:58.188 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.189 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.189 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 6 2020-11-20T12:09:58.189 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops = 100000000 2020-11-20T12:09:58.190 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:09:58.190 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:09:58.190 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:58.190 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:09:58.191 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.191 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:09:58.191 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.191 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:58.191 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.192 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:58.192 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:09:58.192 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.192 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:09:58.193 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:09:58.193 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.193 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:58.193 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:58.193 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.194 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:58.194 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:58.194 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.194 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:09:58.194 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.195 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:58.195 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:09:58.195 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:58.195 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:58.195 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:58.196 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:58.196 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:09:58.196 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:58.196 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:58.197 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:58.197 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.197 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:58.197 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.197 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:09:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:09:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:09:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:09:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:09:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:09:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.200 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:09:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:09:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:09:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:09:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:09:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:09:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:09:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:09:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:09:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:09:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:09:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:09:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:09:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:09:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:09:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:09:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:09:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:09:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:09:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:09:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:09:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:09:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:09:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:09:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:09:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.210 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:09:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:09:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.212 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.212 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:58.212 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.212 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:09:58.212 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.213 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:58.213 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:09:58.213 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:58.213 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:09:58.213 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:58.214 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.214 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:58.214 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:09:58.214 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.214 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:09:58.215 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.215 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:09:58.215 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:09:58.215 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:09:58.215 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:09:58.216 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:58.216 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:58.216 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:09:58.216 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:09:58.216 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.217 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:09:58.217 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.217 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.217 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:09:58.217 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:09:58.218 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:09:58.218 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:09:58.218 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:09:58.218 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:09:58.218 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:09:58.219 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.219 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:09:58.219 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:58.219 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:09:58.219 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:09:58.220 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:09:58.220 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:09:58.220 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:09:58.220 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:09:58.220 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:09:58.221 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:58.221 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.221 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.221 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 6 2020-11-20T12:09:58.221 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops = 100000000 2020-11-20T12:09:58.240 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:58.248 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:58.450 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:09:58.451 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:09:58.451 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:58.451 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:09:58.451 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.452 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:09:58.452 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.452 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:58.452 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.452 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:58.453 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:09:58.453 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.453 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:09:58.453 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:09:58.453 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.454 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:58.454 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:58.454 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.454 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:58.455 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:58.455 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.455 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:09:58.455 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.455 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:58.456 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:09:58.456 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:58.456 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:58.456 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:58.456 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:58.457 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:09:58.457 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:58.457 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:58.457 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:58.457 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.458 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:58.458 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.458 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:09:58.458 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.458 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:09:58.459 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.459 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:09:58.459 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:09:58.459 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:09:58.459 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:58.460 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:09:58.460 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:58.460 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.460 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:09:58.460 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:09:58.461 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.461 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:58.461 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:09:58.461 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.461 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:09:58.462 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:58.462 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.462 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:09:58.462 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:09:58.462 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:58.463 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.463 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:09:58.463 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:09:58.463 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.463 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:58.464 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.464 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:09:58.464 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.465 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:09:58.465 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.465 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:58.465 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:58.465 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.466 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:09:58.466 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:09:58.466 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:09:58.466 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:09:58.466 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:09:58.467 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:09:58.467 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:09:58.467 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:09:58.467 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:09:58.467 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:09:58.468 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:09:58.468 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:09:58.468 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:09:58.468 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:09:58.468 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:58.469 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:58.469 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:58.469 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:58.469 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:09:58.469 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:58.470 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.470 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:09:58.470 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.470 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:09:58.470 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:58.471 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:58.471 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.471 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.471 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:58.471 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.471 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:09:58.472 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.472 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:58.472 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:09:58.472 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:58.472 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:09:58.473 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:58.473 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.473 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:58.473 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:09:58.474 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.474 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:09:58.474 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.474 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:09:58.474 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:09:58.474 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:09:58.475 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:09:58.475 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:58.475 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:58.475 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:09:58.475 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:09:58.476 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.476 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:09:58.476 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.476 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.476 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:09:58.477 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:09:58.477 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:09:58.477 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:09:58.477 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:09:58.477 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:09:58.478 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:09:58.478 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.478 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:09:58.478 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:58.478 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:09:58.479 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:09:58.479 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:09:58.479 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:09:58.479 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:09:58.479 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:09:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:09:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.480 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops = 100000000 2020-11-20T12:09:58.481 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:09:58.481 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:09:58.482 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:58.482 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:09:58.482 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.482 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:09:58.482 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.483 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:58.483 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.483 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:58.483 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:09:58.483 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.484 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:09:58.484 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:09:58.484 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.484 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:58.484 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:58.485 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.485 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:58.485 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:58.485 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.485 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:09:58.486 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.486 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:58.486 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:09:58.486 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:58.486 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:58.487 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:58.487 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:58.487 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:09:58.487 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:58.487 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:58.488 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:58.488 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.488 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:58.488 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.488 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:09:58.489 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.489 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:09:58.489 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.489 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:09:58.489 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:09:58.490 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:09:58.490 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:58.490 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:09:58.490 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:58.490 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.491 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:09:58.491 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:09:58.491 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.491 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:58.491 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:09:58.491 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.492 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:09:58.492 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:58.492 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.492 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:09:58.492 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:09:58.493 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:58.493 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.493 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:09:58.493 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:09:58.493 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.494 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:58.494 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.494 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:09:58.494 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.494 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:09:58.495 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.495 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:58.495 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:58.495 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.495 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:09:58.496 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:09:58.496 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:09:58.496 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:09:58.496 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:09:58.496 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:09:58.497 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:09:58.497 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:09:58.497 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:09:58.497 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:09:58.497 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:09:58.498 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:09:58.498 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:09:58.498 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:09:58.498 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:58.498 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:58.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:58.499 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:58.499 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:09:58.499 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:58.499 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.500 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:09:58.500 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.500 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:09:58.500 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:58.500 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:58.500 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.501 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.501 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:58.501 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.501 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:09:58.501 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.502 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:58.502 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:09:58.502 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:58.502 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:09:58.502 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:58.503 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.503 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:58.503 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:09:58.503 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.503 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:09:58.504 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.504 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:09:58.504 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:09:58.504 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:09:58.504 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:09:58.505 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:58.505 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:58.505 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:09:58.505 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:09:58.505 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.506 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:09:58.506 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.506 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.506 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:09:58.506 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:09:58.507 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:09:58.507 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:09:58.507 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:09:58.507 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:09:58.507 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:09:58.508 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.508 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:09:58.508 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:58.508 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:09:58.508 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:09:58.508 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:09:58.509 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:09:58.509 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:09:58.509 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:09:58.509 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:09:58.510 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:58.510 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.510 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.510 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops = 100000000 2020-11-20T12:09:58.527 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:58.539 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:09:58.742 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:09:58.743 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:09:58.743 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:58.744 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:09:58.744 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.744 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:09:58.745 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.745 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:58.746 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.746 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:58.746 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:09:58.747 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.747 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:09:58.747 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:09:58.748 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.748 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:58.748 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:58.749 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.749 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:58.750 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:58.750 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.750 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:09:58.751 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.751 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:58.751 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:09:58.751 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:58.752 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:58.752 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:58.752 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:58.752 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:09:58.752 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:58.753 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:58.753 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:58.753 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.753 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:58.754 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.754 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:09:58.754 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.755 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:09:58.755 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.755 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:09:58.755 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:09:58.755 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:09:58.756 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:58.756 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:09:58.756 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:58.756 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.757 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:09:58.757 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:09:58.757 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.758 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:58.758 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:09:58.758 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.758 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:09:58.758 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:58.759 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.759 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:09:58.759 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:09:58.759 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:58.760 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.760 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:09:58.760 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:09:58.760 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.761 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:58.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.761 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:09:58.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.761 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:09:58.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.762 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:58.762 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:58.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.763 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:09:58.763 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:09:58.763 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:09:58.763 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:09:58.764 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:09:58.764 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:09:58.764 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:09:58.765 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:09:58.766 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:09:58.766 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:09:58.766 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:09:58.766 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:09:58.767 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:09:58.767 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:09:58.767 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:58.767 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:58.767 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:58.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:58.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:09:58.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:58.768 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.768 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:09:58.769 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.769 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:09:58.769 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:58.769 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:58.769 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.770 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.770 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:58.770 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.770 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:09:58.771 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.771 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:09:58.771 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:09:58.771 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:09:58.771 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:09:58.772 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:58.772 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.772 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:58.772 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:09:58.772 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.773 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:09:58.773 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:09:58.773 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:09:58.773 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:09:58.773 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:09:58.774 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:09:58.774 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:58.774 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:58.774 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:09:58.775 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:09:58.780 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.780 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:09:58.780 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.781 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.782 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:09:58.782 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:09:58.782 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:09:58.782 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:09:58.783 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:09:58.783 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:09:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:09:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:09:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:09:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:09:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:09:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:09:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:09:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:09:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:09:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:09:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:09:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.788 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:09:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:09:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:09:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:09:58.788 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:09:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:09:58.789 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:09:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:09:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:09:58.790 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:09:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:09:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:09:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.791 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:09:58.792 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:09:58.792 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.792 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:09:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:09:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:09:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:09:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:09:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:09:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:09:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:09:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:09:58.794 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:09:58.795 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:09:58.795 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.795 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:09:58.795 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.795 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:09:58.796 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.796 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:09:58.796 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.796 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:09:58.797 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:09:58.798 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:09:58.798 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:09:58.798 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:09:58.798 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:09:58.798 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.799 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:09:58.799 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:09:58.799 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.799 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:09:58.799 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:09:58.800 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.800 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:09:58.800 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:09:58.800 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.800 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:09:58.801 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:09:58.801 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:09:58.801 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.802 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:09:58.802 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:09:58.802 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.802 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:09:58.802 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.803 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:09:58.803 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.803 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:09:58.803 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.803 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:09:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:09:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:09:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:09:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:09:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:09:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:09:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:09:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:09:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:09:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:09:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:09:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:09:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:09:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:09:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:09:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:09:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:09:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:09:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:09:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:09:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.808 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:09:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:09:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:09:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.810 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:09:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:09:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:09:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:09:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:09:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:09:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:09:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:09:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:09:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:09:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:09:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:09:58.813 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:09:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:09:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:09:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:09:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:09:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:09:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:09:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:09:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:09:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.815 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:09:58.815 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:09:58.816 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:09:58.816 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:09:58.816 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:09:58.816 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:09:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:09:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:09:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:09:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:09:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:09:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:09:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:09:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:09:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:09:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:09:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:09:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:09:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:09:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:09:58.821 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:09:59.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.067+0000 7f089359e700 1 Processor -- start 2020-11-20T12:09:59.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.067+0000 7f089359e700 1 -- start start 2020-11-20T12:09:59.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.069+0000 7f089359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f08940bf800 0x7f089414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.069+0000 7f089359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f08940bfbe0 0x7f08941480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.069+0000 7f089359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089414d110 con 0x7f089409f560 2020-11-20T12:09:59.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.069+0000 7f089359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089414cbd0 con 0x7f08940bf800 2020-11-20T12:09:59.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.069+0000 7f089359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089414ce70 con 0x7f08940bfbe0 2020-11-20T12:09:59.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.069+0000 7f0889d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f08940bfbe0 0x7f08941480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.069+0000 7f0889d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f08940bfbe0 0x7f08941480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45090/0 (socket says 172.21.15.187:45090) 2020-11-20T12:09:59.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.069+0000 7f0889d9b700 1 -- 172.21.15.187:0/521550061 learned_addr learned my addr 172.21.15.187:0/521550061 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:59.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.070+0000 7f0889d9b700 1 -- 172.21.15.187:0/521550061 >> v2:172.21.15.5:3300/0 conn(0x7f08940bf800 msgr2=0x7f089414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:59.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.070+0000 7f0889d9b700 1 --2- 172.21.15.187:0/521550061 >> v2:172.21.15.5:3300/0 conn(0x7f08940bf800 0x7f089414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:59.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.070+0000 7f0889d9b700 1 -- 172.21.15.187:0/521550061 >> v1:172.21.15.5:6789/0 conn(0x7f089409f560 legacy=0x7f0894060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:59.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.070+0000 7f0889d9b700 1 -- 172.21.15.187:0/521550061 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08940a4cc0 con 0x7f08940bfbe0 2020-11-20T12:09:59.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.070+0000 7f0889d9b700 1 --2- 172.21.15.187:0/521550061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bfbe0 0x7f08941480c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f08940687b0 tx=0x7f087c006eb0).ready entity=mon.1 client_cookie=522b30f05a8891e7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:59.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.071+0000 7f0891d9b700 1 -- 172.21.15.187:0/521550061 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c009920 con 0x7f08940bfbe0 2020-11-20T12:09:59.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.071+0000 7f0891d9b700 1 -- 172.21.15.187:0/521550061 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c009a80 con 0x7f08940bfbe0 2020-11-20T12:09:59.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.071+0000 7f0891d9b700 1 -- 172.21.15.187:0/521550061 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c009e00 con 0x7f08940bfbe0 2020-11-20T12:09:59.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.071+0000 7f089359e700 1 -- 172.21.15.187:0/521550061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bfbe0 msgr2=0x7f08941480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.071+0000 7f089359e700 1 --2- 172.21.15.187:0/521550061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bfbe0 0x7f08941480c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f08940687b0 tx=0x7f087c006eb0).stop 2020-11-20T12:09:59.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.072+0000 7f089359e700 1 -- 172.21.15.187:0/521550061 shutdown_connections 2020-11-20T12:09:59.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.072+0000 7f089359e700 1 --2- 172.21.15.187:0/521550061 >> v2:172.21.15.5:3300/0 conn(0x7f08940bf800 0x7f089414c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:59.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.072+0000 7f089359e700 1 --2- 172.21.15.187:0/521550061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bfbe0 0x7f08941480c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:59.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.072+0000 7f089359e700 1 -- 172.21.15.187:0/521550061 >> 172.21.15.187:0/521550061 conn(0x7f0894068370 msgr2=0x7f08940bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:59.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.072+0000 7f089359e700 1 -- 172.21.15.187:0/521550061 shutdown_connections 2020-11-20T12:09:59.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.072+0000 7f089359e700 1 -- 172.21.15.187:0/521550061 wait complete. 2020-11-20T12:09:59.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.074+0000 7f089359e700 1 Processor -- start 2020-11-20T12:09:59.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.074+0000 7f089359e700 1 -- start start 2020-11-20T12:09:59.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.075+0000 7f089359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f089409f560 0x7f089413ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.075+0000 7f089359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bf800 0x7f089414a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.075+0000 7f089359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f089414aa20 0x7f08940ba8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.075+0000 7f089359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0894144310 con 0x7f089409f560 2020-11-20T12:09:59.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.075+0000 7f089359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08941445b0 con 0x7f08940bf800 2020-11-20T12:09:59.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.075+0000 7f089359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089413f440 con 0x7f089414aa20 2020-11-20T12:09:59.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f0889d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bf800 0x7f089414a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f0889d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bf800 0x7f089414a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45096/0 (socket says 172.21.15.187:45096) 2020-11-20T12:09:59.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f0889d9b700 1 -- 172.21.15.187:0/2897196606 learned_addr learned my addr 172.21.15.187:0/2897196606 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:59.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f089259c700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f089409f560 0x7f089413ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f0889d9b700 1 -- 172.21.15.187:0/2897196606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f089414aa20 msgr2=0x7f08940ba8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f0889d9b700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f089414aa20 0x7f08940ba8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:59.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f0889d9b700 1 -- 172.21.15.187:0/2897196606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f089409f560 msgr2=0x7f089413ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f0889d9b700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f089409f560 0x7f089413ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:59.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f0889d9b700 1 -- 172.21.15.187:0/2897196606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894149810 con 0x7f08940bf800 2020-11-20T12:09:59.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.076+0000 7f089259c700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f089409f560 0x7f089413ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:09:59.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.077+0000 7f0889d9b700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bf800 0x7f089414a530 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f089414cfb0 tx=0x7f087c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:59.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.077+0000 7f088affd700 1 -- 172.21.15.187:0/2897196606 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c01d5d0 con 0x7f08940bf800 2020-11-20T12:09:59.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.077+0000 7f088affd700 1 -- 172.21.15.187:0/2897196606 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c01d730 con 0x7f08940bf800 2020-11-20T12:09:59.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.077+0000 7f088affd700 1 -- 172.21.15.187:0/2897196606 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c01dab0 con 0x7f08940bf800 2020-11-20T12:09:59.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.077+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08940badc0 con 0x7f08940bf800 2020-11-20T12:09:59.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.077+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08940b4020 con 0x7f08940bf800 2020-11-20T12:09:59.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.078+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0894080050 con 0x7f08940bf800 2020-11-20T12:09:59.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.079+0000 7f088affd700 1 -- 172.21.15.187:0/2897196606 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f087c025030 con 0x7f08940bf800 2020-11-20T12:09:59.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.080+0000 7f088affd700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0870056d20 0x7f08700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.080+0000 7f088affd700 1 -- 172.21.15.187:0/2897196606 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f087c071840 con 0x7f08940bf800 2020-11-20T12:09:59.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.080+0000 7f089259c700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0870056d20 0x7f08700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.081+0000 7f089259c700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0870056d20 0x7f08700591e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f0884007aa0 tx=0x7f0884008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:59.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.084+0000 7f088affd700 1 -- 172.21.15.187:0/2897196606 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f087c046cf0 con 0x7f08940bf800 2020-11-20T12:09:59.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.277+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0894144740 con 0x7f08940bf800 2020-11-20T12:09:59.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.287+0000 7f088affd700 1 -- 172.21.15.187:0/2897196606 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f087c0230a0 con 0x7f08940bf800 2020-11-20T12:09:59.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.289+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0870056d20 msgr2=0x7f08700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0870056d20 0x7f08700591e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f0884007aa0 tx=0x7f0884008040).stop 2020-11-20T12:09:59.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bf800 msgr2=0x7f089414a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bf800 0x7f089414a530 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f089414cfb0 tx=0x7f087c015000).stop 2020-11-20T12:09:59.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 shutdown_connections 2020-11-20T12:09:59.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0870056d20 0x7f08700591e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:59.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f089409f560 0x7f089413ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:59.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f089414aa20 0x7f08940ba8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:59.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 --2- 172.21.15.187:0/2897196606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08940bf800 0x7f089414a530 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:59.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 >> 172.21.15.187:0/2897196606 conn(0x7f0894068370 msgr2=0x7f08940a3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:59.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 shutdown_connections 2020-11-20T12:09:59.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.290+0000 7f089359e700 1 -- 172.21.15.187:0/2897196606 wait complete. 2020-11-20T12:09:59.317 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2020-11-20T12:09:59.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.560+0000 7f4851305700 1 Processor -- start 2020-11-20T12:09:59.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.561+0000 7f4851305700 1 -- start start 2020-11-20T12:09:59.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.562+0000 7f4851305700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b1760 0x7f484c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.562+0000 7f4851305700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f484c065680 0x7f484c14c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.562+0000 7f4851305700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c150390 con 0x7f484c0b1380 2020-11-20T12:09:59.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.562+0000 7f4851305700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484c1511e0 con 0x7f484c065680 2020-11-20T12:09:59.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.562+0000 7f4851305700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c14d760 con 0x7f484c0b1760 2020-11-20T12:09:59.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.563+0000 7f484bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f484c0b1380 0x7f484c150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41708/0 (socket says 172.21.15.187:41708) 2020-11-20T12:09:59.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.563+0000 7f484bfff700 1 -- 172.21.15.187:0/3546628681 learned_addr learned my addr 172.21.15.187:0/3546628681 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:09:59.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.563+0000 7f484b7fe700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b1760 0x7f484c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.563+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354628566 0 0) 0x7f484c150390 con 0x7f484c0b1380 2020-11-20T12:09:59.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4838003240 con 0x7f484c0b1380 2020-11-20T12:09:59.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.563+0000 7f4850b04700 1 --2- 172.21.15.187:0/3546628681 >> v2:172.21.15.187:3300/0 conn(0x7f484c065680 0x7f484c14c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 28834326 0 0) 0x7f4838003240 con 0x7f484c0b1380 2020-11-20T12:09:59.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b1760 msgr2=0x7f484c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:59.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b1760 0x7f484c065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:59.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 >> v2:172.21.15.187:3300/0 conn(0x7f484c065680 msgr2=0x7f484c14c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:59.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 --2- 172.21.15.187:0/3546628681 >> v2:172.21.15.187:3300/0 conn(0x7f484c065680 0x7f484c14c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:59.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c150630 con 0x7f484c0b1380 2020-11-20T12:09:59.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f483c0014e0 con 0x7f484c0b1380 2020-11-20T12:09:59.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 >> v1:172.21.15.187:6789/0 conn(0x7f484c0b1380 legacy=0x7f484c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b1760 0x7f48380037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003d10 0x7f48380044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48380049d0 0x7f4838004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1511e0 con 0x7f48380049d0 2020-11-20T12:09:59.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c14d760 con 0x7f4838003d10 2020-11-20T12:09:59.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.564+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48380042d0 con 0x7f484c0b1760 2020-11-20T12:09:59.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.565+0000 7f484b7fe700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003d10 0x7f48380044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.565+0000 7f484b7fe700 1 -- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b1760 msgr2=0x7f48380037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:59.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.565+0000 7f484b7fe700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b1760 0x7f48380037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:59.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.565+0000 7f484b7fe700 1 -- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48380049d0 msgr2=0x7f4838004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:59.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.565+0000 7f484b7fe700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48380049d0 0x7f4838004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:59.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.565+0000 7f484b7fe700 1 -- 172.21.15.187:0/3546628681 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4838003240 con 0x7f4838003d10 2020-11-20T12:09:59.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.565+0000 7f484b7fe700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003d10 0x7f48380044e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f484000bc00 tx=0x7f4840006df0).ready entity=mon.1 client_cookie=7f0d1d576fd00c5c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:59.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000b310 con 0x7f4838003d10 2020-11-20T12:09:59.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484000eb40 con 0x7f4838003d10 2020-11-20T12:09:59.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f484affd700 1 -- 172.21.15.187:0/3546628681 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000eec0 con 0x7f4838003d10 2020-11-20T12:09:59.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f4851305700 1 -- 172.21.15.187:0/3546628681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003d10 msgr2=0x7f48380044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f4851305700 1 --2- 172.21.15.187:0/3546628681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003d10 0x7f48380044e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f484000bc00 tx=0x7f4840006df0).stop 2020-11-20T12:09:59.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f4850b04700 1 -- 172.21.15.187:0/3546628681 reap_dead start 2020-11-20T12:09:59.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f4851305700 1 -- 172.21.15.187:0/3546628681 shutdown_connections 2020-11-20T12:09:59.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f4851305700 1 -- 172.21.15.187:0/3546628681 >> 172.21.15.187:0/3546628681 conn(0x7f484c0bab30 msgr2=0x7f484c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:59.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f4851305700 1 -- 172.21.15.187:0/3546628681 shutdown_connections 2020-11-20T12:09:59.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.566+0000 7f4851305700 1 -- 172.21.15.187:0/3546628681 wait complete. 2020-11-20T12:09:59.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.568+0000 7f4851305700 1 Processor -- start 2020-11-20T12:09:59.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.568+0000 7f4851305700 1 -- start start 2020-11-20T12:09:59.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.569+0000 7f4851305700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c143ce0 0x7f484c0a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.569+0000 7f4851305700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c09ff10 0x7f484c0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.569+0000 7f4851305700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b06a0 0x7f484c0b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.569+0000 7f4851305700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1440c0 con 0x7f484c143ce0 2020-11-20T12:09:59.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.569+0000 7f4851305700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0a03b0 con 0x7f484c09ff10 2020-11-20T12:09:59.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.569+0000 7f4851305700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0b3c40 con 0x7f484c0b06a0 2020-11-20T12:09:59.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c09ff10 0x7f484c0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c09ff10 0x7f484c0b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45116/0 (socket says 172.21.15.187:45116) 2020-11-20T12:09:59.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484b7fe700 1 -- 172.21.15.187:0/1984089065 learned_addr learned my addr 172.21.15.187:0/1984089065 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:09:59.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484bfff700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c143ce0 0x7f484c0a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484b7fe700 1 -- 172.21.15.187:0/1984089065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b06a0 msgr2=0x7f484c0b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484b7fe700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b06a0 0x7f484c0b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:09:59.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484b7fe700 1 -- 172.21.15.187:0/1984089065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c143ce0 msgr2=0x7f484c0a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:59.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484b7fe700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c143ce0 0x7f484c0a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:59.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.570+0000 7f484b7fe700 1 -- 172.21.15.187:0/1984089065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0b3ee0 con 0x7f484c09ff10 2020-11-20T12:09:59.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.571+0000 7f484b7fe700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c09ff10 0x7f484c0b4380 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f4840006fd0 tx=0x7f4840019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:59.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.571+0000 7f482bfff700 1 -- 172.21.15.187:0/1984089065 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484001c630 con 0x7f484c09ff10 2020-11-20T12:09:59.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.571+0000 7f482bfff700 1 -- 172.21.15.187:0/1984089065 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484000a070 con 0x7f484c09ff10 2020-11-20T12:09:59.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.571+0000 7f482bfff700 1 -- 172.21.15.187:0/1984089065 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484001c7d0 con 0x7f484c09ff10 2020-11-20T12:09:59.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.571+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c149870 con 0x7f484c09ff10 2020-11-20T12:09:59.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.571+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c09dae0 con 0x7f484c09ff10 2020-11-20T12:09:59.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.572+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c0b4070 con 0x7f484c09ff10 2020-11-20T12:09:59.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.573+0000 7f482bfff700 1 -- 172.21.15.187:0/1984089065 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f484002d070 con 0x7f484c09ff10 2020-11-20T12:09:59.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.573+0000 7f482bfff700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:59.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.573+0000 7f482bfff700 1 -- 172.21.15.187:0/1984089065 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4840071ee0 con 0x7f484c09ff10 2020-11-20T12:09:59.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.573+0000 7f484bfff700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:59.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.574+0000 7f484bfff700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f483c001690 tx=0x7f483c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:59.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.577+0000 7f482bfff700 1 -- 172.21.15.187:0/1984089065 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4840047350 con 0x7f484c09ff10 2020-11-20T12:09:59.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:09:59.832+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f484c061a20 con 0x7f484c09ff10 2020-11-20T12:10:00.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.178+0000 7f482bfff700 1 -- 172.21.15.187:0/1984089065 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v267) v1 ==== 134+0+0 (secure 0 0 0) 0x7f484004a870 con 0x7f484c09ff10 2020-11-20T12:10:00.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 msgr2=0x7f48240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f483c001690 tx=0x7f483c009040).stop 2020-11-20T12:10:00.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c09ff10 msgr2=0x7f484c0b4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c09ff10 0x7f484c0b4380 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f4840006fd0 tx=0x7f4840019000).stop 2020-11-20T12:10:00.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 shutdown_connections 2020-11-20T12:10:00.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c143ce0 0x7f484c0a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c0b06a0 0x7f484c0b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:00.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 --2- 172.21.15.187:0/1984089065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c09ff10 0x7f484c0b4380 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 >> 172.21.15.187:0/1984089065 conn(0x7f484c0bab30 msgr2=0x7f484c0b94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:00.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.181+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 shutdown_connections 2020-11-20T12:10:00.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.182+0000 7f4851305700 1 -- 172.21.15.187:0/1984089065 wait complete. 2020-11-20T12:10:00.288 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:10:00.289 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:10:00.289 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2020-11-20T12:10:00.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.535+0000 7f975ffff700 1 Processor -- start 2020-11-20T12:10:00.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.535+0000 7f975ffff700 1 -- start start 2020-11-20T12:10:00.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.536+0000 7f975ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f97601484d0 0x7f9760061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.536+0000 7f975ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97600622c0 0x7f97600b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.536+0000 7f975ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97600b1300 con 0x7f9760066f70 2020-11-20T12:10:00.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.537+0000 7f975ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9760067730 con 0x7f97601484d0 2020-11-20T12:10:00.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.537+0000 7f975ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97600b15a0 con 0x7f97600622c0 2020-11-20T12:10:00.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.537+0000 7f975f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9760066f70 0x7f97600b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41732/0 (socket says 172.21.15.187:41732) 2020-11-20T12:10:00.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.537+0000 7f975f7fe700 1 -- 172.21.15.187:0/2697913461 learned_addr learned my addr 172.21.15.187:0/2697913461 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:00.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.537+0000 7f975effd700 1 --2- 172.21.15.187:0/2697913461 >> v2:172.21.15.5:3300/0 conn(0x7f97601484d0 0x7f9760061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975e7fc700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97600622c0 0x7f97600b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914653215 0 0) 0x7f97600b1300 con 0x7f9760066f70 2020-11-20T12:10:00.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f974c003240 con 0x7f9760066f70 2020-11-20T12:10:00.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3063745315 0 0) 0x7f974c003240 con 0x7f9760066f70 2020-11-20T12:10:00.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97600622c0 msgr2=0x7f97600b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:00.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97600622c0 0x7f97600b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 >> v2:172.21.15.5:3300/0 conn(0x7f97601484d0 msgr2=0x7f9760061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:00.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 --2- 172.21.15.187:0/2697913461 >> v2:172.21.15.5:3300/0 conn(0x7f97601484d0 0x7f9760061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97600b1840 con 0x7f9760066f70 2020-11-20T12:10:00.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f9758001860 con 0x7f9760066f70 2020-11-20T12:10:00.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 >> v1:172.21.15.187:6789/0 conn(0x7f9760066f70 legacy=0x7f97600b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974c003730 0x7f974c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.538+0000 7f975dffb700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f974c0044e0 0x7f974c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.539+0000 7f975dffb700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97600622c0 0x7f974c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.539+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760067730 con 0x7f974c0044e0 2020-11-20T12:10:00.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.539+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97600b15a0 con 0x7f974c003730 2020-11-20T12:10:00.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.539+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c004270 con 0x7f97600622c0 2020-11-20T12:10:00.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.539+0000 7f975effd700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974c003730 0x7f974c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.539+0000 7f975f7fe700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97600622c0 0x7f974c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.539+0000 7f975e7fc700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f974c0044e0 0x7f974c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975effd700 1 -- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97600622c0 msgr2=0x7f974c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:00.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975effd700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97600622c0 0x7f974c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975effd700 1 -- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f974c0044e0 msgr2=0x7f974c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:00.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975effd700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f974c0044e0 0x7f974c004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975effd700 1 -- 172.21.15.187:0/2697913461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c003240 con 0x7f974c003730 2020-11-20T12:10:00.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975effd700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974c003730 0x7f974c003b50 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f975000bc00 tx=0x7f9750006df0).ready entity=mon.1 client_cookie=135303ab90e0278c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:00.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975000b310 con 0x7f974c003730 2020-11-20T12:10:00.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975000eb40 con 0x7f974c003730 2020-11-20T12:10:00.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.540+0000 7f975dffb700 1 -- 172.21.15.187:0/2697913461 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975000eec0 con 0x7f974c003730 2020-11-20T12:10:00.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.541+0000 7f975ffff700 1 -- 172.21.15.187:0/2697913461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974c003730 msgr2=0x7f974c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.541+0000 7f975ffff700 1 --2- 172.21.15.187:0/2697913461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974c003730 0x7f974c003b50 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f975000bc00 tx=0x7f9750006df0).stop 2020-11-20T12:10:00.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.541+0000 7f975f7fe700 1 -- 172.21.15.187:0/2697913461 reap_dead start 2020-11-20T12:10:00.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.541+0000 7f975ffff700 1 -- 172.21.15.187:0/2697913461 shutdown_connections 2020-11-20T12:10:00.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.541+0000 7f975ffff700 1 -- 172.21.15.187:0/2697913461 >> 172.21.15.187:0/2697913461 conn(0x7f97600bab30 msgr2=0x7f97600baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:00.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.541+0000 7f975ffff700 1 -- 172.21.15.187:0/2697913461 shutdown_connections 2020-11-20T12:10:00.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.541+0000 7f975ffff700 1 -- 172.21.15.187:0/2697913461 wait complete. 2020-11-20T12:10:00.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.543+0000 7f975ffff700 1 Processor -- start 2020-11-20T12:10:00.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.543+0000 7f975ffff700 1 -- start start 2020-11-20T12:10:00.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.544+0000 7f975ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f976009e140 0x7f976009cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.544+0000 7f975ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976009e520 0x7f97600b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.544+0000 7f975ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976009f2d0 0x7f976009b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.544+0000 7f975ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976009d1f0 con 0x7f976009e140 2020-11-20T12:10:00.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.544+0000 7f975ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976009b000 con 0x7f976009e520 2020-11-20T12:10:00.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.544+0000 7f975ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976009b2a0 con 0x7f976009f2d0 2020-11-20T12:10:00.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975e7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976009e520 0x7f97600b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975e7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976009e520 0x7f97600b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45136/0 (socket says 172.21.15.187:45136) 2020-11-20T12:10:00.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975e7fc700 1 -- 172.21.15.187:0/1474046199 learned_addr learned my addr 172.21.15.187:0/1474046199 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:00.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975effd700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f976009e140 0x7f976009cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975e7fc700 1 -- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976009f2d0 msgr2=0x7f976009b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975f7fe700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976009f2d0 0x7f976009b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975e7fc700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976009f2d0 0x7f976009b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975e7fc700 1 -- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f976009e140 msgr2=0x7f976009cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.545+0000 7f975e7fc700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f976009e140 0x7f976009cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.546+0000 7f975e7fc700 1 -- 172.21.15.187:0/1474046199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97600b3140 con 0x7f976009e520 2020-11-20T12:10:00.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.546+0000 7f975effd700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f976009e140 0x7f976009cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:00.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.546+0000 7f975e7fc700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976009e520 0x7f97600b2c50 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f9754010c50 tx=0x7f9754008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:00.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.546+0000 7f9746ffd700 1 -- 172.21.15.187:0/1474046199 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975400eab0 con 0x7f976009e520 2020-11-20T12:10:00.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.546+0000 7f9746ffd700 1 -- 172.21.15.187:0/1474046199 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975400ec10 con 0x7f976009e520 2020-11-20T12:10:00.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.546+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976009bc30 con 0x7f976009e520 2020-11-20T12:10:00.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.546+0000 7f9746ffd700 1 -- 172.21.15.187:0/1474046199 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754014470 con 0x7f976009e520 2020-11-20T12:10:00.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.546+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97600b1a40 con 0x7f976009e520 2020-11-20T12:10:00.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.548+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976009b460 con 0x7f976009e520 2020-11-20T12:10:00.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.548+0000 7f9746ffd700 1 -- 172.21.15.187:0/1474046199 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9754024070 con 0x7f976009e520 2020-11-20T12:10:00.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.549+0000 7f9746ffd700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f973c056d20 0x7f973c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.549+0000 7f9746ffd700 1 -- 172.21.15.187:0/1474046199 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f97540709f0 con 0x7f976009e520 2020-11-20T12:10:00.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.549+0000 7f975effd700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f973c056d20 0x7f973c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.550+0000 7f975effd700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f973c056d20 0x7f973c0591e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f976005ea30 tx=0x7f9750019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:00.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.552+0000 7f9746ffd700 1 -- 172.21.15.187:0/1474046199 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9754045d20 con 0x7f976009e520 2020-11-20T12:10:00.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:00.802+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f9760061a20 con 0x7f976009e520 2020-11-20T12:10:01.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.184+0000 7f9746ffd700 1 -- 172.21.15.187:0/1474046199 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v268) v1 ==== 145+0+0 (secure 0 0 0) 0x7f975401b310 con 0x7f976009e520 2020-11-20T12:10:01.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f973c056d20 msgr2=0x7f973c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:01.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f973c056d20 0x7f973c0591e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f976005ea30 tx=0x7f9750019000).stop 2020-11-20T12:10:01.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976009e520 msgr2=0x7f97600b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:01.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976009e520 0x7f97600b2c50 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f9754010c50 tx=0x7f9754008040).stop 2020-11-20T12:10:01.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 shutdown_connections 2020-11-20T12:10:01.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f973c056d20 0x7f973c0591e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f976009e140 0x7f976009cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976009f2d0 0x7f976009b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 --2- 172.21.15.187:0/1474046199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976009e520 0x7f97600b2c50 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.187+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 >> 172.21.15.187:0/1474046199 conn(0x7f97600bab30 msgr2=0x7f9760142ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:01.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.188+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 shutdown_connections 2020-11-20T12:10:01.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.188+0000 7f975ffff700 1 -- 172.21.15.187:0/1474046199 wait complete. 2020-11-20T12:10:01.215 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:10:01.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.447+0000 7f1d39469700 1 Processor -- start 2020-11-20T12:10:01.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.447+0000 7f1d39469700 1 -- start start 2020-11-20T12:10:01.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d39469700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1d340b57b0 0x7f1d34068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:01.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d39469700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1d341482d0 0x7f1d34062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:01.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d39469700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d34062b50 0x7f1d34065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:01.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d39469700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d34144b80 con 0x7f1d341482d0 2020-11-20T12:10:01.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d39469700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d341489e0 con 0x7f1d340b57b0 2020-11-20T12:10:01.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d39469700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d34148c80 con 0x7f1d34062b50 2020-11-20T12:10:01.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d33fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1d340b57b0 0x7f1d34068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:01.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d33fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1d340b57b0 0x7f1d34068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45142/0 (socket says 172.21.15.187:45142) 2020-11-20T12:10:01.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.449+0000 7f1d33fff700 1 -- 172.21.15.187:0/2225110504 learned_addr learned my addr 172.21.15.187:0/2225110504 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:01.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d38c68700 1 --2- 172.21.15.187:0/2225110504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d34062b50 0x7f1d34065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:01.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d33fff700 1 -- 172.21.15.187:0/2225110504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d34062b50 msgr2=0x7f1d34065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:01.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d337fe700 1 --2- 172.21.15.187:0/2225110504 >> v2:172.21.15.5:3300/0 conn(0x7f1d341482d0 0x7f1d34062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:01.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d33fff700 1 --2- 172.21.15.187:0/2225110504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d34062b50 0x7f1d34065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d33fff700 1 -- 172.21.15.187:0/2225110504 >> v2:172.21.15.5:3300/0 conn(0x7f1d341482d0 msgr2=0x7f1d34062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:01.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d33fff700 1 --2- 172.21.15.187:0/2225110504 >> v2:172.21.15.5:3300/0 conn(0x7f1d341482d0 0x7f1d34062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d33fff700 1 -- 172.21.15.187:0/2225110504 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d34148f20 con 0x7f1d340b57b0 2020-11-20T12:10:01.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d38c68700 1 --2- 172.21.15.187:0/2225110504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d34062b50 0x7f1d34065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:01.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d337fe700 1 --2- 172.21.15.187:0/2225110504 >> v2:172.21.15.5:3300/0 conn(0x7f1d341482d0 0x7f1d34062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:01.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d33fff700 1 --2- 172.21.15.187:0/2225110504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d340b57b0 0x7f1d34068230 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f1d340b85e0 tx=0x7f1d24009290).ready entity=mon.1 client_cookie=9594c8c35fc077fd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:01.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d32ffd700 1 -- 172.21.15.187:0/2225110504 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2400bbc0 con 0x7f1d340b57b0 2020-11-20T12:10:01.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d32ffd700 1 -- 172.21.15.187:0/2225110504 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2400bd20 con 0x7f1d340b57b0 2020-11-20T12:10:01.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.450+0000 7f1d32ffd700 1 -- 172.21.15.187:0/2225110504 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d24011610 con 0x7f1d340b57b0 2020-11-20T12:10:01.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 -- 172.21.15.187:0/2225110504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d340b57b0 msgr2=0x7f1d34068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:01.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 --2- 172.21.15.187:0/2225110504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d340b57b0 0x7f1d34068230 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f1d340b85e0 tx=0x7f1d24009290).stop 2020-11-20T12:10:01.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 -- 172.21.15.187:0/2225110504 shutdown_connections 2020-11-20T12:10:01.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 --2- 172.21.15.187:0/2225110504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d34062b50 0x7f1d34065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 --2- 172.21.15.187:0/2225110504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d340b57b0 0x7f1d34068230 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 --2- 172.21.15.187:0/2225110504 >> v2:172.21.15.5:3300/0 conn(0x7f1d341482d0 0x7f1d34062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 -- 172.21.15.187:0/2225110504 >> 172.21.15.187:0/2225110504 conn(0x7f1d340bab30 msgr2=0x7f1d340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:01.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 -- 172.21.15.187:0/2225110504 shutdown_connections 2020-11-20T12:10:01.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.451+0000 7f1d39469700 1 -- 172.21.15.187:0/2225110504 wait complete. 2020-11-20T12:10:01.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.453+0000 7f1d39469700 1 Processor -- start 2020-11-20T12:10:01.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.453+0000 7f1d39469700 1 -- start start 2020-11-20T12:10:01.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d39469700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d34062b50 0x7f1d3409ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:01.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d39469700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d340b57b0 0x7f1d3409f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:01.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d39469700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d341482d0 0x7f1d34140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:01.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d39469700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3409fea0 con 0x7f1d34062b50 2020-11-20T12:10:01.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d39469700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340a0140 con 0x7f1d341482d0 2020-11-20T12:10:01.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d39469700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340a03e0 con 0x7f1d340b57b0 2020-11-20T12:10:01.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d38c68700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d341482d0 0x7f1d34140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:01.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d38c68700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d341482d0 0x7f1d34140bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45152/0 (socket says 172.21.15.187:45152) 2020-11-20T12:10:01.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d38c68700 1 -- 172.21.15.187:0/3685078216 learned_addr learned my addr 172.21.15.187:0/3685078216 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:01.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.454+0000 7f1d337fe700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d340b57b0 0x7f1d3409f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:01.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d38c68700 1 -- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d340b57b0 msgr2=0x7f1d3409f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:01.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d38c68700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d340b57b0 0x7f1d3409f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d38c68700 1 -- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d34062b50 msgr2=0x7f1d3409ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:01.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d33fff700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d34062b50 0x7f1d3409ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:01.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d38c68700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d34062b50 0x7f1d3409ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d38c68700 1 -- 172.21.15.187:0/3685078216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3409d3c0 con 0x7f1d341482d0 2020-11-20T12:10:01.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d33fff700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d34062b50 0x7f1d3409ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:01.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d38c68700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d341482d0 0x7f1d34140bb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f1d2c00c870 tx=0x7f1d2c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:01.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d13fff700 1 -- 172.21.15.187:0/3685078216 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c007620 con 0x7f1d341482d0 2020-11-20T12:10:01.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d13fff700 1 -- 172.21.15.187:0/3685078216 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c007780 con 0x7f1d341482d0 2020-11-20T12:10:01.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d13fff700 1 -- 172.21.15.187:0/3685078216 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c007b20 con 0x7f1d341482d0 2020-11-20T12:10:01.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d340a0880 con 0x7f1d341482d0 2020-11-20T12:10:01.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.455+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3409fa10 con 0x7f1d341482d0 2020-11-20T12:10:01.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.457+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d340a05a0 con 0x7f1d341482d0 2020-11-20T12:10:01.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.457+0000 7f1d13fff700 1 -- 172.21.15.187:0/3685078216 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1d2c01b460 con 0x7f1d341482d0 2020-11-20T12:10:01.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.458+0000 7f1d13fff700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d0c056d50 0x7f1d0c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:01.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.458+0000 7f1d13fff700 1 -- 172.21.15.187:0/3685078216 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1d2c06ff30 con 0x7f1d341482d0 2020-11-20T12:10:01.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.458+0000 7f1d33fff700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d0c056d50 0x7f1d0c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:01.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.459+0000 7f1d33fff700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d0c056d50 0x7f1d0c059210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f1d2400c010 tx=0x7f1d24016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:01.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.462+0000 7f1d13fff700 1 -- 172.21.15.187:0/3685078216 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1d2c0452d0 con 0x7f1d341482d0 2020-11-20T12:10:01.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.645+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f1d34061a20 con 0x7f1d341482d0 2020-11-20T12:10:01.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.655+0000 7f1d13fff700 1 -- 172.21.15.187:0/3685078216 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1d2c0488f0 con 0x7f1d341482d0 2020-11-20T12:10:01.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.657+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d0c056d50 msgr2=0x7f1d0c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:01.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.657+0000 7f1d39469700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d0c056d50 0x7f1d0c059210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f1d2400c010 tx=0x7f1d24016000).stop 2020-11-20T12:10:01.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.657+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d341482d0 msgr2=0x7f1d34140bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:01.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.657+0000 7f1d39469700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d341482d0 0x7f1d34140bb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f1d2c00c870 tx=0x7f1d2c008040).stop 2020-11-20T12:10:01.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.658+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 shutdown_connections 2020-11-20T12:10:01.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.658+0000 7f1d39469700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d0c056d50 0x7f1d0c059210 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.658+0000 7f1d39469700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d34062b50 0x7f1d3409ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.658+0000 7f1d39469700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d341482d0 0x7f1d34140bb0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.658+0000 7f1d39469700 1 --2- 172.21.15.187:0/3685078216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d340b57b0 0x7f1d3409f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:01.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.658+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 >> 172.21.15.187:0/3685078216 conn(0x7f1d340bab30 msgr2=0x7f1d34061ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:01.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.658+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 shutdown_connections 2020-11-20T12:10:01.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:01.658+0000 7f1d39469700 1 -- 172.21.15.187:0/3685078216 wait complete. 2020-11-20T12:10:01.685 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:10:01.686 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:10:01.803 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:10:01.820 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:10:01.944 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:10:01.961 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:10:01.962 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:10:01.962 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:02.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.211+0000 7fb90bfff700 1 Processor -- start 2020-11-20T12:10:02.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.211+0000 7fb90bfff700 1 -- start start 2020-11-20T12:10:02.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.212+0000 7fb90bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb90c065190 0x7fb90c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.212+0000 7fb90bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb90c09ca90 0x7fb90c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.212+0000 7fb90bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c0b13e0 con 0x7fb90c065f90 2020-11-20T12:10:02.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.212+0000 7fb90bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c14ab30 con 0x7fb90c09ca90 2020-11-20T12:10:02.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.212+0000 7fb90bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c062c10 con 0x7fb90c065190 2020-11-20T12:10:02.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.213+0000 7fb90affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb90c065190 0x7fb90c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.213+0000 7fb90affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb90c065190 0x7fb90c065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45158/0 (socket says 172.21.15.187:45158) 2020-11-20T12:10:02.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.213+0000 7fb90affd700 1 -- 172.21.15.187:0/3891742918 learned_addr learned my addr 172.21.15.187:0/3891742918 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:02.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90b7fe700 1 --2- 172.21.15.187:0/3891742918 >> v2:172.21.15.5:3300/0 conn(0x7fb90c09ca90 0x7fb90c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90affd700 1 -- 172.21.15.187:0/3891742918 >> v2:172.21.15.5:3300/0 conn(0x7fb90c09ca90 msgr2=0x7fb90c09cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:02.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90affd700 1 --2- 172.21.15.187:0/3891742918 >> v2:172.21.15.5:3300/0 conn(0x7fb90c09ca90 0x7fb90c09cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90affd700 1 -- 172.21.15.187:0/3891742918 >> v1:172.21.15.5:6789/0 conn(0x7fb90c065f90 legacy=0x7fb90c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:02.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90affd700 1 -- 172.21.15.187:0/3891742918 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c0b1640 con 0x7fb90c065190 2020-11-20T12:10:02.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90b7fe700 1 --2- 172.21.15.187:0/3891742918 >> v2:172.21.15.5:3300/0 conn(0x7fb90c09ca90 0x7fb90c09cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:02.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90affd700 1 --2- 172.21.15.187:0/3891742918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c065190 0x7fb90c065570 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7fb90c05c8b0 tx=0x7fb8fc006eb0).ready entity=mon.1 client_cookie=5b7509b6d3ced84f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:02.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90a7fc700 1 -- 172.21.15.187:0/3891742918 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc009920 con 0x7fb90c065190 2020-11-20T12:10:02.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90a7fc700 1 -- 172.21.15.187:0/3891742918 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc009a80 con 0x7fb90c065190 2020-11-20T12:10:02.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.214+0000 7fb90a7fc700 1 -- 172.21.15.187:0/3891742918 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc009e00 con 0x7fb90c065190 2020-11-20T12:10:02.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.215+0000 7fb90bfff700 1 -- 172.21.15.187:0/3891742918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c065190 msgr2=0x7fb90c065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.215+0000 7fb90bfff700 1 --2- 172.21.15.187:0/3891742918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c065190 0x7fb90c065570 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7fb90c05c8b0 tx=0x7fb8fc006eb0).stop 2020-11-20T12:10:02.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.215+0000 7fb90bfff700 1 -- 172.21.15.187:0/3891742918 shutdown_connections 2020-11-20T12:10:02.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.215+0000 7fb90bfff700 1 --2- 172.21.15.187:0/3891742918 >> v2:172.21.15.5:3300/0 conn(0x7fb90c09ca90 0x7fb90c09cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.215+0000 7fb90bfff700 1 --2- 172.21.15.187:0/3891742918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c065190 0x7fb90c065570 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.215+0000 7fb90bfff700 1 -- 172.21.15.187:0/3891742918 >> 172.21.15.187:0/3891742918 conn(0x7fb90c0bab30 msgr2=0x7fb90c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:02.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.215+0000 7fb90bfff700 1 -- 172.21.15.187:0/3891742918 shutdown_connections 2020-11-20T12:10:02.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.215+0000 7fb90bfff700 1 -- 172.21.15.187:0/3891742918 wait complete. 2020-11-20T12:10:02.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.217+0000 7fb90bfff700 1 Processor -- start 2020-11-20T12:10:02.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.217+0000 7fb90bfff700 1 -- start start 2020-11-20T12:10:02.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.218+0000 7fb90bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb90c065f90 0x7fb90c0b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.218+0000 7fb90bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c09ca90 0x7fb90c0b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.218+0000 7fb90bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb90c148e50 0x7fb90c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.218+0000 7fb90bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c1438b0 con 0x7fb90c148e50 2020-11-20T12:10:02.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.218+0000 7fb90bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c149b10 con 0x7fb90c09ca90 2020-11-20T12:10:02.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.218+0000 7fb90bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c149db0 con 0x7fb90c065f90 2020-11-20T12:10:02.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.219+0000 7fb9027fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c09ca90 0x7fb90c0b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.219+0000 7fb9027fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c09ca90 0x7fb90c0b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45164/0 (socket says 172.21.15.187:45164) 2020-11-20T12:10:02.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.219+0000 7fb9027fc700 1 -- 172.21.15.187:0/834007304 learned_addr learned my addr 172.21.15.187:0/834007304 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:02.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.219+0000 7fb90affd700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb90c065f90 0x7fb90c0b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.219+0000 7fb90b7fe700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb90c148e50 0x7fb90c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9027fc700 1 -- 172.21.15.187:0/834007304 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb90c065f90 msgr2=0x7fb90c0b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9027fc700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb90c065f90 0x7fb90c0b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9027fc700 1 -- 172.21.15.187:0/834007304 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb90c148e50 msgr2=0x7fb90c1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9027fc700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb90c148e50 0x7fb90c1565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9027fc700 1 -- 172.21.15.187:0/834007304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c148980 con 0x7fb90c09ca90 2020-11-20T12:10:02.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9027fc700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c09ca90 0x7fb90c0b3ed0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7fb8f400c760 tx=0x7fb8f4007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:02.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9037fe700 1 -- 172.21.15.187:0/834007304 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f400e560 con 0x7fb90c09ca90 2020-11-20T12:10:02.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9037fe700 1 -- 172.21.15.187:0/834007304 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f400e6c0 con 0x7fb90c09ca90 2020-11-20T12:10:02.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c149790 con 0x7fb90c09ca90 2020-11-20T12:10:02.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb9037fe700 1 -- 172.21.15.187:0/834007304 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f400ea80 con 0x7fb90c09ca90 2020-11-20T12:10:02.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.220+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c156cd0 con 0x7fb90c09ca90 2020-11-20T12:10:02.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.221+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c099d50 con 0x7fb90c09ca90 2020-11-20T12:10:02.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.222+0000 7fb9037fe700 1 -- 172.21.15.187:0/834007304 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb8f4019030 con 0x7fb90c09ca90 2020-11-20T12:10:02.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.222+0000 7fb9037fe700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8e8056d20 0x7fb8e80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.222+0000 7fb9037fe700 1 -- 172.21.15.187:0/834007304 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb8f406f440 con 0x7fb90c09ca90 2020-11-20T12:10:02.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.223+0000 7fb90affd700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8e8056d20 0x7fb8e80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.223+0000 7fb90affd700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8e8056d20 0x7fb8e80591e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fb90c063480 tx=0x7fb8fc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:02.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.227+0000 7fb9037fe700 1 -- 172.21.15.187:0/834007304 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb8f4044890 con 0x7fb90c09ca90 2020-11-20T12:10:02.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.453+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb90c149980 con 0x7fb90c09ca90 2020-11-20T12:10:02.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.454+0000 7fb9037fe700 1 -- 172.21.15.187:0/834007304 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fb8f4047eb0 con 0x7fb90c09ca90 2020-11-20T12:10:02.458 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:02.458 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:10:02.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8e8056d20 msgr2=0x7fb8e80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8e8056d20 0x7fb8e80591e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fb90c063480 tx=0x7fb8fc015000).stop 2020-11-20T12:10:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c09ca90 msgr2=0x7fb90c0b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c09ca90 0x7fb90c0b3ed0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7fb8f400c760 tx=0x7fb8f4007c00).stop 2020-11-20T12:10:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 shutdown_connections 2020-11-20T12:10:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8e8056d20 0x7fb8e80591e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb90c148e50 0x7fb90c1565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb90c065f90 0x7fb90c0b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 --2- 172.21.15.187:0/834007304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb90c09ca90 0x7fb90c0b3ed0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.456+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 >> 172.21.15.187:0/834007304 conn(0x7fb90c0bab30 msgr2=0x7fb90c0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:02.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.457+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 shutdown_connections 2020-11-20T12:10:02.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.457+0000 7fb90bfff700 1 -- 172.21.15.187:0/834007304 wait complete. 2020-11-20T12:10:02.462 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:10:02.487 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:10:02.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.733+0000 7f4f4f59e700 1 Processor -- start 2020-11-20T12:10:02.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.734+0000 7f4f4f59e700 1 -- start start 2020-11-20T12:10:02.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.735+0000 7f4f4f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4f501484d0 0x7f4f50061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.735+0000 7f4f4f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f500b1300 con 0x7f4f50066f70 2020-11-20T12:10:02.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.735+0000 7f4f4f59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f500b15a0 con 0x7f4f500622c0 2020-11-20T12:10:02.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.735+0000 7f4f4f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f50067730 con 0x7f4f501484d0 2020-11-20T12:10:02.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.736+0000 7f4f3ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4f500622c0 0x7f4f500b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41776/0 (socket says 172.21.15.187:41776) 2020-11-20T12:10:02.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.736+0000 7f4f3ffff700 1 -- 172.21.15.187:0/2238715384 learned_addr learned my addr 172.21.15.187:0/2238715384 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:02.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.736+0000 7f4f4e59c700 1 --2- 172.21.15.187:0/2238715384 >> v2:172.21.15.5:3300/0 conn(0x7f4f501484d0 0x7f4f50061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.736+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104288528 0 0) 0x7f4f500b15a0 con 0x7f4f500622c0 2020-11-20T12:10:02.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f30003240 con 0x7f4f500622c0 2020-11-20T12:10:02.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2437927743 0 0) 0x7f4f30003240 con 0x7f4f500622c0 2020-11-20T12:10:02.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 >> v2:172.21.15.5:3300/0 conn(0x7f4f501484d0 msgr2=0x7f4f50061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:02.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 --2- 172.21.15.187:0/2238715384 >> v2:172.21.15.5:3300/0 conn(0x7f4f501484d0 0x7f4f50061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 >> v1:172.21.15.5:6789/0 conn(0x7f4f50066f70 legacy=0x7f4f500b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f500b1840 con 0x7f4f500622c0 2020-11-20T12:10:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4f440014e0 con 0x7f4f500622c0 2020-11-20T12:10:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 >> v1:172.21.15.187:6789/0 conn(0x7f4f500622c0 legacy=0x7f4f500b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f30003780 0x7f4f30003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f300044e0 0x7f4f30004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f3000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50067730 con 0x7f4f30003780 2020-11-20T12:10:02.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f30005300 con 0x7f4f30004e50 2020-11-20T12:10:02.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.737+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3000bc20 con 0x7f4f300044e0 2020-11-20T12:10:02.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4ed9d700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f3000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f3ffff700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f300044e0 0x7f4f30004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4e59c700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f30003780 0x7f4f30003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4ed9d700 1 -- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f300044e0 msgr2=0x7f4f30004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:02.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4ed9d700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f300044e0 0x7f4f30004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4ed9d700 1 -- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f30003780 msgr2=0x7f4f30003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4ed9d700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f30003780 0x7f4f30003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4ed9d700 1 -- 172.21.15.187:0/2238715384 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30003240 con 0x7f4f30004e50 2020-11-20T12:10:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4e59c700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f30003780 0x7f4f30003ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:02.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f3ffff700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f300044e0 0x7f4f30004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4ed9d700 1 -- 172.21.15.187:0/2238715384 reap_dead start 2020-11-20T12:10:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.738+0000 7f4f4ed9d700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f3000b730 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f4f480018f0 tx=0x7f4f48007b90).ready entity=mon.1 client_cookie=8f41291140dbf066 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48007020 con 0x7f4f30004e50 2020-11-20T12:10:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f48007180 con 0x7f4f30004e50 2020-11-20T12:10:02.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4dd9b700 1 -- 172.21.15.187:0/2238715384 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4800c830 con 0x7f4f30004e50 2020-11-20T12:10:02.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4f59e700 1 -- 172.21.15.187:0/2238715384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 msgr2=0x7f4f3000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4f59e700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f3000b730 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f4f480018f0 tx=0x7f4f48007b90).stop 2020-11-20T12:10:02.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4f59e700 1 -- 172.21.15.187:0/2238715384 shutdown_connections 2020-11-20T12:10:02.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4f59e700 1 --2- 172.21.15.187:0/2238715384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f3000b730 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4f59e700 1 -- 172.21.15.187:0/2238715384 >> 172.21.15.187:0/2238715384 conn(0x7f4f500bab30 msgr2=0x7f4f500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:02.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4f59e700 1 -- 172.21.15.187:0/2238715384 shutdown_connections 2020-11-20T12:10:02.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.739+0000 7f4f4f59e700 1 -- 172.21.15.187:0/2238715384 wait complete. 2020-11-20T12:10:02.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.741+0000 7f4f4f59e700 1 Processor -- start 2020-11-20T12:10:02.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.741+0000 7f4f4f59e700 1 -- start start 2020-11-20T12:10:02.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.742+0000 7f4f4f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f5013b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.742+0000 7f4f4f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f500b2120 0x7f4f50138810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.742+0000 7f4f4f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f500b3300 0x7f4f500b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.742+0000 7f4f4f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5013bb10 con 0x7f4f500b3300 2020-11-20T12:10:02.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.742+0000 7f4f4f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50154e90 con 0x7f4f30004e50 2020-11-20T12:10:02.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.742+0000 7f4f4f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5009ef60 con 0x7f4f500b2120 2020-11-20T12:10:02.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f5013b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f5013b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45182/0 (socket says 172.21.15.187:45182) 2020-11-20T12:10:02.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 -- 172.21.15.187:0/1495116139 learned_addr learned my addr 172.21.15.187:0/1495116139 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 -- 172.21.15.187:0/1495116139 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f500b2120 msgr2=0x7f4f50138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f500b2120 0x7f4f50138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 -- 172.21.15.187:0/1495116139 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f500b3300 msgr2=0x7f4f500b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f500b3300 0x7f4f500b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 -- 172.21.15.187:0/1495116139 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5009f1c0 con 0x7f4f30004e50 2020-11-20T12:10:02.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f4e59c700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f5013b620 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f4f4000fbe0 tx=0x7f4f40008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:02.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.743+0000 7f4f3e7fc700 1 -- 172.21.15.187:0/1495116139 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4000d960 con 0x7f4f30004e50 2020-11-20T12:10:02.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.744+0000 7f4f3e7fc700 1 -- 172.21.15.187:0/1495116139 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4000dac0 con 0x7f4f30004e50 2020-11-20T12:10:02.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.744+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5013bd30 con 0x7f4f30004e50 2020-11-20T12:10:02.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.744+0000 7f4f3e7fc700 1 -- 172.21.15.187:0/1495116139 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4000de40 con 0x7f4f30004e50 2020-11-20T12:10:02.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.744+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5013a4f0 con 0x7f4f30004e50 2020-11-20T12:10:02.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.745+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f50061a20 con 0x7f4f30004e50 2020-11-20T12:10:02.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.745+0000 7f4f3e7fc700 1 -- 172.21.15.187:0/1495116139 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4f40014670 con 0x7f4f30004e50 2020-11-20T12:10:02.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.746+0000 7f4f3e7fc700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f2c056d20 0x7f4f2c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:02.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.746+0000 7f4f3e7fc700 1 -- 172.21.15.187:0/1495116139 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4f40070820 con 0x7f4f30004e50 2020-11-20T12:10:02.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.746+0000 7f4f3ffff700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f2c056d20 0x7f4f2c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:02.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.747+0000 7f4f3ffff700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f2c056d20 0x7f4f2c0591e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f4f500b4c70 tx=0x7f4f44009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:02.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.750+0000 7f4f3e7fc700 1 -- 172.21.15.187:0/1495116139 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4f40045c20 con 0x7f4f30004e50 2020-11-20T12:10:02.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.937+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f501488b0 con 0x7f4f30004e50 2020-11-20T12:10:02.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.939+0000 7f4f3e7fc700 1 -- 172.21.15.187:0/1495116139 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16912 (secure 0 0 0) 0x7f4f40018090 con 0x7f4f30004e50 2020-11-20T12:10:02.944 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:02.944 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":61,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:09:18.096938+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:09:05.207488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2020-11-20T12:09:06.244532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/3570428099":"2020-11-21T12:09:18.096834+0000","172.21.15.187:6834/3570428099":"2020-11-21T12:09:18.096834+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:10:02.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f2c056d20 msgr2=0x7f4f2c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f2c056d20 0x7f4f2c0591e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f4f500b4c70 tx=0x7f4f44009040).stop 2020-11-20T12:10:02.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 msgr2=0x7f4f5013b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:02.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f5013b620 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f4f4000fbe0 tx=0x7f4f40008040).stop 2020-11-20T12:10:02.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 shutdown_connections 2020-11-20T12:10:02.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f2c056d20 0x7f4f2c0591e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f500b3300 0x7f4f500b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:02.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f30004e50 0x7f4f5013b620 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:02.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 --2- 172.21.15.187:0/1495116139 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f500b2120 0x7f4f50138810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:02.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.941+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 >> 172.21.15.187:0/1495116139 conn(0x7f4f500bab30 msgr2=0x7f4f500b6140 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:02.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.942+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 shutdown_connections 2020-11-20T12:10:02.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:02.942+0000 7f4f4f59e700 1 -- 172.21.15.187:0/1495116139 wait complete. 2020-11-20T12:10:02.970 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:09:05.207488+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:09:06.244532+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:10:02.972 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:10:03.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.211+0000 7f86aa0d6700 1 Processor -- start 2020-11-20T12:10:03.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.211+0000 7f86aa0d6700 1 -- start start 2020-11-20T12:10:03.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.212+0000 7f86aa0d6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f86a41484d0 0x7f86a4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.212+0000 7f86aa0d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a4066f70 0x7f86a40b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.212+0000 7f86aa0d6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40b1300 con 0x7f86a40622c0 2020-11-20T12:10:03.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.212+0000 7f86aa0d6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a4067730 con 0x7f86a41484d0 2020-11-20T12:10:03.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.212+0000 7f86aa0d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40b15a0 con 0x7f86a4066f70 2020-11-20T12:10:03.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.213+0000 7f86a98d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a4066f70 0x7f86a40b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.214+0000 7f86a98d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a4066f70 0x7f86a40b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:46082/0 (socket says 172.21.15.187:46082) 2020-11-20T12:10:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.214+0000 7f86a88d3700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f86a40622c0 0x7f86a40b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:35306/0 (socket says 172.21.15.187:35306) 2020-11-20T12:10:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.214+0000 7f86a98d5700 1 -- 172.21.15.187:0/2394273747 learned_addr learned my addr 172.21.15.187:0/2394273747 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.214+0000 7f86a90d4700 1 --2- 172.21.15.187:0/2394273747 >> v2:172.21.15.5:3300/0 conn(0x7f86a41484d0 0x7f86a4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.215+0000 7f86a3fff700 1 -- 172.21.15.187:0/2394273747 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986676445 0 0) 0x7f86a40b1300 con 0x7f86a40622c0 2020-11-20T12:10:03.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.215+0000 7f86a3fff700 1 -- 172.21.15.187:0/2394273747 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8690003240 con 0x7f86a40622c0 2020-11-20T12:10:03.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.215+0000 7f86a98d5700 1 -- 172.21.15.187:0/2394273747 >> v2:172.21.15.5:3300/0 conn(0x7f86a41484d0 msgr2=0x7f86a4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:03.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.215+0000 7f86a98d5700 1 --2- 172.21.15.187:0/2394273747 >> v2:172.21.15.5:3300/0 conn(0x7f86a41484d0 0x7f86a4061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.215+0000 7f86a98d5700 1 -- 172.21.15.187:0/2394273747 >> v1:172.21.15.5:6789/0 conn(0x7f86a40622c0 legacy=0x7f86a40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.215+0000 7f86a98d5700 1 -- 172.21.15.187:0/2394273747 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a40b1840 con 0x7f86a4066f70 2020-11-20T12:10:03.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.215+0000 7f86a90d4700 1 --2- 172.21.15.187:0/2394273747 >> v2:172.21.15.5:3300/0 conn(0x7f86a41484d0 0x7f86a4061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:10:03.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.216+0000 7f86a98d5700 1 --2- 172.21.15.187:0/2394273747 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a4066f70 0x7f86a40b5d50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f869c006fd0 tx=0x7f869c006eb0).ready entity=mon.2 client_cookie=6f8cf25fdec6ea6b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:03.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.216+0000 7f86a3fff700 1 -- 172.21.15.187:0/2394273747 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c009720 con 0x7f86a4066f70 2020-11-20T12:10:03.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.216+0000 7f86a3fff700 1 -- 172.21.15.187:0/2394273747 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c009880 con 0x7f86a4066f70 2020-11-20T12:10:03.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.216+0000 7f86a3fff700 1 -- 172.21.15.187:0/2394273747 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c009c00 con 0x7f86a4066f70 2020-11-20T12:10:03.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.217+0000 7f86aa0d6700 1 -- 172.21.15.187:0/2394273747 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a4066f70 msgr2=0x7f86a40b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.217+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/2394273747 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a4066f70 0x7f86a40b5d50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f869c006fd0 tx=0x7f869c006eb0).stop 2020-11-20T12:10:03.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.217+0000 7f86aa0d6700 1 -- 172.21.15.187:0/2394273747 shutdown_connections 2020-11-20T12:10:03.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.217+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/2394273747 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a4066f70 0x7f86a40b5d50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.217+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/2394273747 >> v2:172.21.15.5:3300/0 conn(0x7f86a41484d0 0x7f86a4061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.217+0000 7f86aa0d6700 1 -- 172.21.15.187:0/2394273747 >> 172.21.15.187:0/2394273747 conn(0x7f86a40bab30 msgr2=0x7f86a40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:03.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.217+0000 7f86aa0d6700 1 -- 172.21.15.187:0/2394273747 shutdown_connections 2020-11-20T12:10:03.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.217+0000 7f86aa0d6700 1 -- 172.21.15.187:0/2394273747 wait complete. 2020-11-20T12:10:03.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.219+0000 7f86aa0d6700 1 Processor -- start 2020-11-20T12:10:03.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.219+0000 7f86aa0d6700 1 -- start start 2020-11-20T12:10:03.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.219+0000 7f86aa0d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86a40622c0 0x7f86a409e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.219+0000 7f86aa0d6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86a4066f70 0x7f86a409cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.219+0000 7f86aa0d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a41484d0 0x7f86a40b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.219+0000 7f86aa0d6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4131ac0 con 0x7f86a4066f70 2020-11-20T12:10:03.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86aa0d6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a409e7d0 con 0x7f86a40622c0 2020-11-20T12:10:03.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86aa0d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a413c910 con 0x7f86a41484d0 2020-11-20T12:10:03.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86a90d4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86a40622c0 0x7f86a409e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86a90d4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86a40622c0 0x7f86a409e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45198/0 (socket says 172.21.15.187:45198) 2020-11-20T12:10:03.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86a90d4700 1 -- 172.21.15.187:0/3250933400 learned_addr learned my addr 172.21.15.187:0/3250933400 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:03.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86a90d4700 1 -- 172.21.15.187:0/3250933400 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a41484d0 msgr2=0x7f86a40b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86a90d4700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a41484d0 0x7f86a40b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:03.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86a88d3700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86a4066f70 0x7f86a409cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86a90d4700 1 -- 172.21.15.187:0/3250933400 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86a4066f70 msgr2=0x7f86a409cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.220+0000 7f86a90d4700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86a4066f70 0x7f86a409cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.221+0000 7f86a90d4700 1 -- 172.21.15.187:0/3250933400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a40a08d0 con 0x7f86a40622c0 2020-11-20T12:10:03.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.221+0000 7f86a90d4700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86a40622c0 0x7f86a409e2e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f8694009fa0 tx=0x7f86940091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:03.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.221+0000 7f86a17fa700 1 -- 172.21.15.187:0/3250933400 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869400ce50 con 0x7f86a40622c0 2020-11-20T12:10:03.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.221+0000 7f86a17fa700 1 -- 172.21.15.187:0/3250933400 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86940123f0 con 0x7f86a40622c0 2020-11-20T12:10:03.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.221+0000 7f86a17fa700 1 -- 172.21.15.187:0/3250933400 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694012770 con 0x7f86a40622c0 2020-11-20T12:10:03.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.221+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a409d240 con 0x7f86a40622c0 2020-11-20T12:10:03.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.221+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a40a04a0 con 0x7f86a40622c0 2020-11-20T12:10:03.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.222+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a40a0720 con 0x7f86a40622c0 2020-11-20T12:10:03.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.223+0000 7f86a17fa700 1 -- 172.21.15.187:0/3250933400 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8694022070 con 0x7f86a40622c0 2020-11-20T12:10:03.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.224+0000 7f86a17fa700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f867c056d20 0x7f867c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.224+0000 7f86a17fa700 1 -- 172.21.15.187:0/3250933400 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8694070a00 con 0x7f86a40622c0 2020-11-20T12:10:03.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.224+0000 7f86a88d3700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f867c056d20 0x7f867c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.224+0000 7f86a88d3700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f867c056d20 0x7f867c0591e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f86a4147aa0 tx=0x7f8698009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:03.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.227+0000 7f86a17fa700 1 -- 172.21.15.187:0/3250933400 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8694045e00 con 0x7f86a40622c0 2020-11-20T12:10:03.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.405+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f86a4061a20 con 0x7f86a40622c0 2020-11-20T12:10:03.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.406+0000 7f86a17fa700 1 -- 172.21.15.187:0/3250933400 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v61) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8694049420 con 0x7f86a40622c0 2020-11-20T12:10:03.411 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:10:03.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f867c056d20 msgr2=0x7f867c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f867c056d20 0x7f867c0591e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f86a4147aa0 tx=0x7f8698009040).stop 2020-11-20T12:10:03.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86a40622c0 msgr2=0x7f86a409e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86a40622c0 0x7f86a409e2e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f8694009fa0 tx=0x7f86940091d0).stop 2020-11-20T12:10:03.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 shutdown_connections 2020-11-20T12:10:03.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f867c056d20 0x7f867c0591e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86a41484d0 0x7f86a40b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:03.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86a40622c0 0x7f86a409e2e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 --2- 172.21.15.187:0/3250933400 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86a4066f70 0x7f86a409cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.408+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 >> 172.21.15.187:0/3250933400 conn(0x7f86a40bab30 msgr2=0x7f86a40b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:03.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.409+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 shutdown_connections 2020-11-20T12:10:03.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.409+0000 7f86aa0d6700 1 -- 172.21.15.187:0/3250933400 wait complete. 2020-11-20T12:10:03.435 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:10:03.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.680+0000 7fb478490700 1 Processor -- start 2020-11-20T12:10:03.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.681+0000 7fb478490700 1 -- start start 2020-11-20T12:10:03.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.682+0000 7fb478490700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.682+0000 7fb478490700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb470068030 0x7fb470146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.682+0000 7fb478490700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb470066420 con 0x7fb470065300 2020-11-20T12:10:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.682+0000 7fb478490700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4700b7380 con 0x7fb470068030 2020-11-20T12:10:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.682+0000 7fb478490700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb470061f60 con 0x7fb47009a8a0 2020-11-20T12:10:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.682+0000 7fb47622c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb470065300 0x7fb470062400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41810/0 (socket says 172.21.15.187:41810) 2020-11-20T12:10:03.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.682+0000 7fb47622c700 1 -- 172.21.15.187:0/3806208041 learned_addr learned my addr 172.21.15.187:0/3806208041 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:03.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.683+0000 7fb476a2d700 1 --2- 172.21.15.187:0/3806208041 >> v2:172.21.15.5:3300/0 conn(0x7fb470068030 0x7fb470146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.683+0000 7fb475a2b700 1 --2- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.683+0000 7fb47522a700 1 -- 172.21.15.187:0/3806208041 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839532139 0 0) 0x7fb470066420 con 0x7fb470065300 2020-11-20T12:10:03.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.683+0000 7fb47522a700 1 -- 172.21.15.187:0/3806208041 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb458003240 con 0x7fb470065300 2020-11-20T12:10:03.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.684+0000 7fb476a2d700 1 -- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 msgr2=0x7fb4700656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:03.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.684+0000 7fb476a2d700 1 --2- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700656e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.684+0000 7fb476a2d700 1 -- 172.21.15.187:0/3806208041 >> v1:172.21.15.187:6789/0 conn(0x7fb470065300 legacy=0x7fb470062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.684+0000 7fb47522a700 1 -- 172.21.15.187:0/3806208041 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3754715303 0 0) 0x7fb458003240 con 0x7fb470065300 2020-11-20T12:10:03.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.684+0000 7fb476a2d700 1 -- 172.21.15.187:0/3806208041 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4700b6fd0 con 0x7fb470068030 2020-11-20T12:10:03.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.684+0000 7fb475a2b700 1 --2- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:10:03.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.684+0000 7fb476a2d700 1 --2- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 0x7fb470146950 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fb468009c90 tx=0x7fb4680091d0).ready entity=mon.0 client_cookie=b05a11e511341d47 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:03.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.685+0000 7fb47522a700 1 -- 172.21.15.187:0/3806208041 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c0014e0 con 0x7fb470068030 2020-11-20T12:10:03.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.685+0000 7fb47522a700 1 -- 172.21.15.187:0/3806208041 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46800b740 con 0x7fb470068030 2020-11-20T12:10:03.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.685+0000 7fb47522a700 1 -- 172.21.15.187:0/3806208041 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800bac0 con 0x7fb470068030 2020-11-20T12:10:03.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.685+0000 7fb478490700 1 -- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 msgr2=0x7fb470146950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.685+0000 7fb478490700 1 --2- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 0x7fb470146950 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fb468009c90 tx=0x7fb4680091d0).stop 2020-11-20T12:10:03.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.686+0000 7fb478490700 1 -- 172.21.15.187:0/3806208041 shutdown_connections 2020-11-20T12:10:03.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.686+0000 7fb478490700 1 --2- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 0x7fb470146950 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.686+0000 7fb478490700 1 --2- 172.21.15.187:0/3806208041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.686+0000 7fb478490700 1 -- 172.21.15.187:0/3806208041 >> 172.21.15.187:0/3806208041 conn(0x7fb4700bacf0 msgr2=0x7fb4700bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:03.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.686+0000 7fb478490700 1 -- 172.21.15.187:0/3806208041 shutdown_connections 2020-11-20T12:10:03.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.686+0000 7fb478490700 1 -- 172.21.15.187:0/3806208041 wait complete. 2020-11-20T12:10:03.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.688+0000 7fb478490700 1 Processor -- start 2020-11-20T12:10:03.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.688+0000 7fb478490700 1 -- start start 2020-11-20T12:10:03.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.689+0000 7fb478490700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb470065300 0x7fb4701566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.689+0000 7fb478490700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 0x7fb470145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.689+0000 7fb478490700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.689+0000 7fb478490700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb470156c90 con 0x7fb470068030 2020-11-20T12:10:03.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.689+0000 7fb478490700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47014a650 con 0x7fb470065300 2020-11-20T12:10:03.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.689+0000 7fb478490700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb470146450 con 0x7fb47009a8a0 2020-11-20T12:10:03.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb47622c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb470065300 0x7fb4701566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb47622c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb470065300 0x7fb4701566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45210/0 (socket says 172.21.15.187:45210) 2020-11-20T12:10:03.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb47622c700 1 -- 172.21.15.187:0/297877358 learned_addr learned my addr 172.21.15.187:0/297877358 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:03.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb476a2d700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb475a2b700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 0x7fb470145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb47622c700 1 -- 172.21.15.187:0/297877358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 msgr2=0x7fb4700b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb47622c700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb47622c700 1 -- 172.21.15.187:0/297877358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 msgr2=0x7fb470145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb47622c700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 0x7fb470145f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.690+0000 7fb47622c700 1 -- 172.21.15.187:0/297877358 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47009ec60 con 0x7fb470065300 2020-11-20T12:10:03.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.691+0000 7fb47622c700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb470065300 0x7fb4701566d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fb46c00cb10 tx=0x7fb46c001a70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:03.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.691+0000 7fb45e7fc700 1 -- 172.21.15.187:0/297877358 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c007170 con 0x7fb470065300 2020-11-20T12:10:03.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.691+0000 7fb45e7fc700 1 -- 172.21.15.187:0/297877358 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c0072d0 con 0x7fb470065300 2020-11-20T12:10:03.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.691+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47009ef00 con 0x7fb470065300 2020-11-20T12:10:03.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.691+0000 7fb45e7fc700 1 -- 172.21.15.187:0/297877358 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c004940 con 0x7fb470065300 2020-11-20T12:10:03.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.691+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4700b2ba0 con 0x7fb470065300 2020-11-20T12:10:03.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.692+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb470044f60 con 0x7fb470065300 2020-11-20T12:10:03.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.693+0000 7fb45e7fc700 1 -- 172.21.15.187:0/297877358 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb46c010070 con 0x7fb470065300 2020-11-20T12:10:03.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.693+0000 7fb45e7fc700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb448056d20 0x7fb4480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:03.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.694+0000 7fb45e7fc700 1 -- 172.21.15.187:0/297877358 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb46c06fbe0 con 0x7fb470065300 2020-11-20T12:10:03.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.694+0000 7fb475a2b700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb448056d20 0x7fb4480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:03.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.694+0000 7fb475a2b700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb448056d20 0x7fb4480591e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fb470156bc0 tx=0x7fb460008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:03.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.698+0000 7fb45e7fc700 1 -- 172.21.15.187:0/297877358 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb46c044ed0 con 0x7fb470065300 2020-11-20T12:10:03.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.882+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb47009f090 con 0x7fb470065300 2020-11-20T12:10:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.883+0000 7fb45e7fc700 1 -- 172.21.15.187:0/297877358 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v61) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb46c0484f0 con 0x7fb470065300 2020-11-20T12:10:03.887 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:03.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb448056d20 msgr2=0x7fb4480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb448056d20 0x7fb4480591e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fb470156bc0 tx=0x7fb460008040).stop 2020-11-20T12:10:03.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb470065300 msgr2=0x7fb4701566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:03.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb470065300 0x7fb4701566d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fb46c00cb10 tx=0x7fb46c001a70).stop 2020-11-20T12:10:03.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 shutdown_connections 2020-11-20T12:10:03.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb448056d20 0x7fb4480591e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb47009a8a0 0x7fb4700b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb470065300 0x7fb4701566d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 --2- 172.21.15.187:0/297877358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb470068030 0x7fb470145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 >> 172.21.15.187:0/297877358 conn(0x7fb4700bacf0 msgr2=0x7fb470066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 shutdown_connections 2020-11-20T12:10:03.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:03.886+0000 7fb478490700 1 -- 172.21.15.187:0/297877358 wait complete. 2020-11-20T12:10:03.914 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:10:04.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.162+0000 7ff54c8da700 1 Processor -- start 2020-11-20T12:10:04.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.162+0000 7ff54c8da700 1 -- start start 2020-11-20T12:10:04.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.163+0000 7ff54c8da700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5480bf800 0x7ff54814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.163+0000 7ff54c8da700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff5480bfbe0 0x7ff5481480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.163+0000 7ff54c8da700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff54809f560 0x7ff5481485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.163+0000 7ff54c8da700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54814d110 con 0x7ff54809f560 2020-11-20T12:10:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff54c8da700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff548148aa0 con 0x7ff5480bfbe0 2020-11-20T12:10:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff54c8da700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54814cbd0 con 0x7ff5480bf800 2020-11-20T12:10:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff53effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff5480bfbe0 0x7ff5481480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff53effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff5480bfbe0 0x7ff5481480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45220/0 (socket says 172.21.15.187:45220) 2020-11-20T12:10:04.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff53effd700 1 -- 172.21.15.187:0/4240329629 learned_addr learned my addr 172.21.15.187:0/4240329629 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:04.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff53effd700 1 -- 172.21.15.187:0/4240329629 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5480bf800 msgr2=0x7ff54814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:04.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff5477fe700 1 --2- 172.21.15.187:0/4240329629 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5480bf800 0x7ff54814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff53effd700 1 --2- 172.21.15.187:0/4240329629 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5480bf800 0x7ff54814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff53effd700 1 -- 172.21.15.187:0/4240329629 >> v2:172.21.15.5:3300/0 conn(0x7ff54809f560 msgr2=0x7ff5481485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:04.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff53effd700 1 --2- 172.21.15.187:0/4240329629 >> v2:172.21.15.5:3300/0 conn(0x7ff54809f560 0x7ff5481485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:04.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff53effd700 1 -- 172.21.15.187:0/4240329629 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54814ce70 con 0x7ff5480bfbe0 2020-11-20T12:10:04.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.164+0000 7ff5477fe700 1 --2- 172.21.15.187:0/4240329629 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5480bf800 0x7ff54814c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.165+0000 7ff53effd700 1 --2- 172.21.15.187:0/4240329629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5480bfbe0 0x7ff5481480c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7ff548068830 tx=0x7ff530006eb0).ready entity=mon.1 client_cookie=3505465f0689ae25 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.165+0000 7ff546ffd700 1 -- 172.21.15.187:0/4240329629 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff530009960 con 0x7ff5480bfbe0 2020-11-20T12:10:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.165+0000 7ff546ffd700 1 -- 172.21.15.187:0/4240329629 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff530009ac0 con 0x7ff5480bfbe0 2020-11-20T12:10:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.165+0000 7ff546ffd700 1 -- 172.21.15.187:0/4240329629 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff530009e40 con 0x7ff5480bfbe0 2020-11-20T12:10:04.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 -- 172.21.15.187:0/4240329629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5480bfbe0 msgr2=0x7ff5481480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 --2- 172.21.15.187:0/4240329629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5480bfbe0 0x7ff5481480c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7ff548068830 tx=0x7ff530006eb0).stop 2020-11-20T12:10:04.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 -- 172.21.15.187:0/4240329629 shutdown_connections 2020-11-20T12:10:04.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 --2- 172.21.15.187:0/4240329629 >> v2:172.21.15.5:3300/0 conn(0x7ff54809f560 0x7ff5481485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:04.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 --2- 172.21.15.187:0/4240329629 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5480bf800 0x7ff54814c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 --2- 172.21.15.187:0/4240329629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5480bfbe0 0x7ff5481480c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 -- 172.21.15.187:0/4240329629 >> 172.21.15.187:0/4240329629 conn(0x7ff548068370 msgr2=0x7ff5480bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:04.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 -- 172.21.15.187:0/4240329629 shutdown_connections 2020-11-20T12:10:04.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.166+0000 7ff54c8da700 1 -- 172.21.15.187:0/4240329629 wait complete. 2020-11-20T12:10:04.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.167+0000 7ff54c8da700 1 Processor -- start 2020-11-20T12:10:04.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.168+0000 7ff54c8da700 1 -- start start 2020-11-20T12:10:04.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.168+0000 7ff54c8da700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff54809f560 0x7ff548142e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.168+0000 7ff54c8da700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5480bf800 0x7ff5481582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff54c8da700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff54815b940 0x7ff54815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff54c8da700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff548158b60 con 0x7ff5480bf800 2020-11-20T12:10:04.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff54c8da700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff548158e00 con 0x7ff54815b940 2020-11-20T12:10:04.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff54c8da700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5481590a0 con 0x7ff54809f560 2020-11-20T12:10:04.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff547fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff54815b940 0x7ff54815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff547fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff54815b940 0x7ff54815bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45228/0 (socket says 172.21.15.187:45228) 2020-11-20T12:10:04.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff547fff700 1 -- 172.21.15.187:0/1097942722 learned_addr learned my addr 172.21.15.187:0/1097942722 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:04.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff53effd700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5480bf800 0x7ff5481582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.169+0000 7ff5477fe700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff54809f560 0x7ff548142e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff547fff700 1 -- 172.21.15.187:0/1097942722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff54809f560 msgr2=0x7ff548142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff547fff700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff54809f560 0x7ff548142e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff547fff700 1 -- 172.21.15.187:0/1097942722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5480bf800 msgr2=0x7ff5481582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff547fff700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5480bf800 0x7ff5481582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff547fff700 1 -- 172.21.15.187:0/1097942722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff548143340 con 0x7ff54815b940 2020-11-20T12:10:04.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff547fff700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff54815b940 0x7ff54815bfa0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7ff54000bae0 tx=0x7ff540009220).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:04.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff53ffff700 1 -- 172.21.15.187:0/1097942722 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000c850 con 0x7ff54815b940 2020-11-20T12:10:04.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff53ffff700 1 -- 172.21.15.187:0/1097942722 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54000c9b0 con 0x7ff54815b940 2020-11-20T12:10:04.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.170+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5481587b0 con 0x7ff54815b940 2020-11-20T12:10:04.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.171+0000 7ff53ffff700 1 -- 172.21.15.187:0/1097942722 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000cb60 con 0x7ff54815b940 2020-11-20T12:10:04.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.171+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff548145e80 con 0x7ff54815b940 2020-11-20T12:10:04.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.172+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff548080050 con 0x7ff54815b940 2020-11-20T12:10:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.172+0000 7ff53ffff700 1 -- 172.21.15.187:0/1097942722 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff54001c070 con 0x7ff54815b940 2020-11-20T12:10:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.173+0000 7ff53ffff700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff524056d20 0x7ff5240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.173+0000 7ff53ffff700 1 -- 172.21.15.187:0/1097942722 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff54006f830 con 0x7ff54815b940 2020-11-20T12:10:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.173+0000 7ff5477fe700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff524056d20 0x7ff5240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.174+0000 7ff5477fe700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff524056d20 0x7ff5240591e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7ff53800a8c0 tx=0x7ff538008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:04.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.177+0000 7ff53ffff700 1 -- 172.21.15.187:0/1097942722 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff540044b20 con 0x7ff54815b940 2020-11-20T12:10:04.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.365+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff5480a4d70 con 0x7ff54815b940 2020-11-20T12:10:04.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.366+0000 7ff53ffff700 1 -- 172.21.15.187:0/1097942722 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v61) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff540048140 con 0x7ff54815b940 2020-11-20T12:10:04.370 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:04.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff524056d20 msgr2=0x7ff5240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff524056d20 0x7ff5240591e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7ff53800a8c0 tx=0x7ff538008040).stop 2020-11-20T12:10:04.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff54815b940 msgr2=0x7ff54815bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff54815b940 0x7ff54815bfa0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7ff54000bae0 tx=0x7ff540009220).stop 2020-11-20T12:10:04.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 shutdown_connections 2020-11-20T12:10:04.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff524056d20 0x7ff5240591e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff54815b940 0x7ff54815bfa0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff54809f560 0x7ff548142e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 --2- 172.21.15.187:0/1097942722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5480bf800 0x7ff5481582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.368+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 >> 172.21.15.187:0/1097942722 conn(0x7ff548068370 msgr2=0x7ff5480bef10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:04.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.369+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 shutdown_connections 2020-11-20T12:10:04.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.369+0000 7ff54c8da700 1 -- 172.21.15.187:0/1097942722 wait complete. 2020-11-20T12:10:04.396 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:10:04.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.652+0000 7f233b29b700 1 Processor -- start 2020-11-20T12:10:04.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.653+0000 7f233b29b700 1 -- start start 2020-11-20T12:10:04.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.654+0000 7f233b29b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f23341484d0 0x7f2334061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.654+0000 7f233b29b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2334066f70 0x7f23340b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.654+0000 7f233b29b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23340b1300 con 0x7f23340622c0 2020-11-20T12:10:04.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.654+0000 7f233b29b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2334067730 con 0x7f23341484d0 2020-11-20T12:10:04.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.654+0000 7f233b29b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23340b15a0 con 0x7f2334066f70 2020-11-20T12:10:04.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.655+0000 7f233aa9a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2334066f70 0x7f23340b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.655+0000 7f233aa9a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2334066f70 0x7f23340b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45236/0 (socket says 172.21.15.187:45236) 2020-11-20T12:10:04.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.655+0000 7f233aa9a700 1 -- 172.21.15.187:0/3702206868 learned_addr learned my addr 172.21.15.187:0/3702206868 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:04.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.655+0000 7f233a299700 1 --2- 172.21.15.187:0/3702206868 >> v2:172.21.15.5:3300/0 conn(0x7f23341484d0 0x7f2334061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.655+0000 7f233aa9a700 1 -- 172.21.15.187:0/3702206868 >> v2:172.21.15.5:3300/0 conn(0x7f23341484d0 msgr2=0x7f2334061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:04.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.655+0000 7f233aa9a700 1 --2- 172.21.15.187:0/3702206868 >> v2:172.21.15.5:3300/0 conn(0x7f23341484d0 0x7f2334061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.655+0000 7f233aa9a700 1 -- 172.21.15.187:0/3702206868 >> v1:172.21.15.5:6789/0 conn(0x7f23340622c0 legacy=0x7f23340b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.655+0000 7f233aa9a700 1 -- 172.21.15.187:0/3702206868 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23340b1840 con 0x7f2334066f70 2020-11-20T12:10:04.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.656+0000 7f233a299700 1 --2- 172.21.15.187:0/3702206868 >> v2:172.21.15.5:3300/0 conn(0x7f23341484d0 0x7f2334061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:04.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.656+0000 7f233aa9a700 1 --2- 172.21.15.187:0/3702206868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b5d50 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7f2334045000 tx=0x7f232c009290).ready entity=mon.1 client_cookie=d21bf4890a3b394 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:04.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.656+0000 7f2339297700 1 -- 172.21.15.187:0/3702206868 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c00bbc0 con 0x7f2334066f70 2020-11-20T12:10:04.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.656+0000 7f2339297700 1 -- 172.21.15.187:0/3702206868 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c00bd20 con 0x7f2334066f70 2020-11-20T12:10:04.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.656+0000 7f2339297700 1 -- 172.21.15.187:0/3702206868 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011610 con 0x7f2334066f70 2020-11-20T12:10:04.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.657+0000 7f233b29b700 1 -- 172.21.15.187:0/3702206868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 msgr2=0x7f23340b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.657+0000 7f233b29b700 1 --2- 172.21.15.187:0/3702206868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b5d50 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7f2334045000 tx=0x7f232c009290).stop 2020-11-20T12:10:04.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.657+0000 7f233b29b700 1 -- 172.21.15.187:0/3702206868 shutdown_connections 2020-11-20T12:10:04.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.657+0000 7f233b29b700 1 --2- 172.21.15.187:0/3702206868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b5d50 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.657+0000 7f233b29b700 1 --2- 172.21.15.187:0/3702206868 >> v2:172.21.15.5:3300/0 conn(0x7f23341484d0 0x7f2334061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.657+0000 7f233b29b700 1 -- 172.21.15.187:0/3702206868 >> 172.21.15.187:0/3702206868 conn(0x7f23340bab30 msgr2=0x7f23340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:04.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.657+0000 7f233b29b700 1 -- 172.21.15.187:0/3702206868 shutdown_connections 2020-11-20T12:10:04.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.657+0000 7f233b29b700 1 -- 172.21.15.187:0/3702206868 wait complete. 2020-11-20T12:10:04.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.659+0000 7f233b29b700 1 Processor -- start 2020-11-20T12:10:04.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.659+0000 7f233b29b700 1 -- start start 2020-11-20T12:10:04.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f233b29b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23340622c0 0x7f233409dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f233b29b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f233b29b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23341484d0 0x7f2334139d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f233b29b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233409a5f0 con 0x7f23341484d0 2020-11-20T12:10:04.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f233b29b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23340b4860 con 0x7f2334066f70 2020-11-20T12:10:04.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f233b29b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233409a820 con 0x7f23340622c0 2020-11-20T12:10:04.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f2339a98700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f2339a98700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45238/0 (socket says 172.21.15.187:45238) 2020-11-20T12:10:04.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.660+0000 7f2339a98700 1 -- 172.21.15.187:0/1705999076 learned_addr learned my addr 172.21.15.187:0/1705999076 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:04.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f2339a98700 1 -- 172.21.15.187:0/1705999076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23340622c0 msgr2=0x7f233409dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f233aa9a700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23341484d0 0x7f2334139d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f2339a98700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23340622c0 0x7f233409dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:04.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f2339a98700 1 -- 172.21.15.187:0/1705999076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23341484d0 msgr2=0x7f2334139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f2339a98700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23341484d0 0x7f2334139d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f2339a98700 1 -- 172.21.15.187:0/1705999076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233409f3f0 con 0x7f2334066f70 2020-11-20T12:10:04.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f2339a98700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b3ee0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f232800c710 tx=0x7f2328007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:04.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f23267fc700 1 -- 172.21.15.187:0/1705999076 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328007020 con 0x7f2334066f70 2020-11-20T12:10:04.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f23267fc700 1 -- 172.21.15.187:0/1705999076 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2328007180 con 0x7f2334066f70 2020-11-20T12:10:04.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233409e190 con 0x7f2334066f70 2020-11-20T12:10:04.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f23267fc700 1 -- 172.21.15.187:0/1705999076 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232800e560 con 0x7f2334066f70 2020-11-20T12:10:04.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.661+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23340b4590 con 0x7f2334066f70 2020-11-20T12:10:04.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.663+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23340ba880 con 0x7f2334066f70 2020-11-20T12:10:04.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.663+0000 7f23267fc700 1 -- 172.21.15.187:0/1705999076 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2328010070 con 0x7f2334066f70 2020-11-20T12:10:04.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.664+0000 7f23267fc700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f230c056d20 0x7f230c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:04.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.664+0000 7f23267fc700 1 -- 172.21.15.187:0/1705999076 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f232806f510 con 0x7f2334066f70 2020-11-20T12:10:04.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.664+0000 7f233a299700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f230c056d20 0x7f230c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:04.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.665+0000 7f233a299700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f230c056d20 0x7f230c0591e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f233409ea40 tx=0x7f2330008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:04.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.668+0000 7f23267fc700 1 -- 172.21.15.187:0/1705999076 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2328044910 con 0x7f2334066f70 2020-11-20T12:10:04.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.852+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f233409a9e0 con 0x7f2334066f70 2020-11-20T12:10:04.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.853+0000 7f23267fc700 1 -- 172.21.15.187:0/1705999076 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v61) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2328047f30 con 0x7f2334066f70 2020-11-20T12:10:04.856 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:04.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f230c056d20 msgr2=0x7f230c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f230c056d20 0x7f230c0591e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f233409ea40 tx=0x7f2330008040).stop 2020-11-20T12:10:04.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 msgr2=0x7f23340b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:04.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b3ee0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f232800c710 tx=0x7f2328007c20).stop 2020-11-20T12:10:04.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 shutdown_connections 2020-11-20T12:10:04.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f230c056d20 0x7f230c0591e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23341484d0 0x7f2334139d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f23340622c0 0x7f233409dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:04.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 --2- 172.21.15.187:0/1705999076 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2334066f70 0x7f23340b3ee0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:04.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 >> 172.21.15.187:0/1705999076 conn(0x7f23340bab30 msgr2=0x7f233413a920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:04.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 shutdown_connections 2020-11-20T12:10:04.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:04.856+0000 7f233b29b700 1 -- 172.21.15.187:0/1705999076 wait complete. 2020-11-20T12:10:04.883 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:05.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.105+0000 7f80a759e700 1 Processor -- start 2020-11-20T12:10:05.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.106+0000 7f80a759e700 1 -- start start 2020-11-20T12:10:05.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.107+0000 7f80a759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f80a81484d0 0x7f80a8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.107+0000 7f80a759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a80b1300 con 0x7f80a8066f70 2020-11-20T12:10:05.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.107+0000 7f80a759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a80b15a0 con 0x7f80a80622c0 2020-11-20T12:10:05.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.107+0000 7f80a759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a8067730 con 0x7f80a81484d0 2020-11-20T12:10:05.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.107+0000 7f809dd9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f80a80622c0 0x7f80a80b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41854/0 (socket says 172.21.15.187:41854) 2020-11-20T12:10:05.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.107+0000 7f809dd9b700 1 -- 172.21.15.187:0/2028018637 learned_addr learned my addr 172.21.15.187:0/2028018637 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:05.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864694178 0 0) 0x7f80a80b15a0 con 0x7f80a80622c0 2020-11-20T12:10:05.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8088003240 con 0x7f80a80622c0 2020-11-20T12:10:05.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3046808961 0 0) 0x7f8088003240 con 0x7f80a80622c0 2020-11-20T12:10:05.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 >> v2:172.21.15.5:3300/0 conn(0x7f80a81484d0 msgr2=0x7f80a8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:05.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a659c700 1 --2- 172.21.15.187:0/2028018637 >> v2:172.21.15.5:3300/0 conn(0x7f80a81484d0 0x7f80a8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 --2- 172.21.15.187:0/2028018637 >> v2:172.21.15.5:3300/0 conn(0x7f80a81484d0 0x7f80a8061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 >> v1:172.21.15.5:6789/0 conn(0x7f80a8066f70 legacy=0x7f80a80b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a80b1840 con 0x7f80a80622c0 2020-11-20T12:10:05.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f80900014e0 con 0x7f80a80622c0 2020-11-20T12:10:05.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 >> v1:172.21.15.187:6789/0 conn(0x7f80a80622c0 legacy=0x7f80a80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.108+0000 7f80a5d9b700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8088003730 0x7f8088003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f80a5d9b700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80880044e0 0x7f8088004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f80a5d9b700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8088004e50 0x7f808800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8067730 con 0x7f8088004e50 2020-11-20T12:10:05.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8088004260 con 0x7f80880044e0 2020-11-20T12:10:05.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f808800bc20 con 0x7f8088003730 2020-11-20T12:10:05.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f809dd9b700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80880044e0 0x7f8088004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f80a659c700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8088003730 0x7f8088003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f80a6d9d700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8088004e50 0x7f808800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.109+0000 7f809dd9b700 1 -- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8088003730 msgr2=0x7f8088003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:05.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f809dd9b700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8088003730 0x7f8088003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f809dd9b700 1 -- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8088004e50 msgr2=0x7f808800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:05.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f809dd9b700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8088004e50 0x7f808800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f809dd9b700 1 -- 172.21.15.187:0/2028018637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8088003240 con 0x7f80880044e0 2020-11-20T12:10:05.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f80a6d9d700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8088004e50 0x7f808800b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:05.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f80a6d9d700 1 -- 172.21.15.187:0/2028018637 reap_dead start 2020-11-20T12:10:05.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f809dd9b700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80880044e0 0x7f8088004960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f80a8062720 tx=0x7f80900077d0).ready entity=mon.1 client_cookie=848f0055fe846e60 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:05.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809000ce70 con 0x7f80880044e0 2020-11-20T12:10:05.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80900072e0 con 0x7f80880044e0 2020-11-20T12:10:05.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.110+0000 7f80a5d9b700 1 -- 172.21.15.187:0/2028018637 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090005800 con 0x7f80880044e0 2020-11-20T12:10:05.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.111+0000 7f80a759e700 1 -- 172.21.15.187:0/2028018637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80880044e0 msgr2=0x7f8088004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.111+0000 7f80a759e700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80880044e0 0x7f8088004960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f80a8062720 tx=0x7f80900077d0).stop 2020-11-20T12:10:05.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.111+0000 7f80a759e700 1 -- 172.21.15.187:0/2028018637 shutdown_connections 2020-11-20T12:10:05.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.111+0000 7f80a759e700 1 --2- 172.21.15.187:0/2028018637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80880044e0 0x7f8088004960 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.111+0000 7f80a759e700 1 -- 172.21.15.187:0/2028018637 >> 172.21.15.187:0/2028018637 conn(0x7f80a80bab30 msgr2=0x7f80a80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:05.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.111+0000 7f80a759e700 1 -- 172.21.15.187:0/2028018637 shutdown_connections 2020-11-20T12:10:05.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.111+0000 7f80a759e700 1 -- 172.21.15.187:0/2028018637 wait complete. 2020-11-20T12:10:05.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.112+0000 7f80a759e700 1 Processor -- start 2020-11-20T12:10:05.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.113+0000 7f80a759e700 1 -- start start 2020-11-20T12:10:05.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.113+0000 7f80a759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80880044e0 0x7f80a80a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.113+0000 7f80a759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a80990a0 0x7f80a8141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.113+0000 7f80a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80a8143aa0 0x7f80a809f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.113+0000 7f80a759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a814ac20 con 0x7f80880044e0 2020-11-20T12:10:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.113+0000 7f80a759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80995f0 con 0x7f80a80990a0 2020-11-20T12:10:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.113+0000 7f80a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8099820 con 0x7f80a8143aa0 2020-11-20T12:10:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a80990a0 0x7f80a8141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a80990a0 0x7f80a8141ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45260/0 (socket says 172.21.15.187:45260) 2020-11-20T12:10:05.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 -- 172.21.15.187:0/2253660881 learned_addr learned my addr 172.21.15.187:0/2253660881 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:05.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f80a6d9d700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80a8143aa0 0x7f80a809f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 -- 172.21.15.187:0/2253660881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80a8143aa0 msgr2=0x7f80a809f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80a8143aa0 0x7f80a809f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f80a659c700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80880044e0 0x7f80a80a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 -- 172.21.15.187:0/2253660881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80880044e0 msgr2=0x7f80a80a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80880044e0 0x7f80a80a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 -- 172.21.15.187:0/2253660881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a81452a0 con 0x7f80a80990a0 2020-11-20T12:10:05.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.114+0000 7f809dd9b700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a80990a0 0x7f80a8141ab0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f8090000fb0 tx=0x7f8090017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:05.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.115+0000 7f809effd700 1 -- 172.21.15.187:0/2253660881 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090011470 con 0x7f80a80990a0 2020-11-20T12:10:05.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.115+0000 7f809effd700 1 -- 172.21.15.187:0/2253660881 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80900115d0 con 0x7f80a80990a0 2020-11-20T12:10:05.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.115+0000 7f809effd700 1 -- 172.21.15.187:0/2253660881 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090011950 con 0x7f80a80990a0 2020-11-20T12:10:05.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.115+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a8141fa0 con 0x7f80a80990a0 2020-11-20T12:10:05.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.115+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a80a0120 con 0x7f80a80990a0 2020-11-20T12:10:05.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.116+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a80a0310 con 0x7f80a80990a0 2020-11-20T12:10:05.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.116+0000 7f809effd700 1 -- 172.21.15.187:0/2253660881 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8090011d40 con 0x7f80a80990a0 2020-11-20T12:10:05.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.117+0000 7f809effd700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8084056d20 0x7f80840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.117+0000 7f809effd700 1 -- 172.21.15.187:0/2253660881 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8090071d30 con 0x7f80a80990a0 2020-11-20T12:10:05.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.117+0000 7f80a659c700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8084056d20 0x7f80840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.118+0000 7f80a659c700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8084056d20 0x7f80840591e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f809800a900 tx=0x7f8098008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:05.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.121+0000 7f809effd700 1 -- 172.21.15.187:0/2253660881 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8090047020 con 0x7f80a80990a0 2020-11-20T12:10:05.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.358+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a80a0310 con 0x7f80a80990a0 2020-11-20T12:10:05.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.359+0000 7f809effd700 1 -- 172.21.15.187:0/2253660881 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f809004a640 con 0x7f80a80990a0 2020-11-20T12:10:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:05.363 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2076591993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2076591993},{"type":"v1","addr":"172.21.15.5:6833","nonce":2076591993}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:09:07.256865+0000","modified":"2020-11-20T12:09:27.418850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6178":{"gid":6178,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":6,"addr":"172.21.15.187:6835/609118159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":609118159},{"type":"v1","addr":"172.21.15.187:6835","nonce":609118159}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2020-11-20T12:10:05.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8084056d20 msgr2=0x7f80840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8084056d20 0x7f80840591e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f809800a900 tx=0x7f8098008040).stop 2020-11-20T12:10:05.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a80990a0 msgr2=0x7f80a8141ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a80990a0 0x7f80a8141ab0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f8090000fb0 tx=0x7f8090017000).stop 2020-11-20T12:10:05.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 shutdown_connections 2020-11-20T12:10:05.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8084056d20 0x7f80840591e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80880044e0 0x7f80a80a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80a8143aa0 0x7f80a809f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 --2- 172.21.15.187:0/2253660881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a80990a0 0x7f80a8141ab0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.361+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 >> 172.21.15.187:0/2253660881 conn(0x7f80a80bab30 msgr2=0x7f80a8065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:05.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.362+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 shutdown_connections 2020-11-20T12:10:05.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.362+0000 7f80a759e700 1 -- 172.21.15.187:0/2253660881 wait complete. 2020-11-20T12:10:05.370 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 41 2020-11-20T12:10:05.384 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:10:05.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.620+0000 7f48533cd700 1 Processor -- start 2020-11-20T12:10:05.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.620+0000 7f48533cd700 1 -- start start 2020-11-20T12:10:05.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.621+0000 7f48533cd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f484c0b6650 0x7f484c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.621+0000 7f48533cd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 0x7f484c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.621+0000 7f48533cd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c0b1300 con 0x7f484c1484d0 2020-11-20T12:10:05.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.621+0000 7f48533cd700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484c067730 con 0x7f484c0b6650 2020-11-20T12:10:05.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.621+0000 7f48533cd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0b15a0 con 0x7f484c066f70 2020-11-20T12:10:05.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.622+0000 7f4851bca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f484c0b6650 0x7f484c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.622+0000 7f4851bca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f484c0b6650 0x7f484c0afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45266/0 (socket says 172.21.15.187:45266) 2020-11-20T12:10:05.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.622+0000 7f4851bca700 1 -- 172.21.15.187:0/2595317626 learned_addr learned my addr 172.21.15.187:0/2595317626 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:05.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.622+0000 7f4852bcc700 1 --2- 172.21.15.187:0/2595317626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 0x7f484c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.622+0000 7f4851bca700 1 -- 172.21.15.187:0/2595317626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 msgr2=0x7f484c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:05.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.622+0000 7f4851bca700 1 --2- 172.21.15.187:0/2595317626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 0x7f484c0aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.622+0000 7f4851bca700 1 -- 172.21.15.187:0/2595317626 >> v1:172.21.15.5:6789/0 conn(0x7f484c1484d0 legacy=0x7f484c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:05.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.622+0000 7f4851bca700 1 -- 172.21.15.187:0/2595317626 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0b1840 con 0x7f484c0b6650 2020-11-20T12:10:05.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.623+0000 7f4851bca700 1 --2- 172.21.15.187:0/2595317626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c0afa90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f4840006fd0 tx=0x7f4840006eb0).ready entity=mon.1 client_cookie=9e7062224eb67389 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:05.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.623+0000 7f48513c9700 1 -- 172.21.15.187:0/2595317626 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4840009890 con 0x7f484c0b6650 2020-11-20T12:10:05.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.623+0000 7f48513c9700 1 -- 172.21.15.187:0/2595317626 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48400099f0 con 0x7f484c0b6650 2020-11-20T12:10:05.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.623+0000 7f48513c9700 1 -- 172.21.15.187:0/2595317626 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4840009d70 con 0x7f484c0b6650 2020-11-20T12:10:05.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.623+0000 7f48533cd700 1 -- 172.21.15.187:0/2595317626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 msgr2=0x7f484c0afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.623+0000 7f48533cd700 1 --2- 172.21.15.187:0/2595317626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c0afa90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f4840006fd0 tx=0x7f4840006eb0).stop 2020-11-20T12:10:05.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.624+0000 7f48533cd700 1 -- 172.21.15.187:0/2595317626 shutdown_connections 2020-11-20T12:10:05.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.624+0000 7f48533cd700 1 --2- 172.21.15.187:0/2595317626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 0x7f484c0aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.624+0000 7f48533cd700 1 --2- 172.21.15.187:0/2595317626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c0afa90 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.624+0000 7f48533cd700 1 -- 172.21.15.187:0/2595317626 >> 172.21.15.187:0/2595317626 conn(0x7f484c0bab30 msgr2=0x7f484c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:05.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.624+0000 7f48533cd700 1 -- 172.21.15.187:0/2595317626 shutdown_connections 2020-11-20T12:10:05.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.624+0000 7f48533cd700 1 -- 172.21.15.187:0/2595317626 wait complete. 2020-11-20T12:10:05.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.625+0000 7f48533cd700 1 Processor -- start 2020-11-20T12:10:05.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.626+0000 7f48533cd700 1 -- start start 2020-11-20T12:10:05.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f48533cd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 0x7f484c0b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f48533cd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f48533cd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c1484d0 0x7f484c145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f48533cd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c09b5a0 con 0x7f484c1484d0 2020-11-20T12:10:05.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f48533cd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c09d5d0 con 0x7f484c0b6650 2020-11-20T12:10:05.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f48533cd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c09d330 con 0x7f484c066f70 2020-11-20T12:10:05.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f4851bca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f4851bca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c144d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45274/0 (socket says 172.21.15.187:45274) 2020-11-20T12:10:05.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f4851bca700 1 -- 172.21.15.187:0/1461406007 learned_addr learned my addr 172.21.15.187:0/1461406007 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:05.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f48523cb700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 0x7f484c0b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.627+0000 7f4852bcc700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c1484d0 0x7f484c145210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f4851bca700 1 -- 172.21.15.187:0/1461406007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 msgr2=0x7f484c0b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f4851bca700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 0x7f484c0b32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f4851bca700 1 -- 172.21.15.187:0/1461406007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c1484d0 msgr2=0x7f484c145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f4851bca700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c1484d0 0x7f484c145210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f4851bca700 1 -- 172.21.15.187:0/1461406007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0b37e0 con 0x7f484c0b6650 2020-11-20T12:10:05.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f4851bca700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c144d20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f484c066430 tx=0x7f4840015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:05.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f483e7fc700 1 -- 172.21.15.187:0/1461406007 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484001e460 con 0x7f484c0b6650 2020-11-20T12:10:05.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c0b3970 con 0x7f484c0b6650 2020-11-20T12:10:05.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f483e7fc700 1 -- 172.21.15.187:0/1461406007 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484001e5c0 con 0x7f484c0b6650 2020-11-20T12:10:05.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.628+0000 7f483e7fc700 1 -- 172.21.15.187:0/1461406007 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484001e960 con 0x7f484c0b6650 2020-11-20T12:10:05.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.629+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c0b41e0 con 0x7f484c0b6650 2020-11-20T12:10:05.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.630+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c061a20 con 0x7f484c0b6650 2020-11-20T12:10:05.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.630+0000 7f483e7fc700 1 -- 172.21.15.187:0/1461406007 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4840025030 con 0x7f484c0b6650 2020-11-20T12:10:05.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.631+0000 7f483e7fc700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:05.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.631+0000 7f483e7fc700 1 -- 172.21.15.187:0/1461406007 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4840071d50 con 0x7f484c0b6650 2020-11-20T12:10:05.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.631+0000 7f48523cb700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:05.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.632+0000 7f48523cb700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f4848006fd0 tx=0x7f4848008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:05.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.634+0000 7f483e7fc700 1 -- 172.21.15.187:0/1461406007 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f48400470c0 con 0x7f484c0b6650 2020-11-20T12:10:05.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.819+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f484c0624e0 con 0x7f484c0b6650 2020-11-20T12:10:05.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.821+0000 7f483e7fc700 1 -- 172.21.15.187:0/1461406007 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16912 (secure 0 0 0) 0x7f4840023070 con 0x7f484c0b6650 2020-11-20T12:10:05.826 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:05.826 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":61,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:09:18.096938+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:09:05.207488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2020-11-20T12:09:06.244532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/3570428099":"2020-11-21T12:09:18.096834+0000","172.21.15.187:6834/3570428099":"2020-11-21T12:09:18.096834+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:10:05.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.823+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 msgr2=0x7f48240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.823+0000 7f48533cd700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f4848006fd0 tx=0x7f4848008040).stop 2020-11-20T12:10:05.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.823+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 msgr2=0x7f484c144d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:05.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.823+0000 7f48533cd700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c144d20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f484c066430 tx=0x7f4840015000).stop 2020-11-20T12:10:05.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.824+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 shutdown_connections 2020-11-20T12:10:05.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.824+0000 7f48533cd700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4824056d20 0x7f48240591e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.824+0000 7f48533cd700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f484c1484d0 0x7f484c145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.824+0000 7f48533cd700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f484c066f70 0x7f484c0b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.824+0000 7f48533cd700 1 --2- 172.21.15.187:0/1461406007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484c0b6650 0x7f484c144d20 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:05.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.824+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 >> 172.21.15.187:0/1461406007 conn(0x7f484c0bab30 msgr2=0x7f484c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:05.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.824+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 shutdown_connections 2020-11-20T12:10:05.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:05.824+0000 7f48533cd700 1 -- 172.21.15.187:0/1461406007 wait complete. 2020-11-20T12:10:05.849 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:10:05.850 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:10:06.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.091+0000 7f1d1ab59700 1 Processor -- start 2020-11-20T12:10:06.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.091+0000 7f1d1ab59700 1 -- start start 2020-11-20T12:10:06.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.092+0000 7f1d1ab59700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1d14145cc0 0x7f1d14066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:06.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.092+0000 7f1d1ab59700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d141487e0 0x7f1d14148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:06.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.092+0000 7f1d1ab59700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1414c170 con 0x7f1d140683f0 2020-11-20T12:10:06.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.092+0000 7f1d1ab59700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d1414c410 con 0x7f1d14145cc0 2020-11-20T12:10:06.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.092+0000 7f1d1ab59700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1414c6b0 con 0x7f1d141487e0 2020-11-20T12:10:06.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.093+0000 7f1d1a358700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d141487e0 0x7f1d14148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:06.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.094+0000 7f1d19356700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1d140683f0 0x7f1d14065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:35398/0 (socket says 172.21.15.187:35398) 2020-11-20T12:10:06.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.094+0000 7f1d19356700 1 -- 172.21.15.187:0/3601108677 learned_addr learned my addr 172.21.15.187:0/3601108677 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:06.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.094+0000 7f1d19b57700 1 --2- 172.21.15.187:0/3601108677 >> v2:172.21.15.5:3300/0 conn(0x7f1d14145cc0 0x7f1d14066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:06.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.095+0000 7f1d18b55700 1 -- 172.21.15.187:0/3601108677 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820958612 0 0) 0x7f1d1414c170 con 0x7f1d140683f0 2020-11-20T12:10:06.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.095+0000 7f1d18b55700 1 -- 172.21.15.187:0/3601108677 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d00003240 con 0x7f1d140683f0 2020-11-20T12:10:06.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.095+0000 7f1d19b57700 1 -- 172.21.15.187:0/3601108677 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d141487e0 msgr2=0x7f1d14148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:06.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.095+0000 7f1d19b57700 1 --2- 172.21.15.187:0/3601108677 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d141487e0 0x7f1d14148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:06.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.095+0000 7f1d19b57700 1 -- 172.21.15.187:0/3601108677 >> v1:172.21.15.5:6789/0 conn(0x7f1d140683f0 legacy=0x7f1d14065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:06.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.095+0000 7f1d19b57700 1 -- 172.21.15.187:0/3601108677 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d140658a0 con 0x7f1d14145cc0 2020-11-20T12:10:06.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.095+0000 7f1d19b57700 1 --2- 172.21.15.187:0/3601108677 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14145cc0 0x7f1d14066040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f1d0400b9a0 tx=0x7f1d04006df0).ready entity=mon.0 client_cookie=322adc44ef1bb1b4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:06.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.096+0000 7f1d18b55700 1 -- 172.21.15.187:0/3601108677 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04009680 con 0x7f1d14145cc0 2020-11-20T12:10:06.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.096+0000 7f1d18b55700 1 -- 172.21.15.187:0/3601108677 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d040097e0 con 0x7f1d14145cc0 2020-11-20T12:10:06.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.096+0000 7f1d18b55700 1 -- 172.21.15.187:0/3601108677 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04009b60 con 0x7f1d14145cc0 2020-11-20T12:10:06.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.097+0000 7f1d1ab59700 1 -- 172.21.15.187:0/3601108677 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14145cc0 msgr2=0x7f1d14066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:06.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.097+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/3601108677 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14145cc0 0x7f1d14066040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f1d0400b9a0 tx=0x7f1d04006df0).stop 2020-11-20T12:10:06.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.097+0000 7f1d1ab59700 1 -- 172.21.15.187:0/3601108677 shutdown_connections 2020-11-20T12:10:06.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.097+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/3601108677 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d141487e0 0x7f1d14148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:06.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.097+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/3601108677 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14145cc0 0x7f1d14066040 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:06.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.097+0000 7f1d1ab59700 1 -- 172.21.15.187:0/3601108677 >> 172.21.15.187:0/3601108677 conn(0x7f1d140bab30 msgr2=0x7f1d140baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:06.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.097+0000 7f1d1ab59700 1 -- 172.21.15.187:0/3601108677 shutdown_connections 2020-11-20T12:10:06.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.097+0000 7f1d1ab59700 1 -- 172.21.15.187:0/3601108677 wait complete. 2020-11-20T12:10:06.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.099+0000 7f1d1ab59700 1 Processor -- start 2020-11-20T12:10:06.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.099+0000 7f1d1ab59700 1 -- start start 2020-11-20T12:10:06.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.100+0000 7f1d1ab59700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d140683f0 0x7f1d1415c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:06.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.100+0000 7f1d1ab59700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d141487e0 0x7f1d1409ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:06.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.100+0000 7f1d1ab59700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14143ee0 0x7f1d14144360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:06.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.100+0000 7f1d1ab59700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14144850 con 0x7f1d14143ee0 2020-11-20T12:10:06.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.100+0000 7f1d1ab59700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1414b380 con 0x7f1d141487e0 2020-11-20T12:10:06.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.100+0000 7f1d1ab59700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1414b0e0 con 0x7f1d140683f0 2020-11-20T12:10:06.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19356700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d141487e0 0x7f1d1409ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:06.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19356700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d141487e0 0x7f1d1409ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45290/0 (socket says 172.21.15.187:45290) 2020-11-20T12:10:06.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19356700 1 -- 172.21.15.187:0/164606724 learned_addr learned my addr 172.21.15.187:0/164606724 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:06.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d1a358700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14143ee0 0x7f1d14144360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:06.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19b57700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d140683f0 0x7f1d1415c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:06.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19356700 1 -- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d140683f0 msgr2=0x7f1d1415c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:06.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19356700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d140683f0 0x7f1d1415c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:06.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19356700 1 -- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14143ee0 msgr2=0x7f1d14144360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:06.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19356700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14143ee0 0x7f1d14144360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:06.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19356700 1 -- 172.21.15.187:0/164606724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1409a780 con 0x7f1d141487e0 2020-11-20T12:10:06.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.101+0000 7f1d19b57700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d140683f0 0x7f1d1415c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:06.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.102+0000 7f1d19356700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d141487e0 0x7f1d1409ff20 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f1d1000ca60 tx=0x7f1d10007e50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:06.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.102+0000 7f1d09ffb700 1 -- 172.21.15.187:0/164606724 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d10005b80 con 0x7f1d141487e0 2020-11-20T12:10:06.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.102+0000 7f1d09ffb700 1 -- 172.21.15.187:0/164606724 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d10005ce0 con 0x7f1d141487e0 2020-11-20T12:10:06.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.102+0000 7f1d09ffb700 1 -- 172.21.15.187:0/164606724 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d10007340 con 0x7f1d141487e0 2020-11-20T12:10:06.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.102+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d140afb20 con 0x7f1d141487e0 2020-11-20T12:10:06.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.102+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1414ad60 con 0x7f1d141487e0 2020-11-20T12:10:06.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.103+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1409a910 con 0x7f1d141487e0 2020-11-20T12:10:06.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.104+0000 7f1d09ffb700 1 -- 172.21.15.187:0/164606724 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1d10010030 con 0x7f1d141487e0 2020-11-20T12:10:06.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.104+0000 7f1d09ffb700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cec056d50 0x7f1cec059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:06.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.104+0000 7f1d09ffb700 1 -- 172.21.15.187:0/164606724 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1d10070190 con 0x7f1d141487e0 2020-11-20T12:10:06.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.105+0000 7f1d19b57700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cec056d50 0x7f1cec059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:06.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.105+0000 7f1d19b57700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cec056d50 0x7f1cec059210 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f1d14063360 tx=0x7f1d04015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:06.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.108+0000 7f1d09ffb700 1 -- 172.21.15.187:0/164606724 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1d10045530 con 0x7f1d141487e0 2020-11-20T12:10:06.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:06.339+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1d14061a20 con 0x7f1d141487e0 2020-11-20T12:10:07.222 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:10:07.218+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:10:07.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.234+0000 7f1d09ffb700 1 -- 172.21.15.187:0/164606724 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1d10048b50 con 0x7f1d141487e0 2020-11-20T12:10:07.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cec056d50 msgr2=0x7f1cec059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:07.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cec056d50 0x7f1cec059210 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f1d14063360 tx=0x7f1d04015000).stop 2020-11-20T12:10:07.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d141487e0 msgr2=0x7f1d1409ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:07.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d141487e0 0x7f1d1409ff20 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f1d1000ca60 tx=0x7f1d10007e50).stop 2020-11-20T12:10:07.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 shutdown_connections 2020-11-20T12:10:07.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1cec056d50 0x7f1cec059210 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:07.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d14143ee0 0x7f1d14144360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:07.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1d140683f0 0x7f1d1415c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:07.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 --2- 172.21.15.187:0/164606724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d141487e0 0x7f1d1409ff20 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:07.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.239+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 >> 172.21.15.187:0/164606724 conn(0x7f1d140bab30 msgr2=0x7f1d141450b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:07.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.240+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 shutdown_connections 2020-11-20T12:10:07.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.240+0000 7f1d1ab59700 1 -- 172.21.15.187:0/164606724 wait complete. 2020-11-20T12:10:07.247 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:10:07.284 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:10:07.342 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:10:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.507+0000 7fdb90e57700 1 Processor -- start 2020-11-20T12:10:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.508+0000 7fdb90e57700 1 -- start start 2020-11-20T12:10:07.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.509+0000 7fdb90e57700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdb8c145cc0 0x7fdb8c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.509+0000 7fdb90e57700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c1487e0 0x7fdb8c148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.509+0000 7fdb90e57700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c14c170 con 0x7fdb8c0683f0 2020-11-20T12:10:07.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.509+0000 7fdb90e57700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c14c410 con 0x7fdb8c145cc0 2020-11-20T12:10:07.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.509+0000 7fdb90e57700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c14c6b0 con 0x7fdb8c1487e0 2020-11-20T12:10:07.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.509+0000 7fdb82ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fdb8c0683f0 0x7fdb8c065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41918/0 (socket says 172.21.15.187:41918) 2020-11-20T12:10:07.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.509+0000 7fdb82ffd700 1 -- 172.21.15.187:0/612694527 learned_addr learned my addr 172.21.15.187:0/612694527 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:07.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.510+0000 7fdb8bfff700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c1487e0 0x7fdb8c148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:07.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.510+0000 7fdb8b7fe700 1 --2- 172.21.15.187:0/612694527 >> v2:172.21.15.187:3300/0 conn(0x7fdb8c145cc0 0x7fdb8c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:07.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.510+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657678215 0 0) 0x7fdb8c14c170 con 0x7fdb8c0683f0 2020-11-20T12:10:07.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.510+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb6c003240 con 0x7fdb8c0683f0 2020-11-20T12:10:07.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.510+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2052034928 0 0) 0x7fdb6c003240 con 0x7fdb8c0683f0 2020-11-20T12:10:07.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.510+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c1487e0 msgr2=0x7fdb8c148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:07.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.510+0000 7fdb8affd700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c1487e0 0x7fdb8c148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:07.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 >> v2:172.21.15.187:3300/0 conn(0x7fdb8c145cc0 msgr2=0x7fdb8c066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:07.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 --2- 172.21.15.187:0/612694527 >> v2:172.21.15.187:3300/0 conn(0x7fdb8c145cc0 0x7fdb8c066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:07.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c0658a0 con 0x7fdb8c0683f0 2020-11-20T12:10:07.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fdb740014e0 con 0x7fdb8c0683f0 2020-11-20T12:10:07.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 >> v1:172.21.15.187:6789/0 conn(0x7fdb8c0683f0 legacy=0x7fdb8c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:07.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb6c0037b0 0x7fdb6c003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c1487e0 0x7fdb6c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6c0049d0 0x7fdb6c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c14c410 con 0x7fdb6c0037b0 2020-11-20T12:10:07.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c14c6b0 con 0x7fdb6c0049d0 2020-11-20T12:10:07.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0042a0 con 0x7fdb8c1487e0 2020-11-20T12:10:07.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.511+0000 7fdb8bfff700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6c0049d0 0x7fdb6c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:07.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb82ffd700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c1487e0 0x7fdb6c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:07.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8bfff700 1 -- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c1487e0 msgr2=0x7fdb6c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:07.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8bfff700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c1487e0 0x7fdb6c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:07.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8bfff700 1 -- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb6c0037b0 msgr2=0x7fdb6c003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:07.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8bfff700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb6c0037b0 0x7fdb6c003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:07.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8bfff700 1 -- 172.21.15.187:0/612694527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c003240 con 0x7fdb6c0049d0 2020-11-20T12:10:07.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8bfff700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6c0049d0 0x7fdb6c004e50 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7fdb84009ea0 tx=0x7fdb840091d0).ready entity=mon.1 client_cookie=1ff5482dc0d6c9bb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb840095c0 con 0x7fdb6c0049d0 2020-11-20T12:10:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8400eda0 con 0x7fdb6c0049d0 2020-11-20T12:10:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.512+0000 7fdb8affd700 1 -- 172.21.15.187:0/612694527 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84012600 con 0x7fdb6c0049d0 2020-11-20T12:10:07.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.513+0000 7fdb90e57700 1 -- 172.21.15.187:0/612694527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6c0049d0 msgr2=0x7fdb6c004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.513+0000 7fdb90e57700 1 --2- 172.21.15.187:0/612694527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6c0049d0 0x7fdb6c004e50 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7fdb84009ea0 tx=0x7fdb840091d0).stop 2020-11-20T12:10:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.513+0000 7fdb8bfff700 1 -- 172.21.15.187:0/612694527 reap_dead start 2020-11-20T12:10:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.513+0000 7fdb90e57700 1 -- 172.21.15.187:0/612694527 shutdown_connections 2020-11-20T12:10:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.513+0000 7fdb90e57700 1 -- 172.21.15.187:0/612694527 >> 172.21.15.187:0/612694527 conn(0x7fdb8c0bab30 msgr2=0x7fdb8c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.513+0000 7fdb90e57700 1 -- 172.21.15.187:0/612694527 shutdown_connections 2020-11-20T12:10:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.513+0000 7fdb90e57700 1 -- 172.21.15.187:0/612694527 wait complete. 2020-11-20T12:10:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.515+0000 7fdb90e57700 1 Processor -- start 2020-11-20T12:10:07.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.515+0000 7fdb90e57700 1 -- start start 2020-11-20T12:10:07.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb90e57700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb8c147230 0x7fdb8c0b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb90e57700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c147610 0x7fdb8c15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb90e57700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb8c0b32e0 0x7fdb8c15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb90e57700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c142930 con 0x7fdb8c0b32e0 2020-11-20T12:10:07.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb90e57700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c149090 con 0x7fdb8c147230 2020-11-20T12:10:07.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb90e57700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c1492c0 con 0x7fdb8c147610 2020-11-20T12:10:07.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb8b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb8c147230 0x7fdb8c0b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:07.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb8b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb8c147230 0x7fdb8c0b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45324/0 (socket says 172.21.15.187:45324) 2020-11-20T12:10:07.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb8b7fe700 1 -- 172.21.15.187:0/1557705531 learned_addr learned my addr 172.21.15.187:0/1557705531 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:07.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb8b7fe700 1 -- 172.21.15.187:0/1557705531 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c147610 msgr2=0x7fdb8c15b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:07.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb8b7fe700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c147610 0x7fdb8c15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:07.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb8b7fe700 1 -- 172.21.15.187:0/1557705531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb8c0b32e0 msgr2=0x7fdb8c15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:07.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb8b7fe700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb8c0b32e0 0x7fdb8c15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:07.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.516+0000 7fdb8b7fe700 1 -- 172.21.15.187:0/1557705531 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c149560 con 0x7fdb8c147230 2020-11-20T12:10:07.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.517+0000 7fdb8b7fe700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb8c147230 0x7fdb8c0b4250 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7fdb7c00d6d0 tx=0x7fdb7c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:07.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.517+0000 7fdb83fff700 1 -- 172.21.15.187:0/1557705531 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c00f550 con 0x7fdb8c147230 2020-11-20T12:10:07.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.517+0000 7fdb83fff700 1 -- 172.21.15.187:0/1557705531 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c00f040 con 0x7fdb8c147230 2020-11-20T12:10:07.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.517+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c149800 con 0x7fdb8c147230 2020-11-20T12:10:07.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.517+0000 7fdb83fff700 1 -- 172.21.15.187:0/1557705531 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c00f940 con 0x7fdb8c147230 2020-11-20T12:10:07.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.517+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c0b4740 con 0x7fdb8c147230 2020-11-20T12:10:07.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.518+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8c0b49f0 con 0x7fdb8c147230 2020-11-20T12:10:07.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.519+0000 7fdb83fff700 1 -- 172.21.15.187:0/1557705531 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdb7c019020 con 0x7fdb8c147230 2020-11-20T12:10:07.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.519+0000 7fdb83fff700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb68056d20 0x7fdb680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:07.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.519+0000 7fdb83fff700 1 -- 172.21.15.187:0/1557705531 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fdb7c06fb40 con 0x7fdb8c147230 2020-11-20T12:10:07.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.520+0000 7fdb82ffd700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb68056d20 0x7fdb680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:07.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.520+0000 7fdb82ffd700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb68056d20 0x7fdb680591e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fdb74001610 tx=0x7fdb74001e20).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:07.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.523+0000 7fdb83fff700 1 -- 172.21.15.187:0/1557705531 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdb7c044f90 con 0x7fdb8c147230 2020-11-20T12:10:07.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:07.770+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdb8c149ba0 con 0x7fdb8c147230 2020-11-20T12:10:08.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.245+0000 7fdb83fff700 1 -- 172.21.15.187:0/1557705531 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdb7c0485b0 con 0x7fdb8c147230 2020-11-20T12:10:08.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.249+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb68056d20 msgr2=0x7fdb680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:08.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.249+0000 7fdb90e57700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb68056d20 0x7fdb680591e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fdb74001610 tx=0x7fdb74001e20).stop 2020-11-20T12:10:08.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.249+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb8c147230 msgr2=0x7fdb8c0b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:08.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.249+0000 7fdb90e57700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb8c147230 0x7fdb8c0b4250 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7fdb7c00d6d0 tx=0x7fdb7c008040).stop 2020-11-20T12:10:08.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.250+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 shutdown_connections 2020-11-20T12:10:08.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.250+0000 7fdb90e57700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdb68056d20 0x7fdb680591e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:08.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.250+0000 7fdb90e57700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb8c0b32e0 0x7fdb8c15bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:08.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.250+0000 7fdb90e57700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb8c147230 0x7fdb8c0b4250 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:08.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.250+0000 7fdb90e57700 1 --2- 172.21.15.187:0/1557705531 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb8c147610 0x7fdb8c15b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:08.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.250+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 >> 172.21.15.187:0/1557705531 conn(0x7fdb8c0bab30 msgr2=0x7fdb8c1452f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:08.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.250+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 shutdown_connections 2020-11-20T12:10:08.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.250+0000 7fdb90e57700 1 -- 172.21.15.187:0/1557705531 wait complete. 2020-11-20T12:10:08.284 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:10:08.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.540+0000 7f7bdbfff700 1 Processor -- start 2020-11-20T12:10:08.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.541+0000 7f7bdbfff700 1 -- start start 2020-11-20T12:10:08.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.542+0000 7f7bdbfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7bdc068230 0x7f7bdc1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:08.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.542+0000 7f7bdbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7bdc0b5f50 0x7f7bdc0b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:08.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.542+0000 7f7bdbfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bdc144b80 con 0x7f7bdc0b57b0 2020-11-20T12:10:08.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.542+0000 7f7bdbfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc1487c0 con 0x7f7bdc068230 2020-11-20T12:10:08.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.542+0000 7f7bdbfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc1489e0 con 0x7f7bdc0b5f50 2020-11-20T12:10:08.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.542+0000 7f7bdb7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7bdc0b5f50 0x7f7bdc0b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:08.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.543+0000 7f7bdb7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7bdc0b5f50 0x7f7bdc0b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45334/0 (socket says 172.21.15.187:45334) 2020-11-20T12:10:08.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.543+0000 7f7bdb7fe700 1 -- 172.21.15.187:0/84210378 learned_addr learned my addr 172.21.15.187:0/84210378 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:08.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.543+0000 7f7bd27fc700 1 --2- 172.21.15.187:0/84210378 >> v2:172.21.15.5:3300/0 conn(0x7f7bdc068230 0x7f7bdc1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:08.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.543+0000 7f7bdb7fe700 1 -- 172.21.15.187:0/84210378 >> v2:172.21.15.5:3300/0 conn(0x7f7bdc068230 msgr2=0x7f7bdc1482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:08.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.543+0000 7f7bdb7fe700 1 --2- 172.21.15.187:0/84210378 >> v2:172.21.15.5:3300/0 conn(0x7f7bdc068230 0x7f7bdc1482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:08.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.543+0000 7f7bdb7fe700 1 -- 172.21.15.187:0/84210378 >> v1:172.21.15.5:6789/0 conn(0x7f7bdc0b57b0 legacy=0x7f7bdc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:08.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.543+0000 7f7bdb7fe700 1 -- 172.21.15.187:0/84210378 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc148c80 con 0x7f7bdc0b5f50 2020-11-20T12:10:08.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bdb7fe700 1 --2- 172.21.15.187:0/84210378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc0b5f50 0x7f7bdc0b6380 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f7bdc066230 tx=0x7f7bd4009290).ready entity=mon.1 client_cookie=b9cf24a035a183ae server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:08.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bda7fc700 1 -- 172.21.15.187:0/84210378 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd400bbc0 con 0x7f7bdc0b5f50 2020-11-20T12:10:08.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bda7fc700 1 -- 172.21.15.187:0/84210378 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd400bd20 con 0x7f7bdc0b5f50 2020-11-20T12:10:08.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bda7fc700 1 -- 172.21.15.187:0/84210378 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4011610 con 0x7f7bdc0b5f50 2020-11-20T12:10:08.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bdbfff700 1 -- 172.21.15.187:0/84210378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc0b5f50 msgr2=0x7f7bdc0b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:08.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/84210378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc0b5f50 0x7f7bdc0b6380 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f7bdc066230 tx=0x7f7bd4009290).stop 2020-11-20T12:10:08.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bdbfff700 1 -- 172.21.15.187:0/84210378 shutdown_connections 2020-11-20T12:10:08.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/84210378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc0b5f50 0x7f7bdc0b6380 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/84210378 >> v2:172.21.15.5:3300/0 conn(0x7f7bdc068230 0x7f7bdc1482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.544+0000 7f7bdbfff700 1 -- 172.21.15.187:0/84210378 >> 172.21.15.187:0/84210378 conn(0x7f7bdc0bab30 msgr2=0x7f7bdc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.545+0000 7f7bdbfff700 1 -- 172.21.15.187:0/84210378 shutdown_connections 2020-11-20T12:10:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.545+0000 7f7bdbfff700 1 -- 172.21.15.187:0/84210378 wait complete. 2020-11-20T12:10:08.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.546+0000 7f7bdbfff700 1 Processor -- start 2020-11-20T12:10:08.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.546+0000 7f7bdbfff700 1 -- start start 2020-11-20T12:10:08.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.547+0000 7f7bdbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc068230 0x7f7bdc139dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:08.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.547+0000 7f7bdbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bdc0b57b0 0x7f7bdc09bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:08.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.547+0000 7f7bdbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bdc0b5f50 0x7f7bdc09b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:08.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.547+0000 7f7bdbfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc13a690 con 0x7f7bdc0b57b0 2020-11-20T12:10:08.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.547+0000 7f7bdbfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc099f90 con 0x7f7bdc068230 2020-11-20T12:10:08.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.547+0000 7f7bdbfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc13a8c0 con 0x7f7bdc0b5f50 2020-11-20T12:10:08.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.547+0000 7f7bdaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc068230 0x7f7bdc139dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:08.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bdaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc068230 0x7f7bdc139dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45338/0 (socket says 172.21.15.187:45338) 2020-11-20T12:10:08.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bdaffd700 1 -- 172.21.15.187:0/2673192623 learned_addr learned my addr 172.21.15.187:0/2673192623 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:08.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bd27fc700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bdc0b57b0 0x7f7bdc09bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:08.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bdaffd700 1 -- 172.21.15.187:0/2673192623 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bdc0b5f50 msgr2=0x7f7bdc09b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:08.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bdaffd700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bdc0b5f50 0x7f7bdc09b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:08.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bdaffd700 1 -- 172.21.15.187:0/2673192623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bdc0b57b0 msgr2=0x7f7bdc09bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:08.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bdaffd700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bdc0b57b0 0x7f7bdc09bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:08.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bdaffd700 1 -- 172.21.15.187:0/2673192623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc09c1b0 con 0x7f7bdc068230 2020-11-20T12:10:08.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bdaffd700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc068230 0x7f7bdc139dd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f7bcc001e70 tx=0x7f7bcc007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:08.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.548+0000 7f7bd37fe700 1 -- 172.21.15.187:0/2673192623 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc007020 con 0x7f7bdc068230 2020-11-20T12:10:08.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.549+0000 7f7bd37fe700 1 -- 172.21.15.187:0/2673192623 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc007180 con 0x7f7bdc068230 2020-11-20T12:10:08.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.549+0000 7f7bd37fe700 1 -- 172.21.15.187:0/2673192623 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc00e560 con 0x7f7bdc068230 2020-11-20T12:10:08.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.549+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bdc09a230 con 0x7f7bdc068230 2020-11-20T12:10:08.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.549+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc0a0030 con 0x7f7bdc068230 2020-11-20T12:10:08.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.550+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bdc0a02b0 con 0x7f7bdc068230 2020-11-20T12:10:08.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.550+0000 7f7bd37fe700 1 -- 172.21.15.187:0/2673192623 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7bcc018020 con 0x7f7bdc068230 2020-11-20T12:10:08.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.551+0000 7f7bd37fe700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb8056d20 0x7f7bb80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:08.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.551+0000 7f7bd37fe700 1 -- 172.21.15.187:0/2673192623 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f7bcc06f970 con 0x7f7bdc068230 2020-11-20T12:10:08.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.551+0000 7f7bd27fc700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb8056d20 0x7f7bb80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:08.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.552+0000 7f7bd27fc700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb8056d20 0x7f7bb80591e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f7bc4006fd0 tx=0x7f7bc4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:08.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.555+0000 7f7bd37fe700 1 -- 172.21.15.187:0/2673192623 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7bcc044d10 con 0x7f7bdc068230 2020-11-20T12:10:08.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:08.741+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7bdc061a20 con 0x7f7bdc068230 2020-11-20T12:10:09.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.253+0000 7f7bd37fe700 1 -- 172.21.15.187:0/2673192623 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v63) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7bcc016020 con 0x7f7bdc068230 2020-11-20T12:10:09.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb8056d20 msgr2=0x7f7bb80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:09.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb8056d20 0x7f7bb80591e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f7bc4006fd0 tx=0x7f7bc4008040).stop 2020-11-20T12:10:09.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc068230 msgr2=0x7f7bdc139dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:09.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc068230 0x7f7bdc139dd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f7bcc001e70 tx=0x7f7bcc007c50).stop 2020-11-20T12:10:09.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 shutdown_connections 2020-11-20T12:10:09.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7bb8056d20 0x7f7bb80591e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:09.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7bdc0b5f50 0x7f7bdc09b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:09.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bdc068230 0x7f7bdc139dd0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:09.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 --2- 172.21.15.187:0/2673192623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7bdc0b57b0 0x7f7bdc09bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:09.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.256+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 >> 172.21.15.187:0/2673192623 conn(0x7f7bdc0bab30 msgr2=0x7f7bdc0afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:09.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.257+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 shutdown_connections 2020-11-20T12:10:09.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.257+0000 7f7bdbfff700 1 -- 172.21.15.187:0/2673192623 wait complete. 2020-11-20T12:10:09.266 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:10:09.289 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:10:09.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.542+0000 7eff7f1e5700 1 Processor -- start 2020-11-20T12:10:09.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.542+0000 7eff7f1e5700 1 -- start start 2020-11-20T12:10:09.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.543+0000 7eff7f1e5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7eff780679c0 0x7eff780b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.543+0000 7eff7f1e5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff78145da0 con 0x7eff780b73b0 2020-11-20T12:10:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.543+0000 7eff7f1e5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff78067520 con 0x7eff780b6fd0 2020-11-20T12:10:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.543+0000 7eff7f1e5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff78146010 con 0x7eff780679c0 2020-11-20T12:10:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.543+0000 7eff7cf81700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7eff780b6fd0 0x7eff78066700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41952/0 (socket says 172.21.15.187:41952) 2020-11-20T12:10:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.543+0000 7eff7cf81700 1 -- 172.21.15.187:0/4062594691 learned_addr learned my addr 172.21.15.187:0/4062594691 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff7d782700 1 --2- 172.21.15.187:0/4062594691 >> v2:172.21.15.5:3300/0 conn(0x7eff780679c0 0x7eff780b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238627548 0 0) 0x7eff78067520 con 0x7eff780b6fd0 2020-11-20T12:10:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c003240 con 0x7eff780b6fd0 2020-11-20T12:10:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2191910604 0 0) 0x7eff5c003240 con 0x7eff780b6fd0 2020-11-20T12:10:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 >> v2:172.21.15.5:3300/0 conn(0x7eff780679c0 msgr2=0x7eff780b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 --2- 172.21.15.187:0/4062594691 >> v2:172.21.15.5:3300/0 conn(0x7eff780679c0 0x7eff780b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 >> v1:172.21.15.5:6789/0 conn(0x7eff780b73b0 legacy=0x7eff78066e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff781462b0 con 0x7eff780b6fd0 2020-11-20T12:10:09.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7eff6c0014e0 con 0x7eff780b6fd0 2020-11-20T12:10:09.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.544+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 >> v1:172.21.15.187:6789/0 conn(0x7eff780b6fd0 legacy=0x7eff78066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:09.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.545+0000 7eff777fe700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c003730 0x7eff5c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:09.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.545+0000 7eff777fe700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff5c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:09.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.545+0000 7eff777fe700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff5c004e50 0x7eff5c00d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:09.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.545+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff78146010 con 0x7eff5c003730 2020-11-20T12:10:09.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.545+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c004260 con 0x7eff5c0044e0 2020-11-20T12:10:09.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.545+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00de30 con 0x7eff5c004e50 2020-11-20T12:10:09.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.545+0000 7eff77fff700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff5c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:09.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.545+0000 7eff7cf81700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c003730 0x7eff5c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:09.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.546+0000 7eff7d782700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff5c004e50 0x7eff5c00d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:09.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.546+0000 7eff77fff700 1 -- 172.21.15.187:0/4062594691 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff5c004e50 msgr2=0x7eff5c00d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:09.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.546+0000 7eff77fff700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff5c004e50 0x7eff5c00d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:09.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.546+0000 7eff77fff700 1 -- 172.21.15.187:0/4062594691 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c003730 msgr2=0x7eff5c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:09.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.546+0000 7eff77fff700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c003730 0x7eff5c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.546+0000 7eff7d782700 1 -- 172.21.15.187:0/4062594691 reap_dead start 2020-11-20T12:10:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.546+0000 7eff77fff700 1 -- 172.21.15.187:0/4062594691 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c003240 con 0x7eff5c0044e0 2020-11-20T12:10:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.546+0000 7eff77fff700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff5c004960 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7eff64001920 tx=0x7eff64007b90).ready entity=mon.1 client_cookie=259827f0ba6c46e1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.547+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64007020 con 0x7eff5c0044e0 2020-11-20T12:10:09.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.547+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff64007180 con 0x7eff5c0044e0 2020-11-20T12:10:09.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.547+0000 7eff777fe700 1 -- 172.21.15.187:0/4062594691 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6400c620 con 0x7eff5c0044e0 2020-11-20T12:10:09.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.548+0000 7eff7f1e5700 1 -- 172.21.15.187:0/4062594691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 msgr2=0x7eff5c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:09.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.548+0000 7eff7f1e5700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff5c004960 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7eff64001920 tx=0x7eff64007b90).stop 2020-11-20T12:10:09.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.548+0000 7eff7f1e5700 1 -- 172.21.15.187:0/4062594691 shutdown_connections 2020-11-20T12:10:09.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.548+0000 7eff7f1e5700 1 --2- 172.21.15.187:0/4062594691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff5c004960 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:09.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.548+0000 7eff7f1e5700 1 -- 172.21.15.187:0/4062594691 >> 172.21.15.187:0/4062594691 conn(0x7eff780bacf0 msgr2=0x7eff780bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:09.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.548+0000 7eff7f1e5700 1 -- 172.21.15.187:0/4062594691 shutdown_connections 2020-11-20T12:10:09.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.548+0000 7eff7f1e5700 1 -- 172.21.15.187:0/4062594691 wait complete. 2020-11-20T12:10:09.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.549+0000 7eff7f1e5700 1 Processor -- start 2020-11-20T12:10:09.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.549+0000 7eff7f1e5700 1 -- start start 2020-11-20T12:10:09.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.550+0000 7eff7f1e5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff7809f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:09.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.550+0000 7eff7f1e5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff780b3d60 0x7eff7809a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:09.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.550+0000 7eff7f1e5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff781433d0 0x7eff78143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:09.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.550+0000 7eff7f1e5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff780b4860 con 0x7eff780b3d60 2020-11-20T12:10:09.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.550+0000 7eff7f1e5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7809e580 con 0x7eff5c0044e0 2020-11-20T12:10:09.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.550+0000 7eff7f1e5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff780a0c20 con 0x7eff781433d0 2020-11-20T12:10:09.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff7809f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:09.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff7809f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45358/0 (socket says 172.21.15.187:45358) 2020-11-20T12:10:09.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 -- 172.21.15.187:0/2120161978 learned_addr learned my addr 172.21.15.187:0/2120161978 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:09.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 -- 172.21.15.187:0/2120161978 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff781433d0 msgr2=0x7eff78143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:09.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff781433d0 0x7eff78143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:09.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff77fff700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff780b3d60 0x7eff7809a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:09.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 -- 172.21.15.187:0/2120161978 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff780b3d60 msgr2=0x7eff7809a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:09.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff780b3d60 0x7eff7809a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:09.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 -- 172.21.15.187:0/2120161978 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff780a0980 con 0x7eff5c0044e0 2020-11-20T12:10:09.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.551+0000 7eff7cf81700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff7809f130 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7eff6c00eb40 tx=0x7eff6c000c80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:09.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.552+0000 7eff74ff9700 1 -- 172.21.15.187:0/2120161978 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00c900 con 0x7eff5c0044e0 2020-11-20T12:10:09.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.552+0000 7eff74ff9700 1 -- 172.21.15.187:0/2120161978 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c00ca60 con 0x7eff5c0044e0 2020-11-20T12:10:09.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.552+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff780b43c0 con 0x7eff5c0044e0 2020-11-20T12:10:09.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.552+0000 7eff74ff9700 1 -- 172.21.15.187:0/2120161978 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00cde0 con 0x7eff5c0044e0 2020-11-20T12:10:09.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.552+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff781575c0 con 0x7eff5c0044e0 2020-11-20T12:10:09.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.553+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff780b4550 con 0x7eff5c0044e0 2020-11-20T12:10:09.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.553+0000 7eff74ff9700 1 -- 172.21.15.187:0/2120161978 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7eff6c014640 con 0x7eff5c0044e0 2020-11-20T12:10:09.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.554+0000 7eff74ff9700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff50056d20 0x7eff500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:09.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.554+0000 7eff74ff9700 1 -- 172.21.15.187:0/2120161978 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5807+0+0 (secure 0 0 0) 0x7eff6c06fba0 con 0x7eff5c0044e0 2020-11-20T12:10:09.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.555+0000 7eff77fff700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff50056d20 0x7eff500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:09.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.555+0000 7eff77fff700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff50056d20 0x7eff500591e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7eff64000f90 tx=0x7eff64015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:09.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.558+0000 7eff74ff9700 1 -- 172.21.15.187:0/2120161978 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7eff6c045020 con 0x7eff5c0044e0 2020-11-20T12:10:09.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:09.748+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7eff78044f60 con 0x7eff5c0044e0 2020-11-20T12:10:10.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.265+0000 7eff74ff9700 1 -- 172.21.15.187:0/2120161978 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v64) v1 ==== 169+0+0 (secure 0 0 0) 0x7eff6c048640 con 0x7eff5c0044e0 2020-11-20T12:10:10.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff50056d20 msgr2=0x7eff500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:10.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff50056d20 0x7eff500591e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7eff64000f90 tx=0x7eff64015000).stop 2020-11-20T12:10:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 msgr2=0x7eff7809f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff7809f130 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7eff6c00eb40 tx=0x7eff6c000c80).stop 2020-11-20T12:10:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 shutdown_connections 2020-11-20T12:10:10.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff50056d20 0x7eff500591e0 secure :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0x7eff64000f90 tx=0x7eff64015000).stop 2020-11-20T12:10:10.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff781433d0 0x7eff78143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:10.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff5c0044e0 0x7eff7809f130 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:10.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 --2- 172.21.15.187:0/2120161978 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff780b3d60 0x7eff7809a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:10.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.268+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 >> 172.21.15.187:0/2120161978 conn(0x7eff780bacf0 msgr2=0x7eff780afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:10.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.269+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 shutdown_connections 2020-11-20T12:10:10.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:10.269+0000 7eff7f1e5700 1 -- 172.21.15.187:0/2120161978 wait complete. 2020-11-20T12:10:10.276 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:10:10.299 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:10:10.299 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:10:10.300 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:10:10.300 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:10:10.300 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:10:10.301 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:10:10.302 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:10:10.303 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:10:10.322 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:10:10.324 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:10:10.372 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:10:10.372 DEBUG:teuthology.parallel:result is None 2020-11-20T12:10:10.559 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:10:12.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2020-11-20T12:10:16.332 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:10:16.333 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:10:16.341 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:10:16.342 DEBUG:teuthology.parallel:result is None 2020-11-20T12:10:16.342 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:10:16.547 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:10:16.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.590+0000 7ffb931f1700 1 Processor -- start 2020-11-20T12:10:16.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.591+0000 7ffb931f1700 1 -- start start 2020-11-20T12:10:16.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.592+0000 7ffb931f1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffb8c060170 0x7ffb8c060550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:16.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.592+0000 7ffb931f1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 0x7ffb8c1470e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:16.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.592+0000 7ffb931f1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8c067460 con 0x7ffb8c066ee0 2020-11-20T12:10:16.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.592+0000 7ffb931f1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb8c0b2ac0 con 0x7ffb8c060170 2020-11-20T12:10:16.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.592+0000 7ffb931f1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c09e390 con 0x7ffb8c066260 2020-11-20T12:10:16.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.592+0000 7ffb921ef700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffb8c060170 0x7ffb8c060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:16.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.593+0000 7ffb921ef700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffb8c060170 0x7ffb8c060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45380/0 (socket says 172.21.15.187:45380) 2020-11-20T12:10:16.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.593+0000 7ffb921ef700 1 -- 172.21.15.187:0/3450531955 learned_addr learned my addr 172.21.15.187:0/3450531955 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:16.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.593+0000 7ffb919ee700 1 --2- 172.21.15.187:0/3450531955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 0x7ffb8c1470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:16.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.593+0000 7ffb921ef700 1 -- 172.21.15.187:0/3450531955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 msgr2=0x7ffb8c1470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:16.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.593+0000 7ffb921ef700 1 --2- 172.21.15.187:0/3450531955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 0x7ffb8c1470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:16.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.593+0000 7ffb921ef700 1 -- 172.21.15.187:0/3450531955 >> v1:172.21.15.5:6789/0 conn(0x7ffb8c066ee0 legacy=0x7ffb8c1438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:16.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.593+0000 7ffb921ef700 1 -- 172.21.15.187:0/3450531955 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c1469a0 con 0x7ffb8c060170 2020-11-20T12:10:16.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.594+0000 7ffb921ef700 1 --2- 172.21.15.187:0/3450531955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c060170 0x7ffb8c060550 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7ffb88009d50 tx=0x7ffb88009290).ready entity=mon.1 client_cookie=e7276cb57e0a0c35 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:16.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.594+0000 7ffb911ed700 1 -- 172.21.15.187:0/3450531955 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8800bc80 con 0x7ffb8c060170 2020-11-20T12:10:16.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.594+0000 7ffb911ed700 1 -- 172.21.15.187:0/3450531955 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8800bde0 con 0x7ffb8c060170 2020-11-20T12:10:16.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.594+0000 7ffb911ed700 1 -- 172.21.15.187:0/3450531955 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88011610 con 0x7ffb8c060170 2020-11-20T12:10:16.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.594+0000 7ffb931f1700 1 -- 172.21.15.187:0/3450531955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c060170 msgr2=0x7ffb8c060550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:16.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.594+0000 7ffb931f1700 1 --2- 172.21.15.187:0/3450531955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c060170 0x7ffb8c060550 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7ffb88009d50 tx=0x7ffb88009290).stop 2020-11-20T12:10:16.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.595+0000 7ffb931f1700 1 -- 172.21.15.187:0/3450531955 shutdown_connections 2020-11-20T12:10:16.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.595+0000 7ffb931f1700 1 --2- 172.21.15.187:0/3450531955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c060170 0x7ffb8c060550 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:16.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.595+0000 7ffb931f1700 1 --2- 172.21.15.187:0/3450531955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 0x7ffb8c1470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:16.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.595+0000 7ffb931f1700 1 -- 172.21.15.187:0/3450531955 >> 172.21.15.187:0/3450531955 conn(0x7ffb8c0bbc80 msgr2=0x7ffb8c0bc090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:16.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.595+0000 7ffb931f1700 1 -- 172.21.15.187:0/3450531955 shutdown_connections 2020-11-20T12:10:16.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.595+0000 7ffb931f1700 1 -- 172.21.15.187:0/3450531955 wait complete. 2020-11-20T12:10:16.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.596+0000 7ffb931f1700 1 Processor -- start 2020-11-20T12:10:16.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.597+0000 7ffb931f1700 1 -- start start 2020-11-20T12:10:16.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.597+0000 7ffb931f1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb8c060170 0x7ffb8c15c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:16.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.597+0000 7ffb931f1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 0x7ffb8c1448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:16.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.597+0000 7ffb931f1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c066ee0 0x7ffb8c093f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:16.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.597+0000 7ffb931f1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c15c6c0 con 0x7ffb8c060170 2020-11-20T12:10:16.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.597+0000 7ffb931f1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c093810 con 0x7ffb8c066ee0 2020-11-20T12:10:16.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.597+0000 7ffb931f1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c093ab0 con 0x7ffb8c066260 2020-11-20T12:10:16.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb929f0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c066ee0 0x7ffb8c093f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:16.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb929f0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c066ee0 0x7ffb8c093f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45388/0 (socket says 172.21.15.187:45388) 2020-11-20T12:10:16.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb929f0700 1 -- 172.21.15.187:0/623574178 learned_addr learned my addr 172.21.15.187:0/623574178 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:16.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb919ee700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 0x7ffb8c1448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:16.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb929f0700 1 -- 172.21.15.187:0/623574178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 msgr2=0x7ffb8c1448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:16.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb929f0700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 0x7ffb8c1448d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:16.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb929f0700 1 -- 172.21.15.187:0/623574178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb8c060170 msgr2=0x7ffb8c15c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:16.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb929f0700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb8c060170 0x7ffb8c15c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:16.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.598+0000 7ffb929f0700 1 -- 172.21.15.187:0/623574178 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c093500 con 0x7ffb8c066ee0 2020-11-20T12:10:16.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.599+0000 7ffb929f0700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c066ee0 0x7ffb8c093f50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7ffb8400c710 tx=0x7ffb84007bf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:16.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.599+0000 7ffb7a7fc700 1 -- 172.21.15.187:0/623574178 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8400e560 con 0x7ffb8c066ee0 2020-11-20T12:10:16.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.599+0000 7ffb7a7fc700 1 -- 172.21.15.187:0/623574178 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8400e6c0 con 0x7ffb8c066ee0 2020-11-20T12:10:16.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.599+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c144dc0 con 0x7ffb8c066ee0 2020-11-20T12:10:16.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.599+0000 7ffb7a7fc700 1 -- 172.21.15.187:0/623574178 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8400ea40 con 0x7ffb8c066ee0 2020-11-20T12:10:16.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.599+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c094680 con 0x7ffb8c066ee0 2020-11-20T12:10:16.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.600+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8c093c70 con 0x7ffb8c066ee0 2020-11-20T12:10:16.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.601+0000 7ffb7a7fc700 1 -- 172.21.15.187:0/623574178 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffb84019030 con 0x7ffb8c066ee0 2020-11-20T12:10:16.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.601+0000 7ffb7a7fc700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb64056d20 0x7ffb640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:16.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.601+0000 7ffb7a7fc700 1 -- 172.21.15.187:0/623574178 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5425+0+0 (secure 0 0 0) 0x7ffb8406f030 con 0x7ffb8c066ee0 2020-11-20T12:10:16.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.602+0000 7ffb921ef700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb64056d20 0x7ffb640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:16.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.602+0000 7ffb921ef700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb64056d20 0x7ffb640591e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7ffb88009500 tx=0x7ffb88016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:16.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.605+0000 7ffb7a7fc700 1 -- 172.21.15.187:0/623574178 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffb84043fd0 con 0x7ffb8c066ee0 2020-11-20T12:10:16.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:16.807+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ffb8c144f50 con 0x7ffb8c066ee0 2020-11-20T12:10:17.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.140+0000 7ffb7a7fc700 1 -- 172.21.15.187:0/623574178 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v65) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffb840447b0 con 0x7ffb8c066ee0 2020-11-20T12:10:17.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.143+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb64056d20 msgr2=0x7ffb640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:17.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.143+0000 7ffb931f1700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb64056d20 0x7ffb640591e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7ffb88009500 tx=0x7ffb88016000).stop 2020-11-20T12:10:17.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.143+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c066ee0 msgr2=0x7ffb8c093f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:17.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.143+0000 7ffb931f1700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c066ee0 0x7ffb8c093f50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7ffb8400c710 tx=0x7ffb84007bf0).stop 2020-11-20T12:10:17.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.144+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 shutdown_connections 2020-11-20T12:10:17.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.144+0000 7ffb931f1700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb64056d20 0x7ffb640591e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.144+0000 7ffb931f1700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb8c060170 0x7ffb8c15c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:17.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.144+0000 7ffb931f1700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb8c066ee0 0x7ffb8c093f50 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.144+0000 7ffb931f1700 1 --2- 172.21.15.187:0/623574178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb8c066260 0x7ffb8c1448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.144+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 >> 172.21.15.187:0/623574178 conn(0x7ffb8c0bbc80 msgr2=0x7ffb8c145a90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:17.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.144+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 shutdown_connections 2020-11-20T12:10:17.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.144+0000 7ffb931f1700 1 -- 172.21.15.187:0/623574178 wait complete. 2020-11-20T12:10:17.151 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:10:17.179 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:10:17.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.419+0000 7f382ffff700 1 Processor -- start 2020-11-20T12:10:17.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.419+0000 7f382ffff700 1 -- start start 2020-11-20T12:10:17.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.420+0000 7f382ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3830065190 0x7f3830065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:17.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.420+0000 7f382ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f3830143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:17.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.420+0000 7f382ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3830143c70 0x7f383009ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:17.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.420+0000 7f382ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383014ab30 con 0x7f3830065190 2020-11-20T12:10:17.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.420+0000 7f382ffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38300b13e0 con 0x7f3830143c70 2020-11-20T12:10:17.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.420+0000 7f382ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3830062c10 con 0x7f3830065f90 2020-11-20T12:10:17.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382f7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3830143c70 0x7f383009ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:17.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382f7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3830143c70 0x7f383009ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45398/0 (socket says 172.21.15.187:45398) 2020-11-20T12:10:17.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382f7fe700 1 -- 172.21.15.187:0/778720173 learned_addr learned my addr 172.21.15.187:0/778720173 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:17.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f38267fc700 1 --2- 172.21.15.187:0/778720173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f3830143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:17.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382effd700 1 --2- 172.21.15.187:0/778720173 >> v2:172.21.15.5:3300/0 conn(0x7f3830065190 0x7f3830065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:17.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382f7fe700 1 -- 172.21.15.187:0/778720173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 msgr2=0x7f3830143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:17.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382f7fe700 1 --2- 172.21.15.187:0/778720173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f3830143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382f7fe700 1 -- 172.21.15.187:0/778720173 >> v2:172.21.15.5:3300/0 conn(0x7f3830065190 msgr2=0x7f3830065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:17.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382f7fe700 1 --2- 172.21.15.187:0/778720173 >> v2:172.21.15.5:3300/0 conn(0x7f3830065190 0x7f3830065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.421+0000 7f382f7fe700 1 -- 172.21.15.187:0/778720173 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300b1640 con 0x7f3830143c70 2020-11-20T12:10:17.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.422+0000 7f382effd700 1 --2- 172.21.15.187:0/778720173 >> v2:172.21.15.5:3300/0 conn(0x7f3830065190 0x7f3830065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:17.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.422+0000 7f38267fc700 1 --2- 172.21.15.187:0/778720173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f3830143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:17.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.422+0000 7f382f7fe700 1 --2- 172.21.15.187:0/778720173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f383009ca90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f38300b6390 tx=0x7f3828009290).ready entity=mon.1 client_cookie=3a61e501f87cf7c3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:17.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.422+0000 7f382e7fc700 1 -- 172.21.15.187:0/778720173 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800bc50 con 0x7f3830143c70 2020-11-20T12:10:17.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.422+0000 7f382e7fc700 1 -- 172.21.15.187:0/778720173 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382800bdb0 con 0x7f3830143c70 2020-11-20T12:10:17.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.422+0000 7f382e7fc700 1 -- 172.21.15.187:0/778720173 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828011610 con 0x7f3830143c70 2020-11-20T12:10:17.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 -- 172.21.15.187:0/778720173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 msgr2=0x7f383009ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:17.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 --2- 172.21.15.187:0/778720173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f383009ca90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f38300b6390 tx=0x7f3828009290).stop 2020-11-20T12:10:17.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 -- 172.21.15.187:0/778720173 shutdown_connections 2020-11-20T12:10:17.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 --2- 172.21.15.187:0/778720173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f383009ca90 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 --2- 172.21.15.187:0/778720173 >> v2:172.21.15.5:3300/0 conn(0x7f3830065190 0x7f3830065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 --2- 172.21.15.187:0/778720173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f3830143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 -- 172.21.15.187:0/778720173 >> 172.21.15.187:0/778720173 conn(0x7f38300bab30 msgr2=0x7f38300baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:17.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 -- 172.21.15.187:0/778720173 shutdown_connections 2020-11-20T12:10:17.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.423+0000 7f382ffff700 1 -- 172.21.15.187:0/778720173 wait complete. 2020-11-20T12:10:17.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.425+0000 7f382ffff700 1 Processor -- start 2020-11-20T12:10:17.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.425+0000 7f382ffff700 1 -- start start 2020-11-20T12:10:17.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.426+0000 7f382ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3830065190 0x7f3830146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:17.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.426+0000 7f382ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f383009e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:17.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.426+0000 7f382ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f38300a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:17.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.426+0000 7f382ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383009e9a0 con 0x7f3830065190 2020-11-20T12:10:17.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.426+0000 7f382ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3830149430 con 0x7f3830143c70 2020-11-20T12:10:17.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.426+0000 7f382ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3830148c50 con 0x7f3830065f90 2020-11-20T12:10:17.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f382f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f38300a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:17.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f382f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f38300a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45404/0 (socket says 172.21.15.187:45404) 2020-11-20T12:10:17.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f382f7fe700 1 -- 172.21.15.187:0/1197645606 learned_addr learned my addr 172.21.15.187:0/1197645606 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:17.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f382effd700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3830065190 0x7f3830146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:17.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f38267fc700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f383009e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:17.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f382f7fe700 1 -- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 msgr2=0x7f383009e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:17.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f382f7fe700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f383009e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f382f7fe700 1 -- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3830065190 msgr2=0x7f3830146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:17.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.427+0000 7f382f7fe700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3830065190 0x7f3830146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:17.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.428+0000 7f382f7fe700 1 -- 172.21.15.187:0/1197645606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3830148ef0 con 0x7f3830143c70 2020-11-20T12:10:17.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.428+0000 7f382effd700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3830065190 0x7f3830146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:17.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.428+0000 7f382f7fe700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f38300a01c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f38280092d0 tx=0x7f3828016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:17.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.428+0000 7f38277fe700 1 -- 172.21.15.187:0/1197645606 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800bc50 con 0x7f3830143c70 2020-11-20T12:10:17.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.428+0000 7f38277fe700 1 -- 172.21.15.187:0/1197645606 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828010c40 con 0x7f3830143c70 2020-11-20T12:10:17.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.428+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3830149190 con 0x7f3830143c70 2020-11-20T12:10:17.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.428+0000 7f38277fe700 1 -- 172.21.15.187:0/1197645606 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382801c470 con 0x7f3830143c70 2020-11-20T12:10:17.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.428+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3830146ff0 con 0x7f3830143c70 2020-11-20T12:10:17.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.429+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3830061a20 con 0x7f3830143c70 2020-11-20T12:10:17.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.430+0000 7f38277fe700 1 -- 172.21.15.187:0/1197645606 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3828017070 con 0x7f3830143c70 2020-11-20T12:10:17.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.431+0000 7f38277fe700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f380c056d20 0x7f380c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:17.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.431+0000 7f38277fe700 1 -- 172.21.15.187:0/1197645606 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3828071e20 con 0x7f3830143c70 2020-11-20T12:10:17.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.431+0000 7f382effd700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f380c056d20 0x7f380c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:17.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.432+0000 7f382effd700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f380c056d20 0x7f380c0591e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f382000b870 tx=0x7f3820008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:17.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.437+0000 7f38277fe700 1 -- 172.21.15.187:0/1197645606 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3828047620 con 0x7f3830143c70 2020-11-20T12:10:17.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.686+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f38300afc60 con 0x7f3830143c70 2020-11-20T12:10:18.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.687+0000 7f38277fe700 1 -- 172.21.15.187:0/1197645606 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f3828015030 con 0x7f3830143c70 2020-11-20T12:10:18.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f380c056d20 msgr2=0x7f380c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f380c056d20 0x7f380c0591e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f382000b870 tx=0x7f3820008040).stop 2020-11-20T12:10:18.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 msgr2=0x7f38300a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f38300a01c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f38280092d0 tx=0x7f3828016000).stop 2020-11-20T12:10:18.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 shutdown_connections 2020-11-20T12:10:18.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f380c056d20 0x7f380c0591e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3830065190 0x7f3830146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830143c70 0x7f38300a01c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 --2- 172.21.15.187:0/1197645606 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3830065f90 0x7f383009e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.689+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 >> 172.21.15.187:0/1197645606 conn(0x7f38300bab30 msgr2=0x7f38300ba0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:18.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.690+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 shutdown_connections 2020-11-20T12:10:18.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:17.690+0000 7f382ffff700 1 -- 172.21.15.187:0/1197645606 wait complete. 2020-11-20T12:10:18.206 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:18.207 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:10:18.207 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:10:18.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.208 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:10:18.209 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:10:18.209 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.214 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:10:18.214 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.214 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.214 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.215 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.215 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:10:18.215 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:10:18.216 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.216 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:10:18.216 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.216 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.217 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.217 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:10:18.218 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:10:18.218 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.218 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.218 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.218 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.219 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.219 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:10:18.219 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:10:18.220 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.220 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.220 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.220 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.221 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.221 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:10:18.222 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:10:18.222 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.222 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.222 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.222 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.223 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.223 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:10:18.224 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:10:18.224 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.224 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.224 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.224 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.225 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.225 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:10:18.225 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:10:18.226 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.226 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.226 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.226 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.227 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.227 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:10:18.227 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:10:18.228 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.228 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.228 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.228 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.229 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.229 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:10:18.230 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:10:18.230 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.230 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.230 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.230 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.231 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.231 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:10:18.231 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:10:18.232 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.232 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.232 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.232 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.232 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.233 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:10:18.233 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:10:18.234 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.234 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:10:18.234 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:10:18.234 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:10:18.234 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.235 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:10:18.235 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:10:18.235 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.236 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:10:18.236 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:10:18.236 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:10:18.236 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:10:18.236 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.237 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:10:18.237 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:10:18.237 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.238 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:10:18.238 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:10:18.238 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:10:18.238 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:10:18.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.240 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:10:18.241 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:10:18.241 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.241 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:10:18.241 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:10:18.242 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:10:18.242 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:10:18.242 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.243 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:10:18.243 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:10:18.244 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.244 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:10:18.244 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:10:18.245 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:10:18.245 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.245 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.246 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:10:18.247 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:10:18.247 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.247 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:10:18.248 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:10:18.248 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.251 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:18.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:18.251 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:10:18.252 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:10:18.252 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:10:18.252 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:10:18.252 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:10:18.253 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:10:18.253 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.253 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:18.253 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:10:18.253 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:10:18.254 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:10:18.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.502+0000 7f659af5a700 1 Processor -- start 2020-11-20T12:10:18.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.502+0000 7f659af5a700 1 -- start start 2020-11-20T12:10:18.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.504+0000 7f659af5a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f65941484d0 0x7f6594061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.504+0000 7f659af5a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594066f70 0x7f65940b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.504+0000 7f659af5a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65940b1300 con 0x7f65940622c0 2020-11-20T12:10:18.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.504+0000 7f659af5a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6594067730 con 0x7f65941484d0 2020-11-20T12:10:18.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.504+0000 7f659af5a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65940b15a0 con 0x7f6594066f70 2020-11-20T12:10:18.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.504+0000 7f6599757700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f65940622c0 0x7f65940b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42014/0 (socket says 172.21.15.187:42014) 2020-11-20T12:10:18.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.504+0000 7f6599757700 1 -- 172.21.15.187:0/151096667 learned_addr learned my addr 172.21.15.187:0/151096667 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:18.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.505+0000 7f659a759700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594066f70 0x7f65940b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.505+0000 7f6599f58700 1 --2- 172.21.15.187:0/151096667 >> v2:172.21.15.5:3300/0 conn(0x7f65941484d0 0x7f6594061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.505+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118597588 0 0) 0x7f65940b1300 con 0x7f65940622c0 2020-11-20T12:10:18.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.505+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f657c003240 con 0x7f65940622c0 2020-11-20T12:10:18.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 592257738 0 0) 0x7f657c003240 con 0x7f65940622c0 2020-11-20T12:10:18.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594066f70 msgr2=0x7f65940b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:18.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594066f70 0x7f65940b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 >> v2:172.21.15.5:3300/0 conn(0x7f65941484d0 msgr2=0x7f6594061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:18.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 --2- 172.21.15.187:0/151096667 >> v2:172.21.15.5:3300/0 conn(0x7f65941484d0 0x7f6594061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65940b1840 con 0x7f65940622c0 2020-11-20T12:10:18.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f65840014e0 con 0x7f65940622c0 2020-11-20T12:10:18.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 >> v1:172.21.15.187:6789/0 conn(0x7f65940622c0 legacy=0x7f65940b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657c003730 0x7f657c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594066f70 0x7f657c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657c0049d0 0x7f657c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6594067730 con 0x7f657c0049d0 2020-11-20T12:10:18.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940b15a0 con 0x7f657c003730 2020-11-20T12:10:18.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0041d0 con 0x7f6594066f70 2020-11-20T12:10:18.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.506+0000 7f6599f58700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657c003730 0x7f657c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.507+0000 7f6599757700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594066f70 0x7f657c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.507+0000 7f659a759700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657c0049d0 0x7f657c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.507+0000 7f6599f58700 1 -- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594066f70 msgr2=0x7f657c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:18.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.507+0000 7f6599f58700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594066f70 0x7f657c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.507+0000 7f6599f58700 1 -- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657c0049d0 msgr2=0x7f657c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:18.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.507+0000 7f6599f58700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657c0049d0 0x7f657c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.507+0000 7f6599f58700 1 -- 172.21.15.187:0/151096667 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c003240 con 0x7f657c003730 2020-11-20T12:10:18.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.507+0000 7f6599f58700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657c003730 0x7f657c003b50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f659000bc00 tx=0x7f6590006df0).ready entity=mon.1 client_cookie=60b9eacd98e6b35d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:18.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.508+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659000b310 con 0x7f657c003730 2020-11-20T12:10:18.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.508+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659000eb40 con 0x7f657c003730 2020-11-20T12:10:18.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.508+0000 7f6598f56700 1 -- 172.21.15.187:0/151096667 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659000eec0 con 0x7f657c003730 2020-11-20T12:10:18.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.508+0000 7f659af5a700 1 -- 172.21.15.187:0/151096667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657c003730 msgr2=0x7f657c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.508+0000 7f659af5a700 1 --2- 172.21.15.187:0/151096667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657c003730 0x7f657c003b50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f659000bc00 tx=0x7f6590006df0).stop 2020-11-20T12:10:18.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.508+0000 7f659a759700 1 -- 172.21.15.187:0/151096667 reap_dead start 2020-11-20T12:10:18.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.509+0000 7f659af5a700 1 -- 172.21.15.187:0/151096667 shutdown_connections 2020-11-20T12:10:18.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.509+0000 7f659af5a700 1 -- 172.21.15.187:0/151096667 >> 172.21.15.187:0/151096667 conn(0x7f65940bab30 msgr2=0x7f65940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:18.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.509+0000 7f659af5a700 1 -- 172.21.15.187:0/151096667 shutdown_connections 2020-11-20T12:10:18.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.509+0000 7f659af5a700 1 -- 172.21.15.187:0/151096667 wait complete. 2020-11-20T12:10:18.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.510+0000 7f659af5a700 1 Processor -- start 2020-11-20T12:10:18.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.510+0000 7f659af5a700 1 -- start start 2020-11-20T12:10:18.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.511+0000 7f659af5a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6594158340 0x7f65940b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.511+0000 7f659af5a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f659409eab0 0x7f6594158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.511+0000 7f659af5a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594158c10 0x7f65940990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.511+0000 7f659af5a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940b38a0 con 0x7f6594158340 2020-11-20T12:10:18.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.511+0000 7f659af5a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659409ef30 con 0x7f659409eab0 2020-11-20T12:10:18.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.511+0000 7f659af5a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65940b3510 con 0x7f6594158c10 2020-11-20T12:10:18.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.511+0000 7f6599757700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f659409eab0 0x7f6594158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599757700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f659409eab0 0x7f6594158720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45422/0 (socket says 172.21.15.187:45422) 2020-11-20T12:10:18.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599757700 1 -- 172.21.15.187:0/370776229 learned_addr learned my addr 172.21.15.187:0/370776229 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:18.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599757700 1 -- 172.21.15.187:0/370776229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594158c10 msgr2=0x7f65940990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599f58700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6594158340 0x7f65940b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599757700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594158c10 0x7f65940990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:18.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599757700 1 -- 172.21.15.187:0/370776229 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6594158340 msgr2=0x7f65940b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599757700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6594158340 0x7f65940b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599757700 1 -- 172.21.15.187:0/370776229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65940b3ad0 con 0x7f659409eab0 2020-11-20T12:10:18.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6599757700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f659409eab0 0x7f6594158720 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f658400eb40 tx=0x7f65840014b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:18.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.512+0000 7f6581ffb700 1 -- 172.21.15.187:0/370776229 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658400ca30 con 0x7f659409eab0 2020-11-20T12:10:18.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.513+0000 7f6581ffb700 1 -- 172.21.15.187:0/370776229 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658400cb90 con 0x7f659409eab0 2020-11-20T12:10:18.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.513+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659409d070 con 0x7f659409eab0 2020-11-20T12:10:18.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.513+0000 7f6581ffb700 1 -- 172.21.15.187:0/370776229 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584014430 con 0x7f659409eab0 2020-11-20T12:10:18.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.513+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65940998a0 con 0x7f659409eab0 2020-11-20T12:10:18.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.514+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65940b3650 con 0x7f659409eab0 2020-11-20T12:10:18.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.514+0000 7f6581ffb700 1 -- 172.21.15.187:0/370776229 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f65840147e0 con 0x7f659409eab0 2020-11-20T12:10:18.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.515+0000 7f6581ffb700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f656c056d20 0x7f656c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.515+0000 7f6581ffb700 1 -- 172.21.15.187:0/370776229 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f658406fae0 con 0x7f659409eab0 2020-11-20T12:10:18.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.515+0000 7f6599f58700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f656c056d20 0x7f656c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.516+0000 7f6599f58700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f656c056d20 0x7f656c0591e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f6590008000 tx=0x7f6590019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:18.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.519+0000 7f6581ffb700 1 -- 172.21.15.187:0/370776229 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6584045200 con 0x7f659409eab0 2020-11-20T12:10:18.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.705+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6594061a20 con 0x7f659409eab0 2020-11-20T12:10:18.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.706+0000 7f6581ffb700 1 -- 172.21.15.187:0/370776229 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+13628 (secure 0 0 0) 0x7f6584048820 con 0x7f659409eab0 2020-11-20T12:10:18.711 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:18.711 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":65,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:10:17.129407+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:10:18.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.709+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f656c056d20 msgr2=0x7f656c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.709+0000 7f659af5a700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f656c056d20 0x7f656c0591e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f6590008000 tx=0x7f6590019000).stop 2020-11-20T12:10:18.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.709+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f659409eab0 msgr2=0x7f6594158720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.709+0000 7f659af5a700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f659409eab0 0x7f6594158720 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f658400eb40 tx=0x7f65840014b0).stop 2020-11-20T12:10:18.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.709+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 shutdown_connections 2020-11-20T12:10:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.709+0000 7f659af5a700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f656c056d20 0x7f656c0591e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.709+0000 7f659af5a700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6594158340 0x7f65940b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.710+0000 7f659af5a700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6594158c10 0x7f65940990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.710+0000 7f659af5a700 1 --2- 172.21.15.187:0/370776229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f659409eab0 0x7f6594158720 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.710+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 >> 172.21.15.187:0/370776229 conn(0x7f65940bab30 msgr2=0x7f65940b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:18.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.710+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 shutdown_connections 2020-11-20T12:10:18.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.710+0000 7f659af5a700 1 -- 172.21.15.187:0/370776229 wait complete. 2020-11-20T12:10:18.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:10:18.740 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:10:18.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.981+0000 7f25b6323700 1 Processor -- start 2020-11-20T12:10:18.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.981+0000 7f25b6323700 1 -- start start 2020-11-20T12:10:18.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.982+0000 7f25b6323700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25b0148230 0x7f25b00b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b6323700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 0x7f25b00b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:18.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b6323700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b009a230 con 0x7f25b00afa20 2020-11-20T12:10:18.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b6323700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b00684d0 con 0x7f25b0148230 2020-11-20T12:10:18.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b6323700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00b13a0 con 0x7f25b00aff30 2020-11-20T12:10:18.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b5321700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25b0148230 0x7f25b00b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b5321700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25b0148230 0x7f25b00b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45430/0 (socket says 172.21.15.187:45430) 2020-11-20T12:10:18.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b5321700 1 -- 172.21.15.187:0/1331446530 learned_addr learned my addr 172.21.15.187:0/1331446530 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:18.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b5b22700 1 --2- 172.21.15.187:0/1331446530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 0x7f25b00b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:18.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.983+0000 7f25b5321700 1 -- 172.21.15.187:0/1331446530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 msgr2=0x7f25b00b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:18.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.984+0000 7f25b5321700 1 --2- 172.21.15.187:0/1331446530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 0x7f25b00b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.984+0000 7f25b5321700 1 -- 172.21.15.187:0/1331446530 >> v1:172.21.15.5:6789/0 conn(0x7f25b00afa20 legacy=0x7f25b00650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:18.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.984+0000 7f25b5321700 1 -- 172.21.15.187:0/1331446530 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00657d0 con 0x7f25b0148230 2020-11-20T12:10:18.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.984+0000 7f25b5b22700 1 --2- 172.21.15.187:0/1331446530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 0x7f25b00b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:18.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.984+0000 7f25b5321700 1 --2- 172.21.15.187:0/1331446530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b0148230 0x7f25b00b50c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f25ac009d40 tx=0x7f25ac009290).ready entity=mon.1 client_cookie=adfdcbcfaaaad824 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:18.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.984+0000 7f259ffff700 1 -- 172.21.15.187:0/1331446530 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac00bcd0 con 0x7f25b0148230 2020-11-20T12:10:18.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.984+0000 7f259ffff700 1 -- 172.21.15.187:0/1331446530 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac00be30 con 0x7f25b0148230 2020-11-20T12:10:18.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.984+0000 7f259ffff700 1 -- 172.21.15.187:0/1331446530 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac011610 con 0x7f25b0148230 2020-11-20T12:10:18.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.985+0000 7f25b6323700 1 -- 172.21.15.187:0/1331446530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b0148230 msgr2=0x7f25b00b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:18.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.985+0000 7f25b6323700 1 --2- 172.21.15.187:0/1331446530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b0148230 0x7f25b00b50c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f25ac009d40 tx=0x7f25ac009290).stop 2020-11-20T12:10:18.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.985+0000 7f25b6323700 1 -- 172.21.15.187:0/1331446530 shutdown_connections 2020-11-20T12:10:18.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.985+0000 7f25b6323700 1 --2- 172.21.15.187:0/1331446530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 0x7f25b00b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.985+0000 7f25b6323700 1 --2- 172.21.15.187:0/1331446530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b0148230 0x7f25b00b50c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:18.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.985+0000 7f25b6323700 1 -- 172.21.15.187:0/1331446530 >> 172.21.15.187:0/1331446530 conn(0x7f25b00baac0 msgr2=0x7f25b00baed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:19.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.985+0000 7f25b6323700 1 -- 172.21.15.187:0/1331446530 shutdown_connections 2020-11-20T12:10:19.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.985+0000 7f25b6323700 1 -- 172.21.15.187:0/1331446530 wait complete. 2020-11-20T12:10:19.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.987+0000 7f25b6323700 1 Processor -- start 2020-11-20T12:10:19.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.987+0000 7f25b6323700 1 -- start start 2020-11-20T12:10:19.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.988+0000 7f25b6323700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b00afa20 0x7f25b01488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.988+0000 7f25b6323700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 0x7f25b014d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.988+0000 7f25b6323700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25b0148230 0x7f25b00b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.988+0000 7f25b6323700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00b5710 con 0x7f25b0148230 2020-11-20T12:10:19.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.988+0000 7f25b6323700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00b37e0 con 0x7f25b00afa20 2020-11-20T12:10:19.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.988+0000 7f25b6323700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00b3a80 con 0x7f25b00aff30 2020-11-20T12:10:19.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5321700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b00afa20 0x7f25b01488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5321700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b00afa20 0x7f25b01488e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45434/0 (socket says 172.21.15.187:45434) 2020-11-20T12:10:19.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5321700 1 -- 172.21.15.187:0/2906652340 learned_addr learned my addr 172.21.15.187:0/2906652340 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:19.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5b22700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25b0148230 0x7f25b00b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5321700 1 -- 172.21.15.187:0/2906652340 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 msgr2=0x7f25b014d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5321700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 0x7f25b014d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:19.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5321700 1 -- 172.21.15.187:0/2906652340 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25b0148230 msgr2=0x7f25b00b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5321700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25b0148230 0x7f25b00b5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.989+0000 7f25b5321700 1 -- 172.21.15.187:0/2906652340 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00b72f0 con 0x7f25b00afa20 2020-11-20T12:10:19.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.990+0000 7f25b5321700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b00afa20 0x7f25b01488e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f25b0064160 tx=0x7f25ac016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:19.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.990+0000 7f259d7fa700 1 -- 172.21.15.187:0/2906652340 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac020070 con 0x7f25b00afa20 2020-11-20T12:10:19.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.990+0000 7f259d7fa700 1 -- 172.21.15.187:0/2906652340 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac010780 con 0x7f25b00afa20 2020-11-20T12:10:19.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.990+0000 7f259d7fa700 1 -- 172.21.15.187:0/2906652340 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac010b40 con 0x7f25b00afa20 2020-11-20T12:10:19.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.990+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b00b3460 con 0x7f25b00afa20 2020-11-20T12:10:19.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.990+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b014d5b0 con 0x7f25b00afa20 2020-11-20T12:10:19.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.991+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b00b3620 con 0x7f25b00afa20 2020-11-20T12:10:19.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.992+0000 7f259d7fa700 1 -- 172.21.15.187:0/2906652340 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f25ac017070 con 0x7f25b00afa20 2020-11-20T12:10:19.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.993+0000 7f259d7fa700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2588056d20 0x7f25880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.993+0000 7f259d7fa700 1 -- 172.21.15.187:0/2906652340 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f25ac071bb0 con 0x7f25b00afa20 2020-11-20T12:10:19.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.993+0000 7f25b4b20700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2588056d20 0x7f25880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.994+0000 7f25b4b20700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2588056d20 0x7f25880591e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f25b00b4940 tx=0x7f25a00077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:19.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:18.996+0000 7f259d7fa700 1 -- 172.21.15.187:0/2906652340 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f25ac047330 con 0x7f25b00afa20 2020-11-20T12:10:19.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2020-11-20T12:10:19.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.184+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f25b00b5880 con 0x7f25b00afa20 2020-11-20T12:10:19.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.184+0000 7f259d7fa700 1 -- 172.21.15.187:0/2906652340 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v65) v1 ==== 110+0+10 (secure 0 0 0) 0x7f25ac015070 con 0x7f25b00afa20 2020-11-20T12:10:19.192 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:10:19.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2588056d20 msgr2=0x7f25880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2588056d20 0x7f25880591e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f25b00b4940 tx=0x7f25a00077f0).stop 2020-11-20T12:10:19.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b00afa20 msgr2=0x7f25b01488e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b00afa20 0x7f25b01488e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f25b0064160 tx=0x7f25ac016000).stop 2020-11-20T12:10:19.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 shutdown_connections 2020-11-20T12:10:19.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2588056d20 0x7f25880591e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25b0148230 0x7f25b00b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25b00afa20 0x7f25b01488e0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 --2- 172.21.15.187:0/2906652340 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25b00aff30 0x7f25b014d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:19.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 >> 172.21.15.187:0/2906652340 conn(0x7f25b00baac0 msgr2=0x7f25b0143ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:19.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 shutdown_connections 2020-11-20T12:10:19.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.187+0000 7f25b6323700 1 -- 172.21.15.187:0/2906652340 wait complete. 2020-11-20T12:10:19.213 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:10:19.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.453+0000 7f66615a4700 1 Processor -- start 2020-11-20T12:10:19.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.453+0000 7f66615a4700 1 -- start start 2020-11-20T12:10:19.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.454+0000 7f66615a4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f665c061f90 0x7f665c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.454+0000 7f66615a4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f665c0b67f0 0x7f665c099ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.454+0000 7f66615a4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c066790 con 0x7f665c062370 2020-11-20T12:10:19.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.454+0000 7f66615a4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c0b56b0 con 0x7f665c061f90 2020-11-20T12:10:19.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.454+0000 7f66615a4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c0b59a0 con 0x7f665c0b67f0 2020-11-20T12:10:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.455+0000 7f6660da3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f665c0b67f0 0x7f665c099ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.455+0000 7f66537fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f665c062370 0x7f665c0b60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42048/0 (socket says 172.21.15.187:42048) 2020-11-20T12:10:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.455+0000 7f66537fe700 1 -- 172.21.15.187:0/2202695504 learned_addr learned my addr 172.21.15.187:0/2202695504 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.455+0000 7f665bfff700 1 --2- 172.21.15.187:0/2202695504 >> v2:172.21.15.5:3300/0 conn(0x7f665c061f90 0x7f665c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.455+0000 7f665b7fe700 1 -- 172.21.15.187:0/2202695504 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313437388 0 0) 0x7f665c066790 con 0x7f665c062370 2020-11-20T12:10:19.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.613+0000 7f665b7fe700 1 -- 172.21.15.187:0/2202695504 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6648003240 con 0x7f665c062370 2020-11-20T12:10:19.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.614+0000 7f6660da3700 1 -- 172.21.15.187:0/2202695504 >> v2:172.21.15.5:3300/0 conn(0x7f665c061f90 msgr2=0x7f665c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:19.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.614+0000 7f6660da3700 1 --2- 172.21.15.187:0/2202695504 >> v2:172.21.15.5:3300/0 conn(0x7f665c061f90 0x7f665c0afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.614+0000 7f6660da3700 1 -- 172.21.15.187:0/2202695504 >> v1:172.21.15.187:6789/0 conn(0x7f665c062370 legacy=0x7f665c0b60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.614+0000 7f6660da3700 1 -- 172.21.15.187:0/2202695504 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c0b5c40 con 0x7f665c0b67f0 2020-11-20T12:10:19.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.614+0000 7f6660da3700 1 --2- 172.21.15.187:0/2202695504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c0b67f0 0x7f665c099ad0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f665c067c80 tx=0x7f6654006df0).ready entity=mon.1 client_cookie=749a70c1bc599dee server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:19.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.615+0000 7f665b7fe700 1 -- 172.21.15.187:0/2202695504 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665400b660 con 0x7f665c0b67f0 2020-11-20T12:10:19.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.615+0000 7f665b7fe700 1 -- 172.21.15.187:0/2202695504 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665400b7c0 con 0x7f665c0b67f0 2020-11-20T12:10:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.615+0000 7f665b7fe700 1 -- 172.21.15.187:0/2202695504 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665400bb40 con 0x7f665c0b67f0 2020-11-20T12:10:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.616+0000 7f66615a4700 1 -- 172.21.15.187:0/2202695504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c0b67f0 msgr2=0x7f665c099ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.616+0000 7f66615a4700 1 --2- 172.21.15.187:0/2202695504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c0b67f0 0x7f665c099ad0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f665c067c80 tx=0x7f6654006df0).stop 2020-11-20T12:10:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.616+0000 7f66615a4700 1 -- 172.21.15.187:0/2202695504 shutdown_connections 2020-11-20T12:10:19.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.616+0000 7f66615a4700 1 --2- 172.21.15.187:0/2202695504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c0b67f0 0x7f665c099ad0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.616+0000 7f66615a4700 1 --2- 172.21.15.187:0/2202695504 >> v2:172.21.15.5:3300/0 conn(0x7f665c061f90 0x7f665c0afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.616+0000 7f66615a4700 1 -- 172.21.15.187:0/2202695504 >> 172.21.15.187:0/2202695504 conn(0x7f665c0bab30 msgr2=0x7f665c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:19.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.616+0000 7f66615a4700 1 -- 172.21.15.187:0/2202695504 shutdown_connections 2020-11-20T12:10:19.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.616+0000 7f66615a4700 1 -- 172.21.15.187:0/2202695504 wait complete. 2020-11-20T12:10:19.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.618+0000 7f66615a4700 1 Processor -- start 2020-11-20T12:10:19.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.618+0000 7f66615a4700 1 -- start start 2020-11-20T12:10:19.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.619+0000 7f66615a4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f665c061f90 0x7f665c0939d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.619+0000 7f66615a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c062370 0x7f665c0b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.619+0000 7f66615a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f665c0b67f0 0x7f665c097f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.619+0000 7f66615a4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c093ec0 con 0x7f665c061f90 2020-11-20T12:10:19.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.619+0000 7f66615a4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c092ff0 con 0x7f665c062370 2020-11-20T12:10:19.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.619+0000 7f66615a4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c093290 con 0x7f665c0b67f0 2020-11-20T12:10:19.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.620+0000 7f66537fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c062370 0x7f665c0b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.620+0000 7f665bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f665c061f90 0x7f665c0939d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.620+0000 7f665bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f665c061f90 0x7f665c0939d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:37710/0 (socket says 172.21.15.187:37710) 2020-11-20T12:10:19.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.620+0000 7f66537fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c062370 0x7f665c0b3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45450/0 (socket says 172.21.15.187:45450) 2020-11-20T12:10:19.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.620+0000 7f665bfff700 1 -- 172.21.15.187:0/3577377317 learned_addr learned my addr 172.21.15.187:0/3577377317 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:19.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.620+0000 7f6660da3700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f665c0b67f0 0x7f665c097f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.621+0000 7f66537fe700 1 -- 172.21.15.187:0/3577377317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f665c0b67f0 msgr2=0x7f665c097f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.621+0000 7f66537fe700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f665c0b67f0 0x7f665c097f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.621+0000 7f66537fe700 1 -- 172.21.15.187:0/3577377317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f665c061f90 msgr2=0x7f665c0939d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.621+0000 7f66537fe700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f665c061f90 0x7f665c0939d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.621+0000 7f66537fe700 1 -- 172.21.15.187:0/3577377317 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c093530 con 0x7f665c062370 2020-11-20T12:10:19.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.622+0000 7f66537fe700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c062370 0x7f665c0b3c00 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f664400c6e0 tx=0x7f66440079e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:19.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.623+0000 7f6658ff9700 1 -- 172.21.15.187:0/3577377317 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66440072e0 con 0x7f665c062370 2020-11-20T12:10:19.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.623+0000 7f6658ff9700 1 -- 172.21.15.187:0/3577377317 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6644007460 con 0x7f665c062370 2020-11-20T12:10:19.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.623+0000 7f6658ff9700 1 -- 172.21.15.187:0/3577377317 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6644005840 con 0x7f665c062370 2020-11-20T12:10:19.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.623+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c0962a0 con 0x7f665c062370 2020-11-20T12:10:19.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.623+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c0b4330 con 0x7f665c062370 2020-11-20T12:10:19.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.624+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c0936c0 con 0x7f665c062370 2020-11-20T12:10:19.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.627+0000 7f6658ff9700 1 -- 172.21.15.187:0/3577377317 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6644010070 con 0x7f665c062370 2020-11-20T12:10:19.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.628+0000 7f6658ff9700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6638056d20 0x7f66380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:19.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.628+0000 7f6658ff9700 1 -- 172.21.15.187:0/3577377317 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f664406f190 con 0x7f665c062370 2020-11-20T12:10:19.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.629+0000 7f665bfff700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6638056d20 0x7f66380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:19.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.630+0000 7f665bfff700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6638056d20 0x7f66380591e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f664c0072c0 tx=0x7f664c005e90).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:19.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.633+0000 7f6658ff9700 1 -- 172.21.15.187:0/3577377317 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6644044800 con 0x7f665c062370 2020-11-20T12:10:19.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.823+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f665c061a20 con 0x7f665c062370 2020-11-20T12:10:19.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.824+0000 7f6658ff9700 1 -- 172.21.15.187:0/3577377317 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v65) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6644007b20 con 0x7f665c062370 2020-11-20T12:10:19.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.826+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6638056d20 msgr2=0x7f66380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.826+0000 7f66615a4700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6638056d20 0x7f66380591e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f664c0072c0 tx=0x7f664c005e90).stop 2020-11-20T12:10:19.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.826+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c062370 msgr2=0x7f665c0b3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:19.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.826+0000 7f66615a4700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c062370 0x7f665c0b3c00 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f664400c6e0 tx=0x7f66440079e0).stop 2020-11-20T12:10:19.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.827+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 shutdown_connections 2020-11-20T12:10:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.827+0000 7f66615a4700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6638056d20 0x7f66380591e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.827+0000 7f66615a4700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f665c061f90 0x7f665c0939d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.827+0000 7f66615a4700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f665c0b67f0 0x7f665c097f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.827+0000 7f66615a4700 1 --2- 172.21.15.187:0/3577377317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f665c062370 0x7f665c0b3c00 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.827+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 >> 172.21.15.187:0/3577377317 conn(0x7f665c0bab30 msgr2=0x7f665c09d000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:19.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.827+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 shutdown_connections 2020-11-20T12:10:19.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:19.827+0000 7f66615a4700 1 -- 172.21.15.187:0/3577377317 wait complete. 2020-11-20T12:10:19.843 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:19.857 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:10:19.905 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:10:19.920 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:10:20.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.084+0000 7fbadbfff700 1 Processor -- start 2020-11-20T12:10:20.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.084+0000 7fbadbfff700 1 -- start start 2020-11-20T12:10:20.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.085+0000 7fbadbfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbadc060710 0x7fbadc066fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.085+0000 7fbadbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbadc146200 0x7fbadc067660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.085+0000 7fbadbfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc148e40 con 0x7fbadc060330 2020-11-20T12:10:20.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.085+0000 7fbadbfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbadc1490e0 con 0x7fbadc060710 2020-11-20T12:10:20.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.085+0000 7fbadbfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbadc149380 con 0x7fbadc146200 2020-11-20T12:10:20.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.086+0000 7fbadb7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbadc146200 0x7fbadc067660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.086+0000 7fbadb7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbadc146200 0x7fbadc067660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45458/0 (socket says 172.21.15.187:45458) 2020-11-20T12:10:20.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.086+0000 7fbadb7fe700 1 -- 172.21.15.187:0/4026455826 learned_addr learned my addr 172.21.15.187:0/4026455826 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:20.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.086+0000 7fbada7fc700 1 --2- 172.21.15.187:0/4026455826 >> v2:172.21.15.5:3300/0 conn(0x7fbadc060710 0x7fbadc066fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.086+0000 7fbadb7fe700 1 -- 172.21.15.187:0/4026455826 >> v2:172.21.15.5:3300/0 conn(0x7fbadc060710 msgr2=0x7fbadc066fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:20.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.087+0000 7fbadb7fe700 1 --2- 172.21.15.187:0/4026455826 >> v2:172.21.15.5:3300/0 conn(0x7fbadc060710 0x7fbadc066fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.087+0000 7fbadb7fe700 1 -- 172.21.15.187:0/4026455826 >> v1:172.21.15.5:6789/0 conn(0x7fbadc060330 legacy=0x7fbadc145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:20.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.087+0000 7fbadb7fe700 1 -- 172.21.15.187:0/4026455826 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc066980 con 0x7fbadc146200 2020-11-20T12:10:20.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.087+0000 7fbada7fc700 1 --2- 172.21.15.187:0/4026455826 >> v2:172.21.15.5:3300/0 conn(0x7fbadc060710 0x7fbadc066fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:20.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.087+0000 7fbadb7fe700 1 --2- 172.21.15.187:0/4026455826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc067660 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fbadc0b1430 tx=0x7fbad4009290).ready entity=mon.1 client_cookie=4edc9472abbad816 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:20.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.087+0000 7fbad9ffb700 1 -- 172.21.15.187:0/4026455826 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad400bb80 con 0x7fbadc146200 2020-11-20T12:10:20.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.087+0000 7fbad9ffb700 1 -- 172.21.15.187:0/4026455826 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad400bce0 con 0x7fbadc146200 2020-11-20T12:10:20.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.087+0000 7fbad9ffb700 1 -- 172.21.15.187:0/4026455826 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad4011610 con 0x7fbadc146200 2020-11-20T12:10:20.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.088+0000 7fbadbfff700 1 -- 172.21.15.187:0/4026455826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 msgr2=0x7fbadc067660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.088+0000 7fbadbfff700 1 --2- 172.21.15.187:0/4026455826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc067660 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fbadc0b1430 tx=0x7fbad4009290).stop 2020-11-20T12:10:20.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.088+0000 7fbadbfff700 1 -- 172.21.15.187:0/4026455826 shutdown_connections 2020-11-20T12:10:20.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.088+0000 7fbadbfff700 1 --2- 172.21.15.187:0/4026455826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc067660 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.088+0000 7fbadbfff700 1 --2- 172.21.15.187:0/4026455826 >> v2:172.21.15.5:3300/0 conn(0x7fbadc060710 0x7fbadc066fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.088+0000 7fbadbfff700 1 -- 172.21.15.187:0/4026455826 >> 172.21.15.187:0/4026455826 conn(0x7fbadc0bbc80 msgr2=0x7fbadc0bc090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:20.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.088+0000 7fbadbfff700 1 -- 172.21.15.187:0/4026455826 shutdown_connections 2020-11-20T12:10:20.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.088+0000 7fbadbfff700 1 -- 172.21.15.187:0/4026455826 wait complete. 2020-11-20T12:10:20.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.090+0000 7fbadbfff700 1 Processor -- start 2020-11-20T12:10:20.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.090+0000 7fbadbfff700 1 -- start start 2020-11-20T12:10:20.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.091+0000 7fbadbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbadc060330 0x7fbadc158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.091+0000 7fbadbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbadc060710 0x7fbadc09ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.091+0000 7fbadbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.091+0000 7fbadbfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc0a1430 con 0x7fbadc060710 2020-11-20T12:10:20.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.091+0000 7fbadbfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc0a16d0 con 0x7fbadc146200 2020-11-20T12:10:20.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.091+0000 7fbadbfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc158f40 con 0x7fbadc060330 2020-11-20T12:10:20.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.091+0000 7fbadb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbadb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc0b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45464/0 (socket says 172.21.15.187:45464) 2020-11-20T12:10:20.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbadb7fe700 1 -- 172.21.15.187:0/2571800659 learned_addr learned my addr 172.21.15.187:0/2571800659 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:20.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbada7fc700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbadc060710 0x7fbadc09ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbadb7fe700 1 -- 172.21.15.187:0/2571800659 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbadc060330 msgr2=0x7fbadc158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbadb7fe700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbadc060330 0x7fbadc158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:20.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbadb7fe700 1 -- 172.21.15.187:0/2571800659 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbadc060710 msgr2=0x7fbadc09ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbadb7fe700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbadc060710 0x7fbadc09ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbadb7fe700 1 -- 172.21.15.187:0/2571800659 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc0a1970 con 0x7fbadc146200 2020-11-20T12:10:20.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.092+0000 7fbadb7fe700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc0b2e90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7fbad40092d0 tx=0x7fbad4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:20.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.093+0000 7fbac2ffd700 1 -- 172.21.15.187:0/2571800659 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad400bb80 con 0x7fbadc146200 2020-11-20T12:10:20.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.093+0000 7fbac2ffd700 1 -- 172.21.15.187:0/2571800659 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad400be50 con 0x7fbadc146200 2020-11-20T12:10:20.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.093+0000 7fbac2ffd700 1 -- 172.21.15.187:0/2571800659 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad401c680 con 0x7fbadc146200 2020-11-20T12:10:20.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.093+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbadc09d1e0 con 0x7fbadc146200 2020-11-20T12:10:20.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.093+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbadc0b37a0 con 0x7fbadc146200 2020-11-20T12:10:20.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.094+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbadc0b3990 con 0x7fbadc146200 2020-11-20T12:10:20.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.094+0000 7fbac2ffd700 1 -- 172.21.15.187:0/2571800659 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbad4023030 con 0x7fbadc146200 2020-11-20T12:10:20.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.095+0000 7fbac2ffd700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbab8056d20 0x7fbab80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.095+0000 7fbac2ffd700 1 -- 172.21.15.187:0/2571800659 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fbad40714c0 con 0x7fbadc146200 2020-11-20T12:10:20.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.096+0000 7fbadaffd700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbab8056d20 0x7fbab80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.096+0000 7fbadaffd700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbab8056d20 0x7fbab80591e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fbacc002330 tx=0x7fbacc0077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:20.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.099+0000 7fbac2ffd700 1 -- 172.21.15.187:0/2571800659 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbadc0b3990 con 0x7fbadc146200 2020-11-20T12:10:20.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.289+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbadc0b3990 con 0x7fbadc146200 2020-11-20T12:10:20.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.290+0000 7fbac2ffd700 1 -- 172.21.15.187:0/2571800659 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v65) v1 ==== 84+0+143 (secure 0 0 0) 0x7fbadc0b3990 con 0x7fbadc146200 2020-11-20T12:10:20.295 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:20.295 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:10:20.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:20.296 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:10:20.296 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:10:20.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:20.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:20.297 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:10:20.297 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:10:20.297 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:20.297 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:10:20.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbab8056d20 msgr2=0x7fbab80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbab8056d20 0x7fbab80591e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fbacc002330 tx=0x7fbacc0077f0).stop 2020-11-20T12:10:20.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 msgr2=0x7fbadc0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc0b2e90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7fbad40092d0 tx=0x7fbad4016000).stop 2020-11-20T12:10:20.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 shutdown_connections 2020-11-20T12:10:20.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbab8056d20 0x7fbab80591e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc146200 0x7fbadc0b2e90 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbadc060330 0x7fbadc158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:20.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 --2- 172.21.15.187:0/2571800659 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbadc060710 0x7fbadc09ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 >> 172.21.15.187:0/2571800659 conn(0x7fbadc0bbc80 msgr2=0x7fbadc148080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:20.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 shutdown_connections 2020-11-20T12:10:20.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.293+0000 7fbadbfff700 1 -- 172.21.15.187:0/2571800659 wait complete. 2020-11-20T12:10:20.318 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:10:20.319 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:10:20.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.563+0000 7f0ba4bc4700 1 Processor -- start 2020-11-20T12:10:20.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.563+0000 7f0ba4bc4700 1 -- start start 2020-11-20T12:10:20.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.564+0000 7f0ba4bc4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0ba0080f00 0x7f0ba00be7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.564+0000 7f0ba4bc4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba0148910 0x7f0ba0148d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.564+0000 7f0ba4bc4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba00bee40 con 0x7f0ba00812e0 2020-11-20T12:10:20.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.564+0000 7f0ba4bc4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba014bee0 con 0x7f0ba0080f00 2020-11-20T12:10:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.564+0000 7f0ba4bc4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba014c180 con 0x7f0ba0148910 2020-11-20T12:10:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.565+0000 7f0b9dd9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0ba00812e0 0x7f0ba014c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42074/0 (socket says 172.21.15.187:42074) 2020-11-20T12:10:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.565+0000 7f0b9dd9b700 1 -- 172.21.15.187:0/3079340331 learned_addr learned my addr 172.21.15.187:0/3079340331 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.565+0000 7f0b9e59c700 1 --2- 172.21.15.187:0/3079340331 >> v2:172.21.15.187:3300/0 conn(0x7f0ba0080f00 0x7f0ba00be7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.565+0000 7f0b9ed9d700 1 --2- 172.21.15.187:0/3079340331 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba0148910 0x7f0ba0148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.566+0000 7f0b9e59c700 1 -- 172.21.15.187:0/3079340331 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba0148910 msgr2=0x7f0ba0148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:20.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.566+0000 7f0b9e59c700 1 --2- 172.21.15.187:0/3079340331 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba0148910 0x7f0ba0148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.566+0000 7f0b9e59c700 1 -- 172.21.15.187:0/3079340331 >> v1:172.21.15.187:6789/0 conn(0x7f0ba00812e0 legacy=0x7f0ba014c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.566+0000 7f0b9e59c700 1 -- 172.21.15.187:0/3079340331 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba014cd30 con 0x7f0ba0080f00 2020-11-20T12:10:20.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.566+0000 7f0b9e59c700 1 --2- 172.21.15.187:0/3079340331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0080f00 0x7f0ba00be7a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f0b90009c90 tx=0x7f0b900091d0).ready entity=mon.1 client_cookie=b36f27c6bf7f19c5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:20.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.567+0000 7f0b9d59a700 1 -- 172.21.15.187:0/3079340331 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9000ba70 con 0x7f0ba0080f00 2020-11-20T12:10:20.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.567+0000 7f0b9d59a700 1 -- 172.21.15.187:0/3079340331 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9000bbd0 con 0x7f0ba0080f00 2020-11-20T12:10:20.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.567+0000 7f0b9d59a700 1 -- 172.21.15.187:0/3079340331 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90011400 con 0x7f0ba0080f00 2020-11-20T12:10:20.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.568+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3079340331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0080f00 msgr2=0x7f0ba00be7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.568+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3079340331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0080f00 0x7f0ba00be7a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f0b90009c90 tx=0x7f0b900091d0).stop 2020-11-20T12:10:20.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.568+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3079340331 shutdown_connections 2020-11-20T12:10:20.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.568+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3079340331 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba0148910 0x7f0ba0148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.568+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3079340331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0080f00 0x7f0ba00be7a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.568+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3079340331 >> 172.21.15.187:0/3079340331 conn(0x7f0ba00bf100 msgr2=0x7f0ba00bf510 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:20.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.569+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3079340331 shutdown_connections 2020-11-20T12:10:20.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.569+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3079340331 wait complete. 2020-11-20T12:10:20.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.571+0000 7f0ba4bc4700 1 Processor -- start 2020-11-20T12:10:20.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.571+0000 7f0ba4bc4700 1 -- start start 2020-11-20T12:10:20.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.572+0000 7f0ba4bc4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ba0080f00 0x7f0ba0141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.572+0000 7f0ba4bc4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba00812e0 0x7f0ba01494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.572+0000 7f0ba4bc4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0148910 0x7f0ba00b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.572+0000 7f0ba4bc4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba0142150 con 0x7f0ba0080f00 2020-11-20T12:10:20.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.572+0000 7f0ba4bc4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba0149e60 con 0x7f0ba0148910 2020-11-20T12:10:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.572+0000 7f0ba4bc4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba014a100 con 0x7f0ba00812e0 2020-11-20T12:10:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0148910 0x7f0ba00b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0148910 0x7f0ba00b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45478/0 (socket says 172.21.15.187:45478) 2020-11-20T12:10:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 -- 172.21.15.187:0/3477496366 learned_addr learned my addr 172.21.15.187:0/3477496366 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9e59c700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ba0080f00 0x7f0ba0141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9dd9b700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba00812e0 0x7f0ba01494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 -- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba00812e0 msgr2=0x7f0ba01494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba00812e0 0x7f0ba01494e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 -- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ba0080f00 msgr2=0x7f0ba0141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ba0080f00 0x7f0ba0141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 -- 172.21.15.187:0/3477496366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba014a3a0 con 0x7f0ba0148910 2020-11-20T12:10:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9e59c700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ba0080f00 0x7f0ba0141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.573+0000 7f0b9ed9d700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0148910 0x7f0ba00b54d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7f0b9800eba0 tx=0x7f0b98006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.574+0000 7f0b8e7fc700 1 -- 172.21.15.187:0/3477496366 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9800b8d0 con 0x7f0ba0148910 2020-11-20T12:10:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.574+0000 7f0b8e7fc700 1 -- 172.21.15.187:0/3477496366 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9800ba30 con 0x7f0ba0148910 2020-11-20T12:10:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.574+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba014b6a0 con 0x7f0ba0148910 2020-11-20T12:10:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.574+0000 7f0b8e7fc700 1 -- 172.21.15.187:0/3477496366 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9800bdd0 con 0x7f0ba0148910 2020-11-20T12:10:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.574+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba0149c10 con 0x7f0ba0148910 2020-11-20T12:10:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.575+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba014a530 con 0x7f0ba0148910 2020-11-20T12:10:20.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.576+0000 7f0b8e7fc700 1 -- 172.21.15.187:0/3477496366 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0b98012600 con 0x7f0ba0148910 2020-11-20T12:10:20.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.576+0000 7f0b8e7fc700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0b74056d20 0x7f0b740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:20.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.577+0000 7f0b8e7fc700 1 -- 172.21.15.187:0/3477496366 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0b9806fff0 con 0x7f0ba0148910 2020-11-20T12:10:20.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.577+0000 7f0b9e59c700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0b74056d20 0x7f0b740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:20.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.578+0000 7f0b9e59c700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0b74056d20 0x7f0b740591e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f0b9000b770 tx=0x7f0b90016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:20.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.581+0000 7f0b8e7fc700 1 -- 172.21.15.187:0/3477496366 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0b98045770 con 0x7f0ba0148910 2020-11-20T12:10:20.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:20.763+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f0ba0044f60 con 0x7f0ba0148910 2020-11-20T12:10:21.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.582+0000 7f0b8e7fc700 1 -- 172.21.15.187:0/3477496366 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v67) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0b98048d90 con 0x7f0ba0148910 2020-11-20T12:10:21.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0b74056d20 msgr2=0x7f0b740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:21.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0b74056d20 0x7f0b740591e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f0b9000b770 tx=0x7f0b90016000).stop 2020-11-20T12:10:21.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0148910 msgr2=0x7f0ba00b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:21.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0148910 0x7f0ba00b54d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7f0b9800eba0 tx=0x7f0b98006df0).stop 2020-11-20T12:10:21.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 shutdown_connections 2020-11-20T12:10:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0b74056d20 0x7f0b740591e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ba0080f00 0x7f0ba0141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ba0148910 0x7f0ba00b54d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 --2- 172.21.15.187:0/3477496366 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ba00812e0 0x7f0ba01494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 >> 172.21.15.187:0/3477496366 conn(0x7f0ba00bf100 msgr2=0x7f0ba00bcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:21.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.586+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 shutdown_connections 2020-11-20T12:10:21.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.587+0000 7f0ba4bc4700 1 -- 172.21.15.187:0/3477496366 wait complete. 2020-11-20T12:10:21.593 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:10:21.618 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:10:21.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.868+0000 7f9cba58d700 1 Processor -- start 2020-11-20T12:10:21.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.868+0000 7f9cba58d700 1 -- start start 2020-11-20T12:10:21.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.870+0000 7f9cba58d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9cb4145cc0 0x7f9cb4066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:21.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.870+0000 7f9cba58d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9cb40683f0 0x7f9cb4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:21.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.870+0000 7f9cba58d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb414c170 con 0x7f9cb4065680 2020-11-20T12:10:21.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.870+0000 7f9cba58d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb41497f0 con 0x7f9cb40683f0 2020-11-20T12:10:21.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.870+0000 7f9cba58d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb414c410 con 0x7f9cb4145cc0 2020-11-20T12:10:21.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.870+0000 7f9cb958b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9cb4145cc0 0x7f9cb4066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:21.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.870+0000 7f9cb958b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9cb4145cc0 0x7f9cb4066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45484/0 (socket says 172.21.15.187:45484) 2020-11-20T12:10:21.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.870+0000 7f9cb958b700 1 -- 172.21.15.187:0/1688164032 learned_addr learned my addr 172.21.15.187:0/1688164032 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:21.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.871+0000 7f9cb8d8a700 1 --2- 172.21.15.187:0/1688164032 >> v2:172.21.15.5:3300/0 conn(0x7f9cb40683f0 0x7f9cb4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:21.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.871+0000 7f9cb958b700 1 -- 172.21.15.187:0/1688164032 >> v2:172.21.15.5:3300/0 conn(0x7f9cb40683f0 msgr2=0x7f9cb4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:21.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.871+0000 7f9cb958b700 1 --2- 172.21.15.187:0/1688164032 >> v2:172.21.15.5:3300/0 conn(0x7f9cb40683f0 0x7f9cb4065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:21.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.871+0000 7f9cb958b700 1 -- 172.21.15.187:0/1688164032 >> v1:172.21.15.5:6789/0 conn(0x7f9cb4065680 legacy=0x7f9cb4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:21.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.871+0000 7f9cb958b700 1 -- 172.21.15.187:0/1688164032 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb414c6b0 con 0x7f9cb4145cc0 2020-11-20T12:10:21.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.871+0000 7f9cb8d8a700 1 --2- 172.21.15.187:0/1688164032 >> v2:172.21.15.5:3300/0 conn(0x7f9cb40683f0 0x7f9cb4065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:21.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.871+0000 7f9cb958b700 1 --2- 172.21.15.187:0/1688164032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb4145cc0 0x7f9cb4066040 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f9cb40b85e0 tx=0x7f9ca8006eb0).ready entity=mon.1 client_cookie=71b404e6fc8d5ca6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:21.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.872+0000 7f9ca7fff700 1 -- 172.21.15.187:0/1688164032 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8009890 con 0x7f9cb4145cc0 2020-11-20T12:10:21.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.872+0000 7f9ca7fff700 1 -- 172.21.15.187:0/1688164032 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca80099f0 con 0x7f9cb4145cc0 2020-11-20T12:10:21.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.872+0000 7f9ca7fff700 1 -- 172.21.15.187:0/1688164032 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8009d70 con 0x7f9cb4145cc0 2020-11-20T12:10:21.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.872+0000 7f9cba58d700 1 -- 172.21.15.187:0/1688164032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb4145cc0 msgr2=0x7f9cb4066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:21.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.872+0000 7f9cba58d700 1 --2- 172.21.15.187:0/1688164032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb4145cc0 0x7f9cb4066040 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f9cb40b85e0 tx=0x7f9ca8006eb0).stop 2020-11-20T12:10:21.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.873+0000 7f9cba58d700 1 -- 172.21.15.187:0/1688164032 shutdown_connections 2020-11-20T12:10:21.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.873+0000 7f9cba58d700 1 --2- 172.21.15.187:0/1688164032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb4145cc0 0x7f9cb4066040 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:21.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.873+0000 7f9cba58d700 1 --2- 172.21.15.187:0/1688164032 >> v2:172.21.15.5:3300/0 conn(0x7f9cb40683f0 0x7f9cb4065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:21.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.873+0000 7f9cba58d700 1 -- 172.21.15.187:0/1688164032 >> 172.21.15.187:0/1688164032 conn(0x7f9cb40bab30 msgr2=0x7f9cb40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:21.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.873+0000 7f9cba58d700 1 -- 172.21.15.187:0/1688164032 shutdown_connections 2020-11-20T12:10:21.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.873+0000 7f9cba58d700 1 -- 172.21.15.187:0/1688164032 wait complete. 2020-11-20T12:10:21.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.874+0000 7f9cba58d700 1 Processor -- start 2020-11-20T12:10:21.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.874+0000 7f9cba58d700 1 -- start start 2020-11-20T12:10:21.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.875+0000 7f9cba58d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cb4065680 0x7f9cb40a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:21.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.875+0000 7f9cba58d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb40683f0 0x7f9cb409b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:21.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.875+0000 7f9cba58d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cb409b150 0x7f9cb415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:21.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.875+0000 7f9cba58d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb409dd10 con 0x7f9cb4065680 2020-11-20T12:10:21.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.875+0000 7f9cba58d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb415c660 con 0x7f9cb40683f0 2020-11-20T12:10:21.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.875+0000 7f9cba58d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb415c900 con 0x7f9cb409b150 2020-11-20T12:10:21.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb40683f0 0x7f9cb409b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:21.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb40683f0 0x7f9cb409b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45490/0 (socket says 172.21.15.187:45490) 2020-11-20T12:10:21.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 -- 172.21.15.187:0/12386346 learned_addr learned my addr 172.21.15.187:0/12386346 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:21.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb958b700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cb4065680 0x7f9cb40a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:21.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 -- 172.21.15.187:0/12386346 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cb409b150 msgr2=0x7f9cb415bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:21.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cb409b150 0x7f9cb415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:21.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 -- 172.21.15.187:0/12386346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cb4065680 msgr2=0x7f9cb40a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:21.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cb4065680 0x7f9cb40a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:21.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 -- 172.21.15.187:0/12386346 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb415c160 con 0x7f9cb40683f0 2020-11-20T12:10:21.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.876+0000 7f9cb8d8a700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb40683f0 0x7f9cb409b6f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f9cb000c8f0 tx=0x7f9cb0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:21.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.877+0000 7f9ca57fa700 1 -- 172.21.15.187:0/12386346 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0007710 con 0x7f9cb40683f0 2020-11-20T12:10:21.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.877+0000 7f9ca57fa700 1 -- 172.21.15.187:0/12386346 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0007870 con 0x7f9cb40683f0 2020-11-20T12:10:21.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.877+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb415c410 con 0x7f9cb40683f0 2020-11-20T12:10:21.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.877+0000 7f9ca57fa700 1 -- 172.21.15.187:0/12386346 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0007bf0 con 0x7f9cb40683f0 2020-11-20T12:10:21.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.877+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb4143e00 con 0x7f9cb40683f0 2020-11-20T12:10:21.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.878+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb4061a20 con 0x7f9cb40683f0 2020-11-20T12:10:21.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.878+0000 7f9ca57fa700 1 -- 172.21.15.187:0/12386346 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9cb001b460 con 0x7f9cb40683f0 2020-11-20T12:10:21.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.879+0000 7f9ca57fa700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c8c056d20 0x7f9c8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:21.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.880+0000 7f9cb958b700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c8c056d20 0x7f9c8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:21.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.880+0000 7f9ca57fa700 1 -- 172.21.15.187:0/12386346 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f9cb0070090 con 0x7f9cb40683f0 2020-11-20T12:10:21.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.881+0000 7f9cb958b700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c8c056d20 0x7f9c8c0591e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9cb4063360 tx=0x7f9ca8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:21.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:21.883+0000 7f9ca57fa700 1 -- 172.21.15.187:0/12386346 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9cb0045540 con 0x7f9cb40683f0 2020-11-20T12:10:22.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.068+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f9cb414bdf0 con 0x7f9cb40683f0 2020-11-20T12:10:22.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.591+0000 7f9ca57fa700 1 -- 172.21.15.187:0/12386346 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v68) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9cb0048b60 con 0x7f9cb40683f0 2020-11-20T12:10:22.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.597+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c8c056d20 msgr2=0x7f9c8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:22.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.597+0000 7f9cba58d700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c8c056d20 0x7f9c8c0591e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9cb4063360 tx=0x7f9ca8015000).stop 2020-11-20T12:10:22.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.597+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb40683f0 msgr2=0x7f9cb409b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:22.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.597+0000 7f9cba58d700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb40683f0 0x7f9cb409b6f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f9cb000c8f0 tx=0x7f9cb0008040).stop 2020-11-20T12:10:22.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.598+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 shutdown_connections 2020-11-20T12:10:22.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.599+0000 7f9cba58d700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c8c056d20 0x7f9c8c0591e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:22.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.599+0000 7f9cba58d700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cb4065680 0x7f9cb40a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:22.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.599+0000 7f9cba58d700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cb409b150 0x7f9cb415bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:22.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.599+0000 7f9cba58d700 1 --2- 172.21.15.187:0/12386346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb40683f0 0x7f9cb409b6f0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:22.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.599+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 >> 172.21.15.187:0/12386346 conn(0x7f9cb40bab30 msgr2=0x7f9cb4148ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:22.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.601+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 shutdown_connections 2020-11-20T12:10:22.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:22.601+0000 7f9cba58d700 1 -- 172.21.15.187:0/12386346 wait complete. 2020-11-20T12:10:22.788 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:10:22.789 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:10:23.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.043+0000 7fa1252e9700 1 Processor -- start 2020-11-20T12:10:23.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.043+0000 7fa1252e9700 1 -- start start 2020-11-20T12:10:23.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.044+0000 7fa1252e9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa1200683f0 0x7fa120066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.045+0000 7fa1252e9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12014c170 con 0x7fa120145cc0 2020-11-20T12:10:23.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.045+0000 7fa1252e9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12014c410 con 0x7fa1201487e0 2020-11-20T12:10:23.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.045+0000 7fa1252e9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa12014c6b0 con 0x7fa1200683f0 2020-11-20T12:10:23.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.045+0000 7fa124ae8700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa1201487e0 0x7fa120148c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42106/0 (socket says 172.21.15.187:42106) 2020-11-20T12:10:23.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.045+0000 7fa124ae8700 1 -- 172.21.15.187:0/1714967046 learned_addr learned my addr 172.21.15.187:0/1714967046 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:23.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa1177fe700 1 --2- 172.21.15.187:0/1714967046 >> v2:172.21.15.5:3300/0 conn(0x7fa1200683f0 0x7fa120066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594189698 0 0) 0x7fa12014c410 con 0x7fa1201487e0 2020-11-20T12:10:23.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa100003240 con 0x7fa1201487e0 2020-11-20T12:10:23.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2429579137 0 0) 0x7fa100003240 con 0x7fa1201487e0 2020-11-20T12:10:23.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 >> v2:172.21.15.5:3300/0 conn(0x7fa1200683f0 msgr2=0x7fa120066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:23.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa11f7fe700 1 --2- 172.21.15.187:0/1714967046 >> v2:172.21.15.5:3300/0 conn(0x7fa1200683f0 0x7fa120066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 >> v1:172.21.15.5:6789/0 conn(0x7fa120145cc0 legacy=0x7fa120065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1200658a0 con 0x7fa1201487e0 2020-11-20T12:10:23.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.046+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fa118001860 con 0x7fa1201487e0 2020-11-20T12:10:23.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.047+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 >> v1:172.21.15.187:6789/0 conn(0x7fa1201487e0 legacy=0x7fa120148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.047+0000 7fa11f7fe700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa100003730 0x7fa100003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.047+0000 7fa11f7fe700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1000044e0 0x7fa100004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.047+0000 7fa11f7fe700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa100004e50 0x7fa10000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.047+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12014c6b0 con 0x7fa100004e50 2020-11-20T12:10:23.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.047+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa100004260 con 0x7fa1000044e0 2020-11-20T12:10:23.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.047+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10000bc20 con 0x7fa100003730 2020-11-20T12:10:23.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.047+0000 7fa1177fe700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1000044e0 0x7fa100004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa11ffff700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa100003730 0x7fa100003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa124ae8700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa100004e50 0x7fa10000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa1177fe700 1 -- 172.21.15.187:0/1714967046 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa100003730 msgr2=0x7fa100003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:23.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa1177fe700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa100003730 0x7fa100003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa1177fe700 1 -- 172.21.15.187:0/1714967046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa100004e50 msgr2=0x7fa10000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:23.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa1177fe700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa100004e50 0x7fa10000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa1177fe700 1 -- 172.21.15.187:0/1714967046 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100003240 con 0x7fa1000044e0 2020-11-20T12:10:23.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa124ae8700 1 -- 172.21.15.187:0/1714967046 reap_dead start 2020-11-20T12:10:23.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.048+0000 7fa1177fe700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1000044e0 0x7fa100004960 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7fa10800bc00 tx=0x7fa108006df0).ready entity=mon.1 client_cookie=adf22b783441b84d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:23.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10800b310 con 0x7fa1000044e0 2020-11-20T12:10:23.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10800eb40 con 0x7fa1000044e0 2020-11-20T12:10:23.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa11f7fe700 1 -- 172.21.15.187:0/1714967046 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10800eec0 con 0x7fa1000044e0 2020-11-20T12:10:23.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa1252e9700 1 -- 172.21.15.187:0/1714967046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1000044e0 msgr2=0x7fa100004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa1252e9700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1000044e0 0x7fa100004960 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7fa10800bc00 tx=0x7fa108006df0).stop 2020-11-20T12:10:23.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa1252e9700 1 -- 172.21.15.187:0/1714967046 shutdown_connections 2020-11-20T12:10:23.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa1252e9700 1 --2- 172.21.15.187:0/1714967046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1000044e0 0x7fa100004960 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa1252e9700 1 -- 172.21.15.187:0/1714967046 >> 172.21.15.187:0/1714967046 conn(0x7fa1200bab30 msgr2=0x7fa1200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:23.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa1252e9700 1 -- 172.21.15.187:0/1714967046 shutdown_connections 2020-11-20T12:10:23.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.049+0000 7fa1252e9700 1 -- 172.21.15.187:0/1714967046 wait complete. 2020-11-20T12:10:23.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.051+0000 7fa1252e9700 1 Processor -- start 2020-11-20T12:10:23.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.051+0000 7fa1252e9700 1 -- start start 2020-11-20T12:10:23.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.052+0000 7fa1252e9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1000044e0 0x7fa12009dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.052+0000 7fa1252e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1200a0940 0x7fa120142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.052+0000 7fa1252e9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa120143960 0x7fa120143dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.052+0000 7fa1252e9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa120148e90 con 0x7fa1000044e0 2020-11-20T12:10:23.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.052+0000 7fa1252e9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1201490c0 con 0x7fa120143960 2020-11-20T12:10:23.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.052+0000 7fa1252e9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa120149360 con 0x7fa1200a0940 2020-11-20T12:10:23.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa124ae8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa120143960 0x7fa120143dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa124ae8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa120143960 0x7fa120143dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45512/0 (socket says 172.21.15.187:45512) 2020-11-20T12:10:23.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa124ae8700 1 -- 172.21.15.187:0/75238876 learned_addr learned my addr 172.21.15.187:0/75238876 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:23.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa1177fe700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1200a0940 0x7fa120142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa11ffff700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1000044e0 0x7fa12009dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa124ae8700 1 -- 172.21.15.187:0/75238876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1200a0940 msgr2=0x7fa120142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa124ae8700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1200a0940 0x7fa120142120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa124ae8700 1 -- 172.21.15.187:0/75238876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1000044e0 msgr2=0x7fa12009dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa124ae8700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1000044e0 0x7fa12009dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.053+0000 7fa124ae8700 1 -- 172.21.15.187:0/75238876 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa120149600 con 0x7fa120143960 2020-11-20T12:10:23.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.054+0000 7fa124ae8700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa120143960 0x7fa120143dc0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7fa120061770 tx=0x7fa118001910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:23.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.054+0000 7fa11cff9700 1 -- 172.21.15.187:0/75238876 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11800cb40 con 0x7fa120143960 2020-11-20T12:10:23.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.054+0000 7fa11cff9700 1 -- 172.21.15.187:0/75238876 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11800cca0 con 0x7fa120143960 2020-11-20T12:10:23.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.054+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa120142610 con 0x7fa120143960 2020-11-20T12:10:23.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.054+0000 7fa11cff9700 1 -- 172.21.15.187:0/75238876 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118014430 con 0x7fa120143960 2020-11-20T12:10:23.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.054+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1200b1b70 con 0x7fa120143960 2020-11-20T12:10:23.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.055+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa120149790 con 0x7fa120143960 2020-11-20T12:10:23.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.056+0000 7fa11cff9700 1 -- 172.21.15.187:0/75238876 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa1180147a0 con 0x7fa120143960 2020-11-20T12:10:23.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.056+0000 7fa11cff9700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0fc056d20 0x7fa0fc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.056+0000 7fa11cff9700 1 -- 172.21.15.187:0/75238876 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fa11806fec0 con 0x7fa120143960 2020-11-20T12:10:23.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.057+0000 7fa11ffff700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0fc056d20 0x7fa0fc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.057+0000 7fa11ffff700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0fc056d20 0x7fa0fc0591e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa110000bc0 tx=0x7fa110009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:23.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.060+0000 7fa11cff9700 1 -- 172.21.15.187:0/75238876 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa118045380 con 0x7fa120143960 2020-11-20T12:10:23.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.308+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fa120061a20 con 0x7fa120143960 2020-11-20T12:10:23.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.323+0000 7fa11cff9700 1 -- 172.21.15.187:0/75238876 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 13 and data pool 14 v46) v1 ==== 189+0+0 (secure 0 0 0) 0x7fa1180489a0 con 0x7fa120143960 2020-11-20T12:10:23.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.327+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0fc056d20 msgr2=0x7fa0fc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.327+0000 7fa1252e9700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0fc056d20 0x7fa0fc0591e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa110000bc0 tx=0x7fa110009040).stop 2020-11-20T12:10:23.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.327+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa120143960 msgr2=0x7fa120143dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.327+0000 7fa1252e9700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa120143960 0x7fa120143dc0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7fa120061770 tx=0x7fa118001910).stop 2020-11-20T12:10:23.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.329+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 shutdown_connections 2020-11-20T12:10:23.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.329+0000 7fa1252e9700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0fc056d20 0x7fa0fc0591e0 secure :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0x7fa110000bc0 tx=0x7fa110009040).stop 2020-11-20T12:10:23.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.329+0000 7fa1252e9700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1000044e0 0x7fa12009dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.329+0000 7fa1252e9700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa120143960 0x7fa120143dc0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.329+0000 7fa1252e9700 1 --2- 172.21.15.187:0/75238876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1200a0940 0x7fa120142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.329+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 >> 172.21.15.187:0/75238876 conn(0x7fa1200bab30 msgr2=0x7fa120065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:23.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.337+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 shutdown_connections 2020-11-20T12:10:23.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.341+0000 7fa1252e9700 1 -- 172.21.15.187:0/75238876 wait complete. 2020-11-20T12:10:23.348 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 13 and data pool 14 2020-11-20T12:10:23.371 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:10:23.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.598+0000 7fd22a107700 1 Processor -- start 2020-11-20T12:10:23.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.599+0000 7fd22a107700 1 -- start start 2020-11-20T12:10:23.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.600+0000 7fd22a107700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd224065190 0x7fd224065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.600+0000 7fd22a107700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2240b13e0 con 0x7fd224065f90 2020-11-20T12:10:23.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.600+0000 7fd22a107700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd224062c10 con 0x7fd22409ca90 2020-11-20T12:10:23.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.600+0000 7fd22a107700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22414ab30 con 0x7fd224065190 2020-11-20T12:10:23.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.600+0000 7fd229906700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd22409ca90 0x7fd22409cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42134/0 (socket says 172.21.15.187:42134) 2020-11-20T12:10:23.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.600+0000 7fd229906700 1 -- 172.21.15.187:0/1391929000 learned_addr learned my addr 172.21.15.187:0/1391929000 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:23.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.601+0000 7fd229105700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd224065190 0x7fd224065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.601+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622895628 0 0) 0x7fd224062c10 con 0x7fd22409ca90 2020-11-20T12:10:23.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.601+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd20c003240 con 0x7fd22409ca90 2020-11-20T12:10:23.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.601+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3605710171 0 0) 0x7fd20c003240 con 0x7fd22409ca90 2020-11-20T12:10:23.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd224065190 msgr2=0x7fd224065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:23.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd224065190 0x7fd224065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 >> v1:172.21.15.5:6789/0 conn(0x7fd224065f90 legacy=0x7fd224143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2240b1640 con 0x7fd22409ca90 2020-11-20T12:10:23.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd21c001860 con 0x7fd22409ca90 2020-11-20T12:10:23.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 >> v1:172.21.15.187:6789/0 conn(0x7fd22409ca90 legacy=0x7fd22409cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd224065190 0x7fd20c003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd20c003c40 0x7fd20c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd20c0049d0 0x7fd20c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22414ab30 con 0x7fd20c003c40 2020-11-20T12:10:23.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c0041d0 con 0x7fd20c0049d0 2020-11-20T12:10:23.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c00fb40 con 0x7fd224065190 2020-11-20T12:10:23.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.602+0000 7fd229906700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd20c0049d0 0x7fd20c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd229906700 1 -- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd224065190 msgr2=0x7fd20c003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:23.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd229105700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd224065190 0x7fd20c003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd228904700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd20c003c40 0x7fd20c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd229906700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd224065190 0x7fd20c003730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd229906700 1 -- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd20c003c40 msgr2=0x7fd20c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:23.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd229906700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd20c003c40 0x7fd20c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd229906700 1 -- 172.21.15.187:0/1391929000 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20c003240 con 0x7fd20c0049d0 2020-11-20T12:10:23.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd228904700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd20c003c40 0x7fd20c0044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:23.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd229906700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd20c0049d0 0x7fd20c004e50 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fd21c00c840 tx=0x7fd2240bb650).ready entity=mon.1 client_cookie=d19ba6143cfde6ec server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:23.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c005b80 con 0x7fd20c0049d0 2020-11-20T12:10:23.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c005ce0 con 0x7fd20c0049d0 2020-11-20T12:10:23.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.603+0000 7fd213fff700 1 -- 172.21.15.187:0/1391929000 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c007300 con 0x7fd20c0049d0 2020-11-20T12:10:23.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.604+0000 7fd22a107700 1 -- 172.21.15.187:0/1391929000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd20c0049d0 msgr2=0x7fd20c004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.604+0000 7fd22a107700 1 --2- 172.21.15.187:0/1391929000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd20c0049d0 0x7fd20c004e50 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fd21c00c840 tx=0x7fd2240bb650).stop 2020-11-20T12:10:23.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.604+0000 7fd229906700 1 -- 172.21.15.187:0/1391929000 reap_dead start 2020-11-20T12:10:23.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.604+0000 7fd22a107700 1 -- 172.21.15.187:0/1391929000 shutdown_connections 2020-11-20T12:10:23.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.604+0000 7fd22a107700 1 -- 172.21.15.187:0/1391929000 >> 172.21.15.187:0/1391929000 conn(0x7fd2240bab30 msgr2=0x7fd2240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:23.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.604+0000 7fd22a107700 1 -- 172.21.15.187:0/1391929000 shutdown_connections 2020-11-20T12:10:23.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.604+0000 7fd22a107700 1 -- 172.21.15.187:0/1391929000 wait complete. 2020-11-20T12:10:23.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.606+0000 7fd22a107700 1 Processor -- start 2020-11-20T12:10:23.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.606+0000 7fd22a107700 1 -- start start 2020-11-20T12:10:23.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd22a107700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd224145670 0x7fd2240ba340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd22a107700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd224145a50 0x7fd22409eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd22a107700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd22413e570 0x7fd22413e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd22a107700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22409f3c0 con 0x7fd224145a50 2020-11-20T12:10:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd22a107700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22409e790 con 0x7fd224145670 2020-11-20T12:10:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd22a107700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22409ea30 con 0x7fd22413e570 2020-11-20T12:10:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd229105700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd224145670 0x7fd2240ba340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd229105700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd224145670 0x7fd2240ba340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45536/0 (socket says 172.21.15.187:45536) 2020-11-20T12:10:23.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd229105700 1 -- 172.21.15.187:0/587339250 learned_addr learned my addr 172.21.15.187:0/587339250 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.608+0000 7fd228904700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd224145a50 0x7fd22409eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd229105700 1 -- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd22413e570 msgr2=0x7fd22413e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd229906700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd22413e570 0x7fd22413e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd229105700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd22413e570 0x7fd22413e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd229105700 1 -- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd224145a50 msgr2=0x7fd22409eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd229105700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd224145a50 0x7fd22409eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd229105700 1 -- 172.21.15.187:0/587339250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22409e4b0 con 0x7fd224145670 2020-11-20T12:10:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd228904700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd224145a50 0x7fd22409eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd229105700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd224145670 0x7fd2240ba340 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7fd22000f830 tx=0x7fd220008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd2117fa700 1 -- 172.21.15.187:0/587339250 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220011660 con 0x7fd224145670 2020-11-20T12:10:23.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd2117fa700 1 -- 172.21.15.187:0/587339250 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2200117c0 con 0x7fd224145670 2020-11-20T12:10:23.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd2117fa700 1 -- 172.21.15.187:0/587339250 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220011b40 con 0x7fd224145670 2020-11-20T12:10:23.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.609+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22413fed0 con 0x7fd224145670 2020-11-20T12:10:23.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.610+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2241403f0 con 0x7fd224145670 2020-11-20T12:10:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.611+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2240ba830 con 0x7fd224145670 2020-11-20T12:10:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.611+0000 7fd2117fa700 1 -- 172.21.15.187:0/587339250 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd220013400 con 0x7fd224145670 2020-11-20T12:10:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.612+0000 7fd2117fa700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1fc056d20 0x7fd1fc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.612+0000 7fd2117fa700 1 -- 172.21.15.187:0/587339250 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd220070100 con 0x7fd224145670 2020-11-20T12:10:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.612+0000 7fd228904700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1fc056d20 0x7fd1fc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:23.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.613+0000 7fd228904700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1fc056d20 0x7fd1fc0591e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fd214007fd0 tx=0x7fd214009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:23.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.616+0000 7fd2117fa700 1 -- 172.21.15.187:0/587339250 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd2200454f0 con 0x7fd224145670 2020-11-20T12:10:23.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.798+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd224061a20 con 0x7fd224145670 2020-11-20T12:10:23.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.801+0000 7fd2117fa700 1 -- 172.21.15.187:0/587339250 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16794 (secure 0 0 0) 0x7fd220017090 con 0x7fd224145670 2020-11-20T12:10:23.806 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:23.806 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":69,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:10:23.311441+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:10:20.847618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2020-11-20T12:10:22.150115+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:10:23.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1fc056d20 msgr2=0x7fd1fc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1fc056d20 0x7fd1fc0591e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fd214007fd0 tx=0x7fd214009040).stop 2020-11-20T12:10:23.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd224145670 msgr2=0x7fd2240ba340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:23.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd224145670 0x7fd2240ba340 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7fd22000f830 tx=0x7fd220008040).stop 2020-11-20T12:10:23.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 shutdown_connections 2020-11-20T12:10:23.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1fc056d20 0x7fd1fc0591e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd22413e570 0x7fd22413e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd224145670 0x7fd2240ba340 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 --2- 172.21.15.187:0/587339250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd224145a50 0x7fd22409eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:23.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.803+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 >> 172.21.15.187:0/587339250 conn(0x7fd2240bab30 msgr2=0x7fd2240b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:23.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.804+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 shutdown_connections 2020-11-20T12:10:23.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:23.804+0000 7fd22a107700 1 -- 172.21.15.187:0/587339250 wait complete. 2020-11-20T12:10:23.833 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:10:24.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.071+0000 7fc06dbad700 1 Processor -- start 2020-11-20T12:10:24.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.072+0000 7fc06dbad700 1 -- start start 2020-11-20T12:10:24.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.073+0000 7fc06dbad700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc0680b1380 0x7fc068150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.073+0000 7fc06dbad700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0680b1760 0x7fc068065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.073+0000 7fc06dbad700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc068150390 con 0x7fc068065680 2020-11-20T12:10:24.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.073+0000 7fc06dbad700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc068150630 con 0x7fc0680b1380 2020-11-20T12:10:24.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.073+0000 7fc06dbad700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc068068510 con 0x7fc0680b1760 2020-11-20T12:10:24.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.074+0000 7fc05ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0680b1760 0x7fc068065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.074+0000 7fc05ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0680b1760 0x7fc068065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45546/0 (socket says 172.21.15.187:45546) 2020-11-20T12:10:24.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.074+0000 7fc05ffff700 1 -- 172.21.15.187:0/1003165738 learned_addr learned my addr 172.21.15.187:0/1003165738 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:24.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.074+0000 7fc06cbab700 1 --2- 172.21.15.187:0/1003165738 >> v2:172.21.15.5:3300/0 conn(0x7fc0680b1380 0x7fc068150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.074+0000 7fc05ffff700 1 -- 172.21.15.187:0/1003165738 >> v2:172.21.15.5:3300/0 conn(0x7fc0680b1380 msgr2=0x7fc068150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:24.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.074+0000 7fc05ffff700 1 --2- 172.21.15.187:0/1003165738 >> v2:172.21.15.5:3300/0 conn(0x7fc0680b1380 0x7fc068150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.074+0000 7fc05ffff700 1 -- 172.21.15.187:0/1003165738 >> v1:172.21.15.5:6789/0 conn(0x7fc068065680 legacy=0x7fc068061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:24.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.074+0000 7fc05ffff700 1 -- 172.21.15.187:0/1003165738 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc068150fc0 con 0x7fc0680b1760 2020-11-20T12:10:24.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.075+0000 7fc05ffff700 1 --2- 172.21.15.187:0/1003165738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1760 0x7fc068065190 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7fc068063a20 tx=0x7fc058006eb0).ready entity=mon.1 client_cookie=41c8e5d7d1b100de server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:24.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.075+0000 7fc05f7fe700 1 -- 172.21.15.187:0/1003165738 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058009890 con 0x7fc0680b1760 2020-11-20T12:10:24.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.075+0000 7fc05f7fe700 1 -- 172.21.15.187:0/1003165738 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0580099f0 con 0x7fc0680b1760 2020-11-20T12:10:24.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.075+0000 7fc05f7fe700 1 -- 172.21.15.187:0/1003165738 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058009d70 con 0x7fc0680b1760 2020-11-20T12:10:24.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.075+0000 7fc06dbad700 1 -- 172.21.15.187:0/1003165738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1760 msgr2=0x7fc068065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.075+0000 7fc06dbad700 1 --2- 172.21.15.187:0/1003165738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1760 0x7fc068065190 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7fc068063a20 tx=0x7fc058006eb0).stop 2020-11-20T12:10:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.076+0000 7fc06dbad700 1 -- 172.21.15.187:0/1003165738 shutdown_connections 2020-11-20T12:10:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.076+0000 7fc06dbad700 1 --2- 172.21.15.187:0/1003165738 >> v2:172.21.15.5:3300/0 conn(0x7fc0680b1380 0x7fc068150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.076+0000 7fc06dbad700 1 --2- 172.21.15.187:0/1003165738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1760 0x7fc068065190 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.076+0000 7fc06dbad700 1 -- 172.21.15.187:0/1003165738 >> 172.21.15.187:0/1003165738 conn(0x7fc0680bab30 msgr2=0x7fc0680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:24.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.076+0000 7fc06dbad700 1 -- 172.21.15.187:0/1003165738 shutdown_connections 2020-11-20T12:10:24.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.076+0000 7fc06dbad700 1 -- 172.21.15.187:0/1003165738 wait complete. 2020-11-20T12:10:24.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.077+0000 7fc06dbad700 1 Processor -- start 2020-11-20T12:10:24.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.077+0000 7fc06dbad700 1 -- start start 2020-11-20T12:10:24.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.078+0000 7fc06dbad700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1380 0x7fc06809c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.078+0000 7fc06dbad700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0680b1760 0x7fc0680a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.078+0000 7fc06dbad700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc06813d620 0x7fc06813da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.078+0000 7fc06dbad700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06813ec60 con 0x7fc06813d620 2020-11-20T12:10:24.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.078+0000 7fc06dbad700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06813e280 con 0x7fc0680b1380 2020-11-20T12:10:24.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.078+0000 7fc06dbad700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06813e520 con 0x7fc0680b1760 2020-11-20T12:10:24.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.079+0000 7fc06cbab700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1380 0x7fc06809c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.079+0000 7fc06cbab700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1380 0x7fc06809c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45550/0 (socket says 172.21.15.187:45550) 2020-11-20T12:10:24.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.079+0000 7fc06cbab700 1 -- 172.21.15.187:0/2502457634 learned_addr learned my addr 172.21.15.187:0/2502457634 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:24.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.079+0000 7fc06d3ac700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc06813d620 0x7fc06813da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc05ffff700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0680b1760 0x7fc0680a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc06cbab700 1 -- 172.21.15.187:0/2502457634 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0680b1760 msgr2=0x7fc0680a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc06cbab700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0680b1760 0x7fc0680a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc06cbab700 1 -- 172.21.15.187:0/2502457634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc06813d620 msgr2=0x7fc06813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc06cbab700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc06813d620 0x7fc06813da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc06cbab700 1 -- 172.21.15.187:0/2502457634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06813e7c0 con 0x7fc0680b1380 2020-11-20T12:10:24.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc06cbab700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1380 0x7fc06809c980 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7fc06400db90 tx=0x7fc064008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:24.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc05cff9700 1 -- 172.21.15.187:0/2502457634 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064009820 con 0x7fc0680b1380 2020-11-20T12:10:24.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc05cff9700 1 -- 172.21.15.187:0/2502457634 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc064009980 con 0x7fc0680b1380 2020-11-20T12:10:24.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc05cff9700 1 -- 172.21.15.187:0/2502457634 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064009d20 con 0x7fc0680b1380 2020-11-20T12:10:24.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06813df50 con 0x7fc0680b1380 2020-11-20T12:10:24.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.080+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0680a0920 con 0x7fc0680b1380 2020-11-20T12:10:24.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.081+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06813e950 con 0x7fc0680b1380 2020-11-20T12:10:24.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.082+0000 7fc05cff9700 1 -- 172.21.15.187:0/2502457634 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc064013690 con 0x7fc0680b1380 2020-11-20T12:10:24.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.083+0000 7fc05cff9700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc040056d50 0x7fc040059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.083+0000 7fc05cff9700 1 -- 172.21.15.187:0/2502457634 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc064070690 con 0x7fc0680b1380 2020-11-20T12:10:24.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.083+0000 7fc05ffff700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc040056d50 0x7fc040059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.084+0000 7fc05ffff700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc040056d50 0x7fc040059210 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fc058006ef0 tx=0x7fc058015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:24.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.087+0000 7fc05cff9700 1 -- 172.21.15.187:0/2502457634 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc064045a60 con 0x7fc0680b1380 2020-11-20T12:10:24.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.286+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc068061a20 con 0x7fc0680b1380 2020-11-20T12:10:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.288+0000 7fc05cff9700 1 -- 172.21.15.187:0/2502457634 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16794 (secure 0 0 0) 0x7fc064049080 con 0x7fc0680b1380 2020-11-20T12:10:24.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc040056d50 msgr2=0x7fc040059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc040056d50 0x7fc040059210 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fc058006ef0 tx=0x7fc058015000).stop 2020-11-20T12:10:24.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1380 msgr2=0x7fc06809c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1380 0x7fc06809c980 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7fc06400db90 tx=0x7fc064008040).stop 2020-11-20T12:10:24.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 shutdown_connections 2020-11-20T12:10:24.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc040056d50 0x7fc040059210 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc06813d620 0x7fc06813da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0680b1380 0x7fc06809c980 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 --2- 172.21.15.187:0/2502457634 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0680b1760 0x7fc0680a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 >> 172.21.15.187:0/2502457634 conn(0x7fc0680bab30 msgr2=0x7fc06814f300 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 shutdown_connections 2020-11-20T12:10:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.291+0000 7fc06dbad700 1 -- 172.21.15.187:0/2502457634 wait complete. 2020-11-20T12:10:24.315 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:24.316 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":69,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:10:23.311441+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:10:20.847618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2020-11-20T12:10:22.150115+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:10:24.321 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:10:24.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.557+0000 7f6a8b1a9700 1 Processor -- start 2020-11-20T12:10:24.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.557+0000 7f6a8b1a9700 1 -- start start 2020-11-20T12:10:24.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.559+0000 7f6a8b1a9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6a840b1380 0x7f6a84150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.559+0000 7f6a8b1a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a840b1760 0x7f6a84065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.559+0000 7f6a8b1a9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a84150390 con 0x7f6a84065680 2020-11-20T12:10:24.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.559+0000 7f6a8b1a9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a84150630 con 0x7f6a840b1380 2020-11-20T12:10:24.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.559+0000 7f6a8b1a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84068510 con 0x7f6a840b1760 2020-11-20T12:10:24.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.559+0000 7f6a8a1a7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6a840b1380 0x7f6a84150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.559+0000 7f6a8a9a8700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6a84065680 0x7f6a84061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42168/0 (socket says 172.21.15.187:42168) 2020-11-20T12:10:24.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.559+0000 7f6a8a9a8700 1 -- 172.21.15.187:0/614981049 learned_addr learned my addr 172.21.15.187:0/614981049 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:24.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.560+0000 7f6a899a6700 1 --2- 172.21.15.187:0/614981049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a840b1760 0x7f6a84065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.560+0000 7f6a8a1a7700 1 -- 172.21.15.187:0/614981049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a840b1760 msgr2=0x7f6a84065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:24.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.560+0000 7f6a8a1a7700 1 --2- 172.21.15.187:0/614981049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a840b1760 0x7f6a84065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.560+0000 7f6a8a1a7700 1 -- 172.21.15.187:0/614981049 >> v1:172.21.15.187:6789/0 conn(0x7f6a84065680 legacy=0x7f6a84061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.560+0000 7f6a8a1a7700 1 -- 172.21.15.187:0/614981049 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a84150fc0 con 0x7f6a840b1380 2020-11-20T12:10:24.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.560+0000 7f6a8a1a7700 1 --2- 172.21.15.187:0/614981049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1380 0x7f6a84150ad0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f6a8000b9a0 tx=0x7f6a80006df0).ready entity=mon.1 client_cookie=f13d5c9ca224a453 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:24.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.560+0000 7f6a891a5700 1 -- 172.21.15.187:0/614981049 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a800097b0 con 0x7f6a840b1380 2020-11-20T12:10:24.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.560+0000 7f6a891a5700 1 -- 172.21.15.187:0/614981049 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a80009910 con 0x7f6a840b1380 2020-11-20T12:10:24.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a891a5700 1 -- 172.21.15.187:0/614981049 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80009c90 con 0x7f6a840b1380 2020-11-20T12:10:24.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/614981049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1380 msgr2=0x7f6a84150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/614981049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1380 0x7f6a84150ad0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f6a8000b9a0 tx=0x7f6a80006df0).stop 2020-11-20T12:10:24.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/614981049 shutdown_connections 2020-11-20T12:10:24.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/614981049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1380 0x7f6a84150ad0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/614981049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a840b1760 0x7f6a84065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/614981049 >> 172.21.15.187:0/614981049 conn(0x7f6a840bab30 msgr2=0x7f6a840baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:24.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/614981049 shutdown_connections 2020-11-20T12:10:24.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.561+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/614981049 wait complete. 2020-11-20T12:10:24.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.563+0000 7f6a8b1a9700 1 Processor -- start 2020-11-20T12:10:24.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.563+0000 7f6a8b1a9700 1 -- start start 2020-11-20T12:10:24.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a8b1a9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a840b1380 0x7f6a8414a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a8b1a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1760 0x7f6a8414a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a8b1a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a8409dca0 0x7f6a8409e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a8b1a9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84160e40 con 0x7f6a840b1380 2020-11-20T12:10:24.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a8b1a9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8409f0b0 con 0x7f6a840b1760 2020-11-20T12:10:24.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a8b1a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8409f350 con 0x7f6a8409dca0 2020-11-20T12:10:24.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a899a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1760 0x7f6a8414a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a899a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1760 0x7f6a8414a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45566/0 (socket says 172.21.15.187:45566) 2020-11-20T12:10:24.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a899a6700 1 -- 172.21.15.187:0/167131246 learned_addr learned my addr 172.21.15.187:0/167131246 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:24.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.564+0000 7f6a899a6700 1 -- 172.21.15.187:0/167131246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a8409dca0 msgr2=0x7f6a8409e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a8a1a7700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a840b1380 0x7f6a8414a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a899a6700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a8409dca0 0x7f6a8409e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:24.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a899a6700 1 -- 172.21.15.187:0/167131246 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a840b1380 msgr2=0x7f6a8414a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:24.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a899a6700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a840b1380 0x7f6a8414a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:24.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a899a6700 1 -- 172.21.15.187:0/167131246 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8414ae80 con 0x7f6a840b1760 2020-11-20T12:10:24.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a899a6700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1760 0x7f6a8414a990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f6a7400d8e0 tx=0x7f6a74008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:24.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a727fc700 1 -- 172.21.15.187:0/167131246 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74009790 con 0x7f6a840b1760 2020-11-20T12:10:24.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8409e5f0 con 0x7f6a840b1760 2020-11-20T12:10:24.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a727fc700 1 -- 172.21.15.187:0/167131246 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a740098f0 con 0x7f6a840b1760 2020-11-20T12:10:24.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a727fc700 1 -- 172.21.15.187:0/167131246 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74009c90 con 0x7f6a840b1760 2020-11-20T12:10:24.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.565+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a841609d0 con 0x7f6a840b1760 2020-11-20T12:10:24.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.566+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8409f510 con 0x7f6a840b1760 2020-11-20T12:10:24.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.567+0000 7f6a727fc700 1 -- 172.21.15.187:0/167131246 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6a74013480 con 0x7f6a840b1760 2020-11-20T12:10:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.568+0000 7f6a727fc700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a5c056d50 0x7f6a5c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.568+0000 7f6a727fc700 1 -- 172.21.15.187:0/167131246 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6a7406ff10 con 0x7f6a840b1760 2020-11-20T12:10:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.568+0000 7f6a8a1a7700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a5c056d50 0x7f6a5c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.569+0000 7f6a8a1a7700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a5c056d50 0x7f6a5c059210 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6a80006e30 tx=0x7f6a80015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:24.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.572+0000 7f6a727fc700 1 -- 172.21.15.187:0/167131246 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6a74045300 con 0x7f6a840b1760 2020-11-20T12:10:24.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:24.824+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6a84061a20 con 0x7f6a840b1760 2020-11-20T12:10:25.135 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:10:25.131+0000 7fd06271c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:10:25.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.333+0000 7f6a727fc700 1 -- 172.21.15.187:0/167131246 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6a74048920 con 0x7f6a840b1760 2020-11-20T12:10:25.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a5c056d50 msgr2=0x7f6a5c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a5c056d50 0x7f6a5c059210 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6a80006e30 tx=0x7f6a80015000).stop 2020-11-20T12:10:25.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1760 msgr2=0x7f6a8414a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1760 0x7f6a8414a990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f6a7400d8e0 tx=0x7f6a74008040).stop 2020-11-20T12:10:25.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 shutdown_connections 2020-11-20T12:10:25.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a5c056d50 0x7f6a5c059210 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a840b1380 0x7f6a8414a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a8409dca0 0x7f6a8409e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:25.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 --2- 172.21.15.187:0/167131246 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a840b1760 0x7f6a8414a990 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.336+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 >> 172.21.15.187:0/167131246 conn(0x7f6a840bab30 msgr2=0x7f6a8414f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:25.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.337+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 shutdown_connections 2020-11-20T12:10:25.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.337+0000 7f6a8b1a9700 1 -- 172.21.15.187:0/167131246 wait complete. 2020-11-20T12:10:25.368 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:25.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.611+0000 7f6002442700 1 Processor -- start 2020-11-20T12:10:25.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.611+0000 7f6002442700 1 -- start start 2020-11-20T12:10:25.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.613+0000 7f6002442700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5ffc065570 0x7f5ffc09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:25.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.613+0000 7f6002442700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc0b13e0 con 0x7f5ffc065190 2020-11-20T12:10:25.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.613+0000 7f6002442700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc09d690 con 0x7f5ffc065f90 2020-11-20T12:10:25.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.613+0000 7f6002442700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ffc14ab30 con 0x7f5ffc065570 2020-11-20T12:10:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.613+0000 7f6000c3f700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5ffc065f90 0x7f5ffc09ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42182/0 (socket says 172.21.15.187:42182) 2020-11-20T12:10:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.613+0000 7f6000c3f700 1 -- 172.21.15.187:0/2478655445 learned_addr learned my addr 172.21.15.187:0/2478655445 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.614+0000 7f6001c41700 1 --2- 172.21.15.187:0/2478655445 >> v2:172.21.15.5:3300/0 conn(0x7f5ffc065570 0x7f5ffc09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.614+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134842438 0 0) 0x7f5ffc09d690 con 0x7f5ffc065f90 2020-11-20T12:10:25.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.614+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe4003240 con 0x7f5ffc065f90 2020-11-20T12:10:25.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.614+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925742247 0 0) 0x7f5ffc0b13e0 con 0x7f5ffc065190 2020-11-20T12:10:25.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.614+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc09d690 con 0x7f5ffc065190 2020-11-20T12:10:25.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2310018725 0 0) 0x7f5fe4003240 con 0x7f5ffc065f90 2020-11-20T12:10:25.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 >> v2:172.21.15.5:3300/0 conn(0x7f5ffc065570 msgr2=0x7f5ffc09d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:25.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 --2- 172.21.15.187:0/2478655445 >> v2:172.21.15.5:3300/0 conn(0x7f5ffc065570 0x7f5ffc09d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 >> v1:172.21.15.5:6789/0 conn(0x7f5ffc065190 legacy=0x7f5ffc143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc062c10 con 0x7f5ffc065f90 2020-11-20T12:10:25.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5fec0014e0 con 0x7f5ffc065f90 2020-11-20T12:10:25.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 >> v1:172.21.15.187:6789/0 conn(0x7f5ffc065f90 legacy=0x7f5ffc09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fe4004970 0x7f5fe4004d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:25.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fe4003480 0x7f5fe4003900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:25.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fe4003df0 0x7f5fe400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:25.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc14ab30 con 0x7f5fe4003df0 2020-11-20T12:10:25.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe400bc20 con 0x7f5fe4003480 2020-11-20T12:10:25.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe400bd60 con 0x7f5fe4004970 2020-11-20T12:10:25.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.615+0000 7f6000c3f700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fe4003480 0x7f5fe4003900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:25.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6001c41700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fe4003df0 0x7f5fe400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:25.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6001440700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fe4004970 0x7f5fe4004d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:25.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6000c3f700 1 -- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fe4004970 msgr2=0x7f5fe4004d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:25.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6000c3f700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fe4004970 0x7f5fe4004d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6000c3f700 1 -- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fe4003df0 msgr2=0x7f5fe400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:25.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6000c3f700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fe4003df0 0x7f5fe400b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6000c3f700 1 -- 172.21.15.187:0/2478655445 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe4003240 con 0x7f5fe4003480 2020-11-20T12:10:25.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6001c41700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fe4003df0 0x7f5fe400b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:25.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6001c41700 1 -- 172.21.15.187:0/2478655445 reap_dead start 2020-11-20T12:10:25.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.616+0000 7f6000c3f700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fe4003480 0x7f5fe4003900 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f5fec007fd0 tx=0x7f5ffc0b0fb0).ready entity=mon.1 client_cookie=549600463581853e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:25.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.617+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec007750 con 0x7f5fe4003480 2020-11-20T12:10:25.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.617+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec00ce30 con 0x7f5fe4003480 2020-11-20T12:10:25.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.617+0000 7f5febfff700 1 -- 172.21.15.187:0/2478655445 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec007320 con 0x7f5fe4003480 2020-11-20T12:10:25.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.617+0000 7f6002442700 1 -- 172.21.15.187:0/2478655445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fe4003480 msgr2=0x7f5fe4003900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.617+0000 7f6002442700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fe4003480 0x7f5fe4003900 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f5fec007fd0 tx=0x7f5ffc0b0fb0).stop 2020-11-20T12:10:25.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.617+0000 7f6002442700 1 -- 172.21.15.187:0/2478655445 shutdown_connections 2020-11-20T12:10:25.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.617+0000 7f6002442700 1 --2- 172.21.15.187:0/2478655445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fe4003480 0x7f5fe4003900 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.617+0000 7f6002442700 1 -- 172.21.15.187:0/2478655445 >> 172.21.15.187:0/2478655445 conn(0x7f5ffc0bab30 msgr2=0x7f5ffc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:25.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.618+0000 7f6002442700 1 -- 172.21.15.187:0/2478655445 shutdown_connections 2020-11-20T12:10:25.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.618+0000 7f6002442700 1 -- 172.21.15.187:0/2478655445 wait complete. 2020-11-20T12:10:25.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.619+0000 7f6002442700 1 Processor -- start 2020-11-20T12:10:25.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.619+0000 7f6002442700 1 -- start start 2020-11-20T12:10:25.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.620+0000 7f6002442700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ffc099420 0x7f5ffc0b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:25.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.620+0000 7f6002442700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ffc1483b0 0x7f5ffc149390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:25.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.620+0000 7f6002442700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ffc149880 0x7f5ffc15b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:25.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.620+0000 7f6002442700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc148cc0 con 0x7f5ffc1483b0 2020-11-20T12:10:25.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.620+0000 7f6002442700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc148930 con 0x7f5ffc149880 2020-11-20T12:10:25.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.620+0000 7f6002442700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc148ef0 con 0x7f5ffc099420 2020-11-20T12:10:25.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001c41700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ffc149880 0x7f5ffc15b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001c41700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ffc149880 0x7f5ffc15b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45586/0 (socket says 172.21.15.187:45586) 2020-11-20T12:10:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001c41700 1 -- 172.21.15.187:0/1183744420 learned_addr learned my addr 172.21.15.187:0/1183744420 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001440700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ffc099420 0x7f5ffc0b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6000c3f700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ffc1483b0 0x7f5ffc149390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001c41700 1 -- 172.21.15.187:0/1183744420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ffc099420 msgr2=0x7f5ffc0b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001c41700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ffc099420 0x7f5ffc0b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001c41700 1 -- 172.21.15.187:0/1183744420 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ffc1483b0 msgr2=0x7f5ffc149390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001c41700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ffc1483b0 0x7f5ffc149390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.621+0000 7f6001c41700 1 -- 172.21.15.187:0/1183744420 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc0b3df0 con 0x7f5ffc149880 2020-11-20T12:10:25.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.622+0000 7f6001c41700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ffc149880 0x7f5ffc15b920 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f5ffc066370 tx=0x7f5ff4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:25.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.622+0000 7f5fe97fa700 1 -- 172.21.15.187:0/1183744420 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff401a070 con 0x7f5ffc149880 2020-11-20T12:10:25.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.622+0000 7f5fe97fa700 1 -- 172.21.15.187:0/1183744420 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff400e910 con 0x7f5ffc149880 2020-11-20T12:10:25.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.622+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ffc0b2780 con 0x7f5ffc149880 2020-11-20T12:10:25.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.622+0000 7f5fe97fa700 1 -- 172.21.15.187:0/1183744420 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff400ed10 con 0x7f5ffc149880 2020-11-20T12:10:25.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.622+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ffc149d00 con 0x7f5ffc149880 2020-11-20T12:10:25.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.623+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ffc149f20 con 0x7f5ffc149880 2020-11-20T12:10:25.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.624+0000 7f5fe97fa700 1 -- 172.21.15.187:0/1183744420 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5ff4014640 con 0x7f5ffc149880 2020-11-20T12:10:25.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.624+0000 7f5fe97fa700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fd4056d20 0x7f5fd40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.625+0000 7f5fe97fa700 1 -- 172.21.15.187:0/1183744420 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5ff4070360 con 0x7f5ffc149880 2020-11-20T12:10:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.625+0000 7f6001440700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fd4056d20 0x7f5fd40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.625+0000 7f6001440700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fd4056d20 0x7f5fd40591e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5ff8000bc0 tx=0x7f5ff8007b90).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.628+0000 7f5fe97fa700 1 -- 172.21.15.187:0/1183744420 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5ff40456d0 con 0x7f5ffc149880 2020-11-20T12:10:25.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.871+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ffc061a20 con 0x7f5ffc149880 2020-11-20T12:10:25.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.872+0000 7f5fe97fa700 1 -- 172.21.15.187:0/1183744420 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f5ff4048cf0 con 0x7f5ffc149880 2020-11-20T12:10:25.878 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:25.878 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2299534653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2299534653},{"type":"v1","addr":"172.21.15.5:6833","nonce":2299534653}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:25.326709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6400":{"gid":6400,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/1383561284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1383561284},{"type":"v1","addr":"172.21.15.187:6835","nonce":1383561284}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:25.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fd4056d20 msgr2=0x7f5fd40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fd4056d20 0x7f5fd40591e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5ff8000bc0 tx=0x7f5ff8007b90).stop 2020-11-20T12:10:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ffc149880 msgr2=0x7f5ffc15b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ffc149880 0x7f5ffc15b920 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f5ffc066370 tx=0x7f5ff4008040).stop 2020-11-20T12:10:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 shutdown_connections 2020-11-20T12:10:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5fd4056d20 0x7f5fd40591e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ffc149880 0x7f5ffc15b920 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ffc099420 0x7f5ffc0b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 --2- 172.21.15.187:0/1183744420 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ffc1483b0 0x7f5ffc149390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:25.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 >> 172.21.15.187:0/1183744420 conn(0x7f5ffc0bab30 msgr2=0x7f5ffc0b9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:25.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 shutdown_connections 2020-11-20T12:10:25.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:25.875+0000 7f6002442700 1 -- 172.21.15.187:0/1183744420 wait complete. 2020-11-20T12:10:25.883 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 48 2020-11-20T12:10:25.904 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:10:26.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.140+0000 7fab5c89e700 1 Processor -- start 2020-11-20T12:10:26.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.140+0000 7fab5c89e700 1 -- start start 2020-11-20T12:10:26.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.142+0000 7fab5c89e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fab580b57b0 0x7fab58068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.142+0000 7fab5c89e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab580b5f50 0x7fab580b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.142+0000 7fab5c89e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab58144b80 con 0x7fab581482d0 2020-11-20T12:10:26.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.142+0000 7fab5c89e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab581489e0 con 0x7fab580b57b0 2020-11-20T12:10:26.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.142+0000 7fab5c89e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab58148c80 con 0x7fab580b5f50 2020-11-20T12:10:26.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.142+0000 7fab5759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab580b5f50 0x7fab580b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:26.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.142+0000 7fab5759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab580b5f50 0x7fab580b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45596/0 (socket says 172.21.15.187:45596) 2020-11-20T12:10:26.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.142+0000 7fab5759e700 1 -- 172.21.15.187:0/1754294086 learned_addr learned my addr 172.21.15.187:0/1754294086 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:26.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.143+0000 7fab5759e700 1 -- 172.21.15.187:0/1754294086 >> v2:172.21.15.5:3300/0 conn(0x7fab580b57b0 msgr2=0x7fab58068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:26.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.143+0000 7fab5759e700 1 --2- 172.21.15.187:0/1754294086 >> v2:172.21.15.5:3300/0 conn(0x7fab580b57b0 0x7fab58068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:26.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.143+0000 7fab5759e700 1 -- 172.21.15.187:0/1754294086 >> v1:172.21.15.5:6789/0 conn(0x7fab581482d0 legacy=0x7fab58062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:26.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.143+0000 7fab5759e700 1 -- 172.21.15.187:0/1754294086 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab58148f20 con 0x7fab580b5f50 2020-11-20T12:10:26.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.143+0000 7fab5759e700 1 --2- 172.21.15.187:0/1754294086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b6380 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fab58066230 tx=0x7fab50006eb0).ready entity=mon.1 client_cookie=6f9a3d4b2ff4d366 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:26.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.143+0000 7fab5659c700 1 -- 172.21.15.187:0/1754294086 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50009890 con 0x7fab580b5f50 2020-11-20T12:10:26.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.143+0000 7fab5659c700 1 -- 172.21.15.187:0/1754294086 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab500099f0 con 0x7fab580b5f50 2020-11-20T12:10:26.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.143+0000 7fab5659c700 1 -- 172.21.15.187:0/1754294086 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50009d70 con 0x7fab580b5f50 2020-11-20T12:10:26.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.144+0000 7fab5c89e700 1 -- 172.21.15.187:0/1754294086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 msgr2=0x7fab580b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.144+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1754294086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b6380 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fab58066230 tx=0x7fab50006eb0).stop 2020-11-20T12:10:26.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.144+0000 7fab5c89e700 1 -- 172.21.15.187:0/1754294086 shutdown_connections 2020-11-20T12:10:26.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.144+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1754294086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b6380 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.144+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1754294086 >> v2:172.21.15.5:3300/0 conn(0x7fab580b57b0 0x7fab58068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:26.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.144+0000 7fab5c89e700 1 -- 172.21.15.187:0/1754294086 >> 172.21.15.187:0/1754294086 conn(0x7fab580bab30 msgr2=0x7fab580baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:26.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.144+0000 7fab5c89e700 1 -- 172.21.15.187:0/1754294086 shutdown_connections 2020-11-20T12:10:26.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.144+0000 7fab5c89e700 1 -- 172.21.15.187:0/1754294086 wait complete. 2020-11-20T12:10:26.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.146+0000 7fab5c89e700 1 Processor -- start 2020-11-20T12:10:26.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.146+0000 7fab5c89e700 1 -- start start 2020-11-20T12:10:26.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.147+0000 7fab5c89e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab580b57b0 0x7fab5813dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.147+0000 7fab5c89e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.147+0000 7fab5c89e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab581482d0 0x7fab580b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.147+0000 7fab5c89e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab580a06d0 con 0x7fab580b57b0 2020-11-20T12:10:26.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.147+0000 7fab5c89e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab580a0430 con 0x7fab580b5f50 2020-11-20T12:10:26.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.147+0000 7fab5c89e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5809d330 con 0x7fab581482d0 2020-11-20T12:10:26.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.147+0000 7fab4e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:26.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45600/0 (socket says 172.21.15.187:45600) 2020-11-20T12:10:26.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4e59c700 1 -- 172.21.15.187:0/1214747073 learned_addr learned my addr 172.21.15.187:0/1214747073 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:26.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab56d9d700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab580b57b0 0x7fab5813dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:26.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4e59c700 1 -- 172.21.15.187:0/1214747073 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab581482d0 msgr2=0x7fab580b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4e59c700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab581482d0 0x7fab580b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:26.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4e59c700 1 -- 172.21.15.187:0/1214747073 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab580b57b0 msgr2=0x7fab5813dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4e59c700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab580b57b0 0x7fab5813dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4e59c700 1 -- 172.21.15.187:0/1214747073 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5813dfa0 con 0x7fab580b5f50 2020-11-20T12:10:26.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab56d9d700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab580b57b0 0x7fab5813dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:26.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4e59c700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b2210 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fab400096b0 tx=0x7fab40006d50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:26.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.148+0000 7fab4f7fe700 1 -- 172.21.15.187:0/1214747073 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4000d560 con 0x7fab580b5f50 2020-11-20T12:10:26.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.149+0000 7fab4f7fe700 1 -- 172.21.15.187:0/1214747073 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab40000e90 con 0x7fab580b5f50 2020-11-20T12:10:26.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.149+0000 7fab4f7fe700 1 -- 172.21.15.187:0/1214747073 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4000d910 con 0x7fab580b5f50 2020-11-20T12:10:26.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.149+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5813c220 con 0x7fab580b5f50 2020-11-20T12:10:26.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.149+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5809ced0 con 0x7fab580b5f50 2020-11-20T12:10:26.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.150+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab580b63d0 con 0x7fab580b5f50 2020-11-20T12:10:26.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.150+0000 7fab4f7fe700 1 -- 172.21.15.187:0/1214747073 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fab4000f030 con 0x7fab580b5f50 2020-11-20T12:10:26.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.151+0000 7fab4f7fe700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab34056d20 0x7fab340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.151+0000 7fab4f7fe700 1 -- 172.21.15.187:0/1214747073 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fab4006fc20 con 0x7fab580b5f50 2020-11-20T12:10:26.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.151+0000 7fab56d9d700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab34056d20 0x7fab340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:26.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.152+0000 7fab56d9d700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab34056d20 0x7fab340591e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fab48007aa0 tx=0x7fab48008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:26.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.156+0000 7fab4f7fe700 1 -- 172.21.15.187:0/1214747073 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fab40044ff0 con 0x7fab580b5f50 2020-11-20T12:10:26.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.347+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab5813e510 con 0x7fab580b5f50 2020-11-20T12:10:26.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.349+0000 7fab4f7fe700 1 -- 172.21.15.187:0/1214747073 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16781 (secure 0 0 0) 0x7fab40048610 con 0x7fab580b5f50 2020-11-20T12:10:26.355 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:26.355 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":70,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:10:24.317245+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:10:20.847618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2020-11-20T12:10:22.150115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:10:26.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.352+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab34056d20 msgr2=0x7fab340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.352+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab34056d20 0x7fab340591e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fab48007aa0 tx=0x7fab48008040).stop 2020-11-20T12:10:26.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.352+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 msgr2=0x7fab580b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.352+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b2210 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fab400096b0 tx=0x7fab40006d50).stop 2020-11-20T12:10:26.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.352+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 shutdown_connections 2020-11-20T12:10:26.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.352+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fab34056d20 0x7fab340591e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.353+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab580b57b0 0x7fab5813dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.353+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab581482d0 0x7fab580b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:26.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.353+0000 7fab5c89e700 1 --2- 172.21.15.187:0/1214747073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b5f50 0x7fab580b2210 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.353+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 >> 172.21.15.187:0/1214747073 conn(0x7fab580bab30 msgr2=0x7fab58065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:26.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.353+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 shutdown_connections 2020-11-20T12:10:26.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.353+0000 7fab5c89e700 1 -- 172.21.15.187:0/1214747073 wait complete. 2020-11-20T12:10:26.381 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:10:26.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.623+0000 7f6e295fb700 1 Processor -- start 2020-11-20T12:10:26.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.623+0000 7f6e295fb700 1 -- start start 2020-11-20T12:10:26.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.624+0000 7f6e295fb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e240b57b0 0x7f6e24068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.624+0000 7f6e295fb700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6e240b5f50 0x7f6e240b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.624+0000 7f6e295fb700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e24144b80 con 0x7f6e241482d0 2020-11-20T12:10:26.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.624+0000 7f6e295fb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e241489e0 con 0x7f6e240b57b0 2020-11-20T12:10:26.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.624+0000 7f6e295fb700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e24148c80 con 0x7f6e240b5f50 2020-11-20T12:10:26.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.625+0000 7f6e28dfa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6e240b5f50 0x7f6e240b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:26.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.625+0000 7f6e28dfa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6e240b5f50 0x7f6e240b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45608/0 (socket says 172.21.15.187:45608) 2020-11-20T12:10:26.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.625+0000 7f6e28dfa700 1 -- 172.21.15.187:0/345448921 learned_addr learned my addr 172.21.15.187:0/345448921 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:26.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e23fff700 1 --2- 172.21.15.187:0/345448921 >> v2:172.21.15.5:3300/0 conn(0x7f6e240b57b0 0x7f6e24068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:26.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e28dfa700 1 -- 172.21.15.187:0/345448921 >> v2:172.21.15.5:3300/0 conn(0x7f6e240b57b0 msgr2=0x7f6e24068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:26.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e28dfa700 1 --2- 172.21.15.187:0/345448921 >> v2:172.21.15.5:3300/0 conn(0x7f6e240b57b0 0x7f6e24068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e28dfa700 1 -- 172.21.15.187:0/345448921 >> v1:172.21.15.5:6789/0 conn(0x7f6e241482d0 legacy=0x7f6e24062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:26.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e28dfa700 1 -- 172.21.15.187:0/345448921 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24148f20 con 0x7f6e240b5f50 2020-11-20T12:10:26.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e28dfa700 1 --2- 172.21.15.187:0/345448921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b5f50 0x7f6e240b6380 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f6e24066230 tx=0x7f6e1c009290).ready entity=mon.1 client_cookie=efabd5e33622c498 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:26.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e237fe700 1 -- 172.21.15.187:0/345448921 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c00bb80 con 0x7f6e240b5f50 2020-11-20T12:10:26.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e237fe700 1 -- 172.21.15.187:0/345448921 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c00bce0 con 0x7f6e240b5f50 2020-11-20T12:10:26.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.626+0000 7f6e237fe700 1 -- 172.21.15.187:0/345448921 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c011610 con 0x7f6e240b5f50 2020-11-20T12:10:26.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.627+0000 7f6e295fb700 1 -- 172.21.15.187:0/345448921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b5f50 msgr2=0x7f6e240b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.627+0000 7f6e295fb700 1 --2- 172.21.15.187:0/345448921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b5f50 0x7f6e240b6380 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f6e24066230 tx=0x7f6e1c009290).stop 2020-11-20T12:10:26.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.627+0000 7f6e295fb700 1 -- 172.21.15.187:0/345448921 shutdown_connections 2020-11-20T12:10:26.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.627+0000 7f6e295fb700 1 --2- 172.21.15.187:0/345448921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b5f50 0x7f6e240b6380 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.627+0000 7f6e295fb700 1 --2- 172.21.15.187:0/345448921 >> v2:172.21.15.5:3300/0 conn(0x7f6e240b57b0 0x7f6e24068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.627+0000 7f6e295fb700 1 -- 172.21.15.187:0/345448921 >> 172.21.15.187:0/345448921 conn(0x7f6e240bab30 msgr2=0x7f6e240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:26.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.627+0000 7f6e295fb700 1 -- 172.21.15.187:0/345448921 shutdown_connections 2020-11-20T12:10:26.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.627+0000 7f6e295fb700 1 -- 172.21.15.187:0/345448921 wait complete. 2020-11-20T12:10:26.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.629+0000 7f6e295fb700 1 Processor -- start 2020-11-20T12:10:26.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.629+0000 7f6e295fb700 1 -- start start 2020-11-20T12:10:26.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e295fb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b57b0 0x7f6e240b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e295fb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e240b5f50 0x7f6e240a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e295fb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e241482d0 0x7f6e2415b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e295fb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240a0970 con 0x7f6e241482d0 2020-11-20T12:10:26.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e295fb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24147940 con 0x7f6e240b57b0 2020-11-20T12:10:26.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e295fb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24147be0 con 0x7f6e240b5f50 2020-11-20T12:10:26.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e23fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b57b0 0x7f6e240b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:26.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e23fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b57b0 0x7f6e240b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45612/0 (socket says 172.21.15.187:45612) 2020-11-20T12:10:26.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e23fff700 1 -- 172.21.15.187:0/2387102604 learned_addr learned my addr 172.21.15.187:0/2387102604 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:26.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e23fff700 1 -- 172.21.15.187:0/2387102604 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e240b5f50 msgr2=0x7f6e240a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e23fff700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e240b5f50 0x7f6e240a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:26.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e23fff700 1 -- 172.21.15.187:0/2387102604 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e241482d0 msgr2=0x7f6e2415b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e23fff700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e241482d0 0x7f6e2415b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:26.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.631+0000 7f6e23fff700 1 -- 172.21.15.187:0/2387102604 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24145020 con 0x7f6e240b57b0 2020-11-20T12:10:26.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.632+0000 7f6e23fff700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b57b0 0x7f6e240b2f60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f6e1400c920 tx=0x7f6e14008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:26.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.632+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2387102604 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14007710 con 0x7f6e240b57b0 2020-11-20T12:10:26.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.632+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2387102604 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e14007870 con 0x7f6e240b57b0 2020-11-20T12:10:26.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.632+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2387102604 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14007c10 con 0x7f6e240b57b0 2020-11-20T12:10:26.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.632+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2415c390 con 0x7f6e240b57b0 2020-11-20T12:10:26.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.632+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e24145680 con 0x7f6e240b57b0 2020-11-20T12:10:26.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.633+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e241451b0 con 0x7f6e240b57b0 2020-11-20T12:10:26.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.634+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2387102604 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6e1401b460 con 0x7f6e240b57b0 2020-11-20T12:10:26.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.634+0000 7f6e20ff9700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e00056d20 0x7f6e000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:26.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.635+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2387102604 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6e140701c0 con 0x7f6e240b57b0 2020-11-20T12:10:26.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.635+0000 7f6e1bfff700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e00056d20 0x7f6e000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:26.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.635+0000 7f6e1bfff700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e00056d20 0x7f6e000591e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f6e0c006fd0 tx=0x7f6e0c006930).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:26.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.639+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2387102604 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6e14045530 con 0x7f6e240b57b0 2020-11-20T12:10:26.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.894+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f6e24061a20 con 0x7f6e240b57b0 2020-11-20T12:10:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.906+0000 7f6e20ff9700 1 -- 172.21.15.187:0/2387102604 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6e14048b50 con 0x7f6e240b57b0 2020-11-20T12:10:26.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.908+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e00056d20 msgr2=0x7f6e000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.908+0000 7f6e295fb700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e00056d20 0x7f6e000591e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f6e0c006fd0 tx=0x7f6e0c006930).stop 2020-11-20T12:10:26.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.908+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b57b0 msgr2=0x7f6e240b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:26.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.908+0000 7f6e295fb700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b57b0 0x7f6e240b2f60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f6e1400c920 tx=0x7f6e14008040).stop 2020-11-20T12:10:26.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.909+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 shutdown_connections 2020-11-20T12:10:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.909+0000 7f6e295fb700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e00056d20 0x7f6e000591e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.909+0000 7f6e295fb700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e241482d0 0x7f6e2415b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.909+0000 7f6e295fb700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e240b57b0 0x7f6e240b2f60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.909+0000 7f6e295fb700 1 --2- 172.21.15.187:0/2387102604 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e240b5f50 0x7f6e240a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:26.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.909+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 >> 172.21.15.187:0/2387102604 conn(0x7f6e240bab30 msgr2=0x7f6e240b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:26.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.909+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 shutdown_connections 2020-11-20T12:10:26.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:26.909+0000 7f6e295fb700 1 -- 172.21.15.187:0/2387102604 wait complete. 2020-11-20T12:10:26.917 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:10:26.935 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:27.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.166+0000 7fee672e8700 1 Processor -- start 2020-11-20T12:10:27.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.166+0000 7fee672e8700 1 -- start start 2020-11-20T12:10:27.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.168+0000 7fee672e8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fee60066f70 0x7fee600b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:27.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.168+0000 7fee672e8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee600b1300 con 0x7fee600b6650 2020-11-20T12:10:27.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.168+0000 7fee672e8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee600b15a0 con 0x7fee601484d0 2020-11-20T12:10:27.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.168+0000 7fee672e8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee60067730 con 0x7fee60066f70 2020-11-20T12:10:27.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.168+0000 7fee662e6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fee601484d0 0x7fee60061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42226/0 (socket says 172.21.15.187:42226) 2020-11-20T12:10:27.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.168+0000 7fee662e6700 1 -- 172.21.15.187:0/2319722711 learned_addr learned my addr 172.21.15.187:0/2319722711 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:27.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341969913 0 0) 0x7fee600b15a0 con 0x7fee601484d0 2020-11-20T12:10:27.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee48003240 con 0x7fee601484d0 2020-11-20T12:10:27.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee66ae7700 1 --2- 172.21.15.187:0/2319722711 >> v2:172.21.15.5:3300/0 conn(0x7fee60066f70 0x7fee600b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:27.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1849115885 0 0) 0x7fee48003240 con 0x7fee601484d0 2020-11-20T12:10:27.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 >> v2:172.21.15.5:3300/0 conn(0x7fee60066f70 msgr2=0x7fee600b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:27.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 --2- 172.21.15.187:0/2319722711 >> v2:172.21.15.5:3300/0 conn(0x7fee60066f70 0x7fee600b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 >> v1:172.21.15.5:6789/0 conn(0x7fee600b6650 legacy=0x7fee600afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:27.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee600b1840 con 0x7fee601484d0 2020-11-20T12:10:27.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fee5c0014e0 con 0x7fee601484d0 2020-11-20T12:10:27.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.169+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 >> v1:172.21.15.187:6789/0 conn(0x7fee601484d0 legacy=0x7fee60061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:27.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.170+0000 7fee652e4700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee48003730 0x7fee48003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:27.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.170+0000 7fee652e4700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee480044e0 0x7fee48004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:27.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.170+0000 7fee652e4700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee48004e50 0x7fee4800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:27.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.170+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee60067730 con 0x7fee48004e50 2020-11-20T12:10:27.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.170+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee48004260 con 0x7fee480044e0 2020-11-20T12:10:27.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.170+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4800bc20 con 0x7fee48003730 2020-11-20T12:10:27.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.170+0000 7fee66ae7700 1 --2- 172.21.15.187:0/2319722711 >> v2:172.21.15.5:3300/0 conn(0x7fee60066f70 0x7fee600b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:27.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.170+0000 7fee65ae5700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee480044e0 0x7fee48004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:27.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee66ae7700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee48004e50 0x7fee4800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:27.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee662e6700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee48003730 0x7fee48003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:27.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee65ae5700 1 -- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee48003730 msgr2=0x7fee48003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:27.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee65ae5700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee48003730 0x7fee48003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee65ae5700 1 -- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee48004e50 msgr2=0x7fee4800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:27.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee65ae5700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee48004e50 0x7fee4800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee65ae5700 1 -- 172.21.15.187:0/2319722711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee48003240 con 0x7fee480044e0 2020-11-20T12:10:27.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee66ae7700 1 -- 172.21.15.187:0/2319722711 reap_dead start 2020-11-20T12:10:27.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee662e6700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee48003730 0x7fee48003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:27.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.171+0000 7fee65ae5700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee480044e0 0x7fee48004960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7fee50001920 tx=0x7fee50007b90).ready entity=mon.1 client_cookie=56b735aa516d2255 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:27.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.172+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50007020 con 0x7fee480044e0 2020-11-20T12:10:27.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.172+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee50007180 con 0x7fee480044e0 2020-11-20T12:10:27.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.172+0000 7fee652e4700 1 -- 172.21.15.187:0/2319722711 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5000c830 con 0x7fee480044e0 2020-11-20T12:10:27.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.172+0000 7fee672e8700 1 -- 172.21.15.187:0/2319722711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee480044e0 msgr2=0x7fee48004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:27.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.172+0000 7fee672e8700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee480044e0 0x7fee48004960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7fee50001920 tx=0x7fee50007b90).stop 2020-11-20T12:10:27.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.173+0000 7fee672e8700 1 -- 172.21.15.187:0/2319722711 shutdown_connections 2020-11-20T12:10:27.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.173+0000 7fee672e8700 1 --2- 172.21.15.187:0/2319722711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee480044e0 0x7fee48004960 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.173+0000 7fee672e8700 1 -- 172.21.15.187:0/2319722711 >> 172.21.15.187:0/2319722711 conn(0x7fee600bab30 msgr2=0x7fee600baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:27.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.173+0000 7fee672e8700 1 -- 172.21.15.187:0/2319722711 shutdown_connections 2020-11-20T12:10:27.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.173+0000 7fee672e8700 1 -- 172.21.15.187:0/2319722711 wait complete. 2020-11-20T12:10:27.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.174+0000 7fee672e8700 1 Processor -- start 2020-11-20T12:10:27.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.174+0000 7fee672e8700 1 -- start start 2020-11-20T12:10:27.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.175+0000 7fee672e8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee480044e0 0x7fee6013b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:27.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.175+0000 7fee672e8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6009fd30 0x7fee601386b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:27.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.175+0000 7fee672e8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee600b3420 0x7fee6009ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:27.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.175+0000 7fee672e8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee600a06f0 con 0x7fee600b3420 2020-11-20T12:10:27.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.175+0000 7fee672e8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6009e880 con 0x7fee6009fd30 2020-11-20T12:10:27.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.175+0000 7fee672e8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6009e5e0 con 0x7fee480044e0 2020-11-20T12:10:27.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.175+0000 7fee65ae5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6009fd30 0x7fee601386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:27.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee65ae5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6009fd30 0x7fee601386b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45634/0 (socket says 172.21.15.187:45634) 2020-11-20T12:10:27.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee65ae5700 1 -- 172.21.15.187:0/666212251 learned_addr learned my addr 172.21.15.187:0/666212251 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:27.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee662e6700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee480044e0 0x7fee6013b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:27.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee66ae7700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee600b3420 0x7fee6009ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:27.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee65ae5700 1 -- 172.21.15.187:0/666212251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee480044e0 msgr2=0x7fee6013b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:27.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee65ae5700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee480044e0 0x7fee6013b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee65ae5700 1 -- 172.21.15.187:0/666212251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee600b3420 msgr2=0x7fee6009ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:27.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee65ae5700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee600b3420 0x7fee6009ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee65ae5700 1 -- 172.21.15.187:0/666212251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee600a0370 con 0x7fee6009fd30 2020-11-20T12:10:27.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.176+0000 7fee65ae5700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6009fd30 0x7fee601386b0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7fee50000dd0 tx=0x7fee50015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:27.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.177+0000 7fee4e7fc700 1 -- 172.21.15.187:0/666212251 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50013c80 con 0x7fee6009fd30 2020-11-20T12:10:27.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.177+0000 7fee4e7fc700 1 -- 172.21.15.187:0/666212251 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee50013e00 con 0x7fee6009fd30 2020-11-20T12:10:27.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.177+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6009f760 con 0x7fee6009fd30 2020-11-20T12:10:27.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.177+0000 7fee4e7fc700 1 -- 172.21.15.187:0/666212251 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50020690 con 0x7fee6009fd30 2020-11-20T12:10:27.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.177+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6013ba80 con 0x7fee6009fd30 2020-11-20T12:10:27.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.178+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee600a0560 con 0x7fee6009fd30 2020-11-20T12:10:27.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.178+0000 7fee4e7fc700 1 -- 172.21.15.187:0/666212251 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fee50026020 con 0x7fee6009fd30 2020-11-20T12:10:27.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.179+0000 7fee4e7fc700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee38056d20 0x7fee380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:27.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.179+0000 7fee4e7fc700 1 -- 172.21.15.187:0/666212251 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fee50071600 con 0x7fee6009fd30 2020-11-20T12:10:27.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.180+0000 7fee662e6700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee38056d20 0x7fee380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:27.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.180+0000 7fee662e6700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee38056d20 0x7fee380591e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fee5c000dd0 tx=0x7fee5c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:27.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.183+0000 7fee4e7fc700 1 -- 172.21.15.187:0/666212251 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fee50046a50 con 0x7fee6009fd30 2020-11-20T12:10:27.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.419+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee6009f950 con 0x7fee6009fd30 2020-11-20T12:10:27.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.420+0000 7fee4e7fc700 1 -- 172.21.15.187:0/666212251 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fee5004a070 con 0x7fee6009fd30 2020-11-20T12:10:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:27.425 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2299534653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2299534653},{"type":"v1","addr":"172.21.15.5:6833","nonce":2299534653}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:25.326709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6400":{"gid":6400,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/1383561284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1383561284},{"type":"v1","addr":"172.21.15.187:6835","nonce":1383561284}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:27.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee38056d20 msgr2=0x7fee380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:27.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee38056d20 0x7fee380591e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fee5c000dd0 tx=0x7fee5c009040).stop 2020-11-20T12:10:27.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6009fd30 msgr2=0x7fee601386b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:27.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6009fd30 0x7fee601386b0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7fee50000dd0 tx=0x7fee50015000).stop 2020-11-20T12:10:27.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 shutdown_connections 2020-11-20T12:10:27.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee38056d20 0x7fee380591e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fee600b3420 0x7fee6009ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fee480044e0 0x7fee6013b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 --2- 172.21.15.187:0/666212251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6009fd30 0x7fee601386b0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:27.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 >> 172.21.15.187:0/666212251 conn(0x7fee600bab30 msgr2=0x7fee6009c680 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:27.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 shutdown_connections 2020-11-20T12:10:27.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:27.423+0000 7fee672e8700 1 -- 172.21.15.187:0/666212251 wait complete. 2020-11-20T12:10:27.435 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 48 2020-11-20T12:10:27.649 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:10:23.312121+0000', 'modified': '2020-11-20T12:10:25.326709+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6400}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6400': {'gid': 6400, 'name': 'a', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.187:6835/1383561284', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1383561284}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1383561284}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:10:27.650 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:10:27.650 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:10:27.926 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6400,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.5853995110000003,"mdsmap_epoch":48,"osdmap_epoch":70,"osdmap_epoch_barrier":70,"uptime":17.342916419000002} 2020-11-20T12:10:27.928 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6400, "mdsmap_epoch": 48, "osdmap_epoch": 70, "osdmap_epoch_barrier": 70, "rank_uptime": 4.585399511, "state": "up:active", "uptime": 17.342916419, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:10:27.928 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:10:27.929 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:10:27.929 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:10:27.929 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:10:27.930 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:10:27.930 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:10:27.930 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:10:27.931 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:10:27.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:27.932 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:10:27.988 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:10:27.988 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:10:27.988 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:10:27.989 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:27.989 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:10:27.989 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:27.989 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:10:27.990 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:27.990 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:10:27.990 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:27.990 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:10:27.991 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:27.991 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:10:27.991 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:27.991 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:10:27.991 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42410sec preferred_lft 42410sec 2020-11-20T12:10:27.992 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:10:27.992 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:27.992 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:10:27.993 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:27.993 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:10:27.994 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:27.994 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:10:27.994 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:27.994 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:10:27.995 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:10:27.995 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:10:27.995 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:27.995 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:10:27.996 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:10:27.996 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:10:27.996 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:27.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:27.998 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:10:28.097 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:10:28.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:28.102 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:10:28.160 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:10:28.160 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:10:28.161 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:10:28.161 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:10:28.161 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:10:28.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:28.163 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:10:28.163 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:10:28.163 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:10:28.164 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:10:28.164 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:10:28.165 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:10:28.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:28.434 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:10:28.489 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:10:28.489 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:10:28.490 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:10:28.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:28.491 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:10:28.596 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:10:28.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.840+0000 7f313bfff700 1 Processor -- start 2020-11-20T12:10:28.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.840+0000 7f313bfff700 1 -- start start 2020-11-20T12:10:28.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.841+0000 7f313bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f313c1484d0 0x7f313c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.841+0000 7f313bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c0622c0 0x7f313c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.841+0000 7f313bfff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313c0b1300 con 0x7f313c066f70 2020-11-20T12:10:28.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.841+0000 7f313bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313c067730 con 0x7f313c1484d0 2020-11-20T12:10:28.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.841+0000 7f313bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c0b15a0 con 0x7f313c0622c0 2020-11-20T12:10:28.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.842+0000 7f313b7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f313c066f70 0x7f313c0b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42248/0 (socket says 172.21.15.187:42248) 2020-11-20T12:10:28.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.842+0000 7f313b7fe700 1 -- 172.21.15.187:0/945029113 learned_addr learned my addr 172.21.15.187:0/945029113 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:28.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.842+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773353974 0 0) 0x7f313c0b1300 con 0x7f313c066f70 2020-11-20T12:10:28.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.842+0000 7f313affd700 1 --2- 172.21.15.187:0/945029113 >> v2:172.21.15.5:3300/0 conn(0x7f313c1484d0 0x7f313c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f311c003240 con 0x7f313c066f70 2020-11-20T12:10:28.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f31327fc700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c0622c0 0x7f313c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2808341183 0 0) 0x7f311c003240 con 0x7f313c066f70 2020-11-20T12:10:28.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c0622c0 msgr2=0x7f313c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:28.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c0622c0 0x7f313c0b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:28.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 >> v2:172.21.15.5:3300/0 conn(0x7f313c1484d0 msgr2=0x7f313c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:28.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 --2- 172.21.15.187:0/945029113 >> v2:172.21.15.5:3300/0 conn(0x7f313c1484d0 0x7f313c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:28.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c0b1840 con 0x7f313c066f70 2020-11-20T12:10:28.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f3134001860 con 0x7f313c066f70 2020-11-20T12:10:28.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 >> v1:172.21.15.187:6789/0 conn(0x7f313c066f70 legacy=0x7f313c0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:28.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.843+0000 7f313a7fc700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f311c003730 0x7f311c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.844+0000 7f313a7fc700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c0622c0 0x7f311c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.844+0000 7f313a7fc700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f311c0049d0 0x7f311c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.844+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c067730 con 0x7f311c0049d0 2020-11-20T12:10:28.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.844+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c0b15a0 con 0x7f311c003730 2020-11-20T12:10:28.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.844+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0041d0 con 0x7f313c0622c0 2020-11-20T12:10:28.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.844+0000 7f313affd700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f311c003730 0x7f311c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.844+0000 7f31327fc700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c0622c0 0x7f311c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313affd700 1 -- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c0622c0 msgr2=0x7f311c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:28.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313affd700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c0622c0 0x7f311c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:28.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313b7fe700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f311c0049d0 0x7f311c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313affd700 1 -- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f311c0049d0 msgr2=0x7f311c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:28.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313affd700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f311c0049d0 0x7f311c004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:28.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313affd700 1 -- 172.21.15.187:0/945029113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c003240 con 0x7f311c003730 2020-11-20T12:10:28.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313affd700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f311c003730 0x7f311c003b50 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f312c00ba80 tx=0x7f312c008040).ready entity=mon.1 client_cookie=57a607c94bfbbd40 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:28.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c00d9c0 con 0x7f311c003730 2020-11-20T12:10:28.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c00db20 con 0x7f311c003730 2020-11-20T12:10:28.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.845+0000 7f313a7fc700 1 -- 172.21.15.187:0/945029113 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c00dea0 con 0x7f311c003730 2020-11-20T12:10:28.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.846+0000 7f313bfff700 1 -- 172.21.15.187:0/945029113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f311c003730 msgr2=0x7f311c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:28.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.846+0000 7f313bfff700 1 --2- 172.21.15.187:0/945029113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f311c003730 0x7f311c003b50 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f312c00ba80 tx=0x7f312c008040).stop 2020-11-20T12:10:28.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.846+0000 7f313b7fe700 1 -- 172.21.15.187:0/945029113 reap_dead start 2020-11-20T12:10:28.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.846+0000 7f313bfff700 1 -- 172.21.15.187:0/945029113 shutdown_connections 2020-11-20T12:10:28.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.846+0000 7f313bfff700 1 -- 172.21.15.187:0/945029113 >> 172.21.15.187:0/945029113 conn(0x7f313c0bab30 msgr2=0x7f313c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:28.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.846+0000 7f313bfff700 1 -- 172.21.15.187:0/945029113 shutdown_connections 2020-11-20T12:10:28.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.846+0000 7f313bfff700 1 -- 172.21.15.187:0/945029113 wait complete. 2020-11-20T12:10:28.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.848+0000 7f313bfff700 1 Processor -- start 2020-11-20T12:10:28.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.848+0000 7f313bfff700 1 -- start start 2020-11-20T12:10:28.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c158330 0x7f313c0b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c158710 0x7f313c098e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c099380 0x7f313c143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c09cd30 con 0x7f313c158330 2020-11-20T12:10:28.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c158f20 con 0x7f313c099380 2020-11-20T12:10:28.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c09c5f0 con 0x7f313c158710 2020-11-20T12:10:28.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c099380 0x7f313c143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c099380 0x7f313c143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45656/0 (socket says 172.21.15.187:45656) 2020-11-20T12:10:28.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313b7fe700 1 -- 172.21.15.187:0/3460892314 learned_addr learned my addr 172.21.15.187:0/3460892314 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:28.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.849+0000 7f313affd700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c158330 0x7f313c0b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f313b7fe700 1 -- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c158710 msgr2=0x7f313c098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:28.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f31327fc700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c158710 0x7f313c098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f313b7fe700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c158710 0x7f313c098e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:28.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f313b7fe700 1 -- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c158330 msgr2=0x7f313c0b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:28.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f313b7fe700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c158330 0x7f313c0b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f313b7fe700 1 -- 172.21.15.187:0/3460892314 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c09c890 con 0x7f313c099380 2020-11-20T12:10:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f313affd700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c158330 0x7f313c0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f31327fc700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c158710 0x7f313c098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f313b7fe700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c099380 0x7f313c143780 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f313c066380 tx=0x7f3134009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f31337fe700 1 -- 172.21.15.187:0/3460892314 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313400bbf0 con 0x7f313c099380 2020-11-20T12:10:28.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f31337fe700 1 -- 172.21.15.187:0/3460892314 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313400bd50 con 0x7f313c099380 2020-11-20T12:10:28.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f31337fe700 1 -- 172.21.15.187:0/3460892314 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134004b70 con 0x7f313c099380 2020-11-20T12:10:28.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.850+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313c158b90 con 0x7f313c099380 2020-11-20T12:10:28.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.851+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313c09d290 con 0x7f313c099380 2020-11-20T12:10:28.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.852+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313c0620d0 con 0x7f313c099380 2020-11-20T12:10:28.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.853+0000 7f31337fe700 1 -- 172.21.15.187:0/3460892314 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f313401e070 con 0x7f313c099380 2020-11-20T12:10:28.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.853+0000 7f31337fe700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3118056d50 0x7f3118059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:28.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.853+0000 7f31337fe700 1 -- 172.21.15.187:0/3460892314 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f313406f680 con 0x7f313c099380 2020-11-20T12:10:28.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.854+0000 7f313affd700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3118056d50 0x7f3118059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:28.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.855+0000 7f313affd700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3118056d50 0x7f3118059210 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f313c05ea30 tx=0x7f312c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:28.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:28.858+0000 7f31337fe700 1 -- 172.21.15.187:0/3460892314 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f31340449f0 con 0x7f313c099380 2020-11-20T12:10:29.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.043+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f313c158d80 con 0x7f313c099380 2020-11-20T12:10:29.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.044+0000 7f31337fe700 1 -- 172.21.15.187:0/3460892314 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f313c158d80 con 0x7f313c099380 2020-11-20T12:10:29.049 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:29.050 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":70,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:10:24.317245+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:10:20.847618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2020-11-20T12:10:22.150115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:10:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.046+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3118056d50 msgr2=0x7f3118059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.046+0000 7f313bfff700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3118056d50 0x7f3118059210 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f313c05ea30 tx=0x7f312c008040).stop 2020-11-20T12:10:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c099380 msgr2=0x7f313c143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c099380 0x7f313c143780 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f313c066380 tx=0x7f3134009040).stop 2020-11-20T12:10:29.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 shutdown_connections 2020-11-20T12:10:29.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3118056d50 0x7f3118059210 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c158330 0x7f313c0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c099380 0x7f313c143780 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 --2- 172.21.15.187:0/3460892314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c158710 0x7f313c098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 >> 172.21.15.187:0/3460892314 conn(0x7f313c0bab30 msgr2=0x7f313c0b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:29.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 shutdown_connections 2020-11-20T12:10:29.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.047+0000 7f313bfff700 1 -- 172.21.15.187:0/3460892314 wait complete. 2020-11-20T12:10:29.073 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:10:20.847618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:10:22.150115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:10:29.074 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:10:29.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.315+0000 7f5e348f3700 1 Processor -- start 2020-11-20T12:10:29.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.315+0000 7f5e348f3700 1 -- start start 2020-11-20T12:10:29.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.317+0000 7f5e348f3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5e30065f90 0x7f5e30065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.317+0000 7f5e348f3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5e3009ca90 0x7f5e3009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.317+0000 7f5e348f3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e300b13e0 con 0x7f5e30065190 2020-11-20T12:10:29.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.317+0000 7f5e348f3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3014ab30 con 0x7f5e3009ca90 2020-11-20T12:10:29.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.317+0000 7f5e348f3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e30062c10 con 0x7f5e30065f90 2020-11-20T12:10:29.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.317+0000 7f5e27fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5e30065f90 0x7f5e30065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.317+0000 7f5e2ed9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5e30065190 0x7f5e30143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42266/0 (socket says 172.21.15.187:42266) 2020-11-20T12:10:29.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.317+0000 7f5e2ed9d700 1 -- 172.21.15.187:0/2889779885 learned_addr learned my addr 172.21.15.187:0/2889779885 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:29.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.318+0000 7f5e2f59e700 1 --2- 172.21.15.187:0/2889779885 >> v2:172.21.15.5:3300/0 conn(0x7f5e3009ca90 0x7f5e3009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.318+0000 7f5e27fff700 1 -- 172.21.15.187:0/2889779885 >> v2:172.21.15.5:3300/0 conn(0x7f5e3009ca90 msgr2=0x7f5e3009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:29.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.318+0000 7f5e27fff700 1 --2- 172.21.15.187:0/2889779885 >> v2:172.21.15.5:3300/0 conn(0x7f5e3009ca90 0x7f5e3009cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.318+0000 7f5e27fff700 1 -- 172.21.15.187:0/2889779885 >> v1:172.21.15.187:6789/0 conn(0x7f5e30065190 legacy=0x7f5e30143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.318+0000 7f5e27fff700 1 -- 172.21.15.187:0/2889779885 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e300b1640 con 0x7f5e30065f90 2020-11-20T12:10:29.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.318+0000 7f5e2e59c700 1 -- 172.21.15.187:0/2889779885 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510258994 0 0) 0x7f5e300b13e0 con 0x7f5e30065190 2020-11-20T12:10:29.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.318+0000 7f5e27fff700 1 --2- 172.21.15.187:0/2889779885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065f90 0x7f5e30065570 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f5e300b5fb0 tx=0x7f5e18006df0).ready entity=mon.1 client_cookie=6bcf76cc53c4b372 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:29.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.318+0000 7f5e2e59c700 1 -- 172.21.15.187:0/2889779885 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e18009840 con 0x7f5e30065f90 2020-11-20T12:10:29.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e2e59c700 1 -- 172.21.15.187:0/2889779885 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e180099a0 con 0x7f5e30065f90 2020-11-20T12:10:29.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e2e59c700 1 -- 172.21.15.187:0/2889779885 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e18009d20 con 0x7f5e30065f90 2020-11-20T12:10:29.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e348f3700 1 -- 172.21.15.187:0/2889779885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065f90 msgr2=0x7f5e30065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e348f3700 1 --2- 172.21.15.187:0/2889779885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065f90 0x7f5e30065570 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f5e300b5fb0 tx=0x7f5e18006df0).stop 2020-11-20T12:10:29.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e348f3700 1 -- 172.21.15.187:0/2889779885 shutdown_connections 2020-11-20T12:10:29.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e348f3700 1 --2- 172.21.15.187:0/2889779885 >> v2:172.21.15.5:3300/0 conn(0x7f5e3009ca90 0x7f5e3009cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e348f3700 1 --2- 172.21.15.187:0/2889779885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065f90 0x7f5e30065570 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e348f3700 1 -- 172.21.15.187:0/2889779885 >> 172.21.15.187:0/2889779885 conn(0x7f5e300bab30 msgr2=0x7f5e300baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:29.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e348f3700 1 -- 172.21.15.187:0/2889779885 shutdown_connections 2020-11-20T12:10:29.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.319+0000 7f5e348f3700 1 -- 172.21.15.187:0/2889779885 wait complete. 2020-11-20T12:10:29.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.321+0000 7f5e348f3700 1 Processor -- start 2020-11-20T12:10:29.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.321+0000 7f5e348f3700 1 -- start start 2020-11-20T12:10:29.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e348f3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065190 0x7f5e30145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e348f3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e30065f90 0x7f5e3009e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e348f3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e3009ca90 0x7f5e30149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e348f3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e300a07f0 con 0x7f5e3009ca90 2020-11-20T12:10:29.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e348f3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3009e340 con 0x7f5e30065190 2020-11-20T12:10:29.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e348f3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e30145c70 con 0x7f5e30065f90 2020-11-20T12:10:29.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e2ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065190 0x7f5e30145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e2ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065190 0x7f5e30145780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45666/0 (socket says 172.21.15.187:45666) 2020-11-20T12:10:29.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.322+0000 7f5e2ed9d700 1 -- 172.21.15.187:0/3793513812 learned_addr learned my addr 172.21.15.187:0/3793513812 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:29.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e27fff700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e30065f90 0x7f5e3009e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e2f59e700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e3009ca90 0x7f5e30149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e2ed9d700 1 -- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e30065f90 msgr2=0x7f5e3009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e2ed9d700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e30065f90 0x7f5e3009e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e2ed9d700 1 -- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e3009ca90 msgr2=0x7f5e30149740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e2ed9d700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e3009ca90 0x7f5e30149740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e2ed9d700 1 -- 172.21.15.187:0/3793513812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e300b3dc0 con 0x7f5e30065190 2020-11-20T12:10:29.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e2f59e700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e3009ca90 0x7f5e30149740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:29.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e2ed9d700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065190 0x7f5e30145780 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f5e2000c760 tx=0x7f5e20007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:29.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e26ffd700 1 -- 172.21.15.187:0/3793513812 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20007320 con 0x7f5e30065190 2020-11-20T12:10:29.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e26ffd700 1 -- 172.21.15.187:0/3793513812 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e20007480 con 0x7f5e30065190 2020-11-20T12:10:29.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3009ecd0 con 0x7f5e30065190 2020-11-20T12:10:29.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.323+0000 7f5e26ffd700 1 -- 172.21.15.187:0/3793513812 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20005820 con 0x7f5e30065190 2020-11-20T12:10:29.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.324+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e300a04a0 con 0x7f5e30065190 2020-11-20T12:10:29.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.325+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e30065690 con 0x7f5e30065190 2020-11-20T12:10:29.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.325+0000 7f5e26ffd700 1 -- 172.21.15.187:0/3793513812 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5e20010030 con 0x7f5e30065190 2020-11-20T12:10:29.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.326+0000 7f5e26ffd700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e0c056d20 0x7f5e0c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.326+0000 7f5e26ffd700 1 -- 172.21.15.187:0/3793513812 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5e2006fd30 con 0x7f5e30065190 2020-11-20T12:10:29.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.326+0000 7f5e27fff700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e0c056d20 0x7f5e0c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.327+0000 7f5e27fff700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e0c056d20 0x7f5e0c0591e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f5e300b5fb0 tx=0x7f5e18015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:29.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.331+0000 7f5e26ffd700 1 -- 172.21.15.187:0/3793513812 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5e20045200 con 0x7f5e30065190 2020-11-20T12:10:29.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.525+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5e30061a20 con 0x7f5e30065190 2020-11-20T12:10:29.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.526+0000 7f5e26ffd700 1 -- 172.21.15.187:0/3793513812 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v70) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5e20048820 con 0x7f5e30065190 2020-11-20T12:10:29.532 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:10:29.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.528+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e0c056d20 msgr2=0x7f5e0c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.528+0000 7f5e348f3700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e0c056d20 0x7f5e0c0591e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f5e300b5fb0 tx=0x7f5e18015000).stop 2020-11-20T12:10:29.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.528+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065190 msgr2=0x7f5e30145780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.528+0000 7f5e348f3700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065190 0x7f5e30145780 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f5e2000c760 tx=0x7f5e20007a60).stop 2020-11-20T12:10:29.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.529+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 shutdown_connections 2020-11-20T12:10:29.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.529+0000 7f5e348f3700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5e0c056d20 0x7f5e0c0591e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.529+0000 7f5e348f3700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e3009ca90 0x7f5e30149740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.529+0000 7f5e348f3700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e30065190 0x7f5e30145780 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.529+0000 7f5e348f3700 1 --2- 172.21.15.187:0/3793513812 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e30065f90 0x7f5e3009e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.529+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 >> 172.21.15.187:0/3793513812 conn(0x7f5e300bab30 msgr2=0x7f5e300b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:29.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.529+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 shutdown_connections 2020-11-20T12:10:29.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.529+0000 7f5e348f3700 1 -- 172.21.15.187:0/3793513812 wait complete. 2020-11-20T12:10:29.574 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:10:29.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.836+0000 7fd03ccb3700 1 Processor -- start 2020-11-20T12:10:29.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.836+0000 7fd03ccb3700 1 -- start start 2020-11-20T12:10:29.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.837+0000 7fd03ccb3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b1380 0x7fd038150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.837+0000 7fd03ccb3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0380658a0 con 0x7fd03814c260 2020-11-20T12:10:29.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.837+0000 7fd03ccb3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd038150390 con 0x7fd0380b1760 2020-11-20T12:10:29.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.837+0000 7fd03ccb3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd038150630 con 0x7fd0380b1380 2020-11-20T12:10:29.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.838+0000 7fd036ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd0380b1760 0x7fd038065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42282/0 (socket says 172.21.15.187:42282) 2020-11-20T12:10:29.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.838+0000 7fd036ffd700 1 -- 172.21.15.187:0/2825857846 learned_addr learned my addr 172.21.15.187:0/2825857846 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:29.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.838+0000 7fd0377fe700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b1380 0x7fd038150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.838+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199137771 0 0) 0x7fd038150390 con 0x7fd0380b1760 2020-11-20T12:10:29.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.838+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd024003240 con 0x7fd0380b1760 2020-11-20T12:10:29.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3993880217 0 0) 0x7fd024003240 con 0x7fd0380b1760 2020-11-20T12:10:29.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b1380 msgr2=0x7fd038150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:29.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b1380 0x7fd038150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 >> v1:172.21.15.5:6789/0 conn(0x7fd03814c260 legacy=0x7fd03814c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd038068510 con 0x7fd0380b1760 2020-11-20T12:10:29.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd02c0014e0 con 0x7fd0380b1760 2020-11-20T12:10:29.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 >> v1:172.21.15.187:6789/0 conn(0x7fd0380b1760 legacy=0x7fd038065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd024003730 0x7fd024003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0240044e0 0x7fd024004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b1380 0x7fd024004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd038150630 con 0x7fd0240044e0 2020-11-20T12:10:29.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd024005390 con 0x7fd024003730 2020-11-20T12:10:29.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0240042f0 con 0x7fd0380b1380 2020-11-20T12:10:29.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.839+0000 7fd0377fe700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd024003730 0x7fd024003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd036ffd700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0240044e0 0x7fd024004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd0377fe700 1 -- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b1380 msgr2=0x7fd024004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:29.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd0377fe700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b1380 0x7fd024004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:29.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd0377fe700 1 -- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0240044e0 msgr2=0x7fd024004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:29.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd0377fe700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0240044e0 0x7fd024004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd0377fe700 1 -- 172.21.15.187:0/2825857846 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd024003240 con 0x7fd024003730 2020-11-20T12:10:29.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd036ffd700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0240044e0 0x7fd024004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:29.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd0377fe700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd024003730 0x7fd024003b50 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7fd02800bc00 tx=0x7fd028006df0).ready entity=mon.1 client_cookie=e19d244220fdfaa3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:29.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.840+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02800b310 con 0x7fd024003730 2020-11-20T12:10:29.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02800eae0 con 0x7fd024003730 2020-11-20T12:10:29.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd0367fc700 1 -- 172.21.15.187:0/2825857846 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02800ee60 con 0x7fd024003730 2020-11-20T12:10:29.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2825857846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd024003730 msgr2=0x7fd024003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd03ccb3700 1 --2- 172.21.15.187:0/2825857846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd024003730 0x7fd024003b50 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7fd02800bc00 tx=0x7fd028006df0).stop 2020-11-20T12:10:29.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd037fff700 1 -- 172.21.15.187:0/2825857846 reap_dead start 2020-11-20T12:10:29.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2825857846 shutdown_connections 2020-11-20T12:10:29.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2825857846 >> 172.21.15.187:0/2825857846 conn(0x7fd0380bab30 msgr2=0x7fd0380baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:29.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2825857846 shutdown_connections 2020-11-20T12:10:29.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.841+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2825857846 wait complete. 2020-11-20T12:10:29.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.843+0000 7fd03ccb3700 1 Processor -- start 2020-11-20T12:10:29.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.843+0000 7fd03ccb3700 1 -- start start 2020-11-20T12:10:29.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd03ccb3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b7ab0 0x7fd03815c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd03ccb3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0380b7e90 0x7fd0380b6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd03ccb3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd03809d320 0x7fd03809f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd03ccb3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380b62d0 con 0x7fd0380b7e90 2020-11-20T12:10:29.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd03ccb3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380b6570 con 0x7fd03809d320 2020-11-20T12:10:29.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd03ccb3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380b6810 con 0x7fd0380b7ab0 2020-11-20T12:10:29.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd037fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd03809d320 0x7fd03809f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd037fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd03809d320 0x7fd03809f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45690/0 (socket says 172.21.15.187:45690) 2020-11-20T12:10:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd037fff700 1 -- 172.21.15.187:0/2489685193 learned_addr learned my addr 172.21.15.187:0/2489685193 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.844+0000 7fd036ffd700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0380b7e90 0x7fd0380b6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd0377fe700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b7ab0 0x7fd03815c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd037fff700 1 -- 172.21.15.187:0/2489685193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b7ab0 msgr2=0x7fd03815c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd037fff700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b7ab0 0x7fd03815c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd037fff700 1 -- 172.21.15.187:0/2489685193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0380b7e90 msgr2=0x7fd0380b6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:29.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd037fff700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0380b7e90 0x7fd0380b6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:29.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd037fff700 1 -- 172.21.15.187:0/2489685193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0380a0800 con 0x7fd03809d320 2020-11-20T12:10:29.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd037fff700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd03809d320 0x7fd03809f920 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7fd03814c640 tx=0x7fd030007cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:29.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd0177fe700 1 -- 172.21.15.187:0/2489685193 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd030007e70 con 0x7fd03809d320 2020-11-20T12:10:29.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd0177fe700 1 -- 172.21.15.187:0/2489685193 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd030005a70 con 0x7fd03809d320 2020-11-20T12:10:29.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0380b71a0 con 0x7fd03809d320 2020-11-20T12:10:29.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.845+0000 7fd0177fe700 1 -- 172.21.15.187:0/2489685193 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd030005e30 con 0x7fd03809d320 2020-11-20T12:10:29.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.846+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03809ffd0 con 0x7fd03809d320 2020-11-20T12:10:29.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.847+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0380b69d0 con 0x7fd03809d320 2020-11-20T12:10:29.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.847+0000 7fd0177fe700 1 -- 172.21.15.187:0/2489685193 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd03001c070 con 0x7fd03809d320 2020-11-20T12:10:29.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.848+0000 7fd0177fe700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd010056d20 0x7fd0100591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:29.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.848+0000 7fd0177fe700 1 -- 172.21.15.187:0/2489685193 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd03006f4b0 con 0x7fd03809d320 2020-11-20T12:10:29.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.848+0000 7fd0377fe700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd010056d20 0x7fd0100591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:29.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.849+0000 7fd0377fe700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd010056d20 0x7fd0100591e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fd038063360 tx=0x7fd028019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:29.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:29.852+0000 7fd0177fe700 1 -- 172.21.15.187:0/2489685193 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd0300449b0 con 0x7fd03809d320 2020-11-20T12:10:30.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.042+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd0380a0250 con 0x7fd03809d320 2020-11-20T12:10:30.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.043+0000 7fd0177fe700 1 -- 172.21.15.187:0/2489685193 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v70) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd030047fd0 con 0x7fd03809d320 2020-11-20T12:10:30.047 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:30.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd010056d20 msgr2=0x7fd0100591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd010056d20 0x7fd0100591e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fd038063360 tx=0x7fd028019000).stop 2020-11-20T12:10:30.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd03809d320 msgr2=0x7fd03809f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd03809d320 0x7fd03809f920 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7fd03814c640 tx=0x7fd030007cb0).stop 2020-11-20T12:10:30.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 shutdown_connections 2020-11-20T12:10:30.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd010056d20 0x7fd0100591e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd03809d320 0x7fd03809f920 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0380b7ab0 0x7fd03815c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 --2- 172.21.15.187:0/2489685193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0380b7e90 0x7fd0380b6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 >> 172.21.15.187:0/2489685193 conn(0x7fd0380bab30 msgr2=0x7fd0380b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:30.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 shutdown_connections 2020-11-20T12:10:30.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.045+0000 7fd03ccb3700 1 -- 172.21.15.187:0/2489685193 wait complete. 2020-11-20T12:10:30.071 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:10:30.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.320+0000 7fcd22c16700 1 Processor -- start 2020-11-20T12:10:30.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.321+0000 7fcd22c16700 1 -- start start 2020-11-20T12:10:30.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.322+0000 7fcd22c16700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c14da70 0x7fcd1c0ba5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.322+0000 7fcd22c16700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c14def0 con 0x7fcd1c0ba210 2020-11-20T12:10:30.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.322+0000 7fcd22c16700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c14d090 con 0x7fcd1c0b6cf0 2020-11-20T12:10:30.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.322+0000 7fcd22c16700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0baae0 con 0x7fcd1c14da70 2020-11-20T12:10:30.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.322+0000 7fcd211b3700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcd1c0b6cf0 0x7fcd1c099ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42304/0 (socket says 172.21.15.187:42304) 2020-11-20T12:10:30.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.322+0000 7fcd211b3700 1 -- 172.21.15.187:0/2333408191 learned_addr learned my addr 172.21.15.187:0/2333408191 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:30.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd13fff700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c14da70 0x7fcd1c0ba5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951097917 0 0) 0x7fcd1c14d090 con 0x7fcd1c0b6cf0 2020-11-20T12:10:30.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd08003240 con 0x7fcd1c0b6cf0 2020-11-20T12:10:30.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2401939160 0 0) 0x7fcd08003240 con 0x7fcd1c0b6cf0 2020-11-20T12:10:30.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c14da70 msgr2=0x7fcd1c0ba5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:30.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c14da70 0x7fcd1c0ba5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 >> v1:172.21.15.5:6789/0 conn(0x7fcd1c0ba210 legacy=0x7fcd1c065300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c14d330 con 0x7fcd1c0b6cf0 2020-11-20T12:10:30.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcd140014e0 con 0x7fcd1c0b6cf0 2020-11-20T12:10:30.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 >> v1:172.21.15.187:6789/0 conn(0x7fcd1c0b6cf0 legacy=0x7fcd1c099ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd08003730 0x7fcd08003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.323+0000 7fcd137fe700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c14da70 0x7fcd080044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd137fe700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd080049d0 0x7fcd08004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0baae0 con 0x7fcd080049d0 2020-11-20T12:10:30.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08004220 con 0x7fcd08003730 2020-11-20T12:10:30.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0800b730 con 0x7fcd1c14da70 2020-11-20T12:10:30.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd209b2700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd08003730 0x7fcd08003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd13fff700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c14da70 0x7fcd080044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd211b3700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd080049d0 0x7fcd08004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd209b2700 1 -- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c14da70 msgr2=0x7fcd080044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:30.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd209b2700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c14da70 0x7fcd080044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd209b2700 1 -- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd080049d0 msgr2=0x7fcd08004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:30.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.324+0000 7fcd209b2700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd080049d0 0x7fcd08004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.325+0000 7fcd209b2700 1 -- 172.21.15.187:0/2333408191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08003240 con 0x7fcd08003730 2020-11-20T12:10:30.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.325+0000 7fcd211b3700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd080049d0 0x7fcd08004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:30.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.325+0000 7fcd209b2700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd08003730 0x7fcd08003b50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7fcd18001ea0 tx=0x7fcd18007f10).ready entity=mon.1 client_cookie=7cec8aae27592a54 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:30.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.325+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18001540 con 0x7fcd08003730 2020-11-20T12:10:30.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.325+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1800cb80 con 0x7fcd08003730 2020-11-20T12:10:30.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.325+0000 7fcd137fe700 1 -- 172.21.15.187:0/2333408191 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18006870 con 0x7fcd08003730 2020-11-20T12:10:30.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.326+0000 7fcd22c16700 1 -- 172.21.15.187:0/2333408191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd08003730 msgr2=0x7fcd08003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.326+0000 7fcd22c16700 1 --2- 172.21.15.187:0/2333408191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd08003730 0x7fcd08003b50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7fcd18001ea0 tx=0x7fcd18007f10).stop 2020-11-20T12:10:30.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.326+0000 7fcd211b3700 1 -- 172.21.15.187:0/2333408191 reap_dead start 2020-11-20T12:10:30.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.326+0000 7fcd22c16700 1 -- 172.21.15.187:0/2333408191 shutdown_connections 2020-11-20T12:10:30.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.326+0000 7fcd22c16700 1 -- 172.21.15.187:0/2333408191 >> 172.21.15.187:0/2333408191 conn(0x7fcd1c0bacf0 msgr2=0x7fcd1c0bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:30.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.326+0000 7fcd22c16700 1 -- 172.21.15.187:0/2333408191 shutdown_connections 2020-11-20T12:10:30.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.326+0000 7fcd22c16700 1 -- 172.21.15.187:0/2333408191 wait complete. 2020-11-20T12:10:30.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.327+0000 7fcd22c16700 1 Processor -- start 2020-11-20T12:10:30.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.327+0000 7fcd22c16700 1 -- start start 2020-11-20T12:10:30.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.328+0000 7fcd22c16700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd1c1494f0 0x7fcd1c09c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.328+0000 7fcd22c16700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c09b370 0x7fcd1c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.328+0000 7fcd22c16700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd1c160b50 0x7fcd1c09dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.328+0000 7fcd22c16700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c15fed0 con 0x7fcd1c160b50 2020-11-20T12:10:30.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.328+0000 7fcd22c16700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c160170 con 0x7fcd1c1494f0 2020-11-20T12:10:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.328+0000 7fcd22c16700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c160410 con 0x7fcd1c09b370 2020-11-20T12:10:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd1c1494f0 0x7fcd1c09c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd1c1494f0 0x7fcd1c09c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45706/0 (socket says 172.21.15.187:45706) 2020-11-20T12:10:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 -- 172.21.15.187:0/895568142 learned_addr learned my addr 172.21.15.187:0/895568142 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 -- 172.21.15.187:0/895568142 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c09b370 msgr2=0x7fcd1c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c09b370 0x7fcd1c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 -- 172.21.15.187:0/895568142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd1c160b50 msgr2=0x7fcd1c09dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd1c160b50 0x7fcd1c09dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 -- 172.21.15.187:0/895568142 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c1606b0 con 0x7fcd1c1494f0 2020-11-20T12:10:30.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd209b2700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd1c1494f0 0x7fcd1c09c990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7fcd18001e70 tx=0x7fcd18015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:30.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd10ff9700 1 -- 172.21.15.187:0/895568142 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18014ea0 con 0x7fcd1c1494f0 2020-11-20T12:10:30.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd10ff9700 1 -- 172.21.15.187:0/895568142 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd180125d0 con 0x7fcd1c1494f0 2020-11-20T12:10:30.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c1498d0 con 0x7fcd1c1494f0 2020-11-20T12:10:30.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.329+0000 7fcd10ff9700 1 -- 172.21.15.187:0/895568142 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18012950 con 0x7fcd1c1494f0 2020-11-20T12:10:30.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.330+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c0b3e00 con 0x7fcd1c1494f0 2020-11-20T12:10:30.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.331+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c160840 con 0x7fcd1c1494f0 2020-11-20T12:10:30.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.331+0000 7fcd10ff9700 1 -- 172.21.15.187:0/895568142 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcd18026030 con 0x7fcd1c1494f0 2020-11-20T12:10:30.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.332+0000 7fcd10ff9700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fccf4056d20 0x7fccf40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.332+0000 7fcd10ff9700 1 -- 172.21.15.187:0/895568142 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcd18071330 con 0x7fcd1c1494f0 2020-11-20T12:10:30.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.332+0000 7fcd13fff700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fccf4056d20 0x7fccf40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.333+0000 7fcd13fff700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fccf4056d20 0x7fccf40591e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fcd1c0b45f0 tx=0x7fcd0c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:30.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.336+0000 7fcd10ff9700 1 -- 172.21.15.187:0/895568142 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcd18046830 con 0x7fcd1c1494f0 2020-11-20T12:10:30.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.522+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd1c14def0 con 0x7fcd1c1494f0 2020-11-20T12:10:30.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.523+0000 7fcd10ff9700 1 -- 172.21.15.187:0/895568142 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcd18024020 con 0x7fcd1c1494f0 2020-11-20T12:10:30.529 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:30.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.525+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fccf4056d20 msgr2=0x7fccf40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.525+0000 7fcd22c16700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fccf4056d20 0x7fccf40591e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fcd1c0b45f0 tx=0x7fcd0c008040).stop 2020-11-20T12:10:30.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.525+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd1c1494f0 msgr2=0x7fcd1c09c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.525+0000 7fcd22c16700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd1c1494f0 0x7fcd1c09c990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7fcd18001e70 tx=0x7fcd18015000).stop 2020-11-20T12:10:30.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.526+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 shutdown_connections 2020-11-20T12:10:30.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.526+0000 7fcd22c16700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fccf4056d20 0x7fccf40591e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.526+0000 7fcd22c16700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcd1c160b50 0x7fcd1c09dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:30.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.526+0000 7fcd22c16700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd1c1494f0 0x7fcd1c09c990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.526+0000 7fcd22c16700 1 --2- 172.21.15.187:0/895568142 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcd1c09b370 0x7fcd1c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:30.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.526+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 >> 172.21.15.187:0/895568142 conn(0x7fcd1c0bacf0 msgr2=0x7fcd1c0b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:30.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.526+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 shutdown_connections 2020-11-20T12:10:30.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.526+0000 7fcd22c16700 1 -- 172.21.15.187:0/895568142 wait complete. 2020-11-20T12:10:30.552 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:10:30.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.800+0000 7fd8db59e700 1 Processor -- start 2020-11-20T12:10:30.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.800+0000 7fd8db59e700 1 -- start start 2020-11-20T12:10:30.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.801+0000 7fd8db59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8dc0b6650 0x7fd8dc0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.801+0000 7fd8db59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc066f70 0x7fd8dc0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.801+0000 7fd8db59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc0b1300 con 0x7fd8dc1484d0 2020-11-20T12:10:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.801+0000 7fd8db59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8dc067730 con 0x7fd8dc0b6650 2020-11-20T12:10:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.801+0000 7fd8db59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0b15a0 con 0x7fd8dc066f70 2020-11-20T12:10:30.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.802+0000 7fd8d9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8dc0b6650 0x7fd8dc0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.802+0000 7fd8d9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8dc0b6650 0x7fd8dc0afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45716/0 (socket says 172.21.15.187:45716) 2020-11-20T12:10:30.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.802+0000 7fd8d9d9b700 1 -- 172.21.15.187:0/1613036857 learned_addr learned my addr 172.21.15.187:0/1613036857 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:30.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.802+0000 7fd8d9d9b700 1 -- 172.21.15.187:0/1613036857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc066f70 msgr2=0x7fd8dc0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.802+0000 7fd8d9d9b700 1 --2- 172.21.15.187:0/1613036857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc066f70 0x7fd8dc0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.802+0000 7fd8d9d9b700 1 -- 172.21.15.187:0/1613036857 >> v1:172.21.15.5:6789/0 conn(0x7fd8dc1484d0 legacy=0x7fd8dc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.802+0000 7fd8d9d9b700 1 -- 172.21.15.187:0/1613036857 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc0b1840 con 0x7fd8dc0b6650 2020-11-20T12:10:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.803+0000 7fd8d9d9b700 1 --2- 172.21.15.187:0/1613036857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc0b6650 0x7fd8dc0afa90 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7fd8d0006fd0 tx=0x7fd8d0006eb0).ready entity=mon.1 client_cookie=e3bb0f878d088f66 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:30.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.803+0000 7fd8d959a700 1 -- 172.21.15.187:0/1613036857 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0009890 con 0x7fd8dc0b6650 2020-11-20T12:10:30.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.803+0000 7fd8d959a700 1 -- 172.21.15.187:0/1613036857 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d00099f0 con 0x7fd8dc0b6650 2020-11-20T12:10:30.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.803+0000 7fd8d959a700 1 -- 172.21.15.187:0/1613036857 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0009d70 con 0x7fd8dc0b6650 2020-11-20T12:10:30.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.804+0000 7fd8db59e700 1 -- 172.21.15.187:0/1613036857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc0b6650 msgr2=0x7fd8dc0afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.804+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1613036857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc0b6650 0x7fd8dc0afa90 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7fd8d0006fd0 tx=0x7fd8d0006eb0).stop 2020-11-20T12:10:30.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.804+0000 7fd8db59e700 1 -- 172.21.15.187:0/1613036857 shutdown_connections 2020-11-20T12:10:30.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.804+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1613036857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc066f70 0x7fd8dc0aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:30.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.804+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1613036857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc0b6650 0x7fd8dc0afa90 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.804+0000 7fd8db59e700 1 -- 172.21.15.187:0/1613036857 >> 172.21.15.187:0/1613036857 conn(0x7fd8dc0bab30 msgr2=0x7fd8dc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:30.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.804+0000 7fd8db59e700 1 -- 172.21.15.187:0/1613036857 shutdown_connections 2020-11-20T12:10:30.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.804+0000 7fd8db59e700 1 -- 172.21.15.187:0/1613036857 wait complete. 2020-11-20T12:10:30.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.806+0000 7fd8db59e700 1 Processor -- start 2020-11-20T12:10:30.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.806+0000 7fd8db59e700 1 -- start start 2020-11-20T12:10:30.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8db59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8dc066f70 0x7fd8dc09d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8db59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc0b6650 0x7fd8dc15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8db59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc1484d0 0x7fd8dc09bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8db59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc09d620 con 0x7fd8dc066f70 2020-11-20T12:10:30.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8db59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc09b390 con 0x7fd8dc1484d0 2020-11-20T12:10:30.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8db59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc09b630 con 0x7fd8dc0b6650 2020-11-20T12:10:30.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8dad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc1484d0 0x7fd8dc09bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8dad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc1484d0 0x7fd8dc09bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45724/0 (socket says 172.21.15.187:45724) 2020-11-20T12:10:30.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.807+0000 7fd8dad9d700 1 -- 172.21.15.187:0/1673174887 learned_addr learned my addr 172.21.15.187:0/1673174887 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:30.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.808+0000 7fd8d9d9b700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc0b6650 0x7fd8dc15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.808+0000 7fd8dad9d700 1 -- 172.21.15.187:0/1673174887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc0b6650 msgr2=0x7fd8dc15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.808+0000 7fd8da59c700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8dc066f70 0x7fd8dc09d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.808+0000 7fd8dad9d700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc0b6650 0x7fd8dc15b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.808+0000 7fd8dad9d700 1 -- 172.21.15.187:0/1673174887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8dc066f70 msgr2=0x7fd8dc09d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:30.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.808+0000 7fd8dad9d700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8dc066f70 0x7fd8dc09d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:30.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.808+0000 7fd8dad9d700 1 -- 172.21.15.187:0/1673174887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc09b8d0 con 0x7fd8dc1484d0 2020-11-20T12:10:30.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.808+0000 7fd8dad9d700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc1484d0 0x7fd8dc09bd70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7fd8d400ba30 tx=0x7fd8d4009220).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:30.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.809+0000 7fd8ca7fc700 1 -- 172.21.15.187:0/1673174887 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d400c8b0 con 0x7fd8dc1484d0 2020-11-20T12:10:30.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.809+0000 7fd8ca7fc700 1 -- 172.21.15.187:0/1673174887 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d400ca10 con 0x7fd8dc1484d0 2020-11-20T12:10:30.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.809+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc15bde0 con 0x7fd8dc1484d0 2020-11-20T12:10:30.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.809+0000 7fd8ca7fc700 1 -- 172.21.15.187:0/1673174887 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d400cbc0 con 0x7fd8dc1484d0 2020-11-20T12:10:30.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.809+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc09c4a0 con 0x7fd8dc1484d0 2020-11-20T12:10:30.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.810+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc09ba60 con 0x7fd8dc1484d0 2020-11-20T12:10:30.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.810+0000 7fd8ca7fc700 1 -- 172.21.15.187:0/1673174887 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd8d401c070 con 0x7fd8dc1484d0 2020-11-20T12:10:30.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.811+0000 7fd8ca7fc700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d50 0x7fd8b0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:30.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.811+0000 7fd8ca7fc700 1 -- 172.21.15.187:0/1673174887 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd8d406f7c0 con 0x7fd8dc1484d0 2020-11-20T12:10:30.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.811+0000 7fd8da59c700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d50 0x7fd8b0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:30.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.812+0000 7fd8da59c700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d50 0x7fd8b0059210 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd8cc00a570 tx=0x7fd8cc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:30.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.815+0000 7fd8ca7fc700 1 -- 172.21.15.187:0/1673174887 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd8d4044c10 con 0x7fd8dc1484d0 2020-11-20T12:10:31.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:30.999+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8dc061a20 con 0x7fd8dc1484d0 2020-11-20T12:10:31.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.000+0000 7fd8ca7fc700 1 -- 172.21.15.187:0/1673174887 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd8d401a070 con 0x7fd8dc1484d0 2020-11-20T12:10:31.005 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:31.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d50 msgr2=0x7fd8b0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d50 0x7fd8b0059210 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd8cc00a570 tx=0x7fd8cc008040).stop 2020-11-20T12:10:31.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc1484d0 msgr2=0x7fd8dc09bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc1484d0 0x7fd8dc09bd70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7fd8d400ba30 tx=0x7fd8d4009220).stop 2020-11-20T12:10:31.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 shutdown_connections 2020-11-20T12:10:31.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d50 0x7fd8b0059210 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:31.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8dc066f70 0x7fd8dc09d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:31.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8dc1484d0 0x7fd8dc09bd70 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:31.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 --2- 172.21.15.187:0/1673174887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8dc0b6650 0x7fd8dc15b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:31.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 >> 172.21.15.187:0/1673174887 conn(0x7fd8dc0bab30 msgr2=0x7fd8dc0b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:31.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 shutdown_connections 2020-11-20T12:10:31.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.003+0000 7fd8db59e700 1 -- 172.21.15.187:0/1673174887 wait complete. 2020-11-20T12:10:31.030 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:10:31.030 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:31.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.266+0000 7f6b7ec70700 1 Processor -- start 2020-11-20T12:10:31.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.266+0000 7f6b7ec70700 1 -- start start 2020-11-20T12:10:31.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.268+0000 7f6b7ec70700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6b7814d270 0x7f6b781573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:31.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.268+0000 7f6b7ec70700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b780658a0 con 0x7f6b780b1760 2020-11-20T12:10:31.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.268+0000 7f6b7ec70700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b78150390 con 0x7f6b780b1380 2020-11-20T12:10:31.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.268+0000 7f6b7ec70700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b781511e0 con 0x7f6b7814d270 2020-11-20T12:10:31.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.268+0000 7f6b7d46d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6b780b1760 0x7f6b78065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:35844/0 (socket says 172.21.15.187:35844) 2020-11-20T12:10:31.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.268+0000 7f6b7d46d700 1 -- 172.21.15.187:0/1983953734 learned_addr learned my addr 172.21.15.187:0/1983953734 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:31.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.268+0000 7f6b7dc6e700 1 --1- 172.21.15.187:0/1983953734 >> v1:172.21.15.187:6789/0 conn(0x7f6b780b1380 0x7f6b78150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42336/0 (socket says 172.21.15.187:42336) 2020-11-20T12:10:31.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.268+0000 7f6b7e46f700 1 --2- 172.21.15.187:0/1983953734 >> v2:172.21.15.187:3300/0 conn(0x7f6b7814d270 0x7f6b781573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:31.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.269+0000 7f6b7cc6c700 1 -- 172.21.15.187:0/1983953734 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130521359 0 0) 0x7f6b780658a0 con 0x7f6b780b1760 2020-11-20T12:10:31.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.269+0000 7f6b7cc6c700 1 -- 172.21.15.187:0/1983953734 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b60003240 con 0x7f6b780b1760 2020-11-20T12:10:31.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.270+0000 7f6b7cc6c700 1 -- 172.21.15.187:0/1983953734 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326754711 0 0) 0x7f6b78150390 con 0x7f6b780b1380 2020-11-20T12:10:31.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.270+0000 7f6b7cc6c700 1 -- 172.21.15.187:0/1983953734 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b780658a0 con 0x7f6b780b1380 2020-11-20T12:10:31.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.270+0000 7f6b7e46f700 1 -- 172.21.15.187:0/1983953734 >> v1:172.21.15.187:6789/0 conn(0x7f6b780b1380 legacy=0x7f6b78150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.270+0000 7f6b7e46f700 1 -- 172.21.15.187:0/1983953734 >> v1:172.21.15.5:6789/0 conn(0x7f6b780b1760 legacy=0x7f6b78065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.270+0000 7f6b7e46f700 1 -- 172.21.15.187:0/1983953734 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b78150630 con 0x7f6b7814d270 2020-11-20T12:10:31.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.270+0000 7f6b7e46f700 1 --2- 172.21.15.187:0/1983953734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b781573a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f6b70009c90 tx=0x7f6b700091d0).ready entity=mon.1 client_cookie=16c1eb8bcf98df1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:31.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7cc6c700 1 -- 172.21.15.187:0/1983953734 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7000b9c0 con 0x7f6b7814d270 2020-11-20T12:10:31.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7cc6c700 1 -- 172.21.15.187:0/1983953734 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7000bb20 con 0x7f6b7814d270 2020-11-20T12:10:31.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7cc6c700 1 -- 172.21.15.187:0/1983953734 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7000bea0 con 0x7f6b7814d270 2020-11-20T12:10:31.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1983953734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 msgr2=0x7f6b781573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7ec70700 1 --2- 172.21.15.187:0/1983953734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b781573a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f6b70009c90 tx=0x7f6b700091d0).stop 2020-11-20T12:10:31.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1983953734 shutdown_connections 2020-11-20T12:10:31.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7ec70700 1 --2- 172.21.15.187:0/1983953734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b781573a0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:31.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1983953734 >> 172.21.15.187:0/1983953734 conn(0x7f6b780bab30 msgr2=0x7f6b780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:31.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1983953734 shutdown_connections 2020-11-20T12:10:31.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.271+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1983953734 wait complete. 2020-11-20T12:10:31.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.273+0000 7f6b7ec70700 1 Processor -- start 2020-11-20T12:10:31.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.273+0000 7f6b7ec70700 1 -- start start 2020-11-20T12:10:31.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7ec70700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b780b1380 0x7f6b7809e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:31.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7ec70700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b780b1760 0x7f6b780a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:31.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7ec70700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b7809f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:31.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7ec70700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7809fab0 con 0x7f6b780b1760 2020-11-20T12:10:31.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7ec70700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7809fd50 con 0x7f6b7814d270 2020-11-20T12:10:31.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7ec70700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780a0930 con 0x7f6b780b1380 2020-11-20T12:10:31.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7e46f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b7809f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:31.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7e46f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b7809f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45734/0 (socket says 172.21.15.187:45734) 2020-11-20T12:10:31.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.274+0000 7f6b7e46f700 1 -- 172.21.15.187:0/1198287940 learned_addr learned my addr 172.21.15.187:0/1198287940 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:31.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b7e46f700 1 -- 172.21.15.187:0/1198287940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b780b1380 msgr2=0x7f6b7809e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b7e46f700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b780b1380 0x7f6b7809e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:31.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b7e46f700 1 -- 172.21.15.187:0/1198287940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b780b1760 msgr2=0x7f6b780a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b7e46f700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b780b1760 0x7f6b780a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:31.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b7e46f700 1 -- 172.21.15.187:0/1198287940 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7809cf80 con 0x7f6b7814d270 2020-11-20T12:10:31.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b7e46f700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b7809f170 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f6b70009210 tx=0x7f6b70016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:31.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b69ffb700 1 -- 172.21.15.187:0/1198287940 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70010700 con 0x7f6b7814d270 2020-11-20T12:10:31.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b69ffb700 1 -- 172.21.15.187:0/1198287940 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b70010860 con 0x7f6b7814d270 2020-11-20T12:10:31.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.275+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7809f660 con 0x7f6b7814d270 2020-11-20T12:10:31.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.276+0000 7f6b69ffb700 1 -- 172.21.15.187:0/1198287940 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70010be0 con 0x7f6b7814d270 2020-11-20T12:10:31.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.276+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b78148f60 con 0x7f6b7814d270 2020-11-20T12:10:31.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.277+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7809d110 con 0x7f6b7814d270 2020-11-20T12:10:31.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.277+0000 7f6b69ffb700 1 -- 172.21.15.187:0/1198287940 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6b70017070 con 0x7f6b7814d270 2020-11-20T12:10:31.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.278+0000 7f6b69ffb700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6b50056d50 0x7f6b50059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:31.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.278+0000 7f6b69ffb700 1 -- 172.21.15.187:0/1198287940 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6b700723a0 con 0x7f6b7814d270 2020-11-20T12:10:31.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.278+0000 7f6b7dc6e700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6b50056d50 0x7f6b50059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:31.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.279+0000 7f6b7dc6e700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6b50056d50 0x7f6b50059210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6b6c007fd0 tx=0x7f6b6c007690).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:31.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.282+0000 7f6b69ffb700 1 -- 172.21.15.187:0/1198287940 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6b700476e0 con 0x7f6b7814d270 2020-11-20T12:10:31.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.534+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b7809e6f0 con 0x7f6b7814d270 2020-11-20T12:10:31.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.535+0000 7f6b69ffb700 1 -- 172.21.15.187:0/1198287940 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f6b7809e6f0 con 0x7f6b7814d270 2020-11-20T12:10:31.539 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:31.540 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2299534653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2299534653},{"type":"v1","addr":"172.21.15.5:6833","nonce":2299534653}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:25.326709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6400":{"gid":6400,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/1383561284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1383561284},{"type":"v1","addr":"172.21.15.187:6835","nonce":1383561284}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:31.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.537+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6b50056d50 msgr2=0x7f6b50059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.537+0000 7f6b7ec70700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6b50056d50 0x7f6b50059210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6b6c007fd0 tx=0x7f6b6c007690).stop 2020-11-20T12:10:31.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 msgr2=0x7f6b7809f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:31.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b7809f170 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f6b70009210 tx=0x7f6b70016000).stop 2020-11-20T12:10:31.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 shutdown_connections 2020-11-20T12:10:31.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6b50056d50 0x7f6b50059210 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:31.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b7814d270 0x7f6b7809f170 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:31.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b780b1380 0x7f6b7809e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:31.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 --2- 172.21.15.187:0/1198287940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b780b1760 0x7f6b780a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:31.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 >> 172.21.15.187:0/1198287940 conn(0x7f6b780bab30 msgr2=0x7f6b78065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:31.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 shutdown_connections 2020-11-20T12:10:31.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:31.538+0000 7f6b7ec70700 1 -- 172.21.15.187:0/1198287940 wait complete. 2020-11-20T12:10:31.545 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 48 2020-11-20T12:10:31.768 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:10:23.312121+0000', 'modified': '2020-11-20T12:10:25.326709+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6400}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6400': {'gid': 6400, 'name': 'a', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.187:6835/1383561284', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1383561284}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1383561284}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:10:31.770 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:10:31.770 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:10:32.039 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6400,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":8.6974232889999996,"mdsmap_epoch":48,"osdmap_epoch":70,"osdmap_epoch_barrier":70,"uptime":21.454937700999999} 2020-11-20T12:10:32.040 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6400, "mdsmap_epoch": 48, "osdmap_epoch": 70, "osdmap_epoch_barrier": 70, "rank_uptime": 8.697423289, "state": "up:active", "uptime": 21.454937701, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:10:32.041 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:10:32.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:32.043 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:10:32.095 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:10:32.096 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:10:32.235 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:10:32.235 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:10:32.273 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:10:32.273 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:10:32.285 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:10:32.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:32.286 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:10:32.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:10:32.377 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir dir_1) 2020-11-20T12:10:32.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:10:32.488 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2020-11-20T12:10:32.796 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records in 2020-11-20T12:10:32.796 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records out 2020-11-20T12:10:32.797 INFO:teuthology.orchestra.run.smithi187.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.211602 s, 39.6 MB/s 2020-11-20T12:10:32.804 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:10:32.805 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:10:32.805 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:10:32.805 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:10:32.806 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2020-11-20T12:10:32.806 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:10:32.987 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627777 2020-11-20T12:10:33.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:10:33.174 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir dir_2) 2020-11-20T12:10:33.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:10:33.325 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo ln dir_1/file_a dir_2/file_b) 2020-11-20T12:10:33.449 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:10:33.450 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:10:33.450 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:10:33.451 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:10:33.451 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2020-11-20T12:10:33.451 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:10:33.642 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627777 2020-11-20T12:10:33.878 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:34.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.135+0000 7f13eb59e700 1 Processor -- start 2020-11-20T12:10:34.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.135+0000 7f13eb59e700 1 -- start start 2020-11-20T12:10:34.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.136+0000 7f13eb59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13ec065190 0x7f13ec065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.136+0000 7f13eb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec065f90 0x7f13ec143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.136+0000 7f13eb59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec0b13e0 con 0x7f13ec143c70 2020-11-20T12:10:34.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.136+0000 7f13eb59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ec14ab30 con 0x7f13ec065190 2020-11-20T12:10:34.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.136+0000 7f13eb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec062c10 con 0x7f13ec065f90 2020-11-20T12:10:34.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.137+0000 7f13ea59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13ec065190 0x7f13ec065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.137+0000 7f13ea59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13ec065190 0x7f13ec065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45748/0 (socket says 172.21.15.187:45748) 2020-11-20T12:10:34.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.137+0000 7f13ea59c700 1 -- 172.21.15.187:0/3570874145 learned_addr learned my addr 172.21.15.187:0/3570874145 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:34.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.137+0000 7f13e1d9b700 1 --2- 172.21.15.187:0/3570874145 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec065f90 0x7f13ec143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.138+0000 7f13ea59c700 1 -- 172.21.15.187:0/3570874145 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec065f90 msgr2=0x7f13ec143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:34.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.138+0000 7f13ea59c700 1 --2- 172.21.15.187:0/3570874145 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec065f90 0x7f13ec143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.138+0000 7f13ea59c700 1 -- 172.21.15.187:0/3570874145 >> v1:172.21.15.5:6789/0 conn(0x7f13ec143c70 legacy=0x7f13ec09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:34.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.138+0000 7f13ea59c700 1 -- 172.21.15.187:0/3570874145 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec0b1640 con 0x7f13ec065190 2020-11-20T12:10:34.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.139+0000 7f13ea59c700 1 --2- 172.21.15.187:0/3570874145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec065570 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f13ec14ac70 tx=0x7f13dc009290).ready entity=mon.1 client_cookie=6888c7adb5b9341f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:34.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.139+0000 7f13e9d9b700 1 -- 172.21.15.187:0/3570874145 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc00bc10 con 0x7f13ec065190 2020-11-20T12:10:34.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.139+0000 7f13e9d9b700 1 -- 172.21.15.187:0/3570874145 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc00bd70 con 0x7f13ec065190 2020-11-20T12:10:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.139+0000 7f13e9d9b700 1 -- 172.21.15.187:0/3570874145 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc011610 con 0x7f13ec065190 2020-11-20T12:10:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.140+0000 7f13eb59e700 1 -- 172.21.15.187:0/3570874145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 msgr2=0x7f13ec065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.140+0000 7f13eb59e700 1 --2- 172.21.15.187:0/3570874145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec065570 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f13ec14ac70 tx=0x7f13dc009290).stop 2020-11-20T12:10:34.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.140+0000 7f13eb59e700 1 -- 172.21.15.187:0/3570874145 shutdown_connections 2020-11-20T12:10:34.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.140+0000 7f13eb59e700 1 --2- 172.21.15.187:0/3570874145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec065570 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.140+0000 7f13eb59e700 1 --2- 172.21.15.187:0/3570874145 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec065f90 0x7f13ec143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.140+0000 7f13eb59e700 1 -- 172.21.15.187:0/3570874145 >> 172.21.15.187:0/3570874145 conn(0x7f13ec0bab30 msgr2=0x7f13ec0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:34.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.140+0000 7f13eb59e700 1 -- 172.21.15.187:0/3570874145 shutdown_connections 2020-11-20T12:10:34.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.140+0000 7f13eb59e700 1 -- 172.21.15.187:0/3570874145 wait complete. 2020-11-20T12:10:34.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.142+0000 7f13eb59e700 1 Processor -- start 2020-11-20T12:10:34.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.142+0000 7f13eb59e700 1 -- start start 2020-11-20T12:10:34.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.143+0000 7f13eb59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec1480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.143+0000 7f13eb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13ec065f90 0x7f13ec0b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.143+0000 7f13eb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec143c70 0x7f13ec148c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.143+0000 7f13eb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec09f870 con 0x7f13ec065f90 2020-11-20T12:10:34.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13eb59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec159220 con 0x7f13ec065190 2020-11-20T12:10:34.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13eb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec09fa60 con 0x7f13ec143c70 2020-11-20T12:10:34.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13ea59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec1480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13ea59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec1480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45754/0 (socket says 172.21.15.187:45754) 2020-11-20T12:10:34.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13ea59c700 1 -- 172.21.15.187:0/632617713 learned_addr learned my addr 172.21.15.187:0/632617713 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:34.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13ea59c700 1 -- 172.21.15.187:0/632617713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec143c70 msgr2=0x7f13ec148c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13ea59c700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec143c70 0x7f13ec148c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:34.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13ea59c700 1 -- 172.21.15.187:0/632617713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13ec065f90 msgr2=0x7f13ec0b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13ea59c700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13ec065f90 0x7f13ec0b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:34.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.144+0000 7f13ea59c700 1 -- 172.21.15.187:0/632617713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec158ed0 con 0x7f13ec065190 2020-11-20T12:10:34.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.145+0000 7f13ea59c700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec1480f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f13dc0090f0 tx=0x7f13dc016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:34.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.145+0000 7f13e2ffd700 1 -- 172.21.15.187:0/632617713 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc020070 con 0x7f13ec065190 2020-11-20T12:10:34.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.145+0000 7f13e2ffd700 1 -- 172.21.15.187:0/632617713 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc010700 con 0x7f13ec065190 2020-11-20T12:10:34.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.145+0000 7f13e2ffd700 1 -- 172.21.15.187:0/632617713 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc010aa0 con 0x7f13ec065190 2020-11-20T12:10:34.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.145+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec1485e0 con 0x7f13ec065190 2020-11-20T12:10:34.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.145+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec09dae0 con 0x7f13ec065190 2020-11-20T12:10:34.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.146+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec1593e0 con 0x7f13ec065190 2020-11-20T12:10:34.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.147+0000 7f13e2ffd700 1 -- 172.21.15.187:0/632617713 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f13dc017070 con 0x7f13ec065190 2020-11-20T12:10:34.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.147+0000 7f13e2ffd700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13c8056d20 0x7f13c80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.148+0000 7f13e2ffd700 1 -- 172.21.15.187:0/632617713 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13dc071cc0 con 0x7f13ec065190 2020-11-20T12:10:34.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.148+0000 7f13e1d9b700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13c8056d20 0x7f13c80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.149+0000 7f13e1d9b700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13c8056d20 0x7f13c80591e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f13ec09e060 tx=0x7f13d4006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:34.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.152+0000 7f13e2ffd700 1 -- 172.21.15.187:0/632617713 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f13dc0470f0 con 0x7f13ec065190 2020-11-20T12:10:34.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.397+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13ec09fc20 con 0x7f13ec065190 2020-11-20T12:10:34.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.398+0000 7f13e2ffd700 1 -- 172.21.15.187:0/632617713 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f13dc015070 con 0x7f13ec065190 2020-11-20T12:10:34.403 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:34.403 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2299534653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2299534653},{"type":"v1","addr":"172.21.15.5:6833","nonce":2299534653}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:25.326709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6400":{"gid":6400,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/1383561284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1383561284},{"type":"v1","addr":"172.21.15.187:6835","nonce":1383561284}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:34.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13c8056d20 msgr2=0x7f13c80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13c8056d20 0x7f13c80591e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f13ec09e060 tx=0x7f13d4006df0).stop 2020-11-20T12:10:34.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 msgr2=0x7f13ec1480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec1480f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f13dc0090f0 tx=0x7f13dc016000).stop 2020-11-20T12:10:34.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 shutdown_connections 2020-11-20T12:10:34.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13c8056d20 0x7f13c80591e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13ec143c70 0x7f13ec148c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:34.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13ec065190 0x7f13ec1480f0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 --2- 172.21.15.187:0/632617713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13ec065f90 0x7f13ec0b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:34.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 >> 172.21.15.187:0/632617713 conn(0x7f13ec0bab30 msgr2=0x7f13ec145e50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:34.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 shutdown_connections 2020-11-20T12:10:34.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.401+0000 7f13eb59e700 1 -- 172.21.15.187:0/632617713 wait complete. 2020-11-20T12:10:34.410 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 48 2020-11-20T12:10:34.430 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:10:34.731 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:10:34.732 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:10:34.733 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data getxattr 10000000001.00000000 parent 2020-11-20T12:10:34.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.813+0000 7f2289e18d00 1 Processor -- start 2020-11-20T12:10:34.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.813+0000 7f2289e18d00 1 -- start start 2020-11-20T12:10:34.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.814+0000 7f2289e18d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55ea4472e3a0 0x55ea447737c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.814+0000 7f2289e18d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea44755d40 0x55ea447f9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.814+0000 7f2289e18d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea4476fad0 0x55ea447f97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.814+0000 7f2289e18d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea446a9810 con 0x55ea44755d40 2020-11-20T12:10:34.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.814+0000 7f2289e18d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea447f56c0 con 0x55ea4476fad0 2020-11-20T12:10:34.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.814+0000 7f2289e18d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea447f6930 con 0x55ea4472e3a0 2020-11-20T12:10:34.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f227aebe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea4476fad0 0x55ea447f97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f227aebe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea4476fad0 0x55ea447f97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45772/0 (socket says 172.21.15.187:45772) 2020-11-20T12:10:34.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f227aebe700 1 -- 172.21.15.187:0/644627083 learned_addr learned my addr 172.21.15.187:0/644627083 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:34.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f227a6bd700 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55ea4472e3a0 0x55ea447737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f2279ebc700 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea44755d40 0x55ea447f9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f227aebe700 1 -- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55ea4472e3a0 msgr2=0x55ea447737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:34.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f227aebe700 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55ea4472e3a0 0x55ea447737c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f227aebe700 1 -- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea44755d40 msgr2=0x55ea447f9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:34.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.815+0000 7f227aebe700 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea44755d40 0x55ea447f9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.816+0000 7f227aebe700 1 -- 172.21.15.187:0/644627083 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea446d34e0 con 0x55ea4476fad0 2020-11-20T12:10:34.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.816+0000 7f2279ebc700 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea44755d40 0x55ea447f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:34.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.816+0000 7f227aebe700 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea4476fad0 0x55ea447f97f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f2274002b60 tx=0x7f2274006b40).ready entity=mon.1 client_cookie=f9c9a3fd6337f222 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:34.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.816+0000 7f2278eba700 1 -- 172.21.15.187:0/644627083 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22740040d0 con 0x55ea4476fad0 2020-11-20T12:10:34.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.816+0000 7f2278eba700 1 -- 172.21.15.187:0/644627083 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227400b830 con 0x55ea4476fad0 2020-11-20T12:10:34.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.816+0000 7f2278eba700 1 -- 172.21.15.187:0/644627083 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227400bbf0 con 0x55ea4476fad0 2020-11-20T12:10:34.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 -- 172.21.15.187:0/644627083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea4476fad0 msgr2=0x55ea447f97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea4476fad0 0x55ea447f97f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f2274002b60 tx=0x7f2274006b40).stop 2020-11-20T12:10:34.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 -- 172.21.15.187:0/644627083 shutdown_connections 2020-11-20T12:10:34.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea4476fad0 0x55ea447f97f0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55ea4472e3a0 0x55ea447737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 --2- 172.21.15.187:0/644627083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea44755d40 0x55ea447f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 -- 172.21.15.187:0/644627083 >> 172.21.15.187:0/644627083 conn(0x55ea4472d0e0 msgr2=0x55ea44752e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:34.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 -- 172.21.15.187:0/644627083 shutdown_connections 2020-11-20T12:10:34.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.817+0000 7f2289e18d00 1 -- 172.21.15.187:0/644627083 wait complete. 2020-11-20T12:10:34.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.818+0000 7f2289e18d00 1 Processor -- start 2020-11-20T12:10:34.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.818+0000 7f2289e18d00 1 -- start start 2020-11-20T12:10:34.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.819+0000 7f2289e18d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea4472e3a0 0x55ea447fc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.819+0000 7f2289e18d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea44755d40 0x55ea44756620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.819+0000 7f2289e18d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55ea4476fad0 0x55ea44769570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.819+0000 7f2289e18d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea447f2490 con 0x55ea4472e3a0 2020-11-20T12:10:34.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.819+0000 7f2289e18d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea447f21f0 con 0x55ea44755d40 2020-11-20T12:10:34.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.819+0000 7f2289e18d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea4480fb90 con 0x55ea4476fad0 2020-11-20T12:10:34.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.819+0000 7f2279ebc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea44755d40 0x55ea44756620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f2279ebc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea44755d40 0x55ea44756620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45776/0 (socket says 172.21.15.187:45776) 2020-11-20T12:10:34.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f2279ebc700 1 -- 172.21.15.187:0/407605722 learned_addr learned my addr 172.21.15.187:0/407605722 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:34.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f227a6bd700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea4472e3a0 0x55ea447fc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f2279ebc700 1 -- 172.21.15.187:0/407605722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55ea4476fad0 msgr2=0x55ea44769570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f2279ebc700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55ea4476fad0 0x55ea44769570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:34.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f2279ebc700 1 -- 172.21.15.187:0/407605722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea4472e3a0 msgr2=0x55ea447fc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f2279ebc700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55ea4472e3a0 0x55ea447fc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f2279ebc700 1 -- 172.21.15.187:0/407605722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea446ceec0 con 0x55ea44755d40 2020-11-20T12:10:34.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.820+0000 7f2279ebc700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea44755d40 0x55ea44756620 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f2270007b80 tx=0x7f2270005c90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:34.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.821+0000 7f226affd700 1 -- 172.21.15.187:0/407605722 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227000f070 con 0x55ea44755d40 2020-11-20T12:10:34.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.821+0000 7f226affd700 1 -- 172.21.15.187:0/407605722 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22700069b0 con 0x55ea44755d40 2020-11-20T12:10:34.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.821+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ea446ad250 con 0x55ea44755d40 2020-11-20T12:10:34.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.821+0000 7f226affd700 1 -- 172.21.15.187:0/407605722 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2270006d50 con 0x55ea44755d40 2020-11-20T12:10:34.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.821+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea447f5060 con 0x55ea44755d40 2020-11-20T12:10:34.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.822+0000 7f226affd700 1 -- 172.21.15.187:0/407605722 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2270008650 con 0x55ea44755d40 2020-11-20T12:10:34.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.823+0000 7f226affd700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f225c0559a0 0x7f225c057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.823+0000 7f226affd700 1 -- 172.21.15.187:0/407605722 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f227006f080 con 0x55ea44755d40 2020-11-20T12:10:34.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.824+0000 7f227a6bd700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f225c0559a0 0x7f225c057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.824+0000 7f2289e18d00 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55ea44769a60 0x55ea44810030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:34.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.824+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:1 14.7 14:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected e70) v8 -- 0x55ea44810520 con 0x55ea44769a60 2020-11-20T12:10:34.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.824+0000 7f227aebe700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55ea44769a60 0x55ea44810030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:34.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.824+0000 7f227a6bd700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f225c0559a0 0x7f225c057e60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f226c005b10 tx=0x7f226c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:34.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.825+0000 7f227aebe700 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55ea44769a60 0x55ea44810030 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:34.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.825+0000 7f227aebe700 1 -- 172.21.15.187:0/407605722 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv4 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f227400ba90 con 0x55ea44769a60 2020-11-20T12:10:34.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.825+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55ea44769a60 msgr2=0x55ea44810030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55ea44769a60 0x55ea44810030 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:34.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f225c0559a0 msgr2=0x7f225c057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f225c0559a0 0x7f225c057e60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f226c005b10 tx=0x7f226c015000).stop 2020-11-20T12:10:34.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea44755d40 msgr2=0x55ea44756620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:34.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 --2- 172.21.15.187:0/407605722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ea44755d40 0x55ea44756620 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f2270007b80 tx=0x7f2270005c90).stop 2020-11-20T12:10:34.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f227aebe700 1 -- 172.21.15.187:0/407605722 reap_dead start 2020-11-20T12:10:34.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 shutdown_connections 2020-11-20T12:10:34.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 >> 172.21.15.187:0/407605722 conn(0x55ea4472d0e0 msgr2=0x55ea447fad00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:34.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 shutdown_connections 2020-11-20T12:10:34.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:34.826+0000 7f2289e18d00 1 -- 172.21.15.187:0/407605722 wait complete. 2020-11-20T12:10:34.856 INFO:teuthology.orchestra.run.smithi187.stdout:Wfile_adir_1 2020-11-20T12:10:34.857 INFO:teuthology.orchestra.run.smithi187:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2020-11-20T12:10:34.962 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:10:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "ino": 1099511627777, 2020-11-20T12:10:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "ancestors": [ 2020-11-20T12:10:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1099511627776, 2020-11-20T12:10:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "file_a", 2020-11-20T12:10:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 8 2020-11-20T12:10:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:10:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:10:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1, 2020-11-20T12:10:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "dir_1", 2020-11-20T12:10:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 15 2020-11-20T12:10:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:10:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2020-11-20T12:10:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "pool": 14, 2020-11-20T12:10:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "old_pools": [] 2020-11-20T12:10:34.966 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:10:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:34.970 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:10:34.971 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:10:34.971 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:10:35.123 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:10:35.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:35.124 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:10:35.195 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:35.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.452+0000 7fc00f59e700 1 Processor -- start 2020-11-20T12:10:35.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.453+0000 7fc00f59e700 1 -- start start 2020-11-20T12:10:35.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc0100b57b0 0x7fc010068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:35.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0101482d0 0x7fc010062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:35.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc010062b50 0x7fc010065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:35.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc010144b80 con 0x7fc0100b57b0 2020-11-20T12:10:35.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00f59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0101489e0 con 0x7fc0101482d0 2020-11-20T12:10:35.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc010148c80 con 0x7fc010062b50 2020-11-20T12:10:35.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0101482d0 0x7fc010062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:35.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0101482d0 0x7fc010062660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45784/0 (socket says 172.21.15.187:45784) 2020-11-20T12:10:35.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.454+0000 7fc00dd9b700 1 -- 172.21.15.187:0/2482050230 learned_addr learned my addr 172.21.15.187:0/2482050230 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:35.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00ed9d700 1 --2- 172.21.15.187:0/2482050230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc010062b50 0x7fc010065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:35.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00dd9b700 1 -- 172.21.15.187:0/2482050230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc010062b50 msgr2=0x7fc010065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:35.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00e59c700 1 --2- 172.21.15.187:0/2482050230 >> v2:172.21.15.5:3300/0 conn(0x7fc0100b57b0 0x7fc010068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:35.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00dd9b700 1 --2- 172.21.15.187:0/2482050230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc010062b50 0x7fc010065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00dd9b700 1 -- 172.21.15.187:0/2482050230 >> v2:172.21.15.5:3300/0 conn(0x7fc0100b57b0 msgr2=0x7fc010068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:35.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00dd9b700 1 --2- 172.21.15.187:0/2482050230 >> v2:172.21.15.5:3300/0 conn(0x7fc0100b57b0 0x7fc010068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00dd9b700 1 -- 172.21.15.187:0/2482050230 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc010148f20 con 0x7fc0101482d0 2020-11-20T12:10:35.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00ed9d700 1 --2- 172.21.15.187:0/2482050230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc010062b50 0x7fc010065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:35.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00e59c700 1 --2- 172.21.15.187:0/2482050230 >> v2:172.21.15.5:3300/0 conn(0x7fc0100b57b0 0x7fc010068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:35.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.455+0000 7fc00dd9b700 1 --2- 172.21.15.187:0/2482050230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010062660 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fc01009d820 tx=0x7fc004006eb0).ready entity=mon.1 client_cookie=690a26a78eb896e8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:35.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00d59a700 1 -- 172.21.15.187:0/2482050230 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0040098d0 con 0x7fc0101482d0 2020-11-20T12:10:35.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00d59a700 1 -- 172.21.15.187:0/2482050230 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc004009a30 con 0x7fc0101482d0 2020-11-20T12:10:35.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00d59a700 1 -- 172.21.15.187:0/2482050230 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004009db0 con 0x7fc0101482d0 2020-11-20T12:10:35.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00f59e700 1 -- 172.21.15.187:0/2482050230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 msgr2=0x7fc010062660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:35.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2482050230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010062660 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fc01009d820 tx=0x7fc004006eb0).stop 2020-11-20T12:10:35.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00f59e700 1 -- 172.21.15.187:0/2482050230 shutdown_connections 2020-11-20T12:10:35.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2482050230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc010062b50 0x7fc010065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2482050230 >> v2:172.21.15.5:3300/0 conn(0x7fc0100b57b0 0x7fc010068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2482050230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010062660 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.456+0000 7fc00f59e700 1 -- 172.21.15.187:0/2482050230 >> 172.21.15.187:0/2482050230 conn(0x7fc0100bab30 msgr2=0x7fc0100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:35.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.457+0000 7fc00f59e700 1 -- 172.21.15.187:0/2482050230 shutdown_connections 2020-11-20T12:10:35.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.457+0000 7fc00f59e700 1 -- 172.21.15.187:0/2482050230 wait complete. 2020-11-20T12:10:35.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.458+0000 7fc00f59e700 1 Processor -- start 2020-11-20T12:10:35.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.459+0000 7fc00f59e700 1 -- start start 2020-11-20T12:10:35.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc010062b50 0x7fc0101494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:35.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0100b57b0 0x7fc010142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:35.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010143460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:35.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100b1bb0 con 0x7fc010062b50 2020-11-20T12:10:35.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100b1e50 con 0x7fc0101482d0 2020-11-20T12:10:35.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100b20f0 con 0x7fc0100b57b0 2020-11-20T12:10:35.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010143460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:35.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010143460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45794/0 (socket says 172.21.15.187:45794) 2020-11-20T12:10:35.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00ed9d700 1 -- 172.21.15.187:0/2501672034 learned_addr learned my addr 172.21.15.187:0/2501672034 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:35.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.460+0000 7fc00dd9b700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0100b57b0 0x7fc010142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:35.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00e59c700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc010062b50 0x7fc0101494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:35.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00ed9d700 1 -- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0100b57b0 msgr2=0x7fc010142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:35.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00ed9d700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0100b57b0 0x7fc010142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00ed9d700 1 -- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc010062b50 msgr2=0x7fc0101494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:35.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00ed9d700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc010062b50 0x7fc0101494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00ed9d700 1 -- 172.21.15.187:0/2501672034 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100b2390 con 0x7fc0101482d0 2020-11-20T12:10:35.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00e59c700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc010062b50 0x7fc0101494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:35.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00ed9d700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010143460 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7fc00800cb50 tx=0x7fc008008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:35.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fbffe7fc700 1 -- 172.21.15.187:0/2501672034 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008007900 con 0x7fc0101482d0 2020-11-20T12:10:35.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fbffe7fc700 1 -- 172.21.15.187:0/2501672034 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc008007a60 con 0x7fc0101482d0 2020-11-20T12:10:35.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc010149990 con 0x7fc0101482d0 2020-11-20T12:10:35.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fbffe7fc700 1 -- 172.21.15.187:0/2501672034 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008007de0 con 0x7fc0101482d0 2020-11-20T12:10:35.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.461+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0100b4ca0 con 0x7fc0101482d0 2020-11-20T12:10:35.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.462+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0100b2520 con 0x7fc0101482d0 2020-11-20T12:10:35.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.463+0000 7fbffe7fc700 1 -- 172.21.15.187:0/2501672034 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc008011620 con 0x7fc0101482d0 2020-11-20T12:10:35.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.464+0000 7fbffe7fc700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbfe4056d20 0x7fbfe40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:35.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.464+0000 7fbffe7fc700 1 -- 172.21.15.187:0/2501672034 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc008070400 con 0x7fc0101482d0 2020-11-20T12:10:35.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.464+0000 7fc00e59c700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbfe4056d20 0x7fbfe40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:35.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.465+0000 7fc00e59c700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbfe4056d20 0x7fbfe40591e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fc00000b870 tx=0x7fc000008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:35.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.467+0000 7fbffe7fc700 1 -- 172.21.15.187:0/2501672034 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc008045740 con 0x7fc0101482d0 2020-11-20T12:10:35.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.716+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc010061a20 con 0x7fc0101482d0 2020-11-20T12:10:35.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.717+0000 7fbffe7fc700 1 -- 172.21.15.187:0/2501672034 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fc008048d60 con 0x7fc0101482d0 2020-11-20T12:10:35.727 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:35.727 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2299534653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2299534653},{"type":"v1","addr":"172.21.15.5:6833","nonce":2299534653}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:25.326709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6400":{"gid":6400,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/1383561284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1383561284},{"type":"v1","addr":"172.21.15.187:6835","nonce":1383561284}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:35.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.719+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbfe4056d20 msgr2=0x7fbfe40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:35.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.719+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbfe4056d20 0x7fbfe40591e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fc00000b870 tx=0x7fc000008040).stop 2020-11-20T12:10:35.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.719+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 msgr2=0x7fc010143460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:35.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.719+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010143460 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7fc00800cb50 tx=0x7fc008008040).stop 2020-11-20T12:10:35.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.720+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 shutdown_connections 2020-11-20T12:10:35.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.720+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbfe4056d20 0x7fbfe40591e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.720+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc010062b50 0x7fc0101494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.720+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0101482d0 0x7fc010143460 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.720+0000 7fc00f59e700 1 --2- 172.21.15.187:0/2501672034 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0100b57b0 0x7fc010142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:35.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.720+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 >> 172.21.15.187:0/2501672034 conn(0x7fc0100bab30 msgr2=0x7fc0100b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:35.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.720+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 shutdown_connections 2020-11-20T12:10:35.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:35.720+0000 7fc00f59e700 1 -- 172.21.15.187:0/2501672034 wait complete. 2020-11-20T12:10:35.735 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 48 2020-11-20T12:10:35.748 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:10:36.044 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:10:36.045 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:10:36.046 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:10:36.046 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:10:36.046 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:10:36.047 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:10:36.082 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:10:36.082 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2020-11-20T12:10:36.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.374+0000 7fc709938700 1 Processor -- start 2020-11-20T12:10:36.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.374+0000 7fc709938700 1 -- start start 2020-11-20T12:10:36.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.375+0000 7fc709938700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc7040bf800 0x7fc70414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:36.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.375+0000 7fc709938700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70414d110 con 0x7fc70409f560 2020-11-20T12:10:36.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.376+0000 7fc709938700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70414cbd0 con 0x7fc7040bfbe0 2020-11-20T12:10:36.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.376+0000 7fc709938700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70414ce70 con 0x7fc7040bf800 2020-11-20T12:10:36.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.376+0000 7fc708936700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc7040bf800 0x7fc70414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:36.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.376+0000 7fc6fbfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc7040bfbe0 0x7fc7041480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42404/0 (socket says 172.21.15.187:42404) 2020-11-20T12:10:36.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.376+0000 7fc6fbfff700 1 -- 172.21.15.187:0/3095785881 learned_addr learned my addr 172.21.15.187:0/3095785881 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:36.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.377+0000 7fc708936700 1 -- 172.21.15.187:0/3095785881 >> v1:172.21.15.187:6789/0 conn(0x7fc7040bfbe0 legacy=0x7fc7041480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:36.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.377+0000 7fc708936700 1 -- 172.21.15.187:0/3095785881 >> v1:172.21.15.5:6789/0 conn(0x7fc70409f560 legacy=0x7fc7040a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:36.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.377+0000 7fc708936700 1 -- 172.21.15.187:0/3095785881 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040a4cc0 con 0x7fc7040bf800 2020-11-20T12:10:36.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.377+0000 7fc708936700 1 --2- 172.21.15.187:0/3095785881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc70414c2c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7fc700009c90 tx=0x7fc7000091d0).ready entity=mon.1 client_cookie=66221284f5551a74 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:36.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.377+0000 7fc6fb7fe700 1 -- 172.21.15.187:0/3095785881 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70000bc10 con 0x7fc7040bf800 2020-11-20T12:10:36.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.377+0000 7fc6fb7fe700 1 -- 172.21.15.187:0/3095785881 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70000bd70 con 0x7fc7040bf800 2020-11-20T12:10:36.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.377+0000 7fc6fb7fe700 1 -- 172.21.15.187:0/3095785881 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700011610 con 0x7fc7040bf800 2020-11-20T12:10:36.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.378+0000 7fc709938700 1 -- 172.21.15.187:0/3095785881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 msgr2=0x7fc70414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.378+0000 7fc709938700 1 --2- 172.21.15.187:0/3095785881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc70414c2c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7fc700009c90 tx=0x7fc7000091d0).stop 2020-11-20T12:10:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.378+0000 7fc709938700 1 -- 172.21.15.187:0/3095785881 shutdown_connections 2020-11-20T12:10:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.378+0000 7fc709938700 1 --2- 172.21.15.187:0/3095785881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc70414c2c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.378+0000 7fc709938700 1 -- 172.21.15.187:0/3095785881 >> 172.21.15.187:0/3095785881 conn(0x7fc704068370 msgr2=0x7fc7040bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.378+0000 7fc709938700 1 -- 172.21.15.187:0/3095785881 shutdown_connections 2020-11-20T12:10:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.378+0000 7fc709938700 1 -- 172.21.15.187:0/3095785881 wait complete. 2020-11-20T12:10:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.380+0000 7fc709938700 1 Processor -- start 2020-11-20T12:10:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.380+0000 7fc709938700 1 -- start start 2020-11-20T12:10:36.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.381+0000 7fc709938700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc70409f560 0x7fc7040bbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:36.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.381+0000 7fc709938700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc704141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:36.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.381+0000 7fc709938700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc704146950 0x7fc704141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:36.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.381+0000 7fc709938700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70415cd30 con 0x7fc704146950 2020-11-20T12:10:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.381+0000 7fc709938700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc704136ea0 con 0x7fc7040bf800 2020-11-20T12:10:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.381+0000 7fc709938700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70415c350 con 0x7fc70409f560 2020-11-20T12:10:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc6fbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc704141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc6fbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc704141810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45806/0 (socket says 172.21.15.187:45806) 2020-11-20T12:10:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc6fbfff700 1 -- 172.21.15.187:0/1807465571 learned_addr learned my addr 172.21.15.187:0/1807465571 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:36.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc708936700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc70409f560 0x7fc7040bbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc6fbfff700 1 -- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc70409f560 msgr2=0x7fc7040bbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc6fbfff700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc70409f560 0x7fc7040bbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc709137700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc704146950 0x7fc704141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc6fbfff700 1 -- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc704146950 msgr2=0x7fc704141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc6fbfff700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc704146950 0x7fc704141d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc6fbfff700 1 -- 172.21.15.187:0/1807465571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70415c5f0 con 0x7fc7040bf800 2020-11-20T12:10:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.382+0000 7fc708936700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc70409f560 0x7fc7040bbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.383+0000 7fc6fbfff700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc704141810 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7fc6f400c740 tx=0x7fc6f4007ce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.383+0000 7fc6f8ff9700 1 -- 172.21.15.187:0/1807465571 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4007320 con 0x7fc7040bf800 2020-11-20T12:10:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.383+0000 7fc6f8ff9700 1 -- 172.21.15.187:0/1807465571 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4007480 con 0x7fc7040bf800 2020-11-20T12:10:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.383+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70415c890 con 0x7fc7040bf800 2020-11-20T12:10:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.383+0000 7fc6f8ff9700 1 -- 172.21.15.187:0/1807465571 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4005800 con 0x7fc7040bf800 2020-11-20T12:10:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.383+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70415bdd0 con 0x7fc7040bf800 2020-11-20T12:10:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.384+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7040b5010 con 0x7fc7040bf800 2020-11-20T12:10:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.385+0000 7fc6f8ff9700 1 -- 172.21.15.187:0/1807465571 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc6f4010030 con 0x7fc7040bf800 2020-11-20T12:10:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.386+0000 7fc6f8ff9700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.386+0000 7fc6f8ff9700 1 -- 172.21.15.187:0/1807465571 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc6f406fca0 con 0x7fc7040bf800 2020-11-20T12:10:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.386+0000 7fc708936700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:36.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.387+0000 7fc708936700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc700006db0 tx=0x7fc700016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:36.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.390+0000 7fc6f8ff9700 1 -- 172.21.15.187:0/1807465571 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc6f4074070 con 0x7fc7040bf800 2020-11-20T12:10:36.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:36.647+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fc704080050 con 0x7fc7040bf800 2020-11-20T12:10:37.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.137+0000 7fc6f8ff9700 1 -- 172.21.15.187:0/1807465571 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5433 v49) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc6f40450f0 con 0x7fc7040bf800 2020-11-20T12:10:37.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 msgr2=0x7fc6dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:37.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc700006db0 tx=0x7fc700016000).stop 2020-11-20T12:10:37.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 msgr2=0x7fc704141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:37.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc704141810 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7fc6f400c740 tx=0x7fc6f4007ce0).stop 2020-11-20T12:10:37.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 shutdown_connections 2020-11-20T12:10:37.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6dc056d20 0x7fc6dc0591e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:37.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc704146950 0x7fc704141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:37.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc70409f560 0x7fc7040bbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:37.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 --2- 172.21.15.187:0/1807465571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7040bf800 0x7fc704141810 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:37.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 >> 172.21.15.187:0/1807465571 conn(0x7fc704068370 msgr2=0x7fc7040a4010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:37.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.141+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 shutdown_connections 2020-11-20T12:10:37.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:37.142+0000 7fc709938700 1 -- 172.21.15.187:0/1807465571 wait complete. 2020-11-20T12:10:37.149 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 5433 2020-11-20T12:10:37.177 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:10:37.178 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:10:37.186 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:10:37.186 DEBUG:teuthology.parallel:result is None 2020-11-20T12:10:37.390 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:10:38.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2020-11-20T12:10:42.080 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:10:42.081 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2020-11-20T12:10:42.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.335+0000 7f5591212700 1 Processor -- start 2020-11-20T12:10:42.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.335+0000 7f5591212700 1 -- start start 2020-11-20T12:10:42.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.336+0000 7f5591212700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c1484d0 0x7f558c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:42.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.336+0000 7f5591212700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f558c066f70 0x7f558c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:42.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.336+0000 7f5591212700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c0b1300 con 0x7f558c0622c0 2020-11-20T12:10:42.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.336+0000 7f5591212700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558c067730 con 0x7f558c066f70 2020-11-20T12:10:42.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.337+0000 7f5591212700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c0b15a0 con 0x7f558c1484d0 2020-11-20T12:10:42.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.337+0000 7f55837fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f558c0622c0 0x7f558c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42420/0 (socket says 172.21.15.187:42420) 2020-11-20T12:10:42.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.337+0000 7f55837fe700 1 -- 172.21.15.187:0/1075465736 learned_addr learned my addr 172.21.15.187:0/1075465736 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:42.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.337+0000 7f5590a11700 1 --2- 172.21.15.187:0/1075465736 >> v2:172.21.15.187:3300/0 conn(0x7f558c066f70 0x7f558c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:42.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.337+0000 7f558bfff700 1 --2- 172.21.15.187:0/1075465736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c1484d0 0x7f558c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:42.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.338+0000 7f5590a11700 1 -- 172.21.15.187:0/1075465736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c1484d0 msgr2=0x7f558c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:42.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.338+0000 7f5590a11700 1 --2- 172.21.15.187:0/1075465736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c1484d0 0x7f558c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:42.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.338+0000 7f5590a11700 1 -- 172.21.15.187:0/1075465736 >> v1:172.21.15.187:6789/0 conn(0x7f558c0622c0 legacy=0x7f558c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:42.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.338+0000 7f5590a11700 1 -- 172.21.15.187:0/1075465736 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c0b1840 con 0x7f558c066f70 2020-11-20T12:10:42.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.338+0000 7f558b7fe700 1 -- 172.21.15.187:0/1075465736 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324936222 0 0) 0x7f558c0b1300 con 0x7f558c0622c0 2020-11-20T12:10:42.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.338+0000 7f5590a11700 1 --2- 172.21.15.187:0/1075465736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c066f70 0x7f558c0b5d50 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f5584009c90 tx=0x7f55840091d0).ready entity=mon.1 client_cookie=237492d6e6d71ee server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:42.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.338+0000 7f558b7fe700 1 -- 172.21.15.187:0/1075465736 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558400bbc0 con 0x7f558c066f70 2020-11-20T12:10:42.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f558b7fe700 1 -- 172.21.15.187:0/1075465736 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558400bd20 con 0x7f558c066f70 2020-11-20T12:10:42.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f558b7fe700 1 -- 172.21.15.187:0/1075465736 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584011610 con 0x7f558c066f70 2020-11-20T12:10:42.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f5591212700 1 -- 172.21.15.187:0/1075465736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c066f70 msgr2=0x7f558c0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:42.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f5591212700 1 --2- 172.21.15.187:0/1075465736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c066f70 0x7f558c0b5d50 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f5584009c90 tx=0x7f55840091d0).stop 2020-11-20T12:10:42.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f5591212700 1 -- 172.21.15.187:0/1075465736 shutdown_connections 2020-11-20T12:10:42.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f5591212700 1 --2- 172.21.15.187:0/1075465736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c066f70 0x7f558c0b5d50 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:42.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f5591212700 1 --2- 172.21.15.187:0/1075465736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c1484d0 0x7f558c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:42.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f5591212700 1 -- 172.21.15.187:0/1075465736 >> 172.21.15.187:0/1075465736 conn(0x7f558c0bab30 msgr2=0x7f558c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:42.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f5591212700 1 -- 172.21.15.187:0/1075465736 shutdown_connections 2020-11-20T12:10:42.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.339+0000 7f5591212700 1 -- 172.21.15.187:0/1075465736 wait complete. 2020-11-20T12:10:42.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.341+0000 7f5591212700 1 Processor -- start 2020-11-20T12:10:42.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.341+0000 7f5591212700 1 -- start start 2020-11-20T12:10:42.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f5591212700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c0622c0 0x7f558c15bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:42.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f5591212700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c066f70 0x7f558c141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:42.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f5591212700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f558c1484d0 0x7f558c14df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:42.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f5591212700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c15c1e0 con 0x7f558c1484d0 2020-11-20T12:10:42.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f5591212700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c15b310 con 0x7f558c0622c0 2020-11-20T12:10:42.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f5591212700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c15b5b0 con 0x7f558c066f70 2020-11-20T12:10:42.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f558bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c0622c0 0x7f558c15bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:42.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f558bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c0622c0 0x7f558c15bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45818/0 (socket says 172.21.15.187:45818) 2020-11-20T12:10:42.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.342+0000 7f558bfff700 1 -- 172.21.15.187:0/3263492586 learned_addr learned my addr 172.21.15.187:0/3263492586 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:42.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f55837fe700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c066f70 0x7f558c141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:42.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f5590a11700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f558c1484d0 0x7f558c14df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:42.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f558bfff700 1 -- 172.21.15.187:0/3263492586 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c066f70 msgr2=0x7f558c141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:42.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f558bfff700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c066f70 0x7f558c141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:42.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f558bfff700 1 -- 172.21.15.187:0/3263492586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f558c1484d0 msgr2=0x7f558c14df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:42.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f558bfff700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f558c1484d0 0x7f558c14df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:42.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f558bfff700 1 -- 172.21.15.187:0/3263492586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c15b850 con 0x7f558c0622c0 2020-11-20T12:10:42.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f558bfff700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c0622c0 0x7f558c15bcf0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f557c00d8e0 tx=0x7f557c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:42.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f5588ff9700 1 -- 172.21.15.187:0/3263492586 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c00b130 con 0x7f558c0622c0 2020-11-20T12:10:42.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f5588ff9700 1 -- 172.21.15.187:0/3263492586 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c009790 con 0x7f558c0622c0 2020-11-20T12:10:42.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f5588ff9700 1 -- 172.21.15.187:0/3263492586 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c009b10 con 0x7f558c0622c0 2020-11-20T12:10:42.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.343+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c09e700 con 0x7f558c0622c0 2020-11-20T12:10:42.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.344+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c14e3f0 con 0x7f558c0622c0 2020-11-20T12:10:42.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.345+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558c15b9e0 con 0x7f558c0622c0 2020-11-20T12:10:42.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.345+0000 7f5588ff9700 1 -- 172.21.15.187:0/3263492586 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f557c012420 con 0x7f558c0622c0 2020-11-20T12:10:42.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.346+0000 7f5588ff9700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5568056d20 0x7f55680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:42.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.346+0000 7f5588ff9700 1 -- 172.21.15.187:0/3263492586 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f557c06fb70 con 0x7f558c0622c0 2020-11-20T12:10:42.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.346+0000 7f55837fe700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5568056d20 0x7f55680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:42.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.347+0000 7f55837fe700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5568056d20 0x7f55680591e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f558c145600 tx=0x7f5574009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:42.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.350+0000 7f5588ff9700 1 -- 172.21.15.187:0/3263492586 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f557c044f60 con 0x7f558c0622c0 2020-11-20T12:10:42.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:42.603+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f558c061a20 con 0x7f558c0622c0 2020-11-20T12:10:43.431 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:10:43.427+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:10:43.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.436+0000 7f5588ff9700 1 -- 172.21.15.187:0/3263492586 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6400 v51) v1 ==== 95+0+0 (secure 0 0 0) 0x7f557c048580 con 0x7f558c0622c0 2020-11-20T12:10:43.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5568056d20 msgr2=0x7f55680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5568056d20 0x7f55680591e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f558c145600 tx=0x7f5574009040).stop 2020-11-20T12:10:43.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c0622c0 msgr2=0x7f558c15bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c0622c0 0x7f558c15bcf0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f557c00d8e0 tx=0x7f557c008040).stop 2020-11-20T12:10:43.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 shutdown_connections 2020-11-20T12:10:43.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5568056d20 0x7f55680591e0 secure :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0x7f558c145600 tx=0x7f5574009040).stop 2020-11-20T12:10:43.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f558c1484d0 0x7f558c14df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c0622c0 0x7f558c15bcf0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 --2- 172.21.15.187:0/3263492586 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f558c066f70 0x7f558c141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 >> 172.21.15.187:0/3263492586 conn(0x7f558c0bab30 msgr2=0x7f558c0b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:43.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.440+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 shutdown_connections 2020-11-20T12:10:43.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.441+0000 7f5591212700 1 -- 172.21.15.187:0/3263492586 wait complete. 2020-11-20T12:10:43.449 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 6400 2020-11-20T12:10:43.475 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:10:43.476 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:10:43.481 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:10:43.482 DEBUG:teuthology.parallel:result is None 2020-11-20T12:10:43.482 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:43.634 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:10:43.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.688+0000 7f1f7bfff700 1 Processor -- start 2020-11-20T12:10:43.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.688+0000 7f1f7bfff700 1 -- start start 2020-11-20T12:10:43.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.689+0000 7f1f7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c065570 0x7f1f7c09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:43.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.689+0000 7f1f7bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c0b13e0 con 0x7f1f7c065190 2020-11-20T12:10:43.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.689+0000 7f1f7bfff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c09d690 con 0x7f1f7c065f90 2020-11-20T12:10:43.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.689+0000 7f1f7bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c14ab30 con 0x7f1f7c065570 2020-11-20T12:10:43.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.689+0000 7f1f7a7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1f7c065f90 0x7f1f7c09ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42452/0 (socket says 172.21.15.187:42452) 2020-11-20T12:10:43.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.689+0000 7f1f7a7fc700 1 -- 172.21.15.187:0/235144039 learned_addr learned my addr 172.21.15.187:0/235144039 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:43.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453096595 0 0) 0x7f1f7c09d690 con 0x7f1f7c065f90 2020-11-20T12:10:43.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f7b7fe700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c065570 0x7f1f7c09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:43.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f68003240 con 0x7f1f7c065f90 2020-11-20T12:10:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1432400353 0 0) 0x7f1f68003240 con 0x7f1f7c065f90 2020-11-20T12:10:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c065570 msgr2=0x7f1f7c09d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c065570 0x7f1f7c09d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 >> v1:172.21.15.5:6789/0 conn(0x7f1f7c065190 legacy=0x7f1f7c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c062c10 con 0x7f1f7c065f90 2020-11-20T12:10:43.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1f700014e0 con 0x7f1f7c065f90 2020-11-20T12:10:43.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 >> v1:172.21.15.187:6789/0 conn(0x7f1f7c065f90 legacy=0x7f1f7c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.690+0000 7f1f79ffb700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f68003730 0x7f1f68003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:43.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f79ffb700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f680044e0 0x7f1f68004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:43.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f79ffb700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c065570 0x7f1f68004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:43.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c14ab30 con 0x7f1f680044e0 2020-11-20T12:10:43.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f68005390 con 0x7f1f68003730 2020-11-20T12:10:43.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f680042f0 con 0x7f1f7c065570 2020-11-20T12:10:43.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f7affd700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f68003730 0x7f1f68003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:43.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f7affd700 1 -- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c065570 msgr2=0x7f1f68004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:43.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f7affd700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c065570 0x7f1f68004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:43.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f7affd700 1 -- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f680044e0 msgr2=0x7f1f68004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:43.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f7affd700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f680044e0 0x7f1f68004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:43.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.691+0000 7f1f7affd700 1 -- 172.21.15.187:0/235144039 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f68003240 con 0x7f1f68003730 2020-11-20T12:10:43.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f7affd700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f68003730 0x7f1f68003b50 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f1f6c001e70 tx=0x7f1f6c007f10).ready entity=mon.1 client_cookie=9a424c8f0459b81c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:43.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c001540 con 0x7f1f68003730 2020-11-20T12:10:43.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c003030 con 0x7f1f68003730 2020-11-20T12:10:43.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f79ffb700 1 -- 172.21.15.187:0/235144039 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00cd90 con 0x7f1f68003730 2020-11-20T12:10:43.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f7bfff700 1 -- 172.21.15.187:0/235144039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f68003730 msgr2=0x7f1f68003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f7bfff700 1 --2- 172.21.15.187:0/235144039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f68003730 0x7f1f68003b50 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f1f6c001e70 tx=0x7f1f6c007f10).stop 2020-11-20T12:10:43.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f7b7fe700 1 -- 172.21.15.187:0/235144039 reap_dead start 2020-11-20T12:10:43.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f7bfff700 1 -- 172.21.15.187:0/235144039 shutdown_connections 2020-11-20T12:10:43.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.692+0000 7f1f7bfff700 1 -- 172.21.15.187:0/235144039 >> 172.21.15.187:0/235144039 conn(0x7f1f7c0bab30 msgr2=0x7f1f7c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:43.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.693+0000 7f1f7bfff700 1 -- 172.21.15.187:0/235144039 shutdown_connections 2020-11-20T12:10:43.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.693+0000 7f1f7bfff700 1 -- 172.21.15.187:0/235144039 wait complete. 2020-11-20T12:10:43.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.694+0000 7f1f7bfff700 1 Processor -- start 2020-11-20T12:10:43.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.694+0000 7f1f7bfff700 1 -- start start 2020-11-20T12:10:43.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.695+0000 7f1f7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c159b90 0x7f1f7c0b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:43.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.695+0000 7f1f7bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7c0b4680 0x7f1f7c159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:43.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.695+0000 7f1f7bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f7c143090 0x7f1f7c1434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:43.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.695+0000 7f1f7bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1439e0 con 0x7f1f7c0b4680 2020-11-20T12:10:43.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.695+0000 7f1f7bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1499b0 con 0x7f1f7c143090 2020-11-20T12:10:43.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.695+0000 7f1f7bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c149c50 con 0x7f1f7c159b90 2020-11-20T12:10:43.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.695+0000 7f1f7b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f7c143090 0x7f1f7c1434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:43.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f7c143090 0x7f1f7c1434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45860/0 (socket says 172.21.15.187:45860) 2020-11-20T12:10:43.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7b7fe700 1 -- 172.21.15.187:0/2183468238 learned_addr learned my addr 172.21.15.187:0/2183468238 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:43.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7affd700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c159b90 0x7f1f7c0b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:43.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7b7fe700 1 -- 172.21.15.187:0/2183468238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c159b90 msgr2=0x7f1f7c0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7b7fe700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c159b90 0x7f1f7c0b2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7a7fc700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7c0b4680 0x7f1f7c159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:43.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7b7fe700 1 -- 172.21.15.187:0/2183468238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7c0b4680 msgr2=0x7f1f7c159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7b7fe700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7c0b4680 0x7f1f7c159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7b7fe700 1 -- 172.21.15.187:0/2183468238 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c149ef0 con 0x7f1f7c143090 2020-11-20T12:10:43.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.696+0000 7f1f7b7fe700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f7c143090 0x7f1f7c1434f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f1f7c066370 tx=0x7f1f74006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:43.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.697+0000 7f1f62ffd700 1 -- 172.21.15.187:0/2183468238 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74010650 con 0x7f1f7c143090 2020-11-20T12:10:43.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.697+0000 7f1f62ffd700 1 -- 172.21.15.187:0/2183468238 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f740107b0 con 0x7f1f7c143090 2020-11-20T12:10:43.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.697+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c0b2f00 con 0x7f1f7c143090 2020-11-20T12:10:43.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.697+0000 7f1f62ffd700 1 -- 172.21.15.187:0/2183468238 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74010b50 con 0x7f1f7c143090 2020-11-20T12:10:43.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.697+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c15a460 con 0x7f1f7c143090 2020-11-20T12:10:43.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.698+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7c0b5520 con 0x7f1f7c143090 2020-11-20T12:10:43.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.698+0000 7f1f62ffd700 1 -- 172.21.15.187:0/2183468238 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1f7401c460 con 0x7f1f7c143090 2020-11-20T12:10:43.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.699+0000 7f1f62ffd700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1f58056d20 0x7f1f580591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:43.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.699+0000 7f1f62ffd700 1 -- 172.21.15.187:0/2183468238 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1f74070160 con 0x7f1f7c143090 2020-11-20T12:10:43.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.699+0000 7f1f7affd700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1f58056d20 0x7f1f580591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:43.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.700+0000 7f1f7affd700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1f58056d20 0x7f1f580591e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f1f7c0b6390 tx=0x7f1f6c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:43.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.703+0000 7f1f62ffd700 1 -- 172.21.15.187:0/2183468238 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1f740453f0 con 0x7f1f7c143090 2020-11-20T12:10:43.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.949+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f7c061a20 con 0x7f1f7c143090 2020-11-20T12:10:43.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.951+0000 7f1f62ffd700 1 -- 172.21.15.187:0/2183468238 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f1f74048a10 con 0x7f1f7c143090 2020-11-20T12:10:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:43.954 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:43.435959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:replay","state_seq":2,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1f58056d20 msgr2=0x7f1f580591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1f58056d20 0x7f1f580591e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f1f7c0b6390 tx=0x7f1f6c015000).stop 2020-11-20T12:10:43.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f7c143090 msgr2=0x7f1f7c1434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:43.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f7c143090 0x7f1f7c1434f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f1f7c066370 tx=0x7f1f74006df0).stop 2020-11-20T12:10:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 shutdown_connections 2020-11-20T12:10:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1f58056d20 0x7f1f580591e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f7c143090 0x7f1f7c1434f0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7c159b90 0x7f1f7c0b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 --2- 172.21.15.187:0/2183468238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7c0b4680 0x7f1f7c159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.953+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 >> 172.21.15.187:0/2183468238 conn(0x7f1f7c0bab30 msgr2=0x7f1f7c0b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:43.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.954+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 shutdown_connections 2020-11-20T12:10:43.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:43.954+0000 7f1f7bfff700 1 -- 172.21.15.187:0/2183468238 wait complete. 2020-11-20T12:10:43.960 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 52 2020-11-20T12:10:44.184 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:10:23.312121+0000', 'modified': '2020-11-20T12:10:43.435959+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 72, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5036}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5036': {'gid': 5036, 'name': 'b', 'rank': 0, 'incarnation': 52, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.5:6833/3088698963', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 3088698963}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 3088698963}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:10:44.184 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5036:up:replay 2020-11-20T12:10:45.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2020-11-20T12:10:45.185 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:45.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.435+0000 7f3141e4f700 1 Processor -- start 2020-11-20T12:10:45.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.435+0000 7f3141e4f700 1 -- start start 2020-11-20T12:10:45.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.436+0000 7f3141e4f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065190 0x7f313c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:45.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.436+0000 7f3141e4f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f313c065f90 0x7f313c143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:45.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.436+0000 7f3141e4f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313c0b13e0 con 0x7f313c143c70 2020-11-20T12:10:45.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.437+0000 7f3141e4f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313c14ab30 con 0x7f313c065f90 2020-11-20T12:10:45.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.437+0000 7f3141e4f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c062c10 con 0x7f313c065190 2020-11-20T12:10:45.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.437+0000 7f314164e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f313c143c70 0x7f313c09ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42472/0 (socket says 172.21.15.187:42472) 2020-11-20T12:10:45.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.437+0000 7f314164e700 1 -- 172.21.15.187:0/2279013602 learned_addr learned my addr 172.21.15.187:0/2279013602 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:45.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.437+0000 7f3140e4d700 1 --2- 172.21.15.187:0/2279013602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065190 0x7f313c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:45.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.437+0000 7f3133fff700 1 --2- 172.21.15.187:0/2279013602 >> v2:172.21.15.187:3300/0 conn(0x7f313c065f90 0x7f313c143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:45.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.438+0000 7f31337fe700 1 -- 172.21.15.187:0/2279013602 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217536754 0 0) 0x7f313c0b13e0 con 0x7f313c143c70 2020-11-20T12:10:45.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.438+0000 7f31337fe700 1 -- 172.21.15.187:0/2279013602 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3128003240 con 0x7f313c143c70 2020-11-20T12:10:45.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.438+0000 7f31337fe700 1 -- 172.21.15.187:0/2279013602 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2713904766 0 0) 0x7f3128003240 con 0x7f313c143c70 2020-11-20T12:10:45.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.438+0000 7f3140e4d700 1 -- 172.21.15.187:0/2279013602 >> v2:172.21.15.187:3300/0 conn(0x7f313c065f90 msgr2=0x7f313c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:45.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.438+0000 7f3140e4d700 1 --2- 172.21.15.187:0/2279013602 >> v2:172.21.15.187:3300/0 conn(0x7f313c065f90 0x7f313c143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.439+0000 7f3140e4d700 1 -- 172.21.15.187:0/2279013602 >> v1:172.21.15.187:6789/0 conn(0x7f313c143c70 legacy=0x7f313c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:45.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.439+0000 7f3140e4d700 1 -- 172.21.15.187:0/2279013602 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c0b1640 con 0x7f313c065190 2020-11-20T12:10:45.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.439+0000 7f3133fff700 1 --2- 172.21.15.187:0/2279013602 >> v2:172.21.15.187:3300/0 conn(0x7f313c065f90 0x7f313c143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:10:45.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.439+0000 7f3140e4d700 1 --2- 172.21.15.187:0/2279013602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065190 0x7f313c065570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f313c14ac70 tx=0x7f31380091d0).ready entity=mon.2 client_cookie=b27482a8b48cc272 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:45.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.439+0000 7f31337fe700 1 -- 172.21.15.187:0/2279013602 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31340014e0 con 0x7f313c065190 2020-11-20T12:10:45.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.439+0000 7f31337fe700 1 -- 172.21.15.187:0/2279013602 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313800b710 con 0x7f313c065190 2020-11-20T12:10:45.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.439+0000 7f31337fe700 1 -- 172.21.15.187:0/2279013602 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313800ba90 con 0x7f313c065190 2020-11-20T12:10:45.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.440+0000 7f3141e4f700 1 -- 172.21.15.187:0/2279013602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065190 msgr2=0x7f313c065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:45.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.440+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2279013602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065190 0x7f313c065570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f313c14ac70 tx=0x7f31380091d0).stop 2020-11-20T12:10:45.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.440+0000 7f3141e4f700 1 -- 172.21.15.187:0/2279013602 shutdown_connections 2020-11-20T12:10:45.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.440+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2279013602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065190 0x7f313c065570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.440+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2279013602 >> v2:172.21.15.187:3300/0 conn(0x7f313c065f90 0x7f313c143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.440+0000 7f3141e4f700 1 -- 172.21.15.187:0/2279013602 >> 172.21.15.187:0/2279013602 conn(0x7f313c0bab30 msgr2=0x7f313c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:45.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.440+0000 7f3141e4f700 1 -- 172.21.15.187:0/2279013602 shutdown_connections 2020-11-20T12:10:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.440+0000 7f3141e4f700 1 -- 172.21.15.187:0/2279013602 wait complete. 2020-11-20T12:10:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.442+0000 7f3141e4f700 1 Processor -- start 2020-11-20T12:10:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.442+0000 7f3141e4f700 1 -- start start 2020-11-20T12:10:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f3141e4f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c065190 0x7f313c148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f3141e4f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065f90 0x7f313c151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f3141e4f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c143c70 0x7f313c152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f3141e4f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c1590c0 con 0x7f313c065190 2020-11-20T12:10:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f3141e4f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c09ee10 con 0x7f313c143c70 2020-11-20T12:10:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f3141e4f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c09f0b0 con 0x7f313c065f90 2020-11-20T12:10:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f314164e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c143c70 0x7f313c152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:45.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f314164e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c143c70 0x7f313c152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45874/0 (socket says 172.21.15.187:45874) 2020-11-20T12:10:45.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.443+0000 7f314164e700 1 -- 172.21.15.187:0/2218192280 learned_addr learned my addr 172.21.15.187:0/2218192280 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:45.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f3140e4d700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c065190 0x7f313c148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:45.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f3133fff700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065f90 0x7f313c151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:45.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f314164e700 1 -- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065f90 msgr2=0x7f313c151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:45.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f314164e700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065f90 0x7f313c151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f314164e700 1 -- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c065190 msgr2=0x7f313c148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:45.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f314164e700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c065190 0x7f313c148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f314164e700 1 -- 172.21.15.187:0/2218192280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c142c00 con 0x7f313c143c70 2020-11-20T12:10:45.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f3133fff700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065f90 0x7f313c151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:45.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f314164e700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c143c70 0x7f313c152220 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f313400c7d0 tx=0x7f31340011f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:45.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f3130ff9700 1 -- 172.21.15.187:0/2218192280 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134007170 con 0x7f313c143c70 2020-11-20T12:10:45.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f3130ff9700 1 -- 172.21.15.187:0/2218192280 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31340072d0 con 0x7f313c143c70 2020-11-20T12:10:45.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313c0ba3e0 con 0x7f313c143c70 2020-11-20T12:10:45.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.444+0000 7f3130ff9700 1 -- 172.21.15.187:0/2218192280 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134004940 con 0x7f313c143c70 2020-11-20T12:10:45.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.445+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313c158e30 con 0x7f313c143c70 2020-11-20T12:10:45.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.446+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313c09f270 con 0x7f313c143c70 2020-11-20T12:10:45.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.447+0000 7f3130ff9700 1 -- 172.21.15.187:0/2218192280 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3134010070 con 0x7f313c143c70 2020-11-20T12:10:45.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.448+0000 7f3130ff9700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3114056d20 0x7f31140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:45.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.448+0000 7f3130ff9700 1 -- 172.21.15.187:0/2218192280 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f313406f9a0 con 0x7f313c143c70 2020-11-20T12:10:45.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.448+0000 7f3140e4d700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3114056d20 0x7f31140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:45.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.449+0000 7f3140e4d700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3114056d20 0x7f31140591e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f313c09e790 tx=0x7f3138016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:45.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.452+0000 7f3130ff9700 1 -- 172.21.15.187:0/2218192280 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3134044d50 con 0x7f313c143c70 2020-11-20T12:10:45.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.692+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f313c061a20 con 0x7f313c143c70 2020-11-20T12:10:45.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.693+0000 7f3130ff9700 1 -- 172.21.15.187:0/2218192280 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f3134048370 con 0x7f313c143c70 2020-11-20T12:10:45.697 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:45.697 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:45.442277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:45.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.695+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3114056d20 msgr2=0x7f31140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:45.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.695+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3114056d20 0x7f31140591e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f313c09e790 tx=0x7f3138016000).stop 2020-11-20T12:10:45.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c143c70 msgr2=0x7f313c152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:45.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c143c70 0x7f313c152220 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f313400c7d0 tx=0x7f31340011f0).stop 2020-11-20T12:10:45.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 shutdown_connections 2020-11-20T12:10:45.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3114056d20 0x7f31140591e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f313c065190 0x7f313c148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313c143c70 0x7f313c152220 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 --2- 172.21.15.187:0/2218192280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313c065f90 0x7f313c151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:45.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 >> 172.21.15.187:0/2218192280 conn(0x7f313c0bab30 msgr2=0x7f313c0b7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:45.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 shutdown_connections 2020-11-20T12:10:45.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:45.696+0000 7f3141e4f700 1 -- 172.21.15.187:0/2218192280 wait complete. 2020-11-20T12:10:45.702 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 54 2020-11-20T12:10:45.925 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:10:23.312121+0000', 'modified': '2020-11-20T12:10:45.442277+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 72, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5036}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5036': {'gid': 5036, 'name': 'b', 'rank': 0, 'incarnation': 52, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.5:6833/3088698963', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 3088698963}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 3088698963}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:10:45.925 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5036:up:rejoin 2020-11-20T12:10:46.926 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:47.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.176+0000 7fcb2099d700 1 Processor -- start 2020-11-20T12:10:47.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.177+0000 7fcb2099d700 1 -- start start 2020-11-20T12:10:47.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.178+0000 7fcb2099d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c1484d0 0x7fcb1c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.178+0000 7fcb2099d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcb1c066f70 0x7fcb1c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.178+0000 7fcb2099d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c0b1300 con 0x7fcb1c0622c0 2020-11-20T12:10:47.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.178+0000 7fcb2099d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1c067730 con 0x7fcb1c066f70 2020-11-20T12:10:47.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.178+0000 7fcb2099d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c0b15a0 con 0x7fcb1c1484d0 2020-11-20T12:10:47.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.178+0000 7fcb13fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcb1c0622c0 0x7fcb1c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42484/0 (socket says 172.21.15.187:42484) 2020-11-20T12:10:47.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.178+0000 7fcb13fff700 1 -- 172.21.15.187:0/294928598 learned_addr learned my addr 172.21.15.187:0/294928598 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:47.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.178+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631907060 0 0) 0x7fcb1c0b1300 con 0x7fcb1c0622c0 2020-11-20T12:10:47.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcafc003240 con 0x7fcb1c0622c0 2020-11-20T12:10:47.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1bfff700 1 --2- 172.21.15.187:0/294928598 >> v2:172.21.15.5:3300/0 conn(0x7fcb1c066f70 0x7fcb1c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1b7fe700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c1484d0 0x7fcb1c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 400739910 0 0) 0x7fcafc003240 con 0x7fcb1c0622c0 2020-11-20T12:10:47.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c1484d0 msgr2=0x7fcb1c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:47.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c1484d0 0x7fcb1c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 >> v2:172.21.15.5:3300/0 conn(0x7fcb1c066f70 msgr2=0x7fcb1c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:47.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 --2- 172.21.15.187:0/294928598 >> v2:172.21.15.5:3300/0 conn(0x7fcb1c066f70 0x7fcb1c0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c0b1840 con 0x7fcb1c0622c0 2020-11-20T12:10:47.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcb040014e0 con 0x7fcb1c0622c0 2020-11-20T12:10:47.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 >> v1:172.21.15.187:6789/0 conn(0x7fcb1c0622c0 legacy=0x7fcb1c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:47.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcafc003730 0x7fcafc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c1484d0 0x7fcafc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcafc0049d0 0x7fcafc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c067730 con 0x7fcafc003730 2020-11-20T12:10:47.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c0b15a0 con 0x7fcafc0049d0 2020-11-20T12:10:47.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.179+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0041d0 con 0x7fcb1c1484d0 2020-11-20T12:10:47.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.180+0000 7fcb1bfff700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcafc0049d0 0x7fcafc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.180+0000 7fcb13fff700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c1484d0 0x7fcafc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.180+0000 7fcb1b7fe700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcafc003730 0x7fcafc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.180+0000 7fcb1bfff700 1 -- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c1484d0 msgr2=0x7fcafc0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:47.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.180+0000 7fcb1bfff700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c1484d0 0x7fcafc0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.180+0000 7fcb1bfff700 1 -- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcafc003730 msgr2=0x7fcafc003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:47.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.180+0000 7fcb1bfff700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcafc003730 0x7fcafc003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.180+0000 7fcb1bfff700 1 -- 172.21.15.187:0/294928598 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc003240 con 0x7fcafc0049d0 2020-11-20T12:10:47.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb1bfff700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcafc0049d0 0x7fcafc004e50 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7fcb14009d80 tx=0x7fcb14008040).ready entity=mon.1 client_cookie=f9c6282199d4d22c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:47.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1400c070 con 0x7fcafc0049d0 2020-11-20T12:10:47.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1400ea80 con 0x7fcafc0049d0 2020-11-20T12:10:47.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb1affd700 1 -- 172.21.15.187:0/294928598 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1400ee40 con 0x7fcafc0049d0 2020-11-20T12:10:47.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb2099d700 1 -- 172.21.15.187:0/294928598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcafc0049d0 msgr2=0x7fcafc004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:47.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb2099d700 1 --2- 172.21.15.187:0/294928598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcafc0049d0 0x7fcafc004e50 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7fcb14009d80 tx=0x7fcb14008040).stop 2020-11-20T12:10:47.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb1bfff700 1 -- 172.21.15.187:0/294928598 reap_dead start 2020-11-20T12:10:47.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb2099d700 1 -- 172.21.15.187:0/294928598 shutdown_connections 2020-11-20T12:10:47.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.181+0000 7fcb2099d700 1 -- 172.21.15.187:0/294928598 >> 172.21.15.187:0/294928598 conn(0x7fcb1c0bab30 msgr2=0x7fcb1c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:47.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.182+0000 7fcb2099d700 1 -- 172.21.15.187:0/294928598 shutdown_connections 2020-11-20T12:10:47.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.182+0000 7fcb2099d700 1 -- 172.21.15.187:0/294928598 wait complete. 2020-11-20T12:10:47.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.183+0000 7fcb2099d700 1 Processor -- start 2020-11-20T12:10:47.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.183+0000 7fcb2099d700 1 -- start start 2020-11-20T12:10:47.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.184+0000 7fcb2099d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb1c099190 0x7fcb1c099570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.184+0000 7fcb2099d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c09e5d0 0x7fcb1c09a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.184+0000 7fcb2099d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1c09ea50 0x7fcb1c143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.184+0000 7fcb2099d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c148b60 con 0x7fcb1c09ea50 2020-11-20T12:10:47.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.184+0000 7fcb2099d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c09aea0 con 0x7fcb1c099190 2020-11-20T12:10:47.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.184+0000 7fcb2099d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c09b140 con 0x7fcb1c09e5d0 2020-11-20T12:10:47.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.184+0000 7fcb1b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb1c099190 0x7fcb1c099570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb1c099190 0x7fcb1c099570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45890/0 (socket says 172.21.15.187:45890) 2020-11-20T12:10:47.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1b7fe700 1 -- 172.21.15.187:0/3876441619 learned_addr learned my addr 172.21.15.187:0/3876441619 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:47.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb13fff700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c09e5d0 0x7fcb1c09a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1b7fe700 1 -- 172.21.15.187:0/3876441619 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c09e5d0 msgr2=0x7fcb1c09a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:47.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1bfff700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1c09ea50 0x7fcb1c143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1b7fe700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c09e5d0 0x7fcb1c09a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1b7fe700 1 -- 172.21.15.187:0/3876441619 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1c09ea50 msgr2=0x7fcb1c143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:47.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1b7fe700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1c09ea50 0x7fcb1c143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1b7fe700 1 -- 172.21.15.187:0/3876441619 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c141d10 con 0x7fcb1c099190 2020-11-20T12:10:47.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.185+0000 7fcb1b7fe700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb1c099190 0x7fcb1c099570 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fcb0c009720 tx=0x7fcb0c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:47.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.186+0000 7fcb137fe700 1 -- 172.21.15.187:0/3876441619 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c0133f0 con 0x7fcb1c099190 2020-11-20T12:10:47.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.186+0000 7fcb137fe700 1 -- 172.21.15.187:0/3876441619 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0c013550 con 0x7fcb1c099190 2020-11-20T12:10:47.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.186+0000 7fcb137fe700 1 -- 172.21.15.187:0/3876441619 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c0138d0 con 0x7fcb1c099190 2020-11-20T12:10:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.186+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c09ab60 con 0x7fcb1c099190 2020-11-20T12:10:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.186+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c144210 con 0x7fcb1c099190 2020-11-20T12:10:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.187+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c148cd0 con 0x7fcb1c099190 2020-11-20T12:10:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.188+0000 7fcb137fe700 1 -- 172.21.15.187:0/3876441619 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcb0c019020 con 0x7fcb1c099190 2020-11-20T12:10:47.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.188+0000 7fcb137fe700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaf8056d50 0x7fcaf8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:47.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.188+0000 7fcb137fe700 1 -- 172.21.15.187:0/3876441619 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcb0c06f970 con 0x7fcb1c099190 2020-11-20T12:10:47.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.189+0000 7fcb13fff700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaf8056d50 0x7fcaf8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:47.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.189+0000 7fcb13fff700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaf8056d50 0x7fcaf8059210 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fcb04001e90 tx=0x7fcb04000f60).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:47.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.192+0000 7fcb137fe700 1 -- 172.21.15.187:0/3876441619 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcb1c148cd0 con 0x7fcb1c099190 2020-11-20T12:10:47.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.439+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb1c148cd0 con 0x7fcb1c099190 2020-11-20T12:10:47.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.440+0000 7fcb137fe700 1 -- 172.21.15.187:0/3876441619 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+1782 (secure 0 0 0) 0x7fcb1c148cd0 con 0x7fcb1c099190 2020-11-20T12:10:47.444 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:47.444 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":55,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:47.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaf8056d50 msgr2=0x7fcaf8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:47.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaf8056d50 0x7fcaf8059210 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fcb04001e90 tx=0x7fcb04000f60).stop 2020-11-20T12:10:47.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb1c099190 msgr2=0x7fcb1c099570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:47.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb1c099190 0x7fcb1c099570 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fcb0c009720 tx=0x7fcb0c008040).stop 2020-11-20T12:10:47.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 shutdown_connections 2020-11-20T12:10:47.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcaf8056d50 0x7fcaf8059210 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1c09ea50 0x7fcb1c143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb1c099190 0x7fcb1c099570 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 --2- 172.21.15.187:0/3876441619 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb1c09e5d0 0x7fcb1c09a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:47.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 >> 172.21.15.187:0/3876441619 conn(0x7fcb1c0bab30 msgr2=0x7fcb1c065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:47.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 shutdown_connections 2020-11-20T12:10:47.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:47.443+0000 7fcb2099d700 1 -- 172.21.15.187:0/3876441619 wait complete. 2020-11-20T12:10:47.449 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 55 2020-11-20T12:10:47.674 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:10:23.312121+0000', 'modified': '2020-11-20T12:10:46.446562+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 72, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5036}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5036': {'gid': 5036, 'name': 'b', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.5:6833/3088698963', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 3088698963}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 3088698963}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:10:47.674 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:10:47.675 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2020-11-20T12:10:47.976 INFO:teuthology.orchestra.run.smithi005.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":5036,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.5111356320000002,"mdsmap_epoch":55,"osdmap_epoch":72,"osdmap_epoch_barrier":72,"uptime":10.560903787999999} 2020-11-20T12:10:47.977 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5036, "mdsmap_epoch": 55, "osdmap_epoch": 72, "osdmap_epoch_barrier": 72, "rank_uptime": 4.511135632, "state": "up:active", "uptime": 10.560903788, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:10:47.978 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:10:47.978 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:10:47.979 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:10:47.979 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:10:47.979 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:10:47.980 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:10:47.980 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:10:47.981 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:10:47.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:47.982 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:10:48.045 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:10:48.045 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:10:48.046 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:10:48.046 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:48.046 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:10:48.046 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:48.047 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:10:48.047 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:48.047 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:10:48.047 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:48.047 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:10:48.048 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:48.048 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:10:48.048 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:48.048 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:10:48.049 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42390sec preferred_lft 42390sec 2020-11-20T12:10:48.049 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:10:48.049 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:48.049 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:10:48.050 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:10:48.050 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:10:48.050 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:48.050 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:10:48.050 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:48.051 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:10:48.051 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:10:48.051 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:10:48.051 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:48.051 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:10:48.052 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:10:48.052 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:10:48.052 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:10:48.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:48.053 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:10:48.156 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:10:48.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:48.161 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:10:48.213 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:10:48.214 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:10:48.214 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:10:48.214 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:10:48.214 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:10:48.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:48.216 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:10:48.216 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:10:48.217 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:10:48.217 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:10:48.218 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:10:48.218 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:10:48.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:48.477 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:10:48.527 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:10:48.528 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:10:48.529 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:10:48.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:48.530 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:10:48.626 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:10:48.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.863+0000 7feec1336700 1 Processor -- start 2020-11-20T12:10:48.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.863+0000 7feec1336700 1 -- start start 2020-11-20T12:10:48.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.864+0000 7feec1336700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feebc1484d0 0x7feebc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:48.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.864+0000 7feec1336700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc066f70 0x7feebc0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:48.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.864+0000 7feec1336700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feebc0b1300 con 0x7feebc0622c0 2020-11-20T12:10:48.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.864+0000 7feec1336700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feebc067730 con 0x7feebc1484d0 2020-11-20T12:10:48.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.864+0000 7feec1336700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc0b15a0 con 0x7feebc066f70 2020-11-20T12:10:48.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feebbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feebc1484d0 0x7feebc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:48.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feeb37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7feebc0622c0 0x7feebc0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42506/0 (socket says 172.21.15.187:42506) 2020-11-20T12:10:48.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feeb37fe700 1 -- 172.21.15.187:0/4164437864 learned_addr learned my addr 172.21.15.187:0/4164437864 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:48.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feec0b35700 1 --2- 172.21.15.187:0/4164437864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc066f70 0x7feebc0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:48.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feebbfff700 1 -- 172.21.15.187:0/4164437864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc066f70 msgr2=0x7feebc0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:48.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feebbfff700 1 --2- 172.21.15.187:0/4164437864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc066f70 0x7feebc0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:48.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feebbfff700 1 -- 172.21.15.187:0/4164437864 >> v1:172.21.15.187:6789/0 conn(0x7feebc0622c0 legacy=0x7feebc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:48.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feebbfff700 1 -- 172.21.15.187:0/4164437864 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc0b1840 con 0x7feebc1484d0 2020-11-20T12:10:48.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.865+0000 7feebb7fe700 1 -- 172.21.15.187:0/4164437864 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700154717 0 0) 0x7feebc0b1300 con 0x7feebc0622c0 2020-11-20T12:10:48.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feebbfff700 1 --2- 172.21.15.187:0/4164437864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc1484d0 0x7feebc061dd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7feeac00b9a0 tx=0x7feeac006df0).ready entity=mon.1 client_cookie=bbf4e394a54152d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:48.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feebb7fe700 1 -- 172.21.15.187:0/4164437864 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac0097b0 con 0x7feebc1484d0 2020-11-20T12:10:48.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feebb7fe700 1 -- 172.21.15.187:0/4164437864 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeac009910 con 0x7feebc1484d0 2020-11-20T12:10:48.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feebb7fe700 1 -- 172.21.15.187:0/4164437864 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac009c90 con 0x7feebc1484d0 2020-11-20T12:10:48.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feec1336700 1 -- 172.21.15.187:0/4164437864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc1484d0 msgr2=0x7feebc061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:48.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feec1336700 1 --2- 172.21.15.187:0/4164437864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc1484d0 0x7feebc061dd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7feeac00b9a0 tx=0x7feeac006df0).stop 2020-11-20T12:10:48.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feec1336700 1 -- 172.21.15.187:0/4164437864 shutdown_connections 2020-11-20T12:10:48.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feec1336700 1 --2- 172.21.15.187:0/4164437864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc066f70 0x7feebc0b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:48.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feec1336700 1 --2- 172.21.15.187:0/4164437864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc1484d0 0x7feebc061dd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:48.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.866+0000 7feec1336700 1 -- 172.21.15.187:0/4164437864 >> 172.21.15.187:0/4164437864 conn(0x7feebc0bab30 msgr2=0x7feebc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:48.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.867+0000 7feec1336700 1 -- 172.21.15.187:0/4164437864 shutdown_connections 2020-11-20T12:10:48.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.867+0000 7feec1336700 1 -- 172.21.15.187:0/4164437864 wait complete. 2020-11-20T12:10:48.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.868+0000 7feec1336700 1 Processor -- start 2020-11-20T12:10:48.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.868+0000 7feec1336700 1 -- start start 2020-11-20T12:10:48.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.869+0000 7feec1336700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc0622c0 0x7feebc09e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:48.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.869+0000 7feec1336700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc066f70 0x7feebc0a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:48.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.869+0000 7feec1336700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc09e3f0 0x7feebc144b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:48.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.869+0000 7feec1336700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc09ee50 con 0x7feebc0622c0 2020-11-20T12:10:48.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.869+0000 7feec1336700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc148cf0 con 0x7feebc066f70 2020-11-20T12:10:48.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.869+0000 7feec1336700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc148f90 con 0x7feebc09e3f0 2020-11-20T12:10:48.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc066f70 0x7feebc0a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:48.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc066f70 0x7feebc0a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45908/0 (socket says 172.21.15.187:45908) 2020-11-20T12:10:48.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 -- 172.21.15.187:0/4223015164 learned_addr learned my addr 172.21.15.187:0/4223015164 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:48.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feec0b35700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc09e3f0 0x7feebc144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:48.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 -- 172.21.15.187:0/4223015164 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc09e3f0 msgr2=0x7feebc144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:48.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc09e3f0 0x7feebc144b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:48.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 -- 172.21.15.187:0/4223015164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc0622c0 msgr2=0x7feebc09e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:48.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc0622c0 0x7feebc09e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:48.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 -- 172.21.15.187:0/4223015164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc0a0a80 con 0x7feebc066f70 2020-11-20T12:10:48.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb37fe700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc066f70 0x7feebc0a0590 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7feea400c710 tx=0x7feea4007a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:48.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.870+0000 7feeb8ff9700 1 -- 172.21.15.187:0/4223015164 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4007320 con 0x7feebc066f70 2020-11-20T12:10:48.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.871+0000 7feeb8ff9700 1 -- 172.21.15.187:0/4223015164 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea4007480 con 0x7feebc066f70 2020-11-20T12:10:48.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.871+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feebc09c3e0 con 0x7feebc066f70 2020-11-20T12:10:48.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.871+0000 7feeb8ff9700 1 -- 172.21.15.187:0/4223015164 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4005800 con 0x7feebc066f70 2020-11-20T12:10:48.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.871+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feebc145230 con 0x7feebc066f70 2020-11-20T12:10:48.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.872+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feebc148b30 con 0x7feebc066f70 2020-11-20T12:10:48.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.872+0000 7feeb8ff9700 1 -- 172.21.15.187:0/4223015164 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feea4010030 con 0x7feebc066f70 2020-11-20T12:10:48.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.873+0000 7feeb8ff9700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee98056d20 0x7fee980591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:48.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.873+0000 7feeb8ff9700 1 -- 172.21.15.187:0/4223015164 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7feea406fc40 con 0x7feebc066f70 2020-11-20T12:10:48.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.873+0000 7feebbfff700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee98056d20 0x7fee980591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:48.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.874+0000 7feebbfff700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee98056d20 0x7fee980591e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7feebc066540 tx=0x7feeac015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:48.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:48.877+0000 7feeb8ff9700 1 -- 172.21.15.187:0/4223015164 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feea4045040 con 0x7feebc066f70 2020-11-20T12:10:49.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.063+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feebc149150 con 0x7feebc066f70 2020-11-20T12:10:49.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.064+0000 7feeb8ff9700 1 -- 172.21.15.187:0/4223015164 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16912 (secure 0 0 0) 0x7feea4048660 con 0x7feebc066f70 2020-11-20T12:10:49.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee98056d20 msgr2=0x7fee980591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee98056d20 0x7fee980591e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7feebc066540 tx=0x7feeac015000).stop 2020-11-20T12:10:49.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc066f70 msgr2=0x7feebc0a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc066f70 0x7feebc0a0590 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7feea400c710 tx=0x7feea4007a80).stop 2020-11-20T12:10:49.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 shutdown_connections 2020-11-20T12:10:49.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fee98056d20 0x7fee980591e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feebc0622c0 0x7feebc09e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:49.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feebc09e3f0 0x7feebc144b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 --2- 172.21.15.187:0/4223015164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feebc066f70 0x7feebc0a0590 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 >> 172.21.15.187:0/4223015164 conn(0x7feebc0bab30 msgr2=0x7feebc065220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:49.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 shutdown_connections 2020-11-20T12:10:49.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.067+0000 7feec1336700 1 -- 172.21.15.187:0/4223015164 wait complete. 2020-11-20T12:10:49.078 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:49.079 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":72,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:10:42.605346+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:10:20.847618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2020-11-20T12:10:22.150115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1383561284":"2020-11-21T12:10:42.605289+0000","172.21.15.187:6834/1383561284":"2020-11-21T12:10:42.605289+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:10:49.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:10:20.847618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:10:22.150115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:10:49.095 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:10:49.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.327+0000 7fede2b05700 1 Processor -- start 2020-11-20T12:10:49.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.327+0000 7fede2b05700 1 -- start start 2020-11-20T12:10:49.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.328+0000 7fede2b05700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feddc065f90 0x7feddc065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.328+0000 7fede2b05700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.328+0000 7fede2b05700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc0b13e0 con 0x7feddc065190 2020-11-20T12:10:49.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.329+0000 7fede2b05700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feddc14ab30 con 0x7feddc065f90 2020-11-20T12:10:49.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.329+0000 7fede2b05700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc062c10 con 0x7feddc09ca90 2020-11-20T12:10:49.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.329+0000 7fede1302700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feddc065f90 0x7feddc065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.329+0000 7fede1302700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feddc065f90 0x7feddc065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45914/0 (socket says 172.21.15.187:45914) 2020-11-20T12:10:49.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.329+0000 7fede1302700 1 -- 172.21.15.187:0/1632129656 learned_addr learned my addr 172.21.15.187:0/1632129656 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:49.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.329+0000 7fede2304700 1 --2- 172.21.15.187:0/1632129656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.330+0000 7fede1302700 1 -- 172.21.15.187:0/1632129656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 msgr2=0x7feddc09cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:49.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.330+0000 7fede1302700 1 --2- 172.21.15.187:0/1632129656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc09cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.330+0000 7fede1302700 1 -- 172.21.15.187:0/1632129656 >> v1:172.21.15.5:6789/0 conn(0x7feddc065190 legacy=0x7feddc143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:49.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.330+0000 7fede1302700 1 -- 172.21.15.187:0/1632129656 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0b1640 con 0x7feddc065f90 2020-11-20T12:10:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.330+0000 7fede1302700 1 --2- 172.21.15.187:0/1632129656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc065570 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7fedcc006fd0 tx=0x7fedcc006eb0).ready entity=mon.1 client_cookie=713268dc043c9ff3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.330+0000 7fede0b01700 1 -- 172.21.15.187:0/1632129656 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc009920 con 0x7feddc065f90 2020-11-20T12:10:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.330+0000 7fede0b01700 1 -- 172.21.15.187:0/1632129656 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc009a80 con 0x7feddc065f90 2020-11-20T12:10:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.330+0000 7fede0b01700 1 -- 172.21.15.187:0/1632129656 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc009e00 con 0x7feddc065f90 2020-11-20T12:10:49.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.331+0000 7fede2b05700 1 -- 172.21.15.187:0/1632129656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 msgr2=0x7feddc065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.331+0000 7fede2b05700 1 --2- 172.21.15.187:0/1632129656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc065570 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7fedcc006fd0 tx=0x7fedcc006eb0).stop 2020-11-20T12:10:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.331+0000 7fede2b05700 1 -- 172.21.15.187:0/1632129656 shutdown_connections 2020-11-20T12:10:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.331+0000 7fede2b05700 1 --2- 172.21.15.187:0/1632129656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc09cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.331+0000 7fede2b05700 1 --2- 172.21.15.187:0/1632129656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc065570 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.331+0000 7fede2b05700 1 -- 172.21.15.187:0/1632129656 >> 172.21.15.187:0/1632129656 conn(0x7feddc0bab30 msgr2=0x7feddc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:49.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.331+0000 7fede2b05700 1 -- 172.21.15.187:0/1632129656 shutdown_connections 2020-11-20T12:10:49.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.332+0000 7fede2b05700 1 -- 172.21.15.187:0/1632129656 wait complete. 2020-11-20T12:10:49.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.333+0000 7fede2b05700 1 Processor -- start 2020-11-20T12:10:49.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.334+0000 7fede2b05700 1 -- start start 2020-11-20T12:10:49.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede2b05700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feddc065190 0x7feddc1429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede2b05700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede2b05700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede2b05700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc148ff0 con 0x7feddc065190 2020-11-20T12:10:49.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede2b05700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc149220 con 0x7feddc065f90 2020-11-20T12:10:49.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede2b05700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc1494c0 con 0x7feddc09ca90 2020-11-20T12:10:49.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede1302700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede1302700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45922/0 (socket says 172.21.15.187:45922) 2020-11-20T12:10:49.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.335+0000 7fede1302700 1 -- 172.21.15.187:0/3786184533 learned_addr learned my addr 172.21.15.187:0/3786184533 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:49.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede2304700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede1b03700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feddc065190 0x7feddc1429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede1302700 1 -- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 msgr2=0x7feddc159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede1302700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede1302700 1 -- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feddc065190 msgr2=0x7feddc1429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede1302700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feddc065190 0x7feddc1429a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede1302700 1 -- 172.21.15.187:0/3786184533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc142e90 con 0x7feddc065f90 2020-11-20T12:10:49.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede2304700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:49.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede1302700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc159650 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7fedcc00b1b0 tx=0x7fedcc015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:49.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fedc9ffb700 1 -- 172.21.15.187:0/3786184533 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc01e460 con 0x7feddc065f90 2020-11-20T12:10:49.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fedc9ffb700 1 -- 172.21.15.187:0/3786184533 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc01e5c0 con 0x7feddc065f90 2020-11-20T12:10:49.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fedc9ffb700 1 -- 172.21.15.187:0/3786184533 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc01e940 con 0x7feddc065f90 2020-11-20T12:10:49.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.336+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc0b3a00 con 0x7feddc065f90 2020-11-20T12:10:49.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.337+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc15a030 con 0x7feddc065f90 2020-11-20T12:10:49.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.338+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feddc061a20 con 0x7feddc065f90 2020-11-20T12:10:49.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.338+0000 7fedc9ffb700 1 -- 172.21.15.187:0/3786184533 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fedcc025030 con 0x7feddc065f90 2020-11-20T12:10:49.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.339+0000 7fedc9ffb700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedb4056d20 0x7fedb40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.339+0000 7fedc9ffb700 1 -- 172.21.15.187:0/3786184533 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fedcc071880 con 0x7feddc065f90 2020-11-20T12:10:49.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.339+0000 7fede1b03700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedb4056d20 0x7fedb40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.340+0000 7fede1b03700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedb4056d20 0x7fedb40591e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fedd8002410 tx=0x7fedd8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:49.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.345+0000 7fedc9ffb700 1 -- 172.21.15.187:0/3786184533 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fedcc046cd0 con 0x7feddc065f90 2020-11-20T12:10:49.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.529+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7feddc0afc60 con 0x7feddc065f90 2020-11-20T12:10:49.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.530+0000 7fedc9ffb700 1 -- 172.21.15.187:0/3786184533 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v72) v1 ==== 110+0+10 (secure 0 0 0) 0x7fedcc023070 con 0x7feddc065f90 2020-11-20T12:10:49.534 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:10:49.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedb4056d20 msgr2=0x7fedb40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedb4056d20 0x7fedb40591e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fedd8002410 tx=0x7fedd8009040).stop 2020-11-20T12:10:49.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 msgr2=0x7feddc159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc159650 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7fedcc00b1b0 tx=0x7fedcc015000).stop 2020-11-20T12:10:49.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 shutdown_connections 2020-11-20T12:10:49.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fedb4056d20 0x7fedb40591e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feddc065190 0x7feddc1429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feddc09ca90 0x7feddc159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 --2- 172.21.15.187:0/3786184533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feddc065f90 0x7feddc159650 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 >> 172.21.15.187:0/3786184533 conn(0x7feddc0bab30 msgr2=0x7feddc0aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:49.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 shutdown_connections 2020-11-20T12:10:49.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.532+0000 7fede2b05700 1 -- 172.21.15.187:0/3786184533 wait complete. 2020-11-20T12:10:49.559 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:10:49.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.804+0000 7f85d2d08700 1 Processor -- start 2020-11-20T12:10:49.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.805+0000 7f85d2d08700 1 -- start start 2020-11-20T12:10:49.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.806+0000 7f85d2d08700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1760 0x7f85cc065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.806+0000 7f85d2d08700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f85cc065680 0x7f85cc14c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.806+0000 7f85d2d08700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85cc150390 con 0x7f85cc0b1380 2020-11-20T12:10:49.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.806+0000 7f85d2d08700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85cc1511e0 con 0x7f85cc065680 2020-11-20T12:10:49.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.806+0000 7f85d2d08700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc14d760 con 0x7f85cc0b1760 2020-11-20T12:10:49.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.807+0000 7f85d2507700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f85cc065680 0x7f85cc14c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.807+0000 7f85d2507700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f85cc065680 0x7f85cc14c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45930/0 (socket says 172.21.15.187:45930) 2020-11-20T12:10:49.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.807+0000 7f85d2507700 1 -- 172.21.15.187:0/3013961647 learned_addr learned my addr 172.21.15.187:0/3013961647 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:49.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.807+0000 7f85d2507700 1 -- 172.21.15.187:0/3013961647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1760 msgr2=0x7f85cc065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:49.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.807+0000 7f85d2507700 1 --2- 172.21.15.187:0/3013961647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1760 0x7f85cc065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.807+0000 7f85d2507700 1 -- 172.21.15.187:0/3013961647 >> v1:172.21.15.5:6789/0 conn(0x7f85cc0b1380 legacy=0x7f85cc150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.807+0000 7f85d2507700 1 -- 172.21.15.187:0/3013961647 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc150630 con 0x7f85cc065680 2020-11-20T12:10:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.807+0000 7f85d2507700 1 --2- 172.21.15.187:0/3013961647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc065680 0x7f85cc14c260 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f85c4009d50 tx=0x7f85c4009290).ready entity=mon.1 client_cookie=d23a26d006af1041 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d0d04700 1 -- 172.21.15.187:0/3013961647 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c400bc40 con 0x7f85cc065680 2020-11-20T12:10:49.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d0d04700 1 -- 172.21.15.187:0/3013961647 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c400bda0 con 0x7f85cc065680 2020-11-20T12:10:49.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d0d04700 1 -- 172.21.15.187:0/3013961647 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4011610 con 0x7f85cc065680 2020-11-20T12:10:49.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d2d08700 1 -- 172.21.15.187:0/3013961647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc065680 msgr2=0x7f85cc14c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d2d08700 1 --2- 172.21.15.187:0/3013961647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc065680 0x7f85cc14c260 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f85c4009d50 tx=0x7f85c4009290).stop 2020-11-20T12:10:49.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d2d08700 1 -- 172.21.15.187:0/3013961647 shutdown_connections 2020-11-20T12:10:49.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d2d08700 1 --2- 172.21.15.187:0/3013961647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc065680 0x7f85cc14c260 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d2d08700 1 --2- 172.21.15.187:0/3013961647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1760 0x7f85cc065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:49.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.808+0000 7f85d2d08700 1 -- 172.21.15.187:0/3013961647 >> 172.21.15.187:0/3013961647 conn(0x7f85cc0bab30 msgr2=0x7f85cc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:49.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.809+0000 7f85d2d08700 1 -- 172.21.15.187:0/3013961647 shutdown_connections 2020-11-20T12:10:49.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.809+0000 7f85d2d08700 1 -- 172.21.15.187:0/3013961647 wait complete. 2020-11-20T12:10:49.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.811+0000 7f85d2d08700 1 Processor -- start 2020-11-20T12:10:49.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.811+0000 7f85d2d08700 1 -- start start 2020-11-20T12:10:49.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.812+0000 7f85d2d08700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1380 0x7f85cc0a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.812+0000 7f85d2d08700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc0b1760 0x7f85cc0b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.812+0000 7f85d2d08700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85cc0b7420 0x7f85cc15f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.812+0000 7f85d2d08700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc0b4960 con 0x7f85cc0b7420 2020-11-20T12:10:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.812+0000 7f85d2d08700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc0b3840 con 0x7f85cc0b1760 2020-11-20T12:10:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.812+0000 7f85d2d08700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc0b3ae0 con 0x7f85cc0b1380 2020-11-20T12:10:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1505700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc0b1760 0x7f85cc0b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1505700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc0b1760 0x7f85cc0b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45936/0 (socket says 172.21.15.187:45936) 2020-11-20T12:10:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1505700 1 -- 172.21.15.187:0/2261835994 learned_addr learned my addr 172.21.15.187:0/2261835994 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1d06700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1380 0x7f85cc0a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d2507700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85cc0b7420 0x7f85cc15f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1505700 1 -- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1380 msgr2=0x7f85cc0a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1505700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1380 0x7f85cc0a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1505700 1 -- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85cc0b7420 msgr2=0x7f85cc15f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1505700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85cc0b7420 0x7f85cc15f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d1505700 1 -- 172.21.15.187:0/2261835994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc0b7860 con 0x7f85cc0b1760 2020-11-20T12:10:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.813+0000 7f85d2507700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85cc0b7420 0x7f85cc15f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.814+0000 7f85d1505700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc0b1760 0x7f85cc0b42c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7f85c8009770 tx=0x7f85c8006ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.814+0000 7f85bdffb700 1 -- 172.21.15.187:0/2261835994 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c800d640 con 0x7f85cc0b1760 2020-11-20T12:10:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.814+0000 7f85bdffb700 1 -- 172.21.15.187:0/2261835994 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c8000e90 con 0x7f85cc0b1760 2020-11-20T12:10:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.814+0000 7f85bdffb700 1 -- 172.21.15.187:0/2261835994 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c800da10 con 0x7f85cc0b1760 2020-11-20T12:10:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.814+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85cc15fcb0 con 0x7f85cc0b1760 2020-11-20T12:10:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.814+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85cc09a880 con 0x7f85cc0b1760 2020-11-20T12:10:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.815+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85cc0b7ab0 con 0x7f85cc0b1760 2020-11-20T12:10:49.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.816+0000 7f85bdffb700 1 -- 172.21.15.187:0/2261835994 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f85c8017030 con 0x7f85cc0b1760 2020-11-20T12:10:49.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.816+0000 7f85bdffb700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85a4056d20 0x7f85a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:49.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.817+0000 7f85bdffb700 1 -- 172.21.15.187:0/2261835994 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f85c806f150 con 0x7f85cc0b1760 2020-11-20T12:10:49.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.817+0000 7f85d1d06700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85a4056d20 0x7f85a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:49.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.817+0000 7f85d1d06700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85a4056d20 0x7f85a40591e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f85c0003500 tx=0x7f85c0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:49.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:49.820+0000 7f85bdffb700 1 -- 172.21.15.187:0/2261835994 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f85c8044600 con 0x7f85cc0b1760 2020-11-20T12:10:50.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.006+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f85cc0b47b0 con 0x7f85cc0b1760 2020-11-20T12:10:50.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.007+0000 7f85bdffb700 1 -- 172.21.15.187:0/2261835994 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7f85c8047c20 con 0x7f85cc0b1760 2020-11-20T12:10:50.012 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:50.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.009+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85a4056d20 msgr2=0x7f85a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.009+0000 7f85d2d08700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85a4056d20 0x7f85a40591e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f85c0003500 tx=0x7f85c0009040).stop 2020-11-20T12:10:50.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.009+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc0b1760 msgr2=0x7f85cc0b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.009+0000 7f85d2d08700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc0b1760 0x7f85cc0b42c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7f85c8009770 tx=0x7f85c8006ec0).stop 2020-11-20T12:10:50.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.010+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 shutdown_connections 2020-11-20T12:10:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.010+0000 7f85d2d08700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85a4056d20 0x7f85a40591e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.010+0000 7f85d2d08700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85cc0b7420 0x7f85cc15f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.010+0000 7f85d2d08700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85cc0b1380 0x7f85cc0a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.010+0000 7f85d2d08700 1 --2- 172.21.15.187:0/2261835994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85cc0b1760 0x7f85cc0b42c0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.010+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 >> 172.21.15.187:0/2261835994 conn(0x7f85cc0bab30 msgr2=0x7f85cc14c030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:50.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.010+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 shutdown_connections 2020-11-20T12:10:50.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.010+0000 7f85d2d08700 1 -- 172.21.15.187:0/2261835994 wait complete. 2020-11-20T12:10:50.034 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:10:50.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.275+0000 7fe4d71b8700 1 Processor -- start 2020-11-20T12:10:50.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.276+0000 7fe4d71b8700 1 -- start start 2020-11-20T12:10:50.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.277+0000 7fe4d71b8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe4d0065570 0x7fe4d009d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.277+0000 7fe4d71b8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d00b13e0 con 0x7fe4d0065190 2020-11-20T12:10:50.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.277+0000 7fe4d71b8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d009d690 con 0x7fe4d0065f90 2020-11-20T12:10:50.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.277+0000 7fe4d71b8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d014ab30 con 0x7fe4d0065570 2020-11-20T12:10:50.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.277+0000 7fe4d59b5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe4d0065f90 0x7fe4d009ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42550/0 (socket says 172.21.15.187:42550) 2020-11-20T12:10:50.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.277+0000 7fe4d59b5700 1 -- 172.21.15.187:0/774571414 learned_addr learned my addr 172.21.15.187:0/774571414 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:50.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.278+0000 7fe4d69b7700 1 --2- 172.21.15.187:0/774571414 >> v2:172.21.15.5:3300/0 conn(0x7fe4d0065570 0x7fe4d009d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.278+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366058740 0 0) 0x7fe4d009d690 con 0x7fe4d0065f90 2020-11-20T12:10:50.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.278+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8003240 con 0x7fe4d0065f90 2020-11-20T12:10:50.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.278+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2385954617 0 0) 0x7fe4b8003240 con 0x7fe4d0065f90 2020-11-20T12:10:50.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.278+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 >> v2:172.21.15.5:3300/0 conn(0x7fe4d0065570 msgr2=0x7fe4d009d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:50.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.278+0000 7fe4d51b4700 1 --2- 172.21.15.187:0/774571414 >> v2:172.21.15.5:3300/0 conn(0x7fe4d0065570 0x7fe4d009d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.278+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 >> v1:172.21.15.5:6789/0 conn(0x7fe4d0065190 legacy=0x7fe4d0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.278+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d0062c10 con 0x7fe4d0065f90 2020-11-20T12:10:50.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fe4c00014e0 con 0x7fe4d0065f90 2020-11-20T12:10:50.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 >> v1:172.21.15.187:6789/0 conn(0x7fe4d0065f90 legacy=0x7fe4d009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d51b4700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b8003730 0x7fe4b8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d51b4700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b80044e0 0x7fe4b8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d51b4700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4b8004e50 0x7fe4b800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d014ab30 con 0x7fe4b8004e50 2020-11-20T12:10:50.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b8004280 con 0x7fe4b8003730 2020-11-20T12:10:50.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b800bc20 con 0x7fe4b80044e0 2020-11-20T12:10:50.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.279+0000 7fe4d61b6700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b8003730 0x7fe4b8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d69b7700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4b8004e50 0x7fe4b800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d59b5700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b80044e0 0x7fe4b8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d61b6700 1 -- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b80044e0 msgr2=0x7fe4b8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:50.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d61b6700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b80044e0 0x7fe4b8004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d61b6700 1 -- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4b8004e50 msgr2=0x7fe4b800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:50.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d61b6700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4b8004e50 0x7fe4b800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d61b6700 1 -- 172.21.15.187:0/774571414 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b8003240 con 0x7fe4b8003730 2020-11-20T12:10:50.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d69b7700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4b8004e50 0x7fe4b800b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:50.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d69b7700 1 -- 172.21.15.187:0/774571414 reap_dead start 2020-11-20T12:10:50.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d59b5700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b80044e0 0x7fe4b8004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:50.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d61b6700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b8003730 0x7fe4b8003b70 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7fe4cc001920 tx=0x7fe4cc007b90).ready entity=mon.1 client_cookie=1801fd29751fc4c6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:50.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.280+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc007020 con 0x7fe4b8003730 2020-11-20T12:10:50.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc007180 con 0x7fe4b8003730 2020-11-20T12:10:50.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d51b4700 1 -- 172.21.15.187:0/774571414 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc00c620 con 0x7fe4b8003730 2020-11-20T12:10:50.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d71b8700 1 -- 172.21.15.187:0/774571414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b8003730 msgr2=0x7fe4b8003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d71b8700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b8003730 0x7fe4b8003b70 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7fe4cc001920 tx=0x7fe4cc007b90).stop 2020-11-20T12:10:50.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d71b8700 1 -- 172.21.15.187:0/774571414 shutdown_connections 2020-11-20T12:10:50.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d71b8700 1 --2- 172.21.15.187:0/774571414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b8003730 0x7fe4b8003b70 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d71b8700 1 -- 172.21.15.187:0/774571414 >> 172.21.15.187:0/774571414 conn(0x7fe4d00bab30 msgr2=0x7fe4d00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:50.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d71b8700 1 -- 172.21.15.187:0/774571414 shutdown_connections 2020-11-20T12:10:50.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.281+0000 7fe4d71b8700 1 -- 172.21.15.187:0/774571414 wait complete. 2020-11-20T12:10:50.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.283+0000 7fe4d71b8700 1 Processor -- start 2020-11-20T12:10:50.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.283+0000 7fe4d71b8700 1 -- start start 2020-11-20T12:10:50.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.284+0000 7fe4d71b8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b8003730 0x7fe4d00a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.284+0000 7fe4d71b8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4d00991d0 0x7fe4d0067720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d71b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4d01432c0 0x7fe4d0143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d71b8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00a0910 con 0x7fe4d00991d0 2020-11-20T12:10:50.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d71b8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00a06b0 con 0x7fe4d01432c0 2020-11-20T12:10:50.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d71b8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00b2d80 con 0x7fe4b8003730 2020-11-20T12:10:50.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d69b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4d01432c0 0x7fe4d0143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d69b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4d01432c0 0x7fe4d0143740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45958/0 (socket says 172.21.15.187:45958) 2020-11-20T12:10:50.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d69b7700 1 -- 172.21.15.187:0/702918274 learned_addr learned my addr 172.21.15.187:0/702918274 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:50.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d61b6700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b8003730 0x7fe4d00a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.285+0000 7fe4d59b5700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4d00991d0 0x7fe4d0067720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d69b7700 1 -- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b8003730 msgr2=0x7fe4d00a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d69b7700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b8003730 0x7fe4d00a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d69b7700 1 -- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4d00991d0 msgr2=0x7fe4d0067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d69b7700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4d00991d0 0x7fe4d0067720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d69b7700 1 -- 172.21.15.187:0/702918274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d00b3020 con 0x7fe4d01432c0 2020-11-20T12:10:50.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d61b6700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b8003730 0x7fe4d00a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:50.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d69b7700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4d01432c0 0x7fe4d0143740 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7fe4d0066370 tx=0x7fe4c8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:50.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4be7fc700 1 -- 172.21.15.187:0/702918274 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c800d6f0 con 0x7fe4d01432c0 2020-11-20T12:10:50.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4be7fc700 1 -- 172.21.15.187:0/702918274 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c800d850 con 0x7fe4d01432c0 2020-11-20T12:10:50.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d0067c10 con 0x7fe4d01432c0 2020-11-20T12:10:50.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4be7fc700 1 -- 172.21.15.187:0/702918274 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c800dbf0 con 0x7fe4d01432c0 2020-11-20T12:10:50.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.286+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d00b1df0 con 0x7fe4d01432c0 2020-11-20T12:10:50.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.288+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d00b31b0 con 0x7fe4d01432c0 2020-11-20T12:10:50.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.288+0000 7fe4be7fc700 1 -- 172.21.15.187:0/702918274 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe4c801f460 con 0x7fe4d01432c0 2020-11-20T12:10:50.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.289+0000 7fe4be7fc700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a8056d50 0x7fe4a8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.289+0000 7fe4be7fc700 1 -- 172.21.15.187:0/702918274 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe4c80706b0 con 0x7fe4d01432c0 2020-11-20T12:10:50.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.289+0000 7fe4d61b6700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a8056d50 0x7fe4a8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.290+0000 7fe4d61b6700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a8056d50 0x7fe4a8059210 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe4cc000f90 tx=0x7fe4cc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:50.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.292+0000 7fe4be7fc700 1 -- 172.21.15.187:0/702918274 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe4c8045940 con 0x7fe4d01432c0 2020-11-20T12:10:50.334 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:10:50.344 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:10:50.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.481+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4d0061a20 con 0x7fe4d01432c0 2020-11-20T12:10:50.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.481+0000 7fe4be7fc700 1 -- 172.21.15.187:0/702918274 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe4c8048f60 con 0x7fe4d01432c0 2020-11-20T12:10:50.485 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:50.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.483+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a8056d50 msgr2=0x7fe4a8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.483+0000 7fe4d71b8700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a8056d50 0x7fe4a8059210 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe4cc000f90 tx=0x7fe4cc015000).stop 2020-11-20T12:10:50.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.483+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4d01432c0 msgr2=0x7fe4d0143740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.483+0000 7fe4d71b8700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4d01432c0 0x7fe4d0143740 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7fe4d0066370 tx=0x7fe4c8008040).stop 2020-11-20T12:10:50.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.484+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 shutdown_connections 2020-11-20T12:10:50.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.484+0000 7fe4d71b8700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe4a8056d50 0x7fe4a8059210 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.484+0000 7fe4d71b8700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4d01432c0 0x7fe4d0143740 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.484+0000 7fe4d71b8700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4b8003730 0x7fe4d00a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.484+0000 7fe4d71b8700 1 --2- 172.21.15.187:0/702918274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4d00991d0 0x7fe4d0067720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.484+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 >> 172.21.15.187:0/702918274 conn(0x7fe4d00bab30 msgr2=0x7fe4d00baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:50.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.484+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 shutdown_connections 2020-11-20T12:10:50.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.484+0000 7fe4d71b8700 1 -- 172.21.15.187:0/702918274 wait complete. 2020-11-20T12:10:50.512 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:10:50.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.804+0000 7efe0d2a4700 1 Processor -- start 2020-11-20T12:10:50.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.804+0000 7efe0d2a4700 1 -- start start 2020-11-20T12:10:50.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.805+0000 7efe0d2a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1380 0x7efe08150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.805+0000 7efe0d2a4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efe0814c260 0x7efe0814c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.805+0000 7efe0d2a4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe080658a0 con 0x7efe080b1760 2020-11-20T12:10:50.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.805+0000 7efe0d2a4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe08150390 con 0x7efe0814c260 2020-11-20T12:10:50.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.805+0000 7efe0d2a4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe08150630 con 0x7efe080b1380 2020-11-20T12:10:50.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.806+0000 7efe0caa3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efe0814c260 0x7efe0814c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.806+0000 7efe0caa3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efe0814c260 0x7efe0814c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45964/0 (socket says 172.21.15.187:45964) 2020-11-20T12:10:50.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.806+0000 7efe0caa3700 1 -- 172.21.15.187:0/2199036883 learned_addr learned my addr 172.21.15.187:0/2199036883 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:50.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.806+0000 7efe07fff700 1 --2- 172.21.15.187:0/2199036883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1380 0x7efe08150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.806+0000 7efe0caa3700 1 -- 172.21.15.187:0/2199036883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1380 msgr2=0x7efe08150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:50.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.806+0000 7efe0caa3700 1 --2- 172.21.15.187:0/2199036883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1380 0x7efe08150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.806+0000 7efe0caa3700 1 -- 172.21.15.187:0/2199036883 >> v1:172.21.15.5:6789/0 conn(0x7efe080b1760 legacy=0x7efe08065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:50.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.806+0000 7efe0caa3700 1 -- 172.21.15.187:0/2199036883 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe08068510 con 0x7efe0814c260 2020-11-20T12:10:50.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.807+0000 7efe0caa3700 1 --2- 172.21.15.187:0/2199036883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe0814c260 0x7efe0814c6f0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7efe00006fd0 tx=0x7efe00006eb0).ready entity=mon.1 client_cookie=218a7ec390e754b7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:50.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.807+0000 7efe077fe700 1 -- 172.21.15.187:0/2199036883 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe00009890 con 0x7efe0814c260 2020-11-20T12:10:50.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.807+0000 7efe077fe700 1 -- 172.21.15.187:0/2199036883 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe000099f0 con 0x7efe0814c260 2020-11-20T12:10:50.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.807+0000 7efe077fe700 1 -- 172.21.15.187:0/2199036883 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe00009d70 con 0x7efe0814c260 2020-11-20T12:10:50.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.808+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2199036883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe0814c260 msgr2=0x7efe0814c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.808+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2199036883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe0814c260 0x7efe0814c6f0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7efe00006fd0 tx=0x7efe00006eb0).stop 2020-11-20T12:10:50.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.808+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2199036883 shutdown_connections 2020-11-20T12:10:50.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.808+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2199036883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe0814c260 0x7efe0814c6f0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.808+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2199036883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1380 0x7efe08150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.808+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2199036883 >> 172.21.15.187:0/2199036883 conn(0x7efe080bab30 msgr2=0x7efe080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:50.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.808+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2199036883 shutdown_connections 2020-11-20T12:10:50.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.808+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2199036883 wait complete. 2020-11-20T12:10:50.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.810+0000 7efe0d2a4700 1 Processor -- start 2020-11-20T12:10:50.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.810+0000 7efe0d2a4700 1 -- start start 2020-11-20T12:10:50.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.810+0000 7efe0d2a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b1380 0x7efe0813f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe0d2a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1760 0x7efe0813f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe0d2a4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe0814c260 0x7efe0809dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe0d2a4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe080b49f0 con 0x7efe0814c260 2020-11-20T12:10:50.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe0d2a4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe080b5dd0 con 0x7efe080b1380 2020-11-20T12:10:50.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe0d2a4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe080b42b0 con 0x7efe080b1760 2020-11-20T12:10:50.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe07fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b1380 0x7efe0813f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe07fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b1380 0x7efe0813f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45968/0 (socket says 172.21.15.187:45968) 2020-11-20T12:10:50.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe07fff700 1 -- 172.21.15.187:0/2119063738 learned_addr learned my addr 172.21.15.187:0/2119063738 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:50.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efdff7fe700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1760 0x7efe0813f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.811+0000 7efe0caa3700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe0814c260 0x7efe0809dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe07fff700 1 -- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1760 msgr2=0x7efe0813f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe07fff700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1760 0x7efe0813f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe07fff700 1 -- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe0814c260 msgr2=0x7efe0809dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:50.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe07fff700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe0814c260 0x7efe0809dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:50.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe07fff700 1 -- 172.21.15.187:0/2119063738 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe080b4550 con 0x7efe080b1380 2020-11-20T12:10:50.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efdff7fe700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1760 0x7efe0813f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:50.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe07fff700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b1380 0x7efe0813f3f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7efdf800ec90 tx=0x7efdf8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:50.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe04ff9700 1 -- 172.21.15.187:0/2119063738 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf800c940 con 0x7efe080b1380 2020-11-20T12:10:50.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe04ff9700 1 -- 172.21.15.187:0/2119063738 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf800caa0 con 0x7efe080b1380 2020-11-20T12:10:50.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe04ff9700 1 -- 172.21.15.187:0/2119063738 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf800ce20 con 0x7efe080b1380 2020-11-20T12:10:50.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.812+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe080b5a10 con 0x7efe080b1380 2020-11-20T12:10:50.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.813+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0809f4d0 con 0x7efe080b1380 2020-11-20T12:10:50.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.814+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe080b5c30 con 0x7efe080b1380 2020-11-20T12:10:50.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.814+0000 7efe04ff9700 1 -- 172.21.15.187:0/2119063738 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efdf8013640 con 0x7efe080b1380 2020-11-20T12:10:50.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.815+0000 7efe04ff9700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:50.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.815+0000 7efe04ff9700 1 -- 172.21.15.187:0/2119063738 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7efdf80700a0 con 0x7efe080b1380 2020-11-20T12:10:50.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.815+0000 7efdff7fe700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:50.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.816+0000 7efdff7fe700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7efe080a0400 tx=0x7efdf0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:50.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:50.819+0000 7efe04ff9700 1 -- 172.21.15.187:0/2119063738 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efe080b5c30 con 0x7efe080b1380 2020-11-20T12:10:51.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.004+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efe080b5c00 con 0x7efe080b1380 2020-11-20T12:10:51.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.005+0000 7efe04ff9700 1 -- 172.21.15.187:0/2119063738 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+10 (secure 0 0 0) 0x7efdf80454a0 con 0x7efe080b1380 2020-11-20T12:10:51.009 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:10:51.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 msgr2=0x7efde4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7efe080a0400 tx=0x7efdf0008040).stop 2020-11-20T12:10:51.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b1380 msgr2=0x7efe0813f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b1380 0x7efe0813f3f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7efdf800ec90 tx=0x7efdf8008040).stop 2020-11-20T12:10:51.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 shutdown_connections 2020-11-20T12:10:51.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efde4056d50 0x7efde4059210 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe0814c260 0x7efe0809dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe080b1380 0x7efe0813f3f0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 --2- 172.21.15.187:0/2119063738 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe080b1760 0x7efe0813f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.007+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 >> 172.21.15.187:0/2119063738 conn(0x7efe080bab30 msgr2=0x7efe0814b740 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:51.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.008+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 shutdown_connections 2020-11-20T12:10:51.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.008+0000 7efe0d2a4700 1 -- 172.21.15.187:0/2119063738 wait complete. 2020-11-20T12:10:51.032 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:10:51.032 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:51.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.273+0000 7f8d5af95700 1 Processor -- start 2020-11-20T12:10:51.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.273+0000 7f8d5af95700 1 -- start start 2020-11-20T12:10:51.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.275+0000 7f8d5af95700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 0x7f8d54099ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:51.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.275+0000 7f8d5af95700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d540b0310 con 0x7f8d54062820 2020-11-20T12:10:51.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.275+0000 7f8d5af95700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d54149190 con 0x7f8d54062c00 2020-11-20T12:10:51.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.275+0000 7f8d5af95700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d54149430 con 0x7f8d540b5130 2020-11-20T12:10:51.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.275+0000 7f8d59792700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8d54062c00 0x7f8d540b0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42584/0 (socket says 172.21.15.187:42584) 2020-11-20T12:10:51.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.275+0000 7f8d59792700 1 -- 172.21.15.187:0/1031593922 learned_addr learned my addr 172.21.15.187:0/1031593922 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:51.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d5a794700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 0x7f8d54099ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:51.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678496985 0 0) 0x7f8d54149190 con 0x7f8d54062c00 2020-11-20T12:10:51.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c003240 con 0x7f8d54062c00 2020-11-20T12:10:51.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703901506 0 0) 0x7f8d540b0310 con 0x7f8d54062820 2020-11-20T12:10:51.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d54149190 con 0x7f8d54062820 2020-11-20T12:10:51.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2260153108 0 0) 0x7f8d3c003240 con 0x7f8d54062c00 2020-11-20T12:10:51.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 msgr2=0x7f8d54099ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:51.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 0x7f8d54099ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 >> v1:172.21.15.5:6789/0 conn(0x7f8d54062820 legacy=0x7f8d54144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.276+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54144d30 con 0x7f8d54062c00 2020-11-20T12:10:51.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8d500014e0 con 0x7f8d54062c00 2020-11-20T12:10:51.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 >> v1:172.21.15.187:6789/0 conn(0x7f8d54062c00 legacy=0x7f8d540b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d58f91700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 0x7f8d3c0038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:51.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d58f91700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d3c003db0 0x7f8d3c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:51.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d58f91700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d3c0049d0 0x7f8d3c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:51.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d54149430 con 0x7f8d3c003db0 2020-11-20T12:10:51.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c005340 con 0x7f8d3c0049d0 2020-11-20T12:10:51.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c00b760 con 0x7f8d540b5130 2020-11-20T12:10:51.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.277+0000 7f8d5a794700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d3c0049d0 0x7f8d3c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:51.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.278+0000 7f8d59792700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d3c003db0 0x7f8d3c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:51.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.278+0000 7f8d59f93700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 0x7f8d3c0038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:51.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.278+0000 7f8d5a794700 1 -- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 msgr2=0x7f8d3c0038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:51.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.278+0000 7f8d5a794700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 0x7f8d3c0038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.278+0000 7f8d5a794700 1 -- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d3c003db0 msgr2=0x7f8d3c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:51.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.278+0000 7f8d5a794700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d3c003db0 0x7f8d3c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.278+0000 7f8d5a794700 1 -- 172.21.15.187:0/1031593922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c003240 con 0x7f8d3c0049d0 2020-11-20T12:10:51.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.278+0000 7f8d59f93700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d540b5130 0x7f8d3c0038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:51.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.279+0000 7f8d5a794700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d3c0049d0 0x7f8d3c004e50 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f8d4c009e70 tx=0x7f8d4c0091d0).ready entity=mon.1 client_cookie=3070186edce89212 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:51.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.279+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c0095c0 con 0x7f8d3c0049d0 2020-11-20T12:10:51.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.279+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c00ed70 con 0x7f8d3c0049d0 2020-11-20T12:10:51.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.279+0000 7f8d58f91700 1 -- 172.21.15.187:0/1031593922 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c00eee0 con 0x7f8d3c0049d0 2020-11-20T12:10:51.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.280+0000 7f8d5af95700 1 -- 172.21.15.187:0/1031593922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d3c0049d0 msgr2=0x7f8d3c004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.280+0000 7f8d5af95700 1 --2- 172.21.15.187:0/1031593922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d3c0049d0 0x7f8d3c004e50 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f8d4c009e70 tx=0x7f8d4c0091d0).stop 2020-11-20T12:10:51.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.280+0000 7f8d5a794700 1 -- 172.21.15.187:0/1031593922 reap_dead start 2020-11-20T12:10:51.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.280+0000 7f8d5af95700 1 -- 172.21.15.187:0/1031593922 shutdown_connections 2020-11-20T12:10:51.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.280+0000 7f8d5af95700 1 -- 172.21.15.187:0/1031593922 >> 172.21.15.187:0/1031593922 conn(0x7f8d540bab30 msgr2=0x7f8d540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.280+0000 7f8d5af95700 1 -- 172.21.15.187:0/1031593922 shutdown_connections 2020-11-20T12:10:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.280+0000 7f8d5af95700 1 -- 172.21.15.187:0/1031593922 wait complete. 2020-11-20T12:10:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.281+0000 7f8d5af95700 1 Processor -- start 2020-11-20T12:10:51.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.282+0000 7f8d5af95700 1 -- start start 2020-11-20T12:10:51.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.282+0000 7f8d5af95700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d541372d0 0x7f8d540b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:51.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.282+0000 7f8d5af95700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d541376b0 0x7f8d540b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:51.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.282+0000 7f8d5af95700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5409d270 0x7f8d540b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:51.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.282+0000 7f8d5af95700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540b6430 con 0x7f8d541376b0 2020-11-20T12:10:51.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.282+0000 7f8d5af95700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5409ac00 con 0x7f8d5409d270 2020-11-20T12:10:51.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.282+0000 7f8d5af95700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5409aea0 con 0x7f8d541372d0 2020-11-20T12:10:51.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.283+0000 7f8d5a794700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5409d270 0x7f8d540b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:51.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.283+0000 7f8d5a794700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5409d270 0x7f8d540b5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45992/0 (socket says 172.21.15.187:45992) 2020-11-20T12:10:51.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.283+0000 7f8d5a794700 1 -- 172.21.15.187:0/220421857 learned_addr learned my addr 172.21.15.187:0/220421857 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:51.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.283+0000 7f8d5a794700 1 -- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d541372d0 msgr2=0x7f8d540b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.283+0000 7f8d59f93700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d541372d0 0x7f8d540b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:51.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.283+0000 7f8d5a794700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d541372d0 0x7f8d540b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.283+0000 7f8d5a794700 1 -- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d541376b0 msgr2=0x7f8d540b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.283+0000 7f8d59792700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d541376b0 0x7f8d540b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:51.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d5a794700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d541376b0 0x7f8d540b5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d5a794700 1 -- 172.21.15.187:0/220421857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5409cdd0 con 0x7f8d5409d270 2020-11-20T12:10:51.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d59792700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d541376b0 0x7f8d540b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:10:51.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d5a794700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5409d270 0x7f8d540b5f40 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f8d4c0011a0 tx=0x7f8d4c01a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:51.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d45ffb700 1 -- 172.21.15.187:0/220421857 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c0095c0 con 0x7f8d5409d270 2020-11-20T12:10:51.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d45ffb700 1 -- 172.21.15.187:0/220421857 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c0243e0 con 0x7f8d5409d270 2020-11-20T12:10:51.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5409a880 con 0x7f8d5409d270 2020-11-20T12:10:51.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d45ffb700 1 -- 172.21.15.187:0/220421857 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c024760 con 0x7f8d5409d270 2020-11-20T12:10:51.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.284+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d540b8400 con 0x7f8d5409d270 2020-11-20T12:10:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.285+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5409aa10 con 0x7f8d5409d270 2020-11-20T12:10:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.286+0000 7f8d45ffb700 1 -- 172.21.15.187:0/220421857 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8d4c02c070 con 0x7f8d5409d270 2020-11-20T12:10:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.286+0000 7f8d45ffb700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d2c056d20 0x7f8d2c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.287+0000 7f8d45ffb700 1 -- 172.21.15.187:0/220421857 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8d4c072430 con 0x7f8d5409d270 2020-11-20T12:10:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.287+0000 7f8d59f93700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d2c056d20 0x7f8d2c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:51.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.287+0000 7f8d59f93700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d2c056d20 0x7f8d2c0591e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8d54137c90 tx=0x7f8d48009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:51.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.290+0000 7f8d45ffb700 1 -- 172.21.15.187:0/220421857 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8d4c047830 con 0x7f8d5409d270 2020-11-20T12:10:51.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.533+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d54061a20 con 0x7f8d5409d270 2020-11-20T12:10:51.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.534+0000 7f8d45ffb700 1 -- 172.21.15.187:0/220421857 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f8d4c016020 con 0x7f8d5409d270 2020-11-20T12:10:51.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d2c056d20 msgr2=0x7f8d2c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d2c056d20 0x7f8d2c0591e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8d54137c90 tx=0x7f8d48009040).stop 2020-11-20T12:10:51.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5409d270 msgr2=0x7f8d540b5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:51.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5409d270 0x7f8d540b5f40 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f8d4c0011a0 tx=0x7f8d4c01a000).stop 2020-11-20T12:10:51.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 shutdown_connections 2020-11-20T12:10:51.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d2c056d20 0x7f8d2c0591e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5409d270 0x7f8d540b5f40 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d541372d0 0x7f8d540b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 --2- 172.21.15.187:0/220421857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d541376b0 0x7f8d540b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:51.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 >> 172.21.15.187:0/220421857 conn(0x7f8d540bab30 msgr2=0x7f8d540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:51.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 shutdown_connections 2020-11-20T12:10:51.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:51.537+0000 7f8d5af95700 1 -- 172.21.15.187:0/220421857 wait complete. 2020-11-20T12:10:51.552 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:10:51.553 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:51.553 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:51.771 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:10:23.312121+0000', 'modified': '2020-11-20T12:10:46.446562+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 72, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5036}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5036': {'gid': 5036, 'name': 'b', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.5:6833/3088698963', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 3088698963}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 3088698963}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:10:52.251 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:10:52.252 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2020-11-20T12:10:52.552 INFO:teuthology.orchestra.run.smithi005.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":5036,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":9.0866383880000008,"mdsmap_epoch":55,"osdmap_epoch":72,"osdmap_epoch_barrier":72,"uptime":15.136405097000001} 2020-11-20T12:10:52.553 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5036, "mdsmap_epoch": 55, "osdmap_epoch": 72, "osdmap_epoch_barrier": 72, "rank_uptime": 9.086638388, "state": "up:active", "uptime": 15.136405097, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:10:52.554 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:10:52.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:52.554 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:10:52.621 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:10:52.622 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:10:52.767 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:10:52.768 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:10:52.807 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:10:52.807 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:10:52.816 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:10:52.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:10:52.817 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:10:52.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:10:52.917 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -f dir_1/file_a) 2020-11-20T12:10:53.025 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:53.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.269+0000 7fbd4551f700 1 Processor -- start 2020-11-20T12:10:53.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.269+0000 7fbd4551f700 1 -- start start 2020-11-20T12:10:53.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.270+0000 7fbd4551f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40065f90 0x7fbd40065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.270+0000 7fbd4551f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbd4009ca90 0x7fbd4009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.270+0000 7fbd4551f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd400b13e0 con 0x7fbd40065190 2020-11-20T12:10:53.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.270+0000 7fbd4551f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4014ab30 con 0x7fbd4009ca90 2020-11-20T12:10:53.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.270+0000 7fbd4551f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd40062c10 con 0x7fbd40065f90 2020-11-20T12:10:53.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.271+0000 7fbd3ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbd40065190 0x7fbd40143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42606/0 (socket says 172.21.15.187:42606) 2020-11-20T12:10:53.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.271+0000 7fbd3ffff700 1 -- 172.21.15.187:0/1569135698 learned_addr learned my addr 172.21.15.187:0/1569135698 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:10:53.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.271+0000 7fbd44d1e700 1 --2- 172.21.15.187:0/1569135698 >> v2:172.21.15.5:3300/0 conn(0x7fbd4009ca90 0x7fbd4009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:53.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.271+0000 7fbd3f7fe700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40065f90 0x7fbd40065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:53.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.271+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520879648 0 0) 0x7fbd400b13e0 con 0x7fbd40065190 2020-11-20T12:10:53.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.271+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2c003240 con 0x7fbd40065190 2020-11-20T12:10:53.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3584743000 0 0) 0x7fbd2c003240 con 0x7fbd40065190 2020-11-20T12:10:53.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40065f90 msgr2=0x7fbd40065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:53.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40065f90 0x7fbd40065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 >> v2:172.21.15.5:3300/0 conn(0x7fbd4009ca90 msgr2=0x7fbd4009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:53.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 --2- 172.21.15.187:0/1569135698 >> v2:172.21.15.5:3300/0 conn(0x7fbd4009ca90 0x7fbd4009cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd400b1640 con 0x7fbd40065190 2020-11-20T12:10:53.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fbd300014e0 con 0x7fbd40065190 2020-11-20T12:10:53.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 >> v1:172.21.15.187:6789/0 conn(0x7fbd40065190 legacy=0x7fbd40143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:53.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40065f90 0x7fbd2c0037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd2c003d30 0x7fbd2c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd2c0049d0 0x7fbd2c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4014ab30 con 0x7fbd2c0049d0 2020-11-20T12:10:53.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd40062c10 con 0x7fbd2c003d30 2020-11-20T12:10:53.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.272+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c0042f0 con 0x7fbd40065f90 2020-11-20T12:10:53.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.273+0000 7fbd3f7fe700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd2c003d30 0x7fbd2c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:53.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.273+0000 7fbd3ffff700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40065f90 0x7fbd2c0037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:53.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.273+0000 7fbd3f7fe700 1 -- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40065f90 msgr2=0x7fbd2c0037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:53.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.273+0000 7fbd3f7fe700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40065f90 0x7fbd2c0037d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.273+0000 7fbd3f7fe700 1 -- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd2c0049d0 msgr2=0x7fbd2c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:53.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.273+0000 7fbd3f7fe700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd2c0049d0 0x7fbd2c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:53.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.273+0000 7fbd3f7fe700 1 -- 172.21.15.187:0/1569135698 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c003240 con 0x7fbd2c003d30 2020-11-20T12:10:53.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.273+0000 7fbd3f7fe700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd2c003d30 0x7fbd2c0044e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7fbd3400bc00 tx=0x7fbd34006df0).ready entity=mon.1 client_cookie=14107627b938f5cc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:53.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.274+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3400b310 con 0x7fbd2c003d30 2020-11-20T12:10:53.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.274+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3400eae0 con 0x7fbd2c003d30 2020-11-20T12:10:53.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.274+0000 7fbd3effd700 1 -- 172.21.15.187:0/1569135698 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3400ee60 con 0x7fbd2c003d30 2020-11-20T12:10:53.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.274+0000 7fbd4551f700 1 -- 172.21.15.187:0/1569135698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd2c003d30 msgr2=0x7fbd2c0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:53.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.274+0000 7fbd4551f700 1 --2- 172.21.15.187:0/1569135698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd2c003d30 0x7fbd2c0044e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7fbd3400bc00 tx=0x7fbd34006df0).stop 2020-11-20T12:10:53.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.274+0000 7fbd44d1e700 1 -- 172.21.15.187:0/1569135698 reap_dead start 2020-11-20T12:10:53.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.274+0000 7fbd4551f700 1 -- 172.21.15.187:0/1569135698 shutdown_connections 2020-11-20T12:10:53.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.274+0000 7fbd4551f700 1 -- 172.21.15.187:0/1569135698 >> 172.21.15.187:0/1569135698 conn(0x7fbd400bab30 msgr2=0x7fbd400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:53.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.275+0000 7fbd4551f700 1 -- 172.21.15.187:0/1569135698 shutdown_connections 2020-11-20T12:10:53.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.275+0000 7fbd4551f700 1 -- 172.21.15.187:0/1569135698 wait complete. 2020-11-20T12:10:53.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.277+0000 7fbd4551f700 1 Processor -- start 2020-11-20T12:10:53.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.277+0000 7fbd4551f700 1 -- start start 2020-11-20T12:10:53.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.278+0000 7fbd4551f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd400b4480 0x7fbd40142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.278+0000 7fbd4551f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd400b4860 0x7fbd40143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.278+0000 7fbd4551f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40067150 0x7fbd4015b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.278+0000 7fbd4551f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd40143680 con 0x7fbd400b4860 2020-11-20T12:10:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.278+0000 7fbd4551f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd40149ef0 con 0x7fbd400b4480 2020-11-20T12:10:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.278+0000 7fbd4551f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd400676d0 con 0x7fbd40067150 2020-11-20T12:10:53.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd400b4480 0x7fbd40142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:53.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd400b4480 0x7fbd40142ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46012/0 (socket says 172.21.15.187:46012) 2020-11-20T12:10:53.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 -- 172.21.15.187:0/3480219500 learned_addr learned my addr 172.21.15.187:0/3480219500 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:53.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3f7fe700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd400b4860 0x7fbd40143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:53.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd44d1e700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40067150 0x7fbd4015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:53.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 -- 172.21.15.187:0/3480219500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40067150 msgr2=0x7fbd4015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:53.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40067150 0x7fbd4015b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 -- 172.21.15.187:0/3480219500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd400b4860 msgr2=0x7fbd40143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:53.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd400b4860 0x7fbd40143190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 -- 172.21.15.187:0/3480219500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40067970 con 0x7fbd400b4480 2020-11-20T12:10:53.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.279+0000 7fbd3ffff700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd400b4480 0x7fbd40142ca0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7fbd3000dab0 tx=0x7fbd30009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:53.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.280+0000 7fbd1ffff700 1 -- 172.21.15.187:0/3480219500 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd30007170 con 0x7fbd400b4480 2020-11-20T12:10:53.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.280+0000 7fbd1ffff700 1 -- 172.21.15.187:0/3480219500 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd300072d0 con 0x7fbd400b4480 2020-11-20T12:10:53.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.280+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd40067c10 con 0x7fbd400b4480 2020-11-20T12:10:53.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.280+0000 7fbd1ffff700 1 -- 172.21.15.187:0/3480219500 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3000bb60 con 0x7fbd400b4480 2020-11-20T12:10:53.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.280+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd40146880 con 0x7fbd400b4480 2020-11-20T12:10:53.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.281+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd40061a20 con 0x7fbd400b4480 2020-11-20T12:10:53.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.282+0000 7fbd1ffff700 1 -- 172.21.15.187:0/3480219500 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbd300045c0 con 0x7fbd400b4480 2020-11-20T12:10:53.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.282+0000 7fbd1ffff700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd18056d20 0x7fbd180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:53.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.283+0000 7fbd1ffff700 1 -- 172.21.15.187:0/3480219500 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbd300703a0 con 0x7fbd400b4480 2020-11-20T12:10:53.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.283+0000 7fbd3f7fe700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd18056d20 0x7fbd180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:53.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.284+0000 7fbd3f7fe700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd18056d20 0x7fbd180591e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fbd40045400 tx=0x7fbd34019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:53.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.287+0000 7fbd1ffff700 1 -- 172.21.15.187:0/3480219500 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbd30045720 con 0x7fbd400b4480 2020-11-20T12:10:53.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.530+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd400afc60 con 0x7fbd400b4480 2020-11-20T12:10:53.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.531+0000 7fbd1ffff700 1 -- 172.21.15.187:0/3480219500 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fbd30048d40 con 0x7fbd400b4480 2020-11-20T12:10:53.535 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:53.535 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:53.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.533+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd18056d20 msgr2=0x7fbd180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.533+0000 7fbd4551f700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd18056d20 0x7fbd180591e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fbd40045400 tx=0x7fbd34019000).stop 2020-11-20T12:10:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.533+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd400b4480 msgr2=0x7fbd40142ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.533+0000 7fbd4551f700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd400b4480 0x7fbd40142ca0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7fbd3000dab0 tx=0x7fbd30009040).stop 2020-11-20T12:10:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.534+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 shutdown_connections 2020-11-20T12:10:53.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.534+0000 7fbd4551f700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd18056d20 0x7fbd180591e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.534+0000 7fbd4551f700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd40067150 0x7fbd4015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.534+0000 7fbd4551f700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd400b4480 0x7fbd40142ca0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.534+0000 7fbd4551f700 1 --2- 172.21.15.187:0/3480219500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd400b4860 0x7fbd40143190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:53.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.534+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 >> 172.21.15.187:0/3480219500 conn(0x7fbd400bab30 msgr2=0x7fbd40061fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:53.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.534+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 shutdown_connections 2020-11-20T12:10:53.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:53.534+0000 7fbd4551f700 1 -- 172.21.15.187:0/3480219500 wait complete. 2020-11-20T12:10:53.540 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:10:53.560 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache num_strays 2020-11-20T12:10:53.853 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"num_strays":1}} 2020-11-20T12:10:53.853 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2020-11-20T12:10:53.854 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:54.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.100+0000 7f399e9f8700 1 Processor -- start 2020-11-20T12:10:54.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.100+0000 7f399e9f8700 1 -- start start 2020-11-20T12:10:54.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399e9f8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f39980b1760 0x7f3998065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:54.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399e9f8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3998065680 0x7f399814c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:54.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399e9f8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3998150390 con 0x7f39980b1380 2020-11-20T12:10:54.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399e9f8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39981511e0 con 0x7f3998065680 2020-11-20T12:10:54.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399e9f8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399814d760 con 0x7f39980b1760 2020-11-20T12:10:54.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399d1f5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f39980b1760 0x7f3998065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:54.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399d1f5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f39980b1760 0x7f3998065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46020/0 (socket says 172.21.15.187:46020) 2020-11-20T12:10:54.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399d1f5700 1 -- 172.21.15.187:0/4009044470 learned_addr learned my addr 172.21.15.187:0/4009044470 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:54.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399d1f5700 1 -- 172.21.15.187:0/4009044470 >> v2:172.21.15.5:3300/0 conn(0x7f3998065680 msgr2=0x7f399814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:54.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399d1f5700 1 --2- 172.21.15.187:0/4009044470 >> v2:172.21.15.5:3300/0 conn(0x7f3998065680 0x7f399814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:54.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399d1f5700 1 -- 172.21.15.187:0/4009044470 >> v1:172.21.15.5:6789/0 conn(0x7f39980b1380 legacy=0x7f3998150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:54.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.102+0000 7f399d1f5700 1 -- 172.21.15.187:0/4009044470 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998150630 con 0x7f39980b1760 2020-11-20T12:10:54.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.103+0000 7f399d1f5700 1 --2- 172.21.15.187:0/4009044470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1760 0x7f3998065190 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f3998063a20 tx=0x7f3994006eb0).ready entity=mon.1 client_cookie=1c69bb045143bb99 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:54.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.103+0000 7f399c9f4700 1 -- 172.21.15.187:0/4009044470 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3994009890 con 0x7f39980b1760 2020-11-20T12:10:54.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.103+0000 7f399c9f4700 1 -- 172.21.15.187:0/4009044470 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39940099f0 con 0x7f39980b1760 2020-11-20T12:10:54.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.103+0000 7f399c9f4700 1 -- 172.21.15.187:0/4009044470 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3994009d70 con 0x7f39980b1760 2020-11-20T12:10:54.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.104+0000 7f399e9f8700 1 -- 172.21.15.187:0/4009044470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1760 msgr2=0x7f3998065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:54.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.104+0000 7f399e9f8700 1 --2- 172.21.15.187:0/4009044470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1760 0x7f3998065190 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f3998063a20 tx=0x7f3994006eb0).stop 2020-11-20T12:10:54.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.104+0000 7f399e9f8700 1 -- 172.21.15.187:0/4009044470 shutdown_connections 2020-11-20T12:10:54.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.104+0000 7f399e9f8700 1 --2- 172.21.15.187:0/4009044470 >> v2:172.21.15.5:3300/0 conn(0x7f3998065680 0x7f399814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:54.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.104+0000 7f399e9f8700 1 --2- 172.21.15.187:0/4009044470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1760 0x7f3998065190 secure :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0x7f3998063a20 tx=0x7f3994006eb0).stop 2020-11-20T12:10:54.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.104+0000 7f399e9f8700 1 -- 172.21.15.187:0/4009044470 >> 172.21.15.187:0/4009044470 conn(0x7f39980bab30 msgr2=0x7f39980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:54.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.104+0000 7f399e9f8700 1 -- 172.21.15.187:0/4009044470 shutdown_connections 2020-11-20T12:10:54.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.104+0000 7f399e9f8700 1 -- 172.21.15.187:0/4009044470 wait complete. 2020-11-20T12:10:54.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.105+0000 7f399e9f8700 1 Processor -- start 2020-11-20T12:10:54.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.105+0000 7f399e9f8700 1 -- start start 2020-11-20T12:10:54.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.106+0000 7f399e9f8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1380 0x7f39980b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:54.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.106+0000 7f399e9f8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39980b7260 0x7f399813c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:54.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.106+0000 7f399e9f8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399813cb80 0x7f399815fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:54.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.106+0000 7f399e9f8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399813d000 con 0x7f399813cb80 2020-11-20T12:10:54.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.106+0000 7f399e9f8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980b1ec0 con 0x7f39980b1380 2020-11-20T12:10:54.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.106+0000 7f399e9f8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39980b2160 con 0x7f39980b7260 2020-11-20T12:10:54.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1380 0x7f39980b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:54.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1380 0x7f39980b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46026/0 (socket says 172.21.15.187:46026) 2020-11-20T12:10:54.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 -- 172.21.15.187:0/342156357 learned_addr learned my addr 172.21.15.187:0/342156357 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:10:54.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 -- 172.21.15.187:0/342156357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39980b7260 msgr2=0x7f399813c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:54.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39980b7260 0x7f399813c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:54.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 -- 172.21.15.187:0/342156357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399813cb80 msgr2=0x7f399815fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:54.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399813cb80 0x7f399815fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:54.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 -- 172.21.15.187:0/342156357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39980b2400 con 0x7f39980b1380 2020-11-20T12:10:54.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.107+0000 7f399d9f6700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1380 0x7f39980b8b00 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7f398c0096e0 tx=0x7f398c006d50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:54.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.108+0000 7f3989ffb700 1 -- 172.21.15.187:0/342156357 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c00d560 con 0x7f39980b1380 2020-11-20T12:10:54.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.108+0000 7f3989ffb700 1 -- 172.21.15.187:0/342156357 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c000e90 con 0x7f39980b1380 2020-11-20T12:10:54.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.108+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39980b8ff0 con 0x7f39980b1380 2020-11-20T12:10:54.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.108+0000 7f3989ffb700 1 -- 172.21.15.187:0/342156357 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c00d910 con 0x7f39980b1380 2020-11-20T12:10:54.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.108+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39980b4120 con 0x7f39980b1380 2020-11-20T12:10:54.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.109+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39980b1d00 con 0x7f39980b1380 2020-11-20T12:10:54.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.109+0000 7f3989ffb700 1 -- 172.21.15.187:0/342156357 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f398c00f030 con 0x7f39980b1380 2020-11-20T12:10:54.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.110+0000 7f3989ffb700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:54.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.110+0000 7f3989ffb700 1 -- 172.21.15.187:0/342156357 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f398c06fd00 con 0x7f39980b1380 2020-11-20T12:10:54.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.110+0000 7f399d1f5700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:54.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.111+0000 7f399d1f5700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3998150170 tx=0x7f3994015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:54.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.114+0000 7f3989ffb700 1 -- 172.21.15.187:0/342156357 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f398c073030 con 0x7f39980b1380 2020-11-20T12:10:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.359+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3998061a20 con 0x7f39980b1380 2020-11-20T12:10:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.360+0000 7f3989ffb700 1 -- 172.21.15.187:0/342156357 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f398c045100 con 0x7f39980b1380 2020-11-20T12:10:54.363 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:10:54.363 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:10:54.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.362+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 msgr2=0x7f39700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:54.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.362+0000 7f399e9f8700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3998150170 tx=0x7f3994015000).stop 2020-11-20T12:10:54.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.362+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1380 msgr2=0x7f39980b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:54.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.362+0000 7f399e9f8700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1380 0x7f39980b8b00 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7f398c0096e0 tx=0x7f398c006d50).stop 2020-11-20T12:10:54.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.363+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 shutdown_connections 2020-11-20T12:10:54.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.363+0000 7f399e9f8700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:54.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.363+0000 7f399e9f8700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399813cb80 0x7f399815fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:54.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.363+0000 7f399e9f8700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39980b1380 0x7f39980b8b00 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:54.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.363+0000 7f399e9f8700 1 --2- 172.21.15.187:0/342156357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39980b7260 0x7f399813c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:10:54.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.363+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 >> 172.21.15.187:0/342156357 conn(0x7f39980bab30 msgr2=0x7f39980625a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:54.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.363+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 shutdown_connections 2020-11-20T12:10:54.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:10:54.363+0000 7f399e9f8700 1 -- 172.21.15.187:0/342156357 wait complete. 2020-11-20T12:10:54.369 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:10:54.389 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:10:54.677 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_created":1}} 2020-11-20T12:10:54.678 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2020-11-20T12:11:20.487 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:11:20.506 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:11:24.679 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:11:24.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.772+0000 7f71ad807d00 1 Processor -- start 2020-11-20T12:11:25.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.772+0000 7f71ad807d00 1 -- start start 2020-11-20T12:11:25.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.773+0000 7f71ad807d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e236090 0x56461e249030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.773+0000 7f71ad807d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56461e2dbcf0 0x56461e2103f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.773+0000 7f71ad807d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.773+0000 7f71ad807d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x56461e1bce30 con 0x56461e236090 2020-11-20T12:11:25.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.773+0000 7f71ad807d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x56461e18b810 con 0x56461e2dda60 2020-11-20T12:11:25.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.773+0000 7f71ad807d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x56461e2497d0 con 0x56461e2dbcf0 2020-11-20T12:11:25.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e8ad700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e8ad700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46044/0 (socket says 172.21.15.187:46044) 2020-11-20T12:11:25.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e8ad700 1 -- 172.21.15.187:0/2056376093 learned_addr learned my addr 172.21.15.187:0/2056376093 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:25.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e0ac700 1 --2- 172.21.15.187:0/2056376093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e236090 0x56461e249030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e8ad700 1 -- 172.21.15.187:0/2056376093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56461e2dbcf0 msgr2=0x56461e2103f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:25.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e8ad700 1 --2- 172.21.15.187:0/2056376093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56461e2dbcf0 0x56461e2103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:25.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e8ad700 1 -- 172.21.15.187:0/2056376093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e236090 msgr2=0x56461e249030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:25.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e8ad700 1 --2- 172.21.15.187:0/2056376093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e236090 0x56461e249030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.774+0000 7f719e8ad700 1 -- 172.21.15.187:0/2056376093 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56461e1b0ec0 con 0x56461e2dda60 2020-11-20T12:11:25.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.775+0000 7f719e8ad700 1 --2- 172.21.15.187:0/2056376093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e5960 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f7198001af0 tx=0x7f7198006b40).ready entity=mon.1 client_cookie=a9b5ecb1a6b7fb14 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:25.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.775+0000 7f719d0aa700 1 -- 172.21.15.187:0/2056376093 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71980060d0 con 0x56461e2dda60 2020-11-20T12:11:25.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.775+0000 7f719d0aa700 1 -- 172.21.15.187:0/2056376093 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7198006230 con 0x56461e2dda60 2020-11-20T12:11:25.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.775+0000 7f719d0aa700 1 -- 172.21.15.187:0/2056376093 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719800b880 con 0x56461e2dda60 2020-11-20T12:11:25.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.775+0000 7f71ad807d00 1 -- 172.21.15.187:0/2056376093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 msgr2=0x56461e2e5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.775+0000 7f71ad807d00 1 --2- 172.21.15.187:0/2056376093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e5960 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f7198001af0 tx=0x7f7198006b40).stop 2020-11-20T12:11:25.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.776+0000 7f71ad807d00 1 -- 172.21.15.187:0/2056376093 shutdown_connections 2020-11-20T12:11:25.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.776+0000 7f71ad807d00 1 --2- 172.21.15.187:0/2056376093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e236090 0x56461e249030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.776+0000 7f71ad807d00 1 --2- 172.21.15.187:0/2056376093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e5960 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.776+0000 7f71ad807d00 1 --2- 172.21.15.187:0/2056376093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56461e2dbcf0 0x56461e2103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:25.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.776+0000 7f71ad807d00 1 -- 172.21.15.187:0/2056376093 >> 172.21.15.187:0/2056376093 conn(0x56461e234a90 msgr2=0x56461e210f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:25.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.776+0000 7f71ad807d00 1 -- 172.21.15.187:0/2056376093 shutdown_connections 2020-11-20T12:11:25.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.776+0000 7f71ad807d00 1 -- 172.21.15.187:0/2056376093 wait complete. 2020-11-20T12:11:25.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.777+0000 7f71ad807d00 1 Processor -- start 2020-11-20T12:11:25.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.777+0000 7f71ad807d00 1 -- start start 2020-11-20T12:11:25.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f71ad807d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56461e236090 0x56461e2db350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f71ad807d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e2dbcf0 0x56461e2e4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f71ad807d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f71ad807d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x56461e2f2220 con 0x56461e2dbcf0 2020-11-20T12:11:25.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f71ad807d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x56461e2f24c0 con 0x56461e2dda60 2020-11-20T12:11:25.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f71ad807d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x56461e2f2760 con 0x56461e236090 2020-11-20T12:11:25.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f719e8ad700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f719e8ad700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46050/0 (socket says 172.21.15.187:46050) 2020-11-20T12:11:25.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f719e8ad700 1 -- 172.21.15.187:0/1561122050 learned_addr learned my addr 172.21.15.187:0/1561122050 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:25.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f71958ab700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e2dbcf0 0x56461e2e4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.778+0000 7f719e0ac700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56461e236090 0x56461e2db350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f719e8ad700 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56461e236090 msgr2=0x56461e2db350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f719e8ad700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56461e236090 0x56461e2db350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f719e8ad700 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e2dbcf0 msgr2=0x56461e2e4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f719e8ad700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x56461e2dbcf0 0x56461e2e4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f719e8ad700 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56461e18f250 con 0x56461e2dda60 2020-11-20T12:11:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f719e8ad700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e4560 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f71980012f0 tx=0x7f7198014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f71977fe700 1 -- 172.21.15.187:0/1561122050 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198017030 con 0x56461e2dda60 2020-11-20T12:11:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f71977fe700 1 -- 172.21.15.187:0/1561122050 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7198012c80 con 0x56461e2dda60 2020-11-20T12:11:25.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56461e2db840 con 0x56461e2dda60 2020-11-20T12:11:25.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f71977fe700 1 -- 172.21.15.187:0/1561122050 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198003410 con 0x56461e2dda60 2020-11-20T12:11:25.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.779+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56461e2f2c00 con 0x56461e2dda60 2020-11-20T12:11:25.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.781+0000 7f71977fe700 1 -- 172.21.15.187:0/1561122050 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f719801c070 con 0x56461e2dda60 2020-11-20T12:11:25.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.782+0000 7f71977fe700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71800559a0 0x7f7180057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.782+0000 7f71977fe700 1 -- 172.21.15.187:0/1561122050 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f71980700c0 con 0x56461e2dda60 2020-11-20T12:11:25.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.782+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x56461e2e4a50 0x56461e2e7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.782+0000 7f719e0ac700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71800559a0 0x7f7180057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.782+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:1 14.0 14:00000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x56461e2faf90 con 0x56461e2e4a50 2020-11-20T12:11:25.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.783+0000 7f719e0ac700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71800559a0 0x7f7180057e60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f7190005a50 tx=0x7f7190013000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:25.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.783+0000 7f71958ab700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x56461e2e4a50 0x56461e2e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.784+0000 7f71958ab700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x56461e2e4a50 0x56461e2e7240 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:25.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.786+0000 7f71958ab700 1 -- 172.21.15.187:0/1561122050 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== osd_op_reply(1 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f718800e040 con 0x56461e2e4a50 2020-11-20T12:11:25.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.786+0000 7f71958ab700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f718800db70 0x7f718800f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.786+0000 7f71958ab700 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:2 14.4 14:20000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x56461e2faf90 con 0x7f718800db70 2020-11-20T12:11:25.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.786+0000 7f719e0ac700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f718800db70 0x7f718800f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.787+0000 7f719e0ac700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f718800db70 0x7f718800f810 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:25.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.788+0000 7f719e0ac700 1 -- 172.21.15.187:0/1561122050 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(2 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f719000a540 con 0x7f718800db70 2020-11-20T12:11:25.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.788+0000 7f719e0ac700 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:3 14.2 14:40000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x56461e2faf90 con 0x7f718800db70 2020-11-20T12:11:25.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.789+0000 7f719e0ac700 1 -- 172.21.15.187:0/1561122050 <== osd.2 v2:172.21.15.187:6810/19904 2 ==== osd_op_reply(3 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f719000a540 con 0x7f718800db70 2020-11-20T12:11:25.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.789+0000 7f719e0ac700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f719001e970 0x7f719001d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.789+0000 7f719e0ac700 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:4 14.6 14:60000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x56461e2faf90 con 0x7f719001e970 2020-11-20T12:11:25.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.790+0000 7f71958ab700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f719001e970 0x7f719001d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.792+0000 7f71958ab700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f719001e970 0x7f719001d3c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:25.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.793+0000 7f71958ab700 1 -- 172.21.15.187:0/1561122050 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== osd_op_reply(4 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f718800e040 con 0x7f719001e970 2020-11-20T12:11:25.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.793+0000 7f71958ab700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f71880133f0 0x7f71880137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.793+0000 7f71958ab700 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:5 14.1 14:80000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x7f7188013040 con 0x7f71880133f0 2020-11-20T12:11:25.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.794+0000 7f719e8ad700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f71880133f0 0x7f71880137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.795+0000 7f719e8ad700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f71880133f0 0x7f71880137f0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:25.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.796+0000 7f719e8ad700 1 -- 172.21.15.187:0/1561122050 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(5 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7198047410 con 0x7f71880133f0 2020-11-20T12:11:25.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.796+0000 7f719e8ad700 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:6 14.5 14:a0000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x7f7188013040 con 0x7f719001e970 2020-11-20T12:11:25.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.798+0000 7f71958ab700 1 -- 172.21.15.187:0/1561122050 <== osd.5 v2:172.21.15.5:6809/20155 2 ==== osd_op_reply(6 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f718800e040 con 0x7f719001e970 2020-11-20T12:11:25.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.798+0000 7f71958ab700 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:7 14.3 14:c0000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x7f7188013040 con 0x56461e2e4a50 2020-11-20T12:11:25.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.800+0000 7f71958ab700 1 -- 172.21.15.187:0/1561122050 <== osd.4 v2:172.21.15.5:6808/20156 2 ==== osd_op_reply(7 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f718801f040 con 0x56461e2e4a50 2020-11-20T12:11:25.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.800+0000 7f71958ab700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f7188012890 0x7f718800e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:25.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.800+0000 7f71958ab700 1 -- 172.21.15.187:0/1561122050 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:8 14.7 14:e0000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x7f7188013040 con 0x7f7188012890 2020-11-20T12:11:25.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.800+0000 7f719e0ac700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f7188012890 0x7f718800e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:25.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.801+0000 7f719e0ac700 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f7188012890 0x7f718800e290 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:25.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f719e0ac700 1 -- 172.21.15.187:0/1561122050 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(8 [pgnls start_epoch 72 out=113b] v70'4 uv4 ondisk = 1) v8 ==== 144+0+113 (crc 0 0 0) 0x7f719000a540 con 0x7f7188012890 2020-11-20T12:11:25.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f7188012890 msgr2=0x7f718800e290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f7188012890 0x7f718800e290 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f718800db70 msgr2=0x7f718800f810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f718800db70 0x7f718800f810 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x56461e2e4a50 msgr2=0x56461e2e7240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x56461e2e4a50 0x56461e2e7240 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f719001e970 msgr2=0x7f719001d3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f719001e970 0x7f719001d3c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f719e8ad700 1 -- 172.21.15.187:0/1561122050 reap_dead start 2020-11-20T12:11:25.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f71880133f0 msgr2=0x7f71880137f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f71880133f0 0x7f71880137f0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71800559a0 msgr2=0x7f7180057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.802+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71800559a0 0x7f7180057e60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f7190005a50 tx=0x7f7190013000).stop 2020-11-20T12:11:25.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 msgr2=0x56461e2e4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:25.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e4560 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f71980012f0 tx=0x7f7198014000).stop 2020-11-20T12:11:25.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 shutdown_connections 2020-11-20T12:11:25.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f71880133f0 0x7f71880137f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f719001e970 0x7f719001d3c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56461e2dda60 0x56461e2e4560 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 --2- 172.21.15.187:0/1561122050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71800559a0 0x7f7180057e60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:25.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 >> 172.21.15.187:0/1561122050 conn(0x56461e234a90 msgr2=0x56461e210d30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:25.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 shutdown_connections 2020-11-20T12:11:25.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:24.803+0000 7f71ad807d00 1 -- 172.21.15.187:0/1561122050 wait complete. 2020-11-20T12:11:25.756 INFO:teuthology.orchestra.run.smithi187.stdout:10000000001.00000000 2020-11-20T12:11:25.756 INFO:teuthology.orchestra.run.smithi187.stdout:10000000001.00000001 2020-11-20T12:11:25.757 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2020-11-20T12:11:25.757 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:26.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.021+0000 7f507dcaa700 1 Processor -- start 2020-11-20T12:11:26.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.021+0000 7f507dcaa700 1 -- start start 2020-11-20T12:11:26.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.023+0000 7f507dcaa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f50780b57b0 0x7f5078068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.023+0000 7f507dcaa700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f50781482d0 0x7f5078062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.023+0000 7f507dcaa700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5078144b80 con 0x7f5078062b50 2020-11-20T12:11:26.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.023+0000 7f507dcaa700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50781489e0 con 0x7f50781482d0 2020-11-20T12:11:26.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.023+0000 7f507dcaa700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5078148c80 con 0x7f50780b57b0 2020-11-20T12:11:26.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.024+0000 7f507cca8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f50780b57b0 0x7f5078068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.024+0000 7f507d4a9700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5078062b50 0x7f5078065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42674/0 (socket says 172.21.15.187:42674) 2020-11-20T12:11:26.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.024+0000 7f507d4a9700 1 -- 172.21.15.187:0/3312238236 learned_addr learned my addr 172.21.15.187:0/3312238236 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:26.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.024+0000 7f5077fff700 1 --2- 172.21.15.187:0/3312238236 >> v2:172.21.15.5:3300/0 conn(0x7f50781482d0 0x7f5078062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.024+0000 7f507cca8700 1 -- 172.21.15.187:0/3312238236 >> v2:172.21.15.5:3300/0 conn(0x7f50781482d0 msgr2=0x7f5078062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:26.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.024+0000 7f507cca8700 1 --2- 172.21.15.187:0/3312238236 >> v2:172.21.15.5:3300/0 conn(0x7f50781482d0 0x7f5078062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.024+0000 7f50777fe700 1 -- 172.21.15.187:0/3312238236 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421364428 0 0) 0x7f5078144b80 con 0x7f5078062b50 2020-11-20T12:11:26.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.024+0000 7f507cca8700 1 -- 172.21.15.187:0/3312238236 >> v1:172.21.15.187:6789/0 conn(0x7f5078062b50 legacy=0x7f5078065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.025+0000 7f507cca8700 1 -- 172.21.15.187:0/3312238236 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5078148f20 con 0x7f50780b57b0 2020-11-20T12:11:26.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.025+0000 7f507cca8700 1 --2- 172.21.15.187:0/3312238236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50780b57b0 0x7f5078068230 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f50780b85e0 tx=0x7f506c006df0).ready entity=mon.1 client_cookie=7dfc6ee197b77cce server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:26.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.025+0000 7f50777fe700 1 -- 172.21.15.187:0/3312238236 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c0097b0 con 0x7f50780b57b0 2020-11-20T12:11:26.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.025+0000 7f50777fe700 1 -- 172.21.15.187:0/3312238236 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c009910 con 0x7f50780b57b0 2020-11-20T12:11:26.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.025+0000 7f50777fe700 1 -- 172.21.15.187:0/3312238236 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c009c90 con 0x7f50780b57b0 2020-11-20T12:11:26.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.026+0000 7f507dcaa700 1 -- 172.21.15.187:0/3312238236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50780b57b0 msgr2=0x7f5078068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.026+0000 7f507dcaa700 1 --2- 172.21.15.187:0/3312238236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50780b57b0 0x7f5078068230 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f50780b85e0 tx=0x7f506c006df0).stop 2020-11-20T12:11:26.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.026+0000 7f507dcaa700 1 -- 172.21.15.187:0/3312238236 shutdown_connections 2020-11-20T12:11:26.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.026+0000 7f507dcaa700 1 --2- 172.21.15.187:0/3312238236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50780b57b0 0x7f5078068230 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.026+0000 7f507dcaa700 1 --2- 172.21.15.187:0/3312238236 >> v2:172.21.15.5:3300/0 conn(0x7f50781482d0 0x7f5078062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.026+0000 7f507dcaa700 1 -- 172.21.15.187:0/3312238236 >> 172.21.15.187:0/3312238236 conn(0x7f50780bab30 msgr2=0x7f50780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.026+0000 7f507dcaa700 1 -- 172.21.15.187:0/3312238236 shutdown_connections 2020-11-20T12:11:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.026+0000 7f507dcaa700 1 -- 172.21.15.187:0/3312238236 wait complete. 2020-11-20T12:11:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.027+0000 7f507dcaa700 1 Processor -- start 2020-11-20T12:11:26.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.028+0000 7f507dcaa700 1 -- start start 2020-11-20T12:11:26.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.028+0000 7f507dcaa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5078062b50 0x7f5078144d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507dcaa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50781482d0 0x7f50781454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507dcaa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f507815be20 0x7f5078141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507dcaa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507815c3a0 con 0x7f50781482d0 2020-11-20T12:11:26.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507dcaa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507815c640 con 0x7f507815be20 2020-11-20T12:11:26.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507dcaa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507815c8e0 con 0x7f5078062b50 2020-11-20T12:11:26.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507d4a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f507815be20 0x7f5078141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507d4a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f507815be20 0x7f5078141df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46074/0 (socket says 172.21.15.187:46074) 2020-11-20T12:11:26.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507d4a9700 1 -- 172.21.15.187:0/2502696079 learned_addr learned my addr 172.21.15.187:0/2502696079 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:26.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507cca8700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5078062b50 0x7f5078144d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507d4a9700 1 -- 172.21.15.187:0/2502696079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5078062b50 msgr2=0x7f5078144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507d4a9700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5078062b50 0x7f5078144d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507d4a9700 1 -- 172.21.15.187:0/2502696079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50781482d0 msgr2=0x7f50781454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507d4a9700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50781482d0 0x7f50781454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:26.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.029+0000 7f507d4a9700 1 -- 172.21.15.187:0/2502696079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50781459b0 con 0x7f507815be20 2020-11-20T12:11:26.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.030+0000 7f507d4a9700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f507815be20 0x7f5078141df0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f507000cab0 tx=0x7f5070007d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:26.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.030+0000 7f5074ff9700 1 -- 172.21.15.187:0/2502696079 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070005b80 con 0x7f507815be20 2020-11-20T12:11:26.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.030+0000 7f5074ff9700 1 -- 172.21.15.187:0/2502696079 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5070005ce0 con 0x7f507815be20 2020-11-20T12:11:26.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.030+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507809a670 con 0x7f507815be20 2020-11-20T12:11:26.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.030+0000 7f5074ff9700 1 -- 172.21.15.187:0/2502696079 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070007340 con 0x7f507815be20 2020-11-20T12:11:26.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.030+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5078142520 con 0x7f507815be20 2020-11-20T12:11:26.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.031+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5078066a90 con 0x7f507815be20 2020-11-20T12:11:26.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.032+0000 7f5074ff9700 1 -- 172.21.15.187:0/2502696079 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5070010030 con 0x7f507815be20 2020-11-20T12:11:26.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.032+0000 7f5074ff9700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5050056d50 0x7f5050059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.032+0000 7f5074ff9700 1 -- 172.21.15.187:0/2502696079 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f507006ffc0 con 0x7f507815be20 2020-11-20T12:11:26.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.033+0000 7f507cca8700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5050056d50 0x7f5050059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.033+0000 7f507cca8700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5050056d50 0x7f5050059210 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f5078063890 tx=0x7f506c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:26.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.036+0000 7f5074ff9700 1 -- 172.21.15.187:0/2502696079 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f50700453c0 con 0x7f507815be20 2020-11-20T12:11:26.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.286+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5078066e90 con 0x7f507815be20 2020-11-20T12:11:26.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.287+0000 7f5074ff9700 1 -- 172.21.15.187:0/2502696079 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f50700489e0 con 0x7f507815be20 2020-11-20T12:11:26.292 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:26.292 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:26.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.289+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5050056d50 msgr2=0x7f5050059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.289+0000 7f507dcaa700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5050056d50 0x7f5050059210 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f5078063890 tx=0x7f506c015000).stop 2020-11-20T12:11:26.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.289+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f507815be20 msgr2=0x7f5078141df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f507815be20 0x7f5078141df0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f507000cab0 tx=0x7f5070007d80).stop 2020-11-20T12:11:26.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 shutdown_connections 2020-11-20T12:11:26.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5050056d50 0x7f5050059210 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f507815be20 0x7f5078141df0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5078062b50 0x7f5078144d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 --2- 172.21.15.187:0/2502696079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50781482d0 0x7f50781454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:26.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 >> 172.21.15.187:0/2502696079 conn(0x7f50780bab30 msgr2=0x7f50780b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:26.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 shutdown_connections 2020-11-20T12:11:26.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.290+0000 7f507dcaa700 1 -- 172.21.15.187:0/2502696079 wait complete. 2020-11-20T12:11:26.297 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:26.316 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:11:26.626 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:11:26.627 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:11:26.628 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:26.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.885+0000 7f4a88d73700 1 Processor -- start 2020-11-20T12:11:26.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.885+0000 7f4a88d73700 1 -- start start 2020-11-20T12:11:26.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.887+0000 7f4a88d73700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4a84065f90 0x7f4a84065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.887+0000 7f4a88d73700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a840b13e0 con 0x7f4a8409ca90 2020-11-20T12:11:26.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.887+0000 7f4a88d73700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a84062c10 con 0x7f4a84065190 2020-11-20T12:11:26.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.887+0000 7f4a88d73700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a8414ab30 con 0x7f4a84065f90 2020-11-20T12:11:26.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.887+0000 7f4a837fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4a84065190 0x7f4a84143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42684/0 (socket says 172.21.15.187:42684) 2020-11-20T12:11:26.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.887+0000 7f4a837fe700 1 -- 172.21.15.187:0/3571939767 learned_addr learned my addr 172.21.15.187:0/3571939767 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:26.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a7affd700 1 --2- 172.21.15.187:0/3571939767 >> v2:172.21.15.5:3300/0 conn(0x7f4a84065f90 0x7f4a84065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607688583 0 0) 0x7f4a84062c10 con 0x7f4a84065190 2020-11-20T12:11:26.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a64003240 con 0x7f4a84065190 2020-11-20T12:11:26.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 110360194 0 0) 0x7f4a64003240 con 0x7f4a84065190 2020-11-20T12:11:26.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 >> v2:172.21.15.5:3300/0 conn(0x7f4a84065f90 msgr2=0x7f4a84065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:26.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 --2- 172.21.15.187:0/3571939767 >> v2:172.21.15.5:3300/0 conn(0x7f4a84065f90 0x7f4a84065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 >> v1:172.21.15.5:6789/0 conn(0x7f4a8409ca90 legacy=0x7f4a8409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a840b1640 con 0x7f4a84065190 2020-11-20T12:11:26.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4a740014e0 con 0x7f4a84065190 2020-11-20T12:11:26.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.888+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 >> v1:172.21.15.187:6789/0 conn(0x7f4a84065190 legacy=0x7f4a84143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.889+0000 7f4a82ffd700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a64003780 0x7f4a64003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.889+0000 7f4a82ffd700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a640044e0 0x7f4a64004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.889+0000 7f4a82ffd700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64004e50 0x7f4a6400fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.889+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8414ab30 con 0x7f4a640044e0 2020-11-20T12:11:26.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.889+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64005300 con 0x7f4a64004e50 2020-11-20T12:11:26.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.889+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64010030 con 0x7f4a64003780 2020-11-20T12:11:26.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.889+0000 7f4a83fff700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64004e50 0x7f4a6400fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.889+0000 7f4a837fe700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a64003780 0x7f4a64003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a7affd700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a640044e0 0x7f4a64004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a83fff700 1 -- 172.21.15.187:0/3571939767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a64003780 msgr2=0x7f4a64003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:26.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a83fff700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a64003780 0x7f4a64003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a83fff700 1 -- 172.21.15.187:0/3571939767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a640044e0 msgr2=0x7f4a64004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:26.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a83fff700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a640044e0 0x7f4a64004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a83fff700 1 -- 172.21.15.187:0/3571939767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64003240 con 0x7f4a64004e50 2020-11-20T12:11:26.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a83fff700 1 -- 172.21.15.187:0/3571939767 reap_dead start 2020-11-20T12:11:26.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a83fff700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64004e50 0x7f4a6400fb40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f4a7c001e70 tx=0x7f4a7c007f10).ready entity=mon.1 client_cookie=b11e9306d815d7e5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:26.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c001540 con 0x7f4a64004e50 2020-11-20T12:11:26.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c003030 con 0x7f4a64004e50 2020-11-20T12:11:26.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.890+0000 7f4a82ffd700 1 -- 172.21.15.187:0/3571939767 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c00cdd0 con 0x7f4a64004e50 2020-11-20T12:11:26.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.891+0000 7f4a88d73700 1 -- 172.21.15.187:0/3571939767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64004e50 msgr2=0x7f4a6400fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.891+0000 7f4a88d73700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64004e50 0x7f4a6400fb40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f4a7c001e70 tx=0x7f4a7c007f10).stop 2020-11-20T12:11:26.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.891+0000 7f4a88d73700 1 -- 172.21.15.187:0/3571939767 shutdown_connections 2020-11-20T12:11:26.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.891+0000 7f4a88d73700 1 --2- 172.21.15.187:0/3571939767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a64004e50 0x7f4a6400fb40 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.891+0000 7f4a88d73700 1 -- 172.21.15.187:0/3571939767 >> 172.21.15.187:0/3571939767 conn(0x7f4a840bab30 msgr2=0x7f4a840baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:26.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.891+0000 7f4a88d73700 1 -- 172.21.15.187:0/3571939767 shutdown_connections 2020-11-20T12:11:26.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.891+0000 7f4a88d73700 1 -- 172.21.15.187:0/3571939767 wait complete. 2020-11-20T12:11:26.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.893+0000 7f4a88d73700 1 Processor -- start 2020-11-20T12:11:26.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.893+0000 7f4a88d73700 1 -- start start 2020-11-20T12:11:26.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a88d73700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a64004e50 0x7f4a84157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a88d73700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a84147eb0 0x7f4a84067b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a88d73700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a8409ec60 0x7f4a8413f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a88d73700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8409e280 con 0x7f4a64004e50 2020-11-20T12:11:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a88d73700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8409e520 con 0x7f4a84147eb0 2020-11-20T12:11:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a88d73700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8409e7c0 con 0x7f4a8409ec60 2020-11-20T12:11:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a7affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a84147eb0 0x7f4a84067b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a7affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a84147eb0 0x7f4a84067b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46092/0 (socket says 172.21.15.187:46092) 2020-11-20T12:11:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.894+0000 7f4a7affd700 1 -- 172.21.15.187:0/1721686892 learned_addr learned my addr 172.21.15.187:0/1721686892 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:26.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a837fe700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a64004e50 0x7f4a84157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7affd700 1 -- 172.21.15.187:0/1721686892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a8409ec60 msgr2=0x7f4a8413f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7affd700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a8409ec60 0x7f4a8413f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:26.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7affd700 1 -- 172.21.15.187:0/1721686892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a64004e50 msgr2=0x7f4a84157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:26.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7affd700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a64004e50 0x7f4a84157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:26.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7affd700 1 -- 172.21.15.187:0/1721686892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84158320 con 0x7f4a84147eb0 2020-11-20T12:11:26.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7affd700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a84147eb0 0x7f4a84067b50 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f4a6c006ec0 tx=0x7f4a6c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:26.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7bfff700 1 -- 172.21.15.187:0/1721686892 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c017070 con 0x7f4a84147eb0 2020-11-20T12:11:26.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7bfff700 1 -- 172.21.15.187:0/1721686892 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c011660 con 0x7f4a84147eb0 2020-11-20T12:11:26.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a84068040 con 0x7f4a84147eb0 2020-11-20T12:11:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.895+0000 7f4a7bfff700 1 -- 172.21.15.187:0/1721686892 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c0119e0 con 0x7f4a84147eb0 2020-11-20T12:11:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.896+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8409f300 con 0x7f4a84147eb0 2020-11-20T12:11:26.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.897+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a84061a20 con 0x7f4a84147eb0 2020-11-20T12:11:26.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.897+0000 7f4a7bfff700 1 -- 172.21.15.187:0/1721686892 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4a6c01c410 con 0x7f4a84147eb0 2020-11-20T12:11:26.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.898+0000 7f4a7bfff700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a60056d20 0x7f4a600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:26.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.898+0000 7f4a7bfff700 1 -- 172.21.15.187:0/1721686892 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4a6c06feb0 con 0x7f4a84147eb0 2020-11-20T12:11:26.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.899+0000 7f4a837fe700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a60056d20 0x7f4a600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:26.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.899+0000 7f4a837fe700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a60056d20 0x7f4a600591e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f4a740013b0 tx=0x7f4a840b0fb0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:26.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:26.902+0000 7f4a7bfff700 1 -- 172.21.15.187:0/1721686892 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4a6c045250 con 0x7f4a84147eb0 2020-11-20T12:11:27.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.147+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a840afc60 con 0x7f4a84147eb0 2020-11-20T12:11:27.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.148+0000 7f4a7bfff700 1 -- 172.21.15.187:0/1721686892 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f4a6c048870 con 0x7f4a84147eb0 2020-11-20T12:11:27.153 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:27.153 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:27.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a60056d20 msgr2=0x7f4a600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a60056d20 0x7f4a600591e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f4a740013b0 tx=0x7f4a840b0fb0).stop 2020-11-20T12:11:27.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a84147eb0 msgr2=0x7f4a84067b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a84147eb0 0x7f4a84067b50 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f4a6c006ec0 tx=0x7f4a6c008040).stop 2020-11-20T12:11:27.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 shutdown_connections 2020-11-20T12:11:27.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a60056d20 0x7f4a600591e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a64004e50 0x7f4a84157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a8409ec60 0x7f4a8413f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:27.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 --2- 172.21.15.187:0/1721686892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a84147eb0 0x7f4a84067b50 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 >> 172.21.15.187:0/1721686892 conn(0x7f4a840bab30 msgr2=0x7f4a84141780 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:27.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 shutdown_connections 2020-11-20T12:11:27.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.151+0000 7f4a88d73700 1 -- 172.21.15.187:0/1721686892 wait complete. 2020-11-20T12:11:27.158 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:27.181 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2020-11-20T12:11:27.527 INFO:teuthology.orchestra.run.smithi005.stdout:{"message":"","return_code":0} 2020-11-20T12:11:27.528 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:11:27.528 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data getxattr 10000000001.00000000 parent 2020-11-20T12:11:27.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.615+0000 7f04afe40d00 1 Processor -- start 2020-11-20T12:11:27.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.615+0000 7f04afe40d00 1 -- start start 2020-11-20T12:11:27.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.616+0000 7f04afe40d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d5c3896f00 0x55d5c3893ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.616+0000 7f04afe40d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d5c38972e0 0x55d5c37ed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.616+0000 7f04afe40d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c3897740 0x55d5c3805850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.616+0000 7f04afe40d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c3742810 con 0x55d5c3896f00 2020-11-20T12:11:27.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.616+0000 7f04afe40d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c38949e0 con 0x55d5c3897740 2020-11-20T12:11:27.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.616+0000 7f04afe40d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d5c3895630 con 0x55d5c38972e0 2020-11-20T12:11:27.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.616+0000 7f04a0ee6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c3897740 0x55d5c3805850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a0ee6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c3897740 0x55d5c3805850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46106/0 (socket says 172.21.15.187:46106) 2020-11-20T12:11:27.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a0ee6700 1 -- 172.21.15.187:0/2204837783 learned_addr learned my addr 172.21.15.187:0/2204837783 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:27.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a06e5700 1 --2- 172.21.15.187:0/2204837783 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d5c3896f00 0x55d5c3893ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a0ee6700 1 -- 172.21.15.187:0/2204837783 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d5c38972e0 msgr2=0x55d5c37ed6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:27.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a0ee6700 1 --2- 172.21.15.187:0/2204837783 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d5c38972e0 0x55d5c37ed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:27.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a0ee6700 1 -- 172.21.15.187:0/2204837783 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d5c3896f00 msgr2=0x55d5c3893ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:27.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a0ee6700 1 --2- 172.21.15.187:0/2204837783 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d5c3896f00 0x55d5c3893ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a0ee6700 1 -- 172.21.15.187:0/2204837783 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5c376c4e0 con 0x55d5c3897740 2020-11-20T12:11:27.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.617+0000 7f04a0ee6700 1 --2- 172.21.15.187:0/2204837783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c3897740 0x55d5c3805850 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f0498001b60 tx=0x7f0498006b10).ready entity=mon.1 client_cookie=c41ceadefc8f5b1d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:27.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f049f6e3700 1 -- 172.21.15.187:0/2204837783 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049800b5e0 con 0x55d5c3897740 2020-11-20T12:11:27.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f049f6e3700 1 -- 172.21.15.187:0/2204837783 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049800b740 con 0x55d5c3897740 2020-11-20T12:11:27.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f049f6e3700 1 -- 172.21.15.187:0/2204837783 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049800bac0 con 0x55d5c3897740 2020-11-20T12:11:27.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 -- 172.21.15.187:0/2204837783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c3897740 msgr2=0x55d5c3805850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 --2- 172.21.15.187:0/2204837783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c3897740 0x55d5c3805850 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f0498001b60 tx=0x7f0498006b10).stop 2020-11-20T12:11:27.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 -- 172.21.15.187:0/2204837783 shutdown_connections 2020-11-20T12:11:27.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 --2- 172.21.15.187:0/2204837783 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d5c3896f00 0x55d5c3893ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 --2- 172.21.15.187:0/2204837783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c3897740 0x55d5c3805850 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 --2- 172.21.15.187:0/2204837783 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d5c38972e0 0x55d5c37ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:27.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 -- 172.21.15.187:0/2204837783 >> 172.21.15.187:0/2204837783 conn(0x55d5c37c5f00 msgr2=0x55d5c3808890 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:27.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 -- 172.21.15.187:0/2204837783 shutdown_connections 2020-11-20T12:11:27.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.618+0000 7f04afe40d00 1 -- 172.21.15.187:0/2204837783 wait complete. 2020-11-20T12:11:27.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.619+0000 7f04afe40d00 1 Processor -- start 2020-11-20T12:11:27.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.619+0000 7f04afe40d00 1 -- start start 2020-11-20T12:11:27.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.620+0000 7f04afe40d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d5c3896f00 0x55d5c37c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.620+0000 7f04afe40d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c38972e0 0x55d5c388f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.620+0000 7f04afe40d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d5c3897740 0x55d5c38a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.620+0000 7f04afe40d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c3804930 con 0x55d5c3896f00 2020-11-20T12:11:27.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.620+0000 7f04afe40d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c3804bd0 con 0x55d5c38972e0 2020-11-20T12:11:27.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.620+0000 7f04afe40d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d5c37eb900 con 0x55d5c3897740 2020-11-20T12:11:27.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c38972e0 0x55d5c388f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c38972e0 0x55d5c388f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46112/0 (socket says 172.21.15.187:46112) 2020-11-20T12:11:27.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 -- 172.21.15.187:0/262552424 learned_addr learned my addr 172.21.15.187:0/262552424 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:27.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 -- 172.21.15.187:0/262552424 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d5c3897740 msgr2=0x55d5c38a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55d5c3897740 0x55d5c38a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:27.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 -- 172.21.15.187:0/262552424 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d5c3896f00 msgr2=0x55d5c37c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55d5c3896f00 0x55d5c37c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:27.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 -- 172.21.15.187:0/262552424 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5c3767ec0 con 0x55d5c38972e0 2020-11-20T12:11:27.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.621+0000 7f0497ee4700 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c38972e0 0x55d5c388f510 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f048800c010 tx=0x7f0488016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:27.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.622+0000 7f049dee0700 1 -- 172.21.15.187:0/262552424 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488013cc0 con 0x55d5c38972e0 2020-11-20T12:11:27.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.622+0000 7f049dee0700 1 -- 172.21.15.187:0/262552424 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0488013e20 con 0x55d5c38972e0 2020-11-20T12:11:27.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.622+0000 7f049dee0700 1 -- 172.21.15.187:0/262552424 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048801c650 con 0x55d5c38972e0 2020-11-20T12:11:27.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.622+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d5c3746250 con 0x55d5c38972e0 2020-11-20T12:11:27.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.622+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5c38a5ab0 con 0x55d5c38972e0 2020-11-20T12:11:27.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.624+0000 7f049dee0700 1 -- 172.21.15.187:0/262552424 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f048801ca80 con 0x55d5c38972e0 2020-11-20T12:11:27.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.624+0000 7f049dee0700 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04800559a0 0x7f0480057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.624+0000 7f049dee0700 1 -- 172.21.15.187:0/262552424 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0488070890 con 0x55d5c38972e0 2020-11-20T12:11:27.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.624+0000 7f04a06e5700 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04800559a0 0x7f0480057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.625+0000 7f04afe40d00 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55d5c3805830 0x55d5c38983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.625+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:1 14.7 14:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected e72) v8 -- 0x55d5c388ddf0 con 0x55d5c3805830 2020-11-20T12:11:27.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.625+0000 7f04a0ee6700 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55d5c3805830 0x55d5c38983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.625+0000 7f04a06e5700 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04800559a0 0x7f0480057e60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f0490005fd0 tx=0x7f049000c040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:27.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04a0ee6700 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55d5c3805830 0x55d5c38983b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:27.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04a0ee6700 1 -- 172.21.15.187:0/262552424 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7f0498000f00 con 0x55d5c3805830 2020-11-20T12:11:27.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55d5c3805830 msgr2=0x55d5c38983b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04afe40d00 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55d5c3805830 0x55d5c38983b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04800559a0 msgr2=0x7f0480057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04afe40d00 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04800559a0 0x7f0480057e60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f0490005fd0 tx=0x7f049000c040).stop 2020-11-20T12:11:27.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c38972e0 msgr2=0x55d5c388f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04afe40d00 1 --2- 172.21.15.187:0/262552424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d5c38972e0 0x55d5c388f510 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f048800c010 tx=0x7f0488016000).stop 2020-11-20T12:11:27.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04a0ee6700 1 -- 172.21.15.187:0/262552424 reap_dead start 2020-11-20T12:11:27.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 shutdown_connections 2020-11-20T12:11:27.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.626+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 >> 172.21.15.187:0/262552424 conn(0x55d5c37c5f00 msgr2=0x55d5c3893ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:27.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.627+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 shutdown_connections 2020-11-20T12:11:27.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.627+0000 7f04afe40d00 1 -- 172.21.15.187:0/262552424 wait complete. 2020-11-20T12:11:27.641 INFO:teuthology.orchestra.run.smithi187.stdout:] 10000000001 stray0 2020-11-20T12:11:27.641 INFO:teuthology.orchestra.run.smithi187:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2020-11-20T12:11:27.725 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:11:27.726 INFO:teuthology.orchestra.run.smithi187.stdout: "ino": 1099511627777, 2020-11-20T12:11:27.726 INFO:teuthology.orchestra.run.smithi187.stdout: "ancestors": [ 2020-11-20T12:11:27.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:11:27.727 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1536, 2020-11-20T12:11:27.727 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "10000000001", 2020-11-20T12:11:27.727 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 13 2020-11-20T12:11:27.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:11:27.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:11:27.728 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 256, 2020-11-20T12:11:27.728 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "stray0", 2020-11-20T12:11:27.728 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 5 2020-11-20T12:11:27.729 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:11:27.729 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2020-11-20T12:11:27.729 INFO:teuthology.orchestra.run.smithi187.stdout: "pool": 14, 2020-11-20T12:11:27.729 INFO:teuthology.orchestra.run.smithi187.stdout: "old_pools": [] 2020-11-20T12:11:27.730 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:11:27.730 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:27.735 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:27.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.976+0000 7f37ba002700 1 Processor -- start 2020-11-20T12:11:27.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.976+0000 7f37ba002700 1 -- start start 2020-11-20T12:11:27.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.977+0000 7f37ba002700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4145cc0 0x7f37b4066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.977+0000 7f37ba002700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b414c170 con 0x7f37b41487e0 2020-11-20T12:11:27.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.977+0000 7f37ba002700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b414c410 con 0x7f37b40683f0 2020-11-20T12:11:27.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.977+0000 7f37ba002700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b414c6b0 con 0x7f37b4145cc0 2020-11-20T12:11:27.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.978+0000 7f37abfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f37b40683f0 0x7f37b4065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42730/0 (socket says 172.21.15.187:42730) 2020-11-20T12:11:27.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.978+0000 7f37abfff700 1 -- 172.21.15.187:0/2113890467 learned_addr learned my addr 172.21.15.187:0/2113890467 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:27.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.978+0000 7f37b9000700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4145cc0 0x7f37b4066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392580309 0 0) 0x7f37b414c410 con 0x7f37b40683f0 2020-11-20T12:11:27.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37a0003240 con 0x7f37b40683f0 2020-11-20T12:11:27.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4014121878 0 0) 0x7f37a0003240 con 0x7f37b40683f0 2020-11-20T12:11:27.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4145cc0 msgr2=0x7f37b4066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:27.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4145cc0 0x7f37b4066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 >> v1:172.21.15.5:6789/0 conn(0x7f37b41487e0 legacy=0x7f37b4148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b40658a0 con 0x7f37b40683f0 2020-11-20T12:11:27.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f37a40014e0 con 0x7f37b40683f0 2020-11-20T12:11:27.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 >> v1:172.21.15.187:6789/0 conn(0x7f37b40683f0 legacy=0x7f37b4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37a0003730 0x7f37a0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4145cc0 0x7f37a00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.979+0000 7f37ab7fe700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37a00049d0 0x7f37a0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b414c6b0 con 0x7f37a0003730 2020-11-20T12:11:27.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a00041d0 con 0x7f37a00049d0 2020-11-20T12:11:27.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a000b730 con 0x7f37b4145cc0 2020-11-20T12:11:27.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37b9801700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37a00049d0 0x7f37a0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37abfff700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4145cc0 0x7f37a00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37b9000700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37a0003730 0x7f37a0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37b9801700 1 -- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4145cc0 msgr2=0x7f37a00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:27.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37b9801700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4145cc0 0x7f37a00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37b9801700 1 -- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37a0003730 msgr2=0x7f37a0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:27.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.980+0000 7f37b9801700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37a0003730 0x7f37a0003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.981+0000 7f37b9801700 1 -- 172.21.15.187:0/2113890467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a0003240 con 0x7f37a00049d0 2020-11-20T12:11:27.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.981+0000 7f37b9801700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37a00049d0 0x7f37a0004e50 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f37ac001e70 tx=0x7f37ac007f10).ready entity=mon.1 client_cookie=2e921ea3bbeb38d4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:27.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.981+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac001540 con 0x7f37a00049d0 2020-11-20T12:11:27.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.981+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac003030 con 0x7f37a00049d0 2020-11-20T12:11:27.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.981+0000 7f37ab7fe700 1 -- 172.21.15.187:0/2113890467 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac00cdf0 con 0x7f37a00049d0 2020-11-20T12:11:27.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.982+0000 7f37ba002700 1 -- 172.21.15.187:0/2113890467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37a00049d0 msgr2=0x7f37a0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.982+0000 7f37ba002700 1 --2- 172.21.15.187:0/2113890467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37a00049d0 0x7f37a0004e50 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f37ac001e70 tx=0x7f37ac007f10).stop 2020-11-20T12:11:27.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.982+0000 7f37b9801700 1 -- 172.21.15.187:0/2113890467 reap_dead start 2020-11-20T12:11:27.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.982+0000 7f37ba002700 1 -- 172.21.15.187:0/2113890467 shutdown_connections 2020-11-20T12:11:27.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.982+0000 7f37ba002700 1 -- 172.21.15.187:0/2113890467 >> 172.21.15.187:0/2113890467 conn(0x7f37b40bab30 msgr2=0x7f37b40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:27.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.982+0000 7f37ba002700 1 -- 172.21.15.187:0/2113890467 shutdown_connections 2020-11-20T12:11:27.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.982+0000 7f37ba002700 1 -- 172.21.15.187:0/2113890467 wait complete. 2020-11-20T12:11:27.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.984+0000 7f37ba002700 1 Processor -- start 2020-11-20T12:11:27.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.984+0000 7f37ba002700 1 -- start start 2020-11-20T12:11:27.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.985+0000 7f37ba002700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4147230 0x7f37b409fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.985+0000 7f37ba002700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37b4147610 0x7f37b4142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.985+0000 7f37ba002700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b409e560 0x7f37b4142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:27.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.985+0000 7f37ba002700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a0190 con 0x7f37b4147610 2020-11-20T12:11:27.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.985+0000 7f37ba002700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a0420 con 0x7f37b409e560 2020-11-20T12:11:27.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.985+0000 7f37ba002700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a06c0 con 0x7f37b4147230 2020-11-20T12:11:27.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.985+0000 7f37b9801700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b409e560 0x7f37b4142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.985+0000 7f37b9801700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b409e560 0x7f37b4142500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46136/0 (socket says 172.21.15.187:46136) 2020-11-20T12:11:27.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37b9801700 1 -- 172.21.15.187:0/1497243968 learned_addr learned my addr 172.21.15.187:0/1497243968 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37b9000700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4147230 0x7f37b409fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37abfff700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37b4147610 0x7f37b4142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37b9801700 1 -- 172.21.15.187:0/1497243968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4147230 msgr2=0x7f37b409fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37b9801700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4147230 0x7f37b409fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37b9801700 1 -- 172.21.15.187:0/1497243968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37b4147610 msgr2=0x7f37b4142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:27.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37b9801700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37b4147610 0x7f37b4142010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:27.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37b9801700 1 -- 172.21.15.187:0/1497243968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b40a0960 con 0x7f37b409e560 2020-11-20T12:11:27.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.986+0000 7f37b9801700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b409e560 0x7f37b4142500 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f37ac001e40 tx=0x7f37ac015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:27.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.987+0000 7f37a8ff9700 1 -- 172.21.15.187:0/1497243968 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac001540 con 0x7f37b409e560 2020-11-20T12:11:28.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.987+0000 7f37a8ff9700 1 -- 172.21.15.187:0/1497243968 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac0125d0 con 0x7f37b409e560 2020-11-20T12:11:28.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.987+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b41429f0 con 0x7f37b409e560 2020-11-20T12:11:28.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.987+0000 7f37a8ff9700 1 -- 172.21.15.187:0/1497243968 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac012970 con 0x7f37b409e560 2020-11-20T12:11:28.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.987+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b4148e50 con 0x7f37b409e560 2020-11-20T12:11:28.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.988+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b40b96d0 con 0x7f37b409e560 2020-11-20T12:11:28.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.988+0000 7f37a8ff9700 1 -- 172.21.15.187:0/1497243968 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f37ac026030 con 0x7f37b409e560 2020-11-20T12:11:28.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.989+0000 7f37a8ff9700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f378c056d20 0x7f378c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.989+0000 7f37a8ff9700 1 -- 172.21.15.187:0/1497243968 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f37ac0714b0 con 0x7f37b409e560 2020-11-20T12:11:28.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.989+0000 7f37b9000700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f378c056d20 0x7f378c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:28.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.990+0000 7f37b9000700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f378c056d20 0x7f378c0591e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f37b0006ec0 tx=0x7f37b0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:28.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:27.993+0000 7f37a8ff9700 1 -- 172.21.15.187:0/1497243968 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f37ac0468b0 con 0x7f37b409e560 2020-11-20T12:11:28.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.240+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37b4061a20 con 0x7f37b409e560 2020-11-20T12:11:28.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.241+0000 7f37a8ff9700 1 -- 172.21.15.187:0/1497243968 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f37ac024870 con 0x7f37b409e560 2020-11-20T12:11:28.246 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:28.246 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:28.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f378c056d20 msgr2=0x7f378c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:28.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f378c056d20 0x7f378c0591e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f37b0006ec0 tx=0x7f37b0008040).stop 2020-11-20T12:11:28.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b409e560 msgr2=0x7f37b4142500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:28.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b409e560 0x7f37b4142500 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f37ac001e40 tx=0x7f37ac015000).stop 2020-11-20T12:11:28.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 shutdown_connections 2020-11-20T12:11:28.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f378c056d20 0x7f378c0591e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:28.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b409e560 0x7f37b4142500 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:28.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f37b4147230 0x7f37b409fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:28.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 --2- 172.21.15.187:0/1497243968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37b4147610 0x7f37b4142010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:28.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 >> 172.21.15.187:0/1497243968 conn(0x7f37b40bab30 msgr2=0x7f37b4142d40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:28.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 shutdown_connections 2020-11-20T12:11:28.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.244+0000 7f37ba002700 1 -- 172.21.15.187:0/1497243968 wait complete. 2020-11-20T12:11:28.251 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:28.272 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_reintegrated 2020-11-20T12:11:28.567 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_reintegrated":0}} 2020-11-20T12:11:28.567 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2020-11-20T12:11:28.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:11:28.568 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mv dir_2/file_b dir_2/file_c) 2020-11-20T12:11:28.701 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:28.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.941+0000 7fd4a98cc700 1 Processor -- start 2020-11-20T12:11:28.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.941+0000 7fd4a98cc700 1 -- start start 2020-11-20T12:11:28.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.942+0000 7fd4a98cc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4a41484d0 0x7fd4a4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.942+0000 7fd4a98cc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a4066f70 0x7fd4a40b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.942+0000 7fd4a98cc700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a40b1300 con 0x7fd4a40622c0 2020-11-20T12:11:28.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.942+0000 7fd4a98cc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a4067730 con 0x7fd4a41484d0 2020-11-20T12:11:28.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.942+0000 7fd4a98cc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40b15a0 con 0x7fd4a4066f70 2020-11-20T12:11:28.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.943+0000 7fd49bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd4a40622c0 0x7fd4a40b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42748/0 (socket says 172.21.15.187:42748) 2020-11-20T12:11:28.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.943+0000 7fd49bfff700 1 -- 172.21.15.187:0/129930024 learned_addr learned my addr 172.21.15.187:0/129930024 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:28.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.943+0000 7fd4a88ca700 1 --2- 172.21.15.187:0/129930024 >> v2:172.21.15.5:3300/0 conn(0x7fd4a41484d0 0x7fd4a4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:28.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.943+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431866408 0 0) 0x7fd4a40b1300 con 0x7fd4a40622c0 2020-11-20T12:11:28.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.943+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd490003240 con 0x7fd4a40622c0 2020-11-20T12:11:28.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.943+0000 7fd4a90cb700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a4066f70 0x7fd4a40b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:28.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4137648103 0 0) 0x7fd490003240 con 0x7fd4a40622c0 2020-11-20T12:11:28.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a4066f70 msgr2=0x7fd4a40b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:28.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a4066f70 0x7fd4a40b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:28.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 >> v2:172.21.15.5:3300/0 conn(0x7fd4a41484d0 msgr2=0x7fd4a4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:28.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 --2- 172.21.15.187:0/129930024 >> v2:172.21.15.5:3300/0 conn(0x7fd4a41484d0 0x7fd4a4061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:28.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a40b1840 con 0x7fd4a40622c0 2020-11-20T12:11:28.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd4940014e0 con 0x7fd4a40622c0 2020-11-20T12:11:28.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 >> v1:172.21.15.187:6789/0 conn(0x7fd4a40622c0 legacy=0x7fd4a40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:28.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490003730 0x7fd490003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a4066f70 0x7fd4900044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4900049d0 0x7fd490004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a4067730 con 0x7fd4900049d0 2020-11-20T12:11:28.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40b15a0 con 0x7fd490003730 2020-11-20T12:11:28.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900041d0 con 0x7fd4a4066f70 2020-11-20T12:11:28.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.944+0000 7fd4a88ca700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490003730 0x7fd490003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:28.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd4a90cb700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4900049d0 0x7fd490004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:28.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd49bfff700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a4066f70 0x7fd4900044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd4a88ca700 1 -- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a4066f70 msgr2=0x7fd4900044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd4a88ca700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a4066f70 0x7fd4900044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd4a88ca700 1 -- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4900049d0 msgr2=0x7fd490004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd4a88ca700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4900049d0 0x7fd490004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd4a88ca700 1 -- 172.21.15.187:0/129930024 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd490003240 con 0x7fd490003730 2020-11-20T12:11:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd4a88ca700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490003730 0x7fd490003b50 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7fd4a000bc00 tx=0x7fd4a0006df0).ready entity=mon.1 client_cookie=32d916684938ab2a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.945+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a000b310 con 0x7fd490003730 2020-11-20T12:11:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a000eb40 con 0x7fd490003730 2020-11-20T12:11:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd49b7fe700 1 -- 172.21.15.187:0/129930024 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a000eec0 con 0x7fd490003730 2020-11-20T12:11:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd4a98cc700 1 -- 172.21.15.187:0/129930024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490003730 msgr2=0x7fd490003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:28.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd4a98cc700 1 --2- 172.21.15.187:0/129930024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd490003730 0x7fd490003b50 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7fd4a000bc00 tx=0x7fd4a0006df0).stop 2020-11-20T12:11:28.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd4a90cb700 1 -- 172.21.15.187:0/129930024 reap_dead start 2020-11-20T12:11:28.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd4a98cc700 1 -- 172.21.15.187:0/129930024 shutdown_connections 2020-11-20T12:11:28.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd4a98cc700 1 -- 172.21.15.187:0/129930024 >> 172.21.15.187:0/129930024 conn(0x7fd4a40bab30 msgr2=0x7fd4a40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:28.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd4a98cc700 1 -- 172.21.15.187:0/129930024 shutdown_connections 2020-11-20T12:11:28.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.946+0000 7fd4a98cc700 1 -- 172.21.15.187:0/129930024 wait complete. 2020-11-20T12:11:28.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.948+0000 7fd4a98cc700 1 Processor -- start 2020-11-20T12:11:28.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.949+0000 7fd4a98cc700 1 -- start start 2020-11-20T12:11:28.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a98cc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4a409fdf0 0x7fd4a4144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a98cc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40a01d0 0x7fd4a4145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a98cc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b1a20 0x7fd4a40b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a98cc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a4145770 con 0x7fd4a40b1a20 2020-11-20T12:11:28.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a98cc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40b2950 con 0x7fd4a409fdf0 2020-11-20T12:11:28.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a98cc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a415bdf0 con 0x7fd4a40a01d0 2020-11-20T12:11:28.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a88ca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4a409fdf0 0x7fd4a4144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:28.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a88ca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4a409fdf0 0x7fd4a4144d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46152/0 (socket says 172.21.15.187:46152) 2020-11-20T12:11:28.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a88ca700 1 -- 172.21.15.187:0/4062444378 learned_addr learned my addr 172.21.15.187:0/4062444378 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:28.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a88ca700 1 -- 172.21.15.187:0/4062444378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40a01d0 msgr2=0x7fd4a4145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:28.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a88ca700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40a01d0 0x7fd4a4145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:28.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a88ca700 1 -- 172.21.15.187:0/4062444378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b1a20 msgr2=0x7fd4a40b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:28.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a88ca700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b1a20 0x7fd4a40b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:28.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.950+0000 7fd4a88ca700 1 -- 172.21.15.187:0/4062444378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a415c090 con 0x7fd4a409fdf0 2020-11-20T12:11:28.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.951+0000 7fd4a88ca700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4a409fdf0 0x7fd4a4144d90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7fd4a0016000 tx=0x7fd4a0019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:28.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.951+0000 7fd498ff9700 1 -- 172.21.15.187:0/4062444378 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a001b660 con 0x7fd4a409fdf0 2020-11-20T12:11:28.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.951+0000 7fd498ff9700 1 -- 172.21.15.187:0/4062444378 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a0022070 con 0x7fd4a409fdf0 2020-11-20T12:11:28.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.951+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a415c330 con 0x7fd4a409fdf0 2020-11-20T12:11:28.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.951+0000 7fd498ff9700 1 -- 172.21.15.187:0/4062444378 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a001b7e0 con 0x7fd4a409fdf0 2020-11-20T12:11:28.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.951+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a415bab0 con 0x7fd4a409fdf0 2020-11-20T12:11:28.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.952+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a40ba890 con 0x7fd4a409fdf0 2020-11-20T12:11:28.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.953+0000 7fd498ff9700 1 -- 172.21.15.187:0/4062444378 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd4a001bb50 con 0x7fd4a409fdf0 2020-11-20T12:11:28.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.954+0000 7fd498ff9700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd47c056d20 0x7fd47c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:28.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.954+0000 7fd498ff9700 1 -- 172.21.15.187:0/4062444378 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd4a0072270 con 0x7fd4a409fdf0 2020-11-20T12:11:28.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.954+0000 7fd49bfff700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd47c056d20 0x7fd47c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:28.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.955+0000 7fd49bfff700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd47c056d20 0x7fd47c0591e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd494000f20 tx=0x7fd494009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:28.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:28.958+0000 7fd498ff9700 1 -- 172.21.15.187:0/4062444378 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd4a0047670 con 0x7fd4a409fdf0 2020-11-20T12:11:29.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.202+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4a415c4c0 con 0x7fd4a409fdf0 2020-11-20T12:11:29.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.203+0000 7fd498ff9700 1 -- 172.21.15.187:0/4062444378 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fd4a004ac90 con 0x7fd4a409fdf0 2020-11-20T12:11:29.208 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:29.208 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:29.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd47c056d20 msgr2=0x7fd47c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:29.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd47c056d20 0x7fd47c0591e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd494000f20 tx=0x7fd494009040).stop 2020-11-20T12:11:29.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4a409fdf0 msgr2=0x7fd4a4144d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:29.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4a409fdf0 0x7fd4a4144d90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7fd4a0016000 tx=0x7fd4a0019000).stop 2020-11-20T12:11:29.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 shutdown_connections 2020-11-20T12:11:29.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd47c056d20 0x7fd47c0591e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:29.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a40b1a20 0x7fd4a40b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:29.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4a409fdf0 0x7fd4a4144d90 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:29.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 --2- 172.21.15.187:0/4062444378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a40a01d0 0x7fd4a4145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:29.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 >> 172.21.15.187:0/4062444378 conn(0x7fd4a40bab30 msgr2=0x7fd4a40b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:29.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.206+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 shutdown_connections 2020-11-20T12:11:29.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.207+0000 7fd4a98cc700 1 -- 172.21.15.187:0/4062444378 wait complete. 2020-11-20T12:11:29.213 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:29.235 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache num_strays 2020-11-20T12:11:29.528 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"num_strays":0}} 2020-11-20T12:11:29.528 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2020-11-20T12:11:29.529 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:29.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.773+0000 7f01be049700 1 Processor -- start 2020-11-20T12:11:29.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.773+0000 7f01be049700 1 -- start start 2020-11-20T12:11:29.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.774+0000 7f01be049700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f01b8065570 0x7f01b809d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:29.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.775+0000 7f01be049700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b80b13e0 con 0x7f01b8065f90 2020-11-20T12:11:29.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.775+0000 7f01be049700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b809d690 con 0x7f01b8065190 2020-11-20T12:11:29.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.775+0000 7f01be049700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b814ab30 con 0x7f01b8065570 2020-11-20T12:11:29.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.775+0000 7f01bd047700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f01b8065190 0x7f01b8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42770/0 (socket says 172.21.15.187:42770) 2020-11-20T12:11:29.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.775+0000 7f01bd047700 1 -- 172.21.15.187:0/2834834911 learned_addr learned my addr 172.21.15.187:0/2834834911 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:29.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.775+0000 7f01bd848700 1 --2- 172.21.15.187:0/2834834911 >> v2:172.21.15.5:3300/0 conn(0x7f01b8065570 0x7f01b809d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:29.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.776+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693337302 0 0) 0x7f01b809d690 con 0x7f01b8065190 2020-11-20T12:11:29.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.776+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a0003240 con 0x7f01b8065190 2020-11-20T12:11:29.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.776+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3519869893 0 0) 0x7f01a0003240 con 0x7f01b8065190 2020-11-20T12:11:29.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.776+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 >> v2:172.21.15.5:3300/0 conn(0x7f01b8065570 msgr2=0x7f01b809d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:29.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.776+0000 7f01a7fff700 1 --2- 172.21.15.187:0/2834834911 >> v2:172.21.15.5:3300/0 conn(0x7f01b8065570 0x7f01b809d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:29.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.776+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 >> v1:172.21.15.5:6789/0 conn(0x7f01b8065f90 legacy=0x7f01b809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:29.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.776+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8062c10 con 0x7f01b8065190 2020-11-20T12:11:29.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.776+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f01b40014e0 con 0x7f01b8065190 2020-11-20T12:11:29.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 >> v1:172.21.15.187:6789/0 conn(0x7f01b8065190 legacy=0x7f01b8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01a7fff700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0003730 0x7f01a0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01a7fff700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00044e0 0x7f01a0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01a7fff700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a0004e50 0x7f01a000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b814ab30 con 0x7f01a0004e50 2020-11-20T12:11:29.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a0004280 con 0x7f01a0003730 2020-11-20T12:11:29.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a000bc20 con 0x7f01a00044e0 2020-11-20T12:11:29.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01bd047700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0003730 0x7f01a0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:29.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01bd848700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a0004e50 0x7f01a000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:29.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.777+0000 7f01bc846700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00044e0 0x7f01a0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:29.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01bd047700 1 -- 172.21.15.187:0/2834834911 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00044e0 msgr2=0x7f01a0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:29.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01bd047700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00044e0 0x7f01a0004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:29.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01bd047700 1 -- 172.21.15.187:0/2834834911 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a0004e50 msgr2=0x7f01a000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:29.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01bd047700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a0004e50 0x7f01a000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:29.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01bd047700 1 -- 172.21.15.187:0/2834834911 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a0003240 con 0x7f01a0003730 2020-11-20T12:11:29.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01bd848700 1 -- 172.21.15.187:0/2834834911 reap_dead start 2020-11-20T12:11:29.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01bd047700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0003730 0x7f01a0003b70 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f01b400c440 tx=0x7f01b80b0fb0).ready entity=mon.1 client_cookie=f68c83dbac1b80a1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:29.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b400ce70 con 0x7f01a0003730 2020-11-20T12:11:29.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b40072e0 con 0x7f01a0003730 2020-11-20T12:11:29.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.778+0000 7f01a7fff700 1 -- 172.21.15.187:0/2834834911 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4005820 con 0x7f01a0003730 2020-11-20T12:11:29.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.779+0000 7f01be049700 1 -- 172.21.15.187:0/2834834911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0003730 msgr2=0x7f01a0003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.779+0000 7f01be049700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0003730 0x7f01a0003b70 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f01b400c440 tx=0x7f01b80b0fb0).stop 2020-11-20T12:11:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.779+0000 7f01be049700 1 -- 172.21.15.187:0/2834834911 shutdown_connections 2020-11-20T12:11:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.779+0000 7f01be049700 1 --2- 172.21.15.187:0/2834834911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0003730 0x7f01a0003b70 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.779+0000 7f01be049700 1 -- 172.21.15.187:0/2834834911 >> 172.21.15.187:0/2834834911 conn(0x7f01b80bab30 msgr2=0x7f01b80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:29.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.779+0000 7f01be049700 1 -- 172.21.15.187:0/2834834911 shutdown_connections 2020-11-20T12:11:29.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.779+0000 7f01be049700 1 -- 172.21.15.187:0/2834834911 wait complete. 2020-11-20T12:11:29.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.780+0000 7f01be049700 1 Processor -- start 2020-11-20T12:11:29.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.780+0000 7f01be049700 1 -- start start 2020-11-20T12:11:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.781+0000 7f01be049700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a0003730 0x7f01b80a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.781+0000 7f01be049700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01b809e920 0x7f01b81466a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.781+0000 7f01be049700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01b8146b90 0x7f01b8142f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.781+0000 7f01be049700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b8067c80 con 0x7f01b809e920 2020-11-20T12:11:29.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.781+0000 7f01be049700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b8067f00 con 0x7f01b8146b90 2020-11-20T12:11:29.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.781+0000 7f01be049700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80681a0 con 0x7f01a0003730 2020-11-20T12:11:29.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.781+0000 7f01bd848700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01b8146b90 0x7f01b8142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:29.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd848700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01b8146b90 0x7f01b8142f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46176/0 (socket says 172.21.15.187:46176) 2020-11-20T12:11:29.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd848700 1 -- 172.21.15.187:0/193508942 learned_addr learned my addr 172.21.15.187:0/193508942 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:29.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd047700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a0003730 0x7f01b80a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bc846700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01b809e920 0x7f01b81466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd848700 1 -- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a0003730 msgr2=0x7f01b80a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd848700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a0003730 0x7f01b80a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd848700 1 -- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01b809e920 msgr2=0x7f01b81466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:29.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd848700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01b809e920 0x7f01b81466a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:29.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd848700 1 -- 172.21.15.187:0/193508942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b80a07e0 con 0x7f01b8146b90 2020-11-20T12:11:29.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd047700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a0003730 0x7f01b80a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:29.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bc846700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01b809e920 0x7f01b81466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:29.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01bd848700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01b8146b90 0x7f01b8142f00 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f01b805c8b0 tx=0x7f01b0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:29.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01a57fa700 1 -- 172.21.15.187:0/193508942 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0019070 con 0x7f01b8146b90 2020-11-20T12:11:29.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.782+0000 7f01a57fa700 1 -- 172.21.15.187:0/193508942 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b000eac0 con 0x7f01b8146b90 2020-11-20T12:11:29.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.783+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b81433f0 con 0x7f01b8146b90 2020-11-20T12:11:29.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.783+0000 7f01a57fa700 1 -- 172.21.15.187:0/193508942 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b000ee80 con 0x7f01b8146b90 2020-11-20T12:11:29.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.783+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b815ff30 con 0x7f01b8146b90 2020-11-20T12:11:29.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.784+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b80b7ae0 con 0x7f01b8146b90 2020-11-20T12:11:29.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.784+0000 7f01a57fa700 1 -- 172.21.15.187:0/193508942 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f01b0014680 con 0x7f01b8146b90 2020-11-20T12:11:29.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.785+0000 7f01a57fa700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0190056d20 0x7f01900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:29.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.785+0000 7f01a57fa700 1 -- 172.21.15.187:0/193508942 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f01b00707c0 con 0x7f01b8146b90 2020-11-20T12:11:29.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.785+0000 7f01bd047700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0190056d20 0x7f01900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:29.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.786+0000 7f01bd047700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0190056d20 0x7f01900591e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f01b40078a0 tx=0x7f01b4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:29.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:29.789+0000 7f01a57fa700 1 -- 172.21.15.187:0/193508942 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f01b0045bc0 con 0x7f01b8146b90 2020-11-20T12:11:30.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.015+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01b8068360 con 0x7f01b8146b90 2020-11-20T12:11:30.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.016+0000 7f01a57fa700 1 -- 172.21.15.187:0/193508942 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f01b00491e0 con 0x7f01b8146b90 2020-11-20T12:11:30.021 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:30.021 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:30.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0190056d20 msgr2=0x7f01900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:30.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0190056d20 0x7f01900591e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f01b40078a0 tx=0x7f01b4015000).stop 2020-11-20T12:11:30.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01b8146b90 msgr2=0x7f01b8142f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01b8146b90 0x7f01b8142f00 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f01b805c8b0 tx=0x7f01b0008040).stop 2020-11-20T12:11:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 shutdown_connections 2020-11-20T12:11:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0190056d20 0x7f01900591e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01b8146b90 0x7f01b8142f00 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a0003730 0x7f01b80a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 --2- 172.21.15.187:0/193508942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01b809e920 0x7f01b81466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.018+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 >> 172.21.15.187:0/193508942 conn(0x7f01b80bab30 msgr2=0x7f01b81442a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.019+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 shutdown_connections 2020-11-20T12:11:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.019+0000 7f01be049700 1 -- 172.21.15.187:0/193508942 wait complete. 2020-11-20T12:11:30.026 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:30.044 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_reintegrated 2020-11-20T12:11:30.343 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_reintegrated":1}} 2020-11-20T12:11:30.343 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2020-11-20T12:11:30.344 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:30.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.589+0000 7f3e71b1c700 1 Processor -- start 2020-11-20T12:11:30.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.589+0000 7f3e71b1c700 1 -- start start 2020-11-20T12:11:30.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.591+0000 7f3e71b1c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3e6c0654a0 0x7f3e6c0b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:30.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.591+0000 7f3e71b1c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:30.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.591+0000 7f3e71b1c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c0b08e0 con 0x7f3e6c0b6c10 2020-11-20T12:11:30.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.591+0000 7f3e71b1c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6c0b0b80 con 0x7f3e6c0654a0 2020-11-20T12:11:30.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.591+0000 7f3e71b1c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c0b05f0 con 0x7f3e6c065880 2020-11-20T12:11:30.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.592+0000 7f3e6affd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:30.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.592+0000 7f3e6affd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:47068/0 (socket says 172.21.15.187:47068) 2020-11-20T12:11:30.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.592+0000 7f3e6affd700 1 -- 172.21.15.187:0/3880046082 learned_addr learned my addr 172.21.15.187:0/3880046082 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:30.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.592+0000 7f3e6b7fe700 1 --2- 172.21.15.187:0/3880046082 >> v2:172.21.15.5:3300/0 conn(0x7f3e6c0654a0 0x7f3e6c0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:30.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.593+0000 7f3e6affd700 1 -- 172.21.15.187:0/3880046082 >> v2:172.21.15.5:3300/0 conn(0x7f3e6c0654a0 msgr2=0x7f3e6c0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:30.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.593+0000 7f3e6affd700 1 --2- 172.21.15.187:0/3880046082 >> v2:172.21.15.5:3300/0 conn(0x7f3e6c0654a0 0x7f3e6c0b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.593+0000 7f3e6a7fc700 1 -- 172.21.15.187:0/3880046082 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258998803 0 0) 0x7f3e6c0b08e0 con 0x7f3e6c0b6c10 2020-11-20T12:11:30.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.593+0000 7f3e6affd700 1 -- 172.21.15.187:0/3880046082 >> v1:172.21.15.5:6789/0 conn(0x7f3e6c0b6c10 legacy=0x7f3e6c0b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:30.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.593+0000 7f3e6affd700 1 -- 172.21.15.187:0/3880046082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c0676f0 con 0x7f3e6c065880 2020-11-20T12:11:30.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.593+0000 7f3e6affd700 1 --2- 172.21.15.187:0/3880046082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c144e60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f3e60006fd0 tx=0x7f3e60006e50).ready entity=mon.2 client_cookie=411f354ecb4ddd96 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:30.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.594+0000 7f3e6a7fc700 1 -- 172.21.15.187:0/3880046082 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60006030 con 0x7f3e6c065880 2020-11-20T12:11:30.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.594+0000 7f3e6a7fc700 1 -- 172.21.15.187:0/3880046082 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e60006190 con 0x7f3e6c065880 2020-11-20T12:11:30.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.594+0000 7f3e6a7fc700 1 -- 172.21.15.187:0/3880046082 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60006510 con 0x7f3e6c065880 2020-11-20T12:11:30.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.595+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3880046082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 msgr2=0x7f3e6c144e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:30.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.595+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3880046082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c144e60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f3e60006fd0 tx=0x7f3e60006e50).stop 2020-11-20T12:11:30.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.595+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3880046082 shutdown_connections 2020-11-20T12:11:30.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.595+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3880046082 >> v2:172.21.15.5:3300/0 conn(0x7f3e6c0654a0 0x7f3e6c0b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.595+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3880046082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c144e60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.595+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3880046082 >> 172.21.15.187:0/3880046082 conn(0x7f3e6c0bacd0 msgr2=0x7f3e6c0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:30.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.595+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3880046082 shutdown_connections 2020-11-20T12:11:30.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.595+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3880046082 wait complete. 2020-11-20T12:11:30.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.596+0000 7f3e71b1c700 1 Processor -- start 2020-11-20T12:11:30.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.597+0000 7f3e71b1c700 1 -- start start 2020-11-20T12:11:30.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.597+0000 7f3e71b1c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e6c0654a0 0x7f3e6c0a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:30.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.597+0000 7f3e71b1c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:30.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.597+0000 7f3e71b1c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e6c0b6c10 0x7f3e6c0b9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:30.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.597+0000 7f3e71b1c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c09fbc0 con 0x7f3e6c0b6c10 2020-11-20T12:11:30.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.597+0000 7f3e71b1c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c158f20 con 0x7f3e6c0654a0 2020-11-20T12:11:30.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.597+0000 7f3e71b1c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c09fdb0 con 0x7f3e6c065880 2020-11-20T12:11:30.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e6c0654a0 0x7f3e6c0a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:30.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e6c0654a0 0x7f3e6c0a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46184/0 (socket says 172.21.15.187:46184) 2020-11-20T12:11:30.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6b7fe700 1 -- 172.21.15.187:0/3408285223 learned_addr learned my addr 172.21.15.187:0/3408285223 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:30.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6affd700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:30.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6b7fe700 1 -- 172.21.15.187:0/3408285223 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 msgr2=0x7f3e6c158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:30.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6b7fe700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6b7fe700 1 -- 172.21.15.187:0/3408285223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e6c0b6c10 msgr2=0x7f3e6c0b9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:30.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6b7fe700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e6c0b6c10 0x7f3e6c0b9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:30.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.598+0000 7f3e6b7fe700 1 -- 172.21.15.187:0/3408285223 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c0a0050 con 0x7f3e6c0654a0 2020-11-20T12:11:30.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.599+0000 7f3e6b7fe700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e6c0654a0 0x7f3e6c0a0790 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f3e5c00ebd0 tx=0x7f3e5c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:30.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.599+0000 7f3e4b7fe700 1 -- 172.21.15.187:0/3408285223 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c00ca60 con 0x7f3e6c0654a0 2020-11-20T12:11:30.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.599+0000 7f3e4b7fe700 1 -- 172.21.15.187:0/3408285223 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c00cbc0 con 0x7f3e6c0654a0 2020-11-20T12:11:30.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.599+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c0a02f0 con 0x7f3e6c0654a0 2020-11-20T12:11:30.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.599+0000 7f3e4b7fe700 1 -- 172.21.15.187:0/3408285223 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c0123f0 con 0x7f3e6c0654a0 2020-11-20T12:11:30.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.599+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c09e130 con 0x7f3e6c0654a0 2020-11-20T12:11:30.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.600+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6c063600 con 0x7f3e6c0654a0 2020-11-20T12:11:30.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.601+0000 7f3e4b7fe700 1 -- 172.21.15.187:0/3408285223 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3e5c012760 con 0x7f3e6c0654a0 2020-11-20T12:11:30.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.601+0000 7f3e4b7fe700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e44056d20 0x7f3e440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:30.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.601+0000 7f3e4b7fe700 1 -- 172.21.15.187:0/3408285223 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3e5c070560 con 0x7f3e6c0654a0 2020-11-20T12:11:30.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.602+0000 7f3e6affd700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e44056d20 0x7f3e440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:30.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.602+0000 7f3e6affd700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e44056d20 0x7f3e440591e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f3e6c09f960 tx=0x7f3e60015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:30.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.605+0000 7f3e4b7fe700 1 -- 172.21.15.187:0/3408285223 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3e5c0458e0 con 0x7f3e6c0654a0 2020-11-20T12:11:30.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.844+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e6c09a5d0 con 0x7f3e6c0654a0 2020-11-20T12:11:30.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.846+0000 7f3e4b7fe700 1 -- 172.21.15.187:0/3408285223 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f3e5c048f00 con 0x7f3e6c0654a0 2020-11-20T12:11:30.851 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:30.852 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:30.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.848+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e44056d20 msgr2=0x7f3e440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:30.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.848+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e44056d20 0x7f3e440591e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f3e6c09f960 tx=0x7f3e60015000).stop 2020-11-20T12:11:30.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.848+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e6c0654a0 msgr2=0x7f3e6c0a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:30.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.848+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e6c0654a0 0x7f3e6c0a0790 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f3e5c00ebd0 tx=0x7f3e5c006df0).stop 2020-11-20T12:11:30.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.849+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 shutdown_connections 2020-11-20T12:11:30.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.849+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e44056d20 0x7f3e440591e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.849+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e6c0b6c10 0x7f3e6c0b9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:30.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.849+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e6c0654a0 0x7f3e6c0a0790 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.849+0000 7f3e71b1c700 1 --2- 172.21.15.187:0/3408285223 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e6c065880 0x7f3e6c158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:30.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.849+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 >> 172.21.15.187:0/3408285223 conn(0x7f3e6c0bacd0 msgr2=0x7f3e6c1419a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:30.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.849+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 shutdown_connections 2020-11-20T12:11:30.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:30.849+0000 7f3e71b1c700 1 -- 172.21.15.187:0/3408285223 wait complete. 2020-11-20T12:11:30.859 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:30.877 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2020-11-20T12:11:31.217 INFO:teuthology.orchestra.run.smithi005.stdout:{"message":"","return_code":0} 2020-11-20T12:11:31.218 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:11:31.219 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data getxattr 10000000001.00000000 parent 2020-11-20T12:11:31.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.314+0000 7f3fcf487d00 1 Processor -- start 2020-11-20T12:11:31.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.314+0000 7f3fcf487d00 1 -- start start 2020-11-20T12:11:31.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.315+0000 7f3fcf487d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5573648e1740 0x5573648a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.315+0000 7f3fcf487d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557364968e30 0x55736489f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.315+0000 7f3fcf487d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 0x5573648c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.315+0000 7f3fcf487d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55736481a810 con 0x5573648e1740 2020-11-20T12:11:31.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.315+0000 7f3fcf487d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x557364968100 con 0x557364968e30 2020-11-20T12:11:31.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.315+0000 7f3fcf487d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573648a0f90 con 0x55736496ef00 2020-11-20T12:11:31.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbf52b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557364968e30 0x55736489f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbf52b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557364968e30 0x55736489f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46202/0 (socket says 172.21.15.187:46202) 2020-11-20T12:11:31.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbf52b700 1 -- 172.21.15.187:0/1748191948 learned_addr learned my addr 172.21.15.187:0/1748191948 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:31.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fc052d700 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 0x5573648c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbfd2c700 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5573648e1740 0x5573648a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbf52b700 1 -- 172.21.15.187:0/1748191948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 msgr2=0x5573648c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:31.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbf52b700 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 0x5573648c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbf52b700 1 -- 172.21.15.187:0/1748191948 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5573648e1740 msgr2=0x5573648a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:31.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbf52b700 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5573648e1740 0x5573648a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.316+0000 7f3fbf52b700 1 -- 172.21.15.187:0/1748191948 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573648444e0 con 0x557364968e30 2020-11-20T12:11:31.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.317+0000 7f3fbf52b700 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557364968e30 0x55736489f900 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f3fb4008490 tx=0x7f3fb4015000).ready entity=mon.1 client_cookie=cd0727446d2cca19 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:31.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.317+0000 7f3fbe529700 1 -- 172.21.15.187:0/1748191948 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4008dc0 con 0x557364968e30 2020-11-20T12:11:31.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.317+0000 7f3fbe529700 1 -- 172.21.15.187:0/1748191948 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb4012600 con 0x557364968e30 2020-11-20T12:11:31.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.317+0000 7f3fbe529700 1 -- 172.21.15.187:0/1748191948 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4012980 con 0x557364968e30 2020-11-20T12:11:31.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.317+0000 7f3fcf487d00 1 -- 172.21.15.187:0/1748191948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557364968e30 msgr2=0x55736489f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.317+0000 7f3fcf487d00 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557364968e30 0x55736489f900 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f3fb4008490 tx=0x7f3fb4015000).stop 2020-11-20T12:11:31.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.318+0000 7f3fcf487d00 1 -- 172.21.15.187:0/1748191948 shutdown_connections 2020-11-20T12:11:31.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.318+0000 7f3fcf487d00 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5573648e1740 0x5573648a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.318+0000 7f3fcf487d00 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 0x5573648c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.318+0000 7f3fcf487d00 1 --2- 172.21.15.187:0/1748191948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557364968e30 0x55736489f900 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.318+0000 7f3fcf487d00 1 -- 172.21.15.187:0/1748191948 >> 172.21.15.187:0/1748191948 conn(0x55736489df00 msgr2=0x55736489f010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:31.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.318+0000 7f3fcf487d00 1 -- 172.21.15.187:0/1748191948 shutdown_connections 2020-11-20T12:11:31.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.318+0000 7f3fcf487d00 1 -- 172.21.15.187:0/1748191948 wait complete. 2020-11-20T12:11:31.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.319+0000 7f3fcf487d00 1 Processor -- start 2020-11-20T12:11:31.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.319+0000 7f3fcf487d00 1 -- start start 2020-11-20T12:11:31.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fcf487d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5573648e1740 0x557364969450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fcf487d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557364968e30 0x5573648c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fcf487d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 0x5573649638b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fcf487d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573649675b0 con 0x557364968e30 2020-11-20T12:11:31.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fcf487d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x557364967810 con 0x5573648e1740 2020-11-20T12:11:31.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fcf487d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x557364966270 con 0x55736496ef00 2020-11-20T12:11:31.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fbfd2c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5573648e1740 0x557364969450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fbfd2c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5573648e1740 0x557364969450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46204/0 (socket says 172.21.15.187:46204) 2020-11-20T12:11:31.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fbfd2c700 1 -- 172.21.15.187:0/3947215650 learned_addr learned my addr 172.21.15.187:0/3947215650 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:31.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fc052d700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 0x5573649638b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fbfd2c700 1 -- 172.21.15.187:0/3947215650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 msgr2=0x5573649638b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fbfd2c700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55736496ef00 0x5573649638b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fbfd2c700 1 -- 172.21.15.187:0/3947215650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557364968e30 msgr2=0x5573648c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.320+0000 7f3fbfd2c700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557364968e30 0x5573648c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:31.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.321+0000 7f3fbfd2c700 1 -- 172.21.15.187:0/3947215650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55736483fec0 con 0x5573648e1740 2020-11-20T12:11:31.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.321+0000 7f3fbfd2c700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5573648e1740 0x557364969450 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f3fb000a590 tx=0x7f3fb0001840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:31.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.321+0000 7f3fbcd26700 1 -- 172.21.15.187:0/3947215650 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb000ae40 con 0x5573648e1740 2020-11-20T12:11:31.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.321+0000 7f3fbcd26700 1 -- 172.21.15.187:0/3947215650 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb000e550 con 0x5573648e1740 2020-11-20T12:11:31.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.321+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55736481e250 con 0x5573648e1740 2020-11-20T12:11:31.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.321+0000 7f3fbcd26700 1 -- 172.21.15.187:0/3947215650 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb000e8f0 con 0x5573648e1740 2020-11-20T12:11:31.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.321+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557364966710 con 0x5573648e1740 2020-11-20T12:11:31.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.323+0000 7f3fbcd26700 1 -- 172.21.15.187:0/3947215650 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3fb000d040 con 0x5573648e1740 2020-11-20T12:11:31.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.324+0000 7f3fbcd26700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3fac0559a0 0x7f3fac057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.324+0000 7f3fbcd26700 1 -- 172.21.15.187:0/3947215650 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3fb006ec80 con 0x5573648e1740 2020-11-20T12:11:31.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.324+0000 7f3fcf487d00 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55736496ba70 0x557364969f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.324+0000 7f3fbf52b700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3fac0559a0 0x7f3fac057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.324+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:1 14.7 14:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected e72) v8 -- 0x55736496a440 con 0x55736496ba70 2020-11-20T12:11:31.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.324+0000 7f3fc052d700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55736496ba70 0x557364969f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.325+0000 7f3fbf52b700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3fac0559a0 0x7f3fac057e60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f3fb401f000 tx=0x7f3fb401b000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:31.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.325+0000 7f3fc052d700 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55736496ba70 0x557364969f50 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:31.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fc052d700 1 -- 172.21.15.187:0/3947215650 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv6 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f3fb800ab20 con 0x55736496ba70 2020-11-20T12:11:31.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55736496ba70 msgr2=0x557364969f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55736496ba70 0x557364969f50 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3fac0559a0 msgr2=0x7f3fac057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3fac0559a0 0x7f3fac057e60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f3fb401f000 tx=0x7f3fb401b000).stop 2020-11-20T12:11:31.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5573648e1740 msgr2=0x557364969450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 --2- 172.21.15.187:0/3947215650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5573648e1740 0x557364969450 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f3fb000a590 tx=0x7f3fb0001840).stop 2020-11-20T12:11:31.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fc052d700 1 -- 172.21.15.187:0/3947215650 reap_dead start 2020-11-20T12:11:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 shutdown_connections 2020-11-20T12:11:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 >> 172.21.15.187:0/3947215650 conn(0x55736489df00 msgr2=0x55736489f010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.326+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 shutdown_connections 2020-11-20T12:11:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.327+0000 7f3fcf487d00 1 -- 172.21.15.187:0/3947215650 wait complete. 2020-11-20T12:11:31.341 INFO:teuthology.orchestra.run.smithi187.stdout:W�file_cdir_2! 2020-11-20T12:11:31.342 INFO:teuthology.orchestra.run.smithi187:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2020-11-20T12:11:31.432 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:11:31.433 INFO:teuthology.orchestra.run.smithi187.stdout: "ino": 1099511627777, 2020-11-20T12:11:31.433 INFO:teuthology.orchestra.run.smithi187.stdout: "ancestors": [ 2020-11-20T12:11:31.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:11:31.433 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1099511628278, 2020-11-20T12:11:31.433 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "file_c", 2020-11-20T12:11:31.434 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 17 2020-11-20T12:11:31.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:11:31.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:11:31.434 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1, 2020-11-20T12:11:31.434 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "dir_2", 2020-11-20T12:11:31.435 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 33 2020-11-20T12:11:31.435 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:11:31.435 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2020-11-20T12:11:31.435 INFO:teuthology.orchestra.run.smithi187.stdout: "pool": 14, 2020-11-20T12:11:31.435 INFO:teuthology.orchestra.run.smithi187.stdout: "old_pools": [] 2020-11-20T12:11:31.436 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:11:31.436 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:31.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:11:31.444 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo ln dir_2/file_c dir_2/file_d) 2020-11-20T12:11:31.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:11:31.559 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -f dir_2/file_c) 2020-11-20T12:11:31.670 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:31.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.902+0000 7fa42b08b700 1 Processor -- start 2020-11-20T12:11:31.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.903+0000 7fa42b08b700 1 -- start start 2020-11-20T12:11:31.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.904+0000 7fa42b08b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa4240666e0 0x7fa4240b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.904+0000 7fa42b08b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa424061e80 con 0x7fa42414c750 2020-11-20T12:11:31.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.904+0000 7fa42b08b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4240620e0 con 0x7fa424067d30 2020-11-20T12:11:31.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.904+0000 7fa42b08b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa424062c90 con 0x7fa4240666e0 2020-11-20T12:11:31.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.905+0000 7fa429628700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa424067d30 0x7fa4241503d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42824/0 (socket says 172.21.15.187:42824) 2020-11-20T12:11:31.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.905+0000 7fa429628700 1 -- 172.21.15.187:0/2304154275 learned_addr learned my addr 172.21.15.187:0/2304154275 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:31.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.905+0000 7fa41bfff700 1 --2- 172.21.15.187:0/2304154275 >> v2:172.21.15.5:3300/0 conn(0x7fa4240666e0 0x7fa4240b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198378959 0 0) 0x7fa4240620e0 con 0x7fa424067d30 2020-11-20T12:11:31.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa410003240 con 0x7fa424067d30 2020-11-20T12:11:31.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54402381 0 0) 0x7fa424061e80 con 0x7fa42414c750 2020-11-20T12:11:31.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4240620e0 con 0x7fa42414c750 2020-11-20T12:11:31.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1180696643 0 0) 0x7fa410003240 con 0x7fa424067d30 2020-11-20T12:11:31.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 >> v2:172.21.15.5:3300/0 conn(0x7fa4240666e0 msgr2=0x7fa4240b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:31.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 --2- 172.21.15.187:0/2304154275 >> v2:172.21.15.5:3300/0 conn(0x7fa4240666e0 0x7fa4240b0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 >> v1:172.21.15.5:6789/0 conn(0x7fa42414c750 legacy=0x7fa424062580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4240b1500 con 0x7fa424067d30 2020-11-20T12:11:31.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.906+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fa41c001860 con 0x7fa424067d30 2020-11-20T12:11:31.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.907+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 >> v1:172.21.15.187:6789/0 conn(0x7fa424067d30 legacy=0x7fa4241503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.907+0000 7fa41b7fe700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4100038c0 0x7fa410003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.907+0000 7fa41b7fe700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa4100044e0 0x7fa410004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.907+0000 7fa41b7fe700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa410004e50 0x7fa41000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.907+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa424062c90 con 0x7fa4100038c0 2020-11-20T12:11:31.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.907+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100053b0 con 0x7fa410004e50 2020-11-20T12:11:31.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.907+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41000bca0 con 0x7fa4100044e0 2020-11-20T12:11:31.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.907+0000 7fa429628700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa410004e50 0x7fa41000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.908+0000 7fa41bfff700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa4100044e0 0x7fa410004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.908+0000 7fa428e27700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4100038c0 0x7fa410003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.908+0000 7fa429628700 1 -- 172.21.15.187:0/2304154275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa4100044e0 msgr2=0x7fa410004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:31.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.908+0000 7fa429628700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa4100044e0 0x7fa410004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.908+0000 7fa429628700 1 -- 172.21.15.187:0/2304154275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4100038c0 msgr2=0x7fa410003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:31.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.908+0000 7fa429628700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4100038c0 0x7fa410003cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.908+0000 7fa429628700 1 -- 172.21.15.187:0/2304154275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa410003240 con 0x7fa410004e50 2020-11-20T12:11:31.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.909+0000 7fa429628700 1 -- 172.21.15.187:0/2304154275 reap_dead start 2020-11-20T12:11:31.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.909+0000 7fa429628700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa410004e50 0x7fa41000b730 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7fa41c00c5d0 tx=0x7fa41c007ad0).ready entity=mon.1 client_cookie=d44e7d8b08f1d571 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:31.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.909+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c005b80 con 0x7fa410004e50 2020-11-20T12:11:31.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.909+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c005ce0 con 0x7fa410004e50 2020-11-20T12:11:31.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.909+0000 7fa41b7fe700 1 -- 172.21.15.187:0/2304154275 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c007320 con 0x7fa410004e50 2020-11-20T12:11:31.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.910+0000 7fa42b08b700 1 -- 172.21.15.187:0/2304154275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa410004e50 msgr2=0x7fa41000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.910+0000 7fa42b08b700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa410004e50 0x7fa41000b730 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7fa41c00c5d0 tx=0x7fa41c007ad0).stop 2020-11-20T12:11:31.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.910+0000 7fa42b08b700 1 -- 172.21.15.187:0/2304154275 shutdown_connections 2020-11-20T12:11:31.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.910+0000 7fa42b08b700 1 --2- 172.21.15.187:0/2304154275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa410004e50 0x7fa41000b730 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.910+0000 7fa42b08b700 1 -- 172.21.15.187:0/2304154275 >> 172.21.15.187:0/2304154275 conn(0x7fa4240bacd0 msgr2=0x7fa4240bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:31.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.910+0000 7fa42b08b700 1 -- 172.21.15.187:0/2304154275 shutdown_connections 2020-11-20T12:11:31.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.910+0000 7fa42b08b700 1 -- 172.21.15.187:0/2304154275 wait complete. 2020-11-20T12:11:31.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.912+0000 7fa42b08b700 1 Processor -- start 2020-11-20T12:11:31.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.912+0000 7fa42b08b700 1 -- start start 2020-11-20T12:11:31.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.913+0000 7fa42b08b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa410004e50 0x7fa42415c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.913+0000 7fa42b08b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42414fbe0 0x7fa42415c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.913+0000 7fa42b08b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4240b6c60 0x7fa424151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.913+0000 7fa42b08b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42415d1a0 con 0x7fa4240b6c60 2020-11-20T12:11:31.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.913+0000 7fa42b08b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4240b3cb0 con 0x7fa42414fbe0 2020-11-20T12:11:31.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.913+0000 7fa42b08b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42415cf00 con 0x7fa410004e50 2020-11-20T12:11:31.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa41bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42414fbe0 0x7fa42415c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa41bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42414fbe0 0x7fa42415c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46228/0 (socket says 172.21.15.187:46228) 2020-11-20T12:11:31.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa41bfff700 1 -- 172.21.15.187:0/439049433 learned_addr learned my addr 172.21.15.187:0/439049433 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:31.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa428e27700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa410004e50 0x7fa42415c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa41bfff700 1 -- 172.21.15.187:0/439049433 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa410004e50 msgr2=0x7fa42415c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa41bfff700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa410004e50 0x7fa42415c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:31.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa41bfff700 1 -- 172.21.15.187:0/439049433 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4240b6c60 msgr2=0x7fa424151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:31.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa41bfff700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4240b6c60 0x7fa424151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:31.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.914+0000 7fa41bfff700 1 -- 172.21.15.187:0/439049433 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4240b4110 con 0x7fa42414fbe0 2020-11-20T12:11:31.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.915+0000 7fa41bfff700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42414fbe0 0x7fa42415c8b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7fa414006a30 tx=0x7fa414008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:31.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.915+0000 7fa418ff9700 1 -- 172.21.15.187:0/439049433 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4140117a0 con 0x7fa42414fbe0 2020-11-20T12:11:31.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.915+0000 7fa418ff9700 1 -- 172.21.15.187:0/439049433 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa414011900 con 0x7fa42414fbe0 2020-11-20T12:11:31.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.915+0000 7fa418ff9700 1 -- 172.21.15.187:0/439049433 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414011ca0 con 0x7fa42414fbe0 2020-11-20T12:11:31.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.915+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa424151a80 con 0x7fa42414fbe0 2020-11-20T12:11:31.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.915+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42415f720 con 0x7fa42414fbe0 2020-11-20T12:11:31.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.916+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4240b3b10 con 0x7fa42414fbe0 2020-11-20T12:11:31.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.917+0000 7fa418ff9700 1 -- 172.21.15.187:0/439049433 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa414014440 con 0x7fa42414fbe0 2020-11-20T12:11:31.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.917+0000 7fa418ff9700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa3fc056d20 0x7fa3fc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:31.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.917+0000 7fa418ff9700 1 -- 172.21.15.187:0/439049433 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa4140705f0 con 0x7fa42414fbe0 2020-11-20T12:11:31.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.917+0000 7fa428e27700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa3fc056d20 0x7fa3fc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:31.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.918+0000 7fa428e27700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa3fc056d20 0x7fa3fc0591e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fa420007b60 tx=0x7fa420009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:31.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:31.922+0000 7fa418ff9700 1 -- 172.21.15.187:0/439049433 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa4240b3b10 con 0x7fa42414fbe0 2020-11-20T12:11:32.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.169+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4240b3b10 con 0x7fa42414fbe0 2020-11-20T12:11:32.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.170+0000 7fa418ff9700 1 -- 172.21.15.187:0/439049433 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fa4240b3b10 con 0x7fa42414fbe0 2020-11-20T12:11:32.176 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:32.176 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:32.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa3fc056d20 msgr2=0x7fa3fc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:32.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa3fc056d20 0x7fa3fc0591e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fa420007b60 tx=0x7fa420009040).stop 2020-11-20T12:11:32.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42414fbe0 msgr2=0x7fa42415c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:32.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42414fbe0 0x7fa42415c8b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7fa414006a30 tx=0x7fa414008040).stop 2020-11-20T12:11:32.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 shutdown_connections 2020-11-20T12:11:32.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa3fc056d20 0x7fa3fc0591e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:32.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4240b6c60 0x7fa424151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:32.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa410004e50 0x7fa42415c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:32.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 --2- 172.21.15.187:0/439049433 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42414fbe0 0x7fa42415c8b0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:32.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 >> 172.21.15.187:0/439049433 conn(0x7fa4240bacd0 msgr2=0x7fa42414b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:32.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.172+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 shutdown_connections 2020-11-20T12:11:32.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.173+0000 7fa42b08b700 1 -- 172.21.15.187:0/439049433 wait complete. 2020-11-20T12:11:32.188 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:32.201 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache num_strays 2020-11-20T12:11:32.507 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"num_strays":0}} 2020-11-20T12:11:32.508 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2020-11-20T12:11:32.508 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:32.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.760+0000 7f1b4759e700 1 Processor -- start 2020-11-20T12:11:32.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.760+0000 7f1b4759e700 1 -- start start 2020-11-20T12:11:32.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.761+0000 7f1b4759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48061f90 0x7f1b480afca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:32.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.762+0000 7f1b4759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1b48062370 0x7f1b48145360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:32.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.762+0000 7f1b4759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b480685f0 con 0x7f1b48063dd0 2020-11-20T12:11:32.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.762+0000 7f1b4759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b48145850 con 0x7f1b48062370 2020-11-20T12:11:32.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.762+0000 7f1b4759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b48067e10 con 0x7f1b48061f90 2020-11-20T12:11:32.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.762+0000 7f1b4659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48061f90 0x7f1b480afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:32.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.763+0000 7f1b4659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48061f90 0x7f1b480afca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:47122/0 (socket says 172.21.15.187:47122) 2020-11-20T12:11:32.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.763+0000 7f1b4659c700 1 -- 172.21.15.187:0/512812478 learned_addr learned my addr 172.21.15.187:0/512812478 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:32.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.763+0000 7f1b3dd9b700 1 --2- 172.21.15.187:0/512812478 >> v2:172.21.15.187:3300/0 conn(0x7f1b48062370 0x7f1b48145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:32.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.763+0000 7f1b45d9b700 1 -- 172.21.15.187:0/512812478 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746558587 0 0) 0x7f1b480685f0 con 0x7f1b48063dd0 2020-11-20T12:11:32.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.763+0000 7f1b45d9b700 1 -- 172.21.15.187:0/512812478 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28003240 con 0x7f1b48063dd0 2020-11-20T12:11:32.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.763+0000 7f1b4659c700 1 -- 172.21.15.187:0/512812478 >> v2:172.21.15.187:3300/0 conn(0x7f1b48062370 msgr2=0x7f1b48145360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:32.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.763+0000 7f1b45d9b700 1 -- 172.21.15.187:0/512812478 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 747388145 0 0) 0x7f1b28003240 con 0x7f1b48063dd0 2020-11-20T12:11:32.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.764+0000 7f1b4659c700 1 --2- 172.21.15.187:0/512812478 >> v2:172.21.15.187:3300/0 conn(0x7f1b48062370 0x7f1b48145360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:32.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.764+0000 7f1b4659c700 1 -- 172.21.15.187:0/512812478 >> v1:172.21.15.187:6789/0 conn(0x7f1b48063dd0 legacy=0x7f1b48148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:32.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.764+0000 7f1b4659c700 1 -- 172.21.15.187:0/512812478 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b480680b0 con 0x7f1b48061f90 2020-11-20T12:11:32.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.764+0000 7f1b3dd9b700 1 --2- 172.21.15.187:0/512812478 >> v2:172.21.15.187:3300/0 conn(0x7f1b48062370 0x7f1b48145360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:11:32.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.764+0000 7f1b4659c700 1 --2- 172.21.15.187:0/512812478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48061f90 0x7f1b480afca0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1b38006fd0 tx=0x7f1b38006eb0).ready entity=mon.2 client_cookie=bd08274650773cf8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:32.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.765+0000 7f1b45d9b700 1 -- 172.21.15.187:0/512812478 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40001860 con 0x7f1b48061f90 2020-11-20T12:11:32.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.765+0000 7f1b45d9b700 1 -- 172.21.15.187:0/512812478 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b38009500 con 0x7f1b48061f90 2020-11-20T12:11:32.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.765+0000 7f1b45d9b700 1 -- 172.21.15.187:0/512812478 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b38009880 con 0x7f1b48061f90 2020-11-20T12:11:32.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.765+0000 7f1b4759e700 1 -- 172.21.15.187:0/512812478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48061f90 msgr2=0x7f1b480afca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:32.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.765+0000 7f1b4759e700 1 --2- 172.21.15.187:0/512812478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48061f90 0x7f1b480afca0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1b38006fd0 tx=0x7f1b38006eb0).stop 2020-11-20T12:11:32.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.766+0000 7f1b4759e700 1 -- 172.21.15.187:0/512812478 shutdown_connections 2020-11-20T12:11:32.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.766+0000 7f1b4759e700 1 --2- 172.21.15.187:0/512812478 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48061f90 0x7f1b480afca0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:32.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.766+0000 7f1b4759e700 1 --2- 172.21.15.187:0/512812478 >> v2:172.21.15.187:3300/0 conn(0x7f1b48062370 0x7f1b48145360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:32.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.766+0000 7f1b4759e700 1 -- 172.21.15.187:0/512812478 >> 172.21.15.187:0/512812478 conn(0x7f1b480bab30 msgr2=0x7f1b480baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:32.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.766+0000 7f1b4759e700 1 -- 172.21.15.187:0/512812478 shutdown_connections 2020-11-20T12:11:32.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.766+0000 7f1b4759e700 1 -- 172.21.15.187:0/512812478 wait complete. 2020-11-20T12:11:32.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.767+0000 7f1b4759e700 1 Processor -- start 2020-11-20T12:11:32.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.767+0000 7f1b4759e700 1 -- start start 2020-11-20T12:11:32.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.768+0000 7f1b4759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b48061f90 0x7f1b4813fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:32.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.768+0000 7f1b4759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b48062370 0x7f1b4813e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:32.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.768+0000 7f1b4759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48063dd0 0x7f1b480b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:32.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.768+0000 7f1b4759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4809cb00 con 0x7f1b48062370 2020-11-20T12:11:32.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.768+0000 7f1b4759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b48140130 con 0x7f1b48061f90 2020-11-20T12:11:32.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.768+0000 7f1b4759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4809c860 con 0x7f1b48063dd0 2020-11-20T12:11:32.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b4659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b48061f90 0x7f1b4813fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:32.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b4659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b48061f90 0x7f1b4813fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46240/0 (socket says 172.21.15.187:46240) 2020-11-20T12:11:32.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b4659c700 1 -- 172.21.15.187:0/2767193148 learned_addr learned my addr 172.21.15.187:0/2767193148 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:32.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b46d9d700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48063dd0 0x7f1b480b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:32.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b3dd9b700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b48062370 0x7f1b4813e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:32.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b4659c700 1 -- 172.21.15.187:0/2767193148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48063dd0 msgr2=0x7f1b480b5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:32.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b4659c700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48063dd0 0x7f1b480b5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:32.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b4659c700 1 -- 172.21.15.187:0/2767193148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b48062370 msgr2=0x7f1b4813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:32.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b4659c700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b48062370 0x7f1b4813e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:32.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.769+0000 7f1b4659c700 1 -- 172.21.15.187:0/2767193148 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b481415b0 con 0x7f1b48061f90 2020-11-20T12:11:32.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.770+0000 7f1b4659c700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b48061f90 0x7f1b4813fc40 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f1b48063a20 tx=0x7f1b38015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:32.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.770+0000 7f1b3effd700 1 -- 172.21.15.187:0/2767193148 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b38011a70 con 0x7f1b48061f90 2020-11-20T12:11:32.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.770+0000 7f1b3effd700 1 -- 172.21.15.187:0/2767193148 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b38011bd0 con 0x7f1b48061f90 2020-11-20T12:11:32.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.770+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4813e7e0 con 0x7f1b48061f90 2020-11-20T12:11:32.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.770+0000 7f1b3effd700 1 -- 172.21.15.187:0/2767193148 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3801e460 con 0x7f1b48061f90 2020-11-20T12:11:32.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.770+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4809c4e0 con 0x7f1b48061f90 2020-11-20T12:11:32.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.771+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b48061a20 con 0x7f1b48061f90 2020-11-20T12:11:32.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.772+0000 7f1b3effd700 1 -- 172.21.15.187:0/2767193148 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1b38025030 con 0x7f1b48061f90 2020-11-20T12:11:32.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.773+0000 7f1b3effd700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b24056d20 0x7f1b240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:32.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.773+0000 7f1b3effd700 1 -- 172.21.15.187:0/2767193148 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1b380714a0 con 0x7f1b48061f90 2020-11-20T12:11:32.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.773+0000 7f1b3dd9b700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b24056d20 0x7f1b240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:32.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.774+0000 7f1b3dd9b700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b24056d20 0x7f1b240591e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f1b30006bd0 tx=0x7f1b30008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:32.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:32.777+0000 7f1b3effd700 1 -- 172.21.15.187:0/2767193148 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1b38046840 con 0x7f1b48061f90 2020-11-20T12:11:33.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.022+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b480446d0 con 0x7f1b48061f90 2020-11-20T12:11:33.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.024+0000 7f1b3effd700 1 -- 172.21.15.187:0/2767193148 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f1b38023070 con 0x7f1b48061f90 2020-11-20T12:11:33.028 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:33.028 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b24056d20 msgr2=0x7f1b240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b24056d20 0x7f1b240591e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f1b30006bd0 tx=0x7f1b30008040).stop 2020-11-20T12:11:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b48061f90 msgr2=0x7f1b4813fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b48061f90 0x7f1b4813fc40 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f1b48063a20 tx=0x7f1b38015000).stop 2020-11-20T12:11:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 shutdown_connections 2020-11-20T12:11:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b24056d20 0x7f1b240591e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b48063dd0 0x7f1b480b5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b48061f90 0x7f1b4813fc40 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 --2- 172.21.15.187:0/2767193148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b48062370 0x7f1b4813e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 >> 172.21.15.187:0/2767193148 conn(0x7f1b480bab30 msgr2=0x7f1b480b7210 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:33.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 shutdown_connections 2020-11-20T12:11:33.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.026+0000 7f1b4759e700 1 -- 172.21.15.187:0/2767193148 wait complete. 2020-11-20T12:11:33.034 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:33.054 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_reintegrated 2020-11-20T12:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_reintegrated":3}} 2020-11-20T12:11:33.353 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 3 } } 2020-11-20T12:11:33.353 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:33.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.600+0000 7f07ba386700 1 Processor -- start 2020-11-20T12:11:33.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.600+0000 7f07ba386700 1 -- start start 2020-11-20T12:11:33.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.601+0000 7f07ba386700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f07b40bf800 0x7f07b414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:33.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.601+0000 7f07ba386700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40bfbe0 0x7f07b41480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:33.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.601+0000 7f07ba386700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b414d110 con 0x7f07b409f560 2020-11-20T12:11:33.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.601+0000 7f07ba386700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b414cbd0 con 0x7f07b40bf800 2020-11-20T12:11:33.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.601+0000 7f07ba386700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b414ce70 con 0x7f07b40bfbe0 2020-11-20T12:11:33.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.602+0000 7f07b9384700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f07b40bf800 0x7f07b414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:33.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.602+0000 7f07b9384700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f07b40bf800 0x7f07b414c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46248/0 (socket says 172.21.15.187:46248) 2020-11-20T12:11:33.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.602+0000 7f07b9384700 1 -- 172.21.15.187:0/3775745260 learned_addr learned my addr 172.21.15.187:0/3775745260 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:33.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.602+0000 7f07b8b83700 1 --2- 172.21.15.187:0/3775745260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40bfbe0 0x7f07b41480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:33.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07b9384700 1 -- 172.21.15.187:0/3775745260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40bfbe0 msgr2=0x7f07b41480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:33.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07b9384700 1 --2- 172.21.15.187:0/3775745260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40bfbe0 0x7f07b41480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07b9384700 1 -- 172.21.15.187:0/3775745260 >> v1:172.21.15.5:6789/0 conn(0x7f07b409f560 legacy=0x7f07b4060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:33.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07b9384700 1 -- 172.21.15.187:0/3775745260 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b40a4cc0 con 0x7f07b40bf800 2020-11-20T12:11:33.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07b8b83700 1 --2- 172.21.15.187:0/3775745260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40bfbe0 0x7f07b41480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:33.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07b9384700 1 --2- 172.21.15.187:0/3775745260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40bf800 0x7f07b414c2c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f07b0006fd0 tx=0x7f07b0006eb0).ready entity=mon.1 client_cookie=72859ece7ae550e8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:33.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07a3fff700 1 -- 172.21.15.187:0/3775745260 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0009920 con 0x7f07b40bf800 2020-11-20T12:11:33.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07a3fff700 1 -- 172.21.15.187:0/3775745260 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b0009a80 con 0x7f07b40bf800 2020-11-20T12:11:33.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.603+0000 7f07a3fff700 1 -- 172.21.15.187:0/3775745260 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0009e00 con 0x7f07b40bf800 2020-11-20T12:11:33.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.604+0000 7f07ba386700 1 -- 172.21.15.187:0/3775745260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40bf800 msgr2=0x7f07b414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:33.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.604+0000 7f07ba386700 1 --2- 172.21.15.187:0/3775745260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40bf800 0x7f07b414c2c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f07b0006fd0 tx=0x7f07b0006eb0).stop 2020-11-20T12:11:33.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.604+0000 7f07ba386700 1 -- 172.21.15.187:0/3775745260 shutdown_connections 2020-11-20T12:11:33.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.604+0000 7f07ba386700 1 --2- 172.21.15.187:0/3775745260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40bf800 0x7f07b414c2c0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.604+0000 7f07ba386700 1 --2- 172.21.15.187:0/3775745260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40bfbe0 0x7f07b41480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.604+0000 7f07ba386700 1 -- 172.21.15.187:0/3775745260 >> 172.21.15.187:0/3775745260 conn(0x7f07b4068370 msgr2=0x7f07b40bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:33.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.604+0000 7f07ba386700 1 -- 172.21.15.187:0/3775745260 shutdown_connections 2020-11-20T12:11:33.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.604+0000 7f07ba386700 1 -- 172.21.15.187:0/3775745260 wait complete. 2020-11-20T12:11:33.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.606+0000 7f07ba386700 1 Processor -- start 2020-11-20T12:11:33.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.606+0000 7f07ba386700 1 -- start start 2020-11-20T12:11:33.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07ba386700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b409f560 0x7f07b414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:33.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07ba386700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40baeb0 0x7f07b4143680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07ba386700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40b3e90 0x7f07b4143b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07ba386700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4144060 con 0x7f07b409f560 2020-11-20T12:11:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07ba386700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40ba4d0 con 0x7f07b40b3e90 2020-11-20T12:11:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07ba386700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40ba770 con 0x7f07b40baeb0 2020-11-20T12:11:33.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07b9b85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40b3e90 0x7f07b4143b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:33.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07b9b85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40b3e90 0x7f07b4143b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46258/0 (socket says 172.21.15.187:46258) 2020-11-20T12:11:33.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.607+0000 7f07b9b85700 1 -- 172.21.15.187:0/3048207548 learned_addr learned my addr 172.21.15.187:0/3048207548 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b9384700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b409f560 0x7f07b414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b9b85700 1 -- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40baeb0 msgr2=0x7f07b4143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b8b83700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40baeb0 0x7f07b4143680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b9b85700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40baeb0 0x7f07b4143680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b9b85700 1 -- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b409f560 msgr2=0x7f07b414a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:33.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b9b85700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b409f560 0x7f07b414a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b9b85700 1 -- 172.21.15.187:0/3048207548 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b40baa10 con 0x7f07b40b3e90 2020-11-20T12:11:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b9384700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b409f560 0x7f07b414a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:11:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.608+0000 7f07b9b85700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40b3e90 0x7f07b4143b70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f07ac00cab0 tx=0x7f07ac0023b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.609+0000 7f07a17fa700 1 -- 172.21.15.187:0/3048207548 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac00a8c0 con 0x7f07b40b3e90 2020-11-20T12:11:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.609+0000 7f07a17fa700 1 -- 172.21.15.187:0/3048207548 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac00aa20 con 0x7f07b40b3e90 2020-11-20T12:11:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.609+0000 7f07a17fa700 1 -- 172.21.15.187:0/3048207548 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac00adc0 con 0x7f07b40b3e90 2020-11-20T12:11:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.609+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b40b77f0 con 0x7f07b40b3e90 2020-11-20T12:11:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.609+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b40ba080 con 0x7f07b40b3e90 2020-11-20T12:11:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.610+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b4080050 con 0x7f07b40b3e90 2020-11-20T12:11:33.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.611+0000 7f07a17fa700 1 -- 172.21.15.187:0/3048207548 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f07ac019030 con 0x7f07b40b3e90 2020-11-20T12:11:33.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.611+0000 7f07a17fa700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f078c056d50 0x7f078c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:33.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.611+0000 7f07a17fa700 1 -- 172.21.15.187:0/3048207548 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f07ac06f8c0 con 0x7f07b40b3e90 2020-11-20T12:11:33.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.612+0000 7f07b9384700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f078c056d50 0x7f078c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:33.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.612+0000 7f07b9384700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f078c056d50 0x7f078c059210 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f07b4068830 tx=0x7f07b0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:33.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.615+0000 7f07a17fa700 1 -- 172.21.15.187:0/3048207548 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f07ac043fd0 con 0x7f07b40b3e90 2020-11-20T12:11:33.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.859+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07b40ba300 con 0x7f07b40b3e90 2020-11-20T12:11:33.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.860+0000 7f07a17fa700 1 -- 172.21.15.187:0/3048207548 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f07ac044d70 con 0x7f07b40b3e90 2020-11-20T12:11:33.864 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:33.865 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:33.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.862+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f078c056d50 msgr2=0x7f078c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:33.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.862+0000 7f07ba386700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f078c056d50 0x7f078c059210 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f07b4068830 tx=0x7f07b0015000).stop 2020-11-20T12:11:33.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.862+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40b3e90 msgr2=0x7f07b4143b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:33.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.862+0000 7f07ba386700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40b3e90 0x7f07b4143b70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f07ac00cab0 tx=0x7f07ac0023b0).stop 2020-11-20T12:11:33.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.863+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 shutdown_connections 2020-11-20T12:11:33.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.863+0000 7f07ba386700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f078c056d50 0x7f078c059210 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.863+0000 7f07ba386700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b409f560 0x7f07b414a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.863+0000 7f07ba386700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07b40b3e90 0x7f07b4143b70 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.863+0000 7f07ba386700 1 --2- 172.21.15.187:0/3048207548 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f07b40baeb0 0x7f07b4143680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:33.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.863+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 >> 172.21.15.187:0/3048207548 conn(0x7f07b4068370 msgr2=0x7f07b4146680 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:33.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.863+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 shutdown_connections 2020-11-20T12:11:33.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:33.863+0000 7f07ba386700 1 -- 172.21.15.187:0/3048207548 wait complete. 2020-11-20T12:11:33.871 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:33.889 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2020-11-20T12:11:34.213 INFO:teuthology.orchestra.run.smithi005.stdout:{"message":"","return_code":0} 2020-11-20T12:11:34.213 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:11:34.214 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data getxattr 10000000001.00000000 parent 2020-11-20T12:11:34.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.304+0000 7f968e9d3d00 1 Processor -- start 2020-11-20T12:11:34.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.304+0000 7f968e9d3d00 1 -- start start 2020-11-20T12:11:34.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.305+0000 7f968e9d3d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a223a0 0x55b299aeade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.305+0000 7f968e9d3d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 0x55b299a484c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.305+0000 7f968e9d3d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a63ad0 0x55b299ae8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.305+0000 7f968e9d3d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b299ae9e70 con 0x55b299a49d40 2020-11-20T12:11:34.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.305+0000 7f968e9d3d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b299aefd20 con 0x55b299a223a0 2020-11-20T12:11:34.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.305+0000 7f968e9d3d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b299ae9bd0 con 0x55b299a63ad0 2020-11-20T12:11:34.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967f278700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a223a0 0x55b299aeade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967f278700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a223a0 0x55b299aeade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46268/0 (socket says 172.21.15.187:46268) 2020-11-20T12:11:34.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967f278700 1 -- 172.21.15.187:0/841916189 learned_addr learned my addr 172.21.15.187:0/841916189 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:34.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967fa79700 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a63ad0 0x55b299ae8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f9676a77700 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 0x55b299a484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967f278700 1 -- 172.21.15.187:0/841916189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a63ad0 msgr2=0x55b299ae8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:34.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967f278700 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a63ad0 0x55b299ae8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967f278700 1 -- 172.21.15.187:0/841916189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 msgr2=0x55b299a484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:34.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967f278700 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 0x55b299a484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.306+0000 7f967f278700 1 -- 172.21.15.187:0/841916189 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2999c60c0 con 0x55b299a223a0 2020-11-20T12:11:34.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f967f278700 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a223a0 0x55b299aeade0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f9670009b20 tx=0x7f9670005bc0).ready entity=mon.1 client_cookie=24526caa2d6f0fb2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:34.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f967e276700 1 -- 172.21.15.187:0/841916189 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967000e070 con 0x55b299a223a0 2020-11-20T12:11:34.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f967e276700 1 -- 172.21.15.187:0/841916189 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9670006b30 con 0x55b299a223a0 2020-11-20T12:11:34.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f967e276700 1 -- 172.21.15.187:0/841916189 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670013420 con 0x55b299a223a0 2020-11-20T12:11:34.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f968e9d3d00 1 -- 172.21.15.187:0/841916189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a223a0 msgr2=0x55b299aeade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f968e9d3d00 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a223a0 0x55b299aeade0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f9670009b20 tx=0x7f9670005bc0).stop 2020-11-20T12:11:34.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f968e9d3d00 1 -- 172.21.15.187:0/841916189 shutdown_connections 2020-11-20T12:11:34.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f968e9d3d00 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a63ad0 0x55b299ae8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f968e9d3d00 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a223a0 0x55b299aeade0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f968e9d3d00 1 --2- 172.21.15.187:0/841916189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 0x55b299a484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.307+0000 7f968e9d3d00 1 -- 172.21.15.187:0/841916189 >> 172.21.15.187:0/841916189 conn(0x55b299a210e0 msgr2=0x55b299aea500 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:34.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.308+0000 7f968e9d3d00 1 -- 172.21.15.187:0/841916189 shutdown_connections 2020-11-20T12:11:34.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.308+0000 7f968e9d3d00 1 -- 172.21.15.187:0/841916189 wait complete. 2020-11-20T12:11:34.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.308+0000 7f968e9d3d00 1 Processor -- start 2020-11-20T12:11:34.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.309+0000 7f968e9d3d00 1 -- start start 2020-11-20T12:11:34.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.309+0000 7f968e9d3d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a223a0 0x55b299a49020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.309+0000 7f968e9d3d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 0x55b299b00860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.309+0000 7f968e9d3d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a63ad0 0x55b299ae5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.309+0000 7f968e9d3d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b299a5c420 con 0x55b299a49d40 2020-11-20T12:11:34.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.309+0000 7f968e9d3d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b299b049d0 con 0x55b299a63ad0 2020-11-20T12:11:34.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.309+0000 7f968e9d3d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b299a5deb0 con 0x55b299a223a0 2020-11-20T12:11:34.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967fa79700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a63ad0 0x55b299ae5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967fa79700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a63ad0 0x55b299ae5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46278/0 (socket says 172.21.15.187:46278) 2020-11-20T12:11:34.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967fa79700 1 -- 172.21.15.187:0/3687073866 learned_addr learned my addr 172.21.15.187:0/3687073866 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:34.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f9676a77700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 0x55b299b00860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967f278700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a223a0 0x55b299a49020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967fa79700 1 -- 172.21.15.187:0/3687073866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a223a0 msgr2=0x55b299a49020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967fa79700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55b299a223a0 0x55b299a49020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967fa79700 1 -- 172.21.15.187:0/3687073866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 msgr2=0x55b299b00860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967fa79700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 0x55b299b00860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f967fa79700 1 -- 172.21.15.187:0/3687073866 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2999c74e0 con 0x55b299a63ad0 2020-11-20T12:11:34.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.310+0000 7f9676a77700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55b299a49d40 0x55b299b00860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:34.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.311+0000 7f967fa79700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a63ad0 0x55b299ae5ac0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f967800f000 tx=0x7f9678014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:34.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.311+0000 7f967ca73700 1 -- 172.21.15.187:0/3687073866 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678010e70 con 0x55b299a63ad0 2020-11-20T12:11:34.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.311+0000 7f967ca73700 1 -- 172.21.15.187:0/3687073866 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9678005670 con 0x55b299a63ad0 2020-11-20T12:11:34.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.311+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b2999c2ec0 con 0x55b299a63ad0 2020-11-20T12:11:34.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.311+0000 7f967ca73700 1 -- 172.21.15.187:0/3687073866 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967801d890 con 0x55b299a63ad0 2020-11-20T12:11:34.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.311+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b2999a1250 con 0x55b299a63ad0 2020-11-20T12:11:34.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.313+0000 7f967ca73700 1 -- 172.21.15.187:0/3687073866 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f967800e070 con 0x55b299a63ad0 2020-11-20T12:11:34.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.314+0000 7f967ca73700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96600559a0 0x7f9660057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.314+0000 7f967ca73700 1 -- 172.21.15.187:0/3687073866 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9678071390 con 0x55b299a63ad0 2020-11-20T12:11:34.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.314+0000 7f967f278700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96600559a0 0x7f9660057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.314+0000 7f968e9d3d00 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55b299a661e0 0x55b299a5cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.314+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:1 14.7 14:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected e72) v8 -- 0x55b299ae8ff0 con 0x55b299a661e0 2020-11-20T12:11:34.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.314+0000 7f9676a77700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55b299a661e0 0x55b299a5cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.315+0000 7f967f278700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96600559a0 0x7f9660057e60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f9670005e50 tx=0x7f9670014000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:34.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.315+0000 7f9676a77700 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55b299a661e0 0x55b299a5cd60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:34.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f9676a77700 1 -- 172.21.15.187:0/3687073866 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv7 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f966800d890 con 0x55b299a661e0 2020-11-20T12:11:34.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55b299a661e0 msgr2=0x55b299a5cd60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x55b299a661e0 0x55b299a5cd60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96600559a0 msgr2=0x7f9660057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96600559a0 0x7f9660057e60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f9670005e50 tx=0x7f9670014000).stop 2020-11-20T12:11:34.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a63ad0 msgr2=0x55b299ae5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 --2- 172.21.15.187:0/3687073866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b299a63ad0 0x55b299ae5ac0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f967800f000 tx=0x7f9678014000).stop 2020-11-20T12:11:34.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f967fa79700 1 -- 172.21.15.187:0/3687073866 reap_dead start 2020-11-20T12:11:34.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 shutdown_connections 2020-11-20T12:11:34.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 >> 172.21.15.187:0/3687073866 conn(0x55b299a210e0 msgr2=0x55b299a66710 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:34.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 shutdown_connections 2020-11-20T12:11:34.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.316+0000 7f968e9d3d00 1 -- 172.21.15.187:0/3687073866 wait complete. 2020-11-20T12:11:34.336 INFO:teuthology.orchestra.run.smithi187.stdout:W�file_ddir_2, 2020-11-20T12:11:34.336 INFO:teuthology.orchestra.run.smithi187:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2020-11-20T12:11:34.410 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:11:34.410 INFO:teuthology.orchestra.run.smithi187.stdout: "ino": 1099511627777, 2020-11-20T12:11:34.410 INFO:teuthology.orchestra.run.smithi187.stdout: "ancestors": [ 2020-11-20T12:11:34.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:11:34.411 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1099511628278, 2020-11-20T12:11:34.411 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "file_d", 2020-11-20T12:11:34.411 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 27 2020-11-20T12:11:34.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:11:34.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:11:34.412 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1, 2020-11-20T12:11:34.412 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "dir_2", 2020-11-20T12:11:34.412 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 44 2020-11-20T12:11:34.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:11:34.413 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2020-11-20T12:11:34.413 INFO:teuthology.orchestra.run.smithi187.stdout: "pool": 14, 2020-11-20T12:11:34.413 INFO:teuthology.orchestra.run.smithi187.stdout: "old_pools": [] 2020-11-20T12:11:34.413 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:11:34.414 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:34.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:11:34.415 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -f dir_2/file_d) 2020-11-20T12:11:34.525 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:34.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.763+0000 7fbd4359e700 1 Processor -- start 2020-11-20T12:11:34.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.763+0000 7fbd4359e700 1 -- start start 2020-11-20T12:11:34.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.765+0000 7fbd4359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbd440b4a20 0x7fbd4414ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.765+0000 7fbd4359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd44065ad0 0x7fbd44148830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.765+0000 7fbd4359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4414be00 con 0x7fbd440b4e00 2020-11-20T12:11:34.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.765+0000 7fbd4359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4414c030 con 0x7fbd440b4a20 2020-11-20T12:11:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.765+0000 7fbd4359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4414c2d0 con 0x7fbd44065ad0 2020-11-20T12:11:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.765+0000 7fbd42d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd44065ad0 0x7fbd44148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.766+0000 7fbd42d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd44065ad0 0x7fbd44148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:47174/0 (socket says 172.21.15.187:47174) 2020-11-20T12:11:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.766+0000 7fbd42d9d700 1 -- 172.21.15.187:0/2938412541 learned_addr learned my addr 172.21.15.187:0/2938412541 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:34.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.766+0000 7fbd4259c700 1 --2- 172.21.15.187:0/2938412541 >> v2:172.21.15.5:3300/0 conn(0x7fbd440b4a20 0x7fbd4414ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.767+0000 7fbd41d9b700 1 -- 172.21.15.187:0/2938412541 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609056670 0 0) 0x7fbd4414be00 con 0x7fbd440b4e00 2020-11-20T12:11:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.767+0000 7fbd41d9b700 1 -- 172.21.15.187:0/2938412541 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd24003240 con 0x7fbd440b4e00 2020-11-20T12:11:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.767+0000 7fbd4259c700 1 -- 172.21.15.187:0/2938412541 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd44065ad0 msgr2=0x7fbd44148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.767+0000 7fbd4259c700 1 --2- 172.21.15.187:0/2938412541 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd44065ad0 0x7fbd44148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.767+0000 7fbd4259c700 1 -- 172.21.15.187:0/2938412541 >> v1:172.21.15.5:6789/0 conn(0x7fbd440b4e00 legacy=0x7fbd44145a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.767+0000 7fbd4259c700 1 -- 172.21.15.187:0/2938412541 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4414c570 con 0x7fbd440b4a20 2020-11-20T12:11:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.767+0000 7fbd4259c700 1 --2- 172.21.15.187:0/2938412541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd440b4a20 0x7fbd4414ca10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fbd34009c90 tx=0x7fbd340091d0).ready entity=mon.0 client_cookie=b70fcdd200e7c45 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd41d9b700 1 -- 172.21.15.187:0/2938412541 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3400b9c0 con 0x7fbd440b4a20 2020-11-20T12:11:34.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd41d9b700 1 -- 172.21.15.187:0/2938412541 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3400bb20 con 0x7fbd440b4a20 2020-11-20T12:11:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd41d9b700 1 -- 172.21.15.187:0/2938412541 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3400bea0 con 0x7fbd440b4a20 2020-11-20T12:11:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd4359e700 1 -- 172.21.15.187:0/2938412541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd440b4a20 msgr2=0x7fbd4414ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd4359e700 1 --2- 172.21.15.187:0/2938412541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd440b4a20 0x7fbd4414ca10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fbd34009c90 tx=0x7fbd340091d0).stop 2020-11-20T12:11:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd4359e700 1 -- 172.21.15.187:0/2938412541 shutdown_connections 2020-11-20T12:11:34.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd4359e700 1 --2- 172.21.15.187:0/2938412541 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd44065ad0 0x7fbd44148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd4359e700 1 --2- 172.21.15.187:0/2938412541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd440b4a20 0x7fbd4414ca10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.768+0000 7fbd4359e700 1 -- 172.21.15.187:0/2938412541 >> 172.21.15.187:0/2938412541 conn(0x7fbd440bb3d0 msgr2=0x7fbd440bb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:34.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.769+0000 7fbd4359e700 1 -- 172.21.15.187:0/2938412541 shutdown_connections 2020-11-20T12:11:34.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.769+0000 7fbd4359e700 1 -- 172.21.15.187:0/2938412541 wait complete. 2020-11-20T12:11:34.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.770+0000 7fbd4359e700 1 Processor -- start 2020-11-20T12:11:34.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.771+0000 7fbd4359e700 1 -- start start 2020-11-20T12:11:34.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd4359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd44065ad0 0x7fbd4409eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd4359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd440b4a20 0x7fbd44147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd4359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd440b4e00 0x7fbd44116ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd4359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd44116520 con 0x7fbd44065ad0 2020-11-20T12:11:34.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd4359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd44116780 con 0x7fbd440b4a20 2020-11-20T12:11:34.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd4359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd44116a20 con 0x7fbd440b4e00 2020-11-20T12:11:34.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd39d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd440b4a20 0x7fbd44147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd39d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd440b4a20 0x7fbd44147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46292/0 (socket says 172.21.15.187:46292) 2020-11-20T12:11:34.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd39d9b700 1 -- 172.21.15.187:0/1635974515 learned_addr learned my addr 172.21.15.187:0/1635974515 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:34.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd42d9d700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd440b4e00 0x7fbd44116ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.772+0000 7fbd4259c700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd44065ad0 0x7fbd4409eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd39d9b700 1 -- 172.21.15.187:0/1635974515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd440b4e00 msgr2=0x7fbd44116ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd39d9b700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd440b4e00 0x7fbd44116ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd39d9b700 1 -- 172.21.15.187:0/1635974515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd44065ad0 msgr2=0x7fbd4409eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:34.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd39d9b700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd44065ad0 0x7fbd4409eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:34.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd39d9b700 1 -- 172.21.15.187:0/1635974515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4409f3c0 con 0x7fbd440b4a20 2020-11-20T12:11:34.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd39d9b700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd440b4a20 0x7fbd44147b00 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7fbd2c00c710 tx=0x7fbd2c007ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:34.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd3affd700 1 -- 172.21.15.187:0/1635974515 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c00e560 con 0x7fbd440b4a20 2020-11-20T12:11:34.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd3affd700 1 -- 172.21.15.187:0/1635974515 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c00e6c0 con 0x7fbd440b4a20 2020-11-20T12:11:34.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd3affd700 1 -- 172.21.15.187:0/1635974515 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c00ea60 con 0x7fbd440b4a20 2020-11-20T12:11:34.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd44147ff0 con 0x7fbd440b4a20 2020-11-20T12:11:34.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.773+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd44117570 con 0x7fbd440b4a20 2020-11-20T12:11:34.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.775+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd44116be0 con 0x7fbd440b4a20 2020-11-20T12:11:34.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.775+0000 7fbd3affd700 1 -- 172.21.15.187:0/1635974515 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbd2c019030 con 0x7fbd440b4a20 2020-11-20T12:11:34.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.776+0000 7fbd3affd700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd20056d50 0x7fbd20059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:34.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.776+0000 7fbd3affd700 1 -- 172.21.15.187:0/1635974515 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbd2c06f550 con 0x7fbd440b4a20 2020-11-20T12:11:34.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.776+0000 7fbd4259c700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd20056d50 0x7fbd20059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:34.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.777+0000 7fbd4259c700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd20056d50 0x7fbd20059210 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fbd440baff0 tx=0x7fbd34016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:34.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:34.781+0000 7fbd3affd700 1 -- 172.21.15.187:0/1635974515 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbd2c0448f0 con 0x7fbd440b4a20 2020-11-20T12:11:35.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.026+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd440b8fd0 con 0x7fbd440b4a20 2020-11-20T12:11:35.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.028+0000 7fbd3affd700 1 -- 172.21.15.187:0/1635974515 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fbd2c047f10 con 0x7fbd440b4a20 2020-11-20T12:11:35.032 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:35.032 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:35.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd20056d50 msgr2=0x7fbd20059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:35.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd20056d50 0x7fbd20059210 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fbd440baff0 tx=0x7fbd34016000).stop 2020-11-20T12:11:35.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd440b4a20 msgr2=0x7fbd44147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:35.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd440b4a20 0x7fbd44147b00 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7fbd2c00c710 tx=0x7fbd2c007ae0).stop 2020-11-20T12:11:35.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 shutdown_connections 2020-11-20T12:11:35.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbd20056d50 0x7fbd20059210 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:35.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd44065ad0 0x7fbd4409eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:35.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd440b4e00 0x7fbd44116ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:35.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 --2- 172.21.15.187:0/1635974515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd440b4a20 0x7fbd44147b00 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:35.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.030+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 >> 172.21.15.187:0/1635974515 conn(0x7fbd440bb3d0 msgr2=0x7fbd440bb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:35.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.031+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 shutdown_connections 2020-11-20T12:11:35.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:35.031+0000 7fbd4359e700 1 -- 172.21.15.187:0/1635974515 wait complete. 2020-11-20T12:11:35.037 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:35.056 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:11:35.353 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:11:35.353 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:11:35.354 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2020-11-20T12:11:40.355 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:40.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.608+0000 7f78a6816700 1 Processor -- start 2020-11-20T12:11:40.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.609+0000 7f78a6816700 1 -- start start 2020-11-20T12:11:40.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.610+0000 7f78a6816700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f78a00b57b0 0x7f78a0068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:40.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.610+0000 7f78a6816700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f78a00b5f50 0x7f78a00b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:40.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.610+0000 7f78a6816700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a0144b80 con 0x7f78a01482d0 2020-11-20T12:11:40.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.610+0000 7f78a6816700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a01489e0 con 0x7f78a00b57b0 2020-11-20T12:11:40.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.610+0000 7f78a6816700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a0148c80 con 0x7f78a00b5f50 2020-11-20T12:11:40.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.610+0000 7f78a5013700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f78a01482d0 0x7f78a0062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42904/0 (socket says 172.21.15.187:42904) 2020-11-20T12:11:40.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.610+0000 7f78a5013700 1 -- 172.21.15.187:0/3299357983 learned_addr learned my addr 172.21.15.187:0/3299357983 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:40.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.610+0000 7f78a6015700 1 --2- 172.21.15.187:0/3299357983 >> v2:172.21.15.187:3300/0 conn(0x7f78a00b5f50 0x7f78a00b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:40.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.611+0000 7f78a5814700 1 --2- 172.21.15.187:0/3299357983 >> v2:172.21.15.5:3300/0 conn(0x7f78a00b57b0 0x7f78a0068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:40.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.611+0000 7f78a4812700 1 -- 172.21.15.187:0/3299357983 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711395451 0 0) 0x7f78a0144b80 con 0x7f78a01482d0 2020-11-20T12:11:40.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.611+0000 7f78a4812700 1 -- 172.21.15.187:0/3299357983 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7888003240 con 0x7f78a01482d0 2020-11-20T12:11:40.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.611+0000 7f78a6015700 1 -- 172.21.15.187:0/3299357983 >> v2:172.21.15.5:3300/0 conn(0x7f78a00b57b0 msgr2=0x7f78a0068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:40.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.611+0000 7f78a6015700 1 --2- 172.21.15.187:0/3299357983 >> v2:172.21.15.5:3300/0 conn(0x7f78a00b57b0 0x7f78a0068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.611+0000 7f78a6015700 1 -- 172.21.15.187:0/3299357983 >> v1:172.21.15.187:6789/0 conn(0x7f78a01482d0 legacy=0x7f78a0062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:40.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.611+0000 7f78a6015700 1 -- 172.21.15.187:0/3299357983 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0148f20 con 0x7f78a00b5f50 2020-11-20T12:11:40.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.611+0000 7f78a6015700 1 --2- 172.21.15.187:0/3299357983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b5f50 0x7f78a00b6380 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f78a0066230 tx=0x7f7898006df0).ready entity=mon.1 client_cookie=b568bdb90960d3b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:40.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a4812700 1 -- 172.21.15.187:0/3299357983 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980097b0 con 0x7f78a00b5f50 2020-11-20T12:11:40.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a4812700 1 -- 172.21.15.187:0/3299357983 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898009910 con 0x7f78a00b5f50 2020-11-20T12:11:40.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a4812700 1 -- 172.21.15.187:0/3299357983 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898009c90 con 0x7f78a00b5f50 2020-11-20T12:11:40.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a6816700 1 -- 172.21.15.187:0/3299357983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b5f50 msgr2=0x7f78a00b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:40.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a6816700 1 --2- 172.21.15.187:0/3299357983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b5f50 0x7f78a00b6380 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f78a0066230 tx=0x7f7898006df0).stop 2020-11-20T12:11:40.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a6816700 1 -- 172.21.15.187:0/3299357983 shutdown_connections 2020-11-20T12:11:40.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a6816700 1 --2- 172.21.15.187:0/3299357983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b5f50 0x7f78a00b6380 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a6816700 1 --2- 172.21.15.187:0/3299357983 >> v2:172.21.15.5:3300/0 conn(0x7f78a00b57b0 0x7f78a0068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a6816700 1 -- 172.21.15.187:0/3299357983 >> 172.21.15.187:0/3299357983 conn(0x7f78a00bab30 msgr2=0x7f78a00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:40.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a6816700 1 -- 172.21.15.187:0/3299357983 shutdown_connections 2020-11-20T12:11:40.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.612+0000 7f78a6816700 1 -- 172.21.15.187:0/3299357983 wait complete. 2020-11-20T12:11:40.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.614+0000 7f78a6816700 1 Processor -- start 2020-11-20T12:11:40.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.614+0000 7f78a6816700 1 -- start start 2020-11-20T12:11:40.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a6816700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b57b0 0x7f78a008fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:40.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a6816700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f78a00b5f50 0x7f78a00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:40.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a6816700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f78a01482d0 0x7f78a00b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:40.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a6816700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a008f280 con 0x7f78a01482d0 2020-11-20T12:11:40.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a6816700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a008f520 con 0x7f78a00b57b0 2020-11-20T12:11:40.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a6816700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a008f7c0 con 0x7f78a00b5f50 2020-11-20T12:11:40.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a5814700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b57b0 0x7f78a008fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:40.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a5814700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b57b0 0x7f78a008fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46304/0 (socket says 172.21.15.187:46304) 2020-11-20T12:11:40.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.615+0000 7f78a5814700 1 -- 172.21.15.187:0/2631562415 learned_addr learned my addr 172.21.15.187:0/2631562415 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:40.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a6015700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f78a01482d0 0x7f78a00b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:40.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a5814700 1 -- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f78a00b5f50 msgr2=0x7f78a00b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:40.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a5013700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f78a00b5f50 0x7f78a00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:40.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a5814700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f78a00b5f50 0x7f78a00b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a5814700 1 -- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f78a01482d0 msgr2=0x7f78a00b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:40.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a5814700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f78a01482d0 0x7f78a00b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a5814700 1 -- 172.21.15.187:0/2631562415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a009d430 con 0x7f78a00b57b0 2020-11-20T12:11:40.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a6015700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f78a01482d0 0x7f78a00b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:40.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a5814700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b57b0 0x7f78a008fc60 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f789c00ec90 tx=0x7f789c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:40.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f788dffb700 1 -- 172.21.15.187:0/2631562415 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c00c940 con 0x7f78a00b57b0 2020-11-20T12:11:40.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f788dffb700 1 -- 172.21.15.187:0/2631562415 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789c00caa0 con 0x7f78a00b57b0 2020-11-20T12:11:40.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a00b3bc0 con 0x7f78a00b57b0 2020-11-20T12:11:40.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f788dffb700 1 -- 172.21.15.187:0/2631562415 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c00ce20 con 0x7f78a00b57b0 2020-11-20T12:11:40.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.616+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a008f020 con 0x7f78a00b57b0 2020-11-20T12:11:40.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.618+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a0061a20 con 0x7f78a00b57b0 2020-11-20T12:11:40.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.618+0000 7f788dffb700 1 -- 172.21.15.187:0/2631562415 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f789c013640 con 0x7f78a00b57b0 2020-11-20T12:11:40.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.620+0000 7f788dffb700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7878056d20 0x7f78780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:40.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.620+0000 7f78a5013700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7878056d20 0x7f78780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:40.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.620+0000 7f788dffb700 1 -- 172.21.15.187:0/2631562415 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f789c06fd00 con 0x7f78a00b57b0 2020-11-20T12:11:40.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.621+0000 7f78a5013700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7878056d20 0x7f78780591e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f78900055a0 tx=0x7f7890009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:40.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.630+0000 7f788dffb700 1 -- 172.21.15.187:0/2631562415 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f789c073030 con 0x7f78a00b57b0 2020-11-20T12:11:40.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.878+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78a0147da0 con 0x7f78a00b57b0 2020-11-20T12:11:40.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.880+0000 7f788dffb700 1 -- 172.21.15.187:0/2631562415 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f789c045100 con 0x7f78a00b57b0 2020-11-20T12:11:40.884 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:40.884 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:40.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7878056d20 msgr2=0x7f78780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:40.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7878056d20 0x7f78780591e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f78900055a0 tx=0x7f7890009040).stop 2020-11-20T12:11:40.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b57b0 msgr2=0x7f78a008fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:40.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b57b0 0x7f78a008fc60 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f789c00ec90 tx=0x7f789c008040).stop 2020-11-20T12:11:40.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 shutdown_connections 2020-11-20T12:11:40.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7878056d20 0x7f78780591e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f78a01482d0 0x7f78a00b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78a00b57b0 0x7f78a008fc60 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 --2- 172.21.15.187:0/2631562415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f78a00b5f50 0x7f78a00b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:40.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 >> 172.21.15.187:0/2631562415 conn(0x7f78a00bab30 msgr2=0x7f78a00b9270 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:40.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 shutdown_connections 2020-11-20T12:11:40.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:40.882+0000 7f78a6816700 1 -- 172.21.15.187:0/2631562415 wait complete. 2020-11-20T12:11:40.890 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:40.911 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:11:41.203 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:11:41.203 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:11:41.244 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2020-11-20T12:11:46.245 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:46.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.489+0000 7f8b2d661700 1 Processor -- start 2020-11-20T12:11:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.489+0000 7f8b2d661700 1 -- start start 2020-11-20T12:11:46.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.490+0000 7f8b2d661700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280bf800 0x7f8b2814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.490+0000 7f8b2d661700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8b280bfbe0 0x7f8b281480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.490+0000 7f8b2d661700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2814d110 con 0x7f8b2809f560 2020-11-20T12:11:46.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.490+0000 7f8b2d661700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b2814cbd0 con 0x7f8b280bfbe0 2020-11-20T12:11:46.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.490+0000 7f8b2d661700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2814ce70 con 0x7f8b280bf800 2020-11-20T12:11:46.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.491+0000 7f8b2ce60700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8b2809f560 0x7f8b28060c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42922/0 (socket says 172.21.15.187:42922) 2020-11-20T12:11:46.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.491+0000 7f8b2ce60700 1 -- 172.21.15.187:0/2262046495 learned_addr learned my addr 172.21.15.187:0/2262046495 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:46.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.491+0000 7f8b1f7fe700 1 --2- 172.21.15.187:0/2262046495 >> v2:172.21.15.5:3300/0 conn(0x7f8b280bfbe0 0x7f8b281480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:46.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.491+0000 7f8b27fff700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280bf800 0x7f8b2814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:46.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579839466 0 0) 0x7f8b2814d110 con 0x7f8b2809f560 2020-11-20T12:11:46.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b08003240 con 0x7f8b2809f560 2020-11-20T12:11:46.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 290348800 0 0) 0x7f8b08003240 con 0x7f8b2809f560 2020-11-20T12:11:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280bf800 msgr2=0x7f8b2814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:46.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280bf800 0x7f8b2814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 >> v2:172.21.15.5:3300/0 conn(0x7f8b280bfbe0 msgr2=0x7f8b281480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 --2- 172.21.15.187:0/2262046495 >> v2:172.21.15.5:3300/0 conn(0x7f8b280bfbe0 0x7f8b281480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b280a4cc0 con 0x7f8b2809f560 2020-11-20T12:11:46.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8b20001860 con 0x7f8b2809f560 2020-11-20T12:11:46.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 >> v1:172.21.15.187:6789/0 conn(0x7f8b2809f560 legacy=0x7f8b28060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:46.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280bf800 0x7f8b08003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b08003c40 0x7f8b080044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b080049d0 0x7f8b08004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2814cbd0 con 0x7f8b080049d0 2020-11-20T12:11:46.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2814ce70 con 0x7f8b08003c40 2020-11-20T12:11:46.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.492+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08004200 con 0x7f8b280bf800 2020-11-20T12:11:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.493+0000 7f8b1f7fe700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b08003c40 0x7f8b080044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.493+0000 7f8b27fff700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280bf800 0x7f8b08003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.493+0000 7f8b1f7fe700 1 -- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280bf800 msgr2=0x7f8b08003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.493+0000 7f8b1f7fe700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280bf800 0x7f8b08003730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.493+0000 7f8b1f7fe700 1 -- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b080049d0 msgr2=0x7f8b08004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.493+0000 7f8b1f7fe700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b080049d0 0x7f8b08004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.493+0000 7f8b1f7fe700 1 -- 172.21.15.187:0/2262046495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b08003240 con 0x7f8b08003c40 2020-11-20T12:11:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.494+0000 7f8b1f7fe700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b08003c40 0x7f8b080044e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f8b280bcaa0 tx=0x7f8b10006df0).ready entity=mon.1 client_cookie=678250479ac17d96 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.494+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1000b310 con 0x7f8b08003c40 2020-11-20T12:11:46.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.494+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1000eae0 con 0x7f8b08003c40 2020-11-20T12:11:46.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.494+0000 7f8b277fe700 1 -- 172.21.15.187:0/2262046495 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1000ee60 con 0x7f8b08003c40 2020-11-20T12:11:46.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.495+0000 7f8b2d661700 1 -- 172.21.15.187:0/2262046495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b08003c40 msgr2=0x7f8b080044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:46.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.495+0000 7f8b2d661700 1 --2- 172.21.15.187:0/2262046495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b08003c40 0x7f8b080044e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f8b280bcaa0 tx=0x7f8b10006df0).stop 2020-11-20T12:11:46.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.495+0000 7f8b2ce60700 1 -- 172.21.15.187:0/2262046495 reap_dead start 2020-11-20T12:11:46.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.495+0000 7f8b2d661700 1 -- 172.21.15.187:0/2262046495 shutdown_connections 2020-11-20T12:11:46.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.495+0000 7f8b2d661700 1 -- 172.21.15.187:0/2262046495 >> 172.21.15.187:0/2262046495 conn(0x7f8b28068370 msgr2=0x7f8b280bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:46.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.495+0000 7f8b2d661700 1 -- 172.21.15.187:0/2262046495 shutdown_connections 2020-11-20T12:11:46.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.495+0000 7f8b2d661700 1 -- 172.21.15.187:0/2262046495 wait complete. 2020-11-20T12:11:46.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.497+0000 7f8b2d661700 1 Processor -- start 2020-11-20T12:11:46.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.497+0000 7f8b2d661700 1 -- start start 2020-11-20T12:11:46.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b2d661700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280b6160 0x7f8b2815bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b2d661700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b280b4320 0x7f8b2815c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b2d661700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b280b6540 0x7f8b2815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b2d661700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2815cec0 con 0x7f8b280b6540 2020-11-20T12:11:46.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b2d661700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b280b6aa0 con 0x7f8b280b4320 2020-11-20T12:11:46.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b2d661700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b280b6d40 con 0x7f8b280b6160 2020-11-20T12:11:46.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b1f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b280b4320 0x7f8b2815c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:46.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b1f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b280b4320 0x7f8b2815c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46326/0 (socket says 172.21.15.187:46326) 2020-11-20T12:11:46.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b1f7fe700 1 -- 172.21.15.187:0/3023237475 learned_addr learned my addr 172.21.15.187:0/3023237475 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:46.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b27fff700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280b6160 0x7f8b2815bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:46.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.498+0000 7f8b2ce60700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b280b6540 0x7f8b2815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:46.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b1f7fe700 1 -- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280b6160 msgr2=0x7f8b2815bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:46.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b1f7fe700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280b6160 0x7f8b2815bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b1f7fe700 1 -- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b280b6540 msgr2=0x7f8b2815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:46.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b1f7fe700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b280b6540 0x7f8b2815c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b1f7fe700 1 -- 172.21.15.187:0/3023237475 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b28158970 con 0x7f8b280b4320 2020-11-20T12:11:46.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b2ce60700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b280b6540 0x7f8b2815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:46.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b1f7fe700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b280b4320 0x7f8b2815c4e0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f8b10006e30 tx=0x7f8b10017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:46.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b24ff9700 1 -- 172.21.15.187:0/3023237475 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1001d620 con 0x7f8b280b4320 2020-11-20T12:11:46.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b24ff9700 1 -- 172.21.15.187:0/3023237475 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1001d7c0 con 0x7f8b280b4320 2020-11-20T12:11:46.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b24ff9700 1 -- 172.21.15.187:0/3023237475 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1001db80 con 0x7f8b280b4320 2020-11-20T12:11:46.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b28146b00 con 0x7f8b280b4320 2020-11-20T12:11:46.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.499+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b28147020 con 0x7f8b280b4320 2020-11-20T12:11:46.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.500+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b28080050 con 0x7f8b280b4320 2020-11-20T12:11:46.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.501+0000 7f8b24ff9700 1 -- 172.21.15.187:0/3023237475 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8b1000b800 con 0x7f8b280b4320 2020-11-20T12:11:46.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.502+0000 7f8b24ff9700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b04056d20 0x7f8b040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:46.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.502+0000 7f8b24ff9700 1 -- 172.21.15.187:0/3023237475 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8b100720f0 con 0x7f8b280b4320 2020-11-20T12:11:46.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.502+0000 7f8b27fff700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b04056d20 0x7f8b040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:46.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.503+0000 7f8b27fff700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b04056d20 0x7f8b040591e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8b18006bd0 tx=0x7f8b18008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:46.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.505+0000 7f8b24ff9700 1 -- 172.21.15.187:0/3023237475 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8b100473c0 con 0x7f8b280b4320 2020-11-20T12:11:46.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.745+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b280b6f00 con 0x7f8b280b4320 2020-11-20T12:11:46.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.746+0000 7f8b24ff9700 1 -- 172.21.15.187:0/3023237475 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f8b1002a070 con 0x7f8b280b4320 2020-11-20T12:11:46.750 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:46.751 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:46.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b04056d20 msgr2=0x7f8b040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:46.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b04056d20 0x7f8b040591e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8b18006bd0 tx=0x7f8b18008040).stop 2020-11-20T12:11:46.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b280b4320 msgr2=0x7f8b2815c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:46.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b280b4320 0x7f8b2815c4e0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f8b10006e30 tx=0x7f8b10017000).stop 2020-11-20T12:11:46.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 shutdown_connections 2020-11-20T12:11:46.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b04056d20 0x7f8b040591e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b280b6540 0x7f8b2815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b280b6160 0x7f8b2815bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 --2- 172.21.15.187:0/3023237475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b280b4320 0x7f8b2815c4e0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:46.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 >> 172.21.15.187:0/3023237475 conn(0x7f8b28068370 msgr2=0x7f8b280be3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:46.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.748+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 shutdown_connections 2020-11-20T12:11:46.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:46.749+0000 7f8b2d661700 1 -- 172.21.15.187:0/3023237475 wait complete. 2020-11-20T12:11:46.755 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:46.775 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:11:47.069 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_enqueued":1}} 2020-11-20T12:11:47.070 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2020-11-20T12:11:47.070 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:47.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.318+0000 7f13dffff700 1 Processor -- start 2020-11-20T12:11:47.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.318+0000 7f13dffff700 1 -- start start 2020-11-20T12:11:47.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.319+0000 7f13dffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13e00bf800 0x7f13e014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:47.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.319+0000 7f13dffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bfbe0 0x7f13e01480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:47.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.319+0000 7f13dffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e014d110 con 0x7f13e009f560 2020-11-20T12:11:47.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.319+0000 7f13dffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e014cbd0 con 0x7f13e00bf800 2020-11-20T12:11:47.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.319+0000 7f13dffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e014ce70 con 0x7f13e00bfbe0 2020-11-20T12:11:47.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.320+0000 7f13deffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13e00bf800 0x7f13e014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:47.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.320+0000 7f13deffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13e00bf800 0x7f13e014c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46332/0 (socket says 172.21.15.187:46332) 2020-11-20T12:11:47.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.320+0000 7f13deffd700 1 -- 172.21.15.187:0/2402312067 learned_addr learned my addr 172.21.15.187:0/2402312067 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:47.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.320+0000 7f13d67fc700 1 --2- 172.21.15.187:0/2402312067 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bfbe0 0x7f13e01480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:47.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.320+0000 7f13deffd700 1 -- 172.21.15.187:0/2402312067 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bfbe0 msgr2=0x7f13e01480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:47.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.320+0000 7f13deffd700 1 --2- 172.21.15.187:0/2402312067 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bfbe0 0x7f13e01480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.321+0000 7f13deffd700 1 -- 172.21.15.187:0/2402312067 >> v1:172.21.15.5:6789/0 conn(0x7f13e009f560 legacy=0x7f13e0060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:47.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.321+0000 7f13deffd700 1 -- 172.21.15.187:0/2402312067 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00a4cc0 con 0x7f13e00bf800 2020-11-20T12:11:47.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.321+0000 7f13d67fc700 1 --2- 172.21.15.187:0/2402312067 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bfbe0 0x7f13e01480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:47.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.321+0000 7f13deffd700 1 --2- 172.21.15.187:0/2402312067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e00bf800 0x7f13e014c2c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f13d0006fd0 tx=0x7f13d0006eb0).ready entity=mon.1 client_cookie=bd3ea1acc0d91251 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:47.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.321+0000 7f13de7fc700 1 -- 172.21.15.187:0/2402312067 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0009920 con 0x7f13e00bf800 2020-11-20T12:11:47.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.321+0000 7f13de7fc700 1 -- 172.21.15.187:0/2402312067 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0009a80 con 0x7f13e00bf800 2020-11-20T12:11:47.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.321+0000 7f13de7fc700 1 -- 172.21.15.187:0/2402312067 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0009e00 con 0x7f13e00bf800 2020-11-20T12:11:47.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.322+0000 7f13dffff700 1 -- 172.21.15.187:0/2402312067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e00bf800 msgr2=0x7f13e014c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:47.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.322+0000 7f13dffff700 1 --2- 172.21.15.187:0/2402312067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e00bf800 0x7f13e014c2c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f13d0006fd0 tx=0x7f13d0006eb0).stop 2020-11-20T12:11:47.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.322+0000 7f13dffff700 1 -- 172.21.15.187:0/2402312067 shutdown_connections 2020-11-20T12:11:47.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.322+0000 7f13dffff700 1 --2- 172.21.15.187:0/2402312067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e00bf800 0x7f13e014c2c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.322+0000 7f13dffff700 1 --2- 172.21.15.187:0/2402312067 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bfbe0 0x7f13e01480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.322+0000 7f13dffff700 1 -- 172.21.15.187:0/2402312067 >> 172.21.15.187:0/2402312067 conn(0x7f13e0068370 msgr2=0x7f13e00bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:47.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.322+0000 7f13dffff700 1 -- 172.21.15.187:0/2402312067 shutdown_connections 2020-11-20T12:11:47.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.322+0000 7f13dffff700 1 -- 172.21.15.187:0/2402312067 wait complete. 2020-11-20T12:11:47.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.324+0000 7f13dffff700 1 Processor -- start 2020-11-20T12:11:47.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.324+0000 7f13dffff700 1 -- start start 2020-11-20T12:11:47.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.325+0000 7f13dffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e009f560 0x7f13e0147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:47.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.325+0000 7f13dffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bf800 0x7f13e0133540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:47.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.325+0000 7f13dffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e01327a0 0x7f13e0132c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:47.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.325+0000 7f13dffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e014a3c0 con 0x7f13e01327a0 2020-11-20T12:11:47.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.325+0000 7f13dffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e014a660 con 0x7f13e009f560 2020-11-20T12:11:47.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.325+0000 7f13dffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e014a900 con 0x7f13e00bf800 2020-11-20T12:11:47.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13deffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e009f560 0x7f13e0147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:47.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13deffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e009f560 0x7f13e0147700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46338/0 (socket says 172.21.15.187:46338) 2020-11-20T12:11:47.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13deffd700 1 -- 172.21.15.187:0/3734917007 learned_addr learned my addr 172.21.15.187:0/3734917007 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:47.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13d67fc700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bf800 0x7f13e0133540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:47.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13df7fe700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e01327a0 0x7f13e0132c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:47.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13deffd700 1 -- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bf800 msgr2=0x7f13e0133540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:47.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13deffd700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bf800 0x7f13e0133540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13deffd700 1 -- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e01327a0 msgr2=0x7f13e0132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:47.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13deffd700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e01327a0 0x7f13e0132c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13deffd700 1 -- 172.21.15.187:0/3734917007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e014ada0 con 0x7f13e009f560 2020-11-20T12:11:47.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.326+0000 7f13df7fe700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e01327a0 0x7f13e0132c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:47.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.327+0000 7f13deffd700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e009f560 0x7f13e0147700 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f13e00bca50 tx=0x7f13d0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:47.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.327+0000 7f13d77fe700 1 -- 172.21.15.187:0/3734917007 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d001d5d0 con 0x7f13e009f560 2020-11-20T12:11:47.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.327+0000 7f13d77fe700 1 -- 172.21.15.187:0/3734917007 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d001d730 con 0x7f13e009f560 2020-11-20T12:11:47.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.327+0000 7f13d77fe700 1 -- 172.21.15.187:0/3734917007 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d001dad0 con 0x7f13e009f560 2020-11-20T12:11:47.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.327+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e014a0b0 con 0x7f13e009f560 2020-11-20T12:11:47.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.327+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e0133a30 con 0x7f13e009f560 2020-11-20T12:11:47.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.328+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e0080050 con 0x7f13e009f560 2020-11-20T12:11:47.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.329+0000 7f13d77fe700 1 -- 172.21.15.187:0/3734917007 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f13d0025030 con 0x7f13e009f560 2020-11-20T12:11:47.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.329+0000 7f13d77fe700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13bc056d20 0x7f13bc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:47.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.329+0000 7f13d77fe700 1 -- 172.21.15.187:0/3734917007 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f13d0071a90 con 0x7f13e009f560 2020-11-20T12:11:47.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.330+0000 7f13d67fc700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13bc056d20 0x7f13bc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:47.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.330+0000 7f13d67fc700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13bc056d20 0x7f13bc0591e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f13c8006fd0 tx=0x7f13c8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:47.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.335+0000 7f13d77fe700 1 -- 172.21.15.187:0/3734917007 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f13d0046e30 con 0x7f13e009f560 2020-11-20T12:11:47.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.564+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13e014c7b0 con 0x7f13e009f560 2020-11-20T12:11:47.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.566+0000 7f13d77fe700 1 -- 172.21.15.187:0/3734917007 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f13d0023020 con 0x7f13e009f560 2020-11-20T12:11:47.569 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:47.569 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:47.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13bc056d20 msgr2=0x7f13bc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:47.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13bc056d20 0x7f13bc0591e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f13c8006fd0 tx=0x7f13c8008040).stop 2020-11-20T12:11:47.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e009f560 msgr2=0x7f13e0147700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:47.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e009f560 0x7f13e0147700 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f13e00bca50 tx=0x7f13d0015000).stop 2020-11-20T12:11:47.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 shutdown_connections 2020-11-20T12:11:47.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13bc056d20 0x7f13bc0591e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e01327a0 0x7f13e0132c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e009f560 0x7f13e0147700 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 --2- 172.21.15.187:0/3734917007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13e00bf800 0x7f13e0133540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:47.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 >> 172.21.15.187:0/3734917007 conn(0x7f13e0068370 msgr2=0x7f13e0141f80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:47.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 shutdown_connections 2020-11-20T12:11:47.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:47.568+0000 7f13dffff700 1 -- 172.21.15.187:0/3734917007 wait complete. 2020-11-20T12:11:47.574 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:47.592 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump purge_queue pq_executed 2020-11-20T12:11:47.872 INFO:teuthology.orchestra.run.smithi005.stdout:{"purge_queue":{"pq_executed":1}} 2020-11-20T12:11:47.873 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2020-11-20T12:11:47.874 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:48.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.148+0000 7f364986d700 1 Processor -- start 2020-11-20T12:11:48.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.149+0000 7f364986d700 1 -- start start 2020-11-20T12:11:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.150+0000 7f364986d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f36440b1380 0x7f3644150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.150+0000 7f364986d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f36440b1760 0x7f3644065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.150+0000 7f364986d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3644150390 con 0x7f3644065680 2020-11-20T12:11:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.150+0000 7f364986d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3644150630 con 0x7f36440b1380 2020-11-20T12:11:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.150+0000 7f364986d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3644068510 con 0x7f36440b1760 2020-11-20T12:11:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.151+0000 7f363bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f36440b1760 0x7f3644065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.151+0000 7f363bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f36440b1760 0x7f3644065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46350/0 (socket says 172.21.15.187:46350) 2020-11-20T12:11:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.151+0000 7f363bfff700 1 -- 172.21.15.187:0/1826197581 learned_addr learned my addr 172.21.15.187:0/1826197581 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.151+0000 7f364886b700 1 --2- 172.21.15.187:0/1826197581 >> v2:172.21.15.5:3300/0 conn(0x7f36440b1380 0x7f3644150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.151+0000 7f363bfff700 1 -- 172.21.15.187:0/1826197581 >> v2:172.21.15.5:3300/0 conn(0x7f36440b1380 msgr2=0x7f3644150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.151+0000 7f363bfff700 1 --2- 172.21.15.187:0/1826197581 >> v2:172.21.15.5:3300/0 conn(0x7f36440b1380 0x7f3644150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.151+0000 7f363bfff700 1 -- 172.21.15.187:0/1826197581 >> v1:172.21.15.5:6789/0 conn(0x7f3644065680 legacy=0x7f3644061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.151+0000 7f363bfff700 1 -- 172.21.15.187:0/1826197581 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3644150fc0 con 0x7f36440b1760 2020-11-20T12:11:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.152+0000 7f363bfff700 1 --2- 172.21.15.187:0/1826197581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b1760 0x7f3644065190 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f3644063a20 tx=0x7f3634006eb0).ready entity=mon.1 client_cookie=5d048501335f9cf2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.152+0000 7f363b7fe700 1 -- 172.21.15.187:0/1826197581 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36340098d0 con 0x7f36440b1760 2020-11-20T12:11:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.152+0000 7f363b7fe700 1 -- 172.21.15.187:0/1826197581 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3634009a30 con 0x7f36440b1760 2020-11-20T12:11:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.152+0000 7f363b7fe700 1 -- 172.21.15.187:0/1826197581 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634009db0 con 0x7f36440b1760 2020-11-20T12:11:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.153+0000 7f364986d700 1 -- 172.21.15.187:0/1826197581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b1760 msgr2=0x7f3644065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.153+0000 7f364986d700 1 --2- 172.21.15.187:0/1826197581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b1760 0x7f3644065190 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f3644063a20 tx=0x7f3634006eb0).stop 2020-11-20T12:11:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.153+0000 7f364986d700 1 -- 172.21.15.187:0/1826197581 shutdown_connections 2020-11-20T12:11:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.153+0000 7f364986d700 1 --2- 172.21.15.187:0/1826197581 >> v2:172.21.15.5:3300/0 conn(0x7f36440b1380 0x7f3644150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.153+0000 7f364986d700 1 --2- 172.21.15.187:0/1826197581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b1760 0x7f3644065190 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.153+0000 7f364986d700 1 -- 172.21.15.187:0/1826197581 >> 172.21.15.187:0/1826197581 conn(0x7f36440bab30 msgr2=0x7f36440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.153+0000 7f364986d700 1 -- 172.21.15.187:0/1826197581 shutdown_connections 2020-11-20T12:11:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.153+0000 7f364986d700 1 -- 172.21.15.187:0/1826197581 wait complete. 2020-11-20T12:11:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.155+0000 7f364986d700 1 Processor -- start 2020-11-20T12:11:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.155+0000 7f364986d700 1 -- start start 2020-11-20T12:11:48.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.156+0000 7f364986d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36440b1380 0x7f364409e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.156+0000 7f364986d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36440b1760 0x7f364409c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.156+0000 7f364986d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b7bf0 0x7f3644149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.156+0000 7f364986d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36440b50e0 con 0x7f36440b1760 2020-11-20T12:11:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.156+0000 7f364986d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36440b8380 con 0x7f36440b7bf0 2020-11-20T12:11:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.156+0000 7f364986d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36440b5310 con 0x7f36440b1380 2020-11-20T12:11:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364906c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b7bf0 0x7f3644149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364906c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b7bf0 0x7f3644149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46354/0 (socket says 172.21.15.187:46354) 2020-11-20T12:11:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364906c700 1 -- 172.21.15.187:0/2778968470 learned_addr learned my addr 172.21.15.187:0/2778968470 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f363bfff700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36440b1760 0x7f364409c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364886b700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36440b1380 0x7f364409e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364906c700 1 -- 172.21.15.187:0/2778968470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36440b1380 msgr2=0x7f364409e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364906c700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36440b1380 0x7f364409e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364906c700 1 -- 172.21.15.187:0/2778968470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36440b1760 msgr2=0x7f364409c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364906c700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36440b1760 0x7f364409c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.157+0000 7f364906c700 1 -- 172.21.15.187:0/2778968470 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36440b8070 con 0x7f36440b7bf0 2020-11-20T12:11:48.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.158+0000 7f364906c700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b7bf0 0x7f3644149a20 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f363c00ca60 tx=0x7f363c007f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.158+0000 7f3638ff9700 1 -- 172.21.15.187:0/2778968470 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c00a830 con 0x7f36440b7bf0 2020-11-20T12:11:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.158+0000 7f3638ff9700 1 -- 172.21.15.187:0/2778968470 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363c00a990 con 0x7f36440b7bf0 2020-11-20T12:11:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.158+0000 7f3638ff9700 1 -- 172.21.15.187:0/2778968470 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c00ad10 con 0x7f36440b7bf0 2020-11-20T12:11:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.158+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364409ce70 con 0x7f36440b7bf0 2020-11-20T12:11:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.158+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364414a150 con 0x7f36440b7bf0 2020-11-20T12:11:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.160+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36440b54d0 con 0x7f36440b7bf0 2020-11-20T12:11:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.160+0000 7f3638ff9700 1 -- 172.21.15.187:0/2778968470 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f363c019030 con 0x7f36440b7bf0 2020-11-20T12:11:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.161+0000 7f3638ff9700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f361c056d50 0x7f361c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.161+0000 7f3638ff9700 1 -- 172.21.15.187:0/2778968470 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f363c06f7b0 con 0x7f36440b7bf0 2020-11-20T12:11:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.161+0000 7f364886b700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f361c056d50 0x7f361c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.162+0000 7f364886b700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f361c056d50 0x7f361c059210 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3640009650 tx=0x7f3640008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:48.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.165+0000 7f3638ff9700 1 -- 172.21.15.187:0/2778968470 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f363c043fd0 con 0x7f36440b7bf0 2020-11-20T12:11:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.417+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3644068680 con 0x7f36440b7bf0 2020-11-20T12:11:48.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.418+0000 7f3638ff9700 1 -- 172.21.15.187:0/2778968470 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f363c044c60 con 0x7f36440b7bf0 2020-11-20T12:11:48.422 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:48.422 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:48.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f361c056d50 msgr2=0x7f361c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f361c056d50 0x7f361c059210 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3640009650 tx=0x7f3640008040).stop 2020-11-20T12:11:48.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b7bf0 msgr2=0x7f3644149a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b7bf0 0x7f3644149a20 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f363c00ca60 tx=0x7f363c007f40).stop 2020-11-20T12:11:48.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 shutdown_connections 2020-11-20T12:11:48.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f361c056d50 0x7f361c059210 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b7bf0 0x7f3644149a20 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36440b1380 0x7f364409e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 --2- 172.21.15.187:0/2778968470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36440b1760 0x7f364409c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 >> 172.21.15.187:0/2778968470 conn(0x7f36440bab30 msgr2=0x7f364414f300 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:48.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 shutdown_connections 2020-11-20T12:11:48.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.421+0000 7f364986d700 1 -- 172.21.15.187:0/2778968470 wait complete. 2020-11-20T12:11:48.428 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:48.449 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache 2020-11-20T12:11:48.748 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":3,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2020-11-20T12:11:48.749 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 3 } } 2020-11-20T12:11:48.750 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:49.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.999+0000 7feb558d3700 1 Processor -- start 2020-11-20T12:11:49.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:48.999+0000 7feb558d3700 1 -- start start 2020-11-20T12:11:49.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.001+0000 7feb558d3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feb50065280 0x7feb5014f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.001+0000 7feb558d3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500afcc0 con 0x7feb50065de0 2020-11-20T12:11:49.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.001+0000 7feb558d3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500b00b0 con 0x7feb50065660 2020-11-20T12:11:49.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.001+0000 7feb558d3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb500b0350 con 0x7feb50065280 2020-11-20T12:11:49.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.001+0000 7feb548d1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feb50065280 0x7feb5014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.001+0000 7feb4ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7feb50065660 0x7feb50067840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:42968/0 (socket says 172.21.15.187:42968) 2020-11-20T12:11:49.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.001+0000 7feb4ffff700 1 -- 172.21.15.187:0/674233346 learned_addr learned my addr 172.21.15.187:0/674233346 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:49.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.002+0000 7feb4f7fe700 1 -- 172.21.15.187:0/674233346 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651298266 0 0) 0x7feb500b00b0 con 0x7feb50065660 2020-11-20T12:11:49.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.002+0000 7feb4f7fe700 1 -- 172.21.15.187:0/674233346 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb34003240 con 0x7feb50065660 2020-11-20T12:11:49.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.002+0000 7feb548d1700 1 -- 172.21.15.187:0/674233346 >> v1:172.21.15.187:6789/0 conn(0x7feb50065660 legacy=0x7feb50067840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.002+0000 7feb548d1700 1 -- 172.21.15.187:0/674233346 >> v1:172.21.15.5:6789/0 conn(0x7feb50065de0 legacy=0x7feb50157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:49.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.002+0000 7feb548d1700 1 -- 172.21.15.187:0/674233346 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500b5620 con 0x7feb50065280 2020-11-20T12:11:49.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.002+0000 7feb548d1700 1 --2- 172.21.15.187:0/674233346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065280 0x7feb5014f750 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7feb4400b990 tx=0x7feb44006d90).ready entity=mon.1 client_cookie=f249e1f8580145ff server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.002+0000 7feb4f7fe700 1 -- 172.21.15.187:0/674233346 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44006030 con 0x7feb50065280 2020-11-20T12:11:49.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb4f7fe700 1 -- 172.21.15.187:0/674233346 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb44006190 con 0x7feb50065280 2020-11-20T12:11:49.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb4f7fe700 1 -- 172.21.15.187:0/674233346 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44006510 con 0x7feb50065280 2020-11-20T12:11:49.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb558d3700 1 -- 172.21.15.187:0/674233346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065280 msgr2=0x7feb5014f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb558d3700 1 --2- 172.21.15.187:0/674233346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065280 0x7feb5014f750 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7feb4400b990 tx=0x7feb44006d90).stop 2020-11-20T12:11:49.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb558d3700 1 -- 172.21.15.187:0/674233346 shutdown_connections 2020-11-20T12:11:49.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb558d3700 1 --2- 172.21.15.187:0/674233346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065280 0x7feb5014f750 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb558d3700 1 -- 172.21.15.187:0/674233346 >> 172.21.15.187:0/674233346 conn(0x7feb500baac0 msgr2=0x7feb500baed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:49.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb558d3700 1 -- 172.21.15.187:0/674233346 shutdown_connections 2020-11-20T12:11:49.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.003+0000 7feb558d3700 1 -- 172.21.15.187:0/674233346 wait complete. 2020-11-20T12:11:49.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.005+0000 7feb558d3700 1 Processor -- start 2020-11-20T12:11:49.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.005+0000 7feb558d3700 1 -- start start 2020-11-20T12:11:49.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb558d3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb50065280 0x7feb500620d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb558d3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065660 0x7feb5015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb558d3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb50065de0 0x7feb5015ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb558d3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5015d1f0 con 0x7feb50065de0 2020-11-20T12:11:49.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb558d3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5015ccb0 con 0x7feb50065660 2020-11-20T12:11:49.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb558d3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5015cf50 con 0x7feb50065280 2020-11-20T12:11:49.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb4ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065660 0x7feb5015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb4ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065660 0x7feb5015c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46368/0 (socket says 172.21.15.187:46368) 2020-11-20T12:11:49.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.006+0000 7feb4ffff700 1 -- 172.21.15.187:0/4044484806 learned_addr learned my addr 172.21.15.187:0/4044484806 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:49.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb550d2700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb50065de0 0x7feb5015ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb548d1700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb50065280 0x7feb500620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4ffff700 1 -- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb50065280 msgr2=0x7feb500620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4ffff700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb50065280 0x7feb500620d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4ffff700 1 -- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb50065de0 msgr2=0x7feb5015ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4ffff700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb50065de0 0x7feb5015ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4ffff700 1 -- 172.21.15.187:0/4044484806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5015fa20 con 0x7feb50065660 2020-11-20T12:11:49.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb550d2700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb50065de0 0x7feb5015ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:49.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4ffff700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065660 0x7feb5015c330 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7feb3c00c6a0 tx=0x7feb3c007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4cff9700 1 -- 172.21.15.187:0/4044484806 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0072e0 con 0x7feb50065660 2020-11-20T12:11:49.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4cff9700 1 -- 172.21.15.187:0/4044484806 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c007440 con 0x7feb50065660 2020-11-20T12:11:49.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb4cff9700 1 -- 172.21.15.187:0/4044484806 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c005820 con 0x7feb50065660 2020-11-20T12:11:49.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.007+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb500625c0 con 0x7feb50065660 2020-11-20T12:11:49.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.008+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5015c9e0 con 0x7feb50065660 2020-11-20T12:11:49.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.009+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb500636b0 con 0x7feb50065660 2020-11-20T12:11:49.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.009+0000 7feb4cff9700 1 -- 172.21.15.187:0/4044484806 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feb3c010070 con 0x7feb50065660 2020-11-20T12:11:49.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.010+0000 7feb4cff9700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb28056d20 0x7feb280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.010+0000 7feb4cff9700 1 -- 172.21.15.187:0/4044484806 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7feb3c06f6f0 con 0x7feb50065660 2020-11-20T12:11:49.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.011+0000 7feb548d1700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb28056d20 0x7feb280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.011+0000 7feb548d1700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb28056d20 0x7feb280591e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7feb50150bd0 tx=0x7feb44015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.014+0000 7feb4cff9700 1 -- 172.21.15.187:0/4044484806 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feb3c044af0 con 0x7feb50065660 2020-11-20T12:11:49.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.262+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb500b07f0 con 0x7feb50065660 2020-11-20T12:11:49.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.263+0000 7feb4cff9700 1 -- 172.21.15.187:0/4044484806 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7feb3c018870 con 0x7feb50065660 2020-11-20T12:11:49.266 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:49.267 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:49.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.265+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb28056d20 msgr2=0x7feb280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.265+0000 7feb558d3700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb28056d20 0x7feb280591e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7feb50150bd0 tx=0x7feb44015000).stop 2020-11-20T12:11:49.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.265+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065660 msgr2=0x7feb5015c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.265+0000 7feb558d3700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065660 0x7feb5015c330 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7feb3c00c6a0 tx=0x7feb3c007ac0).stop 2020-11-20T12:11:49.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.266+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 shutdown_connections 2020-11-20T12:11:49.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.266+0000 7feb558d3700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feb28056d20 0x7feb280591e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.266+0000 7feb558d3700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb50065de0 0x7feb5015ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.266+0000 7feb558d3700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb50065280 0x7feb500620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.266+0000 7feb558d3700 1 --2- 172.21.15.187:0/4044484806 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb50065660 0x7feb5015c330 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.266+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 >> 172.21.15.187:0/4044484806 conn(0x7feb500baac0 msgr2=0x7feb5009fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:49.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.266+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 shutdown_connections 2020-11-20T12:11:49.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.266+0000 7feb558d3700 1 -- 172.21.15.187:0/4044484806 wait complete. 2020-11-20T12:11:49.272 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:49.293 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump purge_queue 2020-11-20T12:11:49.577 INFO:teuthology.orchestra.run.smithi005.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed":1,"pq_item_in_journal":0}} 2020-11-20T12:11:49.582 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2020-11-20T12:11:49.583 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:11:49.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.676+0000 7f016ccc5d00 1 Processor -- start 2020-11-20T12:11:49.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.677+0000 7f016ccc5d00 1 -- start start 2020-11-20T12:11:49.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.677+0000 7f016ccc5d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed10ce0 0x55a92ecd2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.677+0000 7f016ccc5d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 0x55a92ecf1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.677+0000 7f016ccc5d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9e3b0 0x55a92ed0bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.677+0000 7f016ccc5d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a92ecd11e0 con 0x55a92ed9e3b0 2020-11-20T12:11:49.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.677+0000 7f016ccc5d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a92ecd1500 con 0x55a92ed10ce0 2020-11-20T12:11:49.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.677+0000 7f016ccc5d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a92ecd17a0 con 0x55a92ed113d0 2020-11-20T12:11:49.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015d56a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed10ce0 0x55a92ecd2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015d56a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed10ce0 0x55a92ecd2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46380/0 (socket says 172.21.15.187:46380) 2020-11-20T12:11:49.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015d56a700 1 -- 172.21.15.187:0/3194381174 learned_addr learned my addr 172.21.15.187:0/3194381174 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:49.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015cd69700 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 0x55a92ecf1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015dd6b700 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9e3b0 0x55a92ed0bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015d56a700 1 -- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 msgr2=0x55a92ecf1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:49.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015d56a700 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 0x55a92ecf1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015d56a700 1 -- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9e3b0 msgr2=0x55a92ed0bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:49.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015d56a700 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9e3b0 0x55a92ed0bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015d56a700 1 -- 172.21.15.187:0/3194381174 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a92ec730c0 con 0x55a92ed10ce0 2020-11-20T12:11:49.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015dd6b700 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9e3b0 0x55a92ed0bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:49.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.678+0000 7f015cd69700 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 0x55a92ecf1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:49.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.679+0000 7f015d56a700 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed10ce0 0x55a92ecd2170 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f0150009b20 tx=0x7f0150005bc0).ready entity=mon.1 client_cookie=cdc2830e0350002 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.679+0000 7f014f7fe700 1 -- 172.21.15.187:0/3194381174 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015000e070 con 0x55a92ed10ce0 2020-11-20T12:11:49.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.679+0000 7f014f7fe700 1 -- 172.21.15.187:0/3194381174 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0150006bc0 con 0x55a92ed10ce0 2020-11-20T12:11:49.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.679+0000 7f014f7fe700 1 -- 172.21.15.187:0/3194381174 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150013420 con 0x55a92ed10ce0 2020-11-20T12:11:49.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.679+0000 7f016ccc5d00 1 -- 172.21.15.187:0/3194381174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed10ce0 msgr2=0x55a92ecd2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.679+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed10ce0 0x55a92ecd2170 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f0150009b20 tx=0x7f0150005bc0).stop 2020-11-20T12:11:49.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.680+0000 7f016ccc5d00 1 -- 172.21.15.187:0/3194381174 shutdown_connections 2020-11-20T12:11:49.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.680+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9e3b0 0x55a92ed0bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.680+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed10ce0 0x55a92ecd2170 secure :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0x7f0150009b20 tx=0x7f0150005bc0).stop 2020-11-20T12:11:49.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.680+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/3194381174 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 0x55a92ecf1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.680+0000 7f016ccc5d00 1 -- 172.21.15.187:0/3194381174 >> 172.21.15.187:0/3194381174 conn(0x55a92eccdee0 msgr2=0x55a92ecd02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:49.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.680+0000 7f016ccc5d00 1 -- 172.21.15.187:0/3194381174 shutdown_connections 2020-11-20T12:11:49.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.680+0000 7f016ccc5d00 1 -- 172.21.15.187:0/3194381174 wait complete. 2020-11-20T12:11:49.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.681+0000 7f016ccc5d00 1 Processor -- start 2020-11-20T12:11:49.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.681+0000 7f016ccc5d00 1 -- start start 2020-11-20T12:11:49.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f016ccc5d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 0x55a92edb0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f016ccc5d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed9e3b0 0x55a92edb1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f016ccc5d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9c080 0x55a92ed99ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f016ccc5d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a92edb16b0 con 0x55a92ed9c080 2020-11-20T12:11:49.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f016ccc5d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a92ed0c170 con 0x55a92ed9e3b0 2020-11-20T12:11:49.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f016ccc5d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a92ed0c610 con 0x55a92ed113d0 2020-11-20T12:11:49.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f015cd69700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed9e3b0 0x55a92edb1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f015cd69700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed9e3b0 0x55a92edb1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46388/0 (socket says 172.21.15.187:46388) 2020-11-20T12:11:49.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f015cd69700 1 -- 172.21.15.187:0/368502212 learned_addr learned my addr 172.21.15.187:0/368502212 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:49.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f015dd6b700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9c080 0x55a92ed99ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.682+0000 7f015cd69700 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 msgr2=0x55a92edb0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f015cd69700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55a92ed113d0 0x55a92edb0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:49.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f015cd69700 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9c080 msgr2=0x55a92ed99ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f015cd69700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9c080 0x55a92ed99ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f015cd69700 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a92ec744e0 con 0x55a92ed9e3b0 2020-11-20T12:11:49.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f015dd6b700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55a92ed9c080 0x55a92ed99ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:49.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f015cd69700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed9e3b0 0x55a92edb1080 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f015400ba90 tx=0x7f0154005c80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f014dffb700 1 -- 172.21.15.187:0/368502212 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154010070 con 0x55a92ed9e3b0 2020-11-20T12:11:49.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f014dffb700 1 -- 172.21.15.187:0/368502212 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154007a20 con 0x55a92ed9e3b0 2020-11-20T12:11:49.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f014dffb700 1 -- 172.21.15.187:0/368502212 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154007da0 con 0x55a92ed9e3b0 2020-11-20T12:11:49.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a92ec6fec0 con 0x55a92ed9e3b0 2020-11-20T12:11:49.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.683+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a92ec4e250 con 0x55a92ed9e3b0 2020-11-20T12:11:49.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.685+0000 7f014dffb700 1 -- 172.21.15.187:0/368502212 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0154015630 con 0x55a92ed9e3b0 2020-11-20T12:11:49.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.686+0000 7f014dffb700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f01400559d0 0x7f0140057e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.686+0000 7f014dffb700 1 -- 172.21.15.187:0/368502212 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f015406f430 con 0x55a92ed9e3b0 2020-11-20T12:11:49.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.686+0000 7f015d56a700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f01400559d0 0x7f0140057e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.686+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a92ed9a1d0 0x55a92ed9a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.686+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:1 14.0 14:00000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x55a92ed9aaa0 con 0x55a92ed9a1d0 2020-11-20T12:11:49.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.687+0000 7f015d56a700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f01400559d0 0x7f0140057e90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x55a92ecf3470 tx=0x7f0150014000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.687+0000 7f015dd6b700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a92ed9a1d0 0x55a92ed9a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.688+0000 7f015dd6b700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a92ed9a1d0 0x55a92ed9a5b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.689+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== osd_op_reply(1 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0158008a80 con 0x55a92ed9a1d0 2020-11-20T12:11:49.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.689+0000 7f015dd6b700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f015801c040 0x7f0158010b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.689+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:2 14.4 14:20000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x55a92ed9aaa0 con 0x7f015801c040 2020-11-20T12:11:49.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.690+0000 7f015d56a700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f015801c040 0x7f0158010b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.691+0000 7f015d56a700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f015801c040 0x7f0158010b00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.691+0000 7f015d56a700 1 -- 172.21.15.187:0/368502212 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(2 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0150006d30 con 0x7f015801c040 2020-11-20T12:11:49.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.691+0000 7f015d56a700 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:3 14.2 14:40000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x55a92ed9aaa0 con 0x7f015801c040 2020-11-20T12:11:49.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.692+0000 7f015d56a700 1 -- 172.21.15.187:0/368502212 <== osd.2 v2:172.21.15.187:6810/19904 2 ==== osd_op_reply(3 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0150006d30 con 0x7f015801c040 2020-11-20T12:11:49.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.692+0000 7f015d56a700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x55a92ed10ce0 0x7f0150028940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.692+0000 7f015d56a700 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:4 14.6 14:60000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x55a92ed9aaa0 con 0x55a92ed10ce0 2020-11-20T12:11:49.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.693+0000 7f015cd69700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x55a92ed10ce0 0x7f0150028940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.694+0000 7f015cd69700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x55a92ed10ce0 0x7f0150028940 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.695+0000 7f015cd69700 1 -- 172.21.15.187:0/368502212 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== osd_op_reply(4 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0154046780 con 0x55a92ed10ce0 2020-11-20T12:11:49.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.695+0000 7f015cd69700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f015401d350 0x7f01540158c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.695+0000 7f015cd69700 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:5 14.1 14:80000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x55a92ed9aaa0 con 0x7f015401d350 2020-11-20T12:11:49.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.696+0000 7f015dd6b700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f015401d350 0x7f01540158c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.698+0000 7f015dd6b700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f015401d350 0x7f01540158c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.699+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(5 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0158008a80 con 0x7f015401d350 2020-11-20T12:11:49.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.699+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:6 14.5 14:a0000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x55a92ed9aaa0 con 0x55a92ed10ce0 2020-11-20T12:11:49.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.700+0000 7f015cd69700 1 -- 172.21.15.187:0/368502212 <== osd.5 v2:172.21.15.5:6809/20155 2 ==== osd_op_reply(6 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0154046780 con 0x55a92ed10ce0 2020-11-20T12:11:49.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.700+0000 7f015cd69700 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:7 14.3 14:c0000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x55a92ed9aaa0 con 0x55a92ed9a1d0 2020-11-20T12:11:49.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.702+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 <== osd.4 v2:172.21.15.5:6808/20156 2 ==== osd_op_reply(7 [pgnls start_epoch 72 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0158008a80 con 0x55a92ed9a1d0 2020-11-20T12:11:49.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.702+0000 7f015dd6b700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f01580106a0 0x7f01580198a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.702+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:8 14.7 14:e0000000::::head [pgnls start_epoch 72 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e72) v8 -- 0x55a92ed9aaa0 con 0x7f01580106a0 2020-11-20T12:11:49.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.702+0000 7f015d56a700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f01580106a0 0x7f01580198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f015d56a700 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f01580106a0 0x7f01580198a0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f015d56a700 1 -- 172.21.15.187:0/368502212 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(8 [pgnls start_epoch 72 out=49b] v72'9 uv9 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0150006d30 con 0x7f01580106a0 2020-11-20T12:11:49.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f01580106a0 msgr2=0x7f01580198a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f01580106a0 0x7f01580198a0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f015801c040 msgr2=0x7f0158010b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f015801c040 0x7f0158010b00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a92ed9a1d0 msgr2=0x55a92ed9a5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55a92ed9a1d0 0x55a92ed9a5b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x55a92ed10ce0 msgr2=0x7f0150028940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x55a92ed10ce0 0x7f0150028940 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f015401d350 msgr2=0x7f01540158c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f015401d350 0x7f01540158c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.703+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 reap_dead start 2020-11-20T12:11:49.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 reap_dead start 2020-11-20T12:11:49.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f01400559d0 msgr2=0x7f0140057e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f01400559d0 0x7f0140057e90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x55a92ecf3470 tx=0x7f0150014000).stop 2020-11-20T12:11:49.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f015dd6b700 1 -- 172.21.15.187:0/368502212 reap_dead start 2020-11-20T12:11:49.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed9e3b0 msgr2=0x55a92edb1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed9e3b0 0x55a92edb1080 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f015400ba90 tx=0x7f0154005c80).stop 2020-11-20T12:11:49.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 shutdown_connections 2020-11-20T12:11:49.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a92ed9e3b0 0x55a92edb1080 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 --2- 172.21.15.187:0/368502212 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f01400559d0 0x7f0140057e90 secure :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0x55a92ecf3470 tx=0x7f0150014000).stop 2020-11-20T12:11:49.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 >> 172.21.15.187:0/368502212 conn(0x55a92eccdee0 msgr2=0x55a92ecd02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:49.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 shutdown_connections 2020-11-20T12:11:49.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.704+0000 7f016ccc5d00 1 -- 172.21.15.187:0/368502212 wait complete. 2020-11-20T12:11:49.722 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2020-11-20T12:11:49.722 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:49.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.967+0000 7f1c7ea3f700 1 Processor -- start 2020-11-20T12:11:49.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.967+0000 7f1c7ea3f700 1 -- start start 2020-11-20T12:11:49.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.968+0000 7f1c7ea3f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1c780bf800 0x7f1c7814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.968+0000 7f1c7ea3f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bfbe0 0x7f1c781480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.969+0000 7f1c7ea3f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7814d110 con 0x7f1c7809f560 2020-11-20T12:11:49.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.969+0000 7f1c7ea3f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7814cbd0 con 0x7f1c780bf800 2020-11-20T12:11:49.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.969+0000 7f1c7ea3f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7814ce70 con 0x7f1c780bfbe0 2020-11-20T12:11:49.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.969+0000 7f1c7da3d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1c780bf800 0x7f1c7814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.969+0000 7f1c7da3d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1c780bf800 0x7f1c7814c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46404/0 (socket says 172.21.15.187:46404) 2020-11-20T12:11:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.969+0000 7f1c7da3d700 1 -- 172.21.15.187:0/1022118680 learned_addr learned my addr 172.21.15.187:0/1022118680 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.969+0000 7f1c7d23c700 1 --2- 172.21.15.187:0/1022118680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bfbe0 0x7f1c781480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7da3d700 1 -- 172.21.15.187:0/1022118680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bfbe0 msgr2=0x7f1c781480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:49.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7da3d700 1 --2- 172.21.15.187:0/1022118680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bfbe0 0x7f1c781480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7da3d700 1 -- 172.21.15.187:0/1022118680 >> v1:172.21.15.5:6789/0 conn(0x7f1c7809f560 legacy=0x7f1c78060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:49.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7da3d700 1 -- 172.21.15.187:0/1022118680 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c780a4cc0 con 0x7f1c780bf800 2020-11-20T12:11:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7d23c700 1 --2- 172.21.15.187:0/1022118680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bfbe0 0x7f1c781480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7da3d700 1 --2- 172.21.15.187:0/1022118680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c780bf800 0x7f1c7814c2c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f1c74006fd0 tx=0x7f1c74006eb0).ready entity=mon.1 client_cookie=37ea55571368eccd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7ca3b700 1 -- 172.21.15.187:0/1022118680 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74009920 con 0x7f1c780bf800 2020-11-20T12:11:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7ca3b700 1 -- 172.21.15.187:0/1022118680 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c74009a80 con 0x7f1c780bf800 2020-11-20T12:11:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.970+0000 7f1c7ca3b700 1 -- 172.21.15.187:0/1022118680 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74009e00 con 0x7f1c780bf800 2020-11-20T12:11:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.971+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1022118680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c780bf800 msgr2=0x7f1c7814c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.971+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1022118680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c780bf800 0x7f1c7814c2c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f1c74006fd0 tx=0x7f1c74006eb0).stop 2020-11-20T12:11:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.971+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1022118680 shutdown_connections 2020-11-20T12:11:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.971+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1022118680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c780bf800 0x7f1c7814c2c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.971+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1022118680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bfbe0 0x7f1c781480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.971+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1022118680 >> 172.21.15.187:0/1022118680 conn(0x7f1c78068370 msgr2=0x7f1c780bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:49.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.971+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1022118680 shutdown_connections 2020-11-20T12:11:49.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.971+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1022118680 wait complete. 2020-11-20T12:11:49.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.972+0000 7f1c7ea3f700 1 Processor -- start 2020-11-20T12:11:49.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.973+0000 7f1c7ea3f700 1 -- start start 2020-11-20T12:11:49.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.973+0000 7f1c7ea3f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7809f560 0x7f1c781413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.973+0000 7f1c7ea3f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bf800 0x7f1c78143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.973+0000 7f1c7ea3f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c78147620 0x7f1c781441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.974+0000 7f1c7ea3f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c78146c40 con 0x7f1c7809f560 2020-11-20T12:11:49.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.974+0000 7f1c7ea3f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c78146ee0 con 0x7f1c78147620 2020-11-20T12:11:49.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.974+0000 7f1c7ea3f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c78147180 con 0x7f1c780bf800 2020-11-20T12:11:49.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.974+0000 7f1c7e23e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c78147620 0x7f1c781441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.974+0000 7f1c7e23e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c78147620 0x7f1c781441a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46410/0 (socket says 172.21.15.187:46410) 2020-11-20T12:11:49.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.974+0000 7f1c7e23e700 1 -- 172.21.15.187:0/1597705203 learned_addr learned my addr 172.21.15.187:0/1597705203 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:49.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.974+0000 7f1c7da3d700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7809f560 0x7f1c781413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.974+0000 7f1c7d23c700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bf800 0x7f1c78143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7e23e700 1 -- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bf800 msgr2=0x7f1c78143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7e23e700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bf800 0x7f1c78143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7e23e700 1 -- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7809f560 msgr2=0x7f1c781413d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7e23e700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7809f560 0x7f1c781413d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7e23e700 1 -- 172.21.15.187:0/1597705203 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c781418c0 con 0x7f1c78147620 2020-11-20T12:11:49.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7da3d700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7809f560 0x7f1c781413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:49.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7e23e700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c78147620 0x7f1c781441a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f1c7000cab0 tx=0x7f1c700023b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c65ffb700 1 -- 172.21.15.187:0/1597705203 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000a830 con 0x7f1c78147620 2020-11-20T12:11:49.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c65ffb700 1 -- 172.21.15.187:0/1597705203 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7000a990 con 0x7f1c78147620 2020-11-20T12:11:49.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c780b41d0 con 0x7f1c78147620 2020-11-20T12:11:49.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c65ffb700 1 -- 172.21.15.187:0/1597705203 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000ad30 con 0x7f1c78147620 2020-11-20T12:11:49.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.975+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c78144850 con 0x7f1c78147620 2020-11-20T12:11:49.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.977+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c78141a50 con 0x7f1c78147620 2020-11-20T12:11:49.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.977+0000 7f1c65ffb700 1 -- 172.21.15.187:0/1597705203 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1c70019030 con 0x7f1c78147620 2020-11-20T12:11:49.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.978+0000 7f1c65ffb700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c50056d50 0x7f1c50059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.978+0000 7f1c65ffb700 1 -- 172.21.15.187:0/1597705203 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1c7006f900 con 0x7f1c78147620 2020-11-20T12:11:49.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.978+0000 7f1c7da3d700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c50056d50 0x7f1c50059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.979+0000 7f1c7da3d700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c50056d50 0x7f1c50059210 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1c74009a50 tx=0x7f1c74015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:49.982+0000 7f1c65ffb700 1 -- 172.21.15.187:0/1597705203 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1c70043fd0 con 0x7f1c78147620 2020-11-20T12:11:50.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.230+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c78080050 con 0x7f1c78147620 2020-11-20T12:11:50.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.231+0000 7f1c65ffb700 1 -- 172.21.15.187:0/1597705203 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f1c70044ca0 con 0x7f1c78147620 2020-11-20T12:11:50.234 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:50.234 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:50.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.233+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c50056d50 msgr2=0x7f1c50059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:50.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.233+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c50056d50 0x7f1c50059210 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1c74009a50 tx=0x7f1c74015000).stop 2020-11-20T12:11:50.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.233+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c78147620 msgr2=0x7f1c781441a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:50.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.233+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c78147620 0x7f1c781441a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f1c7000cab0 tx=0x7f1c700023b0).stop 2020-11-20T12:11:50.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.234+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 shutdown_connections 2020-11-20T12:11:50.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.234+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c50056d50 0x7f1c50059210 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:50.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.234+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7809f560 0x7f1c781413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:50.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.234+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c78147620 0x7f1c781441a0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:50.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.234+0000 7f1c7ea3f700 1 --2- 172.21.15.187:0/1597705203 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c780bf800 0x7f1c78143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:50.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.234+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 >> 172.21.15.187:0/1597705203 conn(0x7f1c78068370 msgr2=0x7f1c780bcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:50.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.234+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 shutdown_connections 2020-11-20T12:11:50.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.234+0000 7f1c7ea3f700 1 -- 172.21.15.187:0/1597705203 wait complete. 2020-11-20T12:11:50.239 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:50.258 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:11:50.549 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_created":3}} 2020-11-20T12:11:50.550 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2020-11-20T12:11:50.550 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:50.659 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:11:50.669 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:11:50.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.798+0000 7ff3d1c25700 1 Processor -- start 2020-11-20T12:11:50.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.799+0000 7ff3d1c25700 1 -- start start 2020-11-20T12:11:50.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.800+0000 7ff3d1c25700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc1484d0 0x7ff3cc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:50.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.800+0000 7ff3d1c25700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff3cc066f70 0x7ff3cc0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:50.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.800+0000 7ff3d1c25700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0b1300 con 0x7ff3cc0622c0 2020-11-20T12:11:50.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.800+0000 7ff3d1c25700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc067730 con 0x7ff3cc066f70 2020-11-20T12:11:50.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.800+0000 7ff3d1c25700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0b15a0 con 0x7ff3cc1484d0 2020-11-20T12:11:50.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.801+0000 7ff3d1424700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff3cc066f70 0x7ff3cc0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:50.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.801+0000 7ff3cbfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff3cc0622c0 0x7ff3cc0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43024/0 (socket says 172.21.15.187:43024) 2020-11-20T12:11:50.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.801+0000 7ff3cbfff700 1 -- 172.21.15.187:0/632927925 learned_addr learned my addr 172.21.15.187:0/632927925 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:50.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.801+0000 7ff3d0c23700 1 --2- 172.21.15.187:0/632927925 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc1484d0 0x7ff3cc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:50.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3d1424700 1 -- 172.21.15.187:0/632927925 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc1484d0 msgr2=0x7ff3cc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:50.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3d1424700 1 --2- 172.21.15.187:0/632927925 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc1484d0 0x7ff3cc061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:50.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3d1424700 1 -- 172.21.15.187:0/632927925 >> v1:172.21.15.187:6789/0 conn(0x7ff3cc0622c0 legacy=0x7ff3cc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:50.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3d1424700 1 -- 172.21.15.187:0/632927925 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0b1840 con 0x7ff3cc066f70 2020-11-20T12:11:50.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3cb7fe700 1 -- 172.21.15.187:0/632927925 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360104589 0 0) 0x7ff3cc0b1300 con 0x7ff3cc0622c0 2020-11-20T12:11:50.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3d1424700 1 --2- 172.21.15.187:0/632927925 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc066f70 0x7ff3cc0b5d50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7ff3c4009c90 tx=0x7ff3c40091d0).ready entity=mon.1 client_cookie=119ce34d5faab09f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:50.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3cb7fe700 1 -- 172.21.15.187:0/632927925 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c400ba70 con 0x7ff3cc066f70 2020-11-20T12:11:50.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3cb7fe700 1 -- 172.21.15.187:0/632927925 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c400bbd0 con 0x7ff3cc066f70 2020-11-20T12:11:50.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.802+0000 7ff3cb7fe700 1 -- 172.21.15.187:0/632927925 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4011400 con 0x7ff3cc066f70 2020-11-20T12:11:50.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.803+0000 7ff3d1c25700 1 -- 172.21.15.187:0/632927925 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc066f70 msgr2=0x7ff3cc0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:50.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.803+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/632927925 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc066f70 0x7ff3cc0b5d50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7ff3c4009c90 tx=0x7ff3c40091d0).stop 2020-11-20T12:11:50.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.803+0000 7ff3d1c25700 1 -- 172.21.15.187:0/632927925 shutdown_connections 2020-11-20T12:11:50.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.803+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/632927925 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc066f70 0x7ff3cc0b5d50 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:50.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.803+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/632927925 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc1484d0 0x7ff3cc061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:50.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.803+0000 7ff3d1c25700 1 -- 172.21.15.187:0/632927925 >> 172.21.15.187:0/632927925 conn(0x7ff3cc0bab30 msgr2=0x7ff3cc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:50.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.803+0000 7ff3d1c25700 1 -- 172.21.15.187:0/632927925 shutdown_connections 2020-11-20T12:11:50.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.803+0000 7ff3d1c25700 1 -- 172.21.15.187:0/632927925 wait complete. 2020-11-20T12:11:50.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.804+0000 7ff3d1c25700 1 Processor -- start 2020-11-20T12:11:50.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.805+0000 7ff3d1c25700 1 -- start start 2020-11-20T12:11:50.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.805+0000 7ff3d1c25700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc0622c0 0x7ff3cc15bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:50.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.805+0000 7ff3d1c25700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3cc066f70 0x7ff3cc141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:50.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.805+0000 7ff3d1c25700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc1484d0 0x7ff3cc14df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:50.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.805+0000 7ff3d1c25700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc15c1e0 con 0x7ff3cc066f70 2020-11-20T12:11:50.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.805+0000 7ff3d1c25700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc15b310 con 0x7ff3cc1484d0 2020-11-20T12:11:50.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.805+0000 7ff3d1c25700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc15b5b0 con 0x7ff3cc0622c0 2020-11-20T12:11:50.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc1484d0 0x7ff3cc14df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:50.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc1484d0 0x7ff3cc14df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46424/0 (socket says 172.21.15.187:46424) 2020-11-20T12:11:50.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 -- 172.21.15.187:0/2495863262 learned_addr learned my addr 172.21.15.187:0/2495863262 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:50.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d0c23700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc0622c0 0x7ff3cc15bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:50.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 -- 172.21.15.187:0/2495863262 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc0622c0 msgr2=0x7ff3cc15bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:50.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc0622c0 0x7ff3cc15bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:50.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 -- 172.21.15.187:0/2495863262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3cc066f70 msgr2=0x7ff3cc141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:50.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3cc066f70 0x7ff3cc141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:50.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 -- 172.21.15.187:0/2495863262 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc15b850 con 0x7ff3cc1484d0 2020-11-20T12:11:50.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.806+0000 7ff3d1424700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc1484d0 0x7ff3cc14df00 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7ff3c4009210 tx=0x7ff3c4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:50.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.807+0000 7ff3c8ff9700 1 -- 172.21.15.187:0/2495863262 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4010780 con 0x7ff3cc1484d0 2020-11-20T12:11:50.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.807+0000 7ff3c8ff9700 1 -- 172.21.15.187:0/2495863262 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c40108e0 con 0x7ff3cc1484d0 2020-11-20T12:11:50.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.807+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc09e700 con 0x7ff3cc1484d0 2020-11-20T12:11:50.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.807+0000 7ff3c8ff9700 1 -- 172.21.15.187:0/2495863262 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4010ca0 con 0x7ff3cc1484d0 2020-11-20T12:11:50.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.807+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc14e3f0 con 0x7ff3cc1484d0 2020-11-20T12:11:50.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.808+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc14e6d0 con 0x7ff3cc1484d0 2020-11-20T12:11:50.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.809+0000 7ff3c8ff9700 1 -- 172.21.15.187:0/2495863262 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff3c4024030 con 0x7ff3cc1484d0 2020-11-20T12:11:50.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.809+0000 7ff3c8ff9700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3a4056d20 0x7ff3a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:50.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.809+0000 7ff3c8ff9700 1 -- 172.21.15.187:0/2495863262 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff3c4071be0 con 0x7ff3cc1484d0 2020-11-20T12:11:50.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.809+0000 7ff3d0c23700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3a4056d20 0x7ff3a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:50.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.810+0000 7ff3d0c23700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3a4056d20 0x7ff3a40591e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7ff3c0006bd0 tx=0x7ff3c0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:50.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:50.813+0000 7ff3c8ff9700 1 -- 172.21.15.187:0/2495863262 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff3c4047000 con 0x7ff3cc1484d0 2020-11-20T12:11:51.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.060+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3cc15b9e0 con 0x7ff3cc1484d0 2020-11-20T12:11:51.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.061+0000 7ff3c8ff9700 1 -- 172.21.15.187:0/2495863262 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7ff3c4022070 con 0x7ff3cc1484d0 2020-11-20T12:11:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:51.066 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:51.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3a4056d20 msgr2=0x7ff3a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:51.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3a4056d20 0x7ff3a40591e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7ff3c0006bd0 tx=0x7ff3c0008040).stop 2020-11-20T12:11:51.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc1484d0 msgr2=0x7ff3cc14df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:51.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc1484d0 0x7ff3cc14df00 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7ff3c4009210 tx=0x7ff3c4016000).stop 2020-11-20T12:11:51.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 shutdown_connections 2020-11-20T12:11:51.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3a4056d20 0x7ff3a40591e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3cc1484d0 0x7ff3cc14df00 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3cc0622c0 0x7ff3cc15bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 --2- 172.21.15.187:0/2495863262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3cc066f70 0x7ff3cc141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:51.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.063+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 >> 172.21.15.187:0/2495863262 conn(0x7ff3cc0bab30 msgr2=0x7ff3cc0b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:51.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.064+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 shutdown_connections 2020-11-20T12:11:51.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:51.064+0000 7ff3d1c25700 1 -- 172.21.15.187:0/2495863262 wait complete. 2020-11-20T12:11:51.076 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:51.092 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:11:51.396 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_enqueued":1}} 2020-11-20T12:11:51.397 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2020-11-20T12:11:51.398 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:11:51.398 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:11:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:11:51.521 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:11:51.632 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:11:51.654 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:11:51.655 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:11:51.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:11:51.656 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:11:51.775 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:11:51.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:11:51.777 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:11:51.832 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:11:52.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.068+0000 7f743495c700 1 Processor -- start 2020-11-20T12:11:52.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.069+0000 7f743495c700 1 -- start start 2020-11-20T12:11:52.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.070+0000 7f743495c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.070+0000 7f743495c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74300622c0 0x7f74300b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.070+0000 7f743495c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74300b1300 con 0x7f7430066f70 2020-11-20T12:11:52.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.070+0000 7f743495c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7430067730 con 0x7f74300622c0 2020-11-20T12:11:52.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.070+0000 7f743495c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74300b15a0 con 0x7f74301484d0 2020-11-20T12:11:52.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.070+0000 7f742659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74300622c0 0x7f74300b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.070+0000 7f742659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74300622c0 0x7f74300b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46434/0 (socket says 172.21.15.187:46434) 2020-11-20T12:11:52.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.070+0000 7f742659c700 1 -- 172.21.15.187:0/3648173283 learned_addr learned my addr 172.21.15.187:0/3648173283 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:52.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.071+0000 7f742ed9d700 1 --2- 172.21.15.187:0/3648173283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.071+0000 7f742659c700 1 -- 172.21.15.187:0/3648173283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 msgr2=0x7f7430061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:52.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.071+0000 7f742659c700 1 --2- 172.21.15.187:0/3648173283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.071+0000 7f742659c700 1 -- 172.21.15.187:0/3648173283 >> v1:172.21.15.5:6789/0 conn(0x7f7430066f70 legacy=0x7f74300b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:52.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.071+0000 7f742659c700 1 -- 172.21.15.187:0/3648173283 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74300b1840 con 0x7f74300622c0 2020-11-20T12:11:52.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.071+0000 7f742659c700 1 --2- 172.21.15.187:0/3648173283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74300622c0 0x7f74300b5640 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f7418006fd0 tx=0x7f7418006eb0).ready entity=mon.1 client_cookie=2e9c43fb6ff60c4f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:52.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.071+0000 7f742e59c700 1 -- 172.21.15.187:0/3648173283 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7418009890 con 0x7f74300622c0 2020-11-20T12:11:52.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f742e59c700 1 -- 172.21.15.187:0/3648173283 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74180099f0 con 0x7f74300622c0 2020-11-20T12:11:52.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f742e59c700 1 -- 172.21.15.187:0/3648173283 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7418009d70 con 0x7f74300622c0 2020-11-20T12:11:52.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f743495c700 1 -- 172.21.15.187:0/3648173283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74300622c0 msgr2=0x7f74300b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f743495c700 1 --2- 172.21.15.187:0/3648173283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74300622c0 0x7f74300b5640 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f7418006fd0 tx=0x7f7418006eb0).stop 2020-11-20T12:11:52.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f743495c700 1 -- 172.21.15.187:0/3648173283 shutdown_connections 2020-11-20T12:11:52.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f743495c700 1 --2- 172.21.15.187:0/3648173283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f743495c700 1 --2- 172.21.15.187:0/3648173283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74300622c0 0x7f74300b5640 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f743495c700 1 -- 172.21.15.187:0/3648173283 >> 172.21.15.187:0/3648173283 conn(0x7f74300bab30 msgr2=0x7f74300baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:52.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f743495c700 1 -- 172.21.15.187:0/3648173283 shutdown_connections 2020-11-20T12:11:52.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.072+0000 7f743495c700 1 -- 172.21.15.187:0/3648173283 wait complete. 2020-11-20T12:11:52.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.074+0000 7f743495c700 1 Processor -- start 2020-11-20T12:11:52.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.074+0000 7f743495c700 1 -- start start 2020-11-20T12:11:52.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.075+0000 7f743495c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74300622c0 0x7f743009dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f743495c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7430066f70 0x7f74300b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f743495c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430065420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f743495c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74300b4380 con 0x7f74300622c0 2020-11-20T12:11:52.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f743495c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74300b45a0 con 0x7f7430066f70 2020-11-20T12:11:52.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f743495c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743009b8c0 con 0x7f74301484d0 2020-11-20T12:11:52.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f742659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7430066f70 0x7f74300b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f742659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7430066f70 0x7f74300b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46438/0 (socket says 172.21.15.187:46438) 2020-11-20T12:11:52.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f742659c700 1 -- 172.21.15.187:0/4036591852 learned_addr learned my addr 172.21.15.187:0/4036591852 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:52.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f742ed9d700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74300622c0 0x7f743009dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.076+0000 7f742f59e700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430065420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f742659c700 1 -- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 msgr2=0x7f7430065420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f742659c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430065420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f742659c700 1 -- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74300622c0 msgr2=0x7f743009dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f742659c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74300622c0 0x7f743009dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f742659c700 1 -- 172.21.15.187:0/4036591852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743009bb60 con 0x7f7430066f70 2020-11-20T12:11:52.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f742ed9d700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74300622c0 0x7f743009dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:52.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f742f59e700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430065420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:52.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f742659c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7430066f70 0x7f74300b3e90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f7430066430 tx=0x7f7418015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:52.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f74277fe700 1 -- 172.21.15.187:0/4036591852 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7418011dd0 con 0x7f7430066f70 2020-11-20T12:11:52.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f74277fe700 1 -- 172.21.15.187:0/4036591852 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741801e460 con 0x7f7430066f70 2020-11-20T12:11:52.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f74277fe700 1 -- 172.21.15.187:0/4036591852 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741801e7e0 con 0x7f7430066f70 2020-11-20T12:11:52.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7430065910 con 0x7f7430066f70 2020-11-20T12:11:52.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.077+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743009d0f0 con 0x7f7430066f70 2020-11-20T12:11:52.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.079+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743009bcf0 con 0x7f7430066f70 2020-11-20T12:11:52.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.079+0000 7f74277fe700 1 -- 172.21.15.187:0/4036591852 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7418025030 con 0x7f7430066f70 2020-11-20T12:11:52.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.080+0000 7f74277fe700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d20 0x7f740c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.080+0000 7f74277fe700 1 -- 172.21.15.187:0/4036591852 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7418071800 con 0x7f7430066f70 2020-11-20T12:11:52.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.080+0000 7f742ed9d700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d20 0x7f740c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.081+0000 7f742ed9d700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d20 0x7f740c0591e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f7420009670 tx=0x7f7420008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:52.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.084+0000 7f74277fe700 1 -- 172.21.15.187:0/4036591852 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7418046c30 con 0x7f7430066f70 2020-11-20T12:11:52.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.272+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7430061a20 con 0x7f7430066f70 2020-11-20T12:11:52.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.282+0000 7f74277fe700 1 -- 172.21.15.187:0/4036591852 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7418023030 con 0x7f7430066f70 2020-11-20T12:11:52.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d20 msgr2=0x7f740c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d20 0x7f740c0591e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f7420009670 tx=0x7f7420008040).stop 2020-11-20T12:11:52.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7430066f70 msgr2=0x7f74300b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7430066f70 0x7f74300b3e90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f7430066430 tx=0x7f7418015000).stop 2020-11-20T12:11:52.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 shutdown_connections 2020-11-20T12:11:52.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f740c056d20 0x7f740c0591e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74300622c0 0x7f743009dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74301484d0 0x7f7430065420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 --2- 172.21.15.187:0/4036591852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7430066f70 0x7f74300b3e90 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 >> 172.21.15.187:0/4036591852 conn(0x7f74300bab30 msgr2=0x7f7430143e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:52.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 shutdown_connections 2020-11-20T12:11:52.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.285+0000 7f743495c700 1 -- 172.21.15.187:0/4036591852 wait complete. 2020-11-20T12:11:52.309 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2020-11-20T12:11:52.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.550+0000 7f8c85487700 1 Processor -- start 2020-11-20T12:11:52.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.550+0000 7f8c85487700 1 -- start start 2020-11-20T12:11:52.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.551+0000 7f8c85487700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8c800652e0 0x7f8c800656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.552+0000 7f8c85487700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 0x7f8c8014a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.552+0000 7f8c85487700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c80149dd0 con 0x7f8c800b7300 2020-11-20T12:11:52.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.552+0000 7f8c85487700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8014a070 con 0x7f8c800652e0 2020-11-20T12:11:52.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.552+0000 7f8c85487700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8014a310 con 0x7f8c800b1590 2020-11-20T12:11:52.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.552+0000 7f8c7f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8c800b7300 0x7f8c80143920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43054/0 (socket says 172.21.15.187:43054) 2020-11-20T12:11:52.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.552+0000 7f8c7f7fe700 1 -- 172.21.15.187:0/578712525 learned_addr learned my addr 172.21.15.187:0/578712525 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:52.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.552+0000 7f8c7effd700 1 --2- 172.21.15.187:0/578712525 >> v2:172.21.15.5:3300/0 conn(0x7f8c800652e0 0x7f8c800656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.552+0000 7f8c767fc700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 0x7f8c8014a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060603675 0 0) 0x7f8c80149dd0 con 0x7f8c800b7300 2020-11-20T12:11:52.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60003240 con 0x7f8c800b7300 2020-11-20T12:11:52.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2694235395 0 0) 0x7f8c60003240 con 0x7f8c800b7300 2020-11-20T12:11:52.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 msgr2=0x7f8c8014a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:52.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 0x7f8c8014a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 >> v2:172.21.15.5:3300/0 conn(0x7f8c800652e0 msgr2=0x7f8c800656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:52.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 --2- 172.21.15.187:0/578712525 >> v2:172.21.15.5:3300/0 conn(0x7f8c800652e0 0x7f8c800656c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8014aca0 con 0x7f8c800b7300 2020-11-20T12:11:52.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8c78001860 con 0x7f8c800b7300 2020-11-20T12:11:52.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.553+0000 7f8c7effd700 1 --2- 172.21.15.187:0/578712525 >> v2:172.21.15.5:3300/0 conn(0x7f8c800652e0 0x7f8c800656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:11:52.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 >> v1:172.21.15.187:6789/0 conn(0x7f8c800b7300 legacy=0x7f8c80143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7e7fc700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 0x7f8c60003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7e7fc700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c60003c40 0x7f8c600044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7e7fc700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c600049d0 0x7f8c60004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8014a070 con 0x7f8c60003c40 2020-11-20T12:11:52.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8014a310 con 0x7f8c600049d0 2020-11-20T12:11:52.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60004200 con 0x7f8c800b1590 2020-11-20T12:11:52.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7f7fe700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c600049d0 0x7f8c60004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c7effd700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 0x7f8c60003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.554+0000 7f8c767fc700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c60003c40 0x7f8c600044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.555+0000 7f8c7f7fe700 1 -- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 msgr2=0x7f8c60003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:52.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.555+0000 7f8c7f7fe700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 0x7f8c60003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.555+0000 7f8c7f7fe700 1 -- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c60003c40 msgr2=0x7f8c600044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:52.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.555+0000 7f8c7f7fe700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c60003c40 0x7f8c600044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.555+0000 7f8c7f7fe700 1 -- 172.21.15.187:0/578712525 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c60003240 con 0x7f8c600049d0 2020-11-20T12:11:52.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.555+0000 7f8c7effd700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c800b1590 0x7f8c60003730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:11:52.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.555+0000 7f8c7f7fe700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c600049d0 0x7f8c60004e50 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f8c7800c840 tx=0x7f8c78007b80).ready entity=mon.1 client_cookie=b527f4714fa58452 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:52.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78005b80 con 0x7f8c600049d0 2020-11-20T12:11:52.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c78005ce0 con 0x7f8c600049d0 2020-11-20T12:11:52.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c7e7fc700 1 -- 172.21.15.187:0/578712525 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78007360 con 0x7f8c600049d0 2020-11-20T12:11:52.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c85487700 1 -- 172.21.15.187:0/578712525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c600049d0 msgr2=0x7f8c60004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c85487700 1 --2- 172.21.15.187:0/578712525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c600049d0 0x7f8c60004e50 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f8c7800c840 tx=0x7f8c78007b80).stop 2020-11-20T12:11:52.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c7f7fe700 1 -- 172.21.15.187:0/578712525 reap_dead start 2020-11-20T12:11:52.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c85487700 1 -- 172.21.15.187:0/578712525 shutdown_connections 2020-11-20T12:11:52.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c85487700 1 -- 172.21.15.187:0/578712525 >> 172.21.15.187:0/578712525 conn(0x7f8c800bacd0 msgr2=0x7f8c800bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:52.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c85487700 1 -- 172.21.15.187:0/578712525 shutdown_connections 2020-11-20T12:11:52.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.556+0000 7f8c85487700 1 -- 172.21.15.187:0/578712525 wait complete. 2020-11-20T12:11:52.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.558+0000 7f8c85487700 1 Processor -- start 2020-11-20T12:11:52.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.559+0000 7f8c85487700 1 -- start start 2020-11-20T12:11:52.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c85487700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c801553e0 0x7f8c800a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c85487700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c801557c0 0x7f8c80148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c85487700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c801565b0 0x7f8c801440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c85487700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c80156a50 con 0x7f8c801565b0 2020-11-20T12:11:52.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c85487700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c80099f20 con 0x7f8c801553e0 2020-11-20T12:11:52.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c85487700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8009a1c0 con 0x7f8c801557c0 2020-11-20T12:11:52.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c7effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c801553e0 0x7f8c800a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c7effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c801553e0 0x7f8c800a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46456/0 (socket says 172.21.15.187:46456) 2020-11-20T12:11:52.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.560+0000 7f8c7effd700 1 -- 172.21.15.187:0/762083964 learned_addr learned my addr 172.21.15.187:0/762083964 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:52.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c767fc700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c801557c0 0x7f8c80148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c7effd700 1 -- 172.21.15.187:0/762083964 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c801557c0 msgr2=0x7f8c80148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c7f7fe700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c801565b0 0x7f8c801440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c7effd700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c801557c0 0x7f8c80148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c7effd700 1 -- 172.21.15.187:0/762083964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c801565b0 msgr2=0x7f8c801440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:52.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c7effd700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c801565b0 0x7f8c801440b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:52.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c7effd700 1 -- 172.21.15.187:0/762083964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c800a0b50 con 0x7f8c801553e0 2020-11-20T12:11:52.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c7effd700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c801553e0 0x7f8c800a0660 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f8c70010cb0 tx=0x7f8c70008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:52.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c777fe700 1 -- 172.21.15.187:0/762083964 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7000ec30 con 0x7f8c801553e0 2020-11-20T12:11:52.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c777fe700 1 -- 172.21.15.187:0/762083964 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7000ed90 con 0x7f8c801553e0 2020-11-20T12:11:52.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c777fe700 1 -- 172.21.15.187:0/762083964 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70014680 con 0x7f8c801553e0 2020-11-20T12:11:52.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.561+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c80148f40 con 0x7f8c801553e0 2020-11-20T12:11:52.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.562+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c80148940 con 0x7f8c801553e0 2020-11-20T12:11:52.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.563+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c800a04d0 con 0x7f8c801553e0 2020-11-20T12:11:52.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.563+0000 7f8c777fe700 1 -- 172.21.15.187:0/762083964 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8c70014ab0 con 0x7f8c801553e0 2020-11-20T12:11:52.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.564+0000 7f8c777fe700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c5c056d20 0x7f8c5c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:52.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.564+0000 7f8c777fe700 1 -- 172.21.15.187:0/762083964 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8c70070980 con 0x7f8c801553e0 2020-11-20T12:11:52.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.564+0000 7f8c767fc700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c5c056d20 0x7f8c5c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:52.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.565+0000 7f8c767fc700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c5c056d20 0x7f8c5c0591e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f8c801452c0 tx=0x7f8c68008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:52.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.569+0000 7f8c777fe700 1 -- 172.21.15.187:0/762083964 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8c70045d80 con 0x7f8c801553e0 2020-11-20T12:11:52.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:52.826+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f8c80063600 con 0x7f8c801553e0 2020-11-20T12:11:53.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.237+0000 7f8c777fe700 1 -- 172.21.15.187:0/762083964 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v351) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8c70018020 con 0x7f8c801553e0 2020-11-20T12:11:53.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c5c056d20 msgr2=0x7f8c5c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:53.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c5c056d20 0x7f8c5c0591e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f8c801452c0 tx=0x7f8c68008040).stop 2020-11-20T12:11:53.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c801553e0 msgr2=0x7f8c800a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:53.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c801553e0 0x7f8c800a0660 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f8c70010cb0 tx=0x7f8c70008040).stop 2020-11-20T12:11:53.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 shutdown_connections 2020-11-20T12:11:53.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c5c056d20 0x7f8c5c0591e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c801565b0 0x7f8c801440b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c801553e0 0x7f8c800a0660 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 --2- 172.21.15.187:0/762083964 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8c801557c0 0x7f8c80148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 >> 172.21.15.187:0/762083964 conn(0x7f8c800bacd0 msgr2=0x7f8c800bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:53.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 shutdown_connections 2020-11-20T12:11:53.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.240+0000 7f8c85487700 1 -- 172.21.15.187:0/762083964 wait complete. 2020-11-20T12:11:53.271 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:11:53.272 INFO:tasks.cephfs_test_runner:Starting test: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:11:53.272 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2020-11-20T12:11:53.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.515+0000 7fbd252fd700 1 Processor -- start 2020-11-20T12:11:53.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.515+0000 7fbd252fd700 1 -- start start 2020-11-20T12:11:53.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.516+0000 7fbd252fd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbd200b57b0 0x7fbd20068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:53.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.516+0000 7fbd252fd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbd201482d0 0x7fbd20062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:53.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.516+0000 7fbd252fd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd20062b50 0x7fbd20065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:53.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.516+0000 7fbd252fd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd20144b80 con 0x7fbd200b57b0 2020-11-20T12:11:53.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.516+0000 7fbd252fd700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd201489e0 con 0x7fbd201482d0 2020-11-20T12:11:53.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.516+0000 7fbd252fd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd20148c80 con 0x7fbd20062b50 2020-11-20T12:11:53.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1f7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbd201482d0 0x7fbd20062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:53.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1f7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbd201482d0 0x7fbd20062660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46466/0 (socket says 172.21.15.187:46466) 2020-11-20T12:11:53.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1f7fe700 1 -- 172.21.15.187:0/4143727362 learned_addr learned my addr 172.21.15.187:0/4143727362 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:53.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1ffff700 1 --2- 172.21.15.187:0/4143727362 >> v2:172.21.15.5:3300/0 conn(0x7fbd200b57b0 0x7fbd20068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:53.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd24afc700 1 --2- 172.21.15.187:0/4143727362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd20062b50 0x7fbd20065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:53.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1f7fe700 1 -- 172.21.15.187:0/4143727362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd20062b50 msgr2=0x7fbd20065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:53.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1f7fe700 1 --2- 172.21.15.187:0/4143727362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd20062b50 0x7fbd20065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1f7fe700 1 -- 172.21.15.187:0/4143727362 >> v2:172.21.15.5:3300/0 conn(0x7fbd200b57b0 msgr2=0x7fbd20068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:53.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1f7fe700 1 --2- 172.21.15.187:0/4143727362 >> v2:172.21.15.5:3300/0 conn(0x7fbd200b57b0 0x7fbd20068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.517+0000 7fbd1f7fe700 1 -- 172.21.15.187:0/4143727362 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd20148f20 con 0x7fbd201482d0 2020-11-20T12:11:53.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.518+0000 7fbd1f7fe700 1 --2- 172.21.15.187:0/4143727362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd201482d0 0x7fbd20062660 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7fbd2009d820 tx=0x7fbd14006eb0).ready entity=mon.1 client_cookie=61ad1008cb3ca864 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:53.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.518+0000 7fbd1effd700 1 -- 172.21.15.187:0/4143727362 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd140098d0 con 0x7fbd201482d0 2020-11-20T12:11:53.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.518+0000 7fbd1effd700 1 -- 172.21.15.187:0/4143727362 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd14009a30 con 0x7fbd201482d0 2020-11-20T12:11:53.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.518+0000 7fbd1effd700 1 -- 172.21.15.187:0/4143727362 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14009db0 con 0x7fbd201482d0 2020-11-20T12:11:53.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 -- 172.21.15.187:0/4143727362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd201482d0 msgr2=0x7fbd20062660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:53.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 --2- 172.21.15.187:0/4143727362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd201482d0 0x7fbd20062660 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7fbd2009d820 tx=0x7fbd14006eb0).stop 2020-11-20T12:11:53.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 -- 172.21.15.187:0/4143727362 shutdown_connections 2020-11-20T12:11:53.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 --2- 172.21.15.187:0/4143727362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd20062b50 0x7fbd20065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 --2- 172.21.15.187:0/4143727362 >> v2:172.21.15.5:3300/0 conn(0x7fbd200b57b0 0x7fbd20068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 --2- 172.21.15.187:0/4143727362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd201482d0 0x7fbd20062660 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:53.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 -- 172.21.15.187:0/4143727362 >> 172.21.15.187:0/4143727362 conn(0x7fbd200bab30 msgr2=0x7fbd200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:53.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 -- 172.21.15.187:0/4143727362 shutdown_connections 2020-11-20T12:11:53.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.519+0000 7fbd252fd700 1 -- 172.21.15.187:0/4143727362 wait complete. 2020-11-20T12:11:53.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.521+0000 7fbd252fd700 1 Processor -- start 2020-11-20T12:11:53.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.521+0000 7fbd252fd700 1 -- start start 2020-11-20T12:11:53.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.522+0000 7fbd252fd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd20062b50 0x7fbd20157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:53.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.522+0000 7fbd252fd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd200b57b0 0x7fbd20157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:53.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.522+0000 7fbd252fd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd201482d0 0x7fbd2015ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:53.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd252fd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2009f4c0 con 0x7fbd200b57b0 2020-11-20T12:11:53.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd252fd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2009eae0 con 0x7fbd20062b50 2020-11-20T12:11:53.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd252fd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2009ed80 con 0x7fbd201482d0 2020-11-20T12:11:53.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd1ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd20062b50 0x7fbd20157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:53.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd1ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd20062b50 0x7fbd20157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46470/0 (socket says 172.21.15.187:46470) 2020-11-20T12:11:53.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd1ffff700 1 -- 172.21.15.187:0/398506650 learned_addr learned my addr 172.21.15.187:0/398506650 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:53.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd1ffff700 1 -- 172.21.15.187:0/398506650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd201482d0 msgr2=0x7fbd2015ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:53.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd1ffff700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd201482d0 0x7fbd2015ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:53.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd1ffff700 1 -- 172.21.15.187:0/398506650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd200b57b0 msgr2=0x7fbd20157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:53.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd1ffff700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd200b57b0 0x7fbd20157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:53.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.523+0000 7fbd1ffff700 1 -- 172.21.15.187:0/398506650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2009f020 con 0x7fbd20062b50 2020-11-20T12:11:53.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.524+0000 7fbd1ffff700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd20062b50 0x7fbd20157720 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7fbd1000db60 tx=0x7fbd10008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:53.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.524+0000 7fbcfffff700 1 -- 172.21.15.187:0/398506650 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10009790 con 0x7fbd20062b50 2020-11-20T12:11:53.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.524+0000 7fbcfffff700 1 -- 172.21.15.187:0/398506650 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd100098f0 con 0x7fbd20062b50 2020-11-20T12:11:53.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.524+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd20158160 con 0x7fbd20062b50 2020-11-20T12:11:53.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.524+0000 7fbcfffff700 1 -- 172.21.15.187:0/398506650 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10009c90 con 0x7fbd20062b50 2020-11-20T12:11:53.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.524+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2009e730 con 0x7fbd20062b50 2020-11-20T12:11:53.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.525+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd200623a0 con 0x7fbd20062b50 2020-11-20T12:11:53.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.526+0000 7fbcfffff700 1 -- 172.21.15.187:0/398506650 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbd10013480 con 0x7fbd20062b50 2020-11-20T12:11:53.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.526+0000 7fbcfffff700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbcf8056d20 0x7fbcf80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:53.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.526+0000 7fbcfffff700 1 -- 172.21.15.187:0/398506650 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbd10070590 con 0x7fbd20062b50 2020-11-20T12:11:53.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.527+0000 7fbd1f7fe700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbcf8056d20 0x7fbcf80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.527+0000 7fbd1f7fe700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbcf8056d20 0x7fbcf80591e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fbd20146310 tx=0x7fbd14015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:53.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.530+0000 7fbcfffff700 1 -- 172.21.15.187:0/398506650 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbd10045820 con 0x7fbd20062b50 2020-11-20T12:11:53.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:53.783+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7fbd20158100 con 0x7fbd20062b50 2020-11-20T12:11:54.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.240+0000 7fbcfffff700 1 -- 172.21.15.187:0/398506650 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v352) v1 ==== 144+0+0 (secure 0 0 0) 0x7fbd10048e40 con 0x7fbd20062b50 2020-11-20T12:11:54.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbcf8056d20 msgr2=0x7fbcf80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbcf8056d20 0x7fbcf80591e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fbd20146310 tx=0x7fbd14015000).stop 2020-11-20T12:11:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd20062b50 msgr2=0x7fbd20157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd20062b50 0x7fbd20157720 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7fbd1000db60 tx=0x7fbd10008040).stop 2020-11-20T12:11:54.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 shutdown_connections 2020-11-20T12:11:54.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbcf8056d20 0x7fbcf80591e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd201482d0 0x7fbd2015ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:54.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd20062b50 0x7fbd20157720 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 --2- 172.21.15.187:0/398506650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd200b57b0 0x7fbd20157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:54.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 >> 172.21.15.187:0/398506650 conn(0x7fbd200bab30 msgr2=0x7fbd20065f20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:54.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.243+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 shutdown_connections 2020-11-20T12:11:54.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.244+0000 7fbd252fd700 1 -- 172.21.15.187:0/398506650 wait complete. 2020-11-20T12:11:54.271 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:11:54.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.525+0000 7f98b114c700 1 Processor -- start 2020-11-20T12:11:54.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.526+0000 7f98b114c700 1 -- start start 2020-11-20T12:11:54.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.527+0000 7f98b114c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f98ac1484d0 0x7f98ac061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.527+0000 7f98b114c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f98ac0622c0 0x7f98ac0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.527+0000 7f98b114c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac0b1300 con 0x7f98ac066f70 2020-11-20T12:11:54.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.527+0000 7f98b114c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac067730 con 0x7f98ac1484d0 2020-11-20T12:11:54.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.527+0000 7f98b114c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac0b15a0 con 0x7f98ac0622c0 2020-11-20T12:11:54.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.528+0000 7f98b094b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f98ac066f70 0x7f98ac0b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43086/0 (socket says 172.21.15.187:43086) 2020-11-20T12:11:54.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.528+0000 7f98b094b700 1 -- 172.21.15.187:0/2843425053 learned_addr learned my addr 172.21.15.187:0/2843425053 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:54.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.528+0000 7f98ab7fe700 1 --2- 172.21.15.187:0/2843425053 >> v2:172.21.15.187:3300/0 conn(0x7f98ac0622c0 0x7f98ac0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.528+0000 7f98abfff700 1 --2- 172.21.15.187:0/2843425053 >> v2:172.21.15.5:3300/0 conn(0x7f98ac1484d0 0x7f98ac061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.528+0000 7f98aaffd700 1 -- 172.21.15.187:0/2843425053 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522486232 0 0) 0x7f98ac0b1300 con 0x7f98ac066f70 2020-11-20T12:11:54.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.528+0000 7f98aaffd700 1 -- 172.21.15.187:0/2843425053 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9898003240 con 0x7f98ac066f70 2020-11-20T12:11:54.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98ab7fe700 1 -- 172.21.15.187:0/2843425053 >> v2:172.21.15.5:3300/0 conn(0x7f98ac1484d0 msgr2=0x7f98ac061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:54.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98ab7fe700 1 --2- 172.21.15.187:0/2843425053 >> v2:172.21.15.5:3300/0 conn(0x7f98ac1484d0 0x7f98ac061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98ab7fe700 1 -- 172.21.15.187:0/2843425053 >> v1:172.21.15.187:6789/0 conn(0x7f98ac066f70 legacy=0x7f98ac0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98ab7fe700 1 -- 172.21.15.187:0/2843425053 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac0b1840 con 0x7f98ac0622c0 2020-11-20T12:11:54.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98abfff700 1 --2- 172.21.15.187:0/2843425053 >> v2:172.21.15.5:3300/0 conn(0x7f98ac1484d0 0x7f98ac061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:11:54.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98ab7fe700 1 --2- 172.21.15.187:0/2843425053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac0622c0 0x7f98ac0b5640 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f98ac068600 tx=0x7f98a0006df0).ready entity=mon.1 client_cookie=a3d5e4e3cff74981 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:54.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98aaffd700 1 -- 172.21.15.187:0/2843425053 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a00097b0 con 0x7f98ac0622c0 2020-11-20T12:11:54.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98aaffd700 1 -- 172.21.15.187:0/2843425053 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a0009910 con 0x7f98ac0622c0 2020-11-20T12:11:54.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.529+0000 7f98aaffd700 1 -- 172.21.15.187:0/2843425053 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0009c90 con 0x7f98ac0622c0 2020-11-20T12:11:54.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.530+0000 7f98b114c700 1 -- 172.21.15.187:0/2843425053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac0622c0 msgr2=0x7f98ac0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.530+0000 7f98b114c700 1 --2- 172.21.15.187:0/2843425053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac0622c0 0x7f98ac0b5640 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f98ac068600 tx=0x7f98a0006df0).stop 2020-11-20T12:11:54.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.530+0000 7f98b114c700 1 -- 172.21.15.187:0/2843425053 shutdown_connections 2020-11-20T12:11:54.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.530+0000 7f98b114c700 1 --2- 172.21.15.187:0/2843425053 >> v2:172.21.15.5:3300/0 conn(0x7f98ac1484d0 0x7f98ac061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.530+0000 7f98b114c700 1 --2- 172.21.15.187:0/2843425053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac0622c0 0x7f98ac0b5640 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.530+0000 7f98b114c700 1 -- 172.21.15.187:0/2843425053 >> 172.21.15.187:0/2843425053 conn(0x7f98ac0bab30 msgr2=0x7f98ac0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:54.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.530+0000 7f98b114c700 1 -- 172.21.15.187:0/2843425053 shutdown_connections 2020-11-20T12:11:54.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.530+0000 7f98b114c700 1 -- 172.21.15.187:0/2843425053 wait complete. 2020-11-20T12:11:54.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.532+0000 7f98b114c700 1 Processor -- start 2020-11-20T12:11:54.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.532+0000 7f98b114c700 1 -- start start 2020-11-20T12:11:54.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98b114c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98ac0622c0 0x7f98ac0b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98b114c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac066f70 0x7f98ac144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98b114c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98ac1484d0 0x7f98ac1584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98b114c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0b4660 con 0x7f98ac1484d0 2020-11-20T12:11:54.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98b114c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0b4900 con 0x7f98ac066f70 2020-11-20T12:11:54.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98b114c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac158020 con 0x7f98ac0622c0 2020-11-20T12:11:54.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98ab7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac066f70 0x7f98ac144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98ab7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac066f70 0x7f98ac144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46488/0 (socket says 172.21.15.187:46488) 2020-11-20T12:11:54.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98ab7fe700 1 -- 172.21.15.187:0/3717336973 learned_addr learned my addr 172.21.15.187:0/3717336973 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:54.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98b094b700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98ac1484d0 0x7f98ac1584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.533+0000 7f98abfff700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98ac0622c0 0x7f98ac0b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98ab7fe700 1 -- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98ac0622c0 msgr2=0x7f98ac0b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98ab7fe700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98ac0622c0 0x7f98ac0b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98ab7fe700 1 -- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98ac1484d0 msgr2=0x7f98ac1584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98ab7fe700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98ac1484d0 0x7f98ac1584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98ab7fe700 1 -- 172.21.15.187:0/3717336973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac0b4240 con 0x7f98ac066f70 2020-11-20T12:11:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98abfff700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98ac0622c0 0x7f98ac0b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98ab7fe700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac066f70 0x7f98ac144350 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f98a0006e30 tx=0x7f98a0017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f988bfff700 1 -- 172.21.15.187:0/3717336973 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a001a3f0 con 0x7f98ac066f70 2020-11-20T12:11:54.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f988bfff700 1 -- 172.21.15.187:0/3717336973 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a001a550 con 0x7f98ac066f70 2020-11-20T12:11:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f988bfff700 1 -- 172.21.15.187:0/3717336973 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a001a8f0 con 0x7f98ac066f70 2020-11-20T12:11:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac144840 con 0x7f98ac066f70 2020-11-20T12:11:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.534+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac1589b0 con 0x7f98ac066f70 2020-11-20T12:11:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.536+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ac0b4490 con 0x7f98ac066f70 2020-11-20T12:11:54.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.536+0000 7f988bfff700 1 -- 172.21.15.187:0/3717336973 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f98a001ac60 con 0x7f98ac066f70 2020-11-20T12:11:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.537+0000 7f988bfff700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9884056d20 0x7f98840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.537+0000 7f988bfff700 1 -- 172.21.15.187:0/3717336973 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f98a0071a70 con 0x7f98ac066f70 2020-11-20T12:11:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.537+0000 7f98abfff700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9884056d20 0x7f98840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.538+0000 7f98abfff700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9884056d20 0x7f98840591e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f989c006bd0 tx=0x7f989c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:54.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.541+0000 7f988bfff700 1 -- 172.21.15.187:0/3717336973 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f98a0073e60 con 0x7f98ac066f70 2020-11-20T12:11:54.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.727+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f98ac061a20 con 0x7f98ac066f70 2020-11-20T12:11:54.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.738+0000 7f988bfff700 1 -- 172.21.15.187:0/3717336973 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f98a0071020 con 0x7f98ac066f70 2020-11-20T12:11:54.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.740+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9884056d20 msgr2=0x7f98840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9884056d20 0x7f98840591e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f989c006bd0 tx=0x7f989c008040).stop 2020-11-20T12:11:54.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac066f70 msgr2=0x7f98ac144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac066f70 0x7f98ac144350 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f98a0006e30 tx=0x7f98a0017000).stop 2020-11-20T12:11:54.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 shutdown_connections 2020-11-20T12:11:54.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9884056d20 0x7f98840591e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98ac1484d0 0x7f98ac1584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98ac0622c0 0x7f98ac0b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 --2- 172.21.15.187:0/3717336973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98ac066f70 0x7f98ac144350 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 >> 172.21.15.187:0/3717336973 conn(0x7f98ac0bab30 msgr2=0x7f98ac0991f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:54.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 shutdown_connections 2020-11-20T12:11:54.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:54.741+0000 7f98b114c700 1 -- 172.21.15.187:0/3717336973 wait complete. 2020-11-20T12:11:54.768 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:11:54.769 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:11:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:11:54.904 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:11:55.012 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:11:55.031 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:11:55.032 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:11:55.032 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:55.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.278+0000 7ff6ce22e700 1 Processor -- start 2020-11-20T12:11:55.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.278+0000 7ff6ce22e700 1 -- start start 2020-11-20T12:11:55.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.279+0000 7ff6ce22e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6c80b57b0 0x7ff6c8068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.279+0000 7ff6ce22e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c81482d0 0x7ff6c8062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.279+0000 7ff6ce22e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c8144b80 con 0x7ff6c8062b50 2020-11-20T12:11:55.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.279+0000 7ff6ce22e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c81489e0 con 0x7ff6c80b57b0 2020-11-20T12:11:55.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.279+0000 7ff6ce22e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8148c80 con 0x7ff6c81482d0 2020-11-20T12:11:55.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cd22c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6c80b57b0 0x7ff6c8068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cd22c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6c80b57b0 0x7ff6c8068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46492/0 (socket says 172.21.15.187:46492) 2020-11-20T12:11:55.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cd22c700 1 -- 172.21.15.187:0/908839585 learned_addr learned my addr 172.21.15.187:0/908839585 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:55.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cca2b700 1 --2- 172.21.15.187:0/908839585 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c81482d0 0x7ff6c8062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cd22c700 1 -- 172.21.15.187:0/908839585 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c81482d0 msgr2=0x7ff6c8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:55.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cd22c700 1 --2- 172.21.15.187:0/908839585 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c81482d0 0x7ff6c8062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cd22c700 1 -- 172.21.15.187:0/908839585 >> v1:172.21.15.5:6789/0 conn(0x7ff6c8062b50 legacy=0x7ff6c8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:55.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cd22c700 1 -- 172.21.15.187:0/908839585 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c8148f20 con 0x7ff6c80b57b0 2020-11-20T12:11:55.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.280+0000 7ff6cca2b700 1 --2- 172.21.15.187:0/908839585 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c81482d0 0x7ff6c8062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:55.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.281+0000 7ff6cd22c700 1 --2- 172.21.15.187:0/908839585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c80b57b0 0x7ff6c8068230 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7ff6c4006fd0 tx=0x7ff6c4006eb0).ready entity=mon.1 client_cookie=bb06ac327a0e6b21 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:55.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.281+0000 7ff6bbfff700 1 -- 172.21.15.187:0/908839585 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4009890 con 0x7ff6c80b57b0 2020-11-20T12:11:55.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.281+0000 7ff6bbfff700 1 -- 172.21.15.187:0/908839585 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c40099f0 con 0x7ff6c80b57b0 2020-11-20T12:11:55.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.281+0000 7ff6bbfff700 1 -- 172.21.15.187:0/908839585 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4009d70 con 0x7ff6c80b57b0 2020-11-20T12:11:55.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.282+0000 7ff6ce22e700 1 -- 172.21.15.187:0/908839585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c80b57b0 msgr2=0x7ff6c8068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.282+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/908839585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c80b57b0 0x7ff6c8068230 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7ff6c4006fd0 tx=0x7ff6c4006eb0).stop 2020-11-20T12:11:55.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.282+0000 7ff6ce22e700 1 -- 172.21.15.187:0/908839585 shutdown_connections 2020-11-20T12:11:55.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.282+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/908839585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c80b57b0 0x7ff6c8068230 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.282+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/908839585 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c81482d0 0x7ff6c8062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.282+0000 7ff6ce22e700 1 -- 172.21.15.187:0/908839585 >> 172.21.15.187:0/908839585 conn(0x7ff6c80bab30 msgr2=0x7ff6c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:55.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.282+0000 7ff6ce22e700 1 -- 172.21.15.187:0/908839585 shutdown_connections 2020-11-20T12:11:55.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.282+0000 7ff6ce22e700 1 -- 172.21.15.187:0/908839585 wait complete. 2020-11-20T12:11:55.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.284+0000 7ff6ce22e700 1 Processor -- start 2020-11-20T12:11:55.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.284+0000 7ff6ce22e700 1 -- start start 2020-11-20T12:11:55.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.285+0000 7ff6ce22e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c8062b50 0x7ff6c8155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.285+0000 7ff6ce22e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6c81482d0 0x7ff6c8142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.285+0000 7ff6ce22e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c809adf0 0x7ff6c809c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.285+0000 7ff6ce22e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8156230 con 0x7ff6c81482d0 2020-11-20T12:11:55.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.285+0000 7ff6ce22e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c809b290 con 0x7ff6c809adf0 2020-11-20T12:11:55.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.285+0000 7ff6ce22e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c809b530 con 0x7ff6c8062b50 2020-11-20T12:11:55.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cda2d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c809adf0 0x7ff6c809c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cda2d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c809adf0 0x7ff6c809c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46502/0 (socket says 172.21.15.187:46502) 2020-11-20T12:11:55.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cda2d700 1 -- 172.21.15.187:0/339920969 learned_addr learned my addr 172.21.15.187:0/339920969 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:55.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cd22c700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c8062b50 0x7ff6c8155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cca2b700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6c81482d0 0x7ff6c8142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cda2d700 1 -- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c8062b50 msgr2=0x7ff6c8155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cda2d700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c8062b50 0x7ff6c8155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cda2d700 1 -- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6c81482d0 msgr2=0x7ff6c8142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cda2d700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6c81482d0 0x7ff6c8142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cda2d700 1 -- 172.21.15.187:0/339920969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c809b7d0 con 0x7ff6c809adf0 2020-11-20T12:11:55.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.286+0000 7ff6cd22c700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c8062b50 0x7ff6c8155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:55.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.287+0000 7ff6cda2d700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c809adf0 0x7ff6c809c520 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7ff6c000cab0 tx=0x7ff6c00023b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:55.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.287+0000 7ff6b97fa700 1 -- 172.21.15.187:0/339920969 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c000a830 con 0x7ff6c809adf0 2020-11-20T12:11:55.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.287+0000 7ff6b97fa700 1 -- 172.21.15.187:0/339920969 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c000a990 con 0x7ff6c809adf0 2020-11-20T12:11:55.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.287+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c80b48d0 con 0x7ff6c809adf0 2020-11-20T12:11:55.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.287+0000 7ff6b97fa700 1 -- 172.21.15.187:0/339920969 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c000ad30 con 0x7ff6c809adf0 2020-11-20T12:11:55.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.287+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c8142ea0 con 0x7ff6c809adf0 2020-11-20T12:11:55.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.288+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c809b960 con 0x7ff6c809adf0 2020-11-20T12:11:55.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.289+0000 7ff6b97fa700 1 -- 172.21.15.187:0/339920969 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff6c0019030 con 0x7ff6c809adf0 2020-11-20T12:11:55.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.290+0000 7ff6b97fa700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6a0056d20 0x7ff6a00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.290+0000 7ff6b97fa700 1 -- 172.21.15.187:0/339920969 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff6c006f8c0 con 0x7ff6c809adf0 2020-11-20T12:11:55.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.290+0000 7ff6cd22c700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6a0056d20 0x7ff6a00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.290+0000 7ff6cd22c700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6a0056d20 0x7ff6a00591e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7ff6c8062fe0 tx=0x7ff6c4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:55.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.294+0000 7ff6b97fa700 1 -- 172.21.15.187:0/339920969 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff6c0043fd0 con 0x7ff6c809adf0 2020-11-20T12:11:55.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.536+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6c80b4a60 con 0x7ff6c809adf0 2020-11-20T12:11:55.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.537+0000 7ff6b97fa700 1 -- 172.21.15.187:0/339920969 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7ff6c0044d70 con 0x7ff6c809adf0 2020-11-20T12:11:55.541 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:55.541 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:55.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.539+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6a0056d20 msgr2=0x7ff6a00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.539+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6a0056d20 0x7ff6a00591e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7ff6c8062fe0 tx=0x7ff6c4015000).stop 2020-11-20T12:11:55.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c809adf0 msgr2=0x7ff6c809c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c809adf0 0x7ff6c809c520 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7ff6c000cab0 tx=0x7ff6c00023b0).stop 2020-11-20T12:11:55.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 shutdown_connections 2020-11-20T12:11:55.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6a0056d20 0x7ff6a00591e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c809adf0 0x7ff6c809c520 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6c8062b50 0x7ff6c8155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 --2- 172.21.15.187:0/339920969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6c81482d0 0x7ff6c8142770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 >> 172.21.15.187:0/339920969 conn(0x7ff6c80bab30 msgr2=0x7ff6c80baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:55.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 shutdown_connections 2020-11-20T12:11:55.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.540+0000 7ff6ce22e700 1 -- 172.21.15.187:0/339920969 wait complete. 2020-11-20T12:11:55.547 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:55.568 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:11:55.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.801+0000 7f6fabfff700 1 Processor -- start 2020-11-20T12:11:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.801+0000 7f6fabfff700 1 -- start start 2020-11-20T12:11:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.802+0000 7f6fabfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 0x7f6fac061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.802+0000 7f6fabfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6fac066f70 0x7f6fac0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.802+0000 7f6fabfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac0b1300 con 0x7f6fac0622c0 2020-11-20T12:11:55.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.803+0000 7f6fabfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac067730 con 0x7f6fac066f70 2020-11-20T12:11:55.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.803+0000 7f6fabfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0b15a0 con 0x7f6fac1484d0 2020-11-20T12:11:55.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.803+0000 7f6faaffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 0x7f6fac061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.803+0000 7f6fab7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6fac066f70 0x7f6fac0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.803+0000 7f6faaffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 0x7f6fac061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:47394/0 (socket says 172.21.15.187:47394) 2020-11-20T12:11:55.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.803+0000 7f6faaffd700 1 -- 172.21.15.187:0/4274775558 learned_addr learned my addr 172.21.15.187:0/4274775558 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:55.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.804+0000 7f6fab7fe700 1 -- 172.21.15.187:0/4274775558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 msgr2=0x7f6fac061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:55.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.804+0000 7f6fab7fe700 1 --2- 172.21.15.187:0/4274775558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 0x7f6fac061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.804+0000 7f6fab7fe700 1 -- 172.21.15.187:0/4274775558 >> v1:172.21.15.5:6789/0 conn(0x7f6fac0622c0 legacy=0x7f6fac0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.804+0000 7f6fab7fe700 1 -- 172.21.15.187:0/4274775558 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac0b1840 con 0x7f6fac066f70 2020-11-20T12:11:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.804+0000 7f6fab7fe700 1 --2- 172.21.15.187:0/4274775558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac066f70 0x7f6fac0b5d50 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f6fa4009c90 tx=0x7f6fa40091d0).ready entity=mon.1 client_cookie=a687dba1a8abd931 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.805+0000 7f6faa7fc700 1 -- 172.21.15.187:0/4274775558 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa400ba70 con 0x7f6fac066f70 2020-11-20T12:11:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.805+0000 7f6faa7fc700 1 -- 172.21.15.187:0/4274775558 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa400bbd0 con 0x7f6fac066f70 2020-11-20T12:11:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.805+0000 7f6faa7fc700 1 -- 172.21.15.187:0/4274775558 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4011400 con 0x7f6fac066f70 2020-11-20T12:11:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.805+0000 7f6fabfff700 1 -- 172.21.15.187:0/4274775558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac066f70 msgr2=0x7f6fac0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.805+0000 7f6fabfff700 1 --2- 172.21.15.187:0/4274775558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac066f70 0x7f6fac0b5d50 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f6fa4009c90 tx=0x7f6fa40091d0).stop 2020-11-20T12:11:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.806+0000 7f6fabfff700 1 -- 172.21.15.187:0/4274775558 shutdown_connections 2020-11-20T12:11:55.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.806+0000 7f6fabfff700 1 --2- 172.21.15.187:0/4274775558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac066f70 0x7f6fac0b5d50 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.806+0000 7f6fabfff700 1 --2- 172.21.15.187:0/4274775558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 0x7f6fac061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:55.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.806+0000 7f6fabfff700 1 -- 172.21.15.187:0/4274775558 >> 172.21.15.187:0/4274775558 conn(0x7f6fac0bab30 msgr2=0x7f6fac0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:55.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.806+0000 7f6fabfff700 1 -- 172.21.15.187:0/4274775558 shutdown_connections 2020-11-20T12:11:55.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.806+0000 7f6fabfff700 1 -- 172.21.15.187:0/4274775558 wait complete. 2020-11-20T12:11:55.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.808+0000 7f6fabfff700 1 Processor -- start 2020-11-20T12:11:55.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.808+0000 7f6fabfff700 1 -- start start 2020-11-20T12:11:55.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.809+0000 7f6fabfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac0622c0 0x7f6fac139850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.809+0000 7f6fabfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fac066f70 0x7f6fac13ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.809+0000 7f6fabfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 0x7f6fac131c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6fabfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0b3820 con 0x7f6fac066f70 2020-11-20T12:11:55.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6fabfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac139e40 con 0x7f6fac0622c0 2020-11-20T12:11:55.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6fabfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac13a0e0 con 0x7f6fac1484d0 2020-11-20T12:11:55.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6faaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac0622c0 0x7f6fac139850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6faaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac0622c0 0x7f6fac139850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46512/0 (socket says 172.21.15.187:46512) 2020-11-20T12:11:55.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6faaffd700 1 -- 172.21.15.187:0/3799328869 learned_addr learned my addr 172.21.15.187:0/3799328869 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:55.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6faaffd700 1 -- 172.21.15.187:0/3799328869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 msgr2=0x7f6fac131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6faaffd700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 0x7f6fac131c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:55.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6faaffd700 1 -- 172.21.15.187:0/3799328869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fac066f70 msgr2=0x7f6fac13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:55.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6faaffd700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fac066f70 0x7f6fac13ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:55.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.810+0000 7f6faaffd700 1 -- 172.21.15.187:0/3799328869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac13a380 con 0x7f6fac0622c0 2020-11-20T12:11:55.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.811+0000 7f6faaffd700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac0622c0 0x7f6fac139850 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f6f9c00f860 tx=0x7f6f9c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:55.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.811+0000 7f6fa2ffd700 1 -- 172.21.15.187:0/3799328869 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00d6c0 con 0x7f6fac0622c0 2020-11-20T12:11:55.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.811+0000 7f6fa2ffd700 1 -- 172.21.15.187:0/3799328869 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c00d820 con 0x7f6fac0622c0 2020-11-20T12:11:55.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.811+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac13d3c0 con 0x7f6fac0622c0 2020-11-20T12:11:55.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.811+0000 7f6fa2ffd700 1 -- 172.21.15.187:0/3799328869 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00dba0 con 0x7f6fac0622c0 2020-11-20T12:11:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.811+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac138bb0 con 0x7f6fac0622c0 2020-11-20T12:11:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.812+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac13a510 con 0x7f6fac0622c0 2020-11-20T12:11:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.813+0000 7f6fa2ffd700 1 -- 172.21.15.187:0/3799328869 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6f9c024070 con 0x7f6fac0622c0 2020-11-20T12:11:55.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.814+0000 7f6fa2ffd700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f88056d20 0x7f6f880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:55.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.814+0000 7f6fa2ffd700 1 -- 172.21.15.187:0/3799328869 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6f9c0705b0 con 0x7f6fac0622c0 2020-11-20T12:11:55.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.814+0000 7f6fa3fff700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f88056d20 0x7f6f880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:55.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.815+0000 7f6fa3fff700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f88056d20 0x7f6f880591e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6fac13d720 tx=0x7f6f940077d0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:55.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:55.817+0000 7f6fa2ffd700 1 -- 172.21.15.187:0/3799328869 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6f9c045930 con 0x7f6fac0622c0 2020-11-20T12:11:56.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.001+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fac138e60 con 0x7f6fac0622c0 2020-11-20T12:11:56.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.002+0000 7f6fa2ffd700 1 -- 172.21.15.187:0/3799328869 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16912 (secure 0 0 0) 0x7f6f9c018070 con 0x7f6fac0622c0 2020-11-20T12:11:56.007 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:56.007 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":72,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:10:42.605346+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:10:20.847618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2020-11-20T12:10:22.150115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1383561284":"2020-11-21T12:10:42.605289+0000","172.21.15.187:6834/1383561284":"2020-11-21T12:10:42.605289+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:11:56.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f88056d20 msgr2=0x7f6f880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f88056d20 0x7f6f880591e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6fac13d720 tx=0x7f6f940077d0).stop 2020-11-20T12:11:56.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac0622c0 msgr2=0x7f6fac139850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac0622c0 0x7f6fac139850 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f6f9c00f860 tx=0x7f6f9c008040).stop 2020-11-20T12:11:56.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 shutdown_connections 2020-11-20T12:11:56.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6f88056d20 0x7f6f880591e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fac1484d0 0x7f6fac131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fac0622c0 0x7f6fac139850 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 --2- 172.21.15.187:0/3799328869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fac066f70 0x7f6fac13ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 >> 172.21.15.187:0/3799328869 conn(0x7f6fac0bab30 msgr2=0x7f6fac0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:56.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 shutdown_connections 2020-11-20T12:11:56.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.005+0000 7f6fabfff700 1 -- 172.21.15.187:0/3799328869 wait complete. 2020-11-20T12:11:56.030 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:10:20.847618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:10:22.150115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:11:56.031 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:11:56.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.270+0000 7f5924e91700 1 Processor -- start 2020-11-20T12:11:56.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.270+0000 7f5924e91700 1 -- start start 2020-11-20T12:11:56.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.272+0000 7f5924e91700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59200bfbe0 0x7f59201480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.272+0000 7f5924e91700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592009f560 0x7f59201485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.272+0000 7f5924e91700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592014d110 con 0x7f59200bf800 2020-11-20T12:11:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.272+0000 7f5924e91700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592014c9d0 con 0x7f59200bfbe0 2020-11-20T12:11:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.272+0000 7f5924e91700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5920148aa0 con 0x7f592009f560 2020-11-20T12:11:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.272+0000 7f591effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59200bfbe0 0x7f59201480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.272+0000 7f591effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59200bfbe0 0x7f59201480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46522/0 (socket says 172.21.15.187:46522) 2020-11-20T12:11:56.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.272+0000 7f591effd700 1 -- 172.21.15.187:0/168126436 learned_addr learned my addr 172.21.15.187:0/168126436 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:56.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.273+0000 7f591ffff700 1 --2- 172.21.15.187:0/168126436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592009f560 0x7f59201485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.273+0000 7f591effd700 1 -- 172.21.15.187:0/168126436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592009f560 msgr2=0x7f59201485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:56.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.273+0000 7f591effd700 1 --2- 172.21.15.187:0/168126436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592009f560 0x7f59201485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.273+0000 7f591effd700 1 -- 172.21.15.187:0/168126436 >> v1:172.21.15.5:6789/0 conn(0x7f59200bf800 legacy=0x7f592014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:56.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.273+0000 7f591effd700 1 -- 172.21.15.187:0/168126436 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592014cbd0 con 0x7f59200bfbe0 2020-11-20T12:11:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.274+0000 7f591effd700 1 --2- 172.21.15.187:0/168126436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200bfbe0 0x7f59201480c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f5914006fd0 tx=0x7f5914006eb0).ready entity=mon.1 client_cookie=e308fb6d39452f86 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.274+0000 7f591e7fc700 1 -- 172.21.15.187:0/168126436 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914009920 con 0x7f59200bfbe0 2020-11-20T12:11:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.274+0000 7f591e7fc700 1 -- 172.21.15.187:0/168126436 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5914009a80 con 0x7f59200bfbe0 2020-11-20T12:11:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.274+0000 7f591e7fc700 1 -- 172.21.15.187:0/168126436 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914009e00 con 0x7f59200bfbe0 2020-11-20T12:11:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.275+0000 7f5924e91700 1 -- 172.21.15.187:0/168126436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200bfbe0 msgr2=0x7f59201480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.275+0000 7f5924e91700 1 --2- 172.21.15.187:0/168126436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200bfbe0 0x7f59201480c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f5914006fd0 tx=0x7f5914006eb0).stop 2020-11-20T12:11:56.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.275+0000 7f5924e91700 1 -- 172.21.15.187:0/168126436 shutdown_connections 2020-11-20T12:11:56.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.275+0000 7f5924e91700 1 --2- 172.21.15.187:0/168126436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592009f560 0x7f59201485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.275+0000 7f5924e91700 1 --2- 172.21.15.187:0/168126436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59200bfbe0 0x7f59201480c0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.275+0000 7f5924e91700 1 -- 172.21.15.187:0/168126436 >> 172.21.15.187:0/168126436 conn(0x7f5920068370 msgr2=0x7f59200bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.275+0000 7f5924e91700 1 -- 172.21.15.187:0/168126436 shutdown_connections 2020-11-20T12:11:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.275+0000 7f5924e91700 1 -- 172.21.15.187:0/168126436 wait complete. 2020-11-20T12:11:56.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.277+0000 7f5924e91700 1 Processor -- start 2020-11-20T12:11:56.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.277+0000 7f5924e91700 1 -- start start 2020-11-20T12:11:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f5924e91700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f592009f560 0x7f5920142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f5924e91700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200bf800 0x7f5920158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f5924e91700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201412f0 0x7f5920146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f5924e91700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592013f0e0 con 0x7f59200bf800 2020-11-20T12:11:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f5924e91700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920158d20 con 0x7f592009f560 2020-11-20T12:11:56.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f5924e91700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592013f310 con 0x7f59201412f0 2020-11-20T12:11:56.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f591f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f592009f560 0x7f5920142980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f591f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f592009f560 0x7f5920142980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46526/0 (socket says 172.21.15.187:46526) 2020-11-20T12:11:56.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f591f7fe700 1 -- 172.21.15.187:0/1389573216 learned_addr learned my addr 172.21.15.187:0/1389573216 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f591f7fe700 1 -- 172.21.15.187:0/1389573216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201412f0 msgr2=0x7f5920146d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f591f7fe700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201412f0 0x7f5920146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f591f7fe700 1 -- 172.21.15.187:0/1389573216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200bf800 msgr2=0x7f5920158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f591f7fe700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200bf800 0x7f5920158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.278+0000 7f591f7fe700 1 -- 172.21.15.187:0/1389573216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592013f5b0 con 0x7f592009f560 2020-11-20T12:11:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.279+0000 7f591f7fe700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f592009f560 0x7f5920142980 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f59100096e0 tx=0x7f5910006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.279+0000 7f58ff7fe700 1 -- 172.21.15.187:0/1389573216 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591000d560 con 0x7f592009f560 2020-11-20T12:11:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.279+0000 7f58ff7fe700 1 -- 172.21.15.187:0/1389573216 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5910000e90 con 0x7f592009f560 2020-11-20T12:11:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.279+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592013f850 con 0x7f592009f560 2020-11-20T12:11:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.279+0000 7f58ff7fe700 1 -- 172.21.15.187:0/1389573216 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591000d910 con 0x7f592009f560 2020-11-20T12:11:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.279+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592013d8a0 con 0x7f592009f560 2020-11-20T12:11:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.280+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5920080050 con 0x7f592009f560 2020-11-20T12:11:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.281+0000 7f58ff7fe700 1 -- 172.21.15.187:0/1389573216 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f591000f030 con 0x7f592009f560 2020-11-20T12:11:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.281+0000 7f58ff7fe700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f8056d20 0x7f58f80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.281+0000 7f58ff7fe700 1 -- 172.21.15.187:0/1389573216 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f591006fcb0 con 0x7f592009f560 2020-11-20T12:11:56.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.282+0000 7f591effd700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f8056d20 0x7f58f80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.282+0000 7f591effd700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f8056d20 0x7f58f80591e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f59200bcd30 tx=0x7f5914015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.285+0000 7f58ff7fe700 1 -- 172.21.15.187:0/1389573216 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5910044fd0 con 0x7f592009f560 2020-11-20T12:11:56.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.470+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f592013f9e0 con 0x7f592009f560 2020-11-20T12:11:56.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.471+0000 7f58ff7fe700 1 -- 172.21.15.187:0/1389573216 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v72) v1 ==== 110+0+10 (secure 0 0 0) 0x7f59100485f0 con 0x7f592009f560 2020-11-20T12:11:56.474 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:11:56.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.473+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f8056d20 msgr2=0x7f58f80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.473+0000 7f5924e91700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f8056d20 0x7f58f80591e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f59200bcd30 tx=0x7f5914015000).stop 2020-11-20T12:11:56.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.473+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f592009f560 msgr2=0x7f5920142980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.473+0000 7f5924e91700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f592009f560 0x7f5920142980 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f59100096e0 tx=0x7f5910006d70).stop 2020-11-20T12:11:56.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.473+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 shutdown_connections 2020-11-20T12:11:56.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.473+0000 7f5924e91700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f8056d20 0x7f58f80591e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.474+0000 7f5924e91700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59201412f0 0x7f5920146d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.474+0000 7f5924e91700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f592009f560 0x7f5920142980 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.474+0000 7f5924e91700 1 --2- 172.21.15.187:0/1389573216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200bf800 0x7f5920158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.474+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 >> 172.21.15.187:0/1389573216 conn(0x7f5920068370 msgr2=0x7f5920147e90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:56.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.474+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 shutdown_connections 2020-11-20T12:11:56.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.474+0000 7f5924e91700 1 -- 172.21.15.187:0/1389573216 wait complete. 2020-11-20T12:11:56.497 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:11:56.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.742+0000 7ff0e7fff700 1 Processor -- start 2020-11-20T12:11:56.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.743+0000 7ff0e7fff700 1 -- start start 2020-11-20T12:11:56.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.744+0000 7ff0e7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff0e81484d0 0x7ff0e8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.744+0000 7ff0e7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0e80622c0 0x7ff0e80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.744+0000 7ff0e7fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e80b1300 con 0x7ff0e8066f70 2020-11-20T12:11:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.744+0000 7ff0e7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e8067730 con 0x7ff0e81484d0 2020-11-20T12:11:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.744+0000 7ff0e7fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e80b15a0 con 0x7ff0e80622c0 2020-11-20T12:11:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.744+0000 7ff0e67fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0e80622c0 0x7ff0e80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.744+0000 7ff0e67fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0e80622c0 0x7ff0e80b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46536/0 (socket says 172.21.15.187:46536) 2020-11-20T12:11:56.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.744+0000 7ff0e67fc700 1 -- 172.21.15.187:0/523529248 learned_addr learned my addr 172.21.15.187:0/523529248 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:56.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e6ffd700 1 --2- 172.21.15.187:0/523529248 >> v2:172.21.15.5:3300/0 conn(0x7ff0e81484d0 0x7ff0e8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e67fc700 1 -- 172.21.15.187:0/523529248 >> v2:172.21.15.5:3300/0 conn(0x7ff0e81484d0 msgr2=0x7ff0e8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:56.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e67fc700 1 --2- 172.21.15.187:0/523529248 >> v2:172.21.15.5:3300/0 conn(0x7ff0e81484d0 0x7ff0e8061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e67fc700 1 -- 172.21.15.187:0/523529248 >> v1:172.21.15.5:6789/0 conn(0x7ff0e8066f70 legacy=0x7ff0e80b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:56.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e67fc700 1 -- 172.21.15.187:0/523529248 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80b1840 con 0x7ff0e80622c0 2020-11-20T12:11:56.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e6ffd700 1 --2- 172.21.15.187:0/523529248 >> v2:172.21.15.5:3300/0 conn(0x7ff0e81484d0 0x7ff0e8061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:56.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e67fc700 1 --2- 172.21.15.187:0/523529248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b5640 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7ff0e8068600 tx=0x7ff0dc006eb0).ready entity=mon.1 client_cookie=6a9629b7a6c2ab1e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e5ffb700 1 -- 172.21.15.187:0/523529248 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc009890 con 0x7ff0e80622c0 2020-11-20T12:11:56.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e5ffb700 1 -- 172.21.15.187:0/523529248 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc0099f0 con 0x7ff0e80622c0 2020-11-20T12:11:56.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.745+0000 7ff0e5ffb700 1 -- 172.21.15.187:0/523529248 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc009d70 con 0x7ff0e80622c0 2020-11-20T12:11:56.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.746+0000 7ff0e7fff700 1 -- 172.21.15.187:0/523529248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 msgr2=0x7ff0e80b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.746+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/523529248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b5640 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7ff0e8068600 tx=0x7ff0dc006eb0).stop 2020-11-20T12:11:56.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.746+0000 7ff0e7fff700 1 -- 172.21.15.187:0/523529248 shutdown_connections 2020-11-20T12:11:56.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.746+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/523529248 >> v2:172.21.15.5:3300/0 conn(0x7ff0e81484d0 0x7ff0e8061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.746+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/523529248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b5640 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.746+0000 7ff0e7fff700 1 -- 172.21.15.187:0/523529248 >> 172.21.15.187:0/523529248 conn(0x7ff0e80bab30 msgr2=0x7ff0e80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:56.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.746+0000 7ff0e7fff700 1 -- 172.21.15.187:0/523529248 shutdown_connections 2020-11-20T12:11:56.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.746+0000 7ff0e7fff700 1 -- 172.21.15.187:0/523529248 wait complete. 2020-11-20T12:11:56.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.748+0000 7ff0e7fff700 1 Processor -- start 2020-11-20T12:11:56.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.748+0000 7ff0e7fff700 1 -- start start 2020-11-20T12:11:56.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.749+0000 7ff0e7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.749+0000 7ff0e7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e8066f70 0x7ff0e8137d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.749+0000 7ff0e7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e81484d0 0x7ff0e809b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.749+0000 7ff0e7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80b3630 con 0x7ff0e81484d0 2020-11-20T12:11:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.749+0000 7ff0e7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80b38d0 con 0x7ff0e80622c0 2020-11-20T12:11:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.749+0000 7ff0e7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80b3b70 con 0x7ff0e8066f70 2020-11-20T12:11:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46540/0 (socket says 172.21.15.187:46540) 2020-11-20T12:11:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 -- 172.21.15.187:0/1149094764 learned_addr learned my addr 172.21.15.187:0/1149094764 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 -- 172.21.15.187:0/1149094764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e8066f70 msgr2=0x7ff0e8137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e8066f70 0x7ff0e8137d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 -- 172.21.15.187:0/1149094764 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e81484d0 msgr2=0x7ff0e809b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e81484d0 0x7ff0e809b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 -- 172.21.15.187:0/1149094764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e809ad90 con 0x7ff0e80622c0 2020-11-20T12:11:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0e6ffd700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b4010 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7ff0d800c8f0 tx=0x7ff0d8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.750+0000 7ff0ceffd700 1 -- 172.21.15.187:0/1149094764 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80077a0 con 0x7ff0e80622c0 2020-11-20T12:11:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.751+0000 7ff0ceffd700 1 -- 172.21.15.187:0/1149094764 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d8007900 con 0x7ff0e80622c0 2020-11-20T12:11:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.751+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e8138240 con 0x7ff0e80622c0 2020-11-20T12:11:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.751+0000 7ff0ceffd700 1 -- 172.21.15.187:0/1149094764 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8007c80 con 0x7ff0e80622c0 2020-11-20T12:11:56.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.751+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e809b760 con 0x7ff0e80622c0 2020-11-20T12:11:56.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.752+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e8061a20 con 0x7ff0e80622c0 2020-11-20T12:11:56.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.752+0000 7ff0ceffd700 1 -- 172.21.15.187:0/1149094764 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff0d8011620 con 0x7ff0e80622c0 2020-11-20T12:11:56.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.753+0000 7ff0ceffd700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0c4056d20 0x7ff0c40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.753+0000 7ff0ceffd700 1 -- 172.21.15.187:0/1149094764 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff0d8070230 con 0x7ff0e80622c0 2020-11-20T12:11:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.754+0000 7ff0e67fc700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0c4056d20 0x7ff0c40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.754+0000 7ff0e67fc700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0c4056d20 0x7ff0c40591e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7ff0e8068600 tx=0x7ff0dc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.757+0000 7ff0ceffd700 1 -- 172.21.15.187:0/1149094764 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff0d8045630 con 0x7ff0e80622c0 2020-11-20T12:11:56.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.943+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff0e81488b0 con 0x7ff0e80622c0 2020-11-20T12:11:56.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.943+0000 7ff0ceffd700 1 -- 172.21.15.187:0/1149094764 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff0d8048c50 con 0x7ff0e80622c0 2020-11-20T12:11:56.947 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:11:56.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0c4056d20 msgr2=0x7ff0c40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0c4056d20 0x7ff0c40591e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7ff0e8068600 tx=0x7ff0dc015000).stop 2020-11-20T12:11:56.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 msgr2=0x7ff0e80b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b4010 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7ff0d800c8f0 tx=0x7ff0d8008040).stop 2020-11-20T12:11:56.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 shutdown_connections 2020-11-20T12:11:56.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0c4056d20 0x7ff0c40591e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e81484d0 0x7ff0e809b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0e80622c0 0x7ff0e80b4010 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 --2- 172.21.15.187:0/1149094764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e8066f70 0x7ff0e8137d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:56.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 >> 172.21.15.187:0/1149094764 conn(0x7ff0e80bab30 msgr2=0x7ff0e8066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:56.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 shutdown_connections 2020-11-20T12:11:56.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:56.946+0000 7ff0e7fff700 1 -- 172.21.15.187:0/1149094764 wait complete. 2020-11-20T12:11:56.975 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:11:57.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.223+0000 7f04e6e0e700 1 Processor -- start 2020-11-20T12:11:57.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.224+0000 7f04e6e0e700 1 -- start start 2020-11-20T12:11:57.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e6e0e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f04e00b57b0 0x7f04e0068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e6e0e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e01482d0 0x7f04e0062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e6e0e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f04e0062b50 0x7f04e0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e6e0e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e0144b80 con 0x7f04e0062b50 2020-11-20T12:11:57.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e6e0e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e01489e0 con 0x7f04e00b57b0 2020-11-20T12:11:57.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e6e0e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e0148c80 con 0x7f04e01482d0 2020-11-20T12:11:57.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e5e0c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f04e00b57b0 0x7f04e0068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e5e0c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f04e00b57b0 0x7f04e0068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46548/0 (socket says 172.21.15.187:46548) 2020-11-20T12:11:57.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.225+0000 7f04e5e0c700 1 -- 172.21.15.187:0/2341536271 learned_addr learned my addr 172.21.15.187:0/2341536271 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:57.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e560b700 1 --2- 172.21.15.187:0/2341536271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e01482d0 0x7f04e0062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e5e0c700 1 -- 172.21.15.187:0/2341536271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e01482d0 msgr2=0x7f04e0062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:57.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e5e0c700 1 --2- 172.21.15.187:0/2341536271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e01482d0 0x7f04e0062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e5e0c700 1 -- 172.21.15.187:0/2341536271 >> v2:172.21.15.5:3300/0 conn(0x7f04e0062b50 msgr2=0x7f04e0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:57.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e660d700 1 --2- 172.21.15.187:0/2341536271 >> v2:172.21.15.5:3300/0 conn(0x7f04e0062b50 0x7f04e0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e5e0c700 1 --2- 172.21.15.187:0/2341536271 >> v2:172.21.15.5:3300/0 conn(0x7f04e0062b50 0x7f04e0065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e5e0c700 1 -- 172.21.15.187:0/2341536271 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e0148f20 con 0x7f04e00b57b0 2020-11-20T12:11:57.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e560b700 1 --2- 172.21.15.187:0/2341536271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e01482d0 0x7f04e0062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:57.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e660d700 1 --2- 172.21.15.187:0/2341536271 >> v2:172.21.15.5:3300/0 conn(0x7f04e0062b50 0x7f04e0065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:57.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.226+0000 7f04e5e0c700 1 --2- 172.21.15.187:0/2341536271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e0068230 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f04e00b85e0 tx=0x7f04dc006eb0).ready entity=mon.1 client_cookie=f6ae168563dca242 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:57.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e4e0a700 1 -- 172.21.15.187:0/2341536271 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc0098d0 con 0x7f04e00b57b0 2020-11-20T12:11:57.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e4e0a700 1 -- 172.21.15.187:0/2341536271 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc009a30 con 0x7f04e00b57b0 2020-11-20T12:11:57.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e4e0a700 1 -- 172.21.15.187:0/2341536271 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc009db0 con 0x7f04e00b57b0 2020-11-20T12:11:57.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e6e0e700 1 -- 172.21.15.187:0/2341536271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 msgr2=0x7f04e0068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/2341536271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e0068230 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f04e00b85e0 tx=0x7f04dc006eb0).stop 2020-11-20T12:11:57.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e6e0e700 1 -- 172.21.15.187:0/2341536271 shutdown_connections 2020-11-20T12:11:57.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/2341536271 >> v2:172.21.15.5:3300/0 conn(0x7f04e0062b50 0x7f04e0065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/2341536271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e0068230 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/2341536271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e01482d0 0x7f04e0062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.227+0000 7f04e6e0e700 1 -- 172.21.15.187:0/2341536271 >> 172.21.15.187:0/2341536271 conn(0x7f04e00bab30 msgr2=0x7f04e00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:57.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.228+0000 7f04e6e0e700 1 -- 172.21.15.187:0/2341536271 shutdown_connections 2020-11-20T12:11:57.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.228+0000 7f04e6e0e700 1 -- 172.21.15.187:0/2341536271 wait complete. 2020-11-20T12:11:57.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.229+0000 7f04e6e0e700 1 Processor -- start 2020-11-20T12:11:57.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.230+0000 7f04e6e0e700 1 -- start start 2020-11-20T12:11:57.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e6e0e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e0062b50 0x7f04e015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e6e0e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e01434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e6e0e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04e01482d0 0x7f04e009f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e6e0e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e009ec30 con 0x7f04e01482d0 2020-11-20T12:11:57.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e6e0e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e009ee60 con 0x7f04e00b57b0 2020-11-20T12:11:57.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e6e0e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e009f100 con 0x7f04e0062b50 2020-11-20T12:11:57.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e560b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e01434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e560b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e01434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46556/0 (socket says 172.21.15.187:46556) 2020-11-20T12:11:57.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.231+0000 7f04e560b700 1 -- 172.21.15.187:0/1936680489 learned_addr learned my addr 172.21.15.187:0/1936680489 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:57.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e660d700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04e01482d0 0x7f04e009f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e5e0c700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e0062b50 0x7f04e015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e560b700 1 -- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e0062b50 msgr2=0x7f04e015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e560b700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e0062b50 0x7f04e015bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e560b700 1 -- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04e01482d0 msgr2=0x7f04e009f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e560b700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04e01482d0 0x7f04e009f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e560b700 1 -- 172.21.15.187:0/1936680489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e01439e0 con 0x7f04e00b57b0 2020-11-20T12:11:57.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e5e0c700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e0062b50 0x7f04e015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:57.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04e560b700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e01434f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f04d000cb50 tx=0x7f04d0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:57.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04cdffb700 1 -- 172.21.15.187:0/1936680489 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d00077d0 con 0x7f04e00b57b0 2020-11-20T12:11:57.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04cdffb700 1 -- 172.21.15.187:0/1936680489 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d0007930 con 0x7f04e00b57b0 2020-11-20T12:11:57.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.232+0000 7f04cdffb700 1 -- 172.21.15.187:0/1936680489 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0007cb0 con 0x7f04e00b57b0 2020-11-20T12:11:57.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.233+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e009fa90 con 0x7f04e00b57b0 2020-11-20T12:11:57.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.233+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e00a0360 con 0x7f04e00b57b0 2020-11-20T12:11:57.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.234+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e015c560 con 0x7f04e00b57b0 2020-11-20T12:11:57.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.234+0000 7f04cdffb700 1 -- 172.21.15.187:0/1936680489 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f04d0011620 con 0x7f04e00b57b0 2020-11-20T12:11:57.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.235+0000 7f04cdffb700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04b8056d20 0x7f04b80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.235+0000 7f04cdffb700 1 -- 172.21.15.187:0/1936680489 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f04d0070290 con 0x7f04e00b57b0 2020-11-20T12:11:57.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.235+0000 7f04e5e0c700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04b8056d20 0x7f04b80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.236+0000 7f04e5e0c700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04b8056d20 0x7f04b80591e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f04dc00b1b0 tx=0x7f04dc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:57.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.239+0000 7f04cdffb700 1 -- 172.21.15.187:0/1936680489 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f04e015c560 con 0x7f04e00b57b0 2020-11-20T12:11:57.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.427+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f04e015c560 con 0x7f04e00b57b0 2020-11-20T12:11:57.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.427+0000 7f04cdffb700 1 -- 172.21.15.187:0/1936680489 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+10 (secure 0 0 0) 0x7f04e015c560 con 0x7f04e00b57b0 2020-11-20T12:11:57.431 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:11:57.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04b8056d20 msgr2=0x7f04b80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04b8056d20 0x7f04b80591e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f04dc00b1b0 tx=0x7f04dc015000).stop 2020-11-20T12:11:57.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 msgr2=0x7f04e01434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e01434f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f04d000cb50 tx=0x7f04d0008040).stop 2020-11-20T12:11:57.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 shutdown_connections 2020-11-20T12:11:57.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f04b8056d20 0x7f04b80591e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04e01482d0 0x7f04e009f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f04e0062b50 0x7f04e015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 --2- 172.21.15.187:0/1936680489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04e00b57b0 0x7f04e01434f0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 >> 172.21.15.187:0/1936680489 conn(0x7f04e00bab30 msgr2=0x7f04e00b7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:57.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 shutdown_connections 2020-11-20T12:11:57.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.430+0000 7f04e6e0e700 1 -- 172.21.15.187:0/1936680489 wait complete. 2020-11-20T12:11:57.458 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:11:57.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.700+0000 7f3fb6365700 1 Processor -- start 2020-11-20T12:11:57.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.701+0000 7f3fb6365700 1 -- start start 2020-11-20T12:11:57.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.702+0000 7f3fb6365700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb0065f90 0x7f3fb0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.702+0000 7f3fb6365700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3fb009ca90 0x7f3fb009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.702+0000 7f3fb6365700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb00b13e0 con 0x7f3fb0065190 2020-11-20T12:11:57.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.702+0000 7f3fb6365700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb014ab30 con 0x7f3fb009ca90 2020-11-20T12:11:57.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.702+0000 7f3fb6365700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb0062c10 con 0x7f3fb0065f90 2020-11-20T12:11:57.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.702+0000 7f3fb5363700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3fb0065190 0x7f3fb0143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43168/0 (socket says 172.21.15.187:43168) 2020-11-20T12:11:57.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.702+0000 7f3fb5363700 1 -- 172.21.15.187:0/2612279061 learned_addr learned my addr 172.21.15.187:0/2612279061 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:57.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.703+0000 7f3fb5b64700 1 --2- 172.21.15.187:0/2612279061 >> v2:172.21.15.187:3300/0 conn(0x7f3fb009ca90 0x7f3fb009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.703+0000 7f3f9ffff700 1 -- 172.21.15.187:0/2612279061 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056330678 0 0) 0x7f3fb00b13e0 con 0x7f3fb0065190 2020-11-20T12:11:57.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.703+0000 7f3f9ffff700 1 -- 172.21.15.187:0/2612279061 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f98003240 con 0x7f3fb0065190 2020-11-20T12:11:57.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.703+0000 7f3fb4b62700 1 --2- 172.21.15.187:0/2612279061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb0065f90 0x7f3fb0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.703+0000 7f3fb5b64700 1 -- 172.21.15.187:0/2612279061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb0065f90 msgr2=0x7f3fb0065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:57.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.704+0000 7f3fb5b64700 1 --2- 172.21.15.187:0/2612279061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb0065f90 0x7f3fb0065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.704+0000 7f3fb5b64700 1 -- 172.21.15.187:0/2612279061 >> v1:172.21.15.187:6789/0 conn(0x7f3fb0065190 legacy=0x7f3fb0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.704+0000 7f3f9ffff700 1 -- 172.21.15.187:0/2612279061 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3260410562 0 0) 0x7f3f98003240 con 0x7f3fb0065190 2020-11-20T12:11:57.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.704+0000 7f3fb5b64700 1 -- 172.21.15.187:0/2612279061 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb00b1640 con 0x7f3fb009ca90 2020-11-20T12:11:57.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.704+0000 7f3fb5b64700 1 --2- 172.21.15.187:0/2612279061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb009ca90 0x7f3fb009cf20 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f3fb014ac70 tx=0x7f3fa80091d0).ready entity=mon.1 client_cookie=80bf73a62f87af97 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:57.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.704+0000 7f3f9ffff700 1 -- 172.21.15.187:0/2612279061 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa800bad0 con 0x7f3fb009ca90 2020-11-20T12:11:57.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.704+0000 7f3f9ffff700 1 -- 172.21.15.187:0/2612279061 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa800bc30 con 0x7f3fb009ca90 2020-11-20T12:11:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.704+0000 7f3f9ffff700 1 -- 172.21.15.187:0/2612279061 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8011400 con 0x7f3fb009ca90 2020-11-20T12:11:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.705+0000 7f3fb6365700 1 -- 172.21.15.187:0/2612279061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb009ca90 msgr2=0x7f3fb009cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.705+0000 7f3fb6365700 1 --2- 172.21.15.187:0/2612279061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb009ca90 0x7f3fb009cf20 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f3fb014ac70 tx=0x7f3fa80091d0).stop 2020-11-20T12:11:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.705+0000 7f3fb6365700 1 -- 172.21.15.187:0/2612279061 shutdown_connections 2020-11-20T12:11:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.705+0000 7f3fb6365700 1 --2- 172.21.15.187:0/2612279061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb009ca90 0x7f3fb009cf20 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.705+0000 7f3fb6365700 1 --2- 172.21.15.187:0/2612279061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb0065f90 0x7f3fb0065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.705+0000 7f3fb6365700 1 -- 172.21.15.187:0/2612279061 >> 172.21.15.187:0/2612279061 conn(0x7f3fb00bab30 msgr2=0x7f3fb00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:57.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.705+0000 7f3fb6365700 1 -- 172.21.15.187:0/2612279061 shutdown_connections 2020-11-20T12:11:57.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.705+0000 7f3fb6365700 1 -- 172.21.15.187:0/2612279061 wait complete. 2020-11-20T12:11:57.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.707+0000 7f3fb6365700 1 Processor -- start 2020-11-20T12:11:57.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.707+0000 7f3fb6365700 1 -- start start 2020-11-20T12:11:57.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb6365700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fb0065190 0x7f3fb0133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb6365700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb0065f90 0x7f3fb0134370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb6365700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb009ca90 0x7f3fb00b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb6365700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb00b4a80 con 0x7f3fb0065190 2020-11-20T12:11:57.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb6365700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb00b47e0 con 0x7f3fb0065f90 2020-11-20T12:11:57.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb6365700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb009eb10 con 0x7f3fb009ca90 2020-11-20T12:11:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb4b62700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb0065f90 0x7f3fb0134370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb4b62700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb0065f90 0x7f3fb0134370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46568/0 (socket says 172.21.15.187:46568) 2020-11-20T12:11:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb4b62700 1 -- 172.21.15.187:0/464208721 learned_addr learned my addr 172.21.15.187:0/464208721 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb4b62700 1 -- 172.21.15.187:0/464208721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb009ca90 msgr2=0x7f3fb00b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb4b62700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb009ca90 0x7f3fb00b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb4b62700 1 -- 172.21.15.187:0/464208721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fb0065190 msgr2=0x7f3fb0133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb4b62700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fb0065190 0x7f3fb0133e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.708+0000 7f3fb4b62700 1 -- 172.21.15.187:0/464208721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb00a0200 con 0x7f3fb0065f90 2020-11-20T12:11:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.709+0000 7f3fb4b62700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb0065f90 0x7f3fb0134370 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f3fa000db60 tx=0x7f3fa0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.709+0000 7f3f9d7fa700 1 -- 172.21.15.187:0/464208721 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0009790 con 0x7f3fb0065f90 2020-11-20T12:11:57.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.709+0000 7f3f9d7fa700 1 -- 172.21.15.187:0/464208721 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa00098f0 con 0x7f3fb0065f90 2020-11-20T12:11:57.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.709+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb009efb0 con 0x7f3fb0065f90 2020-11-20T12:11:57.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.709+0000 7f3f9d7fa700 1 -- 172.21.15.187:0/464208721 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0009c90 con 0x7f3fb0065f90 2020-11-20T12:11:57.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.709+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb009f620 con 0x7f3fb0065f90 2020-11-20T12:11:57.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.710+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb009ecd0 con 0x7f3fb0065f90 2020-11-20T12:11:57.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.711+0000 7f3f9d7fa700 1 -- 172.21.15.187:0/464208721 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3fa0013480 con 0x7f3fb0065f90 2020-11-20T12:11:57.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.711+0000 7f3f9d7fa700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f88056d20 0x7f3f880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:57.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.711+0000 7f3f9d7fa700 1 -- 172.21.15.187:0/464208721 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3fa00706d0 con 0x7f3fb0065f90 2020-11-20T12:11:57.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.712+0000 7f3fb5363700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f88056d20 0x7f3f880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:57.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.712+0000 7f3fb5363700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f88056d20 0x7f3f880591e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f3fac001e80 tx=0x7f3fac001340).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:57.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.715+0000 7f3f9d7fa700 1 -- 172.21.15.187:0/464208721 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3fa0045960 con 0x7f3fb0065f90 2020-11-20T12:11:57.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.900+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3fb0061a20 con 0x7f3fb0065f90 2020-11-20T12:11:57.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.901+0000 7f3f9d7fa700 1 -- 172.21.15.187:0/464208721 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3fa0048f80 con 0x7f3fb0065f90 2020-11-20T12:11:57.904 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:11:57.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.903+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f88056d20 msgr2=0x7f3f880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f88056d20 0x7f3f880591e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f3fac001e80 tx=0x7f3fac001340).stop 2020-11-20T12:11:57.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb0065f90 msgr2=0x7f3fb0134370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:57.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb0065f90 0x7f3fb0134370 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f3fa000db60 tx=0x7f3fa0008040).stop 2020-11-20T12:11:57.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 shutdown_connections 2020-11-20T12:11:57.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f88056d20 0x7f3f880591e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fb0065190 0x7f3fb0133e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:57.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fb009ca90 0x7f3fb00b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:57.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 --2- 172.21.15.187:0/464208721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb0065f90 0x7f3fb0134370 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:57.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 >> 172.21.15.187:0/464208721 conn(0x7f3fb00bab30 msgr2=0x7f3fb00ba070 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:57.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 shutdown_connections 2020-11-20T12:11:57.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:57.904+0000 7f3fb6365700 1 -- 172.21.15.187:0/464208721 wait complete. 2020-11-20T12:11:57.929 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:58.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.169+0000 7eff6cbdf700 1 Processor -- start 2020-11-20T12:11:58.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.169+0000 7eff6cbdf700 1 -- start start 2020-11-20T12:11:58.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff6cbdf700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7eff680b1590 0x7eff680656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff6cbdf700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7eff680b7300 0x7eff68143920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff6cbdf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff68149dd0 con 0x7eff680652e0 2020-11-20T12:11:58.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff6cbdf700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6814a070 con 0x7eff680b7300 2020-11-20T12:11:58.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff6cbdf700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6814a310 con 0x7eff680b1590 2020-11-20T12:11:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff5dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7eff680b1590 0x7eff680656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff5dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7eff680b1590 0x7eff680656c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46578/0 (socket says 172.21.15.187:46578) 2020-11-20T12:11:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff5dd9b700 1 -- 172.21.15.187:0/1660108869 learned_addr learned my addr 172.21.15.187:0/1660108869 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.171+0000 7eff66d9d700 1 --2- 172.21.15.187:0/1660108869 >> v2:172.21.15.5:3300/0 conn(0x7eff680b7300 0x7eff68143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.172+0000 7eff5dd9b700 1 -- 172.21.15.187:0/1660108869 >> v2:172.21.15.5:3300/0 conn(0x7eff680b7300 msgr2=0x7eff68143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.172+0000 7eff5dd9b700 1 --2- 172.21.15.187:0/1660108869 >> v2:172.21.15.5:3300/0 conn(0x7eff680b7300 0x7eff68143920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.172+0000 7eff5dd9b700 1 -- 172.21.15.187:0/1660108869 >> v1:172.21.15.5:6789/0 conn(0x7eff680652e0 legacy=0x7eff6814a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.172+0000 7eff5dd9b700 1 -- 172.21.15.187:0/1660108869 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff680b5380 con 0x7eff680b1590 2020-11-20T12:11:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.173+0000 7eff5dd9b700 1 --2- 172.21.15.187:0/1660108869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680b1590 0x7eff680656c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7eff68149d30 tx=0x7eff50006e50).ready entity=mon.1 client_cookie=cf7268838290e5ca server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.173+0000 7eff65d9b700 1 -- 172.21.15.187:0/1660108869 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50006030 con 0x7eff680b1590 2020-11-20T12:11:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.173+0000 7eff65d9b700 1 -- 172.21.15.187:0/1660108869 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff50006190 con 0x7eff680b1590 2020-11-20T12:11:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.173+0000 7eff65d9b700 1 -- 172.21.15.187:0/1660108869 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50006510 con 0x7eff680b1590 2020-11-20T12:11:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.174+0000 7eff6cbdf700 1 -- 172.21.15.187:0/1660108869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680b1590 msgr2=0x7eff680656c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.174+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/1660108869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680b1590 0x7eff680656c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7eff68149d30 tx=0x7eff50006e50).stop 2020-11-20T12:11:58.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.174+0000 7eff6cbdf700 1 -- 172.21.15.187:0/1660108869 shutdown_connections 2020-11-20T12:11:58.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.174+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/1660108869 >> v2:172.21.15.5:3300/0 conn(0x7eff680b7300 0x7eff68143920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.174+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/1660108869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680b1590 0x7eff680656c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.174+0000 7eff6cbdf700 1 -- 172.21.15.187:0/1660108869 >> 172.21.15.187:0/1660108869 conn(0x7eff680bacd0 msgr2=0x7eff680bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.174+0000 7eff6cbdf700 1 -- 172.21.15.187:0/1660108869 shutdown_connections 2020-11-20T12:11:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.174+0000 7eff6cbdf700 1 -- 172.21.15.187:0/1660108869 wait complete. 2020-11-20T12:11:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.175+0000 7eff6cbdf700 1 Processor -- start 2020-11-20T12:11:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.176+0000 7eff6cbdf700 1 -- start start 2020-11-20T12:11:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.176+0000 7eff6cbdf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680652e0 0x7eff680b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.176+0000 7eff6cbdf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff680b1590 0x7eff68068160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.176+0000 7eff6cbdf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff680b7300 0x7eff6809c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.176+0000 7eff6cbdf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6815c580 con 0x7eff680b1590 2020-11-20T12:11:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.176+0000 7eff6cbdf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff68158220 con 0x7eff680652e0 2020-11-20T12:11:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.176+0000 7eff6cbdf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff681584c0 con 0x7eff680b7300 2020-11-20T12:11:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.177+0000 7eff6659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680652e0 0x7eff680b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.177+0000 7eff6659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680652e0 0x7eff680b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46584/0 (socket says 172.21.15.187:46584) 2020-11-20T12:11:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.177+0000 7eff6659c700 1 -- 172.21.15.187:0/657012476 learned_addr learned my addr 172.21.15.187:0/657012476 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.177+0000 7eff5dd9b700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff680b1590 0x7eff68068160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.177+0000 7eff66d9d700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff680b7300 0x7eff6809c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.177+0000 7eff6659c700 1 -- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff680b7300 msgr2=0x7eff6809c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.177+0000 7eff6659c700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff680b7300 0x7eff6809c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff6659c700 1 -- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff680b1590 msgr2=0x7eff68068160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff6659c700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff680b1590 0x7eff68068160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff6659c700 1 -- 172.21.15.187:0/657012476 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff68068650 con 0x7eff680652e0 2020-11-20T12:11:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff66d9d700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff680b7300 0x7eff6809c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:11:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff6659c700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680652e0 0x7eff680b40d0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7eff5800c710 tx=0x7eff58007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff5effd700 1 -- 172.21.15.187:0/657012476 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5800e560 con 0x7eff680652e0 2020-11-20T12:11:58.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff5effd700 1 -- 172.21.15.187:0/657012476 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5800e6c0 con 0x7eff680652e0 2020-11-20T12:11:58.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff680b3d50 con 0x7eff680652e0 2020-11-20T12:11:58.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff5effd700 1 -- 172.21.15.187:0/657012476 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5800ea40 con 0x7eff680652e0 2020-11-20T12:11:58.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.178+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6809c620 con 0x7eff680652e0 2020-11-20T12:11:58.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.179+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff6809c8d0 con 0x7eff680652e0 2020-11-20T12:11:58.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.180+0000 7eff5effd700 1 -- 172.21.15.187:0/657012476 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7eff58019030 con 0x7eff680652e0 2020-11-20T12:11:58.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.181+0000 7eff5effd700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff44056d20 0x7eff440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.181+0000 7eff5effd700 1 -- 172.21.15.187:0/657012476 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7eff5806f660 con 0x7eff680652e0 2020-11-20T12:11:58.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.181+0000 7eff5dd9b700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff44056d20 0x7eff440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.181+0000 7eff5dd9b700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff44056d20 0x7eff440591e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7eff68149d30 tx=0x7eff50015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:58.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.184+0000 7eff5effd700 1 -- 172.21.15.187:0/657012476 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7eff58044a60 con 0x7eff680652e0 2020-11-20T12:11:58.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.435+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff6815c380 con 0x7eff680652e0 2020-11-20T12:11:58.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.436+0000 7eff5effd700 1 -- 172.21.15.187:0/657012476 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2137 (secure 0 0 0) 0x7eff58017090 con 0x7eff680652e0 2020-11-20T12:11:58.440 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:58.440 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6835/3559036794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3559036794},{"type":"v1","addr":"172.21.15.187:6835","nonce":3559036794}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:10:23.312121+0000","modified":"2020-11-20T12:10:46.446562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":5,"addr":"172.21.15.5:6833/3088698963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3088698963},{"type":"v1","addr":"172.21.15.5:6833","nonce":3088698963}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2020-11-20T12:11:58.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.438+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff44056d20 msgr2=0x7eff440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.438+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff44056d20 0x7eff440591e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7eff68149d30 tx=0x7eff50015000).stop 2020-11-20T12:11:58.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.438+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680652e0 msgr2=0x7eff680b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.438+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680652e0 0x7eff680b40d0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7eff5800c710 tx=0x7eff58007c10).stop 2020-11-20T12:11:58.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.439+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 shutdown_connections 2020-11-20T12:11:58.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.439+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7eff44056d20 0x7eff440591e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.439+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7eff680b7300 0x7eff6809c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.439+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff680652e0 0x7eff680b40d0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.439+0000 7eff6cbdf700 1 --2- 172.21.15.187:0/657012476 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff680b1590 0x7eff68068160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.439+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 >> 172.21.15.187:0/657012476 conn(0x7eff680bacd0 msgr2=0x7eff6809fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:58.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.439+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 shutdown_connections 2020-11-20T12:11:58.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.439+0000 7eff6cbdf700 1 -- 172.21.15.187:0/657012476 wait complete. 2020-11-20T12:11:58.448 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 56 2020-11-20T12:11:58.464 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:11:58.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.728+0000 7f2c80d99700 1 Processor -- start 2020-11-20T12:11:58.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.728+0000 7f2c80d99700 1 -- start start 2020-11-20T12:11:58.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.730+0000 7f2c80d99700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c7c068230 0x7f2c7c1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.730+0000 7f2c80d99700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c144b80 con 0x7f2c7c0b57b0 2020-11-20T12:11:58.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.730+0000 7f2c80d99700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c1489e0 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.730+0000 7f2c80d99700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c1487c0 con 0x7f2c7c068230 2020-11-20T12:11:58.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.730+0000 7f2c73fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c7c068230 0x7f2c7c1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.730+0000 7f2c7bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2c7c0b5f50 0x7f2c7c0b6380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43200/0 (socket says 172.21.15.187:43200) 2020-11-20T12:11:58.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.730+0000 7f2c7bfff700 1 -- 172.21.15.187:0/3360552769 learned_addr learned my addr 172.21.15.187:0/3360552769 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:58.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.731+0000 7f2c73fff700 1 -- 172.21.15.187:0/3360552769 >> v1:172.21.15.187:6789/0 conn(0x7f2c7c0b5f50 legacy=0x7f2c7c0b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.731+0000 7f2c73fff700 1 -- 172.21.15.187:0/3360552769 >> v1:172.21.15.5:6789/0 conn(0x7f2c7c0b57b0 legacy=0x7f2c7c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:58.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.731+0000 7f2c73fff700 1 -- 172.21.15.187:0/3360552769 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c148c80 con 0x7f2c7c068230 2020-11-20T12:11:58.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.731+0000 7f2c73fff700 1 --2- 172.21.15.187:0/3360552769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c068230 0x7f2c7c1482d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f2c6400b9a0 tx=0x7f2c64006df0).ready entity=mon.1 client_cookie=dabe5d7f24e2b639 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:58.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.731+0000 7f2c7affd700 1 -- 172.21.15.187:0/3360552769 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64009770 con 0x7f2c7c068230 2020-11-20T12:11:58.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.731+0000 7f2c7affd700 1 -- 172.21.15.187:0/3360552769 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c640098d0 con 0x7f2c7c068230 2020-11-20T12:11:58.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.731+0000 7f2c7affd700 1 -- 172.21.15.187:0/3360552769 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64009c50 con 0x7f2c7c068230 2020-11-20T12:11:58.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.732+0000 7f2c80d99700 1 -- 172.21.15.187:0/3360552769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c068230 msgr2=0x7f2c7c1482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.732+0000 7f2c80d99700 1 --2- 172.21.15.187:0/3360552769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c068230 0x7f2c7c1482d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f2c6400b9a0 tx=0x7f2c64006df0).stop 2020-11-20T12:11:58.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.732+0000 7f2c80d99700 1 -- 172.21.15.187:0/3360552769 shutdown_connections 2020-11-20T12:11:58.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.732+0000 7f2c80d99700 1 --2- 172.21.15.187:0/3360552769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c068230 0x7f2c7c1482d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.732+0000 7f2c80d99700 1 -- 172.21.15.187:0/3360552769 >> 172.21.15.187:0/3360552769 conn(0x7f2c7c0bab30 msgr2=0x7f2c7c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:58.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.732+0000 7f2c80d99700 1 -- 172.21.15.187:0/3360552769 shutdown_connections 2020-11-20T12:11:58.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.732+0000 7f2c80d99700 1 -- 172.21.15.187:0/3360552769 wait complete. 2020-11-20T12:11:58.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.734+0000 7f2c80d99700 1 Processor -- start 2020-11-20T12:11:58.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.734+0000 7f2c80d99700 1 -- start start 2020-11-20T12:11:58.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.735+0000 7f2c80d99700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c7c068230 0x7f2c7c1466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.735+0000 7f2c80d99700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c7c0b57b0 0x7f2c7c157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.735+0000 7f2c80d99700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c0b5f50 0x7f2c7c15b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.735+0000 7f2c80d99700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c15c7a0 con 0x7f2c7c068230 2020-11-20T12:11:58.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.735+0000 7f2c80d99700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c146cb0 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.735+0000 7f2c80d99700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c146f50 con 0x7f2c7c0b57b0 2020-11-20T12:11:58.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c0b5f50 0x7f2c7c15b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c0b5f50 0x7f2c7c15b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46598/0 (socket says 172.21.15.187:46598) 2020-11-20T12:11:58.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 -- 172.21.15.187:0/2662931570 learned_addr learned my addr 172.21.15.187:0/2662931570 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:58.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 -- 172.21.15.187:0/2662931570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c7c0b57b0 msgr2=0x7f2c7c157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7b7fe700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c7c068230 0x7f2c7c1466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c7c0b57b0 0x7f2c7c157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:58.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 -- 172.21.15.187:0/2662931570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c7c068230 msgr2=0x7f2c7c1466f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c7c068230 0x7f2c7c1466f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 -- 172.21.15.187:0/2662931570 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c1471f0 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.736+0000 7f2c7bfff700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c0b5f50 0x7f2c7c15b860 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7f2c7400ca60 tx=0x7f2c74001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:58.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.737+0000 7f2c737fe700 1 -- 172.21.15.187:0/2662931570 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74005b80 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.737+0000 7f2c737fe700 1 -- 172.21.15.187:0/2662931570 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c74005ce0 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.737+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c158030 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.737+0000 7f2c737fe700 1 -- 172.21.15.187:0/2662931570 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74007320 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.737+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c15bf90 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.738+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c147380 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.738+0000 7f2c737fe700 1 -- 172.21.15.187:0/2662931570 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2c74010030 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.739+0000 7f2c737fe700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c58056d20 0x7f2c580591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:58.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.739+0000 7f2c737fe700 1 -- 172.21.15.187:0/2662931570 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2c7406fef0 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.739+0000 7f2c7b7fe700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c58056d20 0x7f2c580591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:58.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.740+0000 7f2c7b7fe700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c58056d20 0x7f2c580591e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f2c6c002410 tx=0x7f2c6c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:58.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.743+0000 7f2c737fe700 1 -- 172.21.15.187:0/2662931570 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2c740452f0 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.912+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c7c061a20 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.915+0000 7f2c737fe700 1 -- 172.21.15.187:0/2662931570 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16912 (secure 0 0 0) 0x7f2c74048910 con 0x7f2c7c0b5f50 2020-11-20T12:11:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:11:58.919 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":72,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:10:42.605346+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:10:20.847618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2020-11-20T12:10:22.150115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1383561284":"2020-11-21T12:10:42.605289+0000","172.21.15.187:6834/1383561284":"2020-11-21T12:10:42.605289+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:11:58.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c58056d20 msgr2=0x7f2c580591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c58056d20 0x7f2c580591e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f2c6c002410 tx=0x7f2c6c009040).stop 2020-11-20T12:11:58.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c0b5f50 msgr2=0x7f2c7c15b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:58.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c0b5f50 0x7f2c7c15b860 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7f2c7400ca60 tx=0x7f2c74001360).stop 2020-11-20T12:11:58.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 shutdown_connections 2020-11-20T12:11:58.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c58056d20 0x7f2c580591e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c7c068230 0x7f2c7c1466f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c7c0b5f50 0x7f2c7c15b860 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:58.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 --2- 172.21.15.187:0/2662931570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c7c0b57b0 0x7f2c7c157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:58.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.917+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 >> 172.21.15.187:0/2662931570 conn(0x7f2c7c0bab30 msgr2=0x7f2c7c147eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:58.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.918+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 shutdown_connections 2020-11-20T12:11:58.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:58.918+0000 7f2c80d99700 1 -- 172.21.15.187:0/2662931570 wait complete. 2020-11-20T12:11:58.945 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:11:58.946 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:11:59.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.195+0000 7fa4a5c38700 1 Processor -- start 2020-11-20T12:11:59.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.195+0000 7fa4a5c38700 1 -- start start 2020-11-20T12:11:59.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.196+0000 7fa4a5c38700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa4a0065190 0x7fa4a0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.196+0000 7fa4a5c38700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a00b13e0 con 0x7fa4a009ca90 2020-11-20T12:11:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.197+0000 7fa4a5c38700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a0062c10 con 0x7fa4a0065f90 2020-11-20T12:11:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.197+0000 7fa4a5c38700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a014ab30 con 0x7fa4a0065190 2020-11-20T12:11:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.197+0000 7fa4a4c36700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa4a0065190 0x7fa4a0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.197+0000 7fa497fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa4a0065f90 0x7fa4a0143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43212/0 (socket says 172.21.15.187:43212) 2020-11-20T12:11:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.197+0000 7fa497fff700 1 -- 172.21.15.187:0/2260163609 learned_addr learned my addr 172.21.15.187:0/2260163609 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:11:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.198+0000 7fa4a4c36700 1 -- 172.21.15.187:0/2260163609 >> v1:172.21.15.187:6789/0 conn(0x7fa4a0065f90 legacy=0x7fa4a0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.198+0000 7fa4a4c36700 1 -- 172.21.15.187:0/2260163609 >> v1:172.21.15.5:6789/0 conn(0x7fa4a009ca90 legacy=0x7fa4a009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.198+0000 7fa4a4c36700 1 -- 172.21.15.187:0/2260163609 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a00b1640 con 0x7fa4a0065190 2020-11-20T12:11:59.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.198+0000 7fa4a4c36700 1 --2- 172.21.15.187:0/2260163609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0065570 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fa4a014ac70 tx=0x7fa49c006df0).ready entity=mon.1 client_cookie=91c025fcf4fdd32e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:59.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.198+0000 7fa4977fe700 1 -- 172.21.15.187:0/2260163609 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c009800 con 0x7fa4a0065190 2020-11-20T12:11:59.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.198+0000 7fa4977fe700 1 -- 172.21.15.187:0/2260163609 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c009960 con 0x7fa4a0065190 2020-11-20T12:11:59.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.199+0000 7fa4977fe700 1 -- 172.21.15.187:0/2260163609 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c009ce0 con 0x7fa4a0065190 2020-11-20T12:11:59.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.199+0000 7fa4a5c38700 1 -- 172.21.15.187:0/2260163609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 msgr2=0x7fa4a0065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.199+0000 7fa4a5c38700 1 --2- 172.21.15.187:0/2260163609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0065570 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fa4a014ac70 tx=0x7fa49c006df0).stop 2020-11-20T12:11:59.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.199+0000 7fa4a5c38700 1 -- 172.21.15.187:0/2260163609 shutdown_connections 2020-11-20T12:11:59.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.199+0000 7fa4a5c38700 1 --2- 172.21.15.187:0/2260163609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0065570 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.199+0000 7fa4a5c38700 1 -- 172.21.15.187:0/2260163609 >> 172.21.15.187:0/2260163609 conn(0x7fa4a00bab30 msgr2=0x7fa4a00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:59.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.199+0000 7fa4a5c38700 1 -- 172.21.15.187:0/2260163609 shutdown_connections 2020-11-20T12:11:59.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.199+0000 7fa4a5c38700 1 -- 172.21.15.187:0/2260163609 wait complete. 2020-11-20T12:11:59.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.201+0000 7fa4a5c38700 1 Processor -- start 2020-11-20T12:11:59.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.201+0000 7fa4a5c38700 1 -- start start 2020-11-20T12:11:59.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a5c38700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a5c38700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4a0065f90 0x7fa4a009ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a5c38700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa4a009ca90 0x7fa4a0145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a5c38700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a009f440 con 0x7fa4a0065f90 2020-11-20T12:11:59.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a5c38700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a009f660 con 0x7fa4a0065190 2020-11-20T12:11:59.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a5c38700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a009f900 con 0x7fa4a009ca90 2020-11-20T12:11:59.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a4c36700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a4c36700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0147fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46610/0 (socket says 172.21.15.187:46610) 2020-11-20T12:11:59.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.202+0000 7fa4a4c36700 1 -- 172.21.15.187:0/1305040593 learned_addr learned my addr 172.21.15.187:0/1305040593 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:11:59.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa4a4c36700 1 -- 172.21.15.187:0/1305040593 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa4a009ca90 msgr2=0x7fa4a0145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa4a4c36700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa4a009ca90 0x7fa4a0145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:59.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa4a4c36700 1 -- 172.21.15.187:0/1305040593 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4a0065f90 msgr2=0x7fa4a009ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa4a4c36700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4a0065f90 0x7fa4a009ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:59.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa4a4c36700 1 -- 172.21.15.187:0/1305040593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a009fba0 con 0x7fa4a0065190 2020-11-20T12:11:59.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa4a4c36700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0147fa0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7fa49c006fd0 tx=0x7fa49c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:59.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa494ff9700 1 -- 172.21.15.187:0/1305040593 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c011e20 con 0x7fa4a0065190 2020-11-20T12:11:59.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa494ff9700 1 -- 172.21.15.187:0/1305040593 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c01d5d0 con 0x7fa4a0065190 2020-11-20T12:11:59.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa494ff9700 1 -- 172.21.15.187:0/1305040593 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c01d950 con 0x7fa4a0065190 2020-11-20T12:11:59.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a0145530 con 0x7fa4a0065190 2020-11-20T12:11:59.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.203+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a01499b0 con 0x7fa4a0065190 2020-11-20T12:11:59.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.204+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a009fd30 con 0x7fa4a0065190 2020-11-20T12:11:59.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.205+0000 7fa494ff9700 1 -- 172.21.15.187:0/1305040593 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa49c025030 con 0x7fa4a0065190 2020-11-20T12:11:59.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.205+0000 7fa494ff9700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa478056d20 0x7fa4780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.206+0000 7fa494ff9700 1 -- 172.21.15.187:0/1305040593 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa49c071950 con 0x7fa4a0065190 2020-11-20T12:11:59.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.206+0000 7fa497fff700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa478056d20 0x7fa4780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.206+0000 7fa497fff700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa478056d20 0x7fa4780591e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa4900055a0 tx=0x7fa4900077d0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:59.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.209+0000 7fa494ff9700 1 -- 172.21.15.187:0/1305040593 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa49c046e00 con 0x7fa4a0065190 2020-11-20T12:11:59.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:11:59.459+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa4a0061a20 con 0x7fa4a0065190 2020-11-20T12:12:00.274 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:12:00.270+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:12:00.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.287+0000 7fa494ff9700 1 -- 172.21.15.187:0/1305040593 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa49c023030 con 0x7fa4a0065190 2020-11-20T12:12:00.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.290+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa478056d20 msgr2=0x7fa4780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:00.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.290+0000 7fa4a5c38700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa478056d20 0x7fa4780591e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa4900055a0 tx=0x7fa4900077d0).stop 2020-11-20T12:12:00.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.290+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 msgr2=0x7fa4a0147fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:00.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.290+0000 7fa4a5c38700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0147fa0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7fa49c006fd0 tx=0x7fa49c015000).stop 2020-11-20T12:12:00.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.291+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 shutdown_connections 2020-11-20T12:12:00.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.291+0000 7fa4a5c38700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa478056d20 0x7fa4780591e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:00.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.291+0000 7fa4a5c38700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa4a009ca90 0x7fa4a0145040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:00.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.291+0000 7fa4a5c38700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4a0065190 0x7fa4a0147fa0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:00.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.291+0000 7fa4a5c38700 1 --2- 172.21.15.187:0/1305040593 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4a0065f90 0x7fa4a009ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:00.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.291+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 >> 172.21.15.187:0/1305040593 conn(0x7fa4a00bab30 msgr2=0x7fa4a00baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:00.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.291+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 shutdown_connections 2020-11-20T12:12:00.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.291+0000 7fa4a5c38700 1 -- 172.21.15.187:0/1305040593 wait complete. 2020-11-20T12:12:00.297 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:12:00.321 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:12:00.420 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:12:00.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.567+0000 7f15ca619700 1 Processor -- start 2020-11-20T12:12:00.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.567+0000 7f15ca619700 1 -- start start 2020-11-20T12:12:00.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.569+0000 7f15ca619700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 0x7f15c4066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:00.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.569+0000 7f15ca619700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f15c41487e0 0x7f15c4148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:00.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.569+0000 7f15ca619700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c414c170 con 0x7f15c40683f0 2020-11-20T12:12:00.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.569+0000 7f15ca619700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c414c410 con 0x7f15c41487e0 2020-11-20T12:12:00.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.569+0000 7f15ca619700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c414c6b0 con 0x7f15c4145cc0 2020-11-20T12:12:00.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.570+0000 7f15c9617700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 0x7f15c4066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:00.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.570+0000 7f15c8e16700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f15c40683f0 0x7f15c4065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:36734/0 (socket says 172.21.15.187:36734) 2020-11-20T12:12:00.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.570+0000 7f15c8e16700 1 -- 172.21.15.187:0/3369680445 learned_addr learned my addr 172.21.15.187:0/3369680445 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:00.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.570+0000 7f15c9e18700 1 --2- 172.21.15.187:0/3369680445 >> v2:172.21.15.187:3300/0 conn(0x7f15c41487e0 0x7f15c4148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:00.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.570+0000 7f15c9e18700 1 -- 172.21.15.187:0/3369680445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 msgr2=0x7f15c4066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:00.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.570+0000 7f15c9e18700 1 --2- 172.21.15.187:0/3369680445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 0x7f15c4066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:00.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.571+0000 7f15c9e18700 1 -- 172.21.15.187:0/3369680445 >> v1:172.21.15.5:6789/0 conn(0x7f15c40683f0 legacy=0x7f15c4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:00.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.571+0000 7f15c9e18700 1 -- 172.21.15.187:0/3369680445 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c40658a0 con 0x7f15c41487e0 2020-11-20T12:12:00.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.571+0000 7f15c9e18700 1 --2- 172.21.15.187:0/3369680445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c4148c90 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f15bc00b9a0 tx=0x7f15bc006df0).ready entity=mon.1 client_cookie=4fc232db6e2a5e5c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:00.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.571+0000 7f15b3fff700 1 -- 172.21.15.187:0/3369680445 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc0097b0 con 0x7f15c41487e0 2020-11-20T12:12:00.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.571+0000 7f15b3fff700 1 -- 172.21.15.187:0/3369680445 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc009910 con 0x7f15c41487e0 2020-11-20T12:12:00.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.571+0000 7f15b3fff700 1 -- 172.21.15.187:0/3369680445 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc009c90 con 0x7f15c41487e0 2020-11-20T12:12:00.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.572+0000 7f15ca619700 1 -- 172.21.15.187:0/3369680445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 msgr2=0x7f15c4148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:00.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.572+0000 7f15ca619700 1 --2- 172.21.15.187:0/3369680445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c4148c90 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f15bc00b9a0 tx=0x7f15bc006df0).stop 2020-11-20T12:12:00.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.572+0000 7f15ca619700 1 -- 172.21.15.187:0/3369680445 shutdown_connections 2020-11-20T12:12:00.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.572+0000 7f15ca619700 1 --2- 172.21.15.187:0/3369680445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c4148c90 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:00.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.572+0000 7f15ca619700 1 --2- 172.21.15.187:0/3369680445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 0x7f15c4066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:00.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.572+0000 7f15ca619700 1 -- 172.21.15.187:0/3369680445 >> 172.21.15.187:0/3369680445 conn(0x7f15c40bab30 msgr2=0x7f15c40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:00.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.572+0000 7f15ca619700 1 -- 172.21.15.187:0/3369680445 shutdown_connections 2020-11-20T12:12:00.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.572+0000 7f15ca619700 1 -- 172.21.15.187:0/3369680445 wait complete. 2020-11-20T12:12:00.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.574+0000 7f15ca619700 1 Processor -- start 2020-11-20T12:12:00.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.574+0000 7f15ca619700 1 -- start start 2020-11-20T12:12:00.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.575+0000 7f15ca619700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15c40683f0 0x7f15c40b60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:00.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.575+0000 7f15ca619700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 0x7f15c409dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:00.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.575+0000 7f15ca619700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c409dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:00.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.575+0000 7f15ca619700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c40b5db0 con 0x7f15c40683f0 2020-11-20T12:12:00.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.575+0000 7f15ca619700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c40b65e0 con 0x7f15c41487e0 2020-11-20T12:12:00.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.575+0000 7f15ca619700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c409f9e0 con 0x7f15c4145cc0 2020-11-20T12:12:00.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.575+0000 7f15c9e18700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c409dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:00.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9e18700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c409dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46626/0 (socket says 172.21.15.187:46626) 2020-11-20T12:12:00.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9e18700 1 -- 172.21.15.187:0/678833869 learned_addr learned my addr 172.21.15.187:0/678833869 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:00.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c8e16700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 0x7f15c409dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:00.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9617700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15c40683f0 0x7f15c40b60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:00.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9e18700 1 -- 172.21.15.187:0/678833869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 msgr2=0x7f15c409dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:00.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9e18700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 0x7f15c409dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:00.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9e18700 1 -- 172.21.15.187:0/678833869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15c40683f0 msgr2=0x7f15c40b60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:00.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9e18700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15c40683f0 0x7f15c40b60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:00.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9e18700 1 -- 172.21.15.187:0/678833869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c409fc80 con 0x7f15c41487e0 2020-11-20T12:12:00.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.576+0000 7f15c9e18700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c409dfd0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7f15c4063390 tx=0x7f15bc015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:00.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.577+0000 7f15b17fa700 1 -- 172.21.15.187:0/678833869 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc01e460 con 0x7f15c41487e0 2020-11-20T12:12:00.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.577+0000 7f15b17fa700 1 -- 172.21.15.187:0/678833869 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc01e5c0 con 0x7f15c41487e0 2020-11-20T12:12:00.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.577+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c409f720 con 0x7f15c41487e0 2020-11-20T12:12:00.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.577+0000 7f15b17fa700 1 -- 172.21.15.187:0/678833869 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc01e940 con 0x7f15c41487e0 2020-11-20T12:12:00.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.577+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c414b640 con 0x7f15c41487e0 2020-11-20T12:12:00.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.578+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c4061a20 con 0x7f15c41487e0 2020-11-20T12:12:00.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.578+0000 7f15b17fa700 1 -- 172.21.15.187:0/678833869 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f15bc025030 con 0x7f15c41487e0 2020-11-20T12:12:00.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.579+0000 7f15b17fa700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f159c056d20 0x7f159c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:00.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.579+0000 7f15b17fa700 1 -- 172.21.15.187:0/678833869 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f15bc071ae0 con 0x7f15c41487e0 2020-11-20T12:12:00.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.580+0000 7f15c9617700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f159c056d20 0x7f159c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:00.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.580+0000 7f15c9617700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f159c056d20 0x7f159c0591e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f15c0009440 tx=0x7f15c0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:00.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.583+0000 7f15b17fa700 1 -- 172.21.15.187:0/678833869 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f15bc046e20 con 0x7f15c41487e0 2020-11-20T12:12:00.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:00.828+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f15c414bdf0 con 0x7f15c41487e0 2020-11-20T12:12:01.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.476+0000 7f15b17fa700 1 -- 172.21.15.187:0/678833869 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7f15bc023070 con 0x7f15c41487e0 2020-11-20T12:12:01.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f159c056d20 msgr2=0x7f159c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f159c056d20 0x7f159c0591e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f15c0009440 tx=0x7f15c0008040).stop 2020-11-20T12:12:01.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 msgr2=0x7f15c409dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c409dfd0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7f15c4063390 tx=0x7f15bc015000).stop 2020-11-20T12:12:01.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 shutdown_connections 2020-11-20T12:12:01.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f159c056d20 0x7f159c0591e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15c40683f0 0x7f15c40b60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15c41487e0 0x7f15c409dfd0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.479+0000 7f15ca619700 1 --2- 172.21.15.187:0/678833869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15c4145cc0 0x7f15c409dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.480+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 >> 172.21.15.187:0/678833869 conn(0x7f15c40bab30 msgr2=0x7f15c40b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:01.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.480+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 shutdown_connections 2020-11-20T12:12:01.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.480+0000 7f15ca619700 1 -- 172.21.15.187:0/678833869 wait complete. 2020-11-20T12:12:01.507 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:12:01.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.752+0000 7f03e92a5700 1 Processor -- start 2020-11-20T12:12:01.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.752+0000 7f03e92a5700 1 -- start start 2020-11-20T12:12:01.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.753+0000 7f03e92a5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b1380 0x7f03e4150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.753+0000 7f03e92a5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03e40b1760 0x7f03e4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.753+0000 7f03e92a5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e4150390 con 0x7f03e4065680 2020-11-20T12:12:01.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.754+0000 7f03e92a5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e4150630 con 0x7f03e40b1760 2020-11-20T12:12:01.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.754+0000 7f03e92a5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e4068510 con 0x7f03e40b1380 2020-11-20T12:12:01.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.754+0000 7f03e8aa4700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f03e4065680 0x7f03e4061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43240/0 (socket says 172.21.15.187:43240) 2020-11-20T12:12:01.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.754+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1334163507 learned_addr learned my addr 172.21.15.187:0/1334163507 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:01.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.754+0000 7f03e3fff700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b1380 0x7f03e4150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.754+0000 7f03e37fe700 1 --2- 172.21.15.187:0/1334163507 >> v2:172.21.15.5:3300/0 conn(0x7f03e40b1760 0x7f03e4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.754+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094411104 0 0) 0x7f03e4150390 con 0x7f03e4065680 2020-11-20T12:12:01.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d0003240 con 0x7f03e4065680 2020-11-20T12:12:01.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1823920524 0 0) 0x7f03d0003240 con 0x7f03e4065680 2020-11-20T12:12:01.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b1380 msgr2=0x7f03e4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:01.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b1380 0x7f03e4150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 >> v2:172.21.15.5:3300/0 conn(0x7f03e40b1760 msgr2=0x7f03e4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:01.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 --2- 172.21.15.187:0/1334163507 >> v2:172.21.15.5:3300/0 conn(0x7f03e40b1760 0x7f03e4065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e4150fc0 con 0x7f03e4065680 2020-11-20T12:12:01.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f03dc001860 con 0x7f03e4065680 2020-11-20T12:12:01.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 >> v1:172.21.15.187:6789/0 conn(0x7f03e4065680 legacy=0x7f03e4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b1380 0x7f03d00037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03d0003d10 0x7f03d00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03d00049d0 0x7f03d0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e4150630 con 0x7f03d0003d10 2020-11-20T12:12:01.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e4068510 con 0x7f03d00049d0 2020-11-20T12:12:01.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.755+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00042d0 con 0x7f03e40b1380 2020-11-20T12:12:01.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.756+0000 7f03e8aa4700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03d00049d0 0x7f03d0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.756+0000 7f03e37fe700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03d0003d10 0x7f03d00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.756+0000 7f03e3fff700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b1380 0x7f03d00037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.756+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b1380 msgr2=0x7f03d00037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:01.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.756+0000 7f03e8aa4700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b1380 0x7f03d00037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.756+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03d0003d10 msgr2=0x7f03d00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:01.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.756+0000 7f03e8aa4700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03d0003d10 0x7f03d00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.756+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1334163507 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0003240 con 0x7f03d00049d0 2020-11-20T12:12:01.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e8aa4700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03d00049d0 0x7f03d0004e50 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f03dc00c5d0 tx=0x7f03dc007ad0).ready entity=mon.1 client_cookie=82753ef8351f6d0c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:01.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc005b80 con 0x7f03d00049d0 2020-11-20T12:12:01.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc005ce0 con 0x7f03d00049d0 2020-11-20T12:12:01.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e2ffd700 1 -- 172.21.15.187:0/1334163507 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc007300 con 0x7f03d00049d0 2020-11-20T12:12:01.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e92a5700 1 -- 172.21.15.187:0/1334163507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03d00049d0 msgr2=0x7f03d0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e92a5700 1 --2- 172.21.15.187:0/1334163507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03d00049d0 0x7f03d0004e50 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f03dc00c5d0 tx=0x7f03dc007ad0).stop 2020-11-20T12:12:01.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1334163507 reap_dead start 2020-11-20T12:12:01.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e92a5700 1 -- 172.21.15.187:0/1334163507 shutdown_connections 2020-11-20T12:12:01.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.757+0000 7f03e92a5700 1 -- 172.21.15.187:0/1334163507 >> 172.21.15.187:0/1334163507 conn(0x7f03e40bab30 msgr2=0x7f03e40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:01.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.758+0000 7f03e92a5700 1 -- 172.21.15.187:0/1334163507 shutdown_connections 2020-11-20T12:12:01.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.758+0000 7f03e92a5700 1 -- 172.21.15.187:0/1334163507 wait complete. 2020-11-20T12:12:01.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.759+0000 7f03e92a5700 1 Processor -- start 2020-11-20T12:12:01.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.759+0000 7f03e92a5700 1 -- start start 2020-11-20T12:12:01.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e92a5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b4360 0x7f03e414a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e92a5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03e40b4740 0x7f03e414a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e92a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03e409ae20 0x7f03e409dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e92a5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e414ae80 con 0x7f03e40b4740 2020-11-20T12:12:01.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e92a5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e409f8a0 con 0x7f03e409ae20 2020-11-20T12:12:01.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e92a5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e409b330 con 0x7f03e40b4360 2020-11-20T12:12:01.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e8aa4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03e409ae20 0x7f03e409dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e8aa4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03e409ae20 0x7f03e409dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46648/0 (socket says 172.21.15.187:46648) 2020-11-20T12:12:01.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.760+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1695575481 learned_addr learned my addr 172.21.15.187:0/1695575481 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:01.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e37fe700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03e40b4740 0x7f03e414a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e3fff700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b4360 0x7f03e414a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b4360 msgr2=0x7f03e414a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e8aa4700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b4360 0x7f03e414a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03e40b4740 msgr2=0x7f03e414a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e8aa4700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03e40b4740 0x7f03e414a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e8aa4700 1 -- 172.21.15.187:0/1695575481 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e409b5d0 con 0x7f03e409ae20 2020-11-20T12:12:01.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e37fe700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03e40b4740 0x7f03e414a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:01.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e8aa4700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03e409ae20 0x7f03e409dae0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f03dc000f40 tx=0x7f03dc013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:01.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03c3fff700 1 -- 172.21.15.187:0/1695575481 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc017070 con 0x7f03e409ae20 2020-11-20T12:12:01.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03c3fff700 1 -- 172.21.15.187:0/1695575481 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc0072e0 con 0x7f03e409ae20 2020-11-20T12:12:01.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03c3fff700 1 -- 172.21.15.187:0/1695575481 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc01a040 con 0x7f03e409ae20 2020-11-20T12:12:01.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.761+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e409b870 con 0x7f03e409ae20 2020-11-20T12:12:01.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.762+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e409d640 con 0x7f03e409ae20 2020-11-20T12:12:01.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.763+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e409ba00 con 0x7f03e409ae20 2020-11-20T12:12:01.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.763+0000 7f03c3fff700 1 -- 172.21.15.187:0/1695575481 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f03dc010070 con 0x7f03e409ae20 2020-11-20T12:12:01.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.764+0000 7f03c3fff700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03bc056d20 0x7f03bc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.764+0000 7f03c3fff700 1 -- 172.21.15.187:0/1695575481 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f03dc0714c0 con 0x7f03e409ae20 2020-11-20T12:12:01.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.764+0000 7f03e3fff700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03bc056d20 0x7f03bc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.765+0000 7f03e3fff700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03bc056d20 0x7f03bc0591e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f03e414b390 tx=0x7f03d4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:01.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.768+0000 7f03c3fff700 1 -- 172.21.15.187:0/1695575481 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f03dc0467e0 con 0x7f03e409ae20 2020-11-20T12:12:01.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:01.952+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f03e4061a20 con 0x7f03e409ae20 2020-11-20T12:12:02.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.489+0000 7f03c3fff700 1 -- 172.21.15.187:0/1695575481 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v74) v1 ==== 181+0+0 (secure 0 0 0) 0x7f03dc049e00 con 0x7f03e409ae20 2020-11-20T12:12:02.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.492+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03bc056d20 msgr2=0x7f03bc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:02.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.492+0000 7f03e92a5700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03bc056d20 0x7f03bc0591e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f03e414b390 tx=0x7f03d4008040).stop 2020-11-20T12:12:02.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.492+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03e409ae20 msgr2=0x7f03e409dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:02.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.492+0000 7f03e92a5700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03e409ae20 0x7f03e409dae0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f03dc000f40 tx=0x7f03dc013000).stop 2020-11-20T12:12:02.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.493+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 shutdown_connections 2020-11-20T12:12:02.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.493+0000 7f03e92a5700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03bc056d20 0x7f03bc0591e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:02.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.493+0000 7f03e92a5700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03e409ae20 0x7f03e409dae0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:02.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.493+0000 7f03e92a5700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03e40b4360 0x7f03e414a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:02.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.493+0000 7f03e92a5700 1 --2- 172.21.15.187:0/1695575481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03e40b4740 0x7f03e414a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:02.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.493+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 >> 172.21.15.187:0/1695575481 conn(0x7f03e40bab30 msgr2=0x7f03e414b670 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:02.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.493+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 shutdown_connections 2020-11-20T12:12:02.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.494+0000 7f03e92a5700 1 -- 172.21.15.187:0/1695575481 wait complete. 2020-11-20T12:12:02.499 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:12:02.523 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:12:02.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.768+0000 7fe1cc98a700 1 Processor -- start 2020-11-20T12:12:02.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.768+0000 7fe1cc98a700 1 -- start start 2020-11-20T12:12:02.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.769+0000 7fe1cc98a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1c8065190 0x7fe1c8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:02.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.769+0000 7fe1cc98a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe1c8065f90 0x7fe1c8143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:02.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.769+0000 7fe1cc98a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80b13e0 con 0x7fe1c8143c70 2020-11-20T12:12:02.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.769+0000 7fe1cc98a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c814ab30 con 0x7fe1c8065f90 2020-11-20T12:12:02.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.769+0000 7fe1cc98a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c8062c10 con 0x7fe1c8065190 2020-11-20T12:12:02.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.770+0000 7fe1c77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1c8065190 0x7fe1c8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:02.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.770+0000 7fe1c77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1c8065190 0x7fe1c8065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46654/0 (socket says 172.21.15.187:46654) 2020-11-20T12:12:02.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.770+0000 7fe1c77fe700 1 -- 172.21.15.187:0/16620511 learned_addr learned my addr 172.21.15.187:0/16620511 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:02.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.771+0000 7fe1beffd700 1 --2- 172.21.15.187:0/16620511 >> v2:172.21.15.5:3300/0 conn(0x7fe1c8065f90 0x7fe1c8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:02.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.771+0000 7fe1c77fe700 1 -- 172.21.15.187:0/16620511 >> v2:172.21.15.5:3300/0 conn(0x7fe1c8065f90 msgr2=0x7fe1c8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:02.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.771+0000 7fe1c77fe700 1 --2- 172.21.15.187:0/16620511 >> v2:172.21.15.5:3300/0 conn(0x7fe1c8065f90 0x7fe1c8143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:02.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.771+0000 7fe1c77fe700 1 -- 172.21.15.187:0/16620511 >> v1:172.21.15.5:6789/0 conn(0x7fe1c8143c70 legacy=0x7fe1c809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:02.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.771+0000 7fe1c77fe700 1 -- 172.21.15.187:0/16620511 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c80b1640 con 0x7fe1c8065190 2020-11-20T12:12:02.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.771+0000 7fe1c77fe700 1 --2- 172.21.15.187:0/16620511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065190 0x7fe1c8065570 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7fe1c805c8b0 tx=0x7fe1b8006eb0).ready entity=mon.1 client_cookie=be3b161bb967886a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.771+0000 7fe1c6ffd700 1 -- 172.21.15.187:0/16620511 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8009920 con 0x7fe1c8065190 2020-11-20T12:12:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1c6ffd700 1 -- 172.21.15.187:0/16620511 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b8009a80 con 0x7fe1c8065190 2020-11-20T12:12:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1c6ffd700 1 -- 172.21.15.187:0/16620511 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8009e00 con 0x7fe1c8065190 2020-11-20T12:12:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1cc98a700 1 -- 172.21.15.187:0/16620511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065190 msgr2=0x7fe1c8065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:02.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/16620511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065190 0x7fe1c8065570 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7fe1c805c8b0 tx=0x7fe1b8006eb0).stop 2020-11-20T12:12:02.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1cc98a700 1 -- 172.21.15.187:0/16620511 shutdown_connections 2020-11-20T12:12:02.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/16620511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065190 0x7fe1c8065570 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:02.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/16620511 >> v2:172.21.15.5:3300/0 conn(0x7fe1c8065f90 0x7fe1c8143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:02.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1cc98a700 1 -- 172.21.15.187:0/16620511 >> 172.21.15.187:0/16620511 conn(0x7fe1c80bab30 msgr2=0x7fe1c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:02.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1cc98a700 1 -- 172.21.15.187:0/16620511 shutdown_connections 2020-11-20T12:12:02.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.772+0000 7fe1cc98a700 1 -- 172.21.15.187:0/16620511 wait complete. 2020-11-20T12:12:02.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.774+0000 7fe1cc98a700 1 Processor -- start 2020-11-20T12:12:02.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.774+0000 7fe1cc98a700 1 -- start start 2020-11-20T12:12:02.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1cc98a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065f90 0x7fe1c809fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:02.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1cc98a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1c8143c70 0x7fe1c809c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:02.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1cc98a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1c80b1c20 0x7fe1c809e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:02.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1cc98a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80b4b30 con 0x7fe1c8143c70 2020-11-20T12:12:02.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1cc98a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8149f60 con 0x7fe1c8065f90 2020-11-20T12:12:02.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1cc98a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80b4890 con 0x7fe1c80b1c20 2020-11-20T12:12:02.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1c77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065f90 0x7fe1c809fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:02.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1c77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065f90 0x7fe1c809fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46658/0 (socket says 172.21.15.187:46658) 2020-11-20T12:12:02.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.775+0000 7fe1c77fe700 1 -- 172.21.15.187:0/3011150461 learned_addr learned my addr 172.21.15.187:0/3011150461 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:02.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1c77fe700 1 -- 172.21.15.187:0/3011150461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1c80b1c20 msgr2=0x7fe1c809e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:02.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1c77fe700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1c80b1c20 0x7fe1c809e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:02.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1c77fe700 1 -- 172.21.15.187:0/3011150461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1c8143c70 msgr2=0x7fe1c809c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:02.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1c77fe700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1c8143c70 0x7fe1c809c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:02.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1c77fe700 1 -- 172.21.15.187:0/3011150461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c809dc60 con 0x7fe1c8065f90 2020-11-20T12:12:02.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1c77fe700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065f90 0x7fe1c809fcf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7fe1b8006ef0 tx=0x7fe1b8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:02.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1bffff700 1 -- 172.21.15.187:0/3011150461 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b801e460 con 0x7fe1c8065f90 2020-11-20T12:12:02.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1bffff700 1 -- 172.21.15.187:0/3011150461 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b801e5c0 con 0x7fe1c8065f90 2020-11-20T12:12:02.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c80b4510 con 0x7fe1c8065f90 2020-11-20T12:12:02.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.776+0000 7fe1bffff700 1 -- 172.21.15.187:0/3011150461 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b801e940 con 0x7fe1c8065f90 2020-11-20T12:12:02.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.777+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c809cb60 con 0x7fe1c8065f90 2020-11-20T12:12:02.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.778+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c809ddf0 con 0x7fe1c8065f90 2020-11-20T12:12:02.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.778+0000 7fe1bffff700 1 -- 172.21.15.187:0/3011150461 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe1b801a040 con 0x7fe1c8065f90 2020-11-20T12:12:02.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.779+0000 7fe1bffff700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1a4056d20 0x7fe1a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:02.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.779+0000 7fe1bffff700 1 -- 172.21.15.187:0/3011150461 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5807+0+0 (secure 0 0 0) 0x7fe1b8071470 con 0x7fe1c8065f90 2020-11-20T12:12:02.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.779+0000 7fe1beffd700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1a4056d20 0x7fe1a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:02.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.780+0000 7fe1beffd700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1a4056d20 0x7fe1a40591e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe1c80a01e0 tx=0x7fe1b0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:02.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.783+0000 7fe1bffff700 1 -- 172.21.15.187:0/3011150461 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe1b80469f0 con 0x7fe1c8065f90 2020-11-20T12:12:02.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:02.980+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe1c8061a20 con 0x7fe1c8065f90 2020-11-20T12:12:03.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.492+0000 7fe1bffff700 1 -- 172.21.15.187:0/3011150461 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v75) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe1b8023020 con 0x7fe1c8065f90 2020-11-20T12:12:03.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1a4056d20 msgr2=0x7fe1a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1a4056d20 0x7fe1a40591e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe1c80a01e0 tx=0x7fe1b0008040).stop 2020-11-20T12:12:03.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065f90 msgr2=0x7fe1c809fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065f90 0x7fe1c809fcf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7fe1b8006ef0 tx=0x7fe1b8015000).stop 2020-11-20T12:12:03.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 shutdown_connections 2020-11-20T12:12:03.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1a4056d20 0x7fe1a40591e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:03.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1c80b1c20 0x7fe1c809e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:03.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c8065f90 0x7fe1c809fcf0 secure :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0x7fe1b8006ef0 tx=0x7fe1b8015000).stop 2020-11-20T12:12:03.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 --2- 172.21.15.187:0/3011150461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1c8143c70 0x7fe1c809c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.496+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 >> 172.21.15.187:0/3011150461 conn(0x7fe1c80bab30 msgr2=0x7fe1c80b63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.497+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 shutdown_connections 2020-11-20T12:12:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.497+0000 7fe1cc98a700 1 -- 172.21.15.187:0/3011150461 wait complete. 2020-11-20T12:12:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:12:03.529 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:12:03.529 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:12:03.530 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:12:03.530 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:12:03.530 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:12:03.531 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:12:03.531 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:12:03.531 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:12:03.550 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:12:03.551 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:12:03.557 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:12:03.558 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:12:03.559 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:12:03.560 DEBUG:teuthology.parallel:result is None 2020-11-20T12:12:03.606 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:12:03.606 DEBUG:teuthology.parallel:result is None 2020-11-20T12:12:03.607 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:12:03.738 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:12:03.774 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:12:03.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.817+0000 7fac0f341700 1 Processor -- start 2020-11-20T12:12:03.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.817+0000 7fac0f341700 1 -- start start 2020-11-20T12:12:03.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.818+0000 7fac0f341700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080652e0 0x7fac080656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.818+0000 7fac0f341700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fac080b1590 0x7fac0814a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.818+0000 7fac0f341700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac08149dd0 con 0x7fac080b7300 2020-11-20T12:12:03.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.818+0000 7fac0f341700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0814a070 con 0x7fac080b1590 2020-11-20T12:12:03.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.818+0000 7fac0f341700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0814a310 con 0x7fac080652e0 2020-11-20T12:12:03.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0c8dc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fac080b1590 0x7fac0814a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0c8dc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fac080b1590 0x7fac0814a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46686/0 (socket says 172.21.15.187:46686) 2020-11-20T12:12:03.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0c8dc700 1 -- 172.21.15.187:0/654287250 learned_addr learned my addr 172.21.15.187:0/654287250 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:03.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0d0dd700 1 --2- 172.21.15.187:0/654287250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080652e0 0x7fac080656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0c8dc700 1 -- 172.21.15.187:0/654287250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080652e0 msgr2=0x7fac080656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:03.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0c8dc700 1 --2- 172.21.15.187:0/654287250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080652e0 0x7fac080656c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:03.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0c8dc700 1 -- 172.21.15.187:0/654287250 >> v1:172.21.15.5:6789/0 conn(0x7fac080b7300 legacy=0x7fac08143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:03.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0c8dc700 1 -- 172.21.15.187:0/654287250 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0814aca0 con 0x7fac080b1590 2020-11-20T12:12:03.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.819+0000 7fac0c8dc700 1 --2- 172.21.15.187:0/654287250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac0814a7b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fabf800ba70 tx=0x7fabf8006eb0).ready entity=mon.1 client_cookie=4da7a2273a702a86 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:03.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fabf7fff700 1 -- 172.21.15.187:0/654287250 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8009800 con 0x7fac080b1590 2020-11-20T12:12:03.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fabf7fff700 1 -- 172.21.15.187:0/654287250 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf8009960 con 0x7fac080b1590 2020-11-20T12:12:03.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fabf7fff700 1 -- 172.21.15.187:0/654287250 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8009ce0 con 0x7fac080b1590 2020-11-20T12:12:03.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fac0f341700 1 -- 172.21.15.187:0/654287250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 msgr2=0x7fac0814a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fac0f341700 1 --2- 172.21.15.187:0/654287250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac0814a7b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fabf800ba70 tx=0x7fabf8006eb0).stop 2020-11-20T12:12:03.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fac0f341700 1 -- 172.21.15.187:0/654287250 shutdown_connections 2020-11-20T12:12:03.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fac0f341700 1 --2- 172.21.15.187:0/654287250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080652e0 0x7fac080656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:03.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fac0f341700 1 --2- 172.21.15.187:0/654287250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac0814a7b0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:03.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fac0f341700 1 -- 172.21.15.187:0/654287250 >> 172.21.15.187:0/654287250 conn(0x7fac080bacd0 msgr2=0x7fac080bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:03.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fac0f341700 1 -- 172.21.15.187:0/654287250 shutdown_connections 2020-11-20T12:12:03.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.820+0000 7fac0f341700 1 -- 172.21.15.187:0/654287250 wait complete. 2020-11-20T12:12:03.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.822+0000 7fac0f341700 1 Processor -- start 2020-11-20T12:12:03.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.822+0000 7fac0f341700 1 -- start start 2020-11-20T12:12:03.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0f341700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac080652e0 0x7fac080b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0f341700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac08143250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0f341700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080b7300 0x7fac080b9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0f341700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac080b4c30 con 0x7fac080652e0 2020-11-20T12:12:03.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0f341700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0815b830 con 0x7fac080b1590 2020-11-20T12:12:03.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0f341700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0808fbb0 con 0x7fac080b7300 2020-11-20T12:12:03.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0c8dc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac08143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0c8dc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac08143250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46690/0 (socket says 172.21.15.187:46690) 2020-11-20T12:12:03.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.823+0000 7fac0c8dc700 1 -- 172.21.15.187:0/3840342934 learned_addr learned my addr 172.21.15.187:0/3840342934 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:03.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0d0dd700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac080652e0 0x7fac080b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0c8dc700 1 -- 172.21.15.187:0/3840342934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080b7300 msgr2=0x7fac080b9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0c8dc700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080b7300 0x7fac080b9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:03.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0c8dc700 1 -- 172.21.15.187:0/3840342934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac080652e0 msgr2=0x7fac080b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0c8dc700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac080652e0 0x7fac080b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:03.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0c8dc700 1 -- 172.21.15.187:0/3840342934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0815ba60 con 0x7fac080b1590 2020-11-20T12:12:03.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0c8dc700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac08143250 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7fabf8006ef0 tx=0x7fabf8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:03.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fabf57fa700 1 -- 172.21.15.187:0/3840342934 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf801e460 con 0x7fac080b1590 2020-11-20T12:12:03.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fabf57fa700 1 -- 172.21.15.187:0/3840342934 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf801e5c0 con 0x7fac080b1590 2020-11-20T12:12:03.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac08143740 con 0x7fac080b1590 2020-11-20T12:12:03.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fabf57fa700 1 -- 172.21.15.187:0/3840342934 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf801e940 con 0x7fac080b1590 2020-11-20T12:12:03.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.824+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0808ebb0 con 0x7fac080b1590 2020-11-20T12:12:03.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.825+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0815bbf0 con 0x7fac080b1590 2020-11-20T12:12:03.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.826+0000 7fabf57fa700 1 -- 172.21.15.187:0/3840342934 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fabf8025030 con 0x7fac080b1590 2020-11-20T12:12:03.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.826+0000 7fabf57fa700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabe0056d20 0x7fabe00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.827+0000 7fabf57fa700 1 -- 172.21.15.187:0/3840342934 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5425+0+0 (secure 0 0 0) 0x7fabf8071530 con 0x7fac080b1590 2020-11-20T12:12:03.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.827+0000 7fac0d0dd700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabe0056d20 0x7fabe00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.827+0000 7fac0d0dd700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabe0056d20 0x7fabe00591e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fac04009650 tx=0x7fac04008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:03.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:03.830+0000 7fabf57fa700 1 -- 172.21.15.187:0/3840342934 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fabf8046c30 con 0x7fac080b1590 2020-11-20T12:12:04.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.015+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fac08063600 con 0x7fac080b1590 2020-11-20T12:12:04.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.815+0000 7fabf57fa700 1 -- 172.21.15.187:0/3840342934 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v77) v1 ==== 97+0+0 (secure 0 0 0) 0x7fabf8023070 con 0x7fac080b1590 2020-11-20T12:12:04.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabe0056d20 msgr2=0x7fabe00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabe0056d20 0x7fabe00591e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fac04009650 tx=0x7fac04008040).stop 2020-11-20T12:12:04.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 msgr2=0x7fac08143250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac08143250 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7fabf8006ef0 tx=0x7fabf8015000).stop 2020-11-20T12:12:04.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 shutdown_connections 2020-11-20T12:12:04.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabe0056d20 0x7fabe00591e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac080652e0 0x7fac080b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fac080b7300 0x7fac080b9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:04.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.818+0000 7fac0f341700 1 --2- 172.21.15.187:0/3840342934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac080b1590 0x7fac08143250 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.819+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 >> 172.21.15.187:0/3840342934 conn(0x7fac080bacd0 msgr2=0x7fac080b9170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:04.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.819+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 shutdown_connections 2020-11-20T12:12:04.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:04.819+0000 7fac0f341700 1 -- 172.21.15.187:0/3840342934 wait complete. 2020-11-20T12:12:04.825 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:12:04.850 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:12:05.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.093+0000 7fd442fb2700 1 Processor -- start 2020-11-20T12:12:05.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.093+0000 7fd442fb2700 1 -- start start 2020-11-20T12:12:05.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.094+0000 7fd442fb2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c0b57b0 0x7fd43c068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.094+0000 7fd442fb2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd43c144b80 con 0x7fd43c0b5f50 2020-11-20T12:12:05.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.094+0000 7fd442fb2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd43c1489e0 con 0x7fd43c1482d0 2020-11-20T12:12:05.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.094+0000 7fd442fb2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c148c80 con 0x7fd43c0b57b0 2020-11-20T12:12:05.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.095+0000 7fd4417af700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd43c1482d0 0x7fd43c062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43304/0 (socket says 172.21.15.187:43304) 2020-11-20T12:12:05.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.095+0000 7fd4417af700 1 -- 172.21.15.187:0/3313839305 learned_addr learned my addr 172.21.15.187:0/3313839305 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:05.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.095+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126679090 0 0) 0x7fd43c1489e0 con 0x7fd43c1482d0 2020-11-20T12:12:05.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.095+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd424003240 con 0x7fd43c1482d0 2020-11-20T12:12:05.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.095+0000 7fd441fb0700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c0b57b0 0x7fd43c068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 264525099 0 0) 0x7fd424003240 con 0x7fd43c1482d0 2020-11-20T12:12:05.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c0b57b0 msgr2=0x7fd43c068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:05.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c0b57b0 0x7fd43c068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 >> v1:172.21.15.5:6789/0 conn(0x7fd43c0b5f50 legacy=0x7fd43c0b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:05.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43c148f20 con 0x7fd43c1482d0 2020-11-20T12:12:05.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd42c0014e0 con 0x7fd43c1482d0 2020-11-20T12:12:05.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 >> v1:172.21.15.187:6789/0 conn(0x7fd43c1482d0 legacy=0x7fd43c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4240037b0 0x7fd424003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c0b57b0 0x7fd4240044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd424004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c148c80 con 0x7fd4240037b0 2020-11-20T12:12:05.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd424004280 con 0x7fd4240049d0 2020-11-20T12:12:05.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42400d940 con 0x7fd43c0b57b0 2020-11-20T12:12:05.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.096+0000 7fd4427b1700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd424004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd4417af700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c0b57b0 0x7fd4240044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd4427b1700 1 -- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c0b57b0 msgr2=0x7fd4240044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:05.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd4427b1700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c0b57b0 0x7fd4240044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd4427b1700 1 -- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4240037b0 msgr2=0x7fd424003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:05.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd441fb0700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4240037b0 0x7fd424003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd4427b1700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4240037b0 0x7fd424003bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd4427b1700 1 -- 172.21.15.187:0/3313839305 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424003240 con 0x7fd4240049d0 2020-11-20T12:12:05.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd4427b1700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd424004e50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7fd434002db0 tx=0x7fd434007f10).ready entity=mon.1 client_cookie=bedd95d78fbda969 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:05.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434002430 con 0x7fd4240049d0 2020-11-20T12:12:05.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43400ca20 con 0x7fd4240049d0 2020-11-20T12:12:05.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.097+0000 7fd440fae700 1 -- 172.21.15.187:0/3313839305 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43400cde0 con 0x7fd4240049d0 2020-11-20T12:12:05.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.098+0000 7fd442fb2700 1 -- 172.21.15.187:0/3313839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 msgr2=0x7fd424004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.098+0000 7fd442fb2700 1 --2- 172.21.15.187:0/3313839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd424004e50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7fd434002db0 tx=0x7fd434007f10).stop 2020-11-20T12:12:05.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.098+0000 7fd4427b1700 1 -- 172.21.15.187:0/3313839305 reap_dead start 2020-11-20T12:12:05.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.098+0000 7fd442fb2700 1 -- 172.21.15.187:0/3313839305 shutdown_connections 2020-11-20T12:12:05.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.098+0000 7fd442fb2700 1 -- 172.21.15.187:0/3313839305 >> 172.21.15.187:0/3313839305 conn(0x7fd43c0bab30 msgr2=0x7fd43c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:05.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.098+0000 7fd442fb2700 1 -- 172.21.15.187:0/3313839305 shutdown_connections 2020-11-20T12:12:05.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.098+0000 7fd442fb2700 1 -- 172.21.15.187:0/3313839305 wait complete. 2020-11-20T12:12:05.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.100+0000 7fd442fb2700 1 Processor -- start 2020-11-20T12:12:05.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.100+0000 7fd442fb2700 1 -- start start 2020-11-20T12:12:05.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd442fb2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd43c158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd442fb2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c1581c0 0x7fd43c143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd442fb2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd43c144300 0x7fd43c15b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd442fb2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c0b3b60 con 0x7fd43c144300 2020-11-20T12:12:05.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd442fb2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c0b3e00 con 0x7fd4240049d0 2020-11-20T12:12:05.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd442fb2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c0b40a0 con 0x7fd43c1581c0 2020-11-20T12:12:05.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd441fb0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd43c158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd441fb0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd43c158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46710/0 (socket says 172.21.15.187:46710) 2020-11-20T12:12:05.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd441fb0700 1 -- 172.21.15.187:0/383573821 learned_addr learned my addr 172.21.15.187:0/383573821 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:05.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd4427b1700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd43c144300 0x7fd43c15b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd4417af700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c1581c0 0x7fd43c143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd441fb0700 1 -- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c1581c0 msgr2=0x7fd43c143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd441fb0700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c1581c0 0x7fd43c143e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd441fb0700 1 -- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd43c144300 msgr2=0x7fd43c15b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.101+0000 7fd441fb0700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd43c144300 0x7fd43c15b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.102+0000 7fd441fb0700 1 -- 172.21.15.187:0/383573821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43c0b4540 con 0x7fd4240049d0 2020-11-20T12:12:05.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.102+0000 7fd4427b1700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd43c144300 0x7fd43c15b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:05.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.102+0000 7fd441fb0700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd43c158ad0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7fd43800c9f0 tx=0x7fd438008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:05.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.102+0000 7fd429ffb700 1 -- 172.21.15.187:0/383573821 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43800a800 con 0x7fd4240049d0 2020-11-20T12:12:05.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.102+0000 7fd429ffb700 1 -- 172.21.15.187:0/383573821 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43800a960 con 0x7fd4240049d0 2020-11-20T12:12:05.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.102+0000 7fd429ffb700 1 -- 172.21.15.187:0/383573821 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43800ace0 con 0x7fd4240049d0 2020-11-20T12:12:05.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.102+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43c149920 con 0x7fd4240049d0 2020-11-20T12:12:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.102+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43c158640 con 0x7fd4240049d0 2020-11-20T12:12:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.103+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43c0b3970 con 0x7fd4240049d0 2020-11-20T12:12:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.104+0000 7fd429ffb700 1 -- 172.21.15.187:0/383573821 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd43800e070 con 0x7fd4240049d0 2020-11-20T12:12:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.104+0000 7fd429ffb700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd414056d50 0x7fd414059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.104+0000 7fd429ffb700 1 -- 172.21.15.187:0/383573821 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd43806fce0 con 0x7fd4240049d0 2020-11-20T12:12:05.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.105+0000 7fd4417af700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd414056d50 0x7fd414059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.105+0000 7fd4417af700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd414056d50 0x7fd414059210 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fd43c09d8a0 tx=0x7fd42c001530).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:05.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.108+0000 7fd429ffb700 1 -- 172.21.15.187:0/383573821 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd438045400 con 0x7fd4240049d0 2020-11-20T12:12:05.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.362+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd43c061a20 con 0x7fd4240049d0 2020-11-20T12:12:05.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.364+0000 7fd429ffb700 1 -- 172.21.15.187:0/383573821 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+4570 (secure 0 0 0) 0x7fd438015020 con 0x7fd4240049d0 2020-11-20T12:12:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:05.369 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:12:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:12:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:12:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:12:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:12:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:12:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:12:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:12:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:12:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:12:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:12:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:12:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:12:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:12:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:12:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:12:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:12:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:12:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:12:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:12:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:12:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:12:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:12:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:12:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:12:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:12:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:12:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:12:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:12:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:12:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:12:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:12:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:12:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:12:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:12:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:12:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:12:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:12:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:12:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:12:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:12:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:12:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:12:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:12:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:12:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:12:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:12:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:12:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:12:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:12:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:12:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:12:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:12:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:12:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:12:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:12:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:12:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:12:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:12:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:12:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:12:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:12:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:12:05.406 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:12:05.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd414056d50 msgr2=0x7fd414059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd414056d50 0x7fd414059210 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fd43c09d8a0 tx=0x7fd42c001530).stop 2020-11-20T12:12:05.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 msgr2=0x7fd43c158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd43c158ad0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7fd43800c9f0 tx=0x7fd438008040).stop 2020-11-20T12:12:05.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 shutdown_connections 2020-11-20T12:12:05.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd414056d50 0x7fd414059210 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd43c144300 0x7fd43c15b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4240049d0 0x7fd43c158ad0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 --2- 172.21.15.187:0/383573821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd43c1581c0 0x7fd43c143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 >> 172.21.15.187:0/383573821 conn(0x7fd43c0bab30 msgr2=0x7fd43c146d80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:05.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.366+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 shutdown_connections 2020-11-20T12:12:05.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.367+0000 7fd442fb2700 1 -- 172.21.15.187:0/383573821 wait complete. 2020-11-20T12:12:05.410 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:12:05.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.650+0000 7fe89c81e700 1 Processor -- start 2020-11-20T12:12:05.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.651+0000 7fe89c81e700 1 -- start start 2020-11-20T12:12:05.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.652+0000 7fe89c81e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe8980b6650 0x7fe8980afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.652+0000 7fe89c81e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980b1300 con 0x7fe8981484d0 2020-11-20T12:12:05.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.652+0000 7fe89c81e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980b15a0 con 0x7fe898066f70 2020-11-20T12:12:05.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.652+0000 7fe89c81e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe898067730 con 0x7fe8980b6650 2020-11-20T12:12:05.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.653+0000 7fe89759e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe898066f70 0x7fe8980aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43328/0 (socket says 172.21.15.187:43328) 2020-11-20T12:12:05.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.653+0000 7fe89759e700 1 -- 172.21.15.187:0/3778231747 learned_addr learned my addr 172.21.15.187:0/3778231747 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:05.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.653+0000 7fe88e59c700 1 --2- 172.21.15.187:0/3778231747 >> v2:172.21.15.187:3300/0 conn(0x7fe8980b6650 0x7fe8980afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.653+0000 7fe89659c700 1 -- 172.21.15.187:0/3778231747 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300310073 0 0) 0x7fe8980b15a0 con 0x7fe898066f70 2020-11-20T12:12:05.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.653+0000 7fe89659c700 1 -- 172.21.15.187:0/3778231747 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe878003240 con 0x7fe898066f70 2020-11-20T12:12:05.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.654+0000 7fe88e59c700 1 -- 172.21.15.187:0/3778231747 >> v1:172.21.15.187:6789/0 conn(0x7fe898066f70 legacy=0x7fe8980aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.654+0000 7fe88e59c700 1 -- 172.21.15.187:0/3778231747 >> v1:172.21.15.5:6789/0 conn(0x7fe8981484d0 legacy=0x7fe898061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.654+0000 7fe88e59c700 1 -- 172.21.15.187:0/3778231747 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980b1840 con 0x7fe8980b6650 2020-11-20T12:12:05.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.654+0000 7fe88e59c700 1 --2- 172.21.15.187:0/3778231747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8980b6650 0x7fe8980afa90 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7fe88000b9a0 tx=0x7fe880006df0).ready entity=mon.1 client_cookie=74f5527ad82088c5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:05.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.654+0000 7fe89659c700 1 -- 172.21.15.187:0/3778231747 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8800097b0 con 0x7fe8980b6650 2020-11-20T12:12:05.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.655+0000 7fe89659c700 1 -- 172.21.15.187:0/3778231747 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880009910 con 0x7fe8980b6650 2020-11-20T12:12:05.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.655+0000 7fe89659c700 1 -- 172.21.15.187:0/3778231747 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880009c90 con 0x7fe8980b6650 2020-11-20T12:12:05.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.655+0000 7fe89c81e700 1 -- 172.21.15.187:0/3778231747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8980b6650 msgr2=0x7fe8980afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.655+0000 7fe89c81e700 1 --2- 172.21.15.187:0/3778231747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8980b6650 0x7fe8980afa90 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7fe88000b9a0 tx=0x7fe880006df0).stop 2020-11-20T12:12:05.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.655+0000 7fe89c81e700 1 -- 172.21.15.187:0/3778231747 shutdown_connections 2020-11-20T12:12:05.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.655+0000 7fe89c81e700 1 --2- 172.21.15.187:0/3778231747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8980b6650 0x7fe8980afa90 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.655+0000 7fe89c81e700 1 -- 172.21.15.187:0/3778231747 >> 172.21.15.187:0/3778231747 conn(0x7fe8980bab30 msgr2=0x7fe8980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:05.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.655+0000 7fe89c81e700 1 -- 172.21.15.187:0/3778231747 shutdown_connections 2020-11-20T12:12:05.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.656+0000 7fe89c81e700 1 -- 172.21.15.187:0/3778231747 wait complete. 2020-11-20T12:12:05.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.657+0000 7fe89c81e700 1 Processor -- start 2020-11-20T12:12:05.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.658+0000 7fe89c81e700 1 -- start start 2020-11-20T12:12:05.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe89c81e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe898066f70 0x7fe89809b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe89c81e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8981484d0 0x7fe898065790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe89c81e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8981574a0 0x7fe898142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe89c81e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89809b6d0 con 0x7fe8981484d0 2020-11-20T12:12:05.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe89c81e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89809b970 con 0x7fe898066f70 2020-11-20T12:12:05.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe89c81e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89809bc10 con 0x7fe8981574a0 2020-11-20T12:12:05.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe896d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe898066f70 0x7fe89809b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe896d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe898066f70 0x7fe89809b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46724/0 (socket says 172.21.15.187:46724) 2020-11-20T12:12:05.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe896d9d700 1 -- 172.21.15.187:0/3166670131 learned_addr learned my addr 172.21.15.187:0/3166670131 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:05.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe88e59c700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8981484d0 0x7fe898065790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe89759e700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8981574a0 0x7fe898142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe896d9d700 1 -- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8981574a0 msgr2=0x7fe898142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe896d9d700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8981574a0 0x7fe898142bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.659+0000 7fe896d9d700 1 -- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8981484d0 msgr2=0x7fe898065790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe896d9d700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8981484d0 0x7fe898065790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe896d9d700 1 -- 172.21.15.187:0/3166670131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898065c80 con 0x7fe898066f70 2020-11-20T12:12:05.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe88e59c700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8981484d0 0x7fe898065790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:05.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe896d9d700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe898066f70 0x7fe89809b0e0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7fe88800c6e0 tx=0x7fe888007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:05.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe88f7fe700 1 -- 172.21.15.187:0/3166670131 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888007020 con 0x7fe898066f70 2020-11-20T12:12:05.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe88f7fe700 1 -- 172.21.15.187:0/3166670131 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe888007180 con 0x7fe898066f70 2020-11-20T12:12:05.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89809c0b0 con 0x7fe898066f70 2020-11-20T12:12:05.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe88f7fe700 1 -- 172.21.15.187:0/3166670131 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88800e560 con 0x7fe898066f70 2020-11-20T12:12:05.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.660+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe898143260 con 0x7fe898066f70 2020-11-20T12:12:05.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.661+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89809bdd0 con 0x7fe898066f70 2020-11-20T12:12:05.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.662+0000 7fe88f7fe700 1 -- 172.21.15.187:0/3166670131 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe888010070 con 0x7fe898066f70 2020-11-20T12:12:05.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.663+0000 7fe88f7fe700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe874056d20 0x7fe8740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:05.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.663+0000 7fe88f7fe700 1 -- 172.21.15.187:0/3166670131 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe88806ef60 con 0x7fe898066f70 2020-11-20T12:12:05.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.663+0000 7fe88e59c700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe874056d20 0x7fe8740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:05.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.664+0000 7fe88e59c700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe874056d20 0x7fe8740591e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fe8980641f0 tx=0x7fe880015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:05.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.667+0000 7fe88f7fe700 1 -- 172.21.15.187:0/3166670131 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe8880446e0 con 0x7fe898066f70 2020-11-20T12:12:05.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.857+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8981434b0 con 0x7fe898066f70 2020-11-20T12:12:05.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.859+0000 7fe88f7fe700 1 -- 172.21.15.187:0/3166670131 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13628 (secure 0 0 0) 0x7fe888047d00 con 0x7fe898066f70 2020-11-20T12:12:05.863 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:05.863 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":77,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:12:04.803531+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:12:05.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.861+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe874056d20 msgr2=0x7fe8740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.861+0000 7fe89c81e700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe874056d20 0x7fe8740591e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fe8980641f0 tx=0x7fe880015000).stop 2020-11-20T12:12:05.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe898066f70 msgr2=0x7fe89809b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:05.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe898066f70 0x7fe89809b0e0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7fe88800c6e0 tx=0x7fe888007c50).stop 2020-11-20T12:12:05.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 shutdown_connections 2020-11-20T12:12:05.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe874056d20 0x7fe8740591e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8981574a0 0x7fe898142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe898066f70 0x7fe89809b0e0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 --2- 172.21.15.187:0/3166670131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8981484d0 0x7fe898065790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:05.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 >> 172.21.15.187:0/3166670131 conn(0x7fe8980bab30 msgr2=0x7fe8980b9120 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:05.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 shutdown_connections 2020-11-20T12:12:05.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:05.862+0000 7fe89c81e700 1 -- 172.21.15.187:0/3166670131 wait complete. 2020-11-20T12:12:05.889 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:12:05.891 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:12:06.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.140+0000 7f72a1d62700 1 Processor -- start 2020-11-20T12:12:06.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.141+0000 7f72a1d62700 1 -- start start 2020-11-20T12:12:06.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.142+0000 7f72a1d62700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c065570 0x7f729c09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.142+0000 7f72a1d62700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c0b13e0 con 0x7f729c065f90 2020-11-20T12:12:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.142+0000 7f72a1d62700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c09d690 con 0x7f729c065190 2020-11-20T12:12:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.142+0000 7f72a1d62700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c14ab30 con 0x7f729c065570 2020-11-20T12:12:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.142+0000 7f72a0d60700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f729c065190 0x7f729c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43336/0 (socket says 172.21.15.187:43336) 2020-11-20T12:12:06.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.142+0000 7f72a0d60700 1 -- 172.21.15.187:0/2730936661 learned_addr learned my addr 172.21.15.187:0/2730936661 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:06.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72a1561700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c065570 0x7f729c09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519690617 0 0) 0x7f729c09d690 con 0x7f729c065190 2020-11-20T12:12:06.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7288003240 con 0x7f729c065190 2020-11-20T12:12:06.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2059972977 0 0) 0x7f7288003240 con 0x7f729c065190 2020-11-20T12:12:06.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c065570 msgr2=0x7f729c09d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:06.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72937fe700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c065570 0x7f729c09d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 >> v1:172.21.15.5:6789/0 conn(0x7f729c065f90 legacy=0x7f729c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c062c10 con 0x7f729c065190 2020-11-20T12:12:06.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.143+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f72980014e0 con 0x7f729c065190 2020-11-20T12:12:06.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 >> v1:172.21.15.187:6789/0 conn(0x7f729c065190 legacy=0x7f729c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72937fe700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7288003730 0x7f7288003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72937fe700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c065570 0x7f72880044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72937fe700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72880049d0 0x7f7288004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c14ab30 con 0x7f7288003730 2020-11-20T12:12:06.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72880041d0 con 0x7f72880049d0 2020-11-20T12:12:06.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728800b730 con 0x7f729c065570 2020-11-20T12:12:06.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72a1561700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72880049d0 0x7f7288004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72a0d60700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7288003730 0x7f7288003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72a1561700 1 -- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c065570 msgr2=0x7f72880044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:06.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.144+0000 7f72a1561700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c065570 0x7f72880044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:06.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.145+0000 7f72a1561700 1 -- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7288003730 msgr2=0x7f7288003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:06.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.145+0000 7f72a1561700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7288003730 0x7f7288003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.145+0000 7f72a1561700 1 -- 172.21.15.187:0/2730936661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7288003240 con 0x7f72880049d0 2020-11-20T12:12:06.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.145+0000 7f72a1561700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72880049d0 0x7f7288004e50 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f7294009ea0 tx=0x7f72940091d0).ready entity=mon.1 client_cookie=25b1994546c546e8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.145+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940095c0 con 0x7f72880049d0 2020-11-20T12:12:06.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.145+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729400eda0 con 0x7f72880049d0 2020-11-20T12:12:06.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.145+0000 7f72937fe700 1 -- 172.21.15.187:0/2730936661 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294012600 con 0x7f72880049d0 2020-11-20T12:12:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.146+0000 7f72a1d62700 1 -- 172.21.15.187:0/2730936661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72880049d0 msgr2=0x7f7288004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.146+0000 7f72a1d62700 1 --2- 172.21.15.187:0/2730936661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72880049d0 0x7f7288004e50 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f7294009ea0 tx=0x7f72940091d0).stop 2020-11-20T12:12:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.146+0000 7f72a1561700 1 -- 172.21.15.187:0/2730936661 reap_dead start 2020-11-20T12:12:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.146+0000 7f72a1d62700 1 -- 172.21.15.187:0/2730936661 shutdown_connections 2020-11-20T12:12:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.146+0000 7f72a1d62700 1 -- 172.21.15.187:0/2730936661 >> 172.21.15.187:0/2730936661 conn(0x7f729c0bab30 msgr2=0x7f729c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.146+0000 7f72a1d62700 1 -- 172.21.15.187:0/2730936661 shutdown_connections 2020-11-20T12:12:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.146+0000 7f72a1d62700 1 -- 172.21.15.187:0/2730936661 wait complete. 2020-11-20T12:12:06.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.147+0000 7f72a1d62700 1 Processor -- start 2020-11-20T12:12:06.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.148+0000 7f72a1d62700 1 -- start start 2020-11-20T12:12:06.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1d62700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f729c159720 0x7f729c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1d62700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c090290 0x7f729c148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1d62700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c148f90 0x7f729c091e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1d62700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c149410 con 0x7f729c159720 2020-11-20T12:12:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1d62700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c159ff0 con 0x7f729c148f90 2020-11-20T12:12:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1d62700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c15a260 con 0x7f729c090290 2020-11-20T12:12:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1561700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c148f90 0x7f729c091e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1561700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c148f90 0x7f729c091e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46746/0 (socket says 172.21.15.187:46746) 2020-11-20T12:12:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1561700 1 -- 172.21.15.187:0/3305939578 learned_addr learned my addr 172.21.15.187:0/3305939578 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f7293fff700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c090290 0x7f729c148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a0d60700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f729c159720 0x7f729c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1561700 1 -- 172.21.15.187:0/3305939578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c090290 msgr2=0x7f729c148aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1561700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c090290 0x7f729c148aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1561700 1 -- 172.21.15.187:0/3305939578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f729c159720 msgr2=0x7f729c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.149+0000 7f72a1561700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f729c159720 0x7f729c159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.150+0000 7f72a1561700 1 -- 172.21.15.187:0/3305939578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c08f540 con 0x7f729c148f90 2020-11-20T12:12:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.150+0000 7f72a1561700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c148f90 0x7f729c091e50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7f7294009e70 tx=0x7f729401a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.150+0000 7f7290ff9700 1 -- 172.21.15.187:0/3305939578 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294024390 con 0x7f729c148f90 2020-11-20T12:12:06.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.150+0000 7f7290ff9700 1 -- 172.21.15.187:0/3305939578 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72940244f0 con 0x7f729c148f90 2020-11-20T12:12:06.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.150+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c08f9e0 con 0x7f729c148f90 2020-11-20T12:12:06.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.150+0000 7f7290ff9700 1 -- 172.21.15.187:0/3305939578 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940248b0 con 0x7f729c148f90 2020-11-20T12:12:06.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.150+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c090a70 con 0x7f729c148f90 2020-11-20T12:12:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.151+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c15a420 con 0x7f729c148f90 2020-11-20T12:12:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.152+0000 7f7290ff9700 1 -- 172.21.15.187:0/3305939578 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f729402c070 con 0x7f729c148f90 2020-11-20T12:12:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.152+0000 7f7290ff9700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7274056d20 0x7f72740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.153+0000 7f7290ff9700 1 -- 172.21.15.187:0/3305939578 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f72940720c0 con 0x7f729c148f90 2020-11-20T12:12:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.153+0000 7f72a0d60700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7274056d20 0x7f72740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.154+0000 7f72a0d60700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7274056d20 0x7f72740591e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f7298001610 tx=0x7f7298009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.157+0000 7f7290ff9700 1 -- 172.21.15.187:0/3305939578 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7294047840 con 0x7f729c148f90 2020-11-20T12:12:06.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.348+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f729c061a20 con 0x7f729c148f90 2020-11-20T12:12:06.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.349+0000 7f7290ff9700 1 -- 172.21.15.187:0/3305939578 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v77) v1 ==== 110+0+10 (secure 0 0 0) 0x7f7294016070 con 0x7f729c148f90 2020-11-20T12:12:06.352 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:12:06.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7274056d20 msgr2=0x7f72740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7274056d20 0x7f72740591e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f7298001610 tx=0x7f7298009040).stop 2020-11-20T12:12:06.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c148f90 msgr2=0x7f729c091e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c148f90 0x7f729c091e50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7f7294009e70 tx=0x7f729401a000).stop 2020-11-20T12:12:06.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 shutdown_connections 2020-11-20T12:12:06.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7274056d20 0x7f72740591e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f729c159720 0x7f729c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c148f90 0x7f729c091e50 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 --2- 172.21.15.187:0/3305939578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f729c090290 0x7f729c148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 >> 172.21.15.187:0/3305939578 conn(0x7f729c0bab30 msgr2=0x7f729c149c20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:06.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 shutdown_connections 2020-11-20T12:12:06.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.352+0000 7f72a1d62700 1 -- 172.21.15.187:0/3305939578 wait complete. 2020-11-20T12:12:06.378 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:12:06.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.619+0000 7fd6b92e1700 1 Processor -- start 2020-11-20T12:12:06.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.619+0000 7fd6b92e1700 1 -- start start 2020-11-20T12:12:06.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.621+0000 7fd6b92e1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b4067820 0x7fd6b40b1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.621+0000 7fd6b92e1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b40b0f20 con 0x7fd6b4145510 2020-11-20T12:12:06.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.621+0000 7fd6b92e1700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b40b11c0 con 0x7fd6b4067c00 2020-11-20T12:12:06.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.621+0000 7fd6b92e1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40b1460 con 0x7fd6b4067820 2020-11-20T12:12:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.621+0000 7fd6ab7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd6b4067c00 0x7fd6b4144e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43358/0 (socket says 172.21.15.187:43358) 2020-11-20T12:12:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.621+0000 7fd6ab7fe700 1 -- 172.21.15.187:0/4249109773 learned_addr learned my addr 172.21.15.187:0/4249109773 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.622+0000 7fd6b3fff700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b4067820 0x7fd6b40b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.622+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988647321 0 0) 0x7fd6b40b11c0 con 0x7fd6b4067c00 2020-11-20T12:12:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.622+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd694003240 con 0x7fd6b4067c00 2020-11-20T12:12:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.622+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 792051410 0 0) 0x7fd694003240 con 0x7fd6b4067c00 2020-11-20T12:12:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.622+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b4067820 msgr2=0x7fd6b40b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.622+0000 7fd6b37fe700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b4067820 0x7fd6b40b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 >> v1:172.21.15.5:6789/0 conn(0x7fd6b4145510 legacy=0x7fd6b40b66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b40b0be0 con 0x7fd6b4067c00 2020-11-20T12:12:06.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd69c0014e0 con 0x7fd6b4067c00 2020-11-20T12:12:06.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 >> v1:172.21.15.187:6789/0 conn(0x7fd6b4067c00 legacy=0x7fd6b4144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b4067820 0x7fd694003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd694003c90 0x7fd6940044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6940049d0 0x7fd694004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40b1460 con 0x7fd694003c90 2020-11-20T12:12:06.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd694004220 con 0x7fd6940049d0 2020-11-20T12:12:06.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69400b730 con 0x7fd6b4067820 2020-11-20T12:12:06.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.623+0000 7fd6b8ae0700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6940049d0 0x7fd694004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b3fff700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b4067820 0x7fd694003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6ab7fe700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd694003c90 0x7fd6940044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b8ae0700 1 -- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b4067820 msgr2=0x7fd694003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:06.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b8ae0700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b4067820 0x7fd694003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b8ae0700 1 -- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd694003c90 msgr2=0x7fd6940044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:06.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b8ae0700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd694003c90 0x7fd6940044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b8ae0700 1 -- 172.21.15.187:0/4249109773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd694003240 con 0x7fd6940049d0 2020-11-20T12:12:06.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6ab7fe700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd694003c90 0x7fd6940044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:06.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b8ae0700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6940049d0 0x7fd694004e50 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7fd6ac001e70 tx=0x7fd6ac007f10).ready entity=mon.1 client_cookie=45b95612fe9fe89c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac001540 con 0x7fd6940049d0 2020-11-20T12:12:06.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac003030 con 0x7fd6940049d0 2020-11-20T12:12:06.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.624+0000 7fd6b37fe700 1 -- 172.21.15.187:0/4249109773 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00cd90 con 0x7fd6940049d0 2020-11-20T12:12:06.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.625+0000 7fd6b92e1700 1 -- 172.21.15.187:0/4249109773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6940049d0 msgr2=0x7fd694004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.625+0000 7fd6b92e1700 1 --2- 172.21.15.187:0/4249109773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6940049d0 0x7fd694004e50 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7fd6ac001e70 tx=0x7fd6ac007f10).stop 2020-11-20T12:12:06.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.625+0000 7fd6b8ae0700 1 -- 172.21.15.187:0/4249109773 reap_dead start 2020-11-20T12:12:06.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.625+0000 7fd6b92e1700 1 -- 172.21.15.187:0/4249109773 shutdown_connections 2020-11-20T12:12:06.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.625+0000 7fd6b92e1700 1 -- 172.21.15.187:0/4249109773 >> 172.21.15.187:0/4249109773 conn(0x7fd6b40bbc80 msgr2=0x7fd6b40bc090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:06.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.625+0000 7fd6b92e1700 1 -- 172.21.15.187:0/4249109773 shutdown_connections 2020-11-20T12:12:06.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.625+0000 7fd6b92e1700 1 -- 172.21.15.187:0/4249109773 wait complete. 2020-11-20T12:12:06.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.627+0000 7fd6b92e1700 1 Processor -- start 2020-11-20T12:12:06.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.627+0000 7fd6b92e1700 1 -- start start 2020-11-20T12:12:06.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b92e1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6b4149080 0x7fd6b409dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b92e1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6b409cc50 0x7fd6b409d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b92e1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b41431d0 0x7fd6b4143650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b92e1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40b5670 con 0x7fd6b409cc50 2020-11-20T12:12:06.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b92e1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b414a3a0 con 0x7fd6b4149080 2020-11-20T12:12:06.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b92e1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b409e2f0 con 0x7fd6b41431d0 2020-11-20T12:12:06.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6b4149080 0x7fd6b409dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6b4149080 0x7fd6b409dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46762/0 (socket says 172.21.15.187:46762) 2020-11-20T12:12:06.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b3fff700 1 -- 172.21.15.187:0/3904851163 learned_addr learned my addr 172.21.15.187:0/3904851163 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:06.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6ab7fe700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6b409cc50 0x7fd6b409d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b3fff700 1 -- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b41431d0 msgr2=0x7fd6b4143650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b8ae0700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b41431d0 0x7fd6b4143650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.628+0000 7fd6b3fff700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b41431d0 0x7fd6b4143650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b3fff700 1 -- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6b409cc50 msgr2=0x7fd6b409d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b3fff700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6b409cc50 0x7fd6b409d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b3fff700 1 -- 172.21.15.187:0/3904851163 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b40b4f30 con 0x7fd6b4149080 2020-11-20T12:12:06.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6ab7fe700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6b409cc50 0x7fd6b409d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:06.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b3fff700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6b4149080 0x7fd6b409dc30 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fd6a4010c20 tx=0x7fd6a4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b0ff9700 1 -- 172.21.15.187:0/3904851163 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400e960 con 0x7fd6b4149080 2020-11-20T12:12:06.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b0ff9700 1 -- 172.21.15.187:0/3904851163 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a400eac0 con 0x7fd6b4149080 2020-11-20T12:12:06.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b0ff9700 1 -- 172.21.15.187:0/3904851163 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400ee40 con 0x7fd6b4149080 2020-11-20T12:12:06.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b40b51d0 con 0x7fd6b4149080 2020-11-20T12:12:06.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.629+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b4149460 con 0x7fd6b4149080 2020-11-20T12:12:06.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.630+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b409e120 con 0x7fd6b4149080 2020-11-20T12:12:06.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.631+0000 7fd6b0ff9700 1 -- 172.21.15.187:0/3904851163 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd6a4024070 con 0x7fd6b4149080 2020-11-20T12:12:06.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.631+0000 7fd6b0ff9700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd690056d20 0x7fd6900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.632+0000 7fd6b0ff9700 1 -- 172.21.15.187:0/3904851163 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd6a40706b0 con 0x7fd6b4149080 2020-11-20T12:12:06.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.632+0000 7fd6ab7fe700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd690056d20 0x7fd6900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.632+0000 7fd6ab7fe700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd690056d20 0x7fd6900591e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fd6b40b66e0 tx=0x7fd69c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.635+0000 7fd6b0ff9700 1 -- 172.21.15.187:0/3904851163 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd6a4045d20 con 0x7fd6b4149080 2020-11-20T12:12:06.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.824+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd6b409b2e0 con 0x7fd6b4149080 2020-11-20T12:12:06.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.825+0000 7fd6b0ff9700 1 -- 172.21.15.187:0/3904851163 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v77) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd6a4049340 con 0x7fd6b4149080 2020-11-20T12:12:06.828 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:12:06.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd690056d20 msgr2=0x7fd6900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd690056d20 0x7fd6900591e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fd6b40b66e0 tx=0x7fd69c009040).stop 2020-11-20T12:12:06.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6b4149080 msgr2=0x7fd6b409dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6b4149080 0x7fd6b409dc30 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fd6a4010c20 tx=0x7fd6a4008040).stop 2020-11-20T12:12:06.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 shutdown_connections 2020-11-20T12:12:06.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd690056d20 0x7fd6900591e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd6b41431d0 0x7fd6b4143650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6b4149080 0x7fd6b409dc30 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 --2- 172.21.15.187:0/3904851163 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6b409cc50 0x7fd6b409d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.827+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 >> 172.21.15.187:0/3904851163 conn(0x7fd6b40bbc80 msgr2=0x7fd6b40a0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:06.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.828+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 shutdown_connections 2020-11-20T12:12:06.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:06.828+0000 7fd6b92e1700 1 -- 172.21.15.187:0/3904851163 wait complete. 2020-11-20T12:12:06.854 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:12:07.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.095+0000 7fc6f759e700 1 Processor -- start 2020-11-20T12:12:07.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.095+0000 7fc6f759e700 1 -- start start 2020-11-20T12:12:07.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.096+0000 7fc6f759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f8065190 0x7fc6f8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.096+0000 7fc6f759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6f8065f90 0x7fc6f8143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6f759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc6f8143c70 0x7fc6f809ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6f759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f814ab30 con 0x7fc6f8143c70 2020-11-20T12:12:07.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6f759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f80b13e0 con 0x7fc6f8065f90 2020-11-20T12:12:07.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6f759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f8062c10 con 0x7fc6f8065190 2020-11-20T12:12:07.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6edd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6f8065f90 0x7fc6f8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6edd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6f8065f90 0x7fc6f8143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46774/0 (socket says 172.21.15.187:46774) 2020-11-20T12:12:07.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6edd9b700 1 -- 172.21.15.187:0/941432012 learned_addr learned my addr 172.21.15.187:0/941432012 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:07.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6f6d9d700 1 --2- 172.21.15.187:0/941432012 >> v2:172.21.15.5:3300/0 conn(0x7fc6f8143c70 0x7fc6f809ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.097+0000 7fc6f659c700 1 --2- 172.21.15.187:0/941432012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f8065190 0x7fc6f8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.098+0000 7fc6edd9b700 1 -- 172.21.15.187:0/941432012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f8065190 msgr2=0x7fc6f8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:07.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.098+0000 7fc6edd9b700 1 --2- 172.21.15.187:0/941432012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f8065190 0x7fc6f8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.098+0000 7fc6edd9b700 1 -- 172.21.15.187:0/941432012 >> v2:172.21.15.5:3300/0 conn(0x7fc6f8143c70 msgr2=0x7fc6f809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:07.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.098+0000 7fc6edd9b700 1 --2- 172.21.15.187:0/941432012 >> v2:172.21.15.5:3300/0 conn(0x7fc6f8143c70 0x7fc6f809ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.098+0000 7fc6edd9b700 1 -- 172.21.15.187:0/941432012 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f80b1640 con 0x7fc6f8065f90 2020-11-20T12:12:07.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.098+0000 7fc6f659c700 1 --2- 172.21.15.187:0/941432012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f8065190 0x7fc6f8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:07.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.098+0000 7fc6edd9b700 1 --2- 172.21.15.187:0/941432012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8065f90 0x7fc6f8143780 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fc6f80632e0 tx=0x7fc6e0006eb0).ready entity=mon.1 client_cookie=566b318455547d0b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:07.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f5d9b700 1 -- 172.21.15.187:0/941432012 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e00098a0 con 0x7fc6f8065f90 2020-11-20T12:12:07.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f5d9b700 1 -- 172.21.15.187:0/941432012 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0009a00 con 0x7fc6f8065f90 2020-11-20T12:12:07.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f5d9b700 1 -- 172.21.15.187:0/941432012 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0009d80 con 0x7fc6f8065f90 2020-11-20T12:12:07.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f759e700 1 -- 172.21.15.187:0/941432012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8065f90 msgr2=0x7fc6f8143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:07.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f759e700 1 --2- 172.21.15.187:0/941432012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8065f90 0x7fc6f8143780 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fc6f80632e0 tx=0x7fc6e0006eb0).stop 2020-11-20T12:12:07.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f759e700 1 -- 172.21.15.187:0/941432012 shutdown_connections 2020-11-20T12:12:07.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f759e700 1 --2- 172.21.15.187:0/941432012 >> v2:172.21.15.5:3300/0 conn(0x7fc6f8143c70 0x7fc6f809ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f759e700 1 --2- 172.21.15.187:0/941432012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f8065190 0x7fc6f8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f759e700 1 --2- 172.21.15.187:0/941432012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8065f90 0x7fc6f8143780 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f759e700 1 -- 172.21.15.187:0/941432012 >> 172.21.15.187:0/941432012 conn(0x7fc6f80bab30 msgr2=0x7fc6f80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:07.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.099+0000 7fc6f759e700 1 -- 172.21.15.187:0/941432012 shutdown_connections 2020-11-20T12:12:07.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.100+0000 7fc6f759e700 1 -- 172.21.15.187:0/941432012 wait complete. 2020-11-20T12:12:07.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.101+0000 7fc6f759e700 1 Processor -- start 2020-11-20T12:12:07.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.101+0000 7fc6f759e700 1 -- start start 2020-11-20T12:12:07.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.102+0000 7fc6f759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6f8065190 0x7fc6f80b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.102+0000 7fc6f759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8143c70 0x7fc6f80b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.102+0000 7fc6f759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f809dae0 0x7fc6f809df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.102+0000 7fc6f759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f80b76a0 con 0x7fc6f8065190 2020-11-20T12:12:07.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.102+0000 7fc6f759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f80b7940 con 0x7fc6f8143c70 2020-11-20T12:12:07.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.102+0000 7fc6f759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f80b7be0 con 0x7fc6f809dae0 2020-11-20T12:12:07.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.102+0000 7fc6edd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8143c70 0x7fc6f80b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6edd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8143c70 0x7fc6f80b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46780/0 (socket says 172.21.15.187:46780) 2020-11-20T12:12:07.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6edd9b700 1 -- 172.21.15.187:0/420746103 learned_addr learned my addr 172.21.15.187:0/420746103 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:07.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6f659c700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6f8065190 0x7fc6f80b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6f6d9d700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f809dae0 0x7fc6f809df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6edd9b700 1 -- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f809dae0 msgr2=0x7fc6f809df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:07.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6edd9b700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f809dae0 0x7fc6f809df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6edd9b700 1 -- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6f8065190 msgr2=0x7fc6f80b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:07.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6edd9b700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6f8065190 0x7fc6f80b1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6edd9b700 1 -- 172.21.15.187:0/420746103 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f80b7390 con 0x7fc6f8143c70 2020-11-20T12:12:07.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6f6d9d700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f809dae0 0x7fc6f809df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:07.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.103+0000 7fc6edd9b700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8143c70 0x7fc6f80b4380 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7fc6e0010000 tx=0x7fc6e0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:07.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.104+0000 7fc6eeffd700 1 -- 172.21.15.187:0/420746103 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0011c80 con 0x7fc6f8143c70 2020-11-20T12:12:07.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.104+0000 7fc6eeffd700 1 -- 172.21.15.187:0/420746103 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0011de0 con 0x7fc6f8143c70 2020-11-20T12:12:07.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.104+0000 7fc6eeffd700 1 -- 172.21.15.187:0/420746103 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e001e690 con 0x7fc6f8143c70 2020-11-20T12:12:07.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.104+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f80b2400 con 0x7fc6f8143c70 2020-11-20T12:12:07.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.104+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f80b4a30 con 0x7fc6f8143c70 2020-11-20T12:12:07.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.105+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f8061a20 con 0x7fc6f8143c70 2020-11-20T12:12:07.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.106+0000 7fc6eeffd700 1 -- 172.21.15.187:0/420746103 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc6e0025030 con 0x7fc6f8143c70 2020-11-20T12:12:07.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.106+0000 7fc6eeffd700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d4056d20 0x7fc6d40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.106+0000 7fc6eeffd700 1 -- 172.21.15.187:0/420746103 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc6e0071350 con 0x7fc6f8143c70 2020-11-20T12:12:07.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.107+0000 7fc6f659c700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d4056d20 0x7fc6d40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.107+0000 7fc6f659c700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d4056d20 0x7fc6d40591e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc6e800ba80 tx=0x7fc6e8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:07.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.110+0000 7fc6eeffd700 1 -- 172.21.15.187:0/420746103 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc6e0046b80 con 0x7fc6f8143c70 2020-11-20T12:12:07.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.297+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc6f80afc60 con 0x7fc6f8143c70 2020-11-20T12:12:07.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.298+0000 7fc6eeffd700 1 -- 172.21.15.187:0/420746103 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+143 (secure 0 0 0) 0x7fc6e0023070 con 0x7fc6f8143c70 2020-11-20T12:12:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:07.301 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:12:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:12:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:12:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:12:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:12:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:12:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:12:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:12:07.303 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:12:07.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.300+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d4056d20 msgr2=0x7fc6d40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:07.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.300+0000 7fc6f759e700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d4056d20 0x7fc6d40591e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc6e800ba80 tx=0x7fc6e8008040).stop 2020-11-20T12:12:07.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.300+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8143c70 msgr2=0x7fc6f80b4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:07.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.300+0000 7fc6f759e700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8143c70 0x7fc6f80b4380 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7fc6e0010000 tx=0x7fc6e0015000).stop 2020-11-20T12:12:07.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.301+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 shutdown_connections 2020-11-20T12:12:07.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.301+0000 7fc6f759e700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d4056d20 0x7fc6d40591e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.301+0000 7fc6f759e700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6f8065190 0x7fc6f80b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.301+0000 7fc6f759e700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6f809dae0 0x7fc6f809df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.301+0000 7fc6f759e700 1 --2- 172.21.15.187:0/420746103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6f8143c70 0x7fc6f80b4380 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.301+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 >> 172.21.15.187:0/420746103 conn(0x7fc6f80bab30 msgr2=0x7fc6f8061d00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:07.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.301+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 shutdown_connections 2020-11-20T12:12:07.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.301+0000 7fc6f759e700 1 -- 172.21.15.187:0/420746103 wait complete. 2020-11-20T12:12:07.325 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:12:07.325 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:12:07.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.562+0000 7ff58bfff700 1 Processor -- start 2020-11-20T12:12:07.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.562+0000 7ff58bfff700 1 -- start start 2020-11-20T12:12:07.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff58c145cc0 0x7ff58c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff58c0683f0 0x7ff58c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 0x7ff58c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c14c170 con 0x7ff58c0683f0 2020-11-20T12:12:07.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c14c410 con 0x7ff58c145cc0 2020-11-20T12:12:07.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c14c6b0 con 0x7ff58c065680 2020-11-20T12:12:07.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff58c145cc0 0x7ff58c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff58c145cc0 0x7ff58c066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46786/0 (socket says 172.21.15.187:46786) 2020-11-20T12:12:07.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.564+0000 7ff58affd700 1 -- 172.21.15.187:0/1062896626 learned_addr learned my addr 172.21.15.187:0/1062896626 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:07.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.565+0000 7ff5827fc700 1 --2- 172.21.15.187:0/1062896626 >> v2:172.21.15.5:3300/0 conn(0x7ff58c0683f0 0x7ff58c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.565+0000 7ff58b7fe700 1 --2- 172.21.15.187:0/1062896626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 0x7ff58c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.565+0000 7ff58affd700 1 -- 172.21.15.187:0/1062896626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 msgr2=0x7ff58c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:07.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.565+0000 7ff58affd700 1 --2- 172.21.15.187:0/1062896626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 0x7ff58c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.565+0000 7ff58affd700 1 -- 172.21.15.187:0/1062896626 >> v2:172.21.15.5:3300/0 conn(0x7ff58c0683f0 msgr2=0x7ff58c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:07.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.565+0000 7ff58affd700 1 --2- 172.21.15.187:0/1062896626 >> v2:172.21.15.5:3300/0 conn(0x7ff58c0683f0 0x7ff58c065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.565+0000 7ff58affd700 1 -- 172.21.15.187:0/1062896626 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c14bdf0 con 0x7ff58c145cc0 2020-11-20T12:12:07.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.566+0000 7ff58affd700 1 --2- 172.21.15.187:0/1062896626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c066040 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7ff58c0b85e0 tx=0x7ff57c006eb0).ready entity=mon.1 client_cookie=4b10b6b8c0568e6d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:07.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.566+0000 7ff58a7fc700 1 -- 172.21.15.187:0/1062896626 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0098d0 con 0x7ff58c145cc0 2020-11-20T12:12:07.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.566+0000 7ff58a7fc700 1 -- 172.21.15.187:0/1062896626 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c009a30 con 0x7ff58c145cc0 2020-11-20T12:12:07.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.566+0000 7ff58a7fc700 1 -- 172.21.15.187:0/1062896626 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c009db0 con 0x7ff58c145cc0 2020-11-20T12:12:07.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.566+0000 7ff58bfff700 1 -- 172.21.15.187:0/1062896626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 msgr2=0x7ff58c066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:07.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.566+0000 7ff58bfff700 1 --2- 172.21.15.187:0/1062896626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c066040 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7ff58c0b85e0 tx=0x7ff57c006eb0).stop 2020-11-20T12:12:07.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.567+0000 7ff58bfff700 1 -- 172.21.15.187:0/1062896626 shutdown_connections 2020-11-20T12:12:07.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.567+0000 7ff58bfff700 1 --2- 172.21.15.187:0/1062896626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 0x7ff58c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.567+0000 7ff58bfff700 1 --2- 172.21.15.187:0/1062896626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c066040 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.567+0000 7ff58bfff700 1 --2- 172.21.15.187:0/1062896626 >> v2:172.21.15.5:3300/0 conn(0x7ff58c0683f0 0x7ff58c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.567+0000 7ff58bfff700 1 -- 172.21.15.187:0/1062896626 >> 172.21.15.187:0/1062896626 conn(0x7ff58c0bab30 msgr2=0x7ff58c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:07.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.567+0000 7ff58bfff700 1 -- 172.21.15.187:0/1062896626 shutdown_connections 2020-11-20T12:12:07.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.567+0000 7ff58bfff700 1 -- 172.21.15.187:0/1062896626 wait complete. 2020-11-20T12:12:07.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.569+0000 7ff58bfff700 1 Processor -- start 2020-11-20T12:12:07.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.569+0000 7ff58bfff700 1 -- start start 2020-11-20T12:12:07.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.570+0000 7ff58bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 0x7ff58c09fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.570+0000 7ff58bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0683f0 0x7ff58c143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.570+0000 7ff58bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.570+0000 7ff58bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c09b460 con 0x7ff58c0683f0 2020-11-20T12:12:07.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.570+0000 7ff58bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c09aa80 con 0x7ff58c145cc0 2020-11-20T12:12:07.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.570+0000 7ff58bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c09ad20 con 0x7ff58c065680 2020-11-20T12:12:07.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.570+0000 7ff58b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.570+0000 7ff58b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c144070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46794/0 (socket says 172.21.15.187:46794) 2020-11-20T12:12:07.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff58b7fe700 1 -- 172.21.15.187:0/605435274 learned_addr learned my addr 172.21.15.187:0/605435274 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:07.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff58affd700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 0x7ff58c09fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff5827fc700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0683f0 0x7ff58c143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff58b7fe700 1 -- 172.21.15.187:0/605435274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 msgr2=0x7ff58c09fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:07.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff58b7fe700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 0x7ff58c09fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff58b7fe700 1 -- 172.21.15.187:0/605435274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0683f0 msgr2=0x7ff58c143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:07.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff58b7fe700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0683f0 0x7ff58c143b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:07.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff58b7fe700 1 -- 172.21.15.187:0/605435274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c09afc0 con 0x7ff58c145cc0 2020-11-20T12:12:07.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff58b7fe700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c144070 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7ff58400eb70 tx=0x7ff584008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:07.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.571+0000 7ff5837fe700 1 -- 172.21.15.187:0/605435274 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58400c7a0 con 0x7ff58c145cc0 2020-11-20T12:12:07.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.572+0000 7ff5837fe700 1 -- 172.21.15.187:0/605435274 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58400c900 con 0x7ff58c145cc0 2020-11-20T12:12:07.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.572+0000 7ff5837fe700 1 -- 172.21.15.187:0/605435274 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58400cc80 con 0x7ff58c145cc0 2020-11-20T12:12:07.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.572+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58c0a02e0 con 0x7ff58c145cc0 2020-11-20T12:12:07.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.572+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c09a6c0 con 0x7ff58c145cc0 2020-11-20T12:12:07.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.573+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c09a8e0 con 0x7ff58c145cc0 2020-11-20T12:12:07.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.573+0000 7ff5837fe700 1 -- 172.21.15.187:0/605435274 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff584013430 con 0x7ff58c145cc0 2020-11-20T12:12:07.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.574+0000 7ff5837fe700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff568056d20 0x7ff5680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:07.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.574+0000 7ff5837fe700 1 -- 172.21.15.187:0/605435274 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff58406f920 con 0x7ff58c145cc0 2020-11-20T12:12:07.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.574+0000 7ff58affd700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff568056d20 0x7ff5680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:07.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.575+0000 7ff58affd700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff568056d20 0x7ff5680591e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ff57c006ef0 tx=0x7ff57c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:07.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.578+0000 7ff5837fe700 1 -- 172.21.15.187:0/605435274 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff58c09a8e0 con 0x7ff58c145cc0 2020-11-20T12:12:07.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:07.767+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7ff58c09a8e0 con 0x7ff58c145cc0 2020-11-20T12:12:08.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.768+0000 7ff5837fe700 1 -- 172.21.15.187:0/605435274 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff58c09a8e0 con 0x7ff58c145cc0 2020-11-20T12:12:08.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff568056d20 msgr2=0x7ff5680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:08.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff568056d20 0x7ff5680591e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ff57c006ef0 tx=0x7ff57c015000).stop 2020-11-20T12:12:08.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 msgr2=0x7ff58c144070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:08.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c144070 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7ff58400eb70 tx=0x7ff584008040).stop 2020-11-20T12:12:08.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 shutdown_connections 2020-11-20T12:12:08.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff568056d20 0x7ff5680591e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:08.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff58c145cc0 0x7ff58c144070 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:08.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff58c065680 0x7ff58c09fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:08.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 --2- 172.21.15.187:0/605435274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0683f0 0x7ff58c143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:08.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.773+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 >> 172.21.15.187:0/605435274 conn(0x7ff58c0bab30 msgr2=0x7ff58c145040 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:08.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.774+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 shutdown_connections 2020-11-20T12:12:08.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:08.774+0000 7ff58bfff700 1 -- 172.21.15.187:0/605435274 wait complete. 2020-11-20T12:12:08.783 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:12:08.813 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:12:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.057+0000 7f545759e700 1 Processor -- start 2020-11-20T12:12:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.057+0000 7f545759e700 1 -- start start 2020-11-20T12:12:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.058+0000 7f545759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5458065190 0x7f5458065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.058+0000 7f545759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f545809ca90 0x7f545809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.058+0000 7f545759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54580b13e0 con 0x7f5458065f90 2020-11-20T12:12:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.058+0000 7f545759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545814ab30 con 0x7f5458065190 2020-11-20T12:12:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.058+0000 7f545759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5458062c10 con 0x7f545809ca90 2020-11-20T12:12:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.059+0000 7f5456d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f545809ca90 0x7f545809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.059+0000 7f5456d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f545809ca90 0x7f545809cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46802/0 (socket says 172.21.15.187:46802) 2020-11-20T12:12:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.059+0000 7f5456d9d700 1 -- 172.21.15.187:0/2889844898 learned_addr learned my addr 172.21.15.187:0/2889844898 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.059+0000 7f545659c700 1 --2- 172.21.15.187:0/2889844898 >> v2:172.21.15.5:3300/0 conn(0x7f5458065190 0x7f5458065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.059+0000 7f5456d9d700 1 -- 172.21.15.187:0/2889844898 >> v2:172.21.15.5:3300/0 conn(0x7f5458065190 msgr2=0x7f5458065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.059+0000 7f5456d9d700 1 --2- 172.21.15.187:0/2889844898 >> v2:172.21.15.5:3300/0 conn(0x7f5458065190 0x7f5458065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.059+0000 7f5456d9d700 1 -- 172.21.15.187:0/2889844898 >> v1:172.21.15.5:6789/0 conn(0x7f5458065f90 legacy=0x7f5458143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.060+0000 7f5456d9d700 1 -- 172.21.15.187:0/2889844898 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54580b1640 con 0x7f545809ca90 2020-11-20T12:12:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.060+0000 7f5456d9d700 1 --2- 172.21.15.187:0/2889844898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f545809ca90 0x7f545809cf20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7f5458099f90 tx=0x7f5450009290).ready entity=mon.1 client_cookie=c05ad0c6931942ad server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.060+0000 7f5455d9b700 1 -- 172.21.15.187:0/2889844898 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545000bc10 con 0x7f545809ca90 2020-11-20T12:12:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.060+0000 7f5455d9b700 1 -- 172.21.15.187:0/2889844898 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545000bd70 con 0x7f545809ca90 2020-11-20T12:12:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.060+0000 7f5455d9b700 1 -- 172.21.15.187:0/2889844898 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5450011610 con 0x7f545809ca90 2020-11-20T12:12:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.061+0000 7f545759e700 1 -- 172.21.15.187:0/2889844898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f545809ca90 msgr2=0x7f545809cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.061+0000 7f545759e700 1 --2- 172.21.15.187:0/2889844898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f545809ca90 0x7f545809cf20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7f5458099f90 tx=0x7f5450009290).stop 2020-11-20T12:12:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.061+0000 7f545759e700 1 -- 172.21.15.187:0/2889844898 shutdown_connections 2020-11-20T12:12:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.061+0000 7f545759e700 1 --2- 172.21.15.187:0/2889844898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f545809ca90 0x7f545809cf20 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.061+0000 7f545759e700 1 --2- 172.21.15.187:0/2889844898 >> v2:172.21.15.5:3300/0 conn(0x7f5458065190 0x7f5458065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.061+0000 7f545759e700 1 -- 172.21.15.187:0/2889844898 >> 172.21.15.187:0/2889844898 conn(0x7f54580bab30 msgr2=0x7f54580baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.061+0000 7f545759e700 1 -- 172.21.15.187:0/2889844898 shutdown_connections 2020-11-20T12:12:09.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.061+0000 7f545759e700 1 -- 172.21.15.187:0/2889844898 wait complete. 2020-11-20T12:12:09.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.062+0000 7f545759e700 1 Processor -- start 2020-11-20T12:12:09.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.063+0000 7f545759e700 1 -- start start 2020-11-20T12:12:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.063+0000 7f545759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5458065190 0x7f545809ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5458065f90 0x7f5458067b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f545809ca90 0x7f545815b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545815c2a0 con 0x7f5458065f90 2020-11-20T12:12:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5458149e40 con 0x7f5458065190 2020-11-20T12:12:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545814a0e0 con 0x7f545809ca90 2020-11-20T12:12:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5458065190 0x7f545809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5458065190 0x7f545809ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46804/0 (socket says 172.21.15.187:46804) 2020-11-20T12:12:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545659c700 1 -- 172.21.15.187:0/337751465 learned_addr learned my addr 172.21.15.187:0/337751465 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:09.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f544dd9b700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5458065f90 0x7f5458067b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545659c700 1 -- 172.21.15.187:0/337751465 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f545809ca90 msgr2=0x7f545815b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545659c700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f545809ca90 0x7f545815b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:09.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545659c700 1 -- 172.21.15.187:0/337751465 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5458065f90 msgr2=0x7f5458067b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545659c700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5458065f90 0x7f5458067b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.064+0000 7f545659c700 1 -- 172.21.15.187:0/337751465 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54580a0490 con 0x7f5458065190 2020-11-20T12:12:09.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.065+0000 7f545659c700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5458065190 0x7f545809ffa0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f544800db60 tx=0x7f5448008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:09.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.065+0000 7f544effd700 1 -- 172.21.15.187:0/337751465 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448009790 con 0x7f5458065190 2020-11-20T12:12:09.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.065+0000 7f544effd700 1 -- 172.21.15.187:0/337751465 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54480098f0 con 0x7f5458065190 2020-11-20T12:12:09.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.065+0000 7f544effd700 1 -- 172.21.15.187:0/337751465 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448009c90 con 0x7f5458065190 2020-11-20T12:12:09.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.065+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54581498b0 con 0x7f5458065190 2020-11-20T12:12:09.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.065+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5458149b90 con 0x7f5458065190 2020-11-20T12:12:09.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.066+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545814a2a0 con 0x7f5458065190 2020-11-20T12:12:09.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.067+0000 7f544effd700 1 -- 172.21.15.187:0/337751465 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5448013480 con 0x7f5458065190 2020-11-20T12:12:09.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.068+0000 7f544effd700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5434056d20 0x7f54340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.068+0000 7f544effd700 1 -- 172.21.15.187:0/337751465 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f5448070470 con 0x7f5458065190 2020-11-20T12:12:09.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.068+0000 7f544dd9b700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5434056d20 0x7f54340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.069+0000 7f544dd9b700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5434056d20 0x7f54340591e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f545809ed80 tx=0x7f5440009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:09.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.072+0000 7f544effd700 1 -- 172.21.15.187:0/337751465 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5448045960 con 0x7f5458065190 2020-11-20T12:12:09.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.259+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f5458061a20 con 0x7f5458065190 2020-11-20T12:12:09.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.777+0000 7f544effd700 1 -- 172.21.15.187:0/337751465 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v79) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5448048f80 con 0x7f5458065190 2020-11-20T12:12:09.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5434056d20 msgr2=0x7f54340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5434056d20 0x7f54340591e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f545809ed80 tx=0x7f5440009040).stop 2020-11-20T12:12:09.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5458065190 msgr2=0x7f545809ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5458065190 0x7f545809ffa0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f544800db60 tx=0x7f5448008040).stop 2020-11-20T12:12:09.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 shutdown_connections 2020-11-20T12:12:09.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5434056d20 0x7f54340591e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f545809ca90 0x7f545815b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:09.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5458065190 0x7f545809ffa0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.782+0000 7f545759e700 1 --2- 172.21.15.187:0/337751465 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5458065f90 0x7f5458067b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.783+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 >> 172.21.15.187:0/337751465 conn(0x7f54580bab30 msgr2=0x7f54580995f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:09.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.783+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 shutdown_connections 2020-11-20T12:12:09.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:09.783+0000 7f545759e700 1 -- 172.21.15.187:0/337751465 wait complete. 2020-11-20T12:12:09.790 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:12:09.822 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:12:10.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.046+0000 7f2a93fff700 1 Processor -- start 2020-11-20T12:12:10.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.046+0000 7f2a93fff700 1 -- start start 2020-11-20T12:12:10.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.048+0000 7f2a93fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2a94068230 0x7f2a941482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.048+0000 7f2a93fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2a940b5f50 0x7f2a940b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.048+0000 7f2a93fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a94144b80 con 0x7f2a940b57b0 2020-11-20T12:12:10.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.048+0000 7f2a93fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a941487c0 con 0x7f2a94068230 2020-11-20T12:12:10.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.048+0000 7f2a93fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a941489e0 con 0x7f2a940b5f50 2020-11-20T12:12:10.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.048+0000 7f2a937fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2a940b5f50 0x7f2a940b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.049+0000 7f2a8a7fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2a94068230 0x7f2a941482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.049+0000 7f2a8a7fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2a94068230 0x7f2a941482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:39078/0 (socket says 172.21.15.187:39078) 2020-11-20T12:12:10.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.049+0000 7f2a8a7fc700 1 -- 172.21.15.187:0/2662976863 learned_addr learned my addr 172.21.15.187:0/2662976863 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:10.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.049+0000 7f2a92ffd700 1 --1- 172.21.15.187:0/2662976863 >> v1:172.21.15.5:6789/0 conn(0x7f2a940b57b0 0x7f2a94062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:36928/0 (socket says 172.21.15.187:36928) 2020-11-20T12:12:10.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.049+0000 7f2a937fe700 1 -- 172.21.15.187:0/2662976863 >> v2:172.21.15.5:3300/0 conn(0x7f2a94068230 msgr2=0x7f2a941482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:10.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.049+0000 7f2a937fe700 1 --2- 172.21.15.187:0/2662976863 >> v2:172.21.15.5:3300/0 conn(0x7f2a94068230 0x7f2a941482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.049+0000 7f2a937fe700 1 -- 172.21.15.187:0/2662976863 >> v1:172.21.15.5:6789/0 conn(0x7f2a940b57b0 legacy=0x7f2a94062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.050+0000 7f2a937fe700 1 -- 172.21.15.187:0/2662976863 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a94148c80 con 0x7f2a940b5f50 2020-11-20T12:12:10.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.050+0000 7f2a937fe700 1 --2- 172.21.15.187:0/2662976863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a940b5f50 0x7f2a940b6380 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7f2a94066230 tx=0x7f2a8c0091d0).ready entity=mon.1 client_cookie=edc4d82c75890864 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:10.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.050+0000 7f2a927fc700 1 -- 172.21.15.187:0/2662976863 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00ba70 con 0x7f2a940b5f50 2020-11-20T12:12:10.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.050+0000 7f2a927fc700 1 -- 172.21.15.187:0/2662976863 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c00bbd0 con 0x7f2a940b5f50 2020-11-20T12:12:10.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.050+0000 7f2a927fc700 1 -- 172.21.15.187:0/2662976863 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c011400 con 0x7f2a940b5f50 2020-11-20T12:12:10.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.051+0000 7f2a93fff700 1 -- 172.21.15.187:0/2662976863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a940b5f50 msgr2=0x7f2a940b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.051+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2662976863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a940b5f50 0x7f2a940b6380 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7f2a94066230 tx=0x7f2a8c0091d0).stop 2020-11-20T12:12:10.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.051+0000 7f2a93fff700 1 -- 172.21.15.187:0/2662976863 shutdown_connections 2020-11-20T12:12:10.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.051+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2662976863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a940b5f50 0x7f2a940b6380 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.051+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2662976863 >> v2:172.21.15.5:3300/0 conn(0x7f2a94068230 0x7f2a941482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.051+0000 7f2a93fff700 1 -- 172.21.15.187:0/2662976863 >> 172.21.15.187:0/2662976863 conn(0x7f2a940bab30 msgr2=0x7f2a940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:10.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.051+0000 7f2a93fff700 1 -- 172.21.15.187:0/2662976863 shutdown_connections 2020-11-20T12:12:10.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.051+0000 7f2a93fff700 1 -- 172.21.15.187:0/2662976863 wait complete. 2020-11-20T12:12:10.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.053+0000 7f2a93fff700 1 Processor -- start 2020-11-20T12:12:10.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.053+0000 7f2a93fff700 1 -- start start 2020-11-20T12:12:10.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a94068230 0x7f2a94158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a940b57b0 0x7f2a94142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a940b5f50 0x7f2a940a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a93fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94159160 con 0x7f2a940b5f50 2020-11-20T12:12:10.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a93fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9409ddf0 con 0x7f2a94068230 2020-11-20T12:12:10.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a93fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9409e090 con 0x7f2a940b57b0 2020-11-20T12:12:10.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a92ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a94068230 0x7f2a94158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a92ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a94068230 0x7f2a94158c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46818/0 (socket says 172.21.15.187:46818) 2020-11-20T12:12:10.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a92ffd700 1 -- 172.21.15.187:0/2166769430 learned_addr learned my addr 172.21.15.187:0/2166769430 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:10.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a8a7fc700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a940b57b0 0x7f2a94142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.054+0000 7f2a92ffd700 1 -- 172.21.15.187:0/2166769430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a940b57b0 msgr2=0x7f2a94142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a92ffd700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a940b57b0 0x7f2a94142320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a92ffd700 1 -- 172.21.15.187:0/2166769430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a940b5f50 msgr2=0x7f2a940a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a92ffd700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a940b5f50 0x7f2a940a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:10.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a92ffd700 1 -- 172.21.15.187:0/2166769430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9409e330 con 0x7f2a94068230 2020-11-20T12:12:10.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a92ffd700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a94068230 0x7f2a94158c70 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f2a8400c710 tx=0x7f2a84007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:10.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a8b7fe700 1 -- 172.21.15.187:0/2166769430 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84007320 con 0x7f2a94068230 2020-11-20T12:12:10.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a8b7fe700 1 -- 172.21.15.187:0/2166769430 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a84005530 con 0x7f2a94068230 2020-11-20T12:12:10.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9409dae0 con 0x7f2a94068230 2020-11-20T12:12:10.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a8b7fe700 1 -- 172.21.15.187:0/2166769430 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840058b0 con 0x7f2a94068230 2020-11-20T12:12:10.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.055+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a94145370 con 0x7f2a94068230 2020-11-20T12:12:10.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.057+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9409e4c0 con 0x7f2a94068230 2020-11-20T12:12:10.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.057+0000 7f2a8b7fe700 1 -- 172.21.15.187:0/2166769430 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2a84010030 con 0x7f2a94068230 2020-11-20T12:12:10.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.058+0000 7f2a8b7fe700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2a68056d20 0x7f2a680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.058+0000 7f2a8b7fe700 1 -- 172.21.15.187:0/2166769430 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f2a8406fbf0 con 0x7f2a94068230 2020-11-20T12:12:10.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.058+0000 7f2a8a7fc700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2a68056d20 0x7f2a680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.059+0000 7f2a8a7fc700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2a68056d20 0x7f2a680591e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f2a7c006a10 tx=0x7f2a7c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:10.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.062+0000 7f2a8b7fe700 1 -- 172.21.15.187:0/2166769430 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2a840450b0 con 0x7f2a94068230 2020-11-20T12:12:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.311+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f2a94061a20 con 0x7f2a94068230 2020-11-20T12:12:10.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.326+0000 7f2a8b7fe700 1 -- 172.21.15.187:0/2166769430 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 15 and data pool 16 v61) v1 ==== 189+0+0 (secure 0 0 0) 0x7f2a840486d0 con 0x7f2a94068230 2020-11-20T12:12:10.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2a68056d20 msgr2=0x7f2a680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2a68056d20 0x7f2a680591e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f2a7c006a10 tx=0x7f2a7c008040).stop 2020-11-20T12:12:10.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a94068230 msgr2=0x7f2a94158c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a94068230 0x7f2a94158c70 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f2a8400c710 tx=0x7f2a84007c00).stop 2020-11-20T12:12:10.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 shutdown_connections 2020-11-20T12:12:10.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2a68056d20 0x7f2a680591e0 secure :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0x7f2a7c006a10 tx=0x7f2a7c008040).stop 2020-11-20T12:12:10.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a940b5f50 0x7f2a940a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:10.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a94068230 0x7f2a94158c70 secure :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0x7f2a8400c710 tx=0x7f2a84007c00).stop 2020-11-20T12:12:10.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 --2- 172.21.15.187:0/2166769430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a940b57b0 0x7f2a94142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.330+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 >> 172.21.15.187:0/2166769430 conn(0x7f2a940bab30 msgr2=0x7f2a94143080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:10.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.331+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 shutdown_connections 2020-11-20T12:12:10.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.332+0000 7f2a93fff700 1 -- 172.21.15.187:0/2166769430 wait complete. 2020-11-20T12:12:10.339 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 15 and data pool 16 2020-11-20T12:12:10.360 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:12:10.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.591+0000 7f82fad02700 1 Processor -- start 2020-11-20T12:12:10.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.591+0000 7f82fad02700 1 -- start start 2020-11-20T12:12:10.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.592+0000 7f82fad02700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f40b1380 0x7f82f4150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.592+0000 7f82fad02700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f82f414c260 0x7f82f414c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.592+0000 7f82fad02700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40658a0 con 0x7f82f40b1760 2020-11-20T12:12:10.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.592+0000 7f82fad02700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f4150390 con 0x7f82f414c260 2020-11-20T12:12:10.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.592+0000 7f82fad02700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4150630 con 0x7f82f40b1380 2020-11-20T12:12:10.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82f94ff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f82f40b1760 0x7f82f4065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43438/0 (socket says 172.21.15.187:43438) 2020-11-20T12:12:10.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82fa501700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f82f414c260 0x7f82f414c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82f94ff700 1 -- 172.21.15.187:0/1506329119 learned_addr learned my addr 172.21.15.187:0/1506329119 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:10.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82f8cfe700 1 -- 172.21.15.187:0/1506329119 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636499943 0 0) 0x7f82f40658a0 con 0x7f82f40b1760 2020-11-20T12:12:10.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82f8cfe700 1 -- 172.21.15.187:0/1506329119 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82dc003240 con 0x7f82f40b1760 2020-11-20T12:12:10.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82fa501700 1 -- 172.21.15.187:0/1506329119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f40b1380 msgr2=0x7f82f4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:10.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82f9d00700 1 --2- 172.21.15.187:0/1506329119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f40b1380 0x7f82f4150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82fa501700 1 --2- 172.21.15.187:0/1506329119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f40b1380 0x7f82f4150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82fa501700 1 -- 172.21.15.187:0/1506329119 >> v1:172.21.15.187:6789/0 conn(0x7f82f40b1760 legacy=0x7f82f4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.593+0000 7f82fa501700 1 -- 172.21.15.187:0/1506329119 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f4068510 con 0x7f82f414c260 2020-11-20T12:12:10.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.594+0000 7f82fa501700 1 --2- 172.21.15.187:0/1506329119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f414c260 0x7f82f414c6f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7f82ec009c90 tx=0x7f82ec0091d0).ready entity=mon.1 client_cookie=b05549a79d5ac2fb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:10.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.594+0000 7f82f8cfe700 1 -- 172.21.15.187:0/1506329119 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00ba70 con 0x7f82f414c260 2020-11-20T12:12:10.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.594+0000 7f82f8cfe700 1 -- 172.21.15.187:0/1506329119 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec00bbd0 con 0x7f82f414c260 2020-11-20T12:12:10.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.594+0000 7f82f8cfe700 1 -- 172.21.15.187:0/1506329119 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec011400 con 0x7f82f414c260 2020-11-20T12:12:10.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.594+0000 7f82fad02700 1 -- 172.21.15.187:0/1506329119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f414c260 msgr2=0x7f82f414c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.594+0000 7f82fad02700 1 --2- 172.21.15.187:0/1506329119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f414c260 0x7f82f414c6f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7f82ec009c90 tx=0x7f82ec0091d0).stop 2020-11-20T12:12:10.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.595+0000 7f82fad02700 1 -- 172.21.15.187:0/1506329119 shutdown_connections 2020-11-20T12:12:10.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.595+0000 7f82fad02700 1 --2- 172.21.15.187:0/1506329119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f414c260 0x7f82f414c6f0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.595+0000 7f82fad02700 1 --2- 172.21.15.187:0/1506329119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f40b1380 0x7f82f4150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.595+0000 7f82fad02700 1 -- 172.21.15.187:0/1506329119 >> 172.21.15.187:0/1506329119 conn(0x7f82f40bab30 msgr2=0x7f82f40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:10.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.595+0000 7f82fad02700 1 -- 172.21.15.187:0/1506329119 shutdown_connections 2020-11-20T12:12:10.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.595+0000 7f82fad02700 1 -- 172.21.15.187:0/1506329119 wait complete. 2020-11-20T12:12:10.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.596+0000 7f82fad02700 1 Processor -- start 2020-11-20T12:12:10.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.596+0000 7f82fad02700 1 -- start start 2020-11-20T12:12:10.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.597+0000 7f82fad02700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f40b1380 0x7f82f409cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.597+0000 7f82fad02700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f40b1760 0x7f82f4152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.597+0000 7f82fad02700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f414c260 0x7f82f40b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.597+0000 7f82fad02700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4148340 con 0x7f82f40b1380 2020-11-20T12:12:10.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.597+0000 7f82fad02700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b5040 con 0x7f82f40b1760 2020-11-20T12:12:10.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.597+0000 7f82fad02700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b52e0 con 0x7f82f414c260 2020-11-20T12:12:10.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f94ff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f40b1760 0x7f82f4152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f94ff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f40b1760 0x7f82f4152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46840/0 (socket says 172.21.15.187:46840) 2020-11-20T12:12:10.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f94ff700 1 -- 172.21.15.187:0/3683537988 learned_addr learned my addr 172.21.15.187:0/3683537988 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:10.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f9d00700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f40b1380 0x7f82f409cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82fa501700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f414c260 0x7f82f40b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f94ff700 1 -- 172.21.15.187:0/3683537988 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f414c260 msgr2=0x7f82f40b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f94ff700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f414c260 0x7f82f40b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f94ff700 1 -- 172.21.15.187:0/3683537988 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f40b1380 msgr2=0x7f82f409cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f94ff700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f40b1380 0x7f82f409cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.598+0000 7f82f94ff700 1 -- 172.21.15.187:0/3683537988 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40b5580 con 0x7f82f40b1760 2020-11-20T12:12:10.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.599+0000 7f82f94ff700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f40b1760 0x7f82f4152ad0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f82e400c6e0 tx=0x7f82e4007a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:10.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.599+0000 7f82e1ffb700 1 -- 172.21.15.187:0/3683537988 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e40072e0 con 0x7f82f40b1760 2020-11-20T12:12:10.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.599+0000 7f82e1ffb700 1 -- 172.21.15.187:0/3683537988 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e4007440 con 0x7f82f40b1760 2020-11-20T12:12:10.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.599+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f40b5820 con 0x7f82f40b1760 2020-11-20T12:12:10.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.599+0000 7f82e1ffb700 1 -- 172.21.15.187:0/3683537988 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4005820 con 0x7f82f40b1760 2020-11-20T12:12:10.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.599+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f40b6db0 con 0x7f82f40b1760 2020-11-20T12:12:10.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.600+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f40b59b0 con 0x7f82f40b1760 2020-11-20T12:12:10.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.601+0000 7f82e1ffb700 1 -- 172.21.15.187:0/3683537988 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f82e401a060 con 0x7f82f40b1760 2020-11-20T12:12:10.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.601+0000 7f82e1ffb700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f82cc056d50 0x7f82cc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:10.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.601+0000 7f82e1ffb700 1 -- 172.21.15.187:0/3683537988 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f82e406f7e0 con 0x7f82f40b1760 2020-11-20T12:12:10.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.601+0000 7f82f9d00700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f82cc056d50 0x7f82cc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:10.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.602+0000 7f82f9d00700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f82cc056d50 0x7f82cc059210 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f82f0006fd0 tx=0x7f82f0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:10.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.605+0000 7f82e1ffb700 1 -- 172.21.15.187:0/3683537988 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f82e4044bb0 con 0x7f82f40b1760 2020-11-20T12:12:10.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.789+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f4061a20 con 0x7f82f40b1760 2020-11-20T12:12:10.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.792+0000 7f82e1ffb700 1 -- 172.21.15.187:0/3683537988 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16794 (secure 0 0 0) 0x7f82e4018070 con 0x7f82f40b1760 2020-11-20T12:12:10.795 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:10.796 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:12:10.313689+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:12:07.846671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2020-11-20T12:12:09.337957+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:12:10.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f82cc056d50 msgr2=0x7f82cc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f82cc056d50 0x7f82cc059210 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f82f0006fd0 tx=0x7f82f0008040).stop 2020-11-20T12:12:10.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f40b1760 msgr2=0x7f82f4152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f40b1760 0x7f82f4152ad0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f82e400c6e0 tx=0x7f82e4007a20).stop 2020-11-20T12:12:10.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 shutdown_connections 2020-11-20T12:12:10.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f82cc056d50 0x7f82cc059210 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f40b1380 0x7f82f409cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82f414c260 0x7f82f40b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 --2- 172.21.15.187:0/3683537988 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82f40b1760 0x7f82f4152ad0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 >> 172.21.15.187:0/3683537988 conn(0x7f82f40bab30 msgr2=0x7f82f40b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:10.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 shutdown_connections 2020-11-20T12:12:10.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:10.794+0000 7f82fad02700 1 -- 172.21.15.187:0/3683537988 wait complete. 2020-11-20T12:12:10.823 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:12:11.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.068+0000 7f4177fff700 1 Processor -- start 2020-11-20T12:12:11.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.068+0000 7f4177fff700 1 -- start start 2020-11-20T12:12:11.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.069+0000 7f4177fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f41781484d0 0x7f4178061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.070+0000 7f4177fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4178066f70 0x7f41780b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.070+0000 7f4177fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780b1300 con 0x7f41780622c0 2020-11-20T12:12:11.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.070+0000 7f4177fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4178067730 con 0x7f41781484d0 2020-11-20T12:12:11.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.070+0000 7f4177fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41780b15a0 con 0x7f4178066f70 2020-11-20T12:12:11.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.070+0000 7f416ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f41780622c0 0x7f41780b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43452/0 (socket says 172.21.15.187:43452) 2020-11-20T12:12:11.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.070+0000 7f416ffff700 1 -- 172.21.15.187:0/477450272 learned_addr learned my addr 172.21.15.187:0/477450272 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:11.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.070+0000 7f4176ffd700 1 --2- 172.21.15.187:0/477450272 >> v2:172.21.15.5:3300/0 conn(0x7f41781484d0 0x7f4178061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.070+0000 7f41777fe700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4178066f70 0x7f41780b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906312781 0 0) 0x7f41780b1300 con 0x7f41780622c0 2020-11-20T12:12:11.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4158003240 con 0x7f41780622c0 2020-11-20T12:12:11.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3627720801 0 0) 0x7f4158003240 con 0x7f41780622c0 2020-11-20T12:12:11.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4178066f70 msgr2=0x7f41780b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:11.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4178066f70 0x7f41780b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 >> v2:172.21.15.5:3300/0 conn(0x7f41781484d0 msgr2=0x7f4178061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:11.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 --2- 172.21.15.187:0/477450272 >> v2:172.21.15.5:3300/0 conn(0x7f41781484d0 0x7f4178061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41780b1840 con 0x7f41780622c0 2020-11-20T12:12:11.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f41600014e0 con 0x7f41780622c0 2020-11-20T12:12:11.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 >> v1:172.21.15.187:6789/0 conn(0x7f41780622c0 legacy=0x7f41780b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4158003730 0x7f4158003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4178066f70 0x7f41580044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41580049d0 0x7f4158004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4178067730 con 0x7f4158003730 2020-11-20T12:12:11.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41780b15a0 con 0x7f41580049d0 2020-11-20T12:12:11.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.071+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41580041d0 con 0x7f4178066f70 2020-11-20T12:12:11.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.072+0000 7f41777fe700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41580049d0 0x7f4158004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.072+0000 7f4176ffd700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4158003730 0x7f4158003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.072+0000 7f416ffff700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4178066f70 0x7f41580044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.072+0000 7f41777fe700 1 -- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4178066f70 msgr2=0x7f41580044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:11.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.072+0000 7f41777fe700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4178066f70 0x7f41580044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.072+0000 7f41777fe700 1 -- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4158003730 msgr2=0x7f4158003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:11.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.072+0000 7f41777fe700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4158003730 0x7f4158003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.073+0000 7f41777fe700 1 -- 172.21.15.187:0/477450272 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158003240 con 0x7f41580049d0 2020-11-20T12:12:11.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.073+0000 7f41777fe700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41580049d0 0x7f4158004e50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7f4170009e70 tx=0x7f41700091d0).ready entity=mon.1 client_cookie=3041baf195aa531b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:11.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.073+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41700095c0 con 0x7f41580049d0 2020-11-20T12:12:11.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.073+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417000ed70 con 0x7f41580049d0 2020-11-20T12:12:11.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.073+0000 7f41767fc700 1 -- 172.21.15.187:0/477450272 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417000eee0 con 0x7f41580049d0 2020-11-20T12:12:11.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.074+0000 7f4177fff700 1 -- 172.21.15.187:0/477450272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41580049d0 msgr2=0x7f4158004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.074+0000 7f4177fff700 1 --2- 172.21.15.187:0/477450272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41580049d0 0x7f4158004e50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7f4170009e70 tx=0x7f41700091d0).stop 2020-11-20T12:12:11.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.074+0000 7f41777fe700 1 -- 172.21.15.187:0/477450272 reap_dead start 2020-11-20T12:12:11.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.074+0000 7f4177fff700 1 -- 172.21.15.187:0/477450272 shutdown_connections 2020-11-20T12:12:11.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.074+0000 7f4177fff700 1 -- 172.21.15.187:0/477450272 >> 172.21.15.187:0/477450272 conn(0x7f41780bab30 msgr2=0x7f41780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:11.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.074+0000 7f4177fff700 1 -- 172.21.15.187:0/477450272 shutdown_connections 2020-11-20T12:12:11.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.074+0000 7f4177fff700 1 -- 172.21.15.187:0/477450272 wait complete. 2020-11-20T12:12:11.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.076+0000 7f4177fff700 1 Processor -- start 2020-11-20T12:12:11.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.076+0000 7f4177fff700 1 -- start start 2020-11-20T12:12:11.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.077+0000 7f4177fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f417809aa40 0x7f417809c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.077+0000 7f4177fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f417809ae20 0x7f417813faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.077+0000 7f4177fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41780b4080 0x7f417813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.077+0000 7f4177fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41780b44c0 con 0x7f417809aa40 2020-11-20T12:12:11.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.077+0000 7f4177fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41780b23f0 con 0x7f41780b4080 2020-11-20T12:12:11.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.077+0000 7f4177fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41780b46b0 con 0x7f417809ae20 2020-11-20T12:12:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f41777fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41780b4080 0x7f417813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f41777fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41780b4080 0x7f417813efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46862/0 (socket says 172.21.15.187:46862) 2020-11-20T12:12:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f41777fe700 1 -- 172.21.15.187:0/750748047 learned_addr learned my addr 172.21.15.187:0/750748047 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f416ffff700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f417809ae20 0x7f417813faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f4176ffd700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f417809aa40 0x7f417809c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f41777fe700 1 -- 172.21.15.187:0/750748047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f417809ae20 msgr2=0x7f417813faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f41777fe700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f417809ae20 0x7f417813faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f41777fe700 1 -- 172.21.15.187:0/750748047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f417809aa40 msgr2=0x7f417809c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f41777fe700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f417809aa40 0x7f417809c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.078+0000 7f41777fe700 1 -- 172.21.15.187:0/750748047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41780b4950 con 0x7f41780b4080 2020-11-20T12:12:11.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.079+0000 7f41777fe700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41780b4080 0x7f417813efc0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f4170014000 tx=0x7f417001a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:11.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.079+0000 7f416effd700 1 -- 172.21.15.187:0/750748047 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41700095c0 con 0x7f41780b4080 2020-11-20T12:12:11.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.079+0000 7f416effd700 1 -- 172.21.15.187:0/750748047 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417001cc50 con 0x7f41780b4080 2020-11-20T12:12:11.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.079+0000 7f416effd700 1 -- 172.21.15.187:0/750748047 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4170025440 con 0x7f41780b4080 2020-11-20T12:12:11.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.079+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417813ffe0 con 0x7f41780b4080 2020-11-20T12:12:11.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.079+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417813f670 con 0x7f41780b4080 2020-11-20T12:12:11.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.081+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417813f8f0 con 0x7f41780b4080 2020-11-20T12:12:11.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.081+0000 7f416effd700 1 -- 172.21.15.187:0/750748047 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f417002d070 con 0x7f41780b4080 2020-11-20T12:12:11.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.082+0000 7f416effd700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4154056d20 0x7f41540591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.082+0000 7f416effd700 1 -- 172.21.15.187:0/750748047 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4170071c50 con 0x7f41780b4080 2020-11-20T12:12:11.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.082+0000 7f4176ffd700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4154056d20 0x7f41540591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.083+0000 7f4176ffd700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4154056d20 0x7f41540591e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f4168006a90 tx=0x7f4168008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:11.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.086+0000 7f416effd700 1 -- 172.21.15.187:0/750748047 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4170046f90 con 0x7f41780b4080 2020-11-20T12:12:11.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.270+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4178061a20 con 0x7f41780b4080 2020-11-20T12:12:11.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.272+0000 7f416effd700 1 -- 172.21.15.187:0/750748047 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16794 (secure 0 0 0) 0x7f4170016070 con 0x7f41780b4080 2020-11-20T12:12:11.277 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:11.278 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:12:10.313689+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:12:07.846671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2020-11-20T12:12:09.337957+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:12:11.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.274+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4154056d20 msgr2=0x7f41540591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.274+0000 7f4177fff700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4154056d20 0x7f41540591e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f4168006a90 tx=0x7f4168008040).stop 2020-11-20T12:12:11.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.274+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41780b4080 msgr2=0x7f417813efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.274+0000 7f4177fff700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41780b4080 0x7f417813efc0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f4170014000 tx=0x7f417001a000).stop 2020-11-20T12:12:11.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.275+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 shutdown_connections 2020-11-20T12:12:11.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.275+0000 7f4177fff700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4154056d20 0x7f41540591e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.275+0000 7f4177fff700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f417809aa40 0x7f417809c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.275+0000 7f4177fff700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41780b4080 0x7f417813efc0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.275+0000 7f4177fff700 1 --2- 172.21.15.187:0/750748047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f417809ae20 0x7f417813faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.275+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 >> 172.21.15.187:0/750748047 conn(0x7f41780bab30 msgr2=0x7f4178066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.275+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 shutdown_connections 2020-11-20T12:12:11.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.275+0000 7f4177fff700 1 -- 172.21.15.187:0/750748047 wait complete. 2020-11-20T12:12:11.303 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:12:11.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.527+0000 7f4b0324f700 1 Processor -- start 2020-11-20T12:12:11.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.527+0000 7f4b0324f700 1 -- start start 2020-11-20T12:12:11.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.528+0000 7f4b0324f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc1484d0 0x7f4afc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.528+0000 7f4b0324f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4afc066f70 0x7f4afc0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.528+0000 7f4b0324f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4afc0b1300 con 0x7f4afc0622c0 2020-11-20T12:12:11.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.528+0000 7f4b0324f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4afc067730 con 0x7f4afc066f70 2020-11-20T12:12:11.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.528+0000 7f4b0324f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc0b15a0 con 0x7f4afc1484d0 2020-11-20T12:12:11.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.529+0000 7f4b01a4c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4afc0622c0 0x7f4afc0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43476/0 (socket says 172.21.15.187:43476) 2020-11-20T12:12:11.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.529+0000 7f4b01a4c700 1 -- 172.21.15.187:0/1130447588 learned_addr learned my addr 172.21.15.187:0/1130447588 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:11.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.529+0000 7f4b0224d700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc1484d0 0x7f4afc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.529+0000 7f4b02a4e700 1 --2- 172.21.15.187:0/1130447588 >> v2:172.21.15.5:3300/0 conn(0x7f4afc066f70 0x7f4afc0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.529+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981926303 0 0) 0x7f4afc0b1300 con 0x7f4afc0622c0 2020-11-20T12:12:11.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.529+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae8003240 con 0x7f4afc0622c0 2020-11-20T12:12:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3134071613 0 0) 0x7f4ae8003240 con 0x7f4afc0622c0 2020-11-20T12:12:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc1484d0 msgr2=0x7f4afc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc1484d0 0x7f4afc061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 >> v2:172.21.15.5:3300/0 conn(0x7f4afc066f70 msgr2=0x7f4afc0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 --2- 172.21.15.187:0/1130447588 >> v2:172.21.15.5:3300/0 conn(0x7f4afc066f70 0x7f4afc0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc0b1840 con 0x7f4afc0622c0 2020-11-20T12:12:11.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4af8001860 con 0x7f4afc0622c0 2020-11-20T12:12:11.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 >> v1:172.21.15.187:6789/0 conn(0x7f4afc0622c0 legacy=0x7f4afc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae8003730 0x7f4ae8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae80044e0 0x7f4ae8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc1484d0 0x7f4ae8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc067730 con 0x7f4ae8003730 2020-11-20T12:12:11.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc0b15a0 con 0x7f4ae80044e0 2020-11-20T12:12:11.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.530+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae8004270 con 0x7f4afc1484d0 2020-11-20T12:12:11.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b01a4c700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae80044e0 0x7f4ae8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b0224d700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae8003730 0x7f4ae8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b02a4e700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc1484d0 0x7f4ae8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b01a4c700 1 -- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc1484d0 msgr2=0x7f4ae8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:11.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b01a4c700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc1484d0 0x7f4ae8004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b01a4c700 1 -- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae8003730 msgr2=0x7f4ae8003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:11.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b01a4c700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae8003730 0x7f4ae8003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b01a4c700 1 -- 172.21.15.187:0/1130447588 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8003240 con 0x7f4ae80044e0 2020-11-20T12:12:11.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.531+0000 7f4b01a4c700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae80044e0 0x7f4ae8004960 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f4af800c810 tx=0x7f4af8007b80).ready entity=mon.1 client_cookie=c9063d03cd09cc33 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:11.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.532+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8005b80 con 0x7f4ae80044e0 2020-11-20T12:12:11.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.532+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af8005ce0 con 0x7f4ae80044e0 2020-11-20T12:12:11.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.532+0000 7f4b0124b700 1 -- 172.21.15.187:0/1130447588 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af80072e0 con 0x7f4ae80044e0 2020-11-20T12:12:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.532+0000 7f4b0324f700 1 -- 172.21.15.187:0/1130447588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae80044e0 msgr2=0x7f4ae8004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.532+0000 7f4b0324f700 1 --2- 172.21.15.187:0/1130447588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ae80044e0 0x7f4ae8004960 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f4af800c810 tx=0x7f4af8007b80).stop 2020-11-20T12:12:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.532+0000 7f4b02a4e700 1 -- 172.21.15.187:0/1130447588 reap_dead start 2020-11-20T12:12:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.532+0000 7f4b0324f700 1 -- 172.21.15.187:0/1130447588 shutdown_connections 2020-11-20T12:12:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.532+0000 7f4b0324f700 1 -- 172.21.15.187:0/1130447588 >> 172.21.15.187:0/1130447588 conn(0x7f4afc0bab30 msgr2=0x7f4afc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:11.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.533+0000 7f4b0324f700 1 -- 172.21.15.187:0/1130447588 shutdown_connections 2020-11-20T12:12:11.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.533+0000 7f4b0324f700 1 -- 172.21.15.187:0/1130447588 wait complete. 2020-11-20T12:12:11.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.534+0000 7f4b0324f700 1 Processor -- start 2020-11-20T12:12:11.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.534+0000 7f4b0324f700 1 -- start start 2020-11-20T12:12:11.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b0324f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae80044e0 0x7f4afc141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b0324f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc157870 0x7f4afc1462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b0324f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc157cf0 0x7f4afc144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b0324f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc145110 con 0x7f4ae80044e0 2020-11-20T12:12:11.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b0324f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc141c50 con 0x7f4afc157cf0 2020-11-20T12:12:11.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b0324f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc157130 con 0x7f4afc157870 2020-11-20T12:12:11.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b02a4e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc157cf0 0x7f4afc144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b02a4e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc157cf0 0x7f4afc144670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46882/0 (socket says 172.21.15.187:46882) 2020-11-20T12:12:11.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.535+0000 7f4b02a4e700 1 -- 172.21.15.187:0/3101240271 learned_addr learned my addr 172.21.15.187:0/3101240271 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:11.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b0224d700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae80044e0 0x7f4afc141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b01a4c700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc157870 0x7f4afc1462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b02a4e700 1 -- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc157870 msgr2=0x7f4afc1462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b02a4e700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc157870 0x7f4afc1462e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b02a4e700 1 -- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae80044e0 msgr2=0x7f4afc141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:11.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b02a4e700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae80044e0 0x7f4afc141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:11.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b02a4e700 1 -- 172.21.15.187:0/3101240271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc1573d0 con 0x7f4afc157cf0 2020-11-20T12:12:11.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b0224d700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae80044e0 0x7f4afc141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:11.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b02a4e700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc157cf0 0x7f4afc144670 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f4afc068600 tx=0x7f4af4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:11.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4af27fc700 1 -- 172.21.15.187:0/3101240271 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4017070 con 0x7f4afc157cf0 2020-11-20T12:12:11.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4af27fc700 1 -- 172.21.15.187:0/3101240271 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af4012570 con 0x7f4afc157cf0 2020-11-20T12:12:11.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4af27fc700 1 -- 172.21.15.187:0/3101240271 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4012910 con 0x7f4afc157cf0 2020-11-20T12:12:11.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.536+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc09e700 con 0x7f4afc157cf0 2020-11-20T12:12:11.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.537+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4afc146990 con 0x7f4afc157cf0 2020-11-20T12:12:11.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.538+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4afc157560 con 0x7f4afc157cf0 2020-11-20T12:12:11.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.538+0000 7f4af27fc700 1 -- 172.21.15.187:0/3101240271 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4af4012c80 con 0x7f4afc157cf0 2020-11-20T12:12:11.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.539+0000 7f4af27fc700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4ad4056d50 0x7f4ad4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:11.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.539+0000 7f4af27fc700 1 -- 172.21.15.187:0/3101240271 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4af406fcc0 con 0x7f4afc157cf0 2020-11-20T12:12:11.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.539+0000 7f4b0224d700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4ad4056d50 0x7f4ad4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:11.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.540+0000 7f4b0224d700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4ad4056d50 0x7f4ad4059210 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f4aec006fd0 tx=0x7f4aec008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:11.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.542+0000 7f4af27fc700 1 -- 172.21.15.187:0/3101240271 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4af4045110 con 0x7f4afc157cf0 2020-11-20T12:12:11.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:11.787+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4afc061a20 con 0x7f4afc157cf0 2020-11-20T12:12:12.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.342+0000 7f4af27fc700 1 -- 172.21.15.187:0/3101240271 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v64) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4af4048730 con 0x7f4afc157cf0 2020-11-20T12:12:12.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4ad4056d50 msgr2=0x7f4ad4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4ad4056d50 0x7f4ad4059210 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f4aec006fd0 tx=0x7f4aec008040).stop 2020-11-20T12:12:12.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc157cf0 msgr2=0x7f4afc144670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc157cf0 0x7f4afc144670 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f4afc068600 tx=0x7f4af4008040).stop 2020-11-20T12:12:12.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 shutdown_connections 2020-11-20T12:12:12.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4ad4056d50 0x7f4ad4059210 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ae80044e0 0x7f4afc141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc157cf0 0x7f4afc144670 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 --2- 172.21.15.187:0/3101240271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4afc157870 0x7f4afc1462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.346+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 >> 172.21.15.187:0/3101240271 conn(0x7f4afc0bab30 msgr2=0x7f4afc0b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:12.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.347+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 shutdown_connections 2020-11-20T12:12:12.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.347+0000 7f4b0324f700 1 -- 172.21.15.187:0/3101240271 wait complete. 2020-11-20T12:12:12.380 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:12.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.611+0000 7f1d9ffff700 1 Processor -- start 2020-11-20T12:12:12.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.611+0000 7f1d9ffff700 1 -- start start 2020-11-20T12:12:12.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.613+0000 7f1d9ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1da0065190 0x7f1da0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.613+0000 7f1d9ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da009ca90 0x7f1da009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.613+0000 7f1d9ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da00b13e0 con 0x7f1da0065f90 2020-11-20T12:12:12.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.613+0000 7f1d9ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da014ab30 con 0x7f1da0065190 2020-11-20T12:12:12.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.613+0000 7f1d9ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da0062c10 con 0x7f1da009ca90 2020-11-20T12:12:12.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.613+0000 7f1d9e7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1da0065f90 0x7f1da0143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43492/0 (socket says 172.21.15.187:43492) 2020-11-20T12:12:12.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.613+0000 7f1d9e7fc700 1 -- 172.21.15.187:0/816451707 learned_addr learned my addr 172.21.15.187:0/816451707 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:12.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9f7fe700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da009ca90 0x7f1da009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9effd700 1 --2- 172.21.15.187:0/816451707 >> v2:172.21.15.5:3300/0 conn(0x7f1da0065190 0x7f1da0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718157335 0 0) 0x7f1da00b13e0 con 0x7f1da0065f90 2020-11-20T12:12:12.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c003240 con 0x7f1da0065f90 2020-11-20T12:12:12.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 741512217 0 0) 0x7f1d8c003240 con 0x7f1da0065f90 2020-11-20T12:12:12.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da009ca90 msgr2=0x7f1da009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:12.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da009ca90 0x7f1da009cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 >> v2:172.21.15.5:3300/0 conn(0x7f1da0065190 msgr2=0x7f1da0065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:12.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 --2- 172.21.15.187:0/816451707 >> v2:172.21.15.5:3300/0 conn(0x7f1da0065190 0x7f1da0065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da00b1640 con 0x7f1da0065f90 2020-11-20T12:12:12.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.614+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1d940014e0 con 0x7f1da0065f90 2020-11-20T12:12:12.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 >> v1:172.21.15.187:6789/0 conn(0x7f1da0065f90 legacy=0x7f1da0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9dffb700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da009ca90 0x7f1d8c0037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9dffb700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c003d10 0x7f1d8c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9dffb700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d8c0049d0 0x7f1d8c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da014ab30 con 0x7f1d8c0049d0 2020-11-20T12:12:12.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da0062c10 con 0x7f1d8c003d10 2020-11-20T12:12:12.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c0042d0 con 0x7f1da009ca90 2020-11-20T12:12:12.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9e7fc700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c003d10 0x7f1d8c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.615+0000 7f1d9effd700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da009ca90 0x7f1d8c0037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9f7fe700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d8c0049d0 0x7f1d8c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9e7fc700 1 -- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da009ca90 msgr2=0x7f1d8c0037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:12.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9e7fc700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da009ca90 0x7f1d8c0037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9e7fc700 1 -- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d8c0049d0 msgr2=0x7f1d8c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:12.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9e7fc700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d8c0049d0 0x7f1d8c004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9e7fc700 1 -- 172.21.15.187:0/816451707 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c003240 con 0x7f1d8c003d10 2020-11-20T12:12:12.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9e7fc700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c003d10 0x7f1d8c0044e0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f1d9400c1e0 tx=0x7f1da00bb650).ready entity=mon.1 client_cookie=4fdc488fced00224 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:12.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d940076f0 con 0x7f1d8c003d10 2020-11-20T12:12:12.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d940070d0 con 0x7f1d8c003d10 2020-11-20T12:12:12.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.616+0000 7f1d9dffb700 1 -- 172.21.15.187:0/816451707 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94007490 con 0x7f1d8c003d10 2020-11-20T12:12:12.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.617+0000 7f1d9ffff700 1 -- 172.21.15.187:0/816451707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c003d10 msgr2=0x7f1d8c0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.617+0000 7f1d9ffff700 1 --2- 172.21.15.187:0/816451707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c003d10 0x7f1d8c0044e0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f1d9400c1e0 tx=0x7f1da00bb650).stop 2020-11-20T12:12:12.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.617+0000 7f1d9f7fe700 1 -- 172.21.15.187:0/816451707 reap_dead start 2020-11-20T12:12:12.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.617+0000 7f1d9ffff700 1 -- 172.21.15.187:0/816451707 shutdown_connections 2020-11-20T12:12:12.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.617+0000 7f1d9ffff700 1 -- 172.21.15.187:0/816451707 >> 172.21.15.187:0/816451707 conn(0x7f1da00bab30 msgr2=0x7f1da00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:12.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.617+0000 7f1d9ffff700 1 -- 172.21.15.187:0/816451707 shutdown_connections 2020-11-20T12:12:12.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.617+0000 7f1d9ffff700 1 -- 172.21.15.187:0/816451707 wait complete. 2020-11-20T12:12:12.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.619+0000 7f1d9ffff700 1 Processor -- start 2020-11-20T12:12:12.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.619+0000 7f1d9ffff700 1 -- start start 2020-11-20T12:12:12.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da009bc00 0x7f1da0099200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da009bfe0 0x7f1da00996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da00b3390 0x7f1da0066590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da015b810 con 0x7f1da009bfe0 2020-11-20T12:12:12.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da0099be0 con 0x7f1da009bc00 2020-11-20T12:12:12.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da015ba40 con 0x7f1da00b3390 2020-11-20T12:12:12.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da009bc00 0x7f1da0099200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da009bc00 0x7f1da0099200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46898/0 (socket says 172.21.15.187:46898) 2020-11-20T12:12:12.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.620+0000 7f1d9effd700 1 -- 172.21.15.187:0/3844725954 learned_addr learned my addr 172.21.15.187:0/3844725954 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:12.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d9e7fc700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da009bfe0 0x7f1da00996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d9effd700 1 -- 172.21.15.187:0/3844725954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da00b3390 msgr2=0x7f1da0066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d9effd700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da00b3390 0x7f1da0066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:12.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d9effd700 1 -- 172.21.15.187:0/3844725954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da009bfe0 msgr2=0x7f1da00996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d9effd700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da009bfe0 0x7f1da00996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d9effd700 1 -- 172.21.15.187:0/3844725954 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da015bce0 con 0x7f1da009bc00 2020-11-20T12:12:12.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d9effd700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da009bc00 0x7f1da0099200 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f1d900067d0 tx=0x7f1d90008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:12.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d86ffd700 1 -- 172.21.15.187:0/3844725954 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9000ea30 con 0x7f1da009bc00 2020-11-20T12:12:12.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d86ffd700 1 -- 172.21.15.187:0/3844725954 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9000eb90 con 0x7f1da009bc00 2020-11-20T12:12:12.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da015bf80 con 0x7f1da009bc00 2020-11-20T12:12:12.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.621+0000 7f1d86ffd700 1 -- 172.21.15.187:0/3844725954 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90014470 con 0x7f1da009bc00 2020-11-20T12:12:12.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.622+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da015c5e0 con 0x7f1da009bc00 2020-11-20T12:12:12.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.623+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da015c110 con 0x7f1da009bc00 2020-11-20T12:12:12.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.623+0000 7f1d86ffd700 1 -- 172.21.15.187:0/3844725954 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1d90024070 con 0x7f1da009bc00 2020-11-20T12:12:12.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.624+0000 7f1d86ffd700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d7c056d20 0x7f1d7c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.624+0000 7f1d86ffd700 1 -- 172.21.15.187:0/3844725954 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1d900709f0 con 0x7f1da009bc00 2020-11-20T12:12:12.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.624+0000 7f1d9e7fc700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d7c056d20 0x7f1d7c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.625+0000 7f1d9e7fc700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d7c056d20 0x7f1d7c0591e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1da005ec00 tx=0x7f1d94015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:12.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.628+0000 7f1d86ffd700 1 -- 172.21.15.187:0/3844725954 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1d90045d30 con 0x7f1da009bc00 2020-11-20T12:12:12.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.872+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1da0061a20 con 0x7f1da009bc00 2020-11-20T12:12:12.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.873+0000 7f1d86ffd700 1 -- 172.21.15.187:0/3844725954 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f1d90018070 con 0x7f1da009bc00 2020-11-20T12:12:12.877 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:12.878 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:12.329807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6952},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:12.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d7c056d20 msgr2=0x7f1d7c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d7c056d20 0x7f1d7c0591e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1da005ec00 tx=0x7f1d94015000).stop 2020-11-20T12:12:12.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da009bc00 msgr2=0x7f1da0099200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da009bc00 0x7f1da0099200 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f1d900067d0 tx=0x7f1d90008040).stop 2020-11-20T12:12:12.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 shutdown_connections 2020-11-20T12:12:12.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1d7c056d20 0x7f1d7c0591e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1da00b3390 0x7f1da0066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:12.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da009bc00 0x7f1da0099200 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.875+0000 7f1d9ffff700 1 --2- 172.21.15.187:0/3844725954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da009bfe0 0x7f1da00996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.876+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 >> 172.21.15.187:0/3844725954 conn(0x7f1da00bab30 msgr2=0x7f1da00b8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:12.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.876+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 shutdown_connections 2020-11-20T12:12:12.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:12.876+0000 7f1d9ffff700 1 -- 172.21.15.187:0/3844725954 wait complete. 2020-11-20T12:12:12.883 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 64 2020-11-20T12:12:12.901 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:12:13.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.128+0000 7ff7bacc6700 1 Processor -- start 2020-11-20T12:12:13.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.128+0000 7ff7bacc6700 1 -- start start 2020-11-20T12:12:13.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.129+0000 7ff7bacc6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff7b4065cc0 0x7ff7b40b7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.129+0000 7ff7bacc6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b40660a0 0x7ff7b4060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.129+0000 7ff7bacc6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b409b3f0 con 0x7ff7b409bb10 2020-11-20T12:12:13.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.129+0000 7ff7bacc6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b41484a0 con 0x7ff7b4065cc0 2020-11-20T12:12:13.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.129+0000 7ff7bacc6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b409dcd0 con 0x7ff7b40660a0 2020-11-20T12:12:13.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.130+0000 7ff7b9263700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff7b409bb10 0x7ff7b4144440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43514/0 (socket says 172.21.15.187:43514) 2020-11-20T12:12:13.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.130+0000 7ff7b9263700 1 -- 172.21.15.187:0/85637549 learned_addr learned my addr 172.21.15.187:0/85637549 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:13.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.130+0000 7ff7b8a62700 1 --2- 172.21.15.187:0/85637549 >> v2:172.21.15.5:3300/0 conn(0x7ff7b4065cc0 0x7ff7b40b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.130+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011937428 0 0) 0x7ff7b409b3f0 con 0x7ff7b409bb10 2020-11-20T12:12:13.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.130+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a0003240 con 0x7ff7b409bb10 2020-11-20T12:12:13.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2424448675 0 0) 0x7ff7a0003240 con 0x7ff7b409bb10 2020-11-20T12:12:13.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b40660a0 msgr2=0x7ff7b4060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:13.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b40660a0 0x7ff7b4060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:13.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 >> v2:172.21.15.5:3300/0 conn(0x7ff7b4065cc0 msgr2=0x7ff7b40b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:13.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 --2- 172.21.15.187:0/85637549 >> v2:172.21.15.5:3300/0 conn(0x7ff7b4065cc0 0x7ff7b40b7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b409df70 con 0x7ff7b409bb10 2020-11-20T12:12:13.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff7ac001860 con 0x7ff7b409bb10 2020-11-20T12:12:13.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 >> v1:172.21.15.187:6789/0 conn(0x7ff7b409bb10 legacy=0x7ff7b4144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a0003730 0x7ff7a0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b40660a0 0x7ff7a00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a00049d0 0x7ff7a0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b41484a0 con 0x7ff7a0003730 2020-11-20T12:12:13.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b409dcd0 con 0x7ff7a00049d0 2020-11-20T12:12:13.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00041d0 con 0x7ff7b40660a0 2020-11-20T12:12:13.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.131+0000 7ff7b9263700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a00049d0 0x7ff7a0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7abfff700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b40660a0 0x7ff7a00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7b8a62700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a0003730 0x7ff7a0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7b9263700 1 -- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b40660a0 msgr2=0x7ff7a00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:13.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7b9263700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b40660a0 0x7ff7a00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7b9263700 1 -- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a0003730 msgr2=0x7ff7a0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:13.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7b9263700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a0003730 0x7ff7a0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7b9263700 1 -- 172.21.15.187:0/85637549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a0003240 con 0x7ff7a00049d0 2020-11-20T12:12:13.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7abfff700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b40660a0 0x7ff7a00044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:13.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7b8a62700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a0003730 0x7ff7a0003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:13.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.132+0000 7ff7b9263700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a00049d0 0x7ff7a0004e50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7ff7ac00c810 tx=0x7ff7b409d0d0).ready entity=mon.1 client_cookie=61ef98e7ea82561e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:13.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac005b80 con 0x7ff7a00049d0 2020-11-20T12:12:13.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac005ce0 con 0x7ff7a00049d0 2020-11-20T12:12:13.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7ab7fe700 1 -- 172.21.15.187:0/85637549 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac0072e0 con 0x7ff7a00049d0 2020-11-20T12:12:13.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7bacc6700 1 -- 172.21.15.187:0/85637549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a00049d0 msgr2=0x7ff7a0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7bacc6700 1 --2- 172.21.15.187:0/85637549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a00049d0 0x7ff7a0004e50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7ff7ac00c810 tx=0x7ff7b409d0d0).stop 2020-11-20T12:12:13.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7b9263700 1 -- 172.21.15.187:0/85637549 reap_dead start 2020-11-20T12:12:13.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7bacc6700 1 -- 172.21.15.187:0/85637549 shutdown_connections 2020-11-20T12:12:13.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7bacc6700 1 -- 172.21.15.187:0/85637549 >> 172.21.15.187:0/85637549 conn(0x7ff7b40b7fc0 msgr2=0x7ff7b40b83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:13.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7bacc6700 1 -- 172.21.15.187:0/85637549 shutdown_connections 2020-11-20T12:12:13.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.133+0000 7ff7bacc6700 1 -- 172.21.15.187:0/85637549 wait complete. 2020-11-20T12:12:13.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.135+0000 7ff7bacc6700 1 Processor -- start 2020-11-20T12:12:13.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.135+0000 7ff7bacc6700 1 -- start start 2020-11-20T12:12:13.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.136+0000 7ff7bacc6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7b408f5e0 0x7ff7b4157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.136+0000 7ff7bacc6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7b408f9c0 0x7ff7b4143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.136+0000 7ff7bacc6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b408fe40 0x7ff7b4143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.136+0000 7ff7bacc6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b415c280 con 0x7ff7b408f9c0 2020-11-20T12:12:13.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.136+0000 7ff7bacc6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b41583f0 con 0x7ff7b408f5e0 2020-11-20T12:12:13.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.136+0000 7ff7bacc6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b4158690 con 0x7ff7b408fe40 2020-11-20T12:12:13.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7b408f5e0 0x7ff7b4157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7b408f5e0 0x7ff7b4157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46918/0 (socket says 172.21.15.187:46918) 2020-11-20T12:12:13.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 -- 172.21.15.187:0/3727265158 learned_addr learned my addr 172.21.15.187:0/3727265158 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:13.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 -- 172.21.15.187:0/3727265158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b408fe40 msgr2=0x7ff7b4143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b408fe40 0x7ff7b4143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:13.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 -- 172.21.15.187:0/3727265158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7b408f9c0 msgr2=0x7ff7b4143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7b408f9c0 0x7ff7b4143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:13.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 -- 172.21.15.187:0/3727265158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b4158930 con 0x7ff7b408f5e0 2020-11-20T12:12:13.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.137+0000 7ff7b8a62700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7b408f5e0 0x7ff7b4157dd0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7ff7b000e760 tx=0x7ff7b0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:13.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.138+0000 7ff7a8ff9700 1 -- 172.21.15.187:0/3727265158 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0010560 con 0x7ff7b408f5e0 2020-11-20T12:12:13.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.138+0000 7ff7a8ff9700 1 -- 172.21.15.187:0/3727265158 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b00106c0 con 0x7ff7b408f5e0 2020-11-20T12:12:13.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.138+0000 7ff7a8ff9700 1 -- 172.21.15.187:0/3727265158 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0010a40 con 0x7ff7b408f5e0 2020-11-20T12:12:13.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.138+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b40b6b60 con 0x7ff7b408f5e0 2020-11-20T12:12:13.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.138+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b40b70a0 con 0x7ff7b408f5e0 2020-11-20T12:12:13.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.139+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b4092740 con 0x7ff7b408f5e0 2020-11-20T12:12:13.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.140+0000 7ff7a8ff9700 1 -- 172.21.15.187:0/3727265158 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff7b0012070 con 0x7ff7b408f5e0 2020-11-20T12:12:13.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.140+0000 7ff7a8ff9700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff78c056d20 0x7ff78c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.140+0000 7ff7a8ff9700 1 -- 172.21.15.187:0/3727265158 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff7b006f510 con 0x7ff7b408f5e0 2020-11-20T12:12:13.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.141+0000 7ff7abfff700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff78c056d20 0x7ff78c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.141+0000 7ff7abfff700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff78c056d20 0x7ff78c0591e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff7b40a15f0 tx=0x7ff7a4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:13.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.144+0000 7ff7a8ff9700 1 -- 172.21.15.187:0/3727265158 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff7b0044900 con 0x7ff7b408f5e0 2020-11-20T12:12:13.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.329+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7b4044f60 con 0x7ff7b408f5e0 2020-11-20T12:12:13.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.332+0000 7ff7a8ff9700 1 -- 172.21.15.187:0/3727265158 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16781 (secure 0 0 0) 0x7ff7b0047f20 con 0x7ff7b408f5e0 2020-11-20T12:12:13.336 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:13.336 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":81,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:12:11.318992+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:12:07.846671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2020-11-20T12:12:09.337957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:12:13.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff78c056d20 msgr2=0x7ff78c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff78c056d20 0x7ff78c0591e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff7b40a15f0 tx=0x7ff7a4008040).stop 2020-11-20T12:12:13.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7b408f5e0 msgr2=0x7ff7b4157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7b408f5e0 0x7ff7b4157dd0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7ff7b000e760 tx=0x7ff7b0008040).stop 2020-11-20T12:12:13.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 shutdown_connections 2020-11-20T12:12:13.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff78c056d20 0x7ff78c0591e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7b408fe40 0x7ff7b4143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:13.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7b408f5e0 0x7ff7b4157dd0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 --2- 172.21.15.187:0/3727265158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7b408f9c0 0x7ff7b4143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:13.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 >> 172.21.15.187:0/3727265158 conn(0x7ff7b40b7fc0 msgr2=0x7ff7b40601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:13.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 shutdown_connections 2020-11-20T12:12:13.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.334+0000 7ff7bacc6700 1 -- 172.21.15.187:0/3727265158 wait complete. 2020-11-20T12:12:13.359 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:12:13.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.598+0000 7f5f74d2e700 1 Processor -- start 2020-11-20T12:12:13.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.598+0000 7f5f74d2e700 1 -- start start 2020-11-20T12:12:13.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.599+0000 7f5f74d2e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f70065f90 0x7f5f70065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.599+0000 7f5f74d2e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5f7009ca90 0x7f5f7009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.599+0000 7f5f74d2e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f700b13e0 con 0x7f5f70065190 2020-11-20T12:12:13.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.599+0000 7f5f74d2e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7014ab30 con 0x7f5f7009ca90 2020-11-20T12:12:13.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.599+0000 7f5f74d2e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f70062c10 con 0x7f5f70065f90 2020-11-20T12:12:13.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.600+0000 7f5f6ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5f7009ca90 0x7f5f7009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.600+0000 7f5f6ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5f7009ca90 0x7f5f7009cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:39194/0 (socket says 172.21.15.187:39194) 2020-11-20T12:12:13.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.600+0000 7f5f6ffff700 1 -- 172.21.15.187:0/2111690391 learned_addr learned my addr 172.21.15.187:0/2111690391 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:13.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.601+0000 7f5f66ffd700 1 --2- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f70065f90 0x7f5f70065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.601+0000 7f5f6ffff700 1 -- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f70065f90 msgr2=0x7f5f70065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:13.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.601+0000 7f5f6ffff700 1 --2- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f70065f90 0x7f5f70065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.601+0000 7f5f6ffff700 1 -- 172.21.15.187:0/2111690391 >> v1:172.21.15.5:6789/0 conn(0x7f5f70065190 legacy=0x7f5f70143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.601+0000 7f5f6ffff700 1 -- 172.21.15.187:0/2111690391 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f700b1640 con 0x7f5f7009ca90 2020-11-20T12:12:13.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.601+0000 7f5f66ffd700 1 --2- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f70065f90 0x7f5f70065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:13.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.602+0000 7f5f6ffff700 1 --2- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f7009ca90 0x7f5f7009cf20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f5f7014ac70 tx=0x7f5f68006eb0).ready entity=mon.0 client_cookie=1b12e17cfcd8828e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:13.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f6effd700 1 -- 172.21.15.187:0/2111690391 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f68009960 con 0x7f5f7009ca90 2020-11-20T12:12:13.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f6effd700 1 -- 172.21.15.187:0/2111690391 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f68009ac0 con 0x7f5f7009ca90 2020-11-20T12:12:13.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f6effd700 1 -- 172.21.15.187:0/2111690391 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f68009e40 con 0x7f5f7009ca90 2020-11-20T12:12:13.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f74d2e700 1 -- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f7009ca90 msgr2=0x7f5f7009cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f7009ca90 0x7f5f7009cf20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f5f7014ac70 tx=0x7f5f68006eb0).stop 2020-11-20T12:12:13.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f74d2e700 1 -- 172.21.15.187:0/2111690391 shutdown_connections 2020-11-20T12:12:13.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f7009ca90 0x7f5f7009cf20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/2111690391 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f70065f90 0x7f5f70065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.603+0000 7f5f74d2e700 1 -- 172.21.15.187:0/2111690391 >> 172.21.15.187:0/2111690391 conn(0x7f5f700bab30 msgr2=0x7f5f700baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:13.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.604+0000 7f5f74d2e700 1 -- 172.21.15.187:0/2111690391 shutdown_connections 2020-11-20T12:12:13.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.604+0000 7f5f74d2e700 1 -- 172.21.15.187:0/2111690391 wait complete. 2020-11-20T12:12:13.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.605+0000 7f5f74d2e700 1 Processor -- start 2020-11-20T12:12:13.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.606+0000 7f5f74d2e700 1 -- start start 2020-11-20T12:12:13.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.606+0000 7f5f74d2e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f70065190 0x7f5f7009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f74d2e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f70065f90 0x7f5f7015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f74d2e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f7009ca90 0x7f5f7015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f74d2e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f70099240 con 0x7f5f70065f90 2020-11-20T12:12:13.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f74d2e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7009b120 con 0x7f5f70065190 2020-11-20T12:12:13.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f74d2e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7009b290 con 0x7f5f7009ca90 2020-11-20T12:12:13.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f6f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f70065190 0x7f5f7009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f6f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f70065190 0x7f5f7009ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46932/0 (socket says 172.21.15.187:46932) 2020-11-20T12:12:13.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f6f7fe700 1 -- 172.21.15.187:0/4200590006 learned_addr learned my addr 172.21.15.187:0/4200590006 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:13.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f6ffff700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f7009ca90 0x7f5f7015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.607+0000 7f5f6f7fe700 1 -- 172.21.15.187:0/4200590006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f7009ca90 msgr2=0x7f5f7015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f66ffd700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f70065f90 0x7f5f7015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f6f7fe700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f7009ca90 0x7f5f7015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f6f7fe700 1 -- 172.21.15.187:0/4200590006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f70065f90 msgr2=0x7f5f7015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f6f7fe700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f70065f90 0x7f5f7015bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f6f7fe700 1 -- 172.21.15.187:0/4200590006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f70099800 con 0x7f5f70065190 2020-11-20T12:12:13.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f6f7fe700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f70065190 0x7f5f7009ac30 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f5f6000ca90 tx=0x7f5f60001460).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:13.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f67fff700 1 -- 172.21.15.187:0/4200590006 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6000a880 con 0x7f5f70065190 2020-11-20T12:12:13.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f67fff700 1 -- 172.21.15.187:0/4200590006 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6000a9e0 con 0x7f5f70065190 2020-11-20T12:12:13.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f700b4880 con 0x7f5f70065190 2020-11-20T12:12:13.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f67fff700 1 -- 172.21.15.187:0/4200590006 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6000ad60 con 0x7f5f70065190 2020-11-20T12:12:13.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.608+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f700674b0 con 0x7f5f70065190 2020-11-20T12:12:13.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.610+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f70061a20 con 0x7f5f70065190 2020-11-20T12:12:13.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.610+0000 7f5f67fff700 1 -- 172.21.15.187:0/4200590006 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5f60018020 con 0x7f5f70065190 2020-11-20T12:12:13.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.611+0000 7f5f67fff700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f4c056d20 0x7f5f4c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:13.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.611+0000 7f5f67fff700 1 -- 172.21.15.187:0/4200590006 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5f6006fcc0 con 0x7f5f70065190 2020-11-20T12:12:13.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.611+0000 7f5f66ffd700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f4c056d20 0x7f5f4c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:13.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.612+0000 7f5f66ffd700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f4c056d20 0x7f5f4c0591e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f5f58006bd0 tx=0x7f5f58008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:13.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.615+0000 7f5f67fff700 1 -- 172.21.15.187:0/4200590006 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5f60045110 con 0x7f5f70065190 2020-11-20T12:12:13.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.849+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f5f700afc60 con 0x7f5f70065190 2020-11-20T12:12:13.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.859+0000 7f5f67fff700 1 -- 172.21.15.187:0/4200590006 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5f60016030 con 0x7f5f70065190 2020-11-20T12:12:13.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f4c056d20 msgr2=0x7f5f4c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f4c056d20 0x7f5f4c0591e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f5f58006bd0 tx=0x7f5f58008040).stop 2020-11-20T12:12:13.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f70065190 msgr2=0x7f5f7009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:13.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f70065190 0x7f5f7009ac30 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f5f6000ca90 tx=0x7f5f60001460).stop 2020-11-20T12:12:13.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 shutdown_connections 2020-11-20T12:12:13.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f4c056d20 0x7f5f4c0591e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5f7009ca90 0x7f5f7015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f70065190 0x7f5f7009ac30 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 --2- 172.21.15.187:0/4200590006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5f70065f90 0x7f5f7015bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:13.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 >> 172.21.15.187:0/4200590006 conn(0x7f5f700bab30 msgr2=0x7f5f700b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:13.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 shutdown_connections 2020-11-20T12:12:13.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:13.862+0000 7f5f74d2e700 1 -- 172.21.15.187:0/4200590006 wait complete. 2020-11-20T12:12:13.869 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:12:13.889 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:14.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.127+0000 7fce0f59e700 1 Processor -- start 2020-11-20T12:12:14.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.127+0000 7fce0f59e700 1 -- start start 2020-11-20T12:12:14.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.129+0000 7fce0f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fce100bfbe0 0x7fce101480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.129+0000 7fce0f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fce1009f560 0x7fce101485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.129+0000 7fce0f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1014d110 con 0x7fce100bf800 2020-11-20T12:12:14.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.129+0000 7fce0f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1014c9d0 con 0x7fce100bfbe0 2020-11-20T12:12:14.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.129+0000 7fce0f59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce10148aa0 con 0x7fce1009f560 2020-11-20T12:12:14.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.129+0000 7fce0ed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fce1009f560 0x7fce101485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.130+0000 7fce0ed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fce1009f560 0x7fce101485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46944/0 (socket says 172.21.15.187:46944) 2020-11-20T12:12:14.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.130+0000 7fce0ed9d700 1 -- 172.21.15.187:0/3377699024 learned_addr learned my addr 172.21.15.187:0/3377699024 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:14.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.130+0000 7fce0dd9b700 1 --2- 172.21.15.187:0/3377699024 >> v2:172.21.15.5:3300/0 conn(0x7fce100bfbe0 0x7fce101480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.130+0000 7fce0ed9d700 1 -- 172.21.15.187:0/3377699024 >> v2:172.21.15.5:3300/0 conn(0x7fce100bfbe0 msgr2=0x7fce101480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:14.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.130+0000 7fce0ed9d700 1 --2- 172.21.15.187:0/3377699024 >> v2:172.21.15.5:3300/0 conn(0x7fce100bfbe0 0x7fce101480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.130+0000 7fce0ed9d700 1 -- 172.21.15.187:0/3377699024 >> v1:172.21.15.5:6789/0 conn(0x7fce100bf800 legacy=0x7fce1014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:14.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.130+0000 7fce0ed9d700 1 -- 172.21.15.187:0/3377699024 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1014cbd0 con 0x7fce1009f560 2020-11-20T12:12:14.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.130+0000 7fce0ed9d700 1 --2- 172.21.15.187:0/3377699024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce101485b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fce1009f2b0 tx=0x7fce08009290).ready entity=mon.1 client_cookie=d90732da6b1df1c4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:14.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0d59a700 1 -- 172.21.15.187:0/3377699024 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0800bc10 con 0x7fce1009f560 2020-11-20T12:12:14.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0d59a700 1 -- 172.21.15.187:0/3377699024 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0800bd70 con 0x7fce1009f560 2020-11-20T12:12:14.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0d59a700 1 -- 172.21.15.187:0/3377699024 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08011610 con 0x7fce1009f560 2020-11-20T12:12:14.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0f59e700 1 -- 172.21.15.187:0/3377699024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 msgr2=0x7fce101485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0f59e700 1 --2- 172.21.15.187:0/3377699024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce101485b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fce1009f2b0 tx=0x7fce08009290).stop 2020-11-20T12:12:14.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0f59e700 1 -- 172.21.15.187:0/3377699024 shutdown_connections 2020-11-20T12:12:14.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0f59e700 1 --2- 172.21.15.187:0/3377699024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce101485b0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0f59e700 1 --2- 172.21.15.187:0/3377699024 >> v2:172.21.15.5:3300/0 conn(0x7fce100bfbe0 0x7fce101480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0f59e700 1 -- 172.21.15.187:0/3377699024 >> 172.21.15.187:0/3377699024 conn(0x7fce10068370 msgr2=0x7fce100bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:14.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.131+0000 7fce0f59e700 1 -- 172.21.15.187:0/3377699024 shutdown_connections 2020-11-20T12:12:14.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.132+0000 7fce0f59e700 1 -- 172.21.15.187:0/3377699024 wait complete. 2020-11-20T12:12:14.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.133+0000 7fce0f59e700 1 Processor -- start 2020-11-20T12:12:14.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.133+0000 7fce0f59e700 1 -- start start 2020-11-20T12:12:14.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce1014bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fce100bf800 0x7fce10158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce10158f90 0x7fce10145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1015c690 con 0x7fce10158f90 2020-11-20T12:12:14.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce10158600 con 0x7fce1009f560 2020-11-20T12:12:14.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce101582f0 con 0x7fce100bf800 2020-11-20T12:12:14.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce1014bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce1014bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46946/0 (socket says 172.21.15.187:46946) 2020-11-20T12:12:14.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.134+0000 7fce0e59c700 1 -- 172.21.15.187:0/563771787 learned_addr learned my addr 172.21.15.187:0/563771787 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:14.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0ed9d700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce10158f90 0x7fce10145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0dd9b700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fce100bf800 0x7fce10158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0e59c700 1 -- 172.21.15.187:0/563771787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fce100bf800 msgr2=0x7fce10158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0e59c700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fce100bf800 0x7fce10158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0e59c700 1 -- 172.21.15.187:0/563771787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce10158f90 msgr2=0x7fce10145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0e59c700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce10158f90 0x7fce10145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0e59c700 1 -- 172.21.15.187:0/563771787 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1014c2a0 con 0x7fce1009f560 2020-11-20T12:12:14.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0e59c700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce1014bdb0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7fce0000c730 tx=0x7fce00007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:14.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fcdfe7fc700 1 -- 172.21.15.187:0/563771787 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0000e560 con 0x7fce1009f560 2020-11-20T12:12:14.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fcdfe7fc700 1 -- 172.21.15.187:0/563771787 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0000e6e0 con 0x7fce1009f560 2020-11-20T12:12:14.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce101479b0 con 0x7fce1009f560 2020-11-20T12:12:14.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.135+0000 7fcdfe7fc700 1 -- 172.21.15.187:0/563771787 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0000eaa0 con 0x7fce1009f560 2020-11-20T12:12:14.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.136+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce10145850 con 0x7fce1009f560 2020-11-20T12:12:14.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.137+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce10080050 con 0x7fce1009f560 2020-11-20T12:12:14.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.137+0000 7fcdfe7fc700 1 -- 172.21.15.187:0/563771787 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fce00019030 con 0x7fce1009f560 2020-11-20T12:12:14.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.138+0000 7fcdfe7fc700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcde4056d20 0x7fcde40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.138+0000 7fcdfe7fc700 1 -- 172.21.15.187:0/563771787 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fce0006f4f0 con 0x7fce1009f560 2020-11-20T12:12:14.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.138+0000 7fce0dd9b700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcde4056d20 0x7fcde40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.139+0000 7fce0dd9b700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcde4056d20 0x7fcde40591e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fce040095b0 tx=0x7fce04008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:14.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.142+0000 7fcdfe7fc700 1 -- 172.21.15.187:0/563771787 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fce00043fd0 con 0x7fce1009f560 2020-11-20T12:12:14.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.389+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce10147b40 con 0x7fce1009f560 2020-11-20T12:12:14.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.390+0000 7fcdfe7fc700 1 -- 172.21.15.187:0/563771787 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fce000449f0 con 0x7fce1009f560 2020-11-20T12:12:14.395 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:14.395 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:12.329807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6952},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:14.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcde4056d20 msgr2=0x7fcde40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcde4056d20 0x7fcde40591e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fce040095b0 tx=0x7fce04008040).stop 2020-11-20T12:12:14.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 msgr2=0x7fce1014bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce1014bdb0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7fce0000c730 tx=0x7fce00007c30).stop 2020-11-20T12:12:14.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 shutdown_connections 2020-11-20T12:12:14.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcde4056d20 0x7fcde40591e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce10158f90 0x7fce10145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce1009f560 0x7fce1014bdb0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 --2- 172.21.15.187:0/563771787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fce100bf800 0x7fce10158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.392+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 >> 172.21.15.187:0/563771787 conn(0x7fce10068370 msgr2=0x7fce100bb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:14.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.393+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 shutdown_connections 2020-11-20T12:12:14.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:14.393+0000 7fce0f59e700 1 -- 172.21.15.187:0/563771787 wait complete. 2020-11-20T12:12:14.399 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 64 2020-11-20T12:12:14.529 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:12:14.525+0000 7ff4a9e99700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:12:14.619 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:12.329807+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6952}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:12:14.620 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:12:14.620 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:12:14.901 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6952,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.5575237929999997,"mdsmap_epoch":64,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":11.103101000000001} 2020-11-20T12:12:15.141 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6952, "mdsmap_epoch": 64, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 4.557523793, "state": "up:active", "uptime": 11.103101, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:12:15.141 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:12:15.142 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:12:15.142 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:12:15.142 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:12:15.143 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:12:15.143 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:12:15.143 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:12:15.144 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:12:15.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:15.145 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:12:15.207 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:12:15.208 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:12:15.208 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:12:15.208 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:15.208 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:12:15.208 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:15.209 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:12:15.209 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:15.209 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:12:15.209 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:15.210 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:12:15.210 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:15.210 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:12:15.210 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:15.210 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:12:15.211 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42303sec preferred_lft 42303sec 2020-11-20T12:12:15.211 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:12:15.211 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:15.211 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:12:15.211 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:15.212 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:12:15.212 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:15.212 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:12:15.212 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:15.213 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:12:15.214 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:12:15.214 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:12:15.214 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:15.214 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:12:15.214 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:12:15.215 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:12:15.215 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:15.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:15.216 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:12:15.315 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:12:15.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:15.322 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:12:15.382 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:12:15.383 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:12:15.384 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:12:15.384 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:12:15.385 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:12:15.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:15.386 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:12:15.387 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:12:15.387 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:12:15.388 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:12:15.388 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:12:15.388 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:12:15.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:15.638 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:12:15.689 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:12:15.689 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:12:15.690 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:12:15.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:15.691 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:12:15.790 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:12:16.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.026+0000 7f1ffe353700 1 Processor -- start 2020-11-20T12:12:16.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.027+0000 7f1ffe353700 1 -- start start 2020-11-20T12:12:16.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.028+0000 7f1ffe353700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1ff8068230 0x7f1ff81482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.028+0000 7f1ffe353700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff8144b80 con 0x7f1ff80b57b0 2020-11-20T12:12:16.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.028+0000 7f1ffe353700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff81489e0 con 0x7f1ff80b5f50 2020-11-20T12:12:16.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.028+0000 7f1ffe353700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff81487c0 con 0x7f1ff8068230 2020-11-20T12:12:16.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.029+0000 7f1ffcb50700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1ff8068230 0x7f1ff81482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:16.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.029+0000 7f1ffcb50700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1ff8068230 0x7f1ff81482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46956/0 (socket says 172.21.15.187:46956) 2020-11-20T12:12:16.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.029+0000 7f1ffcb50700 1 -- 172.21.15.187:0/1688655586 learned_addr learned my addr 172.21.15.187:0/1688655586 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:16.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.029+0000 7f1ffdb52700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1ff80b5f50 0x7f1ff80b6380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43564/0 (socket says 172.21.15.187:43564) 2020-11-20T12:12:16.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.030+0000 7f1ffcb50700 1 -- 172.21.15.187:0/1688655586 >> v1:172.21.15.187:6789/0 conn(0x7f1ff80b5f50 legacy=0x7f1ff80b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.030+0000 7f1ffcb50700 1 -- 172.21.15.187:0/1688655586 >> v1:172.21.15.5:6789/0 conn(0x7f1ff80b57b0 legacy=0x7f1ff8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.030+0000 7f1ffcb50700 1 -- 172.21.15.187:0/1688655586 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff8148c80 con 0x7f1ff8068230 2020-11-20T12:12:16.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.030+0000 7f1ffcb50700 1 --2- 172.21.15.187:0/1688655586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff81482d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7f1fe8006fd0 tx=0x7f1fe8006eb0).ready entity=mon.1 client_cookie=a8257b87c90ec555 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:16.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.030+0000 7f1fe7fff700 1 -- 172.21.15.187:0/1688655586 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80098d0 con 0x7f1ff8068230 2020-11-20T12:12:16.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.030+0000 7f1fe7fff700 1 -- 172.21.15.187:0/1688655586 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe8009a30 con 0x7f1ff8068230 2020-11-20T12:12:16.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.030+0000 7f1fe7fff700 1 -- 172.21.15.187:0/1688655586 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8009db0 con 0x7f1ff8068230 2020-11-20T12:12:16.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.031+0000 7f1ffe353700 1 -- 172.21.15.187:0/1688655586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 msgr2=0x7f1ff81482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.031+0000 7f1ffe353700 1 --2- 172.21.15.187:0/1688655586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff81482d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7f1fe8006fd0 tx=0x7f1fe8006eb0).stop 2020-11-20T12:12:16.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.031+0000 7f1ffe353700 1 -- 172.21.15.187:0/1688655586 shutdown_connections 2020-11-20T12:12:16.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.031+0000 7f1ffe353700 1 --2- 172.21.15.187:0/1688655586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff81482d0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.031+0000 7f1ffe353700 1 -- 172.21.15.187:0/1688655586 >> 172.21.15.187:0/1688655586 conn(0x7f1ff80bab30 msgr2=0x7f1ff80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:16.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.031+0000 7f1ffe353700 1 -- 172.21.15.187:0/1688655586 shutdown_connections 2020-11-20T12:12:16.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.031+0000 7f1ffe353700 1 -- 172.21.15.187:0/1688655586 wait complete. 2020-11-20T12:12:16.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.033+0000 7f1ffe353700 1 Processor -- start 2020-11-20T12:12:16.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.033+0000 7f1ffe353700 1 -- start start 2020-11-20T12:12:16.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffe353700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffe353700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ff80b57b0 0x7f1ff815bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffe353700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ff80b5f50 0x7f1ff809f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffe353700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8145e90 con 0x7f1ff80b57b0 2020-11-20T12:12:16.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffe353700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff815c850 con 0x7f1ff8068230 2020-11-20T12:12:16.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffe353700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8145bf0 con 0x7f1ff80b5f50 2020-11-20T12:12:16.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffd351700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:16.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffd351700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff809cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46960/0 (socket says 172.21.15.187:46960) 2020-11-20T12:12:16.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffd351700 1 -- 172.21.15.187:0/3392851131 learned_addr learned my addr 172.21.15.187:0/3392851131 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:16.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffd351700 1 -- 172.21.15.187:0/3392851131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ff80b5f50 msgr2=0x7f1ff809f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffd351700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ff80b5f50 0x7f1ff809f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:16.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffd351700 1 -- 172.21.15.187:0/3392851131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ff80b57b0 msgr2=0x7f1ff815bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffd351700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ff80b57b0 0x7f1ff815bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:16.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.034+0000 7f1ffd351700 1 -- 172.21.15.187:0/3392851131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff809c790 con 0x7f1ff8068230 2020-11-20T12:12:16.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.035+0000 7f1ffd351700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff809cc30 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f1ff400c740 tx=0x7f1ff4007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:16.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.035+0000 7f1fe57fa700 1 -- 172.21.15.187:0/3392851131 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff400e560 con 0x7f1ff8068230 2020-11-20T12:12:16.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.035+0000 7f1fe57fa700 1 -- 172.21.15.187:0/3392851131 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff400e6c0 con 0x7f1ff8068230 2020-11-20T12:12:16.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.035+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff809d120 con 0x7f1ff8068230 2020-11-20T12:12:16.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.035+0000 7f1fe57fa700 1 -- 172.21.15.187:0/3392851131 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff400ea60 con 0x7f1ff8068230 2020-11-20T12:12:16.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.035+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff8145750 con 0x7f1ff8068230 2020-11-20T12:12:16.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.036+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff8145a00 con 0x7f1ff8068230 2020-11-20T12:12:16.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.037+0000 7f1fe57fa700 1 -- 172.21.15.187:0/3392851131 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1ff4019030 con 0x7f1ff8068230 2020-11-20T12:12:16.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.037+0000 7f1fe57fa700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fd0056d20 0x7f1fd00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.037+0000 7f1fe57fa700 1 -- 172.21.15.187:0/3392851131 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ff406f5d0 con 0x7f1ff8068230 2020-11-20T12:12:16.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.038+0000 7f1ffcb50700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fd0056d20 0x7f1fd00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:16.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.038+0000 7f1ffcb50700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fd0056d20 0x7f1fd00591e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f1ff809b700 tx=0x7f1fe8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:16.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.041+0000 7f1fe57fa700 1 -- 172.21.15.187:0/3392851131 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1ff40449a0 con 0x7f1ff8068230 2020-11-20T12:12:16.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.228+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ff8061a20 con 0x7f1ff8068230 2020-11-20T12:12:16.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.230+0000 7f1fe57fa700 1 -- 172.21.15.187:0/3392851131 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f1ff4017020 con 0x7f1ff8068230 2020-11-20T12:12:16.235 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:16.235 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":81,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:12:11.318992+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:12:07.846671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2020-11-20T12:12:09.337957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:12:16.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fd0056d20 msgr2=0x7f1fd00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fd0056d20 0x7f1fd00591e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f1ff809b700 tx=0x7f1fe8015000).stop 2020-11-20T12:12:16.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 msgr2=0x7f1ff809cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff809cc30 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f1ff400c740 tx=0x7f1ff4007c10).stop 2020-11-20T12:12:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 shutdown_connections 2020-11-20T12:12:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1fd0056d20 0x7f1fd00591e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ff80b5f50 0x7f1ff809f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:16.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff8068230 0x7f1ff809cc30 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 --2- 172.21.15.187:0/3392851131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ff80b57b0 0x7f1ff815bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:16.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 >> 172.21.15.187:0/3392851131 conn(0x7f1ff80bab30 msgr2=0x7f1ff8147ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:16.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 shutdown_connections 2020-11-20T12:12:16.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.233+0000 7f1ffe353700 1 -- 172.21.15.187:0/3392851131 wait complete. 2020-11-20T12:12:16.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:12:07.846671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:12:09.337957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:12:16.268 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:12:16.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.529+0000 7f0dac8f0700 1 Processor -- start 2020-11-20T12:12:16.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.529+0000 7f0dac8f0700 1 -- start start 2020-11-20T12:12:16.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.531+0000 7f0dac8f0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0da80b57b0 0x7f0da8068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.531+0000 7f0dac8f0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da8144b80 con 0x7f0da81482d0 2020-11-20T12:12:16.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.531+0000 7f0dac8f0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da81489e0 con 0x7f0da80b5f50 2020-11-20T12:12:16.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.531+0000 7f0dac8f0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da8148c80 con 0x7f0da80b57b0 2020-11-20T12:12:16.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.531+0000 7f0da77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0da80b57b0 0x7f0da8068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:16.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.531+0000 7f0da77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0da80b57b0 0x7f0da8068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46968/0 (socket says 172.21.15.187:46968) 2020-11-20T12:12:16.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.531+0000 7f0da77fe700 1 -- 172.21.15.187:0/1334313927 learned_addr learned my addr 172.21.15.187:0/1334313927 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:16.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.532+0000 7f0da77fe700 1 -- 172.21.15.187:0/1334313927 >> v1:172.21.15.187:6789/0 conn(0x7f0da80b5f50 legacy=0x7f0da80b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:16.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.532+0000 7f0da77fe700 1 -- 172.21.15.187:0/1334313927 >> v1:172.21.15.5:6789/0 conn(0x7f0da81482d0 legacy=0x7f0da8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:16.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.532+0000 7f0da77fe700 1 -- 172.21.15.187:0/1334313927 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da8148f20 con 0x7f0da80b57b0 2020-11-20T12:12:16.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.533+0000 7f0da77fe700 1 --2- 172.21.15.187:0/1334313927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da80b57b0 0x7f0da8068230 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7f0d98006fd0 tx=0x7f0d98006eb0).ready entity=mon.1 client_cookie=151af2ade3f98799 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:16.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.533+0000 7f0da6ffd700 1 -- 172.21.15.187:0/1334313927 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d98009890 con 0x7f0da80b57b0 2020-11-20T12:12:16.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.533+0000 7f0da6ffd700 1 -- 172.21.15.187:0/1334313927 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d980099f0 con 0x7f0da80b57b0 2020-11-20T12:12:16.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.533+0000 7f0da6ffd700 1 -- 172.21.15.187:0/1334313927 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d98009d70 con 0x7f0da80b57b0 2020-11-20T12:12:16.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.534+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1334313927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da80b57b0 msgr2=0x7f0da8068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.534+0000 7f0dac8f0700 1 --2- 172.21.15.187:0/1334313927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da80b57b0 0x7f0da8068230 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7f0d98006fd0 tx=0x7f0d98006eb0).stop 2020-11-20T12:12:16.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.534+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1334313927 shutdown_connections 2020-11-20T12:12:16.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.534+0000 7f0dac8f0700 1 --2- 172.21.15.187:0/1334313927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da80b57b0 0x7f0da8068230 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.534+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1334313927 >> 172.21.15.187:0/1334313927 conn(0x7f0da80bab30 msgr2=0x7f0da80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:16.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.534+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1334313927 shutdown_connections 2020-11-20T12:12:16.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.534+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1334313927 wait complete. 2020-11-20T12:12:16.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.535+0000 7f0dac8f0700 1 Processor -- start 2020-11-20T12:12:16.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.536+0000 7f0dac8f0700 1 -- start start 2020-11-20T12:12:16.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0dac8f0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0da80b5f50 0x7f0da80b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0dac8f0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da81482d0 0x7f0da8141b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0dac8f0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0da80b1bb0 0x7f0da815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0dac8f0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da8146ca0 con 0x7f0da80b5f50 2020-11-20T12:12:16.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0dac8f0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da8146f40 con 0x7f0da81482d0 2020-11-20T12:12:16.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0dac8f0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da80b3130 con 0x7f0da80b1bb0 2020-11-20T12:12:16.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0d9effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da81482d0 0x7f0da8141b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:16.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0d9effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da81482d0 0x7f0da8141b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46974/0 (socket says 172.21.15.187:46974) 2020-11-20T12:12:16.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0d9effd700 1 -- 172.21.15.187:0/1975056065 learned_addr learned my addr 172.21.15.187:0/1975056065 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:16.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0d9effd700 1 -- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0da80b1bb0 msgr2=0x7f0da815bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0da7fff700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0da80b1bb0 0x7f0da815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:16.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.537+0000 7f0da77fe700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0da80b5f50 0x7f0da80b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:16.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0d9effd700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0da80b1bb0 0x7f0da815bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0d9effd700 1 -- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0da80b5f50 msgr2=0x7f0da80b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0d9effd700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0da80b5f50 0x7f0da80b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0d9effd700 1 -- 172.21.15.187:0/1975056065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da80b2d10 con 0x7f0da81482d0 2020-11-20T12:12:16.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0da7fff700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0da80b1bb0 0x7f0da815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:16.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0da77fe700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0da80b5f50 0x7f0da80b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:16.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0d9effd700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da81482d0 0x7f0da8141b50 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7f0d900096e0 tx=0x7f0d90006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:16.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0d9ffff700 1 -- 172.21.15.187:0/1975056065 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9000d560 con 0x7f0da81482d0 2020-11-20T12:12:16.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0d9ffff700 1 -- 172.21.15.187:0/1975056065 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d90000e90 con 0x7f0da81482d0 2020-11-20T12:12:16.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da815c010 con 0x7f0da81482d0 2020-11-20T12:12:16.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0d9ffff700 1 -- 172.21.15.187:0/1975056065 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9000d930 con 0x7f0da81482d0 2020-11-20T12:12:16.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.538+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da80b08f0 con 0x7f0da81482d0 2020-11-20T12:12:16.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.540+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da8147100 con 0x7f0da81482d0 2020-11-20T12:12:16.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.540+0000 7f0d9ffff700 1 -- 172.21.15.187:0/1975056065 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0d9000f030 con 0x7f0da81482d0 2020-11-20T12:12:16.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.541+0000 7f0d9ffff700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d84056d20 0x7f0d840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:16.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.541+0000 7f0d9ffff700 1 -- 172.21.15.187:0/1975056065 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0d9006fc70 con 0x7f0da81482d0 2020-11-20T12:12:16.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.541+0000 7f0da77fe700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d84056d20 0x7f0d840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:16.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.542+0000 7f0da77fe700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d84056d20 0x7f0d840591e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f0da8062fe0 tx=0x7f0d98015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:16.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.544+0000 7f0d9ffff700 1 -- 172.21.15.187:0/1975056065 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0d90045040 con 0x7f0da81482d0 2020-11-20T12:12:16.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.730+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0da8061a20 con 0x7f0da81482d0 2020-11-20T12:12:16.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.731+0000 7f0d9ffff700 1 -- 172.21.15.187:0/1975056065 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0d90048660 con 0x7f0da81482d0 2020-11-20T12:12:16.736 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:12:16.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d84056d20 msgr2=0x7f0d840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d84056d20 0x7f0d840591e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f0da8062fe0 tx=0x7f0d98015000).stop 2020-11-20T12:12:16.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da81482d0 msgr2=0x7f0da8141b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:16.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da81482d0 0x7f0da8141b50 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7f0d900096e0 tx=0x7f0d90006d70).stop 2020-11-20T12:12:16.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 shutdown_connections 2020-11-20T12:12:16.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d84056d20 0x7f0d840591e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0da80b5f50 0x7f0da80b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0da80b1bb0 0x7f0da815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 --2- 172.21.15.187:0/1975056065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0da81482d0 0x7f0da8141b50 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:16.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 >> 172.21.15.187:0/1975056065 conn(0x7f0da80bab30 msgr2=0x7f0da8065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:16.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 shutdown_connections 2020-11-20T12:12:16.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:16.734+0000 7f0dac8f0700 1 -- 172.21.15.187:0/1975056065 wait complete. 2020-11-20T12:12:16.762 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:12:17.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.010+0000 7f5ed1ccc700 1 Processor -- start 2020-11-20T12:12:17.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.010+0000 7f5ed1ccc700 1 -- start start 2020-11-20T12:12:17.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.011+0000 7f5ed1ccc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ecc0b57b0 0x7f5ecc068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.011+0000 7f5ed1ccc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0b5f50 0x7f5ecc0b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.011+0000 7f5ed1ccc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc144b80 con 0x7f5ecc1482d0 2020-11-20T12:12:17.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.011+0000 7f5ed1ccc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc1489e0 con 0x7f5ecc0b5f50 2020-11-20T12:12:17.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.011+0000 7f5ed1ccc700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc148c80 con 0x7f5ecc0b57b0 2020-11-20T12:12:17.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.012+0000 7f5ed0cca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ecc0b57b0 0x7f5ecc068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.012+0000 7f5ed0cca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ecc0b57b0 0x7f5ecc068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46984/0 (socket says 172.21.15.187:46984) 2020-11-20T12:12:17.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.012+0000 7f5ed0cca700 1 -- 172.21.15.187:0/1156704293 learned_addr learned my addr 172.21.15.187:0/1156704293 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:17.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.012+0000 7f5ed14cb700 1 --2- 172.21.15.187:0/1156704293 >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0b5f50 0x7f5ecc0b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.012+0000 7f5ed0cca700 1 -- 172.21.15.187:0/1156704293 >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0b5f50 msgr2=0x7f5ecc0b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:17.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.012+0000 7f5ed0cca700 1 --2- 172.21.15.187:0/1156704293 >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0b5f50 0x7f5ecc0b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.013+0000 7f5ed0cca700 1 -- 172.21.15.187:0/1156704293 >> v1:172.21.15.5:6789/0 conn(0x7f5ecc1482d0 legacy=0x7f5ecc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:17.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.013+0000 7f5ed0cca700 1 -- 172.21.15.187:0/1156704293 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc148f20 con 0x7f5ecc0b57b0 2020-11-20T12:12:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.013+0000 7f5ed0cca700 1 --2- 172.21.15.187:0/1156704293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b57b0 0x7f5ecc068230 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7f5ecc0b85e0 tx=0x7f5ec8006eb0).ready entity=mon.1 client_cookie=27d3888ec20710d7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.013+0000 7f5ec37fe700 1 -- 172.21.15.187:0/1156704293 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8009890 con 0x7f5ecc0b57b0 2020-11-20T12:12:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.013+0000 7f5ec37fe700 1 -- 172.21.15.187:0/1156704293 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec80099f0 con 0x7f5ecc0b57b0 2020-11-20T12:12:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.013+0000 7f5ec37fe700 1 -- 172.21.15.187:0/1156704293 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8009d70 con 0x7f5ecc0b57b0 2020-11-20T12:12:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.014+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1156704293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b57b0 msgr2=0x7f5ecc068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.014+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1156704293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b57b0 0x7f5ecc068230 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7f5ecc0b85e0 tx=0x7f5ec8006eb0).stop 2020-11-20T12:12:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.014+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1156704293 shutdown_connections 2020-11-20T12:12:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.014+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1156704293 >> v2:172.21.15.5:3300/0 conn(0x7f5ecc0b5f50 0x7f5ecc0b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.014+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1156704293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc0b57b0 0x7f5ecc068230 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.014+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1156704293 >> 172.21.15.187:0/1156704293 conn(0x7f5ecc0bab30 msgr2=0x7f5ecc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.014+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1156704293 shutdown_connections 2020-11-20T12:12:17.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.014+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1156704293 wait complete. 2020-11-20T12:12:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.016+0000 7f5ed1ccc700 1 Processor -- start 2020-11-20T12:12:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.016+0000 7f5ed1ccc700 1 -- start start 2020-11-20T12:12:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ed1ccc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b5f50 0x7f5ecc138e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ed1ccc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc1482d0 0x7f5ecc139340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ed1ccc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc0b38c0 0x7f5ecc143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ed1ccc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0b0fd0 con 0x7f5ecc0b5f50 2020-11-20T12:12:17.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ed1ccc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc139830 con 0x7f5ecc1482d0 2020-11-20T12:12:17.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ed1ccc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0b4970 con 0x7f5ecc0b38c0 2020-11-20T12:12:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ec3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc1482d0 0x7f5ecc139340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ec3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc1482d0 0x7f5ecc139340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46990/0 (socket says 172.21.15.187:46990) 2020-11-20T12:12:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ec3fff700 1 -- 172.21.15.187:0/1047548807 learned_addr learned my addr 172.21.15.187:0/1047548807 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.018+0000 7f5ed0cca700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b5f50 0x7f5ecc138e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec3fff700 1 -- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc0b38c0 msgr2=0x7f5ecc143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ed14cb700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc0b38c0 0x7f5ecc143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec3fff700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc0b38c0 0x7f5ecc143b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec3fff700 1 -- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b5f50 msgr2=0x7f5ecc138e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec3fff700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b5f50 0x7f5ecc138e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec3fff700 1 -- 172.21.15.187:0/1047548807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0b3420 con 0x7f5ecc1482d0 2020-11-20T12:12:17.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ed14cb700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc0b38c0 0x7f5ecc143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:17.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec3fff700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc1482d0 0x7f5ecc139340 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f5ebc00c730 tx=0x7f5ebc007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/1047548807 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc00e560 con 0x7f5ecc1482d0 2020-11-20T12:12:17.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/1047548807 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc00e6e0 con 0x7f5ecc1482d0 2020-11-20T12:12:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc0b3140 con 0x7f5ecc1482d0 2020-11-20T12:12:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/1047548807 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc00eaa0 con 0x7f5ecc1482d0 2020-11-20T12:12:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.019+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc0b4630 con 0x7f5ecc1482d0 2020-11-20T12:12:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.021+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc061a20 con 0x7f5ecc1482d0 2020-11-20T12:12:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.021+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/1047548807 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5ebc019030 con 0x7f5ecc1482d0 2020-11-20T12:12:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.022+0000 7f5ec0ff9700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.022+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/1047548807 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5ebc06f4b0 con 0x7f5ecc1482d0 2020-11-20T12:12:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.022+0000 7f5ed0cca700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.023+0000 7f5ed0cca700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f5ecc062fe0 tx=0x7f5ec8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.026+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/1047548807 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5ebc043fd0 con 0x7f5ecc1482d0 2020-11-20T12:12:17.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.212+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5ecc0446d0 con 0x7f5ecc1482d0 2020-11-20T12:12:17.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.213+0000 7f5ec0ff9700 1 -- 172.21.15.187:0/1047548807 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5ebc0449b0 con 0x7f5ecc1482d0 2020-11-20T12:12:17.217 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:12:17.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 msgr2=0x7f5ea40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f5ecc062fe0 tx=0x7f5ec8015000).stop 2020-11-20T12:12:17.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc1482d0 msgr2=0x7f5ecc139340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc1482d0 0x7f5ecc139340 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f5ebc00c730 tx=0x7f5ebc007c30).stop 2020-11-20T12:12:17.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 shutdown_connections 2020-11-20T12:12:17.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5ea4056d20 0x7f5ea40591e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ecc0b5f50 0x7f5ecc138e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ecc0b38c0 0x7f5ecc143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 --2- 172.21.15.187:0/1047548807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ecc1482d0 0x7f5ecc139340 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 >> 172.21.15.187:0/1047548807 conn(0x7f5ecc0bab30 msgr2=0x7f5ecc09cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:17.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 shutdown_connections 2020-11-20T12:12:17.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.216+0000 7f5ed1ccc700 1 -- 172.21.15.187:0/1047548807 wait complete. 2020-11-20T12:12:17.242 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:12:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.486+0000 7f199f499700 1 Processor -- start 2020-11-20T12:12:17.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.487+0000 7f199f499700 1 -- start start 2020-11-20T12:12:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.488+0000 7f199f499700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f19981484d0 0x7f1998061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.488+0000 7f199f499700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f19980622c0 0x7f19980b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.488+0000 7f199f499700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 0x7f19980b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.488+0000 7f199f499700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1998067730 con 0x7f19981484d0 2020-11-20T12:12:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.488+0000 7f199f499700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19980b1300 con 0x7f19980622c0 2020-11-20T12:12:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.488+0000 7f199f499700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19980b15a0 con 0x7f1998066f70 2020-11-20T12:12:17.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199dc96700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f19980622c0 0x7f19980b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199dc96700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f19980622c0 0x7f19980b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:46998/0 (socket says 172.21.15.187:46998) 2020-11-20T12:12:17.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199dc96700 1 -- 172.21.15.187:0/4121369711 learned_addr learned my addr 172.21.15.187:0/4121369711 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:17.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199e497700 1 --2- 172.21.15.187:0/4121369711 >> v2:172.21.15.5:3300/0 conn(0x7f19981484d0 0x7f1998061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199ec98700 1 --2- 172.21.15.187:0/4121369711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 0x7f19980b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199dc96700 1 -- 172.21.15.187:0/4121369711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 msgr2=0x7f19980b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199dc96700 1 --2- 172.21.15.187:0/4121369711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 0x7f19980b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199dc96700 1 -- 172.21.15.187:0/4121369711 >> v2:172.21.15.5:3300/0 conn(0x7f19981484d0 msgr2=0x7f1998061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199dc96700 1 --2- 172.21.15.187:0/4121369711 >> v2:172.21.15.5:3300/0 conn(0x7f19981484d0 0x7f1998061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.489+0000 7f199dc96700 1 -- 172.21.15.187:0/4121369711 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19980b1840 con 0x7f19980622c0 2020-11-20T12:12:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.490+0000 7f199dc96700 1 --2- 172.21.15.187:0/4121369711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19980622c0 0x7f19980b5640 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7f1998068600 tx=0x7f1988006eb0).ready entity=mon.1 client_cookie=d79c585c546c4de4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.490+0000 7f199d495700 1 -- 172.21.15.187:0/4121369711 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19880098d0 con 0x7f19980622c0 2020-11-20T12:12:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.490+0000 7f199d495700 1 -- 172.21.15.187:0/4121369711 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1988009a30 con 0x7f19980622c0 2020-11-20T12:12:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.490+0000 7f199d495700 1 -- 172.21.15.187:0/4121369711 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1988009db0 con 0x7f19980622c0 2020-11-20T12:12:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.490+0000 7f199f499700 1 -- 172.21.15.187:0/4121369711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19980622c0 msgr2=0x7f19980b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.490+0000 7f199f499700 1 --2- 172.21.15.187:0/4121369711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19980622c0 0x7f19980b5640 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7f1998068600 tx=0x7f1988006eb0).stop 2020-11-20T12:12:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.491+0000 7f199f499700 1 -- 172.21.15.187:0/4121369711 shutdown_connections 2020-11-20T12:12:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.491+0000 7f199f499700 1 --2- 172.21.15.187:0/4121369711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 0x7f19980b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.491+0000 7f199f499700 1 --2- 172.21.15.187:0/4121369711 >> v2:172.21.15.5:3300/0 conn(0x7f19981484d0 0x7f1998061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.491+0000 7f199f499700 1 --2- 172.21.15.187:0/4121369711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19980622c0 0x7f19980b5640 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.491+0000 7f199f499700 1 -- 172.21.15.187:0/4121369711 >> 172.21.15.187:0/4121369711 conn(0x7f19980bab30 msgr2=0x7f19980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.491+0000 7f199f499700 1 -- 172.21.15.187:0/4121369711 shutdown_connections 2020-11-20T12:12:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.491+0000 7f199f499700 1 -- 172.21.15.187:0/4121369711 wait complete. 2020-11-20T12:12:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.493+0000 7f199f499700 1 Processor -- start 2020-11-20T12:12:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.493+0000 7f199f499700 1 -- start start 2020-11-20T12:12:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.494+0000 7f199f499700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19980622c0 0x7f1998144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.494+0000 7f199f499700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 0x7f199809b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.494+0000 7f199f499700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19981484d0 0x7f199809ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.494+0000 7f199f499700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199809c1e0 con 0x7f19980622c0 2020-11-20T12:12:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.494+0000 7f199f499700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199809e910 con 0x7f19981484d0 2020-11-20T12:12:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.494+0000 7f199f499700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199809ebb0 con 0x7f1998066f70 2020-11-20T12:12:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19981484d0 0x7f199809ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19981484d0 0x7f199809ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47006/0 (socket says 172.21.15.187:47006) 2020-11-20T12:12:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 -- 172.21.15.187:0/2213897038 learned_addr learned my addr 172.21.15.187:0/2213897038 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199dc96700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 0x7f199809b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199e497700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19980622c0 0x7f1998144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 -- 172.21.15.187:0/2213897038 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 msgr2=0x7f199809b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 0x7f199809b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 -- 172.21.15.187:0/2213897038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19980622c0 msgr2=0x7f1998144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19980622c0 0x7f1998144260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 -- 172.21.15.187:0/2213897038 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199809cfa0 con 0x7f19981484d0 2020-11-20T12:12:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.495+0000 7f199ec98700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19981484d0 0x7f199809ae00 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7f199000eb70 tx=0x7f1990008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.496+0000 7f19867fc700 1 -- 172.21.15.187:0/2213897038 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199000c7a0 con 0x7f19981484d0 2020-11-20T12:12:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.496+0000 7f19867fc700 1 -- 172.21.15.187:0/2213897038 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199000c900 con 0x7f19981484d0 2020-11-20T12:12:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.496+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1998144750 con 0x7f19981484d0 2020-11-20T12:12:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.496+0000 7f19867fc700 1 -- 172.21.15.187:0/2213897038 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199000cca0 con 0x7f19981484d0 2020-11-20T12:12:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.496+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199809bf90 con 0x7f19981484d0 2020-11-20T12:12:17.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.497+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199809ed70 con 0x7f19981484d0 2020-11-20T12:12:17.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.498+0000 7f19867fc700 1 -- 172.21.15.187:0/2213897038 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1990013640 con 0x7f19981484d0 2020-11-20T12:12:17.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.499+0000 7f19867fc700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1970056d20 0x7f19700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.499+0000 7f19867fc700 1 -- 172.21.15.187:0/2213897038 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f199006fc90 con 0x7f19981484d0 2020-11-20T12:12:17.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.499+0000 7f199e497700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1970056d20 0x7f19700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.499+0000 7f199e497700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1970056d20 0x7f19700591e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f199400d990 tx=0x7f1994008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.504+0000 7f19867fc700 1 -- 172.21.15.187:0/2213897038 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f19900450e0 con 0x7f19981484d0 2020-11-20T12:12:17.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.692+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1998061a20 con 0x7f19981484d0 2020-11-20T12:12:17.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.692+0000 7f19867fc700 1 -- 172.21.15.187:0/2213897038 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1990048700 con 0x7f19981484d0 2020-11-20T12:12:17.697 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:12:17.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1970056d20 msgr2=0x7f19700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1970056d20 0x7f19700591e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f199400d990 tx=0x7f1994008040).stop 2020-11-20T12:12:17.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19981484d0 msgr2=0x7f199809ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19981484d0 0x7f199809ae00 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7f199000eb70 tx=0x7f1990008040).stop 2020-11-20T12:12:17.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 shutdown_connections 2020-11-20T12:12:17.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1970056d20 0x7f19700591e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19980622c0 0x7f1998144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19981484d0 0x7f199809ae00 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 --2- 172.21.15.187:0/2213897038 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1998066f70 0x7f199809b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 >> 172.21.15.187:0/2213897038 conn(0x7f19980bab30 msgr2=0x7f19980b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:17.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 shutdown_connections 2020-11-20T12:12:17.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.695+0000 7f199f499700 1 -- 172.21.15.187:0/2213897038 wait complete. 2020-11-20T12:12:17.723 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:12:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.968+0000 7f16871e0700 1 Processor -- start 2020-11-20T12:12:17.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.969+0000 7f16871e0700 1 -- start start 2020-11-20T12:12:17.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.970+0000 7f16871e0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1680065f90 0x7f1680065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.970+0000 7f16871e0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f168009ca90 0x7f168009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.970+0000 7f16871e0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16800b13e0 con 0x7f1680065190 2020-11-20T12:12:17.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.970+0000 7f16871e0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168014ab30 con 0x7f1680065f90 2020-11-20T12:12:17.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.970+0000 7f16871e0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1680062c10 con 0x7f168009ca90 2020-11-20T12:12:17.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.970+0000 7f16861de700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1680065190 0x7f1680143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43616/0 (socket says 172.21.15.187:43616) 2020-11-20T12:12:17.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.970+0000 7f16861de700 1 -- 172.21.15.187:0/2718627194 learned_addr learned my addr 172.21.15.187:0/2718627194 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:17.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.971+0000 7f16869df700 1 --2- 172.21.15.187:0/2718627194 >> v2:172.21.15.187:3300/0 conn(0x7f168009ca90 0x7f168009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.971+0000 7f16859dd700 1 --2- 172.21.15.187:0/2718627194 >> v2:172.21.15.5:3300/0 conn(0x7f1680065f90 0x7f1680065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.971+0000 7f16851dc700 1 -- 172.21.15.187:0/2718627194 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525860559 0 0) 0x7f16800b13e0 con 0x7f1680065190 2020-11-20T12:12:17.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.971+0000 7f16851dc700 1 -- 172.21.15.187:0/2718627194 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1668003240 con 0x7f1680065190 2020-11-20T12:12:17.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.971+0000 7f16869df700 1 -- 172.21.15.187:0/2718627194 >> v2:172.21.15.5:3300/0 conn(0x7f1680065f90 msgr2=0x7f1680065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:17.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.971+0000 7f16869df700 1 --2- 172.21.15.187:0/2718627194 >> v2:172.21.15.5:3300/0 conn(0x7f1680065f90 0x7f1680065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.971+0000 7f16869df700 1 -- 172.21.15.187:0/2718627194 >> v1:172.21.15.187:6789/0 conn(0x7f1680065190 legacy=0x7f1680143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.971+0000 7f16869df700 1 -- 172.21.15.187:0/2718627194 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800b1640 con 0x7f168009ca90 2020-11-20T12:12:17.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.972+0000 7f16869df700 1 --2- 172.21.15.187:0/2718627194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168009ca90 0x7f168009cf20 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7f1680099f90 tx=0x7f1678006df0).ready entity=mon.1 client_cookie=f64d9f18eadd9702 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.972+0000 7f16851dc700 1 -- 172.21.15.187:0/2718627194 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16780096d0 con 0x7f168009ca90 2020-11-20T12:12:17.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.972+0000 7f16851dc700 1 -- 172.21.15.187:0/2718627194 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1678009830 con 0x7f168009ca90 2020-11-20T12:12:17.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.972+0000 7f16851dc700 1 -- 172.21.15.187:0/2718627194 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678009bb0 con 0x7f168009ca90 2020-11-20T12:12:17.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.973+0000 7f16871e0700 1 -- 172.21.15.187:0/2718627194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168009ca90 msgr2=0x7f168009cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.973+0000 7f16871e0700 1 --2- 172.21.15.187:0/2718627194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168009ca90 0x7f168009cf20 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7f1680099f90 tx=0x7f1678006df0).stop 2020-11-20T12:12:17.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.973+0000 7f16871e0700 1 -- 172.21.15.187:0/2718627194 shutdown_connections 2020-11-20T12:12:17.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.973+0000 7f16871e0700 1 --2- 172.21.15.187:0/2718627194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168009ca90 0x7f168009cf20 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.973+0000 7f16871e0700 1 --2- 172.21.15.187:0/2718627194 >> v2:172.21.15.5:3300/0 conn(0x7f1680065f90 0x7f1680065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:17.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.973+0000 7f16871e0700 1 -- 172.21.15.187:0/2718627194 >> 172.21.15.187:0/2718627194 conn(0x7f16800bab30 msgr2=0x7f16800baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:17.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.973+0000 7f16871e0700 1 -- 172.21.15.187:0/2718627194 shutdown_connections 2020-11-20T12:12:17.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.973+0000 7f16871e0700 1 -- 172.21.15.187:0/2718627194 wait complete. 2020-11-20T12:12:17.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.975+0000 7f16871e0700 1 Processor -- start 2020-11-20T12:12:17.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.975+0000 7f16871e0700 1 -- start start 2020-11-20T12:12:17.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16871e0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1680065190 0x7f1680142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16871e0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1680065f90 0x7f16800b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16871e0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f168009ca90 0x7f16800b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16871e0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1680142320 con 0x7f168009ca90 2020-11-20T12:12:17.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16871e0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1680142550 con 0x7f1680065190 2020-11-20T12:12:17.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16871e0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16801427f0 con 0x7f1680065f90 2020-11-20T12:12:17.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16861de700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1680065190 0x7f1680142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16861de700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1680065190 0x7f1680142c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47016/0 (socket says 172.21.15.187:47016) 2020-11-20T12:12:17.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16861de700 1 -- 172.21.15.187:0/584275740 learned_addr learned my addr 172.21.15.187:0/584275740 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:17.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16861de700 1 -- 172.21.15.187:0/584275740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1680065f90 msgr2=0x7f16800b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16861de700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1680065f90 0x7f16800b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:17.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16861de700 1 -- 172.21.15.187:0/584275740 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f168009ca90 msgr2=0x7f16800b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:17.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16861de700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f168009ca90 0x7f16800b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:17.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.977+0000 7f16861de700 1 -- 172.21.15.187:0/584275740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680143180 con 0x7f1680065190 2020-11-20T12:12:17.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.978+0000 7f16861de700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1680065190 0x7f1680142c90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7f167c00ca60 tx=0x7f167c007e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.978+0000 7f166e7fc700 1 -- 172.21.15.187:0/584275740 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c005b80 con 0x7f1680065190 2020-11-20T12:12:17.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.978+0000 7f166e7fc700 1 -- 172.21.15.187:0/584275740 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c005ce0 con 0x7f1680065190 2020-11-20T12:12:17.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.978+0000 7f166e7fc700 1 -- 172.21.15.187:0/584275740 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c007340 con 0x7f1680065190 2020-11-20T12:12:17.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.978+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1680149950 con 0x7f1680065190 2020-11-20T12:12:17.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.978+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1680066990 con 0x7f1680065190 2020-11-20T12:12:17.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.980+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1680061a20 con 0x7f1680065190 2020-11-20T12:12:17.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.980+0000 7f166e7fc700 1 -- 172.21.15.187:0/584275740 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f167c010030 con 0x7f1680065190 2020-11-20T12:12:17.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.981+0000 7f166e7fc700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1658056d20 0x7f16580591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:17.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.981+0000 7f166e7fc700 1 -- 172.21.15.187:0/584275740 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f167c070080 con 0x7f1680065190 2020-11-20T12:12:17.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.981+0000 7f16859dd700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1658056d20 0x7f16580591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:17.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.982+0000 7f16859dd700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1658056d20 0x7f16580591e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f168015b4c0 tx=0x7f1670006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:17.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:17.984+0000 7f166e7fc700 1 -- 172.21.15.187:0/584275740 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f167c0454d0 con 0x7f1680065190 2020-11-20T12:12:18.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.170+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16800afc60 con 0x7f1680065190 2020-11-20T12:12:18.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.171+0000 7f166e7fc700 1 -- 172.21.15.187:0/584275740 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7f167c048af0 con 0x7f1680065190 2020-11-20T12:12:18.175 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:12:18.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1658056d20 msgr2=0x7f16580591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:18.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1658056d20 0x7f16580591e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f168015b4c0 tx=0x7f1670006df0).stop 2020-11-20T12:12:18.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1680065190 msgr2=0x7f1680142c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:18.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1680065190 0x7f1680142c90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7f167c00ca60 tx=0x7f167c007e40).stop 2020-11-20T12:12:18.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 shutdown_connections 2020-11-20T12:12:18.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1658056d20 0x7f16580591e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f168009ca90 0x7f16800b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:18.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1680065190 0x7f1680142c90 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 --2- 172.21.15.187:0/584275740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1680065f90 0x7f16800b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:18.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 >> 172.21.15.187:0/584275740 conn(0x7f16800bab30 msgr2=0x7f16800b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:18.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 shutdown_connections 2020-11-20T12:12:18.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.173+0000 7f16871e0700 1 -- 172.21.15.187:0/584275740 wait complete. 2020-11-20T12:12:18.200 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:12:18.201 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:18.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.445+0000 7f8750a29700 1 Processor -- start 2020-11-20T12:12:18.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.445+0000 7f8750a29700 1 -- start start 2020-11-20T12:12:18.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.446+0000 7f8750a29700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c0bf800 0x7f874c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:18.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.446+0000 7f8750a29700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f874c09f560 0x7f874c0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:18.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.446+0000 7f8750a29700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874c14d110 con 0x7f874c0bfbe0 2020-11-20T12:12:18.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.446+0000 7f8750a29700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874c14cbd0 con 0x7f874c09f560 2020-11-20T12:12:18.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.446+0000 7f8750a29700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874c14ce70 con 0x7f874c0bf800 2020-11-20T12:12:18.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.447+0000 7f874ad9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c0bf800 0x7f874c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:18.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.447+0000 7f874b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f874c09f560 0x7f874c0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:18.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.448+0000 7f874ad9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c0bf800 0x7f874c14c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:47912/0 (socket says 172.21.15.187:47912) 2020-11-20T12:12:18.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.448+0000 7f874b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f874c09f560 0x7f874c0a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:39292/0 (socket says 172.21.15.187:39292) 2020-11-20T12:12:18.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.448+0000 7f874ad9d700 1 -- 172.21.15.187:0/366937615 learned_addr learned my addr 172.21.15.187:0/366937615 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:18.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.449+0000 7f874ad9d700 1 -- 172.21.15.187:0/366937615 >> v2:172.21.15.5:3300/0 conn(0x7f874c09f560 msgr2=0x7f874c0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:18.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.449+0000 7f874ad9d700 1 --2- 172.21.15.187:0/366937615 >> v2:172.21.15.5:3300/0 conn(0x7f874c09f560 0x7f874c0a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.449+0000 7f874ad9d700 1 -- 172.21.15.187:0/366937615 >> v1:172.21.15.5:6789/0 conn(0x7f874c0bfbe0 legacy=0x7f874c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:18.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.449+0000 7f874ad9d700 1 -- 172.21.15.187:0/366937615 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c0a4cc0 con 0x7f874c0bf800 2020-11-20T12:12:18.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.449+0000 7f874b59e700 1 --2- 172.21.15.187:0/366937615 >> v2:172.21.15.5:3300/0 conn(0x7f874c09f560 0x7f874c0a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:12:18.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.449+0000 7f874ad9d700 1 --2- 172.21.15.187:0/366937615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c0bf800 0x7f874c14c2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f873c009d50 tx=0x7f873c009290).ready entity=mon.2 client_cookie=ff1b307c27af914d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:18.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.449+0000 7f874a59c700 1 -- 172.21.15.187:0/366937615 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c00bc50 con 0x7f874c0bf800 2020-11-20T12:12:18.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.449+0000 7f874a59c700 1 -- 172.21.15.187:0/366937615 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c00bdb0 con 0x7f874c0bf800 2020-11-20T12:12:18.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f874a59c700 1 -- 172.21.15.187:0/366937615 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c011610 con 0x7f874c0bf800 2020-11-20T12:12:18.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f8750a29700 1 -- 172.21.15.187:0/366937615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c0bf800 msgr2=0x7f874c14c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:18.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f8750a29700 1 --2- 172.21.15.187:0/366937615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c0bf800 0x7f874c14c2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f873c009d50 tx=0x7f873c009290).stop 2020-11-20T12:12:18.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f8750a29700 1 -- 172.21.15.187:0/366937615 shutdown_connections 2020-11-20T12:12:18.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f8750a29700 1 --2- 172.21.15.187:0/366937615 >> v2:172.21.15.5:3300/0 conn(0x7f874c09f560 0x7f874c0a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f8750a29700 1 --2- 172.21.15.187:0/366937615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c0bf800 0x7f874c14c2c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f8750a29700 1 -- 172.21.15.187:0/366937615 >> 172.21.15.187:0/366937615 conn(0x7f874c068370 msgr2=0x7f874c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:18.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f8750a29700 1 -- 172.21.15.187:0/366937615 shutdown_connections 2020-11-20T12:12:18.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.450+0000 7f8750a29700 1 -- 172.21.15.187:0/366937615 wait complete. 2020-11-20T12:12:18.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.452+0000 7f8750a29700 1 Processor -- start 2020-11-20T12:12:18.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.452+0000 7f8750a29700 1 -- start start 2020-11-20T12:12:18.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.453+0000 7f8750a29700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f874c09f560 0x7f874c0b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:18.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.453+0000 7f8750a29700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f874c0bf800 0x7f874c0ba700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:18.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.453+0000 7f8750a29700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c15c170 0x7f874c146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:18.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.453+0000 7f8750a29700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0b72e0 con 0x7f874c09f560 2020-11-20T12:12:18.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.453+0000 7f8750a29700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c15ca10 con 0x7f874c0bf800 2020-11-20T12:12:18.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.453+0000 7f8750a29700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874c15ccb0 con 0x7f874c15c170 2020-11-20T12:12:18.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f874c0bf800 0x7f874c0ba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:18.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f874c0bf800 0x7f874c0ba700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47032/0 (socket says 172.21.15.187:47032) 2020-11-20T12:12:18.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874259c700 1 -- 172.21.15.187:0/2210889066 learned_addr learned my addr 172.21.15.187:0/2210889066 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:18.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874ad9d700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f874c09f560 0x7f874c0b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:18.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874b59e700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c15c170 0x7f874c146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:18.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874259c700 1 -- 172.21.15.187:0/2210889066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c15c170 msgr2=0x7f874c146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:18.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874259c700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c15c170 0x7f874c146c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874259c700 1 -- 172.21.15.187:0/2210889066 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f874c09f560 msgr2=0x7f874c0b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:18.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874259c700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f874c09f560 0x7f874c0b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.454+0000 7f874259c700 1 -- 172.21.15.187:0/2210889066 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c15cf50 con 0x7f874c0bf800 2020-11-20T12:12:18.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.455+0000 7f874259c700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f874c0bf800 0x7f874c0ba700 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f874c0a1bc0 tx=0x7f8734007b70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:18.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.455+0000 7f87437fe700 1 -- 172.21.15.187:0/2210889066 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873400e490 con 0x7f874c0bf800 2020-11-20T12:12:18.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.455+0000 7f87437fe700 1 -- 172.21.15.187:0/2210889066 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873400e5f0 con 0x7f874c0bf800 2020-11-20T12:12:18.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.455+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874c0babf0 con 0x7f874c0bf800 2020-11-20T12:12:18.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.455+0000 7f87437fe700 1 -- 172.21.15.187:0/2210889066 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873400e970 con 0x7f874c0bf800 2020-11-20T12:12:18.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.455+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874c15c5d0 con 0x7f874c0bf800 2020-11-20T12:12:18.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.456+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874c080050 con 0x7f874c0bf800 2020-11-20T12:12:18.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.457+0000 7f87437fe700 1 -- 172.21.15.187:0/2210889066 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8734010030 con 0x7f874c0bf800 2020-11-20T12:12:18.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.457+0000 7f87437fe700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8728056d20 0x7f87280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:18.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.457+0000 7f87437fe700 1 -- 172.21.15.187:0/2210889066 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f873406fe30 con 0x7f874c0bf800 2020-11-20T12:12:18.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.458+0000 7f874ad9d700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8728056d20 0x7f87280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:18.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.458+0000 7f874ad9d700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8728056d20 0x7f87280591e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f874c145910 tx=0x7f873c016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:18.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.461+0000 7f87437fe700 1 -- 172.21.15.187:0/2210889066 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8734045170 con 0x7f874c0bf800 2020-11-20T12:12:18.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.706+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f874c14c7b0 con 0x7f874c0bf800 2020-11-20T12:12:18.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.707+0000 7f87437fe700 1 -- 172.21.15.187:0/2210889066 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f8734048790 con 0x7f874c0bf800 2020-11-20T12:12:18.711 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:18.711 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5159,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:12.329807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6952},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:18.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8728056d20 msgr2=0x7f87280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:18.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8728056d20 0x7f87280591e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f874c145910 tx=0x7f873c016000).stop 2020-11-20T12:12:18.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f874c0bf800 msgr2=0x7f874c0ba700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:18.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f874c0bf800 0x7f874c0ba700 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f874c0a1bc0 tx=0x7f8734007b70).stop 2020-11-20T12:12:18.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 shutdown_connections 2020-11-20T12:12:18.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8728056d20 0x7f87280591e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f874c09f560 0x7f874c0b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f874c15c170 0x7f874c146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 --2- 172.21.15.187:0/2210889066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f874c0bf800 0x7f874c0ba700 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 >> 172.21.15.187:0/2210889066 conn(0x7f874c068370 msgr2=0x7f874c0a4010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 shutdown_connections 2020-11-20T12:12:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:18.710+0000 7f8750a29700 1 -- 172.21.15.187:0/2210889066 wait complete. 2020-11-20T12:12:18.717 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 64 2020-11-20T12:12:18.940 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:12.329807+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6952}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:12:18.941 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:12:18.941 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:12:19.223 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6952,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":8.8784884319999993,"mdsmap_epoch":64,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":15.424062572} 2020-11-20T12:12:19.224 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6952, "mdsmap_epoch": 64, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 8.878488432, "state": "up:active", "uptime": 15.424062572, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:12:19.224 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:12:19.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:19.225 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:12:19.278 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:12:19.279 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:12:19.422 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:12:19.422 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:12:19.464 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:12:19.465 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:12:19.475 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:12:19.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:19.476 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:12:19.582 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2020-11-20T12:12:19.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.821+0000 7fe423fff700 1 Processor -- start 2020-11-20T12:12:19.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.821+0000 7fe423fff700 1 -- start start 2020-11-20T12:12:19.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.822+0000 7fe423fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065190 0x7fe424065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:19.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.823+0000 7fe423fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe42409ca90 0x7fe42409cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:19.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.823+0000 7fe423fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4240b13e0 con 0x7fe424065f90 2020-11-20T12:12:19.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.823+0000 7fe423fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42414ab30 con 0x7fe42409ca90 2020-11-20T12:12:19.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.823+0000 7fe423fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe424062c10 con 0x7fe424065190 2020-11-20T12:12:19.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.824+0000 7fe422ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065190 0x7fe424065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:19.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.824+0000 7fe41a7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe424065f90 0x7fe424143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43652/0 (socket says 172.21.15.187:43652) 2020-11-20T12:12:19.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.824+0000 7fe422ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065190 0x7fe424065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:47930/0 (socket says 172.21.15.187:47930) 2020-11-20T12:12:19.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.824+0000 7fe41a7fc700 1 -- 172.21.15.187:0/1634974543 learned_addr learned my addr 172.21.15.187:0/1634974543 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:19.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.824+0000 7fe4237fe700 1 --2- 172.21.15.187:0/1634974543 >> v2:172.21.15.5:3300/0 conn(0x7fe42409ca90 0x7fe42409cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:19.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.825+0000 7fe4227fc700 1 -- 172.21.15.187:0/1634974543 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441175777 0 0) 0x7fe4240b13e0 con 0x7fe424065f90 2020-11-20T12:12:19.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.825+0000 7fe422ffd700 1 -- 172.21.15.187:0/1634974543 >> v2:172.21.15.5:3300/0 conn(0x7fe42409ca90 msgr2=0x7fe42409cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:19.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.825+0000 7fe422ffd700 1 --2- 172.21.15.187:0/1634974543 >> v2:172.21.15.5:3300/0 conn(0x7fe42409ca90 0x7fe42409cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:19.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.825+0000 7fe422ffd700 1 -- 172.21.15.187:0/1634974543 >> v1:172.21.15.187:6789/0 conn(0x7fe424065f90 legacy=0x7fe424143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:19.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.825+0000 7fe422ffd700 1 -- 172.21.15.187:0/1634974543 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4240b1640 con 0x7fe424065190 2020-11-20T12:12:19.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.825+0000 7fe422ffd700 1 --2- 172.21.15.187:0/1634974543 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065190 0x7fe424065570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fe42414ac70 tx=0x7fe414006df0).ready entity=mon.2 client_cookie=cddf80d1e18a165e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:19.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe4227fc700 1 -- 172.21.15.187:0/1634974543 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414009840 con 0x7fe424065190 2020-11-20T12:12:19.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe4227fc700 1 -- 172.21.15.187:0/1634974543 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4140099a0 con 0x7fe424065190 2020-11-20T12:12:19.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe4227fc700 1 -- 172.21.15.187:0/1634974543 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414009d20 con 0x7fe424065190 2020-11-20T12:12:19.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe423fff700 1 -- 172.21.15.187:0/1634974543 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065190 msgr2=0x7fe424065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:19.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe423fff700 1 --2- 172.21.15.187:0/1634974543 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065190 0x7fe424065570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fe42414ac70 tx=0x7fe414006df0).stop 2020-11-20T12:12:19.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe423fff700 1 -- 172.21.15.187:0/1634974543 shutdown_connections 2020-11-20T12:12:19.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe423fff700 1 --2- 172.21.15.187:0/1634974543 >> v2:172.21.15.5:3300/0 conn(0x7fe42409ca90 0x7fe42409cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:19.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe423fff700 1 --2- 172.21.15.187:0/1634974543 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065190 0x7fe424065570 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:19.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe423fff700 1 -- 172.21.15.187:0/1634974543 >> 172.21.15.187:0/1634974543 conn(0x7fe4240bab30 msgr2=0x7fe4240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:19.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe423fff700 1 -- 172.21.15.187:0/1634974543 shutdown_connections 2020-11-20T12:12:19.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.826+0000 7fe423fff700 1 -- 172.21.15.187:0/1634974543 wait complete. 2020-11-20T12:12:19.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.828+0000 7fe423fff700 1 Processor -- start 2020-11-20T12:12:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.828+0000 7fe423fff700 1 -- start start 2020-11-20T12:12:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe423fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe424065190 0x7fe4240b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe423fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065f90 0x7fe42415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe423fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe42409ca90 0x7fe42409fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe423fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240b4950 con 0x7fe424065190 2020-11-20T12:12:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe423fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe424099420 con 0x7fe42409ca90 2020-11-20T12:12:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe423fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240996c0 con 0x7fe424065f90 2020-11-20T12:12:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe4237fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe42409ca90 0x7fe42409fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe4237fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe42409ca90 0x7fe42409fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47050/0 (socket says 172.21.15.187:47050) 2020-11-20T12:12:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.829+0000 7fe4237fe700 1 -- 172.21.15.187:0/3411464368 learned_addr learned my addr 172.21.15.187:0/3411464368 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.830+0000 7fe422ffd700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe424065190 0x7fe4240b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.830+0000 7fe41a7fc700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065f90 0x7fe42415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.830+0000 7fe4237fe700 1 -- 172.21.15.187:0/3411464368 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065f90 msgr2=0x7fe42415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.830+0000 7fe4237fe700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065f90 0x7fe42415baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.830+0000 7fe4237fe700 1 -- 172.21.15.187:0/3411464368 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe424065190 msgr2=0x7fe4240b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.830+0000 7fe4237fe700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe424065190 0x7fe4240b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.830+0000 7fe4237fe700 1 -- 172.21.15.187:0/3411464368 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe424099960 con 0x7fe42409ca90 2020-11-20T12:12:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.830+0000 7fe4237fe700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe42409ca90 0x7fe42409fef0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7fe41c00ed20 tx=0x7fe41c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.831+0000 7fe41b7fe700 1 -- 172.21.15.187:0/3411464368 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c00cbb0 con 0x7fe42409ca90 2020-11-20T12:12:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.831+0000 7fe41b7fe700 1 -- 172.21.15.187:0/3411464368 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c00cd10 con 0x7fe42409ca90 2020-11-20T12:12:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.831+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42415bf90 con 0x7fe42409ca90 2020-11-20T12:12:19.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.831+0000 7fe41b7fe700 1 -- 172.21.15.187:0/3411464368 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c012620 con 0x7fe42409ca90 2020-11-20T12:12:19.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.831+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42415c490 con 0x7fe42409ca90 2020-11-20T12:12:19.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.832+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe424061a20 con 0x7fe42409ca90 2020-11-20T12:12:19.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.832+0000 7fe41b7fe700 1 -- 172.21.15.187:0/3411464368 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe41c022070 con 0x7fe42409ca90 2020-11-20T12:12:19.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.833+0000 7fe41b7fe700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe400056d20 0x7fe4000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:19.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.833+0000 7fe41b7fe700 1 -- 172.21.15.187:0/3411464368 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe41c070810 con 0x7fe42409ca90 2020-11-20T12:12:19.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.833+0000 7fe422ffd700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe400056d20 0x7fe4000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:19.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.834+0000 7fe422ffd700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe400056d20 0x7fe4000591e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fe414006e30 tx=0x7fe414015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:19.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:19.837+0000 7fe41b7fe700 1 -- 172.21.15.187:0/3411464368 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe41c045b50 con 0x7fe42409ca90 2020-11-20T12:12:20.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.085+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fe4240afc60 con 0x7fe42409ca90 2020-11-20T12:12:20.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.179+0000 7fe41b7fe700 1 -- 172.21.15.187:0/3411464368 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v65) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe41c016070 con 0x7fe42409ca90 2020-11-20T12:12:20.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.183+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe400056d20 msgr2=0x7fe4000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:20.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.183+0000 7fe423fff700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe400056d20 0x7fe4000591e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fe414006e30 tx=0x7fe414015000).stop 2020-11-20T12:12:20.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.183+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe42409ca90 msgr2=0x7fe42409fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:20.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.183+0000 7fe423fff700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe42409ca90 0x7fe42409fef0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7fe41c00ed20 tx=0x7fe41c0091d0).stop 2020-11-20T12:12:20.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.184+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 shutdown_connections 2020-11-20T12:12:20.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.184+0000 7fe423fff700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe400056d20 0x7fe4000591e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.184+0000 7fe423fff700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe424065190 0x7fe4240b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.184+0000 7fe423fff700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe42409ca90 0x7fe42409fef0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.184+0000 7fe423fff700 1 --2- 172.21.15.187:0/3411464368 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe424065f90 0x7fe42415baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.184+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 >> 172.21.15.187:0/3411464368 conn(0x7fe4240bab30 msgr2=0x7fe4240ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:20.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.184+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 shutdown_connections 2020-11-20T12:12:20.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.184+0000 7fe423fff700 1 -- 172.21.15.187:0/3411464368 wait complete. 2020-11-20T12:12:20.221 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:20.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.462+0000 7fe68c805700 1 Processor -- start 2020-11-20T12:12:20.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.463+0000 7fe68c805700 1 -- start start 2020-11-20T12:12:20.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68c805700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe6880654a0 0x7fe6880b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:20.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68c805700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe688065880 0x7fe688144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:20.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68c805700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6880b6c10 0x7fe688145350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:20.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68c805700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6880b08e0 con 0x7fe688065880 2020-11-20T12:12:20.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68c805700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6880b0b80 con 0x7fe6880654a0 2020-11-20T12:12:20.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68c805700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6880b05f0 con 0x7fe6880b6c10 2020-11-20T12:12:20.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe6880654a0 0x7fe6880b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:20.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe6880654a0 0x7fe6880b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47056/0 (socket says 172.21.15.187:47056) 2020-11-20T12:12:20.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.464+0000 7fe68659c700 1 -- 172.21.15.187:0/3909288656 learned_addr learned my addr 172.21.15.187:0/3909288656 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:20.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe686d9d700 1 --2- 172.21.15.187:0/3909288656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6880b6c10 0x7fe688145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:20.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe685d9b700 1 --2- 172.21.15.187:0/3909288656 >> v2:172.21.15.5:3300/0 conn(0x7fe688065880 0x7fe688144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:20.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe68659c700 1 -- 172.21.15.187:0/3909288656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6880b6c10 msgr2=0x7fe688145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:20.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe68659c700 1 --2- 172.21.15.187:0/3909288656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6880b6c10 0x7fe688145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe68659c700 1 -- 172.21.15.187:0/3909288656 >> v2:172.21.15.5:3300/0 conn(0x7fe688065880 msgr2=0x7fe688144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:20.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe68659c700 1 --2- 172.21.15.187:0/3909288656 >> v2:172.21.15.5:3300/0 conn(0x7fe688065880 0x7fe688144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe68659c700 1 -- 172.21.15.187:0/3909288656 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6880b5880 con 0x7fe6880654a0 2020-11-20T12:12:20.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe685d9b700 1 --2- 172.21.15.187:0/3909288656 >> v2:172.21.15.5:3300/0 conn(0x7fe688065880 0x7fe688144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:20.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.465+0000 7fe68659c700 1 --2- 172.21.15.187:0/3909288656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880654a0 0x7fe6880b1650 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7fe6880b8ab0 tx=0x7fe678006eb0).ready entity=mon.1 client_cookie=3d06ec802f06e08b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:20.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68559a700 1 -- 172.21.15.187:0/3909288656 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678009960 con 0x7fe6880654a0 2020-11-20T12:12:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68559a700 1 -- 172.21.15.187:0/3909288656 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678009ac0 con 0x7fe6880654a0 2020-11-20T12:12:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68559a700 1 -- 172.21.15.187:0/3909288656 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678009e40 con 0x7fe6880654a0 2020-11-20T12:12:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 -- 172.21.15.187:0/3909288656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880654a0 msgr2=0x7fe6880b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 --2- 172.21.15.187:0/3909288656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880654a0 0x7fe6880b1650 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7fe6880b8ab0 tx=0x7fe678006eb0).stop 2020-11-20T12:12:20.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 -- 172.21.15.187:0/3909288656 shutdown_connections 2020-11-20T12:12:20.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 --2- 172.21.15.187:0/3909288656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe6880b6c10 0x7fe688145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 --2- 172.21.15.187:0/3909288656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880654a0 0x7fe6880b1650 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 --2- 172.21.15.187:0/3909288656 >> v2:172.21.15.5:3300/0 conn(0x7fe688065880 0x7fe688144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 -- 172.21.15.187:0/3909288656 >> 172.21.15.187:0/3909288656 conn(0x7fe6880bacd0 msgr2=0x7fe6880bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:20.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 -- 172.21.15.187:0/3909288656 shutdown_connections 2020-11-20T12:12:20.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.466+0000 7fe68c805700 1 -- 172.21.15.187:0/3909288656 wait complete. 2020-11-20T12:12:20.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.468+0000 7fe68c805700 1 Processor -- start 2020-11-20T12:12:20.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.469+0000 7fe68c805700 1 -- start start 2020-11-20T12:12:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe68c805700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6880654a0 0x7fe6880b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe68c805700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe688065880 0x7fe68809f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe68c805700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880b6c10 0x7fe6880b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:20.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe68c805700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6880b4c40 con 0x7fe6880654a0 2020-11-20T12:12:20.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe68c805700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68813aa50 con 0x7fe6880b6c10 2020-11-20T12:12:20.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe68c805700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe688156d90 con 0x7fe688065880 2020-11-20T12:12:20.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe686d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880b6c10 0x7fe6880b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:20.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe686d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880b6c10 0x7fe6880b25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47064/0 (socket says 172.21.15.187:47064) 2020-11-20T12:12:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe686d9d700 1 -- 172.21.15.187:0/2584394458 learned_addr learned my addr 172.21.15.187:0/2584394458 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.470+0000 7fe68659c700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6880654a0 0x7fe6880b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe685d9b700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe688065880 0x7fe68809f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:20.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe686d9d700 1 -- 172.21.15.187:0/2584394458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe688065880 msgr2=0x7fe68809f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:20.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe686d9d700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe688065880 0x7fe68809f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe686d9d700 1 -- 172.21.15.187:0/2584394458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6880654a0 msgr2=0x7fe6880b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:20.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe686d9d700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6880654a0 0x7fe6880b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe686d9d700 1 -- 172.21.15.187:0/2584394458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe688157000 con 0x7fe6880b6c10 2020-11-20T12:12:20.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe686d9d700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880b6c10 0x7fe6880b25c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7fe68000cbb0 tx=0x7fe680008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:20.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe6767fc700 1 -- 172.21.15.187:0/2584394458 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680007950 con 0x7fe6880b6c10 2020-11-20T12:12:20.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe6767fc700 1 -- 172.21.15.187:0/2584394458 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe680007ab0 con 0x7fe6880b6c10 2020-11-20T12:12:20.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe6767fc700 1 -- 172.21.15.187:0/2584394458 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680007e50 con 0x7fe6880b6c10 2020-11-20T12:12:20.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.471+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe688157540 con 0x7fe6880b6c10 2020-11-20T12:12:20.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.472+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68809f900 con 0x7fe6880b6c10 2020-11-20T12:12:20.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.473+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe688157390 con 0x7fe6880b6c10 2020-11-20T12:12:20.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.473+0000 7fe6767fc700 1 -- 172.21.15.187:0/2584394458 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe680011620 con 0x7fe6880b6c10 2020-11-20T12:12:20.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.474+0000 7fe6767fc700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe65c056d50 0x7fe65c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:20.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.474+0000 7fe6767fc700 1 -- 172.21.15.187:0/2584394458 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe6800703f0 con 0x7fe6880b6c10 2020-11-20T12:12:20.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.474+0000 7fe68659c700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe65c056d50 0x7fe65c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:20.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.475+0000 7fe68659c700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe65c056d50 0x7fe65c059210 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7fe678006ef0 tx=0x7fe678015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:20.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.478+0000 7fe6767fc700 1 -- 172.21.15.187:0/2584394458 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe6800457c0 con 0x7fe6880b6c10 2020-11-20T12:12:20.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.726+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6881576d0 con 0x7fe6880b6c10 2020-11-20T12:12:20.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.727+0000 7fe6767fc700 1 -- 172.21.15.187:0/2584394458 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2153 (secure 0 0 0) 0x7fe680048de0 con 0x7fe6880b6c10 2020-11-20T12:12:20.731 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:20.731 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:20.179609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:creating","state_seq":2,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:20.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe65c056d50 msgr2=0x7fe65c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:20.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe65c056d50 0x7fe65c059210 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7fe678006ef0 tx=0x7fe678015000).stop 2020-11-20T12:12:20.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880b6c10 msgr2=0x7fe6880b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:20.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880b6c10 0x7fe6880b25c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7fe68000cbb0 tx=0x7fe680008040).stop 2020-11-20T12:12:20.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 shutdown_connections 2020-11-20T12:12:20.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe65c056d50 0x7fe65c059210 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6880654a0 0x7fe6880b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe6880b6c10 0x7fe6880b25c0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 --2- 172.21.15.187:0/2584394458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe688065880 0x7fe68809f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:20.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.729+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 >> 172.21.15.187:0/2584394458 conn(0x7fe6880bacd0 msgr2=0x7fe688099ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:20.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.730+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 shutdown_connections 2020-11-20T12:12:20.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:20.730+0000 7fe68c805700 1 -- 172.21.15.187:0/2584394458 wait complete. 2020-11-20T12:12:20.736 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 66 2020-11-20T12:12:20.757 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2020-11-20T12:12:20.837 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:12:20.846 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:12:25.759 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:26.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.007+0000 7fd36d9b6700 1 Processor -- start 2020-11-20T12:12:26.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.007+0000 7fd36d9b6700 1 -- start start 2020-11-20T12:12:26.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.009+0000 7fd36d9b6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3680652e0 0x7fd3680656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.009+0000 7fd36d9b6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd368149dd0 con 0x7fd3680b1590 2020-11-20T12:12:26.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.009+0000 7fd36d9b6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36814a070 con 0x7fd3680b7300 2020-11-20T12:12:26.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.009+0000 7fd36d9b6700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36814a310 con 0x7fd3680652e0 2020-11-20T12:12:26.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.010+0000 7fd35e7fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd3680b1590 0x7fd36814a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:37188/0 (socket says 172.21.15.187:37188) 2020-11-20T12:12:26.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.010+0000 7fd35e7fc700 1 -- 172.21.15.187:0/1399310832 learned_addr learned my addr 172.21.15.187:0/1399310832 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.010+0000 7fd366ffd700 1 --2- 172.21.15.187:0/1399310832 >> v2:172.21.15.187:3300/0 conn(0x7fd3680652e0 0x7fd3680656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.010+0000 7fd3677fe700 1 --1- 172.21.15.187:0/1399310832 >> v1:172.21.15.187:6789/0 conn(0x7fd3680b7300 0x7fd368143920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43682/0 (socket says 172.21.15.187:43682) 2020-11-20T12:12:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.010+0000 7fd366ffd700 1 -- 172.21.15.187:0/1399310832 >> v1:172.21.15.187:6789/0 conn(0x7fd3680b7300 legacy=0x7fd368143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.010+0000 7fd366ffd700 1 -- 172.21.15.187:0/1399310832 >> v1:172.21.15.5:6789/0 conn(0x7fd3680b1590 legacy=0x7fd36814a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.010+0000 7fd366ffd700 1 -- 172.21.15.187:0/1399310832 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3680b5380 con 0x7fd3680652e0 2020-11-20T12:12:26.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.011+0000 7fd366ffd700 1 --2- 172.21.15.187:0/1399310832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680652e0 0x7fd3680656c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7fd35800b9a0 tx=0x7fd358006df0).ready entity=mon.1 client_cookie=886e407fdae4cb6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.011+0000 7fd3667fc700 1 -- 172.21.15.187:0/1399310832 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3580097b0 con 0x7fd3680652e0 2020-11-20T12:12:26.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.011+0000 7fd3667fc700 1 -- 172.21.15.187:0/1399310832 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd358009910 con 0x7fd3680652e0 2020-11-20T12:12:26.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.011+0000 7fd3667fc700 1 -- 172.21.15.187:0/1399310832 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358009c90 con 0x7fd3680652e0 2020-11-20T12:12:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.011+0000 7fd36d9b6700 1 -- 172.21.15.187:0/1399310832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680652e0 msgr2=0x7fd3680656c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.011+0000 7fd36d9b6700 1 --2- 172.21.15.187:0/1399310832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680652e0 0x7fd3680656c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7fd35800b9a0 tx=0x7fd358006df0).stop 2020-11-20T12:12:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.012+0000 7fd36d9b6700 1 -- 172.21.15.187:0/1399310832 shutdown_connections 2020-11-20T12:12:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.012+0000 7fd36d9b6700 1 --2- 172.21.15.187:0/1399310832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680652e0 0x7fd3680656c0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.012+0000 7fd36d9b6700 1 -- 172.21.15.187:0/1399310832 >> 172.21.15.187:0/1399310832 conn(0x7fd3680bacd0 msgr2=0x7fd3680bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.012+0000 7fd36d9b6700 1 -- 172.21.15.187:0/1399310832 shutdown_connections 2020-11-20T12:12:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.012+0000 7fd36d9b6700 1 -- 172.21.15.187:0/1399310832 wait complete. 2020-11-20T12:12:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.014+0000 7fd36d9b6700 1 Processor -- start 2020-11-20T12:12:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.014+0000 7fd36d9b6700 1 -- start start 2020-11-20T12:12:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd36d9b6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3680652e0 0x7fd368146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd36d9b6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3680b1590 0x7fd3680b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd36d9b6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680b7300 0x7fd368156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd36d9b6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368142ec0 con 0x7fd3680b1590 2020-11-20T12:12:26.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd36d9b6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368149a00 con 0x7fd3680b7300 2020-11-20T12:12:26.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd36d9b6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd368156590 con 0x7fd3680652e0 2020-11-20T12:12:26.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd3677fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680b7300 0x7fd368156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd3677fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680b7300 0x7fd368156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47082/0 (socket says 172.21.15.187:47082) 2020-11-20T12:12:26.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.015+0000 7fd3677fe700 1 -- 172.21.15.187:0/2252625773 learned_addr learned my addr 172.21.15.187:0/2252625773 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:26.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd3677fe700 1 -- 172.21.15.187:0/2252625773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3680652e0 msgr2=0x7fd368146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd3677fe700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3680652e0 0x7fd368146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:26.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd3677fe700 1 -- 172.21.15.187:0/2252625773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3680b1590 msgr2=0x7fd3680b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd3677fe700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3680b1590 0x7fd3680b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:26.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd3677fe700 1 -- 172.21.15.187:0/2252625773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368149730 con 0x7fd3680b7300 2020-11-20T12:12:26.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd3677fe700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680b7300 0x7fd368156a30 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7fd36000ca60 tx=0x7fd360007f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd35f7fe700 1 -- 172.21.15.187:0/2252625773 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360005ac0 con 0x7fd3680b7300 2020-11-20T12:12:26.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd35f7fe700 1 -- 172.21.15.187:0/2252625773 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360005c20 con 0x7fd3680b7300 2020-11-20T12:12:26.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd35f7fe700 1 -- 172.21.15.187:0/2252625773 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360007110 con 0x7fd3680b7300 2020-11-20T12:12:26.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd368156290 con 0x7fd3680b7300 2020-11-20T12:12:26.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.016+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd368147660 con 0x7fd3680b7300 2020-11-20T12:12:26.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.018+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3680b59b0 con 0x7fd3680b7300 2020-11-20T12:12:26.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.018+0000 7fd35f7fe700 1 -- 172.21.15.187:0/2252625773 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd360010030 con 0x7fd3680b7300 2020-11-20T12:12:26.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.019+0000 7fd35f7fe700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd338056d20 0x7fd3380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.019+0000 7fd35f7fe700 1 -- 172.21.15.187:0/2252625773 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd360070040 con 0x7fd3680b7300 2020-11-20T12:12:26.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.019+0000 7fd366ffd700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd338056d20 0x7fd3380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.020+0000 7fd366ffd700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd338056d20 0x7fd3380591e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fd358006e30 tx=0x7fd358015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.023+0000 7fd35f7fe700 1 -- 172.21.15.187:0/2252625773 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd360045490 con 0x7fd3680b7300 2020-11-20T12:12:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.267+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd368156750 con 0x7fd3680b7300 2020-11-20T12:12:26.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.268+0000 7fd35f7fe700 1 -- 172.21.15.187:0/2252625773 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2151 (secure 0 0 0) 0x7fd360048ab0 con 0x7fd3680b7300 2020-11-20T12:12:26.272 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:26.273 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:21.182870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:26.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd338056d20 msgr2=0x7fd3380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd338056d20 0x7fd3380591e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fd358006e30 tx=0x7fd358015000).stop 2020-11-20T12:12:26.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680b7300 msgr2=0x7fd368156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680b7300 0x7fd368156a30 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7fd36000ca60 tx=0x7fd360007f40).stop 2020-11-20T12:12:26.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 shutdown_connections 2020-11-20T12:12:26.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd338056d20 0x7fd3380591e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3680b7300 0x7fd368156a30 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3680652e0 0x7fd368146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:26.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 --2- 172.21.15.187:0/2252625773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3680b1590 0x7fd3680b8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:26.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 >> 172.21.15.187:0/2252625773 conn(0x7fd3680bacd0 msgr2=0x7fd3680b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:26.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 shutdown_connections 2020-11-20T12:12:26.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.271+0000 7fd36d9b6700 1 -- 172.21.15.187:0/2252625773 wait complete. 2020-11-20T12:12:26.278 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 67 2020-11-20T12:12:26.299 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:26.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.538+0000 7fadbf1bc700 1 Processor -- start 2020-11-20T12:12:26.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.538+0000 7fadbf1bc700 1 -- start start 2020-11-20T12:12:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.539+0000 7fadbf1bc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fadb81484d0 0x7fadb8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.539+0000 7fadbf1bc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fadb80622c0 0x7fadb80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.539+0000 7fadbf1bc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8066f70 0x7fadb80b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.539+0000 7fadbf1bc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb8067730 con 0x7fadb80622c0 2020-11-20T12:12:26.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.539+0000 7fadbf1bc700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb80b1300 con 0x7fadb81484d0 2020-11-20T12:12:26.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.539+0000 7fadbf1bc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb80b15a0 con 0x7fadb8066f70 2020-11-20T12:12:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.539+0000 7fadbe1ba700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fadb81484d0 0x7fadb8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.540+0000 7fadbe1ba700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fadb81484d0 0x7fadb8061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47086/0 (socket says 172.21.15.187:47086) 2020-11-20T12:12:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.540+0000 7fadbe1ba700 1 -- 172.21.15.187:0/4232515919 learned_addr learned my addr 172.21.15.187:0/4232515919 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.540+0000 7fadbe1ba700 1 -- 172.21.15.187:0/4232515919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8066f70 msgr2=0x7fadb80b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.540+0000 7fadbe1ba700 1 --2- 172.21.15.187:0/4232515919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8066f70 0x7fadb80b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.540+0000 7fadbe1ba700 1 -- 172.21.15.187:0/4232515919 >> v2:172.21.15.5:3300/0 conn(0x7fadb80622c0 msgr2=0x7fadb80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.540+0000 7fadbe1ba700 1 --2- 172.21.15.187:0/4232515919 >> v2:172.21.15.5:3300/0 conn(0x7fadb80622c0 0x7fadb80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.540+0000 7fadbe1ba700 1 -- 172.21.15.187:0/4232515919 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb80b1840 con 0x7fadb81484d0 2020-11-20T12:12:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.540+0000 7fadbe1ba700 1 --2- 172.21.15.187:0/4232515919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb81484d0 0x7fadb8061dd0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7fadb80b85e0 tx=0x7fadb4006eb0).ready entity=mon.1 client_cookie=ce2ec9ef0adf1011 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbd1b8700 1 -- 172.21.15.187:0/4232515919 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb40098d0 con 0x7fadb81484d0 2020-11-20T12:12:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbd1b8700 1 -- 172.21.15.187:0/4232515919 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb4009a30 con 0x7fadb81484d0 2020-11-20T12:12:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbd1b8700 1 -- 172.21.15.187:0/4232515919 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb4009db0 con 0x7fadb81484d0 2020-11-20T12:12:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 -- 172.21.15.187:0/4232515919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb81484d0 msgr2=0x7fadb8061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/4232515919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb81484d0 0x7fadb8061dd0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7fadb80b85e0 tx=0x7fadb4006eb0).stop 2020-11-20T12:12:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 -- 172.21.15.187:0/4232515919 shutdown_connections 2020-11-20T12:12:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/4232515919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8066f70 0x7fadb80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/4232515919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb81484d0 0x7fadb8061dd0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/4232515919 >> v2:172.21.15.5:3300/0 conn(0x7fadb80622c0 0x7fadb80b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 -- 172.21.15.187:0/4232515919 >> 172.21.15.187:0/4232515919 conn(0x7fadb80bab30 msgr2=0x7fadb80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 -- 172.21.15.187:0/4232515919 shutdown_connections 2020-11-20T12:12:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.541+0000 7fadbf1bc700 1 -- 172.21.15.187:0/4232515919 wait complete. 2020-11-20T12:12:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.543+0000 7fadbf1bc700 1 Processor -- start 2020-11-20T12:12:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.543+0000 7fadbf1bc700 1 -- start start 2020-11-20T12:12:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbf1bc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb80622c0 0x7fadb80a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbf1bc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fadb8066f70 0x7fadb80b9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbf1bc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8132d40 0x7fadb8133180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbf1bc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb80a0910 con 0x7fadb8066f70 2020-11-20T12:12:26.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbf1bc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8133670 con 0x7fadb80622c0 2020-11-20T12:12:26.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbf1bc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb809f800 con 0x7fadb8132d40 2020-11-20T12:12:26.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbe1ba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb80622c0 0x7fadb80a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbe1ba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb80622c0 0x7fadb80a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47092/0 (socket says 172.21.15.187:47092) 2020-11-20T12:12:26.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.544+0000 7fadbe1ba700 1 -- 172.21.15.187:0/869271730 learned_addr learned my addr 172.21.15.187:0/869271730 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:26.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbd9b9700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fadb8066f70 0x7fadb80b9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbe9bb700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8132d40 0x7fadb8133180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbe1ba700 1 -- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8132d40 msgr2=0x7fadb8133180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbe1ba700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8132d40 0x7fadb8133180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbe1ba700 1 -- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fadb8066f70 msgr2=0x7fadb80b9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbe1ba700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fadb8066f70 0x7fadb80b9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbe1ba700 1 -- 172.21.15.187:0/869271730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb809faa0 con 0x7fadb80622c0 2020-11-20T12:12:26.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbd9b9700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fadb8066f70 0x7fadb80b9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbe9bb700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8132d40 0x7fadb8133180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbe1ba700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb80622c0 0x7fadb80a0420 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7fadb4006ef0 tx=0x7fadb4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fada67fc700 1 -- 172.21.15.187:0/869271730 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb401e460 con 0x7fadb80622c0 2020-11-20T12:12:26.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fada67fc700 1 -- 172.21.15.187:0/869271730 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb401e5c0 con 0x7fadb80622c0 2020-11-20T12:12:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fada67fc700 1 -- 172.21.15.187:0/869271730 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb401e960 con 0x7fadb80622c0 2020-11-20T12:12:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.545+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb809fd40 con 0x7fadb80622c0 2020-11-20T12:12:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.546+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb80b9770 con 0x7fadb80622c0 2020-11-20T12:12:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.547+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb809fed0 con 0x7fadb80622c0 2020-11-20T12:12:26.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.547+0000 7fada67fc700 1 -- 172.21.15.187:0/869271730 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fadb4025030 con 0x7fadb80622c0 2020-11-20T12:12:26.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.548+0000 7fada67fc700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad90056d50 0x7fad90059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.548+0000 7fada67fc700 1 -- 172.21.15.187:0/869271730 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fadb40717c0 con 0x7fadb80622c0 2020-11-20T12:12:26.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.548+0000 7fadbd9b9700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad90056d50 0x7fad90059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.549+0000 7fadbd9b9700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad90056d50 0x7fad90059210 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fada8006fd0 tx=0x7fada8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.552+0000 7fada67fc700 1 -- 172.21.15.187:0/869271730 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fadb4046ca0 con 0x7fadb80622c0 2020-11-20T12:12:26.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.801+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadb8061a20 con 0x7fadb80622c0 2020-11-20T12:12:26.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.802+0000 7fada67fc700 1 -- 172.21.15.187:0/869271730 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2151 (secure 0 0 0) 0x7fadb4023070 con 0x7fadb80622c0 2020-11-20T12:12:26.807 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:26.807 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:21.182870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:26.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad90056d50 msgr2=0x7fad90059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad90056d50 0x7fad90059210 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fada8006fd0 tx=0x7fada8008040).stop 2020-11-20T12:12:26.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb80622c0 msgr2=0x7fadb80a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb80622c0 0x7fadb80a0420 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7fadb4006ef0 tx=0x7fadb4015000).stop 2020-11-20T12:12:26.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 shutdown_connections 2020-11-20T12:12:26.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad90056d50 0x7fad90059210 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fadb8132d40 0x7fadb8133180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadb80622c0 0x7fadb80a0420 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 --2- 172.21.15.187:0/869271730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fadb8066f70 0x7fadb80b9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 >> 172.21.15.187:0/869271730 conn(0x7fadb80bab30 msgr2=0x7fadb8065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:26.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 shutdown_connections 2020-11-20T12:12:26.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:26.805+0000 7fadbf1bc700 1 -- 172.21.15.187:0/869271730 wait complete. 2020-11-20T12:12:26.812 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 67 2020-11-20T12:12:26.831 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and b 2020-11-20T12:12:26.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:12:26.832 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo bash -c ' 2020-11-20T12:12:26.833 INFO:teuthology.orchestra.run.smithi187:> mkdir dir_1 dir_2 2020-11-20T12:12:26.834 INFO:teuthology.orchestra.run.smithi187:> touch dir_1/original 2020-11-20T12:12:26.834 INFO:teuthology.orchestra.run.smithi187:> ln dir_1/original dir_2/linkto 2020-11-20T12:12:26.834 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:12:26.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:12:26.970 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.pin -v 1 dir_1) 2020-11-20T12:12:27.081 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:27.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.323+0000 7fbe0f59e700 1 Processor -- start 2020-11-20T12:12:27.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.323+0000 7fbe0f59e700 1 -- start start 2020-11-20T12:12:27.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.324+0000 7fbe0f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbe080b1380 0x7fbe08150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:27.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.325+0000 7fbe0f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe080658a0 con 0x7fbe0814c260 2020-11-20T12:12:27.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.325+0000 7fbe0f59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe08150390 con 0x7fbe080b1760 2020-11-20T12:12:27.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.325+0000 7fbe0f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe08150630 con 0x7fbe080b1380 2020-11-20T12:12:27.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.325+0000 7fbe0dd9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbe080b1760 0x7fbe08065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43706/0 (socket says 172.21.15.187:43706) 2020-11-20T12:12:27.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.325+0000 7fbe0dd9b700 1 -- 172.21.15.187:0/3454646337 learned_addr learned my addr 172.21.15.187:0/3454646337 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:27.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.325+0000 7fbe0e59c700 1 --2- 172.21.15.187:0/3454646337 >> v2:172.21.15.5:3300/0 conn(0x7fbe080b1380 0x7fbe08150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:27.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590435480 0 0) 0x7fbe08150390 con 0x7fbe080b1760 2020-11-20T12:12:27.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf0003240 con 0x7fbe080b1760 2020-11-20T12:12:27.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1281675874 0 0) 0x7fbdf0003240 con 0x7fbe080b1760 2020-11-20T12:12:27.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 >> v2:172.21.15.5:3300/0 conn(0x7fbe080b1380 msgr2=0x7fbe08150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:27.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 --2- 172.21.15.187:0/3454646337 >> v2:172.21.15.5:3300/0 conn(0x7fbe080b1380 0x7fbe08150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:27.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 >> v1:172.21.15.5:6789/0 conn(0x7fbe0814c260 legacy=0x7fbe0814c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:27.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe08068510 con 0x7fbe080b1760 2020-11-20T12:12:27.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fbe000014e0 con 0x7fbe080b1760 2020-11-20T12:12:27.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 >> v1:172.21.15.187:6789/0 conn(0x7fbe080b1760 legacy=0x7fbe08065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:27.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdf0003780 0x7fbdf0003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:27.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf00044e0 0x7fbdf0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:27.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf0004e50 0x7fbdf000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:27.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe08150630 con 0x7fbdf0003780 2020-11-20T12:12:27.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0005300 con 0x7fbdf00044e0 2020-11-20T12:12:27.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.326+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf000bc20 con 0x7fbdf0004e50 2020-11-20T12:12:27.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0dd9b700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf00044e0 0x7fbdf0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:27.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0e59c700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdf0003780 0x7fbdf0003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:27.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0ed9d700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf0004e50 0x7fbdf000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:27.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0dd9b700 1 -- 172.21.15.187:0/3454646337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf0004e50 msgr2=0x7fbdf000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:27.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0dd9b700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf0004e50 0x7fbdf000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:27.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0dd9b700 1 -- 172.21.15.187:0/3454646337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdf0003780 msgr2=0x7fbdf0003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:27.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0dd9b700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdf0003780 0x7fbdf0003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:27.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0dd9b700 1 -- 172.21.15.187:0/3454646337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf0003240 con 0x7fbdf00044e0 2020-11-20T12:12:27.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.327+0000 7fbe0ed9d700 1 -- 172.21.15.187:0/3454646337 reap_dead start 2020-11-20T12:12:27.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.328+0000 7fbe0dd9b700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf00044e0 0x7fbdf0004960 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7fbe0000c210 tx=0x7fbe000076d0).ready entity=mon.1 client_cookie=60ecd95537af3ef server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:27.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.328+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe000070d0 con 0x7fbdf00044e0 2020-11-20T12:12:27.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.328+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe00007230 con 0x7fbdf00044e0 2020-11-20T12:12:27.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.328+0000 7fbe0d59a700 1 -- 172.21.15.187:0/3454646337 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe000055f0 con 0x7fbdf00044e0 2020-11-20T12:12:27.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.328+0000 7fbe0f59e700 1 -- 172.21.15.187:0/3454646337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf00044e0 msgr2=0x7fbdf0004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:27.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.328+0000 7fbe0f59e700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf00044e0 0x7fbdf0004960 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7fbe0000c210 tx=0x7fbe000076d0).stop 2020-11-20T12:12:27.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.329+0000 7fbe0f59e700 1 -- 172.21.15.187:0/3454646337 shutdown_connections 2020-11-20T12:12:27.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.329+0000 7fbe0f59e700 1 --2- 172.21.15.187:0/3454646337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf00044e0 0x7fbdf0004960 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:27.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.329+0000 7fbe0f59e700 1 -- 172.21.15.187:0/3454646337 >> 172.21.15.187:0/3454646337 conn(0x7fbe080bab30 msgr2=0x7fbe080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:27.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.329+0000 7fbe0f59e700 1 -- 172.21.15.187:0/3454646337 shutdown_connections 2020-11-20T12:12:27.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.329+0000 7fbe0f59e700 1 -- 172.21.15.187:0/3454646337 wait complete. 2020-11-20T12:12:27.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.330+0000 7fbe0f59e700 1 Processor -- start 2020-11-20T12:12:27.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.330+0000 7fbe0f59e700 1 -- start start 2020-11-20T12:12:27.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.331+0000 7fbe0f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf00044e0 0x7fbe0809eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:27.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.331+0000 7fbe0f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0809d1d0 0x7fbe080a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:27.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.331+0000 7fbe0f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0814b260 0x7fbe0814b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:27.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.331+0000 7fbe0f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0809f340 con 0x7fbe0809d1d0 2020-11-20T12:12:27.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.331+0000 7fbe0f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0809f5e0 con 0x7fbe0814b260 2020-11-20T12:12:27.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.331+0000 7fbe0f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0809f880 con 0x7fbdf00044e0 2020-11-20T12:12:27.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0814b260 0x7fbe0814b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:27.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0814b260 0x7fbe0814b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47116/0 (socket says 172.21.15.187:47116) 2020-11-20T12:12:27.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 -- 172.21.15.187:0/439582410 learned_addr learned my addr 172.21.15.187:0/439582410 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:27.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0e59c700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf00044e0 0x7fbe0809eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:27.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 -- 172.21.15.187:0/439582410 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf00044e0 msgr2=0x7fbe0809eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:27.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf00044e0 0x7fbe0809eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:27.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 -- 172.21.15.187:0/439582410 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0809d1d0 msgr2=0x7fbe080a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:27.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0809d1d0 0x7fbe080a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:27.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 -- 172.21.15.187:0/439582410 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0814bbb0 con 0x7fbe0814b260 2020-11-20T12:12:27.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbe0ed9d700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0814b260 0x7fbe0814b6c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7fbe080b50b0 tx=0x7fbe04009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:27.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.332+0000 7fbdfa7fc700 1 -- 172.21.15.187:0/439582410 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04003280 con 0x7fbe0814b260 2020-11-20T12:12:27.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.333+0000 7fbdfa7fc700 1 -- 172.21.15.187:0/439582410 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe040033e0 con 0x7fbe0814b260 2020-11-20T12:12:27.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.333+0000 7fbdfa7fc700 1 -- 172.21.15.187:0/439582410 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04001750 con 0x7fbe0814b260 2020-11-20T12:12:27.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.333+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0809f000 con 0x7fbe0814b260 2020-11-20T12:12:27.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.333+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0815c340 con 0x7fbe0814b260 2020-11-20T12:12:27.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.334+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0809f190 con 0x7fbe0814b260 2020-11-20T12:12:27.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.334+0000 7fbdfa7fc700 1 -- 172.21.15.187:0/439582410 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbe04001b40 con 0x7fbe0814b260 2020-11-20T12:12:27.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.335+0000 7fbdfa7fc700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde0056d50 0x7fbde0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:27.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.335+0000 7fbdfa7fc700 1 -- 172.21.15.187:0/439582410 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbe0406fad0 con 0x7fbe0814b260 2020-11-20T12:12:27.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.335+0000 7fbe0e59c700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde0056d50 0x7fbde0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:27.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.336+0000 7fbe0e59c700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde0056d50 0x7fbde0059210 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7fbe0805ea30 tx=0x7fbdfc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:27.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.339+0000 7fbdfa7fc700 1 -- 172.21.15.187:0/439582410 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbe04044f20 con 0x7fbe0814b260 2020-11-20T12:12:27.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.582+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe0809fa40 con 0x7fbe0814b260 2020-11-20T12:12:27.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.584+0000 7fbdfa7fc700 1 -- 172.21.15.187:0/439582410 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2151 (secure 0 0 0) 0x7fbe04048540 con 0x7fbe0814b260 2020-11-20T12:12:27.588 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:27.589 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:21.182870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:27.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde0056d50 msgr2=0x7fbde0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:27.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde0056d50 0x7fbde0059210 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7fbe0805ea30 tx=0x7fbdfc008040).stop 2020-11-20T12:12:27.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0814b260 msgr2=0x7fbe0814b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:27.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0814b260 0x7fbe0814b6c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7fbe080b50b0 tx=0x7fbe04009040).stop 2020-11-20T12:12:27.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 shutdown_connections 2020-11-20T12:12:27.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbde0056d50 0x7fbde0059210 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:27.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe0814b260 0x7fbe0814b6c0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:27.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbdf00044e0 0x7fbe0809eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:27.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 --2- 172.21.15.187:0/439582410 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0809d1d0 0x7fbe080a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:27.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 >> 172.21.15.187:0/439582410 conn(0x7fbe080bab30 msgr2=0x7fbe0814ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:27.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 shutdown_connections 2020-11-20T12:12:27.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:27.586+0000 7fbe0f59e700 1 -- 172.21.15.187:0/439582410 wait complete. 2020-11-20T12:12:27.595 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 67 2020-11-20T12:12:27.615 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:12:27.907 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2020-11-20T12:12:27.908 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:12:27.909 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2020-11-20T12:12:29.911 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:30.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.159+0000 7f91ce0c0700 1 Processor -- start 2020-11-20T12:12:30.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.160+0000 7f91ce0c0700 1 -- start start 2020-11-20T12:12:30.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.161+0000 7f91ce0c0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f91c8065190 0x7f91c8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.161+0000 7f91ce0c0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c809ca90 0x7f91c809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.161+0000 7f91ce0c0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c80b13e0 con 0x7f91c8065f90 2020-11-20T12:12:30.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.161+0000 7f91ce0c0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c814ab30 con 0x7f91c8065190 2020-11-20T12:12:30.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.161+0000 7f91ce0c0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c8062c10 con 0x7f91c809ca90 2020-11-20T12:12:30.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.161+0000 7f91cc8bd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f91c8065f90 0x7f91c8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43726/0 (socket says 172.21.15.187:43726) 2020-11-20T12:12:30.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.161+0000 7f91cc8bd700 1 -- 172.21.15.187:0/3033326130 learned_addr learned my addr 172.21.15.187:0/3033326130 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:30.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863431472 0 0) 0x7f91c80b13e0 con 0x7f91c8065f90 2020-11-20T12:12:30.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b0003240 con 0x7f91c8065f90 2020-11-20T12:12:30.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91cd0be700 1 --2- 172.21.15.187:0/3033326130 >> v2:172.21.15.5:3300/0 conn(0x7f91c8065190 0x7f91c8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91cd8bf700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c809ca90 0x7f91c809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3684098922 0 0) 0x7f91b0003240 con 0x7f91c8065f90 2020-11-20T12:12:30.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c809ca90 msgr2=0x7f91c809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:30.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91b7fff700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c809ca90 0x7f91c809cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 >> v2:172.21.15.5:3300/0 conn(0x7f91c8065190 msgr2=0x7f91c8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:30.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91b7fff700 1 --2- 172.21.15.187:0/3033326130 >> v2:172.21.15.5:3300/0 conn(0x7f91c8065190 0x7f91c8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.162+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c80b1640 con 0x7f91c8065f90 2020-11-20T12:12:30.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91cd0be700 1 --2- 172.21.15.187:0/3033326130 >> v2:172.21.15.5:3300/0 conn(0x7f91c8065190 0x7f91c8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:30.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f91b80014e0 con 0x7f91c8065f90 2020-11-20T12:12:30.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 >> v1:172.21.15.187:6789/0 conn(0x7f91c8065f90 legacy=0x7f91c8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91b7fff700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c809ca90 0x7f91b00037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91b7fff700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91b0003d10 0x7f91b00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91b7fff700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91b00049d0 0x7f91b0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c814ab30 con 0x7f91b00049d0 2020-11-20T12:12:30.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c8062c10 con 0x7f91b0003d10 2020-11-20T12:12:30.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00042d0 con 0x7f91c809ca90 2020-11-20T12:12:30.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.163+0000 7f91cc8bd700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91b0003d10 0x7f91b00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cd0be700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c809ca90 0x7f91b00037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cd8bf700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91b00049d0 0x7f91b0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cc8bd700 1 -- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c809ca90 msgr2=0x7f91b00037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:30.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cc8bd700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c809ca90 0x7f91b00037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cc8bd700 1 -- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91b00049d0 msgr2=0x7f91b0004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:30.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cc8bd700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91b00049d0 0x7f91b0004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cc8bd700 1 -- 172.21.15.187:0/3033326130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b0003240 con 0x7f91b0003d10 2020-11-20T12:12:30.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cd8bf700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91b00049d0 0x7f91b0004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:30.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.164+0000 7f91cc8bd700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91b0003d10 0x7f91b00044e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f91b800c1e0 tx=0x7f91c80bb650).ready entity=mon.1 client_cookie=163eeb9ba0c105eb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:30.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b80076f0 con 0x7f91b0003d10 2020-11-20T12:12:30.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b80070d0 con 0x7f91b0003d10 2020-11-20T12:12:30.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91b7fff700 1 -- 172.21.15.187:0/3033326130 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8007470 con 0x7f91b0003d10 2020-11-20T12:12:30.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91ce0c0700 1 -- 172.21.15.187:0/3033326130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91b0003d10 msgr2=0x7f91b00044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91ce0c0700 1 --2- 172.21.15.187:0/3033326130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91b0003d10 0x7f91b00044e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f91b800c1e0 tx=0x7f91c80bb650).stop 2020-11-20T12:12:30.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91cd8bf700 1 -- 172.21.15.187:0/3033326130 reap_dead start 2020-11-20T12:12:30.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91ce0c0700 1 -- 172.21.15.187:0/3033326130 shutdown_connections 2020-11-20T12:12:30.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91ce0c0700 1 -- 172.21.15.187:0/3033326130 >> 172.21.15.187:0/3033326130 conn(0x7f91c80bab30 msgr2=0x7f91c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:30.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.165+0000 7f91ce0c0700 1 -- 172.21.15.187:0/3033326130 shutdown_connections 2020-11-20T12:12:30.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.166+0000 7f91ce0c0700 1 -- 172.21.15.187:0/3033326130 wait complete. 2020-11-20T12:12:30.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.167+0000 7f91ce0c0700 1 Processor -- start 2020-11-20T12:12:30.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.167+0000 7f91ce0c0700 1 -- start start 2020-11-20T12:12:30.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.168+0000 7f91ce0c0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c815bc40 0x7f91c815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.168+0000 7f91ce0c0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c8149700 0x7f91c80b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.168+0000 7f91ce0c0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91c80b2e10 0x7f91c809fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.168+0000 7f91ce0c0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c815c6a0 con 0x7f91c80b2e10 2020-11-20T12:12:30.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.168+0000 7f91ce0c0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c815c940 con 0x7f91c815bc40 2020-11-20T12:12:30.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.168+0000 7f91ce0c0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c809f9b0 con 0x7f91c8149700 2020-11-20T12:12:30.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.168+0000 7f91cd0be700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c815bc40 0x7f91c815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd0be700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c815bc40 0x7f91c815c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47134/0 (socket says 172.21.15.187:47134) 2020-11-20T12:12:30.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd0be700 1 -- 172.21.15.187:0/2679267852 learned_addr learned my addr 172.21.15.187:0/2679267852 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:30.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cc8bd700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c8149700 0x7f91c80b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd8bf700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91c80b2e10 0x7f91c809fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd0be700 1 -- 172.21.15.187:0/2679267852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c8149700 msgr2=0x7f91c80b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd0be700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c8149700 0x7f91c80b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd0be700 1 -- 172.21.15.187:0/2679267852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91c80b2e10 msgr2=0x7f91c809fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd0be700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91c80b2e10 0x7f91c809fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd0be700 1 -- 172.21.15.187:0/2679267852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c8149dc0 con 0x7f91c815bc40 2020-11-20T12:12:30.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.169+0000 7f91cd0be700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c815bc40 0x7f91c815c020 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7f91c4010f40 tx=0x7f91c4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:30.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.170+0000 7f91b57fa700 1 -- 172.21.15.187:0/2679267852 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c400eca0 con 0x7f91c815bc40 2020-11-20T12:12:30.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.170+0000 7f91b57fa700 1 -- 172.21.15.187:0/2679267852 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c400c030 con 0x7f91c815bc40 2020-11-20T12:12:30.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.170+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c8149fb0 con 0x7f91c815bc40 2020-11-20T12:12:30.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.170+0000 7f91b57fa700 1 -- 172.21.15.187:0/2679267852 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c4014410 con 0x7f91c815bc40 2020-11-20T12:12:30.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.170+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c80a0500 con 0x7f91c815bc40 2020-11-20T12:12:30.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.171+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c815c510 con 0x7f91c815bc40 2020-11-20T12:12:30.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.171+0000 7f91b57fa700 1 -- 172.21.15.187:0/2679267852 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f91c4012040 con 0x7f91c815bc40 2020-11-20T12:12:30.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.172+0000 7f91b57fa700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f91a0056d20 0x7f91a00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.172+0000 7f91b57fa700 1 -- 172.21.15.187:0/2679267852 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f91c4070870 con 0x7f91c815bc40 2020-11-20T12:12:30.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.172+0000 7f91cc8bd700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f91a0056d20 0x7f91a00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.173+0000 7f91cc8bd700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f91a0056d20 0x7f91a00591e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f91c805ec00 tx=0x7f91b8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:30.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.176+0000 7f91b57fa700 1 -- 172.21.15.187:0/2679267852 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f91c4045b50 con 0x7f91c815bc40 2020-11-20T12:12:30.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.428+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91c8065690 con 0x7f91c815bc40 2020-11-20T12:12:30.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.428+0000 7f91b57fa700 1 -- 172.21.15.187:0/2679267852 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2151 (secure 0 0 0) 0x7f91c4018020 con 0x7f91c815bc40 2020-11-20T12:12:30.433 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:30.433 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:21.182870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:30.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f91a0056d20 msgr2=0x7f91a00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f91a0056d20 0x7f91a00591e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f91c805ec00 tx=0x7f91b8015000).stop 2020-11-20T12:12:30.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c815bc40 msgr2=0x7f91c815c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c815bc40 0x7f91c815c020 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7f91c4010f40 tx=0x7f91c4008040).stop 2020-11-20T12:12:30.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 shutdown_connections 2020-11-20T12:12:30.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f91a0056d20 0x7f91a00591e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91c80b2e10 0x7f91c809fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c815bc40 0x7f91c815c020 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 --2- 172.21.15.187:0/2679267852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91c8149700 0x7f91c80b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 >> 172.21.15.187:0/2679267852 conn(0x7f91c80bab30 msgr2=0x7f91c80b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:30.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 shutdown_connections 2020-11-20T12:12:30.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:30.431+0000 7f91ce0c0700 1 -- 172.21.15.187:0/2679267852 wait complete. 2020-11-20T12:12:30.438 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 67 2020-11-20T12:12:30.458 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:12:30.747 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2020-11-20T12:12:30.748 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:12:30.748 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2020-11-20T12:12:32.750 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:33.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.997+0000 7f541a77f700 1 Processor -- start 2020-11-20T12:12:33.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.997+0000 7f541a77f700 1 -- start start 2020-11-20T12:12:33.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.998+0000 7f541a77f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f54140b6650 0x7f54140afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.998+0000 7f541a77f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5414066f70 0x7f54140aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.998+0000 7f541a77f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140b1300 con 0x7f54141484d0 2020-11-20T12:12:33.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.998+0000 7f541a77f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5414067730 con 0x7f54140b6650 2020-11-20T12:12:33.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.998+0000 7f541a77f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54140b15a0 con 0x7f5414066f70 2020-11-20T12:12:33.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.999+0000 7f541977d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f54141484d0 0x7f5414061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43746/0 (socket says 172.21.15.187:43746) 2020-11-20T12:12:33.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.999+0000 7f541977d700 1 -- 172.21.15.187:0/902573742 learned_addr learned my addr 172.21.15.187:0/902573742 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:33.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.999+0000 7f5419f7e700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5414066f70 0x7f54140aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.999+0000 7f5418f7c700 1 --2- 172.21.15.187:0/902573742 >> v2:172.21.15.5:3300/0 conn(0x7f54140b6650 0x7f54140afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.999+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220475183 0 0) 0x7f54140b1300 con 0x7f54141484d0 2020-11-20T12:12:33.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:32.999+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53fc003240 con 0x7f54141484d0 2020-11-20T12:12:33.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 631992248 0 0) 0x7f53fc003240 con 0x7f54141484d0 2020-11-20T12:12:33.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5414066f70 msgr2=0x7f54140aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:33.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5414066f70 0x7f54140aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 >> v2:172.21.15.5:3300/0 conn(0x7f54140b6650 msgr2=0x7f54140afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:33.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 --2- 172.21.15.187:0/902573742 >> v2:172.21.15.5:3300/0 conn(0x7f54140b6650 0x7f54140afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140b1840 con 0x7f54141484d0 2020-11-20T12:12:33.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5410001860 con 0x7f54141484d0 2020-11-20T12:12:33.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 >> v1:172.21.15.187:6789/0 conn(0x7f54141484d0 legacy=0x7f5414061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53fc003730 0x7f53fc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53fc0044e0 0x7f53fc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5414066f70 0x7f53fc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5414067730 con 0x7f53fc003730 2020-11-20T12:12:33.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54140b15a0 con 0x7f53fc0044e0 2020-11-20T12:12:33.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.000+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc004270 con 0x7f5414066f70 2020-11-20T12:12:33.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.001+0000 7f5418f7c700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53fc0044e0 0x7f53fc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.001+0000 7f541977d700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53fc003730 0x7f53fc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.001+0000 7f5419f7e700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5414066f70 0x7f53fc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.001+0000 7f5418f7c700 1 -- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5414066f70 msgr2=0x7f53fc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:33.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.001+0000 7f5418f7c700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5414066f70 0x7f53fc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.001+0000 7f5418f7c700 1 -- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53fc003730 msgr2=0x7f53fc003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:33.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.001+0000 7f5418f7c700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53fc003730 0x7f53fc003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.001+0000 7f5418f7c700 1 -- 172.21.15.187:0/902573742 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc003240 con 0x7f53fc0044e0 2020-11-20T12:12:33.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.002+0000 7f5418f7c700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53fc0044e0 0x7f53fc004960 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f540400bc00 tx=0x7f5404006df0).ready entity=mon.1 client_cookie=7146efe60b45c6ec server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:33.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.002+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540400b310 con 0x7f53fc0044e0 2020-11-20T12:12:33.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.002+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540400eb40 con 0x7f53fc0044e0 2020-11-20T12:12:33.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.002+0000 7f5403fff700 1 -- 172.21.15.187:0/902573742 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540400eec0 con 0x7f53fc0044e0 2020-11-20T12:12:33.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.003+0000 7f541a77f700 1 -- 172.21.15.187:0/902573742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53fc0044e0 msgr2=0x7f53fc004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.003+0000 7f541a77f700 1 --2- 172.21.15.187:0/902573742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53fc0044e0 0x7f53fc004960 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f540400bc00 tx=0x7f5404006df0).stop 2020-11-20T12:12:33.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.003+0000 7f5419f7e700 1 -- 172.21.15.187:0/902573742 reap_dead start 2020-11-20T12:12:33.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.003+0000 7f541a77f700 1 -- 172.21.15.187:0/902573742 shutdown_connections 2020-11-20T12:12:33.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.003+0000 7f541a77f700 1 -- 172.21.15.187:0/902573742 >> 172.21.15.187:0/902573742 conn(0x7f54140bab30 msgr2=0x7f54140baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:33.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.003+0000 7f541a77f700 1 -- 172.21.15.187:0/902573742 shutdown_connections 2020-11-20T12:12:33.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.003+0000 7f541a77f700 1 -- 172.21.15.187:0/902573742 wait complete. 2020-11-20T12:12:33.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.004+0000 7f541a77f700 1 Processor -- start 2020-11-20T12:12:33.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.004+0000 7f541a77f700 1 -- start start 2020-11-20T12:12:33.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.005+0000 7f541a77f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5414159890 0x7f5414134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.005+0000 7f541a77f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5414135270 0x7f5414159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.005+0000 7f541a77f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541415a160 0x7f54140b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.005+0000 7f541a77f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54141356f0 con 0x7f5414159890 2020-11-20T12:12:33.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.005+0000 7f541a77f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541415a5e0 con 0x7f5414135270 2020-11-20T12:12:33.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.005+0000 7f541a77f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54140b2af0 con 0x7f541415a160 2020-11-20T12:12:33.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f5418f7c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5414135270 0x7f5414159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f5418f7c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5414135270 0x7f5414159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47154/0 (socket says 172.21.15.187:47154) 2020-11-20T12:12:33.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f5418f7c700 1 -- 172.21.15.187:0/1850853509 learned_addr learned my addr 172.21.15.187:0/1850853509 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:33.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f5418f7c700 1 -- 172.21.15.187:0/1850853509 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541415a160 msgr2=0x7f54140b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f5418f7c700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541415a160 0x7f54140b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:33.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f5418f7c700 1 -- 172.21.15.187:0/1850853509 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5414159890 msgr2=0x7f5414134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f541977d700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5414159890 0x7f5414134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f5418f7c700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5414159890 0x7f5414134d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.006+0000 7f5418f7c700 1 -- 172.21.15.187:0/1850853509 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140b2d90 con 0x7f5414135270 2020-11-20T12:12:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.007+0000 7f5418f7c700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5414135270 0x7f5414159c70 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f5404006fd0 tx=0x7f5404019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:33.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.007+0000 7f54017fa700 1 -- 172.21.15.187:0/1850853509 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540401e630 con 0x7f5414135270 2020-11-20T12:12:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.007+0000 7f54017fa700 1 -- 172.21.15.187:0/1850853509 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540400a070 con 0x7f5414135270 2020-11-20T12:12:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.007+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54140b3030 con 0x7f5414135270 2020-11-20T12:12:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.007+0000 7f54017fa700 1 -- 172.21.15.187:0/1850853509 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540401e7b0 con 0x7f5414135270 2020-11-20T12:12:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.007+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54140b3750 con 0x7f5414135270 2020-11-20T12:12:33.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.008+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54140b2940 con 0x7f5414135270 2020-11-20T12:12:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.009+0000 7f54017fa700 1 -- 172.21.15.187:0/1850853509 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f540401eb60 con 0x7f5414135270 2020-11-20T12:12:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.009+0000 7f54017fa700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53ec056d20 0x7f53ec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.009+0000 7f54017fa700 1 -- 172.21.15.187:0/1850853509 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5404072430 con 0x7f5414135270 2020-11-20T12:12:33.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.010+0000 7f541977d700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53ec056d20 0x7f53ec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.010+0000 7f541977d700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53ec056d20 0x7f53ec0591e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f54100012a0 tx=0x7f5410005f30).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:33.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.013+0000 7f54017fa700 1 -- 172.21.15.187:0/1850853509 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5404047880 con 0x7f5414135270 2020-11-20T12:12:33.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.263+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5414061a20 con 0x7f5414135270 2020-11-20T12:12:33.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.264+0000 7f54017fa700 1 -- 172.21.15.187:0/1850853509 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2151 (secure 0 0 0) 0x7f540404aea0 con 0x7f5414135270 2020-11-20T12:12:33.269 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:33.269 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:21.182870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:33.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.266+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53ec056d20 msgr2=0x7f53ec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.266+0000 7f541a77f700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53ec056d20 0x7f53ec0591e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f54100012a0 tx=0x7f5410005f30).stop 2020-11-20T12:12:33.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.266+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5414135270 msgr2=0x7f5414159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.266+0000 7f541a77f700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5414135270 0x7f5414159c70 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f5404006fd0 tx=0x7f5404019000).stop 2020-11-20T12:12:33.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.267+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 shutdown_connections 2020-11-20T12:12:33.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.267+0000 7f541a77f700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53ec056d20 0x7f53ec0591e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.267+0000 7f541a77f700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5414159890 0x7f5414134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.267+0000 7f541a77f700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541415a160 0x7f54140b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:33.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.267+0000 7f541a77f700 1 --2- 172.21.15.187:0/1850853509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5414135270 0x7f5414159c70 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.267+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 >> 172.21.15.187:0/1850853509 conn(0x7f54140bab30 msgr2=0x7f541409cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:33.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.267+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 shutdown_connections 2020-11-20T12:12:33.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:33.267+0000 7f541a77f700 1 -- 172.21.15.187:0/1850853509 wait complete. 2020-11-20T12:12:33.282 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 67 2020-11-20T12:12:33.295 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:12:33.590 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2020-11-20T12:12:33.591 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:12:33.592 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2020-11-20T12:12:35.593 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:35.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.837+0000 7fb8aec71700 1 Processor -- start 2020-11-20T12:12:35.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.837+0000 7fb8aec71700 1 -- start start 2020-11-20T12:12:35.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.839+0000 7fb8aec71700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a8065f90 0x7fb8a8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.839+0000 7fb8aec71700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8a809ca90 0x7fb8a809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.839+0000 7fb8aec71700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a80b13e0 con 0x7fb8a8065190 2020-11-20T12:12:35.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.839+0000 7fb8aec71700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a814ab30 con 0x7fb8a809ca90 2020-11-20T12:12:35.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.839+0000 7fb8aec71700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8062c10 con 0x7fb8a8065f90 2020-11-20T12:12:35.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.839+0000 7fb8adc6f700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb8a8065190 0x7fb8a8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43768/0 (socket says 172.21.15.187:43768) 2020-11-20T12:12:35.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.839+0000 7fb8adc6f700 1 -- 172.21.15.187:0/2736370235 learned_addr learned my addr 172.21.15.187:0/2736370235 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:35.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8ae470700 1 --2- 172.21.15.187:0/2736370235 >> v2:172.21.15.5:3300/0 conn(0x7fb8a809ca90 0x7fb8a809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8ad46e700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a8065f90 0x7fb8a8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281744050 0 0) 0x7fb8a80b13e0 con 0x7fb8a8065190 2020-11-20T12:12:35.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb890003240 con 0x7fb8a8065190 2020-11-20T12:12:35.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4114574547 0 0) 0x7fb890003240 con 0x7fb8a8065190 2020-11-20T12:12:35.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a8065f90 msgr2=0x7fb8a8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:35.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a8065f90 0x7fb8a8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 >> v2:172.21.15.5:3300/0 conn(0x7fb8a809ca90 msgr2=0x7fb8a809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:35.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 --2- 172.21.15.187:0/2736370235 >> v2:172.21.15.5:3300/0 conn(0x7fb8a809ca90 0x7fb8a809cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a80b1640 con 0x7fb8a8065190 2020-11-20T12:12:35.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fb8a40014e0 con 0x7fb8a8065190 2020-11-20T12:12:35.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 >> v1:172.21.15.187:6789/0 conn(0x7fb8a8065190 legacy=0x7fb8a8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb890003730 0x7fb890003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8900044e0 0x7fb890004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.840+0000 7fb8acc6d700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a8065f90 0x7fb890004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a814ab30 con 0x7fb8900044e0 2020-11-20T12:12:35.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8062c10 con 0x7fb890003730 2020-11-20T12:12:35.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb890004290 con 0x7fb8a8065f90 2020-11-20T12:12:35.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8adc6f700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb890003730 0x7fb890003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8ae470700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a8065f90 0x7fb890004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8ad46e700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8900044e0 0x7fb890004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8adc6f700 1 -- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a8065f90 msgr2=0x7fb890004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:35.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8adc6f700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a8065f90 0x7fb890004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.841+0000 7fb8adc6f700 1 -- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8900044e0 msgr2=0x7fb890004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:35.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.842+0000 7fb8adc6f700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8900044e0 0x7fb890004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.842+0000 7fb8adc6f700 1 -- 172.21.15.187:0/2736370235 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb890003240 con 0x7fb890003730 2020-11-20T12:12:35.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.842+0000 7fb8ad46e700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8900044e0 0x7fb890004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:35.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.842+0000 7fb8adc6f700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb890003730 0x7fb890003b70 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7fb8a400c440 tx=0x7fb8a80bb650).ready entity=mon.1 client_cookie=e3a2049735325447 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:35.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.842+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a40072e0 con 0x7fb890003730 2020-11-20T12:12:35.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.842+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4007440 con 0x7fb890003730 2020-11-20T12:12:35.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.842+0000 7fb8acc6d700 1 -- 172.21.15.187:0/2736370235 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4005840 con 0x7fb890003730 2020-11-20T12:12:35.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.843+0000 7fb8aec71700 1 -- 172.21.15.187:0/2736370235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb890003730 msgr2=0x7fb890003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.843+0000 7fb8aec71700 1 --2- 172.21.15.187:0/2736370235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb890003730 0x7fb890003b70 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7fb8a400c440 tx=0x7fb8a80bb650).stop 2020-11-20T12:12:35.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.843+0000 7fb8ae470700 1 -- 172.21.15.187:0/2736370235 reap_dead start 2020-11-20T12:12:35.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.843+0000 7fb8aec71700 1 -- 172.21.15.187:0/2736370235 shutdown_connections 2020-11-20T12:12:35.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.843+0000 7fb8aec71700 1 -- 172.21.15.187:0/2736370235 >> 172.21.15.187:0/2736370235 conn(0x7fb8a80bab30 msgr2=0x7fb8a80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:35.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.843+0000 7fb8aec71700 1 -- 172.21.15.187:0/2736370235 shutdown_connections 2020-11-20T12:12:35.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.843+0000 7fb8aec71700 1 -- 172.21.15.187:0/2736370235 wait complete. 2020-11-20T12:12:35.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.845+0000 7fb8aec71700 1 Processor -- start 2020-11-20T12:12:35.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.845+0000 7fb8aec71700 1 -- start start 2020-11-20T12:12:35.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8aec71700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a8142a60 0x7fb8a8142e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8aec71700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a809eb80 0x7fb8a80b2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8aec71700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a80b2950 0x7fb8a809fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8aec71700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a809f020 con 0x7fb8a80b2950 2020-11-20T12:12:35.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8aec71700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a809f2c0 con 0x7fb8a8142a60 2020-11-20T12:12:35.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8aec71700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a809f560 con 0x7fb8a809eb80 2020-11-20T12:12:35.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8adc6f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a8142a60 0x7fb8a8142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8adc6f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a8142a60 0x7fb8a8142e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47174/0 (socket says 172.21.15.187:47174) 2020-11-20T12:12:35.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8adc6f700 1 -- 172.21.15.187:0/3855286828 learned_addr learned my addr 172.21.15.187:0/3855286828 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:35.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8ae470700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a80b2950 0x7fb8a809fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.846+0000 7fb8ad46e700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a809eb80 0x7fb8a80b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8adc6f700 1 -- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a809eb80 msgr2=0x7fb8a80b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8adc6f700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a809eb80 0x7fb8a80b2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8adc6f700 1 -- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a80b2950 msgr2=0x7fb8a809fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8adc6f700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a80b2950 0x7fb8a809fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8adc6f700 1 -- 172.21.15.187:0/3855286828 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a815a450 con 0x7fb8a8142a60 2020-11-20T12:12:35.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8ae470700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a80b2950 0x7fb8a809fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:35.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8adc6f700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a8142a60 0x7fb8a8142e40 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7fb8a400a000 tx=0x7fb8a4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:35.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb895ffb700 1 -- 172.21.15.187:0/3855286828 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400ceb0 con 0x7fb8a8142a60 2020-11-20T12:12:35.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb895ffb700 1 -- 172.21.15.187:0/3855286828 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4001840 con 0x7fb8a8142a60 2020-11-20T12:12:35.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb895ffb700 1 -- 172.21.15.187:0/3855286828 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4013660 con 0x7fb8a8142a60 2020-11-20T12:12:35.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a8143330 con 0x7fb8a8142a60 2020-11-20T12:12:35.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.847+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a80a0130 con 0x7fb8a8142a60 2020-11-20T12:12:35.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.848+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a8143550 con 0x7fb8a8142a60 2020-11-20T12:12:35.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.849+0000 7fb895ffb700 1 -- 172.21.15.187:0/3855286828 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb8a4025020 con 0x7fb8a8142a60 2020-11-20T12:12:35.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.850+0000 7fb895ffb700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb880056d20 0x7fb8800591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.850+0000 7fb895ffb700 1 -- 172.21.15.187:0/3855286828 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb8a40713d0 con 0x7fb8a8142a60 2020-11-20T12:12:35.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.850+0000 7fb8ad46e700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb880056d20 0x7fb8800591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.851+0000 7fb8ad46e700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb880056d20 0x7fb8800591e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fb8a8159650 tx=0x7fb898008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:35.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:35.854+0000 7fb895ffb700 1 -- 172.21.15.187:0/3855286828 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb8a4046710 con 0x7fb8a8142a60 2020-11-20T12:12:36.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.096+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8a80b99a0 con 0x7fb8a8142a60 2020-11-20T12:12:36.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.097+0000 7fb895ffb700 1 -- 172.21.15.187:0/3855286828 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2152 (secure 0 0 0) 0x7fb8a4049d30 con 0x7fb8a8142a60 2020-11-20T12:12:36.102 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:36.102 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:34.321010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:36.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb880056d20 msgr2=0x7fb8800591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:36.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb880056d20 0x7fb8800591e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fb8a8159650 tx=0x7fb898008040).stop 2020-11-20T12:12:36.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a8142a60 msgr2=0x7fb8a8142e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:36.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a8142a60 0x7fb8a8142e40 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7fb8a400a000 tx=0x7fb8a4015000).stop 2020-11-20T12:12:36.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 shutdown_connections 2020-11-20T12:12:36.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb880056d20 0x7fb8800591e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:36.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a80b2950 0x7fb8a809fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:36.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a8142a60 0x7fb8a8142e40 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:36.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 --2- 172.21.15.187:0/3855286828 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a809eb80 0x7fb8a80b2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:36.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 >> 172.21.15.187:0/3855286828 conn(0x7fb8a80bab30 msgr2=0x7fb8a80ba0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:36.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 shutdown_connections 2020-11-20T12:12:36.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.100+0000 7fb8aec71700 1 -- 172.21.15.187:0/3855286828 wait complete. 2020-11-20T12:12:36.108 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 68 2020-11-20T12:12:36.127 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:12:36.419 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}}] 2020-11-20T12:12:36.420 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/dir_1", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false } ] 2020-11-20T12:12:36.421 INFO:tasks.cephfs.cephfs_test_case:[('/dir_1', 1)] =?= [('/dir_1', 1)] 2020-11-20T12:12:36.421 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:12:36.422 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.pin -v 0 dir_2) 2020-11-20T12:12:36.531 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:36.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.767+0000 7fc3fa321700 1 Processor -- start 2020-11-20T12:12:36.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.767+0000 7fc3fa321700 1 -- start start 2020-11-20T12:12:36.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.769+0000 7fc3fa321700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f4068230 0x7fc3f41482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:36.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.769+0000 7fc3fa321700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f4144b80 con 0x7fc3f40b57b0 2020-11-20T12:12:36.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.769+0000 7fc3fa321700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f41489e0 con 0x7fc3f40b5f50 2020-11-20T12:12:36.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.769+0000 7fc3fa321700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f41487c0 con 0x7fc3f4068230 2020-11-20T12:12:36.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.769+0000 7fc3f9b20700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc3f40b5f50 0x7fc3f40b63e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43790/0 (socket says 172.21.15.187:43790) 2020-11-20T12:12:36.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.769+0000 7fc3f9b20700 1 -- 172.21.15.187:0/3017329121 learned_addr learned my addr 172.21.15.187:0/3017329121 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:36.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3f8b1e700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f4068230 0x7fc3f41482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:36.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684461191 0 0) 0x7fc3f41489e0 con 0x7fc3f40b5f50 2020-11-20T12:12:36.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3dc003240 con 0x7fc3f40b5f50 2020-11-20T12:12:36.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 292273892 0 0) 0x7fc3dc003240 con 0x7fc3f40b5f50 2020-11-20T12:12:36.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f4068230 msgr2=0x7fc3f41482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:36.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f4068230 0x7fc3f41482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:36.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 >> v1:172.21.15.5:6789/0 conn(0x7fc3f40b57b0 legacy=0x7fc3f4062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:36.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f4148c80 con 0x7fc3f40b5f50 2020-11-20T12:12:36.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fc3ec0018a0 con 0x7fc3f40b5f50 2020-11-20T12:12:36.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 >> v1:172.21.15.187:6789/0 conn(0x7fc3f40b5f50 legacy=0x7fc3f40b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:36.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f4068230 0x7fc3dc003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:36.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3dc003c20 0x7fc3dc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:36.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3dc0049d0 0x7fc3dc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:36.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f41487c0 con 0x7fc3dc0049d0 2020-11-20T12:12:36.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc0041b0 con 0x7fc3dc003c20 2020-11-20T12:12:36.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.770+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc00d940 con 0x7fc3f4068230 2020-11-20T12:12:36.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.771+0000 7fc3f8b1e700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3dc003c20 0x7fc3dc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:36.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.771+0000 7fc3f9b20700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3dc0049d0 0x7fc3dc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:36.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.771+0000 7fc3f8b1e700 1 -- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f4068230 msgr2=0x7fc3dc003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:36.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.771+0000 7fc3f8b1e700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f4068230 0x7fc3dc003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:36.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.771+0000 7fc3f8b1e700 1 -- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3dc0049d0 msgr2=0x7fc3dc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:36.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.771+0000 7fc3f8b1e700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3dc0049d0 0x7fc3dc004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:36.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.771+0000 7fc3f8b1e700 1 -- 172.21.15.187:0/3017329121 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc003240 con 0x7fc3dc003c20 2020-11-20T12:12:36.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.771+0000 7fc3f8b1e700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3dc003c20 0x7fc3dc0044e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7fc3e800ba80 tx=0x7fc3e8008040).ready entity=mon.1 client_cookie=357ffb2874850d62 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:36.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e800d960 con 0x7fc3dc003c20 2020-11-20T12:12:36.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e800dac0 con 0x7fc3dc003c20 2020-11-20T12:12:36.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3e7fff700 1 -- 172.21.15.187:0/3017329121 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e800de40 con 0x7fc3dc003c20 2020-11-20T12:12:36.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3fa321700 1 -- 172.21.15.187:0/3017329121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3dc003c20 msgr2=0x7fc3dc0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:36.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3fa321700 1 --2- 172.21.15.187:0/3017329121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3dc003c20 0x7fc3dc0044e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7fc3e800ba80 tx=0x7fc3e8008040).stop 2020-11-20T12:12:36.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3f9b20700 1 -- 172.21.15.187:0/3017329121 reap_dead start 2020-11-20T12:12:36.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3fa321700 1 -- 172.21.15.187:0/3017329121 shutdown_connections 2020-11-20T12:12:36.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3fa321700 1 -- 172.21.15.187:0/3017329121 >> 172.21.15.187:0/3017329121 conn(0x7fc3f40bab30 msgr2=0x7fc3f40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:36.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.772+0000 7fc3fa321700 1 -- 172.21.15.187:0/3017329121 shutdown_connections 2020-11-20T12:12:36.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.773+0000 7fc3fa321700 1 -- 172.21.15.187:0/3017329121 wait complete. 2020-11-20T12:12:36.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.774+0000 7fc3fa321700 1 Processor -- start 2020-11-20T12:12:36.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.774+0000 7fc3fa321700 1 -- start start 2020-11-20T12:12:36.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3fa321700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f41458c0 0x7fc3f40b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:36.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3fa321700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4145ca0 0x7fc3f40b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:36.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3fa321700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f409fa70 0x7fc3f40b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:36.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3fa321700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40a02a0 con 0x7fc3f41458c0 2020-11-20T12:12:36.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3fa321700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40b2bc0 con 0x7fc3f4145ca0 2020-11-20T12:12:36.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3fa321700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4147420 con 0x7fc3f409fa70 2020-11-20T12:12:36.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3f8b1e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4145ca0 0x7fc3f40b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:36.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3f8b1e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4145ca0 0x7fc3f40b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47196/0 (socket says 172.21.15.187:47196) 2020-11-20T12:12:36.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.775+0000 7fc3f8b1e700 1 -- 172.21.15.187:0/4213052492 learned_addr learned my addr 172.21.15.187:0/4213052492 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:36.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f931f700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f41458c0 0x7fc3f40b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:36.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f9b20700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f409fa70 0x7fc3f40b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:36.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f8b1e700 1 -- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f409fa70 msgr2=0x7fc3f40b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:36.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f8b1e700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f409fa70 0x7fc3f40b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:36.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f8b1e700 1 -- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f41458c0 msgr2=0x7fc3f40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:36.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f8b1e700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f41458c0 0x7fc3f40b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:36.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f8b1e700 1 -- 172.21.15.187:0/4213052492 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f41476c0 con 0x7fc3f4145ca0 2020-11-20T12:12:36.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f931f700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f41458c0 0x7fc3f40b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:36.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3f8b1e700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4145ca0 0x7fc3f40b3590 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7fc3e8012000 tx=0x7fc3e8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:36.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3e57fa700 1 -- 172.21.15.187:0/4213052492 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e801d620 con 0x7fc3f4145ca0 2020-11-20T12:12:36.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3e57fa700 1 -- 172.21.15.187:0/4213052492 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e801d780 con 0x7fc3f4145ca0 2020-11-20T12:12:36.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f4147b60 con 0x7fc3f4145ca0 2020-11-20T12:12:36.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3e57fa700 1 -- 172.21.15.187:0/4213052492 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e801db00 con 0x7fc3f4145ca0 2020-11-20T12:12:36.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.776+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f41480d0 con 0x7fc3f4145ca0 2020-11-20T12:12:36.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.778+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f4147850 con 0x7fc3f4145ca0 2020-11-20T12:12:36.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.778+0000 7fc3e57fa700 1 -- 172.21.15.187:0/4213052492 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc3e80253f0 con 0x7fc3f4145ca0 2020-11-20T12:12:36.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.779+0000 7fc3e57fa700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3cc056d20 0x7fc3cc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:36.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.779+0000 7fc3e57fa700 1 -- 172.21.15.187:0/4213052492 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc3e8071ec0 con 0x7fc3f4145ca0 2020-11-20T12:12:36.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.779+0000 7fc3f931f700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3cc056d20 0x7fc3cc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:36.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.780+0000 7fc3f931f700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3cc056d20 0x7fc3cc0591e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7fc3f0002510 tx=0x7fc3f0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:36.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:36.783+0000 7fc3e57fa700 1 -- 172.21.15.187:0/4213052492 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc3e80472b0 con 0x7fc3f4145ca0 2020-11-20T12:12:37.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.008+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3f4061a20 con 0x7fc3f4145ca0 2020-11-20T12:12:37.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.009+0000 7fc3e57fa700 1 -- 172.21.15.187:0/4213052492 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2152 (secure 0 0 0) 0x7fc3e804a8d0 con 0x7fc3f4145ca0 2020-11-20T12:12:37.013 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:37.014 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:34.321010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:37.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.011+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3cc056d20 msgr2=0x7fc3cc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:37.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.011+0000 7fc3fa321700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3cc056d20 0x7fc3cc0591e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7fc3f0002510 tx=0x7fc3f0009040).stop 2020-11-20T12:12:37.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.011+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4145ca0 msgr2=0x7fc3f40b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:37.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.011+0000 7fc3fa321700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4145ca0 0x7fc3f40b3590 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7fc3e8012000 tx=0x7fc3e8008040).stop 2020-11-20T12:12:37.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.012+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 shutdown_connections 2020-11-20T12:12:37.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.012+0000 7fc3fa321700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3cc056d20 0x7fc3cc0591e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:37.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.012+0000 7fc3fa321700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f41458c0 0x7fc3f40b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:37.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.012+0000 7fc3fa321700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3f409fa70 0x7fc3f40b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:37.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.012+0000 7fc3fa321700 1 --2- 172.21.15.187:0/4213052492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4145ca0 0x7fc3f40b3590 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:37.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.012+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 >> 172.21.15.187:0/4213052492 conn(0x7fc3f40bab30 msgr2=0x7fc3f40b9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:37.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.012+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 shutdown_connections 2020-11-20T12:12:37.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:37.012+0000 7fc3fa321700 1 -- 172.21.15.187:0/4213052492 wait complete. 2020-11-20T12:12:37.018 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 68 2020-11-20T12:12:37.036 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2020-11-20T12:12:37.283 INFO:teuthology.orchestra.run.smithi187.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"14","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2020-11-20T12:12:37.284 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2020-11-20T12:12:37.284 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2020-11-20T12:12:39.286 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:39.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.538+0000 7f7f308ec700 1 Processor -- start 2020-11-20T12:12:39.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.538+0000 7f7f308ec700 1 -- start start 2020-11-20T12:12:39.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.539+0000 7f7f308ec700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f2c065190 0x7f7f2c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:39.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.540+0000 7f7f308ec700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 0x7f7f2c143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:39.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.540+0000 7f7f308ec700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c0b13e0 con 0x7f7f2c143c70 2020-11-20T12:12:39.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.540+0000 7f7f308ec700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c14ab30 con 0x7f7f2c065190 2020-11-20T12:12:39.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.540+0000 7f7f308ec700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c062c10 con 0x7f7f2c065f90 2020-11-20T12:12:39.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.540+0000 7f7f2ad9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f2c065190 0x7f7f2c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:39.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.540+0000 7f7f2ad9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f2c065190 0x7f7f2c065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47202/0 (socket says 172.21.15.187:47202) 2020-11-20T12:12:39.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.540+0000 7f7f2ad9d700 1 -- 172.21.15.187:0/67891907 learned_addr learned my addr 172.21.15.187:0/67891907 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:39.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.541+0000 7f7f2259c700 1 --2- 172.21.15.187:0/67891907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 0x7f7f2c143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:39.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.541+0000 7f7f2ad9d700 1 -- 172.21.15.187:0/67891907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 msgr2=0x7f7f2c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:39.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.541+0000 7f7f2ad9d700 1 --2- 172.21.15.187:0/67891907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 0x7f7f2c143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:39.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.541+0000 7f7f2ad9d700 1 -- 172.21.15.187:0/67891907 >> v1:172.21.15.5:6789/0 conn(0x7f7f2c143c70 legacy=0x7f7f2c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:39.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.541+0000 7f7f2ad9d700 1 -- 172.21.15.187:0/67891907 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c0b1640 con 0x7f7f2c065190 2020-11-20T12:12:39.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.542+0000 7f7f2ad9d700 1 --2- 172.21.15.187:0/67891907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c065190 0x7f7f2c065570 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f7f2c14ac70 tx=0x7f7f1c009290).ready entity=mon.1 client_cookie=7f48b19afde7729c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:39.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.542+0000 7f7f2a59c700 1 -- 172.21.15.187:0/67891907 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c00bc50 con 0x7f7f2c065190 2020-11-20T12:12:39.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.542+0000 7f7f2a59c700 1 -- 172.21.15.187:0/67891907 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c00bdb0 con 0x7f7f2c065190 2020-11-20T12:12:39.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.542+0000 7f7f2a59c700 1 -- 172.21.15.187:0/67891907 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c011610 con 0x7f7f2c065190 2020-11-20T12:12:39.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.543+0000 7f7f308ec700 1 -- 172.21.15.187:0/67891907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c065190 msgr2=0x7f7f2c065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:39.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.543+0000 7f7f308ec700 1 --2- 172.21.15.187:0/67891907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c065190 0x7f7f2c065570 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f7f2c14ac70 tx=0x7f7f1c009290).stop 2020-11-20T12:12:39.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.543+0000 7f7f308ec700 1 -- 172.21.15.187:0/67891907 shutdown_connections 2020-11-20T12:12:39.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.543+0000 7f7f308ec700 1 --2- 172.21.15.187:0/67891907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c065190 0x7f7f2c065570 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:39.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.543+0000 7f7f308ec700 1 --2- 172.21.15.187:0/67891907 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 0x7f7f2c143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:39.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.543+0000 7f7f308ec700 1 -- 172.21.15.187:0/67891907 >> 172.21.15.187:0/67891907 conn(0x7f7f2c0bab30 msgr2=0x7f7f2c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:39.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.543+0000 7f7f308ec700 1 -- 172.21.15.187:0/67891907 shutdown_connections 2020-11-20T12:12:39.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.543+0000 7f7f308ec700 1 -- 172.21.15.187:0/67891907 wait complete. 2020-11-20T12:12:39.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.545+0000 7f7f308ec700 1 Processor -- start 2020-11-20T12:12:39.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.545+0000 7f7f308ec700 1 -- start start 2020-11-20T12:12:39.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f308ec700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 0x7f7f2c147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:39.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f308ec700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f2c143c70 0x7f7f2c099180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:39.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f308ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c1587d0 0x7f7f2c147e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:39.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f308ec700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c159150 con 0x7f7f2c143c70 2020-11-20T12:12:39.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f308ec700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c1593f0 con 0x7f7f2c1587d0 2020-11-20T12:12:39.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f308ec700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c099670 con 0x7f7f2c065f90 2020-11-20T12:12:39.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f2b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c1587d0 0x7f7f2c147e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:39.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f2b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c1587d0 0x7f7f2c147e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47208/0 (socket says 172.21.15.187:47208) 2020-11-20T12:12:39.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.546+0000 7f7f2b59e700 1 -- 172.21.15.187:0/1089987357 learned_addr learned my addr 172.21.15.187:0/1089987357 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:39.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f2b59e700 1 -- 172.21.15.187:0/1089987357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 msgr2=0x7f7f2c147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:39.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f2b59e700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 0x7f7f2c147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:39.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f2b59e700 1 -- 172.21.15.187:0/1089987357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f2c143c70 msgr2=0x7f7f2c099180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:39.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f2b59e700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f2c143c70 0x7f7f2c099180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:39.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f2b59e700 1 -- 172.21.15.187:0/1089987357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c148b90 con 0x7f7f2c1587d0 2020-11-20T12:12:39.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f2b59e700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c1587d0 0x7f7f2c147e80 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f7f2400c710 tx=0x7f7f24007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:39.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f237fe700 1 -- 172.21.15.187:0/1089987357 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2400e560 con 0x7f7f2c1587d0 2020-11-20T12:12:39.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f237fe700 1 -- 172.21.15.187:0/1089987357 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2400e6c0 con 0x7f7f2c1587d0 2020-11-20T12:12:39.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f237fe700 1 -- 172.21.15.187:0/1089987357 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2400ea40 con 0x7f7f2c1587d0 2020-11-20T12:12:39.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.547+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c158c50 con 0x7f7f2c1587d0 2020-11-20T12:12:39.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.548+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c1485b0 con 0x7f7f2c1587d0 2020-11-20T12:12:39.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.549+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c148800 con 0x7f7f2c1587d0 2020-11-20T12:12:39.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.550+0000 7f7f237fe700 1 -- 172.21.15.187:0/1089987357 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7f24010030 con 0x7f7f2c1587d0 2020-11-20T12:12:39.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.550+0000 7f7f237fe700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f08056d20 0x7f7f080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:39.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.550+0000 7f7f237fe700 1 -- 172.21.15.187:0/1089987357 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7f2406fd40 con 0x7f7f2c1587d0 2020-11-20T12:12:39.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.551+0000 7f7f2ad9d700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f08056d20 0x7f7f080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:39.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.551+0000 7f7f2ad9d700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f08056d20 0x7f7f080591e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f7f2c0632e0 tx=0x7f7f1c016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:39.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.555+0000 7f7f237fe700 1 -- 172.21.15.187:0/1089987357 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7f24045190 con 0x7f7f2c1587d0 2020-11-20T12:12:39.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.802+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f2c159580 con 0x7f7f2c1587d0 2020-11-20T12:12:39.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.803+0000 7f7f237fe700 1 -- 172.21.15.187:0/1089987357 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2152 (secure 0 0 0) 0x7f7f240487b0 con 0x7f7f2c1587d0 2020-11-20T12:12:39.808 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:39.808 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:34.321010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:39.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.805+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f08056d20 msgr2=0x7f7f080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:39.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.805+0000 7f7f308ec700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f08056d20 0x7f7f080591e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f7f2c0632e0 tx=0x7f7f1c016000).stop 2020-11-20T12:12:39.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.805+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c1587d0 msgr2=0x7f7f2c147e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:39.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.805+0000 7f7f308ec700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c1587d0 0x7f7f2c147e80 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f7f2400c710 tx=0x7f7f24007c20).stop 2020-11-20T12:12:39.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.806+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 shutdown_connections 2020-11-20T12:12:39.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.806+0000 7f7f308ec700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f08056d20 0x7f7f080591e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:39.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.806+0000 7f7f308ec700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2c1587d0 0x7f7f2c147e80 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:39.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.806+0000 7f7f308ec700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f2c065f90 0x7f7f2c147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:39.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.806+0000 7f7f308ec700 1 --2- 172.21.15.187:0/1089987357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f2c143c70 0x7f7f2c099180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:39.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.806+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 >> 172.21.15.187:0/1089987357 conn(0x7f7f2c0bab30 msgr2=0x7f7f2c0b9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:39.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.806+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 shutdown_connections 2020-11-20T12:12:39.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:39.806+0000 7f7f308ec700 1 -- 172.21.15.187:0/1089987357 wait complete. 2020-11-20T12:12:39.815 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 68 2020-11-20T12:12:39.835 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2020-11-20T12:12:40.118 INFO:teuthology.orchestra.run.smithi187.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2020-11-20T12:12:40.119 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2020-11-20T12:12:40.120 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2020-11-20T12:12:42.121 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:42.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.371+0000 7f9a0d655700 1 Processor -- start 2020-11-20T12:12:42.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.371+0000 7f9a0d655700 1 -- start start 2020-11-20T12:12:42.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.372+0000 7f9a0d655700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a08068230 0x7f9a081482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:42.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.372+0000 7f9a0d655700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08144b80 con 0x7f9a080b57b0 2020-11-20T12:12:42.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.372+0000 7f9a0d655700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a081489e0 con 0x7f9a080b5f50 2020-11-20T12:12:42.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.372+0000 7f9a0d655700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a081487c0 con 0x7f9a08068230 2020-11-20T12:12:42.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.373+0000 7f9a0ce54700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9a080b5f50 0x7f9a080b63e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43826/0 (socket says 172.21.15.187:43826) 2020-11-20T12:12:42.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.373+0000 7f9a0ce54700 1 -- 172.21.15.187:0/485347500 learned_addr learned my addr 172.21.15.187:0/485347500 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:42.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.373+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956694038 0 0) 0x7f9a081489e0 con 0x7f9a080b5f50 2020-11-20T12:12:42.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.373+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f4003240 con 0x7f9a080b5f50 2020-11-20T12:12:42.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.373+0000 7f9a077fe700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a08068230 0x7f9a081482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:42.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3107259037 0 0) 0x7f99f4003240 con 0x7f9a080b5f50 2020-11-20T12:12:42.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a08068230 msgr2=0x7f9a081482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:42.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a08068230 0x7f9a081482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 >> v1:172.21.15.5:6789/0 conn(0x7f9a080b57b0 legacy=0x7f9a08062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:42.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08148c80 con 0x7f9a080b5f50 2020-11-20T12:12:42.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f9a000018a0 con 0x7f9a080b5f50 2020-11-20T12:12:42.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 >> v1:172.21.15.187:6789/0 conn(0x7f9a080b5f50 legacy=0x7f9a080b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:42.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99f4003730 0x7f99f4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:42.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99f40044e0 0x7f99f4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:42.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a08068230 0x7f99f4004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:42.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a081487c0 con 0x7f99f4003730 2020-11-20T12:12:42.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f4005390 con 0x7f99f40044e0 2020-11-20T12:12:42.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f40042f0 con 0x7f9a08068230 2020-11-20T12:12:42.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.374+0000 7f9a077fe700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99f40044e0 0x7f99f4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:42.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a0ce54700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a08068230 0x7f99f4004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:42.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a07fff700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99f4003730 0x7f99f4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:42.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a077fe700 1 -- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a08068230 msgr2=0x7f99f4004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:42.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a077fe700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a08068230 0x7f99f4004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a077fe700 1 -- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99f4003730 msgr2=0x7f99f4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:42.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a077fe700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99f4003730 0x7f99f4003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a077fe700 1 -- 172.21.15.187:0/485347500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f4003240 con 0x7f99f40044e0 2020-11-20T12:12:42.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a07fff700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99f4003730 0x7f99f4003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:42.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a077fe700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99f40044e0 0x7f99f4004960 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f99fc00ba80 tx=0x7f99fc008040).ready entity=mon.1 client_cookie=4b950b0a9dbce4a9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:42.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.375+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00d960 con 0x7f99f40044e0 2020-11-20T12:12:42.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc00dac0 con 0x7f99f40044e0 2020-11-20T12:12:42.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a06ffd700 1 -- 172.21.15.187:0/485347500 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00de40 con 0x7f99f40044e0 2020-11-20T12:12:42.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a0d655700 1 -- 172.21.15.187:0/485347500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99f40044e0 msgr2=0x7f99f4004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:42.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a0d655700 1 --2- 172.21.15.187:0/485347500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99f40044e0 0x7f99f4004960 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f99fc00ba80 tx=0x7f99fc008040).stop 2020-11-20T12:12:42.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a0ce54700 1 -- 172.21.15.187:0/485347500 reap_dead start 2020-11-20T12:12:42.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a0d655700 1 -- 172.21.15.187:0/485347500 shutdown_connections 2020-11-20T12:12:42.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a0d655700 1 -- 172.21.15.187:0/485347500 >> 172.21.15.187:0/485347500 conn(0x7f9a080bab30 msgr2=0x7f9a080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:42.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a0d655700 1 -- 172.21.15.187:0/485347500 shutdown_connections 2020-11-20T12:12:42.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.376+0000 7f9a0d655700 1 -- 172.21.15.187:0/485347500 wait complete. 2020-11-20T12:12:42.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.378+0000 7f9a0d655700 1 Processor -- start 2020-11-20T12:12:42.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.378+0000 7f9a0d655700 1 -- start start 2020-11-20T12:12:42.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0d655700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a0809f720 0x7f9a08145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:42.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0d655700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a0809fb00 0x7f9a081470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:42.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0d655700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a081475e0 0x7f9a081543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:42.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0d655700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a08153d40 con 0x7f9a0809fb00 2020-11-20T12:12:42.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0d655700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a081580a0 con 0x7f9a081475e0 2020-11-20T12:12:42.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0d655700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a08153f30 con 0x7f9a0809f720 2020-11-20T12:12:42.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0ce54700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a081475e0 0x7f9a081543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:42.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0ce54700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a081475e0 0x7f9a081543d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47232/0 (socket says 172.21.15.187:47232) 2020-11-20T12:12:42.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.379+0000 7f9a0ce54700 1 -- 172.21.15.187:0/1150147958 learned_addr learned my addr 172.21.15.187:0/1150147958 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:42.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a07fff700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a0809f720 0x7f9a08145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:42.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a077fe700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a0809fb00 0x7f9a081470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:42.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a0ce54700 1 -- 172.21.15.187:0/1150147958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a0809f720 msgr2=0x7f9a08145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:42.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a0ce54700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a0809f720 0x7f9a08145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a0ce54700 1 -- 172.21.15.187:0/1150147958 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a0809fb00 msgr2=0x7f9a081470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:42.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a0ce54700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a0809fb00 0x7f9a081470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a0ce54700 1 -- 172.21.15.187:0/1150147958 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08145470 con 0x7f9a081475e0 2020-11-20T12:12:42.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a0ce54700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a081475e0 0x7f9a081543d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f9a00001390 tx=0x7f9a000017e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:42.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f99e7fff700 1 -- 172.21.15.187:0/1150147958 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000ccd0 con 0x7f9a081475e0 2020-11-20T12:12:42.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f99e7fff700 1 -- 172.21.15.187:0/1150147958 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0000ce30 con 0x7f9a081475e0 2020-11-20T12:12:42.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f99e7fff700 1 -- 172.21.15.187:0/1150147958 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a00014660 con 0x7f9a081475e0 2020-11-20T12:12:42.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.380+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a081548c0 con 0x7f9a081475e0 2020-11-20T12:12:42.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.381+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a08157310 con 0x7f9a081475e0 2020-11-20T12:12:42.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.382+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a08158260 con 0x7f9a081475e0 2020-11-20T12:12:42.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.382+0000 7f99e7fff700 1 -- 172.21.15.187:0/1150147958 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9a000133e0 con 0x7f9a081475e0 2020-11-20T12:12:42.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.383+0000 7f99e7fff700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99e0056d20 0x7f99e00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:42.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.383+0000 7f99e7fff700 1 -- 172.21.15.187:0/1150147958 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9a0006fee0 con 0x7f9a081475e0 2020-11-20T12:12:42.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.383+0000 7f9a07fff700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99e0056d20 0x7f99e00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:42.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.384+0000 7f9a07fff700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99e0056d20 0x7f99e00591e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f9a080a01c0 tx=0x7f99f8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:42.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.387+0000 7f99e7fff700 1 -- 172.21.15.187:0/1150147958 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9a000452d0 con 0x7f9a081475e0 2020-11-20T12:12:42.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.638+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a0809fee0 con 0x7f9a081475e0 2020-11-20T12:12:42.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.639+0000 7f99e7fff700 1 -- 172.21.15.187:0/1150147958 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2152 (secure 0 0 0) 0x7f9a000488f0 con 0x7f9a081475e0 2020-11-20T12:12:42.643 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:42.643 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:34.321010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:42.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99e0056d20 msgr2=0x7f99e00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:42.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99e0056d20 0x7f99e00591e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f9a080a01c0 tx=0x7f99f8009040).stop 2020-11-20T12:12:42.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a081475e0 msgr2=0x7f9a081543d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:42.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a081475e0 0x7f9a081543d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f9a00001390 tx=0x7f9a000017e0).stop 2020-11-20T12:12:42.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 shutdown_connections 2020-11-20T12:12:42.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99e0056d20 0x7f99e00591e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a081475e0 0x7f9a081543d0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a0809f720 0x7f9a08145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 --2- 172.21.15.187:0/1150147958 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a0809fb00 0x7f9a081470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.641+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 >> 172.21.15.187:0/1150147958 conn(0x7f9a080bab30 msgr2=0x7f9a080b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.642+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 shutdown_connections 2020-11-20T12:12:42.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:42.642+0000 7f9a0d655700 1 -- 172.21.15.187:0/1150147958 wait complete. 2020-11-20T12:12:42.648 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 68 2020-11-20T12:12:42.667 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2020-11-20T12:12:42.936 INFO:teuthology.orchestra.run.smithi187.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2020-11-20T12:12:42.937 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2020-11-20T12:12:42.938 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2020-11-20T12:12:44.939 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:45.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.187+0000 7fb93661c700 1 Processor -- start 2020-11-20T12:12:45.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.187+0000 7fb93661c700 1 -- start start 2020-11-20T12:12:45.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.188+0000 7fb93661c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb9300b57b0 0x7fb930068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.188+0000 7fb93661c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb9301482d0 0x7fb930062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.188+0000 7fb93661c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 0x7fb930061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.188+0000 7fb93661c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb930144b80 con 0x7fb9301482d0 2020-11-20T12:12:45.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.188+0000 7fb93661c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9301489e0 con 0x7fb9300b57b0 2020-11-20T12:12:45.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.188+0000 7fb93661c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb930148c80 con 0x7fb930062b50 2020-11-20T12:12:45.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.188+0000 7fb93561a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb9300b57b0 0x7fb930068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.189+0000 7fb93561a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb9300b57b0 0x7fb930068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47236/0 (socket says 172.21.15.187:47236) 2020-11-20T12:12:45.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.189+0000 7fb93561a700 1 -- 172.21.15.187:0/337882057 learned_addr learned my addr 172.21.15.187:0/337882057 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:45.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.189+0000 7fb93561a700 1 -- 172.21.15.187:0/337882057 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 msgr2=0x7fb930061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:45.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.189+0000 7fb93561a700 1 --2- 172.21.15.187:0/337882057 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 0x7fb930061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:45.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.189+0000 7fb93561a700 1 -- 172.21.15.187:0/337882057 >> v2:172.21.15.5:3300/0 conn(0x7fb9301482d0 msgr2=0x7fb930062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:45.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.189+0000 7fb93561a700 1 --2- 172.21.15.187:0/337882057 >> v2:172.21.15.5:3300/0 conn(0x7fb9301482d0 0x7fb930062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:45.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.189+0000 7fb93561a700 1 -- 172.21.15.187:0/337882057 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930148f20 con 0x7fb9300b57b0 2020-11-20T12:12:45.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.189+0000 7fb93561a700 1 --2- 172.21.15.187:0/337882057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9300b57b0 0x7fb930068230 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7fb9300b85e0 tx=0x7fb92c006eb0).ready entity=mon.1 client_cookie=aeb6606a54111eb0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb91ffff700 1 -- 172.21.15.187:0/337882057 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c0098d0 con 0x7fb9300b57b0 2020-11-20T12:12:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb91ffff700 1 -- 172.21.15.187:0/337882057 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92c009a30 con 0x7fb9300b57b0 2020-11-20T12:12:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb91ffff700 1 -- 172.21.15.187:0/337882057 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c009db0 con 0x7fb9300b57b0 2020-11-20T12:12:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb93661c700 1 -- 172.21.15.187:0/337882057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9300b57b0 msgr2=0x7fb930068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb93661c700 1 --2- 172.21.15.187:0/337882057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9300b57b0 0x7fb930068230 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7fb9300b85e0 tx=0x7fb92c006eb0).stop 2020-11-20T12:12:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb93661c700 1 -- 172.21.15.187:0/337882057 shutdown_connections 2020-11-20T12:12:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb93661c700 1 --2- 172.21.15.187:0/337882057 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 0x7fb930061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb93661c700 1 --2- 172.21.15.187:0/337882057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9300b57b0 0x7fb930068230 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb93661c700 1 --2- 172.21.15.187:0/337882057 >> v2:172.21.15.5:3300/0 conn(0x7fb9301482d0 0x7fb930062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:45.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.190+0000 7fb93661c700 1 -- 172.21.15.187:0/337882057 >> 172.21.15.187:0/337882057 conn(0x7fb9300bab30 msgr2=0x7fb9300baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:45.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.191+0000 7fb93661c700 1 -- 172.21.15.187:0/337882057 shutdown_connections 2020-11-20T12:12:45.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.191+0000 7fb93661c700 1 -- 172.21.15.187:0/337882057 wait complete. 2020-11-20T12:12:45.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.192+0000 7fb93661c700 1 Processor -- start 2020-11-20T12:12:45.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.192+0000 7fb93661c700 1 -- start start 2020-11-20T12:12:45.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb93661c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 0x7fb93009fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb93661c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9301482d0 0x7fb930158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb93661c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb930146a50 0x7fb930158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb93661c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb930146fd0 con 0x7fb930146a50 2020-11-20T12:12:45.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb93661c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93015bf70 con 0x7fb9301482d0 2020-11-20T12:12:45.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb93661c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300b4bf0 con 0x7fb930062b50 2020-11-20T12:12:45.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb934e19700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9301482d0 0x7fb930158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb934e19700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9301482d0 0x7fb930158820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47244/0 (socket says 172.21.15.187:47244) 2020-11-20T12:12:45.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.193+0000 7fb934e19700 1 -- 172.21.15.187:0/3791122486 learned_addr learned my addr 172.21.15.187:0/3791122486 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:45.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb93561a700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 0x7fb93009fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb935e1b700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb930146a50 0x7fb930158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb934e19700 1 -- 172.21.15.187:0/3791122486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 msgr2=0x7fb93009fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb934e19700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 0x7fb93009fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb934e19700 1 -- 172.21.15.187:0/3791122486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb930146a50 msgr2=0x7fb930158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb934e19700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb930146a50 0x7fb930158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb934e19700 1 -- 172.21.15.187:0/3791122486 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93009fef0 con 0x7fb9301482d0 2020-11-20T12:12:45.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb934e19700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9301482d0 0x7fb930158820 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7fb920009770 tx=0x7fb920006ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:45.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb91d7fa700 1 -- 172.21.15.187:0/3791122486 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92000d6d0 con 0x7fb9301482d0 2020-11-20T12:12:45.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.194+0000 7fb91d7fa700 1 -- 172.21.15.187:0/3791122486 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb920000e90 con 0x7fb9301482d0 2020-11-20T12:12:45.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.195+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb930092e30 con 0x7fb9301482d0 2020-11-20T12:12:45.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.195+0000 7fb91d7fa700 1 -- 172.21.15.187:0/3791122486 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92000daa0 con 0x7fb9301482d0 2020-11-20T12:12:45.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.195+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93015c5d0 con 0x7fb9301482d0 2020-11-20T12:12:45.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.196+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9300623d0 con 0x7fb9301482d0 2020-11-20T12:12:45.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.196+0000 7fb91d7fa700 1 -- 172.21.15.187:0/3791122486 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb920017030 con 0x7fb9301482d0 2020-11-20T12:12:45.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.197+0000 7fb91d7fa700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb908056d20 0x7fb9080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.197+0000 7fb91d7fa700 1 -- 172.21.15.187:0/3791122486 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb92006f390 con 0x7fb9301482d0 2020-11-20T12:12:45.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.197+0000 7fb93561a700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb908056d20 0x7fb9080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.198+0000 7fb93561a700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb908056d20 0x7fb9080591e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fb930063890 tx=0x7fb92c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:45.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.201+0000 7fb91d7fa700 1 -- 172.21.15.187:0/3791122486 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb920044890 con 0x7fb9301482d0 2020-11-20T12:12:45.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.445+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb930092fc0 con 0x7fb9301482d0 2020-11-20T12:12:45.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.446+0000 7fb91d7fa700 1 -- 172.21.15.187:0/3791122486 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2152 (secure 0 0 0) 0x7fb930092fc0 con 0x7fb9301482d0 2020-11-20T12:12:45.451 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:45.451 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:34.321010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:45.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb908056d20 msgr2=0x7fb9080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb908056d20 0x7fb9080591e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fb930063890 tx=0x7fb92c015000).stop 2020-11-20T12:12:45.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9301482d0 msgr2=0x7fb930158820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9301482d0 0x7fb930158820 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7fb920009770 tx=0x7fb920006ec0).stop 2020-11-20T12:12:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 shutdown_connections 2020-11-20T12:12:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb908056d20 0x7fb9080591e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb930146a50 0x7fb930158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb930062b50 0x7fb93009fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 --2- 172.21.15.187:0/3791122486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9301482d0 0x7fb930158820 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 >> 172.21.15.187:0/3791122486 conn(0x7fb9300bab30 msgr2=0x7fb9300baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 shutdown_connections 2020-11-20T12:12:45.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:45.449+0000 7fb93661c700 1 -- 172.21.15.187:0/3791122486 wait complete. 2020-11-20T12:12:45.456 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 68 2020-11-20T12:12:45.477 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2020-11-20T12:12:45.764 INFO:teuthology.orchestra.run.smithi187.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/dir_2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2020-11-20T12:12:45.765 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/dir_2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:12:45.766 INFO:tasks.cephfs.cephfs_test_case:[('/dir_2', 0)] =?= [('/dir_2', 0)] 2020-11-20T12:12:45.766 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:12:45.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:12:45.767 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:12:45.903 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:12:45.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:45.904 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:12:45.959 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2020-11-20T12:12:46.282 INFO:teuthology.orchestra.run.smithi005.stdout:{"message":"","return_code":0} 2020-11-20T12:12:46.283 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:12:46.284 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json cache drop 2020-11-20T12:12:46.564 INFO:teuthology.orchestra.run.smithi005.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":285,"bytes":56946}},"duration":0.01908841} 2020-11-20T12:12:46.565 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 56946, "items": 285 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.01908841, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2020-11-20T12:12:46.566 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:12:46.566 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:12:46.566 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:12:46.567 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:12:46.567 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:12:46.567 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:12:46.568 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:12:46.568 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:12:46.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:46.569 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:12:46.632 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:12:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:12:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:12:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:12:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:46.633 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:12:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:46.634 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:12:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:46.634 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:12:46.635 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:46.635 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:12:46.635 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:46.635 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:12:46.635 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42272sec preferred_lft 42272sec 2020-11-20T12:12:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:12:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:46.636 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:12:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:12:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:12:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:12:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:46.637 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:12:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:12:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:12:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:46.638 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:12:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:12:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:12:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:12:46.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:46.640 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:12:46.752 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:12:46.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:46.758 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:12:46.822 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:12:46.822 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:12:46.823 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:12:46.823 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:12:46.823 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:12:46.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:46.825 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:12:46.825 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:12:46.825 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:12:46.826 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:12:46.826 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:12:46.827 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:12:47.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:47.084 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:12:47.138 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:12:47.138 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:12:47.139 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:12:47.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:47.140 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:12:47.247 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:12:47.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.489+0000 7f36ca3b3700 1 Processor -- start 2020-11-20T12:12:47.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.489+0000 7f36ca3b3700 1 -- start start 2020-11-20T12:12:47.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.490+0000 7f36ca3b3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f36c4065190 0x7f36c4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.490+0000 7f36ca3b3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f36c4065f90 0x7f36c4143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.490+0000 7f36ca3b3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c40b13e0 con 0x7f36c4143c70 2020-11-20T12:12:47.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.490+0000 7f36ca3b3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c414ab30 con 0x7f36c4065f90 2020-11-20T12:12:47.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.490+0000 7f36ca3b3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c4062c10 con 0x7f36c4065190 2020-11-20T12:12:47.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.491+0000 7f36c9bb2700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f36c4143c70 0x7f36c409ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43856/0 (socket says 172.21.15.187:43856) 2020-11-20T12:12:47.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.491+0000 7f36c9bb2700 1 -- 172.21.15.187:0/2238067955 learned_addr learned my addr 172.21.15.187:0/2238067955 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:47.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.491+0000 7f36c93b1700 1 --2- 172.21.15.187:0/2238067955 >> v2:172.21.15.187:3300/0 conn(0x7f36c4065190 0x7f36c4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.491+0000 7f36c8bb0700 1 --2- 172.21.15.187:0/2238067955 >> v2:172.21.15.5:3300/0 conn(0x7f36c4065f90 0x7f36c4143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.491+0000 7f36b3fff700 1 -- 172.21.15.187:0/2238067955 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191133077 0 0) 0x7f36c40b13e0 con 0x7f36c4143c70 2020-11-20T12:12:47.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.491+0000 7f36b3fff700 1 -- 172.21.15.187:0/2238067955 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ac003240 con 0x7f36c4143c70 2020-11-20T12:12:47.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36c93b1700 1 -- 172.21.15.187:0/2238067955 >> v2:172.21.15.5:3300/0 conn(0x7f36c4065f90 msgr2=0x7f36c4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:47.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36c93b1700 1 --2- 172.21.15.187:0/2238067955 >> v2:172.21.15.5:3300/0 conn(0x7f36c4065f90 0x7f36c4143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36c93b1700 1 -- 172.21.15.187:0/2238067955 >> v1:172.21.15.187:6789/0 conn(0x7f36c4143c70 legacy=0x7f36c409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36c93b1700 1 -- 172.21.15.187:0/2238067955 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c40b1640 con 0x7f36c4065190 2020-11-20T12:12:47.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36c8bb0700 1 --2- 172.21.15.187:0/2238067955 >> v2:172.21.15.5:3300/0 conn(0x7f36c4065f90 0x7f36c4143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:47.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36c93b1700 1 --2- 172.21.15.187:0/2238067955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4065190 0x7f36c4065570 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f36c405c8b0 tx=0x7f36c0006df0).ready entity=mon.1 client_cookie=4ab6fcc93c43d0e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:47.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36b3fff700 1 -- 172.21.15.187:0/2238067955 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0009840 con 0x7f36c4065190 2020-11-20T12:12:47.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36b3fff700 1 -- 172.21.15.187:0/2238067955 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c00099a0 con 0x7f36c4065190 2020-11-20T12:12:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.492+0000 7f36b3fff700 1 -- 172.21.15.187:0/2238067955 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0009d20 con 0x7f36c4065190 2020-11-20T12:12:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.493+0000 7f36ca3b3700 1 -- 172.21.15.187:0/2238067955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4065190 msgr2=0x7f36c4065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.493+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/2238067955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4065190 0x7f36c4065570 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f36c405c8b0 tx=0x7f36c0006df0).stop 2020-11-20T12:12:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.493+0000 7f36ca3b3700 1 -- 172.21.15.187:0/2238067955 shutdown_connections 2020-11-20T12:12:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.493+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/2238067955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4065190 0x7f36c4065570 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.493+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/2238067955 >> v2:172.21.15.5:3300/0 conn(0x7f36c4065f90 0x7f36c4143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.493+0000 7f36ca3b3700 1 -- 172.21.15.187:0/2238067955 >> 172.21.15.187:0/2238067955 conn(0x7f36c40bab30 msgr2=0x7f36c40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:47.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.493+0000 7f36ca3b3700 1 -- 172.21.15.187:0/2238067955 shutdown_connections 2020-11-20T12:12:47.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.493+0000 7f36ca3b3700 1 -- 172.21.15.187:0/2238067955 wait complete. 2020-11-20T12:12:47.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.495+0000 7f36ca3b3700 1 Processor -- start 2020-11-20T12:12:47.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.495+0000 7f36ca3b3700 1 -- start start 2020-11-20T12:12:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.496+0000 7f36ca3b3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36c4065190 0x7f36c4148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.496+0000 7f36ca3b3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36c4065f90 0x7f36c41498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.496+0000 7f36ca3b3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4143c70 0x7f36c41475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.496+0000 7f36ca3b3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4148ed0 con 0x7f36c4065f90 2020-11-20T12:12:47.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.496+0000 7f36ca3b3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4149170 con 0x7f36c4143c70 2020-11-20T12:12:47.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.496+0000 7f36ca3b3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4149410 con 0x7f36c4065190 2020-11-20T12:12:47.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4143c70 0x7f36c41475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4143c70 0x7f36c41475c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47260/0 (socket says 172.21.15.187:47260) 2020-11-20T12:12:47.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 -- 172.21.15.187:0/1710862434 learned_addr learned my addr 172.21.15.187:0/1710862434 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:47.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c8bb0700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36c4065f90 0x7f36c41498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c93b1700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36c4065190 0x7f36c4148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 -- 172.21.15.187:0/1710862434 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36c4065190 msgr2=0x7f36c4148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36c4065190 0x7f36c4148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 -- 172.21.15.187:0/1710862434 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36c4065f90 msgr2=0x7f36c41498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36c4065f90 0x7f36c41498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 -- 172.21.15.187:0/1710862434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c4148640 con 0x7f36c4143c70 2020-11-20T12:12:47.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.497+0000 7f36c9bb2700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4143c70 0x7f36c41475c0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f36bc001420 tx=0x7f36bc007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:47.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.498+0000 7f36b17fa700 1 -- 172.21.15.187:0/1710862434 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc005b80 con 0x7f36c4143c70 2020-11-20T12:12:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.498+0000 7f36b17fa700 1 -- 172.21.15.187:0/1710862434 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc005ce0 con 0x7f36c4143c70 2020-11-20T12:12:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.498+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4142770 con 0x7f36c4143c70 2020-11-20T12:12:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.498+0000 7f36b17fa700 1 -- 172.21.15.187:0/1710862434 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc007320 con 0x7f36c4143c70 2020-11-20T12:12:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.498+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c4142c90 con 0x7f36c4143c70 2020-11-20T12:12:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.499+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c41495d0 con 0x7f36c4143c70 2020-11-20T12:12:47.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.500+0000 7f36b17fa700 1 -- 172.21.15.187:0/1710862434 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f36bc010030 con 0x7f36c4143c70 2020-11-20T12:12:47.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.500+0000 7f36b17fa700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f369c056d20 0x7f369c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.500+0000 7f36b17fa700 1 -- 172.21.15.187:0/1710862434 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f36bc06fe80 con 0x7f36c4143c70 2020-11-20T12:12:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.501+0000 7f36c93b1700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f369c056d20 0x7f369c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.501+0000 7f36c93b1700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f369c056d20 0x7f369c0591e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f36c0006e30 tx=0x7f36c0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.504+0000 7f36b17fa700 1 -- 172.21.15.187:0/1710862434 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f36bc0452d0 con 0x7f36c4143c70 2020-11-20T12:12:47.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.696+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36c4061a20 con 0x7f36c4143c70 2020-11-20T12:12:47.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.698+0000 7f36b17fa700 1 -- 172.21.15.187:0/1710862434 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f36bc0488f0 con 0x7f36c4143c70 2020-11-20T12:12:47.701 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:47.701 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":81,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:12:11.318992+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:12:07.846671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2020-11-20T12:12:09.337957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:12:47.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.700+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f369c056d20 msgr2=0x7f369c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.700+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f369c056d20 0x7f369c0591e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f36c0006e30 tx=0x7f36c0015000).stop 2020-11-20T12:12:47.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.700+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4143c70 msgr2=0x7f36c41475c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.700+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4143c70 0x7f36c41475c0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f36bc001420 tx=0x7f36bc007e20).stop 2020-11-20T12:12:47.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.701+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 shutdown_connections 2020-11-20T12:12:47.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.701+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f369c056d20 0x7f369c0591e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.701+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36c4143c70 0x7f36c41475c0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.701+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36c4065190 0x7f36c4148930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.701+0000 7f36ca3b3700 1 --2- 172.21.15.187:0/1710862434 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36c4065f90 0x7f36c41498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.701+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 >> 172.21.15.187:0/1710862434 conn(0x7f36c40bab30 msgr2=0x7f36c409ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:47.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.701+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 shutdown_connections 2020-11-20T12:12:47.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.701+0000 7f36ca3b3700 1 -- 172.21.15.187:0/1710862434 wait complete. 2020-11-20T12:12:47.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:12:07.846671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:12:09.337957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:12:47.728 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:12:47.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.975+0000 7f876ed14700 1 Processor -- start 2020-11-20T12:12:47.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.976+0000 7f876ed14700 1 -- start start 2020-11-20T12:12:47.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.977+0000 7f876ed14700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87680bf800 0x7f876814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.977+0000 7f876ed14700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876814d110 con 0x7f876809f560 2020-11-20T12:12:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.977+0000 7f876ed14700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876814cbd0 con 0x7f87680bfbe0 2020-11-20T12:12:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.977+0000 7f876ed14700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876814ce70 con 0x7f87680bf800 2020-11-20T12:12:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.977+0000 7f876d511700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f87680bfbe0 0x7f87681480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43872/0 (socket says 172.21.15.187:43872) 2020-11-20T12:12:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.977+0000 7f876d511700 1 -- 172.21.15.187:0/109671266 learned_addr learned my addr 172.21.15.187:0/109671266 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.978+0000 7f876dd12700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87680bf800 0x7f876814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.978+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288746097 0 0) 0x7f876814cbd0 con 0x7f87680bfbe0 2020-11-20T12:12:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.978+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8750003240 con 0x7f87680bfbe0 2020-11-20T12:12:47.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.978+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3040343365 0 0) 0x7f8750003240 con 0x7f87680bfbe0 2020-11-20T12:12:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.978+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87680bf800 msgr2=0x7f876814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.978+0000 7f876cd10700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87680bf800 0x7f876814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 >> v1:172.21.15.5:6789/0 conn(0x7f876809f560 legacy=0x7f87680a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680a4cc0 con 0x7f87680bfbe0 2020-11-20T12:12:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f87580014e0 con 0x7f87680bfbe0 2020-11-20T12:12:47.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 >> v1:172.21.15.187:6789/0 conn(0x7f87680bfbe0 legacy=0x7f87681480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8750003730 0x7f8750003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87500044e0 0x7f8750004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87680bf800 0x7f8750004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876814ce70 con 0x7f87500044e0 2020-11-20T12:12:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8750005390 con 0x7f8750003730 2020-11-20T12:12:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87500042f0 con 0x7f87680bf800 2020-11-20T12:12:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.979+0000 7f876dd12700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8750003730 0x7f8750003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876d511700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87500044e0 0x7f8750004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876e513700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87680bf800 0x7f8750004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876dd12700 1 -- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87680bf800 msgr2=0x7f8750004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876dd12700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87680bf800 0x7f8750004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876dd12700 1 -- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87500044e0 msgr2=0x7f8750004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876dd12700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87500044e0 0x7f8750004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876dd12700 1 -- 172.21.15.187:0/109671266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8750003240 con 0x7f8750003730 2020-11-20T12:12:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876d511700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87500044e0 0x7f8750004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876dd12700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8750003730 0x7f8750003b50 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f8764009d80 tx=0x7f8764008040).ready entity=mon.1 client_cookie=70bd989e9be8f534 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876400c070 con 0x7f8750003730 2020-11-20T12:12:47.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876400ea20 con 0x7f8750003730 2020-11-20T12:12:47.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.980+0000 7f876cd10700 1 -- 172.21.15.187:0/109671266 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876400ede0 con 0x7f8750003730 2020-11-20T12:12:47.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.981+0000 7f876ed14700 1 -- 172.21.15.187:0/109671266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8750003730 msgr2=0x7f8750003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.981+0000 7f876ed14700 1 --2- 172.21.15.187:0/109671266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8750003730 0x7f8750003b50 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f8764009d80 tx=0x7f8764008040).stop 2020-11-20T12:12:47.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.981+0000 7f876e513700 1 -- 172.21.15.187:0/109671266 reap_dead start 2020-11-20T12:12:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.981+0000 7f876ed14700 1 -- 172.21.15.187:0/109671266 shutdown_connections 2020-11-20T12:12:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.981+0000 7f876ed14700 1 -- 172.21.15.187:0/109671266 >> 172.21.15.187:0/109671266 conn(0x7f8768068370 msgr2=0x7f87680bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.981+0000 7f876ed14700 1 -- 172.21.15.187:0/109671266 shutdown_connections 2020-11-20T12:12:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.981+0000 7f876ed14700 1 -- 172.21.15.187:0/109671266 wait complete. 2020-11-20T12:12:47.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.983+0000 7f876ed14700 1 Processor -- start 2020-11-20T12:12:47.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.983+0000 7f876ed14700 1 -- start start 2020-11-20T12:12:47.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876ed14700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8768146c80 0x7f87680b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876ed14700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87680b3e90 0x7f87681411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876ed14700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8768141690 0x7f8768147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:47.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876ed14700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680b76c0 con 0x7f87680b3e90 2020-11-20T12:12:47.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876ed14700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680b4b80 con 0x7f8768146c80 2020-11-20T12:12:47.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876ed14700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87680b4e20 con 0x7f8768141690 2020-11-20T12:12:47.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876dd12700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8768146c80 0x7f87680b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876dd12700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8768146c80 0x7f87680b71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47276/0 (socket says 172.21.15.187:47276) 2020-11-20T12:12:47.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876dd12700 1 -- 172.21.15.187:0/3750535050 learned_addr learned my addr 172.21.15.187:0/3750535050 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:47.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876dd12700 1 -- 172.21.15.187:0/3750535050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8768141690 msgr2=0x7f8768147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876dd12700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8768141690 0x7f8768147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:47.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876dd12700 1 -- 172.21.15.187:0/3750535050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87680b3e90 msgr2=0x7f87681411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:47.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876d511700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87680b3e90 0x7f87681411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:47.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876dd12700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87680b3e90 0x7f87681411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:47.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.984+0000 7f876dd12700 1 -- 172.21.15.187:0/3750535050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680b50c0 con 0x7f8768146c80 2020-11-20T12:12:47.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.985+0000 7f876dd12700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8768146c80 0x7f87680b71d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f8764009d50 tx=0x7f8764008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:47.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.985+0000 7f8755ffb700 1 -- 172.21.15.187:0/3750535050 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876401e630 con 0x7f8768146c80 2020-11-20T12:12:47.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.985+0000 7f8755ffb700 1 -- 172.21.15.187:0/3750535050 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876401e790 con 0x7f8768146c80 2020-11-20T12:12:47.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.985+0000 7f8755ffb700 1 -- 172.21.15.187:0/3750535050 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876401eb30 con 0x7f8768146c80 2020-11-20T12:12:48.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.985+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8768147550 con 0x7f8768146c80 2020-11-20T12:12:48.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.985+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8768143c20 con 0x7f8768146c80 2020-11-20T12:12:48.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.986+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8768080050 con 0x7f8768146c80 2020-11-20T12:12:48.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.987+0000 7f8755ffb700 1 -- 172.21.15.187:0/3750535050 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f876402d070 con 0x7f8768146c80 2020-11-20T12:12:48.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.987+0000 7f8755ffb700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8740056d20 0x7f87400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.988+0000 7f8755ffb700 1 -- 172.21.15.187:0/3750535050 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8764071b50 con 0x7f8768146c80 2020-11-20T12:12:48.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.988+0000 7f876d511700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8740056d20 0x7f87400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.988+0000 7f876d511700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8740056d20 0x7f87400591e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f8758000dd0 tx=0x7f8758009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:48.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:47.991+0000 7f8755ffb700 1 -- 172.21.15.187:0/3750535050 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8764047020 con 0x7f8768146c80 2020-11-20T12:12:48.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.177+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f87680b6fd0 con 0x7f8768146c80 2020-11-20T12:12:48.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.179+0000 7f8755ffb700 1 -- 172.21.15.187:0/3750535050 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8764022070 con 0x7f8768146c80 2020-11-20T12:12:48.182 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:12:48.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8740056d20 msgr2=0x7f87400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8740056d20 0x7f87400591e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f8758000dd0 tx=0x7f8758009040).stop 2020-11-20T12:12:48.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8768146c80 msgr2=0x7f87680b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8768146c80 0x7f87680b71d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f8764009d50 tx=0x7f8764008040).stop 2020-11-20T12:12:48.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 shutdown_connections 2020-11-20T12:12:48.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8740056d20 0x7f87400591e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8768141690 0x7f8768147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:48.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8768146c80 0x7f87680b71d0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 --2- 172.21.15.187:0/3750535050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87680b3e90 0x7f87681411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 >> 172.21.15.187:0/3750535050 conn(0x7f8768068370 msgr2=0x7f87680be770 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:48.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.181+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 shutdown_connections 2020-11-20T12:12:48.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.182+0000 7f876ed14700 1 -- 172.21.15.187:0/3750535050 wait complete. 2020-11-20T12:12:48.205 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:12:48.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.447+0000 7fca09f02700 1 Processor -- start 2020-11-20T12:12:48.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.447+0000 7fca09f02700 1 -- start start 2020-11-20T12:12:48.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.449+0000 7fca09f02700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fca04065f90 0x7fca04065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.449+0000 7fca09f02700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca0409ca90 0x7fca0409cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.449+0000 7fca09f02700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca040b13e0 con 0x7fca04065190 2020-11-20T12:12:48.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.449+0000 7fca09f02700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0414ab30 con 0x7fca04065f90 2020-11-20T12:12:48.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.449+0000 7fca09f02700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca04062c10 con 0x7fca0409ca90 2020-11-20T12:12:48.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.449+0000 7fc9fbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fca04065f90 0x7fca04065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.449+0000 7fca08f00700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fca04065190 0x7fca04143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43890/0 (socket says 172.21.15.187:43890) 2020-11-20T12:12:48.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.449+0000 7fca08f00700 1 -- 172.21.15.187:0/3197962748 learned_addr learned my addr 172.21.15.187:0/3197962748 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:48.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fca09701700 1 --2- 172.21.15.187:0/3197962748 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca0409ca90 0x7fca0409cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fc9fb7fe700 1 -- 172.21.15.187:0/3197962748 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835060900 0 0) 0x7fca040b13e0 con 0x7fca04065190 2020-11-20T12:12:48.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fc9fb7fe700 1 -- 172.21.15.187:0/3197962748 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f0003240 con 0x7fca04065190 2020-11-20T12:12:48.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fc9fbfff700 1 -- 172.21.15.187:0/3197962748 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca0409ca90 msgr2=0x7fca0409cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:48.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fc9fbfff700 1 --2- 172.21.15.187:0/3197962748 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca0409ca90 0x7fca0409cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fc9fbfff700 1 -- 172.21.15.187:0/3197962748 >> v1:172.21.15.187:6789/0 conn(0x7fca04065190 legacy=0x7fca04143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fc9fbfff700 1 -- 172.21.15.187:0/3197962748 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca040b1640 con 0x7fca04065f90 2020-11-20T12:12:48.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fc9fbfff700 1 --2- 172.21.15.187:0/3197962748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04065570 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7fc9f400b9a0 tx=0x7fc9f4006df0).ready entity=mon.1 client_cookie=71015c9c5cca96e6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:48.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.450+0000 7fc9fb7fe700 1 -- 172.21.15.187:0/3197962748 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4009840 con 0x7fca04065f90 2020-11-20T12:12:48.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fc9fb7fe700 1 -- 172.21.15.187:0/3197962748 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f40099a0 con 0x7fca04065f90 2020-11-20T12:12:48.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fc9fb7fe700 1 -- 172.21.15.187:0/3197962748 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4009d20 con 0x7fca04065f90 2020-11-20T12:12:48.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fca09f02700 1 -- 172.21.15.187:0/3197962748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 msgr2=0x7fca04065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fca09f02700 1 --2- 172.21.15.187:0/3197962748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04065570 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7fc9f400b9a0 tx=0x7fc9f4006df0).stop 2020-11-20T12:12:48.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fca09f02700 1 -- 172.21.15.187:0/3197962748 shutdown_connections 2020-11-20T12:12:48.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fca09f02700 1 --2- 172.21.15.187:0/3197962748 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca0409ca90 0x7fca0409cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fca09f02700 1 --2- 172.21.15.187:0/3197962748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04065570 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fca09f02700 1 -- 172.21.15.187:0/3197962748 >> 172.21.15.187:0/3197962748 conn(0x7fca040bab30 msgr2=0x7fca040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:48.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fca09f02700 1 -- 172.21.15.187:0/3197962748 shutdown_connections 2020-11-20T12:12:48.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.451+0000 7fca09f02700 1 -- 172.21.15.187:0/3197962748 wait complete. 2020-11-20T12:12:48.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.453+0000 7fca09f02700 1 Processor -- start 2020-11-20T12:12:48.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.453+0000 7fca09f02700 1 -- start start 2020-11-20T12:12:48.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fca09f02700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca04065190 0x7fca040b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fca09f02700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fca09f02700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca0409ca90 0x7fca04143880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fca09f02700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca04149150 con 0x7fca0409ca90 2020-11-20T12:12:48.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fca09f02700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca041493f0 con 0x7fca04065f90 2020-11-20T12:12:48.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fca09f02700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca04149690 con 0x7fca04065190 2020-11-20T12:12:48.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fc9fbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fc9fbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04143390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47292/0 (socket says 172.21.15.187:47292) 2020-11-20T12:12:48.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.454+0000 7fc9fbfff700 1 -- 172.21.15.187:0/3323945118 learned_addr learned my addr 172.21.15.187:0/3323945118 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:48.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fca08f00700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca04065190 0x7fca040b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fca09701700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca0409ca90 0x7fca04143880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9fbfff700 1 -- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca04065190 msgr2=0x7fca040b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9fbfff700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca04065190 0x7fca040b7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9fbfff700 1 -- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca0409ca90 msgr2=0x7fca04143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9fbfff700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca0409ca90 0x7fca04143880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9fbfff700 1 -- 172.21.15.187:0/3323945118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca04148e40 con 0x7fca04065f90 2020-11-20T12:12:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fca08f00700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca04065190 0x7fca040b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fca09701700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca0409ca90 0x7fca04143880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9fbfff700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04143390 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7fc9f4006e30 tx=0x7fc9f4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9f8ff9700 1 -- 172.21.15.187:0/3323945118 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f401e460 con 0x7fca04065f90 2020-11-20T12:12:48.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9f8ff9700 1 -- 172.21.15.187:0/3323945118 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f401e5c0 con 0x7fca04065f90 2020-11-20T12:12:48.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca040b7a40 con 0x7fca04065f90 2020-11-20T12:12:48.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fc9f8ff9700 1 -- 172.21.15.187:0/3323945118 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f401e940 con 0x7fca04065f90 2020-11-20T12:12:48.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.455+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca040b4780 con 0x7fca04065f90 2020-11-20T12:12:48.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.456+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca040b4a30 con 0x7fca04065f90 2020-11-20T12:12:48.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.457+0000 7fc9f8ff9700 1 -- 172.21.15.187:0/3323945118 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc9f4025030 con 0x7fca04065f90 2020-11-20T12:12:48.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.458+0000 7fc9f8ff9700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9dc056d50 0x7fc9dc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.458+0000 7fc9f8ff9700 1 -- 172.21.15.187:0/3323945118 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc9f40718e0 con 0x7fca04065f90 2020-11-20T12:12:48.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.458+0000 7fca08f00700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9dc056d50 0x7fc9dc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.459+0000 7fca08f00700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9dc056d50 0x7fc9dc059210 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fca00005920 tx=0x7fca00009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:48.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.461+0000 7fc9f8ff9700 1 -- 172.21.15.187:0/3323945118 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc9f4046e60 con 0x7fca04065f90 2020-11-20T12:12:48.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.647+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fca04061a20 con 0x7fca04065f90 2020-11-20T12:12:48.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.648+0000 7fc9f8ff9700 1 -- 172.21.15.187:0/3323945118 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc9f4023020 con 0x7fca04065f90 2020-11-20T12:12:48.651 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:12:48.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9dc056d50 msgr2=0x7fc9dc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9dc056d50 0x7fc9dc059210 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fca00005920 tx=0x7fca00009040).stop 2020-11-20T12:12:48.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 msgr2=0x7fca04143390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04143390 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7fc9f4006e30 tx=0x7fc9f4015000).stop 2020-11-20T12:12:48.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 shutdown_connections 2020-11-20T12:12:48.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9dc056d50 0x7fc9dc059210 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca0409ca90 0x7fca04143880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca04065190 0x7fca040b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 --2- 172.21.15.187:0/3323945118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca04065f90 0x7fca04143390 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.650+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 >> 172.21.15.187:0/3323945118 conn(0x7fca040bab30 msgr2=0x7fca040aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.651+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 shutdown_connections 2020-11-20T12:12:48.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.651+0000 7fca09f02700 1 -- 172.21.15.187:0/3323945118 wait complete. 2020-11-20T12:12:48.677 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:12:48.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.927+0000 7fc67d3fd700 1 Processor -- start 2020-11-20T12:12:48.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.928+0000 7fc67d3fd700 1 -- start start 2020-11-20T12:12:48.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.929+0000 7fc67d3fd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc678065f90 0x7fc678065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.929+0000 7fc67d3fd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6780b13e0 con 0x7fc67809ca90 2020-11-20T12:12:48.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.929+0000 7fc67d3fd700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc678062c10 con 0x7fc678065190 2020-11-20T12:12:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.929+0000 7fc67d3fd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc67814ab30 con 0x7fc678065f90 2020-11-20T12:12:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.929+0000 7fc677fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc678065190 0x7fc678143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:43904/0 (socket says 172.21.15.187:43904) 2020-11-20T12:12:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.929+0000 7fc677fff700 1 -- 172.21.15.187:0/3995229687 learned_addr learned my addr 172.21.15.187:0/3995229687 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:48.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.930+0000 7fc66f7fe700 1 --2- 172.21.15.187:0/3995229687 >> v2:172.21.15.5:3300/0 conn(0x7fc678065f90 0x7fc678065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.930+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262145133 0 0) 0x7fc678062c10 con 0x7fc678065190 2020-11-20T12:12:48.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.930+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc658003240 con 0x7fc678065190 2020-11-20T12:12:48.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.930+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2091903550 0 0) 0x7fc658003240 con 0x7fc678065190 2020-11-20T12:12:48.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.930+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 >> v2:172.21.15.5:3300/0 conn(0x7fc678065f90 msgr2=0x7fc678065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.930+0000 7fc6777fe700 1 --2- 172.21.15.187:0/3995229687 >> v2:172.21.15.5:3300/0 conn(0x7fc678065f90 0x7fc678065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.930+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 >> v1:172.21.15.5:6789/0 conn(0x7fc67809ca90 legacy=0x7fc67809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.930+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780b1640 con 0x7fc678065190 2020-11-20T12:12:48.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fc6680014e0 con 0x7fc678065190 2020-11-20T12:12:48.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 >> v1:172.21.15.187:6789/0 conn(0x7fc678065190 legacy=0x7fc678143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc6777fe700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc658003800 0x7fc658003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc6777fe700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6580044e0 0x7fc658004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc6777fe700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc658004e50 0x7fc65800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67814ab30 con 0x7fc658004e50 2020-11-20T12:12:48.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc658005300 con 0x7fc6580044e0 2020-11-20T12:12:48.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65800bc20 con 0x7fc658003800 2020-11-20T12:12:48.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.931+0000 7fc66f7fe700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6580044e0 0x7fc658004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc67cbfc700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc658004e50 0x7fc65800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc677fff700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc658003800 0x7fc658003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc66f7fe700 1 -- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc658003800 msgr2=0x7fc658003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:48.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc66f7fe700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc658003800 0x7fc658003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc66f7fe700 1 -- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc658004e50 msgr2=0x7fc65800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:48.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc66f7fe700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc658004e50 0x7fc65800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc66f7fe700 1 -- 172.21.15.187:0/3995229687 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658003240 con 0x7fc6580044e0 2020-11-20T12:12:48.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc67cbfc700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc658004e50 0x7fc65800b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:48.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc677fff700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc658003800 0x7fc658003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:48.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc67cbfc700 1 -- 172.21.15.187:0/3995229687 reap_dead start 2020-11-20T12:12:48.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc66f7fe700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6580044e0 0x7fc658004960 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7fc66000bc00 tx=0x7fc660006df0).ready entity=mon.1 client_cookie=7d1a20d1085a37fb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:48.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66000b330 con 0x7fc6580044e0 2020-11-20T12:12:48.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66000eb40 con 0x7fc6580044e0 2020-11-20T12:12:48.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.932+0000 7fc6777fe700 1 -- 172.21.15.187:0/3995229687 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66000eec0 con 0x7fc6580044e0 2020-11-20T12:12:48.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.933+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3995229687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6580044e0 msgr2=0x7fc658004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.933+0000 7fc67d3fd700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6580044e0 0x7fc658004960 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7fc66000bc00 tx=0x7fc660006df0).stop 2020-11-20T12:12:48.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.933+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3995229687 shutdown_connections 2020-11-20T12:12:48.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.933+0000 7fc67d3fd700 1 --2- 172.21.15.187:0/3995229687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6580044e0 0x7fc658004960 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.933+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3995229687 >> 172.21.15.187:0/3995229687 conn(0x7fc6780bab30 msgr2=0x7fc6780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:48.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.933+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3995229687 shutdown_connections 2020-11-20T12:12:48.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.933+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3995229687 wait complete. 2020-11-20T12:12:48.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.935+0000 7fc67d3fd700 1 Processor -- start 2020-11-20T12:12:48.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.935+0000 7fc67d3fd700 1 -- start start 2020-11-20T12:12:48.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.936+0000 7fc67d3fd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6580044e0 0x7fc678142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.936+0000 7fc67d3fd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6780b3f70 0x7fc678159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67d3fd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc678159d30 0x7fc67809e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67d3fd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6781492c0 con 0x7fc6580044e0 2020-11-20T12:12:48.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67d3fd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780a0410 con 0x7fc678159d30 2020-11-20T12:12:48.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67d3fd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780a06b0 con 0x7fc6780b3f70 2020-11-20T12:12:48.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67cbfc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc678159d30 0x7fc67809e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67cbfc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc678159d30 0x7fc67809e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47312/0 (socket says 172.21.15.187:47312) 2020-11-20T12:12:48.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67cbfc700 1 -- 172.21.15.187:0/3386878527 learned_addr learned my addr 172.21.15.187:0/3386878527 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:48.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc677fff700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6580044e0 0x7fc678142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67cbfc700 1 -- 172.21.15.187:0/3386878527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6780b3f70 msgr2=0x7fc678159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.937+0000 7fc67cbfc700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6780b3f70 0x7fc678159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:48.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc67cbfc700 1 -- 172.21.15.187:0/3386878527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6580044e0 msgr2=0x7fc678142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:48.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc67cbfc700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6580044e0 0x7fc678142ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:48.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc67cbfc700 1 -- 172.21.15.187:0/3386878527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780a0950 con 0x7fc678159d30 2020-11-20T12:12:48.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc677fff700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6580044e0 0x7fc678142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:48.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc67cbfc700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc678159d30 0x7fc67809e4f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7fc67809ce70 tx=0x7fc670009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:48.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc674ff9700 1 -- 172.21.15.187:0/3386878527 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67000f5d0 con 0x7fc678159d30 2020-11-20T12:12:48.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc674ff9700 1 -- 172.21.15.187:0/3386878527 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67000f730 con 0x7fc678159d30 2020-11-20T12:12:48.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc674ff9700 1 -- 172.21.15.187:0/3386878527 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67000fad0 con 0x7fc678159d30 2020-11-20T12:12:48.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6781433e0 con 0x7fc678159d30 2020-11-20T12:12:48.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.938+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc67815a3f0 con 0x7fc678159d30 2020-11-20T12:12:48.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.940+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc678061a20 con 0x7fc678159d30 2020-11-20T12:12:48.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.940+0000 7fc674ff9700 1 -- 172.21.15.187:0/3386878527 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc67001c460 con 0x7fc678159d30 2020-11-20T12:12:48.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.941+0000 7fc674ff9700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc654056d20 0x7fc6540591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:48.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.941+0000 7fc674ff9700 1 -- 172.21.15.187:0/3386878527 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc670070180 con 0x7fc678159d30 2020-11-20T12:12:48.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.941+0000 7fc677fff700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc654056d20 0x7fc6540591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:48.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.942+0000 7fc677fff700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc654056d20 0x7fc6540591e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fc6680013b0 tx=0x7fc668009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:48.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:48.944+0000 7fc674ff9700 1 -- 172.21.15.187:0/3386878527 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc670045460 con 0x7fc678159d30 2020-11-20T12:12:49.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.117+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc6780afc60 con 0x7fc678159d30 2020-11-20T12:12:49.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.118+0000 7fc674ff9700 1 -- 172.21.15.187:0/3386878527 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc670048a80 con 0x7fc678159d30 2020-11-20T12:12:49.121 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:12:49.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.120+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc654056d20 msgr2=0x7fc6540591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.120+0000 7fc67d3fd700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc654056d20 0x7fc6540591e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fc6680013b0 tx=0x7fc668009040).stop 2020-11-20T12:12:49.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.120+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc678159d30 msgr2=0x7fc67809e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.120+0000 7fc67d3fd700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc678159d30 0x7fc67809e4f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7fc67809ce70 tx=0x7fc670009040).stop 2020-11-20T12:12:49.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.121+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 shutdown_connections 2020-11-20T12:12:49.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.121+0000 7fc67d3fd700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc654056d20 0x7fc6540591e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.121+0000 7fc67d3fd700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6580044e0 0x7fc678142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.121+0000 7fc67d3fd700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc678159d30 0x7fc67809e4f0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.121+0000 7fc67d3fd700 1 --2- 172.21.15.187:0/3386878527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc6780b3f70 0x7fc678159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:49.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.121+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 >> 172.21.15.187:0/3386878527 conn(0x7fc6780bab30 msgr2=0x7fc678145af0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:49.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.121+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 shutdown_connections 2020-11-20T12:12:49.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.121+0000 7fc67d3fd700 1 -- 172.21.15.187:0/3386878527 wait complete. 2020-11-20T12:12:49.147 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:12:49.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.388+0000 7fb4aa074700 1 Processor -- start 2020-11-20T12:12:49.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.388+0000 7fb4aa074700 1 -- start start 2020-11-20T12:12:49.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4aa074700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb4a41484d0 0x7fb4a4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4aa074700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4aa074700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb4a4066f70 0x7fb4a40b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4aa074700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a4067730 con 0x7fb4a4066f70 2020-11-20T12:12:49.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4aa074700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a40b1300 con 0x7fb4a41484d0 2020-11-20T12:12:49.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4aa074700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a40b15a0 con 0x7fb4a40622c0 2020-11-20T12:12:49.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4a9072700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb4a41484d0 0x7fb4a4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4a9072700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb4a41484d0 0x7fb4a4061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47318/0 (socket says 172.21.15.187:47318) 2020-11-20T12:12:49.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.390+0000 7fb4a9072700 1 -- 172.21.15.187:0/1916415888 learned_addr learned my addr 172.21.15.187:0/1916415888 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:49.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.391+0000 7fb4a8871700 1 --2- 172.21.15.187:0/1916415888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.391+0000 7fb4a9873700 1 --2- 172.21.15.187:0/1916415888 >> v2:172.21.15.5:3300/0 conn(0x7fb4a4066f70 0x7fb4a40b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.391+0000 7fb4a9072700 1 -- 172.21.15.187:0/1916415888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 msgr2=0x7fb4a40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:49.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.391+0000 7fb4a9072700 1 --2- 172.21.15.187:0/1916415888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a40b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.391+0000 7fb4a9072700 1 -- 172.21.15.187:0/1916415888 >> v2:172.21.15.5:3300/0 conn(0x7fb4a4066f70 msgr2=0x7fb4a40b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:49.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.391+0000 7fb4a9072700 1 --2- 172.21.15.187:0/1916415888 >> v2:172.21.15.5:3300/0 conn(0x7fb4a4066f70 0x7fb4a40b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.391+0000 7fb4a9072700 1 -- 172.21.15.187:0/1916415888 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a40b1840 con 0x7fb4a41484d0 2020-11-20T12:12:49.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.391+0000 7fb4a9072700 1 --2- 172.21.15.187:0/1916415888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a41484d0 0x7fb4a4061dd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7fb4a40b85e0 tx=0x7fb4a0006eb0).ready entity=mon.1 client_cookie=70971f05609083d3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:49.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.392+0000 7fb493fff700 1 -- 172.21.15.187:0/1916415888 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a00098d0 con 0x7fb4a41484d0 2020-11-20T12:12:49.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.392+0000 7fb493fff700 1 -- 172.21.15.187:0/1916415888 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a0009a30 con 0x7fb4a41484d0 2020-11-20T12:12:49.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.392+0000 7fb493fff700 1 -- 172.21.15.187:0/1916415888 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0009db0 con 0x7fb4a41484d0 2020-11-20T12:12:49.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.392+0000 7fb4aa074700 1 -- 172.21.15.187:0/1916415888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a41484d0 msgr2=0x7fb4a4061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.392+0000 7fb4aa074700 1 --2- 172.21.15.187:0/1916415888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a41484d0 0x7fb4a4061dd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7fb4a40b85e0 tx=0x7fb4a0006eb0).stop 2020-11-20T12:12:49.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.393+0000 7fb4aa074700 1 -- 172.21.15.187:0/1916415888 shutdown_connections 2020-11-20T12:12:49.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.393+0000 7fb4aa074700 1 --2- 172.21.15.187:0/1916415888 >> v2:172.21.15.5:3300/0 conn(0x7fb4a4066f70 0x7fb4a40b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.393+0000 7fb4aa074700 1 --2- 172.21.15.187:0/1916415888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a41484d0 0x7fb4a4061dd0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.393+0000 7fb4aa074700 1 --2- 172.21.15.187:0/1916415888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.393+0000 7fb4aa074700 1 -- 172.21.15.187:0/1916415888 >> 172.21.15.187:0/1916415888 conn(0x7fb4a40bab30 msgr2=0x7fb4a40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.393+0000 7fb4aa074700 1 -- 172.21.15.187:0/1916415888 shutdown_connections 2020-11-20T12:12:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.393+0000 7fb4aa074700 1 -- 172.21.15.187:0/1916415888 wait complete. 2020-11-20T12:12:49.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.394+0000 7fb4aa074700 1 Processor -- start 2020-11-20T12:12:49.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.395+0000 7fb4aa074700 1 -- start start 2020-11-20T12:12:49.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4aa074700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a409ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4aa074700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a4066f70 0x7fb4a409ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4aa074700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4a41484d0 0x7fb4a40b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4aa074700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a409f750 con 0x7fb4a41484d0 2020-11-20T12:12:49.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4aa074700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a409f4b0 con 0x7fb4a4066f70 2020-11-20T12:12:49.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4aa074700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a409fa30 con 0x7fb4a40622c0 2020-11-20T12:12:49.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4a8871700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a4066f70 0x7fb4a409ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4a8871700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a4066f70 0x7fb4a409ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47326/0 (socket says 172.21.15.187:47326) 2020-11-20T12:12:49.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4a8871700 1 -- 172.21.15.187:0/430059734 learned_addr learned my addr 172.21.15.187:0/430059734 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:49.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4a9072700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a409ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4a9873700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4a41484d0 0x7fb4a40b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4a8871700 1 -- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 msgr2=0x7fb4a409ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4a8871700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a409ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.396+0000 7fb4a8871700 1 -- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4a41484d0 msgr2=0x7fb4a40b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4a8871700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4a41484d0 0x7fb4a40b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4a8871700 1 -- 172.21.15.187:0/430059734 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a40b3330 con 0x7fb4a4066f70 2020-11-20T12:12:49.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4a9072700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a409ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4a8871700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a4066f70 0x7fb4a409ab10 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7fb49400d930 tx=0x7fb494008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4917fa700 1 -- 172.21.15.187:0/430059734 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494009730 con 0x7fb4a4066f70 2020-11-20T12:12:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4917fa700 1 -- 172.21.15.187:0/430059734 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb494009890 con 0x7fb4a4066f70 2020-11-20T12:12:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a40b41c0 con 0x7fb4a4066f70 2020-11-20T12:12:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4917fa700 1 -- 172.21.15.187:0/430059734 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494009c30 con 0x7fb4a4066f70 2020-11-20T12:12:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.397+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a415b750 con 0x7fb4a4066f70 2020-11-20T12:12:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.398+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a409fbf0 con 0x7fb4a4066f70 2020-11-20T12:12:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.399+0000 7fb4917fa700 1 -- 172.21.15.187:0/430059734 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb49401c460 con 0x7fb4a4066f70 2020-11-20T12:12:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.400+0000 7fb4917fa700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb47c056d20 0x7fb47c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.400+0000 7fb4917fa700 1 -- 172.21.15.187:0/430059734 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb494070080 con 0x7fb4a4066f70 2020-11-20T12:12:49.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.400+0000 7fb4a9072700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb47c056d20 0x7fb47c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.401+0000 7fb4a9072700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb47c056d20 0x7fb47c0591e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fb4a000b1b0 tx=0x7fb4a0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:49.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.404+0000 7fb4917fa700 1 -- 172.21.15.187:0/430059734 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb494045470 con 0x7fb4a4066f70 2020-11-20T12:12:49.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.577+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb4a4061a20 con 0x7fb4a4066f70 2020-11-20T12:12:49.581 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:12:49.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.577+0000 7fb4917fa700 1 -- 172.21.15.187:0/430059734 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb494048a90 con 0x7fb4a4066f70 2020-11-20T12:12:49.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb47c056d20 msgr2=0x7fb47c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb47c056d20 0x7fb47c0591e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fb4a000b1b0 tx=0x7fb4a0015000).stop 2020-11-20T12:12:49.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a4066f70 msgr2=0x7fb4a409ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a4066f70 0x7fb4a409ab10 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7fb49400d930 tx=0x7fb494008040).stop 2020-11-20T12:12:49.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 shutdown_connections 2020-11-20T12:12:49.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb47c056d20 0x7fb47c0591e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4a41484d0 0x7fb4a40b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4a40622c0 0x7fb4a409ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.579+0000 7fb4aa074700 1 --2- 172.21.15.187:0/430059734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4a4066f70 0x7fb4a409ab10 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.580+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 >> 172.21.15.187:0/430059734 conn(0x7fb4a40bab30 msgr2=0x7fb4a4144ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:49.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.580+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 shutdown_connections 2020-11-20T12:12:49.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.580+0000 7fb4aa074700 1 -- 172.21.15.187:0/430059734 wait complete. 2020-11-20T12:12:49.605 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:12:49.606 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:49.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.846+0000 7fc9e9390700 1 Processor -- start 2020-11-20T12:12:49.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.846+0000 7fc9e9390700 1 -- start start 2020-11-20T12:12:49.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.848+0000 7fc9e9390700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc9e41484d0 0x7fc9e4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.848+0000 7fc9e9390700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc9e40622c0 0x7fc9e40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.848+0000 7fc9e9390700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e4066f70 0x7fc9e40b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.848+0000 7fc9e9390700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e4067730 con 0x7fc9e40622c0 2020-11-20T12:12:49.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.848+0000 7fc9e9390700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e40b1300 con 0x7fc9e41484d0 2020-11-20T12:12:49.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.848+0000 7fc9e9390700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40b15a0 con 0x7fc9e4066f70 2020-11-20T12:12:49.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.849+0000 7fc9e8b8f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e4066f70 0x7fc9e40b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.849+0000 7fc9dbfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc9e40622c0 0x7fc9e40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.849+0000 7fc9dbfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc9e40622c0 0x7fc9e40b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:39596/0 (socket says 172.21.15.187:39596) 2020-11-20T12:12:49.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.849+0000 7fc9dbfff700 1 -- 172.21.15.187:0/3931832752 learned_addr learned my addr 172.21.15.187:0/3931832752 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:49.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.849+0000 7fc9e3fff700 1 --2- 172.21.15.187:0/3931832752 >> v2:172.21.15.187:3300/0 conn(0x7fc9e41484d0 0x7fc9e4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.850+0000 7fc9e8b8f700 1 -- 172.21.15.187:0/3931832752 >> v2:172.21.15.187:3300/0 conn(0x7fc9e41484d0 msgr2=0x7fc9e4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:49.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.850+0000 7fc9e8b8f700 1 --2- 172.21.15.187:0/3931832752 >> v2:172.21.15.187:3300/0 conn(0x7fc9e41484d0 0x7fc9e4061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.850+0000 7fc9e8b8f700 1 -- 172.21.15.187:0/3931832752 >> v2:172.21.15.5:3300/0 conn(0x7fc9e40622c0 msgr2=0x7fc9e40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:49.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.850+0000 7fc9e8b8f700 1 --2- 172.21.15.187:0/3931832752 >> v2:172.21.15.5:3300/0 conn(0x7fc9e40622c0 0x7fc9e40b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.850+0000 7fc9e8b8f700 1 -- 172.21.15.187:0/3931832752 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e40b1840 con 0x7fc9e4066f70 2020-11-20T12:12:49.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.850+0000 7fc9dbfff700 1 --2- 172.21.15.187:0/3931832752 >> v2:172.21.15.5:3300/0 conn(0x7fc9e40622c0 0x7fc9e40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:12:49.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.851+0000 7fc9e8b8f700 1 --2- 172.21.15.187:0/3931832752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e4066f70 0x7fc9e40b5b30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc9e40b85e0 tx=0x7fc9dc0091d0).ready entity=mon.2 client_cookie=2fe7706e476fbfe1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:49.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.851+0000 7fc9e37fe700 1 -- 172.21.15.187:0/3931832752 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc00b9c0 con 0x7fc9e4066f70 2020-11-20T12:12:49.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.851+0000 7fc9e37fe700 1 -- 172.21.15.187:0/3931832752 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9dc00bb20 con 0x7fc9e4066f70 2020-11-20T12:12:49.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.851+0000 7fc9e37fe700 1 -- 172.21.15.187:0/3931832752 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc00bea0 con 0x7fc9e4066f70 2020-11-20T12:12:49.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 -- 172.21.15.187:0/3931832752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e4066f70 msgr2=0x7fc9e40b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 --2- 172.21.15.187:0/3931832752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e4066f70 0x7fc9e40b5b30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc9e40b85e0 tx=0x7fc9dc0091d0).stop 2020-11-20T12:12:49.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 -- 172.21.15.187:0/3931832752 shutdown_connections 2020-11-20T12:12:49.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 --2- 172.21.15.187:0/3931832752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e4066f70 0x7fc9e40b5b30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 --2- 172.21.15.187:0/3931832752 >> v2:172.21.15.187:3300/0 conn(0x7fc9e41484d0 0x7fc9e4061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 --2- 172.21.15.187:0/3931832752 >> v2:172.21.15.5:3300/0 conn(0x7fc9e40622c0 0x7fc9e40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:49.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 -- 172.21.15.187:0/3931832752 >> 172.21.15.187:0/3931832752 conn(0x7fc9e40bab30 msgr2=0x7fc9e40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:49.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 -- 172.21.15.187:0/3931832752 shutdown_connections 2020-11-20T12:12:49.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.852+0000 7fc9e9390700 1 -- 172.21.15.187:0/3931832752 wait complete. 2020-11-20T12:12:49.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.854+0000 7fc9e9390700 1 Processor -- start 2020-11-20T12:12:49.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.854+0000 7fc9e9390700 1 -- start start 2020-11-20T12:12:49.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9e9390700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9e40622c0 0x7fc9e409e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9e9390700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9e4066f70 0x7fc9e409f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9e9390700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e41484d0 0x7fc9e4144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9e9390700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e409fe70 con 0x7fc9e40622c0 2020-11-20T12:12:49.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9e9390700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40a0110 con 0x7fc9e4066f70 2020-11-20T12:12:49.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9e9390700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40a03b0 con 0x7fc9e41484d0 2020-11-20T12:12:49.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9dbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9e4066f70 0x7fc9e409f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9dbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9e4066f70 0x7fc9e409f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47338/0 (socket says 172.21.15.187:47338) 2020-11-20T12:12:49.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.855+0000 7fc9dbfff700 1 -- 172.21.15.187:0/2416194442 learned_addr learned my addr 172.21.15.187:0/2416194442 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:49.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9dbfff700 1 -- 172.21.15.187:0/2416194442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e41484d0 msgr2=0x7fc9e4144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9dbfff700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e41484d0 0x7fc9e4144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:49.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9dbfff700 1 -- 172.21.15.187:0/2416194442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9e40622c0 msgr2=0x7fc9e409e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:49.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9dbfff700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9e40622c0 0x7fc9e409e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:49.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9dbfff700 1 -- 172.21.15.187:0/2416194442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e409fbb0 con 0x7fc9e4066f70 2020-11-20T12:12:49.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9dbfff700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9e4066f70 0x7fc9e409f6c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7fc9cc00ecf0 tx=0x7fc9cc006eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:49.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9e0ff9700 1 -- 172.21.15.187:0/2416194442 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc00cbb0 con 0x7fc9e4066f70 2020-11-20T12:12:49.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9e0ff9700 1 -- 172.21.15.187:0/2416194442 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9cc00cd10 con 0x7fc9e4066f70 2020-11-20T12:12:49.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e4144750 con 0x7fc9e4066f70 2020-11-20T12:12:49.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9e0ff9700 1 -- 172.21.15.187:0/2416194442 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc012600 con 0x7fc9e4066f70 2020-11-20T12:12:49.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.856+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e40a0a90 con 0x7fc9e4066f70 2020-11-20T12:12:49.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.858+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e4061a20 con 0x7fc9e4066f70 2020-11-20T12:12:49.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.858+0000 7fc9e0ff9700 1 -- 172.21.15.187:0/2416194442 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc9cc022070 con 0x7fc9e4066f70 2020-11-20T12:12:49.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.859+0000 7fc9e0ff9700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9c0056d20 0x7fc9c00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:49.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.859+0000 7fc9e0ff9700 1 -- 172.21.15.187:0/2416194442 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc9cc0706d0 con 0x7fc9e4066f70 2020-11-20T12:12:49.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.859+0000 7fc9e3fff700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9c0056d20 0x7fc9c00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:49.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.860+0000 7fc9e3fff700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9c0056d20 0x7fc9c00591e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fc9d4006fd0 tx=0x7fc9d4006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:49.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:49.862+0000 7fc9e0ff9700 1 -- 172.21.15.187:0/2416194442 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc9cc045b20 con 0x7fc9e4066f70 2020-11-20T12:12:50.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.109+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9e41488b0 con 0x7fc9e4066f70 2020-11-20T12:12:51.018 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:12:51.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.110+0000 7fc9e0ff9700 1 -- 172.21.15.187:0/2416194442 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2152 (secure 0 0 0) 0x7fc9cc049140 con 0x7fc9e4066f70 2020-11-20T12:12:51.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9c0056d20 msgr2=0x7fc9c00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:51.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9c0056d20 0x7fc9c00591e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fc9d4006fd0 tx=0x7fc9d4006df0).stop 2020-11-20T12:12:51.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9e4066f70 msgr2=0x7fc9e409f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:51.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9e4066f70 0x7fc9e409f6c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7fc9cc00ecf0 tx=0x7fc9cc006eb0).stop 2020-11-20T12:12:51.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 shutdown_connections 2020-11-20T12:12:51.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9c0056d20 0x7fc9c00591e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:51.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9e40622c0 0x7fc9e409e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:51.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9e41484d0 0x7fc9e4144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:51.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 --2- 172.21.15.187:0/2416194442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9e4066f70 0x7fc9e409f6c0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:51.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 >> 172.21.15.187:0/2416194442 conn(0x7fc9e40bab30 msgr2=0x7fc9e4144b80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:51.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 shutdown_connections 2020-11-20T12:12:51.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:50.112+0000 7fc9e9390700 1 -- 172.21.15.187:0/2416194442 wait complete. 2020-11-20T12:12:51.022 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 68 2020-11-20T12:12:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:51.022 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:34.321010+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:51.031 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:12:51.224 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:34.321010+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:12:51.224 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2020-11-20T12:12:51.225 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2020-11-20T12:12:51.525 INFO:teuthology.orchestra.run.smithi005.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":1,"id":5159,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":31.315681941000001,"mdsmap_epoch":68,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":47.761558381999997} 2020-11-20T12:12:51.526 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5159, "mdsmap_epoch": 68, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 31.315681941, "state": "up:active", "uptime": 47.761558382, "want_state": "up:active", "whoami": 1 } 2020-11-20T12:12:51.527 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:12:51.810 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6952,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":41.466411047999998,"mdsmap_epoch":68,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":48.011989423000003} 2020-11-20T12:12:51.811 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6952, "mdsmap_epoch": 68, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 41.466411048, "state": "up:active", "uptime": 48.011989423, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:12:51.811 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:12:51.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:51.812 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:12:51.867 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:12:51.868 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:12:52.005 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:12:52.005 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:12:52.045 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:12:52.045 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:12:52.053 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:12:52.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:52.054 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:12:52.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:12:52.150 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -f dir_1/original) 2020-11-20T12:12:52.258 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:12:52.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:12:52.259 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:12:53.784 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:12:53.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:12:53.785 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:12:53.840 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:12:54.130 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_created":1}} 2020-11-20T12:12:54.131 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2020-11-20T12:12:54.131 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2020-11-20T12:12:54.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.383+0000 7fab0cd56700 1 Processor -- start 2020-11-20T12:12:54.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.383+0000 7fab0cd56700 1 -- start start 2020-11-20T12:12:54.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab0cd56700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab081484d0 0x7fab08061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:54.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab0cd56700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab080622c0 0x7fab080b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:54.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab0cd56700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fab08066f70 0x7fab080b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:54.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab0cd56700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab08067730 con 0x7fab08066f70 2020-11-20T12:12:54.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab0cd56700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab080b1300 con 0x7fab080622c0 2020-11-20T12:12:54.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab0cd56700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab080b15a0 con 0x7fab081484d0 2020-11-20T12:12:54.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab06ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab080622c0 0x7fab080b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:54.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab06ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab080622c0 0x7fab080b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47352/0 (socket says 172.21.15.187:47352) 2020-11-20T12:12:54.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.385+0000 7fab06ffd700 1 -- 172.21.15.187:0/703780693 learned_addr learned my addr 172.21.15.187:0/703780693 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:54.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab077fe700 1 --2- 172.21.15.187:0/703780693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab081484d0 0x7fab08061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:54.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab06ffd700 1 -- 172.21.15.187:0/703780693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab081484d0 msgr2=0x7fab08061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:54.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab06ffd700 1 --2- 172.21.15.187:0/703780693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab081484d0 0x7fab08061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab06ffd700 1 -- 172.21.15.187:0/703780693 >> v2:172.21.15.5:3300/0 conn(0x7fab08066f70 msgr2=0x7fab080b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab07fff700 1 --2- 172.21.15.187:0/703780693 >> v2:172.21.15.5:3300/0 conn(0x7fab08066f70 0x7fab080b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab06ffd700 1 --2- 172.21.15.187:0/703780693 >> v2:172.21.15.5:3300/0 conn(0x7fab08066f70 0x7fab080b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab06ffd700 1 -- 172.21.15.187:0/703780693 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab080b1840 con 0x7fab080622c0 2020-11-20T12:12:54.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab077fe700 1 --2- 172.21.15.187:0/703780693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab081484d0 0x7fab08061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:54.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.386+0000 7fab06ffd700 1 --2- 172.21.15.187:0/703780693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab080622c0 0x7fab080b5640 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7fab08066380 tx=0x7faafc006eb0).ready entity=mon.1 client_cookie=58587b245098a745 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:54.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab067fc700 1 -- 172.21.15.187:0/703780693 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc0098d0 con 0x7fab080622c0 2020-11-20T12:12:54.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab067fc700 1 -- 172.21.15.187:0/703780693 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc009a30 con 0x7fab080622c0 2020-11-20T12:12:54.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab067fc700 1 -- 172.21.15.187:0/703780693 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc009db0 con 0x7fab080622c0 2020-11-20T12:12:54.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab0cd56700 1 -- 172.21.15.187:0/703780693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab080622c0 msgr2=0x7fab080b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:54.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab0cd56700 1 --2- 172.21.15.187:0/703780693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab080622c0 0x7fab080b5640 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7fab08066380 tx=0x7faafc006eb0).stop 2020-11-20T12:12:54.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab0cd56700 1 -- 172.21.15.187:0/703780693 shutdown_connections 2020-11-20T12:12:54.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab0cd56700 1 --2- 172.21.15.187:0/703780693 >> v2:172.21.15.5:3300/0 conn(0x7fab08066f70 0x7fab080b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab0cd56700 1 --2- 172.21.15.187:0/703780693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab081484d0 0x7fab08061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab0cd56700 1 --2- 172.21.15.187:0/703780693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab080622c0 0x7fab080b5640 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab0cd56700 1 -- 172.21.15.187:0/703780693 >> 172.21.15.187:0/703780693 conn(0x7fab080bab30 msgr2=0x7fab080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.387+0000 7fab0cd56700 1 -- 172.21.15.187:0/703780693 shutdown_connections 2020-11-20T12:12:54.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.388+0000 7fab0cd56700 1 -- 172.21.15.187:0/703780693 wait complete. 2020-11-20T12:12:54.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.389+0000 7fab0cd56700 1 Processor -- start 2020-11-20T12:12:54.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.389+0000 7fab0cd56700 1 -- start start 2020-11-20T12:12:54.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.390+0000 7fab0cd56700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab080622c0 0x7fab0809c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:54.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.390+0000 7fab0cd56700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab08066f70 0x7fab08143550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.390+0000 7fab0cd56700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab081484d0 0x7fab081448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.390+0000 7fab0cd56700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab080b1aa0 con 0x7fab080622c0 2020-11-20T12:12:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.390+0000 7fab0cd56700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab080b1d40 con 0x7fab081484d0 2020-11-20T12:12:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.390+0000 7fab0cd56700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab080b1fe0 con 0x7fab08066f70 2020-11-20T12:12:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.390+0000 7fab07fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab081484d0 0x7fab081448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.390+0000 7fab07fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab081484d0 0x7fab081448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47360/0 (socket says 172.21.15.187:47360) 2020-11-20T12:12:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7fab07fff700 1 -- 172.21.15.187:0/4294261875 learned_addr learned my addr 172.21.15.187:0/4294261875 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7fab06ffd700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab08066f70 0x7fab08143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:54.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7fab07fff700 1 -- 172.21.15.187:0/4294261875 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab08066f70 msgr2=0x7fab08143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:54.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7fab07fff700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab08066f70 0x7fab08143550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:54.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7fab07fff700 1 -- 172.21.15.187:0/4294261875 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab080622c0 msgr2=0x7fab0809c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:54.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7fab07fff700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab080622c0 0x7fab0809c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:54.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7fab07fff700 1 -- 172.21.15.187:0/4294261875 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab081445d0 con 0x7fab081484d0 2020-11-20T12:12:54.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7fab07fff700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab081484d0 0x7fab081448c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7fab0000afd0 tx=0x7fab00008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:54.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7faae77fe700 1 -- 172.21.15.187:0/4294261875 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00007940 con 0x7fab081484d0 2020-11-20T12:12:54.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.391+0000 7faae77fe700 1 -- 172.21.15.187:0/4294261875 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab00007aa0 con 0x7fab081484d0 2020-11-20T12:12:54.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.392+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab08135f50 con 0x7fab081484d0 2020-11-20T12:12:54.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.392+0000 7faae77fe700 1 -- 172.21.15.187:0/4294261875 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00007e40 con 0x7fab081484d0 2020-11-20T12:12:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.392+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab08136410 con 0x7fab081484d0 2020-11-20T12:12:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.393+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab080b21a0 con 0x7fab081484d0 2020-11-20T12:12:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.393+0000 7faae77fe700 1 -- 172.21.15.187:0/4294261875 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fab00011620 con 0x7fab081484d0 2020-11-20T12:12:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.394+0000 7faae77fe700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faae0056d20 0x7faae00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:54.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.394+0000 7faae77fe700 1 -- 172.21.15.187:0/4294261875 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fab00070300 con 0x7fab081484d0 2020-11-20T12:12:54.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.395+0000 7fab077fe700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faae0056d20 0x7faae00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:54.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.395+0000 7fab077fe700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faae0056d20 0x7faae00591e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7faaf800a6d0 tx=0x7faaf8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:54.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.399+0000 7faae77fe700 1 -- 172.21.15.187:0/4294261875 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fab000456d0 con 0x7fab081484d0 2020-11-20T12:12:54.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:54.644+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fab08061a20 con 0x7fab081484d0 2020-11-20T12:12:55.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.184+0000 7faae77fe700 1 -- 172.21.15.187:0/4294261875 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v69) v1 ==== 105+0+0 (secure 0 0 0) 0x7fab00048cf0 con 0x7fab081484d0 2020-11-20T12:12:55.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.188+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faae0056d20 msgr2=0x7faae00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:55.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.188+0000 7fab0cd56700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faae0056d20 0x7faae00591e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7faaf800a6d0 tx=0x7faaf8008040).stop 2020-11-20T12:12:55.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.188+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab081484d0 msgr2=0x7fab081448c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:55.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.188+0000 7fab0cd56700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab081484d0 0x7fab081448c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7fab0000afd0 tx=0x7fab00008040).stop 2020-11-20T12:12:55.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.189+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 shutdown_connections 2020-11-20T12:12:55.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.189+0000 7fab0cd56700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faae0056d20 0x7faae00591e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.189+0000 7fab0cd56700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab080622c0 0x7fab0809c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:55.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.189+0000 7fab0cd56700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab081484d0 0x7fab081448c0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.189+0000 7fab0cd56700 1 --2- 172.21.15.187:0/4294261875 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fab08066f70 0x7fab08143550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.189+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 >> 172.21.15.187:0/4294261875 conn(0x7fab080bab30 msgr2=0x7fab080b0760 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:55.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.189+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 shutdown_connections 2020-11-20T12:12:55.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.190+0000 7fab0cd56700 1 -- 172.21.15.187:0/4294261875 wait complete. 2020-11-20T12:12:55.223 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:55.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.463+0000 7fb9de0d1700 1 Processor -- start 2020-11-20T12:12:55.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.463+0000 7fb9de0d1700 1 -- start start 2020-11-20T12:12:55.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.464+0000 7fb9de0d1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb9d81484d0 0x7fb9d8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:55.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.464+0000 7fb9de0d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d80622c0 0x7fb9d80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:55.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.464+0000 7fb9de0d1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb9d8066f70 0x7fb9d80b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:55.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.464+0000 7fb9de0d1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d8067730 con 0x7fb9d8066f70 2020-11-20T12:12:55.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.464+0000 7fb9de0d1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d80b1300 con 0x7fb9d81484d0 2020-11-20T12:12:55.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.464+0000 7fb9de0d1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80b15a0 con 0x7fb9d80622c0 2020-11-20T12:12:55.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.464+0000 7fb9dd0cf700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb9d81484d0 0x7fb9d8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:55.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9dd0cf700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb9d81484d0 0x7fb9d8061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47364/0 (socket says 172.21.15.187:47364) 2020-11-20T12:12:55.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9dd0cf700 1 -- 172.21.15.187:0/2219824157 learned_addr learned my addr 172.21.15.187:0/2219824157 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:55.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9dd0cf700 1 -- 172.21.15.187:0/2219824157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d80622c0 msgr2=0x7fb9d80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:55.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9dd0cf700 1 --2- 172.21.15.187:0/2219824157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d80622c0 0x7fb9d80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:55.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9dd0cf700 1 -- 172.21.15.187:0/2219824157 >> v2:172.21.15.5:3300/0 conn(0x7fb9d8066f70 msgr2=0x7fb9d80b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:55.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9dd0cf700 1 --2- 172.21.15.187:0/2219824157 >> v2:172.21.15.5:3300/0 conn(0x7fb9d8066f70 0x7fb9d80b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:55.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9dd0cf700 1 -- 172.21.15.187:0/2219824157 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d80b1840 con 0x7fb9d81484d0 2020-11-20T12:12:55.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9dd0cf700 1 --2- 172.21.15.187:0/2219824157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d81484d0 0x7fb9d8061dd0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7fb9d80b85e0 tx=0x7fb9d4009290).ready entity=mon.1 client_cookie=e03233157eaa3b2c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:55.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.465+0000 7fb9c7fff700 1 -- 172.21.15.187:0/2219824157 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d400bbc0 con 0x7fb9d81484d0 2020-11-20T12:12:55.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9c7fff700 1 -- 172.21.15.187:0/2219824157 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d400bd20 con 0x7fb9d81484d0 2020-11-20T12:12:55.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9c7fff700 1 -- 172.21.15.187:0/2219824157 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4011610 con 0x7fb9d81484d0 2020-11-20T12:12:55.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2219824157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d81484d0 msgr2=0x7fb9d8061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:55.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2219824157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d81484d0 0x7fb9d8061dd0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7fb9d80b85e0 tx=0x7fb9d4009290).stop 2020-11-20T12:12:55.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2219824157 shutdown_connections 2020-11-20T12:12:55.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2219824157 >> v2:172.21.15.5:3300/0 conn(0x7fb9d8066f70 0x7fb9d80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:55.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2219824157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d81484d0 0x7fb9d8061dd0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2219824157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d80622c0 0x7fb9d80b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:55.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2219824157 >> 172.21.15.187:0/2219824157 conn(0x7fb9d80bab30 msgr2=0x7fb9d80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:55.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2219824157 shutdown_connections 2020-11-20T12:12:55.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.466+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2219824157 wait complete. 2020-11-20T12:12:55.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.468+0000 7fb9de0d1700 1 Processor -- start 2020-11-20T12:12:55.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.468+0000 7fb9de0d1700 1 -- start start 2020-11-20T12:12:55.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9de0d1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d80622c0 0x7fb9d8099220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:55.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9de0d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d8066f70 0x7fb9d8132c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:55.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9de0d1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9d813e0a0 0x7fb9d81330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:55.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9de0d1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d813e810 con 0x7fb9d813e0a0 2020-11-20T12:12:55.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9de0d1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d81335e0 con 0x7fb9d80622c0 2020-11-20T12:12:55.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9de0d1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d813e570 con 0x7fb9d8066f70 2020-11-20T12:12:55.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9dd0cf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d80622c0 0x7fb9d8099220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:55.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9dd0cf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d80622c0 0x7fb9d8099220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47370/0 (socket says 172.21.15.187:47370) 2020-11-20T12:12:55.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.469+0000 7fb9dd0cf700 1 -- 172.21.15.187:0/2312981134 learned_addr learned my addr 172.21.15.187:0/2312981134 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:55.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dc8ce700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d8066f70 0x7fb9d8132c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:55.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dd8d0700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9d813e0a0 0x7fb9d81330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:55.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dd0cf700 1 -- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d8066f70 msgr2=0x7fb9d8132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:55.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dd0cf700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d8066f70 0x7fb9d8132c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dd0cf700 1 -- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9d813e0a0 msgr2=0x7fb9d81330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:55.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dd0cf700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9d813e0a0 0x7fb9d81330f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dd0cf700 1 -- 172.21.15.187:0/2312981134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d809b070 con 0x7fb9d80622c0 2020-11-20T12:12:55.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dc8ce700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d8066f70 0x7fb9d8132c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:55.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9dd0cf700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d80622c0 0x7fb9d8099220 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7fb9d8064da0 tx=0x7fb9d4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:55.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9c57fa700 1 -- 172.21.15.187:0/2312981134 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d401b6a0 con 0x7fb9d80622c0 2020-11-20T12:12:55.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9c57fa700 1 -- 172.21.15.187:0/2312981134 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d401b800 con 0x7fb9d80622c0 2020-11-20T12:12:55.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9c57fa700 1 -- 172.21.15.187:0/2312981134 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d401bba0 con 0x7fb9d80622c0 2020-11-20T12:12:55.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.470+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d80b6030 con 0x7fb9d80622c0 2020-11-20T12:12:55.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.471+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d809cfd0 con 0x7fb9d80622c0 2020-11-20T12:12:55.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.472+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d8061a20 con 0x7fb9d80622c0 2020-11-20T12:12:55.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.472+0000 7fb9c57fa700 1 -- 172.21.15.187:0/2312981134 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb9d4017070 con 0x7fb9d80622c0 2020-11-20T12:12:55.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.473+0000 7fb9c57fa700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb9b0056d20 0x7fb9b00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:55.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.473+0000 7fb9c57fa700 1 -- 172.21.15.187:0/2312981134 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb9d4071d50 con 0x7fb9d80622c0 2020-11-20T12:12:55.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.473+0000 7fb9dc8ce700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb9b0056d20 0x7fb9b00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:55.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.474+0000 7fb9dc8ce700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb9b0056d20 0x7fb9b00591e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7fb9d809aab0 tx=0x7fb9c8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:55.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.477+0000 7fb9c57fa700 1 -- 172.21.15.187:0/2312981134 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb9d40471a0 con 0x7fb9d80622c0 2020-11-20T12:12:55.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.722+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9d81488b0 con 0x7fb9d80622c0 2020-11-20T12:12:55.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.723+0000 7fb9c57fa700 1 -- 172.21.15.187:0/2312981134 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2154 (secure 0 0 0) 0x7fb9d4015070 con 0x7fb9d80622c0 2020-11-20T12:12:55.726 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:55.727 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:55.184060+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:55.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb9b0056d20 msgr2=0x7fb9b00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:55.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb9b0056d20 0x7fb9b00591e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7fb9d809aab0 tx=0x7fb9c8008040).stop 2020-11-20T12:12:55.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d80622c0 msgr2=0x7fb9d8099220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:55.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d80622c0 0x7fb9d8099220 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7fb9d8064da0 tx=0x7fb9d4016000).stop 2020-11-20T12:12:55.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 shutdown_connections 2020-11-20T12:12:55.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb9b0056d20 0x7fb9b00591e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9d813e0a0 0x7fb9d81330f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9d80622c0 0x7fb9d8099220 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 --2- 172.21.15.187:0/2312981134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb9d8066f70 0x7fb9d8132c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:55.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 >> 172.21.15.187:0/2312981134 conn(0x7fb9d80bab30 msgr2=0x7fb9d80b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:55.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 shutdown_connections 2020-11-20T12:12:55.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:55.726+0000 7fb9de0d1700 1 -- 172.21.15.187:0/2312981134 wait complete. 2020-11-20T12:12:55.732 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 70 2020-11-20T12:12:55.951 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:55.184060+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:12:55.952 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:12:56.954 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.200+0000 7fe1c0d18700 1 Processor -- start 2020-11-20T12:12:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.201+0000 7fe1c0d18700 1 -- start start 2020-11-20T12:12:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.202+0000 7fe1c0d18700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe1bc0b57b0 0x7fe1bc068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.202+0000 7fe1c0d18700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1bc0b5f50 0x7fe1bc0b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.202+0000 7fe1c0d18700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1bc144b80 con 0x7fe1bc1482d0 2020-11-20T12:12:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.202+0000 7fe1c0d18700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1bc1489e0 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.202+0000 7fe1c0d18700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1bc148c80 con 0x7fe1bc0b5f50 2020-11-20T12:12:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.203+0000 7fe1bbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1bc0b5f50 0x7fe1bc0b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.203+0000 7fe1bbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1bc0b5f50 0x7fe1bc0b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47382/0 (socket says 172.21.15.187:47382) 2020-11-20T12:12:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.203+0000 7fe1bbfff700 1 -- 172.21.15.187:0/3793068464 learned_addr learned my addr 172.21.15.187:0/3793068464 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.203+0000 7fe1bb7fe700 1 --2- 172.21.15.187:0/3793068464 >> v2:172.21.15.5:3300/0 conn(0x7fe1bc0b57b0 0x7fe1bc068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.204+0000 7fe1bbfff700 1 -- 172.21.15.187:0/3793068464 >> v2:172.21.15.5:3300/0 conn(0x7fe1bc0b57b0 msgr2=0x7fe1bc068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.204+0000 7fe1bbfff700 1 --2- 172.21.15.187:0/3793068464 >> v2:172.21.15.5:3300/0 conn(0x7fe1bc0b57b0 0x7fe1bc068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.204+0000 7fe1bbfff700 1 -- 172.21.15.187:0/3793068464 >> v1:172.21.15.5:6789/0 conn(0x7fe1bc1482d0 legacy=0x7fe1bc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.204+0000 7fe1bbfff700 1 -- 172.21.15.187:0/3793068464 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc148f20 con 0x7fe1bc0b5f50 2020-11-20T12:12:57.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.205+0000 7fe1bbfff700 1 --2- 172.21.15.187:0/3793068464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b5f50 0x7fe1bc0b6380 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7fe1bc066230 tx=0x7fe1b4009290).ready entity=mon.1 client_cookie=6b41d352302bc1f7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:57.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.206+0000 7fe1baffd700 1 -- 172.21.15.187:0/3793068464 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b400bbc0 con 0x7fe1bc0b5f50 2020-11-20T12:12:57.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.206+0000 7fe1baffd700 1 -- 172.21.15.187:0/3793068464 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b400bd20 con 0x7fe1bc0b5f50 2020-11-20T12:12:57.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.206+0000 7fe1baffd700 1 -- 172.21.15.187:0/3793068464 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4011610 con 0x7fe1bc0b5f50 2020-11-20T12:12:57.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.206+0000 7fe1c0d18700 1 -- 172.21.15.187:0/3793068464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b5f50 msgr2=0x7fe1bc0b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:57.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.206+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/3793068464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b5f50 0x7fe1bc0b6380 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7fe1bc066230 tx=0x7fe1b4009290).stop 2020-11-20T12:12:57.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.207+0000 7fe1c0d18700 1 -- 172.21.15.187:0/3793068464 shutdown_connections 2020-11-20T12:12:57.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.207+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/3793068464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b5f50 0x7fe1bc0b6380 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:57.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.207+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/3793068464 >> v2:172.21.15.5:3300/0 conn(0x7fe1bc0b57b0 0x7fe1bc068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:57.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.207+0000 7fe1c0d18700 1 -- 172.21.15.187:0/3793068464 >> 172.21.15.187:0/3793068464 conn(0x7fe1bc0bab30 msgr2=0x7fe1bc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:57.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.207+0000 7fe1c0d18700 1 -- 172.21.15.187:0/3793068464 shutdown_connections 2020-11-20T12:12:57.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.207+0000 7fe1c0d18700 1 -- 172.21.15.187:0/3793068464 wait complete. 2020-11-20T12:12:57.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.208+0000 7fe1c0d18700 1 Processor -- start 2020-11-20T12:12:57.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.209+0000 7fe1c0d18700 1 -- start start 2020-11-20T12:12:57.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.209+0000 7fe1c0d18700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b57b0 0x7fe1bc09ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:57.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.209+0000 7fe1c0d18700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1bc0b5f50 0x7fe1bc15bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:57.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.209+0000 7fe1c0d18700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1bc1482d0 0x7fe1bc145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:57.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.209+0000 7fe1c0d18700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc09f350 con 0x7fe1bc1482d0 2020-11-20T12:12:57.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.209+0000 7fe1c0d18700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc15c3a0 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.209+0000 7fe1c0d18700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc15c640 con 0x7fe1bc0b5f50 2020-11-20T12:12:57.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1bb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b57b0 0x7fe1bc09ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:57.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1bb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b57b0 0x7fe1bc09ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47384/0 (socket says 172.21.15.187:47384) 2020-11-20T12:12:57.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1bb7fe700 1 -- 172.21.15.187:0/1152943702 learned_addr learned my addr 172.21.15.187:0/1152943702 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:57.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1b2ffd700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1bc0b5f50 0x7fe1bc15bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:57.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1bb7fe700 1 -- 172.21.15.187:0/1152943702 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1bc0b5f50 msgr2=0x7fe1bc15bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:57.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1bb7fe700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1bc0b5f50 0x7fe1bc15bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:57.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1bb7fe700 1 -- 172.21.15.187:0/1152943702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1bc1482d0 msgr2=0x7fe1bc145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:57.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1bb7fe700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1bc1482d0 0x7fe1bc145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.210+0000 7fe1bb7fe700 1 -- 172.21.15.187:0/1152943702 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc15c030 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.211+0000 7fe1bb7fe700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b57b0 0x7fe1bc09ee60 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7fe1ac00ec00 tx=0x7fe1ac006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.211+0000 7fe1b3fff700 1 -- 172.21.15.187:0/1152943702 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac00c2e0 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.211+0000 7fe1b3fff700 1 -- 172.21.15.187:0/1152943702 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac00b9a0 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.211+0000 7fe1b3fff700 1 -- 172.21.15.187:0/1152943702 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac00bd40 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.211+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1bc09e870 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.211+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1bc147590 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.212+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1bc15c800 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.213+0000 7fe1b3fff700 1 -- 172.21.15.187:0/1152943702 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe1ac012600 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.213+0000 7fe1b3fff700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe198056d20 0x7fe1980591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:57.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.213+0000 7fe1b3fff700 1 -- 172.21.15.187:0/1152943702 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe1ac070380 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.213+0000 7fe1b2ffd700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe198056d20 0x7fe1980591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:57.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.214+0000 7fe1b2ffd700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe198056d20 0x7fe1980591e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fe1bc15b8e0 tx=0x7fe1a4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:57.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.217+0000 7fe1b3fff700 1 -- 172.21.15.187:0/1152943702 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe1ac045750 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.459+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1bc0b6440 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.459+0000 7fe1b3fff700 1 -- 172.21.15.187:0/1152943702 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2154 (secure 0 0 0) 0x7fe1ac048d70 con 0x7fe1bc0b57b0 2020-11-20T12:12:57.463 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:57.463 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:55.184060+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:57.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.461+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe198056d20 msgr2=0x7fe1980591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.461+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe198056d20 0x7fe1980591e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fe1bc15b8e0 tx=0x7fe1a4009040).stop 2020-11-20T12:12:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.461+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b57b0 msgr2=0x7fe1bc09ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.461+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b57b0 0x7fe1bc09ee60 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7fe1ac00ec00 tx=0x7fe1ac006df0).stop 2020-11-20T12:12:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.462+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 shutdown_connections 2020-11-20T12:12:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.462+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe198056d20 0x7fe1980591e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:57.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.462+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1bc1482d0 0x7fe1bc145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:57.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.462+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0b57b0 0x7fe1bc09ee60 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:57.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.462+0000 7fe1c0d18700 1 --2- 172.21.15.187:0/1152943702 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1bc0b5f50 0x7fe1bc15bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:57.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.462+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 >> 172.21.15.187:0/1152943702 conn(0x7fe1bc0bab30 msgr2=0x7fe1bc1440d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:57.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.462+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 shutdown_connections 2020-11-20T12:12:57.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:57.462+0000 7fe1c0d18700 1 -- 172.21.15.187:0/1152943702 wait complete. 2020-11-20T12:12:57.467 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 70 2020-11-20T12:12:57.687 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:55.184060+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:12:57.688 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:12:58.689 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:58.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.940+0000 7efdd7fff700 1 Processor -- start 2020-11-20T12:12:58.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.941+0000 7efdd7fff700 1 -- start start 2020-11-20T12:12:58.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.942+0000 7efdd7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efdd80b67f0 0x7efdd8099ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:58.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.942+0000 7efdd7fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd8066790 con 0x7efdd8061f90 2020-11-20T12:12:58.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.942+0000 7efdd7fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd80b56b0 con 0x7efdd8062370 2020-11-20T12:12:58.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.942+0000 7efdd7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd80b59a0 con 0x7efdd80b67f0 2020-11-20T12:12:58.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.942+0000 7efdd67fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7efdd8062370 0x7efdd80b60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44000/0 (socket says 172.21.15.187:44000) 2020-11-20T12:12:58.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.942+0000 7efdd67fc700 1 -- 172.21.15.187:0/1646837429 learned_addr learned my addr 172.21.15.187:0/1646837429 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:12:58.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd77fe700 1 --2- 172.21.15.187:0/1646837429 >> v2:172.21.15.5:3300/0 conn(0x7efdd80b67f0 0x7efdd8099ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:58.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393088038 0 0) 0x7efdd80b56b0 con 0x7efdd8062370 2020-11-20T12:12:58.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc0003240 con 0x7efdd8062370 2020-11-20T12:12:58.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 236963488 0 0) 0x7efdc0003240 con 0x7efdd8062370 2020-11-20T12:12:58.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 >> v2:172.21.15.5:3300/0 conn(0x7efdd80b67f0 msgr2=0x7efdd8099ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:58.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 --2- 172.21.15.187:0/1646837429 >> v2:172.21.15.5:3300/0 conn(0x7efdd80b67f0 0x7efdd8099ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:58.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 >> v1:172.21.15.5:6789/0 conn(0x7efdd8061f90 legacy=0x7efdd80afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:58.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd80b5c40 con 0x7efdd8062370 2020-11-20T12:12:58.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7efdcc0014e0 con 0x7efdd8062370 2020-11-20T12:12:58.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.943+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 >> v1:172.21.15.187:6789/0 conn(0x7efdd8062370 legacy=0x7efdd80b60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:58.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd5ffb700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdc0003480 0x7efdc0003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:58.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd5ffb700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdc00044e0 0x7efdc0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:58.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd5ffb700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdc0004e50 0x7efdc0005700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:58.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd80b59a0 con 0x7efdc0004e50 2020-11-20T12:12:58.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0004250 con 0x7efdc0003480 2020-11-20T12:12:58.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0005bf0 con 0x7efdc00044e0 2020-11-20T12:12:58.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd6ffd700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdc0003480 0x7efdc0003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:58.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd6ffd700 1 -- 172.21.15.187:0/1646837429 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdc00044e0 msgr2=0x7efdc0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:58.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd6ffd700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdc00044e0 0x7efdc0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:58.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd6ffd700 1 -- 172.21.15.187:0/1646837429 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdc0004e50 msgr2=0x7efdc0005700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:58.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd6ffd700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdc0004e50 0x7efdc0005700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:58.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd6ffd700 1 -- 172.21.15.187:0/1646837429 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc0003240 con 0x7efdc0003480 2020-11-20T12:12:58.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.944+0000 7efdd77fe700 1 -- 172.21.15.187:0/1646837429 reap_dead start 2020-11-20T12:12:58.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.945+0000 7efdd6ffd700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdc0003480 0x7efdc0003b80 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7efdc8002860 tx=0x7efdc8007b90).ready entity=mon.1 client_cookie=b88c6e4335073481 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:58.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.945+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8007020 con 0x7efdc0003480 2020-11-20T12:12:58.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.945+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc80071a0 con 0x7efdc0003480 2020-11-20T12:12:58.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.945+0000 7efdd5ffb700 1 -- 172.21.15.187:0/1646837429 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc800c830 con 0x7efdc0003480 2020-11-20T12:12:58.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.946+0000 7efdd7fff700 1 -- 172.21.15.187:0/1646837429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdc0003480 msgr2=0x7efdc0003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:58.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.946+0000 7efdd7fff700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdc0003480 0x7efdc0003b80 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7efdc8002860 tx=0x7efdc8007b90).stop 2020-11-20T12:12:58.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.946+0000 7efdd7fff700 1 -- 172.21.15.187:0/1646837429 shutdown_connections 2020-11-20T12:12:58.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.946+0000 7efdd7fff700 1 --2- 172.21.15.187:0/1646837429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdc0003480 0x7efdc0003b80 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:58.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.946+0000 7efdd7fff700 1 -- 172.21.15.187:0/1646837429 >> 172.21.15.187:0/1646837429 conn(0x7efdd80bab30 msgr2=0x7efdd80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:58.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.946+0000 7efdd7fff700 1 -- 172.21.15.187:0/1646837429 shutdown_connections 2020-11-20T12:12:58.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.946+0000 7efdd7fff700 1 -- 172.21.15.187:0/1646837429 wait complete. 2020-11-20T12:12:58.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.948+0000 7efdd7fff700 1 Processor -- start 2020-11-20T12:12:58.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.948+0000 7efdd7fff700 1 -- start start 2020-11-20T12:12:58.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd80a02f0 0x7efdd808ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:58.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdd80a06d0 0x7efdd816cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:58.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd809d460 0x7efdd816d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:58.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd808ff70 con 0x7efdd80a02f0 2020-11-20T12:12:58.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd806efa0 con 0x7efdd809d460 2020-11-20T12:12:58.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd806f240 con 0x7efdd80a06d0 2020-11-20T12:12:58.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd809d460 0x7efdd816d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:58.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd809d460 0x7efdd816d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47408/0 (socket says 172.21.15.187:47408) 2020-11-20T12:12:58.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.949+0000 7efdd77fe700 1 -- 172.21.15.187:0/3835231230 learned_addr learned my addr 172.21.15.187:0/3835231230 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:12:58.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd6ffd700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd80a02f0 0x7efdd808ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:58.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd67fc700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdd80a06d0 0x7efdd816cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:58.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd77fe700 1 -- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdd80a06d0 msgr2=0x7efdd816cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:58.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd77fe700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdd80a06d0 0x7efdd816cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:58.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd77fe700 1 -- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd80a02f0 msgr2=0x7efdd808ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:58.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd77fe700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd80a02f0 0x7efdd808ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:58.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd77fe700 1 -- 172.21.15.187:0/3835231230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8170ba0 con 0x7efdd809d460 2020-11-20T12:12:58.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd67fc700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdd80a06d0 0x7efdd816cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:58.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdd77fe700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd809d460 0x7efdd816d070 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7efdd8063870 tx=0x7efdd0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:58.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.950+0000 7efdbeffd700 1 -- 172.21.15.187:0/3835231230 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0017070 con 0x7efdd809d460 2020-11-20T12:12:58.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.951+0000 7efdbeffd700 1 -- 172.21.15.187:0/3835231230 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd0013430 con 0x7efdd809d460 2020-11-20T12:12:58.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.951+0000 7efdbeffd700 1 -- 172.21.15.187:0/3835231230 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd00137d0 con 0x7efdd809d460 2020-11-20T12:12:58.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.951+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd806f4e0 con 0x7efdd809d460 2020-11-20T12:12:58.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.951+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd808f150 con 0x7efdd809d460 2020-11-20T12:12:58.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.952+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd8170d30 con 0x7efdd809d460 2020-11-20T12:12:58.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.952+0000 7efdbeffd700 1 -- 172.21.15.187:0/3835231230 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efdd0013bc0 con 0x7efdd809d460 2020-11-20T12:12:58.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.953+0000 7efdbeffd700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdb4056d50 0x7efdb4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:58.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.953+0000 7efdbeffd700 1 -- 172.21.15.187:0/3835231230 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efdd006f980 con 0x7efdd809d460 2020-11-20T12:12:58.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.953+0000 7efdd6ffd700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdb4056d50 0x7efdb4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:58.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.954+0000 7efdd6ffd700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdb4056d50 0x7efdb4059210 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7efdd805ea30 tx=0x7efdc8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:58.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:58.958+0000 7efdbeffd700 1 -- 172.21.15.187:0/3835231230 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efdd0044d70 con 0x7efdd809d460 2020-11-20T12:12:59.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.200+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdd8061a20 con 0x7efdd809d460 2020-11-20T12:12:59.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.201+0000 7efdbeffd700 1 -- 172.21.15.187:0/3835231230 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2154 (secure 0 0 0) 0x7efdd0048390 con 0x7efdd809d460 2020-11-20T12:12:59.204 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:12:59.204 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:55.184060+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:12:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.203+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdb4056d50 msgr2=0x7efdb4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.203+0000 7efdd7fff700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdb4056d50 0x7efdb4059210 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7efdd805ea30 tx=0x7efdc8015000).stop 2020-11-20T12:12:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.203+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd809d460 msgr2=0x7efdd816d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.203+0000 7efdd7fff700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd809d460 0x7efdd816d070 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7efdd8063870 tx=0x7efdd0008040).stop 2020-11-20T12:12:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.204+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 shutdown_connections 2020-11-20T12:12:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.204+0000 7efdd7fff700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdb4056d50 0x7efdb4059210 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.204+0000 7efdd7fff700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd80a02f0 0x7efdd808ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.204+0000 7efdd7fff700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd809d460 0x7efdd816d070 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.204+0000 7efdd7fff700 1 --2- 172.21.15.187:0/3835231230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdd80a06d0 0x7efdd816cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:59.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.204+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 >> 172.21.15.187:0/3835231230 conn(0x7efdd80bab30 msgr2=0x7efdd8067cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:59.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.204+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 shutdown_connections 2020-11-20T12:12:59.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:12:59.204+0000 7efdd7fff700 1 -- 172.21.15.187:0/3835231230 wait complete. 2020-11-20T12:12:59.209 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 70 2020-11-20T12:12:59.429 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:55.184060+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:12:59.429 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:00.431 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:00.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.678+0000 7fefee950700 1 Processor -- start 2020-11-20T12:13:00.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.678+0000 7fefee950700 1 -- start start 2020-11-20T12:13:00.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.679+0000 7fefee950700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fefe81484d0 0x7fefe8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.679+0000 7fefee950700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80b1300 con 0x7fefe80622c0 2020-11-20T12:13:00.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.679+0000 7fefee950700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80b15a0 con 0x7fefe8066f70 2020-11-20T12:13:00.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.679+0000 7fefee950700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe8067730 con 0x7fefe81484d0 2020-11-20T12:13:00.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.679+0000 7fefed94e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fefe81484d0 0x7fefe8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.680+0000 7fefed94e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fefe81484d0 0x7fefe8061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47412/0 (socket says 172.21.15.187:47412) 2020-11-20T12:13:00.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.680+0000 7fefed94e700 1 -- 172.21.15.187:0/4190747878 learned_addr learned my addr 172.21.15.187:0/4190747878 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:00.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.680+0000 7fefed94e700 1 -- 172.21.15.187:0/4190747878 >> v1:172.21.15.187:6789/0 conn(0x7fefe8066f70 legacy=0x7fefe80b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.680+0000 7fefed94e700 1 -- 172.21.15.187:0/4190747878 >> v1:172.21.15.5:6789/0 conn(0x7fefe80622c0 legacy=0x7fefe80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:00.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.680+0000 7fefed94e700 1 -- 172.21.15.187:0/4190747878 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe80b1840 con 0x7fefe81484d0 2020-11-20T12:13:00.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.680+0000 7fefed94e700 1 --2- 172.21.15.187:0/4190747878 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe81484d0 0x7fefe8061dd0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7fefe4009d50 tx=0x7fefe4009290).ready entity=mon.1 client_cookie=37f8a0f6e83da91c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:00.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.681+0000 7fefec94c700 1 -- 172.21.15.187:0/4190747878 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe400bc40 con 0x7fefe81484d0 2020-11-20T12:13:00.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.681+0000 7fefec94c700 1 -- 172.21.15.187:0/4190747878 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe400bda0 con 0x7fefe81484d0 2020-11-20T12:13:00.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.681+0000 7fefec94c700 1 -- 172.21.15.187:0/4190747878 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4011610 con 0x7fefe81484d0 2020-11-20T12:13:00.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.681+0000 7fefee950700 1 -- 172.21.15.187:0/4190747878 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe81484d0 msgr2=0x7fefe8061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.681+0000 7fefee950700 1 --2- 172.21.15.187:0/4190747878 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe81484d0 0x7fefe8061dd0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7fefe4009d50 tx=0x7fefe4009290).stop 2020-11-20T12:13:00.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.681+0000 7fefee950700 1 -- 172.21.15.187:0/4190747878 shutdown_connections 2020-11-20T12:13:00.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.682+0000 7fefee950700 1 --2- 172.21.15.187:0/4190747878 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe81484d0 0x7fefe8061dd0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.682+0000 7fefee950700 1 -- 172.21.15.187:0/4190747878 >> 172.21.15.187:0/4190747878 conn(0x7fefe80bab30 msgr2=0x7fefe80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:00.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.682+0000 7fefee950700 1 -- 172.21.15.187:0/4190747878 shutdown_connections 2020-11-20T12:13:00.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.682+0000 7fefee950700 1 -- 172.21.15.187:0/4190747878 wait complete. 2020-11-20T12:13:00.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.683+0000 7fefee950700 1 Processor -- start 2020-11-20T12:13:00.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.684+0000 7fefee950700 1 -- start start 2020-11-20T12:13:00.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefee950700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefe80622c0 0x7fefe8141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefee950700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe8066f70 0x7fefe80b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefee950700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefe81484d0 0x7fefe80b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefee950700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe809d290 con 0x7fefe80622c0 2020-11-20T12:13:00.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefee950700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe80b33a0 con 0x7fefe8066f70 2020-11-20T12:13:00.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefee950700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe80b3640 con 0x7fefe81484d0 2020-11-20T12:13:00.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefed14d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe8066f70 0x7fefe80b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefed14d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe8066f70 0x7fefe80b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47420/0 (socket says 172.21.15.187:47420) 2020-11-20T12:13:00.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefed14d700 1 -- 172.21.15.187:0/607057669 learned_addr learned my addr 172.21.15.187:0/607057669 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:00.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.685+0000 7fefed94e700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefe80622c0 0x7fefe8141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefed14d700 1 -- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefe81484d0 msgr2=0x7fefe80b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefee14f700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefe81484d0 0x7fefe80b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefed14d700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefe81484d0 0x7fefe80b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefed14d700 1 -- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefe80622c0 msgr2=0x7fefe8141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefed14d700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefe80622c0 0x7fefe8141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefed14d700 1 -- 172.21.15.187:0/607057669 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe809ec10 con 0x7fefe8066f70 2020-11-20T12:13:00.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefed94e700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefe80622c0 0x7fefe8141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:00.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefed14d700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe8066f70 0x7fefe80b1bc0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7fefd800c760 tx=0x7fefd8007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:00.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefd5ffb700 1 -- 172.21.15.187:0/607057669 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd800e560 con 0x7fefe8066f70 2020-11-20T12:13:00.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefd5ffb700 1 -- 172.21.15.187:0/607057669 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd800e6c0 con 0x7fefe8066f70 2020-11-20T12:13:00.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefd5ffb700 1 -- 172.21.15.187:0/607057669 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd800ea80 con 0x7fefe8066f70 2020-11-20T12:13:00.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe80b25a0 con 0x7fefe8066f70 2020-11-20T12:13:00.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.686+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe8141ca0 con 0x7fefe8066f70 2020-11-20T12:13:00.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.688+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe80620a0 con 0x7fefe8066f70 2020-11-20T12:13:00.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.688+0000 7fefd5ffb700 1 -- 172.21.15.187:0/607057669 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fefd8019030 con 0x7fefe8066f70 2020-11-20T12:13:00.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.689+0000 7fefd5ffb700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fefc0056d20 0x7fefc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.689+0000 7fefd5ffb700 1 -- 172.21.15.187:0/607057669 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fefd806f3f0 con 0x7fefe8066f70 2020-11-20T12:13:00.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.689+0000 7fefed94e700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fefc0056d20 0x7fefc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.690+0000 7fefed94e700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fefc0056d20 0x7fefc00591e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fefe4009d50 tx=0x7fefe4016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:00.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.693+0000 7fefd5ffb700 1 -- 172.21.15.187:0/607057669 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fefd8044890 con 0x7fefe8066f70 2020-11-20T12:13:00.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.938+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefe8061a20 con 0x7fefe8066f70 2020-11-20T12:13:00.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.939+0000 7fefd5ffb700 1 -- 172.21.15.187:0/607057669 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7fefd8047eb0 con 0x7fefe8066f70 2020-11-20T12:13:00.944 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:00.945 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:00.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fefc0056d20 msgr2=0x7fefc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fefc0056d20 0x7fefc00591e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fefe4009d50 tx=0x7fefe4016000).stop 2020-11-20T12:13:00.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe8066f70 msgr2=0x7fefe80b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe8066f70 0x7fefe80b1bc0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7fefd800c760 tx=0x7fefd8007c00).stop 2020-11-20T12:13:00.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 shutdown_connections 2020-11-20T12:13:00.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fefc0056d20 0x7fefc00591e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefe80622c0 0x7fefe8141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefe81484d0 0x7fefe80b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 --2- 172.21.15.187:0/607057669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fefe8066f70 0x7fefe80b1bc0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 >> 172.21.15.187:0/607057669 conn(0x7fefe80bab30 msgr2=0x7fefe8144110 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 shutdown_connections 2020-11-20T12:13:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:00.942+0000 7fefee950700 1 -- 172.21.15.187:0/607057669 wait complete. 2020-11-20T12:13:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:01.175 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:01.176 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:02.177 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:02.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.429+0000 7fe304636700 1 Processor -- start 2020-11-20T12:13:02.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.429+0000 7fe304636700 1 -- start start 2020-11-20T12:13:02.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.430+0000 7fe304636700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0654a0 0x7fe2fc0b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:02.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.430+0000 7fe304636700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe2fc065880 0x7fe2fc144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:02.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.431+0000 7fe304636700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc0b08e0 con 0x7fe2fc0b6c10 2020-11-20T12:13:02.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.431+0000 7fe304636700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc0b0b80 con 0x7fe2fc065880 2020-11-20T12:13:02.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.431+0000 7fe304636700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0b05f0 con 0x7fe2fc0654a0 2020-11-20T12:13:02.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.431+0000 7fe3023d2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0654a0 0x7fe2fc0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:02.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.431+0000 7fe301bd1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe2fc065880 0x7fe2fc144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:02.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.432+0000 7fe3023d2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0654a0 0x7fe2fc0b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:48314/0 (socket says 172.21.15.187:48314) 2020-11-20T12:13:02.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.432+0000 7fe3023d2700 1 -- 172.21.15.187:0/4216955380 learned_addr learned my addr 172.21.15.187:0/4216955380 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:02.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.433+0000 7fe301bd1700 1 -- 172.21.15.187:0/4216955380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0654a0 msgr2=0x7fe2fc0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:02.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.433+0000 7fe301bd1700 1 --2- 172.21.15.187:0/4216955380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0654a0 0x7fe2fc0b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.433+0000 7fe301bd1700 1 -- 172.21.15.187:0/4216955380 >> v1:172.21.15.5:6789/0 conn(0x7fe2fc0b6c10 legacy=0x7fe2fc0b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:02.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.433+0000 7fe301bd1700 1 -- 172.21.15.187:0/4216955380 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc0676f0 con 0x7fe2fc065880 2020-11-20T12:13:02.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.433+0000 7fe301bd1700 1 --2- 172.21.15.187:0/4216955380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc065880 0x7fe2fc144e60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fe2ec00dba0 tx=0x7fe2ec006d90).ready entity=mon.0 client_cookie=745465ead08e7680 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:02.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe3013d0700 1 -- 172.21.15.187:0/4216955380 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec009940 con 0x7fe2fc065880 2020-11-20T12:13:02.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe3013d0700 1 -- 172.21.15.187:0/4216955380 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec009aa0 con 0x7fe2fc065880 2020-11-20T12:13:02.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe3013d0700 1 -- 172.21.15.187:0/4216955380 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec009e20 con 0x7fe2fc065880 2020-11-20T12:13:02.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe304636700 1 -- 172.21.15.187:0/4216955380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc065880 msgr2=0x7fe2fc144e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:02.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe304636700 1 --2- 172.21.15.187:0/4216955380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc065880 0x7fe2fc144e60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fe2ec00dba0 tx=0x7fe2ec006d90).stop 2020-11-20T12:13:02.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe304636700 1 -- 172.21.15.187:0/4216955380 shutdown_connections 2020-11-20T12:13:02.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe304636700 1 --2- 172.21.15.187:0/4216955380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0654a0 0x7fe2fc0b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe304636700 1 --2- 172.21.15.187:0/4216955380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc065880 0x7fe2fc144e60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.434+0000 7fe304636700 1 -- 172.21.15.187:0/4216955380 >> 172.21.15.187:0/4216955380 conn(0x7fe2fc0bacd0 msgr2=0x7fe2fc0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:02.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.435+0000 7fe304636700 1 -- 172.21.15.187:0/4216955380 shutdown_connections 2020-11-20T12:13:02.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.435+0000 7fe304636700 1 -- 172.21.15.187:0/4216955380 wait complete. 2020-11-20T12:13:02.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.436+0000 7fe304636700 1 Processor -- start 2020-11-20T12:13:02.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.437+0000 7fe304636700 1 -- start start 2020-11-20T12:13:02.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe304636700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2fc0654a0 0x7fe2fc146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:02.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe304636700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0b6c10 0x7fe2fc15bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:02.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe304636700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc0b27a0 0x7fe2fc15c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:02.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe304636700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc15c6d0 con 0x7fe2fc0b27a0 2020-11-20T12:13:02.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe304636700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0b4690 con 0x7fe2fc0654a0 2020-11-20T12:13:02.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe304636700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0b4930 con 0x7fe2fc0b6c10 2020-11-20T12:13:02.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe3023d2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2fc0654a0 0x7fe2fc146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:02.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe3023d2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2fc0654a0 0x7fe2fc146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47432/0 (socket says 172.21.15.187:47432) 2020-11-20T12:13:02.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.438+0000 7fe3023d2700 1 -- 172.21.15.187:0/1212377984 learned_addr learned my addr 172.21.15.187:0/1212377984 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:02.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.439+0000 7fe302bd3700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc0b27a0 0x7fe2fc15c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:02.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.439+0000 7fe301bd1700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0b6c10 0x7fe2fc15bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:02.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.439+0000 7fe3023d2700 1 -- 172.21.15.187:0/1212377984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0b6c10 msgr2=0x7fe2fc15bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:02.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.439+0000 7fe3023d2700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0b6c10 0x7fe2fc15bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.439+0000 7fe3023d2700 1 -- 172.21.15.187:0/1212377984 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc0b27a0 msgr2=0x7fe2fc15c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:02.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.439+0000 7fe3023d2700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc0b27a0 0x7fe2fc15c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.439+0000 7fe3023d2700 1 -- 172.21.15.187:0/1212377984 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc146900 con 0x7fe2fc0654a0 2020-11-20T12:13:02.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.439+0000 7fe3023d2700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2fc0654a0 0x7fe2fc146410 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7fe2f800ef20 tx=0x7fe2f8009290).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:02.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.440+0000 7fe2ea7fc700 1 -- 172.21.15.187:0/1212377984 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f800cd00 con 0x7fe2fc0654a0 2020-11-20T12:13:02.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.440+0000 7fe2ea7fc700 1 -- 172.21.15.187:0/1212377984 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f800ce60 con 0x7fe2fc0654a0 2020-11-20T12:13:02.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.440+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc09a810 con 0x7fe2fc0654a0 2020-11-20T12:13:02.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.440+0000 7fe2ea7fc700 1 -- 172.21.15.187:0/1212377984 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8012620 con 0x7fe2fc0654a0 2020-11-20T12:13:02.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.440+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc143350 con 0x7fe2fc0654a0 2020-11-20T12:13:02.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.441+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc0b4af0 con 0x7fe2fc0654a0 2020-11-20T12:13:02.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.441+0000 7fe2ea7fc700 1 -- 172.21.15.187:0/1212377984 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe2f8022070 con 0x7fe2fc0654a0 2020-11-20T12:13:02.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.442+0000 7fe2ea7fc700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe2d4056d50 0x7fe2d4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:02.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.442+0000 7fe2ea7fc700 1 -- 172.21.15.187:0/1212377984 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe2f8070a20 con 0x7fe2fc0654a0 2020-11-20T12:13:02.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.442+0000 7fe301bd1700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe2d4056d50 0x7fe2d4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:02.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.443+0000 7fe301bd1700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe2d4056d50 0x7fe2d4059210 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fe2fc09d9c0 tx=0x7fe2ec017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:02.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.446+0000 7fe2ea7fc700 1 -- 172.21.15.187:0/1212377984 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe2f8045e70 con 0x7fe2fc0654a0 2020-11-20T12:13:02.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.691+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2fc063600 con 0x7fe2fc0654a0 2020-11-20T12:13:02.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.692+0000 7fe2ea7fc700 1 -- 172.21.15.187:0/1212377984 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7fe2f8016020 con 0x7fe2fc0654a0 2020-11-20T12:13:02.695 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:02.695 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:02.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe2d4056d50 msgr2=0x7fe2d4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:02.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe2d4056d50 0x7fe2d4059210 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fe2fc09d9c0 tx=0x7fe2ec017000).stop 2020-11-20T12:13:02.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2fc0654a0 msgr2=0x7fe2fc146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:02.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2fc0654a0 0x7fe2fc146410 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7fe2f800ef20 tx=0x7fe2f8009290).stop 2020-11-20T12:13:02.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 shutdown_connections 2020-11-20T12:13:02.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe2d4056d50 0x7fe2d4059210 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2fc0b27a0 0x7fe2fc15c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2fc0654a0 0x7fe2fc146410 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 --2- 172.21.15.187:0/1212377984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2fc0b6c10 0x7fe2fc15bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:02.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 >> 172.21.15.187:0/1212377984 conn(0x7fe2fc0bacd0 msgr2=0x7fe2fc143d40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:02.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 shutdown_connections 2020-11-20T12:13:02.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:02.694+0000 7fe304636700 1 -- 172.21.15.187:0/1212377984 wait complete. 2020-11-20T12:13:02.700 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:02.922 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:02.923 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:03.924 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:04.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.164+0000 7fc841eb9700 1 Processor -- start 2020-11-20T12:13:04.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.164+0000 7fc841eb9700 1 -- start start 2020-11-20T12:13:04.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.165+0000 7fc841eb9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc83c1484d0 0x7fc83c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.165+0000 7fc841eb9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc83c066f70 0x7fc83c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.165+0000 7fc841eb9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0b1300 con 0x7fc83c0622c0 2020-11-20T12:13:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.165+0000 7fc841eb9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c067730 con 0x7fc83c066f70 2020-11-20T12:13:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.165+0000 7fc841eb9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c0b15a0 con 0x7fc83c1484d0 2020-11-20T12:13:04.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.166+0000 7fc840eb7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc83c1484d0 0x7fc83c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:04.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.166+0000 7fc840eb7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc83c1484d0 0x7fc83c061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47440/0 (socket says 172.21.15.187:47440) 2020-11-20T12:13:04.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.166+0000 7fc840eb7700 1 -- 172.21.15.187:0/1149166586 learned_addr learned my addr 172.21.15.187:0/1149166586 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:04.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.166+0000 7fc840eb7700 1 -- 172.21.15.187:0/1149166586 >> v2:172.21.15.5:3300/0 conn(0x7fc83c066f70 msgr2=0x7fc83c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.166+0000 7fc840eb7700 1 --2- 172.21.15.187:0/1149166586 >> v2:172.21.15.5:3300/0 conn(0x7fc83c066f70 0x7fc83c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.166+0000 7fc840eb7700 1 -- 172.21.15.187:0/1149166586 >> v1:172.21.15.5:6789/0 conn(0x7fc83c0622c0 legacy=0x7fc83c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.166+0000 7fc840eb7700 1 -- 172.21.15.187:0/1149166586 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c0b1840 con 0x7fc83c1484d0 2020-11-20T12:13:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.166+0000 7fc840eb7700 1 --2- 172.21.15.187:0/1149166586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c061dd0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7fc83c0b85e0 tx=0x7fc838006eb0).ready entity=mon.1 client_cookie=e750e03cc0238665 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:04.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.167+0000 7fc8337fe700 1 -- 172.21.15.187:0/1149166586 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838009890 con 0x7fc83c1484d0 2020-11-20T12:13:04.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.167+0000 7fc8337fe700 1 -- 172.21.15.187:0/1149166586 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8380099f0 con 0x7fc83c1484d0 2020-11-20T12:13:04.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.167+0000 7fc8337fe700 1 -- 172.21.15.187:0/1149166586 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838009d70 con 0x7fc83c1484d0 2020-11-20T12:13:04.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.167+0000 7fc841eb9700 1 -- 172.21.15.187:0/1149166586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 msgr2=0x7fc83c061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:04.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.167+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1149166586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c061dd0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7fc83c0b85e0 tx=0x7fc838006eb0).stop 2020-11-20T12:13:04.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.168+0000 7fc841eb9700 1 -- 172.21.15.187:0/1149166586 shutdown_connections 2020-11-20T12:13:04.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.168+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1149166586 >> v2:172.21.15.5:3300/0 conn(0x7fc83c066f70 0x7fc83c0b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:04.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.168+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1149166586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c061dd0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:04.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.168+0000 7fc841eb9700 1 -- 172.21.15.187:0/1149166586 >> 172.21.15.187:0/1149166586 conn(0x7fc83c0bab30 msgr2=0x7fc83c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:04.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.168+0000 7fc841eb9700 1 -- 172.21.15.187:0/1149166586 shutdown_connections 2020-11-20T12:13:04.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.168+0000 7fc841eb9700 1 -- 172.21.15.187:0/1149166586 wait complete. 2020-11-20T12:13:04.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.169+0000 7fc841eb9700 1 Processor -- start 2020-11-20T12:13:04.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.169+0000 7fc841eb9700 1 -- start start 2020-11-20T12:13:04.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.170+0000 7fc841eb9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc83c0622c0 0x7fc83c1585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:04.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.170+0000 7fc841eb9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc83c066f70 0x7fc83c1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:04.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.170+0000 7fc841eb9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c0b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:04.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.170+0000 7fc841eb9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0b3a30 con 0x7fc83c0622c0 2020-11-20T12:13:04.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.170+0000 7fc841eb9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0b2230 con 0x7fc83c1484d0 2020-11-20T12:13:04.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.170+0000 7fc841eb9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0b24d0 con 0x7fc83c066f70 2020-11-20T12:13:04.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.170+0000 7fc8416b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c0b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:04.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc8416b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c0b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47448/0 (socket says 172.21.15.187:47448) 2020-11-20T12:13:04.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc8416b8700 1 -- 172.21.15.187:0/1955911011 learned_addr learned my addr 172.21.15.187:0/1955911011 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:04.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc8416b8700 1 -- 172.21.15.187:0/1955911011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc83c066f70 msgr2=0x7fc83c1590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:04.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc8416b8700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc83c066f70 0x7fc83c1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:04.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc8416b8700 1 -- 172.21.15.187:0/1955911011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc83c0622c0 msgr2=0x7fc83c1585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:04.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc8416b8700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc83c0622c0 0x7fc83c1585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:04.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc8416b8700 1 -- 172.21.15.187:0/1955911011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c0b2770 con 0x7fc83c1484d0 2020-11-20T12:13:04.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc8416b8700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c0b1a00 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7fc83400ba00 tx=0x7fc834009220).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:04.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.171+0000 7fc830ff9700 1 -- 172.21.15.187:0/1955911011 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400c910 con 0x7fc83c1484d0 2020-11-20T12:13:04.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.172+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c159600 con 0x7fc83c1484d0 2020-11-20T12:13:04.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.172+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c0b2000 con 0x7fc83c1484d0 2020-11-20T12:13:04.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.172+0000 7fc830ff9700 1 -- 172.21.15.187:0/1955911011 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83400ca70 con 0x7fc83c1484d0 2020-11-20T12:13:04.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.172+0000 7fc830ff9700 1 -- 172.21.15.187:0/1955911011 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400cbe0 con 0x7fc83c1484d0 2020-11-20T12:13:04.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.173+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c0b2900 con 0x7fc83c1484d0 2020-11-20T12:13:04.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.173+0000 7fc830ff9700 1 -- 172.21.15.187:0/1955911011 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc834015420 con 0x7fc83c1484d0 2020-11-20T12:13:04.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.174+0000 7fc830ff9700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc814056d20 0x7fc8140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:04.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.174+0000 7fc830ff9700 1 -- 172.21.15.187:0/1955911011 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc83406fdb0 con 0x7fc83c1484d0 2020-11-20T12:13:04.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.174+0000 7fc840eb7700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc814056d20 0x7fc8140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:04.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.175+0000 7fc840eb7700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc814056d20 0x7fc8140591e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fc838006ef0 tx=0x7fc838015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:04.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.179+0000 7fc830ff9700 1 -- 172.21.15.187:0/1955911011 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc8340451a0 con 0x7fc83c1484d0 2020-11-20T12:13:04.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.425+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc83c061a20 con 0x7fc83c1484d0 2020-11-20T12:13:04.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.426+0000 7fc830ff9700 1 -- 172.21.15.187:0/1955911011 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7fc83401a070 con 0x7fc83c1484d0 2020-11-20T12:13:04.430 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:04.431 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:04.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc814056d20 msgr2=0x7fc8140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:04.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc814056d20 0x7fc8140591e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fc838006ef0 tx=0x7fc838015000).stop 2020-11-20T12:13:04.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 msgr2=0x7fc83c0b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:04.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c0b1a00 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7fc83400ba00 tx=0x7fc834009220).stop 2020-11-20T12:13:04.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 shutdown_connections 2020-11-20T12:13:04.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc814056d20 0x7fc8140591e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:04.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc83c0622c0 0x7fc83c1585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:04.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc83c1484d0 0x7fc83c0b1a00 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:04.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 --2- 172.21.15.187:0/1955911011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc83c066f70 0x7fc83c1590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:04.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 >> 172.21.15.187:0/1955911011 conn(0x7fc83c0bab30 msgr2=0x7fc83c1440a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:04.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.428+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 shutdown_connections 2020-11-20T12:13:04.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:04.429+0000 7fc841eb9700 1 -- 172.21.15.187:0/1955911011 wait complete. 2020-11-20T12:13:04.435 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:04.656 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:04.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:05.658 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:05.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.916+0000 7f76933fc700 1 Processor -- start 2020-11-20T12:13:05.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.917+0000 7f76933fc700 1 -- start start 2020-11-20T12:13:05.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.918+0000 7f76933fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c0bf800 0x7f768c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:05.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.918+0000 7f76933fc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c14d110 con 0x7f768c0bfbe0 2020-11-20T12:13:05.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.918+0000 7f76933fc700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c14cbd0 con 0x7f768c09f560 2020-11-20T12:13:05.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.918+0000 7f76933fc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c14ce70 con 0x7f768c0bf800 2020-11-20T12:13:05.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.919+0000 7f7692bfb700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f768c09f560 0x7f768c0a36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44062/0 (socket says 172.21.15.187:44062) 2020-11-20T12:13:05.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.919+0000 7f7692bfb700 1 -- 172.21.15.187:0/3700492167 learned_addr learned my addr 172.21.15.187:0/3700492167 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:05.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.919+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939005558 0 0) 0x7f768c14cbd0 con 0x7f768c09f560 2020-11-20T12:13:05.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.919+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7674003240 con 0x7f768c09f560 2020-11-20T12:13:05.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.919+0000 7f76923fa700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c0bf800 0x7f768c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:05.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.920+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3077425767 0 0) 0x7f7674003240 con 0x7f768c09f560 2020-11-20T12:13:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.920+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c0bf800 msgr2=0x7f768c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.920+0000 7f76913f8700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c0bf800 0x7f768c14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.920+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 >> v1:172.21.15.5:6789/0 conn(0x7f768c0bfbe0 legacy=0x7f768c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.920+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0a4cc0 con 0x7f768c09f560 2020-11-20T12:13:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.920+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f7684001860 con 0x7f768c09f560 2020-11-20T12:13:05.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.920+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 >> v1:172.21.15.187:6789/0 conn(0x7f768c09f560 legacy=0x7f768c0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:05.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.920+0000 7f76913f8700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7674003730 0x7f7674003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:05.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.921+0000 7f76913f8700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c0bf800 0x7f76740044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:05.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.921+0000 7f76913f8700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76740049d0 0x7f7674004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:05.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.921+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c14ce70 con 0x7f76740049d0 2020-11-20T12:13:05.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.921+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7674004140 con 0x7f7674003730 2020-11-20T12:13:05.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.921+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7674004300 con 0x7f768c0bf800 2020-11-20T12:13:05.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.921+0000 7f76923fa700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7674003730 0x7f7674003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:05.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.921+0000 7f7691bf9700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c0bf800 0x7f76740044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:05.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f7692bfb700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76740049d0 0x7f7674004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:05.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76923fa700 1 -- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c0bf800 msgr2=0x7f76740044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:05.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76923fa700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c0bf800 0x7f76740044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:05.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76923fa700 1 -- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76740049d0 msgr2=0x7f7674004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:05.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76923fa700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76740049d0 0x7f7674004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:05.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76923fa700 1 -- 172.21.15.187:0/3700492167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7674003240 con 0x7f7674003730 2020-11-20T12:13:05.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76923fa700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7674003730 0x7f7674003b10 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f768800ba80 tx=0x7f7688008040).ready entity=mon.1 client_cookie=3e2a12697f96120f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:05.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768800d9c0 con 0x7f7674003730 2020-11-20T12:13:05.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768800db20 con 0x7f7674003730 2020-11-20T12:13:05.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.922+0000 7f76913f8700 1 -- 172.21.15.187:0/3700492167 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768800dea0 con 0x7f7674003730 2020-11-20T12:13:05.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.923+0000 7f76933fc700 1 -- 172.21.15.187:0/3700492167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7674003730 msgr2=0x7f7674003b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:05.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.923+0000 7f76933fc700 1 --2- 172.21.15.187:0/3700492167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7674003730 0x7f7674003b10 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f768800ba80 tx=0x7f7688008040).stop 2020-11-20T12:13:05.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.923+0000 7f7692bfb700 1 -- 172.21.15.187:0/3700492167 reap_dead start 2020-11-20T12:13:05.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.923+0000 7f76933fc700 1 -- 172.21.15.187:0/3700492167 shutdown_connections 2020-11-20T12:13:05.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.923+0000 7f76933fc700 1 -- 172.21.15.187:0/3700492167 >> 172.21.15.187:0/3700492167 conn(0x7f768c068370 msgr2=0x7f768c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:05.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.923+0000 7f76933fc700 1 -- 172.21.15.187:0/3700492167 shutdown_connections 2020-11-20T12:13:05.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.923+0000 7f76933fc700 1 -- 172.21.15.187:0/3700492167 wait complete. 2020-11-20T12:13:05.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.925+0000 7f76933fc700 1 Processor -- start 2020-11-20T12:13:05.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.925+0000 7f76933fc700 1 -- start start 2020-11-20T12:13:05.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.926+0000 7f76933fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f768c14ae10 0x7f768c0b6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:05.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.926+0000 7f76933fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c14b1f0 0x7f768c1464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:05.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.926+0000 7f76933fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f768c0b5160 0x7f768c141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:05.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76933fc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0b4a30 con 0x7f768c0b5160 2020-11-20T12:13:05.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76933fc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0b3e90 con 0x7f768c14ae10 2020-11-20T12:13:05.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76933fc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c15cb60 con 0x7f768c14b1f0 2020-11-20T12:13:05.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76923fa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f768c14ae10 0x7f768c0b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:05.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76923fa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f768c14ae10 0x7f768c0b6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47466/0 (socket says 172.21.15.187:47466) 2020-11-20T12:13:05.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76923fa700 1 -- 172.21.15.187:0/1749282549 learned_addr learned my addr 172.21.15.187:0/1749282549 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:05.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f7692bfb700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f768c0b5160 0x7f768c141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:05.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f7691bf9700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c14b1f0 0x7f768c1464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:05.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76923fa700 1 -- 172.21.15.187:0/1749282549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c14b1f0 msgr2=0x7f768c1464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:05.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76923fa700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c14b1f0 0x7f768c1464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:05.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76923fa700 1 -- 172.21.15.187:0/1749282549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f768c0b5160 msgr2=0x7f768c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:05.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76923fa700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f768c0b5160 0x7f768c141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:05.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.927+0000 7f76923fa700 1 -- 172.21.15.187:0/1749282549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c1474a0 con 0x7f768c14ae10 2020-11-20T12:13:05.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.928+0000 7f76923fa700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f768c14ae10 0x7f768c0b6780 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f7688012000 tx=0x7f7688008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:05.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.928+0000 7f767a7fc700 1 -- 172.21.15.187:0/1749282549 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688006760 con 0x7f768c14ae10 2020-11-20T12:13:05.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.928+0000 7f767a7fc700 1 -- 172.21.15.187:0/1749282549 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768801d620 con 0x7f768c14ae10 2020-11-20T12:13:05.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.928+0000 7f767a7fc700 1 -- 172.21.15.187:0/1749282549 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768801d9c0 con 0x7f768c14ae10 2020-11-20T12:13:05.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.928+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c0b6c70 con 0x7f768c14ae10 2020-11-20T12:13:05.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.928+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c146b50 con 0x7f768c14ae10 2020-11-20T12:13:05.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.929+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c0b6e90 con 0x7f768c14ae10 2020-11-20T12:13:05.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.930+0000 7f767a7fc700 1 -- 172.21.15.187:0/1749282549 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7688023070 con 0x7f768c14ae10 2020-11-20T12:13:05.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.931+0000 7f767a7fc700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7664056d20 0x7f76640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.931+0000 7f767a7fc700 1 -- 172.21.15.187:0/1749282549 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7688071750 con 0x7f768c14ae10 2020-11-20T12:13:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.931+0000 7f7691bf9700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7664056d20 0x7f76640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.932+0000 7f7691bf9700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7664056d20 0x7f76640591e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f768c15bd30 tx=0x7f767c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:05.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:05.934+0000 7f767a7fc700 1 -- 172.21.15.187:0/1749282549 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7688046ba0 con 0x7f768c14ae10 2020-11-20T12:13:06.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.177+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f768c080050 con 0x7f768c14ae10 2020-11-20T12:13:06.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.178+0000 7f767a7fc700 1 -- 172.21.15.187:0/1749282549 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7f768804a1c0 con 0x7f768c14ae10 2020-11-20T12:13:06.181 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:06.181 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:06.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7664056d20 msgr2=0x7f76640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7664056d20 0x7f76640591e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f768c15bd30 tx=0x7f767c009040).stop 2020-11-20T12:13:06.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f768c14ae10 msgr2=0x7f768c0b6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f768c14ae10 0x7f768c0b6780 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f7688012000 tx=0x7f7688008040).stop 2020-11-20T12:13:06.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 shutdown_connections 2020-11-20T12:13:06.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7664056d20 0x7f76640591e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f768c0b5160 0x7f768c141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f768c14ae10 0x7f768c0b6780 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 --2- 172.21.15.187:0/1749282549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f768c14b1f0 0x7f768c1464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.180+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 >> 172.21.15.187:0/1749282549 conn(0x7f768c068370 msgr2=0x7f768c147ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:06.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.181+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 shutdown_connections 2020-11-20T12:13:06.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:06.181+0000 7f76933fc700 1 -- 172.21.15.187:0/1749282549 wait complete. 2020-11-20T12:13:06.187 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:06.410 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:06.410 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:07.411 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:07.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.659+0000 7f0c851f0700 1 Processor -- start 2020-11-20T12:13:07.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.659+0000 7f0c851f0700 1 -- start start 2020-11-20T12:13:07.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.660+0000 7f0c851f0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0c801484d0 0x7f0c80061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:07.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.660+0000 7f0c851f0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c800b1300 con 0x7f0c800622c0 2020-11-20T12:13:07.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.660+0000 7f0c851f0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c800b15a0 con 0x7f0c80066f70 2020-11-20T12:13:07.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.660+0000 7f0c851f0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c80067730 con 0x7f0c801484d0 2020-11-20T12:13:07.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.661+0000 7f0c849ef700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0c80066f70 0x7f0c800b5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44084/0 (socket says 172.21.15.187:44084) 2020-11-20T12:13:07.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.661+0000 7f0c849ef700 1 -- 172.21.15.187:0/315811451 learned_addr learned my addr 172.21.15.187:0/315811451 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:07.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.661+0000 7f0c7ffff700 1 --2- 172.21.15.187:0/315811451 >> v2:172.21.15.5:3300/0 conn(0x7f0c801484d0 0x7f0c80061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:07.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879460498 0 0) 0x7f0c800b15a0 con 0x7f0c80066f70 2020-11-20T12:13:07.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c60003240 con 0x7f0c80066f70 2020-11-20T12:13:07.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3248176796 0 0) 0x7f0c60003240 con 0x7f0c80066f70 2020-11-20T12:13:07.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 >> v2:172.21.15.5:3300/0 conn(0x7f0c801484d0 msgr2=0x7f0c80061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:07.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 --2- 172.21.15.187:0/315811451 >> v2:172.21.15.5:3300/0 conn(0x7f0c801484d0 0x7f0c80061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:07.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 >> v1:172.21.15.5:6789/0 conn(0x7f0c800622c0 legacy=0x7f0c800b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:07.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c800b1840 con 0x7f0c80066f70 2020-11-20T12:13:07.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f0c78001860 con 0x7f0c80066f70 2020-11-20T12:13:07.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 >> v1:172.21.15.187:6789/0 conn(0x7f0c80066f70 legacy=0x7f0c800b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:07.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60003780 0x7f0c60003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:07.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c600044e0 0x7f0c60004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:07.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c60004e50 0x7f0c6000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:07.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c80067730 con 0x7f0c600044e0 2020-11-20T12:13:07.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c60005300 con 0x7f0c60004e50 2020-11-20T12:13:07.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.662+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6000bc20 con 0x7f0c60003780 2020-11-20T12:13:07.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c849ef700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c60004e50 0x7f0c6000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:07.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c777fe700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c600044e0 0x7f0c60004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:07.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c7ffff700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60003780 0x7f0c60003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:07.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c849ef700 1 -- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60003780 msgr2=0x7f0c60003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:07.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c849ef700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60003780 0x7f0c60003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:07.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c849ef700 1 -- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c600044e0 msgr2=0x7f0c60004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:07.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c849ef700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c600044e0 0x7f0c60004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:07.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c849ef700 1 -- 172.21.15.187:0/315811451 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c60003240 con 0x7f0c60004e50 2020-11-20T12:13:07.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c7ffff700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60003780 0x7f0c60003ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:07.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c777fe700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c600044e0 0x7f0c60004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:07.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c849ef700 1 -- 172.21.15.187:0/315811451 reap_dead start 2020-11-20T12:13:07.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.663+0000 7f0c849ef700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c60004e50 0x7f0c6000b730 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7f0c7800c550 tx=0x7f0c78007ad0).ready entity=mon.1 client_cookie=fea5d9e87a76ce0c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:07.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.664+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c78007b40 con 0x7f0c60004e50 2020-11-20T12:13:07.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.664+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c78005b80 con 0x7f0c60004e50 2020-11-20T12:13:07.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.664+0000 7f0c7f7fe700 1 -- 172.21.15.187:0/315811451 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c780070d0 con 0x7f0c60004e50 2020-11-20T12:13:07.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.664+0000 7f0c851f0700 1 -- 172.21.15.187:0/315811451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c60004e50 msgr2=0x7f0c6000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:07.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.664+0000 7f0c851f0700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c60004e50 0x7f0c6000b730 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7f0c7800c550 tx=0x7f0c78007ad0).stop 2020-11-20T12:13:07.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.664+0000 7f0c851f0700 1 -- 172.21.15.187:0/315811451 shutdown_connections 2020-11-20T12:13:07.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.664+0000 7f0c851f0700 1 --2- 172.21.15.187:0/315811451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c60004e50 0x7f0c6000b730 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:07.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.664+0000 7f0c851f0700 1 -- 172.21.15.187:0/315811451 >> 172.21.15.187:0/315811451 conn(0x7f0c800bab30 msgr2=0x7f0c800baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:07.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.665+0000 7f0c851f0700 1 -- 172.21.15.187:0/315811451 shutdown_connections 2020-11-20T12:13:07.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.665+0000 7f0c851f0700 1 -- 172.21.15.187:0/315811451 wait complete. 2020-11-20T12:13:07.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.666+0000 7f0c851f0700 1 Processor -- start 2020-11-20T12:13:07.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.667+0000 7f0c851f0700 1 -- start start 2020-11-20T12:13:07.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c851f0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60004e50 0x7f0c8013be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:07.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c851f0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c80147950 0x7f0c8013c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:07.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c851f0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8009c030 0x7f0c8009c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:07.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c851f0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8013fcb0 con 0x7f0c8009c030 2020-11-20T12:13:07.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c851f0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c800b49b0 con 0x7f0c80147950 2020-11-20T12:13:07.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c851f0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8013f2d0 con 0x7f0c60004e50 2020-11-20T12:13:07.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c777fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c80147950 0x7f0c8013c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:07.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c777fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c80147950 0x7f0c8013c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47488/0 (socket says 172.21.15.187:47488) 2020-11-20T12:13:07.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c777fe700 1 -- 172.21.15.187:0/1352233365 learned_addr learned my addr 172.21.15.187:0/1352233365 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:07.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c777fe700 1 -- 172.21.15.187:0/1352233365 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60004e50 msgr2=0x7f0c8013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:07.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c777fe700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60004e50 0x7f0c8013be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:07.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c777fe700 1 -- 172.21.15.187:0/1352233365 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8009c030 msgr2=0x7f0c8009c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:07.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c777fe700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8009c030 0x7f0c8009c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:07.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.668+0000 7f0c777fe700 1 -- 172.21.15.187:0/1352233365 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8013f570 con 0x7f0c80147950 2020-11-20T12:13:07.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.669+0000 7f0c777fe700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c80147950 0x7f0c8013c350 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7f0c6800d7b0 tx=0x7f0c68009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:07.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.669+0000 7f0c7cff9700 1 -- 172.21.15.187:0/1352233365 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800f5d0 con 0x7f0c80147950 2020-11-20T12:13:07.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.669+0000 7f0c7cff9700 1 -- 172.21.15.187:0/1352233365 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6800f730 con 0x7f0c80147950 2020-11-20T12:13:07.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.669+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8013f810 con 0x7f0c80147950 2020-11-20T12:13:07.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.669+0000 7f0c7cff9700 1 -- 172.21.15.187:0/1352233365 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800fab0 con 0x7f0c80147950 2020-11-20T12:13:07.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.669+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8013d420 con 0x7f0c80147950 2020-11-20T12:13:07.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.670+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8013f9a0 con 0x7f0c80147950 2020-11-20T12:13:07.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.671+0000 7f0c7cff9700 1 -- 172.21.15.187:0/1352233365 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0c6801c460 con 0x7f0c80147950 2020-11-20T12:13:07.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.672+0000 7f0c7cff9700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c5c056d50 0x7f0c5c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:07.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.672+0000 7f0c7cff9700 1 -- 172.21.15.187:0/1352233365 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0c680700c0 con 0x7f0c80147950 2020-11-20T12:13:07.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.672+0000 7f0c7ffff700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c5c056d50 0x7f0c5c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:07.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.672+0000 7f0c7ffff700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c5c056d50 0x7f0c5c059210 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f0c70006d00 tx=0x7f0c70008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:07.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.675+0000 7f0c7cff9700 1 -- 172.21.15.187:0/1352233365 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0c680453a0 con 0x7f0c80147950 2020-11-20T12:13:07.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.925+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c80061a20 con 0x7f0c80147950 2020-11-20T12:13:07.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.926+0000 7f0c7cff9700 1 -- 172.21.15.187:0/1352233365 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7f0c680489c0 con 0x7f0c80147950 2020-11-20T12:13:07.932 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:07.934 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c5c056d50 msgr2=0x7f0c5c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c5c056d50 0x7f0c5c059210 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f0c70006d00 tx=0x7f0c70008040).stop 2020-11-20T12:13:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c80147950 msgr2=0x7f0c8013c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:07.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c80147950 0x7f0c8013c350 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7f0c6800d7b0 tx=0x7f0c68009040).stop 2020-11-20T12:13:07.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 shutdown_connections 2020-11-20T12:13:07.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c5c056d50 0x7f0c5c059210 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:07.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8009c030 0x7f0c8009c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:07.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c60004e50 0x7f0c8013be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 --2- 172.21.15.187:0/1352233365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c80147950 0x7f0c8013c350 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.928+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 >> 172.21.15.187:0/1352233365 conn(0x7f0c800bab30 msgr2=0x7f0c80066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.929+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 shutdown_connections 2020-11-20T12:13:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:07.929+0000 7f0c851f0700 1 -- 172.21.15.187:0/1352233365 wait complete. 2020-11-20T12:13:07.938 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:08.159 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:08.160 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:09.161 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:09.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.410+0000 7f13d352b700 1 Processor -- start 2020-11-20T12:13:09.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.410+0000 7f13d352b700 1 -- start start 2020-11-20T12:13:09.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.411+0000 7f13d352b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc0b6650 0x7f13cc0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:09.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.411+0000 7f13d352b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f13cc066f70 0x7f13cc0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:09.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.411+0000 7f13d352b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13cc0b1300 con 0x7f13cc1484d0 2020-11-20T12:13:09.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.411+0000 7f13d352b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13cc067730 con 0x7f13cc066f70 2020-11-20T12:13:09.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.411+0000 7f13d352b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc0b15a0 con 0x7f13cc0b6650 2020-11-20T12:13:09.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.412+0000 7f13d2d2a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f13cc066f70 0x7f13cc0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:09.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.412+0000 7f13d2d2a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f13cc066f70 0x7f13cc0aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:39762/0 (socket says 172.21.15.187:39762) 2020-11-20T12:13:09.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.412+0000 7f13d2d2a700 1 -- 172.21.15.187:0/396652300 learned_addr learned my addr 172.21.15.187:0/396652300 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:09.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.412+0000 7f13d1d28700 1 --2- 172.21.15.187:0/396652300 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc0b6650 0x7f13cc0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:09.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.413+0000 7f13d2d2a700 1 -- 172.21.15.187:0/396652300 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc0b6650 msgr2=0x7f13cc0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:09.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.413+0000 7f13d2d2a700 1 --2- 172.21.15.187:0/396652300 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc0b6650 0x7f13cc0afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.413+0000 7f13d2d2a700 1 -- 172.21.15.187:0/396652300 >> v1:172.21.15.5:6789/0 conn(0x7f13cc1484d0 legacy=0x7f13cc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:09.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.413+0000 7f13d2d2a700 1 -- 172.21.15.187:0/396652300 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc0b1840 con 0x7f13cc066f70 2020-11-20T12:13:09.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.414+0000 7f13d2d2a700 1 --2- 172.21.15.187:0/396652300 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc066f70 0x7f13cc0aff80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f13c4009d50 tx=0x7f13c4009290).ready entity=mon.0 client_cookie=af61f56ad2e97305 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:09.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.414+0000 7f13d1527700 1 -- 172.21.15.187:0/396652300 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c400bc80 con 0x7f13cc066f70 2020-11-20T12:13:09.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.414+0000 7f13d1527700 1 -- 172.21.15.187:0/396652300 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c400bde0 con 0x7f13cc066f70 2020-11-20T12:13:09.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.414+0000 7f13d1527700 1 -- 172.21.15.187:0/396652300 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4011610 con 0x7f13cc066f70 2020-11-20T12:13:09.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.415+0000 7f13d352b700 1 -- 172.21.15.187:0/396652300 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc066f70 msgr2=0x7f13cc0aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:09.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.415+0000 7f13d352b700 1 --2- 172.21.15.187:0/396652300 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc066f70 0x7f13cc0aff80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f13c4009d50 tx=0x7f13c4009290).stop 2020-11-20T12:13:09.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.416+0000 7f13d352b700 1 -- 172.21.15.187:0/396652300 shutdown_connections 2020-11-20T12:13:09.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.416+0000 7f13d352b700 1 --2- 172.21.15.187:0/396652300 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc066f70 0x7f13cc0aff80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.416+0000 7f13d352b700 1 --2- 172.21.15.187:0/396652300 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc0b6650 0x7f13cc0afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.416+0000 7f13d352b700 1 -- 172.21.15.187:0/396652300 >> 172.21.15.187:0/396652300 conn(0x7f13cc0bab30 msgr2=0x7f13cc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:09.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.416+0000 7f13d352b700 1 -- 172.21.15.187:0/396652300 shutdown_connections 2020-11-20T12:13:09.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.416+0000 7f13d352b700 1 -- 172.21.15.187:0/396652300 wait complete. 2020-11-20T12:13:09.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.418+0000 7f13d352b700 1 Processor -- start 2020-11-20T12:13:09.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.418+0000 7f13d352b700 1 -- start start 2020-11-20T12:13:09.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.419+0000 7f13d352b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc066f70 0x7f13cc09eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:09.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.419+0000 7f13d352b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc0b6650 0x7f13cc142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:09.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.419+0000 7f13d352b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13cc1484d0 0x7f13cc142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:09.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.419+0000 7f13d352b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc157e40 con 0x7f13cc0b6650 2020-11-20T12:13:09.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.419+0000 7f13d352b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc09e670 con 0x7f13cc1484d0 2020-11-20T12:13:09.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.419+0000 7f13d352b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc09b800 con 0x7f13cc066f70 2020-11-20T12:13:09.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13cc1484d0 0x7f13cc142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:09.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13cc1484d0 0x7f13cc142730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47504/0 (socket says 172.21.15.187:47504) 2020-11-20T12:13:09.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 -- 172.21.15.187:0/4135994171 learned_addr learned my addr 172.21.15.187:0/4135994171 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:09.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2529700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc066f70 0x7f13cc09eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:09.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d1d28700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc0b6650 0x7f13cc142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:09.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 -- 172.21.15.187:0/4135994171 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc066f70 msgr2=0x7f13cc09eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:09.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc066f70 0x7f13cc09eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 -- 172.21.15.187:0/4135994171 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc0b6650 msgr2=0x7f13cc142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:09.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc0b6650 0x7f13cc142240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 -- 172.21.15.187:0/4135994171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc09f000 con 0x7f13cc1484d0 2020-11-20T12:13:09.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.420+0000 7f13d2d2a700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13cc1484d0 0x7f13cc142730 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f13cc066430 tx=0x7f13c4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:09.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.421+0000 7f13be7fc700 1 -- 172.21.15.187:0/4135994171 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c400bc80 con 0x7f13cc1484d0 2020-11-20T12:13:09.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.421+0000 7f13be7fc700 1 -- 172.21.15.187:0/4135994171 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c4009e90 con 0x7f13cc1484d0 2020-11-20T12:13:09.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.421+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13cc09f1f0 con 0x7f13cc1484d0 2020-11-20T12:13:09.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.421+0000 7f13be7fc700 1 -- 172.21.15.187:0/4135994171 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4010e10 con 0x7f13cc1484d0 2020-11-20T12:13:09.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.421+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13cc0a0870 con 0x7f13cc1484d0 2020-11-20T12:13:09.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.422+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13cc09e830 con 0x7f13cc1484d0 2020-11-20T12:13:09.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.423+0000 7f13be7fc700 1 -- 172.21.15.187:0/4135994171 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f13c4017070 con 0x7f13cc1484d0 2020-11-20T12:13:09.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.423+0000 7f13be7fc700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13a4056d50 0x7f13a4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:09.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.423+0000 7f13be7fc700 1 -- 172.21.15.187:0/4135994171 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13c4072110 con 0x7f13cc1484d0 2020-11-20T12:13:09.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.424+0000 7f13d2529700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13a4056d50 0x7f13a4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:09.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.424+0000 7f13d2529700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13a4056d50 0x7f13a4059210 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f13c8003450 tx=0x7f13c8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:09.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.427+0000 7f13be7fc700 1 -- 172.21.15.187:0/4135994171 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f13c40475e0 con 0x7f13cc1484d0 2020-11-20T12:13:09.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.673+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13cc158000 con 0x7f13cc1484d0 2020-11-20T12:13:09.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.674+0000 7f13be7fc700 1 -- 172.21.15.187:0/4135994171 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7f13c4015070 con 0x7f13cc1484d0 2020-11-20T12:13:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:09.677 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:09.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13a4056d50 msgr2=0x7f13a4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:09.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13a4056d50 0x7f13a4059210 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f13c8003450 tx=0x7f13c8009040).stop 2020-11-20T12:13:09.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13cc1484d0 msgr2=0x7f13cc142730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:09.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13cc1484d0 0x7f13cc142730 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f13cc066430 tx=0x7f13c4016000).stop 2020-11-20T12:13:09.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 shutdown_connections 2020-11-20T12:13:09.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13a4056d50 0x7f13a4059210 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13cc1484d0 0x7f13cc142730 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13cc066f70 0x7f13cc09eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 --2- 172.21.15.187:0/4135994171 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13cc0b6650 0x7f13cc142240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:09.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 >> 172.21.15.187:0/4135994171 conn(0x7f13cc0bab30 msgr2=0x7f13cc0b0830 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:09.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.676+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 shutdown_connections 2020-11-20T12:13:09.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:09.677+0000 7f13d352b700 1 -- 172.21.15.187:0/4135994171 wait complete. 2020-11-20T12:13:09.682 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:09.902 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:09.903 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:10.904 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:11.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.154+0000 7fb9572b7700 1 Processor -- start 2020-11-20T12:13:11.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.154+0000 7fb9572b7700 1 -- start start 2020-11-20T12:13:11.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.155+0000 7fb9572b7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb9500b1380 0x7fb950150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:11.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.155+0000 7fb9572b7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9500658a0 con 0x7fb95014c260 2020-11-20T12:13:11.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.155+0000 7fb9572b7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb950150390 con 0x7fb9500b1760 2020-11-20T12:13:11.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.155+0000 7fb9572b7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb950150630 con 0x7fb9500b1380 2020-11-20T12:13:11.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.156+0000 7fb955ab4700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb9500b1760 0x7fb950065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44118/0 (socket says 172.21.15.187:44118) 2020-11-20T12:13:11.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.156+0000 7fb955ab4700 1 -- 172.21.15.187:0/1637658070 learned_addr learned my addr 172.21.15.187:0/1637658070 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:11.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.156+0000 7fb9562b5700 1 --2- 172.21.15.187:0/1637658070 >> v2:172.21.15.5:3300/0 conn(0x7fb9500b1380 0x7fb950150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:11.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.156+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133035801 0 0) 0x7fb950150390 con 0x7fb9500b1760 2020-11-20T12:13:11.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb938003240 con 0x7fb9500b1760 2020-11-20T12:13:11.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3610563225 0 0) 0x7fb938003240 con 0x7fb9500b1760 2020-11-20T12:13:11.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 >> v2:172.21.15.5:3300/0 conn(0x7fb9500b1380 msgr2=0x7fb950150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:11.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 --2- 172.21.15.187:0/1637658070 >> v2:172.21.15.5:3300/0 conn(0x7fb9500b1380 0x7fb950150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 >> v1:172.21.15.5:6789/0 conn(0x7fb95014c260 legacy=0x7fb95014c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb950068510 con 0x7fb9500b1760 2020-11-20T12:13:11.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fb9400014e0 con 0x7fb9500b1760 2020-11-20T12:13:11.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 >> v1:172.21.15.187:6789/0 conn(0x7fb9500b1760 legacy=0x7fb950065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb938003730 0x7fb938003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:11.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9380044e0 0x7fb938004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:11.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb938004e50 0x7fb93800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:11.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb950150630 con 0x7fb938004e50 2020-11-20T12:13:11.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938004260 con 0x7fb9380044e0 2020-11-20T12:13:11.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.157+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93800bc20 con 0x7fb938003730 2020-11-20T12:13:11.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb955ab4700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9380044e0 0x7fb938004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:11.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb9562b5700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb938003730 0x7fb938003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:11.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb955ab4700 1 -- 172.21.15.187:0/1637658070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb938003730 msgr2=0x7fb938003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:11.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb956ab6700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb938004e50 0x7fb93800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:11.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb955ab4700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb938003730 0x7fb938003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb955ab4700 1 -- 172.21.15.187:0/1637658070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb938004e50 msgr2=0x7fb93800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:11.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb955ab4700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb938004e50 0x7fb93800b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb955ab4700 1 -- 172.21.15.187:0/1637658070 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb938003240 con 0x7fb9380044e0 2020-11-20T12:13:11.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb956ab6700 1 -- 172.21.15.187:0/1637658070 reap_dead start 2020-11-20T12:13:11.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.158+0000 7fb955ab4700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9380044e0 0x7fb938004960 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7fb94000c410 tx=0x7fb9400077d0).ready entity=mon.1 client_cookie=5df60aa89094f6d5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:11.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94000ce80 con 0x7fb9380044e0 2020-11-20T12:13:11.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9400072e0 con 0x7fb9380044e0 2020-11-20T12:13:11.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9552b3700 1 -- 172.21.15.187:0/1637658070 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940005800 con 0x7fb9380044e0 2020-11-20T12:13:11.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9572b7700 1 -- 172.21.15.187:0/1637658070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9380044e0 msgr2=0x7fb938004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9572b7700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9380044e0 0x7fb938004960 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7fb94000c410 tx=0x7fb9400077d0).stop 2020-11-20T12:13:11.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9572b7700 1 -- 172.21.15.187:0/1637658070 shutdown_connections 2020-11-20T12:13:11.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9572b7700 1 --2- 172.21.15.187:0/1637658070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9380044e0 0x7fb938004960 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9572b7700 1 -- 172.21.15.187:0/1637658070 >> 172.21.15.187:0/1637658070 conn(0x7fb9500bab30 msgr2=0x7fb9500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:11.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9572b7700 1 -- 172.21.15.187:0/1637658070 shutdown_connections 2020-11-20T12:13:11.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.159+0000 7fb9572b7700 1 -- 172.21.15.187:0/1637658070 wait complete. 2020-11-20T12:13:11.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.161+0000 7fb9572b7700 1 Processor -- start 2020-11-20T12:13:11.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.161+0000 7fb9572b7700 1 -- start start 2020-11-20T12:13:11.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb9572b7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9380044e0 0x7fb9500b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:11.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb9572b7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb95009e6b0 0x7fb95015c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:11.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb9572b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb95009eb30 0x7fb95015f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:11.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb9572b7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500b1b40 con 0x7fb9380044e0 2020-11-20T12:13:11.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb9572b7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95009f030 con 0x7fb95009eb30 2020-11-20T12:13:11.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb9572b7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95009f2d0 con 0x7fb95009e6b0 2020-11-20T12:13:11.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb956ab6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb95009eb30 0x7fb95015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:11.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb956ab6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb95009eb30 0x7fb95015f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47524/0 (socket says 172.21.15.187:47524) 2020-11-20T12:13:11.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.162+0000 7fb956ab6700 1 -- 172.21.15.187:0/4020177356 learned_addr learned my addr 172.21.15.187:0/4020177356 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:11.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb956ab6700 1 -- 172.21.15.187:0/4020177356 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb95009e6b0 msgr2=0x7fb95015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb955ab4700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb95009e6b0 0x7fb95015c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:11.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb9562b5700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9380044e0 0x7fb9500b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:11.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb956ab6700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb95009e6b0 0x7fb95015c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb956ab6700 1 -- 172.21.15.187:0/4020177356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9380044e0 msgr2=0x7fb9500b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb956ab6700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9380044e0 0x7fb9500b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb956ab6700 1 -- 172.21.15.187:0/4020177356 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95009f570 con 0x7fb95009eb30 2020-11-20T12:13:11.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb956ab6700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb95009eb30 0x7fb95015f810 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7fb9500b50b0 tx=0x7fb948009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:11.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb93e7fc700 1 -- 172.21.15.187:0/4020177356 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94800b640 con 0x7fb95009eb30 2020-11-20T12:13:11.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb93e7fc700 1 -- 172.21.15.187:0/4020177356 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94800b7a0 con 0x7fb95009eb30 2020-11-20T12:13:11.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb93e7fc700 1 -- 172.21.15.187:0/4020177356 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94800bb20 con 0x7fb95009eb30 2020-11-20T12:13:11.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95015c4b0 con 0x7fb95009eb30 2020-11-20T12:13:11.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.163+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95015ce80 con 0x7fb95009eb30 2020-11-20T12:13:11.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.164+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95015d160 con 0x7fb95009eb30 2020-11-20T12:13:11.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.165+0000 7fb93e7fc700 1 -- 172.21.15.187:0/4020177356 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb94801a030 con 0x7fb95009eb30 2020-11-20T12:13:11.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.166+0000 7fb93e7fc700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb928056d20 0x7fb9280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:11.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.166+0000 7fb93e7fc700 1 -- 172.21.15.187:0/4020177356 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb94806fb00 con 0x7fb95009eb30 2020-11-20T12:13:11.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.166+0000 7fb9562b5700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb928056d20 0x7fb9280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:11.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.167+0000 7fb9562b5700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb928056d20 0x7fb9280591e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fb95009de30 tx=0x7fb94c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:11.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.170+0000 7fb93e7fc700 1 -- 172.21.15.187:0/4020177356 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb948045000 con 0x7fb95009eb30 2020-11-20T12:13:11.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.416+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb950061a20 con 0x7fb95009eb30 2020-11-20T12:13:11.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.417+0000 7fb93e7fc700 1 -- 172.21.15.187:0/4020177356 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7fb948048620 con 0x7fb95009eb30 2020-11-20T12:13:11.421 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:11.421 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:11.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb928056d20 msgr2=0x7fb9280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb928056d20 0x7fb9280591e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fb95009de30 tx=0x7fb94c008040).stop 2020-11-20T12:13:11.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb95009eb30 msgr2=0x7fb95015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb95009eb30 0x7fb95015f810 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7fb9500b50b0 tx=0x7fb948009040).stop 2020-11-20T12:13:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 shutdown_connections 2020-11-20T12:13:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb928056d20 0x7fb9280591e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb9380044e0 0x7fb9500b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb95009eb30 0x7fb95015f810 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 --2- 172.21.15.187:0/4020177356 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb95009e6b0 0x7fb95015c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 >> 172.21.15.187:0/4020177356 conn(0x7fb9500bab30 msgr2=0x7fb95014e610 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:11.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.420+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 shutdown_connections 2020-11-20T12:13:11.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:11.421+0000 7fb9572b7700 1 -- 172.21.15.187:0/4020177356 wait complete. 2020-11-20T12:13:11.426 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:11.648 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:11.648 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:12.650 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:12.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.895+0000 7f89edd72700 1 Processor -- start 2020-11-20T12:13:12.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.895+0000 7f89edd72700 1 -- start start 2020-11-20T12:13:12.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.897+0000 7f89edd72700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89e8065f90 0x7f89e8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.897+0000 7f89edd72700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f89e809ca90 0x7f89e809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.897+0000 7f89edd72700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80b13e0 con 0x7f89e8065190 2020-11-20T12:13:12.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.897+0000 7f89edd72700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e814ab30 con 0x7f89e809ca90 2020-11-20T12:13:12.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.897+0000 7f89edd72700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e8062c10 con 0x7f89e8065f90 2020-11-20T12:13:12.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.897+0000 7f89ecd70700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f89e8065190 0x7f89e8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44134/0 (socket says 172.21.15.187:44134) 2020-11-20T12:13:12.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.897+0000 7f89ecd70700 1 -- 172.21.15.187:0/4029033514 learned_addr learned my addr 172.21.15.187:0/4029033514 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:12.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.897+0000 7f89dffff700 1 --2- 172.21.15.187:0/4029033514 >> v2:172.21.15.187:3300/0 conn(0x7f89e8065f90 0x7f89e8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185594978 0 0) 0x7f89e80b13e0 con 0x7f89e8065190 2020-11-20T12:13:12.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d4003240 con 0x7f89e8065190 2020-11-20T12:13:12.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1106277144 0 0) 0x7f89d4003240 con 0x7f89e8065190 2020-11-20T12:13:12.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89ed571700 1 --2- 172.21.15.187:0/4029033514 >> v2:172.21.15.5:3300/0 conn(0x7f89e809ca90 0x7f89e809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 >> v2:172.21.15.187:3300/0 conn(0x7f89e8065f90 msgr2=0x7f89e8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:12.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 --2- 172.21.15.187:0/4029033514 >> v2:172.21.15.187:3300/0 conn(0x7f89e8065f90 0x7f89e8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:12.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 >> v2:172.21.15.5:3300/0 conn(0x7f89e809ca90 msgr2=0x7f89e809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:12.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 --2- 172.21.15.187:0/4029033514 >> v2:172.21.15.5:3300/0 conn(0x7f89e809ca90 0x7f89e809cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:12.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e80b1640 con 0x7f89e8065190 2020-11-20T12:13:12.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f89e40014e0 con 0x7f89e8065190 2020-11-20T12:13:12.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 >> v1:172.21.15.187:6789/0 conn(0x7f89e8065190 legacy=0x7f89e8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:12.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d4003730 0x7f89d4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89d40044e0 0x7f89d4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89d4004e50 0x7f89d400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e814ab30 con 0x7f89d4004e50 2020-11-20T12:13:12.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8062c10 con 0x7f89d4003730 2020-11-20T12:13:12.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.898+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d4004260 con 0x7f89d40044e0 2020-11-20T12:13:12.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.899+0000 7f89ed571700 1 --2- 172.21.15.187:0/4029033514 >> v2:172.21.15.5:3300/0 conn(0x7f89e809ca90 0x7f89e809cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:12.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.899+0000 7f89ecd70700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d4003730 0x7f89d4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.899+0000 7f89ed571700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89d4004e50 0x7f89d400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.899+0000 7f89dffff700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89d40044e0 0x7f89d4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.900+0000 7f89ecd70700 1 -- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89d40044e0 msgr2=0x7f89d4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:12.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.900+0000 7f89ecd70700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89d40044e0 0x7f89d4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:12.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.900+0000 7f89ecd70700 1 -- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89d4004e50 msgr2=0x7f89d400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:12.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.900+0000 7f89ecd70700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89d4004e50 0x7f89d400b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:12.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.900+0000 7f89ecd70700 1 -- 172.21.15.187:0/4029033514 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d4003240 con 0x7f89d4003730 2020-11-20T12:13:12.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.900+0000 7f89ed571700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89d4004e50 0x7f89d400b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:12.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.900+0000 7f89ed571700 1 -- 172.21.15.187:0/4029033514 reap_dead start 2020-11-20T12:13:12.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.900+0000 7f89ecd70700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d4003730 0x7f89d4003b50 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f89e400c440 tx=0x7f89e80b0fb0).ready entity=mon.1 client_cookie=51670f3044918f9e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:12.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40072e0 con 0x7f89d4003730 2020-11-20T12:13:12.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e4007440 con 0x7f89d4003730 2020-11-20T12:13:12.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89df7fe700 1 -- 172.21.15.187:0/4029033514 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4005840 con 0x7f89d4003730 2020-11-20T12:13:12.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89edd72700 1 -- 172.21.15.187:0/4029033514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d4003730 msgr2=0x7f89d4003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:12.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89edd72700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d4003730 0x7f89d4003b50 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f89e400c440 tx=0x7f89e80b0fb0).stop 2020-11-20T12:13:12.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89edd72700 1 -- 172.21.15.187:0/4029033514 shutdown_connections 2020-11-20T12:13:12.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89edd72700 1 --2- 172.21.15.187:0/4029033514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d4003730 0x7f89d4003b50 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:12.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89edd72700 1 -- 172.21.15.187:0/4029033514 >> 172.21.15.187:0/4029033514 conn(0x7f89e80bab30 msgr2=0x7f89e80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:12.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89edd72700 1 -- 172.21.15.187:0/4029033514 shutdown_connections 2020-11-20T12:13:12.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.901+0000 7f89edd72700 1 -- 172.21.15.187:0/4029033514 wait complete. 2020-11-20T12:13:12.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.903+0000 7f89edd72700 1 Processor -- start 2020-11-20T12:13:12.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.903+0000 7f89edd72700 1 -- start start 2020-11-20T12:13:12.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89edd72700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e8143a60 0x7f89e8144300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89edd72700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e8143e40 0x7f89e8141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89edd72700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89e81418f0 0x7f89e8142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89edd72700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e81447f0 con 0x7f89e8143a60 2020-11-20T12:13:12.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89edd72700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e809e790 con 0x7f89e81418f0 2020-11-20T12:13:12.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89edd72700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e809ea30 con 0x7f89e8143e40 2020-11-20T12:13:12.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89ed571700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89e81418f0 0x7f89e8142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89ed571700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89e81418f0 0x7f89e8142510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47544/0 (socket says 172.21.15.187:47544) 2020-11-20T12:13:12.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.904+0000 7f89ed571700 1 -- 172.21.15.187:0/362706495 learned_addr learned my addr 172.21.15.187:0/362706495 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:12.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89dffff700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e8143e40 0x7f89e8141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89ecd70700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e8143a60 0x7f89e8144300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89ed571700 1 -- 172.21.15.187:0/362706495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e8143e40 msgr2=0x7f89e8141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:12.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89ed571700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e8143e40 0x7f89e8141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:12.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89ed571700 1 -- 172.21.15.187:0/362706495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e8143a60 msgr2=0x7f89e8144300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:12.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89ed571700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e8143a60 0x7f89e8144300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:12.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89ed571700 1 -- 172.21.15.187:0/362706495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e809ecd0 con 0x7f89e81418f0 2020-11-20T12:13:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89ed571700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89e81418f0 0x7f89e8142510 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f89e809ce70 tx=0x7f89e0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89dcff9700 1 -- 172.21.15.187:0/362706495 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e000c820 con 0x7f89e81418f0 2020-11-20T12:13:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89dcff9700 1 -- 172.21.15.187:0/362706495 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e000c980 con 0x7f89e81418f0 2020-11-20T12:13:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e8141d30 con 0x7f89e81418f0 2020-11-20T12:13:12.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.905+0000 7f89dcff9700 1 -- 172.21.15.187:0/362706495 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e000cd20 con 0x7f89e81418f0 2020-11-20T12:13:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.906+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e809e560 con 0x7f89e81418f0 2020-11-20T12:13:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.907+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e8061a20 con 0x7f89e81418f0 2020-11-20T12:13:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.907+0000 7f89dcff9700 1 -- 172.21.15.187:0/362706495 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f89e0012600 con 0x7f89e81418f0 2020-11-20T12:13:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.908+0000 7f89dcff9700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f89c0056d50 0x7f89c0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:12.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.908+0000 7f89dcff9700 1 -- 172.21.15.187:0/362706495 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f89e006fe20 con 0x7f89e81418f0 2020-11-20T12:13:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.908+0000 7f89ecd70700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f89c0056d50 0x7f89c0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.909+0000 7f89ecd70700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f89c0056d50 0x7f89c0059210 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f89e805f200 tx=0x7f89e4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:12.912+0000 7f89dcff9700 1 -- 172.21.15.187:0/362706495 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f89e0045210 con 0x7f89e81418f0 2020-11-20T12:13:13.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.157+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89e80afc60 con 0x7f89e81418f0 2020-11-20T12:13:13.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.158+0000 7f89dcff9700 1 -- 172.21.15.187:0/362706495 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7f89e0016070 con 0x7f89e81418f0 2020-11-20T12:13:13.162 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:13.162 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:13.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f89c0056d50 msgr2=0x7f89c0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f89c0056d50 0x7f89c0059210 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f89e805f200 tx=0x7f89e4015000).stop 2020-11-20T12:13:13.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89e81418f0 msgr2=0x7f89e8142510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89e81418f0 0x7f89e8142510 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f89e809ce70 tx=0x7f89e0008040).stop 2020-11-20T12:13:13.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 shutdown_connections 2020-11-20T12:13:13.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f89c0056d50 0x7f89c0059210 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e8143a60 0x7f89e8144300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89e81418f0 0x7f89e8142510 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 --2- 172.21.15.187:0/362706495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e8143e40 0x7f89e8141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 >> 172.21.15.187:0/362706495 conn(0x7f89e80bab30 msgr2=0x7f89e8061d00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:13.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.161+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 shutdown_connections 2020-11-20T12:13:13.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:13.162+0000 7f89edd72700 1 -- 172.21.15.187:0/362706495 wait complete. 2020-11-20T12:13:13.167 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:13.392 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:14.341 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:15.344 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:15.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.591+0000 7f116f14c700 1 Processor -- start 2020-11-20T12:13:15.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.591+0000 7f116f14c700 1 -- start start 2020-11-20T12:13:15.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.593+0000 7f116f14c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b1760 0x7f1168065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.593+0000 7f116f14c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1168150390 con 0x7f11680b1380 2020-11-20T12:13:15.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.593+0000 7f116f14c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1168150630 con 0x7f1168065680 2020-11-20T12:13:15.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.593+0000 7f116f14c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11681511e0 con 0x7f11680b1760 2020-11-20T12:13:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.593+0000 7f116e94b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1168065680 0x7f116814c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44158/0 (socket says 172.21.15.187:44158) 2020-11-20T12:13:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.593+0000 7f116e94b700 1 -- 172.21.15.187:0/1579652968 learned_addr learned my addr 172.21.15.187:0/1579652968 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d949700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b1760 0x7f1168065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110094244 0 0) 0x7f1168150630 con 0x7f1168065680 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1150003240 con 0x7f1168065680 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 349057526 0 0) 0x7f1150003240 con 0x7f1168065680 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b1760 msgr2=0x7f1168065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d148700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b1760 0x7f1168065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 >> v1:172.21.15.5:6789/0 conn(0x7f11680b1380 legacy=0x7f1168150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168068510 con 0x7f1168065680 2020-11-20T12:13:15.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.594+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1160001860 con 0x7f1168065680 2020-11-20T12:13:15.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 >> v1:172.21.15.187:6789/0 conn(0x7f1168065680 legacy=0x7f116814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116d148700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1150003730 0x7f1150003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116d148700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b1760 0x7f11500044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116d148700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f1150004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11681511e0 con 0x7f1150003730 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150004220 con 0x7f11500049d0 2020-11-20T12:13:15.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115000b730 con 0x7f11680b1760 2020-11-20T12:13:15.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116e94b700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f1150004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116e14a700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1150003730 0x7f1150003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116e94b700 1 -- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b1760 msgr2=0x7f11500044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116e94b700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b1760 0x7f11500044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116e94b700 1 -- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1150003730 msgr2=0x7f1150003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:15.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116e94b700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1150003730 0x7f1150003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.595+0000 7f116e94b700 1 -- 172.21.15.187:0/1579652968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150003240 con 0x7f11500049d0 2020-11-20T12:13:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.596+0000 7f116e94b700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f1150004e50 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f1160007fd0 tx=0x7f1160007a50).ready entity=mon.1 client_cookie=b4cbb4f3007d6de2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.596+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160005b80 con 0x7f11500049d0 2020-11-20T12:13:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.596+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1160005ce0 con 0x7f11500049d0 2020-11-20T12:13:15.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.596+0000 7f116d148700 1 -- 172.21.15.187:0/1579652968 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160007300 con 0x7f11500049d0 2020-11-20T12:13:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.597+0000 7f116f14c700 1 -- 172.21.15.187:0/1579652968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 msgr2=0x7f1150004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.597+0000 7f116f14c700 1 --2- 172.21.15.187:0/1579652968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f1150004e50 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f1160007fd0 tx=0x7f1160007a50).stop 2020-11-20T12:13:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.597+0000 7f116e94b700 1 -- 172.21.15.187:0/1579652968 reap_dead start 2020-11-20T12:13:15.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.597+0000 7f116f14c700 1 -- 172.21.15.187:0/1579652968 shutdown_connections 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.597+0000 7f116f14c700 1 -- 172.21.15.187:0/1579652968 >> 172.21.15.187:0/1579652968 conn(0x7f11680bab30 msgr2=0x7f11680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.597+0000 7f116f14c700 1 -- 172.21.15.187:0/1579652968 shutdown_connections 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.597+0000 7f116f14c700 1 -- 172.21.15.187:0/1579652968 wait complete. 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.599+0000 7f116f14c700 1 Processor -- start 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.599+0000 7f116f14c700 1 -- start start 2020-11-20T12:13:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116f14c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f116814b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116f14c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11680b7650 0x7f11680b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116f14c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b2a30 0x7f11680b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116f14c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116814be30 con 0x7f11680b7650 2020-11-20T12:13:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116f14c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11680b33a0 con 0x7f11500049d0 2020-11-20T12:13:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116f14c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1168160210 con 0x7f11680b2a30 2020-11-20T12:13:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116e14a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f116814b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116e14a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f116814b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47560/0 (socket says 172.21.15.187:47560) 2020-11-20T12:13:15.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.600+0000 7f116e14a700 1 -- 172.21.15.187:0/3402879661 learned_addr learned my addr 172.21.15.187:0/3402879661 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:15.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116e94b700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b2a30 0x7f11680b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116d949700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11680b7650 0x7f11680b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116e14a700 1 -- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b2a30 msgr2=0x7f11680b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116e14a700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b2a30 0x7f11680b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116e14a700 1 -- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11680b7650 msgr2=0x7f11680b7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116e14a700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11680b7650 0x7f11680b7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116e14a700 1 -- 172.21.15.187:0/3402879661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11680b71b0 con 0x7f11500049d0 2020-11-20T12:13:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116e94b700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b2a30 0x7f11680b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116d949700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11680b7650 0x7f11680b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:15.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116e14a700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f116814b940 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f116400d780 tx=0x7f1164009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:15.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f11567fc700 1 -- 172.21.15.187:0/3402879661 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400f5d0 con 0x7f11500049d0 2020-11-20T12:13:15.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f11567fc700 1 -- 172.21.15.187:0/3402879661 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116400f730 con 0x7f11500049d0 2020-11-20T12:13:15.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f11567fc700 1 -- 172.21.15.187:0/3402879661 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400fab0 con 0x7f11500049d0 2020-11-20T12:13:15.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.601+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116814f350 con 0x7f11500049d0 2020-11-20T12:13:15.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.602+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116815f620 con 0x7f11500049d0 2020-11-20T12:13:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.603+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11680b6fe0 con 0x7f11500049d0 2020-11-20T12:13:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.603+0000 7f11567fc700 1 -- 172.21.15.187:0/3402879661 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1164013470 con 0x7f11500049d0 2020-11-20T12:13:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.604+0000 7f11567fc700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1140056d20 0x7f11400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.604+0000 7f11567fc700 1 -- 172.21.15.187:0/3402879661 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f116406fc80 con 0x7f11500049d0 2020-11-20T12:13:15.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.604+0000 7f116d949700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1140056d20 0x7f11400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.605+0000 7f116d949700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1140056d20 0x7f11400591e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f1158006bd0 tx=0x7f1158008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:15.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.608+0000 7f11567fc700 1 -- 172.21.15.187:0/3402879661 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1164044f60 con 0x7f11500049d0 2020-11-20T12:13:15.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.854+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1168160350 con 0x7f11500049d0 2020-11-20T12:13:15.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.854+0000 7f11567fc700 1 -- 172.21.15.187:0/3402879661 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7f1164007bf0 con 0x7f11500049d0 2020-11-20T12:13:15.859 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:15.859 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:15.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1140056d20 msgr2=0x7f11400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1140056d20 0x7f11400591e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f1158006bd0 tx=0x7f1158008040).stop 2020-11-20T12:13:15.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 msgr2=0x7f116814b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f116814b940 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f116400d780 tx=0x7f1164009040).stop 2020-11-20T12:13:15.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 shutdown_connections 2020-11-20T12:13:15.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1140056d20 0x7f11400591e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11680b2a30 0x7f11680b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11500049d0 0x7f116814b940 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 --2- 172.21.15.187:0/3402879661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11680b7650 0x7f11680b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.857+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 >> 172.21.15.187:0/3402879661 conn(0x7f11680bab30 msgr2=0x7f116809dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:15.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.858+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 shutdown_connections 2020-11-20T12:13:15.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:15.858+0000 7f116f14c700 1 -- 172.21.15.187:0/3402879661 wait complete. 2020-11-20T12:13:15.864 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:16.086 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:16.086 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:17.088 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:17.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.338+0000 7fd783fff700 1 Processor -- start 2020-11-20T12:13:17.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.338+0000 7fd783fff700 1 -- start start 2020-11-20T12:13:17.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.339+0000 7fd783fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd784067020 0x7fd78409eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.340+0000 7fd783fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067400 0x7fd784145a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.340+0000 7fd783fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd784062660 con 0x7fd78414a5d0 2020-11-20T12:13:17.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.340+0000 7fd783fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd784062900 con 0x7fd784067020 2020-11-20T12:13:17.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.340+0000 7fd783fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd784060b20 con 0x7fd784067400 2020-11-20T12:13:17.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.341+0000 7fd7827fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067400 0x7fd784145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.341+0000 7fd782ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd784067020 0x7fd78409eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.341+0000 7fd7827fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067400 0x7fd784145a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:48458/0 (socket says 172.21.15.187:48458) 2020-11-20T12:13:17.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.341+0000 7fd7827fc700 1 -- 172.21.15.187:0/2123213006 learned_addr learned my addr 172.21.15.187:0/2123213006 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:17.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.342+0000 7fd7827fc700 1 -- 172.21.15.187:0/2123213006 >> v2:172.21.15.5:3300/0 conn(0x7fd784067020 msgr2=0x7fd78409eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:17.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.342+0000 7fd7827fc700 1 --2- 172.21.15.187:0/2123213006 >> v2:172.21.15.5:3300/0 conn(0x7fd784067020 0x7fd78409eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.342+0000 7fd7827fc700 1 -- 172.21.15.187:0/2123213006 >> v1:172.21.15.5:6789/0 conn(0x7fd78414a5d0 legacy=0x7fd784145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.342+0000 7fd7827fc700 1 -- 172.21.15.187:0/2123213006 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78414aad0 con 0x7fd784067400 2020-11-20T12:13:17.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.342+0000 7fd7827fc700 1 --2- 172.21.15.187:0/2123213006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067400 0x7fd784145a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fd778006fd0 tx=0x7fd778006eb0).ready entity=mon.2 client_cookie=2b5b42ca3b8f5c29 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:17.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.343+0000 7fd781ffb700 1 -- 172.21.15.187:0/2123213006 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd778009810 con 0x7fd784067400 2020-11-20T12:13:17.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.343+0000 7fd781ffb700 1 -- 172.21.15.187:0/2123213006 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd778009970 con 0x7fd784067400 2020-11-20T12:13:17.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.343+0000 7fd781ffb700 1 -- 172.21.15.187:0/2123213006 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd778009cf0 con 0x7fd784067400 2020-11-20T12:13:17.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.344+0000 7fd783fff700 1 -- 172.21.15.187:0/2123213006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067400 msgr2=0x7fd784145a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.344+0000 7fd783fff700 1 --2- 172.21.15.187:0/2123213006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067400 0x7fd784145a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fd778006fd0 tx=0x7fd778006eb0).stop 2020-11-20T12:13:17.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.344+0000 7fd783fff700 1 -- 172.21.15.187:0/2123213006 shutdown_connections 2020-11-20T12:13:17.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.344+0000 7fd783fff700 1 --2- 172.21.15.187:0/2123213006 >> v2:172.21.15.5:3300/0 conn(0x7fd784067020 0x7fd78409eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.344+0000 7fd783fff700 1 --2- 172.21.15.187:0/2123213006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067400 0x7fd784145a30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.344+0000 7fd783fff700 1 -- 172.21.15.187:0/2123213006 >> 172.21.15.187:0/2123213006 conn(0x7fd7840bb3d0 msgr2=0x7fd7840bb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:17.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.344+0000 7fd783fff700 1 -- 172.21.15.187:0/2123213006 shutdown_connections 2020-11-20T12:13:17.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.344+0000 7fd783fff700 1 -- 172.21.15.187:0/2123213006 wait complete. 2020-11-20T12:13:17.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.345+0000 7fd783fff700 1 Processor -- start 2020-11-20T12:13:17.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.346+0000 7fd783fff700 1 -- start start 2020-11-20T12:13:17.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.346+0000 7fd783fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067020 0x7fd784158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.346+0000 7fd783fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd784067400 0x7fd784144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.346+0000 7fd783fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78414a5d0 0x7fd784141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.346+0000 7fd783fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd784142300 con 0x7fd784067400 2020-11-20T12:13:17.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.346+0000 7fd783fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7841445b0 con 0x7fd78414a5d0 2020-11-20T12:13:17.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.346+0000 7fd783fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd784144850 con 0x7fd784067020 2020-11-20T12:13:17.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78414a5d0 0x7fd784141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78414a5d0 0x7fd784141e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47578/0 (socket says 172.21.15.187:47578) 2020-11-20T12:13:17.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 -- 172.21.15.187:0/4099203778 learned_addr learned my addr 172.21.15.187:0/4099203778 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:17.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd782ffd700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067020 0x7fd784158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7827fc700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd784067400 0x7fd784144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 -- 172.21.15.187:0/4099203778 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067020 msgr2=0x7fd784158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067020 0x7fd784158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 -- 172.21.15.187:0/4099203778 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd784067400 msgr2=0x7fd784144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd784067400 0x7fd784144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 -- 172.21.15.187:0/4099203778 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd784144af0 con 0x7fd78414a5d0 2020-11-20T12:13:17.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.347+0000 7fd7837fe700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78414a5d0 0x7fd784141e10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7fd77c00c810 tx=0x7fd77c007f20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:17.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.348+0000 7fd76affd700 1 -- 172.21.15.187:0/4099203778 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c00a640 con 0x7fd78414a5d0 2020-11-20T12:13:17.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.348+0000 7fd76affd700 1 -- 172.21.15.187:0/4099203778 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77c00a7a0 con 0x7fd78414a5d0 2020-11-20T12:13:17.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.348+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd784142b60 con 0x7fd78414a5d0 2020-11-20T12:13:17.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.348+0000 7fd76affd700 1 -- 172.21.15.187:0/4099203778 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c00ab20 con 0x7fd78414a5d0 2020-11-20T12:13:17.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.348+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd784135780 con 0x7fd78414a5d0 2020-11-20T12:13:17.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.349+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd784144400 con 0x7fd78414a5d0 2020-11-20T12:13:17.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.349+0000 7fd76affd700 1 -- 172.21.15.187:0/4099203778 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd77c010070 con 0x7fd78414a5d0 2020-11-20T12:13:17.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.350+0000 7fd76affd700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd760056d20 0x7fd7600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.350+0000 7fd76affd700 1 -- 172.21.15.187:0/4099203778 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd77c06fc60 con 0x7fd78414a5d0 2020-11-20T12:13:17.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.350+0000 7fd782ffd700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd760056d20 0x7fd7600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.351+0000 7fd782ffd700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd760056d20 0x7fd7600591e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd774006bd0 tx=0x7fd774008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:17.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.354+0000 7fd76affd700 1 -- 172.21.15.187:0/4099203778 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd77c0450b0 con 0x7fd78414a5d0 2020-11-20T12:13:17.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.593+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7840b8fd0 con 0x7fd78414a5d0 2020-11-20T12:13:17.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.594+0000 7fd76affd700 1 -- 172.21.15.187:0/4099203778 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2155 (secure 0 0 0) 0x7fd77c0486d0 con 0x7fd78414a5d0 2020-11-20T12:13:17.598 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:17.599 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:12:59.217000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6952,"mds_1":5159},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5159":{"gid":5159,"name":"b","rank":1,"incarnation":66,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/1084867844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1084867844},{"type":"v1","addr":"172.21.15.5:6833","nonce":1084867844}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:17.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd760056d20 msgr2=0x7fd7600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd760056d20 0x7fd7600591e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd774006bd0 tx=0x7fd774008040).stop 2020-11-20T12:13:17.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78414a5d0 msgr2=0x7fd784141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78414a5d0 0x7fd784141e10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7fd77c00c810 tx=0x7fd77c007f20).stop 2020-11-20T12:13:17.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 shutdown_connections 2020-11-20T12:13:17.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd760056d20 0x7fd7600591e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78414a5d0 0x7fd784141e10 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd784067020 0x7fd784158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 --2- 172.21.15.187:0/4099203778 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd784067400 0x7fd784144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 >> 172.21.15.187:0/4099203778 conn(0x7fd7840bb3d0 msgr2=0x7fd7840bb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:17.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 shutdown_connections 2020-11-20T12:13:17.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:17.596+0000 7fd783fff700 1 -- 172.21.15.187:0/4099203778 wait complete. 2020-11-20T12:13:17.603 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 71 2020-11-20T12:13:17.821 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:12:59.217000+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 6952, 'mds_1': 5159}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5159': {'gid': 5159, 'name': 'b', 'rank': 1, 'incarnation': 66, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/1084867844', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 1084867844}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 1084867844}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:17.821 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5159:up:stopping 2020-11-20T12:13:18.823 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:19.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.069+0000 7f3860ea6700 1 Processor -- start 2020-11-20T12:13:19.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.070+0000 7f3860ea6700 1 -- start start 2020-11-20T12:13:19.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.071+0000 7f3860ea6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c0b6650 0x7f385c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:19.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.071+0000 7f3860ea6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f385c066f70 0x7f385c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:19.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.071+0000 7f3860ea6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385c0b1300 con 0x7f385c1484d0 2020-11-20T12:13:19.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.071+0000 7f3860ea6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f385c067730 con 0x7f385c066f70 2020-11-20T12:13:19.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.071+0000 7f3860ea6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0b15a0 con 0x7f385c0b6650 2020-11-20T12:13:19.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.072+0000 7f385bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f385c066f70 0x7f385c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:19.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.072+0000 7f3852ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c0b6650 0x7f385c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:19.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.072+0000 7f385bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f385c066f70 0x7f385c0aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:39848/0 (socket says 172.21.15.187:39848) 2020-11-20T12:13:19.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.072+0000 7f385bfff700 1 -- 172.21.15.187:0/1739000022 learned_addr learned my addr 172.21.15.187:0/1739000022 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:19.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.073+0000 7f385bfff700 1 -- 172.21.15.187:0/1739000022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c0b6650 msgr2=0x7f385c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:19.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.073+0000 7f385bfff700 1 --2- 172.21.15.187:0/1739000022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c0b6650 0x7f385c0afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:19.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.073+0000 7f385bfff700 1 -- 172.21.15.187:0/1739000022 >> v1:172.21.15.5:6789/0 conn(0x7f385c1484d0 legacy=0x7f385c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:19.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.073+0000 7f385bfff700 1 -- 172.21.15.187:0/1739000022 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c0b1840 con 0x7f385c066f70 2020-11-20T12:13:19.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.074+0000 7f385bfff700 1 --2- 172.21.15.187:0/1739000022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c066f70 0x7f385c0aff80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3854009d50 tx=0x7f3854009290).ready entity=mon.0 client_cookie=dcf17bac701a787c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:19.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.074+0000 7f385affd700 1 -- 172.21.15.187:0/1739000022 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400bbc0 con 0x7f385c066f70 2020-11-20T12:13:19.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.074+0000 7f385affd700 1 -- 172.21.15.187:0/1739000022 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385400bd20 con 0x7f385c066f70 2020-11-20T12:13:19.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.074+0000 7f385affd700 1 -- 172.21.15.187:0/1739000022 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854011610 con 0x7f385c066f70 2020-11-20T12:13:19.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.075+0000 7f3860ea6700 1 -- 172.21.15.187:0/1739000022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c066f70 msgr2=0x7f385c0aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:19.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.075+0000 7f3860ea6700 1 --2- 172.21.15.187:0/1739000022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c066f70 0x7f385c0aff80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3854009d50 tx=0x7f3854009290).stop 2020-11-20T12:13:19.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.075+0000 7f3860ea6700 1 -- 172.21.15.187:0/1739000022 shutdown_connections 2020-11-20T12:13:19.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.075+0000 7f3860ea6700 1 --2- 172.21.15.187:0/1739000022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c066f70 0x7f385c0aff80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:19.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.075+0000 7f3860ea6700 1 --2- 172.21.15.187:0/1739000022 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c0b6650 0x7f385c0afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:19.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.075+0000 7f3860ea6700 1 -- 172.21.15.187:0/1739000022 >> 172.21.15.187:0/1739000022 conn(0x7f385c0bab30 msgr2=0x7f385c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:19.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.075+0000 7f3860ea6700 1 -- 172.21.15.187:0/1739000022 shutdown_connections 2020-11-20T12:13:19.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.075+0000 7f3860ea6700 1 -- 172.21.15.187:0/1739000022 wait complete. 2020-11-20T12:13:19.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.076+0000 7f3860ea6700 1 Processor -- start 2020-11-20T12:13:19.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.077+0000 7f3860ea6700 1 -- start start 2020-11-20T12:13:19.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.077+0000 7f3860ea6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f385c066f70 0x7f385c13aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:19.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.077+0000 7f3860ea6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c0b6650 0x7f385c13af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:19.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.077+0000 7f3860ea6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c1484d0 0x7f385c09bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:19.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.077+0000 7f3860ea6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c13b520 con 0x7f385c0b6650 2020-11-20T12:13:19.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.077+0000 7f3860ea6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c13b7c0 con 0x7f385c066f70 2020-11-20T12:13:19.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.077+0000 7f3860ea6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c13ba60 con 0x7f385c1484d0 2020-11-20T12:13:19.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f385c066f70 0x7f385c13aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:19.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f385c066f70 0x7f385c13aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47588/0 (socket says 172.21.15.187:47588) 2020-11-20T12:13:19.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385b7fe700 1 -- 172.21.15.187:0/2881934571 learned_addr learned my addr 172.21.15.187:0/2881934571 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:19.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385bfff700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c1484d0 0x7f385c09bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:19.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f3852ffd700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c0b6650 0x7f385c13af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:19.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385b7fe700 1 -- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c1484d0 msgr2=0x7f385c09bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:19.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385b7fe700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c1484d0 0x7f385c09bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:19.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385b7fe700 1 -- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c0b6650 msgr2=0x7f385c13af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:19.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385b7fe700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c0b6650 0x7f385c13af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:19.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f385b7fe700 1 -- 172.21.15.187:0/2881934571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c138c20 con 0x7f385c066f70 2020-11-20T12:13:19.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.078+0000 7f3852ffd700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c0b6650 0x7f385c13af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:19.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.079+0000 7f385b7fe700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f385c066f70 0x7f385c13aa90 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7f384c00c6e0 tx=0x7f384c007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:19.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.079+0000 7f3853fff700 1 -- 172.21.15.187:0/2881934571 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c007020 con 0x7f385c066f70 2020-11-20T12:13:19.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.079+0000 7f3853fff700 1 -- 172.21.15.187:0/2881934571 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c007180 con 0x7f385c066f70 2020-11-20T12:13:19.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.079+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c13be00 con 0x7f385c066f70 2020-11-20T12:13:19.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.079+0000 7f3853fff700 1 -- 172.21.15.187:0/2881934571 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c00e560 con 0x7f385c066f70 2020-11-20T12:13:19.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.079+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385c1385a0 con 0x7f385c066f70 2020-11-20T12:13:19.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.080+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385c13c150 con 0x7f385c066f70 2020-11-20T12:13:19.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.080+0000 7f3853fff700 1 -- 172.21.15.187:0/2881934571 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f384c010070 con 0x7f385c066f70 2020-11-20T12:13:19.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.081+0000 7f3853fff700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3838056d20 0x7f38380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:19.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.081+0000 7f3853fff700 1 -- 172.21.15.187:0/2881934571 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f384c06f310 con 0x7f385c066f70 2020-11-20T12:13:19.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.081+0000 7f3852ffd700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3838056d20 0x7f38380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:19.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.082+0000 7f3852ffd700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3838056d20 0x7f38380591e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f3844006bd0 tx=0x7f3844008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:19.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.085+0000 7f3853fff700 1 -- 172.21.15.187:0/2881934571 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f384c0445f0 con 0x7f385c066f70 2020-11-20T12:13:19.309 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:13:20.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.337+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f385c13bc20 con 0x7f385c066f70 2020-11-20T12:13:20.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.338+0000 7f3853fff700 1 -- 172.21.15.187:0/2881934571 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+1789 (secure 0 0 0) 0x7f384c047c10 con 0x7f385c066f70 2020-11-20T12:13:20.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3838056d20 msgr2=0x7f38380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3838056d20 0x7f38380591e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f3844006bd0 tx=0x7f3844008040).stop 2020-11-20T12:13:20.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f385c066f70 msgr2=0x7f385c13aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f385c066f70 0x7f385c13aa90 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7f384c00c6e0 tx=0x7f384c007c50).stop 2020-11-20T12:13:20.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 shutdown_connections 2020-11-20T12:13:20.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3838056d20 0x7f38380591e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f385c1484d0 0x7f385c09bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f385c066f70 0x7f385c13aa90 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 --2- 172.21.15.187:0/2881934571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f385c0b6650 0x7f385c13af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 >> 172.21.15.187:0/2881934571 conn(0x7f385c0bab30 msgr2=0x7f385c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:20.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 shutdown_connections 2020-11-20T12:13:20.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:19.341+0000 7f3860ea6700 1 -- 172.21.15.187:0/2881934571 wait complete. 2020-11-20T12:13:20.656 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 72 2020-11-20T12:13:20.657 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:20.657 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:13:19.215710+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6952},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:20.859 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:12:10.314184+0000', 'modified': '2020-11-20T12:13:19.215710+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6952}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6952': {'gid': 6952, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1106173813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1106173813}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1106173813}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:20.860 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:13:20.861 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:13:21.142 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":6952,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":70.799384017999998,"mdsmap_epoch":72,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":77.344967378000007} 2020-11-20T12:13:21.143 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6952, "mdsmap_epoch": 72, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 70.799384018, "state": "up:active", "uptime": 77.344967378, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:13:21.144 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:13:21.191 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:13:21.200 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:13:21.405 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_created":1}} 2020-11-20T12:13:21.406 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2020-11-20T12:13:21.406 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:13:21.407 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:13:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:13:21.547 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:13:21.657 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:13:21.676 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:13:21.677 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:13:21.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.920+0000 7ff0289d9700 1 Processor -- start 2020-11-20T12:13:21.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.920+0000 7ff0289d9700 1 -- start start 2020-11-20T12:13:21.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.921+0000 7ff0289d9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0240bb840 0x7ff0240655c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff0289d9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240bbc20 0x7ff024065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff0289d9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff0240b7580 0x7ff024148e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff0289d9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff024066b40 con 0x7ff0240b7580 2020-11-20T12:13:21.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff0289d9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff024148980 con 0x7ff0240bb840 2020-11-20T12:13:21.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff0289d9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240621c0 con 0x7ff0240bbc20 2020-11-20T12:13:21.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff0237fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0240bb840 0x7ff0240655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff0237fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0240bb840 0x7ff0240655c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47600/0 (socket says 172.21.15.187:47600) 2020-11-20T12:13:21.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff0237fe700 1 -- 172.21.15.187:0/1867680332 learned_addr learned my addr 172.21.15.187:0/1867680332 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:21.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff01bfff700 1 --2- 172.21.15.187:0/1867680332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240bbc20 0x7ff024065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.922+0000 7ff023fff700 1 --2- 172.21.15.187:0/1867680332 >> v2:172.21.15.5:3300/0 conn(0x7ff0240b7580 0x7ff024148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.923+0000 7ff0237fe700 1 -- 172.21.15.187:0/1867680332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240bbc20 msgr2=0x7ff024065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.923+0000 7ff0237fe700 1 --2- 172.21.15.187:0/1867680332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240bbc20 0x7ff024065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.923+0000 7ff0237fe700 1 -- 172.21.15.187:0/1867680332 >> v2:172.21.15.5:3300/0 conn(0x7ff0240b7580 msgr2=0x7ff024148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.923+0000 7ff0237fe700 1 --2- 172.21.15.187:0/1867680332 >> v2:172.21.15.5:3300/0 conn(0x7ff0240b7580 0x7ff024148e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.923+0000 7ff0237fe700 1 -- 172.21.15.187:0/1867680332 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024062330 con 0x7ff0240bb840 2020-11-20T12:13:21.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.923+0000 7ff0237fe700 1 --2- 172.21.15.187:0/1867680332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240bb840 0x7ff0240655c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7ff024068400 tx=0x7ff014006eb0).ready entity=mon.1 client_cookie=8d8ae611fde413ef server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:21.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.923+0000 7ff022ffd700 1 -- 172.21.15.187:0/1867680332 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff014009720 con 0x7ff0240bb840 2020-11-20T12:13:21.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff022ffd700 1 -- 172.21.15.187:0/1867680332 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff014009880 con 0x7ff0240bb840 2020-11-20T12:13:21.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff022ffd700 1 -- 172.21.15.187:0/1867680332 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff014009c00 con 0x7ff0240bb840 2020-11-20T12:13:21.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 -- 172.21.15.187:0/1867680332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240bb840 msgr2=0x7ff0240655c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:21.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1867680332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240bb840 0x7ff0240655c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7ff024068400 tx=0x7ff014006eb0).stop 2020-11-20T12:13:21.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 -- 172.21.15.187:0/1867680332 shutdown_connections 2020-11-20T12:13:21.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1867680332 >> v2:172.21.15.5:3300/0 conn(0x7ff0240b7580 0x7ff024148e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1867680332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240bb840 0x7ff0240655c0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1867680332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240bbc20 0x7ff024065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 -- 172.21.15.187:0/1867680332 >> 172.21.15.187:0/1867680332 conn(0x7ff024060bf0 msgr2=0x7ff0240baf70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:21.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 -- 172.21.15.187:0/1867680332 shutdown_connections 2020-11-20T12:13:21.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.924+0000 7ff0289d9700 1 -- 172.21.15.187:0/1867680332 wait complete. 2020-11-20T12:13:21.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.926+0000 7ff0289d9700 1 Processor -- start 2020-11-20T12:13:21.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.927+0000 7ff0289d9700 1 -- start start 2020-11-20T12:13:21.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.928+0000 7ff0289d9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240b7580 0x7ff0240b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.928+0000 7ff0289d9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240b6980 0x7ff0241151a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.928+0000 7ff0289d9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff024115690 0x7ff024115ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.928+0000 7ff0289d9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff024137810 con 0x7ff024115690 2020-11-20T12:13:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.928+0000 7ff0289d9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff024115fc0 con 0x7ff0240b6980 2020-11-20T12:13:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.928+0000 7ff0289d9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff024137dd0 con 0x7ff0240b7580 2020-11-20T12:13:21.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.928+0000 7ff01bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240b6980 0x7ff0241151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff0237fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240b7580 0x7ff0240b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240b6980 0x7ff0241151a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47604/0 (socket says 172.21.15.187:47604) 2020-11-20T12:13:21.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01bfff700 1 -- 172.21.15.187:0/1613556210 learned_addr learned my addr 172.21.15.187:0/1613556210 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:21.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff023fff700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff024115690 0x7ff024115ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01bfff700 1 -- 172.21.15.187:0/1613556210 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240b7580 msgr2=0x7ff0240b6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01bfff700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240b7580 0x7ff0240b6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01bfff700 1 -- 172.21.15.187:0/1613556210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff024115690 msgr2=0x7ff024115ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:21.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01bfff700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff024115690 0x7ff024115ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01bfff700 1 -- 172.21.15.187:0/1613556210 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024138070 con 0x7ff0240b6980 2020-11-20T12:13:21.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01bfff700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240b6980 0x7ff0241151a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7ff00c00ec00 tx=0x7ff00c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:21.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.929+0000 7ff01b7fe700 1 -- 172.21.15.187:0/1613556210 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c00b8d0 con 0x7ff0240b6980 2020-11-20T12:13:21.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.930+0000 7ff01b7fe700 1 -- 172.21.15.187:0/1613556210 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c00ba30 con 0x7ff0240b6980 2020-11-20T12:13:21.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.930+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff024138310 con 0x7ff0240b6980 2020-11-20T12:13:21.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.930+0000 7ff01b7fe700 1 -- 172.21.15.187:0/1613556210 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c00bdb0 con 0x7ff0240b6980 2020-11-20T12:13:21.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.930+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0241439a0 con 0x7ff0240b6980 2020-11-20T12:13:21.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.931+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff024137bd0 con 0x7ff0240b6980 2020-11-20T12:13:21.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.931+0000 7ff01b7fe700 1 -- 172.21.15.187:0/1613556210 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff00c012600 con 0x7ff0240b6980 2020-11-20T12:13:21.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.932+0000 7ff01b7fe700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d20 0x7ff0000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.932+0000 7ff01b7fe700 1 -- 172.21.15.187:0/1613556210 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff00c0701e0 con 0x7ff0240b6980 2020-11-20T12:13:21.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.933+0000 7ff0237fe700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d20 0x7ff0000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.933+0000 7ff0237fe700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d20 0x7ff0000591e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7ff024065130 tx=0x7ff014015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:21.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:21.936+0000 7ff01b7fe700 1 -- 172.21.15.187:0/1613556210 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff00c045630 con 0x7ff0240b6980 2020-11-20T12:13:22.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.125+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff0240446d0 con 0x7ff0240b6980 2020-11-20T12:13:22.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.135+0000 7ff01b7fe700 1 -- 172.21.15.187:0/1613556210 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff00c048c50 con 0x7ff0240b6980 2020-11-20T12:13:22.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.137+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d20 msgr2=0x7ff0000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:22.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.137+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d20 0x7ff0000591e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7ff024065130 tx=0x7ff014015000).stop 2020-11-20T12:13:22.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.137+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240b6980 msgr2=0x7ff0241151a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:22.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.137+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240b6980 0x7ff0241151a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7ff00c00ec00 tx=0x7ff00c006df0).stop 2020-11-20T12:13:22.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.138+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 shutdown_connections 2020-11-20T12:13:22.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.138+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d20 0x7ff0000591e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.138+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff024115690 0x7ff024115ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.138+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0240b7580 0x7ff0240b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.138+0000 7ff0289d9700 1 --2- 172.21.15.187:0/1613556210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0240b6980 0x7ff0241151a0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.138+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 >> 172.21.15.187:0/1613556210 conn(0x7ff024060bf0 msgr2=0x7ff024142de0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:22.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.138+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 shutdown_connections 2020-11-20T12:13:22.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.138+0000 7ff0289d9700 1 -- 172.21.15.187:0/1613556210 wait complete. 2020-11-20T12:13:22.166 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2020-11-20T12:13:22.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.403+0000 7f5cef3c5700 1 Processor -- start 2020-11-20T12:13:22.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.403+0000 7f5cef3c5700 1 -- start start 2020-11-20T12:13:22.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cef3c5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ce80b1380 0x7f5ce8150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:22.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cef3c5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 0x7f5ce8065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:22.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cef3c5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5ce8065680 0x7f5ce8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:22.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cef3c5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce8150390 con 0x7f5ce8065680 2020-11-20T12:13:22.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cef3c5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce8150630 con 0x7f5ce80b1380 2020-11-20T12:13:22.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cef3c5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8068510 con 0x7f5ce80b1760 2020-11-20T12:13:22.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cee3c3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ce80b1380 0x7f5ce8150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:22.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cee3c3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ce80b1380 0x7f5ce8150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47610/0 (socket says 172.21.15.187:47610) 2020-11-20T12:13:22.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.405+0000 7f5cee3c3700 1 -- 172.21.15.187:0/2417972017 learned_addr learned my addr 172.21.15.187:0/2417972017 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:22.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5cedbc2700 1 --2- 172.21.15.187:0/2417972017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 0x7f5ce8065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:22.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5cee3c3700 1 -- 172.21.15.187:0/2417972017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 msgr2=0x7f5ce8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:22.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5cee3c3700 1 --2- 172.21.15.187:0/2417972017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 0x7f5ce8065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5cee3c3700 1 -- 172.21.15.187:0/2417972017 >> v2:172.21.15.5:3300/0 conn(0x7f5ce8065680 msgr2=0x7f5ce8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:22.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5cee3c3700 1 --2- 172.21.15.187:0/2417972017 >> v2:172.21.15.5:3300/0 conn(0x7f5ce8065680 0x7f5ce8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:22.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5cee3c3700 1 -- 172.21.15.187:0/2417972017 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce8150fc0 con 0x7f5ce80b1380 2020-11-20T12:13:22.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5cee3c3700 1 --2- 172.21.15.187:0/2417972017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b1380 0x7f5ce8150ad0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f5ce80b85e0 tx=0x7f5ce4006eb0).ready entity=mon.1 client_cookie=dfe8c63d56be3c71 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:22.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5ced3c1700 1 -- 172.21.15.187:0/2417972017 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce40098d0 con 0x7f5ce80b1380 2020-11-20T12:13:22.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5ced3c1700 1 -- 172.21.15.187:0/2417972017 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce4009a30 con 0x7f5ce80b1380 2020-11-20T12:13:22.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.406+0000 7f5ced3c1700 1 -- 172.21.15.187:0/2417972017 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4009db0 con 0x7f5ce80b1380 2020-11-20T12:13:22.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 -- 172.21.15.187:0/2417972017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b1380 msgr2=0x7f5ce8150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:22.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/2417972017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b1380 0x7f5ce8150ad0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f5ce80b85e0 tx=0x7f5ce4006eb0).stop 2020-11-20T12:13:22.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 -- 172.21.15.187:0/2417972017 shutdown_connections 2020-11-20T12:13:22.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/2417972017 >> v2:172.21.15.5:3300/0 conn(0x7f5ce8065680 0x7f5ce8061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:22.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/2417972017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b1380 0x7f5ce8150ad0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/2417972017 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 0x7f5ce8065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 -- 172.21.15.187:0/2417972017 >> 172.21.15.187:0/2417972017 conn(0x7f5ce80bab30 msgr2=0x7f5ce80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:22.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 -- 172.21.15.187:0/2417972017 shutdown_connections 2020-11-20T12:13:22.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.407+0000 7f5cef3c5700 1 -- 172.21.15.187:0/2417972017 wait complete. 2020-11-20T12:13:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.409+0000 7f5cef3c5700 1 Processor -- start 2020-11-20T12:13:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.409+0000 7f5cef3c5700 1 -- start start 2020-11-20T12:13:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.410+0000 7f5cef3c5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ce80b1380 0x7f5ce809af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.410+0000 7f5cef3c5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 0x7f5ce80b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:22.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.410+0000 7f5cef3c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b3dd0 0x7f5ce8149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.410+0000 7f5cef3c5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce809b430 con 0x7f5ce80b1380 2020-11-20T12:13:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.410+0000 7f5cef3c5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce809b650 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.410+0000 7f5cef3c5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce809b8f0 con 0x7f5ce80b1760 2020-11-20T12:13:22.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.411+0000 7f5ceebc4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b3dd0 0x7f5ce8149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:22.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.411+0000 7f5ceebc4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b3dd0 0x7f5ce8149950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47620/0 (socket says 172.21.15.187:47620) 2020-11-20T12:13:22.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.411+0000 7f5ceebc4700 1 -- 172.21.15.187:0/1400590589 learned_addr learned my addr 172.21.15.187:0/1400590589 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:22.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.411+0000 7f5cee3c3700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ce80b1380 0x7f5ce809af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:22.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.411+0000 7f5cedbc2700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 0x7f5ce80b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:22.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.411+0000 7f5ceebc4700 1 -- 172.21.15.187:0/1400590589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 msgr2=0x7f5ce80b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:22.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.411+0000 7f5ceebc4700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 0x7f5ce80b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5ceebc4700 1 -- 172.21.15.187:0/1400590589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ce80b1380 msgr2=0x7f5ce809af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:22.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5ceebc4700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ce80b1380 0x7f5ce809af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:22.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5ceebc4700 1 -- 172.21.15.187:0/1400590589 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce809bb90 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5ceebc4700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b3dd0 0x7f5ce8149950 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f5ce000ba30 tx=0x7f5ce0009220).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5cda7fc700 1 -- 172.21.15.187:0/1400590589 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce000c8b0 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5cda7fc700 1 -- 172.21.15.187:0/1400590589 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce000ca10 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5cda7fc700 1 -- 172.21.15.187:0/1400590589 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce000cbc0 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce809c030 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.412+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce8149e40 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.414+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce8061a20 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.414+0000 7f5cda7fc700 1 -- 172.21.15.187:0/1400590589 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5ce001c070 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.415+0000 7f5cda7fc700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5cc0056d20 0x7f5cc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.415+0000 7f5cda7fc700 1 -- 172.21.15.187:0/1400590589 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5ce006f8b0 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.415+0000 7f5cee3c3700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5cc0056d20 0x7f5cc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.416+0000 7f5cee3c3700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5cc0056d20 0x7f5cc00591e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f5ce4006ef0 tx=0x7f5ce4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:22.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.419+0000 7f5cda7fc700 1 -- 172.21.15.187:0/1400590589 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5ce0044c30 con 0x7f5ce80b3dd0 2020-11-20T12:13:22.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:22.675+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7f5ce809bd20 con 0x7f5ce80b3dd0 2020-11-20T12:13:23.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.257+0000 7f5cda7fc700 1 -- 172.21.15.187:0/1400590589 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v429) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5ce001a070 con 0x7f5ce80b3dd0 2020-11-20T12:13:23.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5cc0056d20 msgr2=0x7f5cc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5cc0056d20 0x7f5cc00591e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f5ce4006ef0 tx=0x7f5ce4015000).stop 2020-11-20T12:13:23.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b3dd0 msgr2=0x7f5ce8149950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b3dd0 0x7f5ce8149950 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f5ce000ba30 tx=0x7f5ce0009220).stop 2020-11-20T12:13:23.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 shutdown_connections 2020-11-20T12:13:23.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5cc0056d20 0x7f5cc00591e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ce80b1380 0x7f5ce809af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce80b3dd0 0x7f5ce8149950 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 --2- 172.21.15.187:0/1400590589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ce80b1760 0x7f5ce80b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 >> 172.21.15.187:0/1400590589 conn(0x7f5ce80bab30 msgr2=0x7f5ce8151320 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:23.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 shutdown_connections 2020-11-20T12:13:23.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.261+0000 7f5cef3c5700 1 -- 172.21.15.187:0/1400590589 wait complete. 2020-11-20T12:13:23.293 INFO:tasks.cephfs_test_runner:test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:13:23.293 INFO:tasks.cephfs_test_runner:Starting test: test_mv_cleanup (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:13:23.294 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup' 2020-11-20T12:13:23.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.537+0000 7f80d5260700 1 Processor -- start 2020-11-20T12:13:23.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.537+0000 7f80d5260700 1 -- start start 2020-11-20T12:13:23.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80d5260700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f80d01484d0 0x7f80d0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80d5260700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d0066f70 0x7f80d00b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80d5260700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d00b1300 con 0x7f80d00622c0 2020-11-20T12:13:23.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80d5260700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d0067730 con 0x7f80d01484d0 2020-11-20T12:13:23.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80d5260700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d00b15a0 con 0x7f80d0066f70 2020-11-20T12:13:23.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80c77fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f80d00622c0 0x7f80d00b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44234/0 (socket says 172.21.15.187:44234) 2020-11-20T12:13:23.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80c77fe700 1 -- 172.21.15.187:0/3218050584 learned_addr learned my addr 172.21.15.187:0/3218050584 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:23.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80cffff700 1 --2- 172.21.15.187:0/3218050584 >> v2:172.21.15.187:3300/0 conn(0x7f80d01484d0 0x7f80d0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.539+0000 7f80d4a5f700 1 --2- 172.21.15.187:0/3218050584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d0066f70 0x7f80d00b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cffff700 1 -- 172.21.15.187:0/3218050584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d0066f70 msgr2=0x7f80d00b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:23.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cffff700 1 --2- 172.21.15.187:0/3218050584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d0066f70 0x7f80d00b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cffff700 1 -- 172.21.15.187:0/3218050584 >> v1:172.21.15.187:6789/0 conn(0x7f80d00622c0 legacy=0x7f80d00b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cffff700 1 -- 172.21.15.187:0/3218050584 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d00b1840 con 0x7f80d01484d0 2020-11-20T12:13:23.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cf7fe700 1 -- 172.21.15.187:0/3218050584 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675436137 0 0) 0x7f80d00b1300 con 0x7f80d00622c0 2020-11-20T12:13:23.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cffff700 1 --2- 172.21.15.187:0/3218050584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d0061dd0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f80c000b9a0 tx=0x7f80c0006df0).ready entity=mon.1 client_cookie=b5518eb6259d29c9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:23.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cf7fe700 1 -- 172.21.15.187:0/3218050584 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c00097b0 con 0x7f80d01484d0 2020-11-20T12:13:23.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cf7fe700 1 -- 172.21.15.187:0/3218050584 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c0009910 con 0x7f80d01484d0 2020-11-20T12:13:23.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.540+0000 7f80cf7fe700 1 -- 172.21.15.187:0/3218050584 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c0009c90 con 0x7f80d01484d0 2020-11-20T12:13:23.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.541+0000 7f80d5260700 1 -- 172.21.15.187:0/3218050584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 msgr2=0x7f80d0061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.541+0000 7f80d5260700 1 --2- 172.21.15.187:0/3218050584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d0061dd0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f80c000b9a0 tx=0x7f80c0006df0).stop 2020-11-20T12:13:23.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.541+0000 7f80d5260700 1 -- 172.21.15.187:0/3218050584 shutdown_connections 2020-11-20T12:13:23.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.541+0000 7f80d5260700 1 --2- 172.21.15.187:0/3218050584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d0066f70 0x7f80d00b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.541+0000 7f80d5260700 1 --2- 172.21.15.187:0/3218050584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d0061dd0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.541+0000 7f80d5260700 1 -- 172.21.15.187:0/3218050584 >> 172.21.15.187:0/3218050584 conn(0x7f80d00bab30 msgr2=0x7f80d00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:23.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.541+0000 7f80d5260700 1 -- 172.21.15.187:0/3218050584 shutdown_connections 2020-11-20T12:13:23.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.541+0000 7f80d5260700 1 -- 172.21.15.187:0/3218050584 wait complete. 2020-11-20T12:13:23.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.542+0000 7f80d5260700 1 Processor -- start 2020-11-20T12:13:23.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.543+0000 7f80d5260700 1 -- start start 2020-11-20T12:13:23.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.543+0000 7f80d5260700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d00622c0 0x7f80d013e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.543+0000 7f80d5260700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d0066f70 0x7f80d009f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.543+0000 7f80d5260700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d00b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.543+0000 7f80d5260700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d00b1bc0 con 0x7f80d0066f70 2020-11-20T12:13:23.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.543+0000 7f80d5260700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d00b1e60 con 0x7f80d01484d0 2020-11-20T12:13:23.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.543+0000 7f80d5260700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d00b2100 con 0x7f80d00622c0 2020-11-20T12:13:23.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d00b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d00b25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47634/0 (socket says 172.21.15.187:47634) 2020-11-20T12:13:23.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 -- 172.21.15.187:0/1685259973 learned_addr learned my addr 172.21.15.187:0/1685259973 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:23.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 -- 172.21.15.187:0/1685259973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d00622c0 msgr2=0x7f80d013e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d00622c0 0x7f80d013e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:23.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 -- 172.21.15.187:0/1685259973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d0066f70 msgr2=0x7f80d009f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d0066f70 0x7f80d009f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:23.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 -- 172.21.15.187:0/1685259973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d00b4570 con 0x7f80d01484d0 2020-11-20T12:13:23.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.544+0000 7f80d4a5f700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d00b25a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f80c800eb40 tx=0x7f80c8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:23.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.545+0000 7f80ccff9700 1 -- 172.21.15.187:0/1685259973 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c800c7a0 con 0x7f80d01484d0 2020-11-20T12:13:23.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.545+0000 7f80ccff9700 1 -- 172.21.15.187:0/1685259973 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c800c900 con 0x7f80d01484d0 2020-11-20T12:13:23.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.545+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d013e990 con 0x7f80d01484d0 2020-11-20T12:13:23.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.545+0000 7f80ccff9700 1 -- 172.21.15.187:0/1685259973 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c800cca0 con 0x7f80d01484d0 2020-11-20T12:13:23.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.545+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d009fac0 con 0x7f80d01484d0 2020-11-20T12:13:23.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.546+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d009fce0 con 0x7f80d01484d0 2020-11-20T12:13:23.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.547+0000 7f80ccff9700 1 -- 172.21.15.187:0/1685259973 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f80c8013640 con 0x7f80d01484d0 2020-11-20T12:13:23.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.547+0000 7f80ccff9700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80ac056d20 0x7f80ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.547+0000 7f80ccff9700 1 -- 172.21.15.187:0/1685259973 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f80c806fd30 con 0x7f80d01484d0 2020-11-20T12:13:23.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.547+0000 7f80cffff700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80ac056d20 0x7f80ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.548+0000 7f80cffff700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80ac056d20 0x7f80ac0591e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f80c0006e30 tx=0x7f80c0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:23.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.551+0000 7f80ccff9700 1 -- 172.21.15.187:0/1685259973 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f80c8045100 con 0x7f80d01484d0 2020-11-20T12:13:23.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:23.807+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup"]} v 0) v1 -- 0x7f80d00b22c0 con 0x7f80d01484d0 2020-11-20T12:13:24.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.262+0000 7f80ccff9700 1 -- 172.21.15.187:0/1685259973 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup"]}]=0 v430) v1 ==== 133+0+0 (secure 0 0 0) 0x7f80c8048720 con 0x7f80d01484d0 2020-11-20T12:13:24.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80ac056d20 msgr2=0x7f80ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:24.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80ac056d20 0x7f80ac0591e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f80c0006e30 tx=0x7f80c0015000).stop 2020-11-20T12:13:24.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 msgr2=0x7f80d00b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:24.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d00b25a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f80c800eb40 tx=0x7f80c8008040).stop 2020-11-20T12:13:24.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 shutdown_connections 2020-11-20T12:13:24.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f80ac056d20 0x7f80ac0591e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80d01484d0 0x7f80d00b25a0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f80d00622c0 0x7f80d013e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:24.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 --2- 172.21.15.187:0/1685259973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d0066f70 0x7f80d009f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:24.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 >> 172.21.15.187:0/1685259973 conn(0x7f80d00bab30 msgr2=0x7f80d00b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:24.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 shutdown_connections 2020-11-20T12:13:24.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.266+0000 7f80d5260700 1 -- 172.21.15.187:0/1685259973 wait complete. 2020-11-20T12:13:24.298 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:13:24.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.536+0000 7faaf3c4d700 1 Processor -- start 2020-11-20T12:13:24.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.536+0000 7faaf3c4d700 1 -- start start 2020-11-20T12:13:24.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.537+0000 7faaf3c4d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faaec065300 0x7faaec0656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:24.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.537+0000 7faaf3c4d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faaec09a8a0 0x7faaec062400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:24.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.537+0000 7faaf3c4d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec068030 0x7faaec0628f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:24.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.537+0000 7faaf3c4d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec066420 con 0x7faaec09a8a0 2020-11-20T12:13:24.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.537+0000 7faaf3c4d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec0b7380 con 0x7faaec065300 2020-11-20T12:13:24.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.537+0000 7faaf3c4d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec061f60 con 0x7faaec068030 2020-11-20T12:13:24.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.538+0000 7faaf19e9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faaec065300 0x7faaec0656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:24.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.538+0000 7faaf19e9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faaec065300 0x7faaec0656e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47638/0 (socket says 172.21.15.187:47638) 2020-11-20T12:13:24.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.538+0000 7faaf19e9700 1 -- 172.21.15.187:0/2889990197 learned_addr learned my addr 172.21.15.187:0/2889990197 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:24.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.538+0000 7faaf11e8700 1 --2- 172.21.15.187:0/2889990197 >> v2:172.21.15.5:3300/0 conn(0x7faaec09a8a0 0x7faaec062400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:24.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.538+0000 7faaf21ea700 1 --2- 172.21.15.187:0/2889990197 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec068030 0x7faaec0628f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:24.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.538+0000 7faaf19e9700 1 -- 172.21.15.187:0/2889990197 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec068030 msgr2=0x7faaec0628f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:24.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.538+0000 7faaf19e9700 1 --2- 172.21.15.187:0/2889990197 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec068030 0x7faaec0628f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.538+0000 7faaf19e9700 1 -- 172.21.15.187:0/2889990197 >> v2:172.21.15.5:3300/0 conn(0x7faaec09a8a0 msgr2=0x7faaec062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:24.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.539+0000 7faaf19e9700 1 --2- 172.21.15.187:0/2889990197 >> v2:172.21.15.5:3300/0 conn(0x7faaec09a8a0 0x7faaec062400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.539+0000 7faaf19e9700 1 -- 172.21.15.187:0/2889990197 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec0b6fd0 con 0x7faaec065300 2020-11-20T12:13:24.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.539+0000 7faaf19e9700 1 --2- 172.21.15.187:0/2889990197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec065300 0x7faaec0656e0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7faaec0b1c80 tx=0x7faae8006eb0).ready entity=mon.1 client_cookie=74477a0db0203191 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:24.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.539+0000 7faaf09e7700 1 -- 172.21.15.187:0/2889990197 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8009960 con 0x7faaec065300 2020-11-20T12:13:24.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.539+0000 7faaf09e7700 1 -- 172.21.15.187:0/2889990197 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae8009ac0 con 0x7faaec065300 2020-11-20T12:13:24.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.539+0000 7faaf09e7700 1 -- 172.21.15.187:0/2889990197 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8009e40 con 0x7faaec065300 2020-11-20T12:13:24.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 -- 172.21.15.187:0/2889990197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec065300 msgr2=0x7faaec0656e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:24.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/2889990197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec065300 0x7faaec0656e0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7faaec0b1c80 tx=0x7faae8006eb0).stop 2020-11-20T12:13:24.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 -- 172.21.15.187:0/2889990197 shutdown_connections 2020-11-20T12:13:24.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/2889990197 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec068030 0x7faaec0628f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/2889990197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec065300 0x7faaec0656e0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/2889990197 >> v2:172.21.15.5:3300/0 conn(0x7faaec09a8a0 0x7faaec062400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 -- 172.21.15.187:0/2889990197 >> 172.21.15.187:0/2889990197 conn(0x7faaec0bacf0 msgr2=0x7faaec0bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:24.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 -- 172.21.15.187:0/2889990197 shutdown_connections 2020-11-20T12:13:24.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.540+0000 7faaf3c4d700 1 -- 172.21.15.187:0/2889990197 wait complete. 2020-11-20T12:13:24.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.542+0000 7faaf3c4d700 1 Processor -- start 2020-11-20T12:13:24.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.542+0000 7faaf3c4d700 1 -- start start 2020-11-20T12:13:24.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.543+0000 7faaf3c4d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec065300 0x7faaec15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:24.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.543+0000 7faaf3c4d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaec068030 0x7faaec0a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:24.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.543+0000 7faaf3c4d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec09a8a0 0x7faaec1452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:24.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.543+0000 7faaf3c4d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec08eda0 con 0x7faaec068030 2020-11-20T12:13:24.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.543+0000 7faaf3c4d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec08f040 con 0x7faaec09a8a0 2020-11-20T12:13:24.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.543+0000 7faaf3c4d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec08f2e0 con 0x7faaec065300 2020-11-20T12:13:24.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf21ea700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec09a8a0 0x7faaec1452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:24.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf21ea700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec09a8a0 0x7faaec1452a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47648/0 (socket says 172.21.15.187:47648) 2020-11-20T12:13:24.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf21ea700 1 -- 172.21.15.187:0/882703002 learned_addr learned my addr 172.21.15.187:0/882703002 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:24.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf21ea700 1 -- 172.21.15.187:0/882703002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec065300 msgr2=0x7faaec15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:24.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf19e9700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec065300 0x7faaec15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:24.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf21ea700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec065300 0x7faaec15a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf21ea700 1 -- 172.21.15.187:0/882703002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaec068030 msgr2=0x7faaec0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:24.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf11e8700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaec068030 0x7faaec0a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:24.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf21ea700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaec068030 0x7faaec0a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.544+0000 7faaf21ea700 1 -- 172.21.15.187:0/882703002 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec0a0670 con 0x7faaec09a8a0 2020-11-20T12:13:24.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.545+0000 7faaf21ea700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec09a8a0 0x7faaec1452a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7faae400afd0 tx=0x7faae4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:24.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.545+0000 7faad9ffb700 1 -- 172.21.15.187:0/882703002 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4007940 con 0x7faaec09a8a0 2020-11-20T12:13:24.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.545+0000 7faad9ffb700 1 -- 172.21.15.187:0/882703002 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae4007aa0 con 0x7faaec09a8a0 2020-11-20T12:13:24.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.545+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec08ea20 con 0x7faaec09a8a0 2020-11-20T12:13:24.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.545+0000 7faad9ffb700 1 -- 172.21.15.187:0/882703002 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4007e60 con 0x7faaec09a8a0 2020-11-20T12:13:24.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.545+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec08f780 con 0x7faaec09a8a0 2020-11-20T12:13:24.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.546+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec044f60 con 0x7faaec09a8a0 2020-11-20T12:13:24.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.547+0000 7faad9ffb700 1 -- 172.21.15.187:0/882703002 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faae4011620 con 0x7faaec09a8a0 2020-11-20T12:13:24.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.548+0000 7faad9ffb700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faac4056d20 0x7faac40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:24.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.548+0000 7faad9ffb700 1 -- 172.21.15.187:0/882703002 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faae4070240 con 0x7faaec09a8a0 2020-11-20T12:13:24.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.548+0000 7faaf19e9700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faac4056d20 0x7faac40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:24.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.549+0000 7faaf19e9700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faac4056d20 0x7faac40591e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7faae800b1b0 tx=0x7faae8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:24.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.552+0000 7faad9ffb700 1 -- 172.21.15.187:0/882703002 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faae4045690 con 0x7faaec09a8a0 2020-11-20T12:13:24.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.738+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7faaec08ec10 con 0x7faaec09a8a0 2020-11-20T12:13:24.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.748+0000 7faad9ffb700 1 -- 172.21.15.187:0/882703002 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7faae4048cb0 con 0x7faaec09a8a0 2020-11-20T12:13:24.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.750+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faac4056d20 msgr2=0x7faac40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:24.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.750+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faac4056d20 0x7faac40591e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7faae800b1b0 tx=0x7faae8015000).stop 2020-11-20T12:13:24.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.750+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec09a8a0 msgr2=0x7faaec1452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:24.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.750+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec09a8a0 0x7faaec1452a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7faae400afd0 tx=0x7faae4008040).stop 2020-11-20T12:13:24.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.751+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 shutdown_connections 2020-11-20T12:13:24.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.751+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faac4056d20 0x7faac40591e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.751+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaec09a8a0 0x7faaec1452a0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.751+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaec065300 0x7faaec15a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.751+0000 7faaf3c4d700 1 --2- 172.21.15.187:0/882703002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaec068030 0x7faaec0a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:24.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.751+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 >> 172.21.15.187:0/882703002 conn(0x7faaec0bacf0 msgr2=0x7faaec062320 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:24.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.751+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 shutdown_connections 2020-11-20T12:13:24.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:24.751+0000 7faaf3c4d700 1 -- 172.21.15.187:0/882703002 wait complete. 2020-11-20T12:13:24.782 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:13:24.784 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:13:24.907 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:13:24.927 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:13:25.035 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:13:25.054 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:13:25.054 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:13:25.055 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:25.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.294+0000 7f38488fe700 1 Processor -- start 2020-11-20T12:13:25.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.294+0000 7f38488fe700 1 -- start start 2020-11-20T12:13:25.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.296+0000 7f38488fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f38441484d0 0x7f3844061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.296+0000 7f38488fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3844066f70 0x7f38440b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.296+0000 7f38488fe700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440b1300 con 0x7f38440622c0 2020-11-20T12:13:25.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.296+0000 7f38488fe700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3844067730 con 0x7f38441484d0 2020-11-20T12:13:25.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.296+0000 7f38488fe700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38440b15a0 con 0x7f3844066f70 2020-11-20T12:13:25.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.296+0000 7f3842d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f38441484d0 0x7f3844061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.296+0000 7f3842d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f38441484d0 0x7f3844061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47652/0 (socket says 172.21.15.187:47652) 2020-11-20T12:13:25.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.296+0000 7f3842d9d700 1 -- 172.21.15.187:0/1184906382 learned_addr learned my addr 172.21.15.187:0/1184906382 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:25.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.297+0000 7f384359e700 1 --2- 172.21.15.187:0/1184906382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3844066f70 0x7f38440b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.297+0000 7f3842d9d700 1 -- 172.21.15.187:0/1184906382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3844066f70 msgr2=0x7f38440b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:25.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.297+0000 7f3842d9d700 1 --2- 172.21.15.187:0/1184906382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3844066f70 0x7f38440b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.297+0000 7f3842d9d700 1 -- 172.21.15.187:0/1184906382 >> v1:172.21.15.5:6789/0 conn(0x7f38440622c0 legacy=0x7f38440b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:25.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.297+0000 7f3842d9d700 1 -- 172.21.15.187:0/1184906382 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38440b1840 con 0x7f38441484d0 2020-11-20T12:13:25.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.297+0000 7f3842d9d700 1 --2- 172.21.15.187:0/1184906382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38441484d0 0x7f3844061dd0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f3834006fd0 tx=0x7f3834006eb0).ready entity=mon.1 client_cookie=e56d226d2a2d77fa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:25.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.297+0000 7f384259c700 1 -- 172.21.15.187:0/1184906382 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834009890 con 0x7f38441484d0 2020-11-20T12:13:25.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f384259c700 1 -- 172.21.15.187:0/1184906382 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38340099f0 con 0x7f38441484d0 2020-11-20T12:13:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f384259c700 1 -- 172.21.15.187:0/1184906382 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834009d70 con 0x7f38441484d0 2020-11-20T12:13:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f38488fe700 1 -- 172.21.15.187:0/1184906382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38441484d0 msgr2=0x7f3844061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f38488fe700 1 --2- 172.21.15.187:0/1184906382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38441484d0 0x7f3844061dd0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f3834006fd0 tx=0x7f3834006eb0).stop 2020-11-20T12:13:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f38488fe700 1 -- 172.21.15.187:0/1184906382 shutdown_connections 2020-11-20T12:13:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f38488fe700 1 --2- 172.21.15.187:0/1184906382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3844066f70 0x7f38440b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f38488fe700 1 --2- 172.21.15.187:0/1184906382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38441484d0 0x7f3844061dd0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f38488fe700 1 -- 172.21.15.187:0/1184906382 >> 172.21.15.187:0/1184906382 conn(0x7f38440bab30 msgr2=0x7f38440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:25.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f38488fe700 1 -- 172.21.15.187:0/1184906382 shutdown_connections 2020-11-20T12:13:25.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.298+0000 7f38488fe700 1 -- 172.21.15.187:0/1184906382 wait complete. 2020-11-20T12:13:25.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.300+0000 7f38488fe700 1 Processor -- start 2020-11-20T12:13:25.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.300+0000 7f38488fe700 1 -- start start 2020-11-20T12:13:25.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.301+0000 7f38488fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38440622c0 0x7f3844147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f38488fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3844066f70 0x7f384409dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f38488fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38441484d0 0x7f384409f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f38488fe700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3844160680 con 0x7f38440622c0 2020-11-20T12:13:25.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f38488fe700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3844160920 con 0x7f3844066f70 2020-11-20T12:13:25.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f38488fe700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384415c4f0 con 0x7f38441484d0 2020-11-20T12:13:25.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f383a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3844066f70 0x7f384409dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f383a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3844066f70 0x7f384409dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47660/0 (socket says 172.21.15.187:47660) 2020-11-20T12:13:25.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f383a59c700 1 -- 172.21.15.187:0/3740085920 learned_addr learned my addr 172.21.15.187:0/3740085920 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:25.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f3842d9d700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38440622c0 0x7f3844147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.302+0000 7f384359e700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38441484d0 0x7f384409f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383a59c700 1 -- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38441484d0 msgr2=0x7f384409f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383a59c700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38441484d0 0x7f384409f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383a59c700 1 -- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38440622c0 msgr2=0x7f3844147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383a59c700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38440622c0 0x7f3844147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383a59c700 1 -- 172.21.15.187:0/3740085920 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384409f740 con 0x7f3844066f70 2020-11-20T12:13:25.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f384359e700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38441484d0 0x7f384409f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:25.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383a59c700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3844066f70 0x7f384409dae0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f382c00c760 tx=0x7f382c007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:25.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383b7fe700 1 -- 172.21.15.187:0/3740085920 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00e560 con 0x7f3844066f70 2020-11-20T12:13:25.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383b7fe700 1 -- 172.21.15.187:0/3740085920 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c00e6c0 con 0x7f3844066f70 2020-11-20T12:13:25.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38440a02a0 con 0x7f3844066f70 2020-11-20T12:13:25.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f383b7fe700 1 -- 172.21.15.187:0/3740085920 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00ea80 con 0x7f3844066f70 2020-11-20T12:13:25.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.303+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3844142130 con 0x7f3844066f70 2020-11-20T12:13:25.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.304+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38441604e0 con 0x7f3844066f70 2020-11-20T12:13:25.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.305+0000 7f383b7fe700 1 -- 172.21.15.187:0/3740085920 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f382c019030 con 0x7f3844066f70 2020-11-20T12:13:25.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.306+0000 7f383b7fe700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3820056d20 0x7f38200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.306+0000 7f383b7fe700 1 -- 172.21.15.187:0/3740085920 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f382c06f470 con 0x7f3844066f70 2020-11-20T12:13:25.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.306+0000 7f3842d9d700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3820056d20 0x7f38200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.307+0000 7f3842d9d700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3820056d20 0x7f38200591e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f383400b1b0 tx=0x7f3834015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:25.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.310+0000 7f383b7fe700 1 -- 172.21.15.187:0/3740085920 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f38441604e0 con 0x7f3844066f70 2020-11-20T12:13:25.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.555+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38441604e0 con 0x7f3844066f70 2020-11-20T12:13:25.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.556+0000 7f383b7fe700 1 -- 172.21.15.187:0/3740085920 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f38441604e0 con 0x7f3844066f70 2020-11-20T12:13:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:25.560 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/797250225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":797250225},{"type":"v1","addr":"172.21.15.5:6833","nonce":797250225}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:13:19.215710+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6952},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:25.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.558+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3820056d20 msgr2=0x7f38200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.558+0000 7f38488fe700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3820056d20 0x7f38200591e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f383400b1b0 tx=0x7f3834015000).stop 2020-11-20T12:13:25.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.558+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3844066f70 msgr2=0x7f384409dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.558+0000 7f38488fe700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3844066f70 0x7f384409dae0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f382c00c760 tx=0x7f382c007c00).stop 2020-11-20T12:13:25.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.559+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 shutdown_connections 2020-11-20T12:13:25.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.559+0000 7f38488fe700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3820056d20 0x7f38200591e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.559+0000 7f38488fe700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38440622c0 0x7f3844147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.559+0000 7f38488fe700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38441484d0 0x7f384409f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.559+0000 7f38488fe700 1 --2- 172.21.15.187:0/3740085920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3844066f70 0x7f384409dae0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.559+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 >> 172.21.15.187:0/3740085920 conn(0x7f38440bab30 msgr2=0x7f38440b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:25.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.559+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 shutdown_connections 2020-11-20T12:13:25.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.559+0000 7f38488fe700 1 -- 172.21.15.187:0/3740085920 wait complete. 2020-11-20T12:13:25.565 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 73 2020-11-20T12:13:25.587 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:13:25.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.823+0000 7f907b503700 1 Processor -- start 2020-11-20T12:13:25.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.823+0000 7f907b503700 1 -- start start 2020-11-20T12:13:25.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.824+0000 7f907b503700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90741484d0 0x7f9074061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.824+0000 7f907b503700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90740b1300 con 0x7f9074066f70 2020-11-20T12:13:25.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.824+0000 7f907b503700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90740b15a0 con 0x7f90740622c0 2020-11-20T12:13:25.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.824+0000 7f907b503700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9074067730 con 0x7f90741484d0 2020-11-20T12:13:25.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.825+0000 7f9079d00700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f90740622c0 0x7f90740b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44274/0 (socket says 172.21.15.187:44274) 2020-11-20T12:13:25.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.825+0000 7f9079d00700 1 -- 172.21.15.187:0/1150916467 learned_addr learned my addr 172.21.15.187:0/1150916467 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:25.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.825+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628627756 0 0) 0x7f90740b15a0 con 0x7f90740622c0 2020-11-20T12:13:25.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.825+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c003240 con 0x7f90740622c0 2020-11-20T12:13:25.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.825+0000 7f907a501700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90741484d0 0x7f9074061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1569574520 0 0) 0x7f905c003240 con 0x7f90740622c0 2020-11-20T12:13:25.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90741484d0 msgr2=0x7f9074061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:25.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90741484d0 0x7f9074061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 >> v1:172.21.15.5:6789/0 conn(0x7f9074066f70 legacy=0x7f90740b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740b1840 con 0x7f90740622c0 2020-11-20T12:13:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f9064001520 con 0x7f90740622c0 2020-11-20T12:13:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 >> v1:172.21.15.187:6789/0 conn(0x7f90740622c0 legacy=0x7f90740b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90741484d0 0x7f905c003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f905c003c90 0x7f905c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f905c0049d0 0x7f905c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074067730 con 0x7f905c0049d0 2020-11-20T12:13:25.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c004220 con 0x7f905c003c90 2020-11-20T12:13:25.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905c00d940 con 0x7f90741484d0 2020-11-20T12:13:25.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.826+0000 7f9079d00700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f905c003c90 0x7f905c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f9079d00700 1 -- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90741484d0 msgr2=0x7f905c003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:25.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f9079d00700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90741484d0 0x7f905c003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:25.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f9079d00700 1 -- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f905c0049d0 msgr2=0x7f905c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:25.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f9079d00700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f905c0049d0 0x7f905c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:25.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f9079d00700 1 -- 172.21.15.187:0/1150916467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c003240 con 0x7f905c003c90 2020-11-20T12:13:25.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f9079d00700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f905c003c90 0x7f905c0044e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7f906400c1e0 tx=0x7f9064007750).ready entity=mon.1 client_cookie=91a8359aae0fef1c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:25.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90640070d0 con 0x7f905c003c90 2020-11-20T12:13:25.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064007230 con 0x7f905c003c90 2020-11-20T12:13:25.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.827+0000 7f90794ff700 1 -- 172.21.15.187:0/1150916467 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90640055f0 con 0x7f905c003c90 2020-11-20T12:13:25.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.828+0000 7f907b503700 1 -- 172.21.15.187:0/1150916467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f905c003c90 msgr2=0x7f905c0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.828+0000 7f907b503700 1 --2- 172.21.15.187:0/1150916467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f905c003c90 0x7f905c0044e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7f906400c1e0 tx=0x7f9064007750).stop 2020-11-20T12:13:25.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.828+0000 7f907ad02700 1 -- 172.21.15.187:0/1150916467 reap_dead start 2020-11-20T12:13:25.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.828+0000 7f907b503700 1 -- 172.21.15.187:0/1150916467 shutdown_connections 2020-11-20T12:13:25.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.828+0000 7f907b503700 1 -- 172.21.15.187:0/1150916467 >> 172.21.15.187:0/1150916467 conn(0x7f90740bab30 msgr2=0x7f90740baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:25.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.828+0000 7f907b503700 1 -- 172.21.15.187:0/1150916467 shutdown_connections 2020-11-20T12:13:25.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.828+0000 7f907b503700 1 -- 172.21.15.187:0/1150916467 wait complete. 2020-11-20T12:13:25.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.830+0000 7f907b503700 1 Processor -- start 2020-11-20T12:13:25.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.830+0000 7f907b503700 1 -- start start 2020-11-20T12:13:25.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f907b503700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9074143ed0 0x7f90740b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f907b503700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90741442b0 0x7f907415bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f907b503700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f907409bd10 0x7f907409c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f907b503700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907409c660 con 0x7f907409bd10 2020-11-20T12:13:25.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f907b503700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907415c3e0 con 0x7f90741442b0 2020-11-20T12:13:25.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f907b503700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907415c680 con 0x7f9074143ed0 2020-11-20T12:13:25.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f9079d00700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90741442b0 0x7f907415bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f9079d00700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90741442b0 0x7f907415bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47678/0 (socket says 172.21.15.187:47678) 2020-11-20T12:13:25.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f9079d00700 1 -- 172.21.15.187:0/3805288333 learned_addr learned my addr 172.21.15.187:0/3805288333 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:25.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f907ad02700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f907409bd10 0x7f907409c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.831+0000 7f907a501700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9074143ed0 0x7f90740b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f9079d00700 1 -- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9074143ed0 msgr2=0x7f90740b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f9079d00700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9074143ed0 0x7f90740b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f9079d00700 1 -- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f907409bd10 msgr2=0x7f907409c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:25.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f9079d00700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f907409bd10 0x7f907409c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:25.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f9079d00700 1 -- 172.21.15.187:0/3805288333 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907415c920 con 0x7f90741442b0 2020-11-20T12:13:25.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f907ad02700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f907409bd10 0x7f907409c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:25.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f907a501700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9074143ed0 0x7f90740b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:25.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f9079d00700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90741442b0 0x7f907415bad0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f906400a000 tx=0x7f9064015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:25.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f90627fc700 1 -- 172.21.15.187:0/3805288333 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064020480 con 0x7f90741442b0 2020-11-20T12:13:25.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f90627fc700 1 -- 172.21.15.187:0/3805288333 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90640205e0 con 0x7f90741442b0 2020-11-20T12:13:25.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f90627fc700 1 -- 172.21.15.187:0/3805288333 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064020980 con 0x7f90741442b0 2020-11-20T12:13:25.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90740b2de0 con 0x7f90741442b0 2020-11-20T12:13:25.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.832+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907415bfc0 con 0x7f90741442b0 2020-11-20T12:13:25.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.834+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90740b8940 con 0x7f90741442b0 2020-11-20T12:13:25.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.834+0000 7f90627fc700 1 -- 172.21.15.187:0/3805288333 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9064010070 con 0x7f90741442b0 2020-11-20T12:13:25.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.834+0000 7f90627fc700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f904c056d50 0x7f904c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:25.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.835+0000 7f90627fc700 1 -- 172.21.15.187:0/3805288333 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9064071530 con 0x7f90741442b0 2020-11-20T12:13:25.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.835+0000 7f907a501700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f904c056d50 0x7f904c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:25.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.836+0000 7f907a501700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f904c056d50 0x7f904c059210 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f9070006db0 tx=0x7f9070008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:25.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:25.838+0000 7f90627fc700 1 -- 172.21.15.187:0/3805288333 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9064046980 con 0x7f90741442b0 2020-11-20T12:13:26.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.023+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9074158500 con 0x7f90741442b0 2020-11-20T12:13:26.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.026+0000 7f90627fc700 1 -- 172.21.15.187:0/3805288333 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f9064049fa0 con 0x7f90741442b0 2020-11-20T12:13:26.032 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:26.032 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":81,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:12:11.318992+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:12:07.846671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2020-11-20T12:12:09.337957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:13:26.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f904c056d50 msgr2=0x7f904c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f904c056d50 0x7f904c059210 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f9070006db0 tx=0x7f9070008040).stop 2020-11-20T12:13:26.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90741442b0 msgr2=0x7f907415bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90741442b0 0x7f907415bad0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f906400a000 tx=0x7f9064015000).stop 2020-11-20T12:13:26.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 shutdown_connections 2020-11-20T12:13:26.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f904c056d50 0x7f904c059210 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f907409bd10 0x7f907409c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9074143ed0 0x7f90740b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 --2- 172.21.15.187:0/3805288333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90741442b0 0x7f907415bad0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 >> 172.21.15.187:0/3805288333 conn(0x7f90740bab30 msgr2=0x7f90740b6680 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.029+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 shutdown_connections 2020-11-20T12:13:26.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.030+0000 7f907b503700 1 -- 172.21.15.187:0/3805288333 wait complete. 2020-11-20T12:13:26.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:12:07.846671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:12:09.337957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:13:26.056 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:13:26.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.300+0000 7f4da1409700 1 Processor -- start 2020-11-20T12:13:26.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.300+0000 7f4da1409700 1 -- start start 2020-11-20T12:13:26.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.301+0000 7f4da1409700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d9c0683f0 0x7f4d9c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.301+0000 7f4da1409700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c14c170 con 0x7f4d9c145cc0 2020-11-20T12:13:26.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.301+0000 7f4da1409700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c14c410 con 0x7f4d9c1487e0 2020-11-20T12:13:26.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.301+0000 7f4da1409700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9c14c6b0 con 0x7f4d9c0683f0 2020-11-20T12:13:26.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.302+0000 7f4da0c08700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4d9c1487e0 0x7f4d9c148c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44296/0 (socket says 172.21.15.187:44296) 2020-11-20T12:13:26.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.302+0000 7f4da0c08700 1 -- 172.21.15.187:0/3248141066 learned_addr learned my addr 172.21.15.187:0/3248141066 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:26.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.302+0000 7f4d937fe700 1 --2- 172.21.15.187:0/3248141066 >> v2:172.21.15.5:3300/0 conn(0x7f4d9c0683f0 0x7f4d9c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729398811 0 0) 0x7f4d9c14c410 con 0x7f4d9c1487e0 2020-11-20T12:13:26.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d7c003240 con 0x7f4d9c1487e0 2020-11-20T12:13:26.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1174116103 0 0) 0x7f4d7c003240 con 0x7f4d9c1487e0 2020-11-20T12:13:26.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 >> v2:172.21.15.5:3300/0 conn(0x7f4d9c0683f0 msgr2=0x7f4d9c066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:26.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 --2- 172.21.15.187:0/3248141066 >> v2:172.21.15.5:3300/0 conn(0x7f4d9c0683f0 0x7f4d9c066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 >> v1:172.21.15.5:6789/0 conn(0x7f4d9c145cc0 legacy=0x7f4d9c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c0658a0 con 0x7f4d9c1487e0 2020-11-20T12:13:26.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4d94001860 con 0x7f4d9c1487e0 2020-11-20T12:13:26.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 >> v1:172.21.15.187:6789/0 conn(0x7f4d9c1487e0 legacy=0x7f4d9c148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d7c003780 0x7f4d7c003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d7c0044e0 0x7f4d7c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d7c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c14c6b0 con 0x7f4d7c003780 2020-11-20T12:13:26.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c005300 con 0x7f4d7c004e50 2020-11-20T12:13:26.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.303+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c00bc20 con 0x7f4d7c0044e0 2020-11-20T12:13:26.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.304+0000 7f4da0c08700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d7c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.304+0000 7f4d9bfff700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d7c003780 0x7f4d7c003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.304+0000 7f4d937fe700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d7c0044e0 0x7f4d7c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.304+0000 7f4da0c08700 1 -- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d7c0044e0 msgr2=0x7f4d7c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:26.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.304+0000 7f4da0c08700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d7c0044e0 0x7f4d7c004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.304+0000 7f4da0c08700 1 -- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d7c003780 msgr2=0x7f4d7c003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:26.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.304+0000 7f4da0c08700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d7c003780 0x7f4d7c003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.305+0000 7f4da0c08700 1 -- 172.21.15.187:0/3248141066 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c003240 con 0x7f4d7c004e50 2020-11-20T12:13:26.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.305+0000 7f4d937fe700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d7c0044e0 0x7f4d7c004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:26.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.305+0000 7f4da0c08700 1 -- 172.21.15.187:0/3248141066 reap_dead start 2020-11-20T12:13:26.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.305+0000 7f4da0c08700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d7c00b730 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f4d9400c550 tx=0x7f4d94007a50).ready entity=mon.1 client_cookie=53270c8084b0ba89 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:26.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.305+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94005b80 con 0x7f4d7c004e50 2020-11-20T12:13:26.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.305+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d94005ce0 con 0x7f4d7c004e50 2020-11-20T12:13:26.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.305+0000 7f4d9b7fe700 1 -- 172.21.15.187:0/3248141066 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94007320 con 0x7f4d7c004e50 2020-11-20T12:13:26.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.306+0000 7f4da1409700 1 -- 172.21.15.187:0/3248141066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 msgr2=0x7f4d7c00b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.306+0000 7f4da1409700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d7c00b730 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f4d9400c550 tx=0x7f4d94007a50).stop 2020-11-20T12:13:26.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.306+0000 7f4da1409700 1 -- 172.21.15.187:0/3248141066 shutdown_connections 2020-11-20T12:13:26.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.306+0000 7f4da1409700 1 --2- 172.21.15.187:0/3248141066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d7c00b730 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.306+0000 7f4da1409700 1 -- 172.21.15.187:0/3248141066 >> 172.21.15.187:0/3248141066 conn(0x7f4d9c0bab30 msgr2=0x7f4d9c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:26.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.306+0000 7f4da1409700 1 -- 172.21.15.187:0/3248141066 shutdown_connections 2020-11-20T12:13:26.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.306+0000 7f4da1409700 1 -- 172.21.15.187:0/3248141066 wait complete. 2020-11-20T12:13:26.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.308+0000 7f4da1409700 1 Processor -- start 2020-11-20T12:13:26.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.308+0000 7f4da1409700 1 -- start start 2020-11-20T12:13:26.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.309+0000 7f4da1409700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d9c0b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.309+0000 7f4da1409700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d9c0b3770 0x7f4d9c0b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.309+0000 7f4da1409700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d9c15b690 0x7f4d9c15bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.309+0000 7f4da1409700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c0b1bb0 con 0x7f4d9c0b3770 2020-11-20T12:13:26.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.309+0000 7f4da1409700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c0b1e10 con 0x7f4d7c004e50 2020-11-20T12:13:26.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.309+0000 7f4da1409700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c15af50 con 0x7f4d9c15b690 2020-11-20T12:13:26.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.309+0000 7f4d9bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d9c0b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d9bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d9c0b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47698/0 (socket says 172.21.15.187:47698) 2020-11-20T12:13:26.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d9bfff700 1 -- 172.21.15.187:0/1476239106 learned_addr learned my addr 172.21.15.187:0/1476239106 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:26.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d937fe700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d9c0b3770 0x7f4d9c0b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4da0c08700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d9c15b690 0x7f4d9c15bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d9bfff700 1 -- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d9c15b690 msgr2=0x7f4d9c15bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d9bfff700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d9c15b690 0x7f4d9c15bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d9bfff700 1 -- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d9c0b3770 msgr2=0x7f4d9c0b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d9bfff700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d9c0b3770 0x7f4d9c0b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d9bfff700 1 -- 172.21.15.187:0/1476239106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c15b1f0 con 0x7f4d7c004e50 2020-11-20T12:13:26.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4da0c08700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d9c15b690 0x7f4d9c15bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:26.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.310+0000 7f4d9bfff700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d9c0b2bb0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7f4d8c00e650 tx=0x7f4d8c009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:26.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.311+0000 7f4d98ff9700 1 -- 172.21.15.187:0/1476239106 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c010550 con 0x7f4d7c004e50 2020-11-20T12:13:26.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.311+0000 7f4d98ff9700 1 -- 172.21.15.187:0/1476239106 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c010040 con 0x7f4d7c004e50 2020-11-20T12:13:26.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.311+0000 7f4d98ff9700 1 -- 172.21.15.187:0/1476239106 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c0108c0 con 0x7f4d7c004e50 2020-11-20T12:13:26.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.311+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9c0b0980 con 0x7f4d7c004e50 2020-11-20T12:13:26.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.311+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9c14b470 con 0x7f4d7c004e50 2020-11-20T12:13:26.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.312+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9c061a20 con 0x7f4d7c004e50 2020-11-20T12:13:26.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.312+0000 7f4d98ff9700 1 -- 172.21.15.187:0/1476239106 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4d8c010cb0 con 0x7f4d7c004e50 2020-11-20T12:13:26.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.313+0000 7f4d98ff9700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d78056d20 0x7f4d780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.313+0000 7f4d98ff9700 1 -- 172.21.15.187:0/1476239106 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4d8c06ffa0 con 0x7f4d7c004e50 2020-11-20T12:13:26.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.313+0000 7f4d937fe700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d78056d20 0x7f4d780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.314+0000 7f4d937fe700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d78056d20 0x7f4d780591e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f4d84006a90 tx=0x7f4d84008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:26.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.317+0000 7f4d98ff9700 1 -- 172.21.15.187:0/1476239106 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4d8c0453f0 con 0x7f4d7c004e50 2020-11-20T12:13:26.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.503+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4d9c14bdf0 con 0x7f4d7c004e50 2020-11-20T12:13:26.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.504+0000 7f4d98ff9700 1 -- 172.21.15.187:0/1476239106 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4d8c048a10 con 0x7f4d7c004e50 2020-11-20T12:13:26.508 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:13:26.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.506+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d78056d20 msgr2=0x7f4d780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.506+0000 7f4da1409700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d78056d20 0x7f4d780591e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f4d84006a90 tx=0x7f4d84008040).stop 2020-11-20T12:13:26.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.506+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 msgr2=0x7f4d9c0b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.506+0000 7f4da1409700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d9c0b2bb0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7f4d8c00e650 tx=0x7f4d8c009040).stop 2020-11-20T12:13:26.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.507+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 shutdown_connections 2020-11-20T12:13:26.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.507+0000 7f4da1409700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d78056d20 0x7f4d780591e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.507+0000 7f4da1409700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d9c15b690 0x7f4d9c15bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.507+0000 7f4da1409700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d7c004e50 0x7f4d9c0b2bb0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.507+0000 7f4da1409700 1 --2- 172.21.15.187:0/1476239106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d9c0b3770 0x7f4d9c0b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.507+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 >> 172.21.15.187:0/1476239106 conn(0x7f4d9c0bab30 msgr2=0x7f4d9c065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:26.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.507+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 shutdown_connections 2020-11-20T12:13:26.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.507+0000 7f4da1409700 1 -- 172.21.15.187:0/1476239106 wait complete. 2020-11-20T12:13:26.534 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:13:26.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.777+0000 7f839e0c6700 1 Processor -- start 2020-11-20T12:13:26.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.777+0000 7f839e0c6700 1 -- start start 2020-11-20T12:13:26.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.778+0000 7f839e0c6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f83980b6650 0x7f83980afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.778+0000 7f839e0c6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83980b1300 con 0x7f83981484d0 2020-11-20T12:13:26.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.778+0000 7f839e0c6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83980b15a0 con 0x7f8398066f70 2020-11-20T12:13:26.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.778+0000 7f839e0c6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8398067730 con 0x7f83980b6650 2020-11-20T12:13:26.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.779+0000 7f839d8c5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8398066f70 0x7f83980aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44316/0 (socket says 172.21.15.187:44316) 2020-11-20T12:13:26.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.779+0000 7f839d8c5700 1 -- 172.21.15.187:0/2961270526 learned_addr learned my addr 172.21.15.187:0/2961270526 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:26.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.779+0000 7f839c8c3700 1 --2- 172.21.15.187:0/2961270526 >> v2:172.21.15.5:3300/0 conn(0x7f83980b6650 0x7f83980afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351454321 0 0) 0x7f83980b15a0 con 0x7f8398066f70 2020-11-20T12:13:26.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8380003240 con 0x7f8398066f70 2020-11-20T12:13:26.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3618046256 0 0) 0x7f8380003240 con 0x7f8398066f70 2020-11-20T12:13:26.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 >> v2:172.21.15.5:3300/0 conn(0x7f83980b6650 msgr2=0x7f83980afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:26.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 --2- 172.21.15.187:0/2961270526 >> v2:172.21.15.5:3300/0 conn(0x7f83980b6650 0x7f83980afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 >> v1:172.21.15.5:6789/0 conn(0x7f83981484d0 legacy=0x7f8398061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83980b1840 con 0x7f8398066f70 2020-11-20T12:13:26.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f83900014e0 con 0x7f8398066f70 2020-11-20T12:13:26.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 >> v1:172.21.15.187:6789/0 conn(0x7f8398066f70 legacy=0x7f83980aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8380003730 0x7f8380003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83800044e0 0x7f8380004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380004e50 0x7f838000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8398067730 con 0x7f83800044e0 2020-11-20T12:13:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8380004260 con 0x7f8380003730 2020-11-20T12:13:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.780+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838000bc20 con 0x7f8380004e50 2020-11-20T12:13:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839d0c4700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8380003730 0x7f8380003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839c8c3700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83800044e0 0x7f8380004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839d8c5700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380004e50 0x7f838000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839d0c4700 1 -- 172.21.15.187:0/2961270526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380004e50 msgr2=0x7f838000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839d0c4700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380004e50 0x7f838000b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839d0c4700 1 -- 172.21.15.187:0/2961270526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83800044e0 msgr2=0x7f8380004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839d0c4700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83800044e0 0x7f8380004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839d0c4700 1 -- 172.21.15.187:0/2961270526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8380003240 con 0x7f8380003730 2020-11-20T12:13:26.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.781+0000 7f839d8c5700 1 -- 172.21.15.187:0/2961270526 reap_dead start 2020-11-20T12:13:26.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.782+0000 7f839d0c4700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8380003730 0x7f8380003b50 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f8394001e70 tx=0x7f8394007f10).ready entity=mon.1 client_cookie=699ae08647ab2d4d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:26.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.782+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394001540 con 0x7f8380003730 2020-11-20T12:13:26.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.782+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8394003030 con 0x7f8380003730 2020-11-20T12:13:26.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.782+0000 7f8387fff700 1 -- 172.21.15.187:0/2961270526 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839400cdf0 con 0x7f8380003730 2020-11-20T12:13:26.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.782+0000 7f839e0c6700 1 -- 172.21.15.187:0/2961270526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8380003730 msgr2=0x7f8380003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.782+0000 7f839e0c6700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8380003730 0x7f8380003b50 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f8394001e70 tx=0x7f8394007f10).stop 2020-11-20T12:13:26.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.783+0000 7f839e0c6700 1 -- 172.21.15.187:0/2961270526 shutdown_connections 2020-11-20T12:13:26.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.783+0000 7f839e0c6700 1 --2- 172.21.15.187:0/2961270526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8380003730 0x7f8380003b50 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.783+0000 7f839e0c6700 1 -- 172.21.15.187:0/2961270526 >> 172.21.15.187:0/2961270526 conn(0x7f83980bab30 msgr2=0x7f83980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:26.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.783+0000 7f839e0c6700 1 -- 172.21.15.187:0/2961270526 shutdown_connections 2020-11-20T12:13:26.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.783+0000 7f839e0c6700 1 -- 172.21.15.187:0/2961270526 wait complete. 2020-11-20T12:13:26.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.785+0000 7f839e0c6700 1 Processor -- start 2020-11-20T12:13:26.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.785+0000 7f839e0c6700 1 -- start start 2020-11-20T12:13:26.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839e0c6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380003730 0x7f8398146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839e0c6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f839815bc50 0x7f839815b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839e0c6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8398099200 0x7f8398099680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839e0c6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8398146940 con 0x7f839815bc50 2020-11-20T12:13:26.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839e0c6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8398146bd0 con 0x7f8398099200 2020-11-20T12:13:26.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839e0c6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8398146e70 con 0x7f8380003730 2020-11-20T12:13:26.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839d8c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8398099200 0x7f8398099680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839d8c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8398099200 0x7f8398099680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47722/0 (socket says 172.21.15.187:47722) 2020-11-20T12:13:26.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.786+0000 7f839d8c5700 1 -- 172.21.15.187:0/3898113101 learned_addr learned my addr 172.21.15.187:0/3898113101 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:26.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839c8c3700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f839815bc50 0x7f839815b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839d0c4700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380003730 0x7f8398146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839d8c5700 1 -- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380003730 msgr2=0x7f8398146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839d8c5700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380003730 0x7f8398146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839d8c5700 1 -- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f839815bc50 msgr2=0x7f839815b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:26.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839d8c5700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f839815bc50 0x7f839815b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:26.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839d8c5700 1 -- 172.21.15.187:0/3898113101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8398147110 con 0x7f8398099200 2020-11-20T12:13:26.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839d0c4700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380003730 0x7f8398146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:26.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839d8c5700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8398099200 0x7f8398099680 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7f8390000bc0 tx=0x7f8390009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:26.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f83857fa700 1 -- 172.21.15.187:0/3898113101 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390007030 con 0x7f8398099200 2020-11-20T12:13:26.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f83857fa700 1 -- 172.21.15.187:0/3898113101 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83900071b0 con 0x7f8398099200 2020-11-20T12:13:26.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f83857fa700 1 -- 172.21.15.187:0/3898113101 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839000b7a0 con 0x7f8398099200 2020-11-20T12:13:26.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.787+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839815b910 con 0x7f8398099200 2020-11-20T12:13:26.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.788+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839809f9c0 con 0x7f8398099200 2020-11-20T12:13:26.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.789+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8398099b70 con 0x7f8398099200 2020-11-20T12:13:26.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.789+0000 7f83857fa700 1 -- 172.21.15.187:0/3898113101 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f839001e070 con 0x7f8398099200 2020-11-20T12:13:26.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.790+0000 7f83857fa700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8370056d20 0x7f83700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:26.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.790+0000 7f83857fa700 1 -- 172.21.15.187:0/3898113101 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f839006f240 con 0x7f8398099200 2020-11-20T12:13:26.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.790+0000 7f839d0c4700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8370056d20 0x7f83700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:26.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.791+0000 7f839d0c4700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8370056d20 0x7f83700591e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f8394007f70 tx=0x7f8394015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:26.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.794+0000 7f83857fa700 1 -- 172.21.15.187:0/3898113101 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8390044690 con 0x7f8398099200 2020-11-20T12:13:26.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.980+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8398061a20 con 0x7f8398099200 2020-11-20T12:13:27.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.981+0000 7f83857fa700 1 -- 172.21.15.187:0/3898113101 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8390047cb0 con 0x7f8398099200 2020-11-20T12:13:27.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.983+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8370056d20 msgr2=0x7f83700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.983+0000 7f839e0c6700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8370056d20 0x7f83700591e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f8394007f70 tx=0x7f8394015000).stop 2020-11-20T12:13:27.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.983+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8398099200 msgr2=0x7f8398099680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.983+0000 7f839e0c6700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8398099200 0x7f8398099680 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7f8390000bc0 tx=0x7f8390009040).stop 2020-11-20T12:13:27.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.984+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 shutdown_connections 2020-11-20T12:13:27.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.984+0000 7f839e0c6700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8370056d20 0x7f83700591e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.984+0000 7f839e0c6700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8398099200 0x7f8398099680 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.984+0000 7f839e0c6700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8380003730 0x7f8398146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.984+0000 7f839e0c6700 1 --2- 172.21.15.187:0/3898113101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f839815bc50 0x7f839815b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.984+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 >> 172.21.15.187:0/3898113101 conn(0x7f83980bab30 msgr2=0x7f83980afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:27.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.984+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 shutdown_connections 2020-11-20T12:13:27.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:26.984+0000 7f839e0c6700 1 -- 172.21.15.187:0/3898113101 wait complete. 2020-11-20T12:13:27.149 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:13:27.150 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:13:27.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.411+0000 7f3aeffff700 1 Processor -- start 2020-11-20T12:13:27.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.411+0000 7f3aeffff700 1 -- start start 2020-11-20T12:13:27.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.412+0000 7f3aeffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af00b57b0 0x7f3af0068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.412+0000 7f3aeffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3af01482d0 0x7f3af0062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.412+0000 7f3aeffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3af0062b50 0x7f3af0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.412+0000 7f3aeffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af0144b80 con 0x7f3af0062b50 2020-11-20T12:13:27.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.412+0000 7f3aeffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af01489e0 con 0x7f3af01482d0 2020-11-20T12:13:27.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.412+0000 7f3aeffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0148c80 con 0x7f3af00b57b0 2020-11-20T12:13:27.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.413+0000 7f3ae67fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3af01482d0 0x7f3af0062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.413+0000 7f3ae67fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3af01482d0 0x7f3af0062660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47730/0 (socket says 172.21.15.187:47730) 2020-11-20T12:13:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.413+0000 7f3ae67fc700 1 -- 172.21.15.187:0/362857208 learned_addr learned my addr 172.21.15.187:0/362857208 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.413+0000 7f3aeeffd700 1 --2- 172.21.15.187:0/362857208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af00b57b0 0x7f3af0068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.413+0000 7f3aef7fe700 1 --2- 172.21.15.187:0/362857208 >> v2:172.21.15.5:3300/0 conn(0x7f3af0062b50 0x7f3af0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.414+0000 7f3ae67fc700 1 -- 172.21.15.187:0/362857208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af00b57b0 msgr2=0x7f3af0068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:27.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.414+0000 7f3ae67fc700 1 --2- 172.21.15.187:0/362857208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af00b57b0 0x7f3af0068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.414+0000 7f3ae67fc700 1 -- 172.21.15.187:0/362857208 >> v2:172.21.15.5:3300/0 conn(0x7f3af0062b50 msgr2=0x7f3af0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:27.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.414+0000 7f3ae67fc700 1 --2- 172.21.15.187:0/362857208 >> v2:172.21.15.5:3300/0 conn(0x7f3af0062b50 0x7f3af0065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.414+0000 7f3ae67fc700 1 -- 172.21.15.187:0/362857208 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af0148f20 con 0x7f3af01482d0 2020-11-20T12:13:27.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.414+0000 7f3ae67fc700 1 --2- 172.21.15.187:0/362857208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af01482d0 0x7f3af0062660 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7f3af0062fe0 tx=0x7f3ad8006eb0).ready entity=mon.1 client_cookie=e196ca37df149413 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:27.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aee7fc700 1 -- 172.21.15.187:0/362857208 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8009720 con 0x7f3af01482d0 2020-11-20T12:13:27.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aee7fc700 1 -- 172.21.15.187:0/362857208 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad8009880 con 0x7f3af01482d0 2020-11-20T12:13:27.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aee7fc700 1 -- 172.21.15.187:0/362857208 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8009c00 con 0x7f3af01482d0 2020-11-20T12:13:27.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aeffff700 1 -- 172.21.15.187:0/362857208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af01482d0 msgr2=0x7f3af0062660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aeffff700 1 --2- 172.21.15.187:0/362857208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af01482d0 0x7f3af0062660 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7f3af0062fe0 tx=0x7f3ad8006eb0).stop 2020-11-20T12:13:27.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aeffff700 1 -- 172.21.15.187:0/362857208 shutdown_connections 2020-11-20T12:13:27.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aeffff700 1 --2- 172.21.15.187:0/362857208 >> v2:172.21.15.5:3300/0 conn(0x7f3af0062b50 0x7f3af0065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aeffff700 1 --2- 172.21.15.187:0/362857208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af00b57b0 0x7f3af0068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aeffff700 1 --2- 172.21.15.187:0/362857208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af01482d0 0x7f3af0062660 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.415+0000 7f3aeffff700 1 -- 172.21.15.187:0/362857208 >> 172.21.15.187:0/362857208 conn(0x7f3af00bab30 msgr2=0x7f3af00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:27.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.416+0000 7f3aeffff700 1 -- 172.21.15.187:0/362857208 shutdown_connections 2020-11-20T12:13:27.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.416+0000 7f3aeffff700 1 -- 172.21.15.187:0/362857208 wait complete. 2020-11-20T12:13:27.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.417+0000 7f3aeffff700 1 Processor -- start 2020-11-20T12:13:27.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.418+0000 7f3aeffff700 1 -- start start 2020-11-20T12:13:27.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.418+0000 7f3aeffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3af0062b50 0x7f3af0135d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.418+0000 7f3aeffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af00b57b0 0x7f3af0136260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3aeffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af01482d0 0x7f3af015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3aeffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af009eab0 con 0x7f3af0062b50 2020-11-20T12:13:27.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3aeffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0136750 con 0x7f3af00b57b0 2020-11-20T12:13:27.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3aeffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0145700 con 0x7f3af01482d0 2020-11-20T12:13:27.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3ae67fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af00b57b0 0x7f3af0136260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3ae67fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af00b57b0 0x7f3af0136260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47734/0 (socket says 172.21.15.187:47734) 2020-11-20T12:13:27.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3ae67fc700 1 -- 172.21.15.187:0/1586477179 learned_addr learned my addr 172.21.15.187:0/1586477179 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:27.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3aeeffd700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3af0062b50 0x7f3af0135d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3ae67fc700 1 -- 172.21.15.187:0/1586477179 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af01482d0 msgr2=0x7f3af015bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3ae67fc700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af01482d0 0x7f3af015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:27.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3ae67fc700 1 -- 172.21.15.187:0/1586477179 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3af0062b50 msgr2=0x7f3af0135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3ae67fc700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3af0062b50 0x7f3af0135d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.419+0000 7f3ae67fc700 1 -- 172.21.15.187:0/1586477179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af009c260 con 0x7f3af00b57b0 2020-11-20T12:13:27.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.420+0000 7f3ae67fc700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af00b57b0 0x7f3af0136260 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f3ad8009850 tx=0x7f3ad8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:27.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.420+0000 7f3ae77fe700 1 -- 172.21.15.187:0/1586477179 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8011d70 con 0x7f3af00b57b0 2020-11-20T12:13:27.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.420+0000 7f3ae77fe700 1 -- 172.21.15.187:0/1586477179 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad801e460 con 0x7f3af00b57b0 2020-11-20T12:13:27.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.420+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af009ece0 con 0x7f3af00b57b0 2020-11-20T12:13:27.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.420+0000 7f3ae77fe700 1 -- 172.21.15.187:0/1586477179 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad801e800 con 0x7f3af00b57b0 2020-11-20T12:13:27.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.420+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af009bea0 con 0x7f3af00b57b0 2020-11-20T12:13:27.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.421+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af00b1200 con 0x7f3af00b57b0 2020-11-20T12:13:27.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.422+0000 7f3ae77fe700 1 -- 172.21.15.187:0/1586477179 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3ad8025030 con 0x7f3af00b57b0 2020-11-20T12:13:27.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.423+0000 7f3ae77fe700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3acc056d20 0x7f3acc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.423+0000 7f3ae77fe700 1 -- 172.21.15.187:0/1586477179 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ad80716c0 con 0x7f3af00b57b0 2020-11-20T12:13:27.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.423+0000 7f3aeeffd700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3acc056d20 0x7f3acc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.424+0000 7f3aeeffd700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3acc056d20 0x7f3acc0591e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f3af0062040 tx=0x7f3ae0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:27.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.426+0000 7f3ae77fe700 1 -- 172.21.15.187:0/1586477179 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3ad8046b10 con 0x7f3af00b57b0 2020-11-20T12:13:27.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.610+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3af0061a20 con 0x7f3af00b57b0 2020-11-20T12:13:27.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.611+0000 7f3ae77fe700 1 -- 172.21.15.187:0/1586477179 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3ad804a130 con 0x7f3af00b57b0 2020-11-20T12:13:27.615 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:13:27.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.613+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3acc056d20 msgr2=0x7f3acc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.613+0000 7f3aeffff700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3acc056d20 0x7f3acc0591e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f3af0062040 tx=0x7f3ae0008040).stop 2020-11-20T12:13:27.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.613+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af00b57b0 msgr2=0x7f3af0136260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.613+0000 7f3aeffff700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af00b57b0 0x7f3af0136260 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f3ad8009850 tx=0x7f3ad8015000).stop 2020-11-20T12:13:27.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.614+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 shutdown_connections 2020-11-20T12:13:27.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.614+0000 7f3aeffff700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3acc056d20 0x7f3acc0591e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.614+0000 7f3aeffff700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3af0062b50 0x7f3af0135d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.614+0000 7f3aeffff700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3af01482d0 0x7f3af015bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:27.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.614+0000 7f3aeffff700 1 --2- 172.21.15.187:0/1586477179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af00b57b0 0x7f3af0136260 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.614+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 >> 172.21.15.187:0/1586477179 conn(0x7f3af00bab30 msgr2=0x7f3af00b6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:27.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.614+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 shutdown_connections 2020-11-20T12:13:27.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.614+0000 7f3aeffff700 1 -- 172.21.15.187:0/1586477179 wait complete. 2020-11-20T12:13:27.639 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:13:27.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.886+0000 7f363359e700 1 Processor -- start 2020-11-20T12:13:27.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.886+0000 7f363359e700 1 -- start start 2020-11-20T12:13:27.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.887+0000 7f363359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f36341484d0 0x7f3634061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.887+0000 7f363359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3634066f70 0x7f36340b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.887+0000 7f363359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36340b1300 con 0x7f36340622c0 2020-11-20T12:13:27.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.887+0000 7f363359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3634067730 con 0x7f36341484d0 2020-11-20T12:13:27.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.887+0000 7f363359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36340b15a0 con 0x7f3634066f70 2020-11-20T12:13:27.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.888+0000 7f3632d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3634066f70 0x7f36340b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.888+0000 7f3632d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3634066f70 0x7f36340b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47742/0 (socket says 172.21.15.187:47742) 2020-11-20T12:13:27.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.888+0000 7f3632d9d700 1 -- 172.21.15.187:0/344286365 learned_addr learned my addr 172.21.15.187:0/344286365 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:27.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.888+0000 7f363259c700 1 --2- 172.21.15.187:0/344286365 >> v2:172.21.15.5:3300/0 conn(0x7f36341484d0 0x7f3634061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.888+0000 7f3632d9d700 1 -- 172.21.15.187:0/344286365 >> v2:172.21.15.5:3300/0 conn(0x7f36341484d0 msgr2=0x7f3634061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:27.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.888+0000 7f3632d9d700 1 --2- 172.21.15.187:0/344286365 >> v2:172.21.15.5:3300/0 conn(0x7f36341484d0 0x7f3634061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.888+0000 7f3632d9d700 1 -- 172.21.15.187:0/344286365 >> v1:172.21.15.187:6789/0 conn(0x7f36340622c0 legacy=0x7f36340b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.888+0000 7f3632d9d700 1 -- 172.21.15.187:0/344286365 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36340b1840 con 0x7f3634066f70 2020-11-20T12:13:27.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.889+0000 7f3632d9d700 1 --2- 172.21.15.187:0/344286365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b5d50 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f3634045000 tx=0x7f362c006eb0).ready entity=mon.1 client_cookie=93f913b43b177301 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:27.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.889+0000 7f3631d9b700 1 -- 172.21.15.187:0/344286365 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c0098d0 con 0x7f3634066f70 2020-11-20T12:13:27.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.889+0000 7f3631d9b700 1 -- 172.21.15.187:0/344286365 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362c009a30 con 0x7f3634066f70 2020-11-20T12:13:27.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.889+0000 7f3631d9b700 1 -- 172.21.15.187:0/344286365 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c009db0 con 0x7f3634066f70 2020-11-20T12:13:27.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.889+0000 7f363359e700 1 -- 172.21.15.187:0/344286365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 msgr2=0x7f36340b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.889+0000 7f363359e700 1 --2- 172.21.15.187:0/344286365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b5d50 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f3634045000 tx=0x7f362c006eb0).stop 2020-11-20T12:13:27.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.890+0000 7f363359e700 1 -- 172.21.15.187:0/344286365 shutdown_connections 2020-11-20T12:13:27.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.890+0000 7f363359e700 1 --2- 172.21.15.187:0/344286365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b5d50 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.890+0000 7f363359e700 1 --2- 172.21.15.187:0/344286365 >> v2:172.21.15.5:3300/0 conn(0x7f36341484d0 0x7f3634061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.890+0000 7f363359e700 1 -- 172.21.15.187:0/344286365 >> 172.21.15.187:0/344286365 conn(0x7f36340bab30 msgr2=0x7f36340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:27.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.890+0000 7f363359e700 1 -- 172.21.15.187:0/344286365 shutdown_connections 2020-11-20T12:13:27.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.890+0000 7f363359e700 1 -- 172.21.15.187:0/344286365 wait complete. 2020-11-20T12:13:27.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.892+0000 7f363359e700 1 Processor -- start 2020-11-20T12:13:27.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.892+0000 7f363359e700 1 -- start start 2020-11-20T12:13:27.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f363359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36340622c0 0x7f363413e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f363359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f363359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36341484d0 0x7f363413b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f363359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3634158c80 con 0x7f36340622c0 2020-11-20T12:13:27.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f363359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363413c8d0 con 0x7f3634066f70 2020-11-20T12:13:27.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f363359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363409f7f0 con 0x7f36341484d0 2020-11-20T12:13:27.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f3629d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f3629d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47748/0 (socket says 172.21.15.187:47748) 2020-11-20T12:13:27.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.893+0000 7f3629d9b700 1 -- 172.21.15.187:0/2570814079 learned_addr learned my addr 172.21.15.187:0/2570814079 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:27.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f363259c700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36340622c0 0x7f363413e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f3629d9b700 1 -- 172.21.15.187:0/2570814079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36341484d0 msgr2=0x7f363413b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f3629d9b700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36341484d0 0x7f363413b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:27.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f3629d9b700 1 -- 172.21.15.187:0/2570814079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36340622c0 msgr2=0x7f363413e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:27.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f3629d9b700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36340622c0 0x7f363413e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:27.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f3629d9b700 1 -- 172.21.15.187:0/2570814079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36340b4660 con 0x7f3634066f70 2020-11-20T12:13:27.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f3629d9b700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b4170 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f361c00c710 tx=0x7f361c007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:27.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f362affd700 1 -- 172.21.15.187:0/2570814079 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c007020 con 0x7f3634066f70 2020-11-20T12:13:27.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f362affd700 1 -- 172.21.15.187:0/2570814079 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c007180 con 0x7f3634066f70 2020-11-20T12:13:27.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36340b4910 con 0x7f3634066f70 2020-11-20T12:13:27.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f362affd700 1 -- 172.21.15.187:0/2570814079 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c00e560 con 0x7f3634066f70 2020-11-20T12:13:27.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.894+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363413baf0 con 0x7f3634066f70 2020-11-20T12:13:27.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.896+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3634158e80 con 0x7f3634066f70 2020-11-20T12:13:27.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.896+0000 7f362affd700 1 -- 172.21.15.187:0/2570814079 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f361c010070 con 0x7f3634066f70 2020-11-20T12:13:27.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.897+0000 7f362affd700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3610056d20 0x7f36100591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:27.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.897+0000 7f362affd700 1 -- 172.21.15.187:0/2570814079 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f361c06f460 con 0x7f3634066f70 2020-11-20T12:13:27.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.897+0000 7f363259c700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3610056d20 0x7f36100591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:27.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.898+0000 7f363259c700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3610056d20 0x7f36100591e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f3624006fd0 tx=0x7f3624008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:27.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:27.901+0000 7f362affd700 1 -- 172.21.15.187:0/2570814079 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f361c0448b0 con 0x7f3634066f70 2020-11-20T12:13:28.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.088+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3634061a20 con 0x7f3634066f70 2020-11-20T12:13:28.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.089+0000 7f362affd700 1 -- 172.21.15.187:0/2570814079 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7f361c047ed0 con 0x7f3634066f70 2020-11-20T12:13:28.094 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:13:28.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.091+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3610056d20 msgr2=0x7f36100591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.091+0000 7f363359e700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3610056d20 0x7f36100591e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f3624006fd0 tx=0x7f3624008040).stop 2020-11-20T12:13:28.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.091+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 msgr2=0x7f36340b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.091+0000 7f363359e700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b4170 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f361c00c710 tx=0x7f361c007c20).stop 2020-11-20T12:13:28.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.092+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 shutdown_connections 2020-11-20T12:13:28.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.092+0000 7f363359e700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3610056d20 0x7f36100591e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.092+0000 7f363359e700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36340622c0 0x7f363413e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.092+0000 7f363359e700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36341484d0 0x7f363413b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:28.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.092+0000 7f363359e700 1 --2- 172.21.15.187:0/2570814079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3634066f70 0x7f36340b4170 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.092+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 >> 172.21.15.187:0/2570814079 conn(0x7f36340bab30 msgr2=0x7f363413a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:28.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.092+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 shutdown_connections 2020-11-20T12:13:28.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.092+0000 7f363359e700 1 -- 172.21.15.187:0/2570814079 wait complete. 2020-11-20T12:13:28.122 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:28.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.371+0000 7f2089744700 1 Processor -- start 2020-11-20T12:13:28.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.372+0000 7f2089744700 1 -- start start 2020-11-20T12:13:28.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.373+0000 7f2089744700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20840b6650 0x7f20840afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.373+0000 7f2089744700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20840b1300 con 0x7f2084066f70 2020-11-20T12:13:28.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.373+0000 7f2089744700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20840b15a0 con 0x7f20841484d0 2020-11-20T12:13:28.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.373+0000 7f2089744700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2084067730 con 0x7f20840b6650 2020-11-20T12:13:28.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.373+0000 7f2083fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f20841484d0 0x7f2084061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44360/0 (socket says 172.21.15.187:44360) 2020-11-20T12:13:28.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.373+0000 7f2083fff700 1 -- 172.21.15.187:0/3429086143 learned_addr learned my addr 172.21.15.187:0/3429086143 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:28.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986099135 0 0) 0x7f20840b15a0 con 0x7f20841484d0 2020-11-20T12:13:28.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2064003240 con 0x7f20841484d0 2020-11-20T12:13:28.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f207b7fe700 1 --2- 172.21.15.187:0/3429086143 >> v2:172.21.15.5:3300/0 conn(0x7f20840b6650 0x7f20840afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3174962827 0 0) 0x7f2064003240 con 0x7f20841484d0 2020-11-20T12:13:28.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 >> v2:172.21.15.5:3300/0 conn(0x7f20840b6650 msgr2=0x7f20840afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:28.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f20837fe700 1 --2- 172.21.15.187:0/3429086143 >> v2:172.21.15.5:3300/0 conn(0x7f20840b6650 0x7f20840afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 >> v1:172.21.15.5:6789/0 conn(0x7f2084066f70 legacy=0x7f20840aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:28.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20840b1840 con 0x7f20841484d0 2020-11-20T12:13:28.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.374+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f20740014e0 con 0x7f20841484d0 2020-11-20T12:13:28.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 >> v1:172.21.15.187:6789/0 conn(0x7f20841484d0 legacy=0x7f2084061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f20837fe700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2064003800 0x7f2064003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f20837fe700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20640044e0 0x7f2064004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f20837fe700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f206400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2084067730 con 0x7f2064003800 2020-11-20T12:13:28.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2064005300 con 0x7f2064004e50 2020-11-20T12:13:28.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206400de30 con 0x7f20640044e0 2020-11-20T12:13:28.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f2088f43700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f206400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f2083fff700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2064003800 0x7f2064003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f207b7fe700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20640044e0 0x7f2064004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f2088f43700 1 -- 172.21.15.187:0/3429086143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20640044e0 msgr2=0x7f2064004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:28.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.375+0000 7f2088f43700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20640044e0 0x7f2064004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.376+0000 7f2088f43700 1 -- 172.21.15.187:0/3429086143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2064003800 msgr2=0x7f2064003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:28.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.376+0000 7f2088f43700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2064003800 0x7f2064003c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.376+0000 7f2088f43700 1 -- 172.21.15.187:0/3429086143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2064003240 con 0x7f2064004e50 2020-11-20T12:13:28.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.376+0000 7f2088f43700 1 -- 172.21.15.187:0/3429086143 reap_dead start 2020-11-20T12:13:28.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.376+0000 7f2088f43700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f206400b730 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f207c002d80 tx=0x7f207c007f10).ready entity=mon.1 client_cookie=f27791a6c61c5a1f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:28.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.376+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c002430 con 0x7f2064004e50 2020-11-20T12:13:28.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.376+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c00c960 con 0x7f2064004e50 2020-11-20T12:13:28.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.376+0000 7f20837fe700 1 -- 172.21.15.187:0/3429086143 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c00cd20 con 0x7f2064004e50 2020-11-20T12:13:28.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.377+0000 7f2089744700 1 -- 172.21.15.187:0/3429086143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 msgr2=0x7f206400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.377+0000 7f2089744700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f206400b730 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f207c002d80 tx=0x7f207c007f10).stop 2020-11-20T12:13:28.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.377+0000 7f2089744700 1 -- 172.21.15.187:0/3429086143 shutdown_connections 2020-11-20T12:13:28.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.377+0000 7f2089744700 1 --2- 172.21.15.187:0/3429086143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f206400b730 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.377+0000 7f2089744700 1 -- 172.21.15.187:0/3429086143 >> 172.21.15.187:0/3429086143 conn(0x7f20840bab30 msgr2=0x7f20840baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:28.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.377+0000 7f2089744700 1 -- 172.21.15.187:0/3429086143 shutdown_connections 2020-11-20T12:13:28.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.377+0000 7f2089744700 1 -- 172.21.15.187:0/3429086143 wait complete. 2020-11-20T12:13:28.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.378+0000 7f2089744700 1 Processor -- start 2020-11-20T12:13:28.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.378+0000 7f2089744700 1 -- start start 2020-11-20T12:13:28.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.379+0000 7f2089744700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f2084142f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.379+0000 7f2089744700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f208413a370 0x7f20841398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.379+0000 7f2089744700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2084139d90 0x7f208409e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.379+0000 7f2089744700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2084143480 con 0x7f2084139d90 2020-11-20T12:13:28.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.379+0000 7f2089744700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840a0300 con 0x7f2064004e50 2020-11-20T12:13:28.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.379+0000 7f2089744700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20840a05a0 con 0x7f208413a370 2020-11-20T12:13:28.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.380+0000 7f2083fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f2084142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.380+0000 7f2083fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f2084142f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47766/0 (socket says 172.21.15.187:47766) 2020-11-20T12:13:28.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.380+0000 7f2083fff700 1 -- 172.21.15.187:0/1409090612 learned_addr learned my addr 172.21.15.187:0/1409090612 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:28.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.380+0000 7f2083fff700 1 -- 172.21.15.187:0/1409090612 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f208413a370 msgr2=0x7f20841398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.380+0000 7f2083fff700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f208413a370 0x7f20841398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:28.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.380+0000 7f2088f43700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2084139d90 0x7f208409e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.380+0000 7f2083fff700 1 -- 172.21.15.187:0/1409090612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2084139d90 msgr2=0x7f208409e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.380+0000 7f2083fff700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2084139d90 0x7f208409e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.381+0000 7f2083fff700 1 -- 172.21.15.187:0/1409090612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20840a0840 con 0x7f2064004e50 2020-11-20T12:13:28.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.381+0000 7f2083fff700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f2084142f90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7f207400ea50 tx=0x7f2074001e50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:28.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.381+0000 7f2080ff9700 1 -- 172.21.15.187:0/1409090612 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207400a730 con 0x7f2064004e50 2020-11-20T12:13:28.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.381+0000 7f2080ff9700 1 -- 172.21.15.187:0/1409090612 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2074010040 con 0x7f2064004e50 2020-11-20T12:13:28.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.381+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20840a0020 con 0x7f2064004e50 2020-11-20T12:13:28.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.381+0000 7f2080ff9700 1 -- 172.21.15.187:0/1409090612 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207400aac0 con 0x7f2064004e50 2020-11-20T12:13:28.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.381+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20840b3c30 con 0x7f2064004e50 2020-11-20T12:13:28.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.382+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2084066900 con 0x7f2064004e50 2020-11-20T12:13:28.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.383+0000 7f2080ff9700 1 -- 172.21.15.187:0/1409090612 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f207401b020 con 0x7f2064004e50 2020-11-20T12:13:28.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.384+0000 7f2080ff9700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2060056d20 0x7f20600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.384+0000 7f2080ff9700 1 -- 172.21.15.187:0/1409090612 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f207406f700 con 0x7f2064004e50 2020-11-20T12:13:28.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.384+0000 7f207b7fe700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2060056d20 0x7f20600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.385+0000 7f207b7fe700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2060056d20 0x7f20600591e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f206c006a70 tx=0x7f206c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:28.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.388+0000 7f2080ff9700 1 -- 172.21.15.187:0/1409090612 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2074044ad0 con 0x7f2064004e50 2020-11-20T12:13:28.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.633+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20840a09d0 con 0x7f2064004e50 2020-11-20T12:13:28.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.634+0000 7f2080ff9700 1 -- 172.21.15.187:0/1409090612 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f20740480f0 con 0x7f2064004e50 2020-11-20T12:13:28.638 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:28.639 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/797250225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":797250225},{"type":"v1","addr":"172.21.15.5:6833","nonce":797250225}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:12:10.314184+0000","modified":"2020-11-20T12:13:19.215710+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6952},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6952":{"gid":6952,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1106173813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1106173813},{"type":"v1","addr":"172.21.15.187:6835","nonce":1106173813}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2020-11-20T12:13:28.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2060056d20 msgr2=0x7f20600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2060056d20 0x7f20600591e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f206c006a70 tx=0x7f206c008040).stop 2020-11-20T12:13:28.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 msgr2=0x7f2084142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f2084142f90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7f207400ea50 tx=0x7f2074001e50).stop 2020-11-20T12:13:28.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 shutdown_connections 2020-11-20T12:13:28.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2060056d20 0x7f20600591e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2084139d90 0x7f208409e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2064004e50 0x7f2084142f90 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 --2- 172.21.15.187:0/1409090612 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f208413a370 0x7f20841398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:28.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 >> 172.21.15.187:0/1409090612 conn(0x7f20840bab30 msgr2=0x7f2084144ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:28.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 shutdown_connections 2020-11-20T12:13:28.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.637+0000 7f2089744700 1 -- 172.21.15.187:0/1409090612 wait complete. 2020-11-20T12:13:28.643 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 73 2020-11-20T12:13:28.663 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:13:28.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.895+0000 7f4985966700 1 Processor -- start 2020-11-20T12:13:28.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.895+0000 7f4985966700 1 -- start start 2020-11-20T12:13:28.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.896+0000 7f4985966700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49800b6650 0x7f49800afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.896+0000 7f4985966700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4980066f70 0x7f49800aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.896+0000 7f4985966700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49800b1300 con 0x7f49801484d0 2020-11-20T12:13:28.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.896+0000 7f4985966700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4980067730 con 0x7f4980066f70 2020-11-20T12:13:28.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.896+0000 7f4985966700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49800b15a0 con 0x7f49800b6650 2020-11-20T12:13:28.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.897+0000 7f497ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49800b6650 0x7f49800afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.898+0000 7f497ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49800b6650 0x7f49800afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:48664/0 (socket says 172.21.15.187:48664) 2020-11-20T12:13:28.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.898+0000 7f497ffff700 1 -- 172.21.15.187:0/2870699761 learned_addr learned my addr 172.21.15.187:0/2870699761 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:28.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.898+0000 7f4985165700 1 --2- 172.21.15.187:0/2870699761 >> v2:172.21.15.5:3300/0 conn(0x7f4980066f70 0x7f49800aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.898+0000 7f497ffff700 1 -- 172.21.15.187:0/2870699761 >> v2:172.21.15.5:3300/0 conn(0x7f4980066f70 msgr2=0x7f49800aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:28.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.898+0000 7f497ffff700 1 --2- 172.21.15.187:0/2870699761 >> v2:172.21.15.5:3300/0 conn(0x7f4980066f70 0x7f49800aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.898+0000 7f497ffff700 1 -- 172.21.15.187:0/2870699761 >> v1:172.21.15.5:6789/0 conn(0x7f49801484d0 legacy=0x7f4980061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.898+0000 7f497ffff700 1 -- 172.21.15.187:0/2870699761 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49800b1840 con 0x7f49800b6650 2020-11-20T12:13:28.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.899+0000 7f497ffff700 1 --2- 172.21.15.187:0/2870699761 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49800b6650 0x7f49800afa90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f496c006fd0 tx=0x7f496c006eb0).ready entity=mon.2 client_cookie=760b2a31d5d389c8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:28.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.899+0000 7f497f7fe700 1 -- 172.21.15.187:0/2870699761 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c009810 con 0x7f49800b6650 2020-11-20T12:13:28.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.899+0000 7f497f7fe700 1 -- 172.21.15.187:0/2870699761 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496c009970 con 0x7f49800b6650 2020-11-20T12:13:28.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.899+0000 7f497f7fe700 1 -- 172.21.15.187:0/2870699761 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c009cf0 con 0x7f49800b6650 2020-11-20T12:13:28.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.900+0000 7f4985966700 1 -- 172.21.15.187:0/2870699761 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49800b6650 msgr2=0x7f49800afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.900+0000 7f4985966700 1 --2- 172.21.15.187:0/2870699761 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49800b6650 0x7f49800afa90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f496c006fd0 tx=0x7f496c006eb0).stop 2020-11-20T12:13:28.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.900+0000 7f4985966700 1 -- 172.21.15.187:0/2870699761 shutdown_connections 2020-11-20T12:13:28.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.900+0000 7f4985966700 1 --2- 172.21.15.187:0/2870699761 >> v2:172.21.15.5:3300/0 conn(0x7f4980066f70 0x7f49800aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.900+0000 7f4985966700 1 --2- 172.21.15.187:0/2870699761 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49800b6650 0x7f49800afa90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.900+0000 7f4985966700 1 -- 172.21.15.187:0/2870699761 >> 172.21.15.187:0/2870699761 conn(0x7f49800bab30 msgr2=0x7f49800baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:28.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.900+0000 7f4985966700 1 -- 172.21.15.187:0/2870699761 shutdown_connections 2020-11-20T12:13:28.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.900+0000 7f4985966700 1 -- 172.21.15.187:0/2870699761 wait complete. 2020-11-20T12:13:28.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.902+0000 7f4985966700 1 Processor -- start 2020-11-20T12:13:28.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.903+0000 7f4985966700 1 -- start start 2020-11-20T12:13:28.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4985966700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4980066f70 0x7f498009f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4985966700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49800b6650 0x7f4980143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4985966700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49801484d0 0x7f498015fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4985966700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4980145a40 con 0x7f49800b6650 2020-11-20T12:13:28.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4985966700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498009fa00 con 0x7f4980066f70 2020-11-20T12:13:28.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4985966700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498009f7a0 con 0x7f49801484d0 2020-11-20T12:13:28.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4984964700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4980066f70 0x7f498009f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4984964700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4980066f70 0x7f498009f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47780/0 (socket says 172.21.15.187:47780) 2020-11-20T12:13:28.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.904+0000 7f4984964700 1 -- 172.21.15.187:0/3368959283 learned_addr learned my addr 172.21.15.187:0/3368959283 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:28.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f497ffff700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49800b6650 0x7f4980143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4985165700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49801484d0 0x7f498015fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4984964700 1 -- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49801484d0 msgr2=0x7f498015fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4984964700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49801484d0 0x7f498015fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4984964700 1 -- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49800b6650 msgr2=0x7f4980143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:28.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4984964700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49800b6650 0x7f4980143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:28.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4984964700 1 -- 172.21.15.187:0/3368959283 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498009a9c0 con 0x7f4980066f70 2020-11-20T12:13:28.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4985165700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49801484d0 0x7f498015fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:28.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f497ffff700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49800b6650 0x7f4980143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:28.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4984964700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4980066f70 0x7f498009f2b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f497400c710 tx=0x7f4974007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:28.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f497cff9700 1 -- 172.21.15.187:0/3368959283 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497400e560 con 0x7f4980066f70 2020-11-20T12:13:28.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f497cff9700 1 -- 172.21.15.187:0/3368959283 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497400e6c0 con 0x7f4980066f70 2020-11-20T12:13:28.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.905+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498009ac60 con 0x7f4980066f70 2020-11-20T12:13:28.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.906+0000 7f497cff9700 1 -- 172.21.15.187:0/3368959283 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497400ea40 con 0x7f4980066f70 2020-11-20T12:13:28.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.906+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4980160020 con 0x7f4980066f70 2020-11-20T12:13:28.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.907+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498009ae40 con 0x7f4980066f70 2020-11-20T12:13:28.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.907+0000 7f497cff9700 1 -- 172.21.15.187:0/3368959283 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4974010030 con 0x7f4980066f70 2020-11-20T12:13:28.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.908+0000 7f497cff9700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4958056d20 0x7f49580591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:28.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.908+0000 7f497cff9700 1 -- 172.21.15.187:0/3368959283 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f497406fdc0 con 0x7f4980066f70 2020-11-20T12:13:28.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.908+0000 7f497ffff700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4958056d20 0x7f49580591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:28.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.909+0000 7f497ffff700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4958056d20 0x7f49580591e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f49800b4460 tx=0x7f496c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:28.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:28.912+0000 7f497cff9700 1 -- 172.21.15.187:0/3368959283 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f498009ae40 con 0x7f4980066f70 2020-11-20T12:13:29.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.099+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4980160300 con 0x7f4980066f70 2020-11-20T12:13:29.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.100+0000 7f497cff9700 1 -- 172.21.15.187:0/3368959283 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f49740451b0 con 0x7f4980066f70 2020-11-20T12:13:29.105 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:29.105 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":81,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:12:11.318992+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:12:07.846671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2020-11-20T12:12:09.337957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:13:29.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4958056d20 msgr2=0x7f49580591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:29.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4958056d20 0x7f49580591e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f49800b4460 tx=0x7f496c015000).stop 2020-11-20T12:13:29.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4980066f70 msgr2=0x7f498009f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:29.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4980066f70 0x7f498009f2b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f497400c710 tx=0x7f4974007c00).stop 2020-11-20T12:13:29.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 shutdown_connections 2020-11-20T12:13:29.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4958056d20 0x7f49580591e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f49801484d0 0x7f498015fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4980066f70 0x7f498009f2b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.102+0000 7f4985966700 1 --2- 172.21.15.187:0/3368959283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49800b6650 0x7f4980143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.103+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 >> 172.21.15.187:0/3368959283 conn(0x7f49800bab30 msgr2=0x7f4980145120 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:29.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.103+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 shutdown_connections 2020-11-20T12:13:29.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.103+0000 7f4985966700 1 -- 172.21.15.187:0/3368959283 wait complete. 2020-11-20T12:13:29.130 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:13:29.130 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:13:29.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.378+0000 7fbdef00c700 1 Processor -- start 2020-11-20T12:13:29.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.379+0000 7fbdef00c700 1 -- start start 2020-11-20T12:13:29.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.380+0000 7fbdef00c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde81484d0 0x7fbde8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.380+0000 7fbdef00c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbde80622c0 0x7fbde80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.380+0000 7fbdef00c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde80b1300 con 0x7fbde8066f70 2020-11-20T12:13:29.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.380+0000 7fbdef00c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde8067730 con 0x7fbde80622c0 2020-11-20T12:13:29.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.380+0000 7fbdef00c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80b15a0 con 0x7fbde81484d0 2020-11-20T12:13:29.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.380+0000 7fbdee80b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbde8066f70 0x7fbde80b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44398/0 (socket says 172.21.15.187:44398) 2020-11-20T12:13:29.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.380+0000 7fbdee80b700 1 -- 172.21.15.187:0/2407164615 learned_addr learned my addr 172.21.15.187:0/2407164615 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:29.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.381+0000 7fbded809700 1 --2- 172.21.15.187:0/2407164615 >> v2:172.21.15.5:3300/0 conn(0x7fbde80622c0 0x7fbde80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:29.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.381+0000 7fbdee00a700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde81484d0 0x7fbde8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:29.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.381+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947414859 0 0) 0x7fbde80b1300 con 0x7fbde8066f70 2020-11-20T12:13:29.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.381+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd0003240 con 0x7fbde8066f70 2020-11-20T12:13:29.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3787063829 0 0) 0x7fbdd0003240 con 0x7fbde8066f70 2020-11-20T12:13:29.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde81484d0 msgr2=0x7fbde8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:29.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde81484d0 0x7fbde8061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 >> v2:172.21.15.5:3300/0 conn(0x7fbde80622c0 msgr2=0x7fbde80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:29.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 --2- 172.21.15.187:0/2407164615 >> v2:172.21.15.5:3300/0 conn(0x7fbde80622c0 0x7fbde80b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80b1840 con 0x7fbde8066f70 2020-11-20T12:13:29.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fbde0001860 con 0x7fbde8066f70 2020-11-20T12:13:29.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 >> v1:172.21.15.187:6789/0 conn(0x7fbde8066f70 legacy=0x7fbde80b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:29.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdd0003730 0x7fbdd0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde81484d0 0x7fbdd00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdd00049d0 0x7fbdd0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8067730 con 0x7fbdd0003730 2020-11-20T12:13:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80b15a0 con 0x7fbdd00049d0 2020-11-20T12:13:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00041d0 con 0x7fbde81484d0 2020-11-20T12:13:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.382+0000 7fbdee80b700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdd00049d0 0x7fbdd0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbded809700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde81484d0 0x7fbdd00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbdee80b700 1 -- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde81484d0 msgr2=0x7fbdd00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:29.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbdee80b700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde81484d0 0x7fbdd00044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbdee80b700 1 -- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdd0003730 msgr2=0x7fbdd0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:29.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbdee80b700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdd0003730 0x7fbdd0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:29.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbdee80b700 1 -- 172.21.15.187:0/2407164615 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd0003240 con 0x7fbdd00049d0 2020-11-20T12:13:29.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbdee80b700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdd00049d0 0x7fbdd0004e50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7fbde000c840 tx=0x7fbde0007b80).ready entity=mon.1 client_cookie=d81253a5d61d8e1c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:29.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0005b80 con 0x7fbdd00049d0 2020-11-20T12:13:29.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0005ce0 con 0x7fbdd00049d0 2020-11-20T12:13:29.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.383+0000 7fbded008700 1 -- 172.21.15.187:0/2407164615 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0007320 con 0x7fbdd00049d0 2020-11-20T12:13:29.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.384+0000 7fbdef00c700 1 -- 172.21.15.187:0/2407164615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdd00049d0 msgr2=0x7fbdd0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:29.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.384+0000 7fbdef00c700 1 --2- 172.21.15.187:0/2407164615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdd00049d0 0x7fbdd0004e50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7fbde000c840 tx=0x7fbde0007b80).stop 2020-11-20T12:13:29.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.384+0000 7fbdee80b700 1 -- 172.21.15.187:0/2407164615 reap_dead start 2020-11-20T12:13:29.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.384+0000 7fbdef00c700 1 -- 172.21.15.187:0/2407164615 shutdown_connections 2020-11-20T12:13:29.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.384+0000 7fbdef00c700 1 -- 172.21.15.187:0/2407164615 >> 172.21.15.187:0/2407164615 conn(0x7fbde80bab30 msgr2=0x7fbde80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:29.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.384+0000 7fbdef00c700 1 -- 172.21.15.187:0/2407164615 shutdown_connections 2020-11-20T12:13:29.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.384+0000 7fbdef00c700 1 -- 172.21.15.187:0/2407164615 wait complete. 2020-11-20T12:13:29.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.386+0000 7fbdef00c700 1 Processor -- start 2020-11-20T12:13:29.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.386+0000 7fbdef00c700 1 -- start start 2020-11-20T12:13:29.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdef00c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde8157210 0x7fbde809e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdef00c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbde81575f0 0x7fbde8153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdef00c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbde809ecf0 0x7fbde8153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdef00c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80b45d0 con 0x7fbde81575f0 2020-11-20T12:13:29.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdef00c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde809fd70 con 0x7fbde809ecf0 2020-11-20T12:13:29.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdef00c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde809f990 con 0x7fbde8157210 2020-11-20T12:13:29.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdee80b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbde809ecf0 0x7fbde8153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:29.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdee80b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbde809ecf0 0x7fbde8153650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47804/0 (socket says 172.21.15.187:47804) 2020-11-20T12:13:29.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.387+0000 7fbdee80b700 1 -- 172.21.15.187:0/1658065704 learned_addr learned my addr 172.21.15.187:0/1658065704 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:29.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbded809700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbde81575f0 0x7fbde8153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:29.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdee00a700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde8157210 0x7fbde809e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:29.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdee80b700 1 -- 172.21.15.187:0/1658065704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde8157210 msgr2=0x7fbde809e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:29.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdee80b700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde8157210 0x7fbde809e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdee80b700 1 -- 172.21.15.187:0/1658065704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbde81575f0 msgr2=0x7fbde8153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:29.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdee80b700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbde81575f0 0x7fbde8153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:29.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdee80b700 1 -- 172.21.15.187:0/1658065704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8157a70 con 0x7fbde809ecf0 2020-11-20T12:13:29.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdee80b700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbde809ecf0 0x7fbde8153650 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7fbde00011e0 tx=0x7fbde0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:29.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdd67fc700 1 -- 172.21.15.187:0/1658065704 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00078a0 con 0x7fbde809ecf0 2020-11-20T12:13:29.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdd67fc700 1 -- 172.21.15.187:0/1658065704 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0005e50 con 0x7fbde809ecf0 2020-11-20T12:13:29.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdd67fc700 1 -- 172.21.15.187:0/1658065704 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0013ac0 con 0x7fbde809ecf0 2020-11-20T12:13:29.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde8157d10 con 0x7fbde809ecf0 2020-11-20T12:13:29.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.388+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde809d400 con 0x7fbde809ecf0 2020-11-20T12:13:29.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.389+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde8157ea0 con 0x7fbde809ecf0 2020-11-20T12:13:29.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.390+0000 7fbdd67fc700 1 -- 172.21.15.187:0/1658065704 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbde0010030 con 0x7fbde809ecf0 2020-11-20T12:13:29.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.391+0000 7fbdd67fc700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbdc0056d20 0x7fbdc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:29.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.391+0000 7fbdd67fc700 1 -- 172.21.15.187:0/1658065704 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbde0071980 con 0x7fbde809ecf0 2020-11-20T12:13:29.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.391+0000 7fbdee00a700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbdc0056d20 0x7fbdc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:29.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.392+0000 7fbdee00a700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbdc0056d20 0x7fbdc00591e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fbde4006fd0 tx=0x7fbde4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:29.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.395+0000 7fbdd67fc700 1 -- 172.21.15.187:0/1658065704 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbde0046d50 con 0x7fbde809ecf0 2020-11-20T12:13:29.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:29.649+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbde8061a20 con 0x7fbde809ecf0 2020-11-20T12:13:30.297 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:13:30.293+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:13:30.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.309+0000 7fbdd67fc700 1 -- 172.21.15.187:0/1658065704 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbde0024070 con 0x7fbde809ecf0 2020-11-20T12:13:30.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.312+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbdc0056d20 msgr2=0x7fbdc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:30.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.312+0000 7fbdef00c700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbdc0056d20 0x7fbdc00591e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fbde4006fd0 tx=0x7fbde4008040).stop 2020-11-20T12:13:30.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.312+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbde809ecf0 msgr2=0x7fbde8153650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:30.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.312+0000 7fbdef00c700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbde809ecf0 0x7fbde8153650 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7fbde00011e0 tx=0x7fbde0016000).stop 2020-11-20T12:13:30.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.312+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 shutdown_connections 2020-11-20T12:13:30.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.313+0000 7fbdef00c700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbdc0056d20 0x7fbdc00591e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:30.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.313+0000 7fbdef00c700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbde809ecf0 0x7fbde8153650 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:30.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.313+0000 7fbdef00c700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbde8157210 0x7fbde809e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:30.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.313+0000 7fbdef00c700 1 --2- 172.21.15.187:0/1658065704 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbde81575f0 0x7fbde8153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:30.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.313+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 >> 172.21.15.187:0/1658065704 conn(0x7fbde80bab30 msgr2=0x7fbde80b61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:30.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.313+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 shutdown_connections 2020-11-20T12:13:30.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:30.313+0000 7fbdef00c700 1 -- 172.21.15.187:0/1658065704 wait complete. 2020-11-20T12:13:30.842 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:13:30.842 INFO:tasks.ceph.mds.a.smithi187.stderr: -5946> 2020-11-20T12:12:14.525+0000 7ff4a9e99700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:13:30.843 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:13:30.843 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:13:31.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.104+0000 7f44a698b700 1 Processor -- start 2020-11-20T12:13:31.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.104+0000 7f44a698b700 1 -- start start 2020-11-20T12:13:31.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.106+0000 7f44a698b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a00bfbe0 0x7f44a01480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:31.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.106+0000 7f44a698b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f44a009f560 0x7f44a01485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:31.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.106+0000 7f44a698b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a014d110 con 0x7f44a00bf800 2020-11-20T12:13:31.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.106+0000 7f44a698b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a014c9d0 con 0x7f44a009f560 2020-11-20T12:13:31.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.106+0000 7f44a698b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0148aa0 con 0x7f44a00bfbe0 2020-11-20T12:13:31.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.107+0000 7f44a5188700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a00bfbe0 0x7f44a01480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:31.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.107+0000 7f44a5188700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a00bfbe0 0x7f44a01480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:48710/0 (socket says 172.21.15.187:48710) 2020-11-20T12:13:31.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.107+0000 7f44a5188700 1 -- 172.21.15.187:0/459944048 learned_addr learned my addr 172.21.15.187:0/459944048 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:31.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.107+0000 7f44a618a700 1 --2- 172.21.15.187:0/459944048 >> v2:172.21.15.5:3300/0 conn(0x7f44a009f560 0x7f44a01485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:31.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.108+0000 7f44a5188700 1 -- 172.21.15.187:0/459944048 >> v2:172.21.15.5:3300/0 conn(0x7f44a009f560 msgr2=0x7f44a01485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:31.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.108+0000 7f44a5188700 1 --2- 172.21.15.187:0/459944048 >> v2:172.21.15.5:3300/0 conn(0x7f44a009f560 0x7f44a01485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:31.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.108+0000 7f44a5188700 1 -- 172.21.15.187:0/459944048 >> v1:172.21.15.5:6789/0 conn(0x7f44a00bf800 legacy=0x7f44a014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:31.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.108+0000 7f44a5188700 1 -- 172.21.15.187:0/459944048 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a014cbd0 con 0x7f44a00bfbe0 2020-11-20T12:13:31.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.108+0000 7f44a5188700 1 --2- 172.21.15.187:0/459944048 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a00bfbe0 0x7f44a01480c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f449000ba20 tx=0x7f4490006eb0).ready entity=mon.2 client_cookie=514641d2f1ff59f5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:31.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a4987700 1 -- 172.21.15.187:0/459944048 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490009860 con 0x7f44a00bfbe0 2020-11-20T12:13:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a4987700 1 -- 172.21.15.187:0/459944048 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44900099c0 con 0x7f44a00bfbe0 2020-11-20T12:13:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a4987700 1 -- 172.21.15.187:0/459944048 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490009d40 con 0x7f44a00bfbe0 2020-11-20T12:13:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a698b700 1 -- 172.21.15.187:0/459944048 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a00bfbe0 msgr2=0x7f44a01480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a698b700 1 --2- 172.21.15.187:0/459944048 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a00bfbe0 0x7f44a01480c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f449000ba20 tx=0x7f4490006eb0).stop 2020-11-20T12:13:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a698b700 1 -- 172.21.15.187:0/459944048 shutdown_connections 2020-11-20T12:13:31.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a698b700 1 --2- 172.21.15.187:0/459944048 >> v2:172.21.15.5:3300/0 conn(0x7f44a009f560 0x7f44a01485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:31.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a698b700 1 --2- 172.21.15.187:0/459944048 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a00bfbe0 0x7f44a01480c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:31.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a698b700 1 -- 172.21.15.187:0/459944048 >> 172.21.15.187:0/459944048 conn(0x7f44a0068370 msgr2=0x7f44a00bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:31.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a698b700 1 -- 172.21.15.187:0/459944048 shutdown_connections 2020-11-20T12:13:31.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.109+0000 7f44a698b700 1 -- 172.21.15.187:0/459944048 wait complete. 2020-11-20T12:13:31.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.111+0000 7f44a698b700 1 Processor -- start 2020-11-20T12:13:31.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.111+0000 7f44a698b700 1 -- start start 2020-11-20T12:13:31.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a698b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a009f560 0x7f44a015c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:31.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a698b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f44a00bf800 0x7f44a00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:31.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a698b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44a00b5b70 0x7f44a014ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:31.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a698b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00b94f0 con 0x7f44a00bf800 2020-11-20T12:13:31.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a698b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00bad60 con 0x7f44a00b5b70 2020-11-20T12:13:31.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a698b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00bb000 con 0x7f44a009f560 2020-11-20T12:13:31.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a618a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44a00b5b70 0x7f44a014ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:31.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a618a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44a00b5b70 0x7f44a014ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47830/0 (socket says 172.21.15.187:47830) 2020-11-20T12:13:31.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.112+0000 7f44a618a700 1 -- 172.21.15.187:0/4838967 learned_addr learned my addr 172.21.15.187:0/4838967 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:31.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.113+0000 7f44a5188700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f44a00bf800 0x7f44a00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:31.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.113+0000 7f44a5989700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a009f560 0x7f44a015c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:31.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.113+0000 7f44a618a700 1 -- 172.21.15.187:0/4838967 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a009f560 msgr2=0x7f44a015c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:31.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.113+0000 7f44a618a700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a009f560 0x7f44a015c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:31.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.113+0000 7f44a618a700 1 -- 172.21.15.187:0/4838967 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f44a00bf800 msgr2=0x7f44a00b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:31.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.113+0000 7f44a618a700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f44a00bf800 0x7f44a00b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:31.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.113+0000 7f44a618a700 1 -- 172.21.15.187:0/4838967 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a00baa50 con 0x7f44a00b5b70 2020-11-20T12:13:31.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.113+0000 7f44a618a700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44a00b5b70 0x7f44a014ac20 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f449800ebd0 tx=0x7f4498006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:31.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.114+0000 7f448dffb700 1 -- 172.21.15.187:0/4838967 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449800c820 con 0x7f44a00b5b70 2020-11-20T12:13:31.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.114+0000 7f448dffb700 1 -- 172.21.15.187:0/4838967 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449800c980 con 0x7f44a00b5b70 2020-11-20T12:13:31.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.114+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a00b85b0 con 0x7f44a00b5b70 2020-11-20T12:13:31.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.114+0000 7f448dffb700 1 -- 172.21.15.187:0/4838967 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449800cd20 con 0x7f44a00b5b70 2020-11-20T12:13:31.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.114+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a00b6ce0 con 0x7f44a00b5b70 2020-11-20T12:13:31.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.115+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a00b8740 con 0x7f44a00b5b70 2020-11-20T12:13:31.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.115+0000 7f448dffb700 1 -- 172.21.15.187:0/4838967 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4498012600 con 0x7f44a00b5b70 2020-11-20T12:13:31.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.116+0000 7f448dffb700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4478056d20 0x7f44780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:31.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.116+0000 7f448dffb700 1 -- 172.21.15.187:0/4838967 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4498070410 con 0x7f44a00b5b70 2020-11-20T12:13:31.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.117+0000 7f44a5989700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4478056d20 0x7f44780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:31.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.117+0000 7f44a5989700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4478056d20 0x7f44780591e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f44a00b4130 tx=0x7f449c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:31.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.120+0000 7f448dffb700 1 -- 172.21.15.187:0/4838967 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f44a00b8740 con 0x7f44a00b5b70 2020-11-20T12:13:31.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:31.367+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f44a00b8740 con 0x7f44a00b5b70 2020-11-20T12:13:32.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.318+0000 7f448dffb700 1 -- 172.21.15.187:0/4838967 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7f44a00b8740 con 0x7f44a00b5b70 2020-11-20T12:13:32.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4478056d20 msgr2=0x7f44780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:32.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4478056d20 0x7f44780591e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f44a00b4130 tx=0x7f449c009040).stop 2020-11-20T12:13:32.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44a00b5b70 msgr2=0x7f44a014ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:32.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44a00b5b70 0x7f44a014ac20 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f449800ebd0 tx=0x7f4498006df0).stop 2020-11-20T12:13:32.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 shutdown_connections 2020-11-20T12:13:32.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4478056d20 0x7f44780591e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44a00b5b70 0x7f44a014ac20 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44a009f560 0x7f44a015c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 --2- 172.21.15.187:0/4838967 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f44a00bf800 0x7f44a00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.322+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 >> 172.21.15.187:0/4838967 conn(0x7f44a0068370 msgr2=0x7f44a014bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:32.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.323+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 shutdown_connections 2020-11-20T12:13:32.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.323+0000 7f44a698b700 1 -- 172.21.15.187:0/4838967 wait complete. 2020-11-20T12:13:32.358 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:13:32.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.606+0000 7fad9be00700 1 Processor -- start 2020-11-20T12:13:32.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.607+0000 7fad9be00700 1 -- start start 2020-11-20T12:13:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.608+0000 7fad9be00700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fad94066210 0x7fad94145010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.608+0000 7fad9be00700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fad940b52f0 0x7fad94145500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.608+0000 7fad9be00700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad94148b60 con 0x7fad94065e30 2020-11-20T12:13:32.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.608+0000 7fad9be00700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad94149540 con 0x7fad94066210 2020-11-20T12:13:32.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.608+0000 7fad9be00700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad940b72c0 con 0x7fad940b52f0 2020-11-20T12:13:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9a39d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fad940b52f0 0x7fad94145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9a39d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fad940b52f0 0x7fad94145500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47838/0 (socket says 172.21.15.187:47838) 2020-11-20T12:13:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9a39d700 1 -- 172.21.15.187:0/4112397937 learned_addr learned my addr 172.21.15.187:0/4112397937 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:32.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9939b700 1 --2- 172.21.15.187:0/4112397937 >> v2:172.21.15.5:3300/0 conn(0x7fad94066210 0x7fad94145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:32.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9a39d700 1 -- 172.21.15.187:0/4112397937 >> v2:172.21.15.5:3300/0 conn(0x7fad94066210 msgr2=0x7fad94145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:32.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9a39d700 1 --2- 172.21.15.187:0/4112397937 >> v2:172.21.15.5:3300/0 conn(0x7fad94066210 0x7fad94145010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9a39d700 1 -- 172.21.15.187:0/4112397937 >> v1:172.21.15.5:6789/0 conn(0x7fad94065e30 legacy=0x7fad94099da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:32.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9a39d700 1 -- 172.21.15.187:0/4112397937 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94148e00 con 0x7fad940b52f0 2020-11-20T12:13:32.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.609+0000 7fad9939b700 1 --2- 172.21.15.187:0/4112397937 >> v2:172.21.15.5:3300/0 conn(0x7fad94066210 0x7fad94145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:32.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.610+0000 7fad9a39d700 1 --2- 172.21.15.187:0/4112397937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad94145500 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7fad940baba0 tx=0x7fad8c009290).ready entity=mon.1 client_cookie=bb59dec924fe501d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:32.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.610+0000 7fad98b9a700 1 -- 172.21.15.187:0/4112397937 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00bb80 con 0x7fad940b52f0 2020-11-20T12:13:32.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.610+0000 7fad98b9a700 1 -- 172.21.15.187:0/4112397937 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c00bce0 con 0x7fad940b52f0 2020-11-20T12:13:32.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.610+0000 7fad98b9a700 1 -- 172.21.15.187:0/4112397937 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c011610 con 0x7fad940b52f0 2020-11-20T12:13:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.611+0000 7fad9be00700 1 -- 172.21.15.187:0/4112397937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 msgr2=0x7fad94145500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.611+0000 7fad9be00700 1 --2- 172.21.15.187:0/4112397937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad94145500 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7fad940baba0 tx=0x7fad8c009290).stop 2020-11-20T12:13:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.611+0000 7fad9be00700 1 -- 172.21.15.187:0/4112397937 shutdown_connections 2020-11-20T12:13:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.611+0000 7fad9be00700 1 --2- 172.21.15.187:0/4112397937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad94145500 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.611+0000 7fad9be00700 1 --2- 172.21.15.187:0/4112397937 >> v2:172.21.15.5:3300/0 conn(0x7fad94066210 0x7fad94145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.611+0000 7fad9be00700 1 -- 172.21.15.187:0/4112397937 >> 172.21.15.187:0/4112397937 conn(0x7fad940bacf0 msgr2=0x7fad940bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:32.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.611+0000 7fad9be00700 1 -- 172.21.15.187:0/4112397937 shutdown_connections 2020-11-20T12:13:32.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.611+0000 7fad9be00700 1 -- 172.21.15.187:0/4112397937 wait complete. 2020-11-20T12:13:32.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.613+0000 7fad9be00700 1 Processor -- start 2020-11-20T12:13:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.613+0000 7fad9be00700 1 -- start start 2020-11-20T12:13:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.614+0000 7fad9be00700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad94065e30 0x7fad940b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.614+0000 7fad9be00700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad94066210 0x7fad94139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.614+0000 7fad9be00700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad9413a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:32.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.614+0000 7fad9be00700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad94144db0 con 0x7fad94066210 2020-11-20T12:13:32.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.614+0000 7fad9be00700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9413a880 con 0x7fad940b52f0 2020-11-20T12:13:32.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.614+0000 7fad9be00700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad940b3950 con 0x7fad94065e30 2020-11-20T12:13:32.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9a39d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad9413a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:32.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9a39d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad9413a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47844/0 (socket says 172.21.15.187:47844) 2020-11-20T12:13:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9a39d700 1 -- 172.21.15.187:0/2391891114 learned_addr learned my addr 172.21.15.187:0/2391891114 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad99b9c700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad94065e30 0x7fad940b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9a39d700 1 -- 172.21.15.187:0/2391891114 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad94065e30 msgr2=0x7fad940b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9939b700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad94066210 0x7fad94139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9a39d700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad94065e30 0x7fad940b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9a39d700 1 -- 172.21.15.187:0/2391891114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad94066210 msgr2=0x7fad94139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:32.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9a39d700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad94066210 0x7fad94139ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:32.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.615+0000 7fad9a39d700 1 -- 172.21.15.187:0/2391891114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9409acd0 con 0x7fad940b52f0 2020-11-20T12:13:32.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.616+0000 7fad9a39d700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad9413a390 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7fad8c0092d0 tx=0x7fad8c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:32.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.616+0000 7fad85ffb700 1 -- 172.21.15.187:0/2391891114 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c020070 con 0x7fad940b52f0 2020-11-20T12:13:32.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.616+0000 7fad85ffb700 1 -- 172.21.15.187:0/2391891114 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c00be50 con 0x7fad940b52f0 2020-11-20T12:13:32.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.616+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9409af70 con 0x7fad940b52f0 2020-11-20T12:13:32.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.616+0000 7fad85ffb700 1 -- 172.21.15.187:0/2391891114 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c010db0 con 0x7fad940b52f0 2020-11-20T12:13:32.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.616+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad940b35a0 con 0x7fad940b52f0 2020-11-20T12:13:32.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.617+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9409b130 con 0x7fad940b52f0 2020-11-20T12:13:32.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.618+0000 7fad85ffb700 1 -- 172.21.15.187:0/2391891114 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fad8c017070 con 0x7fad940b52f0 2020-11-20T12:13:32.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.619+0000 7fad85ffb700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad6c056d20 0x7fad6c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:32.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.619+0000 7fad85ffb700 1 -- 172.21.15.187:0/2391891114 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fad8c0720e0 con 0x7fad940b52f0 2020-11-20T12:13:32.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.619+0000 7fad99b9c700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad6c056d20 0x7fad6c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:32.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.620+0000 7fad99b9c700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad6c056d20 0x7fad6c0591e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fad9000a570 tx=0x7fad90008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:32.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.623+0000 7fad85ffb700 1 -- 172.21.15.187:0/2391891114 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fad8c0473b0 con 0x7fad940b52f0 2020-11-20T12:13:32.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:32.811+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fad94044f60 con 0x7fad940b52f0 2020-11-20T12:13:33.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.326+0000 7fad85ffb700 1 -- 172.21.15.187:0/2391891114 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) v1 ==== 181+0+0 (secure 0 0 0) 0x7fad8c015030 con 0x7fad940b52f0 2020-11-20T12:13:33.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.329+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad6c056d20 msgr2=0x7fad6c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:33.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.329+0000 7fad9be00700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad6c056d20 0x7fad6c0591e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fad9000a570 tx=0x7fad90008040).stop 2020-11-20T12:13:33.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.329+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 msgr2=0x7fad9413a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:33.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.329+0000 7fad9be00700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad9413a390 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7fad8c0092d0 tx=0x7fad8c016000).stop 2020-11-20T12:13:33.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.330+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 shutdown_connections 2020-11-20T12:13:33.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.330+0000 7fad9be00700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad6c056d20 0x7fad6c0591e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:33.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.330+0000 7fad9be00700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad940b52f0 0x7fad9413a390 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:33.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.330+0000 7fad9be00700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad94065e30 0x7fad940b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:33.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.330+0000 7fad9be00700 1 --2- 172.21.15.187:0/2391891114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad94066210 0x7fad94139ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:33.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.330+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 >> 172.21.15.187:0/2391891114 conn(0x7fad940bacf0 msgr2=0x7fad94065390 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:33.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.330+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 shutdown_connections 2020-11-20T12:13:33.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.330+0000 7fad9be00700 1 -- 172.21.15.187:0/2391891114 wait complete. 2020-11-20T12:13:33.337 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:13:33.360 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:13:33.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.607+0000 7f140e598700 1 Processor -- start 2020-11-20T12:13:33.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.607+0000 7f140e598700 1 -- start start 2020-11-20T12:13:33.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.608+0000 7f140e598700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080812e0 0x7f14080be7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.608+0000 7f140e598700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1408148910 0x7f1408148da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.608+0000 7f140e598700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14080bee40 con 0x7f1408080f00 2020-11-20T12:13:33.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.608+0000 7f140e598700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140814bee0 con 0x7f1408148910 2020-11-20T12:13:33.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.608+0000 7f140e598700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140814c180 con 0x7f14080812e0 2020-11-20T12:13:33.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.608+0000 7f1407fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1408080f00 0x7f140814c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44454/0 (socket says 172.21.15.187:44454) 2020-11-20T12:13:33.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.608+0000 7f1407fff700 1 -- 172.21.15.187:0/4109592146 learned_addr learned my addr 172.21.15.187:0/4109592146 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:33.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f14077fe700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080812e0 0x7f14080be7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:33.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f140cb35700 1 --2- 172.21.15.187:0/4109592146 >> v2:172.21.15.5:3300/0 conn(0x7f1408148910 0x7f1408148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:33.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009235186 0 0) 0x7f14080bee40 con 0x7f1408080f00 2020-11-20T12:13:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e8003240 con 0x7f1408080f00 2020-11-20T12:13:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2780833131 0 0) 0x7f13e8003240 con 0x7f1408080f00 2020-11-20T12:13:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080812e0 msgr2=0x7f14080be7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f1406ffd700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080812e0 0x7f14080be7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 >> v2:172.21.15.5:3300/0 conn(0x7f1408148910 msgr2=0x7f1408148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:33.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.609+0000 7f1406ffd700 1 --2- 172.21.15.187:0/4109592146 >> v2:172.21.15.5:3300/0 conn(0x7f1408148910 0x7f1408148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:33.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140814cd30 con 0x7f1408080f00 2020-11-20T12:13:33.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f13f80014e0 con 0x7f1408080f00 2020-11-20T12:13:33.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 >> v1:172.21.15.187:6789/0 conn(0x7f1408080f00 legacy=0x7f140814c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e80037b0 0x7f13e8003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e80044e0 0x7f13e8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080812e0 0x7f13e8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140814bee0 con 0x7f13e80044e0 2020-11-20T12:13:33.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140814c180 con 0x7f13e80037b0 2020-11-20T12:13:33.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e8005340 con 0x7f14080812e0 2020-11-20T12:13:33.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f140cb35700 1 --2- 172.21.15.187:0/4109592146 >> v2:172.21.15.5:3300/0 conn(0x7f1408148910 0x7f1408148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:33.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1407fff700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e80037b0 0x7f13e8003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:33.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1407fff700 1 -- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080812e0 msgr2=0x7f13e8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:33.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1407fff700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080812e0 0x7f13e8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1407fff700 1 -- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e80044e0 msgr2=0x7f13e8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1407fff700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13e80044e0 0x7f13e8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.610+0000 7f1407fff700 1 -- 172.21.15.187:0/4109592146 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e8003240 con 0x7f13e80037b0 2020-11-20T12:13:33.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.611+0000 7f1407fff700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e80037b0 0x7f13e8003bd0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f13f800c210 tx=0x7f13f8007750).ready entity=mon.1 client_cookie=a3a62fba5878aed1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.611+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f80070d0 con 0x7f13e80037b0 2020-11-20T12:13:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.611+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f8007230 con 0x7f13e80037b0 2020-11-20T12:13:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.611+0000 7f1406ffd700 1 -- 172.21.15.187:0/4109592146 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f80055f0 con 0x7f13e80037b0 2020-11-20T12:13:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.611+0000 7f140e598700 1 -- 172.21.15.187:0/4109592146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e80037b0 msgr2=0x7f13e8003bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:33.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.612+0000 7f140e598700 1 --2- 172.21.15.187:0/4109592146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e80037b0 0x7f13e8003bd0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f13f800c210 tx=0x7f13f8007750).stop 2020-11-20T12:13:33.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.612+0000 7f140cb35700 1 -- 172.21.15.187:0/4109592146 reap_dead start 2020-11-20T12:13:33.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.612+0000 7f140e598700 1 -- 172.21.15.187:0/4109592146 shutdown_connections 2020-11-20T12:13:33.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.612+0000 7f140e598700 1 -- 172.21.15.187:0/4109592146 >> 172.21.15.187:0/4109592146 conn(0x7f14080bf100 msgr2=0x7f14080bf510 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:33.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.612+0000 7f140e598700 1 -- 172.21.15.187:0/4109592146 shutdown_connections 2020-11-20T12:13:33.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.612+0000 7f140e598700 1 -- 172.21.15.187:0/4109592146 wait complete. 2020-11-20T12:13:33.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.613+0000 7f140e598700 1 Processor -- start 2020-11-20T12:13:33.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.613+0000 7f140e598700 1 -- start start 2020-11-20T12:13:33.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.614+0000 7f140e598700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080a4440 0x7f14080a4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.614+0000 7f140e598700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14080a2ef0 0x7f140815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.614+0000 7f140e598700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14080a4d10 0x7f140815c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.614+0000 7f140e598700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14080a3390 con 0x7f14080a4d10 2020-11-20T12:13:33.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.614+0000 7f140e598700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14080a3630 con 0x7f14080a2ef0 2020-11-20T12:13:33.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.614+0000 7f140e598700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14080a38d0 con 0x7f14080a4440 2020-11-20T12:13:33.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.614+0000 7f14077fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14080a2ef0 0x7f140815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:33.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f14077fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14080a2ef0 0x7f140815bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47862/0 (socket says 172.21.15.187:47862) 2020-11-20T12:13:33.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f14077fe700 1 -- 172.21.15.187:0/2320598942 learned_addr learned my addr 172.21.15.187:0/2320598942 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:33.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f140cb35700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14080a4d10 0x7f140815c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:33.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f14077fe700 1 -- 172.21.15.187:0/2320598942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080a4440 msgr2=0x7f14080a4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:33.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f14077fe700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080a4440 0x7f14080a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:33.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f14077fe700 1 -- 172.21.15.187:0/2320598942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14080a4d10 msgr2=0x7f140815c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:33.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f14077fe700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14080a4d10 0x7f140815c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:33.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f14077fe700 1 -- 172.21.15.187:0/2320598942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14080a3d70 con 0x7f14080a2ef0 2020-11-20T12:13:33.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f14077fe700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14080a2ef0 0x7f140815bc40 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f13f00067d0 tx=0x7f13f0006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:33.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.615+0000 7f13fffff700 1 -- 172.21.15.187:0/2320598942 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0016070 con 0x7f14080a2ef0 2020-11-20T12:13:33.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.616+0000 7f13fffff700 1 -- 172.21.15.187:0/2320598942 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f000c7a0 con 0x7f14080a2ef0 2020-11-20T12:13:33.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.616+0000 7f13fffff700 1 -- 172.21.15.187:0/2320598942 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f000cb40 con 0x7f14080a2ef0 2020-11-20T12:13:33.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.616+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14080a4020 con 0x7f14080a2ef0 2020-11-20T12:13:33.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.616+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140815c7e0 con 0x7f14080a2ef0 2020-11-20T12:13:33.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.617+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14080a3a90 con 0x7f14080a2ef0 2020-11-20T12:13:33.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.617+0000 7f13fffff700 1 -- 172.21.15.187:0/2320598942 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f13f001b410 con 0x7f14080a2ef0 2020-11-20T12:13:33.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.618+0000 7f13fffff700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13e4056d20 0x7f13e40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:33.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.618+0000 7f13fffff700 1 -- 172.21.15.187:0/2320598942 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f13f006fec0 con 0x7f14080a2ef0 2020-11-20T12:13:33.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.619+0000 7f1407fff700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13e4056d20 0x7f13e40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:33.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.619+0000 7f1407fff700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13e4056d20 0x7f13e40591e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f140805f730 tx=0x7f13f8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:33.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.622+0000 7f13fffff700 1 -- 172.21.15.187:0/2320598942 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f13f0045300 con 0x7f14080a2ef0 2020-11-20T12:13:33.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:33.808+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1408044f60 con 0x7f14080a2ef0 2020-11-20T12:13:34.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.340+0000 7f13fffff700 1 -- 172.21.15.187:0/2320598942 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7f13f0048920 con 0x7f14080a2ef0 2020-11-20T12:13:34.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.343+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13e4056d20 msgr2=0x7f13e40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:34.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.343+0000 7f140e598700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13e4056d20 0x7f13e40591e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f140805f730 tx=0x7f13f8015000).stop 2020-11-20T12:13:34.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.343+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14080a2ef0 msgr2=0x7f140815bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:34.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.343+0000 7f140e598700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14080a2ef0 0x7f140815bc40 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f13f00067d0 tx=0x7f13f0006df0).stop 2020-11-20T12:13:34.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.344+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 shutdown_connections 2020-11-20T12:13:34.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.344+0000 7f140e598700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f13e4056d20 0x7f13e40591e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:34.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.344+0000 7f140e598700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14080a4d10 0x7f140815c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:34.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.344+0000 7f140e598700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f14080a4440 0x7f14080a4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:34.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.344+0000 7f140e598700 1 --2- 172.21.15.187:0/2320598942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14080a2ef0 0x7f140815bc40 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:34.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.344+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 >> 172.21.15.187:0/2320598942 conn(0x7f14080bf100 msgr2=0x7f14080b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:34.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.344+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 shutdown_connections 2020-11-20T12:13:34.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.344+0000 7f140e598700 1 -- 172.21.15.187:0/2320598942 wait complete. 2020-11-20T12:13:34.351 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:13:34.373 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:13:34.374 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:13:34.374 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:13:34.375 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:13:34.376 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:13:34.376 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:13:34.377 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:13:34.377 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:13:34.393 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:13:34.394 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:13:34.405 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:13:34.405 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:13:34.430 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:13:34.431 DEBUG:teuthology.parallel:result is None 2020-11-20T12:13:34.443 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:13:34.444 DEBUG:teuthology.parallel:result is None 2020-11-20T12:13:34.445 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:13:34.606 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:13:34.634 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:13:34.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.664+0000 7f4d49b8a700 1 Processor -- start 2020-11-20T12:13:34.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.665+0000 7f4d49b8a700 1 -- start start 2020-11-20T12:13:34.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.666+0000 7f4d49b8a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d440b1760 0x7f4d44065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:34.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.666+0000 7f4d49b8a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4d44065680 0x7f4d4414c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:34.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.666+0000 7f4d49b8a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d44150390 con 0x7f4d440b1380 2020-11-20T12:13:34.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.666+0000 7f4d49b8a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d441511e0 con 0x7f4d440b1760 2020-11-20T12:13:34.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.666+0000 7f4d49b8a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4414d760 con 0x7f4d44065680 2020-11-20T12:13:34.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.666+0000 7f4d49389700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4d44065680 0x7f4d4414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:34.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.666+0000 7f4d49389700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4d44065680 0x7f4d4414c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47890/0 (socket says 172.21.15.187:47890) 2020-11-20T12:13:34.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.666+0000 7f4d49389700 1 -- 172.21.15.187:0/3578777960 learned_addr learned my addr 172.21.15.187:0/3578777960 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:34.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d43fff700 1 --2- 172.21.15.187:0/3578777960 >> v2:172.21.15.5:3300/0 conn(0x7f4d440b1760 0x7f4d44065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:34.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d49389700 1 -- 172.21.15.187:0/3578777960 >> v2:172.21.15.5:3300/0 conn(0x7f4d440b1760 msgr2=0x7f4d44065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:34.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d49389700 1 --2- 172.21.15.187:0/3578777960 >> v2:172.21.15.5:3300/0 conn(0x7f4d440b1760 0x7f4d44065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:34.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d49389700 1 -- 172.21.15.187:0/3578777960 >> v1:172.21.15.5:6789/0 conn(0x7f4d440b1380 legacy=0x7f4d44150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:34.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d49389700 1 -- 172.21.15.187:0/3578777960 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44150630 con 0x7f4d44065680 2020-11-20T12:13:34.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d43fff700 1 --2- 172.21.15.187:0/3578777960 >> v2:172.21.15.5:3300/0 conn(0x7f4d440b1760 0x7f4d44065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:34.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d49389700 1 --2- 172.21.15.187:0/3578777960 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d44065680 0x7f4d4414c260 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f4d44061740 tx=0x7f4d3c009290).ready entity=mon.1 client_cookie=dca828fb5597fe07 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:34.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d437fe700 1 -- 172.21.15.187:0/3578777960 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00bb80 con 0x7f4d44065680 2020-11-20T12:13:34.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d437fe700 1 -- 172.21.15.187:0/3578777960 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c00bce0 con 0x7f4d44065680 2020-11-20T12:13:34.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.667+0000 7f4d437fe700 1 -- 172.21.15.187:0/3578777960 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c011610 con 0x7f4d44065680 2020-11-20T12:13:34.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.668+0000 7f4d49b8a700 1 -- 172.21.15.187:0/3578777960 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d44065680 msgr2=0x7f4d4414c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:34.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.668+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/3578777960 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d44065680 0x7f4d4414c260 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f4d44061740 tx=0x7f4d3c009290).stop 2020-11-20T12:13:34.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.668+0000 7f4d49b8a700 1 -- 172.21.15.187:0/3578777960 shutdown_connections 2020-11-20T12:13:34.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.668+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/3578777960 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d44065680 0x7f4d4414c260 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:34.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.668+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/3578777960 >> v2:172.21.15.5:3300/0 conn(0x7f4d440b1760 0x7f4d44065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:34.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.668+0000 7f4d49b8a700 1 -- 172.21.15.187:0/3578777960 >> 172.21.15.187:0/3578777960 conn(0x7f4d440bab30 msgr2=0x7f4d440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:34.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.668+0000 7f4d49b8a700 1 -- 172.21.15.187:0/3578777960 shutdown_connections 2020-11-20T12:13:34.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.668+0000 7f4d49b8a700 1 -- 172.21.15.187:0/3578777960 wait complete. 2020-11-20T12:13:34.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.670+0000 7f4d49b8a700 1 Processor -- start 2020-11-20T12:13:34.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.670+0000 7f4d49b8a700 1 -- start start 2020-11-20T12:13:34.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d49b8a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d440b1380 0x7f4d4415ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:34.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d49b8a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d440b1760 0x7f4d4414a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:34.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d49b8a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d440b50e0 0x7f4d4414a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:34.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d49b8a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4415cf20 con 0x7f4d440b50e0 2020-11-20T12:13:34.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d49b8a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b7aa0 con 0x7f4d440b1380 2020-11-20T12:13:34.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d49b8a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b7d40 con 0x7f4d440b1760 2020-11-20T12:13:34.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d48b88700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d440b1380 0x7f4d4415ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:34.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d48b88700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d440b1380 0x7f4d4415ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47892/0 (socket says 172.21.15.187:47892) 2020-11-20T12:13:34.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d48b88700 1 -- 172.21.15.187:0/2713276387 learned_addr learned my addr 172.21.15.187:0/2713276387 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:34.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d49389700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d440b50e0 0x7f4d4414a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:34.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.671+0000 7f4d43fff700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d440b1760 0x7f4d4414a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:34.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d48b88700 1 -- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d440b1760 msgr2=0x7f4d4414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:34.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d48b88700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d440b1760 0x7f4d4414a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:34.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d48b88700 1 -- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d440b50e0 msgr2=0x7f4d4414a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:34.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d48b88700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d440b50e0 0x7f4d4414a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:34.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d48b88700 1 -- 172.21.15.187:0/2713276387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4409a800 con 0x7f4d440b1380 2020-11-20T12:13:34.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d43fff700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d440b1760 0x7f4d4414a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:34.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d49389700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d440b50e0 0x7f4d4414a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:34.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d48b88700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d440b1380 0x7f4d4415ca30 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7f4d3800c730 tx=0x7f4d38007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:34.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d40ff9700 1 -- 172.21.15.187:0/2713276387 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800e560 con 0x7f4d440b1380 2020-11-20T12:13:34.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d40ff9700 1 -- 172.21.15.187:0/2713276387 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3800e6c0 con 0x7f4d440b1380 2020-11-20T12:13:34.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d40ff9700 1 -- 172.21.15.187:0/2713276387 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800ea80 con 0x7f4d440b1380 2020-11-20T12:13:34.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4409aa50 con 0x7f4d440b1380 2020-11-20T12:13:34.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.672+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4409aef0 con 0x7f4d440b1380 2020-11-20T12:13:34.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.674+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d44061a20 con 0x7f4d440b1380 2020-11-20T12:13:34.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.674+0000 7f4d40ff9700 1 -- 172.21.15.187:0/2713276387 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4d38010070 con 0x7f4d440b1380 2020-11-20T12:13:34.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.675+0000 7f4d40ff9700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d1c056d20 0x7f4d1c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:34.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.675+0000 7f4d40ff9700 1 -- 172.21.15.187:0/2713276387 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f4d3806f840 con 0x7f4d440b1380 2020-11-20T12:13:34.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.675+0000 7f4d43fff700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d1c056d20 0x7f4d1c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:34.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.676+0000 7f4d43fff700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d1c056d20 0x7f4d1c0591e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f4d3000b7d0 tx=0x7f4d30008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:34.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.679+0000 7f4d40ff9700 1 -- 172.21.15.187:0/2713276387 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4d38048e60 con 0x7f4d440b1380 2020-11-20T12:13:34.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:34.885+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4d440446d0 con 0x7f4d440b1380 2020-11-20T12:13:35.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.342+0000 7f4d40ff9700 1 -- 172.21.15.187:0/2713276387 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v85) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4d38044f70 con 0x7f4d440b1380 2020-11-20T12:13:35.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.346+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d1c056d20 msgr2=0x7f4d1c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.346+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d1c056d20 0x7f4d1c0591e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f4d3000b7d0 tx=0x7f4d30008040).stop 2020-11-20T12:13:35.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.346+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d440b1380 msgr2=0x7f4d4415ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.346+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d440b1380 0x7f4d4415ca30 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7f4d3800c730 tx=0x7f4d38007c30).stop 2020-11-20T12:13:35.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.347+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 shutdown_connections 2020-11-20T12:13:35.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.347+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d1c056d20 0x7f4d1c0591e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.347+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d440b50e0 0x7f4d4414a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.347+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d440b1380 0x7f4d4415ca30 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.347+0000 7f4d49b8a700 1 --2- 172.21.15.187:0/2713276387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d440b1760 0x7f4d4414a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.347+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 >> 172.21.15.187:0/2713276387 conn(0x7f4d440bab30 msgr2=0x7f4d440b0810 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:35.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.347+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 shutdown_connections 2020-11-20T12:13:35.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.347+0000 7f4d49b8a700 1 -- 172.21.15.187:0/2713276387 wait complete. 2020-11-20T12:13:35.353 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:13:35.382 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:13:35.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.624+0000 7efd70a85700 1 Processor -- start 2020-11-20T12:13:35.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.624+0000 7efd70a85700 1 -- start start 2020-11-20T12:13:35.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.626+0000 7efd70a85700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efd6c0b1380 0x7efd6c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:35.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.626+0000 7efd70a85700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6c0658a0 con 0x7efd6c0b1760 2020-11-20T12:13:35.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.626+0000 7efd70a85700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6c150390 con 0x7efd6c14c260 2020-11-20T12:13:35.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.626+0000 7efd70a85700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6c150630 con 0x7efd6c0b1380 2020-11-20T12:13:35.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.626+0000 7efd6bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7efd6c14c260 0x7efd6c14c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44510/0 (socket says 172.21.15.187:44510) 2020-11-20T12:13:35.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.626+0000 7efd6bfff700 1 -- 172.21.15.187:0/2594630741 learned_addr learned my addr 172.21.15.187:0/2594630741 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:35.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.626+0000 7efd6b7fe700 1 --2- 172.21.15.187:0/2594630741 >> v2:172.21.15.187:3300/0 conn(0x7efd6c0b1380 0x7efd6c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:35.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.627+0000 7efd6affd700 1 -- 172.21.15.187:0/2594630741 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505251011 0 0) 0x7efd6c150390 con 0x7efd6c14c260 2020-11-20T12:13:35.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.627+0000 7efd6affd700 1 -- 172.21.15.187:0/2594630741 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c003240 con 0x7efd6c14c260 2020-11-20T12:13:35.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.627+0000 7efd6b7fe700 1 -- 172.21.15.187:0/2594630741 >> v1:172.21.15.187:6789/0 conn(0x7efd6c14c260 legacy=0x7efd6c14c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.627+0000 7efd6affd700 1 -- 172.21.15.187:0/2594630741 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3647964496 0 0) 0x7efd4c003240 con 0x7efd6c14c260 2020-11-20T12:13:35.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.627+0000 7efd6b7fe700 1 -- 172.21.15.187:0/2594630741 >> v1:172.21.15.5:6789/0 conn(0x7efd6c0b1760 legacy=0x7efd6c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.627+0000 7efd6b7fe700 1 -- 172.21.15.187:0/2594630741 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c068510 con 0x7efd6c0b1380 2020-11-20T12:13:35.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd6b7fe700 1 --2- 172.21.15.187:0/2594630741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c0b1380 0x7efd6c150ad0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7efd5c00b920 tx=0x7efd5c006df0).ready entity=mon.1 client_cookie=1904af3521036148 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:35.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd6affd700 1 -- 172.21.15.187:0/2594630741 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64001860 con 0x7efd6c0b1380 2020-11-20T12:13:35.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd6affd700 1 -- 172.21.15.187:0/2594630741 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c011400 con 0x7efd6c0b1380 2020-11-20T12:13:35.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd6affd700 1 -- 172.21.15.187:0/2594630741 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c011780 con 0x7efd6c0b1380 2020-11-20T12:13:35.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd70a85700 1 -- 172.21.15.187:0/2594630741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c0b1380 msgr2=0x7efd6c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd70a85700 1 --2- 172.21.15.187:0/2594630741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c0b1380 0x7efd6c150ad0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7efd5c00b920 tx=0x7efd5c006df0).stop 2020-11-20T12:13:35.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd70a85700 1 -- 172.21.15.187:0/2594630741 shutdown_connections 2020-11-20T12:13:35.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd70a85700 1 --2- 172.21.15.187:0/2594630741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c0b1380 0x7efd6c150ad0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.628+0000 7efd70a85700 1 -- 172.21.15.187:0/2594630741 >> 172.21.15.187:0/2594630741 conn(0x7efd6c0bab30 msgr2=0x7efd6c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:35.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.629+0000 7efd70a85700 1 -- 172.21.15.187:0/2594630741 shutdown_connections 2020-11-20T12:13:35.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.629+0000 7efd70a85700 1 -- 172.21.15.187:0/2594630741 wait complete. 2020-11-20T12:13:35.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.630+0000 7efd70a85700 1 Processor -- start 2020-11-20T12:13:35.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.630+0000 7efd70a85700 1 -- start start 2020-11-20T12:13:35.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.631+0000 7efd70a85700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd6c0b1380 0x7efd6c09f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:35.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.631+0000 7efd70a85700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd6c0b1760 0x7efd6c09dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:35.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.631+0000 7efd70a85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c14c260 0x7efd6c09dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:35.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.631+0000 7efd70a85700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c0b5ca0 con 0x7efd6c0b1380 2020-11-20T12:13:35.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.631+0000 7efd70a85700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c0b5f40 con 0x7efd6c14c260 2020-11-20T12:13:35.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.631+0000 7efd70a85700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c0b4380 con 0x7efd6c0b1760 2020-11-20T12:13:35.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c14c260 0x7efd6c09dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:35.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c14c260 0x7efd6c09dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47910/0 (socket says 172.21.15.187:47910) 2020-11-20T12:13:35.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6bfff700 1 -- 172.21.15.187:0/3700078913 learned_addr learned my addr 172.21.15.187:0/3700078913 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd62ffd700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd6c0b1760 0x7efd6c09dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6b7fe700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd6c0b1380 0x7efd6c09f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6bfff700 1 -- 172.21.15.187:0/3700078913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd6c0b1760 msgr2=0x7efd6c09dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6bfff700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd6c0b1760 0x7efd6c09dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6bfff700 1 -- 172.21.15.187:0/3700078913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd6c0b1380 msgr2=0x7efd6c09f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6bfff700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd6c0b1380 0x7efd6c09f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.632+0000 7efd6bfff700 1 -- 172.21.15.187:0/3700078913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c0b4540 con 0x7efd6c14c260 2020-11-20T12:13:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.633+0000 7efd6bfff700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c14c260 0x7efd6c09dfd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7efd6400ce60 tx=0x7efd64001dc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.633+0000 7efd63fff700 1 -- 172.21.15.187:0/3700078913 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6400aaa0 con 0x7efd6c14c260 2020-11-20T12:13:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.633+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6c0b3f40 con 0x7efd6c14c260 2020-11-20T12:13:35.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.633+0000 7efd63fff700 1 -- 172.21.15.187:0/3700078913 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6400ac00 con 0x7efd6c14c260 2020-11-20T12:13:35.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.633+0000 7efd63fff700 1 -- 172.21.15.187:0/3700078913 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64007170 con 0x7efd6c14c260 2020-11-20T12:13:35.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.633+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6c09e700 con 0x7efd6c14c260 2020-11-20T12:13:35.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.634+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6c09e950 con 0x7efd6c14c260 2020-11-20T12:13:35.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.635+0000 7efd63fff700 1 -- 172.21.15.187:0/3700078913 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efd6401b060 con 0x7efd6c14c260 2020-11-20T12:13:35.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.636+0000 7efd63fff700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efd48056d20 0x7efd480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:35.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.636+0000 7efd63fff700 1 -- 172.21.15.187:0/3700078913 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efd6406fb30 con 0x7efd6c14c260 2020-11-20T12:13:35.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.636+0000 7efd6b7fe700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efd48056d20 0x7efd480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:35.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.636+0000 7efd6b7fe700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efd48056d20 0x7efd480591e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7efd5c006e30 tx=0x7efd5c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:35.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.639+0000 7efd63fff700 1 -- 172.21.15.187:0/3700078913 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efd6c09e950 con 0x7efd6c14c260 2020-11-20T12:13:35.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.887+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efd6c09e950 con 0x7efd6c14c260 2020-11-20T12:13:35.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.888+0000 7efd63fff700 1 -- 172.21.15.187:0/3700078913 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4570 (secure 0 0 0) 0x7efd6c09e950 con 0x7efd6c14c260 2020-11-20T12:13:35.893 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:35.893 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:13:35.893 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:13:35.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.893 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:13:35.894 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:13:35.894 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.894 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:13:35.894 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.895 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.895 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.895 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.896 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:13:35.896 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:13:35.896 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.897 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:13:35.897 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.897 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.897 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.898 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.898 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:13:35.898 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:13:35.899 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.899 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.899 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.899 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.899 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.901 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:13:35.901 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:13:35.901 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.902 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.902 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.902 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.902 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.903 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:13:35.903 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:13:35.903 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.904 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.904 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.904 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.904 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.905 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:13:35.905 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:13:35.905 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.905 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.906 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.906 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.906 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.907 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:13:35.907 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:13:35.907 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.907 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.908 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.908 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.908 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.909 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:13:35.909 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:13:35.909 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.909 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.909 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.910 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.910 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:13:35.911 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:13:35.911 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.911 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.911 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.911 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.912 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.912 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:13:35.912 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:13:35.913 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.913 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.913 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.913 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.914 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.917 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:13:35.917 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:13:35.917 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.917 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:13:35.918 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:13:35.918 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:13:35.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.919 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:13:35.919 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:13:35.919 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.919 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:13:35.920 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:13:35.920 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:13:35.920 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:13:35.920 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.921 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:13:35.921 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:13:35.921 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.922 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:13:35.922 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:13:35.922 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:13:35.922 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:13:35.922 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.923 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:13:35.923 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:13:35.923 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.924 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:13:35.924 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:13:35.924 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:13:35.924 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:13:35.924 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.925 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:13:35.925 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:13:35.926 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.926 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:13:35.926 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:13:35.926 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:13:35.926 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.927 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.927 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:13:35.927 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:13:35.928 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.928 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:13:35.928 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:13:35.928 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.928 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:35.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:35.929 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:13:35.929 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:13:35.930 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:13:35.930 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:13:35.930 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:13:35.930 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:13:35.930 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.931 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:35.931 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:13:35.931 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:13:35.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.890+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efd48056d20 msgr2=0x7efd480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efd48056d20 0x7efd480591e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7efd5c006e30 tx=0x7efd5c015000).stop 2020-11-20T12:13:35.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c14c260 msgr2=0x7efd6c09dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:35.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c14c260 0x7efd6c09dfd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7efd6400ce60 tx=0x7efd64001dc0).stop 2020-11-20T12:13:35.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 shutdown_connections 2020-11-20T12:13:35.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efd48056d20 0x7efd480591e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd6c0b1380 0x7efd6c09f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd6c14c260 0x7efd6c09dfd0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 --2- 172.21.15.187:0/3700078913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd6c0b1760 0x7efd6c09dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:35.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 >> 172.21.15.187:0/3700078913 conn(0x7efd6c0bab30 msgr2=0x7efd6c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:35.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 shutdown_connections 2020-11-20T12:13:35.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:35.891+0000 7efd70a85700 1 -- 172.21.15.187:0/3700078913 wait complete. 2020-11-20T12:13:35.936 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:13:36.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.194+0000 7ff5be546700 1 Processor -- start 2020-11-20T12:13:36.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.194+0000 7ff5be546700 1 -- start start 2020-11-20T12:13:36.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.196+0000 7ff5be546700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff5b80b1380 0x7ff5b8150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.196+0000 7ff5be546700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff5b80b1760 0x7ff5b8065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.196+0000 7ff5be546700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b8150390 con 0x7ff5b8065680 2020-11-20T12:13:36.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.196+0000 7ff5be546700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b8150630 con 0x7ff5b80b1380 2020-11-20T12:13:36.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.196+0000 7ff5be546700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b8068510 con 0x7ff5b80b1760 2020-11-20T12:13:36.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.196+0000 7ff5bcd43700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff5b80b1760 0x7ff5b8065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.196+0000 7ff5bcd43700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff5b80b1760 0x7ff5b8065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47918/0 (socket says 172.21.15.187:47918) 2020-11-20T12:13:36.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.196+0000 7ff5bcd43700 1 -- 172.21.15.187:0/394457767 learned_addr learned my addr 172.21.15.187:0/394457767 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:36.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.197+0000 7ff5bd544700 1 --2- 172.21.15.187:0/394457767 >> v2:172.21.15.5:3300/0 conn(0x7ff5b80b1380 0x7ff5b8150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.197+0000 7ff5bcd43700 1 -- 172.21.15.187:0/394457767 >> v2:172.21.15.5:3300/0 conn(0x7ff5b80b1380 msgr2=0x7ff5b8150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:36.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.197+0000 7ff5bcd43700 1 --2- 172.21.15.187:0/394457767 >> v2:172.21.15.5:3300/0 conn(0x7ff5b80b1380 0x7ff5b8150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.197+0000 7ff5bcd43700 1 -- 172.21.15.187:0/394457767 >> v1:172.21.15.187:6789/0 conn(0x7ff5b8065680 legacy=0x7ff5b8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:36.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.197+0000 7ff5bcd43700 1 -- 172.21.15.187:0/394457767 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8150fc0 con 0x7ff5b80b1760 2020-11-20T12:13:36.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.197+0000 7ff5bcd43700 1 --2- 172.21.15.187:0/394457767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b8065190 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7ff5b8063a20 tx=0x7ff5a8006eb0).ready entity=mon.1 client_cookie=b760e64817dc321a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:36.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.197+0000 7ff5a7fff700 1 -- 172.21.15.187:0/394457767 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8009890 con 0x7ff5b80b1760 2020-11-20T12:13:36.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5a7fff700 1 -- 172.21.15.187:0/394457767 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a80099f0 con 0x7ff5b80b1760 2020-11-20T12:13:36.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5a7fff700 1 -- 172.21.15.187:0/394457767 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8009d70 con 0x7ff5b80b1760 2020-11-20T12:13:36.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5be546700 1 -- 172.21.15.187:0/394457767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 msgr2=0x7ff5b8065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5be546700 1 --2- 172.21.15.187:0/394457767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b8065190 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7ff5b8063a20 tx=0x7ff5a8006eb0).stop 2020-11-20T12:13:36.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5be546700 1 -- 172.21.15.187:0/394457767 shutdown_connections 2020-11-20T12:13:36.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5be546700 1 --2- 172.21.15.187:0/394457767 >> v2:172.21.15.5:3300/0 conn(0x7ff5b80b1380 0x7ff5b8150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5be546700 1 --2- 172.21.15.187:0/394457767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b8065190 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5be546700 1 -- 172.21.15.187:0/394457767 >> 172.21.15.187:0/394457767 conn(0x7ff5b80bab30 msgr2=0x7ff5b80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:36.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5be546700 1 -- 172.21.15.187:0/394457767 shutdown_connections 2020-11-20T12:13:36.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.198+0000 7ff5be546700 1 -- 172.21.15.187:0/394457767 wait complete. 2020-11-20T12:13:36.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.200+0000 7ff5be546700 1 Processor -- start 2020-11-20T12:13:36.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.200+0000 7ff5be546700 1 -- start start 2020-11-20T12:13:36.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5be546700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b80b1380 0x7ff5b809c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5be546700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b80a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5be546700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b813d620 0x7ff5b813da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5be546700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b813ec60 con 0x7ff5b813d620 2020-11-20T12:13:36.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5be546700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b813e280 con 0x7ff5b80b1760 2020-11-20T12:13:36.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5be546700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b813e520 con 0x7ff5b80b1380 2020-11-20T12:13:36.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5bcd43700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b80a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5bcd43700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b80a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47922/0 (socket says 172.21.15.187:47922) 2020-11-20T12:13:36.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.201+0000 7ff5bcd43700 1 -- 172.21.15.187:0/699908841 learned_addr learned my addr 172.21.15.187:0/699908841 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:36.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5bcd43700 1 -- 172.21.15.187:0/699908841 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b80b1380 msgr2=0x7ff5b809c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5bcd43700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b80b1380 0x7ff5b809c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:36.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5bcd43700 1 -- 172.21.15.187:0/699908841 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b813d620 msgr2=0x7ff5b813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5bcd43700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b813d620 0x7ff5b813da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:36.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5bcd43700 1 -- 172.21.15.187:0/699908841 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b813e7c0 con 0x7ff5b80b1760 2020-11-20T12:13:36.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5bcd43700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b80a0430 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7ff5a8006fd0 tx=0x7ff5a8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:36.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5a57fa700 1 -- 172.21.15.187:0/699908841 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8011e50 con 0x7ff5b80b1760 2020-11-20T12:13:36.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5a57fa700 1 -- 172.21.15.187:0/699908841 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a801e460 con 0x7ff5b80b1760 2020-11-20T12:13:36.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b813df50 con 0x7ff5b80b1760 2020-11-20T12:13:36.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5a57fa700 1 -- 172.21.15.187:0/699908841 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a801e7e0 con 0x7ff5b80b1760 2020-11-20T12:13:36.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.202+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b80a0920 con 0x7ff5b80b1760 2020-11-20T12:13:36.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.203+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b813e950 con 0x7ff5b80b1760 2020-11-20T12:13:36.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.204+0000 7ff5a57fa700 1 -- 172.21.15.187:0/699908841 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff5a8025030 con 0x7ff5b80b1760 2020-11-20T12:13:36.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.205+0000 7ff5a57fa700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.205+0000 7ff5a57fa700 1 -- 172.21.15.187:0/699908841 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff5a8071370 con 0x7ff5b80b1760 2020-11-20T12:13:36.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.205+0000 7ff5bd544700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.205+0000 7ff5bd544700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7ff5b400b740 tx=0x7ff5b4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:36.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.208+0000 7ff5a57fa700 1 -- 172.21.15.187:0/699908841 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff5a8046ba0 con 0x7ff5b80b1760 2020-11-20T12:13:36.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.383+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b8061a20 con 0x7ff5b80b1760 2020-11-20T12:13:36.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.385+0000 7ff5a57fa700 1 -- 172.21.15.187:0/699908841 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13628 (secure 0 0 0) 0x7ff5a8030550 con 0x7ff5b80b1760 2020-11-20T12:13:36.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 msgr2=0x7ff590059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7ff5b400b740 tx=0x7ff5b4008040).stop 2020-11-20T12:13:36.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 msgr2=0x7ff5b80a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b80a0430 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7ff5a8006fd0 tx=0x7ff5a8015000).stop 2020-11-20T12:13:36.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 shutdown_connections 2020-11-20T12:13:36.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff590056d50 0x7ff590059210 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5b813d620 0x7ff5b813da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:36.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5b80b1380 0x7ff5b809c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:36.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 --2- 172.21.15.187:0/699908841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5b80b1760 0x7ff5b80a0430 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 >> 172.21.15.187:0/699908841 conn(0x7ff5b80bab30 msgr2=0x7ff5b814f300 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:36.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 shutdown_connections 2020-11-20T12:13:36.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.388+0000 7ff5be546700 1 -- 172.21.15.187:0/699908841 wait complete. 2020-11-20T12:13:36.530 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:36.530 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":85,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:13:35.332534+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:13:36.531 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:13:36.532 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:13:36.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.780+0000 7f6e46e81700 1 Processor -- start 2020-11-20T12:13:36.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.780+0000 7f6e46e81700 1 -- start start 2020-11-20T12:13:36.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.781+0000 7f6e46e81700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40065190 0x7f6e40065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.782+0000 7f6e46e81700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e40065f90 0x7f6e40143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.782+0000 7f6e46e81700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e400b13e0 con 0x7f6e40143c70 2020-11-20T12:13:36.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.782+0000 7f6e46e81700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4014ab30 con 0x7f6e40065f90 2020-11-20T12:13:36.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.782+0000 7f6e46e81700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40062c10 con 0x7f6e40065190 2020-11-20T12:13:36.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.782+0000 7f6e46680700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6e40143c70 0x7f6e4009ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44534/0 (socket says 172.21.15.187:44534) 2020-11-20T12:13:36.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.782+0000 7f6e46680700 1 -- 172.21.15.187:0/2464118190 learned_addr learned my addr 172.21.15.187:0/2464118190 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:36.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.782+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433398995 0 0) 0x7f6e400b13e0 con 0x7f6e40143c70 2020-11-20T12:13:36.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e28003240 con 0x7f6e40143c70 2020-11-20T12:13:36.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e45e7f700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40065190 0x7f6e40065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e4567e700 1 --2- 172.21.15.187:0/2464118190 >> v2:172.21.15.5:3300/0 conn(0x7f6e40065f90 0x7f6e40143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4155102910 0 0) 0x7f6e28003240 con 0x7f6e40143c70 2020-11-20T12:13:36.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40065190 msgr2=0x7f6e40065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:36.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40065190 0x7f6e40065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 >> v2:172.21.15.5:3300/0 conn(0x7f6e40065f90 msgr2=0x7f6e40143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:36.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 --2- 172.21.15.187:0/2464118190 >> v2:172.21.15.5:3300/0 conn(0x7f6e40065f90 0x7f6e40143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e400b1640 con 0x7f6e40143c70 2020-11-20T12:13:36.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6e380014e0 con 0x7f6e40143c70 2020-11-20T12:13:36.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e45e7f700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40065190 0x7f6e40065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:36.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 >> v1:172.21.15.187:6789/0 conn(0x7f6e40143c70 legacy=0x7f6e4009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e280037b0 0x7f6e28003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e280044e0 0x7f6e28004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40065190 0x7f6e28004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4014ab30 con 0x7f6e280037b0 2020-11-20T12:13:36.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40062c10 con 0x7f6e280044e0 2020-11-20T12:13:36.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28005340 con 0x7f6e40065190 2020-11-20T12:13:36.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.783+0000 7f6e4567e700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e280044e0 0x7f6e28004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.784+0000 7f6e4567e700 1 -- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40065190 msgr2=0x7f6e28004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:36.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.784+0000 7f6e4567e700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40065190 0x7f6e28004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:36.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.784+0000 7f6e4567e700 1 -- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e280037b0 msgr2=0x7f6e28003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:36.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.784+0000 7f6e45e7f700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e280037b0 0x7f6e28003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.784+0000 7f6e4567e700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e280037b0 0x7f6e28003bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.784+0000 7f6e4567e700 1 -- 172.21.15.187:0/2464118190 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e28003240 con 0x7f6e280044e0 2020-11-20T12:13:36.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.784+0000 7f6e4567e700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e280044e0 0x7f6e28004960 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f6e3000ba50 tx=0x7f6e30008040).ready entity=mon.1 client_cookie=2f18d0bd4ffe02e3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:36.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.784+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3000d9c0 con 0x7f6e280044e0 2020-11-20T12:13:36.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3000db20 con 0x7f6e280044e0 2020-11-20T12:13:36.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e44e7d700 1 -- 172.21.15.187:0/2464118190 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3000dea0 con 0x7f6e280044e0 2020-11-20T12:13:36.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e46e81700 1 -- 172.21.15.187:0/2464118190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e280044e0 msgr2=0x7f6e28004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e46e81700 1 --2- 172.21.15.187:0/2464118190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e280044e0 0x7f6e28004960 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f6e3000ba50 tx=0x7f6e30008040).stop 2020-11-20T12:13:36.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e46680700 1 -- 172.21.15.187:0/2464118190 reap_dead start 2020-11-20T12:13:36.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e46e81700 1 -- 172.21.15.187:0/2464118190 shutdown_connections 2020-11-20T12:13:36.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e46e81700 1 -- 172.21.15.187:0/2464118190 >> 172.21.15.187:0/2464118190 conn(0x7f6e400bab30 msgr2=0x7f6e400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:36.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e46e81700 1 -- 172.21.15.187:0/2464118190 shutdown_connections 2020-11-20T12:13:36.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.785+0000 7f6e46e81700 1 -- 172.21.15.187:0/2464118190 wait complete. 2020-11-20T12:13:36.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.787+0000 7f6e46e81700 1 Processor -- start 2020-11-20T12:13:36.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.787+0000 7f6e46e81700 1 -- start start 2020-11-20T12:13:36.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.788+0000 7f6e46e81700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e40145340 0x7f6e40142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.788+0000 7f6e46e81700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40145720 0x7f6e40143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.788+0000 7f6e46e81700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4009ea00 0x7f6e4009fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.788+0000 7f6e46e81700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4009f490 con 0x7f6e4009ea00 2020-11-20T12:13:36.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.788+0000 7f6e46e81700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4009f730 con 0x7f6e40145340 2020-11-20T12:13:36.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.788+0000 7f6e46e81700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e400b24f0 con 0x7f6e40145720 2020-11-20T12:13:36.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.789+0000 7f6e45e7f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e40145340 0x7f6e40142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.789+0000 7f6e45e7f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e40145340 0x7f6e40142ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47940/0 (socket says 172.21.15.187:47940) 2020-11-20T12:13:36.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.789+0000 7f6e45e7f700 1 -- 172.21.15.187:0/1272072353 learned_addr learned my addr 172.21.15.187:0/1272072353 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:36.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.789+0000 7f6e4567e700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40145720 0x7f6e40143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.789+0000 7f6e46680700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4009ea00 0x7f6e4009fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e45e7f700 1 -- 172.21.15.187:0/1272072353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40145720 msgr2=0x7f6e40143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e45e7f700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40145720 0x7f6e40143190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e45e7f700 1 -- 172.21.15.187:0/1272072353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4009ea00 msgr2=0x7f6e4009fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e45e7f700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4009ea00 0x7f6e4009fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e45e7f700 1 -- 172.21.15.187:0/1272072353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4009eea0 con 0x7f6e40145340 2020-11-20T12:13:36.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e45e7f700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e40145340 0x7f6e40142ca0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f6e3c00fbe0 tx=0x7f6e3c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:36.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e2dffb700 1 -- 172.21.15.187:0/1272072353 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c009830 con 0x7f6e40145340 2020-11-20T12:13:36.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e2dffb700 1 -- 172.21.15.187:0/1272072353 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c00c030 con 0x7f6e40145340 2020-11-20T12:13:36.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e2dffb700 1 -- 172.21.15.187:0/1272072353 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c009ba0 con 0x7f6e40145340 2020-11-20T12:13:36.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e40146640 con 0x7f6e40145340 2020-11-20T12:13:36.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.790+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4009dae0 con 0x7f6e40145340 2020-11-20T12:13:36.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.791+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4009f300 con 0x7f6e40145340 2020-11-20T12:13:36.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.792+0000 7f6e2dffb700 1 -- 172.21.15.187:0/1272072353 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6e3c019070 con 0x7f6e40145340 2020-11-20T12:13:36.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.793+0000 7f6e2dffb700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e18056d50 0x7f6e18059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:36.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.793+0000 7f6e2dffb700 1 -- 172.21.15.187:0/1272072353 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f6e3c06fc40 con 0x7f6e40145340 2020-11-20T12:13:36.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.793+0000 7f6e4567e700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e18056d50 0x7f6e18059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:36.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.794+0000 7f6e4567e700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e18056d50 0x7f6e18059210 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f6e400b6390 tx=0x7f6e30008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:36.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.798+0000 7f6e2dffb700 1 -- 172.21.15.187:0/1272072353 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6e3c0453c0 con 0x7f6e40145340 2020-11-20T12:13:36.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.982+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f6e40061a20 con 0x7f6e40145340 2020-11-20T12:13:36.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.983+0000 7f6e2dffb700 1 -- 172.21.15.187:0/1272072353 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v85) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6e3c0489e0 con 0x7f6e40145340 2020-11-20T12:13:36.987 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:13:36.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.985+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e18056d50 msgr2=0x7f6e18059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.985+0000 7f6e46e81700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e18056d50 0x7f6e18059210 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f6e400b6390 tx=0x7f6e30008040).stop 2020-11-20T12:13:36.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.985+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e40145340 msgr2=0x7f6e40142ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:36.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.985+0000 7f6e46e81700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e40145340 0x7f6e40142ca0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f6e3c00fbe0 tx=0x7f6e3c008040).stop 2020-11-20T12:13:36.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.986+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 shutdown_connections 2020-11-20T12:13:36.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.986+0000 7f6e46e81700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6e18056d50 0x7f6e18059210 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.986+0000 7f6e46e81700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4009ea00 0x7f6e4009fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.986+0000 7f6e46e81700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e40145340 0x7f6e40142ca0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.986+0000 7f6e46e81700 1 --2- 172.21.15.187:0/1272072353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e40145720 0x7f6e40143190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:36.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.986+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 >> 172.21.15.187:0/1272072353 conn(0x7f6e400bab30 msgr2=0x7f6e4009cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:36.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.986+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 shutdown_connections 2020-11-20T12:13:36.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:36.986+0000 7f6e46e81700 1 -- 172.21.15.187:0/1272072353 wait complete. 2020-11-20T12:13:37.011 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:13:37.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.248+0000 7f5ed5032700 1 Processor -- start 2020-11-20T12:13:37.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.248+0000 7f5ed5032700 1 -- start start 2020-11-20T12:13:37.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.250+0000 7f5ed5032700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ed0065190 0x7f5ed0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.250+0000 7f5ed5032700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed009ca90 0x7f5ed009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.250+0000 7f5ed5032700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00b13e0 con 0x7f5ed0065f90 2020-11-20T12:13:37.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.250+0000 7f5ed5032700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed014ab30 con 0x7f5ed0065190 2020-11-20T12:13:37.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.250+0000 7f5ed5032700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0062c10 con 0x7f5ed009ca90 2020-11-20T12:13:37.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.250+0000 7f5ecffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ed0065190 0x7f5ed0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.250+0000 7f5ecffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ed0065190 0x7f5ed0065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47948/0 (socket says 172.21.15.187:47948) 2020-11-20T12:13:37.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.250+0000 7f5ecffff700 1 -- 172.21.15.187:0/2823726882 learned_addr learned my addr 172.21.15.187:0/2823726882 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:37.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.251+0000 7f5ed4831700 1 --2- 172.21.15.187:0/2823726882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed009ca90 0x7f5ed009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.251+0000 7f5ecffff700 1 -- 172.21.15.187:0/2823726882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed009ca90 msgr2=0x7f5ed009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:37.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.251+0000 7f5ecffff700 1 --2- 172.21.15.187:0/2823726882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed009ca90 0x7f5ed009cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.251+0000 7f5ecffff700 1 -- 172.21.15.187:0/2823726882 >> v1:172.21.15.5:6789/0 conn(0x7f5ed0065f90 legacy=0x7f5ed0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:37.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.251+0000 7f5ecffff700 1 -- 172.21.15.187:0/2823726882 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed00b1640 con 0x7f5ed0065190 2020-11-20T12:13:37.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.251+0000 7f5ecffff700 1 --2- 172.21.15.187:0/2823726882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065190 0x7f5ed0065570 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f5ec0006fd0 tx=0x7f5ec0006eb0).ready entity=mon.1 client_cookie=3bb3c78ec1169ce0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:37.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ecf7fe700 1 -- 172.21.15.187:0/2823726882 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0009920 con 0x7f5ed0065190 2020-11-20T12:13:37.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ecf7fe700 1 -- 172.21.15.187:0/2823726882 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec0009a80 con 0x7f5ed0065190 2020-11-20T12:13:37.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ecf7fe700 1 -- 172.21.15.187:0/2823726882 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0009e00 con 0x7f5ed0065190 2020-11-20T12:13:37.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ed5032700 1 -- 172.21.15.187:0/2823726882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065190 msgr2=0x7f5ed0065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2823726882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065190 0x7f5ed0065570 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f5ec0006fd0 tx=0x7f5ec0006eb0).stop 2020-11-20T12:13:37.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ed5032700 1 -- 172.21.15.187:0/2823726882 shutdown_connections 2020-11-20T12:13:37.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2823726882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed009ca90 0x7f5ed009cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2823726882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065190 0x7f5ed0065570 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ed5032700 1 -- 172.21.15.187:0/2823726882 >> 172.21.15.187:0/2823726882 conn(0x7f5ed00bab30 msgr2=0x7f5ed00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:37.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.252+0000 7f5ed5032700 1 -- 172.21.15.187:0/2823726882 shutdown_connections 2020-11-20T12:13:37.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.253+0000 7f5ed5032700 1 -- 172.21.15.187:0/2823726882 wait complete. 2020-11-20T12:13:37.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.254+0000 7f5ed5032700 1 Processor -- start 2020-11-20T12:13:37.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.254+0000 7f5ed5032700 1 -- start start 2020-11-20T12:13:37.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.255+0000 7f5ed5032700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed0065190 0x7f5ed00b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.255+0000 7f5ed5032700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065f90 0x7f5ed01456b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.255+0000 7f5ed5032700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ed009ca90 0x7f5ed00a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.255+0000 7f5ed5032700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00b4a90 con 0x7f5ed009ca90 2020-11-20T12:13:37.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.255+0000 7f5ed5032700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0149ed0 con 0x7f5ed0065f90 2020-11-20T12:13:37.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.255+0000 7f5ed5032700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00b47f0 con 0x7f5ed0065190 2020-11-20T12:13:37.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065f90 0x7f5ed01456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065f90 0x7f5ed01456b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47956/0 (socket says 172.21.15.187:47956) 2020-11-20T12:13:37.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 -- 172.21.15.187:0/2571763298 learned_addr learned my addr 172.21.15.187:0/2571763298 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:37.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ecffff700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed0065190 0x7f5ed00b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ed4831700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ed009ca90 0x7f5ed00a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 -- 172.21.15.187:0/2571763298 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed0065190 msgr2=0x7f5ed00b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed0065190 0x7f5ed00b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 -- 172.21.15.187:0/2571763298 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ed009ca90 msgr2=0x7f5ed00a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ed009ca90 0x7f5ed00a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 -- 172.21.15.187:0/2571763298 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed009fcc0 con 0x7f5ed0065f90 2020-11-20T12:13:37.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.256+0000 7f5ec77fe700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065f90 0x7f5ed01456b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f5eb80096b0 tx=0x7f5eb8006de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:37.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.257+0000 7f5eccff9700 1 -- 172.21.15.187:0/2571763298 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb800d560 con 0x7f5ed0065f90 2020-11-20T12:13:37.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.257+0000 7f5eccff9700 1 -- 172.21.15.187:0/2571763298 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb800d6c0 con 0x7f5ed0065f90 2020-11-20T12:13:37.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.257+0000 7f5eccff9700 1 -- 172.21.15.187:0/2571763298 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb800da80 con 0x7f5ed0065f90 2020-11-20T12:13:37.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.257+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed00b2d90 con 0x7f5ed0065f90 2020-11-20T12:13:37.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.257+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed00b4590 con 0x7f5ed0065f90 2020-11-20T12:13:37.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.258+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed0061a20 con 0x7f5ed0065f90 2020-11-20T12:13:37.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.258+0000 7f5eccff9700 1 -- 172.21.15.187:0/2571763298 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5eb800f070 con 0x7f5ed0065f90 2020-11-20T12:13:37.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.259+0000 7f5eccff9700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5eac056d20 0x7f5eac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.259+0000 7f5eccff9700 1 -- 172.21.15.187:0/2571763298 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5eb806f640 con 0x7f5ed0065f90 2020-11-20T12:13:37.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.259+0000 7f5ecffff700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5eac056d20 0x7f5eac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.260+0000 7f5ecffff700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5eac056d20 0x7f5eac0591e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f5ec000b1b0 tx=0x7f5ec0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:37.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.263+0000 7f5eccff9700 1 -- 172.21.15.187:0/2571763298 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5eb8044c50 con 0x7f5ed0065f90 2020-11-20T12:13:37.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.450+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5ed00afc60 con 0x7f5ed0065f90 2020-11-20T12:13:37.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.451+0000 7f5eccff9700 1 -- 172.21.15.187:0/2571763298 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v85) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5eb8048270 con 0x7f5ed0065f90 2020-11-20T12:13:37.455 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:13:37.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5eac056d20 msgr2=0x7f5eac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5eac056d20 0x7f5eac0591e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f5ec000b1b0 tx=0x7f5ec0015000).stop 2020-11-20T12:13:37.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065f90 msgr2=0x7f5ed01456b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065f90 0x7f5ed01456b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f5eb80096b0 tx=0x7f5eb8006de0).stop 2020-11-20T12:13:37.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 shutdown_connections 2020-11-20T12:13:37.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5eac056d20 0x7f5eac0591e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ed009ca90 0x7f5ed00a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ed0065190 0x7f5ed00b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 --2- 172.21.15.187:0/2571763298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ed0065f90 0x7f5ed01456b0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 >> 172.21.15.187:0/2571763298 conn(0x7f5ed00bab30 msgr2=0x7f5ed00aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:37.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 shutdown_connections 2020-11-20T12:13:37.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.454+0000 7f5ed5032700 1 -- 172.21.15.187:0/2571763298 wait complete. 2020-11-20T12:13:37.482 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:13:37.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.732+0000 7ff6016a9700 1 Processor -- start 2020-11-20T12:13:37.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.732+0000 7ff6016a9700 1 -- start start 2020-11-20T12:13:37.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.733+0000 7ff6016a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc1484d0 0x7ff5fc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.733+0000 7ff6016a9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0b1300 con 0x7ff5fc066f70 2020-11-20T12:13:37.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.733+0000 7ff6016a9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0b15a0 con 0x7ff5fc0622c0 2020-11-20T12:13:37.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.733+0000 7ff6016a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc067730 con 0x7ff5fc1484d0 2020-11-20T12:13:37.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.734+0000 7ff5f37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff5fc0622c0 0x7ff5fc0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44570/0 (socket says 172.21.15.187:44570) 2020-11-20T12:13:37.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.734+0000 7ff5f37fe700 1 -- 172.21.15.187:0/989044215 learned_addr learned my addr 172.21.15.187:0/989044215 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:37.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.734+0000 7ff5fbfff700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc1484d0 0x7ff5fc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830795798 0 0) 0x7ff5fc0b15a0 con 0x7ff5fc0622c0 2020-11-20T12:13:37.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5dc003240 con 0x7ff5fc0622c0 2020-11-20T12:13:37.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1587240922 0 0) 0x7ff5dc003240 con 0x7ff5fc0622c0 2020-11-20T12:13:37.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc1484d0 msgr2=0x7ff5fc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:37.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc1484d0 0x7ff5fc061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 >> v1:172.21.15.5:6789/0 conn(0x7ff5fc066f70 legacy=0x7ff5fc0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0b1840 con 0x7ff5fc0622c0 2020-11-20T12:13:37.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff5e4001520 con 0x7ff5fc0622c0 2020-11-20T12:13:37.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 >> v1:172.21.15.187:6789/0 conn(0x7ff5fc0622c0 legacy=0x7ff5fc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5dc003730 0x7ff5dc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5dc0044e0 0x7ff5dc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc1484d0 0x7ff5dc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc067730 con 0x7ff5dc0044e0 2020-11-20T12:13:37.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc005390 con 0x7ff5dc003730 2020-11-20T12:13:37.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.735+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0042f0 con 0x7ff5fc1484d0 2020-11-20T12:13:37.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.736+0000 7ff5fbfff700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5dc003730 0x7ff5dc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.736+0000 7ff5f37fe700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5dc0044e0 0x7ff5dc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.736+0000 7ff600ea8700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc1484d0 0x7ff5dc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.737+0000 7ff5fbfff700 1 -- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc1484d0 msgr2=0x7ff5dc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:37.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.737+0000 7ff5fbfff700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc1484d0 0x7ff5dc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.737+0000 7ff5fbfff700 1 -- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5dc0044e0 msgr2=0x7ff5dc004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:37.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.737+0000 7ff5fbfff700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5dc0044e0 0x7ff5dc004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.737+0000 7ff5fbfff700 1 -- 172.21.15.187:0/989044215 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc003240 con 0x7ff5dc003730 2020-11-20T12:13:37.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.737+0000 7ff5fbfff700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5dc003730 0x7ff5dc003b50 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7ff5ec00bc00 tx=0x7ff5ec006df0).ready entity=mon.1 client_cookie=78b6a6aa5c1730da server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:37.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.737+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec00b310 con 0x7ff5dc003730 2020-11-20T12:13:37.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.738+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec00e9d0 con 0x7ff5dc003730 2020-11-20T12:13:37.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.738+0000 7ff5fb7fe700 1 -- 172.21.15.187:0/989044215 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec00ed50 con 0x7ff5dc003730 2020-11-20T12:13:37.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.738+0000 7ff6016a9700 1 -- 172.21.15.187:0/989044215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5dc003730 msgr2=0x7ff5dc003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.738+0000 7ff6016a9700 1 --2- 172.21.15.187:0/989044215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5dc003730 0x7ff5dc003b50 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7ff5ec00bc00 tx=0x7ff5ec006df0).stop 2020-11-20T12:13:37.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.738+0000 7ff600ea8700 1 -- 172.21.15.187:0/989044215 reap_dead start 2020-11-20T12:13:37.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.738+0000 7ff6016a9700 1 -- 172.21.15.187:0/989044215 shutdown_connections 2020-11-20T12:13:37.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.738+0000 7ff6016a9700 1 -- 172.21.15.187:0/989044215 >> 172.21.15.187:0/989044215 conn(0x7ff5fc0bab30 msgr2=0x7ff5fc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:37.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.738+0000 7ff6016a9700 1 -- 172.21.15.187:0/989044215 shutdown_connections 2020-11-20T12:13:37.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.739+0000 7ff6016a9700 1 -- 172.21.15.187:0/989044215 wait complete. 2020-11-20T12:13:37.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.740+0000 7ff6016a9700 1 Processor -- start 2020-11-20T12:13:37.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.740+0000 7ff6016a9700 1 -- start start 2020-11-20T12:13:37.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff6016a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc0b76e0 0x7ff5fc09b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff6016a9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5fc0b7ac0 0x7ff5fc1444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff6016a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5fc1449a0 0x7ff5fc15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff6016a9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc09bc40 con 0x7ff5fc0b7ac0 2020-11-20T12:13:37.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff6016a9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc14d340 con 0x7ff5fc1449a0 2020-11-20T12:13:37.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff6016a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc14d5e0 con 0x7ff5fc0b76e0 2020-11-20T12:13:37.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff600ea8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5fc1449a0 0x7ff5fc15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff600ea8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5fc1449a0 0x7ff5fc15b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47978/0 (socket says 172.21.15.187:47978) 2020-11-20T12:13:37.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.741+0000 7ff600ea8700 1 -- 172.21.15.187:0/4213753996 learned_addr learned my addr 172.21.15.187:0/4213753996 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:37.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff5f37fe700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5fc0b7ac0 0x7ff5fc1444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff600ea8700 1 -- 172.21.15.187:0/4213753996 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc0b76e0 msgr2=0x7ff5fc09b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff5fbfff700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc0b76e0 0x7ff5fc09b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff600ea8700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc0b76e0 0x7ff5fc09b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff600ea8700 1 -- 172.21.15.187:0/4213753996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5fc0b7ac0 msgr2=0x7ff5fc1444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff600ea8700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5fc0b7ac0 0x7ff5fc1444b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff600ea8700 1 -- 172.21.15.187:0/4213753996 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc14d880 con 0x7ff5fc1449a0 2020-11-20T12:13:37.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff600ea8700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5fc1449a0 0x7ff5fc15b650 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7ff5fc0b85e0 tx=0x7ff5f4007f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:37.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff5f8ff9700 1 -- 172.21.15.187:0/4213753996 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4017070 con 0x7ff5fc1449a0 2020-11-20T12:13:37.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff5f8ff9700 1 -- 172.21.15.187:0/4213753996 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f4002020 con 0x7ff5fc1449a0 2020-11-20T12:13:37.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.742+0000 7ff5f8ff9700 1 -- 172.21.15.187:0/4213753996 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40023e0 con 0x7ff5fc1449a0 2020-11-20T12:13:37.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.743+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc09b520 con 0x7ff5fc1449a0 2020-11-20T12:13:37.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.743+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc14cfc0 con 0x7ff5fc1449a0 2020-11-20T12:13:37.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.744+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc14d150 con 0x7ff5fc1449a0 2020-11-20T12:13:37.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.744+0000 7ff5f8ff9700 1 -- 172.21.15.187:0/4213753996 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff5f4006a80 con 0x7ff5fc1449a0 2020-11-20T12:13:37.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.745+0000 7ff5f8ff9700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5d8056d50 0x7ff5d8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:37.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.745+0000 7ff5f8ff9700 1 -- 172.21.15.187:0/4213753996 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff5f406f530 con 0x7ff5fc1449a0 2020-11-20T12:13:37.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.745+0000 7ff5fbfff700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5d8056d50 0x7ff5d8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:37.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.746+0000 7ff5fbfff700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5d8056d50 0x7ff5d8059210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7ff5fc066540 tx=0x7ff5ec019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:37.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.749+0000 7ff5f8ff9700 1 -- 172.21.15.187:0/4213753996 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff5f4044ba0 con 0x7ff5fc1449a0 2020-11-20T12:13:37.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.940+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff5fc14da10 con 0x7ff5fc1449a0 2020-11-20T12:13:37.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.941+0000 7ff5f8ff9700 1 -- 172.21.15.187:0/4213753996 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) v1 ==== 84+0+143 (secure 0 0 0) 0x7ff5f40481c0 con 0x7ff5fc1449a0 2020-11-20T12:13:37.945 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:37.945 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:13:37.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:37.946 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:13:37.946 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:13:37.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:13:37.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:37.947 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:13:37.947 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:13:37.947 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:37.947 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:13:37.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.943+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5d8056d50 msgr2=0x7ff5d8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.943+0000 7ff6016a9700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5d8056d50 0x7ff5d8059210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7ff5fc066540 tx=0x7ff5ec019000).stop 2020-11-20T12:13:37.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5fc1449a0 msgr2=0x7ff5fc15b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:37.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5fc1449a0 0x7ff5fc15b650 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7ff5fc0b85e0 tx=0x7ff5f4007f10).stop 2020-11-20T12:13:37.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 shutdown_connections 2020-11-20T12:13:37.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5d8056d50 0x7ff5d8059210 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5fc1449a0 0x7ff5fc15b650 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff5fc0b76e0 0x7ff5fc09b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 --2- 172.21.15.187:0/4213753996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5fc0b7ac0 0x7ff5fc1444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:37.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 >> 172.21.15.187:0/4213753996 conn(0x7ff5fc0bab30 msgr2=0x7ff5fc1458d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:37.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 shutdown_connections 2020-11-20T12:13:37.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:37.944+0000 7ff6016a9700 1 -- 172.21.15.187:0/4213753996 wait complete. 2020-11-20T12:13:37.972 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:13:37.973 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:13:38.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.209+0000 7f466b510700 1 Processor -- start 2020-11-20T12:13:38.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.209+0000 7f466b510700 1 -- start start 2020-11-20T12:13:38.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.211+0000 7f466b510700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4664145cc0 0x7f4664066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.211+0000 7f466b510700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466414c170 con 0x7f46641487e0 2020-11-20T12:13:38.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.211+0000 7f466b510700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466414c410 con 0x7f46640683f0 2020-11-20T12:13:38.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.211+0000 7f466b510700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466414c6b0 con 0x7f4664145cc0 2020-11-20T12:13:38.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.211+0000 7f4669d0d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f46640683f0 0x7f4664065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44588/0 (socket says 172.21.15.187:44588) 2020-11-20T12:13:38.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.211+0000 7f4669d0d700 1 -- 172.21.15.187:0/2406928028 learned_addr learned my addr 172.21.15.187:0/2406928028 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:38.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990526237 0 0) 0x7f466414c410 con 0x7f46640683f0 2020-11-20T12:13:38.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466a50e700 1 --2- 172.21.15.187:0/2406928028 >> v2:172.21.15.5:3300/0 conn(0x7f4664145cc0 0x7f4664066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f464c003240 con 0x7f46640683f0 2020-11-20T12:13:38.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3388767764 0 0) 0x7f464c003240 con 0x7f46640683f0 2020-11-20T12:13:38.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 >> v2:172.21.15.5:3300/0 conn(0x7f4664145cc0 msgr2=0x7f4664066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:38.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 --2- 172.21.15.187:0/2406928028 >> v2:172.21.15.5:3300/0 conn(0x7f4664145cc0 0x7f4664066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 >> v1:172.21.15.5:6789/0 conn(0x7f46641487e0 legacy=0x7f4664148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46640658a0 con 0x7f46640683f0 2020-11-20T12:13:38.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f46540014e0 con 0x7f46640683f0 2020-11-20T12:13:38.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.212+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 >> v1:172.21.15.187:6789/0 conn(0x7f46640683f0 legacy=0x7f4664065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.213+0000 7f466950c700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f464c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.213+0000 7f466950c700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f464c0044e0 0x7f464c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.213+0000 7f466950c700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f464c004e50 0x7f464c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.213+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466414c6b0 con 0x7f464c0044e0 2020-11-20T12:13:38.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.213+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c005300 con 0x7f464c003800 2020-11-20T12:13:38.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.213+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c00bc20 con 0x7f464c004e50 2020-11-20T12:13:38.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.213+0000 7f466a50e700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f464c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.213+0000 7f4669d0d700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f464c0044e0 0x7f464c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466a50e700 1 -- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f464c004e50 msgr2=0x7f464c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:38.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466ad0f700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f464c004e50 0x7f464c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466a50e700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f464c004e50 0x7f464c00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466a50e700 1 -- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f464c0044e0 msgr2=0x7f464c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:38.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466a50e700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f464c0044e0 0x7f464c004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466a50e700 1 -- 172.21.15.187:0/2406928028 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c003240 con 0x7f464c003800 2020-11-20T12:13:38.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466ad0f700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f464c004e50 0x7f464c00b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:38.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f4669d0d700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f464c0044e0 0x7f464c004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:38.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466ad0f700 1 -- 172.21.15.187:0/2406928028 reap_dead start 2020-11-20T12:13:38.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.214+0000 7f466a50e700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f464c003c20 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f466000bc30 tx=0x7f4660006df0).ready entity=mon.1 client_cookie=916724254f520408 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:38.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.215+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466000b310 con 0x7f464c003800 2020-11-20T12:13:38.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.215+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466000eb40 con 0x7f464c003800 2020-11-20T12:13:38.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.215+0000 7f466950c700 1 -- 172.21.15.187:0/2406928028 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466000eec0 con 0x7f464c003800 2020-11-20T12:13:38.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.215+0000 7f466b510700 1 -- 172.21.15.187:0/2406928028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 msgr2=0x7f464c003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.215+0000 7f466b510700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f464c003c20 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f466000bc30 tx=0x7f4660006df0).stop 2020-11-20T12:13:38.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.216+0000 7f466b510700 1 -- 172.21.15.187:0/2406928028 shutdown_connections 2020-11-20T12:13:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.216+0000 7f466b510700 1 --2- 172.21.15.187:0/2406928028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f464c003c20 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.216+0000 7f466b510700 1 -- 172.21.15.187:0/2406928028 >> 172.21.15.187:0/2406928028 conn(0x7f46640bab30 msgr2=0x7f46640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.216+0000 7f466b510700 1 -- 172.21.15.187:0/2406928028 shutdown_connections 2020-11-20T12:13:38.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.216+0000 7f466b510700 1 -- 172.21.15.187:0/2406928028 wait complete. 2020-11-20T12:13:38.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.217+0000 7f466b510700 1 Processor -- start 2020-11-20T12:13:38.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.217+0000 7f466b510700 1 -- start start 2020-11-20T12:13:38.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.218+0000 7f466b510700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f46640b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.218+0000 7f466b510700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f466415b570 0x7f466415bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.218+0000 7f466b510700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f466414b730 0x7f466415c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.218+0000 7f466b510700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466409d580 con 0x7f466414b730 2020-11-20T12:13:38.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.218+0000 7f466b510700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466409aef0 con 0x7f464c003800 2020-11-20T12:13:38.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.218+0000 7f466b510700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466409b190 con 0x7f466415b570 2020-11-20T12:13:38.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f46640b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f46640b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47994/0 (socket says 172.21.15.187:47994) 2020-11-20T12:13:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 -- 172.21.15.187:0/1210244527 learned_addr learned my addr 172.21.15.187:0/1210244527 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f4669d0d700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f466415b570 0x7f466415bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 -- 172.21.15.187:0/1210244527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f466415b570 msgr2=0x7f466415bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466ad0f700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f466414b730 0x7f466415c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f466415b570 0x7f466415bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 -- 172.21.15.187:0/1210244527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f466414b730 msgr2=0x7f466415c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f466414b730 0x7f466415c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 -- 172.21.15.187:0/1210244527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46640b66c0 con 0x7f464c003800 2020-11-20T12:13:38.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.219+0000 7f466a50e700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f46640b61d0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f4660006e30 tx=0x7f4660019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:38.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.220+0000 7f46527fc700 1 -- 172.21.15.187:0/1210244527 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466001c640 con 0x7f464c003800 2020-11-20T12:13:38.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.220+0000 7f46527fc700 1 -- 172.21.15.187:0/1210244527 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466001c7a0 con 0x7f464c003800 2020-11-20T12:13:38.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.220+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466409d200 con 0x7f464c003800 2020-11-20T12:13:38.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.220+0000 7f46527fc700 1 -- 172.21.15.187:0/1210244527 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466001c950 con 0x7f464c003800 2020-11-20T12:13:38.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.220+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46640b1bb0 con 0x7f464c003800 2020-11-20T12:13:38.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.221+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4664061a20 con 0x7f464c003800 2020-11-20T12:13:38.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.221+0000 7f46527fc700 1 -- 172.21.15.187:0/1210244527 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f466001cd40 con 0x7f464c003800 2020-11-20T12:13:38.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.222+0000 7f46527fc700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f463c056d20 0x7f463c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.222+0000 7f46527fc700 1 -- 172.21.15.187:0/1210244527 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4660071f90 con 0x7f464c003800 2020-11-20T12:13:38.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.222+0000 7f4669d0d700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f463c056d20 0x7f463c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.223+0000 7f4669d0d700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f463c056d20 0x7f463c0591e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f4654001d60 tx=0x7f4654000e90).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:38.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.226+0000 7f46527fc700 1 -- 172.21.15.187:0/1210244527 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4660047680 con 0x7f464c003800 2020-11-20T12:13:38.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.412+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f466409d390 con 0x7f464c003800 2020-11-20T12:13:38.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.675+0000 7f46527fc700 1 -- 172.21.15.187:0/1210244527 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v86) v1 ==== 133+0+0 (secure 0 0 0) 0x7f466004aca0 con 0x7f464c003800 2020-11-20T12:13:38.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.679+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f463c056d20 msgr2=0x7f463c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.679+0000 7f466b510700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f463c056d20 0x7f463c0591e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f4654001d60 tx=0x7f4654000e90).stop 2020-11-20T12:13:38.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.679+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 msgr2=0x7f46640b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.679+0000 7f466b510700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f46640b61d0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f4660006e30 tx=0x7f4660019000).stop 2020-11-20T12:13:38.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.680+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 shutdown_connections 2020-11-20T12:13:38.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.680+0000 7f466b510700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f463c056d20 0x7f463c0591e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.680+0000 7f466b510700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f466414b730 0x7f466415c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.680+0000 7f466b510700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f464c003800 0x7f46640b61d0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.680+0000 7f466b510700 1 --2- 172.21.15.187:0/1210244527 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f466415b570 0x7f466415bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.680+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 >> 172.21.15.187:0/1210244527 conn(0x7f46640bab30 msgr2=0x7f46640625a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:38.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.680+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 shutdown_connections 2020-11-20T12:13:38.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.680+0000 7f466b510700 1 -- 172.21.15.187:0/1210244527 wait complete. 2020-11-20T12:13:38.687 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:13:38.709 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:13:38.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.946+0000 7f18f359e700 1 Processor -- start 2020-11-20T12:13:38.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.946+0000 7f18f359e700 1 -- start start 2020-11-20T12:13:38.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.947+0000 7f18f359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f18f4065570 0x7f18f409d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.947+0000 7f18f359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f40b13e0 con 0x7f18f4065f90 2020-11-20T12:13:38.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.947+0000 7f18f359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f409d690 con 0x7f18f4065190 2020-11-20T12:13:38.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.947+0000 7f18f359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f414ab30 con 0x7f18f4065570 2020-11-20T12:13:38.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.948+0000 7f18f259c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f18f4065190 0x7f18f4143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44608/0 (socket says 172.21.15.187:44608) 2020-11-20T12:13:38.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.948+0000 7f18f259c700 1 -- 172.21.15.187:0/3813202426 learned_addr learned my addr 172.21.15.187:0/3813202426 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:38.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.948+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717911801 0 0) 0x7f18f409d690 con 0x7f18f4065190 2020-11-20T12:13:38.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.948+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d4003240 con 0x7f18f4065190 2020-11-20T12:13:38.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.948+0000 7f18f2d9d700 1 --2- 172.21.15.187:0/3813202426 >> v2:172.21.15.5:3300/0 conn(0x7f18f4065570 0x7f18f409d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1353233009 0 0) 0x7f18d4003240 con 0x7f18f4065190 2020-11-20T12:13:38.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 >> v2:172.21.15.5:3300/0 conn(0x7f18f4065570 msgr2=0x7f18f409d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:38.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 --2- 172.21.15.187:0/3813202426 >> v2:172.21.15.5:3300/0 conn(0x7f18f4065570 0x7f18f409d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 >> v1:172.21.15.5:6789/0 conn(0x7f18f4065f90 legacy=0x7f18f409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:38.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f4062c10 con 0x7f18f4065190 2020-11-20T12:13:38.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f18e40014e0 con 0x7f18f4065190 2020-11-20T12:13:38.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 >> v1:172.21.15.187:6789/0 conn(0x7f18f4065190 legacy=0x7f18f4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18d4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40044e0 0x7f18d4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d4004e50 0x7f18d400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f414ab30 con 0x7f18d40044e0 2020-11-20T12:13:38.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4004280 con 0x7f18d4003730 2020-11-20T12:13:38.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.949+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d400bc20 con 0x7f18d4004e50 2020-11-20T12:13:38.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18f259c700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18d4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18e9d9b700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40044e0 0x7f18d4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18f2d9d700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d4004e50 0x7f18d400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18f259c700 1 -- 172.21.15.187:0/3813202426 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d4004e50 msgr2=0x7f18d400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:38.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18f259c700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18d4004e50 0x7f18d400b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18f259c700 1 -- 172.21.15.187:0/3813202426 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40044e0 msgr2=0x7f18d4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:38.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18f259c700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40044e0 0x7f18d4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18f259c700 1 -- 172.21.15.187:0/3813202426 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4003240 con 0x7f18d4003730 2020-11-20T12:13:38.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.950+0000 7f18f2d9d700 1 -- 172.21.15.187:0/3813202426 reap_dead start 2020-11-20T12:13:38.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.951+0000 7f18f259c700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18d4003b70 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f18e400c440 tx=0x7f18f40b0fb0).ready entity=mon.1 client_cookie=dbb06542c4d2004 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:38.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.951+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e40072e0 con 0x7f18d4003730 2020-11-20T12:13:38.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.951+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4007440 con 0x7f18d4003730 2020-11-20T12:13:38.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.951+0000 7f18f1d9b700 1 -- 172.21.15.187:0/3813202426 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4005820 con 0x7f18d4003730 2020-11-20T12:13:38.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.951+0000 7f18f359e700 1 -- 172.21.15.187:0/3813202426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 msgr2=0x7f18d4003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.951+0000 7f18f359e700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18d4003b70 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f18e400c440 tx=0x7f18f40b0fb0).stop 2020-11-20T12:13:38.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.952+0000 7f18f359e700 1 -- 172.21.15.187:0/3813202426 shutdown_connections 2020-11-20T12:13:38.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.952+0000 7f18f359e700 1 --2- 172.21.15.187:0/3813202426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18d4003b70 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.952+0000 7f18f359e700 1 -- 172.21.15.187:0/3813202426 >> 172.21.15.187:0/3813202426 conn(0x7f18f40bab30 msgr2=0x7f18f40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:38.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.952+0000 7f18f359e700 1 -- 172.21.15.187:0/3813202426 shutdown_connections 2020-11-20T12:13:38.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.952+0000 7f18f359e700 1 -- 172.21.15.187:0/3813202426 wait complete. 2020-11-20T12:13:38.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.953+0000 7f18f359e700 1 Processor -- start 2020-11-20T12:13:38.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.954+0000 7f18f359e700 1 -- start start 2020-11-20T12:13:38.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18f4099260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18f41466a0 0x7f18f41430e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18f41435d0 0x7f18f4066e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4067ef0 con 0x7f18f41435d0 2020-11-20T12:13:38.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4068190 con 0x7f18d4003730 2020-11-20T12:13:38.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4068430 con 0x7f18f41466a0 2020-11-20T12:13:38.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18f4099260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18f4099260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48016/0 (socket says 172.21.15.187:48016) 2020-11-20T12:13:38.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f259c700 1 -- 172.21.15.187:0/1439224353 learned_addr learned my addr 172.21.15.187:0/1439224353 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:38.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18e9d9b700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18f41466a0 0x7f18f41430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f259c700 1 -- 172.21.15.187:0/1439224353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18f41466a0 msgr2=0x7f18f41430e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f259c700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18f41466a0 0x7f18f41430e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:38.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f259c700 1 -- 172.21.15.187:0/1439224353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18f41435d0 msgr2=0x7f18f4066e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:38.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f259c700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18f41435d0 0x7f18f4066e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:38.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.955+0000 7f18f259c700 1 -- 172.21.15.187:0/1439224353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f409e9c0 con 0x7f18d4003730 2020-11-20T12:13:38.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.956+0000 7f18f259c700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18f4099260 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f18e4008fd0 tx=0x7f18e4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:38.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.956+0000 7f18eaffd700 1 -- 172.21.15.187:0/1439224353 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4005320 con 0x7f18d4003730 2020-11-20T12:13:38.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.956+0000 7f18eaffd700 1 -- 172.21.15.187:0/1439224353 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4005480 con 0x7f18d4003730 2020-11-20T12:13:38.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.956+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f4099750 con 0x7f18d4003730 2020-11-20T12:13:38.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.956+0000 7f18eaffd700 1 -- 172.21.15.187:0/1439224353 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4004500 con 0x7f18d4003730 2020-11-20T12:13:38.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.956+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f4067330 con 0x7f18d4003730 2020-11-20T12:13:38.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.957+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f4061a20 con 0x7f18d4003730 2020-11-20T12:13:38.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.958+0000 7f18eaffd700 1 -- 172.21.15.187:0/1439224353 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f18e4010070 con 0x7f18d4003730 2020-11-20T12:13:38.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.959+0000 7f18eaffd700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18d0056d20 0x7f18d00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:38.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.959+0000 7f18eaffd700 1 -- 172.21.15.187:0/1439224353 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f18e40716b0 con 0x7f18d4003730 2020-11-20T12:13:38.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.959+0000 7f18e9d9b700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18d0056d20 0x7f18d00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:38.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.960+0000 7f18e9d9b700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18d0056d20 0x7f18d00591e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f18dc000bc0 tx=0x7f18dc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:38.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:38.964+0000 7f18eaffd700 1 -- 172.21.15.187:0/1439224353 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f18e4046c40 con 0x7f18d4003730 2020-11-20T12:13:39.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.150+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f18f40bb8d0 con 0x7f18d4003730 2020-11-20T12:13:39.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.695+0000 7f18eaffd700 1 -- 172.21.15.187:0/1439224353 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v87) v1 ==== 125+0+0 (secure 0 0 0) 0x7f18e404a260 con 0x7f18d4003730 2020-11-20T12:13:39.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18d0056d20 msgr2=0x7f18d00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:39.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18d0056d20 0x7f18d00591e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f18dc000bc0 tx=0x7f18dc009040).stop 2020-11-20T12:13:39.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 msgr2=0x7f18f4099260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:39.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18f4099260 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f18e4008fd0 tx=0x7f18e4015000).stop 2020-11-20T12:13:39.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 shutdown_connections 2020-11-20T12:13:39.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f18d0056d20 0x7f18d00591e0 secure :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0x7f18dc000bc0 tx=0x7f18dc009040).stop 2020-11-20T12:13:39.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18f41435d0 0x7f18f4066e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:39.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18d4003730 0x7f18f4099260 secure :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0x7f18e4008fd0 tx=0x7f18e4015000).stop 2020-11-20T12:13:39.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 --2- 172.21.15.187:0/1439224353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18f41466a0 0x7f18f41430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:39.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.704+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 >> 172.21.15.187:0/1439224353 conn(0x7f18f40bab30 msgr2=0x7f18f40b6400 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:39.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.707+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 shutdown_connections 2020-11-20T12:13:39.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.707+0000 7f18f359e700 1 -- 172.21.15.187:0/1439224353 wait complete. 2020-11-20T12:13:39.714 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:13:39.743 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:13:39.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.982+0000 7fb15474f700 1 Processor -- start 2020-11-20T12:13:39.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.982+0000 7fb15474f700 1 -- start start 2020-11-20T12:13:39.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.983+0000 7fb15474f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb14c0b73b0 0x7fb14c066e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:39.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.983+0000 7fb15474f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14c146010 con 0x7fb14c0b6fd0 2020-11-20T12:13:39.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.983+0000 7fb15474f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14c1462b0 con 0x7fb14c067300 2020-11-20T12:13:39.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.983+0000 7fb15474f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb14c146550 con 0x7fb14c0b73b0 2020-11-20T12:13:39.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.984+0000 7fb152cec700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb14c067300 0x7fb14c144d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44634/0 (socket says 172.21.15.187:44634) 2020-11-20T12:13:39.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.984+0000 7fb152cec700 1 -- 172.21.15.187:0/708276576 learned_addr learned my addr 172.21.15.187:0/708276576 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:39.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.984+0000 7fb151cea700 1 --2- 172.21.15.187:0/708276576 >> v2:172.21.15.5:3300/0 conn(0x7fb14c0b73b0 0x7fb14c066e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:39.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.985+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070543699 0 0) 0x7fb14c1462b0 con 0x7fb14c067300 2020-11-20T12:13:39.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.985+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb134003240 con 0x7fb14c067300 2020-11-20T12:13:39.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 942700240 0 0) 0x7fb134003240 con 0x7fb14c067300 2020-11-20T12:13:39.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 >> v2:172.21.15.5:3300/0 conn(0x7fb14c0b73b0 msgr2=0x7fb14c066e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:39.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 --2- 172.21.15.187:0/708276576 >> v2:172.21.15.5:3300/0 conn(0x7fb14c0b73b0 0x7fb14c066e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:39.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 >> v1:172.21.15.5:6789/0 conn(0x7fb14c0b6fd0 legacy=0x7fb14c066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:39.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c0b6650 con 0x7fb14c067300 2020-11-20T12:13:39.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fb1440014e0 con 0x7fb14c067300 2020-11-20T12:13:39.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 >> v1:172.21.15.187:6789/0 conn(0x7fb14c067300 legacy=0x7fb14c144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:39.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb134003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:39.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1340044e0 0x7fb134004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:39.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb134004e50 0x7fb13400d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:39.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c146550 con 0x7fb134004e50 2020-11-20T12:13:39.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb134004260 con 0x7fb134003730 2020-11-20T12:13:39.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13400de30 con 0x7fb1340044e0 2020-11-20T12:13:39.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.986+0000 7fb1524eb700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb134003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:39.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb151cea700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1340044e0 0x7fb134004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:39.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1524eb700 1 -- 172.21.15.187:0/708276576 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1340044e0 msgr2=0x7fb134004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:39.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1524eb700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1340044e0 0x7fb134004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:39.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1524eb700 1 -- 172.21.15.187:0/708276576 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb134004e50 msgr2=0x7fb13400d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:39.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1524eb700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb134004e50 0x7fb13400d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:39.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1524eb700 1 -- 172.21.15.187:0/708276576 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb134003240 con 0x7fb134003730 2020-11-20T12:13:39.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb152cec700 1 -- 172.21.15.187:0/708276576 reap_dead start 2020-11-20T12:13:39.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1524eb700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb134003b50 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7fb148001e70 tx=0x7fb148007f10).ready entity=mon.1 client_cookie=4f56114ff9c15cf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:39.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148001540 con 0x7fb134003730 2020-11-20T12:13:39.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb148015070 con 0x7fb134003730 2020-11-20T12:13:39.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.987+0000 7fb1514e9700 1 -- 172.21.15.187:0/708276576 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14800cbc0 con 0x7fb134003730 2020-11-20T12:13:39.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.988+0000 7fb15474f700 1 -- 172.21.15.187:0/708276576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 msgr2=0x7fb134003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:39.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.988+0000 7fb15474f700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb134003b50 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7fb148001e70 tx=0x7fb148007f10).stop 2020-11-20T12:13:39.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.988+0000 7fb15474f700 1 -- 172.21.15.187:0/708276576 shutdown_connections 2020-11-20T12:13:39.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.988+0000 7fb15474f700 1 --2- 172.21.15.187:0/708276576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb134003b50 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:39.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.988+0000 7fb15474f700 1 -- 172.21.15.187:0/708276576 >> 172.21.15.187:0/708276576 conn(0x7fb14c0bacf0 msgr2=0x7fb14c0bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:39.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.988+0000 7fb15474f700 1 -- 172.21.15.187:0/708276576 shutdown_connections 2020-11-20T12:13:39.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.988+0000 7fb15474f700 1 -- 172.21.15.187:0/708276576 wait complete. 2020-11-20T12:13:40.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.989+0000 7fb15474f700 1 Processor -- start 2020-11-20T12:13:40.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.990+0000 7fb15474f700 1 -- start start 2020-11-20T12:13:40.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.990+0000 7fb15474f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb14c142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.990+0000 7fb15474f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb14c142740 0x7fb14c157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.990+0000 7fb15474f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb14c157b50 0x7fb14c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.990+0000 7fb15474f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c139be0 con 0x7fb14c157b50 2020-11-20T12:13:40.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.990+0000 7fb15474f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c09d280 con 0x7fb134003730 2020-11-20T12:13:40.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.990+0000 7fb15474f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c09c8a0 con 0x7fb14c142740 2020-11-20T12:13:40.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb14c142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb14c142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48036/0 (socket says 172.21.15.187:48036) 2020-11-20T12:13:40.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 -- 172.21.15.187:0/2622473009 learned_addr learned my addr 172.21.15.187:0/2622473009 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:40.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 -- 172.21.15.187:0/2622473009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb14c142740 msgr2=0x7fb14c157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb14c142740 0x7fb14c157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 -- 172.21.15.187:0/2622473009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb14c157b50 msgr2=0x7fb14c0b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb14c157b50 0x7fb14c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 -- 172.21.15.187:0/2622473009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c09cb40 con 0x7fb134003730 2020-11-20T12:13:40.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.991+0000 7fb1524eb700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb14c142250 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fb148007f70 tx=0x7fb148017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:40.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.992+0000 7fb13a7fc700 1 -- 172.21.15.187:0/2622473009 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148006870 con 0x7fb134003730 2020-11-20T12:13:40.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.992+0000 7fb13a7fc700 1 -- 172.21.15.187:0/2622473009 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb148003030 con 0x7fb134003730 2020-11-20T12:13:40.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.992+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14c09cde0 con 0x7fb134003730 2020-11-20T12:13:40.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.992+0000 7fb13a7fc700 1 -- 172.21.15.187:0/2622473009 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148006c20 con 0x7fb134003730 2020-11-20T12:13:40.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.992+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14c157fb0 con 0x7fb134003730 2020-11-20T12:13:40.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.993+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14c09cf70 con 0x7fb134003730 2020-11-20T12:13:40.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.994+0000 7fb13a7fc700 1 -- 172.21.15.187:0/2622473009 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb148007030 con 0x7fb134003730 2020-11-20T12:13:40.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.994+0000 7fb13a7fc700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb124056d20 0x7fb1240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.994+0000 7fb13a7fc700 1 -- 172.21.15.187:0/2622473009 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fb148075800 con 0x7fb134003730 2020-11-20T12:13:40.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.995+0000 7fb151cea700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb124056d20 0x7fb1240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.995+0000 7fb151cea700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb124056d20 0x7fb1240591e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7fb13c006a90 tx=0x7fb13c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:40.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:39.998+0000 7fb13a7fc700 1 -- 172.21.15.187:0/2622473009 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb14804abb0 con 0x7fb134003730 2020-11-20T12:13:40.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.249+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fb14c044f60 con 0x7fb134003730 2020-11-20T12:13:40.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.264+0000 7fb13a7fc700 1 -- 172.21.15.187:0/2622473009 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 17 and data pool 18 v77) v1 ==== 189+0+0 (secure 0 0 0) 0x7fb14804e1d0 con 0x7fb134003730 2020-11-20T12:13:40.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb124056d20 msgr2=0x7fb1240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb124056d20 0x7fb1240591e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7fb13c006a90 tx=0x7fb13c008040).stop 2020-11-20T12:13:40.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 msgr2=0x7fb14c142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb14c142250 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fb148007f70 tx=0x7fb148017000).stop 2020-11-20T12:13:40.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 shutdown_connections 2020-11-20T12:13:40.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb124056d20 0x7fb1240591e0 secure :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0x7fb13c006a90 tx=0x7fb13c008040).stop 2020-11-20T12:13:40.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb14c157b50 0x7fb14c0b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb134003730 0x7fb14c142250 secure :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0x7fb148007f70 tx=0x7fb148017000).stop 2020-11-20T12:13:40.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 --2- 172.21.15.187:0/2622473009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb14c142740 0x7fb14c157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.269+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 >> 172.21.15.187:0/2622473009 conn(0x7fb14c0bacf0 msgr2=0x7fb14c1432f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:40.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.270+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 shutdown_connections 2020-11-20T12:13:40.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.270+0000 7fb15474f700 1 -- 172.21.15.187:0/2622473009 wait complete. 2020-11-20T12:13:40.278 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 17 and data pool 18 2020-11-20T12:13:40.301 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:13:40.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.498+0000 7ffac6d63700 1 Processor -- start 2020-11-20T12:13:40.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.498+0000 7ffac6d63700 1 -- start start 2020-11-20T12:13:40.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.499+0000 7ffac6d63700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffac00683f0 0x7ffac0066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.499+0000 7ffac6d63700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 0x7ffac0148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.499+0000 7ffac6d63700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac014c170 con 0x7ffac0145cc0 2020-11-20T12:13:40.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.499+0000 7ffac6d63700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac014c410 con 0x7ffac00683f0 2020-11-20T12:13:40.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.499+0000 7ffac6d63700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac014c6b0 con 0x7ffac01487e0 2020-11-20T12:13:40.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.500+0000 7ffac5560700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffac00683f0 0x7ffac0066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.500+0000 7ffac5d61700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ffac0145cc0 0x7ffac0065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44658/0 (socket says 172.21.15.187:44658) 2020-11-20T12:13:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.500+0000 7ffac5d61700 1 -- 172.21.15.187:0/2584112344 learned_addr learned my addr 172.21.15.187:0/2584112344 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.500+0000 7ffac6562700 1 --2- 172.21.15.187:0/2584112344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 0x7ffac0148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac5560700 1 -- 172.21.15.187:0/2584112344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 msgr2=0x7ffac0148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac5560700 1 --2- 172.21.15.187:0/2584112344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 0x7ffac0148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac5560700 1 -- 172.21.15.187:0/2584112344 >> v1:172.21.15.187:6789/0 conn(0x7ffac0145cc0 legacy=0x7ffac0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac5560700 1 -- 172.21.15.187:0/2584112344 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac00658a0 con 0x7ffac00683f0 2020-11-20T12:13:40.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac4d5f700 1 -- 172.21.15.187:0/2584112344 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429080863 0 0) 0x7ffac014c170 con 0x7ffac0145cc0 2020-11-20T12:13:40.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac5560700 1 --2- 172.21.15.187:0/2584112344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac0066040 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7ffab000b9a0 tx=0x7ffab0006df0).ready entity=mon.1 client_cookie=4520ad3534157dd3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:40.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac4d5f700 1 -- 172.21.15.187:0/2584112344 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab00097b0 con 0x7ffac00683f0 2020-11-20T12:13:40.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac4d5f700 1 -- 172.21.15.187:0/2584112344 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab0009910 con 0x7ffac00683f0 2020-11-20T12:13:40.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.501+0000 7ffac4d5f700 1 -- 172.21.15.187:0/2584112344 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0009c90 con 0x7ffac00683f0 2020-11-20T12:13:40.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.502+0000 7ffac6d63700 1 -- 172.21.15.187:0/2584112344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 msgr2=0x7ffac0066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.502+0000 7ffac6d63700 1 --2- 172.21.15.187:0/2584112344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac0066040 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7ffab000b9a0 tx=0x7ffab0006df0).stop 2020-11-20T12:13:40.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.502+0000 7ffac6d63700 1 -- 172.21.15.187:0/2584112344 shutdown_connections 2020-11-20T12:13:40.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.502+0000 7ffac6d63700 1 --2- 172.21.15.187:0/2584112344 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 0x7ffac0148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.502+0000 7ffac6d63700 1 --2- 172.21.15.187:0/2584112344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac0066040 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.502+0000 7ffac6d63700 1 -- 172.21.15.187:0/2584112344 >> 172.21.15.187:0/2584112344 conn(0x7ffac00bab30 msgr2=0x7ffac00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:40.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.502+0000 7ffac6d63700 1 -- 172.21.15.187:0/2584112344 shutdown_connections 2020-11-20T12:13:40.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.502+0000 7ffac6d63700 1 -- 172.21.15.187:0/2584112344 wait complete. 2020-11-20T12:13:40.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.503+0000 7ffac6d63700 1 Processor -- start 2020-11-20T12:13:40.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.504+0000 7ffac6d63700 1 -- start start 2020-11-20T12:13:40.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac6d63700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac015c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac6d63700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffac0145cc0 0x7ffac0144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac6d63700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 0x7ffac009f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac6d63700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00b0690 con 0x7ffac0145cc0 2020-11-20T12:13:40.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac6d63700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00b0930 con 0x7ffac00683f0 2020-11-20T12:13:40.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac6d63700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00b0bd0 con 0x7ffac01487e0 2020-11-20T12:13:40.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac5d61700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac5d61700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac015c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48058/0 (socket says 172.21.15.187:48058) 2020-11-20T12:13:40.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac5d61700 1 -- 172.21.15.187:0/3646014989 learned_addr learned my addr 172.21.15.187:0/3646014989 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:40.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac5d61700 1 -- 172.21.15.187:0/3646014989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 msgr2=0x7ffac009f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac5d61700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 0x7ffac009f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac5d61700 1 -- 172.21.15.187:0/3646014989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffac0145cc0 msgr2=0x7ffac0144f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac5d61700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffac0145cc0 0x7ffac0144f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.505+0000 7ffac5d61700 1 -- 172.21.15.187:0/3646014989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac015d090 con 0x7ffac00683f0 2020-11-20T12:13:40.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.506+0000 7ffac5d61700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac015c150 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7ffabc00c720 tx=0x7ffabc007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:40.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.506+0000 7ffaadffb700 1 -- 172.21.15.187:0/3646014989 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc007320 con 0x7ffac00683f0 2020-11-20T12:13:40.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.506+0000 7ffaadffb700 1 -- 172.21.15.187:0/3646014989 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc007480 con 0x7ffac00683f0 2020-11-20T12:13:40.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.506+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac009d320 con 0x7ffac00683f0 2020-11-20T12:13:40.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.506+0000 7ffaadffb700 1 -- 172.21.15.187:0/3646014989 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc005800 con 0x7ffac00683f0 2020-11-20T12:13:40.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.506+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac0145650 con 0x7ffac00683f0 2020-11-20T12:13:40.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.507+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac00b0d90 con 0x7ffac00683f0 2020-11-20T12:13:40.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.508+0000 7ffaadffb700 1 -- 172.21.15.187:0/3646014989 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffabc010030 con 0x7ffac00683f0 2020-11-20T12:13:40.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.509+0000 7ffaadffb700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa98056d50 0x7ffa98059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.509+0000 7ffaadffb700 1 -- 172.21.15.187:0/3646014989 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffabc06fca0 con 0x7ffac00683f0 2020-11-20T12:13:40.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.509+0000 7ffac5560700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa98056d50 0x7ffa98059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.510+0000 7ffac5560700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa98056d50 0x7ffa98059210 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7ffac009c8f0 tx=0x7ffab0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:40.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.512+0000 7ffaadffb700 1 -- 172.21.15.187:0/3646014989 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffabc074070 con 0x7ffac00683f0 2020-11-20T12:13:40.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.683+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffac0061a20 con 0x7ffac00683f0 2020-11-20T12:13:40.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.685+0000 7ffaadffb700 1 -- 172.21.15.187:0/3646014989 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16794 (secure 0 0 0) 0x7ffabc0450f0 con 0x7ffac00683f0 2020-11-20T12:13:40.690 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:40.690 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":88,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:13:40.252772+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:13:38.491662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2020-11-20T12:13:39.231237+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:13:40.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.687+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa98056d50 msgr2=0x7ffa98059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.687+0000 7ffac6d63700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa98056d50 0x7ffa98059210 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7ffac009c8f0 tx=0x7ffab0015000).stop 2020-11-20T12:13:40.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.687+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 msgr2=0x7ffac015c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.687+0000 7ffac6d63700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac015c150 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7ffabc00c720 tx=0x7ffabc007ac0).stop 2020-11-20T12:13:40.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.688+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 shutdown_connections 2020-11-20T12:13:40.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.688+0000 7ffac6d63700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa98056d50 0x7ffa98059210 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.688+0000 7ffac6d63700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffac01487e0 0x7ffac009f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.688+0000 7ffac6d63700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffac00683f0 0x7ffac015c150 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.688+0000 7ffac6d63700 1 --2- 172.21.15.187:0/3646014989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffac0145cc0 0x7ffac0144f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.688+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 >> 172.21.15.187:0/3646014989 conn(0x7ffac00bab30 msgr2=0x7ffac014b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:40.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.688+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 shutdown_connections 2020-11-20T12:13:40.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.688+0000 7ffac6d63700 1 -- 172.21.15.187:0/3646014989 wait complete. 2020-11-20T12:13:40.716 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:13:40.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.939+0000 7ff8b9fe2700 1 Processor -- start 2020-11-20T12:13:40.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.939+0000 7ff8b9fe2700 1 -- start start 2020-11-20T12:13:40.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b9fe2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8b40b57b0 0x7ff8b4068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b9fe2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff8b41482d0 0x7ff8b4062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b9fe2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b4062b50 0x7ff8b4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b9fe2700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b4144b80 con 0x7ff8b41482d0 2020-11-20T12:13:40.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b9fe2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b41489e0 con 0x7ff8b40b57b0 2020-11-20T12:13:40.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b9fe2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4148c80 con 0x7ff8b4062b50 2020-11-20T12:13:40.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b8fe0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8b40b57b0 0x7ff8b4068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b8fe0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8b40b57b0 0x7ff8b4068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48068/0 (socket says 172.21.15.187:48068) 2020-11-20T12:13:40.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b8fe0700 1 -- 172.21.15.187:0/2511892997 learned_addr learned my addr 172.21.15.187:0/2511892997 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:40.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b97e1700 1 --2- 172.21.15.187:0/2511892997 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b4062b50 0x7ff8b4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.941+0000 7ff8b3fff700 1 --2- 172.21.15.187:0/2511892997 >> v2:172.21.15.5:3300/0 conn(0x7ff8b41482d0 0x7ff8b4062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.942+0000 7ff8b8fe0700 1 -- 172.21.15.187:0/2511892997 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b4062b50 msgr2=0x7ff8b4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:40.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.942+0000 7ff8b8fe0700 1 --2- 172.21.15.187:0/2511892997 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b4062b50 0x7ff8b4065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.942+0000 7ff8b8fe0700 1 -- 172.21.15.187:0/2511892997 >> v2:172.21.15.5:3300/0 conn(0x7ff8b41482d0 msgr2=0x7ff8b4062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.942+0000 7ff8b8fe0700 1 --2- 172.21.15.187:0/2511892997 >> v2:172.21.15.5:3300/0 conn(0x7ff8b41482d0 0x7ff8b4062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.942+0000 7ff8b8fe0700 1 -- 172.21.15.187:0/2511892997 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4148f20 con 0x7ff8b40b57b0 2020-11-20T12:13:40.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.942+0000 7ff8b8fe0700 1 --2- 172.21.15.187:0/2511892997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b40b57b0 0x7ff8b4068230 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7ff8b40b85e0 tx=0x7ff8a8006eb0).ready entity=mon.1 client_cookie=78bcf45917a9f0d1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:40.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.943+0000 7ff8b37fe700 1 -- 172.21.15.187:0/2511892997 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80098d0 con 0x7ff8b40b57b0 2020-11-20T12:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.943+0000 7ff8b37fe700 1 -- 172.21.15.187:0/2511892997 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a8009a30 con 0x7ff8b40b57b0 2020-11-20T12:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.943+0000 7ff8b37fe700 1 -- 172.21.15.187:0/2511892997 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8009db0 con 0x7ff8b40b57b0 2020-11-20T12:13:40.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.944+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2511892997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b40b57b0 msgr2=0x7ff8b4068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.944+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2511892997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b40b57b0 0x7ff8b4068230 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7ff8b40b85e0 tx=0x7ff8a8006eb0).stop 2020-11-20T12:13:40.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.944+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2511892997 shutdown_connections 2020-11-20T12:13:40.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.944+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2511892997 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b4062b50 0x7ff8b4065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.944+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2511892997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b40b57b0 0x7ff8b4068230 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.944+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2511892997 >> v2:172.21.15.5:3300/0 conn(0x7ff8b41482d0 0x7ff8b4062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.944+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2511892997 >> 172.21.15.187:0/2511892997 conn(0x7ff8b40bab30 msgr2=0x7ff8b40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:40.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.945+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2511892997 shutdown_connections 2020-11-20T12:13:40.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.945+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2511892997 wait complete. 2020-11-20T12:13:40.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.946+0000 7ff8b9fe2700 1 Processor -- start 2020-11-20T12:13:40.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.947+0000 7ff8b9fe2700 1 -- start start 2020-11-20T12:13:40.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b9fe2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8b4062b50 0x7ff8b41468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b9fe2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b40b57b0 0x7ff8b40b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b9fe2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b41482d0 0x7ff8b40b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b9fe2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4158560 con 0x7ff8b4062b50 2020-11-20T12:13:40.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b9fe2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4158800 con 0x7ff8b41482d0 2020-11-20T12:13:40.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b9fe2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4158aa0 con 0x7ff8b40b57b0 2020-11-20T12:13:40.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b97e1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b41482d0 0x7ff8b40b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b97e1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b41482d0 0x7ff8b40b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48076/0 (socket says 172.21.15.187:48076) 2020-11-20T12:13:40.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b97e1700 1 -- 172.21.15.187:0/2112163562 learned_addr learned my addr 172.21.15.187:0/2112163562 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:40.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.948+0000 7ff8b8fe0700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8b4062b50 0x7ff8b41468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b97e1700 1 -- 172.21.15.187:0/2112163562 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b40b57b0 msgr2=0x7ff8b40b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b97e1700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b40b57b0 0x7ff8b40b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:40.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b97e1700 1 -- 172.21.15.187:0/2112163562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8b4062b50 msgr2=0x7ff8b41468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:40.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b97e1700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8b4062b50 0x7ff8b41468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:40.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b97e1700 1 -- 172.21.15.187:0/2112163562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4158f40 con 0x7ff8b41482d0 2020-11-20T12:13:40.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b97e1700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b41482d0 0x7ff8b40b4380 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7ff8ac00eb70 tx=0x7ff8ac008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:40.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b0ff9700 1 -- 172.21.15.187:0/2112163562 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00c830 con 0x7ff8b41482d0 2020-11-20T12:13:40.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b0ff9700 1 -- 172.21.15.187:0/2112163562 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac00c990 con 0x7ff8b41482d0 2020-11-20T12:13:40.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b0ff9700 1 -- 172.21.15.187:0/2112163562 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00cd10 con 0x7ff8b41482d0 2020-11-20T12:13:40.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b4146de0 con 0x7ff8b41482d0 2020-11-20T12:13:40.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.949+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b41580c0 con 0x7ff8b41482d0 2020-11-20T12:13:40.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.950+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b4158c60 con 0x7ff8b41482d0 2020-11-20T12:13:40.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.951+0000 7ff8b0ff9700 1 -- 172.21.15.187:0/2112163562 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff8ac013640 con 0x7ff8b41482d0 2020-11-20T12:13:40.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.951+0000 7ff8b0ff9700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff88c056d20 0x7ff88c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:40.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.951+0000 7ff8b0ff9700 1 -- 172.21.15.187:0/2112163562 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff8ac06fde0 con 0x7ff8b41482d0 2020-11-20T12:13:40.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.952+0000 7ff8b8fe0700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff88c056d20 0x7ff88c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:40.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.952+0000 7ff8b8fe0700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff88c056d20 0x7ff88c0591e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7ff8a8006ef0 tx=0x7ff8a8017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:40.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:40.955+0000 7ff8b0ff9700 1 -- 172.21.15.187:0/2112163562 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff8ac073030 con 0x7ff8b41482d0 2020-11-20T12:13:41.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.126+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8b4061a20 con 0x7ff8b41482d0 2020-11-20T12:13:41.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.128+0000 7ff8b0ff9700 1 -- 172.21.15.187:0/2112163562 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16794 (secure 0 0 0) 0x7ff8ac0451b0 con 0x7ff8b41482d0 2020-11-20T12:13:41.132 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:41.133 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":88,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:13:40.252772+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:13:38.491662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2020-11-20T12:13:39.231237+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:13:41.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.130+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff88c056d20 msgr2=0x7ff88c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:41.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.130+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff88c056d20 0x7ff88c0591e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7ff8a8006ef0 tx=0x7ff8a8017000).stop 2020-11-20T12:13:41.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.130+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b41482d0 msgr2=0x7ff8b40b4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:41.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.130+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b41482d0 0x7ff8b40b4380 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7ff8ac00eb70 tx=0x7ff8ac008040).stop 2020-11-20T12:13:41.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.131+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 shutdown_connections 2020-11-20T12:13:41.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.131+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff88c056d20 0x7ff88c0591e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:41.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.131+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8b4062b50 0x7ff8b41468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:41.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.131+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8b41482d0 0x7ff8b40b4380 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:41.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.131+0000 7ff8b9fe2700 1 --2- 172.21.15.187:0/2112163562 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8b40b57b0 0x7ff8b40b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:41.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.131+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 >> 172.21.15.187:0/2112163562 conn(0x7ff8b40bab30 msgr2=0x7ff8b40b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:41.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.131+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 shutdown_connections 2020-11-20T12:13:41.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.131+0000 7ff8b9fe2700 1 -- 172.21.15.187:0/2112163562 wait complete. 2020-11-20T12:13:41.157 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:13:41.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.394+0000 7f6a4f4d4700 1 Processor -- start 2020-11-20T12:13:41.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.394+0000 7f6a4f4d4700 1 -- start start 2020-11-20T12:13:41.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.395+0000 7f6a4f4d4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065570 0x7f6a4809d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:41.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.395+0000 7f6a4f4d4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a480b13e0 con 0x7f6a48065f90 2020-11-20T12:13:41.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.396+0000 7f6a4f4d4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4809d690 con 0x7f6a48065190 2020-11-20T12:13:41.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.396+0000 7f6a4f4d4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4814ab30 con 0x7f6a48065570 2020-11-20T12:13:41.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.396+0000 7f6a4e4d2700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6a48065190 0x7f6a48143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44690/0 (socket says 172.21.15.187:44690) 2020-11-20T12:13:41.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.396+0000 7f6a4e4d2700 1 -- 172.21.15.187:0/3656699118 learned_addr learned my addr 172.21.15.187:0/3656699118 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:41.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.396+0000 7f6a4ecd3700 1 --2- 172.21.15.187:0/3656699118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065570 0x7f6a4809d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:41.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.397+0000 7f6a4ecd3700 1 -- 172.21.15.187:0/3656699118 >> v1:172.21.15.187:6789/0 conn(0x7f6a48065190 legacy=0x7f6a48143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:41.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.397+0000 7f6a4ecd3700 1 -- 172.21.15.187:0/3656699118 >> v1:172.21.15.5:6789/0 conn(0x7f6a48065f90 legacy=0x7f6a4809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:41.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.397+0000 7f6a4ecd3700 1 -- 172.21.15.187:0/3656699118 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a48062c10 con 0x7f6a48065570 2020-11-20T12:13:41.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.398+0000 7f6a4ecd3700 1 --2- 172.21.15.187:0/3656699118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065570 0x7f6a4809d1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6a40009cc0 tx=0x7f6a400091d0).ready entity=mon.2 client_cookie=299e78929f827fd6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:41.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.401+0000 7f6a4d4d0700 1 -- 172.21.15.187:0/3656699118 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4000bc50 con 0x7f6a48065570 2020-11-20T12:13:41.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.401+0000 7f6a4d4d0700 1 -- 172.21.15.187:0/3656699118 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4000bdb0 con 0x7f6a48065570 2020-11-20T12:13:41.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.401+0000 7f6a4d4d0700 1 -- 172.21.15.187:0/3656699118 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40011610 con 0x7f6a48065570 2020-11-20T12:13:41.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.401+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/3656699118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065570 msgr2=0x7f6a4809d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:41.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.401+0000 7f6a4f4d4700 1 --2- 172.21.15.187:0/3656699118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065570 0x7f6a4809d1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6a40009cc0 tx=0x7f6a400091d0).stop 2020-11-20T12:13:41.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.402+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/3656699118 shutdown_connections 2020-11-20T12:13:41.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.402+0000 7f6a4f4d4700 1 --2- 172.21.15.187:0/3656699118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065570 0x7f6a4809d1a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:41.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.402+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/3656699118 >> 172.21.15.187:0/3656699118 conn(0x7f6a480bab30 msgr2=0x7f6a480baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:41.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.402+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/3656699118 shutdown_connections 2020-11-20T12:13:41.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.402+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/3656699118 wait complete. 2020-11-20T12:13:41.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.404+0000 7f6a4f4d4700 1 Processor -- start 2020-11-20T12:13:41.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.404+0000 7f6a4f4d4700 1 -- start start 2020-11-20T12:13:41.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4f4d4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a48065190 0x7f6a4809fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:41.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4f4d4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a48065570 0x7f6a4815c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:41.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4f4d4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065f90 0x7f6a4809eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:41.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4f4d4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4809fc20 con 0x7f6a48065570 2020-11-20T12:13:41.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4f4d4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481473f0 con 0x7f6a48065190 2020-11-20T12:13:41.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4f4d4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a48147690 con 0x7f6a48065f90 2020-11-20T12:13:41.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4e4d2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a48065190 0x7f6a4809fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:41.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4e4d2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a48065190 0x7f6a4809fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48086/0 (socket says 172.21.15.187:48086) 2020-11-20T12:13:41.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.405+0000 7f6a4e4d2700 1 -- 172.21.15.187:0/1965001832 learned_addr learned my addr 172.21.15.187:0/1965001832 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:41.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.406+0000 7f6a4ecd3700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065f90 0x7f6a4809eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:41.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.406+0000 7f6a4dcd1700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a48065570 0x7f6a4815c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:41.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.406+0000 7f6a4e4d2700 1 -- 172.21.15.187:0/1965001832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065f90 msgr2=0x7f6a4809eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:41.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.406+0000 7f6a4e4d2700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065f90 0x7f6a4809eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:41.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.406+0000 7f6a4e4d2700 1 -- 172.21.15.187:0/1965001832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a48065570 msgr2=0x7f6a4815c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:41.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.406+0000 7f6a4e4d2700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a48065570 0x7f6a4815c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:41.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.406+0000 7f6a4e4d2700 1 -- 172.21.15.187:0/1965001832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4809f060 con 0x7f6a48065190 2020-11-20T12:13:41.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.406+0000 7f6a4e4d2700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a48065190 0x7f6a4809fe00 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7f6a4400c710 tx=0x7f6a44007c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:41.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.407+0000 7f6a367fc700 1 -- 172.21.15.187:0/1965001832 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44007320 con 0x7f6a48065190 2020-11-20T12:13:41.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.407+0000 7f6a367fc700 1 -- 172.21.15.187:0/1965001832 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a44007480 con 0x7f6a48065190 2020-11-20T12:13:41.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.407+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a481470e0 con 0x7f6a48065190 2020-11-20T12:13:41.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.407+0000 7f6a367fc700 1 -- 172.21.15.187:0/1965001832 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44005800 con 0x7f6a48065190 2020-11-20T12:13:41.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.407+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a48147d70 con 0x7f6a48065190 2020-11-20T12:13:41.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.408+0000 7f6a367fc700 1 -- 172.21.15.187:0/1965001832 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6a44010030 con 0x7f6a48065190 2020-11-20T12:13:41.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.408+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a48147850 con 0x7f6a48065190 2020-11-20T12:13:41.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.409+0000 7f6a367fc700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a20056d20 0x7f6a200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:41.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.409+0000 7f6a367fc700 1 -- 172.21.15.187:0/1965001832 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6a4406fc90 con 0x7f6a48065190 2020-11-20T12:13:41.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.409+0000 7f6a4dcd1700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a20056d20 0x7f6a200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:41.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.410+0000 7f6a4dcd1700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a20056d20 0x7f6a200591e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f6a480685e0 tx=0x7f6a38009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:41.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.413+0000 7f6a367fc700 1 -- 172.21.15.187:0/1965001832 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6a440450e0 con 0x7f6a48065190 2020-11-20T12:13:41.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:41.661+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6a48061a20 con 0x7f6a48065190 2020-11-20T12:13:43.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.411+0000 7f6a367fc700 1 -- 172.21.15.187:0/1965001832 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v80) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6a44048700 con 0x7f6a48065190 2020-11-20T12:13:43.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.414+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a20056d20 msgr2=0x7f6a200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.414+0000 7f6a4f4d4700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a20056d20 0x7f6a200591e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f6a480685e0 tx=0x7f6a38009040).stop 2020-11-20T12:13:43.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.414+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a48065190 msgr2=0x7f6a4809fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.414+0000 7f6a4f4d4700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a48065190 0x7f6a4809fe00 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7f6a4400c710 tx=0x7f6a44007c40).stop 2020-11-20T12:13:43.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.415+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 shutdown_connections 2020-11-20T12:13:43.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.415+0000 7f6a4f4d4700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6a20056d20 0x7f6a200591e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.415+0000 7f6a4f4d4700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6a48065f90 0x7f6a4809eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.415+0000 7f6a4f4d4700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a48065190 0x7f6a4809fe00 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.415+0000 7f6a4f4d4700 1 --2- 172.21.15.187:0/1965001832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a48065570 0x7f6a4815c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.415+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 >> 172.21.15.187:0/1965001832 conn(0x7f6a480bab30 msgr2=0x7f6a481458a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:43.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.415+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 shutdown_connections 2020-11-20T12:13:43.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:42.415+0000 7f6a4f4d4700 1 -- 172.21.15.187:0/1965001832 wait complete. 2020-11-20T12:13:43.169 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:43.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.419+0000 7ff38d036700 1 Processor -- start 2020-11-20T12:13:43.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.419+0000 7ff38d036700 1 -- start start 2020-11-20T12:13:43.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.421+0000 7ff38d036700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff3880bb940 0x7ff3880b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.421+0000 7ff38d036700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff3880bbd20 0x7ff3880b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.421+0000 7ff38d036700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3881497f0 con 0x7ff388063110 2020-11-20T12:13:43.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.421+0000 7ff38d036700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3880b7ac0 con 0x7ff3880bb940 2020-11-20T12:13:43.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.421+0000 7ff38d036700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff388149470 con 0x7ff3880bbd20 2020-11-20T12:13:43.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.421+0000 7ff38659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff3880bbd20 0x7ff3880b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.421+0000 7ff38759e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff388063110 0x7ff3880b8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44704/0 (socket says 172.21.15.187:44704) 2020-11-20T12:13:43.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.421+0000 7ff38759e700 1 -- 172.21.15.187:0/2879918424 learned_addr learned my addr 172.21.15.187:0/2879918424 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:43.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.422+0000 7ff386d9d700 1 --2- 172.21.15.187:0/2879918424 >> v2:172.21.15.5:3300/0 conn(0x7ff3880bb940 0x7ff3880b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.422+0000 7ff38659c700 1 -- 172.21.15.187:0/2879918424 >> v2:172.21.15.5:3300/0 conn(0x7ff3880bb940 msgr2=0x7ff3880b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:43.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.422+0000 7ff38659c700 1 --2- 172.21.15.187:0/2879918424 >> v2:172.21.15.5:3300/0 conn(0x7ff3880bb940 0x7ff3880b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.422+0000 7ff38659c700 1 -- 172.21.15.187:0/2879918424 >> v1:172.21.15.187:6789/0 conn(0x7ff388063110 legacy=0x7ff3880b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.422+0000 7ff385d9b700 1 -- 172.21.15.187:0/2879918424 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783788505 0 0) 0x7ff3881497f0 con 0x7ff388063110 2020-11-20T12:13:43.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.422+0000 7ff38659c700 1 -- 172.21.15.187:0/2879918424 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff388062700 con 0x7ff3880bbd20 2020-11-20T12:13:43.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.422+0000 7ff38659c700 1 --2- 172.21.15.187:0/2879918424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3880bbd20 0x7ff3880b7f60 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7ff388067d10 tx=0x7ff37c006df0).ready entity=mon.1 client_cookie=5b2071f5f1b3372e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:43.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.423+0000 7ff385d9b700 1 -- 172.21.15.187:0/2879918424 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0097b0 con 0x7ff3880bbd20 2020-11-20T12:13:43.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.423+0000 7ff385d9b700 1 -- 172.21.15.187:0/2879918424 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c009910 con 0x7ff3880bbd20 2020-11-20T12:13:43.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.423+0000 7ff385d9b700 1 -- 172.21.15.187:0/2879918424 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c009c90 con 0x7ff3880bbd20 2020-11-20T12:13:43.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.423+0000 7ff38d036700 1 -- 172.21.15.187:0/2879918424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3880bbd20 msgr2=0x7ff3880b7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.423+0000 7ff38d036700 1 --2- 172.21.15.187:0/2879918424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3880bbd20 0x7ff3880b7f60 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7ff388067d10 tx=0x7ff37c006df0).stop 2020-11-20T12:13:43.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.423+0000 7ff38d036700 1 -- 172.21.15.187:0/2879918424 shutdown_connections 2020-11-20T12:13:43.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.423+0000 7ff38d036700 1 --2- 172.21.15.187:0/2879918424 >> v2:172.21.15.5:3300/0 conn(0x7ff3880bb940 0x7ff3880b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.423+0000 7ff38d036700 1 --2- 172.21.15.187:0/2879918424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3880bbd20 0x7ff3880b7f60 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.424+0000 7ff38d036700 1 -- 172.21.15.187:0/2879918424 >> 172.21.15.187:0/2879918424 conn(0x7ff38809d640 msgr2=0x7ff38809da50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:43.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.424+0000 7ff38d036700 1 -- 172.21.15.187:0/2879918424 shutdown_connections 2020-11-20T12:13:43.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.424+0000 7ff38d036700 1 -- 172.21.15.187:0/2879918424 wait complete. 2020-11-20T12:13:43.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.425+0000 7ff38d036700 1 Processor -- start 2020-11-20T12:13:43.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.425+0000 7ff38d036700 1 -- start start 2020-11-20T12:13:43.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.426+0000 7ff38d036700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff388063110 0x7ff38814e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.426+0000 7ff38d036700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3880bb940 0x7ff38815b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.426+0000 7ff38d036700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3880bbd20 0x7ff38815bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.426+0000 7ff38d036700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff388158500 con 0x7ff3880bb940 2020-11-20T12:13:43.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.426+0000 7ff38d036700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881587a0 con 0x7ff388063110 2020-11-20T12:13:43.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.426+0000 7ff38d036700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff388158a40 con 0x7ff3880bbd20 2020-11-20T12:13:43.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.426+0000 7ff386d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff388063110 0x7ff38814e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff386d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff388063110 0x7ff38814e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48100/0 (socket says 172.21.15.187:48100) 2020-11-20T12:13:43.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff386d9d700 1 -- 172.21.15.187:0/800356553 learned_addr learned my addr 172.21.15.187:0/800356553 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:43.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff38659c700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3880bb940 0x7ff38815b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff38759e700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3880bbd20 0x7ff38815bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff386d9d700 1 -- 172.21.15.187:0/800356553 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3880bbd20 msgr2=0x7ff38815bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff386d9d700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3880bbd20 0x7ff38815bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff386d9d700 1 -- 172.21.15.187:0/800356553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3880bb940 msgr2=0x7ff38815b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff386d9d700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3880bb940 0x7ff38815b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff386d9d700 1 -- 172.21.15.187:0/800356553 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880b43b0 con 0x7ff388063110 2020-11-20T12:13:43.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.427+0000 7ff386d9d700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff388063110 0x7ff38814e660 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7ff37800db60 tx=0x7ff378008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:43.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.428+0000 7ff36effd700 1 -- 172.21.15.187:0/800356553 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378009790 con 0x7ff388063110 2020-11-20T12:13:43.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.428+0000 7ff36effd700 1 -- 172.21.15.187:0/800356553 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3780098f0 con 0x7ff388063110 2020-11-20T12:13:43.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.428+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff388158ee0 con 0x7ff388063110 2020-11-20T12:13:43.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.428+0000 7ff36effd700 1 -- 172.21.15.187:0/800356553 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378009c90 con 0x7ff388063110 2020-11-20T12:13:43.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.428+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3881582a0 con 0x7ff388063110 2020-11-20T12:13:43.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.429+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3880b14f0 con 0x7ff388063110 2020-11-20T12:13:43.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.430+0000 7ff36effd700 1 -- 172.21.15.187:0/800356553 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff378013480 con 0x7ff388063110 2020-11-20T12:13:43.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.430+0000 7ff36effd700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff364056d20 0x7ff3640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.430+0000 7ff36effd700 1 -- 172.21.15.187:0/800356553 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff378070660 con 0x7ff388063110 2020-11-20T12:13:43.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.431+0000 7ff38659c700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff364056d20 0x7ff3640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.431+0000 7ff38659c700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff364056d20 0x7ff3640591e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff38815b590 tx=0x7ff37c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:43.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.434+0000 7ff36effd700 1 -- 172.21.15.187:0/800356553 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff378045940 con 0x7ff388063110 2020-11-20T12:13:43.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.686+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff388158c00 con 0x7ff388063110 2020-11-20T12:13:43.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.686+0000 7ff36effd700 1 -- 172.21.15.187:0/800356553 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7ff378048f60 con 0x7ff388063110 2020-11-20T12:13:43.691 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:43.691 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:13:43.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff364056d20 msgr2=0x7ff3640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff364056d20 0x7ff3640591e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff38815b590 tx=0x7ff37c015000).stop 2020-11-20T12:13:43.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff388063110 msgr2=0x7ff38814e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff388063110 0x7ff38814e660 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7ff37800db60 tx=0x7ff378008040).stop 2020-11-20T12:13:43.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 shutdown_connections 2020-11-20T12:13:43.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff364056d20 0x7ff3640591e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3880bbd20 0x7ff38815bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff388063110 0x7ff38814e660 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 --2- 172.21.15.187:0/800356553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3880bb940 0x7ff38815b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 >> 172.21.15.187:0/800356553 conn(0x7ff38809d640 msgr2=0x7ff3880b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 shutdown_connections 2020-11-20T12:13:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.690+0000 7ff38d036700 1 -- 172.21.15.187:0/800356553 wait complete. 2020-11-20T12:13:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:13:43.718 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:13:43.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.948+0000 7f494759e700 1 Processor -- start 2020-11-20T12:13:43.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.949+0000 7f494759e700 1 -- start start 2020-11-20T12:13:43.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.950+0000 7f494759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4948065f90 0x7f4948065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.950+0000 7f494759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49480b13e0 con 0x7f494809ca90 2020-11-20T12:13:43.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.950+0000 7f494759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4948062c10 con 0x7f4948065190 2020-11-20T12:13:43.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.950+0000 7f494759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494814ab30 con 0x7f4948065f90 2020-11-20T12:13:43.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.950+0000 7f4945d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4948065f90 0x7f4948065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.950+0000 7f494659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4948065190 0x7f4948143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44714/0 (socket says 172.21.15.187:44714) 2020-11-20T12:13:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.950+0000 7f494659c700 1 -- 172.21.15.187:0/3368819467 learned_addr learned my addr 172.21.15.187:0/3368819467 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.951+0000 7f4945d9b700 1 -- 172.21.15.187:0/3368819467 >> v1:172.21.15.187:6789/0 conn(0x7f4948065190 legacy=0x7f4948143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.951+0000 7f4945d9b700 1 -- 172.21.15.187:0/3368819467 >> v1:172.21.15.5:6789/0 conn(0x7f494809ca90 legacy=0x7f494809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:43.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.951+0000 7f4945d9b700 1 -- 172.21.15.187:0/3368819467 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49480b1640 con 0x7f4948065f90 2020-11-20T12:13:43.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.951+0000 7f4945d9b700 1 --2- 172.21.15.187:0/3368819467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065f90 0x7f4948065570 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f494814ac70 tx=0x7f493c006df0).ready entity=mon.1 client_cookie=6d3636fe4fd1d956 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:43.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.951+0000 7f494559a700 1 -- 172.21.15.187:0/3368819467 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c009800 con 0x7f4948065f90 2020-11-20T12:13:43.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494559a700 1 -- 172.21.15.187:0/3368819467 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c009960 con 0x7f4948065f90 2020-11-20T12:13:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494559a700 1 -- 172.21.15.187:0/3368819467 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c009ce0 con 0x7f4948065f90 2020-11-20T12:13:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494759e700 1 -- 172.21.15.187:0/3368819467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065f90 msgr2=0x7f4948065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494759e700 1 --2- 172.21.15.187:0/3368819467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065f90 0x7f4948065570 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f494814ac70 tx=0x7f493c006df0).stop 2020-11-20T12:13:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494759e700 1 -- 172.21.15.187:0/3368819467 shutdown_connections 2020-11-20T12:13:43.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494759e700 1 --2- 172.21.15.187:0/3368819467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065f90 0x7f4948065570 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:43.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494759e700 1 -- 172.21.15.187:0/3368819467 >> 172.21.15.187:0/3368819467 conn(0x7f49480bab30 msgr2=0x7f49480baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:43.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494759e700 1 -- 172.21.15.187:0/3368819467 shutdown_connections 2020-11-20T12:13:43.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.952+0000 7f494759e700 1 -- 172.21.15.187:0/3368819467 wait complete. 2020-11-20T12:13:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.954+0000 7f494759e700 1 Processor -- start 2020-11-20T12:13:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.954+0000 7f494759e700 1 -- start start 2020-11-20T12:13:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.955+0000 7f494759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065190 0x7f4948156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.955+0000 7f494759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4948065f90 0x7f494809fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f494809ca90 0x7f49480b9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49480a02e0 con 0x7f4948065f90 2020-11-20T12:13:43.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4948156410 con 0x7f4948065190 2020-11-20T12:13:43.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49481566b0 con 0x7f494809ca90 2020-11-20T12:13:43.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065190 0x7f4948156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065190 0x7f4948156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48114/0 (socket says 172.21.15.187:48114) 2020-11-20T12:13:43.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 -- 172.21.15.187:0/2243166310 learned_addr learned my addr 172.21.15.187:0/2243166310 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:43.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 -- 172.21.15.187:0/2243166310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f494809ca90 msgr2=0x7f49480b9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f494809ca90 0x7f49480b9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:43.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 -- 172.21.15.187:0/2243166310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4948065f90 msgr2=0x7f494809fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:43.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4948065f90 0x7f494809fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:43.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 -- 172.21.15.187:0/2243166310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4948156950 con 0x7f4948065190 2020-11-20T12:13:43.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.956+0000 7f494659c700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065190 0x7f4948156df0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f493800c740 tx=0x7f4938007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:43.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.957+0000 7f49367fc700 1 -- 172.21.15.187:0/2243166310 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938007320 con 0x7f4948065190 2020-11-20T12:13:43.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.957+0000 7f49367fc700 1 -- 172.21.15.187:0/2243166310 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4938007480 con 0x7f4948065190 2020-11-20T12:13:43.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.957+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49480ba410 con 0x7f4948065190 2020-11-20T12:13:43.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.957+0000 7f49367fc700 1 -- 172.21.15.187:0/2243166310 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938005800 con 0x7f4948065190 2020-11-20T12:13:43.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.957+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4948149430 con 0x7f4948065190 2020-11-20T12:13:43.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.958+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4948156ae0 con 0x7f4948065190 2020-11-20T12:13:43.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.959+0000 7f49367fc700 1 -- 172.21.15.187:0/2243166310 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4938010030 con 0x7f4948065190 2020-11-20T12:13:43.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.959+0000 7f49367fc700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f491c056d20 0x7f491c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:43.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.959+0000 7f49367fc700 1 -- 172.21.15.187:0/2243166310 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f493806fd10 con 0x7f4948065190 2020-11-20T12:13:43.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.960+0000 7f4945d9b700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f491c056d20 0x7f491c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:43.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.960+0000 7f4945d9b700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f491c056d20 0x7f491c0591e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f494814ac70 tx=0x7f493c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:43.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:43.963+0000 7f49367fc700 1 -- 172.21.15.187:0/2243166310 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4938075020 con 0x7f4948065190 2020-11-20T12:13:44.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.151+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4948061a20 con 0x7f4948065190 2020-11-20T12:13:44.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.153+0000 7f49367fc700 1 -- 172.21.15.187:0/2243166310 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f4938045050 con 0x7f4948065190 2020-11-20T12:13:44.157 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:44.158 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":89,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:13:41.257868+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:13:38.491662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2020-11-20T12:13:39.231237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:13:44.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f491c056d20 msgr2=0x7f491c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f491c056d20 0x7f491c0591e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f494814ac70 tx=0x7f493c015000).stop 2020-11-20T12:13:44.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065190 msgr2=0x7f4948156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065190 0x7f4948156df0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f493800c740 tx=0x7f4938007c00).stop 2020-11-20T12:13:44.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 shutdown_connections 2020-11-20T12:13:44.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f491c056d20 0x7f491c0591e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f494809ca90 0x7f49480b9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:44.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4948065190 0x7f4948156df0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 --2- 172.21.15.187:0/2243166310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4948065f90 0x7f494809fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:44.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.155+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 >> 172.21.15.187:0/2243166310 conn(0x7f49480bab30 msgr2=0x7f49481497a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:44.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.156+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 shutdown_connections 2020-11-20T12:13:44.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.156+0000 7f494759e700 1 -- 172.21.15.187:0/2243166310 wait complete. 2020-11-20T12:13:44.182 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:13:44.405 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:13:44.401+0000 7f912e67b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:13:44.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.432+0000 7f2f3956c700 1 Processor -- start 2020-11-20T12:13:44.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.432+0000 7f2f3956c700 1 -- start start 2020-11-20T12:13:44.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.433+0000 7f2f3956c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f34065190 0x7f2f34065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.433+0000 7f2f3956c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f340b13e0 con 0x7f2f34065f90 2020-11-20T12:13:44.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.433+0000 7f2f3956c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f34062c10 con 0x7f2f3409ca90 2020-11-20T12:13:44.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.433+0000 7f2f3956c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3414ab30 con 0x7f2f34065190 2020-11-20T12:13:44.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.434+0000 7f2f38d6b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2f3409ca90 0x7f2f3409cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44730/0 (socket says 172.21.15.187:44730) 2020-11-20T12:13:44.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.434+0000 7f2f38d6b700 1 -- 172.21.15.187:0/3151869231 learned_addr learned my addr 172.21.15.187:0/3151869231 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:44.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.434+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124691915 0 0) 0x7f2f34062c10 con 0x7f2f3409ca90 2020-11-20T12:13:44.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.434+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f20003240 con 0x7f2f3409ca90 2020-11-20T12:13:44.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.434+0000 7f2f33fff700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f34065190 0x7f2f34065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 919713656 0 0) 0x7f2f20003240 con 0x7f2f3409ca90 2020-11-20T12:13:44.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f34065190 msgr2=0x7f2f34065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:44.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f34065190 0x7f2f34065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 >> v1:172.21.15.5:6789/0 conn(0x7f2f34065f90 legacy=0x7f2f34143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f340b1640 con 0x7f2f3409ca90 2020-11-20T12:13:44.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f2f2c001860 con 0x7f2f3409ca90 2020-11-20T12:13:44.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 >> v1:172.21.15.187:6789/0 conn(0x7f2f3409ca90 legacy=0x7f2f3409cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f20003730 0x7f2f20003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f34065190 0x7f2f200044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f200049d0 0x7f2f20004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3414ab30 con 0x7f2f200049d0 2020-11-20T12:13:44.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f20004220 con 0x7f2f20003730 2020-11-20T12:13:44.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.435+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2000b730 con 0x7f2f34065190 2020-11-20T12:13:44.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.436+0000 7f2f33fff700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f20003730 0x7f2f20003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.436+0000 7f2f38d6b700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f200049d0 0x7f2f20004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.436+0000 7f2f337fe700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f34065190 0x7f2f200044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f33fff700 1 -- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f34065190 msgr2=0x7f2f200044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:44.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f33fff700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f34065190 0x7f2f200044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f33fff700 1 -- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f200049d0 msgr2=0x7f2f20004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:44.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f33fff700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f200049d0 0x7f2f20004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f33fff700 1 -- 172.21.15.187:0/3151869231 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20003240 con 0x7f2f20003730 2020-11-20T12:13:44.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f33fff700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f20003730 0x7f2f20003b50 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f2f2400bab0 tx=0x7f2f24008040).ready entity=mon.1 client_cookie=ca240ee3e20e59be server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:44.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400d8f0 con 0x7f2f20003730 2020-11-20T12:13:44.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2400da50 con 0x7f2f20003730 2020-11-20T12:13:44.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.437+0000 7f2f32ffd700 1 -- 172.21.15.187:0/3151869231 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400ddd0 con 0x7f2f20003730 2020-11-20T12:13:44.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.438+0000 7f2f3956c700 1 -- 172.21.15.187:0/3151869231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f20003730 msgr2=0x7f2f20003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.438+0000 7f2f3956c700 1 --2- 172.21.15.187:0/3151869231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f20003730 0x7f2f20003b50 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f2f2400bab0 tx=0x7f2f24008040).stop 2020-11-20T12:13:44.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.438+0000 7f2f38d6b700 1 -- 172.21.15.187:0/3151869231 reap_dead start 2020-11-20T12:13:44.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.438+0000 7f2f3956c700 1 -- 172.21.15.187:0/3151869231 shutdown_connections 2020-11-20T12:13:44.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.438+0000 7f2f3956c700 1 -- 172.21.15.187:0/3151869231 >> 172.21.15.187:0/3151869231 conn(0x7f2f340bab30 msgr2=0x7f2f340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:44.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.438+0000 7f2f3956c700 1 -- 172.21.15.187:0/3151869231 shutdown_connections 2020-11-20T12:13:44.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.438+0000 7f2f3956c700 1 -- 172.21.15.187:0/3151869231 wait complete. 2020-11-20T12:13:44.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.440+0000 7f2f3956c700 1 Processor -- start 2020-11-20T12:13:44.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.440+0000 7f2f3956c700 1 -- start start 2020-11-20T12:13:44.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f3956c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f340b3800 0x7f2f340b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f3956c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f340b3be0 0x7f2f34144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f3956c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f34145310 0x7f2f340b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f3956c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340b32b0 con 0x7f2f340b3800 2020-11-20T12:13:44.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f3956c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3409f8a0 con 0x7f2f34145310 2020-11-20T12:13:44.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f3956c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3409fb40 con 0x7f2f340b3be0 2020-11-20T12:13:44.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f38d6b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f34145310 0x7f2f340b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f38d6b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f34145310 0x7f2f340b1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48138/0 (socket says 172.21.15.187:48138) 2020-11-20T12:13:44.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f38d6b700 1 -- 172.21.15.187:0/1110526834 learned_addr learned my addr 172.21.15.187:0/1110526834 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:44.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f38d6b700 1 -- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f340b3be0 msgr2=0x7f2f34144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f33fff700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f340b3800 0x7f2f340b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f337fe700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f340b3be0 0x7f2f34144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.441+0000 7f2f38d6b700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f340b3be0 0x7f2f34144e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f38d6b700 1 -- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f340b3800 msgr2=0x7f2f340b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f38d6b700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f340b3800 0x7f2f340b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f38d6b700 1 -- 172.21.15.187:0/1110526834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3409fde0 con 0x7f2f34145310 2020-11-20T12:13:44.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f33fff700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f340b3800 0x7f2f340b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:44.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f38d6b700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f34145310 0x7f2f340b1b80 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f2f34066440 tx=0x7f2f340bb650).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:44.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f13fff700 1 -- 172.21.15.187:0/1110526834 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c00fd40 con 0x7f2f34145310 2020-11-20T12:13:44.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f13fff700 1 -- 172.21.15.187:0/1110526834 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c00fea0 con 0x7f2f34145310 2020-11-20T12:13:44.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f340a0280 con 0x7f2f34145310 2020-11-20T12:13:44.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f13fff700 1 -- 172.21.15.187:0/1110526834 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c015640 con 0x7f2f34145310 2020-11-20T12:13:44.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.442+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f340b4180 con 0x7f2f34145310 2020-11-20T12:13:44.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.443+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3409ff70 con 0x7f2f34145310 2020-11-20T12:13:44.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.444+0000 7f2f13fff700 1 -- 172.21.15.187:0/1110526834 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2f2c0159b0 con 0x7f2f34145310 2020-11-20T12:13:44.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.444+0000 7f2f13fff700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f0c056d20 0x7f2f0c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.445+0000 7f2f13fff700 1 -- 172.21.15.187:0/1110526834 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2f2c06ff30 con 0x7f2f34145310 2020-11-20T12:13:44.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.445+0000 7f2f33fff700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f0c056d20 0x7f2f0c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.445+0000 7f2f33fff700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f0c056d20 0x7f2f0c0591e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f2f3405f730 tx=0x7f2f24008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:44.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.448+0000 7f2f13fff700 1 -- 172.21.15.187:0/1110526834 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2f2c045320 con 0x7f2f34145310 2020-11-20T12:13:44.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.703+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f2f34061a20 con 0x7f2f34145310 2020-11-20T12:13:44.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.713+0000 7f2f13fff700 1 -- 172.21.15.187:0/1110526834 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2f2c01d030 con 0x7f2f34145310 2020-11-20T12:13:44.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f0c056d20 msgr2=0x7f2f0c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f0c056d20 0x7f2f0c0591e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f2f3405f730 tx=0x7f2f24008040).stop 2020-11-20T12:13:44.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f34145310 msgr2=0x7f2f340b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f34145310 0x7f2f340b1b80 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f2f34066440 tx=0x7f2f340bb650).stop 2020-11-20T12:13:44.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 shutdown_connections 2020-11-20T12:13:44.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f0c056d20 0x7f2f0c0591e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f340b3800 0x7f2f340b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f34145310 0x7f2f340b1b80 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 --2- 172.21.15.187:0/1110526834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f340b3be0 0x7f2f34144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 >> 172.21.15.187:0/1110526834 conn(0x7f2f340bab30 msgr2=0x7f2f3409aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:44.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.716+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 shutdown_connections 2020-11-20T12:13:44.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.717+0000 7f2f3956c700 1 -- 172.21.15.187:0/1110526834 wait complete. 2020-11-20T12:13:44.723 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:13:44.744 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:44.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.983+0000 7f5a0122b700 1 Processor -- start 2020-11-20T12:13:44.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.984+0000 7f5a0122b700 1 -- start start 2020-11-20T12:13:44.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.985+0000 7f5a0122b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0bf800 0x7f59fc14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.985+0000 7f5a0122b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59fc09f560 0x7f59fc0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.985+0000 7f5a0122b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc14d110 con 0x7f59fc0bfbe0 2020-11-20T12:13:44.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.985+0000 7f5a0122b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc14cbd0 con 0x7f59fc09f560 2020-11-20T12:13:44.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.985+0000 7f5a0122b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc14ce70 con 0x7f59fc0bf800 2020-11-20T12:13:44.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.985+0000 7f59fb7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f59fc0bfbe0 0x7f59fc1480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44748/0 (socket says 172.21.15.187:44748) 2020-11-20T12:13:44.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.986+0000 7f59fb7fe700 1 -- 172.21.15.187:0/2602108215 learned_addr learned my addr 172.21.15.187:0/2602108215 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:44.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.986+0000 7f59fbfff700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0bf800 0x7f59fc14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.986+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613468618 0 0) 0x7f59fc14d110 con 0x7f59fc0bfbe0 2020-11-20T12:13:44.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.986+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e8003240 con 0x7f59fc0bfbe0 2020-11-20T12:13:44.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.986+0000 7f5a00a2a700 1 --2- 172.21.15.187:0/2602108215 >> v2:172.21.15.187:3300/0 conn(0x7f59fc09f560 0x7f59fc0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:44.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.986+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 368787485 0 0) 0x7f59e8003240 con 0x7f59fc0bfbe0 2020-11-20T12:13:44.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0bf800 msgr2=0x7f59fc14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:44.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0bf800 0x7f59fc14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 >> v2:172.21.15.187:3300/0 conn(0x7f59fc09f560 msgr2=0x7f59fc0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:44.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 --2- 172.21.15.187:0/2602108215 >> v2:172.21.15.187:3300/0 conn(0x7f59fc09f560 0x7f59fc0a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:44.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0a4cc0 con 0x7f59fc0bfbe0 2020-11-20T12:13:44.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f59f00014e0 con 0x7f59fc0bfbe0 2020-11-20T12:13:44.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f5a00a2a700 1 --2- 172.21.15.187:0/2602108215 >> v2:172.21.15.187:3300/0 conn(0x7f59fc09f560 0x7f59fc0a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:13:44.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 >> v1:172.21.15.187:6789/0 conn(0x7f59fc0bfbe0 legacy=0x7f59fc1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:44.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0bf800 0x7f59e8003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59e8003c40 0x7f59e80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59e80049d0 0x7f59e8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:44.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc14cbd0 con 0x7f59e80049d0 2020-11-20T12:13:44.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc14ce70 con 0x7f59e8003c40 2020-11-20T12:13:44.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e8004200 con 0x7f59fc0bf800 2020-11-20T12:13:44.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.987+0000 7f59fb7fe700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59e8003c40 0x7f59e80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:45.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59fbfff700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0bf800 0x7f59e8003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:45.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f5a00a2a700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59e80049d0 0x7f59e8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:45.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59fb7fe700 1 -- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0bf800 msgr2=0x7f59e8003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:45.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59fb7fe700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0bf800 0x7f59e8003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:45.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59fb7fe700 1 -- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59e80049d0 msgr2=0x7f59e8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:45.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59fb7fe700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59e80049d0 0x7f59e8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:45.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59fb7fe700 1 -- 172.21.15.187:0/2602108215 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8003240 con 0x7f59e8003c40 2020-11-20T12:13:45.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59fb7fe700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59e8003c40 0x7f59e80044e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f59fc0bcaa0 tx=0x7f59f0007820).ready entity=mon.1 client_cookie=dede9e8ad77e6a47 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:45.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f000ce50 con 0x7f59e8003c40 2020-11-20T12:13:45.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f00072e0 con 0x7f59e8003c40 2020-11-20T12:13:45.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.988+0000 7f59faffd700 1 -- 172.21.15.187:0/2602108215 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0005820 con 0x7f59e8003c40 2020-11-20T12:13:45.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.989+0000 7f5a0122b700 1 -- 172.21.15.187:0/2602108215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59e8003c40 msgr2=0x7f59e80044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:45.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.989+0000 7f5a0122b700 1 --2- 172.21.15.187:0/2602108215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59e8003c40 0x7f59e80044e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f59fc0bcaa0 tx=0x7f59f0007820).stop 2020-11-20T12:13:45.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.989+0000 7f5a00a2a700 1 -- 172.21.15.187:0/2602108215 reap_dead start 2020-11-20T12:13:45.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.989+0000 7f5a0122b700 1 -- 172.21.15.187:0/2602108215 shutdown_connections 2020-11-20T12:13:45.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.989+0000 7f5a0122b700 1 -- 172.21.15.187:0/2602108215 >> 172.21.15.187:0/2602108215 conn(0x7f59fc068370 msgr2=0x7f59fc0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:45.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.989+0000 7f5a0122b700 1 -- 172.21.15.187:0/2602108215 shutdown_connections 2020-11-20T12:13:45.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.989+0000 7f5a0122b700 1 -- 172.21.15.187:0/2602108215 wait complete. 2020-11-20T12:13:45.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.991+0000 7f5a0122b700 1 Processor -- start 2020-11-20T12:13:45.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.991+0000 7f5a0122b700 1 -- start start 2020-11-20T12:13:45.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.992+0000 7f5a0122b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0a37c0 0x7f59fc0b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:45.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.992+0000 7f5a0122b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59fc0a3ba0 0x7f59fc15c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:45.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.992+0000 7f5a0122b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59fc15ca70 0x7f59fc1588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:45.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.992+0000 7f5a0122b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0b48c0 con 0x7f59fc15ca70 2020-11-20T12:13:45.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.992+0000 7f5a0122b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc15be40 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.992+0000 7f5a0122b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc15c0e0 con 0x7f59fc0a37c0 2020-11-20T12:13:45.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59fc0a3ba0 0x7f59fc15c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:45.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59fc0a3ba0 0x7f59fc15c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48156/0 (socket says 172.21.15.187:48156) 2020-11-20T12:13:45.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 -- 172.21.15.187:0/1391109550 learned_addr learned my addr 172.21.15.187:0/1391109550 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:45.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f5a00a2a700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59fc15ca70 0x7f59fc1588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:45.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fbfff700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0a37c0 0x7f59fc0b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:45.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 -- 172.21.15.187:0/1391109550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0a37c0 msgr2=0x7f59fc0b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:45.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0a37c0 0x7f59fc0b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:45.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 -- 172.21.15.187:0/1391109550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59fc15ca70 msgr2=0x7f59fc1588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:45.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59fc15ca70 0x7f59fc1588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:45.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 -- 172.21.15.187:0/1391109550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0b54d0 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.993+0000 7f59fb7fe700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59fc0a3ba0 0x7f59fc15c580 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f59f0007a90 tx=0x7f59f0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:45.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.994+0000 7f59dbfff700 1 -- 172.21.15.187:0/1391109550 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0013450 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.994+0000 7f59dbfff700 1 -- 172.21.15.187:0/1391109550 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f00135b0 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.994+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc0b5720 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.994+0000 7f59dbfff700 1 -- 172.21.15.187:0/1391109550 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0013970 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.994+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc159020 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.995+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc15be40 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.996+0000 7f59dbfff700 1 -- 172.21.15.187:0/1391109550 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f59f0026020 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.996+0000 7f59dbfff700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59d4056d20 0x7f59d40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:45.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.997+0000 7f59dbfff700 1 -- 172.21.15.187:0/1391109550 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f59f00714e0 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.997+0000 7f59fbfff700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59d4056d20 0x7f59d40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:45.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:44.997+0000 7f59fbfff700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59d4056d20 0x7f59d40591e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f59ec006a30 tx=0x7f59ec008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:45.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.000+0000 7f59dbfff700 1 -- 172.21.15.187:0/1391109550 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f59f0046930 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.249+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59fc09ea10 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.250+0000 7f59dbfff700 1 -- 172.21.15.187:0/1391109550 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f59f0049f50 con 0x7f59fc0a3ba0 2020-11-20T12:13:45.254 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:45.254 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:13:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59d4056d20 msgr2=0x7f59d40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59d4056d20 0x7f59d40591e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f59ec006a30 tx=0x7f59ec008040).stop 2020-11-20T12:13:45.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59fc0a3ba0 msgr2=0x7f59fc15c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:45.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59fc0a3ba0 0x7f59fc15c580 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f59f0007a90 tx=0x7f59f0015000).stop 2020-11-20T12:13:45.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 shutdown_connections 2020-11-20T12:13:45.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59d4056d20 0x7f59d40591e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59fc15ca70 0x7f59fc1588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59fc0a37c0 0x7f59fc0b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 --2- 172.21.15.187:0/1391109550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59fc0a3ba0 0x7f59fc15c580 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.252+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 >> 172.21.15.187:0/1391109550 conn(0x7f59fc068370 msgr2=0x7f59fc060dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.253+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 shutdown_connections 2020-11-20T12:13:45.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:45.253+0000 7f5a0122b700 1 -- 172.21.15.187:0/1391109550 wait complete. 2020-11-20T12:13:45.259 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:13:45.479 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:13:40.253243+0000', 'modified': '2020-11-20T12:13:42.397341+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7408}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7408': {'gid': 7408, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1880661915', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1880661915}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1880661915}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:45.480 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:13:45.480 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:13:45.749 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":7408,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":5.4705229449999999,"mdsmap_epoch":80,"osdmap_epoch":89,"osdmap_epoch_barrier":89,"uptime":11.121520362} 2020-11-20T12:13:45.750 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 7408, "mdsmap_epoch": 80, "osdmap_epoch": 89, "osdmap_epoch_barrier": 89, "rank_uptime": 5.470522945, "state": "up:active", "uptime": 11.121520362, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:13:45.750 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:13:45.750 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:13:45.751 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:13:45.751 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:13:45.751 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:13:45.752 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:13:45.752 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:13:45.752 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:13:45.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:13:45.753 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:13:45.812 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:13:46.159 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:13:46.162 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:13:46.163 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:13:46.163 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:13:46.164 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:13:46.167 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:13:46.167 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:13:46.168 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:13:46.169 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:13:46.169 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:13:46.170 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:13:46.171 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:13:46.172 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:13:46.173 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:13:46.175 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42213sec preferred_lft 42213sec 2020-11-20T12:13:46.175 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:13:46.176 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:13:46.178 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:13:46.179 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:13:46.179 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:13:46.180 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:13:46.181 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:13:46.182 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:13:46.183 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:13:46.183 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:13:46.184 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:13:46.185 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:13:46.185 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:13:46.185 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:13:46.186 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:13:46.186 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:13:46.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:13:46.187 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:13:46.310 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:13:46.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:13:46.315 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:13:46.373 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:13:46.373 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:13:46.374 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:13:46.374 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:13:46.374 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:13:46.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:13:46.375 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:13:46.375 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:13:46.376 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:13:46.376 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:13:46.377 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:13:46.377 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:13:46.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:13:46.631 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:13:46.681 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:13:46.682 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:13:46.683 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:13:46.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:13:46.684 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:13:46.775 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:13:47.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.014+0000 7feb30ef0700 1 Processor -- start 2020-11-20T12:13:47.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.014+0000 7feb30ef0700 1 -- start start 2020-11-20T12:13:47.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.016+0000 7feb30ef0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feb2c1484d0 0x7feb2c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.016+0000 7feb30ef0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c0b1300 con 0x7feb2c066f70 2020-11-20T12:13:47.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.016+0000 7feb30ef0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c0b15a0 con 0x7feb2c0622c0 2020-11-20T12:13:47.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.016+0000 7feb30ef0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c067730 con 0x7feb2c1484d0 2020-11-20T12:13:47.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.016+0000 7feb22ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7feb2c0622c0 0x7feb2c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44768/0 (socket says 172.21.15.187:44768) 2020-11-20T12:13:47.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.016+0000 7feb22ffd700 1 -- 172.21.15.187:0/3395016214 learned_addr learned my addr 172.21.15.187:0/3395016214 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:47.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.017+0000 7feb2b7fe700 1 --2- 172.21.15.187:0/3395016214 >> v2:172.21.15.187:3300/0 conn(0x7feb2c1484d0 0x7feb2c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.017+0000 7feb2affd700 1 -- 172.21.15.187:0/3395016214 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982847934 0 0) 0x7feb2c0b15a0 con 0x7feb2c0622c0 2020-11-20T12:13:47.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.017+0000 7feb2affd700 1 -- 172.21.15.187:0/3395016214 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb10003240 con 0x7feb2c0622c0 2020-11-20T12:13:47.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.017+0000 7feb2b7fe700 1 -- 172.21.15.187:0/3395016214 >> v1:172.21.15.187:6789/0 conn(0x7feb2c0622c0 legacy=0x7feb2c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.017+0000 7feb2b7fe700 1 -- 172.21.15.187:0/3395016214 >> v1:172.21.15.5:6789/0 conn(0x7feb2c066f70 legacy=0x7feb2c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:47.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.017+0000 7feb2b7fe700 1 -- 172.21.15.187:0/3395016214 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0b1840 con 0x7feb2c1484d0 2020-11-20T12:13:47.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.018+0000 7feb2b7fe700 1 --2- 172.21.15.187:0/3395016214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c1484d0 0x7feb2c061dd0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7feb1c00b9a0 tx=0x7feb1c006df0).ready entity=mon.1 client_cookie=69c8f8d86b706cd6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.018+0000 7feb2affd700 1 -- 172.21.15.187:0/3395016214 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0097b0 con 0x7feb2c1484d0 2020-11-20T12:13:47.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.018+0000 7feb2affd700 1 -- 172.21.15.187:0/3395016214 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c009910 con 0x7feb2c1484d0 2020-11-20T12:13:47.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.018+0000 7feb2affd700 1 -- 172.21.15.187:0/3395016214 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c009c90 con 0x7feb2c1484d0 2020-11-20T12:13:47.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.018+0000 7feb30ef0700 1 -- 172.21.15.187:0/3395016214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c1484d0 msgr2=0x7feb2c061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.018+0000 7feb30ef0700 1 --2- 172.21.15.187:0/3395016214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c1484d0 0x7feb2c061dd0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7feb1c00b9a0 tx=0x7feb1c006df0).stop 2020-11-20T12:13:47.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.019+0000 7feb30ef0700 1 -- 172.21.15.187:0/3395016214 shutdown_connections 2020-11-20T12:13:47.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.019+0000 7feb30ef0700 1 --2- 172.21.15.187:0/3395016214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c1484d0 0x7feb2c061dd0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.019+0000 7feb30ef0700 1 -- 172.21.15.187:0/3395016214 >> 172.21.15.187:0/3395016214 conn(0x7feb2c0bab30 msgr2=0x7feb2c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:47.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.019+0000 7feb30ef0700 1 -- 172.21.15.187:0/3395016214 shutdown_connections 2020-11-20T12:13:47.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.019+0000 7feb30ef0700 1 -- 172.21.15.187:0/3395016214 wait complete. 2020-11-20T12:13:47.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.020+0000 7feb30ef0700 1 Processor -- start 2020-11-20T12:13:47.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.020+0000 7feb30ef0700 1 -- start start 2020-11-20T12:13:47.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.021+0000 7feb30ef0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb2c0622c0 0x7feb2c09dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.021+0000 7feb30ef0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c066f70 0x7feb2c134e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.021+0000 7feb30ef0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb2c1484d0 0x7feb2c135e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.021+0000 7feb30ef0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c09d580 con 0x7feb2c1484d0 2020-11-20T12:13:47.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.021+0000 7feb30ef0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c09dfd0 con 0x7feb2c066f70 2020-11-20T12:13:47.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.021+0000 7feb30ef0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c1369e0 con 0x7feb2c0622c0 2020-11-20T12:13:47.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c066f70 0x7feb2c134e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c066f70 0x7feb2c134e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48170/0 (socket says 172.21.15.187:48170) 2020-11-20T12:13:47.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 -- 172.21.15.187:0/3657594030 learned_addr learned my addr 172.21.15.187:0/3657594030 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:47.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 -- 172.21.15.187:0/3657594030 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb2c0622c0 msgr2=0x7feb2c09dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb2c0622c0 0x7feb2c09dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:47.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 -- 172.21.15.187:0/3657594030 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb2c1484d0 msgr2=0x7feb2c135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb2bfff700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb2c1484d0 0x7feb2c135e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb2c1484d0 0x7feb2c135e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 -- 172.21.15.187:0/3657594030 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c136c80 con 0x7feb2c066f70 2020-11-20T12:13:47.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.022+0000 7feb22ffd700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c066f70 0x7feb2c134e70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7feb1400c710 tx=0x7feb14007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.023+0000 7feb23fff700 1 -- 172.21.15.187:0/3657594030 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14007320 con 0x7feb2c066f70 2020-11-20T12:13:47.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.023+0000 7feb23fff700 1 -- 172.21.15.187:0/3657594030 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb14007480 con 0x7feb2c066f70 2020-11-20T12:13:47.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.023+0000 7feb23fff700 1 -- 172.21.15.187:0/3657594030 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14005800 con 0x7feb2c066f70 2020-11-20T12:13:47.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.023+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c136f20 con 0x7feb2c066f70 2020-11-20T12:13:47.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.023+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c13e3e0 con 0x7feb2c066f70 2020-11-20T12:13:47.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.024+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c13e6c0 con 0x7feb2c066f70 2020-11-20T12:13:47.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.024+0000 7feb23fff700 1 -- 172.21.15.187:0/3657594030 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feb14010030 con 0x7feb2c066f70 2020-11-20T12:13:47.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.025+0000 7feb23fff700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feafc056d20 0x7feafc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.025+0000 7feb23fff700 1 -- 172.21.15.187:0/3657594030 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feb1406fdd0 con 0x7feb2c066f70 2020-11-20T12:13:47.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.025+0000 7feb2b7fe700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feafc056d20 0x7feafc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.026+0000 7feb2b7fe700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feafc056d20 0x7feafc0591e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7feb1c00b9a0 tx=0x7feb1c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.029+0000 7feb23fff700 1 -- 172.21.15.187:0/3657594030 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feb14045110 con 0x7feb2c066f70 2020-11-20T12:13:47.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.215+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb2c061a20 con 0x7feb2c066f70 2020-11-20T12:13:47.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.217+0000 7feb23fff700 1 -- 172.21.15.187:0/3657594030 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16781 (secure 0 0 0) 0x7feb14048730 con 0x7feb2c066f70 2020-11-20T12:13:47.222 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:47.222 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":89,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:13:41.257868+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:13:38.491662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2020-11-20T12:13:39.231237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:13:47.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.219+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feafc056d20 msgr2=0x7feafc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.219+0000 7feb30ef0700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feafc056d20 0x7feafc0591e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7feb1c00b9a0 tx=0x7feb1c015000).stop 2020-11-20T12:13:47.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c066f70 msgr2=0x7feb2c134e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c066f70 0x7feb2c134e70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7feb1400c710 tx=0x7feb14007aa0).stop 2020-11-20T12:13:47.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 shutdown_connections 2020-11-20T12:13:47.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feafc056d20 0x7feafc0591e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb2c1484d0 0x7feb2c135e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb2c0622c0 0x7feb2c09dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:47.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 --2- 172.21.15.187:0/3657594030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb2c066f70 0x7feb2c134e70 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 >> 172.21.15.187:0/3657594030 conn(0x7feb2c0bab30 msgr2=0x7feb2c0b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:47.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 shutdown_connections 2020-11-20T12:13:47.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.220+0000 7feb30ef0700 1 -- 172.21.15.187:0/3657594030 wait complete. 2020-11-20T12:13:47.246 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:13:38.491662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:13:39.231237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:13:47.247 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:13:47.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.490+0000 7f655359e700 1 Processor -- start 2020-11-20T12:13:47.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.490+0000 7f655359e700 1 -- start start 2020-11-20T12:13:47.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.491+0000 7f655359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f65540b1760 0x7f6554065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.491+0000 7f655359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6554065680 0x7f655414c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.491+0000 7f655359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554150390 con 0x7f65540b1380 2020-11-20T12:13:47.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.491+0000 7f655359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65541511e0 con 0x7f6554065680 2020-11-20T12:13:47.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.491+0000 7f655359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655414d760 con 0x7f65540b1760 2020-11-20T12:13:47.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.492+0000 7f655259c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f65540b1380 0x7f6554150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44782/0 (socket says 172.21.15.187:44782) 2020-11-20T12:13:47.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.492+0000 7f655259c700 1 -- 172.21.15.187:0/66787277 learned_addr learned my addr 172.21.15.187:0/66787277 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:47.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.492+0000 7f6549d9b700 1 --2- 172.21.15.187:0/66787277 >> v2:172.21.15.187:3300/0 conn(0x7f65540b1760 0x7f6554065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.492+0000 7f6552d9d700 1 --2- 172.21.15.187:0/66787277 >> v2:172.21.15.5:3300/0 conn(0x7f6554065680 0x7f655414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.492+0000 7f6551d9b700 1 -- 172.21.15.187:0/66787277 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539983666 0 0) 0x7f6554150390 con 0x7f65540b1380 2020-11-20T12:13:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.492+0000 7f6551d9b700 1 -- 172.21.15.187:0/66787277 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6534003240 con 0x7f65540b1380 2020-11-20T12:13:47.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.493+0000 7f6549d9b700 1 -- 172.21.15.187:0/66787277 >> v2:172.21.15.5:3300/0 conn(0x7f6554065680 msgr2=0x7f655414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:47.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.493+0000 7f6551d9b700 1 -- 172.21.15.187:0/66787277 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 266348708 0 0) 0x7f6534003240 con 0x7f65540b1380 2020-11-20T12:13:47.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.493+0000 7f6549d9b700 1 --2- 172.21.15.187:0/66787277 >> v2:172.21.15.5:3300/0 conn(0x7f6554065680 0x7f655414c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.493+0000 7f6549d9b700 1 -- 172.21.15.187:0/66787277 >> v1:172.21.15.187:6789/0 conn(0x7f65540b1380 legacy=0x7f6554150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.493+0000 7f6549d9b700 1 -- 172.21.15.187:0/66787277 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554150630 con 0x7f65540b1760 2020-11-20T12:13:47.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.493+0000 7f6552d9d700 1 --2- 172.21.15.187:0/66787277 >> v2:172.21.15.5:3300/0 conn(0x7f6554065680 0x7f655414c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:47.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.493+0000 7f6549d9b700 1 --2- 172.21.15.187:0/66787277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1760 0x7f6554065190 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7f6554063a20 tx=0x7f653c006df0).ready entity=mon.1 client_cookie=613b9a861b4ee8b7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.494+0000 7f6551d9b700 1 -- 172.21.15.187:0/66787277 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65440014e0 con 0x7f65540b1760 2020-11-20T12:13:47.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.494+0000 7f6551d9b700 1 -- 172.21.15.187:0/66787277 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c0094a0 con 0x7f65540b1760 2020-11-20T12:13:47.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.494+0000 7f6551d9b700 1 -- 172.21.15.187:0/66787277 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c009820 con 0x7f65540b1760 2020-11-20T12:13:47.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.494+0000 7f655359e700 1 -- 172.21.15.187:0/66787277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1760 msgr2=0x7f6554065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.494+0000 7f655359e700 1 --2- 172.21.15.187:0/66787277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1760 0x7f6554065190 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7f6554063a20 tx=0x7f653c006df0).stop 2020-11-20T12:13:47.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.495+0000 7f655359e700 1 -- 172.21.15.187:0/66787277 shutdown_connections 2020-11-20T12:13:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.495+0000 7f655359e700 1 --2- 172.21.15.187:0/66787277 >> v2:172.21.15.5:3300/0 conn(0x7f6554065680 0x7f655414c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.495+0000 7f655359e700 1 --2- 172.21.15.187:0/66787277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1760 0x7f6554065190 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.495+0000 7f655359e700 1 -- 172.21.15.187:0/66787277 >> 172.21.15.187:0/66787277 conn(0x7f65540bab30 msgr2=0x7f65540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.495+0000 7f655359e700 1 -- 172.21.15.187:0/66787277 shutdown_connections 2020-11-20T12:13:47.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.495+0000 7f655359e700 1 -- 172.21.15.187:0/66787277 wait complete. 2020-11-20T12:13:47.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.497+0000 7f655359e700 1 Processor -- start 2020-11-20T12:13:47.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.497+0000 7f655359e700 1 -- start start 2020-11-20T12:13:47.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.498+0000 7f655359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1380 0x7f65540b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.498+0000 7f655359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65540b0530 0x7f655409aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.498+0000 7f655359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65540b4100 0x7f655409af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.498+0000 7f655359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655409e8f0 con 0x7f65540b4100 2020-11-20T12:13:47.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.498+0000 7f655359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655409eb90 con 0x7f65540b1380 2020-11-20T12:13:47.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.498+0000 7f655359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655409e570 con 0x7f65540b0530 2020-11-20T12:13:47.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.498+0000 7f655259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1380 0x7f65540b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.498+0000 7f655259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1380 0x7f65540b0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48182/0 (socket says 172.21.15.187:48182) 2020-11-20T12:13:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f655259c700 1 -- 172.21.15.187:0/955422538 learned_addr learned my addr 172.21.15.187:0/955422538 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f6552d9d700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65540b4100 0x7f655409af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f6549d9b700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65540b0530 0x7f655409aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f655259c700 1 -- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65540b0530 msgr2=0x7f655409aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f655259c700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65540b0530 0x7f655409aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f655259c700 1 -- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65540b4100 msgr2=0x7f655409af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f655259c700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65540b4100 0x7f655409af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f655259c700 1 -- 172.21.15.187:0/955422538 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65540a07f0 con 0x7f65540b1380 2020-11-20T12:13:47.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f6549d9b700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65540b0530 0x7f655409aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:47.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.499+0000 7f655259c700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1380 0x7f65540b0040 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f654400cae0 tx=0x7f6544001a70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.500+0000 7f654affd700 1 -- 172.21.15.187:0/955422538 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544007170 con 0x7f65540b1380 2020-11-20T12:13:47.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.500+0000 7f654affd700 1 -- 172.21.15.187:0/955422538 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65440072d0 con 0x7f65540b1380 2020-11-20T12:13:47.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.500+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65540a0980 con 0x7f65540b1380 2020-11-20T12:13:47.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.500+0000 7f654affd700 1 -- 172.21.15.187:0/955422538 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654400aaf0 con 0x7f65540b1380 2020-11-20T12:13:47.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.500+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655409df40 con 0x7f65540b1380 2020-11-20T12:13:47.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.501+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655409ed00 con 0x7f65540b1380 2020-11-20T12:13:47.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.502+0000 7f654affd700 1 -- 172.21.15.187:0/955422538 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6544010070 con 0x7f65540b1380 2020-11-20T12:13:47.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.502+0000 7f654affd700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6530056d20 0x7f65300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.503+0000 7f654affd700 1 -- 172.21.15.187:0/955422538 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f654406fb70 con 0x7f65540b1380 2020-11-20T12:13:47.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.503+0000 7f6549d9b700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6530056d20 0x7f65300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.503+0000 7f6549d9b700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6530056d20 0x7f65300591e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f6554150170 tx=0x7f653c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.506+0000 7f654affd700 1 -- 172.21.15.187:0/955422538 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f655409ed00 con 0x7f65540b1380 2020-11-20T12:13:47.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.694+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f655409ecd0 con 0x7f65540b1380 2020-11-20T12:13:47.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.695+0000 7f654affd700 1 -- 172.21.15.187:0/955422538 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v89) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6544044fc0 con 0x7f65540b1380 2020-11-20T12:13:47.699 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:13:47.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6530056d20 msgr2=0x7f65300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6530056d20 0x7f65300591e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f6554150170 tx=0x7f653c015000).stop 2020-11-20T12:13:47.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1380 msgr2=0x7f65540b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1380 0x7f65540b0040 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f654400cae0 tx=0x7f6544001a70).stop 2020-11-20T12:13:47.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 shutdown_connections 2020-11-20T12:13:47.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6530056d20 0x7f65300591e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65540b4100 0x7f655409af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65540b1380 0x7f65540b0040 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 --2- 172.21.15.187:0/955422538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65540b0530 0x7f655409aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 >> 172.21.15.187:0/955422538 conn(0x7f65540bab30 msgr2=0x7f6554151680 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:47.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.697+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 shutdown_connections 2020-11-20T12:13:47.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.698+0000 7f655359e700 1 -- 172.21.15.187:0/955422538 wait complete. 2020-11-20T12:13:47.724 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:13:47.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.971+0000 7f6fd359e700 1 Processor -- start 2020-11-20T12:13:47.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.972+0000 7f6fd359e700 1 -- start start 2020-11-20T12:13:47.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.973+0000 7f6fd359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6fd40b6650 0x7f6fd40afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.973+0000 7f6fd359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6fd4066f70 0x7f6fd40aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.973+0000 7f6fd359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40b1300 con 0x7f6fd41484d0 2020-11-20T12:13:47.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.973+0000 7f6fd359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd4067730 con 0x7f6fd40b6650 2020-11-20T12:13:47.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.973+0000 7f6fd359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd40b15a0 con 0x7f6fd4066f70 2020-11-20T12:13:47.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.974+0000 7f6fd259c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6fd41484d0 0x7f6fd4061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44798/0 (socket says 172.21.15.187:44798) 2020-11-20T12:13:47.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.974+0000 7f6fd259c700 1 -- 172.21.15.187:0/1036334408 learned_addr learned my addr 172.21.15.187:0/1036334408 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:47.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.974+0000 7f6fd2d9d700 1 --2- 172.21.15.187:0/1036334408 >> v2:172.21.15.187:3300/0 conn(0x7f6fd4066f70 0x7f6fd40aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.974+0000 7f6fcbfff700 1 --2- 172.21.15.187:0/1036334408 >> v2:172.21.15.5:3300/0 conn(0x7f6fd40b6650 0x7f6fd40afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.974+0000 7f6fd1d9b700 1 -- 172.21.15.187:0/1036334408 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883728241 0 0) 0x7f6fd40b1300 con 0x7f6fd41484d0 2020-11-20T12:13:47.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.974+0000 7f6fd1d9b700 1 -- 172.21.15.187:0/1036334408 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb4003240 con 0x7f6fd41484d0 2020-11-20T12:13:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.974+0000 7f6fd2d9d700 1 -- 172.21.15.187:0/1036334408 >> v2:172.21.15.5:3300/0 conn(0x7f6fd40b6650 msgr2=0x7f6fd40afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.974+0000 7f6fd2d9d700 1 --2- 172.21.15.187:0/1036334408 >> v2:172.21.15.5:3300/0 conn(0x7f6fd40b6650 0x7f6fd40afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.975+0000 7f6fd2d9d700 1 -- 172.21.15.187:0/1036334408 >> v1:172.21.15.187:6789/0 conn(0x7f6fd41484d0 legacy=0x7f6fd4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.975+0000 7f6fd2d9d700 1 -- 172.21.15.187:0/1036334408 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40b1840 con 0x7f6fd4066f70 2020-11-20T12:13:47.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.975+0000 7f6fd2d9d700 1 --2- 172.21.15.187:0/1036334408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd40aff80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7f6fd4045000 tx=0x7f6fcc0091d0).ready entity=mon.1 client_cookie=42ba3aff96071766 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.975+0000 7f6fd1d9b700 1 -- 172.21.15.187:0/1036334408 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00b9c0 con 0x7f6fd4066f70 2020-11-20T12:13:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.975+0000 7f6fd1d9b700 1 -- 172.21.15.187:0/1036334408 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc00bb20 con 0x7f6fd4066f70 2020-11-20T12:13:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.975+0000 7f6fd1d9b700 1 -- 172.21.15.187:0/1036334408 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00bea0 con 0x7f6fd4066f70 2020-11-20T12:13:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.976+0000 7f6fd359e700 1 -- 172.21.15.187:0/1036334408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 msgr2=0x7f6fd40aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.976+0000 7f6fd359e700 1 --2- 172.21.15.187:0/1036334408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd40aff80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7f6fd4045000 tx=0x7f6fcc0091d0).stop 2020-11-20T12:13:47.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.976+0000 7f6fd359e700 1 -- 172.21.15.187:0/1036334408 shutdown_connections 2020-11-20T12:13:47.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.976+0000 7f6fd359e700 1 --2- 172.21.15.187:0/1036334408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd40aff80 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.976+0000 7f6fd359e700 1 --2- 172.21.15.187:0/1036334408 >> v2:172.21.15.5:3300/0 conn(0x7f6fd40b6650 0x7f6fd40afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.976+0000 7f6fd359e700 1 -- 172.21.15.187:0/1036334408 >> 172.21.15.187:0/1036334408 conn(0x7f6fd40bab30 msgr2=0x7f6fd40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:47.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.976+0000 7f6fd359e700 1 -- 172.21.15.187:0/1036334408 shutdown_connections 2020-11-20T12:13:47.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.976+0000 7f6fd359e700 1 -- 172.21.15.187:0/1036334408 wait complete. 2020-11-20T12:13:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.977+0000 7f6fd359e700 1 Processor -- start 2020-11-20T12:13:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.978+0000 7f6fd359e700 1 -- start start 2020-11-20T12:13:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.978+0000 7f6fd359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd409fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.978+0000 7f6fd359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd40b6650 0x7f6fd409f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.978+0000 7f6fd359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd41484d0 0x7f6fd40b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.978+0000 7f6fd359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b2a40 con 0x7f6fd41484d0 2020-11-20T12:13:47.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.978+0000 7f6fd359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd409ddf0 con 0x7f6fd4066f70 2020-11-20T12:13:47.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.978+0000 7f6fd359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd409e090 con 0x7f6fd40b6650 2020-11-20T12:13:47.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd409fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd409fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48196/0 (socket says 172.21.15.187:48196) 2020-11-20T12:13:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 -- 172.21.15.187:0/4277407446 learned_addr learned my addr 172.21.15.187:0/4277407446 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 -- 172.21.15.187:0/4277407446 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd40b6650 msgr2=0x7f6fd409f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd40b6650 0x7f6fd409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:47.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd2d9d700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd41484d0 0x7f6fd40b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 -- 172.21.15.187:0/4277407446 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd41484d0 msgr2=0x7f6fd40b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd41484d0 0x7f6fd40b2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 -- 172.21.15.187:0/4277407446 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40a02a0 con 0x7f6fd4066f70 2020-11-20T12:13:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.979+0000 7f6fd259c700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd409fdb0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f6fc400c710 tx=0x7f6fc4007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.980+0000 7f6fca7fc700 1 -- 172.21.15.187:0/4277407446 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc40072e0 con 0x7f6fd4066f70 2020-11-20T12:13:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.980+0000 7f6fca7fc700 1 -- 172.21.15.187:0/4277407446 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc4007440 con 0x7f6fd4066f70 2020-11-20T12:13:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.980+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd40b39d0 con 0x7f6fd4066f70 2020-11-20T12:13:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.980+0000 7f6fca7fc700 1 -- 172.21.15.187:0/4277407446 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4005800 con 0x7f6fd4066f70 2020-11-20T12:13:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.980+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd40b2790 con 0x7f6fd4066f70 2020-11-20T12:13:47.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.981+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd4063d60 con 0x7f6fd4066f70 2020-11-20T12:13:47.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.981+0000 7f6fca7fc700 1 -- 172.21.15.187:0/4277407446 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6fc4010070 con 0x7f6fd4066f70 2020-11-20T12:13:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.982+0000 7f6fca7fc700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fb0056d20 0x7f6fb00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.982+0000 7f6fca7fc700 1 -- 172.21.15.187:0/4277407446 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6fc406f8d0 con 0x7f6fd4066f70 2020-11-20T12:13:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.982+0000 7f6fcbfff700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fb0056d20 0x7f6fb00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.983+0000 7f6fcbfff700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fb0056d20 0x7f6fb00591e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f6fd415a6c0 tx=0x7f6fbc006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:47.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:47.987+0000 7f6fca7fc700 1 -- 172.21.15.187:0/4277407446 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6fc4044c90 con 0x7f6fd4066f70 2020-11-20T12:13:48.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.175+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6fd4063fd0 con 0x7f6fd4066f70 2020-11-20T12:13:48.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.176+0000 7f6fca7fc700 1 -- 172.21.15.187:0/4277407446 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v89) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6fc40482b0 con 0x7f6fd4066f70 2020-11-20T12:13:48.180 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:13:48.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fb0056d20 msgr2=0x7f6fb00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fb0056d20 0x7f6fb00591e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f6fd415a6c0 tx=0x7f6fbc006df0).stop 2020-11-20T12:13:48.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 msgr2=0x7f6fd409fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd409fdb0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f6fc400c710 tx=0x7f6fc4007ad0).stop 2020-11-20T12:13:48.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 shutdown_connections 2020-11-20T12:13:48.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fb0056d20 0x7f6fb00591e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd41484d0 0x7f6fd40b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd4066f70 0x7f6fd409fdb0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 --2- 172.21.15.187:0/4277407446 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd40b6650 0x7f6fd409f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:48.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.178+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 >> 172.21.15.187:0/4277407446 conn(0x7f6fd40bab30 msgr2=0x7f6fd40b8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:48.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.179+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 shutdown_connections 2020-11-20T12:13:48.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.179+0000 7f6fd359e700 1 -- 172.21.15.187:0/4277407446 wait complete. 2020-11-20T12:13:48.207 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:13:48.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.456+0000 7f1e0b398700 1 Processor -- start 2020-11-20T12:13:48.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.457+0000 7f1e0b398700 1 -- start start 2020-11-20T12:13:48.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.458+0000 7f1e0b398700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e04065190 0x7f1e04065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.458+0000 7f1e0b398700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e0409ca90 0x7f1e0409cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.458+0000 7f1e0b398700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e040b13e0 con 0x7f1e04065f90 2020-11-20T12:13:48.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.458+0000 7f1e0b398700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0414ab30 con 0x7f1e04065190 2020-11-20T12:13:48.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.458+0000 7f1e0b398700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04062c10 con 0x7f1e0409ca90 2020-11-20T12:13:48.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.458+0000 7f1e0a396700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e04065190 0x7f1e04065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.458+0000 7f1e09b95700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1e04065f90 0x7f1e04143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44810/0 (socket says 172.21.15.187:44810) 2020-11-20T12:13:48.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.459+0000 7f1e09b95700 1 -- 172.21.15.187:0/3580925122 learned_addr learned my addr 172.21.15.187:0/3580925122 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:48.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.459+0000 7f1e0ab97700 1 --2- 172.21.15.187:0/3580925122 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e0409ca90 0x7f1e0409cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.459+0000 7f1e0a396700 1 -- 172.21.15.187:0/3580925122 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e0409ca90 msgr2=0x7f1e0409cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:48.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.459+0000 7f1e0a396700 1 --2- 172.21.15.187:0/3580925122 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e0409ca90 0x7f1e0409cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.459+0000 7f1e0a396700 1 -- 172.21.15.187:0/3580925122 >> v1:172.21.15.187:6789/0 conn(0x7f1e04065f90 legacy=0x7f1e04143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.459+0000 7f1e0a396700 1 -- 172.21.15.187:0/3580925122 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e040b1640 con 0x7f1e04065190 2020-11-20T12:13:48.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.459+0000 7f1e09394700 1 -- 172.21.15.187:0/3580925122 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617318144 0 0) 0x7f1e040b13e0 con 0x7f1e04065f90 2020-11-20T12:13:48.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.459+0000 7f1e0a396700 1 --2- 172.21.15.187:0/3580925122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e04065190 0x7f1e04065570 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f1e0000b9a0 tx=0x7f1e00006df0).ready entity=mon.1 client_cookie=a339b90b3862d2cc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e09394700 1 -- 172.21.15.187:0/3580925122 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00009840 con 0x7f1e04065190 2020-11-20T12:13:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e09394700 1 -- 172.21.15.187:0/3580925122 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e000099a0 con 0x7f1e04065190 2020-11-20T12:13:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e09394700 1 -- 172.21.15.187:0/3580925122 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00009d20 con 0x7f1e04065190 2020-11-20T12:13:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e0b398700 1 -- 172.21.15.187:0/3580925122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e04065190 msgr2=0x7f1e04065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e0b398700 1 --2- 172.21.15.187:0/3580925122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e04065190 0x7f1e04065570 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f1e0000b9a0 tx=0x7f1e00006df0).stop 2020-11-20T12:13:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e0b398700 1 -- 172.21.15.187:0/3580925122 shutdown_connections 2020-11-20T12:13:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e0b398700 1 --2- 172.21.15.187:0/3580925122 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e0409ca90 0x7f1e0409cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e0b398700 1 --2- 172.21.15.187:0/3580925122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e04065190 0x7f1e04065570 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.460+0000 7f1e0b398700 1 -- 172.21.15.187:0/3580925122 >> 172.21.15.187:0/3580925122 conn(0x7f1e040bab30 msgr2=0x7f1e040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:48.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.461+0000 7f1e0b398700 1 -- 172.21.15.187:0/3580925122 shutdown_connections 2020-11-20T12:13:48.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.461+0000 7f1e0b398700 1 -- 172.21.15.187:0/3580925122 wait complete. 2020-11-20T12:13:48.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.462+0000 7f1e0b398700 1 Processor -- start 2020-11-20T12:13:48.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.463+0000 7f1e0b398700 1 -- start start 2020-11-20T12:13:48.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0b398700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e04065190 0x7f1e04143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0b398700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e04065f90 0x7f1e0409f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0b398700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0409ca90 0x7f1e040b7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0b398700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04149740 con 0x7f1e04065190 2020-11-20T12:13:48.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0b398700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e041499e0 con 0x7f1e0409ca90 2020-11-20T12:13:48.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0b398700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04143580 con 0x7f1e04065f90 2020-11-20T12:13:48.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0ab97700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0409ca90 0x7f1e040b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0ab97700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0409ca90 0x7f1e040b7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48214/0 (socket says 172.21.15.187:48214) 2020-11-20T12:13:48.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.464+0000 7f1e0ab97700 1 -- 172.21.15.187:0/2118759243 learned_addr learned my addr 172.21.15.187:0/2118759243 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:48.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e0a396700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e04065190 0x7f1e04143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e09b95700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e04065f90 0x7f1e0409f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e0ab97700 1 -- 172.21.15.187:0/2118759243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e04065f90 msgr2=0x7f1e0409f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e0ab97700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e04065f90 0x7f1e0409f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e0ab97700 1 -- 172.21.15.187:0/2118759243 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e04065190 msgr2=0x7f1e04143090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e0ab97700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e04065190 0x7f1e04143090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e0ab97700 1 -- 172.21.15.187:0/2118759243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e040b4650 con 0x7f1e0409ca90 2020-11-20T12:13:48.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e0ab97700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0409ca90 0x7f1e040b7610 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f1dfc00db40 tx=0x7f1dfc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:48.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1df27fc700 1 -- 172.21.15.187:0/2118759243 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc009a10 con 0x7f1e0409ca90 2020-11-20T12:13:48.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1df27fc700 1 -- 172.21.15.187:0/2118759243 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc009b70 con 0x7f1e0409ca90 2020-11-20T12:13:48.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0409fdf0 con 0x7f1e0409ca90 2020-11-20T12:13:48.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.465+0000 7f1df27fc700 1 -- 172.21.15.187:0/2118759243 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc013480 con 0x7f1e0409ca90 2020-11-20T12:13:48.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.466+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e040b7d40 con 0x7f1e0409ca90 2020-11-20T12:13:48.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.467+0000 7f1df27fc700 1 -- 172.21.15.187:0/2118759243 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1dfc0137f0 con 0x7f1e0409ca90 2020-11-20T12:13:48.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.467+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e040b47e0 con 0x7f1e0409ca90 2020-11-20T12:13:48.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.468+0000 7f1df27fc700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ddc056d20 0x7f1ddc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.468+0000 7f1df27fc700 1 -- 172.21.15.187:0/2118759243 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1dfc0702b0 con 0x7f1e0409ca90 2020-11-20T12:13:48.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.469+0000 7f1e0a396700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ddc056d20 0x7f1ddc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.469+0000 7f1e0a396700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ddc056d20 0x7f1ddc0591e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f1e0000b150 tx=0x7f1e00015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:48.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.473+0000 7f1df27fc700 1 -- 172.21.15.187:0/2118759243 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1dfc045700 con 0x7f1e0409ca90 2020-11-20T12:13:48.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.657+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e04149b70 con 0x7f1e0409ca90 2020-11-20T12:13:48.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.657+0000 7f1df27fc700 1 -- 172.21.15.187:0/2118759243 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1dfc048d20 con 0x7f1e0409ca90 2020-11-20T12:13:48.662 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:13:48.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ddc056d20 msgr2=0x7f1ddc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ddc056d20 0x7f1ddc0591e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f1e0000b150 tx=0x7f1e00015000).stop 2020-11-20T12:13:48.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0409ca90 msgr2=0x7f1e040b7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0409ca90 0x7f1e040b7610 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f1dfc00db40 tx=0x7f1dfc008040).stop 2020-11-20T12:13:48.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 shutdown_connections 2020-11-20T12:13:48.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ddc056d20 0x7f1ddc0591e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e04065190 0x7f1e04143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e0409ca90 0x7f1e040b7610 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 --2- 172.21.15.187:0/2118759243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e04065f90 0x7f1e0409f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 >> 172.21.15.187:0/2118759243 conn(0x7f1e040bab30 msgr2=0x7f1e0409b080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:48.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 shutdown_connections 2020-11-20T12:13:48.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.660+0000 7f1e0b398700 1 -- 172.21.15.187:0/2118759243 wait complete. 2020-11-20T12:13:48.686 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:13:48.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.930+0000 7f55f96a5700 1 Processor -- start 2020-11-20T12:13:48.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.930+0000 7f55f96a5700 1 -- start start 2020-11-20T12:13:48.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.931+0000 7f55f96a5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f55f40ba9b0 0x7f55f4148400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.931+0000 7f55f96a5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f414be70 con 0x7f55f40675d0 2020-11-20T12:13:48.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.931+0000 7f55f96a5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f414c110 con 0x7f55f40679b0 2020-11-20T12:13:48.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.931+0000 7f55f96a5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f414c3b0 con 0x7f55f40ba9b0 2020-11-20T12:13:48.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.932+0000 7f55eb7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f55f40679b0 0x7f55f4147cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44826/0 (socket says 172.21.15.187:44826) 2020-11-20T12:13:48.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.932+0000 7f55eb7fe700 1 -- 172.21.15.187:0/2053436666 learned_addr learned my addr 172.21.15.187:0/2053436666 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:48.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.932+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994100734 0 0) 0x7f55f414c110 con 0x7f55f40679b0 2020-11-20T12:13:48.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.932+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d4003240 con 0x7f55f40679b0 2020-11-20T12:13:48.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.932+0000 7f55f8ea4700 1 --2- 172.21.15.187:0/2053436666 >> v2:172.21.15.5:3300/0 conn(0x7f55f40ba9b0 0x7f55f4148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3647941549 0 0) 0x7f55d4003240 con 0x7f55f40679b0 2020-11-20T12:13:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 >> v2:172.21.15.5:3300/0 conn(0x7f55f40ba9b0 msgr2=0x7f55f4148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 --2- 172.21.15.187:0/2053436666 >> v2:172.21.15.5:3300/0 conn(0x7f55f40ba9b0 0x7f55f4148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 >> v1:172.21.15.5:6789/0 conn(0x7f55f40675d0 legacy=0x7f55f414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f40bcb60 con 0x7f55f40679b0 2020-11-20T12:13:48.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f55dc0014e0 con 0x7f55f40679b0 2020-11-20T12:13:48.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 >> v1:172.21.15.187:6789/0 conn(0x7f55f40679b0 legacy=0x7f55f4147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55d4003730 0x7f55d4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d40044e0 0x7f55d4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55d4004e50 0x7f55d400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f414c3b0 con 0x7f55d4004e50 2020-11-20T12:13:48.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d4004260 con 0x7f55d40044e0 2020-11-20T12:13:48.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d400bc20 con 0x7f55d4003730 2020-11-20T12:13:48.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.933+0000 7f55eb7fe700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d40044e0 0x7f55d4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55f8ea4700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55d4004e50 0x7f55d400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55eb7fe700 1 -- 172.21.15.187:0/2053436666 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55d4003730 msgr2=0x7f55d4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55eb7fe700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55d4003730 0x7f55d4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55eb7fe700 1 -- 172.21.15.187:0/2053436666 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55d4004e50 msgr2=0x7f55d400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55eb7fe700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55d4004e50 0x7f55d400b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55eb7fe700 1 -- 172.21.15.187:0/2053436666 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d4003240 con 0x7f55d40044e0 2020-11-20T12:13:48.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55f8ea4700 1 -- 172.21.15.187:0/2053436666 reap_dead start 2020-11-20T12:13:48.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55eb7fe700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d40044e0 0x7f55d4004960 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f55dc00c410 tx=0x7f55dc0077d0).ready entity=mon.1 client_cookie=b66000d7d8318eb3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:48.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00ce80 con 0x7f55d40044e0 2020-11-20T12:13:48.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc0072e0 con 0x7f55d40044e0 2020-11-20T12:13:48.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.934+0000 7f55f37fe700 1 -- 172.21.15.187:0/2053436666 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc005800 con 0x7f55d40044e0 2020-11-20T12:13:48.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.935+0000 7f55f96a5700 1 -- 172.21.15.187:0/2053436666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d40044e0 msgr2=0x7f55d4004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.935+0000 7f55f96a5700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d40044e0 0x7f55d4004960 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f55dc00c410 tx=0x7f55dc0077d0).stop 2020-11-20T12:13:48.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.935+0000 7f55f96a5700 1 -- 172.21.15.187:0/2053436666 shutdown_connections 2020-11-20T12:13:48.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.935+0000 7f55f96a5700 1 --2- 172.21.15.187:0/2053436666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d40044e0 0x7f55d4004960 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.935+0000 7f55f96a5700 1 -- 172.21.15.187:0/2053436666 >> 172.21.15.187:0/2053436666 conn(0x7f55f40bce50 msgr2=0x7f55f40bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:48.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.935+0000 7f55f96a5700 1 -- 172.21.15.187:0/2053436666 shutdown_connections 2020-11-20T12:13:48.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.935+0000 7f55f96a5700 1 -- 172.21.15.187:0/2053436666 wait complete. 2020-11-20T12:13:48.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.937+0000 7f55f96a5700 1 Processor -- start 2020-11-20T12:13:48.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.937+0000 7f55f96a5700 1 -- start start 2020-11-20T12:13:48.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f96a5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55d40044e0 0x7f55f40b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f96a5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55f4147750 0x7f55f40b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f96a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f40b6210 0x7f55f40b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f96a5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f40b5230 con 0x7f55f4147750 2020-11-20T12:13:48.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f96a5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f40b54d0 con 0x7f55f40b6210 2020-11-20T12:13:48.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f96a5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f40b5770 con 0x7f55d40044e0 2020-11-20T12:13:48.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f8ea4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f40b6210 0x7f55f40b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f8ea4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f40b6210 0x7f55f40b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48234/0 (socket says 172.21.15.187:48234) 2020-11-20T12:13:48.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f8ea4700 1 -- 172.21.15.187:0/710360912 learned_addr learned my addr 172.21.15.187:0/710360912 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:48.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.938+0000 7f55f3fff700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55d40044e0 0x7f55f40b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f8ea4700 1 -- 172.21.15.187:0/710360912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55d40044e0 msgr2=0x7f55f40b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f8ea4700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55d40044e0 0x7f55f40b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55eb7fe700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55f4147750 0x7f55f40b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f8ea4700 1 -- 172.21.15.187:0/710360912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55f4147750 msgr2=0x7f55f40b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:48.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f8ea4700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55f4147750 0x7f55f40b5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:48.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f8ea4700 1 -- 172.21.15.187:0/710360912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f4147bd0 con 0x7f55f40b6210 2020-11-20T12:13:48.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f8ea4700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f40b6210 0x7f55f40b6670 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f55f405c6d0 tx=0x7f55ec008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:48.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f0ff9700 1 -- 172.21.15.187:0/710360912 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec00c070 con 0x7f55f40b6210 2020-11-20T12:13:48.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f0ff9700 1 -- 172.21.15.187:0/710360912 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec0133f0 con 0x7f55f40b6210 2020-11-20T12:13:48.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f409d810 con 0x7f55f40b6210 2020-11-20T12:13:48.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f0ff9700 1 -- 172.21.15.187:0/710360912 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec0137b0 con 0x7f55f40b6210 2020-11-20T12:13:48.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.939+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f409dd50 con 0x7f55f40b6210 2020-11-20T12:13:48.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.941+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f4147d60 con 0x7f55f40b6210 2020-11-20T12:13:48.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.941+0000 7f55f0ff9700 1 -- 172.21.15.187:0/710360912 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f55ec013b60 con 0x7f55f40b6210 2020-11-20T12:13:48.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.942+0000 7f55f0ff9700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f55d0056d20 0x7f55d00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:48.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.942+0000 7f55f0ff9700 1 -- 172.21.15.187:0/710360912 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f55ec06f340 con 0x7f55f40b6210 2020-11-20T12:13:48.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.942+0000 7f55f3fff700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f55d0056d20 0x7f55d00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:48.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.943+0000 7f55f3fff700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f55d0056d20 0x7f55d00591e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f55e4007fd0 tx=0x7f55e4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:48.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:48.945+0000 7f55f0ff9700 1 -- 172.21.15.187:0/710360912 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f55f4147d60 con 0x7f55f40b6210 2020-11-20T12:13:49.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.129+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55f4147d60 con 0x7f55f40b6210 2020-11-20T12:13:49.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.129+0000 7f55f0ff9700 1 -- 172.21.15.187:0/710360912 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+10 (secure 0 0 0) 0x7f55f4147d60 con 0x7f55f40b6210 2020-11-20T12:13:49.132 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:13:49.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f55d0056d20 msgr2=0x7f55d00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:49.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f55d0056d20 0x7f55d00591e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f55e4007fd0 tx=0x7f55e4009040).stop 2020-11-20T12:13:49.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f40b6210 msgr2=0x7f55f40b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:49.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f40b6210 0x7f55f40b6670 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f55f405c6d0 tx=0x7f55ec008040).stop 2020-11-20T12:13:49.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 shutdown_connections 2020-11-20T12:13:49.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f55d0056d20 0x7f55d00591e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f40b6210 0x7f55f40b6670 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55d40044e0 0x7f55f40b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 --2- 172.21.15.187:0/710360912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55f4147750 0x7f55f40b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 >> 172.21.15.187:0/710360912 conn(0x7f55f40bce50 msgr2=0x7f55f4148090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:49.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 shutdown_connections 2020-11-20T12:13:49.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.132+0000 7f55f96a5700 1 -- 172.21.15.187:0/710360912 wait complete. 2020-11-20T12:13:49.158 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:13:49.159 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:49.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.396+0000 7fe9fb59e700 1 Processor -- start 2020-11-20T12:13:49.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.396+0000 7fe9fb59e700 1 -- start start 2020-11-20T12:13:49.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.397+0000 7fe9fb59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe9fc1484d0 0x7fe9fc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.397+0000 7fe9fb59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe9fc0622c0 0x7fe9fc0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.397+0000 7fe9fb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe9fc066f70 0x7fe9fc0b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.397+0000 7fe9fb59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9fc067730 con 0x7fe9fc1484d0 2020-11-20T12:13:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.397+0000 7fe9fb59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9fc0b1300 con 0x7fe9fc0622c0 2020-11-20T12:13:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.397+0000 7fe9fb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0b15a0 con 0x7fe9fc066f70 2020-11-20T12:13:49.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9f9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe9fc0622c0 0x7fe9fc0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:49.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9f9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe9fc0622c0 0x7fe9fc0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48240/0 (socket says 172.21.15.187:48240) 2020-11-20T12:13:49.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9f9d9b700 1 -- 172.21.15.187:0/24762131 learned_addr learned my addr 172.21.15.187:0/24762131 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:49.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9f9d9b700 1 -- 172.21.15.187:0/24762131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe9fc066f70 msgr2=0x7fe9fc0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:49.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9f9d9b700 1 --2- 172.21.15.187:0/24762131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe9fc066f70 0x7fe9fc0b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:49.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9f9d9b700 1 -- 172.21.15.187:0/24762131 >> v2:172.21.15.5:3300/0 conn(0x7fe9fc1484d0 msgr2=0x7fe9fc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:49.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9fa59c700 1 --2- 172.21.15.187:0/24762131 >> v2:172.21.15.5:3300/0 conn(0x7fe9fc1484d0 0x7fe9fc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:49.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9f9d9b700 1 --2- 172.21.15.187:0/24762131 >> v2:172.21.15.5:3300/0 conn(0x7fe9fc1484d0 0x7fe9fc061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.398+0000 7fe9f9d9b700 1 -- 172.21.15.187:0/24762131 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0b1840 con 0x7fe9fc0622c0 2020-11-20T12:13:49.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.399+0000 7fe9f9d9b700 1 --2- 172.21.15.187:0/24762131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc0622c0 0x7fe9fc0b5640 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7fe9fc068600 tx=0x7fe9f0006eb0).ready entity=mon.1 client_cookie=d3d542f15cea5816 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:49.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.399+0000 7fe9f959a700 1 -- 172.21.15.187:0/24762131 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00098d0 con 0x7fe9fc0622c0 2020-11-20T12:13:49.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.399+0000 7fe9f959a700 1 -- 172.21.15.187:0/24762131 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f0009a30 con 0x7fe9fc0622c0 2020-11-20T12:13:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.399+0000 7fe9f959a700 1 -- 172.21.15.187:0/24762131 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0009db0 con 0x7fe9fc0622c0 2020-11-20T12:13:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.399+0000 7fe9fb59e700 1 -- 172.21.15.187:0/24762131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc0622c0 msgr2=0x7fe9fc0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.399+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/24762131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc0622c0 0x7fe9fc0b5640 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7fe9fc068600 tx=0x7fe9f0006eb0).stop 2020-11-20T12:13:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.400+0000 7fe9fb59e700 1 -- 172.21.15.187:0/24762131 shutdown_connections 2020-11-20T12:13:49.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.400+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/24762131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe9fc066f70 0x7fe9fc0b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.400+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/24762131 >> v2:172.21.15.5:3300/0 conn(0x7fe9fc1484d0 0x7fe9fc061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.400+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/24762131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc0622c0 0x7fe9fc0b5640 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.400+0000 7fe9fb59e700 1 -- 172.21.15.187:0/24762131 >> 172.21.15.187:0/24762131 conn(0x7fe9fc0bab30 msgr2=0x7fe9fc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.400+0000 7fe9fb59e700 1 -- 172.21.15.187:0/24762131 shutdown_connections 2020-11-20T12:13:49.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.400+0000 7fe9fb59e700 1 -- 172.21.15.187:0/24762131 wait complete. 2020-11-20T12:13:49.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.401+0000 7fe9fb59e700 1 Processor -- start 2020-11-20T12:13:49.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.401+0000 7fe9fb59e700 1 -- start start 2020-11-20T12:13:49.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.402+0000 7fe9fb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0622c0 0x7fe9fc0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:49.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.402+0000 7fe9fb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe9fc066f70 0x7fe9fc144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:49.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.402+0000 7fe9fb59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc1484d0 0x7fe9fc144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:49.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.402+0000 7fe9fb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc144b20 con 0x7fe9fc0622c0 2020-11-20T12:13:49.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.402+0000 7fe9fb59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc09fec0 con 0x7fe9fc1484d0 2020-11-20T12:13:49.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.402+0000 7fe9fb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0a0160 con 0x7fe9fc066f70 2020-11-20T12:13:49.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc1484d0 0x7fe9fc144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:49.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc1484d0 0x7fe9fc144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48248/0 (socket says 172.21.15.187:48248) 2020-11-20T12:13:49.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 -- 172.21.15.187:0/417608294 learned_addr learned my addr 172.21.15.187:0/417608294 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:49.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 -- 172.21.15.187:0/417608294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe9fc066f70 msgr2=0x7fe9fc144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:49.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe9fc066f70 0x7fe9fc144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:49.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fa59c700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0622c0 0x7fe9fc0afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:49.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 -- 172.21.15.187:0/417608294 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0622c0 msgr2=0x7fe9fc0afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:49.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0622c0 0x7fe9fc0afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 -- 172.21.15.187:0/417608294 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0a0400 con 0x7fe9fc1484d0 2020-11-20T12:13:49.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.403+0000 7fe9fad9d700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc1484d0 0x7fe9fc144630 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7fe9f400ba30 tx=0x7fe9f4009220).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:49.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.404+0000 7fe9ea7fc700 1 -- 172.21.15.187:0/417608294 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f400c8b0 con 0x7fe9fc1484d0 2020-11-20T12:13:49.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.404+0000 7fe9ea7fc700 1 -- 172.21.15.187:0/417608294 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f400ca10 con 0x7fe9fc1484d0 2020-11-20T12:13:49.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.404+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc0b0140 con 0x7fe9fc1484d0 2020-11-20T12:13:49.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.404+0000 7fe9ea7fc700 1 -- 172.21.15.187:0/417608294 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f400cbc0 con 0x7fe9fc1484d0 2020-11-20T12:13:49.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.404+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc0a0ae0 con 0x7fe9fc1484d0 2020-11-20T12:13:49.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.405+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc0a0590 con 0x7fe9fc1484d0 2020-11-20T12:13:49.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.406+0000 7fe9ea7fc700 1 -- 172.21.15.187:0/417608294 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe9f401c070 con 0x7fe9fc1484d0 2020-11-20T12:13:49.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.406+0000 7fe9ea7fc700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe9d0056d20 0x7fe9d00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:49.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.406+0000 7fe9ea7fc700 1 -- 172.21.15.187:0/417608294 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe9f406f790 con 0x7fe9fc1484d0 2020-11-20T12:13:49.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.406+0000 7fe9fa59c700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe9d0056d20 0x7fe9d00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:49.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.407+0000 7fe9fa59c700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe9d0056d20 0x7fe9d00591e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fe9fc09f750 tx=0x7fe9ec008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:49.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.410+0000 7fe9ea7fc700 1 -- 172.21.15.187:0/417608294 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe9f4044be0 con 0x7fe9fc1484d0 2020-11-20T12:13:49.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.656+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9fc061a20 con 0x7fe9fc1484d0 2020-11-20T12:13:49.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.657+0000 7fe9ea7fc700 1 -- 172.21.15.187:0/417608294 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fe9f401a070 con 0x7fe9fc1484d0 2020-11-20T12:13:49.662 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:49.664 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:13:49.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.659+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe9d0056d20 msgr2=0x7fe9d00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:49.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.659+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe9d0056d20 0x7fe9d00591e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fe9fc09f750 tx=0x7fe9ec008040).stop 2020-11-20T12:13:49.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.659+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc1484d0 msgr2=0x7fe9fc144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:49.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.659+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc1484d0 0x7fe9fc144630 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7fe9f400ba30 tx=0x7fe9f4009220).stop 2020-11-20T12:13:49.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.660+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 shutdown_connections 2020-11-20T12:13:49.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.660+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe9d0056d20 0x7fe9d00591e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.660+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0622c0 0x7fe9fc0afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.660+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9fc1484d0 0x7fe9fc144630 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:49.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.660+0000 7fe9fb59e700 1 --2- 172.21.15.187:0/417608294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe9fc066f70 0x7fe9fc144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:49.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.660+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 >> 172.21.15.187:0/417608294 conn(0x7fe9fc0bab30 msgr2=0x7fe9fc0b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:49.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.660+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 shutdown_connections 2020-11-20T12:13:49.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:49.660+0000 7fe9fb59e700 1 -- 172.21.15.187:0/417608294 wait complete. 2020-11-20T12:13:49.667 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:13:49.886 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:13:40.253243+0000', 'modified': '2020-11-20T12:13:42.397341+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7408}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7408': {'gid': 7408, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1880661915', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1880661915}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1880661915}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:13:49.886 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:13:49.887 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:13:50.172 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":7408,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":9.8901729730000003,"mdsmap_epoch":80,"osdmap_epoch":89,"osdmap_epoch_barrier":89,"uptime":15.541169928} 2020-11-20T12:13:50.173 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 7408, "mdsmap_epoch": 80, "osdmap_epoch": 89, "osdmap_epoch_barrier": 89, "rank_uptime": 9.890172973, "state": "up:active", "uptime": 15.541169928, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:13:50.173 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:13:50.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:13:50.174 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:13:50.225 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:13:50.225 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:13:50.359 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:13:50.360 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:13:50.401 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:13:50.401 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:13:50.414 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:13:50.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:13:50.415 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:13:50.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:13:50.507 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo dd if=/dev/urandom of=file_a bs=1M conv=fdatasync count=8 seek=0) 2020-11-20T12:13:50.834 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records in 2020-11-20T12:13:50.835 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records out 2020-11-20T12:13:50.836 INFO:teuthology.orchestra.run.smithi187.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.196722 s, 42.6 MB/s 2020-11-20T12:13:50.841 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:13:50.842 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:13:50.842 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:13:50.843 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:13:50.843 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/file_a").st_ino) 2020-11-20T12:13:50.843 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627776 2020-11-20T12:13:51.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:13:51.220 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo dd if=/dev/urandom of=file_b bs=1M conv=fdatasync count=8 seek=0) 2020-11-20T12:13:51.343 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:13:51.352 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:13:51.475 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records in 2020-11-20T12:13:51.475 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records out 2020-11-20T12:13:51.475 INFO:teuthology.orchestra.run.smithi187.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.153068 s, 54.8 MB/s 2020-11-20T12:13:51.482 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:13:51.482 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:13:51.482 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:13:51.483 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:13:51.483 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/file_b").st_ino) 2020-11-20T12:13:51.484 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:13:51.681 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627777 2020-11-20T12:13:51.866 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:52.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.158+0000 7f7770821700 1 Processor -- start 2020-11-20T12:13:52.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.158+0000 7f7770821700 1 -- start start 2020-11-20T12:13:52.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.159+0000 7f7770821700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f776c0b1760 0x7f776c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.159+0000 7f7770821700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f776c065680 0x7f776c14c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.159+0000 7f7770821700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c150390 con 0x7f776c0b1380 2020-11-20T12:13:52.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.159+0000 7f7770821700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776c1511e0 con 0x7f776c065680 2020-11-20T12:13:52.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.159+0000 7f7770821700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776c14d760 con 0x7f776c0b1760 2020-11-20T12:13:52.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.160+0000 7f776259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f776c0b1760 0x7f776c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.160+0000 7f776259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f776c0b1760 0x7f776c065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48264/0 (socket says 172.21.15.187:48264) 2020-11-20T12:13:52.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.160+0000 7f776259c700 1 -- 172.21.15.187:0/2368176407 learned_addr learned my addr 172.21.15.187:0/2368176407 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:52.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.160+0000 7f776b59e700 1 --2- 172.21.15.187:0/2368176407 >> v2:172.21.15.5:3300/0 conn(0x7f776c065680 0x7f776c14c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.160+0000 7f776259c700 1 -- 172.21.15.187:0/2368176407 >> v2:172.21.15.5:3300/0 conn(0x7f776c065680 msgr2=0x7f776c14c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:52.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.160+0000 7f776259c700 1 --2- 172.21.15.187:0/2368176407 >> v2:172.21.15.5:3300/0 conn(0x7f776c065680 0x7f776c14c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.160+0000 7f776259c700 1 -- 172.21.15.187:0/2368176407 >> v1:172.21.15.5:6789/0 conn(0x7f776c0b1380 legacy=0x7f776c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:52.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.160+0000 7f776259c700 1 -- 172.21.15.187:0/2368176407 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c150630 con 0x7f776c0b1760 2020-11-20T12:13:52.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.161+0000 7f776259c700 1 --2- 172.21.15.187:0/2368176407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c0b1760 0x7f776c065190 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f776c063a20 tx=0x7f7754006eb0).ready entity=mon.1 client_cookie=e7c64d00f8bd852b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:52.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.161+0000 7f776a59c700 1 -- 172.21.15.187:0/2368176407 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754009890 con 0x7f776c0b1760 2020-11-20T12:13:52.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.161+0000 7f776a59c700 1 -- 172.21.15.187:0/2368176407 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77540099f0 con 0x7f776c0b1760 2020-11-20T12:13:52.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.161+0000 7f776a59c700 1 -- 172.21.15.187:0/2368176407 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754009d70 con 0x7f776c0b1760 2020-11-20T12:13:52.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.162+0000 7f7770821700 1 -- 172.21.15.187:0/2368176407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c0b1760 msgr2=0x7f776c065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.162+0000 7f7770821700 1 --2- 172.21.15.187:0/2368176407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c0b1760 0x7f776c065190 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f776c063a20 tx=0x7f7754006eb0).stop 2020-11-20T12:13:52.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.162+0000 7f7770821700 1 -- 172.21.15.187:0/2368176407 shutdown_connections 2020-11-20T12:13:52.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.162+0000 7f7770821700 1 --2- 172.21.15.187:0/2368176407 >> v2:172.21.15.5:3300/0 conn(0x7f776c065680 0x7f776c14c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.162+0000 7f7770821700 1 --2- 172.21.15.187:0/2368176407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c0b1760 0x7f776c065190 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.162+0000 7f7770821700 1 -- 172.21.15.187:0/2368176407 >> 172.21.15.187:0/2368176407 conn(0x7f776c0bab30 msgr2=0x7f776c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:52.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.162+0000 7f7770821700 1 -- 172.21.15.187:0/2368176407 shutdown_connections 2020-11-20T12:13:52.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.162+0000 7f7770821700 1 -- 172.21.15.187:0/2368176407 wait complete. 2020-11-20T12:13:52.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.163+0000 7f7770821700 1 Processor -- start 2020-11-20T12:13:52.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.164+0000 7f7770821700 1 -- start start 2020-11-20T12:13:52.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.164+0000 7f7770821700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f776c0b1380 0x7f776c0b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.164+0000 7f7770821700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f776c0b1760 0x7f776c0b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.164+0000 7f7770821700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c14ea30 0x7f776c148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.164+0000 7f7770821700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c09bc80 con 0x7f776c0b1380 2020-11-20T12:13:52.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.164+0000 7f7770821700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0a0590 con 0x7f776c14ea30 2020-11-20T12:13:52.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.164+0000 7f7770821700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0a0830 con 0x7f776c0b1760 2020-11-20T12:13:52.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c14ea30 0x7f776c148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c14ea30 0x7f776c148870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48270/0 (socket says 172.21.15.187:48270) 2020-11-20T12:13:52.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776b59e700 1 -- 172.21.15.187:0/1105310910 learned_addr learned my addr 172.21.15.187:0/1105310910 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:52.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776259c700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f776c0b1760 0x7f776c0b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776b59e700 1 -- 172.21.15.187:0/1105310910 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f776c0b1760 msgr2=0x7f776c0b23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776b59e700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f776c0b1760 0x7f776c0b23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776ad9d700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f776c0b1380 0x7f776c0b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776b59e700 1 -- 172.21.15.187:0/1105310910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f776c0b1380 msgr2=0x7f776c0b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776b59e700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f776c0b1380 0x7f776c0b4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.165+0000 7f776b59e700 1 -- 172.21.15.187:0/1105310910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c09bf20 con 0x7f776c14ea30 2020-11-20T12:13:52.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.166+0000 7f776b59e700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c14ea30 0x7f776c148870 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f776400afd0 tx=0x7f7764008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:52.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.166+0000 7f77637fe700 1 -- 172.21.15.187:0/1105310910 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764007940 con 0x7f776c14ea30 2020-11-20T12:13:52.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.166+0000 7f77637fe700 1 -- 172.21.15.187:0/1105310910 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7764007aa0 con 0x7f776c14ea30 2020-11-20T12:13:52.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.166+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776c14eeb0 con 0x7f776c14ea30 2020-11-20T12:13:52.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.166+0000 7f77637fe700 1 -- 172.21.15.187:0/1105310910 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764007e40 con 0x7f776c14ea30 2020-11-20T12:13:52.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.166+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c148fa0 con 0x7f776c14ea30 2020-11-20T12:13:52.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.167+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776c061a20 con 0x7f776c14ea30 2020-11-20T12:13:52.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.168+0000 7f77637fe700 1 -- 172.21.15.187:0/1105310910 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7764011620 con 0x7f776c14ea30 2020-11-20T12:13:52.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.168+0000 7f77637fe700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f773c056d20 0x7f773c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.168+0000 7f77637fe700 1 -- 172.21.15.187:0/1105310910 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f77640703a0 con 0x7f776c14ea30 2020-11-20T12:13:52.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.169+0000 7f776ad9d700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f773c056d20 0x7f773c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.169+0000 7f776ad9d700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f773c056d20 0x7f773c0591e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f775c00a5f0 tx=0x7f775c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:52.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.172+0000 7f77637fe700 1 -- 172.21.15.187:0/1105310910 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7764045770 con 0x7f776c14ea30 2020-11-20T12:13:52.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.421+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f776c0b49f0 con 0x7f776c14ea30 2020-11-20T12:13:52.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.423+0000 7f77637fe700 1 -- 172.21.15.187:0/1105310910 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f7764048d90 con 0x7f776c14ea30 2020-11-20T12:13:52.426 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:52.426 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:13:52.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f773c056d20 msgr2=0x7f773c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f773c056d20 0x7f773c0591e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f775c00a5f0 tx=0x7f775c008040).stop 2020-11-20T12:13:52.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c14ea30 msgr2=0x7f776c148870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c14ea30 0x7f776c148870 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f776400afd0 tx=0x7f7764008040).stop 2020-11-20T12:13:52.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 shutdown_connections 2020-11-20T12:13:52.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f773c056d20 0x7f773c0591e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f776c0b1380 0x7f776c0b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776c14ea30 0x7f776c148870 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 --2- 172.21.15.187:0/1105310910 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f776c0b1760 0x7f776c0b23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.425+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 >> 172.21.15.187:0/1105310910 conn(0x7f776c0bab30 msgr2=0x7f776c1492c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:52.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.426+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 shutdown_connections 2020-11-20T12:13:52.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.426+0000 7f7770821700 1 -- 172.21.15.187:0/1105310910 wait complete. 2020-11-20T12:13:52.431 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:13:52.454 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:13:52.768 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:13:52.769 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:13:52.769 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data getxattr 10000000000.00000000 parent 2020-11-20T12:13:52.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.874+0000 7fbafad67d00 1 Processor -- start 2020-11-20T12:13:52.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.874+0000 7fbafad67d00 1 -- start start 2020-11-20T12:13:52.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.875+0000 7fbafad67d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f4f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.875+0000 7fbafad67d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 0x557274f50250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.875+0000 7fbafad67d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557275004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.875+0000 7fbafad67d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x557274ed8e30 con 0x557274ff5a00 2020-11-20T12:13:52.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.875+0000 7fbafad67d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x557274ea7810 con 0x7fbad80039e0 2020-11-20T12:13:52.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.875+0000 7fbafad67d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x557274ff8e90 con 0x5572750043c0 2020-11-20T12:13:52.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.875+0000 7fbaeb60c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f4f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbae2e0b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 0x557274f50250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaeb60c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f4f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48290/0 (socket says 172.21.15.187:48290) 2020-11-20T12:13:52.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaeb60c700 1 -- 172.21.15.187:0/1875703036 learned_addr learned my addr 172.21.15.187:0/1875703036 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:52.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaebe0d700 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557275004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaeb60c700 1 -- 172.21.15.187:0/1875703036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 msgr2=0x557275004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:52.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaeb60c700 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557275004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaeb60c700 1 -- 172.21.15.187:0/1875703036 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 msgr2=0x557274f50250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:52.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaeb60c700 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 0x557274f50250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaeb60c700 1 -- 172.21.15.187:0/1875703036 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557274eccec0 con 0x7fbad80039e0 2020-11-20T12:13:52.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaebe0d700 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557275004840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:52.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.876+0000 7fbaeb60c700 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f4f890 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7fbadc003a70 tx=0x7fbadc008040).ready entity=mon.1 client_cookie=8cab1eb9ba3229f4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:52.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbaea60a700 1 -- 172.21.15.187:0/1875703036 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc0030d0 con 0x7fbad80039e0 2020-11-20T12:13:52.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbaea60a700 1 -- 172.21.15.187:0/1875703036 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc00f040 con 0x7fbad80039e0 2020-11-20T12:13:52.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbaea60a700 1 -- 172.21.15.187:0/1875703036 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc001780 con 0x7fbad80039e0 2020-11-20T12:13:52.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbafad67d00 1 -- 172.21.15.187:0/1875703036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 msgr2=0x557274f4f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbafad67d00 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f4f890 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7fbadc003a70 tx=0x7fbadc008040).stop 2020-11-20T12:13:52.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbafad67d00 1 -- 172.21.15.187:0/1875703036 shutdown_connections 2020-11-20T12:13:52.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbafad67d00 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557275004840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbafad67d00 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f4f890 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.877+0000 7fbafad67d00 1 --2- 172.21.15.187:0/1875703036 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 0x557274f50250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.878+0000 7fbafad67d00 1 -- 172.21.15.187:0/1875703036 >> 172.21.15.187:0/1875703036 conn(0x557274f2af00 msgr2=0x557274f4ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:52.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.878+0000 7fbafad67d00 1 -- 172.21.15.187:0/1875703036 shutdown_connections 2020-11-20T12:13:52.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.878+0000 7fbafad67d00 1 -- 172.21.15.187:0/1875703036 wait complete. 2020-11-20T12:13:52.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.879+0000 7fbafad67d00 1 Processor -- start 2020-11-20T12:13:52.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.879+0000 7fbafad67d00 1 -- start start 2020-11-20T12:13:52.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.879+0000 7fbafad67d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 0x557274f65290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbafad67d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557274ff1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbafad67d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f65fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbafad67d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x557274f65780 con 0x557274ff5a00 2020-11-20T12:13:52.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbafad67d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x557274f648b0 con 0x7fbad80039e0 2020-11-20T12:13:52.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbafad67d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x557274f64b50 con 0x5572750043c0 2020-11-20T12:13:52.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbaebe0d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f65fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbaebe0d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f65fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48300/0 (socket says 172.21.15.187:48300) 2020-11-20T12:13:52.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbaebe0d700 1 -- 172.21.15.187:0/1071025619 learned_addr learned my addr 172.21.15.187:0/1071025619 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:52.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbaeb60c700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 0x557274f65290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbae2e0b700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557274ff1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbaebe0d700 1 -- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 msgr2=0x557274ff1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbaebe0d700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557274ff1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbaebe0d700 1 -- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 msgr2=0x557274f65290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.880+0000 7fbaebe0d700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 0x557274f65290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbaebe0d700 1 -- 172.21.15.187:0/1071025619 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557274eab250 con 0x7fbad80039e0 2020-11-20T12:13:52.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbaeb60c700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557274ff5a00 0x557274f65290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:52.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbae2e0b700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5572750043c0 0x557274ff1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:52.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbaebe0d700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f65fc0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7fbae4009fd0 tx=0x557274f310f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:52.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbae8e07700 1 -- 172.21.15.187:0/1071025619 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae400fc10 con 0x7fbad80039e0 2020-11-20T12:13:52.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbae8e07700 1 -- 172.21.15.187:0/1071025619 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae400fdb0 con 0x7fbad80039e0 2020-11-20T12:13:52.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557274f64df0 con 0x7fbad80039e0 2020-11-20T12:13:52.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbae8e07700 1 -- 172.21.15.187:0/1071025619 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae401a690 con 0x7fbad80039e0 2020-11-20T12:13:52.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.881+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557274ff1720 con 0x7fbad80039e0 2020-11-20T12:13:52.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.883+0000 7fbae8e07700 1 -- 172.21.15.187:0/1071025619 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbae401aac0 con 0x7fbad80039e0 2020-11-20T12:13:52.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.884+0000 7fbae8e07700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbacc0559a0 0x7fbacc057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.884+0000 7fbae8e07700 1 -- 172.21.15.187:0/1071025619 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbae40712e0 con 0x7fbad80039e0 2020-11-20T12:13:52.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.884+0000 7fbaeb60c700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbacc0559a0 0x7fbacc057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.884+0000 7fbafad67d00 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x557274f664b0 0x5572750178f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:52.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.884+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:1 18.0 18:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected e89) v8 -- 0x557274ff3420 con 0x557274f664b0 2020-11-20T12:13:52.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.885+0000 7fbaeb60c700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbacc0559a0 0x7fbacc057e60 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fbadc003960 tx=0x7fbadc014000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:52.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.885+0000 7fbae2e0b700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x557274f664b0 0x5572750178f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:52.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.886+0000 7fbae2e0b700 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x557274f664b0 0x5572750178f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:52.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbae2e0b700 1 -- 172.21.15.187:0/1071025619 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (62) out=62b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+62 (crc 0 0 0) 0x7fbad4008b10 con 0x557274f664b0 2020-11-20T12:13:52.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x557274f664b0 msgr2=0x5572750178f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbafad67d00 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x557274f664b0 0x5572750178f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:52.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbacc0559a0 msgr2=0x7fbacc057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbafad67d00 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbacc0559a0 0x7fbacc057e60 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fbadc003960 tx=0x7fbadc014000).stop 2020-11-20T12:13:52.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 msgr2=0x557274f65fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:52.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbafad67d00 1 --2- 172.21.15.187:0/1071025619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbad80039e0 0x557274f65fc0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7fbae4009fd0 tx=0x557274f310f0).stop 2020-11-20T12:13:52.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbaebe0d700 1 -- 172.21.15.187:0/1071025619 reap_dead start 2020-11-20T12:13:52.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 shutdown_connections 2020-11-20T12:13:52.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.888+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 >> 172.21.15.187:0/1071025619 conn(0x557274f2af00 msgr2=0x557274ffc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:52.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.889+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 shutdown_connections 2020-11-20T12:13:52.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:52.889+0000 7fbafad67d00 1 -- 172.21.15.187:0/1071025619 wait complete. 2020-11-20T12:13:52.900 INFO:teuthology.orchestra.run.smithi187.stdout:8file_a 2020-11-20T12:13:52.901 INFO:teuthology.orchestra.run.smithi187:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2020-11-20T12:13:52.997 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:13:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: "ino": 1099511627776, 2020-11-20T12:13:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: "ancestors": [ 2020-11-20T12:13:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:52.999 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1, 2020-11-20T12:13:52.999 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "file_a", 2020-11-20T12:13:52.999 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 6 2020-11-20T12:13:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2020-11-20T12:13:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: "pool": 18, 2020-11-20T12:13:53.001 INFO:teuthology.orchestra.run.smithi187.stdout: "old_pools": [] 2020-11-20T12:13:53.001 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:13:53.001 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:53.004 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data getxattr 10000000001.00000000 parent 2020-11-20T12:13:53.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.103+0000 7fda4654cd00 1 Processor -- start 2020-11-20T12:13:53.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.103+0000 7fda4654cd00 1 -- start start 2020-11-20T12:13:53.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda4654cd00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888db600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda4654cd00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b888de3f0 0x560b8884eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda4654cd00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b8884f450 0x560b8884f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda4654cd00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b887bce30 con 0x560b8884f450 2020-11-20T12:13:53.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda4654cd00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b8878b810 con 0x560b8880fd80 2020-11-20T12:13:53.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda4654cd00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b888dc050 con 0x560b888de3f0 2020-11-20T12:13:53.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda36df1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888db600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda36df1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888db600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48318/0 (socket says 172.21.15.187:48318) 2020-11-20T12:13:53.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda36df1700 1 -- 172.21.15.187:0/2047976327 learned_addr learned my addr 172.21.15.187:0/2047976327 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:53.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda36df1700 1 -- 172.21.15.187:0/2047976327 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b888de3f0 msgr2=0x560b8884eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:53.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda36df1700 1 --2- 172.21.15.187:0/2047976327 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b888de3f0 0x560b8884eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:53.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda375f2700 1 --2- 172.21.15.187:0/2047976327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b8884f450 0x560b8884f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda36df1700 1 -- 172.21.15.187:0/2047976327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b8884f450 msgr2=0x560b8884f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:53.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda36df1700 1 --2- 172.21.15.187:0/2047976327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b8884f450 0x560b8884f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.104+0000 7fda36df1700 1 -- 172.21.15.187:0/2047976327 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b887b0ec0 con 0x560b8880fd80 2020-11-20T12:13:53.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.105+0000 7fda36df1700 1 --2- 172.21.15.187:0/2047976327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888db600 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7fda28009b40 tx=0x7fda28005ca0).ready entity=mon.1 client_cookie=22fd3adeb2538701 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:53.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.105+0000 7fda355ee700 1 -- 172.21.15.187:0/2047976327 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28008b80 con 0x560b8880fd80 2020-11-20T12:13:53.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.105+0000 7fda355ee700 1 -- 172.21.15.187:0/2047976327 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda28010040 con 0x560b8880fd80 2020-11-20T12:13:53.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.105+0000 7fda355ee700 1 -- 172.21.15.187:0/2047976327 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280143f0 con 0x560b8880fd80 2020-11-20T12:13:53.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 -- 172.21.15.187:0/2047976327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 msgr2=0x560b888db600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 --2- 172.21.15.187:0/2047976327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888db600 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7fda28009b40 tx=0x7fda28005ca0).stop 2020-11-20T12:13:53.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 -- 172.21.15.187:0/2047976327 shutdown_connections 2020-11-20T12:13:53.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 --2- 172.21.15.187:0/2047976327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b8884f450 0x560b8884f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 --2- 172.21.15.187:0/2047976327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888db600 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 --2- 172.21.15.187:0/2047976327 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b888de3f0 0x560b8884eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:53.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 -- 172.21.15.187:0/2047976327 >> 172.21.15.187:0/2047976327 conn(0x560b8880ef00 msgr2=0x560b888de970 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:53.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 -- 172.21.15.187:0/2047976327 shutdown_connections 2020-11-20T12:13:53.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.106+0000 7fda4654cd00 1 -- 172.21.15.187:0/2047976327 wait complete. 2020-11-20T12:13:53.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.107+0000 7fda4654cd00 1 Processor -- start 2020-11-20T12:13:53.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.107+0000 7fda4654cd00 1 -- start start 2020-11-20T12:13:53.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda4654cd00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888e4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda4654cd00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b8884f450 0x560b888e31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda4654cd00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b888de3f0 0x560b888db280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda4654cd00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b888e0c20 con 0x560b888de3f0 2020-11-20T12:13:53.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda4654cd00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b888e9b70 con 0x560b8880fd80 2020-11-20T12:13:53.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda4654cd00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b888e9da0 con 0x560b8884f450 2020-11-20T12:13:53.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda36df1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda36df1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888e4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48324/0 (socket says 172.21.15.187:48324) 2020-11-20T12:13:53.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.108+0000 7fda36df1700 1 -- 172.21.15.187:0/1835201955 learned_addr learned my addr 172.21.15.187:0/1835201955 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:53.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda36df1700 1 -- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b8884f450 msgr2=0x560b888e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda36df1700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b8884f450 0x560b888e31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:53.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda36df1700 1 -- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b888de3f0 msgr2=0x560b888db280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda375f2700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b888de3f0 0x560b888db280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda36df1700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x560b888de3f0 0x560b888db280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda36df1700 1 -- 172.21.15.187:0/1835201955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b8878f250 con 0x560b8880fd80 2020-11-20T12:13:53.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda36df1700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888e4320 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x560b888dd8c0 tx=0x7fda28017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:53.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda277fe700 1 -- 172.21.15.187:0/1835201955 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280059e0 con 0x560b8880fd80 2020-11-20T12:13:53.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda277fe700 1 -- 172.21.15.187:0/1835201955 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2801d410 con 0x560b8880fd80 2020-11-20T12:13:53.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda277fe700 1 -- 172.21.15.187:0/1835201955 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2801d790 con 0x560b8880fd80 2020-11-20T12:13:53.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.109+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560b888e3be0 con 0x560b8880fd80 2020-11-20T12:13:53.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.110+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b888e4080 con 0x560b8880fd80 2020-11-20T12:13:53.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.111+0000 7fda277fe700 1 -- 172.21.15.187:0/1835201955 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fda28024030 con 0x560b8880fd80 2020-11-20T12:13:53.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.112+0000 7fda277fe700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda1c0559a0 0x7fda1c057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.112+0000 7fda277fe700 1 -- 172.21.15.187:0/1835201955 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fda28070030 con 0x560b8880fd80 2020-11-20T12:13:53.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.112+0000 7fda365f0700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda1c0559a0 0x7fda1c057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.112+0000 7fda4654cd00 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x560b888e1970 0x560b8884d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.112+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:1 18.7 18:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected e89) v8 -- 0x560b888e36d0 con 0x560b888e1970 2020-11-20T12:13:53.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.113+0000 7fda365f0700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda1c0559a0 0x7fda1c057e60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fda2c000dd0 tx=0x7fda2c001660).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:53.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.113+0000 7fda375f2700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x560b888e1970 0x560b8884d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.114+0000 7fda375f2700 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x560b888e1970 0x560b8884d520 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:53.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda375f2700 1 -- 172.21.15.187:0/1835201955 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (62) out=62b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+62 (crc 0 0 0) 0x7fda30018040 con 0x560b888e1970 2020-11-20T12:13:53.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x560b888e1970 msgr2=0x560b8884d520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x560b888e1970 0x560b8884d520 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda1c0559a0 msgr2=0x7fda1c057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda1c0559a0 0x7fda1c057e60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fda2c000dd0 tx=0x7fda2c001660).stop 2020-11-20T12:13:53.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 msgr2=0x560b888e4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 --2- 172.21.15.187:0/1835201955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560b8880fd80 0x560b888e4320 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x560b888dd8c0 tx=0x7fda28017000).stop 2020-11-20T12:13:53.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda375f2700 1 -- 172.21.15.187:0/1835201955 reap_dead start 2020-11-20T12:13:53.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 shutdown_connections 2020-11-20T12:13:53.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 >> 172.21.15.187:0/1835201955 conn(0x560b8880ef00 msgr2=0x560b888105e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:53.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 shutdown_connections 2020-11-20T12:13:53.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.116+0000 7fda4654cd00 1 -- 172.21.15.187:0/1835201955 wait complete. 2020-11-20T12:13:53.139 INFO:teuthology.orchestra.run.smithi187.stdout:8file_b  2020-11-20T12:13:53.139 INFO:teuthology.orchestra.run.smithi187:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2020-11-20T12:13:53.239 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:13:53.240 INFO:teuthology.orchestra.run.smithi187.stdout: "ino": 1099511627777, 2020-11-20T12:13:53.240 INFO:teuthology.orchestra.run.smithi187.stdout: "ancestors": [ 2020-11-20T12:13:53.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:13:53.241 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1, 2020-11-20T12:13:53.241 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "file_b", 2020-11-20T12:13:53.242 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 12 2020-11-20T12:13:53.242 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:13:53.242 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2020-11-20T12:13:53.243 INFO:teuthology.orchestra.run.smithi187.stdout: "pool": 18, 2020-11-20T12:13:53.243 INFO:teuthology.orchestra.run.smithi187.stdout: "old_pools": [] 2020-11-20T12:13:53.244 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:13:53.244 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:53.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:13:53.247 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mv file_a file_b) 2020-11-20T12:13:53.358 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:53.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.599+0000 7fad52528700 1 Processor -- start 2020-11-20T12:13:53.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.599+0000 7fad52528700 1 -- start start 2020-11-20T12:13:53.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad52528700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fad4c065190 0x7fad4c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad52528700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fad4c09ca90 0x7fad4c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad52528700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c0b13e0 con 0x7fad4c065f90 2020-11-20T12:13:53.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad52528700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c14ab30 con 0x7fad4c065190 2020-11-20T12:13:53.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad52528700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c062c10 con 0x7fad4c09ca90 2020-11-20T12:13:53.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad50d25700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fad4c065f90 0x7fad4c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44942/0 (socket says 172.21.15.187:44942) 2020-11-20T12:13:53.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad51d27700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fad4c09ca90 0x7fad4c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad50d25700 1 -- 172.21.15.187:0/1410088746 learned_addr learned my addr 172.21.15.187:0/1410088746 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:53.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.601+0000 7fad51526700 1 --2- 172.21.15.187:0/1410088746 >> v2:172.21.15.5:3300/0 conn(0x7fad4c065190 0x7fad4c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.602+0000 7fad51d27700 1 -- 172.21.15.187:0/1410088746 >> v2:172.21.15.5:3300/0 conn(0x7fad4c065190 msgr2=0x7fad4c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:53.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.602+0000 7fad51d27700 1 --2- 172.21.15.187:0/1410088746 >> v2:172.21.15.5:3300/0 conn(0x7fad4c065190 0x7fad4c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.602+0000 7fad51d27700 1 -- 172.21.15.187:0/1410088746 >> v1:172.21.15.187:6789/0 conn(0x7fad4c065f90 legacy=0x7fad4c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.602+0000 7fad51d27700 1 -- 172.21.15.187:0/1410088746 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c0b1640 con 0x7fad4c09ca90 2020-11-20T12:13:53.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.602+0000 7fad3bfff700 1 -- 172.21.15.187:0/1410088746 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056424769 0 0) 0x7fad4c0b13e0 con 0x7fad4c065f90 2020-11-20T12:13:53.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.602+0000 7fad51d27700 1 --2- 172.21.15.187:0/1410088746 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c09ca90 0x7fad4c09cf20 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0x7fad4c099f90 tx=0x7fad440091d0).ready entity=mon.1 client_cookie=acebcdc2a676c049 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:53.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.602+0000 7fad3bfff700 1 -- 172.21.15.187:0/1410088746 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4400bc50 con 0x7fad4c09ca90 2020-11-20T12:13:53.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad3bfff700 1 -- 172.21.15.187:0/1410088746 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4400bdb0 con 0x7fad4c09ca90 2020-11-20T12:13:53.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad3bfff700 1 -- 172.21.15.187:0/1410088746 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44011610 con 0x7fad4c09ca90 2020-11-20T12:13:53.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad52528700 1 -- 172.21.15.187:0/1410088746 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c09ca90 msgr2=0x7fad4c09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad52528700 1 --2- 172.21.15.187:0/1410088746 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c09ca90 0x7fad4c09cf20 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0x7fad4c099f90 tx=0x7fad440091d0).stop 2020-11-20T12:13:53.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad52528700 1 -- 172.21.15.187:0/1410088746 shutdown_connections 2020-11-20T12:13:53.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad52528700 1 --2- 172.21.15.187:0/1410088746 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c09ca90 0x7fad4c09cf20 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad52528700 1 --2- 172.21.15.187:0/1410088746 >> v2:172.21.15.5:3300/0 conn(0x7fad4c065190 0x7fad4c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad52528700 1 -- 172.21.15.187:0/1410088746 >> 172.21.15.187:0/1410088746 conn(0x7fad4c0bab30 msgr2=0x7fad4c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:53.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad52528700 1 -- 172.21.15.187:0/1410088746 shutdown_connections 2020-11-20T12:13:53.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.603+0000 7fad52528700 1 -- 172.21.15.187:0/1410088746 wait complete. 2020-11-20T12:13:53.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.605+0000 7fad52528700 1 Processor -- start 2020-11-20T12:13:53.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.605+0000 7fad52528700 1 -- start start 2020-11-20T12:13:53.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.606+0000 7fad52528700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c065190 0x7fad4c143460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.606+0000 7fad52528700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c065f90 0x7fad4c0b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.606+0000 7fad52528700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c09ca90 0x7fad4c144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.606+0000 7fad52528700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c14acc0 con 0x7fad4c065f90 2020-11-20T12:13:53.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.606+0000 7fad52528700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c14aef0 con 0x7fad4c065190 2020-11-20T12:13:53.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.606+0000 7fad52528700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c14b190 con 0x7fad4c09ca90 2020-11-20T12:13:53.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c065190 0x7fad4c143460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c065190 0x7fad4c143460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48340/0 (socket says 172.21.15.187:48340) 2020-11-20T12:13:53.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 -- 172.21.15.187:0/4199797513 learned_addr learned my addr 172.21.15.187:0/4199797513 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:53.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad50d25700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c065f90 0x7fad4c0b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 -- 172.21.15.187:0/4199797513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c09ca90 msgr2=0x7fad4c144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c09ca90 0x7fad4c144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:53.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 -- 172.21.15.187:0/4199797513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c065f90 msgr2=0x7fad4c0b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c065f90 0x7fad4c0b3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 -- 172.21.15.187:0/4199797513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c143950 con 0x7fad4c065190 2020-11-20T12:13:53.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.607+0000 7fad51526700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c065190 0x7fad4c143460 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fad4800db90 tx=0x7fad48008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:53.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.608+0000 7fad397fa700 1 -- 172.21.15.187:0/4199797513 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48009790 con 0x7fad4c065190 2020-11-20T12:13:53.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.608+0000 7fad397fa700 1 -- 172.21.15.187:0/4199797513 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad480098f0 con 0x7fad4c065190 2020-11-20T12:13:53.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.608+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c0b35c0 con 0x7fad4c065190 2020-11-20T12:13:53.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.608+0000 7fad397fa700 1 -- 172.21.15.187:0/4199797513 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48009c90 con 0x7fad4c065190 2020-11-20T12:13:53.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.608+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c149b90 con 0x7fad4c065190 2020-11-20T12:13:53.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.609+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c061a20 con 0x7fad4c065190 2020-11-20T12:13:53.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.609+0000 7fad397fa700 1 -- 172.21.15.187:0/4199797513 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fad48013480 con 0x7fad4c065190 2020-11-20T12:13:53.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.610+0000 7fad397fa700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad24056d20 0x7fad240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:53.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.610+0000 7fad397fa700 1 -- 172.21.15.187:0/4199797513 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fad48070540 con 0x7fad4c065190 2020-11-20T12:13:53.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.610+0000 7fad50d25700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad24056d20 0x7fad240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.611+0000 7fad50d25700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad24056d20 0x7fad240591e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fad4c15b5a0 tx=0x7fad3c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.614+0000 7fad397fa700 1 -- 172.21.15.187:0/4199797513 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fad48045930 con 0x7fad4c065190 2020-11-20T12:13:53.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.862+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad4c0afc60 con 0x7fad4c065190 2020-11-20T12:13:53.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.863+0000 7fad397fa700 1 -- 172.21.15.187:0/4199797513 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fad48048f50 con 0x7fad4c065190 2020-11-20T12:13:53.868 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:53.868 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:13:53.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad24056d20 msgr2=0x7fad240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad24056d20 0x7fad240591e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fad4c15b5a0 tx=0x7fad3c009040).stop 2020-11-20T12:13:53.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c065190 msgr2=0x7fad4c143460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:53.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c065190 0x7fad4c143460 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fad4800db90 tx=0x7fad48008040).stop 2020-11-20T12:13:53.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 shutdown_connections 2020-11-20T12:13:53.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad24056d20 0x7fad240591e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad4c09ca90 0x7fad4c144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:53.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad4c065190 0x7fad4c143460 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 --2- 172.21.15.187:0/4199797513 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad4c065f90 0x7fad4c0b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:53.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 >> 172.21.15.187:0/4199797513 conn(0x7fad4c0bab30 msgr2=0x7fad4c09c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:53.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 shutdown_connections 2020-11-20T12:13:53.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:53.866+0000 7fad52528700 1 -- 172.21.15.187:0/4199797513 wait complete. 2020-11-20T12:13:53.876 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:13:53.894 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:13:54.170 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_created":1}} 2020-11-20T12:13:54.171 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2020-11-20T12:13:54.172 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:54.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.414+0000 7f952d82a700 1 Processor -- start 2020-11-20T12:13:54.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.414+0000 7f952d82a700 1 -- start start 2020-11-20T12:13:54.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.415+0000 7f952d82a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9528065f90 0x7f9528065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:54.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.415+0000 7f952d82a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f952809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:54.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.415+0000 7f952d82a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95280b13e0 con 0x7f9528065190 2020-11-20T12:13:54.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.415+0000 7f952d82a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952814ab30 con 0x7f9528065f90 2020-11-20T12:13:54.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.415+0000 7f952d82a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9528062c10 con 0x7f952809ca90 2020-11-20T12:13:54.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.416+0000 7f952c828700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9528065190 0x7f9528143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44954/0 (socket says 172.21.15.187:44954) 2020-11-20T12:13:54.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.416+0000 7f952c828700 1 -- 172.21.15.187:0/3843607575 learned_addr learned my addr 172.21.15.187:0/3843607575 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:13:54.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.416+0000 7f951ffff700 1 --2- 172.21.15.187:0/3843607575 >> v2:172.21.15.5:3300/0 conn(0x7f9528065f90 0x7f9528065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:54.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.416+0000 7f952d029700 1 --2- 172.21.15.187:0/3843607575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f952809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:54.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.417+0000 7f951f7fe700 1 -- 172.21.15.187:0/3843607575 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194227293 0 0) 0x7f95280b13e0 con 0x7f9528065190 2020-11-20T12:13:54.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.417+0000 7f951f7fe700 1 -- 172.21.15.187:0/3843607575 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9514003240 con 0x7f9528065190 2020-11-20T12:13:54.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.417+0000 7f952d029700 1 -- 172.21.15.187:0/3843607575 >> v2:172.21.15.5:3300/0 conn(0x7f9528065f90 msgr2=0x7f9528065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:54.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.417+0000 7f952d029700 1 --2- 172.21.15.187:0/3843607575 >> v2:172.21.15.5:3300/0 conn(0x7f9528065f90 0x7f9528065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.417+0000 7f952d029700 1 -- 172.21.15.187:0/3843607575 >> v1:172.21.15.187:6789/0 conn(0x7f9528065190 legacy=0x7f9528143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:54.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.417+0000 7f952d029700 1 -- 172.21.15.187:0/3843607575 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95280b1640 con 0x7f952809ca90 2020-11-20T12:13:54.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.418+0000 7f952d029700 1 --2- 172.21.15.187:0/3843607575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f952809cf40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f952000b9a0 tx=0x7f9520006df0).ready entity=mon.2 client_cookie=7c5063f7126653ce server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:54.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.418+0000 7f951f7fe700 1 -- 172.21.15.187:0/3843607575 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520009710 con 0x7f952809ca90 2020-11-20T12:13:54.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.418+0000 7f951f7fe700 1 -- 172.21.15.187:0/3843607575 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9520009870 con 0x7f952809ca90 2020-11-20T12:13:54.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f951f7fe700 1 -- 172.21.15.187:0/3843607575 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520009bf0 con 0x7f952809ca90 2020-11-20T12:13:54.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f952d82a700 1 -- 172.21.15.187:0/3843607575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 msgr2=0x7f952809cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:54.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f952d82a700 1 --2- 172.21.15.187:0/3843607575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f952809cf40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f952000b9a0 tx=0x7f9520006df0).stop 2020-11-20T12:13:54.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f952d82a700 1 -- 172.21.15.187:0/3843607575 shutdown_connections 2020-11-20T12:13:54.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f952d82a700 1 --2- 172.21.15.187:0/3843607575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f952809cf40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f952d82a700 1 --2- 172.21.15.187:0/3843607575 >> v2:172.21.15.5:3300/0 conn(0x7f9528065f90 0x7f9528065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f952d82a700 1 -- 172.21.15.187:0/3843607575 >> 172.21.15.187:0/3843607575 conn(0x7f95280bab30 msgr2=0x7f95280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:54.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f952d82a700 1 -- 172.21.15.187:0/3843607575 shutdown_connections 2020-11-20T12:13:54.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.419+0000 7f952d82a700 1 -- 172.21.15.187:0/3843607575 wait complete. 2020-11-20T12:13:54.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.421+0000 7f952d82a700 1 Processor -- start 2020-11-20T12:13:54.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.421+0000 7f952d82a700 1 -- start start 2020-11-20T12:13:54.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.422+0000 7f952d82a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9528065190 0x7f9528139040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:54.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.422+0000 7f952d82a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9528065f90 0x7f9528139530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:54.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.422+0000 7f952d82a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f95280a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:54.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.422+0000 7f952d82a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95280b3650 con 0x7f9528065f90 2020-11-20T12:13:54.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.422+0000 7f952d82a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95280b38f0 con 0x7f9528065190 2020-11-20T12:13:54.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.422+0000 7f952d82a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95280b25b0 con 0x7f952809ca90 2020-11-20T12:13:54.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952c828700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9528065190 0x7f9528139040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:54.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952c828700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9528065190 0x7f9528139040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48354/0 (socket says 172.21.15.187:48354) 2020-11-20T12:13:54.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952c828700 1 -- 172.21.15.187:0/1784419977 learned_addr learned my addr 172.21.15.187:0/1784419977 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:13:54.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f951ffff700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9528065f90 0x7f9528139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:54.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952c828700 1 -- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 msgr2=0x7f95280a00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:54.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952d029700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f95280a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:54.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952c828700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f95280a00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952c828700 1 -- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9528065f90 msgr2=0x7f9528139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:54.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952c828700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9528065f90 0x7f9528139530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f952c828700 1 -- 172.21.15.187:0/1784419977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95280b2770 con 0x7f9528065190 2020-11-20T12:13:54.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.423+0000 7f951ffff700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9528065f90 0x7f9528139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:54.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.424+0000 7f952c828700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9528065190 0x7f9528139040 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f952400ca60 tx=0x7f9524007e50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:54.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.424+0000 7f951cff9700 1 -- 172.21.15.187:0/1784419977 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524005b80 con 0x7f9528065190 2020-11-20T12:13:54.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.424+0000 7f951cff9700 1 -- 172.21.15.187:0/1784419977 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9524005ce0 con 0x7f9528065190 2020-11-20T12:13:54.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.424+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95280b1ff0 con 0x7f9528065190 2020-11-20T12:13:54.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.424+0000 7f951cff9700 1 -- 172.21.15.187:0/1784419977 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524007340 con 0x7f9528065190 2020-11-20T12:13:54.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.424+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95280a07d0 con 0x7f9528065190 2020-11-20T12:13:54.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.425+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95280b22f0 con 0x7f9528065190 2020-11-20T12:13:54.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.426+0000 7f951cff9700 1 -- 172.21.15.187:0/1784419977 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9524010030 con 0x7f9528065190 2020-11-20T12:13:54.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.426+0000 7f951cff9700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9500056d50 0x7f9500059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:54.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.426+0000 7f951cff9700 1 -- 172.21.15.187:0/1784419977 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f952406ffe0 con 0x7f9528065190 2020-11-20T12:13:54.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.427+0000 7f951ffff700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9500056d50 0x7f9500059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:54.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.427+0000 7f951ffff700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9500056d50 0x7f9500059210 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f9518006bd0 tx=0x7f9518008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:54.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.432+0000 7f951cff9700 1 -- 172.21.15.187:0/1784419977 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f95240453d0 con 0x7f9528065190 2020-11-20T12:13:54.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.681+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95280b2900 con 0x7f9528065190 2020-11-20T12:13:54.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.682+0000 7f951cff9700 1 -- 172.21.15.187:0/1784419977 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f95240489f0 con 0x7f9528065190 2020-11-20T12:13:54.685 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:13:54.685 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:13:54.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.684+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9500056d50 msgr2=0x7f9500059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:54.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.684+0000 7f952d82a700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9500056d50 0x7f9500059210 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f9518006bd0 tx=0x7f9518008040).stop 2020-11-20T12:13:54.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.684+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9528065190 msgr2=0x7f9528139040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:54.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.684+0000 7f952d82a700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9528065190 0x7f9528139040 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f952400ca60 tx=0x7f9524007e50).stop 2020-11-20T12:13:54.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.685+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 shutdown_connections 2020-11-20T12:13:54.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.685+0000 7f952d82a700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9500056d50 0x7f9500059210 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.685+0000 7f952d82a700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f952809ca90 0x7f95280a00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.685+0000 7f952d82a700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9528065190 0x7f9528139040 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.685+0000 7f952d82a700 1 --2- 172.21.15.187:0/1784419977 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9528065f90 0x7f9528139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:54.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.685+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 >> 172.21.15.187:0/1784419977 conn(0x7f95280bab30 msgr2=0x7f952809c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:54.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.685+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 shutdown_connections 2020-11-20T12:13:54.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:13:54.685+0000 7f952d82a700 1 -- 172.21.15.187:0/1784419977 wait complete. 2020-11-20T12:13:54.691 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:13:54.712 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:13:54.974 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:13:54.975 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:13:54.976 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2020-11-20T12:13:59.977 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:00.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.224+0000 7f6fcf59e700 1 Processor -- start 2020-11-20T12:14:00.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.224+0000 7f6fcf59e700 1 -- start start 2020-11-20T12:14:00.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.225+0000 7f6fcf59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6fd0065190 0x7f6fd0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:00.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.225+0000 7f6fcf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065f90 0x7f6fd0143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:00.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.225+0000 7f6fcf59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00b13e0 con 0x7f6fd0143c70 2020-11-20T12:14:00.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.225+0000 7f6fcf59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd014ab30 con 0x7f6fd0065190 2020-11-20T12:14:00.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.225+0000 7f6fcf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0062c10 con 0x7f6fd0065f90 2020-11-20T12:14:00.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.226+0000 7f6fce59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6fd0065190 0x7f6fd0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:00.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.226+0000 7f6fcdd9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065f90 0x7f6fd0143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:00.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.226+0000 7f6fced9d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6fd0143c70 0x7f6fd009ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:38482/0 (socket says 172.21.15.187:38482) 2020-11-20T12:14:00.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.226+0000 7f6fced9d700 1 -- 172.21.15.187:0/2241193596 learned_addr learned my addr 172.21.15.187:0/2241193596 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:00.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.227+0000 7f6fcdd9b700 1 -- 172.21.15.187:0/2241193596 >> v2:172.21.15.5:3300/0 conn(0x7f6fd0065190 msgr2=0x7f6fd0065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:00.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.227+0000 7f6fcdd9b700 1 --2- 172.21.15.187:0/2241193596 >> v2:172.21.15.5:3300/0 conn(0x7f6fd0065190 0x7f6fd0065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.227+0000 7f6fcdd9b700 1 -- 172.21.15.187:0/2241193596 >> v1:172.21.15.5:6789/0 conn(0x7f6fd0143c70 legacy=0x7f6fd009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:00.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.227+0000 7f6fcd59a700 1 -- 172.21.15.187:0/2241193596 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327300723 0 0) 0x7f6fd00b13e0 con 0x7f6fd0143c70 2020-11-20T12:14:00.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.227+0000 7f6fcdd9b700 1 -- 172.21.15.187:0/2241193596 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00b1640 con 0x7f6fd0065f90 2020-11-20T12:14:00.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.228+0000 7f6fcdd9b700 1 --2- 172.21.15.187:0/2241193596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065f90 0x7f6fd0143780 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6fd014ac70 tx=0x7f6fc4006df0).ready entity=mon.2 client_cookie=a4b5dd792ce9a61e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:00.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.228+0000 7f6fcd59a700 1 -- 172.21.15.187:0/2241193596 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4009840 con 0x7f6fd0065f90 2020-11-20T12:14:00.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.228+0000 7f6fcd59a700 1 -- 172.21.15.187:0/2241193596 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc40099a0 con 0x7f6fd0065f90 2020-11-20T12:14:00.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcd59a700 1 -- 172.21.15.187:0/2241193596 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4009d20 con 0x7f6fd0065f90 2020-11-20T12:14:00.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2241193596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065f90 msgr2=0x7f6fd0143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:00.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2241193596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065f90 0x7f6fd0143780 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6fd014ac70 tx=0x7f6fc4006df0).stop 2020-11-20T12:14:00.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2241193596 shutdown_connections 2020-11-20T12:14:00.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2241193596 >> v2:172.21.15.5:3300/0 conn(0x7f6fd0065190 0x7f6fd0065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2241193596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065f90 0x7f6fd0143780 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2241193596 >> 172.21.15.187:0/2241193596 conn(0x7f6fd00bab30 msgr2=0x7f6fd00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:00.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2241193596 shutdown_connections 2020-11-20T12:14:00.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.229+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2241193596 wait complete. 2020-11-20T12:14:00.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.231+0000 7f6fcf59e700 1 Processor -- start 2020-11-20T12:14:00.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.231+0000 7f6fcf59e700 1 -- start start 2020-11-20T12:14:00.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065190 0x7f6fd0148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:00.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcf59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd0065f90 0x7f6fd00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:00.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcf59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd0143c70 0x7f6fd009bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:00.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcf59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00a0a40 con 0x7f6fd0143c70 2020-11-20T12:14:00.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcf59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0147600 con 0x7f6fd0065f90 2020-11-20T12:14:00.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd01478a0 con 0x7f6fd0065190 2020-11-20T12:14:00.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcdd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd0065f90 0x7f6fd00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:00.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcdd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd0065f90 0x7f6fd00b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48368/0 (socket says 172.21.15.187:48368) 2020-11-20T12:14:00.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.232+0000 7f6fcdd9b700 1 -- 172.21.15.187:0/2447618690 learned_addr learned my addr 172.21.15.187:0/2447618690 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:00.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fce59c700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065190 0x7f6fd0148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:00.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fcdd9b700 1 -- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065190 msgr2=0x7f6fd0148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fcdd9b700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065190 0x7f6fd0148f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fced9d700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd0143c70 0x7f6fd009bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fcdd9b700 1 -- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd0143c70 msgr2=0x7f6fd009bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fcdd9b700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd0143c70 0x7f6fd009bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fcdd9b700 1 -- 172.21.15.187:0/2447618690 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0147b40 con 0x7f6fd0065f90 2020-11-20T12:14:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fced9d700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd0143c70 0x7f6fd009bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fcdd9b700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd0065f90 0x7f6fd00b1b40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f6fc4006e30 tx=0x7f6fc4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fbe7fc700 1 -- 172.21.15.187:0/2447618690 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4011e50 con 0x7f6fd0065f90 2020-11-20T12:14:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fbe7fc700 1 -- 172.21.15.187:0/2447618690 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc401e460 con 0x7f6fd0065f90 2020-11-20T12:14:00.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd009c0f0 con 0x7f6fd0065f90 2020-11-20T12:14:00.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fbe7fc700 1 -- 172.21.15.187:0/2447618690 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc401e820 con 0x7f6fd0065f90 2020-11-20T12:14:00.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.233+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd01472c0 con 0x7f6fd0065f90 2020-11-20T12:14:00.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.235+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0061a20 con 0x7f6fd0065f90 2020-11-20T12:14:00.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.235+0000 7f6fbe7fc700 1 -- 172.21.15.187:0/2447618690 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6fc4025030 con 0x7f6fd0065f90 2020-11-20T12:14:00.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.236+0000 7f6fbe7fc700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fa4056d50 0x7f6fa4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:00.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.236+0000 7f6fbe7fc700 1 -- 172.21.15.187:0/2447618690 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6fc4071800 con 0x7f6fd0065f90 2020-11-20T12:14:00.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.236+0000 7f6fce59c700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fa4056d50 0x7f6fa4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:00.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.237+0000 7f6fce59c700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fa4056d50 0x7f6fa4059210 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f6fc0006bd0 tx=0x7f6fc0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:00.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.240+0000 7f6fbe7fc700 1 -- 172.21.15.187:0/2447618690 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6fc4046ce0 con 0x7f6fd0065f90 2020-11-20T12:14:00.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.482+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fd00afc60 con 0x7f6fd0065f90 2020-11-20T12:14:00.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.483+0000 7f6fbe7fc700 1 -- 172.21.15.187:0/2447618690 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f6fc4023020 con 0x7f6fd0065f90 2020-11-20T12:14:00.487 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:00.487 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:14:00.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fa4056d50 msgr2=0x7f6fa4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:00.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fa4056d50 0x7f6fa4059210 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f6fc0006bd0 tx=0x7f6fc0008040).stop 2020-11-20T12:14:00.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd0065f90 msgr2=0x7f6fd00b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:00.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd0065f90 0x7f6fd00b1b40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f6fc4006e30 tx=0x7f6fc4015000).stop 2020-11-20T12:14:00.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 shutdown_connections 2020-11-20T12:14:00.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6fa4056d50 0x7f6fa4059210 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6fd0143c70 0x7f6fd009bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6fd0065190 0x7f6fd0148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 --2- 172.21.15.187:0/2447618690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fd0065f90 0x7f6fd00b1b40 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:00.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 >> 172.21.15.187:0/2447618690 conn(0x7f6fd00bab30 msgr2=0x7f6fd01498b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:00.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 shutdown_connections 2020-11-20T12:14:00.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:00.486+0000 7f6fcf59e700 1 -- 172.21.15.187:0/2447618690 wait complete. 2020-11-20T12:14:00.494 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:14:00.510 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:14:00.814 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:14:00.815 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:14:00.816 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2020-11-20T12:14:05.819 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:06.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.082+0000 7fdea857f700 1 Processor -- start 2020-11-20T12:14:06.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.082+0000 7fdea857f700 1 -- start start 2020-11-20T12:14:06.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.083+0000 7fdea857f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdea00661f0 0x7fdea00b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.083+0000 7fdea857f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0099ce0 0x7fdea00b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.083+0000 7fdea857f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea00b5df0 con 0x7fdea0065e10 2020-11-20T12:14:06.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.083+0000 7fdea857f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea009a440 con 0x7fdea00661f0 2020-11-20T12:14:06.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.083+0000 7fdea857f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea009a1a0 con 0x7fdea0099ce0 2020-11-20T12:14:06.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.084+0000 7fdea5b1a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdea00661f0 0x7fdea00b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.084+0000 7fdea5b1a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdea00661f0 0x7fdea00b62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48380/0 (socket says 172.21.15.187:48380) 2020-11-20T12:14:06.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.084+0000 7fdea5b1a700 1 -- 172.21.15.187:0/1017993866 learned_addr learned my addr 172.21.15.187:0/1017993866 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:06.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.084+0000 7fdea6b1c700 1 --2- 172.21.15.187:0/1017993866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0099ce0 0x7fdea00b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.085+0000 7fdea5b1a700 1 -- 172.21.15.187:0/1017993866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0099ce0 msgr2=0x7fdea00b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:06.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.085+0000 7fdea5b1a700 1 --2- 172.21.15.187:0/1017993866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0099ce0 0x7fdea00b67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.085+0000 7fdea5b1a700 1 -- 172.21.15.187:0/1017993866 >> v1:172.21.15.5:6789/0 conn(0x7fdea0065e10 legacy=0x7fdea0144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.085+0000 7fdea5b1a700 1 -- 172.21.15.187:0/1017993866 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea00b0160 con 0x7fdea00661f0 2020-11-20T12:14:06.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.085+0000 7fdea5b1a700 1 --2- 172.21.15.187:0/1017993866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea00661f0 0x7fdea00b62e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7fde94006fd0 tx=0x7fde94006e50).ready entity=mon.1 client_cookie=6b2b0ee80314bd3c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:06.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.085+0000 7fdea5319700 1 -- 172.21.15.187:0/1017993866 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94009900 con 0x7fdea00661f0 2020-11-20T12:14:06.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.085+0000 7fdea5319700 1 -- 172.21.15.187:0/1017993866 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde94009a60 con 0x7fdea00661f0 2020-11-20T12:14:06.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.085+0000 7fdea5319700 1 -- 172.21.15.187:0/1017993866 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94009de0 con 0x7fdea00661f0 2020-11-20T12:14:06.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.086+0000 7fdea857f700 1 -- 172.21.15.187:0/1017993866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea00661f0 msgr2=0x7fdea00b62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.086+0000 7fdea857f700 1 --2- 172.21.15.187:0/1017993866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea00661f0 0x7fdea00b62e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7fde94006fd0 tx=0x7fde94006e50).stop 2020-11-20T12:14:06.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.086+0000 7fdea857f700 1 -- 172.21.15.187:0/1017993866 shutdown_connections 2020-11-20T12:14:06.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.086+0000 7fdea857f700 1 --2- 172.21.15.187:0/1017993866 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0099ce0 0x7fdea00b67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.086+0000 7fdea857f700 1 --2- 172.21.15.187:0/1017993866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea00661f0 0x7fdea00b62e0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.086+0000 7fdea857f700 1 -- 172.21.15.187:0/1017993866 >> 172.21.15.187:0/1017993866 conn(0x7fdea00bacd0 msgr2=0x7fdea00bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:06.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.086+0000 7fdea857f700 1 -- 172.21.15.187:0/1017993866 shutdown_connections 2020-11-20T12:14:06.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.086+0000 7fdea857f700 1 -- 172.21.15.187:0/1017993866 wait complete. 2020-11-20T12:14:06.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.088+0000 7fdea857f700 1 Processor -- start 2020-11-20T12:14:06.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.088+0000 7fdea857f700 1 -- start start 2020-11-20T12:14:06.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.089+0000 7fdea857f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0065e10 0x7fdea009ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea857f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdea00661f0 0x7fdea00a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea857f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea0099ce0 0x7fdea0136160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea857f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea009dee0 con 0x7fdea00661f0 2020-11-20T12:14:06.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea857f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea009e180 con 0x7fdea0099ce0 2020-11-20T12:14:06.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea857f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea009e420 con 0x7fdea0065e10 2020-11-20T12:14:06.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea6b1c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea0099ce0 0x7fdea0136160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea6b1c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea0099ce0 0x7fdea0136160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48388/0 (socket says 172.21.15.187:48388) 2020-11-20T12:14:06.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea6b1c700 1 -- 172.21.15.187:0/3491420803 learned_addr learned my addr 172.21.15.187:0/3491420803 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:06.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea5b1a700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdea00661f0 0x7fdea00a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.090+0000 7fdea631b700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0065e10 0x7fdea009ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea6b1c700 1 -- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0065e10 msgr2=0x7fdea009ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea6b1c700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0065e10 0x7fdea009ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea6b1c700 1 -- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdea00661f0 msgr2=0x7fdea00a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea6b1c700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdea00661f0 0x7fdea00a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea6b1c700 1 -- 172.21.15.187:0/3491420803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea009cf00 con 0x7fdea0099ce0 2020-11-20T12:14:06.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea631b700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0065e10 0x7fdea009ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:06.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea6b1c700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea0099ce0 0x7fdea0136160 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7fde9800eb40 tx=0x7fde98008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:06.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fde927fc700 1 -- 172.21.15.187:0/3491420803 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9800c830 con 0x7fdea0099ce0 2020-11-20T12:14:06.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fde927fc700 1 -- 172.21.15.187:0/3491420803 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9800c990 con 0x7fdea0099ce0 2020-11-20T12:14:06.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea00a0a60 con 0x7fdea0099ce0 2020-11-20T12:14:06.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fde927fc700 1 -- 172.21.15.187:0/3491420803 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9800cd30 con 0x7fdea0099ce0 2020-11-20T12:14:06.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.091+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea00b4ed0 con 0x7fdea0099ce0 2020-11-20T12:14:06.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.092+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea009e5e0 con 0x7fdea0099ce0 2020-11-20T12:14:06.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.093+0000 7fde927fc700 1 -- 172.21.15.187:0/3491420803 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fde98013640 con 0x7fdea0099ce0 2020-11-20T12:14:06.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.094+0000 7fde927fc700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde78056d20 0x7fde780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.094+0000 7fde927fc700 1 -- 172.21.15.187:0/3491420803 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fde9806fd40 con 0x7fdea0099ce0 2020-11-20T12:14:06.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.094+0000 7fdea631b700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde78056d20 0x7fde780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.095+0000 7fdea631b700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde78056d20 0x7fde780591e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7fde9c001430 tx=0x7fde9c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:06.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.098+0000 7fde927fc700 1 -- 172.21.15.187:0/3491420803 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fde98045110 con 0x7fdea0099ce0 2020-11-20T12:14:06.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.341+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdea0063600 con 0x7fdea0099ce0 2020-11-20T12:14:06.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.342+0000 7fde927fc700 1 -- 172.21.15.187:0/3491420803 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fde98048730 con 0x7fdea0099ce0 2020-11-20T12:14:06.346 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:06.346 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:14:06.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.344+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde78056d20 msgr2=0x7fde780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.344+0000 7fdea857f700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde78056d20 0x7fde780591e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7fde9c001430 tx=0x7fde9c009040).stop 2020-11-20T12:14:06.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.344+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea0099ce0 msgr2=0x7fdea0136160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.344+0000 7fdea857f700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea0099ce0 0x7fdea0136160 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7fde9800eb40 tx=0x7fde98008040).stop 2020-11-20T12:14:06.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.345+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 shutdown_connections 2020-11-20T12:14:06.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.345+0000 7fdea857f700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde78056d20 0x7fde780591e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.345+0000 7fdea857f700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdea0099ce0 0x7fdea0136160 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.345+0000 7fdea857f700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdea0065e10 0x7fdea009ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.345+0000 7fdea857f700 1 --2- 172.21.15.187:0/3491420803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdea00661f0 0x7fdea00a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.345+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 >> 172.21.15.187:0/3491420803 conn(0x7fdea00bacd0 msgr2=0x7fdea0142580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:06.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.345+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 shutdown_connections 2020-11-20T12:14:06.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.345+0000 7fdea857f700 1 -- 172.21.15.187:0/3491420803 wait complete. 2020-11-20T12:14:06.352 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:14:06.370 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:14:06.639 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":1}} 2020-11-20T12:14:06.640 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2020-11-20T12:14:06.641 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:06.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.916+0000 7fecd2654700 1 Processor -- start 2020-11-20T12:14:06.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.917+0000 7fecd2654700 1 -- start start 2020-11-20T12:14:06.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.918+0000 7fecd2654700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7feccc0b6650 0x7feccc0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.918+0000 7fecd2654700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feccc0b1300 con 0x7feccc1484d0 2020-11-20T12:14:06.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.918+0000 7fecd2654700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feccc0b15a0 con 0x7feccc066f70 2020-11-20T12:14:06.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.918+0000 7fecd2654700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feccc067730 con 0x7feccc0b6650 2020-11-20T12:14:06.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.918+0000 7fecd1e53700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7feccc066f70 0x7feccc0aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45002/0 (socket says 172.21.15.187:45002) 2020-11-20T12:14:06.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.918+0000 7fecd1e53700 1 -- 172.21.15.187:0/3045051777 learned_addr learned my addr 172.21.15.187:0/3045051777 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:06.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.919+0000 7fecd0e51700 1 --2- 172.21.15.187:0/3045051777 >> v2:172.21.15.5:3300/0 conn(0x7feccc0b6650 0x7feccc0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.919+0000 7fecbbfff700 1 -- 172.21.15.187:0/3045051777 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860028827 0 0) 0x7feccc0b15a0 con 0x7feccc066f70 2020-11-20T12:14:06.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.919+0000 7fecbbfff700 1 -- 172.21.15.187:0/3045051777 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb4003240 con 0x7feccc066f70 2020-11-20T12:14:06.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.920+0000 7fecd0e51700 1 -- 172.21.15.187:0/3045051777 >> v1:172.21.15.187:6789/0 conn(0x7feccc066f70 legacy=0x7feccc0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.920+0000 7fecd0e51700 1 -- 172.21.15.187:0/3045051777 >> v1:172.21.15.5:6789/0 conn(0x7feccc1484d0 legacy=0x7feccc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.920+0000 7fecd0e51700 1 -- 172.21.15.187:0/3045051777 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0b1840 con 0x7feccc0b6650 2020-11-20T12:14:06.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.920+0000 7fecd0e51700 1 --2- 172.21.15.187:0/3045051777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 0x7feccc0afa90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fecbc00b9a0 tx=0x7fecbc006df0).ready entity=mon.0 client_cookie=4cd4a822878396e1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:06.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.920+0000 7fecbbfff700 1 -- 172.21.15.187:0/3045051777 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc0097b0 con 0x7feccc0b6650 2020-11-20T12:14:06.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecbbfff700 1 -- 172.21.15.187:0/3045051777 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecbc009910 con 0x7feccc0b6650 2020-11-20T12:14:06.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecbbfff700 1 -- 172.21.15.187:0/3045051777 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc009c90 con 0x7feccc0b6650 2020-11-20T12:14:06.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecd2654700 1 -- 172.21.15.187:0/3045051777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 msgr2=0x7feccc0afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecd2654700 1 --2- 172.21.15.187:0/3045051777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 0x7feccc0afa90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fecbc00b9a0 tx=0x7fecbc006df0).stop 2020-11-20T12:14:06.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecd2654700 1 -- 172.21.15.187:0/3045051777 shutdown_connections 2020-11-20T12:14:06.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecd2654700 1 --2- 172.21.15.187:0/3045051777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 0x7feccc0afa90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecd2654700 1 -- 172.21.15.187:0/3045051777 >> 172.21.15.187:0/3045051777 conn(0x7feccc0bab30 msgr2=0x7feccc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:06.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecd2654700 1 -- 172.21.15.187:0/3045051777 shutdown_connections 2020-11-20T12:14:06.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.921+0000 7fecd2654700 1 -- 172.21.15.187:0/3045051777 wait complete. 2020-11-20T12:14:06.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.923+0000 7fecd2654700 1 Processor -- start 2020-11-20T12:14:06.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.923+0000 7fecd2654700 1 -- start start 2020-11-20T12:14:06.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd2654700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feccc066f70 0x7feccc139690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd2654700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 0x7feccc0a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd2654700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feccc1484d0 0x7feccc09e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd2654700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc09de60 con 0x7feccc0b6650 2020-11-20T12:14:06.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd2654700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc09e100 con 0x7feccc1484d0 2020-11-20T12:14:06.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd2654700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc09e3a0 con 0x7feccc066f70 2020-11-20T12:14:06.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd1e53700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feccc1484d0 0x7feccc09e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd1e53700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feccc1484d0 0x7feccc09e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48402/0 (socket says 172.21.15.187:48402) 2020-11-20T12:14:06.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.924+0000 7fecd1e53700 1 -- 172.21.15.187:0/2181415810 learned_addr learned my addr 172.21.15.187:0/2181415810 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:06.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd0e51700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 0x7feccc0a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd1e53700 1 -- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feccc066f70 msgr2=0x7feccc139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd1652700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feccc066f70 0x7feccc139690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd1e53700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feccc066f70 0x7feccc139690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd1e53700 1 -- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 msgr2=0x7feccc0a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:06.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd1e53700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 0x7feccc0a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:06.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd1e53700 1 -- 172.21.15.187:0/2181415810 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc13c460 con 0x7feccc1484d0 2020-11-20T12:14:06.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd0e51700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 0x7feccc0a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:06.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd1652700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feccc066f70 0x7feccc139690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:06.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd1e53700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feccc1484d0 0x7feccc09e840 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7fecc400c6e0 tx=0x7fecc4007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:06.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecb97fa700 1 -- 172.21.15.187:0/2181415810 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4007320 con 0x7feccc1484d0 2020-11-20T12:14:06.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecb97fa700 1 -- 172.21.15.187:0/2181415810 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc4007480 con 0x7feccc1484d0 2020-11-20T12:14:06.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feccc0a09f0 con 0x7feccc1484d0 2020-11-20T12:14:06.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecb97fa700 1 -- 172.21.15.187:0/2181415810 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4005800 con 0x7feccc1484d0 2020-11-20T12:14:06.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.925+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feccc09ed30 con 0x7feccc1484d0 2020-11-20T12:14:06.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.927+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feccc09e560 con 0x7feccc1484d0 2020-11-20T12:14:06.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.927+0000 7fecb97fa700 1 -- 172.21.15.187:0/2181415810 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fecc4010030 con 0x7feccc1484d0 2020-11-20T12:14:06.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.928+0000 7fecb97fa700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feca4056d50 0x7feca4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:06.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.928+0000 7fecb97fa700 1 -- 172.21.15.187:0/2181415810 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fecc406fc10 con 0x7feccc1484d0 2020-11-20T12:14:06.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.928+0000 7fecd1652700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feca4056d50 0x7feca4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:06.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.929+0000 7fecd1652700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feca4056d50 0x7feca4059210 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7fecc8001b20 tx=0x7fecc8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:06.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:06.931+0000 7fecb97fa700 1 -- 172.21.15.187:0/2181415810 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fecc4044fe0 con 0x7feccc1484d0 2020-11-20T12:14:07.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.179+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feccc061a20 con 0x7feccc1484d0 2020-11-20T12:14:07.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.180+0000 7fecb97fa700 1 -- 172.21.15.187:0/2181415810 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fecc4048600 con 0x7feccc1484d0 2020-11-20T12:14:07.183 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:07.184 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:14:07.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.182+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feca4056d50 msgr2=0x7feca4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:07.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.182+0000 7fecd2654700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feca4056d50 0x7feca4059210 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7fecc8001b20 tx=0x7fecc8009040).stop 2020-11-20T12:14:07.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.182+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feccc1484d0 msgr2=0x7feccc09e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:07.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.182+0000 7fecd2654700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feccc1484d0 0x7feccc09e840 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7fecc400c6e0 tx=0x7fecc4007a60).stop 2020-11-20T12:14:07.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.183+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 shutdown_connections 2020-11-20T12:14:07.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.183+0000 7fecd2654700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7feca4056d50 0x7feca4059210 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:07.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.183+0000 7fecd2654700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feccc1484d0 0x7feccc09e840 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:07.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.183+0000 7fecd2654700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feccc066f70 0x7feccc139690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:07.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.183+0000 7fecd2654700 1 --2- 172.21.15.187:0/2181415810 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feccc0b6650 0x7feccc0a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:07.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.183+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 >> 172.21.15.187:0/2181415810 conn(0x7feccc0bab30 msgr2=0x7feccc0b9120 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:07.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.183+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 shutdown_connections 2020-11-20T12:14:07.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.183+0000 7fecd2654700 1 -- 172.21.15.187:0/2181415810 wait complete. 2020-11-20T12:14:07.188 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:14:07.207 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2020-11-20T12:14:07.470 INFO:teuthology.orchestra.run.smithi187.stdout:{"purge_queue":{"pq_executed":1}} 2020-11-20T12:14:07.471 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2020-11-20T12:14:07.472 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:07.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.729+0000 7f683e3b7700 1 Processor -- start 2020-11-20T12:14:07.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.729+0000 7f683e3b7700 1 -- start start 2020-11-20T12:14:07.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.730+0000 7f683e3b7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6838066f70 0x7f68380b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:07.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.730+0000 7f683e3b7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68380b1300 con 0x7f68381484d0 2020-11-20T12:14:07.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.731+0000 7f683e3b7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68380b15a0 con 0x7f68380b6650 2020-11-20T12:14:07.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.731+0000 7f683e3b7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6838067730 con 0x7f6838066f70 2020-11-20T12:14:07.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.731+0000 7f683cbb4700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f68380b6650 0x7f68380afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45012/0 (socket says 172.21.15.187:45012) 2020-11-20T12:14:07.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.731+0000 7f683cbb4700 1 -- 172.21.15.187:0/2989282805 learned_addr learned my addr 172.21.15.187:0/2989282805 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:07.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.731+0000 7f683dbb6700 1 --2- 172.21.15.187:0/2989282805 >> v2:172.21.15.187:3300/0 conn(0x7f6838066f70 0x7f68380b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:07.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.731+0000 7f6827fff700 1 -- 172.21.15.187:0/2989282805 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337917155 0 0) 0x7f68380b15a0 con 0x7f68380b6650 2020-11-20T12:14:07.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.731+0000 7f6827fff700 1 -- 172.21.15.187:0/2989282805 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6820003240 con 0x7f68380b6650 2020-11-20T12:14:07.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.732+0000 7f683dbb6700 1 -- 172.21.15.187:0/2989282805 >> v1:172.21.15.187:6789/0 conn(0x7f68380b6650 legacy=0x7f68380afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:07.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.732+0000 7f6827fff700 1 -- 172.21.15.187:0/2989282805 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3484151163 0 0) 0x7f6820003240 con 0x7f68380b6650 2020-11-20T12:14:07.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.732+0000 7f683dbb6700 1 -- 172.21.15.187:0/2989282805 >> v1:172.21.15.5:6789/0 conn(0x7f68381484d0 legacy=0x7f6838061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:07.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.732+0000 7f683dbb6700 1 -- 172.21.15.187:0/2989282805 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68380b1840 con 0x7f6838066f70 2020-11-20T12:14:07.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.732+0000 7f683dbb6700 1 --2- 172.21.15.187:0/2989282805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6838066f70 0x7f68380b01a0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f6830009c10 tx=0x7f68300091d0).ready entity=mon.1 client_cookie=5889f06f2de9f026 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:07.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.732+0000 7f6827fff700 1 -- 172.21.15.187:0/2989282805 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280014e0 con 0x7f6838066f70 2020-11-20T12:14:07.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.732+0000 7f6827fff700 1 -- 172.21.15.187:0/2989282805 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683000b790 con 0x7f6838066f70 2020-11-20T12:14:07.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.732+0000 7f6827fff700 1 -- 172.21.15.187:0/2989282805 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683000bb10 con 0x7f6838066f70 2020-11-20T12:14:07.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.733+0000 7f683e3b7700 1 -- 172.21.15.187:0/2989282805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6838066f70 msgr2=0x7f68380b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:07.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.733+0000 7f683e3b7700 1 --2- 172.21.15.187:0/2989282805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6838066f70 0x7f68380b01a0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f6830009c10 tx=0x7f68300091d0).stop 2020-11-20T12:14:07.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.733+0000 7f683e3b7700 1 -- 172.21.15.187:0/2989282805 shutdown_connections 2020-11-20T12:14:07.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.733+0000 7f683e3b7700 1 --2- 172.21.15.187:0/2989282805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6838066f70 0x7f68380b01a0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:07.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.733+0000 7f683e3b7700 1 -- 172.21.15.187:0/2989282805 >> 172.21.15.187:0/2989282805 conn(0x7f68380bab30 msgr2=0x7f68380baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:07.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.733+0000 7f683e3b7700 1 -- 172.21.15.187:0/2989282805 shutdown_connections 2020-11-20T12:14:07.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.733+0000 7f683e3b7700 1 -- 172.21.15.187:0/2989282805 wait complete. 2020-11-20T12:14:07.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.734+0000 7f683e3b7700 1 Processor -- start 2020-11-20T12:14:07.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.735+0000 7f683e3b7700 1 -- start start 2020-11-20T12:14:07.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.735+0000 7f683e3b7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6838066f70 0x7f683815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:07.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.735+0000 7f683e3b7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68380b6650 0x7f68380ba140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:07.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.735+0000 7f683e3b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68381484d0 0x7f68380b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:07.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.735+0000 7f683e3b7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683815c2b0 con 0x7f68380b6650 2020-11-20T12:14:07.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.735+0000 7f683e3b7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380b2060 con 0x7f68381484d0 2020-11-20T12:14:07.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.735+0000 7f683e3b7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68380b2300 con 0x7f6838066f70 2020-11-20T12:14:07.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68381484d0 0x7f68380b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:07.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68381484d0 0x7f68380b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48414/0 (socket says 172.21.15.187:48414) 2020-11-20T12:14:07.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 -- 172.21.15.187:0/4020178711 learned_addr learned my addr 172.21.15.187:0/4020178711 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:07.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 -- 172.21.15.187:0/4020178711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6838066f70 msgr2=0x7f683815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:07.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6838066f70 0x7f683815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:07.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 -- 172.21.15.187:0/4020178711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68380b6650 msgr2=0x7f68380ba140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:07.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68380b6650 0x7f68380ba140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:07.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 -- 172.21.15.187:0/4020178711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68380b25a0 con 0x7f68381484d0 2020-11-20T12:14:07.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.736+0000 7f683dbb6700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68381484d0 0x7f68380b2ce0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f6830011000 tx=0x7f6830016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:07.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.737+0000 7f68257fa700 1 -- 172.21.15.187:0/4020178711 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830006ea0 con 0x7f68381484d0 2020-11-20T12:14:07.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.737+0000 7f68257fa700 1 -- 172.21.15.187:0/4020178711 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683001c5f0 con 0x7f68381484d0 2020-11-20T12:14:07.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.737+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68380b2840 con 0x7f68381484d0 2020-11-20T12:14:07.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.737+0000 7f68257fa700 1 -- 172.21.15.187:0/4020178711 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683001c970 con 0x7f68381484d0 2020-11-20T12:14:07.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.737+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68380b1c40 con 0x7f68381484d0 2020-11-20T12:14:07.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.738+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68380b29d0 con 0x7f68381484d0 2020-11-20T12:14:07.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.739+0000 7f68257fa700 1 -- 172.21.15.187:0/4020178711 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6830017070 con 0x7f68381484d0 2020-11-20T12:14:07.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.739+0000 7f68257fa700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6810056d20 0x7f68100591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:07.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.739+0000 7f68257fa700 1 -- 172.21.15.187:0/4020178711 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f68300721a0 con 0x7f68381484d0 2020-11-20T12:14:07.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.740+0000 7f683d3b5700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6810056d20 0x7f68100591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:07.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.740+0000 7f683d3b5700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6810056d20 0x7f68100591e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f6834006fd0 tx=0x7f6834006930).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:07.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.743+0000 7f68257fa700 1 -- 172.21.15.187:0/4020178711 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f68300475f0 con 0x7f68381484d0 2020-11-20T12:14:07.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.990+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6838061a20 con 0x7f68381484d0 2020-11-20T12:14:07.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.991+0000 7f68257fa700 1 -- 172.21.15.187:0/4020178711 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f6830015070 con 0x7f68381484d0 2020-11-20T12:14:07.995 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:07.996 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:14:07.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6810056d20 msgr2=0x7f68100591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:07.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6810056d20 0x7f68100591e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f6834006fd0 tx=0x7f6834006930).stop 2020-11-20T12:14:08.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68381484d0 msgr2=0x7f68380b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:08.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68381484d0 0x7f68380b2ce0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f6830011000 tx=0x7f6830016000).stop 2020-11-20T12:14:08.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 shutdown_connections 2020-11-20T12:14:08.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6810056d20 0x7f68100591e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68381484d0 0x7f68380b2ce0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6838066f70 0x7f683815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:08.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 --2- 172.21.15.187:0/4020178711 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68380b6650 0x7f68380ba140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:08.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 >> 172.21.15.187:0/4020178711 conn(0x7f68380bab30 msgr2=0x7f68380afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:08.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 shutdown_connections 2020-11-20T12:14:08.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:07.993+0000 7f683e3b7700 1 -- 172.21.15.187:0/4020178711 wait complete. 2020-11-20T12:14:08.003 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:14:08.019 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache 2020-11-20T12:14:08.287 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1,"strays_enqueued":1,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2020-11-20T12:14:08.288 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 0 } } 2020-11-20T12:14:08.288 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:08.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.523+0000 7ff14b59e700 1 Processor -- start 2020-11-20T12:14:08.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.523+0000 7ff14b59e700 1 -- start start 2020-11-20T12:14:08.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.525+0000 7ff14b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 0x7ff14c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:08.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.525+0000 7ff14b59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff14c065f90 0x7ff14c143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:08.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.525+0000 7ff14b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0b13e0 con 0x7ff14c143c70 2020-11-20T12:14:08.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.525+0000 7ff14b59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c14ab30 con 0x7ff14c065f90 2020-11-20T12:14:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.525+0000 7ff14b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c062c10 con 0x7ff14c065190 2020-11-20T12:14:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.525+0000 7ff149d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff14c065f90 0x7ff14c143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.525+0000 7ff149d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff14c065f90 0x7ff14c143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48422/0 (socket says 172.21.15.187:48422) 2020-11-20T12:14:08.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.525+0000 7ff149d9b700 1 -- 172.21.15.187:0/3182935459 learned_addr learned my addr 172.21.15.187:0/3182935459 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:08.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.526+0000 7ff14a59c700 1 --2- 172.21.15.187:0/3182935459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 0x7ff14c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:08.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.526+0000 7ff149d9b700 1 -- 172.21.15.187:0/3182935459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 msgr2=0x7ff14c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:08.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.526+0000 7ff149d9b700 1 --2- 172.21.15.187:0/3182935459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 0x7ff14c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.526+0000 7ff149d9b700 1 -- 172.21.15.187:0/3182935459 >> v1:172.21.15.5:6789/0 conn(0x7ff14c143c70 legacy=0x7ff14c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:08.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.526+0000 7ff149d9b700 1 -- 172.21.15.187:0/3182935459 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0b1640 con 0x7ff14c065f90 2020-11-20T12:14:08.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.526+0000 7ff149d9b700 1 --2- 172.21.15.187:0/3182935459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c065f90 0x7ff14c143780 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7ff14c14ac70 tx=0x7ff140006eb0).ready entity=mon.1 client_cookie=34626a1980be70e7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:08.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14959a700 1 -- 172.21.15.187:0/3182935459 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140009960 con 0x7ff14c065f90 2020-11-20T12:14:08.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14959a700 1 -- 172.21.15.187:0/3182935459 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff140009ac0 con 0x7ff14c065f90 2020-11-20T12:14:08.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14959a700 1 -- 172.21.15.187:0/3182935459 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140009e40 con 0x7ff14c065f90 2020-11-20T12:14:08.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14b59e700 1 -- 172.21.15.187:0/3182935459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c065f90 msgr2=0x7ff14c143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:08.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3182935459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c065f90 0x7ff14c143780 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7ff14c14ac70 tx=0x7ff140006eb0).stop 2020-11-20T12:14:08.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14b59e700 1 -- 172.21.15.187:0/3182935459 shutdown_connections 2020-11-20T12:14:08.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3182935459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 0x7ff14c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3182935459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c065f90 0x7ff14c143780 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14b59e700 1 -- 172.21.15.187:0/3182935459 >> 172.21.15.187:0/3182935459 conn(0x7ff14c0bab30 msgr2=0x7ff14c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:08.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14b59e700 1 -- 172.21.15.187:0/3182935459 shutdown_connections 2020-11-20T12:14:08.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.527+0000 7ff14b59e700 1 -- 172.21.15.187:0/3182935459 wait complete. 2020-11-20T12:14:08.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.529+0000 7ff14b59e700 1 Processor -- start 2020-11-20T12:14:08.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.529+0000 7ff14b59e700 1 -- start start 2020-11-20T12:14:08.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.530+0000 7ff14b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 0x7ff14c147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:08.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.530+0000 7ff14b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff14c065f90 0x7ff14c09bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:08.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.530+0000 7ff14b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c143c70 0x7ff14c142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:08.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.530+0000 7ff14b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c147a40 con 0x7ff14c065f90 2020-11-20T12:14:08.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.530+0000 7ff14b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b4350 con 0x7ff14c143c70 2020-11-20T12:14:08.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.530+0000 7ff14b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b45f0 con 0x7ff14c065190 2020-11-20T12:14:08.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.530+0000 7ff14ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c143c70 0x7ff14c142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:08.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff14ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c143c70 0x7ff14c142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48430/0 (socket says 172.21.15.187:48430) 2020-11-20T12:14:08.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff14ad9d700 1 -- 172.21.15.187:0/3772860347 learned_addr learned my addr 172.21.15.187:0/3772860347 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:08.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff149d9b700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff14c065f90 0x7ff14c09bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:08.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff14ad9d700 1 -- 172.21.15.187:0/3772860347 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 msgr2=0x7ff14c147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:08.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff14ad9d700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 0x7ff14c147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:08.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff14ad9d700 1 -- 172.21.15.187:0/3772860347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff14c065f90 msgr2=0x7ff14c09bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:08.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff14ad9d700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff14c065f90 0x7ff14c09bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff14ad9d700 1 -- 172.21.15.187:0/3772860347 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0b4890 con 0x7ff14c143c70 2020-11-20T12:14:08.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff14ad9d700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c143c70 0x7ff14c142de0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7ff14400ca90 tx=0x7ff144001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:08.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.531+0000 7ff13a7fc700 1 -- 172.21.15.187:0/3772860347 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400a7c0 con 0x7ff14c143c70 2020-11-20T12:14:08.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.532+0000 7ff13a7fc700 1 -- 172.21.15.187:0/3772860347 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14400a920 con 0x7ff14c143c70 2020-11-20T12:14:08.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.532+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c147c10 con 0x7ff14c143c70 2020-11-20T12:14:08.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.532+0000 7ff13a7fc700 1 -- 172.21.15.187:0/3772860347 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400acc0 con 0x7ff14c143c70 2020-11-20T12:14:08.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.532+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c09c0f0 con 0x7ff14c143c70 2020-11-20T12:14:08.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.533+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c0b4a20 con 0x7ff14c143c70 2020-11-20T12:14:08.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.533+0000 7ff13a7fc700 1 -- 172.21.15.187:0/3772860347 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff144010070 con 0x7ff14c143c70 2020-11-20T12:14:08.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.534+0000 7ff13a7fc700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff120056d70 0x7ff120059230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:08.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.534+0000 7ff13a7fc700 1 -- 172.21.15.187:0/3772860347 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff14406fd20 con 0x7ff14c143c70 2020-11-20T12:14:08.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.534+0000 7ff14a59c700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff120056d70 0x7ff120059230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:08.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.535+0000 7ff14a59c700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff120056d70 0x7ff120059230 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7ff13c00b740 tx=0x7ff13c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:08.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.538+0000 7ff13a7fc700 1 -- 172.21.15.187:0/3772860347 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff1440451f0 con 0x7ff14c143c70 2020-11-20T12:14:08.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.790+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff14c061a20 con 0x7ff14c143c70 2020-11-20T12:14:08.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.790+0000 7ff13a7fc700 1 -- 172.21.15.187:0/3772860347 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7ff144048810 con 0x7ff14c143c70 2020-11-20T12:14:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:08.795 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:14:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff120056d70 msgr2=0x7ff120059230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:08.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff120056d70 0x7ff120059230 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7ff13c00b740 tx=0x7ff13c008040).stop 2020-11-20T12:14:08.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c143c70 msgr2=0x7ff14c142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:08.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c143c70 0x7ff14c142de0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7ff14400ca90 tx=0x7ff144001360).stop 2020-11-20T12:14:08.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 shutdown_connections 2020-11-20T12:14:08.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff120056d70 0x7ff120059230 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff14c143c70 0x7ff14c142de0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff14c065190 0x7ff14c147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:08.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 --2- 172.21.15.187:0/3772860347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff14c065f90 0x7ff14c09bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:08.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 >> 172.21.15.187:0/3772860347 conn(0x7ff14c0bab30 msgr2=0x7ff14c0b9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:08.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 shutdown_connections 2020-11-20T12:14:08.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:08.793+0000 7ff14b59e700 1 -- 172.21.15.187:0/3772860347 wait complete. 2020-11-20T12:14:08.804 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:14:08.817 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue 2020-11-20T12:14:09.096 INFO:teuthology.orchestra.run.smithi187.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":5,"pq_executing":0,"pq_executing_high_water":1,"pq_executed":1,"pq_item_in_journal":0}} 2020-11-20T12:14:09.097 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 5, "pq_item_in_journal": 0 } } 2020-11-20T12:14:09.098 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:14:09.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.187+0000 7fd21e68bd00 1 Processor -- start 2020-11-20T12:14:09.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.187+0000 7fd21e68bd00 1 -- start start 2020-11-20T12:14:09.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd21e68bd00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5611129ed380 0x561112a167b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd21e68bd00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a14d20 0x561112ab98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd21e68bd00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 0x561112ac4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd21e68bd00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x561112968810 con 0x561112a14d20 2020-11-20T12:14:09.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd21e68bd00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x561112a31770 con 0x5611129ed380 2020-11-20T12:14:09.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd21e68bd00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x561112a27a20 con 0x561112a2eab0 2020-11-20T12:14:09.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd20ef30700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5611129ed380 0x561112a167b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd20ef30700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5611129ed380 0x561112a167b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48440/0 (socket says 172.21.15.187:48440) 2020-11-20T12:14:09.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.188+0000 7fd20ef30700 1 -- 172.21.15.187:0/40204011 learned_addr learned my addr 172.21.15.187:0/40204011 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:09.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20f731700 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 0x561112ac4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20e72f700 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a14d20 0x561112ab98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20ef30700 1 -- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 msgr2=0x561112ac4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:09.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20ef30700 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 0x561112ac4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20ef30700 1 -- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a14d20 msgr2=0x561112ab98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:09.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20ef30700 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a14d20 0x561112ab98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20ef30700 1 -- 172.21.15.187:0/40204011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611129924e0 con 0x5611129ed380 2020-11-20T12:14:09.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20f731700 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 0x561112ac4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:09.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20e72f700 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a14d20 0x561112ab98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:09.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.189+0000 7fd20ef30700 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5611129ed380 0x561112a167b0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7fd20000b610 tx=0x7fd200001760).ready entity=mon.1 client_cookie=6dada0660d46483b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd20d72d700 1 -- 172.21.15.187:0/40204011 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20000d5a0 con 0x5611129ed380 2020-11-20T12:14:09.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd20d72d700 1 -- 172.21.15.187:0/40204011 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20000d700 con 0x5611129ed380 2020-11-20T12:14:09.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd20d72d700 1 -- 172.21.15.187:0/40204011 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20000da80 con 0x5611129ed380 2020-11-20T12:14:09.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd21e68bd00 1 -- 172.21.15.187:0/40204011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5611129ed380 msgr2=0x561112a167b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5611129ed380 0x561112a167b0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7fd20000b610 tx=0x7fd200001760).stop 2020-11-20T12:14:09.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd21e68bd00 1 -- 172.21.15.187:0/40204011 shutdown_connections 2020-11-20T12:14:09.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 0x561112ac4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5611129ed380 0x561112a167b0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/40204011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a14d20 0x561112ab98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.190+0000 7fd21e68bd00 1 -- 172.21.15.187:0/40204011 >> 172.21.15.187:0/40204011 conn(0x5611129ec0c0 msgr2=0x5611129ed760 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:09.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.191+0000 7fd21e68bd00 1 -- 172.21.15.187:0/40204011 shutdown_connections 2020-11-20T12:14:09.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.191+0000 7fd21e68bd00 1 -- 172.21.15.187:0/40204011 wait complete. 2020-11-20T12:14:09.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.191+0000 7fd21e68bd00 1 Processor -- start 2020-11-20T12:14:09.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.192+0000 7fd21e68bd00 1 -- start start 2020-11-20T12:14:09.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd21e68bd00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561112a14d20 0x561112ab9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd21e68bd00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 0x561112acf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd21e68bd00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a2d8d0 0x561112acfca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd21e68bd00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x561112ab1000 con 0x561112a2d8d0 2020-11-20T12:14:09.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd21e68bd00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x561112ab9f00 con 0x561112a14d20 2020-11-20T12:14:09.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd21e68bd00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x561112aba070 con 0x561112a2eab0 2020-11-20T12:14:09.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd20ef30700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561112a14d20 0x561112ab9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd20ef30700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561112a14d20 0x561112ab9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48446/0 (socket says 172.21.15.187:48446) 2020-11-20T12:14:09.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 learned_addr learned my addr 172.21.15.187:0/1851814830 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:09.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 msgr2=0x561112acf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x561112a2eab0 0x561112acf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:09.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a2d8d0 msgr2=0x561112acfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x561112a2d8d0 0x561112acfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:09.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.193+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56111298dec0 con 0x561112a14d20 2020-11-20T12:14:09.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.194+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561112a14d20 0x561112ab9a10 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7fd20000b5e0 tx=0x561112a32530).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.194+0000 7fd1ff7fe700 1 -- 172.21.15.187:0/1851814830 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20000eea0 con 0x561112a14d20 2020-11-20T12:14:09.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.194+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56111296c250 con 0x561112a14d20 2020-11-20T12:14:09.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.194+0000 7fd1ff7fe700 1 -- 172.21.15.187:0/1851814830 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd200001400 con 0x561112a14d20 2020-11-20T12:14:09.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.194+0000 7fd1ff7fe700 1 -- 172.21.15.187:0/1851814830 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20001b880 con 0x561112a14d20 2020-11-20T12:14:09.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.194+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561112a27bb0 con 0x561112a14d20 2020-11-20T12:14:09.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.196+0000 7fd1ff7fe700 1 -- 172.21.15.187:0/1851814830 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd20001bbf0 con 0x561112a14d20 2020-11-20T12:14:09.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.197+0000 7fd1ff7fe700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1f40559a0 0x7fd1f4057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.197+0000 7fd1ff7fe700 1 -- 172.21.15.187:0/1851814830 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd200071850 con 0x561112a14d20 2020-11-20T12:14:09.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.197+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x561112ab7380 0x561112ab77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.197+0000 7fd20e72f700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1f40559a0 0x7fd1f4057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.197+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:1 18.0 18:00000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x561112ab7cb0 con 0x561112ab7380 2020-11-20T12:14:09.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.198+0000 7fd20f731700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x561112ab7380 0x561112ab77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.198+0000 7fd20e72f700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1f40559a0 0x7fd1f4057e60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x561112a32150 tx=0x7fd204013000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.199+0000 7fd20f731700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x561112ab7380 0x561112ab77c0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.201+0000 7fd20f731700 1 -- 172.21.15.187:0/1851814830 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(1 [pgnls start_epoch 89 out=81b] v89'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7fd2080086e0 con 0x561112ab7380 2020-11-20T12:14:09.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.201+0000 7fd20f731700 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:2 18.4 18:20000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x561112ab7cb0 con 0x561112ab7380 2020-11-20T12:14:09.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.202+0000 7fd20f731700 1 -- 172.21.15.187:0/1851814830 <== osd.6 v2:172.21.15.5:6800/20157 2 ==== osd_op_reply(2 [pgnls start_epoch 89 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd2080086e0 con 0x561112ab7380 2020-11-20T12:14:09.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.202+0000 7fd20f731700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7fd2080128d0 0x7fd208011350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.202+0000 7fd20f731700 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- osd_op(unknown.0.0:3 18.2 18:40000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x561112ab7cb0 con 0x7fd2080128d0 2020-11-20T12:14:09.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.202+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7fd2080128d0 0x7fd208011350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.203+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7fd2080128d0 0x7fd208011350 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.203+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 <== osd.0 v2:172.21.15.187:6802/19903 1 ==== osd_op_reply(3 [pgnls start_epoch 89 out=81b] v89'1 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7fd200006030 con 0x7fd2080128d0 2020-11-20T12:14:09.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.203+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x5611129ed380 0x7fd200071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.204+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:4 18.6 18:60000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x561112ab7cb0 con 0x5611129ed380 2020-11-20T12:14:09.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.204+0000 7fd20e72f700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x5611129ed380 0x7fd200071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.205+0000 7fd20e72f700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x5611129ed380 0x7fd200071050 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.205+0000 7fd20e72f700 1 -- 172.21.15.187:0/1851814830 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(4 [pgnls start_epoch 89 out=49b] v89'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd204000b80 con 0x5611129ed380 2020-11-20T12:14:09.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.205+0000 7fd20e72f700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fd20401d930 0x7fd204011390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.205+0000 7fd20e72f700 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:5 18.1 18:80000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x561112ab7cb0 con 0x7fd20401d930 2020-11-20T12:14:09.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.206+0000 7fd20f731700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fd20401d930 0x7fd204011390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.208+0000 7fd20f731700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fd20401d930 0x7fd204011390 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.209+0000 7fd20f731700 1 -- 172.21.15.187:0/1851814830 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== osd_op_reply(5 [pgnls start_epoch 89 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd2080086e0 con 0x7fd20401d930 2020-11-20T12:14:09.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.209+0000 7fd20f731700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fd208015950 0x7fd2080102c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.209+0000 7fd20f731700 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:6 18.5 18:a0000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x561112ab7cb0 con 0x7fd208015950 2020-11-20T12:14:09.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.210+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fd208015950 0x7fd2080102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.211+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fd208015950 0x7fd2080102c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.212+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(6 [pgnls start_epoch 89 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd200006030 con 0x7fd208015950 2020-11-20T12:14:09.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.212+0000 7fd20ef30700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7fd200049d40 0x7fd200047f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.212+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:7 18.3 18:c0000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x561112ab7cb0 con 0x7fd200049d40 2020-11-20T12:14:09.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.213+0000 7fd20e72f700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7fd200049d40 0x7fd200047f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.215+0000 7fd20e72f700 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7fd200049d40 0x7fd200047f40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.216+0000 7fd20e72f700 1 -- 172.21.15.187:0/1851814830 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== osd_op_reply(7 [pgnls start_epoch 89 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd204000b80 con 0x7fd200049d40 2020-11-20T12:14:09.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.216+0000 7fd20e72f700 1 -- 172.21.15.187:0/1851814830 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:8 18.7 18:e0000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x561112ab7cb0 con 0x7fd208015950 2020-11-20T12:14:09.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd20ef30700 1 -- 172.21.15.187:0/1851814830 <== osd.7 v2:172.21.15.5:6814/20158 2 ==== osd_op_reply(8 [pgnls start_epoch 89 out=49b] v89'5 uv5 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd200006030 con 0x7fd208015950 2020-11-20T12:14:09.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7fd2080128d0 msgr2=0x7fd208011350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7fd2080128d0 0x7fd208011350 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x5611129ed380 msgr2=0x7fd200071050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x5611129ed380 0x7fd200071050 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7fd200049d40 msgr2=0x7fd200047f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7fd200049d40 0x7fd200047f40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fd20401d930 msgr2=0x7fd204011390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fd20401d930 0x7fd204011390 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x561112ab7380 msgr2=0x561112ab77c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x561112ab7380 0x561112ab77c0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd20f731700 1 -- 172.21.15.187:0/1851814830 reap_dead start 2020-11-20T12:14:09.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.217+0000 7fd20f731700 1 -- 172.21.15.187:0/1851814830 reap_dead start 2020-11-20T12:14:09.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fd208015950 msgr2=0x7fd2080102c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fd208015950 0x7fd2080102c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1f40559a0 msgr2=0x7fd1f4057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1f40559a0 0x7fd1f4057e60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x561112a32150 tx=0x7fd204013000).stop 2020-11-20T12:14:09.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561112a14d20 msgr2=0x561112ab9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561112a14d20 0x561112ab9a10 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7fd20000b5e0 tx=0x561112a32530).stop 2020-11-20T12:14:09.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 shutdown_connections 2020-11-20T12:14:09.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fd208015950 0x7fd2080102c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561112a14d20 0x561112ab9a10 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x561112ab7380 0x561112ab77c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 --2- 172.21.15.187:0/1851814830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd1f40559a0 0x7fd1f4057e60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 >> 172.21.15.187:0/1851814830 conn(0x5611129ec0c0 msgr2=0x561112a2dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:09.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 shutdown_connections 2020-11-20T12:14:09.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.218+0000 7fd21e68bd00 1 -- 172.21.15.187:0/1851814830 wait complete. 2020-11-20T12:14:09.233 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000000 2020-11-20T12:14:09.233 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000001 2020-11-20T12:14:09.235 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2020-11-20T12:14:09.236 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:09.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.484+0000 7f503759e700 1 Processor -- start 2020-11-20T12:14:09.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.484+0000 7f503759e700 1 -- start start 2020-11-20T12:14:09.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.485+0000 7f503759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f50381484d0 0x7f5038061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.485+0000 7f503759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f50380622c0 0x7f50380b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.485+0000 7f503759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50380b1300 con 0x7f5038066f70 2020-11-20T12:14:09.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.486+0000 7f503759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5038067730 con 0x7f50381484d0 2020-11-20T12:14:09.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.486+0000 7f503759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50380b15a0 con 0x7f50380622c0 2020-11-20T12:14:09.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.486+0000 7f5036d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5038066f70 0x7f50380b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45072/0 (socket says 172.21.15.187:45072) 2020-11-20T12:14:09.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.486+0000 7f5036d9d700 1 -- 172.21.15.187:0/2242813158 learned_addr learned my addr 172.21.15.187:0/2242813158 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:09.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.486+0000 7f503659c700 1 --2- 172.21.15.187:0/2242813158 >> v2:172.21.15.187:3300/0 conn(0x7f50381484d0 0x7f5038061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.486+0000 7f502dd9b700 1 --2- 172.21.15.187:0/2242813158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f50380622c0 0x7f50380b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.487+0000 7f5035d9b700 1 -- 172.21.15.187:0/2242813158 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930898290 0 0) 0x7f50380b1300 con 0x7f5038066f70 2020-11-20T12:14:09.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.487+0000 7f5035d9b700 1 -- 172.21.15.187:0/2242813158 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5018003240 con 0x7f5038066f70 2020-11-20T12:14:09.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.487+0000 7f503659c700 1 -- 172.21.15.187:0/2242813158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f50380622c0 msgr2=0x7f50380b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:09.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.487+0000 7f503659c700 1 --2- 172.21.15.187:0/2242813158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f50380622c0 0x7f50380b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.487+0000 7f5035d9b700 1 -- 172.21.15.187:0/2242813158 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2452846315 0 0) 0x7f5018003240 con 0x7f5038066f70 2020-11-20T12:14:09.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.487+0000 7f503659c700 1 -- 172.21.15.187:0/2242813158 >> v1:172.21.15.187:6789/0 conn(0x7f5038066f70 legacy=0x7f50380b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.487+0000 7f503659c700 1 -- 172.21.15.187:0/2242813158 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50380b1840 con 0x7f50381484d0 2020-11-20T12:14:09.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.487+0000 7f503659c700 1 --2- 172.21.15.187:0/2242813158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50381484d0 0x7f5038061dd0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f5028009c90 tx=0x7f50280091d0).ready entity=mon.1 client_cookie=564a419ff10d375c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f5035d9b700 1 -- 172.21.15.187:0/2242813158 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50300014e0 con 0x7f50381484d0 2020-11-20T12:14:09.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f5035d9b700 1 -- 172.21.15.187:0/2242813158 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502800b760 con 0x7f50381484d0 2020-11-20T12:14:09.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f5035d9b700 1 -- 172.21.15.187:0/2242813158 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502800bae0 con 0x7f50381484d0 2020-11-20T12:14:09.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f503759e700 1 -- 172.21.15.187:0/2242813158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50381484d0 msgr2=0x7f5038061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f503759e700 1 --2- 172.21.15.187:0/2242813158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50381484d0 0x7f5038061dd0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f5028009c90 tx=0x7f50280091d0).stop 2020-11-20T12:14:09.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f503759e700 1 -- 172.21.15.187:0/2242813158 shutdown_connections 2020-11-20T12:14:09.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f503759e700 1 --2- 172.21.15.187:0/2242813158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50381484d0 0x7f5038061dd0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f503759e700 1 --2- 172.21.15.187:0/2242813158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f50380622c0 0x7f50380b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f503759e700 1 -- 172.21.15.187:0/2242813158 >> 172.21.15.187:0/2242813158 conn(0x7f50380bab30 msgr2=0x7f50380baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:09.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f503759e700 1 -- 172.21.15.187:0/2242813158 shutdown_connections 2020-11-20T12:14:09.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.488+0000 7f503759e700 1 -- 172.21.15.187:0/2242813158 wait complete. 2020-11-20T12:14:09.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.490+0000 7f503759e700 1 Processor -- start 2020-11-20T12:14:09.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.490+0000 7f503759e700 1 -- start start 2020-11-20T12:14:09.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50380622c0 0x7f503815b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5038066f70 0x7f50380b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50381484d0 0x7f5038137770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503815bf40 con 0x7f50381484d0 2020-11-20T12:14:09.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503815c1e0 con 0x7f50380622c0 2020-11-20T12:14:09.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503815c480 con 0x7f5038066f70 2020-11-20T12:14:09.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50380622c0 0x7f503815b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50380622c0 0x7f503815b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48472/0 (socket says 172.21.15.187:48472) 2020-11-20T12:14:09.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.491+0000 7f503659c700 1 -- 172.21.15.187:0/3354780534 learned_addr learned my addr 172.21.15.187:0/3354780534 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:09.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f503659c700 1 -- 172.21.15.187:0/3354780534 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5038066f70 msgr2=0x7f50380b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f503659c700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5038066f70 0x7f50380b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:09.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f503659c700 1 -- 172.21.15.187:0/3354780534 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50381484d0 msgr2=0x7f5038137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f503659c700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50381484d0 0x7f5038137770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:09.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f503659c700 1 -- 172.21.15.187:0/3354780534 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503815bc00 con 0x7f50380622c0 2020-11-20T12:14:09.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f503659c700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50380622c0 0x7f503815b710 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7f502800b670 tx=0x7f5028016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f502effd700 1 -- 172.21.15.187:0/3354780534 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502801d450 con 0x7f50380622c0 2020-11-20T12:14:09.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f502effd700 1 -- 172.21.15.187:0/3354780534 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502801d5d0 con 0x7f50380622c0 2020-11-20T12:14:09.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50380b4310 con 0x7f50380622c0 2020-11-20T12:14:09.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f502effd700 1 -- 172.21.15.187:0/3354780534 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502801d990 con 0x7f50380622c0 2020-11-20T12:14:09.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.492+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503809d430 con 0x7f50380622c0 2020-11-20T12:14:09.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.493+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503815bd90 con 0x7f50380622c0 2020-11-20T12:14:09.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.494+0000 7f502effd700 1 -- 172.21.15.187:0/3354780534 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5028017040 con 0x7f50380622c0 2020-11-20T12:14:09.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.495+0000 7f502effd700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5014056d20 0x7f50140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:09.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.495+0000 7f502effd700 1 -- 172.21.15.187:0/3354780534 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5028071f70 con 0x7f50380622c0 2020-11-20T12:14:09.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.495+0000 7f502dd9b700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5014056d20 0x7f50140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:09.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.496+0000 7f502dd9b700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5014056d20 0x7f50140591e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f503809d820 tx=0x7f5020008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:09.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.499+0000 7f502effd700 1 -- 172.21.15.187:0/3354780534 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f50280472b0 con 0x7f50380622c0 2020-11-20T12:14:09.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.747+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5038061a20 con 0x7f50380622c0 2020-11-20T12:14:09.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.748+0000 7f502effd700 1 -- 172.21.15.187:0/3354780534 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f5028021020 con 0x7f50380622c0 2020-11-20T12:14:09.751 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:09.751 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:14:09.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5014056d20 msgr2=0x7f50140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5014056d20 0x7f50140591e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f503809d820 tx=0x7f5020008040).stop 2020-11-20T12:14:09.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50380622c0 msgr2=0x7f503815b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:09.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50380622c0 0x7f503815b710 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7f502800b670 tx=0x7f5028016000).stop 2020-11-20T12:14:09.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 shutdown_connections 2020-11-20T12:14:09.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5014056d20 0x7f50140591e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50381484d0 0x7f5038137770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:09.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50380622c0 0x7f503815b710 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:09.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 --2- 172.21.15.187:0/3354780534 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5038066f70 0x7f50380b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:09.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.750+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 >> 172.21.15.187:0/3354780534 conn(0x7f50380bab30 msgr2=0x7f50380b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:09.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.751+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 shutdown_connections 2020-11-20T12:14:09.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:09.751+0000 7f503759e700 1 -- 172.21.15.187:0/3354780534 wait complete. 2020-11-20T12:14:09.756 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:14:09.778 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:14:10.080 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:14:10.080 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:14:10.081 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data getxattr 10000000000.00000000 parent 2020-11-20T12:14:10.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.171+0000 7fda4027ed00 1 Processor -- start 2020-11-20T12:14:10.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.171+0000 7fda4027ed00 1 -- start start 2020-11-20T12:14:10.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.172+0000 7fda4027ed00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc4363830 0x55bcc4361370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.172+0000 7fda4027ed00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc43761a0 0x55bcc433d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.172+0000 7fda4027ed00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc437f570 0x55bcc44199a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.172+0000 7fda4027ed00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc440b3f0 con 0x55bcc43761a0 2020-11-20T12:14:10.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.172+0000 7fda4027ed00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc440b690 con 0x55bcc437f570 2020-11-20T12:14:10.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.172+0000 7fda4027ed00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcc44073b0 con 0x55bcc4363830 2020-11-20T12:14:10.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc437f570 0x55bcc44199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc437f570 0x55bcc44199a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48490/0 (socket says 172.21.15.187:48490) 2020-11-20T12:14:10.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 -- 172.21.15.187:0/2342638448 learned_addr learned my addr 172.21.15.187:0/2342638448 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:10.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda30b23700 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc4363830 0x55bcc4361370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda28322700 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc43761a0 0x55bcc433d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 -- 172.21.15.187:0/2342638448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc4363830 msgr2=0x55bcc4361370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:10.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc4363830 0x55bcc4361370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 -- 172.21.15.187:0/2342638448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc43761a0 msgr2=0x55bcc433d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:10.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc43761a0 0x55bcc433d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 -- 172.21.15.187:0/2342638448 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcc42e30c0 con 0x55bcc437f570 2020-11-20T12:14:10.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.173+0000 7fda31324700 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc437f570 0x55bcc44199a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7fda2c00b9a0 tx=0x7fda2c007040).ready entity=mon.1 client_cookie=56904368d83396be server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda2b7fe700 1 -- 172.21.15.187:0/2342638448 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c0098d0 con 0x55bcc437f570 2020-11-20T12:14:10.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda2b7fe700 1 -- 172.21.15.187:0/2342638448 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c00b030 con 0x55bcc437f570 2020-11-20T12:14:10.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda2b7fe700 1 -- 172.21.15.187:0/2342638448 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c009c40 con 0x55bcc437f570 2020-11-20T12:14:10.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 -- 172.21.15.187:0/2342638448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc437f570 msgr2=0x55bcc44199a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc437f570 0x55bcc44199a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7fda2c00b9a0 tx=0x7fda2c007040).stop 2020-11-20T12:14:10.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 -- 172.21.15.187:0/2342638448 shutdown_connections 2020-11-20T12:14:10.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc437f570 0x55bcc44199a0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc4363830 0x55bcc4361370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 --2- 172.21.15.187:0/2342638448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc43761a0 0x55bcc433d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 -- 172.21.15.187:0/2342638448 >> 172.21.15.187:0/2342638448 conn(0x55bcc433fab0 msgr2=0x55bcc4407d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 -- 172.21.15.187:0/2342638448 shutdown_connections 2020-11-20T12:14:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.174+0000 7fda4027ed00 1 -- 172.21.15.187:0/2342638448 wait complete. 2020-11-20T12:14:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.175+0000 7fda4027ed00 1 Processor -- start 2020-11-20T12:14:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.175+0000 7fda4027ed00 1 -- start start 2020-11-20T12:14:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.176+0000 7fda4027ed00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc4363830 0x55bcc4406de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.176+0000 7fda4027ed00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc43761a0 0x55bcc4405ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.176+0000 7fda4027ed00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc437f570 0x55bcc4402ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.176+0000 7fda4027ed00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc4421b00 con 0x55bcc4363830 2020-11-20T12:14:10.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.176+0000 7fda4027ed00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc4403190 con 0x55bcc43761a0 2020-11-20T12:14:10.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.176+0000 7fda4027ed00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcc4421800 con 0x55bcc437f570 2020-11-20T12:14:10.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc43761a0 0x55bcc4405ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc43761a0 0x55bcc4405ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48494/0 (socket says 172.21.15.187:48494) 2020-11-20T12:14:10.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 -- 172.21.15.187:0/1667095745 learned_addr learned my addr 172.21.15.187:0/1667095745 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:10.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda30b23700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc4363830 0x55bcc4406de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda31324700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc437f570 0x55bcc4402ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 -- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc437f570 msgr2=0x55bcc4402ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55bcc437f570 0x55bcc4402ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 -- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc4363830 msgr2=0x55bcc4406de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55bcc4363830 0x55bcc4406de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 -- 172.21.15.187:0/1667095745 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcc42e44e0 con 0x55bcc43761a0 2020-11-20T12:14:10.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.177+0000 7fda28322700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc43761a0 0x55bcc4405ad0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x55bcc43434b0 tx=0x7fda18015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.178+0000 7fda29ffb700 1 -- 172.21.15.187:0/1667095745 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18005330 con 0x55bcc43761a0 2020-11-20T12:14:10.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.178+0000 7fda29ffb700 1 -- 172.21.15.187:0/1667095745 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda18012e10 con 0x55bcc43761a0 2020-11-20T12:14:10.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.178+0000 7fda29ffb700 1 -- 172.21.15.187:0/1667095745 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1801d600 con 0x55bcc43761a0 2020-11-20T12:14:10.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.178+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bcc42dfec0 con 0x55bcc43761a0 2020-11-20T12:14:10.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.178+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcc42be250 con 0x55bcc43761a0 2020-11-20T12:14:10.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.179+0000 7fda29ffb700 1 -- 172.21.15.187:0/1667095745 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fda1801a030 con 0x55bcc43761a0 2020-11-20T12:14:10.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.180+0000 7fda29ffb700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda100559a0 0x7fda10057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.180+0000 7fda29ffb700 1 -- 172.21.15.187:0/1667095745 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fda18070ec0 con 0x55bcc43761a0 2020-11-20T12:14:10.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.180+0000 7fda30b23700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda100559a0 0x7fda10057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.180+0000 7fda4027ed00 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x55bcc440c6d0 0x55bcc4408960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.180+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:1 18.0 18:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected e89) v8 -- 0x55bcc440d470 con 0x55bcc440c6d0 2020-11-20T12:14:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.181+0000 7fda30b23700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda100559a0 0x7fda10057e60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fda20008940 tx=0x7fda20005b30).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.181+0000 7fda31324700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x55bcc440c6d0 0x55bcc4408960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.183+0000 7fda31324700 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x55bcc440c6d0 0x55bcc4408960 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.184+0000 7fda31324700 1 -- 172.21.15.187:0/1667095745 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (62) out=62b] v0'0 uv4 ondisk = 0) v8 ==== 164+0+62 (crc 0 0 0) 0x7fda2c018040 con 0x55bcc440c6d0 2020-11-20T12:14:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x55bcc440c6d0 msgr2=0x55bcc4408960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x55bcc440c6d0 0x55bcc4408960 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda100559a0 msgr2=0x7fda10057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda100559a0 0x7fda10057e60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fda20008940 tx=0x7fda20005b30).stop 2020-11-20T12:14:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc43761a0 msgr2=0x55bcc4405ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 --2- 172.21.15.187:0/1667095745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bcc43761a0 0x55bcc4405ad0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x55bcc43434b0 tx=0x7fda18015000).stop 2020-11-20T12:14:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda31324700 1 -- 172.21.15.187:0/1667095745 reap_dead start 2020-11-20T12:14:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 shutdown_connections 2020-11-20T12:14:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 >> 172.21.15.187:0/1667095745 conn(0x55bcc433fab0 msgr2=0x55bcc4361140 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:10.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 shutdown_connections 2020-11-20T12:14:10.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.185+0000 7fda4027ed00 1 -- 172.21.15.187:0/1667095745 wait complete. 2020-11-20T12:14:10.197 INFO:teuthology.orchestra.run.smithi187.stdout:8file_b 2020-11-20T12:14:10.197 INFO:teuthology.orchestra.run.smithi187:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2020-11-20T12:14:10.299 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:14:10.299 INFO:teuthology.orchestra.run.smithi187.stdout: "ino": 1099511627776, 2020-11-20T12:14:10.299 INFO:teuthology.orchestra.run.smithi187.stdout: "ancestors": [ 2020-11-20T12:14:10.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:10.300 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1, 2020-11-20T12:14:10.300 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "file_b", 2020-11-20T12:14:10.300 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 14 2020-11-20T12:14:10.301 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:10.301 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2020-11-20T12:14:10.301 INFO:teuthology.orchestra.run.smithi187.stdout: "pool": 18, 2020-11-20T12:14:10.302 INFO:teuthology.orchestra.run.smithi187.stdout: "old_pools": [] 2020-11-20T12:14:10.302 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:14:10.302 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:10.305 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:14:10.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.391+0000 7f5382609d00 1 Processor -- start 2020-11-20T12:14:10.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.391+0000 7f5382609d00 1 -- start start 2020-11-20T12:14:10.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f5382609d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5591d48f6380 0x5591d493a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f5382609d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d491dd20 0x5591d4930810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f5382609d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d4937ab0 0x5591d49c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f5382609d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591d4871810 con 0x5591d491dd20 2020-11-20T12:14:10.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f5382609d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591d4930d00 con 0x5591d4937ab0 2020-11-20T12:14:10.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f5382609d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5591d493a650 con 0x5591d48f6380 2020-11-20T12:14:10.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f53736af700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d4937ab0 0x5591d49c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f53736af700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d4937ab0 0x5591d49c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48512/0 (socket says 172.21.15.187:48512) 2020-11-20T12:14:10.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.392+0000 7f53736af700 1 -- 172.21.15.187:0/3552252297 learned_addr learned my addr 172.21.15.187:0/3552252297 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:10.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.393+0000 7f536a6ad700 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d491dd20 0x5591d4930810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.393+0000 7f5372eae700 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5591d48f6380 0x5591d493a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.393+0000 7f53736af700 1 -- 172.21.15.187:0/3552252297 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5591d48f6380 msgr2=0x5591d493a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:10.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.393+0000 7f53736af700 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5591d48f6380 0x5591d493a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.393+0000 7f53736af700 1 -- 172.21.15.187:0/3552252297 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d491dd20 msgr2=0x5591d4930810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:10.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.393+0000 7f53736af700 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d491dd20 0x5591d4930810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.393+0000 7f53736af700 1 -- 172.21.15.187:0/3552252297 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591d489b4e0 con 0x5591d4937ab0 2020-11-20T12:14:10.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.393+0000 7f53736af700 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d4937ab0 0x5591d49c2770 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f536c00aab0 tx=0x7f536c005c00).ready entity=mon.1 client_cookie=56b4d7911a96e273 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5371eac700 1 -- 172.21.15.187:0/3552252297 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c00f070 con 0x5591d4937ab0 2020-11-20T12:14:10.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5371eac700 1 -- 172.21.15.187:0/3552252297 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c008c00 con 0x5591d4937ab0 2020-11-20T12:14:10.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5371eac700 1 -- 172.21.15.187:0/3552252297 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c013410 con 0x5591d4937ab0 2020-11-20T12:14:10.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5382609d00 1 -- 172.21.15.187:0/3552252297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d4937ab0 msgr2=0x5591d49c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5382609d00 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d4937ab0 0x5591d49c2770 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f536c00aab0 tx=0x7f536c005c00).stop 2020-11-20T12:14:10.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5382609d00 1 -- 172.21.15.187:0/3552252297 shutdown_connections 2020-11-20T12:14:10.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5382609d00 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d4937ab0 0x5591d49c2770 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5382609d00 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5591d48f6380 0x5591d493a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5382609d00 1 --2- 172.21.15.187:0/3552252297 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d491dd20 0x5591d4930810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.394+0000 7f5382609d00 1 -- 172.21.15.187:0/3552252297 >> 172.21.15.187:0/3552252297 conn(0x5591d48f50c0 msgr2=0x5591d491c290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:10.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.395+0000 7f5382609d00 1 -- 172.21.15.187:0/3552252297 shutdown_connections 2020-11-20T12:14:10.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.395+0000 7f5382609d00 1 -- 172.21.15.187:0/3552252297 wait complete. 2020-11-20T12:14:10.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.395+0000 7f5382609d00 1 Processor -- start 2020-11-20T12:14:10.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.396+0000 7f5382609d00 1 -- start start 2020-11-20T12:14:10.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.396+0000 7f5382609d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5591d48f6380 0x5591d4937220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.396+0000 7f5382609d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d491dd20 0x5591d49bc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.396+0000 7f5382609d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d4937ab0 0x5591d4933060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.396+0000 7f5382609d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591d49bcc60 con 0x5591d4937ab0 2020-11-20T12:14:10.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.396+0000 7f5382609d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591d49bcf00 con 0x5591d491dd20 2020-11-20T12:14:10.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.396+0000 7f5382609d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5591d49bd1a0 con 0x5591d48f6380 2020-11-20T12:14:10.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f536a6ad700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d491dd20 0x5591d49bc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f536a6ad700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d491dd20 0x5591d49bc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48516/0 (socket says 172.21.15.187:48516) 2020-11-20T12:14:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f536a6ad700 1 -- 172.21.15.187:0/1291892998 learned_addr learned my addr 172.21.15.187:0/1291892998 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f53736af700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d4937ab0 0x5591d4933060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f536a6ad700 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5591d48f6380 msgr2=0x5591d4937220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f536a6ad700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5591d48f6380 0x5591d4937220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f536a6ad700 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d4937ab0 msgr2=0x5591d4933060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f536a6ad700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5591d4937ab0 0x5591d4933060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.397+0000 7f536a6ad700 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591d4896ec0 con 0x5591d491dd20 2020-11-20T12:14:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.398+0000 7f536a6ad700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d491dd20 0x5591d49bc430 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7f535c005c80 tx=0x7f535c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.398+0000 7f536bfff700 1 -- 172.21.15.187:0/1291892998 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c005350 con 0x5591d491dd20 2020-11-20T12:14:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.398+0000 7f536bfff700 1 -- 172.21.15.187:0/1291892998 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c01e480 con 0x5591d491dd20 2020-11-20T12:14:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.398+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5591d4875250 con 0x5591d491dd20 2020-11-20T12:14:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.398+0000 7f536bfff700 1 -- 172.21.15.187:0/1291892998 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c01e820 con 0x5591d491dd20 2020-11-20T12:14:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.398+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5591d4933550 con 0x5591d491dd20 2020-11-20T12:14:10.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.400+0000 7f536bfff700 1 -- 172.21.15.187:0/1291892998 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f535c011390 con 0x5591d491dd20 2020-11-20T12:14:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.400+0000 7f536bfff700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53540559a0 0x7f5354057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.400+0000 7f536bfff700 1 -- 172.21.15.187:0/1291892998 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f535c070500 con 0x5591d491dd20 2020-11-20T12:14:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.401+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5591d4931850 0x5591d49bdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.401+0000 7f5372eae700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53540559a0 0x7f5354057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.401+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:1 18.0 18:00000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x5591d49c1790 con 0x5591d4931850 2020-11-20T12:14:10.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.401+0000 7f5372eae700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53540559a0 0x7f5354057e60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f53640044b0 tx=0x7f5364008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.401+0000 7f53736af700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5591d4931850 0x5591d49bdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.403+0000 7f53736af700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5591d4931850 0x5591d49bdb50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.404+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(1 [pgnls start_epoch 89 out=81b] v89'4 uv4 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f536c008d70 con 0x5591d4931850 2020-11-20T12:14:10.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.404+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:2 18.4 18:20000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x5591d49c1790 con 0x5591d4931850 2020-11-20T12:14:10.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.405+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 <== osd.6 v2:172.21.15.5:6800/20157 2 ==== osd_op_reply(2 [pgnls start_epoch 89 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536c008d70 con 0x5591d4931850 2020-11-20T12:14:10.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.405+0000 7f53736af700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f536c01e960 0x7f536c01d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.405+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- osd_op(unknown.0.0:3 18.2 18:40000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x5591d49c1790 con 0x7f536c01e960 2020-11-20T12:14:10.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.406+0000 7f5372eae700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f536c01e960 0x7f536c01d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.406+0000 7f5372eae700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f536c01e960 0x7f536c01d3e0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.407+0000 7f5372eae700 1 -- 172.21.15.187:0/1291892998 <== osd.0 v2:172.21.15.187:6802/19903 1 ==== osd_op_reply(3 [pgnls start_epoch 89 out=81b] v89'1 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f53640044e0 con 0x7f536c01e960 2020-11-20T12:14:10.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.407+0000 7f5372eae700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f5364004c20 0x7f5364017920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.407+0000 7f5372eae700 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:4 18.6 18:60000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x5591d49c1790 con 0x7f5364004c20 2020-11-20T12:14:10.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.407+0000 7f536a6ad700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f5364004c20 0x7f5364017920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.408+0000 7f536a6ad700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f5364004c20 0x7f5364017920 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.409+0000 7f536a6ad700 1 -- 172.21.15.187:0/1291892998 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(4 [pgnls start_epoch 89 out=49b] v89'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c047a00 con 0x7f5364004c20 2020-11-20T12:14:10.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.409+0000 7f536a6ad700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f535c048b50 0x7f535c047cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.409+0000 7f536a6ad700 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:5 18.1 18:80000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x5591d49c1790 con 0x7f535c048b50 2020-11-20T12:14:10.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.410+0000 7f53736af700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f535c048b50 0x7f535c047cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.411+0000 7f53736af700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f535c048b50 0x7f535c047cc0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.412+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== osd_op_reply(5 [pgnls start_epoch 89 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536c008d70 con 0x7f535c048b50 2020-11-20T12:14:10.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.412+0000 7f53736af700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f536c013910 0x7f536c02a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.412+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:6 18.5 18:a0000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x5591d49c1790 con 0x7f536c013910 2020-11-20T12:14:10.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.413+0000 7f5372eae700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f536c013910 0x7f536c02a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.414+0000 7f5372eae700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f536c013910 0x7f536c02a8b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.415+0000 7f5372eae700 1 -- 172.21.15.187:0/1291892998 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(6 [pgnls start_epoch 89 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53640044e0 con 0x7f536c013910 2020-11-20T12:14:10.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.415+0000 7f5372eae700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f53640128d0 0x7f5364014380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:10.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.416+0000 7f5372eae700 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:7 18.3 18:c0000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x5591d49c1790 con 0x7f53640128d0 2020-11-20T12:14:10.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.418+0000 7f536a6ad700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f53640128d0 0x7f5364014380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:10.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.420+0000 7f536a6ad700 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f53640128d0 0x7f5364014380 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:10.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.421+0000 7f536a6ad700 1 -- 172.21.15.187:0/1291892998 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== osd_op_reply(7 [pgnls start_epoch 89 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c047a00 con 0x7f53640128d0 2020-11-20T12:14:10.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.421+0000 7f536a6ad700 1 -- 172.21.15.187:0/1291892998 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:8 18.7 18:e0000000::::head [pgnls start_epoch 89 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e89) v8 -- 0x5591d49c1790 con 0x7f536c013910 2020-11-20T12:14:10.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.422+0000 7f5372eae700 1 -- 172.21.15.187:0/1291892998 <== osd.7 v2:172.21.15.5:6814/20158 2 ==== osd_op_reply(8 [pgnls start_epoch 89 out=49b] v89'5 uv5 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53640044e0 con 0x7f536c013910 2020-11-20T12:14:10.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f536c01e960 msgr2=0x7f536c01d3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f536c01e960 0x7f536c01d3e0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f5364004c20 msgr2=0x7f5364017920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f5364004c20 0x7f5364017920 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f53640128d0 msgr2=0x7f5364014380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f53640128d0 0x7f5364014380 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f535c048b50 msgr2=0x7f535c047cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f535c048b50 0x7f535c047cc0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5591d4931850 msgr2=0x5591d49bdb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5591d4931850 0x5591d49bdb50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f536c013910 msgr2=0x7f536c02a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f536c013910 0x7f536c02a8b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 reap_dead start 2020-11-20T12:14:10.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 reap_dead start 2020-11-20T12:14:10.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 reap_dead start 2020-11-20T12:14:10.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f53736af700 1 -- 172.21.15.187:0/1291892998 reap_dead start 2020-11-20T12:14:10.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53540559a0 msgr2=0x7f5354057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53540559a0 0x7f5354057e60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f53640044b0 tx=0x7f5364008040).stop 2020-11-20T12:14:10.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d491dd20 msgr2=0x5591d49bc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:10.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d491dd20 0x5591d49bc430 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7f535c005c80 tx=0x7f535c015000).stop 2020-11-20T12:14:10.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 shutdown_connections 2020-11-20T12:14:10.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5591d491dd20 0x5591d49bc430 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 --2- 172.21.15.187:0/1291892998 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53540559a0 0x7f5354057e60 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:10.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.423+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 >> 172.21.15.187:0/1291892998 conn(0x5591d48f50c0 msgr2=0x5591d491c290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:10.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.424+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 shutdown_connections 2020-11-20T12:14:10.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:10.424+0000 7f5382609d00 1 -- 172.21.15.187:0/1291892998 wait complete. 2020-11-20T12:14:10.432 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000000 2020-11-20T12:14:10.433 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000001 2020-11-20T12:14:10.433 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 8388608 found 2020-11-20T12:14:10.434 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:14:10.434 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:14:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:14:10.575 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:14:10.685 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:14:10.703 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:14:10.704 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:14:10.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:14:10.705 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:14:10.838 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:14:10.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:10.839 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:14:10.893 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:14:11.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.131+0000 7f6efa033700 1 Processor -- start 2020-11-20T12:14:11.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.131+0000 7f6efa033700 1 -- start start 2020-11-20T12:14:11.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.132+0000 7f6efa033700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6ef4066f70 0x7f6ef40b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.132+0000 7f6efa033700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef40b1300 con 0x7f6ef41484d0 2020-11-20T12:14:11.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.132+0000 7f6efa033700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef40b15a0 con 0x7f6ef40b6650 2020-11-20T12:14:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.132+0000 7f6efa033700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef4067730 con 0x7f6ef4066f70 2020-11-20T12:14:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.132+0000 7f6ef8830700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6ef40b6650 0x7f6ef40afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45142/0 (socket says 172.21.15.187:45142) 2020-11-20T12:14:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ef8830700 1 -- 172.21.15.187:0/3288860390 learned_addr learned my addr 172.21.15.187:0/3288860390 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ef9832700 1 --2- 172.21.15.187:0/3288860390 >> v2:172.21.15.5:3300/0 conn(0x7f6ef4066f70 0x7f6ef40b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967840067 0 0) 0x7f6ef40b15a0 con 0x7f6ef40b6650 2020-11-20T12:14:11.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6edc003240 con 0x7f6ef40b6650 2020-11-20T12:14:11.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1155195192 0 0) 0x7f6edc003240 con 0x7f6ef40b6650 2020-11-20T12:14:11.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 >> v2:172.21.15.5:3300/0 conn(0x7f6ef4066f70 msgr2=0x7f6ef40b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:11.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ee7fff700 1 --2- 172.21.15.187:0/3288860390 >> v2:172.21.15.5:3300/0 conn(0x7f6ef4066f70 0x7f6ef40b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 >> v1:172.21.15.5:6789/0 conn(0x7f6ef41484d0 legacy=0x7f6ef4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.133+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef40b1840 con 0x7f6ef40b6650 2020-11-20T12:14:11.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6ef00014e0 con 0x7f6ef40b6650 2020-11-20T12:14:11.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 >> v1:172.21.15.187:6789/0 conn(0x7f6ef40b6650 legacy=0x7f6ef40afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ee7fff700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6edc003730 0x7f6edc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ee7fff700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6edc0044e0 0x7f6edc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ee7fff700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc004e50 0x7f6edc00d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef4067730 con 0x7f6edc004e50 2020-11-20T12:14:11.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc004260 con 0x7f6edc0044e0 2020-11-20T12:14:11.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc00de30 con 0x7f6edc003730 2020-11-20T12:14:11.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ef8830700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6edc0044e0 0x7f6edc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.134+0000 7f6ef9832700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc004e50 0x7f6edc00d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ef9031700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6edc003730 0x7f6edc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ef8830700 1 -- 172.21.15.187:0/3288860390 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6edc003730 msgr2=0x7f6edc003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:11.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ef8830700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6edc003730 0x7f6edc003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ef8830700 1 -- 172.21.15.187:0/3288860390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc004e50 msgr2=0x7f6edc00d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:11.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ef8830700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc004e50 0x7f6edc00d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ef8830700 1 -- 172.21.15.187:0/3288860390 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc003240 con 0x7f6edc0044e0 2020-11-20T12:14:11.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ef9832700 1 -- 172.21.15.187:0/3288860390 reap_dead start 2020-11-20T12:14:11.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ef8830700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6edc0044e0 0x7f6edc004960 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f6ef0007e90 tx=0x7f6ef00077d0).ready entity=mon.1 client_cookie=a19b0eaf8f58c030 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:11.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000ce70 con 0x7f6edc0044e0 2020-11-20T12:14:11.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef00072e0 con 0x7f6edc0044e0 2020-11-20T12:14:11.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.135+0000 7f6ee7fff700 1 -- 172.21.15.187:0/3288860390 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0005800 con 0x7f6edc0044e0 2020-11-20T12:14:11.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.136+0000 7f6efa033700 1 -- 172.21.15.187:0/3288860390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6edc0044e0 msgr2=0x7f6edc004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.136+0000 7f6efa033700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6edc0044e0 0x7f6edc004960 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f6ef0007e90 tx=0x7f6ef00077d0).stop 2020-11-20T12:14:11.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.136+0000 7f6efa033700 1 -- 172.21.15.187:0/3288860390 shutdown_connections 2020-11-20T12:14:11.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.136+0000 7f6efa033700 1 --2- 172.21.15.187:0/3288860390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6edc0044e0 0x7f6edc004960 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.136+0000 7f6efa033700 1 -- 172.21.15.187:0/3288860390 >> 172.21.15.187:0/3288860390 conn(0x7f6ef40bab30 msgr2=0x7f6ef40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:11.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.136+0000 7f6efa033700 1 -- 172.21.15.187:0/3288860390 shutdown_connections 2020-11-20T12:14:11.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.136+0000 7f6efa033700 1 -- 172.21.15.187:0/3288860390 wait complete. 2020-11-20T12:14:11.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.138+0000 7f6efa033700 1 Processor -- start 2020-11-20T12:14:11.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.138+0000 7f6efa033700 1 -- start start 2020-11-20T12:14:11.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6efa033700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc0044e0 0x7f6ef4154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6efa033700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ef40a0730 0x7f6ef409dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6efa033700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ef409dfd0 0x7f6ef40b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6efa033700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef409ed80 con 0x7f6edc0044e0 2020-11-20T12:14:11.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6efa033700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef40a0bb0 con 0x7f6ef40a0730 2020-11-20T12:14:11.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6efa033700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef41547f0 con 0x7f6ef409dfd0 2020-11-20T12:14:11.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef8830700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ef40a0730 0x7f6ef409dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef8830700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ef40a0730 0x7f6ef409dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48548/0 (socket says 172.21.15.187:48548) 2020-11-20T12:14:11.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef8830700 1 -- 172.21.15.187:0/1182721222 learned_addr learned my addr 172.21.15.187:0/1182721222 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:11.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef9031700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc0044e0 0x7f6ef4154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef8830700 1 -- 172.21.15.187:0/1182721222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ef409dfd0 msgr2=0x7f6ef40b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef8830700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ef409dfd0 0x7f6ef40b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:11.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef8830700 1 -- 172.21.15.187:0/1182721222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc0044e0 msgr2=0x7f6ef4154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef8830700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc0044e0 0x7f6ef4154240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.139+0000 7f6ef8830700 1 -- 172.21.15.187:0/1182721222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef409ef80 con 0x7f6ef40a0730 2020-11-20T12:14:11.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.140+0000 7f6ef8830700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ef40a0730 0x7f6ef409dae0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f6edc0043f0 tx=0x7f6ef0017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:11.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.140+0000 7f6ee57fa700 1 -- 172.21.15.187:0/1182721222 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0011560 con 0x7f6ef40a0730 2020-11-20T12:14:11.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.140+0000 7f6ee57fa700 1 -- 172.21.15.187:0/1182721222 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef00116c0 con 0x7f6ef40a0730 2020-11-20T12:14:11.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.140+0000 7f6ee57fa700 1 -- 172.21.15.187:0/1182721222 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0011a40 con 0x7f6ef40a0730 2020-11-20T12:14:11.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.140+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef40a03b0 con 0x7f6ef40a0730 2020-11-20T12:14:11.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.140+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef40b46d0 con 0x7f6ef40a0730 2020-11-20T12:14:11.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.141+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef409f110 con 0x7f6ef40a0730 2020-11-20T12:14:11.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.142+0000 7f6ee57fa700 1 -- 172.21.15.187:0/1182721222 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6ef002c070 con 0x7f6ef40a0730 2020-11-20T12:14:11.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.142+0000 7f6ee57fa700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ecc056d50 0x7f6ecc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.143+0000 7f6ee57fa700 1 -- 172.21.15.187:0/1182721222 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6ef00755b0 con 0x7f6ef40a0730 2020-11-20T12:14:11.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.143+0000 7f6ef9031700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ecc056d50 0x7f6ecc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.143+0000 7f6ef9031700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ecc056d50 0x7f6ecc059210 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f6ee80034f0 tx=0x7f6ee8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:11.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.146+0000 7f6ee57fa700 1 -- 172.21.15.187:0/1182721222 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6ef004a980 con 0x7f6ef40a0730 2020-11-20T12:14:11.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.336+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6ef4061a20 con 0x7f6ef40a0730 2020-11-20T12:14:11.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.346+0000 7f6ee57fa700 1 -- 172.21.15.187:0/1182721222 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ef0001c40 con 0x7f6ef40a0730 2020-11-20T12:14:11.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.348+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ecc056d50 msgr2=0x7f6ecc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.348+0000 7f6efa033700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ecc056d50 0x7f6ecc059210 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f6ee80034f0 tx=0x7f6ee8009040).stop 2020-11-20T12:14:11.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.348+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ef40a0730 msgr2=0x7f6ef409dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.348+0000 7f6efa033700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ef40a0730 0x7f6ef409dae0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f6edc0043f0 tx=0x7f6ef0017000).stop 2020-11-20T12:14:11.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.349+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 shutdown_connections 2020-11-20T12:14:11.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.349+0000 7f6efa033700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ecc056d50 0x7f6ecc059210 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.349+0000 7f6efa033700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6edc0044e0 0x7f6ef4154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.349+0000 7f6efa033700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ef409dfd0 0x7f6ef40b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:11.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.349+0000 7f6efa033700 1 --2- 172.21.15.187:0/1182721222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ef40a0730 0x7f6ef409dae0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.349+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 >> 172.21.15.187:0/1182721222 conn(0x7f6ef40bab30 msgr2=0x7f6ef4143190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:11.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.349+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 shutdown_connections 2020-11-20T12:14:11.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.349+0000 7f6efa033700 1 -- 172.21.15.187:0/1182721222 wait complete. 2020-11-20T12:14:11.377 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup' 2020-11-20T12:14:11.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.624+0000 7fcc174f7700 1 Processor -- start 2020-11-20T12:14:11.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.624+0000 7fcc174f7700 1 -- start start 2020-11-20T12:14:11.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.625+0000 7fcc174f7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc10065880 0x7fcc100b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.625+0000 7fcc174f7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100b08e0 con 0x7fcc100654a0 2020-11-20T12:14:11.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.625+0000 7fcc174f7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100b0b80 con 0x7fcc100b6c10 2020-11-20T12:14:11.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.625+0000 7fcc174f7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100b05f0 con 0x7fcc10065880 2020-11-20T12:14:11.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.626+0000 7fcc15a94700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcc100b6c10 0x7fcc100666e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45162/0 (socket says 172.21.15.187:45162) 2020-11-20T12:14:11.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.626+0000 7fcc15a94700 1 -- 172.21.15.187:0/1284076920 learned_addr learned my addr 172.21.15.187:0/1284076920 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:11.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.626+0000 7fcc14a92700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc10065880 0x7fcc100b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277074938 0 0) 0x7fcc100b0b80 con 0x7fcc100b6c10 2020-11-20T12:14:11.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbfc003240 con 0x7fcc100b6c10 2020-11-20T12:14:11.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 956659503 0 0) 0x7fcbfc003240 con 0x7fcc100b6c10 2020-11-20T12:14:11.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc10065880 msgr2=0x7fcc100b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:11.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc10065880 0x7fcc100b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 >> v1:172.21.15.5:6789/0 conn(0x7fcc100654a0 legacy=0x7fcc10144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10145570 con 0x7fcc100b6c10 2020-11-20T12:14:11.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcc08001860 con 0x7fcc100b6c10 2020-11-20T12:14:11.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 >> v1:172.21.15.187:6789/0 conn(0x7fcc100b6c10 legacy=0x7fcc100666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.627+0000 7fcc0ffff700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbfc003730 0x7fcbfc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.628+0000 7fcc0ffff700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc10065880 0x7fcbfc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.628+0000 7fcc0ffff700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbfc0049d0 0x7fcbfc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.628+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100b05f0 con 0x7fcbfc0049d0 2020-11-20T12:14:11.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.628+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc004220 con 0x7fcbfc003730 2020-11-20T12:14:11.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.628+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc00d940 con 0x7fcc10065880 2020-11-20T12:14:11.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.628+0000 7fcc15a94700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbfc0049d0 0x7fcbfc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.628+0000 7fcc15293700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbfc003730 0x7fcbfc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.629+0000 7fcc14a92700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc10065880 0x7fcbfc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.629+0000 7fcc15293700 1 -- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc10065880 msgr2=0x7fcbfc0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:11.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.629+0000 7fcc15293700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc10065880 0x7fcbfc0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.629+0000 7fcc15293700 1 -- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbfc0049d0 msgr2=0x7fcbfc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:11.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.629+0000 7fcc15293700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbfc0049d0 0x7fcbfc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.629+0000 7fcc15293700 1 -- 172.21.15.187:0/1284076920 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc003240 con 0x7fcbfc003730 2020-11-20T12:14:11.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.629+0000 7fcc15a94700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbfc0049d0 0x7fcbfc004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:14:11.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.630+0000 7fcc15293700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbfc003730 0x7fcbfc003b50 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7fcc00001920 tx=0x7fcc00007b90).ready entity=mon.1 client_cookie=5280ae67b272bba4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:11.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.630+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00007020 con 0x7fcbfc003730 2020-11-20T12:14:11.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.630+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00007180 con 0x7fcbfc003730 2020-11-20T12:14:11.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.630+0000 7fcc0ffff700 1 -- 172.21.15.187:0/1284076920 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0000c830 con 0x7fcbfc003730 2020-11-20T12:14:11.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.631+0000 7fcc174f7700 1 -- 172.21.15.187:0/1284076920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbfc003730 msgr2=0x7fcbfc003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.631+0000 7fcc174f7700 1 --2- 172.21.15.187:0/1284076920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbfc003730 0x7fcbfc003b50 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7fcc00001920 tx=0x7fcc00007b90).stop 2020-11-20T12:14:11.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.631+0000 7fcc15a94700 1 -- 172.21.15.187:0/1284076920 reap_dead start 2020-11-20T12:14:11.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.631+0000 7fcc174f7700 1 -- 172.21.15.187:0/1284076920 shutdown_connections 2020-11-20T12:14:11.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.631+0000 7fcc174f7700 1 -- 172.21.15.187:0/1284076920 >> 172.21.15.187:0/1284076920 conn(0x7fcc100bacd0 msgr2=0x7fcc100bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:11.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.631+0000 7fcc174f7700 1 -- 172.21.15.187:0/1284076920 shutdown_connections 2020-11-20T12:14:11.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.631+0000 7fcc174f7700 1 -- 172.21.15.187:0/1284076920 wait complete. 2020-11-20T12:14:11.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.632+0000 7fcc174f7700 1 Processor -- start 2020-11-20T12:14:11.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.632+0000 7fcc174f7700 1 -- start start 2020-11-20T12:14:11.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.633+0000 7fcc174f7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc1015bba0 0x7fcc1015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.633+0000 7fcc174f7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc1009d3f0 0x7fcc101408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.633+0000 7fcc174f7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc10140de0 0x7fcc10098e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.633+0000 7fcc174f7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1009cc60 con 0x7fcc10140de0 2020-11-20T12:14:11.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.633+0000 7fcc174f7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1009cf00 con 0x7fcc1009d3f0 2020-11-20T12:14:11.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.633+0000 7fcc174f7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1009d1a0 con 0x7fcc1015bba0 2020-11-20T12:14:11.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc14a92700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc1009d3f0 0x7fcc101408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc14a92700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc1009d3f0 0x7fcc101408f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48568/0 (socket says 172.21.15.187:48568) 2020-11-20T12:14:11.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc14a92700 1 -- 172.21.15.187:0/652303936 learned_addr learned my addr 172.21.15.187:0/652303936 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:11.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc15293700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc1015bba0 0x7fcc1015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc15a94700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc10140de0 0x7fcc10098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc14a92700 1 -- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc1015bba0 msgr2=0x7fcc1015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc14a92700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc1015bba0 0x7fcc1015bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc14a92700 1 -- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc10140de0 msgr2=0x7fcc10098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:11.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc14a92700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc10140de0 0x7fcc10098e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:11.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc14a92700 1 -- 172.21.15.187:0/652303936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10141260 con 0x7fcc1009d3f0 2020-11-20T12:14:11.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.634+0000 7fcc15a94700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc10140de0 0x7fcc10098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:11.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.635+0000 7fcc14a92700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc1009d3f0 0x7fcc101408f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7fcc0400ba00 tx=0x7fcc04008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:11.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.635+0000 7fcc0d7fa700 1 -- 172.21.15.187:0/652303936 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04011930 con 0x7fcc1009d3f0 2020-11-20T12:14:11.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.635+0000 7fcc0d7fa700 1 -- 172.21.15.187:0/652303936 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc04011a90 con 0x7fcc1009d3f0 2020-11-20T12:14:11.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.635+0000 7fcc0d7fa700 1 -- 172.21.15.187:0/652303936 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04011e10 con 0x7fcc1009d3f0 2020-11-20T12:14:11.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.635+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1015c470 con 0x7fcc1009d3f0 2020-11-20T12:14:11.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.635+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc100995c0 con 0x7fcc1009d3f0 2020-11-20T12:14:11.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.636+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1015c660 con 0x7fcc1009d3f0 2020-11-20T12:14:11.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.637+0000 7fcc0d7fa700 1 -- 172.21.15.187:0/652303936 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcc04024070 con 0x7fcc1009d3f0 2020-11-20T12:14:11.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.638+0000 7fcc0d7fa700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbe8056d50 0x7fcbe8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:11.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.638+0000 7fcc0d7fa700 1 -- 172.21.15.187:0/652303936 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcc040709b0 con 0x7fcc1009d3f0 2020-11-20T12:14:11.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.638+0000 7fcc15293700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbe8056d50 0x7fcbe8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:11.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.639+0000 7fcc15293700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbe8056d50 0x7fcbe8059210 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fcc100647e0 tx=0x7fcc00015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:11.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.641+0000 7fcc0d7fa700 1 -- 172.21.15.187:0/652303936 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc04045d80 con 0x7fcc1009d3f0 2020-11-20T12:14:11.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:11.898+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup"]} v 0) v1 -- 0x7fcc1009caa0 con 0x7fcc1009d3f0 2020-11-20T12:14:12.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.803+0000 7fcc0d7fa700 1 -- 172.21.15.187:0/652303936 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup"]}]=0 v471) v1 ==== 130+0+0 (secure 0 0 0) 0x7fcc04018070 con 0x7fcc1009d3f0 2020-11-20T12:14:12.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbe8056d50 msgr2=0x7fcbe8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:12.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbe8056d50 0x7fcbe8059210 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fcc100647e0 tx=0x7fcc00015000).stop 2020-11-20T12:14:12.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc1009d3f0 msgr2=0x7fcc101408f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:12.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc1009d3f0 0x7fcc101408f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7fcc0400ba00 tx=0x7fcc04008040).stop 2020-11-20T12:14:12.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 shutdown_connections 2020-11-20T12:14:12.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbe8056d50 0x7fcbe8059210 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:12.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc10140de0 0x7fcc10098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:12.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc1015bba0 0x7fcc1015bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:12.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 --2- 172.21.15.187:0/652303936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc1009d3f0 0x7fcc101408f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:12.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.806+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 >> 172.21.15.187:0/652303936 conn(0x7fcc100bacd0 msgr2=0x7fcc10141d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:12.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.807+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 shutdown_connections 2020-11-20T12:14:12.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:12.807+0000 7fcc174f7700 1 -- 172.21.15.187:0/652303936 wait complete. 2020-11-20T12:14:12.836 INFO:tasks.cephfs_test_runner:test_mv_cleanup (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:14:12.837 INFO:tasks.cephfs_test_runner:Starting test: test_mv_hardlink_cleanup (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:14:12.838 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup' 2020-11-20T12:14:13.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.070+0000 7f51509c2700 1 Processor -- start 2020-11-20T12:14:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.070+0000 7f51509c2700 1 -- start start 2020-11-20T12:14:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.071+0000 7f51509c2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f514c0bd620 0x7f514c067e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.072+0000 7f51509c2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c067450 con 0x7f514c061330 2020-11-20T12:14:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.072+0000 7f51509c2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c0676f0 con 0x7f514c0bda00 2020-11-20T12:14:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.072+0000 7f51509c2700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c067990 con 0x7f514c0bd620 2020-11-20T12:14:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.072+0000 7f5143fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f514c0bda00 0x7f514c145a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45182/0 (socket says 172.21.15.187:45182) 2020-11-20T12:14:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.072+0000 7f5143fff700 1 -- 172.21.15.187:0/3021533459 learned_addr learned my addr 172.21.15.187:0/3021533459 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:13.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.072+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082257132 0 0) 0x7f514c0676f0 con 0x7f514c0bda00 2020-11-20T12:14:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f512c003240 con 0x7f514c0bda00 2020-11-20T12:14:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514b7fe700 1 --2- 172.21.15.187:0/3021533459 >> v2:172.21.15.5:3300/0 conn(0x7f514c0bd620 0x7f514c067e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3727851262 0 0) 0x7f512c003240 con 0x7f514c0bda00 2020-11-20T12:14:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 >> v2:172.21.15.5:3300/0 conn(0x7f514c0bd620 msgr2=0x7f514c067e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 --2- 172.21.15.187:0/3021533459 >> v2:172.21.15.5:3300/0 conn(0x7f514c0bd620 0x7f514c067e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 >> v1:172.21.15.5:6789/0 conn(0x7f514c061330 legacy=0x7f514c0a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c146170 con 0x7f514c0bda00 2020-11-20T12:14:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f51340014e0 con 0x7f514c0bda00 2020-11-20T12:14:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 >> v1:172.21.15.187:6789/0 conn(0x7f514c0bda00 legacy=0x7f514c145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f512c003800 0x7f512c003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f512c0044e0 0x7f512c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512c004e50 0x7f512c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c067990 con 0x7f512c004e50 2020-11-20T12:14:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c005300 con 0x7f512c003800 2020-11-20T12:14:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.073+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c00bc20 con 0x7f512c0044e0 2020-11-20T12:14:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.074+0000 7f514b7fe700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f512c003800 0x7f512c003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.074+0000 7f514b7fe700 1 -- 172.21.15.187:0/3021533459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f512c0044e0 msgr2=0x7f512c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:13.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.074+0000 7f514b7fe700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f512c0044e0 0x7f512c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:13.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.074+0000 7f514b7fe700 1 -- 172.21.15.187:0/3021533459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512c004e50 msgr2=0x7f512c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.074+0000 7f514b7fe700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512c004e50 0x7f512c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.074+0000 7f514b7fe700 1 -- 172.21.15.187:0/3021533459 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c003240 con 0x7f512c003800 2020-11-20T12:14:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.074+0000 7f514bfff700 1 -- 172.21.15.187:0/3021533459 reap_dead start 2020-11-20T12:14:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.074+0000 7f514b7fe700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f512c003800 0x7f512c003c00 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f513c009d80 tx=0x7f513c008040).ready entity=mon.1 client_cookie=12375aa69763bbf7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:13.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.075+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c00c070 con 0x7f512c003800 2020-11-20T12:14:13.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.075+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c00ea20 con 0x7f512c003800 2020-11-20T12:14:13.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.075+0000 7f514affd700 1 -- 172.21.15.187:0/3021533459 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c00ede0 con 0x7f512c003800 2020-11-20T12:14:13.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.075+0000 7f51509c2700 1 -- 172.21.15.187:0/3021533459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f512c003800 msgr2=0x7f512c003c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:13.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.075+0000 7f51509c2700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f512c003800 0x7f512c003c00 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f513c009d80 tx=0x7f513c008040).stop 2020-11-20T12:14:13.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.075+0000 7f51509c2700 1 -- 172.21.15.187:0/3021533459 shutdown_connections 2020-11-20T12:14:13.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.075+0000 7f51509c2700 1 --2- 172.21.15.187:0/3021533459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f512c003800 0x7f512c003c00 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:13.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.075+0000 7f51509c2700 1 -- 172.21.15.187:0/3021533459 >> 172.21.15.187:0/3021533459 conn(0x7f514c060c10 msgr2=0x7f514c0bcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:13.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.076+0000 7f51509c2700 1 -- 172.21.15.187:0/3021533459 shutdown_connections 2020-11-20T12:14:13.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.076+0000 7f51509c2700 1 -- 172.21.15.187:0/3021533459 wait complete. 2020-11-20T12:14:13.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.078+0000 7f51509c2700 1 Processor -- start 2020-11-20T12:14:13.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.078+0000 7f51509c2700 1 -- start start 2020-11-20T12:14:13.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.079+0000 7f51509c2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512c003800 0x7f514c12de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:13.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.079+0000 7f51509c2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c1309f0 0x7f514c158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:13.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.079+0000 7f51509c2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f514c15b070 0x7f514c147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:13.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.079+0000 7f51509c2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c158820 con 0x7f512c003800 2020-11-20T12:14:13.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.079+0000 7f51509c2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c15c460 con 0x7f514c1309f0 2020-11-20T12:14:13.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.079+0000 7f51509c2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c15c700 con 0x7f514c15b070 2020-11-20T12:14:13.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.079+0000 7f5143fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c1309f0 0x7f514c158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:13.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f5143fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c1309f0 0x7f514c158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48588/0 (socket says 172.21.15.187:48588) 2020-11-20T12:14:13.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f5143fff700 1 -- 172.21.15.187:0/3267462874 learned_addr learned my addr 172.21.15.187:0/3267462874 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:13.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f514b7fe700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512c003800 0x7f514c12de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:13.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f514bfff700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f514c15b070 0x7f514c147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:13.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f5143fff700 1 -- 172.21.15.187:0/3267462874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f514c15b070 msgr2=0x7f514c147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:13.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f5143fff700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f514c15b070 0x7f514c147440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:13.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f5143fff700 1 -- 172.21.15.187:0/3267462874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512c003800 msgr2=0x7f514c12de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:13.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f5143fff700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512c003800 0x7f514c12de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:13.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f5143fff700 1 -- 172.21.15.187:0/3267462874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c0b1e60 con 0x7f514c1309f0 2020-11-20T12:14:13.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.080+0000 7f5143fff700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c1309f0 0x7f514c158330 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f513400ca60 tx=0x7f5134001a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:13.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.081+0000 7f51437fe700 1 -- 172.21.15.187:0/3267462874 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134007170 con 0x7f514c1309f0 2020-11-20T12:14:13.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.081+0000 7f51437fe700 1 -- 172.21.15.187:0/3267462874 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5134007310 con 0x7f514c1309f0 2020-11-20T12:14:13.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.081+0000 7f51437fe700 1 -- 172.21.15.187:0/3267462874 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513400a9a0 con 0x7f514c1309f0 2020-11-20T12:14:13.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.081+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c12dc00 con 0x7f514c1309f0 2020-11-20T12:14:13.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.081+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c147930 con 0x7f514c1309f0 2020-11-20T12:14:13.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.082+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c147be0 con 0x7f514c1309f0 2020-11-20T12:14:13.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.083+0000 7f51437fe700 1 -- 172.21.15.187:0/3267462874 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5134010070 con 0x7f514c1309f0 2020-11-20T12:14:13.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.083+0000 7f51437fe700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5128056d50 0x7f5128059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:13.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.083+0000 7f51437fe700 1 -- 172.21.15.187:0/3267462874 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f513406f900 con 0x7f514c1309f0 2020-11-20T12:14:13.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.083+0000 7f514b7fe700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5128056d50 0x7f5128059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:13.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.084+0000 7f514b7fe700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5128056d50 0x7f5128059210 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f513c009d80 tx=0x7f513c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:13.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.087+0000 7f51437fe700 1 -- 172.21.15.187:0/3267462874 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5134044cd0 con 0x7f514c1309f0 2020-11-20T12:14:13.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.343+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup"]} v 0) v1 -- 0x7f514c0b1ff0 con 0x7f514c1309f0 2020-11-20T12:14:13.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.814+0000 7f51437fe700 1 -- 172.21.15.187:0/3267462874 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup"]}]=0 v472) v1 ==== 142+0+0 (secure 0 0 0) 0x7f51340482f0 con 0x7f514c1309f0 2020-11-20T12:14:13.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5128056d50 msgr2=0x7f5128059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:13.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5128056d50 0x7f5128059210 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f513c009d80 tx=0x7f513c008040).stop 2020-11-20T12:14:13.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c1309f0 msgr2=0x7f514c158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:13.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c1309f0 0x7f514c158330 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f513400ca60 tx=0x7f5134001a40).stop 2020-11-20T12:14:13.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 shutdown_connections 2020-11-20T12:14:13.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5128056d50 0x7f5128059210 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:13.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512c003800 0x7f514c12de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:13.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f514c15b070 0x7f514c147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:13.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 --2- 172.21.15.187:0/3267462874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c1309f0 0x7f514c158330 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:13.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.817+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 >> 172.21.15.187:0/3267462874 conn(0x7f514c060c10 msgr2=0x7f514c149160 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:13.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.818+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 shutdown_connections 2020-11-20T12:14:13.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:13.818+0000 7f51509c2700 1 -- 172.21.15.187:0/3267462874 wait complete. 2020-11-20T12:14:13.845 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:14:14.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.088+0000 7f7ccf308700 1 Processor -- start 2020-11-20T12:14:14.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.088+0000 7f7ccf308700 1 -- start start 2020-11-20T12:14:14.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.090+0000 7f7ccf308700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7cc80b6fd0 0x7f7cc8066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.090+0000 7f7ccf308700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7cc8067300 0x7f7cc8144d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.090+0000 7f7ccf308700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc8146010 con 0x7f7cc80b73b0 2020-11-20T12:14:14.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.090+0000 7f7ccf308700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc81462b0 con 0x7f7cc80b6fd0 2020-11-20T12:14:14.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.090+0000 7f7ccf308700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc8146550 con 0x7f7cc8067300 2020-11-20T12:14:14.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.090+0000 7f7ccd8a5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7cc8067300 0x7f7cc8144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.090+0000 7f7ccd8a5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7cc8067300 0x7f7cc8144d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48596/0 (socket says 172.21.15.187:48596) 2020-11-20T12:14:14.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.090+0000 7f7ccd8a5700 1 -- 172.21.15.187:0/164556984 learned_addr learned my addr 172.21.15.187:0/164556984 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.091+0000 7f7ccd8a5700 1 -- 172.21.15.187:0/164556984 >> v2:172.21.15.5:3300/0 conn(0x7f7cc80b6fd0 msgr2=0x7f7cc8066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.091+0000 7f7ccd0a4700 1 --2- 172.21.15.187:0/164556984 >> v2:172.21.15.5:3300/0 conn(0x7f7cc80b6fd0 0x7f7cc8066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.091+0000 7f7ccd8a5700 1 --2- 172.21.15.187:0/164556984 >> v2:172.21.15.5:3300/0 conn(0x7f7cc80b6fd0 0x7f7cc8066700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.091+0000 7f7ccd8a5700 1 -- 172.21.15.187:0/164556984 >> v1:172.21.15.5:6789/0 conn(0x7f7cc80b73b0 legacy=0x7f7cc8066bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:14.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.091+0000 7f7ccd8a5700 1 -- 172.21.15.187:0/164556984 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc80b6650 con 0x7f7cc8067300 2020-11-20T12:14:14.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.091+0000 7f7ccd8a5700 1 --2- 172.21.15.187:0/164556984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc8067300 0x7f7cc8144d90 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7f7cc809d9f0 tx=0x7f7cc0009290).ready entity=mon.1 client_cookie=92d8b525d7593128 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:14.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7cb7fff700 1 -- 172.21.15.187:0/164556984 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000bb80 con 0x7f7cc8067300 2020-11-20T12:14:14.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7cb7fff700 1 -- 172.21.15.187:0/164556984 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc000bce0 con 0x7f7cc8067300 2020-11-20T12:14:14.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7cb7fff700 1 -- 172.21.15.187:0/164556984 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0011610 con 0x7f7cc8067300 2020-11-20T12:14:14.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7ccf308700 1 -- 172.21.15.187:0/164556984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc8067300 msgr2=0x7f7cc8144d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7ccf308700 1 --2- 172.21.15.187:0/164556984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc8067300 0x7f7cc8144d90 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7f7cc809d9f0 tx=0x7f7cc0009290).stop 2020-11-20T12:14:14.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7ccf308700 1 -- 172.21.15.187:0/164556984 shutdown_connections 2020-11-20T12:14:14.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7ccf308700 1 --2- 172.21.15.187:0/164556984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc8067300 0x7f7cc8144d90 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7ccf308700 1 --2- 172.21.15.187:0/164556984 >> v2:172.21.15.5:3300/0 conn(0x7f7cc80b6fd0 0x7f7cc8066700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7ccf308700 1 -- 172.21.15.187:0/164556984 >> 172.21.15.187:0/164556984 conn(0x7f7cc80bacf0 msgr2=0x7f7cc80bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:14.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.092+0000 7f7ccf308700 1 -- 172.21.15.187:0/164556984 shutdown_connections 2020-11-20T12:14:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.093+0000 7f7ccf308700 1 -- 172.21.15.187:0/164556984 wait complete. 2020-11-20T12:14:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.094+0000 7f7ccf308700 1 Processor -- start 2020-11-20T12:14:14.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.094+0000 7f7ccf308700 1 -- start start 2020-11-20T12:14:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.095+0000 7f7ccf308700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cc8067300 0x7f7cc8145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.095+0000 7f7ccf308700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cc80b6fd0 0x7f7cc813b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.095+0000 7f7ccf308700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc80b73b0 0x7f7cc8133360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.095+0000 7f7ccf308700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc809d1b0 con 0x7f7cc80b6fd0 2020-11-20T12:14:14.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.095+0000 7f7ccf308700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc809d450 con 0x7f7cc80b73b0 2020-11-20T12:14:14.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.095+0000 7f7ccf308700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc809cee0 con 0x7f7cc8067300 2020-11-20T12:14:14.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd8a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc80b73b0 0x7f7cc8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd8a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc80b73b0 0x7f7cc8133360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48604/0 (socket says 172.21.15.187:48604) 2020-11-20T12:14:14.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd8a5700 1 -- 172.21.15.187:0/237616326 learned_addr learned my addr 172.21.15.187:0/237616326 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:14.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd0a4700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cc8067300 0x7f7cc8145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd8a5700 1 -- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cc8067300 msgr2=0x7f7cc8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccc8a3700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cc80b6fd0 0x7f7cc813b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd8a5700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cc8067300 0x7f7cc8145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd8a5700 1 -- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cc80b6fd0 msgr2=0x7f7cc813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd8a5700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cc80b6fd0 0x7f7cc813b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd8a5700 1 -- 172.21.15.187:0/237616326 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc8145c60 con 0x7f7cc80b73b0 2020-11-20T12:14:14.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.096+0000 7f7ccd0a4700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cc8067300 0x7f7cc8145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:14.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.097+0000 7f7ccd8a5700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc80b73b0 0x7f7cc8133360 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7f7cc00092d0 tx=0x7f7cc0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:14.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.097+0000 7f7cb57fa700 1 -- 172.21.15.187:0/237616326 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000bb80 con 0x7f7cc80b73b0 2020-11-20T12:14:14.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.097+0000 7f7cb57fa700 1 -- 172.21.15.187:0/237616326 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc000be50 con 0x7f7cc80b73b0 2020-11-20T12:14:14.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.097+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc813bde0 con 0x7f7cc80b73b0 2020-11-20T12:14:14.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.097+0000 7f7cb57fa700 1 -- 172.21.15.187:0/237616326 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0010e70 con 0x7f7cc80b73b0 2020-11-20T12:14:14.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.097+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc8133850 con 0x7f7cc80b73b0 2020-11-20T12:14:14.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.098+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc809d610 con 0x7f7cc80b73b0 2020-11-20T12:14:14.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.099+0000 7f7cb57fa700 1 -- 172.21.15.187:0/237616326 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7cc0017070 con 0x7f7cc80b73b0 2020-11-20T12:14:14.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.100+0000 7f7cb57fa700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ca0056d20 0x7f7ca00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.100+0000 7f7cb57fa700 1 -- 172.21.15.187:0/237616326 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7cc0072290 con 0x7f7cc80b73b0 2020-11-20T12:14:14.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.100+0000 7f7ccd0a4700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ca0056d20 0x7f7ca00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.101+0000 7f7ccd0a4700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ca0056d20 0x7f7ca00591e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f7cc400b7f0 tx=0x7f7cc4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:14.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.104+0000 7f7cb57fa700 1 -- 172.21.15.187:0/237616326 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7cc0047610 con 0x7f7cc80b73b0 2020-11-20T12:14:14.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.295+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f7cc80b67e0 con 0x7f7cc80b73b0 2020-11-20T12:14:14.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.304+0000 7f7cb57fa700 1 -- 172.21.15.187:0/237616326 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7cc0015020 con 0x7f7cc80b73b0 2020-11-20T12:14:14.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ca0056d20 msgr2=0x7f7ca00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ca0056d20 0x7f7ca00591e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f7cc400b7f0 tx=0x7f7cc4008040).stop 2020-11-20T12:14:14.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc80b73b0 msgr2=0x7f7cc8133360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc80b73b0 0x7f7cc8133360 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7f7cc00092d0 tx=0x7f7cc0016000).stop 2020-11-20T12:14:14.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 shutdown_connections 2020-11-20T12:14:14.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ca0056d20 0x7f7ca00591e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cc80b73b0 0x7f7cc8133360 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cc8067300 0x7f7cc8145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 --2- 172.21.15.187:0/237616326 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cc80b6fd0 0x7f7cc813b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 >> 172.21.15.187:0/237616326 conn(0x7f7cc80bacf0 msgr2=0x7f7cc813d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:14.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 shutdown_connections 2020-11-20T12:14:14.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.307+0000 7f7ccf308700 1 -- 172.21.15.187:0/237616326 wait complete. 2020-11-20T12:14:14.332 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:14:14.332 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:14:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:14:14.468 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:14:14.586 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:14:14.598 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:14:14.599 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:14:14.599 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:14.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.850+0000 7f57d3fff700 1 Processor -- start 2020-11-20T12:14:14.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.850+0000 7f57d3fff700 1 -- start start 2020-11-20T12:14:14.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.852+0000 7f57d3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1380 0x7f57d4150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.852+0000 7f57d3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f57d40b1760 0x7f57d4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.852+0000 7f57d3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d4150390 con 0x7f57d4065680 2020-11-20T12:14:14.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.852+0000 7f57d3fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d4150630 con 0x7f57d40b1760 2020-11-20T12:14:14.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.852+0000 7f57d3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d4068510 con 0x7f57d40b1380 2020-11-20T12:14:14.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.852+0000 7f57d27fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f57d40b1760 0x7f57d4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.852+0000 7f57d27fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f57d40b1760 0x7f57d4065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48612/0 (socket says 172.21.15.187:48612) 2020-11-20T12:14:14.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.853+0000 7f57d27fc700 1 -- 172.21.15.187:0/2692527675 learned_addr learned my addr 172.21.15.187:0/2692527675 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:14.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.853+0000 7f57d2ffd700 1 --2- 172.21.15.187:0/2692527675 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1380 0x7f57d4150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.853+0000 7f57d27fc700 1 -- 172.21.15.187:0/2692527675 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1380 msgr2=0x7f57d4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:14.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.853+0000 7f57d27fc700 1 --2- 172.21.15.187:0/2692527675 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1380 0x7f57d4150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.853+0000 7f57d27fc700 1 -- 172.21.15.187:0/2692527675 >> v1:172.21.15.5:6789/0 conn(0x7f57d4065680 legacy=0x7f57d4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.853+0000 7f57d27fc700 1 -- 172.21.15.187:0/2692527675 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d4150fc0 con 0x7f57d40b1760 2020-11-20T12:14:14.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.853+0000 7f57d27fc700 1 --2- 172.21.15.187:0/2692527675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1760 0x7f57d4065190 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7f57c8009d50 tx=0x7f57c8009290).ready entity=mon.1 client_cookie=7ff6cab3c7127f89 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:14.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.853+0000 7f57d1ffb700 1 -- 172.21.15.187:0/2692527675 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c800bc80 con 0x7f57d40b1760 2020-11-20T12:14:14.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d1ffb700 1 -- 172.21.15.187:0/2692527675 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c800bde0 con 0x7f57d40b1760 2020-11-20T12:14:14.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d1ffb700 1 -- 172.21.15.187:0/2692527675 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c8011610 con 0x7f57d40b1760 2020-11-20T12:14:14.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d3fff700 1 -- 172.21.15.187:0/2692527675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1760 msgr2=0x7f57d4065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d3fff700 1 --2- 172.21.15.187:0/2692527675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1760 0x7f57d4065190 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7f57c8009d50 tx=0x7f57c8009290).stop 2020-11-20T12:14:14.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d3fff700 1 -- 172.21.15.187:0/2692527675 shutdown_connections 2020-11-20T12:14:14.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d3fff700 1 --2- 172.21.15.187:0/2692527675 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1380 0x7f57d4150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d3fff700 1 --2- 172.21.15.187:0/2692527675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1760 0x7f57d4065190 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d3fff700 1 -- 172.21.15.187:0/2692527675 >> 172.21.15.187:0/2692527675 conn(0x7f57d40bab30 msgr2=0x7f57d40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:14.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d3fff700 1 -- 172.21.15.187:0/2692527675 shutdown_connections 2020-11-20T12:14:14.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.854+0000 7f57d3fff700 1 -- 172.21.15.187:0/2692527675 wait complete. 2020-11-20T12:14:14.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.856+0000 7f57d3fff700 1 Processor -- start 2020-11-20T12:14:14.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.856+0000 7f57d3fff700 1 -- start start 2020-11-20T12:14:14.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.857+0000 7f57d3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1380 0x7f57d40b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.857+0000 7f57d3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1760 0x7f57d41496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.857+0000 7f57d3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40a08a0 0x7f57d4149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.857+0000 7f57d3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d414a0c0 con 0x7f57d40a08a0 2020-11-20T12:14:14.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.857+0000 7f57d3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d409e5c0 con 0x7f57d40b1380 2020-11-20T12:14:14.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.857+0000 7f57d3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d409e860 con 0x7f57d40b1760 2020-11-20T12:14:14.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1380 0x7f57d40b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1380 0x7f57d40b3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48614/0 (socket says 172.21.15.187:48614) 2020-11-20T12:14:14.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d2ffd700 1 -- 172.21.15.187:0/1833493159 learned_addr learned my addr 172.21.15.187:0/1833493159 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:14.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d2ffd700 1 -- 172.21.15.187:0/1833493159 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1760 msgr2=0x7f57d41496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d37fe700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40a08a0 0x7f57d4149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d27fc700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1760 0x7f57d41496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d2ffd700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1760 0x7f57d41496e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d2ffd700 1 -- 172.21.15.187:0/1833493159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40a08a0 msgr2=0x7f57d4149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:14.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d2ffd700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40a08a0 0x7f57d4149bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:14.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.858+0000 7f57d2ffd700 1 -- 172.21.15.187:0/1833493159 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d40b7ef0 con 0x7f57d40b1380 2020-11-20T12:14:14.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.859+0000 7f57d2ffd700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1380 0x7f57d40b3e50 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7f57c400db60 tx=0x7f57c4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:14.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.859+0000 7f57baffd700 1 -- 172.21.15.187:0/1833493159 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c4009790 con 0x7f57d40b1380 2020-11-20T12:14:14.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.859+0000 7f57baffd700 1 -- 172.21.15.187:0/1833493159 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c40098f0 con 0x7f57d40b1380 2020-11-20T12:14:14.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.859+0000 7f57baffd700 1 -- 172.21.15.187:0/1833493159 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c4009c90 con 0x7f57d40b1380 2020-11-20T12:14:14.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.859+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d409eb00 con 0x7f57d40b1380 2020-11-20T12:14:14.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.859+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d40b4340 con 0x7f57d40b1380 2020-11-20T12:14:14.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.860+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d4061a20 con 0x7f57d40b1380 2020-11-20T12:14:14.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.861+0000 7f57baffd700 1 -- 172.21.15.187:0/1833493159 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f57c4013480 con 0x7f57d40b1380 2020-11-20T12:14:14.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.861+0000 7f57baffd700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57b0056d50 0x7f57b0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:14.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.862+0000 7f57baffd700 1 -- 172.21.15.187:0/1833493159 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f57c4070690 con 0x7f57d40b1380 2020-11-20T12:14:14.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.862+0000 7f57d27fc700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57b0056d50 0x7f57b0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:14.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.862+0000 7f57d27fc700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57b0056d50 0x7f57b0059210 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f57d4150170 tx=0x7f57c8016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:14.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:14.865+0000 7f57baffd700 1 -- 172.21.15.187:0/1833493159 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f57c4045970 con 0x7f57d40b1380 2020-11-20T12:14:15.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.108+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57d40b8080 con 0x7f57d40b1380 2020-11-20T12:14:15.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.109+0000 7f57baffd700 1 -- 172.21.15.187:0/1833493159 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f57c4048f90 con 0x7f57d40b1380 2020-11-20T12:14:15.113 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:15.113 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:14:15.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57b0056d50 msgr2=0x7f57b0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57b0056d50 0x7f57b0059210 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f57d4150170 tx=0x7f57c8016000).stop 2020-11-20T12:14:15.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1380 msgr2=0x7f57d40b3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1380 0x7f57d40b3e50 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7f57c400db60 tx=0x7f57c4008040).stop 2020-11-20T12:14:15.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 shutdown_connections 2020-11-20T12:14:15.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57b0056d50 0x7f57b0059210 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40a08a0 0x7f57d4149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40b1380 0x7f57d40b3e50 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 --2- 172.21.15.187:0/1833493159 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40b1760 0x7f57d41496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 >> 172.21.15.187:0/1833493159 conn(0x7f57d40bab30 msgr2=0x7f57d414a870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:15.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 shutdown_connections 2020-11-20T12:14:15.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.112+0000 7f57d3fff700 1 -- 172.21.15.187:0/1833493159 wait complete. 2020-11-20T12:14:15.118 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:14:15.138 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:15.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.380+0000 7f70da250700 1 Processor -- start 2020-11-20T12:14:15.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.380+0000 7f70da250700 1 -- start start 2020-11-20T12:14:15.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.381+0000 7f70da250700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70d4065190 0x7f70d4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.381+0000 7f70da250700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d4143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.381+0000 7f70da250700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d40b13e0 con 0x7f70d4143c70 2020-11-20T12:14:15.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.381+0000 7f70da250700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d414ab30 con 0x7f70d4065190 2020-11-20T12:14:15.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.382+0000 7f70da250700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d4062c10 con 0x7f70d4065f90 2020-11-20T12:14:15.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.382+0000 7f70d924e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70d4065190 0x7f70d4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.382+0000 7f70d924e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70d4065190 0x7f70d4065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48622/0 (socket says 172.21.15.187:48622) 2020-11-20T12:14:15.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.382+0000 7f70d924e700 1 -- 172.21.15.187:0/1414595542 learned_addr learned my addr 172.21.15.187:0/1414595542 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:15.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.382+0000 7f70d8a4d700 1 --2- 172.21.15.187:0/1414595542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d4143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.382+0000 7f70d924e700 1 -- 172.21.15.187:0/1414595542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 msgr2=0x7f70d4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:15.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.382+0000 7f70d924e700 1 --2- 172.21.15.187:0/1414595542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d4143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.382+0000 7f70d924e700 1 -- 172.21.15.187:0/1414595542 >> v1:172.21.15.5:6789/0 conn(0x7f70d4143c70 legacy=0x7f70d409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:15.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.383+0000 7f70d924e700 1 -- 172.21.15.187:0/1414595542 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d40b1640 con 0x7f70d4065190 2020-11-20T12:14:15.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.383+0000 7f70d924e700 1 --2- 172.21.15.187:0/1414595542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4065190 0x7f70d4065570 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f70d414ac70 tx=0x7f70d0006eb0).ready entity=mon.1 client_cookie=38da385519e8852f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:15.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.383+0000 7f70c3fff700 1 -- 172.21.15.187:0/1414595542 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0009920 con 0x7f70d4065190 2020-11-20T12:14:15.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.383+0000 7f70c3fff700 1 -- 172.21.15.187:0/1414595542 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d0009a80 con 0x7f70d4065190 2020-11-20T12:14:15.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.383+0000 7f70c3fff700 1 -- 172.21.15.187:0/1414595542 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0009e00 con 0x7f70d4065190 2020-11-20T12:14:15.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.384+0000 7f70da250700 1 -- 172.21.15.187:0/1414595542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4065190 msgr2=0x7f70d4065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.384+0000 7f70da250700 1 --2- 172.21.15.187:0/1414595542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4065190 0x7f70d4065570 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f70d414ac70 tx=0x7f70d0006eb0).stop 2020-11-20T12:14:15.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.384+0000 7f70da250700 1 -- 172.21.15.187:0/1414595542 shutdown_connections 2020-11-20T12:14:15.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.384+0000 7f70da250700 1 --2- 172.21.15.187:0/1414595542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4065190 0x7f70d4065570 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.384+0000 7f70da250700 1 --2- 172.21.15.187:0/1414595542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d4143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.384+0000 7f70da250700 1 -- 172.21.15.187:0/1414595542 >> 172.21.15.187:0/1414595542 conn(0x7f70d40bab30 msgr2=0x7f70d40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:15.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.384+0000 7f70da250700 1 -- 172.21.15.187:0/1414595542 shutdown_connections 2020-11-20T12:14:15.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.384+0000 7f70da250700 1 -- 172.21.15.187:0/1414595542 wait complete. 2020-11-20T12:14:15.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.386+0000 7f70da250700 1 Processor -- start 2020-11-20T12:14:15.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.386+0000 7f70da250700 1 -- start start 2020-11-20T12:14:15.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.387+0000 7f70da250700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70d4065190 0x7f70d40b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.387+0000 7f70da250700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d40994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.387+0000 7f70da250700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4143c70 0x7f70d415b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.387+0000 7f70da250700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d4067670 con 0x7f70d4065190 2020-11-20T12:14:15.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.387+0000 7f70da250700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d40b1b40 con 0x7f70d4143c70 2020-11-20T12:14:15.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.387+0000 7f70da250700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d41468d0 con 0x7f70d4065f90 2020-11-20T12:14:15.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4143c70 0x7f70d415b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4143c70 0x7f70d415b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48630/0 (socket says 172.21.15.187:48630) 2020-11-20T12:14:15.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 -- 172.21.15.187:0/1096378550 learned_addr learned my addr 172.21.15.187:0/1096378550 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:15.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d8a4d700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d40994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 -- 172.21.15.187:0/1096378550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 msgr2=0x7f70d40994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d40994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 -- 172.21.15.187:0/1096378550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70d4065190 msgr2=0x7f70d40b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70d4065190 0x7f70d40b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:15.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 -- 172.21.15.187:0/1096378550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d40b1de0 con 0x7f70d4143c70 2020-11-20T12:14:15.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d8a4d700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d40994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:15.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.388+0000 7f70d9a4f700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4143c70 0x7f70d415b9a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f70cc00bae0 tx=0x7f70cc0090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:15.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.389+0000 7f70c17fa700 1 -- 172.21.15.187:0/1096378550 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc00c7d0 con 0x7f70d4143c70 2020-11-20T12:14:15.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.389+0000 7f70c17fa700 1 -- 172.21.15.187:0/1096378550 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc00c950 con 0x7f70d4143c70 2020-11-20T12:14:15.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.389+0000 7f70c17fa700 1 -- 172.21.15.187:0/1096378550 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc00cb00 con 0x7f70d4143c70 2020-11-20T12:14:15.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.389+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d4066670 con 0x7f70d4143c70 2020-11-20T12:14:15.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.389+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d40b2ba0 con 0x7f70d4143c70 2020-11-20T12:14:15.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.390+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d4061a20 con 0x7f70d4143c70 2020-11-20T12:14:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.391+0000 7f70c17fa700 1 -- 172.21.15.187:0/1096378550 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f70cc01c070 con 0x7f70d4143c70 2020-11-20T12:14:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.391+0000 7f70c17fa700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70ac056d50 0x7f70ac059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.391+0000 7f70c17fa700 1 -- 172.21.15.187:0/1096378550 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f70cc06f6c0 con 0x7f70d4143c70 2020-11-20T12:14:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.392+0000 7f70d924e700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70ac056d50 0x7f70ac059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.392+0000 7f70d924e700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70ac056d50 0x7f70ac059210 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f70d000b1b0 tx=0x7f70d0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:15.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.396+0000 7f70c17fa700 1 -- 172.21.15.187:0/1096378550 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f70cc044ad0 con 0x7f70d4143c70 2020-11-20T12:14:15.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.582+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70d40afc60 con 0x7f70d4143c70 2020-11-20T12:14:15.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.583+0000 7f70c17fa700 1 -- 172.21.15.187:0/1096378550 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f70cc0480f0 con 0x7f70d4143c70 2020-11-20T12:14:15.586 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:15.586 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":89,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:13:41.257868+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:13:38.491662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2020-11-20T12:13:39.231237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:15.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70ac056d50 msgr2=0x7f70ac059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70ac056d50 0x7f70ac059210 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f70d000b1b0 tx=0x7f70d0015000).stop 2020-11-20T12:14:15.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4143c70 msgr2=0x7f70d415b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4143c70 0x7f70d415b9a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f70cc00bae0 tx=0x7f70cc0090f0).stop 2020-11-20T12:14:15.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 shutdown_connections 2020-11-20T12:14:15.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70ac056d50 0x7f70ac059210 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70d4065190 0x7f70d40b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:15.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d4143c70 0x7f70d415b9a0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 --2- 172.21.15.187:0/1096378550 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70d4065f90 0x7f70d40994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.585+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 >> 172.21.15.187:0/1096378550 conn(0x7f70d40bab30 msgr2=0x7f70d40b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:15.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.586+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 shutdown_connections 2020-11-20T12:14:15.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.586+0000 7f70da250700 1 -- 172.21.15.187:0/1096378550 wait complete. 2020-11-20T12:14:15.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:13:38.491662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:13:39.231237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:14:15.615 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:14:15.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.867+0000 7f16fd773700 1 Processor -- start 2020-11-20T12:14:15.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.867+0000 7f16fd773700 1 -- start start 2020-11-20T12:14:15.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.868+0000 7f16fd773700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f16f8060330 0x7f16f8067c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.868+0000 7f16fd773700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f8060710 0x7f16f8066540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.868+0000 7f16fd773700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80b65d0 con 0x7f16f80b8000 2020-11-20T12:14:15.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.868+0000 7f16fd773700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f80677d0 con 0x7f16f8060330 2020-11-20T12:14:15.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.868+0000 7f16fd773700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80b5f30 con 0x7f16f8060710 2020-11-20T12:14:15.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.869+0000 7f16fcf72700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f16f80b8000 0x7f16f8145ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45242/0 (socket says 172.21.15.187:45242) 2020-11-20T12:14:15.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.869+0000 7f16fcf72700 1 -- 172.21.15.187:0/3864947879 learned_addr learned my addr 172.21.15.187:0/3864947879 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:15.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.869+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767488217 0 0) 0x7f16f80b65d0 con 0x7f16f80b8000 2020-11-20T12:14:15.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.869+0000 7f16ef7fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f8060710 0x7f16f8066540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.869+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8003240 con 0x7f16f80b8000 2020-11-20T12:14:15.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.869+0000 7f16f7fff700 1 --2- 172.21.15.187:0/3864947879 >> v2:172.21.15.187:3300/0 conn(0x7f16f8060330 0x7f16f8067c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1402918539 0 0) 0x7f16d8003240 con 0x7f16f80b8000 2020-11-20T12:14:15.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f8060710 msgr2=0x7f16f8066540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:15.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f8060710 0x7f16f8066540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 >> v2:172.21.15.187:3300/0 conn(0x7f16f8060330 msgr2=0x7f16f8067c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:15.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 --2- 172.21.15.187:0/3864947879 >> v2:172.21.15.187:3300/0 conn(0x7f16f8060330 0x7f16f8067c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80b6250 con 0x7f16f80b8000 2020-11-20T12:14:15.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f16f0001860 con 0x7f16f80b8000 2020-11-20T12:14:15.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f7fff700 1 --2- 172.21.15.187:0/3864947879 >> v2:172.21.15.187:3300/0 conn(0x7f16f8060330 0x7f16f8067c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:15.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 >> v1:172.21.15.187:6789/0 conn(0x7f16f80b8000 legacy=0x7f16f8145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f8060710 0x7f16d8003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16d8003c40 0x7f16d80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16d80049d0 0x7f16d8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80677d0 con 0x7f16d80049d0 2020-11-20T12:14:15.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80b5f30 con 0x7f16d8003c40 2020-11-20T12:14:15.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8004200 con 0x7f16f8060710 2020-11-20T12:14:15.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.870+0000 7f16ef7fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16d8003c40 0x7f16d80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.871+0000 7f16fcf72700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16d80049d0 0x7f16d8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.871+0000 7f16f7fff700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f8060710 0x7f16d8003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.871+0000 7f16ef7fe700 1 -- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f8060710 msgr2=0x7f16d8003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:15.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.871+0000 7f16ef7fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f8060710 0x7f16d8003730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.871+0000 7f16ef7fe700 1 -- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16d80049d0 msgr2=0x7f16d8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:15.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.871+0000 7f16ef7fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16d80049d0 0x7f16d8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.871+0000 7f16ef7fe700 1 -- 172.21.15.187:0/3864947879 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8003240 con 0x7f16d8003c40 2020-11-20T12:14:15.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.871+0000 7f16ef7fe700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16d8003c40 0x7f16d80044e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f16e000ba50 tx=0x7f16e0008040).ready entity=mon.1 client_cookie=e80c20d5c8527e5f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:15.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.872+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000d9c0 con 0x7f16d8003c40 2020-11-20T12:14:15.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.872+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e000db20 con 0x7f16d8003c40 2020-11-20T12:14:15.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.872+0000 7f16f77fe700 1 -- 172.21.15.187:0/3864947879 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000dea0 con 0x7f16d8003c40 2020-11-20T12:14:15.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.872+0000 7f16fd773700 1 -- 172.21.15.187:0/3864947879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16d8003c40 msgr2=0x7f16d80044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.872+0000 7f16fd773700 1 --2- 172.21.15.187:0/3864947879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16d8003c40 0x7f16d80044e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f16e000ba50 tx=0x7f16e0008040).stop 2020-11-20T12:14:15.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.872+0000 7f16fcf72700 1 -- 172.21.15.187:0/3864947879 reap_dead start 2020-11-20T12:14:15.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.872+0000 7f16fd773700 1 -- 172.21.15.187:0/3864947879 shutdown_connections 2020-11-20T12:14:15.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.872+0000 7f16fd773700 1 -- 172.21.15.187:0/3864947879 >> 172.21.15.187:0/3864947879 conn(0x7f16f80bbc50 msgr2=0x7f16f80bc060 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:15.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.873+0000 7f16fd773700 1 -- 172.21.15.187:0/3864947879 shutdown_connections 2020-11-20T12:14:15.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.873+0000 7f16fd773700 1 -- 172.21.15.187:0/3864947879 wait complete. 2020-11-20T12:14:15.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.874+0000 7f16fd773700 1 Processor -- start 2020-11-20T12:14:15.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.875+0000 7f16fd773700 1 -- start start 2020-11-20T12:14:15.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16fd773700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f8136c40 0x7f16f81360b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16fd773700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f8137020 0x7f16f815b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16fd773700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f81365a0 0x7f16f815bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16fd773700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80b55c0 con 0x7f16f8137020 2020-11-20T12:14:15.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16fd773700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80b5860 con 0x7f16f8136c40 2020-11-20T12:14:15.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16fd773700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f81473a0 con 0x7f16f81365a0 2020-11-20T12:14:15.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16f7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f8136c40 0x7f16f81360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16f7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f8136c40 0x7f16f81360b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48648/0 (socket says 172.21.15.187:48648) 2020-11-20T12:14:15.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.876+0000 7f16f7fff700 1 -- 172.21.15.187:0/1814995982 learned_addr learned my addr 172.21.15.187:0/1814995982 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:15.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16ef7fe700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f8137020 0x7f16f815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16fcf72700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f81365a0 0x7f16f815bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f7fff700 1 -- 172.21.15.187:0/1814995982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f81365a0 msgr2=0x7f16f815bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f7fff700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f81365a0 0x7f16f815bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f7fff700 1 -- 172.21.15.187:0/1814995982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f8137020 msgr2=0x7f16f815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f7fff700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f8137020 0x7f16f815b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f7fff700 1 -- 172.21.15.187:0/1814995982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8137470 con 0x7f16f8136c40 2020-11-20T12:14:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f7fff700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f8136c40 0x7f16f81360b0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f16e800e840 tx=0x7f16e8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f4ff9700 1 -- 172.21.15.187:0/1814995982 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8017070 con 0x7f16f8136c40 2020-11-20T12:14:15.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f4ff9700 1 -- 172.21.15.187:0/1814995982 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e8013430 con 0x7f16f8136c40 2020-11-20T12:14:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f80b5190 con 0x7f16f8136c40 2020-11-20T12:14:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.877+0000 7f16f4ff9700 1 -- 172.21.15.187:0/1814995982 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e80137d0 con 0x7f16f8136c40 2020-11-20T12:14:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.878+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f80ba0a0 con 0x7f16f8136c40 2020-11-20T12:14:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.879+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f8147560 con 0x7f16f8136c40 2020-11-20T12:14:15.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.879+0000 7f16f4ff9700 1 -- 172.21.15.187:0/1814995982 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f16e8013b40 con 0x7f16f8136c40 2020-11-20T12:14:15.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.880+0000 7f16f4ff9700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16d4056d20 0x7f16d40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:15.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.880+0000 7f16f4ff9700 1 -- 172.21.15.187:0/1814995982 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f16e806f980 con 0x7f16f8136c40 2020-11-20T12:14:15.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.880+0000 7f16ef7fe700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16d4056d20 0x7f16d40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:15.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.881+0000 7f16ef7fe700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16d4056d20 0x7f16d40591e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f16f805ec00 tx=0x7f16e0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:15.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:15.884+0000 7f16f4ff9700 1 -- 172.21.15.187:0/1814995982 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f16e8044d70 con 0x7f16f8136c40 2020-11-20T12:14:16.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.065+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f16f80649c0 con 0x7f16f8136c40 2020-11-20T12:14:16.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.067+0000 7f16f4ff9700 1 -- 172.21.15.187:0/1814995982 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v89) v1 ==== 110+0+10 (secure 0 0 0) 0x7f16e8048390 con 0x7f16f8136c40 2020-11-20T12:14:16.070 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:14:16.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16d4056d20 msgr2=0x7f16d40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16d4056d20 0x7f16d40591e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f16f805ec00 tx=0x7f16e0008040).stop 2020-11-20T12:14:16.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f8136c40 msgr2=0x7f16f81360b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f8136c40 0x7f16f81360b0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f16e800e840 tx=0x7f16e8008040).stop 2020-11-20T12:14:16.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 shutdown_connections 2020-11-20T12:14:16.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16d4056d20 0x7f16d40591e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f16f81365a0 0x7f16f815bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16f8136c40 0x7f16f81360b0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 --2- 172.21.15.187:0/1814995982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f16f8137020 0x7f16f815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.069+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 >> 172.21.15.187:0/1814995982 conn(0x7f16f80bbc50 msgr2=0x7f16f8143170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:16.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.070+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 shutdown_connections 2020-11-20T12:14:16.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.070+0000 7f16fd773700 1 -- 172.21.15.187:0/1814995982 wait complete. 2020-11-20T12:14:16.096 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:14:16.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.335+0000 7f864e795700 1 Processor -- start 2020-11-20T12:14:16.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.335+0000 7f864e795700 1 -- start start 2020-11-20T12:14:16.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.337+0000 7f864e795700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86480679b0 0x7f8648147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.337+0000 7f864e795700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f86480ba9b0 0x7f86481481e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.337+0000 7f864e795700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864814be70 con 0x7f86480675d0 2020-11-20T12:14:16.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.337+0000 7f864e795700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864814c110 con 0x7f86480ba9b0 2020-11-20T12:14:16.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.337+0000 7f864e795700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864814c3b0 con 0x7f86480679b0 2020-11-20T12:14:16.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.337+0000 7f864d793700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f86480675d0 0x7f864814c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45260/0 (socket says 172.21.15.187:45260) 2020-11-20T12:14:16.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.337+0000 7f864d793700 1 -- 172.21.15.187:0/4042614151 learned_addr learned my addr 172.21.15.187:0/4042614151 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:16.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.337+0000 7f864cf92700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86480679b0 0x7f8648147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498515464 0 0) 0x7f864814be70 con 0x7f86480675d0 2020-11-20T12:14:16.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f864df94700 1 --2- 172.21.15.187:0/4042614151 >> v2:172.21.15.187:3300/0 conn(0x7f86480ba9b0 0x7f86481481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630003240 con 0x7f86480675d0 2020-11-20T12:14:16.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2123272128 0 0) 0x7f8630003240 con 0x7f86480675d0 2020-11-20T12:14:16.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86480679b0 msgr2=0x7f8648147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:16.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86480679b0 0x7f8648147cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 >> v2:172.21.15.187:3300/0 conn(0x7f86480ba9b0 msgr2=0x7f86481481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:16.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 --2- 172.21.15.187:0/4042614151 >> v2:172.21.15.187:3300/0 conn(0x7f86480ba9b0 0x7f86481481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86481486d0 con 0x7f86480675d0 2020-11-20T12:14:16.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f86440014e0 con 0x7f86480675d0 2020-11-20T12:14:16.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 >> v1:172.21.15.187:6789/0 conn(0x7f86480675d0 legacy=0x7f864814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8630003730 0x7f8630003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86300044e0 0x7f8630004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86480679b0 0x7f8630004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864814c110 con 0x7f86300044e0 2020-11-20T12:14:16.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864814c3b0 con 0x7f8630003730 2020-11-20T12:14:16.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.338+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8630004270 con 0x7f86480679b0 2020-11-20T12:14:16.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f864d793700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8630003730 0x7f8630003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f864d793700 1 -- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86480679b0 msgr2=0x7f8630004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:16.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f864d793700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86480679b0 0x7f8630004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:16.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f864d793700 1 -- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86300044e0 msgr2=0x7f8630004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:16.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f864d793700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86300044e0 0x7f8630004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:16.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f864d793700 1 -- 172.21.15.187:0/4042614151 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630003240 con 0x7f8630003730 2020-11-20T12:14:16.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f864d793700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8630003730 0x7f8630003b50 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f864400c440 tx=0x7f86480bbee0).ready entity=mon.1 client_cookie=a491d254b0cabfec server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:16.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86440072e0 con 0x7f8630003730 2020-11-20T12:14:16.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.339+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644007440 con 0x7f8630003730 2020-11-20T12:14:16.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.340+0000 7f8637fff700 1 -- 172.21.15.187:0/4042614151 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644005840 con 0x7f8630003730 2020-11-20T12:14:16.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.340+0000 7f864e795700 1 -- 172.21.15.187:0/4042614151 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8630003730 msgr2=0x7f8630003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.340+0000 7f864e795700 1 --2- 172.21.15.187:0/4042614151 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8630003730 0x7f8630003b50 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f864400c440 tx=0x7f86480bbee0).stop 2020-11-20T12:14:16.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.340+0000 7f864df94700 1 -- 172.21.15.187:0/4042614151 reap_dead start 2020-11-20T12:14:16.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.340+0000 7f864e795700 1 -- 172.21.15.187:0/4042614151 shutdown_connections 2020-11-20T12:14:16.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.340+0000 7f864e795700 1 -- 172.21.15.187:0/4042614151 >> 172.21.15.187:0/4042614151 conn(0x7f86480bce50 msgr2=0x7f86480bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:16.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.340+0000 7f864e795700 1 -- 172.21.15.187:0/4042614151 shutdown_connections 2020-11-20T12:14:16.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.340+0000 7f864e795700 1 -- 172.21.15.187:0/4042614151 wait complete. 2020-11-20T12:14:16.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.342+0000 7f864e795700 1 Processor -- start 2020-11-20T12:14:16.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.342+0000 7f864e795700 1 -- start start 2020-11-20T12:14:16.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864e795700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f864815b9d0 0x7f86480a2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864e795700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f864815bdb0 0x7f8648145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864e795700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86481460c0 0x7f8648148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864e795700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864809f9d0 con 0x7f86481460c0 2020-11-20T12:14:16.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864e795700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864809fe00 con 0x7f864815b9d0 2020-11-20T12:14:16.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864e795700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864809d7b0 con 0x7f864815bdb0 2020-11-20T12:14:16.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864d793700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f864815b9d0 0x7f86480a2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864d793700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f864815b9d0 0x7f86480a2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48666/0 (socket says 172.21.15.187:48666) 2020-11-20T12:14:16.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.343+0000 7f864d793700 1 -- 172.21.15.187:0/4266772021 learned_addr learned my addr 172.21.15.187:0/4266772021 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:16.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f864d793700 1 -- 172.21.15.187:0/4266772021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f864815bdb0 msgr2=0x7f8648145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f864d793700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f864815bdb0 0x7f8648145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:16.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f864d793700 1 -- 172.21.15.187:0/4266772021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86481460c0 msgr2=0x7f8648148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f864d793700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86481460c0 0x7f8648148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:16.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f864d793700 1 -- 172.21.15.187:0/4266772021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864809da50 con 0x7f864815b9d0 2020-11-20T12:14:16.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f864d793700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f864815b9d0 0x7f86480a2650 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f86300053e0 tx=0x7f8644015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:16.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f86357fa700 1 -- 172.21.15.187:0/4266772021 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644001780 con 0x7f864815b9d0 2020-11-20T12:14:16.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f86357fa700 1 -- 172.21.15.187:0/4266772021 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644005bb0 con 0x7f864815b9d0 2020-11-20T12:14:16.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f86357fa700 1 -- 172.21.15.187:0/4266772021 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86440148c0 con 0x7f864815b9d0 2020-11-20T12:14:16.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864809dcf0 con 0x7f864815b9d0 2020-11-20T12:14:16.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.344+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86480a2b40 con 0x7f864815b9d0 2020-11-20T12:14:16.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.345+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86480446d0 con 0x7f864815b9d0 2020-11-20T12:14:16.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.346+0000 7f86357fa700 1 -- 172.21.15.187:0/4266772021 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8644025030 con 0x7f864815b9d0 2020-11-20T12:14:16.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.346+0000 7f86357fa700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8620056d50 0x7f8620059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.347+0000 7f86357fa700 1 -- 172.21.15.187:0/4266772021 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8644070b20 con 0x7f864815b9d0 2020-11-20T12:14:16.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.347+0000 7f864cf92700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8620056d50 0x7f8620059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.347+0000 7f864cf92700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8620056d50 0x7f8620059210 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f864814a440 tx=0x7f8638006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:16.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.350+0000 7f86357fa700 1 -- 172.21.15.187:0/4266772021 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8644046020 con 0x7f864815b9d0 2020-11-20T12:14:16.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.534+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8648065830 con 0x7f864815b9d0 2020-11-20T12:14:16.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.535+0000 7f86357fa700 1 -- 172.21.15.187:0/4266772021 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v89) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8644023070 con 0x7f864815b9d0 2020-11-20T12:14:16.539 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:16.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8620056d50 msgr2=0x7f8620059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8620056d50 0x7f8620059210 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f864814a440 tx=0x7f8638006df0).stop 2020-11-20T12:14:16.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f864815b9d0 msgr2=0x7f86480a2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f864815b9d0 0x7f86480a2650 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f86300053e0 tx=0x7f8644015000).stop 2020-11-20T12:14:16.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 shutdown_connections 2020-11-20T12:14:16.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8620056d50 0x7f8620059210 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86481460c0 0x7f8648148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:16.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f864815b9d0 0x7f86480a2650 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 --2- 172.21.15.187:0/4266772021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f864815bdb0 0x7f8648145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:16.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.537+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 >> 172.21.15.187:0/4266772021 conn(0x7f86480bce50 msgr2=0x7f86480b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:16.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.538+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 shutdown_connections 2020-11-20T12:14:16.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.538+0000 7f864e795700 1 -- 172.21.15.187:0/4266772021 wait complete. 2020-11-20T12:14:16.565 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:14:16.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.787+0000 7f976ec6f700 1 Processor -- start 2020-11-20T12:14:16.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.787+0000 7f976ec6f700 1 -- start start 2020-11-20T12:14:16.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.789+0000 7f976ec6f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f97680bfbe0 0x7f97681480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.789+0000 7f976ec6f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976814d110 con 0x7f976809f560 2020-11-20T12:14:16.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.789+0000 7f976ec6f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976814cbd0 con 0x7f97680bf800 2020-11-20T12:14:16.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.789+0000 7f976ec6f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976814c9d0 con 0x7f97680bfbe0 2020-11-20T12:14:16.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.789+0000 7f976dc6d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f97680bf800 0x7f976814c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45280/0 (socket says 172.21.15.187:45280) 2020-11-20T12:14:16.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.789+0000 7f976dc6d700 1 -- 172.21.15.187:0/2056372688 learned_addr learned my addr 172.21.15.187:0/2056372688 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:16.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.790+0000 7f976d46c700 1 --2- 172.21.15.187:0/2056372688 >> v2:172.21.15.5:3300/0 conn(0x7f97680bfbe0 0x7f97681480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.790+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955217395 0 0) 0x7f976814cbd0 con 0x7f97680bf800 2020-11-20T12:14:16.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.790+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9750003240 con 0x7f97680bf800 2020-11-20T12:14:16.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.790+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 471324520 0 0) 0x7f9750003240 con 0x7f97680bf800 2020-11-20T12:14:16.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.790+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 >> v2:172.21.15.5:3300/0 conn(0x7f97680bfbe0 msgr2=0x7f97681480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:16.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.790+0000 7f976cc6b700 1 --2- 172.21.15.187:0/2056372688 >> v2:172.21.15.5:3300/0 conn(0x7f97680bfbe0 0x7f97681480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.790+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 >> v1:172.21.15.5:6789/0 conn(0x7f976809f560 legacy=0x7f97680a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.790+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976814ce70 con 0x7f97680bf800 2020-11-20T12:14:16.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f97640018a0 con 0x7f97680bf800 2020-11-20T12:14:16.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 >> v1:172.21.15.187:6789/0 conn(0x7f97680bf800 legacy=0x7f976814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976cc6b700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9750003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976cc6b700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97500044e0 0x7f9750004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976cc6b700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9750004e50 0x7f975000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976814c9d0 con 0x7f9750004e50 2020-11-20T12:14:16.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750004260 con 0x7f9750003730 2020-11-20T12:14:16.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975000bc20 con 0x7f97500044e0 2020-11-20T12:14:16.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976dc6d700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9750003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.791+0000 7f976e46e700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9750004e50 0x7f975000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976d46c700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97500044e0 0x7f9750004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976dc6d700 1 -- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97500044e0 msgr2=0x7f9750004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:16.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976dc6d700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97500044e0 0x7f9750004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976dc6d700 1 -- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9750004e50 msgr2=0x7f975000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:16.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976dc6d700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9750004e50 0x7f975000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976dc6d700 1 -- 172.21.15.187:0/2056372688 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750003240 con 0x7f9750003730 2020-11-20T12:14:16.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976e46e700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9750004e50 0x7f975000b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:16.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976e46e700 1 -- 172.21.15.187:0/2056372688 reap_dead start 2020-11-20T12:14:16.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976dc6d700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9750003b50 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f976400c840 tx=0x7f976814d000).ready entity=mon.1 client_cookie=be0cf32a90301b91 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:16.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764007b50 con 0x7f9750003730 2020-11-20T12:14:16.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9764005b80 con 0x7f9750003730 2020-11-20T12:14:16.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.792+0000 7f976cc6b700 1 -- 172.21.15.187:0/2056372688 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764007110 con 0x7f9750003730 2020-11-20T12:14:16.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.793+0000 7f976ec6f700 1 -- 172.21.15.187:0/2056372688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 msgr2=0x7f9750003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.793+0000 7f976ec6f700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9750003b50 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f976400c840 tx=0x7f976814d000).stop 2020-11-20T12:14:16.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.793+0000 7f976ec6f700 1 -- 172.21.15.187:0/2056372688 shutdown_connections 2020-11-20T12:14:16.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.793+0000 7f976ec6f700 1 --2- 172.21.15.187:0/2056372688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9750003b50 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.793+0000 7f976ec6f700 1 -- 172.21.15.187:0/2056372688 >> 172.21.15.187:0/2056372688 conn(0x7f9768068370 msgr2=0x7f97680bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:16.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.793+0000 7f976ec6f700 1 -- 172.21.15.187:0/2056372688 shutdown_connections 2020-11-20T12:14:16.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.793+0000 7f976ec6f700 1 -- 172.21.15.187:0/2056372688 wait complete. 2020-11-20T12:14:16.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.794+0000 7f976ec6f700 1 Processor -- start 2020-11-20T12:14:16.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.794+0000 7f976ec6f700 1 -- start start 2020-11-20T12:14:16.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.795+0000 7f976ec6f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9768133600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.795+0000 7f976ec6f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768149820 0x7f97681327a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.795+0000 7f976ec6f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976814ba00 0x7f9768132c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.795+0000 7f976ec6f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768144a20 con 0x7f9768149820 2020-11-20T12:14:16.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.795+0000 7f976ec6f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768133180 con 0x7f9750003730 2020-11-20T12:14:16.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.795+0000 7f976ec6f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97680b4b40 con 0x7f976814ba00 2020-11-20T12:14:16.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9768133600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9768133600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48686/0 (socket says 172.21.15.187:48686) 2020-11-20T12:14:16.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 -- 172.21.15.187:0/3472170242 learned_addr learned my addr 172.21.15.187:0/3472170242 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:16.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 -- 172.21.15.187:0/3472170242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976814ba00 msgr2=0x7f9768132c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976d46c700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768149820 0x7f97681327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976814ba00 0x7f9768132c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:16.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 -- 172.21.15.187:0/3472170242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768149820 msgr2=0x7f97681327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768149820 0x7f97681327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 -- 172.21.15.187:0/3472170242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97680b48a0 con 0x7f9750003730 2020-11-20T12:14:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.796+0000 7f976dc6d700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9768133600 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f97500043f0 tx=0x7f9764014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.797+0000 7f9755ffb700 1 -- 172.21.15.187:0/3472170242 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764004a90 con 0x7f9750003730 2020-11-20T12:14:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.797+0000 7f9755ffb700 1 -- 172.21.15.187:0/3472170242 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9764004bf0 con 0x7f9750003730 2020-11-20T12:14:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.797+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976814a3f0 con 0x7f9750003730 2020-11-20T12:14:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.797+0000 7f9755ffb700 1 -- 172.21.15.187:0/3472170242 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764005040 con 0x7f9750003730 2020-11-20T12:14:16.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.797+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976815bc00 con 0x7f9750003730 2020-11-20T12:14:16.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.798+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9768080050 con 0x7f9750003730 2020-11-20T12:14:16.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.798+0000 7f9755ffb700 1 -- 172.21.15.187:0/3472170242 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9764025030 con 0x7f9750003730 2020-11-20T12:14:16.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.799+0000 7f9755ffb700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:16.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.799+0000 7f9755ffb700 1 -- 172.21.15.187:0/3472170242 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9764071830 con 0x7f9750003730 2020-11-20T12:14:16.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.799+0000 7f976d46c700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:16.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.800+0000 7f976d46c700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f976814b7a0 tx=0x7f9758008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:16.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.802+0000 7f9755ffb700 1 -- 172.21.15.187:0/3472170242 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9764046d30 con 0x7f9750003730 2020-11-20T12:14:16.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.989+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f97680a4d60 con 0x7f9750003730 2020-11-20T12:14:16.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.991+0000 7f9755ffb700 1 -- 172.21.15.187:0/3472170242 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9764023020 con 0x7f9750003730 2020-11-20T12:14:16.995 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:16.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 msgr2=0x7f97400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f976814b7a0 tx=0x7f9758008040).stop 2020-11-20T12:14:16.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 msgr2=0x7f9768133600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:16.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9768133600 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f97500043f0 tx=0x7f9764014000).stop 2020-11-20T12:14:16.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 shutdown_connections 2020-11-20T12:14:16.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9740056d20 0x7f97400591e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f976814ba00 0x7f9768132c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:16.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9750003730 0x7f9768133600 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 --2- 172.21.15.187:0/3472170242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768149820 0x7f97681327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:16.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 >> 172.21.15.187:0/3472170242 conn(0x7f9768068370 msgr2=0x7f9768060c90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:16.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 shutdown_connections 2020-11-20T12:14:16.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:16.993+0000 7f976ec6f700 1 -- 172.21.15.187:0/3472170242 wait complete. 2020-11-20T12:14:17.018 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:14:17.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.262+0000 7f7dc0bbc700 1 Processor -- start 2020-11-20T12:14:17.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.263+0000 7f7dc0bbc700 1 -- start start 2020-11-20T12:14:17.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.264+0000 7f7dc0bbc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 0x7f7dbc066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.264+0000 7f7dc0bbc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7dbc1487e0 0x7f7dbc148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.264+0000 7f7dc0bbc700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc14c170 con 0x7f7dbc0683f0 2020-11-20T12:14:17.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.264+0000 7f7dc0bbc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc14c410 con 0x7f7dbc1487e0 2020-11-20T12:14:17.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.264+0000 7f7dc0bbc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc14c6b0 con 0x7f7dbc145cc0 2020-11-20T12:14:17.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.265+0000 7f7db2ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7dbc0683f0 0x7f7dbc065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45300/0 (socket says 172.21.15.187:45300) 2020-11-20T12:14:17.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.265+0000 7f7db2ffd700 1 -- 172.21.15.187:0/3705362743 learned_addr learned my addr 172.21.15.187:0/3705362743 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:17.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.265+0000 7f7dbbfff700 1 --2- 172.21.15.187:0/3705362743 >> v2:172.21.15.5:3300/0 conn(0x7f7dbc1487e0 0x7f7dbc148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.265+0000 7f7dbb7fe700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 0x7f7dbc066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.265+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005344454 0 0) 0x7f7dbc14c170 con 0x7f7dbc0683f0 2020-11-20T12:14:17.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.265+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9c003240 con 0x7f7dbc0683f0 2020-11-20T12:14:17.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3051418868 0 0) 0x7f7d9c003240 con 0x7f7dbc0683f0 2020-11-20T12:14:17.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 msgr2=0x7f7dbc066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:17.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 0x7f7dbc066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 >> v2:172.21.15.5:3300/0 conn(0x7f7dbc1487e0 msgr2=0x7f7dbc148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:17.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 --2- 172.21.15.187:0/3705362743 >> v2:172.21.15.5:3300/0 conn(0x7f7dbc1487e0 0x7f7dbc148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0658a0 con 0x7f7dbc0683f0 2020-11-20T12:14:17.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f7da40014e0 con 0x7f7dbc0683f0 2020-11-20T12:14:17.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 >> v1:172.21.15.187:6789/0 conn(0x7f7dbc0683f0 legacy=0x7f7dbc065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 0x7f7d9c0037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d9c003d10 0x7f7d9c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d9c0049d0 0x7f7d9c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc14c410 con 0x7f7d9c003d10 2020-11-20T12:14:17.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc14c6b0 con 0x7f7d9c0049d0 2020-11-20T12:14:17.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.266+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0042d0 con 0x7f7dbc145cc0 2020-11-20T12:14:17.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7dbbfff700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d9c0049d0 0x7f7d9c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7dbb7fe700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 0x7f7d9c0037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7db2ffd700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d9c003d10 0x7f7d9c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7dbbfff700 1 -- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 msgr2=0x7f7d9c0037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:17.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7dbbfff700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 0x7f7d9c0037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7dbbfff700 1 -- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d9c003d10 msgr2=0x7f7d9c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:17.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7dbbfff700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d9c003d10 0x7f7d9c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7dbbfff700 1 -- 172.21.15.187:0/3705362743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c003240 con 0x7f7d9c0049d0 2020-11-20T12:14:17.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7dbb7fe700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc145cc0 0x7f7d9c0037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:17.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.267+0000 7f7db2ffd700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d9c003d10 0x7f7d9c0044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:17.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.268+0000 7f7dbbfff700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d9c0049d0 0x7f7d9c004e50 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7f7db4009ea0 tx=0x7f7db40091d0).ready entity=mon.1 client_cookie=e8508ebb6056f522 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:17.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.268+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db40095c0 con 0x7f7d9c0049d0 2020-11-20T12:14:17.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.268+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db400eda0 con 0x7f7d9c0049d0 2020-11-20T12:14:17.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.268+0000 7f7dbaffd700 1 -- 172.21.15.187:0/3705362743 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db4012600 con 0x7f7d9c0049d0 2020-11-20T12:14:17.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.269+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/3705362743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d9c0049d0 msgr2=0x7f7d9c004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.269+0000 7f7dc0bbc700 1 --2- 172.21.15.187:0/3705362743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d9c0049d0 0x7f7d9c004e50 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7f7db4009ea0 tx=0x7f7db40091d0).stop 2020-11-20T12:14:17.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.269+0000 7f7dbbfff700 1 -- 172.21.15.187:0/3705362743 reap_dead start 2020-11-20T12:14:17.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.269+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/3705362743 shutdown_connections 2020-11-20T12:14:17.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.269+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/3705362743 >> 172.21.15.187:0/3705362743 conn(0x7f7dbc0bab30 msgr2=0x7f7dbc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:17.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.269+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/3705362743 shutdown_connections 2020-11-20T12:14:17.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.269+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/3705362743 wait complete. 2020-11-20T12:14:17.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.271+0000 7f7dc0bbc700 1 Processor -- start 2020-11-20T12:14:17.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.271+0000 7f7dc0bbc700 1 -- start start 2020-11-20T12:14:17.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dc0bbc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dbc0b61d0 0x7f7dbc09dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dc0bbc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc0b65b0 0x7f7dbc0b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dc0bbc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dbc09e1e0 0x7f7dbc15bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dc0bbc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc09e670 con 0x7f7dbc09e1e0 2020-11-20T12:14:17.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dc0bbc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc15b610 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dc0bbc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0a0880 con 0x7f7dbc0b65b0 2020-11-20T12:14:17.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dbb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dbc0b61d0 0x7f7dbc09dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dbb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dbc0b61d0 0x7f7dbc09dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48708/0 (socket says 172.21.15.187:48708) 2020-11-20T12:14:17.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.273+0000 7f7dbb7fe700 1 -- 172.21.15.187:0/2114734202 learned_addr learned my addr 172.21.15.187:0/2114734202 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:17.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7db2ffd700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc0b65b0 0x7f7dbc0b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7dbbfff700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dbc09e1e0 0x7f7dbc15bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7dbb7fe700 1 -- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc0b65b0 msgr2=0x7f7dbc0b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7dbb7fe700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc0b65b0 0x7f7dbc0b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7dbb7fe700 1 -- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dbc09e1e0 msgr2=0x7f7dbc15bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7dbb7fe700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dbc09e1e0 0x7f7dbc15bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7dbb7fe700 1 -- 172.21.15.187:0/2114734202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0b3560 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7db2ffd700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc0b65b0 0x7f7dbc0b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:17.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7dbb7fe700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dbc0b61d0 0x7f7dbc09dcf0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7f7dac010c20 tx=0x7f7dac008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:17.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7db3fff700 1 -- 172.21.15.187:0/2114734202 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac00ea30 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7db3fff700 1 -- 172.21.15.187:0/2114734202 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac00eb90 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7db3fff700 1 -- 172.21.15.187:0/2114734202 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac014470 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.274+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc0b3810 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.275+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc15c160 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.276+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc15c410 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.276+0000 7f7db3fff700 1 -- 172.21.15.187:0/2114734202 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7dac024070 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.277+0000 7f7db3fff700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d98056d50 0x7f7d98059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.277+0000 7f7db3fff700 1 -- 172.21.15.187:0/2114734202 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7dac0708f0 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.278+0000 7f7db2ffd700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d98056d50 0x7f7d98059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.278+0000 7f7db2ffd700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d98056d50 0x7f7d98059210 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f7dbc14b4f0 tx=0x7f7da4001530).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:17.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.281+0000 7f7db3fff700 1 -- 172.21.15.187:0/2114734202 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7dac045dc0 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.470+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7dbc061a20 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.471+0000 7f7db3fff700 1 -- 172.21.15.187:0/2114734202 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7dac0180a0 con 0x7f7dbc0b61d0 2020-11-20T12:14:17.475 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:17.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d98056d50 msgr2=0x7f7d98059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d98056d50 0x7f7d98059210 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f7dbc14b4f0 tx=0x7f7da4001530).stop 2020-11-20T12:14:17.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dbc0b61d0 msgr2=0x7f7dbc09dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dbc0b61d0 0x7f7dbc09dcf0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7f7dac010c20 tx=0x7f7dac008040).stop 2020-11-20T12:14:17.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 shutdown_connections 2020-11-20T12:14:17.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d98056d50 0x7f7d98059210 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dbc09e1e0 0x7f7dbc15bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dbc0b61d0 0x7f7dbc09dcf0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 --2- 172.21.15.187:0/2114734202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7dbc0b65b0 0x7f7dbc0b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.473+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 >> 172.21.15.187:0/2114734202 conn(0x7f7dbc0bab30 msgr2=0x7f7dbc0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:17.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.474+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 shutdown_connections 2020-11-20T12:14:17.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.474+0000 7f7dc0bbc700 1 -- 172.21.15.187:0/2114734202 wait complete. 2020-11-20T12:14:17.503 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:17.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.744+0000 7f2d0ffff700 1 Processor -- start 2020-11-20T12:14:17.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.744+0000 7f2d0ffff700 1 -- start start 2020-11-20T12:14:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.746+0000 7f2d0ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d10065570 0x7f2d1009d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.746+0000 7f2d0ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d100b13e0 con 0x7f2d10065f90 2020-11-20T12:14:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.746+0000 7f2d0ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1009d690 con 0x7f2d10065190 2020-11-20T12:14:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.746+0000 7f2d0ffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1014ab30 con 0x7f2d10065570 2020-11-20T12:14:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.746+0000 7f2d0effd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2d10065190 0x7f2d10143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45322/0 (socket says 172.21.15.187:45322) 2020-11-20T12:14:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.746+0000 7f2d0effd700 1 -- 172.21.15.187:0/146388397 learned_addr learned my addr 172.21.15.187:0/146388397 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0f7fe700 1 --2- 172.21.15.187:0/146388397 >> v2:172.21.15.187:3300/0 conn(0x7f2d10065570 0x7f2d1009d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334511091 0 0) 0x7f2d1009d690 con 0x7f2d10065190 2020-11-20T12:14:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf0003240 con 0x7f2d10065190 2020-11-20T12:14:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 147953129 0 0) 0x7f2cf0003240 con 0x7f2d10065190 2020-11-20T12:14:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 >> v2:172.21.15.187:3300/0 conn(0x7f2d10065570 msgr2=0x7f2d1009d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 --2- 172.21.15.187:0/146388397 >> v2:172.21.15.187:3300/0 conn(0x7f2d10065570 0x7f2d1009d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 >> v1:172.21.15.5:6789/0 conn(0x7f2d10065f90 legacy=0x7f2d1009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10062c10 con 0x7f2d10065190 2020-11-20T12:14:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f2d000014e0 con 0x7f2d10065190 2020-11-20T12:14:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 >> v1:172.21.15.187:6789/0 conn(0x7f2d10065190 legacy=0x7f2d10143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2cf0003730 0x7f2cf0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2cf0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2cf0004e50 0x7f2cf000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.747+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1014ab30 con 0x7f2cf0003730 2020-11-20T12:14:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf0004280 con 0x7f2cf00044e0 2020-11-20T12:14:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf000bc20 con 0x7f2cf0004e50 2020-11-20T12:14:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d067fc700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2cf0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d0effd700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2cf0003730 0x7f2cf0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d0f7fe700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2cf0004e50 0x7f2cf000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d067fc700 1 -- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2cf0004e50 msgr2=0x7f2cf000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d067fc700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2cf0004e50 0x7f2cf000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d067fc700 1 -- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2cf0003730 msgr2=0x7f2cf0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:17.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d067fc700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2cf0003730 0x7f2cf0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d067fc700 1 -- 172.21.15.187:0/146388397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf0003240 con 0x7f2cf00044e0 2020-11-20T12:14:17.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.748+0000 7f2d0effd700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2cf0003730 0x7f2cf0003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:17.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.749+0000 7f2d0f7fe700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2cf0004e50 0x7f2cf000b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:17.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.749+0000 7f2d0f7fe700 1 -- 172.21.15.187:0/146388397 reap_dead start 2020-11-20T12:14:17.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.749+0000 7f2d067fc700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2cf0004960 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7f2cf80018f0 tx=0x7f2cf8007b90).ready entity=mon.1 client_cookie=46910c8a1f598cdf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:17.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.749+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8007020 con 0x7f2cf00044e0 2020-11-20T12:14:17.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.749+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf8007180 con 0x7f2cf00044e0 2020-11-20T12:14:17.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.749+0000 7f2d0e7fc700 1 -- 172.21.15.187:0/146388397 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf800c830 con 0x7f2cf00044e0 2020-11-20T12:14:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.749+0000 7f2d0ffff700 1 -- 172.21.15.187:0/146388397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 msgr2=0x7f2cf0004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.749+0000 7f2d0ffff700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2cf0004960 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7f2cf80018f0 tx=0x7f2cf8007b90).stop 2020-11-20T12:14:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.750+0000 7f2d0ffff700 1 -- 172.21.15.187:0/146388397 shutdown_connections 2020-11-20T12:14:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.750+0000 7f2d0ffff700 1 --2- 172.21.15.187:0/146388397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2cf0004960 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.750+0000 7f2d0ffff700 1 -- 172.21.15.187:0/146388397 >> 172.21.15.187:0/146388397 conn(0x7f2d100bab30 msgr2=0x7f2d100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:17.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.750+0000 7f2d0ffff700 1 -- 172.21.15.187:0/146388397 shutdown_connections 2020-11-20T12:14:17.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.750+0000 7f2d0ffff700 1 -- 172.21.15.187:0/146388397 wait complete. 2020-11-20T12:14:17.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.752+0000 7f2d0ffff700 1 Processor -- start 2020-11-20T12:14:17.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.752+0000 7f2d0ffff700 1 -- start start 2020-11-20T12:14:17.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2d10158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d101488a0 0x7f2d10147820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d101490f0 0x7f2d1009f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100a0a00 con 0x7f2d101490f0 2020-11-20T12:14:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10147ec0 con 0x7f2cf00044e0 2020-11-20T12:14:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10148160 con 0x7f2d101488a0 2020-11-20T12:14:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2d10158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2d10158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48726/0 (socket says 172.21.15.187:48726) 2020-11-20T12:14:17.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.753+0000 7f2d0effd700 1 -- 172.21.15.187:0/2003412825 learned_addr learned my addr 172.21.15.187:0/2003412825 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d067fc700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d101488a0 0x7f2d10147820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d0effd700 1 -- 172.21.15.187:0/2003412825 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d101488a0 msgr2=0x7f2d10147820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d0effd700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d101488a0 0x7f2d10147820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d0effd700 1 -- 172.21.15.187:0/2003412825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d101490f0 msgr2=0x7f2d1009f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:17.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d0effd700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d101490f0 0x7f2d1009f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d0effd700 1 -- 172.21.15.187:0/2003412825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10148400 con 0x7f2cf00044e0 2020-11-20T12:14:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d067fc700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d101488a0 0x7f2d10147820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d0effd700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2d10158340 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f2d0000dad0 tx=0x7f2d00009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d077fe700 1 -- 172.21.15.187:0/2003412825 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00007170 con 0x7f2cf00044e0 2020-11-20T12:14:17.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d077fe700 1 -- 172.21.15.187:0/2003412825 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d000072d0 con 0x7f2cf00044e0 2020-11-20T12:14:17.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d077fe700 1 -- 172.21.15.187:0/2003412825 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000bb40 con 0x7f2cf00044e0 2020-11-20T12:14:17.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.754+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d10149530 con 0x7f2cf00044e0 2020-11-20T12:14:17.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.755+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1009fb90 con 0x7f2cf00044e0 2020-11-20T12:14:17.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.756+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d10147d10 con 0x7f2cf00044e0 2020-11-20T12:14:17.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.757+0000 7f2d077fe700 1 -- 172.21.15.187:0/2003412825 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2d00004500 con 0x7f2cf00044e0 2020-11-20T12:14:17.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.757+0000 7f2d077fe700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d50 0x7f2cec059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:17.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.757+0000 7f2d077fe700 1 -- 172.21.15.187:0/2003412825 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2d00070240 con 0x7f2cf00044e0 2020-11-20T12:14:17.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.758+0000 7f2d067fc700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d50 0x7f2cec059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:17.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.758+0000 7f2d067fc700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d50 0x7f2cec059210 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f2d100b4a00 tx=0x7f2cf8016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:17.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:17.762+0000 7f2d077fe700 1 -- 172.21.15.187:0/2003412825 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2d00045630 con 0x7f2cf00044e0 2020-11-20T12:14:18.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.010+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d101496c0 con 0x7f2cf00044e0 2020-11-20T12:14:18.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.011+0000 7f2d077fe700 1 -- 172.21.15.187:0/2003412825 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f2d00048c50 con 0x7f2cf00044e0 2020-11-20T12:14:18.015 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:18.016 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1713004164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1713004164},{"type":"v1","addr":"172.21.15.5:6833","nonce":1713004164}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:13:40.253243+0000","modified":"2020-11-20T12:13:42.397341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7408":{"gid":7408,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1880661915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1880661915},{"type":"v1","addr":"172.21.15.187:6835","nonce":1880661915}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2020-11-20T12:14:18.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.013+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d50 msgr2=0x7f2cec059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.013+0000 7f2d0ffff700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d50 0x7f2cec059210 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f2d100b4a00 tx=0x7f2cf8016000).stop 2020-11-20T12:14:18.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.013+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 msgr2=0x7f2d10158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.013+0000 7f2d0ffff700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2d10158340 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f2d0000dad0 tx=0x7f2d00009040).stop 2020-11-20T12:14:18.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.014+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 shutdown_connections 2020-11-20T12:14:18.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.014+0000 7f2d0ffff700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d50 0x7f2cec059210 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.014+0000 7f2d0ffff700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d101490f0 0x7f2d1009f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:18.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.014+0000 7f2d0ffff700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cf00044e0 0x7f2d10158340 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.014+0000 7f2d0ffff700 1 --2- 172.21.15.187:0/2003412825 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d101488a0 0x7f2d10147820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.014+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 >> 172.21.15.187:0/2003412825 conn(0x7f2d100bab30 msgr2=0x7f2d1009ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:18.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.014+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 shutdown_connections 2020-11-20T12:14:18.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.014+0000 7f2d0ffff700 1 -- 172.21.15.187:0/2003412825 wait complete. 2020-11-20T12:14:18.022 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 80 2020-11-20T12:14:18.040 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:18.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.278+0000 7f81bce8d700 1 Processor -- start 2020-11-20T12:14:18.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.278+0000 7f81bce8d700 1 -- start start 2020-11-20T12:14:18.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.279+0000 7f81bce8d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f81b80b6c10 0x7f81b80b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.279+0000 7f81bce8d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b80b08e0 con 0x7f81b80654a0 2020-11-20T12:14:18.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.279+0000 7f81bce8d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b80b0b80 con 0x7f81b8065880 2020-11-20T12:14:18.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.279+0000 7f81bce8d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b80b05f0 con 0x7f81b80b6c10 2020-11-20T12:14:18.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.280+0000 7f81b5d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f81b8065880 0x7f81b80666e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45342/0 (socket says 172.21.15.187:45342) 2020-11-20T12:14:18.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.280+0000 7f81b5d9b700 1 -- 172.21.15.187:0/3045602846 learned_addr learned my addr 172.21.15.187:0/3045602846 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:18.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.280+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608670801 0 0) 0x7f81b80b0b80 con 0x7f81b8065880 2020-11-20T12:14:18.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.280+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f819c003240 con 0x7f81b8065880 2020-11-20T12:14:18.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.280+0000 7f81b6d9d700 1 --2- 172.21.15.187:0/3045602846 >> v2:172.21.15.5:3300/0 conn(0x7f81b80b6c10 0x7f81b80b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1810332444 0 0) 0x7f819c003240 con 0x7f81b8065880 2020-11-20T12:14:18.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 >> v2:172.21.15.5:3300/0 conn(0x7f81b80b6c10 msgr2=0x7f81b80b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:18.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 --2- 172.21.15.187:0/3045602846 >> v2:172.21.15.5:3300/0 conn(0x7f81b80b6c10 0x7f81b80b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 >> v1:172.21.15.5:6789/0 conn(0x7f81b80654a0 legacy=0x7f81b8144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b8145570 con 0x7f81b8065880 2020-11-20T12:14:18.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f81ac001660 con 0x7f81b8065880 2020-11-20T12:14:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 >> v1:172.21.15.187:6789/0 conn(0x7f81b8065880 legacy=0x7f81b80666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c003730 0x7f819c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.281+0000 7f81b559a700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f819c0044e0 0x7f819c004940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b559a700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819c004e30 0x7f819c00d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80b05f0 con 0x7f819c003730 2020-11-20T12:14:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c0041f0 con 0x7f819c0044e0 2020-11-20T12:14:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819c00de30 con 0x7f819c004e30 2020-11-20T12:14:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b5d9b700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f819c0044e0 0x7f819c004940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b659c700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c003730 0x7f819c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b5d9b700 1 -- 172.21.15.187:0/3045602846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819c004e30 msgr2=0x7f819c00d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b5d9b700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819c004e30 0x7f819c00d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b5d9b700 1 -- 172.21.15.187:0/3045602846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c003730 msgr2=0x7f819c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.282+0000 7f81b5d9b700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c003730 0x7f819c003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.283+0000 7f81b5d9b700 1 -- 172.21.15.187:0/3045602846 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c003240 con 0x7f819c0044e0 2020-11-20T12:14:18.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.283+0000 7f81b6d9d700 1 -- 172.21.15.187:0/3045602846 reap_dead start 2020-11-20T12:14:18.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.283+0000 7f81b5d9b700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f819c0044e0 0x7f819c004940 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f81ac00c600 tx=0x7f81ac007b10).ready entity=mon.1 client_cookie=bf5a29da737dba21 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:18.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.283+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac007b80 con 0x7f819c0044e0 2020-11-20T12:14:18.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.283+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac00e560 con 0x7f819c0044e0 2020-11-20T12:14:18.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.283+0000 7f81b559a700 1 -- 172.21.15.187:0/3045602846 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac00e920 con 0x7f819c0044e0 2020-11-20T12:14:18.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.284+0000 7f81bce8d700 1 -- 172.21.15.187:0/3045602846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f819c0044e0 msgr2=0x7f819c004940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.284+0000 7f81bce8d700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f819c0044e0 0x7f819c004940 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f81ac00c600 tx=0x7f81ac007b10).stop 2020-11-20T12:14:18.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.284+0000 7f81bce8d700 1 -- 172.21.15.187:0/3045602846 shutdown_connections 2020-11-20T12:14:18.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.284+0000 7f81bce8d700 1 --2- 172.21.15.187:0/3045602846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f819c0044e0 0x7f819c004940 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.284+0000 7f81bce8d700 1 -- 172.21.15.187:0/3045602846 >> 172.21.15.187:0/3045602846 conn(0x7f81b80bacd0 msgr2=0x7f81b80bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:18.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.284+0000 7f81bce8d700 1 -- 172.21.15.187:0/3045602846 shutdown_connections 2020-11-20T12:14:18.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.284+0000 7f81bce8d700 1 -- 172.21.15.187:0/3045602846 wait complete. 2020-11-20T12:14:18.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.286+0000 7f81bce8d700 1 Processor -- start 2020-11-20T12:14:18.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.286+0000 7f81bce8d700 1 -- start start 2020-11-20T12:14:18.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81bce8d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c0044e0 0x7f81b809ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81bce8d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81b80b1c10 0x7f81b8155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81bce8d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b81564c0 0x7f81b815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81bce8d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80b41d0 con 0x7f819c0044e0 2020-11-20T12:14:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81bce8d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b809c0b0 con 0x7f81b80b1c10 2020-11-20T12:14:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81bce8d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b809c670 con 0x7f81b81564c0 2020-11-20T12:14:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b5d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81b80b1c10 0x7f81b8155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b5d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81b80b1c10 0x7f81b8155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48748/0 (socket says 172.21.15.187:48748) 2020-11-20T12:14:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b5d9b700 1 -- 172.21.15.187:0/946402295 learned_addr learned my addr 172.21.15.187:0/946402295 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b659c700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c0044e0 0x7f81b809ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b5d9b700 1 -- 172.21.15.187:0/946402295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b81564c0 msgr2=0x7f81b815fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b5d9b700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b81564c0 0x7f81b815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:18.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b5d9b700 1 -- 172.21.15.187:0/946402295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c0044e0 msgr2=0x7f81b809ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b5d9b700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c0044e0 0x7f81b809ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.287+0000 7f81b5d9b700 1 -- 172.21.15.187:0/946402295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b80b2090 con 0x7f81b80b1c10 2020-11-20T12:14:18.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.288+0000 7f81b5d9b700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81b80b1c10 0x7f81b8155fd0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f819c004380 tx=0x7f81ac017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.288+0000 7f81a67fc700 1 -- 172.21.15.187:0/946402295 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac00eea0 con 0x7f81b80b1c10 2020-11-20T12:14:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.288+0000 7f81a67fc700 1 -- 172.21.15.187:0/946402295 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac007030 con 0x7f81b80b1c10 2020-11-20T12:14:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.288+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b809c350 con 0x7f81b80b1c10 2020-11-20T12:14:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.288+0000 7f81a67fc700 1 -- 172.21.15.187:0/946402295 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac0073b0 con 0x7f81b80b1c10 2020-11-20T12:14:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.288+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b81601e0 con 0x7f81b80b1c10 2020-11-20T12:14:18.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.289+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b809c4e0 con 0x7f81b80b1c10 2020-11-20T12:14:18.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.290+0000 7f81a67fc700 1 -- 172.21.15.187:0/946402295 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f81ac010070 con 0x7f81b80b1c10 2020-11-20T12:14:18.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.290+0000 7f81a67fc700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f818c056d20 0x7f818c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.290+0000 7f81a67fc700 1 -- 172.21.15.187:0/946402295 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f81ac071460 con 0x7f81b80b1c10 2020-11-20T12:14:18.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.291+0000 7f81b659c700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f818c056d20 0x7f818c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.291+0000 7f81b659c700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f818c056d20 0x7f818c0591e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f81b805f730 tx=0x7f81a8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:18.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.294+0000 7f81a67fc700 1 -- 172.21.15.187:0/946402295 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f81ac0468b0 con 0x7f81b80b1c10 2020-11-20T12:14:18.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.478+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81b8063600 con 0x7f81b80b1c10 2020-11-20T12:14:18.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.480+0000 7f81a67fc700 1 -- 172.21.15.187:0/946402295 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f81ac049ed0 con 0x7f81b80b1c10 2020-11-20T12:14:18.485 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:18.485 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":89,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:13:41.257868+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:13:38.491662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2020-11-20T12:13:39.231237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:18.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f818c056d20 msgr2=0x7f818c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f818c056d20 0x7f818c0591e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f81b805f730 tx=0x7f81a8009040).stop 2020-11-20T12:14:18.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81b80b1c10 msgr2=0x7f81b8155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81b80b1c10 0x7f81b8155fd0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f819c004380 tx=0x7f81ac017000).stop 2020-11-20T12:14:18.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 shutdown_connections 2020-11-20T12:14:18.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f818c056d20 0x7f818c0591e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f819c0044e0 0x7f81b809ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b81564c0 0x7f81b815fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:18.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 --2- 172.21.15.187:0/946402295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81b80b1c10 0x7f81b8155fd0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 >> 172.21.15.187:0/946402295 conn(0x7f81b80bacd0 msgr2=0x7f81b8144c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:18.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 shutdown_connections 2020-11-20T12:14:18.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.483+0000 7f81bce8d700 1 -- 172.21.15.187:0/946402295 wait complete. 2020-11-20T12:14:18.512 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:14:18.512 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:14:18.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.747+0000 7f3572bb2700 1 Processor -- start 2020-11-20T12:14:18.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.747+0000 7f3572bb2700 1 -- start start 2020-11-20T12:14:18.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.748+0000 7f3572bb2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f356c066f70 0x7f356c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.748+0000 7f3572bb2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0b1300 con 0x7f356c1484d0 2020-11-20T12:14:18.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.748+0000 7f3572bb2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0b15a0 con 0x7f356c0b6650 2020-11-20T12:14:18.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.749+0000 7f3572bb2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c067730 con 0x7f356c066f70 2020-11-20T12:14:18.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.749+0000 7f35723b1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f356c066f70 0x7f356c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.749+0000 7f35713af700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f356c0b6650 0x7f356c0afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45362/0 (socket says 172.21.15.187:45362) 2020-11-20T12:14:18.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.749+0000 7f35713af700 1 -- 172.21.15.187:0/2590322083 learned_addr learned my addr 172.21.15.187:0/2590322083 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:18.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.749+0000 7f35723b1700 1 -- 172.21.15.187:0/2590322083 >> v1:172.21.15.187:6789/0 conn(0x7f356c0b6650 legacy=0x7f356c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.749+0000 7f35723b1700 1 -- 172.21.15.187:0/2590322083 >> v1:172.21.15.5:6789/0 conn(0x7f356c1484d0 legacy=0x7f356c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:18.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.749+0000 7f35723b1700 1 -- 172.21.15.187:0/2590322083 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0b1840 con 0x7f356c066f70 2020-11-20T12:14:18.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.750+0000 7f35723b1700 1 --2- 172.21.15.187:0/2590322083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b01a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f3564009c90 tx=0x7f35640091d0).ready entity=mon.1 client_cookie=7a309d29b27bb1b9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:18.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.750+0000 7f3570bae700 1 -- 172.21.15.187:0/2590322083 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356400bb80 con 0x7f356c066f70 2020-11-20T12:14:18.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.750+0000 7f3570bae700 1 -- 172.21.15.187:0/2590322083 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356400bce0 con 0x7f356c066f70 2020-11-20T12:14:18.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.750+0000 7f3570bae700 1 -- 172.21.15.187:0/2590322083 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564011610 con 0x7f356c066f70 2020-11-20T12:14:18.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.751+0000 7f3572bb2700 1 -- 172.21.15.187:0/2590322083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 msgr2=0x7f356c0b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.751+0000 7f3572bb2700 1 --2- 172.21.15.187:0/2590322083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b01a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f3564009c90 tx=0x7f35640091d0).stop 2020-11-20T12:14:18.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.751+0000 7f3572bb2700 1 -- 172.21.15.187:0/2590322083 shutdown_connections 2020-11-20T12:14:18.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.751+0000 7f3572bb2700 1 --2- 172.21.15.187:0/2590322083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b01a0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:18.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.751+0000 7f3572bb2700 1 -- 172.21.15.187:0/2590322083 >> 172.21.15.187:0/2590322083 conn(0x7f356c0bab30 msgr2=0x7f356c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:18.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.751+0000 7f3572bb2700 1 -- 172.21.15.187:0/2590322083 shutdown_connections 2020-11-20T12:14:18.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.751+0000 7f3572bb2700 1 -- 172.21.15.187:0/2590322083 wait complete. 2020-11-20T12:14:18.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.752+0000 7f3572bb2700 1 Processor -- start 2020-11-20T12:14:18.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.753+0000 7f3572bb2700 1 -- start start 2020-11-20T12:14:18.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.753+0000 7f3572bb2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.753+0000 7f3572bb2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f356c0b6650 0x7f356c09cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.753+0000 7f3572bb2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f356c1484d0 0x7f356c09af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.753+0000 7f3572bb2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c09ef70 con 0x7f356c1484d0 2020-11-20T12:14:18.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.753+0000 7f3572bb2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c09f180 con 0x7f356c066f70 2020-11-20T12:14:18.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.753+0000 7f3572bb2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c09f420 con 0x7f356c0b6650 2020-11-20T12:14:18.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48760/0 (socket says 172.21.15.187:48760) 2020-11-20T12:14:18.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 -- 172.21.15.187:0/1723282737 learned_addr learned my addr 172.21.15.187:0/1723282737 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:18.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 -- 172.21.15.187:0/1723282737 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f356c0b6650 msgr2=0x7f356c09cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f356c0b6650 0x7f356c09cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:18.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 -- 172.21.15.187:0/1723282737 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f356c1484d0 msgr2=0x7f356c09af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:18.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f356c1484d0 0x7f356c09af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:18.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 -- 172.21.15.187:0/1723282737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c09d160 con 0x7f356c066f70 2020-11-20T12:14:18.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3571bb0700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b2cd0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f35680096e0 tx=0x7f3568006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:18.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.754+0000 7f3559ffb700 1 -- 172.21.15.187:0/1723282737 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356800d560 con 0x7f356c066f70 2020-11-20T12:14:18.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.755+0000 7f3559ffb700 1 -- 172.21.15.187:0/1723282737 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3568000e90 con 0x7f356c066f70 2020-11-20T12:14:18.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.755+0000 7f3559ffb700 1 -- 172.21.15.187:0/1723282737 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356800d930 con 0x7f356c066f70 2020-11-20T12:14:18.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.755+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c09eab0 con 0x7f356c066f70 2020-11-20T12:14:18.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.755+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c0b3ec0 con 0x7f356c066f70 2020-11-20T12:14:18.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.756+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c061a20 con 0x7f356c066f70 2020-11-20T12:14:18.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.756+0000 7f3559ffb700 1 -- 172.21.15.187:0/1723282737 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f356800f030 con 0x7f356c066f70 2020-11-20T12:14:18.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.757+0000 7f3559ffb700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3544056d20 0x7f35440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:18.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.757+0000 7f3559ffb700 1 -- 172.21.15.187:0/1723282737 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f356806fb50 con 0x7f356c066f70 2020-11-20T12:14:18.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.757+0000 7f35713af700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3544056d20 0x7f35440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:18.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.758+0000 7f35713af700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3544056d20 0x7f35440591e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f356c0b27d0 tx=0x7f355c007800).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:18.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:18.761+0000 7f3559ffb700 1 -- 172.21.15.187:0/1723282737 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3568044e90 con 0x7f356c066f70 2020-11-20T12:14:19.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.010+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f356c0b3ec0 con 0x7f356c066f70 2020-11-20T12:14:19.847 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:14:19.843+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:14:19.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.858+0000 7f3559ffb700 1 -- 172.21.15.187:0/1723282737 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v81) v1 ==== 161+0+0 (secure 0 0 0) 0x7f356c0b3ec0 con 0x7f356c066f70 2020-11-20T12:14:19.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3544056d20 msgr2=0x7f35440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:19.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3544056d20 0x7f35440591e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f356c0b27d0 tx=0x7f355c007800).stop 2020-11-20T12:14:19.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 msgr2=0x7f356c0b2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:19.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b2cd0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f35680096e0 tx=0x7f3568006d70).stop 2020-11-20T12:14:19.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 shutdown_connections 2020-11-20T12:14:19.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3544056d20 0x7f35440591e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:19.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f356c1484d0 0x7f356c09af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:19.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f356c066f70 0x7f356c0b2cd0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:19.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 --2- 172.21.15.187:0/1723282737 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f356c0b6650 0x7f356c09cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:19.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.862+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 >> 172.21.15.187:0/1723282737 conn(0x7f356c0bab30 msgr2=0x7f356c142c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:19.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.863+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 shutdown_connections 2020-11-20T12:14:19.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:19.863+0000 7f3572bb2700 1 -- 172.21.15.187:0/1723282737 wait complete. 2020-11-20T12:14:19.871 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:14:19.879 INFO:tasks.ceph.mds.a.smithi187.stderr: -3864> 2020-11-20T12:13:44.401+0000 7f912e67b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:14:19.896 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:14:19.945 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:14:20.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.138+0000 7ffad7335700 1 Processor -- start 2020-11-20T12:14:20.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.138+0000 7ffad7335700 1 -- start start 2020-11-20T12:14:20.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad7335700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffad0065190 0x7ffad0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:20.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad7335700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 0x7ffad0143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:20.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad7335700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffad0143c70 0x7ffad009ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:20.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad7335700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad014ab30 con 0x7ffad0065190 2020-11-20T12:14:20.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad7335700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00b13e0 con 0x7ffad0143c70 2020-11-20T12:14:20.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad7335700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad0062c10 con 0x7ffad0065f90 2020-11-20T12:14:20.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad6b34700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffad0143c70 0x7ffad009ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:20.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad6b34700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffad0143c70 0x7ffad009ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48784/0 (socket says 172.21.15.187:48784) 2020-11-20T12:14:20.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.140+0000 7ffad6b34700 1 -- 172.21.15.187:0/3951918480 learned_addr learned my addr 172.21.15.187:0/3951918480 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:20.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.141+0000 7ffad6333700 1 --2- 172.21.15.187:0/3951918480 >> v2:172.21.15.5:3300/0 conn(0x7ffad0065190 0x7ffad0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:20.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.141+0000 7ffad5b32700 1 --2- 172.21.15.187:0/3951918480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 0x7ffad0143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:20.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.141+0000 7ffad6b34700 1 -- 172.21.15.187:0/3951918480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 msgr2=0x7ffad0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:20.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.141+0000 7ffad6b34700 1 --2- 172.21.15.187:0/3951918480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 0x7ffad0143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:20.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.141+0000 7ffad6b34700 1 -- 172.21.15.187:0/3951918480 >> v2:172.21.15.5:3300/0 conn(0x7ffad0065190 msgr2=0x7ffad0065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:20.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.141+0000 7ffad6b34700 1 --2- 172.21.15.187:0/3951918480 >> v2:172.21.15.5:3300/0 conn(0x7ffad0065190 0x7ffad0065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:20.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.141+0000 7ffad6b34700 1 -- 172.21.15.187:0/3951918480 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00b1640 con 0x7ffad0143c70 2020-11-20T12:14:20.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.141+0000 7ffad6333700 1 --2- 172.21.15.187:0/3951918480 >> v2:172.21.15.5:3300/0 conn(0x7ffad0065190 0x7ffad0065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:20.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.142+0000 7ffad6b34700 1 --2- 172.21.15.187:0/3951918480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009ca90 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7ffad00b6390 tx=0x7ffac8009290).ready entity=mon.1 client_cookie=9245614248d5864e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:20.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.142+0000 7ffad5331700 1 -- 172.21.15.187:0/3951918480 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac800bc50 con 0x7ffad0143c70 2020-11-20T12:14:20.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.142+0000 7ffad5331700 1 -- 172.21.15.187:0/3951918480 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac800bdb0 con 0x7ffad0143c70 2020-11-20T12:14:20.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.142+0000 7ffad5331700 1 -- 172.21.15.187:0/3951918480 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8011610 con 0x7ffad0143c70 2020-11-20T12:14:20.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 -- 172.21.15.187:0/3951918480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 msgr2=0x7ffad009ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:20.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 --2- 172.21.15.187:0/3951918480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009ca90 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7ffad00b6390 tx=0x7ffac8009290).stop 2020-11-20T12:14:20.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 -- 172.21.15.187:0/3951918480 shutdown_connections 2020-11-20T12:14:20.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 --2- 172.21.15.187:0/3951918480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009ca90 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:20.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 --2- 172.21.15.187:0/3951918480 >> v2:172.21.15.5:3300/0 conn(0x7ffad0065190 0x7ffad0065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:20.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 --2- 172.21.15.187:0/3951918480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 0x7ffad0143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:20.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 -- 172.21.15.187:0/3951918480 >> 172.21.15.187:0/3951918480 conn(0x7ffad00bab30 msgr2=0x7ffad00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:20.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 -- 172.21.15.187:0/3951918480 shutdown_connections 2020-11-20T12:14:20.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.143+0000 7ffad7335700 1 -- 172.21.15.187:0/3951918480 wait complete. 2020-11-20T12:14:20.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.145+0000 7ffad7335700 1 Processor -- start 2020-11-20T12:14:20.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.145+0000 7ffad7335700 1 -- start start 2020-11-20T12:14:20.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad7335700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffad0065190 0x7ffad009dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:20.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad7335700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 0x7ffad0068210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:20.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad7335700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:20.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad7335700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad014a150 con 0x7ffad0065190 2020-11-20T12:14:20.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad7335700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad0067cf0 con 0x7ffad0143c70 2020-11-20T12:14:20.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad7335700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad0067f90 con 0x7ffad0065f90 2020-11-20T12:14:20.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad6b34700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:20.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad6b34700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48790/0 (socket says 172.21.15.187:48790) 2020-11-20T12:14:20.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad6b34700 1 -- 172.21.15.187:0/74788575 learned_addr learned my addr 172.21.15.187:0/74788575 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:20.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.146+0000 7ffad5b32700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 0x7ffad0068210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:20.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad6333700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffad0065190 0x7ffad009dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:20.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad6b34700 1 -- 172.21.15.187:0/74788575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 msgr2=0x7ffad0068210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:20.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad6b34700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 0x7ffad0068210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:20.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad6b34700 1 -- 172.21.15.187:0/74788575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffad0065190 msgr2=0x7ffad009dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:20.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad6b34700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffad0065190 0x7ffad009dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:20.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad6b34700 1 -- 172.21.15.187:0/74788575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00b43d0 con 0x7ffad0143c70 2020-11-20T12:14:20.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad6b34700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009e4d0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7ffad014ac70 tx=0x7ffac8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:20.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffabe7fc700 1 -- 172.21.15.187:0/74788575 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8003290 con 0x7ffad0143c70 2020-11-20T12:14:20.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffabe7fc700 1 -- 172.21.15.187:0/74788575 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac8011650 con 0x7ffad0143c70 2020-11-20T12:14:20.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad009f6d0 con 0x7ffad0143c70 2020-11-20T12:14:20.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffabe7fc700 1 -- 172.21.15.187:0/74788575 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8010e10 con 0x7ffad0143c70 2020-11-20T12:14:20.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.147+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad009ed30 con 0x7ffad0143c70 2020-11-20T12:14:20.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.148+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad00b4560 con 0x7ffad0143c70 2020-11-20T12:14:20.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.149+0000 7ffabe7fc700 1 -- 172.21.15.187:0/74788575 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffac8017070 con 0x7ffad0143c70 2020-11-20T12:14:20.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.150+0000 7ffabe7fc700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffaa8056d20 0x7ffaa80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:20.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.150+0000 7ffabe7fc700 1 -- 172.21.15.187:0/74788575 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ffac8072150 con 0x7ffad0143c70 2020-11-20T12:14:20.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.150+0000 7ffad6333700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffaa8056d20 0x7ffaa80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:20.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.151+0000 7ffad6333700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffaa8056d20 0x7ffaa80591e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7ffacc006bd0 tx=0x7ffacc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:20.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.154+0000 7ffabe7fc700 1 -- 172.21.15.187:0/74788575 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffac8047570 con 0x7ffad0143c70 2020-11-20T12:14:20.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:20.387+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffad0061a20 con 0x7ffad0143c70 2020-11-20T12:14:21.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.263+0000 7ffabe7fc700 1 -- 172.21.15.187:0/74788575 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v82) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffac8015020 con 0x7ffad0143c70 2020-11-20T12:14:21.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffaa8056d20 msgr2=0x7ffaa80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:21.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffaa8056d20 0x7ffaa80591e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7ffacc006bd0 tx=0x7ffacc008040).stop 2020-11-20T12:14:21.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 msgr2=0x7ffad009e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:21.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009e4d0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7ffad014ac70 tx=0x7ffac8016000).stop 2020-11-20T12:14:21.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 shutdown_connections 2020-11-20T12:14:21.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffaa8056d20 0x7ffaa80591e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffad0065190 0x7ffad009dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffad0143c70 0x7ffad009e4d0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 --2- 172.21.15.187:0/74788575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffad0065f90 0x7ffad0068210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.267+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 >> 172.21.15.187:0/74788575 conn(0x7ffad00bab30 msgr2=0x7ffad0142da0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:21.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.268+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 shutdown_connections 2020-11-20T12:14:21.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.268+0000 7ffad7335700 1 -- 172.21.15.187:0/74788575 wait complete. 2020-11-20T12:14:21.299 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:14:21.541 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:14:21.551 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:14:21.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.596+0000 7f4f65c71700 1 Processor -- start 2020-11-20T12:14:21.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.597+0000 7f4f65c71700 1 -- start start 2020-11-20T12:14:21.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.598+0000 7f4f65c71700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4f600bf800 0x7f4f6014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.598+0000 7f4f65c71700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600bfbe0 0x7f4f601480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.598+0000 7f4f65c71700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6014d110 con 0x7f4f6009f560 2020-11-20T12:14:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.598+0000 7f4f65c71700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6014cbd0 con 0x7f4f600bf800 2020-11-20T12:14:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.598+0000 7f4f65c71700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6014ce70 con 0x7f4f600bfbe0 2020-11-20T12:14:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.598+0000 7f4f64c6f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4f600bf800 0x7f4f6014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:21.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.599+0000 7f4f64c6f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4f600bf800 0x7f4f6014c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48794/0 (socket says 172.21.15.187:48794) 2020-11-20T12:14:21.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.599+0000 7f4f64c6f700 1 -- 172.21.15.187:0/1831918447 learned_addr learned my addr 172.21.15.187:0/1831918447 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:21.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.599+0000 7f4f5ffff700 1 --2- 172.21.15.187:0/1831918447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600bfbe0 0x7f4f601480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:21.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.599+0000 7f4f64c6f700 1 -- 172.21.15.187:0/1831918447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600bfbe0 msgr2=0x7f4f601480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:21.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.599+0000 7f4f64c6f700 1 --2- 172.21.15.187:0/1831918447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600bfbe0 0x7f4f601480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.599+0000 7f4f64c6f700 1 -- 172.21.15.187:0/1831918447 >> v1:172.21.15.5:6789/0 conn(0x7f4f6009f560 legacy=0x7f4f60060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:21.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.599+0000 7f4f64c6f700 1 -- 172.21.15.187:0/1831918447 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f600a4cc0 con 0x7f4f600bf800 2020-11-20T12:14:21.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.599+0000 7f4f64c6f700 1 --2- 172.21.15.187:0/1831918447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f600bf800 0x7f4f6014c2c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f4f54006fd0 tx=0x7f4f54006eb0).ready entity=mon.1 client_cookie=6b3dae1b0feddb0d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:21.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f5f7fe700 1 -- 172.21.15.187:0/1831918447 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54009960 con 0x7f4f600bf800 2020-11-20T12:14:21.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f5f7fe700 1 -- 172.21.15.187:0/1831918447 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f54009ac0 con 0x7f4f600bf800 2020-11-20T12:14:21.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f5f7fe700 1 -- 172.21.15.187:0/1831918447 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54009e40 con 0x7f4f600bf800 2020-11-20T12:14:21.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f65c71700 1 -- 172.21.15.187:0/1831918447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f600bf800 msgr2=0x7f4f6014c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:21.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f65c71700 1 --2- 172.21.15.187:0/1831918447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f600bf800 0x7f4f6014c2c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f4f54006fd0 tx=0x7f4f54006eb0).stop 2020-11-20T12:14:21.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f65c71700 1 -- 172.21.15.187:0/1831918447 shutdown_connections 2020-11-20T12:14:21.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f65c71700 1 --2- 172.21.15.187:0/1831918447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f600bf800 0x7f4f6014c2c0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f65c71700 1 --2- 172.21.15.187:0/1831918447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600bfbe0 0x7f4f601480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f65c71700 1 -- 172.21.15.187:0/1831918447 >> 172.21.15.187:0/1831918447 conn(0x7f4f60068370 msgr2=0x7f4f600bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:21.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.600+0000 7f4f65c71700 1 -- 172.21.15.187:0/1831918447 shutdown_connections 2020-11-20T12:14:21.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.601+0000 7f4f65c71700 1 -- 172.21.15.187:0/1831918447 wait complete. 2020-11-20T12:14:21.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.602+0000 7f4f65c71700 1 Processor -- start 2020-11-20T12:14:21.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.603+0000 7f4f65c71700 1 -- start start 2020-11-20T12:14:21.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f65c71700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f6009f560 0x7f4f600b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:21.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f65c71700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f600bf800 0x7f4f600b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:21.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f65c71700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600ba4b0 0x7f4f60141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:21.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f65c71700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f600b7ba0 con 0x7f4f600bf800 2020-11-20T12:14:21.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f65c71700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f600b5000 con 0x7f4f6009f560 2020-11-20T12:14:21.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f65c71700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f600b3e90 con 0x7f4f600ba4b0 2020-11-20T12:14:21.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f64c6f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f6009f560 0x7f4f600b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:21.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f64c6f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f6009f560 0x7f4f600b76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48800/0 (socket says 172.21.15.187:48800) 2020-11-20T12:14:21.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f64c6f700 1 -- 172.21.15.187:0/244048496 learned_addr learned my addr 172.21.15.187:0/244048496 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:21.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f65470700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600ba4b0 0x7f4f60141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:21.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f5ffff700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f600bf800 0x7f4f600b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:21.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f64c6f700 1 -- 172.21.15.187:0/244048496 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600ba4b0 msgr2=0x7f4f60141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:21.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f64c6f700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600ba4b0 0x7f4f60141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f64c6f700 1 -- 172.21.15.187:0/244048496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f600bf800 msgr2=0x7f4f600b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:21.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.604+0000 7f4f64c6f700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f600bf800 0x7f4f600b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:21.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.605+0000 7f4f64c6f700 1 -- 172.21.15.187:0/244048496 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f600b4130 con 0x7f4f6009f560 2020-11-20T12:14:21.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.605+0000 7f4f64c6f700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f6009f560 0x7f4f600b76b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f4f54000dd0 tx=0x7f4f54015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:21.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.605+0000 7f4f5cff9700 1 -- 172.21.15.187:0/244048496 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5401e460 con 0x7f4f6009f560 2020-11-20T12:14:21.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.605+0000 7f4f5cff9700 1 -- 172.21.15.187:0/244048496 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5401e5c0 con 0x7f4f6009f560 2020-11-20T12:14:21.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.605+0000 7f4f5cff9700 1 -- 172.21.15.187:0/244048496 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5401e960 con 0x7f4f6009f560 2020-11-20T12:14:21.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.605+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f600b43d0 con 0x7f4f6009f560 2020-11-20T12:14:21.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.605+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f600b46d0 con 0x7f4f6009f560 2020-11-20T12:14:21.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.606+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f60080050 con 0x7f4f6009f560 2020-11-20T12:14:21.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.607+0000 7f4f5cff9700 1 -- 172.21.15.187:0/244048496 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4f54025030 con 0x7f4f6009f560 2020-11-20T12:14:21.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.608+0000 7f4f5cff9700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f38056d20 0x7f4f380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:21.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.608+0000 7f4f5cff9700 1 -- 172.21.15.187:0/244048496 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4f54071860 con 0x7f4f6009f560 2020-11-20T12:14:21.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.608+0000 7f4f5ffff700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f38056d20 0x7f4f380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:21.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.609+0000 7f4f5ffff700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f38056d20 0x7f4f380591e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f4f600619c0 tx=0x7f4f4c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:21.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.615+0000 7f4f5cff9700 1 -- 172.21.15.187:0/244048496 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4f54046d10 con 0x7f4f6009f560 2020-11-20T12:14:21.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:21.802+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4f6014c7b0 con 0x7f4f6009f560 2020-11-20T12:14:22.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.276+0000 7f4f5cff9700 1 -- 172.21.15.187:0/244048496 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v91) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4f54023020 con 0x7f4f6009f560 2020-11-20T12:14:22.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.279+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f38056d20 msgr2=0x7f4f380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:22.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.279+0000 7f4f65c71700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f38056d20 0x7f4f380591e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f4f600619c0 tx=0x7f4f4c008040).stop 2020-11-20T12:14:22.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.279+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f6009f560 msgr2=0x7f4f600b76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:22.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.279+0000 7f4f65c71700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f6009f560 0x7f4f600b76b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f4f54000dd0 tx=0x7f4f54015000).stop 2020-11-20T12:14:22.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.280+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 shutdown_connections 2020-11-20T12:14:22.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.280+0000 7f4f65c71700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f38056d20 0x7f4f380591e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:22.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.280+0000 7f4f65c71700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f600ba4b0 0x7f4f60141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:22.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.280+0000 7f4f65c71700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f6009f560 0x7f4f600b76b0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:22.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.280+0000 7f4f65c71700 1 --2- 172.21.15.187:0/244048496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f600bf800 0x7f4f600b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:22.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.280+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 >> 172.21.15.187:0/244048496 conn(0x7f4f60068370 msgr2=0x7f4f600bef10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:22.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.280+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 shutdown_connections 2020-11-20T12:14:22.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.280+0000 7f4f65c71700 1 -- 172.21.15.187:0/244048496 wait complete. 2020-11-20T12:14:22.287 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:14:22.312 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:14:22.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.553+0000 7f0ffd1d5700 1 Processor -- start 2020-11-20T12:14:22.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.554+0000 7f0ffd1d5700 1 -- start start 2020-11-20T12:14:22.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.555+0000 7f0ffd1d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ff8066f70 0x7f0ff80b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:22.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.555+0000 7f0ffd1d5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff80b1300 con 0x7f0ff81484d0 2020-11-20T12:14:22.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.555+0000 7f0ffd1d5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff80b15a0 con 0x7f0ff80b6650 2020-11-20T12:14:22.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.555+0000 7f0ffd1d5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8067730 con 0x7f0ff8066f70 2020-11-20T12:14:22.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.555+0000 7f0fef7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0ff80b6650 0x7f0ff80afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45416/0 (socket says 172.21.15.187:45416) 2020-11-20T12:14:22.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.555+0000 7f0fef7fe700 1 -- 172.21.15.187:0/2797171054 learned_addr learned my addr 172.21.15.187:0/2797171054 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:22.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.556+0000 7f0ffc9d4700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ff8066f70 0x7f0ff80b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:22.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.556+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218910227 0 0) 0x7f0ff80b15a0 con 0x7f0ff80b6650 2020-11-20T12:14:22.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.556+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd8003240 con 0x7f0ff80b6650 2020-11-20T12:14:22.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.556+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4193199697 0 0) 0x7f0fd8003240 con 0x7f0ff80b6650 2020-11-20T12:14:22.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.556+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ff8066f70 msgr2=0x7f0ff80b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:22.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.556+0000 7f0ff77fe700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ff8066f70 0x7f0ff80b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:22.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.556+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 >> v1:172.21.15.5:6789/0 conn(0x7f0ff81484d0 legacy=0x7f0ff8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:22.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.556+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80b1840 con 0x7f0ff80b6650 2020-11-20T12:14:22.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f0fe0001520 con 0x7f0ff80b6650 2020-11-20T12:14:22.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 >> v1:172.21.15.187:6789/0 conn(0x7f0ff80b6650 legacy=0x7f0ff80afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:22.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff77fe700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fd8003730 0x7f0fd8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:22.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff77fe700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fd80044e0 0x7f0fd8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:22.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff77fe700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ff8066f70 0x7f0fd8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:22.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8067730 con 0x7f0fd80044e0 2020-11-20T12:14:22.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8005340 con 0x7f0fd8003730 2020-11-20T12:14:22.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd800b730 con 0x7f0ff8066f70 2020-11-20T12:14:22.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff7fff700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fd8003730 0x7f0fd8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:22.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff7fff700 1 -- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ff8066f70 msgr2=0x7f0fd8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:22.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff7fff700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ff8066f70 0x7f0fd8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:22.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff7fff700 1 -- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fd80044e0 msgr2=0x7f0fd8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:22.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.557+0000 7f0ff7fff700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fd80044e0 0x7f0fd8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:22.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.558+0000 7f0ff7fff700 1 -- 172.21.15.187:0/2797171054 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd8003240 con 0x7f0fd8003730 2020-11-20T12:14:22.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.558+0000 7f0ff7fff700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fd8003730 0x7f0fd8003b50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f0fe8001920 tx=0x7f0fe8007b90).ready entity=mon.1 client_cookie=1ec0d6bd91a2732 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:22.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.558+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8007020 con 0x7f0fd8003730 2020-11-20T12:14:22.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.558+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe8007180 con 0x7f0fd8003730 2020-11-20T12:14:22.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.558+0000 7f0ff77fe700 1 -- 172.21.15.187:0/2797171054 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe800c620 con 0x7f0fd8003730 2020-11-20T12:14:22.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.558+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/2797171054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fd8003730 msgr2=0x7f0fd8003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:22.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.558+0000 7f0ffd1d5700 1 --2- 172.21.15.187:0/2797171054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fd8003730 0x7f0fd8003b50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f0fe8001920 tx=0x7f0fe8007b90).stop 2020-11-20T12:14:22.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.559+0000 7f0ffc9d4700 1 -- 172.21.15.187:0/2797171054 reap_dead start 2020-11-20T12:14:22.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.559+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/2797171054 shutdown_connections 2020-11-20T12:14:22.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.559+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/2797171054 >> 172.21.15.187:0/2797171054 conn(0x7f0ff80bab30 msgr2=0x7f0ff80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:22.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.559+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/2797171054 shutdown_connections 2020-11-20T12:14:22.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.559+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/2797171054 wait complete. 2020-11-20T12:14:22.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.560+0000 7f0ffd1d5700 1 Processor -- start 2020-11-20T12:14:22.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.560+0000 7f0ffd1d5700 1 -- start start 2020-11-20T12:14:22.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.561+0000 7f0ffd1d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fd8003730 0x7f0ff80afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:22.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.561+0000 7f0ffd1d5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ff809f620 0x7f0ff8144740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:22.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.561+0000 7f0ffd1d5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ff809d000 0x7f0ff8141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:22.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.561+0000 7f0ffd1d5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff80b9810 con 0x7f0ff809f620 2020-11-20T12:14:22.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.561+0000 7f0ffd1d5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff809e670 con 0x7f0ff809d000 2020-11-20T12:14:22.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.561+0000 7f0ffd1d5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff81435d0 con 0x7f0fd8003730 2020-11-20T12:14:22.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.561+0000 7f0ffc9d4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ff809d000 0x7f0ff8141ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:22.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ffc9d4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ff809d000 0x7f0ff8141ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48824/0 (socket says 172.21.15.187:48824) 2020-11-20T12:14:22.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ffc9d4700 1 -- 172.21.15.187:0/37273994 learned_addr learned my addr 172.21.15.187:0/37273994 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:22.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ff7fff700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fd8003730 0x7f0ff80afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:22.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0fef7fe700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ff809f620 0x7f0ff8144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:22.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ffc9d4700 1 -- 172.21.15.187:0/37273994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fd8003730 msgr2=0x7f0ff80afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:22.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ffc9d4700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fd8003730 0x7f0ff80afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:22.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ffc9d4700 1 -- 172.21.15.187:0/37273994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ff809f620 msgr2=0x7f0ff8144740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:22.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ffc9d4700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ff809f620 0x7f0ff8144740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:22.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ffc9d4700 1 -- 172.21.15.187:0/37273994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80b0140 con 0x7f0ff809d000 2020-11-20T12:14:22.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.562+0000 7f0ffc9d4700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ff809d000 0x7f0ff8141ff0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f0ff8099f10 tx=0x7f0ff0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:22.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.563+0000 7f0ff4ff9700 1 -- 172.21.15.187:0/37273994 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff000c440 con 0x7f0ff809d000 2020-11-20T12:14:22.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.563+0000 7f0ff4ff9700 1 -- 172.21.15.187:0/37273994 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff000c5a0 con 0x7f0ff809d000 2020-11-20T12:14:22.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.563+0000 7f0ff4ff9700 1 -- 172.21.15.187:0/37273994 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff000c960 con 0x7f0ff809d000 2020-11-20T12:14:22.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.563+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff8144c30 con 0x7f0ff809d000 2020-11-20T12:14:22.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.563+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff81426a0 con 0x7f0ff809d000 2020-11-20T12:14:22.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.564+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff8142920 con 0x7f0ff809d000 2020-11-20T12:14:22.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.564+0000 7f0ff4ff9700 1 -- 172.21.15.187:0/37273994 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0ff001b030 con 0x7f0ff809d000 2020-11-20T12:14:22.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.565+0000 7f0ff4ff9700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fd4056d50 0x7f0fd4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:22.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.565+0000 7f0ff4ff9700 1 -- 172.21.15.187:0/37273994 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f0ff006f4f0 con 0x7f0ff809d000 2020-11-20T12:14:22.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.566+0000 7f0ff7fff700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fd4056d50 0x7f0fd4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:22.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.566+0000 7f0ff7fff700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fd4056d50 0x7f0fd4059210 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f0fe8000fd0 tx=0x7f0fe8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:22.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.569+0000 7f0ff4ff9700 1 -- 172.21.15.187:0/37273994 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0ff0044a20 con 0x7f0ff809d000 2020-11-20T12:14:22.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:22.756+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0ff8143790 con 0x7f0ff809d000 2020-11-20T12:14:23.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.280+0000 7f0ff4ff9700 1 -- 172.21.15.187:0/37273994 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v92) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0ff0048040 con 0x7f0ff809d000 2020-11-20T12:14:23.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.284+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fd4056d50 msgr2=0x7f0fd4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:23.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.284+0000 7f0ffd1d5700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fd4056d50 0x7f0fd4059210 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f0fe8000fd0 tx=0x7f0fe8015000).stop 2020-11-20T12:14:23.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.284+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ff809d000 msgr2=0x7f0ff8141ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:23.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.284+0000 7f0ffd1d5700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ff809d000 0x7f0ff8141ff0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f0ff8099f10 tx=0x7f0ff0008040).stop 2020-11-20T12:14:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.285+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 shutdown_connections 2020-11-20T12:14:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.285+0000 7f0ffd1d5700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fd4056d50 0x7f0fd4059210 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.285+0000 7f0ffd1d5700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ff809d000 0x7f0ff8141ff0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.285+0000 7f0ffd1d5700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fd8003730 0x7f0ff80afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.285+0000 7f0ffd1d5700 1 --2- 172.21.15.187:0/37273994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ff809f620 0x7f0ff8144740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.285+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 >> 172.21.15.187:0/37273994 conn(0x7f0ff80bab30 msgr2=0x7f0ff80b7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:23.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.285+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 shutdown_connections 2020-11-20T12:14:23.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.286+0000 7f0ffd1d5700 1 -- 172.21.15.187:0/37273994 wait complete. 2020-11-20T12:14:23.292 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:14:23.319 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:14:23.319 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:14:23.319 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:14:23.320 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:14:23.320 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:14:23.320 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:14:23.321 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:14:23.321 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:14:23.339 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:14:23.339 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:14:23.353 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:14:23.354 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:14:23.387 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:14:23.388 DEBUG:teuthology.parallel:result is None 2020-11-20T12:14:23.403 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:14:23.403 DEBUG:teuthology.parallel:result is None 2020-11-20T12:14:23.404 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:14:23.556 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:14:23.600 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:14:23.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.619+0000 7fa47bfff700 1 Processor -- start 2020-11-20T12:14:23.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.619+0000 7fa47bfff700 1 -- start start 2020-11-20T12:14:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.620+0000 7fa47bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa47c0675d0 0x7fa47c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.620+0000 7fa47bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 0x7fa47c147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.620+0000 7fa47bfff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c14be70 con 0x7fa47c0ba9b0 2020-11-20T12:14:23.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.620+0000 7fa47bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47c14c110 con 0x7fa47c0675d0 2020-11-20T12:14:23.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.620+0000 7fa47bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c14c3b0 con 0x7fa47c0679b0 2020-11-20T12:14:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.620+0000 7fa47b7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa47c0ba9b0 0x7fa47c1481e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45448/0 (socket says 172.21.15.187:45448) 2020-11-20T12:14:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.620+0000 7fa47b7fe700 1 -- 172.21.15.187:0/1293155788 learned_addr learned my addr 172.21.15.187:0/1293155788 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa47affd700 1 --2- 172.21.15.187:0/1293155788 >> v2:172.21.15.5:3300/0 conn(0x7fa47c0675d0 0x7fa47c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa47a7fc700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 0x7fa47c147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575010326 0 0) 0x7fa47c14be70 con 0x7fa47c0ba9b0 2020-11-20T12:14:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa468003240 con 0x7fa47c0ba9b0 2020-11-20T12:14:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 902125467 0 0) 0x7fa468003240 con 0x7fa47c0ba9b0 2020-11-20T12:14:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 msgr2=0x7fa47c147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:23.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 0x7fa47c147cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 >> v2:172.21.15.5:3300/0 conn(0x7fa47c0675d0 msgr2=0x7fa47c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:23.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 --2- 172.21.15.187:0/1293155788 >> v2:172.21.15.5:3300/0 conn(0x7fa47c0675d0 0x7fa47c14c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c14cd40 con 0x7fa47c0ba9b0 2020-11-20T12:14:23.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fa474001860 con 0x7fa47c0ba9b0 2020-11-20T12:14:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 >> v1:172.21.15.187:6789/0 conn(0x7fa47c0ba9b0 legacy=0x7fa47c1481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.621+0000 7fa479ffb700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4680037b0 0x7fa468003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa479ffb700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 0x7fa4680044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa479ffb700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4680049d0 0x7fa468004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c14c110 con 0x7fa4680049d0 2020-11-20T12:14:23.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c14c3b0 con 0x7fa4680037b0 2020-11-20T12:14:23.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680042a0 con 0x7fa47c0679b0 2020-11-20T12:14:23.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47affd700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4680037b0 0x7fa468003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47a7fc700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 0x7fa4680044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47b7fe700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4680049d0 0x7fa468004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47affd700 1 -- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 msgr2=0x7fa4680044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:23.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47affd700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 0x7fa4680044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47affd700 1 -- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4680049d0 msgr2=0x7fa468004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:23.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47affd700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4680049d0 0x7fa468004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47affd700 1 -- 172.21.15.187:0/1293155788 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa468003240 con 0x7fa4680037b0 2020-11-20T12:14:23.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47a7fc700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c0679b0 0x7fa4680044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:23.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.622+0000 7fa47b7fe700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4680049d0 0x7fa468004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:23.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.623+0000 7fa47affd700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4680037b0 0x7fa468003bd0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7fa46c00bc00 tx=0x7fa46c006df0).ready entity=mon.1 client_cookie=92bfaf528222476b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:23.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.623+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c00b310 con 0x7fa4680037b0 2020-11-20T12:14:23.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.623+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c00eb40 con 0x7fa4680037b0 2020-11-20T12:14:23.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.623+0000 7fa479ffb700 1 -- 172.21.15.187:0/1293155788 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c00eec0 con 0x7fa4680037b0 2020-11-20T12:14:23.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.624+0000 7fa47bfff700 1 -- 172.21.15.187:0/1293155788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4680037b0 msgr2=0x7fa468003bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:23.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.624+0000 7fa47bfff700 1 --2- 172.21.15.187:0/1293155788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4680037b0 0x7fa468003bd0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7fa46c00bc00 tx=0x7fa46c006df0).stop 2020-11-20T12:14:23.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.624+0000 7fa47b7fe700 1 -- 172.21.15.187:0/1293155788 reap_dead start 2020-11-20T12:14:23.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.624+0000 7fa47bfff700 1 -- 172.21.15.187:0/1293155788 shutdown_connections 2020-11-20T12:14:23.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.624+0000 7fa47bfff700 1 -- 172.21.15.187:0/1293155788 >> 172.21.15.187:0/1293155788 conn(0x7fa47c0bce50 msgr2=0x7fa47c0bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:23.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.624+0000 7fa47bfff700 1 -- 172.21.15.187:0/1293155788 shutdown_connections 2020-11-20T12:14:23.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.624+0000 7fa47bfff700 1 -- 172.21.15.187:0/1293155788 wait complete. 2020-11-20T12:14:23.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.625+0000 7fa47bfff700 1 Processor -- start 2020-11-20T12:14:23.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.625+0000 7fa47bfff700 1 -- start start 2020-11-20T12:14:23.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.626+0000 7fa47bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c062040 0x7fa47c0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.626+0000 7fa47bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47c062420 0x7fa47c0a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.626+0000 7fa47bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa47c1489f0 0x7fa47c143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.626+0000 7fa47bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c0b4a50 con 0x7fa47c1489f0 2020-11-20T12:14:23.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.626+0000 7fa47bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c0b4cf0 con 0x7fa47c062420 2020-11-20T12:14:23.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.626+0000 7fa47bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c09fe00 con 0x7fa47c062040 2020-11-20T12:14:23.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.626+0000 7fa47a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47c062420 0x7fa47c0a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47c062420 0x7fa47c0a1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48854/0 (socket says 172.21.15.187:48854) 2020-11-20T12:14:23.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47a7fc700 1 -- 172.21.15.187:0/2805644759 learned_addr learned my addr 172.21.15.187:0/2805644759 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:23.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47affd700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c062040 0x7fa47c0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47b7fe700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa47c1489f0 0x7fa47c143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47a7fc700 1 -- 172.21.15.187:0/2805644759 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c062040 msgr2=0x7fa47c0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:23.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47a7fc700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c062040 0x7fa47c0b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47a7fc700 1 -- 172.21.15.187:0/2805644759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa47c1489f0 msgr2=0x7fa47c143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:23.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47a7fc700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa47c1489f0 0x7fa47c143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:23.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47a7fc700 1 -- 172.21.15.187:0/2805644759 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c0b4710 con 0x7fa47c062420 2020-11-20T12:14:23.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47a7fc700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47c062420 0x7fa47c0a1800 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7fa470010c20 tx=0x7fa470008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:23.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa462ffd700 1 -- 172.21.15.187:0/2805644759 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47000ea90 con 0x7fa47c062420 2020-11-20T12:14:23.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa462ffd700 1 -- 172.21.15.187:0/2805644759 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47000ebf0 con 0x7fa47c062420 2020-11-20T12:14:23.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.627+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47c0a1cf0 con 0x7fa47c062420 2020-11-20T12:14:23.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.628+0000 7fa462ffd700 1 -- 172.21.15.187:0/2805644759 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470014470 con 0x7fa47c062420 2020-11-20T12:14:23.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.628+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47c1441b0 con 0x7fa47c062420 2020-11-20T12:14:23.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.629+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47c09c0d0 con 0x7fa47c062420 2020-11-20T12:14:23.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.629+0000 7fa462ffd700 1 -- 172.21.15.187:0/2805644759 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa470024070 con 0x7fa47c062420 2020-11-20T12:14:23.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.630+0000 7fa462ffd700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa458056d20 0x7fa4580591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:23.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.630+0000 7fa462ffd700 1 -- 172.21.15.187:0/2805644759 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa470070610 con 0x7fa47c062420 2020-11-20T12:14:23.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.630+0000 7fa47affd700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa458056d20 0x7fa4580591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:23.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.631+0000 7fa47affd700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa458056d20 0x7fa4580591e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fa47c05ea30 tx=0x7fa46c019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:23.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.634+0000 7fa462ffd700 1 -- 172.21.15.187:0/2805644759 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa470045d40 con 0x7fa47c062420 2020-11-20T12:14:23.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:23.841+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa47c0bcb60 con 0x7fa47c062420 2020-11-20T12:14:24.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.294+0000 7fa462ffd700 1 -- 172.21.15.187:0/2805644759 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v93) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa470018020 con 0x7fa47c062420 2020-11-20T12:14:24.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa458056d20 msgr2=0x7fa4580591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa458056d20 0x7fa4580591e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fa47c05ea30 tx=0x7fa46c019000).stop 2020-11-20T12:14:24.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47c062420 msgr2=0x7fa47c0a1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47c062420 0x7fa47c0a1800 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7fa470010c20 tx=0x7fa470008040).stop 2020-11-20T12:14:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 shutdown_connections 2020-11-20T12:14:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa458056d20 0x7fa4580591e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:24.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa47c1489f0 0x7fa47c143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa47c062040 0x7fa47c0b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 --2- 172.21.15.187:0/2805644759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47c062420 0x7fa47c0a1800 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 >> 172.21.15.187:0/2805644759 conn(0x7fa47c0bce50 msgr2=0x7fa47c145ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 shutdown_connections 2020-11-20T12:14:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.297+0000 7fa47bfff700 1 -- 172.21.15.187:0/2805644759 wait complete. 2020-11-20T12:14:24.304 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:14:24.325 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:14:24.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.568+0000 7f6e1a3df700 1 Processor -- start 2020-11-20T12:14:24.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.568+0000 7f6e1a3df700 1 -- start start 2020-11-20T12:14:24.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.569+0000 7f6e1a3df700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6e140b1760 0x7f6e14065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:24.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.569+0000 7f6e1a3df700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e14150390 con 0x7f6e14065680 2020-11-20T12:14:24.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.569+0000 7f6e1a3df700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e14150630 con 0x7f6e140b1380 2020-11-20T12:14:24.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.569+0000 7f6e1a3df700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e141511e0 con 0x7f6e140b1760 2020-11-20T12:14:24.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.570+0000 7f6e193dd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6e140b1380 0x7f6e14150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45466/0 (socket says 172.21.15.187:45466) 2020-11-20T12:14:24.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.570+0000 7f6e193dd700 1 -- 172.21.15.187:0/3038980675 learned_addr learned my addr 172.21.15.187:0/3038980675 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:24.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.570+0000 7f6e18bdc700 1 --2- 172.21.15.187:0/3038980675 >> v2:172.21.15.187:3300/0 conn(0x7f6e140b1760 0x7f6e14065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:24.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.570+0000 7f6e03fff700 1 -- 172.21.15.187:0/3038980675 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236496174 0 0) 0x7f6e14150630 con 0x7f6e140b1380 2020-11-20T12:14:24.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.570+0000 7f6e03fff700 1 -- 172.21.15.187:0/3038980675 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc003240 con 0x7f6e140b1380 2020-11-20T12:14:24.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.571+0000 7f6e18bdc700 1 -- 172.21.15.187:0/3038980675 >> v1:172.21.15.187:6789/0 conn(0x7f6e140b1380 legacy=0x7f6e14150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.571+0000 7f6e18bdc700 1 -- 172.21.15.187:0/3038980675 >> v1:172.21.15.5:6789/0 conn(0x7f6e14065680 legacy=0x7f6e1414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.571+0000 7f6e18bdc700 1 -- 172.21.15.187:0/3038980675 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e14068510 con 0x7f6e140b1760 2020-11-20T12:14:24.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.571+0000 7f6e18bdc700 1 --2- 172.21.15.187:0/3038980675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e14065190 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f6e0400b9a0 tx=0x7f6e04006df0).ready entity=mon.1 client_cookie=62f007b8ed57528f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:24.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.571+0000 7f6e03fff700 1 -- 172.21.15.187:0/3038980675 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e040097b0 con 0x7f6e140b1760 2020-11-20T12:14:24.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.571+0000 7f6e03fff700 1 -- 172.21.15.187:0/3038980675 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e04009910 con 0x7f6e140b1760 2020-11-20T12:14:24.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.571+0000 7f6e03fff700 1 -- 172.21.15.187:0/3038980675 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04009c90 con 0x7f6e140b1760 2020-11-20T12:14:24.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.572+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3038980675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 msgr2=0x7f6e14065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.572+0000 7f6e1a3df700 1 --2- 172.21.15.187:0/3038980675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e14065190 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f6e0400b9a0 tx=0x7f6e04006df0).stop 2020-11-20T12:14:24.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.572+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3038980675 shutdown_connections 2020-11-20T12:14:24.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.572+0000 7f6e1a3df700 1 --2- 172.21.15.187:0/3038980675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e14065190 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:24.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.572+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3038980675 >> 172.21.15.187:0/3038980675 conn(0x7f6e140bab30 msgr2=0x7f6e140baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:24.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.572+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3038980675 shutdown_connections 2020-11-20T12:14:24.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.572+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3038980675 wait complete. 2020-11-20T12:14:24.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.574+0000 7f6e1a3df700 1 Processor -- start 2020-11-20T12:14:24.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.574+0000 7f6e1a3df700 1 -- start start 2020-11-20T12:14:24.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.574+0000 7f6e1a3df700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e140b1380 0x7f6e140a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:24.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.574+0000 7f6e1a3df700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e140b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:24.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.574+0000 7f6e1a3df700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e14160350 0x7f6e1414f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:24.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e1a3df700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e140b4200 con 0x7f6e14160350 2020-11-20T12:14:24.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e1a3df700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1415f970 con 0x7f6e140b1760 2020-11-20T12:14:24.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e1a3df700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1415fc10 con 0x7f6e140b1380 2020-11-20T12:14:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e18bdc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e140b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e18bdc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e140b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48868/0 (socket says 172.21.15.187:48868) 2020-11-20T12:14:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e18bdc700 1 -- 172.21.15.187:0/3744727715 learned_addr learned my addr 172.21.15.187:0/3744727715 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e18bdc700 1 -- 172.21.15.187:0/3744727715 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e140b1380 msgr2=0x7f6e140a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e18bdc700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e140b1380 0x7f6e140a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:24.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e18bdc700 1 -- 172.21.15.187:0/3744727715 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e14160350 msgr2=0x7f6e1414f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e18bdc700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e14160350 0x7f6e1414f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:24.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.575+0000 7f6e18bdc700 1 -- 172.21.15.187:0/3744727715 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1415feb0 con 0x7f6e140b1760 2020-11-20T12:14:24.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.576+0000 7f6e18bdc700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e140b3d10 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7f6e04006e30 tx=0x7f6e04015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:24.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.576+0000 7f6e017fa700 1 -- 172.21.15.187:0/3744727715 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04011e50 con 0x7f6e140b1760 2020-11-20T12:14:24.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.576+0000 7f6e017fa700 1 -- 172.21.15.187:0/3744727715 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0401e460 con 0x7f6e140b1760 2020-11-20T12:14:24.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.576+0000 7f6e017fa700 1 -- 172.21.15.187:0/3744727715 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0401e7e0 con 0x7f6e140b1760 2020-11-20T12:14:24.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.576+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e14160790 con 0x7f6e140b1760 2020-11-20T12:14:24.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.576+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e140b5310 con 0x7f6e140b1760 2020-11-20T12:14:24.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.577+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e14160040 con 0x7f6e140b1760 2020-11-20T12:14:24.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.578+0000 7f6e017fa700 1 -- 172.21.15.187:0/3744727715 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6e04025030 con 0x7f6e140b1760 2020-11-20T12:14:24.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.578+0000 7f6e017fa700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dec056d20 0x7f6dec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:24.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.578+0000 7f6e017fa700 1 -- 172.21.15.187:0/3744727715 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f6e04071430 con 0x7f6e140b1760 2020-11-20T12:14:24.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.579+0000 7f6e193dd700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dec056d20 0x7f6dec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:24.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.579+0000 7f6e193dd700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dec056d20 0x7f6dec0591e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f6e100055a0 tx=0x7f6e100076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:24.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.582+0000 7f6e017fa700 1 -- 172.21.15.187:0/3744727715 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6e04046c60 con 0x7f6e140b1760 2020-11-20T12:14:24.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.834+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6e14061a20 con 0x7f6e140b1760 2020-11-20T12:14:24.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.836+0000 7f6e017fa700 1 -- 172.21.15.187:0/3744727715 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f6e04023070 con 0x7f6e140b1760 2020-11-20T12:14:24.840 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:24.840 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:14:24.840 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:14:24.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.841 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:14:24.841 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:14:24.841 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.841 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:14:24.842 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.842 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.842 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.842 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.843 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:14:24.843 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:14:24.844 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.844 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:14:24.844 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.844 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.845 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.845 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.846 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:14:24.846 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:14:24.846 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.846 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.846 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.847 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.847 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.847 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:14:24.848 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:14:24.848 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.848 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.848 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.849 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.849 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.849 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:14:24.850 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:14:24.850 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.850 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.850 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.850 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.851 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.851 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:14:24.852 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:14:24.852 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.852 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.852 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.852 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.853 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.854 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:14:24.854 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:14:24.854 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.854 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.855 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.855 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.855 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.856 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:14:24.856 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:14:24.856 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.856 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.857 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.857 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.857 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.858 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:14:24.858 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:14:24.858 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.858 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.859 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.859 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.859 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.860 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:14:24.860 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:14:24.860 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.860 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.861 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.861 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.861 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.862 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:14:24.862 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:14:24.862 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.862 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:14:24.863 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:14:24.863 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:14:24.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.864 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:14:24.864 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:14:24.864 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.865 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:14:24.865 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:14:24.865 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:14:24.865 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:14:24.865 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.867 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:14:24.867 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:14:24.867 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.868 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:14:24.868 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:14:24.868 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:14:24.868 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:14:24.869 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.869 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:14:24.869 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:14:24.870 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.870 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:14:24.870 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:14:24.870 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:14:24.870 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:14:24.871 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.871 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:14:24.872 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:14:24.872 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.872 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:14:24.872 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:14:24.872 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:14:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.873 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:14:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:14:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:14:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:14:24.874 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.875 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:24.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:24.875 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:14:24.876 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:14:24.876 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:14:24.876 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:14:24.876 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:14:24.876 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:14:24.877 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.877 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:24.877 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:14:24.877 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:14:24.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dec056d20 msgr2=0x7f6dec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dec056d20 0x7f6dec0591e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f6e100055a0 tx=0x7f6e100076a0).stop 2020-11-20T12:14:24.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 msgr2=0x7f6e140b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:24.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e140b3d10 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7f6e04006e30 tx=0x7f6e04015000).stop 2020-11-20T12:14:24.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 shutdown_connections 2020-11-20T12:14:24.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dec056d20 0x7f6dec0591e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:24.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e14160350 0x7f6e1414f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:24.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e140b1380 0x7f6e140a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:24.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 --2- 172.21.15.187:0/3744727715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e140b1760 0x7f6e140b3d10 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:24.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 >> 172.21.15.187:0/3744727715 conn(0x7f6e140bab30 msgr2=0x7f6e1414b670 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:24.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 shutdown_connections 2020-11-20T12:14:24.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:24.838+0000 7f6e1a3df700 1 -- 172.21.15.187:0/3744727715 wait complete. 2020-11-20T12:14:24.882 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:25.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.119+0000 7fc50359e700 1 Processor -- start 2020-11-20T12:14:25.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.120+0000 7fc50359e700 1 -- start start 2020-11-20T12:14:25.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.121+0000 7fc50359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc5041484d0 0x7fc504061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.121+0000 7fc50359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504066f70 0x7fc5040b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.121+0000 7fc50359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5040b1300 con 0x7fc5040622c0 2020-11-20T12:14:25.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.121+0000 7fc50359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc504067730 con 0x7fc5041484d0 2020-11-20T12:14:25.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.121+0000 7fc50359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5040b15a0 con 0x7fc504066f70 2020-11-20T12:14:25.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.121+0000 7fc501d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc5040622c0 0x7fc5040b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45480/0 (socket says 172.21.15.187:45480) 2020-11-20T12:14:25.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.121+0000 7fc501d9b700 1 -- 172.21.15.187:0/179018060 learned_addr learned my addr 172.21.15.187:0/179018060 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:25.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.122+0000 7fc50259c700 1 --2- 172.21.15.187:0/179018060 >> v2:172.21.15.5:3300/0 conn(0x7fc5041484d0 0x7fc504061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.122+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864903555 0 0) 0x7fc5040b1300 con 0x7fc5040622c0 2020-11-20T12:14:25.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.122+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e8003240 con 0x7fc5040622c0 2020-11-20T12:14:25.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.122+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2017164080 0 0) 0x7fc4e8003240 con 0x7fc5040622c0 2020-11-20T12:14:25.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504066f70 msgr2=0x7fc5040b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:25.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc502d9d700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504066f70 0x7fc5040b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504066f70 0x7fc5040b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 >> v2:172.21.15.5:3300/0 conn(0x7fc5041484d0 msgr2=0x7fc504061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:25.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 --2- 172.21.15.187:0/179018060 >> v2:172.21.15.5:3300/0 conn(0x7fc5041484d0 0x7fc504061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5040b1840 con 0x7fc5040622c0 2020-11-20T12:14:25.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fc4f80014e0 con 0x7fc5040622c0 2020-11-20T12:14:25.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50259c700 1 --2- 172.21.15.187:0/179018060 >> v2:172.21.15.5:3300/0 conn(0x7fc5041484d0 0x7fc504061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:25.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 >> v1:172.21.15.187:6789/0 conn(0x7fc5040622c0 legacy=0x7fc5040b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4e8003730 0x7fc4e8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504066f70 0x7fc4e80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e80049d0 0x7fc4e8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc504067730 con 0x7fc4e8003730 2020-11-20T12:14:25.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5040b15a0 con 0x7fc4e80049d0 2020-11-20T12:14:25.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e80041d0 con 0x7fc504066f70 2020-11-20T12:14:25.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.123+0000 7fc502d9d700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e80049d0 0x7fc4e8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc50259c700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4e8003730 0x7fc4e8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc501d9b700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504066f70 0x7fc4e80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc502d9d700 1 -- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504066f70 msgr2=0x7fc4e80044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:25.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc502d9d700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504066f70 0x7fc4e80044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc502d9d700 1 -- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4e8003730 msgr2=0x7fc4e8003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:25.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc502d9d700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4e8003730 0x7fc4e8003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc502d9d700 1 -- 172.21.15.187:0/179018060 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e8003240 con 0x7fc4e80049d0 2020-11-20T12:14:25.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc502d9d700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e80049d0 0x7fc4e8004e50 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7fc4fc00ad60 tx=0x7fc4fc008040).ready entity=mon.1 client_cookie=b3353ffe8d268014 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:25.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc00ccb0 con 0x7fc4e80049d0 2020-11-20T12:14:25.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc00ce10 con 0x7fc4e80049d0 2020-11-20T12:14:25.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.124+0000 7fc50159a700 1 -- 172.21.15.187:0/179018060 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc011620 con 0x7fc4e80049d0 2020-11-20T12:14:25.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.125+0000 7fc50359e700 1 -- 172.21.15.187:0/179018060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e80049d0 msgr2=0x7fc4e8004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.125+0000 7fc50359e700 1 --2- 172.21.15.187:0/179018060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e80049d0 0x7fc4e8004e50 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7fc4fc00ad60 tx=0x7fc4fc008040).stop 2020-11-20T12:14:25.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.125+0000 7fc502d9d700 1 -- 172.21.15.187:0/179018060 reap_dead start 2020-11-20T12:14:25.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.125+0000 7fc50359e700 1 -- 172.21.15.187:0/179018060 shutdown_connections 2020-11-20T12:14:25.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.125+0000 7fc50359e700 1 -- 172.21.15.187:0/179018060 >> 172.21.15.187:0/179018060 conn(0x7fc5040bab30 msgr2=0x7fc5040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:25.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.125+0000 7fc50359e700 1 -- 172.21.15.187:0/179018060 shutdown_connections 2020-11-20T12:14:25.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.125+0000 7fc50359e700 1 -- 172.21.15.187:0/179018060 wait complete. 2020-11-20T12:14:25.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.127+0000 7fc50359e700 1 Processor -- start 2020-11-20T12:14:25.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.127+0000 7fc50359e700 1 -- start start 2020-11-20T12:14:25.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc50359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc504144fb0 0x7fc50414de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc50359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504145390 0x7fc50409ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc50359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc50409bfd0 0x7fc5041413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc50359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50414e330 con 0x7fc504144fb0 2020-11-20T12:14:25.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc50359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50414e550 con 0x7fc50409bfd0 2020-11-20T12:14:25.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc50359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50409dae0 con 0x7fc504145390 2020-11-20T12:14:25.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc502d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc50409bfd0 0x7fc5041413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc502d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc50409bfd0 0x7fc5041413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48890/0 (socket says 172.21.15.187:48890) 2020-11-20T12:14:25.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.128+0000 7fc502d9d700 1 -- 172.21.15.187:0/2848755772 learned_addr learned my addr 172.21.15.187:0/2848755772 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:25.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.129+0000 7fc50259c700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc504144fb0 0x7fc50414de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.129+0000 7fc501d9b700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504145390 0x7fc50409ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.129+0000 7fc502d9d700 1 -- 172.21.15.187:0/2848755772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504145390 msgr2=0x7fc50409ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.129+0000 7fc502d9d700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504145390 0x7fc50409ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.129+0000 7fc502d9d700 1 -- 172.21.15.187:0/2848755772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc504144fb0 msgr2=0x7fc50414de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.129+0000 7fc502d9d700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc504144fb0 0x7fc50414de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.129+0000 7fc502d9d700 1 -- 172.21.15.187:0/2848755772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50409dd80 con 0x7fc50409bfd0 2020-11-20T12:14:25.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.130+0000 7fc502d9d700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc50409bfd0 0x7fc5041413e0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7fc5040673e0 tx=0x7fc4fc018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:25.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.130+0000 7fc4f27fc700 1 -- 172.21.15.187:0/2848755772 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc00ccb0 con 0x7fc50409bfd0 2020-11-20T12:14:25.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.130+0000 7fc4f27fc700 1 -- 172.21.15.187:0/2848755772 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc01ab00 con 0x7fc50409bfd0 2020-11-20T12:14:25.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.130+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50409f320 con 0x7fc50409bfd0 2020-11-20T12:14:25.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.130+0000 7fc4f27fc700 1 -- 172.21.15.187:0/2848755772 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc01aea0 con 0x7fc50409bfd0 2020-11-20T12:14:25.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.130+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc504141a90 con 0x7fc50409bfd0 2020-11-20T12:14:25.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.131+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5040b7300 con 0x7fc50409bfd0 2020-11-20T12:14:25.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.132+0000 7fc4f27fc700 1 -- 172.21.15.187:0/2848755772 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc4fc02b070 con 0x7fc50409bfd0 2020-11-20T12:14:25.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.132+0000 7fc4f27fc700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4d8056d20 0x7fc4d80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.132+0000 7fc4f27fc700 1 -- 172.21.15.187:0/2848755772 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc4fc0716d0 con 0x7fc50409bfd0 2020-11-20T12:14:25.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.132+0000 7fc50259c700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4d8056d20 0x7fc4d80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.133+0000 7fc50259c700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4d8056d20 0x7fc4d80591e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7fc4f4006a90 tx=0x7fc4f400b040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:25.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.136+0000 7fc4f27fc700 1 -- 172.21.15.187:0/2848755772 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc4fc046e50 con 0x7fc50409bfd0 2020-11-20T12:14:25.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.321+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc50409df10 con 0x7fc50409bfd0 2020-11-20T12:14:25.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.323+0000 7fc4f27fc700 1 -- 172.21.15.187:0/2848755772 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+13628 (secure 0 0 0) 0x7fc4fc04a470 con 0x7fc50409bfd0 2020-11-20T12:14:25.327 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:25.327 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":93,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:14:24.276130+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:25.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.325+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4d8056d20 msgr2=0x7fc4d80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.325+0000 7fc50359e700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4d8056d20 0x7fc4d80591e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7fc4f4006a90 tx=0x7fc4f400b040).stop 2020-11-20T12:14:25.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.325+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc50409bfd0 msgr2=0x7fc5041413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.325+0000 7fc50359e700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc50409bfd0 0x7fc5041413e0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7fc5040673e0 tx=0x7fc4fc018000).stop 2020-11-20T12:14:25.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.326+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 shutdown_connections 2020-11-20T12:14:25.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.326+0000 7fc50359e700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4d8056d20 0x7fc4d80591e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.326+0000 7fc50359e700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc504144fb0 0x7fc50414de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.326+0000 7fc50359e700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc50409bfd0 0x7fc5041413e0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.326+0000 7fc50359e700 1 --2- 172.21.15.187:0/2848755772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc504145390 0x7fc50409ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.326+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 >> 172.21.15.187:0/2848755772 conn(0x7fc5040bab30 msgr2=0x7fc5040b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:25.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.326+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 shutdown_connections 2020-11-20T12:14:25.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.326+0000 7fc50359e700 1 -- 172.21.15.187:0/2848755772 wait complete. 2020-11-20T12:14:25.358 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:14:25.359 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:14:25.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.609+0000 7f862ef97700 1 Processor -- start 2020-11-20T12:14:25.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.609+0000 7f862ef97700 1 -- start start 2020-11-20T12:14:25.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.610+0000 7f862ef97700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8628066f70 0x7f86280b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.611+0000 7f862ef97700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280b1300 con 0x7f86281484d0 2020-11-20T12:14:25.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.611+0000 7f862ef97700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280b15a0 con 0x7f86280b6650 2020-11-20T12:14:25.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.611+0000 7f862ef97700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8628067730 con 0x7f8628066f70 2020-11-20T12:14:25.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.611+0000 7f862e796700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8628066f70 0x7f86280b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.611+0000 7f862d794700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f86280b6650 0x7f86280afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45504/0 (socket says 172.21.15.187:45504) 2020-11-20T12:14:25.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.611+0000 7f862df95700 1 --1- 172.21.15.187:0/690823499 >> v1:172.21.15.5:6789/0 conn(0x7f86281484d0 0x7f8628061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:39010/0 (socket says 172.21.15.187:39010) 2020-11-20T12:14:25.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.611+0000 7f862d794700 1 -- 172.21.15.187:0/690823499 learned_addr learned my addr 172.21.15.187:0/690823499 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:25.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.612+0000 7f862cf93700 1 -- 172.21.15.187:0/690823499 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897354422 0 0) 0x7f86280b1300 con 0x7f86281484d0 2020-11-20T12:14:25.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.612+0000 7f862cf93700 1 -- 172.21.15.187:0/690823499 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8610003240 con 0x7f86281484d0 2020-11-20T12:14:25.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.613+0000 7f862cf93700 1 -- 172.21.15.187:0/690823499 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778179451 0 0) 0x7f86280b15a0 con 0x7f86280b6650 2020-11-20T12:14:25.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.613+0000 7f862cf93700 1 -- 172.21.15.187:0/690823499 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86280b1300 con 0x7f86280b6650 2020-11-20T12:14:25.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.613+0000 7f862e796700 1 -- 172.21.15.187:0/690823499 >> v1:172.21.15.187:6789/0 conn(0x7f86280b6650 legacy=0x7f86280afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.613+0000 7f862e796700 1 -- 172.21.15.187:0/690823499 >> v1:172.21.15.5:6789/0 conn(0x7f86281484d0 legacy=0x7f8628061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.613+0000 7f862e796700 1 -- 172.21.15.187:0/690823499 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280b1840 con 0x7f8628066f70 2020-11-20T12:14:25.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.613+0000 7f862e796700 1 --2- 172.21.15.187:0/690823499 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8628066f70 0x7f86280b01a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f862000b9d0 tx=0x7f8620006df0).ready entity=mon.2 client_cookie=26ea8c90c5bb3e7c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:25.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.614+0000 7f862cf93700 1 -- 172.21.15.187:0/690823499 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620009680 con 0x7f8628066f70 2020-11-20T12:14:25.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.614+0000 7f862cf93700 1 -- 172.21.15.187:0/690823499 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86200097e0 con 0x7f8628066f70 2020-11-20T12:14:25.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.614+0000 7f862cf93700 1 -- 172.21.15.187:0/690823499 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620009b60 con 0x7f8628066f70 2020-11-20T12:14:25.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.614+0000 7f862ef97700 1 -- 172.21.15.187:0/690823499 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8628066f70 msgr2=0x7f86280b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.614+0000 7f862ef97700 1 --2- 172.21.15.187:0/690823499 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8628066f70 0x7f86280b01a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f862000b9d0 tx=0x7f8620006df0).stop 2020-11-20T12:14:25.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.614+0000 7f862ef97700 1 -- 172.21.15.187:0/690823499 shutdown_connections 2020-11-20T12:14:25.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.614+0000 7f862ef97700 1 --2- 172.21.15.187:0/690823499 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8628066f70 0x7f86280b01a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.614+0000 7f862ef97700 1 -- 172.21.15.187:0/690823499 >> 172.21.15.187:0/690823499 conn(0x7f86280bab30 msgr2=0x7f86280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:25.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.615+0000 7f862ef97700 1 -- 172.21.15.187:0/690823499 shutdown_connections 2020-11-20T12:14:25.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.615+0000 7f862ef97700 1 -- 172.21.15.187:0/690823499 wait complete. 2020-11-20T12:14:25.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.616+0000 7f862ef97700 1 Processor -- start 2020-11-20T12:14:25.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.617+0000 7f862ef97700 1 -- start start 2020-11-20T12:14:25.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862ef97700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8628066f70 0x7f862809c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862ef97700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86280b6650 0x7f862809dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862ef97700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86281484d0 0x7f8628142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862ef97700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86281378f0 con 0x7f8628066f70 2020-11-20T12:14:25.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862ef97700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86280b33a0 con 0x7f86280b6650 2020-11-20T12:14:25.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862ef97700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86280b3640 con 0x7f86281484d0 2020-11-20T12:14:25.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862d794700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86280b6650 0x7f862809dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862d794700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86280b6650 0x7f862809dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48902/0 (socket says 172.21.15.187:48902) 2020-11-20T12:14:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862d794700 1 -- 172.21.15.187:0/2524719994 learned_addr learned my addr 172.21.15.187:0/2524719994 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.618+0000 7f862df95700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8628066f70 0x7f862809c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f862d794700 1 -- 172.21.15.187:0/2524719994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86281484d0 msgr2=0x7f8628142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f862d794700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86281484d0 0x7f8628142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:25.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f862d794700 1 -- 172.21.15.187:0/2524719994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8628066f70 msgr2=0x7f862809c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f862d794700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8628066f70 0x7f862809c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f862d794700 1 -- 172.21.15.187:0/2524719994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280b38e0 con 0x7f86280b6650 2020-11-20T12:14:25.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f862d794700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86280b6650 0x7f862809dae0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7f861800c6a0 tx=0x7f8618007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:25.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f8615ffb700 1 -- 172.21.15.187:0/2524719994 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8618007320 con 0x7f86280b6650 2020-11-20T12:14:25.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f8615ffb700 1 -- 172.21.15.187:0/2524719994 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8618007480 con 0x7f86280b6650 2020-11-20T12:14:25.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862809d360 con 0x7f86280b6650 2020-11-20T12:14:25.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f8615ffb700 1 -- 172.21.15.187:0/2524719994 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8618005820 con 0x7f86280b6650 2020-11-20T12:14:25.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.619+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86280b2f00 con 0x7f86280b6650 2020-11-20T12:14:25.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.620+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86280b31e0 con 0x7f86280b6650 2020-11-20T12:14:25.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.621+0000 7f8615ffb700 1 -- 172.21.15.187:0/2524719994 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8618010030 con 0x7f86280b6650 2020-11-20T12:14:25.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.622+0000 7f8615ffb700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8600056d20 0x7f86000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:25.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.622+0000 7f8615ffb700 1 -- 172.21.15.187:0/2524719994 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f861806f9c0 con 0x7f86280b6650 2020-11-20T12:14:25.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.622+0000 7f862df95700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8600056d20 0x7f86000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:25.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.623+0000 7f862df95700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8600056d20 0x7f86000591e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f8624005f70 tx=0x7f8624007c70).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:25.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.626+0000 7f8615ffb700 1 -- 172.21.15.187:0/2524719994 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f86180450e0 con 0x7f86280b6650 2020-11-20T12:14:25.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.812+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f86280b3a70 con 0x7f86280b6650 2020-11-20T12:14:25.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.813+0000 7f8615ffb700 1 -- 172.21.15.187:0/2524719994 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v93) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8618048700 con 0x7f86280b6650 2020-11-20T12:14:25.817 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:14:25.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.815+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8600056d20 msgr2=0x7f86000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.815+0000 7f862ef97700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8600056d20 0x7f86000591e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f8624005f70 tx=0x7f8624007c70).stop 2020-11-20T12:14:25.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.815+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86280b6650 msgr2=0x7f862809dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:25.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.815+0000 7f862ef97700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86280b6650 0x7f862809dae0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7f861800c6a0 tx=0x7f8618007ac0).stop 2020-11-20T12:14:25.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.816+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 shutdown_connections 2020-11-20T12:14:25.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.816+0000 7f862ef97700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8600056d20 0x7f86000591e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.816+0000 7f862ef97700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8628066f70 0x7f862809c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.816+0000 7f862ef97700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86281484d0 0x7f8628142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:25.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.816+0000 7f862ef97700 1 --2- 172.21.15.187:0/2524719994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86280b6650 0x7f862809dae0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:25.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.816+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 >> 172.21.15.187:0/2524719994 conn(0x7f86280bab30 msgr2=0x7f8628144f50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:25.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.816+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 shutdown_connections 2020-11-20T12:14:25.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:25.816+0000 7f862ef97700 1 -- 172.21.15.187:0/2524719994 wait complete. 2020-11-20T12:14:25.841 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:14:26.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.096+0000 7fbe8490a700 1 Processor -- start 2020-11-20T12:14:26.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.096+0000 7fbe8490a700 1 -- start start 2020-11-20T12:14:26.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.098+0000 7fbe8490a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbe800b1380 0x7fbe80150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.098+0000 7fbe8490a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b1760 0x7fbe80065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.098+0000 7fbe8490a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe80150390 con 0x7fbe80065680 2020-11-20T12:14:26.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.098+0000 7fbe8490a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe80150630 con 0x7fbe800b1380 2020-11-20T12:14:26.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.098+0000 7fbe8490a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe80068510 con 0x7fbe800b1760 2020-11-20T12:14:26.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.098+0000 7fbe7ed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbe800b1380 0x7fbe80150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7ed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbe800b1380 0x7fbe80150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48908/0 (socket says 172.21.15.187:48908) 2020-11-20T12:14:26.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7ed9d700 1 -- 172.21.15.187:0/1840483061 learned_addr learned my addr 172.21.15.187:0/1840483061 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:26.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7f59e700 1 --1- 172.21.15.187:0/1840483061 >> v1:172.21.15.5:6789/0 conn(0x7fbe80065680 0x7fbe80061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:39028/0 (socket says 172.21.15.187:39028) 2020-11-20T12:14:26.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7659c700 1 --2- 172.21.15.187:0/1840483061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b1760 0x7fbe80065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7ed9d700 1 -- 172.21.15.187:0/1840483061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b1760 msgr2=0x7fbe80065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:26.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7ed9d700 1 --2- 172.21.15.187:0/1840483061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b1760 0x7fbe80065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7ed9d700 1 -- 172.21.15.187:0/1840483061 >> v1:172.21.15.5:6789/0 conn(0x7fbe80065680 legacy=0x7fbe80061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7ed9d700 1 -- 172.21.15.187:0/1840483061 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe80150fc0 con 0x7fbe800b1380 2020-11-20T12:14:26.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.099+0000 7fbe7659c700 1 --2- 172.21.15.187:0/1840483061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b1760 0x7fbe80065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:26.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.100+0000 7fbe7ed9d700 1 --2- 172.21.15.187:0/1840483061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1380 0x7fbe80150ad0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7fbe70006fd0 tx=0x7fbe70006eb0).ready entity=mon.1 client_cookie=94d3bdab5f467da server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:26.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.100+0000 7fbe7e59c700 1 -- 172.21.15.187:0/1840483061 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700098d0 con 0x7fbe800b1380 2020-11-20T12:14:26.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.100+0000 7fbe7e59c700 1 -- 172.21.15.187:0/1840483061 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70009a30 con 0x7fbe800b1380 2020-11-20T12:14:26.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.100+0000 7fbe7e59c700 1 -- 172.21.15.187:0/1840483061 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70009db0 con 0x7fbe800b1380 2020-11-20T12:14:26.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.100+0000 7fbe8490a700 1 -- 172.21.15.187:0/1840483061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1380 msgr2=0x7fbe80150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.100+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1840483061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1380 0x7fbe80150ad0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7fbe70006fd0 tx=0x7fbe70006eb0).stop 2020-11-20T12:14:26.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.101+0000 7fbe8490a700 1 -- 172.21.15.187:0/1840483061 shutdown_connections 2020-11-20T12:14:26.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.101+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1840483061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1380 0x7fbe80150ad0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.101+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1840483061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b1760 0x7fbe80065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.101+0000 7fbe8490a700 1 -- 172.21.15.187:0/1840483061 >> 172.21.15.187:0/1840483061 conn(0x7fbe800bab30 msgr2=0x7fbe800baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:26.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.101+0000 7fbe8490a700 1 -- 172.21.15.187:0/1840483061 shutdown_connections 2020-11-20T12:14:26.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.101+0000 7fbe8490a700 1 -- 172.21.15.187:0/1840483061 wait complete. 2020-11-20T12:14:26.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.102+0000 7fbe8490a700 1 Processor -- start 2020-11-20T12:14:26.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.102+0000 7fbe8490a700 1 -- start start 2020-11-20T12:14:26.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.103+0000 7fbe8490a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1760 0x7fbe800b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.103+0000 7fbe8490a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe800a09d0 0x7fbe801453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.103+0000 7fbe8490a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b7a90 0x7fbe801458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.103+0000 7fbe8490a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8009ecb0 con 0x7fbe800a09d0 2020-11-20T12:14:26.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.103+0000 7fbe8490a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8009dae0 con 0x7fbe800b1760 2020-11-20T12:14:26.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.103+0000 7fbe8490a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8009e770 con 0x7fbe800b7a90 2020-11-20T12:14:26.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.103+0000 7fbe7ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1760 0x7fbe800b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1760 0x7fbe800b3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48914/0 (socket says 172.21.15.187:48914) 2020-11-20T12:14:26.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7ed9d700 1 -- 172.21.15.187:0/1696931592 learned_addr learned my addr 172.21.15.187:0/1696931592 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:26.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7ed9d700 1 -- 172.21.15.187:0/1696931592 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b7a90 msgr2=0x7fbe801458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7ed9d700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b7a90 0x7fbe801458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:26.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7ed9d700 1 -- 172.21.15.187:0/1696931592 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe800a09d0 msgr2=0x7fbe801453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7659c700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe800a09d0 0x7fbe801453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7ed9d700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe800a09d0 0x7fbe801453e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7ed9d700 1 -- 172.21.15.187:0/1696931592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8009ea10 con 0x7fbe800b1760 2020-11-20T12:14:26.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe7ed9d700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1760 0x7fbe800b3e50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7fbe8014dd90 tx=0x7fbe70015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:26.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.104+0000 7fbe777fe700 1 -- 172.21.15.187:0/1696931592 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70011e30 con 0x7fbe800b1760 2020-11-20T12:14:26.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.105+0000 7fbe777fe700 1 -- 172.21.15.187:0/1696931592 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7001e460 con 0x7fbe800b1760 2020-11-20T12:14:26.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.105+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8009dcd0 con 0x7fbe800b1760 2020-11-20T12:14:26.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.105+0000 7fbe777fe700 1 -- 172.21.15.187:0/1696931592 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7001e800 con 0x7fbe800b1760 2020-11-20T12:14:26.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.105+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe80145dc0 con 0x7fbe800b1760 2020-11-20T12:14:26.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.106+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8009de60 con 0x7fbe800b1760 2020-11-20T12:14:26.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.106+0000 7fbe777fe700 1 -- 172.21.15.187:0/1696931592 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbe70025030 con 0x7fbe800b1760 2020-11-20T12:14:26.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.107+0000 7fbe777fe700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbe5c056d20 0x7fbe5c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.107+0000 7fbe777fe700 1 -- 172.21.15.187:0/1696931592 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fbe70071570 con 0x7fbe800b1760 2020-11-20T12:14:26.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.107+0000 7fbe7659c700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbe5c056d20 0x7fbe5c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.108+0000 7fbe7659c700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbe5c056d20 0x7fbe5c0591e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fbe68006fd0 tx=0x7fbe68008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:26.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.111+0000 7fbe777fe700 1 -- 172.21.15.187:0/1696931592 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbe70046c90 con 0x7fbe800b1760 2020-11-20T12:14:26.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.294+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbe80061a20 con 0x7fbe800b1760 2020-11-20T12:14:26.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.297+0000 7fbe777fe700 1 -- 172.21.15.187:0/1696931592 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v93) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbe70023070 con 0x7fbe800b1760 2020-11-20T12:14:26.301 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:26.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbe5c056d20 msgr2=0x7fbe5c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbe5c056d20 0x7fbe5c0591e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fbe68006fd0 tx=0x7fbe68008040).stop 2020-11-20T12:14:26.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1760 msgr2=0x7fbe800b3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1760 0x7fbe800b3e50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7fbe8014dd90 tx=0x7fbe70015000).stop 2020-11-20T12:14:26.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 shutdown_connections 2020-11-20T12:14:26.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbe5c056d20 0x7fbe5c0591e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe800b7a90 0x7fbe801458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:26.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe800b1760 0x7fbe800b3e50 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 --2- 172.21.15.187:0/1696931592 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe800a09d0 0x7fbe801453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 >> 172.21.15.187:0/1696931592 conn(0x7fbe800bab30 msgr2=0x7fbe8014b630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:26.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 shutdown_connections 2020-11-20T12:14:26.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.300+0000 7fbe8490a700 1 -- 172.21.15.187:0/1696931592 wait complete. 2020-11-20T12:14:26.328 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:14:26.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.572+0000 7fd94af33700 1 Processor -- start 2020-11-20T12:14:26.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.572+0000 7fd94af33700 1 -- start start 2020-11-20T12:14:26.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.574+0000 7fd94af33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd944065f90 0x7fd944065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.574+0000 7fd94af33700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9440b13e0 con 0x7fd944065190 2020-11-20T12:14:26.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.574+0000 7fd94af33700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd944062c10 con 0x7fd94409ca90 2020-11-20T12:14:26.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.574+0000 7fd94af33700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94414ab30 con 0x7fd944065f90 2020-11-20T12:14:26.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.574+0000 7fd94a732700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd94409ca90 0x7fd94409cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45530/0 (socket says 172.21.15.187:45530) 2020-11-20T12:14:26.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.574+0000 7fd94a732700 1 -- 172.21.15.187:0/3015548120 learned_addr learned my addr 172.21.15.187:0/3015548120 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:26.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.575+0000 7fd949730700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd944065f90 0x7fd944065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.575+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041939216 0 0) 0x7fd944062c10 con 0x7fd94409ca90 2020-11-20T12:14:26.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.575+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd92c003240 con 0x7fd94409ca90 2020-11-20T12:14:26.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.575+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2491107749 0 0) 0x7fd92c003240 con 0x7fd94409ca90 2020-11-20T12:14:26.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.575+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd944065f90 msgr2=0x7fd944065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:26.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.575+0000 7fd948f2f700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd944065f90 0x7fd944065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 >> v1:172.21.15.5:6789/0 conn(0x7fd944065190 legacy=0x7fd944143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9440b1640 con 0x7fd94409ca90 2020-11-20T12:14:26.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd93c001860 con 0x7fd94409ca90 2020-11-20T12:14:26.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 >> v1:172.21.15.187:6789/0 conn(0x7fd94409ca90 legacy=0x7fd94409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd92c003730 0x7fd92c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c0040b0 0x7fd92c004580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004bb0 0x7fd92c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94414ab30 con 0x7fd92c0040b0 2020-11-20T12:14:26.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c0051b0 con 0x7fd92c004bb0 2020-11-20T12:14:26.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.576+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c005370 con 0x7fd92c003730 2020-11-20T12:14:26.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.577+0000 7fd949f31700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd92c003730 0x7fd92c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.577+0000 7fd949730700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c0040b0 0x7fd92c004580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.577+0000 7fd94a732700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004bb0 0x7fd92c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd94a732700 1 -- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd92c003730 msgr2=0x7fd92c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:26.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd94a732700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd92c003730 0x7fd92c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd94a732700 1 -- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c0040b0 msgr2=0x7fd92c004580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:26.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd94a732700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c0040b0 0x7fd92c004580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd94a732700 1 -- 172.21.15.187:0/3015548120 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c003240 con 0x7fd92c004bb0 2020-11-20T12:14:26.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd94a732700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004bb0 0x7fd92c00b730 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7fd93c00c870 tx=0x7fd9440bb650).ready entity=mon.1 client_cookie=a73ba13a5ac0017e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:26.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c005b80 con 0x7fd92c004bb0 2020-11-20T12:14:26.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c005ce0 con 0x7fd92c004bb0 2020-11-20T12:14:26.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.578+0000 7fd948f2f700 1 -- 172.21.15.187:0/3015548120 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c007300 con 0x7fd92c004bb0 2020-11-20T12:14:26.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.579+0000 7fd94af33700 1 -- 172.21.15.187:0/3015548120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004bb0 msgr2=0x7fd92c00b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.579+0000 7fd94af33700 1 --2- 172.21.15.187:0/3015548120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004bb0 0x7fd92c00b730 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7fd93c00c870 tx=0x7fd9440bb650).stop 2020-11-20T12:14:26.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.579+0000 7fd94a732700 1 -- 172.21.15.187:0/3015548120 reap_dead start 2020-11-20T12:14:26.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.579+0000 7fd94af33700 1 -- 172.21.15.187:0/3015548120 shutdown_connections 2020-11-20T12:14:26.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.579+0000 7fd94af33700 1 -- 172.21.15.187:0/3015548120 >> 172.21.15.187:0/3015548120 conn(0x7fd9440bab30 msgr2=0x7fd9440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:26.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.579+0000 7fd94af33700 1 -- 172.21.15.187:0/3015548120 shutdown_connections 2020-11-20T12:14:26.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.579+0000 7fd94af33700 1 -- 172.21.15.187:0/3015548120 wait complete. 2020-11-20T12:14:26.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.581+0000 7fd94af33700 1 Processor -- start 2020-11-20T12:14:26.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.581+0000 7fd94af33700 1 -- start start 2020-11-20T12:14:26.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd94af33700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd944145dd0 0x7fd9440b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd94af33700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94413d710 0x7fd9440b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd94af33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9441461b0 0x7fd9440b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd94af33700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94413ce20 con 0x7fd944145dd0 2020-11-20T12:14:26.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd94af33700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94413ad10 con 0x7fd94413d710 2020-11-20T12:14:26.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd94af33700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94413d4f0 con 0x7fd9441461b0 2020-11-20T12:14:26.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd949730700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94413d710 0x7fd9440b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd949730700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94413d710 0x7fd9440b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48936/0 (socket says 172.21.15.187:48936) 2020-11-20T12:14:26.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.582+0000 7fd949730700 1 -- 172.21.15.187:0/2638725281 learned_addr learned my addr 172.21.15.187:0/2638725281 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:26.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd949f31700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd944145dd0 0x7fd9440b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd94a732700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9441461b0 0x7fd9440b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd949730700 1 -- 172.21.15.187:0/2638725281 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9441461b0 msgr2=0x7fd9440b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd949730700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9441461b0 0x7fd9440b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd949730700 1 -- 172.21.15.187:0/2638725281 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd944145dd0 msgr2=0x7fd9440b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd949730700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd944145dd0 0x7fd9440b40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd949730700 1 -- 172.21.15.187:0/2638725281 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9440b2850 con 0x7fd94413d710 2020-11-20T12:14:26.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd949730700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94413d710 0x7fd9440b2be0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7fd9340109c0 tx=0x7fd934008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:26.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd931ffb700 1 -- 172.21.15.187:0/2638725281 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd934018070 con 0x7fd94413d710 2020-11-20T12:14:26.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd931ffb700 1 -- 172.21.15.187:0/2638725281 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93400d9c0 con 0x7fd94413d710 2020-11-20T12:14:26.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94413d050 con 0x7fd94413d710 2020-11-20T12:14:26.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.583+0000 7fd931ffb700 1 -- 172.21.15.187:0/2638725281 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93400dd80 con 0x7fd94413d710 2020-11-20T12:14:26.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.584+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9440b4620 con 0x7fd94413d710 2020-11-20T12:14:26.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.585+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd944061a20 con 0x7fd94413d710 2020-11-20T12:14:26.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.585+0000 7fd931ffb700 1 -- 172.21.15.187:0/2638725281 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd934014680 con 0x7fd94413d710 2020-11-20T12:14:26.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.586+0000 7fd931ffb700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:26.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.586+0000 7fd931ffb700 1 -- 172.21.15.187:0/2638725281 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd93406ff50 con 0x7fd94413d710 2020-11-20T12:14:26.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.586+0000 7fd949f31700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:26.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.587+0000 7fd949f31700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fd940007b60 tx=0x7fd940009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:26.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.590+0000 7fd931ffb700 1 -- 172.21.15.187:0/2638725281 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd9340456d0 con 0x7fd94413d710 2020-11-20T12:14:26.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.781+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd94409d630 con 0x7fd94413d710 2020-11-20T12:14:26.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.782+0000 7fd931ffb700 1 -- 172.21.15.187:0/2638725281 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v93) v1 ==== 84+0+143 (secure 0 0 0) 0x7fd934048cf0 con 0x7fd94413d710 2020-11-20T12:14:26.786 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:26.786 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:14:26.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:26.787 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:14:26.787 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:14:26.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:14:26.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:26.788 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:14:26.788 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:14:26.788 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:26.788 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:14:26.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.784+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 msgr2=0x7fd91c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.784+0000 7fd94af33700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fd940007b60 tx=0x7fd940009040).stop 2020-11-20T12:14:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.784+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94413d710 msgr2=0x7fd9440b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94413d710 0x7fd9440b2be0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7fd9340109c0 tx=0x7fd934008040).stop 2020-11-20T12:14:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 shutdown_connections 2020-11-20T12:14:26.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd944145dd0 0x7fd9440b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9441461b0 0x7fd9440b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 --2- 172.21.15.187:0/2638725281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94413d710 0x7fd9440b2be0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 >> 172.21.15.187:0/2638725281 conn(0x7fd9440bab30 msgr2=0x7fd94409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:26.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 shutdown_connections 2020-11-20T12:14:26.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:26.785+0000 7fd94af33700 1 -- 172.21.15.187:0/2638725281 wait complete. 2020-11-20T12:14:26.814 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:14:26.815 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:14:27.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.067+0000 7fc8effff700 1 Processor -- start 2020-11-20T12:14:27.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.068+0000 7fc8effff700 1 -- start start 2020-11-20T12:14:27.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.069+0000 7fc8effff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc8f00b1380 0x7fc8f0150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.069+0000 7fc8effff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.069+0000 7fc8effff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f0150390 con 0x7fc8f0065680 2020-11-20T12:14:27.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.069+0000 7fc8effff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f0150630 con 0x7fc8f00b1380 2020-11-20T12:14:27.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.069+0000 7fc8effff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0068510 con 0x7fc8f00b1760 2020-11-20T12:14:27.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8ef7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc8f0065680 0x7fc8f0061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45550/0 (socket says 172.21.15.187:45550) 2020-11-20T12:14:27.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8ef7fe700 1 -- 172.21.15.187:0/4237373697 learned_addr learned my addr 172.21.15.187:0/4237373697 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:27.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8eeffd700 1 --2- 172.21.15.187:0/4237373697 >> v2:172.21.15.187:3300/0 conn(0x7fc8f00b1380 0x7fc8f0150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8ee7fc700 1 --2- 172.21.15.187:0/4237373697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8eeffd700 1 -- 172.21.15.187:0/4237373697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 msgr2=0x7fc8f0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:27.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8eeffd700 1 --2- 172.21.15.187:0/4237373697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f0065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8eeffd700 1 -- 172.21.15.187:0/4237373697 >> v1:172.21.15.187:6789/0 conn(0x7fc8f0065680 legacy=0x7fc8f0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8eeffd700 1 -- 172.21.15.187:0/4237373697 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f0150fc0 con 0x7fc8f00b1380 2020-11-20T12:14:27.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8ee7fc700 1 --2- 172.21.15.187:0/4237373697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f0065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:27.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.070+0000 7fc8edffb700 1 -- 172.21.15.187:0/4237373697 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441815543 0 0) 0x7fc8f0150390 con 0x7fc8f0065680 2020-11-20T12:14:27.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8eeffd700 1 --2- 172.21.15.187:0/4237373697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f00b1380 0x7fc8f0150ad0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7fc8e000b9a0 tx=0x7fc8e0006df0).ready entity=mon.1 client_cookie=f51da523061672ae server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:27.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8edffb700 1 -- 172.21.15.187:0/4237373697 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e00097b0 con 0x7fc8f00b1380 2020-11-20T12:14:27.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8edffb700 1 -- 172.21.15.187:0/4237373697 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e0009910 con 0x7fc8f00b1380 2020-11-20T12:14:27.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8edffb700 1 -- 172.21.15.187:0/4237373697 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0009c90 con 0x7fc8f00b1380 2020-11-20T12:14:27.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8effff700 1 -- 172.21.15.187:0/4237373697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f00b1380 msgr2=0x7fc8f0150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8effff700 1 --2- 172.21.15.187:0/4237373697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f00b1380 0x7fc8f0150ad0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7fc8e000b9a0 tx=0x7fc8e0006df0).stop 2020-11-20T12:14:27.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8effff700 1 -- 172.21.15.187:0/4237373697 shutdown_connections 2020-11-20T12:14:27.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8effff700 1 --2- 172.21.15.187:0/4237373697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f00b1380 0x7fc8f0150ad0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8effff700 1 --2- 172.21.15.187:0/4237373697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f0065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.071+0000 7fc8effff700 1 -- 172.21.15.187:0/4237373697 >> 172.21.15.187:0/4237373697 conn(0x7fc8f00bab30 msgr2=0x7fc8f00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:27.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.072+0000 7fc8effff700 1 -- 172.21.15.187:0/4237373697 shutdown_connections 2020-11-20T12:14:27.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.072+0000 7fc8effff700 1 -- 172.21.15.187:0/4237373697 wait complete. 2020-11-20T12:14:27.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.073+0000 7fc8effff700 1 Processor -- start 2020-11-20T12:14:27.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.073+0000 7fc8effff700 1 -- start start 2020-11-20T12:14:27.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.074+0000 7fc8effff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8f00b1380 0x7fc8f0148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.074+0000 7fc8effff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f015f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.074+0000 7fc8effff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f0149450 0x7fc8f015fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.074+0000 7fc8effff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00b2b10 con 0x7fc8f00b1380 2020-11-20T12:14:27.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.074+0000 7fc8effff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f009dbc0 con 0x7fc8f0149450 2020-11-20T12:14:27.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.074+0000 7fc8effff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f015d420 con 0x7fc8f00b1760 2020-11-20T12:14:27.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ef7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f0149450 0x7fc8f015fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ef7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f0149450 0x7fc8f015fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48952/0 (socket says 172.21.15.187:48952) 2020-11-20T12:14:27.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ef7fe700 1 -- 172.21.15.187:0/3205675432 learned_addr learned my addr 172.21.15.187:0/3205675432 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:27.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ee7fc700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f015f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ef7fe700 1 -- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 msgr2=0x7fc8f015f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8eeffd700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8f00b1380 0x7fc8f0148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ef7fe700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f015f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ef7fe700 1 -- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8f00b1380 msgr2=0x7fc8f0148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ef7fe700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8f00b1380 0x7fc8f0148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ef7fe700 1 -- 172.21.15.187:0/3205675432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f009cad0 con 0x7fc8f0149450 2020-11-20T12:14:27.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.075+0000 7fc8ee7fc700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f015f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:27.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.076+0000 7fc8ef7fe700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f0149450 0x7fc8f015fed0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7fc8e800cb10 tx=0x7fc8e8007d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:27.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.076+0000 7fc8d6ffd700 1 -- 172.21.15.187:0/3205675432 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8005b80 con 0x7fc8f0149450 2020-11-20T12:14:27.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.076+0000 7fc8d6ffd700 1 -- 172.21.15.187:0/3205675432 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e8005ce0 con 0x7fc8f0149450 2020-11-20T12:14:27.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.076+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f00b2d40 con 0x7fc8f0149450 2020-11-20T12:14:27.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.076+0000 7fc8d6ffd700 1 -- 172.21.15.187:0/3205675432 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8007320 con 0x7fc8f0149450 2020-11-20T12:14:27.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.076+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f009c550 con 0x7fc8f0149450 2020-11-20T12:14:27.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.077+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f00b2ed0 con 0x7fc8f0149450 2020-11-20T12:14:27.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.078+0000 7fc8d6ffd700 1 -- 172.21.15.187:0/3205675432 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc8e8010030 con 0x7fc8f0149450 2020-11-20T12:14:27.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.078+0000 7fc8d6ffd700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc8cc056d20 0x7fc8cc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.079+0000 7fc8d6ffd700 1 -- 172.21.15.187:0/3205675432 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc8e806fc90 con 0x7fc8f0149450 2020-11-20T12:14:27.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.079+0000 7fc8eeffd700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc8cc056d20 0x7fc8cc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.079+0000 7fc8eeffd700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc8cc056d20 0x7fc8cc0591e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7fc8e0006e30 tx=0x7fc8e0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:27.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.083+0000 7fc8d6ffd700 1 -- 172.21.15.187:0/3205675432 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc8e8045410 con 0x7fc8f0149450 2020-11-20T12:14:27.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.268+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fc8f0061a20 con 0x7fc8f0149450 2020-11-20T12:14:27.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.407+0000 7fc8d6ffd700 1 -- 172.21.15.187:0/3205675432 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v94) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc8e8048a30 con 0x7fc8f0149450 2020-11-20T12:14:27.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.413+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc8cc056d20 msgr2=0x7fc8cc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.413+0000 7fc8effff700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc8cc056d20 0x7fc8cc0591e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7fc8e0006e30 tx=0x7fc8e0015000).stop 2020-11-20T12:14:27.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.413+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f0149450 msgr2=0x7fc8f015fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.413+0000 7fc8effff700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f0149450 0x7fc8f015fed0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7fc8e800cb10 tx=0x7fc8e8007d80).stop 2020-11-20T12:14:27.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.414+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 shutdown_connections 2020-11-20T12:14:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.414+0000 7fc8effff700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc8cc056d20 0x7fc8cc0591e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.414+0000 7fc8effff700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8f00b1380 0x7fc8f0148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.414+0000 7fc8effff700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f0149450 0x7fc8f015fed0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.414+0000 7fc8effff700 1 --2- 172.21.15.187:0/3205675432 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc8f00b1760 0x7fc8f015f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.414+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 >> 172.21.15.187:0/3205675432 conn(0x7fc8f00bab30 msgr2=0x7fc8f014f250 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:27.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.415+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 shutdown_connections 2020-11-20T12:14:27.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.417+0000 7fc8effff700 1 -- 172.21.15.187:0/3205675432 wait complete. 2020-11-20T12:14:27.422 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:14:27.448 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:14:27.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.697+0000 7f900f435700 1 Processor -- start 2020-11-20T12:14:27.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.697+0000 7f900f435700 1 -- start start 2020-11-20T12:14:27.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.698+0000 7f900f435700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f900814d270 0x7f90081573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.698+0000 7f900f435700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90080658a0 con 0x7f90080b1380 2020-11-20T12:14:27.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.698+0000 7f900f435700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9008150390 con 0x7f90080b1760 2020-11-20T12:14:27.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.698+0000 7f900f435700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90081511e0 con 0x7f900814d270 2020-11-20T12:14:27.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.698+0000 7f900dc32700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f90080b1760 0x7f9008065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45562/0 (socket says 172.21.15.187:45562) 2020-11-20T12:14:27.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.698+0000 7f900dc32700 1 -- 172.21.15.187:0/2312907110 learned_addr learned my addr 172.21.15.187:0/2312907110 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:27.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188736720 0 0) 0x7f9008150390 con 0x7f90080b1760 2020-11-20T12:14:27.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0003240 con 0x7f90080b1760 2020-11-20T12:14:27.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900ec34700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f900814d270 0x7f90081573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1380665438 0 0) 0x7f8ff0003240 con 0x7f90080b1760 2020-11-20T12:14:27.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f900814d270 msgr2=0x7f90081573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:27.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f900814d270 0x7f90081573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:27.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 >> v1:172.21.15.5:6789/0 conn(0x7f90080b1380 legacy=0x7f9008150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:27.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9008150630 con 0x7f90080b1760 2020-11-20T12:14:27.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8ff80014e0 con 0x7f90080b1760 2020-11-20T12:14:27.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 >> v1:172.21.15.187:6789/0 conn(0x7f90080b1760 legacy=0x7f9008065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.699+0000 7f900d431700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f900814d270 0x7f8ff00037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900d431700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ff0003cf0 0x7f8ff00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900d431700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ff00049d0 0x7f8ff0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90081511e0 con 0x7f8ff0003cf0 2020-11-20T12:14:27.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff0004280 con 0x7f8ff00049d0 2020-11-20T12:14:27.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff000b730 con 0x7f900814d270 2020-11-20T12:14:27.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900ec34700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ff00049d0 0x7f8ff0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900ec34700 1 -- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f900814d270 msgr2=0x7f8ff00037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:27.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900ec34700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f900814d270 0x7f8ff00037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:27.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900ec34700 1 -- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ff0003cf0 msgr2=0x7f8ff00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:27.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900ec34700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ff0003cf0 0x7f8ff00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:27.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900ec34700 1 -- 172.21.15.187:0/2312907110 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff0003240 con 0x7f8ff00049d0 2020-11-20T12:14:27.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.700+0000 7f900ec34700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ff00049d0 0x7f8ff0004e50 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7f9000009d80 tx=0x7f9000008040).ready entity=mon.1 client_cookie=21df1d25f6694361 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:27.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900000c070 con 0x7f8ff00049d0 2020-11-20T12:14:27.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900000ea80 con 0x7f8ff00049d0 2020-11-20T12:14:27.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900d431700 1 -- 172.21.15.187:0/2312907110 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900000ee40 con 0x7f8ff00049d0 2020-11-20T12:14:27.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900f435700 1 -- 172.21.15.187:0/2312907110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ff00049d0 msgr2=0x7f8ff0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900f435700 1 --2- 172.21.15.187:0/2312907110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ff00049d0 0x7f8ff0004e50 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7f9000009d80 tx=0x7f9000008040).stop 2020-11-20T12:14:27.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900ec34700 1 -- 172.21.15.187:0/2312907110 reap_dead start 2020-11-20T12:14:27.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900f435700 1 -- 172.21.15.187:0/2312907110 shutdown_connections 2020-11-20T12:14:27.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900f435700 1 -- 172.21.15.187:0/2312907110 >> 172.21.15.187:0/2312907110 conn(0x7f90080bab30 msgr2=0x7f90080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:27.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.701+0000 7f900f435700 1 -- 172.21.15.187:0/2312907110 shutdown_connections 2020-11-20T12:14:27.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.702+0000 7f900f435700 1 -- 172.21.15.187:0/2312907110 wait complete. 2020-11-20T12:14:27.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.703+0000 7f900f435700 1 Processor -- start 2020-11-20T12:14:27.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.703+0000 7f900f435700 1 -- start start 2020-11-20T12:14:27.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.704+0000 7f900f435700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90081524b0 0x7f90080b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.704+0000 7f900f435700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9008152890 0x7f90080b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.704+0000 7f900f435700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90080b2030 0x7f90080b2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.704+0000 7f900f435700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900814b1a0 con 0x7f90080b2030 2020-11-20T12:14:27.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.704+0000 7f900f435700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90080b2980 con 0x7f9008152890 2020-11-20T12:14:27.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.704+0000 7f900f435700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900814b3d0 con 0x7f90081524b0 2020-11-20T12:14:27.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9008152890 0x7f90080b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9008152890 0x7f90080b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48972/0 (socket says 172.21.15.187:48972) 2020-11-20T12:14:27.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 -- 172.21.15.187:0/1175795654 learned_addr learned my addr 172.21.15.187:0/1175795654 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:27.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 -- 172.21.15.187:0/1175795654 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90081524b0 msgr2=0x7f90080b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90081524b0 0x7f90080b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:27.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 -- 172.21.15.187:0/1175795654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90080b2030 msgr2=0x7f90080b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:27.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90080b2030 0x7f90080b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:27.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 -- 172.21.15.187:0/1175795654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90080b2d60 con 0x7f9008152890 2020-11-20T12:14:27.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f900dc32700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9008152890 0x7f90080b1b40 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f8ff800ca70 tx=0x7f8ff8000d90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:27.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f8ff67fc700 1 -- 172.21.15.187:0/1175795654 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8007170 con 0x7f9008152890 2020-11-20T12:14:27.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f8ff67fc700 1 -- 172.21.15.187:0/1175795654 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff8007310 con 0x7f9008152890 2020-11-20T12:14:27.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.705+0000 7f8ff67fc700 1 -- 172.21.15.187:0/1175795654 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff800aa40 con 0x7f9008152890 2020-11-20T12:14:27.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.706+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900809d5b0 con 0x7f9008152890 2020-11-20T12:14:27.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.706+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9008149730 con 0x7f9008152890 2020-11-20T12:14:27.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.707+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90080b3120 con 0x7f9008152890 2020-11-20T12:14:27.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.707+0000 7f8ff67fc700 1 -- 172.21.15.187:0/1175795654 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8ff8010070 con 0x7f9008152890 2020-11-20T12:14:27.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.708+0000 7f8ff67fc700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fe0056d20 0x7f8fe00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:27.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.708+0000 7f8ff67fc700 1 -- 172.21.15.187:0/1175795654 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f8ff806f920 con 0x7f9008152890 2020-11-20T12:14:27.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.708+0000 7f900e433700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fe0056d20 0x7f8fe00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:27.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.709+0000 7f900e433700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fe0056d20 0x7f8fe00591e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f90080b4c60 tx=0x7f90040069b0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:27.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.712+0000 7f8ff67fc700 1 -- 172.21.15.187:0/1175795654 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8ff8044e30 con 0x7f9008152890 2020-11-20T12:14:27.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:27.881+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f90080b2ef0 con 0x7f9008152890 2020-11-20T12:14:28.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.666+0000 7f8ff67fc700 1 -- 172.21.15.187:0/1175795654 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v96) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8ff8048450 con 0x7f9008152890 2020-11-20T12:14:28.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.670+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fe0056d20 msgr2=0x7f8fe00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:28.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.670+0000 7f900f435700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fe0056d20 0x7f8fe00591e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f90080b4c60 tx=0x7f90040069b0).stop 2020-11-20T12:14:28.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.670+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9008152890 msgr2=0x7f90080b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:28.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.670+0000 7f900f435700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9008152890 0x7f90080b1b40 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f8ff800ca70 tx=0x7f8ff8000d90).stop 2020-11-20T12:14:28.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.671+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 shutdown_connections 2020-11-20T12:14:28.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.671+0000 7f900f435700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fe0056d20 0x7f8fe00591e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:28.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.671+0000 7f900f435700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90080b2030 0x7f90080b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:28.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.671+0000 7f900f435700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f90081524b0 0x7f90080b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:28.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.671+0000 7f900f435700 1 --2- 172.21.15.187:0/1175795654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9008152890 0x7f90080b1b40 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:28.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.671+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 >> 172.21.15.187:0/1175795654 conn(0x7f90080bab30 msgr2=0x7f90080afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:28.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.671+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 shutdown_connections 2020-11-20T12:14:28.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.672+0000 7f900f435700 1 -- 172.21.15.187:0/1175795654 wait complete. 2020-11-20T12:14:28.680 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:14:28.708 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:14:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.960+0000 7f103dc04700 1 Processor -- start 2020-11-20T12:14:29.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.960+0000 7f103dc04700 1 -- start start 2020-11-20T12:14:29.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103dc04700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f10381484d0 0x7f1038061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103dc04700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10380622c0 0x7f10380b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103dc04700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1038066f70 0x7f10380b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103dc04700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1038067730 con 0x7f1038066f70 2020-11-20T12:14:29.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103dc04700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10380b1300 con 0x7f10381484d0 2020-11-20T12:14:29.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103dc04700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10380b15a0 con 0x7f10380622c0 2020-11-20T12:14:29.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103cc02700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f10381484d0 0x7f1038061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103cc02700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f10381484d0 0x7f1038061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48978/0 (socket says 172.21.15.187:48978) 2020-11-20T12:14:29.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.962+0000 7f103cc02700 1 -- 172.21.15.187:0/3971213804 learned_addr learned my addr 172.21.15.187:0/3971213804 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:29.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f1037fff700 1 --2- 172.21.15.187:0/3971213804 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10380622c0 0x7f10380b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f103d403700 1 --2- 172.21.15.187:0/3971213804 >> v2:172.21.15.5:3300/0 conn(0x7f1038066f70 0x7f10380b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f103cc02700 1 -- 172.21.15.187:0/3971213804 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10380622c0 msgr2=0x7f10380b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:29.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f103cc02700 1 --2- 172.21.15.187:0/3971213804 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10380622c0 0x7f10380b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f103cc02700 1 -- 172.21.15.187:0/3971213804 >> v2:172.21.15.5:3300/0 conn(0x7f1038066f70 msgr2=0x7f10380b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:29.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f103cc02700 1 --2- 172.21.15.187:0/3971213804 >> v2:172.21.15.5:3300/0 conn(0x7f1038066f70 0x7f10380b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f103cc02700 1 -- 172.21.15.187:0/3971213804 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10380b1840 con 0x7f10381484d0 2020-11-20T12:14:29.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f1037fff700 1 --2- 172.21.15.187:0/3971213804 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10380622c0 0x7f10380b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:29.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.963+0000 7f103cc02700 1 --2- 172.21.15.187:0/3971213804 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10381484d0 0x7f1038061dd0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7f10380b85e0 tx=0x7f102c006eb0).ready entity=mon.1 client_cookie=184166587a4ad0c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:29.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.964+0000 7f10377fe700 1 -- 172.21.15.187:0/3971213804 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c0098d0 con 0x7f10381484d0 2020-11-20T12:14:29.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.964+0000 7f10377fe700 1 -- 172.21.15.187:0/3971213804 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c009a30 con 0x7f10381484d0 2020-11-20T12:14:29.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.964+0000 7f10377fe700 1 -- 172.21.15.187:0/3971213804 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c009db0 con 0x7f10381484d0 2020-11-20T12:14:29.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.964+0000 7f103dc04700 1 -- 172.21.15.187:0/3971213804 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10381484d0 msgr2=0x7f1038061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.964+0000 7f103dc04700 1 --2- 172.21.15.187:0/3971213804 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10381484d0 0x7f1038061dd0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7f10380b85e0 tx=0x7f102c006eb0).stop 2020-11-20T12:14:29.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.964+0000 7f103dc04700 1 -- 172.21.15.187:0/3971213804 shutdown_connections 2020-11-20T12:14:29.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.965+0000 7f103dc04700 1 --2- 172.21.15.187:0/3971213804 >> v2:172.21.15.5:3300/0 conn(0x7f1038066f70 0x7f10380b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.965+0000 7f103dc04700 1 --2- 172.21.15.187:0/3971213804 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10381484d0 0x7f1038061dd0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.965+0000 7f103dc04700 1 --2- 172.21.15.187:0/3971213804 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10380622c0 0x7f10380b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.965+0000 7f103dc04700 1 -- 172.21.15.187:0/3971213804 >> 172.21.15.187:0/3971213804 conn(0x7f10380bab30 msgr2=0x7f10380baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:29.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.965+0000 7f103dc04700 1 -- 172.21.15.187:0/3971213804 shutdown_connections 2020-11-20T12:14:29.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.965+0000 7f103dc04700 1 -- 172.21.15.187:0/3971213804 wait complete. 2020-11-20T12:14:29.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.966+0000 7f103dc04700 1 Processor -- start 2020-11-20T12:14:29.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.966+0000 7f103dc04700 1 -- start start 2020-11-20T12:14:29.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.967+0000 7f103dc04700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10380622c0 0x7f10380b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.967+0000 7f103dc04700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038066f70 0x7f10380b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.967+0000 7f103dc04700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10381484d0 0x7f103809cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.967+0000 7f103dc04700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103809b340 con 0x7f10380622c0 2020-11-20T12:14:29.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.967+0000 7f103dc04700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103809b570 con 0x7f1038066f70 2020-11-20T12:14:29.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.967+0000 7f103dc04700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103809b810 con 0x7f10381484d0 2020-11-20T12:14:29.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038066f70 0x7f10380b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038066f70 0x7f10380b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48986/0 (socket says 172.21.15.187:48986) 2020-11-20T12:14:29.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 -- 172.21.15.187:0/3837764552 learned_addr learned my addr 172.21.15.187:0/3837764552 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:29.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 -- 172.21.15.187:0/3837764552 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10381484d0 msgr2=0x7f103809cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10381484d0 0x7f103809cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:29.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 -- 172.21.15.187:0/3837764552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10380622c0 msgr2=0x7f10380b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10380622c0 0x7f10380b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:29.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 -- 172.21.15.187:0/3837764552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103809bab0 con 0x7f1038066f70 2020-11-20T12:14:29.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1037fff700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038066f70 0x7f10380b2e60 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f102400db60 tx=0x7f1024008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:29.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.968+0000 7f1034ff9700 1 -- 172.21.15.187:0/3837764552 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024009820 con 0x7f1038066f70 2020-11-20T12:14:29.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.969+0000 7f1034ff9700 1 -- 172.21.15.187:0/3837764552 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1024009980 con 0x7f1038066f70 2020-11-20T12:14:29.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.969+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103809d470 con 0x7f1038066f70 2020-11-20T12:14:29.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.969+0000 7f1034ff9700 1 -- 172.21.15.187:0/3837764552 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024009d00 con 0x7f1038066f70 2020-11-20T12:14:29.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.969+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1038145c40 con 0x7f1038066f70 2020-11-20T12:14:29.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.970+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103809bc40 con 0x7f1038066f70 2020-11-20T12:14:29.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.970+0000 7f1034ff9700 1 -- 172.21.15.187:0/3837764552 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1024013690 con 0x7f1038066f70 2020-11-20T12:14:29.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.971+0000 7f1034ff9700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1010056d20 0x7f10100591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.971+0000 7f1034ff9700 1 -- 172.21.15.187:0/3837764552 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f10240702c0 con 0x7f1038066f70 2020-11-20T12:14:29.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.971+0000 7f103cc02700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1010056d20 0x7f10100591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.972+0000 7f103cc02700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1010056d20 0x7f10100591e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f102c00b1b0 tx=0x7f102c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:29.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:28.975+0000 7f1034ff9700 1 -- 172.21.15.187:0/3837764552 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1024045780 con 0x7f1038066f70 2020-11-20T12:14:29.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.222+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1038061a20 con 0x7f1038066f70 2020-11-20T12:14:29.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.236+0000 7f1034ff9700 1 -- 172.21.15.187:0/3837764552 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 19 and data pool 20 v84) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1024009e60 con 0x7f1038066f70 2020-11-20T12:14:29.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1010056d20 msgr2=0x7f10100591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1010056d20 0x7f10100591e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f102c00b1b0 tx=0x7f102c015000).stop 2020-11-20T12:14:29.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038066f70 msgr2=0x7f10380b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038066f70 0x7f10380b2e60 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f102400db60 tx=0x7f1024008040).stop 2020-11-20T12:14:29.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 shutdown_connections 2020-11-20T12:14:29.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1010056d20 0x7f10100591e0 secure :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0x7f102c00b1b0 tx=0x7f102c015000).stop 2020-11-20T12:14:29.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10380622c0 0x7f10380b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:29.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10381484d0 0x7f103809cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:29.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 --2- 172.21.15.187:0/3837764552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038066f70 0x7f10380b2e60 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.240+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 >> 172.21.15.187:0/3837764552 conn(0x7f10380bab30 msgr2=0x7f10380b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:29.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.241+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 shutdown_connections 2020-11-20T12:14:29.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.241+0000 7f103dc04700 1 -- 172.21.15.187:0/3837764552 wait complete. 2020-11-20T12:14:29.248 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 19 and data pool 20 2020-11-20T12:14:29.273 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:29.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.489+0000 7f670c8b3700 1 Processor -- start 2020-11-20T12:14:29.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.490+0000 7f670c8b3700 1 -- start start 2020-11-20T12:14:29.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.491+0000 7f670c8b3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080654a0 0x7f67080b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.491+0000 7f670c8b3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6708065880 0x7f6708144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.491+0000 7f670c8b3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67080b08e0 con 0x7f67080b6c10 2020-11-20T12:14:29.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.491+0000 7f670c8b3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67080b0b80 con 0x7f6708065880 2020-11-20T12:14:29.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.491+0000 7f670c8b3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67080b05f0 con 0x7f67080654a0 2020-11-20T12:14:29.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.492+0000 7f670659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080654a0 0x7f67080b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.492+0000 7f66fdd9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6708065880 0x7f6708144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.492+0000 7f670659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080654a0 0x7f67080b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:49894/0 (socket says 172.21.15.187:49894) 2020-11-20T12:14:29.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.492+0000 7f670659c700 1 -- 172.21.15.187:0/2039682532 learned_addr learned my addr 172.21.15.187:0/2039682532 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:29.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.493+0000 7f670659c700 1 -- 172.21.15.187:0/2039682532 >> v2:172.21.15.5:3300/0 conn(0x7f6708065880 msgr2=0x7f6708144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:29.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.493+0000 7f670659c700 1 --2- 172.21.15.187:0/2039682532 >> v2:172.21.15.5:3300/0 conn(0x7f6708065880 0x7f6708144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.493+0000 7f670659c700 1 -- 172.21.15.187:0/2039682532 >> v1:172.21.15.5:6789/0 conn(0x7f67080b6c10 legacy=0x7f67080b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.493+0000 7f670659c700 1 -- 172.21.15.187:0/2039682532 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080676f0 con 0x7f67080654a0 2020-11-20T12:14:29.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.494+0000 7f670659c700 1 --2- 172.21.15.187:0/2039682532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080654a0 0x7f67080b1650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f66f8006fd0 tx=0x7f66f8006eb0).ready entity=mon.2 client_cookie=2350d24790f806b1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:29.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.494+0000 7f6705d9b700 1 -- 172.21.15.187:0/2039682532 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8009810 con 0x7f67080654a0 2020-11-20T12:14:29.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.494+0000 7f6705d9b700 1 -- 172.21.15.187:0/2039682532 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f8009970 con 0x7f67080654a0 2020-11-20T12:14:29.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.494+0000 7f6705d9b700 1 -- 172.21.15.187:0/2039682532 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8009cf0 con 0x7f67080654a0 2020-11-20T12:14:29.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.495+0000 7f670c8b3700 1 -- 172.21.15.187:0/2039682532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080654a0 msgr2=0x7f67080b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.495+0000 7f670c8b3700 1 --2- 172.21.15.187:0/2039682532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080654a0 0x7f67080b1650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f66f8006fd0 tx=0x7f66f8006eb0).stop 2020-11-20T12:14:29.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.495+0000 7f670c8b3700 1 -- 172.21.15.187:0/2039682532 shutdown_connections 2020-11-20T12:14:29.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.495+0000 7f670c8b3700 1 --2- 172.21.15.187:0/2039682532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080654a0 0x7f67080b1650 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.495+0000 7f670c8b3700 1 --2- 172.21.15.187:0/2039682532 >> v2:172.21.15.5:3300/0 conn(0x7f6708065880 0x7f6708144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.495+0000 7f670c8b3700 1 -- 172.21.15.187:0/2039682532 >> 172.21.15.187:0/2039682532 conn(0x7f67080bacd0 msgr2=0x7f67080bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:29.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.495+0000 7f670c8b3700 1 -- 172.21.15.187:0/2039682532 shutdown_connections 2020-11-20T12:14:29.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.495+0000 7f670c8b3700 1 -- 172.21.15.187:0/2039682532 wait complete. 2020-11-20T12:14:29.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.497+0000 7f670c8b3700 1 Processor -- start 2020-11-20T12:14:29.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.497+0000 7f670c8b3700 1 -- start start 2020-11-20T12:14:29.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f670c8b3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6708065880 0x7f670809f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f670c8b3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080b6c10 0x7f6708099600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f670c8b3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67081487d0 0x7f6708099af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f670c8b3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6708147c40 con 0x7f6708065880 2020-11-20T12:14:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f670c8b3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6708147e30 con 0x7f67081487d0 2020-11-20T12:14:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f670c8b3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6708148090 con 0x7f67080b6c10 2020-11-20T12:14:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f6706d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67081487d0 0x7f6708099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f6706d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67081487d0 0x7f6708099af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49014/0 (socket says 172.21.15.187:49014) 2020-11-20T12:14:29.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f6706d9d700 1 -- 172.21.15.187:0/502101523 learned_addr learned my addr 172.21.15.187:0/502101523 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:29.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f670659c700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6708065880 0x7f670809f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.498+0000 7f66fdd9b700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080b6c10 0x7f6708099600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f6706d9d700 1 -- 172.21.15.187:0/502101523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080b6c10 msgr2=0x7f6708099600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f6706d9d700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080b6c10 0x7f6708099600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f6706d9d700 1 -- 172.21.15.187:0/502101523 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6708065880 msgr2=0x7f670809f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f6706d9d700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6708065880 0x7f670809f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f6706d9d700 1 -- 172.21.15.187:0/502101523 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708148330 con 0x7f67081487d0 2020-11-20T12:14:29.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f6706d9d700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67081487d0 0x7f6708099af0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f670000ca90 tx=0x7f6700001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:29.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f66feffd700 1 -- 172.21.15.187:0/502101523 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670000a7c0 con 0x7f67081487d0 2020-11-20T12:14:29.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f66feffd700 1 -- 172.21.15.187:0/502101523 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670000a920 con 0x7f67081487d0 2020-11-20T12:14:29.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f66feffd700 1 -- 172.21.15.187:0/502101523 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670000acc0 con 0x7f67081487d0 2020-11-20T12:14:29.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670809f5c0 con 0x7f67081487d0 2020-11-20T12:14:29.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.499+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67081521f0 con 0x7f67081487d0 2020-11-20T12:14:29.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.500+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708063600 con 0x7f67081487d0 2020-11-20T12:14:29.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.501+0000 7f66feffd700 1 -- 172.21.15.187:0/502101523 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6700010070 con 0x7f67081487d0 2020-11-20T12:14:29.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.502+0000 7f66feffd700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f66e4056d20 0x7f66e40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.502+0000 7f66feffd700 1 -- 172.21.15.187:0/502101523 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f670006fe10 con 0x7f67081487d0 2020-11-20T12:14:29.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.502+0000 7f670659c700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f66e4056d20 0x7f66e40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.503+0000 7f670659c700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f66e4056d20 0x7f66e40591e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f67080b8ab0 tx=0x7f66f8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:29.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.506+0000 7f66feffd700 1 -- 172.21.15.187:0/502101523 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6700045260 con 0x7f67081487d0 2020-11-20T12:14:29.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.692+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f670809a5d0 con 0x7f67081487d0 2020-11-20T12:14:29.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.695+0000 7f66feffd700 1 -- 172.21.15.187:0/502101523 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16794 (secure 0 0 0) 0x7f6700048880 con 0x7f67081487d0 2020-11-20T12:14:29.699 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:29.700 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":97,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:14:29.225080+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:14:27.347251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2020-11-20T12:14:27.963950+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:29.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.697+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f66e4056d20 msgr2=0x7f66e40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.697+0000 7f670c8b3700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f66e4056d20 0x7f66e40591e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f67080b8ab0 tx=0x7f66f8015000).stop 2020-11-20T12:14:29.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.697+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67081487d0 msgr2=0x7f6708099af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.697+0000 7f670c8b3700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67081487d0 0x7f6708099af0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f670000ca90 tx=0x7f6700001360).stop 2020-11-20T12:14:29.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.697+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 shutdown_connections 2020-11-20T12:14:29.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.697+0000 7f670c8b3700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f66e4056d20 0x7f66e40591e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.697+0000 7f670c8b3700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6708065880 0x7f670809f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.697+0000 7f670c8b3700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67081487d0 0x7f6708099af0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.698+0000 7f670c8b3700 1 --2- 172.21.15.187:0/502101523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67080b6c10 0x7f6708099600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:29.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.698+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 >> 172.21.15.187:0/502101523 conn(0x7f67080bacd0 msgr2=0x7f67080afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:29.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.698+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 shutdown_connections 2020-11-20T12:14:29.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.698+0000 7f670c8b3700 1 -- 172.21.15.187:0/502101523 wait complete. 2020-11-20T12:14:29.724 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:29.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.986+0000 7f796b0b8700 1 Processor -- start 2020-11-20T12:14:29.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.987+0000 7f796b0b8700 1 -- start start 2020-11-20T12:14:29.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.988+0000 7f796b0b8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f79641482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:29.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.988+0000 7f796b0b8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7964144b80 con 0x7f79640b5f50 2020-11-20T12:14:29.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.988+0000 7f796b0b8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79641489e0 con 0x7f79640b57b0 2020-11-20T12:14:29.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.988+0000 7f796b0b8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79641487c0 con 0x7f7964068230 2020-11-20T12:14:29.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.988+0000 7f796a0b6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f79640b57b0 0x7f7964062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45628/0 (socket says 172.21.15.187:45628) 2020-11-20T12:14:29.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.989+0000 7f796a0b6700 1 -- 172.21.15.187:0/2919120940 learned_addr learned my addr 172.21.15.187:0/2919120940 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:29.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.989+0000 7f79698b5700 1 --2- 172.21.15.187:0/2919120940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f79641482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:29.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.989+0000 7f79690b4700 1 -- 172.21.15.187:0/2919120940 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769357889 0 0) 0x7f79641489e0 con 0x7f79640b57b0 2020-11-20T12:14:29.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.990+0000 7f79690b4700 1 -- 172.21.15.187:0/2919120940 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f794c004350 con 0x7f79640b57b0 2020-11-20T12:14:29.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.990+0000 7f79698b5700 1 -- 172.21.15.187:0/2919120940 >> v1:172.21.15.187:6789/0 conn(0x7f79640b57b0 legacy=0x7f7964062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.990+0000 7f79698b5700 1 -- 172.21.15.187:0/2919120940 >> v1:172.21.15.5:6789/0 conn(0x7f79640b5f50 legacy=0x7f79640b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:29.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.990+0000 7f79698b5700 1 -- 172.21.15.187:0/2919120940 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7964148c80 con 0x7f7964068230 2020-11-20T12:14:29.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.990+0000 7f79698b5700 1 --2- 172.21.15.187:0/2919120940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f79641482d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f796000b9d0 tx=0x7f7960006df0).ready entity=mon.2 client_cookie=722ffb400f721ddf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:29.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.991+0000 7f79690b4700 1 -- 172.21.15.187:0/2919120940 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79600097b0 con 0x7f7964068230 2020-11-20T12:14:30.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.991+0000 7f79690b4700 1 -- 172.21.15.187:0/2919120940 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7960009910 con 0x7f7964068230 2020-11-20T12:14:30.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.991+0000 7f79690b4700 1 -- 172.21.15.187:0/2919120940 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960009c90 con 0x7f7964068230 2020-11-20T12:14:30.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.991+0000 7f796b0b8700 1 -- 172.21.15.187:0/2919120940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 msgr2=0x7f79641482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:30.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.991+0000 7f796b0b8700 1 --2- 172.21.15.187:0/2919120940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f79641482d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f796000b9d0 tx=0x7f7960006df0).stop 2020-11-20T12:14:30.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.992+0000 7f796b0b8700 1 -- 172.21.15.187:0/2919120940 shutdown_connections 2020-11-20T12:14:30.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.992+0000 7f796b0b8700 1 --2- 172.21.15.187:0/2919120940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f79641482d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.992+0000 7f796b0b8700 1 -- 172.21.15.187:0/2919120940 >> 172.21.15.187:0/2919120940 conn(0x7f79640bab30 msgr2=0x7f79640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:30.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.992+0000 7f796b0b8700 1 -- 172.21.15.187:0/2919120940 shutdown_connections 2020-11-20T12:14:30.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.992+0000 7f796b0b8700 1 -- 172.21.15.187:0/2919120940 wait complete. 2020-11-20T12:14:30.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.994+0000 7f796b0b8700 1 Processor -- start 2020-11-20T12:14:30.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.994+0000 7f796b0b8700 1 -- start start 2020-11-20T12:14:30.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.995+0000 7f796b0b8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f7964144d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.995+0000 7f796b0b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79640b57b0 0x7f796409a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.995+0000 7f796b0b8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79640b5f50 0x7f79641456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.995+0000 7f796b0b8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796409d5b0 con 0x7f79640b5f50 2020-11-20T12:14:30.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.995+0000 7f796b0b8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79640a07d0 con 0x7f79640b57b0 2020-11-20T12:14:30.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.995+0000 7f796b0b8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79641453a0 con 0x7f7964068230 2020-11-20T12:14:30.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f79698b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79640b57b0 0x7f796409a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f79698b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79640b57b0 0x7f796409a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49026/0 (socket says 172.21.15.187:49026) 2020-11-20T12:14:30.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f79698b5700 1 -- 172.21.15.187:0/1849705682 learned_addr learned my addr 172.21.15.187:0/1849705682 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:30.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f796a0b6700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f7964144d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f796a8b7700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79640b5f50 0x7f79641456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f79698b5700 1 -- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 msgr2=0x7f7964144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:30.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f79698b5700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f7964144d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f79698b5700 1 -- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79640b5f50 msgr2=0x7f79641456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:30.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f79698b5700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79640b5f50 0x7f79641456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f79698b5700 1 -- 172.21.15.187:0/1849705682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796409d110 con 0x7f79640b57b0 2020-11-20T12:14:30.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.996+0000 7f796a8b7700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79640b5f50 0x7f79641456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:30.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.997+0000 7f79698b5700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79640b57b0 0x7f796409a3c0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7f7960006e30 tx=0x7f7960015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:30.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.997+0000 7f79567fc700 1 -- 172.21.15.187:0/1849705682 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960011e50 con 0x7f79640b57b0 2020-11-20T12:14:30.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.997+0000 7f79567fc700 1 -- 172.21.15.187:0/1849705682 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796001e460 con 0x7f79640b57b0 2020-11-20T12:14:30.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.997+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796409ce00 con 0x7f79640b57b0 2020-11-20T12:14:30.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.997+0000 7f79567fc700 1 -- 172.21.15.187:0/1849705682 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796001e7e0 con 0x7f79640b57b0 2020-11-20T12:14:30.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.997+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7964145b90 con 0x7f79640b57b0 2020-11-20T12:14:30.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.999+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7964061a20 con 0x7f79640b57b0 2020-11-20T12:14:30.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:29.999+0000 7f79567fc700 1 -- 172.21.15.187:0/1849705682 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7960025030 con 0x7f79640b57b0 2020-11-20T12:14:30.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.000+0000 7f79567fc700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f793c056d20 0x7f793c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.000+0000 7f79567fc700 1 -- 172.21.15.187:0/1849705682 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f79600717c0 con 0x7f79640b57b0 2020-11-20T12:14:30.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.000+0000 7f796a0b6700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f793c056d20 0x7f793c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.001+0000 7f796a0b6700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f793c056d20 0x7f793c0591e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f79580055a0 tx=0x7f7958009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:30.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.004+0000 7f79567fc700 1 -- 172.21.15.187:0/1849705682 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7960046bf0 con 0x7f79640b57b0 2020-11-20T12:14:30.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.188+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f796409d2a0 con 0x7f79640b57b0 2020-11-20T12:14:30.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.189+0000 7f79567fc700 1 -- 172.21.15.187:0/1849705682 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16794 (secure 0 0 0) 0x7f79600311b0 con 0x7f79640b57b0 2020-11-20T12:14:30.193 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:30.194 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":97,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:14:29.225080+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:14:27.347251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2020-11-20T12:14:27.963950+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:30.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.191+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f793c056d20 msgr2=0x7f793c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:30.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.191+0000 7f796b0b8700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f793c056d20 0x7f793c0591e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f79580055a0 tx=0x7f7958009040).stop 2020-11-20T12:14:30.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.191+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79640b57b0 msgr2=0x7f796409a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:30.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.191+0000 7f796b0b8700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79640b57b0 0x7f796409a3c0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7f7960006e30 tx=0x7f7960015000).stop 2020-11-20T12:14:30.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.192+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 shutdown_connections 2020-11-20T12:14:30.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.192+0000 7f796b0b8700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f793c056d20 0x7f793c0591e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.192+0000 7f796b0b8700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79640b5f50 0x7f79641456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.192+0000 7f796b0b8700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7964068230 0x7f7964144d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.192+0000 7f796b0b8700 1 --2- 172.21.15.187:0/1849705682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79640b57b0 0x7f796409a3c0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.192+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 >> 172.21.15.187:0/1849705682 conn(0x7f79640bab30 msgr2=0x7f79640b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:30.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.192+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 shutdown_connections 2020-11-20T12:14:30.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.192+0000 7f796b0b8700 1 -- 172.21.15.187:0/1849705682 wait complete. 2020-11-20T12:14:30.216 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:14:30.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.453+0000 7fd96e0ab700 1 Processor -- start 2020-11-20T12:14:30.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.453+0000 7fd96e0ab700 1 -- start start 2020-11-20T12:14:30.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.454+0000 7fd96e0ab700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9680b6650 0x7fd9680afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.454+0000 7fd96e0ab700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd968066f70 0x7fd9680aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.454+0000 7fd96e0ab700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9680b1300 con 0x7fd9681484d0 2020-11-20T12:14:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.454+0000 7fd96e0ab700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd968067730 con 0x7fd968066f70 2020-11-20T12:14:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.454+0000 7fd96e0ab700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9680b15a0 con 0x7fd9680b6650 2020-11-20T12:14:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.455+0000 7fd96c8a8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9680b6650 0x7fd9680afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.455+0000 7fd96c8a8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9680b6650 0x7fd9680afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49032/0 (socket says 172.21.15.187:49032) 2020-11-20T12:14:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.455+0000 7fd96c8a8700 1 -- 172.21.15.187:0/3134418271 learned_addr learned my addr 172.21.15.187:0/3134418271 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:30.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.455+0000 7fd96d8aa700 1 --2- 172.21.15.187:0/3134418271 >> v2:172.21.15.5:3300/0 conn(0x7fd968066f70 0x7fd9680aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.455+0000 7fd96c8a8700 1 -- 172.21.15.187:0/3134418271 >> v2:172.21.15.5:3300/0 conn(0x7fd968066f70 msgr2=0x7fd9680aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:30.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.455+0000 7fd96c8a8700 1 --2- 172.21.15.187:0/3134418271 >> v2:172.21.15.5:3300/0 conn(0x7fd968066f70 0x7fd9680aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.455+0000 7fd96c8a8700 1 -- 172.21.15.187:0/3134418271 >> v1:172.21.15.187:6789/0 conn(0x7fd9681484d0 legacy=0x7fd968061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:30.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.456+0000 7fd96c8a8700 1 -- 172.21.15.187:0/3134418271 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9680b1840 con 0x7fd9680b6650 2020-11-20T12:14:30.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.456+0000 7fd96d8aa700 1 --2- 172.21.15.187:0/3134418271 >> v2:172.21.15.5:3300/0 conn(0x7fd968066f70 0x7fd9680aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:30.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.456+0000 7fd96c8a8700 1 --2- 172.21.15.187:0/3134418271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9680b6650 0x7fd9680afa90 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fd968068600 tx=0x7fd958006eb0).ready entity=mon.1 client_cookie=d346817726a4ba9a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:30.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.456+0000 7fd957fff700 1 -- 172.21.15.187:0/3134418271 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9580098d0 con 0x7fd9680b6650 2020-11-20T12:14:30.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.456+0000 7fd957fff700 1 -- 172.21.15.187:0/3134418271 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd958009a30 con 0x7fd9680b6650 2020-11-20T12:14:30.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.456+0000 7fd957fff700 1 -- 172.21.15.187:0/3134418271 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958009db0 con 0x7fd9680b6650 2020-11-20T12:14:30.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.457+0000 7fd96e0ab700 1 -- 172.21.15.187:0/3134418271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9680b6650 msgr2=0x7fd9680afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:30.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.457+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/3134418271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9680b6650 0x7fd9680afa90 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fd968068600 tx=0x7fd958006eb0).stop 2020-11-20T12:14:30.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.457+0000 7fd96e0ab700 1 -- 172.21.15.187:0/3134418271 shutdown_connections 2020-11-20T12:14:30.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.457+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/3134418271 >> v2:172.21.15.5:3300/0 conn(0x7fd968066f70 0x7fd9680aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.457+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/3134418271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9680b6650 0x7fd9680afa90 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.457+0000 7fd96e0ab700 1 -- 172.21.15.187:0/3134418271 >> 172.21.15.187:0/3134418271 conn(0x7fd9680bab30 msgr2=0x7fd9680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:30.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.457+0000 7fd96e0ab700 1 -- 172.21.15.187:0/3134418271 shutdown_connections 2020-11-20T12:14:30.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.457+0000 7fd96e0ab700 1 -- 172.21.15.187:0/3134418271 wait complete. 2020-11-20T12:14:30.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.458+0000 7fd96e0ab700 1 Processor -- start 2020-11-20T12:14:30.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.458+0000 7fd96e0ab700 1 -- start start 2020-11-20T12:14:30.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.459+0000 7fd96e0ab700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd968066f70 0x7fd96813e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.459+0000 7fd96e0ab700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9680b6650 0x7fd96809fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.459+0000 7fd96e0ab700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9681484d0 0x7fd96815b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.459+0000 7fd96e0ab700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96815c210 con 0x7fd9681484d0 2020-11-20T12:14:30.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.459+0000 7fd96e0ab700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96815c4b0 con 0x7fd968066f70 2020-11-20T12:14:30.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.459+0000 7fd96e0ab700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96815bca0 con 0x7fd9680b6650 2020-11-20T12:14:30.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd968066f70 0x7fd96813e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd968066f70 0x7fd96813e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49040/0 (socket says 172.21.15.187:49040) 2020-11-20T12:14:30.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 -- 172.21.15.187:0/2713810991 learned_addr learned my addr 172.21.15.187:0/2713810991 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:30.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d8aa700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9681484d0 0x7fd96815b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 -- 172.21.15.187:0/2713810991 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9680b6650 msgr2=0x7fd96809fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:30.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9680b6650 0x7fd96809fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:30.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 -- 172.21.15.187:0/2713810991 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9681484d0 msgr2=0x7fd96815b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:30.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9681484d0 0x7fd96815b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:30.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 -- 172.21.15.187:0/2713810991 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96815be60 con 0x7fd968066f70 2020-11-20T12:14:30.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.460+0000 7fd96d0a9700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd968066f70 0x7fd96813e410 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fd96400c6e0 tx=0x7fd964007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:30.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.461+0000 7fd9557fa700 1 -- 172.21.15.187:0/2713810991 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964007020 con 0x7fd968066f70 2020-11-20T12:14:30.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.461+0000 7fd9557fa700 1 -- 172.21.15.187:0/2713810991 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd964007180 con 0x7fd968066f70 2020-11-20T12:14:30.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.461+0000 7fd9557fa700 1 -- 172.21.15.187:0/2713810991 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96400e560 con 0x7fd968066f70 2020-11-20T12:14:30.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.461+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9680a0610 con 0x7fd968066f70 2020-11-20T12:14:30.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.461+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd968135f30 con 0x7fd968066f70 2020-11-20T12:14:30.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.462+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96809ff50 con 0x7fd968066f70 2020-11-20T12:14:30.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.462+0000 7fd9557fa700 1 -- 172.21.15.187:0/2713810991 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd964010070 con 0x7fd968066f70 2020-11-20T12:14:30.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.463+0000 7fd9557fa700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd940056d20 0x7fd9400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:30.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.463+0000 7fd9557fa700 1 -- 172.21.15.187:0/2713810991 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd96406f2e0 con 0x7fd968066f70 2020-11-20T12:14:30.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.463+0000 7fd96c8a8700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd940056d20 0x7fd9400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:30.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.464+0000 7fd96c8a8700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd940056d20 0x7fd9400591e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fd96815b550 tx=0x7fd958015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:30.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.467+0000 7fd9557fa700 1 -- 172.21.15.187:0/2713810991 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd964044730 con 0x7fd968066f70 2020-11-20T12:14:30.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:30.709+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd96815c050 con 0x7fd968066f70 2020-11-20T12:14:31.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.439+0000 7fd9557fa700 1 -- 172.21.15.187:0/2713810991 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v87) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd964001e60 con 0x7fd968066f70 2020-11-20T12:14:31.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd940056d20 msgr2=0x7fd9400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:31.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd940056d20 0x7fd9400591e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fd96815b550 tx=0x7fd958015000).stop 2020-11-20T12:14:31.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd968066f70 msgr2=0x7fd96813e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:31.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd968066f70 0x7fd96813e410 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fd96400c6e0 tx=0x7fd964007c50).stop 2020-11-20T12:14:31.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 shutdown_connections 2020-11-20T12:14:31.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd940056d20 0x7fd9400591e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9681484d0 0x7fd96815b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd968066f70 0x7fd96813e410 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 --2- 172.21.15.187:0/2713810991 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9680b6650 0x7fd96809fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:31.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.442+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 >> 172.21.15.187:0/2713810991 conn(0x7fd9680bab30 msgr2=0x7fd9680b6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:31.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.443+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 shutdown_connections 2020-11-20T12:14:31.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.443+0000 7fd96e0ab700 1 -- 172.21.15.187:0/2713810991 wait complete. 2020-11-20T12:14:31.473 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:31.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.711+0000 7fec8e607700 1 Processor -- start 2020-11-20T12:14:31.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.711+0000 7fec8e607700 1 -- start start 2020-11-20T12:14:31.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.713+0000 7fec8e607700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec880b4e00 0x7fec8814ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.713+0000 7fec8e607700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec88065ad0 0x7fec88148830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.713+0000 7fec8e607700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8814be00 con 0x7fec880b4a20 2020-11-20T12:14:31.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.713+0000 7fec8e607700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8814c030 con 0x7fec880b4e00 2020-11-20T12:14:31.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.713+0000 7fec8e607700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8814c2d0 con 0x7fec88065ad0 2020-11-20T12:14:31.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.713+0000 7fec8d605700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fec880b4a20 0x7fec88145a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45652/0 (socket says 172.21.15.187:45652) 2020-11-20T12:14:31.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.713+0000 7fec8d605700 1 -- 172.21.15.187:0/1773524016 learned_addr learned my addr 172.21.15.187:0/1773524016 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:31.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.713+0000 7fec8ce04700 1 --2- 172.21.15.187:0/1773524016 >> v2:172.21.15.187:3300/0 conn(0x7fec880b4e00 0x7fec8814ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:31.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec8de06700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec88065ad0 0x7fec88148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:31.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311921381 0 0) 0x7fec8814be00 con 0x7fec880b4a20 2020-11-20T12:14:31.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec70003240 con 0x7fec880b4a20 2020-11-20T12:14:31.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3251227338 0 0) 0x7fec70003240 con 0x7fec880b4a20 2020-11-20T12:14:31.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec88065ad0 msgr2=0x7fec88148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:31.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec88065ad0 0x7fec88148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 >> v2:172.21.15.187:3300/0 conn(0x7fec880b4e00 msgr2=0x7fec8814ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:31.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 --2- 172.21.15.187:0/1773524016 >> v2:172.21.15.187:3300/0 conn(0x7fec880b4e00 0x7fec8814ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8814c570 con 0x7fec880b4a20 2020-11-20T12:14:31.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fec840014e0 con 0x7fec880b4a20 2020-11-20T12:14:31.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 >> v1:172.21.15.187:6789/0 conn(0x7fec880b4a20 legacy=0x7fec88145a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:31.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec700037b0 0x7fec70003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.714+0000 7fec77fff700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec700044e0 0x7fec70004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec77fff700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec88065ad0 0x7fec70004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8814c030 con 0x7fec700037b0 2020-11-20T12:14:31.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8814c2d0 con 0x7fec700044e0 2020-11-20T12:14:31.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec70005340 con 0x7fec88065ad0 2020-11-20T12:14:31.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec8ce04700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec700044e0 0x7fec70004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:31.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec8d605700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec700037b0 0x7fec70003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:31.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec8ce04700 1 -- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec88065ad0 msgr2=0x7fec70004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:31.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec8ce04700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec88065ad0 0x7fec70004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:31.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec8ce04700 1 -- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec700037b0 msgr2=0x7fec70003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:31.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec8ce04700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec700037b0 0x7fec70003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.715+0000 7fec8ce04700 1 -- 172.21.15.187:0/1773524016 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec70003240 con 0x7fec700044e0 2020-11-20T12:14:31.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.716+0000 7fec8ce04700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec700044e0 0x7fec70004960 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7fec7800bc00 tx=0x7fec78006df0).ready entity=mon.1 client_cookie=432daa09c9374ec0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:31.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.716+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7800b310 con 0x7fec700044e0 2020-11-20T12:14:31.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.716+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7800eb40 con 0x7fec700044e0 2020-11-20T12:14:31.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.716+0000 7fec77fff700 1 -- 172.21.15.187:0/1773524016 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7800eec0 con 0x7fec700044e0 2020-11-20T12:14:31.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.716+0000 7fec8e607700 1 -- 172.21.15.187:0/1773524016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec700044e0 msgr2=0x7fec70004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:31.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.716+0000 7fec8e607700 1 --2- 172.21.15.187:0/1773524016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec700044e0 0x7fec70004960 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7fec7800bc00 tx=0x7fec78006df0).stop 2020-11-20T12:14:31.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.716+0000 7fec8de06700 1 -- 172.21.15.187:0/1773524016 reap_dead start 2020-11-20T12:14:31.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.717+0000 7fec8e607700 1 -- 172.21.15.187:0/1773524016 shutdown_connections 2020-11-20T12:14:31.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.717+0000 7fec8e607700 1 -- 172.21.15.187:0/1773524016 >> 172.21.15.187:0/1773524016 conn(0x7fec880bb3d0 msgr2=0x7fec880bb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:31.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.717+0000 7fec8e607700 1 -- 172.21.15.187:0/1773524016 shutdown_connections 2020-11-20T12:14:31.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.717+0000 7fec8e607700 1 -- 172.21.15.187:0/1773524016 wait complete. 2020-11-20T12:14:31.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.718+0000 7fec8e607700 1 Processor -- start 2020-11-20T12:14:31.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.718+0000 7fec8e607700 1 -- start start 2020-11-20T12:14:31.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8e607700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec8815b910 0x7fec8815bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8e607700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec88142c80 0x7fec881430e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8e607700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec88115060 0x7fec881154a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8e607700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec880a11c0 con 0x7fec88115060 2020-11-20T12:14:31.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8e607700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec88147860 con 0x7fec88142c80 2020-11-20T12:14:31.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8e607700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec88146e80 con 0x7fec8815b910 2020-11-20T12:14:31.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8ce04700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec88142c80 0x7fec881430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:31.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8ce04700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec88142c80 0x7fec881430e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49060/0 (socket says 172.21.15.187:49060) 2020-11-20T12:14:31.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.719+0000 7fec8ce04700 1 -- 172.21.15.187:0/2092198930 learned_addr learned my addr 172.21.15.187:0/2092198930 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:31.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8de06700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec88115060 0x7fec881154a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:31.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8d605700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec8815b910 0x7fec8815bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:31.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8ce04700 1 -- 172.21.15.187:0/2092198930 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec8815b910 msgr2=0x7fec8815bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:31.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8ce04700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec8815b910 0x7fec8815bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8ce04700 1 -- 172.21.15.187:0/2092198930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec88115060 msgr2=0x7fec881154a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:31.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8ce04700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec88115060 0x7fec881154a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8ce04700 1 -- 172.21.15.187:0/2092198930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec88147120 con 0x7fec88142c80 2020-11-20T12:14:31.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8ce04700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec88142c80 0x7fec881430e0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7fec78006e30 tx=0x7fec78019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:31.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec757fa700 1 -- 172.21.15.187:0/2092198930 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7801b660 con 0x7fec88142c80 2020-11-20T12:14:31.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec757fa700 1 -- 172.21.15.187:0/2092198930 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec78022070 con 0x7fec88142c80 2020-11-20T12:14:31.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec881473c0 con 0x7fec88142c80 2020-11-20T12:14:31.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec757fa700 1 -- 172.21.15.187:0/2092198930 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7801b7e0 con 0x7fec88142c80 2020-11-20T12:14:31.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.720+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec88148230 con 0x7fec88142c80 2020-11-20T12:14:31.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.722+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec88147550 con 0x7fec88142c80 2020-11-20T12:14:31.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.722+0000 7fec757fa700 1 -- 172.21.15.187:0/2092198930 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fec7802e070 con 0x7fec88142c80 2020-11-20T12:14:31.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.723+0000 7fec757fa700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fec60056d20 0x7fec600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:31.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.723+0000 7fec757fa700 1 -- 172.21.15.187:0/2092198930 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fec780725b0 con 0x7fec88142c80 2020-11-20T12:14:31.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.723+0000 7fec8d605700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fec60056d20 0x7fec600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:31.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.724+0000 7fec8d605700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fec60056d20 0x7fec600591e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fec880b54f0 tx=0x7fec840014a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:31.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.727+0000 7fec757fa700 1 -- 172.21.15.187:0/2092198930 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fec78047980 con 0x7fec88142c80 2020-11-20T12:14:31.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.952+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec880b8fd0 con 0x7fec88142c80 2020-11-20T12:14:31.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.953+0000 7fec757fa700 1 -- 172.21.15.187:0/2092198930 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fec7804afa0 con 0x7fec88142c80 2020-11-20T12:14:31.957 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:31.958 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:31.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fec60056d20 msgr2=0x7fec600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:31.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fec60056d20 0x7fec600591e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fec880b54f0 tx=0x7fec840014a0).stop 2020-11-20T12:14:31.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec88142c80 msgr2=0x7fec881430e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:31.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec88142c80 0x7fec881430e0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7fec78006e30 tx=0x7fec78019000).stop 2020-11-20T12:14:31.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 shutdown_connections 2020-11-20T12:14:31.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fec60056d20 0x7fec600591e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec88115060 0x7fec881154a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec8815b910 0x7fec8815bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 --2- 172.21.15.187:0/2092198930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec88142c80 0x7fec881430e0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:31.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 >> 172.21.15.187:0/2092198930 conn(0x7fec880bb3d0 msgr2=0x7fec88144d70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:31.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 shutdown_connections 2020-11-20T12:14:31.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:31.956+0000 7fec8e607700 1 -- 172.21.15.187:0/2092198930 wait complete. 2020-11-20T12:14:31.963 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:31.982 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:32.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.214+0000 7f2f2af6a700 1 Processor -- start 2020-11-20T12:14:32.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.215+0000 7f2f2af6a700 1 -- start start 2020-11-20T12:14:32.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.216+0000 7f2f2af6a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 0x7f2f24148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.216+0000 7f2f2af6a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2414c170 con 0x7f2f24145cc0 2020-11-20T12:14:32.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.216+0000 7f2f2af6a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2414c410 con 0x7f2f240683f0 2020-11-20T12:14:32.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.216+0000 7f2f2af6a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2414c6b0 con 0x7f2f24066040 2020-11-20T12:14:32.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.216+0000 7f2f29767700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2f240683f0 0x7f2f241487e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45674/0 (socket says 172.21.15.187:45674) 2020-11-20T12:14:32.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.216+0000 7f2f29767700 1 -- 172.21.15.187:0/1951893989 learned_addr learned my addr 172.21.15.187:0/1951893989 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:32.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f2a769700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 0x7f2f24148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225153174 0 0) 0x7f2f2414c410 con 0x7f2f240683f0 2020-11-20T12:14:32.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f0c003240 con 0x7f2f240683f0 2020-11-20T12:14:32.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4178557786 0 0) 0x7f2f0c003240 con 0x7f2f240683f0 2020-11-20T12:14:32.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 msgr2=0x7f2f24148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:32.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 0x7f2f24148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 >> v1:172.21.15.5:6789/0 conn(0x7f2f24145cc0 legacy=0x7f2f24065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f240658a0 con 0x7f2f240683f0 2020-11-20T12:14:32.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f2f140014e0 con 0x7f2f240683f0 2020-11-20T12:14:32.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 >> v1:172.21.15.187:6789/0 conn(0x7f2f240683f0 legacy=0x7f2f241487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 0x7f2f0c003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f0c003c40 0x7f2f0c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.217+0000 7f2f28f66700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f0c0049d0 0x7f2f0c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2414c6b0 con 0x7f2f0c003c40 2020-11-20T12:14:32.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c0041d0 con 0x7f2f0c0049d0 2020-11-20T12:14:32.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c00b730 con 0x7f2f24066040 2020-11-20T12:14:32.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f2a769700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f0c0049d0 0x7f2f0c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f29767700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f0c003c40 0x7f2f0c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f29f68700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 0x7f2f0c003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f2a769700 1 -- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 msgr2=0x7f2f0c003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:32.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f2a769700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 0x7f2f0c003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f2a769700 1 -- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f0c003c40 msgr2=0x7f2f0c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:32.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f2a769700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f0c003c40 0x7f2f0c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f2a769700 1 -- 172.21.15.187:0/1951893989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c003240 con 0x7f2f0c0049d0 2020-11-20T12:14:32.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f29f68700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f24066040 0x7f2f0c003730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:32.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.218+0000 7f2f29767700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f0c003c40 0x7f2f0c0044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:32.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.219+0000 7f2f2a769700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f0c0049d0 0x7f2f0c004e50 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7f2f1c00bc00 tx=0x7f2f1c006df0).ready entity=mon.1 client_cookie=76e09145c742926b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:32.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.219+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c00b310 con 0x7f2f0c0049d0 2020-11-20T12:14:32.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.219+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c00eae0 con 0x7f2f0c0049d0 2020-11-20T12:14:32.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.219+0000 7f2f28f66700 1 -- 172.21.15.187:0/1951893989 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c00ee60 con 0x7f2f0c0049d0 2020-11-20T12:14:32.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.220+0000 7f2f2af6a700 1 -- 172.21.15.187:0/1951893989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f0c0049d0 msgr2=0x7f2f0c004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.220+0000 7f2f2af6a700 1 --2- 172.21.15.187:0/1951893989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f0c0049d0 0x7f2f0c004e50 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7f2f1c00bc00 tx=0x7f2f1c006df0).stop 2020-11-20T12:14:32.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.220+0000 7f2f2a769700 1 -- 172.21.15.187:0/1951893989 reap_dead start 2020-11-20T12:14:32.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.220+0000 7f2f2af6a700 1 -- 172.21.15.187:0/1951893989 shutdown_connections 2020-11-20T12:14:32.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.220+0000 7f2f2af6a700 1 -- 172.21.15.187:0/1951893989 >> 172.21.15.187:0/1951893989 conn(0x7f2f240bab30 msgr2=0x7f2f240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:32.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.220+0000 7f2f2af6a700 1 -- 172.21.15.187:0/1951893989 shutdown_connections 2020-11-20T12:14:32.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.220+0000 7f2f2af6a700 1 -- 172.21.15.187:0/1951893989 wait complete. 2020-11-20T12:14:32.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.221+0000 7f2f2af6a700 1 Processor -- start 2020-11-20T12:14:32.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.221+0000 7f2f2af6a700 1 -- start start 2020-11-20T12:14:32.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.222+0000 7f2f2af6a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f2409cd90 0x7f2f2409bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.222+0000 7f2f2af6a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f2409d170 0x7f2f24143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.222+0000 7f2f2af6a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f240a0180 0x7f2f24144200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.222+0000 7f2f2af6a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2409c490 con 0x7f2f240a0180 2020-11-20T12:14:32.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.222+0000 7f2f2af6a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2409c6e0 con 0x7f2f2409d170 2020-11-20T12:14:32.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.222+0000 7f2f2af6a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2409dff0 con 0x7f2f2409cd90 2020-11-20T12:14:32.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f2409d170 0x7f2f24143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f2409d170 0x7f2f24143d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49080/0 (socket says 172.21.15.187:49080) 2020-11-20T12:14:32.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 -- 172.21.15.187:0/391008844 learned_addr learned my addr 172.21.15.187:0/391008844 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:32.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29f68700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f2409cd90 0x7f2f2409bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 -- 172.21.15.187:0/391008844 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f2409cd90 msgr2=0x7f2f2409bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f2409cd90 0x7f2f2409bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 -- 172.21.15.187:0/391008844 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f240a0180 msgr2=0x7f2f24144200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f240a0180 0x7f2f24144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:32.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 -- 172.21.15.187:0/391008844 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2409e290 con 0x7f2f2409d170 2020-11-20T12:14:32.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.223+0000 7f2f29767700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f2409d170 0x7f2f24143d10 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f2f1400dab0 tx=0x7f2f14009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:32.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.224+0000 7f2f11ffb700 1 -- 172.21.15.187:0/391008844 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f14007170 con 0x7f2f2409d170 2020-11-20T12:14:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.224+0000 7f2f11ffb700 1 -- 172.21.15.187:0/391008844 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f140072d0 con 0x7f2f2409d170 2020-11-20T12:14:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.224+0000 7f2f11ffb700 1 -- 172.21.15.187:0/391008844 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1400bb40 con 0x7f2f2409d170 2020-11-20T12:14:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.224+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f240b2b80 con 0x7f2f2409d170 2020-11-20T12:14:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.224+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2415b460 con 0x7f2f2409d170 2020-11-20T12:14:32.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.225+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2409e420 con 0x7f2f2409d170 2020-11-20T12:14:32.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.226+0000 7f2f11ffb700 1 -- 172.21.15.187:0/391008844 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2f140045c0 con 0x7f2f2409d170 2020-11-20T12:14:32.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.226+0000 7f2f11ffb700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2efc056d20 0x7f2efc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.226+0000 7f2f11ffb700 1 -- 172.21.15.187:0/391008844 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2f14070290 con 0x7f2f2409d170 2020-11-20T12:14:32.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.227+0000 7f2f29f68700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2efc056d20 0x7f2efc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.227+0000 7f2f29f68700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2efc056d20 0x7f2efc0591e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f2f20000f40 tx=0x7f2f20009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:32.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.230+0000 7f2f11ffb700 1 -- 172.21.15.187:0/391008844 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2f14045680 con 0x7f2f2409d170 2020-11-20T12:14:32.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.415+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f24061a20 con 0x7f2f2409d170 2020-11-20T12:14:32.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.418+0000 7f2f11ffb700 1 -- 172.21.15.187:0/391008844 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f2f14001190 con 0x7f2f2409d170 2020-11-20T12:14:32.423 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:32.423 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":98,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:14:30.229422+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:14:27.347251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2020-11-20T12:14:27.963950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:32.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2efc056d20 msgr2=0x7f2efc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2efc056d20 0x7f2efc0591e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f2f20000f40 tx=0x7f2f20009040).stop 2020-11-20T12:14:32.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f2409d170 msgr2=0x7f2f24143d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f2409d170 0x7f2f24143d10 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f2f1400dab0 tx=0x7f2f14009040).stop 2020-11-20T12:14:32.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 shutdown_connections 2020-11-20T12:14:32.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2efc056d20 0x7f2efc0591e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f240a0180 0x7f2f24144200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:32.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f2409cd90 0x7f2f2409bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 --2- 172.21.15.187:0/391008844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f2409d170 0x7f2f24143d10 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 >> 172.21.15.187:0/391008844 conn(0x7f2f240bab30 msgr2=0x7f2f240b93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:32.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 shutdown_connections 2020-11-20T12:14:32.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.420+0000 7f2f2af6a700 1 -- 172.21.15.187:0/391008844 wait complete. 2020-11-20T12:14:32.449 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:14:32.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.699+0000 7ffabce5b700 1 Processor -- start 2020-11-20T12:14:32.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.699+0000 7ffabce5b700 1 -- start start 2020-11-20T12:14:32.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.700+0000 7ffabce5b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffab81487c0 0x7ffab8066e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.700+0000 7ffabce5b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab8062280 con 0x7ffab8061de0 2020-11-20T12:14:32.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.700+0000 7ffabce5b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab8062520 con 0x7ffab81483e0 2020-11-20T12:14:32.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.700+0000 7ffabce5b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab80627c0 con 0x7ffab81487c0 2020-11-20T12:14:32.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.700+0000 7ffab659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ffab81483e0 0x7ffab8065300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45692/0 (socket says 172.21.15.187:45692) 2020-11-20T12:14:32.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.700+0000 7ffab659c700 1 -- 172.21.15.187:0/3229187576 learned_addr learned my addr 172.21.15.187:0/3229187576 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:32.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.701+0000 7ffab6d9d700 1 --2- 172.21.15.187:0/3229187576 >> v2:172.21.15.187:3300/0 conn(0x7ffab81487c0 0x7ffab8066e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.701+0000 7ffab559a700 1 -- 172.21.15.187:0/3229187576 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976213093 0 0) 0x7ffab8062520 con 0x7ffab81483e0 2020-11-20T12:14:32.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.701+0000 7ffab559a700 1 -- 172.21.15.187:0/3229187576 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c003240 con 0x7ffab81483e0 2020-11-20T12:14:32.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.701+0000 7ffab6d9d700 1 -- 172.21.15.187:0/3229187576 >> v1:172.21.15.187:6789/0 conn(0x7ffab81483e0 legacy=0x7ffab8065300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.701+0000 7ffab6d9d700 1 -- 172.21.15.187:0/3229187576 >> v1:172.21.15.5:6789/0 conn(0x7ffab8061de0 legacy=0x7ffab8066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.701+0000 7ffab6d9d700 1 -- 172.21.15.187:0/3229187576 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8065a10 con 0x7ffab81487c0 2020-11-20T12:14:32.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.702+0000 7ffab6d9d700 1 --2- 172.21.15.187:0/3229187576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab81487c0 0x7ffab8066e10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7ffab0009c90 tx=0x7ffab00091d0).ready entity=mon.1 client_cookie=fe1eb17029afb2d2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:32.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.702+0000 7ffab559a700 1 -- 172.21.15.187:0/3229187576 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab000ba70 con 0x7ffab81487c0 2020-11-20T12:14:32.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.702+0000 7ffab559a700 1 -- 172.21.15.187:0/3229187576 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab000bbd0 con 0x7ffab81487c0 2020-11-20T12:14:32.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.702+0000 7ffab559a700 1 -- 172.21.15.187:0/3229187576 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0011400 con 0x7ffab81487c0 2020-11-20T12:14:32.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.703+0000 7ffabce5b700 1 -- 172.21.15.187:0/3229187576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab81487c0 msgr2=0x7ffab8066e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.703+0000 7ffabce5b700 1 --2- 172.21.15.187:0/3229187576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab81487c0 0x7ffab8066e10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7ffab0009c90 tx=0x7ffab00091d0).stop 2020-11-20T12:14:32.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.703+0000 7ffabce5b700 1 -- 172.21.15.187:0/3229187576 shutdown_connections 2020-11-20T12:14:32.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.703+0000 7ffabce5b700 1 --2- 172.21.15.187:0/3229187576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab81487c0 0x7ffab8066e10 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.703+0000 7ffabce5b700 1 -- 172.21.15.187:0/3229187576 >> 172.21.15.187:0/3229187576 conn(0x7ffab80bacf0 msgr2=0x7ffab80bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:32.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.703+0000 7ffabce5b700 1 -- 172.21.15.187:0/3229187576 shutdown_connections 2020-11-20T12:14:32.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.703+0000 7ffabce5b700 1 -- 172.21.15.187:0/3229187576 wait complete. 2020-11-20T12:14:32.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.704+0000 7ffabce5b700 1 Processor -- start 2020-11-20T12:14:32.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.704+0000 7ffabce5b700 1 -- start start 2020-11-20T12:14:32.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.705+0000 7ffabce5b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab8061de0 0x7ffab8146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.705+0000 7ffabce5b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffab81483e0 0x7ffab8146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.705+0000 7ffabce5b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffab81487c0 0x7ffab8158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.705+0000 7ffabce5b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab8149e10 con 0x7ffab81483e0 2020-11-20T12:14:32.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.705+0000 7ffabce5b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab814a0b0 con 0x7ffab8061de0 2020-11-20T12:14:32.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.705+0000 7ffabce5b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab814a350 con 0x7ffab81487c0 2020-11-20T12:14:32.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab8061de0 0x7ffab8146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab8061de0 0x7ffab8146240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49092/0 (socket says 172.21.15.187:49092) 2020-11-20T12:14:32.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 -- 172.21.15.187:0/1078984033 learned_addr learned my addr 172.21.15.187:0/1078984033 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:32.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 -- 172.21.15.187:0/1078984033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffab81487c0 msgr2=0x7ffab8158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffab81487c0 0x7ffab8158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:32.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 -- 172.21.15.187:0/1078984033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffab81483e0 msgr2=0x7ffab8146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffab81483e0 0x7ffab8146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:32.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 -- 172.21.15.187:0/1078984033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab814a7f0 con 0x7ffab8061de0 2020-11-20T12:14:32.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffab659c700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab8061de0 0x7ffab8146240 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7ffaa800c6f0 tx=0x7ffaa8007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:32.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.706+0000 7ffaa67fc700 1 -- 172.21.15.187:0/1078984033 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8007320 con 0x7ffab8061de0 2020-11-20T12:14:32.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.707+0000 7ffaa67fc700 1 -- 172.21.15.187:0/1078984033 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa8007480 con 0x7ffab8061de0 2020-11-20T12:14:32.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.707+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab814b300 con 0x7ffab8061de0 2020-11-20T12:14:32.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.707+0000 7ffaa67fc700 1 -- 172.21.15.187:0/1078984033 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8005800 con 0x7ffab8061de0 2020-11-20T12:14:32.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.707+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab8158b40 con 0x7ffab8061de0 2020-11-20T12:14:32.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.708+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab814a510 con 0x7ffab8061de0 2020-11-20T12:14:32.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.708+0000 7ffaa67fc700 1 -- 172.21.15.187:0/1078984033 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffaa8010030 con 0x7ffab8061de0 2020-11-20T12:14:32.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.709+0000 7ffaa67fc700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa8c056d20 0x7ffa8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:32.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.709+0000 7ffaa67fc700 1 -- 172.21.15.187:0/1078984033 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffaa806fc80 con 0x7ffab8061de0 2020-11-20T12:14:32.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.709+0000 7ffab5d9b700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa8c056d20 0x7ffa8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:32.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.710+0000 7ffab5d9b700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa8c056d20 0x7ffa8c0591e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7ffab80b5680 tx=0x7ffaac007810).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:32.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.713+0000 7ffaa67fc700 1 -- 172.21.15.187:0/1078984033 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffaa8045070 con 0x7ffab8061de0 2020-11-20T12:14:32.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.967+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7ffab8044f60 con 0x7ffab8061de0 2020-11-20T12:14:32.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.978+0000 7ffaa67fc700 1 -- 172.21.15.187:0/1078984033 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7ffaa8048690 con 0x7ffab8061de0 2020-11-20T12:14:32.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa8c056d20 msgr2=0x7ffa8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa8c056d20 0x7ffa8c0591e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7ffab80b5680 tx=0x7ffaac007810).stop 2020-11-20T12:14:32.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab8061de0 msgr2=0x7ffab8146240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:32.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab8061de0 0x7ffab8146240 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7ffaa800c6f0 tx=0x7ffaa8007ad0).stop 2020-11-20T12:14:32.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 shutdown_connections 2020-11-20T12:14:32.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffa8c056d20 0x7ffa8c0591e0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffab81487c0 0x7ffab8158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:32.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffab8061de0 0x7ffab8146240 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:32.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 --2- 172.21.15.187:0/1078984033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffab81483e0 0x7ffab8146730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:32.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 >> 172.21.15.187:0/1078984033 conn(0x7ffab80bacf0 msgr2=0x7ffab80b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:32.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 shutdown_connections 2020-11-20T12:14:32.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:32.981+0000 7ffabce5b700 1 -- 172.21.15.187:0/1078984033 wait complete. 2020-11-20T12:14:32.988 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:14:33.008 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:33.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.250+0000 7ff4a9616700 1 Processor -- start 2020-11-20T12:14:33.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.250+0000 7ff4a9616700 1 -- start start 2020-11-20T12:14:33.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.252+0000 7ff4a9616700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a40630a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:33.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.252+0000 7ff4a9616700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a40b1230 con 0x7ff4a4067070 2020-11-20T12:14:33.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.252+0000 7ff4a9616700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a40626c0 con 0x7ff4a4067450 2020-11-20T12:14:33.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.252+0000 7ff4a9616700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a4062960 con 0x7ff4a40607e0 2020-11-20T12:14:33.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.253+0000 7ff4a37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff4a4067450 0x7ff4a40b0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45708/0 (socket says 172.21.15.187:45708) 2020-11-20T12:14:33.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.253+0000 7ff4a37fe700 1 -- 172.21.15.187:0/3371847305 learned_addr learned my addr 172.21.15.187:0/3371847305 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:33.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.253+0000 7ff4a8e15700 1 --2- 172.21.15.187:0/3371847305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a40630a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:33.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.253+0000 7ff4a2ffd700 1 -- 172.21.15.187:0/3371847305 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136404744 0 0) 0x7ff4a40626c0 con 0x7ff4a4067450 2020-11-20T12:14:33.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.253+0000 7ff4a2ffd700 1 -- 172.21.15.187:0/3371847305 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff490003240 con 0x7ff4a4067450 2020-11-20T12:14:33.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.254+0000 7ff4a8e15700 1 -- 172.21.15.187:0/3371847305 >> v1:172.21.15.187:6789/0 conn(0x7ff4a4067450 legacy=0x7ff4a40b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:33.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.254+0000 7ff4a2ffd700 1 -- 172.21.15.187:0/3371847305 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4141277154 0 0) 0x7ff490003240 con 0x7ff4a4067450 2020-11-20T12:14:33.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.254+0000 7ff4a8e15700 1 -- 172.21.15.187:0/3371847305 >> v1:172.21.15.5:6789/0 conn(0x7ff4a4067070 legacy=0x7ff4a414c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:33.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.254+0000 7ff4a8e15700 1 -- 172.21.15.187:0/3371847305 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a4062c00 con 0x7ff4a40607e0 2020-11-20T12:14:33.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.254+0000 7ff4a8e15700 1 --2- 172.21.15.187:0/3371847305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a40630a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff49c009c40 tx=0x7ff49c0091d0).ready entity=mon.2 client_cookie=ed0177df42c788b4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:33.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.255+0000 7ff4a2ffd700 1 -- 172.21.15.187:0/3371847305 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498001520 con 0x7ff4a40607e0 2020-11-20T12:14:33.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.255+0000 7ff4a2ffd700 1 -- 172.21.15.187:0/3371847305 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49c00b710 con 0x7ff4a40607e0 2020-11-20T12:14:33.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.255+0000 7ff4a2ffd700 1 -- 172.21.15.187:0/3371847305 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00ba90 con 0x7ff4a40607e0 2020-11-20T12:14:33.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.256+0000 7ff4a9616700 1 -- 172.21.15.187:0/3371847305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 msgr2=0x7ff4a40630a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:33.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.256+0000 7ff4a9616700 1 --2- 172.21.15.187:0/3371847305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a40630a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff49c009c40 tx=0x7ff49c0091d0).stop 2020-11-20T12:14:33.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.256+0000 7ff4a9616700 1 -- 172.21.15.187:0/3371847305 shutdown_connections 2020-11-20T12:14:33.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.256+0000 7ff4a9616700 1 --2- 172.21.15.187:0/3371847305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a40630a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:33.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.256+0000 7ff4a9616700 1 -- 172.21.15.187:0/3371847305 >> 172.21.15.187:0/3371847305 conn(0x7ff4a40bb470 msgr2=0x7ff4a40bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:33.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.256+0000 7ff4a9616700 1 -- 172.21.15.187:0/3371847305 shutdown_connections 2020-11-20T12:14:33.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.256+0000 7ff4a9616700 1 -- 172.21.15.187:0/3371847305 wait complete. 2020-11-20T12:14:33.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.258+0000 7ff4a9616700 1 Processor -- start 2020-11-20T12:14:33.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.258+0000 7ff4a9616700 1 -- start start 2020-11-20T12:14:33.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a9616700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a415b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:33.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a9616700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4a4067070 0x7ff4a4140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:33.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a9616700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4a4067450 0x7ff4a4145c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:33.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a9616700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a415bc80 con 0x7ff4a4067450 2020-11-20T12:14:33.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a9616700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a40a01f0 con 0x7ff4a4067070 2020-11-20T12:14:33.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a9616700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a415bea0 con 0x7ff4a40607e0 2020-11-20T12:14:33.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4a4067070 0x7ff4a4140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:33.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4a4067070 0x7ff4a4140c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49108/0 (socket says 172.21.15.187:49108) 2020-11-20T12:14:33.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.259+0000 7ff4a37fe700 1 -- 172.21.15.187:0/2142635096 learned_addr learned my addr 172.21.15.187:0/2142635096 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:33.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a3fff700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a415b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:33.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a8e15700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4a4067450 0x7ff4a4145c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:33.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a37fe700 1 -- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 msgr2=0x7ff4a415b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:33.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a37fe700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a415b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:33.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a37fe700 1 -- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4a4067450 msgr2=0x7ff4a4145c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:33.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a37fe700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4a4067450 0x7ff4a4145c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:33.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a37fe700 1 -- 172.21.15.187:0/2142635096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a415c140 con 0x7ff4a4067070 2020-11-20T12:14:33.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a3fff700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a415b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:33.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a37fe700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4a4067070 0x7ff4a4140c20 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7ff49800c830 tx=0x7ff4980019e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:33.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff483fff700 1 -- 172.21.15.187:0/2142635096 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498007170 con 0x7ff4a4067070 2020-11-20T12:14:33.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff483fff700 1 -- 172.21.15.187:0/2142635096 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4980072d0 con 0x7ff4a4067070 2020-11-20T12:14:33.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff483fff700 1 -- 172.21.15.187:0/2142635096 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498004960 con 0x7ff4a4067070 2020-11-20T12:14:33.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.260+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a4141110 con 0x7ff4a4067070 2020-11-20T12:14:33.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.261+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a4146330 con 0x7ff4a4067070 2020-11-20T12:14:33.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.262+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a40605e0 con 0x7ff4a4067070 2020-11-20T12:14:33.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.263+0000 7ff483fff700 1 -- 172.21.15.187:0/2142635096 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff498010070 con 0x7ff4a4067070 2020-11-20T12:14:33.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.263+0000 7ff483fff700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff47c056d20 0x7ff47c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:33.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.264+0000 7ff483fff700 1 -- 172.21.15.187:0/2142635096 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff49806fab0 con 0x7ff4a4067070 2020-11-20T12:14:33.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.264+0000 7ff4a3fff700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff47c056d20 0x7ff47c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:33.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.264+0000 7ff4a3fff700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff47c056d20 0x7ff47c0591e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7ff494000f70 tx=0x7ff494009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:33.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.267+0000 7ff483fff700 1 -- 172.21.15.187:0/2142635096 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff498044f00 con 0x7ff4a4067070 2020-11-20T12:14:33.426 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:14:33.422+0000 7ffa74972700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:14:33.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.511+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4a4063610 con 0x7ff4a4067070 2020-11-20T12:14:33.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.511+0000 7ff483fff700 1 -- 172.21.15.187:0/2142635096 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7ff498048520 con 0x7ff4a4067070 2020-11-20T12:14:33.515 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:33.516 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:33.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.513+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff47c056d20 msgr2=0x7ff47c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:33.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.513+0000 7ff4a9616700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff47c056d20 0x7ff47c0591e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7ff494000f70 tx=0x7ff494009040).stop 2020-11-20T12:14:33.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.513+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4a4067070 msgr2=0x7ff4a4140c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:33.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.513+0000 7ff4a9616700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4a4067070 0x7ff4a4140c20 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7ff49800c830 tx=0x7ff4980019e0).stop 2020-11-20T12:14:33.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.514+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 shutdown_connections 2020-11-20T12:14:33.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.514+0000 7ff4a9616700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff47c056d20 0x7ff47c0591e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:33.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.514+0000 7ff4a9616700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4a4067450 0x7ff4a4145c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:33.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.514+0000 7ff4a9616700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff4a40607e0 0x7ff4a415b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:33.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.514+0000 7ff4a9616700 1 --2- 172.21.15.187:0/2142635096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4a4067070 0x7ff4a4140c20 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:33.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.514+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 >> 172.21.15.187:0/2142635096 conn(0x7ff4a40bb470 msgr2=0x7ff4a40b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:33.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.514+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 shutdown_connections 2020-11-20T12:14:33.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:33.514+0000 7ff4a9616700 1 -- 172.21.15.187:0/2142635096 wait complete. 2020-11-20T12:14:33.521 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:33.742 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:14:29.225459+0000', 'modified': '2020-11-20T12:14:31.424296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7801}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7801': {'gid': 7801, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/2663757618', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2663757618}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2663757618}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:14:33.742 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:14:33.743 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:14:34.012 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":7801,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.7574771069999997,"mdsmap_epoch":87,"osdmap_epoch":98,"osdmap_epoch_barrier":98,"uptime":10.431406548} 2020-11-20T12:14:34.012 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 7801, "mdsmap_epoch": 87, "osdmap_epoch": 98, "osdmap_epoch_barrier": 98, "rank_uptime": 4.757477107, "state": "up:active", "uptime": 10.431406548, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:14:34.013 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:14:34.013 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:14:34.013 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:14:34.014 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:14:34.014 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:14:34.014 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:14:34.015 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:14:34.015 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:14:34.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:34.016 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:14:34.066 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:14:34.066 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:14:34.067 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:14:34.067 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:14:34.067 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:14:34.067 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:14:34.067 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:14:34.068 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:14:34.068 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:14:34.068 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:14:34.068 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:14:34.068 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:14:34.069 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:14:34.069 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:14:34.069 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:14:34.069 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42164sec preferred_lft 42164sec 2020-11-20T12:14:34.070 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:14:34.070 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:14:34.070 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:14:34.070 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:14:34.070 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:14:34.071 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:14:34.071 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:14:34.071 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:14:34.071 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:14:34.072 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:14:34.072 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:14:34.072 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:14:34.072 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:14:34.072 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:14:34.073 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:14:34.073 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:14:34.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:34.074 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:14:34.163 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:14:34.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:34.167 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:14:34.216 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:14:34.217 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:14:34.217 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:14:34.217 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:14:34.217 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:14:34.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:34.219 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:14:34.219 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:14:34.220 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:14:34.220 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:14:34.220 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:14:34.221 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:14:34.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:34.470 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:14:34.519 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:14:34.519 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:14:34.520 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:14:34.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:34.521 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:14:34.624 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:34.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.957+0000 7f6386f83700 1 Processor -- start 2020-11-20T12:14:34.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.957+0000 7f6386f83700 1 -- start start 2020-11-20T12:14:34.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6386f83700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380068230 0x7f63801482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6386f83700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f63800b5f50 0x7f63800b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6386f83700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6380144b80 con 0x7f63800b57b0 2020-11-20T12:14:34.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6386f83700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63801487c0 con 0x7f63800b5f50 2020-11-20T12:14:34.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6386f83700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63801489e0 con 0x7f6380068230 2020-11-20T12:14:34.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6385f81700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f63800b57b0 0x7f6380062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45720/0 (socket says 172.21.15.187:45720) 2020-11-20T12:14:34.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6385f81700 1 -- 172.21.15.187:0/2390702267 learned_addr learned my addr 172.21.15.187:0/2390702267 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:34.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245790941 0 0) 0x7f6380144b80 con 0x7f63800b57b0 2020-11-20T12:14:34.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6368003240 con 0x7f63800b57b0 2020-11-20T12:14:34.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.959+0000 7f6386782700 1 --2- 172.21.15.187:0/2390702267 >> v2:172.21.15.5:3300/0 conn(0x7f63800b5f50 0x7f63800b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:34.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6385780700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380068230 0x7f63801482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:34.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2908305330 0 0) 0x7f6368003240 con 0x7f63800b57b0 2020-11-20T12:14:34.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380068230 msgr2=0x7f63801482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:34.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380068230 0x7f63801482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:34.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 >> v2:172.21.15.5:3300/0 conn(0x7f63800b5f50 msgr2=0x7f63800b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:34.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 --2- 172.21.15.187:0/2390702267 >> v2:172.21.15.5:3300/0 conn(0x7f63800b5f50 0x7f63800b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:34.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6380148c80 con 0x7f63800b57b0 2020-11-20T12:14:34.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f637c0014e0 con 0x7f63800b57b0 2020-11-20T12:14:34.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 >> v1:172.21.15.187:6789/0 conn(0x7f63800b57b0 legacy=0x7f6380062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:34.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380068230 0x7f6368003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6368003c40 0x7f63680044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63680049d0 0x7f6368004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63801487c0 con 0x7f63680049d0 2020-11-20T12:14:34.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63801489e0 con 0x7f6368003c40 2020-11-20T12:14:34.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6368004200 con 0x7f6380068230 2020-11-20T12:14:34.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.960+0000 7f6385780700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6368003c40 0x7f63680044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:34.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.961+0000 7f6385f81700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380068230 0x7f6368003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:34.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.961+0000 7f6385780700 1 -- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380068230 msgr2=0x7f6368003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:34.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.961+0000 7f6385780700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380068230 0x7f6368003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:34.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.961+0000 7f6386782700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63680049d0 0x7f6368004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:34.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.961+0000 7f6385780700 1 -- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63680049d0 msgr2=0x7f6368004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:34.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.961+0000 7f6385780700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63680049d0 0x7f6368004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:34.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.961+0000 7f6385780700 1 -- 172.21.15.187:0/2390702267 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6368003240 con 0x7f6368003c40 2020-11-20T12:14:34.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.961+0000 7f6385780700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6368003c40 0x7f63680044e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f637000ba80 tx=0x7f6370008040).ready entity=mon.1 client_cookie=26b81798654a8699 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:34.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.962+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637000d9c0 con 0x7f6368003c40 2020-11-20T12:14:34.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.962+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637000db20 con 0x7f6368003c40 2020-11-20T12:14:34.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.962+0000 7f6384f7f700 1 -- 172.21.15.187:0/2390702267 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637000dea0 con 0x7f6368003c40 2020-11-20T12:14:34.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.963+0000 7f6386f83700 1 -- 172.21.15.187:0/2390702267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6368003c40 msgr2=0x7f63680044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:34.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.963+0000 7f6386f83700 1 --2- 172.21.15.187:0/2390702267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6368003c40 0x7f63680044e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f637000ba80 tx=0x7f6370008040).stop 2020-11-20T12:14:34.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.963+0000 7f6386782700 1 -- 172.21.15.187:0/2390702267 reap_dead start 2020-11-20T12:14:34.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.963+0000 7f6386f83700 1 -- 172.21.15.187:0/2390702267 shutdown_connections 2020-11-20T12:14:34.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.963+0000 7f6386f83700 1 -- 172.21.15.187:0/2390702267 >> 172.21.15.187:0/2390702267 conn(0x7f63800bab30 msgr2=0x7f63800baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:34.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.963+0000 7f6386f83700 1 -- 172.21.15.187:0/2390702267 shutdown_connections 2020-11-20T12:14:34.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.963+0000 7f6386f83700 1 -- 172.21.15.187:0/2390702267 wait complete. 2020-11-20T12:14:34.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.964+0000 7f6386f83700 1 Processor -- start 2020-11-20T12:14:34.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.964+0000 7f6386f83700 1 -- start start 2020-11-20T12:14:34.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.965+0000 7f6386f83700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f638009f8c0 0x7f6380147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.965+0000 7f6386f83700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638009fca0 0x7f6380147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.965+0000 7f6386f83700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380144d70 0x7f6380147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.965+0000 7f6386f83700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638015b720 con 0x7f638009f8c0 2020-11-20T12:14:34.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.965+0000 7f6386f83700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638015b9c0 con 0x7f638009fca0 2020-11-20T12:14:34.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.965+0000 7f6386f83700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638015bc60 con 0x7f6380144d70 2020-11-20T12:14:34.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638009fca0 0x7f6380147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:34.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638009fca0 0x7f6380147620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49128/0 (socket says 172.21.15.187:49128) 2020-11-20T12:14:34.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 -- 172.21.15.187:0/3043570193 learned_addr learned my addr 172.21.15.187:0/3043570193 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:34.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 -- 172.21.15.187:0/3043570193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380144d70 msgr2=0x7f6380147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:34.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380144d70 0x7f6380147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:34.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 -- 172.21.15.187:0/3043570193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f638009f8c0 msgr2=0x7f6380147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:34.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f638009f8c0 0x7f6380147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:34.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 -- 172.21.15.187:0/3043570193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638015bf00 con 0x7f638009fca0 2020-11-20T12:14:34.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f6385780700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638009fca0 0x7f6380147620 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f637000ba50 tx=0x7f6370008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:34.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.966+0000 7f636dffb700 1 -- 172.21.15.187:0/3043570193 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637001d620 con 0x7f638009fca0 2020-11-20T12:14:34.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.967+0000 7f636dffb700 1 -- 172.21.15.187:0/3043570193 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637001d780 con 0x7f638009fca0 2020-11-20T12:14:34.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.967+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638015c3a0 con 0x7f638009fca0 2020-11-20T12:14:34.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.967+0000 7f636dffb700 1 -- 172.21.15.187:0/3043570193 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637001db20 con 0x7f638009fca0 2020-11-20T12:14:34.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.967+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63800b1dc0 con 0x7f638009fca0 2020-11-20T12:14:34.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.968+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638015c090 con 0x7f638009fca0 2020-11-20T12:14:34.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.968+0000 7f636dffb700 1 -- 172.21.15.187:0/3043570193 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f63700123e0 con 0x7f638009fca0 2020-11-20T12:14:34.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.969+0000 7f636dffb700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6358056d20 0x7f63580591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:34.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.969+0000 7f636dffb700 1 -- 172.21.15.187:0/3043570193 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6370071be0 con 0x7f638009fca0 2020-11-20T12:14:34.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.969+0000 7f6385f81700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6358056d20 0x7f63580591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:34.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.970+0000 7f6385f81700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6358056d20 0x7f63580591e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f637c000dd0 tx=0x7f637c000bf0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:34.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:34.973+0000 7f636dffb700 1 -- 172.21.15.187:0/3043570193 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6370046fd0 con 0x7f638009fca0 2020-11-20T12:14:35.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.159+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6380061a20 con 0x7f638009fca0 2020-11-20T12:14:35.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.161+0000 7f636dffb700 1 -- 172.21.15.187:0/3043570193 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f637004a5f0 con 0x7f638009fca0 2020-11-20T12:14:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:35.165 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":98,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:14:30.229422+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:14:27.347251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2020-11-20T12:14:27.963950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:35.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6358056d20 msgr2=0x7f63580591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6358056d20 0x7f63580591e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f637c000dd0 tx=0x7f637c000bf0).stop 2020-11-20T12:14:35.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638009fca0 msgr2=0x7f6380147620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638009fca0 0x7f6380147620 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f637000ba50 tx=0x7f6370008040).stop 2020-11-20T12:14:35.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 shutdown_connections 2020-11-20T12:14:35.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6358056d20 0x7f63580591e0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f638009f8c0 0x7f6380147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:35.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6380144d70 0x7f6380147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:35.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 --2- 172.21.15.187:0/3043570193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638009fca0 0x7f6380147620 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.163+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 >> 172.21.15.187:0/3043570193 conn(0x7f63800bab30 msgr2=0x7f63800b9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:35.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.164+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 shutdown_connections 2020-11-20T12:14:35.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.164+0000 7f6386f83700 1 -- 172.21.15.187:0/3043570193 wait complete. 2020-11-20T12:14:35.189 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:14:27.347251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:14:27.963950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:14:35.190 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:14:35.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.427+0000 7fe73df0b700 1 Processor -- start 2020-11-20T12:14:35.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.427+0000 7fe73df0b700 1 -- start start 2020-11-20T12:14:35.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.429+0000 7fe73df0b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe7381484d0 0x7fe738061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.429+0000 7fe73df0b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7380622c0 0x7fe7380b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.429+0000 7fe73df0b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7380b1300 con 0x7fe738066f70 2020-11-20T12:14:35.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.429+0000 7fe73df0b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe738067730 con 0x7fe7381484d0 2020-11-20T12:14:35.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.429+0000 7fe73df0b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7380b15a0 con 0x7fe7380622c0 2020-11-20T12:14:35.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.429+0000 7fe73d70a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe738066f70 0x7fe7380b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45744/0 (socket says 172.21.15.187:45744) 2020-11-20T12:14:35.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.429+0000 7fe73d70a700 1 -- 172.21.15.187:0/2947239421 learned_addr learned my addr 172.21.15.187:0/2947239421 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:35.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.430+0000 7fe737fff700 1 --2- 172.21.15.187:0/2947239421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7380622c0 0x7fe7380b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.430+0000 7fe73cf09700 1 --2- 172.21.15.187:0/2947239421 >> v2:172.21.15.5:3300/0 conn(0x7fe7381484d0 0x7fe738061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.430+0000 7fe7377fe700 1 -- 172.21.15.187:0/2947239421 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226218827 0 0) 0x7fe7380b1300 con 0x7fe738066f70 2020-11-20T12:14:35.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.430+0000 7fe7377fe700 1 -- 172.21.15.187:0/2947239421 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c003240 con 0x7fe738066f70 2020-11-20T12:14:35.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.431+0000 7fe737fff700 1 -- 172.21.15.187:0/2947239421 >> v2:172.21.15.5:3300/0 conn(0x7fe7381484d0 msgr2=0x7fe738061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:35.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.431+0000 7fe737fff700 1 --2- 172.21.15.187:0/2947239421 >> v2:172.21.15.5:3300/0 conn(0x7fe7381484d0 0x7fe738061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.431+0000 7fe737fff700 1 -- 172.21.15.187:0/2947239421 >> v1:172.21.15.187:6789/0 conn(0x7fe738066f70 legacy=0x7fe7380b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.431+0000 7fe7377fe700 1 -- 172.21.15.187:0/2947239421 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 875388225 0 0) 0x7fe71c003240 con 0x7fe738066f70 2020-11-20T12:14:35.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.431+0000 7fe737fff700 1 -- 172.21.15.187:0/2947239421 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7380b1840 con 0x7fe7380622c0 2020-11-20T12:14:35.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.431+0000 7fe737fff700 1 --2- 172.21.15.187:0/2947239421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7380622c0 0x7fe7380b5640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fe72400b9a0 tx=0x7fe724006df0).ready entity=mon.2 client_cookie=71e2a8cfe7e8f930 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:35.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.432+0000 7fe7377fe700 1 -- 172.21.15.187:0/2947239421 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730001860 con 0x7fe7380622c0 2020-11-20T12:14:35.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.432+0000 7fe7377fe700 1 -- 172.21.15.187:0/2947239421 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe724011400 con 0x7fe7380622c0 2020-11-20T12:14:35.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.432+0000 7fe7377fe700 1 -- 172.21.15.187:0/2947239421 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724011780 con 0x7fe7380622c0 2020-11-20T12:14:35.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.432+0000 7fe73df0b700 1 -- 172.21.15.187:0/2947239421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7380622c0 msgr2=0x7fe7380b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.433+0000 7fe73df0b700 1 --2- 172.21.15.187:0/2947239421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7380622c0 0x7fe7380b5640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fe72400b9a0 tx=0x7fe724006df0).stop 2020-11-20T12:14:35.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.433+0000 7fe73df0b700 1 -- 172.21.15.187:0/2947239421 shutdown_connections 2020-11-20T12:14:35.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.433+0000 7fe73df0b700 1 --2- 172.21.15.187:0/2947239421 >> v2:172.21.15.5:3300/0 conn(0x7fe7381484d0 0x7fe738061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.433+0000 7fe73df0b700 1 --2- 172.21.15.187:0/2947239421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7380622c0 0x7fe7380b5640 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.433+0000 7fe73df0b700 1 -- 172.21.15.187:0/2947239421 >> 172.21.15.187:0/2947239421 conn(0x7fe7380bab30 msgr2=0x7fe7380baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:35.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.433+0000 7fe73df0b700 1 -- 172.21.15.187:0/2947239421 shutdown_connections 2020-11-20T12:14:35.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.433+0000 7fe73df0b700 1 -- 172.21.15.187:0/2947239421 wait complete. 2020-11-20T12:14:35.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.435+0000 7fe73df0b700 1 Processor -- start 2020-11-20T12:14:35.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.435+0000 7fe73df0b700 1 -- start start 2020-11-20T12:14:35.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73df0b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7380622c0 0x7fe73812bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73df0b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe738066f70 0x7fe73812d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73df0b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7381484d0 0x7fe73809dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73df0b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73812cf70 con 0x7fe7381484d0 2020-11-20T12:14:35.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73df0b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73812d210 con 0x7fe7380622c0 2020-11-20T12:14:35.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73df0b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73812d4b0 con 0x7fe738066f70 2020-11-20T12:14:35.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73cf09700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7380622c0 0x7fe73812bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73cf09700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7380622c0 0x7fe73812bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49140/0 (socket says 172.21.15.187:49140) 2020-11-20T12:14:35.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.436+0000 7fe73cf09700 1 -- 172.21.15.187:0/51093393 learned_addr learned my addr 172.21.15.187:0/51093393 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:35.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe737fff700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe738066f70 0x7fe73812d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe73d70a700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7381484d0 0x7fe73809dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe73cf09700 1 -- 172.21.15.187:0/51093393 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe738066f70 msgr2=0x7fe73812d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe73cf09700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe738066f70 0x7fe73812d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe73cf09700 1 -- 172.21.15.187:0/51093393 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7381484d0 msgr2=0x7fe73809dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe73cf09700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7381484d0 0x7fe73809dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe73cf09700 1 -- 172.21.15.187:0/51093393 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73812cba0 con 0x7fe7380622c0 2020-11-20T12:14:35.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe73cf09700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7380622c0 0x7fe73812bb90 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7fe72c00ebd0 tx=0x7fe72c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:35.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe734ff9700 1 -- 172.21.15.187:0/51093393 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00c2e0 con 0x7fe7380622c0 2020-11-20T12:14:35.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe734ff9700 1 -- 172.21.15.187:0/51093393 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c00b9a0 con 0x7fe7380622c0 2020-11-20T12:14:35.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe734ff9700 1 -- 172.21.15.187:0/51093393 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00bd40 con 0x7fe7380622c0 2020-11-20T12:14:35.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.437+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe738130d30 con 0x7fe7380622c0 2020-11-20T12:14:35.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.438+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73809dfd0 con 0x7fe7380622c0 2020-11-20T12:14:35.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.439+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73812cd90 con 0x7fe7380622c0 2020-11-20T12:14:35.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.439+0000 7fe734ff9700 1 -- 172.21.15.187:0/51093393 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe72c012600 con 0x7fe7380622c0 2020-11-20T12:14:35.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.440+0000 7fe734ff9700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe710056d20 0x7fe7100591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.440+0000 7fe734ff9700 1 -- 172.21.15.187:0/51093393 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe72c070230 con 0x7fe7380622c0 2020-11-20T12:14:35.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.440+0000 7fe737fff700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe710056d20 0x7fe7100591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.441+0000 7fe737fff700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe710056d20 0x7fe7100591e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fe72400b9a0 tx=0x7fe724013000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:35.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.444+0000 7fe734ff9700 1 -- 172.21.15.187:0/51093393 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe72c045680 con 0x7fe7380622c0 2020-11-20T12:14:35.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.633+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe738061a20 con 0x7fe7380622c0 2020-11-20T12:14:35.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.635+0000 7fe734ff9700 1 -- 172.21.15.187:0/51093393 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v98) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe72c048ca0 con 0x7fe7380622c0 2020-11-20T12:14:35.638 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:14:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe710056d20 msgr2=0x7fe7100591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe710056d20 0x7fe7100591e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fe72400b9a0 tx=0x7fe724013000).stop 2020-11-20T12:14:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7380622c0 msgr2=0x7fe73812bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7380622c0 0x7fe73812bb90 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7fe72c00ebd0 tx=0x7fe72c006df0).stop 2020-11-20T12:14:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 shutdown_connections 2020-11-20T12:14:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe710056d20 0x7fe7100591e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7381484d0 0x7fe73809dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7380622c0 0x7fe73812bb90 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 --2- 172.21.15.187:0/51093393 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe738066f70 0x7fe73812d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 >> 172.21.15.187:0/51093393 conn(0x7fe7380bab30 msgr2=0x7fe7380b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:35.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 shutdown_connections 2020-11-20T12:14:35.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.637+0000 7fe73df0b700 1 -- 172.21.15.187:0/51093393 wait complete. 2020-11-20T12:14:35.665 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:14:35.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.905+0000 7ffbba543700 1 Processor -- start 2020-11-20T12:14:35.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.905+0000 7ffbba543700 1 -- start start 2020-11-20T12:14:35.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.907+0000 7ffbba543700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffbb4065190 0x7ffbb4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.907+0000 7ffbba543700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb409ca90 0x7ffbb409cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.907+0000 7ffbba543700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb40b13e0 con 0x7ffbb4065f90 2020-11-20T12:14:35.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.907+0000 7ffbba543700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb414ab30 con 0x7ffbb4065190 2020-11-20T12:14:35.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.907+0000 7ffbba543700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb4062c10 con 0x7ffbb409ca90 2020-11-20T12:14:35.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.907+0000 7ffbb8d40700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ffbb4065f90 0x7ffbb4143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45756/0 (socket says 172.21.15.187:45756) 2020-11-20T12:14:35.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.907+0000 7ffbb8d40700 1 -- 172.21.15.187:0/4288534598 learned_addr learned my addr 172.21.15.187:0/4288534598 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:35.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.908+0000 7ffbb9d42700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb409ca90 0x7ffbb409cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.908+0000 7ffbb9541700 1 --2- 172.21.15.187:0/4288534598 >> v2:172.21.15.5:3300/0 conn(0x7ffbb4065190 0x7ffbb4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.908+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936317260 0 0) 0x7ffbb40b13e0 con 0x7ffbb4065f90 2020-11-20T12:14:35.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.908+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb9c003240 con 0x7ffbb4065f90 2020-11-20T12:14:35.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3313217038 0 0) 0x7ffb9c003240 con 0x7ffbb4065f90 2020-11-20T12:14:35.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb409ca90 msgr2=0x7ffbb409cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:35.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb409ca90 0x7ffbb409cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 >> v2:172.21.15.5:3300/0 conn(0x7ffbb4065190 msgr2=0x7ffbb4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:35.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 --2- 172.21.15.187:0/4288534598 >> v2:172.21.15.5:3300/0 conn(0x7ffbb4065190 0x7ffbb4065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb40b1640 con 0x7ffbb4065f90 2020-11-20T12:14:35.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ffba80014e0 con 0x7ffbb4065f90 2020-11-20T12:14:35.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 >> v1:172.21.15.187:6789/0 conn(0x7ffbb4065f90 legacy=0x7ffbb4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb409ca90 0x7ffb9c003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c003c40 0x7ffb9c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0049d0 0x7ffb9c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb414ab30 con 0x7ffb9c0049d0 2020-11-20T12:14:35.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb4062c10 con 0x7ffb9c003c40 2020-11-20T12:14:35.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c004200 con 0x7ffbb409ca90 2020-11-20T12:14:35.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.909+0000 7ffbb8d40700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c003c40 0x7ffb9c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.910+0000 7ffbb9541700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb409ca90 0x7ffb9c003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.910+0000 7ffbb9d42700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0049d0 0x7ffb9c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.910+0000 7ffbb8d40700 1 -- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb409ca90 msgr2=0x7ffb9c003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:35.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.910+0000 7ffbb8d40700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb409ca90 0x7ffb9c003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.910+0000 7ffbb8d40700 1 -- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0049d0 msgr2=0x7ffb9c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:35.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.910+0000 7ffbb8d40700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0049d0 0x7ffb9c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.910+0000 7ffbb8d40700 1 -- 172.21.15.187:0/4288534598 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c003240 con 0x7ffb9c003c40 2020-11-20T12:14:35.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.910+0000 7ffbb9d42700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb9c0049d0 0x7ffb9c004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:14:35.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.911+0000 7ffbb8d40700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c003c40 0x7ffb9c0044e0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7ffba800c410 tx=0x7ffbb40bb650).ready entity=mon.1 client_cookie=e17b338a39003d50 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:35.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.911+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba80070d0 con 0x7ffb9c003c40 2020-11-20T12:14:35.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.911+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba8007230 con 0x7ffb9c003c40 2020-11-20T12:14:35.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.911+0000 7ffba7fff700 1 -- 172.21.15.187:0/4288534598 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba80055f0 con 0x7ffb9c003c40 2020-11-20T12:14:35.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.912+0000 7ffbba543700 1 -- 172.21.15.187:0/4288534598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c003c40 msgr2=0x7ffb9c0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.912+0000 7ffbba543700 1 --2- 172.21.15.187:0/4288534598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb9c003c40 0x7ffb9c0044e0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7ffba800c410 tx=0x7ffbb40bb650).stop 2020-11-20T12:14:35.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.912+0000 7ffbb9d42700 1 -- 172.21.15.187:0/4288534598 reap_dead start 2020-11-20T12:14:35.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.912+0000 7ffbba543700 1 -- 172.21.15.187:0/4288534598 shutdown_connections 2020-11-20T12:14:35.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.912+0000 7ffbba543700 1 -- 172.21.15.187:0/4288534598 >> 172.21.15.187:0/4288534598 conn(0x7ffbb40bab30 msgr2=0x7ffbb40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:35.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.912+0000 7ffbba543700 1 -- 172.21.15.187:0/4288534598 shutdown_connections 2020-11-20T12:14:35.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.912+0000 7ffbba543700 1 -- 172.21.15.187:0/4288534598 wait complete. 2020-11-20T12:14:35.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.913+0000 7ffbba543700 1 Processor -- start 2020-11-20T12:14:35.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.914+0000 7ffbba543700 1 -- start start 2020-11-20T12:14:35.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.914+0000 7ffbba543700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40b2ec0 0x7ffbb40b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.914+0000 7ffbba543700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb40b32a0 0x7ffbb4144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.914+0000 7ffbba543700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb4145370 0x7ffbb409eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.914+0000 7ffbba543700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb414b3f0 con 0x7ffbb40b32a0 2020-11-20T12:14:35.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.914+0000 7ffbba543700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb414b690 con 0x7ffbb4145370 2020-11-20T12:14:35.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.914+0000 7ffbba543700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb414b930 con 0x7ffbb40b2ec0 2020-11-20T12:14:35.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9d42700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb4145370 0x7ffbb409eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9d42700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb4145370 0x7ffbb409eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49164/0 (socket says 172.21.15.187:49164) 2020-11-20T12:14:35.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9d42700 1 -- 172.21.15.187:0/3482133405 learned_addr learned my addr 172.21.15.187:0/3482133405 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:35.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9541700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40b2ec0 0x7ffbb40b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb8d40700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb40b32a0 0x7ffbb4144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9d42700 1 -- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40b2ec0 msgr2=0x7ffbb40b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9d42700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40b2ec0 0x7ffbb40b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9d42700 1 -- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb40b32a0 msgr2=0x7ffbb4144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:35.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9d42700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb40b32a0 0x7ffbb4144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:35.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9d42700 1 -- 172.21.15.187:0/3482133405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb41457d0 con 0x7ffbb4145370 2020-11-20T12:14:35.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.916+0000 7ffbb8d40700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb40b32a0 0x7ffbb4144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:35.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.915+0000 7ffbb9541700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40b2ec0 0x7ffbb40b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:35.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.916+0000 7ffbb9d42700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb4145370 0x7ffbb409eb80 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7ffbb40b1b80 tx=0x7ffbac008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:35.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.916+0000 7ffba57fa700 1 -- 172.21.15.187:0/3482133405 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac00ea30 con 0x7ffbb4145370 2020-11-20T12:14:35.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.916+0000 7ffba57fa700 1 -- 172.21.15.187:0/3482133405 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac00ebb0 con 0x7ffbb4145370 2020-11-20T12:14:35.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.916+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb40b4890 con 0x7ffbb4145370 2020-11-20T12:14:35.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.916+0000 7ffba57fa700 1 -- 172.21.15.187:0/3482133405 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac014430 con 0x7ffbb4145370 2020-11-20T12:14:35.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.916+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb409f070 con 0x7ffbb4145370 2020-11-20T12:14:35.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.917+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4061a20 con 0x7ffbb4145370 2020-11-20T12:14:35.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.918+0000 7ffba57fa700 1 -- 172.21.15.187:0/3482133405 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffbac01e030 con 0x7ffbb4145370 2020-11-20T12:14:35.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.918+0000 7ffba57fa700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:35.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.918+0000 7ffba57fa700 1 -- 172.21.15.187:0/3482133405 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffbac06fca0 con 0x7ffbb4145370 2020-11-20T12:14:35.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.919+0000 7ffbb9541700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:35.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.919+0000 7ffbb9541700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7ffbb0006fd0 tx=0x7ffbb0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:35.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:35.922+0000 7ffba57fa700 1 -- 172.21.15.187:0/3482133405 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffbac045120 con 0x7ffbb4145370 2020-11-20T12:14:36.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.107+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ffbb40afc60 con 0x7ffbb4145370 2020-11-20T12:14:36.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.108+0000 7ffba57fa700 1 -- 172.21.15.187:0/3482133405 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v98) v1 ==== 92+0+10 (secure 0 0 0) 0x7ffbac048740 con 0x7ffbb4145370 2020-11-20T12:14:36.112 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:36.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 msgr2=0x7ffb8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7ffbb0006fd0 tx=0x7ffbb0008040).stop 2020-11-20T12:14:36.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb4145370 msgr2=0x7ffbb409eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb4145370 0x7ffbb409eb80 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7ffbb40b1b80 tx=0x7ffbac008040).stop 2020-11-20T12:14:36.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 shutdown_connections 2020-11-20T12:14:36.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb8c056d20 0x7ffb8c0591e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb4145370 0x7ffbb409eb80 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb40b2ec0 0x7ffbb40b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 --2- 172.21.15.187:0/3482133405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb40b32a0 0x7ffbb4144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 >> 172.21.15.187:0/3482133405 conn(0x7ffbb40bab30 msgr2=0x7ffbb4145b90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:36.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 shutdown_connections 2020-11-20T12:14:36.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.111+0000 7ffbba543700 1 -- 172.21.15.187:0/3482133405 wait complete. 2020-11-20T12:14:36.136 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:14:36.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.396+0000 7f386bfff700 1 Processor -- start 2020-11-20T12:14:36.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.396+0000 7f386bfff700 1 -- start start 2020-11-20T12:14:36.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.397+0000 7f386bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.397+0000 7f386bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f386c0bfbe0 0x7f386c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.397+0000 7f386bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f386c09f560 0x7f386c1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.397+0000 7f386bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f386c14d110 con 0x7f386c0bfbe0 2020-11-20T12:14:36.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.397+0000 7f386bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f386c148aa0 con 0x7f386c09f560 2020-11-20T12:14:36.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.397+0000 7f386bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386c14cbd0 con 0x7f386c0bf800 2020-11-20T12:14:36.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f386c09f560 0x7f386c1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f386c09f560 0x7f386c1485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49172/0 (socket says 172.21.15.187:49172) 2020-11-20T12:14:36.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386b7fe700 1 -- 172.21.15.187:0/2230053965 learned_addr learned my addr 172.21.15.187:0/2230053965 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:36.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386affd700 1 --2- 172.21.15.187:0/2230053965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386b7fe700 1 -- 172.21.15.187:0/2230053965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 msgr2=0x7f386c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:36.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386b7fe700 1 --2- 172.21.15.187:0/2230053965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386b7fe700 1 -- 172.21.15.187:0/2230053965 >> v2:172.21.15.5:3300/0 conn(0x7f386c0bfbe0 msgr2=0x7f386c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:36.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f38627fc700 1 --2- 172.21.15.187:0/2230053965 >> v2:172.21.15.5:3300/0 conn(0x7f386c0bfbe0 0x7f386c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386b7fe700 1 --2- 172.21.15.187:0/2230053965 >> v2:172.21.15.5:3300/0 conn(0x7f386c0bfbe0 0x7f386c1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.398+0000 7f386b7fe700 1 -- 172.21.15.187:0/2230053965 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c14ce70 con 0x7f386c09f560 2020-11-20T12:14:36.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.399+0000 7f386affd700 1 --2- 172.21.15.187:0/2230053965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:36.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.399+0000 7f38627fc700 1 --2- 172.21.15.187:0/2230053965 >> v2:172.21.15.5:3300/0 conn(0x7f386c0bfbe0 0x7f386c1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:36.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.399+0000 7f386b7fe700 1 --2- 172.21.15.187:0/2230053965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c09f560 0x7f386c1485b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f386c080b40 tx=0x7f3864006eb0).ready entity=mon.1 client_cookie=1d6d93fe9468a48d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:36.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.399+0000 7f386a7fc700 1 -- 172.21.15.187:0/2230053965 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864009960 con 0x7f386c09f560 2020-11-20T12:14:36.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.399+0000 7f386a7fc700 1 -- 172.21.15.187:0/2230053965 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3864009ac0 con 0x7f386c09f560 2020-11-20T12:14:36.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.399+0000 7f386a7fc700 1 -- 172.21.15.187:0/2230053965 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864009e40 con 0x7f386c09f560 2020-11-20T12:14:36.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 -- 172.21.15.187:0/2230053965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c09f560 msgr2=0x7f386c1485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 --2- 172.21.15.187:0/2230053965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c09f560 0x7f386c1485b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f386c080b40 tx=0x7f3864006eb0).stop 2020-11-20T12:14:36.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 -- 172.21.15.187:0/2230053965 shutdown_connections 2020-11-20T12:14:36.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 --2- 172.21.15.187:0/2230053965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c09f560 0x7f386c1485b0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 --2- 172.21.15.187:0/2230053965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 --2- 172.21.15.187:0/2230053965 >> v2:172.21.15.5:3300/0 conn(0x7f386c0bfbe0 0x7f386c1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 -- 172.21.15.187:0/2230053965 >> 172.21.15.187:0/2230053965 conn(0x7f386c068370 msgr2=0x7f386c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 -- 172.21.15.187:0/2230053965 shutdown_connections 2020-11-20T12:14:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.400+0000 7f386bfff700 1 -- 172.21.15.187:0/2230053965 wait complete. 2020-11-20T12:14:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.402+0000 7f386bfff700 1 Processor -- start 2020-11-20T12:14:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.402+0000 7f386bfff700 1 -- start start 2020-11-20T12:14:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.403+0000 7f386bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f386c09f560 0x7f386c157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.403+0000 7f386bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.403+0000 7f386bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c15c260 0x7f386c0bb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.403+0000 7f386bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c141360 con 0x7f386c09f560 2020-11-20T12:14:36.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.403+0000 7f386bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c158610 con 0x7f386c15c260 2020-11-20T12:14:36.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.403+0000 7f386bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386c1588b0 con 0x7f386c0bf800 2020-11-20T12:14:36.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.404+0000 7f386b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c15c260 0x7f386c0bb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.404+0000 7f386b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c15c260 0x7f386c0bb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49178/0 (socket says 172.21.15.187:49178) 2020-11-20T12:14:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.404+0000 7f386b7fe700 1 -- 172.21.15.187:0/911559165 learned_addr learned my addr 172.21.15.187:0/911559165 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.404+0000 7f38627fc700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386b7fe700 1 -- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 msgr2=0x7f386c15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386b7fe700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386affd700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f386c09f560 0x7f386c157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386b7fe700 1 -- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f386c09f560 msgr2=0x7f386c157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386b7fe700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f386c09f560 0x7f386c157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386b7fe700 1 -- 172.21.15.187:0/911559165 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c158b50 con 0x7f386c15c260 2020-11-20T12:14:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386affd700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f386c09f560 0x7f386c157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386b7fe700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c15c260 0x7f386c0bb700 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f3864006fd0 tx=0x7f3864015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f38637fe700 1 -- 172.21.15.187:0/911559165 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38640106c0 con 0x7f386c15c260 2020-11-20T12:14:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f38637fe700 1 -- 172.21.15.187:0/911559165 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3864010820 con 0x7f386c15c260 2020-11-20T12:14:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386c0bbbf0 con 0x7f386c15c260 2020-11-20T12:14:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f38637fe700 1 -- 172.21.15.187:0/911559165 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864010bc0 con 0x7f386c15c260 2020-11-20T12:14:36.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.405+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386c146b70 con 0x7f386c15c260 2020-11-20T12:14:36.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.406+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386c080050 con 0x7f386c15c260 2020-11-20T12:14:36.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.407+0000 7f38637fe700 1 -- 172.21.15.187:0/911559165 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3864025030 con 0x7f386c15c260 2020-11-20T12:14:36.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.408+0000 7f38637fe700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3848056d20 0x7f38480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.408+0000 7f38637fe700 1 -- 172.21.15.187:0/911559165 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3864071a20 con 0x7f386c15c260 2020-11-20T12:14:36.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.408+0000 7f386affd700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3848056d20 0x7f38480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.409+0000 7f386affd700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3848056d20 0x7f38480591e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f385c008b60 tx=0x7f385c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:36.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.412+0000 7f38637fe700 1 -- 172.21.15.187:0/911559165 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3864046f20 con 0x7f386c15c260 2020-11-20T12:14:36.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.599+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f386c158f10 con 0x7f386c15c260 2020-11-20T12:14:36.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.600+0000 7f38637fe700 1 -- 172.21.15.187:0/911559165 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3864023070 con 0x7f386c15c260 2020-11-20T12:14:36.604 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:36.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.602+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3848056d20 msgr2=0x7f38480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.602+0000 7f386bfff700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3848056d20 0x7f38480591e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f385c008b60 tx=0x7f385c008040).stop 2020-11-20T12:14:36.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.602+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c15c260 msgr2=0x7f386c0bb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.602+0000 7f386bfff700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c15c260 0x7f386c0bb700 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f3864006fd0 tx=0x7f3864015000).stop 2020-11-20T12:14:36.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.603+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 shutdown_connections 2020-11-20T12:14:36.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.603+0000 7f386bfff700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3848056d20 0x7f38480591e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.603+0000 7f386bfff700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f386c09f560 0x7f386c157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.603+0000 7f386bfff700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f386c15c260 0x7f386c0bb700 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.603+0000 7f386bfff700 1 --2- 172.21.15.187:0/911559165 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f386c0bf800 0x7f386c15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.603+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 >> 172.21.15.187:0/911559165 conn(0x7f386c068370 msgr2=0x7f386c060c90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:36.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.603+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 shutdown_connections 2020-11-20T12:14:36.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.603+0000 7f386bfff700 1 -- 172.21.15.187:0/911559165 wait complete. 2020-11-20T12:14:36.630 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:14:36.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.872+0000 7fbd11ccd700 1 Processor -- start 2020-11-20T12:14:36.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.872+0000 7fbd11ccd700 1 -- start start 2020-11-20T12:14:36.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.874+0000 7fbd11ccd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbd0c1484d0 0x7fbd0c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.874+0000 7fbd11ccd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c0622c0 0x7fbd0c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.874+0000 7fbd11ccd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c0b1300 con 0x7fbd0c066f70 2020-11-20T12:14:36.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.874+0000 7fbd11ccd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0c067730 con 0x7fbd0c1484d0 2020-11-20T12:14:36.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.874+0000 7fbd11ccd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0b15a0 con 0x7fbd0c0622c0 2020-11-20T12:14:36.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.875+0000 7fbd03fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c0622c0 0x7fbd0c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.875+0000 7fbd03fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c0622c0 0x7fbd0c0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:50070/0 (socket says 172.21.15.187:50070) 2020-11-20T12:14:36.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.875+0000 7fbd03fff700 1 -- 172.21.15.187:0/2476704877 learned_addr learned my addr 172.21.15.187:0/2476704877 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:36.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.875+0000 7fbd10ccb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbd0c1484d0 0x7fbd0c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.876+0000 7fbd03fff700 1 -- 172.21.15.187:0/2476704877 >> v2:172.21.15.5:3300/0 conn(0x7fbd0c1484d0 msgr2=0x7fbd0c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:36.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.876+0000 7fbd03fff700 1 --2- 172.21.15.187:0/2476704877 >> v2:172.21.15.5:3300/0 conn(0x7fbd0c1484d0 0x7fbd0c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.876+0000 7fbd03fff700 1 -- 172.21.15.187:0/2476704877 >> v1:172.21.15.5:6789/0 conn(0x7fbd0c066f70 legacy=0x7fbd0c0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.876+0000 7fbd03fff700 1 -- 172.21.15.187:0/2476704877 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c0b1840 con 0x7fbd0c0622c0 2020-11-20T12:14:36.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.876+0000 7fbd03fff700 1 --2- 172.21.15.187:0/2476704877 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c0622c0 0x7fbd0c0b5640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbcfc006fd0 tx=0x7fbcfc006f40).ready entity=mon.2 client_cookie=fc0f44972c872efb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:36.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd037fe700 1 -- 172.21.15.187:0/2476704877 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00dc60 con 0x7fbd0c0622c0 2020-11-20T12:14:36.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd037fe700 1 -- 172.21.15.187:0/2476704877 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc00ddc0 con 0x7fbd0c0622c0 2020-11-20T12:14:36.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd037fe700 1 -- 172.21.15.187:0/2476704877 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc010680 con 0x7fbd0c0622c0 2020-11-20T12:14:36.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd11ccd700 1 -- 172.21.15.187:0/2476704877 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c0622c0 msgr2=0x7fbd0c0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/2476704877 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c0622c0 0x7fbd0c0b5640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbcfc006fd0 tx=0x7fbcfc006f40).stop 2020-11-20T12:14:36.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd11ccd700 1 -- 172.21.15.187:0/2476704877 shutdown_connections 2020-11-20T12:14:36.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/2476704877 >> v2:172.21.15.5:3300/0 conn(0x7fbd0c1484d0 0x7fbd0c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/2476704877 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c0622c0 0x7fbd0c0b5640 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd11ccd700 1 -- 172.21.15.187:0/2476704877 >> 172.21.15.187:0/2476704877 conn(0x7fbd0c0bab30 msgr2=0x7fbd0c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:36.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.877+0000 7fbd11ccd700 1 -- 172.21.15.187:0/2476704877 shutdown_connections 2020-11-20T12:14:36.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.878+0000 7fbd11ccd700 1 -- 172.21.15.187:0/2476704877 wait complete. 2020-11-20T12:14:36.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.879+0000 7fbd11ccd700 1 Processor -- start 2020-11-20T12:14:36.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.879+0000 7fbd11ccd700 1 -- start start 2020-11-20T12:14:36.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.880+0000 7fbd11ccd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd0c0622c0 0x7fbd0c0b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.880+0000 7fbd11ccd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c066f70 0x7fbd0c158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.880+0000 7fbd11ccd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd0c1484d0 0x7fbd0c149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.880+0000 7fbd11ccd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c158530 con 0x7fbd0c1484d0 2020-11-20T12:14:36.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.880+0000 7fbd11ccd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c143d90 con 0x7fbd0c0622c0 2020-11-20T12:14:36.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.880+0000 7fbd11ccd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c144030 con 0x7fbd0c066f70 2020-11-20T12:14:36.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd0c0622c0 0x7fbd0c0b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd0c0622c0 0x7fbd0c0b2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49188/0 (socket says 172.21.15.187:49188) 2020-11-20T12:14:36.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 -- 172.21.15.187:0/24786079 learned_addr learned my addr 172.21.15.187:0/24786079 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:36.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd03fff700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c066f70 0x7fbd0c158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd114cc700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd0c1484d0 0x7fbd0c149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 -- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c066f70 msgr2=0x7fbd0c158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c066f70 0x7fbd0c158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 -- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd0c1484d0 msgr2=0x7fbd0c149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:36.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd0c1484d0 0x7fbd0c149d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:36.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 -- 172.21.15.187:0/24786079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c1442d0 con 0x7fbd0c0622c0 2020-11-20T12:14:36.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd03fff700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c066f70 0x7fbd0c158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:36.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.881+0000 7fbd10ccb700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd0c0622c0 0x7fbd0c0b2bc0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7fbd0800ebd0 tx=0x7fbd08006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:36.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.882+0000 7fbd00ff9700 1 -- 172.21.15.187:0/24786079 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0800ba30 con 0x7fbd0c0622c0 2020-11-20T12:14:36.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.882+0000 7fbd00ff9700 1 -- 172.21.15.187:0/24786079 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0800bb90 con 0x7fbd0c0622c0 2020-11-20T12:14:36.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.882+0000 7fbd00ff9700 1 -- 172.21.15.187:0/24786079 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd080123f0 con 0x7fbd0c0622c0 2020-11-20T12:14:36.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.882+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c09ef70 con 0x7fbd0c0622c0 2020-11-20T12:14:36.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.882+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c0a0490 con 0x7fbd0c0622c0 2020-11-20T12:14:36.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.883+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c061a20 con 0x7fbd0c0622c0 2020-11-20T12:14:36.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.884+0000 7fbd00ff9700 1 -- 172.21.15.187:0/24786079 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbd08012760 con 0x7fbd0c0622c0 2020-11-20T12:14:36.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.884+0000 7fbd00ff9700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbce4056d20 0x7fbce40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:36.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.884+0000 7fbd00ff9700 1 -- 172.21.15.187:0/24786079 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbd080704a0 con 0x7fbd0c0622c0 2020-11-20T12:14:36.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.885+0000 7fbd03fff700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbce4056d20 0x7fbce40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:36.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.885+0000 7fbd03fff700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbce4056d20 0x7fbce40591e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fbcfc00d960 tx=0x7fbcfc006f40).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:36.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:36.888+0000 7fbd00ff9700 1 -- 172.21.15.187:0/24786079 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbd08045870 con 0x7fbd0c0622c0 2020-11-20T12:14:37.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.069+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd0c1488b0 con 0x7fbd0c0622c0 2020-11-20T12:14:37.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.070+0000 7fbd00ff9700 1 -- 172.21.15.187:0/24786079 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+10 (secure 0 0 0) 0x7fbd08048e90 con 0x7fbd0c0622c0 2020-11-20T12:14:37.074 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:37.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbce4056d20 msgr2=0x7fbce40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:37.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbce4056d20 0x7fbce40591e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fbcfc00d960 tx=0x7fbcfc006f40).stop 2020-11-20T12:14:37.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd0c0622c0 msgr2=0x7fbd0c0b2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:37.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd0c0622c0 0x7fbd0c0b2bc0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7fbd0800ebd0 tx=0x7fbd08006df0).stop 2020-11-20T12:14:37.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 shutdown_connections 2020-11-20T12:14:37.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbce4056d20 0x7fbce40591e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd0c1484d0 0x7fbd0c149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd0c0622c0 0x7fbd0c0b2bc0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 --2- 172.21.15.187:0/24786079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbd0c066f70 0x7fbd0c158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.072+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 >> 172.21.15.187:0/24786079 conn(0x7fbd0c0bab30 msgr2=0x7fbd0c0ba260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:37.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.073+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 shutdown_connections 2020-11-20T12:14:37.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.073+0000 7fbd11ccd700 1 -- 172.21.15.187:0/24786079 wait complete. 2020-11-20T12:14:37.096 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:14:37.097 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:37.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.323+0000 7f32d92d7700 1 Processor -- start 2020-11-20T12:14:37.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.324+0000 7f32d92d7700 1 -- start start 2020-11-20T12:14:37.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.325+0000 7f32d92d7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32d41484d0 0x7f32d4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:37.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.325+0000 7f32d92d7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f32d40622c0 0x7f32d40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:37.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.325+0000 7f32d92d7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40b1300 con 0x7f32d4066f70 2020-11-20T12:14:37.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.325+0000 7f32d92d7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d4067730 con 0x7f32d40622c0 2020-11-20T12:14:37.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.325+0000 7f32d92d7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d40b15a0 con 0x7f32d41484d0 2020-11-20T12:14:37.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.325+0000 7f32d3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32d41484d0 0x7f32d4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:37.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.325+0000 7f32d8ad6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f32d4066f70 0x7f32d40b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45806/0 (socket says 172.21.15.187:45806) 2020-11-20T12:14:37.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.325+0000 7f32d8ad6700 1 -- 172.21.15.187:0/670703629 learned_addr learned my addr 172.21.15.187:0/670703629 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:37.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.326+0000 7f32d37fe700 1 --2- 172.21.15.187:0/670703629 >> v2:172.21.15.5:3300/0 conn(0x7f32d40622c0 0x7f32d40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:37.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.326+0000 7f32d2ffd700 1 -- 172.21.15.187:0/670703629 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706887258 0 0) 0x7f32d40b1300 con 0x7f32d4066f70 2020-11-20T12:14:37.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.326+0000 7f32d2ffd700 1 -- 172.21.15.187:0/670703629 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c0003240 con 0x7f32d4066f70 2020-11-20T12:14:37.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.326+0000 7f32d3fff700 1 -- 172.21.15.187:0/670703629 >> v2:172.21.15.5:3300/0 conn(0x7f32d40622c0 msgr2=0x7f32d40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:37.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.326+0000 7f32d3fff700 1 --2- 172.21.15.187:0/670703629 >> v2:172.21.15.5:3300/0 conn(0x7f32d40622c0 0x7f32d40b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.326+0000 7f32d3fff700 1 -- 172.21.15.187:0/670703629 >> v1:172.21.15.187:6789/0 conn(0x7f32d4066f70 legacy=0x7f32d40b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:37.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.326+0000 7f32d3fff700 1 -- 172.21.15.187:0/670703629 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40b1840 con 0x7f32d41484d0 2020-11-20T12:14:37.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d3fff700 1 --2- 172.21.15.187:0/670703629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d41484d0 0x7f32d4061dd0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f32d40b85e0 tx=0x7f32c4006df0).ready entity=mon.1 client_cookie=5c1d239d09a70777 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:37.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d2ffd700 1 -- 172.21.15.187:0/670703629 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c40097b0 con 0x7f32d41484d0 2020-11-20T12:14:37.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d2ffd700 1 -- 172.21.15.187:0/670703629 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c4009910 con 0x7f32d41484d0 2020-11-20T12:14:37.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d2ffd700 1 -- 172.21.15.187:0/670703629 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4009c90 con 0x7f32d41484d0 2020-11-20T12:14:37.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d92d7700 1 -- 172.21.15.187:0/670703629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d41484d0 msgr2=0x7f32d4061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:37.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d92d7700 1 --2- 172.21.15.187:0/670703629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d41484d0 0x7f32d4061dd0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f32d40b85e0 tx=0x7f32c4006df0).stop 2020-11-20T12:14:37.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d92d7700 1 -- 172.21.15.187:0/670703629 shutdown_connections 2020-11-20T12:14:37.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d92d7700 1 --2- 172.21.15.187:0/670703629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d41484d0 0x7f32d4061dd0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d92d7700 1 --2- 172.21.15.187:0/670703629 >> v2:172.21.15.5:3300/0 conn(0x7f32d40622c0 0x7f32d40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.327+0000 7f32d92d7700 1 -- 172.21.15.187:0/670703629 >> 172.21.15.187:0/670703629 conn(0x7f32d40bab30 msgr2=0x7f32d40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:37.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.328+0000 7f32d92d7700 1 -- 172.21.15.187:0/670703629 shutdown_connections 2020-11-20T12:14:37.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.328+0000 7f32d92d7700 1 -- 172.21.15.187:0/670703629 wait complete. 2020-11-20T12:14:37.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.329+0000 7f32d92d7700 1 Processor -- start 2020-11-20T12:14:37.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.330+0000 7f32d92d7700 1 -- start start 2020-11-20T12:14:37.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.330+0000 7f32d92d7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32d40622c0 0x7f32d4134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:37.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.330+0000 7f32d92d7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d4066f70 0x7f32d4135480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:37.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.330+0000 7f32d92d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32d40a0710 0x7f32d413d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:37.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.330+0000 7f32d92d7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4135970 con 0x7f32d40622c0 2020-11-20T12:14:37.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.330+0000 7f32d92d7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d409e590 con 0x7f32d4066f70 2020-11-20T12:14:37.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.330+0000 7f32d92d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d409e830 con 0x7f32d40a0710 2020-11-20T12:14:37.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d4066f70 0x7f32d4135480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:37.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d4066f70 0x7f32d4135480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49204/0 (socket says 172.21.15.187:49204) 2020-11-20T12:14:37.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 -- 172.21.15.187:0/4149620693 learned_addr learned my addr 172.21.15.187:0/4149620693 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:37.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 -- 172.21.15.187:0/4149620693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32d40a0710 msgr2=0x7f32d413d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:37.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32d40a0710 0x7f32d413d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:37.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 -- 172.21.15.187:0/4149620693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32d40622c0 msgr2=0x7f32d4134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:37.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32d40622c0 0x7f32d4134f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:37.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 -- 172.21.15.187:0/4149620693 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d409ead0 con 0x7f32d4066f70 2020-11-20T12:14:37.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.331+0000 7f32d37fe700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d4066f70 0x7f32d4135480 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f32c800db60 tx=0x7f32c8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:37.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.332+0000 7f32b3fff700 1 -- 172.21.15.187:0/4149620693 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c800b170 con 0x7f32d4066f70 2020-11-20T12:14:37.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.332+0000 7f32b3fff700 1 -- 172.21.15.187:0/4149620693 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c8009790 con 0x7f32d4066f70 2020-11-20T12:14:37.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.332+0000 7f32b3fff700 1 -- 172.21.15.187:0/4149620693 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8009b50 con 0x7f32d4066f70 2020-11-20T12:14:37.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.332+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d409ed70 con 0x7f32d4066f70 2020-11-20T12:14:37.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.332+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d413d5d0 con 0x7f32d4066f70 2020-11-20T12:14:37.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.333+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d4061a20 con 0x7f32d4066f70 2020-11-20T12:14:37.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.333+0000 7f32b3fff700 1 -- 172.21.15.187:0/4149620693 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f32c801c460 con 0x7f32d4066f70 2020-11-20T12:14:37.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.334+0000 7f32b3fff700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f32ac056d20 0x7f32ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:37.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.334+0000 7f32b3fff700 1 -- 172.21.15.187:0/4149620693 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f32c8070370 con 0x7f32d4066f70 2020-11-20T12:14:37.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.334+0000 7f32d3fff700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f32ac056d20 0x7f32ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:37.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.335+0000 7f32d3fff700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f32ac056d20 0x7f32ac0591e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f32d4066540 tx=0x7f32c4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:37.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.338+0000 7f32b3fff700 1 -- 172.21.15.187:0/4149620693 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f32c8045650 con 0x7f32d4066f70 2020-11-20T12:14:37.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.583+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32d41488b0 con 0x7f32d4066f70 2020-11-20T12:14:37.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.584+0000 7f32b3fff700 1 -- 172.21.15.187:0/4149620693 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f32c8048c70 con 0x7f32d4066f70 2020-11-20T12:14:37.588 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:37.588 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:37.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f32ac056d20 msgr2=0x7f32ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:37.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f32ac056d20 0x7f32ac0591e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f32d4066540 tx=0x7f32c4015000).stop 2020-11-20T12:14:37.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d4066f70 msgr2=0x7f32d4135480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d4066f70 0x7f32d4135480 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f32c800db60 tx=0x7f32c8008040).stop 2020-11-20T12:14:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 shutdown_connections 2020-11-20T12:14:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f32ac056d20 0x7f32ac0591e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32d40622c0 0x7f32d4134f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:37.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32d40a0710 0x7f32d413d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:37.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 --2- 172.21.15.187:0/4149620693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d4066f70 0x7f32d4135480 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:37.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.586+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 >> 172.21.15.187:0/4149620693 conn(0x7f32d40bab30 msgr2=0x7f32d40b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:37.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.587+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 shutdown_connections 2020-11-20T12:14:37.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:37.587+0000 7f32d92d7700 1 -- 172.21.15.187:0/4149620693 wait complete. 2020-11-20T12:14:37.593 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:37.816 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:14:29.225459+0000', 'modified': '2020-11-20T12:14:31.424296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7801}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7801': {'gid': 7801, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/2663757618', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2663757618}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2663757618}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:14:37.816 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:14:37.817 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:14:38.096 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":7801,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":8.8391649979999993,"mdsmap_epoch":87,"osdmap_epoch":98,"osdmap_epoch_barrier":98,"uptime":14.513102848000001} 2020-11-20T12:14:38.096 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 7801, "mdsmap_epoch": 87, "osdmap_epoch": 98, "osdmap_epoch_barrier": 98, "rank_uptime": 8.839164998, "state": "up:active", "uptime": 14.513102848, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:14:38.097 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:14:38.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:38.098 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:14:38.152 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:14:38.153 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:14:38.293 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:14:38.293 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:14:38.331 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:14:38.332 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:14:38.338 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:14:38.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:38.339 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:14:38.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:14:38.430 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo dd if=/dev/urandom of=file_a bs=1M conv=fdatasync count=8 seek=0) 2020-11-20T12:14:38.726 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records in 2020-11-20T12:14:38.726 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records out 2020-11-20T12:14:38.726 INFO:teuthology.orchestra.run.smithi187.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.183673 s, 45.7 MB/s 2020-11-20T12:14:38.734 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:14:38.734 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:14:38.734 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:14:38.735 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:14:38.735 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/file_a").st_ino) 2020-11-20T12:14:38.736 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:14:38.920 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627776 2020-11-20T12:14:39.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:14:39.108 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo dd if=/dev/urandom of=file_b bs=1M conv=fdatasync count=8 seek=0) 2020-11-20T12:14:39.393 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records in 2020-11-20T12:14:39.394 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records out 2020-11-20T12:14:39.394 INFO:teuthology.orchestra.run.smithi187.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.180799 s, 46.4 MB/s 2020-11-20T12:14:39.401 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:14:39.402 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:14:39.402 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:14:39.402 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:14:39.403 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/file_b").st_ino) 2020-11-20T12:14:39.403 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:14:39.578 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627777 2020-11-20T12:14:39.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:14:39.766 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo ln file_b linkto_b) 2020-11-20T12:14:39.877 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:14:39.877 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:14:39.878 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:14:39.878 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:14:39.879 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/linkto_b").st_ino) 2020-11-20T12:14:39.879 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:14:40.063 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627777 2020-11-20T12:14:40.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:14:40.251 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mv file_a file_b) 2020-11-20T12:14:40.425 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:40.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.668+0000 7f794f401700 1 Processor -- start 2020-11-20T12:14:40.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.669+0000 7f794f401700 1 -- start start 2020-11-20T12:14:40.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.670+0000 7f794f401700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f79481484d0 0x7f7948061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:40.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.670+0000 7f794f401700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79480622c0 0x7f79480b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:40.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.670+0000 7f794f401700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7948066f70 0x7f79480b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:40.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.670+0000 7f794f401700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7948067730 con 0x7f7948066f70 2020-11-20T12:14:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.670+0000 7f794f401700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79480b1300 con 0x7f79481484d0 2020-11-20T12:14:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.670+0000 7f794f401700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79480b15a0 con 0x7f79480622c0 2020-11-20T12:14:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794e3ff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f79481484d0 0x7f7948061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794e3ff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f79481484d0 0x7f7948061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49220/0 (socket says 172.21.15.187:49220) 2020-11-20T12:14:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794e3ff700 1 -- 172.21.15.187:0/3286589798 learned_addr learned my addr 172.21.15.187:0/3286589798 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:40.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794dbfe700 1 --2- 172.21.15.187:0/3286589798 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79480622c0 0x7f79480b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:40.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794ec00700 1 --2- 172.21.15.187:0/3286589798 >> v2:172.21.15.5:3300/0 conn(0x7f7948066f70 0x7f79480b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:40.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794e3ff700 1 -- 172.21.15.187:0/3286589798 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79480622c0 msgr2=0x7f79480b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:40.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794e3ff700 1 --2- 172.21.15.187:0/3286589798 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79480622c0 0x7f79480b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794e3ff700 1 -- 172.21.15.187:0/3286589798 >> v2:172.21.15.5:3300/0 conn(0x7f7948066f70 msgr2=0x7f79480b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:40.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.671+0000 7f794e3ff700 1 --2- 172.21.15.187:0/3286589798 >> v2:172.21.15.5:3300/0 conn(0x7f7948066f70 0x7f79480b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.672+0000 7f794e3ff700 1 -- 172.21.15.187:0/3286589798 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480b1840 con 0x7f79481484d0 2020-11-20T12:14:40.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.672+0000 7f794e3ff700 1 --2- 172.21.15.187:0/3286589798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79481484d0 0x7f7948061dd0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7f79480b85e0 tx=0x7f7944006eb0).ready entity=mon.1 client_cookie=915d9ae06b52070c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:40.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.672+0000 7f794d3fd700 1 -- 172.21.15.187:0/3286589798 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79440098d0 con 0x7f79481484d0 2020-11-20T12:14:40.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.672+0000 7f794d3fd700 1 -- 172.21.15.187:0/3286589798 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7944009a30 con 0x7f79481484d0 2020-11-20T12:14:40.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.672+0000 7f794d3fd700 1 -- 172.21.15.187:0/3286589798 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944009db0 con 0x7f79481484d0 2020-11-20T12:14:40.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 -- 172.21.15.187:0/3286589798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79481484d0 msgr2=0x7f7948061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:40.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 --2- 172.21.15.187:0/3286589798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79481484d0 0x7f7948061dd0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7f79480b85e0 tx=0x7f7944006eb0).stop 2020-11-20T12:14:40.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 -- 172.21.15.187:0/3286589798 shutdown_connections 2020-11-20T12:14:40.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 --2- 172.21.15.187:0/3286589798 >> v2:172.21.15.5:3300/0 conn(0x7f7948066f70 0x7f79480b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 --2- 172.21.15.187:0/3286589798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79481484d0 0x7f7948061dd0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 --2- 172.21.15.187:0/3286589798 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79480622c0 0x7f79480b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 -- 172.21.15.187:0/3286589798 >> 172.21.15.187:0/3286589798 conn(0x7f79480bab30 msgr2=0x7f79480baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:40.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 -- 172.21.15.187:0/3286589798 shutdown_connections 2020-11-20T12:14:40.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.673+0000 7f794f401700 1 -- 172.21.15.187:0/3286589798 wait complete. 2020-11-20T12:14:40.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.675+0000 7f794f401700 1 Processor -- start 2020-11-20T12:14:40.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.675+0000 7f794f401700 1 -- start start 2020-11-20T12:14:40.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.676+0000 7f794f401700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480622c0 0x7f7948159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:40.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.676+0000 7f794f401700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7948066f70 0x7f794809e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:40.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.676+0000 7f794f401700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79481484d0 0x7f7948139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:40.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.676+0000 7f794f401700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794815a070 con 0x7f79481484d0 2020-11-20T12:14:40.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.676+0000 7f794f401700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79480b49f0 con 0x7f79480622c0 2020-11-20T12:14:40.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.676+0000 7f794f401700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7948159dd0 con 0x7f7948066f70 2020-11-20T12:14:40.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794e3ff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480622c0 0x7f7948159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:40.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794e3ff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480622c0 0x7f7948159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49226/0 (socket says 172.21.15.187:49226) 2020-11-20T12:14:40.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794e3ff700 1 -- 172.21.15.187:0/4142583211 learned_addr learned my addr 172.21.15.187:0/4142583211 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:40.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794dbfe700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7948066f70 0x7f794809e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:40.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794e3ff700 1 -- 172.21.15.187:0/4142583211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7948066f70 msgr2=0x7f794809e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:40.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794e3ff700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7948066f70 0x7f794809e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794e3ff700 1 -- 172.21.15.187:0/4142583211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79481484d0 msgr2=0x7f7948139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:40.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794e3ff700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79481484d0 0x7f7948139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:40.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.677+0000 7f794e3ff700 1 -- 172.21.15.187:0/4142583211 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480b46e0 con 0x7f79480622c0 2020-11-20T12:14:40.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.678+0000 7f794e3ff700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480622c0 0x7f7948159500 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f794400c010 tx=0x7f7944015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:40.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.678+0000 7f79367fc700 1 -- 172.21.15.187:0/4142583211 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794401e460 con 0x7f79480622c0 2020-11-20T12:14:40.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.678+0000 7f79367fc700 1 -- 172.21.15.187:0/4142583211 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794401e5c0 con 0x7f79480622c0 2020-11-20T12:14:40.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.678+0000 7f79367fc700 1 -- 172.21.15.187:0/4142583211 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794401e940 con 0x7f79480622c0 2020-11-20T12:14:40.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.678+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79481599f0 con 0x7f79480622c0 2020-11-20T12:14:40.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.678+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794809e870 con 0x7f79480622c0 2020-11-20T12:14:40.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.679+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7948099e60 con 0x7f79480622c0 2020-11-20T12:14:40.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.680+0000 7f79367fc700 1 -- 172.21.15.187:0/4142583211 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f794401a040 con 0x7f79480622c0 2020-11-20T12:14:40.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.681+0000 7f79367fc700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7920056d20 0x7f79200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:40.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.681+0000 7f79367fc700 1 -- 172.21.15.187:0/4142583211 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7944071390 con 0x7f79480622c0 2020-11-20T12:14:40.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.681+0000 7f794dbfe700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7920056d20 0x7f79200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:40.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.682+0000 7f794dbfe700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7920056d20 0x7f79200591e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f794809bb00 tx=0x7f7938008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:40.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.685+0000 7f79367fc700 1 -- 172.21.15.187:0/4142583211 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7944046890 con 0x7f79480622c0 2020-11-20T12:14:40.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.931+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7948061a20 con 0x7f79480622c0 2020-11-20T12:14:40.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.932+0000 7f79367fc700 1 -- 172.21.15.187:0/4142583211 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f7944023070 con 0x7f79480622c0 2020-11-20T12:14:40.936 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:40.937 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:40.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7920056d20 msgr2=0x7f79200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:40.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7920056d20 0x7f79200591e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f794809bb00 tx=0x7f7938008040).stop 2020-11-20T12:14:40.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480622c0 msgr2=0x7f7948159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:40.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480622c0 0x7f7948159500 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f794400c010 tx=0x7f7944015000).stop 2020-11-20T12:14:40.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 shutdown_connections 2020-11-20T12:14:40.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7920056d20 0x7f79200591e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79481484d0 0x7f7948139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:40.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480622c0 0x7f7948159500 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 --2- 172.21.15.187:0/4142583211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7948066f70 0x7f794809e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:40.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 >> 172.21.15.187:0/4142583211 conn(0x7f79480bab30 msgr2=0x7f7948143d60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:40.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 shutdown_connections 2020-11-20T12:14:40.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:40.935+0000 7f794f401700 1 -- 172.21.15.187:0/4142583211 wait complete. 2020-11-20T12:14:40.942 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:40.960 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2020-11-20T12:14:41.232 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"num_strays":1}} 2020-11-20T12:14:41.234 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2020-11-20T12:14:41.236 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2020-11-20T12:14:46.238 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.490+0000 7f6153fff700 1 Processor -- start 2020-11-20T12:14:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.490+0000 7f6153fff700 1 -- start start 2020-11-20T12:14:46.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.492+0000 7f6153fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61540b1380 0x7f6154150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.492+0000 7f6153fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f615414c260 0x7f615414c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.492+0000 7f6153fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61540658a0 con 0x7f61540b1760 2020-11-20T12:14:46.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.492+0000 7f6153fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6154150390 con 0x7f61540b1380 2020-11-20T12:14:46.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.492+0000 7f6153fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6154150630 con 0x7f615414c260 2020-11-20T12:14:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.492+0000 7f614a7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f61540b1760 0x7f6154065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45840/0 (socket says 172.21.15.187:45840) 2020-11-20T12:14:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.492+0000 7f614a7fc700 1 -- 172.21.15.187:0/452234811 learned_addr learned my addr 172.21.15.187:0/452234811 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.492+0000 7f61537fe700 1 --2- 172.21.15.187:0/452234811 >> v2:172.21.15.187:3300/0 conn(0x7f615414c260 0x7f615414c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f6152ffd700 1 --2- 172.21.15.187:0/452234811 >> v2:172.21.15.5:3300/0 conn(0x7f61540b1380 0x7f6154150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965133518 0 0) 0x7f61540658a0 con 0x7f61540b1760 2020-11-20T12:14:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6134003240 con 0x7f61540b1760 2020-11-20T12:14:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4021036051 0 0) 0x7f6134003240 con 0x7f61540b1760 2020-11-20T12:14:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 >> v2:172.21.15.187:3300/0 conn(0x7f615414c260 msgr2=0x7f615414c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 --2- 172.21.15.187:0/452234811 >> v2:172.21.15.187:3300/0 conn(0x7f615414c260 0x7f615414c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 >> v2:172.21.15.5:3300/0 conn(0x7f61540b1380 msgr2=0x7f6154150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 --2- 172.21.15.187:0/452234811 >> v2:172.21.15.5:3300/0 conn(0x7f61540b1380 0x7f6154150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6154068510 con 0x7f61540b1760 2020-11-20T12:14:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f613c0014e0 con 0x7f61540b1760 2020-11-20T12:14:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 >> v1:172.21.15.187:6789/0 conn(0x7f61540b1760 legacy=0x7f6154065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6134003730 0x7f6134003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61340044e0 0x7f6134004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6134004e50 0x7f613400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6154150390 con 0x7f6134004e50 2020-11-20T12:14:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6154150630 con 0x7f61340044e0 2020-11-20T12:14:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.493+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6134004260 con 0x7f6134003730 2020-11-20T12:14:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.494+0000 7f614a7fc700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61340044e0 0x7f6134004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.494+0000 7f6152ffd700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6134003730 0x7f6134003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.495+0000 7f61537fe700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6134004e50 0x7f613400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.495+0000 7f614a7fc700 1 -- 172.21.15.187:0/452234811 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6134003730 msgr2=0x7f6134003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:46.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.495+0000 7f614a7fc700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6134003730 0x7f6134003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.495+0000 7f614a7fc700 1 -- 172.21.15.187:0/452234811 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6134004e50 msgr2=0x7f613400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:46.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.495+0000 7f614a7fc700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6134004e50 0x7f613400b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.495+0000 7f614a7fc700 1 -- 172.21.15.187:0/452234811 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6134003240 con 0x7f61340044e0 2020-11-20T12:14:46.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.495+0000 7f61537fe700 1 -- 172.21.15.187:0/452234811 reap_dead start 2020-11-20T12:14:46.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.495+0000 7f614a7fc700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61340044e0 0x7f6134004960 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7f613c00c440 tx=0x7f613c007800).ready entity=mon.1 client_cookie=d2d6f953f3bd770a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:46.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00ce00 con 0x7f61340044e0 2020-11-20T12:14:46.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c0072e0 con 0x7f61340044e0 2020-11-20T12:14:46.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f61527fc700 1 -- 172.21.15.187:0/452234811 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c005840 con 0x7f61340044e0 2020-11-20T12:14:46.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f6153fff700 1 -- 172.21.15.187:0/452234811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61340044e0 msgr2=0x7f6134004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:46.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f6153fff700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61340044e0 0x7f6134004960 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7f613c00c440 tx=0x7f613c007800).stop 2020-11-20T12:14:46.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f6153fff700 1 -- 172.21.15.187:0/452234811 shutdown_connections 2020-11-20T12:14:46.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f6153fff700 1 --2- 172.21.15.187:0/452234811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61340044e0 0x7f6134004960 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f6153fff700 1 -- 172.21.15.187:0/452234811 >> 172.21.15.187:0/452234811 conn(0x7f61540bab30 msgr2=0x7f61540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:46.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f6153fff700 1 -- 172.21.15.187:0/452234811 shutdown_connections 2020-11-20T12:14:46.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.496+0000 7f6153fff700 1 -- 172.21.15.187:0/452234811 wait complete. 2020-11-20T12:14:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.498+0000 7f6153fff700 1 Processor -- start 2020-11-20T12:14:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.498+0000 7f6153fff700 1 -- start start 2020-11-20T12:14:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f6153fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61340044e0 0x7f615409cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f6153fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61540b3fc0 0x7f615409d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f6153fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f615415c680 0x7f615415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f6153fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540b4510 con 0x7f61340044e0 2020-11-20T12:14:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f6153fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540b64f0 con 0x7f61540b3fc0 2020-11-20T12:14:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f6153fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615409a920 con 0x7f615415c680 2020-11-20T12:14:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f614a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61540b3fc0 0x7f615409d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f614a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61540b3fc0 0x7f615409d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49248/0 (socket says 172.21.15.187:49248) 2020-11-20T12:14:46.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.499+0000 7f614a7fc700 1 -- 172.21.15.187:0/634449025 learned_addr learned my addr 172.21.15.187:0/634449025 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:46.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f6152ffd700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61340044e0 0x7f615409cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f61537fe700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f615415c680 0x7f615415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614a7fc700 1 -- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f615415c680 msgr2=0x7f615415cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:46.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614a7fc700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f615415c680 0x7f615415cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614a7fc700 1 -- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61340044e0 msgr2=0x7f615409cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:46.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614a7fc700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61340044e0 0x7f615409cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614a7fc700 1 -- 172.21.15.187:0/634449025 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615415cff0 con 0x7f61540b3fc0 2020-11-20T12:14:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f6152ffd700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61340044e0 0x7f615409cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614a7fc700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61540b3fc0 0x7f615409d040 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f613c008fd0 tx=0x7f613c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614b7fe700 1 -- 172.21.15.187:0/634449025 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c0055e0 con 0x7f61540b3fc0 2020-11-20T12:14:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614b7fe700 1 -- 172.21.15.187:0/634449025 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c001680 con 0x7f61540b3fc0 2020-11-20T12:14:46.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615415fa00 con 0x7f61540b3fc0 2020-11-20T12:14:46.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.500+0000 7f614b7fe700 1 -- 172.21.15.187:0/634449025 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c014b10 con 0x7f61540b3fc0 2020-11-20T12:14:46.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.501+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615415ff10 con 0x7f61540b3fc0 2020-11-20T12:14:46.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.502+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615415d180 con 0x7f61540b3fc0 2020-11-20T12:14:46.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.502+0000 7f614b7fe700 1 -- 172.21.15.187:0/634449025 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f613c025030 con 0x7f61540b3fc0 2020-11-20T12:14:46.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.503+0000 7f614b7fe700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6130056d20 0x7f61300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:46.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.503+0000 7f614b7fe700 1 -- 172.21.15.187:0/634449025 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f613c071100 con 0x7f61540b3fc0 2020-11-20T12:14:46.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.503+0000 7f6152ffd700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6130056d20 0x7f61300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:46.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.504+0000 7f6152ffd700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6130056d20 0x7f61300591e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f6144009440 tx=0x7f6144008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:46.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.507+0000 7f614b7fe700 1 -- 172.21.15.187:0/634449025 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f613c046580 con 0x7f61540b3fc0 2020-11-20T12:14:46.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.749+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6154061a20 con 0x7f61540b3fc0 2020-11-20T12:14:46.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.750+0000 7f614b7fe700 1 -- 172.21.15.187:0/634449025 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f613c023070 con 0x7f61540b3fc0 2020-11-20T12:14:46.754 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:46.754 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:46.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.752+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6130056d20 msgr2=0x7f61300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:46.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.752+0000 7f6153fff700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6130056d20 0x7f61300591e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f6144009440 tx=0x7f6144008040).stop 2020-11-20T12:14:46.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61540b3fc0 msgr2=0x7f615409d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:46.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61540b3fc0 0x7f615409d040 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f613c008fd0 tx=0x7f613c015000).stop 2020-11-20T12:14:46.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 shutdown_connections 2020-11-20T12:14:46.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6130056d20 0x7f61300591e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61340044e0 0x7f615409cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f615415c680 0x7f615415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 --2- 172.21.15.187:0/634449025 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61540b3fc0 0x7f615409d040 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:46.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 >> 172.21.15.187:0/634449025 conn(0x7f61540bab30 msgr2=0x7f615414b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:46.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 shutdown_connections 2020-11-20T12:14:46.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:46.753+0000 7f6153fff700 1 -- 172.21.15.187:0/634449025 wait complete. 2020-11-20T12:14:46.760 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:46.781 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2020-11-20T12:14:47.044 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"num_strays":0}} 2020-11-20T12:14:47.045 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2020-11-20T12:14:47.045 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:47.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.289+0000 7fc3aa759700 1 Processor -- start 2020-11-20T12:14:47.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.289+0000 7fc3aa759700 1 -- start start 2020-11-20T12:14:47.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.290+0000 7fc3aa759700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc3a40b6650 0x7fc3a40afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:47.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.290+0000 7fc3aa759700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc3a4066f70 0x7fc3a40aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:47.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.290+0000 7fc3aa759700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a40b1300 con 0x7fc3a41484d0 2020-11-20T12:14:47.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.290+0000 7fc3aa759700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a4067730 con 0x7fc3a40b6650 2020-11-20T12:14:47.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.290+0000 7fc3aa759700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a40b15a0 con 0x7fc3a4066f70 2020-11-20T12:14:47.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.291+0000 7fc3a9f58700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc3a4066f70 0x7fc3a40aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:47.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.291+0000 7fc3a9f58700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc3a4066f70 0x7fc3a40aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49258/0 (socket says 172.21.15.187:49258) 2020-11-20T12:14:47.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.291+0000 7fc3a9f58700 1 -- 172.21.15.187:0/2864036272 learned_addr learned my addr 172.21.15.187:0/2864036272 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:47.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.291+0000 7fc3a8f56700 1 --2- 172.21.15.187:0/2864036272 >> v2:172.21.15.5:3300/0 conn(0x7fc3a40b6650 0x7fc3a40afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:47.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.292+0000 7fc3a9f58700 1 -- 172.21.15.187:0/2864036272 >> v2:172.21.15.5:3300/0 conn(0x7fc3a40b6650 msgr2=0x7fc3a40afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:47.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.292+0000 7fc3a9f58700 1 --2- 172.21.15.187:0/2864036272 >> v2:172.21.15.5:3300/0 conn(0x7fc3a40b6650 0x7fc3a40afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.292+0000 7fc3a9f58700 1 -- 172.21.15.187:0/2864036272 >> v1:172.21.15.5:6789/0 conn(0x7fc3a41484d0 legacy=0x7fc3a4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:47.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.292+0000 7fc3a9f58700 1 -- 172.21.15.187:0/2864036272 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a40b1840 con 0x7fc3a4066f70 2020-11-20T12:14:47.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.292+0000 7fc3a9f58700 1 --2- 172.21.15.187:0/2864036272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a4066f70 0x7fc3a40aff80 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7fc3a4045000 tx=0x7fc39c006eb0).ready entity=mon.1 client_cookie=ea197dbe36ba552f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:47.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc393fff700 1 -- 172.21.15.187:0/2864036272 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c009810 con 0x7fc3a4066f70 2020-11-20T12:14:47.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc393fff700 1 -- 172.21.15.187:0/2864036272 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c009970 con 0x7fc3a4066f70 2020-11-20T12:14:47.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc393fff700 1 -- 172.21.15.187:0/2864036272 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c009cf0 con 0x7fc3a4066f70 2020-11-20T12:14:47.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc3aa759700 1 -- 172.21.15.187:0/2864036272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a4066f70 msgr2=0x7fc3a40aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:47.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc3aa759700 1 --2- 172.21.15.187:0/2864036272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a4066f70 0x7fc3a40aff80 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7fc3a4045000 tx=0x7fc39c006eb0).stop 2020-11-20T12:14:47.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc3aa759700 1 -- 172.21.15.187:0/2864036272 shutdown_connections 2020-11-20T12:14:47.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc3aa759700 1 --2- 172.21.15.187:0/2864036272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a4066f70 0x7fc3a40aff80 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc3aa759700 1 --2- 172.21.15.187:0/2864036272 >> v2:172.21.15.5:3300/0 conn(0x7fc3a40b6650 0x7fc3a40afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc3aa759700 1 -- 172.21.15.187:0/2864036272 >> 172.21.15.187:0/2864036272 conn(0x7fc3a40bab30 msgr2=0x7fc3a40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:47.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc3aa759700 1 -- 172.21.15.187:0/2864036272 shutdown_connections 2020-11-20T12:14:47.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.293+0000 7fc3aa759700 1 -- 172.21.15.187:0/2864036272 wait complete. 2020-11-20T12:14:47.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.295+0000 7fc3aa759700 1 Processor -- start 2020-11-20T12:14:47.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.295+0000 7fc3aa759700 1 -- start start 2020-11-20T12:14:47.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3aa759700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a4066f70 0x7fc3a4158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:47.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3aa759700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a40b6650 0x7fc3a40b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:47.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3aa759700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a41484d0 0x7fc3a40b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:47.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3aa759700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a41590d0 con 0x7fc3a41484d0 2020-11-20T12:14:47.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3aa759700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b9830 con 0x7fc3a40b6650 2020-11-20T12:14:47.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3aa759700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b1b10 con 0x7fc3a4066f70 2020-11-20T12:14:47.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3a8f56700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a40b6650 0x7fc3a40b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:47.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3a8f56700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a40b6650 0x7fc3a40b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49262/0 (socket says 172.21.15.187:49262) 2020-11-20T12:14:47.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.296+0000 7fc3a8f56700 1 -- 172.21.15.187:0/1131226966 learned_addr learned my addr 172.21.15.187:0/1131226966 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:47.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.297+0000 7fc3a9757700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a4066f70 0x7fc3a4158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:47.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.297+0000 7fc3a9f58700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a41484d0 0x7fc3a40b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:47.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.297+0000 7fc3a8f56700 1 -- 172.21.15.187:0/1131226966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a4066f70 msgr2=0x7fc3a4158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:47.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.297+0000 7fc3a8f56700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a4066f70 0x7fc3a4158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.297+0000 7fc3a8f56700 1 -- 172.21.15.187:0/1131226966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a41484d0 msgr2=0x7fc3a40b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:47.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.297+0000 7fc3a8f56700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a41484d0 0x7fc3a40b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.297+0000 7fc3a8f56700 1 -- 172.21.15.187:0/1131226966 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a40b1db0 con 0x7fc3a40b6650 2020-11-20T12:14:47.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.297+0000 7fc3a8f56700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a40b6650 0x7fc3a40b4410 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7fc39400eba0 tx=0x7fc394006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:47.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.298+0000 7fc3917fa700 1 -- 172.21.15.187:0/1131226966 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39400ca60 con 0x7fc3a40b6650 2020-11-20T12:14:47.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.298+0000 7fc3917fa700 1 -- 172.21.15.187:0/1131226966 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39400cbc0 con 0x7fc3a40b6650 2020-11-20T12:14:47.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.298+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a40b4900 con 0x7fc3a40b6650 2020-11-20T12:14:47.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.298+0000 7fc3917fa700 1 -- 172.21.15.187:0/1131226966 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3940123f0 con 0x7fc3a40b6650 2020-11-20T12:14:47.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.298+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a40b2900 con 0x7fc3a40b6650 2020-11-20T12:14:47.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.299+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a4061a20 con 0x7fc3a40b6650 2020-11-20T12:14:47.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.300+0000 7fc3917fa700 1 -- 172.21.15.187:0/1131226966 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc3940127a0 con 0x7fc3a40b6650 2020-11-20T12:14:47.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.300+0000 7fc3917fa700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc37c056d20 0x7fc37c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:47.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.301+0000 7fc3917fa700 1 -- 172.21.15.187:0/1131226966 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc394070440 con 0x7fc3a40b6650 2020-11-20T12:14:47.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.301+0000 7fc3a9757700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc37c056d20 0x7fc37c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:47.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.301+0000 7fc3a9757700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc37c056d20 0x7fc37c0591e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7fc3a0000f40 tx=0x7fc3a0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:47.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.305+0000 7fc3917fa700 1 -- 172.21.15.187:0/1131226966 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc394045890 con 0x7fc3a40b6650 2020-11-20T12:14:47.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.532+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3a40624e0 con 0x7fc3a40b6650 2020-11-20T12:14:47.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.533+0000 7fc3917fa700 1 -- 172.21.15.187:0/1131226966 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7fc394048eb0 con 0x7fc3a40b6650 2020-11-20T12:14:47.537 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:47.537 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:47.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc37c056d20 msgr2=0x7fc37c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:47.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc37c056d20 0x7fc37c0591e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7fc3a0000f40 tx=0x7fc3a0009040).stop 2020-11-20T12:14:47.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a40b6650 msgr2=0x7fc3a40b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:47.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a40b6650 0x7fc3a40b4410 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7fc39400eba0 tx=0x7fc394006df0).stop 2020-11-20T12:14:47.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 shutdown_connections 2020-11-20T12:14:47.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc37c056d20 0x7fc37c0591e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a41484d0 0x7fc3a40b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a4066f70 0x7fc3a4158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 --2- 172.21.15.187:0/1131226966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a40b6650 0x7fc3a40b4410 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:47.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 >> 172.21.15.187:0/1131226966 conn(0x7fc3a40bab30 msgr2=0x7fc3a4142cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:47.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 shutdown_connections 2020-11-20T12:14:47.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:47.535+0000 7fc3aa759700 1 -- 172.21.15.187:0/1131226966 wait complete. 2020-11-20T12:14:47.542 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:47.562 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:14:47.840 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_created":1}} 2020-11-20T12:14:47.841 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2020-11-20T12:14:47.842 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:48.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.068+0000 7f3c0aec2700 1 Processor -- start 2020-11-20T12:14:48.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.068+0000 7f3c0aec2700 1 -- start start 2020-11-20T12:14:48.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.069+0000 7f3c0aec2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3c040b57b0 0x7f3c04068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.069+0000 7f3c0aec2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3c040b5f50 0x7f3c040b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.069+0000 7f3c0aec2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c04144b80 con 0x7f3c041482d0 2020-11-20T12:14:48.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.069+0000 7f3c0aec2700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c041489e0 con 0x7f3c040b5f50 2020-11-20T12:14:48.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.069+0000 7f3c0aec2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c04148c80 con 0x7f3c040b57b0 2020-11-20T12:14:48.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.070+0000 7f3c09ec0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3c040b57b0 0x7f3c04068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.070+0000 7f3c09ec0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3c040b57b0 0x7f3c04068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49268/0 (socket says 172.21.15.187:49268) 2020-11-20T12:14:48.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.070+0000 7f3c09ec0700 1 -- 172.21.15.187:0/1330670474 learned_addr learned my addr 172.21.15.187:0/1330670474 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:48.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.070+0000 7f3c0a6c1700 1 --2- 172.21.15.187:0/1330670474 >> v2:172.21.15.5:3300/0 conn(0x7f3c040b5f50 0x7f3c040b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.070+0000 7f3c09ec0700 1 -- 172.21.15.187:0/1330670474 >> v2:172.21.15.5:3300/0 conn(0x7f3c040b5f50 msgr2=0x7f3c040b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:48.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.070+0000 7f3c09ec0700 1 --2- 172.21.15.187:0/1330670474 >> v2:172.21.15.5:3300/0 conn(0x7f3c040b5f50 0x7f3c040b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.070+0000 7f3c09ec0700 1 -- 172.21.15.187:0/1330670474 >> v1:172.21.15.5:6789/0 conn(0x7f3c041482d0 legacy=0x7f3c04062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:48.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.070+0000 7f3c09ec0700 1 -- 172.21.15.187:0/1330670474 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c04148f20 con 0x7f3c040b57b0 2020-11-20T12:14:48.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.071+0000 7f3c09ec0700 1 --2- 172.21.15.187:0/1330670474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c040b57b0 0x7f3c04068230 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f3c040b85e0 tx=0x7f3c00006f40).ready entity=mon.1 client_cookie=64fc349f3fd642ca server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.071+0000 7f3c08ebe700 1 -- 172.21.15.187:0/1330670474 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000caf0 con 0x7f3c040b57b0 2020-11-20T12:14:48.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.071+0000 7f3c08ebe700 1 -- 172.21.15.187:0/1330670474 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0000cc50 con 0x7f3c040b57b0 2020-11-20T12:14:48.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.071+0000 7f3c08ebe700 1 -- 172.21.15.187:0/1330670474 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c00010410 con 0x7f3c040b57b0 2020-11-20T12:14:48.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.072+0000 7f3c0aec2700 1 -- 172.21.15.187:0/1330670474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c040b57b0 msgr2=0x7f3c04068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.072+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/1330670474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c040b57b0 0x7f3c04068230 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f3c040b85e0 tx=0x7f3c00006f40).stop 2020-11-20T12:14:48.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.072+0000 7f3c0aec2700 1 -- 172.21.15.187:0/1330670474 shutdown_connections 2020-11-20T12:14:48.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.072+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/1330670474 >> v2:172.21.15.5:3300/0 conn(0x7f3c040b5f50 0x7f3c040b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.072+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/1330670474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c040b57b0 0x7f3c04068230 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.072+0000 7f3c0aec2700 1 -- 172.21.15.187:0/1330670474 >> 172.21.15.187:0/1330670474 conn(0x7f3c040bab30 msgr2=0x7f3c040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:48.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.072+0000 7f3c0aec2700 1 -- 172.21.15.187:0/1330670474 shutdown_connections 2020-11-20T12:14:48.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.072+0000 7f3c0aec2700 1 -- 172.21.15.187:0/1330670474 wait complete. 2020-11-20T12:14:48.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.074+0000 7f3c0aec2700 1 Processor -- start 2020-11-20T12:14:48.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.074+0000 7f3c0aec2700 1 -- start start 2020-11-20T12:14:48.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0aec2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c040b5f50 0x7f3c0413bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0aec2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c041482d0 0x7f3c041453b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0aec2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c04145c50 0x7f3c041460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0aec2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040b77c0 con 0x7f3c040b5f50 2020-11-20T12:14:48.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0aec2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040b7a60 con 0x7f3c04145c50 2020-11-20T12:14:48.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0aec2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041585e0 con 0x7f3c041482d0 2020-11-20T12:14:48.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0a6c1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c04145c50 0x7f3c041460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0a6c1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c04145c50 0x7f3c041460d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49278/0 (socket says 172.21.15.187:49278) 2020-11-20T12:14:48.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.075+0000 7f3c0a6c1700 1 -- 172.21.15.187:0/24157861 learned_addr learned my addr 172.21.15.187:0/24157861 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:48.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c09ec0700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c040b5f50 0x7f3c0413bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c096bf700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c041482d0 0x7f3c041453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c0a6c1700 1 -- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c041482d0 msgr2=0x7f3c041453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c0a6c1700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c041482d0 0x7f3c041453b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c0a6c1700 1 -- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c040b5f50 msgr2=0x7f3c0413bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c0a6c1700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c040b5f50 0x7f3c0413bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c0a6c1700 1 -- 172.21.15.187:0/24157861 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0413c4a0 con 0x7f3c04145c50 2020-11-20T12:14:48.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c09ec0700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c040b5f50 0x7f3c0413bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:48.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c0a6c1700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c04145c50 0x7f3c041460d0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f3bfc00afd0 tx=0x7f3bfc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3bf5ffb700 1 -- 172.21.15.187:0/24157861 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc007940 con 0x7f3c04145c50 2020-11-20T12:14:48.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3bf5ffb700 1 -- 172.21.15.187:0/24157861 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc007aa0 con 0x7f3c04145c50 2020-11-20T12:14:48.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c041458a0 con 0x7f3c04145c50 2020-11-20T12:14:48.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.076+0000 7f3bf5ffb700 1 -- 172.21.15.187:0/24157861 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc007e60 con 0x7f3c04145c50 2020-11-20T12:14:48.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.077+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c040b6540 con 0x7f3c04145c50 2020-11-20T12:14:48.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.078+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c04061a20 con 0x7f3c04145c50 2020-11-20T12:14:48.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.078+0000 7f3bf5ffb700 1 -- 172.21.15.187:0/24157861 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3bfc011620 con 0x7f3c04145c50 2020-11-20T12:14:48.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.079+0000 7f3bf5ffb700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bdc056d20 0x7f3bdc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.079+0000 7f3bf5ffb700 1 -- 172.21.15.187:0/24157861 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3bfc070240 con 0x7f3c04145c50 2020-11-20T12:14:48.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.079+0000 7f3c09ec0700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bdc056d20 0x7f3bdc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.080+0000 7f3c09ec0700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bdc056d20 0x7f3bdc0591e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f3c04063890 tx=0x7f3c00006f40).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.083+0000 7f3bf5ffb700 1 -- 172.21.15.187:0/24157861 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3bfc045690 con 0x7f3c04145c50 2020-11-20T12:14:48.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.323+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c040446d0 con 0x7f3c04145c50 2020-11-20T12:14:48.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.324+0000 7f3bf5ffb700 1 -- 172.21.15.187:0/24157861 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f3bfc048cb0 con 0x7f3c04145c50 2020-11-20T12:14:48.328 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:48.329 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:48.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.326+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bdc056d20 msgr2=0x7f3bdc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.326+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bdc056d20 0x7f3bdc0591e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f3c04063890 tx=0x7f3c00006f40).stop 2020-11-20T12:14:48.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.326+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c04145c50 msgr2=0x7f3c041460d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.326+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c04145c50 0x7f3c041460d0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f3bfc00afd0 tx=0x7f3bfc008040).stop 2020-11-20T12:14:48.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.327+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 shutdown_connections 2020-11-20T12:14:48.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.327+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bdc056d20 0x7f3bdc0591e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.327+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c040b5f50 0x7f3c0413bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.327+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c04145c50 0x7f3c041460d0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.327+0000 7f3c0aec2700 1 --2- 172.21.15.187:0/24157861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c041482d0 0x7f3c041453b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.327+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 >> 172.21.15.187:0/24157861 conn(0x7f3c040bab30 msgr2=0x7f3c040659b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:48.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.327+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 shutdown_connections 2020-11-20T12:14:48.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.327+0000 7f3c0aec2700 1 -- 172.21.15.187:0/24157861 wait complete. 2020-11-20T12:14:48.334 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:48.351 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2020-11-20T12:14:48.609 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_reintegrated":1}} 2020-11-20T12:14:48.610 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2020-11-20T12:14:48.610 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:14:48.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.695+0000 7f930e847d00 1 Processor -- start 2020-11-20T12:14:48.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.695+0000 7f930e847d00 1 -- start start 2020-11-20T12:14:48.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.696+0000 7f930e847d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56055493de80 0x5605549e1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.696+0000 7f930e847d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e0eb0 0x5605549de280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.696+0000 7f930e847d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5605549e5060 0x5605549c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.696+0000 7f930e847d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x560554891810 con 0x5605549e0eb0 2020-11-20T12:14:48.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.696+0000 7f930e847d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x56055493c5d0 con 0x5605549e5060 2020-11-20T12:14:48.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.696+0000 7f930e847d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605549e4920 con 0x56055493de80 2020-11-20T12:14:48.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff8ed700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5605549e5060 0x5605549c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff8ed700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5605549e5060 0x5605549c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49292/0 (socket says 172.21.15.187:49292) 2020-11-20T12:14:48.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff8ed700 1 -- 172.21.15.187:0/367145639 learned_addr learned my addr 172.21.15.187:0/367145639 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:48.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56055493de80 0x5605549e1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92f68eb700 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e0eb0 0x5605549de280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff8ed700 1 -- 172.21.15.187:0/367145639 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56055493de80 msgr2=0x5605549e1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:48.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56055493de80 0x5605549e1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff8ed700 1 -- 172.21.15.187:0/367145639 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e0eb0 msgr2=0x5605549de280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:48.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e0eb0 0x5605549de280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.697+0000 7f92ff8ed700 1 -- 172.21.15.187:0/367145639 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605548bb4e0 con 0x5605549e5060 2020-11-20T12:14:48.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.698+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5605549e5060 0x5605549c7430 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7f92f8009ad0 tx=0x7f92f8005b10).ready entity=mon.1 client_cookie=f565a782741e0113 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.698+0000 7f92fe0ea700 1 -- 172.21.15.187:0/367145639 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f800e070 con 0x5605549e5060 2020-11-20T12:14:48.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.698+0000 7f92fe0ea700 1 -- 172.21.15.187:0/367145639 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f8006ac0 con 0x5605549e5060 2020-11-20T12:14:48.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.698+0000 7f92fe0ea700 1 -- 172.21.15.187:0/367145639 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8006e40 con 0x5605549e5060 2020-11-20T12:14:48.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.698+0000 7f930e847d00 1 -- 172.21.15.187:0/367145639 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5605549e5060 msgr2=0x5605549c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.698+0000 7f930e847d00 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5605549e5060 0x5605549c7430 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7f92f8009ad0 tx=0x7f92f8005b10).stop 2020-11-20T12:14:48.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.699+0000 7f930e847d00 1 -- 172.21.15.187:0/367145639 shutdown_connections 2020-11-20T12:14:48.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.699+0000 7f930e847d00 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5605549e5060 0x5605549c7430 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.699+0000 7f930e847d00 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x56055493de80 0x5605549e1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.699+0000 7f930e847d00 1 --2- 172.21.15.187:0/367145639 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e0eb0 0x5605549de280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.699+0000 7f930e847d00 1 -- 172.21.15.187:0/367145639 >> 172.21.15.187:0/367145639 conn(0x560554916a90 msgr2=0x56055493bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:48.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.699+0000 7f930e847d00 1 -- 172.21.15.187:0/367145639 shutdown_connections 2020-11-20T12:14:48.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.699+0000 7f930e847d00 1 -- 172.21.15.187:0/367145639 wait complete. 2020-11-20T12:14:48.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.699+0000 7f930e847d00 1 Processor -- start 2020-11-20T12:14:48.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.700+0000 7f930e847d00 1 -- start start 2020-11-20T12:14:48.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.700+0000 7f930e847d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56055493de80 0x5605549c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.700+0000 7f930e847d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5605549e0eb0 0x5605549c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.700+0000 7f930e847d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e5060 0x5605549f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.700+0000 7f930e847d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605549cab10 con 0x5605549e5060 2020-11-20T12:14:48.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.700+0000 7f930e847d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605549c6e60 con 0x56055493de80 2020-11-20T12:14:48.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.700+0000 7f930e847d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605549cad00 con 0x5605549e0eb0 2020-11-20T12:14:48.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56055493de80 0x5605549c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56055493de80 0x5605549c6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49294/0 (socket says 172.21.15.187:49294) 2020-11-20T12:14:48.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 learned_addr learned my addr 172.21.15.187:0/200770482 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:48.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5605549e0eb0 0x5605549c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5605549e0eb0 msgr2=0x5605549c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5605549e0eb0 0x5605549c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e5060 msgr2=0x5605549f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e5060 0x5605549f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5605549e5060 0x5605549f5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605548b6ec0 con 0x56055493de80 2020-11-20T12:14:48.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5605549e0eb0 0x5605549c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:48.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.701+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56055493de80 0x5605549c6480 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7f92f000f000 tx=0x7f92f0014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.702+0000 7f92fc8e7700 1 -- 172.21.15.187:0/200770482 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0005670 con 0x56055493de80 2020-11-20T12:14:48.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.702+0000 7f92fc8e7700 1 -- 172.21.15.187:0/200770482 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f0010cf0 con 0x56055493de80 2020-11-20T12:14:48.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.702+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560554895250 con 0x56055493de80 2020-11-20T12:14:48.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.702+0000 7f92fc8e7700 1 -- 172.21.15.187:0/200770482 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f001d640 con 0x56055493de80 2020-11-20T12:14:48.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.702+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605549cb440 con 0x56055493de80 2020-11-20T12:14:48.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.703+0000 7f92fc8e7700 1 -- 172.21.15.187:0/200770482 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f92f000e070 con 0x56055493de80 2020-11-20T12:14:48.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.704+0000 7f92fc8e7700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e00559a0 0x7f92e0057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.704+0000 7f92fc8e7700 1 -- 172.21.15.187:0/200770482 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f92f00713b0 con 0x56055493de80 2020-11-20T12:14:48.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.704+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5605549f65b0 0x5605549c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.704+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:1 20.0 20:00000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x5605549e61f0 con 0x5605549f65b0 2020-11-20T12:14:48.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.704+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e00559a0 0x7f92e0057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.705+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e00559a0 0x7f92e0057e60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x5605549e34a0 tx=0x7f92e80025f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.705+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5605549f65b0 0x5605549c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.706+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5605549f65b0 0x5605549c8760 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.707+0000 7f92ff8ed700 1 -- 172.21.15.187:0/200770482 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(1 [pgnls start_epoch 98 out=81b] v98'2 uv2 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f92f8006dc0 con 0x5605549f65b0 2020-11-20T12:14:48.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.707+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f92f80078e0 0x7f92f80123c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.707+0000 7f92ff8ed700 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:2 20.4 20:20000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x5605549e61f0 con 0x7f92f80078e0 2020-11-20T12:14:48.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.708+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f92f80078e0 0x7f92f80123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.709+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f92f80078e0 0x7f92f80123c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.711+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== osd_op_reply(2 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92f0048750 con 0x7f92f80078e0 2020-11-20T12:14:48.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.711+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f92f0026030 0x7f92f0049310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.711+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] -- osd_op(unknown.0.0:3 20.2 20:40000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x5605549e61f0 con 0x7f92f0026030 2020-11-20T12:14:48.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.711+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f92f0026030 0x7f92f0049310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.712+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f92f0026030 0x7f92f0049310 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.712+0000 7f92f68eb700 1 -- 172.21.15.187:0/200770482 <== osd.3 v2:172.21.15.187:6826/19935 1 ==== osd_op_reply(3 [pgnls start_epoch 98 out=81b] v98'1 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f92e8010790 con 0x7f92f0026030 2020-11-20T12:14:48.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.712+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f92e800f9e0 0x7f92e800e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.713+0000 7f92f68eb700 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:4 20.6 20:60000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x5605549e61f0 con 0x7f92e800f9e0 2020-11-20T12:14:48.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.713+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f92e800f9e0 0x7f92e800e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.715+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f92e800f9e0 0x7f92e800e2f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.716+0000 7f92ff8ed700 1 -- 172.21.15.187:0/200770482 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(4 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92f8006dc0 con 0x7f92e800f9e0 2020-11-20T12:14:48.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.716+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f92f8023460 0x7f92f8023880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.716+0000 7f92ff8ed700 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- osd_op(unknown.0.0:5 20.1 20:80000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x5605549e61f0 con 0x7f92f8023460 2020-11-20T12:14:48.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.717+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f92f8023460 0x7f92f8023880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.717+0000 7f92ff0ec700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f92f8023460 0x7f92f8023880 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.718+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 <== osd.0 v2:172.21.15.187:6802/19903 1 ==== osd_op_reply(5 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92f0048750 con 0x7f92f8023460 2020-11-20T12:14:48.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.718+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:6 20.5 20:a0000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x5605549e61f0 con 0x7f92f80078e0 2020-11-20T12:14:48.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.719+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 <== osd.5 v2:172.21.15.5:6809/20155 2 ==== osd_op_reply(6 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92f0048750 con 0x7f92f80078e0 2020-11-20T12:14:48.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.719+0000 7f92ff0ec700 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:7 20.3 20:c0000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x5605549e61f0 con 0x7f92e800f9e0 2020-11-20T12:14:48.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.720+0000 7f92ff8ed700 1 -- 172.21.15.187:0/200770482 <== osd.7 v2:172.21.15.5:6814/20158 2 ==== osd_op_reply(7 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92f8006dc0 con 0x7f92e800f9e0 2020-11-20T12:14:48.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.720+0000 7f92ff8ed700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f92f8021950 0x7f92f80203d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.720+0000 7f92ff8ed700 1 -- 172.21.15.187:0/200770482 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:8 20.7 20:e0000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x5605549e61f0 con 0x7f92f8021950 2020-11-20T12:14:48.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.721+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f92f8021950 0x7f92f80203d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.721+0000 7f92f68eb700 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f92f8021950 0x7f92f80203d0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f92f68eb700 1 -- 172.21.15.187:0/200770482 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(8 [pgnls start_epoch 98 out=113b] v98'2 uv2 ondisk = 1) v8 ==== 144+0+113 (crc 0 0 0) 0x7f92e8010790 con 0x7f92f8021950 2020-11-20T12:14:48.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f92f8023460 msgr2=0x7f92f8023880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f92f8023460 0x7f92f8023880 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f92f8021950 msgr2=0x7f92f80203d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f92f8021950 0x7f92f80203d0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f92f0026030 msgr2=0x7f92f0049310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f92f0026030 0x7f92f0049310 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f92f80078e0 msgr2=0x7f92f80123c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f92f80078e0 0x7f92f80123c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5605549f65b0 msgr2=0x5605549c8760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f92ff8ed700 1 -- 172.21.15.187:0/200770482 reap_dead start 2020-11-20T12:14:48.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5605549f65b0 0x5605549c8760 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f92e800f9e0 msgr2=0x7f92e800e2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f92e800f9e0 0x7f92e800e2f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e00559a0 msgr2=0x7f92e0057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e00559a0 0x7f92e0057e60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x5605549e34a0 tx=0x7f92e80025f0).stop 2020-11-20T12:14:48.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56055493de80 msgr2=0x5605549c6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56055493de80 0x5605549c6480 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7f92f000f000 tx=0x7f92f0014000).stop 2020-11-20T12:14:48.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 shutdown_connections 2020-11-20T12:14:48.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f92e800f9e0 0x7f92e800e2f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56055493de80 0x5605549c6480 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x5605549f65b0 0x5605549c8760 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 --2- 172.21.15.187:0/200770482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e00559a0 0x7f92e0057e60 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.722+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 >> 172.21.15.187:0/200770482 conn(0x560554916a90 msgr2=0x5605549de280 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:48.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.723+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 shutdown_connections 2020-11-20T12:14:48.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.723+0000 7f930e847d00 1 -- 172.21.15.187:0/200770482 wait complete. 2020-11-20T12:14:48.734 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000000 2020-11-20T12:14:48.734 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000001 2020-11-20T12:14:48.734 INFO:teuthology.orchestra.run.smithi187.stdout:10000000001.00000000 2020-11-20T12:14:48.734 INFO:teuthology.orchestra.run.smithi187.stdout:10000000001.00000001 2020-11-20T12:14:48.735 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 8388608 found 2020-11-20T12:14:48.736 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:14:48.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.819+0000 7f9ce3dfad00 1 Processor -- start 2020-11-20T12:14:48.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.819+0000 7f9ce3dfad00 1 -- start start 2020-11-20T12:14:48.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9ce3dfad00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22e87d0 0x558ce22e9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9ce3dfad00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce22eb8c0 0x558ce235e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9ce3dfad00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce2378880 0x558ce235f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9ce3dfad00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ce2225810 con 0x558ce2378880 2020-11-20T12:14:48.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9ce3dfad00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ce235e080 con 0x558ce22e87d0 2020-11-20T12:14:48.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9ce3dfad00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ce22e9620 con 0x558ce22eb8c0 2020-11-20T12:14:48.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9cd469f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22e87d0 0x558ce22e9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9cd469f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22e87d0 0x558ce22e9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49320/0 (socket says 172.21.15.187:49320) 2020-11-20T12:14:48.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.820+0000 7f9cd469f700 1 -- 172.21.15.187:0/3092562709 learned_addr learned my addr 172.21.15.187:0/3092562709 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:48.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce22eb8c0 0x558ce235e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce2378880 0x558ce235f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd469f700 1 -- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce22eb8c0 msgr2=0x558ce235e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:48.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd469f700 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce22eb8c0 0x558ce235e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd469f700 1 -- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce2378880 msgr2=0x558ce235f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:48.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd469f700 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce2378880 0x558ce235f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd469f700 1 -- 172.21.15.187:0/3092562709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ce224f4e0 con 0x558ce22e87d0 2020-11-20T12:14:48.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce2378880 0x558ce235f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:48.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd469f700 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22e87d0 0x558ce22e9ac0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7f9cc400b4e0 tx=0x7f9cc4001880).ready entity=mon.1 client_cookie=a5f25a6c7c566046 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd2e9c700 1 -- 172.21.15.187:0/3092562709 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc400d550 con 0x558ce22e87d0 2020-11-20T12:14:48.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd2e9c700 1 -- 172.21.15.187:0/3092562709 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc400d6b0 con 0x558ce22e87d0 2020-11-20T12:14:48.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.821+0000 7f9cd2e9c700 1 -- 172.21.15.187:0/3092562709 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc400da30 con 0x558ce22e87d0 2020-11-20T12:14:48.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/3092562709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22e87d0 msgr2=0x558ce22e9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22e87d0 0x558ce22e9ac0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7f9cc400b4e0 tx=0x7f9cc4001880).stop 2020-11-20T12:14:48.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/3092562709 shutdown_connections 2020-11-20T12:14:48.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce2378880 0x558ce235f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22e87d0 0x558ce22e9ac0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/3092562709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce22eb8c0 0x558ce235e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/3092562709 >> 172.21.15.187:0/3092562709 conn(0x558ce22aaa90 msgr2=0x558ce22d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:48.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/3092562709 shutdown_connections 2020-11-20T12:14:48.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.822+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/3092562709 wait complete. 2020-11-20T12:14:48.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.823+0000 7f9ce3dfad00 1 Processor -- start 2020-11-20T12:14:48.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.823+0000 7f9ce3dfad00 1 -- start start 2020-11-20T12:14:48.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9ce3dfad00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce22e87d0 0x558ce238c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9ce3dfad00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22eb8c0 0x558ce235c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9ce3dfad00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce2378880 0x558ce235ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9ce3dfad00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ce2378000 con 0x558ce22e87d0 2020-11-20T12:14:48.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9ce3dfad00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ce235cf10 con 0x558ce22eb8c0 2020-11-20T12:14:48.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9ce3dfad00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x558ce238cb10 con 0x558ce2378880 2020-11-20T12:14:48.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9cd3e9e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22eb8c0 0x558ce235c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9cd3e9e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22eb8c0 0x558ce235c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49328/0 (socket says 172.21.15.187:49328) 2020-11-20T12:14:48.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.824+0000 7f9cd3e9e700 1 -- 172.21.15.187:0/1258933435 learned_addr learned my addr 172.21.15.187:0/1258933435 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:48.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd469f700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce22e87d0 0x558ce238c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce2378880 0x558ce235ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd3e9e700 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce2378880 msgr2=0x558ce235ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x558ce2378880 0x558ce235ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd3e9e700 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce22e87d0 msgr2=0x558ce238c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x558ce22e87d0 0x558ce238c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd3e9e700 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ce224aec0 con 0x558ce22eb8c0 2020-11-20T12:14:48.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22eb8c0 0x558ce235c530 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f9cc8005fd0 tx=0x7f9cc800c040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd1699700 1 -- 172.21.15.187:0/1258933435 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc800f030 con 0x558ce22eb8c0 2020-11-20T12:14:48.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd1699700 1 -- 172.21.15.187:0/1258933435 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc8010040 con 0x558ce22eb8c0 2020-11-20T12:14:48.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9cd1699700 1 -- 172.21.15.187:0/1258933435 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8015670 con 0x558ce22eb8c0 2020-11-20T12:14:48.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.825+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558ce2229250 con 0x558ce22eb8c0 2020-11-20T12:14:48.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.826+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ce2377bd0 con 0x558ce22eb8c0 2020-11-20T12:14:48.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.827+0000 7f9cd1699700 1 -- 172.21.15.187:0/1258933435 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9cc801c030 con 0x558ce22eb8c0 2020-11-20T12:14:48.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.828+0000 7f9cd1699700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9cc00559a0 0x7f9cc0057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.828+0000 7f9cd1699700 1 -- 172.21.15.187:0/1258933435 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9cc806f240 con 0x558ce22eb8c0 2020-11-20T12:14:48.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.828+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x558ce235d150 0x558ce22cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.828+0000 7f9cd469f700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9cc00559a0 0x7f9cc0057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.828+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:1 20.0 20:00000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x558ce22cd820 con 0x558ce235d150 2020-11-20T12:14:48.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.829+0000 7f9cd469f700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9cc00559a0 0x7f9cc0057e60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f9cc40018e0 tx=0x7f9cc4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.829+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x558ce235d150 0x558ce22cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.830+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x558ce235d150 0x558ce22cd330 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.831+0000 7f9cd4ea0700 1 -- 172.21.15.187:0/1258933435 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(1 [pgnls start_epoch 98 out=81b] v98'2 uv2 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f9ccc008910 con 0x558ce235d150 2020-11-20T12:14:48.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.832+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f9ccc01c3f0 0x7f9ccc01c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.832+0000 7f9cd4ea0700 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:2 20.4 20:20000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x558ce22cd820 con 0x7f9ccc01c3f0 2020-11-20T12:14:48.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.832+0000 7f9cd469f700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f9ccc01c3f0 0x7f9ccc01c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.834+0000 7f9cd469f700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f9ccc01c3f0 0x7f9ccc01c830 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.835+0000 7f9cd469f700 1 -- 172.21.15.187:0/1258933435 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== osd_op_reply(2 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9cc400d920 con 0x7f9ccc01c3f0 2020-11-20T12:14:48.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.835+0000 7f9cd469f700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f9cc401d410 0x7f9cc401d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.835+0000 7f9cd469f700 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] -- osd_op(unknown.0.0:3 20.2 20:40000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x558ce22cd820 con 0x7f9cc401d410 2020-11-20T12:14:48.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.835+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f9cc401d410 0x7f9cc401d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.836+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f9cc401d410 0x7f9cc401d810 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.836+0000 7f9cd3e9e700 1 -- 172.21.15.187:0/1258933435 <== osd.3 v2:172.21.15.187:6826/19935 1 ==== osd_op_reply(3 [pgnls start_epoch 98 out=81b] v98'1 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f9cc8012070 con 0x7f9cc401d410 2020-11-20T12:14:48.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.836+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f9cc801f060 0x7f9cc8015a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.836+0000 7f9cd3e9e700 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:4 20.6 20:60000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x558ce22cd820 con 0x7f9cc801f060 2020-11-20T12:14:48.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.837+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f9cc801f060 0x7f9cc8015a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.838+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f9cc801f060 0x7f9cc8015a60 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.840+0000 7f9cd4ea0700 1 -- 172.21.15.187:0/1258933435 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(4 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ccc008910 con 0x7f9cc801f060 2020-11-20T12:14:48.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.840+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f9ccc00fa20 0x7f9ccc01b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.840+0000 7f9cd4ea0700 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- osd_op(unknown.0.0:5 20.1 20:80000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x558ce22cd820 con 0x7f9ccc00fa20 2020-11-20T12:14:48.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.840+0000 7f9cd469f700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f9ccc00fa20 0x7f9ccc01b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.841+0000 7f9cd469f700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f9ccc00fa20 0x7f9ccc01b380 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.841+0000 7f9cd469f700 1 -- 172.21.15.187:0/1258933435 <== osd.0 v2:172.21.15.187:6802/19903 1 ==== osd_op_reply(5 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9cc400d920 con 0x7f9ccc00fa20 2020-11-20T12:14:48.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.841+0000 7f9cd469f700 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:6 20.5 20:a0000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x558ce22cd820 con 0x7f9ccc01c3f0 2020-11-20T12:14:48.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.843+0000 7f9cd469f700 1 -- 172.21.15.187:0/1258933435 <== osd.5 v2:172.21.15.5:6809/20155 2 ==== osd_op_reply(6 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9cc400d920 con 0x7f9ccc01c3f0 2020-11-20T12:14:48.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.843+0000 7f9cd469f700 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:7 20.3 20:c0000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x558ce22cd820 con 0x7f9cc801f060 2020-11-20T12:14:48.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.844+0000 7f9cd4ea0700 1 -- 172.21.15.187:0/1258933435 <== osd.7 v2:172.21.15.5:6814/20158 2 ==== osd_op_reply(7 [pgnls start_epoch 98 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9ccc008910 con 0x7f9cc801f060 2020-11-20T12:14:48.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.844+0000 7f9cd4ea0700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f9ccc02d3c0 0x7f9ccc02d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:48.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.844+0000 7f9cd4ea0700 1 -- 172.21.15.187:0/1258933435 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:8 20.7 20:e0000000::::head [pgnls start_epoch 98 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98) v8 -- 0x558ce22cd820 con 0x7f9ccc02d3c0 2020-11-20T12:14:48.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.845+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f9ccc02d3c0 0x7f9ccc02d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:48.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9cd3e9e700 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f9ccc02d3c0 0x7f9ccc02d7e0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:48.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9cd3e9e700 1 -- 172.21.15.187:0/1258933435 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(8 [pgnls start_epoch 98 out=113b] v98'2 uv2 ondisk = 1) v8 ==== 144+0+113 (crc 0 0 0) 0x7f9cc8012070 con 0x7f9ccc02d3c0 2020-11-20T12:14:48.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f9ccc00fa20 msgr2=0x7f9ccc01b380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f9ccc00fa20 0x7f9ccc01b380 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f9ccc02d3c0 msgr2=0x7f9ccc02d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f9ccc02d3c0 0x7f9ccc02d7e0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f9cc401d410 msgr2=0x7f9cc401d810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f9cc401d410 0x7f9cc401d810 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f9ccc01c3f0 msgr2=0x7f9ccc01c830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f9ccc01c3f0 0x7f9ccc01c830 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x558ce235d150 msgr2=0x558ce22cd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x558ce235d150 0x558ce22cd330 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.846+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f9cc801f060 msgr2=0x7f9cc8015a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f9cc801f060 0x7f9cc8015a60 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9cd4ea0700 1 -- 172.21.15.187:0/1258933435 reap_dead start 2020-11-20T12:14:48.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9cd4ea0700 1 -- 172.21.15.187:0/1258933435 reap_dead start 2020-11-20T12:14:48.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9cc00559a0 msgr2=0x7f9cc0057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9cc00559a0 0x7f9cc0057e60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f9cc40018e0 tx=0x7f9cc4015000).stop 2020-11-20T12:14:48.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22eb8c0 msgr2=0x558ce235c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:48.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22eb8c0 0x558ce235c530 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f9cc8005fd0 tx=0x7f9cc800c040).stop 2020-11-20T12:14:48.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 shutdown_connections 2020-11-20T12:14:48.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x558ce22eb8c0 0x558ce235c530 secure :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0x7f9cc8005fd0 tx=0x7f9cc800c040).stop 2020-11-20T12:14:48.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 --2- 172.21.15.187:0/1258933435 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9cc00559a0 0x7f9cc0057e60 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:48.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 >> 172.21.15.187:0/1258933435 conn(0x558ce22aaa90 msgr2=0x558ce22e7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:48.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 shutdown_connections 2020-11-20T12:14:48.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:48.847+0000 7f9ce3dfad00 1 -- 172.21.15.187:0/1258933435 wait complete. 2020-11-20T12:14:48.859 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000000 2020-11-20T12:14:48.859 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000001 2020-11-20T12:14:48.859 INFO:teuthology.orchestra.run.smithi187.stdout:10000000001.00000000 2020-11-20T12:14:48.860 INFO:teuthology.orchestra.run.smithi187.stdout:10000000001.00000001 2020-11-20T12:14:48.861 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2020-11-20T12:14:48.861 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:49.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.094+0000 7ff50b59e700 1 Processor -- start 2020-11-20T12:14:49.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.094+0000 7ff50b59e700 1 -- start start 2020-11-20T12:14:49.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.096+0000 7ff50b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c0683f0 0x7ff50c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.096+0000 7ff50b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff50c1487e0 0x7ff50c148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.096+0000 7ff50b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50c14c170 con 0x7ff50c145cc0 2020-11-20T12:14:49.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.096+0000 7ff50b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50c14c410 con 0x7ff50c1487e0 2020-11-20T12:14:49.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.096+0000 7ff50b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c14c6b0 con 0x7ff50c0683f0 2020-11-20T12:14:49.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.097+0000 7ff50ad9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff50c1487e0 0x7ff50c148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.097+0000 7ff501d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c0683f0 0x7ff50c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.097+0000 7ff501d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c0683f0 0x7ff50c066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:50236/0 (socket says 172.21.15.187:50236) 2020-11-20T12:14:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.097+0000 7ff50ad9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff50c1487e0 0x7ff50c148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:41614/0 (socket says 172.21.15.187:41614) 2020-11-20T12:14:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.097+0000 7ff501d9b700 1 -- 172.21.15.187:0/1920645317 learned_addr learned my addr 172.21.15.187:0/1920645317 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.098+0000 7ff501d9b700 1 -- 172.21.15.187:0/1920645317 >> v2:172.21.15.5:3300/0 conn(0x7ff50c1487e0 msgr2=0x7ff50c148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:49.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.098+0000 7ff501d9b700 1 --2- 172.21.15.187:0/1920645317 >> v2:172.21.15.5:3300/0 conn(0x7ff50c1487e0 0x7ff50c148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.098+0000 7ff501d9b700 1 -- 172.21.15.187:0/1920645317 >> v1:172.21.15.5:6789/0 conn(0x7ff50c145cc0 legacy=0x7ff50c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.098+0000 7ff501d9b700 1 -- 172.21.15.187:0/1920645317 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c0658a0 con 0x7ff50c0683f0 2020-11-20T12:14:49.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.098+0000 7ff501d9b700 1 --2- 172.21.15.187:0/1920645317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c0683f0 0x7ff50c066040 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7ff4f4006fd0 tx=0x7ff4f4006eb0).ready entity=mon.2 client_cookie=6da1c71b79a85926 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:49.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.099+0000 7ff509d9b700 1 -- 172.21.15.187:0/1920645317 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4009810 con 0x7ff50c0683f0 2020-11-20T12:14:49.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.099+0000 7ff509d9b700 1 -- 172.21.15.187:0/1920645317 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f4009970 con 0x7ff50c0683f0 2020-11-20T12:14:49.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.099+0000 7ff509d9b700 1 -- 172.21.15.187:0/1920645317 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4009cf0 con 0x7ff50c0683f0 2020-11-20T12:14:49.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.100+0000 7ff50b59e700 1 -- 172.21.15.187:0/1920645317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c0683f0 msgr2=0x7ff50c066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.100+0000 7ff50b59e700 1 --2- 172.21.15.187:0/1920645317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c0683f0 0x7ff50c066040 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7ff4f4006fd0 tx=0x7ff4f4006eb0).stop 2020-11-20T12:14:49.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.100+0000 7ff50b59e700 1 -- 172.21.15.187:0/1920645317 shutdown_connections 2020-11-20T12:14:49.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.100+0000 7ff50b59e700 1 --2- 172.21.15.187:0/1920645317 >> v2:172.21.15.5:3300/0 conn(0x7ff50c1487e0 0x7ff50c148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.100+0000 7ff50b59e700 1 --2- 172.21.15.187:0/1920645317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c0683f0 0x7ff50c066040 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.100+0000 7ff50b59e700 1 -- 172.21.15.187:0/1920645317 >> 172.21.15.187:0/1920645317 conn(0x7ff50c0bab30 msgr2=0x7ff50c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:49.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.100+0000 7ff50b59e700 1 -- 172.21.15.187:0/1920645317 shutdown_connections 2020-11-20T12:14:49.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.100+0000 7ff50b59e700 1 -- 172.21.15.187:0/1920645317 wait complete. 2020-11-20T12:14:49.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.101+0000 7ff50b59e700 1 Processor -- start 2020-11-20T12:14:49.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.101+0000 7ff50b59e700 1 -- start start 2020-11-20T12:14:49.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.102+0000 7ff50b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50c0683f0 0x7ff50c143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.102+0000 7ff50b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c145cc0 0x7ff50c144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.102+0000 7ff50b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c1487e0 0x7ff50c15b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.102+0000 7ff50b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c144620 con 0x7ff50c0683f0 2020-11-20T12:14:49.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.102+0000 7ff50b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c09bc50 con 0x7ff50c1487e0 2020-11-20T12:14:49.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.102+0000 7ff50b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c09bef0 con 0x7ff50c145cc0 2020-11-20T12:14:49.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c1487e0 0x7ff50c15b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c1487e0 0x7ff50c15b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49356/0 (socket says 172.21.15.187:49356) 2020-11-20T12:14:49.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50ad9d700 1 -- 172.21.15.187:0/660030737 learned_addr learned my addr 172.21.15.187:0/660030737 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:49.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff501d9b700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c145cc0 0x7ff50c144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50a59c700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50c0683f0 0x7ff50c143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50ad9d700 1 -- 172.21.15.187:0/660030737 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c145cc0 msgr2=0x7ff50c144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50ad9d700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c145cc0 0x7ff50c144130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50ad9d700 1 -- 172.21.15.187:0/660030737 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50c0683f0 msgr2=0x7ff50c143c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50ad9d700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50c0683f0 0x7ff50c143c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.103+0000 7ff50ad9d700 1 -- 172.21.15.187:0/660030737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c09c190 con 0x7ff50c1487e0 2020-11-20T12:14:49.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.104+0000 7ff50ad9d700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c1487e0 0x7ff50c15b870 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7ff50400ed50 tx=0x7ff5040091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:49.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.104+0000 7ff502ffd700 1 -- 172.21.15.187:0/660030737 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50400cc70 con 0x7ff50c1487e0 2020-11-20T12:14:49.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.104+0000 7ff502ffd700 1 -- 172.21.15.187:0/660030737 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50400cdd0 con 0x7ff50c1487e0 2020-11-20T12:14:49.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.104+0000 7ff502ffd700 1 -- 172.21.15.187:0/660030737 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504012620 con 0x7ff50c1487e0 2020-11-20T12:14:49.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.104+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c0b3f30 con 0x7ff50c1487e0 2020-11-20T12:14:49.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.104+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c15bfa0 con 0x7ff50c1487e0 2020-11-20T12:14:49.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.105+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c0b4120 con 0x7ff50c1487e0 2020-11-20T12:14:49.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.106+0000 7ff502ffd700 1 -- 172.21.15.187:0/660030737 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff504022070 con 0x7ff50c1487e0 2020-11-20T12:14:49.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.106+0000 7ff502ffd700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4e8056d20 0x7ff4e80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.106+0000 7ff502ffd700 1 -- 172.21.15.187:0/660030737 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff504070780 con 0x7ff50c1487e0 2020-11-20T12:14:49.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.106+0000 7ff50a59c700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4e8056d20 0x7ff4e80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.107+0000 7ff50a59c700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4e8056d20 0x7ff4e80591e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7ff4fc001410 tx=0x7ff4fc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:49.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.110+0000 7ff502ffd700 1 -- 172.21.15.187:0/660030737 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff50c0b4120 con 0x7ff50c1487e0 2020-11-20T12:14:49.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.358+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff50c0b1bb0 con 0x7ff50c1487e0 2020-11-20T12:14:49.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.359+0000 7ff502ffd700 1 -- 172.21.15.187:0/660030737 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7ff50c0b1bb0 con 0x7ff50c1487e0 2020-11-20T12:14:49.363 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:49.363 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:49.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.361+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4e8056d20 msgr2=0x7ff4e80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4e8056d20 0x7ff4e80591e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7ff4fc001410 tx=0x7ff4fc009040).stop 2020-11-20T12:14:49.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c1487e0 msgr2=0x7ff50c15b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c1487e0 0x7ff50c15b870 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7ff50400ed50 tx=0x7ff5040091d0).stop 2020-11-20T12:14:49.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 shutdown_connections 2020-11-20T12:14:49.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4e8056d20 0x7ff4e80591e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50c0683f0 0x7ff50c143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c1487e0 0x7ff50c15b870 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 --2- 172.21.15.187:0/660030737 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50c145cc0 0x7ff50c144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 >> 172.21.15.187:0/660030737 conn(0x7ff50c0bab30 msgr2=0x7ff50c0baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 shutdown_connections 2020-11-20T12:14:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.362+0000 7ff50b59e700 1 -- 172.21.15.187:0/660030737 wait complete. 2020-11-20T12:14:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:49.387 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:14:49.688 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:14:49.689 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:14:49.689 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data getxattr 10000000001.00000000 parent 2020-11-20T12:14:49.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.775+0000 7f4504e34d00 1 Processor -- start 2020-11-20T12:14:49.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.775+0000 7f4504e34d00 1 -- start start 2020-11-20T12:14:49.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f4504e34d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad353a0 0x563eaadfc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f4504e34d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 0x563eaadedd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f4504e34d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 0x563eaad5b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f4504e34d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eaae02d70 con 0x563eaad76ad0 2020-11-20T12:14:49.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f4504e34d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eaae02ad0 con 0x563eaad353a0 2020-11-20T12:14:49.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f4504e34d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x563eaae00b80 con 0x563eaad5cd40 2020-11-20T12:14:49.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f44f56d9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad353a0 0x563eaadfc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f44f56d9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad353a0 0x563eaadfc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49372/0 (socket says 172.21.15.187:49372) 2020-11-20T12:14:49.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.776+0000 7f44f56d9700 1 -- 172.21.15.187:0/2540528823 learned_addr learned my addr 172.21.15.187:0/2540528823 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:49.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44f4ed8700 1 --2- 172.21.15.187:0/2540528823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 0x563eaadedd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44f56d9700 1 -- 172.21.15.187:0/2540528823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 msgr2=0x563eaadedd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:49.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44f56d9700 1 --2- 172.21.15.187:0/2540528823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 0x563eaadedd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44f56d9700 1 -- 172.21.15.187:0/2540528823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 msgr2=0x563eaad5b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:49.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44f56d9700 1 --2- 172.21.15.187:0/2540528823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 0x563eaad5b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:49.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44f56d9700 1 -- 172.21.15.187:0/2540528823 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563eaacd90c0 con 0x563eaad353a0 2020-11-20T12:14:49.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44f56d9700 1 --2- 172.21.15.187:0/2540528823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad353a0 0x563eaadfc9c0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f44e800aae0 tx=0x7f44e8005cc0).ready entity=mon.1 client_cookie=9951617ff66d17cf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:49.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44e77fe700 1 -- 172.21.15.187:0/2540528823 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e800f070 con 0x563eaad353a0 2020-11-20T12:14:49.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44e77fe700 1 -- 172.21.15.187:0/2540528823 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8008c40 con 0x563eaad353a0 2020-11-20T12:14:49.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.777+0000 7f44e77fe700 1 -- 172.21.15.187:0/2540528823 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8013410 con 0x563eaad353a0 2020-11-20T12:14:49.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 -- 172.21.15.187:0/2540528823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad353a0 msgr2=0x563eaadfc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 --2- 172.21.15.187:0/2540528823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad353a0 0x563eaadfc9c0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f44e800aae0 tx=0x7f44e8005cc0).stop 2020-11-20T12:14:49.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 -- 172.21.15.187:0/2540528823 shutdown_connections 2020-11-20T12:14:49.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 --2- 172.21.15.187:0/2540528823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 0x563eaad5b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:49.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 --2- 172.21.15.187:0/2540528823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad353a0 0x563eaadfc9c0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 --2- 172.21.15.187:0/2540528823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 0x563eaadedd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 -- 172.21.15.187:0/2540528823 >> 172.21.15.187:0/2540528823 conn(0x563eaad340e0 msgr2=0x563eaadfd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:49.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 -- 172.21.15.187:0/2540528823 shutdown_connections 2020-11-20T12:14:49.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.778+0000 7f4504e34d00 1 -- 172.21.15.187:0/2540528823 wait complete. 2020-11-20T12:14:49.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.779+0000 7f4504e34d00 1 Processor -- start 2020-11-20T12:14:49.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.779+0000 7f4504e34d00 1 -- start start 2020-11-20T12:14:49.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f4504e34d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 0x563eaae14460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f4504e34d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 0x563eaad5c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f4504e34d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad5a8f0 0x563eaadff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f4504e34d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eaadfef30 con 0x563eaad76ad0 2020-11-20T12:14:49.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f4504e34d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eaadff1d0 con 0x563eaad5a8f0 2020-11-20T12:14:49.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f4504e34d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x563eaad71200 con 0x563eaad5cd40 2020-11-20T12:14:49.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f44f5eda700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad5a8f0 0x563eaadff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f44f5eda700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad5a8f0 0x563eaadff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49378/0 (socket says 172.21.15.187:49378) 2020-11-20T12:14:49.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.780+0000 7f44f5eda700 1 -- 172.21.15.187:0/1678085976 learned_addr learned my addr 172.21.15.187:0/1678085976 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:49.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f4ed8700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 0x563eaad5c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f5eda700 1 -- 172.21.15.187:0/1678085976 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 msgr2=0x563eaae14460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f56d9700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 0x563eaae14460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f5eda700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563eaad5cd40 0x563eaae14460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f5eda700 1 -- 172.21.15.187:0/1678085976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 msgr2=0x563eaad5c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f5eda700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 0x563eaad5c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f5eda700 1 -- 172.21.15.187:0/1678085976 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563eaacda4e0 con 0x563eaad5a8f0 2020-11-20T12:14:49.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f4ed8700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563eaad76ad0 0x563eaad5c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:49.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44f5eda700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad5a8f0 0x563eaadff670 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f44f000f000 tx=0x7f44f0014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:49.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44e5ffb700 1 -- 172.21.15.187:0/1678085976 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f0010c40 con 0x563eaad5a8f0 2020-11-20T12:14:49.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44e5ffb700 1 -- 172.21.15.187:0/1678085976 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f0010da0 con 0x563eaad5a8f0 2020-11-20T12:14:49.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f44e5ffb700 1 -- 172.21.15.187:0/1678085976 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f001c680 con 0x563eaad5a8f0 2020-11-20T12:14:49.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.781+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563eaacd5ec0 con 0x563eaad5a8f0 2020-11-20T12:14:49.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.782+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563eaacb4250 con 0x563eaad5a8f0 2020-11-20T12:14:49.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.783+0000 7f44e5ffb700 1 -- 172.21.15.187:0/1678085976 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f44f0023030 con 0x563eaad5a8f0 2020-11-20T12:14:49.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.784+0000 7f44e5ffb700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44d80559a0 0x7f44d8057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.784+0000 7f44e5ffb700 1 -- 172.21.15.187:0/1678085976 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f44f00705c0 con 0x563eaad5a8f0 2020-11-20T12:14:49.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.784+0000 7f44f56d9700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44d80559a0 0x7f44d8057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.784+0000 7f4504e34d00 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x563eaad6f520 0x563eaad6f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.784+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:1 20.7 20:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected e98) v8 -- 0x563eaadea470 con 0x563eaad6f520 2020-11-20T12:14:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.784+0000 7f44f4ed8700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x563eaad6f520 0x563eaad6f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.785+0000 7f44f56d9700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44d80559a0 0x7f44d8057e60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x563eaadfdd40 tx=0x7f44e8018000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:49.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.785+0000 7f44f4ed8700 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x563eaad6f520 0x563eaad6f900 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:49.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f44f4ed8700 1 -- 172.21.15.187:0/1678085976 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (64) out=64b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+64 (crc 0 0 0) 0x7f44ec008850 con 0x563eaad6f520 2020-11-20T12:14:49.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x563eaad6f520 msgr2=0x563eaad6f900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x563eaad6f520 0x563eaad6f900 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:49.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44d80559a0 msgr2=0x7f44d8057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44d80559a0 0x7f44d8057e60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x563eaadfdd40 tx=0x7f44e8018000).stop 2020-11-20T12:14:49.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad5a8f0 msgr2=0x563eaadff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:49.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 --2- 172.21.15.187:0/1678085976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x563eaad5a8f0 0x563eaadff670 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f44f000f000 tx=0x7f44f0014000).stop 2020-11-20T12:14:49.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f44f5eda700 1 -- 172.21.15.187:0/1678085976 reap_dead start 2020-11-20T12:14:49.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 shutdown_connections 2020-11-20T12:14:49.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 >> 172.21.15.187:0/1678085976 conn(0x563eaad340e0 msgr2=0x563eaadfd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:49.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 shutdown_connections 2020-11-20T12:14:49.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:49.786+0000 7f4504e34d00 1 -- 172.21.15.187:0/1678085976 wait complete. 2020-11-20T12:14:49.801 INFO:teuthology.orchestra.run.smithi187.stdout::linkto_b 2020-11-20T12:14:49.802 INFO:teuthology.orchestra.run.smithi187:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2020-11-20T12:14:49.882 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:14:49.882 INFO:teuthology.orchestra.run.smithi187.stdout: "ino": 1099511627777, 2020-11-20T12:14:49.883 INFO:teuthology.orchestra.run.smithi187.stdout: "ancestors": [ 2020-11-20T12:14:49.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:14:49.883 INFO:teuthology.orchestra.run.smithi187.stdout: "dirino": 1, 2020-11-20T12:14:49.883 INFO:teuthology.orchestra.run.smithi187.stdout: "dname": "linkto_b", 2020-11-20T12:14:49.883 INFO:teuthology.orchestra.run.smithi187.stdout: "version": 23 2020-11-20T12:14:49.884 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:14:49.884 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2020-11-20T12:14:49.884 INFO:teuthology.orchestra.run.smithi187.stdout: "pool": 20, 2020-11-20T12:14:49.884 INFO:teuthology.orchestra.run.smithi187.stdout: "old_pools": [] 2020-11-20T12:14:49.884 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:14:49.885 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:49.886 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:14:49.886 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:14:50.004 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:14:50.029 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:14:50.141 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:14:50.153 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:14:50.154 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:14:50.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:14:50.155 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:14:50.298 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:14:50.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:14:50.299 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:14:50.353 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:14:50.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.590+0000 7f3b2a6e9700 1 Processor -- start 2020-11-20T12:14:50.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.590+0000 7f3b2a6e9700 1 -- start start 2020-11-20T12:14:50.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.590+0000 7f3b2a6e9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b241484d0 0x7f3b24061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:50.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b2a6e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 0x7f3b240b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:50.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b2a6e9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240b1300 con 0x7f3b24066f70 2020-11-20T12:14:50.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b2a6e9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b24067730 con 0x7f3b241484d0 2020-11-20T12:14:50.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b2a6e9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240b15a0 con 0x7f3b240622c0 2020-11-20T12:14:50.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b29ee8700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3b24066f70 0x7f3b240b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:45994/0 (socket says 172.21.15.187:45994) 2020-11-20T12:14:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b29ee8700 1 -- 172.21.15.187:0/4285535486 learned_addr learned my addr 172.21.15.187:0/4285535486 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b296e7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b241484d0 0x7f3b24061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b28ee6700 1 --2- 172.21.15.187:0/4285535486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 0x7f3b240b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b13fff700 1 -- 172.21.15.187:0/4285535486 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173360867 0 0) 0x7f3b240b1300 con 0x7f3b24066f70 2020-11-20T12:14:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b13fff700 1 -- 172.21.15.187:0/4285535486 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b0c003240 con 0x7f3b24066f70 2020-11-20T12:14:50.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b296e7700 1 -- 172.21.15.187:0/4285535486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 msgr2=0x7f3b240b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:50.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b296e7700 1 --2- 172.21.15.187:0/4285535486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 0x7f3b240b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:50.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b296e7700 1 -- 172.21.15.187:0/4285535486 >> v1:172.21.15.187:6789/0 conn(0x7f3b24066f70 legacy=0x7f3b240b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:50.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b296e7700 1 -- 172.21.15.187:0/4285535486 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240b1840 con 0x7f3b241484d0 2020-11-20T12:14:50.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.592+0000 7f3b296e7700 1 --2- 172.21.15.187:0/4285535486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b241484d0 0x7f3b24061dd0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7f3b2000b9a0 tx=0x7f3b20006df0).ready entity=mon.1 client_cookie=dee7c2ea87c00e11 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:50.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.593+0000 7f3b13fff700 1 -- 172.21.15.187:0/4285535486 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20009680 con 0x7f3b241484d0 2020-11-20T12:14:50.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.593+0000 7f3b13fff700 1 -- 172.21.15.187:0/4285535486 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b200097e0 con 0x7f3b241484d0 2020-11-20T12:14:50.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.593+0000 7f3b13fff700 1 -- 172.21.15.187:0/4285535486 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20009b60 con 0x7f3b241484d0 2020-11-20T12:14:50.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.593+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/4285535486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b241484d0 msgr2=0x7f3b24061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.593+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/4285535486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b241484d0 0x7f3b24061dd0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7f3b2000b9a0 tx=0x7f3b20006df0).stop 2020-11-20T12:14:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.594+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/4285535486 shutdown_connections 2020-11-20T12:14:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.594+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/4285535486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b241484d0 0x7f3b24061dd0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:50.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.594+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/4285535486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 0x7f3b240b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:50.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.594+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/4285535486 >> 172.21.15.187:0/4285535486 conn(0x7f3b240bab30 msgr2=0x7f3b240baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:50.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.594+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/4285535486 shutdown_connections 2020-11-20T12:14:50.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.594+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/4285535486 wait complete. 2020-11-20T12:14:50.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.596+0000 7f3b2a6e9700 1 Processor -- start 2020-11-20T12:14:50.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.596+0000 7f3b2a6e9700 1 -- start start 2020-11-20T12:14:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b2a6e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 0x7f3b24134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b2a6e9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b24066f70 0x7f3b24135270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b2a6e9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b241484d0 0x7f3b24159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b2a6e9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24136810 con 0x7f3b241484d0 2020-11-20T12:14:50.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b2a6e9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24135760 con 0x7f3b24066f70 2020-11-20T12:14:50.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b2a6e9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24136a00 con 0x7f3b240622c0 2020-11-20T12:14:50.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b28ee6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b24066f70 0x7f3b24135270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:50.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b28ee6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b24066f70 0x7f3b24135270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49392/0 (socket says 172.21.15.187:49392) 2020-11-20T12:14:50.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.597+0000 7f3b28ee6700 1 -- 172.21.15.187:0/1309996045 learned_addr learned my addr 172.21.15.187:0/1309996045 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:50.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b28ee6700 1 -- 172.21.15.187:0/1309996045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 msgr2=0x7f3b24134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:50.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b28ee6700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 0x7f3b24134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:50.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b28ee6700 1 -- 172.21.15.187:0/1309996045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b241484d0 msgr2=0x7f3b24159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:50.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b28ee6700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b241484d0 0x7f3b24159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:50.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b28ee6700 1 -- 172.21.15.187:0/1309996045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b24136ca0 con 0x7f3b24066f70 2020-11-20T12:14:50.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b28ee6700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b24066f70 0x7f3b24135270 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f3b1400eba0 tx=0x7f3b14006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:50.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b117fa700 1 -- 172.21.15.187:0/1309996045 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1400c9b0 con 0x7f3b24066f70 2020-11-20T12:14:50.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b117fa700 1 -- 172.21.15.187:0/1309996045 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1400cb10 con 0x7f3b24066f70 2020-11-20T12:14:50.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b24138690 con 0x7f3b24066f70 2020-11-20T12:14:50.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b117fa700 1 -- 172.21.15.187:0/1309996045 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1400ce90 con 0x7f3b24066f70 2020-11-20T12:14:50.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.598+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b24159d70 con 0x7f3b24066f70 2020-11-20T12:14:50.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.600+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b24136e30 con 0x7f3b24066f70 2020-11-20T12:14:50.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.600+0000 7f3b117fa700 1 -- 172.21.15.187:0/1309996045 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3b1401a030 con 0x7f3b24066f70 2020-11-20T12:14:50.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.601+0000 7f3b117fa700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3afc056d50 0x7f3afc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:50.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.601+0000 7f3b117fa700 1 -- 172.21.15.187:0/1309996045 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3b1406fbe0 con 0x7f3b24066f70 2020-11-20T12:14:50.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.601+0000 7f3b296e7700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3afc056d50 0x7f3afc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:50.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.602+0000 7f3b296e7700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3afc056d50 0x7f3afc059210 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f3b20006e30 tx=0x7f3b20015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:50.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.605+0000 7f3b117fa700 1 -- 172.21.15.187:0/1309996045 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3b140450e0 con 0x7f3b24066f70 2020-11-20T12:14:50.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.793+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3b24138820 con 0x7f3b24066f70 2020-11-20T12:14:50.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.803+0000 7f3b117fa700 1 -- 172.21.15.187:0/1309996045 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3b24138820 con 0x7f3b24066f70 2020-11-20T12:14:50.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3afc056d50 msgr2=0x7f3afc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:50.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3afc056d50 0x7f3afc059210 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f3b20006e30 tx=0x7f3b20015000).stop 2020-11-20T12:14:50.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b24066f70 msgr2=0x7f3b24135270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:50.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b24066f70 0x7f3b24135270 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f3b1400eba0 tx=0x7f3b14006df0).stop 2020-11-20T12:14:50.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 shutdown_connections 2020-11-20T12:14:50.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3afc056d50 0x7f3afc059210 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:50.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b241484d0 0x7f3b24159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:50.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b240622c0 0x7f3b24134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:50.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 --2- 172.21.15.187:0/1309996045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b24066f70 0x7f3b24135270 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:50.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 >> 172.21.15.187:0/1309996045 conn(0x7f3b240bab30 msgr2=0x7f3b24065770 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:50.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 shutdown_connections 2020-11-20T12:14:50.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:50.807+0000 7f3b2a6e9700 1 -- 172.21.15.187:0/1309996045 wait complete. 2020-11-20T12:14:50.835 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup' 2020-11-20T12:14:51.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.080+0000 7f38cf067700 1 Processor -- start 2020-11-20T12:14:51.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.080+0000 7f38cf067700 1 -- start start 2020-11-20T12:14:51.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.082+0000 7f38cf067700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f38c80b57b0 0x7f38c8068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:51.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.082+0000 7f38cf067700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c80b5f50 0x7f38c80b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:51.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.082+0000 7f38cf067700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c8144b80 con 0x7f38c81482d0 2020-11-20T12:14:51.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.082+0000 7f38cf067700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c81489e0 con 0x7f38c80b57b0 2020-11-20T12:14:51.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.082+0000 7f38cf067700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8148c80 con 0x7f38c80b5f50 2020-11-20T12:14:51.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.083+0000 7f38ce065700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f38c80b57b0 0x7f38c8068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:51.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.083+0000 7f38cd864700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f38c81482d0 0x7f38c8062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:39514/0 (socket says 172.21.15.187:39514) 2020-11-20T12:14:51.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.083+0000 7f38cd864700 1 -- 172.21.15.187:0/4255426093 learned_addr learned my addr 172.21.15.187:0/4255426093 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:51.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.083+0000 7f38ce866700 1 --2- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c80b5f50 0x7f38c80b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:51.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.084+0000 7f38cd063700 1 -- 172.21.15.187:0/4255426093 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080283324 0 0) 0x7f38c8144b80 con 0x7f38c81482d0 2020-11-20T12:14:51.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.084+0000 7f38cd063700 1 -- 172.21.15.187:0/4255426093 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b0003240 con 0x7f38c81482d0 2020-11-20T12:14:51.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.084+0000 7f38ce065700 1 -- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c80b5f50 msgr2=0x7f38c80b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:51.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.084+0000 7f38ce065700 1 --2- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c80b5f50 0x7f38c80b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:51.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.084+0000 7f38ce065700 1 -- 172.21.15.187:0/4255426093 >> v1:172.21.15.5:6789/0 conn(0x7f38c81482d0 legacy=0x7f38c8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:51.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.084+0000 7f38ce065700 1 -- 172.21.15.187:0/4255426093 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c8148f20 con 0x7f38c80b57b0 2020-11-20T12:14:51.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.084+0000 7f38ce866700 1 --2- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c80b5f50 0x7f38c80b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:14:51.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.085+0000 7f38ce065700 1 --2- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 0x7f38c8068230 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f38c400b9a0 tx=0x7f38c4006df0).ready entity=mon.0 client_cookie=9bab446da38bd28 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:51.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.085+0000 7f38cd063700 1 -- 172.21.15.187:0/4255426093 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4009680 con 0x7f38c80b57b0 2020-11-20T12:14:51.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.085+0000 7f38cd063700 1 -- 172.21.15.187:0/4255426093 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c40097e0 con 0x7f38c80b57b0 2020-11-20T12:14:51.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.085+0000 7f38cd063700 1 -- 172.21.15.187:0/4255426093 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4009b60 con 0x7f38c80b57b0 2020-11-20T12:14:51.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.086+0000 7f38cf067700 1 -- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 msgr2=0x7f38c8068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:51.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.086+0000 7f38cf067700 1 --2- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 0x7f38c8068230 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f38c400b9a0 tx=0x7f38c4006df0).stop 2020-11-20T12:14:51.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.086+0000 7f38cf067700 1 -- 172.21.15.187:0/4255426093 shutdown_connections 2020-11-20T12:14:51.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.086+0000 7f38cf067700 1 --2- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c80b5f50 0x7f38c80b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:51.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.086+0000 7f38cf067700 1 --2- 172.21.15.187:0/4255426093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 0x7f38c8068230 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:51.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.086+0000 7f38cf067700 1 -- 172.21.15.187:0/4255426093 >> 172.21.15.187:0/4255426093 conn(0x7f38c80bab30 msgr2=0x7f38c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:51.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.086+0000 7f38cf067700 1 -- 172.21.15.187:0/4255426093 shutdown_connections 2020-11-20T12:14:51.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.086+0000 7f38cf067700 1 -- 172.21.15.187:0/4255426093 wait complete. 2020-11-20T12:14:51.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.088+0000 7f38cf067700 1 Processor -- start 2020-11-20T12:14:51.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.088+0000 7f38cf067700 1 -- start start 2020-11-20T12:14:51.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.089+0000 7f38cf067700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 0x7f38c809bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:51.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38cf067700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c80b5f50 0x7f38c809c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:51.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38cf067700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c81482d0 0x7f38c809c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:51.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38cf067700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c809b7b0 con 0x7f38c80b57b0 2020-11-20T12:14:51.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38cf067700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c809ba50 con 0x7f38c80b5f50 2020-11-20T12:14:51.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38cf067700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c815c610 con 0x7f38c81482d0 2020-11-20T12:14:51.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38cd864700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c80b5f50 0x7f38c809c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:51.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38cd864700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c80b5f50 0x7f38c809c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49404/0 (socket says 172.21.15.187:49404) 2020-11-20T12:14:51.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38cd864700 1 -- 172.21.15.187:0/1262209581 learned_addr learned my addr 172.21.15.187:0/1262209581 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:51.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.090+0000 7f38ce065700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 0x7f38c809bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:51.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38ce866700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c81482d0 0x7f38c809c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:51.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38cd864700 1 -- 172.21.15.187:0/1262209581 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c81482d0 msgr2=0x7f38c809c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:51.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38cd864700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c81482d0 0x7f38c809c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:51.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38cd864700 1 -- 172.21.15.187:0/1262209581 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 msgr2=0x7f38c809bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:51.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38cd864700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 0x7f38c809bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:51.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38cd864700 1 -- 172.21.15.187:0/1262209581 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c809d5b0 con 0x7f38c80b5f50 2020-11-20T12:14:51.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38cd864700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c80b5f50 0x7f38c809c3e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f38b800c6e0 tx=0x7f38b8007ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:51.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38b67fc700 1 -- 172.21.15.187:0/1262209581 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b80062e0 con 0x7f38c80b5f50 2020-11-20T12:14:51.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38b67fc700 1 -- 172.21.15.187:0/1262209581 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b8004500 con 0x7f38c80b5f50 2020-11-20T12:14:51.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38b67fc700 1 -- 172.21.15.187:0/1262209581 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b80048a0 con 0x7f38c80b5f50 2020-11-20T12:14:51.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c815c130 con 0x7f38c80b5f50 2020-11-20T12:14:51.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.091+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c80a0530 con 0x7f38c80b5f50 2020-11-20T12:14:51.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.093+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c809bc10 con 0x7f38c80b5f50 2020-11-20T12:14:51.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.093+0000 7f38b67fc700 1 -- 172.21.15.187:0/1262209581 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f38b8010070 con 0x7f38c80b5f50 2020-11-20T12:14:51.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.094+0000 7f38b67fc700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f38a0056d20 0x7f38a00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:51.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.094+0000 7f38b67fc700 1 -- 172.21.15.187:0/1262209581 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f38b806f6c0 con 0x7f38c80b5f50 2020-11-20T12:14:51.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.095+0000 7f38ce065700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f38a0056d20 0x7f38a00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:51.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.095+0000 7f38ce065700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f38a0056d20 0x7f38a00591e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f38c4006e30 tx=0x7f38c4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:51.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.098+0000 7f38b67fc700 1 -- 172.21.15.187:0/1262209581 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f38b8044b10 con 0x7f38c80b5f50 2020-11-20T12:14:51.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:51.351+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup"]} v 0) v1 -- 0x7f38c8061a20 con 0x7f38c80b5f50 2020-11-20T12:14:51.693 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:14:51.702 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:14:52.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.310+0000 7f38b67fc700 1 -- 172.21.15.187:0/1262209581 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup"]}]=0 v507) v1 ==== 139+0+0 (secure 0 0 0) 0x7f38b8048130 con 0x7f38c80b5f50 2020-11-20T12:14:52.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f38a0056d20 msgr2=0x7f38a00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:52.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f38a0056d20 0x7f38a00591e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f38c4006e30 tx=0x7f38c4015000).stop 2020-11-20T12:14:52.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c80b5f50 msgr2=0x7f38c809c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:52.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c80b5f50 0x7f38c809c3e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f38b800c6e0 tx=0x7f38b8007ae0).stop 2020-11-20T12:14:52.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 shutdown_connections 2020-11-20T12:14:52.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f38a0056d20 0x7f38a00591e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38c80b57b0 0x7f38c809bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38c81482d0 0x7f38c809c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 --2- 172.21.15.187:0/1262209581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c80b5f50 0x7f38c809c3e0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.313+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 >> 172.21.15.187:0/1262209581 conn(0x7f38c80bab30 msgr2=0x7f38c8065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:52.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.314+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 shutdown_connections 2020-11-20T12:14:52.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.314+0000 7f38cf067700 1 -- 172.21.15.187:0/1262209581 wait complete. 2020-11-20T12:14:52.347 INFO:tasks.cephfs_test_runner:test_mv_hardlink_cleanup (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:14:52.348 INFO:tasks.cephfs_test_runner:Starting test: test_open_inode (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:14:52.349 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_open_inode' 2020-11-20T12:14:52.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.599+0000 7f2acdf11700 1 Processor -- start 2020-11-20T12:14:52.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.599+0000 7f2acdf11700 1 -- start start 2020-11-20T12:14:52.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.600+0000 7f2acdf11700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac80bf800 0x7f2ac814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.600+0000 7f2acdf11700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2ac809f560 0x7f2ac80a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.600+0000 7f2acdf11700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac814d110 con 0x7f2ac80bfbe0 2020-11-20T12:14:52.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.601+0000 7f2acdf11700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac814cbd0 con 0x7f2ac809f560 2020-11-20T12:14:52.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.601+0000 7f2acdf11700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac814ce70 con 0x7f2ac80bf800 2020-11-20T12:14:52.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.601+0000 7f2ac7fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2ac80bfbe0 0x7f2ac81480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46018/0 (socket says 172.21.15.187:46018) 2020-11-20T12:14:52.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.601+0000 7f2ac7fff700 1 -- 172.21.15.187:0/2782351350 learned_addr learned my addr 172.21.15.187:0/2782351350 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:52.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.601+0000 7f2accf0f700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac80bf800 0x7f2ac814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.601+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49929048 0 0) 0x7f2ac814d110 con 0x7f2ac80bfbe0 2020-11-20T12:14:52.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa8003240 con 0x7f2ac80bfbe0 2020-11-20T12:14:52.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2acd710700 1 --2- 172.21.15.187:0/2782351350 >> v2:172.21.15.5:3300/0 conn(0x7f2ac809f560 0x7f2ac80a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3394245369 0 0) 0x7f2aa8003240 con 0x7f2ac80bfbe0 2020-11-20T12:14:52.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac80bf800 msgr2=0x7f2ac814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:52.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac80bf800 0x7f2ac814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 >> v2:172.21.15.5:3300/0 conn(0x7f2ac809f560 msgr2=0x7f2ac80a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:52.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 --2- 172.21.15.187:0/2782351350 >> v2:172.21.15.5:3300/0 conn(0x7f2ac809f560 0x7f2ac80a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80a4cc0 con 0x7f2ac80bfbe0 2020-11-20T12:14:52.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f2ab00014e0 con 0x7f2ac80bfbe0 2020-11-20T12:14:52.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 >> v1:172.21.15.187:6789/0 conn(0x7f2ac80bfbe0 legacy=0x7f2ac81480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:52.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2aa80037b0 0x7f2aa8003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac80bf800 0x7f2aa80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aa80049d0 0x7f2aa8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac814cbd0 con 0x7f2aa80037b0 2020-11-20T12:14:52.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac814ce70 con 0x7f2aa80049d0 2020-11-20T12:14:52.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.602+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80042a0 con 0x7f2ac80bf800 2020-11-20T12:14:52.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.603+0000 7f2acd710700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aa80049d0 0x7f2aa8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.603+0000 7f2ac7fff700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac80bf800 0x7f2aa80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.603+0000 7f2accf0f700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2aa80037b0 0x7f2aa8003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.604+0000 7f2acd710700 1 -- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac80bf800 msgr2=0x7f2aa80044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:52.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.604+0000 7f2acd710700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac80bf800 0x7f2aa80044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.604+0000 7f2acd710700 1 -- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2aa80037b0 msgr2=0x7f2aa8003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:52.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.604+0000 7f2acd710700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2aa80037b0 0x7f2aa8003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.604+0000 7f2acd710700 1 -- 172.21.15.187:0/2782351350 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8003240 con 0x7f2aa80049d0 2020-11-20T12:14:52.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.604+0000 7f2acd710700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aa80049d0 0x7f2aa8004e50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f2ab800ca20 tx=0x7f2ab8008040).ready entity=mon.1 client_cookie=4567731800022c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:52.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.605+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800a840 con 0x7f2aa80049d0 2020-11-20T12:14:52.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.605+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab800c030 con 0x7f2aa80049d0 2020-11-20T12:14:52.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.605+0000 7f2ac77fe700 1 -- 172.21.15.187:0/2782351350 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800abb0 con 0x7f2aa80049d0 2020-11-20T12:14:52.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.605+0000 7f2acdf11700 1 -- 172.21.15.187:0/2782351350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aa80049d0 msgr2=0x7f2aa8004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:52.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.605+0000 7f2acdf11700 1 --2- 172.21.15.187:0/2782351350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aa80049d0 0x7f2aa8004e50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f2ab800ca20 tx=0x7f2ab8008040).stop 2020-11-20T12:14:52.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.605+0000 7f2acd710700 1 -- 172.21.15.187:0/2782351350 reap_dead start 2020-11-20T12:14:52.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.605+0000 7f2acdf11700 1 -- 172.21.15.187:0/2782351350 shutdown_connections 2020-11-20T12:14:52.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.605+0000 7f2acdf11700 1 -- 172.21.15.187:0/2782351350 >> 172.21.15.187:0/2782351350 conn(0x7f2ac8068370 msgr2=0x7f2ac80bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:52.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.606+0000 7f2acdf11700 1 -- 172.21.15.187:0/2782351350 shutdown_connections 2020-11-20T12:14:52.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.606+0000 7f2acdf11700 1 -- 172.21.15.187:0/2782351350 wait complete. 2020-11-20T12:14:52.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.607+0000 7f2acdf11700 1 Processor -- start 2020-11-20T12:14:52.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.607+0000 7f2acdf11700 1 -- start start 2020-11-20T12:14:52.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.608+0000 7f2acdf11700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac815bbb0 0x7f2ac80b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.608+0000 7f2acdf11700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8146700 0x7f2ac815bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.608+0000 7f2acdf11700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ac815c480 0x7f2ac8140460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.608+0000 7f2acdf11700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8146b80 con 0x7f2ac8146700 2020-11-20T12:14:52.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.608+0000 7f2acdf11700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8145fc0 con 0x7f2ac815c480 2020-11-20T12:14:52.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.608+0000 7f2acdf11700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8146260 con 0x7f2ac815bbb0 2020-11-20T12:14:52.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.608+0000 7f2acd710700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ac815c480 0x7f2ac8140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2acd710700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ac815c480 0x7f2ac8140460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49428/0 (socket says 172.21.15.187:49428) 2020-11-20T12:14:52.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2acd710700 1 -- 172.21.15.187:0/4267869579 learned_addr learned my addr 172.21.15.187:0/4267869579 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:52.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2accf0f700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac815bbb0 0x7f2ac80b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2ac7fff700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8146700 0x7f2ac815bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2acd710700 1 -- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac815bbb0 msgr2=0x7f2ac80b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:52.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2acd710700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac815bbb0 0x7f2ac80b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2acd710700 1 -- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8146700 msgr2=0x7f2ac815bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:52.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2acd710700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8146700 0x7f2ac815bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:52.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2acd710700 1 -- 172.21.15.187:0/4267869579 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac8140950 con 0x7f2ac815c480 2020-11-20T12:14:52.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2ac7fff700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8146700 0x7f2ac815bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:52.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.609+0000 7f2acd710700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ac815c480 0x7f2ac8140460 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f2ab800c9f0 tx=0x7f2ab8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:52.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.610+0000 7f2ac4ff9700 1 -- 172.21.15.187:0/4267869579 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab80115e0 con 0x7f2ac815c480 2020-11-20T12:14:52.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.610+0000 7f2ac4ff9700 1 -- 172.21.15.187:0/4267869579 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8011740 con 0x7f2ac815c480 2020-11-20T12:14:52.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.610+0000 7f2ac4ff9700 1 -- 172.21.15.187:0/4267869579 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8011ae0 con 0x7f2ac815c480 2020-11-20T12:14:52.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.610+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac8140ba0 con 0x7f2ac815c480 2020-11-20T12:14:52.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.610+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac8149f90 con 0x7f2ac815c480 2020-11-20T12:14:52.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.611+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac8145e00 con 0x7f2ac815c480 2020-11-20T12:14:52.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.611+0000 7f2ac4ff9700 1 -- 172.21.15.187:0/4267869579 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2ab800c5f0 con 0x7f2ac815c480 2020-11-20T12:14:52.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.612+0000 7f2ac4ff9700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2aa4056d50 0x7f2aa4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:52.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.612+0000 7f2ac4ff9700 1 -- 172.21.15.187:0/4267869579 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2ab8071b20 con 0x7f2ac815c480 2020-11-20T12:14:52.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.612+0000 7f2accf0f700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2aa4056d50 0x7f2aa4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:52.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.613+0000 7f2accf0f700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2aa4056d50 0x7f2aa4059210 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f2ac00090e0 tx=0x7f2ac0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:52.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.616+0000 7f2ac4ff9700 1 -- 172.21.15.187:0/4267869579 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2ab804ae60 con 0x7f2ac815c480 2020-11-20T12:14:52.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:52.867+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_open_inode"]} v 0) v1 -- 0x7f2ac8080050 con 0x7f2ac815c480 2020-11-20T12:14:53.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.315+0000 7f2ac4ff9700 1 -- 172.21.15.187:0/4267869579 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_open_inode"]}]=0 v508) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2ab8046f70 con 0x7f2ac815c480 2020-11-20T12:14:53.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2aa4056d50 msgr2=0x7f2aa4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:53.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2aa4056d50 0x7f2aa4059210 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f2ac00090e0 tx=0x7f2ac0008040).stop 2020-11-20T12:14:53.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ac815c480 msgr2=0x7f2ac8140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:53.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ac815c480 0x7f2ac8140460 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f2ab800c9f0 tx=0x7f2ab8008040).stop 2020-11-20T12:14:53.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 shutdown_connections 2020-11-20T12:14:53.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2aa4056d50 0x7f2aa4059210 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ac815c480 0x7f2ac8140460 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ac815bbb0 0x7f2ac80b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 --2- 172.21.15.187:0/4267869579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8146700 0x7f2ac815bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 >> 172.21.15.187:0/4267869579 conn(0x7f2ac8068370 msgr2=0x7f2ac81416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:53.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 shutdown_connections 2020-11-20T12:14:53.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.318+0000 7f2acdf11700 1 -- 172.21.15.187:0/4267869579 wait complete. 2020-11-20T12:14:53.349 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:14:53.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.596+0000 7f0f75f2b700 1 Processor -- start 2020-11-20T12:14:53.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.597+0000 7f0f75f2b700 1 -- start start 2020-11-20T12:14:53.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.598+0000 7f0f75f2b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0f700812e0 0x7f0f700be7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:53.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.598+0000 7f0f75f2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70148910 0x7f0f70148d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:53.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.598+0000 7f0f75f2b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f700bee40 con 0x7f0f70080f00 2020-11-20T12:14:53.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.598+0000 7f0f75f2b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7014bee0 con 0x7f0f700812e0 2020-11-20T12:14:53.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.598+0000 7f0f75f2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7014c180 con 0x7f0f70148910 2020-11-20T12:14:53.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.599+0000 7f0f6f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0f70080f00 0x7f0f7014c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46038/0 (socket says 172.21.15.187:46038) 2020-11-20T12:14:53.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.599+0000 7f0f66ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0f700812e0 0x7f0f700be7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:53.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.599+0000 7f0f6f7fe700 1 -- 172.21.15.187:0/3705799591 learned_addr learned my addr 172.21.15.187:0/3705799591 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:53.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.599+0000 7f0f6ffff700 1 --2- 172.21.15.187:0/3705799591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70148910 0x7f0f70148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:53.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f6effd700 1 -- 172.21.15.187:0/3705799591 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217665375 0 0) 0x7f0f700bee40 con 0x7f0f70080f00 2020-11-20T12:14:53.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f6effd700 1 -- 172.21.15.187:0/3705799591 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f50003240 con 0x7f0f70080f00 2020-11-20T12:14:53.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f66ffd700 1 -- 172.21.15.187:0/3705799591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70148910 msgr2=0x7f0f70148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:53.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f66ffd700 1 --2- 172.21.15.187:0/3705799591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70148910 0x7f0f70148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f66ffd700 1 -- 172.21.15.187:0/3705799591 >> v1:172.21.15.187:6789/0 conn(0x7f0f70080f00 legacy=0x7f0f7014c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:53.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f66ffd700 1 -- 172.21.15.187:0/3705799591 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7014cd30 con 0x7f0f700812e0 2020-11-20T12:14:53.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f66ffd700 1 --2- 172.21.15.187:0/3705799591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f700812e0 0x7f0f700be7a0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7f0f5800b9a0 tx=0x7f0f58006df0).ready entity=mon.1 client_cookie=16c674baa319866c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:53.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f6effd700 1 -- 172.21.15.187:0/3705799591 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f58009680 con 0x7f0f700812e0 2020-11-20T12:14:53.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f6effd700 1 -- 172.21.15.187:0/3705799591 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f580097e0 con 0x7f0f700812e0 2020-11-20T12:14:53.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.600+0000 7f0f6effd700 1 -- 172.21.15.187:0/3705799591 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f58009b60 con 0x7f0f700812e0 2020-11-20T12:14:53.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.601+0000 7f0f75f2b700 1 -- 172.21.15.187:0/3705799591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f700812e0 msgr2=0x7f0f700be7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:53.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.601+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/3705799591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f700812e0 0x7f0f700be7a0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7f0f5800b9a0 tx=0x7f0f58006df0).stop 2020-11-20T12:14:53.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.601+0000 7f0f75f2b700 1 -- 172.21.15.187:0/3705799591 shutdown_connections 2020-11-20T12:14:53.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.601+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/3705799591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70148910 0x7f0f70148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.601+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/3705799591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f700812e0 0x7f0f700be7a0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.601+0000 7f0f75f2b700 1 -- 172.21.15.187:0/3705799591 >> 172.21.15.187:0/3705799591 conn(0x7f0f700bf100 msgr2=0x7f0f700bf510 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:53.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.601+0000 7f0f75f2b700 1 -- 172.21.15.187:0/3705799591 shutdown_connections 2020-11-20T12:14:53.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.601+0000 7f0f75f2b700 1 -- 172.21.15.187:0/3705799591 wait complete. 2020-11-20T12:14:53.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.603+0000 7f0f75f2b700 1 Processor -- start 2020-11-20T12:14:53.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.603+0000 7f0f75f2b700 1 -- start start 2020-11-20T12:14:53.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.603+0000 7f0f75f2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70080f00 0x7f0f700b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:53.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.603+0000 7f0f75f2b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f70148910 0x7f0f700a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:53.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.603+0000 7f0f75f2b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f70147e10 0x7f0f7015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:53.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.603+0000 7f0f75f2b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7015c270 con 0x7f0f70147e10 2020-11-20T12:14:53.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f75f2b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7015c510 con 0x7f0f70148910 2020-11-20T12:14:53.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f75f2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7015c7b0 con 0x7f0f70080f00 2020-11-20T12:14:53.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f70148910 0x7f0f700a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:53.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f70148910 0x7f0f700a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49440/0 (socket says 172.21.15.187:49440) 2020-11-20T12:14:53.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 -- 172.21.15.187:0/1366054747 learned_addr learned my addr 172.21.15.187:0/1366054747 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:53.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 -- 172.21.15.187:0/1366054747 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70080f00 msgr2=0x7f0f700b6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:53.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70080f00 0x7f0f700b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:53.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 -- 172.21.15.187:0/1366054747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f70147e10 msgr2=0x7f0f7015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:53.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f70147e10 0x7f0f7015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:53.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 -- 172.21.15.187:0/1366054747 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f700a07b0 con 0x7f0f70148910 2020-11-20T12:14:53.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.604+0000 7f0f66ffd700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f70148910 0x7f0f700a02c0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f0f58010000 tx=0x7f0f58015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:53.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.605+0000 7f0f67fff700 1 -- 172.21.15.187:0/1366054747 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f58011d50 con 0x7f0f70148910 2020-11-20T12:14:53.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.605+0000 7f0f67fff700 1 -- 172.21.15.187:0/1366054747 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5801e460 con 0x7f0f70148910 2020-11-20T12:14:53.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.605+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7015bf30 con 0x7f0f70148910 2020-11-20T12:14:53.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.605+0000 7f0f67fff700 1 -- 172.21.15.187:0/1366054747 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5801e7e0 con 0x7f0f70148910 2020-11-20T12:14:53.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.605+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f700b79f0 con 0x7f0f70148910 2020-11-20T12:14:53.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.606+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f70148780 con 0x7f0f70148910 2020-11-20T12:14:53.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.606+0000 7f0f67fff700 1 -- 172.21.15.187:0/1366054747 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0f58025030 con 0x7f0f70148910 2020-11-20T12:14:53.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.607+0000 7f0f67fff700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0f4c056d20 0x7f0f4c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:53.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.607+0000 7f0f67fff700 1 -- 172.21.15.187:0/1366054747 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0f58071570 con 0x7f0f70148910 2020-11-20T12:14:53.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.608+0000 7f0f6f7fe700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0f4c056d20 0x7f0f4c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:53.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.608+0000 7f0f6f7fe700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0f4c056d20 0x7f0f4c0591e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f0f600055a0 tx=0x7f0f600076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:53.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.611+0000 7f0f67fff700 1 -- 172.21.15.187:0/1366054747 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0f58046a70 con 0x7f0f70148910 2020-11-20T12:14:53.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.797+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f0f7015c0c0 con 0x7f0f70148910 2020-11-20T12:14:53.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.807+0000 7f0f67fff700 1 -- 172.21.15.187:0/1366054747 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0f58023020 con 0x7f0f70148910 2020-11-20T12:14:53.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0f4c056d20 msgr2=0x7f0f4c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:53.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0f4c056d20 0x7f0f4c0591e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f0f600055a0 tx=0x7f0f600076a0).stop 2020-11-20T12:14:53.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f70148910 msgr2=0x7f0f700a02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:53.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f70148910 0x7f0f700a02c0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f0f58010000 tx=0x7f0f58015000).stop 2020-11-20T12:14:53.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 shutdown_connections 2020-11-20T12:14:53.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0f4c056d20 0x7f0f4c0591e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f70147e10 0x7f0f7015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:53.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f70080f00 0x7f0f700b6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:53.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 --2- 172.21.15.187:0/1366054747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f70148910 0x7f0f700a02c0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:53.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 >> 172.21.15.187:0/1366054747 conn(0x7f0f700bf100 msgr2=0x7f0f700bcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:53.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.809+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 shutdown_connections 2020-11-20T12:14:53.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:53.810+0000 7f0f75f2b700 1 -- 172.21.15.187:0/1366054747 wait complete. 2020-11-20T12:14:53.833 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:14:53.834 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:14:53.952 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:14:53.972 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:14:54.082 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:14:54.098 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:14:54.099 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:14:54.100 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:54.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.337+0000 7ff0cad85700 1 Processor -- start 2020-11-20T12:14:54.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.337+0000 7ff0cad85700 1 -- start start 2020-11-20T12:14:54.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.338+0000 7ff0cad85700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0c412da00 0x7ff0c412de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.339+0000 7ff0cad85700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c4063140 con 0x7ff0c4067020 2020-11-20T12:14:54.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.339+0000 7ff0cad85700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40633e0 con 0x7ff0c4067400 2020-11-20T12:14:54.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.339+0000 7ff0cad85700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c4148230 con 0x7ff0c412da00 2020-11-20T12:14:54.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.339+0000 7ff0c9582700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff0c4067400 0x7ff0c4144c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46052/0 (socket says 172.21.15.187:46052) 2020-11-20T12:14:54.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.339+0000 7ff0c9582700 1 -- 172.21.15.187:0/3258743573 learned_addr learned my addr 172.21.15.187:0/3258743573 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:54.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.339+0000 7ff0ca584700 1 --2- 172.21.15.187:0/3258743573 >> v2:172.21.15.187:3300/0 conn(0x7ff0c412da00 0x7ff0c412de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.339+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128363081 0 0) 0x7ff0c40633e0 con 0x7ff0c4067400 2020-11-20T12:14:54.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.339+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ac003240 con 0x7ff0c4067400 2020-11-20T12:14:54.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1563008671 0 0) 0x7ff0ac003240 con 0x7ff0c4067400 2020-11-20T12:14:54.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 >> v2:172.21.15.187:3300/0 conn(0x7ff0c412da00 msgr2=0x7ff0c412de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:54.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 --2- 172.21.15.187:0/3258743573 >> v2:172.21.15.187:3300/0 conn(0x7ff0c412da00 0x7ff0c412de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 >> v1:172.21.15.5:6789/0 conn(0x7ff0c4067020 legacy=0x7ff0c4066440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:54.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4061590 con 0x7ff0c4067400 2020-11-20T12:14:54.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff0b40014e0 con 0x7ff0c4067400 2020-11-20T12:14:54.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 >> v1:172.21.15.187:6789/0 conn(0x7ff0c4067400 legacy=0x7ff0c4144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac003730 0x7ff0ac003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0ac0044e0 0x7ff0ac004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0ac004e50 0x7ff0ac00c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c4148230 con 0x7ff0ac003730 2020-11-20T12:14:54.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac004260 con 0x7ff0ac0044e0 2020-11-20T12:14:54.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.340+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac005700 con 0x7ff0ac004e50 2020-11-20T12:14:54.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0c9582700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0ac0044e0 0x7ff0ac004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0ca584700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0ac004e50 0x7ff0ac00c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0c9582700 1 -- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0ac004e50 msgr2=0x7ff0ac00c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0c9582700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0ac004e50 0x7ff0ac00c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0c9d83700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac003730 0x7ff0ac003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0c9582700 1 -- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac003730 msgr2=0x7ff0ac003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:54.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0c9582700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac003730 0x7ff0ac003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0c9582700 1 -- 172.21.15.187:0/3258743573 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac003240 con 0x7ff0ac0044e0 2020-11-20T12:14:54.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0ca584700 1 -- 172.21.15.187:0/3258743573 reap_dead start 2020-11-20T12:14:54.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.341+0000 7ff0c9d83700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac003730 0x7ff0ac003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:54.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0c9582700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0ac0044e0 0x7ff0ac004960 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7ff0b400c410 tx=0x7ff0b40077d0).ready entity=mon.1 client_cookie=5282810321bf2f19 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:54.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b400ce00 con 0x7ff0ac0044e0 2020-11-20T12:14:54.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b40072e0 con 0x7ff0ac0044e0 2020-11-20T12:14:54.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0c8d81700 1 -- 172.21.15.187:0/3258743573 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4005800 con 0x7ff0ac0044e0 2020-11-20T12:14:54.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0cad85700 1 -- 172.21.15.187:0/3258743573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0ac0044e0 msgr2=0x7ff0ac004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0cad85700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0ac0044e0 0x7ff0ac004960 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7ff0b400c410 tx=0x7ff0b40077d0).stop 2020-11-20T12:14:54.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0cad85700 1 -- 172.21.15.187:0/3258743573 shutdown_connections 2020-11-20T12:14:54.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0cad85700 1 --2- 172.21.15.187:0/3258743573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0ac0044e0 0x7ff0ac004960 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.342+0000 7ff0cad85700 1 -- 172.21.15.187:0/3258743573 >> 172.21.15.187:0/3258743573 conn(0x7ff0c40bb3d0 msgr2=0x7ff0c40bb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:54.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.343+0000 7ff0cad85700 1 -- 172.21.15.187:0/3258743573 shutdown_connections 2020-11-20T12:14:54.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.343+0000 7ff0cad85700 1 -- 172.21.15.187:0/3258743573 wait complete. 2020-11-20T12:14:54.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.344+0000 7ff0cad85700 1 Processor -- start 2020-11-20T12:14:54.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.344+0000 7ff0cad85700 1 -- start start 2020-11-20T12:14:54.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.345+0000 7ff0cad85700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac0044e0 0x7ff0c4136620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.345+0000 7ff0cad85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0c4136b10 0x7ff0c415ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.345+0000 7ff0cad85700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0c415cf60 0x7ff0c415fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.345+0000 7ff0cad85700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40b1290 con 0x7ff0ac0044e0 2020-11-20T12:14:54.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.345+0000 7ff0cad85700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c409f6e0 con 0x7ff0c4136b10 2020-11-20T12:14:54.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.345+0000 7ff0cad85700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40b3380 con 0x7ff0c415cf60 2020-11-20T12:14:54.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.345+0000 7ff0c9582700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0c4136b10 0x7ff0c415ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9582700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0c4136b10 0x7ff0c415ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49460/0 (socket says 172.21.15.187:49460) 2020-11-20T12:14:54.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9582700 1 -- 172.21.15.187:0/200959275 learned_addr learned my addr 172.21.15.187:0/200959275 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:54.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9d83700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac0044e0 0x7ff0c4136620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0ca584700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0c415cf60 0x7ff0c415fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9582700 1 -- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0c415cf60 msgr2=0x7ff0c415fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9582700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0c415cf60 0x7ff0c415fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9582700 1 -- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac0044e0 msgr2=0x7ff0c4136620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9582700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac0044e0 0x7ff0c4136620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9582700 1 -- 172.21.15.187:0/200959275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40b3620 con 0x7ff0c4136b10 2020-11-20T12:14:54.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0ca584700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0c415cf60 0x7ff0c415fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:54.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0c9582700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0c4136b10 0x7ff0c415ca70 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7ff0b40078e0 tx=0x7ff0b4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:54.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.346+0000 7ff0b1ffb700 1 -- 172.21.15.187:0/200959275 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b40055e0 con 0x7ff0c4136b10 2020-11-20T12:14:54.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.347+0000 7ff0b1ffb700 1 -- 172.21.15.187:0/200959275 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b4005740 con 0x7ff0c4136b10 2020-11-20T12:14:54.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.347+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c40b3000 con 0x7ff0c4136b10 2020-11-20T12:14:54.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.347+0000 7ff0b1ffb700 1 -- 172.21.15.187:0/200959275 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4013680 con 0x7ff0c4136b10 2020-11-20T12:14:54.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.347+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c415d3a0 con 0x7ff0c4136b10 2020-11-20T12:14:54.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.348+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c40b9c90 con 0x7ff0c4136b10 2020-11-20T12:14:54.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.348+0000 7ff0b1ffb700 1 -- 172.21.15.187:0/200959275 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff0b4026070 con 0x7ff0c4136b10 2020-11-20T12:14:54.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.349+0000 7ff0b1ffb700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff09c056d20 0x7ff09c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.349+0000 7ff0b1ffb700 1 -- 172.21.15.187:0/200959275 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff0b4070cd0 con 0x7ff0c4136b10 2020-11-20T12:14:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.349+0000 7ff0c9d83700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff09c056d20 0x7ff09c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.350+0000 7ff0c9d83700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff09c056d20 0x7ff09c0591e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7ff0c412d650 tx=0x7ff0c0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.353+0000 7ff0b1ffb700 1 -- 172.21.15.187:0/200959275 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff0b40460a0 con 0x7ff0c4136b10 2020-11-20T12:14:54.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.596+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0c40b3190 con 0x7ff0c4136b10 2020-11-20T12:14:54.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.597+0000 7ff0b1ffb700 1 -- 172.21.15.187:0/200959275 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7ff0b40496c0 con 0x7ff0c4136b10 2020-11-20T12:14:54.600 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:54.600 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:54.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff09c056d20 msgr2=0x7ff09c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff09c056d20 0x7ff09c0591e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7ff0c412d650 tx=0x7ff0c0009040).stop 2020-11-20T12:14:54.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0c4136b10 msgr2=0x7ff0c415ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0c4136b10 0x7ff0c415ca70 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7ff0b40078e0 tx=0x7ff0b4015000).stop 2020-11-20T12:14:54.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 shutdown_connections 2020-11-20T12:14:54.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff09c056d20 0x7ff09c0591e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac0044e0 0x7ff0c4136620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0c415cf60 0x7ff0c415fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 --2- 172.21.15.187:0/200959275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0c4136b10 0x7ff0c415ca70 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.599+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 >> 172.21.15.187:0/200959275 conn(0x7ff0c40bb3d0 msgr2=0x7ff0c41474d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:54.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.600+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 shutdown_connections 2020-11-20T12:14:54.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.600+0000 7ff0cad85700 1 -- 172.21.15.187:0/200959275 wait complete. 2020-11-20T12:14:54.605 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:54.625 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:54.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.850+0000 7f6df7059700 1 Processor -- start 2020-11-20T12:14:54.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.850+0000 7f6df7059700 1 -- start start 2020-11-20T12:14:54.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.851+0000 7f6df7059700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6df00b1760 0x7f6df0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.851+0000 7f6df7059700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df0065680 0x7f6df014c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.851+0000 7f6df7059700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df0150390 con 0x7f6df00b1380 2020-11-20T12:14:54.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.851+0000 7f6df7059700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df01511e0 con 0x7f6df00b1760 2020-11-20T12:14:54.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.851+0000 7f6df7059700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df014d760 con 0x7f6df0065680 2020-11-20T12:14:54.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.852+0000 7f6df6057700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6df00b1380 0x7f6df0150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:39582/0 (socket says 172.21.15.187:39582) 2020-11-20T12:14:54.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.852+0000 7f6df6057700 1 -- 172.21.15.187:0/2749787945 learned_addr learned my addr 172.21.15.187:0/2749787945 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:54.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.852+0000 7f6df5856700 1 --2- 172.21.15.187:0/2749787945 >> v2:172.21.15.187:3300/0 conn(0x7f6df00b1760 0x7f6df0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.853+0000 7f6df6858700 1 --2- 172.21.15.187:0/2749787945 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df0065680 0x7f6df014c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.853+0000 7f6df5055700 1 -- 172.21.15.187:0/2749787945 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561448861 0 0) 0x7f6df0150390 con 0x7f6df00b1380 2020-11-20T12:14:54.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.853+0000 7f6df5055700 1 -- 172.21.15.187:0/2749787945 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd8003240 con 0x7f6df00b1380 2020-11-20T12:14:54.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.853+0000 7f6df5856700 1 -- 172.21.15.187:0/2749787945 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df0065680 msgr2=0x7f6df014c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:54.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.853+0000 7f6df5856700 1 --2- 172.21.15.187:0/2749787945 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df0065680 0x7f6df014c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.853+0000 7f6df5856700 1 -- 172.21.15.187:0/2749787945 >> v1:172.21.15.5:6789/0 conn(0x7f6df00b1380 legacy=0x7f6df0150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.853+0000 7f6df5856700 1 -- 172.21.15.187:0/2749787945 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0150630 con 0x7f6df00b1760 2020-11-20T12:14:54.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df5856700 1 --2- 172.21.15.187:0/2749787945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0065190 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7f6de000b9d0 tx=0x7f6de0006df0).ready entity=mon.1 client_cookie=6d02f7bc07a28868 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:54.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df5055700 1 -- 172.21.15.187:0/2749787945 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de0009770 con 0x7f6df00b1760 2020-11-20T12:14:54.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df5055700 1 -- 172.21.15.187:0/2749787945 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de00098d0 con 0x7f6df00b1760 2020-11-20T12:14:54.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df5055700 1 -- 172.21.15.187:0/2749787945 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de0009c50 con 0x7f6df00b1760 2020-11-20T12:14:54.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df7059700 1 -- 172.21.15.187:0/2749787945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 msgr2=0x7f6df0065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df7059700 1 --2- 172.21.15.187:0/2749787945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0065190 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7f6de000b9d0 tx=0x7f6de0006df0).stop 2020-11-20T12:14:54.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df7059700 1 -- 172.21.15.187:0/2749787945 shutdown_connections 2020-11-20T12:14:54.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df7059700 1 --2- 172.21.15.187:0/2749787945 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df0065680 0x7f6df014c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.854+0000 7f6df7059700 1 --2- 172.21.15.187:0/2749787945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0065190 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.855+0000 7f6df7059700 1 -- 172.21.15.187:0/2749787945 >> 172.21.15.187:0/2749787945 conn(0x7f6df00bab30 msgr2=0x7f6df00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:54.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.855+0000 7f6df7059700 1 -- 172.21.15.187:0/2749787945 shutdown_connections 2020-11-20T12:14:54.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.855+0000 7f6df7059700 1 -- 172.21.15.187:0/2749787945 wait complete. 2020-11-20T12:14:54.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.856+0000 7f6df7059700 1 Processor -- start 2020-11-20T12:14:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.856+0000 7f6df7059700 1 -- start start 2020-11-20T12:14:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df7059700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df00b1380 0x7f6df00b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df7059700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df7059700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6df00b7710 0x7f6df014a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df7059700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00b9b00 con 0x7f6df00b7710 2020-11-20T12:14:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df7059700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00b9da0 con 0x7f6df00b1760 2020-11-20T12:14:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df7059700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00ba040 con 0x7f6df00b1380 2020-11-20T12:14:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df5856700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df5856700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0149b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49474/0 (socket says 172.21.15.187:49474) 2020-11-20T12:14:54.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.857+0000 7f6df5856700 1 -- 172.21.15.187:0/3995406337 learned_addr learned my addr 172.21.15.187:0/3995406337 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:54.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df5856700 1 -- 172.21.15.187:0/3995406337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df00b1380 msgr2=0x7f6df00b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df6057700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df00b1380 0x7f6df00b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df5856700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df00b1380 0x7f6df00b40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:54.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df5856700 1 -- 172.21.15.187:0/3995406337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6df00b7710 msgr2=0x7f6df014a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:54.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df5856700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6df00b7710 0x7f6df014a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:54.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df5856700 1 -- 172.21.15.187:0/3995406337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df00b9810 con 0x7f6df00b1760 2020-11-20T12:14:54.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df5856700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0149b20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f6de0002fc0 tx=0x7f6de0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:54.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6dde7fc700 1 -- 172.21.15.187:0/3995406337 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de0011d50 con 0x7f6df00b1760 2020-11-20T12:14:54.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6dde7fc700 1 -- 172.21.15.187:0/3995406337 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de001e460 con 0x7f6df00b1760 2020-11-20T12:14:54.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df00b4590 con 0x7f6df00b1760 2020-11-20T12:14:54.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6dde7fc700 1 -- 172.21.15.187:0/3995406337 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de001e7e0 con 0x7f6df00b1760 2020-11-20T12:14:54.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.858+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df00b8b00 con 0x7f6df00b1760 2020-11-20T12:14:54.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.859+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df0061a20 con 0x7f6df00b1760 2020-11-20T12:14:54.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.860+0000 7f6dde7fc700 1 -- 172.21.15.187:0/3995406337 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6de0025030 con 0x7f6df00b1760 2020-11-20T12:14:54.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.860+0000 7f6dde7fc700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dc8056d20 0x7f6dc80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:54.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.861+0000 7f6dde7fc700 1 -- 172.21.15.187:0/3995406337 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6de00719d0 con 0x7f6df00b1760 2020-11-20T12:14:54.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.861+0000 7f6df6057700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dc8056d20 0x7f6dc80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:54.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.861+0000 7f6df6057700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dc8056d20 0x7f6dc80591e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f6dec0055a0 tx=0x7f6dec009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:54.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:54.865+0000 7f6dde7fc700 1 -- 172.21.15.187:0/3995406337 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6de0046d10 con 0x7f6df00b1760 2020-11-20T12:14:55.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.051+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6df0068510 con 0x7f6df00b1760 2020-11-20T12:14:55.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.054+0000 7f6dde7fc700 1 -- 172.21.15.187:0/3995406337 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f6de0023070 con 0x7f6df00b1760 2020-11-20T12:14:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:55.057 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":98,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:14:30.229422+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:14:27.347251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2020-11-20T12:14:27.963950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:55.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dc8056d20 msgr2=0x7f6dc80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dc8056d20 0x7f6dc80591e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f6dec0055a0 tx=0x7f6dec009040).stop 2020-11-20T12:14:55.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 msgr2=0x7f6df0149b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0149b20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f6de0002fc0 tx=0x7f6de0015000).stop 2020-11-20T12:14:55.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 shutdown_connections 2020-11-20T12:14:55.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6dc8056d20 0x7f6dc80591e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6df00b7710 0x7f6df014a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:55.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6df00b1380 0x7f6df00b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 --2- 172.21.15.187:0/3995406337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6df00b1760 0x7f6df0149b20 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 >> 172.21.15.187:0/3995406337 conn(0x7f6df00bab30 msgr2=0x7f6df00baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:55.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 shutdown_connections 2020-11-20T12:14:55.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.056+0000 7f6df7059700 1 -- 172.21.15.187:0/3995406337 wait complete. 2020-11-20T12:14:55.084 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:14:27.347251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:14:27.963950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:14:55.085 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:14:55.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.321+0000 7f20aa57e700 1 Processor -- start 2020-11-20T12:14:55.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.321+0000 7f20aa57e700 1 -- start start 2020-11-20T12:14:55.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.323+0000 7f20aa57e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20a4065f90 0x7f20a4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.323+0000 7f20aa57e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20a409ca90 0x7f20a409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.323+0000 7f20aa57e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a40b13e0 con 0x7f20a4065190 2020-11-20T12:14:55.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.323+0000 7f20aa57e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a414ab30 con 0x7f20a4065f90 2020-11-20T12:14:55.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.323+0000 7f20aa57e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a4062c10 con 0x7f20a409ca90 2020-11-20T12:14:55.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.323+0000 7f20a9d7d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20a409ca90 0x7f20a409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.323+0000 7f20a9d7d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20a409ca90 0x7f20a409cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49484/0 (socket says 172.21.15.187:49484) 2020-11-20T12:14:55.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.323+0000 7f20a9d7d700 1 -- 172.21.15.187:0/2539914944 learned_addr learned my addr 172.21.15.187:0/2539914944 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:55.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.324+0000 7f20a9d7d700 1 -- 172.21.15.187:0/2539914944 >> v2:172.21.15.5:3300/0 conn(0x7f20a4065f90 msgr2=0x7f20a4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:55.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.324+0000 7f20a9d7d700 1 --2- 172.21.15.187:0/2539914944 >> v2:172.21.15.5:3300/0 conn(0x7f20a4065f90 0x7f20a4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:55.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.324+0000 7f20a9d7d700 1 -- 172.21.15.187:0/2539914944 >> v1:172.21.15.5:6789/0 conn(0x7f20a4065190 legacy=0x7f20a4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:55.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.324+0000 7f20a9d7d700 1 -- 172.21.15.187:0/2539914944 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a40b1640 con 0x7f20a409ca90 2020-11-20T12:14:55.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.324+0000 7f20a9d7d700 1 --2- 172.21.15.187:0/2539914944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a409cf20 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f20a4099f90 tx=0x7f209c009290).ready entity=mon.1 client_cookie=7650847e6714c9ac server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:55.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.325+0000 7f2093fff700 1 -- 172.21.15.187:0/2539914944 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c00bc10 con 0x7f20a409ca90 2020-11-20T12:14:55.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.328+0000 7f2093fff700 1 -- 172.21.15.187:0/2539914944 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c00bd70 con 0x7f20a409ca90 2020-11-20T12:14:55.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.328+0000 7f2093fff700 1 -- 172.21.15.187:0/2539914944 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c011610 con 0x7f20a409ca90 2020-11-20T12:14:55.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.329+0000 7f20aa57e700 1 -- 172.21.15.187:0/2539914944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 msgr2=0x7f20a409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.329+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2539914944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a409cf20 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f20a4099f90 tx=0x7f209c009290).stop 2020-11-20T12:14:55.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.329+0000 7f20aa57e700 1 -- 172.21.15.187:0/2539914944 shutdown_connections 2020-11-20T12:14:55.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.329+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2539914944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a409cf20 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.329+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2539914944 >> v2:172.21.15.5:3300/0 conn(0x7f20a4065f90 0x7f20a4065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:55.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.329+0000 7f20aa57e700 1 -- 172.21.15.187:0/2539914944 >> 172.21.15.187:0/2539914944 conn(0x7f20a40bab30 msgr2=0x7f20a40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:55.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.329+0000 7f20aa57e700 1 -- 172.21.15.187:0/2539914944 shutdown_connections 2020-11-20T12:14:55.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.329+0000 7f20aa57e700 1 -- 172.21.15.187:0/2539914944 wait complete. 2020-11-20T12:14:55.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.331+0000 7f20aa57e700 1 Processor -- start 2020-11-20T12:14:55.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.331+0000 7f20aa57e700 1 -- start start 2020-11-20T12:14:55.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.332+0000 7f20aa57e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a4065190 0x7f20a409ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20aa57e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a4065f90 0x7f20a4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20aa57e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a4099680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20aa57e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a40b2ef0 con 0x7f20a4065190 2020-11-20T12:14:55.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20aa57e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4067870 con 0x7f20a409ca90 2020-11-20T12:14:55.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20aa57e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4067b10 con 0x7f20a4065f90 2020-11-20T12:14:55.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a9d7d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a4099680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a9d7d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a4099680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49490/0 (socket says 172.21.15.187:49490) 2020-11-20T12:14:55.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a9d7d700 1 -- 172.21.15.187:0/2786233839 learned_addr learned my addr 172.21.15.187:0/2786233839 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:55.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a957c700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a4065190 0x7f20a409ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a8d7b700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a4065f90 0x7f20a4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a9d7d700 1 -- 172.21.15.187:0/2786233839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a4065f90 msgr2=0x7f20a4099190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a9d7d700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a4065f90 0x7f20a4099190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a9d7d700 1 -- 172.21.15.187:0/2786233839 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a4065190 msgr2=0x7f20a409ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a9d7d700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a4065190 0x7f20a409ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.333+0000 7f20a9d7d700 1 -- 172.21.15.187:0/2786233839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a40a04a0 con 0x7f20a409ca90 2020-11-20T12:14:55.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.334+0000 7f20a9d7d700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a4099680 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f209c0106d0 tx=0x7f209c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:55.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.334+0000 7f20917fa700 1 -- 172.21.15.187:0/2786233839 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c00bc10 con 0x7f20a409ca90 2020-11-20T12:14:55.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.334+0000 7f20917fa700 1 -- 172.21.15.187:0/2786233839 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c010bf0 con 0x7f20a409ca90 2020-11-20T12:14:55.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.334+0000 7f20917fa700 1 -- 172.21.15.187:0/2786233839 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c00bee0 con 0x7f20a409ca90 2020-11-20T12:14:55.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.334+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a40674b0 con 0x7f20a409ca90 2020-11-20T12:14:55.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.334+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a40a0a10 con 0x7f20a409ca90 2020-11-20T12:14:55.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.335+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a4067d10 con 0x7f20a409ca90 2020-11-20T12:14:55.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.336+0000 7f20917fa700 1 -- 172.21.15.187:0/2786233839 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f209c017070 con 0x7f20a409ca90 2020-11-20T12:14:55.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.336+0000 7f20917fa700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f207c056d20 0x7f207c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.336+0000 7f20917fa700 1 -- 172.21.15.187:0/2786233839 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f209c0721e0 con 0x7f20a409ca90 2020-11-20T12:14:55.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.337+0000 7f20a957c700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f207c056d20 0x7f207c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.337+0000 7f20a957c700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f207c056d20 0x7f207c0591e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f20a0002410 tx=0x7f20a0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:55.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.340+0000 7f20917fa700 1 -- 172.21.15.187:0/2786233839 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f209c047630 con 0x7f20a409ca90 2020-11-20T12:14:55.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.527+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f20a4061a20 con 0x7f20a409ca90 2020-11-20T12:14:55.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.528+0000 7f20917fa700 1 -- 172.21.15.187:0/2786233839 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v98) v1 ==== 110+0+10 (secure 0 0 0) 0x7f209c0150b0 con 0x7f20a409ca90 2020-11-20T12:14:55.531 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:14:55.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f207c056d20 msgr2=0x7f207c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f207c056d20 0x7f207c0591e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f20a0002410 tx=0x7f20a0009040).stop 2020-11-20T12:14:55.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 msgr2=0x7f20a4099680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a4099680 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f209c0106d0 tx=0x7f209c016000).stop 2020-11-20T12:14:55.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 shutdown_connections 2020-11-20T12:14:55.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f207c056d20 0x7f207c0591e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a4065190 0x7f20a409ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a409ca90 0x7f20a4099680 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 --2- 172.21.15.187:0/2786233839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a4065f90 0x7f20a4099190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 >> 172.21.15.187:0/2786233839 conn(0x7f20a40bab30 msgr2=0x7f20a4143070 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:55.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.530+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 shutdown_connections 2020-11-20T12:14:55.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.531+0000 7f20aa57e700 1 -- 172.21.15.187:0/2786233839 wait complete. 2020-11-20T12:14:55.556 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:14:55.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.800+0000 7f703ffff700 1 Processor -- start 2020-11-20T12:14:55.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.800+0000 7f703ffff700 1 -- start start 2020-11-20T12:14:55.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.801+0000 7f703ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7040065190 0x7f7040065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.801+0000 7f703ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70400b13e0 con 0x7f704009ca90 2020-11-20T12:14:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.801+0000 7f703ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7040062c10 con 0x7f7040065f90 2020-11-20T12:14:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.801+0000 7f703ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704014ab30 con 0x7f7040065190 2020-11-20T12:14:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.802+0000 7f70367fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7040065f90 0x7f7040143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46102/0 (socket says 172.21.15.187:46102) 2020-11-20T12:14:55.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.802+0000 7f70367fc700 1 -- 172.21.15.187:0/2897931537 learned_addr learned my addr 172.21.15.187:0/2897931537 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:55.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.802+0000 7f703effd700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7040065190 0x7f7040065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.802+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050097066 0 0) 0x7f7040062c10 con 0x7f7040065f90 2020-11-20T12:14:55.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7020003240 con 0x7f7040065f90 2020-11-20T12:14:55.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1855384088 0 0) 0x7f7020003240 con 0x7f7040065f90 2020-11-20T12:14:55.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7040065190 msgr2=0x7f7040065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:55.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7040065190 0x7f7040065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 >> v1:172.21.15.5:6789/0 conn(0x7f704009ca90 legacy=0x7f704009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:55.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70400b1640 con 0x7f7040065f90 2020-11-20T12:14:55.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f70280014e0 con 0x7f7040065f90 2020-11-20T12:14:55.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 >> v1:172.21.15.187:6789/0 conn(0x7f7040065f90 legacy=0x7f7040143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70200037b0 0x7f7020003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7040065190 0x7f70200044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200049d0 0x7f7020004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704014ab30 con 0x7f70200049d0 2020-11-20T12:14:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020004280 con 0x7f70200037b0 2020-11-20T12:14:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.803+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702000fb40 con 0x7f7040065190 2020-11-20T12:14:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f703effd700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70200037b0 0x7f7020003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f70367fc700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7040065190 0x7f70200044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f703effd700 1 -- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7040065190 msgr2=0x7f70200044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f703effd700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7040065190 0x7f70200044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f703effd700 1 -- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200049d0 msgr2=0x7f7020004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f703f7fe700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200049d0 0x7f7020004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f703effd700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200049d0 0x7f7020004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f703effd700 1 -- 172.21.15.187:0/2897931537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020003240 con 0x7f70200037b0 2020-11-20T12:14:55.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.804+0000 7f703f7fe700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200049d0 0x7f7020004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:55.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703effd700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70200037b0 0x7f7020003bb0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f703000bc00 tx=0x7f7030006df0).ready entity=mon.1 client_cookie=409c814d43691021 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:55.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703000b310 con 0x7f70200037b0 2020-11-20T12:14:55.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000eb40 con 0x7f70200037b0 2020-11-20T12:14:55.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703e7fc700 1 -- 172.21.15.187:0/2897931537 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703000eec0 con 0x7f70200037b0 2020-11-20T12:14:55.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703ffff700 1 -- 172.21.15.187:0/2897931537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70200037b0 msgr2=0x7f7020003bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703ffff700 1 --2- 172.21.15.187:0/2897931537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70200037b0 0x7f7020003bb0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f703000bc00 tx=0x7f7030006df0).stop 2020-11-20T12:14:55.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703f7fe700 1 -- 172.21.15.187:0/2897931537 reap_dead start 2020-11-20T12:14:55.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703ffff700 1 -- 172.21.15.187:0/2897931537 shutdown_connections 2020-11-20T12:14:55.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.805+0000 7f703ffff700 1 -- 172.21.15.187:0/2897931537 >> 172.21.15.187:0/2897931537 conn(0x7f70400bab30 msgr2=0x7f70400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:55.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.806+0000 7f703ffff700 1 -- 172.21.15.187:0/2897931537 shutdown_connections 2020-11-20T12:14:55.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.806+0000 7f703ffff700 1 -- 172.21.15.187:0/2897931537 wait complete. 2020-11-20T12:14:55.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.807+0000 7f703ffff700 1 Processor -- start 2020-11-20T12:14:55.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.807+0000 7f703ffff700 1 -- start start 2020-11-20T12:14:55.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.808+0000 7f703ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200037b0 0x7f7040143250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.808+0000 7f703ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7040146630 0x7f7040146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.808+0000 7f703ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f704009e890 0x7f7040144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.808+0000 7f703ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70400b37b0 con 0x7f70200037b0 2020-11-20T12:14:55.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.808+0000 7f703ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70400b39e0 con 0x7f7040146630 2020-11-20T12:14:55.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.808+0000 7f703ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70400b3c80 con 0x7f704009e890 2020-11-20T12:14:55.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f70367fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7040146630 0x7f7040146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f70367fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7040146630 0x7f7040146cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49508/0 (socket says 172.21.15.187:49508) 2020-11-20T12:14:55.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f70367fc700 1 -- 172.21.15.187:0/912241607 learned_addr learned my addr 172.21.15.187:0/912241607 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:55.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f703f7fe700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f704009e890 0x7f7040144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f70367fc700 1 -- 172.21.15.187:0/912241607 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f704009e890 msgr2=0x7f7040144f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f703effd700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200037b0 0x7f7040143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f70367fc700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f704009e890 0x7f7040144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f70367fc700 1 -- 172.21.15.187:0/912241607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200037b0 msgr2=0x7f7040143250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:55.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f70367fc700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200037b0 0x7f7040143250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:55.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.809+0000 7f70367fc700 1 -- 172.21.15.187:0/912241607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040143740 con 0x7f7040146630 2020-11-20T12:14:55.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.810+0000 7f70367fc700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7040146630 0x7f7040146cf0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f7028000f90 tx=0x7f7028009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:55.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.810+0000 7f70377fe700 1 -- 172.21.15.187:0/912241607 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7028007170 con 0x7f7040146630 2020-11-20T12:14:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.810+0000 7f70377fe700 1 -- 172.21.15.187:0/912241607 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70280072d0 con 0x7f7040146630 2020-11-20T12:14:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.810+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70401471e0 con 0x7f7040146630 2020-11-20T12:14:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.810+0000 7f70377fe700 1 -- 172.21.15.187:0/912241607 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702800ba50 con 0x7f7040146630 2020-11-20T12:14:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.810+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7040145460 con 0x7f7040146630 2020-11-20T12:14:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.811+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7040145740 con 0x7f7040146630 2020-11-20T12:14:55.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.812+0000 7f70377fe700 1 -- 172.21.15.187:0/912241607 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f70280045c0 con 0x7f7040146630 2020-11-20T12:14:55.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.813+0000 7f70377fe700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f701c056d20 0x7f701c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:55.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.813+0000 7f70377fe700 1 -- 172.21.15.187:0/912241607 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7028070110 con 0x7f7040146630 2020-11-20T12:14:55.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.813+0000 7f703effd700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f701c056d20 0x7f701c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:55.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.814+0000 7f703effd700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f701c056d20 0x7f701c0591e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f703000e630 tx=0x7f7030019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:55.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:55.816+0000 7f70377fe700 1 -- 172.21.15.187:0/912241607 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7028045560 con 0x7f7040146630 2020-11-20T12:14:56.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.000+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f70400baf40 con 0x7f7040146630 2020-11-20T12:14:56.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.001+0000 7f70377fe700 1 -- 172.21.15.187:0/912241607 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v98) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7028048b80 con 0x7f7040146630 2020-11-20T12:14:56.004 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:56.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f701c056d20 msgr2=0x7f701c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f701c056d20 0x7f701c0591e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f703000e630 tx=0x7f7030019000).stop 2020-11-20T12:14:56.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7040146630 msgr2=0x7f7040146cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7040146630 0x7f7040146cf0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f7028000f90 tx=0x7f7028009040).stop 2020-11-20T12:14:56.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 shutdown_connections 2020-11-20T12:14:56.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f701c056d20 0x7f701c0591e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70200037b0 0x7f7040143250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f704009e890 0x7f7040144f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 --2- 172.21.15.187:0/912241607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7040146630 0x7f7040146cf0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.003+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 >> 172.21.15.187:0/912241607 conn(0x7f70400bab30 msgr2=0x7f704009c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:56.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.004+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 shutdown_connections 2020-11-20T12:14:56.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.004+0000 7f703ffff700 1 -- 172.21.15.187:0/912241607 wait complete. 2020-11-20T12:14:56.027 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:14:56.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.272+0000 7fe1a727c700 1 Processor -- start 2020-11-20T12:14:56.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.272+0000 7fe1a727c700 1 -- start start 2020-11-20T12:14:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.273+0000 7fe1a727c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1a00b57b0 0x7fe1a0068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.273+0000 7fe1a727c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 0x7fe1a0062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a727c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a0144b80 con 0x7fe1a0062b50 2020-11-20T12:14:56.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a727c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a01489e0 con 0x7fe1a00b57b0 2020-11-20T12:14:56.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a727c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a0148c80 con 0x7fe1a01482d0 2020-11-20T12:14:56.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a6a7b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe1a0062b50 0x7fe1a0065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46124/0 (socket says 172.21.15.187:46124) 2020-11-20T12:14:56.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a6a7b700 1 -- 172.21.15.187:0/1613639395 learned_addr learned my addr 172.21.15.187:0/1613639395 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:56.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a627a700 1 --2- 172.21.15.187:0/1613639395 >> v2:172.21.15.187:3300/0 conn(0x7fe1a00b57b0 0x7fe1a0068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a5a79700 1 --2- 172.21.15.187:0/1613639395 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 0x7fe1a0062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a5278700 1 -- 172.21.15.187:0/1613639395 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719882622 0 0) 0x7fe1a0144b80 con 0x7fe1a0062b50 2020-11-20T12:14:56.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a5278700 1 -- 172.21.15.187:0/1613639395 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe188003240 con 0x7fe1a0062b50 2020-11-20T12:14:56.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a627a700 1 -- 172.21.15.187:0/1613639395 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 msgr2=0x7fe1a0062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:56.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a627a700 1 --2- 172.21.15.187:0/1613639395 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 0x7fe1a0062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a627a700 1 -- 172.21.15.187:0/1613639395 >> v1:172.21.15.187:6789/0 conn(0x7fe1a0062b50 legacy=0x7fe1a0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.274+0000 7fe1a627a700 1 -- 172.21.15.187:0/1613639395 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a0148f20 con 0x7fe1a00b57b0 2020-11-20T12:14:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.275+0000 7fe1a627a700 1 --2- 172.21.15.187:0/1613639395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00b57b0 0x7fe1a0068230 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fe19c00b9a0 tx=0x7fe19c006df0).ready entity=mon.1 client_cookie=79aa3429c3e6572f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.275+0000 7fe1a5278700 1 -- 172.21.15.187:0/1613639395 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c0097b0 con 0x7fe1a00b57b0 2020-11-20T12:14:56.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.275+0000 7fe1a5278700 1 -- 172.21.15.187:0/1613639395 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19c009910 con 0x7fe1a00b57b0 2020-11-20T12:14:56.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.275+0000 7fe1a5278700 1 -- 172.21.15.187:0/1613639395 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c009c90 con 0x7fe1a00b57b0 2020-11-20T12:14:56.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.275+0000 7fe1a727c700 1 -- 172.21.15.187:0/1613639395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00b57b0 msgr2=0x7fe1a0068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.275+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1613639395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00b57b0 0x7fe1a0068230 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fe19c00b9a0 tx=0x7fe19c006df0).stop 2020-11-20T12:14:56.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.276+0000 7fe1a727c700 1 -- 172.21.15.187:0/1613639395 shutdown_connections 2020-11-20T12:14:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.276+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1613639395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00b57b0 0x7fe1a0068230 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.276+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1613639395 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 0x7fe1a0062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.276+0000 7fe1a727c700 1 -- 172.21.15.187:0/1613639395 >> 172.21.15.187:0/1613639395 conn(0x7fe1a00bab30 msgr2=0x7fe1a00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.276+0000 7fe1a727c700 1 -- 172.21.15.187:0/1613639395 shutdown_connections 2020-11-20T12:14:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.276+0000 7fe1a727c700 1 -- 172.21.15.187:0/1613639395 wait complete. 2020-11-20T12:14:56.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.277+0000 7fe1a727c700 1 Processor -- start 2020-11-20T12:14:56.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.277+0000 7fe1a727c700 1 -- start start 2020-11-20T12:14:56.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.278+0000 7fe1a727c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a0062b50 0x7fe1a009e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.278+0000 7fe1a727c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1a00b57b0 0x7fe1a009cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.278+0000 7fe1a727c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 0x7fe1a009d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.278+0000 7fe1a727c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a009eb10 con 0x7fe1a00b57b0 2020-11-20T12:14:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.278+0000 7fe1a727c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a013b9e0 con 0x7fe1a0062b50 2020-11-20T12:14:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.278+0000 7fe1a727c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a013b710 con 0x7fe1a01482d0 2020-11-20T12:14:56.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a0062b50 0x7fe1a009e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a0062b50 0x7fe1a009e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49520/0 (socket says 172.21.15.187:49520) 2020-11-20T12:14:56.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 -- 172.21.15.187:0/1418560987 learned_addr learned my addr 172.21.15.187:0/1418560987 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:56.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 -- 172.21.15.187:0/1418560987 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 msgr2=0x7fe1a009d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 0x7fe1a009d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 -- 172.21.15.187:0/1418560987 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1a00b57b0 msgr2=0x7fe1a009cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1a00b57b0 0x7fe1a009cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 -- 172.21.15.187:0/1418560987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a013be80 con 0x7fe1a0062b50 2020-11-20T12:14:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1a627a700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a0062b50 0x7fe1a009e4e0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7fe19c006e30 tx=0x7fe19c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:56.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.279+0000 7fe1927fc700 1 -- 172.21.15.187:0/1418560987 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c01e460 con 0x7fe1a0062b50 2020-11-20T12:14:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.280+0000 7fe1927fc700 1 -- 172.21.15.187:0/1418560987 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19c01e5c0 con 0x7fe1a0062b50 2020-11-20T12:14:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.280+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a013c070 con 0x7fe1a0062b50 2020-11-20T12:14:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.280+0000 7fe1927fc700 1 -- 172.21.15.187:0/1418560987 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c01e940 con 0x7fe1a0062b50 2020-11-20T12:14:56.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.280+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a015ba70 con 0x7fe1a0062b50 2020-11-20T12:14:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.281+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a013bba0 con 0x7fe1a0062b50 2020-11-20T12:14:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.281+0000 7fe1927fc700 1 -- 172.21.15.187:0/1418560987 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe19c025030 con 0x7fe1a0062b50 2020-11-20T12:14:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.282+0000 7fe1927fc700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe178056d20 0x7fe1780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.282+0000 7fe1927fc700 1 -- 172.21.15.187:0/1418560987 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe19c071860 con 0x7fe1a0062b50 2020-11-20T12:14:56.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.282+0000 7fe1a5a79700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe178056d20 0x7fe1780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.283+0000 7fe1a5a79700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe178056d20 0x7fe1780591e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fe1a009e470 tx=0x7fe194008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:56.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.286+0000 7fe1927fc700 1 -- 172.21.15.187:0/1418560987 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe19c046c50 con 0x7fe1a0062b50 2020-11-20T12:14:56.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.466+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe1a0061a20 con 0x7fe1a0062b50 2020-11-20T12:14:56.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.467+0000 7fe1927fc700 1 -- 172.21.15.187:0/1418560987 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe19c023020 con 0x7fe1a0062b50 2020-11-20T12:14:56.470 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:56.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe178056d20 msgr2=0x7fe1780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe178056d20 0x7fe1780591e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fe1a009e470 tx=0x7fe194008040).stop 2020-11-20T12:14:56.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a0062b50 msgr2=0x7fe1a009e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a0062b50 0x7fe1a009e4e0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7fe19c006e30 tx=0x7fe19c015000).stop 2020-11-20T12:14:56.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 shutdown_connections 2020-11-20T12:14:56.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe178056d20 0x7fe1780591e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1a01482d0 0x7fe1a009d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:56.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a0062b50 0x7fe1a009e4e0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 --2- 172.21.15.187:0/1418560987 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1a00b57b0 0x7fe1a009cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:14:56.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 >> 172.21.15.187:0/1418560987 conn(0x7fe1a00bab30 msgr2=0x7fe1a00b9710 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:56.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 shutdown_connections 2020-11-20T12:14:56.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.469+0000 7fe1a727c700 1 -- 172.21.15.187:0/1418560987 wait complete. 2020-11-20T12:14:56.498 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:14:56.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.737+0000 7f4525fa8700 1 Processor -- start 2020-11-20T12:14:56.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.737+0000 7f4525fa8700 1 -- start start 2020-11-20T12:14:56.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.738+0000 7f4525fa8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f4520067e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.738+0000 7f4525fa8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f45200bda00 0x7f4520149270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f4525fa8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4520149760 con 0x7f4520061330 2020-11-20T12:14:56.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f4525fa8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4520067450 con 0x7f45200bda00 2020-11-20T12:14:56.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f4525fa8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45200676f0 con 0x7f45200bd620 2020-11-20T12:14:56.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f451ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f45200bda00 0x7f4520149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f451ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f45200bda00 0x7f4520149270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49528/0 (socket says 172.21.15.187:49528) 2020-11-20T12:14:56.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f451ffff700 1 -- 172.21.15.187:0/3431654459 learned_addr learned my addr 172.21.15.187:0/3431654459 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:56.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f4524fa6700 1 --2- 172.21.15.187:0/3431654459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f4520067e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f451ffff700 1 -- 172.21.15.187:0/3431654459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 msgr2=0x7f4520067e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f451ffff700 1 --2- 172.21.15.187:0/3431654459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f4520067e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.739+0000 7f451ffff700 1 -- 172.21.15.187:0/3431654459 >> v1:172.21.15.5:6789/0 conn(0x7f4520061330 legacy=0x7f4520145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.740+0000 7f451ffff700 1 -- 172.21.15.187:0/3431654459 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520067990 con 0x7f45200bda00 2020-11-20T12:14:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.740+0000 7f4524fa6700 1 --2- 172.21.15.187:0/3431654459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f4520067e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:56.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.740+0000 7f451ffff700 1 --2- 172.21.15.187:0/3431654459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45200bda00 0x7f4520149270 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f450c006fd0 tx=0x7f450c006eb0).ready entity=mon.1 client_cookie=7c8765ff48570256 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:56.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.740+0000 7f451f7fe700 1 -- 172.21.15.187:0/3431654459 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c009920 con 0x7f45200bda00 2020-11-20T12:14:56.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.740+0000 7f451f7fe700 1 -- 172.21.15.187:0/3431654459 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c009a80 con 0x7f45200bda00 2020-11-20T12:14:56.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.740+0000 7f451f7fe700 1 -- 172.21.15.187:0/3431654459 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c009e00 con 0x7f45200bda00 2020-11-20T12:14:56.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.741+0000 7f4525fa8700 1 -- 172.21.15.187:0/3431654459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45200bda00 msgr2=0x7f4520149270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.741+0000 7f4525fa8700 1 --2- 172.21.15.187:0/3431654459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45200bda00 0x7f4520149270 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f450c006fd0 tx=0x7f450c006eb0).stop 2020-11-20T12:14:56.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.741+0000 7f4525fa8700 1 -- 172.21.15.187:0/3431654459 shutdown_connections 2020-11-20T12:14:56.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.741+0000 7f4525fa8700 1 --2- 172.21.15.187:0/3431654459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f4520067e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.741+0000 7f4525fa8700 1 --2- 172.21.15.187:0/3431654459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45200bda00 0x7f4520149270 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.741+0000 7f4525fa8700 1 -- 172.21.15.187:0/3431654459 >> 172.21.15.187:0/3431654459 conn(0x7f4520060c10 msgr2=0x7f45200bcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:56.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.741+0000 7f4525fa8700 1 -- 172.21.15.187:0/3431654459 shutdown_connections 2020-11-20T12:14:56.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.741+0000 7f4525fa8700 1 -- 172.21.15.187:0/3431654459 wait complete. 2020-11-20T12:14:56.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.742+0000 7f4525fa8700 1 Processor -- start 2020-11-20T12:14:56.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.743+0000 7f4525fa8700 1 -- start start 2020-11-20T12:14:56.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.743+0000 7f4525fa8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4520061330 0x7f452015bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.743+0000 7f4525fa8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f452012cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.743+0000 7f4525fa8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452012d390 0x7f45200b8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.743+0000 7f4525fa8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45200b6930 con 0x7f4520061330 2020-11-20T12:14:56.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.743+0000 7f4525fa8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452015c480 con 0x7f452012d390 2020-11-20T12:14:56.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.743+0000 7f4525fa8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45200b6b60 con 0x7f45200bd620 2020-11-20T12:14:56.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.744+0000 7f45257a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452012d390 0x7f45200b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.744+0000 7f45257a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452012d390 0x7f45200b8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49538/0 (socket says 172.21.15.187:49538) 2020-11-20T12:14:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.744+0000 7f45257a7700 1 -- 172.21.15.187:0/1810455202 learned_addr learned my addr 172.21.15.187:0/1810455202 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.744+0000 7f4524fa6700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4520061330 0x7f452015bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.744+0000 7f451ffff700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f452012cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f45257a7700 1 -- 172.21.15.187:0/1810455202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 msgr2=0x7f452012cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f45257a7700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f452012cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f45257a7700 1 -- 172.21.15.187:0/1810455202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4520061330 msgr2=0x7f452015bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f45257a7700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4520061330 0x7f452015bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f45257a7700 1 -- 172.21.15.187:0/1810455202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45200b6e00 con 0x7f452012d390 2020-11-20T12:14:56.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f45257a7700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452012d390 0x7f45200b8460 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f451800cab0 tx=0x7f45180023b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f451cff9700 1 -- 172.21.15.187:0/1810455202 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451800a830 con 0x7f452012d390 2020-11-20T12:14:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f451cff9700 1 -- 172.21.15.187:0/1810455202 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451800a990 con 0x7f452012d390 2020-11-20T12:14:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f451cff9700 1 -- 172.21.15.187:0/1810455202 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451800ad30 con 0x7f452012d390 2020-11-20T12:14:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45200b70a0 con 0x7f452012d390 2020-11-20T12:14:56.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.745+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45200b89d0 con 0x7f452012d390 2020-11-20T12:14:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.746+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45200b8c80 con 0x7f452012d390 2020-11-20T12:14:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.747+0000 7f451cff9700 1 -- 172.21.15.187:0/1810455202 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4518019030 con 0x7f452012d390 2020-11-20T12:14:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.748+0000 7f451cff9700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44f8056d20 0x7f44f80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.748+0000 7f451cff9700 1 -- 172.21.15.187:0/1810455202 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f451806f850 con 0x7f452012d390 2020-11-20T12:14:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.748+0000 7f4524fa6700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44f8056d20 0x7f44f80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:56.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.748+0000 7f4524fa6700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44f8056d20 0x7f44f80591e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f451400a8c0 tx=0x7f4514008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:56.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.751+0000 7f451cff9700 1 -- 172.21.15.187:0/1810455202 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4518043fd0 con 0x7f452012d390 2020-11-20T12:14:56.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.936+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f45200446d0 con 0x7f452012d390 2020-11-20T12:14:56.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.937+0000 7f451cff9700 1 -- 172.21.15.187:0/1810455202 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4518044d50 con 0x7f452012d390 2020-11-20T12:14:56.940 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:14:56.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44f8056d20 msgr2=0x7f44f80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44f8056d20 0x7f44f80591e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f451400a8c0 tx=0x7f4514008040).stop 2020-11-20T12:14:56.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452012d390 msgr2=0x7f45200b8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:56.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452012d390 0x7f45200b8460 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f451800cab0 tx=0x7f45180023b0).stop 2020-11-20T12:14:56.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 shutdown_connections 2020-11-20T12:14:56.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f44f8056d20 0x7f44f80591e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4520061330 0x7f452015bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452012d390 0x7f45200b8460 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 --2- 172.21.15.187:0/1810455202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45200bd620 0x7f452012cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:56.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 >> 172.21.15.187:0/1810455202 conn(0x7f4520060c10 msgr2=0x7f45200bc660 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:56.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 shutdown_connections 2020-11-20T12:14:56.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:56.940+0000 7f4525fa8700 1 -- 172.21.15.187:0/1810455202 wait complete. 2020-11-20T12:14:56.967 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:14:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.195+0000 7f76af59e700 1 Processor -- start 2020-11-20T12:14:57.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.195+0000 7f76af59e700 1 -- start start 2020-11-20T12:14:57.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.197+0000 7f76af59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f76b00bf800 0x7f76b014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.197+0000 7f76af59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f76b00bfbe0 0x7f76b01480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.197+0000 7f76af59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b014d110 con 0x7f76b009f560 2020-11-20T12:14:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.197+0000 7f76af59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b014cbd0 con 0x7f76b00bfbe0 2020-11-20T12:14:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.197+0000 7f76af59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b014ce70 con 0x7f76b00bf800 2020-11-20T12:14:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.197+0000 7f76ae59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f76b00bf800 0x7f76b014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.197+0000 7f76ae59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f76b00bf800 0x7f76b014c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49542/0 (socket says 172.21.15.187:49542) 2020-11-20T12:14:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.197+0000 7f76ae59c700 1 -- 172.21.15.187:0/3919586355 learned_addr learned my addr 172.21.15.187:0/3919586355 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.198+0000 7f76a7fff700 1 --2- 172.21.15.187:0/3919586355 >> v2:172.21.15.5:3300/0 conn(0x7f76b00bfbe0 0x7f76b01480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.198+0000 7f76ae59c700 1 -- 172.21.15.187:0/3919586355 >> v2:172.21.15.5:3300/0 conn(0x7f76b00bfbe0 msgr2=0x7f76b01480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.198+0000 7f76ae59c700 1 --2- 172.21.15.187:0/3919586355 >> v2:172.21.15.5:3300/0 conn(0x7f76b00bfbe0 0x7f76b01480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.198+0000 7f76ae59c700 1 -- 172.21.15.187:0/3919586355 >> v1:172.21.15.5:6789/0 conn(0x7f76b009f560 legacy=0x7f76b0060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.198+0000 7f76ae59c700 1 -- 172.21.15.187:0/3919586355 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b00a4cc0 con 0x7f76b00bf800 2020-11-20T12:14:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.198+0000 7f76a7fff700 1 --2- 172.21.15.187:0/3919586355 >> v2:172.21.15.5:3300/0 conn(0x7f76b00bfbe0 0x7f76b01480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.198+0000 7f76ae59c700 1 --2- 172.21.15.187:0/3919586355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b014c2c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f76b0068780 tx=0x7f76a0006eb0).ready entity=mon.1 client_cookie=86fc26968ca48102 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:57.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76add9b700 1 -- 172.21.15.187:0/3919586355 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a0009920 con 0x7f76b00bf800 2020-11-20T12:14:57.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76add9b700 1 -- 172.21.15.187:0/3919586355 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a0009a80 con 0x7f76b00bf800 2020-11-20T12:14:57.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76add9b700 1 -- 172.21.15.187:0/3919586355 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a0009e00 con 0x7f76b00bf800 2020-11-20T12:14:57.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76af59e700 1 -- 172.21.15.187:0/3919586355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 msgr2=0x7f76b014c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76af59e700 1 --2- 172.21.15.187:0/3919586355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b014c2c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f76b0068780 tx=0x7f76a0006eb0).stop 2020-11-20T12:14:57.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76af59e700 1 -- 172.21.15.187:0/3919586355 shutdown_connections 2020-11-20T12:14:57.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76af59e700 1 --2- 172.21.15.187:0/3919586355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b014c2c0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76af59e700 1 --2- 172.21.15.187:0/3919586355 >> v2:172.21.15.5:3300/0 conn(0x7f76b00bfbe0 0x7f76b01480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76af59e700 1 -- 172.21.15.187:0/3919586355 >> 172.21.15.187:0/3919586355 conn(0x7f76b0068370 msgr2=0x7f76b00bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:57.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76af59e700 1 -- 172.21.15.187:0/3919586355 shutdown_connections 2020-11-20T12:14:57.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.199+0000 7f76af59e700 1 -- 172.21.15.187:0/3919586355 wait complete. 2020-11-20T12:14:57.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.202+0000 7f76af59e700 1 Processor -- start 2020-11-20T12:14:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.202+0000 7f76af59e700 1 -- start start 2020-11-20T12:14:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76af59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76b009f560 0x7f76b0065d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76af59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b0141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76af59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f76b0141500 0x7f76b0160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76af59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b0066270 con 0x7f76b009f560 2020-11-20T12:14:57.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76af59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b0145f60 con 0x7f76b00bf800 2020-11-20T12:14:57.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76af59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b0066450 con 0x7f76b0141500 2020-11-20T12:14:57.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76a7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b0141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76a7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b0141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49548/0 (socket says 172.21.15.187:49548) 2020-11-20T12:14:57.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.203+0000 7f76a7fff700 1 -- 172.21.15.187:0/1232155299 learned_addr learned my addr 172.21.15.187:0/1232155299 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:57.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76aed9d700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f76b0141500 0x7f76b0160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a7fff700 1 -- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f76b0141500 msgr2=0x7f76b0160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a7fff700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f76b0141500 0x7f76b0160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76ae59c700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76b009f560 0x7f76b0065d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a7fff700 1 -- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76b009f560 msgr2=0x7f76b0065d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a7fff700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76b009f560 0x7f76b0065d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a7fff700 1 -- 172.21.15.187:0/1232155299 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b0143d50 con 0x7f76b00bf800 2020-11-20T12:14:57.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76aed9d700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f76b0141500 0x7f76b0160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:57.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76ae59c700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76b009f560 0x7f76b0065d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:57.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a7fff700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b0141010 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7f769800c920 tx=0x7f7698008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:57.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a67fc700 1 -- 172.21.15.187:0/1232155299 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698007710 con 0x7f76b00bf800 2020-11-20T12:14:57.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a67fc700 1 -- 172.21.15.187:0/1232155299 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7698007870 con 0x7f76b00bf800 2020-11-20T12:14:57.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b0143ff0 con 0x7f76b00bf800 2020-11-20T12:14:57.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.204+0000 7f76a67fc700 1 -- 172.21.15.187:0/1232155299 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698007bf0 con 0x7f76b00bf800 2020-11-20T12:14:57.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.205+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b0160880 con 0x7f76b00bf800 2020-11-20T12:14:57.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.206+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b0080050 con 0x7f76b00bf800 2020-11-20T12:14:57.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.206+0000 7f76a67fc700 1 -- 172.21.15.187:0/1232155299 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f769801b460 con 0x7f76b00bf800 2020-11-20T12:14:57.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.207+0000 7f76a67fc700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f768c056d20 0x7f768c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.207+0000 7f76a67fc700 1 -- 172.21.15.187:0/1232155299 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f76980700b0 con 0x7f76b00bf800 2020-11-20T12:14:57.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.207+0000 7f76ae59c700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f768c056d20 0x7f768c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.208+0000 7f76ae59c700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f768c056d20 0x7f768c0591e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f76a000b1b0 tx=0x7f76a0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:57.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.210+0000 7f76a67fc700 1 -- 172.21.15.187:0/1232155299 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f76980454a0 con 0x7f76b00bf800 2020-11-20T12:14:57.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.439+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76b0144180 con 0x7f76b00bf800 2020-11-20T12:14:57.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.439+0000 7f76a67fc700 1 -- 172.21.15.187:0/1232155299 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2136 (secure 0 0 0) 0x7f7698048ac0 con 0x7f76b00bf800 2020-11-20T12:14:57.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.441+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f768c056d20 msgr2=0x7f768c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.441+0000 7f76af59e700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f768c056d20 0x7f768c0591e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f76a000b1b0 tx=0x7f76a0015000).stop 2020-11-20T12:14:57.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 msgr2=0x7f76b0141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b0141010 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7f769800c920 tx=0x7f7698008040).stop 2020-11-20T12:14:57.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 shutdown_connections 2020-11-20T12:14:57.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f768c056d20 0x7f768c0591e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f76b009f560 0x7f76b0065d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f76b0141500 0x7f76b0160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 --2- 172.21.15.187:0/1232155299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b00bf800 0x7f76b0141010 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 >> 172.21.15.187:0/1232155299 conn(0x7f76b0068370 msgr2=0x7f76b00be120 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 shutdown_connections 2020-11-20T12:14:57.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.442+0000 7f76af59e700 1 -- 172.21.15.187:0/1232155299 wait complete. 2020-11-20T12:14:57.466 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 87 2020-11-20T12:14:57.466 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:57.466 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3285928237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3285928237},{"type":"v1","addr":"172.21.15.5:6833","nonce":3285928237}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:14:29.225459+0000","modified":"2020-11-20T12:14:31.424296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2663757618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2663757618},{"type":"v1","addr":"172.21.15.187:6835","nonce":2663757618}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2020-11-20T12:14:57.468 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:14:57.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.707+0000 7f1cb1065700 1 Processor -- start 2020-11-20T12:14:57.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.707+0000 7f1cb1065700 1 -- start start 2020-11-20T12:14:57.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.708+0000 7f1cb1065700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1cac0661f0 0x7f1cac0b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.708+0000 7f1cb1065700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac099ce0 0x7f1cac0b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.708+0000 7f1cb1065700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac0b5df0 con 0x7f1cac065e10 2020-11-20T12:14:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.709+0000 7f1cb1065700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac09a440 con 0x7f1cac0661f0 2020-11-20T12:14:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.709+0000 7f1cb1065700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac09a1a0 con 0x7f1cac099ce0 2020-11-20T12:14:57.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.709+0000 7f1ca259c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1cac0661f0 0x7f1cac0b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.709+0000 7f1cab59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac099ce0 0x7f1cac0b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.710+0000 7f1ca259c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1cac0661f0 0x7f1cac0b62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:41822/0 (socket says 172.21.15.187:41822) 2020-11-20T12:14:57.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.710+0000 7f1ca259c700 1 -- 172.21.15.187:0/777223951 learned_addr learned my addr 172.21.15.187:0/777223951 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:57.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.709+0000 7f1caad9d700 1 --1- 172.21.15.187:0/777223951 >> v1:172.21.15.5:6789/0 conn(0x7f1cac065e10 0x7f1cac144470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:39672/0 (socket says 172.21.15.187:39672) 2020-11-20T12:14:57.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.710+0000 7f1ca259c700 1 -- 172.21.15.187:0/777223951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac099ce0 msgr2=0x7f1cac0b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:57.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.710+0000 7f1ca259c700 1 --2- 172.21.15.187:0/777223951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac099ce0 0x7f1cac0b67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.711+0000 7f1ca259c700 1 -- 172.21.15.187:0/777223951 >> v1:172.21.15.5:6789/0 conn(0x7f1cac065e10 legacy=0x7f1cac144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.711+0000 7f1ca259c700 1 -- 172.21.15.187:0/777223951 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0b0160 con 0x7f1cac0661f0 2020-11-20T12:14:57.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.711+0000 7f1ca259c700 1 --2- 172.21.15.187:0/777223951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0661f0 0x7f1cac0b62e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f1c94006fd0 tx=0x7f1c94006e50).ready entity=mon.0 client_cookie=e434788a50dd321b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.713+0000 7f1caa59c700 1 -- 172.21.15.187:0/777223951 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94006030 con 0x7f1cac0661f0 2020-11-20T12:14:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.713+0000 7f1caa59c700 1 -- 172.21.15.187:0/777223951 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94006190 con 0x7f1cac0661f0 2020-11-20T12:14:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.713+0000 7f1caa59c700 1 -- 172.21.15.187:0/777223951 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94006510 con 0x7f1cac0661f0 2020-11-20T12:14:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.713+0000 7f1cb1065700 1 -- 172.21.15.187:0/777223951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0661f0 msgr2=0x7f1cac0b62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.713+0000 7f1cb1065700 1 --2- 172.21.15.187:0/777223951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0661f0 0x7f1cac0b62e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f1c94006fd0 tx=0x7f1c94006e50).stop 2020-11-20T12:14:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.713+0000 7f1cb1065700 1 -- 172.21.15.187:0/777223951 shutdown_connections 2020-11-20T12:14:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.713+0000 7f1cb1065700 1 --2- 172.21.15.187:0/777223951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac099ce0 0x7f1cac0b67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.714+0000 7f1cb1065700 1 --2- 172.21.15.187:0/777223951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0661f0 0x7f1cac0b62e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.714+0000 7f1cb1065700 1 -- 172.21.15.187:0/777223951 >> 172.21.15.187:0/777223951 conn(0x7f1cac0bacd0 msgr2=0x7f1cac0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:57.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.714+0000 7f1cb1065700 1 -- 172.21.15.187:0/777223951 shutdown_connections 2020-11-20T12:14:57.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.714+0000 7f1cb1065700 1 -- 172.21.15.187:0/777223951 wait complete. 2020-11-20T12:14:57.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.715+0000 7f1cb1065700 1 Processor -- start 2020-11-20T12:14:57.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.716+0000 7f1cb1065700 1 -- start start 2020-11-20T12:14:57.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1cb1065700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac065e10 0x7f1cac099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1cb1065700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cac0661f0 0x7f1cac09bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1cb1065700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac099ce0 0x7f1cac0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1cb1065700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac099a70 con 0x7f1cac099ce0 2020-11-20T12:14:57.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1cb1065700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac09c4c0 con 0x7f1cac0661f0 2020-11-20T12:14:57.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1cb1065700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0b1ef0 con 0x7f1cac065e10 2020-11-20T12:14:57.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1ca259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cac0661f0 0x7f1cac09bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1ca259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cac0661f0 0x7f1cac09bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49564/0 (socket says 172.21.15.187:49564) 2020-11-20T12:14:57.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.717+0000 7f1ca259c700 1 -- 172.21.15.187:0/1345483260 learned_addr learned my addr 172.21.15.187:0/1345483260 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:57.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1cab59e700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac099ce0 0x7f1cac0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1caad9d700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac065e10 0x7f1cac099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1ca259c700 1 -- 172.21.15.187:0/1345483260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac065e10 msgr2=0x7f1cac099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1ca259c700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac065e10 0x7f1cac099580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1ca259c700 1 -- 172.21.15.187:0/1345483260 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac099ce0 msgr2=0x7f1cac0b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1ca259c700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac099ce0 0x7f1cac0b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1ca259c700 1 -- 172.21.15.187:0/1345483260 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac09c6b0 con 0x7f1cac0661f0 2020-11-20T12:14:57.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1ca259c700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cac0661f0 0x7f1cac09bfd0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f1c94006be0 tx=0x7f1c94015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:57.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1ca37fe700 1 -- 172.21.15.187:0/1345483260 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94005a20 con 0x7f1cac0661f0 2020-11-20T12:14:57.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1ca37fe700 1 -- 172.21.15.187:0/1345483260 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94005b80 con 0x7f1cac0661f0 2020-11-20T12:14:57.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.718+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac1489f0 con 0x7f1cac0661f0 2020-11-20T12:14:57.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.719+0000 7f1ca37fe700 1 -- 172.21.15.187:0/1345483260 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9401e460 con 0x7f1cac0661f0 2020-11-20T12:14:57.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.719+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac0b43e0 con 0x7f1cac0661f0 2020-11-20T12:14:57.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.720+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac063600 con 0x7f1cac0661f0 2020-11-20T12:14:57.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.720+0000 7f1ca37fe700 1 -- 172.21.15.187:0/1345483260 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1c94025030 con 0x7f1cac0661f0 2020-11-20T12:14:57.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.721+0000 7f1ca37fe700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c88056d50 0x7f1c88059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:57.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.721+0000 7f1ca37fe700 1 -- 172.21.15.187:0/1345483260 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1c94071750 con 0x7f1cac0661f0 2020-11-20T12:14:57.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.721+0000 7f1caad9d700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c88056d50 0x7f1c88059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:57.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.722+0000 7f1caad9d700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c88056d50 0x7f1c88059210 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f1c9c007fd0 tx=0x7f1c9c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:57.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.725+0000 7f1ca37fe700 1 -- 172.21.15.187:0/1345483260 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1c94046c20 con 0x7f1cac0661f0 2020-11-20T12:14:57.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.895+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cac0666e0 con 0x7f1cac0661f0 2020-11-20T12:14:57.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.897+0000 7f1ca37fe700 1 -- 172.21.15.187:0/1345483260 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16781 (secure 0 0 0) 0x7f1c94023030 con 0x7f1cac0661f0 2020-11-20T12:14:57.901 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:14:57.901 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":98,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:14:30.229422+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:14:27.347251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2020-11-20T12:14:27.963950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:14:57.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c88056d50 msgr2=0x7f1c88059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c88056d50 0x7f1c88059210 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f1c9c007fd0 tx=0x7f1c9c009040).stop 2020-11-20T12:14:57.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cac0661f0 msgr2=0x7f1cac09bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:57.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cac0661f0 0x7f1cac09bfd0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f1c94006be0 tx=0x7f1c94015000).stop 2020-11-20T12:14:57.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 shutdown_connections 2020-11-20T12:14:57.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c88056d50 0x7f1c88059210 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac099ce0 0x7f1cac0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1cac065e10 0x7f1cac099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 --2- 172.21.15.187:0/1345483260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cac0661f0 0x7f1cac09bfd0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:57.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 >> 172.21.15.187:0/1345483260 conn(0x7f1cac0bacd0 msgr2=0x7f1cac140690 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:57.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 shutdown_connections 2020-11-20T12:14:57.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:57.900+0000 7f1cb1065700 1 -- 172.21.15.187:0/1345483260 wait complete. 2020-11-20T12:14:57.927 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:14:57.928 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:14:58.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.169+0000 7f4e4db52700 1 Processor -- start 2020-11-20T12:14:58.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.169+0000 7f4e4db52700 1 -- start start 2020-11-20T12:14:58.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.170+0000 7f4e4db52700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4e480bcab0 0x7f4e4814ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:58.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.170+0000 7f4e4db52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e48147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:58.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.170+0000 7f4e4db52700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4814cf70 con 0x7f4e480bc6d0 2020-11-20T12:14:58.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.170+0000 7f4e4db52700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4814c0a0 con 0x7f4e480bcab0 2020-11-20T12:14:58.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.170+0000 7f4e4db52700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4814c340 con 0x7f4e480664d0 2020-11-20T12:14:58.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.170+0000 7f4e477fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4e480bc6d0 0x7f4e48066d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46176/0 (socket says 172.21.15.187:46176) 2020-11-20T12:14:58.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.170+0000 7f4e477fe700 1 -- 172.21.15.187:0/2123698470 learned_addr learned my addr 172.21.15.187:0/2123698470 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:58.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.170+0000 7f4e3effd700 1 --2- 172.21.15.187:0/2123698470 >> v2:172.21.15.187:3300/0 conn(0x7f4e480bcab0 0x7f4e4814ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:58.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.171+0000 7f4e47fff700 1 --2- 172.21.15.187:0/2123698470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e48147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:58.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.171+0000 7f4e46ffd700 1 -- 172.21.15.187:0/2123698470 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971522262 0 0) 0x7f4e4814cf70 con 0x7f4e480bc6d0 2020-11-20T12:14:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.171+0000 7f4e46ffd700 1 -- 172.21.15.187:0/2123698470 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e28003240 con 0x7f4e480bc6d0 2020-11-20T12:14:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.171+0000 7f4e3effd700 1 -- 172.21.15.187:0/2123698470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 msgr2=0x7f4e48147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.171+0000 7f4e3effd700 1 --2- 172.21.15.187:0/2123698470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e48147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.171+0000 7f4e3effd700 1 -- 172.21.15.187:0/2123698470 >> v1:172.21.15.187:6789/0 conn(0x7f4e480bc6d0 legacy=0x7f4e48066d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.171+0000 7f4e3effd700 1 -- 172.21.15.187:0/2123698470 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4814c5e0 con 0x7f4e480bcab0 2020-11-20T12:14:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.171+0000 7f4e3effd700 1 --2- 172.21.15.187:0/2123698470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e4814ca80 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f4e3000b9a0 tx=0x7f4e30006df0).ready entity=mon.1 client_cookie=64bc72cc0d39428b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e46ffd700 1 -- 172.21.15.187:0/2123698470 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e300097b0 con 0x7f4e480bcab0 2020-11-20T12:14:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e46ffd700 1 -- 172.21.15.187:0/2123698470 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e30009910 con 0x7f4e480bcab0 2020-11-20T12:14:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e46ffd700 1 -- 172.21.15.187:0/2123698470 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30009c90 con 0x7f4e480bcab0 2020-11-20T12:14:58.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e4db52700 1 -- 172.21.15.187:0/2123698470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 msgr2=0x7f4e4814ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e4db52700 1 --2- 172.21.15.187:0/2123698470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e4814ca80 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f4e3000b9a0 tx=0x7f4e30006df0).stop 2020-11-20T12:14:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e4db52700 1 -- 172.21.15.187:0/2123698470 shutdown_connections 2020-11-20T12:14:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e4db52700 1 --2- 172.21.15.187:0/2123698470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e48147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e4db52700 1 --2- 172.21.15.187:0/2123698470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e4814ca80 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:58.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.172+0000 7f4e4db52700 1 -- 172.21.15.187:0/2123698470 >> 172.21.15.187:0/2123698470 conn(0x7f4e48061d50 msgr2=0x7f4e480bbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:58.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.173+0000 7f4e4db52700 1 -- 172.21.15.187:0/2123698470 shutdown_connections 2020-11-20T12:14:58.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.173+0000 7f4e4db52700 1 -- 172.21.15.187:0/2123698470 wait complete. 2020-11-20T12:14:58.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.175+0000 7f4e4db52700 1 Processor -- start 2020-11-20T12:14:58.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.175+0000 7f4e4db52700 1 -- start start 2020-11-20T12:14:58.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e4db52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e4814aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e4db52700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e480bc6d0 0x7f4e480b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e4db52700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e481407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e4db52700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e481476c0 con 0x7f4e480bc6d0 2020-11-20T12:14:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e4db52700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480b3a20 con 0x7f4e480bcab0 2020-11-20T12:14:58.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e4db52700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480b3750 con 0x7f4e480664d0 2020-11-20T12:14:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e47fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e481407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e47fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e481407f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49580/0 (socket says 172.21.15.187:49580) 2020-11-20T12:14:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.176+0000 7f4e47fff700 1 -- 172.21.15.187:0/768349382 learned_addr learned my addr 172.21.15.187:0/768349382 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:58.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e477fe700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e4814aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e3effd700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e480bc6d0 0x7f4e480b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e47fff700 1 -- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 msgr2=0x7f4e4814aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e47fff700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e4814aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e47fff700 1 -- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e480bc6d0 msgr2=0x7f4e480b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e47fff700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e480bc6d0 0x7f4e480b4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e47fff700 1 -- 172.21.15.187:0/768349382 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4814b3a0 con 0x7f4e480bcab0 2020-11-20T12:14:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e477fe700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e4814aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:14:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e47fff700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e481407f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f4e4000eb40 tx=0x7f4e40008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e3ffff700 1 -- 172.21.15.187:0/768349382 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4000c7a0 con 0x7f4e480bcab0 2020-11-20T12:14:58.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e3ffff700 1 -- 172.21.15.187:0/768349382 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4000c900 con 0x7f4e480bcab0 2020-11-20T12:14:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e48147300 con 0x7f4e480bcab0 2020-11-20T12:14:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.177+0000 7f4e3ffff700 1 -- 172.21.15.187:0/768349382 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4000cca0 con 0x7f4e480bcab0 2020-11-20T12:14:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.178+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e481466b0 con 0x7f4e480bcab0 2020-11-20T12:14:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.178+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e480b3be0 con 0x7f4e480bcab0 2020-11-20T12:14:58.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.179+0000 7f4e3ffff700 1 -- 172.21.15.187:0/768349382 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4e40013640 con 0x7f4e480bcab0 2020-11-20T12:14:58.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.180+0000 7f4e3ffff700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e24056d20 0x7f4e240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:58.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.180+0000 7f4e3ffff700 1 -- 172.21.15.187:0/768349382 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4e4006fdc0 con 0x7f4e480bcab0 2020-11-20T12:14:58.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.180+0000 7f4e477fe700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e24056d20 0x7f4e240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:58.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.181+0000 7f4e477fe700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e24056d20 0x7f4e240591e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f4e380055a0 tx=0x7f4e38009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:58.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.184+0000 7f4e3ffff700 1 -- 172.21.15.187:0/768349382 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4e40045100 con 0x7f4e480bcab0 2020-11-20T12:14:58.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:58.417+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4e48044f60 con 0x7f4e480bcab0 2020-11-20T12:14:59.355 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:14:59.351+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:14:59.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.367+0000 7f4e3ffff700 1 -- 172.21.15.187:0/768349382 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4e40048720 con 0x7f4e480bcab0 2020-11-20T12:14:59.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e24056d20 msgr2=0x7f4e240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:59.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e24056d20 0x7f4e240591e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f4e380055a0 tx=0x7f4e38009040).stop 2020-11-20T12:14:59.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 msgr2=0x7f4e481407f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:59.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e481407f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f4e4000eb40 tx=0x7f4e40008040).stop 2020-11-20T12:14:59.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 shutdown_connections 2020-11-20T12:14:59.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e24056d20 0x7f4e240591e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4e480bcab0 0x7f4e481407f0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e480664d0 0x7f4e4814aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 --2- 172.21.15.187:0/768349382 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e480bc6d0 0x7f4e480b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 >> 172.21.15.187:0/768349382 conn(0x7f4e48061d50 msgr2=0x7f4e480b99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:59.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 shutdown_connections 2020-11-20T12:14:59.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.371+0000 7f4e4db52700 1 -- 172.21.15.187:0/768349382 wait complete. 2020-11-20T12:14:59.377 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:14:59.395 INFO:tasks.ceph.mds.a.smithi187.stderr: -4128> 2020-11-20T12:14:33.422+0000 7ffa74972700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:14:59.402 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:14:59.472 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:14:59.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.624+0000 7f41fffff700 1 Processor -- start 2020-11-20T12:14:59.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.624+0000 7f41fffff700 1 -- start start 2020-11-20T12:14:59.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.625+0000 7f41fffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4200145cc0 0x7f4200066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:59.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.625+0000 7f41fffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f4200065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:59.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.625+0000 7f41fffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420014c170 con 0x7f4200065680 2020-11-20T12:14:59.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.625+0000 7f41fffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42001497f0 con 0x7f4200145cc0 2020-11-20T12:14:59.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.625+0000 7f41fffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420014c410 con 0x7f42000683f0 2020-11-20T12:14:59.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.626+0000 7f41fe7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f4200065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:59.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.626+0000 7f41ff7fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4200065680 0x7f4200061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:39714/0 (socket says 172.21.15.187:39714) 2020-11-20T12:14:59.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.626+0000 7f41ff7fe700 1 -- 172.21.15.187:0/2288100213 learned_addr learned my addr 172.21.15.187:0/2288100213 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:14:59.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.626+0000 7f41feffd700 1 --2- 172.21.15.187:0/2288100213 >> v2:172.21.15.5:3300/0 conn(0x7f4200145cc0 0x7f4200066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:59.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.627+0000 7f41fdffb700 1 -- 172.21.15.187:0/2288100213 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593233902 0 0) 0x7f420014c170 con 0x7f4200065680 2020-11-20T12:14:59.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.627+0000 7f41fdffb700 1 -- 172.21.15.187:0/2288100213 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec003240 con 0x7f4200065680 2020-11-20T12:14:59.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.628+0000 7f41feffd700 1 -- 172.21.15.187:0/2288100213 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 msgr2=0x7f4200065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:14:59.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.628+0000 7f41feffd700 1 --2- 172.21.15.187:0/2288100213 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f4200065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.628+0000 7f41feffd700 1 -- 172.21.15.187:0/2288100213 >> v1:172.21.15.5:6789/0 conn(0x7f4200065680 legacy=0x7f4200061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:59.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.628+0000 7f41feffd700 1 -- 172.21.15.187:0/2288100213 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420014c6b0 con 0x7f4200145cc0 2020-11-20T12:14:59.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.628+0000 7f41fe7fc700 1 --2- 172.21.15.187:0/2288100213 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f4200065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:14:59.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.628+0000 7f41feffd700 1 --2- 172.21.15.187:0/2288100213 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200145cc0 0x7f4200066040 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f41f000b9a0 tx=0x7f41f0006df0).ready entity=mon.0 client_cookie=e04b33b751391bb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:59.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.629+0000 7f41fdffb700 1 -- 172.21.15.187:0/2288100213 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0009680 con 0x7f4200145cc0 2020-11-20T12:14:59.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.629+0000 7f41fdffb700 1 -- 172.21.15.187:0/2288100213 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f00097e0 con 0x7f4200145cc0 2020-11-20T12:14:59.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.629+0000 7f41fdffb700 1 -- 172.21.15.187:0/2288100213 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0009b60 con 0x7f4200145cc0 2020-11-20T12:14:59.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.630+0000 7f41fffff700 1 -- 172.21.15.187:0/2288100213 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200145cc0 msgr2=0x7f4200066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:59.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.630+0000 7f41fffff700 1 --2- 172.21.15.187:0/2288100213 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200145cc0 0x7f4200066040 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f41f000b9a0 tx=0x7f41f0006df0).stop 2020-11-20T12:14:59.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.630+0000 7f41fffff700 1 -- 172.21.15.187:0/2288100213 shutdown_connections 2020-11-20T12:14:59.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.630+0000 7f41fffff700 1 --2- 172.21.15.187:0/2288100213 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200145cc0 0x7f4200066040 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.630+0000 7f41fffff700 1 --2- 172.21.15.187:0/2288100213 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f4200065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.630+0000 7f41fffff700 1 -- 172.21.15.187:0/2288100213 >> 172.21.15.187:0/2288100213 conn(0x7f42000bab30 msgr2=0x7f42000baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:14:59.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.630+0000 7f41fffff700 1 -- 172.21.15.187:0/2288100213 shutdown_connections 2020-11-20T12:14:59.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.630+0000 7f41fffff700 1 -- 172.21.15.187:0/2288100213 wait complete. 2020-11-20T12:14:59.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.632+0000 7f41fffff700 1 Processor -- start 2020-11-20T12:14:59.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.632+0000 7f41fffff700 1 -- start start 2020-11-20T12:14:59.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.633+0000 7f41fffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200065680 0x7f420009e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:59.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.633+0000 7f41fffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f420014f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:59.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.633+0000 7f41fffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f420009dae0 0x7f42001438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:59.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.633+0000 7f41fffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420009e1b0 con 0x7f4200065680 2020-11-20T12:14:59.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.633+0000 7f41fffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420009e410 con 0x7f420009dae0 2020-11-20T12:14:59.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.633+0000 7f41fffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420009a670 con 0x7f42000683f0 2020-11-20T12:14:59.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.633+0000 7f41ff7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f420009dae0 0x7f42001438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:59.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41ff7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f420009dae0 0x7f42001438c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49604/0 (socket says 172.21.15.187:49604) 2020-11-20T12:14:59.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41ff7fe700 1 -- 172.21.15.187:0/2763338430 learned_addr learned my addr 172.21.15.187:0/2763338430 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:14:59.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41feffd700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200065680 0x7f420009e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:59.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41fe7fc700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f420014f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:59.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41ff7fe700 1 -- 172.21.15.187:0/2763338430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 msgr2=0x7f420014f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:59.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41ff7fe700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f420014f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41ff7fe700 1 -- 172.21.15.187:0/2763338430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200065680 msgr2=0x7f420009e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:14:59.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41ff7fe700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200065680 0x7f420009e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:14:59.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41ff7fe700 1 -- 172.21.15.187:0/2763338430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420009a910 con 0x7f420009dae0 2020-11-20T12:14:59.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.634+0000 7f41ff7fe700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f420009dae0 0x7f42001438c0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f41f8001420 tx=0x7f41f8007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:59.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.635+0000 7f41e6ffd700 1 -- 172.21.15.187:0/2763338430 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8005b80 con 0x7f420009dae0 2020-11-20T12:14:59.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.635+0000 7f41e6ffd700 1 -- 172.21.15.187:0/2763338430 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f8005ce0 con 0x7f420009dae0 2020-11-20T12:14:59.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.635+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420014fe60 con 0x7f420009dae0 2020-11-20T12:14:59.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.635+0000 7f41e6ffd700 1 -- 172.21.15.187:0/2763338430 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8007320 con 0x7f420009dae0 2020-11-20T12:14:59.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.635+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4200143ff0 con 0x7f420009dae0 2020-11-20T12:14:59.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.636+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420009aaa0 con 0x7f420009dae0 2020-11-20T12:14:59.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.636+0000 7f41e6ffd700 1 -- 172.21.15.187:0/2763338430 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f41f8010030 con 0x7f420009dae0 2020-11-20T12:14:59.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.637+0000 7f41e6ffd700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f41dc056d20 0x7f41dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:14:59.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.637+0000 7f41e6ffd700 1 -- 172.21.15.187:0/2763338430 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f41f8070070 con 0x7f420009dae0 2020-11-20T12:14:59.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.637+0000 7f41feffd700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f41dc056d20 0x7f41dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:14:59.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.638+0000 7f41feffd700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f41dc056d20 0x7f41dc0591e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f4200063360 tx=0x7f41f0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:14:59.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.641+0000 7f41e6ffd700 1 -- 172.21.15.187:0/2763338430 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f41f8045470 con 0x7f420009dae0 2020-11-20T12:14:59.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:14:59.887+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4200061a20 con 0x7f420009dae0 2020-11-20T12:15:00.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.378+0000 7f41e6ffd700 1 -- 172.21.15.187:0/2763338430 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v89) v1 ==== 104+0+0 (secure 0 0 0) 0x7f41f8048a90 con 0x7f420009dae0 2020-11-20T12:15:00.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f41dc056d20 msgr2=0x7f41dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:00.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f41dc056d20 0x7f41dc0591e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f4200063360 tx=0x7f41f0015000).stop 2020-11-20T12:15:00.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f420009dae0 msgr2=0x7f42001438c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:00.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f420009dae0 0x7f42001438c0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f41f8001420 tx=0x7f41f8007e20).stop 2020-11-20T12:15:00.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 shutdown_connections 2020-11-20T12:15:00.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f41dc056d20 0x7f41dc0591e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:00.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4200065680 0x7f420009e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:00.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f420009dae0 0x7f42001438c0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:00.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 --2- 172.21.15.187:0/2763338430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42000683f0 0x7f420014f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:00.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.381+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 >> 172.21.15.187:0/2763338430 conn(0x7f42000bab30 msgr2=0x7f42001460a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:00.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.382+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 shutdown_connections 2020-11-20T12:15:00.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.382+0000 7f41fffff700 1 -- 172.21.15.187:0/2763338430 wait complete. 2020-11-20T12:15:00.413 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:15:00.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.656+0000 7f8feaf6b700 1 Processor -- start 2020-11-20T12:15:00.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.656+0000 7f8feaf6b700 1 -- start start 2020-11-20T12:15:00.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.658+0000 7f8feaf6b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8fe414d270 0x7f8fe41573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:00.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.658+0000 7f8feaf6b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe40658a0 con 0x7f8fe40b1380 2020-11-20T12:15:00.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.658+0000 7f8feaf6b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe4150390 con 0x7f8fe40b1760 2020-11-20T12:15:00.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.658+0000 7f8feaf6b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe41511e0 con 0x7f8fe414d270 2020-11-20T12:15:00.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.658+0000 7f8fea76a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8fe414d270 0x7f8fe41573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.658+0000 7f8fe9768700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8fe40b1760 0x7f8fe4065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46216/0 (socket says 172.21.15.187:46216) 2020-11-20T12:15:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.658+0000 7f8fe9768700 1 -- 172.21.15.187:0/2258744287 learned_addr learned my addr 172.21.15.187:0/2258744287 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.659+0000 7f8fea76a700 1 -- 172.21.15.187:0/2258744287 >> v1:172.21.15.187:6789/0 conn(0x7f8fe40b1760 legacy=0x7f8fe4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.659+0000 7f8fea76a700 1 -- 172.21.15.187:0/2258744287 >> v1:172.21.15.5:6789/0 conn(0x7f8fe40b1380 legacy=0x7f8fe4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.659+0000 7f8fea76a700 1 -- 172.21.15.187:0/2258744287 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe4150630 con 0x7f8fe414d270 2020-11-20T12:15:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.659+0000 7f8fea76a700 1 --2- 172.21.15.187:0/2258744287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe41573a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7f8fdc009c90 tx=0x7f8fdc0091d0).ready entity=mon.1 client_cookie=ecdc28f9a09f49b7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.659+0000 7f8fe8f67700 1 -- 172.21.15.187:0/2258744287 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc00bb80 con 0x7f8fe414d270 2020-11-20T12:15:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.659+0000 7f8fe8f67700 1 -- 172.21.15.187:0/2258744287 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc00bce0 con 0x7f8fe414d270 2020-11-20T12:15:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.659+0000 7f8fe8f67700 1 -- 172.21.15.187:0/2258744287 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc011610 con 0x7f8fe414d270 2020-11-20T12:15:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.660+0000 7f8feaf6b700 1 -- 172.21.15.187:0/2258744287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 msgr2=0x7f8fe41573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.660+0000 7f8feaf6b700 1 --2- 172.21.15.187:0/2258744287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe41573a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7f8fdc009c90 tx=0x7f8fdc0091d0).stop 2020-11-20T12:15:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.660+0000 7f8feaf6b700 1 -- 172.21.15.187:0/2258744287 shutdown_connections 2020-11-20T12:15:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.660+0000 7f8feaf6b700 1 --2- 172.21.15.187:0/2258744287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe41573a0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.660+0000 7f8feaf6b700 1 -- 172.21.15.187:0/2258744287 >> 172.21.15.187:0/2258744287 conn(0x7f8fe40bab30 msgr2=0x7f8fe40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.660+0000 7f8feaf6b700 1 -- 172.21.15.187:0/2258744287 shutdown_connections 2020-11-20T12:15:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.660+0000 7f8feaf6b700 1 -- 172.21.15.187:0/2258744287 wait complete. 2020-11-20T12:15:00.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.662+0000 7f8feaf6b700 1 Processor -- start 2020-11-20T12:15:00.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.662+0000 7f8feaf6b700 1 -- start start 2020-11-20T12:15:00.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.663+0000 7f8feaf6b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe40b1380 0x7f8fe409fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:00.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.663+0000 7f8feaf6b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8fe40b1760 0x7f8fe40a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:00.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8feaf6b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe413d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:00.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8feaf6b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe409c840 con 0x7f8fe40b1380 2020-11-20T12:15:00.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8feaf6b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe409cae0 con 0x7f8fe414d270 2020-11-20T12:15:00.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8feaf6b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe40a0730 con 0x7f8fe40b1760 2020-11-20T12:15:00.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fea76a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe413d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:00.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fea76a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe413d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49618/0 (socket says 172.21.15.187:49618) 2020-11-20T12:15:00.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fea76a700 1 -- 172.21.15.187:0/3152526692 learned_addr learned my addr 172.21.15.187:0/3152526692 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:00.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fe9f69700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe40b1380 0x7f8fe409fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:00.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fe9768700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8fe40b1760 0x7f8fe40a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:00.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fea76a700 1 -- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8fe40b1760 msgr2=0x7f8fe40a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:00.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fea76a700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8fe40b1760 0x7f8fe40a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:00.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fea76a700 1 -- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe40b1380 msgr2=0x7f8fe409fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:00.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fea76a700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe40b1380 0x7f8fe409fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:00.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fea76a700 1 -- 172.21.15.187:0/3152526692 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe40b4520 con 0x7f8fe414d270 2020-11-20T12:15:00.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.664+0000 7f8fe9f69700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe40b1380 0x7f8fe409fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:00.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.665+0000 7f8fea76a700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe413d1e0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f8fdc009210 tx=0x7f8fdc016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:00.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.665+0000 7f8fd1ffb700 1 -- 172.21.15.187:0/3152526692 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc00be50 con 0x7f8fe414d270 2020-11-20T12:15:00.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.665+0000 7f8fd1ffb700 1 -- 172.21.15.187:0/3152526692 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc010c00 con 0x7f8fe414d270 2020-11-20T12:15:00.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.665+0000 7f8fd1ffb700 1 -- 172.21.15.187:0/3152526692 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc01d480 con 0x7f8fe414d270 2020-11-20T12:15:00.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.665+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe409c020 con 0x7f8fe414d270 2020-11-20T12:15:00.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.665+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe413d6d0 con 0x7f8fe414d270 2020-11-20T12:15:00.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.667+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe409c6b0 con 0x7f8fe414d270 2020-11-20T12:15:00.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.667+0000 7f8fd1ffb700 1 -- 172.21.15.187:0/3152526692 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8fdc017070 con 0x7f8fe414d270 2020-11-20T12:15:00.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.668+0000 7f8fd1ffb700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fbc056d20 0x7f8fbc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:00.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.668+0000 7f8fd1ffb700 1 -- 172.21.15.187:0/3152526692 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8fdc072780 con 0x7f8fe414d270 2020-11-20T12:15:00.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.668+0000 7f8fe9f69700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fbc056d20 0x7f8fbc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:00.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.669+0000 7f8fe9f69700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fbc056d20 0x7f8fbc0591e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f8fe0002510 tx=0x7f8fe0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:00.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.672+0000 7f8fd1ffb700 1 -- 172.21.15.187:0/3152526692 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8fdc047b80 con 0x7f8fe414d270 2020-11-20T12:15:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:00.857+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8fe4061a20 con 0x7f8fe414d270 2020-11-20T12:15:01.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.386+0000 7f8fd1ffb700 1 -- 172.21.15.187:0/3152526692 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8fdc015020 con 0x7f8fe414d270 2020-11-20T12:15:01.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.389+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fbc056d20 msgr2=0x7f8fbc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:01.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.389+0000 7f8feaf6b700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fbc056d20 0x7f8fbc0591e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f8fe0002510 tx=0x7f8fe0009040).stop 2020-11-20T12:15:01.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.389+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 msgr2=0x7f8fe413d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:01.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.389+0000 7f8feaf6b700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe413d1e0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f8fdc009210 tx=0x7f8fdc016000).stop 2020-11-20T12:15:01.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.390+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 shutdown_connections 2020-11-20T12:15:01.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.390+0000 7f8feaf6b700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fbc056d20 0x7f8fbc0591e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.390+0000 7f8feaf6b700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe40b1380 0x7f8fe409fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.390+0000 7f8feaf6b700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fe414d270 0x7f8fe413d1e0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.390+0000 7f8feaf6b700 1 --2- 172.21.15.187:0/3152526692 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8fe40b1760 0x7f8fe40a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.390+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 >> 172.21.15.187:0/3152526692 conn(0x7f8fe40bab30 msgr2=0x7f8fe4149190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:01.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.390+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 shutdown_connections 2020-11-20T12:15:01.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.390+0000 7f8feaf6b700 1 -- 172.21.15.187:0/3152526692 wait complete. 2020-11-20T12:15:01.396 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:15:01.425 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:15:01.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.670+0000 7f017b58a700 1 Processor -- start 2020-11-20T12:15:01.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.670+0000 7f017b58a700 1 -- start start 2020-11-20T12:15:01.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.672+0000 7f017b58a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f017414d270 0x7f01741573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:01.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.672+0000 7f017b58a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01740658a0 con 0x7f01740b1760 2020-11-20T12:15:01.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.672+0000 7f017b58a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0174150390 con 0x7f01740b1380 2020-11-20T12:15:01.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.672+0000 7f017b58a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01741511e0 con 0x7f017414d270 2020-11-20T12:15:01.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.672+0000 7f017a588700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f01740b1380 0x7f0174150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46230/0 (socket says 172.21.15.187:46230) 2020-11-20T12:15:01.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.672+0000 7f017a588700 1 -- 172.21.15.187:0/764829289 learned_addr learned my addr 172.21.15.187:0/764829289 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:01.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.673+0000 7f017ad89700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f017414d270 0x7f01741573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:01.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.673+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161816445 0 0) 0x7f0174150390 con 0x7f01740b1380 2020-11-20T12:15:01.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.673+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f015c003240 con 0x7f01740b1380 2020-11-20T12:15:01.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.673+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 239911441 0 0) 0x7f015c003240 con 0x7f01740b1380 2020-11-20T12:15:01.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.673+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f017414d270 msgr2=0x7f01741573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:01.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.673+0000 7f0179586700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f017414d270 0x7f01741573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.673+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 >> v1:172.21.15.5:6789/0 conn(0x7f01740b1760 legacy=0x7f0174065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:01.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0174150630 con 0x7f01740b1380 2020-11-20T12:15:01.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f01700014e0 con 0x7f01740b1380 2020-11-20T12:15:01.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 >> v1:172.21.15.187:6789/0 conn(0x7f01740b1380 legacy=0x7f0174150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:01.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f015c003730 0x7f015c003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:01.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f015c004070 0x7f015c004540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:01.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f017414d270 0x7f015c004bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:01.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01741511e0 con 0x7f015c004070 2020-11-20T12:15:01.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c005140 con 0x7f015c003730 2020-11-20T12:15:01.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c005300 con 0x7f017414d270 2020-11-20T12:15:01.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.674+0000 7f017a588700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f015c003730 0x7f015c003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:01.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f017ad89700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f017414d270 0x7f015c004bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:01.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f0179d87700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f015c004070 0x7f015c004540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:01.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f017a588700 1 -- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f017414d270 msgr2=0x7f015c004bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:01.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f017a588700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f017414d270 0x7f015c004bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f017a588700 1 -- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f015c004070 msgr2=0x7f015c004540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:01.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f017a588700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f015c004070 0x7f015c004540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f017a588700 1 -- 172.21.15.187:0/764829289 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c003240 con 0x7f015c003730 2020-11-20T12:15:01.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f017a588700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f015c003730 0x7f015c003b10 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7f017000c440 tx=0x7f0170007750).ready entity=mon.1 client_cookie=c043d3c648fcdef0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:01.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01700070d0 con 0x7f015c003730 2020-11-20T12:15:01.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0170007230 con 0x7f015c003730 2020-11-20T12:15:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.675+0000 7f0179586700 1 -- 172.21.15.187:0/764829289 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01700055f0 con 0x7f015c003730 2020-11-20T12:15:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.676+0000 7f017b58a700 1 -- 172.21.15.187:0/764829289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f015c003730 msgr2=0x7f015c003b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.676+0000 7f017b58a700 1 --2- 172.21.15.187:0/764829289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f015c003730 0x7f015c003b10 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7f017000c440 tx=0x7f0170007750).stop 2020-11-20T12:15:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.676+0000 7f017ad89700 1 -- 172.21.15.187:0/764829289 reap_dead start 2020-11-20T12:15:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.676+0000 7f017b58a700 1 -- 172.21.15.187:0/764829289 shutdown_connections 2020-11-20T12:15:01.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.676+0000 7f017b58a700 1 -- 172.21.15.187:0/764829289 >> 172.21.15.187:0/764829289 conn(0x7f01740bab30 msgr2=0x7f01740baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:01.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.676+0000 7f017b58a700 1 -- 172.21.15.187:0/764829289 shutdown_connections 2020-11-20T12:15:01.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.677+0000 7f017b58a700 1 -- 172.21.15.187:0/764829289 wait complete. 2020-11-20T12:15:01.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.679+0000 7f017b58a700 1 Processor -- start 2020-11-20T12:15:01.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.679+0000 7f017b58a700 1 -- start start 2020-11-20T12:15:01.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f017b58a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01740b52d0 0x7f0174147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:01.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f017b58a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01740b56b0 0x7f017415f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:01.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f017b58a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0174148440 0x7f017415fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:01.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f017b58a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017409e750 con 0x7f01740b52d0 2020-11-20T12:15:01.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f017b58a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017409e9f0 con 0x7f01740b56b0 2020-11-20T12:15:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f017b58a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01740b5c50 con 0x7f0174148440 2020-11-20T12:15:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f0179d87700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01740b56b0 0x7f017415f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f0179d87700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01740b56b0 0x7f017415f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49636/0 (socket says 172.21.15.187:49636) 2020-11-20T12:15:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.680+0000 7f0179d87700 1 -- 172.21.15.187:0/3780792443 learned_addr learned my addr 172.21.15.187:0/3780792443 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:01.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f017a588700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01740b52d0 0x7f0174147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f017ad89700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0174148440 0x7f017415fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f0179d87700 1 -- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0174148440 msgr2=0x7f017415fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f0179d87700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0174148440 0x7f017415fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f0179d87700 1 -- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01740b52d0 msgr2=0x7f0174147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:01.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f0179d87700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01740b52d0 0x7f0174147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:01.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f0179d87700 1 -- 172.21.15.187:0/3780792443 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01740b4020 con 0x7f01740b56b0 2020-11-20T12:15:01.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f017ad89700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0174148440 0x7f017415fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:01.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f0179d87700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01740b56b0 0x7f017415f840 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7f016800eb30 tx=0x7f0168009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:01.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f01667fc700 1 -- 172.21.15.187:0/3780792443 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800c750 con 0x7f01740b56b0 2020-11-20T12:15:01.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f01667fc700 1 -- 172.21.15.187:0/3780792443 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016800c8b0 con 0x7f01740b56b0 2020-11-20T12:15:01.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f01667fc700 1 -- 172.21.15.187:0/3780792443 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800cc30 con 0x7f01740b56b0 2020-11-20T12:15:01.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.681+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01740b45f0 con 0x7f01740b56b0 2020-11-20T12:15:01.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.682+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0174160220 con 0x7f01740b56b0 2020-11-20T12:15:01.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.683+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017409e5a0 con 0x7f01740b56b0 2020-11-20T12:15:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.683+0000 7f01667fc700 1 -- 172.21.15.187:0/3780792443 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f016801b030 con 0x7f01740b56b0 2020-11-20T12:15:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.684+0000 7f01667fc700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f014c056d20 0x7f014c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.684+0000 7f01667fc700 1 -- 172.21.15.187:0/3780792443 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f016806fbc0 con 0x7f01740b56b0 2020-11-20T12:15:01.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.684+0000 7f017a588700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f014c056d20 0x7f014c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:01.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.685+0000 7f017a588700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f014c056d20 0x7f014c0591e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f017405ea30 tx=0x7f0170015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:01.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.688+0000 7f01667fc700 1 -- 172.21.15.187:0/3780792443 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0168045220 con 0x7f01740b56b0 2020-11-20T12:15:01.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:01.879+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0174061a20 con 0x7f01740b56b0 2020-11-20T12:15:02.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.398+0000 7f01667fc700 1 -- 172.21.15.187:0/3780792443 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0168048840 con 0x7f01740b56b0 2020-11-20T12:15:02.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.401+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f014c056d20 msgr2=0x7f014c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.401+0000 7f017b58a700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f014c056d20 0x7f014c0591e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f017405ea30 tx=0x7f0170015000).stop 2020-11-20T12:15:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.401+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01740b56b0 msgr2=0x7f017415f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.401+0000 7f017b58a700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01740b56b0 0x7f017415f840 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7f016800eb30 tx=0x7f0168009040).stop 2020-11-20T12:15:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.402+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 shutdown_connections 2020-11-20T12:15:02.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.402+0000 7f017b58a700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f014c056d20 0x7f014c0591e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:02.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.402+0000 7f017b58a700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01740b52d0 0x7f0174147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:02.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.402+0000 7f017b58a700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0174148440 0x7f017415fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:02.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.402+0000 7f017b58a700 1 --2- 172.21.15.187:0/3780792443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01740b56b0 0x7f017415f840 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:02.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.402+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 >> 172.21.15.187:0/3780792443 conn(0x7f01740bab30 msgr2=0x7f017414e220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:02.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.402+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 shutdown_connections 2020-11-20T12:15:02.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:02.402+0000 7f017b58a700 1 -- 172.21.15.187:0/3780792443 wait complete. 2020-11-20T12:15:02.408 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:15:02.433 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:15:02.433 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:15:02.434 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:15:02.434 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:15:02.434 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:15:02.435 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:15:02.435 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:15:02.435 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:15:02.457 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:15:02.457 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:15:02.463 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:15:02.464 DEBUG:teuthology.parallel:result is None 2020-11-20T12:15:02.645 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:15:07.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2020-11-20T12:15:08.468 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:15:08.468 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:15:08.482 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:15:08.483 DEBUG:teuthology.parallel:result is None 2020-11-20T12:15:08.483 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:15:08.682 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:15:08.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.739+0000 7fb6ee879700 1 Processor -- start 2020-11-20T12:15:08.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.739+0000 7fb6ee879700 1 -- start start 2020-11-20T12:15:08.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.741+0000 7fb6ee879700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb6e80b1380 0x7fb6e8150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.741+0000 7fb6ee879700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e80658a0 con 0x7fb6e814c260 2020-11-20T12:15:08.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.741+0000 7fb6ee879700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e8150390 con 0x7fb6e80b1760 2020-11-20T12:15:08.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.741+0000 7fb6ee879700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e8150630 con 0x7fb6e80b1380 2020-11-20T12:15:08.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.741+0000 7fb6ed076700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb6e80b1760 0x7fb6e8065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46268/0 (socket says 172.21.15.187:46268) 2020-11-20T12:15:08.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.741+0000 7fb6ed076700 1 -- 172.21.15.187:0/1106385265 learned_addr learned my addr 172.21.15.187:0/1106385265 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:08.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.742+0000 7fb6ed877700 1 --2- 172.21.15.187:0/1106385265 >> v2:172.21.15.5:3300/0 conn(0x7fb6e80b1380 0x7fb6e8150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.742+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188270241 0 0) 0x7fb6e8150390 con 0x7fb6e80b1760 2020-11-20T12:15:08.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.742+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d0003240 con 0x7fb6e80b1760 2020-11-20T12:15:08.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.742+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1170300454 0 0) 0x7fb6d0003240 con 0x7fb6e80b1760 2020-11-20T12:15:08.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.742+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 >> v2:172.21.15.5:3300/0 conn(0x7fb6e80b1380 msgr2=0x7fb6e8150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:08.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.742+0000 7fb6ec875700 1 --2- 172.21.15.187:0/1106385265 >> v2:172.21.15.5:3300/0 conn(0x7fb6e80b1380 0x7fb6e8150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.742+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 >> v1:172.21.15.5:6789/0 conn(0x7fb6e814c260 legacy=0x7fb6e814c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8068510 con 0x7fb6e80b1760 2020-11-20T12:15:08.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fb6d80014e0 con 0x7fb6e80b1760 2020-11-20T12:15:08.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 >> v1:172.21.15.187:6789/0 conn(0x7fb6e80b1760 legacy=0x7fb6e8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6d0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6d00044e0 0x7fb6d0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6d0004e50 0x7fb6d000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e8150630 con 0x7fb6d00044e0 2020-11-20T12:15:08.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d0004260 con 0x7fb6d0003730 2020-11-20T12:15:08.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d000bc20 con 0x7fb6d0004e50 2020-11-20T12:15:08.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ed877700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6d0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ed877700 1 -- 172.21.15.187:0/1106385265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6d0004e50 msgr2=0x7fb6d000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:08.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.743+0000 7fb6ee078700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6d0004e50 0x7fb6d000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ed877700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6d0004e50 0x7fb6d000b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ed877700 1 -- 172.21.15.187:0/1106385265 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6d00044e0 msgr2=0x7fb6d0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:08.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ed877700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6d00044e0 0x7fb6d0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:08.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ed877700 1 -- 172.21.15.187:0/1106385265 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d0003240 con 0x7fb6d0003730 2020-11-20T12:15:08.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ee078700 1 -- 172.21.15.187:0/1106385265 reap_dead start 2020-11-20T12:15:08.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ed877700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6d0003b50 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7fb6e400bc00 tx=0x7fb6e4006df0).ready entity=mon.1 client_cookie=9a11e54a3183d09e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:08.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e400b310 con 0x7fb6d0003730 2020-11-20T12:15:08.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e400eb40 con 0x7fb6d0003730 2020-11-20T12:15:08.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.744+0000 7fb6ec875700 1 -- 172.21.15.187:0/1106385265 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e400eec0 con 0x7fb6d0003730 2020-11-20T12:15:08.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.745+0000 7fb6ee879700 1 -- 172.21.15.187:0/1106385265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 msgr2=0x7fb6d0003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.745+0000 7fb6ee879700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6d0003b50 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7fb6e400bc00 tx=0x7fb6e4006df0).stop 2020-11-20T12:15:08.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.745+0000 7fb6ee879700 1 -- 172.21.15.187:0/1106385265 shutdown_connections 2020-11-20T12:15:08.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.745+0000 7fb6ee879700 1 --2- 172.21.15.187:0/1106385265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6d0003b50 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.745+0000 7fb6ee879700 1 -- 172.21.15.187:0/1106385265 >> 172.21.15.187:0/1106385265 conn(0x7fb6e80bab30 msgr2=0x7fb6e80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:08.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.745+0000 7fb6ee879700 1 -- 172.21.15.187:0/1106385265 shutdown_connections 2020-11-20T12:15:08.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.745+0000 7fb6ee879700 1 -- 172.21.15.187:0/1106385265 wait complete. 2020-11-20T12:15:08.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.747+0000 7fb6ee879700 1 Processor -- start 2020-11-20T12:15:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.747+0000 7fb6ee879700 1 -- start start 2020-11-20T12:15:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.747+0000 7fb6ee879700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6e814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.747+0000 7fb6ee879700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6e80b6840 0x7fb6e80b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ee879700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6e80b3000 0x7fb6e80b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ee879700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e80b6f50 con 0x7fb6e80b6840 2020-11-20T12:15:08.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ee879700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e80b6cc0 con 0x7fb6d0003730 2020-11-20T12:15:08.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ee879700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e80b7260 con 0x7fb6e80b3000 2020-11-20T12:15:08.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed877700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6e814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed877700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6e814b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49674/0 (socket says 172.21.15.187:49674) 2020-11-20T12:15:08.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed877700 1 -- 172.21.15.187:0/1737943562 learned_addr learned my addr 172.21.15.187:0/1737943562 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:08.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ee078700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6e80b3000 0x7fb6e80b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed877700 1 -- 172.21.15.187:0/1737943562 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6e80b3000 msgr2=0x7fb6e80b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed877700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6e80b3000 0x7fb6e80b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed076700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6e80b6840 0x7fb6e80b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed877700 1 -- 172.21.15.187:0/1737943562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6e80b6840 msgr2=0x7fb6e80b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed877700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6e80b6840 0x7fb6e80b2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.748+0000 7fb6ed877700 1 -- 172.21.15.187:0/1737943562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e80b7500 con 0x7fb6d0003730 2020-11-20T12:15:08.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.749+0000 7fb6ed877700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6e814b410 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7fb6e4012d90 tx=0x7fb6e4019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:08.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.749+0000 7fb6d5ffb700 1 -- 172.21.15.187:0/1737943562 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e401d620 con 0x7fb6d0003730 2020-11-20T12:15:08.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.749+0000 7fb6d5ffb700 1 -- 172.21.15.187:0/1737943562 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e4021070 con 0x7fb6d0003730 2020-11-20T12:15:08.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.749+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e80b7e90 con 0x7fb6d0003730 2020-11-20T12:15:08.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.749+0000 7fb6d5ffb700 1 -- 172.21.15.187:0/1737943562 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e401d7a0 con 0x7fb6d0003730 2020-11-20T12:15:08.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.749+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e815c130 con 0x7fb6d0003730 2020-11-20T12:15:08.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.750+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e80b7690 con 0x7fb6d0003730 2020-11-20T12:15:08.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.751+0000 7fb6d5ffb700 1 -- 172.21.15.187:0/1737943562 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb6e402d070 con 0x7fb6d0003730 2020-11-20T12:15:08.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.751+0000 7fb6d5ffb700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb6c0056d20 0x7fb6c00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.751+0000 7fb6d5ffb700 1 -- 172.21.15.187:0/1737943562 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb6e4071ea0 con 0x7fb6d0003730 2020-11-20T12:15:08.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.752+0000 7fb6ed076700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb6c0056d20 0x7fb6c00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.752+0000 7fb6ed076700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb6c0056d20 0x7fb6c00591e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fb6d8001690 tx=0x7fb6d8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:08.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.755+0000 7fb6d5ffb700 1 -- 172.21.15.187:0/1737943562 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb6e40475d0 con 0x7fb6d0003730 2020-11-20T12:15:08.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:08.959+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb6e8061a20 con 0x7fb6d0003730 2020-11-20T12:15:09.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.812+0000 7fb6d5ffb700 1 -- 172.21.15.187:0/1737943562 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v102) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb6e404abf0 con 0x7fb6d0003730 2020-11-20T12:15:09.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb6c0056d20 msgr2=0x7fb6c00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:09.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb6c0056d20 0x7fb6c00591e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fb6d8001690 tx=0x7fb6d8009040).stop 2020-11-20T12:15:09.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 msgr2=0x7fb6e814b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6e814b410 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7fb6e4012d90 tx=0x7fb6e4019000).stop 2020-11-20T12:15:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 shutdown_connections 2020-11-20T12:15:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb6c0056d20 0x7fb6c00591e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb6e80b3000 0x7fb6e80b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:09.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6d0003730 0x7fb6e814b410 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 --2- 172.21.15.187:0/1737943562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6e80b6840 0x7fb6e80b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.815+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 >> 172.21.15.187:0/1737943562 conn(0x7fb6e80bab30 msgr2=0x7fb6e814c640 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.816+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 shutdown_connections 2020-11-20T12:15:09.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:09.816+0000 7fb6ee879700 1 -- 172.21.15.187:0/1737943562 wait complete. 2020-11-20T12:15:09.821 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:15:09.848 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:15:10.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.089+0000 7f526d674700 1 Processor -- start 2020-11-20T12:15:10.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.089+0000 7f526d674700 1 -- start start 2020-11-20T12:15:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.090+0000 7f526d674700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268065190 0x7f5268065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.090+0000 7f526d674700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f526809ca90 0x7f526809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.090+0000 7f526d674700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52680b13e0 con 0x7f5268065f90 2020-11-20T12:15:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.090+0000 7f526d674700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526814ab30 con 0x7f526809ca90 2020-11-20T12:15:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.090+0000 7f526d674700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5268062c10 con 0x7f5268065190 2020-11-20T12:15:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.091+0000 7f525f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5268065f90 0x7f5268143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46288/0 (socket says 172.21.15.187:46288) 2020-11-20T12:15:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.091+0000 7f525f7fe700 1 -- 172.21.15.187:0/1258295797 learned_addr learned my addr 172.21.15.187:0/1258295797 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.091+0000 7f526ce73700 1 --2- 172.21.15.187:0/1258295797 >> v2:172.21.15.5:3300/0 conn(0x7f526809ca90 0x7f526809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.091+0000 7f5267fff700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268065190 0x7f5268065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.091+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617994521 0 0) 0x7f52680b13e0 con 0x7f5268065f90 2020-11-20T12:15:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.091+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5248003240 con 0x7f5268065f90 2020-11-20T12:15:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1887887840 0 0) 0x7f5248003240 con 0x7f5268065f90 2020-11-20T12:15:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268065190 msgr2=0x7f5268065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268065190 0x7f5268065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 >> v2:172.21.15.5:3300/0 conn(0x7f526809ca90 msgr2=0x7f526809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 --2- 172.21.15.187:0/1258295797 >> v2:172.21.15.5:3300/0 conn(0x7f526809ca90 0x7f526809cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52680b1640 con 0x7f5268065f90 2020-11-20T12:15:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f52500014e0 con 0x7f5268065f90 2020-11-20T12:15:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 >> v1:172.21.15.187:6789/0 conn(0x7f5268065f90 legacy=0x7f5268143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5248003730 0x7f5248003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52480044e0 0x7f5248004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268065190 0x7f5248004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526814ab30 con 0x7f5248003730 2020-11-20T12:15:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268062c10 con 0x7f52480044e0 2020-11-20T12:15:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5248004290 con 0x7f5268065190 2020-11-20T12:15:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.092+0000 7f525f7fe700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52480044e0 0x7f5248004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.093+0000 7f526ce73700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268065190 0x7f5248004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.093+0000 7f5267fff700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5248003730 0x7f5248003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.093+0000 7f525f7fe700 1 -- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268065190 msgr2=0x7f5248004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.093+0000 7f525f7fe700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268065190 0x7f5248004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.093+0000 7f525f7fe700 1 -- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5248003730 msgr2=0x7f5248003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.093+0000 7f525f7fe700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5248003730 0x7f5248003b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.093+0000 7f525f7fe700 1 -- 172.21.15.187:0/1258295797 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248003240 con 0x7f52480044e0 2020-11-20T12:15:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.093+0000 7f525f7fe700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52480044e0 0x7f5248004960 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f525000c410 tx=0x7f52680bb650).ready entity=mon.1 client_cookie=5ea897682237a567 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525000ce80 con 0x7f52480044e0 2020-11-20T12:15:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52500072e0 con 0x7f52480044e0 2020-11-20T12:15:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f52677fe700 1 -- 172.21.15.187:0/1258295797 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250005820 con 0x7f52480044e0 2020-11-20T12:15:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f526d674700 1 -- 172.21.15.187:0/1258295797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52480044e0 msgr2=0x7f5248004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f526d674700 1 --2- 172.21.15.187:0/1258295797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52480044e0 0x7f5248004960 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f525000c410 tx=0x7f52680bb650).stop 2020-11-20T12:15:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f526ce73700 1 -- 172.21.15.187:0/1258295797 reap_dead start 2020-11-20T12:15:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f526d674700 1 -- 172.21.15.187:0/1258295797 shutdown_connections 2020-11-20T12:15:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f526d674700 1 -- 172.21.15.187:0/1258295797 >> 172.21.15.187:0/1258295797 conn(0x7f52680bab30 msgr2=0x7f52680baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f526d674700 1 -- 172.21.15.187:0/1258295797 shutdown_connections 2020-11-20T12:15:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.094+0000 7f526d674700 1 -- 172.21.15.187:0/1258295797 wait complete. 2020-11-20T12:15:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.096+0000 7f526d674700 1 Processor -- start 2020-11-20T12:15:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.096+0000 7f526d674700 1 -- start start 2020-11-20T12:15:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526d674700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5268146800 0x7f526815bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526d674700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268146be0 0x7f5268149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526d674700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52680b2cc0 0x7f5268149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526d674700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52680b46d0 con 0x7f5268146800 2020-11-20T12:15:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526d674700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526815b870 con 0x7f52680b2cc0 2020-11-20T12:15:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526d674700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526815bb10 con 0x7f5268146be0 2020-11-20T12:15:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526ce73700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52680b2cc0 0x7f5268149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526ce73700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52680b2cc0 0x7f5268149950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49698/0 (socket says 172.21.15.187:49698) 2020-11-20T12:15:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.097+0000 7f526ce73700 1 -- 172.21.15.187:0/1187102990 learned_addr learned my addr 172.21.15.187:0/1187102990 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f5267fff700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5268146800 0x7f526815bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f525f7fe700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268146be0 0x7f5268149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f526ce73700 1 -- 172.21.15.187:0/1187102990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268146be0 msgr2=0x7f5268149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f526ce73700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268146be0 0x7f5268149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f526ce73700 1 -- 172.21.15.187:0/1187102990 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5268146800 msgr2=0x7f526815bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f526ce73700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5268146800 0x7f526815bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f526ce73700 1 -- 172.21.15.187:0/1187102990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52680b42e0 con 0x7f52680b2cc0 2020-11-20T12:15:10.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f526ce73700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52680b2cc0 0x7f5268149950 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7f526809ce70 tx=0x7f5260008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f5264ff9700 1 -- 172.21.15.187:0/1187102990 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526000e9d0 con 0x7f52680b2cc0 2020-11-20T12:15:10.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f5264ff9700 1 -- 172.21.15.187:0/1187102990 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526000eb30 con 0x7f52680b2cc0 2020-11-20T12:15:10.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52680b1b80 con 0x7f52680b2cc0 2020-11-20T12:15:10.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.098+0000 7f5264ff9700 1 -- 172.21.15.187:0/1187102990 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5260014430 con 0x7f52680b2cc0 2020-11-20T12:15:10.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.099+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526815c6e0 con 0x7f52680b2cc0 2020-11-20T12:15:10.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.100+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52680b4530 con 0x7f52680b2cc0 2020-11-20T12:15:10.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.100+0000 7f5264ff9700 1 -- 172.21.15.187:0/1187102990 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5260014820 con 0x7f52680b2cc0 2020-11-20T12:15:10.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.101+0000 7f5264ff9700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5244056d20 0x7f52440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.101+0000 7f5264ff9700 1 -- 172.21.15.187:0/1187102990 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5260070350 con 0x7f52680b2cc0 2020-11-20T12:15:10.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.101+0000 7f5267fff700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5244056d20 0x7f52440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.102+0000 7f5267fff700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5244056d20 0x7f52440591e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f5258006d00 tx=0x7f5258008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.105+0000 7f5264ff9700 1 -- 172.21.15.187:0/1187102990 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f52680b4530 con 0x7f52680b2cc0 2020-11-20T12:15:10.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.362+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f52680b4530 con 0x7f52680b2cc0 2020-11-20T12:15:10.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.363+0000 7f5264ff9700 1 -- 172.21.15.187:0/1187102990 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f52680b4530 con 0x7f52680b2cc0 2020-11-20T12:15:10.366 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:10.366 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:15:10.366 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:15:10.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.367 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:15:10.367 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:15:10.367 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.368 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:10.368 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.368 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.368 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.369 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.369 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:15:10.370 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:15:10.370 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.370 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:10.370 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.371 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.371 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.371 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.372 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:15:10.372 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:15:10.372 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.372 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.373 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.373 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.374 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.374 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:15:10.375 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:15:10.375 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.375 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.376 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.376 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.376 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.377 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:15:10.377 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:15:10.377 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.378 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.378 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.378 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.378 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.379 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:15:10.379 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:15:10.380 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.380 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.380 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.380 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.380 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.381 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:15:10.381 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:15:10.382 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.382 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.382 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.382 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.383 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.383 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:15:10.384 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:15:10.384 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.384 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.384 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.385 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.385 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.386 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:15:10.386 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:15:10.386 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.386 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.387 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.387 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.388 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:15:10.388 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:15:10.388 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.388 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.389 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.389 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.389 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.390 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:15:10.390 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:15:10.390 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.391 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:10.391 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:15:10.391 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:15:10.391 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.392 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:15:10.392 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:15:10.393 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.393 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:10.393 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:15:10.393 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:15:10.394 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:15:10.394 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.395 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:15:10.395 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:15:10.395 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.395 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:10.396 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:15:10.396 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:15:10.396 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:15:10.396 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.397 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:15:10.397 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:15:10.397 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.398 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:10.398 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:15:10.398 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:15:10.398 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:15:10.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.399 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:15:10.399 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:15:10.400 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.400 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:15:10.400 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:10.400 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:10.401 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.401 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.402 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:15:10.402 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:15:10.402 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.402 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:15:10.402 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:15:10.403 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.403 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:10.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:10.403 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:15:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:15:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:15:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:15:10.405 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:10.405 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.405 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:10.405 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:15:10.406 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:15:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5244056d20 msgr2=0x7f52440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5244056d20 0x7f52440591e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f5258006d00 tx=0x7f5258008040).stop 2020-11-20T12:15:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52680b2cc0 msgr2=0x7f5268149950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52680b2cc0 0x7f5268149950 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7f526809ce70 tx=0x7f5260008040).stop 2020-11-20T12:15:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 shutdown_connections 2020-11-20T12:15:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5244056d20 0x7f52440591e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5268146800 0x7f526815bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52680b2cc0 0x7f5268149950 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 --2- 172.21.15.187:0/1187102990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5268146be0 0x7f5268149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 >> 172.21.15.187:0/1187102990 conn(0x7f52680bab30 msgr2=0x7f526809cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 shutdown_connections 2020-11-20T12:15:10.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.366+0000 7f526d674700 1 -- 172.21.15.187:0/1187102990 wait complete. 2020-11-20T12:15:10.410 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:15:10.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.642+0000 7f3bbb34a700 1 Processor -- start 2020-11-20T12:15:10.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.643+0000 7f3bbb34a700 1 -- start start 2020-11-20T12:15:10.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.644+0000 7f3bbb34a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4068230 0x7f3bb41482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.644+0000 7f3bbb34a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3bb40b5f50 0x7f3bb40b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.644+0000 7f3bbb34a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb4144b80 con 0x7f3bb40b57b0 2020-11-20T12:15:10.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.644+0000 7f3bbb34a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb41487c0 con 0x7f3bb40b5f50 2020-11-20T12:15:10.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.644+0000 7f3bbb34a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb41489e0 con 0x7f3bb4068230 2020-11-20T12:15:10.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.645+0000 7f3bb9b47700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4068230 0x7f3bb41482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.645+0000 7f3bbab49700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3bb40b5f50 0x7f3bb40b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.645+0000 7f3bb9b47700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4068230 0x7f3bb41482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:50592/0 (socket says 172.21.15.187:50592) 2020-11-20T12:15:10.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.645+0000 7f3bb9b47700 1 -- 172.21.15.187:0/1953275208 learned_addr learned my addr 172.21.15.187:0/1953275208 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:10.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.646+0000 7f3bb9b47700 1 -- 172.21.15.187:0/1953275208 >> v2:172.21.15.5:3300/0 conn(0x7f3bb40b5f50 msgr2=0x7f3bb40b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:10.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.646+0000 7f3bb9b47700 1 --2- 172.21.15.187:0/1953275208 >> v2:172.21.15.5:3300/0 conn(0x7f3bb40b5f50 0x7f3bb40b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.646+0000 7f3bb9b47700 1 -- 172.21.15.187:0/1953275208 >> v1:172.21.15.5:6789/0 conn(0x7f3bb40b57b0 legacy=0x7f3bb4062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.646+0000 7f3bb9346700 1 -- 172.21.15.187:0/1953275208 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562769208 0 0) 0x7f3bb4144b80 con 0x7f3bb40b57b0 2020-11-20T12:15:10.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.646+0000 7f3bb9b47700 1 -- 172.21.15.187:0/1953275208 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb4148c80 con 0x7f3bb4068230 2020-11-20T12:15:10.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.647+0000 7f3bbab49700 1 --2- 172.21.15.187:0/1953275208 >> v2:172.21.15.5:3300/0 conn(0x7f3bb40b5f50 0x7f3bb40b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:15:10.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.647+0000 7f3bb9b47700 1 --2- 172.21.15.187:0/1953275208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4068230 0x7f3bb41482d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f3ba8006fd0 tx=0x7f3ba8006eb0).ready entity=mon.2 client_cookie=1f8cb0fe1d3e4a7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bb9346700 1 -- 172.21.15.187:0/1953275208 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8009810 con 0x7f3bb4068230 2020-11-20T12:15:10.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bb9346700 1 -- 172.21.15.187:0/1953275208 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba8009970 con 0x7f3bb4068230 2020-11-20T12:15:10.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bb9346700 1 -- 172.21.15.187:0/1953275208 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8009cf0 con 0x7f3bb4068230 2020-11-20T12:15:10.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bbb34a700 1 -- 172.21.15.187:0/1953275208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4068230 msgr2=0x7f3bb41482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/1953275208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4068230 0x7f3bb41482d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f3ba8006fd0 tx=0x7f3ba8006eb0).stop 2020-11-20T12:15:10.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bbb34a700 1 -- 172.21.15.187:0/1953275208 shutdown_connections 2020-11-20T12:15:10.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/1953275208 >> v2:172.21.15.5:3300/0 conn(0x7f3bb40b5f50 0x7f3bb40b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/1953275208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb4068230 0x7f3bb41482d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bbb34a700 1 -- 172.21.15.187:0/1953275208 >> 172.21.15.187:0/1953275208 conn(0x7f3bb40bab30 msgr2=0x7f3bb40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:10.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bbb34a700 1 -- 172.21.15.187:0/1953275208 shutdown_connections 2020-11-20T12:15:10.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.648+0000 7f3bbb34a700 1 -- 172.21.15.187:0/1953275208 wait complete. 2020-11-20T12:15:10.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.650+0000 7f3bbb34a700 1 Processor -- start 2020-11-20T12:15:10.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.651+0000 7f3bbb34a700 1 -- start start 2020-11-20T12:15:10.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bbb34a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4068230 0x7f3bb408f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bbb34a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb40b57b0 0x7f3bb4155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bbb34a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb40b5f50 0x7f3bb4155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bbb34a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4156020 con 0x7f3bb40b5f50 2020-11-20T12:15:10.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bbb34a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4090980 con 0x7f3bb4068230 2020-11-20T12:15:10.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bbb34a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb41481f0 con 0x7f3bb40b57b0 2020-11-20T12:15:10.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bba348700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4068230 0x7f3bb408f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bba348700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4068230 0x7f3bb408f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49708/0 (socket says 172.21.15.187:49708) 2020-11-20T12:15:10.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.652+0000 7f3bba348700 1 -- 172.21.15.187:0/2883595113 learned_addr learned my addr 172.21.15.187:0/2883595113 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:10.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3bbab49700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb40b5f50 0x7f3bb4155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3bb9b47700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb40b57b0 0x7f3bb4155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3bba348700 1 -- 172.21.15.187:0/2883595113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb40b57b0 msgr2=0x7f3bb4155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3bba348700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb40b57b0 0x7f3bb4155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3bba348700 1 -- 172.21.15.187:0/2883595113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb40b5f50 msgr2=0x7f3bb4155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3bba348700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb40b5f50 0x7f3bb4155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3bba348700 1 -- 172.21.15.187:0/2883595113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb408f540 con 0x7f3bb4068230 2020-11-20T12:15:10.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3bba348700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4068230 0x7f3bb408f9e0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f3bb000c6e0 tx=0x7f3bb0007a70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.653+0000 7f3ba67fc700 1 -- 172.21.15.187:0/2883595113 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb000e560 con 0x7f3bb4068230 2020-11-20T12:15:10.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.654+0000 7f3ba67fc700 1 -- 172.21.15.187:0/2883595113 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb000e6c0 con 0x7f3bb4068230 2020-11-20T12:15:10.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.654+0000 7f3ba67fc700 1 -- 172.21.15.187:0/2883595113 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb000ea80 con 0x7f3bb4068230 2020-11-20T12:15:10.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.654+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb4147d50 con 0x7f3bb4068230 2020-11-20T12:15:10.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.654+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb4159ad0 con 0x7f3bb4068230 2020-11-20T12:15:10.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.655+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb4148050 con 0x7f3bb4068230 2020-11-20T12:15:10.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.656+0000 7f3ba67fc700 1 -- 172.21.15.187:0/2883595113 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3bb0010070 con 0x7f3bb4068230 2020-11-20T12:15:10.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.656+0000 7f3ba67fc700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b8c056d20 0x7f3b8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.657+0000 7f3ba67fc700 1 -- 172.21.15.187:0/2883595113 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3bb006f490 con 0x7f3bb4068230 2020-11-20T12:15:10.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.657+0000 7f3bb9b47700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b8c056d20 0x7f3b8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.657+0000 7f3bb9b47700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b8c056d20 0x7f3b8c0591e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f3bb4146c40 tx=0x7f3ba8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.660+0000 7f3ba67fc700 1 -- 172.21.15.187:0/2883595113 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3bb4148050 con 0x7f3bb4068230 2020-11-20T12:15:10.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.849+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bb4148050 con 0x7f3bb4068230 2020-11-20T12:15:10.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.851+0000 7f3ba67fc700 1 -- 172.21.15.187:0/2883595113 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+13629 (secure 0 0 0) 0x7f3bb4148050 con 0x7f3bb4068230 2020-11-20T12:15:10.854 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:10.854 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":102,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:15:09.796672+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:15:10.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b8c056d20 msgr2=0x7f3b8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b8c056d20 0x7f3b8c0591e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f3bb4146c40 tx=0x7f3ba8015000).stop 2020-11-20T12:15:10.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4068230 msgr2=0x7f3bb408f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4068230 0x7f3bb408f9e0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f3bb000c6e0 tx=0x7f3bb0007a70).stop 2020-11-20T12:15:10.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 shutdown_connections 2020-11-20T12:15:10.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b8c056d20 0x7f3b8c0591e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bb40b5f50 0x7f3bb4155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bb4068230 0x7f3bb408f9e0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 --2- 172.21.15.187:0/2883595113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bb40b57b0 0x7f3bb4155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 >> 172.21.15.187:0/2883595113 conn(0x7f3bb40bab30 msgr2=0x7f3bb4065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:10.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 shutdown_connections 2020-11-20T12:15:10.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:10.853+0000 7f3bbb34a700 1 -- 172.21.15.187:0/2883595113 wait complete. 2020-11-20T12:15:10.881 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:15:10.882 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:15:11.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.121+0000 7f63ba165700 1 Processor -- start 2020-11-20T12:15:11.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.121+0000 7f63ba165700 1 -- start start 2020-11-20T12:15:11.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.123+0000 7f63ba165700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b4060330 0x7f63b40662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.123+0000 7f63ba165700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f63b40676a0 0x7f63b40b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.123+0000 7f63ba165700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b409b540 con 0x7f63b4060710 2020-11-20T12:15:11.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.123+0000 7f63ba165700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b40b7da0 con 0x7f63b40676a0 2020-11-20T12:15:11.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.123+0000 7f63ba165700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40b8040 con 0x7f63b4060330 2020-11-20T12:15:11.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.124+0000 7f63b9163700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b4060330 0x7f63b40662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.124+0000 7f63b9964700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f63b40676a0 0x7f63b40b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.124+0000 7f63b9163700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b4060330 0x7f63b40662a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:50604/0 (socket says 172.21.15.187:50604) 2020-11-20T12:15:11.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.124+0000 7f63b9163700 1 -- 172.21.15.187:0/1685291962 learned_addr learned my addr 172.21.15.187:0/1685291962 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:11.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.124+0000 7f63b9964700 1 -- 172.21.15.187:0/1685291962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b4060330 msgr2=0x7f63b40662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:11.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.124+0000 7f63b9964700 1 --2- 172.21.15.187:0/1685291962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b4060330 0x7f63b40662a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.124+0000 7f63b9964700 1 -- 172.21.15.187:0/1685291962 >> v1:172.21.15.5:6789/0 conn(0x7f63b4060710 legacy=0x7f63b41480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.125+0000 7f63b9964700 1 -- 172.21.15.187:0/1685291962 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40b82e0 con 0x7f63b40676a0 2020-11-20T12:15:11.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.125+0000 7f63b9964700 1 --2- 172.21.15.187:0/1685291962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b40676a0 0x7f63b40b6b00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7f63ac00b9a0 tx=0x7f63ac006df0).ready entity=mon.1 client_cookie=77c08ec726bb1f40 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:11.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.125+0000 7f63a3fff700 1 -- 172.21.15.187:0/1685291962 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac009800 con 0x7f63b40676a0 2020-11-20T12:15:11.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.125+0000 7f63a3fff700 1 -- 172.21.15.187:0/1685291962 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac009960 con 0x7f63b40676a0 2020-11-20T12:15:11.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.125+0000 7f63a3fff700 1 -- 172.21.15.187:0/1685291962 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac009ce0 con 0x7f63b40676a0 2020-11-20T12:15:11.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.126+0000 7f63ba165700 1 -- 172.21.15.187:0/1685291962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b40676a0 msgr2=0x7f63b40b6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.126+0000 7f63ba165700 1 --2- 172.21.15.187:0/1685291962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b40676a0 0x7f63b40b6b00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7f63ac00b9a0 tx=0x7f63ac006df0).stop 2020-11-20T12:15:11.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.126+0000 7f63ba165700 1 -- 172.21.15.187:0/1685291962 shutdown_connections 2020-11-20T12:15:11.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.126+0000 7f63ba165700 1 --2- 172.21.15.187:0/1685291962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b40676a0 0x7f63b40b6b00 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.126+0000 7f63ba165700 1 --2- 172.21.15.187:0/1685291962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b4060330 0x7f63b40662a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.126+0000 7f63ba165700 1 -- 172.21.15.187:0/1685291962 >> 172.21.15.187:0/1685291962 conn(0x7f63b40bbc20 msgr2=0x7f63b40bc030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:11.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.126+0000 7f63ba165700 1 -- 172.21.15.187:0/1685291962 shutdown_connections 2020-11-20T12:15:11.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.126+0000 7f63ba165700 1 -- 172.21.15.187:0/1685291962 wait complete. 2020-11-20T12:15:11.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.128+0000 7f63ba165700 1 Processor -- start 2020-11-20T12:15:11.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.128+0000 7f63ba165700 1 -- start start 2020-11-20T12:15:11.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63ba165700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b4060330 0x7f63b40a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63ba165700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b4060710 0x7f63b41473e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63ba165700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b40676a0 0x7f63b409ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63ba165700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4158ca0 con 0x7f63b4060710 2020-11-20T12:15:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63ba165700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b414b230 con 0x7f63b4060330 2020-11-20T12:15:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63ba165700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b41478d0 con 0x7f63b40676a0 2020-11-20T12:15:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63b9163700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b4060330 0x7f63b40a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63b9163700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b4060330 0x7f63b40a1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49722/0 (socket says 172.21.15.187:49722) 2020-11-20T12:15:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.129+0000 7f63b9163700 1 -- 172.21.15.187:0/2923230306 learned_addr learned my addr 172.21.15.187:0/2923230306 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63b9964700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b40676a0 0x7f63b409ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63b8962700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b4060710 0x7f63b41473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63b9163700 1 -- 172.21.15.187:0/2923230306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b40676a0 msgr2=0x7f63b409ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63b9163700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b40676a0 0x7f63b409ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63b9163700 1 -- 172.21.15.187:0/2923230306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b4060710 msgr2=0x7f63b41473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63b9163700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b4060710 0x7f63b41473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63b9163700 1 -- 172.21.15.187:0/2923230306 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4158950 con 0x7f63b4060330 2020-11-20T12:15:11.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63b9163700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b4060330 0x7f63b40a1480 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f63b000ef20 tx=0x7f63b0009290).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.130+0000 7f63a17fa700 1 -- 172.21.15.187:0/2923230306 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b000cc70 con 0x7f63b4060330 2020-11-20T12:15:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.131+0000 7f63a17fa700 1 -- 172.21.15.187:0/2923230306 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b000cdd0 con 0x7f63b4060330 2020-11-20T12:15:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.131+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b409f0c0 con 0x7f63b4060330 2020-11-20T12:15:11.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.131+0000 7f63a17fa700 1 -- 172.21.15.187:0/2923230306 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b0012620 con 0x7f63b4060330 2020-11-20T12:15:11.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.131+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b408ec60 con 0x7f63b4060330 2020-11-20T12:15:11.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.132+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b409acf0 con 0x7f63b4060330 2020-11-20T12:15:11.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.132+0000 7f63a17fa700 1 -- 172.21.15.187:0/2923230306 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f63b0022070 con 0x7f63b4060330 2020-11-20T12:15:11.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.133+0000 7f63a17fa700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f638c056d20 0x7f638c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.133+0000 7f63a17fa700 1 -- 172.21.15.187:0/2923230306 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f63b00705b0 con 0x7f63b4060330 2020-11-20T12:15:11.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.133+0000 7f63b8962700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f638c056d20 0x7f638c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.134+0000 7f63b8962700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f638c056d20 0x7f638c0591e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f63b40a1410 tx=0x7f63a4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:11.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.137+0000 7f63a17fa700 1 -- 172.21.15.187:0/2923230306 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f63b0045d30 con 0x7f63b4060330 2020-11-20T12:15:11.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.326+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f63b4158ae0 con 0x7f63b4060330 2020-11-20T12:15:11.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.327+0000 7f63a17fa700 1 -- 172.21.15.187:0/2923230306 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v102) v1 ==== 110+0+10 (secure 0 0 0) 0x7f63b0016020 con 0x7f63b4060330 2020-11-20T12:15:11.330 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:15:11.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f638c056d20 msgr2=0x7f638c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f638c056d20 0x7f638c0591e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f63b40a1410 tx=0x7f63a4009040).stop 2020-11-20T12:15:11.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b4060330 msgr2=0x7f63b40a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b4060330 0x7f63b40a1480 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f63b000ef20 tx=0x7f63b0009290).stop 2020-11-20T12:15:11.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 shutdown_connections 2020-11-20T12:15:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f638c056d20 0x7f638c0591e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f63b40676a0 0x7f63b409ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b4060330 0x7f63b40a1480 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 --2- 172.21.15.187:0/2923230306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f63b4060710 0x7f63b41473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 >> 172.21.15.187:0/2923230306 conn(0x7f63b40bbc20 msgr2=0x7f63b40b6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:11.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 shutdown_connections 2020-11-20T12:15:11.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.329+0000 7f63ba165700 1 -- 172.21.15.187:0/2923230306 wait complete. 2020-11-20T12:15:11.354 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:15:11.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.598+0000 7f0627fff700 1 Processor -- start 2020-11-20T12:15:11.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.598+0000 7f0627fff700 1 -- start start 2020-11-20T12:15:11.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.599+0000 7f0627fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0628065190 0x7f0628065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.599+0000 7f0627fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f062809ca90 0x7f062809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.599+0000 7f0627fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06280b13e0 con 0x7f0628065f90 2020-11-20T12:15:11.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.599+0000 7f0627fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f062814ab30 con 0x7f062809ca90 2020-11-20T12:15:11.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.599+0000 7f0627fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0628062c10 con 0x7f0628065190 2020-11-20T12:15:11.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.600+0000 7f0626ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0628065190 0x7f0628065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.600+0000 7f061e7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0628065f90 0x7f0628143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46338/0 (socket says 172.21.15.187:46338) 2020-11-20T12:15:11.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.600+0000 7f061e7fc700 1 -- 172.21.15.187:0/3798261704 learned_addr learned my addr 172.21.15.187:0/3798261704 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:11.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.600+0000 7f0626ffd700 1 -- 172.21.15.187:0/3798261704 >> v2:172.21.15.5:3300/0 conn(0x7f062809ca90 msgr2=0x7f062809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:11.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.600+0000 7f0626ffd700 1 --2- 172.21.15.187:0/3798261704 >> v2:172.21.15.5:3300/0 conn(0x7f062809ca90 0x7f062809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:11.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.600+0000 7f0626ffd700 1 -- 172.21.15.187:0/3798261704 >> v1:172.21.15.187:6789/0 conn(0x7f0628065f90 legacy=0x7f0628143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.600+0000 7f0626ffd700 1 -- 172.21.15.187:0/3798261704 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06280b1640 con 0x7f0628065190 2020-11-20T12:15:11.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.600+0000 7f06267fc700 1 -- 172.21.15.187:0/3798261704 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998349620 0 0) 0x7f06280b13e0 con 0x7f0628065f90 2020-11-20T12:15:11.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f0626ffd700 1 --2- 172.21.15.187:0/3798261704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f0628065570 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f062805c8b0 tx=0x7f0618006df0).ready entity=mon.1 client_cookie=dd3ff0efaccb72c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:11.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f06267fc700 1 -- 172.21.15.187:0/3798261704 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0618009840 con 0x7f0628065190 2020-11-20T12:15:11.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f06267fc700 1 -- 172.21.15.187:0/3798261704 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06180099a0 con 0x7f0628065190 2020-11-20T12:15:11.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f06267fc700 1 -- 172.21.15.187:0/3798261704 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0618009d20 con 0x7f0628065190 2020-11-20T12:15:11.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f0627fff700 1 -- 172.21.15.187:0/3798261704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 msgr2=0x7f0628065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f0627fff700 1 --2- 172.21.15.187:0/3798261704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f0628065570 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f062805c8b0 tx=0x7f0618006df0).stop 2020-11-20T12:15:11.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f0627fff700 1 -- 172.21.15.187:0/3798261704 shutdown_connections 2020-11-20T12:15:11.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f0627fff700 1 --2- 172.21.15.187:0/3798261704 >> v2:172.21.15.5:3300/0 conn(0x7f062809ca90 0x7f062809cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:11.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.601+0000 7f0627fff700 1 --2- 172.21.15.187:0/3798261704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f0628065570 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.602+0000 7f0627fff700 1 -- 172.21.15.187:0/3798261704 >> 172.21.15.187:0/3798261704 conn(0x7f06280bab30 msgr2=0x7f06280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:11.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.602+0000 7f0627fff700 1 -- 172.21.15.187:0/3798261704 shutdown_connections 2020-11-20T12:15:11.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.602+0000 7f0627fff700 1 -- 172.21.15.187:0/3798261704 wait complete. 2020-11-20T12:15:11.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.603+0000 7f0627fff700 1 Processor -- start 2020-11-20T12:15:11.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.603+0000 7f0627fff700 1 -- start start 2020-11-20T12:15:11.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.604+0000 7f0627fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f062815b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.604+0000 7f0627fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0628065f90 0x7f06281450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.604+0000 7f0627fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f062809ca90 0x7f0628143160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.604+0000 7f0627fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0628147ee0 con 0x7f0628065f90 2020-11-20T12:15:11.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.604+0000 7f0627fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062815be80 con 0x7f0628065190 2020-11-20T12:15:11.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.604+0000 7f0627fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062815c120 con 0x7f062809ca90 2020-11-20T12:15:11.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.604+0000 7f0626ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f062815b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f0626ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f062815b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49736/0 (socket says 172.21.15.187:49736) 2020-11-20T12:15:11.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f0626ffd700 1 -- 172.21.15.187:0/639726997 learned_addr learned my addr 172.21.15.187:0/639726997 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:11.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f061e7fc700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0628065f90 0x7f06281450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f0626ffd700 1 -- 172.21.15.187:0/639726997 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f062809ca90 msgr2=0x7f0628143160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f0626ffd700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f062809ca90 0x7f0628143160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:11.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f0626ffd700 1 -- 172.21.15.187:0/639726997 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0628065f90 msgr2=0x7f06281450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f0626ffd700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0628065f90 0x7f06281450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f0626ffd700 1 -- 172.21.15.187:0/639726997 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062815c3c0 con 0x7f0628065190 2020-11-20T12:15:11.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f0626ffd700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f062815b850 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f0618000dd0 tx=0x7f0618015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:11.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.605+0000 7f061f7fe700 1 -- 172.21.15.187:0/639726997 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061801e460 con 0x7f0628065190 2020-11-20T12:15:11.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.606+0000 7f061f7fe700 1 -- 172.21.15.187:0/639726997 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061801e5c0 con 0x7f0628065190 2020-11-20T12:15:11.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.606+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06281455a0 con 0x7f0628065190 2020-11-20T12:15:11.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.606+0000 7f061f7fe700 1 -- 172.21.15.187:0/639726997 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061801e980 con 0x7f0628065190 2020-11-20T12:15:11.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.606+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06281477a0 con 0x7f0628065190 2020-11-20T12:15:11.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.607+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f062815c780 con 0x7f0628065190 2020-11-20T12:15:11.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.607+0000 7f061f7fe700 1 -- 172.21.15.187:0/639726997 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0618025030 con 0x7f0628065190 2020-11-20T12:15:11.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.608+0000 7f061f7fe700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0604056d50 0x7f0604059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:11.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.608+0000 7f061f7fe700 1 -- 172.21.15.187:0/639726997 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0618071a50 con 0x7f0628065190 2020-11-20T12:15:11.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.608+0000 7f061e7fc700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0604056d50 0x7f0604059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:11.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.609+0000 7f061e7fc700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0604056d50 0x7f0604059210 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f062809c7c0 tx=0x7f0610009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:11.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.612+0000 7f061f7fe700 1 -- 172.21.15.187:0/639726997 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f06180470c0 con 0x7f0628065190 2020-11-20T12:15:11.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.800+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f062815c550 con 0x7f0628065190 2020-11-20T12:15:11.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.801+0000 7f061f7fe700 1 -- 172.21.15.187:0/639726997 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v102) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0618023070 con 0x7f0628065190 2020-11-20T12:15:11.804 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:15:11.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0604056d50 msgr2=0x7f0604059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0604056d50 0x7f0604059210 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f062809c7c0 tx=0x7f0610009040).stop 2020-11-20T12:15:11.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 msgr2=0x7f062815b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:11.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f062815b850 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f0618000dd0 tx=0x7f0618015000).stop 2020-11-20T12:15:11.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 shutdown_connections 2020-11-20T12:15:11.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0604056d50 0x7f0604059210 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f062809ca90 0x7f0628143160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:11.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628065190 0x7f062815b850 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 --2- 172.21.15.187:0/639726997 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0628065f90 0x7f06281450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:11.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.803+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 >> 172.21.15.187:0/639726997 conn(0x7f06280bab30 msgr2=0x7f06280ba270 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:11.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.804+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 shutdown_connections 2020-11-20T12:15:11.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:11.804+0000 7f0627fff700 1 -- 172.21.15.187:0/639726997 wait complete. 2020-11-20T12:15:11.830 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:15:12.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.085+0000 7ff2b0bf3700 1 Processor -- start 2020-11-20T12:15:12.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.085+0000 7ff2b0bf3700 1 -- start start 2020-11-20T12:15:12.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.086+0000 7ff2b0bf3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0652e0 0x7ff2ac0656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.086+0000 7ff2b0bf3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff2ac0b1590 0x7ff2ac14a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.086+0000 7ff2b0bf3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac149dd0 con 0x7ff2ac0b7300 2020-11-20T12:15:12.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.086+0000 7ff2b0bf3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2ac14a070 con 0x7ff2ac0b1590 2020-11-20T12:15:12.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.086+0000 7ff2b0bf3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac14a310 con 0x7ff2ac0652e0 2020-11-20T12:15:12.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.087+0000 7ff2a1d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff2ac0b1590 0x7ff2ac14a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.087+0000 7ff2a1d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff2ac0b1590 0x7ff2ac14a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49748/0 (socket says 172.21.15.187:49748) 2020-11-20T12:15:12.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.087+0000 7ff2a1d9b700 1 -- 172.21.15.187:0/1924596630 learned_addr learned my addr 172.21.15.187:0/1924596630 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:12.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.088+0000 7ff2aa59c700 1 --2- 172.21.15.187:0/1924596630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0652e0 0x7ff2ac0656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.088+0000 7ff2a1d9b700 1 -- 172.21.15.187:0/1924596630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0652e0 msgr2=0x7ff2ac0656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:12.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.088+0000 7ff2a1d9b700 1 --2- 172.21.15.187:0/1924596630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0652e0 0x7ff2ac0656c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.088+0000 7ff2a1d9b700 1 -- 172.21.15.187:0/1924596630 >> v1:172.21.15.5:6789/0 conn(0x7ff2ac0b7300 legacy=0x7ff2ac143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.088+0000 7ff2a1d9b700 1 -- 172.21.15.187:0/1924596630 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac14aca0 con 0x7ff2ac0b1590 2020-11-20T12:15:12.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.088+0000 7ff2a1d9b700 1 --2- 172.21.15.187:0/1924596630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b1590 0x7ff2ac14a7b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7ff29400ba70 tx=0x7ff294006e50).ready entity=mon.1 client_cookie=a56b6ae348c1c57c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:12.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.088+0000 7ff2a9d9b700 1 -- 172.21.15.187:0/1924596630 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294006030 con 0x7ff2ac0b1590 2020-11-20T12:15:12.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2a9d9b700 1 -- 172.21.15.187:0/1924596630 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff294006190 con 0x7ff2ac0b1590 2020-11-20T12:15:12.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2a9d9b700 1 -- 172.21.15.187:0/1924596630 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294006510 con 0x7ff2ac0b1590 2020-11-20T12:15:12.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/1924596630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b1590 msgr2=0x7ff2ac14a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/1924596630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b1590 0x7ff2ac14a7b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7ff29400ba70 tx=0x7ff294006e50).stop 2020-11-20T12:15:12.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/1924596630 shutdown_connections 2020-11-20T12:15:12.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/1924596630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0652e0 0x7ff2ac0656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/1924596630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b1590 0x7ff2ac14a7b0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/1924596630 >> 172.21.15.187:0/1924596630 conn(0x7ff2ac0bacd0 msgr2=0x7ff2ac0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:12.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/1924596630 shutdown_connections 2020-11-20T12:15:12.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.089+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/1924596630 wait complete. 2020-11-20T12:15:12.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.091+0000 7ff2b0bf3700 1 Processor -- start 2020-11-20T12:15:12.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.091+0000 7ff2b0bf3700 1 -- start start 2020-11-20T12:15:12.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2b0bf3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2ac0652e0 0x7ff2ac142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2b0bf3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0b1590 0x7ff2ac148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2b0bf3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b7300 0x7ff2ac158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2b0bf3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac15b9d0 con 0x7ff2ac0652e0 2020-11-20T12:15:12.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2b0bf3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac0b3880 con 0x7ff2ac0b7300 2020-11-20T12:15:12.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2b0bf3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac142850 con 0x7ff2ac0b1590 2020-11-20T12:15:12.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2aad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b7300 0x7ff2ac158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2aad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b7300 0x7ff2ac158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49752/0 (socket says 172.21.15.187:49752) 2020-11-20T12:15:12.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.092+0000 7ff2aad9d700 1 -- 172.21.15.187:0/3091966688 learned_addr learned my addr 172.21.15.187:0/3091966688 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:12.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2a1d9b700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0b1590 0x7ff2ac148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2aad9d700 1 -- 172.21.15.187:0/3091966688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0b1590 msgr2=0x7ff2ac148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2aad9d700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0b1590 0x7ff2ac148580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2aad9d700 1 -- 172.21.15.187:0/3091966688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2ac0652e0 msgr2=0x7ff2ac142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2aad9d700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2ac0652e0 0x7ff2ac142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:12.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2aad9d700 1 -- 172.21.15.187:0/3091966688 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac08ea90 con 0x7ff2ac0b7300 2020-11-20T12:15:12.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2a1d9b700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0b1590 0x7ff2ac148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:12.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2aad9d700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b7300 0x7ff2ac158330 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7ff2a400ca90 tx=0x7ff2a4001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:12.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2a2ffd700 1 -- 172.21.15.187:0/3091966688 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a400a730 con 0x7ff2ac0b7300 2020-11-20T12:15:12.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2a2ffd700 1 -- 172.21.15.187:0/3091966688 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a400a890 con 0x7ff2ac0b7300 2020-11-20T12:15:12.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2a2ffd700 1 -- 172.21.15.187:0/3091966688 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a400ac30 con 0x7ff2ac0b7300 2020-11-20T12:15:12.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.093+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac0684d0 con 0x7ff2ac0b7300 2020-11-20T12:15:12.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.094+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ac148c30 con 0x7ff2ac0b7300 2020-11-20T12:15:12.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.095+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac15bb90 con 0x7ff2ac0b7300 2020-11-20T12:15:12.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.095+0000 7ff2a2ffd700 1 -- 172.21.15.187:0/3091966688 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff2a4010070 con 0x7ff2ac0b7300 2020-11-20T12:15:12.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.096+0000 7ff2a2ffd700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff288056d20 0x7ff2880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.096+0000 7ff2a2ffd700 1 -- 172.21.15.187:0/3091966688 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff2a406f990 con 0x7ff2ac0b7300 2020-11-20T12:15:12.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.096+0000 7ff2aa59c700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff288056d20 0x7ff2880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.097+0000 7ff2aa59c700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff288056d20 0x7ff2880591e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7ff29c00b740 tx=0x7ff29c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:12.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.100+0000 7ff2a2ffd700 1 -- 172.21.15.187:0/3091966688 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff2a4045110 con 0x7ff2ac0b7300 2020-11-20T12:15:12.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.289+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff2ac0686c0 con 0x7ff2ac0b7300 2020-11-20T12:15:12.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.290+0000 7ff2a2ffd700 1 -- 172.21.15.187:0/3091966688 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v102) v1 ==== 84+0+143 (secure 0 0 0) 0x7ff2a4048730 con 0x7ff2ac0b7300 2020-11-20T12:15:12.293 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:12.293 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:15:12.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:12.294 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:15:12.294 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:15:12.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:12.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:12.295 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:15:12.295 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:15:12.295 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:12.295 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:15:12.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.292+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff288056d20 msgr2=0x7ff2880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.292+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff288056d20 0x7ff2880591e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7ff29c00b740 tx=0x7ff29c008040).stop 2020-11-20T12:15:12.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.292+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b7300 msgr2=0x7ff2ac158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.292+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b7300 0x7ff2ac158330 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7ff2a400ca90 tx=0x7ff2a4001360).stop 2020-11-20T12:15:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.293+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 shutdown_connections 2020-11-20T12:15:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.293+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff288056d20 0x7ff2880591e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.293+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2ac0652e0 0x7ff2ac142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:12.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.293+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2ac0b7300 0x7ff2ac158330 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.293+0000 7ff2b0bf3700 1 --2- 172.21.15.187:0/3091966688 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff2ac0b1590 0x7ff2ac148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.293+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 >> 172.21.15.187:0/3091966688 conn(0x7ff2ac0bacd0 msgr2=0x7ff2ac14a510 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:12.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.293+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 shutdown_connections 2020-11-20T12:15:12.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.293+0000 7ff2b0bf3700 1 -- 172.21.15.187:0/3091966688 wait complete. 2020-11-20T12:15:12.320 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:15:12.320 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:15:12.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.565+0000 7fb059863700 1 Processor -- start 2020-11-20T12:15:12.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.565+0000 7fb059863700 1 -- start start 2020-11-20T12:15:12.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb059863700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb0540ba210 0x7fb05414da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb059863700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb054065300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb059863700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb0540657f0 0x7fb0540b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb059863700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0540baa70 con 0x7fb0540ba210 2020-11-20T12:15:12.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb059863700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05414d090 con 0x7fb0540657f0 2020-11-20T12:15:12.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb059863700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb05414d330 con 0x7fb0540ba5f0 2020-11-20T12:15:12.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb0537fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb0540657f0 0x7fb0540b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb0537fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb0540657f0 0x7fb0540b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49762/0 (socket says 172.21.15.187:49762) 2020-11-20T12:15:12.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.567+0000 7fb0537fe700 1 -- 172.21.15.187:0/4240486728 learned_addr learned my addr 172.21.15.187:0/4240486728 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:12.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb04a7fc700 1 --2- 172.21.15.187:0/4240486728 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb054065300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb052ffd700 1 --2- 172.21.15.187:0/4240486728 >> v2:172.21.15.5:3300/0 conn(0x7fb0540ba210 0x7fb05414da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb0537fe700 1 -- 172.21.15.187:0/4240486728 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 msgr2=0x7fb054065300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:12.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb0537fe700 1 --2- 172.21.15.187:0/4240486728 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb054065300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb0537fe700 1 -- 172.21.15.187:0/4240486728 >> v2:172.21.15.5:3300/0 conn(0x7fb0540ba210 msgr2=0x7fb05414da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:12.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb0537fe700 1 --2- 172.21.15.187:0/4240486728 >> v2:172.21.15.5:3300/0 conn(0x7fb0540ba210 0x7fb05414da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb0537fe700 1 -- 172.21.15.187:0/4240486728 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05414d5d0 con 0x7fb0540657f0 2020-11-20T12:15:12.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb052ffd700 1 --2- 172.21.15.187:0/4240486728 >> v2:172.21.15.5:3300/0 conn(0x7fb0540ba210 0x7fb05414da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:12.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb0537fe700 1 --2- 172.21.15.187:0/4240486728 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540657f0 0x7fb0540b6cf0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7fb0540682c0 tx=0x7fb04c009290).ready entity=mon.1 client_cookie=3497413b9a1b06f1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:12.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb0527fc700 1 -- 172.21.15.187:0/4240486728 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c00bbc0 con 0x7fb0540657f0 2020-11-20T12:15:12.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.568+0000 7fb0527fc700 1 -- 172.21.15.187:0/4240486728 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c00bd20 con 0x7fb0540657f0 2020-11-20T12:15:12.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb0527fc700 1 -- 172.21.15.187:0/4240486728 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c011610 con 0x7fb0540657f0 2020-11-20T12:15:12.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 -- 172.21.15.187:0/4240486728 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540657f0 msgr2=0x7fb0540b6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 --2- 172.21.15.187:0/4240486728 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540657f0 0x7fb0540b6cf0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7fb0540682c0 tx=0x7fb04c009290).stop 2020-11-20T12:15:12.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 -- 172.21.15.187:0/4240486728 shutdown_connections 2020-11-20T12:15:12.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 --2- 172.21.15.187:0/4240486728 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540657f0 0x7fb0540b6cf0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 --2- 172.21.15.187:0/4240486728 >> v2:172.21.15.5:3300/0 conn(0x7fb0540ba210 0x7fb05414da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 --2- 172.21.15.187:0/4240486728 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb054065300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 -- 172.21.15.187:0/4240486728 >> 172.21.15.187:0/4240486728 conn(0x7fb0540bacf0 msgr2=0x7fb0540bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:12.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 -- 172.21.15.187:0/4240486728 shutdown_connections 2020-11-20T12:15:12.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.569+0000 7fb059863700 1 -- 172.21.15.187:0/4240486728 wait complete. 2020-11-20T12:15:12.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.571+0000 7fb059863700 1 Processor -- start 2020-11-20T12:15:12.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.571+0000 7fb059863700 1 -- start start 2020-11-20T12:15:12.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb059863700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0540657f0 0x7fb05415d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb059863700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540ba210 0x7fb0541527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb059863700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb0540b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb059863700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05409fca0 con 0x7fb0540657f0 2020-11-20T12:15:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb059863700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05409ff40 con 0x7fb0540ba210 2020-11-20T12:15:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb059863700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0540a01e0 con 0x7fb0540ba5f0 2020-11-20T12:15:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb04a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540ba210 0x7fb0541527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb04a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540ba210 0x7fb0541527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49766/0 (socket says 172.21.15.187:49766) 2020-11-20T12:15:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.572+0000 7fb04a7fc700 1 -- 172.21.15.187:0/2233645839 learned_addr learned my addr 172.21.15.187:0/2233645839 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:12.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb052ffd700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0540657f0 0x7fb05415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb0537fe700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb0540b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb04a7fc700 1 -- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 msgr2=0x7fb0540b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb04a7fc700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb0540b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb04a7fc700 1 -- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0540657f0 msgr2=0x7fb05415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:12.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb04a7fc700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0540657f0 0x7fb05415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:12.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb04a7fc700 1 -- 172.21.15.187:0/2233645839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb054152c90 con 0x7fb0540ba210 2020-11-20T12:15:12.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb0537fe700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb0540b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:12.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.573+0000 7fb04a7fc700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540ba210 0x7fb0541527a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7fb03c00db60 tx=0x7fb03c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:12.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.574+0000 7fb04b7fe700 1 -- 172.21.15.187:0/2233645839 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c009790 con 0x7fb0540ba210 2020-11-20T12:15:12.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.574+0000 7fb04b7fe700 1 -- 172.21.15.187:0/2233645839 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c0098f0 con 0x7fb0540ba210 2020-11-20T12:15:12.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.574+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05409e100 con 0x7fb0540ba210 2020-11-20T12:15:12.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.574+0000 7fb04b7fe700 1 -- 172.21.15.187:0/2233645839 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c009c90 con 0x7fb0540ba210 2020-11-20T12:15:12.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.574+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0540a08c0 con 0x7fb0540ba210 2020-11-20T12:15:12.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.575+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0540a03a0 con 0x7fb0540ba210 2020-11-20T12:15:12.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.575+0000 7fb04b7fe700 1 -- 172.21.15.187:0/2233645839 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb03c013480 con 0x7fb0540ba210 2020-11-20T12:15:12.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.577+0000 7fb04b7fe700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb030056d20 0x7fb0300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.577+0000 7fb04b7fe700 1 -- 172.21.15.187:0/2233645839 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb03c070220 con 0x7fb0540ba210 2020-11-20T12:15:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.577+0000 7fb052ffd700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb030056d20 0x7fb0300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.578+0000 7fb052ffd700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb030056d20 0x7fb0300591e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fb04400a8c0 tx=0x7fb044008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.581+0000 7fb04b7fe700 1 -- 172.21.15.187:0/2233645839 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb03c045940 con 0x7fb0540ba210 2020-11-20T12:15:12.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:12.762+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fb054044f60 con 0x7fb0540ba210 2020-11-20T12:15:13.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.713+0000 7fb04b7fe700 1 -- 172.21.15.187:0/2233645839 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v104) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb03c048f60 con 0x7fb0540ba210 2020-11-20T12:15:13.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb030056d20 msgr2=0x7fb0300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:13.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb030056d20 0x7fb0300591e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fb04400a8c0 tx=0x7fb044008040).stop 2020-11-20T12:15:13.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540ba210 msgr2=0x7fb0541527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:13.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540ba210 0x7fb0541527a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7fb03c00db60 tx=0x7fb03c008040).stop 2020-11-20T12:15:13.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 shutdown_connections 2020-11-20T12:15:13.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb030056d20 0x7fb0300591e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0540657f0 0x7fb05415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0540ba5f0 0x7fb0540b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 --2- 172.21.15.187:0/2233645839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0540ba210 0x7fb0541527a0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 >> 172.21.15.187:0/2233645839 conn(0x7fb0540bacf0 msgr2=0x7fb054067570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:13.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.717+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 shutdown_connections 2020-11-20T12:15:13.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.718+0000 7fb059863700 1 -- 172.21.15.187:0/2233645839 wait complete. 2020-11-20T12:15:13.723 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:15:13.750 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:15:13.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.989+0000 7f72cb55f700 1 Processor -- start 2020-11-20T12:15:13.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.989+0000 7f72cb55f700 1 -- start start 2020-11-20T12:15:13.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.990+0000 7f72cb55f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f72c4065f90 0x7f72c4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:13.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.990+0000 7f72cb55f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c409ca90 0x7f72c409cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:13.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.990+0000 7f72cb55f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c40b13e0 con 0x7f72c4065190 2020-11-20T12:15:13.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.990+0000 7f72cb55f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72c414ab30 con 0x7f72c4065f90 2020-11-20T12:15:13.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.990+0000 7f72cb55f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4062c10 con 0x7f72c409ca90 2020-11-20T12:15:13.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.991+0000 7f72ca55d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f72c4065190 0x7f72c4143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46378/0 (socket says 172.21.15.187:46378) 2020-11-20T12:15:14.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.991+0000 7f72ca55d700 1 -- 172.21.15.187:0/2703735706 learned_addr learned my addr 172.21.15.187:0/2703735706 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:14.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.991+0000 7f72c9d5c700 1 --2- 172.21.15.187:0/2703735706 >> v2:172.21.15.5:3300/0 conn(0x7f72c4065f90 0x7f72c4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:14.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.991+0000 7f72cad5e700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c409ca90 0x7f72c409cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:14.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.991+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031280288 0 0) 0x7f72c40b13e0 con 0x7f72c4065190 2020-11-20T12:15:14.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.991+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac003240 con 0x7f72c4065190 2020-11-20T12:15:14.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 774500702 0 0) 0x7f72ac003240 con 0x7f72c4065190 2020-11-20T12:15:14.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c409ca90 msgr2=0x7f72c409cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:14.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c409ca90 0x7f72c409cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:14.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 >> v2:172.21.15.5:3300/0 conn(0x7f72c4065f90 msgr2=0x7f72c4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:14.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 --2- 172.21.15.187:0/2703735706 >> v2:172.21.15.5:3300/0 conn(0x7f72c4065f90 0x7f72c4065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:14.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c40b1640 con 0x7f72c4065190 2020-11-20T12:15:14.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f72c00014e0 con 0x7f72c4065190 2020-11-20T12:15:14.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 >> v1:172.21.15.187:6789/0 conn(0x7f72c4065190 legacy=0x7f72c4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:14.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c409ca90 0x7f72ac003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:14.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72ac003c40 0x7f72ac0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:14.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72ac0049d0 0x7f72ac004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:14.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c414ab30 con 0x7f72ac003c40 2020-11-20T12:15:14.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4062c10 con 0x7f72ac0049d0 2020-11-20T12:15:14.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.992+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac004200 con 0x7f72c409ca90 2020-11-20T12:15:14.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.993+0000 7f72cad5e700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72ac0049d0 0x7f72ac004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.993+0000 7f72c9d5c700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72ac003c40 0x7f72ac0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.993+0000 7f72cad5e700 1 -- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c409ca90 msgr2=0x7f72ac003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.993+0000 7f72cad5e700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c409ca90 0x7f72ac003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.993+0000 7f72cad5e700 1 -- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72ac003c40 msgr2=0x7f72ac0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.993+0000 7f72cad5e700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72ac003c40 0x7f72ac0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.993+0000 7f72cad5e700 1 -- 172.21.15.187:0/2703735706 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac003240 con 0x7f72ac0049d0 2020-11-20T12:15:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.993+0000 7f72c9d5c700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72ac003c40 0x7f72ac0044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.994+0000 7f72cad5e700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72ac0049d0 0x7f72ac004e50 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7f72bc009ea0 tx=0x7f72bc0091d0).ready entity=mon.1 client_cookie=531a178b5f988caa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.994+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc0095c0 con 0x7f72ac0049d0 2020-11-20T12:15:14.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.994+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72bc00eda0 con 0x7f72ac0049d0 2020-11-20T12:15:14.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.994+0000 7f72c955b700 1 -- 172.21.15.187:0/2703735706 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc012600 con 0x7f72ac0049d0 2020-11-20T12:15:14.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.995+0000 7f72cb55f700 1 -- 172.21.15.187:0/2703735706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72ac0049d0 msgr2=0x7f72ac004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:14.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.995+0000 7f72cb55f700 1 --2- 172.21.15.187:0/2703735706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72ac0049d0 0x7f72ac004e50 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7f72bc009ea0 tx=0x7f72bc0091d0).stop 2020-11-20T12:15:14.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.995+0000 7f72cad5e700 1 -- 172.21.15.187:0/2703735706 reap_dead start 2020-11-20T12:15:14.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.995+0000 7f72cb55f700 1 -- 172.21.15.187:0/2703735706 shutdown_connections 2020-11-20T12:15:14.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.995+0000 7f72cb55f700 1 -- 172.21.15.187:0/2703735706 >> 172.21.15.187:0/2703735706 conn(0x7f72c40bab30 msgr2=0x7f72c40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:14.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.995+0000 7f72cb55f700 1 -- 172.21.15.187:0/2703735706 shutdown_connections 2020-11-20T12:15:14.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.995+0000 7f72cb55f700 1 -- 172.21.15.187:0/2703735706 wait complete. 2020-11-20T12:15:14.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.997+0000 7f72cb55f700 1 Processor -- start 2020-11-20T12:15:14.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.997+0000 7f72cb55f700 1 -- start start 2020-11-20T12:15:14.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.998+0000 7f72cb55f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c40b7ae0 0x7f72c41497c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:14.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.998+0000 7f72cb55f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c40b7ec0 0x7f72c40993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:14.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.998+0000 7f72cb55f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72c40998b0 0x7f72c409dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:14.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.998+0000 7f72cb55f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c409f890 con 0x7f72c40998b0 2020-11-20T12:15:14.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.998+0000 7f72cb55f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c409fb30 con 0x7f72c40b7ec0 2020-11-20T12:15:14.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.998+0000 7f72cb55f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c409fdd0 con 0x7f72c40b7ae0 2020-11-20T12:15:14.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.998+0000 7f72c9d5c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c40b7ec0 0x7f72c40993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:14.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72c9d5c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c40b7ec0 0x7f72c40993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49786/0 (socket says 172.21.15.187:49786) 2020-11-20T12:15:14.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72c9d5c700 1 -- 172.21.15.187:0/2089734341 learned_addr learned my addr 172.21.15.187:0/2089734341 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72c9d5c700 1 -- 172.21.15.187:0/2089734341 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c40b7ae0 msgr2=0x7f72c41497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72c9d5c700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c40b7ae0 0x7f72c41497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72c9d5c700 1 -- 172.21.15.187:0/2089734341 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72c40998b0 msgr2=0x7f72c409dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72c9d5c700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72c40998b0 0x7f72c409dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72c9d5c700 1 -- 172.21.15.187:0/2089734341 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c40a0270 con 0x7f72c40b7ec0 2020-11-20T12:15:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72c9d5c700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c40b7ec0 0x7f72c40993c0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f72b4006ec0 tx=0x7f72b4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:13.999+0000 7f72b27fc700 1 -- 172.21.15.187:0/2089734341 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4017070 con 0x7f72c40b7ec0 2020-11-20T12:15:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.000+0000 7f72b27fc700 1 -- 172.21.15.187:0/2089734341 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b4011660 con 0x7f72c40b7ec0 2020-11-20T12:15:14.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.000+0000 7f72b27fc700 1 -- 172.21.15.187:0/2089734341 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b40119e0 con 0x7f72c40b7ec0 2020-11-20T12:15:14.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.000+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c40b4610 con 0x7f72c40b7ec0 2020-11-20T12:15:14.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.000+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c409dfd0 con 0x7f72c40b7ec0 2020-11-20T12:15:14.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.001+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c409ff90 con 0x7f72c40b7ec0 2020-11-20T12:15:14.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.001+0000 7f72b27fc700 1 -- 172.21.15.187:0/2089734341 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f72b401c410 con 0x7f72c40b7ec0 2020-11-20T12:15:14.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.002+0000 7f72b27fc700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f729c056d20 0x7f729c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:14.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.002+0000 7f72b27fc700 1 -- 172.21.15.187:0/2089734341 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f72b406fbe0 con 0x7f72c40b7ec0 2020-11-20T12:15:14.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.002+0000 7f72ca55d700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f729c056d20 0x7f729c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:14.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.003+0000 7f72ca55d700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f729c056d20 0x7f729c0591e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f72c00011a0 tx=0x7f72c40bb650).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:14.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.006+0000 7f72b27fc700 1 -- 172.21.15.187:0/2089734341 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f72b40451a0 con 0x7f72c40b7ec0 2020-11-20T12:15:14.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.191+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f72c4061a20 con 0x7f72c40b7ec0 2020-11-20T12:15:14.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2020-11-20T12:15:14.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.719+0000 7f72b27fc700 1 -- 172.21.15.187:0/2089734341 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v105) v1 ==== 125+0+0 (secure 0 0 0) 0x7f72b40487c0 con 0x7f72c40b7ec0 2020-11-20T12:15:14.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.724+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f729c056d20 msgr2=0x7f729c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:14.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.724+0000 7f72cb55f700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f729c056d20 0x7f729c0591e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f72c00011a0 tx=0x7f72c40bb650).stop 2020-11-20T12:15:14.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.724+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c40b7ec0 msgr2=0x7f72c40993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:14.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.724+0000 7f72cb55f700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c40b7ec0 0x7f72c40993c0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f72b4006ec0 tx=0x7f72b4008040).stop 2020-11-20T12:15:14.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.726+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 shutdown_connections 2020-11-20T12:15:14.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.726+0000 7f72cb55f700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f729c056d20 0x7f729c0591e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:14.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.726+0000 7f72cb55f700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f72c40998b0 0x7f72c409dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:14.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.726+0000 7f72cb55f700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f72c40b7ae0 0x7f72c41497c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:14.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.726+0000 7f72cb55f700 1 --2- 172.21.15.187:0/2089734341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c40b7ec0 0x7f72c40993c0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:14.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.726+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 >> 172.21.15.187:0/2089734341 conn(0x7f72c40bab30 msgr2=0x7f72c40b9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:14.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.726+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 shutdown_connections 2020-11-20T12:15:14.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.727+0000 7f72cb55f700 1 -- 172.21.15.187:0/2089734341 wait complete. 2020-11-20T12:15:14.736 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:15:14.757 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:15:14.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.995+0000 7f0e95364700 1 Processor -- start 2020-11-20T12:15:14.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.995+0000 7f0e95364700 1 -- start start 2020-11-20T12:15:14.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.997+0000 7f0e95364700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e901484d0 0x7f0e90061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.997+0000 7f0e95364700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0e900622c0 0x7f0e900b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.997+0000 7f0e95364700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e900b1300 con 0x7f0e90066f70 2020-11-20T12:15:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.997+0000 7f0e95364700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e90067730 con 0x7f0e900622c0 2020-11-20T12:15:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.997+0000 7f0e95364700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900b15a0 con 0x7f0e901484d0 2020-11-20T12:15:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.998+0000 7f0e94b63700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0e90066f70 0x7f0e900b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:39912/0 (socket says 172.21.15.187:39912) 2020-11-20T12:15:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.998+0000 7f0e8ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e901484d0 0x7f0e90061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.998+0000 7f0e94b63700 1 -- 172.21.15.187:0/3097170786 learned_addr learned my addr 172.21.15.187:0/3097170786 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.998+0000 7f0e877fe700 1 --2- 172.21.15.187:0/3097170786 >> v2:172.21.15.5:3300/0 conn(0x7f0e900622c0 0x7f0e900b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.999+0000 7f0e8ffff700 1 -- 172.21.15.187:0/3097170786 >> v2:172.21.15.5:3300/0 conn(0x7f0e900622c0 msgr2=0x7f0e900b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.999+0000 7f0e8ffff700 1 --2- 172.21.15.187:0/3097170786 >> v2:172.21.15.5:3300/0 conn(0x7f0e900622c0 0x7f0e900b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.999+0000 7f0e8ffff700 1 -- 172.21.15.187:0/3097170786 >> v1:172.21.15.5:6789/0 conn(0x7f0e90066f70 legacy=0x7f0e900b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.999+0000 7f0e8ffff700 1 -- 172.21.15.187:0/3097170786 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e900b1840 con 0x7f0e901484d0 2020-11-20T12:15:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:14.999+0000 7f0e8ffff700 1 --2- 172.21.15.187:0/3097170786 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e901484d0 0x7f0e90061dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0e8000b9a0 tx=0x7f0e80006df0).ready entity=mon.2 client_cookie=ef1587bc20b237b8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.000+0000 7f0e8f7fe700 1 -- 172.21.15.187:0/3097170786 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e800097b0 con 0x7f0e901484d0 2020-11-20T12:15:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.000+0000 7f0e8f7fe700 1 -- 172.21.15.187:0/3097170786 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e80009910 con 0x7f0e901484d0 2020-11-20T12:15:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.000+0000 7f0e8f7fe700 1 -- 172.21.15.187:0/3097170786 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80009c90 con 0x7f0e901484d0 2020-11-20T12:15:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.000+0000 7f0e95364700 1 -- 172.21.15.187:0/3097170786 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e901484d0 msgr2=0x7f0e90061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.000+0000 7f0e95364700 1 --2- 172.21.15.187:0/3097170786 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e901484d0 0x7f0e90061dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0e8000b9a0 tx=0x7f0e80006df0).stop 2020-11-20T12:15:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.001+0000 7f0e95364700 1 -- 172.21.15.187:0/3097170786 shutdown_connections 2020-11-20T12:15:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.001+0000 7f0e95364700 1 --2- 172.21.15.187:0/3097170786 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e901484d0 0x7f0e90061dd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.001+0000 7f0e95364700 1 --2- 172.21.15.187:0/3097170786 >> v2:172.21.15.5:3300/0 conn(0x7f0e900622c0 0x7f0e900b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.001+0000 7f0e95364700 1 -- 172.21.15.187:0/3097170786 >> 172.21.15.187:0/3097170786 conn(0x7f0e900bab30 msgr2=0x7f0e900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.001+0000 7f0e95364700 1 -- 172.21.15.187:0/3097170786 shutdown_connections 2020-11-20T12:15:15.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.001+0000 7f0e95364700 1 -- 172.21.15.187:0/3097170786 wait complete. 2020-11-20T12:15:15.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.002+0000 7f0e95364700 1 Processor -- start 2020-11-20T12:15:15.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.003+0000 7f0e95364700 1 -- start start 2020-11-20T12:15:15.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.003+0000 7f0e95364700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900622c0 0x7f0e901430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.003+0000 7f0e95364700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e90066f70 0x7f0e9009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e95364700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e901484d0 0x7f0e90065460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e95364700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e901466f0 con 0x7f0e901484d0 2020-11-20T12:15:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e95364700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e90146410 con 0x7f0e90066f70 2020-11-20T12:15:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e95364700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e901435a0 con 0x7f0e900622c0 2020-11-20T12:15:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e877fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e90066f70 0x7f0e9009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e877fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e90066f70 0x7f0e9009f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49802/0 (socket says 172.21.15.187:49802) 2020-11-20T12:15:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e877fe700 1 -- 172.21.15.187:0/686824986 learned_addr learned my addr 172.21.15.187:0/686824986 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e94b63700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e901484d0 0x7f0e90065460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e8ffff700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900622c0 0x7f0e901430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.004+0000 7f0e877fe700 1 -- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900622c0 msgr2=0x7f0e901430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e877fe700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900622c0 0x7f0e901430b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e877fe700 1 -- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e901484d0 msgr2=0x7f0e90065460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e877fe700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e901484d0 0x7f0e90065460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e877fe700 1 -- 172.21.15.187:0/686824986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9009fad0 con 0x7f0e90066f70 2020-11-20T12:15:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e94b63700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e901484d0 0x7f0e90065460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e877fe700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e90066f70 0x7f0e9009f5e0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f0e7800ebd0 tx=0x7f0e78006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e8cff9700 1 -- 172.21.15.187:0/686824986 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800b8d0 con 0x7f0e90066f70 2020-11-20T12:15:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e8cff9700 1 -- 172.21.15.187:0/686824986 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7800ba30 con 0x7f0e90066f70 2020-11-20T12:15:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e90147710 con 0x7f0e90066f70 2020-11-20T12:15:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e8cff9700 1 -- 172.21.15.187:0/686824986 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800bdb0 con 0x7f0e90066f70 2020-11-20T12:15:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.005+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e90065b10 con 0x7f0e90066f70 2020-11-20T12:15:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.006+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9009fcc0 con 0x7f0e90066f70 2020-11-20T12:15:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.007+0000 7f0e8cff9700 1 -- 172.21.15.187:0/686824986 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0e7801a030 con 0x7f0e90066f70 2020-11-20T12:15:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.008+0000 7f0e8cff9700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d20 0x7f0e6c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.008+0000 7f0e8cff9700 1 -- 172.21.15.187:0/686824986 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f0e7806fab0 con 0x7f0e90066f70 2020-11-20T12:15:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.008+0000 7f0e8ffff700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d20 0x7f0e6c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.009+0000 7f0e8ffff700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d20 0x7f0e6c0591e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f0e80006e30 tx=0x7f0e80015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.012+0000 7f0e8cff9700 1 -- 172.21.15.187:0/686824986 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0e78045020 con 0x7f0e90066f70 2020-11-20T12:15:15.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.257+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f0e901478a0 con 0x7f0e90066f70 2020-11-20T12:15:15.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.274+0000 7f0e8cff9700 1 -- 172.21.15.187:0/686824986 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 21 and data pool 22 v92) v1 ==== 189+0+0 (secure 0 0 0) 0x7f0e901478a0 con 0x7f0e90066f70 2020-11-20T12:15:15.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.281+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d20 msgr2=0x7f0e6c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.281+0000 7f0e95364700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d20 0x7f0e6c0591e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f0e80006e30 tx=0x7f0e80015000).stop 2020-11-20T12:15:15.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.281+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e90066f70 msgr2=0x7f0e9009f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.281+0000 7f0e95364700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e90066f70 0x7f0e9009f5e0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f0e7800ebd0 tx=0x7f0e78006df0).stop 2020-11-20T12:15:15.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.282+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 shutdown_connections 2020-11-20T12:15:15.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.282+0000 7f0e95364700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0e6c056d20 0x7f0e6c0591e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.282+0000 7f0e95364700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e901484d0 0x7f0e90065460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.282+0000 7f0e95364700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0e900622c0 0x7f0e901430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.282+0000 7f0e95364700 1 --2- 172.21.15.187:0/686824986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e90066f70 0x7f0e9009f5e0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.282+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 >> 172.21.15.187:0/686824986 conn(0x7f0e900bab30 msgr2=0x7f0e90144ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:15.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.283+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 shutdown_connections 2020-11-20T12:15:15.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.283+0000 7f0e95364700 1 -- 172.21.15.187:0/686824986 wait complete. 2020-11-20T12:15:15.290 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 21 and data pool 22 2020-11-20T12:15:15.312 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:15:15.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.531+0000 7f65fd82d700 1 Processor -- start 2020-11-20T12:15:15.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.532+0000 7f65fd82d700 1 -- start start 2020-11-20T12:15:15.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.533+0000 7f65fd82d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f65f80b57b0 0x7f65f8068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.533+0000 7f65fd82d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81482d0 0x7f65f8062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.533+0000 7f65fd82d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f8144b80 con 0x7f65f8062b50 2020-11-20T12:15:15.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.533+0000 7f65fd82d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f81489e0 con 0x7f65f80b57b0 2020-11-20T12:15:15.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.533+0000 7f65fd82d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f8148c80 con 0x7f65f81482d0 2020-11-20T12:15:15.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.534+0000 7f65fd02c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f65f8062b50 0x7f65f8065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46422/0 (socket says 172.21.15.187:46422) 2020-11-20T12:15:15.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.534+0000 7f65fd02c700 1 -- 172.21.15.187:0/4054406821 learned_addr learned my addr 172.21.15.187:0/4054406821 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:15.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.534+0000 7f65fc82b700 1 --2- 172.21.15.187:0/4054406821 >> v2:172.21.15.5:3300/0 conn(0x7f65f80b57b0 0x7f65f8068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.534+0000 7f65effff700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81482d0 0x7f65f8062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.534+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731176116 0 0) 0x7f65f8144b80 con 0x7f65f8062b50 2020-11-20T12:15:15.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.534+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e4003240 con 0x7f65f8062b50 2020-11-20T12:15:15.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3616192490 0 0) 0x7f65e4003240 con 0x7f65f8062b50 2020-11-20T12:15:15.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81482d0 msgr2=0x7f65f8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:15.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81482d0 0x7f65f8062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 >> v2:172.21.15.5:3300/0 conn(0x7f65f80b57b0 msgr2=0x7f65f8068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:15.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 --2- 172.21.15.187:0/4054406821 >> v2:172.21.15.5:3300/0 conn(0x7f65f80b57b0 0x7f65f8068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f8148f20 con 0x7f65f8062b50 2020-11-20T12:15:15.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f65f0001860 con 0x7f65f8062b50 2020-11-20T12:15:15.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 >> v1:172.21.15.187:6789/0 conn(0x7f65f8062b50 legacy=0x7f65f8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81482d0 0x7f65e40037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65e4003d10 0x7f65e40044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65e40049d0 0x7f65e4004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f81489e0 con 0x7f65e4003d10 2020-11-20T12:15:15.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f8148c80 con 0x7f65e40049d0 2020-11-20T12:15:15.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.535+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40042d0 con 0x7f65f81482d0 2020-11-20T12:15:15.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.536+0000 7f65fd02c700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65e40049d0 0x7f65e4004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.536+0000 7f65fc82b700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81482d0 0x7f65e40037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.536+0000 7f65effff700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65e4003d10 0x7f65e40044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.536+0000 7f65fd02c700 1 -- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81482d0 msgr2=0x7f65e40037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:15.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.536+0000 7f65fd02c700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81482d0 0x7f65e40037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.536+0000 7f65fd02c700 1 -- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65e4003d10 msgr2=0x7f65e40044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:15.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.536+0000 7f65fd02c700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65e4003d10 0x7f65e40044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.536+0000 7f65fd02c700 1 -- 172.21.15.187:0/4054406821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e4003240 con 0x7f65e40049d0 2020-11-20T12:15:15.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.537+0000 7f65fd02c700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65e40049d0 0x7f65e4004e50 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7f65f000c600 tx=0x7f65f0007ad0).ready entity=mon.1 client_cookie=8e30d35d16480d47 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.537+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f0005b80 con 0x7f65e40049d0 2020-11-20T12:15:15.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.537+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f0005ce0 con 0x7f65e40049d0 2020-11-20T12:15:15.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.537+0000 7f65ef7fe700 1 -- 172.21.15.187:0/4054406821 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f0007320 con 0x7f65e40049d0 2020-11-20T12:15:15.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.538+0000 7f65fd82d700 1 -- 172.21.15.187:0/4054406821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65e40049d0 msgr2=0x7f65e4004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.538+0000 7f65fd82d700 1 --2- 172.21.15.187:0/4054406821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65e40049d0 0x7f65e4004e50 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7f65f000c600 tx=0x7f65f0007ad0).stop 2020-11-20T12:15:15.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.538+0000 7f65fd02c700 1 -- 172.21.15.187:0/4054406821 reap_dead start 2020-11-20T12:15:15.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.538+0000 7f65fd82d700 1 -- 172.21.15.187:0/4054406821 shutdown_connections 2020-11-20T12:15:15.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.538+0000 7f65fd82d700 1 -- 172.21.15.187:0/4054406821 >> 172.21.15.187:0/4054406821 conn(0x7f65f80bab30 msgr2=0x7f65f80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:15.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.538+0000 7f65fd82d700 1 -- 172.21.15.187:0/4054406821 shutdown_connections 2020-11-20T12:15:15.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.538+0000 7f65fd82d700 1 -- 172.21.15.187:0/4054406821 wait complete. 2020-11-20T12:15:15.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.540+0000 7f65fd82d700 1 Processor -- start 2020-11-20T12:15:15.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.540+0000 7f65fd82d700 1 -- start start 2020-11-20T12:15:15.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65fd82d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65f80b7590 0x7f65f80b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65fd82d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65f80b7970 0x7f65f8158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65fd82d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81593e0 0x7f65f8140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65fd82d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f8158bd0 con 0x7f65f80b7590 2020-11-20T12:15:15.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65fd82d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80b39d0 con 0x7f65f80b7970 2020-11-20T12:15:15.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65fd82d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80b36c0 con 0x7f65f81593e0 2020-11-20T12:15:15.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65effff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65f80b7970 0x7f65f8158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65effff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65f80b7970 0x7f65f8158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49826/0 (socket says 172.21.15.187:49826) 2020-11-20T12:15:15.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.541+0000 7f65effff700 1 -- 172.21.15.187:0/2152225634 learned_addr learned my addr 172.21.15.187:0/2152225634 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:15.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65fc82b700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65f80b7590 0x7f65f80b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65fd02c700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81593e0 0x7f65f8140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65effff700 1 -- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81593e0 msgr2=0x7f65f8140de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65effff700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81593e0 0x7f65f8140de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65effff700 1 -- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65f80b7590 msgr2=0x7f65f80b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65effff700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65f80b7590 0x7f65f80b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65effff700 1 -- 172.21.15.187:0/2152225634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f80b4be0 con 0x7f65f80b7970 2020-11-20T12:15:15.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65fc82b700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65f80b7590 0x7f65f80b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:15.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65effff700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65f80b7970 0x7f65f8158ef0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7f65e8010cf0 tx=0x7f65e8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.542+0000 7f65ecff9700 1 -- 172.21.15.187:0/2152225634 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e800ec70 con 0x7f65f80b7970 2020-11-20T12:15:15.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.543+0000 7f65ecff9700 1 -- 172.21.15.187:0/2152225634 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e800edd0 con 0x7f65f80b7970 2020-11-20T12:15:15.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.543+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f8158760 con 0x7f65f80b7970 2020-11-20T12:15:15.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.543+0000 7f65ecff9700 1 -- 172.21.15.187:0/2152225634 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e8013680 con 0x7f65f80b7970 2020-11-20T12:15:15.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.543+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f8141490 con 0x7f65f80b7970 2020-11-20T12:15:15.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.544+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f8061a20 con 0x7f65f80b7970 2020-11-20T12:15:15.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.544+0000 7f65ecff9700 1 -- 172.21.15.187:0/2152225634 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f65e8023070 con 0x7f65f80b7970 2020-11-20T12:15:15.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.545+0000 7f65ecff9700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65d0056d20 0x7f65d00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.545+0000 7f65ecff9700 1 -- 172.21.15.187:0/2152225634 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f65e8070580 con 0x7f65f80b7970 2020-11-20T12:15:15.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.545+0000 7f65fc82b700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65d0056d20 0x7f65d00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.546+0000 7f65fc82b700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65d0056d20 0x7f65d00591e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f65f4006fd0 tx=0x7f65f4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.549+0000 7f65ecff9700 1 -- 172.21.15.187:0/2152225634 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f65e8045950 con 0x7f65f80b7970 2020-11-20T12:15:15.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.735+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65f80446d0 con 0x7f65f80b7970 2020-11-20T12:15:15.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.736+0000 7f65ecff9700 1 -- 172.21.15.187:0/2152225634 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16803 (secure 0 0 0) 0x7f65e8048f70 con 0x7f65f80b7970 2020-11-20T12:15:15.739 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:15.740 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":106,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:15:15.259066+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:15:12.843783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2020-11-20T12:15:14.270455+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:15:15.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65d0056d20 msgr2=0x7f65d00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65d0056d20 0x7f65d00591e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f65f4006fd0 tx=0x7f65f4008040).stop 2020-11-20T12:15:15.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65f80b7970 msgr2=0x7f65f8158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65f80b7970 0x7f65f8158ef0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7f65e8010cf0 tx=0x7f65e8008040).stop 2020-11-20T12:15:15.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 shutdown_connections 2020-11-20T12:15:15.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f65d0056d20 0x7f65d00591e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65f80b7590 0x7f65f80b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f65f81593e0 0x7f65f8140de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 --2- 172.21.15.187:0/2152225634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65f80b7970 0x7f65f8158ef0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 >> 172.21.15.187:0/2152225634 conn(0x7f65f80bab30 msgr2=0x7f65f8143e10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:15.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 shutdown_connections 2020-11-20T12:15:15.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.739+0000 7f65fd82d700 1 -- 172.21.15.187:0/2152225634 wait complete. 2020-11-20T12:15:15.764 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:15:16.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:15.999+0000 7f68cb025700 1 Processor -- start 2020-11-20T12:15:16.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.000+0000 7f68cb025700 1 -- start start 2020-11-20T12:15:16.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.001+0000 7f68cb025700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f68c4065190 0x7f68c4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.001+0000 7f68cb025700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f68c4065f90 0x7f68c4143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.001+0000 7f68cb025700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c40b13e0 con 0x7f68c4143c70 2020-11-20T12:15:16.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.001+0000 7f68cb025700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c414ab30 con 0x7f68c4065f90 2020-11-20T12:15:16.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.001+0000 7f68cb025700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c4062c10 con 0x7f68c4065190 2020-11-20T12:15:16.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68ca023700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f68c4065190 0x7f68c4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68ca023700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f68c4065190 0x7f68c4065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49832/0 (socket says 172.21.15.187:49832) 2020-11-20T12:15:16.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68ca023700 1 -- 172.21.15.187:0/3114112691 learned_addr learned my addr 172.21.15.187:0/3114112691 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:16.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68c9822700 1 --2- 172.21.15.187:0/3114112691 >> v2:172.21.15.5:3300/0 conn(0x7f68c4065f90 0x7f68c4143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68ca023700 1 -- 172.21.15.187:0/3114112691 >> v2:172.21.15.5:3300/0 conn(0x7f68c4065f90 msgr2=0x7f68c4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:16.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68ca023700 1 --2- 172.21.15.187:0/3114112691 >> v2:172.21.15.5:3300/0 conn(0x7f68c4065f90 0x7f68c4143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68ca023700 1 -- 172.21.15.187:0/3114112691 >> v1:172.21.15.5:6789/0 conn(0x7f68c4143c70 legacy=0x7f68c409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:16.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68ca023700 1 -- 172.21.15.187:0/3114112691 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c40b1640 con 0x7f68c4065190 2020-11-20T12:15:16.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.002+0000 7f68c9822700 1 --2- 172.21.15.187:0/3114112691 >> v2:172.21.15.5:3300/0 conn(0x7f68c4065f90 0x7f68c4143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:16.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.003+0000 7f68ca023700 1 --2- 172.21.15.187:0/3114112691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4065190 0x7f68c4065570 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f68c405c8b0 tx=0x7f68c0009290).ready entity=mon.1 client_cookie=a5119196b496cf83 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:16.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.003+0000 7f68c9021700 1 -- 172.21.15.187:0/3114112691 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c000bc10 con 0x7f68c4065190 2020-11-20T12:15:16.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.003+0000 7f68c9021700 1 -- 172.21.15.187:0/3114112691 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c000bd70 con 0x7f68c4065190 2020-11-20T12:15:16.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.003+0000 7f68c9021700 1 -- 172.21.15.187:0/3114112691 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0011610 con 0x7f68c4065190 2020-11-20T12:15:16.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.003+0000 7f68cb025700 1 -- 172.21.15.187:0/3114112691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4065190 msgr2=0x7f68c4065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.003+0000 7f68cb025700 1 --2- 172.21.15.187:0/3114112691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4065190 0x7f68c4065570 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f68c405c8b0 tx=0x7f68c0009290).stop 2020-11-20T12:15:16.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.004+0000 7f68cb025700 1 -- 172.21.15.187:0/3114112691 shutdown_connections 2020-11-20T12:15:16.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.004+0000 7f68cb025700 1 --2- 172.21.15.187:0/3114112691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4065190 0x7f68c4065570 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.004+0000 7f68cb025700 1 --2- 172.21.15.187:0/3114112691 >> v2:172.21.15.5:3300/0 conn(0x7f68c4065f90 0x7f68c4143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.004+0000 7f68cb025700 1 -- 172.21.15.187:0/3114112691 >> 172.21.15.187:0/3114112691 conn(0x7f68c40bab30 msgr2=0x7f68c40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:16.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.004+0000 7f68cb025700 1 -- 172.21.15.187:0/3114112691 shutdown_connections 2020-11-20T12:15:16.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.004+0000 7f68cb025700 1 -- 172.21.15.187:0/3114112691 wait complete. 2020-11-20T12:15:16.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.006+0000 7f68cb025700 1 Processor -- start 2020-11-20T12:15:16.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.006+0000 7f68cb025700 1 -- start start 2020-11-20T12:15:16.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.007+0000 7f68cb025700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68c4065190 0x7f68c40b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.007+0000 7f68cb025700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68c4065f90 0x7f68c4137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.007+0000 7f68cb025700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4143c70 0x7f68c4142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.007+0000 7f68cb025700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c4138530 con 0x7f68c4065f90 2020-11-20T12:15:16.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.007+0000 7f68cb025700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40b2d80 con 0x7f68c4143c70 2020-11-20T12:15:16.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.007+0000 7f68cb025700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40b3020 con 0x7f68c4065190 2020-11-20T12:15:16.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca824700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4143c70 0x7f68c4142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca023700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68c4065190 0x7f68c40b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca824700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4143c70 0x7f68c4142bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49842/0 (socket says 172.21.15.187:49842) 2020-11-20T12:15:16.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca824700 1 -- 172.21.15.187:0/3250874840 learned_addr learned my addr 172.21.15.187:0/3250874840 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:16.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68c9822700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68c4065f90 0x7f68c4137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca824700 1 -- 172.21.15.187:0/3250874840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68c4065190 msgr2=0x7f68c40b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca824700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68c4065190 0x7f68c40b3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca824700 1 -- 172.21.15.187:0/3250874840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68c4065f90 msgr2=0x7f68c4137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca824700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68c4065f90 0x7f68c4137b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68ca824700 1 -- 172.21.15.187:0/3250874840 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c40b32c0 con 0x7f68c4143c70 2020-11-20T12:15:16.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.008+0000 7f68c9822700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68c4065f90 0x7f68c4137b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:16.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.009+0000 7f68ca824700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4143c70 0x7f68c4142bd0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f68bc0096e0 tx=0x7f68bc006cd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:16.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.009+0000 7f68b27fc700 1 -- 172.21.15.187:0/3250874840 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc00d490 con 0x7f68c4143c70 2020-11-20T12:15:16.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.009+0000 7f68b27fc700 1 -- 172.21.15.187:0/3250874840 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68bc000e90 con 0x7f68c4143c70 2020-11-20T12:15:16.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.009+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c40b3c50 con 0x7f68c4143c70 2020-11-20T12:15:16.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.009+0000 7f68b27fc700 1 -- 172.21.15.187:0/3250874840 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc00d860 con 0x7f68c4143c70 2020-11-20T12:15:16.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.009+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c4143140 con 0x7f68c4143c70 2020-11-20T12:15:16.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.010+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c4061a20 con 0x7f68c4143c70 2020-11-20T12:15:16.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.011+0000 7f68b27fc700 1 -- 172.21.15.187:0/3250874840 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f68bc00f030 con 0x7f68c4143c70 2020-11-20T12:15:16.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.011+0000 7f68b27fc700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f689c056d20 0x7f689c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.011+0000 7f68b27fc700 1 -- 172.21.15.187:0/3250874840 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f68bc06fbd0 con 0x7f68c4143c70 2020-11-20T12:15:16.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.012+0000 7f68ca023700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f689c056d20 0x7f689c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.012+0000 7f68ca023700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f689c056d20 0x7f689c0591e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f68c00092d0 tx=0x7f68c0016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:16.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.015+0000 7f68b27fc700 1 -- 172.21.15.187:0/3250874840 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f68bc044f10 con 0x7f68c4143c70 2020-11-20T12:15:16.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.199+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68c40afc60 con 0x7f68c4143c70 2020-11-20T12:15:16.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.201+0000 7f68b27fc700 1 -- 172.21.15.187:0/3250874840 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16803 (secure 0 0 0) 0x7f68bc048530 con 0x7f68c4143c70 2020-11-20T12:15:16.204 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:16.204 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":106,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:15:15.259066+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:15:12.843783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2020-11-20T12:15:14.270455+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:15:16.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.203+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f689c056d20 msgr2=0x7f689c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.203+0000 7f68cb025700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f689c056d20 0x7f689c0591e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f68c00092d0 tx=0x7f68c0016000).stop 2020-11-20T12:15:16.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.203+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4143c70 msgr2=0x7f68c4142bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.203+0000 7f68cb025700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4143c70 0x7f68c4142bd0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f68bc0096e0 tx=0x7f68bc006cd0).stop 2020-11-20T12:15:16.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.204+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 shutdown_connections 2020-11-20T12:15:16.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.204+0000 7f68cb025700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f689c056d20 0x7f689c0591e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.204+0000 7f68cb025700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68c4143c70 0x7f68c4142bd0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.204+0000 7f68cb025700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68c4065190 0x7f68c40b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.204+0000 7f68cb025700 1 --2- 172.21.15.187:0/3250874840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68c4065f90 0x7f68c4137b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.204+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 >> 172.21.15.187:0/3250874840 conn(0x7f68c40bab30 msgr2=0x7f68c409cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:16.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.204+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 shutdown_connections 2020-11-20T12:15:16.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.204+0000 7f68cb025700 1 -- 172.21.15.187:0/3250874840 wait complete. 2020-11-20T12:15:16.230 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:15:16.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.477+0000 7ff40f06b700 1 Processor -- start 2020-11-20T12:15:16.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.478+0000 7ff40f06b700 1 -- start start 2020-11-20T12:15:16.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.479+0000 7ff40f06b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff4080bf800 0x7ff40814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.479+0000 7ff40f06b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40814d110 con 0x7ff4080bfbe0 2020-11-20T12:15:16.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.479+0000 7ff40f06b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40814cbd0 con 0x7ff40809f560 2020-11-20T12:15:16.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.479+0000 7ff40f06b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40814ce70 con 0x7ff4080bf800 2020-11-20T12:15:16.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.479+0000 7ff40e069700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff4080bf800 0x7ff40814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.479+0000 7ff40e86a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff40809f560 0x7ff4080a36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46456/0 (socket says 172.21.15.187:46456) 2020-11-20T12:15:16.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.479+0000 7ff40e86a700 1 -- 172.21.15.187:0/3444273604 learned_addr learned my addr 172.21.15.187:0/3444273604 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:16.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.480+0000 7ff40e069700 1 -- 172.21.15.187:0/3444273604 >> v1:172.21.15.187:6789/0 conn(0x7ff40809f560 legacy=0x7ff4080a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.480+0000 7ff40e069700 1 -- 172.21.15.187:0/3444273604 >> v1:172.21.15.5:6789/0 conn(0x7ff4080bfbe0 legacy=0x7ff4081480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:16.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.480+0000 7ff40e069700 1 -- 172.21.15.187:0/3444273604 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4080a4cc0 con 0x7ff4080bf800 2020-11-20T12:15:16.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.480+0000 7ff40e069700 1 --2- 172.21.15.187:0/3444273604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4080bf800 0x7ff40814c2c0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7ff40400b9a0 tx=0x7ff404006df0).ready entity=mon.1 client_cookie=519b164af33e2cfc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:16.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40d067700 1 -- 172.21.15.187:0/3444273604 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404009800 con 0x7ff4080bf800 2020-11-20T12:15:16.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40d067700 1 -- 172.21.15.187:0/3444273604 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff404009960 con 0x7ff4080bf800 2020-11-20T12:15:16.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40d067700 1 -- 172.21.15.187:0/3444273604 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404009ce0 con 0x7ff4080bf800 2020-11-20T12:15:16.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40f06b700 1 -- 172.21.15.187:0/3444273604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4080bf800 msgr2=0x7ff40814c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40f06b700 1 --2- 172.21.15.187:0/3444273604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4080bf800 0x7ff40814c2c0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7ff40400b9a0 tx=0x7ff404006df0).stop 2020-11-20T12:15:16.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40f06b700 1 -- 172.21.15.187:0/3444273604 shutdown_connections 2020-11-20T12:15:16.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40f06b700 1 --2- 172.21.15.187:0/3444273604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4080bf800 0x7ff40814c2c0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40f06b700 1 -- 172.21.15.187:0/3444273604 >> 172.21.15.187:0/3444273604 conn(0x7ff408068370 msgr2=0x7ff4080bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:16.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40f06b700 1 -- 172.21.15.187:0/3444273604 shutdown_connections 2020-11-20T12:15:16.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.481+0000 7ff40f06b700 1 -- 172.21.15.187:0/3444273604 wait complete. 2020-11-20T12:15:16.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.483+0000 7ff40f06b700 1 Processor -- start 2020-11-20T12:15:16.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.483+0000 7ff40f06b700 1 -- start start 2020-11-20T12:15:16.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40f06b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff40809f560 0x7ff4080b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40f06b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4080bf800 0x7ff408142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40f06b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff408144800 0x7ff4081411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40f06b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080b5170 con 0x7ff4080bf800 2020-11-20T12:15:16.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40f06b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080b6410 con 0x7ff40809f560 2020-11-20T12:15:16.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40f06b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff408144ca0 con 0x7ff408144800 2020-11-20T12:15:16.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40e069700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff40809f560 0x7ff4080b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40e069700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff40809f560 0x7ff4080b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49852/0 (socket says 172.21.15.187:49852) 2020-11-20T12:15:16.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.484+0000 7ff40e069700 1 -- 172.21.15.187:0/2552462836 learned_addr learned my addr 172.21.15.187:0/2552462836 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:16.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff40d868700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4080bf800 0x7ff408142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff40e86a700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff408144800 0x7ff4081411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff40e069700 1 -- 172.21.15.187:0/2552462836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff408144800 msgr2=0x7ff4081411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff40e069700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff408144800 0x7ff4081411a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff40e069700 1 -- 172.21.15.187:0/2552462836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4080bf800 msgr2=0x7ff408142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:16.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff40e069700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4080bf800 0x7ff408142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:16.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff40e069700 1 -- 172.21.15.187:0/2552462836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4080b6600 con 0x7ff40809f560 2020-11-20T12:15:16.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff40e069700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff40809f560 0x7ff4080b3e90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7ff40400b150 tx=0x7ff404015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:16.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.485+0000 7ff3f67fc700 1 -- 172.21.15.187:0/2552462836 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40401d5d0 con 0x7ff40809f560 2020-11-20T12:15:16.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.486+0000 7ff3f67fc700 1 -- 172.21.15.187:0/2552462836 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40401d730 con 0x7ff40809f560 2020-11-20T12:15:16.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.486+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4080b4380 con 0x7ff40809f560 2020-11-20T12:15:16.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.486+0000 7ff3f67fc700 1 -- 172.21.15.187:0/2552462836 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40401dad0 con 0x7ff40809f560 2020-11-20T12:15:16.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.486+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff408145e30 con 0x7ff40809f560 2020-11-20T12:15:16.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.487+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff408080050 con 0x7ff40809f560 2020-11-20T12:15:16.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.487+0000 7ff3f67fc700 1 -- 172.21.15.187:0/2552462836 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff404025030 con 0x7ff40809f560 2020-11-20T12:15:16.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.488+0000 7ff3f67fc700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3e0056d20 0x7ff3e00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:16.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.488+0000 7ff3f67fc700 1 -- 172.21.15.187:0/2552462836 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff404071940 con 0x7ff40809f560 2020-11-20T12:15:16.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.488+0000 7ff40d868700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3e0056d20 0x7ff3e00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:16.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.489+0000 7ff40d868700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3e0056d20 0x7ff3e00591e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7ff4080b5f10 tx=0x7ff3f8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:16.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.492+0000 7ff3f67fc700 1 -- 172.21.15.187:0/2552462836 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff404046d90 con 0x7ff40809f560 2020-11-20T12:15:16.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:16.737+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff40814c7b0 con 0x7ff40809f560 2020-11-20T12:15:17.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.293+0000 7ff3f67fc700 1 -- 172.21.15.187:0/2552462836 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v95) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff404023070 con 0x7ff40809f560 2020-11-20T12:15:17.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3e0056d20 msgr2=0x7ff3e00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3e0056d20 0x7ff3e00591e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7ff4080b5f10 tx=0x7ff3f8008040).stop 2020-11-20T12:15:17.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff40809f560 msgr2=0x7ff4080b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff40809f560 0x7ff4080b3e90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7ff40400b150 tx=0x7ff404015000).stop 2020-11-20T12:15:17.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 shutdown_connections 2020-11-20T12:15:17.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3e0056d20 0x7ff3e00591e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff408144800 0x7ff4081411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff40809f560 0x7ff4080b3e90 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 --2- 172.21.15.187:0/2552462836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4080bf800 0x7ff408142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.296+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 >> 172.21.15.187:0/2552462836 conn(0x7ff408068370 msgr2=0x7ff408147000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:17.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.297+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 shutdown_connections 2020-11-20T12:15:17.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.297+0000 7ff40f06b700 1 -- 172.21.15.187:0/2552462836 wait complete. 2020-11-20T12:15:17.328 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:17.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.569+0000 7f5c1608c700 1 Processor -- start 2020-11-20T12:15:17.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.569+0000 7f5c1608c700 1 -- start start 2020-11-20T12:15:17.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.570+0000 7f5c1608c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5c10065190 0x7f5c10065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.571+0000 7f5c1608c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100b13e0 con 0x7f5c10065f90 2020-11-20T12:15:17.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.571+0000 7f5c1608c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c10062c10 con 0x7f5c1009ca90 2020-11-20T12:15:17.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.571+0000 7f5c1608c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1014ab30 con 0x7f5c10065190 2020-11-20T12:15:17.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.571+0000 7f5c1508a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5c10065190 0x7f5c10065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.571+0000 7f5c1588b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5c1009ca90 0x7f5c1009cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46470/0 (socket says 172.21.15.187:46470) 2020-11-20T12:15:17.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.571+0000 7f5c1588b700 1 -- 172.21.15.187:0/3551385881 learned_addr learned my addr 172.21.15.187:0/3551385881 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:17.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.572+0000 7f5c1508a700 1 -- 172.21.15.187:0/3551385881 >> v1:172.21.15.187:6789/0 conn(0x7f5c1009ca90 legacy=0x7f5c1009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.572+0000 7f5c1508a700 1 -- 172.21.15.187:0/3551385881 >> v1:172.21.15.5:6789/0 conn(0x7f5c10065f90 legacy=0x7f5c10143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:17.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.572+0000 7f5c1508a700 1 -- 172.21.15.187:0/3551385881 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c100b1640 con 0x7f5c10065190 2020-11-20T12:15:17.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.572+0000 7f5c1508a700 1 --2- 172.21.15.187:0/3551385881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c10065190 0x7f5c10065570 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f5c1014ac70 tx=0x7f5c0c0091d0).ready entity=mon.1 client_cookie=436e085b4c164711 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:17.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.572+0000 7f5bfffff700 1 -- 172.21.15.187:0/3551385881 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c00bc10 con 0x7f5c10065190 2020-11-20T12:15:17.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.572+0000 7f5bfffff700 1 -- 172.21.15.187:0/3551385881 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c00bd70 con 0x7f5c10065190 2020-11-20T12:15:17.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.572+0000 7f5bfffff700 1 -- 172.21.15.187:0/3551385881 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c011610 con 0x7f5c10065190 2020-11-20T12:15:17.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.573+0000 7f5c1608c700 1 -- 172.21.15.187:0/3551385881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c10065190 msgr2=0x7f5c10065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.573+0000 7f5c1608c700 1 --2- 172.21.15.187:0/3551385881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c10065190 0x7f5c10065570 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f5c1014ac70 tx=0x7f5c0c0091d0).stop 2020-11-20T12:15:17.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.573+0000 7f5c1608c700 1 -- 172.21.15.187:0/3551385881 shutdown_connections 2020-11-20T12:15:17.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.573+0000 7f5c1608c700 1 --2- 172.21.15.187:0/3551385881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c10065190 0x7f5c10065570 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.573+0000 7f5c1608c700 1 -- 172.21.15.187:0/3551385881 >> 172.21.15.187:0/3551385881 conn(0x7f5c100bab30 msgr2=0x7f5c100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:17.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.573+0000 7f5c1608c700 1 -- 172.21.15.187:0/3551385881 shutdown_connections 2020-11-20T12:15:17.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.573+0000 7f5c1608c700 1 -- 172.21.15.187:0/3551385881 wait complete. 2020-11-20T12:15:17.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.574+0000 7f5c1608c700 1 Processor -- start 2020-11-20T12:15:17.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.575+0000 7f5c1608c700 1 -- start start 2020-11-20T12:15:17.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.575+0000 7f5c1608c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c10065190 0x7f5c100b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.575+0000 7f5c1608c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c10065f90 0x7f5c100b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.575+0000 7f5c1608c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c1009ca90 0x7f5c1015be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1608c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1015cca0 con 0x7f5c10065f90 2020-11-20T12:15:17.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1608c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c101488d0 con 0x7f5c1009ca90 2020-11-20T12:15:17.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1608c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1015ca00 con 0x7f5c10065190 2020-11-20T12:15:17.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1588b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c1009ca90 0x7f5c1015be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1588b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c1009ca90 0x7f5c1015be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49870/0 (socket says 172.21.15.187:49870) 2020-11-20T12:15:17.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1588b700 1 -- 172.21.15.187:0/731618245 learned_addr learned my addr 172.21.15.187:0/731618245 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:17.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1508a700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c10065190 0x7f5c100b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1588b700 1 -- 172.21.15.187:0/731618245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c10065190 msgr2=0x7f5c100b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1588b700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c10065190 0x7f5c100b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1588b700 1 -- 172.21.15.187:0/731618245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c10065f90 msgr2=0x7f5c100b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1588b700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c10065f90 0x7f5c100b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:17.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.576+0000 7f5c1588b700 1 -- 172.21.15.187:0/731618245 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c10148d30 con 0x7f5c1009ca90 2020-11-20T12:15:17.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.577+0000 7f5c1588b700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c1009ca90 0x7f5c1015be80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f5c0800c710 tx=0x7f5c08007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:17.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.577+0000 7f5bfd7fa700 1 -- 172.21.15.187:0/731618245 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080072e0 con 0x7f5c1009ca90 2020-11-20T12:15:17.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.577+0000 7f5bfd7fa700 1 -- 172.21.15.187:0/731618245 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c08007440 con 0x7f5c1009ca90 2020-11-20T12:15:17.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.577+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c10146720 con 0x7f5c1009ca90 2020-11-20T12:15:17.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.577+0000 7f5bfd7fa700 1 -- 172.21.15.187:0/731618245 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08005800 con 0x7f5c1009ca90 2020-11-20T12:15:17.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.577+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c100992e0 con 0x7f5c1009ca90 2020-11-20T12:15:17.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.578+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c10148a90 con 0x7f5c1009ca90 2020-11-20T12:15:17.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.579+0000 7f5bfd7fa700 1 -- 172.21.15.187:0/731618245 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5c08010070 con 0x7f5c1009ca90 2020-11-20T12:15:17.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.579+0000 7f5bfd7fa700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5be8056d20 0x7f5be80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.579+0000 7f5bfd7fa700 1 -- 172.21.15.187:0/731618245 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5c0806f620 con 0x7f5c1009ca90 2020-11-20T12:15:17.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.580+0000 7f5c1508a700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5be8056d20 0x7f5be80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.580+0000 7f5c1508a700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5be8056d20 0x7f5be80591e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f5c0c009fd0 tx=0x7f5c0c016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:17.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.583+0000 7f5bfd7fa700 1 -- 172.21.15.187:0/731618245 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5c10148a90 con 0x7f5c1009ca90 2020-11-20T12:15:17.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.830+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c101468b0 con 0x7f5c1009ca90 2020-11-20T12:15:17.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.831+0000 7f5bfd7fa700 1 -- 172.21.15.187:0/731618245 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f5c101468b0 con 0x7f5c1009ca90 2020-11-20T12:15:17.834 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:17.834 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5be8056d20 msgr2=0x7f5be80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5be8056d20 0x7f5be80591e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f5c0c009fd0 tx=0x7f5c0c016000).stop 2020-11-20T12:15:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c1009ca90 msgr2=0x7f5c1015be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c1009ca90 0x7f5c1015be80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f5c0800c710 tx=0x7f5c08007c30).stop 2020-11-20T12:15:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 shutdown_connections 2020-11-20T12:15:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5be8056d20 0x7f5be80591e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c1009ca90 0x7f5c1015be80 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5c10065190 0x7f5c100b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 --2- 172.21.15.187:0/731618245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c10065f90 0x7f5c100b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.834+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 >> 172.21.15.187:0/731618245 conn(0x7f5c100bab30 msgr2=0x7f5c100b77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.835+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 shutdown_connections 2020-11-20T12:15:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:17.835+0000 7f5c1608c700 1 -- 172.21.15.187:0/731618245 wait complete. 2020-11-20T12:15:17.840 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:17.861 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:15:18.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.106+0000 7f3e209aa700 1 Processor -- start 2020-11-20T12:15:18.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.107+0000 7f3e209aa700 1 -- start start 2020-11-20T12:15:18.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.108+0000 7f3e209aa700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3e1c1484d0 0x7f3e1c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.108+0000 7f3e209aa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c066f70 0x7f3e1c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.108+0000 7f3e209aa700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1c0b1300 con 0x7f3e1c0622c0 2020-11-20T12:15:18.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.108+0000 7f3e209aa700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e1c067730 con 0x7f3e1c1484d0 2020-11-20T12:15:18.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.108+0000 7f3e209aa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c0b15a0 con 0x7f3e1c066f70 2020-11-20T12:15:18.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.108+0000 7f3e1259c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3e1c0622c0 0x7f3e1c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46480/0 (socket says 172.21.15.187:46480) 2020-11-20T12:15:18.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.108+0000 7f3e1259c700 1 -- 172.21.15.187:0/1433174321 learned_addr learned my addr 172.21.15.187:0/1433174321 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:18.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1ad9d700 1 --2- 172.21.15.187:0/1433174321 >> v2:172.21.15.5:3300/0 conn(0x7f3e1c1484d0 0x7f3e1c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392134058 0 0) 0x7f3e1c0b1300 con 0x7f3e1c0622c0 2020-11-20T12:15:18.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e00003240 con 0x7f3e1c0622c0 2020-11-20T12:15:18.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1b59e700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c066f70 0x7f3e1c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1117029712 0 0) 0x7f3e00003240 con 0x7f3e1c0622c0 2020-11-20T12:15:18.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c066f70 msgr2=0x7f3e1c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:18.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c066f70 0x7f3e1c0b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 >> v2:172.21.15.5:3300/0 conn(0x7f3e1c1484d0 msgr2=0x7f3e1c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:18.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 --2- 172.21.15.187:0/1433174321 >> v2:172.21.15.5:3300/0 conn(0x7f3e1c1484d0 0x7f3e1c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c0b1840 con 0x7f3e1c0622c0 2020-11-20T12:15:18.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.109+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f3e040014e0 con 0x7f3e1c0622c0 2020-11-20T12:15:18.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1ad9d700 1 --2- 172.21.15.187:0/1433174321 >> v2:172.21.15.5:3300/0 conn(0x7f3e1c1484d0 0x7f3e1c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:18.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 >> v1:172.21.15.187:6789/0 conn(0x7f3e1c0622c0 legacy=0x7f3e1c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1a59c700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c066f70 0x7f3e00003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1a59c700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e00003c40 0x7f3e000044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1a59c700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e000049d0 0x7f3e00004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c067730 con 0x7f3e000049d0 2020-11-20T12:15:18.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c0b15a0 con 0x7f3e00003c40 2020-11-20T12:15:18.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00004200 con 0x7f3e1c066f70 2020-11-20T12:15:18.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1259c700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e00003c40 0x7f3e000044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1ad9d700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c066f70 0x7f3e00003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1259c700 1 -- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c066f70 msgr2=0x7f3e00003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:18.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1259c700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c066f70 0x7f3e00003730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1259c700 1 -- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e000049d0 msgr2=0x7f3e00004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:18.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1259c700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e000049d0 0x7f3e00004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:18.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.110+0000 7f3e1259c700 1 -- 172.21.15.187:0/1433174321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00003240 con 0x7f3e00003c40 2020-11-20T12:15:18.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.111+0000 7f3e1259c700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e00003c40 0x7f3e000044e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f3e0400c410 tx=0x7f3e04007800).ready entity=mon.1 client_cookie=673b4aa9ae5a03a1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:18.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.111+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0400ce80 con 0x7f3e00003c40 2020-11-20T12:15:18.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.111+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e040072e0 con 0x7f3e00003c40 2020-11-20T12:15:18.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.111+0000 7f3e1a59c700 1 -- 172.21.15.187:0/1433174321 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e04005820 con 0x7f3e00003c40 2020-11-20T12:15:18.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.111+0000 7f3e209aa700 1 -- 172.21.15.187:0/1433174321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e00003c40 msgr2=0x7f3e000044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.111+0000 7f3e209aa700 1 --2- 172.21.15.187:0/1433174321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e00003c40 0x7f3e000044e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f3e0400c410 tx=0x7f3e04007800).stop 2020-11-20T12:15:18.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.111+0000 7f3e1b59e700 1 -- 172.21.15.187:0/1433174321 reap_dead start 2020-11-20T12:15:18.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.112+0000 7f3e209aa700 1 -- 172.21.15.187:0/1433174321 shutdown_connections 2020-11-20T12:15:18.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.112+0000 7f3e209aa700 1 -- 172.21.15.187:0/1433174321 >> 172.21.15.187:0/1433174321 conn(0x7f3e1c0bab30 msgr2=0x7f3e1c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:18.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.112+0000 7f3e209aa700 1 -- 172.21.15.187:0/1433174321 shutdown_connections 2020-11-20T12:15:18.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.112+0000 7f3e209aa700 1 -- 172.21.15.187:0/1433174321 wait complete. 2020-11-20T12:15:18.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.114+0000 7f3e209aa700 1 Processor -- start 2020-11-20T12:15:18.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.114+0000 7f3e209aa700 1 -- start start 2020-11-20T12:15:18.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e209aa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e1c0b2640 0x7f3e1c09ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e209aa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e1c0b2a20 0x7f3e1c09ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e209aa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c09f1c0 0x7f3e1c09bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e209aa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c1398f0 con 0x7f3e1c0b2640 2020-11-20T12:15:18.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e209aa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c09e860 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e209aa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c0b3180 con 0x7f3e1c09f1c0 2020-11-20T12:15:18.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e1259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e1c0b2a20 0x7f3e1c09ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e1259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e1c0b2a20 0x7f3e1c09ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49888/0 (socket says 172.21.15.187:49888) 2020-11-20T12:15:18.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.115+0000 7f3e1259c700 1 -- 172.21.15.187:0/2728356712 learned_addr learned my addr 172.21.15.187:0/2728356712 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:18.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e1ad9d700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e1c0b2640 0x7f3e1c09ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e1259c700 1 -- 172.21.15.187:0/2728356712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c09f1c0 msgr2=0x7f3e1c09bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e1259c700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c09f1c0 0x7f3e1c09bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:18.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e1259c700 1 -- 172.21.15.187:0/2728356712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e1c0b2640 msgr2=0x7f3e1c09ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e1259c700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e1c0b2640 0x7f3e1c09ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e1259c700 1 -- 172.21.15.187:0/2728356712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c0b3420 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e1ad9d700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e1c0b2640 0x7f3e1c09ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:18.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e1259c700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e1c0b2a20 0x7f3e1c09ecd0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f3e000041d0 tx=0x7f3e04015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:18.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e137fe700 1 -- 172.21.15.187:0/2728356712 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e04006d20 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e137fe700 1 -- 172.21.15.187:0/2728356712 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e04006e80 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1c0b2ea0 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e137fe700 1 -- 172.21.15.187:0/2728356712 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e040056d0 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.116+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1c09c130 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.118+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e1c061a20 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.118+0000 7f3e137fe700 1 -- 172.21.15.187:0/2728356712 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3e04025020 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.118+0000 7f3e137fe700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dec056d20 0x7f3dec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.119+0000 7f3e137fe700 1 -- 172.21.15.187:0/2728356712 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e04071430 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.119+0000 7f3e1ad9d700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dec056d20 0x7f3dec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.120+0000 7f3e1ad9d700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dec056d20 0x7f3dec0591e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f3e0c00d890 tx=0x7f3e0c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:18.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.123+0000 7f3e137fe700 1 -- 172.21.15.187:0/2728356712 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3e04046830 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.309+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e1c1488b0 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.311+0000 7f3e137fe700 1 -- 172.21.15.187:0/2728356712 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16791 (secure 0 0 0) 0x7f3e04049e50 con 0x7f3e1c0b2a20 2020-11-20T12:15:18.315 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:18.315 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:15:17.285828+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:15:12.843783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2020-11-20T12:15:14.270455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:15:18.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.313+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dec056d20 msgr2=0x7f3dec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dec056d20 0x7f3dec0591e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f3e0c00d890 tx=0x7f3e0c008040).stop 2020-11-20T12:15:18.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e1c0b2a20 msgr2=0x7f3e1c09ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e1c0b2a20 0x7f3e1c09ecd0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f3e000041d0 tx=0x7f3e04015000).stop 2020-11-20T12:15:18.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 shutdown_connections 2020-11-20T12:15:18.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3dec056d20 0x7f3dec0591e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e1c0b2640 0x7f3e1c09ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e1c09f1c0 0x7f3e1c09bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:18.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 --2- 172.21.15.187:0/2728356712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e1c0b2a20 0x7f3e1c09ecd0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 >> 172.21.15.187:0/2728356712 conn(0x7f3e1c0bab30 msgr2=0x7f3e1c0b6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:18.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 shutdown_connections 2020-11-20T12:15:18.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.314+0000 7f3e209aa700 1 -- 172.21.15.187:0/2728356712 wait complete. 2020-11-20T12:15:18.339 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:15:18.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.590+0000 7feceeb7b700 1 Processor -- start 2020-11-20T12:15:18.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.590+0000 7feceeb7b700 1 -- start start 2020-11-20T12:15:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.591+0000 7feceeb7b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece8066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.591+0000 7feceeb7b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fece81487e0 0x7fece8148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.591+0000 7feceeb7b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece814c170 con 0x7fece80683f0 2020-11-20T12:15:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.591+0000 7feceeb7b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece814c410 con 0x7fece81487e0 2020-11-20T12:15:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.591+0000 7feceeb7b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece814c6b0 con 0x7fece8145cc0 2020-11-20T12:15:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.592+0000 7fecedb79700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece8066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.592+0000 7fecedb79700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece8066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:50782/0 (socket says 172.21.15.187:50782) 2020-11-20T12:15:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.592+0000 7fecedb79700 1 -- 172.21.15.187:0/3254900663 learned_addr learned my addr 172.21.15.187:0/3254900663 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.592+0000 7fecee37a700 1 --2- 172.21.15.187:0/3254900663 >> v2:172.21.15.5:3300/0 conn(0x7fece81487e0 0x7fece8148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.593+0000 7fecedb79700 1 -- 172.21.15.187:0/3254900663 >> v2:172.21.15.5:3300/0 conn(0x7fece81487e0 msgr2=0x7fece8148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.593+0000 7fecedb79700 1 --2- 172.21.15.187:0/3254900663 >> v2:172.21.15.5:3300/0 conn(0x7fece81487e0 0x7fece8148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.593+0000 7fecedb79700 1 -- 172.21.15.187:0/3254900663 >> v1:172.21.15.5:6789/0 conn(0x7fece80683f0 legacy=0x7fece8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.594+0000 7fecedb79700 1 -- 172.21.15.187:0/3254900663 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece80658a0 con 0x7fece8145cc0 2020-11-20T12:15:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.594+0000 7fecedb79700 1 --2- 172.21.15.187:0/3254900663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece8066040 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fece4006fd0 tx=0x7fece4006eb0).ready entity=mon.2 client_cookie=3db4e141e58ea5ac server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.595+0000 7fececb77700 1 -- 172.21.15.187:0/3254900663 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4009810 con 0x7fece8145cc0 2020-11-20T12:15:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.595+0000 7fececb77700 1 -- 172.21.15.187:0/3254900663 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece4009970 con 0x7fece8145cc0 2020-11-20T12:15:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.595+0000 7fececb77700 1 -- 172.21.15.187:0/3254900663 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4009cf0 con 0x7fece8145cc0 2020-11-20T12:15:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.596+0000 7feceeb7b700 1 -- 172.21.15.187:0/3254900663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 msgr2=0x7fece8066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.596+0000 7feceeb7b700 1 --2- 172.21.15.187:0/3254900663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece8066040 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fece4006fd0 tx=0x7fece4006eb0).stop 2020-11-20T12:15:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.596+0000 7feceeb7b700 1 -- 172.21.15.187:0/3254900663 shutdown_connections 2020-11-20T12:15:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.596+0000 7feceeb7b700 1 --2- 172.21.15.187:0/3254900663 >> v2:172.21.15.5:3300/0 conn(0x7fece81487e0 0x7fece8148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.596+0000 7feceeb7b700 1 --2- 172.21.15.187:0/3254900663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece8066040 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.596+0000 7feceeb7b700 1 -- 172.21.15.187:0/3254900663 >> 172.21.15.187:0/3254900663 conn(0x7fece80bab30 msgr2=0x7fece80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.596+0000 7feceeb7b700 1 -- 172.21.15.187:0/3254900663 shutdown_connections 2020-11-20T12:15:18.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.596+0000 7feceeb7b700 1 -- 172.21.15.187:0/3254900663 wait complete. 2020-11-20T12:15:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.598+0000 7feceeb7b700 1 Processor -- start 2020-11-20T12:15:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.598+0000 7feceeb7b700 1 -- start start 2020-11-20T12:15:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7feceeb7b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fece80683f0 0x7fece80b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7feceeb7b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece809f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7feceeb7b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fece81487e0 0x7fece809c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7feceeb7b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80b3d80 con 0x7fece80683f0 2020-11-20T12:15:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7feceeb7b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece809f980 con 0x7fece81487e0 2020-11-20T12:15:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7feceeb7b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece809f530 con 0x7fece8145cc0 2020-11-20T12:15:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7fecee37a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fece81487e0 0x7fece809c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7fecee37a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fece81487e0 0x7fece809c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49904/0 (socket says 172.21.15.187:49904) 2020-11-20T12:15:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7fecee37a700 1 -- 172.21.15.187:0/1177201880 learned_addr learned my addr 172.21.15.187:0/1177201880 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7feced378700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece809f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.600+0000 7fecedb79700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fece80683f0 0x7fece80b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecee37a700 1 -- 172.21.15.187:0/1177201880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 msgr2=0x7fece809f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecee37a700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece809f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecee37a700 1 -- 172.21.15.187:0/1177201880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fece80683f0 msgr2=0x7fece80b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecee37a700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fece80683f0 0x7fece80b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecee37a700 1 -- 172.21.15.187:0/1177201880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece809c520 con 0x7fece81487e0 2020-11-20T12:15:18.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecee37a700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fece81487e0 0x7fece809c030 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7fece000ed20 tx=0x7fece00091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:18.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecd5ffb700 1 -- 172.21.15.187:0/1177201880 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece000cbb0 con 0x7fece81487e0 2020-11-20T12:15:18.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecd5ffb700 1 -- 172.21.15.187:0/1177201880 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece000cd10 con 0x7fece81487e0 2020-11-20T12:15:18.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7fecd5ffb700 1 -- 172.21.15.187:0/1177201880 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0012620 con 0x7fece81487e0 2020-11-20T12:15:18.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.601+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece80b3830 con 0x7fece81487e0 2020-11-20T12:15:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.602+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece8144cc0 con 0x7fece81487e0 2020-11-20T12:15:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.603+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece80b3ef0 con 0x7fece81487e0 2020-11-20T12:15:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.603+0000 7fecd5ffb700 1 -- 172.21.15.187:0/1177201880 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fece0022070 con 0x7fece81487e0 2020-11-20T12:15:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.604+0000 7fecd5ffb700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecc0056d20 0x7fecc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.604+0000 7fecd5ffb700 1 -- 172.21.15.187:0/1177201880 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fece0070730 con 0x7fece81487e0 2020-11-20T12:15:18.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.604+0000 7fecedb79700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecc0056d20 0x7fecc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:18.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.605+0000 7fecedb79700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecc0056d20 0x7fecc00591e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fece400b1b0 tx=0x7fece4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:18.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.608+0000 7fecd5ffb700 1 -- 172.21.15.187:0/1177201880 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fece0045b80 con 0x7fece81487e0 2020-11-20T12:15:18.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.861+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fece8061a20 con 0x7fece81487e0 2020-11-20T12:15:18.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.872+0000 7fecd5ffb700 1 -- 172.21.15.187:0/1177201880 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 184+0+0 (secure 0 0 0) 0x7fece0016020 con 0x7fece81487e0 2020-11-20T12:15:18.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecc0056d20 msgr2=0x7fecc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecc0056d20 0x7fecc00591e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fece400b1b0 tx=0x7fece4015000).stop 2020-11-20T12:15:18.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fece81487e0 msgr2=0x7fece809c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fece81487e0 0x7fece809c030 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7fece000ed20 tx=0x7fece00091d0).stop 2020-11-20T12:15:18.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 shutdown_connections 2020-11-20T12:15:18.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecc0056d20 0x7fecc00591e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fece80683f0 0x7fece80b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fece81487e0 0x7fece809c030 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 --2- 172.21.15.187:0/1177201880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fece8145cc0 0x7fece809f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.874+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 >> 172.21.15.187:0/1177201880 conn(0x7fece80bab30 msgr2=0x7fece80b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:18.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.875+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 shutdown_connections 2020-11-20T12:15:18.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:18.875+0000 7feceeb7b700 1 -- 172.21.15.187:0/1177201880 wait complete. 2020-11-20T12:15:18.886 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:15:18.902 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:19.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.150+0000 7f9664ca1700 1 Processor -- start 2020-11-20T12:15:19.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.151+0000 7f9664ca1700 1 -- start start 2020-11-20T12:15:19.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.152+0000 7f9664ca1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f96600bfbe0 0x7f96601480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.152+0000 7f9664ca1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f966009f560 0x7f96601485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.152+0000 7f9664ca1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966014d110 con 0x7f96600bf800 2020-11-20T12:15:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.152+0000 7f9664ca1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f966014c9d0 con 0x7f96600bfbe0 2020-11-20T12:15:19.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.152+0000 7f9664ca1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9660148aa0 con 0x7f966009f560 2020-11-20T12:15:19.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.152+0000 7f965ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f966009f560 0x7f96601485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:19.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.152+0000 7f965ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f966009f560 0x7f96601485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49910/0 (socket says 172.21.15.187:49910) 2020-11-20T12:15:19.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.152+0000 7f965ffff700 1 -- 172.21.15.187:0/843418271 learned_addr learned my addr 172.21.15.187:0/843418271 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f965ffff700 1 -- 172.21.15.187:0/843418271 >> v2:172.21.15.5:3300/0 conn(0x7f96600bfbe0 msgr2=0x7f96601480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f965ffff700 1 --2- 172.21.15.187:0/843418271 >> v2:172.21.15.5:3300/0 conn(0x7f96600bfbe0 0x7f96601480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f965ffff700 1 -- 172.21.15.187:0/843418271 >> v1:172.21.15.5:6789/0 conn(0x7f96600bf800 legacy=0x7f966014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f965ffff700 1 -- 172.21.15.187:0/843418271 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966014cbd0 con 0x7f966009f560 2020-11-20T12:15:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f965ffff700 1 --2- 172.21.15.187:0/843418271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966009f560 0x7f96601485b0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7f966009f2b0 tx=0x7f9658009290).ready entity=mon.1 client_cookie=83cf72e941c890b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:19.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f965effd700 1 -- 172.21.15.187:0/843418271 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965800bc10 con 0x7f966009f560 2020-11-20T12:15:19.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f965effd700 1 -- 172.21.15.187:0/843418271 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965800bd70 con 0x7f966009f560 2020-11-20T12:15:19.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f965effd700 1 -- 172.21.15.187:0/843418271 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658011610 con 0x7f966009f560 2020-11-20T12:15:19.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f9664ca1700 1 -- 172.21.15.187:0/843418271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966009f560 msgr2=0x7f96601485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:19.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f9664ca1700 1 --2- 172.21.15.187:0/843418271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966009f560 0x7f96601485b0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7f966009f2b0 tx=0x7f9658009290).stop 2020-11-20T12:15:19.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f9664ca1700 1 -- 172.21.15.187:0/843418271 shutdown_connections 2020-11-20T12:15:19.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f9664ca1700 1 --2- 172.21.15.187:0/843418271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966009f560 0x7f96601485b0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:19.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f9664ca1700 1 --2- 172.21.15.187:0/843418271 >> v2:172.21.15.5:3300/0 conn(0x7f96600bfbe0 0x7f96601480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:19.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f9664ca1700 1 -- 172.21.15.187:0/843418271 >> 172.21.15.187:0/843418271 conn(0x7f9660068370 msgr2=0x7f96600bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:19.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f9664ca1700 1 -- 172.21.15.187:0/843418271 shutdown_connections 2020-11-20T12:15:19.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.153+0000 7f9664ca1700 1 -- 172.21.15.187:0/843418271 wait complete. 2020-11-20T12:15:19.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.155+0000 7f9664ca1700 1 Processor -- start 2020-11-20T12:15:19.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.155+0000 7f9664ca1700 1 -- start start 2020-11-20T12:15:19.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f9664ca1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f966009f560 0x7f966014c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:19.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f9664ca1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96600bf800 0x7f96601578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:19.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f9664ca1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96601458c0 0x7f9660157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:19.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f9664ca1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9660159320 con 0x7f966009f560 2020-11-20T12:15:19.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f9664ca1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9660158940 con 0x7f96601458c0 2020-11-20T12:15:19.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f9664ca1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9660158be0 con 0x7f96600bf800 2020-11-20T12:15:19.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f965ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96601458c0 0x7f9660157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:19.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f965ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96601458c0 0x7f9660157de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49918/0 (socket says 172.21.15.187:49918) 2020-11-20T12:15:19.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f965ffff700 1 -- 172.21.15.187:0/1869386484 learned_addr learned my addr 172.21.15.187:0/1869386484 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:19.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f965f7fe700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f966009f560 0x7f966014c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:19.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.156+0000 7f9656ffd700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96600bf800 0x7f96601578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:19.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f965ffff700 1 -- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96600bf800 msgr2=0x7f96601578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:19.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f965ffff700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96600bf800 0x7f96601578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:19.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f965ffff700 1 -- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f966009f560 msgr2=0x7f966014c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:19.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f965ffff700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f966009f560 0x7f966014c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:19.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f965ffff700 1 -- 172.21.15.187:0/1869386484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660158e80 con 0x7f96601458c0 2020-11-20T12:15:19.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f965f7fe700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f966009f560 0x7f966014c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:19.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f965ffff700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96601458c0 0x7f9660157de0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7f9660148be0 tx=0x7f9658016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:19.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f9657fff700 1 -- 172.21.15.187:0/1869386484 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658020070 con 0x7f96601458c0 2020-11-20T12:15:19.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f9657fff700 1 -- 172.21.15.187:0/1869386484 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965801c470 con 0x7f96601458c0 2020-11-20T12:15:19.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966015c770 con 0x7f96601458c0 2020-11-20T12:15:19.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f9657fff700 1 -- 172.21.15.187:0/1869386484 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965800bc10 con 0x7f96601458c0 2020-11-20T12:15:19.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.157+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9660158350 con 0x7f96601458c0 2020-11-20T12:15:19.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.159+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9660080050 con 0x7f96601458c0 2020-11-20T12:15:19.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.159+0000 7f9657fff700 1 -- 172.21.15.187:0/1869386484 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9658017070 con 0x7f96601458c0 2020-11-20T12:15:19.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.160+0000 7f9657fff700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f963c056d20 0x7f963c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:19.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.160+0000 7f9657fff700 1 -- 172.21.15.187:0/1869386484 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f96580722a0 con 0x7f96601458c0 2020-11-20T12:15:19.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.160+0000 7f965f7fe700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f963c056d20 0x7f963c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:19.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.161+0000 7f965f7fe700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f963c056d20 0x7f963c0591e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f9650002510 tx=0x7f9650009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:19.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.164+0000 7f9657fff700 1 -- 172.21.15.187:0/1869386484 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f96580475e0 con 0x7f96601458c0 2020-11-20T12:15:19.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.416+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9660159010 con 0x7f96601458c0 2020-11-20T12:15:19.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.416+0000 7f9657fff700 1 -- 172.21.15.187:0/1869386484 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f9658015020 con 0x7f96601458c0 2020-11-20T12:15:19.421 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:19.421 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:19.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f963c056d20 msgr2=0x7f963c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:19.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f963c056d20 0x7f963c0591e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f9650002510 tx=0x7f9650009040).stop 2020-11-20T12:15:19.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96601458c0 msgr2=0x7f9660157de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:19.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96601458c0 0x7f9660157de0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7f9660148be0 tx=0x7f9658016000).stop 2020-11-20T12:15:19.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 shutdown_connections 2020-11-20T12:15:19.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f963c056d20 0x7f963c0591e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:19.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f966009f560 0x7f966014c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:19.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96601458c0 0x7f9660157de0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:19.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.419+0000 7f9664ca1700 1 --2- 172.21.15.187:0/1869386484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96600bf800 0x7f96601578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:19.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.420+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 >> 172.21.15.187:0/1869386484 conn(0x7f9660068370 msgr2=0x7f96601426c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:19.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.420+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 shutdown_connections 2020-11-20T12:15:19.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:19.420+0000 7f9664ca1700 1 -- 172.21.15.187:0/1869386484 wait complete. 2020-11-20T12:15:19.426 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:19.646 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:15:15.259490+0000', 'modified': '2020-11-20T12:15:17.285078+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8125}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8125': {'gid': 8125, 'name': 'a', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.187:6835/2812802720', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2812802720}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2812802720}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:15:19.647 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:15:19.647 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:15:19.922 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8125,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.6378625219999998,"mdsmap_epoch":95,"osdmap_epoch":108,"osdmap_epoch_barrier":107,"uptime":17.255206201} 2020-11-20T12:15:19.923 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8125, "mdsmap_epoch": 95, "osdmap_epoch": 108, "osdmap_epoch_barrier": 107, "rank_uptime": 4.637862522, "state": "up:active", "uptime": 17.255206201, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:15:19.924 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:15:19.924 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:15:19.924 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:15:19.925 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:15:19.925 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:15:19.925 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:15:19.926 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:15:19.926 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:15:19.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:19.927 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:15:19.980 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:15:19.981 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:15:19.981 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:15:19.981 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:15:19.981 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:15:19.982 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:15:19.982 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:15:19.982 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:15:19.982 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:15:19.983 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:15:19.983 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:15:19.983 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:15:19.983 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:15:19.983 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:15:19.984 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:15:19.984 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42118sec preferred_lft 42118sec 2020-11-20T12:15:19.984 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:15:19.984 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:15:19.985 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:15:19.985 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:15:19.985 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:15:19.985 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:15:19.985 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:15:19.986 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:15:19.986 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:15:19.986 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:15:19.986 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:15:19.986 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:15:19.987 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:15:19.987 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:15:19.987 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:15:19.987 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:15:19.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:19.989 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:15:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:15:20.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:20.090 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:15:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:15:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:15:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:15:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:15:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:15:20.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:20.151 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:15:20.151 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:15:20.152 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:15:20.152 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:15:20.152 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:15:20.153 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:15:20.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:20.410 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:15:20.479 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:15:20.479 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:15:20.480 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:15:20.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:20.481 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:15:20.590 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:15:20.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.830+0000 7f9e6e27f700 1 Processor -- start 2020-11-20T12:15:20.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.831+0000 7f9e6e27f700 1 -- start start 2020-11-20T12:15:20.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.832+0000 7f9e6e27f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e680654a0 0x7f9e680b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.832+0000 7f9e6e27f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e680b08e0 con 0x7f9e68065880 2020-11-20T12:15:20.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.832+0000 7f9e6e27f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e680b0b80 con 0x7f9e680b6c10 2020-11-20T12:15:20.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.832+0000 7f9e6e27f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e680b05f0 con 0x7f9e680654a0 2020-11-20T12:15:20.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.833+0000 7f9e6c81c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9e680b6c10 0x7f9e680666e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46532/0 (socket says 172.21.15.187:46532) 2020-11-20T12:15:20.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.833+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1256952765 learned_addr learned my addr 172.21.15.187:0/1256952765 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:20.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.833+0000 7f9e67fff700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e680654a0 0x7f9e680b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.833+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542639327 0 0) 0x7f9e680b0b80 con 0x7f9e680b6c10 2020-11-20T12:15:20.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.833+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e48003240 con 0x7f9e680b6c10 2020-11-20T12:15:20.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 679977262 0 0) 0x7f9e48003240 con 0x7f9e680b6c10 2020-11-20T12:15:20.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e680654a0 msgr2=0x7f9e680b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:20.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e680654a0 0x7f9e680b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 >> v1:172.21.15.5:6789/0 conn(0x7f9e68065880 legacy=0x7f9e68144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e68145570 con 0x7f9e680b6c10 2020-11-20T12:15:20.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f9e60001860 con 0x7f9e680b6c10 2020-11-20T12:15:20.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 >> v1:172.21.15.187:6789/0 conn(0x7f9e680b6c10 legacy=0x7f9e680666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e48003730 0x7f9e48003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e680654a0 0x7f9e480044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e480049d0 0x7f9e48004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e680b05f0 con 0x7f9e48003730 2020-11-20T12:15:20.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e48004220 con 0x7f9e480049d0 2020-11-20T12:15:20.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800b730 con 0x7f9e680654a0 2020-11-20T12:15:20.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.834+0000 7f9e6c81c700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e480049d0 0x7f9e48004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e5f7fe700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e680654a0 0x7f9e480044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e67fff700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e48003730 0x7f9e48003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e680654a0 msgr2=0x7f9e480044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:20.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e6c81c700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e680654a0 0x7f9e480044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e48003730 msgr2=0x7f9e48003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:20.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e6c81c700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e48003730 0x7f9e48003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1256952765 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e48003240 con 0x7f9e480049d0 2020-11-20T12:15:20.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e6c81c700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e480049d0 0x7f9e48004e50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f9e6000c550 tx=0x7f9e60007a50).ready entity=mon.1 client_cookie=10db86fbb5880a24 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:20.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60005b80 con 0x7f9e480049d0 2020-11-20T12:15:20.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60005ce0 con 0x7f9e480049d0 2020-11-20T12:15:20.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.835+0000 7f9e677fe700 1 -- 172.21.15.187:0/1256952765 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60007300 con 0x7f9e480049d0 2020-11-20T12:15:20.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.836+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1256952765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e480049d0 msgr2=0x7f9e48004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.836+0000 7f9e6e27f700 1 --2- 172.21.15.187:0/1256952765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e480049d0 0x7f9e48004e50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f9e6000c550 tx=0x7f9e60007a50).stop 2020-11-20T12:15:20.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.836+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1256952765 reap_dead start 2020-11-20T12:15:20.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.836+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1256952765 shutdown_connections 2020-11-20T12:15:20.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.836+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1256952765 >> 172.21.15.187:0/1256952765 conn(0x7f9e680bacd0 msgr2=0x7f9e680bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:20.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.836+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1256952765 shutdown_connections 2020-11-20T12:15:20.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.836+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1256952765 wait complete. 2020-11-20T12:15:20.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.838+0000 7f9e6e27f700 1 Processor -- start 2020-11-20T12:15:20.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.838+0000 7f9e6e27f700 1 -- start start 2020-11-20T12:15:20.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6e27f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e68137a20 0x7f9e6809bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6e27f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e68137e00 0x7f9e6809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6e27f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6809fc90 0x7f9e680a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6e27f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e68144930 con 0x7f9e68137a20 2020-11-20T12:15:20.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6e27f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6809c580 con 0x7f9e6809fc90 2020-11-20T12:15:20.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6e27f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6809c820 con 0x7f9e68137e00 2020-11-20T12:15:20.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6c81c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6809fc90 0x7f9e680a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6c81c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6809fc90 0x7f9e680a0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49936/0 (socket says 172.21.15.187:49936) 2020-11-20T12:15:20.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1010503250 learned_addr learned my addr 172.21.15.187:0/1010503250 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:20.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e67fff700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e68137a20 0x7f9e6809bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1010503250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e68137e00 msgr2=0x7f9e6809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e5f7fe700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e68137e00 0x7f9e6809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.839+0000 7f9e6c81c700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e68137e00 0x7f9e6809cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1010503250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e68137a20 msgr2=0x7f9e6809bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e6c81c700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e68137a20 0x7f9e6809bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e6c81c700 1 -- 172.21.15.187:0/1010503250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6809cac0 con 0x7f9e6809fc90 2020-11-20T12:15:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e6c81c700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6809fc90 0x7f9e680a0110 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f9e60000f40 tx=0x7f9e60015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e64ff9700 1 -- 172.21.15.187:0/1010503250 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600072e0 con 0x7f9e6809fc90 2020-11-20T12:15:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e64ff9700 1 -- 172.21.15.187:0/1010503250 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e600138b0 con 0x7f9e6809fc90 2020-11-20T12:15:20.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6813afc0 con 0x7f9e6809fc90 2020-11-20T12:15:20.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e64ff9700 1 -- 172.21.15.187:0/1010503250 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60013c50 con 0x7f9e6809fc90 2020-11-20T12:15:20.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.840+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6809d610 con 0x7f9e6809fc90 2020-11-20T12:15:20.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.841+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e68063600 con 0x7f9e6809fc90 2020-11-20T12:15:20.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.842+0000 7f9e64ff9700 1 -- 172.21.15.187:0/1010503250 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9e6001d070 con 0x7f9e6809fc90 2020-11-20T12:15:20.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.842+0000 7f9e64ff9700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e44056d20 0x7f9e440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.842+0000 7f9e64ff9700 1 -- 172.21.15.187:0/1010503250 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9e60071800 con 0x7f9e6809fc90 2020-11-20T12:15:20.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.843+0000 7f9e67fff700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e44056d20 0x7f9e440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.843+0000 7f9e67fff700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e44056d20 0x7f9e440591e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f9e58006a90 tx=0x7f9e58008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:20.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:20.846+0000 7f9e64ff9700 1 -- 172.21.15.187:0/1010503250 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9e60046c50 con 0x7f9e6809fc90 2020-11-20T12:15:21.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.034+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e6809d610 con 0x7f9e6809fc90 2020-11-20T12:15:21.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.037+0000 7f9e64ff9700 1 -- 172.21.15.187:0/1010503250 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16791 (secure 0 0 0) 0x7f9e6809d610 con 0x7f9e6809fc90 2020-11-20T12:15:21.042 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:21.043 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:15:17.285828+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:15:12.843783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2020-11-20T12:15:14.270455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:15:21.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.039+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e44056d20 msgr2=0x7f9e440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.039+0000 7f9e6e27f700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e44056d20 0x7f9e440591e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f9e58006a90 tx=0x7f9e58008040).stop 2020-11-20T12:15:21.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.039+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6809fc90 msgr2=0x7f9e680a0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.039+0000 7f9e6e27f700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6809fc90 0x7f9e680a0110 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f9e60000f40 tx=0x7f9e60015000).stop 2020-11-20T12:15:21.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.040+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 shutdown_connections 2020-11-20T12:15:21.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.040+0000 7f9e6e27f700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e44056d20 0x7f9e440591e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.040+0000 7f9e6e27f700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e68137a20 0x7f9e6809bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.040+0000 7f9e6e27f700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6809fc90 0x7f9e680a0110 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.040+0000 7f9e6e27f700 1 --2- 172.21.15.187:0/1010503250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e68137e00 0x7f9e6809cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.040+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 >> 172.21.15.187:0/1010503250 conn(0x7f9e680bacd0 msgr2=0x7f9e68067210 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:21.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.040+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 shutdown_connections 2020-11-20T12:15:21.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.040+0000 7f9e6e27f700 1 -- 172.21.15.187:0/1010503250 wait complete. 2020-11-20T12:15:21.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:15:12.843783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:15:14.270455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:15:21.068 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:15:21.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.315+0000 7fd4a18b4700 1 Processor -- start 2020-11-20T12:15:21.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.316+0000 7fd4a18b4700 1 -- start start 2020-11-20T12:15:21.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.317+0000 7fd4a18b4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 0x7fd49c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.317+0000 7fd4a18b4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd49c09ca90 0x7fd49c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.317+0000 7fd4a18b4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0b13e0 con 0x7fd49c065190 2020-11-20T12:15:21.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.317+0000 7fd4a18b4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c14ab30 con 0x7fd49c09ca90 2020-11-20T12:15:21.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.317+0000 7fd4a18b4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c062c10 con 0x7fd49c065f90 2020-11-20T12:15:21.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.317+0000 7fd4a10b3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd49c09ca90 0x7fd49c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.318+0000 7fd4a10b3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd49c09ca90 0x7fd49c09cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49946/0 (socket says 172.21.15.187:49946) 2020-11-20T12:15:21.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.318+0000 7fd4a10b3700 1 -- 172.21.15.187:0/727946572 learned_addr learned my addr 172.21.15.187:0/727946572 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:21.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.318+0000 7fd493fff700 1 --2- 172.21.15.187:0/727946572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 0x7fd49c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.318+0000 7fd4a10b3700 1 -- 172.21.15.187:0/727946572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 msgr2=0x7fd49c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:21.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.318+0000 7fd4a10b3700 1 --2- 172.21.15.187:0/727946572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 0x7fd49c065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.318+0000 7fd4a10b3700 1 -- 172.21.15.187:0/727946572 >> v1:172.21.15.5:6789/0 conn(0x7fd49c065190 legacy=0x7fd49c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:21.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.318+0000 7fd4a10b3700 1 -- 172.21.15.187:0/727946572 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c0b1640 con 0x7fd49c09ca90 2020-11-20T12:15:21.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.318+0000 7fd4a10b3700 1 --2- 172.21.15.187:0/727946572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c09cf20 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7fd49c14ac70 tx=0x7fd494006eb0).ready entity=mon.1 client_cookie=9113ee8322d25b39 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:21.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4937fe700 1 -- 172.21.15.187:0/727946572 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494009920 con 0x7fd49c09ca90 2020-11-20T12:15:21.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4937fe700 1 -- 172.21.15.187:0/727946572 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd494009a80 con 0x7fd49c09ca90 2020-11-20T12:15:21.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4937fe700 1 -- 172.21.15.187:0/727946572 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494009e00 con 0x7fd49c09ca90 2020-11-20T12:15:21.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4a18b4700 1 -- 172.21.15.187:0/727946572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 msgr2=0x7fd49c09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/727946572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c09cf20 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7fd49c14ac70 tx=0x7fd494006eb0).stop 2020-11-20T12:15:21.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4a18b4700 1 -- 172.21.15.187:0/727946572 shutdown_connections 2020-11-20T12:15:21.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/727946572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c09cf20 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/727946572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 0x7fd49c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.319+0000 7fd4a18b4700 1 -- 172.21.15.187:0/727946572 >> 172.21.15.187:0/727946572 conn(0x7fd49c0bab30 msgr2=0x7fd49c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:21.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.320+0000 7fd4a18b4700 1 -- 172.21.15.187:0/727946572 shutdown_connections 2020-11-20T12:15:21.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.320+0000 7fd4a18b4700 1 -- 172.21.15.187:0/727946572 wait complete. 2020-11-20T12:15:21.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.321+0000 7fd4a18b4700 1 Processor -- start 2020-11-20T12:15:21.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.322+0000 7fd4a18b4700 1 -- start start 2020-11-20T12:15:21.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.322+0000 7fd4a18b4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd49c065190 0x7fd49c143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.322+0000 7fd4a18b4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 0x7fd49c0990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.322+0000 7fd4a18b4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c0995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.322+0000 7fd4a18b4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c066e40 con 0x7fd49c065190 2020-11-20T12:15:21.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.322+0000 7fd4a18b4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0a0950 con 0x7fd49c09ca90 2020-11-20T12:15:21.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.322+0000 7fd4a18b4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c09e060 con 0x7fd49c065f90 2020-11-20T12:15:21.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.323+0000 7fd4a10b3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c0995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.323+0000 7fd4a10b3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c0995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49952/0 (socket says 172.21.15.187:49952) 2020-11-20T12:15:21.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.323+0000 7fd4a10b3700 1 -- 172.21.15.187:0/3313992515 learned_addr learned my addr 172.21.15.187:0/3313992515 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:21.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.323+0000 7fd493fff700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 0x7fd49c0990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.323+0000 7fd4a10b3700 1 -- 172.21.15.187:0/3313992515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 msgr2=0x7fd49c0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.323+0000 7fd4a10b3700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 0x7fd49c0990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.323+0000 7fd4a10b3700 1 -- 172.21.15.187:0/3313992515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd49c065190 msgr2=0x7fd49c143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.323+0000 7fd4a10b3700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd49c065190 0x7fd49c143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:21.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.324+0000 7fd4a10b3700 1 -- 172.21.15.187:0/3313992515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c09e300 con 0x7fd49c09ca90 2020-11-20T12:15:21.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.324+0000 7fd4a10b3700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c0995b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fd49400b1b0 tx=0x7fd494015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:21.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.324+0000 7fd490ff9700 1 -- 172.21.15.187:0/3313992515 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49401d5d0 con 0x7fd49c09ca90 2020-11-20T12:15:21.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.324+0000 7fd490ff9700 1 -- 172.21.15.187:0/3313992515 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49401d730 con 0x7fd49c09ca90 2020-11-20T12:15:21.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.324+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c09e5a0 con 0x7fd49c09ca90 2020-11-20T12:15:21.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.324+0000 7fd490ff9700 1 -- 172.21.15.187:0/3313992515 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49401dad0 con 0x7fd49c09ca90 2020-11-20T12:15:21.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.324+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c143750 con 0x7fd49c09ca90 2020-11-20T12:15:21.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.326+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c061a20 con 0x7fd49c09ca90 2020-11-20T12:15:21.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.326+0000 7fd490ff9700 1 -- 172.21.15.187:0/3313992515 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd494025030 con 0x7fd49c09ca90 2020-11-20T12:15:21.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.327+0000 7fd490ff9700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd474056d20 0x7fd4740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.327+0000 7fd490ff9700 1 -- 172.21.15.187:0/3313992515 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd494071b50 con 0x7fd49c09ca90 2020-11-20T12:15:21.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.327+0000 7fd4a08b2700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd474056d20 0x7fd4740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.327+0000 7fd4a08b2700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd474056d20 0x7fd4740591e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fd498006fd0 tx=0x7fd498006d30).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:21.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.331+0000 7fd490ff9700 1 -- 172.21.15.187:0/3313992515 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd494047030 con 0x7fd49c09ca90 2020-11-20T12:15:21.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.521+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd49c09e730 con 0x7fd49c09ca90 2020-11-20T12:15:21.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.522+0000 7fd490ff9700 1 -- 172.21.15.187:0/3313992515 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v108) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd494023030 con 0x7fd49c09ca90 2020-11-20T12:15:21.526 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:15:21.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd474056d20 msgr2=0x7fd4740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd474056d20 0x7fd4740591e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fd498006fd0 tx=0x7fd498006d30).stop 2020-11-20T12:15:21.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 msgr2=0x7fd49c0995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c0995b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fd49400b1b0 tx=0x7fd494015000).stop 2020-11-20T12:15:21.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 shutdown_connections 2020-11-20T12:15:21.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd474056d20 0x7fd4740591e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd49c065190 0x7fd49c143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:21.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd49c09ca90 0x7fd49c0995b0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 --2- 172.21.15.187:0/3313992515 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd49c065f90 0x7fd49c0990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.524+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 >> 172.21.15.187:0/3313992515 conn(0x7fd49c0bab30 msgr2=0x7fd49c1427c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:21.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.525+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 shutdown_connections 2020-11-20T12:15:21.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.525+0000 7fd4a18b4700 1 -- 172.21.15.187:0/3313992515 wait complete. 2020-11-20T12:15:21.553 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:15:21.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.798+0000 7f7f38d2a700 1 Processor -- start 2020-11-20T12:15:21.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.798+0000 7f7f38d2a700 1 -- start start 2020-11-20T12:15:21.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.799+0000 7f7f38d2a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f34065300 0x7f7f340656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.799+0000 7f7f38d2a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f3409a8a0 0x7f7f34062400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.800+0000 7f7f38d2a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f34066420 con 0x7f7f34068030 2020-11-20T12:15:21.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.800+0000 7f7f38d2a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f340b7380 con 0x7f7f34065300 2020-11-20T12:15:21.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.800+0000 7f7f38d2a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f34061f60 con 0x7f7f3409a8a0 2020-11-20T12:15:21.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.800+0000 7f7f3259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f34065300 0x7f7f340656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.800+0000 7f7f3259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f34065300 0x7f7f340656e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49956/0 (socket says 172.21.15.187:49956) 2020-11-20T12:15:21.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.800+0000 7f7f3259c700 1 -- 172.21.15.187:0/4019781701 learned_addr learned my addr 172.21.15.187:0/4019781701 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:21.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.800+0000 7f7f29d9b700 1 --2- 172.21.15.187:0/4019781701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f3409a8a0 0x7f7f34062400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.801+0000 7f7f3259c700 1 -- 172.21.15.187:0/4019781701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f3409a8a0 msgr2=0x7f7f34062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:21.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.801+0000 7f7f3259c700 1 --2- 172.21.15.187:0/4019781701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f3409a8a0 0x7f7f34062400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.801+0000 7f7f3259c700 1 -- 172.21.15.187:0/4019781701 >> v1:172.21.15.5:6789/0 conn(0x7f7f34068030 legacy=0x7f7f34146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.801+0000 7f7f3259c700 1 -- 172.21.15.187:0/4019781701 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f340b6fd0 con 0x7f7f34065300 2020-11-20T12:15:21.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.801+0000 7f7f3259c700 1 --2- 172.21.15.187:0/4019781701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f34065300 0x7f7f340656e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f7f24006fd0 tx=0x7f7f24006eb0).ready entity=mon.1 client_cookie=4ea558a6582a2ae6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:21.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.801+0000 7f7f31d9b700 1 -- 172.21.15.187:0/4019781701 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24009960 con 0x7f7f34065300 2020-11-20T12:15:21.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f31d9b700 1 -- 172.21.15.187:0/4019781701 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f24009ac0 con 0x7f7f34065300 2020-11-20T12:15:21.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f31d9b700 1 -- 172.21.15.187:0/4019781701 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24009e40 con 0x7f7f34065300 2020-11-20T12:15:21.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f38d2a700 1 -- 172.21.15.187:0/4019781701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f34065300 msgr2=0x7f7f340656e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/4019781701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f34065300 0x7f7f340656e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f7f24006fd0 tx=0x7f7f24006eb0).stop 2020-11-20T12:15:21.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f38d2a700 1 -- 172.21.15.187:0/4019781701 shutdown_connections 2020-11-20T12:15:21.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/4019781701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f34065300 0x7f7f340656e0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/4019781701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f3409a8a0 0x7f7f34062400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f38d2a700 1 -- 172.21.15.187:0/4019781701 >> 172.21.15.187:0/4019781701 conn(0x7f7f340bacf0 msgr2=0x7f7f340bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:21.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f38d2a700 1 -- 172.21.15.187:0/4019781701 shutdown_connections 2020-11-20T12:15:21.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.802+0000 7f7f38d2a700 1 -- 172.21.15.187:0/4019781701 wait complete. 2020-11-20T12:15:21.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.804+0000 7f7f38d2a700 1 Processor -- start 2020-11-20T12:15:21.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.804+0000 7f7f38d2a700 1 -- start start 2020-11-20T12:15:21.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.805+0000 7f7f38d2a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f34065300 0x7f7f340b9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.805+0000 7f7f38d2a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f34068030 0x7f7f3414a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.805+0000 7f7f38d2a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f3409a8a0 0x7f7f3415b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.805+0000 7f7f38d2a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340ba6f0 con 0x7f7f34065300 2020-11-20T12:15:21.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.805+0000 7f7f38d2a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f34144f50 con 0x7f7f3409a8a0 2020-11-20T12:15:21.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.805+0000 7f7f38d2a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340ba1b0 con 0x7f7f34068030 2020-11-20T12:15:21.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f32d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f3409a8a0 0x7f7f3415b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f32d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f3409a8a0 0x7f7f3415b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49966/0 (socket says 172.21.15.187:49966) 2020-11-20T12:15:21.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f32d9d700 1 -- 172.21.15.187:0/2728463568 learned_addr learned my addr 172.21.15.187:0/2728463568 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:21.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f3259c700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f34065300 0x7f7f340b9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f29d9b700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f34068030 0x7f7f3414a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f32d9d700 1 -- 172.21.15.187:0/2728463568 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f34068030 msgr2=0x7f7f3414a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f32d9d700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f34068030 0x7f7f3414a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f32d9d700 1 -- 172.21.15.187:0/2728463568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f34065300 msgr2=0x7f7f340b9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:21.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.806+0000 7f7f32d9d700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f34065300 0x7f7f340b9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:21.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.807+0000 7f7f32d9d700 1 -- 172.21.15.187:0/2728463568 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f340ba450 con 0x7f7f3409a8a0 2020-11-20T12:15:21.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.807+0000 7f7f32d9d700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f3409a8a0 0x7f7f3415b8d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f7f2c00ca60 tx=0x7f7f2c007f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:21.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.807+0000 7f7f2affd700 1 -- 172.21.15.187:0/2728463568 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c00a830 con 0x7f7f3409a8a0 2020-11-20T12:15:21.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.807+0000 7f7f2affd700 1 -- 172.21.15.187:0/2728463568 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c00a990 con 0x7f7f3409a8a0 2020-11-20T12:15:21.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.807+0000 7f7f2affd700 1 -- 172.21.15.187:0/2728463568 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c00ad10 con 0x7f7f3409a8a0 2020-11-20T12:15:21.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.807+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3414a650 con 0x7f7f3409a8a0 2020-11-20T12:15:21.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.807+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3415c000 con 0x7f7f3409a8a0 2020-11-20T12:15:21.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.808+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f34044f60 con 0x7f7f3409a8a0 2020-11-20T12:15:21.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.809+0000 7f7f2affd700 1 -- 172.21.15.187:0/2728463568 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7f2c019030 con 0x7f7f3409a8a0 2020-11-20T12:15:21.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.810+0000 7f7f2affd700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f10056d50 0x7f7f10059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:21.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.810+0000 7f7f2affd700 1 -- 172.21.15.187:0/2728463568 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7f2c06f8b0 con 0x7f7f3409a8a0 2020-11-20T12:15:21.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.810+0000 7f7f3259c700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f10056d50 0x7f7f10059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:21.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.811+0000 7f7f3259c700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f10056d50 0x7f7f10059210 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f7f340a0ec0 tx=0x7f7f24015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:21.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:21.814+0000 7f7f2affd700 1 -- 172.21.15.187:0/2728463568 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7f2c043fd0 con 0x7f7f3409a8a0 2020-11-20T12:15:21.851 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:15:21.865 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:15:22.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.004+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7f3405cae0 con 0x7f7f3409a8a0 2020-11-20T12:15:22.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.004+0000 7f7f2affd700 1 -- 172.21.15.187:0/2728463568 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v108) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7f2c044ca0 con 0x7f7f3409a8a0 2020-11-20T12:15:22.009 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:15:22.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f10056d50 msgr2=0x7f7f10059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f10056d50 0x7f7f10059210 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f7f340a0ec0 tx=0x7f7f24015000).stop 2020-11-20T12:15:22.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f3409a8a0 msgr2=0x7f7f3415b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f3409a8a0 0x7f7f3415b8d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f7f2c00ca60 tx=0x7f7f2c007f40).stop 2020-11-20T12:15:22.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 shutdown_connections 2020-11-20T12:15:22.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7f10056d50 0x7f7f10059210 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f34065300 0x7f7f340b9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f3409a8a0 0x7f7f3415b8d0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 --2- 172.21.15.187:0/2728463568 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7f34068030 0x7f7f3414a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 >> 172.21.15.187:0/2728463568 conn(0x7f7f340bacf0 msgr2=0x7f7f34067900 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:22.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 shutdown_connections 2020-11-20T12:15:22.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.007+0000 7f7f38d2a700 1 -- 172.21.15.187:0/2728463568 wait complete. 2020-11-20T12:15:22.032 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:15:22.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.318+0000 7fe37759e700 1 Processor -- start 2020-11-20T12:15:22.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.319+0000 7fe37759e700 1 -- start start 2020-11-20T12:15:22.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.320+0000 7fe37759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe378065190 0x7fe378065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.320+0000 7fe37759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3780b13e0 con 0x7fe37809ca90 2020-11-20T12:15:22.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.320+0000 7fe37759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe378062c10 con 0x7fe378065f90 2020-11-20T12:15:22.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.320+0000 7fe37759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37814ab30 con 0x7fe378065190 2020-11-20T12:15:22.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.321+0000 7fe375d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe378065f90 0x7fe378143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46578/0 (socket says 172.21.15.187:46578) 2020-11-20T12:15:22.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.321+0000 7fe375d9b700 1 -- 172.21.15.187:0/3874055818 learned_addr learned my addr 172.21.15.187:0/3874055818 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:22.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.321+0000 7fe37659c700 1 --2- 172.21.15.187:0/3874055818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe378065190 0x7fe378065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.322+0000 7fe37659c700 1 -- 172.21.15.187:0/3874055818 >> v1:172.21.15.187:6789/0 conn(0x7fe378065f90 legacy=0x7fe378143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.322+0000 7fe37659c700 1 -- 172.21.15.187:0/3874055818 >> v1:172.21.15.5:6789/0 conn(0x7fe37809ca90 legacy=0x7fe37809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.322+0000 7fe37559a700 1 -- 172.21.15.187:0/3874055818 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874797515 0 0) 0x7fe3780b13e0 con 0x7fe37809ca90 2020-11-20T12:15:22.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.322+0000 7fe37659c700 1 -- 172.21.15.187:0/3874055818 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3780b1640 con 0x7fe378065190 2020-11-20T12:15:22.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.323+0000 7fe37659c700 1 --2- 172.21.15.187:0/3874055818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe378065190 0x7fe378065570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe36800b9d0 tx=0x7fe368006df0).ready entity=mon.2 client_cookie=7633a2a9972b82fc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.330+0000 7fe37559a700 1 -- 172.21.15.187:0/3874055818 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368009900 con 0x7fe378065190 2020-11-20T12:15:22.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.330+0000 7fe37559a700 1 -- 172.21.15.187:0/3874055818 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe368009a60 con 0x7fe378065190 2020-11-20T12:15:22.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.330+0000 7fe37559a700 1 -- 172.21.15.187:0/3874055818 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368009de0 con 0x7fe378065190 2020-11-20T12:15:22.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.330+0000 7fe37759e700 1 -- 172.21.15.187:0/3874055818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe378065190 msgr2=0x7fe378065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.330+0000 7fe37759e700 1 --2- 172.21.15.187:0/3874055818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe378065190 0x7fe378065570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe36800b9d0 tx=0x7fe368006df0).stop 2020-11-20T12:15:22.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.331+0000 7fe37759e700 1 -- 172.21.15.187:0/3874055818 shutdown_connections 2020-11-20T12:15:22.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.331+0000 7fe37759e700 1 --2- 172.21.15.187:0/3874055818 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe378065190 0x7fe378065570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.331+0000 7fe37759e700 1 -- 172.21.15.187:0/3874055818 >> 172.21.15.187:0/3874055818 conn(0x7fe3780bab30 msgr2=0x7fe3780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:22.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.331+0000 7fe37759e700 1 -- 172.21.15.187:0/3874055818 shutdown_connections 2020-11-20T12:15:22.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.331+0000 7fe37759e700 1 -- 172.21.15.187:0/3874055818 wait complete. 2020-11-20T12:15:22.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.333+0000 7fe37759e700 1 Processor -- start 2020-11-20T12:15:22.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.333+0000 7fe37759e700 1 -- start start 2020-11-20T12:15:22.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.334+0000 7fe37759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe378065190 0x7fe37808ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.334+0000 7fe37759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe378065f90 0x7fe378144db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.334+0000 7fe37759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe37809ca90 0x7fe3781452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.334+0000 7fe37759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37815a580 con 0x7fe378065f90 2020-11-20T12:15:22.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.334+0000 7fe37759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3780b4460 con 0x7fe378065190 2020-11-20T12:15:22.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.334+0000 7fe37759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3780b4700 con 0x7fe37809ca90 2020-11-20T12:15:22.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.335+0000 7fe37659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe378065190 0x7fe37808ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.335+0000 7fe37659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe378065190 0x7fe37808ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49976/0 (socket says 172.21.15.187:49976) 2020-11-20T12:15:22.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.335+0000 7fe37659c700 1 -- 172.21.15.187:0/3256853152 learned_addr learned my addr 172.21.15.187:0/3256853152 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:22.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.335+0000 7fe376d9d700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe37809ca90 0x7fe3781452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.335+0000 7fe375d9b700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe378065f90 0x7fe378144db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe37659c700 1 -- 172.21.15.187:0/3256853152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe37809ca90 msgr2=0x7fe3781452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe37659c700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe37809ca90 0x7fe3781452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe37659c700 1 -- 172.21.15.187:0/3256853152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe378065f90 msgr2=0x7fe378144db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe37659c700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe378065f90 0x7fe378144db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe37659c700 1 -- 172.21.15.187:0/3256853152 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe378149f40 con 0x7fe378065190 2020-11-20T12:15:22.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe37659c700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe378065190 0x7fe37808ebe0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7fe3780b1570 tx=0x7fe368015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe3667fc700 1 -- 172.21.15.187:0/3256853152 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368011ea0 con 0x7fe378065190 2020-11-20T12:15:22.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe3667fc700 1 -- 172.21.15.187:0/3256853152 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36801d5d0 con 0x7fe378065190 2020-11-20T12:15:22.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe3667fc700 1 -- 172.21.15.187:0/3256853152 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36801d950 con 0x7fe378065190 2020-11-20T12:15:22.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37808f0d0 con 0x7fe378065190 2020-11-20T12:15:22.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.336+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37815a260 con 0x7fe378065190 2020-11-20T12:15:22.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.338+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe378061a20 con 0x7fe378065190 2020-11-20T12:15:22.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.338+0000 7fe3667fc700 1 -- 172.21.15.187:0/3256853152 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe368025030 con 0x7fe378065190 2020-11-20T12:15:22.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.339+0000 7fe3667fc700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe34c056d50 0x7fe34c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.339+0000 7fe3667fc700 1 -- 172.21.15.187:0/3256853152 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe3680719a0 con 0x7fe378065190 2020-11-20T12:15:22.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.339+0000 7fe375d9b700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe34c056d50 0x7fe34c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.340+0000 7fe375d9b700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe34c056d50 0x7fe34c059210 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fe378099b30 tx=0x7fe36c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.343+0000 7fe3667fc700 1 -- 172.21.15.187:0/3256853152 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe368046ea0 con 0x7fe378065190 2020-11-20T12:15:22.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.531+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe3780bb8d0 con 0x7fe378065190 2020-11-20T12:15:22.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.532+0000 7fe3667fc700 1 -- 172.21.15.187:0/3256853152 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe368023070 con 0x7fe378065190 2020-11-20T12:15:22.536 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:15:22.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe34c056d50 msgr2=0x7fe34c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe34c056d50 0x7fe34c059210 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fe378099b30 tx=0x7fe36c009040).stop 2020-11-20T12:15:22.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe378065190 msgr2=0x7fe37808ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe378065190 0x7fe37808ebe0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7fe3780b1570 tx=0x7fe368015000).stop 2020-11-20T12:15:22.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 shutdown_connections 2020-11-20T12:15:22.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe34c056d50 0x7fe34c059210 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe37809ca90 0x7fe3781452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe378065190 0x7fe37808ebe0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 --2- 172.21.15.187:0/3256853152 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe378065f90 0x7fe378144db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 >> 172.21.15.187:0/3256853152 conn(0x7fe3780bab30 msgr2=0x7fe37809b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:22.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 shutdown_connections 2020-11-20T12:15:22.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.535+0000 7fe37759e700 1 -- 172.21.15.187:0/3256853152 wait complete. 2020-11-20T12:15:22.565 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:15:22.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.811+0000 7fda63fff700 1 Processor -- start 2020-11-20T12:15:22.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.812+0000 7fda63fff700 1 -- start start 2020-11-20T12:15:22.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.813+0000 7fda63fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda64065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.813+0000 7fda63fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fda64065680 0x7fda6414c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.813+0000 7fda63fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda64150390 con 0x7fda640b1380 2020-11-20T12:15:22.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.813+0000 7fda63fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda641511e0 con 0x7fda64065680 2020-11-20T12:15:22.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.813+0000 7fda63fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6414d760 con 0x7fda640b1760 2020-11-20T12:15:22.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.813+0000 7fda637fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fda64065680 0x7fda6414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.814+0000 7fda637fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fda64065680 0x7fda6414c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49988/0 (socket says 172.21.15.187:49988) 2020-11-20T12:15:22.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.814+0000 7fda637fe700 1 -- 172.21.15.187:0/598581084 learned_addr learned my addr 172.21.15.187:0/598581084 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:22.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.814+0000 7fda5a7fc700 1 --2- 172.21.15.187:0/598581084 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda64065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.814+0000 7fda637fe700 1 -- 172.21.15.187:0/598581084 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 msgr2=0x7fda64065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:22.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.814+0000 7fda637fe700 1 --2- 172.21.15.187:0/598581084 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda64065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.814+0000 7fda637fe700 1 -- 172.21.15.187:0/598581084 >> v1:172.21.15.5:6789/0 conn(0x7fda640b1380 legacy=0x7fda64150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:22.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.814+0000 7fda637fe700 1 -- 172.21.15.187:0/598581084 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda64150630 con 0x7fda64065680 2020-11-20T12:15:22.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.814+0000 7fda637fe700 1 --2- 172.21.15.187:0/598581084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda64065680 0x7fda6414c260 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7fda5c006fd0 tx=0x7fda5c006eb0).ready entity=mon.1 client_cookie=961d62a36dfa9349 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.815+0000 7fda627fc700 1 -- 172.21.15.187:0/598581084 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c009890 con 0x7fda64065680 2020-11-20T12:15:22.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.815+0000 7fda627fc700 1 -- 172.21.15.187:0/598581084 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c0099f0 con 0x7fda64065680 2020-11-20T12:15:22.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.815+0000 7fda627fc700 1 -- 172.21.15.187:0/598581084 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c009d70 con 0x7fda64065680 2020-11-20T12:15:22.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.815+0000 7fda63fff700 1 -- 172.21.15.187:0/598581084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda64065680 msgr2=0x7fda6414c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.815+0000 7fda63fff700 1 --2- 172.21.15.187:0/598581084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda64065680 0x7fda6414c260 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7fda5c006fd0 tx=0x7fda5c006eb0).stop 2020-11-20T12:15:22.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.816+0000 7fda63fff700 1 -- 172.21.15.187:0/598581084 shutdown_connections 2020-11-20T12:15:22.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.816+0000 7fda63fff700 1 --2- 172.21.15.187:0/598581084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda64065680 0x7fda6414c260 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.816+0000 7fda63fff700 1 --2- 172.21.15.187:0/598581084 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda64065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.816+0000 7fda63fff700 1 -- 172.21.15.187:0/598581084 >> 172.21.15.187:0/598581084 conn(0x7fda640bab30 msgr2=0x7fda640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:22.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.816+0000 7fda63fff700 1 -- 172.21.15.187:0/598581084 shutdown_connections 2020-11-20T12:15:22.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.816+0000 7fda63fff700 1 -- 172.21.15.187:0/598581084 wait complete. 2020-11-20T12:15:22.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.818+0000 7fda63fff700 1 Processor -- start 2020-11-20T12:15:22.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.818+0000 7fda63fff700 1 -- start start 2020-11-20T12:15:22.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.819+0000 7fda63fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda640b1380 0x7fda6415fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.819+0000 7fda63fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda640b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.819+0000 7fda63fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda6415ff70 0x7fda640b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.819+0000 7fda63fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda641604e0 con 0x7fda6415ff70 2020-11-20T12:15:22.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.819+0000 7fda63fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64160780 con 0x7fda640b1380 2020-11-20T12:15:22.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.819+0000 7fda63fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640b36c0 con 0x7fda640b1760 2020-11-20T12:15:22.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda5a7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda640b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda62ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda640b1380 0x7fda6415fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda637fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda6415ff70 0x7fda640b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda5a7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda640b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:50878/0 (socket says 172.21.15.187:50878) 2020-11-20T12:15:22.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda5a7fc700 1 -- 172.21.15.187:0/27494363 learned_addr learned my addr 172.21.15.187:0/27494363 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:22.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda62ffd700 1 -- 172.21.15.187:0/27494363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 msgr2=0x7fda640b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda62ffd700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda640b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda62ffd700 1 -- 172.21.15.187:0/27494363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda6415ff70 msgr2=0x7fda640b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda62ffd700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda6415ff70 0x7fda640b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.820+0000 7fda62ffd700 1 -- 172.21.15.187:0/27494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640b4560 con 0x7fda640b1380 2020-11-20T12:15:22.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.821+0000 7fda62ffd700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda640b1380 0x7fda6415fa80 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7fda5400cae0 tx=0x7fda54007ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.821+0000 7fda5b7fe700 1 -- 172.21.15.187:0/27494363 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5400a520 con 0x7fda640b1380 2020-11-20T12:15:22.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.821+0000 7fda5b7fe700 1 -- 172.21.15.187:0/27494363 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5400a680 con 0x7fda640b1380 2020-11-20T12:15:22.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.821+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda640b4810 con 0x7fda640b1380 2020-11-20T12:15:22.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.821+0000 7fda5b7fe700 1 -- 172.21.15.187:0/27494363 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5400aa20 con 0x7fda640b1380 2020-11-20T12:15:22.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.821+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda640b4210 con 0x7fda640b1380 2020-11-20T12:15:22.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.822+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda640b3880 con 0x7fda640b1380 2020-11-20T12:15:22.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.823+0000 7fda5b7fe700 1 -- 172.21.15.187:0/27494363 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fda54019030 con 0x7fda640b1380 2020-11-20T12:15:22.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.824+0000 7fda5b7fe700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda40056d20 0x7fda400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.824+0000 7fda5b7fe700 1 -- 172.21.15.187:0/27494363 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fda5406f610 con 0x7fda640b1380 2020-11-20T12:15:22.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.824+0000 7fda5a7fc700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda40056d20 0x7fda400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.825+0000 7fda5a7fc700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda40056d20 0x7fda400591e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fda640b0d40 tx=0x7fda4c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:22.828+0000 7fda5b7fe700 1 -- 172.21.15.187:0/27494363 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fda54043fd0 con 0x7fda640b1380 2020-11-20T12:15:23.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.012+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fda64061a20 con 0x7fda640b1380 2020-11-20T12:15:23.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.013+0000 7fda5b7fe700 1 -- 172.21.15.187:0/27494363 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+10 (secure 0 0 0) 0x7fda54044a00 con 0x7fda640b1380 2020-11-20T12:15:23.017 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:15:23.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda40056d20 msgr2=0x7fda400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda40056d20 0x7fda400591e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fda640b0d40 tx=0x7fda4c008040).stop 2020-11-20T12:15:23.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda640b1380 msgr2=0x7fda6415fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda640b1380 0x7fda6415fa80 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7fda5400cae0 tx=0x7fda54007ef0).stop 2020-11-20T12:15:23.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 shutdown_connections 2020-11-20T12:15:23.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fda40056d20 0x7fda400591e0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda6415ff70 0x7fda640b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda640b1380 0x7fda6415fa80 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 --2- 172.21.15.187:0/27494363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fda640b1760 0x7fda640b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.015+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 >> 172.21.15.187:0/27494363 conn(0x7fda640bab30 msgr2=0x7fda640ba260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:23.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.016+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 shutdown_connections 2020-11-20T12:15:23.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.016+0000 7fda63fff700 1 -- 172.21.15.187:0/27494363 wait complete. 2020-11-20T12:15:23.043 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:15:23.043 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:23.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.273+0000 7fd9f9117700 1 Processor -- start 2020-11-20T12:15:23.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.273+0000 7fd9f9117700 1 -- start start 2020-11-20T12:15:23.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f9117700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9f4065190 0x7fd9f4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:23.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f9117700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9f4065f90 0x7fd9f4143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:23.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f9117700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 0x7fd9f409ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:23.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f9117700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f414ab30 con 0x7fd9f4065f90 2020-11-20T12:15:23.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f9117700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f40b13e0 con 0x7fd9f4065190 2020-11-20T12:15:23.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f9117700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f4062c10 con 0x7fd9f4143c70 2020-11-20T12:15:23.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9f4065190 0x7fd9f4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:23.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9f4065190 0x7fd9f4065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50000/0 (socket says 172.21.15.187:50000) 2020-11-20T12:15:23.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.275+0000 7fd9f3fff700 1 -- 172.21.15.187:0/4265102549 learned_addr learned my addr 172.21.15.187:0/4265102549 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:23.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f37fe700 1 --2- 172.21.15.187:0/4265102549 >> v2:172.21.15.5:3300/0 conn(0x7fd9f4065f90 0x7fd9f4143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:23.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f8916700 1 --2- 172.21.15.187:0/4265102549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 0x7fd9f409ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:23.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f3fff700 1 -- 172.21.15.187:0/4265102549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 msgr2=0x7fd9f409ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:23.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f3fff700 1 --2- 172.21.15.187:0/4265102549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 0x7fd9f409ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f3fff700 1 -- 172.21.15.187:0/4265102549 >> v2:172.21.15.5:3300/0 conn(0x7fd9f4065f90 msgr2=0x7fd9f4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:23.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f3fff700 1 --2- 172.21.15.187:0/4265102549 >> v2:172.21.15.5:3300/0 conn(0x7fd9f4065f90 0x7fd9f4143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f3fff700 1 -- 172.21.15.187:0/4265102549 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40b1640 con 0x7fd9f4065190 2020-11-20T12:15:23.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f37fe700 1 --2- 172.21.15.187:0/4265102549 >> v2:172.21.15.5:3300/0 conn(0x7fd9f4065f90 0x7fd9f4143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:23.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.276+0000 7fd9f3fff700 1 --2- 172.21.15.187:0/4265102549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065190 0x7fd9f4065570 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7fd9f405c8b0 tx=0x7fd9e4006eb0).ready entity=mon.1 client_cookie=974ae0328c610cd4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:23.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f2ffd700 1 -- 172.21.15.187:0/4265102549 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4009960 con 0x7fd9f4065190 2020-11-20T12:15:23.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f2ffd700 1 -- 172.21.15.187:0/4265102549 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e4009ac0 con 0x7fd9f4065190 2020-11-20T12:15:23.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f2ffd700 1 -- 172.21.15.187:0/4265102549 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4009e40 con 0x7fd9f4065190 2020-11-20T12:15:23.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f9117700 1 -- 172.21.15.187:0/4265102549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065190 msgr2=0x7fd9f4065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f9117700 1 --2- 172.21.15.187:0/4265102549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065190 0x7fd9f4065570 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7fd9f405c8b0 tx=0x7fd9e4006eb0).stop 2020-11-20T12:15:23.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f9117700 1 -- 172.21.15.187:0/4265102549 shutdown_connections 2020-11-20T12:15:23.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f9117700 1 --2- 172.21.15.187:0/4265102549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 0x7fd9f409ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f9117700 1 --2- 172.21.15.187:0/4265102549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065190 0x7fd9f4065570 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f9117700 1 --2- 172.21.15.187:0/4265102549 >> v2:172.21.15.5:3300/0 conn(0x7fd9f4065f90 0x7fd9f4143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.277+0000 7fd9f9117700 1 -- 172.21.15.187:0/4265102549 >> 172.21.15.187:0/4265102549 conn(0x7fd9f40bab30 msgr2=0x7fd9f40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:23.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.278+0000 7fd9f9117700 1 -- 172.21.15.187:0/4265102549 shutdown_connections 2020-11-20T12:15:23.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.278+0000 7fd9f9117700 1 -- 172.21.15.187:0/4265102549 wait complete. 2020-11-20T12:15:23.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.279+0000 7fd9f9117700 1 Processor -- start 2020-11-20T12:15:23.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.279+0000 7fd9f9117700 1 -- start start 2020-11-20T12:15:23.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.280+0000 7fd9f9117700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9f4065190 0x7fd9f4146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:23.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.280+0000 7fd9f9117700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065f90 0x7fd9f4141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:23.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.280+0000 7fd9f9117700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 0x7fd9f4145180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.280+0000 7fd9f9117700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f409f2b0 con 0x7fd9f4065190 2020-11-20T12:15:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.280+0000 7fd9f9117700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f409f550 con 0x7fd9f4065f90 2020-11-20T12:15:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.280+0000 7fd9f9117700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f409f7f0 con 0x7fd9f4143c70 2020-11-20T12:15:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065f90 0x7fd9f4141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:23.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065f90 0x7fd9f4141050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50006/0 (socket says 172.21.15.187:50006) 2020-11-20T12:15:23.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 -- 172.21.15.187:0/1523814033 learned_addr learned my addr 172.21.15.187:0/1523814033 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:23.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f3fff700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9f4065190 0x7fd9f4146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:23.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 -- 172.21.15.187:0/1523814033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 msgr2=0x7fd9f4145180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 0x7fd9f4145180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:23.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 -- 172.21.15.187:0/1523814033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9f4065190 msgr2=0x7fd9f4146860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9f4065190 0x7fd9f4146860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 -- 172.21.15.187:0/1523814033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f4146d50 con 0x7fd9f4065f90 2020-11-20T12:15:23.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f3fff700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9f4065190 0x7fd9f4146860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:23.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.281+0000 7fd9f37fe700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065f90 0x7fd9f4141050 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7fd9e800cb50 tx=0x7fd9e8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:23.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.282+0000 7fd9d3fff700 1 -- 172.21.15.187:0/1523814033 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80077d0 con 0x7fd9f4065f90 2020-11-20T12:15:23.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.282+0000 7fd9d3fff700 1 -- 172.21.15.187:0/1523814033 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e8007930 con 0x7fd9f4065f90 2020-11-20T12:15:23.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.282+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f4141540 con 0x7fd9f4065f90 2020-11-20T12:15:23.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.282+0000 7fd9d3fff700 1 -- 172.21.15.187:0/1523814033 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8007cb0 con 0x7fd9f4065f90 2020-11-20T12:15:23.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.282+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f409ee80 con 0x7fd9f4065f90 2020-11-20T12:15:23.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.283+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f409f9b0 con 0x7fd9f4065f90 2020-11-20T12:15:23.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.284+0000 7fd9d3fff700 1 -- 172.21.15.187:0/1523814033 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd9e8011620 con 0x7fd9f4065f90 2020-11-20T12:15:23.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.284+0000 7fd9d3fff700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9cc056d20 0x7fd9cc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:23.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.284+0000 7fd9d3fff700 1 -- 172.21.15.187:0/1523814033 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd9e806ffa0 con 0x7fd9f4065f90 2020-11-20T12:15:23.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.285+0000 7fd9f3fff700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9cc056d20 0x7fd9cc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:23.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.285+0000 7fd9f3fff700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9cc056d20 0x7fd9cc0591e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7fd9e400b1b0 tx=0x7fd9e4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:23.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.288+0000 7fd9d3fff700 1 -- 172.21.15.187:0/1523814033 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd9e8045390 con 0x7fd9f4065f90 2020-11-20T12:15:23.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.530+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9f4061a20 con 0x7fd9f4065f90 2020-11-20T12:15:23.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.532+0000 7fd9d3fff700 1 -- 172.21.15.187:0/1523814033 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fd9e80489b0 con 0x7fd9f4065f90 2020-11-20T12:15:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:23.536 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:23.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9cc056d20 msgr2=0x7fd9cc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9cc056d20 0x7fd9cc0591e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7fd9e400b1b0 tx=0x7fd9e4015000).stop 2020-11-20T12:15:23.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065f90 msgr2=0x7fd9f4141050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065f90 0x7fd9f4141050 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7fd9e800cb50 tx=0x7fd9e8008040).stop 2020-11-20T12:15:23.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 shutdown_connections 2020-11-20T12:15:23.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd9cc056d20 0x7fd9cc0591e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9f4065190 0x7fd9f4146860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9f4143c70 0x7fd9f4145180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:23.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 --2- 172.21.15.187:0/1523814033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9f4065f90 0x7fd9f4141050 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 >> 172.21.15.187:0/1523814033 conn(0x7fd9f40bab30 msgr2=0x7fd9f4141f10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:23.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 shutdown_connections 2020-11-20T12:15:23.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:23.534+0000 7fd9f9117700 1 -- 172.21.15.187:0/1523814033 wait complete. 2020-11-20T12:15:23.541 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:23.761 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:15:15.259490+0000', 'modified': '2020-11-20T12:15:17.285078+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8125}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8125': {'gid': 8125, 'name': 'a', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.187:6835/2812802720', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2812802720}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2812802720}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:15:24.034 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:15:24.035 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:15:24.319 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8125,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":9.0316118089999993,"mdsmap_epoch":95,"osdmap_epoch":108,"osdmap_epoch_barrier":107,"uptime":21.648958987} 2020-11-20T12:15:24.320 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8125, "mdsmap_epoch": 95, "osdmap_epoch": 108, "osdmap_epoch_barrier": 107, "rank_uptime": 9.031611809, "state": "up:active", "uptime": 21.648958987, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:15:24.321 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:15:24.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:24.322 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:15:24.376 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:15:24.377 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:15:24.513 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:15:24.514 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:15:24.552 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:15:24.552 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:15:24.559 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:15:24.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:24.560 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:15:24.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:24.669 INFO:teuthology.orchestra.run.smithi187:> sudo python3 -c ' 2020-11-20T12:15:24.669 INFO:teuthology.orchestra.run.smithi187:> import glob 2020-11-20T12:15:24.670 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:15:24.670 INFO:teuthology.orchestra.run.smithi187:> import json 2020-11-20T12:15:24.671 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:15:24.671 INFO:teuthology.orchestra.run.smithi187:> def get_id_to_dir(): 2020-11-20T12:15:24.671 INFO:teuthology.orchestra.run.smithi187:> result = {} 2020-11-20T12:15:24.672 INFO:teuthology.orchestra.run.smithi187:> for dir in glob.glob("/sys/kernel/debug/ceph/*"): 2020-11-20T12:15:24.672 INFO:teuthology.orchestra.run.smithi187:> mds_sessions_lines = open(os.path.join(dir, "mds_sessions")).readlines() 2020-11-20T12:15:24.672 INFO:teuthology.orchestra.run.smithi187:> client_id = mds_sessions_lines[1].split()[1].strip('"'"'"'"'"') 2020-11-20T12:15:24.673 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:15:24.673 INFO:teuthology.orchestra.run.smithi187:> result[client_id] = dir 2020-11-20T12:15:24.674 INFO:teuthology.orchestra.run.smithi187:> return result 2020-11-20T12:15:24.674 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:15:24.674 INFO:teuthology.orchestra.run.smithi187:> print(json.dumps(get_id_to_dir())) 2020-11-20T12:15:24.675 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:15:24.810 INFO:teuthology.orchestra.run.smithi187.stdout:{"0": "/sys/kernel/debug/ceph/099efcab-1650-4909-9b03-145da903b26d.client8260"} 2020-11-20T12:15:24.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:24.820 INFO:teuthology.orchestra.run.smithi187:> sudo python3 -c ' 2020-11-20T12:15:24.821 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:15:24.822 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:15:24.822 INFO:teuthology.orchestra.run.smithi187:> print(open(os.path.join("/sys/kernel/debug/ceph/099efcab-1650-4909-9b03-145da903b26d.client8260", "mds_sessions")).read()) 2020-11-20T12:15:24.823 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:15:24.948 INFO:teuthology.orchestra.run.smithi187.stdout:global_id 8260 2020-11-20T12:15:24.949 INFO:teuthology.orchestra.run.smithi187.stdout:name "0" 2020-11-20T12:15:24.949 INFO:teuthology.orchestra.run.smithi187.stdout:mds.0 open 2020-11-20T12:15:24.949 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:24.958 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:15:24.958 INFO:teuthology.orchestra.run.smithi187:> import time 2020-11-20T12:15:24.959 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:15:24.959 INFO:teuthology.orchestra.run.smithi187:> with open("/home/ubuntu/cephtest/mnt.0/open_file", '"'"'w'"'"') as f: 2020-11-20T12:15:24.960 INFO:teuthology.orchestra.run.smithi187:> f.write('"'"'content'"'"') 2020-11-20T12:15:24.960 INFO:teuthology.orchestra.run.smithi187:> f.flush() 2020-11-20T12:15:24.960 INFO:teuthology.orchestra.run.smithi187:> f.write('"'"'content2'"'"') 2020-11-20T12:15:24.961 INFO:teuthology.orchestra.run.smithi187:> while True: 2020-11-20T12:15:24.961 INFO:teuthology.orchestra.run.smithi187:> time.sleep(1) 2020-11-20T12:15:24.961 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:15:24.970 INFO:teuthology.orchestra.run.smithi187:> sudo ls /home/ubuntu/cephtest/mnt.0/open_file 2020-11-20T12:15:25.053 INFO:teuthology.orchestra.run.smithi187.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/open_file': No such file or directory 2020-11-20T12:15:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-11-20T12:15:26.059 INFO:teuthology.orchestra.run.smithi187:> sudo ls /home/ubuntu/cephtest/mnt.0/open_file 2020-11-20T12:15:26.178 INFO:teuthology.orchestra.run.smithi187.stdout:/home/ubuntu/cephtest/mnt.0/open_file 2020-11-20T12:15:26.182 DEBUG:tasks.cephfs.mount:File open_file became visible from 0 after 1s 2020-11-20T12:15:26.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:15:26.184 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo dd if=/dev/urandom of=open_file bs=1M conv=fdatasync count=8 seek=0) 2020-11-20T12:15:26.499 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records in 2020-11-20T12:15:26.499 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records out 2020-11-20T12:15:26.499 INFO:teuthology.orchestra.run.smithi187.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.186475 s, 45.0 MB/s 2020-11-20T12:15:26.507 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:15:26.507 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:15:26.508 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:15:26.508 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:15:26.508 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/open_file").st_ino) 2020-11-20T12:15:26.509 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:15:26.701 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627776 2020-11-20T12:15:26.890 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:27.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.135+0000 7ff02de2f700 1 Processor -- start 2020-11-20T12:15:27.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.135+0000 7ff02de2f700 1 -- start start 2020-11-20T12:15:27.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.136+0000 7ff02de2f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff02814d270 0x7ff0281573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.136+0000 7ff02de2f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0280658a0 con 0x7ff0280b1760 2020-11-20T12:15:27.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.136+0000 7ff02de2f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff028150390 con 0x7ff0280b1380 2020-11-20T12:15:27.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.136+0000 7ff02de2f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0281511e0 con 0x7ff02814d270 2020-11-20T12:15:27.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.137+0000 7ff02d62e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff02814d270 0x7ff0281573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.137+0000 7ff02d62e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff02814d270 0x7ff0281573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50022/0 (socket says 172.21.15.187:50022) 2020-11-20T12:15:27.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.137+0000 7ff02d62e700 1 -- 172.21.15.187:0/2846450545 learned_addr learned my addr 172.21.15.187:0/2846450545 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:27.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.138+0000 7ff02d62e700 1 -- 172.21.15.187:0/2846450545 >> v1:172.21.15.187:6789/0 conn(0x7ff0280b1380 legacy=0x7ff028150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.138+0000 7ff0277fe700 1 -- 172.21.15.187:0/2846450545 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240289431 0 0) 0x7ff028150390 con 0x7ff0280b1380 2020-11-20T12:15:27.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.138+0000 7ff02d62e700 1 -- 172.21.15.187:0/2846450545 >> v1:172.21.15.5:6789/0 conn(0x7ff0280b1760 legacy=0x7ff028065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:27.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.138+0000 7ff02d62e700 1 -- 172.21.15.187:0/2846450545 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff028150630 con 0x7ff02814d270 2020-11-20T12:15:27.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.138+0000 7ff02d62e700 1 --2- 172.21.15.187:0/2846450545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff02814d270 0x7ff0281573a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7ff020009d50 tx=0x7ff020009290).ready entity=mon.1 client_cookie=9ca2137cb6d50558 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:27.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.139+0000 7ff0277fe700 1 -- 172.21.15.187:0/2846450545 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02000bc40 con 0x7ff02814d270 2020-11-20T12:15:27.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.139+0000 7ff0277fe700 1 -- 172.21.15.187:0/2846450545 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02000bda0 con 0x7ff02814d270 2020-11-20T12:15:27.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.139+0000 7ff0277fe700 1 -- 172.21.15.187:0/2846450545 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020011610 con 0x7ff02814d270 2020-11-20T12:15:27.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.139+0000 7ff02de2f700 1 -- 172.21.15.187:0/2846450545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff02814d270 msgr2=0x7ff0281573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.139+0000 7ff02de2f700 1 --2- 172.21.15.187:0/2846450545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff02814d270 0x7ff0281573a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7ff020009d50 tx=0x7ff020009290).stop 2020-11-20T12:15:27.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.139+0000 7ff02de2f700 1 -- 172.21.15.187:0/2846450545 shutdown_connections 2020-11-20T12:15:27.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.139+0000 7ff02de2f700 1 --2- 172.21.15.187:0/2846450545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff02814d270 0x7ff0281573a0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.140+0000 7ff02de2f700 1 -- 172.21.15.187:0/2846450545 >> 172.21.15.187:0/2846450545 conn(0x7ff0280bab30 msgr2=0x7ff0280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:27.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.140+0000 7ff02de2f700 1 -- 172.21.15.187:0/2846450545 shutdown_connections 2020-11-20T12:15:27.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.140+0000 7ff02de2f700 1 -- 172.21.15.187:0/2846450545 wait complete. 2020-11-20T12:15:27.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.142+0000 7ff02de2f700 1 Processor -- start 2020-11-20T12:15:27.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.142+0000 7ff02de2f700 1 -- start start 2020-11-20T12:15:27.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff02de2f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0280b1380 0x7ff028148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff02de2f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0280b1760 0x7ff0280b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff02de2f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02814d270 0x7ff0280b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff02de2f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0280b32e0 con 0x7ff02814d270 2020-11-20T12:15:27.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff02de2f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0280b3580 con 0x7ff0280b1760 2020-11-20T12:15:27.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff02de2f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0280b3820 con 0x7ff0280b1380 2020-11-20T12:15:27.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff027fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0280b1760 0x7ff0280b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff027fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0280b1760 0x7ff0280b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50028/0 (socket says 172.21.15.187:50028) 2020-11-20T12:15:27.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.143+0000 7ff027fff700 1 -- 172.21.15.187:0/1263747775 learned_addr learned my addr 172.21.15.187:0/1263747775 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:27.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff02d62e700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02814d270 0x7ff0280b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff02ce2d700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0280b1380 0x7ff028148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff027fff700 1 -- 172.21.15.187:0/1263747775 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0280b1380 msgr2=0x7ff028148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff027fff700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0280b1380 0x7ff028148110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff027fff700 1 -- 172.21.15.187:0/1263747775 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02814d270 msgr2=0x7ff0280b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff027fff700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02814d270 0x7ff0280b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff027fff700 1 -- 172.21.15.187:0/1263747775 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0280b5200 con 0x7ff0280b1760 2020-11-20T12:15:27.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff027fff700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0280b1760 0x7ff0280b4010 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7ff01400c730 tx=0x7ff014007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:27.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff024ff9700 1 -- 172.21.15.187:0/1263747775 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01400e560 con 0x7ff0280b1760 2020-11-20T12:15:27.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff024ff9700 1 -- 172.21.15.187:0/1263747775 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01400e6c0 con 0x7ff0280b1760 2020-11-20T12:15:27.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.144+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0280b4500 con 0x7ff0280b1760 2020-11-20T12:15:27.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.145+0000 7ff024ff9700 1 -- 172.21.15.187:0/1263747775 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01400ea80 con 0x7ff0280b1760 2020-11-20T12:15:27.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.145+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff028149ad0 con 0x7ff0280b1760 2020-11-20T12:15:27.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.146+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0280b39e0 con 0x7ff0280b1760 2020-11-20T12:15:27.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.146+0000 7ff024ff9700 1 -- 172.21.15.187:0/1263747775 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff014010070 con 0x7ff0280b1760 2020-11-20T12:15:27.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.147+0000 7ff024ff9700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d50 0x7ff000059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.148+0000 7ff024ff9700 1 -- 172.21.15.187:0/1263747775 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff01406fb00 con 0x7ff0280b1760 2020-11-20T12:15:27.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.148+0000 7ff02ce2d700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d50 0x7ff000059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.148+0000 7ff02ce2d700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d50 0x7ff000059210 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7ff01c0011a0 tx=0x7ff01c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:27.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.152+0000 7ff024ff9700 1 -- 172.21.15.187:0/1263747775 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff014044f50 con 0x7ff0280b1760 2020-11-20T12:15:27.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.399+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff028061a20 con 0x7ff0280b1760 2020-11-20T12:15:27.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.400+0000 7ff024ff9700 1 -- 172.21.15.187:0/1263747775 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7ff014048570 con 0x7ff0280b1760 2020-11-20T12:15:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:27.405 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:27.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.403+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d50 msgr2=0x7ff000059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.403+0000 7ff02de2f700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d50 0x7ff000059210 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7ff01c0011a0 tx=0x7ff01c009040).stop 2020-11-20T12:15:27.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.403+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0280b1760 msgr2=0x7ff0280b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.403+0000 7ff02de2f700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0280b1760 0x7ff0280b4010 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7ff01400c730 tx=0x7ff014007c30).stop 2020-11-20T12:15:27.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.404+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 shutdown_connections 2020-11-20T12:15:27.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.404+0000 7ff02de2f700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff000056d50 0x7ff000059210 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.404+0000 7ff02de2f700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02814d270 0x7ff0280b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.404+0000 7ff02de2f700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0280b1380 0x7ff028148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.404+0000 7ff02de2f700 1 --2- 172.21.15.187:0/1263747775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0280b1760 0x7ff0280b4010 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.404+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 >> 172.21.15.187:0/1263747775 conn(0x7ff0280bab30 msgr2=0x7ff0280afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:27.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.404+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 shutdown_connections 2020-11-20T12:15:27.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:27.404+0000 7ff02de2f700 1 -- 172.21.15.187:0/1263747775 wait complete. 2020-11-20T12:15:27.410 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:27.434 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2020-11-20T12:15:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:[{"id":8260,"entity":{"name":{"type":"client","num":8260},"addr":{"type":"v1","addr":"192.168.0.1:0","nonce":4134631550}},"state":"open","num_leases":1,"num_caps":2,"request_load_avg":8,"uptime":3.1259500930000002,"requests_in_flight":0,"completed_requests":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":2.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":4.9694600695128779,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"inst":"client.8260 v1:192.168.0.1:0/4134631550","completed_requests":[{"tid":9,"created_ino":"0x0"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"used_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x0000000000007bff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000001f"}},"entity_id":"0","hostname":"smithi187","kernel_version":"5.10.0-rc2-ceph-gac4f7b07f403","root":"/"}}] 2020-11-20T12:15:27.688 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "entity_id": "0", "hostname": "smithi187", "kernel_version": "5.10.0-rc2-ceph-gac4f7b07f403", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000001f" } }, "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 9 } ], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 4134631550, "type": "v1" }, "name": { "num": 8260, "type": "client" } }, "id": 8260, "inst": "client.8260 v1:192.168.0.1:0/4134631550", "num_caps": 2, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 2.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 4.969460069512878 }, "state": "open", "uptime": 3.125950093, "used_inos": [] } ] 2020-11-20T12:15:27.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:15:27.689 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -f open_file) 2020-11-20T12:15:27.791 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:28.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.019+0000 7f621116e700 1 Processor -- start 2020-11-20T12:15:28.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.020+0000 7f621116e700 1 -- start start 2020-11-20T12:15:28.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.021+0000 7f621116e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f620c14d270 0x7f620c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.021+0000 7f621116e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620c0658a0 con 0x7f620c0b1380 2020-11-20T12:15:28.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.021+0000 7f621116e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620c150390 con 0x7f620c0b1760 2020-11-20T12:15:28.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.021+0000 7f621116e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620c1511e0 con 0x7f620c14d270 2020-11-20T12:15:28.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.021+0000 7f621096d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f620c14d270 0x7f620c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.021+0000 7f621096d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f620c14d270 0x7f620c1573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50038/0 (socket says 172.21.15.187:50038) 2020-11-20T12:15:28.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.021+0000 7f621096d700 1 -- 172.21.15.187:0/1865461128 learned_addr learned my addr 172.21.15.187:0/1865461128 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:28.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.023+0000 7f621096d700 1 -- 172.21.15.187:0/1865461128 >> v1:172.21.15.187:6789/0 conn(0x7f620c0b1760 legacy=0x7f620c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.023+0000 7f621096d700 1 -- 172.21.15.187:0/1865461128 >> v1:172.21.15.5:6789/0 conn(0x7f620c0b1380 legacy=0x7f620c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.023+0000 7f621096d700 1 -- 172.21.15.187:0/1865461128 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c150630 con 0x7f620c14d270 2020-11-20T12:15:28.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.022+0000 7f620b7fe700 1 -- 172.21.15.187:0/1865461128 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422056101 0 0) 0x7f620c150390 con 0x7f620c0b1760 2020-11-20T12:15:28.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.023+0000 7f621096d700 1 --2- 172.21.15.187:0/1865461128 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c14d270 0x7f620c1573a0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7f6204009d50 tx=0x7f6204009290).ready entity=mon.1 client_cookie=fc414c6054f94a48 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.023+0000 7f620b7fe700 1 -- 172.21.15.187:0/1865461128 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620400bc80 con 0x7f620c14d270 2020-11-20T12:15:28.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.023+0000 7f620b7fe700 1 -- 172.21.15.187:0/1865461128 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620400bde0 con 0x7f620c14d270 2020-11-20T12:15:28.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.023+0000 7f620b7fe700 1 -- 172.21.15.187:0/1865461128 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6204011610 con 0x7f620c14d270 2020-11-20T12:15:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.024+0000 7f621116e700 1 -- 172.21.15.187:0/1865461128 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c14d270 msgr2=0x7f620c1573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.024+0000 7f621116e700 1 --2- 172.21.15.187:0/1865461128 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c14d270 0x7f620c1573a0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7f6204009d50 tx=0x7f6204009290).stop 2020-11-20T12:15:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.024+0000 7f621116e700 1 -- 172.21.15.187:0/1865461128 shutdown_connections 2020-11-20T12:15:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.024+0000 7f621116e700 1 --2- 172.21.15.187:0/1865461128 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c14d270 0x7f620c1573a0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.024+0000 7f621116e700 1 -- 172.21.15.187:0/1865461128 >> 172.21.15.187:0/1865461128 conn(0x7f620c0bab30 msgr2=0x7f620c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:28.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.024+0000 7f621116e700 1 -- 172.21.15.187:0/1865461128 shutdown_connections 2020-11-20T12:15:28.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.024+0000 7f621116e700 1 -- 172.21.15.187:0/1865461128 wait complete. 2020-11-20T12:15:28.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.026+0000 7f621116e700 1 Processor -- start 2020-11-20T12:15:28.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.026+0000 7f621116e700 1 -- start start 2020-11-20T12:15:28.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.026+0000 7f621116e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f620c0b1380 0x7f620c09c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.026+0000 7f621116e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c0b1760 0x7f620c09bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.026+0000 7f621116e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f620c14d270 0x7f620c147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.026+0000 7f621116e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0b26d0 con 0x7f620c0b1380 2020-11-20T12:15:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f621116e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0b2970 con 0x7f620c0b1760 2020-11-20T12:15:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f621116e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0b2c10 con 0x7f620c14d270 2020-11-20T12:15:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f62037fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c0b1760 0x7f620c09bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f62037fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c0b1760 0x7f620c09bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50040/0 (socket says 172.21.15.187:50040) 2020-11-20T12:15:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f62037fe700 1 -- 172.21.15.187:0/185275462 learned_addr learned my addr 172.21.15.187:0/185275462 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f62037fe700 1 -- 172.21.15.187:0/185275462 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f620c14d270 msgr2=0x7f620c147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f621096d700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f620c14d270 0x7f620c147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f62037fe700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f620c14d270 0x7f620c147ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f62037fe700 1 -- 172.21.15.187:0/185275462 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f620c0b1380 msgr2=0x7f620c09c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f62037fe700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f620c0b1380 0x7f620c09c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.027+0000 7f62037fe700 1 -- 172.21.15.187:0/185275462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c09cc30 con 0x7f620c0b1760 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.028+0000 7f62037fe700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c0b1760 0x7f620c09bea0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7f61f400c6e0 tx=0x7f61f4007a70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.028+0000 7f6208ff9700 1 -- 172.21.15.187:0/185275462 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f400e560 con 0x7f620c0b1760 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.028+0000 7f6208ff9700 1 -- 172.21.15.187:0/185275462 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f400e6c0 con 0x7f620c0b1760 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.028+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c0b4580 con 0x7f620c0b1760 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.028+0000 7f6208ff9700 1 -- 172.21.15.187:0/185275462 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f400ea80 con 0x7f620c0b1760 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.028+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c0b30b0 con 0x7f620c0b1760 2020-11-20T12:15:28.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.029+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c0b2510 con 0x7f620c0b1760 2020-11-20T12:15:28.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.030+0000 7f6208ff9700 1 -- 172.21.15.187:0/185275462 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f61f4010070 con 0x7f620c0b1760 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.031+0000 7f6208ff9700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61e8056d20 0x7f61e80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.031+0000 7f6208ff9700 1 -- 172.21.15.187:0/185275462 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f61f406f840 con 0x7f620c0b1760 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.031+0000 7f620bfff700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61e8056d20 0x7f61e80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.032+0000 7f620bfff700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61e8056d20 0x7f61e80591e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f61fc001330 tx=0x7f61fc007810).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.035+0000 7f6208ff9700 1 -- 172.21.15.187:0/185275462 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f61f4044c30 con 0x7f620c0b1760 2020-11-20T12:15:28.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.264+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f620c066310 con 0x7f620c0b1760 2020-11-20T12:15:28.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.265+0000 7f6208ff9700 1 -- 172.21.15.187:0/185275462 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f61f4048250 con 0x7f620c0b1760 2020-11-20T12:15:28.268 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:28.269 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:28.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61e8056d20 msgr2=0x7f61e80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61e8056d20 0x7f61e80591e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f61fc001330 tx=0x7f61fc007810).stop 2020-11-20T12:15:28.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c0b1760 msgr2=0x7f620c09bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c0b1760 0x7f620c09bea0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7f61f400c6e0 tx=0x7f61f4007a70).stop 2020-11-20T12:15:28.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 shutdown_connections 2020-11-20T12:15:28.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f61e8056d20 0x7f61e80591e0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f620c0b1380 0x7f620c09c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:28.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f620c14d270 0x7f620c147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 --2- 172.21.15.187:0/185275462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c0b1760 0x7f620c09bea0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 >> 172.21.15.187:0/185275462 conn(0x7f620c0bab30 msgr2=0x7f620c149010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:28.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.267+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 shutdown_connections 2020-11-20T12:15:28.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.268+0000 7f621116e700 1 -- 172.21.15.187:0/185275462 wait complete. 2020-11-20T12:15:28.274 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:28.292 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2020-11-20T12:15:28.545 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"num_strays":1}} 2020-11-20T12:15:28.546 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2020-11-20T12:15:28.546 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:28.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.774+0000 7ff3dd337700 1 Processor -- start 2020-11-20T12:15:28.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.774+0000 7ff3dd337700 1 -- start start 2020-11-20T12:15:28.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.775+0000 7ff3dd337700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8065190 0x7ff3d8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.775+0000 7ff3dd337700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff3d8065f90 0x7ff3d8143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.775+0000 7ff3dd337700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff3d8143c70 0x7ff3d809ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.776+0000 7ff3dd337700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d814ab30 con 0x7ff3d8143c70 2020-11-20T12:15:28.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.776+0000 7ff3dd337700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d80b13e0 con 0x7ff3d8065f90 2020-11-20T12:15:28.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.776+0000 7ff3dd337700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8062c10 con 0x7ff3d8065190 2020-11-20T12:15:28.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.776+0000 7ff3cf7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff3d8065f90 0x7ff3d8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.776+0000 7ff3cf7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff3d8065f90 0x7ff3d8143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50052/0 (socket says 172.21.15.187:50052) 2020-11-20T12:15:28.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.776+0000 7ff3cf7fe700 1 -- 172.21.15.187:0/4148551544 learned_addr learned my addr 172.21.15.187:0/4148551544 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:28.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.776+0000 7ff3dcb36700 1 --2- 172.21.15.187:0/4148551544 >> v2:172.21.15.5:3300/0 conn(0x7ff3d8143c70 0x7ff3d809ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.776+0000 7ff3d7fff700 1 --2- 172.21.15.187:0/4148551544 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8065190 0x7ff3d8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.777+0000 7ff3cf7fe700 1 -- 172.21.15.187:0/4148551544 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8065190 msgr2=0x7ff3d8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:28.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.777+0000 7ff3cf7fe700 1 --2- 172.21.15.187:0/4148551544 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8065190 0x7ff3d8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.777+0000 7ff3cf7fe700 1 -- 172.21.15.187:0/4148551544 >> v2:172.21.15.5:3300/0 conn(0x7ff3d8143c70 msgr2=0x7ff3d809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:28.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.777+0000 7ff3cf7fe700 1 --2- 172.21.15.187:0/4148551544 >> v2:172.21.15.5:3300/0 conn(0x7ff3d8143c70 0x7ff3d809ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.777+0000 7ff3cf7fe700 1 -- 172.21.15.187:0/4148551544 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80b1640 con 0x7ff3d8065f90 2020-11-20T12:15:28.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.777+0000 7ff3cf7fe700 1 --2- 172.21.15.187:0/4148551544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065f90 0x7ff3d8143780 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7ff3d80632e0 tx=0x7ff3c0006eb0).ready entity=mon.1 client_cookie=d0ef95567eafd54f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3d77fe700 1 -- 172.21.15.187:0/4148551544 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0009960 con 0x7ff3d8065f90 2020-11-20T12:15:28.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3d77fe700 1 -- 172.21.15.187:0/4148551544 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c0009ac0 con 0x7ff3d8065f90 2020-11-20T12:15:28.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3d77fe700 1 -- 172.21.15.187:0/4148551544 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0009e40 con 0x7ff3d8065f90 2020-11-20T12:15:28.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3dd337700 1 -- 172.21.15.187:0/4148551544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065f90 msgr2=0x7ff3d8143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3dd337700 1 --2- 172.21.15.187:0/4148551544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065f90 0x7ff3d8143780 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7ff3d80632e0 tx=0x7ff3c0006eb0).stop 2020-11-20T12:15:28.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3dd337700 1 -- 172.21.15.187:0/4148551544 shutdown_connections 2020-11-20T12:15:28.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3dd337700 1 --2- 172.21.15.187:0/4148551544 >> v2:172.21.15.5:3300/0 conn(0x7ff3d8143c70 0x7ff3d809ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3dd337700 1 --2- 172.21.15.187:0/4148551544 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8065190 0x7ff3d8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3dd337700 1 --2- 172.21.15.187:0/4148551544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065f90 0x7ff3d8143780 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3dd337700 1 -- 172.21.15.187:0/4148551544 >> 172.21.15.187:0/4148551544 conn(0x7ff3d80bab30 msgr2=0x7ff3d80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:28.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.778+0000 7ff3dd337700 1 -- 172.21.15.187:0/4148551544 shutdown_connections 2020-11-20T12:15:28.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.779+0000 7ff3dd337700 1 -- 172.21.15.187:0/4148551544 wait complete. 2020-11-20T12:15:28.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.780+0000 7ff3dd337700 1 Processor -- start 2020-11-20T12:15:28.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.780+0000 7ff3dd337700 1 -- start start 2020-11-20T12:15:28.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.781+0000 7ff3dd337700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065190 0x7ff3d809f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.781+0000 7ff3dd337700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3d8065f90 0x7ff3d8142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.781+0000 7ff3dd337700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8143c70 0x7ff3d809e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.781+0000 7ff3dd337700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d809fc70 con 0x7ff3d8065f90 2020-11-20T12:15:28.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.781+0000 7ff3dd337700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80b4070 con 0x7ff3d8065190 2020-11-20T12:15:28.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.781+0000 7ff3dd337700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80b4310 con 0x7ff3d8143c70 2020-11-20T12:15:28.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065190 0x7ff3d809f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065190 0x7ff3d809f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50056/0 (socket says 172.21.15.187:50056) 2020-11-20T12:15:28.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3d7fff700 1 -- 172.21.15.187:0/2103586295 learned_addr learned my addr 172.21.15.187:0/2103586295 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:28.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3cf7fe700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3d8065f90 0x7ff3d8142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3dcb36700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8143c70 0x7ff3d809e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3d7fff700 1 -- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8143c70 msgr2=0x7ff3d809e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3d7fff700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8143c70 0x7ff3d809e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3d7fff700 1 -- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3d8065f90 msgr2=0x7ff3d8142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3d7fff700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3d8065f90 0x7ff3d8142930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3d7fff700 1 -- 172.21.15.187:0/2103586295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8142e20 con 0x7ff3d8065190 2020-11-20T12:15:28.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3cf7fe700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3d8065f90 0x7ff3d8142930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:28.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.782+0000 7ff3dcb36700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8143c70 0x7ff3d809e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:28.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.783+0000 7ff3d7fff700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065190 0x7ff3d809f780 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7ff3c800db60 tx=0x7ff3c8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.783+0000 7ff3d4ff9700 1 -- 172.21.15.187:0/2103586295 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8009820 con 0x7ff3d8065190 2020-11-20T12:15:28.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.783+0000 7ff3d4ff9700 1 -- 172.21.15.187:0/2103586295 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c8009980 con 0x7ff3d8065190 2020-11-20T12:15:28.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.783+0000 7ff3d4ff9700 1 -- 172.21.15.187:0/2103586295 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8009d20 con 0x7ff3d8065190 2020-11-20T12:15:28.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.783+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d809eba0 con 0x7ff3d8065190 2020-11-20T12:15:28.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.783+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d8148fe0 con 0x7ff3d8065190 2020-11-20T12:15:28.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.784+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d809e4f0 con 0x7ff3d8065190 2020-11-20T12:15:28.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.785+0000 7ff3d4ff9700 1 -- 172.21.15.187:0/2103586295 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff3c8013690 con 0x7ff3d8065190 2020-11-20T12:15:28.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.786+0000 7ff3d4ff9700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3b4056d20 0x7ff3b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.786+0000 7ff3d4ff9700 1 -- 172.21.15.187:0/2103586295 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff3c8070610 con 0x7ff3d8065190 2020-11-20T12:15:28.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.786+0000 7ff3cf7fe700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3b4056d20 0x7ff3b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.787+0000 7ff3cf7fe700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3b4056d20 0x7ff3b40591e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7ff3d80632e0 tx=0x7ff3c0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:28.790+0000 7ff3d4ff9700 1 -- 172.21.15.187:0/2103586295 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff3c8045a00 con 0x7ff3d8065190 2020-11-20T12:15:29.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.042+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3d8061a20 con 0x7ff3d8065190 2020-11-20T12:15:29.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.043+0000 7ff3d4ff9700 1 -- 172.21.15.187:0/2103586295 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7ff3c8049020 con 0x7ff3d8065190 2020-11-20T12:15:29.047 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:29.047 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:29.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.045+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3b4056d20 msgr2=0x7ff3b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:29.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.045+0000 7ff3dd337700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3b4056d20 0x7ff3b40591e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7ff3d80632e0 tx=0x7ff3c0015000).stop 2020-11-20T12:15:29.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065190 msgr2=0x7ff3d809f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:29.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065190 0x7ff3d809f780 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7ff3c800db60 tx=0x7ff3c8008040).stop 2020-11-20T12:15:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 shutdown_connections 2020-11-20T12:15:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3b4056d20 0x7ff3b40591e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3d8143c70 0x7ff3d809e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3d8065190 0x7ff3d809f780 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 --2- 172.21.15.187:0/2103586295 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3d8065f90 0x7ff3d8142930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 >> 172.21.15.187:0/2103586295 conn(0x7ff3d80bab30 msgr2=0x7ff3d80ba2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:29.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 shutdown_connections 2020-11-20T12:15:29.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.046+0000 7ff3dd337700 1 -- 172.21.15.187:0/2103586295 wait complete. 2020-11-20T12:15:29.053 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:29.072 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:15:29.339 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_created":1}} 2020-11-20T12:15:29.418 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2020-11-20T12:15:29.419 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:29.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.675+0000 7fcbb3360700 1 Processor -- start 2020-11-20T12:15:29.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.675+0000 7fcbb3360700 1 -- start start 2020-11-20T12:15:29.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.677+0000 7fcbb3360700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcbac065190 0x7fcbac065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:29.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.677+0000 7fcbb3360700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac09ca90 0x7fcbac09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:29.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.677+0000 7fcbb3360700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac0b13e0 con 0x7fcbac065f90 2020-11-20T12:15:29.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.677+0000 7fcbb3360700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbac14ab30 con 0x7fcbac065190 2020-11-20T12:15:29.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.677+0000 7fcbb3360700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac062c10 con 0x7fcbac09ca90 2020-11-20T12:15:29.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.678+0000 7fcbb2b5f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac09ca90 0x7fcbac09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:29.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.678+0000 7fcbb235e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcbac065190 0x7fcbac065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:29.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.678+0000 7fcbb2b5f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac09ca90 0x7fcbac09cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:50954/0 (socket says 172.21.15.187:50954) 2020-11-20T12:15:29.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.678+0000 7fcbb2b5f700 1 -- 172.21.15.187:0/1657077453 learned_addr learned my addr 172.21.15.187:0/1657077453 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:29.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.679+0000 7fcbb2b5f700 1 -- 172.21.15.187:0/1657077453 >> v2:172.21.15.5:3300/0 conn(0x7fcbac065190 msgr2=0x7fcbac065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:29.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.679+0000 7fcbb2b5f700 1 --2- 172.21.15.187:0/1657077453 >> v2:172.21.15.5:3300/0 conn(0x7fcbac065190 0x7fcbac065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.679+0000 7fcbb2b5f700 1 -- 172.21.15.187:0/1657077453 >> v1:172.21.15.5:6789/0 conn(0x7fcbac065f90 legacy=0x7fcbac143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:29.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.679+0000 7fcbb2b5f700 1 -- 172.21.15.187:0/1657077453 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0b1640 con 0x7fcbac09ca90 2020-11-20T12:15:29.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.679+0000 7fcbb2b5f700 1 --2- 172.21.15.187:0/1657077453 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac09ca90 0x7fcbac09cf40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fcba4009d50 tx=0x7fcba4009290).ready entity=mon.2 client_cookie=bce1b3a2982d250e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:29.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.680+0000 7fcbb135c700 1 -- 172.21.15.187:0/1657077453 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba400bc50 con 0x7fcbac09ca90 2020-11-20T12:15:29.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.680+0000 7fcbb135c700 1 -- 172.21.15.187:0/1657077453 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba400bdb0 con 0x7fcbac09ca90 2020-11-20T12:15:29.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.680+0000 7fcbb135c700 1 -- 172.21.15.187:0/1657077453 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4011610 con 0x7fcbac09ca90 2020-11-20T12:15:29.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.680+0000 7fcbb3360700 1 -- 172.21.15.187:0/1657077453 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac09ca90 msgr2=0x7fcbac09cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:29.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.680+0000 7fcbb3360700 1 --2- 172.21.15.187:0/1657077453 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac09ca90 0x7fcbac09cf40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fcba4009d50 tx=0x7fcba4009290).stop 2020-11-20T12:15:29.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.681+0000 7fcbb3360700 1 -- 172.21.15.187:0/1657077453 shutdown_connections 2020-11-20T12:15:29.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.681+0000 7fcbb3360700 1 --2- 172.21.15.187:0/1657077453 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac09ca90 0x7fcbac09cf40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.681+0000 7fcbb3360700 1 --2- 172.21.15.187:0/1657077453 >> v2:172.21.15.5:3300/0 conn(0x7fcbac065190 0x7fcbac065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.681+0000 7fcbb3360700 1 -- 172.21.15.187:0/1657077453 >> 172.21.15.187:0/1657077453 conn(0x7fcbac0bab30 msgr2=0x7fcbac0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:29.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.681+0000 7fcbb3360700 1 -- 172.21.15.187:0/1657077453 shutdown_connections 2020-11-20T12:15:29.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.681+0000 7fcbb3360700 1 -- 172.21.15.187:0/1657077453 wait complete. 2020-11-20T12:15:29.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.683+0000 7fcbb3360700 1 Processor -- start 2020-11-20T12:15:29.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.683+0000 7fcbb3360700 1 -- start start 2020-11-20T12:15:29.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.684+0000 7fcbb3360700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbac065190 0x7fcbac099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:29.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.684+0000 7fcbb3360700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac065f90 0x7fcbac0b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:29.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.684+0000 7fcbb3360700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbac09ca90 0x7fcbac1454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:29.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.684+0000 7fcbb3360700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0b2120 con 0x7fcbac065190 2020-11-20T12:15:29.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.684+0000 7fcbb3360700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0b23c0 con 0x7fcbac09ca90 2020-11-20T12:15:29.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.684+0000 7fcbb3360700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0b2660 con 0x7fcbac065f90 2020-11-20T12:15:29.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb2b5f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbac09ca90 0x7fcbac1454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:29.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb2b5f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbac09ca90 0x7fcbac1454a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50072/0 (socket says 172.21.15.187:50072) 2020-11-20T12:15:29.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb2b5f700 1 -- 172.21.15.187:0/424590852 learned_addr learned my addr 172.21.15.187:0/424590852 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:29.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb1b5d700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac065f90 0x7fcbac0b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:29.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb235e700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbac065190 0x7fcbac099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:29.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb2b5f700 1 -- 172.21.15.187:0/424590852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac065f90 msgr2=0x7fcbac0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:29.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb2b5f700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac065f90 0x7fcbac0b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb2b5f700 1 -- 172.21.15.187:0/424590852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbac065190 msgr2=0x7fcbac099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:29.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb2b5f700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbac065190 0x7fcbac099280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.685+0000 7fcbb2b5f700 1 -- 172.21.15.187:0/424590852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac099770 con 0x7fcbac09ca90 2020-11-20T12:15:29.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.686+0000 7fcbb2b5f700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbac09ca90 0x7fcbac1454a0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7fcba4009fd0 tx=0x7fcba4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:29.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.686+0000 7fcb9e7fc700 1 -- 172.21.15.187:0/424590852 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba400bc50 con 0x7fcbac09ca90 2020-11-20T12:15:29.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.686+0000 7fcb9e7fc700 1 -- 172.21.15.187:0/424590852 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba4010c40 con 0x7fcbac09ca90 2020-11-20T12:15:29.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.686+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac067470 con 0x7fcbac09ca90 2020-11-20T12:15:29.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.686+0000 7fcb9e7fc700 1 -- 172.21.15.187:0/424590852 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba401c470 con 0x7fcbac09ca90 2020-11-20T12:15:29.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.686+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac09f9f0 con 0x7fcbac09ca90 2020-11-20T12:15:29.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.687+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac0b2820 con 0x7fcbac09ca90 2020-11-20T12:15:29.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.688+0000 7fcb9e7fc700 1 -- 172.21.15.187:0/424590852 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcba4017070 con 0x7fcbac09ca90 2020-11-20T12:15:29.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.688+0000 7fcb9e7fc700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcb84056d20 0x7fcb840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:29.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.688+0000 7fcb9e7fc700 1 -- 172.21.15.187:0/424590852 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcba4072240 con 0x7fcbac09ca90 2020-11-20T12:15:29.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.689+0000 7fcbb235e700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcb84056d20 0x7fcb840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:29.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.690+0000 7fcbb235e700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcb84056d20 0x7fcb840591e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fcbac0a0b70 tx=0x7fcba0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:29.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.692+0000 7fcb9e7fc700 1 -- 172.21.15.187:0/424590852 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcba4047690 con 0x7fcbac09ca90 2020-11-20T12:15:29.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.935+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbac09fbe0 con 0x7fcbac09ca90 2020-11-20T12:15:29.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.936+0000 7fcb9e7fc700 1 -- 172.21.15.187:0/424590852 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fcba4015020 con 0x7fcbac09ca90 2020-11-20T12:15:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:29.941 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:29.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcb84056d20 msgr2=0x7fcb840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:29.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcb84056d20 0x7fcb840591e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fcbac0a0b70 tx=0x7fcba0008040).stop 2020-11-20T12:15:29.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbac09ca90 msgr2=0x7fcbac1454a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:29.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbac09ca90 0x7fcbac1454a0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7fcba4009fd0 tx=0x7fcba4016000).stop 2020-11-20T12:15:29.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 shutdown_connections 2020-11-20T12:15:29.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcb84056d20 0x7fcb840591e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbac065190 0x7fcbac099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbac09ca90 0x7fcbac1454a0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 --2- 172.21.15.187:0/424590852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcbac065f90 0x7fcbac0b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:29.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.939+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 >> 172.21.15.187:0/424590852 conn(0x7fcbac0bab30 msgr2=0x7fcbac141a70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:29.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.940+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 shutdown_connections 2020-11-20T12:15:29.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:29.940+0000 7fcbb3360700 1 -- 172.21.15.187:0/424590852 wait complete. 2020-11-20T12:15:29.946 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:29.964 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:15:30.206 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:15:30.206 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:15:30.207 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:30.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.436+0000 7f99e759e700 1 Processor -- start 2020-11-20T12:15:30.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.437+0000 7f99e759e700 1 -- start start 2020-11-20T12:15:30.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.438+0000 7f99e759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e80b1380 0x7f99e8150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.438+0000 7f99e759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f99e80b1760 0x7f99e8065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.438+0000 7f99e759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e8150390 con 0x7f99e8065680 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.438+0000 7f99e759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e8150630 con 0x7f99e80b1760 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.438+0000 7f99e759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8068510 con 0x7f99e80b1380 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.438+0000 7f99e6d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f99e8065680 0x7f99e8061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46686/0 (socket says 172.21.15.187:46686) 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.438+0000 7f99e6d9d700 1 -- 172.21.15.187:0/2180213291 learned_addr learned my addr 172.21.15.187:0/2180213291 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.439+0000 7f99ddd9b700 1 --2- 172.21.15.187:0/2180213291 >> v2:172.21.15.5:3300/0 conn(0x7f99e80b1760 0x7f99e8065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.439+0000 7f99e659c700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e80b1380 0x7f99e8150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.439+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502193524 0 0) 0x7f99e8150390 con 0x7f99e8065680 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.439+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c8003240 con 0x7f99e8065680 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1298175663 0 0) 0x7f99c8003240 con 0x7f99e8065680 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e80b1380 msgr2=0x7f99e8150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e80b1380 0x7f99e8150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 >> v2:172.21.15.5:3300/0 conn(0x7f99e80b1760 msgr2=0x7f99e8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 --2- 172.21.15.187:0/2180213291 >> v2:172.21.15.5:3300/0 conn(0x7f99e80b1760 0x7f99e8065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8150fc0 con 0x7f99e8065680 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f99e0001860 con 0x7f99e8065680 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 >> v1:172.21.15.187:6789/0 conn(0x7f99e8065680 legacy=0x7f99e8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e80b1380 0x7f99c80037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c8003d10 0x7f99c80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99c80049d0 0x7f99c8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8150630 con 0x7f99c80049d0 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8068510 con 0x7f99c8003d10 2020-11-20T12:15:30.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c80042d0 con 0x7f99e80b1380 2020-11-20T12:15:30.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.440+0000 7f99ddd9b700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c8003d10 0x7f99c80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99e6d9d700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99c80049d0 0x7f99c8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99e659c700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e80b1380 0x7f99c80037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99ddd9b700 1 -- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e80b1380 msgr2=0x7f99c80037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:30.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99ddd9b700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e80b1380 0x7f99c80037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99ddd9b700 1 -- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99c80049d0 msgr2=0x7f99c8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:30.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99ddd9b700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99c80049d0 0x7f99c8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99ddd9b700 1 -- 172.21.15.187:0/2180213291 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c8003240 con 0x7f99c8003d10 2020-11-20T12:15:30.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99ddd9b700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c8003d10 0x7f99c80044e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7f99e8067750 tx=0x7f99d0006df0).ready entity=mon.1 client_cookie=b8c1bd5c8be9dcf1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:30.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.441+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000b310 con 0x7f99c8003d10 2020-11-20T12:15:30.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d000eae0 con 0x7f99c8003d10 2020-11-20T12:15:30.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e5d9b700 1 -- 172.21.15.187:0/2180213291 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000ee60 con 0x7f99c8003d10 2020-11-20T12:15:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e759e700 1 -- 172.21.15.187:0/2180213291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c8003d10 msgr2=0x7f99c80044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e759e700 1 --2- 172.21.15.187:0/2180213291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c8003d10 0x7f99c80044e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7f99e8067750 tx=0x7f99d0006df0).stop 2020-11-20T12:15:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e6d9d700 1 -- 172.21.15.187:0/2180213291 reap_dead start 2020-11-20T12:15:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e759e700 1 -- 172.21.15.187:0/2180213291 shutdown_connections 2020-11-20T12:15:30.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e759e700 1 -- 172.21.15.187:0/2180213291 >> 172.21.15.187:0/2180213291 conn(0x7f99e80bab30 msgr2=0x7f99e80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e759e700 1 -- 172.21.15.187:0/2180213291 shutdown_connections 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.442+0000 7f99e759e700 1 -- 172.21.15.187:0/2180213291 wait complete. 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.444+0000 7f99e759e700 1 Processor -- start 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.444+0000 7f99e759e700 1 -- start start 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.444+0000 7f99e759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99e815ff90 0x7f99e80b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e8160370 0x7f99e814a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99e81607d0 0x7f99e814ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e814b340 con 0x7f99e81607d0 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80a0370 con 0x7f99e815ff90 2020-11-20T12:15:30.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80b3aa0 con 0x7f99e8160370 2020-11-20T12:15:30.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99e815ff90 0x7f99e80b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99e815ff90 0x7f99e80b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50088/0 (socket says 172.21.15.187:50088) 2020-11-20T12:15:30.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 -- 172.21.15.187:0/4006922969 learned_addr learned my addr 172.21.15.187:0/4006922969 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:30.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 -- 172.21.15.187:0/4006922969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e8160370 msgr2=0x7f99e814a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e8160370 0x7f99e814a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:30.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 -- 172.21.15.187:0/4006922969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99e81607d0 msgr2=0x7f99e814ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99e81607d0 0x7f99e814ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 -- 172.21.15.187:0/4006922969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e80b3d40 con 0x7f99e815ff90 2020-11-20T12:15:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.445+0000 7f99e659c700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99e815ff90 0x7f99e80b34f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f99d8006d00 tx=0x7f99d8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:30.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.446+0000 7f99deffd700 1 -- 172.21.15.187:0/4006922969 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8011660 con 0x7f99e815ff90 2020-11-20T12:15:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.446+0000 7f99deffd700 1 -- 172.21.15.187:0/4006922969 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d80117c0 con 0x7f99e815ff90 2020-11-20T12:15:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.446+0000 7f99deffd700 1 -- 172.21.15.187:0/4006922969 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8011b40 con 0x7f99e815ff90 2020-11-20T12:15:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.446+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e80b3fe0 con 0x7f99e815ff90 2020-11-20T12:15:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.446+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e809ef40 con 0x7f99e815ff90 2020-11-20T12:15:30.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.447+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e80b4170 con 0x7f99e815ff90 2020-11-20T12:15:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.447+0000 7f99deffd700 1 -- 172.21.15.187:0/4006922969 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f99d8013400 con 0x7f99e815ff90 2020-11-20T12:15:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.448+0000 7f99deffd700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99c4056d20 0x7f99c40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.448+0000 7f99deffd700 1 -- 172.21.15.187:0/4006922969 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f99d8070260 con 0x7f99e815ff90 2020-11-20T12:15:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.448+0000 7f99ddd9b700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99c4056d20 0x7f99c40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.449+0000 7f99ddd9b700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99c4056d20 0x7f99c40591e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f99d0016000 tx=0x7f99d0017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:30.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.452+0000 7f99deffd700 1 -- 172.21.15.187:0/4006922969 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f99d8045540 con 0x7f99e815ff90 2020-11-20T12:15:30.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.699+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99e8061a20 con 0x7f99e815ff90 2020-11-20T12:15:30.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.700+0000 7f99deffd700 1 -- 172.21.15.187:0/4006922969 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f99d8017020 con 0x7f99e815ff90 2020-11-20T12:15:30.704 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:30.704 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:30.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99c4056d20 msgr2=0x7f99c40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99c4056d20 0x7f99c40591e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f99d0016000 tx=0x7f99d0017000).stop 2020-11-20T12:15:30.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99e815ff90 msgr2=0x7f99e80b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99e815ff90 0x7f99e80b34f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f99d8006d00 tx=0x7f99d8008040).stop 2020-11-20T12:15:30.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 shutdown_connections 2020-11-20T12:15:30.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f99c4056d20 0x7f99c40591e0 secure :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0x7f99d0016000 tx=0x7f99d0017000).stop 2020-11-20T12:15:30.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f99e81607d0 0x7f99e814ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:30.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99e815ff90 0x7f99e80b34f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 --2- 172.21.15.187:0/4006922969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f99e8160370 0x7f99e814a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:30.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 >> 172.21.15.187:0/4006922969 conn(0x7f99e80bab30 msgr2=0x7f99e814b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:30.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 shutdown_connections 2020-11-20T12:15:30.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:30.703+0000 7f99e759e700 1 -- 172.21.15.187:0/4006922969 wait complete. 2020-11-20T12:15:30.709 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:30.729 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2020-11-20T12:15:31.001 INFO:teuthology.orchestra.run.smithi187.stdout:[{"id":8260,"entity":{"name":{"type":"client","num":8260},"addr":{"type":"v1","addr":"192.168.0.1:0","nonce":4134631550}},"state":"open","num_leases":1,"num_caps":2,"request_load_avg":9,"uptime":6.4388971990000003,"requests_in_flight":0,"completed_requests":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":2.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":5.9242116665920781,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"inst":"client.8260 v1:192.168.0.1:0/4134631550","completed_requests":[{"tid":10,"created_ino":"0x0"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"used_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x0000000000007bff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000001f"}},"entity_id":"0","hostname":"smithi187","kernel_version":"5.10.0-rc2-ceph-gac4f7b07f403","root":"/"}}] 2020-11-20T12:15:31.002 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "entity_id": "0", "hostname": "smithi187", "kernel_version": "5.10.0-rc2-ceph-gac4f7b07f403", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000001f" } }, "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 10 } ], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 4134631550, "type": "v1" }, "name": { "num": 8260, "type": "client" } }, "id": 8260, "inst": "client.8260 v1:192.168.0.1:0/4134631550", "num_caps": 2, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 2.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 5.924211666592078 }, "state": "open", "uptime": 6.438897199, "used_inos": [] } ] 2020-11-20T12:15:31.003 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:15:31.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.077+0000 7fe08629dd00 1 Processor -- start 2020-11-20T12:15:31.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.078+0000 7fe08629dd00 1 -- start start 2020-11-20T12:15:31.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.078+0000 7fe08629dd00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55717caf7380 0x55717caf7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.078+0000 7fe08629dd00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb1ed20 0x55717cb1f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.078+0000 7fe08629dd00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb38ab0 0x55717cbbe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.078+0000 7fe08629dd00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55717ca72810 con 0x55717cb1ed20 2020-11-20T12:15:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.078+0000 7fe08629dd00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55717cb20930 con 0x55717cb38ab0 2020-11-20T12:15:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.078+0000 7fe08629dd00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55717cb20b90 con 0x55717caf7380 2020-11-20T12:15:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe077343700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb38ab0 0x55717cbbe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe077343700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb38ab0 0x55717cbbe4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50106/0 (socket says 172.21.15.187:50106) 2020-11-20T12:15:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe077343700 1 -- 172.21.15.187:0/1411400214 learned_addr learned my addr 172.21.15.187:0/1411400214 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe076341700 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb1ed20 0x55717cb1f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe077343700 1 -- 172.21.15.187:0/1411400214 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55717caf7380 msgr2=0x55717caf7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe077343700 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55717caf7380 0x55717caf7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:31.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe077343700 1 -- 172.21.15.187:0/1411400214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb1ed20 msgr2=0x55717cb1f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:31.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe077343700 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb1ed20 0x55717cb1f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe077343700 1 -- 172.21.15.187:0/1411400214 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55717ca9c4e0 con 0x55717cb38ab0 2020-11-20T12:15:31.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.079+0000 7fe076341700 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb1ed20 0x55717cb1f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:31.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe077343700 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb38ab0 0x55717cbbe4f0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7fe07000a830 tx=0x7fe070014000).ready entity=mon.1 client_cookie=96b8bc6419d0070d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe07533f700 1 -- 172.21.15.187:0/1411400214 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070020070 con 0x55717cb38ab0 2020-11-20T12:15:31.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe07533f700 1 -- 172.21.15.187:0/1411400214 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0700057a0 con 0x55717cb38ab0 2020-11-20T12:15:31.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe07533f700 1 -- 172.21.15.187:0/1411400214 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07001c660 con 0x55717cb38ab0 2020-11-20T12:15:31.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe08629dd00 1 -- 172.21.15.187:0/1411400214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb38ab0 msgr2=0x55717cbbe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb38ab0 0x55717cbbe4f0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7fe07000a830 tx=0x7fe070014000).stop 2020-11-20T12:15:31.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe08629dd00 1 -- 172.21.15.187:0/1411400214 shutdown_connections 2020-11-20T12:15:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb38ab0 0x55717cbbe4f0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55717caf7380 0x55717caf7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1411400214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb1ed20 0x55717cb1f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.080+0000 7fe08629dd00 1 -- 172.21.15.187:0/1411400214 >> 172.21.15.187:0/1411400214 conn(0x55717caf60c0 msgr2=0x55717cb1caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:31.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.081+0000 7fe08629dd00 1 -- 172.21.15.187:0/1411400214 shutdown_connections 2020-11-20T12:15:31.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.081+0000 7fe08629dd00 1 -- 172.21.15.187:0/1411400214 wait complete. 2020-11-20T12:15:31.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.081+0000 7fe08629dd00 1 Processor -- start 2020-11-20T12:15:31.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.081+0000 7fe08629dd00 1 -- start start 2020-11-20T12:15:31.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.082+0000 7fe08629dd00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55717caf7380 0x55717cbc1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.082+0000 7fe08629dd00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb1ed20 0x55717cbc4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.082+0000 7fe08629dd00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb38ab0 0x55717cbbac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.082+0000 7fe08629dd00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55717cbc1700 con 0x55717cb38ab0 2020-11-20T12:15:31.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.082+0000 7fe08629dd00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55717cbc69c0 con 0x55717cb1ed20 2020-11-20T12:15:31.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.082+0000 7fe08629dd00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55717cbc2490 con 0x55717caf7380 2020-11-20T12:15:31.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb1ed20 0x55717cbc4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb1ed20 0x55717cbc4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50110/0 (socket says 172.21.15.187:50110) 2020-11-20T12:15:31.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 -- 172.21.15.187:0/1219882541 learned_addr learned my addr 172.21.15.187:0/1219882541 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:31.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55717caf7380 msgr2=0x55717cbc1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55717caf7380 0x55717cbc1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:31.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe077343700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb38ab0 0x55717cbbac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb38ab0 msgr2=0x55717cbbac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55717cb38ab0 0x55717cbbac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55717ca97ec0 con 0x55717cb1ed20 2020-11-20T12:15:31.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.083+0000 7fe076341700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb1ed20 0x55717cbc4650 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7fe06c00ba60 tx=0x7fe06c005bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.084+0000 7fe0677fe700 1 -- 172.21.15.187:0/1219882541 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c010070 con 0x55717cb1ed20 2020-11-20T12:15:31.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.084+0000 7fe0677fe700 1 -- 172.21.15.187:0/1219882541 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06c0078f0 con 0x55717cb1ed20 2020-11-20T12:15:31.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.084+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55717ca76250 con 0x55717cb1ed20 2020-11-20T12:15:31.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.084+0000 7fe0677fe700 1 -- 172.21.15.187:0/1219882541 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c007c70 con 0x55717cb1ed20 2020-11-20T12:15:31.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.084+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55717cbc4b40 con 0x55717cb1ed20 2020-11-20T12:15:31.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.085+0000 7fe0677fe700 1 -- 172.21.15.187:0/1219882541 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe06c015630 con 0x55717cb1ed20 2020-11-20T12:15:31.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.086+0000 7fe0677fe700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c0559a0 0x7fe05c057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.086+0000 7fe0677fe700 1 -- 172.21.15.187:0/1219882541 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe06c06f3d0 con 0x55717cb1ed20 2020-11-20T12:15:31.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.086+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x55717cbbb5f0 0x55717cbc9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.086+0000 7fe076b42700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c0559a0 0x7fe05c057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.086+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:1 22.0 22:00000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x55717cbc9b10 con 0x55717cbbb5f0 2020-11-20T12:15:31.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.087+0000 7fe077343700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x55717cbbb5f0 0x55717cbc9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.087+0000 7fe076b42700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c0559a0 0x7fe05c057e60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fe06800b1a0 tx=0x7fe0680023a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.087+0000 7fe077343700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x55717cbbb5f0 0x55717cbc9620 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.088+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(1 [pgnls start_epoch 108 out=81b] v108'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7fe070022070 con 0x55717cbbb5f0 2020-11-20T12:15:31.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.088+0000 7fe077343700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe07001c070 0x7fe070027940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.088+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:2 22.4 22:20000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x55717cbc9b10 con 0x7fe07001c070 2020-11-20T12:15:31.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.089+0000 7fe076b42700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe07001c070 0x7fe070027940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.090+0000 7fe076b42700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe07001c070 0x7fe070027940 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.091+0000 7fe076b42700 1 -- 172.21.15.187:0/1219882541 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== osd_op_reply(2 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0680134f0 con 0x7fe07001c070 2020-11-20T12:15:31.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.091+0000 7fe076b42700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fe068010940 0x7fe06800f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.091+0000 7fe076b42700 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:3 22.2 22:40000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x55717cbc9b10 con 0x7fe068010940 2020-11-20T12:15:31.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.092+0000 7fe076341700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fe068010940 0x7fe06800f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.093+0000 7fe076341700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fe068010940 0x7fe06800f3c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.095+0000 7fe076341700 1 -- 172.21.15.187:0/1219882541 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(3 [pgnls start_epoch 108 out=81b] v108'1 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7fe06c046770 con 0x7fe068010940 2020-11-20T12:15:31.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.095+0000 7fe076341700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fe06c0478c0 0x7fe06c0158c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.095+0000 7fe076341700 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] -- osd_op(unknown.0.0:4 22.6 22:60000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x55717cbc9b10 con 0x7fe06c0478c0 2020-11-20T12:15:31.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.095+0000 7fe077343700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fe06c0478c0 0x7fe06c0158c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.096+0000 7fe077343700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fe06c0478c0 0x7fe06c0158c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.096+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 <== osd.3 v2:172.21.15.187:6826/19935 1 ==== osd_op_reply(4 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe070022070 con 0x7fe06c0478c0 2020-11-20T12:15:31.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.096+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:5 22.1 22:80000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x55717cbc9b10 con 0x55717cbbb5f0 2020-11-20T12:15:31.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.097+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 <== osd.2 v2:172.21.15.187:6810/19904 2 ==== osd_op_reply(5 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe070022070 con 0x55717cbbb5f0 2020-11-20T12:15:31.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.097+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:6 22.5 22:a0000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x55717cbc9b10 con 0x55717cbbb5f0 2020-11-20T12:15:31.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.097+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 <== osd.2 v2:172.21.15.187:6810/19904 3 ==== osd_op_reply(6 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe070022070 con 0x55717cbbb5f0 2020-11-20T12:15:31.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.097+0000 7fe077343700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7fe070036040 0x7fe070036aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.097+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:7 22.3 22:c0000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x55717cbc9b10 con 0x7fe070036040 2020-11-20T12:15:31.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.097+0000 7fe076b42700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7fe070036040 0x7fe070036aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.098+0000 7fe076b42700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7fe070036040 0x7fe070036aa0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.099+0000 7fe076b42700 1 -- 172.21.15.187:0/1219882541 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(7 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe06801c040 con 0x7fe070036040 2020-11-20T12:15:31.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.099+0000 7fe076b42700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fe068022920 0x7fe0680213a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.099+0000 7fe076b42700 1 -- 172.21.15.187:0/1219882541 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:8 22.7 22:e0000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x55717cbc9b10 con 0x7fe068022920 2020-11-20T12:15:31.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.100+0000 7fe076341700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fe068022920 0x7fe0680213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.101+0000 7fe076341700 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fe068022920 0x7fe0680213a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe076341700 1 -- 172.21.15.187:0/1219882541 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(8 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe06c046770 con 0x7fe068022920 2020-11-20T12:15:31.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7fe070036040 msgr2=0x7fe070036aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7fe070036040 0x7fe070036aa0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x55717cbbb5f0 msgr2=0x55717cbc9620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x55717cbbb5f0 0x55717cbc9620 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fe06c0478c0 msgr2=0x7fe06c0158c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7fe06c0478c0 0x7fe06c0158c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe07001c070 msgr2=0x7fe070027940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe07001c070 0x7fe070027940 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fe068010940 msgr2=0x7fe06800f3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7fe068010940 0x7fe06800f3c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fe068022920 msgr2=0x7fe0680213a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7fe068022920 0x7fe0680213a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.102+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 reap_dead start 2020-11-20T12:15:31.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c0559a0 msgr2=0x7fe05c057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c0559a0 0x7fe05c057e60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fe06800b1a0 tx=0x7fe0680023a0).stop 2020-11-20T12:15:31.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb1ed20 msgr2=0x55717cbc4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb1ed20 0x55717cbc4650 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7fe06c00ba60 tx=0x7fe06c005bc0).stop 2020-11-20T12:15:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe077343700 1 -- 172.21.15.187:0/1219882541 reap_dead start 2020-11-20T12:15:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 shutdown_connections 2020-11-20T12:15:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 --2- 172.21.15.187:0/1219882541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55717cb1ed20 0x55717cbc4650 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 >> 172.21.15.187:0/1219882541 conn(0x55717caf60c0 msgr2=0x55717caf7940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:31.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 shutdown_connections 2020-11-20T12:15:31.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.103+0000 7fe08629dd00 1 -- 172.21.15.187:0/1219882541 wait complete. 2020-11-20T12:15:31.120 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000000 2020-11-20T12:15:31.120 INFO:teuthology.orchestra.run.smithi187.stdout:10000000000.00000001 2020-11-20T12:15:31.121 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 8388608 found 2020-11-20T12:15:31.153 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:31.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.442+0000 7fd7ed07f700 1 Processor -- start 2020-11-20T12:15:31.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.443+0000 7fd7ed07f700 1 -- start start 2020-11-20T12:15:31.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.444+0000 7fd7ed07f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80679b0 0x7fd7e8147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.444+0000 7fd7ed07f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd7e80ba9b0 0x7fd7e81481e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.444+0000 7fd7ed07f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e814be70 con 0x7fd7e80675d0 2020-11-20T12:15:31.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.444+0000 7fd7ed07f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7e814c110 con 0x7fd7e80ba9b0 2020-11-20T12:15:31.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.444+0000 7fd7ed07f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e814c3b0 con 0x7fd7e80679b0 2020-11-20T12:15:31.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.445+0000 7fd7e7fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd7e80675d0 0x7fd7e814c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:40244/0 (socket says 172.21.15.187:40244) 2020-11-20T12:15:31.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.445+0000 7fd7e7fff700 1 -- 172.21.15.187:0/2065098253 learned_addr learned my addr 172.21.15.187:0/2065098253 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:31.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.445+0000 7fd7df7fe700 1 --2- 172.21.15.187:0/2065098253 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80679b0 0x7fd7e8147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.445+0000 7fd7ec87e700 1 --2- 172.21.15.187:0/2065098253 >> v2:172.21.15.5:3300/0 conn(0x7fd7e80ba9b0 0x7fd7e81481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.446+0000 7fd7e77fe700 1 -- 172.21.15.187:0/2065098253 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721841351 0 0) 0x7fd7e814be70 con 0x7fd7e80675d0 2020-11-20T12:15:31.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.446+0000 7fd7e77fe700 1 -- 172.21.15.187:0/2065098253 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c8003240 con 0x7fd7e80675d0 2020-11-20T12:15:31.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.446+0000 7fd7df7fe700 1 -- 172.21.15.187:0/2065098253 >> v2:172.21.15.5:3300/0 conn(0x7fd7e80ba9b0 msgr2=0x7fd7e81481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:31.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.446+0000 7fd7df7fe700 1 --2- 172.21.15.187:0/2065098253 >> v2:172.21.15.5:3300/0 conn(0x7fd7e80ba9b0 0x7fd7e81481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.446+0000 7fd7df7fe700 1 -- 172.21.15.187:0/2065098253 >> v1:172.21.15.5:6789/0 conn(0x7fd7e80675d0 legacy=0x7fd7e814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.446+0000 7fd7df7fe700 1 -- 172.21.15.187:0/2065098253 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e81486d0 con 0x7fd7e80679b0 2020-11-20T12:15:31.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.447+0000 7fd7df7fe700 1 --2- 172.21.15.187:0/2065098253 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80679b0 0x7fd7e8147cf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd7d000b9a0 tx=0x7fd7d0006df0).ready entity=mon.2 client_cookie=19f7df4358e6704d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.447+0000 7fd7e77fe700 1 -- 172.21.15.187:0/2065098253 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0009680 con 0x7fd7e80679b0 2020-11-20T12:15:31.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.447+0000 7fd7e77fe700 1 -- 172.21.15.187:0/2065098253 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d00097e0 con 0x7fd7e80679b0 2020-11-20T12:15:31.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.447+0000 7fd7e77fe700 1 -- 172.21.15.187:0/2065098253 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0009b60 con 0x7fd7e80679b0 2020-11-20T12:15:31.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.448+0000 7fd7ed07f700 1 -- 172.21.15.187:0/2065098253 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80679b0 msgr2=0x7fd7e8147cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.448+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/2065098253 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80679b0 0x7fd7e8147cf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd7d000b9a0 tx=0x7fd7d0006df0).stop 2020-11-20T12:15:31.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.448+0000 7fd7ed07f700 1 -- 172.21.15.187:0/2065098253 shutdown_connections 2020-11-20T12:15:31.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.448+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/2065098253 >> v2:172.21.15.5:3300/0 conn(0x7fd7e80ba9b0 0x7fd7e81481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.448+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/2065098253 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80679b0 0x7fd7e8147cf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.448+0000 7fd7ed07f700 1 -- 172.21.15.187:0/2065098253 >> 172.21.15.187:0/2065098253 conn(0x7fd7e80bce50 msgr2=0x7fd7e80bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:31.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.448+0000 7fd7ed07f700 1 -- 172.21.15.187:0/2065098253 shutdown_connections 2020-11-20T12:15:31.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.448+0000 7fd7ed07f700 1 -- 172.21.15.187:0/2065098253 wait complete. 2020-11-20T12:15:31.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.450+0000 7fd7ed07f700 1 Processor -- start 2020-11-20T12:15:31.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.450+0000 7fd7ed07f700 1 -- start start 2020-11-20T12:15:31.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.451+0000 7fd7ed07f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80675d0 0x7fd7e8158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.451+0000 7fd7ed07f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e80679b0 0x7fd7e81475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.451+0000 7fd7ed07f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7e80ba9b0 0x7fd7e80a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.451+0000 7fd7ed07f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8158d70 con 0x7fd7e80ba9b0 2020-11-20T12:15:31.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.451+0000 7fd7ed07f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e809f1f0 con 0x7fd7e80679b0 2020-11-20T12:15:31.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.451+0000 7fd7ed07f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e809f490 con 0x7fd7e80675d0 2020-11-20T12:15:31.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7df7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e80679b0 0x7fd7e81475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7df7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e80679b0 0x7fd7e81475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50136/0 (socket says 172.21.15.187:50136) 2020-11-20T12:15:31.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7df7fe700 1 -- 172.21.15.187:0/572115467 learned_addr learned my addr 172.21.15.187:0/572115467 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:31.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7e7fff700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80675d0 0x7fd7e8158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7ec87e700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7e80ba9b0 0x7fd7e80a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7df7fe700 1 -- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80675d0 msgr2=0x7fd7e8158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7df7fe700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80675d0 0x7fd7e8158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7df7fe700 1 -- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7e80ba9b0 msgr2=0x7fd7e80a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7df7fe700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7e80ba9b0 0x7fd7e80a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7df7fe700 1 -- 172.21.15.187:0/572115467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e809f730 con 0x7fd7e80679b0 2020-11-20T12:15:31.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7e7fff700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80675d0 0x7fd7e8158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:31.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.452+0000 7fd7ec87e700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7e80ba9b0 0x7fd7e80a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:31.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.453+0000 7fd7df7fe700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e80679b0 0x7fd7e81475d0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7fd7d000c010 tx=0x7fd7d0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.453+0000 7fd7e4ff9700 1 -- 172.21.15.187:0/572115467 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0006b20 con 0x7fd7e80679b0 2020-11-20T12:15:31.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.453+0000 7fd7e4ff9700 1 -- 172.21.15.187:0/572115467 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d001c5c0 con 0x7fd7e80679b0 2020-11-20T12:15:31.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.453+0000 7fd7e4ff9700 1 -- 172.21.15.187:0/572115467 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d001c960 con 0x7fd7e80679b0 2020-11-20T12:15:31.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.453+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e8147ac0 con 0x7fd7e80679b0 2020-11-20T12:15:31.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.453+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e80a2010 con 0x7fd7e80679b0 2020-11-20T12:15:31.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.454+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e809c080 con 0x7fd7e80679b0 2020-11-20T12:15:31.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.455+0000 7fd7e4ff9700 1 -- 172.21.15.187:0/572115467 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd7d000f070 con 0x7fd7e80679b0 2020-11-20T12:15:31.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.455+0000 7fd7e4ff9700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7c4056d20 0x7fd7c40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:31.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.455+0000 7fd7e4ff9700 1 -- 172.21.15.187:0/572115467 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd7d0071ea0 con 0x7fd7e80679b0 2020-11-20T12:15:31.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.456+0000 7fd7e7fff700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7c4056d20 0x7fd7c40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:31.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.456+0000 7fd7e7fff700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7c4056d20 0x7fd7c40591e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fd7d80055a0 tx=0x7fd7d8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:31.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.459+0000 7fd7e4ff9700 1 -- 172.21.15.187:0/572115467 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd7d0047270 con 0x7fd7e80679b0 2020-11-20T12:15:31.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.703+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7e809f8c0 con 0x7fd7e80679b0 2020-11-20T12:15:31.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.704+0000 7fd7e4ff9700 1 -- 172.21.15.187:0/572115467 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fd7d004a890 con 0x7fd7e80679b0 2020-11-20T12:15:31.709 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:31.709 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:31.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7c4056d20 msgr2=0x7fd7c40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7c4056d20 0x7fd7c40591e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fd7d80055a0 tx=0x7fd7d8009040).stop 2020-11-20T12:15:31.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e80679b0 msgr2=0x7fd7e81475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:31.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e80679b0 0x7fd7e81475d0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7fd7d000c010 tx=0x7fd7d0015000).stop 2020-11-20T12:15:31.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 shutdown_connections 2020-11-20T12:15:31.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd7c4056d20 0x7fd7c40591e0 secure :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0x7fd7d80055a0 tx=0x7fd7d8009040).stop 2020-11-20T12:15:31.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7e80ba9b0 0x7fd7e80a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7e80675d0 0x7fd7e8158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 --2- 172.21.15.187:0/572115467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e80679b0 0x7fd7e81475d0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:31.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 >> 172.21.15.187:0/572115467 conn(0x7fd7e80bce50 msgr2=0x7fd7e80b9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:31.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.707+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 shutdown_connections 2020-11-20T12:15:31.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:31.708+0000 7fd7ed07f700 1 -- 172.21.15.187:0/572115467 wait complete. 2020-11-20T12:15:31.716 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:31.734 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2020-11-20T12:15:32.019 INFO:teuthology.orchestra.run.smithi187.stdout:[{"id":8260,"entity":{"name":{"type":"client","num":8260},"addr":{"type":"v1","addr":"192.168.0.1:0","nonce":4134631550}},"state":"open","num_leases":1,"num_caps":2,"request_load_avg":9,"uptime":7.4548809790000004,"requests_in_flight":0,"completed_requests":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":2.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":5.9103213574435101,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"inst":"client.8260 v1:192.168.0.1:0/4134631550","completed_requests":[{"tid":10,"created_ino":"0x0"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"used_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x0000000000007bff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000001f"}},"entity_id":"0","hostname":"smithi187","kernel_version":"5.10.0-rc2-ceph-gac4f7b07f403","root":"/"}}] 2020-11-20T12:15:32.020 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "entity_id": "0", "hostname": "smithi187", "kernel_version": "5.10.0-rc2-ceph-gac4f7b07f403", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000001f" } }, "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 10 } ], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 4134631550, "type": "v1" }, "name": { "num": 8260, "type": "client" } }, "id": 8260, "inst": "client.8260 v1:192.168.0.1:0/4134631550", "num_caps": 2, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 2.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 5.91032135744351 }, "state": "open", "uptime": 7.454880979, "used_inos": [] } ] 2020-11-20T12:15:32.021 DEBUG:tasks.ceph_test_case:wait_until_equal: 2 != 1, waiting (timeout=60)... 2020-11-20T12:15:37.022 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:37.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.273+0000 7f33ffc09700 1 Processor -- start 2020-11-20T12:15:37.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.273+0000 7f33ffc09700 1 -- start start 2020-11-20T12:15:37.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.275+0000 7f33ffc09700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33f809a150 0x7f33f80652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:37.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.275+0000 7f33ffc09700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f80b7290 0x7f33f8061de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:37.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.275+0000 7f33ffc09700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f8146c10 con 0x7f33f80657d0 2020-11-20T12:15:37.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.275+0000 7f33ffc09700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f8146eb0 con 0x7f33f809a150 2020-11-20T12:15:37.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.275+0000 7f33ffc09700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f8147150 con 0x7f33f80b7290 2020-11-20T12:15:37.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fd1a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f80b7290 0x7f33f8061de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:37.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fd9a5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33f809a150 0x7f33f80652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:37.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fe1a6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f33f80657d0 0x7f33f80622d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46756/0 (socket says 172.21.15.187:46756) 2020-11-20T12:15:37.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fe1a6700 1 -- 172.21.15.187:0/3593709474 learned_addr learned my addr 172.21.15.187:0/3593709474 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:37.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fd9a5700 1 -- 172.21.15.187:0/3593709474 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f80b7290 msgr2=0x7f33f8061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:37.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fd9a5700 1 --2- 172.21.15.187:0/3593709474 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f80b7290 0x7f33f8061de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fd9a5700 1 -- 172.21.15.187:0/3593709474 >> v1:172.21.15.187:6789/0 conn(0x7f33f80657d0 legacy=0x7f33f80622d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:37.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fd9a5700 1 -- 172.21.15.187:0/3593709474 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f8146950 con 0x7f33f809a150 2020-11-20T12:15:37.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.276+0000 7f33fc9a3700 1 -- 172.21.15.187:0/3593709474 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463497066 0 0) 0x7f33f8146c10 con 0x7f33f80657d0 2020-11-20T12:15:37.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.277+0000 7f33fd9a5700 1 --2- 172.21.15.187:0/3593709474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f809a150 0x7f33f80652e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f33f400b9a0 tx=0x7f33f4006df0).ready entity=mon.1 client_cookie=62ad73f2c87a490f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:37.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.277+0000 7f33fc9a3700 1 -- 172.21.15.187:0/3593709474 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f40097b0 con 0x7f33f809a150 2020-11-20T12:15:37.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.277+0000 7f33fc9a3700 1 -- 172.21.15.187:0/3593709474 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f4009910 con 0x7f33f809a150 2020-11-20T12:15:37.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.277+0000 7f33fc9a3700 1 -- 172.21.15.187:0/3593709474 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4009c90 con 0x7f33f809a150 2020-11-20T12:15:37.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.278+0000 7f33ffc09700 1 -- 172.21.15.187:0/3593709474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f809a150 msgr2=0x7f33f80652e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:37.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.278+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3593709474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f809a150 0x7f33f80652e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f33f400b9a0 tx=0x7f33f4006df0).stop 2020-11-20T12:15:37.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.278+0000 7f33ffc09700 1 -- 172.21.15.187:0/3593709474 shutdown_connections 2020-11-20T12:15:37.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.278+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3593709474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f809a150 0x7f33f80652e0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.278+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3593709474 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f80b7290 0x7f33f8061de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.278+0000 7f33ffc09700 1 -- 172.21.15.187:0/3593709474 >> 172.21.15.187:0/3593709474 conn(0x7f33f80bacd0 msgr2=0x7f33f80bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:37.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.278+0000 7f33ffc09700 1 -- 172.21.15.187:0/3593709474 shutdown_connections 2020-11-20T12:15:37.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.278+0000 7f33ffc09700 1 -- 172.21.15.187:0/3593709474 wait complete. 2020-11-20T12:15:37.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.279+0000 7f33ffc09700 1 Processor -- start 2020-11-20T12:15:37.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.279+0000 7f33ffc09700 1 -- start start 2020-11-20T12:15:37.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.280+0000 7f33ffc09700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33f80657d0 0x7f33f80b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:37.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.280+0000 7f33ffc09700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f809a150 0x7f33f8143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:37.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.280+0000 7f33ffc09700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f8068140 0x7f33f8143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:37.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.280+0000 7f33ffc09700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f815bca0 con 0x7f33f80657d0 2020-11-20T12:15:37.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.280+0000 7f33ffc09700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f815bf40 con 0x7f33f8068140 2020-11-20T12:15:37.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.280+0000 7f33ffc09700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f815c1e0 con 0x7f33f809a150 2020-11-20T12:15:37.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fe1a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f8068140 0x7f33f8143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:37.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fe1a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f8068140 0x7f33f8143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50158/0 (socket says 172.21.15.187:50158) 2020-11-20T12:15:37.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fe1a6700 1 -- 172.21.15.187:0/3672835751 learned_addr learned my addr 172.21.15.187:0/3672835751 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:37.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fd9a5700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33f80657d0 0x7f33f80b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:37.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fd1a4700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f809a150 0x7f33f8143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:37.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fe1a6700 1 -- 172.21.15.187:0/3672835751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f809a150 msgr2=0x7f33f8143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:37.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fe1a6700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f809a150 0x7f33f8143110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fe1a6700 1 -- 172.21.15.187:0/3672835751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33f80657d0 msgr2=0x7f33f80b7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:37.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fe1a6700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33f80657d0 0x7f33f80b7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.281+0000 7f33fe1a6700 1 -- 172.21.15.187:0/3672835751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f80b82d0 con 0x7f33f8068140 2020-11-20T12:15:37.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.282+0000 7f33fe1a6700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f8068140 0x7f33f8143600 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f33f000cab0 tx=0x7f33f0007de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:37.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.282+0000 7f33e5ffb700 1 -- 172.21.15.187:0/3672835751 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0005c50 con 0x7f33f8068140 2020-11-20T12:15:37.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.282+0000 7f33e5ffb700 1 -- 172.21.15.187:0/3672835751 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f0005db0 con 0x7f33f8068140 2020-11-20T12:15:37.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.282+0000 7f33e5ffb700 1 -- 172.21.15.187:0/3672835751 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0007340 con 0x7f33f8068140 2020-11-20T12:15:37.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.282+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f815b920 con 0x7f33f8068140 2020-11-20T12:15:37.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.282+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f809e3a0 con 0x7f33f8068140 2020-11-20T12:15:37.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.283+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f815c3a0 con 0x7f33f8068140 2020-11-20T12:15:37.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.284+0000 7f33e5ffb700 1 -- 172.21.15.187:0/3672835751 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f33f0010030 con 0x7f33f8068140 2020-11-20T12:15:37.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.284+0000 7f33e5ffb700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33d0056d20 0x7f33d00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:37.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.284+0000 7f33e5ffb700 1 -- 172.21.15.187:0/3672835751 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f33f0070230 con 0x7f33f8068140 2020-11-20T12:15:37.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.285+0000 7f33fd9a5700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33d0056d20 0x7f33d00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:37.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.285+0000 7f33fd9a5700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33d0056d20 0x7f33d00591e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f33f4006e30 tx=0x7f33f4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:37.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.288+0000 7f33e5ffb700 1 -- 172.21.15.187:0/3672835751 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f33f0045570 con 0x7f33f8068140 2020-11-20T12:15:37.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.533+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33f815bb10 con 0x7f33f8068140 2020-11-20T12:15:37.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.534+0000 7f33e5ffb700 1 -- 172.21.15.187:0/3672835751 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f33f0048b90 con 0x7f33f8068140 2020-11-20T12:15:37.538 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:37.539 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:37.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33d0056d20 msgr2=0x7f33d00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:37.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33d0056d20 0x7f33d00591e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f33f4006e30 tx=0x7f33f4015000).stop 2020-11-20T12:15:37.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f8068140 msgr2=0x7f33f8143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:37.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f8068140 0x7f33f8143600 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f33f000cab0 tx=0x7f33f0007de0).stop 2020-11-20T12:15:37.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 shutdown_connections 2020-11-20T12:15:37.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33d0056d20 0x7f33d00591e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33f80657d0 0x7f33f80b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33f8068140 0x7f33f8143600 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 --2- 172.21.15.187:0/3672835751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33f809a150 0x7f33f8143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 >> 172.21.15.187:0/3672835751 conn(0x7f33f80bacd0 msgr2=0x7f33f80b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 shutdown_connections 2020-11-20T12:15:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:37.537+0000 7f33ffc09700 1 -- 172.21.15.187:0/3672835751 wait complete. 2020-11-20T12:15:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:37.563 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session ls 2020-11-20T12:15:37.834 INFO:teuthology.orchestra.run.smithi187.stdout:[{"id":8260,"entity":{"name":{"type":"client","num":8260},"addr":{"type":"v1","addr":"192.168.0.1:0","nonce":4134631550}},"state":"open","num_leases":1,"num_caps":1,"request_load_avg":8,"uptime":13.272788092000001,"requests_in_flight":0,"completed_requests":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0.98065468332979,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":2.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":5.8314051122694162,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"inst":"client.8260 v1:192.168.0.1:0/4134631550","completed_requests":[{"tid":10,"created_ino":"0x0"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"used_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x0000000000007bff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000001f"}},"entity_id":"0","hostname":"smithi187","kernel_version":"5.10.0-rc2-ceph-gac4f7b07f403","root":"/"}}] 2020-11-20T12:15:37.835 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "client_features": { "feature_bits": "0x0000000000007bff" }, "entity_id": "0", "hostname": "smithi187", "kernel_version": "5.10.0-rc2-ceph-gac4f7b07f403", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000001f" } }, "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 10 } ], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 4134631550, "type": "v1" }, "name": { "num": 8260, "type": "client" } }, "id": 8260, "inst": "client.8260 v1:192.168.0.1:0/4134631550", "num_caps": 1, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 2.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0.98065468332979 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 5.831405112269416 }, "state": "open", "uptime": 13.272788092, "used_inos": [] } ] 2020-11-20T12:15:37.836 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:38.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.076+0000 7f119154c700 1 Processor -- start 2020-11-20T12:15:38.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.076+0000 7f119154c700 1 -- start start 2020-11-20T12:15:38.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.078+0000 7f119154c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f118c0b6650 0x7f118c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.078+0000 7f119154c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f118c066f70 0x7f118c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.078+0000 7f119154c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0b1300 con 0x7f118c1484d0 2020-11-20T12:15:38.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.078+0000 7f119154c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118c067730 con 0x7f118c0b6650 2020-11-20T12:15:38.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.078+0000 7f119154c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118c0b15a0 con 0x7f118c066f70 2020-11-20T12:15:38.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.078+0000 7f1190d4b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f118c066f70 0x7f118c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:38.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.078+0000 7f118bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f118c1484d0 0x7f118c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46768/0 (socket says 172.21.15.187:46768) 2020-11-20T12:15:38.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.078+0000 7f118bfff700 1 -- 172.21.15.187:0/3030436793 learned_addr learned my addr 172.21.15.187:0/3030436793 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:38.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.079+0000 7f11837fe700 1 --2- 172.21.15.187:0/3030436793 >> v2:172.21.15.5:3300/0 conn(0x7f118c0b6650 0x7f118c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:38.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.079+0000 7f118b7fe700 1 -- 172.21.15.187:0/3030436793 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865028602 0 0) 0x7f118c0b1300 con 0x7f118c1484d0 2020-11-20T12:15:38.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.079+0000 7f118b7fe700 1 -- 172.21.15.187:0/3030436793 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f116c003240 con 0x7f118c1484d0 2020-11-20T12:15:38.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.079+0000 7f1190d4b700 1 -- 172.21.15.187:0/3030436793 >> v2:172.21.15.5:3300/0 conn(0x7f118c0b6650 msgr2=0x7f118c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:38.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.079+0000 7f1190d4b700 1 --2- 172.21.15.187:0/3030436793 >> v2:172.21.15.5:3300/0 conn(0x7f118c0b6650 0x7f118c0afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:38.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.079+0000 7f1190d4b700 1 -- 172.21.15.187:0/3030436793 >> v1:172.21.15.187:6789/0 conn(0x7f118c1484d0 legacy=0x7f118c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.079+0000 7f1190d4b700 1 -- 172.21.15.187:0/3030436793 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c0b1840 con 0x7f118c066f70 2020-11-20T12:15:38.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.079+0000 7f1190d4b700 1 --2- 172.21.15.187:0/3030436793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c0aff80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f118c045000 tx=0x7f11840091d0).ready entity=mon.1 client_cookie=ca947641877113b8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:38.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f118b7fe700 1 -- 172.21.15.187:0/3030436793 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118400ba70 con 0x7f118c066f70 2020-11-20T12:15:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f118b7fe700 1 -- 172.21.15.187:0/3030436793 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118400bbd0 con 0x7f118c066f70 2020-11-20T12:15:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f118b7fe700 1 -- 172.21.15.187:0/3030436793 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184011400 con 0x7f118c066f70 2020-11-20T12:15:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f119154c700 1 -- 172.21.15.187:0/3030436793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 msgr2=0x7f118c0aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f119154c700 1 --2- 172.21.15.187:0/3030436793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c0aff80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f118c045000 tx=0x7f11840091d0).stop 2020-11-20T12:15:38.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f119154c700 1 -- 172.21.15.187:0/3030436793 shutdown_connections 2020-11-20T12:15:38.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f119154c700 1 --2- 172.21.15.187:0/3030436793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c0aff80 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:38.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f119154c700 1 --2- 172.21.15.187:0/3030436793 >> v2:172.21.15.5:3300/0 conn(0x7f118c0b6650 0x7f118c0afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:38.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.080+0000 7f119154c700 1 -- 172.21.15.187:0/3030436793 >> 172.21.15.187:0/3030436793 conn(0x7f118c0bab30 msgr2=0x7f118c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:38.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.081+0000 7f119154c700 1 -- 172.21.15.187:0/3030436793 shutdown_connections 2020-11-20T12:15:38.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.081+0000 7f119154c700 1 -- 172.21.15.187:0/3030436793 wait complete. 2020-11-20T12:15:38.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.082+0000 7f119154c700 1 Processor -- start 2020-11-20T12:15:38.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.082+0000 7f119154c700 1 -- start start 2020-11-20T12:15:38.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.083+0000 7f119154c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c13a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.083+0000 7f119154c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f118c0b6650 0x7f118c13a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.083+0000 7f119154c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f118c1484d0 0x7f118c0b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.083+0000 7f119154c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c09c670 con 0x7f118c1484d0 2020-11-20T12:15:38.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.083+0000 7f119154c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c09c910 con 0x7f118c066f70 2020-11-20T12:15:38.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.083+0000 7f119154c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c09cbb0 con 0x7f118c0b6650 2020-11-20T12:15:38.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.083+0000 7f118bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c13a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:38.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f118bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c13a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50168/0 (socket says 172.21.15.187:50168) 2020-11-20T12:15:38.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f118bfff700 1 -- 172.21.15.187:0/348322471 learned_addr learned my addr 172.21.15.187:0/348322471 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:38.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f1190d4b700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f118c1484d0 0x7f118c0b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:38.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f118bfff700 1 -- 172.21.15.187:0/348322471 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f118c0b6650 msgr2=0x7f118c13a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f118bfff700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f118c0b6650 0x7f118c13a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:38.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f118bfff700 1 -- 172.21.15.187:0/348322471 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f118c1484d0 msgr2=0x7f118c0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f118bfff700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f118c1484d0 0x7f118c0b40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:38.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f118bfff700 1 -- 172.21.15.187:0/348322471 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c09b640 con 0x7f118c066f70 2020-11-20T12:15:38.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f118bfff700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c13a410 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f117c00c6e0 tx=0x7f117c007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:38.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f1188ff9700 1 -- 172.21.15.187:0/348322471 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c0072e0 con 0x7f118c066f70 2020-11-20T12:15:38.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f1188ff9700 1 -- 172.21.15.187:0/348322471 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c007440 con 0x7f118c066f70 2020-11-20T12:15:38.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.084+0000 7f1188ff9700 1 -- 172.21.15.187:0/348322471 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c005820 con 0x7f118c066f70 2020-11-20T12:15:38.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.085+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c09b890 con 0x7f118c066f70 2020-11-20T12:15:38.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.085+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c0b4750 con 0x7f118c066f70 2020-11-20T12:15:38.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.086+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c09cd70 con 0x7f118c066f70 2020-11-20T12:15:38.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.086+0000 7f1188ff9700 1 -- 172.21.15.187:0/348322471 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f117c010070 con 0x7f118c066f70 2020-11-20T12:15:38.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.087+0000 7f1188ff9700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1168056d50 0x7f1168059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.087+0000 7f1188ff9700 1 -- 172.21.15.187:0/348322471 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f117c06f540 con 0x7f118c066f70 2020-11-20T12:15:38.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.087+0000 7f11837fe700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1168056d50 0x7f1168059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:38.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.088+0000 7f11837fe700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1168056d50 0x7f1168059210 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f118c0619f0 tx=0x7f1174008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:38.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.091+0000 7f1188ff9700 1 -- 172.21.15.187:0/348322471 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f117c044930 con 0x7f118c066f70 2020-11-20T12:15:38.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.340+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f118c0b49a0 con 0x7f118c066f70 2020-11-20T12:15:38.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.341+0000 7f1188ff9700 1 -- 172.21.15.187:0/348322471 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f117c018020 con 0x7f118c066f70 2020-11-20T12:15:38.345 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:38.345 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:38.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1168056d50 msgr2=0x7f1168059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1168056d50 0x7f1168059210 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f118c0619f0 tx=0x7f1174008040).stop 2020-11-20T12:15:38.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 msgr2=0x7f118c13a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c13a410 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f117c00c6e0 tx=0x7f117c007ac0).stop 2020-11-20T12:15:38.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 shutdown_connections 2020-11-20T12:15:38.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1168056d50 0x7f1168059210 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:38.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f118c1484d0 0x7f118c0b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:38.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f118c066f70 0x7f118c13a410 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:38.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 --2- 172.21.15.187:0/348322471 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f118c0b6650 0x7f118c13a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:38.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.343+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 >> 172.21.15.187:0/348322471 conn(0x7f118c0bab30 msgr2=0x7f118c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:38.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.344+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 shutdown_connections 2020-11-20T12:15:38.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.344+0000 7f119154c700 1 -- 172.21.15.187:0/348322471 wait complete. 2020-11-20T12:15:38.350 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:38.370 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:15:38.643 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":1}} 2020-11-20T12:15:38.643 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2020-11-20T12:15:38.644 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:38.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.884+0000 7fd8d8dc1700 1 Processor -- start 2020-11-20T12:15:38.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.884+0000 7fd8d8dc1700 1 -- start start 2020-11-20T12:15:38.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.885+0000 7fd8d8dc1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8d40bf800 0x7fd8d414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.886+0000 7fd8d8dc1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d414d110 con 0x7fd8d40bfbe0 2020-11-20T12:15:38.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.886+0000 7fd8d8dc1700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d414cbd0 con 0x7fd8d409f560 2020-11-20T12:15:38.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.886+0000 7fd8d8dc1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d414ce70 con 0x7fd8d40bf800 2020-11-20T12:15:38.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.886+0000 7fd8d37fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8d40bf800 0x7fd8d414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:38.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.886+0000 7fd8d37fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd8d40bf800 0x7fd8d414c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50176/0 (socket says 172.21.15.187:50176) 2020-11-20T12:15:38.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.886+0000 7fd8d37fe700 1 -- 172.21.15.187:0/3989435090 learned_addr learned my addr 172.21.15.187:0/3989435090 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:38.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.887+0000 7fd8d37fe700 1 -- 172.21.15.187:0/3989435090 >> v1:172.21.15.187:6789/0 conn(0x7fd8d409f560 legacy=0x7fd8d40a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.887+0000 7fd8d37fe700 1 -- 172.21.15.187:0/3989435090 >> v1:172.21.15.5:6789/0 conn(0x7fd8d40bfbe0 legacy=0x7fd8d41480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:38.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.887+0000 7fd8d37fe700 1 -- 172.21.15.187:0/3989435090 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40a4cc0 con 0x7fd8d40bf800 2020-11-20T12:15:38.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.887+0000 7fd8d37fe700 1 --2- 172.21.15.187:0/3989435090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d414c2c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7fd8c4006fd0 tx=0x7fd8c4006eb0).ready entity=mon.1 client_cookie=d781d5ea3c0242d5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:38.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.887+0000 7fd8d2ffd700 1 -- 172.21.15.187:0/3989435090 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4009920 con 0x7fd8d40bf800 2020-11-20T12:15:38.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d2ffd700 1 -- 172.21.15.187:0/3989435090 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c4009a80 con 0x7fd8d40bf800 2020-11-20T12:15:38.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d2ffd700 1 -- 172.21.15.187:0/3989435090 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4009e00 con 0x7fd8d40bf800 2020-11-20T12:15:38.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/3989435090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 msgr2=0x7fd8d414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d8dc1700 1 --2- 172.21.15.187:0/3989435090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d414c2c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7fd8c4006fd0 tx=0x7fd8c4006eb0).stop 2020-11-20T12:15:38.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/3989435090 shutdown_connections 2020-11-20T12:15:38.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d8dc1700 1 --2- 172.21.15.187:0/3989435090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d414c2c0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:38.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/3989435090 >> 172.21.15.187:0/3989435090 conn(0x7fd8d4068370 msgr2=0x7fd8d40bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:38.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/3989435090 shutdown_connections 2020-11-20T12:15:38.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.888+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/3989435090 wait complete. 2020-11-20T12:15:38.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.890+0000 7fd8d8dc1700 1 Processor -- start 2020-11-20T12:15:38.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.890+0000 7fd8d8dc1700 1 -- start start 2020-11-20T12:15:38.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8d8dc1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8d409f560 0x7fd8d40ba750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8d8dc1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d40b7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8d8dc1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8d4146bb0 0x7fd8d4146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8d8dc1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40b4b80 con 0x7fd8d409f560 2020-11-20T12:15:38.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8d8dc1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40b41a0 con 0x7fd8d40bf800 2020-11-20T12:15:38.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8d8dc1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40b4440 con 0x7fd8d4146bb0 2020-11-20T12:15:38.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8cbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d40b7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:38.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8cbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d40b7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50182/0 (socket says 172.21.15.187:50182) 2020-11-20T12:15:38.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.891+0000 7fd8cbfff700 1 -- 172.21.15.187:0/620204905 learned_addr learned my addr 172.21.15.187:0/620204905 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:38.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cbfff700 1 -- 172.21.15.187:0/620204905 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8d4146bb0 msgr2=0x7fd8d4146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cbfff700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8d4146bb0 0x7fd8d4146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:38.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cbfff700 1 -- 172.21.15.187:0/620204905 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8d409f560 msgr2=0x7fd8d40ba750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:38.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cbfff700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8d409f560 0x7fd8d40ba750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:38.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cbfff700 1 -- 172.21.15.187:0/620204905 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40b46e0 con 0x7fd8d40bf800 2020-11-20T12:15:38.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cbfff700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d40b7710 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7fd8bc009710 tx=0x7fd8bc006d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:38.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cb7fe700 1 -- 172.21.15.187:0/620204905 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc00d560 con 0x7fd8d40bf800 2020-11-20T12:15:38.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cb7fe700 1 -- 172.21.15.187:0/620204905 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc000e90 con 0x7fd8d40bf800 2020-11-20T12:15:38.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d40b3e90 con 0x7fd8d40bf800 2020-11-20T12:15:38.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8cb7fe700 1 -- 172.21.15.187:0/620204905 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc00d930 con 0x7fd8d40bf800 2020-11-20T12:15:38.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.892+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d40b7c00 con 0x7fd8d40bf800 2020-11-20T12:15:38.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.893+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d4080050 con 0x7fd8d40bf800 2020-11-20T12:15:38.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.894+0000 7fd8cb7fe700 1 -- 172.21.15.187:0/620204905 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd8bc00f030 con 0x7fd8d40bf800 2020-11-20T12:15:38.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.894+0000 7fd8cb7fe700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d20 0x7fd8b00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:38.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.895+0000 7fd8cb7fe700 1 -- 172.21.15.187:0/620204905 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd8bc06fdc0 con 0x7fd8d40bf800 2020-11-20T12:15:38.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.895+0000 7fd8d37fe700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d20 0x7fd8b00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:38.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.895+0000 7fd8d37fe700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d20 0x7fd8b00591e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fd8c4006ef0 tx=0x7fd8c4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:38.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:38.898+0000 7fd8cb7fe700 1 -- 172.21.15.187:0/620204905 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd8bc045100 con 0x7fd8d40bf800 2020-11-20T12:15:39.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.144+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8d40b4870 con 0x7fd8d40bf800 2020-11-20T12:15:39.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.145+0000 7fd8cb7fe700 1 -- 172.21.15.187:0/620204905 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fd8bc048720 con 0x7fd8d40bf800 2020-11-20T12:15:39.149 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:39.149 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:39.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d20 msgr2=0x7fd8b00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:39.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d20 0x7fd8b00591e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fd8c4006ef0 tx=0x7fd8c4015000).stop 2020-11-20T12:15:39.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 msgr2=0x7fd8d40b7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:39.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d40b7710 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7fd8bc009710 tx=0x7fd8bc006d70).stop 2020-11-20T12:15:39.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 shutdown_connections 2020-11-20T12:15:39.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd8b0056d20 0x7fd8b00591e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:39.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8d409f560 0x7fd8d40ba750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:39.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8d4146bb0 0x7fd8d4146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:39.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 --2- 172.21.15.187:0/620204905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8d40bf800 0x7fd8d40b7710 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:39.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 >> 172.21.15.187:0/620204905 conn(0x7fd8d4068370 msgr2=0x7fd8d40a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:39.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 shutdown_connections 2020-11-20T12:15:39.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.148+0000 7fd8d8dc1700 1 -- 172.21.15.187:0/620204905 wait complete. 2020-11-20T12:15:39.154 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:39.174 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2020-11-20T12:15:39.449 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"num_strays":0}} 2020-11-20T12:15:39.450 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2020-11-20T12:15:39.450 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:39.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.697+0000 7fba83fff700 1 Processor -- start 2020-11-20T12:15:39.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.697+0000 7fba83fff700 1 -- start start 2020-11-20T12:15:39.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.699+0000 7fba83fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba841484d0 0x7fba84061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:39.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.699+0000 7fba83fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840b1300 con 0x7fba840622c0 2020-11-20T12:15:39.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.699+0000 7fba83fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840b15a0 con 0x7fba84066f70 2020-11-20T12:15:39.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.699+0000 7fba83fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba84067730 con 0x7fba841484d0 2020-11-20T12:15:39.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.699+0000 7fba82ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba841484d0 0x7fba84061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:39.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.699+0000 7fba837fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fba84066f70 0x7fba840b5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46798/0 (socket says 172.21.15.187:46798) 2020-11-20T12:15:39.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.699+0000 7fba837fe700 1 -- 172.21.15.187:0/515646131 learned_addr learned my addr 172.21.15.187:0/515646131 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:39.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.700+0000 7fba82ffd700 1 -- 172.21.15.187:0/515646131 >> v1:172.21.15.187:6789/0 conn(0x7fba84066f70 legacy=0x7fba840b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:39.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.700+0000 7fba82ffd700 1 -- 172.21.15.187:0/515646131 >> v1:172.21.15.5:6789/0 conn(0x7fba840622c0 legacy=0x7fba840b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:39.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.700+0000 7fba82ffd700 1 -- 172.21.15.187:0/515646131 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba840b1840 con 0x7fba841484d0 2020-11-20T12:15:39.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.700+0000 7fba82ffd700 1 --2- 172.21.15.187:0/515646131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba841484d0 0x7fba84061dd0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7fba7400b9a0 tx=0x7fba74006df0).ready entity=mon.1 client_cookie=2f8e314a8254131d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:39.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.700+0000 7fba827fc700 1 -- 172.21.15.187:0/515646131 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74009770 con 0x7fba841484d0 2020-11-20T12:15:39.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.700+0000 7fba827fc700 1 -- 172.21.15.187:0/515646131 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba740098d0 con 0x7fba841484d0 2020-11-20T12:15:39.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.701+0000 7fba827fc700 1 -- 172.21.15.187:0/515646131 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74009c50 con 0x7fba841484d0 2020-11-20T12:15:39.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.701+0000 7fba83fff700 1 -- 172.21.15.187:0/515646131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba841484d0 msgr2=0x7fba84061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:39.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.701+0000 7fba83fff700 1 --2- 172.21.15.187:0/515646131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba841484d0 0x7fba84061dd0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7fba7400b9a0 tx=0x7fba74006df0).stop 2020-11-20T12:15:39.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.701+0000 7fba83fff700 1 -- 172.21.15.187:0/515646131 shutdown_connections 2020-11-20T12:15:39.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.701+0000 7fba83fff700 1 --2- 172.21.15.187:0/515646131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba841484d0 0x7fba84061dd0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:39.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.701+0000 7fba83fff700 1 -- 172.21.15.187:0/515646131 >> 172.21.15.187:0/515646131 conn(0x7fba840bab30 msgr2=0x7fba840baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:39.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.701+0000 7fba83fff700 1 -- 172.21.15.187:0/515646131 shutdown_connections 2020-11-20T12:15:39.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.701+0000 7fba83fff700 1 -- 172.21.15.187:0/515646131 wait complete. 2020-11-20T12:15:39.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.703+0000 7fba83fff700 1 Processor -- start 2020-11-20T12:15:39.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.703+0000 7fba83fff700 1 -- start start 2020-11-20T12:15:39.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba83fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba840622c0 0x7fba8409b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:39.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba83fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba84066f70 0x7fba84159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:39.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba83fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba841484d0 0x7fba840b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:39.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba83fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba84144ba0 con 0x7fba84066f70 2020-11-20T12:15:39.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba83fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba84143940 con 0x7fba840622c0 2020-11-20T12:15:39.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba83fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba84099230 con 0x7fba841484d0 2020-11-20T12:15:39.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba82ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba840622c0 0x7fba8409b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:39.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba82ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba840622c0 0x7fba8409b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50196/0 (socket says 172.21.15.187:50196) 2020-11-20T12:15:39.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.704+0000 7fba82ffd700 1 -- 172.21.15.187:0/948471946 learned_addr learned my addr 172.21.15.187:0/948471946 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:39.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba82ffd700 1 -- 172.21.15.187:0/948471946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba841484d0 msgr2=0x7fba840b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:39.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba82ffd700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba841484d0 0x7fba840b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:39.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba7a7fc700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba84066f70 0x7fba84159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:39.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba82ffd700 1 -- 172.21.15.187:0/948471946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba84066f70 msgr2=0x7fba84159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:39.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba82ffd700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba84066f70 0x7fba84159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:39.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba82ffd700 1 -- 172.21.15.187:0/948471946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84099460 con 0x7fba840622c0 2020-11-20T12:15:39.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba82ffd700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba840622c0 0x7fba8409b040 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7fba84066430 tx=0x7fba74015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:39.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba7b7fe700 1 -- 172.21.15.187:0/948471946 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7401e460 con 0x7fba840622c0 2020-11-20T12:15:39.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba7b7fe700 1 -- 172.21.15.187:0/948471946 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7401e5c0 con 0x7fba840622c0 2020-11-20T12:15:39.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba84099700 con 0x7fba840622c0 2020-11-20T12:15:39.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba7b7fe700 1 -- 172.21.15.187:0/948471946 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7401e940 con 0x7fba840622c0 2020-11-20T12:15:39.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.705+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba840b3b90 con 0x7fba840622c0 2020-11-20T12:15:39.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.707+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba84099890 con 0x7fba840622c0 2020-11-20T12:15:39.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.707+0000 7fba7b7fe700 1 -- 172.21.15.187:0/948471946 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fba74025030 con 0x7fba840622c0 2020-11-20T12:15:39.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.708+0000 7fba7b7fe700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba60056d20 0x7fba600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:39.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.708+0000 7fba7b7fe700 1 -- 172.21.15.187:0/948471946 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fba74071880 con 0x7fba840622c0 2020-11-20T12:15:39.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.708+0000 7fba7a7fc700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba60056d20 0x7fba600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:39.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.709+0000 7fba7a7fc700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba60056d20 0x7fba600591e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fba840a0d60 tx=0x7fba6c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:39.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.712+0000 7fba7b7fe700 1 -- 172.21.15.187:0/948471946 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fba74046d00 con 0x7fba840622c0 2020-11-20T12:15:39.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.957+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba84061a20 con 0x7fba840622c0 2020-11-20T12:15:39.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.958+0000 7fba7b7fe700 1 -- 172.21.15.187:0/948471946 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fba74023070 con 0x7fba840622c0 2020-11-20T12:15:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:39.963 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:39.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba60056d20 msgr2=0x7fba600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:39.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba60056d20 0x7fba600591e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fba840a0d60 tx=0x7fba6c008040).stop 2020-11-20T12:15:39.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba840622c0 msgr2=0x7fba8409b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba840622c0 0x7fba8409b040 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7fba84066430 tx=0x7fba74015000).stop 2020-11-20T12:15:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 shutdown_connections 2020-11-20T12:15:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba60056d20 0x7fba600591e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba841484d0 0x7fba840b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba840622c0 0x7fba8409b040 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 --2- 172.21.15.187:0/948471946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba84066f70 0x7fba84159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:39.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.961+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 >> 172.21.15.187:0/948471946 conn(0x7fba840bab30 msgr2=0x7fba840b6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:39.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.962+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 shutdown_connections 2020-11-20T12:15:39.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:39.962+0000 7fba83fff700 1 -- 172.21.15.187:0/948471946 wait complete. 2020-11-20T12:15:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:39.991 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2020-11-20T12:15:40.266 INFO:teuthology.orchestra.run.smithi187.stdout:{"purge_queue":{"pq_executed":1}} 2020-11-20T12:15:40.267 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2020-11-20T12:15:40.267 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:15:40.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.353+0000 7f4de56c7d00 1 Processor -- start 2020-11-20T12:15:40.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.353+0000 7f4de56c7d00 1 -- start start 2020-11-20T12:15:40.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.354+0000 7f4de56c7d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97e43720 0x555c97e26760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.354+0000 7f4de56c7d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 0x555c97e3f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.354+0000 7f4de56c7d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x555c97ed5e80 0x555c97e288b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.354+0000 7f4de56c7d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c97dade30 con 0x555c97ed0ee0 2020-11-20T12:15:40.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.354+0000 7f4de56c7d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c97d7c810 con 0x555c97e43720 2020-11-20T12:15:40.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.354+0000 7f4de56c7d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c97e3d5e0 con 0x555c97ed5e80 2020-11-20T12:15:40.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.354+0000 7f4dd5f6c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97e43720 0x555c97e26760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd5f6c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97e43720 0x555c97e26760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50210/0 (socket says 172.21.15.187:50210) 2020-11-20T12:15:40.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd5f6c700 1 -- 172.21.15.187:0/3032924040 learned_addr learned my addr 172.21.15.187:0/3032924040 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:40.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd5f6c700 1 -- 172.21.15.187:0/3032924040 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x555c97ed5e80 msgr2=0x555c97e288b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:40.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/3032924040 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x555c97ed5e80 0x555c97e288b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:40.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/3032924040 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 0x555c97e3f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd5f6c700 1 -- 172.21.15.187:0/3032924040 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 msgr2=0x555c97e3f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:40.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/3032924040 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 0x555c97e3f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd5f6c700 1 -- 172.21.15.187:0/3032924040 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c97da1ec0 con 0x555c97e43720 2020-11-20T12:15:40.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/3032924040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97e43720 0x555c97e26760 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7f4dc800b610 tx=0x7f4dc80017e0).ready entity=mon.1 client_cookie=7f5a621a1055b60a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.355+0000 7f4dd4f6a700 1 -- 172.21.15.187:0/3032924040 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800d5a0 con 0x555c97e43720 2020-11-20T12:15:40.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4dd4f6a700 1 -- 172.21.15.187:0/3032924040 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc800d700 con 0x555c97e43720 2020-11-20T12:15:40.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4dd4f6a700 1 -- 172.21.15.187:0/3032924040 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800daa0 con 0x555c97e43720 2020-11-20T12:15:40.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 -- 172.21.15.187:0/3032924040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97e43720 msgr2=0x555c97e26760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/3032924040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97e43720 0x555c97e26760 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7f4dc800b610 tx=0x7f4dc80017e0).stop 2020-11-20T12:15:40.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 -- 172.21.15.187:0/3032924040 shutdown_connections 2020-11-20T12:15:40.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/3032924040 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x555c97ed5e80 0x555c97e288b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:40.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/3032924040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97e43720 0x555c97e26760 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/3032924040 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 0x555c97e3f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 -- 172.21.15.187:0/3032924040 >> 172.21.15.187:0/3032924040 conn(0x555c97dffee0 msgr2=0x555c97ed12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:40.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 -- 172.21.15.187:0/3032924040 shutdown_connections 2020-11-20T12:15:40.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.356+0000 7f4de56c7d00 1 -- 172.21.15.187:0/3032924040 wait complete. 2020-11-20T12:15:40.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.357+0000 7f4de56c7d00 1 Processor -- start 2020-11-20T12:15:40.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.357+0000 7f4de56c7d00 1 -- start start 2020-11-20T12:15:40.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4de56c7d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x555c97e43720 0x555c97e268d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4de56c7d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 0x555c97ec7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4de56c7d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97ed5e80 0x555c97ec76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4de56c7d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c97ecc850 con 0x555c97ed0ee0 2020-11-20T12:15:40.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4de56c7d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c97ec7be0 con 0x555c97ed5e80 2020-11-20T12:15:40.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4de56c7d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c97ecb500 con 0x555c97e43720 2020-11-20T12:15:40.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4dd676d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97ed5e80 0x555c97ec76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4dd676d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97ed5e80 0x555c97ec76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50220/0 (socket says 172.21.15.187:50220) 2020-11-20T12:15:40.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.358+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 learned_addr learned my addr 172.21.15.187:0/2465364031 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:40.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.359+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 0x555c97ec7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.359+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x555c97e43720 0x555c97e268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.359+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x555c97e43720 msgr2=0x555c97e268d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.359+0000 7f4dd676d700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x555c97e43720 0x555c97e268d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.359+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 msgr2=0x555c97ec7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.359+0000 7f4dd676d700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x555c97ed0ee0 0x555c97ec7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.359+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c97d80250 con 0x555c97ed5e80 2020-11-20T12:15:40.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.359+0000 7f4dd676d700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97ed5e80 0x555c97ec76f0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x555c97df0690 tx=0x7f4dd0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.360+0000 7f4dceffd700 1 -- 172.21.15.187:0/2465364031 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0008640 con 0x555c97ed5e80 2020-11-20T12:15:40.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.360+0000 7f4dceffd700 1 -- 172.21.15.187:0/2465364031 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd0005850 con 0x555c97ed5e80 2020-11-20T12:15:40.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.360+0000 7f4dceffd700 1 -- 172.21.15.187:0/2465364031 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd001b840 con 0x555c97ed5e80 2020-11-20T12:15:40.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.360+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555c97ecca40 con 0x555c97ed5e80 2020-11-20T12:15:40.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.360+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c97eccf80 con 0x555c97ed5e80 2020-11-20T12:15:40.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.361+0000 7f4dceffd700 1 -- 172.21.15.187:0/2465364031 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4dd0023030 con 0x555c97ed5e80 2020-11-20T12:15:40.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.362+0000 7f4dceffd700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4db80559a0 0x7f4db8057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.362+0000 7f4dceffd700 1 -- 172.21.15.187:0/2465364031 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4dd0070630 con 0x555c97ed5e80 2020-11-20T12:15:40.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.362+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4db80559a0 0x7f4db8057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.362+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x555c97ed1da0 0x555c97ee2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.362+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:1 22.0 22:00000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x555c97e016a0 con 0x555c97ed1da0 2020-11-20T12:15:40.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.363+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x555c97ed1da0 0x555c97ee2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.363+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4db80559a0 0x7f4db8057e60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f4dc8000f20 tx=0x555c97e02600).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.364+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x555c97ed1da0 0x555c97ee2fa0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.364+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== osd_op_reply(1 [pgnls start_epoch 108 out=49b] v108'4 uv4 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4dc000e520 con 0x555c97ed1da0 2020-11-20T12:15:40.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.364+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f4dc0010880 0x7f4dc000f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.364+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- osd_op(unknown.0.0:2 22.4 22:20000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x555c97e016a0 con 0x7f4dc0010880 2020-11-20T12:15:40.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.365+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f4dc0010880 0x7f4dc000f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.367+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f4dc0010880 0x7f4dc000f300 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.368+0000 7f4dd5f6c700 1 -- 172.21.15.187:0/2465364031 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== osd_op_reply(2 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4dc8006030 con 0x7f4dc0010880 2020-11-20T12:15:40.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.368+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f4dc8019030 0x7f4dc800e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.368+0000 7f4dd5f6c700 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:3 22.2 22:40000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x555c97e016a0 con 0x7f4dc8019030 2020-11-20T12:15:40.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.368+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f4dc8019030 0x7f4dc800e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.370+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f4dc8019030 0x7f4dc800e430 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.372+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(3 [pgnls start_epoch 108 out=49b] v108'3 uv2 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4dc000e520 con 0x7f4dc8019030 2020-11-20T12:15:40.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.372+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f4dc000e7f0 0x7f4dc0013900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.372+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] -- osd_op(unknown.0.0:4 22.6 22:60000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x555c97e016a0 con 0x7f4dc000e7f0 2020-11-20T12:15:40.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.372+0000 7f4dd676d700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f4dc000e7f0 0x7f4dc0013900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.373+0000 7f4dd676d700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f4dc000e7f0 0x7f4dc0013900 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.373+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 <== osd.3 v2:172.21.15.187:6826/19935 1 ==== osd_op_reply(4 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4dd0047b90 con 0x7f4dc000e7f0 2020-11-20T12:15:40.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.373+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:5 22.1 22:80000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x555c97e016a0 con 0x555c97ed1da0 2020-11-20T12:15:40.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.374+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 <== osd.2 v2:172.21.15.187:6810/19904 2 ==== osd_op_reply(5 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4dc000e520 con 0x555c97ed1da0 2020-11-20T12:15:40.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.374+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- osd_op(unknown.0.0:6 22.5 22:a0000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x555c97e016a0 con 0x555c97ed1da0 2020-11-20T12:15:40.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.374+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 <== osd.2 v2:172.21.15.187:6810/19904 3 ==== osd_op_reply(6 [pgnls start_epoch 108 out=49b] v108'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4dc000e520 con 0x555c97ed1da0 2020-11-20T12:15:40.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.374+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f4dc0003560 0x7f4dc001f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.374+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:7 22.3 22:c0000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x555c97e016a0 con 0x7f4dc0003560 2020-11-20T12:15:40.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.375+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f4dc0003560 0x7f4dc001f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.375+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f4dc0003560 0x7f4dc001f2b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.376+0000 7f4dd5f6c700 1 -- 172.21.15.187:0/2465364031 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(7 [pgnls start_epoch 108 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4dc8006030 con 0x7f4dc0003560 2020-11-20T12:15:40.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.376+0000 7f4dd5f6c700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f4dc8019920 0x7f4dc80283f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.376+0000 7f4dd5f6c700 1 -- 172.21.15.187:0/2465364031 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:8 22.7 22:e0000000::::head [pgnls start_epoch 108 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108) v8 -- 0x555c97e016a0 con 0x7f4dc8019920 2020-11-20T12:15:40.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.376+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f4dc8019920 0x7f4dc80283f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.378+0000 7f4dcd76b700 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f4dc8019920 0x7f4dc80283f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4dcd76b700 1 -- 172.21.15.187:0/2465364031 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(8 [pgnls start_epoch 108 out=49b] v108'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4dc000e520 con 0x7f4dc8019920 2020-11-20T12:15:40.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f4dc0003560 msgr2=0x7f4dc001f2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f4dc0003560 0x7f4dc001f2b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x555c97ed1da0 msgr2=0x555c97ee2fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x555c97ed1da0 0x555c97ee2fa0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f4dc000e7f0 msgr2=0x7f4dc0013900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f4dc000e7f0 0x7f4dc0013900 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f4dc0010880 msgr2=0x7f4dc000f300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7f4dc0010880 0x7f4dc000f300 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f4dc8019030 msgr2=0x7f4dc800e430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f4dc8019030 0x7f4dc800e430 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 reap_dead start 2020-11-20T12:15:40.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f4dc8019920 msgr2=0x7f4dc80283f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.379+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f4dc8019920 0x7f4dc80283f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 reap_dead start 2020-11-20T12:15:40.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4dd676d700 1 -- 172.21.15.187:0/2465364031 reap_dead start 2020-11-20T12:15:40.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4db80559a0 msgr2=0x7f4db8057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4db80559a0 0x7f4db8057e60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f4dc8000f20 tx=0x555c97e02600).stop 2020-11-20T12:15:40.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97ed5e80 msgr2=0x555c97ec76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97ed5e80 0x555c97ec76f0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x555c97df0690 tx=0x7f4dd0015000).stop 2020-11-20T12:15:40.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 shutdown_connections 2020-11-20T12:15:40.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f4dc8019920 0x7f4dc80283f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555c97ed5e80 0x555c97ec76f0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 --2- 172.21.15.187:0/2465364031 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4db80559a0 0x7f4db8057e60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 >> 172.21.15.187:0/2465364031 conn(0x555c97dffee0 msgr2=0x555c97ed12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:40.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.380+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 shutdown_connections 2020-11-20T12:15:40.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.381+0000 7f4de56c7d00 1 -- 172.21.15.187:0/2465364031 wait complete. 2020-11-20T12:15:40.393 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 8388608 are absent 2020-11-20T12:15:40.393 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2020-11-20T12:15:40.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.624+0000 7f7ae2db6700 1 Processor -- start 2020-11-20T12:15:40.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.624+0000 7f7ae2db6700 1 -- start start 2020-11-20T12:15:40.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.626+0000 7f7ae2db6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7adc065350 0x7f7adc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.626+0000 7f7ae2db6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc065730 0x7f7adc0622c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.626+0000 7f7ae2db6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7adc14c110 con 0x7f7adc0b1790 2020-11-20T12:15:40.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.626+0000 7f7ae2db6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7adc14c3b0 con 0x7f7adc065350 2020-11-20T12:15:40.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.626+0000 7f7ae2db6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc14c650 con 0x7f7adc065730 2020-11-20T12:15:40.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.627+0000 7f7ae25b5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7adc0b1790 0x7f7adc147d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46846/0 (socket says 172.21.15.187:46846) 2020-11-20T12:15:40.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.627+0000 7f7ae25b5700 1 -- 172.21.15.187:0/1373510880 learned_addr learned my addr 172.21.15.187:0/1373510880 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:40.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.627+0000 7f7ae1db4700 1 --2- 172.21.15.187:0/1373510880 >> v2:172.21.15.5:3300/0 conn(0x7f7adc065350 0x7f7adc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.627+0000 7f7ae15b3700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc065730 0x7f7adc0622c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.627+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242308669 0 0) 0x7f7adc14c110 con 0x7f7adc0b1790 2020-11-20T12:15:40.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.627+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ac4003240 con 0x7f7adc0b1790 2020-11-20T12:15:40.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 869813116 0 0) 0x7f7ac4003240 con 0x7f7adc0b1790 2020-11-20T12:15:40.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc065730 msgr2=0x7f7adc0622c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:40.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc065730 0x7f7adc0622c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 >> v2:172.21.15.5:3300/0 conn(0x7f7adc065350 msgr2=0x7f7adc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:40.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 --2- 172.21.15.187:0/1373510880 >> v2:172.21.15.5:3300/0 conn(0x7f7adc065350 0x7f7adc061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc14be00 con 0x7f7adc0b1790 2020-11-20T12:15:40.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f7ad40018b0 con 0x7f7adc0b1790 2020-11-20T12:15:40.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 >> v1:172.21.15.187:6789/0 conn(0x7f7adc0b1790 legacy=0x7f7adc147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ac4003730 0x7f7ac4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ac40044e0 0x7f7ac4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc065730 0x7f7ac4004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc14c3b0 con 0x7f7ac40044e0 2020-11-20T12:15:40.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc14c650 con 0x7f7ac4003730 2020-11-20T12:15:40.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac4004270 con 0x7f7adc065730 2020-11-20T12:15:40.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.628+0000 7f7ae1db4700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ac4003730 0x7f7ac4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.629+0000 7f7ae25b5700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc065730 0x7f7ac4004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.629+0000 7f7ae15b3700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ac40044e0 0x7f7ac4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.629+0000 7f7ae1db4700 1 -- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc065730 msgr2=0x7f7ac4004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:40.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.629+0000 7f7ae1db4700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc065730 0x7f7ac4004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.629+0000 7f7ae1db4700 1 -- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ac40044e0 msgr2=0x7f7ac4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:40.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.629+0000 7f7ae1db4700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ac40044e0 0x7f7ac4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.629+0000 7f7ae1db4700 1 -- 172.21.15.187:0/1373510880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac4003240 con 0x7f7ac4003730 2020-11-20T12:15:40.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.629+0000 7f7ae1db4700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ac4003730 0x7f7ac4003b50 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f7ad800bb90 tx=0x7f7ad8008040).ready entity=mon.1 client_cookie=2b2d3f19225eb7d0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800b2c0 con 0x7f7ac4003730 2020-11-20T12:15:40.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad800ea90 con 0x7f7ac4003730 2020-11-20T12:15:40.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae0db2700 1 -- 172.21.15.187:0/1373510880 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800ee10 con 0x7f7ac4003730 2020-11-20T12:15:40.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae2db6700 1 -- 172.21.15.187:0/1373510880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ac4003730 msgr2=0x7f7ac4003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae2db6700 1 --2- 172.21.15.187:0/1373510880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ac4003730 0x7f7ac4003b50 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f7ad800bb90 tx=0x7f7ad8008040).stop 2020-11-20T12:15:40.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae25b5700 1 -- 172.21.15.187:0/1373510880 reap_dead start 2020-11-20T12:15:40.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae2db6700 1 -- 172.21.15.187:0/1373510880 shutdown_connections 2020-11-20T12:15:40.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae2db6700 1 -- 172.21.15.187:0/1373510880 >> 172.21.15.187:0/1373510880 conn(0x7f7adc0bab10 msgr2=0x7f7adc0baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:40.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae2db6700 1 -- 172.21.15.187:0/1373510880 shutdown_connections 2020-11-20T12:15:40.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.630+0000 7f7ae2db6700 1 -- 172.21.15.187:0/1373510880 wait complete. 2020-11-20T12:15:40.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.632+0000 7f7ae2db6700 1 Processor -- start 2020-11-20T12:15:40.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.632+0000 7f7ae2db6700 1 -- start start 2020-11-20T12:15:40.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae2db6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc1571a0 0x7f7adc157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae2db6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7adc0b3b10 0x7f7adc157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae2db6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7adc0b57c0 0x7f7adc0b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae2db6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0b63c0 con 0x7f7adc0b3b10 2020-11-20T12:15:40.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae2db6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0b6660 con 0x7f7adc0b57c0 2020-11-20T12:15:40.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae2db6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0b60f0 con 0x7f7adc1571a0 2020-11-20T12:15:40.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae25b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7adc0b57c0 0x7f7adc0b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae25b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7adc0b57c0 0x7f7adc0b5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50252/0 (socket says 172.21.15.187:50252) 2020-11-20T12:15:40.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.633+0000 7f7ae25b5700 1 -- 172.21.15.187:0/3241783282 learned_addr learned my addr 172.21.15.187:0/3241783282 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:40.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae1db4700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc1571a0 0x7f7adc157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae15b3700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7adc0b3b10 0x7f7adc157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae25b5700 1 -- 172.21.15.187:0/3241783282 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc1571a0 msgr2=0x7f7adc157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae25b5700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc1571a0 0x7f7adc157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae25b5700 1 -- 172.21.15.187:0/3241783282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7adc0b3b10 msgr2=0x7f7adc157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae25b5700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7adc0b3b10 0x7f7adc157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae25b5700 1 -- 172.21.15.187:0/3241783282 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc0b43d0 con 0x7f7adc0b57c0 2020-11-20T12:15:40.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae25b5700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7adc0b57c0 0x7f7adc0b5c00 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f7ad4001b70 tx=0x7f7ad4001260).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ac9ffb700 1 -- 172.21.15.187:0/3241783282 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad400cb40 con 0x7f7adc0b57c0 2020-11-20T12:15:40.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ac9ffb700 1 -- 172.21.15.187:0/3241783282 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad400cca0 con 0x7f7adc0b57c0 2020-11-20T12:15:40.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ac9ffb700 1 -- 172.21.15.187:0/3241783282 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4014430 con 0x7f7adc0b57c0 2020-11-20T12:15:40.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc13db20 con 0x7f7adc0b57c0 2020-11-20T12:15:40.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.634+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc13e040 con 0x7f7adc0b57c0 2020-11-20T12:15:40.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.635+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc061a20 con 0x7f7adc0b57c0 2020-11-20T12:15:40.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.636+0000 7f7ac9ffb700 1 -- 172.21.15.187:0/3241783282 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7ad4014820 con 0x7f7adc0b57c0 2020-11-20T12:15:40.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.637+0000 7f7ac9ffb700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ab4056d20 0x7f7ab40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:40.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.637+0000 7f7ac9ffb700 1 -- 172.21.15.187:0/3241783282 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7ad406fe40 con 0x7f7adc0b57c0 2020-11-20T12:15:40.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.637+0000 7f7ae1db4700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ab4056d20 0x7f7ab40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:40.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.638+0000 7f7ae1db4700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ab4056d20 0x7f7ab40591e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f7adc05ea30 tx=0x7f7ad801a000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:40.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.641+0000 7f7ac9ffb700 1 -- 172.21.15.187:0/3241783282 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7ad4045210 con 0x7f7adc0b57c0 2020-11-20T12:15:40.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.898+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f7adc14caf0 con 0x7f7adc0b57c0 2020-11-20T12:15:40.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.899+0000 7f7ac9ffb700 1 -- 172.21.15.187:0/3241783282 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1943 (secure 0 0 0) 0x7f7ad4048830 con 0x7f7adc0b57c0 2020-11-20T12:15:40.903 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:40.904 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:15:40.904 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:15:40.904 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:15:40.904 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 772938199040, 2020-11-20T12:15:40.905 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 155914240, 2020-11-20T12:15:40.905 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 155914240, 2020-11-20T12:15:40.905 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 0.00020167563343420625, 2020-11-20T12:15:40.905 INFO:teuthology.orchestra.run.smithi187.stdout: "num_osds": 8, 2020-11-20T12:15:40.905 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_osds": 8, 2020-11-20T12:15:40.906 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_omap_osds": 8 2020-11-20T12:15:40.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:40.906 INFO:teuthology.orchestra.run.smithi187.stdout: "stats_by_class": { 2020-11-20T12:15:40.906 INFO:teuthology.orchestra.run.smithi187.stdout: "ssd": { 2020-11-20T12:15:40.907 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:15:40.907 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 772938199040, 2020-11-20T12:15:40.907 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 155914240, 2020-11-20T12:15:40.907 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 155914240, 2020-11-20T12:15:40.907 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 0.00020167563343420625 2020-11-20T12:15:40.908 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:40.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:40.908 INFO:teuthology.orchestra.run.smithi187.stdout: "pools": [ 2020-11-20T12:15:40.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:40.908 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "device_health_metrics", 2020-11-20T12:15:40.909 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 1, 2020-11-20T12:15:40.909 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:15:40.909 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:15:40.909 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 2, 2020-11-20T12:15:40.910 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:15:40.910 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:15:40.910 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:15:40.910 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367135064064 2020-11-20T12:15:40.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:40.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:40.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:40.911 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "rbd", 2020-11-20T12:15:40.911 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 2, 2020-11-20T12:15:40.911 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:15:40.912 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:15:40.912 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:15:40.912 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:15:40.912 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:15:40.912 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:15:40.913 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367135064064 2020-11-20T12:15:40.913 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:40.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:40.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:40.914 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_metadata", 2020-11-20T12:15:40.914 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 21, 2020-11-20T12:15:40.914 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:15:40.914 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 21049, 2020-11-20T12:15:40.914 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 23, 2020-11-20T12:15:40.915 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 104, 2020-11-20T12:15:40.915 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 106496, 2020-11-20T12:15:40.915 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 1.4503650902497611e-07, 2020-11-20T12:15:40.915 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367135064064 2020-11-20T12:15:40.915 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:40.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:40.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:40.916 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_data", 2020-11-20T12:15:40.916 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 22, 2020-11-20T12:15:40.916 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:15:40.917 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:15:40.917 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:15:40.917 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:15:40.917 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:15:40.918 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:15:40.918 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367135064064 2020-11-20T12:15:40.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:40.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:40.918 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:15:40.919 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:15:40.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.901+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ab4056d20 msgr2=0x7f7ab40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.901+0000 7f7ae2db6700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ab4056d20 0x7f7ab40591e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f7adc05ea30 tx=0x7f7ad801a000).stop 2020-11-20T12:15:40.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.901+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7adc0b57c0 msgr2=0x7f7adc0b5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:40.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.901+0000 7f7ae2db6700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7adc0b57c0 0x7f7adc0b5c00 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f7ad4001b70 tx=0x7f7ad4001260).stop 2020-11-20T12:15:40.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.901+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 shutdown_connections 2020-11-20T12:15:40.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.902+0000 7f7ae2db6700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ab4056d20 0x7f7ab40591e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.902+0000 7f7ae2db6700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7adc0b57c0 0x7f7adc0b5c00 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.902+0000 7f7ae2db6700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7adc1571a0 0x7f7adc157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.902+0000 7f7ae2db6700 1 --2- 172.21.15.187:0/3241783282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7adc0b3b10 0x7f7adc157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:40.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.902+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 >> 172.21.15.187:0/3241783282 conn(0x7f7adc0bab10 msgr2=0x7f7adc0afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:40.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.902+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 shutdown_connections 2020-11-20T12:15:40.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:40.902+0000 7f7ae2db6700 1 -- 172.21.15.187:0/3241783282 wait complete. 2020-11-20T12:15:40.928 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2020-11-20T12:15:40.929 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:15:40.929 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:15:41.050 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:15:41.073 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:15:41.179 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:15:41.204 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:15:41.205 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:15:41.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:15:41.206 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:15:41.328 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:15:41.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:15:41.329 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:15:41.385 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:15:41.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.625+0000 7f8e3a359700 1 Processor -- start 2020-11-20T12:15:41.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.625+0000 7f8e3a359700 1 -- start start 2020-11-20T12:15:41.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.626+0000 7f8e3a359700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8e340bf800 0x7f8e3414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:41.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.627+0000 7f8e3a359700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3414d110 con 0x7f8e3409f560 2020-11-20T12:15:41.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.627+0000 7f8e3a359700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3414cbd0 con 0x7f8e340bfbe0 2020-11-20T12:15:41.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.627+0000 7f8e3a359700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3414ce70 con 0x7f8e340bf800 2020-11-20T12:15:41.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.627+0000 7f8e38b56700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8e340bfbe0 0x7f8e341480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46864/0 (socket says 172.21.15.187:46864) 2020-11-20T12:15:41.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.627+0000 7f8e38b56700 1 -- 172.21.15.187:0/1501419427 learned_addr learned my addr 172.21.15.187:0/1501419427 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:41.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.627+0000 7f8e39357700 1 --2- 172.21.15.187:0/1501419427 >> v2:172.21.15.5:3300/0 conn(0x7f8e340bf800 0x7f8e3414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:41.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293949561 0 0) 0x7f8e3414cbd0 con 0x7f8e340bfbe0 2020-11-20T12:15:41.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e1c003240 con 0x7f8e340bfbe0 2020-11-20T12:15:41.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 166407938 0 0) 0x7f8e1c003240 con 0x7f8e340bfbe0 2020-11-20T12:15:41.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 >> v2:172.21.15.5:3300/0 conn(0x7f8e340bf800 msgr2=0x7f8e3414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:41.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 --2- 172.21.15.187:0/1501419427 >> v2:172.21.15.5:3300/0 conn(0x7f8e340bf800 0x7f8e3414c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:41.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 >> v1:172.21.15.5:6789/0 conn(0x7f8e3409f560 legacy=0x7f8e340a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:41.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e340a4cc0 con 0x7f8e340bfbe0 2020-11-20T12:15:41.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8e24001520 con 0x7f8e340bfbe0 2020-11-20T12:15:41.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 >> v1:172.21.15.187:6789/0 conn(0x7f8e340bfbe0 legacy=0x7f8e341480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:41.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e1c003730 0x7f8e1c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:41.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e1c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:41.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1c004e50 0x7f8e1c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:41.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3414ce70 con 0x7f8e1c004e50 2020-11-20T12:15:41.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c004260 con 0x7f8e1c0044e0 2020-11-20T12:15:41.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.628+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c00bc20 con 0x7f8e1c003730 2020-11-20T12:15:41.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.629+0000 7f8e38b56700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e1c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:41.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.629+0000 7f8e38b56700 1 -- 172.21.15.187:0/1501419427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e1c003730 msgr2=0x7f8e1c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:41.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.629+0000 7f8e38b56700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e1c003730 0x7f8e1c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:41.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.629+0000 7f8e38b56700 1 -- 172.21.15.187:0/1501419427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1c004e50 msgr2=0x7f8e1c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:41.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.629+0000 7f8e38b56700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1c004e50 0x7f8e1c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:41.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.629+0000 7f8e38b56700 1 -- 172.21.15.187:0/1501419427 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c003240 con 0x7f8e1c0044e0 2020-11-20T12:15:41.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.629+0000 7f8e39b58700 1 -- 172.21.15.187:0/1501419427 reap_dead start 2020-11-20T12:15:41.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.629+0000 7f8e38b56700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e1c004960 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7f8e2400c440 tx=0x7f8e240077d0).ready entity=mon.1 client_cookie=81f24f878ebfa481 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:41.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.630+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e240072e0 con 0x7f8e1c0044e0 2020-11-20T12:15:41.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.630+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e24007440 con 0x7f8e1c0044e0 2020-11-20T12:15:41.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.630+0000 7f8e23fff700 1 -- 172.21.15.187:0/1501419427 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e24005840 con 0x7f8e1c0044e0 2020-11-20T12:15:41.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.631+0000 7f8e3a359700 1 -- 172.21.15.187:0/1501419427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 msgr2=0x7f8e1c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:41.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.631+0000 7f8e3a359700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e1c004960 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7f8e2400c440 tx=0x7f8e240077d0).stop 2020-11-20T12:15:41.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.631+0000 7f8e3a359700 1 -- 172.21.15.187:0/1501419427 shutdown_connections 2020-11-20T12:15:41.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.631+0000 7f8e3a359700 1 --2- 172.21.15.187:0/1501419427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e1c004960 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:41.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.631+0000 7f8e3a359700 1 -- 172.21.15.187:0/1501419427 >> 172.21.15.187:0/1501419427 conn(0x7f8e34068370 msgr2=0x7f8e340bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:41.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.631+0000 7f8e3a359700 1 -- 172.21.15.187:0/1501419427 shutdown_connections 2020-11-20T12:15:41.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.631+0000 7f8e3a359700 1 -- 172.21.15.187:0/1501419427 wait complete. 2020-11-20T12:15:41.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.633+0000 7f8e3a359700 1 Processor -- start 2020-11-20T12:15:41.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.633+0000 7f8e3a359700 1 -- start start 2020-11-20T12:15:41.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.634+0000 7f8e3a359700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e34144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:41.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.634+0000 7f8e3a359700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e34147430 0x7f8e34140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:41.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.634+0000 7f8e3a359700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e34130890 0x7f8e34130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:41.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.634+0000 7f8e3a359700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e34131200 con 0x7f8e34130890 2020-11-20T12:15:41.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.634+0000 7f8e3a359700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e34144f10 con 0x7f8e1c0044e0 2020-11-20T12:15:41.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.634+0000 7f8e3a359700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e341451b0 con 0x7f8e34147430 2020-11-20T12:15:41.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.634+0000 7f8e39357700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e34144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:41.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39357700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e34144750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50268/0 (socket says 172.21.15.187:50268) 2020-11-20T12:15:41.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39357700 1 -- 172.21.15.187:0/761466003 learned_addr learned my addr 172.21.15.187:0/761466003 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:41.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e38b56700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e34147430 0x7f8e34140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:41.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39357700 1 -- 172.21.15.187:0/761466003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e34147430 msgr2=0x7f8e34140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:41.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39357700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e34147430 0x7f8e34140380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:41.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39357700 1 -- 172.21.15.187:0/761466003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e34130890 msgr2=0x7f8e34130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:41.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39b58700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e34130890 0x7f8e34130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:41.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39357700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e34130890 0x7f8e34130d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:41.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39357700 1 -- 172.21.15.187:0/761466003 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e34145450 con 0x7f8e1c0044e0 2020-11-20T12:15:41.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.635+0000 7f8e39357700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e34144750 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f8e3000dfd0 tx=0x7f8e30008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:41.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.636+0000 7f8e217fa700 1 -- 172.21.15.187:0/761466003 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30016070 con 0x7f8e1c0044e0 2020-11-20T12:15:41.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.636+0000 7f8e217fa700 1 -- 172.21.15.187:0/761466003 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3000f5e0 con 0x7f8e1c0044e0 2020-11-20T12:15:41.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.636+0000 7f8e217fa700 1 -- 172.21.15.187:0/761466003 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3000f960 con 0x7f8e1c0044e0 2020-11-20T12:15:41.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.636+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e34144c40 con 0x7f8e1c0044e0 2020-11-20T12:15:41.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.636+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e341458f0 con 0x7f8e1c0044e0 2020-11-20T12:15:41.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.637+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e34080050 con 0x7f8e1c0044e0 2020-11-20T12:15:41.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.637+0000 7f8e217fa700 1 -- 172.21.15.187:0/761466003 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8e30012420 con 0x7f8e1c0044e0 2020-11-20T12:15:41.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.638+0000 7f8e217fa700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e0c056d20 0x7f8e0c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:41.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.638+0000 7f8e217fa700 1 -- 172.21.15.187:0/761466003 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e3006fd50 con 0x7f8e1c0044e0 2020-11-20T12:15:41.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.639+0000 7f8e38b56700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e0c056d20 0x7f8e0c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:41.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.639+0000 7f8e38b56700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e0c056d20 0x7f8e0c0591e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f8e3414bfb0 tx=0x7f8e24015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:41.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.643+0000 7f8e217fa700 1 -- 172.21.15.187:0/761466003 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8e30045140 con 0x7f8e1c0044e0 2020-11-20T12:15:41.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.830+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8e3414c2c0 con 0x7f8e1c0044e0 2020-11-20T12:15:41.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.841+0000 7f8e217fa700 1 -- 172.21.15.187:0/761466003 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8e30048760 con 0x7f8e1c0044e0 2020-11-20T12:15:41.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.843+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e0c056d20 msgr2=0x7f8e0c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:41.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.843+0000 7f8e3a359700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e0c056d20 0x7f8e0c0591e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f8e3414bfb0 tx=0x7f8e24015000).stop 2020-11-20T12:15:41.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.843+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 msgr2=0x7f8e34144750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:41.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.843+0000 7f8e3a359700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e34144750 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f8e3000dfd0 tx=0x7f8e30008040).stop 2020-11-20T12:15:41.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.844+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 shutdown_connections 2020-11-20T12:15:41.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.844+0000 7f8e3a359700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8e0c056d20 0x7f8e0c0591e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:41.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.844+0000 7f8e3a359700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e34130890 0x7f8e34130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:41.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.844+0000 7f8e3a359700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e1c0044e0 0x7f8e34144750 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:41.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.844+0000 7f8e3a359700 1 --2- 172.21.15.187:0/761466003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e34147430 0x7f8e34140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:41.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.844+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 >> 172.21.15.187:0/761466003 conn(0x7f8e34068370 msgr2=0x7f8e34141870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:41.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.844+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 shutdown_connections 2020-11-20T12:15:41.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:41.844+0000 7f8e3a359700 1 -- 172.21.15.187:0/761466003 wait complete. 2020-11-20T12:15:41.870 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_open_inode' 2020-11-20T12:15:42.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.110+0000 7f023299a700 1 Processor -- start 2020-11-20T12:15:42.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.111+0000 7f023299a700 1 -- start start 2020-11-20T12:15:42.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.112+0000 7f023299a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f022c065f90 0x7f022c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:42.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.112+0000 7f023299a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c09ca90 0x7f022c09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:42.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.112+0000 7f023299a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022c0b13e0 con 0x7f022c065190 2020-11-20T12:15:42.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.112+0000 7f023299a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022c14ab30 con 0x7f022c065f90 2020-11-20T12:15:42.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.112+0000 7f023299a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022c062c10 con 0x7f022c09ca90 2020-11-20T12:15:42.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.113+0000 7f0232199700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c09ca90 0x7f022c09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:42.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.113+0000 7f0232199700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c09ca90 0x7f022c09cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:51168/0 (socket says 172.21.15.187:51168) 2020-11-20T12:15:42.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.113+0000 7f0232199700 1 -- 172.21.15.187:0/431364755 learned_addr learned my addr 172.21.15.187:0/431364755 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:42.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.113+0000 7f0231197700 1 --2- 172.21.15.187:0/431364755 >> v2:172.21.15.5:3300/0 conn(0x7f022c065f90 0x7f022c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:42.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.114+0000 7f0232199700 1 -- 172.21.15.187:0/431364755 >> v2:172.21.15.5:3300/0 conn(0x7f022c065f90 msgr2=0x7f022c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:42.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.114+0000 7f0232199700 1 --2- 172.21.15.187:0/431364755 >> v2:172.21.15.5:3300/0 conn(0x7f022c065f90 0x7f022c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:42.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.114+0000 7f0232199700 1 -- 172.21.15.187:0/431364755 >> v1:172.21.15.5:6789/0 conn(0x7f022c065190 legacy=0x7f022c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:42.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.114+0000 7f0232199700 1 -- 172.21.15.187:0/431364755 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c0b1640 con 0x7f022c09ca90 2020-11-20T12:15:42.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.115+0000 7f0232199700 1 --2- 172.21.15.187:0/431364755 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c09ca90 0x7f022c09cf40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0224006fd0 tx=0x7f0224006eb0).ready entity=mon.2 client_cookie=b49163c118155f32 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:42.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.115+0000 7f0230996700 1 -- 172.21.15.187:0/431364755 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02240098a0 con 0x7f022c09ca90 2020-11-20T12:15:42.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.115+0000 7f0230996700 1 -- 172.21.15.187:0/431364755 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0224009a00 con 0x7f022c09ca90 2020-11-20T12:15:42.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.115+0000 7f0230996700 1 -- 172.21.15.187:0/431364755 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224009d80 con 0x7f022c09ca90 2020-11-20T12:15:42.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.116+0000 7f023299a700 1 -- 172.21.15.187:0/431364755 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c09ca90 msgr2=0x7f022c09cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:42.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.116+0000 7f023299a700 1 --2- 172.21.15.187:0/431364755 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c09ca90 0x7f022c09cf40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0224006fd0 tx=0x7f0224006eb0).stop 2020-11-20T12:15:42.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.116+0000 7f023299a700 1 -- 172.21.15.187:0/431364755 shutdown_connections 2020-11-20T12:15:42.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.116+0000 7f023299a700 1 --2- 172.21.15.187:0/431364755 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c09ca90 0x7f022c09cf40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:42.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.116+0000 7f023299a700 1 --2- 172.21.15.187:0/431364755 >> v2:172.21.15.5:3300/0 conn(0x7f022c065f90 0x7f022c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:42.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.116+0000 7f023299a700 1 -- 172.21.15.187:0/431364755 >> 172.21.15.187:0/431364755 conn(0x7f022c0bab30 msgr2=0x7f022c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:42.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.116+0000 7f023299a700 1 -- 172.21.15.187:0/431364755 shutdown_connections 2020-11-20T12:15:42.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.116+0000 7f023299a700 1 -- 172.21.15.187:0/431364755 wait complete. 2020-11-20T12:15:42.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.118+0000 7f023299a700 1 Processor -- start 2020-11-20T12:15:42.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.118+0000 7f023299a700 1 -- start start 2020-11-20T12:15:42.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.119+0000 7f023299a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c065190 0x7f022c145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:42.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.119+0000 7f023299a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065f90 0x7f022c140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:42.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.119+0000 7f023299a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c09ca90 0x7f022c141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:42.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.119+0000 7f023299a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0b3e80 con 0x7f022c09ca90 2020-11-20T12:15:42.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.119+0000 7f023299a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0b4120 con 0x7f022c065190 2020-11-20T12:15:42.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.119+0000 7f023299a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022c145630 con 0x7f022c065f90 2020-11-20T12:15:42.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.120+0000 7f0231998700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c065190 0x7f022c145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:42.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.120+0000 7f0231998700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c065190 0x7f022c145140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50282/0 (socket says 172.21.15.187:50282) 2020-11-20T12:15:42.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.120+0000 7f0231998700 1 -- 172.21.15.187:0/325826190 learned_addr learned my addr 172.21.15.187:0/325826190 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:42.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.120+0000 7f0231197700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065f90 0x7f022c140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:42.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.120+0000 7f0232199700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c09ca90 0x7f022c141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:42.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0231998700 1 -- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065f90 msgr2=0x7f022c140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:42.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0231998700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065f90 0x7f022c140d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:42.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0231998700 1 -- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c09ca90 msgr2=0x7f022c141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:42.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0231998700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c09ca90 0x7f022c141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:42.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0231998700 1 -- 172.21.15.187:0/325826190 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c0b2480 con 0x7f022c065190 2020-11-20T12:15:42.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0232199700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c09ca90 0x7f022c141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:42.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0231998700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c065190 0x7f022c145140 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f022800ca90 tx=0x7f0228007f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:42.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0219ffb700 1 -- 172.21.15.187:0/325826190 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022800a7c0 con 0x7f022c065190 2020-11-20T12:15:42.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0219ffb700 1 -- 172.21.15.187:0/325826190 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022800a920 con 0x7f022c065190 2020-11-20T12:15:42.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c0b4860 con 0x7f022c065190 2020-11-20T12:15:42.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.121+0000 7f0219ffb700 1 -- 172.21.15.187:0/325826190 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022800aca0 con 0x7f022c065190 2020-11-20T12:15:42.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.122+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c0b3ba0 con 0x7f022c065190 2020-11-20T12:15:42.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.123+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c0b2610 con 0x7f022c065190 2020-11-20T12:15:42.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.123+0000 7f0219ffb700 1 -- 172.21.15.187:0/325826190 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0228019030 con 0x7f022c065190 2020-11-20T12:15:42.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.124+0000 7f0219ffb700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:42.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.124+0000 7f0219ffb700 1 -- 172.21.15.187:0/325826190 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f022806f970 con 0x7f022c065190 2020-11-20T12:15:42.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.124+0000 7f0231197700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:42.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.125+0000 7f0231197700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f022c0b3240 tx=0x7f021c006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:42.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.128+0000 7f0219ffb700 1 -- 172.21.15.187:0/325826190 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0228044dc0 con 0x7f022c065190 2020-11-20T12:15:42.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.381+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_open_inode"]} v 0) v1 -- 0x7f022c061a20 con 0x7f022c065190 2020-11-20T12:15:42.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.879+0000 7f0219ffb700 1 -- 172.21.15.187:0/325826190 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_open_inode"]}]=0 v552) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0228017020 con 0x7f022c065190 2020-11-20T12:15:43.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.883+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 msgr2=0x7f02040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:43.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.883+0000 7f023299a700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f022c0b3240 tx=0x7f021c006df0).stop 2020-11-20T12:15:43.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.883+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c065190 msgr2=0x7f022c145140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:43.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.883+0000 7f023299a700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c065190 0x7f022c145140 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f022800ca90 tx=0x7f0228007f90).stop 2020-11-20T12:15:43.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.884+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 shutdown_connections 2020-11-20T12:15:43.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.884+0000 7f023299a700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0204056d20 0x7f02040591e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.884+0000 7f023299a700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f022c09ca90 0x7f022c141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.884+0000 7f023299a700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f022c065190 0x7f022c145140 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.884+0000 7f023299a700 1 --2- 172.21.15.187:0/325826190 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f022c065f90 0x7f022c140d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.884+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 >> 172.21.15.187:0/325826190 conn(0x7f022c0bab30 msgr2=0x7f022c141a70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:43.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.884+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 shutdown_connections 2020-11-20T12:15:43.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:42.884+0000 7f023299a700 1 -- 172.21.15.187:0/325826190 wait complete. 2020-11-20T12:15:43.676 INFO:tasks.cephfs_test_runner:test_open_inode (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:15:43.677 INFO:tasks.cephfs_test_runner:Starting test: test_ops_throttle (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:15:43.678 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_ops_throttle' 2020-11-20T12:15:43.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.927+0000 7f4a512cd700 1 Processor -- start 2020-11-20T12:15:43.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.928+0000 7f4a512cd700 1 -- start start 2020-11-20T12:15:43.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.929+0000 7f4a512cd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4a4c1484d0 0x7f4a4c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.929+0000 7f4a512cd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c0622c0 0x7f4a4c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.929+0000 7f4a512cd700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4c0b1300 con 0x7f4a4c066f70 2020-11-20T12:15:43.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.929+0000 7f4a512cd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4c067730 con 0x7f4a4c1484d0 2020-11-20T12:15:43.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.929+0000 7f4a512cd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0b15a0 con 0x7f4a4c0622c0 2020-11-20T12:15:43.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.929+0000 7f4a50acc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4a4c066f70 0x7f4a4c0b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46900/0 (socket says 172.21.15.187:46900) 2020-11-20T12:15:43.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.930+0000 7f4a50acc700 1 -- 172.21.15.187:0/873817943 learned_addr learned my addr 172.21.15.187:0/873817943 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:43.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.930+0000 7f4a4bfff700 1 --2- 172.21.15.187:0/873817943 >> v2:172.21.15.5:3300/0 conn(0x7f4a4c1484d0 0x7f4a4c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.930+0000 7f4a437fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c0622c0 0x7f4a4c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.930+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63830925 0 0) 0x7f4a4c0b1300 con 0x7f4a4c066f70 2020-11-20T12:15:43.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.930+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a2c003240 con 0x7f4a4c066f70 2020-11-20T12:15:43.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.930+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2625520065 0 0) 0x7f4a2c003240 con 0x7f4a4c066f70 2020-11-20T12:15:43.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.930+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c0622c0 msgr2=0x7f4a4c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:43.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c0622c0 0x7f4a4c0b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 >> v2:172.21.15.5:3300/0 conn(0x7f4a4c1484d0 msgr2=0x7f4a4c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:43.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 --2- 172.21.15.187:0/873817943 >> v2:172.21.15.5:3300/0 conn(0x7f4a4c1484d0 0x7f4a4c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c0b1840 con 0x7f4a4c066f70 2020-11-20T12:15:43.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4a44001860 con 0x7f4a4c066f70 2020-11-20T12:15:43.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 >> v1:172.21.15.187:6789/0 conn(0x7f4a4c066f70 legacy=0x7f4a4c0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:43.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c0622c0 0x7f4a2c003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a2c003c40 0x7f4a2c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a2c0049d0 0x7f4a2c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c067730 con 0x7f4a2c0049d0 2020-11-20T12:15:43.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0b15a0 con 0x7f4a2c003c40 2020-11-20T12:15:43.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c004200 con 0x7f4a4c0622c0 2020-11-20T12:15:43.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a437fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a2c003c40 0x7f4a2c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.931+0000 7f4a4bfff700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c0622c0 0x7f4a2c003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a50acc700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a2c0049d0 0x7f4a2c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a437fe700 1 -- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c0622c0 msgr2=0x7f4a2c003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:43.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a437fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c0622c0 0x7f4a2c003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a437fe700 1 -- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a2c0049d0 msgr2=0x7f4a2c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:43.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a437fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a2c0049d0 0x7f4a2c004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a437fe700 1 -- 172.21.15.187:0/873817943 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c003240 con 0x7f4a2c003c40 2020-11-20T12:15:43.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a437fe700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a2c003c40 0x7f4a2c0044e0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f4a4c062720 tx=0x7f4a34006df0).ready entity=mon.1 client_cookie=a19e2c512868f562 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:43.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3400b330 con 0x7f4a2c003c40 2020-11-20T12:15:43.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3400eae0 con 0x7f4a2c003c40 2020-11-20T12:15:43.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.932+0000 7f4a4b7fe700 1 -- 172.21.15.187:0/873817943 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3400ee60 con 0x7f4a2c003c40 2020-11-20T12:15:43.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.933+0000 7f4a512cd700 1 -- 172.21.15.187:0/873817943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a2c003c40 msgr2=0x7f4a2c0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:43.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.933+0000 7f4a512cd700 1 --2- 172.21.15.187:0/873817943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a2c003c40 0x7f4a2c0044e0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f4a4c062720 tx=0x7f4a34006df0).stop 2020-11-20T12:15:43.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.933+0000 7f4a50acc700 1 -- 172.21.15.187:0/873817943 reap_dead start 2020-11-20T12:15:43.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.933+0000 7f4a512cd700 1 -- 172.21.15.187:0/873817943 shutdown_connections 2020-11-20T12:15:43.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.933+0000 7f4a512cd700 1 -- 172.21.15.187:0/873817943 >> 172.21.15.187:0/873817943 conn(0x7f4a4c0bab30 msgr2=0x7f4a4c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:43.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.933+0000 7f4a512cd700 1 -- 172.21.15.187:0/873817943 shutdown_connections 2020-11-20T12:15:43.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.933+0000 7f4a512cd700 1 -- 172.21.15.187:0/873817943 wait complete. 2020-11-20T12:15:43.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.934+0000 7f4a512cd700 1 Processor -- start 2020-11-20T12:15:43.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.935+0000 7f4a512cd700 1 -- start start 2020-11-20T12:15:43.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.935+0000 7f4a512cd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c14d9f0 0x7f4a4c14ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.935+0000 7f4a512cd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4c0b4050 0x7f4a4c1586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.935+0000 7f4a512cd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4c09c660 0x7f4a4c0a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.935+0000 7f4a512cd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c09dae0 con 0x7f4a4c09c660 2020-11-20T12:15:43.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.935+0000 7f4a512cd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0b45d0 con 0x7f4a4c0b4050 2020-11-20T12:15:43.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.935+0000 7f4a512cd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0b4870 con 0x7f4a4c14d9f0 2020-11-20T12:15:43.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4c0b4050 0x7f4a4c1586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4c0b4050 0x7f4a4c1586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50306/0 (socket says 172.21.15.187:50306) 2020-11-20T12:15:43.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 -- 172.21.15.187:0/200790179 learned_addr learned my addr 172.21.15.187:0/200790179 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a4bfff700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c14d9f0 0x7f4a4c14ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a50acc700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4c09c660 0x7f4a4c0a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 -- 172.21.15.187:0/200790179 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c14d9f0 msgr2=0x7f4a4c14ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:43.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c14d9f0 0x7f4a4c14ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 -- 172.21.15.187:0/200790179 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4c09c660 msgr2=0x7f4a4c0a0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4c09c660 0x7f4a4c0a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 -- 172.21.15.187:0/200790179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c158b90 con 0x7f4a4c0b4050 2020-11-20T12:15:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.936+0000 7f4a437fe700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4c0b4050 0x7f4a4c1586a0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7f4a34012000 tx=0x7f4a34016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.937+0000 7f4a48ff9700 1 -- 172.21.15.187:0/200790179 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a340125c0 con 0x7f4a4c0b4050 2020-11-20T12:15:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.937+0000 7f4a48ff9700 1 -- 172.21.15.187:0/200790179 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a34012720 con 0x7f4a4c0b4050 2020-11-20T12:15:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.937+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4c09cae0 con 0x7f4a4c0b4050 2020-11-20T12:15:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.937+0000 7f4a48ff9700 1 -- 172.21.15.187:0/200790179 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34012aa0 con 0x7f4a4c0b4050 2020-11-20T12:15:43.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.937+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4c0a06d0 con 0x7f4a4c0b4050 2020-11-20T12:15:43.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.938+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4c09cd80 con 0x7f4a4c0b4050 2020-11-20T12:15:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.939+0000 7f4a48ff9700 1 -- 172.21.15.187:0/200790179 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4a3401f430 con 0x7f4a4c0b4050 2020-11-20T12:15:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.939+0000 7f4a48ff9700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a28056d20 0x7f4a280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.939+0000 7f4a48ff9700 1 -- 172.21.15.187:0/200790179 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4a34071c60 con 0x7f4a4c0b4050 2020-11-20T12:15:43.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.940+0000 7f4a4bfff700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a28056d20 0x7f4a280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:43.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.940+0000 7f4a4bfff700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a28056d20 0x7f4a280591e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f4a3c006a30 tx=0x7f4a3c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:43.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:43.944+0000 7f4a48ff9700 1 -- 172.21.15.187:0/200790179 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4a4c09cd80 con 0x7f4a4c0b4050 2020-11-20T12:15:44.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.197+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_ops_throttle"]} v 0) v1 -- 0x7f4a4c09cd80 con 0x7f4a4c0b4050 2020-11-20T12:15:44.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.896+0000 7f4a48ff9700 1 -- 172.21.15.187:0/200790179 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_ops_throttle"]}]=0 v554) v1 ==== 135+0+0 (secure 0 0 0) 0x7f4a4c09cd80 con 0x7f4a4c0b4050 2020-11-20T12:15:44.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.899+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a28056d20 msgr2=0x7f4a280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:44.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.899+0000 7f4a512cd700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a28056d20 0x7f4a280591e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f4a3c006a30 tx=0x7f4a3c008040).stop 2020-11-20T12:15:44.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.899+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4c0b4050 msgr2=0x7f4a4c1586a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:44.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.899+0000 7f4a512cd700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4c0b4050 0x7f4a4c1586a0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7f4a34012000 tx=0x7f4a34016000).stop 2020-11-20T12:15:44.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.899+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 shutdown_connections 2020-11-20T12:15:44.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.899+0000 7f4a512cd700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a28056d20 0x7f4a280591e0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:44.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.899+0000 7f4a512cd700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4c09c660 0x7f4a4c0a0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:44.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.900+0000 7f4a512cd700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a4c14d9f0 0x7f4a4c14ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:44.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.900+0000 7f4a512cd700 1 --2- 172.21.15.187:0/200790179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4c0b4050 0x7f4a4c1586a0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:44.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.900+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 >> 172.21.15.187:0/200790179 conn(0x7f4a4c0bab30 msgr2=0x7f4a4c0b6440 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:44.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.900+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 shutdown_connections 2020-11-20T12:15:44.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:44.900+0000 7f4a512cd700 1 -- 172.21.15.187:0/200790179 wait complete. 2020-11-20T12:15:44.932 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:15:45.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.180+0000 7f7dbe15a700 1 Processor -- start 2020-11-20T12:15:45.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.181+0000 7f7dbe15a700 1 -- start start 2020-11-20T12:15:45.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.182+0000 7f7dbe15a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065f90 0x7f7db8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.182+0000 7f7dbe15a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7db809ca90 0x7f7db809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.182+0000 7f7dbe15a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db80b13e0 con 0x7f7db8065190 2020-11-20T12:15:45.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.182+0000 7f7dbe15a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db814ab30 con 0x7f7db809ca90 2020-11-20T12:15:45.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.182+0000 7f7dbe15a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db8062c10 con 0x7f7db8065f90 2020-11-20T12:15:45.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.183+0000 7f7dbd158700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7db8065190 0x7f7db8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46916/0 (socket says 172.21.15.187:46916) 2020-11-20T12:15:45.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.183+0000 7f7dbd158700 1 -- 172.21.15.187:0/3012032342 learned_addr learned my addr 172.21.15.187:0/3012032342 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:45.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.183+0000 7f7dbc957700 1 --2- 172.21.15.187:0/3012032342 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065f90 0x7f7db8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.183+0000 7f7dbd959700 1 --2- 172.21.15.187:0/3012032342 >> v2:172.21.15.5:3300/0 conn(0x7f7db809ca90 0x7f7db809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.184+0000 7f7da7fff700 1 -- 172.21.15.187:0/3012032342 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716236449 0 0) 0x7f7db80b13e0 con 0x7f7db8065190 2020-11-20T12:15:45.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.184+0000 7f7da7fff700 1 -- 172.21.15.187:0/3012032342 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da0003240 con 0x7f7db8065190 2020-11-20T12:15:45.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.184+0000 7f7da7fff700 1 -- 172.21.15.187:0/3012032342 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 519081914 0 0) 0x7f7da0003240 con 0x7f7db8065190 2020-11-20T12:15:45.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.184+0000 7f7dbc957700 1 -- 172.21.15.187:0/3012032342 >> v2:172.21.15.5:3300/0 conn(0x7f7db809ca90 msgr2=0x7f7db809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:45.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.184+0000 7f7dbc957700 1 --2- 172.21.15.187:0/3012032342 >> v2:172.21.15.5:3300/0 conn(0x7f7db809ca90 0x7f7db809cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.184+0000 7f7dbc957700 1 -- 172.21.15.187:0/3012032342 >> v1:172.21.15.187:6789/0 conn(0x7f7db8065190 legacy=0x7f7db8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.184+0000 7f7dbc957700 1 -- 172.21.15.187:0/3012032342 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db80b1640 con 0x7f7db8065f90 2020-11-20T12:15:45.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.184+0000 7f7dbc957700 1 --2- 172.21.15.187:0/3012032342 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065f90 0x7f7db8065570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7db814ac70 tx=0x7f7da8006df0).ready entity=mon.2 client_cookie=14e8fb9e441b2e6b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:45.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.185+0000 7f7da7fff700 1 -- 172.21.15.187:0/3012032342 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db40014e0 con 0x7f7db8065f90 2020-11-20T12:15:45.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.185+0000 7f7da7fff700 1 -- 172.21.15.187:0/3012032342 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da8011400 con 0x7f7db8065f90 2020-11-20T12:15:45.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.185+0000 7f7da7fff700 1 -- 172.21.15.187:0/3012032342 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8011780 con 0x7f7db8065f90 2020-11-20T12:15:45.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.185+0000 7f7dbe15a700 1 -- 172.21.15.187:0/3012032342 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065f90 msgr2=0x7f7db8065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.186+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/3012032342 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065f90 0x7f7db8065570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7db814ac70 tx=0x7f7da8006df0).stop 2020-11-20T12:15:45.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.186+0000 7f7dbe15a700 1 -- 172.21.15.187:0/3012032342 shutdown_connections 2020-11-20T12:15:45.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.186+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/3012032342 >> v2:172.21.15.5:3300/0 conn(0x7f7db809ca90 0x7f7db809cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.186+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/3012032342 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065f90 0x7f7db8065570 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.186+0000 7f7dbe15a700 1 -- 172.21.15.187:0/3012032342 >> 172.21.15.187:0/3012032342 conn(0x7f7db80bab30 msgr2=0x7f7db80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:45.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.186+0000 7f7dbe15a700 1 -- 172.21.15.187:0/3012032342 shutdown_connections 2020-11-20T12:15:45.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.186+0000 7f7dbe15a700 1 -- 172.21.15.187:0/3012032342 wait complete. 2020-11-20T12:15:45.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.188+0000 7f7dbe15a700 1 Processor -- start 2020-11-20T12:15:45.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.188+0000 7f7dbe15a700 1 -- start start 2020-11-20T12:15:45.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbe15a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065190 0x7f7db80b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbe15a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7db8065f90 0x7f7db8099160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbe15a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7db809ca90 0x7f7db8099650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbe15a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db815c6b0 con 0x7f7db8065f90 2020-11-20T12:15:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbe15a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db8146670 con 0x7f7db809ca90 2020-11-20T12:15:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbe15a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db815cae0 con 0x7f7db8065190 2020-11-20T12:15:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbd959700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7db809ca90 0x7f7db8099650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbd959700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7db809ca90 0x7f7db8099650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50320/0 (socket says 172.21.15.187:50320) 2020-11-20T12:15:45.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.189+0000 7f7dbd959700 1 -- 172.21.15.187:0/526443524 learned_addr learned my addr 172.21.15.187:0/526443524 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:45.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7dbd158700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065190 0x7f7db80b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7dbc957700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7db8065f90 0x7f7db8099160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7dbd959700 1 -- 172.21.15.187:0/526443524 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065190 msgr2=0x7f7db80b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7dbd959700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065190 0x7f7db80b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7dbd959700 1 -- 172.21.15.187:0/526443524 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7db8065f90 msgr2=0x7f7db8099160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7dbd959700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7db8065f90 0x7f7db8099160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7dbd959700 1 -- 172.21.15.187:0/526443524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db8146910 con 0x7f7db809ca90 2020-11-20T12:15:45.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7dbd959700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7db809ca90 0x7f7db8099650 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7f7db000ed50 tx=0x7f7db00091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:45.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.190+0000 7f7da57fa700 1 -- 172.21.15.187:0/526443524 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db000cbb0 con 0x7f7db809ca90 2020-11-20T12:15:45.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.191+0000 7f7da57fa700 1 -- 172.21.15.187:0/526443524 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db000cd10 con 0x7f7db809ca90 2020-11-20T12:15:45.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.191+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db80b4830 con 0x7f7db809ca90 2020-11-20T12:15:45.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.191+0000 7f7da57fa700 1 -- 172.21.15.187:0/526443524 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0012620 con 0x7f7db809ca90 2020-11-20T12:15:45.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.191+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db80b2020 con 0x7f7db809ca90 2020-11-20T12:15:45.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.192+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db815c7f0 con 0x7f7db809ca90 2020-11-20T12:15:45.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.192+0000 7f7da57fa700 1 -- 172.21.15.187:0/526443524 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7db0022070 con 0x7f7db809ca90 2020-11-20T12:15:45.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.193+0000 7f7da57fa700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d90056d20 0x7f7d900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.193+0000 7f7da57fa700 1 -- 172.21.15.187:0/526443524 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7db0070690 con 0x7f7db809ca90 2020-11-20T12:15:45.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.193+0000 7f7dbd158700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d90056d20 0x7f7d900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.194+0000 7f7dbd158700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d90056d20 0x7f7d900591e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f7db4001e90 tx=0x7f7db4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:45.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.197+0000 7f7da57fa700 1 -- 172.21.15.187:0/526443524 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7db0045ae0 con 0x7f7db809ca90 2020-11-20T12:15:45.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.387+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f7db8061a20 con 0x7f7db809ca90 2020-11-20T12:15:45.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.397+0000 7f7da57fa700 1 -- 172.21.15.187:0/526443524 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7db0016020 con 0x7f7db809ca90 2020-11-20T12:15:45.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d90056d20 msgr2=0x7f7d900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d90056d20 0x7f7d900591e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f7db4001e90 tx=0x7f7db4009040).stop 2020-11-20T12:15:45.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7db809ca90 msgr2=0x7f7db8099650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7db809ca90 0x7f7db8099650 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7f7db000ed50 tx=0x7f7db00091d0).stop 2020-11-20T12:15:45.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 shutdown_connections 2020-11-20T12:15:45.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7d90056d20 0x7f7d900591e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7db809ca90 0x7f7db8099650 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7db8065190 0x7f7db80b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 --2- 172.21.15.187:0/526443524 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7db8065f90 0x7f7db8099160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 >> 172.21.15.187:0/526443524 conn(0x7f7db80bab30 msgr2=0x7f7db80afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:45.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 shutdown_connections 2020-11-20T12:15:45.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.400+0000 7f7dbe15a700 1 -- 172.21.15.187:0/526443524 wait complete. 2020-11-20T12:15:45.428 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:15:45.429 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:15:45.545 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:15:45.573 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:15:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:15:45.701 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:15:45.702 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:15:45.702 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:45.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.936+0000 7f5ab8cbb700 1 Processor -- start 2020-11-20T12:15:45.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.936+0000 7f5ab8cbb700 1 -- start start 2020-11-20T12:15:45.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.938+0000 7f5ab8cbb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5ab4066040 0x7f5ab4148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.938+0000 7f5ab8cbb700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab414c170 con 0x7f5ab40683f0 2020-11-20T12:15:45.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.938+0000 7f5ab8cbb700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab414c410 con 0x7f5ab4145cc0 2020-11-20T12:15:45.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.938+0000 7f5ab8cbb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab414c6b0 con 0x7f5ab4066040 2020-11-20T12:15:45.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.938+0000 7f5ab37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5ab4145cc0 0x7f5ab4065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46932/0 (socket says 172.21.15.187:46932) 2020-11-20T12:15:45.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.938+0000 7f5ab37fe700 1 -- 172.21.15.187:0/80358012 learned_addr learned my addr 172.21.15.187:0/80358012 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:45.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.939+0000 7f5ab3fff700 1 --2- 172.21.15.187:0/80358012 >> v2:172.21.15.5:3300/0 conn(0x7f5ab4066040 0x7f5ab4148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.939+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527285142 0 0) 0x7f5ab414c410 con 0x7f5ab4145cc0 2020-11-20T12:15:45.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.939+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a94003240 con 0x7f5ab4145cc0 2020-11-20T12:15:45.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.939+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873095864 0 0) 0x7f5ab414c170 con 0x7f5ab40683f0 2020-11-20T12:15:45.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.939+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab414c410 con 0x7f5ab40683f0 2020-11-20T12:15:45.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.939+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4258973761 0 0) 0x7f5a94003240 con 0x7f5ab4145cc0 2020-11-20T12:15:45.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 >> v2:172.21.15.5:3300/0 conn(0x7f5ab4066040 msgr2=0x7f5ab4148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:45.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 --2- 172.21.15.187:0/80358012 >> v2:172.21.15.5:3300/0 conn(0x7f5ab4066040 0x7f5ab4148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 >> v1:172.21.15.5:6789/0 conn(0x7f5ab40683f0 legacy=0x7f5ab41487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab40658a0 con 0x7f5ab4145cc0 2020-11-20T12:15:45.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5aa4001860 con 0x7f5ab4145cc0 2020-11-20T12:15:45.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 >> v1:172.21.15.187:6789/0 conn(0x7f5ab4145cc0 legacy=0x7f5ab4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a94003910 0x7f5a94003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a940044e0 0x7f5a94004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a94004e50 0x7f5a9400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab414c6b0 con 0x7f5a94003910 2020-11-20T12:15:45.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9400bc20 con 0x7f5a940044e0 2020-11-20T12:15:45.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9400bd60 con 0x7f5a94004e50 2020-11-20T12:15:45.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.940+0000 7f5aaaffd700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a940044e0 0x7f5a94004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5ab3fff700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a94004e50 0x7f5a9400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5aaaffd700 1 -- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a94004e50 msgr2=0x7f5a9400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:45.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5aaaffd700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a94004e50 0x7f5a9400b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5aaaffd700 1 -- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a94003910 msgr2=0x7f5a94003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:45.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5ab37fe700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a94003910 0x7f5a94003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5aaaffd700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a94003910 0x7f5a94003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5ab3fff700 1 -- 172.21.15.187:0/80358012 reap_dead start 2020-11-20T12:15:45.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5aaaffd700 1 -- 172.21.15.187:0/80358012 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a94003240 con 0x7f5a940044e0 2020-11-20T12:15:45.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5ab37fe700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a94003910 0x7f5a94003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:45.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5aaaffd700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a940044e0 0x7f5a94004960 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f5a9c001a80 tx=0x7f5a9c007b90).ready entity=mon.1 client_cookie=125ce3f6d7819ee3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:45.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.941+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c007020 con 0x7f5a940044e0 2020-11-20T12:15:45.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c007180 con 0x7f5a940044e0 2020-11-20T12:15:45.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab2ffd700 1 -- 172.21.15.187:0/80358012 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c00c920 con 0x7f5a940044e0 2020-11-20T12:15:45.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/80358012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a940044e0 msgr2=0x7f5a94004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab8cbb700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a940044e0 0x7f5a94004960 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f5a9c001a80 tx=0x7f5a9c007b90).stop 2020-11-20T12:15:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/80358012 shutdown_connections 2020-11-20T12:15:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab8cbb700 1 --2- 172.21.15.187:0/80358012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a940044e0 0x7f5a94004960 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/80358012 >> 172.21.15.187:0/80358012 conn(0x7f5ab40bab30 msgr2=0x7f5ab40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/80358012 shutdown_connections 2020-11-20T12:15:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.942+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/80358012 wait complete. 2020-11-20T12:15:45.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.944+0000 7f5ab8cbb700 1 Processor -- start 2020-11-20T12:15:45.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.944+0000 7f5ab8cbb700 1 -- start start 2020-11-20T12:15:45.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.945+0000 7f5ab8cbb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a940044e0 0x7f5ab41442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5ab8cbb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ab409e1b0 0x7f5ab40a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5ab8cbb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ab415b6c0 0x7f5ab40a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5ab8cbb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab40a0b60 con 0x7f5a940044e0 2020-11-20T12:15:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5ab8cbb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab409d650 con 0x7f5ab409e1b0 2020-11-20T12:15:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5ab8cbb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab415bc40 con 0x7f5ab415b6c0 2020-11-20T12:15:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5aaaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ab409e1b0 0x7f5ab40a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5aaaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ab409e1b0 0x7f5ab40a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50338/0 (socket says 172.21.15.187:50338) 2020-11-20T12:15:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5aaaffd700 1 -- 172.21.15.187:0/2226944208 learned_addr learned my addr 172.21.15.187:0/2226944208 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5aaaffd700 1 -- 172.21.15.187:0/2226944208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ab415b6c0 msgr2=0x7f5ab40a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5aaaffd700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ab415b6c0 0x7f5ab40a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5aaaffd700 1 -- 172.21.15.187:0/2226944208 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a940044e0 msgr2=0x7f5ab41442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5aaaffd700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a940044e0 0x7f5ab41442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.946+0000 7f5aaaffd700 1 -- 172.21.15.187:0/2226944208 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab415bee0 con 0x7f5ab409e1b0 2020-11-20T12:15:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.947+0000 7f5aaaffd700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ab409e1b0 0x7f5ab40a0180 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7f5a9c012b90 tx=0x7f5a9c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:45.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.947+0000 7f5aabfff700 1 -- 172.21.15.187:0/2226944208 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c001ea0 con 0x7f5ab409e1b0 2020-11-20T12:15:45.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.947+0000 7f5aabfff700 1 -- 172.21.15.187:0/2226944208 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c01c5b0 con 0x7f5ab409e1b0 2020-11-20T12:15:45.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.947+0000 7f5aabfff700 1 -- 172.21.15.187:0/2226944208 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c00c920 con 0x7f5ab409e1b0 2020-11-20T12:15:45.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.947+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab415c180 con 0x7f5ab409e1b0 2020-11-20T12:15:45.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.947+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab409fa60 con 0x7f5ab409e1b0 2020-11-20T12:15:45.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.948+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab415c310 con 0x7f5ab409e1b0 2020-11-20T12:15:45.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.949+0000 7f5aabfff700 1 -- 172.21.15.187:0/2226944208 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5a9c010070 con 0x7f5ab409e1b0 2020-11-20T12:15:45.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.949+0000 7f5aabfff700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a90056d50 0x7f5a90059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:45.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.949+0000 7f5aabfff700 1 -- 172.21.15.187:0/2226944208 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5a9c071410 con 0x7f5ab409e1b0 2020-11-20T12:15:45.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.950+0000 7f5ab37fe700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a90056d50 0x7f5a90059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:45.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.950+0000 7f5ab37fe700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a90056d50 0x7f5a90059210 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f5aa4001990 tx=0x7f5aa4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:45.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:45.953+0000 7f5aabfff700 1 -- 172.21.15.187:0/2226944208 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5a9c046860 con 0x7f5ab409e1b0 2020-11-20T12:15:46.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.196+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ab4061a20 con 0x7f5ab409e1b0 2020-11-20T12:15:46.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.197+0000 7f5aabfff700 1 -- 172.21.15.187:0/2226944208 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f5a9c049e80 con 0x7f5ab409e1b0 2020-11-20T12:15:46.201 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:46.202 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:46.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a90056d50 msgr2=0x7f5a90059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a90056d50 0x7f5a90059210 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f5aa4001990 tx=0x7f5aa4009040).stop 2020-11-20T12:15:46.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ab409e1b0 msgr2=0x7f5ab40a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ab409e1b0 0x7f5ab40a0180 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7f5a9c012b90 tx=0x7f5a9c016000).stop 2020-11-20T12:15:46.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 shutdown_connections 2020-11-20T12:15:46.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5a90056d50 0x7f5a90059210 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a940044e0 0x7f5ab41442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:46.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ab415b6c0 0x7f5ab40a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:46.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 --2- 172.21.15.187:0/2226944208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ab409e1b0 0x7f5ab40a0180 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 >> 172.21.15.187:0/2226944208 conn(0x7f5ab40bab30 msgr2=0x7f5ab40afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:46.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 shutdown_connections 2020-11-20T12:15:46.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.200+0000 7f5ab8cbb700 1 -- 172.21.15.187:0/2226944208 wait complete. 2020-11-20T12:15:46.207 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:46.228 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:15:46.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.467+0000 7ff1d3fff700 1 Processor -- start 2020-11-20T12:15:46.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.467+0000 7ff1d3fff700 1 -- start start 2020-11-20T12:15:46.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.468+0000 7ff1d3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40b6650 0x7ff1d40afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.468+0000 7ff1d3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d40b1300 con 0x7ff1d4066f70 2020-11-20T12:15:46.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.468+0000 7ff1d3fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d40b15a0 con 0x7ff1d41484d0 2020-11-20T12:15:46.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.468+0000 7ff1d3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d4067730 con 0x7ff1d40b6650 2020-11-20T12:15:46.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.469+0000 7ff1d2ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff1d41484d0 0x7ff1d4061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46950/0 (socket says 172.21.15.187:46950) 2020-11-20T12:15:46.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.469+0000 7ff1d2ffd700 1 -- 172.21.15.187:0/965529408 learned_addr learned my addr 172.21.15.187:0/965529408 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:46.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.469+0000 7ff1ca7fc700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40b6650 0x7ff1d40afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.469+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294790656 0 0) 0x7ff1d40b15a0 con 0x7ff1d41484d0 2020-11-20T12:15:46.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b4003240 con 0x7ff1d41484d0 2020-11-20T12:15:46.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 823145657 0 0) 0x7ff1b4003240 con 0x7ff1d41484d0 2020-11-20T12:15:46.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40b6650 msgr2=0x7ff1d40afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:46.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40b6650 0x7ff1d40afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 >> v1:172.21.15.5:6789/0 conn(0x7ff1d4066f70 legacy=0x7ff1d40aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d40b1840 con 0x7ff1d41484d0 2020-11-20T12:15:46.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff1c4001520 con 0x7ff1d41484d0 2020-11-20T12:15:46.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 >> v1:172.21.15.187:6789/0 conn(0x7ff1d41484d0 legacy=0x7ff1d4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b4003730 0x7ff1b4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b40044e0 0x7ff1b4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40b6650 0x7ff1b4004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d4067730 con 0x7ff1b40044e0 2020-11-20T12:15:46.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b4005390 con 0x7ff1b4003730 2020-11-20T12:15:46.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.470+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b4004340 con 0x7ff1d40b6650 2020-11-20T12:15:46.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.471+0000 7ff1d2ffd700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b4003730 0x7ff1b4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.471+0000 7ff1d37fe700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40b6650 0x7ff1b4004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.471+0000 7ff1ca7fc700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b40044e0 0x7ff1b4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.471+0000 7ff1d2ffd700 1 -- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40b6650 msgr2=0x7ff1b4004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:46.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.471+0000 7ff1d2ffd700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40b6650 0x7ff1b4004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.471+0000 7ff1d2ffd700 1 -- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b40044e0 msgr2=0x7ff1b4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:46.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.471+0000 7ff1d2ffd700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b40044e0 0x7ff1b4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.471+0000 7ff1d2ffd700 1 -- 172.21.15.187:0/965529408 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b4003240 con 0x7ff1b4003730 2020-11-20T12:15:46.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.472+0000 7ff1d2ffd700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b4003730 0x7ff1b4003b50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7ff1c400c210 tx=0x7ff1c4007750).ready entity=mon.1 client_cookie=e247105145e7e4a7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:46.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.472+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40070d0 con 0x7ff1b4003730 2020-11-20T12:15:46.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.472+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c4007230 con 0x7ff1b4003730 2020-11-20T12:15:46.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.472+0000 7ff1d27fc700 1 -- 172.21.15.187:0/965529408 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40055f0 con 0x7ff1b4003730 2020-11-20T12:15:46.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.473+0000 7ff1d3fff700 1 -- 172.21.15.187:0/965529408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b4003730 msgr2=0x7ff1b4003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.473+0000 7ff1d3fff700 1 --2- 172.21.15.187:0/965529408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b4003730 0x7ff1b4003b50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7ff1c400c210 tx=0x7ff1c4007750).stop 2020-11-20T12:15:46.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.473+0000 7ff1d37fe700 1 -- 172.21.15.187:0/965529408 reap_dead start 2020-11-20T12:15:46.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.473+0000 7ff1d3fff700 1 -- 172.21.15.187:0/965529408 shutdown_connections 2020-11-20T12:15:46.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.473+0000 7ff1d3fff700 1 -- 172.21.15.187:0/965529408 >> 172.21.15.187:0/965529408 conn(0x7ff1d40bab30 msgr2=0x7ff1d40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:46.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.473+0000 7ff1d3fff700 1 -- 172.21.15.187:0/965529408 shutdown_connections 2020-11-20T12:15:46.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.473+0000 7ff1d3fff700 1 -- 172.21.15.187:0/965529408 wait complete. 2020-11-20T12:15:46.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.475+0000 7ff1d3fff700 1 Processor -- start 2020-11-20T12:15:46.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.475+0000 7ff1d3fff700 1 -- start start 2020-11-20T12:15:46.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.476+0000 7ff1d3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40659d0 0x7ff1d4065db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.476+0000 7ff1d3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d409b830 0x7ff1d4130950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.476+0000 7ff1d3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1d4131b00 0x7ff1d4131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.476+0000 7ff1d3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d409d0d0 con 0x7ff1d4131b00 2020-11-20T12:15:46.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.476+0000 7ff1d3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d40b44b0 con 0x7ff1d409b830 2020-11-20T12:15:46.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.476+0000 7ff1d3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d409d2c0 con 0x7ff1d40659d0 2020-11-20T12:15:46.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d409b830 0x7ff1d4130950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d409b830 0x7ff1d4130950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50358/0 (socket says 172.21.15.187:50358) 2020-11-20T12:15:46.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 -- 172.21.15.187:0/1952941074 learned_addr learned my addr 172.21.15.187:0/1952941074 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:46.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1d37fe700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1d4131b00 0x7ff1d4131f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1d2ffd700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40659d0 0x7ff1d4065db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 -- 172.21.15.187:0/1952941074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40659d0 msgr2=0x7ff1d4065db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40659d0 0x7ff1d4065db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 -- 172.21.15.187:0/1952941074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1d4131b00 msgr2=0x7ff1d4131f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1d4131b00 0x7ff1d4131f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 -- 172.21.15.187:0/1952941074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d409d560 con 0x7ff1d409b830 2020-11-20T12:15:46.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.477+0000 7ff1ca7fc700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d409b830 0x7ff1d4130950 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7ff1bc010c20 tx=0x7ff1bc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:46.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.478+0000 7ff1cb7fe700 1 -- 172.21.15.187:0/1952941074 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc00b360 con 0x7ff1d409b830 2020-11-20T12:15:46.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.478+0000 7ff1cb7fe700 1 -- 172.21.15.187:0/1952941074 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc00ea30 con 0x7ff1d409b830 2020-11-20T12:15:46.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.478+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d41312c0 con 0x7ff1d409b830 2020-11-20T12:15:46.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.478+0000 7ff1cb7fe700 1 -- 172.21.15.187:0/1952941074 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc00edb0 con 0x7ff1d409b830 2020-11-20T12:15:46.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.478+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d4131760 con 0x7ff1d409b830 2020-11-20T12:15:46.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.479+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d4131450 con 0x7ff1d409b830 2020-11-20T12:15:46.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.480+0000 7ff1cb7fe700 1 -- 172.21.15.187:0/1952941074 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff1bc024070 con 0x7ff1d409b830 2020-11-20T12:15:46.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.481+0000 7ff1cb7fe700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1b0056d20 0x7ff1b00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.481+0000 7ff1cb7fe700 1 -- 172.21.15.187:0/1952941074 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff1bc070960 con 0x7ff1d409b830 2020-11-20T12:15:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.481+0000 7ff1d2ffd700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1b0056d20 0x7ff1b00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.482+0000 7ff1d2ffd700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1b0056d20 0x7ff1b00591e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7ff1d40641f0 tx=0x7ff1c4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.484+0000 7ff1cb7fe700 1 -- 172.21.15.187:0/1952941074 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff1bc045ca0 con 0x7ff1d409b830 2020-11-20T12:15:46.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.669+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1d4061a20 con 0x7ff1d409b830 2020-11-20T12:15:46.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.672+0000 7ff1cb7fe700 1 -- 172.21.15.187:0/1952941074 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16791 (secure 0 0 0) 0x7ff1bc018070 con 0x7ff1d409b830 2020-11-20T12:15:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:46.676 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:15:17.285828+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:15:12.843783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2020-11-20T12:15:14.270455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:15:46.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1b0056d20 msgr2=0x7ff1b00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1b0056d20 0x7ff1b00591e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7ff1d40641f0 tx=0x7ff1c4015000).stop 2020-11-20T12:15:46.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d409b830 msgr2=0x7ff1d4130950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d409b830 0x7ff1d4130950 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7ff1bc010c20 tx=0x7ff1bc008040).stop 2020-11-20T12:15:46.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 shutdown_connections 2020-11-20T12:15:46.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff1b0056d20 0x7ff1b00591e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1d4131b00 0x7ff1d4131f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1d40659d0 0x7ff1d4065db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 --2- 172.21.15.187:0/1952941074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d409b830 0x7ff1d4130950 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.674+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 >> 172.21.15.187:0/1952941074 conn(0x7ff1d40bab30 msgr2=0x7ff1d40b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:46.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.675+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 shutdown_connections 2020-11-20T12:15:46.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.675+0000 7ff1d3fff700 1 -- 172.21.15.187:0/1952941074 wait complete. 2020-11-20T12:15:46.699 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:15:12.843783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:15:14.270455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:15:46.700 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:15:46.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.941+0000 7f754955a700 1 Processor -- start 2020-11-20T12:15:46.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.941+0000 7f754955a700 1 -- start start 2020-11-20T12:15:46.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.942+0000 7f754955a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f7544065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f754955a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f754409ca90 0x7f754409cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f754955a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440b13e0 con 0x7f7544065f90 2020-11-20T12:15:46.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f754955a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754414ab30 con 0x7f754409ca90 2020-11-20T12:15:46.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f754955a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7544062c10 con 0x7f7544065190 2020-11-20T12:15:46.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f753b7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7544065f90 0x7f7544143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:46970/0 (socket says 172.21.15.187:46970) 2020-11-20T12:15:46.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f753b7fe700 1 -- 172.21.15.187:0/4140523970 learned_addr learned my addr 172.21.15.187:0/4140523970 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:46.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f7548d59700 1 --2- 172.21.15.187:0/4140523970 >> v2:172.21.15.187:3300/0 conn(0x7f754409ca90 0x7f754409cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f7543fff700 1 --2- 172.21.15.187:0/4140523970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f7544065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.943+0000 7f75437fe700 1 -- 172.21.15.187:0/4140523970 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001514994 0 0) 0x7f75440b13e0 con 0x7f7544065f90 2020-11-20T12:15:46.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.944+0000 7f75437fe700 1 -- 172.21.15.187:0/4140523970 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7524003240 con 0x7f7544065f90 2020-11-20T12:15:46.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.944+0000 7f7548d59700 1 -- 172.21.15.187:0/4140523970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 msgr2=0x7f7544065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:46.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.944+0000 7f7548d59700 1 --2- 172.21.15.187:0/4140523970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f7544065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.944+0000 7f7548d59700 1 -- 172.21.15.187:0/4140523970 >> v1:172.21.15.187:6789/0 conn(0x7f7544065f90 legacy=0x7f7544143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.944+0000 7f7548d59700 1 -- 172.21.15.187:0/4140523970 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440b1640 con 0x7f754409ca90 2020-11-20T12:15:46.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.944+0000 7f7548d59700 1 --2- 172.21.15.187:0/4140523970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f754409cf40 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f754414ac70 tx=0x7f753c006df0).ready entity=mon.1 client_cookie=656b907ddc09ad0d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:46.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.944+0000 7f75437fe700 1 -- 172.21.15.187:0/4140523970 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c0096d0 con 0x7f754409ca90 2020-11-20T12:15:46.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.945+0000 7f75437fe700 1 -- 172.21.15.187:0/4140523970 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c009830 con 0x7f754409ca90 2020-11-20T12:15:46.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.945+0000 7f75437fe700 1 -- 172.21.15.187:0/4140523970 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c009bb0 con 0x7f754409ca90 2020-11-20T12:15:46.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.945+0000 7f754955a700 1 -- 172.21.15.187:0/4140523970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 msgr2=0x7f754409cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.945+0000 7f754955a700 1 --2- 172.21.15.187:0/4140523970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f754409cf40 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f754414ac70 tx=0x7f753c006df0).stop 2020-11-20T12:15:46.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.945+0000 7f754955a700 1 -- 172.21.15.187:0/4140523970 shutdown_connections 2020-11-20T12:15:46.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.945+0000 7f754955a700 1 --2- 172.21.15.187:0/4140523970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f754409cf40 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.945+0000 7f754955a700 1 --2- 172.21.15.187:0/4140523970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f7544065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.945+0000 7f754955a700 1 -- 172.21.15.187:0/4140523970 >> 172.21.15.187:0/4140523970 conn(0x7f75440bab30 msgr2=0x7f75440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:46.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.946+0000 7f754955a700 1 -- 172.21.15.187:0/4140523970 shutdown_connections 2020-11-20T12:15:46.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.946+0000 7f754955a700 1 -- 172.21.15.187:0/4140523970 wait complete. 2020-11-20T12:15:46.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.947+0000 7f754955a700 1 Processor -- start 2020-11-20T12:15:46.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.947+0000 7f754955a700 1 -- start start 2020-11-20T12:15:46.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f754955a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f75440b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f754955a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7544065f90 0x7f75440b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f754955a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f7544142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f754955a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754414acc0 con 0x7f7544065f90 2020-11-20T12:15:46.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f754955a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754409dec0 con 0x7f754409ca90 2020-11-20T12:15:46.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f754955a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754414aef0 con 0x7f7544065190 2020-11-20T12:15:46.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f7548d59700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f7544142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f7548d59700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f7544142e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50374/0 (socket says 172.21.15.187:50374) 2020-11-20T12:15:46.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.948+0000 7f7548d59700 1 -- 172.21.15.187:0/731957415 learned_addr learned my addr 172.21.15.187:0/731957415 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:46.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7543fff700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f75440b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f753b7fe700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7544065f90 0x7f75440b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7548d59700 1 -- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 msgr2=0x7f75440b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7548d59700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f75440b7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7548d59700 1 -- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7544065f90 msgr2=0x7f75440b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:46.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7548d59700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7544065f90 0x7f75440b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:46.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7548d59700 1 -- 172.21.15.187:0/731957415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440b7a40 con 0x7f754409ca90 2020-11-20T12:15:46.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7543fff700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f75440b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:46.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7548d59700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f7544142e00 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7f75440b1570 tx=0x7f753c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:46.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7540ff9700 1 -- 172.21.15.187:0/731957415 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c021070 con 0x7f754409ca90 2020-11-20T12:15:46.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f7540ff9700 1 -- 172.21.15.187:0/731957415 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c011e70 con 0x7f754409ca90 2020-11-20T12:15:46.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.949+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7544149950 con 0x7f754409ca90 2020-11-20T12:15:46.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.950+0000 7f7540ff9700 1 -- 172.21.15.187:0/731957415 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c01c7d0 con 0x7f754409ca90 2020-11-20T12:15:46.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.950+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754409dca0 con 0x7f754409ca90 2020-11-20T12:15:46.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.951+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7544061a20 con 0x7f754409ca90 2020-11-20T12:15:46.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.951+0000 7f7540ff9700 1 -- 172.21.15.187:0/731957415 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f753c00f070 con 0x7f754409ca90 2020-11-20T12:15:46.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.952+0000 7f7540ff9700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7520056d50 0x7f7520059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:46.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.952+0000 7f7540ff9700 1 -- 172.21.15.187:0/731957415 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f753c071de0 con 0x7f754409ca90 2020-11-20T12:15:46.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.952+0000 7f7543fff700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7520056d50 0x7f7520059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:46.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.953+0000 7f7543fff700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7520056d50 0x7f7520059210 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f7534006f50 tx=0x7f7534008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:46.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:46.955+0000 7f7540ff9700 1 -- 172.21.15.187:0/731957415 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f753c047230 con 0x7f754409ca90 2020-11-20T12:15:47.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.140+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f754409e050 con 0x7f754409ca90 2020-11-20T12:15:47.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.141+0000 7f7540ff9700 1 -- 172.21.15.187:0/731957415 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v108) v1 ==== 110+0+10 (secure 0 0 0) 0x7f753c04a850 con 0x7f754409ca90 2020-11-20T12:15:47.145 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:15:47.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7520056d50 msgr2=0x7f7520059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7520056d50 0x7f7520059210 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f7534006f50 tx=0x7f7534008040).stop 2020-11-20T12:15:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 msgr2=0x7f7544142e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f7544142e00 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7f75440b1570 tx=0x7f753c015000).stop 2020-11-20T12:15:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 shutdown_connections 2020-11-20T12:15:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7520056d50 0x7f7520059210 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754409ca90 0x7f7544142e00 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7544065190 0x7f75440b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 --2- 172.21.15.187:0/731957415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7544065f90 0x7f75440b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.143+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 >> 172.21.15.187:0/731957415 conn(0x7f75440bab30 msgr2=0x7f75440ba070 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:47.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.144+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 shutdown_connections 2020-11-20T12:15:47.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.144+0000 7f754955a700 1 -- 172.21.15.187:0/731957415 wait complete. 2020-11-20T12:15:47.172 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:15:47.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.421+0000 7f3000833700 1 Processor -- start 2020-11-20T12:15:47.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.421+0000 7f3000833700 1 -- start start 2020-11-20T12:15:47.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.422+0000 7f3000833700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc1484d0 0x7f2ffc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.422+0000 7f3000833700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2ffc0622c0 0x7f2ffc0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.422+0000 7f3000833700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2ffc066f70 0x7f2ffc0b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.422+0000 7f3000833700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc067730 con 0x7f2ffc0622c0 2020-11-20T12:15:47.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.422+0000 7f3000833700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc0b1300 con 0x7f2ffc066f70 2020-11-20T12:15:47.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.422+0000 7f3000833700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc0b15a0 con 0x7f2ffc1484d0 2020-11-20T12:15:47.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.423+0000 7f2ffbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2ffc066f70 0x7f2ffc0b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.423+0000 7f2ffbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2ffc066f70 0x7f2ffc0b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50380/0 (socket says 172.21.15.187:50380) 2020-11-20T12:15:47.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.423+0000 7f2ffbfff700 1 -- 172.21.15.187:0/2666758759 learned_addr learned my addr 172.21.15.187:0/2666758759 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.423+0000 7f2ffb7fe700 1 --2- 172.21.15.187:0/2666758759 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc1484d0 0x7f2ffc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.423+0000 7f2ff2ffd700 1 --2- 172.21.15.187:0/2666758759 >> v2:172.21.15.5:3300/0 conn(0x7f2ffc0622c0 0x7f2ffc0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffbfff700 1 -- 172.21.15.187:0/2666758759 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc1484d0 msgr2=0x7f2ffc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffbfff700 1 --2- 172.21.15.187:0/2666758759 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc1484d0 0x7f2ffc061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffbfff700 1 -- 172.21.15.187:0/2666758759 >> v2:172.21.15.5:3300/0 conn(0x7f2ffc0622c0 msgr2=0x7f2ffc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:47.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffbfff700 1 --2- 172.21.15.187:0/2666758759 >> v2:172.21.15.5:3300/0 conn(0x7f2ffc0622c0 0x7f2ffc0b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffbfff700 1 -- 172.21.15.187:0/2666758759 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc0b1840 con 0x7f2ffc066f70 2020-11-20T12:15:47.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffbfff700 1 --2- 172.21.15.187:0/2666758759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc066f70 0x7f2ffc0b5b30 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f2ffc09d820 tx=0x7f2ff4009290).ready entity=mon.1 client_cookie=6e7428eede87dba1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:47.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffaffd700 1 -- 172.21.15.187:0/2666758759 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff400bc40 con 0x7f2ffc066f70 2020-11-20T12:15:47.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffaffd700 1 -- 172.21.15.187:0/2666758759 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff400bda0 con 0x7f2ffc066f70 2020-11-20T12:15:47.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.424+0000 7f2ffaffd700 1 -- 172.21.15.187:0/2666758759 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4011610 con 0x7f2ffc066f70 2020-11-20T12:15:47.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 -- 172.21.15.187:0/2666758759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc066f70 msgr2=0x7f2ffc0b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 --2- 172.21.15.187:0/2666758759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc066f70 0x7f2ffc0b5b30 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f2ffc09d820 tx=0x7f2ff4009290).stop 2020-11-20T12:15:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 -- 172.21.15.187:0/2666758759 shutdown_connections 2020-11-20T12:15:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 --2- 172.21.15.187:0/2666758759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc066f70 0x7f2ffc0b5b30 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 --2- 172.21.15.187:0/2666758759 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc1484d0 0x7f2ffc061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 --2- 172.21.15.187:0/2666758759 >> v2:172.21.15.5:3300/0 conn(0x7f2ffc0622c0 0x7f2ffc0b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 -- 172.21.15.187:0/2666758759 >> 172.21.15.187:0/2666758759 conn(0x7f2ffc0bab30 msgr2=0x7f2ffc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:47.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 -- 172.21.15.187:0/2666758759 shutdown_connections 2020-11-20T12:15:47.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.425+0000 7f3000833700 1 -- 172.21.15.187:0/2666758759 wait complete. 2020-11-20T12:15:47.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.427+0000 7f3000833700 1 Processor -- start 2020-11-20T12:15:47.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.427+0000 7f3000833700 1 -- start start 2020-11-20T12:15:47.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.428+0000 7f3000833700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc0622c0 0x7f2ffc0b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.428+0000 7f3000833700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc066f70 0x7f2ffc09e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.428+0000 7f3000833700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ffc1484d0 0x7f2ffc133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.428+0000 7f3000833700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc0b3ae0 con 0x7f2ffc1484d0 2020-11-20T12:15:47.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.428+0000 7f3000833700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc09ddf0 con 0x7f2ffc0622c0 2020-11-20T12:15:47.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.428+0000 7f3000833700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc09e090 con 0x7f2ffc066f70 2020-11-20T12:15:47.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.428+0000 7f2ffb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc0622c0 0x7f2ffc0b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ffb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc0622c0 0x7f2ffc0b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50384/0 (socket says 172.21.15.187:50384) 2020-11-20T12:15:47.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ffb7fe700 1 -- 172.21.15.187:0/2193406964 learned_addr learned my addr 172.21.15.187:0/2193406964 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:47.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ff2ffd700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc066f70 0x7f2ffc09e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ffb7fe700 1 -- 172.21.15.187:0/2193406964 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc066f70 msgr2=0x7f2ffc09e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ffb7fe700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc066f70 0x7f2ffc09e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ffb7fe700 1 -- 172.21.15.187:0/2193406964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ffc1484d0 msgr2=0x7f2ffc133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ffb7fe700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ffc1484d0 0x7f2ffc133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:47.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ffb7fe700 1 -- 172.21.15.187:0/2193406964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc09e330 con 0x7f2ffc0622c0 2020-11-20T12:15:47.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.429+0000 7f2ffb7fe700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc0622c0 0x7f2ffc0b35f0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7f2fec00d8e0 tx=0x7f2fec008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:47.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.430+0000 7f2ff3fff700 1 -- 172.21.15.187:0/2193406964 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec009790 con 0x7f2ffc0622c0 2020-11-20T12:15:47.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.430+0000 7f2ff3fff700 1 -- 172.21.15.187:0/2193406964 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec0098f0 con 0x7f2ffc0622c0 2020-11-20T12:15:47.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.430+0000 7f2ff3fff700 1 -- 172.21.15.187:0/2193406964 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec009c90 con 0x7f2ffc0622c0 2020-11-20T12:15:47.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.430+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc09dae0 con 0x7f2ffc0622c0 2020-11-20T12:15:47.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.430+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc09ecc0 con 0x7f2ffc0622c0 2020-11-20T12:15:47.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.431+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc09e4c0 con 0x7f2ffc0622c0 2020-11-20T12:15:47.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.432+0000 7f2ff3fff700 1 -- 172.21.15.187:0/2193406964 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2fec013480 con 0x7f2ffc0622c0 2020-11-20T12:15:47.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.433+0000 7f2ff3fff700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fd8056d20 0x7f2fd80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.433+0000 7f2ff3fff700 1 -- 172.21.15.187:0/2193406964 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2fec06feb0 con 0x7f2ffc0622c0 2020-11-20T12:15:47.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.433+0000 7f2ff2ffd700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fd8056d20 0x7f2fd80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.433+0000 7f2ff2ffd700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fd8056d20 0x7f2fd80591e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f2ffc0b4830 tx=0x7f2fe4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:47.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.437+0000 7f2ff3fff700 1 -- 172.21.15.187:0/2193406964 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2fec0452a0 con 0x7f2ffc0622c0 2020-11-20T12:15:47.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.629+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2ffc061a20 con 0x7f2ffc0622c0 2020-11-20T12:15:47.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.630+0000 7f2ff3fff700 1 -- 172.21.15.187:0/2193406964 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v108) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2fec0488c0 con 0x7f2ffc0622c0 2020-11-20T12:15:47.634 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:15:47.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fd8056d20 msgr2=0x7f2fd80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fd8056d20 0x7f2fd80591e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f2ffc0b4830 tx=0x7f2fe4008040).stop 2020-11-20T12:15:47.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc0622c0 msgr2=0x7f2ffc0b35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc0622c0 0x7f2ffc0b35f0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7f2fec00d8e0 tx=0x7f2fec008040).stop 2020-11-20T12:15:47.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 shutdown_connections 2020-11-20T12:15:47.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2fd8056d20 0x7f2fd80591e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ffc1484d0 0x7f2ffc133680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:47.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ffc0622c0 0x7f2ffc0b35f0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 --2- 172.21.15.187:0/2193406964 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ffc066f70 0x7f2ffc09e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 >> 172.21.15.187:0/2193406964 conn(0x7f2ffc0bab30 msgr2=0x7f2ffc0ba2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:47.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.632+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 shutdown_connections 2020-11-20T12:15:47.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.633+0000 7f3000833700 1 -- 172.21.15.187:0/2193406964 wait complete. 2020-11-20T12:15:47.660 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:15:47.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.899+0000 7f8a2359e700 1 Processor -- start 2020-11-20T12:15:47.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.900+0000 7f8a2359e700 1 -- start start 2020-11-20T12:15:47.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.901+0000 7f8a2359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8a240bf800 0x7f8a2414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.901+0000 7f8a2359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2414d110 con 0x7f8a2409f560 2020-11-20T12:15:47.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.901+0000 7f8a2359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2414cbd0 con 0x7f8a240bfbe0 2020-11-20T12:15:47.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.901+0000 7f8a2359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2414ce70 con 0x7f8a240bf800 2020-11-20T12:15:47.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.902+0000 7f8a19d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8a240bfbe0 0x7f8a241480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47000/0 (socket says 172.21.15.187:47000) 2020-11-20T12:15:47.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.902+0000 7f8a19d9b700 1 -- 172.21.15.187:0/772704072 learned_addr learned my addr 172.21.15.187:0/772704072 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:47.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.902+0000 7f8a2259c700 1 --2- 172.21.15.187:0/772704072 >> v2:172.21.15.187:3300/0 conn(0x7f8a240bf800 0x7f8a2414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.902+0000 7f8a2259c700 1 -- 172.21.15.187:0/772704072 >> v1:172.21.15.187:6789/0 conn(0x7f8a240bfbe0 legacy=0x7f8a241480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.902+0000 7f8a2259c700 1 -- 172.21.15.187:0/772704072 >> v1:172.21.15.5:6789/0 conn(0x7f8a2409f560 legacy=0x7f8a240a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:47.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.902+0000 7f8a2259c700 1 -- 172.21.15.187:0/772704072 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a240a4cc0 con 0x7f8a240bf800 2020-11-20T12:15:47.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.903+0000 7f8a2259c700 1 --2- 172.21.15.187:0/772704072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a240bf800 0x7f8a2414c2c0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7f8a1400b9a0 tx=0x7f8a14006df0).ready entity=mon.1 client_cookie=1b48d2d6a6f154ca server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:47.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.903+0000 7f8a21d9b700 1 -- 172.21.15.187:0/772704072 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14009800 con 0x7f8a240bf800 2020-11-20T12:15:47.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.903+0000 7f8a21d9b700 1 -- 172.21.15.187:0/772704072 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14009960 con 0x7f8a240bf800 2020-11-20T12:15:47.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.903+0000 7f8a21d9b700 1 -- 172.21.15.187:0/772704072 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14009ce0 con 0x7f8a240bf800 2020-11-20T12:15:47.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.904+0000 7f8a2359e700 1 -- 172.21.15.187:0/772704072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a240bf800 msgr2=0x7f8a2414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.904+0000 7f8a2359e700 1 --2- 172.21.15.187:0/772704072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a240bf800 0x7f8a2414c2c0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7f8a1400b9a0 tx=0x7f8a14006df0).stop 2020-11-20T12:15:47.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.904+0000 7f8a2359e700 1 -- 172.21.15.187:0/772704072 shutdown_connections 2020-11-20T12:15:47.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.904+0000 7f8a2359e700 1 --2- 172.21.15.187:0/772704072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a240bf800 0x7f8a2414c2c0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:47.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.904+0000 7f8a2359e700 1 -- 172.21.15.187:0/772704072 >> 172.21.15.187:0/772704072 conn(0x7f8a24068370 msgr2=0x7f8a240bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:47.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.904+0000 7f8a2359e700 1 -- 172.21.15.187:0/772704072 shutdown_connections 2020-11-20T12:15:47.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.904+0000 7f8a2359e700 1 -- 172.21.15.187:0/772704072 wait complete. 2020-11-20T12:15:47.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.905+0000 7f8a2359e700 1 Processor -- start 2020-11-20T12:15:47.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.906+0000 7f8a2359e700 1 -- start start 2020-11-20T12:15:47.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.906+0000 7f8a2359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a2409f560 0x7f8a240b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.906+0000 7f8a2359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a240bf800 0x7f8a24143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.906+0000 7f8a2359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a240ba300 0x7f8a24158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a241587a0 con 0x7f8a240bf800 2020-11-20T12:15:47.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240b4d50 con 0x7f8a2409f560 2020-11-20T12:15:47.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240ba880 con 0x7f8a240ba300 2020-11-20T12:15:47.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a2409f560 0x7f8a240b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a2409f560 0x7f8a240b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50398/0 (socket says 172.21.15.187:50398) 2020-11-20T12:15:47.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 -- 172.21.15.187:0/3187995863 learned_addr learned my addr 172.21.15.187:0/3187995863 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:47.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 -- 172.21.15.187:0/3187995863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a240ba300 msgr2=0x7f8a24158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a240ba300 0x7f8a24158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:47.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 -- 172.21.15.187:0/3187995863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a240bf800 msgr2=0x7f8a24143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:47.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a240bf800 0x7f8a24143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:47.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 -- 172.21.15.187:0/3187995863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a240bab20 con 0x7f8a2409f560 2020-11-20T12:15:47.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.907+0000 7f8a2259c700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a2409f560 0x7f8a240b4860 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7f8a240bca50 tx=0x7f8a14015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:47.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.908+0000 7f8a1affd700 1 -- 172.21.15.187:0/3187995863 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1401d6a0 con 0x7f8a2409f560 2020-11-20T12:15:47.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.908+0000 7f8a1affd700 1 -- 172.21.15.187:0/3187995863 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1401d800 con 0x7f8a2409f560 2020-11-20T12:15:47.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.908+0000 7f8a1affd700 1 -- 172.21.15.187:0/3187995863 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1401dba0 con 0x7f8a2409f560 2020-11-20T12:15:47.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.908+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a24143780 con 0x7f8a2409f560 2020-11-20T12:15:47.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.908+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a241592c0 con 0x7f8a2409f560 2020-11-20T12:15:47.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.909+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24080050 con 0x7f8a2409f560 2020-11-20T12:15:47.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.909+0000 7f8a1affd700 1 -- 172.21.15.187:0/3187995863 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8a14025030 con 0x7f8a2409f560 2020-11-20T12:15:47.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.910+0000 7f8a1affd700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a00056d20 0x7f8a000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:47.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.910+0000 7f8a1affd700 1 -- 172.21.15.187:0/3187995863 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8a14071970 con 0x7f8a2409f560 2020-11-20T12:15:47.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.910+0000 7f8a19d9b700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a00056d20 0x7f8a000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:47.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.911+0000 7f8a19d9b700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a00056d20 0x7f8a000591e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f8a0c000bc0 tx=0x7f8a0c007880).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:47.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:47.914+0000 7f8a1affd700 1 -- 172.21.15.187:0/3187995863 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8a14046e10 con 0x7f8a2409f560 2020-11-20T12:15:48.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.116+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a2414c7b0 con 0x7f8a2409f560 2020-11-20T12:15:48.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.117+0000 7f8a1affd700 1 -- 172.21.15.187:0/3187995863 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8a14023020 con 0x7f8a2409f560 2020-11-20T12:15:48.121 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:15:48.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a00056d20 msgr2=0x7f8a000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:48.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a00056d20 0x7f8a000591e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f8a0c000bc0 tx=0x7f8a0c007880).stop 2020-11-20T12:15:48.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a2409f560 msgr2=0x7f8a240b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:48.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a2409f560 0x7f8a240b4860 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7f8a240bca50 tx=0x7f8a14015000).stop 2020-11-20T12:15:48.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 shutdown_connections 2020-11-20T12:15:48.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a00056d20 0x7f8a000591e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:48.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a240ba300 0x7f8a24158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:48.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a2409f560 0x7f8a240b4860 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:48.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 --2- 172.21.15.187:0/3187995863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a240bf800 0x7f8a24143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:48.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.119+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 >> 172.21.15.187:0/3187995863 conn(0x7f8a24068370 msgr2=0x7f8a240a4080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:48.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.120+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 shutdown_connections 2020-11-20T12:15:48.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.120+0000 7f8a2359e700 1 -- 172.21.15.187:0/3187995863 wait complete. 2020-11-20T12:15:48.146 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:15:48.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.397+0000 7f18b759e700 1 Processor -- start 2020-11-20T12:15:49.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.397+0000 7f18b759e700 1 -- start start 2020-11-20T12:15:49.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.398+0000 7f18b759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f18b8068230 0x7f18b81482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.398+0000 7f18b759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80b5f50 0x7f18b80b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.398+0000 7f18b759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b8144b80 con 0x7f18b80b57b0 2020-11-20T12:15:49.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.398+0000 7f18b759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b81487c0 con 0x7f18b8068230 2020-11-20T12:15:49.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.398+0000 7f18b759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b81489e0 con 0x7f18b80b5f50 2020-11-20T12:15:49.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.399+0000 7f18b659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f18b80b57b0 0x7f18b8062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47012/0 (socket says 172.21.15.187:47012) 2020-11-20T12:15:49.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.399+0000 7f18b659c700 1 -- 172.21.15.187:0/3902624051 learned_addr learned my addr 172.21.15.187:0/3902624051 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:49.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.399+0000 7f18b6d9d700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80b5f50 0x7f18b80b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.399+0000 7f18add9b700 1 --2- 172.21.15.187:0/3902624051 >> v2:172.21.15.5:3300/0 conn(0x7f18b8068230 0x7f18b81482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.399+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408150860 0 0) 0x7f18b8144b80 con 0x7f18b80b57b0 2020-11-20T12:15:49.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.399+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898003240 con 0x7f18b80b57b0 2020-11-20T12:15:49.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 434348611 0 0) 0x7f1898003240 con 0x7f18b80b57b0 2020-11-20T12:15:49.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80b5f50 msgr2=0x7f18b80b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80b5f50 0x7f18b80b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 >> v2:172.21.15.5:3300/0 conn(0x7f18b8068230 msgr2=0x7f18b81482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 --2- 172.21.15.187:0/3902624051 >> v2:172.21.15.5:3300/0 conn(0x7f18b8068230 0x7f18b81482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b8148c80 con 0x7f18b80b57b0 2020-11-20T12:15:49.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f18a80014e0 con 0x7f18b80b57b0 2020-11-20T12:15:49.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 >> v1:172.21.15.187:6789/0 conn(0x7f18b80b57b0 legacy=0x7f18b8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1898003730 0x7f1898003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80b5f50 0x7f18980044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1898004040 0x7f18980049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b81487c0 con 0x7f1898004040 2020-11-20T12:15:49.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b81489e0 con 0x7f1898003730 2020-11-20T12:15:49.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.400+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1898005010 con 0x7f18b80b5f50 2020-11-20T12:15:49.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.401+0000 7f18b659c700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1898003730 0x7f1898003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.401+0000 7f18b659c700 1 -- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80b5f50 msgr2=0x7f18980044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.401+0000 7f18b659c700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80b5f50 0x7f18980044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:49.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.401+0000 7f18b659c700 1 -- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1898004040 msgr2=0x7f18980049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.401+0000 7f18b659c700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1898004040 0x7f18980049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:49.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.401+0000 7f18b659c700 1 -- 172.21.15.187:0/3902624051 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1898003240 con 0x7f1898003730 2020-11-20T12:15:49.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.401+0000 7f18b659c700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1898003730 0x7f1898003b50 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7f18a800c440 tx=0x7f18a8007800).ready entity=mon.1 client_cookie=93018dd04a07121b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.402+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a80072e0 con 0x7f1898003730 2020-11-20T12:15:49.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.402+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a8007440 con 0x7f1898003730 2020-11-20T12:15:49.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.402+0000 7f18b5d9b700 1 -- 172.21.15.187:0/3902624051 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8005820 con 0x7f1898003730 2020-11-20T12:15:49.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.402+0000 7f18b759e700 1 -- 172.21.15.187:0/3902624051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1898003730 msgr2=0x7f1898003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.402+0000 7f18b759e700 1 --2- 172.21.15.187:0/3902624051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1898003730 0x7f1898003b50 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7f18a800c440 tx=0x7f18a8007800).stop 2020-11-20T12:15:49.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.402+0000 7f18b6d9d700 1 -- 172.21.15.187:0/3902624051 reap_dead start 2020-11-20T12:15:49.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.402+0000 7f18b759e700 1 -- 172.21.15.187:0/3902624051 shutdown_connections 2020-11-20T12:15:49.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.402+0000 7f18b759e700 1 -- 172.21.15.187:0/3902624051 >> 172.21.15.187:0/3902624051 conn(0x7f18b80bab30 msgr2=0x7f18b80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:49.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.403+0000 7f18b759e700 1 -- 172.21.15.187:0/3902624051 shutdown_connections 2020-11-20T12:15:49.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.403+0000 7f18b759e700 1 -- 172.21.15.187:0/3902624051 wait complete. 2020-11-20T12:15:49.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.405+0000 7f18b759e700 1 Processor -- start 2020-11-20T12:15:49.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.405+0000 7f18b759e700 1 -- start start 2020-11-20T12:15:49.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.406+0000 7f18b759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18b809ff80 0x7f18b80b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.406+0000 7f18b759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80a0360 0x7f18b81460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.406+0000 7f18b759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18b80b49a0 0x7f18b8146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.406+0000 7f18b759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8145550 con 0x7f18b809ff80 2020-11-20T12:15:49.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.406+0000 7f18b759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b81485e0 con 0x7f18b80b49a0 2020-11-20T12:15:49.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.406+0000 7f18b759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b80b2100 con 0x7f18b80a0360 2020-11-20T12:15:49.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.406+0000 7f18b6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18b80b49a0 0x7f18b8146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18b80b49a0 0x7f18b8146590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50422/0 (socket says 172.21.15.187:50422) 2020-11-20T12:15:49.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b6d9d700 1 -- 172.21.15.187:0/847562230 learned_addr learned my addr 172.21.15.187:0/847562230 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:49.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18add9b700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80a0360 0x7f18b81460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b659c700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18b809ff80 0x7f18b80b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b6d9d700 1 -- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80a0360 msgr2=0x7f18b81460a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b6d9d700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80a0360 0x7f18b81460a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b6d9d700 1 -- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18b809ff80 msgr2=0x7f18b80b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b6d9d700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18b809ff80 0x7f18b80b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b6d9d700 1 -- 172.21.15.187:0/847562230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b8144e10 con 0x7f18b80b49a0 2020-11-20T12:15:49.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18add9b700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80a0360 0x7f18b81460a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:49.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18b6d9d700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18b80b49a0 0x7f18b8146590 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f18b8064070 tx=0x7f18b00091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.407+0000 7f18aeffd700 1 -- 172.21.15.187:0/847562230 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b000c820 con 0x7f18b80b49a0 2020-11-20T12:15:49.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.408+0000 7f18aeffd700 1 -- 172.21.15.187:0/847562230 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b000c980 con 0x7f18b80b49a0 2020-11-20T12:15:49.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.408+0000 7f18aeffd700 1 -- 172.21.15.187:0/847562230 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b000cd20 con 0x7f18b80b49a0 2020-11-20T12:15:49.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.408+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b81450b0 con 0x7f18b80b49a0 2020-11-20T12:15:49.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.408+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b80b1da0 con 0x7f18b80b49a0 2020-11-20T12:15:49.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.409+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b8145240 con 0x7f18b80b49a0 2020-11-20T12:15:49.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.409+0000 7f18aeffd700 1 -- 172.21.15.187:0/847562230 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f18b0012600 con 0x7f18b80b49a0 2020-11-20T12:15:49.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.410+0000 7f18aeffd700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1894056d20 0x7f18940591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.410+0000 7f18aeffd700 1 -- 172.21.15.187:0/847562230 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f18b0070470 con 0x7f18b80b49a0 2020-11-20T12:15:49.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.410+0000 7f18b659c700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1894056d20 0x7f18940591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.411+0000 7f18b659c700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1894056d20 0x7f18940591e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f18b805e630 tx=0x7f18a8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.414+0000 7f18aeffd700 1 -- 172.21.15.187:0/847562230 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f18b0045860 con 0x7f18b80b49a0 2020-11-20T12:15:49.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.599+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18b8061a20 con 0x7f18b80b49a0 2020-11-20T12:15:49.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.600+0000 7f18aeffd700 1 -- 172.21.15.187:0/847562230 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+10 (secure 0 0 0) 0x7f18b0048e80 con 0x7f18b80b49a0 2020-11-20T12:15:49.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1894056d20 msgr2=0x7f18940591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1894056d20 0x7f18940591e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f18b805e630 tx=0x7f18a8015000).stop 2020-11-20T12:15:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18b80b49a0 msgr2=0x7f18b8146590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18b80b49a0 0x7f18b8146590 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f18b8064070 tx=0x7f18b00091d0).stop 2020-11-20T12:15:49.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 shutdown_connections 2020-11-20T12:15:49.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1894056d20 0x7f18940591e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18b809ff80 0x7f18b80b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18b80b49a0 0x7f18b8146590 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 --2- 172.21.15.187:0/847562230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f18b80a0360 0x7f18b81460a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 >> 172.21.15.187:0/847562230 conn(0x7f18b80bab30 msgr2=0x7f18b80b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:49.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 shutdown_connections 2020-11-20T12:15:49.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:48.603+0000 7f18b759e700 1 -- 172.21.15.187:0/847562230 wait complete. 2020-11-20T12:15:49.105 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:15:49.106 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:49.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.357+0000 7f0869da2700 1 Processor -- start 2020-11-20T12:15:49.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.357+0000 7f0869da2700 1 -- start start 2020-11-20T12:15:49.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.358+0000 7f0869da2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f08640b6650 0x7f08640afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.358+0000 7f0869da2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0864066f70 0x7f08640aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.358+0000 7f0869da2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08640b1300 con 0x7f08641484d0 2020-11-20T12:15:49.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.359+0000 7f0869da2700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0864067730 con 0x7f08640b6650 2020-11-20T12:15:49.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.359+0000 7f0869da2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08640b15a0 con 0x7f0864066f70 2020-11-20T12:15:49.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.359+0000 7f0868da0700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f08641484d0 0x7f0864061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47032/0 (socket says 172.21.15.187:47032) 2020-11-20T12:15:49.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.359+0000 7f0868da0700 1 -- 172.21.15.187:0/3247599752 learned_addr learned my addr 172.21.15.187:0/3247599752 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:49.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.359+0000 7f0863fff700 1 --2- 172.21.15.187:0/3247599752 >> v2:172.21.15.5:3300/0 conn(0x7f08640b6650 0x7f08640afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.359+0000 7f08695a1700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0864066f70 0x7f08640aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.359+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811023067 0 0) 0x7f08640b1300 con 0x7f08641484d0 2020-11-20T12:15:49.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0848003240 con 0x7f08641484d0 2020-11-20T12:15:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1904621563 0 0) 0x7f0848003240 con 0x7f08641484d0 2020-11-20T12:15:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0864066f70 msgr2=0x7f08640aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0864066f70 0x7f08640aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 >> v2:172.21.15.5:3300/0 conn(0x7f08640b6650 msgr2=0x7f08640afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 --2- 172.21.15.187:0/3247599752 >> v2:172.21.15.5:3300/0 conn(0x7f08640b6650 0x7f08640afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08640b1840 con 0x7f08641484d0 2020-11-20T12:15:49.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f08580014e0 con 0x7f08641484d0 2020-11-20T12:15:49.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 >> v1:172.21.15.187:6789/0 conn(0x7f08641484d0 legacy=0x7f0864061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0864066f70 0x7f0848003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0848003c40 0x7f08480044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08480049d0 0x7f0848004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0864067730 con 0x7f08480049d0 2020-11-20T12:15:49.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08640b15a0 con 0x7f0848003c40 2020-11-20T12:15:49.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.360+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0848004200 con 0x7f0864066f70 2020-11-20T12:15:49.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.361+0000 7f0863fff700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0848003c40 0x7f08480044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.361+0000 7f08695a1700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08480049d0 0x7f0848004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.361+0000 7f0868da0700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0864066f70 0x7f0848003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.361+0000 7f0863fff700 1 -- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0864066f70 msgr2=0x7f0848003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.361+0000 7f0863fff700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0864066f70 0x7f0848003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.361+0000 7f0863fff700 1 -- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08480049d0 msgr2=0x7f0848004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.361+0000 7f0863fff700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08480049d0 0x7f0848004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.361+0000 7f0863fff700 1 -- 172.21.15.187:0/3247599752 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0848003240 con 0x7f0848003c40 2020-11-20T12:15:49.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.362+0000 7f0863fff700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0848003c40 0x7f08480044e0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f085000bc00 tx=0x7f0850006df0).ready entity=mon.1 client_cookie=9607b2ca4eb9b379 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.362+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085000b310 con 0x7f0848003c40 2020-11-20T12:15:49.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.362+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085000eb40 con 0x7f0848003c40 2020-11-20T12:15:49.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.362+0000 7f08637fe700 1 -- 172.21.15.187:0/3247599752 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085000eec0 con 0x7f0848003c40 2020-11-20T12:15:49.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.362+0000 7f0869da2700 1 -- 172.21.15.187:0/3247599752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0848003c40 msgr2=0x7f08480044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.362+0000 7f0869da2700 1 --2- 172.21.15.187:0/3247599752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0848003c40 0x7f08480044e0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f085000bc00 tx=0x7f0850006df0).stop 2020-11-20T12:15:49.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.363+0000 7f08695a1700 1 -- 172.21.15.187:0/3247599752 reap_dead start 2020-11-20T12:15:49.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.363+0000 7f0869da2700 1 -- 172.21.15.187:0/3247599752 shutdown_connections 2020-11-20T12:15:49.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.363+0000 7f0869da2700 1 -- 172.21.15.187:0/3247599752 >> 172.21.15.187:0/3247599752 conn(0x7f08640bab30 msgr2=0x7f08640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:49.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.363+0000 7f0869da2700 1 -- 172.21.15.187:0/3247599752 shutdown_connections 2020-11-20T12:15:49.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.363+0000 7f0869da2700 1 -- 172.21.15.187:0/3247599752 wait complete. 2020-11-20T12:15:49.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.364+0000 7f0869da2700 1 Processor -- start 2020-11-20T12:15:49.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.364+0000 7f0869da2700 1 -- start start 2020-11-20T12:15:49.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.365+0000 7f0869da2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08641470b0 0x7f08640b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.365+0000 7f0869da2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0864147490 0x7f08640b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.365+0000 7f0869da2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f086409e4b0 0x7f0864136b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.365+0000 7f0869da2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08640b4080 con 0x7f08641470b0 2020-11-20T12:15:49.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.365+0000 7f0869da2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08640b42a0 con 0x7f0864147490 2020-11-20T12:15:49.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.365+0000 7f0869da2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08640b4540 con 0x7f086409e4b0 2020-11-20T12:15:49.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0863fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0864147490 0x7f08640b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0863fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0864147490 0x7f08640b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50442/0 (socket says 172.21.15.187:50442) 2020-11-20T12:15:49.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0863fff700 1 -- 172.21.15.187:0/2929415145 learned_addr learned my addr 172.21.15.187:0/2929415145 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:49.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f08695a1700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f086409e4b0 0x7f0864136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0868da0700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08641470b0 0x7f08640b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0863fff700 1 -- 172.21.15.187:0/2929415145 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f086409e4b0 msgr2=0x7f0864136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0863fff700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f086409e4b0 0x7f0864136b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0863fff700 1 -- 172.21.15.187:0/2929415145 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08641470b0 msgr2=0x7f08640b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0863fff700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08641470b0 0x7f08640b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.366+0000 7f0863fff700 1 -- 172.21.15.187:0/2929415145 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08640b47e0 con 0x7f0864147490 2020-11-20T12:15:49.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.367+0000 7f0863fff700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0864147490 0x7f08640b3b90 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7f0850006fd0 tx=0x7f0850019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.367+0000 7f0860ff9700 1 -- 172.21.15.187:0/2929415145 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085001d620 con 0x7f0864147490 2020-11-20T12:15:49.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.367+0000 7f0860ff9700 1 -- 172.21.15.187:0/2929415145 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0850021070 con 0x7f0864147490 2020-11-20T12:15:49.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.367+0000 7f0860ff9700 1 -- 172.21.15.187:0/2929415145 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085001d7e0 con 0x7f0864147490 2020-11-20T12:15:49.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.367+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0864137000 con 0x7f0864147490 2020-11-20T12:15:49.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.367+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0864137520 con 0x7f0864147490 2020-11-20T12:15:49.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.368+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864061a20 con 0x7f0864147490 2020-11-20T12:15:49.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.369+0000 7f0860ff9700 1 -- 172.21.15.187:0/2929415145 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f085001dc10 con 0x7f0864147490 2020-11-20T12:15:49.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.369+0000 7f0860ff9700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f083c056d20 0x7f083c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.369+0000 7f0860ff9700 1 -- 172.21.15.187:0/2929415145 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0850071e20 con 0x7f0864147490 2020-11-20T12:15:49.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.370+0000 7f0868da0700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f083c056d20 0x7f083c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.370+0000 7f0868da0700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f083c056d20 0x7f083c0591e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f0858000f20 tx=0x7f0858005bb0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.374+0000 7f0860ff9700 1 -- 172.21.15.187:0/2929415145 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0850047270 con 0x7f0864147490 2020-11-20T12:15:49.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.622+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0864137520 con 0x7f0864147490 2020-11-20T12:15:49.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.622+0000 7f0860ff9700 1 -- 172.21.15.187:0/2929415145 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f0864137520 con 0x7f0864147490 2020-11-20T12:15:49.628 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:49.628 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":95,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2915858185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2915858185},{"type":"v1","addr":"172.21.15.5:6833","nonce":2915858185}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:15:15.259490+0000","modified":"2020-11-20T12:15:17.285078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8125":{"gid":8125,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/2812802720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2812802720},{"type":"v1","addr":"172.21.15.187:6835","nonce":2812802720}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2020-11-20T12:15:49.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.625+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f083c056d20 msgr2=0x7f083c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.625+0000 7f0869da2700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f083c056d20 0x7f083c0591e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f0858000f20 tx=0x7f0858005bb0).stop 2020-11-20T12:15:49.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.625+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0864147490 msgr2=0x7f08640b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.625+0000 7f0869da2700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0864147490 0x7f08640b3b90 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7f0850006fd0 tx=0x7f0850019000).stop 2020-11-20T12:15:49.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.626+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 shutdown_connections 2020-11-20T12:15:49.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.626+0000 7f0869da2700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f083c056d20 0x7f083c0591e0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.626+0000 7f0869da2700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08641470b0 0x7f08640b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.626+0000 7f0869da2700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f086409e4b0 0x7f0864136b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.626+0000 7f0869da2700 1 --2- 172.21.15.187:0/2929415145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0864147490 0x7f08640b3b90 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.626+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 >> 172.21.15.187:0/2929415145 conn(0x7f08640bab30 msgr2=0x7f08640afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:49.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.626+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 shutdown_connections 2020-11-20T12:15:49.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.626+0000 7f0869da2700 1 -- 172.21.15.187:0/2929415145 wait complete. 2020-11-20T12:15:49.637 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 95 2020-11-20T12:15:49.652 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:15:49.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.905+0000 7f8a68b6f700 1 Processor -- start 2020-11-20T12:15:49.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.905+0000 7f8a68b6f700 1 -- start start 2020-11-20T12:15:49.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.907+0000 7f8a68b6f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8a640bfbe0 0x7f8a641480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.907+0000 7f8a68b6f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6414d110 con 0x7f8a6409f560 2020-11-20T12:15:49.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.907+0000 7f8a68b6f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6414cbd0 con 0x7f8a640bf800 2020-11-20T12:15:49.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.907+0000 7f8a68b6f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6414c9d0 con 0x7f8a640bfbe0 2020-11-20T12:15:49.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.907+0000 7f8a5affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8a640bfbe0 0x7f8a641480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.907+0000 7f8a637fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8a640bf800 0x7f8a6414c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47052/0 (socket says 172.21.15.187:47052) 2020-11-20T12:15:49.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.907+0000 7f8a637fe700 1 -- 172.21.15.187:0/2760996950 learned_addr learned my addr 172.21.15.187:0/2760996950 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:49.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.908+0000 7f8a5affd700 1 -- 172.21.15.187:0/2760996950 >> v1:172.21.15.187:6789/0 conn(0x7f8a640bf800 legacy=0x7f8a6414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.908+0000 7f8a5affd700 1 -- 172.21.15.187:0/2760996950 >> v1:172.21.15.5:6789/0 conn(0x7f8a6409f560 legacy=0x7f8a640a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:49.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.908+0000 7f8a62ffd700 1 -- 172.21.15.187:0/2760996950 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288724994 0 0) 0x7f8a6414cbd0 con 0x7f8a640bf800 2020-11-20T12:15:49.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.908+0000 7f8a5affd700 1 -- 172.21.15.187:0/2760996950 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6414ce70 con 0x7f8a640bfbe0 2020-11-20T12:15:49.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.908+0000 7f8a5affd700 1 --2- 172.21.15.187:0/2760996950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bfbe0 0x7f8a641480c0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f8a4c00b9a0 tx=0x7f8a4c006df0).ready entity=mon.1 client_cookie=b828ad1f8c5bf3b2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.908+0000 7f8a62ffd700 1 -- 172.21.15.187:0/2760996950 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c009800 con 0x7f8a640bfbe0 2020-11-20T12:15:49.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.908+0000 7f8a62ffd700 1 -- 172.21.15.187:0/2760996950 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c009960 con 0x7f8a640bfbe0 2020-11-20T12:15:49.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.908+0000 7f8a62ffd700 1 -- 172.21.15.187:0/2760996950 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c009ce0 con 0x7f8a640bfbe0 2020-11-20T12:15:49.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.909+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2760996950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bfbe0 msgr2=0x7f8a641480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.909+0000 7f8a68b6f700 1 --2- 172.21.15.187:0/2760996950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bfbe0 0x7f8a641480c0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f8a4c00b9a0 tx=0x7f8a4c006df0).stop 2020-11-20T12:15:49.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.909+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2760996950 shutdown_connections 2020-11-20T12:15:49.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.909+0000 7f8a68b6f700 1 --2- 172.21.15.187:0/2760996950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bfbe0 0x7f8a641480c0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.909+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2760996950 >> 172.21.15.187:0/2760996950 conn(0x7f8a64068370 msgr2=0x7f8a640bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:49.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.909+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2760996950 shutdown_connections 2020-11-20T12:15:49.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.909+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2760996950 wait complete. 2020-11-20T12:15:49.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.911+0000 7f8a68b6f700 1 Processor -- start 2020-11-20T12:15:49.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.911+0000 7f8a68b6f700 1 -- start start 2020-11-20T12:15:49.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a68b6f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a6409f560 0x7f8a64145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a68b6f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bf800 0x7f8a64142c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a68b6f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a64146c90 0x7f8a64134960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a68b6f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a64146550 con 0x7f8a64146c90 2020-11-20T12:15:49.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a68b6f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a641467f0 con 0x7f8a640bf800 2020-11-20T12:15:49.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a68b6f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a64143120 con 0x7f8a6409f560 2020-11-20T12:15:49.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a5affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bf800 0x7f8a64142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a5affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bf800 0x7f8a64142c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50456/0 (socket says 172.21.15.187:50456) 2020-11-20T12:15:49.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.912+0000 7f8a5affd700 1 -- 172.21.15.187:0/2635505197 learned_addr learned my addr 172.21.15.187:0/2635505197 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:49.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a637fe700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a6409f560 0x7f8a64145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a63fff700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a64146c90 0x7f8a64134960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5affd700 1 -- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a6409f560 msgr2=0x7f8a64145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5affd700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a6409f560 0x7f8a64145ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5affd700 1 -- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a64146c90 msgr2=0x7f8a64134960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:49.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5affd700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a64146c90 0x7f8a64134960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:49.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5affd700 1 -- 172.21.15.187:0/2635505197 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a64147640 con 0x7f8a640bf800 2020-11-20T12:15:49.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a63fff700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a64146c90 0x7f8a64134960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:49.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5affd700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bf800 0x7f8a64142c30 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7f8a4c00b150 tx=0x7f8a4c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5bfff700 1 -- 172.21.15.187:0/2635505197 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c01d6a0 con 0x7f8a640bf800 2020-11-20T12:15:49.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5bfff700 1 -- 172.21.15.187:0/2635505197 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c01d800 con 0x7f8a640bf800 2020-11-20T12:15:49.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a5bfff700 1 -- 172.21.15.187:0/2635505197 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c01db80 con 0x7f8a640bf800 2020-11-20T12:15:49.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a64145b00 con 0x7f8a640bf800 2020-11-20T12:15:49.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.913+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a641472b0 con 0x7f8a640bf800 2020-11-20T12:15:49.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.915+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a64080050 con 0x7f8a640bf800 2020-11-20T12:15:49.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.915+0000 7f8a5bfff700 1 -- 172.21.15.187:0/2635505197 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8a4c025030 con 0x7f8a640bf800 2020-11-20T12:15:49.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.916+0000 7f8a5bfff700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a40056d50 0x7f8a40059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:49.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.916+0000 7f8a5bfff700 1 -- 172.21.15.187:0/2635505197 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8a4c0718a0 con 0x7f8a640bf800 2020-11-20T12:15:49.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.916+0000 7f8a637fe700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a40056d50 0x7f8a40059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:49.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.917+0000 7f8a637fe700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a40056d50 0x7f8a40059210 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f8a54000bc0 tx=0x7f8a54009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:49.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:49.920+0000 7f8a5bfff700 1 -- 172.21.15.187:0/2635505197 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8a4c046da0 con 0x7f8a640bf800 2020-11-20T12:15:50.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.106+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a64146980 con 0x7f8a640bf800 2020-11-20T12:15:50.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.107+0000 7f8a5bfff700 1 -- 172.21.15.187:0/2635505197 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16791 (secure 0 0 0) 0x7f8a4c0311b0 con 0x7f8a640bf800 2020-11-20T12:15:50.111 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:50.111 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:15:17.285828+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:15:12.843783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2020-11-20T12:15:14.270455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:15:50.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a40056d50 msgr2=0x7f8a40059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:50.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a40056d50 0x7f8a40059210 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f8a54000bc0 tx=0x7f8a54009040).stop 2020-11-20T12:15:50.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bf800 msgr2=0x7f8a64142c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:50.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bf800 0x7f8a64142c30 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7f8a4c00b150 tx=0x7f8a4c015000).stop 2020-11-20T12:15:50.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 shutdown_connections 2020-11-20T12:15:50.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8a40056d50 0x7f8a40059210 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:50.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a64146c90 0x7f8a64134960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:50.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a6409f560 0x7f8a64145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:50.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 --2- 172.21.15.187:0/2635505197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a640bf800 0x7f8a64142c30 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:50.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.109+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 >> 172.21.15.187:0/2635505197 conn(0x7f8a64068370 msgr2=0x7f8a640a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:50.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.110+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 shutdown_connections 2020-11-20T12:15:50.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.110+0000 7f8a68b6f700 1 -- 172.21.15.187:0/2635505197 wait complete. 2020-11-20T12:15:50.137 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:15:50.137 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:15:50.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.371+0000 7fcc3e552700 1 Processor -- start 2020-11-20T12:15:50.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.371+0000 7fcc3e552700 1 -- start start 2020-11-20T12:15:50.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.372+0000 7fcc3e552700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcc380683f0 0x7fcc38066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:50.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.372+0000 7fcc3e552700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 0x7fcc38148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:50.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.372+0000 7fcc3e552700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3814c170 con 0x7fcc38145cc0 2020-11-20T12:15:50.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.372+0000 7fcc3e552700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3814c410 con 0x7fcc380683f0 2020-11-20T12:15:50.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.372+0000 7fcc3e552700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3814c6b0 con 0x7fcc381487e0 2020-11-20T12:15:50.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc3d550700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcc38145cc0 0x7fcc38065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47066/0 (socket says 172.21.15.187:47066) 2020-11-20T12:15:50.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc3d550700 1 -- 172.21.15.187:0/3955070929 learned_addr learned my addr 172.21.15.187:0/3955070929 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:50.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc3cd4f700 1 --2- 172.21.15.187:0/3955070929 >> v2:172.21.15.187:3300/0 conn(0x7fcc380683f0 0x7fcc38066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:50.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc3dd51700 1 --2- 172.21.15.187:0/3955070929 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 0x7fcc38148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:50.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc27fff700 1 -- 172.21.15.187:0/3955070929 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096862177 0 0) 0x7fcc3814c170 con 0x7fcc38145cc0 2020-11-20T12:15:50.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc27fff700 1 -- 172.21.15.187:0/3955070929 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc20003240 con 0x7fcc38145cc0 2020-11-20T12:15:50.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc3cd4f700 1 -- 172.21.15.187:0/3955070929 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 msgr2=0x7fcc38148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:50.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc3cd4f700 1 --2- 172.21.15.187:0/3955070929 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 0x7fcc38148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:50.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.373+0000 7fcc3cd4f700 1 -- 172.21.15.187:0/3955070929 >> v1:172.21.15.187:6789/0 conn(0x7fcc38145cc0 legacy=0x7fcc38065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:50.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.374+0000 7fcc3cd4f700 1 -- 172.21.15.187:0/3955070929 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc380658a0 con 0x7fcc380683f0 2020-11-20T12:15:50.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.374+0000 7fcc3cd4f700 1 --2- 172.21.15.187:0/3955070929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc380683f0 0x7fcc38066040 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7fcc2800b9a0 tx=0x7fcc28006df0).ready entity=mon.1 client_cookie=d931973528b9853e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:50.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.374+0000 7fcc27fff700 1 -- 172.21.15.187:0/3955070929 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc280097b0 con 0x7fcc380683f0 2020-11-20T12:15:50.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.374+0000 7fcc27fff700 1 -- 172.21.15.187:0/3955070929 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc28009910 con 0x7fcc380683f0 2020-11-20T12:15:50.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.374+0000 7fcc27fff700 1 -- 172.21.15.187:0/3955070929 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28009c90 con 0x7fcc380683f0 2020-11-20T12:15:50.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.375+0000 7fcc3e552700 1 -- 172.21.15.187:0/3955070929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc380683f0 msgr2=0x7fcc38066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:50.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.375+0000 7fcc3e552700 1 --2- 172.21.15.187:0/3955070929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc380683f0 0x7fcc38066040 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7fcc2800b9a0 tx=0x7fcc28006df0).stop 2020-11-20T12:15:50.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.375+0000 7fcc3e552700 1 -- 172.21.15.187:0/3955070929 shutdown_connections 2020-11-20T12:15:50.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.375+0000 7fcc3e552700 1 --2- 172.21.15.187:0/3955070929 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 0x7fcc38148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:50.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.375+0000 7fcc3e552700 1 --2- 172.21.15.187:0/3955070929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc380683f0 0x7fcc38066040 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:50.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.375+0000 7fcc3e552700 1 -- 172.21.15.187:0/3955070929 >> 172.21.15.187:0/3955070929 conn(0x7fcc380bab30 msgr2=0x7fcc380baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:50.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.375+0000 7fcc3e552700 1 -- 172.21.15.187:0/3955070929 shutdown_connections 2020-11-20T12:15:50.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.375+0000 7fcc3e552700 1 -- 172.21.15.187:0/3955070929 wait complete. 2020-11-20T12:15:50.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.377+0000 7fcc3e552700 1 Processor -- start 2020-11-20T12:15:50.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.377+0000 7fcc3e552700 1 -- start start 2020-11-20T12:15:50.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.378+0000 7fcc3e552700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc380683f0 0x7fcc3809f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:50.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.378+0000 7fcc3e552700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc38145cc0 0x7fcc38158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:50.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.378+0000 7fcc3e552700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 0x7fcc38158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:50.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.378+0000 7fcc3e552700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380b02d0 con 0x7fcc380683f0 2020-11-20T12:15:50.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.378+0000 7fcc3e552700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380b0570 con 0x7fcc38145cc0 2020-11-20T12:15:50.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.378+0000 7fcc3e552700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380b0810 con 0x7fcc381487e0 2020-11-20T12:15:50.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3cd4f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc38145cc0 0x7fcc38158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:50.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3cd4f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc38145cc0 0x7fcc38158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50468/0 (socket says 172.21.15.187:50468) 2020-11-20T12:15:50.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3cd4f700 1 -- 172.21.15.187:0/3267545046 learned_addr learned my addr 172.21.15.187:0/3267545046 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:50.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3d550700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc380683f0 0x7fcc3809f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:50.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3cd4f700 1 -- 172.21.15.187:0/3267545046 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 msgr2=0x7fcc38158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:50.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3cd4f700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 0x7fcc38158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:50.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3cd4f700 1 -- 172.21.15.187:0/3267545046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc380683f0 msgr2=0x7fcc3809f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:50.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3cd4f700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc380683f0 0x7fcc3809f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:50.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3cd4f700 1 -- 172.21.15.187:0/3267545046 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3809f930 con 0x7fcc38145cc0 2020-11-20T12:15:50.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.379+0000 7fcc3d550700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc380683f0 0x7fcc3809f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:50.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.380+0000 7fcc3cd4f700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc38145cc0 0x7fcc38158140 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7fcc28006e30 tx=0x7fcc28015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:50.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.380+0000 7fcc257fa700 1 -- 172.21.15.187:0/3267545046 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28011d50 con 0x7fcc38145cc0 2020-11-20T12:15:50.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.380+0000 7fcc257fa700 1 -- 172.21.15.187:0/3267545046 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2801e460 con 0x7fcc38145cc0 2020-11-20T12:15:50.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.380+0000 7fcc257fa700 1 -- 172.21.15.187:0/3267545046 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2801e7e0 con 0x7fcc38145cc0 2020-11-20T12:15:50.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.380+0000 7fcc3e552700 1 -- 172.21.15.187:0/3267545046 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc38158b20 con 0x7fcc38145cc0 2020-11-20T12:15:50.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.380+0000 7fcc3e552700 1 -- 172.21.15.187:0/3267545046 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc380b0e70 con 0x7fcc38145cc0 2020-11-20T12:15:50.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.382+0000 7fcc16ffd700 1 -- 172.21.15.187:0/3267545046 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc08004f90 con 0x7fcc38145cc0 2020-11-20T12:15:50.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.382+0000 7fcc257fa700 1 -- 172.21.15.187:0/3267545046 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcc28025030 con 0x7fcc38145cc0 2020-11-20T12:15:50.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.383+0000 7fcc257fa700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc10056d70 0x7fcc10059230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:50.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.383+0000 7fcc257fa700 1 -- 172.21.15.187:0/3267545046 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcc280717f0 con 0x7fcc38145cc0 2020-11-20T12:15:50.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.383+0000 7fcc3d550700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc10056d70 0x7fcc10059230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:50.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.384+0000 7fcc3d550700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc10056d70 0x7fcc10059230 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7fcc340055a0 tx=0x7fcc34009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:50.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.386+0000 7fcc257fa700 1 -- 172.21.15.187:0/3267545046 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc28046c60 con 0x7fcc38145cc0 2020-11-20T12:15:50.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:50.628+0000 7fcc16ffd700 1 -- 172.21.15.187:0/3267545046 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcc08003af0 con 0x7fcc38145cc0 2020-11-20T12:15:51.321 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:15:51.315+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:15:51.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.333+0000 7fcc257fa700 1 -- 172.21.15.187:0/3267545046 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v96) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcc28023020 con 0x7fcc38145cc0 2020-11-20T12:15:51.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 -- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc10056d70 msgr2=0x7fcc10059230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:51.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc10056d70 0x7fcc10059230 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7fcc340055a0 tx=0x7fcc34009040).stop 2020-11-20T12:15:51.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 -- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc38145cc0 msgr2=0x7fcc38158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:51.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc38145cc0 0x7fcc38158140 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7fcc28006e30 tx=0x7fcc28015000).stop 2020-11-20T12:15:51.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 -- 172.21.15.187:0/3267545046 shutdown_connections 2020-11-20T12:15:51.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcc10056d70 0x7fcc10059230 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:51.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc380683f0 0x7fcc3809f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:51.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc381487e0 0x7fcc38158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:51.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 --2- 172.21.15.187:0/3267545046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc38145cc0 0x7fcc38158140 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:51.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.336+0000 7fcc16ffd700 1 -- 172.21.15.187:0/3267545046 >> 172.21.15.187:0/3267545046 conn(0x7fcc380bab30 msgr2=0x7fcc380b9160 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:51.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.337+0000 7fcc16ffd700 1 -- 172.21.15.187:0/3267545046 shutdown_connections 2020-11-20T12:15:51.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.337+0000 7fcc16ffd700 1 -- 172.21.15.187:0/3267545046 wait complete. 2020-11-20T12:15:51.343 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:15:51.371 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:15:51.423 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:15:51.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.591+0000 7f2aef4be700 1 Processor -- start 2020-11-20T12:15:51.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.591+0000 7f2aef4be700 1 -- start start 2020-11-20T12:15:51.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.592+0000 7f2aef4be700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2ae80bfbe0 0x7f2ae81480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:51.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.592+0000 7f2aef4be700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae814d110 con 0x7f2ae809f560 2020-11-20T12:15:51.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.592+0000 7f2aef4be700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae814cbd0 con 0x7f2ae80bf800 2020-11-20T12:15:51.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.592+0000 7f2aef4be700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae814c9d0 con 0x7f2ae80bfbe0 2020-11-20T12:15:51.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.593+0000 7f2aee4bc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2ae80bf800 0x7f2ae814c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47092/0 (socket says 172.21.15.187:47092) 2020-11-20T12:15:51.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.593+0000 7f2aee4bc700 1 -- 172.21.15.187:0/3737062439 learned_addr learned my addr 172.21.15.187:0/3737062439 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:51.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.593+0000 7f2aedcbb700 1 --2- 172.21.15.187:0/3737062439 >> v2:172.21.15.187:3300/0 conn(0x7f2ae80bfbe0 0x7f2ae81480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:51.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.593+0000 7f2aed4ba700 1 -- 172.21.15.187:0/3737062439 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200830334 0 0) 0x7f2ae814cbd0 con 0x7f2ae80bf800 2020-11-20T12:15:51.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.593+0000 7f2aed4ba700 1 -- 172.21.15.187:0/3737062439 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad0003240 con 0x7f2ae80bf800 2020-11-20T12:15:51.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.593+0000 7f2aedcbb700 1 -- 172.21.15.187:0/3737062439 >> v1:172.21.15.187:6789/0 conn(0x7f2ae80bf800 legacy=0x7f2ae814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:51.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.593+0000 7f2aedcbb700 1 -- 172.21.15.187:0/3737062439 >> v1:172.21.15.5:6789/0 conn(0x7f2ae809f560 legacy=0x7f2ae80a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:51.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.593+0000 7f2aedcbb700 1 -- 172.21.15.187:0/3737062439 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae814ce70 con 0x7f2ae80bfbe0 2020-11-20T12:15:51.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.594+0000 7f2aedcbb700 1 --2- 172.21.15.187:0/3737062439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae80bfbe0 0x7f2ae81480c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f2ad800b9a0 tx=0x7f2ad8006df0).ready entity=mon.1 client_cookie=6c12ec246c50a727 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:51.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.595+0000 7f2aed4ba700 1 -- 172.21.15.187:0/3737062439 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8009800 con 0x7f2ae80bfbe0 2020-11-20T12:15:51.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.595+0000 7f2aed4ba700 1 -- 172.21.15.187:0/3737062439 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad8009960 con 0x7f2ae80bfbe0 2020-11-20T12:15:51.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.595+0000 7f2aed4ba700 1 -- 172.21.15.187:0/3737062439 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8009ce0 con 0x7f2ae80bfbe0 2020-11-20T12:15:51.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.595+0000 7f2aef4be700 1 -- 172.21.15.187:0/3737062439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae80bfbe0 msgr2=0x7f2ae81480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:51.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.596+0000 7f2aef4be700 1 --2- 172.21.15.187:0/3737062439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae80bfbe0 0x7f2ae81480c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f2ad800b9a0 tx=0x7f2ad8006df0).stop 2020-11-20T12:15:51.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.596+0000 7f2aef4be700 1 -- 172.21.15.187:0/3737062439 shutdown_connections 2020-11-20T12:15:51.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.596+0000 7f2aef4be700 1 --2- 172.21.15.187:0/3737062439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae80bfbe0 0x7f2ae81480c0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:51.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.596+0000 7f2aef4be700 1 -- 172.21.15.187:0/3737062439 >> 172.21.15.187:0/3737062439 conn(0x7f2ae8068370 msgr2=0x7f2ae80bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:51.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.596+0000 7f2aef4be700 1 -- 172.21.15.187:0/3737062439 shutdown_connections 2020-11-20T12:15:51.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.596+0000 7f2aef4be700 1 -- 172.21.15.187:0/3737062439 wait complete. 2020-11-20T12:15:51.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.598+0000 7f2aef4be700 1 Processor -- start 2020-11-20T12:15:51.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.598+0000 7f2aef4be700 1 -- start start 2020-11-20T12:15:51.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aef4be700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae809f560 0x7f2ae80b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:51.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aef4be700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ae80bf800 0x7f2ae80bac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:51.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aef4be700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ae80ba730 0x7f2ae815bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:51.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aef4be700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae80bb170 con 0x7f2ae80ba730 2020-11-20T12:15:51.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aef4be700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae815c710 con 0x7f2ae809f560 2020-11-20T12:15:51.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aef4be700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae815c9b0 con 0x7f2ae80bf800 2020-11-20T12:15:51.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aee4bc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae809f560 0x7f2ae80b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:51.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aee4bc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae809f560 0x7f2ae80b6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50492/0 (socket says 172.21.15.187:50492) 2020-11-20T12:15:51.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.599+0000 7f2aee4bc700 1 -- 172.21.15.187:0/2611958256 learned_addr learned my addr 172.21.15.187:0/2611958256 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:51.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aedcbb700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ae80bf800 0x7f2ae80bac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:51.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aee4bc700 1 -- 172.21.15.187:0/2611958256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ae80bf800 msgr2=0x7f2ae80bac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:51.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aeecbd700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ae80ba730 0x7f2ae815bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:51.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aee4bc700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ae80bf800 0x7f2ae80bac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:51.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aee4bc700 1 -- 172.21.15.187:0/2611958256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ae80ba730 msgr2=0x7f2ae815bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:51.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aee4bc700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ae80ba730 0x7f2ae815bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:51.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aee4bc700 1 -- 172.21.15.187:0/2611958256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae815c1e0 con 0x7f2ae809f560 2020-11-20T12:15:51.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aee4bc700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae809f560 0x7f2ae80b6cc0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7f2ae400ca90 tx=0x7f2ae4007de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:51.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2ad67fc700 1 -- 172.21.15.187:0/2611958256 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4005b80 con 0x7f2ae809f560 2020-11-20T12:15:51.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2ad67fc700 1 -- 172.21.15.187:0/2611958256 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae4005ce0 con 0x7f2ae809f560 2020-11-20T12:15:51.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae815c490 con 0x7f2ae809f560 2020-11-20T12:15:51.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2ad67fc700 1 -- 172.21.15.187:0/2611958256 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4007340 con 0x7f2ae809f560 2020-11-20T12:15:51.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.600+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae81475d0 con 0x7f2ae809f560 2020-11-20T12:15:51.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.602+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae81480c0 con 0x7f2ae809f560 2020-11-20T12:15:51.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.602+0000 7f2ad67fc700 1 -- 172.21.15.187:0/2611958256 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2ae4010030 con 0x7f2ae809f560 2020-11-20T12:15:51.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.603+0000 7f2ad67fc700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2ac0056d20 0x7f2ac00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:51.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.603+0000 7f2ad67fc700 1 -- 172.21.15.187:0/2611958256 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2ae40700b0 con 0x7f2ae809f560 2020-11-20T12:15:51.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.603+0000 7f2aedcbb700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2ac0056d20 0x7f2ac00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:51.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.603+0000 7f2aedcbb700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2ac0056d20 0x7f2ac00591e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f2ad800be60 tx=0x7f2ad8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:51.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.607+0000 7f2ad67fc700 1 -- 172.21.15.187:0/2611958256 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2ae40454b0 con 0x7f2ae809f560 2020-11-20T12:15:51.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:51.861+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2ae8080050 con 0x7f2ae809f560 2020-11-20T12:15:52.025 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:15:52.034 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:15:52.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.344+0000 7f2ad67fc700 1 -- 172.21.15.187:0/2611958256 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v97) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2ae4048ad0 con 0x7f2ae809f560 2020-11-20T12:15:52.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.347+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2ac0056d20 msgr2=0x7f2ac00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:52.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.347+0000 7f2aef4be700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2ac0056d20 0x7f2ac00591e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f2ad800be60 tx=0x7f2ad8015000).stop 2020-11-20T12:15:52.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.347+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae809f560 msgr2=0x7f2ae80b6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:52.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.347+0000 7f2aef4be700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae809f560 0x7f2ae80b6cc0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7f2ae400ca90 tx=0x7f2ae4007de0).stop 2020-11-20T12:15:52.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.348+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 shutdown_connections 2020-11-20T12:15:52.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.348+0000 7f2aef4be700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2ac0056d20 0x7f2ac00591e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:52.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.348+0000 7f2aef4be700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ae80ba730 0x7f2ae815bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:52.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.348+0000 7f2aef4be700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ae809f560 0x7f2ae80b6cc0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:52.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.348+0000 7f2aef4be700 1 --2- 172.21.15.187:0/2611958256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ae80bf800 0x7f2ae80bac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:52.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.348+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 >> 172.21.15.187:0/2611958256 conn(0x7f2ae8068370 msgr2=0x7f2ae8141e10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:52.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.348+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 shutdown_connections 2020-11-20T12:15:52.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.348+0000 7f2aef4be700 1 -- 172.21.15.187:0/2611958256 wait complete. 2020-11-20T12:15:52.380 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:15:52.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.620+0000 7fe28b59e700 1 Processor -- start 2020-11-20T12:15:52.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.620+0000 7fe28b59e700 1 -- start start 2020-11-20T12:15:52.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.622+0000 7fe28b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c14ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:52.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.622+0000 7fe28b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe28c0b4e00 0x7fe28c065ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:52.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.622+0000 7fe28b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c14be00 con 0x7fe28c145a20 2020-11-20T12:15:52.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.622+0000 7fe28b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28c065fc0 con 0x7fe28c0b4e00 2020-11-20T12:15:52.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.622+0000 7fe28b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c14c030 con 0x7fe28c0b4a20 2020-11-20T12:15:52.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.623+0000 7fe289d9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe28c0b4e00 0x7fe28c065ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:52.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.623+0000 7fe28a59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c14ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:52.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.623+0000 7fe28a59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c14ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:51388/0 (socket says 172.21.15.187:51388) 2020-11-20T12:15:52.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.623+0000 7fe28a59c700 1 -- 172.21.15.187:0/3905513616 learned_addr learned my addr 172.21.15.187:0/3905513616 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:52.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.624+0000 7fe28a59c700 1 -- 172.21.15.187:0/3905513616 >> v2:172.21.15.5:3300/0 conn(0x7fe28c0b4e00 msgr2=0x7fe28c065ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:52.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.624+0000 7fe28a59c700 1 --2- 172.21.15.187:0/3905513616 >> v2:172.21.15.5:3300/0 conn(0x7fe28c0b4e00 0x7fe28c065ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:52.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.624+0000 7fe28a59c700 1 -- 172.21.15.187:0/3905513616 >> v1:172.21.15.5:6789/0 conn(0x7fe28c145a20 legacy=0x7fe28c148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:52.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.624+0000 7fe28a59c700 1 -- 172.21.15.187:0/3905513616 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c14c2d0 con 0x7fe28c0b4a20 2020-11-20T12:15:52.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.624+0000 7fe28a59c700 1 --2- 172.21.15.187:0/3905513616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c14ca10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe27c006fd0 tx=0x7fe27c006eb0).ready entity=mon.2 client_cookie=7dcde6323a522b63 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:52.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.625+0000 7fe28959a700 1 -- 172.21.15.187:0/3905513616 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c009810 con 0x7fe28c0b4a20 2020-11-20T12:15:52.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.625+0000 7fe28959a700 1 -- 172.21.15.187:0/3905513616 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c009970 con 0x7fe28c0b4a20 2020-11-20T12:15:52.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.625+0000 7fe28959a700 1 -- 172.21.15.187:0/3905513616 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c009cf0 con 0x7fe28c0b4a20 2020-11-20T12:15:52.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.625+0000 7fe28b59e700 1 -- 172.21.15.187:0/3905513616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 msgr2=0x7fe28c14ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:52.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.625+0000 7fe28b59e700 1 --2- 172.21.15.187:0/3905513616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c14ca10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe27c006fd0 tx=0x7fe27c006eb0).stop 2020-11-20T12:15:52.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.626+0000 7fe28b59e700 1 -- 172.21.15.187:0/3905513616 shutdown_connections 2020-11-20T12:15:52.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.626+0000 7fe28b59e700 1 --2- 172.21.15.187:0/3905513616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c14ca10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:52.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.626+0000 7fe28b59e700 1 --2- 172.21.15.187:0/3905513616 >> v2:172.21.15.5:3300/0 conn(0x7fe28c0b4e00 0x7fe28c065ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:52.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.626+0000 7fe28b59e700 1 -- 172.21.15.187:0/3905513616 >> 172.21.15.187:0/3905513616 conn(0x7fe28c0bb3d0 msgr2=0x7fe28c0bb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:52.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.626+0000 7fe28b59e700 1 -- 172.21.15.187:0/3905513616 shutdown_connections 2020-11-20T12:15:52.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.626+0000 7fe28b59e700 1 -- 172.21.15.187:0/3905513616 wait complete. 2020-11-20T12:15:52.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.627+0000 7fe28b59e700 1 Processor -- start 2020-11-20T12:15:52.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.627+0000 7fe28b59e700 1 -- start start 2020-11-20T12:15:52.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.628+0000 7fe28b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c15b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:52.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.628+0000 7fe28b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe28c0b4e00 0x7fe28c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:52.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.628+0000 7fe28b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe28c145a20 0x7fe28c149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:52.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.628+0000 7fe28b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c115f80 con 0x7fe28c145a20 2020-11-20T12:15:52.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.628+0000 7fe28b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c1161a0 con 0x7fe28c0b4e00 2020-11-20T12:15:52.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.628+0000 7fe28b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c116440 con 0x7fe28c0b4a20 2020-11-20T12:15:52.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.628+0000 7fe289d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe28c0b4e00 0x7fe28c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:52.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.628+0000 7fe289d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe28c0b4e00 0x7fe28c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50506/0 (socket says 172.21.15.187:50506) 2020-11-20T12:15:52.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe289d9b700 1 -- 172.21.15.187:0/2335400095 learned_addr learned my addr 172.21.15.187:0/2335400095 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:52.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe289d9b700 1 -- 172.21.15.187:0/2335400095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 msgr2=0x7fe28c15b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:52.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe289d9b700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c15b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:52.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe289d9b700 1 -- 172.21.15.187:0/2335400095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe28c145a20 msgr2=0x7fe28c149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:52.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe289d9b700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe28c145a20 0x7fe28c149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:52.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe289d9b700 1 -- 172.21.15.187:0/2335400095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c15c3d0 con 0x7fe28c0b4e00 2020-11-20T12:15:52.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe289d9b700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe28c0b4e00 0x7fe28c15bee0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7fe28000eba0 tx=0x7fe280006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:52.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe27a7fc700 1 -- 172.21.15.187:0/2335400095 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28000c8f0 con 0x7fe28c0b4e00 2020-11-20T12:15:52.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe27a7fc700 1 -- 172.21.15.187:0/2335400095 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28000ca50 con 0x7fe28c0b4e00 2020-11-20T12:15:52.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.629+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c14a920 con 0x7fe28c0b4e00 2020-11-20T12:15:52.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.630+0000 7fe27a7fc700 1 -- 172.21.15.187:0/2335400095 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28000cdd0 con 0x7fe28c0b4e00 2020-11-20T12:15:52.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.630+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c149940 con 0x7fe28c0b4e00 2020-11-20T12:15:52.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.631+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c116600 con 0x7fe28c0b4e00 2020-11-20T12:15:52.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.631+0000 7fe27a7fc700 1 -- 172.21.15.187:0/2335400095 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe28001a030 con 0x7fe28c0b4e00 2020-11-20T12:15:52.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.632+0000 7fe27a7fc700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe260056d50 0x7fe260059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:52.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.632+0000 7fe27a7fc700 1 -- 172.21.15.187:0/2335400095 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe28006fc40 con 0x7fe28c0b4e00 2020-11-20T12:15:52.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.632+0000 7fe28a59c700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe260056d50 0x7fe260059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:52.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.633+0000 7fe28a59c700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe260056d50 0x7fe260059210 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fe27c006ef0 tx=0x7fe27c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:52.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.636+0000 7fe27a7fc700 1 -- 172.21.15.187:0/2335400095 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe280044fe0 con 0x7fe28c0b4e00 2020-11-20T12:15:52.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:52.822+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe28c0b8fd0 con 0x7fe28c0b4e00 2020-11-20T12:15:53.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.348+0000 7fe27a7fc700 1 -- 172.21.15.187:0/2335400095 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe280048600 con 0x7fe28c0b4e00 2020-11-20T12:15:53.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.352+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe260056d50 msgr2=0x7fe260059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:53.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.352+0000 7fe28b59e700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe260056d50 0x7fe260059210 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fe27c006ef0 tx=0x7fe27c015000).stop 2020-11-20T12:15:53.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.352+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe28c0b4e00 msgr2=0x7fe28c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:53.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.352+0000 7fe28b59e700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe28c0b4e00 0x7fe28c15bee0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7fe28000eba0 tx=0x7fe280006df0).stop 2020-11-20T12:15:53.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.353+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 shutdown_connections 2020-11-20T12:15:53.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.353+0000 7fe28b59e700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe260056d50 0x7fe260059210 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:53.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.353+0000 7fe28b59e700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe28c145a20 0x7fe28c149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:53.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.353+0000 7fe28b59e700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe28c0b4a20 0x7fe28c15b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:53.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.353+0000 7fe28b59e700 1 --2- 172.21.15.187:0/2335400095 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe28c0b4e00 0x7fe28c15bee0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:53.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.353+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 >> 172.21.15.187:0/2335400095 conn(0x7fe28c0bb3d0 msgr2=0x7fe28c0b9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:53.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.353+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 shutdown_connections 2020-11-20T12:15:53.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.353+0000 7fe28b59e700 1 -- 172.21.15.187:0/2335400095 wait complete. 2020-11-20T12:15:53.359 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:15:53.383 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:15:53.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.633+0000 7f7445a24700 1 Processor -- start 2020-11-20T12:15:53.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.633+0000 7f7445a24700 1 -- start start 2020-11-20T12:15:53.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.634+0000 7f7445a24700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f74400b5180 0x7f74400a1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:53.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.634+0000 7f7445a24700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74400a1640 0x7f74401521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:53.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.634+0000 7f7445a24700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74400b9ef0 con 0x7f7440148550 2020-11-20T12:15:53.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.634+0000 7f7445a24700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400ba190 con 0x7f74400b5180 2020-11-20T12:15:53.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.634+0000 7f7445a24700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400ba430 con 0x7f74400a1640 2020-11-20T12:15:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.635+0000 7f7445223700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74400a1640 0x7f74401521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.635+0000 7f7444a22700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7440148550 0x7f74400b9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47120/0 (socket says 172.21.15.187:47120) 2020-11-20T12:15:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.635+0000 7f7444a22700 1 -- 172.21.15.187:0/309956772 learned_addr learned my addr 172.21.15.187:0/309956772 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:53.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.635+0000 7f7437fff700 1 --2- 172.21.15.187:0/309956772 >> v2:172.21.15.5:3300/0 conn(0x7f74400b5180 0x7f74400a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:53.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f74377fe700 1 -- 172.21.15.187:0/309956772 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668389644 0 0) 0x7f74400b9ef0 con 0x7f7440148550 2020-11-20T12:15:53.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f74377fe700 1 -- 172.21.15.187:0/309956772 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f742c003240 con 0x7f7440148550 2020-11-20T12:15:53.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f7445223700 1 -- 172.21.15.187:0/309956772 >> v2:172.21.15.5:3300/0 conn(0x7f74400b5180 msgr2=0x7f74400a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:53.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f7445223700 1 --2- 172.21.15.187:0/309956772 >> v2:172.21.15.5:3300/0 conn(0x7f74400b5180 0x7f74400a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:53.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f7445223700 1 -- 172.21.15.187:0/309956772 >> v1:172.21.15.187:6789/0 conn(0x7f7440148550 legacy=0x7f74400b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:53.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f7445223700 1 -- 172.21.15.187:0/309956772 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7440148930 con 0x7f74400a1640 2020-11-20T12:15:53.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f7445223700 1 --2- 172.21.15.187:0/309956772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74400a1640 0x7f74401521a0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f7440068780 tx=0x7f7438006df0).ready entity=mon.1 client_cookie=3f2f423d728f6263 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:53.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f74377fe700 1 -- 172.21.15.187:0/309956772 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74380097b0 con 0x7f74400a1640 2020-11-20T12:15:53.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f74377fe700 1 -- 172.21.15.187:0/309956772 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438009910 con 0x7f74400a1640 2020-11-20T12:15:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.636+0000 7f74377fe700 1 -- 172.21.15.187:0/309956772 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438009c90 con 0x7f74400a1640 2020-11-20T12:15:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.637+0000 7f7445a24700 1 -- 172.21.15.187:0/309956772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74400a1640 msgr2=0x7f74401521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.637+0000 7f7445a24700 1 --2- 172.21.15.187:0/309956772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74400a1640 0x7f74401521a0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f7440068780 tx=0x7f7438006df0).stop 2020-11-20T12:15:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.637+0000 7f7445a24700 1 -- 172.21.15.187:0/309956772 shutdown_connections 2020-11-20T12:15:53.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.637+0000 7f7445a24700 1 --2- 172.21.15.187:0/309956772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74400a1640 0x7f74401521a0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.637+0000 7f7445a24700 1 --2- 172.21.15.187:0/309956772 >> v2:172.21.15.5:3300/0 conn(0x7f74400b5180 0x7f74400a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.637+0000 7f7445a24700 1 -- 172.21.15.187:0/309956772 >> 172.21.15.187:0/309956772 conn(0x7f7440068320 msgr2=0x7f74400bef70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.637+0000 7f7445a24700 1 -- 172.21.15.187:0/309956772 shutdown_connections 2020-11-20T12:15:53.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.637+0000 7f7445a24700 1 -- 172.21.15.187:0/309956772 wait complete. 2020-11-20T12:15:53.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.639+0000 7f7445a24700 1 Processor -- start 2020-11-20T12:15:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.639+0000 7f7445a24700 1 -- start start 2020-11-20T12:15:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445a24700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74400a1640 0x7f7440141650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445a24700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74400b5180 0x7f74400bdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445a24700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7440148550 0x7f7440140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445a24700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7440144d70 con 0x7f74400a1640 2020-11-20T12:15:53.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445a24700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74400a43a0 con 0x7f7440148550 2020-11-20T12:15:53.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445a24700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74400a4640 con 0x7f74400b5180 2020-11-20T12:15:53.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445223700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7440148550 0x7f7440140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:53.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445223700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7440148550 0x7f7440140760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50524/0 (socket says 172.21.15.187:50524) 2020-11-20T12:15:53.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445223700 1 -- 172.21.15.187:0/85814578 learned_addr learned my addr 172.21.15.187:0/85814578 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:53.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445223700 1 -- 172.21.15.187:0/85814578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74400b5180 msgr2=0x7f74400bdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:53.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.640+0000 7f7445223700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74400b5180 0x7f74400bdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:53.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7445223700 1 -- 172.21.15.187:0/85814578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74400a1640 msgr2=0x7f7440141650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:53.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7445223700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74400a1640 0x7f7440141650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:53.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7445223700 1 -- 172.21.15.187:0/85814578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74400a48e0 con 0x7f7440148550 2020-11-20T12:15:53.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7445223700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7440148550 0x7f7440140760 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7f74400ba570 tx=0x7f7438015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:53.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7434ff9700 1 -- 172.21.15.187:0/85814578 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743801e460 con 0x7f7440148550 2020-11-20T12:15:53.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7434ff9700 1 -- 172.21.15.187:0/85814578 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743801e5c0 con 0x7f7440148550 2020-11-20T12:15:53.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7434ff9700 1 -- 172.21.15.187:0/85814578 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743801e940 con 0x7f7440148550 2020-11-20T12:15:53.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74400a4b80 con 0x7f7440148550 2020-11-20T12:15:53.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.641+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74400be270 con 0x7f7440148550 2020-11-20T12:15:53.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.643+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74400be490 con 0x7f7440148550 2020-11-20T12:15:53.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.643+0000 7f7434ff9700 1 -- 172.21.15.187:0/85814578 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7438025030 con 0x7f7440148550 2020-11-20T12:15:53.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.644+0000 7f7434ff9700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7418056d20 0x7f74180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:53.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.644+0000 7f7434ff9700 1 -- 172.21.15.187:0/85814578 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f7438071700 con 0x7f7440148550 2020-11-20T12:15:53.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.644+0000 7f7444a22700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7418056d20 0x7f74180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:53.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.644+0000 7f7444a22700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7418056d20 0x7f74180591e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f743c005920 tx=0x7f743c007a20).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:53.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.647+0000 7f7434ff9700 1 -- 172.21.15.187:0/85814578 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7438046d60 con 0x7f7440148550 2020-11-20T12:15:53.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:53.840+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f74400805f0 con 0x7f7440148550 2020-11-20T12:15:54.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.362+0000 7f7434ff9700 1 -- 172.21.15.187:0/85814578 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7438023070 con 0x7f7440148550 2020-11-20T12:15:55.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.365+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7418056d20 msgr2=0x7f74180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:55.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.365+0000 7f7445a24700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7418056d20 0x7f74180591e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f743c005920 tx=0x7f743c007a20).stop 2020-11-20T12:15:55.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.365+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7440148550 msgr2=0x7f7440140760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:55.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.365+0000 7f7445a24700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7440148550 0x7f7440140760 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7f74400ba570 tx=0x7f7438015000).stop 2020-11-20T12:15:55.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.366+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 shutdown_connections 2020-11-20T12:15:55.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.366+0000 7f7445a24700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7418056d20 0x7f74180591e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:55.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.366+0000 7f7445a24700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74400a1640 0x7f7440141650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:55.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.366+0000 7f7445a24700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7440148550 0x7f7440140760 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:55.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.366+0000 7f7445a24700 1 --2- 172.21.15.187:0/85814578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f74400b5180 0x7f74400bdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:55.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.366+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 >> 172.21.15.187:0/85814578 conn(0x7f7440068320 msgr2=0x7f7440143b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:55.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.366+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 shutdown_connections 2020-11-20T12:15:55.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:54.366+0000 7f7445a24700 1 -- 172.21.15.187:0/85814578 wait complete. 2020-11-20T12:15:55.998 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:15:55.999 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:15:55.999 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:15:56.000 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:15:56.000 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:15:56.001 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:15:56.001 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:15:56.002 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:15:56.002 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:15:56.027 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:15:56.028 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:15:56.036 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:15:56.037 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:15:56.046 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:15:56.046 DEBUG:teuthology.parallel:result is None 2020-11-20T12:15:56.081 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:15:56.082 DEBUG:teuthology.parallel:result is None 2020-11-20T12:15:56.083 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:15:56.248 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:15:56.248 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:15:56.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.303+0000 7f083bfff700 1 Processor -- start 2020-11-20T12:15:56.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.304+0000 7f083bfff700 1 -- start start 2020-11-20T12:15:56.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f083c1484d0 0x7f083c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:56.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f083c066f70 0x7f083c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:56.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c0b1300 con 0x7f083c0622c0 2020-11-20T12:15:56.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083c067730 con 0x7f083c1484d0 2020-11-20T12:15:56.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083c0b15a0 con 0x7f083c066f70 2020-11-20T12:15:56.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f083c066f70 0x7f083c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:56.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f083c066f70 0x7f083c0b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50544/0 (socket says 172.21.15.187:50544) 2020-11-20T12:15:56.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083b7fe700 1 -- 172.21.15.187:0/456106789 learned_addr learned my addr 172.21.15.187:0/456106789 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:56.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.305+0000 7f083affd700 1 --2- 172.21.15.187:0/456106789 >> v2:172.21.15.5:3300/0 conn(0x7f083c1484d0 0x7f083c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:56.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.306+0000 7f083b7fe700 1 -- 172.21.15.187:0/456106789 >> v2:172.21.15.5:3300/0 conn(0x7f083c1484d0 msgr2=0x7f083c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:56.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.306+0000 7f083b7fe700 1 --2- 172.21.15.187:0/456106789 >> v2:172.21.15.5:3300/0 conn(0x7f083c1484d0 0x7f083c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:56.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.306+0000 7f083b7fe700 1 -- 172.21.15.187:0/456106789 >> v1:172.21.15.5:6789/0 conn(0x7f083c0622c0 legacy=0x7f083c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:56.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.306+0000 7f083b7fe700 1 -- 172.21.15.187:0/456106789 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c0b1840 con 0x7f083c066f70 2020-11-20T12:15:56.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.306+0000 7f083b7fe700 1 --2- 172.21.15.187:0/456106789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c066f70 0x7f083c0b5d50 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7f083c045000 tx=0x7f0834009290).ready entity=mon.1 client_cookie=2808dea01c807dc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:56.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.306+0000 7f0839ffb700 1 -- 172.21.15.187:0/456106789 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083400bb80 con 0x7f083c066f70 2020-11-20T12:15:56.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.306+0000 7f0839ffb700 1 -- 172.21.15.187:0/456106789 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083400bce0 con 0x7f083c066f70 2020-11-20T12:15:56.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f0839ffb700 1 -- 172.21.15.187:0/456106789 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834011610 con 0x7f083c066f70 2020-11-20T12:15:56.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f083bfff700 1 -- 172.21.15.187:0/456106789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c066f70 msgr2=0x7f083c0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:56.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f083bfff700 1 --2- 172.21.15.187:0/456106789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c066f70 0x7f083c0b5d50 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7f083c045000 tx=0x7f0834009290).stop 2020-11-20T12:15:56.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f083bfff700 1 -- 172.21.15.187:0/456106789 shutdown_connections 2020-11-20T12:15:56.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f083bfff700 1 --2- 172.21.15.187:0/456106789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c066f70 0x7f083c0b5d50 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:56.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f083bfff700 1 --2- 172.21.15.187:0/456106789 >> v2:172.21.15.5:3300/0 conn(0x7f083c1484d0 0x7f083c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:56.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f083bfff700 1 -- 172.21.15.187:0/456106789 >> 172.21.15.187:0/456106789 conn(0x7f083c0bab30 msgr2=0x7f083c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:56.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f083bfff700 1 -- 172.21.15.187:0/456106789 shutdown_connections 2020-11-20T12:15:56.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.307+0000 7f083bfff700 1 -- 172.21.15.187:0/456106789 wait complete. 2020-11-20T12:15:56.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.309+0000 7f083bfff700 1 Processor -- start 2020-11-20T12:15:56.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.309+0000 7f083bfff700 1 -- start start 2020-11-20T12:15:56.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c0622c0 0x7f083c13aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:56.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f083c066f70 0x7f083c09b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:56.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f083c1484d0 0x7f083c0a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:56.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c13afd0 con 0x7f083c066f70 2020-11-20T12:15:56.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c147080 con 0x7f083c0622c0 2020-11-20T12:15:56.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c1472e0 con 0x7f083c1484d0 2020-11-20T12:15:56.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c0622c0 0x7f083c13aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:56.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c0622c0 0x7f083c13aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50546/0 (socket says 172.21.15.187:50546) 2020-11-20T12:15:56.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083affd700 1 -- 172.21.15.187:0/3702380153 learned_addr learned my addr 172.21.15.187:0/3702380153 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:56.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083b7fe700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f083c1484d0 0x7f083c0a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:56.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083affd700 1 -- 172.21.15.187:0/3702380153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f083c1484d0 msgr2=0x7f083c0a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:56.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083affd700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f083c1484d0 0x7f083c0a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:56.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083a7fc700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f083c066f70 0x7f083c09b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:56.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083affd700 1 -- 172.21.15.187:0/3702380153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f083c066f70 msgr2=0x7f083c09b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:56.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.310+0000 7f083affd700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f083c066f70 0x7f083c09b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:56.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.311+0000 7f083affd700 1 -- 172.21.15.187:0/3702380153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c147580 con 0x7f083c0622c0 2020-11-20T12:15:56.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.311+0000 7f083affd700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c0622c0 0x7f083c13aae0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 rx=0x7f082c00d8e0 tx=0x7f082c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:56.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.311+0000 7f0822ffd700 1 -- 172.21.15.187:0/3702380153 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00b130 con 0x7f083c0622c0 2020-11-20T12:15:56.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.311+0000 7f0822ffd700 1 -- 172.21.15.187:0/3702380153 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c009790 con 0x7f083c0622c0 2020-11-20T12:15:56.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.311+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c09b6c0 con 0x7f083c0622c0 2020-11-20T12:15:56.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.311+0000 7f0822ffd700 1 -- 172.21.15.187:0/3702380153 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c009b10 con 0x7f083c0622c0 2020-11-20T12:15:56.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.311+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c0b38e0 con 0x7f083c0622c0 2020-11-20T12:15:56.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.312+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083c061a20 con 0x7f083c0622c0 2020-11-20T12:15:56.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.313+0000 7f0822ffd700 1 -- 172.21.15.187:0/3702380153 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f082c01c460 con 0x7f083c0622c0 2020-11-20T12:15:56.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.313+0000 7f0822ffd700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0818056d20 0x7f08180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:56.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.314+0000 7f0822ffd700 1 -- 172.21.15.187:0/3702380153 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f082c06fc90 con 0x7f083c0622c0 2020-11-20T12:15:56.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.314+0000 7f083a7fc700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0818056d20 0x7f08180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:56.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.314+0000 7f083a7fc700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0818056d20 0x7f08180591e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f083c13f4a0 tx=0x7f0830009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:56.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.318+0000 7f0822ffd700 1 -- 172.21.15.187:0/3702380153 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f082c045360 con 0x7f083c0622c0 2020-11-20T12:15:56.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:56.521+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f083c1488b0 con 0x7f083c0622c0 2020-11-20T12:15:57.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.490+0000 7f0822ffd700 1 -- 172.21.15.187:0/3702380153 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v112) v1 ==== 97+0+0 (secure 0 0 0) 0x7f082c048980 con 0x7f083c0622c0 2020-11-20T12:15:57.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0818056d20 msgr2=0x7f08180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:57.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0818056d20 0x7f08180591e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f083c13f4a0 tx=0x7f0830009040).stop 2020-11-20T12:15:57.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c0622c0 msgr2=0x7f083c13aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:57.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c0622c0 0x7f083c13aae0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 rx=0x7f082c00d8e0 tx=0x7f082c008040).stop 2020-11-20T12:15:57.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 shutdown_connections 2020-11-20T12:15:57.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0818056d20 0x7f08180591e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f083c1484d0 0x7f083c0a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f083c0622c0 0x7f083c13aae0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 --2- 172.21.15.187:0/3702380153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f083c066f70 0x7f083c09b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 >> 172.21.15.187:0/3702380153 conn(0x7f083c0bab30 msgr2=0x7f083c13ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:57.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.493+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 shutdown_connections 2020-11-20T12:15:57.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.494+0000 7f083bfff700 1 -- 172.21.15.187:0/3702380153 wait complete. 2020-11-20T12:15:57.499 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:15:57.526 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:15:57.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.767+0000 7f154b59e700 1 Processor -- start 2020-11-20T12:15:57.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.767+0000 7f154b59e700 1 -- start start 2020-11-20T12:15:57.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:57.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f154c0b1760 0x7f154c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:57.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154b59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f154c065680 0x7f154c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:57.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c150390 con 0x7f154c0b1760 2020-11-20T12:15:57.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154b59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c150630 con 0x7f154c065680 2020-11-20T12:15:57.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c068510 con 0x7f154c0b1380 2020-11-20T12:15:57.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154ad9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f154c065680 0x7f154c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:57.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154ad9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f154c065680 0x7f154c061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50558/0 (socket says 172.21.15.187:50558) 2020-11-20T12:15:57.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.769+0000 7f154ad9d700 1 -- 172.21.15.187:0/1826003008 learned_addr learned my addr 172.21.15.187:0/1826003008 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:57.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f1549d9b700 1 --2- 172.21.15.187:0/1826003008 >> v2:172.21.15.5:3300/0 conn(0x7f154c0b1760 0x7f154c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:57.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154a59c700 1 --2- 172.21.15.187:0/1826003008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:57.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154ad9d700 1 -- 172.21.15.187:0/1826003008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 msgr2=0x7f154c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:57.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154ad9d700 1 --2- 172.21.15.187:0/1826003008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154ad9d700 1 -- 172.21.15.187:0/1826003008 >> v2:172.21.15.5:3300/0 conn(0x7f154c0b1760 msgr2=0x7f154c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:57.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154ad9d700 1 --2- 172.21.15.187:0/1826003008 >> v2:172.21.15.5:3300/0 conn(0x7f154c0b1760 0x7f154c065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154ad9d700 1 -- 172.21.15.187:0/1826003008 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c150fc0 con 0x7f154c065680 2020-11-20T12:15:57.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154a59c700 1 --2- 172.21.15.187:0/1826003008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:57.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154ad9d700 1 --2- 172.21.15.187:0/1826003008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c065680 0x7f154c061dd0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7f154c0b50e0 tx=0x7f1544009290).ready entity=mon.1 client_cookie=e03bca7f69ed097 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:57.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.770+0000 7f154959a700 1 -- 172.21.15.187:0/1826003008 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154400bbc0 con 0x7f154c065680 2020-11-20T12:15:57.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154959a700 1 -- 172.21.15.187:0/1826003008 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154400bd20 con 0x7f154c065680 2020-11-20T12:15:57.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154959a700 1 -- 172.21.15.187:0/1826003008 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544011610 con 0x7f154c065680 2020-11-20T12:15:57.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 -- 172.21.15.187:0/1826003008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c065680 msgr2=0x7f154c061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:57.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 --2- 172.21.15.187:0/1826003008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c065680 0x7f154c061dd0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7f154c0b50e0 tx=0x7f1544009290).stop 2020-11-20T12:15:57.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 -- 172.21.15.187:0/1826003008 shutdown_connections 2020-11-20T12:15:57.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 --2- 172.21.15.187:0/1826003008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c065680 0x7f154c061dd0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 --2- 172.21.15.187:0/1826003008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 --2- 172.21.15.187:0/1826003008 >> v2:172.21.15.5:3300/0 conn(0x7f154c0b1760 0x7f154c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 -- 172.21.15.187:0/1826003008 >> 172.21.15.187:0/1826003008 conn(0x7f154c0bab30 msgr2=0x7f154c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:57.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 -- 172.21.15.187:0/1826003008 shutdown_connections 2020-11-20T12:15:57.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.771+0000 7f154b59e700 1 -- 172.21.15.187:0/1826003008 wait complete. 2020-11-20T12:15:57.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.773+0000 7f154b59e700 1 Processor -- start 2020-11-20T12:15:57.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.773+0000 7f154b59e700 1 -- start start 2020-11-20T12:15:57.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c09a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:57.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f154c0b1760 0x7f154c0b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:57.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c0b36a0 0x7f154c0b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:57.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0b4900 con 0x7f154c0b1760 2020-11-20T12:15:57.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c09b610 con 0x7f154c0b36a0 2020-11-20T12:15:57.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c09dae0 con 0x7f154c0b1380 2020-11-20T12:15:57.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c0b36a0 0x7f154c0b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:57.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c0b36a0 0x7f154c0b7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50564/0 (socket says 172.21.15.187:50564) 2020-11-20T12:15:57.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.774+0000 7f154ad9d700 1 -- 172.21.15.187:0/1455412357 learned_addr learned my addr 172.21.15.187:0/1455412357 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:57.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f1549d9b700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f154c0b1760 0x7f154c0b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:57.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f154a59c700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c09a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:57.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f154ad9d700 1 -- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 msgr2=0x7f154c09a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:57.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f154ad9d700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c09a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f154ad9d700 1 -- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f154c0b1760 msgr2=0x7f154c0b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:57.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f154ad9d700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f154c0b1760 0x7f154c0b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:57.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f154ad9d700 1 -- 172.21.15.187:0/1455412357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c09dd10 con 0x7f154c0b36a0 2020-11-20T12:15:57.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f1549d9b700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f154c0b1760 0x7f154c0b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:57.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f154a59c700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c09a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:57.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f154ad9d700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c0b36a0 0x7f154c0b7480 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 rx=0x7f154c150770 tx=0x7f1544016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:57.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f153a7fc700 1 -- 172.21.15.187:0/1455412357 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544011650 con 0x7f154c0b36a0 2020-11-20T12:15:57.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f153a7fc700 1 -- 172.21.15.187:0/1455412357 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154400be90 con 0x7f154c0b36a0 2020-11-20T12:15:57.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.775+0000 7f153a7fc700 1 -- 172.21.15.187:0/1455412357 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544010e10 con 0x7f154c0b36a0 2020-11-20T12:15:57.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.776+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c09dfb0 con 0x7f154c0b36a0 2020-11-20T12:15:57.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.776+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c15b5f0 con 0x7f154c0b36a0 2020-11-20T12:15:57.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.777+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c061a20 con 0x7f154c0b36a0 2020-11-20T12:15:57.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.777+0000 7f153a7fc700 1 -- 172.21.15.187:0/1455412357 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1544017070 con 0x7f154c0b36a0 2020-11-20T12:15:57.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.778+0000 7f153a7fc700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1520056d20 0x7f15200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:57.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.778+0000 7f153a7fc700 1 -- 172.21.15.187:0/1455412357 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f1544071e20 con 0x7f154c0b36a0 2020-11-20T12:15:57.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.779+0000 7f154a59c700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1520056d20 0x7f15200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:57.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.779+0000 7f154a59c700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1520056d20 0x7f15200591e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f153c00b870 tx=0x7f153c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:57.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:57.782+0000 7f153a7fc700 1 -- 172.21.15.187:0/1455412357 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1544047620 con 0x7f154c0b36a0 2020-11-20T12:15:58.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.033+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f154c0446d0 con 0x7f154c0b36a0 2020-11-20T12:15:58.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.034+0000 7f153a7fc700 1 -- 172.21.15.187:0/1455412357 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f1544015070 con 0x7f154c0b36a0 2020-11-20T12:15:58.037 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:58.038 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:15:58.038 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:15:58.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.039 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:15:58.039 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:15:58.039 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.040 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:58.040 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.040 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.040 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.040 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.041 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:15:58.041 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:15:58.042 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.042 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:58.042 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.042 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.042 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.043 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.043 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:15:58.044 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:15:58.044 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.044 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.044 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.045 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.045 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.046 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:15:58.046 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:15:58.046 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.046 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.046 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.047 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.047 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.048 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:15:58.048 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:15:58.048 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.049 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.051 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.052 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.052 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.052 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:15:58.053 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:15:58.053 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.053 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.053 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.054 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.054 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.054 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:15:58.055 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:15:58.055 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.055 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.055 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.055 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.056 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.056 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:15:58.056 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:15:58.057 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.057 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.057 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.057 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.058 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.058 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:15:58.058 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:15:58.059 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.059 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.059 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.059 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.059 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.060 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:15:58.060 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:15:58.060 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.061 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.061 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.061 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.061 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.062 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:15:58.062 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:15:58.062 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.062 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:58.063 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:15:58.063 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:15:58.063 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.064 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:15:58.064 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:15:58.064 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.064 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:58.065 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:15:58.065 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:15:58.065 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:15:58.065 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.066 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:15:58.066 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:15:58.066 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.066 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:58.067 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:15:58.067 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:15:58.067 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:15:58.067 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.068 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:15:58.068 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:15:58.068 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.068 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:15:58.069 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:15:58.069 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:15:58.069 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:15:58.069 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.070 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:15:58.070 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:15:58.070 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.071 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:15:58.071 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:15:58.071 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:15:58.071 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.071 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:15:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:15:58.072 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.073 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:15:58.073 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:15:58.073 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.073 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:58.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:58.074 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:15:58.074 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:15:58.074 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:15:58.075 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:15:58.075 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:15:58.075 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:15:58.075 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.075 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:58.076 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:15:58.076 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:15:58.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.036+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1520056d20 msgr2=0x7f15200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.036+0000 7f154b59e700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1520056d20 0x7f15200591e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f153c00b870 tx=0x7f153c008040).stop 2020-11-20T12:15:58.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.036+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c0b36a0 msgr2=0x7f154c0b7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.036+0000 7f154b59e700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c0b36a0 0x7f154c0b7480 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 rx=0x7f154c150770 tx=0x7f1544016000).stop 2020-11-20T12:15:58.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.037+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 shutdown_connections 2020-11-20T12:15:58.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.037+0000 7f154b59e700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1520056d20 0x7f15200591e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.037+0000 7f154b59e700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f154c0b36a0 0x7f154c0b7480 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.037+0000 7f154b59e700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f154c0b1380 0x7f154c09a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.037+0000 7f154b59e700 1 --2- 172.21.15.187:0/1455412357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f154c0b1760 0x7f154c0b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.037+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 >> 172.21.15.187:0/1455412357 conn(0x7f154c0bab30 msgr2=0x7f154c14b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:58.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.037+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 shutdown_connections 2020-11-20T12:15:58.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.038+0000 7f154b59e700 1 -- 172.21.15.187:0/1455412357 wait complete. 2020-11-20T12:15:58.080 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:15:58.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.325+0000 7f39992f9700 1 Processor -- start 2020-11-20T12:15:58.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.325+0000 7f39992f9700 1 -- start start 2020-11-20T12:15:58.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.326+0000 7f39992f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f3994061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.326+0000 7f39992f9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3994066f70 0x7f39940b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.326+0000 7f39992f9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39940b1300 con 0x7f39940622c0 2020-11-20T12:15:58.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f39992f9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3994067730 con 0x7f3994066f70 2020-11-20T12:15:58.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f39992f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39940b15a0 con 0x7f39941484d0 2020-11-20T12:15:58.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f3998af8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3994066f70 0x7f39940b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f398b7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f39940622c0 0x7f39940b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47176/0 (socket says 172.21.15.187:47176) 2020-11-20T12:15:58.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f398b7fe700 1 -- 172.21.15.187:0/2692253393 learned_addr learned my addr 172.21.15.187:0/2692253393 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f3993fff700 1 --2- 172.21.15.187:0/2692253393 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f3994061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f3998af8700 1 -- 172.21.15.187:0/2692253393 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 msgr2=0x7f3994061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f3998af8700 1 --2- 172.21.15.187:0/2692253393 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f3994061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f3998af8700 1 -- 172.21.15.187:0/2692253393 >> v1:172.21.15.187:6789/0 conn(0x7f39940622c0 legacy=0x7f39940b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.327+0000 7f39937fe700 1 -- 172.21.15.187:0/2692253393 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027971406 0 0) 0x7f39940b1300 con 0x7f39940622c0 2020-11-20T12:15:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.328+0000 7f3998af8700 1 -- 172.21.15.187:0/2692253393 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39940b1840 con 0x7f3994066f70 2020-11-20T12:15:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.328+0000 7f3998af8700 1 --2- 172.21.15.187:0/2692253393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f39940b5d50 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 rx=0x7f398c009c90 tx=0x7f398c0091d0).ready entity=mon.1 client_cookie=9fdd5dcd7cffb56d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.328+0000 7f39937fe700 1 -- 172.21.15.187:0/2692253393 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c00bbc0 con 0x7f3994066f70 2020-11-20T12:15:58.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.328+0000 7f39937fe700 1 -- 172.21.15.187:0/2692253393 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c00bd20 con 0x7f3994066f70 2020-11-20T12:15:58.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.328+0000 7f39937fe700 1 -- 172.21.15.187:0/2692253393 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c011610 con 0x7f3994066f70 2020-11-20T12:15:58.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.329+0000 7f39992f9700 1 -- 172.21.15.187:0/2692253393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 msgr2=0x7f39940b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.329+0000 7f39992f9700 1 --2- 172.21.15.187:0/2692253393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f39940b5d50 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 rx=0x7f398c009c90 tx=0x7f398c0091d0).stop 2020-11-20T12:15:58.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.329+0000 7f39992f9700 1 -- 172.21.15.187:0/2692253393 shutdown_connections 2020-11-20T12:15:58.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.329+0000 7f39992f9700 1 --2- 172.21.15.187:0/2692253393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f39940b5d50 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.329+0000 7f39992f9700 1 --2- 172.21.15.187:0/2692253393 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f3994061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.329+0000 7f39992f9700 1 -- 172.21.15.187:0/2692253393 >> 172.21.15.187:0/2692253393 conn(0x7f39940bab30 msgr2=0x7f39940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:58.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.329+0000 7f39992f9700 1 -- 172.21.15.187:0/2692253393 shutdown_connections 2020-11-20T12:15:58.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.329+0000 7f39992f9700 1 -- 172.21.15.187:0/2692253393 wait complete. 2020-11-20T12:15:58.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.330+0000 7f39992f9700 1 Processor -- start 2020-11-20T12:15:58.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.331+0000 7f39992f9700 1 -- start start 2020-11-20T12:15:58.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.331+0000 7f39992f9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39940622c0 0x7f399415bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f39992f9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f3994141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f39992f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f399414df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f39992f9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399415c1e0 con 0x7f39940622c0 2020-11-20T12:15:58.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f39992f9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399415b310 con 0x7f3994066f70 2020-11-20T12:15:58.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f39992f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399415b5b0 con 0x7f39941484d0 2020-11-20T12:15:58.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f398b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f3994141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f398b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f3994141760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50576/0 (socket says 172.21.15.187:50576) 2020-11-20T12:15:58.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f398b7fe700 1 -- 172.21.15.187:0/3062344175 learned_addr learned my addr 172.21.15.187:0/3062344175 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:58.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f3993fff700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39940622c0 0x7f399415bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.332+0000 7f3998af8700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f399414df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f398b7fe700 1 -- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 msgr2=0x7f399414df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f398b7fe700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f399414df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f398b7fe700 1 -- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39940622c0 msgr2=0x7f399415bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f398b7fe700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39940622c0 0x7f399415bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f398b7fe700 1 -- 172.21.15.187:0/3062344175 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399415b850 con 0x7f3994066f70 2020-11-20T12:15:58.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f3998af8700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f399414df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:58.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f398b7fe700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f3994141760 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f397c00c6e0 tx=0x7f397c007a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:58.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f3990ff9700 1 -- 172.21.15.187:0/3062344175 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c007320 con 0x7f3994066f70 2020-11-20T12:15:58.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f3990ff9700 1 -- 172.21.15.187:0/3062344175 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397c007480 con 0x7f3994066f70 2020-11-20T12:15:58.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f3990ff9700 1 -- 172.21.15.187:0/3062344175 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c005800 con 0x7f3994066f70 2020-11-20T12:15:58.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399409e700 con 0x7f3994066f70 2020-11-20T12:15:58.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.333+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399414e3f0 con 0x7f3994066f70 2020-11-20T12:15:58.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.334+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399415b9e0 con 0x7f3994066f70 2020-11-20T12:15:58.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.335+0000 7f3990ff9700 1 -- 172.21.15.187:0/3062344175 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f397c010030 con 0x7f3994066f70 2020-11-20T12:15:58.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.336+0000 7f3990ff9700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.336+0000 7f3990ff9700 1 -- 172.21.15.187:0/3062344175 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f397c06f870 con 0x7f3994066f70 2020-11-20T12:15:58.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.336+0000 7f3993fff700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.337+0000 7f3993fff700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f398400d8f0 tx=0x7f3984008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:58.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.341+0000 7f3990ff9700 1 -- 172.21.15.187:0/3062344175 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f397c044ff0 con 0x7f3994066f70 2020-11-20T12:15:58.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.530+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3994061a20 con 0x7f3994066f70 2020-11-20T12:15:58.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.531+0000 7f3990ff9700 1 -- 172.21.15.187:0/3062344175 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+13636 (secure 0 0 0) 0x7f397c007b10 con 0x7f3994066f70 2020-11-20T12:15:58.534 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:58.534 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":112,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:15:57.471181+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:15:58.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 msgr2=0x7f39700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f398400d8f0 tx=0x7f3984008040).stop 2020-11-20T12:15:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 msgr2=0x7f3994141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f3994141760 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f397c00c6e0 tx=0x7f397c007a80).stop 2020-11-20T12:15:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 shutdown_connections 2020-11-20T12:15:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3970056d20 0x7f39700591e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39940622c0 0x7f399415bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39941484d0 0x7f399414df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 --2- 172.21.15.187:0/3062344175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3994066f70 0x7f3994141760 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 >> 172.21.15.187:0/3062344175 conn(0x7f39940bab30 msgr2=0x7f39940b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:58.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 shutdown_connections 2020-11-20T12:15:58.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.534+0000 7f39992f9700 1 -- 172.21.15.187:0/3062344175 wait complete. 2020-11-20T12:15:58.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:15:58.562 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:15:58.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.807+0000 7fecdef2c700 1 Processor -- start 2020-11-20T12:15:58.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.807+0000 7fecdef2c700 1 -- start start 2020-11-20T12:15:58.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.808+0000 7fecdef2c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd814d270 0x7fecd81573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.808+0000 7fecdef2c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd80658a0 con 0x7fecd80b1760 2020-11-20T12:15:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.809+0000 7fecdef2c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd8150390 con 0x7fecd80b1380 2020-11-20T12:15:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.809+0000 7fecdef2c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd81511e0 con 0x7fecd814d270 2020-11-20T12:15:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.809+0000 7fecddf2a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fecd80b1380 0x7fecd8150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47188/0 (socket says 172.21.15.187:47188) 2020-11-20T12:15:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.809+0000 7fecddf2a700 1 -- 172.21.15.187:0/1835143143 learned_addr learned my addr 172.21.15.187:0/1835143143 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:58.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.809+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797917844 0 0) 0x7fecd8150390 con 0x7fecd80b1380 2020-11-20T12:15:58.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.809+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc0003240 con 0x7fecd80b1380 2020-11-20T12:15:58.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.809+0000 7fecde72b700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd814d270 0x7fecd81573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 304868061 0 0) 0x7fecc0003240 con 0x7fecd80b1380 2020-11-20T12:15:58.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd814d270 msgr2=0x7fecd81573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:58.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd814d270 0x7fecd81573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 >> v1:172.21.15.5:6789/0 conn(0x7fecd80b1760 legacy=0x7fecd8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd8150630 con 0x7fecd80b1380 2020-11-20T12:15:58.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fecd4001860 con 0x7fecd80b1380 2020-11-20T12:15:58.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 >> v1:172.21.15.187:6789/0 conn(0x7fecd80b1380 legacy=0x7fecd8150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc0003730 0x7fecc0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecc00044e0 0x7fecc0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecc0004e50 0x7fecc000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd81511e0 con 0x7fecc0004e50 2020-11-20T12:15:58.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc00041c0 con 0x7fecc0003730 2020-11-20T12:15:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.810+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc000bc20 con 0x7fecc00044e0 2020-11-20T12:15:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecddf2a700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc0003730 0x7fecc0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecdd729700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecc00044e0 0x7fecc0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecde72b700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecc0004e50 0x7fecc000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecddf2a700 1 -- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecc00044e0 msgr2=0x7fecc0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:58.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecddf2a700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecc00044e0 0x7fecc0004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecddf2a700 1 -- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecc0004e50 msgr2=0x7fecc000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:58.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecddf2a700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecc0004e50 0x7fecc000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecddf2a700 1 -- 172.21.15.187:0/1835143143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0003240 con 0x7fecc0003730 2020-11-20T12:15:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecdd729700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecc00044e0 0x7fecc0004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.811+0000 7fecde72b700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecc0004e50 0x7fecc000b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.812+0000 7fecddf2a700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc0003730 0x7fecc0003b50 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7fecd400c620 tx=0x7fecd4007b50).ready entity=mon.1 client_cookie=2f24f8a98bef5127 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:58.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.812+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd4005b80 con 0x7fecc0003730 2020-11-20T12:15:58.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.812+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd4005ce0 con 0x7fecc0003730 2020-11-20T12:15:58.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.812+0000 7fecdcf28700 1 -- 172.21.15.187:0/1835143143 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd40072e0 con 0x7fecc0003730 2020-11-20T12:15:58.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.812+0000 7fecdef2c700 1 -- 172.21.15.187:0/1835143143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc0003730 msgr2=0x7fecc0003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.812+0000 7fecdef2c700 1 --2- 172.21.15.187:0/1835143143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc0003730 0x7fecc0003b50 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7fecd400c620 tx=0x7fecd4007b50).stop 2020-11-20T12:15:58.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.813+0000 7fecde72b700 1 -- 172.21.15.187:0/1835143143 reap_dead start 2020-11-20T12:15:58.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.813+0000 7fecdef2c700 1 -- 172.21.15.187:0/1835143143 shutdown_connections 2020-11-20T12:15:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.813+0000 7fecdef2c700 1 -- 172.21.15.187:0/1835143143 >> 172.21.15.187:0/1835143143 conn(0x7fecd80bab30 msgr2=0x7fecd80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.813+0000 7fecdef2c700 1 -- 172.21.15.187:0/1835143143 shutdown_connections 2020-11-20T12:15:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.813+0000 7fecdef2c700 1 -- 172.21.15.187:0/1835143143 wait complete. 2020-11-20T12:15:58.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.814+0000 7fecdef2c700 1 Processor -- start 2020-11-20T12:15:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.815+0000 7fecdef2c700 1 -- start start 2020-11-20T12:15:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.815+0000 7fecdef2c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd8160000 0x7fecd809e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.815+0000 7fecdef2c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd80b60e0 0x7fecd81603e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.815+0000 7fecdef2c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd80a0670 0x7fecd809d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.815+0000 7fecdef2c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd809d5a0 con 0x7fecd8160000 2020-11-20T12:15:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.815+0000 7fecdef2c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd81609f0 con 0x7fecd80a0670 2020-11-20T12:15:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.815+0000 7fecdef2c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd8160c90 con 0x7fecd80b60e0 2020-11-20T12:15:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.816+0000 7fecde72b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd80a0670 0x7fecd809d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.816+0000 7fecde72b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd80a0670 0x7fecd809d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50598/0 (socket says 172.21.15.187:50598) 2020-11-20T12:15:58.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.816+0000 7fecde72b700 1 -- 172.21.15.187:0/659039120 learned_addr learned my addr 172.21.15.187:0/659039120 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:58.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.816+0000 7fecddf2a700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd8160000 0x7fecd809e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.816+0000 7fecdd729700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd80b60e0 0x7fecd81603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecde72b700 1 -- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd80b60e0 msgr2=0x7fecd81603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecde72b700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd80b60e0 0x7fecd81603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecde72b700 1 -- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd8160000 msgr2=0x7fecd809e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:58.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecde72b700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd8160000 0x7fecd809e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:58.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecde72b700 1 -- 172.21.15.187:0/659039120 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd809f350 con 0x7fecd80a0670 2020-11-20T12:15:58.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecddf2a700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd8160000 0x7fecd809e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:58.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecde72b700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd80a0670 0x7fecd809d0b0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 rx=0x7fecd80639b0 tx=0x7fecd0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecc5ffb700 1 -- 172.21.15.187:0/659039120 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0018070 con 0x7fecd80a0670 2020-11-20T12:15:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecc5ffb700 1 -- 172.21.15.187:0/659039120 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd0013610 con 0x7fecd80a0670 2020-11-20T12:15:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecc5ffb700 1 -- 172.21.15.187:0/659039120 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd00139f0 con 0x7fecd80a0670 2020-11-20T12:15:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd809f5a0 con 0x7fecd80a0670 2020-11-20T12:15:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.817+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd809fa40 con 0x7fecd80a0670 2020-11-20T12:15:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.819+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd8061a20 con 0x7fecd80a0670 2020-11-20T12:15:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.819+0000 7fecc5ffb700 1 -- 172.21.15.187:0/659039120 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fecd001d410 con 0x7fecd80a0670 2020-11-20T12:15:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.820+0000 7fecc5ffb700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecb0056d20 0x7fecb00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:58.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.820+0000 7fecc5ffb700 1 -- 172.21.15.187:0/659039120 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fecd006f840 con 0x7fecd80a0670 2020-11-20T12:15:58.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.820+0000 7fecddf2a700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecb0056d20 0x7fecb00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:58.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.821+0000 7fecddf2a700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecb0056d20 0x7fecb00591e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7fecd805e630 tx=0x7fecd4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:58.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:58.824+0000 7fecc5ffb700 1 -- 172.21.15.187:0/659039120 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fecd0044e50 con 0x7fecd80a0670 2020-11-20T12:15:59.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.011+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fecd80446d0 con 0x7fecd80a0670 2020-11-20T12:15:59.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.013+0000 7fecc5ffb700 1 -- 172.21.15.187:0/659039120 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v112) v1 ==== 110+0+10 (secure 0 0 0) 0x7fecd0048470 con 0x7fecd80a0670 2020-11-20T12:15:59.016 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:15:59.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.015+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecb0056d20 msgr2=0x7fecb00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.015+0000 7fecdef2c700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecb0056d20 0x7fecb00591e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7fecd805e630 tx=0x7fecd4015000).stop 2020-11-20T12:15:59.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.015+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd80a0670 msgr2=0x7fecd809d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.015+0000 7fecdef2c700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd80a0670 0x7fecd809d0b0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 rx=0x7fecd80639b0 tx=0x7fecd0008040).stop 2020-11-20T12:15:59.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.016+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 shutdown_connections 2020-11-20T12:15:59.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.016+0000 7fecdef2c700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecb0056d20 0x7fecb00591e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.016+0000 7fecdef2c700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd8160000 0x7fecd809e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.016+0000 7fecdef2c700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd80a0670 0x7fecd809d0b0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.016+0000 7fecdef2c700 1 --2- 172.21.15.187:0/659039120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd80b60e0 0x7fecd81603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.016+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 >> 172.21.15.187:0/659039120 conn(0x7fecd80bab30 msgr2=0x7fecd81492c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:59.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.016+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 shutdown_connections 2020-11-20T12:15:59.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.016+0000 7fecdef2c700 1 -- 172.21.15.187:0/659039120 wait complete. 2020-11-20T12:15:59.043 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:15:59.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.293+0000 7f928aa23700 1 Processor -- start 2020-11-20T12:15:59.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.293+0000 7f928aa23700 1 -- start start 2020-11-20T12:15:59.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.294+0000 7f928aa23700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f92840b6650 0x7f92840afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.294+0000 7f928aa23700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92840b1300 con 0x7f92841484d0 2020-11-20T12:15:59.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.295+0000 7f928aa23700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92840b15a0 con 0x7f9284066f70 2020-11-20T12:15:59.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.295+0000 7f928aa23700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9284067730 con 0x7f92840b6650 2020-11-20T12:15:59.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.295+0000 7f928a222700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9284066f70 0x7f92840aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47210/0 (socket says 172.21.15.187:47210) 2020-11-20T12:15:59.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.295+0000 7f928a222700 1 -- 172.21.15.187:0/1818649800 learned_addr learned my addr 172.21.15.187:0/1818649800 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:15:59.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.295+0000 7f9289220700 1 --2- 172.21.15.187:0/1818649800 >> v2:172.21.15.187:3300/0 conn(0x7f92840b6650 0x7f92840afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141747849 0 0) 0x7f92840b15a0 con 0x7f9284066f70 2020-11-20T12:15:59.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c003240 con 0x7f9284066f70 2020-11-20T12:15:59.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1885247192 0 0) 0x7f926c003240 con 0x7f9284066f70 2020-11-20T12:15:59.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 >> v2:172.21.15.187:3300/0 conn(0x7f92840b6650 msgr2=0x7f92840afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:59.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 --2- 172.21.15.187:0/1818649800 >> v2:172.21.15.187:3300/0 conn(0x7f92840b6650 0x7f92840afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 >> v1:172.21.15.5:6789/0 conn(0x7f92841484d0 legacy=0x7f9284061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92840b1840 con 0x7f9284066f70 2020-11-20T12:15:59.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f927c0014e0 con 0x7f9284066f70 2020-11-20T12:15:59.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 >> v1:172.21.15.187:6789/0 conn(0x7f9284066f70 legacy=0x7f92840aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f926c003780 0x7f926c003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f926c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f926c004e50 0x7f926c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284067730 con 0x7f926c004e50 2020-11-20T12:15:59.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c005300 con 0x7f926c0044e0 2020-11-20T12:15:59.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.296+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c00de30 con 0x7f926c003780 2020-11-20T12:15:59.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f9289220700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f926c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f928a222700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f926c004e50 0x7f926c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f9289a21700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f926c003780 0x7f926c003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f9289220700 1 -- 172.21.15.187:0/1818649800 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f926c003780 msgr2=0x7f926c003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:59.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f9289220700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f926c003780 0x7f926c003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f9289220700 1 -- 172.21.15.187:0/1818649800 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f926c004e50 msgr2=0x7f926c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:59.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f9289220700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f926c004e50 0x7f926c00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f9289220700 1 -- 172.21.15.187:0/1818649800 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c003240 con 0x7f926c0044e0 2020-11-20T12:15:59.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.297+0000 7f928a222700 1 -- 172.21.15.187:0/1818649800 reap_dead start 2020-11-20T12:15:59.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.298+0000 7f9289220700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f926c004960 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 rx=0x7f927400bc00 tx=0x7f9274006df0).ready entity=mon.1 client_cookie=8a2cadafcdcd5c81 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:59.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.298+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400b310 con 0x7f926c0044e0 2020-11-20T12:15:59.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.298+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927400ea90 con 0x7f926c0044e0 2020-11-20T12:15:59.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.298+0000 7f9288a1f700 1 -- 172.21.15.187:0/1818649800 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400ee10 con 0x7f926c0044e0 2020-11-20T12:15:59.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.299+0000 7f928aa23700 1 -- 172.21.15.187:0/1818649800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 msgr2=0x7f926c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.299+0000 7f928aa23700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f926c004960 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 rx=0x7f927400bc00 tx=0x7f9274006df0).stop 2020-11-20T12:15:59.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.299+0000 7f928aa23700 1 -- 172.21.15.187:0/1818649800 shutdown_connections 2020-11-20T12:15:59.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.299+0000 7f928aa23700 1 --2- 172.21.15.187:0/1818649800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f926c004960 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.299+0000 7f928aa23700 1 -- 172.21.15.187:0/1818649800 >> 172.21.15.187:0/1818649800 conn(0x7f92840bab30 msgr2=0x7f92840baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:59.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.299+0000 7f928aa23700 1 -- 172.21.15.187:0/1818649800 shutdown_connections 2020-11-20T12:15:59.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.299+0000 7f928aa23700 1 -- 172.21.15.187:0/1818649800 wait complete. 2020-11-20T12:15:59.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.301+0000 7f928aa23700 1 Processor -- start 2020-11-20T12:15:59.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.301+0000 7f928aa23700 1 -- start start 2020-11-20T12:15:59.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.302+0000 7f928aa23700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f9284157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.302+0000 7f928aa23700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92840a05c0 0x7f92840b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.302+0000 7f928aa23700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92840b2bc0 0x7f928409c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.302+0000 7f928aa23700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92840b4ad0 con 0x7f92840a05c0 2020-11-20T12:15:59.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.302+0000 7f928aa23700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284157aa0 con 0x7f926c0044e0 2020-11-20T12:15:59.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.302+0000 7f928aa23700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9284157d40 con 0x7f92840b2bc0 2020-11-20T12:15:59.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289a21700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f9284157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289a21700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f9284157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50614/0 (socket says 172.21.15.187:50614) 2020-11-20T12:15:59.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289a21700 1 -- 172.21.15.187:0/3243684361 learned_addr learned my addr 172.21.15.187:0/3243684361 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:59.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289a21700 1 -- 172.21.15.187:0/3243684361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92840b2bc0 msgr2=0x7f928409c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f928a222700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92840b2bc0 0x7f928409c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289220700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92840a05c0 0x7f92840b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289a21700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92840b2bc0 0x7f928409c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289a21700 1 -- 172.21.15.187:0/3243684361 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92840a05c0 msgr2=0x7f92840b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289a21700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92840a05c0 0x7f92840b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.303+0000 7f9289a21700 1 -- 172.21.15.187:0/3243684361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928409c270 con 0x7f926c0044e0 2020-11-20T12:15:59.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.304+0000 7f9289a21700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f9284157160 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f928000eae0 tx=0x7f9280009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:59.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.304+0000 7f9271ffb700 1 -- 172.21.15.187:0/3243684361 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280003320 con 0x7f926c0044e0 2020-11-20T12:15:59.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.304+0000 7f9271ffb700 1 -- 172.21.15.187:0/3243684361 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928000c6f0 con 0x7f926c0044e0 2020-11-20T12:15:59.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.304+0000 7f9271ffb700 1 -- 172.21.15.187:0/3243684361 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928000ca70 con 0x7f926c0044e0 2020-11-20T12:15:59.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.304+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9284157650 con 0x7f926c0044e0 2020-11-20T12:15:59.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.304+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928409dd20 con 0x7f926c0044e0 2020-11-20T12:15:59.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.306+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9284061a20 con 0x7f926c0044e0 2020-11-20T12:15:59.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.306+0000 7f9271ffb700 1 -- 172.21.15.187:0/3243684361 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f92800014e0 con 0x7f926c0044e0 2020-11-20T12:15:59.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.307+0000 7f9271ffb700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f925c056d20 0x7f925c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.307+0000 7f9271ffb700 1 -- 172.21.15.187:0/3243684361 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f928006fba0 con 0x7f926c0044e0 2020-11-20T12:15:59.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.307+0000 7f9289220700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f925c056d20 0x7f925c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.308+0000 7f9289220700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f925c056d20 0x7f925c0591e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f928409eff0 tx=0x7f9274019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:59.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.311+0000 7f9271ffb700 1 -- 172.21.15.187:0/3243684361 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f92800452c0 con 0x7f926c0044e0 2020-11-20T12:15:59.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.498+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f92841488b0 con 0x7f926c0044e0 2020-11-20T12:15:59.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.504+0000 7f9271ffb700 1 -- 172.21.15.187:0/3243684361 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v112) v1 ==== 92+0+10 (secure 0 0 0) 0x7f92800488e0 con 0x7f926c0044e0 2020-11-20T12:15:59.507 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:15:59.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.507+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f925c056d20 msgr2=0x7f925c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.507+0000 7f928aa23700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f925c056d20 0x7f925c0591e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f928409eff0 tx=0x7f9274019000).stop 2020-11-20T12:15:59.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.507+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 msgr2=0x7f9284157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.507+0000 7f928aa23700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f9284157160 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f928000eae0 tx=0x7f9280009040).stop 2020-11-20T12:15:59.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.508+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 shutdown_connections 2020-11-20T12:15:59.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.508+0000 7f928aa23700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f925c056d20 0x7f925c0591e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.508+0000 7f928aa23700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92840b2bc0 0x7f928409c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.508+0000 7f928aa23700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f926c0044e0 0x7f9284157160 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.508+0000 7f928aa23700 1 --2- 172.21.15.187:0/3243684361 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92840a05c0 0x7f92840b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.508+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 >> 172.21.15.187:0/3243684361 conn(0x7f92840bab30 msgr2=0x7f9284147360 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:59.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.508+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 shutdown_connections 2020-11-20T12:15:59.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.508+0000 7f928aa23700 1 -- 172.21.15.187:0/3243684361 wait complete. 2020-11-20T12:15:59.535 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:15:59.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.770+0000 7faf0ecc5700 1 Processor -- start 2020-11-20T12:15:59.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.771+0000 7faf0ecc5700 1 -- start start 2020-11-20T12:15:59.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.772+0000 7faf0ecc5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf08065190 0x7faf08065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.772+0000 7faf0ecc5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0809ca90 0x7faf0809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.772+0000 7faf0ecc5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf080b13e0 con 0x7faf08065f90 2020-11-20T12:15:59.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.772+0000 7faf0ecc5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0814ab30 con 0x7faf08065190 2020-11-20T12:15:59.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.772+0000 7faf0ecc5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf08062c10 con 0x7faf0809ca90 2020-11-20T12:15:59.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.773+0000 7faf0dcc3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf08065190 0x7faf08065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.773+0000 7faf0e4c4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0809ca90 0x7faf0809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.773+0000 7faf0dcc3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf08065190 0x7faf08065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:42888/0 (socket says 172.21.15.187:42888) 2020-11-20T12:15:59.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.773+0000 7faf0d4c2700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7faf08065f90 0x7faf08143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:40738/0 (socket says 172.21.15.187:40738) 2020-11-20T12:15:59.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.773+0000 7faf0dcc3700 1 -- 172.21.15.187:0/3699811057 learned_addr learned my addr 172.21.15.187:0/3699811057 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:59.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.773+0000 7faf0e4c4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0809ca90 0x7faf0809cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:51514/0 (socket says 172.21.15.187:51514) 2020-11-20T12:15:59.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.774+0000 7faf0ccc1700 1 -- 172.21.15.187:0/3699811057 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184651659 0 0) 0x7faf080b13e0 con 0x7faf08065f90 2020-11-20T12:15:59.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.774+0000 7faf0ccc1700 1 -- 172.21.15.187:0/3699811057 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef0003240 con 0x7faf08065f90 2020-11-20T12:15:59.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.774+0000 7faf0dcc3700 1 -- 172.21.15.187:0/3699811057 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0809ca90 msgr2=0x7faf0809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:59.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.774+0000 7faf0dcc3700 1 --2- 172.21.15.187:0/3699811057 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0809ca90 0x7faf0809cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.774+0000 7faf0dcc3700 1 -- 172.21.15.187:0/3699811057 >> v1:172.21.15.5:6789/0 conn(0x7faf08065f90 legacy=0x7faf08143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.774+0000 7faf0dcc3700 1 -- 172.21.15.187:0/3699811057 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf080b1640 con 0x7faf08065190 2020-11-20T12:15:59.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.774+0000 7faf0e4c4700 1 --2- 172.21.15.187:0/3699811057 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0809ca90 0x7faf0809cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:15:59.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.775+0000 7faf0dcc3700 1 --2- 172.21.15.187:0/3699811057 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065190 0x7faf08065570 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7faf04006fd0 tx=0x7faf04006eb0).ready entity=mon.0 client_cookie=e09de6c9bac18b48 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:59.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.775+0000 7faf0ccc1700 1 -- 172.21.15.187:0/3699811057 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf040097b0 con 0x7faf08065190 2020-11-20T12:15:59.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ccc1700 1 -- 172.21.15.187:0/3699811057 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf04009910 con 0x7faf08065190 2020-11-20T12:15:59.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ccc1700 1 -- 172.21.15.187:0/3699811057 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04009c90 con 0x7faf08065190 2020-11-20T12:15:59.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ecc5700 1 -- 172.21.15.187:0/3699811057 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065190 msgr2=0x7faf08065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/3699811057 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065190 0x7faf08065570 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7faf04006fd0 tx=0x7faf04006eb0).stop 2020-11-20T12:15:59.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ecc5700 1 -- 172.21.15.187:0/3699811057 shutdown_connections 2020-11-20T12:15:59.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/3699811057 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0809ca90 0x7faf0809cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/3699811057 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065190 0x7faf08065570 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ecc5700 1 -- 172.21.15.187:0/3699811057 >> 172.21.15.187:0/3699811057 conn(0x7faf080bab30 msgr2=0x7faf080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:59.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ecc5700 1 -- 172.21.15.187:0/3699811057 shutdown_connections 2020-11-20T12:15:59.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.776+0000 7faf0ecc5700 1 -- 172.21.15.187:0/3699811057 wait complete. 2020-11-20T12:15:59.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.778+0000 7faf0ecc5700 1 Processor -- start 2020-11-20T12:15:59.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.778+0000 7faf0ecc5700 1 -- start start 2020-11-20T12:15:59.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.779+0000 7faf0ecc5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065f90 0x7faf08068260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.779+0000 7faf0ecc5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf0809ca90 0x7faf080b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.779+0000 7faf0ecc5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0815bcc0 0x7faf0815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.779+0000 7faf0ecc5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08142bd0 con 0x7faf08065f90 2020-11-20T12:15:59.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.779+0000 7faf0ecc5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf080b25a0 con 0x7faf0809ca90 2020-11-20T12:15:59.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.779+0000 7faf0ecc5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf080b4700 con 0x7faf0815bcc0 2020-11-20T12:15:59.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.780+0000 7faf0d4c2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf0809ca90 0x7faf080b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.780+0000 7faf0d4c2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf0809ca90 0x7faf080b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50630/0 (socket says 172.21.15.187:50630) 2020-11-20T12:15:59.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.780+0000 7faf0d4c2700 1 -- 172.21.15.187:0/2687789493 learned_addr learned my addr 172.21.15.187:0/2687789493 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:15:59.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.780+0000 7faf0dcc3700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065f90 0x7faf08068260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.780+0000 7faf0e4c4700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0815bcc0 0x7faf0815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faf0d4c2700 1 -- 172.21.15.187:0/2687789493 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0815bcc0 msgr2=0x7faf0815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faf0d4c2700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0815bcc0 0x7faf0815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faf0d4c2700 1 -- 172.21.15.187:0/2687789493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065f90 msgr2=0x7faf08068260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faf0d4c2700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065f90 0x7faf08068260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faf0d4c2700 1 -- 172.21.15.187:0/2687789493 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf080b49a0 con 0x7faf0809ca90 2020-11-20T12:15:59.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faf0d4c2700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf0809ca90 0x7faf080b20b0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7faef800c710 tx=0x7faef8007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:59.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faef5ffb700 1 -- 172.21.15.187:0/2687789493 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8007320 con 0x7faf0809ca90 2020-11-20T12:15:59.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faef5ffb700 1 -- 172.21.15.187:0/2687789493 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef8005530 con 0x7faf0809ca90 2020-11-20T12:15:59.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0809eb80 con 0x7faf0809ca90 2020-11-20T12:15:59.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faef5ffb700 1 -- 172.21.15.187:0/2687789493 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef80058b0 con 0x7faf0809ca90 2020-11-20T12:15:59.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.781+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0815c840 con 0x7faf0809ca90 2020-11-20T12:15:59.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.782+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0814a250 con 0x7faf0809ca90 2020-11-20T12:15:59.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.783+0000 7faef5ffb700 1 -- 172.21.15.187:0/2687789493 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faef8010030 con 0x7faf0809ca90 2020-11-20T12:15:59.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.783+0000 7faef5ffb700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee0056d20 0x7faee00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:59.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.784+0000 7faef5ffb700 1 -- 172.21.15.187:0/2687789493 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5253+0+0 (secure 0 0 0) 0x7faef806fa10 con 0x7faf0809ca90 2020-11-20T12:15:59.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.784+0000 7faf0dcc3700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee0056d20 0x7faee00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:59.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.785+0000 7faf0dcc3700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee0056d20 0x7faee00591e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7faf080632e0 tx=0x7faf04015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:59.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.789+0000 7faef5ffb700 1 -- 172.21.15.187:0/2687789493 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faef8045190 con 0x7faf0809ca90 2020-11-20T12:15:59.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.959+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7faf08061a20 con 0x7faf0809ca90 2020-11-20T12:15:59.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.961+0000 7faef5ffb700 1 -- 172.21.15.187:0/2687789493 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v112) v1 ==== 84+0+143 (secure 0 0 0) 0x7faef80487b0 con 0x7faf0809ca90 2020-11-20T12:15:59.964 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:15:59.964 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:15:59.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:59.965 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:15:59.965 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:15:59.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:15:59.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:15:59.966 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:15:59.966 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:15:59.966 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:15:59.967 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:15:59.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee0056d20 msgr2=0x7faee00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee0056d20 0x7faee00591e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7faf080632e0 tx=0x7faf04015000).stop 2020-11-20T12:15:59.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf0809ca90 msgr2=0x7faf080b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:59.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf0809ca90 0x7faf080b20b0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7faef800c710 tx=0x7faef8007aa0).stop 2020-11-20T12:15:59.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 shutdown_connections 2020-11-20T12:15:59.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee0056d20 0x7faee00591e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf08065f90 0x7faf08068260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf0815bcc0 0x7faf0815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 --2- 172.21.15.187:0/2687789493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf0809ca90 0x7faf080b20b0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:59.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 >> 172.21.15.187:0/2687789493 conn(0x7faf080bab30 msgr2=0x7faf0809bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:59.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 shutdown_connections 2020-11-20T12:15:59.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:15:59.963+0000 7faf0ecc5700 1 -- 172.21.15.187:0/2687789493 wait complete. 2020-11-20T12:15:59.987 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:15:59.988 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:16:00.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.224+0000 7fb8a60d1700 1 Processor -- start 2020-11-20T12:16:00.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.224+0000 7fb8a60d1700 1 -- start start 2020-11-20T12:16:00.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.225+0000 7fb8a60d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a01484d0 0x7fb8a0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:00.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.225+0000 7fb8a60d1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8a0066f70 0x7fb8a00b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:00.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.225+0000 7fb8a60d1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00b1300 con 0x7fb8a00622c0 2020-11-20T12:16:00.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.225+0000 7fb8a60d1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a0067730 con 0x7fb8a0066f70 2020-11-20T12:16:00.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.225+0000 7fb8a60d1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00b15a0 con 0x7fb8a01484d0 2020-11-20T12:16:00.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.226+0000 7fb8a58d0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8a0066f70 0x7fb8a00b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:00.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.226+0000 7fb8a50cf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a01484d0 0x7fb8a0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:00.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.226+0000 7fb8a58d0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8a0066f70 0x7fb8a00b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:42904/0 (socket says 172.21.15.187:42904) 2020-11-20T12:16:00.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.226+0000 7fb8a58d0700 1 -- 172.21.15.187:0/2730551520 learned_addr learned my addr 172.21.15.187:0/2730551520 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:00.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.227+0000 7fb8a58d0700 1 -- 172.21.15.187:0/2730551520 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a01484d0 msgr2=0x7fb8a0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:00.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.227+0000 7fb8a58d0700 1 --2- 172.21.15.187:0/2730551520 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a01484d0 0x7fb8a0061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:00.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.227+0000 7fb8a58d0700 1 -- 172.21.15.187:0/2730551520 >> v1:172.21.15.5:6789/0 conn(0x7fb8a00622c0 legacy=0x7fb8a00b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:00.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.227+0000 7fb8a58d0700 1 -- 172.21.15.187:0/2730551520 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00b1840 con 0x7fb8a0066f70 2020-11-20T12:16:00.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.228+0000 7fb8a58d0700 1 --2- 172.21.15.187:0/2730551520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 0x7fb8a00b5d50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fb898009d50 tx=0x7fb898009290).ready entity=mon.0 client_cookie=d20b74f3142b8f2f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:00.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.228+0000 7fb88ffff700 1 -- 172.21.15.187:0/2730551520 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89800bbc0 con 0x7fb8a0066f70 2020-11-20T12:16:00.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.229+0000 7fb88ffff700 1 -- 172.21.15.187:0/2730551520 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89800bd20 con 0x7fb8a0066f70 2020-11-20T12:16:00.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.229+0000 7fb88ffff700 1 -- 172.21.15.187:0/2730551520 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898011610 con 0x7fb8a0066f70 2020-11-20T12:16:00.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.229+0000 7fb8a60d1700 1 -- 172.21.15.187:0/2730551520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 msgr2=0x7fb8a00b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:00.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.229+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/2730551520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 0x7fb8a00b5d50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fb898009d50 tx=0x7fb898009290).stop 2020-11-20T12:16:00.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.230+0000 7fb8a60d1700 1 -- 172.21.15.187:0/2730551520 shutdown_connections 2020-11-20T12:16:00.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.230+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/2730551520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 0x7fb8a00b5d50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:00.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.230+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/2730551520 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a01484d0 0x7fb8a0061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:00.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.230+0000 7fb8a60d1700 1 -- 172.21.15.187:0/2730551520 >> 172.21.15.187:0/2730551520 conn(0x7fb8a00bab30 msgr2=0x7fb8a00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:00.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.230+0000 7fb8a60d1700 1 -- 172.21.15.187:0/2730551520 shutdown_connections 2020-11-20T12:16:00.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.230+0000 7fb8a60d1700 1 -- 172.21.15.187:0/2730551520 wait complete. 2020-11-20T12:16:00.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.231+0000 7fb8a60d1700 1 Processor -- start 2020-11-20T12:16:00.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.232+0000 7fb8a60d1700 1 -- start start 2020-11-20T12:16:00.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.232+0000 7fb8a60d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a00622c0 0x7fb8a00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:00.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a60d1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 0x7fb8a0158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:00.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a60d1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a01484d0 0x7fb8a009b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a60d1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a009e870 con 0x7fb8a0066f70 2020-11-20T12:16:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a60d1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a0157f10 con 0x7fb8a01484d0 2020-11-20T12:16:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a60d1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a01581b0 con 0x7fb8a00622c0 2020-11-20T12:16:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a58d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a01484d0 0x7fb8a009b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a58d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a01484d0 0x7fb8a009b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50646/0 (socket says 172.21.15.187:50646) 2020-11-20T12:16:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a58d0700 1 -- 172.21.15.187:0/1557776280 learned_addr learned my addr 172.21.15.187:0/1557776280 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a50cf700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a00622c0 0x7fb8a00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.233+0000 7fb8a48ce700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 0x7fb8a0158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:00.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb8a58d0700 1 -- 172.21.15.187:0/1557776280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a00622c0 msgr2=0x7fb8a00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:00.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb8a58d0700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a00622c0 0x7fb8a00b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:00.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb8a58d0700 1 -- 172.21.15.187:0/1557776280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 msgr2=0x7fb8a0158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:00.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb8a58d0700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 0x7fb8a0158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:00.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb8a58d0700 1 -- 172.21.15.187:0/1557776280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00b4310 con 0x7fb8a01484d0 2020-11-20T12:16:00.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb8a58d0700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a01484d0 0x7fb8a009b7f0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 rx=0x7fb898009fd0 tx=0x7fb898016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:00.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb88d7fa700 1 -- 172.21.15.187:0/1557776280 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89800bbc0 con 0x7fb8a01484d0 2020-11-20T12:16:00.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb88d7fa700 1 -- 172.21.15.187:0/1557776280 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89800be90 con 0x7fb8a01484d0 2020-11-20T12:16:00.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a009e4f0 con 0x7fb8a01484d0 2020-11-20T12:16:00.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.234+0000 7fb88d7fa700 1 -- 172.21.15.187:0/1557776280 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898010e50 con 0x7fb8a01484d0 2020-11-20T12:16:00.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.235+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a009d1d0 con 0x7fb8a01484d0 2020-11-20T12:16:00.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.236+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a009e9e0 con 0x7fb8a01484d0 2020-11-20T12:16:00.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.236+0000 7fb88d7fa700 1 -- 172.21.15.187:0/1557776280 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb898017070 con 0x7fb8a01484d0 2020-11-20T12:16:00.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.237+0000 7fb88d7fa700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb878056d20 0x7fb8780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:00.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.237+0000 7fb88d7fa700 1 -- 172.21.15.187:0/1557776280 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb898071e30 con 0x7fb8a01484d0 2020-11-20T12:16:00.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.237+0000 7fb8a50cf700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb878056d20 0x7fb8780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:00.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.238+0000 7fb8a50cf700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb878056d20 0x7fb8780591e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fb8a009f390 tx=0x7fb89c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:00.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.241+0000 7fb88d7fa700 1 -- 172.21.15.187:0/1557776280 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb8a009e9e0 con 0x7fb8a01484d0 2020-11-20T12:16:00.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:00.429+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fb8a00b44a0 con 0x7fb8a01484d0 2020-11-20T12:16:01.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.305+0000 7fb88d7fa700 1 -- 172.21.15.187:0/1557776280 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v115) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb8a00b44a0 con 0x7fb8a01484d0 2020-11-20T12:16:01.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb878056d20 msgr2=0x7fb8780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:01.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb878056d20 0x7fb8780591e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fb8a009f390 tx=0x7fb89c008040).stop 2020-11-20T12:16:01.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a01484d0 msgr2=0x7fb8a009b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:01.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a01484d0 0x7fb8a009b7f0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 rx=0x7fb898009fd0 tx=0x7fb898016000).stop 2020-11-20T12:16:01.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 shutdown_connections 2020-11-20T12:16:01.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb878056d20 0x7fb8780591e0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:01.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8a01484d0 0x7fb8a009b7f0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:01.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8a00622c0 0x7fb8a00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:01.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 --2- 172.21.15.187:0/1557776280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8a0066f70 0x7fb8a0158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:01.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.310+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 >> 172.21.15.187:0/1557776280 conn(0x7fb8a00bab30 msgr2=0x7fb8a0065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:01.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.311+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 shutdown_connections 2020-11-20T12:16:01.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:01.311+0000 7fb8a60d1700 1 -- 172.21.15.187:0/1557776280 wait complete. 2020-11-20T12:16:01.939 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:16:01.940 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:16:02.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.196+0000 7f95e1e90700 1 Processor -- start 2020-11-20T12:16:02.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.196+0000 7f95e1e90700 1 -- start start 2020-11-20T12:16:02.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.198+0000 7f95e1e90700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95dc0b1380 0x7f95dc150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:02.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.198+0000 7f95e1e90700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95dc0658a0 con 0x7f95dc0b1760 2020-11-20T12:16:02.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.198+0000 7f95e1e90700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95dc150390 con 0x7f95dc14c260 2020-11-20T12:16:02.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.198+0000 7f95e1e90700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95dc150630 con 0x7f95dc0b1380 2020-11-20T12:16:02.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.198+0000 7f95e168f700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f95dc14c260 0x7f95dc14c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47262/0 (socket says 172.21.15.187:47262) 2020-11-20T12:16:02.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.198+0000 7f95e168f700 1 -- 172.21.15.187:0/141106556 learned_addr learned my addr 172.21.15.187:0/141106556 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:02.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.198+0000 7f95e0e8e700 1 --2- 172.21.15.187:0/141106556 >> v2:172.21.15.187:3300/0 conn(0x7f95dc0b1380 0x7f95dc150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:02.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.199+0000 7f95d37fe700 1 -- 172.21.15.187:0/141106556 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477674728 0 0) 0x7f95dc150390 con 0x7f95dc14c260 2020-11-20T12:16:02.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.199+0000 7f95d37fe700 1 -- 172.21.15.187:0/141106556 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c8003240 con 0x7f95dc14c260 2020-11-20T12:16:02.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.199+0000 7f95e0e8e700 1 -- 172.21.15.187:0/141106556 >> v1:172.21.15.187:6789/0 conn(0x7f95dc14c260 legacy=0x7f95dc14c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:02.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.199+0000 7f95e0e8e700 1 -- 172.21.15.187:0/141106556 >> v1:172.21.15.5:6789/0 conn(0x7f95dc0b1760 legacy=0x7f95dc065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:02.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.199+0000 7f95e0e8e700 1 -- 172.21.15.187:0/141106556 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc068510 con 0x7f95dc0b1380 2020-11-20T12:16:02.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.199+0000 7f95e0e8e700 1 --2- 172.21.15.187:0/141106556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1380 0x7f95dc150ad0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 rx=0x7f95d8009c90 tx=0x7f95d80091d0).ready entity=mon.1 client_cookie=d9557d8711f5cd71 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:02.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.199+0000 7f95d37fe700 1 -- 172.21.15.187:0/141106556 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d800ba70 con 0x7f95dc0b1380 2020-11-20T12:16:02.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.200+0000 7f95d37fe700 1 -- 172.21.15.187:0/141106556 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d800bbd0 con 0x7f95dc0b1380 2020-11-20T12:16:02.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.200+0000 7f95d37fe700 1 -- 172.21.15.187:0/141106556 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8011400 con 0x7f95dc0b1380 2020-11-20T12:16:02.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.200+0000 7f95e1e90700 1 -- 172.21.15.187:0/141106556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1380 msgr2=0x7f95dc150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:02.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.200+0000 7f95e1e90700 1 --2- 172.21.15.187:0/141106556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1380 0x7f95dc150ad0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 rx=0x7f95d8009c90 tx=0x7f95d80091d0).stop 2020-11-20T12:16:02.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.200+0000 7f95e1e90700 1 -- 172.21.15.187:0/141106556 shutdown_connections 2020-11-20T12:16:02.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.200+0000 7f95e1e90700 1 --2- 172.21.15.187:0/141106556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1380 0x7f95dc150ad0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:02.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.200+0000 7f95e1e90700 1 -- 172.21.15.187:0/141106556 >> 172.21.15.187:0/141106556 conn(0x7f95dc0bab30 msgr2=0x7f95dc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:02.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.201+0000 7f95e1e90700 1 -- 172.21.15.187:0/141106556 shutdown_connections 2020-11-20T12:16:02.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.201+0000 7f95e1e90700 1 -- 172.21.15.187:0/141106556 wait complete. 2020-11-20T12:16:02.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.203+0000 7f95e1e90700 1 Processor -- start 2020-11-20T12:16:02.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.203+0000 7f95e1e90700 1 -- start start 2020-11-20T12:16:02.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95e1e90700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc0b1380 0x7f95dc15d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:02.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95e1e90700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1760 0x7f95dc0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:02.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95e1e90700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc14c260 0x7f95dc1493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:02.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95e1e90700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0a02e0 con 0x7f95dc0b1380 2020-11-20T12:16:02.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95e1e90700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0a0a50 con 0x7f95dc0b1760 2020-11-20T12:16:02.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95e1e90700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0a0510 con 0x7f95dc14c260 2020-11-20T12:16:02.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95d3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1760 0x7f95dc0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:02.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95d3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1760 0x7f95dc0b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50660/0 (socket says 172.21.15.187:50660) 2020-11-20T12:16:02.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.204+0000 7f95d3fff700 1 -- 172.21.15.187:0/904035313 learned_addr learned my addr 172.21.15.187:0/904035313 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:02.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95e0e8e700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc0b1380 0x7f95dc15d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:02.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95e168f700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc14c260 0x7f95dc1493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:02.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d3fff700 1 -- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc14c260 msgr2=0x7f95dc1493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:02.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d3fff700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc14c260 0x7f95dc1493c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:02.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d3fff700 1 -- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc0b1380 msgr2=0x7f95dc15d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:02.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d3fff700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc0b1380 0x7f95dc15d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:02.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d3fff700 1 -- 172.21.15.187:0/904035313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc0a07b0 con 0x7f95dc0b1760 2020-11-20T12:16:02.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95e168f700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc14c260 0x7f95dc1493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:02.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d3fff700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1760 0x7f95dc0b33a0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7f95cc00c710 tx=0x7f95cc007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:02.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d0ff9700 1 -- 172.21.15.187:0/904035313 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc00e560 con 0x7f95dc0b1760 2020-11-20T12:16:02.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d0ff9700 1 -- 172.21.15.187:0/904035313 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc00e6c0 con 0x7f95dc0b1760 2020-11-20T12:16:02.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95d0ff9700 1 -- 172.21.15.187:0/904035313 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc00ea40 con 0x7f95dc0b1760 2020-11-20T12:16:02.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.205+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95dc15d560 con 0x7f95dc0b1760 2020-11-20T12:16:02.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.206+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95dc15c930 con 0x7f95dc0b1760 2020-11-20T12:16:02.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.207+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95dc061a20 con 0x7f95dc0b1760 2020-11-20T12:16:02.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.207+0000 7f95d0ff9700 1 -- 172.21.15.187:0/904035313 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f95cc019030 con 0x7f95dc0b1760 2020-11-20T12:16:02.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.208+0000 7f95d0ff9700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b4056d50 0x7f95b4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:02.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.208+0000 7f95d0ff9700 1 -- 172.21.15.187:0/904035313 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f95cc06f3e0 con 0x7f95dc0b1760 2020-11-20T12:16:02.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.209+0000 7f95e0e8e700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b4056d50 0x7f95b4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:02.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.209+0000 7f95e0e8e700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b4056d50 0x7f95b4059210 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f95d8009210 tx=0x7f95d8016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:02.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.212+0000 7f95d0ff9700 1 -- 172.21.15.187:0/904035313 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f95cc044a00 con 0x7f95dc0b1760 2020-11-20T12:16:02.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:02.408+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f95dc1509f0 con 0x7f95dc0b1760 2020-11-20T12:16:03.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.315+0000 7f95d0ff9700 1 -- 172.21.15.187:0/904035313 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v117) v1 ==== 125+0+0 (secure 0 0 0) 0x7f95cc017020 con 0x7f95dc0b1760 2020-11-20T12:16:03.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b4056d50 msgr2=0x7f95b4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:03.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b4056d50 0x7f95b4059210 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f95d8009210 tx=0x7f95d8016000).stop 2020-11-20T12:16:03.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1760 msgr2=0x7f95dc0b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:03.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1760 0x7f95dc0b33a0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7f95cc00c710 tx=0x7f95cc007c10).stop 2020-11-20T12:16:03.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 shutdown_connections 2020-11-20T12:16:03.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b4056d50 0x7f95b4059210 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc0b1380 0x7f95dc15d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc14c260 0x7f95dc1493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 --2- 172.21.15.187:0/904035313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc0b1760 0x7f95dc0b33a0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.318+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 >> 172.21.15.187:0/904035313 conn(0x7f95dc0bab30 msgr2=0x7f95dc14e400 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:03.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.319+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 shutdown_connections 2020-11-20T12:16:03.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.319+0000 7f95e1e90700 1 -- 172.21.15.187:0/904035313 wait complete. 2020-11-20T12:16:03.324 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:16:03.350 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:16:03.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.599+0000 7fa42bfff700 1 Processor -- start 2020-11-20T12:16:03.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.599+0000 7fa42bfff700 1 -- start start 2020-11-20T12:16:03.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.600+0000 7fa42bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa42c0bf800 0x7fa42c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:03.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.600+0000 7fa42bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c09f560 0x7fa42c0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:03.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.600+0000 7fa42bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c14d110 con 0x7fa42c0bfbe0 2020-11-20T12:16:03.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.600+0000 7fa42bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42c14cbd0 con 0x7fa42c0bf800 2020-11-20T12:16:03.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.600+0000 7fa42bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c14ce70 con 0x7fa42c09f560 2020-11-20T12:16:03.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.601+0000 7fa42affd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa42c0bf800 0x7fa42c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:03.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.601+0000 7fa42b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c09f560 0x7fa42c0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:03.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.601+0000 7fa42b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c09f560 0x7fa42c0a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:51558/0 (socket says 172.21.15.187:51558) 2020-11-20T12:16:03.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.601+0000 7fa42b7fe700 1 -- 172.21.15.187:0/2321790986 learned_addr learned my addr 172.21.15.187:0/2321790986 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:03.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.602+0000 7fa42b7fe700 1 -- 172.21.15.187:0/2321790986 >> v2:172.21.15.5:3300/0 conn(0x7fa42c0bf800 msgr2=0x7fa42c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:03.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.602+0000 7fa42b7fe700 1 --2- 172.21.15.187:0/2321790986 >> v2:172.21.15.5:3300/0 conn(0x7fa42c0bf800 0x7fa42c14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.602+0000 7fa42b7fe700 1 -- 172.21.15.187:0/2321790986 >> v1:172.21.15.5:6789/0 conn(0x7fa42c0bfbe0 legacy=0x7fa42c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:03.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.602+0000 7fa42b7fe700 1 -- 172.21.15.187:0/2321790986 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c0a4cc0 con 0x7fa42c09f560 2020-11-20T12:16:03.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.603+0000 7fa42b7fe700 1 --2- 172.21.15.187:0/2321790986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c09f560 0x7fa42c0a36e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa424009d50 tx=0x7fa424009290).ready entity=mon.2 client_cookie=3041c40b1a5500fa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:03.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.603+0000 7fa429ffb700 1 -- 172.21.15.187:0/2321790986 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42400bc50 con 0x7fa42c09f560 2020-11-20T12:16:03.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.603+0000 7fa429ffb700 1 -- 172.21.15.187:0/2321790986 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42400bdb0 con 0x7fa42c09f560 2020-11-20T12:16:03.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa429ffb700 1 -- 172.21.15.187:0/2321790986 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa424011610 con 0x7fa42c09f560 2020-11-20T12:16:03.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa42bfff700 1 -- 172.21.15.187:0/2321790986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c09f560 msgr2=0x7fa42c0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:03.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa42bfff700 1 --2- 172.21.15.187:0/2321790986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c09f560 0x7fa42c0a36e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa424009d50 tx=0x7fa424009290).stop 2020-11-20T12:16:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa42bfff700 1 -- 172.21.15.187:0/2321790986 shutdown_connections 2020-11-20T12:16:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa42bfff700 1 --2- 172.21.15.187:0/2321790986 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c09f560 0x7fa42c0a36e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa42bfff700 1 --2- 172.21.15.187:0/2321790986 >> v2:172.21.15.5:3300/0 conn(0x7fa42c0bf800 0x7fa42c14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa42bfff700 1 -- 172.21.15.187:0/2321790986 >> 172.21.15.187:0/2321790986 conn(0x7fa42c068370 msgr2=0x7fa42c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:03.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa42bfff700 1 -- 172.21.15.187:0/2321790986 shutdown_connections 2020-11-20T12:16:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.604+0000 7fa42bfff700 1 -- 172.21.15.187:0/2321790986 wait complete. 2020-11-20T12:16:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.606+0000 7fa42bfff700 1 Processor -- start 2020-11-20T12:16:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.606+0000 7fa42bfff700 1 -- start start 2020-11-20T12:16:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.607+0000 7fa42bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa42c09f560 0x7fa42c0bd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.607+0000 7fa42bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42c0bf800 0x7fa42c137a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.607+0000 7fa42bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c158510 0x7fa42c137f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.607+0000 7fa42bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c145710 con 0x7fa42c09f560 2020-11-20T12:16:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.607+0000 7fa42bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c1459b0 con 0x7fa42c0bf800 2020-11-20T12:16:03.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.607+0000 7fa42bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c0bd710 con 0x7fa42c158510 2020-11-20T12:16:03.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.607+0000 7fa42a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42c0bf800 0x7fa42c137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:03.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42c0bf800 0x7fa42c137a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50674/0 (socket says 172.21.15.187:50674) 2020-11-20T12:16:03.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42a7fc700 1 -- 172.21.15.187:0/3151602708 learned_addr learned my addr 172.21.15.187:0/3151602708 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:03.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42affd700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa42c09f560 0x7fa42c0bd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:03.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42b7fe700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c158510 0x7fa42c137f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:03.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42a7fc700 1 -- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c158510 msgr2=0x7fa42c137f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:03.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42a7fc700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c158510 0x7fa42c137f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42a7fc700 1 -- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa42c09f560 msgr2=0x7fa42c0bd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:03.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42a7fc700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa42c09f560 0x7fa42c0bd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42a7fc700 1 -- 172.21.15.187:0/3151602708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c144670 con 0x7fa42c0bf800 2020-11-20T12:16:03.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42b7fe700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c158510 0x7fa42c137f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:03.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.608+0000 7fa42a7fc700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42c0bf800 0x7fa42c137a20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 rx=0x7fa42000c6e0 tx=0x7fa420007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:03.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.609+0000 7fa412ffd700 1 -- 172.21.15.187:0/3151602708 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420007020 con 0x7fa42c0bf800 2020-11-20T12:16:03.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.609+0000 7fa412ffd700 1 -- 172.21.15.187:0/3151602708 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa420007180 con 0x7fa42c0bf800 2020-11-20T12:16:03.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.609+0000 7fa412ffd700 1 -- 172.21.15.187:0/3151602708 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42000e560 con 0x7fa42c0bf800 2020-11-20T12:16:03.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.609+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42c14ad90 con 0x7fa42c0bf800 2020-11-20T12:16:03.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.609+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42c0b40d0 con 0x7fa42c0bf800 2020-11-20T12:16:03.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.610+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42c14af80 con 0x7fa42c0bf800 2020-11-20T12:16:03.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.611+0000 7fa412ffd700 1 -- 172.21.15.187:0/3151602708 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa420010070 con 0x7fa42c0bf800 2020-11-20T12:16:03.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.611+0000 7fa412ffd700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa408056d20 0x7fa4080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:03.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.612+0000 7fa412ffd700 1 -- 172.21.15.187:0/3151602708 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fa42006f220 con 0x7fa42c0bf800 2020-11-20T12:16:03.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.612+0000 7fa42affd700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa408056d20 0x7fa4080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:03.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.613+0000 7fa42affd700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa408056d20 0x7fa4080591e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7fa41c006fd0 tx=0x7fa41c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:03.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.616+0000 7fa412ffd700 1 -- 172.21.15.187:0/3151602708 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa4200446e0 con 0x7fa42c0bf800 2020-11-20T12:16:03.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.864+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fa42c080050 con 0x7fa42c0bf800 2020-11-20T12:16:03.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.879+0000 7fa412ffd700 1 -- 172.21.15.187:0/3151602708 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 23 and data pool 24 v100) v1 ==== 189+0+0 (secure 0 0 0) 0x7fa420047d00 con 0x7fa42c0bf800 2020-11-20T12:16:03.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa408056d20 msgr2=0x7fa4080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:03.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa408056d20 0x7fa4080591e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7fa41c006fd0 tx=0x7fa41c008040).stop 2020-11-20T12:16:03.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42c0bf800 msgr2=0x7fa42c137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:03.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42c0bf800 0x7fa42c137a20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 rx=0x7fa42000c6e0 tx=0x7fa420007c50).stop 2020-11-20T12:16:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 shutdown_connections 2020-11-20T12:16:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa408056d20 0x7fa4080591e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa42c09f560 0x7fa42c0bd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa42c158510 0x7fa42c137f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 --2- 172.21.15.187:0/3151602708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa42c0bf800 0x7fa42c137a20 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:03.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 >> 172.21.15.187:0/3151602708 conn(0x7fa42c068370 msgr2=0x7fa42c143d70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:03.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 shutdown_connections 2020-11-20T12:16:03.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:03.883+0000 7fa42bfff700 1 -- 172.21.15.187:0/3151602708 wait complete. 2020-11-20T12:16:03.894 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 23 and data pool 24 2020-11-20T12:16:03.911 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:16:04.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.121+0000 7f81cf59e700 1 Processor -- start 2020-11-20T12:16:04.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.121+0000 7f81cf59e700 1 -- start start 2020-11-20T12:16:04.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.122+0000 7f81cf59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f81d0065190 0x7f81d0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.123+0000 7f81cf59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d00b13e0 con 0x7f81d009ca90 2020-11-20T12:16:04.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.123+0000 7f81cf59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d0062c10 con 0x7f81d0065f90 2020-11-20T12:16:04.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.123+0000 7f81cf59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d014ab30 con 0x7f81d0065190 2020-11-20T12:16:04.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.123+0000 7f81c5d9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f81d0065f90 0x7f81d0143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47298/0 (socket says 172.21.15.187:47298) 2020-11-20T12:16:04.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.123+0000 7f81c5d9b700 1 -- 172.21.15.187:0/1254433848 learned_addr learned my addr 172.21.15.187:0/1254433848 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:04.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.123+0000 7f81ce59c700 1 --2- 172.21.15.187:0/1254433848 >> v2:172.21.15.187:3300/0 conn(0x7f81d0065190 0x7f81d0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.124+0000 7f81ce59c700 1 -- 172.21.15.187:0/1254433848 >> v1:172.21.15.187:6789/0 conn(0x7f81d0065f90 legacy=0x7f81d0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.124+0000 7f81ce59c700 1 -- 172.21.15.187:0/1254433848 >> v1:172.21.15.5:6789/0 conn(0x7f81d009ca90 legacy=0x7f81d009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.124+0000 7f81ce59c700 1 -- 172.21.15.187:0/1254433848 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d00b1640 con 0x7f81d0065190 2020-11-20T12:16:04.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.124+0000 7f81ce59c700 1 --2- 172.21.15.187:0/1254433848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d0065190 0x7f81d0065570 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7f81d014ac70 tx=0x7f81c0006df0).ready entity=mon.1 client_cookie=1f0b94b1e67072f3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:04.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.125+0000 7f81cdd9b700 1 -- 172.21.15.187:0/1254433848 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c0009840 con 0x7f81d0065190 2020-11-20T12:16:04.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.125+0000 7f81cdd9b700 1 -- 172.21.15.187:0/1254433848 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c00099a0 con 0x7f81d0065190 2020-11-20T12:16:04.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.125+0000 7f81cdd9b700 1 -- 172.21.15.187:0/1254433848 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c0009d20 con 0x7f81d0065190 2020-11-20T12:16:04.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.125+0000 7f81cf59e700 1 -- 172.21.15.187:0/1254433848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d0065190 msgr2=0x7f81d0065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.125+0000 7f81cf59e700 1 --2- 172.21.15.187:0/1254433848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d0065190 0x7f81d0065570 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7f81d014ac70 tx=0x7f81c0006df0).stop 2020-11-20T12:16:04.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.125+0000 7f81cf59e700 1 -- 172.21.15.187:0/1254433848 shutdown_connections 2020-11-20T12:16:04.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.125+0000 7f81cf59e700 1 --2- 172.21.15.187:0/1254433848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d0065190 0x7f81d0065570 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.125+0000 7f81cf59e700 1 -- 172.21.15.187:0/1254433848 >> 172.21.15.187:0/1254433848 conn(0x7f81d00bab30 msgr2=0x7f81d00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:04.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.126+0000 7f81cf59e700 1 -- 172.21.15.187:0/1254433848 shutdown_connections 2020-11-20T12:16:04.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.126+0000 7f81cf59e700 1 -- 172.21.15.187:0/1254433848 wait complete. 2020-11-20T12:16:04.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.128+0000 7f81cf59e700 1 Processor -- start 2020-11-20T12:16:04.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.128+0000 7f81cf59e700 1 -- start start 2020-11-20T12:16:04.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81cf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81d0065190 0x7f81d009f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81cf59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81d0065f90 0x7f81d0066590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81cf59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d009ca90 0x7f81d0066a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81cf59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d00ba2a0 con 0x7f81d0065f90 2020-11-20T12:16:04.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81cf59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d0068520 con 0x7f81d009ca90 2020-11-20T12:16:04.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81cf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d00b9b60 con 0x7f81d0065190 2020-11-20T12:16:04.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81ced9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d009ca90 0x7f81d0066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81ced9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d009ca90 0x7f81d0066a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50700/0 (socket says 172.21.15.187:50700) 2020-11-20T12:16:04.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.129+0000 7f81ced9d700 1 -- 172.21.15.187:0/3486610160 learned_addr learned my addr 172.21.15.187:0/3486610160 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:04.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81c5d9b700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81d0065f90 0x7f81d0066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81ce59c700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81d0065190 0x7f81d009f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81ced9d700 1 -- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81d0065190 msgr2=0x7f81d009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81ced9d700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81d0065190 0x7f81d009f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81ced9d700 1 -- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81d0065f90 msgr2=0x7f81d0066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81ced9d700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81d0065f90 0x7f81d0066590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81ced9d700 1 -- 172.21.15.187:0/3486610160 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d00b9e00 con 0x7f81d009ca90 2020-11-20T12:16:04.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81ce59c700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81d0065190 0x7f81d009f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:04.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81ced9d700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d009ca90 0x7f81d0066a80 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7f81c800ca90 tx=0x7f81c8001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81c6ffd700 1 -- 172.21.15.187:0/3486610160 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c80013d0 con 0x7f81d009ca90 2020-11-20T12:16:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.130+0000 7f81c6ffd700 1 -- 172.21.15.187:0/3486610160 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c800a730 con 0x7f81d009ca90 2020-11-20T12:16:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.131+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d009e950 con 0x7f81d009ca90 2020-11-20T12:16:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.131+0000 7f81c6ffd700 1 -- 172.21.15.187:0/3486610160 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c800aad0 con 0x7f81d009ca90 2020-11-20T12:16:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.131+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d0068080 con 0x7f81d009ca90 2020-11-20T12:16:04.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.132+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d0068330 con 0x7f81d009ca90 2020-11-20T12:16:04.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.133+0000 7f81c6ffd700 1 -- 172.21.15.187:0/3486610160 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f81c8010070 con 0x7f81d009ca90 2020-11-20T12:16:04.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.133+0000 7f81c6ffd700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ac056d20 0x7f81ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.133+0000 7f81c6ffd700 1 -- 172.21.15.187:0/3486610160 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f81c806fc60 con 0x7f81d009ca90 2020-11-20T12:16:04.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.134+0000 7f81ce59c700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ac056d20 0x7f81ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.134+0000 7f81ce59c700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ac056d20 0x7f81ac0591e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f81c0009fd0 tx=0x7f81c0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:04.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.137+0000 7f81c6ffd700 1 -- 172.21.15.187:0/3486610160 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f81c80450b0 con 0x7f81d009ca90 2020-11-20T12:16:04.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.322+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81d0061a20 con 0x7f81d009ca90 2020-11-20T12:16:04.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.325+0000 7f81c6ffd700 1 -- 172.21.15.187:0/3486610160 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16804 (secure 0 0 0) 0x7f81c80486d0 con 0x7f81d009ca90 2020-11-20T12:16:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:04.329 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":118,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:16:03.867052+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:16:00.509740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2020-11-20T12:16:02.486990+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:16:04.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ac056d20 msgr2=0x7f81ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ac056d20 0x7f81ac0591e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f81c0009fd0 tx=0x7f81c0015000).stop 2020-11-20T12:16:04.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d009ca90 msgr2=0x7f81d0066a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d009ca90 0x7f81d0066a80 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7f81c800ca90 tx=0x7f81c8001360).stop 2020-11-20T12:16:04.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 shutdown_connections 2020-11-20T12:16:04.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ac056d20 0x7f81ac0591e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d009ca90 0x7f81d0066a80 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81d0065190 0x7f81d009f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 --2- 172.21.15.187:0/3486610160 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81d0065f90 0x7f81d0066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 >> 172.21.15.187:0/3486610160 conn(0x7f81d00bab30 msgr2=0x7f81d009aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:04.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 shutdown_connections 2020-11-20T12:16:04.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.327+0000 7f81cf59e700 1 -- 172.21.15.187:0/3486610160 wait complete. 2020-11-20T12:16:04.354 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:16:04.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.588+0000 7f4fe14a3700 1 Processor -- start 2020-11-20T12:16:04.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.588+0000 7f4fe14a3700 1 -- start start 2020-11-20T12:16:04.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.590+0000 7f4fe14a3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4fdc068230 0x7f4fdc1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.590+0000 7f4fe14a3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fdc144b80 con 0x7f4fdc0b57b0 2020-11-20T12:16:04.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.590+0000 7f4fe14a3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fdc1489e0 con 0x7f4fdc0b5f50 2020-11-20T12:16:04.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.590+0000 7f4fe14a3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fdc1487c0 con 0x7f4fdc068230 2020-11-20T12:16:04.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.590+0000 7f4fe0ca2700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4fdc0b5f50 0x7f4fdc0b6380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47314/0 (socket says 172.21.15.187:47314) 2020-11-20T12:16:04.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.590+0000 7f4fe0ca2700 1 -- 172.21.15.187:0/3045473834 learned_addr learned my addr 172.21.15.187:0/3045473834 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:04.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.590+0000 7f4fd3fff700 1 --2- 172.21.15.187:0/3045473834 >> v2:172.21.15.5:3300/0 conn(0x7f4fdc068230 0x7f4fdc1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847603056 0 0) 0x7f4fdc1489e0 con 0x7f4fdc0b5f50 2020-11-20T12:16:04.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fbc003240 con 0x7f4fdc0b5f50 2020-11-20T12:16:04.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3199006791 0 0) 0x7f4fbc003240 con 0x7f4fdc0b5f50 2020-11-20T12:16:04.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 >> v2:172.21.15.5:3300/0 conn(0x7f4fdc068230 msgr2=0x7f4fdc1482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:04.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 --2- 172.21.15.187:0/3045473834 >> v2:172.21.15.5:3300/0 conn(0x7f4fdc068230 0x7f4fdc1482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 >> v1:172.21.15.5:6789/0 conn(0x7f4fdc0b57b0 legacy=0x7f4fdc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc148c80 con 0x7f4fdc0b5f50 2020-11-20T12:16:04.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4fd4001860 con 0x7f4fdc0b5f50 2020-11-20T12:16:04.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 >> v1:172.21.15.187:6789/0 conn(0x7f4fdc0b5f50 legacy=0x7f4fdc0b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fbc003730 0x7f4fbc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fbc0044e0 0x7f4fbc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fbc004e50 0x7f4fbc00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc1487c0 con 0x7f4fbc004e50 2020-11-20T12:16:04.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc004260 con 0x7f4fbc003730 2020-11-20T12:16:04.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.591+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc00de30 con 0x7f4fbc0044e0 2020-11-20T12:16:04.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fdbfff700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fbc003730 0x7f4fbc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fe0ca2700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fbc004e50 0x7f4fbc00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fd3fff700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fbc0044e0 0x7f4fbc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fdbfff700 1 -- 172.21.15.187:0/3045473834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fbc0044e0 msgr2=0x7f4fbc004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:04.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fdbfff700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fbc0044e0 0x7f4fbc004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fdbfff700 1 -- 172.21.15.187:0/3045473834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fbc004e50 msgr2=0x7f4fbc00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:04.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fdbfff700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fbc004e50 0x7f4fbc00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fdbfff700 1 -- 172.21.15.187:0/3045473834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc003240 con 0x7f4fbc003730 2020-11-20T12:16:04.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.592+0000 7f4fe0ca2700 1 -- 172.21.15.187:0/3045473834 reap_dead start 2020-11-20T12:16:04.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.593+0000 7f4fdbfff700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fbc003730 0x7f4fbc003b50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7f4fcc001920 tx=0x7f4fcc007b90).ready entity=mon.1 client_cookie=3923925eb82faa5f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:04.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.593+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc007020 con 0x7f4fbc003730 2020-11-20T12:16:04.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.593+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc007180 con 0x7f4fbc003730 2020-11-20T12:16:04.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.593+0000 7f4fdb7fe700 1 -- 172.21.15.187:0/3045473834 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc00c620 con 0x7f4fbc003730 2020-11-20T12:16:04.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.594+0000 7f4fe14a3700 1 -- 172.21.15.187:0/3045473834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fbc003730 msgr2=0x7f4fbc003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.594+0000 7f4fe14a3700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fbc003730 0x7f4fbc003b50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7f4fcc001920 tx=0x7f4fcc007b90).stop 2020-11-20T12:16:04.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.594+0000 7f4fe14a3700 1 -- 172.21.15.187:0/3045473834 shutdown_connections 2020-11-20T12:16:04.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.594+0000 7f4fe14a3700 1 --2- 172.21.15.187:0/3045473834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fbc003730 0x7f4fbc003b50 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.594+0000 7f4fe14a3700 1 -- 172.21.15.187:0/3045473834 >> 172.21.15.187:0/3045473834 conn(0x7f4fdc0bab30 msgr2=0x7f4fdc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:04.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.594+0000 7f4fe14a3700 1 -- 172.21.15.187:0/3045473834 shutdown_connections 2020-11-20T12:16:04.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.594+0000 7f4fe14a3700 1 -- 172.21.15.187:0/3045473834 wait complete. 2020-11-20T12:16:04.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.595+0000 7f4fe14a3700 1 Processor -- start 2020-11-20T12:16:04.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.596+0000 7f4fe14a3700 1 -- start start 2020-11-20T12:16:04.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.596+0000 7f4fe14a3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fdc09f580 0x7f4fdc147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.596+0000 7f4fe14a3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fdc09f960 0x7f4fdc0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.596+0000 7f4fe14a3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fdc1450c0 0x7f4fdc0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.596+0000 7f4fe14a3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0b1d30 con 0x7f4fdc09f960 2020-11-20T12:16:04.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.596+0000 7f4fe14a3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0b1fd0 con 0x7f4fdc09f580 2020-11-20T12:16:04.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.596+0000 7f4fe14a3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc147bf0 con 0x7f4fdc1450c0 2020-11-20T12:16:04.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fdbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fdc09f580 0x7f4fdc147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fdbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fdc09f580 0x7f4fdc147700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50716/0 (socket says 172.21.15.187:50716) 2020-11-20T12:16:04.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fdbfff700 1 -- 172.21.15.187:0/342993241 learned_addr learned my addr 172.21.15.187:0/342993241 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:04.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fd3fff700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fdc09f960 0x7f4fdc0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fdbfff700 1 -- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fdc1450c0 msgr2=0x7f4fdc0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fe0ca2700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fdc1450c0 0x7f4fdc0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fdbfff700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fdc1450c0 0x7f4fdc0b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fdbfff700 1 -- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fdc09f960 msgr2=0x7f4fdc0b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fdbfff700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fdc09f960 0x7f4fdc0b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fdbfff700 1 -- 172.21.15.187:0/342993241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc0b2f10 con 0x7f4fdc09f580 2020-11-20T12:16:04.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.597+0000 7f4fd3fff700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fdc09f960 0x7f4fdc0b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:04.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.598+0000 7f4fdbfff700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fdc09f580 0x7f4fdc147700 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 rx=0x7f4fcc000f90 tx=0x7f4fcc015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:04.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.598+0000 7f4fd8ff9700 1 -- 172.21.15.187:0/342993241 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc013c80 con 0x7f4fdc09f580 2020-11-20T12:16:04.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.598+0000 7f4fd8ff9700 1 -- 172.21.15.187:0/342993241 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc013de0 con 0x7f4fdc09f580 2020-11-20T12:16:04.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.598+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc0b3100 con 0x7f4fdc09f580 2020-11-20T12:16:04.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.598+0000 7f4fd8ff9700 1 -- 172.21.15.187:0/342993241 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc020690 con 0x7f4fdc09f580 2020-11-20T12:16:04.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.598+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc0b35c0 con 0x7f4fdc09f580 2020-11-20T12:16:04.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.599+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc0b37b0 con 0x7f4fdc09f580 2020-11-20T12:16:04.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.600+0000 7f4fd8ff9700 1 -- 172.21.15.187:0/342993241 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4fcc026020 con 0x7f4fdc09f580 2020-11-20T12:16:04.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.601+0000 7f4fd8ff9700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fb8056d20 0x7f4fb80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:04.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.601+0000 7f4fd8ff9700 1 -- 172.21.15.187:0/342993241 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4fcc0715d0 con 0x7f4fdc09f580 2020-11-20T12:16:04.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.601+0000 7f4fd3fff700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fb8056d20 0x7f4fb80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:04.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.602+0000 7f4fd3fff700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fb8056d20 0x7f4fb80591e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f4fc4006a30 tx=0x7f4fc4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:04.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.605+0000 7f4fd8ff9700 1 -- 172.21.15.187:0/342993241 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4fcc046a20 con 0x7f4fdc09f580 2020-11-20T12:16:04.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.788+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fdc147e30 con 0x7f4fdc09f580 2020-11-20T12:16:04.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.790+0000 7f4fd8ff9700 1 -- 172.21.15.187:0/342993241 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16804 (secure 0 0 0) 0x7f4fcc04a040 con 0x7f4fdc09f580 2020-11-20T12:16:04.794 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:04.795 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":118,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:16:03.867052+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:16:00.509740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2020-11-20T12:16:02.486990+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:16:04.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fb8056d20 msgr2=0x7f4fb80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fb8056d20 0x7f4fb80591e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f4fc4006a30 tx=0x7f4fc4008040).stop 2020-11-20T12:16:04.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fdc09f580 msgr2=0x7f4fdc147700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:04.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fdc09f580 0x7f4fdc147700 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 rx=0x7f4fcc000f90 tx=0x7f4fcc015000).stop 2020-11-20T12:16:04.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 shutdown_connections 2020-11-20T12:16:04.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fb8056d20 0x7f4fb80591e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fdc1450c0 0x7f4fdc0b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fdc09f580 0x7f4fdc147700 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 --2- 172.21.15.187:0/342993241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fdc09f960 0x7f4fdc0b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:04.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 >> 172.21.15.187:0/342993241 conn(0x7f4fdc0bab30 msgr2=0x7f4fdc0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:04.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.792+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 shutdown_connections 2020-11-20T12:16:04.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:04.793+0000 7f4fe14a3700 1 -- 172.21.15.187:0/342993241 wait complete. 2020-11-20T12:16:04.817 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:16:05.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.043+0000 7f0a6ae0d700 1 Processor -- start 2020-11-20T12:16:05.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.043+0000 7f0a6ae0d700 1 -- start start 2020-11-20T12:16:05.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.044+0000 7f0a6ae0d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0a640bbaa0 0x7f0a640a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:05.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a6ae0d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a641426f0 0x7f0a64142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:05.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a6ae0d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a64149930 con 0x7f0a640bbe80 2020-11-20T12:16:05.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a6ae0d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a640a3560 con 0x7f0a640bbaa0 2020-11-20T12:16:05.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a6ae0d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6414a330 con 0x7f0a641426f0 2020-11-20T12:16:05.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a69e0b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0a640bbaa0 0x7f0a640a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:05.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a6960a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0a640bbe80 0x7f0a64149c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47332/0 (socket says 172.21.15.187:47332) 2020-11-20T12:16:05.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a6960a700 1 -- 172.21.15.187:0/3451887450 learned_addr learned my addr 172.21.15.187:0/3451887450 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:05.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a6a60c700 1 --2- 172.21.15.187:0/3451887450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a641426f0 0x7f0a64142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:05.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.045+0000 7f0a69e0b700 1 -- 172.21.15.187:0/3451887450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a641426f0 msgr2=0x7f0a64142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:05.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.046+0000 7f0a69e0b700 1 --2- 172.21.15.187:0/3451887450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a641426f0 0x7f0a64142b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.046+0000 7f0a69e0b700 1 -- 172.21.15.187:0/3451887450 >> v1:172.21.15.187:6789/0 conn(0x7f0a640bbe80 legacy=0x7f0a64149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:05.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.046+0000 7f0a69e0b700 1 -- 172.21.15.187:0/3451887450 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a640a3250 con 0x7f0a640bbaa0 2020-11-20T12:16:05.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.046+0000 7f0a6a60c700 1 --2- 172.21.15.187:0/3451887450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a641426f0 0x7f0a64142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:05.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.046+0000 7f0a69e0b700 1 --2- 172.21.15.187:0/3451887450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbaa0 0x7f0a640a4bf0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 rx=0x7f0a6000b9a0 tx=0x7f0a60006df0).ready entity=mon.1 client_cookie=7d0c5937773af90e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:05.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.046+0000 7f0a68e09700 1 -- 172.21.15.187:0/3451887450 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600097b0 con 0x7f0a640bbaa0 2020-11-20T12:16:05.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.046+0000 7f0a68e09700 1 -- 172.21.15.187:0/3451887450 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60009910 con 0x7f0a640bbaa0 2020-11-20T12:16:05.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.046+0000 7f0a68e09700 1 -- 172.21.15.187:0/3451887450 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60009c90 con 0x7f0a640bbaa0 2020-11-20T12:16:05.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.047+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/3451887450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbaa0 msgr2=0x7f0a640a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:05.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.047+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/3451887450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbaa0 0x7f0a640a4bf0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 rx=0x7f0a6000b9a0 tx=0x7f0a60006df0).stop 2020-11-20T12:16:05.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.047+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/3451887450 shutdown_connections 2020-11-20T12:16:05.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.047+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/3451887450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a641426f0 0x7f0a64142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.047+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/3451887450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbaa0 0x7f0a640a4bf0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.047+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/3451887450 >> 172.21.15.187:0/3451887450 conn(0x7f0a64061bd0 msgr2=0x7f0a640678a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:05.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.047+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/3451887450 shutdown_connections 2020-11-20T12:16:05.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.047+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/3451887450 wait complete. 2020-11-20T12:16:05.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.048+0000 7f0a6ae0d700 1 Processor -- start 2020-11-20T12:16:05.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.048+0000 7f0a6ae0d700 1 -- start start 2020-11-20T12:16:05.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.049+0000 7f0a6ae0d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a640bbaa0 0x7f0a641582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:05.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.049+0000 7f0a6ae0d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbe80 0x7f0a64134990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:05.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.049+0000 7f0a6ae0d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a641426f0 0x7f0a640b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:05.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.049+0000 7f0a6ae0d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a640b55b0 con 0x7f0a641426f0 2020-11-20T12:16:05.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.049+0000 7f0a6ae0d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a64134e80 con 0x7f0a640bbe80 2020-11-20T12:16:05.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.049+0000 7f0a6ae0d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a64145be0 con 0x7f0a640bbaa0 2020-11-20T12:16:05.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.049+0000 7f0a6960a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbe80 0x7f0a64134990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:05.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6960a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbe80 0x7f0a64134990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50732/0 (socket says 172.21.15.187:50732) 2020-11-20T12:16:05.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6960a700 1 -- 172.21.15.187:0/1703811542 learned_addr learned my addr 172.21.15.187:0/1703811542 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:05.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6a60c700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a641426f0 0x7f0a640b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:05.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a69e0b700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a640bbaa0 0x7f0a641582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:05.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6960a700 1 -- 172.21.15.187:0/1703811542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a640bbaa0 msgr2=0x7f0a641582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:05.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6960a700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a640bbaa0 0x7f0a641582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6960a700 1 -- 172.21.15.187:0/1703811542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a641426f0 msgr2=0x7f0a640b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:05.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6960a700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a641426f0 0x7f0a640b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6960a700 1 -- 172.21.15.187:0/1703811542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a64145da0 con 0x7f0a640bbe80 2020-11-20T12:16:05.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.050+0000 7f0a6960a700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbe80 0x7f0a64134990 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7f0a5400c710 tx=0x7f0a54007c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:05.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.051+0000 7f0a51ffb700 1 -- 172.21.15.187:0/1703811542 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54007320 con 0x7f0a640bbe80 2020-11-20T12:16:05.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.051+0000 7f0a51ffb700 1 -- 172.21.15.187:0/1703811542 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a54007480 con 0x7f0a640bbe80 2020-11-20T12:16:05.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.051+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a640b4e10 con 0x7f0a640bbe80 2020-11-20T12:16:05.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.051+0000 7f0a51ffb700 1 -- 172.21.15.187:0/1703811542 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54005800 con 0x7f0a640bbe80 2020-11-20T12:16:05.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.051+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a640b52d0 con 0x7f0a640bbe80 2020-11-20T12:16:05.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.052+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a640b59f0 con 0x7f0a640bbe80 2020-11-20T12:16:05.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.052+0000 7f0a51ffb700 1 -- 172.21.15.187:0/1703811542 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0a54010030 con 0x7f0a640bbe80 2020-11-20T12:16:05.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.053+0000 7f0a51ffb700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0a3c056d20 0x7f0a3c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:05.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.053+0000 7f0a51ffb700 1 -- 172.21.15.187:0/1703811542 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0a5406fb10 con 0x7f0a640bbe80 2020-11-20T12:16:05.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.053+0000 7f0a69e0b700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0a3c056d20 0x7f0a3c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:05.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.054+0000 7f0a69e0b700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0a3c056d20 0x7f0a3c0591e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f0a6000b150 tx=0x7f0a60015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:05.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.057+0000 7f0a51ffb700 1 -- 172.21.15.187:0/1703811542 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0a54044ee0 con 0x7f0a640bbe80 2020-11-20T12:16:05.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.294+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0a64145da0 con 0x7f0a640bbe80 2020-11-20T12:16:05.541 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:16:05.537+0000 7f42cae08700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:16:05.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.884+0000 7f0a51ffb700 1 -- 172.21.15.187:0/1703811542 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v102) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0a64145da0 con 0x7f0a640bbe80 2020-11-20T12:16:05.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0a3c056d20 msgr2=0x7f0a3c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:05.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0a3c056d20 0x7f0a3c0591e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f0a6000b150 tx=0x7f0a60015000).stop 2020-11-20T12:16:05.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbe80 msgr2=0x7f0a64134990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:05.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbe80 0x7f0a64134990 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7f0a5400c710 tx=0x7f0a54007c40).stop 2020-11-20T12:16:05.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 shutdown_connections 2020-11-20T12:16:05.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0a3c056d20 0x7f0a3c0591e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a641426f0 0x7f0a640b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a640bbaa0 0x7f0a641582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 --2- 172.21.15.187:0/1703811542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a640bbe80 0x7f0a64134990 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:05.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 >> 172.21.15.187:0/1703811542 conn(0x7f0a64061bd0 msgr2=0x7f0a64140d30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:05.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 shutdown_connections 2020-11-20T12:16:05.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:05.888+0000 7f0a6ae0d700 1 -- 172.21.15.187:0/1703811542 wait complete. 2020-11-20T12:16:05.919 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.157+0000 7fb50c81e700 1 Processor -- start 2020-11-20T12:16:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.158+0000 7fb50c81e700 1 -- start start 2020-11-20T12:16:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.159+0000 7fb50c81e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb50814d270 0x7fb5081573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.159+0000 7fb50c81e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5080658a0 con 0x7fb5080b1760 2020-11-20T12:16:06.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.159+0000 7fb50c81e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb508150390 con 0x7fb5080b1380 2020-11-20T12:16:06.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.159+0000 7fb50c81e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5081511e0 con 0x7fb50814d270 2020-11-20T12:16:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.159+0000 7fb5077fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb5080b1380 0x7fb508150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47344/0 (socket says 172.21.15.187:47344) 2020-11-20T12:16:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.159+0000 7fb5077fe700 1 -- 172.21.15.187:0/2971874344 learned_addr learned my addr 172.21.15.187:0/2971874344 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.160+0000 7fb507fff700 1 --2- 172.21.15.187:0/2971874344 >> v2:172.21.15.187:3300/0 conn(0x7fb50814d270 0x7fb5081573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:06.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.160+0000 7fb506ffd700 1 -- 172.21.15.187:0/2971874344 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803814682 0 0) 0x7fb508150390 con 0x7fb5080b1380 2020-11-20T12:16:06.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.160+0000 7fb506ffd700 1 -- 172.21.15.187:0/2971874344 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e8003240 con 0x7fb5080b1380 2020-11-20T12:16:06.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.160+0000 7fb507fff700 1 -- 172.21.15.187:0/2971874344 >> v1:172.21.15.187:6789/0 conn(0x7fb5080b1380 legacy=0x7fb508150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:06.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.160+0000 7fb507fff700 1 -- 172.21.15.187:0/2971874344 >> v1:172.21.15.5:6789/0 conn(0x7fb5080b1760 legacy=0x7fb508065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:06.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.160+0000 7fb507fff700 1 -- 172.21.15.187:0/2971874344 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508150630 con 0x7fb50814d270 2020-11-20T12:16:06.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.161+0000 7fb507fff700 1 --2- 172.21.15.187:0/2971874344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb50814d270 0x7fb5081573a0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7fb500009c90 tx=0x7fb5000091d0).ready entity=mon.1 client_cookie=c84d9a3a9c5de25c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:06.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.161+0000 7fb506ffd700 1 -- 172.21.15.187:0/2971874344 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50000ba70 con 0x7fb50814d270 2020-11-20T12:16:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.161+0000 7fb506ffd700 1 -- 172.21.15.187:0/2971874344 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50000bbd0 con 0x7fb50814d270 2020-11-20T12:16:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.161+0000 7fb506ffd700 1 -- 172.21.15.187:0/2971874344 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb500011400 con 0x7fb50814d270 2020-11-20T12:16:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.161+0000 7fb50c81e700 1 -- 172.21.15.187:0/2971874344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb50814d270 msgr2=0x7fb5081573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.161+0000 7fb50c81e700 1 --2- 172.21.15.187:0/2971874344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb50814d270 0x7fb5081573a0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7fb500009c90 tx=0x7fb5000091d0).stop 2020-11-20T12:16:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.162+0000 7fb50c81e700 1 -- 172.21.15.187:0/2971874344 shutdown_connections 2020-11-20T12:16:06.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.162+0000 7fb50c81e700 1 --2- 172.21.15.187:0/2971874344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb50814d270 0x7fb5081573a0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:06.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.162+0000 7fb50c81e700 1 -- 172.21.15.187:0/2971874344 >> 172.21.15.187:0/2971874344 conn(0x7fb5080bab30 msgr2=0x7fb5080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:06.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.162+0000 7fb50c81e700 1 -- 172.21.15.187:0/2971874344 shutdown_connections 2020-11-20T12:16:06.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.162+0000 7fb50c81e700 1 -- 172.21.15.187:0/2971874344 wait complete. 2020-11-20T12:16:06.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.163+0000 7fb50c81e700 1 Processor -- start 2020-11-20T12:16:06.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.163+0000 7fb50c81e700 1 -- start start 2020-11-20T12:16:06.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.164+0000 7fb50c81e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080b1380 0x7fb5080b7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:06.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.164+0000 7fb50c81e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080b1760 0x7fb5080b6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:06.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.164+0000 7fb50c81e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb50814d270 0x7fb508148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:06.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.164+0000 7fb50c81e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5080b60e0 con 0x7fb5080b1760 2020-11-20T12:16:06.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.164+0000 7fb50c81e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5080b6380 con 0x7fb5080b1380 2020-11-20T12:16:06.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.164+0000 7fb50c81e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5080b6620 con 0x7fb50814d270 2020-11-20T12:16:06.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080b1380 0x7fb5080b7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:06.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080b1380 0x7fb5080b7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50744/0 (socket says 172.21.15.187:50744) 2020-11-20T12:16:06.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 -- 172.21.15.187:0/1380399276 learned_addr learned my addr 172.21.15.187:0/1380399276 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:06.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb4feffd700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080b1760 0x7fb5080b6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:06.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 -- 172.21.15.187:0/1380399276 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb50814d270 msgr2=0x7fb508148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:06.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb50814d270 0x7fb508148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:06.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 -- 172.21.15.187:0/1380399276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080b1760 msgr2=0x7fb5080b6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:06.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080b1760 0x7fb5080b6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 -- 172.21.15.187:0/1380399276 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508148e00 con 0x7fb5080b1380 2020-11-20T12:16:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.165+0000 7fb5077fe700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080b1380 0x7fb5080b7020 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 rx=0x7fb4f800c6f0 tx=0x7fb4f8007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.166+0000 7fb4fffff700 1 -- 172.21.15.187:0/1380399276 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8007320 con 0x7fb5080b1380 2020-11-20T12:16:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.166+0000 7fb4fffff700 1 -- 172.21.15.187:0/1380399276 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f8007480 con 0x7fb5080b1380 2020-11-20T12:16:06.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.166+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5080b5cf0 con 0x7fb5080b1380 2020-11-20T12:16:06.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.166+0000 7fb4fffff700 1 -- 172.21.15.187:0/1380399276 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8005800 con 0x7fb5080b1380 2020-11-20T12:16:06.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.166+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5080b43a0 con 0x7fb5080b1380 2020-11-20T12:16:06.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.167+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5080b67e0 con 0x7fb5080b1380 2020-11-20T12:16:06.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.167+0000 7fb4fffff700 1 -- 172.21.15.187:0/1380399276 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb4f8010030 con 0x7fb5080b1380 2020-11-20T12:16:06.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.168+0000 7fb4fffff700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4e4056d20 0x7fb4e40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:06.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.168+0000 7fb4fffff700 1 -- 172.21.15.187:0/1380399276 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb4f806fd30 con 0x7fb5080b1380 2020-11-20T12:16:06.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.168+0000 7fb4feffd700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4e4056d20 0x7fb4e40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:06.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.169+0000 7fb4feffd700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4e4056d20 0x7fb4e40591e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fb5080b3b80 tx=0x7fb4f0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:06.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.172+0000 7fb4fffff700 1 -- 172.21.15.187:0/1380399276 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb4f8045120 con 0x7fb5080b1380 2020-11-20T12:16:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.420+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb508061a20 con 0x7fb5080b1380 2020-11-20T12:16:06.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.422+0000 7fb4fffff700 1 -- 172.21.15.187:0/1380399276 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2141 (secure 0 0 0) 0x7fb4f8048740 con 0x7fb5080b1380 2020-11-20T12:16:06.425 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:06.426 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":102,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2092825437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2092825437},{"type":"v1","addr":"172.21.15.5:6833","nonce":2092825437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":99}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:05.876197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8473},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8473":{"gid":8473,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3143922581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3143922581},{"type":"v1","addr":"172.21.15.187:6835","nonce":3143922581}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:06.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4e4056d20 msgr2=0x7fb4e40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:08.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4e4056d20 0x7fb4e40591e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fb5080b3b80 tx=0x7fb4f0009040).stop 2020-11-20T12:16:08.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080b1380 msgr2=0x7fb5080b7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:08.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080b1380 0x7fb5080b7020 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 rx=0x7fb4f800c6f0 tx=0x7fb4f8007ad0).stop 2020-11-20T12:16:08.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 shutdown_connections 2020-11-20T12:16:08.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb4e4056d20 0x7fb4e40591e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:08.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb50814d270 0x7fb508148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:08.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5080b1380 0x7fb5080b7020 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:08.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 --2- 172.21.15.187:0/1380399276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5080b1760 0x7fb5080b6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:08.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 >> 172.21.15.187:0/1380399276 conn(0x7fb5080bab30 msgr2=0x7fb508149090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:08.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 shutdown_connections 2020-11-20T12:16:08.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:06.424+0000 7fb50c81e700 1 -- 172.21.15.187:0/1380399276 wait complete. 2020-11-20T12:16:08.895 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 102 2020-11-20T12:16:08.896 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:16:09.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.142+0000 7ff11d4b0700 1 Processor -- start 2020-11-20T12:16:09.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.142+0000 7ff11d4b0700 1 -- start start 2020-11-20T12:16:09.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.143+0000 7ff11d4b0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff118066f70 0x7ff1180b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.143+0000 7ff11d4b0700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1180b1300 con 0x7ff1181484d0 2020-11-20T12:16:09.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.143+0000 7ff11d4b0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1180b15a0 con 0x7ff1180b6650 2020-11-20T12:16:09.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.143+0000 7ff11d4b0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff118067730 con 0x7ff118066f70 2020-11-20T12:16:09.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.143+0000 7ff10f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff1180b6650 0x7ff1180afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47360/0 (socket says 172.21.15.187:47360) 2020-11-20T12:16:09.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.143+0000 7ff10f7fe700 1 -- 172.21.15.187:0/1754732667 learned_addr learned my addr 172.21.15.187:0/1754732667 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:09.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028770692 0 0) 0x7ff1180b15a0 con 0x7ff1180b6650 2020-11-20T12:16:09.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0f8003240 con 0x7ff1180b6650 2020-11-20T12:16:09.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff11ccaf700 1 --2- 172.21.15.187:0/1754732667 >> v2:172.21.15.5:3300/0 conn(0x7ff118066f70 0x7ff1180b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 760555455 0 0) 0x7ff0f8003240 con 0x7ff1180b6650 2020-11-20T12:16:09.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 >> v2:172.21.15.5:3300/0 conn(0x7ff118066f70 msgr2=0x7ff1180b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:09.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 --2- 172.21.15.187:0/1754732667 >> v2:172.21.15.5:3300/0 conn(0x7ff118066f70 0x7ff1180b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 >> v1:172.21.15.5:6789/0 conn(0x7ff1181484d0 legacy=0x7ff118061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1180b1840 con 0x7ff1180b6650 2020-11-20T12:16:09.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff1000014e0 con 0x7ff1180b6650 2020-11-20T12:16:09.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 >> v1:172.21.15.187:6789/0 conn(0x7ff1180b6650 legacy=0x7ff1180afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0f8003730 0x7ff0f8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0f80044e0 0x7ff0f8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0f8004e50 0x7ff0f800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff118067730 con 0x7ff0f8004e50 2020-11-20T12:16:09.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8004260 con 0x7ff0f80044e0 2020-11-20T12:16:09.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.144+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f800bc20 con 0x7ff0f8003730 2020-11-20T12:16:09.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff10f7fe700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0f80044e0 0x7ff0f8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff117fff700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0f8003730 0x7ff0f8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff10f7fe700 1 -- 172.21.15.187:0/1754732667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0f8003730 msgr2=0x7ff0f8003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:09.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff10f7fe700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0f8003730 0x7ff0f8003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff10f7fe700 1 -- 172.21.15.187:0/1754732667 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0f8004e50 msgr2=0x7ff0f800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:09.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff10f7fe700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0f8004e50 0x7ff0f800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff10f7fe700 1 -- 172.21.15.187:0/1754732667 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f8003240 con 0x7ff0f80044e0 2020-11-20T12:16:09.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff117fff700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0f8003730 0x7ff0f8003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:09.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff11ccaf700 1 -- 172.21.15.187:0/1754732667 reap_dead start 2020-11-20T12:16:09.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.145+0000 7ff10f7fe700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0f80044e0 0x7ff0f8004960 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7ff10000c410 tx=0x7ff1000077d0).ready entity=mon.1 client_cookie=f4c6f73d920bbd7e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:09.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.146+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10000ce80 con 0x7ff0f80044e0 2020-11-20T12:16:09.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.146+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1000072e0 con 0x7ff0f80044e0 2020-11-20T12:16:09.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.146+0000 7ff1177fe700 1 -- 172.21.15.187:0/1754732667 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100005800 con 0x7ff0f80044e0 2020-11-20T12:16:09.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.146+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1754732667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0f80044e0 msgr2=0x7ff0f8004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.146+0000 7ff11d4b0700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0f80044e0 0x7ff0f8004960 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7ff10000c410 tx=0x7ff1000077d0).stop 2020-11-20T12:16:09.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.146+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1754732667 shutdown_connections 2020-11-20T12:16:09.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.146+0000 7ff11d4b0700 1 --2- 172.21.15.187:0/1754732667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0f80044e0 0x7ff0f8004960 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.146+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1754732667 >> 172.21.15.187:0/1754732667 conn(0x7ff1180bab30 msgr2=0x7ff1180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:09.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.147+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1754732667 shutdown_connections 2020-11-20T12:16:09.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.147+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1754732667 wait complete. 2020-11-20T12:16:09.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.148+0000 7ff11d4b0700 1 Processor -- start 2020-11-20T12:16:09.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.148+0000 7ff11d4b0700 1 -- start start 2020-11-20T12:16:09.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.149+0000 7ff11d4b0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0f80044e0 0x7ff11809f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.149+0000 7ff11d4b0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11809fa40 0x7ff118147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.149+0000 7ff11d4b0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff11813ca80 0x7ff11813cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.149+0000 7ff11d4b0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11813add0 con 0x7ff0f80044e0 2020-11-20T12:16:09.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.149+0000 7ff11d4b0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11813b070 con 0x7ff11813ca80 2020-11-20T12:16:09.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.149+0000 7ff11d4b0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11813b310 con 0x7ff11809fa40 2020-11-20T12:16:09.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.149+0000 7ff11ccaf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff11813ca80 0x7ff11813cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff11ccaf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff11813ca80 0x7ff11813cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50766/0 (socket says 172.21.15.187:50766) 2020-11-20T12:16:09.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff11ccaf700 1 -- 172.21.15.187:0/1462550220 learned_addr learned my addr 172.21.15.187:0/1462550220 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:09.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff11ccaf700 1 -- 172.21.15.187:0/1462550220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11809fa40 msgr2=0x7ff118147300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff11ccaf700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11809fa40 0x7ff118147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff11ccaf700 1 -- 172.21.15.187:0/1462550220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0f80044e0 msgr2=0x7ff11809f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff11ccaf700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0f80044e0 0x7ff11809f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff11ccaf700 1 -- 172.21.15.187:0/1462550220 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1181477f0 con 0x7ff11813ca80 2020-11-20T12:16:09.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff11ccaf700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff11813ca80 0x7ff11813cf00 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7ff118044fd0 tx=0x7ff110008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:09.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.150+0000 7ff114ff9700 1 -- 172.21.15.187:0/1462550220 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110017070 con 0x7ff11813ca80 2020-11-20T12:16:09.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.151+0000 7ff114ff9700 1 -- 172.21.15.187:0/1462550220 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff110013430 con 0x7ff11813ca80 2020-11-20T12:16:09.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.151+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11809dcb0 con 0x7ff11813ca80 2020-11-20T12:16:09.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.151+0000 7ff114ff9700 1 -- 172.21.15.187:0/1462550220 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1100137b0 con 0x7ff11813ca80 2020-11-20T12:16:09.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.151+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1180b2140 con 0x7ff11813ca80 2020-11-20T12:16:09.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.152+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff118061a20 con 0x7ff11813ca80 2020-11-20T12:16:09.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.152+0000 7ff114ff9700 1 -- 172.21.15.187:0/1462550220 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff110013ba0 con 0x7ff11813ca80 2020-11-20T12:16:09.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.153+0000 7ff114ff9700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0f4056d20 0x7ff0f40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.153+0000 7ff114ff9700 1 -- 172.21.15.187:0/1462550220 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff11006f980 con 0x7ff11813ca80 2020-11-20T12:16:09.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.153+0000 7ff117fff700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0f4056d20 0x7ff0f40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.154+0000 7ff117fff700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0f4056d20 0x7ff0f40591e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7ff11805ea30 tx=0x7ff108009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:09.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.157+0000 7ff114ff9700 1 -- 172.21.15.187:0/1462550220 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff110044d70 con 0x7ff11813ca80 2020-11-20T12:16:09.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.345+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1181488b0 con 0x7ff11813ca80 2020-11-20T12:16:09.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.348+0000 7ff114ff9700 1 -- 172.21.15.187:0/1462550220 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16791 (secure 0 0 0) 0x7ff110048390 con 0x7ff11813ca80 2020-11-20T12:16:09.351 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:09.351 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":119,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:16:04.871789+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:16:00.509740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2020-11-20T12:16:02.486990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:16:09.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0f4056d20 msgr2=0x7ff0f40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0f4056d20 0x7ff0f40591e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7ff11805ea30 tx=0x7ff108009040).stop 2020-11-20T12:16:09.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff11813ca80 msgr2=0x7ff11813cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff11813ca80 0x7ff11813cf00 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7ff118044fd0 tx=0x7ff110008040).stop 2020-11-20T12:16:09.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 shutdown_connections 2020-11-20T12:16:09.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff0f4056d20 0x7ff0f40591e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0f80044e0 0x7ff11809f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff11813ca80 0x7ff11813cf00 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 --2- 172.21.15.187:0/1462550220 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11809fa40 0x7ff118147300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 >> 172.21.15.187:0/1462550220 conn(0x7ff1180bab30 msgr2=0x7ff1180ba220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 shutdown_connections 2020-11-20T12:16:09.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.350+0000 7ff11d4b0700 1 -- 172.21.15.187:0/1462550220 wait complete. 2020-11-20T12:16:09.378 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:16:09.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.620+0000 7facbec53700 1 Processor -- start 2020-11-20T12:16:09.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.621+0000 7facbec53700 1 -- start start 2020-11-20T12:16:09.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.622+0000 7facbec53700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7facb8065190 0x7facb8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.622+0000 7facbec53700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb809ca90 0x7facb809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.622+0000 7facbec53700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb80b13e0 con 0x7facb8065f90 2020-11-20T12:16:09.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.622+0000 7facbec53700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb814ab30 con 0x7facb8065190 2020-11-20T12:16:09.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.622+0000 7facbec53700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb8062c10 con 0x7facb809ca90 2020-11-20T12:16:09.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.623+0000 7facbd450700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7facb8065f90 0x7facb8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47380/0 (socket says 172.21.15.187:47380) 2020-11-20T12:16:09.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.623+0000 7facbe452700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb809ca90 0x7facb809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.623+0000 7facbd450700 1 -- 172.21.15.187:0/695262707 learned_addr learned my addr 172.21.15.187:0/695262707 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:09.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.623+0000 7facbdc51700 1 --2- 172.21.15.187:0/695262707 >> v2:172.21.15.5:3300/0 conn(0x7facb8065190 0x7facb8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.624+0000 7facbcc4f700 1 -- 172.21.15.187:0/695262707 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990465576 0 0) 0x7facb80b13e0 con 0x7facb8065f90 2020-11-20T12:16:09.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.624+0000 7facbcc4f700 1 -- 172.21.15.187:0/695262707 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca0003240 con 0x7facb8065f90 2020-11-20T12:16:09.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.624+0000 7facbe452700 1 -- 172.21.15.187:0/695262707 >> v2:172.21.15.5:3300/0 conn(0x7facb8065190 msgr2=0x7facb8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:09.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.624+0000 7facbe452700 1 --2- 172.21.15.187:0/695262707 >> v2:172.21.15.5:3300/0 conn(0x7facb8065190 0x7facb8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.624+0000 7facbe452700 1 -- 172.21.15.187:0/695262707 >> v1:172.21.15.187:6789/0 conn(0x7facb8065f90 legacy=0x7facb8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.624+0000 7facbe452700 1 -- 172.21.15.187:0/695262707 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb80b1640 con 0x7facb809ca90 2020-11-20T12:16:09.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.625+0000 7facbe452700 1 --2- 172.21.15.187:0/695262707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb809ca90 0x7facb809cf40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7facb0009c90 tx=0x7facb00091d0).ready entity=mon.2 client_cookie=66fe6cdf774388ff server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:09.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.625+0000 7facbcc4f700 1 -- 172.21.15.187:0/695262707 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb000ba50 con 0x7facb809ca90 2020-11-20T12:16:09.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.625+0000 7facbcc4f700 1 -- 172.21.15.187:0/695262707 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb000bbb0 con 0x7facb809ca90 2020-11-20T12:16:09.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.625+0000 7facbcc4f700 1 -- 172.21.15.187:0/695262707 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0011400 con 0x7facb809ca90 2020-11-20T12:16:09.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.626+0000 7facbec53700 1 -- 172.21.15.187:0/695262707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb809ca90 msgr2=0x7facb809cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.626+0000 7facbec53700 1 --2- 172.21.15.187:0/695262707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb809ca90 0x7facb809cf40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7facb0009c90 tx=0x7facb00091d0).stop 2020-11-20T12:16:09.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.626+0000 7facbec53700 1 -- 172.21.15.187:0/695262707 shutdown_connections 2020-11-20T12:16:09.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.626+0000 7facbec53700 1 --2- 172.21.15.187:0/695262707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb809ca90 0x7facb809cf40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.626+0000 7facbec53700 1 --2- 172.21.15.187:0/695262707 >> v2:172.21.15.5:3300/0 conn(0x7facb8065190 0x7facb8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.626+0000 7facbec53700 1 -- 172.21.15.187:0/695262707 >> 172.21.15.187:0/695262707 conn(0x7facb80bab30 msgr2=0x7facb80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:09.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.626+0000 7facbec53700 1 -- 172.21.15.187:0/695262707 shutdown_connections 2020-11-20T12:16:09.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.626+0000 7facbec53700 1 -- 172.21.15.187:0/695262707 wait complete. 2020-11-20T12:16:09.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.627+0000 7facbec53700 1 Processor -- start 2020-11-20T12:16:09.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.628+0000 7facbec53700 1 -- start start 2020-11-20T12:16:09.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.628+0000 7facbec53700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7facb8065190 0x7facb80b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.628+0000 7facbec53700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb8065f90 0x7facb809ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.628+0000 7facbec53700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facb809ca90 0x7facb8143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.628+0000 7facbec53700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb80b2e30 con 0x7facb8065190 2020-11-20T12:16:09.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.628+0000 7facbec53700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb80a0380 con 0x7facb809ca90 2020-11-20T12:16:09.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.628+0000 7facbec53700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb80a05e0 con 0x7facb8065f90 2020-11-20T12:16:09.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facb809ca90 0x7facb8143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facb809ca90 0x7facb8143100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50782/0 (socket says 172.21.15.187:50782) 2020-11-20T12:16:09.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 -- 172.21.15.187:0/765414881 learned_addr learned my addr 172.21.15.187:0/765414881 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:09.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 -- 172.21.15.187:0/765414881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb8065f90 msgr2=0x7facb809ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb8065f90 0x7facb809ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 -- 172.21.15.187:0/765414881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7facb8065190 msgr2=0x7facb80b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7facb8065190 0x7facb80b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 -- 172.21.15.187:0/765414881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb80b2170 con 0x7facb809ca90 2020-11-20T12:16:09.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.629+0000 7facbe452700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facb809ca90 0x7facb8143100 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 rx=0x7facb0009210 tx=0x7facb0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:09.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.630+0000 7faca5ffb700 1 -- 172.21.15.187:0/765414881 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0006ea0 con 0x7facb809ca90 2020-11-20T12:16:09.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.630+0000 7faca5ffb700 1 -- 172.21.15.187:0/765414881 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb0010780 con 0x7facb809ca90 2020-11-20T12:16:09.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.630+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb809dae0 con 0x7facb809ca90 2020-11-20T12:16:09.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.630+0000 7faca5ffb700 1 -- 172.21.15.187:0/765414881 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0010b20 con 0x7facb809ca90 2020-11-20T12:16:09.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.630+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb809fb90 con 0x7facb809ca90 2020-11-20T12:16:09.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.631+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb80a07a0 con 0x7facb809ca90 2020-11-20T12:16:09.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.631+0000 7faca5ffb700 1 -- 172.21.15.187:0/765414881 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7facb0017070 con 0x7facb809ca90 2020-11-20T12:16:09.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.632+0000 7faca5ffb700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fac90056d50 0x7fac90059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:09.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.632+0000 7faca5ffb700 1 -- 172.21.15.187:0/765414881 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7facb0072460 con 0x7facb809ca90 2020-11-20T12:16:09.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.633+0000 7facbdc51700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fac90056d50 0x7fac90059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:09.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.633+0000 7facbdc51700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fac90056d50 0x7fac90059210 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7facb4006fd0 tx=0x7facb4006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:09.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.636+0000 7faca5ffb700 1 -- 172.21.15.187:0/765414881 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7facb00477a0 con 0x7facb809ca90 2020-11-20T12:16:09.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.900+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7facb80a09a0 con 0x7facb809ca90 2020-11-20T12:16:09.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.911+0000 7faca5ffb700 1 -- 172.21.15.187:0/765414881 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 184+0+0 (secure 0 0 0) 0x7facb004adc0 con 0x7facb809ca90 2020-11-20T12:16:09.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fac90056d50 msgr2=0x7fac90059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fac90056d50 0x7fac90059210 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7facb4006fd0 tx=0x7facb4006df0).stop 2020-11-20T12:16:09.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facb809ca90 msgr2=0x7facb8143100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:09.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facb809ca90 0x7facb8143100 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 rx=0x7facb0009210 tx=0x7facb0016000).stop 2020-11-20T12:16:09.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 shutdown_connections 2020-11-20T12:16:09.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fac90056d50 0x7fac90059210 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7facb8065190 0x7facb80b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facb809ca90 0x7facb8143100 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:09.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 --2- 172.21.15.187:0/765414881 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7facb8065f90 0x7facb809ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:09.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.913+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 >> 172.21.15.187:0/765414881 conn(0x7facb80bab30 msgr2=0x7facb809c240 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:09.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.914+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 shutdown_connections 2020-11-20T12:16:09.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:09.914+0000 7facbec53700 1 -- 172.21.15.187:0/765414881 wait complete. 2020-11-20T12:16:09.921 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:16:09.940 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:10.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.178+0000 7ff5049a9700 1 Processor -- start 2020-11-20T12:16:10.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.179+0000 7ff5049a9700 1 -- start start 2020-11-20T12:16:10.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.180+0000 7ff5049a9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff500065190 0x7ff500065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:10.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.180+0000 7ff5049a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50009ca90 0x7ff50009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.180+0000 7ff5049a9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5000b13e0 con 0x7ff500065f90 2020-11-20T12:16:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.180+0000 7ff5049a9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50014ab30 con 0x7ff500065190 2020-11-20T12:16:10.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.180+0000 7ff5049a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff500062c10 con 0x7ff50009ca90 2020-11-20T12:16:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.181+0000 7ff4fed9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff500065190 0x7ff500065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.181+0000 7ff4fed9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff500065190 0x7ff500065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:43050/0 (socket says 172.21.15.187:43050) 2020-11-20T12:16:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.181+0000 7ff4fed9d700 1 -- 172.21.15.187:0/1978804306 learned_addr learned my addr 172.21.15.187:0/1978804306 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:10.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.181+0000 7ff4ff59e700 1 --2- 172.21.15.187:0/1978804306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50009ca90 0x7ff50009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:10.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.182+0000 7ff4fed9d700 1 -- 172.21.15.187:0/1978804306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50009ca90 msgr2=0x7ff50009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:10.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.182+0000 7ff4fed9d700 1 --2- 172.21.15.187:0/1978804306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50009ca90 0x7ff50009cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.182+0000 7ff4fed9d700 1 -- 172.21.15.187:0/1978804306 >> v1:172.21.15.5:6789/0 conn(0x7ff500065f90 legacy=0x7ff500143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:10.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.182+0000 7ff4fed9d700 1 -- 172.21.15.187:0/1978804306 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5000b1640 con 0x7ff500065190 2020-11-20T12:16:10.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.183+0000 7ff4fed9d700 1 --2- 172.21.15.187:0/1978804306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff500065190 0x7ff500065570 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7ff4f0006fd0 tx=0x7ff4f0006eb0).ready entity=mon.0 client_cookie=916c78238ad74579 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:10.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.183+0000 7ff4fe59c700 1 -- 172.21.15.187:0/1978804306 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00098a0 con 0x7ff500065190 2020-11-20T12:16:10.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.183+0000 7ff4fe59c700 1 -- 172.21.15.187:0/1978804306 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f0009a00 con 0x7ff500065190 2020-11-20T12:16:10.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.183+0000 7ff4fe59c700 1 -- 172.21.15.187:0/1978804306 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0009d80 con 0x7ff500065190 2020-11-20T12:16:10.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.184+0000 7ff5049a9700 1 -- 172.21.15.187:0/1978804306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff500065190 msgr2=0x7ff500065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:10.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.184+0000 7ff5049a9700 1 --2- 172.21.15.187:0/1978804306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff500065190 0x7ff500065570 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7ff4f0006fd0 tx=0x7ff4f0006eb0).stop 2020-11-20T12:16:10.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.184+0000 7ff5049a9700 1 -- 172.21.15.187:0/1978804306 shutdown_connections 2020-11-20T12:16:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.184+0000 7ff5049a9700 1 --2- 172.21.15.187:0/1978804306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff50009ca90 0x7ff50009cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.184+0000 7ff5049a9700 1 --2- 172.21.15.187:0/1978804306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff500065190 0x7ff500065570 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.184+0000 7ff5049a9700 1 -- 172.21.15.187:0/1978804306 >> 172.21.15.187:0/1978804306 conn(0x7ff5000bab30 msgr2=0x7ff5000baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:10.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.185+0000 7ff5049a9700 1 -- 172.21.15.187:0/1978804306 shutdown_connections 2020-11-20T12:16:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.185+0000 7ff5049a9700 1 -- 172.21.15.187:0/1978804306 wait complete. 2020-11-20T12:16:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.186+0000 7ff5049a9700 1 Processor -- start 2020-11-20T12:16:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.186+0000 7ff5049a9700 1 -- start start 2020-11-20T12:16:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.187+0000 7ff5049a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff500065190 0x7ff5000b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:10.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.187+0000 7ff5049a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff500065f90 0x7ff5000b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.187+0000 7ff5049a9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50009ca90 0x7ff50009eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.187+0000 7ff5049a9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000671c0 con 0x7ff50009ca90 2020-11-20T12:16:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.187+0000 7ff5049a9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff500067460 con 0x7ff500065190 2020-11-20T12:16:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.187+0000 7ff5049a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff500067700 con 0x7ff500065f90 2020-11-20T12:16:10.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff500065190 0x7ff5000b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:10.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff500065190 0x7ff5000b2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50794/0 (socket says 172.21.15.187:50794) 2020-11-20T12:16:10.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 -- 172.21.15.187:0/3998548610 learned_addr learned my addr 172.21.15.187:0/3998548610 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:10.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4f659c700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff500065f90 0x7ff5000b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:10.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4ff59e700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50009ca90 0x7ff50009eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:10.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 -- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff500065f90 msgr2=0x7ff5000b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:10.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff500065f90 0x7ff5000b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 -- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50009ca90 msgr2=0x7ff50009eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:10.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50009ca90 0x7ff50009eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 -- 172.21.15.187:0/3998548610 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5000b45e0 con 0x7ff500065190 2020-11-20T12:16:10.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4f659c700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff500065f90 0x7ff5000b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:10.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.188+0000 7ff4fed9d700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff500065190 0x7ff5000b2730 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 rx=0x7ff4f000b1b0 tx=0x7ff4f0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:10.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.189+0000 7ff4f77fe700 1 -- 172.21.15.187:0/3998548610 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0011e50 con 0x7ff500065190 2020-11-20T12:16:10.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.189+0000 7ff4f77fe700 1 -- 172.21.15.187:0/3998548610 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f001e460 con 0x7ff500065190 2020-11-20T12:16:10.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.189+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff500066e40 con 0x7ff500065190 2020-11-20T12:16:10.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.189+0000 7ff4f77fe700 1 -- 172.21.15.187:0/3998548610 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f001e7e0 con 0x7ff500065190 2020-11-20T12:16:10.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.189+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50009efa0 con 0x7ff500065190 2020-11-20T12:16:10.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.190+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5000678c0 con 0x7ff500065190 2020-11-20T12:16:10.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.191+0000 7ff4f77fe700 1 -- 172.21.15.187:0/3998548610 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff4f0025030 con 0x7ff500065190 2020-11-20T12:16:10.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.191+0000 7ff4f77fe700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4dc056d20 0x7ff4dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:10.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.191+0000 7ff4f77fe700 1 -- 172.21.15.187:0/3998548610 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff4f00717f0 con 0x7ff500065190 2020-11-20T12:16:10.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.191+0000 7ff4f659c700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4dc056d20 0x7ff4dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:10.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.192+0000 7ff4f659c700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4dc056d20 0x7ff4dc0591e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7ff5000b3c60 tx=0x7ff4e8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:10.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.195+0000 7ff4f77fe700 1 -- 172.21.15.187:0/3998548610 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff4f0046cf0 con 0x7ff500065190 2020-11-20T12:16:10.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.444+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff500061a20 con 0x7ff500065190 2020-11-20T12:16:10.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.445+0000 7ff4f77fe700 1 -- 172.21.15.187:0/3998548610 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2141 (secure 0 0 0) 0x7ff4f0023070 con 0x7ff500065190 2020-11-20T12:16:10.448 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:10.449 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":102,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2092825437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2092825437},{"type":"v1","addr":"172.21.15.5:6833","nonce":2092825437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":99}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:05.876197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8473},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8473":{"gid":8473,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3143922581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3143922581},{"type":"v1","addr":"172.21.15.187:6835","nonce":3143922581}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:10.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.448+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4dc056d20 msgr2=0x7ff4dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:10.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.448+0000 7ff5049a9700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4dc056d20 0x7ff4dc0591e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7ff5000b3c60 tx=0x7ff4e8009040).stop 2020-11-20T12:16:10.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.448+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff500065190 msgr2=0x7ff5000b2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:10.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.448+0000 7ff5049a9700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff500065190 0x7ff5000b2730 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 rx=0x7ff4f000b1b0 tx=0x7ff4f0015000).stop 2020-11-20T12:16:10.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.449+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 shutdown_connections 2020-11-20T12:16:10.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.449+0000 7ff5049a9700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff4dc056d20 0x7ff4dc0591e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.449+0000 7ff5049a9700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff50009ca90 0x7ff50009eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.449+0000 7ff5049a9700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff500065190 0x7ff5000b2730 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.449+0000 7ff5049a9700 1 --2- 172.21.15.187:0/3998548610 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff500065f90 0x7ff5000b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:10.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.449+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 >> 172.21.15.187:0/3998548610 conn(0x7ff5000bab30 msgr2=0x7ff5000aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:10.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.449+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 shutdown_connections 2020-11-20T12:16:10.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:10.449+0000 7ff5049a9700 1 -- 172.21.15.187:0/3998548610 wait complete. 2020-11-20T12:16:10.456 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 102 2020-11-20T12:16:10.675 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:16:03.867608+0000', 'modified': '2020-11-20T12:16:05.876197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8473}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8473': {'gid': 8473, 'name': 'a', 'rank': 0, 'incarnation': 100, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3143922581', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3143922581}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3143922581}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:16:10.676 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:16:10.676 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:16:10.953 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8473,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":7.0563070730000002,"mdsmap_epoch":102,"osdmap_epoch":119,"osdmap_epoch_barrier":119,"uptime":14.680806072999999} 2020-11-20T12:16:10.953 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8473, "mdsmap_epoch": 102, "osdmap_epoch": 119, "osdmap_epoch_barrier": 119, "rank_uptime": 7.056307073, "state": "up:active", "uptime": 14.680806073, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:16:10.954 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:16:10.954 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:16:10.955 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:16:10.955 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:16:10.955 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:16:10.956 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:16:10.956 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:16:10.956 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:16:10.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:16:10.957 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:16:11.018 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:16:11.019 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:16:11.019 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:16:11.019 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:16:11.019 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:16:11.019 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:16:11.020 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:16:11.020 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:16:11.020 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:16:11.020 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:16:11.021 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:16:11.021 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:16:11.021 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:16:11.021 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:16:11.021 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:16:11.022 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42067sec preferred_lft 42067sec 2020-11-20T12:16:11.022 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:16:11.022 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:16:11.022 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:16:11.023 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:16:11.023 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:16:11.023 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:16:11.023 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:16:11.023 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:16:11.024 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:16:11.024 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:16:11.024 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:16:11.024 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:16:11.024 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:16:11.025 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:16:11.025 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:16:11.025 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:16:11.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:16:11.026 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:16:11.128 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:16:11.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:16:11.134 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:16:11.194 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:16:11.195 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:16:11.196 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:16:11.197 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:16:11.197 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:16:11.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:16:11.200 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:16:11.201 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:16:11.203 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:16:11.203 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:16:11.204 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:16:11.205 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:16:11.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:16:11.475 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:16:11.526 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:16:11.527 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:16:11.528 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:16:11.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:16:11.529 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:16:11.634 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:16:11.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.877+0000 7f4020c1c700 1 Processor -- start 2020-11-20T12:16:11.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.878+0000 7f4020c1c700 1 -- start start 2020-11-20T12:16:11.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.879+0000 7f4020c1c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f401c0b57b0 0x7f401c068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:11.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.879+0000 7f4020c1c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 0x7f401c062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:11.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.879+0000 7f4020c1c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c144b80 con 0x7f401c062b50 2020-11-20T12:16:11.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.879+0000 7f4020c1c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401c1489e0 con 0x7f401c0b57b0 2020-11-20T12:16:11.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.879+0000 7f4020c1c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c148c80 con 0x7f401c1482d0 2020-11-20T12:16:11.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.880+0000 7f401b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f401c0b57b0 0x7f401c068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:11.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.880+0000 7f401bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f401c062b50 0x7f401c065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47410/0 (socket says 172.21.15.187:47410) 2020-11-20T12:16:11.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.880+0000 7f401bfff700 1 -- 172.21.15.187:0/255993173 learned_addr learned my addr 172.21.15.187:0/255993173 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:11.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.880+0000 7f4012ffd700 1 --2- 172.21.15.187:0/255993173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 0x7f401c062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:11.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.880+0000 7f401affd700 1 -- 172.21.15.187:0/255993173 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904911788 0 0) 0x7f401c144b80 con 0x7f401c062b50 2020-11-20T12:16:11.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.881+0000 7f401b7fe700 1 -- 172.21.15.187:0/255993173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 msgr2=0x7f401c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:11.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.881+0000 7f401b7fe700 1 --2- 172.21.15.187:0/255993173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 0x7f401c062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:11.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.881+0000 7f401b7fe700 1 -- 172.21.15.187:0/255993173 >> v1:172.21.15.187:6789/0 conn(0x7f401c062b50 legacy=0x7f401c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:11.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.881+0000 7f401b7fe700 1 -- 172.21.15.187:0/255993173 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c148f20 con 0x7f401c0b57b0 2020-11-20T12:16:11.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.881+0000 7f401b7fe700 1 --2- 172.21.15.187:0/255993173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c0b57b0 0x7f401c068230 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 rx=0x7f400c00b9a0 tx=0x7f400c006df0).ready entity=mon.1 client_cookie=c5aed3b01adf9c23 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:11.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.881+0000 7f401affd700 1 -- 172.21.15.187:0/255993173 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c0097b0 con 0x7f401c0b57b0 2020-11-20T12:16:11.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.881+0000 7f401affd700 1 -- 172.21.15.187:0/255993173 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c009910 con 0x7f401c0b57b0 2020-11-20T12:16:11.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.881+0000 7f401affd700 1 -- 172.21.15.187:0/255993173 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c009c90 con 0x7f401c0b57b0 2020-11-20T12:16:11.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.882+0000 7f4020c1c700 1 -- 172.21.15.187:0/255993173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c0b57b0 msgr2=0x7f401c068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:11.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.882+0000 7f4020c1c700 1 --2- 172.21.15.187:0/255993173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c0b57b0 0x7f401c068230 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 rx=0x7f400c00b9a0 tx=0x7f400c006df0).stop 2020-11-20T12:16:11.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.882+0000 7f4020c1c700 1 -- 172.21.15.187:0/255993173 shutdown_connections 2020-11-20T12:16:11.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.882+0000 7f4020c1c700 1 --2- 172.21.15.187:0/255993173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c0b57b0 0x7f401c068230 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:11.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.882+0000 7f4020c1c700 1 --2- 172.21.15.187:0/255993173 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 0x7f401c062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:11.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.882+0000 7f4020c1c700 1 -- 172.21.15.187:0/255993173 >> 172.21.15.187:0/255993173 conn(0x7f401c0bab30 msgr2=0x7f401c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:11.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.882+0000 7f4020c1c700 1 -- 172.21.15.187:0/255993173 shutdown_connections 2020-11-20T12:16:11.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.882+0000 7f4020c1c700 1 -- 172.21.15.187:0/255993173 wait complete. 2020-11-20T12:16:11.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.883+0000 7f4020c1c700 1 Processor -- start 2020-11-20T12:16:11.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.884+0000 7f4020c1c700 1 -- start start 2020-11-20T12:16:11.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.884+0000 7f4020c1c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c062b50 0x7f401c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:11.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.884+0000 7f4020c1c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f401c0b57b0 0x7f401c09d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:11.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f4020c1c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 0x7f401c157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:11.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f4020c1c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c157330 con 0x7f401c0b57b0 2020-11-20T12:16:11.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f4020c1c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c157590 con 0x7f401c062b50 2020-11-20T12:16:11.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f4020c1c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c154b90 con 0x7f401c1482d0 2020-11-20T12:16:11.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c062b50 0x7f401c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:11.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c062b50 0x7f401c1424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50806/0 (socket says 172.21.15.187:50806) 2020-11-20T12:16:11.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 -- 172.21.15.187:0/2547704321 learned_addr learned my addr 172.21.15.187:0/2547704321 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:11.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 -- 172.21.15.187:0/2547704321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 msgr2=0x7f401c157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:11.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 0x7f401c157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:11.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 -- 172.21.15.187:0/2547704321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f401c0b57b0 msgr2=0x7f401c09d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:11.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f401c0b57b0 0x7f401c09d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:11.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 -- 172.21.15.187:0/2547704321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c09d4f0 con 0x7f401c062b50 2020-11-20T12:16:11.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.885+0000 7f401b7fe700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c062b50 0x7f401c1424e0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 rx=0x7f400c006e30 tx=0x7f400c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:11.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.886+0000 7f4013fff700 1 -- 172.21.15.187:0/2547704321 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c01e460 con 0x7f401c062b50 2020-11-20T12:16:11.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.886+0000 7f4013fff700 1 -- 172.21.15.187:0/2547704321 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c01e5c0 con 0x7f401c062b50 2020-11-20T12:16:11.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.886+0000 7f4013fff700 1 -- 172.21.15.187:0/2547704321 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c01e960 con 0x7f401c062b50 2020-11-20T12:16:11.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.886+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c142c60 con 0x7f401c062b50 2020-11-20T12:16:11.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.886+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c1548e0 con 0x7f401c062b50 2020-11-20T12:16:11.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.887+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401c157760 con 0x7f401c062b50 2020-11-20T12:16:11.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.888+0000 7f4013fff700 1 -- 172.21.15.187:0/2547704321 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f400c025030 con 0x7f401c062b50 2020-11-20T12:16:11.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.888+0000 7f4013fff700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ff8056d20 0x7f3ff80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:11.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.888+0000 7f4013fff700 1 -- 172.21.15.187:0/2547704321 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f400c0719a0 con 0x7f401c062b50 2020-11-20T12:16:11.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.889+0000 7f4012ffd700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ff8056d20 0x7f3ff80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:11.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.889+0000 7f4012ffd700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ff8056d20 0x7f3ff80591e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f401c062b20 tx=0x7f4004008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:11.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:11.893+0000 7f4013fff700 1 -- 172.21.15.187:0/2547704321 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f400c046e40 con 0x7f401c062b50 2020-11-20T12:16:12.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.080+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f401c061a20 con 0x7f401c062b50 2020-11-20T12:16:12.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.082+0000 7f4013fff700 1 -- 172.21.15.187:0/2547704321 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16791 (secure 0 0 0) 0x7f400c023070 con 0x7f401c062b50 2020-11-20T12:16:12.085 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:12.086 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":119,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:16:04.871789+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:16:00.509740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2020-11-20T12:16:02.486990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:16:12.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ff8056d20 msgr2=0x7f3ff80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:12.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ff8056d20 0x7f3ff80591e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f401c062b20 tx=0x7f4004008040).stop 2020-11-20T12:16:12.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c062b50 msgr2=0x7f401c1424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:12.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c062b50 0x7f401c1424e0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 rx=0x7f400c006e30 tx=0x7f400c015000).stop 2020-11-20T12:16:12.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 shutdown_connections 2020-11-20T12:16:12.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3ff8056d20 0x7f3ff80591e0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:12.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f401c1482d0 0x7f401c157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:12.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f401c062b50 0x7f401c1424e0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:12.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 --2- 172.21.15.187:0/2547704321 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f401c0b57b0 0x7f401c09d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:12.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 >> 172.21.15.187:0/2547704321 conn(0x7f401c0bab30 msgr2=0x7f401c0b9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:12.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 shutdown_connections 2020-11-20T12:16:12.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.085+0000 7f4020c1c700 1 -- 172.21.15.187:0/2547704321 wait complete. 2020-11-20T12:16:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:16:00.509740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:16:02.486990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:16:12.564 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:16:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.829+0000 7f55a0a39700 1 Processor -- start 2020-11-20T12:16:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.829+0000 7f55a0a39700 1 -- start start 2020-11-20T12:16:12.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.830+0000 7f55a0a39700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 0x7f559c068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:12.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.830+0000 7f55a0a39700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c144b80 con 0x7f559c0b5f50 2020-11-20T12:16:12.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.830+0000 7f55a0a39700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c1489e0 con 0x7f559c1482d0 2020-11-20T12:16:12.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.830+0000 7f55a0a39700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c148c80 con 0x7f559c0b57b0 2020-11-20T12:16:12.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.831+0000 7f559a59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f559c1482d0 0x7f559c062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47424/0 (socket says 172.21.15.187:47424) 2020-11-20T12:16:12.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.831+0000 7f559a59c700 1 -- 172.21.15.187:0/849095370 learned_addr learned my addr 172.21.15.187:0/849095370 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:12.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.831+0000 7f559ad9d700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 0x7f559c068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:12.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009052675 0 0) 0x7f559c1489e0 con 0x7f559c1482d0 2020-11-20T12:16:12.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5588004350 con 0x7f559c1482d0 2020-11-20T12:16:12.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1047200460 0 0) 0x7f5588004350 con 0x7f559c1482d0 2020-11-20T12:16:12.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 msgr2=0x7f559c068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:12.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 0x7f559c068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:12.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 >> v1:172.21.15.5:6789/0 conn(0x7f559c0b5f50 legacy=0x7f559c0b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:12.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c148f20 con 0x7f559c1482d0 2020-11-20T12:16:12.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f55900014e0 con 0x7f559c1482d0 2020-11-20T12:16:12.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 >> v1:172.21.15.187:6789/0 conn(0x7f559c1482d0 legacy=0x7f559c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:12.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 0x7f5588004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:12.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5588004d30 0x7f55880055f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:12.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.832+0000 7f5599d9b700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5588005ae0 0x7f5588005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:12.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c148c80 con 0x7f5588005ae0 2020-11-20T12:16:12.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588005250 con 0x7f5588004d30 2020-11-20T12:16:12.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5588005410 con 0x7f559c0b57b0 2020-11-20T12:16:12.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f559a59c700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5588004d30 0x7f55880055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:12.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f559ad9d700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 0x7f5588004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:12.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f559a59c700 1 -- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 msgr2=0x7f5588004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:12.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f559a59c700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 0x7f5588004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:12.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f559a59c700 1 -- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5588005ae0 msgr2=0x7f5588005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:12.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f559a59c700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5588005ae0 0x7f5588005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:12.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f559a59c700 1 -- 172.21.15.187:0/849095370 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588004350 con 0x7f5588004d30 2020-11-20T12:16:12.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.833+0000 7f559ad9d700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b57b0 0x7f5588004840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:12.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f559a59c700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5588004d30 0x7f55880055f0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 rx=0x7f559000c410 tx=0x7f55900077d0).ready entity=mon.1 client_cookie=31b7589fb159f2e6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:12.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559000ce80 con 0x7f5588004d30 2020-11-20T12:16:12.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55900072e0 con 0x7f5588004d30 2020-11-20T12:16:12.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f5599d9b700 1 -- 172.21.15.187:0/849095370 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590005820 con 0x7f5588004d30 2020-11-20T12:16:12.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f55a0a39700 1 -- 172.21.15.187:0/849095370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5588004d30 msgr2=0x7f55880055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:12.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f55a0a39700 1 --2- 172.21.15.187:0/849095370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5588004d30 0x7f55880055f0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 rx=0x7f559000c410 tx=0x7f55900077d0).stop 2020-11-20T12:16:12.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f559b59e700 1 -- 172.21.15.187:0/849095370 reap_dead start 2020-11-20T12:16:12.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f55a0a39700 1 -- 172.21.15.187:0/849095370 shutdown_connections 2020-11-20T12:16:12.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.834+0000 7f55a0a39700 1 -- 172.21.15.187:0/849095370 >> 172.21.15.187:0/849095370 conn(0x7f559c0bab30 msgr2=0x7f559c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:12.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.835+0000 7f55a0a39700 1 -- 172.21.15.187:0/849095370 shutdown_connections 2020-11-20T12:16:12.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.835+0000 7f55a0a39700 1 -- 172.21.15.187:0/849095370 wait complete. 2020-11-20T12:16:12.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.836+0000 7f55a0a39700 1 Processor -- start 2020-11-20T12:16:12.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.836+0000 7f55a0a39700 1 -- start start 2020-11-20T12:16:12.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f55a0a39700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f559c146170 0x7f559c092e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:12.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f55a0a39700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b3140 0x7f559c093320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:12.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f55a0a39700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f559c146d00 0x7f559c15b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:12.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f55a0a39700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c159120 con 0x7f559c146d00 2020-11-20T12:16:12.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f55a0a39700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c147bc0 con 0x7f559c146170 2020-11-20T12:16:12.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f55a0a39700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c093930 con 0x7f559c0b3140 2020-11-20T12:16:12.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f559ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f559c146170 0x7f559c092e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:12.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f559ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f559c146170 0x7f559c092e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50826/0 (socket says 172.21.15.187:50826) 2020-11-20T12:16:12.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.837+0000 7f559ad9d700 1 -- 172.21.15.187:0/2291596399 learned_addr learned my addr 172.21.15.187:0/2291596399 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:12.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f559a59c700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b3140 0x7f559c093320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:12.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f559ad9d700 1 -- 172.21.15.187:0/2291596399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b3140 msgr2=0x7f559c093320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:12.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f559ad9d700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b3140 0x7f559c093320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:12.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f559ad9d700 1 -- 172.21.15.187:0/2291596399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f559c146d00 msgr2=0x7f559c15b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:12.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f559b59e700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f559c146d00 0x7f559c15b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:12.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f559ad9d700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f559c146d00 0x7f559c15b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:12.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f559ad9d700 1 -- 172.21.15.187:0/2291596399 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c093bd0 con 0x7f559c146170 2020-11-20T12:16:12.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f559ad9d700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f559c146170 0x7f559c092e30 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f558c00f830 tx=0x7f558c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:12.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f5582ffd700 1 -- 172.21.15.187:0/2291596399 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c017070 con 0x7f559c146170 2020-11-20T12:16:12.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f5582ffd700 1 -- 172.21.15.187:0/2291596399 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c00d6c0 con 0x7f559c146170 2020-11-20T12:16:12.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f5582ffd700 1 -- 172.21.15.187:0/2291596399 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c00da40 con 0x7f559c146170 2020-11-20T12:16:12.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.838+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c158c80 con 0x7f559c146170 2020-11-20T12:16:12.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.839+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c15bc20 con 0x7f559c146170 2020-11-20T12:16:12.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.840+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c0b8490 con 0x7f559c146170 2020-11-20T12:16:12.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.840+0000 7f5582ffd700 1 -- 172.21.15.187:0/2291596399 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f558c01c410 con 0x7f559c146170 2020-11-20T12:16:12.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.841+0000 7f5582ffd700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5578056d20 0x7f55780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:12.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.841+0000 7f5582ffd700 1 -- 172.21.15.187:0/2291596399 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f558c070170 con 0x7f559c146170 2020-11-20T12:16:12.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.841+0000 7f559a59c700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5578056d20 0x7f55780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:12.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.842+0000 7f559a59c700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5578056d20 0x7f55780591e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f559c05f200 tx=0x7f5590015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:12.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:12.845+0000 7f5582ffd700 1 -- 172.21.15.187:0/2291596399 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f558c045450 con 0x7f559c146170 2020-11-20T12:16:13.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.032+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f559c158e10 con 0x7f559c146170 2020-11-20T12:16:13.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.033+0000 7f5582ffd700 1 -- 172.21.15.187:0/2291596399 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v119) v1 ==== 110+0+10 (secure 0 0 0) 0x7f558c048a70 con 0x7f559c146170 2020-11-20T12:16:13.037 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:16:13.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5578056d20 msgr2=0x7f55780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5578056d20 0x7f55780591e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f559c05f200 tx=0x7f5590015000).stop 2020-11-20T12:16:13.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f559c146170 msgr2=0x7f559c092e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f559c146170 0x7f559c092e30 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f558c00f830 tx=0x7f558c008040).stop 2020-11-20T12:16:13.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 shutdown_connections 2020-11-20T12:16:13.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5578056d20 0x7f55780591e0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f559c146d00 0x7f559c15b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f559c146170 0x7f559c092e30 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 --2- 172.21.15.187:0/2291596399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f559c0b3140 0x7f559c093320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 >> 172.21.15.187:0/2291596399 conn(0x7f559c0bab30 msgr2=0x7f559c0b6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:13.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.036+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 shutdown_connections 2020-11-20T12:16:13.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.037+0000 7f55a0a39700 1 -- 172.21.15.187:0/2291596399 wait complete. 2020-11-20T12:16:13.062 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:16:13.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.310+0000 7f3c3a9e3700 1 Processor -- start 2020-11-20T12:16:13.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.311+0000 7f3c3a9e3700 1 -- start start 2020-11-20T12:16:13.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.312+0000 7f3c3a9e3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3c340b57b0 0x7f3c34068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.312+0000 7f3c3a9e3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c34144b80 con 0x7f3c340b5f50 2020-11-20T12:16:13.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.312+0000 7f3c3a9e3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c341489e0 con 0x7f3c341482d0 2020-11-20T12:16:13.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.312+0000 7f3c3a9e3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c34148c80 con 0x7f3c340b57b0 2020-11-20T12:16:13.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.312+0000 7f3c391e0700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3c341482d0 0x7f3c34062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47442/0 (socket says 172.21.15.187:47442) 2020-11-20T12:16:13.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.312+0000 7f3c391e0700 1 -- 172.21.15.187:0/15447969 learned_addr learned my addr 172.21.15.187:0/15447969 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:13.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c399e1700 1 --2- 172.21.15.187:0/15447969 >> v2:172.21.15.187:3300/0 conn(0x7f3c340b57b0 0x7f3c34068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38446703 0 0) 0x7f3c341489e0 con 0x7f3c341482d0 2020-11-20T12:16:13.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c003240 con 0x7f3c341482d0 2020-11-20T12:16:13.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1446273699 0 0) 0x7f3c1c003240 con 0x7f3c341482d0 2020-11-20T12:16:13.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 >> v2:172.21.15.187:3300/0 conn(0x7f3c340b57b0 msgr2=0x7f3c34068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:13.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c389df700 1 --2- 172.21.15.187:0/15447969 >> v2:172.21.15.187:3300/0 conn(0x7f3c340b57b0 0x7f3c34068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 >> v1:172.21.15.5:6789/0 conn(0x7f3c340b5f50 legacy=0x7f3c340b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c34148f20 con 0x7f3c341482d0 2020-11-20T12:16:13.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.313+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f3c240014e0 con 0x7f3c341482d0 2020-11-20T12:16:13.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 >> v1:172.21.15.187:6789/0 conn(0x7f3c341482d0 legacy=0x7f3c34062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c389df700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c1c003730 0x7f3c1c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c389df700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c1c0044e0 0x7f3c1c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c389df700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c1c004e50 0x7f3c1c005700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c34148c80 con 0x7f3c1c004e50 2020-11-20T12:16:13.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c004260 con 0x7f3c1c003730 2020-11-20T12:16:13.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c005bf0 con 0x7f3c1c0044e0 2020-11-20T12:16:13.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c399e1700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c1c003730 0x7f3c1c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c3a1e2700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c1c004e50 0x7f3c1c005700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.314+0000 7f3c399e1700 1 -- 172.21.15.187:0/15447969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c1c0044e0 msgr2=0x7f3c1c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:13.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c399e1700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c1c0044e0 0x7f3c1c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:13.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c399e1700 1 -- 172.21.15.187:0/15447969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c1c004e50 msgr2=0x7f3c1c005700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:13.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c399e1700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c1c004e50 0x7f3c1c005700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c399e1700 1 -- 172.21.15.187:0/15447969 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c003240 con 0x7f3c1c003730 2020-11-20T12:16:13.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c3a1e2700 1 -- 172.21.15.187:0/15447969 reap_dead start 2020-11-20T12:16:13.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c399e1700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c1c003730 0x7f3c1c003b50 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7f3c3000bc00 tx=0x7f3c30006df0).ready entity=mon.1 client_cookie=c2fe8fc5212ab673 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:13.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3000b310 con 0x7f3c1c003730 2020-11-20T12:16:13.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3000eb40 con 0x7f3c1c003730 2020-11-20T12:16:13.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.315+0000 7f3c389df700 1 -- 172.21.15.187:0/15447969 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3000eec0 con 0x7f3c1c003730 2020-11-20T12:16:13.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.316+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/15447969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c1c003730 msgr2=0x7f3c1c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.316+0000 7f3c3a9e3700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c1c003730 0x7f3c1c003b50 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7f3c3000bc00 tx=0x7f3c30006df0).stop 2020-11-20T12:16:13.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.316+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/15447969 shutdown_connections 2020-11-20T12:16:13.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.316+0000 7f3c3a9e3700 1 --2- 172.21.15.187:0/15447969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c1c003730 0x7f3c1c003b50 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.316+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/15447969 >> 172.21.15.187:0/15447969 conn(0x7f3c340bab30 msgr2=0x7f3c340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:13.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.316+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/15447969 shutdown_connections 2020-11-20T12:16:13.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.316+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/15447969 wait complete. 2020-11-20T12:16:13.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.317+0000 7f3c3a9e3700 1 Processor -- start 2020-11-20T12:16:13.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.318+0000 7f3c3a9e3700 1 -- start start 2020-11-20T12:16:13.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.318+0000 7f3c3a9e3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c34142fb0 0x7f3c3415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.318+0000 7f3c3a9e3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c34143390 0x7f3c3409c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.318+0000 7f3c3a9e3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c3409cd20 0x7f3c34158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.318+0000 7f3c3a9e3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3415c360 con 0x7f3c3409cd20 2020-11-20T12:16:13.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.318+0000 7f3c3a9e3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3415c600 con 0x7f3c34143390 2020-11-20T12:16:13.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.318+0000 7f3c3a9e3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3415c8a0 con 0x7f3c34142fb0 2020-11-20T12:16:13.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c391e0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c34143390 0x7f3c3409c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c391e0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c34143390 0x7f3c3409c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50848/0 (socket says 172.21.15.187:50848) 2020-11-20T12:16:13.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c391e0700 1 -- 172.21.15.187:0/1703074771 learned_addr learned my addr 172.21.15.187:0/1703074771 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:13.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c399e1700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c34142fb0 0x7f3c3415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c391e0700 1 -- 172.21.15.187:0/1703074771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c34142fb0 msgr2=0x7f3c3415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c391e0700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c34142fb0 0x7f3c3415ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c391e0700 1 -- 172.21.15.187:0/1703074771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c3409cd20 msgr2=0x7f3c34158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c391e0700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c3409cd20 0x7f3c34158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:13.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.319+0000 7f3c391e0700 1 -- 172.21.15.187:0/1703074771 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340b1db0 con 0x7f3c34143390 2020-11-20T12:16:13.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.320+0000 7f3c391e0700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c34143390 0x7f3c3409c830 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7f3c2400ca90 tx=0x7f3c34068110).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:13.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.320+0000 7f3c21ffb700 1 -- 172.21.15.187:0/1703074771 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24007170 con 0x7f3c34143390 2020-11-20T12:16:13.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.320+0000 7f3c21ffb700 1 -- 172.21.15.187:0/1703074771 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c240072d0 con 0x7f3c34143390 2020-11-20T12:16:13.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.320+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3415bf40 con 0x7f3c34143390 2020-11-20T12:16:13.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.320+0000 7f3c21ffb700 1 -- 172.21.15.187:0/1703074771 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400aba0 con 0x7f3c34143390 2020-11-20T12:16:13.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.320+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3409d3c0 con 0x7f3c34143390 2020-11-20T12:16:13.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.321+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c34061a20 con 0x7f3c34143390 2020-11-20T12:16:13.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.322+0000 7f3c21ffb700 1 -- 172.21.15.187:0/1703074771 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3c2401b060 con 0x7f3c34143390 2020-11-20T12:16:13.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.322+0000 7f3c21ffb700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c0c056d50 0x7f3c0c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.323+0000 7f3c21ffb700 1 -- 172.21.15.187:0/1703074771 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3c2406fa60 con 0x7f3c34143390 2020-11-20T12:16:13.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.323+0000 7f3c399e1700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c0c056d50 0x7f3c0c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.323+0000 7f3c399e1700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c0c056d50 0x7f3c0c059210 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f3c3405e630 tx=0x7f3c30019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:13.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.326+0000 7f3c21ffb700 1 -- 172.21.15.187:0/1703074771 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3c24044eb0 con 0x7f3c34143390 2020-11-20T12:16:13.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.511+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3c3409d630 con 0x7f3c34143390 2020-11-20T12:16:13.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.512+0000 7f3c21ffb700 1 -- 172.21.15.187:0/1703074771 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v119) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3c24019030 con 0x7f3c34143390 2020-11-20T12:16:13.519 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:16:13.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.514+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c0c056d50 msgr2=0x7f3c0c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.514+0000 7f3c3a9e3700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c0c056d50 0x7f3c0c059210 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f3c3405e630 tx=0x7f3c30019000).stop 2020-11-20T12:16:13.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.514+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c34143390 msgr2=0x7f3c3409c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.514+0000 7f3c3a9e3700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c34143390 0x7f3c3409c830 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7f3c2400ca90 tx=0x7f3c34068110).stop 2020-11-20T12:16:13.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.515+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 shutdown_connections 2020-11-20T12:16:13.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.515+0000 7f3c3a9e3700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c0c056d50 0x7f3c0c059210 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.515+0000 7f3c3a9e3700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c3409cd20 0x7f3c34158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:13.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.515+0000 7f3c3a9e3700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c34142fb0 0x7f3c3415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.515+0000 7f3c3a9e3700 1 --2- 172.21.15.187:0/1703074771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c34143390 0x7f3c3409c830 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.515+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 >> 172.21.15.187:0/1703074771 conn(0x7f3c340bab30 msgr2=0x7f3c340b96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:13.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.515+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 shutdown_connections 2020-11-20T12:16:13.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.515+0000 7f3c3a9e3700 1 -- 172.21.15.187:0/1703074771 wait complete. 2020-11-20T12:16:13.541 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:16:13.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.794+0000 7fa3bd7bf700 1 Processor -- start 2020-11-20T12:16:13.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.795+0000 7fa3bd7bf700 1 -- start start 2020-11-20T12:16:13.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.796+0000 7fa3bd7bf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 0x7fa3b809d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.796+0000 7fa3bd7bf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b80b13e0 con 0x7fa3b8065190 2020-11-20T12:16:13.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.796+0000 7fa3bd7bf700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b809d690 con 0x7fa3b8065f90 2020-11-20T12:16:13.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.796+0000 7fa3bd7bf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b814ab30 con 0x7fa3b8065570 2020-11-20T12:16:13.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.797+0000 7fa3b77fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa3b8065f90 0x7fa3b809ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47462/0 (socket says 172.21.15.187:47462) 2020-11-20T12:16:13.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.797+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1452085200 learned_addr learned my addr 172.21.15.187:0/1452085200 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:13.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.797+0000 7fa3bcfbe700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 0x7fa3b809d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.797+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160010956 0 0) 0x7fa3b809d690 con 0x7fa3b8065f90 2020-11-20T12:16:13.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.797+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a4003240 con 0x7fa3b8065f90 2020-11-20T12:16:13.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1722870855 0 0) 0x7fa3a4003240 con 0x7fa3b8065f90 2020-11-20T12:16:13.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 msgr2=0x7fa3b809d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:13.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 0x7fa3b809d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 >> v1:172.21.15.5:6789/0 conn(0x7fa3b8065190 legacy=0x7fa3b8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8062c10 con 0x7fa3b8065f90 2020-11-20T12:16:13.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fa3ac0014e0 con 0x7fa3b8065f90 2020-11-20T12:16:13.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 >> v1:172.21.15.187:6789/0 conn(0x7fa3b8065f90 legacy=0x7fa3b809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a4003730 0x7fa3a4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 0x7fa3a40044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3a40049d0 0x7fa3a4004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b814ab30 con 0x7fa3a4003730 2020-11-20T12:16:13.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a40041d0 con 0x7fa3a40049d0 2020-11-20T12:16:13.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a400b730 con 0x7fa3b8065570 2020-11-20T12:16:13.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.798+0000 7fa3bcfbe700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3a40049d0 0x7fa3a4004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3b7fff700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a4003730 0x7fa3a4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 0x7fa3a40044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3bcfbe700 1 -- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 msgr2=0x7fa3a40044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:13.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3bcfbe700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 0x7fa3a40044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3bcfbe700 1 -- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a4003730 msgr2=0x7fa3a4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:13.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3bcfbe700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a4003730 0x7fa3a4003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3bcfbe700 1 -- 172.21.15.187:0/1452085200 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a4003240 con 0x7fa3a40049d0 2020-11-20T12:16:13.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b8065570 0x7fa3a40044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:13.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3bcfbe700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3a40049d0 0x7fa3a4004e50 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 rx=0x7fa3b0009ea0 tx=0x7fa3b00091d0).ready entity=mon.1 client_cookie=bdefff0eb6e7375b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:13.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.799+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00095c0 con 0x7fa3a40049d0 2020-11-20T12:16:13.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b000eda0 con 0x7fa3a40049d0 2020-11-20T12:16:13.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3b6ffd700 1 -- 172.21.15.187:0/1452085200 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0012600 con 0x7fa3a40049d0 2020-11-20T12:16:13.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1452085200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3a40049d0 msgr2=0x7fa3a4004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3bd7bf700 1 --2- 172.21.15.187:0/1452085200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3a40049d0 0x7fa3a4004e50 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 rx=0x7fa3b0009ea0 tx=0x7fa3b00091d0).stop 2020-11-20T12:16:13.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3bcfbe700 1 -- 172.21.15.187:0/1452085200 reap_dead start 2020-11-20T12:16:13.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1452085200 shutdown_connections 2020-11-20T12:16:13.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1452085200 >> 172.21.15.187:0/1452085200 conn(0x7fa3b80bab30 msgr2=0x7fa3b80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:13.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1452085200 shutdown_connections 2020-11-20T12:16:13.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.800+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1452085200 wait complete. 2020-11-20T12:16:13.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.802+0000 7fa3bd7bf700 1 Processor -- start 2020-11-20T12:16:13.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.802+0000 7fa3bd7bf700 1 -- start start 2020-11-20T12:16:13.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3bd7bf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b81599b0 0x7fa3b8159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3bd7bf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b8148ec0 0x7fa3b80b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3bd7bf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3b8142ed0 0x7fa3b8143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3bd7bf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b80b1b40 con 0x7fa3b8142ed0 2020-11-20T12:16:13.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3bd7bf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8148930 con 0x7fa3b8148ec0 2020-11-20T12:16:13.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3bd7bf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8148bd0 con 0x7fa3b81599b0 2020-11-20T12:16:13.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3b77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b8148ec0 0x7fa3b80b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3b77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b8148ec0 0x7fa3b80b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50868/0 (socket says 172.21.15.187:50868) 2020-11-20T12:16:13.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.803+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1765585363 learned_addr learned my addr 172.21.15.187:0/1765585363 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:13.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3bcfbe700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3b8142ed0 0x7fa3b8143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3b7fff700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b81599b0 0x7fa3b8159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b81599b0 msgr2=0x7fa3b8159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b81599b0 0x7fa3b8159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3b8142ed0 msgr2=0x7fa3b8143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:13.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3b8142ed0 0x7fa3b8143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:13.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3b77fe700 1 -- 172.21.15.187:0/1765585363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b815a610 con 0x7fa3b8148ec0 2020-11-20T12:16:13.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3bcfbe700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3b8142ed0 0x7fa3b8143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:13.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3b77fe700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b8148ec0 0x7fa3b80b4580 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7fa3ac000dd0 tx=0x7fa3ac009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:13.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa397fff700 1 -- 172.21.15.187:0/1765585363 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac007170 con 0x7fa3b8148ec0 2020-11-20T12:16:13.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa397fff700 1 -- 172.21.15.187:0/1765585363 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac0072d0 con 0x7fa3b8148ec0 2020-11-20T12:16:13.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa397fff700 1 -- 172.21.15.187:0/1765585363 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac00bb60 con 0x7fa3b8148ec0 2020-11-20T12:16:13.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.804+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b8143820 con 0x7fa3b8148ec0 2020-11-20T12:16:13.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.805+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b8144670 con 0x7fa3b8148ec0 2020-11-20T12:16:13.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.806+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b8061a20 con 0x7fa3b8148ec0 2020-11-20T12:16:13.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.806+0000 7fa397fff700 1 -- 172.21.15.187:0/1765585363 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa3ac0045c0 con 0x7fa3b8148ec0 2020-11-20T12:16:13.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.807+0000 7fa397fff700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa390056d50 0x7fa390059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:13.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.807+0000 7fa397fff700 1 -- 172.21.15.187:0/1765585363 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa3ac070300 con 0x7fa3b8148ec0 2020-11-20T12:16:13.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.807+0000 7fa3b7fff700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa390056d50 0x7fa390059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:13.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.808+0000 7fa3b7fff700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa390056d50 0x7fa390059210 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fa3a8000bc0 tx=0x7fa3a8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:13.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:13.811+0000 7fa397fff700 1 -- 172.21.15.187:0/1765585363 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa3ac0455e0 con 0x7fa3b8148ec0 2020-11-20T12:16:14.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.002+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3b809d5d0 con 0x7fa3b8148ec0 2020-11-20T12:16:14.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.004+0000 7fa397fff700 1 -- 172.21.15.187:0/1765585363 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v119) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa3ac001190 con 0x7fa3b8148ec0 2020-11-20T12:16:14.006 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:16:14.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa390056d50 msgr2=0x7fa390059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa390056d50 0x7fa390059210 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fa3a8000bc0 tx=0x7fa3a8009040).stop 2020-11-20T12:16:14.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b8148ec0 msgr2=0x7fa3b80b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b8148ec0 0x7fa3b80b4580 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7fa3ac000dd0 tx=0x7fa3ac009040).stop 2020-11-20T12:16:14.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 shutdown_connections 2020-11-20T12:16:14.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa390056d50 0x7fa390059210 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3b8142ed0 0x7fa3b8143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3b81599b0 0x7fa3b8159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 --2- 172.21.15.187:0/1765585363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b8148ec0 0x7fa3b80b4580 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.006+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 >> 172.21.15.187:0/1765585363 conn(0x7fa3b80bab30 msgr2=0x7fa3b809ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:14.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.007+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 shutdown_connections 2020-11-20T12:16:14.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.007+0000 7fa3bd7bf700 1 -- 172.21.15.187:0/1765585363 wait complete. 2020-11-20T12:16:14.034 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:16:14.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.274+0000 7f5968910700 1 Processor -- start 2020-11-20T12:16:14.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.275+0000 7f5968910700 1 -- start start 2020-11-20T12:16:14.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.276+0000 7f5968910700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f596414d270 0x7f59641573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.276+0000 7f5968910700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59640658a0 con 0x7f59640b1760 2020-11-20T12:16:14.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.276+0000 7f5968910700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5964150390 con 0x7f59640b1380 2020-11-20T12:16:14.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.276+0000 7f5968910700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59641511e0 con 0x7f596414d270 2020-11-20T12:16:14.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.276+0000 7f5962d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f59640b1380 0x7f5964150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47480/0 (socket says 172.21.15.187:47480) 2020-11-20T12:16:14.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.276+0000 7f5962d9d700 1 -- 172.21.15.187:0/3536690382 learned_addr learned my addr 172.21.15.187:0/3536690382 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:14.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.276+0000 7f596359e700 1 --2- 172.21.15.187:0/3536690382 >> v2:172.21.15.187:3300/0 conn(0x7f596414d270 0x7f59641573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.277+0000 7f5961d9b700 1 -- 172.21.15.187:0/3536690382 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600740208 0 0) 0x7f5964150390 con 0x7f59640b1380 2020-11-20T12:16:14.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.277+0000 7f5961d9b700 1 -- 172.21.15.187:0/3536690382 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5950003240 con 0x7f59640b1380 2020-11-20T12:16:14.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.277+0000 7f596359e700 1 -- 172.21.15.187:0/3536690382 >> v1:172.21.15.187:6789/0 conn(0x7f59640b1380 legacy=0x7f5964150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.277+0000 7f596359e700 1 -- 172.21.15.187:0/3536690382 >> v1:172.21.15.5:6789/0 conn(0x7f59640b1760 legacy=0x7f5964065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.277+0000 7f596359e700 1 -- 172.21.15.187:0/3536690382 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5964150630 con 0x7f596414d270 2020-11-20T12:16:14.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.277+0000 7f596359e700 1 --2- 172.21.15.187:0/3536690382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f596414d270 0x7f59641573a0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f595c009c90 tx=0x7f595c0091d0).ready entity=mon.1 client_cookie=3db8698db552efa8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:14.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5961d9b700 1 -- 172.21.15.187:0/3536690382 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c00ba70 con 0x7f596414d270 2020-11-20T12:16:14.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5961d9b700 1 -- 172.21.15.187:0/3536690382 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c00bbd0 con 0x7f596414d270 2020-11-20T12:16:14.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5961d9b700 1 -- 172.21.15.187:0/3536690382 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c011400 con 0x7f596414d270 2020-11-20T12:16:14.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5968910700 1 -- 172.21.15.187:0/3536690382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f596414d270 msgr2=0x7f59641573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5968910700 1 --2- 172.21.15.187:0/3536690382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f596414d270 0x7f59641573a0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f595c009c90 tx=0x7f595c0091d0).stop 2020-11-20T12:16:14.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5968910700 1 -- 172.21.15.187:0/3536690382 shutdown_connections 2020-11-20T12:16:14.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5968910700 1 --2- 172.21.15.187:0/3536690382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f596414d270 0x7f59641573a0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5968910700 1 -- 172.21.15.187:0/3536690382 >> 172.21.15.187:0/3536690382 conn(0x7f59640bab30 msgr2=0x7f59640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:14.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.278+0000 7f5968910700 1 -- 172.21.15.187:0/3536690382 shutdown_connections 2020-11-20T12:16:14.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.279+0000 7f5968910700 1 -- 172.21.15.187:0/3536690382 wait complete. 2020-11-20T12:16:14.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.281+0000 7f5968910700 1 Processor -- start 2020-11-20T12:16:14.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.281+0000 7f5968910700 1 -- start start 2020-11-20T12:16:14.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f5968910700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59640b1380 0x7f596415cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f5968910700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59640b1760 0x7f596415b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f5968910700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f596414d270 0x7f596409fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f5968910700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596409e830 con 0x7f596414d270 2020-11-20T12:16:14.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f5968910700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5964148470 con 0x7f59640b1760 2020-11-20T12:16:14.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f5968910700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596415d100 con 0x7f59640b1380 2020-11-20T12:16:14.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f596259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59640b1760 0x7f596415b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f596259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59640b1760 0x7f596415b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50882/0 (socket says 172.21.15.187:50882) 2020-11-20T12:16:14.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.282+0000 7f596259c700 1 -- 172.21.15.187:0/1230001023 learned_addr learned my addr 172.21.15.187:0/1230001023 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:14.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.283+0000 7f596359e700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f596414d270 0x7f596409fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.283+0000 7f5962d9d700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59640b1380 0x7f596415cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.283+0000 7f596259c700 1 -- 172.21.15.187:0/1230001023 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59640b1380 msgr2=0x7f596415cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.283+0000 7f596259c700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59640b1380 0x7f596415cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.283+0000 7f596259c700 1 -- 172.21.15.187:0/1230001023 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f596414d270 msgr2=0x7f596409fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.283+0000 7f596259c700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f596414d270 0x7f596409fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.283+0000 7f596259c700 1 -- 172.21.15.187:0/1230001023 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59640b40d0 con 0x7f59640b1760 2020-11-20T12:16:14.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.283+0000 7f596259c700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59640b1760 0x7f596415b9a0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 rx=0x7f595800c6e0 tx=0x7f5958007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:14.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.284+0000 7f594affd700 1 -- 172.21.15.187:0/1230001023 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958007020 con 0x7f59640b1760 2020-11-20T12:16:14.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.284+0000 7f594affd700 1 -- 172.21.15.187:0/1230001023 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5958007180 con 0x7f59640b1760 2020-11-20T12:16:14.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.284+0000 7f594affd700 1 -- 172.21.15.187:0/1230001023 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595800e560 con 0x7f59640b1760 2020-11-20T12:16:14.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.284+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59640b4370 con 0x7f59640b1760 2020-11-20T12:16:14.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.284+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596415c050 con 0x7f59640b1760 2020-11-20T12:16:14.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.285+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59640b4500 con 0x7f59640b1760 2020-11-20T12:16:14.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.285+0000 7f594affd700 1 -- 172.21.15.187:0/1230001023 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5958018020 con 0x7f59640b1760 2020-11-20T12:16:14.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.286+0000 7f594affd700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5940056d20 0x7f59400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.286+0000 7f594affd700 1 -- 172.21.15.187:0/1230001023 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f595806f890 con 0x7f59640b1760 2020-11-20T12:16:14.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.287+0000 7f5962d9d700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5940056d20 0x7f59400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.287+0000 7f5962d9d700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5940056d20 0x7f59400591e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f59640b3770 tx=0x7f5954009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:14.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.291+0000 7f594affd700 1 -- 172.21.15.187:0/1230001023 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5958044ce0 con 0x7f59640b1760 2020-11-20T12:16:14.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.482+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5964061a20 con 0x7f59640b1760 2020-11-20T12:16:14.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.483+0000 7f594affd700 1 -- 172.21.15.187:0/1230001023 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v119) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5958048300 con 0x7f59640b1760 2020-11-20T12:16:14.486 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:16:14.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.485+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5940056d20 msgr2=0x7f59400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.485+0000 7f5968910700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5940056d20 0x7f59400591e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f59640b3770 tx=0x7f5954009040).stop 2020-11-20T12:16:14.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.485+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59640b1760 msgr2=0x7f596415b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.485+0000 7f5968910700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59640b1760 0x7f596415b9a0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 rx=0x7f595800c6e0 tx=0x7f5958007c50).stop 2020-11-20T12:16:14.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.486+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 shutdown_connections 2020-11-20T12:16:14.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.486+0000 7f5968910700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5940056d20 0x7f59400591e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.486+0000 7f5968910700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f596414d270 0x7f596409fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.486+0000 7f5968910700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59640b1380 0x7f596415cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.486+0000 7f5968910700 1 --2- 172.21.15.187:0/1230001023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59640b1760 0x7f596415b9a0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.486+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 >> 172.21.15.187:0/1230001023 conn(0x7f59640bab30 msgr2=0x7f596414f090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:14.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.486+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 shutdown_connections 2020-11-20T12:16:14.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.486+0000 7f5968910700 1 -- 172.21.15.187:0/1230001023 wait complete. 2020-11-20T12:16:14.511 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:16:14.512 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:14.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.753+0000 7f5892a52700 1 Processor -- start 2020-11-20T12:16:14.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.753+0000 7f5892a52700 1 -- start start 2020-11-20T12:16:14.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.754+0000 7f5892a52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c1484d0 0x7f588c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.754+0000 7f5892a52700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f588c0622c0 0x7f588c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.754+0000 7f5892a52700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f588c066f70 0x7f588c0b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.754+0000 7f5892a52700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c067730 con 0x7f588c0622c0 2020-11-20T12:16:14.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.754+0000 7f5892a52700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c0b1300 con 0x7f588c066f70 2020-11-20T12:16:14.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.754+0000 7f5892a52700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0b15a0 con 0x7f588c1484d0 2020-11-20T12:16:14.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.755+0000 7f5892251700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f588c066f70 0x7f588c0b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.755+0000 7f5892251700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f588c066f70 0x7f588c0b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50892/0 (socket says 172.21.15.187:50892) 2020-11-20T12:16:14.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.755+0000 7f5892251700 1 -- 172.21.15.187:0/1440860882 learned_addr learned my addr 172.21.15.187:0/1440860882 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:14.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.755+0000 7f5891a50700 1 --2- 172.21.15.187:0/1440860882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c1484d0 0x7f588c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.755+0000 7f589124f700 1 --2- 172.21.15.187:0/1440860882 >> v2:172.21.15.5:3300/0 conn(0x7f588c0622c0 0x7f588c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5892251700 1 -- 172.21.15.187:0/1440860882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c1484d0 msgr2=0x7f588c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:14.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5892251700 1 --2- 172.21.15.187:0/1440860882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c1484d0 0x7f588c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5892251700 1 -- 172.21.15.187:0/1440860882 >> v2:172.21.15.5:3300/0 conn(0x7f588c0622c0 msgr2=0x7f588c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:14.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5892251700 1 --2- 172.21.15.187:0/1440860882 >> v2:172.21.15.5:3300/0 conn(0x7f588c0622c0 0x7f588c0b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5892251700 1 -- 172.21.15.187:0/1440860882 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0b1840 con 0x7f588c066f70 2020-11-20T12:16:14.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5892251700 1 --2- 172.21.15.187:0/1440860882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c066f70 0x7f588c0b5b30 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 rx=0x7f588c09d820 tx=0x7f5884009290).ready entity=mon.1 client_cookie=7932a780745cf135 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:14.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5890a4e700 1 -- 172.21.15.187:0/1440860882 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400bbc0 con 0x7f588c066f70 2020-11-20T12:16:14.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5890a4e700 1 -- 172.21.15.187:0/1440860882 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588400bd20 con 0x7f588c066f70 2020-11-20T12:16:14.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.756+0000 7f5890a4e700 1 -- 172.21.15.187:0/1440860882 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884011610 con 0x7f588c066f70 2020-11-20T12:16:14.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 -- 172.21.15.187:0/1440860882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c066f70 msgr2=0x7f588c0b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 --2- 172.21.15.187:0/1440860882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c066f70 0x7f588c0b5b30 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 rx=0x7f588c09d820 tx=0x7f5884009290).stop 2020-11-20T12:16:14.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 -- 172.21.15.187:0/1440860882 shutdown_connections 2020-11-20T12:16:14.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 --2- 172.21.15.187:0/1440860882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c066f70 0x7f588c0b5b30 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 --2- 172.21.15.187:0/1440860882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c1484d0 0x7f588c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 --2- 172.21.15.187:0/1440860882 >> v2:172.21.15.5:3300/0 conn(0x7f588c0622c0 0x7f588c0b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 -- 172.21.15.187:0/1440860882 >> 172.21.15.187:0/1440860882 conn(0x7f588c0bab30 msgr2=0x7f588c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:14.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 -- 172.21.15.187:0/1440860882 shutdown_connections 2020-11-20T12:16:14.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.757+0000 7f5892a52700 1 -- 172.21.15.187:0/1440860882 wait complete. 2020-11-20T12:16:14.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.759+0000 7f5892a52700 1 Processor -- start 2020-11-20T12:16:14.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.759+0000 7f5892a52700 1 -- start start 2020-11-20T12:16:14.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892a52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c0622c0 0x7f588c09f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892a52700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f588c066f70 0x7f588c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892a52700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c1484d0 0x7f588c09b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892a52700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c09f620 con 0x7f588c066f70 2020-11-20T12:16:14.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892a52700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c09f8b0 con 0x7f588c1484d0 2020-11-20T12:16:14.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892a52700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c09fb50 con 0x7f588c0622c0 2020-11-20T12:16:14.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892251700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c1484d0 0x7f588c09b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892251700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c1484d0 0x7f588c09b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50898/0 (socket says 172.21.15.187:50898) 2020-11-20T12:16:14.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892251700 1 -- 172.21.15.187:0/3013682434 learned_addr learned my addr 172.21.15.187:0/3013682434 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:14.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f5892251700 1 -- 172.21.15.187:0/3013682434 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c0622c0 msgr2=0x7f588c09f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.760+0000 7f589124f700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f588c066f70 0x7f588c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5891a50700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c0622c0 0x7f588c09f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5892251700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c0622c0 0x7f588c09f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5892251700 1 -- 172.21.15.187:0/3013682434 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f588c066f70 msgr2=0x7f588c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:14.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5892251700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f588c066f70 0x7f588c0a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:14.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5892251700 1 -- 172.21.15.187:0/3013682434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c09fdf0 con 0x7f588c1484d0 2020-11-20T12:16:14.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5892251700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c1484d0 0x7f588c09b0e0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7f58840092d0 tx=0x7f5884016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:14.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5879ffb700 1 -- 172.21.15.187:0/3013682434 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400bbc0 con 0x7f588c1484d0 2020-11-20T12:16:14.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5879ffb700 1 -- 172.21.15.187:0/3013682434 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588400be90 con 0x7f588c1484d0 2020-11-20T12:16:14.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5879ffb700 1 -- 172.21.15.187:0/3013682434 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884010e50 con 0x7f588c1484d0 2020-11-20T12:16:14.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c0a0780 con 0x7f588c1484d0 2020-11-20T12:16:14.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.761+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c09b790 con 0x7f588c1484d0 2020-11-20T12:16:14.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.762+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c061a20 con 0x7f588c1484d0 2020-11-20T12:16:14.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.763+0000 7f5879ffb700 1 -- 172.21.15.187:0/3013682434 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5884017070 con 0x7f588c1484d0 2020-11-20T12:16:14.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.764+0000 7f5879ffb700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5864056d20 0x7f58640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:14.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.764+0000 7f5879ffb700 1 -- 172.21.15.187:0/3013682434 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5884072210 con 0x7f588c1484d0 2020-11-20T12:16:14.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.764+0000 7f5891a50700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5864056d20 0x7f58640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:14.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.765+0000 7f5891a50700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5864056d20 0x7f58640591e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f58880095b0 tx=0x7f5888008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:14.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:14.767+0000 7f5879ffb700 1 -- 172.21.15.187:0/3013682434 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f58840476e0 con 0x7f588c1484d0 2020-11-20T12:16:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.008+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f588c09ff80 con 0x7f588c1484d0 2020-11-20T12:16:15.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.009+0000 7f5879ffb700 1 -- 172.21.15.187:0/3013682434 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2141 (secure 0 0 0) 0x7f5884015020 con 0x7f588c1484d0 2020-11-20T12:16:15.012 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:15.012 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":102,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2092825437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2092825437},{"type":"v1","addr":"172.21.15.5:6833","nonce":2092825437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":99}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:05.876197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8473},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8473":{"gid":8473,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3143922581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3143922581},{"type":"v1","addr":"172.21.15.187:6835","nonce":3143922581}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.011+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5864056d20 msgr2=0x7f58640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.011+0000 7f5892a52700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5864056d20 0x7f58640591e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f58880095b0 tx=0x7f5888008040).stop 2020-11-20T12:16:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.011+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c1484d0 msgr2=0x7f588c09b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:15.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.011+0000 7f5892a52700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c1484d0 0x7f588c09b0e0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7f58840092d0 tx=0x7f5884016000).stop 2020-11-20T12:16:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.012+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 shutdown_connections 2020-11-20T12:16:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.012+0000 7f5892a52700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5864056d20 0x7f58640591e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:15.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.012+0000 7f5892a52700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f588c1484d0 0x7f588c09b0e0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.012+0000 7f5892a52700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f588c0622c0 0x7f588c09f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.012+0000 7f5892a52700 1 --2- 172.21.15.187:0/3013682434 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f588c066f70 0x7f588c0a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.012+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 >> 172.21.15.187:0/3013682434 conn(0x7f588c0bab30 msgr2=0x7f588c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.012+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 shutdown_connections 2020-11-20T12:16:15.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:15.012+0000 7f5892a52700 1 -- 172.21.15.187:0/3013682434 wait complete. 2020-11-20T12:16:15.017 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 102 2020-11-20T12:16:15.236 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:16:03.867608+0000', 'modified': '2020-11-20T12:16:05.876197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8473}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8473': {'gid': 8473, 'name': 'a', 'rank': 0, 'incarnation': 100, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3143922581', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3143922581}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3143922581}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:16:15.237 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:16:15.237 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:16:15.514 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8473,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":11.618057678,"mdsmap_epoch":102,"osdmap_epoch":119,"osdmap_epoch_barrier":119,"uptime":19.242559605} 2020-11-20T12:16:15.515 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8473, "mdsmap_epoch": 102, "osdmap_epoch": 119, "osdmap_epoch_barrier": 119, "rank_uptime": 11.618057678, "state": "up:active", "uptime": 19.242559605, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:16:15.516 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:16:15.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:16:15.517 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:16:15.563 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:16:15.564 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:16:15.696 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:16:15.697 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:16:15.731 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:16:15.732 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:16:15.741 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:16:15.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:16:15.742 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:16:15.837 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:16:15.846 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:16:16.042 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:16:16.043 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:16:16.043 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:16:16.043 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:16:16.044 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.044 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:16:16.045 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.045 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:16:16.046 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.046 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:16:16.046 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:16:16.047 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.047 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:16:16.047 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:16:16.048 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.048 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:16:16.049 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:16:16.049 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.049 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:16:16.050 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:16:16.050 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.050 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:16:16.051 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.051 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:16:16.051 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:16:16.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:16:16.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:16:16.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:16:16.053 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:16:16.053 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:16:16.054 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:16:16.054 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:16:16.054 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:16:16.055 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.055 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:16:16.055 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.056 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:16:16.056 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.057 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:16:16.057 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.057 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:16:16.058 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:16:16.058 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:16:16.058 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:16:16.059 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:16:16.059 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:16:16.059 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.060 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:16:16.060 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:16:16.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.061 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:16:16.061 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:16:16.062 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.062 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:16:16.062 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:16:16.063 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.063 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:16:16.063 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:16:16.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:16:16.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.065 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:16:16.065 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:16:16.065 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.066 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:16:16.066 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.066 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:16:16.067 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.067 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:16:16.068 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.068 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:16:16.068 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:16:16.069 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.069 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:16:16.070 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:16:16.070 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:16:16.070 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:16:16.071 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:16:16.071 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:16:16.071 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:16:16.072 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:16:16.072 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:16:16.073 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:16:16.073 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:16:16.073 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:16:16.074 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:16:16.074 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:16:16.074 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:16:16.075 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:16:16.075 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:16:16.075 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:16:16.076 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:16:16.076 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:16:16.076 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.077 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:16:16.077 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:16.077 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:16:16.078 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:16:16.078 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:16:16.079 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:16.079 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:16.079 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:16:16.080 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.080 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:16:16.080 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:16.081 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:16:16.081 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:16:16.081 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:16:16.082 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:16:16.082 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:16:16.082 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:16.083 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:16:16.083 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:16:16.084 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:16.084 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:16:16.084 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.085 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:16:16.085 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:16:16.085 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:16:16.086 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:16:16.086 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:16:16.086 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:16:16.087 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:16:16.087 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:16:16.087 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:16.088 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:16:16.088 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:16:16.089 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:16:16.089 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:16:16.089 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:16:16.090 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:16:16.090 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:16:16.090 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:16:16.091 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:16:16.091 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:16:16.091 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:16.092 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:16:16.092 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:16:16.092 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:16:16.093 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:16:16.093 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:16:16.094 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:16:16.094 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:16:16.094 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:16:16.095 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:16:16.095 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:16:16.095 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:16:16.096 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:16:16.096 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 100000000 2020-11-20T12:16:16.098 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:16:16.098 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:16:16.099 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:16:16.099 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:16:16.099 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.099 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:16:16.100 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.100 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:16:16.100 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.101 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:16:16.101 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:16:16.101 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.102 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:16:16.102 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:16:16.102 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.103 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:16:16.103 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:16:16.103 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.103 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:16:16.103 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:16:16.104 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.104 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:16:16.104 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.104 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:16:16.104 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:16:16.105 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:16:16.105 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:16:16.105 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:16:16.105 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:16:16.106 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:16:16.106 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:16:16.106 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:16:16.106 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:16:16.106 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.107 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:16:16.107 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.107 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:16:16.107 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.107 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:16:16.108 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.108 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:16:16.108 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:16:16.108 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:16:16.108 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:16:16.109 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:16:16.109 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:16:16.109 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.109 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:16:16.109 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:16:16.110 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.110 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:16:16.110 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:16:16.110 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.110 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:16:16.111 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:16:16.111 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.111 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:16:16.111 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:16:16.111 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:16:16.112 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.112 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:16:16.112 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:16:16.112 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.112 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:16:16.113 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.113 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:16:16.113 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.113 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:16:16.113 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.114 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:16:16.114 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:16:16.114 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.114 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:16:16.114 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:16:16.115 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:16:16.115 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:16:16.115 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:16:16.115 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:16:16.115 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:16:16.115 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:16:16.116 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:16:16.116 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:16:16.116 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:16:16.116 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:16:16.116 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:16:16.117 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:16:16.117 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:16:16.117 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:16:16.117 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:16:16.118 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:16:16.118 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:16:16.118 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:16:16.118 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.118 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:16:16.118 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:16.119 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:16:16.119 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:16:16.119 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:16:16.119 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:16.119 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:16.120 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:16:16.120 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.120 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:16:16.120 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:16.120 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:16:16.121 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:16:16.121 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:16:16.121 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:16:16.121 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:16:16.121 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:16.122 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:16:16.122 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:16:16.122 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:16.122 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:16:16.122 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.123 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:16:16.123 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:16:16.123 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:16:16.123 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:16:16.123 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:16:16.124 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:16:16.124 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:16:16.124 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:16:16.124 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:16.124 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:16:16.125 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:16:16.125 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:16:16.125 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:16:16.125 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:16:16.125 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:16:16.126 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:16:16.126 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:16:16.126 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:16:16.126 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:16:16.126 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:16.127 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:16:16.127 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:16:16.127 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:16:16.127 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:16:16.127 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:16:16.128 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:16:16.128 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:16:16.128 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:16:16.128 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:16:16.128 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:16:16.129 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:16:16.129 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:16:16.129 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 100000000 2020-11-20T12:16:16.131 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:16:16.139 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:16:16.333 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:16:16.333 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:16:16.333 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:16:16.333 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:16:16.334 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.334 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:16:16.334 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.334 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:16:16.335 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.335 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:16:16.335 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:16:16.335 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.336 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:16:16.336 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:16:16.336 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.336 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:16:16.336 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:16:16.337 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.337 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:16:16.337 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:16:16.337 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.337 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:16:16.338 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.338 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:16:16.338 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:16:16.338 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:16:16.338 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:16:16.339 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:16:16.339 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:16:16.339 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:16:16.339 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:16:16.339 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:16:16.340 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:16:16.340 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.340 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:16:16.340 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.340 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:16:16.341 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.341 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:16:16.341 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.342 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:16:16.342 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:16:16.342 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:16:16.342 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:16:16.343 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:16:16.343 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:16:16.343 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.343 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:16:16.343 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:16:16.344 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.344 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:16:16.344 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:16:16.344 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.344 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:16:16.345 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:16:16.345 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.345 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:16:16.345 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:16:16.345 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:16:16.346 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.346 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:16:16.346 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:16:16.346 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.346 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:16:16.347 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.347 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:16:16.347 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.347 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:16:16.347 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:16:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:16:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:16:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:16:16.349 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:16:16.349 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:16:16.349 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:16:16.349 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:16:16.349 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:16:16.350 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:16:16.350 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:16:16.350 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:16:16.350 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:16:16.351 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:16:16.351 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:16:16.351 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:16:16.351 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:16:16.351 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:16:16.351 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:16:16.352 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:16:16.352 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:16:16.352 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:16:16.352 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.352 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:16:16.353 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:16.353 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:16:16.353 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:16:16.353 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:16:16.353 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:16.354 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:16.354 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:16:16.354 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.354 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:16:16.354 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:16:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:16:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:16:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:16:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:16:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:16:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:16:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:16:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:16.357 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:16:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:16:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:16:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:16:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:16:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:16:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:16:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:16:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:16:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:16:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:16:16.359 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:16:16.359 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:16:16.360 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:16:16.360 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:16:16.360 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:16:16.360 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:16:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:16:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:16:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:16:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:16:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:16:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:16:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:16:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:16:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:16:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:16:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:16:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:16:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:16:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 100000000 2020-11-20T12:16:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops = 16 2020-11-20T12:16:16.364 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:16:16.364 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:16:16.364 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:16:16.365 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:16:16.365 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.365 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:16:16.365 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.366 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:16:16.366 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.366 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:16:16.366 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:16:16.366 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.367 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:16:16.367 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:16:16.367 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.367 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:16:16.367 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:16:16.368 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.368 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:16:16.368 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:16:16.368 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.368 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:16:16.369 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:16:16.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:16:16.370 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:16:16.370 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:16:16.370 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:16:16.370 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:16:16.370 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:16:16.371 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:16:16.371 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:16:16.371 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:16:16.371 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:16:16.372 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.372 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:16:16.372 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.372 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:16:16.372 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.372 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:16:16.373 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:16:16.373 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:16:16.373 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:16:16.374 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:16:16.374 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:16:16.374 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.374 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:16:16.375 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:16:16.375 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.375 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:16:16.375 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:16:16.375 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.376 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:16:16.376 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:16:16.376 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.376 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:16:16.376 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:16:16.377 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:16:16.377 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.377 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:16:16.378 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:16:16.378 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.378 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:16:16.378 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.378 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:16:16.379 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.379 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:16:16.379 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.379 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:16:16.379 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:16:16.380 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.380 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:16:16.380 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:16:16.380 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:16:16.380 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:16:16.380 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:16:16.381 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:16:16.381 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:16:16.381 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:16:16.381 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:16:16.381 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:16:16.382 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:16:16.382 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:16:16.382 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:16:16.382 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:16:16.382 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:16:16.383 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:16:16.383 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:16:16.383 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:16:16.383 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:16:16.383 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:16:16.384 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.384 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:16:16.384 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:16.384 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:16:16.384 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:16:16.385 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:16:16.385 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:16.385 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:16.385 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:16:16.385 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.386 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:16:16.386 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:16.386 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:16:16.386 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:16:16.386 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:16:16.387 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:16:16.387 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:16:16.387 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:16.387 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:16:16.387 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:16:16.388 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:16.388 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:16:16.388 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:16.388 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:16:16.388 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:16:16.389 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:16:16.389 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:16:16.389 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:16:16.389 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:16:16.389 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:16:16.390 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:16:16.390 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:16.391 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:16:16.391 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:16:16.392 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:16:16.392 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:16:16.393 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:16:16.393 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:16:16.393 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:16:16.393 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:16:16.394 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:16:16.394 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:16:16.394 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:16.395 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:16:16.395 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:16:16.396 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:16:16.396 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:16:16.396 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:16:16.397 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:16:16.397 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:16:16.398 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:16:16.398 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:16:16.398 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:16:16.399 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:16:16.399 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:16:16.399 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 100000000 2020-11-20T12:16:16.399 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops = 16 2020-11-20T12:16:16.401 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:16:16.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.665+0000 7fdbff52f700 1 Processor -- start 2020-11-20T12:16:16.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.665+0000 7fdbff52f700 1 -- start start 2020-11-20T12:16:16.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.666+0000 7fdbff52f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdbf809cba0 0x7fdbf8099e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:16.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.666+0000 7fdbff52f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 0x7fdbf8061f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:16.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.666+0000 7fdbff52f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf8062470 con 0x7fdbf8145dc0 2020-11-20T12:16:16.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.666+0000 7fdbff52f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf8067500 con 0x7fdbf809cba0 2020-11-20T12:16:16.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.666+0000 7fdbff52f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80677a0 con 0x7fdbf809cf80 2020-11-20T12:16:16.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.667+0000 7fdbfe52d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdbf809cba0 0x7fdbf8099e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:16.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.667+0000 7fdbfe52d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdbf809cba0 0x7fdbf8099e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50908/0 (socket says 172.21.15.187:50908) 2020-11-20T12:16:16.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.667+0000 7fdbfe52d700 1 -- 172.21.15.187:0/611339838 learned_addr learned my addr 172.21.15.187:0/611339838 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:16.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.667+0000 7fdbfdd2c700 1 --2- 172.21.15.187:0/611339838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 0x7fdbf8061f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:16.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.668+0000 7fdbfe52d700 1 -- 172.21.15.187:0/611339838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 msgr2=0x7fdbf8061f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:16.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.668+0000 7fdbfe52d700 1 --2- 172.21.15.187:0/611339838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 0x7fdbf8061f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.668+0000 7fdbfe52d700 1 -- 172.21.15.187:0/611339838 >> v1:172.21.15.5:6789/0 conn(0x7fdbf8145dc0 legacy=0x7fdbf8146250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:16.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.668+0000 7fdbfe52d700 1 -- 172.21.15.187:0/611339838 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf809e010 con 0x7fdbf809cba0 2020-11-20T12:16:16.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.668+0000 7fdbfe52d700 1 --2- 172.21.15.187:0/611339838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf809cba0 0x7fdbf8099e10 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 rx=0x7fdbf4006fd0 tx=0x7fdbf4006eb0).ready entity=mon.1 client_cookie=e827dde68216d866 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:16.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbfd52b700 1 -- 172.21.15.187:0/611339838 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40098d0 con 0x7fdbf809cba0 2020-11-20T12:16:16.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbfd52b700 1 -- 172.21.15.187:0/611339838 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4009a30 con 0x7fdbf809cba0 2020-11-20T12:16:16.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbfd52b700 1 -- 172.21.15.187:0/611339838 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4009db0 con 0x7fdbf809cba0 2020-11-20T12:16:16.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbff52f700 1 -- 172.21.15.187:0/611339838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf809cba0 msgr2=0x7fdbf8099e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:16.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbff52f700 1 --2- 172.21.15.187:0/611339838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf809cba0 0x7fdbf8099e10 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 rx=0x7fdbf4006fd0 tx=0x7fdbf4006eb0).stop 2020-11-20T12:16:16.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbff52f700 1 -- 172.21.15.187:0/611339838 shutdown_connections 2020-11-20T12:16:16.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbff52f700 1 --2- 172.21.15.187:0/611339838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf809cba0 0x7fdbf8099e10 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbff52f700 1 --2- 172.21.15.187:0/611339838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 0x7fdbf8061f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.669+0000 7fdbff52f700 1 -- 172.21.15.187:0/611339838 >> 172.21.15.187:0/611339838 conn(0x7fdbf80b76c0 msgr2=0x7fdbf80b7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:16.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.670+0000 7fdbff52f700 1 -- 172.21.15.187:0/611339838 shutdown_connections 2020-11-20T12:16:16.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.670+0000 7fdbff52f700 1 -- 172.21.15.187:0/611339838 wait complete. 2020-11-20T12:16:16.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.671+0000 7fdbff52f700 1 Processor -- start 2020-11-20T12:16:16.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.671+0000 7fdbff52f700 1 -- start start 2020-11-20T12:16:16.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.672+0000 7fdbff52f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 0x7fdbf8142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:16.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.672+0000 7fdbff52f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf8145dc0 0x7fdbf815b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:16.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.672+0000 7fdbff52f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbf814ac50 0x7fdbf815bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:16.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.673+0000 7fdbff52f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf814b0d0 con 0x7fdbf814ac50 2020-11-20T12:16:16.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.673+0000 7fdbff52f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf814b370 con 0x7fdbf8145dc0 2020-11-20T12:16:16.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.673+0000 7fdbff52f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf809a870 con 0x7fdbf809cf80 2020-11-20T12:16:16.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.673+0000 7fdbfdd2c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf8145dc0 0x7fdbf815b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:16.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.673+0000 7fdbfdd2c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf8145dc0 0x7fdbf815b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50914/0 (socket says 172.21.15.187:50914) 2020-11-20T12:16:16.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.673+0000 7fdbfdd2c700 1 -- 172.21.15.187:0/1565553133 learned_addr learned my addr 172.21.15.187:0/1565553133 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:16.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.673+0000 7fdbfe52d700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 0x7fdbf8142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:16.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.673+0000 7fdbfed2e700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbf814ac50 0x7fdbf815bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:16.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbfdd2c700 1 -- 172.21.15.187:0/1565553133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 msgr2=0x7fdbf8142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:16.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbfdd2c700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 0x7fdbf8142d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbfdd2c700 1 -- 172.21.15.187:0/1565553133 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbf814ac50 msgr2=0x7fdbf815bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:16.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbfdd2c700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbf814ac50 0x7fdbf815bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbfdd2c700 1 -- 172.21.15.187:0/1565553133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf815c0f0 con 0x7fdbf8145dc0 2020-11-20T12:16:16.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbfdd2c700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf8145dc0 0x7fdbf815b710 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7fdbe800db60 tx=0x7fdbe8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:16.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbe67fc700 1 -- 172.21.15.187:0/1565553133 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8009790 con 0x7fdbf8145dc0 2020-11-20T12:16:16.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbe67fc700 1 -- 172.21.15.187:0/1565553133 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe80098f0 con 0x7fdbf8145dc0 2020-11-20T12:16:16.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf814b740 con 0x7fdbf8145dc0 2020-11-20T12:16:16.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbe67fc700 1 -- 172.21.15.187:0/1565553133 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8009c90 con 0x7fdbf8145dc0 2020-11-20T12:16:16.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.674+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf81434c0 con 0x7fdbf8145dc0 2020-11-20T12:16:16.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.675+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf809aa30 con 0x7fdbf8145dc0 2020-11-20T12:16:16.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.676+0000 7fdbe67fc700 1 -- 172.21.15.187:0/1565553133 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdbe8013480 con 0x7fdbf8145dc0 2020-11-20T12:16:16.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.677+0000 7fdbe67fc700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdbd0056d20 0x7fdbd00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:16.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.677+0000 7fdbe67fc700 1 -- 172.21.15.187:0/1565553133 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdbe8070550 con 0x7fdbf8145dc0 2020-11-20T12:16:16.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.677+0000 7fdbfe52d700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdbd0056d20 0x7fdbd00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:16.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.677+0000 7fdbfe52d700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdbd0056d20 0x7fdbd00591e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fdbf8069850 tx=0x7fdbf4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:16.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.680+0000 7fdbe67fc700 1 -- 172.21.15.187:0/1565553133 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdbe8045940 con 0x7fdbf8145dc0 2020-11-20T12:16:16.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.869+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdbf8069ca0 con 0x7fdbf8145dc0 2020-11-20T12:16:16.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.870+0000 7fdbe67fc700 1 -- 172.21.15.187:0/1565553133 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v119) v1 ==== 100+0+10 (secure 0 0 0) 0x7fdbe8048f60 con 0x7fdbf8145dc0 2020-11-20T12:16:16.873 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:16:16.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.873+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdbd0056d20 msgr2=0x7fdbd00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:16.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.873+0000 7fdbff52f700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdbd0056d20 0x7fdbd00591e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fdbf8069850 tx=0x7fdbf4015000).stop 2020-11-20T12:16:16.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.873+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf8145dc0 msgr2=0x7fdbf815b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:16.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.873+0000 7fdbff52f700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf8145dc0 0x7fdbf815b710 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7fdbe800db60 tx=0x7fdbe8008040).stop 2020-11-20T12:16:16.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.874+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 shutdown_connections 2020-11-20T12:16:16.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.874+0000 7fdbff52f700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdbd0056d20 0x7fdbd00591e0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.874+0000 7fdbff52f700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbf814ac50 0x7fdbf815bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.874+0000 7fdbff52f700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbf809cf80 0x7fdbf8142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.874+0000 7fdbff52f700 1 --2- 172.21.15.187:0/1565553133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf8145dc0 0x7fdbf815b710 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:16.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.874+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 >> 172.21.15.187:0/1565553133 conn(0x7fdbf80b76c0 msgr2=0x7fdbf8144e60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:16.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.874+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 shutdown_connections 2020-11-20T12:16:16.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:16.874+0000 7fdbff52f700 1 -- 172.21.15.187:0/1565553133 wait complete. 2020-11-20T12:16:16.901 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:16:16.910 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:16:17.108 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:16:17.109 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:16:17.109 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:16:17.109 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:16:17.109 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.110 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:16:17.110 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.110 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:16:17.110 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.111 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:16:17.111 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:16:17.111 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.111 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:16:17.111 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:16:17.112 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.112 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:16:17.112 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:16:17.112 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.112 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:16:17.113 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:16:17.113 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.113 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:16:17.113 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.114 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:16:17.114 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:16:17.114 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:16:17.114 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:16:17.114 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:16:17.115 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:16:17.115 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:16:17.115 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:16:17.115 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:16:17.116 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:16:17.116 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.116 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:16:17.116 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.117 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:16:17.117 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.118 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:16:17.118 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.118 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:16:17.119 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:16:17.119 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:16:17.119 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:16:17.120 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:16:17.120 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:16:17.121 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.121 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:16:17.122 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:16:17.122 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.122 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:16:17.123 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:16:17.123 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.124 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:16:17.124 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:16:17.124 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.125 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:16:17.125 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:16:17.126 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:16:17.126 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.126 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:16:17.127 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:16:17.127 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.128 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:16:17.128 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.128 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:16:17.129 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.129 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:16:17.130 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.130 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:16:17.130 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:16:17.131 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.131 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:16:17.132 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:16:17.132 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:16:17.133 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:16:17.133 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:16:17.133 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:16:17.134 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:16:17.134 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:16:17.135 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:16:17.135 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:16:17.135 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:16:17.136 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:16:17.136 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:16:17.137 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:16:17.137 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:16:17.138 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:16:17.138 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:16:17.138 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:16:17.139 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:16:17.139 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:16:17.140 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.140 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:16:17.140 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:17.141 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:16:17.141 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:16:17.142 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:16:17.142 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:17.142 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:17.142 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:16:17.143 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.143 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:16:17.143 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:17.143 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:16:17.144 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:16:17.144 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:16:17.144 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:16:17.144 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:16:17.144 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:17.145 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:16:17.145 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:16:17.145 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:17.146 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:16:17.146 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:17.147 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:16:17.147 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:16:17.148 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:16:17.148 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:16:17.148 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:16:17.149 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:16:17.149 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:16:17.150 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:16:17.150 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:17.150 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:16:17.151 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:16:17.151 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:16:17.152 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:16:17.152 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:16:17.152 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:16:17.153 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:16:17.153 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:16:17.154 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:16:17.154 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:16:17.154 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:16:17.155 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:16:17.155 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:16:17.156 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:16:17.156 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:16:17.156 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:16:17.157 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:16:17.157 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:16:17.158 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:16:17.158 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:16:17.158 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:16:17.159 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:16:17.159 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:16:17.160 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 100000000 2020-11-20T12:16:17.160 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops = 16 2020-11-20T12:16:17.160 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops_per_pg = 2.0 2020-11-20T12:16:17.161 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:16:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:16:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:16:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:16:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:16:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:16:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:16:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:16:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:16:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:16:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:16:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:16:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:16:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:16:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:16:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:16:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:16:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:16:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:16:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:16:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:16:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:16:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:16:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:16:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:16:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:16:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:16:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:16:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:16:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:16:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:16:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:16:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:16:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:16:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.176 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:16:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:16:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:16:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:16:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:16:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:16:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:16:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:16:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:16:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:16:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:16:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:16:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:16:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:16:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:16:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:16:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.182 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:16:17.182 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:16:17.182 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:16:17.182 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:16:17.182 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:16:17.183 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:16:17.183 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:16:17.183 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:16:17.183 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:16:17.184 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:16:17.184 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:16:17.184 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:16:17.184 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:16:17.184 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:16:17.185 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:16:17.185 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:16:17.185 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:16:17.185 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:16:17.185 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:16:17.186 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:16:17.186 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.186 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:16:17.186 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:17.186 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:16:17.187 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:16:17.187 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:16:17.187 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:17.187 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:17.187 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:16:17.188 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.188 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:16:17.188 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:17.188 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:16:17.188 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:16:17.189 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:16:17.189 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:16:17.189 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:16:17.189 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:16:17.190 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:16:17.190 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:16:17.190 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:16:17.190 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:16:17.190 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:16:17.191 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:16:17.191 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:16:17.191 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:16:17.191 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:16:17.191 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:16:17.192 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:16:17.192 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:16:17.192 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:16:17.192 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:17.192 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:16:17.193 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:16:17.193 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:16:17.193 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:16:17.193 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:16:17.193 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:16:17.194 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:16:17.194 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:16:17.194 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:16:17.194 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:16:17.194 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:16:17.195 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:16:17.195 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:16:17.195 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:16:17.195 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:16:17.195 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:16:17.196 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:16:17.196 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:16:17.196 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:16:17.196 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:16:17.196 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:16:17.197 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:16:17.197 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:16:17.197 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 100000000 2020-11-20T12:16:17.197 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops = 16 2020-11-20T12:16:17.198 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops_per_pg = 2.0 2020-11-20T12:16:17.199 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:16:17.200 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:16:17.200 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:16:17.200 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:16:17.229 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:16:17.230 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2020-11-20T12:16:17.231 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:16:17.231 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2020-11-20T12:16:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.495+0000 7fcfa18fd700 1 Processor -- start 2020-11-20T12:16:17.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.495+0000 7fcfa18fd700 1 -- start start 2020-11-20T12:16:17.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.496+0000 7f25e244d700 1 Processor -- start 2020-11-20T12:16:17.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.496+0000 7f25e244d700 1 -- start start 2020-11-20T12:16:17.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7f25e244d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25dc1460f0 0x7f25dc0624e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7f25e244d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc1464d0 0x7f25dc063d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7f25e244d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc146b60 con 0x7f25dc147540 2020-11-20T12:16:17.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7f25e244d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc1479a0 con 0x7f25dc1460f0 2020-11-20T12:16:17.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7f25e244d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc146950 con 0x7f25dc1464d0 2020-11-20T12:16:17.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7f25e144b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25dc1460f0 0x7f25dc0624e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7f25e144b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25dc1460f0 0x7f25dc0624e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50926/0 (socket says 172.21.15.187:50926) 2020-11-20T12:16:17.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7f25e144b700 1 -- 172.21.15.187:0/2235179042 learned_addr learned my addr 172.21.15.187:0/2235179042 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:17.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7f25e144b700 1 -- 172.21.15.187:0/2235179042 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc1464d0 msgr2=0x7f25dc063d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:17.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7f25e144b700 1 --2- 172.21.15.187:0/2235179042 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc1464d0 0x7f25dc063d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:17.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7f25e144b700 1 -- 172.21.15.187:0/2235179042 >> v1:172.21.15.5:6789/0 conn(0x7f25dc147540 legacy=0x7f25dc062da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:17.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7f25e144b700 1 -- 172.21.15.187:0/2235179042 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc146e00 con 0x7f25dc1460f0 2020-11-20T12:16:17.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7f25e144b700 1 --2- 172.21.15.187:0/2235179042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1460f0 0x7f25dc0624e0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 rx=0x7f25d8006fd0 tx=0x7f25d8006eb0).ready entity=mon.1 client_cookie=8b704749c2f19251 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:17.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25cbfff700 1 -- 172.21.15.187:0/2235179042 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8009890 con 0x7f25dc1460f0 2020-11-20T12:16:17.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25cbfff700 1 -- 172.21.15.187:0/2235179042 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d80099f0 con 0x7f25dc1460f0 2020-11-20T12:16:17.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25cbfff700 1 -- 172.21.15.187:0/2235179042 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8009d70 con 0x7f25dc1460f0 2020-11-20T12:16:17.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25e244d700 1 -- 172.21.15.187:0/2235179042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1460f0 msgr2=0x7f25dc0624e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:17.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25e244d700 1 --2- 172.21.15.187:0/2235179042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1460f0 0x7f25dc0624e0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 rx=0x7f25d8006fd0 tx=0x7f25d8006eb0).stop 2020-11-20T12:16:17.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25e244d700 1 -- 172.21.15.187:0/2235179042 shutdown_connections 2020-11-20T12:16:17.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25e244d700 1 --2- 172.21.15.187:0/2235179042 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc1464d0 0x7f25dc063d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:17.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25e244d700 1 --2- 172.21.15.187:0/2235179042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1460f0 0x7f25dc0624e0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:17.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25e244d700 1 -- 172.21.15.187:0/2235179042 >> 172.21.15.187:0/2235179042 conn(0x7f25dc0b88a0 msgr2=0x7f25dc0b8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:17.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7f25e244d700 1 -- 172.21.15.187:0/2235179042 shutdown_connections 2020-11-20T12:16:17.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.501+0000 7f25e244d700 1 -- 172.21.15.187:0/2235179042 wait complete. 2020-11-20T12:16:17.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.502+0000 7f25e244d700 1 Processor -- start 2020-11-20T12:16:17.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.502+0000 7f25e244d700 1 -- start start 2020-11-20T12:16:17.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7f25e244d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25dc1460f0 0x7f25dc144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7f25e244d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1464d0 0x7f25dc1192d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7f25e244d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc147540 0x7f25dc0a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7f25e244d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc13b2e0 con 0x7f25dc1460f0 2020-11-20T12:16:17.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7f25e244d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc13b580 con 0x7f25dc1464d0 2020-11-20T12:16:17.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7f25e244d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc13b820 con 0x7f25dc147540 2020-11-20T12:16:17.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7f25e0c4a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1464d0 0x7f25dc1192d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e0c4a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1464d0 0x7f25dc1192d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50940/0 (socket says 172.21.15.187:50940) 2020-11-20T12:16:17.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e0c4a700 1 -- 172.21.15.187:0/3565604064 learned_addr learned my addr 172.21.15.187:0/3565604064 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:17.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e144b700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25dc1460f0 0x7f25dc144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e1c4c700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc147540 0x7f25dc0a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e0c4a700 1 -- 172.21.15.187:0/3565604064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc147540 msgr2=0x7f25dc0a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:17.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e0c4a700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc147540 0x7f25dc0a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:17.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e0c4a700 1 -- 172.21.15.187:0/3565604064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25dc1460f0 msgr2=0x7f25dc144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:17.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e0c4a700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25dc1460f0 0x7f25dc144af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:17.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e0c4a700 1 -- 172.21.15.187:0/3565604064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc13bcc0 con 0x7f25dc1464d0 2020-11-20T12:16:17.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7f25e0c4a700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1464d0 0x7f25dc1192d0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7f25cc0096e0 tx=0x7f25cc006d40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:17.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7f25c97fa700 1 -- 172.21.15.187:0/3565604064 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc00d560 con 0x7f25dc1464d0 2020-11-20T12:16:17.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7f25c97fa700 1 -- 172.21.15.187:0/3565604064 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc000e90 con 0x7f25dc1464d0 2020-11-20T12:16:17.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7f25c97fa700 1 -- 172.21.15.187:0/3565604064 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc00d910 con 0x7f25dc1464d0 2020-11-20T12:16:17.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc144fe0 con 0x7f25dc1464d0 2020-11-20T12:16:17.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc0b8380 con 0x7f25dc1464d0 2020-11-20T12:16:17.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.506+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc065350 con 0x7f25dc1464d0 2020-11-20T12:16:17.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.507+0000 7f25c97fa700 1 -- 172.21.15.187:0/3565604064 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f25cc00f030 con 0x7f25dc1464d0 2020-11-20T12:16:17.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.508+0000 7f25c97fa700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f25b4056d20 0x7f25b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.508+0000 7f25c97fa700 1 -- 172.21.15.187:0/3565604064 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f25cc06fba0 con 0x7f25dc1464d0 2020-11-20T12:16:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.509+0000 7f25e144b700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f25b4056d20 0x7f25b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.509+0000 7f25e144b700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f25b4056d20 0x7f25b40591e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f25d800b1b0 tx=0x7f25d8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.516+0000 7f25c97fa700 1 -- 172.21.15.187:0/3565604064 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f25cc044ff0 con 0x7f25dc1464d0 2020-11-20T12:16:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.806+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f25dc13b9e0 con 0x7f25dc1464d0 2020-11-20T12:16:17.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.496+0000 7fcfa18fd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c0ba010 0x7fcf9c061870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.496+0000 7fcfa18fd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c061130 con 0x7fcf9c14c1a0 2020-11-20T12:16:17.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.496+0000 7fcfa18fd700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c0613d0 con 0x7fcf9c0b4070 2020-11-20T12:16:17.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.496+0000 7fcfa18fd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0bb7b0 con 0x7fcf9c0ba010 2020-11-20T12:16:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.497+0000 7fcfa10fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcf9c0b4070 0x7fcf9c0b4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47530/0 (socket says 172.21.15.187:47530) 2020-11-20T12:16:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.497+0000 7fcfa10fc700 1 -- 172.21.15.187:0/127862912 learned_addr learned my addr 172.21.15.187:0/127862912 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.497+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430220426 0 0) 0x7fcf9c0613d0 con 0x7fcf9c0b4070 2020-11-20T12:16:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.497+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf80003240 con 0x7fcf9c0b4070 2020-11-20T12:16:17.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.497+0000 7fcf9bfff700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c0ba010 0x7fcf9c061870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1323616342 0 0) 0x7fcf80003240 con 0x7fcf9c0b4070 2020-11-20T12:16:17.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c0ba010 msgr2=0x7fcf9c061870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:17.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c0ba010 0x7fcf9c061870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:17.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 >> v1:172.21.15.5:6789/0 conn(0x7fcf9c14c1a0 legacy=0x7fcf9c14c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:17.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0bb8f0 con 0x7fcf9c0b4070 2020-11-20T12:16:17.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcf94001860 con 0x7fcf9c0b4070 2020-11-20T12:16:17.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 >> v1:172.21.15.187:6789/0 conn(0x7fcf9c0b4070 legacy=0x7fcf9c0b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:17.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf80003730 0x7fcf80003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c0ba010 0x7fcf800044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800049d0 0x7fcf80004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0bb7b0 con 0x7fcf80003730 2020-11-20T12:16:17.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80004220 con 0x7fcf800049d0 2020-11-20T12:16:17.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.498+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8000b730 con 0x7fcf9c0ba010 2020-11-20T12:16:17.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7fcfa10fc700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800049d0 0x7fcf80004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7fcfa08fb700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf80003730 0x7fcf80003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7fcfa10fc700 1 -- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c0ba010 msgr2=0x7fcf800044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:17.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7fcfa10fc700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c0ba010 0x7fcf800044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:17.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7fcfa10fc700 1 -- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf80003730 msgr2=0x7fcf80003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:17.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7fcfa10fc700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf80003730 0x7fcf80003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:17.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7fcfa10fc700 1 -- 172.21.15.187:0/127862912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf80003240 con 0x7fcf800049d0 2020-11-20T12:16:17.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.499+0000 7fcfa08fb700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf80003730 0x7fcf80003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:17.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7fcfa10fc700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800049d0 0x7fcf80004e50 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 rx=0x7fcf9400c550 tx=0x7fcf94007a50).ready entity=mon.1 client_cookie=666db30e3e579d5b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:17.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94005b80 con 0x7fcf800049d0 2020-11-20T12:16:17.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94005ce0 con 0x7fcf800049d0 2020-11-20T12:16:17.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7fcf9b7fe700 1 -- 172.21.15.187:0/127862912 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94007320 con 0x7fcf800049d0 2020-11-20T12:16:17.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7fcfa18fd700 1 -- 172.21.15.187:0/127862912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800049d0 msgr2=0x7fcf80004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:17.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7fcfa18fd700 1 --2- 172.21.15.187:0/127862912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800049d0 0x7fcf80004e50 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 rx=0x7fcf9400c550 tx=0x7fcf94007a50).stop 2020-11-20T12:16:17.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.500+0000 7fcfa10fc700 1 -- 172.21.15.187:0/127862912 reap_dead start 2020-11-20T12:16:17.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.501+0000 7fcfa18fd700 1 -- 172.21.15.187:0/127862912 shutdown_connections 2020-11-20T12:16:17.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.501+0000 7fcfa18fd700 1 -- 172.21.15.187:0/127862912 >> 172.21.15.187:0/127862912 conn(0x7fcf9c0bf110 msgr2=0x7fcf9c0bf520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:17.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.501+0000 7fcfa18fd700 1 -- 172.21.15.187:0/127862912 shutdown_connections 2020-11-20T12:16:17.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.501+0000 7fcfa18fd700 1 -- 172.21.15.187:0/127862912 wait complete. 2020-11-20T12:16:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.502+0000 7fcfa18fd700 1 Processor -- start 2020-11-20T12:16:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.502+0000 7fcfa18fd700 1 -- start start 2020-11-20T12:16:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7fcfa18fd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf9c130ff0 0x7fcf9c0a42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7fcfa18fd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c1313d0 0x7fcf9c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7fcfa18fd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf9c15b3d0 0x7fcf9c1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7fcfa18fd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0a49d0 con 0x7fcf9c15b3d0 2020-11-20T12:16:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7fcfa18fd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0b68b0 con 0x7fcf9c130ff0 2020-11-20T12:16:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.503+0000 7fcfa18fd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0a4c00 con 0x7fcf9c1313d0 2020-11-20T12:16:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa08fb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf9c130ff0 0x7fcf9c0a42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa08fb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf9c130ff0 0x7fcf9c0a42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50944/0 (socket says 172.21.15.187:50944) 2020-11-20T12:16:17.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa08fb700 1 -- 172.21.15.187:0/38885094 learned_addr learned my addr 172.21.15.187:0/38885094 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:17.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa10fc700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf9c15b3d0 0x7fcf9c1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa08fb700 1 -- 172.21.15.187:0/38885094 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c1313d0 msgr2=0x7fcf9c0b8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:17.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa08fb700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c1313d0 0x7fcf9c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:17.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa08fb700 1 -- 172.21.15.187:0/38885094 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf9c15b3d0 msgr2=0x7fcf9c1573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:17.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa08fb700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf9c15b3d0 0x7fcf9c1573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:17.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.504+0000 7fcfa08fb700 1 -- 172.21.15.187:0/38885094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0a4ea0 con 0x7fcf9c130ff0 2020-11-20T12:16:17.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7fcfa08fb700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf9c130ff0 0x7fcf9c0a42a0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 rx=0x7fcf9000d7b0 tx=0x7fcf90009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:17.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7fcf98ff9700 1 -- 172.21.15.187:0/38885094 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000f5d0 con 0x7fcf9c130ff0 2020-11-20T12:16:17.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7fcf98ff9700 1 -- 172.21.15.187:0/38885094 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9000f730 con 0x7fcf9c130ff0 2020-11-20T12:16:17.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7fcf98ff9700 1 -- 172.21.15.187:0/38885094 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000fab0 con 0x7fcf9c130ff0 2020-11-20T12:16:17.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c15b850 con 0x7fcf9c130ff0 2020-11-20T12:16:17.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.505+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c157a90 con 0x7fcf9c130ff0 2020-11-20T12:16:17.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.507+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9c0b6a20 con 0x7fcf9c130ff0 2020-11-20T12:16:17.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.507+0000 7fcf98ff9700 1 -- 172.21.15.187:0/38885094 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcf9001c460 con 0x7fcf9c130ff0 2020-11-20T12:16:17.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.508+0000 7fcf98ff9700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf74056d70 0x7fcf74059230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:17.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.508+0000 7fcf98ff9700 1 -- 172.21.15.187:0/38885094 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcf9006ff70 con 0x7fcf9c130ff0 2020-11-20T12:16:17.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.509+0000 7fcf9bfff700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf74056d70 0x7fcf74059230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:17.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.509+0000 7fcf9bfff700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf74056d70 0x7fcf74059230 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7fcf88006d00 tx=0x7fcf88006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:17.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.512+0000 7fcf98ff9700 1 -- 172.21.15.187:0/38885094 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcf90045360 con 0x7fcf9c130ff0 2020-11-20T12:16:17.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:17.796+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fcf9c044f60 con 0x7fcf9c130ff0 2020-11-20T12:16:18.375 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:16:18.372+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:16:18.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.389+0000 7fcf98ff9700 1 -- 172.21.15.187:0/38885094 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 8473 v103) v1 ==== 95+0+0 (secure 0 0 0) 0x7fcf900077d0 con 0x7fcf9c130ff0 2020-11-20T12:16:18.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.389+0000 7f25c97fa700 1 -- 172.21.15.187:0/3565604064 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5726 v103) v1 ==== 95+0+0 (secure 0 0 0) 0x7f25cc006e60 con 0x7f25dc1464d0 2020-11-20T12:16:18.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf74056d70 msgr2=0x7fcf74059230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf74056d70 0x7fcf74059230 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7fcf88006d00 tx=0x7fcf88006df0).stop 2020-11-20T12:16:18.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf9c130ff0 msgr2=0x7fcf9c0a42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf9c130ff0 0x7fcf9c0a42a0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 rx=0x7fcf9000d7b0 tx=0x7fcf90009040).stop 2020-11-20T12:16:18.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 shutdown_connections 2020-11-20T12:16:18.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcf74056d70 0x7fcf74059230 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf9c15b3d0 0x7fcf9c1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf9c130ff0 0x7fcf9c0a42a0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 --2- 172.21.15.187:0/38885094 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcf9c1313d0 0x7fcf9c0b8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:18.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 >> 172.21.15.187:0/38885094 conn(0x7fcf9c0bf110 msgr2=0x7fcf9c0bcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:18.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 shutdown_connections 2020-11-20T12:16:18.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f25b4056d20 msgr2=0x7f25b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7f25e244d700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f25b4056d20 0x7f25b40591e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f25d800b1b0 tx=0x7f25d8015000).stop 2020-11-20T12:16:18.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1464d0 msgr2=0x7f25dc1192d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.393+0000 7f25e244d700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1464d0 0x7f25dc1192d0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7f25cc0096e0 tx=0x7f25cc006d40).stop 2020-11-20T12:16:18.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 shutdown_connections 2020-11-20T12:16:18.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7f25e244d700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f25b4056d20 0x7f25b40591e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7f25e244d700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25dc1460f0 0x7f25dc144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7f25e244d700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25dc147540 0x7f25dc0a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7fcfa18fd700 1 -- 172.21.15.187:0/38885094 wait complete. 2020-11-20T12:16:18.402 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 8473 2020-11-20T12:16:18.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7f25e244d700 1 --2- 172.21.15.187:0/3565604064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25dc1464d0 0x7f25dc1192d0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 >> 172.21.15.187:0/3565604064 conn(0x7f25dc0b88a0 msgr2=0x7f25dc145910 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:18.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 shutdown_connections 2020-11-20T12:16:18.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.394+0000 7f25e244d700 1 -- 172.21.15.187:0/3565604064 wait complete. 2020-11-20T12:16:18.403 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 5726 2020-11-20T12:16:18.422 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:16:18.423 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:16:18.424 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:16:18.425 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:16:18.437 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:16:18.438 DEBUG:teuthology.parallel:result is None 2020-11-20T12:16:18.442 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:16:18.442 DEBUG:teuthology.parallel:result is None 2020-11-20T12:16:18.443 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:18.604 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:16:18.641 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:16:18.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.668+0000 7f98aace3700 1 Processor -- start 2020-11-20T12:16:18.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.668+0000 7f98aace3700 1 -- start start 2020-11-20T12:16:18.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.669+0000 7f98aace3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067d70 0x7f98a414a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:18.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.669+0000 7f98aace3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f98a40afe40 0x7f98a4146a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:18.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.669+0000 7f98aace3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a4061650 con 0x7f98a4067990 2020-11-20T12:16:18.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.669+0000 7f98aace3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a414ad60 con 0x7f98a40afe40 2020-11-20T12:16:18.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.669+0000 7f98aace3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a412f850 con 0x7f98a4067d70 2020-11-20T12:16:18.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a8a7f700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f98a4067990 0x7f98a40b0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:41082/0 (socket says 172.21.15.187:41082) 2020-11-20T12:16:18.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a9280700 1 --2- 172.21.15.187:0/289577766 >> v2:172.21.15.5:3300/0 conn(0x7f98a40afe40 0x7f98a4146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:18.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a8a7f700 1 -- 172.21.15.187:0/289577766 learned_addr learned my addr 172.21.15.187:0/289577766 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:18.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a3fff700 1 --2- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067d70 0x7f98a414a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a37fe700 1 -- 172.21.15.187:0/289577766 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942009027 0 0) 0x7f98a4061650 con 0x7f98a4067990 2020-11-20T12:16:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a37fe700 1 -- 172.21.15.187:0/289577766 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9888003240 con 0x7f98a4067990 2020-11-20T12:16:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a9280700 1 -- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067d70 msgr2=0x7f98a414a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a9280700 1 --2- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067d70 0x7f98a414a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a9280700 1 -- 172.21.15.187:0/289577766 >> v1:172.21.15.5:6789/0 conn(0x7f98a4067990 legacy=0x7f98a40b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a9280700 1 -- 172.21.15.187:0/289577766 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a412faf0 con 0x7f98a40afe40 2020-11-20T12:16:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.670+0000 7f98a3fff700 1 --2- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067d70 0x7f98a414a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:16:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.671+0000 7f98a9280700 1 --2- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 0x7f98a4146a30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f989c009c90 tx=0x7f989c0091d0).ready entity=mon.0 client_cookie=a37ae17622e742fd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.671+0000 7f98a37fe700 1 -- 172.21.15.187:0/289577766 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c00b9c0 con 0x7f98a40afe40 2020-11-20T12:16:18.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.671+0000 7f98a37fe700 1 -- 172.21.15.187:0/289577766 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c00bb20 con 0x7f98a40afe40 2020-11-20T12:16:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.671+0000 7f98a37fe700 1 -- 172.21.15.187:0/289577766 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c00bea0 con 0x7f98a40afe40 2020-11-20T12:16:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.671+0000 7f98aace3700 1 -- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 msgr2=0x7f98a4146a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.671+0000 7f98aace3700 1 --2- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 0x7f98a4146a30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f989c009c90 tx=0x7f989c0091d0).stop 2020-11-20T12:16:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.672+0000 7f98aace3700 1 -- 172.21.15.187:0/289577766 shutdown_connections 2020-11-20T12:16:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.672+0000 7f98aace3700 1 --2- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 0x7f98a4146a30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.672+0000 7f98aace3700 1 --2- 172.21.15.187:0/289577766 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067d70 0x7f98a414a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.672+0000 7f98aace3700 1 -- 172.21.15.187:0/289577766 >> 172.21.15.187:0/289577766 conn(0x7f98a40bb950 msgr2=0x7f98a40b7920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.672+0000 7f98aace3700 1 -- 172.21.15.187:0/289577766 shutdown_connections 2020-11-20T12:16:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.672+0000 7f98aace3700 1 -- 172.21.15.187:0/289577766 wait complete. 2020-11-20T12:16:18.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.673+0000 7f98aace3700 1 Processor -- start 2020-11-20T12:16:18.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.673+0000 7f98aace3700 1 -- start start 2020-11-20T12:16:18.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.674+0000 7f98aace3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067990 0x7f98a4125790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:18.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.674+0000 7f98aace3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a4067d70 0x7f98a4145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:18.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.674+0000 7f98aace3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 0x7f98a4145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:18.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.674+0000 7f98aace3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a41464a0 con 0x7f98a40afe40 2020-11-20T12:16:18.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.674+0000 7f98aace3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a4125050 con 0x7f98a4067d70 2020-11-20T12:16:18.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.674+0000 7f98aace3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a41252f0 con 0x7f98a4067990 2020-11-20T12:16:18.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.674+0000 7f98a3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a4067d70 0x7f98a4145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:18.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a4067d70 0x7f98a4145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50974/0 (socket says 172.21.15.187:50974) 2020-11-20T12:16:18.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a3fff700 1 -- 172.21.15.187:0/3581438577 learned_addr learned my addr 172.21.15.187:0/3581438577 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:18.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a8a7f700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067990 0x7f98a4125790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:18.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a9280700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 0x7f98a4145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:18.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a3fff700 1 -- 172.21.15.187:0/3581438577 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067990 msgr2=0x7f98a4125790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a3fff700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067990 0x7f98a4125790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a3fff700 1 -- 172.21.15.187:0/3581438577 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 msgr2=0x7f98a4145fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a3fff700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 0x7f98a4145fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a3fff700 1 -- 172.21.15.187:0/3581438577 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4125c80 con 0x7f98a4067d70 2020-11-20T12:16:18.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.675+0000 7f98a3fff700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a4067d70 0x7f98a4145ac0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 rx=0x7f989000ed20 tx=0x7f9890006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:18.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.676+0000 7f98a0ff9700 1 -- 172.21.15.187:0/3581438577 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989000b570 con 0x7f98a4067d70 2020-11-20T12:16:18.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.676+0000 7f98a0ff9700 1 -- 172.21.15.187:0/3581438577 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989000cbb0 con 0x7f98a4067d70 2020-11-20T12:16:18.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.676+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a41519c0 con 0x7f98a4067d70 2020-11-20T12:16:18.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.676+0000 7f98a0ff9700 1 -- 172.21.15.187:0/3581438577 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98900123f0 con 0x7f98a4067d70 2020-11-20T12:16:18.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.676+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a414bc10 con 0x7f98a4067d70 2020-11-20T12:16:18.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.677+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a4124ea0 con 0x7f98a4067d70 2020-11-20T12:16:18.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.677+0000 7f98a0ff9700 1 -- 172.21.15.187:0/3581438577 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f98900127a0 con 0x7f98a4067d70 2020-11-20T12:16:18.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.678+0000 7f98a0ff9700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f987c056d20 0x7f987c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:18.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.678+0000 7f98a0ff9700 1 -- 172.21.15.187:0/3581438577 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9890070610 con 0x7f98a4067d70 2020-11-20T12:16:18.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.678+0000 7f98a8a7f700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f987c056d20 0x7f987c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:18.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.679+0000 7f98a8a7f700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f987c056d20 0x7f987c0591e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f98980055a0 tx=0x7f9898009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:18.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.682+0000 7f98a0ff9700 1 -- 172.21.15.187:0/3581438577 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9890045900 con 0x7f98a4067d70 2020-11-20T12:16:18.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.928+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98a41254b0 con 0x7f98a4067d70 2020-11-20T12:16:18.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.928+0000 7f98a0ff9700 1 -- 172.21.15.187:0/3581438577 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+1428 (secure 0 0 0) 0x7f9890048f20 con 0x7f98a4067d70 2020-11-20T12:16:18.932 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:18.932 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":103,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:18.372630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:18.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f987c056d20 msgr2=0x7f987c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f987c056d20 0x7f987c0591e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f98980055a0 tx=0x7f9898009040).stop 2020-11-20T12:16:18.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a4067d70 msgr2=0x7f98a4145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:18.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a4067d70 0x7f98a4145ac0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 rx=0x7f989000ed20 tx=0x7f9890006df0).stop 2020-11-20T12:16:18.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 shutdown_connections 2020-11-20T12:16:18.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f987c056d20 0x7f987c0591e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98a40afe40 0x7f98a4145fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98a4067990 0x7f98a4125790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 --2- 172.21.15.187:0/3581438577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a4067d70 0x7f98a4145ac0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:18.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 >> 172.21.15.187:0/3581438577 conn(0x7f98a40bb950 msgr2=0x7f98a4062880 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:18.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 shutdown_connections 2020-11-20T12:16:18.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:18.931+0000 7f98aace3700 1 -- 172.21.15.187:0/3581438577 wait complete. 2020-11-20T12:16:18.939 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 103 2020-11-20T12:16:19.159 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:16:03.867608+0000', 'modified': '2020-11-20T12:16:18.372630+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:16:19.159 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2020-11-20T12:16:20.161 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:20.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.406+0000 7fc70827a700 1 Processor -- start 2020-11-20T12:16:20.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.406+0000 7fc70827a700 1 -- start start 2020-11-20T12:16:20.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.408+0000 7fc70827a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc7000b81f0 0x7fc700065870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:20.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.408+0000 7fc70827a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc70009f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:20.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.408+0000 7fc70827a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc700067920 con 0x7fc7000b85d0 2020-11-20T12:16:20.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.408+0000 7fc70827a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc700148b00 con 0x7fc7000b81f0 2020-11-20T12:16:20.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.408+0000 7fc70827a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc700148cf0 con 0x7fc7001498a0 2020-11-20T12:16:20.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.409+0000 7fc706016700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc7000b81f0 0x7fc700065870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:20.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.409+0000 7fc706817700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc70009f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:20.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.409+0000 7fc706016700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc7000b81f0 0x7fc700065870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:43246/0 (socket says 172.21.15.187:43246) 2020-11-20T12:16:20.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.409+0000 7fc706016700 1 -- 172.21.15.187:0/1942474003 learned_addr learned my addr 172.21.15.187:0/1942474003 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:20.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.410+0000 7fc706817700 1 -- 172.21.15.187:0/1942474003 >> v2:172.21.15.5:3300/0 conn(0x7fc7000b81f0 msgr2=0x7fc700065870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:20.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.410+0000 7fc706817700 1 --2- 172.21.15.187:0/1942474003 >> v2:172.21.15.5:3300/0 conn(0x7fc7000b81f0 0x7fc700065870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.410+0000 7fc706817700 1 -- 172.21.15.187:0/1942474003 >> v1:172.21.15.5:6789/0 conn(0x7fc7000b85d0 legacy=0x7fc700149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:20.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.410+0000 7fc706817700 1 -- 172.21.15.187:0/1942474003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700067110 con 0x7fc7001498a0 2020-11-20T12:16:20.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.410+0000 7fc706016700 1 --2- 172.21.15.187:0/1942474003 >> v2:172.21.15.5:3300/0 conn(0x7fc7000b81f0 0x7fc700065870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:16:20.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.410+0000 7fc706817700 1 --2- 172.21.15.187:0/1942474003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc70009f220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc6f8009c90 tx=0x7fc6f80091d0).ready entity=mon.2 client_cookie=abe157477ecf8eaa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:20.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc705014700 1 -- 172.21.15.187:0/1942474003 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f800bb00 con 0x7fc7001498a0 2020-11-20T12:16:20.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc705014700 1 -- 172.21.15.187:0/1942474003 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f800bc60 con 0x7fc7001498a0 2020-11-20T12:16:20.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc705014700 1 -- 172.21.15.187:0/1942474003 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8011400 con 0x7fc7001498a0 2020-11-20T12:16:20.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc70827a700 1 -- 172.21.15.187:0/1942474003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 msgr2=0x7fc70009f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:20.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc70827a700 1 --2- 172.21.15.187:0/1942474003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc70009f220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc6f8009c90 tx=0x7fc6f80091d0).stop 2020-11-20T12:16:20.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc70827a700 1 -- 172.21.15.187:0/1942474003 shutdown_connections 2020-11-20T12:16:20.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc70827a700 1 --2- 172.21.15.187:0/1942474003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc70009f220 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc70827a700 1 --2- 172.21.15.187:0/1942474003 >> v2:172.21.15.5:3300/0 conn(0x7fc7000b81f0 0x7fc700065870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc70827a700 1 -- 172.21.15.187:0/1942474003 >> 172.21.15.187:0/1942474003 conn(0x7fc7000bb950 msgr2=0x7fc7000b7920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:20.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc70827a700 1 -- 172.21.15.187:0/1942474003 shutdown_connections 2020-11-20T12:16:20.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.411+0000 7fc70827a700 1 -- 172.21.15.187:0/1942474003 wait complete. 2020-11-20T12:16:20.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.413+0000 7fc70827a700 1 Processor -- start 2020-11-20T12:16:20.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.413+0000 7fc70827a700 1 -- start start 2020-11-20T12:16:20.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.414+0000 7fc70827a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7000b81f0 0x7fc7001354f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:20.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.414+0000 7fc70827a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7000b85d0 0x7fc700134860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:20.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.414+0000 7fc70827a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc700134d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:20.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.414+0000 7fc70827a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc700135e40 con 0x7fc7000b81f0 2020-11-20T12:16:20.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.414+0000 7fc70827a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000b62c0 con 0x7fc7000b85d0 2020-11-20T12:16:20.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.414+0000 7fc70827a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000b6560 con 0x7fc7001498a0 2020-11-20T12:16:20.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.414+0000 7fc705815700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7000b85d0 0x7fc700134860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:20.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc705815700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7000b85d0 0x7fc700134860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:50988/0 (socket says 172.21.15.187:50988) 2020-11-20T12:16:20.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc705815700 1 -- 172.21.15.187:0/149082081 learned_addr learned my addr 172.21.15.187:0/149082081 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:20.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc706817700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc700134d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:20.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc706016700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7000b81f0 0x7fc7001354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:20.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc705815700 1 -- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 msgr2=0x7fc700134d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:20.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc705815700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc700134d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc705815700 1 -- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7000b81f0 msgr2=0x7fc7001354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:20.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc705815700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7000b81f0 0x7fc7001354f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc705815700 1 -- 172.21.15.187:0/149082081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7001359e0 con 0x7fc7000b85d0 2020-11-20T12:16:20.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.415+0000 7fc706016700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7000b81f0 0x7fc7001354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:16:20.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.416+0000 7fc705815700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7000b85d0 0x7fc700134860 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7fc6f000c710 tx=0x7fc6f0007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:20.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.416+0000 7fc6ee7fc700 1 -- 172.21.15.187:0/149082081 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f000e560 con 0x7fc7000b85d0 2020-11-20T12:16:20.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.416+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc700145690 con 0x7fc7000b85d0 2020-11-20T12:16:20.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.416+0000 7fc6ee7fc700 1 -- 172.21.15.187:0/149082081 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f000e6c0 con 0x7fc7000b85d0 2020-11-20T12:16:20.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.416+0000 7fc6ee7fc700 1 -- 172.21.15.187:0/149082081 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f000ea40 con 0x7fc7000b85d0 2020-11-20T12:16:20.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.416+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7000b53e0 con 0x7fc7000b85d0 2020-11-20T12:16:20.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.417+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7000b6110 con 0x7fc7000b85d0 2020-11-20T12:16:20.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.418+0000 7fc6ee7fc700 1 -- 172.21.15.187:0/149082081 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc6f0019030 con 0x7fc7000b85d0 2020-11-20T12:16:20.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.418+0000 7fc6ee7fc700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d8056d20 0x7fc6d80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:20.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.418+0000 7fc6ee7fc700 1 -- 172.21.15.187:0/149082081 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc6f006f6b0 con 0x7fc7000b85d0 2020-11-20T12:16:20.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.419+0000 7fc706016700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d8056d20 0x7fc6d80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:20.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.419+0000 7fc706016700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d8056d20 0x7fc6d80591e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7fc7001489e0 tx=0x7fc6fc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:20.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.423+0000 7fc6ee7fc700 1 -- 172.21.15.187:0/149082081 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc6f0044ab0 con 0x7fc7000b85d0 2020-11-20T12:16:20.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.669+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc700069660 con 0x7fc7000b85d0 2020-11-20T12:16:20.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.670+0000 7fc6ee7fc700 1 -- 172.21.15.187:0/149082081 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+1428 (secure 0 0 0) 0x7fc6f0017090 con 0x7fc7000b85d0 2020-11-20T12:16:20.673 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:20.673 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":103,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:18.372630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:20.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.672+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d8056d20 msgr2=0x7fc6d80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:20.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.672+0000 7fc70827a700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d8056d20 0x7fc6d80591e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7fc7001489e0 tx=0x7fc6fc008040).stop 2020-11-20T12:16:20.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.672+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7000b85d0 msgr2=0x7fc700134860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:20.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.672+0000 7fc70827a700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7000b85d0 0x7fc700134860 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7fc6f000c710 tx=0x7fc6f0007c10).stop 2020-11-20T12:16:20.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.673+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 shutdown_connections 2020-11-20T12:16:20.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.673+0000 7fc70827a700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc6d8056d20 0x7fc6d80591e0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.673+0000 7fc70827a700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc7000b81f0 0x7fc7001354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.673+0000 7fc70827a700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc7001498a0 0x7fc700134d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.673+0000 7fc70827a700 1 --2- 172.21.15.187:0/149082081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7000b85d0 0x7fc700134860 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:20.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.673+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 >> 172.21.15.187:0/149082081 conn(0x7fc7000bb950 msgr2=0x7fc700141c40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:20.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.673+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 shutdown_connections 2020-11-20T12:16:20.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:20.673+0000 7fc70827a700 1 -- 172.21.15.187:0/149082081 wait complete. 2020-11-20T12:16:20.678 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 103 2020-11-20T12:16:20.903 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:16:03.867608+0000', 'modified': '2020-11-20T12:16:18.372630+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:16:20.904 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2020-11-20T12:16:21.906 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:22.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.154+0000 7fc4f62b2700 1 Processor -- start 2020-11-20T12:16:22.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.154+0000 7fc4f62b2700 1 -- start start 2020-11-20T12:16:22.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.155+0000 7fc4f62b2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc4f0065c90 0x7fc4f0150b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:22.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.155+0000 7fc4f62b2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f009a410 0x7fc4f0151080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:22.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.155+0000 7fc4f62b2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00b57b0 con 0x7fc4f0066070 2020-11-20T12:16:22.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.155+0000 7fc4f62b2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f01506f0 con 0x7fc4f0065c90 2020-11-20T12:16:22.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.155+0000 7fc4f62b2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00b1570 con 0x7fc4f009a410 2020-11-20T12:16:22.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.156+0000 7fc4f52b0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc4f0065c90 0x7fc4f0150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:22.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.156+0000 7fc4f4aaf700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc4f0066070 0x7fc4f0099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47600/0 (socket says 172.21.15.187:47600) 2020-11-20T12:16:22.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.156+0000 7fc4f4aaf700 1 -- 172.21.15.187:0/4103642034 learned_addr learned my addr 172.21.15.187:0/4103642034 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:22.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.157+0000 7fc4f5ab1700 1 --2- 172.21.15.187:0/4103642034 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f009a410 0x7fc4f0151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:22.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.157+0000 7fc4f52b0700 1 -- 172.21.15.187:0/4103642034 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f009a410 msgr2=0x7fc4f0151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:22.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.157+0000 7fc4f52b0700 1 --2- 172.21.15.187:0/4103642034 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f009a410 0x7fc4f0151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.157+0000 7fc4f52b0700 1 -- 172.21.15.187:0/4103642034 >> v1:172.21.15.187:6789/0 conn(0x7fc4f0066070 legacy=0x7fc4f0099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:22.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.157+0000 7fc4f52b0700 1 -- 172.21.15.187:0/4103642034 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f00b16e0 con 0x7fc4f0065c90 2020-11-20T12:16:22.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.157+0000 7fc4dffff700 1 -- 172.21.15.187:0/4103642034 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121522264 0 0) 0x7fc4f00b57b0 con 0x7fc4f0066070 2020-11-20T12:16:22.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.157+0000 7fc4f52b0700 1 --2- 172.21.15.187:0/4103642034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f0065c90 0x7fc4f0150b90 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7fc4ec009c90 tx=0x7fc4ec0091d0).ready entity=mon.1 client_cookie=162441cc742702fa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:22.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4dffff700 1 -- 172.21.15.187:0/4103642034 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec00bbc0 con 0x7fc4f0065c90 2020-11-20T12:16:22.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4dffff700 1 -- 172.21.15.187:0/4103642034 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ec00bd20 con 0x7fc4f0065c90 2020-11-20T12:16:22.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4dffff700 1 -- 172.21.15.187:0/4103642034 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec011610 con 0x7fc4f0065c90 2020-11-20T12:16:22.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4f62b2700 1 -- 172.21.15.187:0/4103642034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f0065c90 msgr2=0x7fc4f0150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:22.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/4103642034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f0065c90 0x7fc4f0150b90 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7fc4ec009c90 tx=0x7fc4ec0091d0).stop 2020-11-20T12:16:22.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4f62b2700 1 -- 172.21.15.187:0/4103642034 shutdown_connections 2020-11-20T12:16:22.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/4103642034 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f009a410 0x7fc4f0151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/4103642034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f0065c90 0x7fc4f0150b90 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4f62b2700 1 -- 172.21.15.187:0/4103642034 >> 172.21.15.187:0/4103642034 conn(0x7fc4f00b8280 msgr2=0x7fc4f00653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:22.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4f62b2700 1 -- 172.21.15.187:0/4103642034 shutdown_connections 2020-11-20T12:16:22.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.158+0000 7fc4f62b2700 1 -- 172.21.15.187:0/4103642034 wait complete. 2020-11-20T12:16:22.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.160+0000 7fc4f62b2700 1 Processor -- start 2020-11-20T12:16:22.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.160+0000 7fc4f62b2700 1 -- start start 2020-11-20T12:16:22.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.161+0000 7fc4f62b2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4f0065c90 0x7fc4f009dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:22.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.161+0000 7fc4f62b2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f0066070 0x7fc4f009b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:22.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.161+0000 7fc4f62b2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f009a410 0x7fc4f014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:22.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.161+0000 7fc4f62b2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00b35d0 con 0x7fc4f0065c90 2020-11-20T12:16:22.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.161+0000 7fc4f62b2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f009aad0 con 0x7fc4f009a410 2020-11-20T12:16:22.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.161+0000 7fc4f62b2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f009ad70 con 0x7fc4f0066070 2020-11-20T12:16:22.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.161+0000 7fc4f5ab1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f009a410 0x7fc4f014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:22.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f5ab1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f009a410 0x7fc4f014c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51004/0 (socket says 172.21.15.187:51004) 2020-11-20T12:16:22.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f5ab1700 1 -- 172.21.15.187:0/14679191 learned_addr learned my addr 172.21.15.187:0/14679191 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:22.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f4aaf700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f0066070 0x7fc4f009b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:22.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f52b0700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4f0065c90 0x7fc4f009dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:22.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f5ab1700 1 -- 172.21.15.187:0/14679191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f0066070 msgr2=0x7fc4f009b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:22.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f5ab1700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f0066070 0x7fc4f009b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f5ab1700 1 -- 172.21.15.187:0/14679191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4f0065c90 msgr2=0x7fc4f009dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:22.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f5ab1700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4f0065c90 0x7fc4f009dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f5ab1700 1 -- 172.21.15.187:0/14679191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f009d610 con 0x7fc4f009a410 2020-11-20T12:16:22.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.162+0000 7fc4f5ab1700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f009a410 0x7fc4f014c800 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7fc4e800d790 tx=0x7fc4e8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:22.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.163+0000 7fc4dd7fa700 1 -- 172.21.15.187:0/14679191 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8009630 con 0x7fc4f009a410 2020-11-20T12:16:22.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.163+0000 7fc4dd7fa700 1 -- 172.21.15.187:0/14679191 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e8009790 con 0x7fc4f009a410 2020-11-20T12:16:22.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.163+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f014ccf0 con 0x7fc4f009a410 2020-11-20T12:16:22.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.163+0000 7fc4dd7fa700 1 -- 172.21.15.187:0/14679191 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8009b10 con 0x7fc4f009a410 2020-11-20T12:16:22.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.163+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f0149c00 con 0x7fc4f009a410 2020-11-20T12:16:22.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.164+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f0068ef0 con 0x7fc4f009a410 2020-11-20T12:16:22.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.164+0000 7fc4dd7fa700 1 -- 172.21.15.187:0/14679191 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc4e8013480 con 0x7fc4f009a410 2020-11-20T12:16:22.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.165+0000 7fc4dd7fa700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4c8056d20 0x7fc4c80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:22.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.165+0000 7fc4dd7fa700 1 -- 172.21.15.187:0/14679191 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc4e8070080 con 0x7fc4f009a410 2020-11-20T12:16:22.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.165+0000 7fc4f52b0700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4c8056d20 0x7fc4c80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:22.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.166+0000 7fc4f52b0700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4c8056d20 0x7fc4c80591e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7fc4ec009210 tx=0x7fc4ec016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:22.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.169+0000 7fc4dd7fa700 1 -- 172.21.15.187:0/14679191 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc4e8045310 con 0x7fc4f009a410 2020-11-20T12:16:22.182 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:16:22.192 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:16:22.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.422+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4f009af30 con 0x7fc4f009a410 2020-11-20T12:16:22.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.423+0000 7fc4dd7fa700 1 -- 172.21.15.187:0/14679191 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+1428 (secure 0 0 0) 0x7fc4e8048930 con 0x7fc4f009a410 2020-11-20T12:16:22.427 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:22.427 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":103,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:18.372630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4c8056d20 msgr2=0x7fc4c80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4c8056d20 0x7fc4c80591e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7fc4ec009210 tx=0x7fc4ec016000).stop 2020-11-20T12:16:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f009a410 msgr2=0x7fc4f014c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:22.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f009a410 0x7fc4f014c800 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7fc4e800d790 tx=0x7fc4e8008040).stop 2020-11-20T12:16:22.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 shutdown_connections 2020-11-20T12:16:22.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc4c8056d20 0x7fc4c80591e0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4f0065c90 0x7fc4f009dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f009a410 0x7fc4f014c800 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 --2- 172.21.15.187:0/14679191 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4f0066070 0x7fc4f009b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 >> 172.21.15.187:0/14679191 conn(0x7fc4f00b8280 msgr2=0x7fc4f00afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:22.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.425+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 shutdown_connections 2020-11-20T12:16:22.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:22.426+0000 7fc4f62b2700 1 -- 172.21.15.187:0/14679191 wait complete. 2020-11-20T12:16:22.432 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 103 2020-11-20T12:16:22.653 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:16:03.867608+0000', 'modified': '2020-11-20T12:16:18.372630+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:16:22.653 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2020-11-20T12:16:23.654 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:23.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.904+0000 7f9da885a700 1 Processor -- start 2020-11-20T12:16:23.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.904+0000 7f9da885a700 1 -- start start 2020-11-20T12:16:23.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.905+0000 7f9da885a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da40b00b0 0x7f9da413e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:23.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.905+0000 7f9da885a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9da413ee00 0x7f9da40b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:23.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.905+0000 7f9da885a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da4148770 con 0x7f9da4148f90 2020-11-20T12:16:23.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.905+0000 7f9da885a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da40b5490 con 0x7f9da413ee00 2020-11-20T12:16:23.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.905+0000 7f9da885a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da409a3f0 con 0x7f9da40b00b0 2020-11-20T12:16:23.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.905+0000 7f9da37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9da4148f90 0x7f9da4099ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47624/0 (socket says 172.21.15.187:47624) 2020-11-20T12:16:23.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.905+0000 7f9da37fe700 1 -- 172.21.15.187:0/3513130663 learned_addr learned my addr 172.21.15.187:0/3513130663 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:16:23.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.905+0000 7f9da3fff700 1 --2- 172.21.15.187:0/3513130663 >> v2:172.21.15.187:3300/0 conn(0x7f9da413ee00 0x7f9da40b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:23.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.906+0000 7f9d9affd700 1 --2- 172.21.15.187:0/3513130663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da40b00b0 0x7f9da413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:23.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.906+0000 7f9da3fff700 1 -- 172.21.15.187:0/3513130663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da40b00b0 msgr2=0x7f9da413e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:23.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.906+0000 7f9da3fff700 1 --2- 172.21.15.187:0/3513130663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da40b00b0 0x7f9da413e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:23.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.906+0000 7f9da3fff700 1 -- 172.21.15.187:0/3513130663 >> v1:172.21.15.187:6789/0 conn(0x7f9da4148f90 legacy=0x7f9da4099ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:23.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.906+0000 7f9da2ffd700 1 -- 172.21.15.187:0/3513130663 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873969191 0 0) 0x7f9da4148770 con 0x7f9da4148f90 2020-11-20T12:16:23.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.906+0000 7f9da3fff700 1 -- 172.21.15.187:0/3513130663 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da4066750 con 0x7f9da413ee00 2020-11-20T12:16:23.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.906+0000 7f9da3fff700 1 --2- 172.21.15.187:0/3513130663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da40b5930 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 rx=0x7f9d9c009c90 tx=0x7f9d9c0091d0).ready entity=mon.1 client_cookie=7bb2f0a5d2e0491d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:23.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.906+0000 7f9da2ffd700 1 -- 172.21.15.187:0/3513130663 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c00bbc0 con 0x7f9da413ee00 2020-11-20T12:16:23.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da2ffd700 1 -- 172.21.15.187:0/3513130663 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c00bd20 con 0x7f9da413ee00 2020-11-20T12:16:23.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da2ffd700 1 -- 172.21.15.187:0/3513130663 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c011610 con 0x7f9da413ee00 2020-11-20T12:16:23.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da885a700 1 -- 172.21.15.187:0/3513130663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 msgr2=0x7f9da40b5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:23.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da885a700 1 --2- 172.21.15.187:0/3513130663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da40b5930 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 rx=0x7f9d9c009c90 tx=0x7f9d9c0091d0).stop 2020-11-20T12:16:23.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da885a700 1 -- 172.21.15.187:0/3513130663 shutdown_connections 2020-11-20T12:16:23.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da885a700 1 --2- 172.21.15.187:0/3513130663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da40b5930 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:23.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da885a700 1 --2- 172.21.15.187:0/3513130663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da40b00b0 0x7f9da413e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:23.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da885a700 1 -- 172.21.15.187:0/3513130663 >> 172.21.15.187:0/3513130663 conn(0x7f9da40b8280 msgr2=0x7f9da40653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:23.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da885a700 1 -- 172.21.15.187:0/3513130663 shutdown_connections 2020-11-20T12:16:23.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.907+0000 7f9da885a700 1 -- 172.21.15.187:0/3513130663 wait complete. 2020-11-20T12:16:23.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.909+0000 7f9da885a700 1 Processor -- start 2020-11-20T12:16:23.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.909+0000 7f9da885a700 1 -- start start 2020-11-20T12:16:23.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9da885a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9da40b00b0 0x7f9da40b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:23.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9da885a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da415cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:23.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9da885a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da4148f90 0x7f9da4144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:23.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9da885a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da415d140 con 0x7f9da40b00b0 2020-11-20T12:16:23.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9da885a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da415c510 con 0x7f9da413ee00 2020-11-20T12:16:23.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9da885a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da415c7b0 con 0x7f9da4148f90 2020-11-20T12:16:23.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9d9affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da415cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:23.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9d9affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da415cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51026/0 (socket says 172.21.15.187:51026) 2020-11-20T12:16:23.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9d9affd700 1 -- 172.21.15.187:0/1088654836 learned_addr learned my addr 172.21.15.187:0/1088654836 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:23.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9da3fff700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da4148f90 0x7f9da4144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:23.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.910+0000 7f9d9affd700 1 -- 172.21.15.187:0/1088654836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da4148f90 msgr2=0x7f9da4144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:23.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9d9affd700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da4148f90 0x7f9da4144d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:23.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9da37fe700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9da40b00b0 0x7f9da40b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:23.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9d9affd700 1 -- 172.21.15.187:0/1088654836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9da40b00b0 msgr2=0x7f9da40b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:23.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9d9affd700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9da40b00b0 0x7f9da40b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:23.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9d9affd700 1 -- 172.21.15.187:0/1088654836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da415c1c0 con 0x7f9da413ee00 2020-11-20T12:16:23.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9d9affd700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da415cc50 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 rx=0x7f9d8c00db60 tx=0x7f9d8c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:23.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9d9bfff700 1 -- 172.21.15.187:0/1088654836 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c009820 con 0x7f9da413ee00 2020-11-20T12:16:23.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9d9bfff700 1 -- 172.21.15.187:0/1088654836 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c009980 con 0x7f9da413ee00 2020-11-20T12:16:23.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da409b890 con 0x7f9da413ee00 2020-11-20T12:16:23.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9d9bfff700 1 -- 172.21.15.187:0/1088654836 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c009d40 con 0x7f9da413ee00 2020-11-20T12:16:23.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.911+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da4099fc0 con 0x7f9da413ee00 2020-11-20T12:16:23.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.912+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da4068ef0 con 0x7f9da413ee00 2020-11-20T12:16:23.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.913+0000 7f9d9bfff700 1 -- 172.21.15.187:0/1088654836 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9d8c012630 con 0x7f9da413ee00 2020-11-20T12:16:23.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.913+0000 7f9d9bfff700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9d80056d20 0x7f9d800591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:23.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.914+0000 7f9d9bfff700 1 -- 172.21.15.187:0/1088654836 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9d8c070360 con 0x7f9da413ee00 2020-11-20T12:16:23.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.914+0000 7f9da37fe700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9d80056d20 0x7f9d800591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:23.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.914+0000 7f9da37fe700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9d80056d20 0x7f9d800591e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f9da409c630 tx=0x7f9d94009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:23.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:23.917+0000 7f9d9bfff700 1 -- 172.21.15.187:0/1088654836 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9d8c045760 con 0x7f9da413ee00 2020-11-20T12:16:24.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.169+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9da415c970 con 0x7f9da413ee00 2020-11-20T12:16:24.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.170+0000 7f9d9bfff700 1 -- 172.21.15.187:0/1088654836 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f9d8c048d80 con 0x7f9da413ee00 2020-11-20T12:16:24.174 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:24.175 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":105,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:23.429277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:replay","state_seq":2,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:24.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9d80056d20 msgr2=0x7f9d800591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:24.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9d80056d20 0x7f9d800591e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f9da409c630 tx=0x7f9d94009040).stop 2020-11-20T12:16:24.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 msgr2=0x7f9da415cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:24.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da415cc50 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 rx=0x7f9d8c00db60 tx=0x7f9d8c008040).stop 2020-11-20T12:16:24.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 shutdown_connections 2020-11-20T12:16:24.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9d80056d20 0x7f9d800591e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9da40b00b0 0x7f9da40b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9da4148f90 0x7f9da4144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 --2- 172.21.15.187:0/1088654836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9da413ee00 0x7f9da415cc50 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 >> 172.21.15.187:0/1088654836 conn(0x7f9da40b8280 msgr2=0x7f9da40600f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:24.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 shutdown_connections 2020-11-20T12:16:24.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:24.173+0000 7f9da885a700 1 -- 172.21.15.187:0/1088654836 wait complete. 2020-11-20T12:16:24.179 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 105 2020-11-20T12:16:24.401 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:16:03.867608+0000', 'modified': '2020-11-20T12:16:23.429277+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8647}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8647': {'gid': 8647, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.187:6835/3515729375', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3515729375}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3515729375}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:16:24.401 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8647:up:replay 2020-11-20T12:16:25.403 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.647+0000 7f8cb4b4d700 1 Processor -- start 2020-11-20T12:16:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.647+0000 7f8cb4b4d700 1 -- start start 2020-11-20T12:16:25.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.649+0000 7f8cb4b4d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cb0065c90 0x7f8cb0150b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:25.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.649+0000 7f8cb4b4d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb009a410 0x7f8cb0151080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:25.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.649+0000 7f8cb4b4d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb00b57b0 con 0x7f8cb0066070 2020-11-20T12:16:25.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.649+0000 7f8cb4b4d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb01506f0 con 0x7f8cb0065c90 2020-11-20T12:16:25.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.649+0000 7f8cb4b4d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00b1570 con 0x7f8cb009a410 2020-11-20T12:16:25.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.649+0000 7f8caf7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cb0065c90 0x7f8cb0150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.649+0000 7f8caf7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cb0065c90 0x7f8cb0150b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51034/0 (socket says 172.21.15.187:51034) 2020-11-20T12:16:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.649+0000 7f8caf7fe700 1 -- 172.21.15.187:0/190663233 learned_addr learned my addr 172.21.15.187:0/190663233 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.650+0000 7f8caf7fe700 1 -- 172.21.15.187:0/190663233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb009a410 msgr2=0x7f8cb0151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.650+0000 7f8caffff700 1 --2- 172.21.15.187:0/190663233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb009a410 0x7f8cb0151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:25.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.650+0000 7f8caf7fe700 1 --2- 172.21.15.187:0/190663233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb009a410 0x7f8cb0151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.650+0000 7f8caf7fe700 1 -- 172.21.15.187:0/190663233 >> v1:172.21.15.5:6789/0 conn(0x7f8cb0066070 legacy=0x7f8cb0099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:25.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.650+0000 7f8caf7fe700 1 -- 172.21.15.187:0/190663233 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb00b16e0 con 0x7f8cb0065c90 2020-11-20T12:16:25.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.651+0000 7f8caf7fe700 1 --2- 172.21.15.187:0/190663233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb0065c90 0x7f8cb0150b90 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 rx=0x7f8ca0009d50 tx=0x7f8ca0009290).ready entity=mon.1 client_cookie=7a485e88614cf2a2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:25.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.651+0000 7f8caeffd700 1 -- 172.21.15.187:0/190663233 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca000bc40 con 0x7f8cb0065c90 2020-11-20T12:16:25.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.651+0000 7f8caeffd700 1 -- 172.21.15.187:0/190663233 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca000bda0 con 0x7f8cb0065c90 2020-11-20T12:16:25.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.651+0000 7f8caeffd700 1 -- 172.21.15.187:0/190663233 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0011610 con 0x7f8cb0065c90 2020-11-20T12:16:25.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.652+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/190663233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb0065c90 msgr2=0x7f8cb0150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:25.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.652+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/190663233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb0065c90 0x7f8cb0150b90 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 rx=0x7f8ca0009d50 tx=0x7f8ca0009290).stop 2020-11-20T12:16:25.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.652+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/190663233 shutdown_connections 2020-11-20T12:16:25.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.652+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/190663233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb009a410 0x7f8cb0151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.652+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/190663233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb0065c90 0x7f8cb0150b90 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.652+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/190663233 >> 172.21.15.187:0/190663233 conn(0x7f8cb00b8280 msgr2=0x7f8cb00653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:25.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.652+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/190663233 shutdown_connections 2020-11-20T12:16:25.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.652+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/190663233 wait complete. 2020-11-20T12:16:25.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.654+0000 7f8cb4b4d700 1 Processor -- start 2020-11-20T12:16:25.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.654+0000 7f8cb4b4d700 1 -- start start 2020-11-20T12:16:25.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.655+0000 7f8cb4b4d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb0066070 0x7f8cb014b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:25.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.655+0000 7f8cb4b4d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb009a410 0x7f8cb009c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:25.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.655+0000 7f8cb4b4d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cb015fd90 0x7f8cb01601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:25.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.655+0000 7f8cb4b4d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb009cc20 con 0x7f8cb015fd90 2020-11-20T12:16:25.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.655+0000 7f8cb4b4d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb009ce40 con 0x7f8cb009a410 2020-11-20T12:16:25.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.655+0000 7f8cb4b4d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00b4e20 con 0x7f8cb0066070 2020-11-20T12:16:25.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.655+0000 7f8ca6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb009a410 0x7f8cb009c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:25.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8ca6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb009a410 0x7f8cb009c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51042/0 (socket says 172.21.15.187:51042) 2020-11-20T12:16:25.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8ca6ffd700 1 -- 172.21.15.187:0/194349524 learned_addr learned my addr 172.21.15.187:0/194349524 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:25.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8caf7fe700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb0066070 0x7f8cb014b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:25.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8caffff700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cb015fd90 0x7f8cb01601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:25.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8ca6ffd700 1 -- 172.21.15.187:0/194349524 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb0066070 msgr2=0x7f8cb014b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:25.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8ca6ffd700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb0066070 0x7f8cb014b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8ca6ffd700 1 -- 172.21.15.187:0/194349524 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cb015fd90 msgr2=0x7f8cb01601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:25.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8ca6ffd700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cb015fd90 0x7f8cb01601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.656+0000 7f8ca6ffd700 1 -- 172.21.15.187:0/194349524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb00a0690 con 0x7f8cb009a410 2020-11-20T12:16:25.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.657+0000 7f8ca6ffd700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb009a410 0x7f8cb009c730 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 rx=0x7f8c9800c760 tx=0x7f8c98007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:25.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.657+0000 7f8ca7fff700 1 -- 172.21.15.187:0/194349524 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9800e560 con 0x7f8cb009a410 2020-11-20T12:16:25.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.657+0000 7f8ca7fff700 1 -- 172.21.15.187:0/194349524 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9800e6e0 con 0x7f8cb009a410 2020-11-20T12:16:25.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.657+0000 7f8ca7fff700 1 -- 172.21.15.187:0/194349524 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9800eaa0 con 0x7f8cb009a410 2020-11-20T12:16:25.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.657+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb00b46e0 con 0x7f8cb009a410 2020-11-20T12:16:25.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.657+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb01606e0 con 0x7f8cb009a410 2020-11-20T12:16:25.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.659+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb0068ef0 con 0x7f8cb009a410 2020-11-20T12:16:25.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.659+0000 7f8ca7fff700 1 -- 172.21.15.187:0/194349524 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8c98010070 con 0x7f8cb009a410 2020-11-20T12:16:25.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.660+0000 7f8ca7fff700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c8c056d20 0x7f8c8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:25.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.660+0000 7f8ca7fff700 1 -- 172.21.15.187:0/194349524 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8c9806fc20 con 0x7f8cb009a410 2020-11-20T12:16:25.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.660+0000 7f8caf7fe700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c8c056d20 0x7f8c8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:25.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.661+0000 7f8caf7fe700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c8c056d20 0x7f8c8c0591e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f8cb00695d0 tx=0x7f8ca0016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:25.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.664+0000 7f8ca7fff700 1 -- 172.21.15.187:0/194349524 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8c98045020 con 0x7f8cb009a410 2020-11-20T12:16:25.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.907+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cb00b4b40 con 0x7f8cb009a410 2020-11-20T12:16:25.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.907+0000 7f8ca7fff700 1 -- 172.21.15.187:0/194349524 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f8c98048640 con 0x7f8cb009a410 2020-11-20T12:16:25.911 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:25.912 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":107,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:25.444342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:25.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.909+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c8c056d20 msgr2=0x7f8c8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:25.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c8c056d20 0x7f8c8c0591e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f8cb00695d0 tx=0x7f8ca0016000).stop 2020-11-20T12:16:25.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb009a410 msgr2=0x7f8cb009c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:25.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb009a410 0x7f8cb009c730 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 rx=0x7f8c9800c760 tx=0x7f8c98007c30).stop 2020-11-20T12:16:25.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 shutdown_connections 2020-11-20T12:16:25.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c8c056d20 0x7f8c8c0591e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cb015fd90 0x7f8cb01601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cb0066070 0x7f8cb014b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 --2- 172.21.15.187:0/194349524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cb009a410 0x7f8cb009c730 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:25.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 >> 172.21.15.187:0/194349524 conn(0x7f8cb00b8280 msgr2=0x7f8cb00629b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:25.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 shutdown_connections 2020-11-20T12:16:25.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:25.910+0000 7f8cb4b4d700 1 -- 172.21.15.187:0/194349524 wait complete. 2020-11-20T12:16:25.917 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 107 2020-11-20T12:16:26.138 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:16:03.867608+0000', 'modified': '2020-11-20T12:16:25.444342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8647}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8647': {'gid': 8647, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.187:6835/3515729375', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3515729375}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3515729375}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:16:26.138 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8647:up:rejoin 2020-11-20T12:16:27.140 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:27.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.376+0000 7fe2c3fff700 1 Processor -- start 2020-11-20T12:16:27.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.376+0000 7fe2c3fff700 1 -- start start 2020-11-20T12:16:27.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.377+0000 7fe2c3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe2c4066070 0x7fe2c414cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.378+0000 7fe2c3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c414c2a0 con 0x7fe2c4065c90 2020-11-20T12:16:27.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.378+0000 7fe2c3fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c414c540 con 0x7fe2c40b6dc0 2020-11-20T12:16:27.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.378+0000 7fe2c3fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c414c7e0 con 0x7fe2c4066070 2020-11-20T12:16:27.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.378+0000 7fe2ba7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe2c4066070 0x7fe2c414cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:27.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.378+0000 7fe2ba7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe2c4066070 0x7fe2c414cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51052/0 (socket says 172.21.15.187:51052) 2020-11-20T12:16:27.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.378+0000 7fe2ba7fc700 1 -- 172.21.15.187:0/1882169775 learned_addr learned my addr 172.21.15.187:0/1882169775 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:27.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.378+0000 7fe2ba7fc700 1 -- 172.21.15.187:0/1882169775 >> v1:172.21.15.187:6789/0 conn(0x7fe2c40b6dc0 legacy=0x7fe2c4145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:27.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.379+0000 7fe2ba7fc700 1 -- 172.21.15.187:0/1882169775 >> v1:172.21.15.5:6789/0 conn(0x7fe2c4065c90 legacy=0x7fe2c40b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:27.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.379+0000 7fe2ba7fc700 1 -- 172.21.15.187:0/1882169775 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c4066bb0 con 0x7fe2c4066070 2020-11-20T12:16:27.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.379+0000 7fe2ba7fc700 1 --2- 172.21.15.187:0/1882169775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4066070 0x7fe2c414cc80 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7fe2ac006fd0 tx=0x7fe2ac006eb0).ready entity=mon.1 client_cookie=26c5d1ba6f84096 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:27.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.379+0000 7fe2c27fc700 1 -- 172.21.15.187:0/1882169775 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac009890 con 0x7fe2c4066070 2020-11-20T12:16:27.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.379+0000 7fe2c27fc700 1 -- 172.21.15.187:0/1882169775 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac0099f0 con 0x7fe2c4066070 2020-11-20T12:16:27.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.379+0000 7fe2c27fc700 1 -- 172.21.15.187:0/1882169775 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac009d70 con 0x7fe2c4066070 2020-11-20T12:16:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.380+0000 7fe2c3fff700 1 -- 172.21.15.187:0/1882169775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4066070 msgr2=0x7fe2c414cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.380+0000 7fe2c3fff700 1 --2- 172.21.15.187:0/1882169775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4066070 0x7fe2c414cc80 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7fe2ac006fd0 tx=0x7fe2ac006eb0).stop 2020-11-20T12:16:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.380+0000 7fe2c3fff700 1 -- 172.21.15.187:0/1882169775 shutdown_connections 2020-11-20T12:16:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.380+0000 7fe2c3fff700 1 --2- 172.21.15.187:0/1882169775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4066070 0x7fe2c414cc80 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.380+0000 7fe2c3fff700 1 -- 172.21.15.187:0/1882169775 >> 172.21.15.187:0/1882169775 conn(0x7fe2c40b8280 msgr2=0x7fe2c40653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:27.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.380+0000 7fe2c3fff700 1 -- 172.21.15.187:0/1882169775 shutdown_connections 2020-11-20T12:16:27.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.380+0000 7fe2c3fff700 1 -- 172.21.15.187:0/1882169775 wait complete. 2020-11-20T12:16:27.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.381+0000 7fe2c3fff700 1 Processor -- start 2020-11-20T12:16:27.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.382+0000 7fe2c3fff700 1 -- start start 2020-11-20T12:16:27.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.382+0000 7fe2c3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4065c90 0x7fe2c40b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.382+0000 7fe2c3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c4066070 0x7fe2c4092ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.382+0000 7fe2c3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c40b6dc0 0x7fe2c409d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.382+0000 7fe2c3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40b3e10 con 0x7fe2c4066070 2020-11-20T12:16:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.382+0000 7fe2c3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40b40b0 con 0x7fe2c4065c90 2020-11-20T12:16:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40b4350 con 0x7fe2c40b6dc0 2020-11-20T12:16:27.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4065c90 0x7fe2c40b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:27.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4065c90 0x7fe2c40b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51056/0 (socket says 172.21.15.187:51056) 2020-11-20T12:16:27.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 -- 172.21.15.187:0/3765435529 learned_addr learned my addr 172.21.15.187:0/3765435529 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:16:27.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 -- 172.21.15.187:0/3765435529 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c40b6dc0 msgr2=0x7fe2c409d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c40b6dc0 0x7fe2c409d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:27.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 -- 172.21.15.187:0/3765435529 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c4066070 msgr2=0x7fe2c4092ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c4066070 0x7fe2c4092ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:27.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 -- 172.21.15.187:0/3765435529 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c40934e0 con 0x7fe2c4065c90 2020-11-20T12:16:27.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.383+0000 7fe2c2ffd700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4065c90 0x7fe2c40b47f0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 rx=0x7fe2b400c730 tx=0x7fe2b4007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:27.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.384+0000 7fe2bb7fe700 1 -- 172.21.15.187:0/3765435529 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b400e560 con 0x7fe2c4065c90 2020-11-20T12:16:27.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.384+0000 7fe2bb7fe700 1 -- 172.21.15.187:0/3765435529 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b400e6c0 con 0x7fe2c4065c90 2020-11-20T12:16:27.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.384+0000 7fe2bb7fe700 1 -- 172.21.15.187:0/3765435529 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b400ea80 con 0x7fe2c4065c90 2020-11-20T12:16:27.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.384+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c409d5e0 con 0x7fe2c4065c90 2020-11-20T12:16:27.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.384+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c409a930 con 0x7fe2c4065c90 2020-11-20T12:16:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.385+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c4068ef0 con 0x7fe2c4065c90 2020-11-20T12:16:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.386+0000 7fe2bb7fe700 1 -- 172.21.15.187:0/3765435529 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe2b4010070 con 0x7fe2c4065c90 2020-11-20T12:16:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.386+0000 7fe2bb7fe700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe294056d70 0x7fe294059230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.386+0000 7fe2bb7fe700 1 -- 172.21.15.187:0/3765435529 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe2b406fb30 con 0x7fe2c4065c90 2020-11-20T12:16:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.386+0000 7fe2ba7fc700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe294056d70 0x7fe294059230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.387+0000 7fe2ba7fc700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe294056d70 0x7fe294059230 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fe2c40b1390 tx=0x7fe2ac015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.390+0000 7fe2bb7fe700 1 -- 172.21.15.187:0/3765435529 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe2b4044fb0 con 0x7fe2c4065c90 2020-11-20T12:16:27.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.636+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2c409d7d0 con 0x7fe2c4065c90 2020-11-20T12:16:27.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.637+0000 7fe2bb7fe700 1 -- 172.21.15.187:0/3765435529 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7fe2b40485d0 con 0x7fe2c4065c90 2020-11-20T12:16:27.641 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:16:27.642 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:16:27.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.639+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe294056d70 msgr2=0x7fe294059230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.639+0000 7fe2c3fff700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe294056d70 0x7fe294059230 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fe2c40b1390 tx=0x7fe2ac015000).stop 2020-11-20T12:16:27.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.639+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4065c90 msgr2=0x7fe2c40b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.639+0000 7fe2c3fff700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4065c90 0x7fe2c40b47f0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 rx=0x7fe2b400c730 tx=0x7fe2b4007c30).stop 2020-11-20T12:16:27.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.640+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 shutdown_connections 2020-11-20T12:16:27.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.640+0000 7fe2c3fff700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe294056d70 0x7fe294059230 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.640+0000 7fe2c3fff700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c40b6dc0 0x7fe2c409d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:27.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.640+0000 7fe2c3fff700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2c4065c90 0x7fe2c40b47f0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.640+0000 7fe2c3fff700 1 --2- 172.21.15.187:0/3765435529 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c4066070 0x7fe2c4092ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:16:27.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.640+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 >> 172.21.15.187:0/3765435529 conn(0x7fe2c40b8280 msgr2=0x7fe2c4061de0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:27.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.640+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 shutdown_connections 2020-11-20T12:16:27.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:16:27.640+0000 7fe2c3fff700 1 -- 172.21.15.187:0/3765435529 wait complete. 2020-11-20T12:16:27.651 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:16:27.868 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:16:03.867608+0000', 'modified': '2020-11-20T12:16:26.456436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8647}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8647': {'gid': 8647, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.187:6835/3515729375', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3515729375}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3515729375}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:16:27.870 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:16:27.871 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:16:28.155 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8647,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.6915516139999998,"mdsmap_epoch":108,"osdmap_epoch":121,"osdmap_epoch_barrier":121,"uptime":9.5256697740000007} 2020-11-20T12:16:28.156 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8647, "mdsmap_epoch": 108, "osdmap_epoch": 121, "osdmap_epoch_barrier": 121, "rank_uptime": 4.691551614, "state": "up:active", "uptime": 9.525669774, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:16:28.157 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:16:28.157 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:16:28.157 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:16:28.158 INFO:teuthology.orchestra.run.smithi187:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2020-11-20T12:16:28.158 INFO:teuthology.orchestra.run.smithi187:> subdir = "delete_me" 2020-11-20T12:16:28.159 INFO:teuthology.orchestra.run.smithi187:> size_unit = 1048576 2020-11-20T12:16:28.159 INFO:teuthology.orchestra.run.smithi187:> file_multiplier = 100 2020-11-20T12:16:28.159 INFO:teuthology.orchestra.run.smithi187:> os.mkdir(os.path.join(mountpoint, subdir)) 2020-11-20T12:16:28.160 INFO:teuthology.orchestra.run.smithi187:> for i in range(0, file_multiplier): 2020-11-20T12:16:28.160 INFO:teuthology.orchestra.run.smithi187:> for size in range(0, 16*size_unit, size_unit): 2020-11-20T12:16:28.160 INFO:teuthology.orchestra.run.smithi187:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2020-11-20T12:16:28.161 INFO:teuthology.orchestra.run.smithi187:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2020-11-20T12:16:28.161 INFO:teuthology.orchestra.run.smithi187:> f.write(size * '"'"'x'"'"') 2020-11-20T12:16:28.162 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:16:28.448 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:16:28.444+0000 7ff82432c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:16:52.338 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:16:52.347 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:22.038 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_max_purge_ops 2020-11-20T12:17:22.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:17:22.407 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -rf delete_me) 2020-11-20T12:17:22.499 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:22.510 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:22.719 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_max_purge_ops":"16"} 2020-11-20T12:17:22.719 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "16" } 2020-11-20T12:17:22.720 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_files 2020-11-20T12:17:23.005 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_max_purge_files":"100000000"} 2020-11-20T12:17:23.006 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "100000000" } 2020-11-20T12:17:23.007 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:23.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.232+0000 7fd637fff700 1 Processor -- start 2020-11-20T12:17:23.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.232+0000 7fd637fff700 1 -- start start 2020-11-20T12:17:23.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.233+0000 7fd637fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd638062fb0 0x7fd6380b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:23.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.233+0000 7fd637fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd638067ee0 0x7fd63814c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:23.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.233+0000 7fd637fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63814c960 con 0x7fd638062bd0 2020-11-20T12:17:23.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.233+0000 7fd637fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63814c780 con 0x7fd638067ee0 2020-11-20T12:17:23.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.233+0000 7fd637fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63814cc00 con 0x7fd638062fb0 2020-11-20T12:17:23.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.234+0000 7fd636ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd638062bd0 0x7fd6380677d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47680/0 (socket says 172.21.15.187:47680) 2020-11-20T12:17:23.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.234+0000 7fd636ffd700 1 -- 172.21.15.187:0/551135084 learned_addr learned my addr 172.21.15.187:0/551135084 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:23.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.234+0000 7fd6367fc700 1 --2- 172.21.15.187:0/551135084 >> v2:172.21.15.187:3300/0 conn(0x7fd638062fb0 0x7fd6380b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:23.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.234+0000 7fd635ffb700 1 -- 172.21.15.187:0/551135084 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161342572 0 0) 0x7fd63814c960 con 0x7fd638062bd0 2020-11-20T12:17:23.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.234+0000 7fd6377fe700 1 --2- 172.21.15.187:0/551135084 >> v2:172.21.15.5:3300/0 conn(0x7fd638067ee0 0x7fd63814c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:23.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.234+0000 7fd635ffb700 1 -- 172.21.15.187:0/551135084 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd618003240 con 0x7fd638062bd0 2020-11-20T12:17:23.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.235+0000 7fd6367fc700 1 -- 172.21.15.187:0/551135084 >> v2:172.21.15.5:3300/0 conn(0x7fd638067ee0 msgr2=0x7fd63814c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:23.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.235+0000 7fd6367fc700 1 --2- 172.21.15.187:0/551135084 >> v2:172.21.15.5:3300/0 conn(0x7fd638067ee0 0x7fd63814c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.235+0000 7fd6367fc700 1 -- 172.21.15.187:0/551135084 >> v1:172.21.15.187:6789/0 conn(0x7fd638062bd0 legacy=0x7fd6380677d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.235+0000 7fd6367fc700 1 -- 172.21.15.187:0/551135084 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63814cea0 con 0x7fd638062fb0 2020-11-20T12:17:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.235+0000 7fd6367fc700 1 --2- 172.21.15.187:0/551135084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062fb0 0x7fd6380b35c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7fd6380633e0 tx=0x7fd620006df0).ready entity=mon.1 client_cookie=df5c5ab7d321d6ca server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.235+0000 7fd635ffb700 1 -- 172.21.15.187:0/551135084 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620009840 con 0x7fd638062fb0 2020-11-20T12:17:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.235+0000 7fd635ffb700 1 -- 172.21.15.187:0/551135084 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6200099a0 con 0x7fd638062fb0 2020-11-20T12:17:23.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.235+0000 7fd635ffb700 1 -- 172.21.15.187:0/551135084 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620009d20 con 0x7fd638062fb0 2020-11-20T12:17:23.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.236+0000 7fd637fff700 1 -- 172.21.15.187:0/551135084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062fb0 msgr2=0x7fd6380b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:23.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.236+0000 7fd637fff700 1 --2- 172.21.15.187:0/551135084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062fb0 0x7fd6380b35c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7fd6380633e0 tx=0x7fd620006df0).stop 2020-11-20T12:17:23.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.236+0000 7fd637fff700 1 -- 172.21.15.187:0/551135084 shutdown_connections 2020-11-20T12:17:23.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.236+0000 7fd637fff700 1 --2- 172.21.15.187:0/551135084 >> v2:172.21.15.5:3300/0 conn(0x7fd638067ee0 0x7fd63814c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:23.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.236+0000 7fd637fff700 1 --2- 172.21.15.187:0/551135084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062fb0 0x7fd6380b35c0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:23.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.236+0000 7fd637fff700 1 -- 172.21.15.187:0/551135084 >> 172.21.15.187:0/551135084 conn(0x7fd6380b9990 msgr2=0x7fd6380b9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:23.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.236+0000 7fd637fff700 1 -- 172.21.15.187:0/551135084 shutdown_connections 2020-11-20T12:17:23.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.236+0000 7fd637fff700 1 -- 172.21.15.187:0/551135084 wait complete. 2020-11-20T12:17:23.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.238+0000 7fd637fff700 1 Processor -- start 2020-11-20T12:17:23.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.238+0000 7fd637fff700 1 -- start start 2020-11-20T12:17:23.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd637fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062bd0 0x7fd638148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:23.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd637fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd638062fb0 0x7fd6380bc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:23.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd637fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd638067ee0 0x7fd6380939c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:23.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd637fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b69f0 con 0x7fd638067ee0 2020-11-20T12:17:23.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd637fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b6750 con 0x7fd638062bd0 2020-11-20T12:17:23.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd637fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd638093520 con 0x7fd638062fb0 2020-11-20T12:17:23.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd636ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062bd0 0x7fd638148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:23.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd636ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062bd0 0x7fd638148d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51080/0 (socket says 172.21.15.187:51080) 2020-11-20T12:17:23.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.239+0000 7fd636ffd700 1 -- 172.21.15.187:0/3973905499 learned_addr learned my addr 172.21.15.187:0/3973905499 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:23.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.240+0000 7fd636ffd700 1 -- 172.21.15.187:0/3973905499 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd638062fb0 msgr2=0x7fd6380bc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:23.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.240+0000 7fd636ffd700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd638062fb0 0x7fd6380bc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:23.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.240+0000 7fd636ffd700 1 -- 172.21.15.187:0/3973905499 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd638067ee0 msgr2=0x7fd6380939c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:23.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.240+0000 7fd636ffd700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd638067ee0 0x7fd6380939c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:23.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.240+0000 7fd636ffd700 1 -- 172.21.15.187:0/3973905499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd638149220 con 0x7fd638062bd0 2020-11-20T12:17:23.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.241+0000 7fd636ffd700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062bd0 0x7fd638148d30 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 rx=0x7fd62800c6a0 tx=0x7fd628007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:23.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.241+0000 7fd62effd700 1 -- 172.21.15.187:0/3973905499 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280072e0 con 0x7fd638062bd0 2020-11-20T12:17:23.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.241+0000 7fd62effd700 1 -- 172.21.15.187:0/3973905499 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628007440 con 0x7fd638062bd0 2020-11-20T12:17:23.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.241+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63814b620 con 0x7fd638062bd0 2020-11-20T12:17:23.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.241+0000 7fd62effd700 1 -- 172.21.15.187:0/3973905499 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628005840 con 0x7fd638062bd0 2020-11-20T12:17:23.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.242+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6380b6320 con 0x7fd638062bd0 2020-11-20T12:17:23.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.244+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6380b65a0 con 0x7fd638062bd0 2020-11-20T12:17:23.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.244+0000 7fd62effd700 1 -- 172.21.15.187:0/3973905499 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd628010070 con 0x7fd638062bd0 2020-11-20T12:17:23.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.245+0000 7fd62effd700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd614056d20 0x7fd6140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:23.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.245+0000 7fd62effd700 1 -- 172.21.15.187:0/3973905499 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd62806f600 con 0x7fd638062bd0 2020-11-20T12:17:23.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.245+0000 7fd6367fc700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd614056d20 0x7fd6140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:23.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.246+0000 7fd6367fc700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd614056d20 0x7fd6140591e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fd6380b7a20 tx=0x7fd620015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:23.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.249+0000 7fd62effd700 1 -- 172.21.15.187:0/3973905499 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd628044a00 con 0x7fd638062bd0 2020-11-20T12:17:23.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.506+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6380936e0 con 0x7fd638062bd0 2020-11-20T12:17:23.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.507+0000 7fd62effd700 1 -- 172.21.15.187:0/3973905499 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7fd628007b50 con 0x7fd638062bd0 2020-11-20T12:17:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:23.510 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:23.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd614056d20 msgr2=0x7fd6140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:23.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd614056d20 0x7fd6140591e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fd6380b7a20 tx=0x7fd620015000).stop 2020-11-20T12:17:23.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062bd0 msgr2=0x7fd638148d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:23.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062bd0 0x7fd638148d30 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 rx=0x7fd62800c6a0 tx=0x7fd628007ac0).stop 2020-11-20T12:17:23.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 shutdown_connections 2020-11-20T12:17:23.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd614056d20 0x7fd6140591e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:23.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd638067ee0 0x7fd6380939c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:23.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd638062bd0 0x7fd638148d30 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:23.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 --2- 172.21.15.187:0/3973905499 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd638062fb0 0x7fd6380bc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:23.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 >> 172.21.15.187:0/3973905499 conn(0x7fd6380b9990 msgr2=0x7fd6380a2870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:23.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.510+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 shutdown_connections 2020-11-20T12:17:23.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:23.511+0000 7fd637fff700 1 -- 172.21.15.187:0/3973905499 wait complete. 2020-11-20T12:17:23.516 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:23.535 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:23.809 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":852,"msgr_send_messages":854,"msgr_recv_bytes":216328,"msgr_send_bytes":8417029,"msgr_created_connections":6,"msgr_active_connections":0,"msgr_running_total_time":0.271782462,"msgr_running_send_time":0.110798933,"msgr_running_recv_time":0.055677471,"msgr_running_fast_dispatch_time":0.081181730,"msgr_send_messages_queue_lat":{"avgcount":852,"sum":0.081903780,"avgtime":0.000096131},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7181,"msgr_send_messages":9560,"msgr_recv_bytes":2058178,"msgr_send_bytes":8789196,"msgr_created_connections":5,"msgr_active_connections":2,"msgr_running_total_time":2.515884805,"msgr_running_send_time":1.881369012,"msgr_running_recv_time":0.758982405,"msgr_running_fast_dispatch_time":0.057286297,"msgr_send_messages_queue_lat":{"avgcount":9559,"sum":1.409784311,"avgtime":0.000147482},"msgr_handle_ack_lat":{"avgcount":7729,"sum":0.005803988,"avgtime":0.000000750}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":65,"msgr_send_messages":62,"msgr_recv_bytes":97412,"msgr_send_bytes":292052,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.025449210,"msgr_running_send_time":0.008831004,"msgr_running_recv_time":0.018355769,"msgr_running_fast_dispatch_time":0.004188727,"msgr_send_messages_queue_lat":{"avgcount":62,"sum":0.016001956,"avgtime":0.000258096},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1226,"complete_latency":{"avgcount":3065,"sum":2.556182029,"avgtime":0.000833990}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000222226,"avgtime":0.000055556}},"mds":{"request":2391,"reply":2391,"reply_latency":{"avgcount":2391,"sum":3.621400359,"avgtime":0.001514596},"forward":0,"dir_fetch":12,"dir_commit":73,"dir_split":9,"dir_merge":0,"inodes":1611,"inodes_top":0,"inodes_bottom":1611,"inodes_pin_tail":0,"inodes_pinned":1611,"inodes_expired":0,"inodes_with_caps":1602,"caps":1602,"subtrees":2,"traverse":2396,"traverse_hit":2393,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":250689,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1599,"root_rbytes":10756292608,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":36,"ceph_cap_op_grant":3301,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3940,"handle_client_caps_dirty":1290,"handle_client_cap_release":0,"process_request_cap_release":790},"mds_cache":{"num_strays":724,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":724,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":5857,"evex":0,"evtrm":0,"ev":5861,"evexg":0,"evexd":0,"segadd":7,"segex":0,"segtrm":0,"seg":8,"segexg":0,"segexd":0,"expos":4194304,"wrpos":16480918,"rdpos":4195632,"jlat":{"avgcount":1531,"sum":18.179650699,"avgtime":0.011874363},"replayed":4},"mds_mem":{"ino":1614,"ino+":1614,"ino-":0,"dir":76,"dir+":85,"dir-":9,"dn":2335,"dn+":2335,"dn-":0,"cap":1602,"cap+":1602,"cap-":0,"rss":73420,"heap":356572},"mds_server":{"dispatch_client_request":2430,"dispatch_server_request":0,"handle_client_request":2391,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":724,"sum":0.843665451,"avgtime":0.001165283},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":1948,"average_load":1948,"avg_session_uptime":1226},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18057476,"buffer_anon_items":6811,"buffer_meta_bytes":1760,"buffer_meta_items":20,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":7937059,"mds_co_items":98603,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1663,"op_send_bytes":12972850,"op_resend":0,"op_reply":1663,"oplen_avg":{"avgcount":1663,"sum":1785},"op":1663,"op_r":26,"op_w":1637,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":1531,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":15,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":5,"osd_session_open":5,"osd_session_close":0,"osd_laggy":0,"omap_wr":170,"omap_rd":28,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":8094,"get_sum":1742688,"get_or_fail_fail":0,"get_or_fail_success":8094,"take":0,"take_sum":0,"put":8094,"put_sum":1742688,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1663,"take_sum":12974736,"put":1635,"put_sum":12974736,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1663,"take_sum":1663,"put":1663,"put_sum":1663,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":179004,"max":3758096384,"get_started":0,"get":5857,"get_sum":12284504,"get_or_fail_fail":0,"get_or_fail_success":5857,"take":0,"take_sum":0,"put":1531,"put_sum":12105500,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:23.811 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 0, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 216328, "msgr_recv_messages": 852, "msgr_running_fast_dispatch_time": 0.08118173, "msgr_running_recv_time": 0.055677471, "msgr_running_send_time": 0.110798933, "msgr_running_total_time": 0.271782462, "msgr_send_bytes": 8417029, "msgr_send_messages": 854, "msgr_send_messages_queue_lat": { "avgcount": 852, "avgtime": 9.6131e-05, "sum": 0.08190378 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 2, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 7729, "avgtime": 7.5e-07, "sum": 0.005803988 }, "msgr_recv_bytes": 2058178, "msgr_recv_messages": 7181, "msgr_running_fast_dispatch_time": 0.057286297, "msgr_running_recv_time": 0.758982405, "msgr_running_send_time": 1.881369012, "msgr_running_total_time": 2.515884805, "msgr_send_bytes": 8789196, "msgr_send_messages": 9560, "msgr_send_messages_queue_lat": { "avgcount": 9559, "avgtime": 0.000147482, "sum": 1.409784311 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 97412, "msgr_recv_messages": 65, "msgr_running_fast_dispatch_time": 0.004188727, "msgr_running_recv_time": 0.018355769, "msgr_running_send_time": 0.008831004, "msgr_running_total_time": 0.02544921, "msgr_send_bytes": 292052, "msgr_send_messages": 62, "msgr_send_messages_queue_lat": { "avgcount": 62, "avgtime": 0.000258096, "sum": 0.016001956 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3065, "avgtime": 0.00083399, "sum": 2.556182029 }, "queue_len": 1226 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 5.5556e-05, "sum": 0.000222226 }, "queue_len": 0 }, "mds": { "caps": 1602, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3301, "ceph_cap_op_revoke": 36, "ceph_cap_op_trunc": 0, "dir_commit": 73, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3940, "handle_client_caps_dirty": 1290, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1611, "inodes_bottom": 1611, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 1611, "inodes_top": 0, "inodes_with_caps": 1602, "load_cent": 250689, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 790, "q": 0, "reply": 2391, "reply_latency": { "avgcount": 2391, "avgtime": 0.001514596, "sum": 3.621400359 }, "request": 2391, "root_rbytes": 10756292608, "root_rfiles": 1599, "root_rsnaps": 0, "subtrees": 2, "traverse": 2396, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2393, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 724, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 724, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 5861, "evadd": 5857, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 1531, "avgtime": 0.011874363, "sum": 18.179650699 }, "rdpos": 4195632, "replayed": 4, "seg": 8, "segadd": 7, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 16480918 }, "mds_mem": { "cap": 1602, "cap+": 1602, "cap-": 0, "dir": 76, "dir+": 85, "dir-": 9, "dn": 2335, "dn+": 2335, "dn-": 0, "heap": 356572, "ino": 1614, "ino+": 1614, "ino-": 0, "rss": 73420 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2430, "dispatch_server_request": 0, "handle_client_request": 2391, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 724, "avgtime": 0.001165283, "sum": 0.843665451 } }, "mds_sessions": { "average_load": 1948, "avg_session_uptime": 1226, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 1948 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18057476, "buffer_anon_items": 6811, "buffer_meta_bytes": 1760, "buffer_meta_items": 20, "mds_co_bytes": 7937059, "mds_co_items": 98603, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 28, "omap_wr": 170, "op": 1663, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 26, "op_reply": 1663, "op_resend": 0, "op_rmw": 0, "op_send": 1663, "op_send_bytes": 12972850, "op_w": 1637, "oplen_avg": { "avgcount": 1663, "sum": 1785 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 5, "osd_sessions": 5, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 15, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1531, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8094, "get_or_fail_fail": 0, "get_or_fail_success": 8094, "get_started": 0, "get_sum": 1742688, "max": 104857600, "put": 8094, "put_sum": 1742688, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1635, "put_sum": 12974736, "take": 1663, "take_sum": 12974736, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1663, "put_sum": 1663, "take": 1663, "take_sum": 1663, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 5857, "get_or_fail_fail": 0, "get_or_fail_success": 5857, "get_started": 0, "get_sum": 12284504, "max": 3758096384, "put": 1531, "put_sum": 12105500, "take": 0, "take_sum": 0, "val": 179004, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:23.812 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/724, 0/724 2020-11-20T12:17:24.813 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:25.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.039+0000 7f4eb7247700 1 Processor -- start 2020-11-20T12:17:25.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.039+0000 7f4eb7247700 1 -- start start 2020-11-20T12:17:25.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.040+0000 7f4eb7247700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b1fc0 0x7f4eb00b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:25.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.040+0000 7f4eb7247700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4eb00b76a0 0x7f4eb0147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:25.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.040+0000 7f4eb7247700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00b7bb0 con 0x7f4eb00b1be0 2020-11-20T12:17:25.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.040+0000 7f4eb7247700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb0065c40 con 0x7f4eb00b76a0 2020-11-20T12:17:25.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.040+0000 7f4eb7247700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb01484a0 con 0x7f4eb00b1fc0 2020-11-20T12:17:25.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.041+0000 7f4eb5a44700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b1fc0 0x7f4eb00b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:25.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.041+0000 7f4eb6245700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4eb00b1be0 0x7f4eb0060ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:41204/0 (socket says 172.21.15.187:41204) 2020-11-20T12:17:25.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.041+0000 7f4eb6245700 1 -- 172.21.15.187:0/143885537 learned_addr learned my addr 172.21.15.187:0/143885537 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:25.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.041+0000 7f4eb6a46700 1 --2- 172.21.15.187:0/143885537 >> v2:172.21.15.5:3300/0 conn(0x7f4eb00b76a0 0x7f4eb0147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:25.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.042+0000 7f4eb5243700 1 -- 172.21.15.187:0/143885537 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005591821 0 0) 0x7f4eb00b7bb0 con 0x7f4eb00b1be0 2020-11-20T12:17:25.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.042+0000 7f4eb5243700 1 -- 172.21.15.187:0/143885537 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e98003240 con 0x7f4eb00b1be0 2020-11-20T12:17:25.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.042+0000 7f4eb5a44700 1 -- 172.21.15.187:0/143885537 >> v2:172.21.15.5:3300/0 conn(0x7f4eb00b76a0 msgr2=0x7f4eb0147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:25.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.042+0000 7f4eb5a44700 1 --2- 172.21.15.187:0/143885537 >> v2:172.21.15.5:3300/0 conn(0x7f4eb00b76a0 0x7f4eb0147fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:25.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.042+0000 7f4eb5a44700 1 -- 172.21.15.187:0/143885537 >> v1:172.21.15.5:6789/0 conn(0x7f4eb00b1be0 legacy=0x7f4eb0060ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:25.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.042+0000 7f4eb5a44700 1 -- 172.21.15.187:0/143885537 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb01485e0 con 0x7f4eb00b1fc0 2020-11-20T12:17:25.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.043+0000 7f4eb5a44700 1 --2- 172.21.15.187:0/143885537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b1fc0 0x7f4eb00b5b50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f4ea000b9a0 tx=0x7f4ea0006df0).ready entity=mon.2 client_cookie=9173e1b1d7ffb432 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:25.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.043+0000 7f4eb5243700 1 -- 172.21.15.187:0/143885537 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0009710 con 0x7f4eb00b1fc0 2020-11-20T12:17:25.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.043+0000 7f4eb5243700 1 -- 172.21.15.187:0/143885537 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea0009870 con 0x7f4eb00b1fc0 2020-11-20T12:17:25.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.043+0000 7f4eb5243700 1 -- 172.21.15.187:0/143885537 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0009bf0 con 0x7f4eb00b1fc0 2020-11-20T12:17:25.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.044+0000 7f4eb7247700 1 -- 172.21.15.187:0/143885537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b1fc0 msgr2=0x7f4eb00b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:25.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.044+0000 7f4eb7247700 1 --2- 172.21.15.187:0/143885537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b1fc0 0x7f4eb00b5b50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f4ea000b9a0 tx=0x7f4ea0006df0).stop 2020-11-20T12:17:25.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.044+0000 7f4eb7247700 1 -- 172.21.15.187:0/143885537 shutdown_connections 2020-11-20T12:17:25.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.044+0000 7f4eb7247700 1 --2- 172.21.15.187:0/143885537 >> v2:172.21.15.5:3300/0 conn(0x7f4eb00b76a0 0x7f4eb0147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:25.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.044+0000 7f4eb7247700 1 --2- 172.21.15.187:0/143885537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b1fc0 0x7f4eb00b5b50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:25.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.044+0000 7f4eb7247700 1 -- 172.21.15.187:0/143885537 >> 172.21.15.187:0/143885537 conn(0x7f4eb00b7ff0 msgr2=0x7f4eb00b8400 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:25.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.044+0000 7f4eb7247700 1 -- 172.21.15.187:0/143885537 shutdown_connections 2020-11-20T12:17:25.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.044+0000 7f4eb7247700 1 -- 172.21.15.187:0/143885537 wait complete. 2020-11-20T12:17:25.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.047+0000 7f4eb7247700 1 Processor -- start 2020-11-20T12:17:25.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.048+0000 7f4eb7247700 1 -- start start 2020-11-20T12:17:25.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.049+0000 7f4eb7247700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eb00b1be0 0x7f4eb0146620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:25.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb7247700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4eb00b1fc0 0x7f4eb0158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:25.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb7247700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b76a0 0x7f4eb0141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:25.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb7247700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb014b780 con 0x7f4eb00b1fc0 2020-11-20T12:17:25.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb7247700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00b50c0 con 0x7f4eb00b1be0 2020-11-20T12:17:25.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb7247700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00b5360 con 0x7f4eb00b76a0 2020-11-20T12:17:25.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb6245700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eb00b1be0 0x7f4eb0146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:25.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb6245700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eb00b1be0 0x7f4eb0146620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51094/0 (socket says 172.21.15.187:51094) 2020-11-20T12:17:25.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb6245700 1 -- 172.21.15.187:0/2121793642 learned_addr learned my addr 172.21.15.187:0/2121793642 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:25.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb5a44700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4eb00b1fc0 0x7f4eb0158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:25.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb6245700 1 -- 172.21.15.187:0/2121793642 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b76a0 msgr2=0x7f4eb0141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:25.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb6245700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b76a0 0x7f4eb0141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:25.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb6245700 1 -- 172.21.15.187:0/2121793642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4eb00b1fc0 msgr2=0x7f4eb0158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:25.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb6245700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4eb00b1fc0 0x7f4eb0158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:25.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.050+0000 7f4eb6245700 1 -- 172.21.15.187:0/2121793642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb01582a0 con 0x7f4eb00b1be0 2020-11-20T12:17:25.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.051+0000 7f4eb6245700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eb00b1be0 0x7f4eb0146620 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 rx=0x7f4eac00c6a0 tx=0x7f4eac007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:25.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.051+0000 7f4e9e7fc700 1 -- 172.21.15.187:0/2121793642 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac0072e0 con 0x7f4eb00b1be0 2020-11-20T12:17:25.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.051+0000 7f4e9e7fc700 1 -- 172.21.15.187:0/2121793642 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eac007440 con 0x7f4eb00b1be0 2020-11-20T12:17:25.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.051+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb0146b10 con 0x7f4eb00b1be0 2020-11-20T12:17:25.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.051+0000 7f4e9e7fc700 1 -- 172.21.15.187:0/2121793642 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac005820 con 0x7f4eb00b1be0 2020-11-20T12:17:25.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.051+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb00b3240 con 0x7f4eb00b1be0 2020-11-20T12:17:25.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.052+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb00631a0 con 0x7f4eb00b1be0 2020-11-20T12:17:25.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.053+0000 7f4e9e7fc700 1 -- 172.21.15.187:0/2121793642 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4eac010070 con 0x7f4eb00b1be0 2020-11-20T12:17:25.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.053+0000 7f4e9e7fc700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e88056d20 0x7f4e880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:25.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.054+0000 7f4e9e7fc700 1 -- 172.21.15.187:0/2121793642 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4eac06f6f0 con 0x7f4eb00b1be0 2020-11-20T12:17:25.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.054+0000 7f4eb5a44700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e88056d20 0x7f4e880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:25.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.055+0000 7f4eb5a44700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e88056d20 0x7f4e880591e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f4eb014ba60 tx=0x7f4ea0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:25.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.057+0000 7f4e9e7fc700 1 -- 172.21.15.187:0/2121793642 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4eac044af0 con 0x7f4eb00b1be0 2020-11-20T12:17:25.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.297+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eb0158430 con 0x7f4eb00b1be0 2020-11-20T12:17:25.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.297+0000 7f4e9e7fc700 1 -- 172.21.15.187:0/2121793642 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f4eac048110 con 0x7f4eb00b1be0 2020-11-20T12:17:25.303 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:25.303 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:25.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e88056d20 msgr2=0x7f4e880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:25.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e88056d20 0x7f4e880591e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f4eb014ba60 tx=0x7f4ea0015000).stop 2020-11-20T12:17:25.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eb00b1be0 msgr2=0x7f4eb0146620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:25.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eb00b1be0 0x7f4eb0146620 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 rx=0x7f4eac00c6a0 tx=0x7f4eac007ac0).stop 2020-11-20T12:17:25.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 shutdown_connections 2020-11-20T12:17:25.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4e88056d20 0x7f4e880591e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:25.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4eb00b76a0 0x7f4eb0141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eb00b1be0 0x7f4eb0146620 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 --2- 172.21.15.187:0/2121793642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4eb00b1fc0 0x7f4eb0158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 >> 172.21.15.187:0/2121793642 conn(0x7f4eb00b7ff0 msgr2=0x7f4eb0066000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 shutdown_connections 2020-11-20T12:17:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:25.300+0000 7f4eb7247700 1 -- 172.21.15.187:0/2121793642 wait complete. 2020-11-20T12:17:25.310 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:25.327 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:25.619 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":882,"msgr_send_messages":889,"msgr_recv_bytes":287389,"msgr_send_bytes":8966937,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.281680558,"msgr_running_send_time":0.114855490,"msgr_running_recv_time":0.059740920,"msgr_running_fast_dispatch_time":0.082139072,"msgr_send_messages_queue_lat":{"avgcount":887,"sum":0.108673719,"avgtime":0.000122518},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7490,"msgr_send_messages":11179,"msgr_recv_bytes":2149292,"msgr_send_bytes":9227656,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":2.656919792,"msgr_running_send_time":1.995058135,"msgr_running_recv_time":0.782258542,"msgr_running_fast_dispatch_time":0.057641905,"msgr_send_messages_queue_lat":{"avgcount":11178,"sum":1.490289049,"avgtime":0.000133323},"msgr_handle_ack_lat":{"avgcount":8024,"sum":0.006091704,"avgtime":0.000000759}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":76,"msgr_send_messages":82,"msgr_recv_bytes":100208,"msgr_send_bytes":298657,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.027125934,"msgr_running_send_time":0.009516311,"msgr_running_recv_time":0.018604645,"msgr_running_fast_dispatch_time":0.004414162,"msgr_send_messages_queue_lat":{"avgcount":82,"sum":0.018176870,"avgtime":0.000221669},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3073,"sum":3.000299330,"avgtime":0.000976342}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":39,"sum":0.105460601,"avgtime":0.002704117}},"mds":{"request":2686,"reply":2685,"reply_latency":{"avgcount":2685,"sum":4.170740558,"avgtime":0.001553348},"forward":0,"dir_fetch":12,"dir_commit":73,"dir_split":9,"dir_merge":0,"inodes":1611,"inodes_top":0,"inodes_bottom":1611,"inodes_pin_tail":0,"inodes_pinned":1611,"inodes_expired":0,"inodes_with_caps":973,"caps":973,"subtrees":2,"traverse":2691,"traverse_hit":2688,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":250689,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":875,"root_rbytes":5907677184,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":36,"ceph_cap_op_grant":3959,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3940,"handle_client_caps_dirty":1290,"handle_client_cap_release":4,"process_request_cap_release":1085},"mds_cache":{"num_strays":390,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1019,"strays_enqueued":629,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":7022,"evex":0,"evtrm":0,"ev":7026,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":10,"segexg":0,"segexd":0,"expos":4194304,"wrpos":18079449,"rdpos":4195632,"jlat":{"avgcount":1534,"sum":18.208044290,"avgtime":0.011869650},"replayed":4},"mds_mem":{"ino":1614,"ino+":1614,"ino-":0,"dir":76,"dir+":85,"dir-":9,"dn":2629,"dn+":2629,"dn-":0,"cap":973,"cap+":1602,"cap-":629,"rss":80296,"heap":356572},"mds_server":{"dispatch_client_request":2725,"dispatch_server_request":0,"handle_client_request":2686,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1018,"sum":1.393005650,"avgtime":0.001368374},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2198,"average_load":2198,"avg_session_uptime":1226},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18057476,"buffer_anon_items":6811,"buffer_meta_bytes":1760,"buffer_meta_items":20,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":7937059,"mds_co_items":98603,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":14,"op_laggy":0,"op_send":1726,"op_send_bytes":13511771,"op_resend":0,"op_reply":1712,"oplen_avg":{"avgcount":1726,"sum":1848},"op":1726,"op_r":29,"op_w":1697,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":11,"osdop_write":1537,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":68,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":170,"omap_rd":28,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":18,"pq_executing_ops_high_water":18,"pq_executing":8,"pq_executing_high_water":9,"pq_executed":15,"pq_item_in_journal":616},"throttle-msgr_dispatch_throttler-mds":{"val":175,"max":104857600,"get_started":0,"get":8444,"get_sum":1880907,"get_or_fail_fail":0,"get_or_fail_success":8444,"take":0,"take_sum":0,"put":8443,"put_sum":1880732,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1726,"take_sum":13577186,"put":1645,"put_sum":13577186,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":14,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1726,"take_sum":1726,"put":1712,"put_sum":1712,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":629,"get_sum":63529,"get_or_fail_fail":0,"get_or_fail_success":629,"take":0,"take_sum":0,"put":3,"put_sum":63529,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":1302233,"max":3758096384,"get_started":0,"get":7022,"get_sum":13883035,"get_or_fail_fail":0,"get_or_fail_success":7022,"take":0,"take_sum":0,"put":1534,"put_sum":12580802,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:25.621 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 287389, "msgr_recv_messages": 882, "msgr_running_fast_dispatch_time": 0.082139072, "msgr_running_recv_time": 0.05974092, "msgr_running_send_time": 0.11485549, "msgr_running_total_time": 0.281680558, "msgr_send_bytes": 8966937, "msgr_send_messages": 889, "msgr_send_messages_queue_lat": { "avgcount": 887, "avgtime": 0.000122518, "sum": 0.108673719 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8024, "avgtime": 7.59e-07, "sum": 0.006091704 }, "msgr_recv_bytes": 2149292, "msgr_recv_messages": 7490, "msgr_running_fast_dispatch_time": 0.057641905, "msgr_running_recv_time": 0.782258542, "msgr_running_send_time": 1.995058135, "msgr_running_total_time": 2.656919792, "msgr_send_bytes": 9227656, "msgr_send_messages": 11179, "msgr_send_messages_queue_lat": { "avgcount": 11178, "avgtime": 0.000133323, "sum": 1.490289049 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 100208, "msgr_recv_messages": 76, "msgr_running_fast_dispatch_time": 0.004414162, "msgr_running_recv_time": 0.018604645, "msgr_running_send_time": 0.009516311, "msgr_running_total_time": 0.027125934, "msgr_send_bytes": 298657, "msgr_send_messages": 82, "msgr_send_messages_queue_lat": { "avgcount": 82, "avgtime": 0.000221669, "sum": 0.01817687 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3073, "avgtime": 0.000976342, "sum": 3.00029933 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 39, "avgtime": 0.002704117, "sum": 0.105460601 }, "queue_len": 0 }, "mds": { "caps": 973, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3959, "ceph_cap_op_revoke": 36, "ceph_cap_op_trunc": 0, "dir_commit": 73, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 4, "handle_client_caps": 3940, "handle_client_caps_dirty": 1290, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1611, "inodes_bottom": 1611, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 1611, "inodes_top": 0, "inodes_with_caps": 973, "load_cent": 250689, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1085, "q": 0, "reply": 2685, "reply_latency": { "avgcount": 2685, "avgtime": 0.001553348, "sum": 4.170740558 }, "request": 2686, "root_rbytes": 5907677184, "root_rfiles": 875, "root_rsnaps": 0, "subtrees": 2, "traverse": 2691, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2688, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 390, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1019, "strays_enqueued": 629, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7026, "evadd": 7022, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 1534, "avgtime": 0.01186965, "sum": 18.20804429 }, "rdpos": 4195632, "replayed": 4, "seg": 10, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 18079449 }, "mds_mem": { "cap": 973, "cap+": 1602, "cap-": 629, "dir": 76, "dir+": 85, "dir-": 9, "dn": 2629, "dn+": 2629, "dn-": 0, "heap": 356572, "ino": 1614, "ino+": 1614, "ino-": 0, "rss": 80296 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2725, "dispatch_server_request": 0, "handle_client_request": 2686, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1018, "avgtime": 0.001368374, "sum": 1.39300565 } }, "mds_sessions": { "average_load": 2198, "avg_session_uptime": 1226, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2198 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18057476, "buffer_anon_items": 6811, "buffer_meta_bytes": 1760, "buffer_meta_items": 20, "mds_co_bytes": 7937059, "mds_co_items": 98603, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 28, "omap_wr": 170, "op": 1726, "op_active": 14, "op_laggy": 0, "op_pg": 0, "op_r": 29, "op_reply": 1712, "op_resend": 0, "op_rmw": 0, "op_send": 1726, "op_send_bytes": 13511771, "op_w": 1697, "oplen_avg": { "avgcount": 1726, "sum": 1848 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 68, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 11, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1537, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 15, "pq_executing": 8, "pq_executing_high_water": 9, "pq_executing_ops": 18, "pq_executing_ops_high_water": 18, "pq_item_in_journal": 616 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8444, "get_or_fail_fail": 0, "get_or_fail_success": 8444, "get_started": 0, "get_sum": 1880907, "max": 104857600, "put": 8443, "put_sum": 1880732, "take": 0, "take_sum": 0, "val": 175, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1645, "put_sum": 13577186, "take": 1726, "take_sum": 13577186, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1712, "put_sum": 1712, "take": 1726, "take_sum": 1726, "val": 14, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 629, "get_or_fail_fail": 0, "get_or_fail_success": 629, "get_started": 0, "get_sum": 63529, "max": 3758096384, "put": 3, "put_sum": 63529, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 7022, "get_or_fail_fail": 0, "get_or_fail_success": 7022, "get_started": 0, "get_sum": 13883035, "max": 3758096384, "put": 1534, "put_sum": 12580802, "take": 0, "take_sum": 0, "val": 1302233, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:25.621 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 8/390, 15/1019 2020-11-20T12:17:26.622 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:26.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.921+0000 7f0de7fff700 1 Processor -- start 2020-11-20T12:17:26.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.921+0000 7f0de7fff700 1 -- start start 2020-11-20T12:17:26.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0de8065ad0 0x7f0de8065eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:26.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de80b16a0 0x7f0de80afc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:26.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de7fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0de8142950 0x7f0de8142de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:26.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de80b0fa0 con 0x7f0de8065ad0 2020-11-20T12:17:26.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de7fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de8149ac0 con 0x7f0de8142950 2020-11-20T12:17:26.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de8149d60 con 0x7f0de80b16a0 2020-11-20T12:17:26.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0de8142950 0x7f0de8142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:26.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0de8142950 0x7f0de8142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51110/0 (socket says 172.21.15.187:51110) 2020-11-20T12:17:26.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.922+0000 7f0de77fe700 1 -- 172.21.15.187:0/1795964829 learned_addr learned my addr 172.21.15.187:0/1795964829 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:26.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.923+0000 7f0de6ffd700 1 --2- 172.21.15.187:0/1795964829 >> v2:172.21.15.5:3300/0 conn(0x7f0de8065ad0 0x7f0de8065eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:26.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.923+0000 7f0de67fc700 1 --2- 172.21.15.187:0/1795964829 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de80b16a0 0x7f0de80afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:26.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.923+0000 7f0de77fe700 1 -- 172.21.15.187:0/1795964829 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de80b16a0 msgr2=0x7f0de80afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:26.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.923+0000 7f0de77fe700 1 --2- 172.21.15.187:0/1795964829 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de80b16a0 0x7f0de80afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:26.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.923+0000 7f0de77fe700 1 -- 172.21.15.187:0/1795964829 >> v2:172.21.15.5:3300/0 conn(0x7f0de8065ad0 msgr2=0x7f0de8065eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:26.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.923+0000 7f0de77fe700 1 --2- 172.21.15.187:0/1795964829 >> v2:172.21.15.5:3300/0 conn(0x7f0de8065ad0 0x7f0de8065eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:26.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.923+0000 7f0de77fe700 1 -- 172.21.15.187:0/1795964829 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de81459c0 con 0x7f0de8142950 2020-11-20T12:17:26.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.923+0000 7f0de77fe700 1 --2- 172.21.15.187:0/1795964829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8142950 0x7f0de8142de0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 rx=0x7f0de80b58c0 tx=0x7f0de0009290).ready entity=mon.1 client_cookie=586f3c7bfc1c5127 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:26.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.924+0000 7f0de5ffb700 1 -- 172.21.15.187:0/1795964829 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de000bba0 con 0x7f0de8142950 2020-11-20T12:17:26.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.924+0000 7f0de5ffb700 1 -- 172.21.15.187:0/1795964829 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de000bd00 con 0x7f0de8142950 2020-11-20T12:17:26.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.924+0000 7f0de5ffb700 1 -- 172.21.15.187:0/1795964829 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de0011610 con 0x7f0de8142950 2020-11-20T12:17:26.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.924+0000 7f0de7fff700 1 -- 172.21.15.187:0/1795964829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8142950 msgr2=0x7f0de8142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:26.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.924+0000 7f0de7fff700 1 --2- 172.21.15.187:0/1795964829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8142950 0x7f0de8142de0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 rx=0x7f0de80b58c0 tx=0x7f0de0009290).stop 2020-11-20T12:17:26.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.925+0000 7f0de7fff700 1 -- 172.21.15.187:0/1795964829 shutdown_connections 2020-11-20T12:17:26.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.925+0000 7f0de7fff700 1 --2- 172.21.15.187:0/1795964829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8142950 0x7f0de8142de0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:26.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.925+0000 7f0de7fff700 1 --2- 172.21.15.187:0/1795964829 >> v2:172.21.15.5:3300/0 conn(0x7f0de8065ad0 0x7f0de8065eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:26.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.925+0000 7f0de7fff700 1 --2- 172.21.15.187:0/1795964829 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de80b16a0 0x7f0de80afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:26.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.925+0000 7f0de7fff700 1 -- 172.21.15.187:0/1795964829 >> 172.21.15.187:0/1795964829 conn(0x7f0de80b8280 msgr2=0x7f0de80653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:26.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.925+0000 7f0de7fff700 1 -- 172.21.15.187:0/1795964829 shutdown_connections 2020-11-20T12:17:26.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.925+0000 7f0de7fff700 1 -- 172.21.15.187:0/1795964829 wait complete. 2020-11-20T12:17:26.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.926+0000 7f0de7fff700 1 Processor -- start 2020-11-20T12:17:26.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.927+0000 7f0de7fff700 1 -- start start 2020-11-20T12:17:26.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.927+0000 7f0de7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8065ad0 0x7f0de80b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:26.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.927+0000 7f0de7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0de80b16a0 0x7f0de80b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:26.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.927+0000 7f0de7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de8142950 0x7f0de8062590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:26.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.927+0000 7f0de7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80b4590 con 0x7f0de80b16a0 2020-11-20T12:17:26.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.927+0000 7f0de7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80b29d0 con 0x7f0de8065ad0 2020-11-20T12:17:26.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.927+0000 7f0de7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80b35c0 con 0x7f0de8142950 2020-11-20T12:17:26.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.928+0000 7f0de6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8065ad0 0x7f0de80b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:26.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.928+0000 7f0de6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8065ad0 0x7f0de80b40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51112/0 (socket says 172.21.15.187:51112) 2020-11-20T12:17:26.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.928+0000 7f0de6ffd700 1 -- 172.21.15.187:0/2098107789 learned_addr learned my addr 172.21.15.187:0/2098107789 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:26.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.928+0000 7f0de77fe700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de8142950 0x7f0de8062590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:26.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.928+0000 7f0de67fc700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0de80b16a0 0x7f0de80b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:26.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.928+0000 7f0de6ffd700 1 -- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de8142950 msgr2=0x7f0de8062590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:26.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.928+0000 7f0de6ffd700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de8142950 0x7f0de8062590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:26.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0de6ffd700 1 -- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0de80b16a0 msgr2=0x7f0de80b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:26.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0de6ffd700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0de80b16a0 0x7f0de80b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:26.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0de6ffd700 1 -- 172.21.15.187:0/2098107789 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de80b2c30 con 0x7f0de8065ad0 2020-11-20T12:17:26.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0de67fc700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0de80b16a0 0x7f0de80b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:17:26.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0de6ffd700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8065ad0 0x7f0de80b40a0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 rx=0x7f0dd800ec00 tx=0x7f0dd8006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:26.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0dceffd700 1 -- 172.21.15.187:0/2098107789 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800c9b0 con 0x7f0de8065ad0 2020-11-20T12:17:26.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0dceffd700 1 -- 172.21.15.187:0/2098107789 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd800cb10 con 0x7f0de8065ad0 2020-11-20T12:17:26.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0dceffd700 1 -- 172.21.15.187:0/2098107789 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800ce90 con 0x7f0de8065ad0 2020-11-20T12:17:26.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de80b4870 con 0x7f0de8065ad0 2020-11-20T12:17:26.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.929+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de80b3a00 con 0x7f0de8065ad0 2020-11-20T12:17:26.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.931+0000 7f0dceffd700 1 -- 172.21.15.187:0/2098107789 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0dd8012600 con 0x7f0de8065ad0 2020-11-20T12:17:26.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.931+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de8044f60 con 0x7f0de8065ad0 2020-11-20T12:17:26.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.932+0000 7f0dceffd700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0dc4056d50 0x7f0dc4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:26.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.932+0000 7f0dceffd700 1 -- 172.21.15.187:0/2098107789 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0dd8070390 con 0x7f0de8065ad0 2020-11-20T12:17:26.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.932+0000 7f0de67fc700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0dc4056d50 0x7f0dc4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:26.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.933+0000 7f0de67fc700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0dc4056d50 0x7f0dc4059210 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f0de80ba460 tx=0x7f0ddc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:26.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:26.938+0000 7f0dceffd700 1 -- 172.21.15.187:0/2098107789 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0dd8045730 con 0x7f0de8065ad0 2020-11-20T12:17:27.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.220+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0de8061250 con 0x7f0de8065ad0 2020-11-20T12:17:27.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.221+0000 7f0dceffd700 1 -- 172.21.15.187:0/2098107789 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f0dd8048d50 con 0x7f0de8065ad0 2020-11-20T12:17:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:27.225 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:27.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0dc4056d50 msgr2=0x7f0dc4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:27.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0dc4056d50 0x7f0dc4059210 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f0de80ba460 tx=0x7f0ddc008040).stop 2020-11-20T12:17:27.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8065ad0 msgr2=0x7f0de80b40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:27.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8065ad0 0x7f0de80b40a0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 rx=0x7f0dd800ec00 tx=0x7f0dd8006df0).stop 2020-11-20T12:17:27.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 shutdown_connections 2020-11-20T12:17:27.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0dc4056d50 0x7f0dc4059210 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:27.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0de8142950 0x7f0de8062590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:27.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0de8065ad0 0x7f0de80b40a0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:27.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 --2- 172.21.15.187:0/2098107789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0de80b16a0 0x7f0de80b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 >> 172.21.15.187:0/2098107789 conn(0x7f0de80b8280 msgr2=0x7f0de80666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 shutdown_connections 2020-11-20T12:17:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:27.223+0000 7f0de7fff700 1 -- 172.21.15.187:0/2098107789 wait complete. 2020-11-20T12:17:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:27.248 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:27.473 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1534,"msgr_send_messages":1536,"msgr_recv_bytes":452997,"msgr_send_bytes":9173330,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.334029772,"msgr_running_send_time":0.134691456,"msgr_running_recv_time":0.074984723,"msgr_running_fast_dispatch_time":0.095834470,"msgr_send_messages_queue_lat":{"avgcount":1534,"sum":0.191451637,"avgtime":0.000124805},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8663,"msgr_send_messages":14007,"msgr_recv_bytes":2456912,"msgr_send_bytes":13795653,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":2.946033653,"msgr_running_send_time":2.212387926,"msgr_running_recv_time":0.916479869,"msgr_running_fast_dispatch_time":0.071775956,"msgr_send_messages_queue_lat":{"avgcount":14006,"sum":1.893808232,"avgtime":0.000135214},"msgr_handle_ack_lat":{"avgcount":8608,"sum":0.006273384,"avgtime":0.000000728}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":453,"msgr_send_messages":450,"msgr_recv_bytes":195966,"msgr_send_bytes":416049,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.061974666,"msgr_running_send_time":0.021349353,"msgr_running_recv_time":0.027483587,"msgr_running_fast_dispatch_time":0.013078424,"msgr_send_messages_queue_lat":{"avgcount":450,"sum":0.099650190,"avgtime":0.000221444},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3078,"sum":3.531309690,"avgtime":0.001147274}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1541,"sum":0.160478562,"avgtime":0.000104139}},"mds":{"request":3268,"reply":3267,"reply_latency":{"avgcount":3267,"sum":5.027767722,"avgtime":0.001538955},"forward":0,"dir_fetch":12,"dir_commit":73,"dir_split":9,"dir_merge":0,"inodes":982,"inodes_top":0,"inodes_bottom":982,"inodes_pin_tail":0,"inodes_pinned":982,"inodes_expired":0,"inodes_with_caps":911,"caps":911,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":250689,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":477,"root_rbytes":3188719616,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":43,"ceph_cap_op_grant":4805,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4150,"handle_client_caps_dirty":1500,"handle_client_cap_release":5,"process_request_cap_release":1668},"mds_cache":{"num_strays":971,"num_strays_delayed":62,"num_strays_enqueuing":0,"strays_created":1600,"strays_enqueued":629,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":8271,"evex":0,"evtrm":0,"ev":8275,"evexg":0,"evexd":0,"segadd":10,"segex":0,"segtrm":0,"seg":11,"segexg":0,"segexd":0,"expos":4194304,"wrpos":20891592,"rdpos":4195632,"jlat":{"avgcount":1536,"sum":18.273538842,"avgtime":0.011896835},"replayed":4},"mds_mem":{"ino":985,"ino+":1614,"ino-":629,"dir":76,"dir+":85,"dir-":9,"dn":2583,"dn+":3212,"dn-":629,"cap":911,"cap+":1602,"cap-":691,"rss":87952,"heap":356572},"mds_server":{"dispatch_client_request":3308,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2748,"average_load":2748,"avg_session_uptime":1226},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18057476,"buffer_anon_items":6811,"buffer_meta_bytes":1760,"buffer_meta_items":20,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":7937059,"mds_co_items":98603,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3119,"op_send_bytes":17626327,"op_resend":0,"op_reply":3119,"oplen_avg":{"avgcount":3119,"sum":3241},"op":3119,"op_r":29,"op_w":3090,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":11,"osdop_write":1539,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":1457,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":170,"omap_rd":28,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":629,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":10646,"get_sum":2264433,"get_or_fail_fail":0,"get_or_fail_success":10646,"take":0,"take_sum":0,"put":10646,"put_sum":2264433,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3119,"take_sum":17691742,"put":1649,"put_sum":17691742,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3119,"take_sum":3119,"put":3119,"put_sum":3119,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":629,"get_sum":63529,"get_or_fail_fail":0,"get_or_fail_success":629,"take":0,"take_sum":0,"put":3,"put_sum":63529,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":0,"max":3758096384,"get_started":0,"get":8271,"get_sum":16695178,"get_or_fail_fail":0,"get_or_fail_success":8271,"take":0,"take_sum":0,"put":1536,"put_sum":16695178,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:27.476 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 452997, "msgr_recv_messages": 1534, "msgr_running_fast_dispatch_time": 0.09583447, "msgr_running_recv_time": 0.074984723, "msgr_running_send_time": 0.134691456, "msgr_running_total_time": 0.334029772, "msgr_send_bytes": 9173330, "msgr_send_messages": 1536, "msgr_send_messages_queue_lat": { "avgcount": 1534, "avgtime": 0.000124805, "sum": 0.191451637 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8608, "avgtime": 7.28e-07, "sum": 0.006273384 }, "msgr_recv_bytes": 2456912, "msgr_recv_messages": 8663, "msgr_running_fast_dispatch_time": 0.071775956, "msgr_running_recv_time": 0.916479869, "msgr_running_send_time": 2.212387926, "msgr_running_total_time": 2.946033653, "msgr_send_bytes": 13795653, "msgr_send_messages": 14007, "msgr_send_messages_queue_lat": { "avgcount": 14006, "avgtime": 0.000135214, "sum": 1.893808232 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 195966, "msgr_recv_messages": 453, "msgr_running_fast_dispatch_time": 0.013078424, "msgr_running_recv_time": 0.027483587, "msgr_running_send_time": 0.021349353, "msgr_running_total_time": 0.061974666, "msgr_send_bytes": 416049, "msgr_send_messages": 450, "msgr_send_messages_queue_lat": { "avgcount": 450, "avgtime": 0.000221444, "sum": 0.09965019 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3078, "avgtime": 0.001147274, "sum": 3.53130969 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1541, "avgtime": 0.000104139, "sum": 0.160478562 }, "queue_len": 0 }, "mds": { "caps": 911, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4805, "ceph_cap_op_revoke": 43, "ceph_cap_op_trunc": 0, "dir_commit": 73, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 5, "handle_client_caps": 4150, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 982, "inodes_bottom": 982, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 982, "inodes_top": 0, "inodes_with_caps": 911, "load_cent": 250689, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3267, "reply_latency": { "avgcount": 3267, "avgtime": 0.001538955, "sum": 5.027767722 }, "request": 3268, "root_rbytes": 3188719616, "root_rfiles": 477, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 971, "num_strays_delayed": 62, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1600, "strays_enqueued": 629, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8275, "evadd": 8271, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 1536, "avgtime": 0.011896835, "sum": 18.273538842 }, "rdpos": 4195632, "replayed": 4, "seg": 11, "segadd": 10, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 20891592 }, "mds_mem": { "cap": 911, "cap+": 1602, "cap-": 691, "dir": 76, "dir+": 85, "dir-": 9, "dn": 2583, "dn+": 3212, "dn-": 629, "heap": 356572, "ino": 985, "ino+": 1614, "ino-": 629, "rss": 87952 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3308, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2748, "avg_session_uptime": 1226, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2748 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18057476, "buffer_anon_items": 6811, "buffer_meta_bytes": 1760, "buffer_meta_items": 20, "mds_co_bytes": 7937059, "mds_co_items": 98603, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 28, "omap_wr": 170, "op": 3119, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 29, "op_reply": 3119, "op_resend": 0, "op_rmw": 0, "op_send": 3119, "op_send_bytes": 17626327, "op_w": 3090, "oplen_avg": { "avgcount": 3119, "sum": 3241 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1457, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 11, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1539, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 629, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10646, "get_or_fail_fail": 0, "get_or_fail_success": 10646, "get_started": 0, "get_sum": 2264433, "max": 104857600, "put": 10646, "put_sum": 2264433, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1649, "put_sum": 17691742, "take": 3119, "take_sum": 17691742, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3119, "put_sum": 3119, "take": 3119, "take_sum": 3119, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 629, "get_or_fail_fail": 0, "get_or_fail_success": 629, "get_started": 0, "get_sum": 63529, "max": 3758096384, "put": 3, "put_sum": 63529, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 8271, "get_or_fail_fail": 0, "get_or_fail_success": 8271, "get_started": 0, "get_sum": 16695178, "max": 3758096384, "put": 1536, "put_sum": 16695178, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:27.476 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/971, 629/1600 2020-11-20T12:17:28.076 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:28.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.315+0000 7f08931ee700 1 Processor -- start 2020-11-20T12:17:28.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.315+0000 7f08931ee700 1 -- start start 2020-11-20T12:17:28.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.316+0000 7f08931ee700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f088c0bc460 0x7f088c14c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.316+0000 7f08931ee700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f088c0bc840 0x7f088c1485e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.316+0000 7f08931ee700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088c14cc10 con 0x7f088c0b96f0 2020-11-20T12:17:28.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.316+0000 7f08931ee700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c14c9f0 con 0x7f088c0bc840 2020-11-20T12:17:28.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.316+0000 7f08931ee700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c14ceb0 con 0x7f088c0bc460 2020-11-20T12:17:28.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08921ec700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f088c0bc460 0x7f088c14c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08929ed700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f088c0b96f0 0x7f088c0b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47730/0 (socket says 172.21.15.187:47730) 2020-11-20T12:17:28.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08929ed700 1 -- 172.21.15.187:0/3071405715 learned_addr learned my addr 172.21.15.187:0/3071405715 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:28.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08919eb700 1 --2- 172.21.15.187:0/3071405715 >> v2:172.21.15.5:3300/0 conn(0x7f088c0bc840 0x7f088c1485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08921ec700 1 -- 172.21.15.187:0/3071405715 >> v2:172.21.15.5:3300/0 conn(0x7f088c0bc840 msgr2=0x7f088c1485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:28.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08921ec700 1 --2- 172.21.15.187:0/3071405715 >> v2:172.21.15.5:3300/0 conn(0x7f088c0bc840 0x7f088c1485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08921ec700 1 -- 172.21.15.187:0/3071405715 >> v1:172.21.15.187:6789/0 conn(0x7f088c0b96f0 legacy=0x7f088c0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08921ec700 1 -- 172.21.15.187:0/3071405715 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c14d150 con 0x7f088c0bc460 2020-11-20T12:17:28.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.317+0000 7f08919eb700 1 --2- 172.21.15.187:0/3071405715 >> v2:172.21.15.5:3300/0 conn(0x7f088c0bc840 0x7f088c1485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:28.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.318+0000 7f08921ec700 1 --2- 172.21.15.187:0/3071405715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0bc460 0x7f088c14c500 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f088c069e30 tx=0x7f0888006df0).ready entity=mon.1 client_cookie=4c6c490800ad894d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:28.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.318+0000 7f08911ea700 1 -- 172.21.15.187:0/3071405715 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888009840 con 0x7f088c0bc460 2020-11-20T12:17:28.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.318+0000 7f08911ea700 1 -- 172.21.15.187:0/3071405715 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08880099a0 con 0x7f088c0bc460 2020-11-20T12:17:28.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.318+0000 7f08911ea700 1 -- 172.21.15.187:0/3071405715 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888009d20 con 0x7f088c0bc460 2020-11-20T12:17:28.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.319+0000 7f08931ee700 1 -- 172.21.15.187:0/3071405715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0bc460 msgr2=0x7f088c14c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.319+0000 7f08931ee700 1 --2- 172.21.15.187:0/3071405715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0bc460 0x7f088c14c500 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f088c069e30 tx=0x7f0888006df0).stop 2020-11-20T12:17:28.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.319+0000 7f08931ee700 1 -- 172.21.15.187:0/3071405715 shutdown_connections 2020-11-20T12:17:28.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.319+0000 7f08931ee700 1 --2- 172.21.15.187:0/3071405715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0bc460 0x7f088c14c500 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.319+0000 7f08931ee700 1 --2- 172.21.15.187:0/3071405715 >> v2:172.21.15.5:3300/0 conn(0x7f088c0bc840 0x7f088c1485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.319+0000 7f08931ee700 1 -- 172.21.15.187:0/3071405715 >> 172.21.15.187:0/3071405715 conn(0x7f088c0bfb90 msgr2=0x7f088c0bbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:28.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.319+0000 7f08931ee700 1 -- 172.21.15.187:0/3071405715 shutdown_connections 2020-11-20T12:17:28.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.319+0000 7f08931ee700 1 -- 172.21.15.187:0/3071405715 wait complete. 2020-11-20T12:17:28.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.321+0000 7f08931ee700 1 Processor -- start 2020-11-20T12:17:28.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.321+0000 7f08931ee700 1 -- start start 2020-11-20T12:17:28.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.322+0000 7f08931ee700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b96f0 0x7f088c1182e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.322+0000 7f08931ee700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c0a4480 0x7f088c0b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.322+0000 7f08931ee700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c0b7e00 0x7f088c11a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.322+0000 7f08931ee700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c1187d0 con 0x7f088c0a4480 2020-11-20T12:17:28.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.322+0000 7f08931ee700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c0b8fc0 con 0x7f088c0b96f0 2020-11-20T12:17:28.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.322+0000 7f08931ee700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c15c9b0 con 0x7f088c0b7e00 2020-11-20T12:17:28.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08921ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b96f0 0x7f088c1182e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08921ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b96f0 0x7f088c1182e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51126/0 (socket says 172.21.15.187:51126) 2020-11-20T12:17:28.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08921ec700 1 -- 172.21.15.187:0/1208137491 learned_addr learned my addr 172.21.15.187:0/1208137491 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:28.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08929ed700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c0b7e00 0x7f088c11a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08921ec700 1 -- 172.21.15.187:0/1208137491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c0b7e00 msgr2=0x7f088c11a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08921ec700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c0b7e00 0x7f088c11a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08921ec700 1 -- 172.21.15.187:0/1208137491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c0a4480 msgr2=0x7f088c0b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08921ec700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c0a4480 0x7f088c0b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:28.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.323+0000 7f08921ec700 1 -- 172.21.15.187:0/1208137491 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c0b8b20 con 0x7f088c0b96f0 2020-11-20T12:17:28.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.324+0000 7f08921ec700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b96f0 0x7f088c1182e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7f088800c010 tx=0x7f0888015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:28.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.324+0000 7f087a7fc700 1 -- 172.21.15.187:0/1208137491 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088801e460 con 0x7f088c0b96f0 2020-11-20T12:17:28.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.324+0000 7f087a7fc700 1 -- 172.21.15.187:0/1208137491 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088801e5c0 con 0x7f088c0b96f0 2020-11-20T12:17:28.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.324+0000 7f087a7fc700 1 -- 172.21.15.187:0/1208137491 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088801e940 con 0x7f088c0b96f0 2020-11-20T12:17:28.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.324+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088c15c6a0 con 0x7f088c0b96f0 2020-11-20T12:17:28.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.324+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088c0b85c0 con 0x7f088c0b96f0 2020-11-20T12:17:28.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.325+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088c0b8980 con 0x7f088c0b96f0 2020-11-20T12:17:28.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.326+0000 7f087a7fc700 1 -- 172.21.15.187:0/1208137491 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f088801a040 con 0x7f088c0b96f0 2020-11-20T12:17:28.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.327+0000 7f087a7fc700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0864056d20 0x7f08640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.327+0000 7f087a7fc700 1 -- 172.21.15.187:0/1208137491 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0888071aa0 con 0x7f088c0b96f0 2020-11-20T12:17:28.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.327+0000 7f08919eb700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0864056d20 0x7f08640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.328+0000 7f08919eb700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0864056d20 0x7f08640591e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f088c0a5170 tx=0x7f087c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:28.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.331+0000 7f087a7fc700 1 -- 172.21.15.187:0/1208137491 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f088c0b8980 con 0x7f088c0b96f0 2020-11-20T12:17:28.477 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:28.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.591+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f088c0b8980 con 0x7f088c0b96f0 2020-11-20T12:17:28.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.591+0000 7f087a7fc700 1 -- 172.21.15.187:0/1208137491 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f088c0b8980 con 0x7f088c0b96f0 2020-11-20T12:17:28.595 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:28.596 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:28.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.594+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0864056d20 msgr2=0x7f08640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.594+0000 7f08931ee700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0864056d20 0x7f08640591e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f088c0a5170 tx=0x7f087c008040).stop 2020-11-20T12:17:28.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.594+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b96f0 msgr2=0x7f088c1182e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.594+0000 7f08931ee700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b96f0 0x7f088c1182e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7f088800c010 tx=0x7f0888015000).stop 2020-11-20T12:17:28.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.595+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 shutdown_connections 2020-11-20T12:17:28.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.595+0000 7f08931ee700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0864056d20 0x7f08640591e0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.595+0000 7f08931ee700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c0b7e00 0x7f088c11a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.595+0000 7f08931ee700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b96f0 0x7f088c1182e0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.595+0000 7f08931ee700 1 --2- 172.21.15.187:0/1208137491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c0a4480 0x7f088c0b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:28.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.595+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 >> 172.21.15.187:0/1208137491 conn(0x7f088c0bfb90 msgr2=0x7f088c0b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:28.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.595+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 shutdown_connections 2020-11-20T12:17:28.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.595+0000 7f08931ee700 1 -- 172.21.15.187:0/1208137491 wait complete. 2020-11-20T12:17:28.601 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:28.626 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:17:28.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.755+0000 7fdea2e07700 1 Processor -- start 2020-11-20T12:17:28.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.755+0000 7fdea2e07700 1 -- start start 2020-11-20T12:17:28.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.757+0000 7fdea2e07700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fde9c0ba170 0x7fde9c1490d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.757+0000 7fdea2e07700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9c0b9a00 con 0x7fde9c1485e0 2020-11-20T12:17:28.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.757+0000 7fdea2e07700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9c14ca20 con 0x7fde9c14c4a0 2020-11-20T12:17:28.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.757+0000 7fdea2e07700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9c14ccc0 con 0x7fde9c0ba170 2020-11-20T12:17:28.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.757+0000 7fdea1604700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fde9c14c4a0 0x7fde9c1489c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47742/0 (socket says 172.21.15.187:47742) 2020-11-20T12:17:28.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.757+0000 7fdea1604700 1 -- 172.21.15.187:0/3374769325 learned_addr learned my addr 172.21.15.187:0/3374769325 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:28.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.757+0000 7fdea2606700 1 --2- 172.21.15.187:0/3374769325 >> v2:172.21.15.187:3300/0 conn(0x7fde9c0ba170 0x7fde9c1490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.759+0000 7fdea2606700 1 -- 172.21.15.187:0/3374769325 >> v1:172.21.15.187:6789/0 conn(0x7fde9c14c4a0 legacy=0x7fde9c1489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.759+0000 7fdea2606700 1 -- 172.21.15.187:0/3374769325 >> v1:172.21.15.5:6789/0 conn(0x7fde9c1485e0 legacy=0x7fde9c0bc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:28.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.759+0000 7fdea2606700 1 -- 172.21.15.187:0/3374769325 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c14cf60 con 0x7fde9c0ba170 2020-11-20T12:17:28.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.758+0000 7fdea0e03700 1 -- 172.21.15.187:0/3374769325 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552888954 0 0) 0x7fde9c14ca20 con 0x7fde9c14c4a0 2020-11-20T12:17:28.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.759+0000 7fdea2606700 1 --2- 172.21.15.187:0/3374769325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c0ba170 0x7fde9c1490d0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7fde940135e0 tx=0x7fde94011a70).ready entity=mon.1 client_cookie=bd883415fc091ff6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:28.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.760+0000 7fdea0e03700 1 -- 172.21.15.187:0/3374769325 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94016070 con 0x7fde9c0ba170 2020-11-20T12:17:28.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.760+0000 7fdea0e03700 1 -- 172.21.15.187:0/3374769325 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde940112e0 con 0x7fde9c0ba170 2020-11-20T12:17:28.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.760+0000 7fdea0e03700 1 -- 172.21.15.187:0/3374769325 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94011660 con 0x7fde9c0ba170 2020-11-20T12:17:28.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.761+0000 7fdea2e07700 1 -- 172.21.15.187:0/3374769325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c0ba170 msgr2=0x7fde9c1490d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.761+0000 7fdea2e07700 1 --2- 172.21.15.187:0/3374769325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c0ba170 0x7fde9c1490d0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7fde940135e0 tx=0x7fde94011a70).stop 2020-11-20T12:17:28.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.762+0000 7fdea2e07700 1 -- 172.21.15.187:0/3374769325 shutdown_connections 2020-11-20T12:17:28.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.762+0000 7fdea2e07700 1 --2- 172.21.15.187:0/3374769325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c0ba170 0x7fde9c1490d0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.762+0000 7fdea2e07700 1 -- 172.21.15.187:0/3374769325 >> 172.21.15.187:0/3374769325 conn(0x7fde9c0bfbb0 msgr2=0x7fde9c0bbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:28.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.763+0000 7fdea2e07700 1 -- 172.21.15.187:0/3374769325 shutdown_connections 2020-11-20T12:17:28.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.763+0000 7fdea2e07700 1 -- 172.21.15.187:0/3374769325 wait complete. 2020-11-20T12:17:28.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.765+0000 7fdea2e07700 1 Processor -- start 2020-11-20T12:17:28.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.765+0000 7fdea2e07700 1 -- start start 2020-11-20T12:17:28.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea2e07700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fde9c0ba170 0x7fde9c143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea2e07700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fde9c1485e0 0x7fde9c14a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea2e07700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c14c4a0 0x7fde9c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea2e07700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c157e00 con 0x7fde9c0ba170 2020-11-20T12:17:28.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea2e07700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c1580a0 con 0x7fde9c14c4a0 2020-11-20T12:17:28.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea2e07700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c158340 con 0x7fde9c1485e0 2020-11-20T12:17:28.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea1e05700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fde9c0ba170 0x7fde9c143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea1e05700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fde9c0ba170 0x7fde9c143a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:43404/0 (socket says 172.21.15.187:43404) 2020-11-20T12:17:28.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea1e05700 1 -- 172.21.15.187:0/3424133865 learned_addr learned my addr 172.21.15.187:0/3424133865 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:28.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea1604700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fde9c1485e0 0x7fde9c14a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.767+0000 7fdea2606700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c14c4a0 0x7fde9c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.768+0000 7fdea1e05700 1 -- 172.21.15.187:0/3424133865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fde9c1485e0 msgr2=0x7fde9c14a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.768+0000 7fdea1e05700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fde9c1485e0 0x7fde9c14a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.768+0000 7fdea1e05700 1 -- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c14c4a0 msgr2=0x7fde9c0a43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:28.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.768+0000 7fdea1e05700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c14c4a0 0x7fde9c0a43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:28.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.768+0000 7fdea1e05700 1 -- 172.21.15.187:0/3424133865 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c143f80 con 0x7fde9c0ba170 2020-11-20T12:17:28.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.769+0000 7fdea2606700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c14c4a0 0x7fde9c0a43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:17:28.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.769+0000 7fdea1e05700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fde9c0ba170 0x7fde9c143a90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fde980096e0 tx=0x7fde98006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:28.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.769+0000 7fde89ffb700 1 -- 172.21.15.187:0/3424133865 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9800d490 con 0x7fde9c0ba170 2020-11-20T12:17:28.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.769+0000 7fde89ffb700 1 -- 172.21.15.187:0/3424133865 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9800d610 con 0x7fde9c0ba170 2020-11-20T12:17:28.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.769+0000 7fde89ffb700 1 -- 172.21.15.187:0/3424133865 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9800d9d0 con 0x7fde9c0ba170 2020-11-20T12:17:28.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.769+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9c14a660 con 0x7fde9c0ba170 2020-11-20T12:17:28.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.770+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9c1445a0 con 0x7fde9c0ba170 2020-11-20T12:17:28.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.772+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9c068630 con 0x7fde9c0ba170 2020-11-20T12:17:28.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.772+0000 7fde89ffb700 1 -- 172.21.15.187:0/3424133865 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fde9800f070 con 0x7fde9c0ba170 2020-11-20T12:17:28.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.774+0000 7fde89ffb700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde74056d20 0x7fde740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:28.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.774+0000 7fde89ffb700 1 -- 172.21.15.187:0/3424133865 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fde9806f820 con 0x7fde9c0ba170 2020-11-20T12:17:28.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.774+0000 7fdea1604700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde74056d20 0x7fde740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:28.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.775+0000 7fdea1604700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde74056d20 0x7fde740591e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7fde9c1573b0 tx=0x7fde8c007c20).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:28.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:28.781+0000 7fde89ffb700 1 -- 172.21.15.187:0/3424133865 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fde98044b10 con 0x7fde9c0ba170 2020-11-20T12:17:29.113 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:17:29.113 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:17:29.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.161+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde9c080210 con 0x7fde9c0ba170 2020-11-20T12:17:29.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.163+0000 7fde89ffb700 1 -- 172.21.15.187:0/3424133865 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7fde98048130 con 0x7fde9c0ba170 2020-11-20T12:17:29.166 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:29.167 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:29.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.165+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde74056d20 msgr2=0x7fde740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:29.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.165+0000 7fdea2e07700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde74056d20 0x7fde740591e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7fde9c1573b0 tx=0x7fde8c007c20).stop 2020-11-20T12:17:29.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.165+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fde9c0ba170 msgr2=0x7fde9c143a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:29.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.165+0000 7fdea2e07700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fde9c0ba170 0x7fde9c143a90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fde980096e0 tx=0x7fde98006df0).stop 2020-11-20T12:17:29.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.166+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 shutdown_connections 2020-11-20T12:17:29.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.166+0000 7fdea2e07700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fde74056d20 0x7fde740591e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:29.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.166+0000 7fdea2e07700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fde9c0ba170 0x7fde9c143a90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:29.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.166+0000 7fdea2e07700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde9c14c4a0 0x7fde9c0a43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:29.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.166+0000 7fdea2e07700 1 --2- 172.21.15.187:0/3424133865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fde9c1485e0 0x7fde9c14a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:29.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.166+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 >> 172.21.15.187:0/3424133865 conn(0x7fde9c0bfbb0 msgr2=0x7fde9c0bd660 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:29.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.166+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 shutdown_connections 2020-11-20T12:17:29.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:29.166+0000 7fdea2e07700 1 -- 172.21.15.187:0/3424133865 wait complete. 2020-11-20T12:17:29.172 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:29.194 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:29.515 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2111,"msgr_send_messages":2118,"msgr_recv_bytes":683893,"msgr_send_bytes":9711225,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.399123107,"msgr_running_send_time":0.166502209,"msgr_running_recv_time":0.086505034,"msgr_running_fast_dispatch_time":0.112812143,"msgr_send_messages_queue_lat":{"avgcount":2116,"sum":0.418086630,"avgtime":0.000197583},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8994,"msgr_send_messages":14338,"msgr_recv_bytes":2562424,"msgr_send_bytes":14120284,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":2.987873099,"msgr_running_send_time":2.228275896,"msgr_running_recv_time":0.927820743,"msgr_running_fast_dispatch_time":0.085027782,"msgr_send_messages_queue_lat":{"avgcount":14337,"sum":1.913265517,"avgtime":0.000133449},"msgr_handle_ack_lat":{"avgcount":8614,"sum":0.006399116,"avgtime":0.000000742}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":749,"msgr_send_messages":749,"msgr_recv_bytes":276648,"msgr_send_bytes":562491,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.089831456,"msgr_running_send_time":0.031177528,"msgr_running_recv_time":0.034080849,"msgr_running_fast_dispatch_time":0.022148917,"msgr_send_messages_queue_lat":{"avgcount":749,"sum":0.134935324,"avgtime":0.000180153},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3116,"sum":3.693394642,"avgtime":0.001185299}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2592,"sum":0.302892033,"avgtime":0.000116856}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":759,"inodes_top":0,"inodes_bottom":759,"inodes_pin_tail":0,"inodes_pinned":759,"inodes_expired":0,"inodes_with_caps":227,"caps":227,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":250689,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4813,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4158,"handle_client_caps_dirty":1500,"handle_client_cap_release":10,"process_request_cap_release":1668},"mds_cache":{"num_strays":226,"num_strays_delayed":226,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1375,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9022,"evex":9025,"evtrm":9025,"ev":9026,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21350328,"rdpos":4195632,"jlat":{"avgcount":1541,"sum":18.353403841,"avgtime":0.011910060},"replayed":4},"mds_mem":{"ino":985,"ino+":1614,"ino-":629,"dir":76,"dir+":85,"dir-":9,"dn":2583,"dn+":3212,"dn-":629,"cap":227,"cap+":1602,"cap-":1375,"rss":94024,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1231},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8730025,"buffer_anon_items":3005,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":2722085,"mds_co_items":55174,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":11,"op_laggy":0,"op_send":4318,"op_send_bytes":18371750,"op_resend":0,"op_reply":4307,"oplen_avg":{"avgcount":4318,"sum":4925},"op":4318,"op_r":34,"op_w":4284,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":16,"osdop_write":1549,"osdop_writefull":13,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":2388,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":193,"omap_rd":28,"omap_del":14},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":16,"pq_executing_ops_high_water":22,"pq_executing":7,"pq_executing_high_water":11,"pq_executed":1024,"pq_item_in_journal":355},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":11850,"get_sum":2573351,"get_or_fail_fail":0,"get_or_fail_success":11850,"take":0,"take_sum":0,"put":11850,"put_sum":2573351,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4318,"take_sum":18512511,"put":1917,"put_sum":18512511,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":11,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4318,"take_sum":4318,"put":4307,"put_sum":4307,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1375,"get_sum":138875,"get_or_fail_fail":0,"get_or_fail_success":1375,"take":0,"take_sum":0,"put":8,"put_sum":138875,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":0,"max":3758096384,"get_started":0,"get":9022,"get_sum":17153914,"get_or_fail_fail":0,"get_or_fail_success":9022,"take":0,"take_sum":0,"put":1541,"put_sum":17153914,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:29.516 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 683893, "msgr_recv_messages": 2111, "msgr_running_fast_dispatch_time": 0.112812143, "msgr_running_recv_time": 0.086505034, "msgr_running_send_time": 0.166502209, "msgr_running_total_time": 0.399123107, "msgr_send_bytes": 9711225, "msgr_send_messages": 2118, "msgr_send_messages_queue_lat": { "avgcount": 2116, "avgtime": 0.000197583, "sum": 0.41808663 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8614, "avgtime": 7.42e-07, "sum": 0.006399116 }, "msgr_recv_bytes": 2562424, "msgr_recv_messages": 8994, "msgr_running_fast_dispatch_time": 0.085027782, "msgr_running_recv_time": 0.927820743, "msgr_running_send_time": 2.228275896, "msgr_running_total_time": 2.987873099, "msgr_send_bytes": 14120284, "msgr_send_messages": 14338, "msgr_send_messages_queue_lat": { "avgcount": 14337, "avgtime": 0.000133449, "sum": 1.913265517 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 276648, "msgr_recv_messages": 749, "msgr_running_fast_dispatch_time": 0.022148917, "msgr_running_recv_time": 0.034080849, "msgr_running_send_time": 0.031177528, "msgr_running_total_time": 0.089831456, "msgr_send_bytes": 562491, "msgr_send_messages": 749, "msgr_send_messages_queue_lat": { "avgcount": 749, "avgtime": 0.000180153, "sum": 0.134935324 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3116, "avgtime": 0.001185299, "sum": 3.693394642 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2592, "avgtime": 0.000116856, "sum": 0.302892033 }, "queue_len": 0 }, "mds": { "caps": 227, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4813, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 10, "handle_client_caps": 4158, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 759, "inodes_bottom": 759, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 759, "inodes_top": 0, "inodes_with_caps": 227, "load_cent": 250689, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 226, "num_strays_delayed": 226, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1375, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9026, "evadd": 9022, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1541, "avgtime": 0.01191006, "sum": 18.353403841 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21350328 }, "mds_mem": { "cap": 227, "cap+": 1602, "cap-": 1375, "dir": 76, "dir+": 85, "dir-": 9, "dn": 2583, "dn+": 3212, "dn-": 629, "heap": 356572, "ino": 985, "ino+": 1614, "ino-": 629, "rss": 94024 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1231, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8730025, "buffer_anon_items": 3005, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 2722085, "mds_co_items": 55174, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 14, "omap_rd": 28, "omap_wr": 193, "op": 4318, "op_active": 11, "op_laggy": 0, "op_pg": 0, "op_r": 34, "op_reply": 4307, "op_resend": 0, "op_rmw": 0, "op_send": 4318, "op_send_bytes": 18371750, "op_w": 4284, "oplen_avg": { "avgcount": 4318, "sum": 4925 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 2388, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 16, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1549, "osdop_writefull": 13, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1024, "pq_executing": 7, "pq_executing_high_water": 11, "pq_executing_ops": 16, "pq_executing_ops_high_water": 22, "pq_item_in_journal": 355 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11850, "get_or_fail_fail": 0, "get_or_fail_success": 11850, "get_started": 0, "get_sum": 2573351, "max": 104857600, "put": 11850, "put_sum": 2573351, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1917, "put_sum": 18512511, "take": 4318, "take_sum": 18512511, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4307, "put_sum": 4307, "take": 4318, "take_sum": 4318, "val": 11, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1375, "get_or_fail_fail": 0, "get_or_fail_success": 1375, "get_started": 0, "get_sum": 138875, "max": 3758096384, "put": 8, "put_sum": 138875, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9022, "get_or_fail_fail": 0, "get_or_fail_success": 9022, "get_started": 0, "get_sum": 17153914, "max": 3758096384, "put": 1541, "put_sum": 17153914, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:29.517 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 7/226, 1024/1601 2020-11-20T12:17:30.518 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:30.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.757+0000 7f62d8c0e700 1 Processor -- start 2020-11-20T12:17:30.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.758+0000 7f62d8c0e700 1 -- start start 2020-11-20T12:17:30.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.759+0000 7f62d8c0e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 0x7f62d4065eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:30.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.759+0000 7f62d8c0e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f62d40b16a0 0x7f62d40afc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:30.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.759+0000 7f62d8c0e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d40b0fa0 con 0x7f62d4142950 2020-11-20T12:17:30.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.759+0000 7f62d8c0e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d4149ac0 con 0x7f62d40b16a0 2020-11-20T12:17:30.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.759+0000 7f62d8c0e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d4149d60 con 0x7f62d4065ad0 2020-11-20T12:17:30.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.759+0000 7f62d2ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f62d40b16a0 0x7f62d40afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:30.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d2ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f62d40b16a0 0x7f62d40afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51152/0 (socket says 172.21.15.187:51152) 2020-11-20T12:17:30.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d2ffd700 1 -- 172.21.15.187:0/1031228876 learned_addr learned my addr 172.21.15.187:0/1031228876 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:30.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d37fe700 1 --2- 172.21.15.187:0/1031228876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 0x7f62d4065eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:30.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d2ffd700 1 -- 172.21.15.187:0/1031228876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 msgr2=0x7f62d4065eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:30.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d2ffd700 1 --2- 172.21.15.187:0/1031228876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 0x7f62d4065eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:30.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d2ffd700 1 -- 172.21.15.187:0/1031228876 >> v1:172.21.15.187:6789/0 conn(0x7f62d4142950 legacy=0x7f62d4145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:30.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d2ffd700 1 -- 172.21.15.187:0/1031228876 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d41459c0 con 0x7f62d40b16a0 2020-11-20T12:17:30.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d27fc700 1 -- 172.21.15.187:0/1031228876 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186522367 0 0) 0x7f62d40b0fa0 con 0x7f62d4142950 2020-11-20T12:17:30.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.760+0000 7f62d2ffd700 1 --2- 172.21.15.187:0/1031228876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d40afc70 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 rx=0x7f62c8006fd0 tx=0x7f62c8006eb0).ready entity=mon.1 client_cookie=5cf90068cb313f51 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:30.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d27fc700 1 -- 172.21.15.187:0/1031228876 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8009960 con 0x7f62d40b16a0 2020-11-20T12:17:30.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d27fc700 1 -- 172.21.15.187:0/1031228876 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c8009ac0 con 0x7f62d40b16a0 2020-11-20T12:17:30.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d27fc700 1 -- 172.21.15.187:0/1031228876 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8009e40 con 0x7f62d40b16a0 2020-11-20T12:17:30.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1031228876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 msgr2=0x7f62d40afc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:30.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1031228876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d40afc70 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 rx=0x7f62c8006fd0 tx=0x7f62c8006eb0).stop 2020-11-20T12:17:30.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1031228876 shutdown_connections 2020-11-20T12:17:30.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1031228876 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 0x7f62d4065eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:30.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1031228876 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d40afc70 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:30.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.761+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1031228876 >> 172.21.15.187:0/1031228876 conn(0x7f62d40b8280 msgr2=0x7f62d40653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:30.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.762+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1031228876 shutdown_connections 2020-11-20T12:17:30.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.762+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1031228876 wait complete. 2020-11-20T12:17:30.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.763+0000 7f62d8c0e700 1 Processor -- start 2020-11-20T12:17:30.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.764+0000 7f62d8c0e700 1 -- start start 2020-11-20T12:17:30.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d8c0e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 0x7f62d4159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:30.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d8c0e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d4140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:30.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d8c0e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62d4142950 0x7f62d4153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:30.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d8c0e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d4159dd0 con 0x7f62d4142950 2020-11-20T12:17:30.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d8c0e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d415a070 con 0x7f62d40b16a0 2020-11-20T12:17:30.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d8c0e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d415a310 con 0x7f62d4065ad0 2020-11-20T12:17:30.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d4140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:30.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d4140380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51156/0 (socket says 172.21.15.187:51156) 2020-11-20T12:17:30.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d2ffd700 1 -- 172.21.15.187:0/1182723927 learned_addr learned my addr 172.21.15.187:0/1182723927 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:30.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d37fe700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 0x7f62d4159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:30.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d3fff700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62d4142950 0x7f62d4153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:30.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.765+0000 7f62d2ffd700 1 -- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 msgr2=0x7f62d4159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:30.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62d2ffd700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 0x7f62d4159730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:30.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62d2ffd700 1 -- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62d4142950 msgr2=0x7f62d4153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:30.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62d2ffd700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62d4142950 0x7f62d4153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:30.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62d2ffd700 1 -- 172.21.15.187:0/1182723927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4140870 con 0x7f62d40b16a0 2020-11-20T12:17:30.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62d3fff700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62d4142950 0x7f62d4153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:30.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62d2ffd700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d4140380 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f62c8010000 tx=0x7f62c8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:30.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62b37fe700 1 -- 172.21.15.187:0/1182723927 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8011e50 con 0x7f62d40b16a0 2020-11-20T12:17:30.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62b37fe700 1 -- 172.21.15.187:0/1182723927 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c801e460 con 0x7f62d40b16a0 2020-11-20T12:17:30.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d40b4b30 con 0x7f62d40b16a0 2020-11-20T12:17:30.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62b37fe700 1 -- 172.21.15.187:0/1182723927 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c801e7e0 con 0x7f62d40b16a0 2020-11-20T12:17:30.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.766+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d40b5ab0 con 0x7f62d40b16a0 2020-11-20T12:17:30.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.767+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d4159c20 con 0x7f62d40b16a0 2020-11-20T12:17:30.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.768+0000 7f62b37fe700 1 -- 172.21.15.187:0/1182723927 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f62c8025030 con 0x7f62d40b16a0 2020-11-20T12:17:30.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.768+0000 7f62b37fe700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f62ac056d20 0x7f62ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:30.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.768+0000 7f62b37fe700 1 -- 172.21.15.187:0/1182723927 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f62c80718d0 con 0x7f62d40b16a0 2020-11-20T12:17:30.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.769+0000 7f62d37fe700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f62ac056d20 0x7f62ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:30.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.769+0000 7f62d37fe700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f62ac056d20 0x7f62ac0591e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f62d40b9570 tx=0x7f62c4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:30.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:30.772+0000 7f62b37fe700 1 -- 172.21.15.187:0/1182723927 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f62c8046d00 con 0x7f62d40b16a0 2020-11-20T12:17:31.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.005+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62d4068ef0 con 0x7f62d40b16a0 2020-11-20T12:17:31.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.006+0000 7f62b37fe700 1 -- 172.21.15.187:0/1182723927 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f62c8023070 con 0x7f62d40b16a0 2020-11-20T12:17:31.010 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:31.011 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:31.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f62ac056d20 msgr2=0x7f62ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:31.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f62ac056d20 0x7f62ac0591e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f62d40b9570 tx=0x7f62c4008040).stop 2020-11-20T12:17:31.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 msgr2=0x7f62d4140380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:31.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d4140380 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f62c8010000 tx=0x7f62c8015000).stop 2020-11-20T12:17:31.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 shutdown_connections 2020-11-20T12:17:31.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f62ac056d20 0x7f62ac0591e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:31.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62d4142950 0x7f62d4153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:31.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f62d4065ad0 0x7f62d4159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:31.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 --2- 172.21.15.187:0/1182723927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62d40b16a0 0x7f62d4140380 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:31.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 >> 172.21.15.187:0/1182723927 conn(0x7f62d40b8280 msgr2=0x7f62d409d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:31.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 shutdown_connections 2020-11-20T12:17:31.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:31.009+0000 7f62d8c0e700 1 -- 172.21.15.187:0/1182723927 wait complete. 2020-11-20T12:17:31.015 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:31.033 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:31.300 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2532,"msgr_send_messages":2534,"msgr_recv_bytes":790827,"msgr_send_bytes":9843929,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.434994590,"msgr_running_send_time":0.179458338,"msgr_running_recv_time":0.096439456,"msgr_running_fast_dispatch_time":0.123305668,"msgr_send_messages_queue_lat":{"avgcount":2532,"sum":0.453228646,"avgtime":0.000179000},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9205,"msgr_send_messages":14544,"msgr_recv_bytes":2615902,"msgr_send_bytes":14186080,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.008723292,"msgr_running_send_time":2.236253386,"msgr_running_recv_time":0.933416904,"msgr_running_fast_dispatch_time":0.090613868,"msgr_send_messages_queue_lat":{"avgcount":14543,"sum":1.923340657,"avgtime":0.000132251},"msgr_handle_ack_lat":{"avgcount":8614,"sum":0.006399116,"avgtime":0.000000742}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":941,"msgr_send_messages":938,"msgr_recv_bytes":325416,"msgr_send_bytes":622782,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.105485023,"msgr_running_send_time":0.036289069,"msgr_running_recv_time":0.038417529,"msgr_running_fast_dispatch_time":0.026992374,"msgr_send_messages_queue_lat":{"avgcount":938,"sum":0.144866692,"avgtime":0.000154442},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3116,"sum":3.693394642,"avgtime":0.001185299}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3565,"sum":0.334199743,"avgtime":0.000093744}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":759,"inodes_top":0,"inodes_bottom":759,"inodes_pin_tail":0,"inodes_pinned":759,"inodes_expired":0,"inodes_with_caps":227,"caps":227,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":250689,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4813,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4158,"handle_client_caps_dirty":1500,"handle_client_cap_release":10,"process_request_cap_release":1668},"mds_cache":{"num_strays":226,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1375,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9022,"evex":9025,"evtrm":9025,"ev":9026,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21350328,"rdpos":4195632,"jlat":{"avgcount":1541,"sum":18.353403841,"avgtime":0.011910060},"replayed":4},"mds_mem":{"ino":239,"ino+":1614,"ino-":1375,"dir":76,"dir+":85,"dir-":9,"dn":237,"dn+":3212,"dn-":2975,"cap":227,"cap+":1602,"cap-":1375,"rss":94024,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1231},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8730025,"buffer_anon_items":3005,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":2722085,"mds_co_items":55174,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5129,"op_send_bytes":18371840,"op_resend":0,"op_reply":5129,"oplen_avg":{"avgcount":5129,"sum":5736},"op":5129,"op_r":34,"op_w":5095,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":16,"osdop_write":1549,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3198,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":193,"omap_rd":28,"omap_del":14},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":22,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1375,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12674,"get_sum":2708403,"get_or_fail_fail":0,"get_or_fail_success":12674,"take":0,"take_sum":0,"put":12674,"put_sum":2708403,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5129,"take_sum":18512601,"put":1918,"put_sum":18512601,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5129,"take_sum":5129,"put":5129,"put_sum":5129,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1375,"get_sum":138875,"get_or_fail_fail":0,"get_or_fail_success":1375,"take":0,"take_sum":0,"put":8,"put_sum":138875,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":0,"max":3758096384,"get_started":0,"get":9022,"get_sum":17153914,"get_or_fail_fail":0,"get_or_fail_success":9022,"take":0,"take_sum":0,"put":1541,"put_sum":17153914,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:31.302 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 790827, "msgr_recv_messages": 2532, "msgr_running_fast_dispatch_time": 0.123305668, "msgr_running_recv_time": 0.096439456, "msgr_running_send_time": 0.179458338, "msgr_running_total_time": 0.43499459, "msgr_send_bytes": 9843929, "msgr_send_messages": 2534, "msgr_send_messages_queue_lat": { "avgcount": 2532, "avgtime": 0.000179, "sum": 0.453228646 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8614, "avgtime": 7.42e-07, "sum": 0.006399116 }, "msgr_recv_bytes": 2615902, "msgr_recv_messages": 9205, "msgr_running_fast_dispatch_time": 0.090613868, "msgr_running_recv_time": 0.933416904, "msgr_running_send_time": 2.236253386, "msgr_running_total_time": 3.008723292, "msgr_send_bytes": 14186080, "msgr_send_messages": 14544, "msgr_send_messages_queue_lat": { "avgcount": 14543, "avgtime": 0.000132251, "sum": 1.923340657 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 325416, "msgr_recv_messages": 941, "msgr_running_fast_dispatch_time": 0.026992374, "msgr_running_recv_time": 0.038417529, "msgr_running_send_time": 0.036289069, "msgr_running_total_time": 0.105485023, "msgr_send_bytes": 622782, "msgr_send_messages": 938, "msgr_send_messages_queue_lat": { "avgcount": 938, "avgtime": 0.000154442, "sum": 0.144866692 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3116, "avgtime": 0.001185299, "sum": 3.693394642 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3565, "avgtime": 9.3744e-05, "sum": 0.334199743 }, "queue_len": 0 }, "mds": { "caps": 227, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4813, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 10, "handle_client_caps": 4158, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 759, "inodes_bottom": 759, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 759, "inodes_top": 0, "inodes_with_caps": 227, "load_cent": 250689, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 226, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1375, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9026, "evadd": 9022, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1541, "avgtime": 0.01191006, "sum": 18.353403841 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21350328 }, "mds_mem": { "cap": 227, "cap+": 1602, "cap-": 1375, "dir": 76, "dir+": 85, "dir-": 9, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 239, "ino+": 1614, "ino-": 1375, "rss": 94024 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1231, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8730025, "buffer_anon_items": 3005, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 2722085, "mds_co_items": 55174, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 14, "omap_rd": 28, "omap_wr": 193, "op": 5129, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 34, "op_reply": 5129, "op_resend": 0, "op_rmw": 0, "op_send": 5129, "op_send_bytes": 18371840, "op_w": 5095, "oplen_avg": { "avgcount": 5129, "sum": 5736 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3198, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 16, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1549, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1375, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 22, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12674, "get_or_fail_fail": 0, "get_or_fail_success": 12674, "get_started": 0, "get_sum": 2708403, "max": 104857600, "put": 12674, "put_sum": 2708403, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1918, "put_sum": 18512601, "take": 5129, "take_sum": 18512601, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5129, "put_sum": 5129, "take": 5129, "take_sum": 5129, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1375, "get_or_fail_fail": 0, "get_or_fail_success": 1375, "get_started": 0, "get_sum": 138875, "max": 3758096384, "put": 8, "put_sum": 138875, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9022, "get_or_fail_fail": 0, "get_or_fail_success": 9022, "get_started": 0, "get_sum": 17153914, "max": 3758096384, "put": 1541, "put_sum": 17153914, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:31.303 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/226, 1375/1601 2020-11-20T12:17:32.305 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:32.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.551+0000 7f6f04f57700 1 Processor -- start 2020-11-20T12:17:32.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.551+0000 7f6f04f57700 1 -- start start 2020-11-20T12:17:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.552+0000 7f6f04f57700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00065ad0 0x7f6f00065eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.553+0000 7f6f04f57700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000b0fa0 con 0x7f6f00142950 2020-11-20T12:17:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.553+0000 7f6f04f57700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000b0380 con 0x7f6f000b16a0 2020-11-20T12:17:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.553+0000 7f6f04f57700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00149ac0 con 0x7f6f00065ad0 2020-11-20T12:17:32.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.553+0000 7f6efeffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6f000b16a0 0x7f6f000afc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47770/0 (socket says 172.21.15.187:47770) 2020-11-20T12:17:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.553+0000 7f6efeffd700 1 -- 172.21.15.187:0/3645693082 learned_addr learned my addr 172.21.15.187:0/3645693082 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.553+0000 7f6eff7fe700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00065ad0 0x7f6f00065eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.553+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573712976 0 0) 0x7f6f000b0380 con 0x7f6f000b16a0 2020-11-20T12:17:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec003240 con 0x7f6f000b16a0 2020-11-20T12:17:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2308297545 0 0) 0x7f6eec003240 con 0x7f6f000b16a0 2020-11-20T12:17:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00065ad0 msgr2=0x7f6f00065eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00065ad0 0x7f6f00065eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 >> v1:172.21.15.5:6789/0 conn(0x7f6f00142950 legacy=0x7f6f00145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:32.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00149d60 con 0x7f6f000b16a0 2020-11-20T12:17:32.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6ef40014e0 con 0x7f6f000b16a0 2020-11-20T12:17:32.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 >> v1:172.21.15.187:6789/0 conn(0x7f6f000b16a0 legacy=0x7f6f000afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:32.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6eec0037b0 0x7f6eec003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:32.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6eec0044e0 0x7f6eec004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:32.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00065ad0 0x7f6eec004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:32.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00149ac0 con 0x7f6eec0044e0 2020-11-20T12:17:32.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec005340 con 0x7f6eec0037b0 2020-11-20T12:17:32.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.554+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec00d940 con 0x7f6f00065ad0 2020-11-20T12:17:32.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.555+0000 7f6eff7fe700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6eec0037b0 0x7f6eec003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:32.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.555+0000 7f6eff7fe700 1 -- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00065ad0 msgr2=0x7f6eec004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:32.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.555+0000 7f6eff7fe700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00065ad0 0x7f6eec004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:32.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.555+0000 7f6efeffd700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6eec0044e0 0x7f6eec004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:32.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.555+0000 7f6eff7fe700 1 -- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6eec0044e0 msgr2=0x7f6eec004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:32.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.555+0000 7f6eff7fe700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6eec0044e0 0x7f6eec004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:32.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.555+0000 7f6eff7fe700 1 -- 172.21.15.187:0/3645693082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec003240 con 0x7f6eec0037b0 2020-11-20T12:17:32.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.555+0000 7f6eff7fe700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6eec0037b0 0x7f6eec003bd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7f6ef000ba80 tx=0x7f6ef0008040).ready entity=mon.1 client_cookie=4c87d0ce1f1d1aa5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:32.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000d9c0 con 0x7f6eec0037b0 2020-11-20T12:17:32.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef000db20 con 0x7f6eec0037b0 2020-11-20T12:17:32.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6efe7fc700 1 -- 172.21.15.187:0/3645693082 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000dea0 con 0x7f6eec0037b0 2020-11-20T12:17:32.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6f04f57700 1 -- 172.21.15.187:0/3645693082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6eec0037b0 msgr2=0x7f6eec003bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:32.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6f04f57700 1 --2- 172.21.15.187:0/3645693082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6eec0037b0 0x7f6eec003bd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7f6ef000ba80 tx=0x7f6ef0008040).stop 2020-11-20T12:17:32.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6efffff700 1 -- 172.21.15.187:0/3645693082 reap_dead start 2020-11-20T12:17:32.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6f04f57700 1 -- 172.21.15.187:0/3645693082 shutdown_connections 2020-11-20T12:17:32.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6f04f57700 1 -- 172.21.15.187:0/3645693082 >> 172.21.15.187:0/3645693082 conn(0x7f6f000b8280 msgr2=0x7f6f000653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:32.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6f04f57700 1 -- 172.21.15.187:0/3645693082 shutdown_connections 2020-11-20T12:17:32.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.556+0000 7f6f04f57700 1 -- 172.21.15.187:0/3645693082 wait complete. 2020-11-20T12:17:32.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.558+0000 7f6f04f57700 1 Processor -- start 2020-11-20T12:17:32.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.558+0000 7f6f04f57700 1 -- start start 2020-11-20T12:17:32.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.559+0000 7f6f04f57700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f000b2500 0x7f6f000b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:32.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.559+0000 7f6f04f57700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00147d40 0x7f6f00145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:32.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.559+0000 7f6f04f57700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f00140290 0x7f6f00140710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:32.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6f04f57700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f000b2f60 con 0x7f6f000b2500 2020-11-20T12:17:32.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6f04f57700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f000b3200 con 0x7f6f00140290 2020-11-20T12:17:32.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6f04f57700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00146270 con 0x7f6f00147d40 2020-11-20T12:17:32.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6efffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f00140290 0x7f6f00140710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:32.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6efffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f00140290 0x7f6f00140710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51176/0 (socket says 172.21.15.187:51176) 2020-11-20T12:17:32.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6efffff700 1 -- 172.21.15.187:0/3252866193 learned_addr learned my addr 172.21.15.187:0/3252866193 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:32.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6efffff700 1 -- 172.21.15.187:0/3252866193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00147d40 msgr2=0x7f6f00145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:32.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6efffff700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00147d40 0x7f6f00145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:32.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6efffff700 1 -- 172.21.15.187:0/3252866193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f000b2500 msgr2=0x7f6f000b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:32.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6efffff700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f000b2500 0x7f6f000b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:32.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.560+0000 7f6efffff700 1 -- 172.21.15.187:0/3252866193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00146430 con 0x7f6f00140290 2020-11-20T12:17:32.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.561+0000 7f6efffff700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f00140290 0x7f6f00140710 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f6f00142d30 tx=0x7f6ef8007d40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:32.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.561+0000 7f6edf7fe700 1 -- 172.21.15.187:0/3252866193 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8007320 con 0x7f6f00140290 2020-11-20T12:17:32.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.561+0000 7f6edf7fe700 1 -- 172.21.15.187:0/3252866193 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef80074a0 con 0x7f6f00140290 2020-11-20T12:17:32.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.562+0000 7f6edf7fe700 1 -- 172.21.15.187:0/3252866193 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8005860 con 0x7f6f00140290 2020-11-20T12:17:32.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.562+0000 7f6f04f57700 1 -- 172.21.15.187:0/3252866193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0009e9e0 con 0x7f6f00140290 2020-11-20T12:17:32.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.562+0000 7f6f04f57700 1 -- 172.21.15.187:0/3252866193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0009ee80 con 0x7f6f00140290 2020-11-20T12:17:32.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.564+0000 7f6edd7fa700 1 -- 172.21.15.187:0/3252866193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed0004f90 con 0x7f6f00140290 2020-11-20T12:17:32.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.564+0000 7f6edf7fe700 1 -- 172.21.15.187:0/3252866193 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6ef8010070 con 0x7f6f00140290 2020-11-20T12:17:32.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.565+0000 7f6edf7fe700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ed8056d20 0x7f6ed80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:32.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.565+0000 7f6edf7fe700 1 -- 172.21.15.187:0/3252866193 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6ef806f650 con 0x7f6f00140290 2020-11-20T12:17:32.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.565+0000 7f6eff7fe700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ed8056d20 0x7f6ed80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:32.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.565+0000 7f6eff7fe700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ed8056d20 0x7f6ed80591e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f6f0005d960 tx=0x7f6ef0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:32.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.569+0000 7f6edf7fe700 1 -- 172.21.15.187:0/3252866193 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6ef80449f0 con 0x7f6f00140290 2020-11-20T12:17:32.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.821+0000 7f6edd7fa700 1 -- 172.21.15.187:0/3252866193 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ed0001660 con 0x7f6f00140290 2020-11-20T12:17:32.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.822+0000 7f6edf7fe700 1 -- 172.21.15.187:0/3252866193 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f6ef8048010 con 0x7f6f00140290 2020-11-20T12:17:32.826 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:32.826 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:32.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 -- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ed8056d20 msgr2=0x7f6ed80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:32.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ed8056d20 0x7f6ed80591e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f6f0005d960 tx=0x7f6ef0008040).stop 2020-11-20T12:17:32.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 -- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f00140290 msgr2=0x7f6f00140710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:32.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f00140290 0x7f6f00140710 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f6f00142d30 tx=0x7f6ef8007d40).stop 2020-11-20T12:17:32.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 -- 172.21.15.187:0/3252866193 shutdown_connections 2020-11-20T12:17:32.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6ed8056d20 0x7f6ed80591e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:32.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f000b2500 0x7f6f000b28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:32.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f00140290 0x7f6f00140710 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:32.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 --2- 172.21.15.187:0/3252866193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f00147d40 0x7f6f00145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:32.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 -- 172.21.15.187:0/3252866193 >> 172.21.15.187:0/3252866193 conn(0x7f6f000b8280 msgr2=0x7f6f000b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:32.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 -- 172.21.15.187:0/3252866193 shutdown_connections 2020-11-20T12:17:32.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:32.825+0000 7f6edd7fa700 1 -- 172.21.15.187:0/3252866193 wait complete. 2020-11-20T12:17:32.832 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:32.853 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:33.124 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2558,"msgr_send_messages":2560,"msgr_recv_bytes":797524,"msgr_send_bytes":9852316,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.437423668,"msgr_running_send_time":0.180331672,"msgr_running_recv_time":0.097256311,"msgr_running_fast_dispatch_time":0.123871944,"msgr_send_messages_queue_lat":{"avgcount":2558,"sum":0.454264995,"avgtime":0.000177586},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9241,"msgr_send_messages":14577,"msgr_recv_bytes":2624782,"msgr_send_bytes":14196689,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.011484436,"msgr_running_send_time":2.237313886,"msgr_running_recv_time":0.934355247,"msgr_running_fast_dispatch_time":0.091344145,"msgr_send_messages_queue_lat":{"avgcount":14576,"sum":1.924091373,"avgtime":0.000132004},"msgr_handle_ack_lat":{"avgcount":8614,"sum":0.006399116,"avgtime":0.000000742}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":950,"msgr_send_messages":947,"msgr_recv_bytes":327702,"msgr_send_bytes":625653,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.106203850,"msgr_running_send_time":0.036627710,"msgr_running_recv_time":0.038573686,"msgr_running_fast_dispatch_time":0.027133361,"msgr_send_messages_queue_lat":{"avgcount":947,"sum":0.146116767,"avgtime":0.000154294},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3116,"sum":3.693394642,"avgtime":0.001185299}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3571,"sum":0.335669930,"avgtime":0.000093998}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":759,"inodes_top":0,"inodes_bottom":759,"inodes_pin_tail":0,"inodes_pinned":759,"inodes_expired":0,"inodes_with_caps":227,"caps":227,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":250689,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4813,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4158,"handle_client_caps_dirty":1500,"handle_client_cap_release":11,"process_request_cap_release":1668},"mds_cache":{"num_strays":225,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1376,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9023,"evex":9025,"evtrm":9025,"ev":2,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21351759,"rdpos":4195632,"jlat":{"avgcount":1541,"sum":18.353403841,"avgtime":0.011910060},"replayed":4},"mds_mem":{"ino":239,"ino+":1614,"ino-":1375,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":226,"cap+":1602,"cap-":1376,"rss":94024,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1231},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8730025,"buffer_anon_items":3005,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":2722085,"mds_co_items":55174,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5197,"op_send_bytes":18372039,"op_resend":0,"op_reply":5197,"oplen_avg":{"avgcount":5197,"sum":5804},"op":5197,"op_r":35,"op_w":5162,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":17,"osdop_write":1550,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3263,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":193,"omap_rd":28,"omap_del":14},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1376,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12745,"get_sum":2719924,"get_or_fail_fail":0,"get_or_fail_success":12745,"take":0,"take_sum":0,"put":12745,"put_sum":2719924,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5197,"take_sum":18512909,"put":1921,"put_sum":18512909,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5197,"take_sum":5197,"put":5197,"put_sum":5197,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1376,"get_sum":138984,"get_or_fail_fail":0,"get_or_fail_success":1376,"take":0,"take_sum":0,"put":9,"put_sum":138984,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":1431,"max":3758096384,"get_started":0,"get":9023,"get_sum":17155345,"get_or_fail_fail":0,"get_or_fail_success":9023,"take":0,"take_sum":0,"put":1541,"put_sum":17153914,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:33.126 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 797524, "msgr_recv_messages": 2558, "msgr_running_fast_dispatch_time": 0.123871944, "msgr_running_recv_time": 0.097256311, "msgr_running_send_time": 0.180331672, "msgr_running_total_time": 0.437423668, "msgr_send_bytes": 9852316, "msgr_send_messages": 2560, "msgr_send_messages_queue_lat": { "avgcount": 2558, "avgtime": 0.000177586, "sum": 0.454264995 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8614, "avgtime": 7.42e-07, "sum": 0.006399116 }, "msgr_recv_bytes": 2624782, "msgr_recv_messages": 9241, "msgr_running_fast_dispatch_time": 0.091344145, "msgr_running_recv_time": 0.934355247, "msgr_running_send_time": 2.237313886, "msgr_running_total_time": 3.011484436, "msgr_send_bytes": 14196689, "msgr_send_messages": 14577, "msgr_send_messages_queue_lat": { "avgcount": 14576, "avgtime": 0.000132004, "sum": 1.924091373 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 327702, "msgr_recv_messages": 950, "msgr_running_fast_dispatch_time": 0.027133361, "msgr_running_recv_time": 0.038573686, "msgr_running_send_time": 0.03662771, "msgr_running_total_time": 0.10620385, "msgr_send_bytes": 625653, "msgr_send_messages": 947, "msgr_send_messages_queue_lat": { "avgcount": 947, "avgtime": 0.000154294, "sum": 0.146116767 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3116, "avgtime": 0.001185299, "sum": 3.693394642 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3571, "avgtime": 9.3998e-05, "sum": 0.33566993 }, "queue_len": 0 }, "mds": { "caps": 227, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4813, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 11, "handle_client_caps": 4158, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 759, "inodes_bottom": 759, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 759, "inodes_top": 0, "inodes_with_caps": 227, "load_cent": 250689, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 225, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1376, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 2, "evadd": 9023, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1541, "avgtime": 0.01191006, "sum": 18.353403841 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21351759 }, "mds_mem": { "cap": 226, "cap+": 1602, "cap-": 1376, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 239, "ino+": 1614, "ino-": 1375, "rss": 94024 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1231, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8730025, "buffer_anon_items": 3005, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 2722085, "mds_co_items": 55174, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 14, "omap_rd": 28, "omap_wr": 193, "op": 5197, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 35, "op_reply": 5197, "op_resend": 0, "op_rmw": 0, "op_send": 5197, "op_send_bytes": 18372039, "op_w": 5162, "oplen_avg": { "avgcount": 5197, "sum": 5804 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3263, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1550, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1376, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12745, "get_or_fail_fail": 0, "get_or_fail_success": 12745, "get_started": 0, "get_sum": 2719924, "max": 104857600, "put": 12745, "put_sum": 2719924, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1921, "put_sum": 18512909, "take": 5197, "take_sum": 18512909, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5197, "put_sum": 5197, "take": 5197, "take_sum": 5197, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1376, "get_or_fail_fail": 0, "get_or_fail_success": 1376, "get_started": 0, "get_sum": 138984, "max": 3758096384, "put": 9, "put_sum": 138984, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9023, "get_or_fail_fail": 0, "get_or_fail_success": 9023, "get_started": 0, "get_sum": 17155345, "max": 3758096384, "put": 1541, "put_sum": 17153914, "take": 0, "take_sum": 0, "val": 1431, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:33.126 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/225, 1376/1601 2020-11-20T12:17:34.128 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:34.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.375+0000 7f20fb59e700 1 Processor -- start 2020-11-20T12:17:34.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.375+0000 7f20fb59e700 1 -- start start 2020-11-20T12:17:34.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.377+0000 7f20fb59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20fc0a1490 0x7f20fc0b9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:34.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.377+0000 7f20fb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:34.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.377+0000 7f20fb59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc14d140 con 0x7f20fc0a1870 2020-11-20T12:17:34.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.377+0000 7f20fb59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20fc14cea0 con 0x7f20fc0a1490 2020-11-20T12:17:34.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.377+0000 7f20fb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc148da0 con 0x7f20fc0ba2a0 2020-11-20T12:17:34.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.377+0000 7f20fa59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20fc0a1490 0x7f20fc0b9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:34.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.377+0000 7f20fa59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20fc0a1490 0x7f20fc0b9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51182/0 (socket says 172.21.15.187:51182) 2020-11-20T12:17:34.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.377+0000 7f20fa59c700 1 -- 172.21.15.187:0/351539974 learned_addr learned my addr 172.21.15.187:0/351539974 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:34.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.378+0000 7f20fad9d700 1 --2- 172.21.15.187:0/351539974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:34.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.378+0000 7f20fa59c700 1 -- 172.21.15.187:0/351539974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 msgr2=0x7f20fc149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:34.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.378+0000 7f20fa59c700 1 --2- 172.21.15.187:0/351539974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.378+0000 7f20fa59c700 1 -- 172.21.15.187:0/351539974 >> v1:172.21.15.5:6789/0 conn(0x7f20fc0a1870 legacy=0x7f20fc09fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:34.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.378+0000 7f20fa59c700 1 -- 172.21.15.187:0/351539974 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc0a04e0 con 0x7f20fc0a1490 2020-11-20T12:17:34.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.378+0000 7f20fad9d700 1 --2- 172.21.15.187:0/351539974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:34.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.378+0000 7f20fa59c700 1 --2- 172.21.15.187:0/351539974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1490 0x7f20fc0b9710 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7f20ec006fd0 tx=0x7f20ec006eb0).ready entity=mon.1 client_cookie=d011fd7759ead0f6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:34.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.379+0000 7f20f959a700 1 -- 172.21.15.187:0/351539974 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec009890 con 0x7f20fc0a1490 2020-11-20T12:17:34.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.379+0000 7f20f959a700 1 -- 172.21.15.187:0/351539974 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ec0099f0 con 0x7f20fc0a1490 2020-11-20T12:17:34.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.379+0000 7f20f959a700 1 -- 172.21.15.187:0/351539974 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec009d70 con 0x7f20fc0a1490 2020-11-20T12:17:34.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.380+0000 7f20fb59e700 1 -- 172.21.15.187:0/351539974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1490 msgr2=0x7f20fc0b9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:34.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.380+0000 7f20fb59e700 1 --2- 172.21.15.187:0/351539974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1490 0x7f20fc0b9710 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7f20ec006fd0 tx=0x7f20ec006eb0).stop 2020-11-20T12:17:34.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.380+0000 7f20fb59e700 1 -- 172.21.15.187:0/351539974 shutdown_connections 2020-11-20T12:17:34.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.380+0000 7f20fb59e700 1 --2- 172.21.15.187:0/351539974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.380+0000 7f20fb59e700 1 --2- 172.21.15.187:0/351539974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1490 0x7f20fc0b9710 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.380+0000 7f20fb59e700 1 -- 172.21.15.187:0/351539974 >> 172.21.15.187:0/351539974 conn(0x7f20fc0bf110 msgr2=0x7f20fc0bf520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:34.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.380+0000 7f20fb59e700 1 -- 172.21.15.187:0/351539974 shutdown_connections 2020-11-20T12:17:34.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.380+0000 7f20fb59e700 1 -- 172.21.15.187:0/351539974 wait complete. 2020-11-20T12:17:34.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.382+0000 7f20fb59e700 1 Processor -- start 2020-11-20T12:17:34.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.382+0000 7f20fb59e700 1 -- start start 2020-11-20T12:17:34.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.382+0000 7f20fb59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1870 0x7f20fc0a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:34.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:34.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20fc0bd770 0x7f20fc139a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:34.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc14a000 con 0x7f20fc0bd770 2020-11-20T12:17:34.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fb59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc14a220 con 0x7f20fc0a1870 2020-11-20T12:17:34.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc14a4c0 con 0x7f20fc0ba2a0 2020-11-20T12:17:34.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fa59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1870 0x7f20fc0a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:34.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fa59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1870 0x7f20fc0a3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51188/0 (socket says 172.21.15.187:51188) 2020-11-20T12:17:34.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fa59c700 1 -- 172.21.15.187:0/1734077930 learned_addr learned my addr 172.21.15.187:0/1734077930 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:34.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fa59c700 1 -- 172.21.15.187:0/1734077930 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 msgr2=0x7f20fc149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:34.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20f9d9b700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:34.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fad9d700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20fc0bd770 0x7f20fc139a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:34.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fa59c700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fa59c700 1 -- 172.21.15.187:0/1734077930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20fc0bd770 msgr2=0x7f20fc139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:34.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fa59c700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20fc0bd770 0x7f20fc139a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.383+0000 7f20fa59c700 1 -- 172.21.15.187:0/1734077930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc14a760 con 0x7f20fc0a1870 2020-11-20T12:17:34.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.384+0000 7f20fa59c700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1870 0x7f20fc0a3fd0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f20fc0bfc50 tx=0x7f20ec015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.384+0000 7f20ea7fc700 1 -- 172.21.15.187:0/1734077930 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec01e460 con 0x7f20fc0a1870 2020-11-20T12:17:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.384+0000 7f20ea7fc700 1 -- 172.21.15.187:0/1734077930 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ec01e5c0 con 0x7f20fc0a1870 2020-11-20T12:17:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.384+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20fc0bdbf0 con 0x7f20fc0a1870 2020-11-20T12:17:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.384+0000 7f20ea7fc700 1 -- 172.21.15.187:0/1734077930 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec01e960 con 0x7f20fc0a1870 2020-11-20T12:17:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.384+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20fc13a190 con 0x7f20fc0a1870 2020-11-20T12:17:34.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.386+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20fc13a440 con 0x7f20fc0a1870 2020-11-20T12:17:34.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.386+0000 7f20ea7fc700 1 -- 172.21.15.187:0/1734077930 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f20ec025030 con 0x7f20fc0a1870 2020-11-20T12:17:34.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.387+0000 7f20ea7fc700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f20d0056d20 0x7f20d00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:34.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.387+0000 7f20ea7fc700 1 -- 172.21.15.187:0/1734077930 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f20ec071c00 con 0x7f20fc0a1870 2020-11-20T12:17:34.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.387+0000 7f20f9d9b700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f20d0056d20 0x7f20d00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:34.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.388+0000 7f20f9d9b700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f20d0056d20 0x7f20d00591e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f20fc0b8dd0 tx=0x7f20f0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:34.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.391+0000 7f20ea7fc700 1 -- 172.21.15.187:0/1734077930 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f20ec077020 con 0x7f20fc0a1870 2020-11-20T12:17:34.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.638+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20fc14ab00 con 0x7f20fc0a1870 2020-11-20T12:17:34.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.639+0000 7f20ea7fc700 1 -- 172.21.15.187:0/1734077930 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f20ec077870 con 0x7f20fc0a1870 2020-11-20T12:17:34.643 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:34.643 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:34.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f20d0056d20 msgr2=0x7f20d00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:34.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f20d0056d20 0x7f20d00591e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f20fc0b8dd0 tx=0x7f20f0009040).stop 2020-11-20T12:17:34.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1870 msgr2=0x7f20fc0a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:34.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1870 0x7f20fc0a3fd0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f20fc0bfc50 tx=0x7f20ec015000).stop 2020-11-20T12:17:34.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 shutdown_connections 2020-11-20T12:17:34.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f20d0056d20 0x7f20d00591e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20fc0bd770 0x7f20fc139a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20fc0a1870 0x7f20fc0a3fd0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 --2- 172.21.15.187:0/1734077930 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20fc0ba2a0 0x7f20fc149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:34.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 >> 172.21.15.187:0/1734077930 conn(0x7f20fc0bf110 msgr2=0x7f20fc147db0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:34.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.641+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 shutdown_connections 2020-11-20T12:17:34.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:34.642+0000 7f20fb59e700 1 -- 172.21.15.187:0/1734077930 wait complete. 2020-11-20T12:17:34.648 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:34.670 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:34.940 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2559,"msgr_send_messages":2561,"msgr_recv_bytes":797770,"msgr_send_bytes":9854058,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.437699528,"msgr_running_send_time":0.180457533,"msgr_running_recv_time":0.097306754,"msgr_running_fast_dispatch_time":0.123942413,"msgr_send_messages_queue_lat":{"avgcount":2559,"sum":0.454356387,"avgtime":0.000177552},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9243,"msgr_send_messages":14578,"msgr_recv_bytes":2625182,"msgr_send_bytes":14198385,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.012260993,"msgr_running_send_time":2.237746954,"msgr_running_recv_time":0.934461863,"msgr_running_fast_dispatch_time":0.091544758,"msgr_send_messages_queue_lat":{"avgcount":14577,"sum":1.924216441,"avgtime":0.000132003},"msgr_handle_ack_lat":{"avgcount":8614,"sum":0.006399116,"avgtime":0.000000742}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":952,"msgr_send_messages":949,"msgr_recv_bytes":328266,"msgr_send_bytes":626627,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.106979786,"msgr_running_send_time":0.036915233,"msgr_running_recv_time":0.038788621,"msgr_running_fast_dispatch_time":0.027282911,"msgr_send_messages_queue_lat":{"avgcount":949,"sum":0.146370231,"avgtime":0.000154236},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3119,"sum":3.693745184,"avgtime":0.001184272}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3571,"sum":0.335669930,"avgtime":0.000093998}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":236,"inodes_top":0,"inodes_bottom":236,"inodes_pin_tail":0,"inodes_pinned":236,"inodes_expired":0,"inodes_with_caps":226,"caps":226,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":250689,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4813,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4158,"handle_client_caps_dirty":1500,"handle_client_cap_release":11,"process_request_cap_release":1668},"mds_cache":{"num_strays":225,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1376,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9023,"evex":9025,"evtrm":9025,"ev":2,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21351759,"rdpos":4195632,"jlat":{"avgcount":1542,"sum":18.361485643,"avgtime":0.011907578},"replayed":4},"mds_mem":{"ino":238,"ino+":1614,"ino-":1376,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":226,"cap+":1602,"cap-":1376,"rss":73752,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1236},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":4593623,"buffer_anon_items":1424,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":799053,"mds_co_items":14046,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5199,"op_send_bytes":18373531,"op_resend":0,"op_reply":5199,"oplen_avg":{"avgcount":5199,"sum":5807},"op":5199,"op_r":35,"op_w":5164,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":17,"osdop_write":1551,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3263,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":194,"omap_rd":28,"omap_del":15},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1376,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12750,"get_sum":2720664,"get_or_fail_fail":0,"get_or_fail_success":12750,"take":0,"take_sum":0,"put":12750,"put_sum":2720664,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5199,"take_sum":18514401,"put":1923,"put_sum":18514401,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5199,"take_sum":5199,"put":5199,"put_sum":5199,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1376,"get_sum":138984,"get_or_fail_fail":0,"get_or_fail_success":1376,"take":0,"take_sum":0,"put":9,"put_sum":138984,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":0,"max":3758096384,"get_started":0,"get":9023,"get_sum":17155345,"get_or_fail_fail":0,"get_or_fail_success":9023,"take":0,"take_sum":0,"put":1542,"put_sum":17155345,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:34.942 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 797770, "msgr_recv_messages": 2559, "msgr_running_fast_dispatch_time": 0.123942413, "msgr_running_recv_time": 0.097306754, "msgr_running_send_time": 0.180457533, "msgr_running_total_time": 0.437699528, "msgr_send_bytes": 9854058, "msgr_send_messages": 2561, "msgr_send_messages_queue_lat": { "avgcount": 2559, "avgtime": 0.000177552, "sum": 0.454356387 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8614, "avgtime": 7.42e-07, "sum": 0.006399116 }, "msgr_recv_bytes": 2625182, "msgr_recv_messages": 9243, "msgr_running_fast_dispatch_time": 0.091544758, "msgr_running_recv_time": 0.934461863, "msgr_running_send_time": 2.237746954, "msgr_running_total_time": 3.012260993, "msgr_send_bytes": 14198385, "msgr_send_messages": 14578, "msgr_send_messages_queue_lat": { "avgcount": 14577, "avgtime": 0.000132003, "sum": 1.924216441 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 328266, "msgr_recv_messages": 952, "msgr_running_fast_dispatch_time": 0.027282911, "msgr_running_recv_time": 0.038788621, "msgr_running_send_time": 0.036915233, "msgr_running_total_time": 0.106979786, "msgr_send_bytes": 626627, "msgr_send_messages": 949, "msgr_send_messages_queue_lat": { "avgcount": 949, "avgtime": 0.000154236, "sum": 0.146370231 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3119, "avgtime": 0.001184272, "sum": 3.693745184 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3571, "avgtime": 9.3998e-05, "sum": 0.33566993 }, "queue_len": 0 }, "mds": { "caps": 226, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4813, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 11, "handle_client_caps": 4158, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 236, "inodes_bottom": 236, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 236, "inodes_top": 0, "inodes_with_caps": 226, "load_cent": 250689, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 225, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1376, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 2, "evadd": 9023, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1542, "avgtime": 0.011907578, "sum": 18.361485643 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21351759 }, "mds_mem": { "cap": 226, "cap+": 1602, "cap-": 1376, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 238, "ino+": 1614, "ino-": 1376, "rss": 73752 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1236, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 4593623, "buffer_anon_items": 1424, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 799053, "mds_co_items": 14046, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 15, "omap_rd": 28, "omap_wr": 194, "op": 5199, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 35, "op_reply": 5199, "op_resend": 0, "op_rmw": 0, "op_send": 5199, "op_send_bytes": 18373531, "op_w": 5164, "oplen_avg": { "avgcount": 5199, "sum": 5807 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3263, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1551, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1376, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12750, "get_or_fail_fail": 0, "get_or_fail_success": 12750, "get_started": 0, "get_sum": 2720664, "max": 104857600, "put": 12750, "put_sum": 2720664, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1923, "put_sum": 18514401, "take": 5199, "take_sum": 18514401, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5199, "put_sum": 5199, "take": 5199, "take_sum": 5199, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1376, "get_or_fail_fail": 0, "get_or_fail_success": 1376, "get_started": 0, "get_sum": 138984, "max": 3758096384, "put": 9, "put_sum": 138984, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9023, "get_or_fail_fail": 0, "get_or_fail_success": 9023, "get_started": 0, "get_sum": 17155345, "max": 3758096384, "put": 1542, "put_sum": 17155345, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:34.943 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/225, 1376/1601 2020-11-20T12:17:35.944 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:36.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.192+0000 7f2f3ffff700 1 Processor -- start 2020-11-20T12:17:36.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.192+0000 7f2f3ffff700 1 -- start start 2020-11-20T12:17:36.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.193+0000 7f2f3ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40065eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:36.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.193+0000 7f2f3ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400b0fa0 con 0x7f2f40142950 2020-11-20T12:17:36.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.193+0000 7f2f3ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400b0380 con 0x7f2f40065ad0 2020-11-20T12:17:36.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.193+0000 7f2f3ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40149ac0 con 0x7f2f400b16a0 2020-11-20T12:17:36.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.194+0000 7f2f3effd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2f40065ad0 0x7f2f400afc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47804/0 (socket says 172.21.15.187:47804) 2020-11-20T12:17:36.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.194+0000 7f2f3effd700 1 -- 172.21.15.187:0/3830004962 learned_addr learned my addr 172.21.15.187:0/3830004962 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:36.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.194+0000 7f2f3e7fc700 1 --2- 172.21.15.187:0/3830004962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40065eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:36.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.195+0000 7f2f3dffb700 1 -- 172.21.15.187:0/3830004962 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720431198 0 0) 0x7f2f400b0380 con 0x7f2f40065ad0 2020-11-20T12:17:36.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.195+0000 7f2f3dffb700 1 -- 172.21.15.187:0/3830004962 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f2c003240 con 0x7f2f40065ad0 2020-11-20T12:17:36.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.195+0000 7f2f3e7fc700 1 -- 172.21.15.187:0/3830004962 >> v1:172.21.15.187:6789/0 conn(0x7f2f40065ad0 legacy=0x7f2f400afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:36.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.195+0000 7f2f3e7fc700 1 -- 172.21.15.187:0/3830004962 >> v1:172.21.15.5:6789/0 conn(0x7f2f40142950 legacy=0x7f2f40142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:36.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.195+0000 7f2f3e7fc700 1 -- 172.21.15.187:0/3830004962 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40149d60 con 0x7f2f400b16a0 2020-11-20T12:17:36.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.196+0000 7f2f3e7fc700 1 --2- 172.21.15.187:0/3830004962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40065eb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f2f3400b9d0 tx=0x7f2f34006df0).ready entity=mon.2 client_cookie=20be45c1d4b2d351 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:36.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.196+0000 7f2f3dffb700 1 -- 172.21.15.187:0/3830004962 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f340096b0 con 0x7f2f400b16a0 2020-11-20T12:17:36.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3dffb700 1 -- 172.21.15.187:0/3830004962 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f34009810 con 0x7f2f400b16a0 2020-11-20T12:17:36.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3dffb700 1 -- 172.21.15.187:0/3830004962 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34009b90 con 0x7f2f400b16a0 2020-11-20T12:17:36.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3ffff700 1 -- 172.21.15.187:0/3830004962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 msgr2=0x7f2f40065eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:36.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3ffff700 1 --2- 172.21.15.187:0/3830004962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40065eb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f2f3400b9d0 tx=0x7f2f34006df0).stop 2020-11-20T12:17:36.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3ffff700 1 -- 172.21.15.187:0/3830004962 shutdown_connections 2020-11-20T12:17:36.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3ffff700 1 --2- 172.21.15.187:0/3830004962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40065eb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:36.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3ffff700 1 -- 172.21.15.187:0/3830004962 >> 172.21.15.187:0/3830004962 conn(0x7f2f400b8280 msgr2=0x7f2f400653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:36.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3ffff700 1 -- 172.21.15.187:0/3830004962 shutdown_connections 2020-11-20T12:17:36.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.197+0000 7f2f3ffff700 1 -- 172.21.15.187:0/3830004962 wait complete. 2020-11-20T12:17:36.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.199+0000 7f2f3ffff700 1 Processor -- start 2020-11-20T12:17:36.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.199+0000 7f2f3ffff700 1 -- start start 2020-11-20T12:17:36.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.200+0000 7f2f3ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f40065ad0 0x7f2f400b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:36.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.200+0000 7f2f3ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:36.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.200+0000 7f2f3ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f40142950 0x7f2f40141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:36.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.200+0000 7f2f3ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400b4a70 con 0x7f2f40142950 2020-11-20T12:17:36.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.200+0000 7f2f3ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4009f740 con 0x7f2f40065ad0 2020-11-20T12:17:36.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.200+0000 7f2f3ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400b4d10 con 0x7f2f400b16a0 2020-11-20T12:17:36.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f40065ad0 0x7f2f400b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:36.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f40065ad0 0x7f2f400b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51202/0 (socket says 172.21.15.187:51202) 2020-11-20T12:17:36.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3effd700 1 -- 172.21.15.187:0/2787787482 learned_addr learned my addr 172.21.15.187:0/2787787482 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:36.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3e7fc700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:36.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3effd700 1 -- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 msgr2=0x7f2f40141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:36.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3effd700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:36.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3f7fe700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f40142950 0x7f2f40141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:36.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3effd700 1 -- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f40142950 msgr2=0x7f2f40141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:36.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3effd700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f40142950 0x7f2f40141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:36.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3effd700 1 -- 172.21.15.187:0/2787787482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40148810 con 0x7f2f40065ad0 2020-11-20T12:17:36.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.201+0000 7f2f3e7fc700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:36.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.202+0000 7f2f3effd700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f40065ad0 0x7f2f400b2870 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 rx=0x7f2f3000c6f0 tx=0x7f2f30007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:36.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.202+0000 7f2f26ffd700 1 -- 172.21.15.187:0/2787787482 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30007320 con 0x7f2f40065ad0 2020-11-20T12:17:36.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.202+0000 7f2f26ffd700 1 -- 172.21.15.187:0/2787787482 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f30007480 con 0x7f2f40065ad0 2020-11-20T12:17:36.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.202+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f400b45d0 con 0x7f2f40065ad0 2020-11-20T12:17:36.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.202+0000 7f2f26ffd700 1 -- 172.21.15.187:0/2787787482 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30005800 con 0x7f2f40065ad0 2020-11-20T12:17:36.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.202+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f400b2f20 con 0x7f2f40065ad0 2020-11-20T12:17:36.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.203+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f40068ef0 con 0x7f2f40065ad0 2020-11-20T12:17:36.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.204+0000 7f2f26ffd700 1 -- 172.21.15.187:0/2787787482 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2f30010030 con 0x7f2f40065ad0 2020-11-20T12:17:36.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.205+0000 7f2f26ffd700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f1c056d20 0x7f2f1c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:36.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.205+0000 7f2f26ffd700 1 -- 172.21.15.187:0/2787787482 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2f3006fd30 con 0x7f2f40065ad0 2020-11-20T12:17:36.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.205+0000 7f2f3e7fc700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f1c056d20 0x7f2f1c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:36.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.206+0000 7f2f3e7fc700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f1c056d20 0x7f2f1c0591e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f2f3400b920 tx=0x7f2f34015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:36.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.209+0000 7f2f26ffd700 1 -- 172.21.15.187:0/2787787482 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2f30045150 con 0x7f2f40065ad0 2020-11-20T12:17:36.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.486+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f40044f60 con 0x7f2f40065ad0 2020-11-20T12:17:36.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.487+0000 7f2f26ffd700 1 -- 172.21.15.187:0/2787787482 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f2f30048770 con 0x7f2f40065ad0 2020-11-20T12:17:36.491 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:36.492 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:36.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f1c056d20 msgr2=0x7f2f1c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:36.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f1c056d20 0x7f2f1c0591e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f2f3400b920 tx=0x7f2f34015000).stop 2020-11-20T12:17:36.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f40065ad0 msgr2=0x7f2f400b2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:36.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f40065ad0 0x7f2f400b2870 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 rx=0x7f2f3000c6f0 tx=0x7f2f30007ad0).stop 2020-11-20T12:17:36.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 shutdown_connections 2020-11-20T12:17:36.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2f1c056d20 0x7f2f1c0591e0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:36.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f40142950 0x7f2f40141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:36.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f40065ad0 0x7f2f400b2870 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:36.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 --2- 172.21.15.187:0/2787787482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f400b16a0 0x7f2f40141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:36.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 >> 172.21.15.187:0/2787787482 conn(0x7f2f400b8280 msgr2=0x7f2f400b90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:36.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.490+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 shutdown_connections 2020-11-20T12:17:36.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:36.491+0000 7f2f3ffff700 1 -- 172.21.15.187:0/2787787482 wait complete. 2020-11-20T12:17:36.497 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:36.518 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:36.789 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2559,"msgr_send_messages":2561,"msgr_recv_bytes":797770,"msgr_send_bytes":9854058,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.437699528,"msgr_running_send_time":0.180457533,"msgr_running_recv_time":0.097306754,"msgr_running_fast_dispatch_time":0.123942413,"msgr_send_messages_queue_lat":{"avgcount":2559,"sum":0.454356387,"avgtime":0.000177552},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9245,"msgr_send_messages":14578,"msgr_recv_bytes":2625582,"msgr_send_bytes":14198385,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.012921360,"msgr_running_send_time":2.238046586,"msgr_running_recv_time":0.934636791,"msgr_running_fast_dispatch_time":0.091720855,"msgr_send_messages_queue_lat":{"avgcount":14577,"sum":1.924216441,"avgtime":0.000132003},"msgr_handle_ack_lat":{"avgcount":8614,"sum":0.006399116,"avgtime":0.000000742}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":952,"msgr_send_messages":949,"msgr_recv_bytes":328266,"msgr_send_bytes":626627,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.106979786,"msgr_running_send_time":0.036915233,"msgr_running_recv_time":0.038788621,"msgr_running_fast_dispatch_time":0.027282911,"msgr_send_messages_queue_lat":{"avgcount":949,"sum":0.146370231,"avgtime":0.000154236},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3119,"sum":3.693745184,"avgtime":0.001184272}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3571,"sum":0.335669930,"avgtime":0.000093998}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":236,"inodes_top":0,"inodes_bottom":236,"inodes_pin_tail":0,"inodes_pinned":236,"inodes_expired":0,"inodes_with_caps":226,"caps":226,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":250689,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4813,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4158,"handle_client_caps_dirty":1500,"handle_client_cap_release":11,"process_request_cap_release":1668},"mds_cache":{"num_strays":225,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1376,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9023,"evex":9025,"evtrm":9025,"ev":2,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21351759,"rdpos":4195632,"jlat":{"avgcount":1542,"sum":18.361485643,"avgtime":0.011907578},"replayed":4},"mds_mem":{"ino":238,"ino+":1614,"ino-":1376,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":226,"cap+":1602,"cap-":1376,"rss":73752,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1236},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":4593623,"buffer_anon_items":1424,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":799053,"mds_co_items":14046,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5199,"op_send_bytes":18373531,"op_resend":0,"op_reply":5199,"oplen_avg":{"avgcount":5199,"sum":5807},"op":5199,"op_r":35,"op_w":5164,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":17,"osdop_write":1551,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3263,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":194,"omap_rd":28,"omap_del":15},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1376,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12752,"get_sum":2720916,"get_or_fail_fail":0,"get_or_fail_success":12752,"take":0,"take_sum":0,"put":12752,"put_sum":2720916,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5199,"take_sum":18514401,"put":1923,"put_sum":18514401,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5199,"take_sum":5199,"put":5199,"put_sum":5199,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1376,"get_sum":138984,"get_or_fail_fail":0,"get_or_fail_success":1376,"take":0,"take_sum":0,"put":9,"put_sum":138984,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":0,"max":3758096384,"get_started":0,"get":9023,"get_sum":17155345,"get_or_fail_fail":0,"get_or_fail_success":9023,"take":0,"take_sum":0,"put":1542,"put_sum":17155345,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:36.790 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 797770, "msgr_recv_messages": 2559, "msgr_running_fast_dispatch_time": 0.123942413, "msgr_running_recv_time": 0.097306754, "msgr_running_send_time": 0.180457533, "msgr_running_total_time": 0.437699528, "msgr_send_bytes": 9854058, "msgr_send_messages": 2561, "msgr_send_messages_queue_lat": { "avgcount": 2559, "avgtime": 0.000177552, "sum": 0.454356387 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8614, "avgtime": 7.42e-07, "sum": 0.006399116 }, "msgr_recv_bytes": 2625582, "msgr_recv_messages": 9245, "msgr_running_fast_dispatch_time": 0.091720855, "msgr_running_recv_time": 0.934636791, "msgr_running_send_time": 2.238046586, "msgr_running_total_time": 3.01292136, "msgr_send_bytes": 14198385, "msgr_send_messages": 14578, "msgr_send_messages_queue_lat": { "avgcount": 14577, "avgtime": 0.000132003, "sum": 1.924216441 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 328266, "msgr_recv_messages": 952, "msgr_running_fast_dispatch_time": 0.027282911, "msgr_running_recv_time": 0.038788621, "msgr_running_send_time": 0.036915233, "msgr_running_total_time": 0.106979786, "msgr_send_bytes": 626627, "msgr_send_messages": 949, "msgr_send_messages_queue_lat": { "avgcount": 949, "avgtime": 0.000154236, "sum": 0.146370231 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3119, "avgtime": 0.001184272, "sum": 3.693745184 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3571, "avgtime": 9.3998e-05, "sum": 0.33566993 }, "queue_len": 0 }, "mds": { "caps": 226, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4813, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 11, "handle_client_caps": 4158, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 236, "inodes_bottom": 236, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 236, "inodes_top": 0, "inodes_with_caps": 226, "load_cent": 250689, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 225, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1376, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 2, "evadd": 9023, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1542, "avgtime": 0.011907578, "sum": 18.361485643 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21351759 }, "mds_mem": { "cap": 226, "cap+": 1602, "cap-": 1376, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 238, "ino+": 1614, "ino-": 1376, "rss": 73752 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1236, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 4593623, "buffer_anon_items": 1424, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 799053, "mds_co_items": 14046, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 15, "omap_rd": 28, "omap_wr": 194, "op": 5199, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 35, "op_reply": 5199, "op_resend": 0, "op_rmw": 0, "op_send": 5199, "op_send_bytes": 18373531, "op_w": 5164, "oplen_avg": { "avgcount": 5199, "sum": 5807 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3263, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1551, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1376, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12752, "get_or_fail_fail": 0, "get_or_fail_success": 12752, "get_started": 0, "get_sum": 2720916, "max": 104857600, "put": 12752, "put_sum": 2720916, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1923, "put_sum": 18514401, "take": 5199, "take_sum": 18514401, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5199, "put_sum": 5199, "take": 5199, "take_sum": 5199, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1376, "get_or_fail_fail": 0, "get_or_fail_success": 1376, "get_started": 0, "get_sum": 138984, "max": 3758096384, "put": 9, "put_sum": 138984, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9023, "get_or_fail_fail": 0, "get_or_fail_success": 9023, "get_started": 0, "get_sum": 17155345, "max": 3758096384, "put": 1542, "put_sum": 17155345, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:36.791 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/225, 1376/1601 2020-11-20T12:17:37.792 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:38.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.041+0000 7f120088c700 1 Processor -- start 2020-11-20T12:17:38.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.041+0000 7f120088c700 1 -- start start 2020-11-20T12:17:38.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.042+0000 7f120088c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f11fc143060 0x7f11fc065eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.042+0000 7f120088c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0b0fa0 con 0x7f11fc065ad0 2020-11-20T12:17:38.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.042+0000 7f120088c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0b0380 con 0x7f11fc0b16a0 2020-11-20T12:17:38.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.042+0000 7f120088c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc149ac0 con 0x7f11fc143060 2020-11-20T12:17:38.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.043+0000 7f11fa59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f11fc0b16a0 0x7f11fc142950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47818/0 (socket says 172.21.15.187:47818) 2020-11-20T12:17:38.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.043+0000 7f11fa59c700 1 -- 172.21.15.187:0/1894886131 learned_addr learned my addr 172.21.15.187:0/1894886131 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:38.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.043+0000 7f11fb59e700 1 --2- 172.21.15.187:0/1894886131 >> v2:172.21.15.5:3300/0 conn(0x7f11fc143060 0x7f11fc065eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128150683 0 0) 0x7f11fc0b0380 con 0x7f11fc0b16a0 2020-11-20T12:17:38.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e8003240 con 0x7f11fc0b16a0 2020-11-20T12:17:38.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 291609183 0 0) 0x7f11e8003240 con 0x7f11fc0b16a0 2020-11-20T12:17:38.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 >> v2:172.21.15.5:3300/0 conn(0x7f11fc143060 msgr2=0x7f11fc065eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:38.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 --2- 172.21.15.187:0/1894886131 >> v2:172.21.15.5:3300/0 conn(0x7f11fc143060 0x7f11fc065eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 >> v1:172.21.15.5:6789/0 conn(0x7f11fc065ad0 legacy=0x7f11fc0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc149d60 con 0x7f11fc0b16a0 2020-11-20T12:17:38.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f11f00014e0 con 0x7f11fc0b16a0 2020-11-20T12:17:38.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 >> v1:172.21.15.187:6789/0 conn(0x7f11fc0b16a0 legacy=0x7f11fc142950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11e8003730 0x7f11e8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11e8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.044+0000 7f11f9d9b700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11e8004e50 0x7f11e800d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc149ac0 con 0x7f11e8004e50 2020-11-20T12:17:38.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8004260 con 0x7f11e80044e0 2020-11-20T12:17:38.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e800de30 con 0x7f11e8003730 2020-11-20T12:17:38.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fa59c700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11e8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fad9d700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11e8003730 0x7f11e8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fb59e700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11e8004e50 0x7f11e800d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fa59c700 1 -- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11e8003730 msgr2=0x7f11e8003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:38.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fa59c700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11e8003730 0x7f11e8003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fa59c700 1 -- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11e8004e50 msgr2=0x7f11e800d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:38.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fa59c700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11e8004e50 0x7f11e800d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fa59c700 1 -- 172.21.15.187:0/1894886131 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e8003240 con 0x7f11e80044e0 2020-11-20T12:17:38.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fad9d700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11e8003730 0x7f11e8003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:38.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.045+0000 7f11fb59e700 1 -- 172.21.15.187:0/1894886131 reap_dead start 2020-11-20T12:17:38.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.046+0000 7f11fa59c700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11e8004960 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f11f000c440 tx=0x7f11fc0631b0).ready entity=mon.1 client_cookie=23d4e5bf4d1bce43 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.046+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f00072e0 con 0x7f11e80044e0 2020-11-20T12:17:38.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.046+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f0007440 con 0x7f11e80044e0 2020-11-20T12:17:38.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.046+0000 7f11f9d9b700 1 -- 172.21.15.187:0/1894886131 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0005820 con 0x7f11e80044e0 2020-11-20T12:17:38.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.047+0000 7f120088c700 1 -- 172.21.15.187:0/1894886131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 msgr2=0x7f11e8004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.047+0000 7f120088c700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11e8004960 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f11f000c440 tx=0x7f11fc0631b0).stop 2020-11-20T12:17:38.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.047+0000 7f120088c700 1 -- 172.21.15.187:0/1894886131 shutdown_connections 2020-11-20T12:17:38.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.047+0000 7f120088c700 1 --2- 172.21.15.187:0/1894886131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11e8004960 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.047+0000 7f120088c700 1 -- 172.21.15.187:0/1894886131 >> 172.21.15.187:0/1894886131 conn(0x7f11fc0b8280 msgr2=0x7f11fc0653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:38.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.047+0000 7f120088c700 1 -- 172.21.15.187:0/1894886131 shutdown_connections 2020-11-20T12:17:38.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.047+0000 7f120088c700 1 -- 172.21.15.187:0/1894886131 wait complete. 2020-11-20T12:17:38.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.048+0000 7f120088c700 1 Processor -- start 2020-11-20T12:17:38.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.048+0000 7f120088c700 1 -- start start 2020-11-20T12:17:38.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.049+0000 7f120088c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11fc09c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.049+0000 7f120088c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11fc0b28a0 0x7f11fc09c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.049+0000 7f120088c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11fc154fa0 0x7f11fc155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.049+0000 7f120088c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc09cad0 con 0x7f11fc0b28a0 2020-11-20T12:17:38.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.049+0000 7f120088c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc09fee0 con 0x7f11e80044e0 2020-11-20T12:17:38.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.049+0000 7f120088c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0a0140 con 0x7f11fc154fa0 2020-11-20T12:17:38.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.049+0000 7f11fad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11fc09c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.050+0000 7f11fad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11fc09c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51222/0 (socket says 172.21.15.187:51222) 2020-11-20T12:17:38.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.050+0000 7f11fad9d700 1 -- 172.21.15.187:0/2728032182 learned_addr learned my addr 172.21.15.187:0/2728032182 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:38.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.050+0000 7f11fad9d700 1 -- 172.21.15.187:0/2728032182 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11fc154fa0 msgr2=0x7f11fc155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.050+0000 7f11fad9d700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11fc154fa0 0x7f11fc155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:38.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.050+0000 7f11fad9d700 1 -- 172.21.15.187:0/2728032182 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11fc0b28a0 msgr2=0x7f11fc09c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.050+0000 7f11fad9d700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11fc0b28a0 0x7f11fc09c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:38.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.050+0000 7f11fad9d700 1 -- 172.21.15.187:0/2728032182 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0a03e0 con 0x7f11e80044e0 2020-11-20T12:17:38.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.050+0000 7f11fad9d700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11fc09c0f0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f11ec00d780 tx=0x7f11ec009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.051+0000 7f11e2ffd700 1 -- 172.21.15.187:0/2728032182 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec00f5d0 con 0x7f11e80044e0 2020-11-20T12:17:38.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.051+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc155910 con 0x7f11e80044e0 2020-11-20T12:17:38.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.051+0000 7f11e2ffd700 1 -- 172.21.15.187:0/2728032182 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec00f730 con 0x7f11e80044e0 2020-11-20T12:17:38.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.051+0000 7f11e2ffd700 1 -- 172.21.15.187:0/2728032182 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec00fab0 con 0x7f11e80044e0 2020-11-20T12:17:38.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.051+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc15b930 con 0x7f11e80044e0 2020-11-20T12:17:38.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.052+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc068ef0 con 0x7f11e80044e0 2020-11-20T12:17:38.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.052+0000 7f11e2ffd700 1 -- 172.21.15.187:0/2728032182 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f11ec01c460 con 0x7f11e80044e0 2020-11-20T12:17:38.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.053+0000 7f11e2ffd700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f11d8056d50 0x7f11d8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.053+0000 7f11e2ffd700 1 -- 172.21.15.187:0/2728032182 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f11ec0701b0 con 0x7f11e80044e0 2020-11-20T12:17:38.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.053+0000 7f11fa59c700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f11d8056d50 0x7f11d8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.054+0000 7f11fa59c700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f11d8056d50 0x7f11d8059210 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f11fc09ecc0 tx=0x7f11f0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.057+0000 7f11e2ffd700 1 -- 172.21.15.187:0/2728032182 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f11ec045440 con 0x7f11e80044e0 2020-11-20T12:17:38.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.302+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11fc044f60 con 0x7f11e80044e0 2020-11-20T12:17:38.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.303+0000 7f11e2ffd700 1 -- 172.21.15.187:0/2728032182 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f11ec007c20 con 0x7f11e80044e0 2020-11-20T12:17:38.307 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:38.307 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:38.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f11d8056d50 msgr2=0x7f11d8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f11d8056d50 0x7f11d8059210 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f11fc09ecc0 tx=0x7f11f0015000).stop 2020-11-20T12:17:38.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 msgr2=0x7f11fc09c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11fc09c0f0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f11ec00d780 tx=0x7f11ec009040).stop 2020-11-20T12:17:38.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 shutdown_connections 2020-11-20T12:17:38.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f11d8056d50 0x7f11d8059210 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f11fc154fa0 0x7f11fc155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:38.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11e80044e0 0x7f11fc09c0f0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.305+0000 7f120088c700 1 --2- 172.21.15.187:0/2728032182 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11fc0b28a0 0x7f11fc09c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:38.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.306+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 >> 172.21.15.187:0/2728032182 conn(0x7f11fc0b8280 msgr2=0x7f11fc062050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:38.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.306+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 shutdown_connections 2020-11-20T12:17:38.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:38.306+0000 7f120088c700 1 -- 172.21.15.187:0/2728032182 wait complete. 2020-11-20T12:17:38.313 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:38.331 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:38.661 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2559,"msgr_send_messages":2562,"msgr_recv_bytes":797770,"msgr_send_bytes":9860771,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.437810066,"msgr_running_send_time":0.180556233,"msgr_running_recv_time":0.097306754,"msgr_running_fast_dispatch_time":0.123942413,"msgr_send_messages_queue_lat":{"avgcount":2560,"sum":0.454421844,"avgtime":0.000177508},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9252,"msgr_send_messages":14580,"msgr_recv_bytes":2627324,"msgr_send_bytes":14200184,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.015182413,"msgr_running_send_time":2.238665960,"msgr_running_recv_time":0.936311954,"msgr_running_fast_dispatch_time":0.091916976,"msgr_send_messages_queue_lat":{"avgcount":14579,"sum":1.924403590,"avgtime":0.000131998},"msgr_handle_ack_lat":{"avgcount":8615,"sum":0.006420846,"avgtime":0.000000745}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":953,"msgr_send_messages":951,"msgr_recv_bytes":328538,"msgr_send_bytes":627631,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.107335901,"msgr_running_send_time":0.037106739,"msgr_running_recv_time":0.038853104,"msgr_running_fast_dispatch_time":0.027339416,"msgr_send_messages_queue_lat":{"avgcount":951,"sum":0.146964640,"avgtime":0.000154536},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3119,"sum":3.693745184,"avgtime":0.001184272}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3571,"sum":0.335669930,"avgtime":0.000093998}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":235,"inodes_top":0,"inodes_bottom":235,"inodes_pin_tail":0,"inodes_pinned":235,"inodes_expired":0,"inodes_with_caps":226,"caps":226,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":6190,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4813,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4162,"handle_client_caps_dirty":1500,"handle_client_cap_release":11,"process_request_cap_release":1668},"mds_cache":{"num_strays":225,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1376,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9026,"evex":9025,"evtrm":9025,"ev":5,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21358161,"rdpos":4195632,"jlat":{"avgcount":1542,"sum":18.361485643,"avgtime":0.011907578},"replayed":4},"mds_mem":{"ino":238,"ino+":1614,"ino-":1376,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":226,"cap+":1602,"cap-":1376,"rss":73752,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1241},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":4101589,"buffer_anon_items":1300,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":800948,"mds_co_items":14071,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":2,"op_laggy":0,"op_send":5201,"op_send_bytes":18380024,"op_resend":0,"op_reply":5199,"oplen_avg":{"avgcount":5201,"sum":5810},"op":5201,"op_r":35,"op_w":5166,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":17,"osdop_write":1552,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3263,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":195,"omap_rd":28,"omap_del":16},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1376,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":236,"max":104857600,"get_started":0,"get":12760,"get_sum":2722270,"get_or_fail_fail":0,"get_or_fail_success":12760,"take":0,"take_sum":0,"put":12759,"put_sum":2722034,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":6493,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5201,"take_sum":18520894,"put":1923,"put_sum":18514401,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":2,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5201,"take_sum":5201,"put":5199,"put_sum":5199,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1376,"get_sum":138984,"get_or_fail_fail":0,"get_or_fail_success":1376,"take":0,"take_sum":0,"put":9,"put_sum":138984,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":0,"max":3758096384,"get_started":0,"get":9026,"get_sum":17161747,"get_or_fail_fail":0,"get_or_fail_success":9026,"take":0,"take_sum":0,"put":1543,"put_sum":17161747,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:38.663 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 797770, "msgr_recv_messages": 2559, "msgr_running_fast_dispatch_time": 0.123942413, "msgr_running_recv_time": 0.097306754, "msgr_running_send_time": 0.180556233, "msgr_running_total_time": 0.437810066, "msgr_send_bytes": 9860771, "msgr_send_messages": 2562, "msgr_send_messages_queue_lat": { "avgcount": 2560, "avgtime": 0.000177508, "sum": 0.454421844 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8615, "avgtime": 7.45e-07, "sum": 0.006420846 }, "msgr_recv_bytes": 2627324, "msgr_recv_messages": 9252, "msgr_running_fast_dispatch_time": 0.091916976, "msgr_running_recv_time": 0.936311954, "msgr_running_send_time": 2.23866596, "msgr_running_total_time": 3.015182413, "msgr_send_bytes": 14200184, "msgr_send_messages": 14580, "msgr_send_messages_queue_lat": { "avgcount": 14579, "avgtime": 0.000131998, "sum": 1.92440359 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 328538, "msgr_recv_messages": 953, "msgr_running_fast_dispatch_time": 0.027339416, "msgr_running_recv_time": 0.038853104, "msgr_running_send_time": 0.037106739, "msgr_running_total_time": 0.107335901, "msgr_send_bytes": 627631, "msgr_send_messages": 951, "msgr_send_messages_queue_lat": { "avgcount": 951, "avgtime": 0.000154536, "sum": 0.14696464 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3119, "avgtime": 0.001184272, "sum": 3.693745184 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3571, "avgtime": 9.3998e-05, "sum": 0.33566993 }, "queue_len": 0 }, "mds": { "caps": 226, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4813, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 11, "handle_client_caps": 4162, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 235, "inodes_bottom": 235, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 235, "inodes_top": 0, "inodes_with_caps": 226, "load_cent": 6190, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 225, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1376, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 5, "evadd": 9026, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1542, "avgtime": 0.011907578, "sum": 18.361485643 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21358161 }, "mds_mem": { "cap": 226, "cap+": 1602, "cap-": 1376, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 238, "ino+": 1614, "ino-": 1376, "rss": 73752 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1241, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 4101589, "buffer_anon_items": 1300, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 800948, "mds_co_items": 14071, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 16, "omap_rd": 28, "omap_wr": 195, "op": 5201, "op_active": 2, "op_laggy": 0, "op_pg": 0, "op_r": 35, "op_reply": 5199, "op_resend": 0, "op_rmw": 0, "op_send": 5201, "op_send_bytes": 18380024, "op_w": 5166, "oplen_avg": { "avgcount": 5201, "sum": 5810 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3263, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1552, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1376, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12760, "get_or_fail_fail": 0, "get_or_fail_success": 12760, "get_started": 0, "get_sum": 2722270, "max": 104857600, "put": 12759, "put_sum": 2722034, "take": 0, "take_sum": 0, "val": 236, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1923, "put_sum": 18514401, "take": 5201, "take_sum": 18520894, "val": 6493, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5199, "put_sum": 5199, "take": 5201, "take_sum": 5201, "val": 2, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1376, "get_or_fail_fail": 0, "get_or_fail_success": 1376, "get_started": 0, "get_sum": 138984, "max": 3758096384, "put": 9, "put_sum": 138984, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9026, "get_or_fail_fail": 0, "get_or_fail_success": 9026, "get_started": 0, "get_sum": 17161747, "max": 3758096384, "put": 1543, "put_sum": 17161747, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:38.664 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/225, 1376/1601 2020-11-20T12:17:39.665 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:39.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.909+0000 7fdced195700 1 Processor -- start 2020-11-20T12:17:39.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.909+0000 7fdced195700 1 -- start start 2020-11-20T12:17:39.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.911+0000 7fdced195700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdce80bc460 0x7fdce814c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.911+0000 7fdced195700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce80bc840 0x7fdce81485e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.911+0000 7fdced195700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce814cc10 con 0x7fdce80b96f0 2020-11-20T12:17:39.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.911+0000 7fdced195700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce814c9f0 con 0x7fdce80bc460 2020-11-20T12:17:39.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.911+0000 7fdced195700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce814ceb0 con 0x7fdce80bc840 2020-11-20T12:17:39.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.911+0000 7fdcec994700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fdce80b96f0 0x7fdce80b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47840/0 (socket says 172.21.15.187:47840) 2020-11-20T12:17:39.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.911+0000 7fdcec994700 1 -- 172.21.15.187:0/2384494982 learned_addr learned my addr 172.21.15.187:0/2384494982 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:39.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.912+0000 7fdce7fff700 1 --2- 172.21.15.187:0/2384494982 >> v2:172.21.15.5:3300/0 conn(0x7fdce80bc460 0x7fdce814c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:39.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.912+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881142220 0 0) 0x7fdce814cc10 con 0x7fdce80b96f0 2020-11-20T12:17:39.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.912+0000 7fdce77fe700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce80bc840 0x7fdce81485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:39.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.912+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd4003240 con 0x7fdce80b96f0 2020-11-20T12:17:39.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.912+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4195971703 0 0) 0x7fdcd4003240 con 0x7fdce80b96f0 2020-11-20T12:17:39.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.912+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce80bc840 msgr2=0x7fdce81485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:39.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.912+0000 7fdce6ffd700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce80bc840 0x7fdce81485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:39.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 >> v2:172.21.15.5:3300/0 conn(0x7fdce80bc460 msgr2=0x7fdce814c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:39.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 --2- 172.21.15.187:0/2384494982 >> v2:172.21.15.5:3300/0 conn(0x7fdce80bc460 0x7fdce814c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:39.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce814d150 con 0x7fdce80b96f0 2020-11-20T12:17:39.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fdce0001860 con 0x7fdce80b96f0 2020-11-20T12:17:39.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 >> v1:172.21.15.187:6789/0 conn(0x7fdce80b96f0 legacy=0x7fdce80b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:39.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdcd4003730 0x7fdcd4003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd4004020 0x7fdcd4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdcd4004b00 0x7fdcd4004f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce814c9f0 con 0x7fdcd4004b00 2020-11-20T12:17:39.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce814ceb0 con 0x7fdcd4004020 2020-11-20T12:17:39.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd400b880 con 0x7fdcd4003730 2020-11-20T12:17:39.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce77fe700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd4004020 0x7fdcd4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:39.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce77fe700 1 -- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdcd4003730 msgr2=0x7fdcd4003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:39.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce77fe700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdcd4003730 0x7fdcd4003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:39.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce77fe700 1 -- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdcd4004b00 msgr2=0x7fdcd4004f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:39.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce77fe700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdcd4004b00 0x7fdcd4004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:39.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.913+0000 7fdce77fe700 1 -- 172.21.15.187:0/2384494982 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4003240 con 0x7fdcd4004020 2020-11-20T12:17:39.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdce77fe700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd4004020 0x7fdcd4004500 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7fdcdc00bc00 tx=0x7fdcdc006df0).ready entity=mon.1 client_cookie=3219d707b481b55b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:39.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc00b310 con 0x7fdcd4004020 2020-11-20T12:17:39.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcdc00eb40 con 0x7fdcd4004020 2020-11-20T12:17:39.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdce6ffd700 1 -- 172.21.15.187:0/2384494982 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc00eec0 con 0x7fdcd4004020 2020-11-20T12:17:39.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdced195700 1 -- 172.21.15.187:0/2384494982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd4004020 msgr2=0x7fdcd4004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:39.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdced195700 1 --2- 172.21.15.187:0/2384494982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd4004020 0x7fdcd4004500 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7fdcdc00bc00 tx=0x7fdcdc006df0).stop 2020-11-20T12:17:39.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdcec994700 1 -- 172.21.15.187:0/2384494982 reap_dead start 2020-11-20T12:17:39.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdced195700 1 -- 172.21.15.187:0/2384494982 shutdown_connections 2020-11-20T12:17:39.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.914+0000 7fdced195700 1 -- 172.21.15.187:0/2384494982 >> 172.21.15.187:0/2384494982 conn(0x7fdce80bfb90 msgr2=0x7fdce80bbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:39.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.915+0000 7fdced195700 1 -- 172.21.15.187:0/2384494982 shutdown_connections 2020-11-20T12:17:39.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.915+0000 7fdced195700 1 -- 172.21.15.187:0/2384494982 wait complete. 2020-11-20T12:17:39.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.916+0000 7fdced195700 1 Processor -- start 2020-11-20T12:17:39.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.916+0000 7fdced195700 1 -- start start 2020-11-20T12:17:39.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdced195700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce8061430 0x7fdce815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdced195700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdce8061810 0x7fdce8158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdced195700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdce80b88e0 0x7fdce8147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdced195700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce813bb80 con 0x7fdce80b88e0 2020-11-20T12:17:39.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdced195700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce815c1d0 con 0x7fdce8061810 2020-11-20T12:17:39.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdced195700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8158010 con 0x7fdce8061430 2020-11-20T12:17:39.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdce77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdce8061810 0x7fdce8158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:39.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdce77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdce8061810 0x7fdce8158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51244/0 (socket says 172.21.15.187:51244) 2020-11-20T12:17:39.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.917+0000 7fdce77fe700 1 -- 172.21.15.187:0/1744766313 learned_addr learned my addr 172.21.15.187:0/1744766313 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:39.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdce7fff700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce8061430 0x7fdce815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:39.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdcec994700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdce80b88e0 0x7fdce8147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:39.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdce77fe700 1 -- 172.21.15.187:0/1744766313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce8061430 msgr2=0x7fdce815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:39.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdce77fe700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce8061430 0x7fdce815bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:39.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdce77fe700 1 -- 172.21.15.187:0/1744766313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdce80b88e0 msgr2=0x7fdce8147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:39.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdce77fe700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdce80b88e0 0x7fdce8147250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:39.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdce77fe700 1 -- 172.21.15.187:0/1744766313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce81582b0 con 0x7fdce8061810 2020-11-20T12:17:39.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdce77fe700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdce8061810 0x7fdce8158750 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7fdcd400b850 tx=0x7fdcdc019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:39.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdcc7fff700 1 -- 172.21.15.187:0/1744766313 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc01d620 con 0x7fdce8061810 2020-11-20T12:17:39.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdcc7fff700 1 -- 172.21.15.187:0/1744766313 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcdc021070 con 0x7fdce8061810 2020-11-20T12:17:39.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce8158c40 con 0x7fdce8061810 2020-11-20T12:17:39.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdcc7fff700 1 -- 172.21.15.187:0/1744766313 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc01d780 con 0x7fdce8061810 2020-11-20T12:17:39.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.918+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce8147740 con 0x7fdce8061810 2020-11-20T12:17:39.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.920+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce8068630 con 0x7fdce8061810 2020-11-20T12:17:39.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.920+0000 7fdcc7fff700 1 -- 172.21.15.187:0/1744766313 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdcdc02d070 con 0x7fdce8061810 2020-11-20T12:17:39.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.920+0000 7fdcc7fff700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdcc0056d50 0x7fdcc0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:39.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.920+0000 7fdcc7fff700 1 -- 172.21.15.187:0/1744766313 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fdcdc072380 con 0x7fdce8061810 2020-11-20T12:17:39.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.921+0000 7fdce7fff700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdcc0056d50 0x7fdcc0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:39.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.921+0000 7fdce7fff700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdcc0056d50 0x7fdcc0059210 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fdcd8006a30 tx=0x7fdcd8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:39.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:39.924+0000 7fdcc7fff700 1 -- 172.21.15.187:0/1744766313 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdcdc047800 con 0x7fdce8061810 2020-11-20T12:17:40.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.147+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdce8080210 con 0x7fdce8061810 2020-11-20T12:17:40.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.148+0000 7fdcc7fff700 1 -- 172.21.15.187:0/1744766313 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7fdcdc04ad20 con 0x7fdce8061810 2020-11-20T12:17:40.152 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:40.152 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:40.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdcc0056d50 msgr2=0x7fdcc0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:40.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdcc0056d50 0x7fdcc0059210 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fdcd8006a30 tx=0x7fdcd8008040).stop 2020-11-20T12:17:40.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdce8061810 msgr2=0x7fdce8158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:40.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdce8061810 0x7fdce8158750 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7fdcd400b850 tx=0x7fdcdc019000).stop 2020-11-20T12:17:40.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 shutdown_connections 2020-11-20T12:17:40.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdcc0056d50 0x7fdcc0059210 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdce80b88e0 0x7fdce8147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdce8061430 0x7fdce815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 --2- 172.21.15.187:0/1744766313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdce8061810 0x7fdce8158750 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.150+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 >> 172.21.15.187:0/1744766313 conn(0x7fdce80bfb90 msgr2=0x7fdce80a1030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:40.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.151+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 shutdown_connections 2020-11-20T12:17:40.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:40.151+0000 7fdced195700 1 -- 172.21.15.187:0/1744766313 wait complete. 2020-11-20T12:17:40.157 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:40.178 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:40.442 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2566,"msgr_send_messages":2568,"msgr_recv_bytes":799811,"msgr_send_bytes":9862956,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.440646734,"msgr_running_send_time":0.181165219,"msgr_running_recv_time":0.097877679,"msgr_running_fast_dispatch_time":0.125481083,"msgr_send_messages_queue_lat":{"avgcount":2566,"sum":0.454830685,"avgtime":0.000177252},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9317,"msgr_send_messages":14585,"msgr_recv_bytes":2647112,"msgr_send_bytes":14201885,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.022123491,"msgr_running_send_time":2.240843171,"msgr_running_recv_time":0.943309025,"msgr_running_fast_dispatch_time":0.092123353,"msgr_send_messages_queue_lat":{"avgcount":14584,"sum":1.925166479,"avgtime":0.000132005},"msgr_handle_ack_lat":{"avgcount":8617,"sum":0.006474858,"avgtime":0.000000751}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":955,"msgr_send_messages":952,"msgr_recv_bytes":329084,"msgr_send_bytes":627950,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.107688788,"msgr_running_send_time":0.037191021,"msgr_running_recv_time":0.038958367,"msgr_running_fast_dispatch_time":0.027472876,"msgr_send_messages_queue_lat":{"avgcount":952,"sum":0.147048319,"avgtime":0.000154462},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3122,"sum":3.694791784,"avgtime":0.001183469}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3585,"sum":0.336993806,"avgtime":0.000094001}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":235,"inodes_top":0,"inodes_bottom":235,"inodes_pin_tail":0,"inodes_pinned":235,"inodes_expired":0,"inodes_with_caps":226,"caps":226,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":6190,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4816,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4223,"handle_client_caps_dirty":1500,"handle_client_cap_release":12,"process_request_cap_release":1668},"mds_cache":{"num_strays":222,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1379,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9091,"evex":9025,"evtrm":9025,"ev":70,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21480882,"rdpos":4195632,"jlat":{"avgcount":1543,"sum":18.386338098,"avgtime":0.011915967},"replayed":4},"mds_mem":{"ino":238,"ino+":1614,"ino-":1376,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":222,"cap+":1602,"cap-":1380,"rss":73752,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1241},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":4101589,"buffer_anon_items":1300,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":800948,"mds_co_items":14071,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5210,"op_send_bytes":18380417,"op_resend":0,"op_reply":5210,"oplen_avg":{"avgcount":5210,"sum":5819},"op":5210,"op_r":37,"op_w":5173,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":19,"osdop_write":1554,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3267,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":195,"omap_rd":28,"omap_del":16},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1379,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12834,"get_sum":2738993,"get_or_fail_fail":0,"get_or_fail_success":12834,"take":0,"take_sum":0,"put":12834,"put_sum":2738993,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5210,"take_sum":18521590,"put":1930,"put_sum":18521590,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5210,"take_sum":5210,"put":5210,"put_sum":5210,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1379,"get_sum":139287,"get_or_fail_fail":0,"get_or_fail_success":1379,"take":0,"take_sum":0,"put":11,"put_sum":139287,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":122721,"max":3758096384,"get_started":0,"get":9091,"get_sum":17284468,"get_or_fail_fail":0,"get_or_fail_success":9091,"take":0,"take_sum":0,"put":1543,"put_sum":17161747,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:40.444 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 799811, "msgr_recv_messages": 2566, "msgr_running_fast_dispatch_time": 0.125481083, "msgr_running_recv_time": 0.097877679, "msgr_running_send_time": 0.181165219, "msgr_running_total_time": 0.440646734, "msgr_send_bytes": 9862956, "msgr_send_messages": 2568, "msgr_send_messages_queue_lat": { "avgcount": 2566, "avgtime": 0.000177252, "sum": 0.454830685 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8617, "avgtime": 7.51e-07, "sum": 0.006474858 }, "msgr_recv_bytes": 2647112, "msgr_recv_messages": 9317, "msgr_running_fast_dispatch_time": 0.092123353, "msgr_running_recv_time": 0.943309025, "msgr_running_send_time": 2.240843171, "msgr_running_total_time": 3.022123491, "msgr_send_bytes": 14201885, "msgr_send_messages": 14585, "msgr_send_messages_queue_lat": { "avgcount": 14584, "avgtime": 0.000132005, "sum": 1.925166479 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 329084, "msgr_recv_messages": 955, "msgr_running_fast_dispatch_time": 0.027472876, "msgr_running_recv_time": 0.038958367, "msgr_running_send_time": 0.037191021, "msgr_running_total_time": 0.107688788, "msgr_send_bytes": 627950, "msgr_send_messages": 952, "msgr_send_messages_queue_lat": { "avgcount": 952, "avgtime": 0.000154462, "sum": 0.147048319 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3122, "avgtime": 0.001183469, "sum": 3.694791784 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3585, "avgtime": 9.4001e-05, "sum": 0.336993806 }, "queue_len": 0 }, "mds": { "caps": 226, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4816, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 12, "handle_client_caps": 4223, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 235, "inodes_bottom": 235, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 235, "inodes_top": 0, "inodes_with_caps": 226, "load_cent": 6190, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 222, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1379, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 70, "evadd": 9091, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1543, "avgtime": 0.011915967, "sum": 18.386338098 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21480882 }, "mds_mem": { "cap": 222, "cap+": 1602, "cap-": 1380, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 238, "ino+": 1614, "ino-": 1376, "rss": 73752 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1241, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 4101589, "buffer_anon_items": 1300, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 800948, "mds_co_items": 14071, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 16, "omap_rd": 28, "omap_wr": 195, "op": 5210, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 37, "op_reply": 5210, "op_resend": 0, "op_rmw": 0, "op_send": 5210, "op_send_bytes": 18380417, "op_w": 5173, "oplen_avg": { "avgcount": 5210, "sum": 5819 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3267, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 19, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1554, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1379, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12834, "get_or_fail_fail": 0, "get_or_fail_success": 12834, "get_started": 0, "get_sum": 2738993, "max": 104857600, "put": 12834, "put_sum": 2738993, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1930, "put_sum": 18521590, "take": 5210, "take_sum": 18521590, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5210, "put_sum": 5210, "take": 5210, "take_sum": 5210, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1379, "get_or_fail_fail": 0, "get_or_fail_success": 1379, "get_started": 0, "get_sum": 139287, "max": 3758096384, "put": 11, "put_sum": 139287, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9091, "get_or_fail_fail": 0, "get_or_fail_success": 9091, "get_started": 0, "get_sum": 17284468, "max": 3758096384, "put": 1543, "put_sum": 17161747, "take": 0, "take_sum": 0, "val": 122721, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:40.444 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/222, 1379/1601 2020-11-20T12:17:41.446 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:41.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.696+0000 7f11899ca700 1 Processor -- start 2020-11-20T12:17:41.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.696+0000 7f11899ca700 1 -- start start 2020-11-20T12:17:41.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.697+0000 7f11899ca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 0x7f1184065eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:41.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.697+0000 7f11899ca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f11840b16a0 0x7f11840afc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:41.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.697+0000 7f11899ca700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11840b0fa0 con 0x7f1184142950 2020-11-20T12:17:41.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.697+0000 7f11899ca700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1184149ac0 con 0x7f11840b16a0 2020-11-20T12:17:41.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.697+0000 7f11899ca700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1184149d60 con 0x7f1184065ad0 2020-11-20T12:17:41.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.698+0000 7f1183fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f11840b16a0 0x7f11840afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:41.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.698+0000 7f1183fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f11840b16a0 0x7f11840afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51252/0 (socket says 172.21.15.187:51252) 2020-11-20T12:17:41.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.698+0000 7f1183fff700 1 -- 172.21.15.187:0/2710884212 learned_addr learned my addr 172.21.15.187:0/2710884212 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:41.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.698+0000 7f11889c8700 1 --2- 172.21.15.187:0/2710884212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 0x7f1184065eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:41.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.698+0000 7f1183fff700 1 -- 172.21.15.187:0/2710884212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 msgr2=0x7f1184065eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:41.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.698+0000 7f1183fff700 1 --2- 172.21.15.187:0/2710884212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 0x7f1184065eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.698+0000 7f1183fff700 1 -- 172.21.15.187:0/2710884212 >> v1:172.21.15.5:6789/0 conn(0x7f1184142950 legacy=0x7f1184145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:41.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.698+0000 7f1183fff700 1 -- 172.21.15.187:0/2710884212 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11841459c0 con 0x7f11840b16a0 2020-11-20T12:17:41.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.699+0000 7f11889c8700 1 --2- 172.21.15.187:0/2710884212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 0x7f1184065eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:41.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.699+0000 7f1183fff700 1 --2- 172.21.15.187:0/2710884212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840afc70 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7f1174006fd0 tx=0x7f1174006eb0).ready entity=mon.1 client_cookie=32b7fa054e1fcd6a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:41.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.699+0000 7f11837fe700 1 -- 172.21.15.187:0/2710884212 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174009920 con 0x7f11840b16a0 2020-11-20T12:17:41.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.699+0000 7f11837fe700 1 -- 172.21.15.187:0/2710884212 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174009a80 con 0x7f11840b16a0 2020-11-20T12:17:41.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.699+0000 7f11837fe700 1 -- 172.21.15.187:0/2710884212 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174009e00 con 0x7f11840b16a0 2020-11-20T12:17:41.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.700+0000 7f11899ca700 1 -- 172.21.15.187:0/2710884212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 msgr2=0x7f11840afc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:41.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.700+0000 7f11899ca700 1 --2- 172.21.15.187:0/2710884212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840afc70 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7f1174006fd0 tx=0x7f1174006eb0).stop 2020-11-20T12:17:41.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.700+0000 7f11899ca700 1 -- 172.21.15.187:0/2710884212 shutdown_connections 2020-11-20T12:17:41.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.700+0000 7f11899ca700 1 --2- 172.21.15.187:0/2710884212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 0x7f1184065eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.700+0000 7f11899ca700 1 --2- 172.21.15.187:0/2710884212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840afc70 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.700+0000 7f11899ca700 1 -- 172.21.15.187:0/2710884212 >> 172.21.15.187:0/2710884212 conn(0x7f11840b8280 msgr2=0x7f11840653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:41.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.700+0000 7f11899ca700 1 -- 172.21.15.187:0/2710884212 shutdown_connections 2020-11-20T12:17:41.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.700+0000 7f11899ca700 1 -- 172.21.15.187:0/2710884212 wait complete. 2020-11-20T12:17:41.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.701+0000 7f11899ca700 1 Processor -- start 2020-11-20T12:17:41.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.702+0000 7f11899ca700 1 -- start start 2020-11-20T12:17:41.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.702+0000 7f11899ca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 0x7f1184158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:41.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.702+0000 7f11899ca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:41.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.702+0000 7f11899ca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1184142950 0x7f118409e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:41.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f11899ca700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11841477c0 con 0x7f1184142950 2020-11-20T12:17:41.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f11899ca700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118409de40 con 0x7f11840b16a0 2020-11-20T12:17:41.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f11899ca700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118409e0e0 con 0x7f1184065ad0 2020-11-20T12:17:41.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f1183fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:41.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f1183fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51258/0 (socket says 172.21.15.187:51258) 2020-11-20T12:17:41.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f1183fff700 1 -- 172.21.15.187:0/2903405643 learned_addr learned my addr 172.21.15.187:0/2903405643 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:41.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f1183fff700 1 -- 172.21.15.187:0/2903405643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 msgr2=0x7f1184158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:41.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f1183fff700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 0x7f1184158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:41.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f1183fff700 1 -- 172.21.15.187:0/2903405643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1184142950 msgr2=0x7f118409e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:41.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f1183fff700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1184142950 0x7f118409e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:41.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.703+0000 7f1183fff700 1 -- 172.21.15.187:0/2903405643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11840b3680 con 0x7f11840b16a0 2020-11-20T12:17:41.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.704+0000 7f1183fff700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840b3190 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7f1174006ef0 tx=0x7f1174015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:41.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.704+0000 7f1180ff9700 1 -- 172.21.15.187:0/2903405643 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117401e460 con 0x7f11840b16a0 2020-11-20T12:17:41.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.704+0000 7f1180ff9700 1 -- 172.21.15.187:0/2903405643 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117401e5e0 con 0x7f11840b16a0 2020-11-20T12:17:41.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.704+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118409ea70 con 0x7f11840b16a0 2020-11-20T12:17:41.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.704+0000 7f1180ff9700 1 -- 172.21.15.187:0/2903405643 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117401e9a0 con 0x7f11840b16a0 2020-11-20T12:17:41.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.704+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118409ed40 con 0x7f11840b16a0 2020-11-20T12:17:41.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.705+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1184068ef0 con 0x7f11840b16a0 2020-11-20T12:17:41.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.706+0000 7f1180ff9700 1 -- 172.21.15.187:0/2903405643 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1174025030 con 0x7f11840b16a0 2020-11-20T12:17:41.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.706+0000 7f1180ff9700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1160056d20 0x7f11600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:41.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.707+0000 7f1180ff9700 1 -- 172.21.15.187:0/2903405643 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1174071810 con 0x7f11840b16a0 2020-11-20T12:17:41.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.707+0000 7f11889c8700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1160056d20 0x7f11600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:41.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.708+0000 7f11889c8700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1160056d20 0x7f11600591e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f116c00a7c0 tx=0x7f116c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:41.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.711+0000 7f1180ff9700 1 -- 172.21.15.187:0/2903405643 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1174046cc0 con 0x7f11840b16a0 2020-11-20T12:17:41.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.957+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f118409e2a0 con 0x7f11840b16a0 2020-11-20T12:17:41.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.958+0000 7f1180ff9700 1 -- 172.21.15.187:0/2903405643 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f1174023070 con 0x7f11840b16a0 2020-11-20T12:17:41.963 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:41.964 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:41.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.960+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1160056d20 msgr2=0x7f11600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:41.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.960+0000 7f11899ca700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1160056d20 0x7f11600591e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f116c00a7c0 tx=0x7f116c008040).stop 2020-11-20T12:17:41.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.960+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 msgr2=0x7f11840b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:41.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.960+0000 7f11899ca700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840b3190 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7f1174006ef0 tx=0x7f1174015000).stop 2020-11-20T12:17:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.961+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 shutdown_connections 2020-11-20T12:17:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.961+0000 7f11899ca700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1160056d20 0x7f11600591e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.961+0000 7f11899ca700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1184142950 0x7f118409e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.961+0000 7f11899ca700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1184065ad0 0x7f1184158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.961+0000 7f11899ca700 1 --2- 172.21.15.187:0/2903405643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11840b16a0 0x7f11840b3190 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.961+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 >> 172.21.15.187:0/2903405643 conn(0x7f11840b8280 msgr2=0x7f11840600f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:41.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.961+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 shutdown_connections 2020-11-20T12:17:41.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:41.961+0000 7f11899ca700 1 -- 172.21.15.187:0/2903405643 wait complete. 2020-11-20T12:17:41.969 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:41.990 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:42.262 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2566,"msgr_send_messages":2568,"msgr_recv_bytes":799811,"msgr_send_bytes":9862956,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.440646734,"msgr_running_send_time":0.181165219,"msgr_running_recv_time":0.097877679,"msgr_running_fast_dispatch_time":0.125481083,"msgr_send_messages_queue_lat":{"avgcount":2566,"sum":0.454830685,"avgtime":0.000177252},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9319,"msgr_send_messages":14585,"msgr_recv_bytes":2647512,"msgr_send_bytes":14201885,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.022642330,"msgr_running_send_time":2.241078824,"msgr_running_recv_time":0.943413487,"msgr_running_fast_dispatch_time":0.092293171,"msgr_send_messages_queue_lat":{"avgcount":14584,"sum":1.925166479,"avgtime":0.000132005},"msgr_handle_ack_lat":{"avgcount":8617,"sum":0.006474858,"avgtime":0.000000751}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":956,"msgr_send_messages":953,"msgr_recv_bytes":329356,"msgr_send_bytes":628510,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.108113741,"msgr_running_send_time":0.037392666,"msgr_running_recv_time":0.039056382,"msgr_running_fast_dispatch_time":0.027530062,"msgr_send_messages_queue_lat":{"avgcount":953,"sum":0.147097313,"avgtime":0.000154351},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3122,"sum":3.694791784,"avgtime":0.001183469}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3585,"sum":0.336993806,"avgtime":0.000094001}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":235,"inodes_top":0,"inodes_bottom":235,"inodes_pin_tail":0,"inodes_pinned":235,"inodes_expired":0,"inodes_with_caps":226,"caps":226,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":6190,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4816,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4223,"handle_client_caps_dirty":1500,"handle_client_cap_release":12,"process_request_cap_release":1668},"mds_cache":{"num_strays":222,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1379,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9091,"evex":9025,"evtrm":9025,"ev":70,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21480882,"rdpos":4195632,"jlat":{"avgcount":1543,"sum":18.386338098,"avgtime":0.011915967},"replayed":4},"mds_mem":{"ino":238,"ino+":1614,"ino-":1376,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":222,"cap+":1602,"cap-":1380,"rss":73752,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1241},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":4101589,"buffer_anon_items":1300,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":800948,"mds_co_items":14071,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5210,"op_send_bytes":18380417,"op_resend":0,"op_reply":5210,"oplen_avg":{"avgcount":5210,"sum":5819},"op":5210,"op_r":37,"op_w":5173,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":19,"osdop_write":1554,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3267,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":195,"omap_rd":28,"omap_del":16},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1379,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12837,"get_sum":2739375,"get_or_fail_fail":0,"get_or_fail_success":12837,"take":0,"take_sum":0,"put":12837,"put_sum":2739375,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5210,"take_sum":18521590,"put":1930,"put_sum":18521590,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5210,"take_sum":5210,"put":5210,"put_sum":5210,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1379,"get_sum":139287,"get_or_fail_fail":0,"get_or_fail_success":1379,"take":0,"take_sum":0,"put":11,"put_sum":139287,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":122721,"max":3758096384,"get_started":0,"get":9091,"get_sum":17284468,"get_or_fail_fail":0,"get_or_fail_success":9091,"take":0,"take_sum":0,"put":1543,"put_sum":17161747,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:42.263 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 799811, "msgr_recv_messages": 2566, "msgr_running_fast_dispatch_time": 0.125481083, "msgr_running_recv_time": 0.097877679, "msgr_running_send_time": 0.181165219, "msgr_running_total_time": 0.440646734, "msgr_send_bytes": 9862956, "msgr_send_messages": 2568, "msgr_send_messages_queue_lat": { "avgcount": 2566, "avgtime": 0.000177252, "sum": 0.454830685 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8617, "avgtime": 7.51e-07, "sum": 0.006474858 }, "msgr_recv_bytes": 2647512, "msgr_recv_messages": 9319, "msgr_running_fast_dispatch_time": 0.092293171, "msgr_running_recv_time": 0.943413487, "msgr_running_send_time": 2.241078824, "msgr_running_total_time": 3.02264233, "msgr_send_bytes": 14201885, "msgr_send_messages": 14585, "msgr_send_messages_queue_lat": { "avgcount": 14584, "avgtime": 0.000132005, "sum": 1.925166479 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 329356, "msgr_recv_messages": 956, "msgr_running_fast_dispatch_time": 0.027530062, "msgr_running_recv_time": 0.039056382, "msgr_running_send_time": 0.037392666, "msgr_running_total_time": 0.108113741, "msgr_send_bytes": 628510, "msgr_send_messages": 953, "msgr_send_messages_queue_lat": { "avgcount": 953, "avgtime": 0.000154351, "sum": 0.147097313 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3122, "avgtime": 0.001183469, "sum": 3.694791784 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3585, "avgtime": 9.4001e-05, "sum": 0.336993806 }, "queue_len": 0 }, "mds": { "caps": 226, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4816, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 12, "handle_client_caps": 4223, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 235, "inodes_bottom": 235, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 235, "inodes_top": 0, "inodes_with_caps": 226, "load_cent": 6190, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 222, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1379, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 70, "evadd": 9091, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1543, "avgtime": 0.011915967, "sum": 18.386338098 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21480882 }, "mds_mem": { "cap": 222, "cap+": 1602, "cap-": 1380, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 238, "ino+": 1614, "ino-": 1376, "rss": 73752 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1241, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 4101589, "buffer_anon_items": 1300, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 800948, "mds_co_items": 14071, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 16, "omap_rd": 28, "omap_wr": 195, "op": 5210, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 37, "op_reply": 5210, "op_resend": 0, "op_rmw": 0, "op_send": 5210, "op_send_bytes": 18380417, "op_w": 5173, "oplen_avg": { "avgcount": 5210, "sum": 5819 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3267, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 19, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1554, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1379, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12837, "get_or_fail_fail": 0, "get_or_fail_success": 12837, "get_started": 0, "get_sum": 2739375, "max": 104857600, "put": 12837, "put_sum": 2739375, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1930, "put_sum": 18521590, "take": 5210, "take_sum": 18521590, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5210, "put_sum": 5210, "take": 5210, "take_sum": 5210, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1379, "get_or_fail_fail": 0, "get_or_fail_success": 1379, "get_started": 0, "get_sum": 139287, "max": 3758096384, "put": 11, "put_sum": 139287, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9091, "get_or_fail_fail": 0, "get_or_fail_success": 9091, "get_started": 0, "get_sum": 17284468, "max": 3758096384, "put": 1543, "put_sum": 17161747, "take": 0, "take_sum": 0, "val": 122721, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:42.264 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/222, 1379/1601 2020-11-20T12:17:43.265 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:43.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.512+0000 7fcdd5257700 1 Processor -- start 2020-11-20T12:17:43.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.513+0000 7fcdd5257700 1 -- start start 2020-11-20T12:17:43.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.514+0000 7fcdd5257700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdd0148f90 0x7fcdd0099ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.514+0000 7fcdd5257700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcdd00b00b0 0x7fcdd013f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.514+0000 7fcdd5257700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd0148770 con 0x7fcdd009a1d0 2020-11-20T12:17:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.514+0000 7fcdd5257700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd00b5490 con 0x7fcdd00b00b0 2020-11-20T12:17:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.514+0000 7fcdd5257700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd013f510 con 0x7fcdd0148f90 2020-11-20T12:17:43.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.514+0000 7fcdcffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdd0148f90 0x7fcdd0099ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.515+0000 7fcdcffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcdd0148f90 0x7fcdd0099ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51264/0 (socket says 172.21.15.187:51264) 2020-11-20T12:17:43.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.515+0000 7fcdcffff700 1 -- 172.21.15.187:0/2299975625 learned_addr learned my addr 172.21.15.187:0/2299975625 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:43.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.515+0000 7fcdd4a56700 1 --2- 172.21.15.187:0/2299975625 >> v2:172.21.15.5:3300/0 conn(0x7fcdd00b00b0 0x7fcdd013f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.515+0000 7fcdcffff700 1 -- 172.21.15.187:0/2299975625 >> v2:172.21.15.5:3300/0 conn(0x7fcdd00b00b0 msgr2=0x7fcdd013f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:43.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.515+0000 7fcdcffff700 1 --2- 172.21.15.187:0/2299975625 >> v2:172.21.15.5:3300/0 conn(0x7fcdd00b00b0 0x7fcdd013f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.515+0000 7fcdcffff700 1 -- 172.21.15.187:0/2299975625 >> v1:172.21.15.187:6789/0 conn(0x7fcdd009a1d0 legacy=0x7fcdd013e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:43.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.516+0000 7fcdcffff700 1 -- 172.21.15.187:0/2299975625 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0066750 con 0x7fcdd0148f90 2020-11-20T12:17:43.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.516+0000 7fcdcffff700 1 --2- 172.21.15.187:0/2299975625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd0148f90 0x7fcdd0099ce0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7fcdd00b8ae0 tx=0x7fcdc0006eb0).ready entity=mon.1 client_cookie=9d4b505201e19a97 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:43.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.516+0000 7fcdcf7fe700 1 -- 172.21.15.187:0/2299975625 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00098d0 con 0x7fcdd0148f90 2020-11-20T12:17:43.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.516+0000 7fcdcf7fe700 1 -- 172.21.15.187:0/2299975625 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc0009a30 con 0x7fcdd0148f90 2020-11-20T12:17:43.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.516+0000 7fcdcf7fe700 1 -- 172.21.15.187:0/2299975625 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0009db0 con 0x7fcdd0148f90 2020-11-20T12:17:43.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.517+0000 7fcdd5257700 1 -- 172.21.15.187:0/2299975625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd0148f90 msgr2=0x7fcdd0099ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.517+0000 7fcdd5257700 1 --2- 172.21.15.187:0/2299975625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd0148f90 0x7fcdd0099ce0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7fcdd00b8ae0 tx=0x7fcdc0006eb0).stop 2020-11-20T12:17:43.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.517+0000 7fcdd5257700 1 -- 172.21.15.187:0/2299975625 shutdown_connections 2020-11-20T12:17:43.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.517+0000 7fcdd5257700 1 --2- 172.21.15.187:0/2299975625 >> v2:172.21.15.5:3300/0 conn(0x7fcdd00b00b0 0x7fcdd013f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.517+0000 7fcdd5257700 1 --2- 172.21.15.187:0/2299975625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd0148f90 0x7fcdd0099ce0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.517+0000 7fcdd5257700 1 -- 172.21.15.187:0/2299975625 >> 172.21.15.187:0/2299975625 conn(0x7fcdd00b8280 msgr2=0x7fcdd00653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:43.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.517+0000 7fcdd5257700 1 -- 172.21.15.187:0/2299975625 shutdown_connections 2020-11-20T12:17:43.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.517+0000 7fcdd5257700 1 -- 172.21.15.187:0/2299975625 wait complete. 2020-11-20T12:17:43.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.519+0000 7fcdd5257700 1 Processor -- start 2020-11-20T12:17:43.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.519+0000 7fcdd5257700 1 -- start start 2020-11-20T12:17:43.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd5257700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd009a1d0 0x7fcdd0138c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd5257700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd00b00b0 0x7fcdd0133990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd5257700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd013ea20 0x7fcdd0132e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd5257700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd0139130 con 0x7fcdd009a1d0 2020-11-20T12:17:43.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd5257700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd0138850 con 0x7fcdd013ea20 2020-11-20T12:17:43.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd5257700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd01385b0 con 0x7fcdd00b00b0 2020-11-20T12:17:43.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd4a56700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd013ea20 0x7fcdd0132e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd4a56700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd013ea20 0x7fcdd0132e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51272/0 (socket says 172.21.15.187:51272) 2020-11-20T12:17:43.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd4a56700 1 -- 172.21.15.187:0/1364115942 learned_addr learned my addr 172.21.15.187:0/1364115942 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:43.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdcffff700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd009a1d0 0x7fcdd0138c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.520+0000 7fcdd4a56700 1 -- 172.21.15.187:0/1364115942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd00b00b0 msgr2=0x7fcdd0133990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdd4a56700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd00b00b0 0x7fcdd0133990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:43.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdd4a56700 1 -- 172.21.15.187:0/1364115942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd009a1d0 msgr2=0x7fcdd0138c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdd4a56700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd009a1d0 0x7fcdd0138c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdd4a56700 1 -- 172.21.15.187:0/1364115942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0136680 con 0x7fcdd013ea20 2020-11-20T12:17:43.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdd4a56700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd013ea20 0x7fcdd0132e50 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 rx=0x7fcdc800cb80 tx=0x7fcdc8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:43.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdccff9700 1 -- 172.21.15.187:0/1364115942 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc80078a0 con 0x7fcdd013ea20 2020-11-20T12:17:43.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdccff9700 1 -- 172.21.15.187:0/1364115942 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc8007a00 con 0x7fcdd013ea20 2020-11-20T12:17:43.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd0133e80 con 0x7fcdd013ea20 2020-11-20T12:17:43.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdccff9700 1 -- 172.21.15.187:0/1364115942 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8007da0 con 0x7fcdd013ea20 2020-11-20T12:17:43.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.521+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd0135a00 con 0x7fcdd013ea20 2020-11-20T12:17:43.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.522+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd0136880 con 0x7fcdd013ea20 2020-11-20T12:17:43.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.523+0000 7fcdccff9700 1 -- 172.21.15.187:0/1364115942 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcdc8011620 con 0x7fcdd013ea20 2020-11-20T12:17:43.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.524+0000 7fcdccff9700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdac056d20 0x7fcdac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.524+0000 7fcdccff9700 1 -- 172.21.15.187:0/1364115942 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcdc8070550 con 0x7fcdd013ea20 2020-11-20T12:17:43.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.524+0000 7fcdcffff700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdac056d20 0x7fcdac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.525+0000 7fcdcffff700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdac056d20 0x7fcdac0591e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fcdd0064da0 tx=0x7fcdc0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:43.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.528+0000 7fcdccff9700 1 -- 172.21.15.187:0/1364115942 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcdd0136880 con 0x7fcdd013ea20 2020-11-20T12:17:43.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.775+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd0136880 con 0x7fcdd013ea20 2020-11-20T12:17:43.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.776+0000 7fcdccff9700 1 -- 172.21.15.187:0/1364115942 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7fcdd0136880 con 0x7fcdd013ea20 2020-11-20T12:17:43.780 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:43.780 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:43.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.778+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdac056d20 msgr2=0x7fcdac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.778+0000 7fcdd5257700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdac056d20 0x7fcdac0591e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fcdd0064da0 tx=0x7fcdc0015000).stop 2020-11-20T12:17:43.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.778+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd013ea20 msgr2=0x7fcdd0132e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.778+0000 7fcdd5257700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd013ea20 0x7fcdd0132e50 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 rx=0x7fcdc800cb80 tx=0x7fcdc8008040).stop 2020-11-20T12:17:43.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.779+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 shutdown_connections 2020-11-20T12:17:43.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.779+0000 7fcdd5257700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcdac056d20 0x7fcdac0591e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.779+0000 7fcdd5257700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd009a1d0 0x7fcdd0138c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.779+0000 7fcdd5257700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdd013ea20 0x7fcdd0132e50 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.779+0000 7fcdd5257700 1 --2- 172.21.15.187:0/1364115942 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcdd00b00b0 0x7fcdd0133990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:43.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.779+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 >> 172.21.15.187:0/1364115942 conn(0x7fcdd00b8280 msgr2=0x7fcdd00600f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:43.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.779+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 shutdown_connections 2020-11-20T12:17:43.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:43.779+0000 7fcdd5257700 1 -- 172.21.15.187:0/1364115942 wait complete. 2020-11-20T12:17:43.786 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:43.806 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:44.077 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2568,"msgr_send_messages":2570,"msgr_recv_bytes":800303,"msgr_send_bytes":9987404,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.441768004,"msgr_running_send_time":0.181917202,"msgr_running_recv_time":0.097968749,"msgr_running_fast_dispatch_time":0.125710479,"msgr_send_messages_queue_lat":{"avgcount":2568,"sum":0.455107718,"avgtime":0.000177222},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9322,"msgr_send_messages":14586,"msgr_recv_bytes":2649458,"msgr_send_bytes":14203581,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.023593473,"msgr_running_send_time":2.241605415,"msgr_running_recv_time":0.943719011,"msgr_running_fast_dispatch_time":0.092453408,"msgr_send_messages_queue_lat":{"avgcount":14585,"sum":1.925254295,"avgtime":0.000132002},"msgr_handle_ack_lat":{"avgcount":8617,"sum":0.006474858,"avgtime":0.000000751}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":957,"msgr_send_messages":954,"msgr_recv_bytes":329648,"msgr_send_bytes":629839,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.108367201,"msgr_running_send_time":0.037491428,"msgr_running_recv_time":0.039111955,"msgr_running_fast_dispatch_time":0.027600164,"msgr_send_messages_queue_lat":{"avgcount":954,"sum":0.147178965,"avgtime":0.000154275},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3127,"sum":3.736767472,"avgtime":0.001195000}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3585,"sum":0.336993806,"avgtime":0.000094001}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":235,"inodes_top":0,"inodes_bottom":235,"inodes_pin_tail":0,"inodes_pinned":235,"inodes_expired":0,"inodes_with_caps":161,"caps":161,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":6190,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4816,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4223,"handle_client_caps_dirty":1500,"handle_client_cap_release":13,"process_request_cap_release":1668},"mds_cache":{"num_strays":222,"num_strays_delayed":62,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1379,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9092,"evex":9025,"evtrm":9025,"ev":71,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21481987,"rdpos":4195632,"jlat":{"avgcount":1545,"sum":18.403514674,"avgtime":0.011911659},"replayed":4},"mds_mem":{"ino":238,"ino+":1614,"ino-":1376,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":161,"cap+":1602,"cap-":1441,"rss":72688,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1246},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":3829589,"buffer_anon_items":1232,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":891257,"mds_co_items":15703,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5213,"op_send_bytes":18505219,"op_resend":0,"op_reply":5213,"oplen_avg":{"avgcount":5213,"sum":5823},"op":5213,"op_r":37,"op_w":5176,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":19,"osdop_write":1556,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3267,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":196,"omap_rd":28,"omap_del":17},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1379,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12843,"get_sum":2741613,"get_or_fail_fail":0,"get_or_fail_success":12843,"take":0,"take_sum":0,"put":12843,"put_sum":2741613,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5213,"take_sum":18646392,"put":1933,"put_sum":18646392,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5213,"take_sum":5213,"put":5213,"put_sum":5213,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1379,"get_sum":139287,"get_or_fail_fail":0,"get_or_fail_success":1379,"take":0,"take_sum":0,"put":11,"put_sum":139287,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":0,"max":3758096384,"get_started":0,"get":9092,"get_sum":17285573,"get_or_fail_fail":0,"get_or_fail_success":9092,"take":0,"take_sum":0,"put":1545,"put_sum":17285573,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:44.079 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 800303, "msgr_recv_messages": 2568, "msgr_running_fast_dispatch_time": 0.125710479, "msgr_running_recv_time": 0.097968749, "msgr_running_send_time": 0.181917202, "msgr_running_total_time": 0.441768004, "msgr_send_bytes": 9987404, "msgr_send_messages": 2570, "msgr_send_messages_queue_lat": { "avgcount": 2568, "avgtime": 0.000177222, "sum": 0.455107718 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8617, "avgtime": 7.51e-07, "sum": 0.006474858 }, "msgr_recv_bytes": 2649458, "msgr_recv_messages": 9322, "msgr_running_fast_dispatch_time": 0.092453408, "msgr_running_recv_time": 0.943719011, "msgr_running_send_time": 2.241605415, "msgr_running_total_time": 3.023593473, "msgr_send_bytes": 14203581, "msgr_send_messages": 14586, "msgr_send_messages_queue_lat": { "avgcount": 14585, "avgtime": 0.000132002, "sum": 1.925254295 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 329648, "msgr_recv_messages": 957, "msgr_running_fast_dispatch_time": 0.027600164, "msgr_running_recv_time": 0.039111955, "msgr_running_send_time": 0.037491428, "msgr_running_total_time": 0.108367201, "msgr_send_bytes": 629839, "msgr_send_messages": 954, "msgr_send_messages_queue_lat": { "avgcount": 954, "avgtime": 0.000154275, "sum": 0.147178965 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3127, "avgtime": 0.001195, "sum": 3.736767472 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3585, "avgtime": 9.4001e-05, "sum": 0.336993806 }, "queue_len": 0 }, "mds": { "caps": 161, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4816, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 13, "handle_client_caps": 4223, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 235, "inodes_bottom": 235, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 235, "inodes_top": 0, "inodes_with_caps": 161, "load_cent": 6190, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 222, "num_strays_delayed": 62, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1379, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 71, "evadd": 9092, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1545, "avgtime": 0.011911659, "sum": 18.403514674 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21481987 }, "mds_mem": { "cap": 161, "cap+": 1602, "cap-": 1441, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 238, "ino+": 1614, "ino-": 1376, "rss": 72688 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1246, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 3829589, "buffer_anon_items": 1232, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 891257, "mds_co_items": 15703, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 17, "omap_rd": 28, "omap_wr": 196, "op": 5213, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 37, "op_reply": 5213, "op_resend": 0, "op_rmw": 0, "op_send": 5213, "op_send_bytes": 18505219, "op_w": 5176, "oplen_avg": { "avgcount": 5213, "sum": 5823 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3267, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 19, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1556, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1379, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12843, "get_or_fail_fail": 0, "get_or_fail_success": 12843, "get_started": 0, "get_sum": 2741613, "max": 104857600, "put": 12843, "put_sum": 2741613, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1933, "put_sum": 18646392, "take": 5213, "take_sum": 18646392, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5213, "put_sum": 5213, "take": 5213, "take_sum": 5213, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1379, "get_or_fail_fail": 0, "get_or_fail_success": 1379, "get_started": 0, "get_sum": 139287, "max": 3758096384, "put": 11, "put_sum": 139287, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9092, "get_or_fail_fail": 0, "get_or_fail_success": 9092, "get_started": 0, "get_sum": 17285573, "max": 3758096384, "put": 1545, "put_sum": 17285573, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:44.080 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/222, 1379/1601 2020-11-20T12:17:45.080 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:45.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.424+0000 7f90033fd700 1 Processor -- start 2020-11-20T12:17:46.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.424+0000 7f90033fd700 1 -- start start 2020-11-20T12:17:46.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f90033fd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8ffc067d00 0x7f8ffc1456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:46.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f90033fd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8ffc145bb0 0x7f8ffc14e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:46.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f90033fd700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc09d610 con 0x7f8ffc09ef90 2020-11-20T12:17:46.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f90033fd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc13f500 con 0x7f8ffc145bb0 2020-11-20T12:17:46.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f90033fd700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc09f510 con 0x7f8ffc067d00 2020-11-20T12:17:46.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f9000998700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8ffc067d00 0x7f8ffc1456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:46.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f9001199700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8ffc09ef90 0x7f8ffc144fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47884/0 (socket says 172.21.15.187:47884) 2020-11-20T12:17:46.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f9000998700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8ffc067d00 0x7f8ffc1456c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51280/0 (socket says 172.21.15.187:51280) 2020-11-20T12:17:46.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.426+0000 7f9001199700 1 -- 172.21.15.187:0/2950218668 learned_addr learned my addr 172.21.15.187:0/2950218668 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:46.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.427+0000 7f900199a700 1 --2- 172.21.15.187:0/2950218668 >> v2:172.21.15.5:3300/0 conn(0x7f8ffc145bb0 0x7f8ffc14e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:46.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.427+0000 7f9000998700 1 -- 172.21.15.187:0/2950218668 >> v2:172.21.15.5:3300/0 conn(0x7f8ffc145bb0 msgr2=0x7f8ffc14e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:46.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.427+0000 7f9000998700 1 --2- 172.21.15.187:0/2950218668 >> v2:172.21.15.5:3300/0 conn(0x7f8ffc145bb0 0x7f8ffc14e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:46.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.427+0000 7f9000998700 1 -- 172.21.15.187:0/2950218668 >> v1:172.21.15.187:6789/0 conn(0x7f8ffc09ef90 legacy=0x7f8ffc144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:46.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.427+0000 7f8febfff700 1 -- 172.21.15.187:0/2950218668 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963589584 0 0) 0x7f8ffc09d610 con 0x7f8ffc09ef90 2020-11-20T12:17:46.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.427+0000 7f9000998700 1 -- 172.21.15.187:0/2950218668 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc09f7b0 con 0x7f8ffc067d00 2020-11-20T12:17:46.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.427+0000 7f9000998700 1 --2- 172.21.15.187:0/2950218668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc067d00 0x7f8ffc1456c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f8ffc061b60 tx=0x7f8fec006df0).ready entity=mon.1 client_cookie=cd40b14372de2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:46.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.427+0000 7f8febfff700 1 -- 172.21.15.187:0/2950218668 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec0097b0 con 0x7f8ffc067d00 2020-11-20T12:17:46.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.428+0000 7f8febfff700 1 -- 172.21.15.187:0/2950218668 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec009910 con 0x7f8ffc067d00 2020-11-20T12:17:46.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.428+0000 7f8febfff700 1 -- 172.21.15.187:0/2950218668 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec009c90 con 0x7f8ffc067d00 2020-11-20T12:17:46.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.428+0000 7f90033fd700 1 -- 172.21.15.187:0/2950218668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc067d00 msgr2=0x7f8ffc1456c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:46.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.428+0000 7f90033fd700 1 --2- 172.21.15.187:0/2950218668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc067d00 0x7f8ffc1456c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f8ffc061b60 tx=0x7f8fec006df0).stop 2020-11-20T12:17:46.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.428+0000 7f90033fd700 1 -- 172.21.15.187:0/2950218668 shutdown_connections 2020-11-20T12:17:46.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.428+0000 7f90033fd700 1 --2- 172.21.15.187:0/2950218668 >> v2:172.21.15.5:3300/0 conn(0x7f8ffc145bb0 0x7f8ffc14e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:46.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.428+0000 7f90033fd700 1 --2- 172.21.15.187:0/2950218668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc067d00 0x7f8ffc1456c0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:46.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.428+0000 7f90033fd700 1 -- 172.21.15.187:0/2950218668 >> 172.21.15.187:0/2950218668 conn(0x7f8ffc0bb950 msgr2=0x7f8ffc0b7920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:46.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.429+0000 7f90033fd700 1 -- 172.21.15.187:0/2950218668 shutdown_connections 2020-11-20T12:17:46.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.429+0000 7f90033fd700 1 -- 172.21.15.187:0/2950218668 wait complete. 2020-11-20T12:17:46.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.432+0000 7f90033fd700 1 Processor -- start 2020-11-20T12:17:46.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.434+0000 7f90033fd700 1 -- start start 2020-11-20T12:17:46.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.435+0000 7f90033fd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc09ef90 0x7f8ffc141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:46.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.435+0000 7f90033fd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ffc145bb0 0x7f8ffc146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:46.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.435+0000 7f90033fd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ffc143b10 0x7f8ffc0b6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:46.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.435+0000 7f90033fd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc146cf0 con 0x7f8ffc145bb0 2020-11-20T12:17:46.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.435+0000 7f90033fd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc141a30 con 0x7f8ffc09ef90 2020-11-20T12:17:46.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.435+0000 7f90033fd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc146520 con 0x7f8ffc143b10 2020-11-20T12:17:46.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.436+0000 7f9001199700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc09ef90 0x7f8ffc141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:46.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.436+0000 7f9001199700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc09ef90 0x7f8ffc141540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51284/0 (socket says 172.21.15.187:51284) 2020-11-20T12:17:46.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.436+0000 7f9001199700 1 -- 172.21.15.187:0/1962516447 learned_addr learned my addr 172.21.15.187:0/1962516447 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:46.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f9001199700 1 -- 172.21.15.187:0/1962516447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ffc143b10 msgr2=0x7f8ffc0b6630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T12:17:46.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f9001199700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ffc143b10 0x7f8ffc0b6630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:46.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f9001199700 1 -- 172.21.15.187:0/1962516447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ffc145bb0 msgr2=0x7f8ffc146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:46.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f9001199700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ffc145bb0 0x7f8ffc146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:46.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f9001199700 1 -- 172.21.15.187:0/1962516447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc1467b0 con 0x7f8ffc09ef90 2020-11-20T12:17:46.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f9001199700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc09ef90 0x7f8ffc141540 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 rx=0x7f8ff800c720 tx=0x7f8ff8007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:46.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f8fe97fa700 1 -- 172.21.15.187:0/1962516447 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8007320 con 0x7f8ffc09ef90 2020-11-20T12:17:46.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f8fe97fa700 1 -- 172.21.15.187:0/1962516447 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff8007480 con 0x7f8ffc09ef90 2020-11-20T12:17:46.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f8fe97fa700 1 -- 172.21.15.187:0/1962516447 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8005800 con 0x7f8ffc09ef90 2020-11-20T12:17:46.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.437+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc146a50 con 0x7f8ffc09ef90 2020-11-20T12:17:46.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.438+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc13ac80 con 0x7f8ffc09ef90 2020-11-20T12:17:46.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.441+0000 7f8fe97fa700 1 -- 172.21.15.187:0/1962516447 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8ff8010030 con 0x7f8ffc09ef90 2020-11-20T12:17:46.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.442+0000 7f8fe97fa700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fd4056d50 0x7f8fd4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:46.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.442+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc15b7d0 con 0x7f8ffc09ef90 2020-11-20T12:17:46.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.444+0000 7f9000998700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fd4056d50 0x7f8fd4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:46.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.444+0000 7f8fe97fa700 1 -- 172.21.15.187:0/1962516447 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8ff8048ed0 con 0x7f8ffc09ef90 2020-11-20T12:17:46.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.446+0000 7f9000998700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fd4056d50 0x7f8fd4059210 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f8ffc0680e0 tx=0x7f8fec015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:46.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.452+0000 7f8fe97fa700 1 -- 172.21.15.187:0/1962516447 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8ff8048ca0 con 0x7f8ffc09ef90 2020-11-20T12:17:46.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.765+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ffc069660 con 0x7f8ffc09ef90 2020-11-20T12:17:46.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.766+0000 7f8fe97fa700 1 -- 172.21.15.187:0/1962516447 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f8ff8048890 con 0x7f8ffc09ef90 2020-11-20T12:17:46.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.768+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fd4056d50 msgr2=0x7f8fd4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:46.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.768+0000 7f90033fd700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fd4056d50 0x7f8fd4059210 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f8ffc0680e0 tx=0x7f8fec015000).stop 2020-11-20T12:17:46.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.768+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc09ef90 msgr2=0x7f8ffc141540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:46.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.768+0000 7f90033fd700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc09ef90 0x7f8ffc141540 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 rx=0x7f8ff800c720 tx=0x7f8ff8007ac0).stop 2020-11-20T12:17:46.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.769+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 shutdown_connections 2020-11-20T12:17:46.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.769+0000 7f90033fd700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8fd4056d50 0x7f8fd4059210 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:46.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.769+0000 7f90033fd700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ffc143b10 0x7f8ffc0b6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:46.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.769+0000 7f90033fd700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ffc09ef90 0x7f8ffc141540 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:46.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.769+0000 7f90033fd700 1 --2- 172.21.15.187:0/1962516447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ffc145bb0 0x7f8ffc146030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:46.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.769+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 >> 172.21.15.187:0/1962516447 conn(0x7f8ffc0bb950 msgr2=0x7f8ffc062060 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:46.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.769+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 shutdown_connections 2020-11-20T12:17:46.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:45.769+0000 7f90033fd700 1 -- 172.21.15.187:0/1962516447 wait complete. 2020-11-20T12:17:46.623 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:46.624 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:46.625 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:46.899 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2643,"msgr_send_messages":2645,"msgr_recv_bytes":825583,"msgr_send_bytes":10017559,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.450230119,"msgr_running_send_time":0.184385805,"msgr_running_recv_time":0.100475763,"msgr_running_fast_dispatch_time":0.128437061,"msgr_send_messages_queue_lat":{"avgcount":2643,"sum":0.464499314,"avgtime":0.000175746},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9535,"msgr_send_messages":14639,"msgr_recv_bytes":2714986,"msgr_send_bytes":14220650,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.043113917,"msgr_running_send_time":2.248578876,"msgr_running_recv_time":0.963002263,"msgr_running_fast_dispatch_time":0.095096928,"msgr_send_messages_queue_lat":{"avgcount":14638,"sum":1.934190245,"avgtime":0.000132134},"msgr_handle_ack_lat":{"avgcount":8626,"sum":0.006569689,"avgtime":0.000000761}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":991,"msgr_send_messages":988,"msgr_recv_bytes":338302,"msgr_send_bytes":640926,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.113182095,"msgr_running_send_time":0.038712290,"msgr_running_recv_time":0.040490518,"msgr_running_fast_dispatch_time":0.029341499,"msgr_send_messages_queue_lat":{"avgcount":988,"sum":0.158238762,"avgtime":0.000160160},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3127,"sum":3.736767472,"avgtime":0.001195000}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3742,"sum":0.356752088,"avgtime":0.000095337}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":235,"inodes_top":0,"inodes_bottom":235,"inodes_pin_tail":0,"inodes_pinned":235,"inodes_expired":0,"inodes_with_caps":161,"caps":161,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":6190,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4826,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4383,"handle_client_caps_dirty":1500,"handle_client_cap_release":20,"process_request_cap_release":1668},"mds_cache":{"num_strays":160,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1441,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9314,"evex":9025,"evtrm":9025,"ev":293,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21832295,"rdpos":4195632,"jlat":{"avgcount":1545,"sum":18.403514674,"avgtime":0.011911659},"replayed":4},"mds_mem":{"ino":235,"ino+":1614,"ino-":1379,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":1,"cap+":1602,"cap-":1601,"rss":72688,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1246},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":3829589,"buffer_anon_items":1232,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":891257,"mds_co_items":15703,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5364,"op_send_bytes":18511571,"op_resend":0,"op_reply":5364,"oplen_avg":{"avgcount":5364,"sum":5974},"op":5364,"op_r":39,"op_w":5325,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":21,"osdop_write":1558,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3413,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":196,"omap_rd":28,"omap_del":17},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1441,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13165,"get_sum":2814763,"get_or_fail_fail":0,"get_or_fail_success":13165,"take":0,"take_sum":0,"put":13165,"put_sum":2814763,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5364,"take_sum":18659006,"put":1938,"put_sum":18659006,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5364,"take_sum":5364,"put":5364,"put_sum":5364,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1441,"get_sum":145549,"get_or_fail_fail":0,"get_or_fail_success":1441,"take":0,"take_sum":0,"put":13,"put_sum":145549,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":350308,"max":3758096384,"get_started":0,"get":9314,"get_sum":17635881,"get_or_fail_fail":0,"get_or_fail_success":9314,"take":0,"take_sum":0,"put":1545,"put_sum":17285573,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:46.903 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 825583, "msgr_recv_messages": 2643, "msgr_running_fast_dispatch_time": 0.128437061, "msgr_running_recv_time": 0.100475763, "msgr_running_send_time": 0.184385805, "msgr_running_total_time": 0.450230119, "msgr_send_bytes": 10017559, "msgr_send_messages": 2645, "msgr_send_messages_queue_lat": { "avgcount": 2643, "avgtime": 0.000175746, "sum": 0.464499314 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8626, "avgtime": 7.61e-07, "sum": 0.006569689 }, "msgr_recv_bytes": 2714986, "msgr_recv_messages": 9535, "msgr_running_fast_dispatch_time": 0.095096928, "msgr_running_recv_time": 0.963002263, "msgr_running_send_time": 2.248578876, "msgr_running_total_time": 3.043113917, "msgr_send_bytes": 14220650, "msgr_send_messages": 14639, "msgr_send_messages_queue_lat": { "avgcount": 14638, "avgtime": 0.000132134, "sum": 1.934190245 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 338302, "msgr_recv_messages": 991, "msgr_running_fast_dispatch_time": 0.029341499, "msgr_running_recv_time": 0.040490518, "msgr_running_send_time": 0.03871229, "msgr_running_total_time": 0.113182095, "msgr_send_bytes": 640926, "msgr_send_messages": 988, "msgr_send_messages_queue_lat": { "avgcount": 988, "avgtime": 0.00016016, "sum": 0.158238762 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3127, "avgtime": 0.001195, "sum": 3.736767472 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3742, "avgtime": 9.5337e-05, "sum": 0.356752088 }, "queue_len": 0 }, "mds": { "caps": 161, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4826, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 20, "handle_client_caps": 4383, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 235, "inodes_bottom": 235, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 235, "inodes_top": 0, "inodes_with_caps": 161, "load_cent": 6190, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 160, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1441, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 293, "evadd": 9314, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1545, "avgtime": 0.011911659, "sum": 18.403514674 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21832295 }, "mds_mem": { "cap": 1, "cap+": 1602, "cap-": 1601, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 235, "ino+": 1614, "ino-": 1379, "rss": 72688 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1246, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 3829589, "buffer_anon_items": 1232, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 891257, "mds_co_items": 15703, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 17, "omap_rd": 28, "omap_wr": 196, "op": 5364, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 39, "op_reply": 5364, "op_resend": 0, "op_rmw": 0, "op_send": 5364, "op_send_bytes": 18511571, "op_w": 5325, "oplen_avg": { "avgcount": 5364, "sum": 5974 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3413, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 21, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1558, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1441, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13165, "get_or_fail_fail": 0, "get_or_fail_success": 13165, "get_started": 0, "get_sum": 2814763, "max": 104857600, "put": 13165, "put_sum": 2814763, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1938, "put_sum": 18659006, "take": 5364, "take_sum": 18659006, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5364, "put_sum": 5364, "take": 5364, "take_sum": 5364, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1441, "get_or_fail_fail": 0, "get_or_fail_success": 1441, "get_started": 0, "get_sum": 145549, "max": 3758096384, "put": 13, "put_sum": 145549, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9314, "get_or_fail_fail": 0, "get_or_fail_success": 9314, "get_started": 0, "get_sum": 17635881, "max": 3758096384, "put": 1545, "put_sum": 17285573, "take": 0, "take_sum": 0, "val": 350308, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:46.904 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/160, 1441/1601 2020-11-20T12:17:47.906 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.153+0000 7ff7e2a4e700 1 Processor -- start 2020-11-20T12:17:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.153+0000 7ff7e2a4e700 1 -- start start 2020-11-20T12:17:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.154+0000 7ff7e2a4e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7dc065c90 0x7ff7dc14cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.154+0000 7ff7e2a4e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc066070 0x7ff7dc0b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.154+0000 7ff7e2a4e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7dc14c2a0 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.154+0000 7ff7e2a4e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7dc14c540 con 0x7ff7dc065c90 2020-11-20T12:17:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.154+0000 7ff7e2a4e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc14c7e0 con 0x7ff7dc066070 2020-11-20T12:17:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.155+0000 7ff7e1a4c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7dc065c90 0x7ff7dc14cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.155+0000 7ff7e1a4c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7dc065c90 0x7ff7dc14cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51290/0 (socket says 172.21.15.187:51290) 2020-11-20T12:17:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.155+0000 7ff7e1a4c700 1 -- 172.21.15.187:0/2577985265 learned_addr learned my addr 172.21.15.187:0/2577985265 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.155+0000 7ff7e124b700 1 --2- 172.21.15.187:0/2577985265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc066070 0x7ff7dc0b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.156+0000 7ff7e1a4c700 1 -- 172.21.15.187:0/2577985265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc066070 msgr2=0x7ff7dc0b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:48.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.156+0000 7ff7e1a4c700 1 --2- 172.21.15.187:0/2577985265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc066070 0x7ff7dc0b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.156+0000 7ff7e1a4c700 1 -- 172.21.15.187:0/2577985265 >> v1:172.21.15.5:6789/0 conn(0x7ff7dc0b6dc0 legacy=0x7ff7dc145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:48.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.156+0000 7ff7e1a4c700 1 -- 172.21.15.187:0/2577985265 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc0b5800 con 0x7ff7dc065c90 2020-11-20T12:17:48.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.156+0000 7ff7e1a4c700 1 --2- 172.21.15.187:0/2577985265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc065c90 0x7ff7dc14cc80 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 rx=0x7ff7d8006fd0 tx=0x7ff7d8006eb0).ready entity=mon.1 client_cookie=f0932082d42e37c6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:48.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.156+0000 7ff7e0a4a700 1 -- 172.21.15.187:0/2577985265 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d8009890 con 0x7ff7dc065c90 2020-11-20T12:17:48.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.156+0000 7ff7e0a4a700 1 -- 172.21.15.187:0/2577985265 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d80099f0 con 0x7ff7dc065c90 2020-11-20T12:17:48.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.156+0000 7ff7e0a4a700 1 -- 172.21.15.187:0/2577985265 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d8009d70 con 0x7ff7dc065c90 2020-11-20T12:17:48.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.157+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/2577985265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc065c90 msgr2=0x7ff7dc14cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.157+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/2577985265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc065c90 0x7ff7dc14cc80 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 rx=0x7ff7d8006fd0 tx=0x7ff7d8006eb0).stop 2020-11-20T12:17:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.157+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/2577985265 shutdown_connections 2020-11-20T12:17:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.157+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/2577985265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc065c90 0x7ff7dc14cc80 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.157+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/2577985265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc066070 0x7ff7dc0b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.157+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/2577985265 >> 172.21.15.187:0/2577985265 conn(0x7ff7dc0b8280 msgr2=0x7ff7dc0653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.157+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/2577985265 shutdown_connections 2020-11-20T12:17:48.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.157+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/2577985265 wait complete. 2020-11-20T12:17:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.159+0000 7ff7e2a4e700 1 Processor -- start 2020-11-20T12:17:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.159+0000 7ff7e2a4e700 1 -- start start 2020-11-20T12:17:48.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e2a4e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc065c90 0x7ff7dc0b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e2a4e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7dc066070 0x7ff7dc0a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e2a4e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc0b6dc0 0x7ff7dc0a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e2a4e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0b28b0 con 0x7ff7dc066070 2020-11-20T12:17:48.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e2a4e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0b2b50 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e2a4e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0b2df0 con 0x7ff7dc065c90 2020-11-20T12:17:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e224d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc0b6dc0 0x7ff7dc0a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e224d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc0b6dc0 0x7ff7dc0a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51300/0 (socket says 172.21.15.187:51300) 2020-11-20T12:17:48.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e224d700 1 -- 172.21.15.187:0/3237454335 learned_addr learned my addr 172.21.15.187:0/3237454335 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e224d700 1 -- 172.21.15.187:0/3237454335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc065c90 msgr2=0x7ff7dc0b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e224d700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc065c90 0x7ff7dc0b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e224d700 1 -- 172.21.15.187:0/3237454335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7dc066070 msgr2=0x7ff7dc0a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e224d700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7dc066070 0x7ff7dc0a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.160+0000 7ff7e224d700 1 -- 172.21.15.187:0/3237454335 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc0b4b10 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.161+0000 7ff7e224d700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc0b6dc0 0x7ff7dc0a07e0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7ff7d400bae0 tx=0x7ff7d40090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:48.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.161+0000 7ff7c9ffb700 1 -- 172.21.15.187:0/3237454335 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d400c830 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.161+0000 7ff7c9ffb700 1 -- 172.21.15.187:0/3237454335 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d400c9b0 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.161+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7dc0b1de0 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.161+0000 7ff7c9ffb700 1 -- 172.21.15.187:0/3237454335 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d400cb60 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.161+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7dc0b2640 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.162+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7dc068ef0 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.163+0000 7ff7c9ffb700 1 -- 172.21.15.187:0/3237454335 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff7d401c070 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.163+0000 7ff7c9ffb700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7b4056d20 0x7ff7b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.163+0000 7ff7c9ffb700 1 -- 172.21.15.187:0/3237454335 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff7d406f730 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.164+0000 7ff7e1a4c700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7b4056d20 0x7ff7b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.164+0000 7ff7e1a4c700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7b4056d20 0x7ff7b40591e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7ff7d8006ef0 tx=0x7ff7d8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:48.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.168+0000 7ff7c9ffb700 1 -- 172.21.15.187:0/3237454335 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff7d4044a20 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.412+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7dc0b2fb0 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.413+0000 7ff7c9ffb700 1 -- 172.21.15.187:0/3237454335 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7ff7d4048040 con 0x7ff7dc0b6dc0 2020-11-20T12:17:48.417 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:48.417 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:48.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7b4056d20 msgr2=0x7ff7b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7b4056d20 0x7ff7b40591e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7ff7d8006ef0 tx=0x7ff7d8015000).stop 2020-11-20T12:17:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc0b6dc0 msgr2=0x7ff7dc0a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc0b6dc0 0x7ff7dc0a07e0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7ff7d400bae0 tx=0x7ff7d40090f0).stop 2020-11-20T12:17:48.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 shutdown_connections 2020-11-20T12:17:48.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff7b4056d20 0x7ff7b40591e0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7dc0b6dc0 0x7ff7dc0a07e0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7dc065c90 0x7ff7dc0b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 --2- 172.21.15.187:0/3237454335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7dc066070 0x7ff7dc0a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 >> 172.21.15.187:0/3237454335 conn(0x7ff7dc0b8280 msgr2=0x7ff7dc14c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 shutdown_connections 2020-11-20T12:17:48.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:48.416+0000 7ff7e2a4e700 1 -- 172.21.15.187:0/3237454335 wait complete. 2020-11-20T12:17:48.423 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:48.444 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:48.727 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2644,"msgr_send_messages":2646,"msgr_recv_bytes":825829,"msgr_send_bytes":10162359,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.452471614,"msgr_running_send_time":0.186203377,"msgr_running_recv_time":0.100504242,"msgr_running_fast_dispatch_time":0.128804388,"msgr_send_messages_queue_lat":{"avgcount":2644,"sum":0.464743887,"avgtime":0.000175773},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9538,"msgr_send_messages":14641,"msgr_recv_bytes":2715632,"msgr_send_bytes":14222747,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.043975409,"msgr_running_send_time":2.249006469,"msgr_running_recv_time":0.963148510,"msgr_running_fast_dispatch_time":0.095339255,"msgr_send_messages_queue_lat":{"avgcount":14640,"sum":1.934364678,"avgtime":0.000132128},"msgr_handle_ack_lat":{"avgcount":8626,"sum":0.006569689,"avgtime":0.000000761}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":992,"msgr_send_messages":989,"msgr_recv_bytes":338594,"msgr_send_bytes":643767,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.113397253,"msgr_running_send_time":0.038766339,"msgr_running_recv_time":0.040565450,"msgr_running_fast_dispatch_time":0.029410012,"msgr_send_messages_queue_lat":{"avgcount":989,"sum":0.158292057,"avgtime":0.000160052},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3131,"sum":3.810735479,"avgtime":0.001217098}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3742,"sum":0.356752088,"avgtime":0.000095337}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":232,"inodes_top":0,"inodes_bottom":232,"inodes_pin_tail":0,"inodes_pinned":232,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":12,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4826,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4383,"handle_client_caps_dirty":1500,"handle_client_cap_release":20,"process_request_cap_release":1668},"mds_cache":{"num_strays":160,"num_strays_delayed":160,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1441,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9314,"evex":9025,"evtrm":9025,"ev":293,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21832295,"rdpos":4195632,"jlat":{"avgcount":1546,"sum":18.418870032,"avgtime":0.011913887},"replayed":4},"mds_mem":{"ino":235,"ino+":1614,"ino-":1379,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":1,"cap+":1602,"cap-":1601,"rss":72688,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1251},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2601487,"buffer_anon_items":922,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1029568,"mds_co_items":18254,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5367,"op_send_bytes":18864457,"op_resend":0,"op_reply":5367,"oplen_avg":{"avgcount":5367,"sum":5978},"op":5367,"op_r":39,"op_w":5328,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":21,"osdop_write":1559,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3413,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":197,"omap_rd":28,"omap_del":18},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1441,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13170,"get_sum":2815529,"get_or_fail_fail":0,"get_or_fail_success":13170,"take":0,"take_sum":0,"put":13170,"put_sum":2815529,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5367,"take_sum":19011892,"put":1941,"put_sum":19011892,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5367,"take_sum":5367,"put":5367,"put_sum":5367,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1441,"get_sum":145549,"get_or_fail_fail":0,"get_or_fail_success":1441,"take":0,"take_sum":0,"put":13,"put_sum":145549,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":0,"max":3758096384,"get_started":0,"get":9314,"get_sum":17635881,"get_or_fail_fail":0,"get_or_fail_success":9314,"take":0,"take_sum":0,"put":1546,"put_sum":17635881,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:48.729 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 825829, "msgr_recv_messages": 2644, "msgr_running_fast_dispatch_time": 0.128804388, "msgr_running_recv_time": 0.100504242, "msgr_running_send_time": 0.186203377, "msgr_running_total_time": 0.452471614, "msgr_send_bytes": 10162359, "msgr_send_messages": 2646, "msgr_send_messages_queue_lat": { "avgcount": 2644, "avgtime": 0.000175773, "sum": 0.464743887 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8626, "avgtime": 7.61e-07, "sum": 0.006569689 }, "msgr_recv_bytes": 2715632, "msgr_recv_messages": 9538, "msgr_running_fast_dispatch_time": 0.095339255, "msgr_running_recv_time": 0.96314851, "msgr_running_send_time": 2.249006469, "msgr_running_total_time": 3.043975409, "msgr_send_bytes": 14222747, "msgr_send_messages": 14641, "msgr_send_messages_queue_lat": { "avgcount": 14640, "avgtime": 0.000132128, "sum": 1.934364678 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 338594, "msgr_recv_messages": 992, "msgr_running_fast_dispatch_time": 0.029410012, "msgr_running_recv_time": 0.04056545, "msgr_running_send_time": 0.038766339, "msgr_running_total_time": 0.113397253, "msgr_send_bytes": 643767, "msgr_send_messages": 989, "msgr_send_messages_queue_lat": { "avgcount": 989, "avgtime": 0.000160052, "sum": 0.158292057 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3131, "avgtime": 0.001217098, "sum": 3.810735479 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3742, "avgtime": 9.5337e-05, "sum": 0.356752088 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4826, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 20, "handle_client_caps": 4383, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 232, "inodes_bottom": 232, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 232, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 12, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 160, "num_strays_delayed": 160, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1441, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 293, "evadd": 9314, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1546, "avgtime": 0.011913887, "sum": 18.418870032 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21832295 }, "mds_mem": { "cap": 1, "cap+": 1602, "cap-": 1601, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 235, "ino+": 1614, "ino-": 1379, "rss": 72688 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1251, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2601487, "buffer_anon_items": 922, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1029568, "mds_co_items": 18254, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 18, "omap_rd": 28, "omap_wr": 197, "op": 5367, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 39, "op_reply": 5367, "op_resend": 0, "op_rmw": 0, "op_send": 5367, "op_send_bytes": 18864457, "op_w": 5328, "oplen_avg": { "avgcount": 5367, "sum": 5978 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3413, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 21, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1559, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1441, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13170, "get_or_fail_fail": 0, "get_or_fail_success": 13170, "get_started": 0, "get_sum": 2815529, "max": 104857600, "put": 13170, "put_sum": 2815529, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1941, "put_sum": 19011892, "take": 5367, "take_sum": 19011892, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5367, "put_sum": 5367, "take": 5367, "take_sum": 5367, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1441, "get_or_fail_fail": 0, "get_or_fail_success": 1441, "get_started": 0, "get_sum": 145549, "max": 3758096384, "put": 13, "put_sum": 145549, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9314, "get_or_fail_fail": 0, "get_or_fail_success": 9314, "get_started": 0, "get_sum": 17635881, "max": 3758096384, "put": 1546, "put_sum": 17635881, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:48.729 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/160, 1441/1601 2020-11-20T12:17:49.731 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:50.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.030+0000 7ffbb87fe700 1 Processor -- start 2020-11-20T12:17:50.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.030+0000 7ffbb87fe700 1 -- start start 2020-11-20T12:17:50.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.031+0000 7ffbb87fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffbb0142b10 0x7ffbb014caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.031+0000 7ffbb87fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb0063860 0x7ffbb0063cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.032+0000 7ffbb5d99700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ffbb00606e0 0x7ffbb0149390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47912/0 (socket says 172.21.15.187:47912) 2020-11-20T12:17:50.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.032+0000 7ffbb659a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffbb0142b10 0x7ffbb014caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.032+0000 7ffbb659a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffbb0142b10 0x7ffbb014caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:43568/0 (socket says 172.21.15.187:43568) 2020-11-20T12:17:50.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.032+0000 7ffbb659a700 1 -- 172.21.15.187:0/1274438273 learned_addr learned my addr 172.21.15.187:0/1274438273 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:50.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.032+0000 7ffbb87fe700 1 -- 172.21.15.187:0/1274438273 --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb0148c50 con 0x7ffbb00606e0 2020-11-20T12:17:50.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.032+0000 7ffbb87fe700 1 -- 172.21.15.187:0/1274438273 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb0148ef0 con 0x7ffbb0142b10 2020-11-20T12:17:50.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.032+0000 7ffbb87fe700 1 -- 172.21.15.187:0/1274438273 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0149aa0 con 0x7ffbb0063860 2020-11-20T12:17:50.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.032+0000 7ffbb6d9b700 1 --2- 172.21.15.187:0/1274438273 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb0063860 0x7ffbb0063cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.033+0000 7ffbb659a700 1 -- 172.21.15.187:0/1274438273 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb0063860 msgr2=0x7ffbb0063cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:50.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.033+0000 7ffbb659a700 1 --2- 172.21.15.187:0/1274438273 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb0063860 0x7ffbb0063cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.033+0000 7ffbb5598700 1 -- 172.21.15.187:0/1274438273 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015989356 0 0) 0x7ffbb0148c50 con 0x7ffbb00606e0 2020-11-20T12:17:50.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.033+0000 7ffbb659a700 1 -- 172.21.15.187:0/1274438273 >> v1:172.21.15.187:6789/0 conn(0x7ffbb00606e0 legacy=0x7ffbb0149390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.033+0000 7ffbb659a700 1 -- 172.21.15.187:0/1274438273 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0060e30 con 0x7ffbb0142b10 2020-11-20T12:17:50.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.033+0000 7ffbb659a700 1 --2- 172.21.15.187:0/1274438273 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0142b10 0x7ffbb014caa0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7ffbac006fd0 tx=0x7ffbac006eb0).ready entity=mon.0 client_cookie=e6689d5a22c68a3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.034+0000 7ffbb5598700 1 -- 172.21.15.187:0/1274438273 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac009810 con 0x7ffbb0142b10 2020-11-20T12:17:50.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.034+0000 7ffbb5598700 1 -- 172.21.15.187:0/1274438273 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac009970 con 0x7ffbb0142b10 2020-11-20T12:17:50.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.034+0000 7ffbb5598700 1 -- 172.21.15.187:0/1274438273 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac009cf0 con 0x7ffbb0142b10 2020-11-20T12:17:50.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.036+0000 7ffbb87fe700 1 -- 172.21.15.187:0/1274438273 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0142b10 msgr2=0x7ffbb014caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.036+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/1274438273 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0142b10 0x7ffbb014caa0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7ffbac006fd0 tx=0x7ffbac006eb0).stop 2020-11-20T12:17:50.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.036+0000 7ffbb87fe700 1 -- 172.21.15.187:0/1274438273 shutdown_connections 2020-11-20T12:17:50.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.036+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/1274438273 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb0063860 0x7ffbb0063cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.036+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/1274438273 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0142b10 0x7ffbb014caa0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.036+0000 7ffbb87fe700 1 -- 172.21.15.187:0/1274438273 >> 172.21.15.187:0/1274438273 conn(0x7ffbb009d640 msgr2=0x7ffbb009da50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:50.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.036+0000 7ffbb87fe700 1 -- 172.21.15.187:0/1274438273 shutdown_connections 2020-11-20T12:17:50.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.036+0000 7ffbb87fe700 1 -- 172.21.15.187:0/1274438273 wait complete. 2020-11-20T12:17:50.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.038+0000 7ffbb87fe700 1 Processor -- start 2020-11-20T12:17:50.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.038+0000 7ffbb87fe700 1 -- start start 2020-11-20T12:17:50.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.039+0000 7ffbb87fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb00606e0 0x7ffbb0139300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.039+0000 7ffbb87fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0063860 0x7ffbb0157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.039+0000 7ffbb87fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb0142b10 0x7ffbb0157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.040+0000 7ffbb5d99700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0063860 0x7ffbb0157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.040+0000 7ffbb5d99700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0063860 0x7ffbb0157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:43574/0 (socket says 172.21.15.187:43574) 2020-11-20T12:17:50.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.040+0000 7ffbb659a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb00606e0 0x7ffbb0139300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.040+0000 7ffbb659a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb00606e0 0x7ffbb0139300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:52200/0 (socket says 172.21.15.187:52200) 2020-11-20T12:17:50.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.041+0000 7ffbb6d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb0142b10 0x7ffbb0157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.041+0000 7ffbb6d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb0142b10 0x7ffbb0157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51314/0 (socket says 172.21.15.187:51314) 2020-11-20T12:17:50.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.041+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb01397f0 con 0x7ffbb0063860 2020-11-20T12:17:50.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.041+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0157a30 con 0x7ffbb0142b10 2020-11-20T12:17:50.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.041+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb00b8050 con 0x7ffbb00606e0 2020-11-20T12:17:50.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.041+0000 7ffbb5d99700 1 -- 172.21.15.187:0/2860004758 learned_addr learned my addr 172.21.15.187:0/2860004758 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:50.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.042+0000 7ffbb5d99700 1 -- 172.21.15.187:0/2860004758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb00606e0 msgr2=0x7ffbb0139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.042+0000 7ffbb5d99700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb00606e0 0x7ffbb0139300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.042+0000 7ffbb5d99700 1 -- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb0142b10 msgr2=0x7ffbb0157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.042+0000 7ffbb5d99700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb0142b10 0x7ffbb0157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.042+0000 7ffbb5d99700 1 -- 172.21.15.187:0/2860004758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb00b8280 con 0x7ffbb0063860 2020-11-20T12:17:50.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.043+0000 7ffbb659a700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb00606e0 0x7ffbb0139300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:17:50.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.043+0000 7ffbb6d9b700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb0142b10 0x7ffbb0157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:17:50.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.043+0000 7ffbb5d99700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0063860 0x7ffbb0157050 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7ffba000c710 tx=0x7ffba0007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.044+0000 7ffb9e7fc700 1 -- 172.21.15.187:0/2860004758 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba00072e0 con 0x7ffbb0063860 2020-11-20T12:17:50.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.044+0000 7ffb9e7fc700 1 -- 172.21.15.187:0/2860004758 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba0005530 con 0x7ffbb0063860 2020-11-20T12:17:50.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.044+0000 7ffb9e7fc700 1 -- 172.21.15.187:0/2860004758 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba00058b0 con 0x7ffbb0063860 2020-11-20T12:17:50.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.045+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb00b8520 con 0x7ffbb0063860 2020-11-20T12:17:50.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.045+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb01433f0 con 0x7ffbb0063860 2020-11-20T12:17:50.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.046+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb00b86b0 con 0x7ffbb0063860 2020-11-20T12:17:50.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.051+0000 7ffb9e7fc700 1 -- 172.21.15.187:0/2860004758 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffba0010070 con 0x7ffbb0063860 2020-11-20T12:17:50.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.052+0000 7ffb9e7fc700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb88056d20 0x7ffb880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.052+0000 7ffb9e7fc700 1 -- 172.21.15.187:0/2860004758 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ffba006f820 con 0x7ffbb0063860 2020-11-20T12:17:50.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.052+0000 7ffbb659a700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb88056d20 0x7ffb880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.053+0000 7ffbb659a700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb88056d20 0x7ffb880591e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7ffbac00b1b0 tx=0x7ffbac015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.057+0000 7ffb9e7fc700 1 -- 172.21.15.187:0/2860004758 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffba0044b90 con 0x7ffbb0063860 2020-11-20T12:17:50.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.316+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbb0069660 con 0x7ffbb0063860 2020-11-20T12:17:50.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.318+0000 7ffb9e7fc700 1 -- 172.21.15.187:0/2860004758 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7ffba00481b0 con 0x7ffbb0063860 2020-11-20T12:17:50.322 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:50.322 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:50.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb88056d20 msgr2=0x7ffb880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb88056d20 0x7ffb880591e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7ffbac00b1b0 tx=0x7ffbac015000).stop 2020-11-20T12:17:50.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0063860 msgr2=0x7ffbb0157050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0063860 0x7ffbb0157050 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7ffba000c710 tx=0x7ffba0007ae0).stop 2020-11-20T12:17:50.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 shutdown_connections 2020-11-20T12:17:50.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ffb88056d20 0x7ffb880591e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbb0142b10 0x7ffbb0157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffbb00606e0 0x7ffbb0139300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 --2- 172.21.15.187:0/2860004758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffbb0063860 0x7ffbb0157050 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.320+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 >> 172.21.15.187:0/2860004758 conn(0x7ffbb009d640 msgr2=0x7ffbb0142170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:50.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.321+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 shutdown_connections 2020-11-20T12:17:50.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.321+0000 7ffbb87fe700 1 -- 172.21.15.187:0/2860004758 wait complete. 2020-11-20T12:17:50.327 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:50.348 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:50.607 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2843,"msgr_send_messages":2845,"msgr_recv_bytes":892487,"msgr_send_bytes":10241952,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.476220937,"msgr_running_send_time":0.195027731,"msgr_running_recv_time":0.107018799,"msgr_running_fast_dispatch_time":0.135331239,"msgr_send_messages_queue_lat":{"avgcount":2843,"sum":0.481399503,"avgtime":0.000169327},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9637,"msgr_send_messages":14739,"msgr_recv_bytes":2740716,"msgr_send_bytes":14254091,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.058520881,"msgr_running_send_time":2.254352369,"msgr_running_recv_time":0.966408726,"msgr_running_fast_dispatch_time":0.098658345,"msgr_send_messages_queue_lat":{"avgcount":14738,"sum":1.946177607,"avgtime":0.000132051},"msgr_handle_ack_lat":{"avgcount":8626,"sum":0.006569689,"avgtime":0.000000761}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1093,"msgr_send_messages":1090,"msgr_recv_bytes":364266,"msgr_send_bytes":676227,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.126109474,"msgr_running_send_time":0.043024291,"msgr_running_recv_time":0.043579447,"msgr_running_fast_dispatch_time":0.032704293,"msgr_send_messages_queue_lat":{"avgcount":1090,"sum":0.163816876,"avgtime":0.000150290},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3131,"sum":3.810735479,"avgtime":0.001217098}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4196,"sum":0.420337324,"avgtime":0.000100175}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":232,"inodes_top":0,"inodes_bottom":232,"inodes_pin_tail":0,"inodes_pinned":232,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":12,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4826,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4383,"handle_client_caps_dirty":1500,"handle_client_cap_release":20,"process_request_cap_release":1668},"mds_cache":{"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1601,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9474,"evex":9025,"evtrm":9025,"ev":453,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21928135,"rdpos":4195632,"jlat":{"avgcount":1546,"sum":18.418870032,"avgtime":0.011913887},"replayed":4},"mds_mem":{"ino":173,"ino+":1614,"ino-":1441,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":1,"cap+":1602,"cap-":1601,"rss":72688,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1251},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2601487,"buffer_anon_items":922,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1029568,"mds_co_items":18254,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5764,"op_send_bytes":18880707,"op_resend":0,"op_reply":5764,"oplen_avg":{"avgcount":5764,"sum":6375},"op":5764,"op_r":42,"op_w":5722,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":24,"osdop_write":1562,"osdop_writefull":19,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3803,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":197,"omap_rd":28,"omap_del":18},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1601,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13569,"get_sum":2896997,"get_or_fail_fail":0,"get_or_fail_success":13569,"take":0,"take_sum":0,"put":13569,"put_sum":2896997,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5764,"take_sum":19044302,"put":1948,"put_sum":19044302,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5764,"take_sum":5764,"put":5764,"put_sum":5764,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1601,"get_sum":161709,"get_or_fail_fail":0,"get_or_fail_success":1601,"take":0,"take_sum":0,"put":16,"put_sum":161709,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":95840,"max":3758096384,"get_started":0,"get":9474,"get_sum":17731721,"get_or_fail_fail":0,"get_or_fail_success":9474,"take":0,"take_sum":0,"put":1546,"put_sum":17635881,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:50.609 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 892487, "msgr_recv_messages": 2843, "msgr_running_fast_dispatch_time": 0.135331239, "msgr_running_recv_time": 0.107018799, "msgr_running_send_time": 0.195027731, "msgr_running_total_time": 0.476220937, "msgr_send_bytes": 10241952, "msgr_send_messages": 2845, "msgr_send_messages_queue_lat": { "avgcount": 2843, "avgtime": 0.000169327, "sum": 0.481399503 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8626, "avgtime": 7.61e-07, "sum": 0.006569689 }, "msgr_recv_bytes": 2740716, "msgr_recv_messages": 9637, "msgr_running_fast_dispatch_time": 0.098658345, "msgr_running_recv_time": 0.966408726, "msgr_running_send_time": 2.254352369, "msgr_running_total_time": 3.058520881, "msgr_send_bytes": 14254091, "msgr_send_messages": 14739, "msgr_send_messages_queue_lat": { "avgcount": 14738, "avgtime": 0.000132051, "sum": 1.946177607 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 364266, "msgr_recv_messages": 1093, "msgr_running_fast_dispatch_time": 0.032704293, "msgr_running_recv_time": 0.043579447, "msgr_running_send_time": 0.043024291, "msgr_running_total_time": 0.126109474, "msgr_send_bytes": 676227, "msgr_send_messages": 1090, "msgr_send_messages_queue_lat": { "avgcount": 1090, "avgtime": 0.00015029, "sum": 0.163816876 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3131, "avgtime": 0.001217098, "sum": 3.810735479 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4196, "avgtime": 0.000100175, "sum": 0.420337324 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4826, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 20, "handle_client_caps": 4383, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 232, "inodes_bottom": 232, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 232, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 12, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1601, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 453, "evadd": 9474, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1546, "avgtime": 0.011913887, "sum": 18.418870032 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21928135 }, "mds_mem": { "cap": 1, "cap+": 1602, "cap-": 1601, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 173, "ino+": 1614, "ino-": 1441, "rss": 72688 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1251, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2601487, "buffer_anon_items": 922, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1029568, "mds_co_items": 18254, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 18, "omap_rd": 28, "omap_wr": 197, "op": 5764, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 42, "op_reply": 5764, "op_resend": 0, "op_rmw": 0, "op_send": 5764, "op_send_bytes": 18880707, "op_w": 5722, "oplen_avg": { "avgcount": 5764, "sum": 6375 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3803, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 24, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1562, "osdop_writefull": 19, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1601, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13569, "get_or_fail_fail": 0, "get_or_fail_success": 13569, "get_started": 0, "get_sum": 2896997, "max": 104857600, "put": 13569, "put_sum": 2896997, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1948, "put_sum": 19044302, "take": 5764, "take_sum": 19044302, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5764, "put_sum": 5764, "take": 5764, "take_sum": 5764, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1601, "get_or_fail_fail": 0, "get_or_fail_success": 1601, "get_started": 0, "get_sum": 161709, "max": 3758096384, "put": 16, "put_sum": 161709, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9474, "get_or_fail_fail": 0, "get_or_fail_success": 9474, "get_started": 0, "get_sum": 17731721, "max": 3758096384, "put": 1546, "put_sum": 17635881, "take": 0, "take_sum": 0, "val": 95840, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:50.610 INFO:tasks.cephfs.test_strays:Complete purge in 14 seconds 2020-11-20T12:17:50.610 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:50.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.854+0000 7f7cd9d17700 1 Processor -- start 2020-11-20T12:17:50.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.854+0000 7f7cd9d17700 1 -- start start 2020-11-20T12:17:50.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.856+0000 7f7cd9d17700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7cd409b420 0x7f7cd4063c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.856+0000 7f7cd9d17700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd4062f90 con 0x7f7cd40607d0 2020-11-20T12:17:50.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.856+0000 7f7cd9d17700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd4063230 con 0x7f7cd4067ec0 2020-11-20T12:17:50.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.856+0000 7f7cd9d17700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd40634d0 con 0x7f7cd409b420 2020-11-20T12:17:50.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.856+0000 7f7cd8d15700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7cd4067ec0 0x7f7cd409ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47924/0 (socket says 172.21.15.187:47924) 2020-11-20T12:17:50.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.856+0000 7f7cd8d15700 1 -- 172.21.15.187:0/1969471841 learned_addr learned my addr 172.21.15.187:0/1969471841 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:50.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd9516700 1 --2- 172.21.15.187:0/1969471841 >> v2:172.21.15.5:3300/0 conn(0x7f7cd409b420 0x7f7cd4063c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548093876 0 0) 0x7f7cd4063230 con 0x7f7cd4067ec0 2020-11-20T12:17:50.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb8003240 con 0x7f7cd4067ec0 2020-11-20T12:17:50.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3045727197 0 0) 0x7f7cb8003240 con 0x7f7cd4067ec0 2020-11-20T12:17:50.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 >> v2:172.21.15.5:3300/0 conn(0x7f7cd409b420 msgr2=0x7f7cd4063c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:50.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 --2- 172.21.15.187:0/1969471841 >> v2:172.21.15.5:3300/0 conn(0x7f7cd409b420 0x7f7cd4063c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 >> v1:172.21.15.5:6789/0 conn(0x7f7cd40607d0 legacy=0x7f7cd40b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:50.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd40682a0 con 0x7f7cd4067ec0 2020-11-20T12:17:50.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f7cc80014e0 con 0x7f7cd4067ec0 2020-11-20T12:17:50.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 >> v1:172.21.15.187:6789/0 conn(0x7f7cd4067ec0 legacy=0x7f7cd409ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb8003730 0x7f7cb8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb80044e0 0x7f7cb8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb8004e50 0x7f7cb800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40634d0 con 0x7f7cb8003730 2020-11-20T12:17:50.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8004260 con 0x7f7cb80044e0 2020-11-20T12:17:50.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.857+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb800bc20 con 0x7f7cb8004e50 2020-11-20T12:17:50.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd3fff700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb80044e0 0x7f7cb8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd8d15700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb8003730 0x7f7cb8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd9516700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb8004e50 0x7f7cb800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd3fff700 1 -- 172.21.15.187:0/1969471841 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb8004e50 msgr2=0x7f7cb800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:50.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd3fff700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb8004e50 0x7f7cb800b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd3fff700 1 -- 172.21.15.187:0/1969471841 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb8003730 msgr2=0x7f7cb8003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:50.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd3fff700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb8003730 0x7f7cb8003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd3fff700 1 -- 172.21.15.187:0/1969471841 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb8003240 con 0x7f7cb80044e0 2020-11-20T12:17:50.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.858+0000 7f7cd9516700 1 -- 172.21.15.187:0/1969471841 reap_dead start 2020-11-20T12:17:50.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.859+0000 7f7cd3fff700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb80044e0 0x7f7cb8004960 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 rx=0x7f7cc00018f0 tx=0x7f7cc0007b90).ready entity=mon.1 client_cookie=a8e183b5b798ad62 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.859+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0007020 con 0x7f7cb80044e0 2020-11-20T12:17:50.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.859+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0007180 con 0x7f7cb80044e0 2020-11-20T12:17:50.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.859+0000 7f7cd37fe700 1 -- 172.21.15.187:0/1969471841 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000c830 con 0x7f7cb80044e0 2020-11-20T12:17:50.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.859+0000 7f7cd9d17700 1 -- 172.21.15.187:0/1969471841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb80044e0 msgr2=0x7f7cb8004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.859+0000 7f7cd9d17700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb80044e0 0x7f7cb8004960 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 rx=0x7f7cc00018f0 tx=0x7f7cc0007b90).stop 2020-11-20T12:17:50.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.860+0000 7f7cd9d17700 1 -- 172.21.15.187:0/1969471841 shutdown_connections 2020-11-20T12:17:50.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.860+0000 7f7cd9d17700 1 --2- 172.21.15.187:0/1969471841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb80044e0 0x7f7cb8004960 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.860+0000 7f7cd9d17700 1 -- 172.21.15.187:0/1969471841 >> 172.21.15.187:0/1969471841 conn(0x7f7cd40b9320 msgr2=0x7f7cd40677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:50.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.860+0000 7f7cd9d17700 1 -- 172.21.15.187:0/1969471841 shutdown_connections 2020-11-20T12:17:50.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.860+0000 7f7cd9d17700 1 -- 172.21.15.187:0/1969471841 wait complete. 2020-11-20T12:17:50.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.861+0000 7f7cd9d17700 1 Processor -- start 2020-11-20T12:17:50.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.861+0000 7f7cd9d17700 1 -- start start 2020-11-20T12:17:50.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.862+0000 7f7cd9d17700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb80044e0 0x7f7cd40a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.862+0000 7f7cd9d17700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cd414bfc0 0x7f7cd40b5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.862+0000 7f7cd9d17700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cd4143d00 0x7f7cd4144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.862+0000 7f7cd9d17700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40b3610 con 0x7f7cd414bfc0 2020-11-20T12:17:50.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.862+0000 7f7cd9d17700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40b38b0 con 0x7f7cd4143d00 2020-11-20T12:17:50.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.862+0000 7f7cd9d17700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40b3b50 con 0x7f7cb80044e0 2020-11-20T12:17:50.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cd4143d00 0x7f7cd4144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cd4143d00 0x7f7cd4144140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51334/0 (socket says 172.21.15.187:51334) 2020-11-20T12:17:50.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 -- 172.21.15.187:0/3663680743 learned_addr learned my addr 172.21.15.187:0/3663680743 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:50.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd8d15700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb80044e0 0x7f7cd40a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd3fff700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cd414bfc0 0x7f7cd40b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 -- 172.21.15.187:0/3663680743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb80044e0 msgr2=0x7f7cd40a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb80044e0 0x7f7cd40a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 -- 172.21.15.187:0/3663680743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cd414bfc0 msgr2=0x7f7cd40b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cd414bfc0 0x7f7cd40b5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 -- 172.21.15.187:0/3663680743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd409f9f0 con 0x7f7cd4143d00 2020-11-20T12:17:50.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.863+0000 7f7cd9516700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cd4143d00 0x7f7cd4144140 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 rx=0x7f7ccc0090e0 tx=0x7f7ccc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.864+0000 7f7cd0ff9700 1 -- 172.21.15.187:0/3663680743 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc018070 con 0x7f7cd4143d00 2020-11-20T12:17:50.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.864+0000 7f7cd0ff9700 1 -- 172.21.15.187:0/3663680743 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc00d830 con 0x7f7cd4143d00 2020-11-20T12:17:50.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.864+0000 7f7cd0ff9700 1 -- 172.21.15.187:0/3663680743 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc00dbf0 con 0x7f7cd4143d00 2020-11-20T12:17:50.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.864+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd414c440 con 0x7f7cd4143d00 2020-11-20T12:17:50.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.864+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd40b31e0 con 0x7f7cd4143d00 2020-11-20T12:17:50.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.865+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd40b3d10 con 0x7f7cd4143d00 2020-11-20T12:17:50.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.866+0000 7f7cd0ff9700 1 -- 172.21.15.187:0/3663680743 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7ccc01d410 con 0x7f7cd4143d00 2020-11-20T12:17:50.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.866+0000 7f7cd0ff9700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cac056d50 0x7f7cac059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.866+0000 7f7cd0ff9700 1 -- 172.21.15.187:0/3663680743 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7ccc0701f0 con 0x7f7cd4143d00 2020-11-20T12:17:50.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.866+0000 7f7cd8d15700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cac056d50 0x7f7cac059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.867+0000 7f7cd8d15700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cac056d50 0x7f7cac059210 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f7cc8000fb0 tx=0x7f7cc8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:50.871+0000 7f7cd0ff9700 1 -- 172.21.15.187:0/3663680743 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7ccc045510 con 0x7f7cd4143d00 2020-11-20T12:17:51.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.122+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cd4044f60 con 0x7f7cd4143d00 2020-11-20T12:17:51.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.122+0000 7f7cd0ff9700 1 -- 172.21.15.187:0/3663680743 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f7ccc048b30 con 0x7f7cd4143d00 2020-11-20T12:17:51.126 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:51.127 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:51.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.125+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cac056d50 msgr2=0x7f7cac059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:51.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.125+0000 7f7cd9d17700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cac056d50 0x7f7cac059210 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f7cc8000fb0 tx=0x7f7cc8009040).stop 2020-11-20T12:17:51.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cd4143d00 msgr2=0x7f7cd4144140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:51.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cd4143d00 0x7f7cd4144140 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 rx=0x7f7ccc0090e0 tx=0x7f7ccc008040).stop 2020-11-20T12:17:51.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 shutdown_connections 2020-11-20T12:17:51.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cac056d50 0x7f7cac059210 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:51.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cd4143d00 0x7f7cd4144140 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:51.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7cb80044e0 0x7f7cd40a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:51.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 --2- 172.21.15.187:0/3663680743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cd414bfc0 0x7f7cd40b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:51.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 >> 172.21.15.187:0/3663680743 conn(0x7f7cd40b9320 msgr2=0x7f7cd40bb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:51.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 shutdown_connections 2020-11-20T12:17:51.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:51.126+0000 7f7cd9d17700 1 -- 172.21.15.187:0/3663680743 wait complete. 2020-11-20T12:17:51.133 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:51.154 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2020-11-20T12:17:51.426 INFO:teuthology.orchestra.run.smithi187.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2843,"msgr_send_messages":2845,"msgr_recv_bytes":892487,"msgr_send_bytes":10241952,"msgr_created_connections":7,"msgr_active_connections":1,"msgr_running_total_time":0.476220937,"msgr_running_send_time":0.195027731,"msgr_running_recv_time":0.107018799,"msgr_running_fast_dispatch_time":0.135331239,"msgr_send_messages_queue_lat":{"avgcount":2843,"sum":0.481399503,"avgtime":0.000169327},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9638,"msgr_send_messages":14739,"msgr_recv_bytes":2740916,"msgr_send_bytes":14254091,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":3.058709707,"msgr_running_send_time":2.254443995,"msgr_running_recv_time":0.966448611,"msgr_running_fast_dispatch_time":0.098711224,"msgr_send_messages_queue_lat":{"avgcount":14738,"sum":1.946177607,"avgtime":0.000132051},"msgr_handle_ack_lat":{"avgcount":8626,"sum":0.006569689,"avgtime":0.000000761}},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1093,"msgr_send_messages":1090,"msgr_recv_bytes":364266,"msgr_send_bytes":676227,"msgr_created_connections":4,"msgr_active_connections":2,"msgr_running_total_time":0.126109474,"msgr_running_send_time":0.043024291,"msgr_running_recv_time":0.043579447,"msgr_running_fast_dispatch_time":0.032704293,"msgr_send_messages_queue_lat":{"avgcount":1090,"sum":0.163816876,"avgtime":0.000150290},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3131,"sum":3.810735479,"avgtime":0.001217098}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4196,"sum":0.420337324,"avgtime":0.000100175}},"mds":{"request":3268,"reply":3268,"reply_latency":{"avgcount":3268,"sum":6.389932592,"avgtime":0.001955303},"forward":0,"dir_fetch":12,"dir_commit":85,"dir_split":9,"dir_merge":0,"inodes":232,"inodes_top":0,"inodes_bottom":232,"inodes_pin_tail":0,"inodes_pinned":232,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":3273,"traverse_hit":3270,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":12,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":44,"ceph_cap_op_grant":4826,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4383,"handle_client_caps_dirty":1500,"handle_client_cap_release":20,"process_request_cap_release":1668},"mds_cache":{"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1601,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":9,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":9474,"evex":9025,"evtrm":9025,"ev":453,"evexg":0,"evexd":9025,"segadd":12,"segex":12,"segtrm":12,"seg":13,"segexg":0,"segexd":12,"expos":21349436,"wrpos":21928135,"rdpos":4195632,"jlat":{"avgcount":1546,"sum":18.418870032,"avgtime":0.011913887},"replayed":4},"mds_mem":{"ino":173,"ino+":1614,"ino-":1441,"dir":12,"dir+":85,"dir-":73,"dn":237,"dn+":3212,"dn-":2975,"cap":1,"cap+":1602,"cap-":1601,"rss":72688,"heap":356572},"mds_server":{"dispatch_client_request":3310,"dispatch_server_request":0,"handle_client_request":3268,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.708117084,"avgtime":0.001692573},"req_getattr_latency":{"avgcount":2,"sum":0.000713489,"avgtime":0.000356744},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.007156462,"avgtime":0.007156462},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":64,"sum":0.061747873,"avgtime":0.000964810},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":1.362164870,"avgtime":1.362164870},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":2.250032814,"avgtime":0.001406270},"cap_revoke_eviction":0,"cap_acquisition_throttle":0},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2707,"average_load":2707,"avg_session_uptime":1251},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_Buffer_bytes":0,"bluestore_Buffer_items":0,"bluestore_Extent_bytes":0,"bluestore_Extent_items":0,"bluestore_Blob_bytes":0,"bluestore_Blob_items":0,"bluestore_SharedBlob_bytes":0,"bluestore_SharedBlob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2601487,"buffer_anon_items":922,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6128,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1029568,"mds_co_items":18254,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5764,"op_send_bytes":18880707,"op_resend":0,"op_reply":5764,"oplen_avg":{"avgcount":5764,"sum":6375},"op":5764,"op_r":42,"op_w":5722,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":232,"osdop_read":24,"osdop_write":1562,"osdop_writefull":19,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":1,"osdop_delete":3803,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":14,"osdop_setxattr":464,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":197,"omap_rd":28,"omap_del":18},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":65,"pq_executing":0,"pq_executing_high_water":11,"pq_executed":1601,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13570,"get_sum":2897123,"get_or_fail_fail":0,"get_or_fail_success":13570,"take":0,"take_sum":0,"put":13570,"put_sum":2897123,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5764,"take_sum":19044302,"put":1948,"put_sum":19044302,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5764,"take_sum":5764,"put":5764,"put_sum":5764,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1601,"get_sum":161709,"get_or_fail_fail":0,"get_or_fail_success":1601,"take":0,"take_sum":0,"put":16,"put_sum":161709,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5650f2b94140":{"val":95840,"max":3758096384,"get_started":0,"get":9474,"get_sum":17731721,"get_or_fail_fail":0,"get_or_fail_success":9474,"take":0,"take_sum":0,"put":1546,"put_sum":17635881,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2020-11-20T12:17:51.429 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 892487, "msgr_recv_messages": 2843, "msgr_running_fast_dispatch_time": 0.135331239, "msgr_running_recv_time": 0.107018799, "msgr_running_send_time": 0.195027731, "msgr_running_total_time": 0.476220937, "msgr_send_bytes": 10241952, "msgr_send_messages": 2845, "msgr_send_messages_queue_lat": { "avgcount": 2843, "avgtime": 0.000169327, "sum": 0.481399503 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 8626, "avgtime": 7.61e-07, "sum": 0.006569689 }, "msgr_recv_bytes": 2740916, "msgr_recv_messages": 9638, "msgr_running_fast_dispatch_time": 0.098711224, "msgr_running_recv_time": 0.966448611, "msgr_running_send_time": 2.254443995, "msgr_running_total_time": 3.058709707, "msgr_send_bytes": 14254091, "msgr_send_messages": 14739, "msgr_send_messages_queue_lat": { "avgcount": 14738, "avgtime": 0.000132051, "sum": 1.946177607 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 2, "msgr_created_connections": 4, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 364266, "msgr_recv_messages": 1093, "msgr_running_fast_dispatch_time": 0.032704293, "msgr_running_recv_time": 0.043579447, "msgr_running_send_time": 0.043024291, "msgr_running_total_time": 0.126109474, "msgr_send_bytes": 676227, "msgr_send_messages": 1090, "msgr_send_messages_queue_lat": { "avgcount": 1090, "avgtime": 0.00015029, "sum": 0.163816876 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3131, "avgtime": 0.001217098, "sum": 3.810735479 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4196, "avgtime": 0.000100175, "sum": 0.420337324 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4826, "ceph_cap_op_revoke": 44, "ceph_cap_op_trunc": 0, "dir_commit": 85, "dir_fetch": 12, "dir_merge": 0, "dir_split": 9, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 20, "handle_client_caps": 4383, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 232, "inodes_bottom": 232, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 232, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 12, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1668, "q": 0, "reply": 3268, "reply_latency": { "avgcount": 3268, "avgtime": 0.001955303, "sum": 6.389932592 }, "request": 3268, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "subtrees": 2, "traverse": 3273, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3270, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 9, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1601, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 453, "evadd": 9474, "evex": 9025, "evexd": 9025, "evexg": 0, "evtrm": 9025, "expos": 21349436, "jlat": { "avgcount": 1546, "avgtime": 0.011913887, "sum": 18.418870032 }, "rdpos": 4195632, "replayed": 4, "seg": 13, "segadd": 12, "segex": 12, "segexd": 12, "segexg": 0, "segtrm": 12, "wrpos": 21928135 }, "mds_mem": { "cap": 1, "cap+": 1602, "cap-": 1601, "dir": 12, "dir+": 85, "dir-": 73, "dn": 237, "dn+": 3212, "dn-": 2975, "heap": 356572, "ino": 173, "ino+": 1614, "ino-": 1441, "rss": 72688 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3310, "dispatch_server_request": 0, "handle_client_request": 3268, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001692573, "sum": 2.708117084 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000356744, "sum": 0.000713489 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.007156462, "sum": 0.007156462 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 64, "avgtime": 0.00096481, "sum": 0.061747873 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 1.36216487, "sum": 1.36216487 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.00140627, "sum": 2.250032814 } }, "mds_sessions": { "average_load": 2707, "avg_session_uptime": 1251, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2707 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2601487, "buffer_anon_items": 922, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1029568, "mds_co_items": 18254, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6128, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 18, "omap_rd": 28, "omap_wr": 197, "op": 5764, "op_active": 0, "op_laggy": 0, "op_pg": 0, "op_r": 42, "op_reply": 5764, "op_resend": 0, "op_rmw": 0, "op_send": 5764, "op_send_bytes": 18880707, "op_w": 5722, "oplen_avg": { "avgcount": 5764, "sum": 6375 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 232, "osdop_delete": 3803, "osdop_getxattr": 14, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 24, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 464, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 1, "osdop_watch": 0, "osdop_write": 1562, "osdop_writefull": 19, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1601, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 65, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13570, "get_or_fail_fail": 0, "get_or_fail_success": 13570, "get_started": 0, "get_sum": 2897123, "max": 104857600, "put": 13570, "put_sum": 2897123, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1948, "put_sum": 19044302, "take": 5764, "take_sum": 19044302, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5764, "put_sum": 5764, "take": 5764, "take_sum": 5764, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1601, "get_or_fail_fail": 0, "get_or_fail_success": 1601, "get_started": 0, "get_sum": 161709, "max": 3758096384, "put": 16, "put_sum": 161709, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5650f2b94140": { "get": 9474, "get_or_fail_fail": 0, "get_or_fail_success": 9474, "get_started": 0, "get_sum": 17731721, "max": 3758096384, "put": 1546, "put_sum": 17635881, "take": 0, "take_sum": 0, "val": 95840, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2020-11-20T12:17:51.430 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:17:51.431 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:17:51.566 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:17:51.587 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:17:51.702 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:17:51.722 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:17:51.723 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:17:51.724 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:17:51.724 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:17:51.887 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:17:51.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:17:51.889 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:17:51.950 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:17:51.959 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:17:52.148 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:17:52.149 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:17:52.149 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:17:52.149 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:17:52.150 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.150 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:17:52.150 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.150 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:17:52.150 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.151 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:17:52.151 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:17:52.151 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.151 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:17:52.151 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:17:52.152 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.152 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:17:52.152 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:17:52.152 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.152 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:17:52.153 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:17:52.153 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.153 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:17:52.153 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.153 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:17:52.154 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:17:52.154 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:17:52.154 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:17:52.154 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:17:52.154 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:17:52.155 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:17:52.155 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:17:52.155 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:17:52.155 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:17:52.156 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.156 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:17:52.156 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.156 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:17:52.156 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.157 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:17:52.157 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.157 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:17:52.157 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:17:52.157 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:17:52.158 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:17:52.158 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:17:52.158 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:17:52.158 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.158 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:17:52.159 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:17:52.159 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.159 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:17:52.159 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:17:52.159 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.160 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:17:52.160 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:17:52.160 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.160 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:17:52.161 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:17:52.161 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:17:52.161 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.161 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:17:52.161 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:17:52.162 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.162 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:17:52.162 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.162 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:17:52.162 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.163 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:17:52.163 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.163 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:17:52.163 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:17:52.163 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.164 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:17:52.164 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:17:52.164 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:17:52.164 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:17:52.164 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:17:52.165 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:17:52.165 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:17:52.165 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:17:52.165 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:17:52.165 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:17:52.166 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:17:52.166 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:17:52.166 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:17:52.166 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:17:52.166 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:17:52.167 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:17:52.167 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:17:52.167 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:17:52.167 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:17:52.167 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:17:52.168 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.168 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:17:52.168 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:52.168 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:17:52.168 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:17:52.169 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:17:52.169 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.169 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.169 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:17:52.169 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.170 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:17:52.170 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:52.170 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:17:52.170 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:17:52.170 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:17:52.171 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:17:52.171 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:17:52.171 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.171 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:17:52.171 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:17:52.172 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.172 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:17:52.172 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.172 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:17:52.173 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:17:52.173 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:17:52.173 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:17:52.173 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:17:52.173 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:17:52.174 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:17:52.174 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:17:52.174 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:52.174 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:17:52.174 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.175 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.175 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:17:52.175 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:17:52.175 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:17:52.175 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:17:52.176 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:17:52.176 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:17:52.176 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:17:52.176 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:52.177 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:17:52.177 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:17:52.177 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:17:52.177 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:17:52.177 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:17:52.178 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:17:52.178 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:17:52.178 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:17:52.178 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:17:52.179 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:17:52.179 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.179 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.179 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 100000000 2020-11-20T12:17:52.179 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops = 16 2020-11-20T12:17:52.180 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:17:52.180 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:17:52.181 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:17:52.181 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:17:52.181 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.181 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:17:52.181 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.182 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:17:52.182 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.182 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:17:52.182 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:17:52.182 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.183 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:17:52.183 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:17:52.183 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.183 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:17:52.184 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:17:52.184 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.184 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:17:52.184 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:17:52.184 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:17:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:17:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:17:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:17:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:17:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:17:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:17:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:17:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:17:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:17:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:17:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:17:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:17:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:17:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:17:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:17:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:17:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:17:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:17:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:17:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.190 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:17:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:17:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:17:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:17:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:17:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:17:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:17:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:17:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:17:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:17:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:17:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:17:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:17:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:17:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:17:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:17:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:17:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:17:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:17:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:17:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:17:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:17:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:17:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:17:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:17:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:17:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:17:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:17:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:17:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:17:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:17:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:17:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:17:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:17:52.199 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:17:52.199 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:17:52.199 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.199 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:17:52.199 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:17:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:17:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:17:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.201 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:17:52.201 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.201 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:17:52.201 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.201 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:17:52.202 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:17:52.202 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:17:52.202 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:17:52.202 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:17:52.203 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.203 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:17:52.203 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:17:52.203 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.203 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:17:52.203 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.204 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:17:52.204 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:17:52.204 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:17:52.204 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:17:52.204 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:17:52.205 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:17:52.205 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:17:52.205 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:17:52.205 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.206 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:17:52.206 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.206 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.206 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:17:52.206 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:17:52.207 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:17:52.207 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:17:52.207 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:17:52.207 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:17:52.207 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:17:52.208 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.208 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:17:52.208 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:17:52.208 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:17:52.208 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:17:52.209 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:17:52.209 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:17:52.209 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:17:52.209 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:17:52.209 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:17:52.210 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:17:52.210 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.210 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.210 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 100000000 2020-11-20T12:17:52.210 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops = 16 2020-11-20T12:17:52.224 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:17:52.233 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:17:52.431 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:17:52.432 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:17:52.432 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:17:52.432 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:17:52.433 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.433 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:17:52.433 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.433 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:17:52.434 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.434 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:17:52.435 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:17:52.435 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.435 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:17:52.436 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:17:52.436 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.436 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:17:52.436 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:17:52.437 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.437 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:17:52.437 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:17:52.437 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.438 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:17:52.438 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.438 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:17:52.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:17:52.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:17:52.439 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:17:52.440 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:17:52.440 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:17:52.440 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:17:52.440 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:17:52.441 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:17:52.441 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:17:52.441 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.441 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:17:52.441 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.442 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:17:52.442 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.442 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:17:52.442 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.443 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:17:52.443 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:17:52.443 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:17:52.443 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:17:52.444 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:17:52.444 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:17:52.444 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.444 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:17:52.445 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:17:52.445 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.445 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:17:52.445 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:17:52.446 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.446 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:17:52.446 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:17:52.447 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.447 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:17:52.447 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:17:52.447 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:17:52.448 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.448 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:17:52.448 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:17:52.449 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.449 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:17:52.449 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.450 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:17:52.450 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.450 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:17:52.450 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.451 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:17:52.451 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:17:52.451 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.451 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:17:52.452 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:17:52.452 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:17:52.452 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:17:52.452 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:17:52.453 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:17:52.453 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:17:52.453 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:17:52.454 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:17:52.454 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:17:52.454 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:17:52.454 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:17:52.455 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:17:52.455 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:17:52.455 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:17:52.455 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:17:52.455 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:17:52.456 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:17:52.456 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:17:52.456 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:17:52.457 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.457 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:17:52.457 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:52.457 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:17:52.458 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:17:52.458 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:17:52.458 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:17:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.459 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:17:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:17:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:17:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:17:52.461 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:17:52.461 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:17:52.461 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.462 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:17:52.462 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:17:52.462 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.462 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:17:52.463 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.463 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:17:52.463 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:17:52.463 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:17:52.464 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:17:52.464 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:17:52.464 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:17:52.464 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:17:52.465 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:17:52.465 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:52.465 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:17:52.465 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.466 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.466 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:17:52.466 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:17:52.466 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:17:52.467 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:17:52.467 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:17:52.467 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:17:52.467 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:17:52.468 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:52.468 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:17:52.468 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:17:52.468 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:17:52.468 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:17:52.469 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:17:52.469 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:17:52.469 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:17:52.469 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:17:52.470 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:17:52.470 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:17:52.470 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.470 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.471 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_ops = 16 2020-11-20T12:17:52.471 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:17:52.472 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:17:52.472 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:17:52.472 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:17:52.473 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.473 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:17:52.473 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.473 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:17:52.474 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.474 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:17:52.474 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:17:52.474 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.474 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:17:52.475 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:17:52.475 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.475 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:17:52.475 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:17:52.476 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.476 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:17:52.476 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:17:52.476 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.477 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:17:52.477 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.477 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:17:52.477 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:17:52.478 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:17:52.478 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:17:52.478 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:17:52.478 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:17:52.479 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:17:52.479 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:17:52.479 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:17:52.479 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:17:52.480 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.480 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:17:52.480 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.480 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:17:52.480 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.481 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:17:52.481 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.481 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:17:52.482 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:17:52.482 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:17:52.483 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:17:52.483 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:17:52.483 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:17:52.483 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.484 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:17:52.484 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:17:52.484 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.484 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:17:52.485 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:17:52.485 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.485 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:17:52.485 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:17:52.486 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.486 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:17:52.486 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:17:52.486 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:17:52.486 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.487 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:17:52.487 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:17:52.487 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.487 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:17:52.488 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.488 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:17:52.488 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.488 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:17:52.488 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.489 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:17:52.489 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:17:52.489 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.489 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:17:52.490 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:17:52.490 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:17:52.490 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:17:52.490 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:17:52.491 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:17:52.491 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:17:52.491 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:17:52.491 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:17:52.492 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:17:52.492 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:17:52.492 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:17:52.492 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:17:52.493 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:17:52.493 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:17:52.493 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:17:52.493 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:17:52.494 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:17:52.494 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:17:52.494 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:17:52.494 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.494 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:17:52.495 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.495 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:17:52.496 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:17:52.496 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:17:52.496 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.496 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.497 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:17:52.497 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.497 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:17:52.497 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.498 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:17:52.498 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:17:52.498 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:17:52.498 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:17:52.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:17:52.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:17:52.499 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:17:52.500 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.500 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:17:52.500 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.500 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:17:52.501 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:17:52.501 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:17:52.502 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:17:52.502 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:17:52.502 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:17:52.502 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:17:52.503 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:17:52.503 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.503 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:17:52.504 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.504 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.504 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:17:52.504 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:17:52.505 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:17:52.505 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:17:52.505 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:17:52.505 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:17:52.505 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:17:52.506 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.506 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:17:52.506 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:17:52.506 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:17:52.507 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:17:52.507 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:17:52.507 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:17:52.507 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:17:52.508 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:17:52.508 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:17:52.508 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:17:52.508 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.508 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.509 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_ops = 16 2020-11-20T12:17:52.511 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:17:52.521 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:17:52.658 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:52.665 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:52.721 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:17:52.721 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:17:52.722 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:17:52.722 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:17:52.722 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.723 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:17:52.723 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.723 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:17:52.724 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.724 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:17:52.724 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:17:52.725 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.725 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:17:52.725 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:17:52.726 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.726 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:17:52.727 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:17:52.727 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.727 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:17:52.728 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:17:52.728 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.728 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:17:52.729 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.729 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:17:52.729 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:17:52.730 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:17:52.730 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:17:52.730 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:17:52.731 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:17:52.731 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:17:52.731 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:17:52.732 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:17:52.732 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:17:52.733 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.733 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:17:52.733 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.734 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:17:52.734 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.734 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:17:52.735 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.735 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:17:52.736 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:17:52.736 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:17:52.736 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:17:52.737 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:17:52.738 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:17:52.738 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.738 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:17:52.738 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:17:52.739 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.739 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:17:52.739 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:17:52.739 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.740 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:17:52.740 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:17:52.740 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.740 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:17:52.740 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:17:52.741 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:17:52.741 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.741 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:17:52.741 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:17:52.741 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.742 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:17:52.742 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.742 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:17:52.742 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.742 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:17:52.743 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.743 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:17:52.743 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:17:52.743 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.743 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:17:52.744 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:17:52.744 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:17:52.744 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:17:52.744 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:17:52.744 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:17:52.745 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:17:52.745 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:17:52.745 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:17:52.745 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:17:52.745 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:17:52.746 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:17:52.746 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:17:52.747 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:17:52.747 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:17:52.747 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:17:52.748 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:17:52.748 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:17:52.748 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:17:52.749 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:17:52.749 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.749 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:17:52.749 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.749 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:17:52.750 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:17:52.750 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:17:52.751 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.751 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.751 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:17:52.752 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.752 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:17:52.752 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.752 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:17:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:17:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:17:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:17:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:17:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:17:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:17:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:17:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:17:52.755 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:17:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:17:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:17:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:17:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:17:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:17:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:17:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:17:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:17:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.757 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:17:52.758 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:17:52.758 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:17:52.758 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:17:52.758 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:17:52.758 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:17:52.759 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:17:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:17:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:17:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:17:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:17:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:17:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:17:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:17:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:17:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:17:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:17:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:17:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:17:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:17:52.986 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:17:52.986 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:17:52.987 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:17:52.987 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:17:52.988 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.989 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:17:52.990 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.990 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:17:52.990 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.990 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:17:52.991 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:17:52.991 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.991 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:17:52.991 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:17:52.992 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.992 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:17:52.992 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:17:52.992 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.993 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:17:52.993 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:17:52.993 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.993 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:17:52.993 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.994 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:17:52.994 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:17:52.994 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:17:52.994 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:17:52.994 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:17:52.995 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:17:52.995 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:17:52.995 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:17:52.995 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:17:52.995 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:17:52.996 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.996 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:17:52.996 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.996 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:17:52.996 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.997 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:17:52.997 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.997 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:17:52.997 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:17:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:17:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:17:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:17:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:17:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.999 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:17:52.999 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:17:52.999 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:52.999 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:17:52.999 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:17:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:17:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:17:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:17:53.001 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:17:53.001 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:17:53.001 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.001 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:17:53.002 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:17:53.002 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.002 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:17:53.002 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:17:53.004 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:17:53.004 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.004 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:17:53.005 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:17:53.005 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:17:53.005 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:17:53.005 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:17:53.006 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:17:53.006 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:17:53.006 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:17:53.006 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:17:53.006 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:17:53.007 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:17:53.007 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:17:53.007 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:17:53.007 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:17:53.007 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:17:53.008 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:17:53.008 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:17:53.008 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:17:53.008 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:17:53.009 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:17:53.009 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.009 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:17:53.009 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:53.009 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:17:53.010 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:17:53.010 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:17:53.010 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:53.010 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:53.011 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:17:53.011 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.011 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:17:53.011 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:53.011 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:17:53.012 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:17:53.012 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:17:53.012 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:17:53.012 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:17:53.012 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:17:53.013 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:17:53.013 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:17:53.013 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:17:53.013 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:17:53.013 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:53.014 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:17:53.014 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:17:53.014 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:17:53.014 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:17:53.014 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:17:53.015 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:17:53.015 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:17:53.015 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:17:53.015 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:53.015 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:17:53.016 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:17:53.016 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:17:53.016 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:17:53.016 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:17:53.016 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:17:53.017 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:17:53.017 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:17:53.017 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:17:53.017 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:17:53.017 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:17:53.018 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:17:53.018 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:17:53.018 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:17:53.018 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:17:53.019 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:17:53.019 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:17:53.019 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:17:53.019 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:17:53.019 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:17:53.020 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:17:53.020 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:17:53.020 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:17:53.049 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:17:53.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.288+0000 7f27cffff700 1 Processor -- start 2020-11-20T12:17:53.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.288+0000 7f27cffff700 1 -- start start 2020-11-20T12:17:53.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.290+0000 7f27cffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f27d014d270 0x7f27d01573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.290+0000 7f27cffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d00658a0 con 0x7f27d00b1760 2020-11-20T12:17:53.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.290+0000 7f27cffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d0150390 con 0x7f27d00b1380 2020-11-20T12:17:53.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.290+0000 7f27cffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d01511e0 con 0x7f27d014d270 2020-11-20T12:17:53.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.290+0000 7f27cf7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f27d014d270 0x7f27d01573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.290+0000 7f27ceffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f27d00b1380 0x7f27d0150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47944/0 (socket says 172.21.15.187:47944) 2020-11-20T12:17:53.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.290+0000 7f27ceffd700 1 -- 172.21.15.187:0/3578448434 learned_addr learned my addr 172.21.15.187:0/3578448434 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:53.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.291+0000 7f27cf7fe700 1 -- 172.21.15.187:0/3578448434 >> v1:172.21.15.187:6789/0 conn(0x7f27d00b1380 legacy=0x7f27d0150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.291+0000 7f27cf7fe700 1 -- 172.21.15.187:0/3578448434 >> v1:172.21.15.5:6789/0 conn(0x7f27d00b1760 legacy=0x7f27d0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.291+0000 7f27cdffb700 1 -- 172.21.15.187:0/3578448434 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295408148 0 0) 0x7f27d0150390 con 0x7f27d00b1380 2020-11-20T12:17:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.291+0000 7f27cf7fe700 1 -- 172.21.15.187:0/3578448434 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d0150630 con 0x7f27d014d270 2020-11-20T12:17:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.291+0000 7f27cf7fe700 1 --2- 172.21.15.187:0/3578448434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d014d270 0x7f27d01573a0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7f27c8009c90 tx=0x7f27c80091d0).ready entity=mon.1 client_cookie=4b51f828b7afc910 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.291+0000 7f27cdffb700 1 -- 172.21.15.187:0/3578448434 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c800bb80 con 0x7f27d014d270 2020-11-20T12:17:53.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.291+0000 7f27cdffb700 1 -- 172.21.15.187:0/3578448434 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c800bce0 con 0x7f27d014d270 2020-11-20T12:17:53.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.291+0000 7f27cdffb700 1 -- 172.21.15.187:0/3578448434 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8011610 con 0x7f27d014d270 2020-11-20T12:17:53.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.292+0000 7f27cffff700 1 -- 172.21.15.187:0/3578448434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d014d270 msgr2=0x7f27d01573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.292+0000 7f27cffff700 1 --2- 172.21.15.187:0/3578448434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d014d270 0x7f27d01573a0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7f27c8009c90 tx=0x7f27c80091d0).stop 2020-11-20T12:17:53.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.292+0000 7f27cffff700 1 -- 172.21.15.187:0/3578448434 shutdown_connections 2020-11-20T12:17:53.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.292+0000 7f27cffff700 1 --2- 172.21.15.187:0/3578448434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d014d270 0x7f27d01573a0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.292+0000 7f27cffff700 1 -- 172.21.15.187:0/3578448434 >> 172.21.15.187:0/3578448434 conn(0x7f27d00bab30 msgr2=0x7f27d00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:53.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.292+0000 7f27cffff700 1 -- 172.21.15.187:0/3578448434 shutdown_connections 2020-11-20T12:17:53.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.292+0000 7f27cffff700 1 -- 172.21.15.187:0/3578448434 wait complete. 2020-11-20T12:17:53.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.294+0000 7f27cffff700 1 Processor -- start 2020-11-20T12:17:53.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.294+0000 7f27cffff700 1 -- start start 2020-11-20T12:17:53.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.295+0000 7f27cffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27d00b1380 0x7f27d00a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.295+0000 7f27cffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d00b1760 0x7f27d009ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.295+0000 7f27cffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27d014d270 0x7f27d0148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.295+0000 7f27cffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d00a0550 con 0x7f27d00b1380 2020-11-20T12:17:53.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.295+0000 7f27cffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d009f3f0 con 0x7f27d00b1760 2020-11-20T12:17:53.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.295+0000 7f27cffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d009f680 con 0x7f27d014d270 2020-11-20T12:17:53.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ce7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d00b1760 0x7f27d009ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ce7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d00b1760 0x7f27d009ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51344/0 (socket says 172.21.15.187:51344) 2020-11-20T12:17:53.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ce7fc700 1 -- 172.21.15.187:0/3573554801 learned_addr learned my addr 172.21.15.187:0/3573554801 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:53.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27cf7fe700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27d014d270 0x7f27d0148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ceffd700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27d00b1380 0x7f27d00a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ce7fc700 1 -- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27d014d270 msgr2=0x7f27d0148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ce7fc700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27d014d270 0x7f27d0148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ce7fc700 1 -- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27d00b1380 msgr2=0x7f27d00a0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ce7fc700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27d00b1380 0x7f27d00a0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ce7fc700 1 -- 172.21.15.187:0/3573554801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d009f920 con 0x7f27d00b1760 2020-11-20T12:17:53.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.296+0000 7f27ceffd700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27d00b1380 0x7f27d00a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:53.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.297+0000 7f27ce7fc700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d00b1760 0x7f27d009ef00 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f27c400c730 tx=0x7f27c4007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.297+0000 7f27b6ffd700 1 -- 172.21.15.187:0/3573554801 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c400e560 con 0x7f27d00b1760 2020-11-20T12:17:53.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.297+0000 7f27b6ffd700 1 -- 172.21.15.187:0/3573554801 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c400e6e0 con 0x7f27d00b1760 2020-11-20T12:17:53.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.297+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d009fbc0 con 0x7f27d00b1760 2020-11-20T12:17:53.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.297+0000 7f27b6ffd700 1 -- 172.21.15.187:0/3573554801 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c400eaa0 con 0x7f27d00b1760 2020-11-20T12:17:53.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.297+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d0149690 con 0x7f27d00b1760 2020-11-20T12:17:53.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.298+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d0061a20 con 0x7f27d00b1760 2020-11-20T12:17:53.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.299+0000 7f27b6ffd700 1 -- 172.21.15.187:0/3573554801 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f27c4019030 con 0x7f27d00b1760 2020-11-20T12:17:53.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.299+0000 7f27b6ffd700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f27ac056d20 0x7f27ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.300+0000 7f27b6ffd700 1 -- 172.21.15.187:0/3573554801 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f27c406f410 con 0x7f27d00b1760 2020-11-20T12:17:53.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.300+0000 7f27ceffd700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f27ac056d20 0x7f27ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.300+0000 7f27ceffd700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f27ac056d20 0x7f27ac0591e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f27c0001550 tx=0x7f27c0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.303+0000 7f27b6ffd700 1 -- 172.21.15.187:0/3573554801 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f27c4043fd0 con 0x7f27d00b1760 2020-11-20T12:17:53.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.489+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f27d0149690 con 0x7f27d00b1760 2020-11-20T12:17:53.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.499+0000 7f27b6ffd700 1 -- 172.21.15.187:0/3573554801 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f27d0149690 con 0x7f27d00b1760 2020-11-20T12:17:53.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.501+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f27ac056d20 msgr2=0x7f27ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.501+0000 7f27cffff700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f27ac056d20 0x7f27ac0591e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f27c0001550 tx=0x7f27c0009040).stop 2020-11-20T12:17:53.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.501+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d00b1760 msgr2=0x7f27d009ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.501+0000 7f27cffff700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d00b1760 0x7f27d009ef00 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f27c400c730 tx=0x7f27c4007c30).stop 2020-11-20T12:17:53.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.502+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 shutdown_connections 2020-11-20T12:17:53.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.502+0000 7f27cffff700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f27ac056d20 0x7f27ac0591e0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.502+0000 7f27cffff700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27d00b1380 0x7f27d00a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.502+0000 7f27cffff700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27d014d270 0x7f27d0148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.502+0000 7f27cffff700 1 --2- 172.21.15.187:0/3573554801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d00b1760 0x7f27d009ef00 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.502+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 >> 172.21.15.187:0/3573554801 conn(0x7f27d00bab30 msgr2=0x7f27d014dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:53.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.502+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 shutdown_connections 2020-11-20T12:17:53.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.502+0000 7f27cffff700 1 -- 172.21.15.187:0/3573554801 wait complete. 2020-11-20T12:17:53.527 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_ops_throttle' 2020-11-20T12:17:53.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.761+0000 7fe753fff700 1 Processor -- start 2020-11-20T12:17:53.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.761+0000 7fe753fff700 1 -- start start 2020-11-20T12:17:53.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.762+0000 7fe753fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.763+0000 7fe753fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe7541482d0 0x7fe754062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.763+0000 7fe753fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe754144b80 con 0x7fe754062b50 2020-11-20T12:17:53.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.763+0000 7fe753fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7541489e0 con 0x7fe7541482d0 2020-11-20T12:17:53.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.763+0000 7fe753fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe754148c80 con 0x7fe7540b57b0 2020-11-20T12:17:53.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.763+0000 7fe7527fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe7541482d0 0x7fe754062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.764+0000 7fe752ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.764+0000 7fe7527fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe7541482d0 0x7fe754062660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:43618/0 (socket says 172.21.15.187:43618) 2020-11-20T12:17:53.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.764+0000 7fe7527fc700 1 -- 172.21.15.187:0/915512530 learned_addr learned my addr 172.21.15.187:0/915512530 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:53.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.764+0000 7fe7537fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe754062b50 0x7fe754065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:41472/0 (socket says 172.21.15.187:41472) 2020-11-20T12:17:53.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.765+0000 7fe752ffd700 1 -- 172.21.15.187:0/915512530 >> v2:172.21.15.5:3300/0 conn(0x7fe7541482d0 msgr2=0x7fe754062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:53.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.765+0000 7fe752ffd700 1 --2- 172.21.15.187:0/915512530 >> v2:172.21.15.5:3300/0 conn(0x7fe7541482d0 0x7fe754062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.765+0000 7fe752ffd700 1 -- 172.21.15.187:0/915512530 >> v1:172.21.15.5:6789/0 conn(0x7fe754062b50 legacy=0x7fe754065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.765+0000 7fe752ffd700 1 -- 172.21.15.187:0/915512530 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe754148f20 con 0x7fe7540b57b0 2020-11-20T12:17:53.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.765+0000 7fe752ffd700 1 --2- 172.21.15.187:0/915512530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754068230 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe74400b9a0 tx=0x7fe744006df0).ready entity=mon.2 client_cookie=d69ba802affb867e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.766+0000 7fe751ffb700 1 -- 172.21.15.187:0/915512530 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7440097b0 con 0x7fe7540b57b0 2020-11-20T12:17:53.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.766+0000 7fe751ffb700 1 -- 172.21.15.187:0/915512530 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe744009910 con 0x7fe7540b57b0 2020-11-20T12:17:53.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.766+0000 7fe751ffb700 1 -- 172.21.15.187:0/915512530 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744009c90 con 0x7fe7540b57b0 2020-11-20T12:17:53.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.767+0000 7fe753fff700 1 -- 172.21.15.187:0/915512530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 msgr2=0x7fe754068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.767+0000 7fe753fff700 1 --2- 172.21.15.187:0/915512530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754068230 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe74400b9a0 tx=0x7fe744006df0).stop 2020-11-20T12:17:53.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.767+0000 7fe753fff700 1 -- 172.21.15.187:0/915512530 shutdown_connections 2020-11-20T12:17:53.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.767+0000 7fe753fff700 1 --2- 172.21.15.187:0/915512530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754068230 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.767+0000 7fe753fff700 1 --2- 172.21.15.187:0/915512530 >> v2:172.21.15.5:3300/0 conn(0x7fe7541482d0 0x7fe754062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.767+0000 7fe753fff700 1 -- 172.21.15.187:0/915512530 >> 172.21.15.187:0/915512530 conn(0x7fe7540bab30 msgr2=0x7fe7540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:53.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.767+0000 7fe753fff700 1 -- 172.21.15.187:0/915512530 shutdown_connections 2020-11-20T12:17:53.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.767+0000 7fe753fff700 1 -- 172.21.15.187:0/915512530 wait complete. 2020-11-20T12:17:53.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.769+0000 7fe753fff700 1 Processor -- start 2020-11-20T12:17:53.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.769+0000 7fe753fff700 1 -- start start 2020-11-20T12:17:53.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.770+0000 7fe753fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe754062b50 0x7fe754138e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.770+0000 7fe753fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.770+0000 7fe753fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7541482d0 0x7fe75413c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.770+0000 7fe753fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541468d0 con 0x7fe754062b50 2020-11-20T12:17:53.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.770+0000 7fe753fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe754146b70 con 0x7fe7541482d0 2020-11-20T12:17:53.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.770+0000 7fe753fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe754147500 con 0x7fe7540b57b0 2020-11-20T12:17:53.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7537fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7541482d0 0x7fe75413c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7537fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7541482d0 0x7fe75413c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51360/0 (socket says 172.21.15.187:51360) 2020-11-20T12:17:53.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7537fe700 1 -- 172.21.15.187:0/1440119922 learned_addr learned my addr 172.21.15.187:0/1440119922 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:53.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe752ffd700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe754062b50 0x7fe754138e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7527fc700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7537fe700 1 -- 172.21.15.187:0/1440119922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 msgr2=0x7fe754147010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7537fe700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754147010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7537fe700 1 -- 172.21.15.187:0/1440119922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe754062b50 msgr2=0x7fe754138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7537fe700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe754062b50 0x7fe754138e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.771+0000 7fe7537fe700 1 -- 172.21.15.187:0/1440119922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75413c950 con 0x7fe7541482d0 2020-11-20T12:17:53.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.772+0000 7fe7537fe700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7541482d0 0x7fe75413c460 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 rx=0x7fe74c00ca90 tx=0x7fe74c007f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.772+0000 7fe73affd700 1 -- 172.21.15.187:0/1440119922 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c005ac0 con 0x7fe7541482d0 2020-11-20T12:17:53.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.772+0000 7fe73affd700 1 -- 172.21.15.187:0/1440119922 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c005c20 con 0x7fe7541482d0 2020-11-20T12:17:53.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.772+0000 7fe73affd700 1 -- 172.21.15.187:0/1440119922 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c007110 con 0x7fe7541482d0 2020-11-20T12:17:53.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.772+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7541479c0 con 0x7fe7541482d0 2020-11-20T12:17:53.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.772+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7540b2490 con 0x7fe7541482d0 2020-11-20T12:17:53.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.774+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7540b2620 con 0x7fe7541482d0 2020-11-20T12:17:53.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.774+0000 7fe73affd700 1 -- 172.21.15.187:0/1440119922 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe74c010030 con 0x7fe7541482d0 2020-11-20T12:17:53.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.775+0000 7fe73affd700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe730056d20 0x7fe7300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.775+0000 7fe73affd700 1 -- 172.21.15.187:0/1440119922 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe74c06ff90 con 0x7fe7541482d0 2020-11-20T12:17:53.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.775+0000 7fe752ffd700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe730056d20 0x7fe7300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.776+0000 7fe752ffd700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe730056d20 0x7fe7300591e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fe744006e30 tx=0x7fe744015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:53.779+0000 7fe73affd700 1 -- 172.21.15.187:0/1440119922 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe74c045390 con 0x7fe7541482d0 2020-11-20T12:17:54.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.030+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_ops_throttle"]} v 0) v1 -- 0x7fe754061a20 con 0x7fe7541482d0 2020-11-20T12:17:54.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.557+0000 7fe73affd700 1 -- 172.21.15.187:0/1440119922 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_ops_throttle"]}]=0 v647) v1 ==== 132+0+0 (secure 0 0 0) 0x7fe74c0489b0 con 0x7fe7541482d0 2020-11-20T12:17:54.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.560+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe730056d20 msgr2=0x7fe7300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:54.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.560+0000 7fe753fff700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe730056d20 0x7fe7300591e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fe744006e30 tx=0x7fe744015000).stop 2020-11-20T12:17:54.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.560+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7541482d0 msgr2=0x7fe75413c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:54.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.560+0000 7fe753fff700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7541482d0 0x7fe75413c460 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 rx=0x7fe74c00ca90 tx=0x7fe74c007f10).stop 2020-11-20T12:17:54.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.561+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 shutdown_connections 2020-11-20T12:17:54.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.561+0000 7fe753fff700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe730056d20 0x7fe7300591e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.561+0000 7fe753fff700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe754062b50 0x7fe754138e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.561+0000 7fe753fff700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7541482d0 0x7fe75413c460 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.561+0000 7fe753fff700 1 --2- 172.21.15.187:0/1440119922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7540b57b0 0x7fe754147010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.561+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 >> 172.21.15.187:0/1440119922 conn(0x7fe7540bab30 msgr2=0x7fe7540b9630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:54.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.561+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 shutdown_connections 2020-11-20T12:17:54.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.561+0000 7fe753fff700 1 -- 172.21.15.187:0/1440119922 wait complete. 2020-11-20T12:17:54.589 INFO:tasks.cephfs_test_runner:test_ops_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:17:54.590 INFO:tasks.cephfs_test_runner:Starting test: test_purge_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:17:54.591 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown' 2020-11-20T12:17:54.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.838+0000 7fcaebfff700 1 Processor -- start 2020-11-20T12:17:54.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.839+0000 7fcaebfff700 1 -- start start 2020-11-20T12:17:54.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.840+0000 7fcaebfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcaec1484d0 0x7fcaec061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.840+0000 7fcaebfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec066f70 0x7fcaec0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.840+0000 7fcaebfff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaec0b1300 con 0x7fcaec0622c0 2020-11-20T12:17:54.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.840+0000 7fcaebfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaec067730 con 0x7fcaec1484d0 2020-11-20T12:17:54.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.840+0000 7fcaebfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec0b15a0 con 0x7fcaec066f70 2020-11-20T12:17:54.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.840+0000 7fcae27fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcaec0622c0 0x7fcaec0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47974/0 (socket says 172.21.15.187:47974) 2020-11-20T12:17:54.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.840+0000 7fcae27fc700 1 -- 172.21.15.187:0/4095690837 learned_addr learned my addr 172.21.15.187:0/4095690837 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:54.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.841+0000 7fcaeb7fe700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec066f70 0x7fcaec0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.841+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/4095690837 >> v2:172.21.15.5:3300/0 conn(0x7fcaec1484d0 0x7fcaec061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.841+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847543162 0 0) 0x7fcaec0b1300 con 0x7fcaec0622c0 2020-11-20T12:17:54.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.841+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcacc003240 con 0x7fcaec0622c0 2020-11-20T12:17:54.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4011912507 0 0) 0x7fcacc003240 con 0x7fcaec0622c0 2020-11-20T12:17:54.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec066f70 msgr2=0x7fcaec0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:54.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec066f70 0x7fcaec0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 >> v2:172.21.15.5:3300/0 conn(0x7fcaec1484d0 msgr2=0x7fcaec061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:54.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 --2- 172.21.15.187:0/4095690837 >> v2:172.21.15.5:3300/0 conn(0x7fcaec1484d0 0x7fcaec061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaec0b1840 con 0x7fcaec0622c0 2020-11-20T12:17:54.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcad40014e0 con 0x7fcaec0622c0 2020-11-20T12:17:54.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 >> v1:172.21.15.187:6789/0 conn(0x7fcaec0622c0 legacy=0x7fcaec0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:54.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc0037b0 0x7fcacc003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec066f70 0x7fcacc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.842+0000 7fcaea7fc700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc0049d0 0x7fcacc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.843+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec067730 con 0x7fcacc0049d0 2020-11-20T12:17:54.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.843+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec0b15a0 con 0x7fcacc0037b0 2020-11-20T12:17:54.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.843+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc0042a0 con 0x7fcaec066f70 2020-11-20T12:17:54.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.843+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/4095690837 >> v2:172.21.15.5:3300/0 conn(0x7fcaec1484d0 0x7fcaec061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:17:54.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.843+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc0037b0 0x7fcacc003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.843+0000 7fcae27fc700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec066f70 0x7fcacc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.844+0000 7fcaeb7fe700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc0049d0 0x7fcacc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.844+0000 7fcaeaffd700 1 -- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec066f70 msgr2=0x7fcacc0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:54.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.844+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec066f70 0x7fcacc0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.844+0000 7fcaeaffd700 1 -- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc0049d0 msgr2=0x7fcacc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:54.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.844+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcacc0049d0 0x7fcacc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.844+0000 7fcaeaffd700 1 -- 172.21.15.187:0/4095690837 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc003240 con 0x7fcacc0037b0 2020-11-20T12:17:54.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.844+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc0037b0 0x7fcacc003bd0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 rx=0x7fcadc00be60 tx=0x7fcadc006df0).ready entity=mon.1 client_cookie=67319300328f3018 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:54.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.845+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00b590 con 0x7fcacc0037b0 2020-11-20T12:17:54.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.845+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcadc00ecf0 con 0x7fcacc0037b0 2020-11-20T12:17:54.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.845+0000 7fcaea7fc700 1 -- 172.21.15.187:0/4095690837 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc012600 con 0x7fcacc0037b0 2020-11-20T12:17:54.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.845+0000 7fcaebfff700 1 -- 172.21.15.187:0/4095690837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc0037b0 msgr2=0x7fcacc003bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:54.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.845+0000 7fcaebfff700 1 --2- 172.21.15.187:0/4095690837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcacc0037b0 0x7fcacc003bd0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 rx=0x7fcadc00be60 tx=0x7fcadc006df0).stop 2020-11-20T12:17:54.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.845+0000 7fcaeb7fe700 1 -- 172.21.15.187:0/4095690837 reap_dead start 2020-11-20T12:17:54.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.845+0000 7fcaebfff700 1 -- 172.21.15.187:0/4095690837 shutdown_connections 2020-11-20T12:17:54.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.845+0000 7fcaebfff700 1 -- 172.21.15.187:0/4095690837 >> 172.21.15.187:0/4095690837 conn(0x7fcaec0bab30 msgr2=0x7fcaec0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:54.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.846+0000 7fcaebfff700 1 -- 172.21.15.187:0/4095690837 shutdown_connections 2020-11-20T12:17:54.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.846+0000 7fcaebfff700 1 -- 172.21.15.187:0/4095690837 wait complete. 2020-11-20T12:17:54.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.847+0000 7fcaebfff700 1 Processor -- start 2020-11-20T12:17:54.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.847+0000 7fcaebfff700 1 -- start start 2020-11-20T12:17:54.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.848+0000 7fcaebfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaec13f1a0 0x7fcaec09c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.848+0000 7fcaebfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec13f580 0x7fcaec13e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.848+0000 7fcaebfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec0b4380 0x7fcaec13e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.848+0000 7fcaebfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec09c6a0 con 0x7fcaec13f1a0 2020-11-20T12:17:54.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.848+0000 7fcaebfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec09c8c0 con 0x7fcaec13f580 2020-11-20T12:17:54.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.848+0000 7fcaebfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec13fa70 con 0x7fcaec0b4380 2020-11-20T12:17:54.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcae27fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec13f580 0x7fcaec13e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcae27fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec13f580 0x7fcaec13e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51380/0 (socket says 172.21.15.187:51380) 2020-11-20T12:17:54.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcae27fc700 1 -- 172.21.15.187:0/3034276637 learned_addr learned my addr 172.21.15.187:0/3034276637 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:54.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaec13f1a0 0x7fcaec09c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcae27fc700 1 -- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec0b4380 msgr2=0x7fcaec13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:54.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcaeb7fe700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec0b4380 0x7fcaec13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcae27fc700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec0b4380 0x7fcaec13e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcae27fc700 1 -- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaec13f1a0 msgr2=0x7fcaec09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:54.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.849+0000 7fcae27fc700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaec13f1a0 0x7fcaec09c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:54.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcae27fc700 1 -- 172.21.15.187:0/3034276637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaec13fd10 con 0x7fcaec13f580 2020-11-20T12:17:54.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaec13f1a0 0x7fcaec09c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:54.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcaeb7fe700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec0b4380 0x7fcaec13e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcae27fc700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec13f580 0x7fcaec13e040 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 rx=0x7fcad4004bc0 tx=0x7fcad4007750).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcae37fe700 1 -- 172.21.15.187:0/3034276637 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad400f9b0 con 0x7fcaec13f580 2020-11-20T12:17:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcae37fe700 1 -- 172.21.15.187:0/3034276637 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad400fb10 con 0x7fcaec13f580 2020-11-20T12:17:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaec1402e0 con 0x7fcaec13f580 2020-11-20T12:17:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcae37fe700 1 -- 172.21.15.187:0/3034276637 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad400feb0 con 0x7fcaec13f580 2020-11-20T12:17:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.850+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaec0b25c0 con 0x7fcaec13f580 2020-11-20T12:17:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.851+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaec13fea0 con 0x7fcaec13f580 2020-11-20T12:17:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.852+0000 7fcae37fe700 1 -- 172.21.15.187:0/3034276637 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcad401b030 con 0x7fcaec13f580 2020-11-20T12:17:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.853+0000 7fcae37fe700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcac8056d20 0x7fcac80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:54.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.853+0000 7fcae37fe700 1 -- 172.21.15.187:0/3034276637 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcad406f8d0 con 0x7fcaec13f580 2020-11-20T12:17:54.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.853+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcac8056d20 0x7fcac80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:54.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.854+0000 7fcaeaffd700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcac8056d20 0x7fcac80591e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fcadc008000 tx=0x7fcadc019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:54.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:54.857+0000 7fcae37fe700 1 -- 172.21.15.187:0/3034276637 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcad4044d80 con 0x7fcaec13f580 2020-11-20T12:17:55.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.111+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown"]} v 0) v1 -- 0x7fcaec1400d0 con 0x7fcaec13f580 2020-11-20T12:17:55.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.561+0000 7fcae37fe700 1 -- 172.21.15.187:0/3034276637 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown"]}]=0 v648) v1 ==== 140+0+0 (secure 0 0 0) 0x7fcad4019020 con 0x7fcaec13f580 2020-11-20T12:17:55.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcac8056d20 msgr2=0x7fcac80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcac8056d20 0x7fcac80591e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fcadc008000 tx=0x7fcadc019000).stop 2020-11-20T12:17:55.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec13f580 msgr2=0x7fcaec13e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec13f580 0x7fcaec13e040 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 rx=0x7fcad4004bc0 tx=0x7fcad4007750).stop 2020-11-20T12:17:55.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 shutdown_connections 2020-11-20T12:17:55.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcac8056d20 0x7fcac80591e0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaec13f1a0 0x7fcaec09c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcaec0b4380 0x7fcaec13e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 --2- 172.21.15.187:0/3034276637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec13f580 0x7fcaec13e040 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 >> 172.21.15.187:0/3034276637 conn(0x7fcaec0bab30 msgr2=0x7fcaec066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:55.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 shutdown_connections 2020-11-20T12:17:55.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.565+0000 7fcaebfff700 1 -- 172.21.15.187:0/3034276637 wait complete. 2020-11-20T12:17:55.597 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:17:55.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.841+0000 7fe555f66700 1 Processor -- start 2020-11-20T12:17:55.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.841+0000 7fe555f66700 1 -- start start 2020-11-20T12:17:55.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.843+0000 7fe555f66700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe550145cc0 0x7fe550066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.843+0000 7fe555f66700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe5500683f0 0x7fe550065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.843+0000 7fe555f66700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55014c170 con 0x7fe550065680 2020-11-20T12:17:55.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.843+0000 7fe555f66700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5501497f0 con 0x7fe550145cc0 2020-11-20T12:17:55.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.843+0000 7fe555f66700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55014c410 con 0x7fe5500683f0 2020-11-20T12:17:55.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.843+0000 7fe54ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe5500683f0 0x7fe550065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.843+0000 7fe555765700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe550065680 0x7fe550061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:47996/0 (socket says 172.21.15.187:47996) 2020-11-20T12:17:55.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.843+0000 7fe555765700 1 -- 172.21.15.187:0/2049537036 learned_addr learned my addr 172.21.15.187:0/2049537036 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:55.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe554f64700 1 --2- 172.21.15.187:0/2049537036 >> v2:172.21.15.5:3300/0 conn(0x7fe550145cc0 0x7fe550066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe54ffff700 1 -- 172.21.15.187:0/2049537036 >> v2:172.21.15.5:3300/0 conn(0x7fe550145cc0 msgr2=0x7fe550066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:55.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe54ffff700 1 --2- 172.21.15.187:0/2049537036 >> v2:172.21.15.5:3300/0 conn(0x7fe550145cc0 0x7fe550066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe54ffff700 1 -- 172.21.15.187:0/2049537036 >> v1:172.21.15.187:6789/0 conn(0x7fe550065680 legacy=0x7fe550061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe54f7fe700 1 -- 172.21.15.187:0/2049537036 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822486174 0 0) 0x7fe55014c170 con 0x7fe550065680 2020-11-20T12:17:55.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe54ffff700 1 -- 172.21.15.187:0/2049537036 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55014c6b0 con 0x7fe5500683f0 2020-11-20T12:17:55.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe54ffff700 1 --2- 172.21.15.187:0/2049537036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe550065190 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 rx=0x7fe5500674b0 tx=0x7fe53c006df0).ready entity=mon.1 client_cookie=d0a9f9db4c00f27a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:55.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe54f7fe700 1 -- 172.21.15.187:0/2049537036 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c0097b0 con 0x7fe5500683f0 2020-11-20T12:17:55.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.844+0000 7fe54f7fe700 1 -- 172.21.15.187:0/2049537036 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c009910 con 0x7fe5500683f0 2020-11-20T12:17:55.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.845+0000 7fe54f7fe700 1 -- 172.21.15.187:0/2049537036 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c009c90 con 0x7fe5500683f0 2020-11-20T12:17:55.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.845+0000 7fe555f66700 1 -- 172.21.15.187:0/2049537036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 msgr2=0x7fe550065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.845+0000 7fe555f66700 1 --2- 172.21.15.187:0/2049537036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe550065190 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 rx=0x7fe5500674b0 tx=0x7fe53c006df0).stop 2020-11-20T12:17:55.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.845+0000 7fe555f66700 1 -- 172.21.15.187:0/2049537036 shutdown_connections 2020-11-20T12:17:55.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.845+0000 7fe555f66700 1 --2- 172.21.15.187:0/2049537036 >> v2:172.21.15.5:3300/0 conn(0x7fe550145cc0 0x7fe550066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.845+0000 7fe555f66700 1 --2- 172.21.15.187:0/2049537036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe550065190 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.845+0000 7fe555f66700 1 -- 172.21.15.187:0/2049537036 >> 172.21.15.187:0/2049537036 conn(0x7fe5500bab30 msgr2=0x7fe5500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:55.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.846+0000 7fe555f66700 1 -- 172.21.15.187:0/2049537036 shutdown_connections 2020-11-20T12:17:55.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.846+0000 7fe555f66700 1 -- 172.21.15.187:0/2049537036 wait complete. 2020-11-20T12:17:55.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.847+0000 7fe555f66700 1 Processor -- start 2020-11-20T12:17:55.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.847+0000 7fe555f66700 1 -- start start 2020-11-20T12:17:55.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.848+0000 7fe555f66700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe550065680 0x7fe5501450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.848+0000 7fe555f66700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe5500b6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.848+0000 7fe555f66700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe550145cc0 0x7fe5500b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.848+0000 7fe555f66700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55009d6f0 con 0x7fe550145cc0 2020-11-20T12:17:55.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.848+0000 7fe555f66700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55009d450 con 0x7fe5500683f0 2020-11-20T12:17:55.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.848+0000 7fe555f66700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55009a780 con 0x7fe550065680 2020-11-20T12:17:55.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe54ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe5500b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe54ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe5500b6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51394/0 (socket says 172.21.15.187:51394) 2020-11-20T12:17:55.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe54ffff700 1 -- 172.21.15.187:0/4136531910 learned_addr learned my addr 172.21.15.187:0/4136531910 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:55.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe554f64700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe550065680 0x7fe5501450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe555765700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe550145cc0 0x7fe5500b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe54ffff700 1 -- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe550065680 msgr2=0x7fe5501450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe54ffff700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe550065680 0x7fe5501450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe54ffff700 1 -- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe550145cc0 msgr2=0x7fe5500b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe54ffff700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe550145cc0 0x7fe5500b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe54ffff700 1 -- 172.21.15.187:0/4136531910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5501455a0 con 0x7fe5500683f0 2020-11-20T12:17:55.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.849+0000 7fe555765700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe550145cc0 0x7fe5500b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:55.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.850+0000 7fe54ffff700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe5500b6700 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7fe53c00c010 tx=0x7fe53c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:55.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.850+0000 7fe54cff9700 1 -- 172.21.15.187:0/4136531910 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c011dd0 con 0x7fe5500683f0 2020-11-20T12:17:55.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.850+0000 7fe54cff9700 1 -- 172.21.15.187:0/4136531910 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c01e460 con 0x7fe5500683f0 2020-11-20T12:17:55.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.850+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55009ff20 con 0x7fe5500683f0 2020-11-20T12:17:55.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.850+0000 7fe54cff9700 1 -- 172.21.15.187:0/4136531910 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c01e820 con 0x7fe5500683f0 2020-11-20T12:17:55.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.850+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55009d100 con 0x7fe5500683f0 2020-11-20T12:17:55.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.851+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe550061a20 con 0x7fe5500683f0 2020-11-20T12:17:55.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.852+0000 7fe54cff9700 1 -- 172.21.15.187:0/4136531910 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe53c025030 con 0x7fe5500683f0 2020-11-20T12:17:55.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.852+0000 7fe54cff9700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe528056d20 0x7fe5280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.852+0000 7fe54cff9700 1 -- 172.21.15.187:0/4136531910 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe53c0716e0 con 0x7fe5500683f0 2020-11-20T12:17:55.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.853+0000 7fe554f64700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe528056d20 0x7fe5280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.853+0000 7fe554f64700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe528056d20 0x7fe5280591e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fe54400d8f0 tx=0x7fe544008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:55.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:55.857+0000 7fe54cff9700 1 -- 172.21.15.187:0/4136531910 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe53c046c10 con 0x7fe5500683f0 2020-11-20T12:17:56.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.045+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7fe55014bdf0 con 0x7fe5500683f0 2020-11-20T12:17:56.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.056+0000 7fe54cff9700 1 -- 172.21.15.187:0/4136531910 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe53c023070 con 0x7fe5500683f0 2020-11-20T12:17:56.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe528056d20 msgr2=0x7fe5280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:56.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe528056d20 0x7fe5280591e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fe54400d8f0 tx=0x7fe544008040).stop 2020-11-20T12:17:56.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 msgr2=0x7fe5500b6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:56.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe5500b6700 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7fe53c00c010 tx=0x7fe53c015000).stop 2020-11-20T12:17:56.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 shutdown_connections 2020-11-20T12:17:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe528056d20 0x7fe5280591e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe550145cc0 0x7fe5500b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe550065680 0x7fe5501450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 --2- 172.21.15.187:0/4136531910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5500683f0 0x7fe5500b6700 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:56.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.058+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 >> 172.21.15.187:0/4136531910 conn(0x7fe5500bab30 msgr2=0x7fe5500afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:56.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.059+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 shutdown_connections 2020-11-20T12:17:56.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:56.059+0000 7fe555f66700 1 -- 172.21.15.187:0/4136531910 wait complete. 2020-11-20T12:17:56.088 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:17:56.088 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:17:56.212 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:17:56.814 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:17:56.923 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:17:56.946 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:17:56.946 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:17:56.947 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:57.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.188+0000 7fa63f59e700 1 Processor -- start 2020-11-20T12:17:57.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.188+0000 7fa63f59e700 1 -- start start 2020-11-20T12:17:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.190+0000 7fa63f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640065190 0x7fa640065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.190+0000 7fa63f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa640065f90 0x7fa640143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.190+0000 7fa63f59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6400b13e0 con 0x7fa640143c70 2020-11-20T12:17:57.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.190+0000 7fa63f59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64014ab30 con 0x7fa640065f90 2020-11-20T12:17:57.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.190+0000 7fa63f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa640062c10 con 0x7fa640065190 2020-11-20T12:17:57.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.190+0000 7fa635d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa640065f90 0x7fa640143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.190+0000 7fa635d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa640065f90 0x7fa640143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51402/0 (socket says 172.21.15.187:51402) 2020-11-20T12:17:57.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.190+0000 7fa635d9b700 1 -- 172.21.15.187:0/762786423 learned_addr learned my addr 172.21.15.187:0/762786423 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:57.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.191+0000 7fa63e59c700 1 --2- 172.21.15.187:0/762786423 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640065190 0x7fa640065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.191+0000 7fa635d9b700 1 -- 172.21.15.187:0/762786423 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640065190 msgr2=0x7fa640065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.191+0000 7fa635d9b700 1 --2- 172.21.15.187:0/762786423 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640065190 0x7fa640065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.191+0000 7fa635d9b700 1 -- 172.21.15.187:0/762786423 >> v1:172.21.15.187:6789/0 conn(0x7fa640143c70 legacy=0x7fa64009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.191+0000 7fa635d9b700 1 -- 172.21.15.187:0/762786423 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6400b1640 con 0x7fa640065f90 2020-11-20T12:17:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.191+0000 7fa635d9b700 1 --2- 172.21.15.187:0/762786423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065f90 0x7fa640143780 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 rx=0x7fa64014ac70 tx=0x7fa628006eb0).ready entity=mon.1 client_cookie=f5294dd300c7040 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.191+0000 7fa63dd9b700 1 -- 172.21.15.187:0/762786423 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628009920 con 0x7fa640065f90 2020-11-20T12:17:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63dd9b700 1 -- 172.21.15.187:0/762786423 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628009a80 con 0x7fa640065f90 2020-11-20T12:17:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63dd9b700 1 -- 172.21.15.187:0/762786423 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628009e00 con 0x7fa640065f90 2020-11-20T12:17:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63f59e700 1 -- 172.21.15.187:0/762786423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065f90 msgr2=0x7fa640143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63f59e700 1 --2- 172.21.15.187:0/762786423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065f90 0x7fa640143780 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 rx=0x7fa64014ac70 tx=0x7fa628006eb0).stop 2020-11-20T12:17:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63f59e700 1 -- 172.21.15.187:0/762786423 shutdown_connections 2020-11-20T12:17:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63f59e700 1 --2- 172.21.15.187:0/762786423 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640065190 0x7fa640065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63f59e700 1 --2- 172.21.15.187:0/762786423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065f90 0x7fa640143780 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63f59e700 1 -- 172.21.15.187:0/762786423 >> 172.21.15.187:0/762786423 conn(0x7fa6400bab30 msgr2=0x7fa6400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63f59e700 1 -- 172.21.15.187:0/762786423 shutdown_connections 2020-11-20T12:17:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.192+0000 7fa63f59e700 1 -- 172.21.15.187:0/762786423 wait complete. 2020-11-20T12:17:57.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.194+0000 7fa63f59e700 1 Processor -- start 2020-11-20T12:17:57.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.194+0000 7fa63f59e700 1 -- start start 2020-11-20T12:17:57.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.195+0000 7fa63f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065190 0x7fa640142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa640065f90 0x7fa640066ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640143c70 0x7fa640066590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa640148f70 con 0x7fa640065f90 2020-11-20T12:17:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64015c660 con 0x7fa640065190 2020-11-20T12:17:57.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6401491a0 con 0x7fa640143c70 2020-11-20T12:17:57.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065190 0x7fa640142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065190 0x7fa640142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51406/0 (socket says 172.21.15.187:51406) 2020-11-20T12:17:57.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63e59c700 1 -- 172.21.15.187:0/2922421776 learned_addr learned my addr 172.21.15.187:0/2922421776 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:57.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63e59c700 1 -- 172.21.15.187:0/2922421776 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640143c70 msgr2=0x7fa640066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63e59c700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640143c70 0x7fa640066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa635d9b700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa640065f90 0x7fa640066ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63e59c700 1 -- 172.21.15.187:0/2922421776 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa640065f90 msgr2=0x7fa640066ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63e59c700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa640065f90 0x7fa640066ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.196+0000 7fa63e59c700 1 -- 172.21.15.187:0/2922421776 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64015c2a0 con 0x7fa640065190 2020-11-20T12:17:57.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.197+0000 7fa63e59c700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065190 0x7fa640142930 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7fa63000db60 tx=0x7fa630008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.197+0000 7fa636ffd700 1 -- 172.21.15.187:0/2922421776 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa630009790 con 0x7fa640065190 2020-11-20T12:17:57.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.197+0000 7fa636ffd700 1 -- 172.21.15.187:0/2922421776 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6300098f0 con 0x7fa640065190 2020-11-20T12:17:57.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.197+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64015c430 con 0x7fa640065190 2020-11-20T12:17:57.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.197+0000 7fa636ffd700 1 -- 172.21.15.187:0/2922421776 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa630009c90 con 0x7fa640065190 2020-11-20T12:17:57.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.197+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa640066a80 con 0x7fa640065190 2020-11-20T12:17:57.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.198+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa640061a20 con 0x7fa640065190 2020-11-20T12:17:57.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.199+0000 7fa636ffd700 1 -- 172.21.15.187:0/2922421776 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa630013480 con 0x7fa640065190 2020-11-20T12:17:57.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.199+0000 7fa636ffd700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa61c056d50 0x7fa61c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.199+0000 7fa636ffd700 1 -- 172.21.15.187:0/2922421776 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa6300705f0 con 0x7fa640065190 2020-11-20T12:17:57.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.200+0000 7fa635d9b700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa61c056d50 0x7fa61c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.200+0000 7fa635d9b700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa61c056d50 0x7fa61c059210 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fa64014ac70 tx=0x7fa628015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.203+0000 7fa636ffd700 1 -- 172.21.15.187:0/2922421776 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa630045910 con 0x7fa640065190 2020-11-20T12:17:57.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.453+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6400afc60 con 0x7fa640065190 2020-11-20T12:17:57.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.454+0000 7fa636ffd700 1 -- 172.21.15.187:0/2922421776 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7fa630048f30 con 0x7fa640065190 2020-11-20T12:17:57.460 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:57.460 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:17:57.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa61c056d50 msgr2=0x7fa61c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa61c056d50 0x7fa61c059210 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fa64014ac70 tx=0x7fa628015000).stop 2020-11-20T12:17:57.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065190 msgr2=0x7fa640142930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065190 0x7fa640142930 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7fa63000db60 tx=0x7fa630008040).stop 2020-11-20T12:17:57.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 shutdown_connections 2020-11-20T12:17:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa61c056d50 0x7fa61c059210 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa640143c70 0x7fa640066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:57.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa640065190 0x7fa640142930 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 --2- 172.21.15.187:0/2922421776 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa640065f90 0x7fa640066ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 >> 172.21.15.187:0/2922421776 conn(0x7fa6400bab30 msgr2=0x7fa6400aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:57.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 shutdown_connections 2020-11-20T12:17:57.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.457+0000 7fa63f59e700 1 -- 172.21.15.187:0/2922421776 wait complete. 2020-11-20T12:17:57.468 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:17:57.485 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:17:57.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.734+0000 7fb0e2faa700 1 Processor -- start 2020-11-20T12:17:57.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.736+0000 7fb0e2faa700 1 -- start start 2020-11-20T12:17:57.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0e2faa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb0dc0661f0 0x7fb0dc0b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0e2faa700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0dc0b5df0 con 0x7fb0dc065e10 2020-11-20T12:17:57.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0e2faa700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0dc09a440 con 0x7fb0dc099ce0 2020-11-20T12:17:57.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0e2faa700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0dc09a1a0 con 0x7fb0dc0661f0 2020-11-20T12:17:57.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0d3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb0dc0661f0 0x7fb0dc0b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0e1547700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb0dc099ce0 0x7fb0dc0b67d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48022/0 (socket says 172.21.15.187:48022) 2020-11-20T12:17:57.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0e1547700 1 -- 172.21.15.187:0/2914053977 learned_addr learned my addr 172.21.15.187:0/2914053977 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:57.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0d3fff700 1 -- 172.21.15.187:0/2914053977 >> v1:172.21.15.187:6789/0 conn(0x7fb0dc099ce0 legacy=0x7fb0dc0b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0d3fff700 1 -- 172.21.15.187:0/2914053977 >> v1:172.21.15.5:6789/0 conn(0x7fb0dc065e10 legacy=0x7fb0dc144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:57.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0d3fff700 1 -- 172.21.15.187:0/2914053977 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc0b0160 con 0x7fb0dc0661f0 2020-11-20T12:17:57.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.737+0000 7fb0d3fff700 1 --2- 172.21.15.187:0/2914053977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc0661f0 0x7fb0dc0b62e0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 rx=0x7fb0cc00b990 tx=0x7fb0cc006d90).ready entity=mon.1 client_cookie=5cdcc9b8d30d6054 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.738+0000 7fb0d37fe700 1 -- 172.21.15.187:0/2914053977 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc006030 con 0x7fb0dc0661f0 2020-11-20T12:17:57.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.738+0000 7fb0d37fe700 1 -- 172.21.15.187:0/2914053977 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc006190 con 0x7fb0dc0661f0 2020-11-20T12:17:57.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.738+0000 7fb0d37fe700 1 -- 172.21.15.187:0/2914053977 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc006510 con 0x7fb0dc0661f0 2020-11-20T12:17:57.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.738+0000 7fb0e2faa700 1 -- 172.21.15.187:0/2914053977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc0661f0 msgr2=0x7fb0dc0b62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.738+0000 7fb0e2faa700 1 --2- 172.21.15.187:0/2914053977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc0661f0 0x7fb0dc0b62e0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 rx=0x7fb0cc00b990 tx=0x7fb0cc006d90).stop 2020-11-20T12:17:57.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.738+0000 7fb0e2faa700 1 -- 172.21.15.187:0/2914053977 shutdown_connections 2020-11-20T12:17:57.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.738+0000 7fb0e2faa700 1 --2- 172.21.15.187:0/2914053977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc0661f0 0x7fb0dc0b62e0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.738+0000 7fb0e2faa700 1 -- 172.21.15.187:0/2914053977 >> 172.21.15.187:0/2914053977 conn(0x7fb0dc0bacd0 msgr2=0x7fb0dc0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:57.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.739+0000 7fb0e2faa700 1 -- 172.21.15.187:0/2914053977 shutdown_connections 2020-11-20T12:17:57.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.739+0000 7fb0e2faa700 1 -- 172.21.15.187:0/2914053977 wait complete. 2020-11-20T12:17:57.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.740+0000 7fb0e2faa700 1 Processor -- start 2020-11-20T12:17:57.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.741+0000 7fb0e2faa700 1 -- start start 2020-11-20T12:17:57.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e2faa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc065e10 0x7fb0dc1344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e2faa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0dc0661f0 0x7fb0dc0b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e2faa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0dc099ce0 0x7fb0dc09cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e2faa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc09dc80 con 0x7fb0dc099ce0 2020-11-20T12:17:57.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e2faa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc152cd0 con 0x7fb0dc065e10 2020-11-20T12:17:57.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e2faa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc152f70 con 0x7fb0dc0661f0 2020-11-20T12:17:57.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e0d46700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc065e10 0x7fb0dc1344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e0d46700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc065e10 0x7fb0dc1344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51420/0 (socket says 172.21.15.187:51420) 2020-11-20T12:17:57.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.742+0000 7fb0e0d46700 1 -- 172.21.15.187:0/1923212680 learned_addr learned my addr 172.21.15.187:0/1923212680 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:57.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0e0d46700 1 -- 172.21.15.187:0/1923212680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0dc0661f0 msgr2=0x7fb0dc0b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0e0d46700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0dc0661f0 0x7fb0dc0b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:57.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0e0d46700 1 -- 172.21.15.187:0/1923212680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0dc099ce0 msgr2=0x7fb0dc09cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0e0d46700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0dc099ce0 0x7fb0dc09cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:57.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0e0d46700 1 -- 172.21.15.187:0/1923212680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc153210 con 0x7fb0dc065e10 2020-11-20T12:17:57.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0e0d46700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc065e10 0x7fb0dc1344c0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7fb0d800c730 tx=0x7fb0d8007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0d0ff9700 1 -- 172.21.15.187:0/1923212680 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800e560 con 0x7fb0dc065e10 2020-11-20T12:17:57.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0d0ff9700 1 -- 172.21.15.187:0/1923212680 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d800e6c0 con 0x7fb0dc065e10 2020-11-20T12:17:57.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0dc1349b0 con 0x7fb0dc065e10 2020-11-20T12:17:57.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0d0ff9700 1 -- 172.21.15.187:0/1923212680 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800ea80 con 0x7fb0dc065e10 2020-11-20T12:17:57.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.743+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0dc0b2d50 con 0x7fb0dc065e10 2020-11-20T12:17:57.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.745+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0dc063600 con 0x7fb0dc065e10 2020-11-20T12:17:57.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.745+0000 7fb0d0ff9700 1 -- 172.21.15.187:0/1923212680 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb0d8010070 con 0x7fb0dc065e10 2020-11-20T12:17:57.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.746+0000 7fb0d0ff9700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb0b4056d20 0x7fb0b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.746+0000 7fb0d0ff9700 1 -- 172.21.15.187:0/1923212680 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb0d806fbd0 con 0x7fb0dc065e10 2020-11-20T12:17:57.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.746+0000 7fb0d3fff700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb0b4056d20 0x7fb0b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.747+0000 7fb0d3fff700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb0b4056d20 0x7fb0b40591e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fb0dc0b50a0 tx=0x7fb0cc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.750+0000 7fb0d0ff9700 1 -- 172.21.15.187:0/1923212680 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb0d8044fd0 con 0x7fb0dc065e10 2020-11-20T12:17:57.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.935+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0dc0666e0 con 0x7fb0dc065e10 2020-11-20T12:17:57.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.937+0000 7fb0d0ff9700 1 -- 172.21.15.187:0/1923212680 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16922 (secure 0 0 0) 0x7fb0d80485f0 con 0x7fb0dc065e10 2020-11-20T12:17:57.942 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:17:57.943 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":121,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:16:17.809383+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:16:00.509740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2020-11-20T12:16:02.486990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/3143922581":"2020-11-21T12:16:17.798582+0000","172.21.15.187:6834/3143922581":"2020-11-21T12:16:17.798582+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:17:57.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb0b4056d20 msgr2=0x7fb0b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb0b4056d20 0x7fb0b40591e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fb0dc0b50a0 tx=0x7fb0cc015000).stop 2020-11-20T12:17:57.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc065e10 msgr2=0x7fb0dc1344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc065e10 0x7fb0dc1344c0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7fb0d800c730 tx=0x7fb0d8007c30).stop 2020-11-20T12:17:57.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 shutdown_connections 2020-11-20T12:17:57.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb0b4056d20 0x7fb0b40591e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0dc099ce0 0x7fb0dc09cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:57.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0dc065e10 0x7fb0dc1344c0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 --2- 172.21.15.187:0/1923212680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0dc0661f0 0x7fb0dc0b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:57.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.940+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 >> 172.21.15.187:0/1923212680 conn(0x7fb0dc0bacd0 msgr2=0x7fb0dc1443d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:57.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.941+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 shutdown_connections 2020-11-20T12:17:57.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:57.941+0000 7fb0e2faa700 1 -- 172.21.15.187:0/1923212680 wait complete. 2020-11-20T12:17:57.967 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:16:00.509740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:16:02.486990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:17:57.968 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:17:58.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.211+0000 7fd71bfff700 1 Processor -- start 2020-11-20T12:17:58.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.211+0000 7fd71bfff700 1 -- start start 2020-11-20T12:17:58.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.212+0000 7fd71bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd71c0676a0 0x7fd71c0b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.212+0000 7fd71bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c09b540 con 0x7fd71c060710 2020-11-20T12:17:58.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.212+0000 7fd71bfff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c0b7da0 con 0x7fd71c060330 2020-11-20T12:17:58.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.212+0000 7fd71bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd71c0b8040 con 0x7fd71c0676a0 2020-11-20T12:17:58.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.213+0000 7fd71affd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd71c060330 0x7fd71c0662a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48034/0 (socket says 172.21.15.187:48034) 2020-11-20T12:17:58.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.213+0000 7fd71affd700 1 -- 172.21.15.187:0/4251022787 learned_addr learned my addr 172.21.15.187:0/4251022787 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:58.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.213+0000 7fd71b7fe700 1 --2- 172.21.15.187:0/4251022787 >> v2:172.21.15.187:3300/0 conn(0x7fd71c0676a0 0x7fd71c0b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.213+0000 7fd71b7fe700 1 -- 172.21.15.187:0/4251022787 >> v1:172.21.15.187:6789/0 conn(0x7fd71c060330 legacy=0x7fd71c0662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.213+0000 7fd71b7fe700 1 -- 172.21.15.187:0/4251022787 >> v1:172.21.15.5:6789/0 conn(0x7fd71c060710 legacy=0x7fd71c1480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:58.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.213+0000 7fd71b7fe700 1 -- 172.21.15.187:0/4251022787 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c0b82e0 con 0x7fd71c0676a0 2020-11-20T12:17:58.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.214+0000 7fd71b7fe700 1 --2- 172.21.15.187:0/4251022787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c0676a0 0x7fd71c0b6b00 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7fd714009c90 tx=0x7fd7140091d0).ready entity=mon.1 client_cookie=b2e846191305d2ee server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:58.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.214+0000 7fd719ffb700 1 -- 172.21.15.187:0/4251022787 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71400bc10 con 0x7fd71c0676a0 2020-11-20T12:17:58.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.214+0000 7fd719ffb700 1 -- 172.21.15.187:0/4251022787 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71400bd70 con 0x7fd71c0676a0 2020-11-20T12:17:58.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.214+0000 7fd719ffb700 1 -- 172.21.15.187:0/4251022787 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714011610 con 0x7fd71c0676a0 2020-11-20T12:17:58.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.214+0000 7fd71bfff700 1 -- 172.21.15.187:0/4251022787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c0676a0 msgr2=0x7fd71c0b6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.214+0000 7fd71bfff700 1 --2- 172.21.15.187:0/4251022787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c0676a0 0x7fd71c0b6b00 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7fd714009c90 tx=0x7fd7140091d0).stop 2020-11-20T12:17:58.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.214+0000 7fd71bfff700 1 -- 172.21.15.187:0/4251022787 shutdown_connections 2020-11-20T12:17:58.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.214+0000 7fd71bfff700 1 --2- 172.21.15.187:0/4251022787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c0676a0 0x7fd71c0b6b00 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.215+0000 7fd71bfff700 1 -- 172.21.15.187:0/4251022787 >> 172.21.15.187:0/4251022787 conn(0x7fd71c0bbc20 msgr2=0x7fd71c0bc030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:58.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.215+0000 7fd71bfff700 1 -- 172.21.15.187:0/4251022787 shutdown_connections 2020-11-20T12:17:58.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.215+0000 7fd71bfff700 1 -- 172.21.15.187:0/4251022787 wait complete. 2020-11-20T12:17:58.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.216+0000 7fd71bfff700 1 Processor -- start 2020-11-20T12:17:58.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.216+0000 7fd71bfff700 1 -- start start 2020-11-20T12:17:58.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.217+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c060330 0x7fd71c143ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.217+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd71c060710 0x7fd71c1443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.217+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd71c0676a0 0x7fd71c15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.217+0000 7fd71bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c1448d0 con 0x7fd71c0676a0 2020-11-20T12:17:58.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.217+0000 7fd71bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c0a1470 con 0x7fd71c060330 2020-11-20T12:17:58.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.217+0000 7fd71bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c0a11d0 con 0x7fd71c060710 2020-11-20T12:17:58.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.217+0000 7fd71affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c060330 0x7fd71c143ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c060330 0x7fd71c143ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51434/0 (socket says 172.21.15.187:51434) 2020-11-20T12:17:58.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71affd700 1 -- 172.21.15.187:0/890479241 learned_addr learned my addr 172.21.15.187:0/890479241 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:58.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71b7fe700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd71c0676a0 0x7fd71c15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71a7fc700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd71c060710 0x7fd71c1443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71affd700 1 -- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd71c060710 msgr2=0x7fd71c1443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71affd700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd71c060710 0x7fd71c1443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71affd700 1 -- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd71c0676a0 msgr2=0x7fd71c15b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71affd700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd71c0676a0 0x7fd71c15b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71affd700 1 -- 172.21.15.187:0/890479241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c14c350 con 0x7fd71c060330 2020-11-20T12:17:58.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71a7fc700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd71c060710 0x7fd71c1443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:58.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.218+0000 7fd71affd700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c060330 0x7fd71c143ef0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7fd70c00c740 tx=0x7fd70c007c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:58.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.219+0000 7fd702ffd700 1 -- 172.21.15.187:0/890479241 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c007320 con 0x7fd71c060330 2020-11-20T12:17:58.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.219+0000 7fd702ffd700 1 -- 172.21.15.187:0/890479241 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c007480 con 0x7fd71c060330 2020-11-20T12:17:58.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.219+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71c14c7f0 con 0x7fd71c060330 2020-11-20T12:17:58.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.219+0000 7fd702ffd700 1 -- 172.21.15.187:0/890479241 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c005800 con 0x7fd71c060330 2020-11-20T12:17:58.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.219+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71c0a0da0 con 0x7fd71c060330 2020-11-20T12:17:58.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.220+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71c14c4e0 con 0x7fd71c060330 2020-11-20T12:17:58.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.221+0000 7fd702ffd700 1 -- 172.21.15.187:0/890479241 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd70c010030 con 0x7fd71c060330 2020-11-20T12:17:58.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.221+0000 7fd702ffd700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd6f8056d20 0x7fd6f80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.221+0000 7fd702ffd700 1 -- 172.21.15.187:0/890479241 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd70c06fc20 con 0x7fd71c060330 2020-11-20T12:17:58.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.222+0000 7fd71a7fc700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd6f8056d20 0x7fd6f80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.222+0000 7fd71a7fc700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd6f8056d20 0x7fd6f80591e0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fd710002510 tx=0x7fd710009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:58.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.225+0000 7fd702ffd700 1 -- 172.21.15.187:0/890479241 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd70c044fc0 con 0x7fd71c060330 2020-11-20T12:17:58.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.415+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd71c09e6e0 con 0x7fd71c060330 2020-11-20T12:17:58.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.415+0000 7fd702ffd700 1 -- 172.21.15.187:0/890479241 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v121) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd70c0485e0 con 0x7fd71c060330 2020-11-20T12:17:58.420 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:17:58.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.417+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd6f8056d20 msgr2=0x7fd6f80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd6f8056d20 0x7fd6f80591e0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fd710002510 tx=0x7fd710009040).stop 2020-11-20T12:17:58.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c060330 msgr2=0x7fd71c143ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c060330 0x7fd71c143ef0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7fd70c00c740 tx=0x7fd70c007c40).stop 2020-11-20T12:17:58.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 shutdown_connections 2020-11-20T12:17:58.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd6f8056d20 0x7fd6f80591e0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd71c0676a0 0x7fd71c15b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd71c060330 0x7fd71c143ef0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 --2- 172.21.15.187:0/890479241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd71c060710 0x7fd71c1443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 >> 172.21.15.187:0/890479241 conn(0x7fd71c0bbc20 msgr2=0x7fd71c067c20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:58.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 shutdown_connections 2020-11-20T12:17:58.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.418+0000 7fd71bfff700 1 -- 172.21.15.187:0/890479241 wait complete. 2020-11-20T12:17:58.446 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:17:58.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.694+0000 7f0857449700 1 Processor -- start 2020-11-20T12:17:58.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.694+0000 7f0857449700 1 -- start start 2020-11-20T12:17:58.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.696+0000 7f0857449700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f08501484d0 0x7f0850061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.696+0000 7f0857449700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f08500622c0 0x7f08500b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.696+0000 7f0857449700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500b1300 con 0x7f0850066f70 2020-11-20T12:17:58.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.696+0000 7f0857449700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0850067730 con 0x7f08501484d0 2020-11-20T12:17:58.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.696+0000 7f0857449700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08500b15a0 con 0x7f08500622c0 2020-11-20T12:17:58.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.696+0000 7f0855c46700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f08500622c0 0x7f08500b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.696+0000 7f0856c48700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0850066f70 0x7f08500b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48052/0 (socket says 172.21.15.187:48052) 2020-11-20T12:17:58.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.696+0000 7f0856c48700 1 -- 172.21.15.187:0/523459627 learned_addr learned my addr 172.21.15.187:0/523459627 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:58.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.697+0000 7f0856447700 1 --2- 172.21.15.187:0/523459627 >> v2:172.21.15.5:3300/0 conn(0x7f08501484d0 0x7f0850061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.697+0000 7f0855445700 1 -- 172.21.15.187:0/523459627 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242303220 0 0) 0x7f08500b1300 con 0x7f0850066f70 2020-11-20T12:17:58.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.697+0000 7f0855445700 1 -- 172.21.15.187:0/523459627 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0838003240 con 0x7f0850066f70 2020-11-20T12:17:58.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.697+0000 7f0855c46700 1 -- 172.21.15.187:0/523459627 >> v2:172.21.15.5:3300/0 conn(0x7f08501484d0 msgr2=0x7f0850061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:58.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.697+0000 7f0855c46700 1 --2- 172.21.15.187:0/523459627 >> v2:172.21.15.5:3300/0 conn(0x7f08501484d0 0x7f0850061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.697+0000 7f0855c46700 1 -- 172.21.15.187:0/523459627 >> v1:172.21.15.187:6789/0 conn(0x7f0850066f70 legacy=0x7f08500b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.697+0000 7f0855c46700 1 -- 172.21.15.187:0/523459627 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500b1840 con 0x7f08500622c0 2020-11-20T12:17:58.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.697+0000 7f0855c46700 1 --2- 172.21.15.187:0/523459627 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f08500b5640 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 rx=0x7f0850068600 tx=0x7f0840006df0).ready entity=mon.1 client_cookie=86e780b1f5c89f60 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:58.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.698+0000 7f0855445700 1 -- 172.21.15.187:0/523459627 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08400097b0 con 0x7f08500622c0 2020-11-20T12:17:58.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.698+0000 7f0855445700 1 -- 172.21.15.187:0/523459627 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0840009910 con 0x7f08500622c0 2020-11-20T12:17:58.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.698+0000 7f0855445700 1 -- 172.21.15.187:0/523459627 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840009c90 con 0x7f08500622c0 2020-11-20T12:17:58.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.699+0000 7f0857449700 1 -- 172.21.15.187:0/523459627 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 msgr2=0x7f08500b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.699+0000 7f0857449700 1 --2- 172.21.15.187:0/523459627 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f08500b5640 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 rx=0x7f0850068600 tx=0x7f0840006df0).stop 2020-11-20T12:17:58.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.699+0000 7f0857449700 1 -- 172.21.15.187:0/523459627 shutdown_connections 2020-11-20T12:17:58.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.699+0000 7f0857449700 1 --2- 172.21.15.187:0/523459627 >> v2:172.21.15.5:3300/0 conn(0x7f08501484d0 0x7f0850061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.699+0000 7f0857449700 1 --2- 172.21.15.187:0/523459627 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f08500b5640 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.699+0000 7f0857449700 1 -- 172.21.15.187:0/523459627 >> 172.21.15.187:0/523459627 conn(0x7f08500bab30 msgr2=0x7f08500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:58.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.699+0000 7f0857449700 1 -- 172.21.15.187:0/523459627 shutdown_connections 2020-11-20T12:17:58.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.699+0000 7f0857449700 1 -- 172.21.15.187:0/523459627 wait complete. 2020-11-20T12:17:58.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.700+0000 7f0857449700 1 Processor -- start 2020-11-20T12:17:58.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.700+0000 7f0857449700 1 -- start start 2020-11-20T12:17:58.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.701+0000 7f0857449700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f085009dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.701+0000 7f0857449700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0850066f70 0x7f0850156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.701+0000 7f0857449700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08501484d0 0x7f08500b7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.701+0000 7f0857449700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085009b270 con 0x7f08501484d0 2020-11-20T12:17:58.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.701+0000 7f0857449700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085009b510 con 0x7f08500622c0 2020-11-20T12:17:58.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.701+0000 7f0857449700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085009b7b0 con 0x7f0850066f70 2020-11-20T12:17:58.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f085009dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f085009dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51448/0 (socket says 172.21.15.187:51448) 2020-11-20T12:17:58.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 -- 172.21.15.187:0/1297544561 learned_addr learned my addr 172.21.15.187:0/1297544561 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:58.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 -- 172.21.15.187:0/1297544561 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0850066f70 msgr2=0x7f0850156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0850066f70 0x7f0850156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:58.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 -- 172.21.15.187:0/1297544561 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08501484d0 msgr2=0x7f08500b7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08501484d0 0x7f08500b7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:58.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 -- 172.21.15.187:0/1297544561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0850156a60 con 0x7f08500622c0 2020-11-20T12:17:58.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.702+0000 7f0856447700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f085009dae0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 rx=0x7f084c00d8e0 tx=0x7f084c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:58.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.703+0000 7f083e7fc700 1 -- 172.21.15.187:0/1297544561 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c009790 con 0x7f08500622c0 2020-11-20T12:17:58.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.703+0000 7f083e7fc700 1 -- 172.21.15.187:0/1297544561 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084c0098f0 con 0x7f08500622c0 2020-11-20T12:17:58.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.703+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085009aef0 con 0x7f08500622c0 2020-11-20T12:17:58.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.703+0000 7f083e7fc700 1 -- 172.21.15.187:0/1297544561 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c009c70 con 0x7f08500622c0 2020-11-20T12:17:58.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.703+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08500b7a10 con 0x7f08500622c0 2020-11-20T12:17:58.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.704+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085009b0e0 con 0x7f08500622c0 2020-11-20T12:17:58.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.704+0000 7f083e7fc700 1 -- 172.21.15.187:0/1297544561 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f084c013480 con 0x7f08500622c0 2020-11-20T12:17:58.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.705+0000 7f083e7fc700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0828056d50 0x7f0828059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:58.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.705+0000 7f083e7fc700 1 -- 172.21.15.187:0/1297544561 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f084c06fff0 con 0x7f08500622c0 2020-11-20T12:17:58.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.705+0000 7f0855c46700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0828056d50 0x7f0828059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:58.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.706+0000 7f0855c46700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0828056d50 0x7f0828059210 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f0850068600 tx=0x7f0840015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:58.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.709+0000 7f083e7fc700 1 -- 172.21.15.187:0/1297544561 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f084c0452c0 con 0x7f08500622c0 2020-11-20T12:17:58.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.894+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f085009b970 con 0x7f08500622c0 2020-11-20T12:17:58.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.896+0000 7f083e7fc700 1 -- 172.21.15.187:0/1297544561 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v121) v1 ==== 92+0+10 (secure 0 0 0) 0x7f084c0488e0 con 0x7f08500622c0 2020-11-20T12:17:58.899 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:17:58.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0828056d50 msgr2=0x7f0828059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0828056d50 0x7f0828059210 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f0850068600 tx=0x7f0840015000).stop 2020-11-20T12:17:58.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 msgr2=0x7f085009dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f085009dae0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 rx=0x7f084c00d8e0 tx=0x7f084c008040).stop 2020-11-20T12:17:58.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 shutdown_connections 2020-11-20T12:17:58.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0828056d50 0x7f0828059210 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08501484d0 0x7f08500b7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:58.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08500622c0 0x7f085009dae0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 --2- 172.21.15.187:0/1297544561 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0850066f70 0x7f0850156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:58.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 >> 172.21.15.187:0/1297544561 conn(0x7f08500bab30 msgr2=0x7f08500b60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:58.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.897+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 shutdown_connections 2020-11-20T12:17:58.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:58.898+0000 7f0857449700 1 -- 172.21.15.187:0/1297544561 wait complete. 2020-11-20T12:17:58.922 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:17:59.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.163+0000 7fe8f07ee700 1 Processor -- start 2020-11-20T12:17:59.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.164+0000 7fe8f07ee700 1 -- start start 2020-11-20T12:17:59.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8f07ee700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe8e8065300 0x7fe8e80656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8f07ee700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e8066420 con 0x7fe8e809a8a0 2020-11-20T12:17:59.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8f07ee700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e80b7380 con 0x7fe8e8068030 2020-11-20T12:17:59.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8f07ee700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e8061f60 con 0x7fe8e8065300 2020-11-20T12:17:59.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8eed8b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe8e8068030 0x7fe8e8146950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48066/0 (socket says 172.21.15.187:48066) 2020-11-20T12:17:59.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8eed8b700 1 -- 172.21.15.187:0/3247334394 learned_addr learned my addr 172.21.15.187:0/3247334394 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:59.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8ee58a700 1 --2- 172.21.15.187:0/3247334394 >> v2:172.21.15.5:3300/0 conn(0x7fe8e8065300 0x7fe8e80656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721954771 0 0) 0x7fe8e80b7380 con 0x7fe8e8068030 2020-11-20T12:17:59.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.165+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0003240 con 0x7fe8e8068030 2020-11-20T12:17:59.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3582387990 0 0) 0x7fe8d0003240 con 0x7fe8e8068030 2020-11-20T12:17:59.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 >> v2:172.21.15.5:3300/0 conn(0x7fe8e8065300 msgr2=0x7fe8e80656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:59.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 --2- 172.21.15.187:0/3247334394 >> v2:172.21.15.5:3300/0 conn(0x7fe8e8065300 0x7fe8e80656e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 >> v1:172.21.15.5:6789/0 conn(0x7fe8e809a8a0 legacy=0x7fe8e8062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e80b6fd0 con 0x7fe8e8068030 2020-11-20T12:17:59.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fe8e00014e0 con 0x7fe8e8068030 2020-11-20T12:17:59.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 >> v1:172.21.15.187:6789/0 conn(0x7fe8e8068030 legacy=0x7fe8e8146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8d0003730 0x7fe8d0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d00044e0 0x7fe8d0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d0004e50 0x7fe8d000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8061f60 con 0x7fe8d0003730 2020-11-20T12:17:59.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0004260 con 0x7fe8d0004e50 2020-11-20T12:17:59.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d000bc20 con 0x7fe8d00044e0 2020-11-20T12:17:59.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8eed8b700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d0004e50 0x7fe8d000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ee58a700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8d0003730 0x7fe8d0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8eed8b700 1 -- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d00044e0 msgr2=0x7fe8d0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:59.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8edd89700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d00044e0 0x7fe8d0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8eed8b700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d00044e0 0x7fe8d0004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8eed8b700 1 -- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8d0003730 msgr2=0x7fe8d0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:59.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8eed8b700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8d0003730 0x7fe8d0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8eed8b700 1 -- 172.21.15.187:0/3247334394 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0003240 con 0x7fe8d0004e50 2020-11-20T12:17:59.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8ee58a700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8d0003730 0x7fe8d0003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:59.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8eed8b700 1 -- 172.21.15.187:0/3247334394 reap_dead start 2020-11-20T12:17:59.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.167+0000 7fe8eed8b700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d0004e50 0x7fe8d000b730 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7fe8e000c410 tx=0x7fe8e00077d0).ready entity=mon.1 client_cookie=b436b915188a051d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:59.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e000ce80 con 0x7fe8d0004e50 2020-11-20T12:17:59.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e00072e0 con 0x7fe8d0004e50 2020-11-20T12:17:59.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8ed588700 1 -- 172.21.15.187:0/3247334394 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0005800 con 0x7fe8d0004e50 2020-11-20T12:17:59.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8f07ee700 1 -- 172.21.15.187:0/3247334394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d0004e50 msgr2=0x7fe8d000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8f07ee700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d0004e50 0x7fe8d000b730 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7fe8e000c410 tx=0x7fe8e00077d0).stop 2020-11-20T12:17:59.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8f07ee700 1 -- 172.21.15.187:0/3247334394 shutdown_connections 2020-11-20T12:17:59.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8f07ee700 1 --2- 172.21.15.187:0/3247334394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d0004e50 0x7fe8d000b730 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8f07ee700 1 -- 172.21.15.187:0/3247334394 >> 172.21.15.187:0/3247334394 conn(0x7fe8e80bacf0 msgr2=0x7fe8e80bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:59.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.169+0000 7fe8f07ee700 1 -- 172.21.15.187:0/3247334394 shutdown_connections 2020-11-20T12:17:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.170+0000 7fe8f07ee700 1 -- 172.21.15.187:0/3247334394 wait complete. 2020-11-20T12:17:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.171+0000 7fe8f07ee700 1 Processor -- start 2020-11-20T12:17:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.171+0000 7fe8f07ee700 1 -- start start 2020-11-20T12:17:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.172+0000 7fe8f07ee700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d0004e50 0x7fe8e8155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.172+0000 7fe8f07ee700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e8093440 0x7fe8e808edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.172+0000 7fe8f07ee700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8e8092c10 0x7fe8e813e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.172+0000 7fe8f07ee700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8156320 con 0x7fe8e8092c10 2020-11-20T12:17:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.172+0000 7fe8f07ee700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e808f2a0 con 0x7fe8e8093440 2020-11-20T12:17:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.172+0000 7fe8f07ee700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e808f530 con 0x7fe8d0004e50 2020-11-20T12:17:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.172+0000 7fe8edd89700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e8093440 0x7fe8e808edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.172+0000 7fe8edd89700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e8093440 0x7fe8e808edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51468/0 (socket says 172.21.15.187:51468) 2020-11-20T12:17:59.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8edd89700 1 -- 172.21.15.187:0/2213973372 learned_addr learned my addr 172.21.15.187:0/2213973372 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:59.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8eed8b700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8e8092c10 0x7fe8e813e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8ee58a700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d0004e50 0x7fe8e8155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8edd89700 1 -- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d0004e50 msgr2=0x7fe8e8155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8edd89700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d0004e50 0x7fe8e8155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8edd89700 1 -- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8e8092c10 msgr2=0x7fe8e813e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8edd89700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8e8092c10 0x7fe8e813e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8edd89700 1 -- 172.21.15.187:0/2213973372 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e808f7d0 con 0x7fe8e8093440 2020-11-20T12:17:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8eed8b700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8e8092c10 0x7fe8e813e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8edd89700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e8093440 0x7fe8e808edb0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 rx=0x7fe8d800d780 tx=0x7fe8d8009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8d67fc700 1 -- 172.21.15.187:0/2213973372 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d800f5d0 con 0x7fe8e8093440 2020-11-20T12:17:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.173+0000 7fe8d67fc700 1 -- 172.21.15.187:0/2213973372 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d800f730 con 0x7fe8e8093440 2020-11-20T12:17:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.174+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e808fa70 con 0x7fe8e8093440 2020-11-20T12:17:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.174+0000 7fe8d67fc700 1 -- 172.21.15.187:0/2213973372 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d800fab0 con 0x7fe8e8093440 2020-11-20T12:17:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.174+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e813e760 con 0x7fe8e8093440 2020-11-20T12:17:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.175+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e813ea40 con 0x7fe8e8093440 2020-11-20T12:17:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.175+0000 7fe8d67fc700 1 -- 172.21.15.187:0/2213973372 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe8d8012410 con 0x7fe8e8093440 2020-11-20T12:17:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.176+0000 7fe8d67fc700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe8c0056d50 0x7fe8c0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.176+0000 7fe8d67fc700 1 -- 172.21.15.187:0/2213973372 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe8d806fea0 con 0x7fe8e8093440 2020-11-20T12:17:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.176+0000 7fe8ee58a700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe8c0056d50 0x7fe8c0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.177+0000 7fe8ee58a700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe8c0056d50 0x7fe8c0059210 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7fe8e40091a0 tx=0x7fe8e4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:59.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.180+0000 7fe8d67fc700 1 -- 172.21.15.187:0/2213973372 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe8d8045240 con 0x7fe8e8093440 2020-11-20T12:17:59.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.368+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe8e808fc00 con 0x7fe8e8093440 2020-11-20T12:17:59.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.370+0000 7fe8d67fc700 1 -- 172.21.15.187:0/2213973372 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe8d8048860 con 0x7fe8e8093440 2020-11-20T12:17:59.372 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:17:59.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.372+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe8c0056d50 msgr2=0x7fe8c0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.372+0000 7fe8f07ee700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe8c0056d50 0x7fe8c0059210 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7fe8e40091a0 tx=0x7fe8e4008040).stop 2020-11-20T12:17:59.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.372+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e8093440 msgr2=0x7fe8e808edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.372+0000 7fe8f07ee700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e8093440 0x7fe8e808edb0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 rx=0x7fe8d800d780 tx=0x7fe8d8009040).stop 2020-11-20T12:17:59.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.373+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 shutdown_connections 2020-11-20T12:17:59.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.373+0000 7fe8f07ee700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe8c0056d50 0x7fe8c0059210 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.373+0000 7fe8f07ee700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8e8092c10 0x7fe8e813e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.373+0000 7fe8f07ee700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe8d0004e50 0x7fe8e8155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.373+0000 7fe8f07ee700 1 --2- 172.21.15.187:0/2213973372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e8093440 0x7fe8e808edb0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.373+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 >> 172.21.15.187:0/2213973372 conn(0x7fe8e80bacf0 msgr2=0x7fe8e809a110 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:59.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.373+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 shutdown_connections 2020-11-20T12:17:59.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.373+0000 7fe8f07ee700 1 -- 172.21.15.187:0/2213973372 wait complete. 2020-11-20T12:17:59.399 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:17:59.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.646+0000 7f00352b0700 1 Processor -- start 2020-11-20T12:17:59.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.646+0000 7f00352b0700 1 -- start start 2020-11-20T12:17:59.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.648+0000 7f00352b0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0030065190 0x7f0030065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.648+0000 7f00352b0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f003009ca90 0x7f003009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.648+0000 7f00352b0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00300b13e0 con 0x7f0030065f90 2020-11-20T12:17:59.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.648+0000 7f00352b0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003014ab30 con 0x7f0030065190 2020-11-20T12:17:59.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.648+0000 7f00352b0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0030062c10 con 0x7f003009ca90 2020-11-20T12:17:59.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.648+0000 7f00277fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0030065f90 0x7f0030143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48084/0 (socket says 172.21.15.187:48084) 2020-11-20T12:17:59.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.648+0000 7f00277fe700 1 -- 172.21.15.187:0/410863061 learned_addr learned my addr 172.21.15.187:0/410863061 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:17:59.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.648+0000 7f002ffff700 1 --2- 172.21.15.187:0/410863061 >> v2:172.21.15.187:3300/0 conn(0x7f0030065190 0x7f0030065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.649+0000 7f0034aaf700 1 --2- 172.21.15.187:0/410863061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f003009ca90 0x7f003009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.649+0000 7f002f7fe700 1 -- 172.21.15.187:0/410863061 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422518072 0 0) 0x7f00300b13e0 con 0x7f0030065f90 2020-11-20T12:17:59.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.649+0000 7f002f7fe700 1 -- 172.21.15.187:0/410863061 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0010003240 con 0x7f0030065f90 2020-11-20T12:17:59.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.650+0000 7f002ffff700 1 -- 172.21.15.187:0/410863061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f003009ca90 msgr2=0x7f003009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:59.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.650+0000 7f002ffff700 1 --2- 172.21.15.187:0/410863061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f003009ca90 0x7f003009cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.650+0000 7f002ffff700 1 -- 172.21.15.187:0/410863061 >> v1:172.21.15.187:6789/0 conn(0x7f0030065f90 legacy=0x7f0030143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.650+0000 7f002ffff700 1 -- 172.21.15.187:0/410863061 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00300b1640 con 0x7f0030065190 2020-11-20T12:17:59.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.650+0000 7f002ffff700 1 --2- 172.21.15.187:0/410863061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f0030065570 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 rx=0x7f002000b9a0 tx=0x7f0020006df0).ready entity=mon.1 client_cookie=3efb86206a582acc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:59.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.650+0000 7f002f7fe700 1 -- 172.21.15.187:0/410863061 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020009710 con 0x7f0030065190 2020-11-20T12:17:59.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.651+0000 7f002f7fe700 1 -- 172.21.15.187:0/410863061 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0020009870 con 0x7f0030065190 2020-11-20T12:17:59.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.651+0000 7f002f7fe700 1 -- 172.21.15.187:0/410863061 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020009bf0 con 0x7f0030065190 2020-11-20T12:17:59.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.651+0000 7f00352b0700 1 -- 172.21.15.187:0/410863061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 msgr2=0x7f0030065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.651+0000 7f00352b0700 1 --2- 172.21.15.187:0/410863061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f0030065570 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 rx=0x7f002000b9a0 tx=0x7f0020006df0).stop 2020-11-20T12:17:59.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.651+0000 7f00352b0700 1 -- 172.21.15.187:0/410863061 shutdown_connections 2020-11-20T12:17:59.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.651+0000 7f00352b0700 1 --2- 172.21.15.187:0/410863061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f003009ca90 0x7f003009cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.651+0000 7f00352b0700 1 --2- 172.21.15.187:0/410863061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f0030065570 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.651+0000 7f00352b0700 1 -- 172.21.15.187:0/410863061 >> 172.21.15.187:0/410863061 conn(0x7f00300bab30 msgr2=0x7f00300baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:59.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.652+0000 7f00352b0700 1 -- 172.21.15.187:0/410863061 shutdown_connections 2020-11-20T12:17:59.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.652+0000 7f00352b0700 1 -- 172.21.15.187:0/410863061 wait complete. 2020-11-20T12:17:59.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.653+0000 7f00352b0700 1 Processor -- start 2020-11-20T12:17:59.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.653+0000 7f00352b0700 1 -- start start 2020-11-20T12:17:59.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f00352b0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f00300b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f00352b0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0030065f90 0x7f00301436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f00352b0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f003009ca90 0x7f0030145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f00352b0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0030149140 con 0x7f003009ca90 2020-11-20T12:17:59.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f00352b0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00301493e0 con 0x7f0030065190 2020-11-20T12:17:59.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f00352b0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0030149680 con 0x7f0030065f90 2020-11-20T12:17:59.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f002ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f00300b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f002ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f00300b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51482/0 (socket says 172.21.15.187:51482) 2020-11-20T12:17:59.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.654+0000 7f002ffff700 1 -- 172.21.15.187:0/2932122327 learned_addr learned my addr 172.21.15.187:0/2932122327 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:17:59.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002ffff700 1 -- 172.21.15.187:0/2932122327 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0030065f90 msgr2=0x7f00301436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002ffff700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0030065f90 0x7f00301436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:59.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002ffff700 1 -- 172.21.15.187:0/2932122327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f003009ca90 msgr2=0x7f0030145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002ffff700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f003009ca90 0x7f0030145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:59.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002ffff700 1 -- 172.21.15.187:0/2932122327 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0030143be0 con 0x7f0030065190 2020-11-20T12:17:59.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002ffff700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f00300b46d0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f002000b150 tx=0x7f0020015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:59.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002cff9700 1 -- 172.21.15.187:0/2932122327 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020011ea0 con 0x7f0030065190 2020-11-20T12:17:59.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002cff9700 1 -- 172.21.15.187:0/2932122327 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002001d450 con 0x7f0030065190 2020-11-20T12:17:59.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f002cff9700 1 -- 172.21.15.187:0/2932122327 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002001d7f0 con 0x7f0030065190 2020-11-20T12:17:59.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.655+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00300b2750 con 0x7f0030065190 2020-11-20T12:17:59.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.656+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00301454f0 con 0x7f0030065190 2020-11-20T12:17:59.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.657+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0030149840 con 0x7f0030065190 2020-11-20T12:17:59.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.657+0000 7f002cff9700 1 -- 172.21.15.187:0/2932122327 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f002000f070 con 0x7f0030065190 2020-11-20T12:17:59.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.658+0000 7f002cff9700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f000c056d20 0x7f000c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:59.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.658+0000 7f002cff9700 1 -- 172.21.15.187:0/2932122327 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0020071420 con 0x7f0030065190 2020-11-20T12:17:59.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.658+0000 7f00277fe700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f000c056d20 0x7f000c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:59.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.659+0000 7f00277fe700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f000c056d20 0x7f000c0591e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f00180055a0 tx=0x7f00180076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:59.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.663+0000 7f002cff9700 1 -- 172.21.15.187:0/2932122327 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f00200467c0 con 0x7f0030065190 2020-11-20T12:17:59.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.847+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0030148f40 con 0x7f0030065190 2020-11-20T12:17:59.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.848+0000 7f002cff9700 1 -- 172.21.15.187:0/2932122327 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0020022020 con 0x7f0030065190 2020-11-20T12:17:59.851 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:17:59.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f000c056d20 msgr2=0x7f000c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f000c056d20 0x7f000c0591e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f00180055a0 tx=0x7f00180076a0).stop 2020-11-20T12:17:59.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 msgr2=0x7f00300b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:59.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f00300b46d0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f002000b150 tx=0x7f0020015000).stop 2020-11-20T12:17:59.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 shutdown_connections 2020-11-20T12:17:59.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f000c056d20 0x7f000c0591e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f003009ca90 0x7f0030145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:59.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0030065190 0x7f00300b46d0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:59.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 --2- 172.21.15.187:0/2932122327 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0030065f90 0x7f00301436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:59.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 >> 172.21.15.187:0/2932122327 conn(0x7f00300bab30 msgr2=0x7f00300aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:59.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 shutdown_connections 2020-11-20T12:17:59.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:17:59.851+0000 7f00352b0700 1 -- 172.21.15.187:0/2932122327 wait complete. 2020-11-20T12:17:59.879 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:00.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.118+0000 7f9c92c17700 1 Processor -- start 2020-11-20T12:18:00.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.119+0000 7f9c92c17700 1 -- start start 2020-11-20T12:18:00.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.120+0000 7f9c92c17700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9c8c065570 0x7f9c8c09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.120+0000 7f9c92c17700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8c0b13e0 con 0x7f9c8c065f90 2020-11-20T12:18:00.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.120+0000 7f9c92c17700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8c09d690 con 0x7f9c8c065190 2020-11-20T12:18:00.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.120+0000 7f9c92c17700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c8c14ab30 con 0x7f9c8c065570 2020-11-20T12:18:00.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.120+0000 7f9c91c15700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9c8c065190 0x7f9c8c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48096/0 (socket says 172.21.15.187:48096) 2020-11-20T12:18:00.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.120+0000 7f9c91c15700 1 -- 172.21.15.187:0/3943097440 learned_addr learned my addr 172.21.15.187:0/3943097440 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:00.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.121+0000 7f9c92416700 1 --2- 172.21.15.187:0/3943097440 >> v2:172.21.15.5:3300/0 conn(0x7f9c8c065570 0x7f9c8c09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.121+0000 7f9c90c13700 1 -- 172.21.15.187:0/3943097440 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786895533 0 0) 0x7f9c8c09d690 con 0x7f9c8c065190 2020-11-20T12:18:00.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.121+0000 7f9c90c13700 1 -- 172.21.15.187:0/3943097440 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c74003240 con 0x7f9c8c065190 2020-11-20T12:18:00.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.122+0000 7f9c92416700 1 -- 172.21.15.187:0/3943097440 >> v1:172.21.15.187:6789/0 conn(0x7f9c8c065190 legacy=0x7f9c8c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.122+0000 7f9c92416700 1 -- 172.21.15.187:0/3943097440 >> v1:172.21.15.5:6789/0 conn(0x7f9c8c065f90 legacy=0x7f9c8c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.122+0000 7f9c92416700 1 -- 172.21.15.187:0/3943097440 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c062c10 con 0x7f9c8c065570 2020-11-20T12:18:00.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.122+0000 7f9c92416700 1 --2- 172.21.15.187:0/3943097440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065570 0x7f9c8c09d1a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f9c8c09d820 tx=0x7f9c840091d0).ready entity=mon.0 client_cookie=3922b84b4858a63b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:00.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.122+0000 7f9c90c13700 1 -- 172.21.15.187:0/3943097440 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8400bad0 con 0x7f9c8c065570 2020-11-20T12:18:00.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.122+0000 7f9c90c13700 1 -- 172.21.15.187:0/3943097440 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8400bc30 con 0x7f9c8c065570 2020-11-20T12:18:00.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.123+0000 7f9c90c13700 1 -- 172.21.15.187:0/3943097440 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84011400 con 0x7f9c8c065570 2020-11-20T12:18:00.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.123+0000 7f9c92c17700 1 -- 172.21.15.187:0/3943097440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065570 msgr2=0x7f9c8c09d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.123+0000 7f9c92c17700 1 --2- 172.21.15.187:0/3943097440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065570 0x7f9c8c09d1a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f9c8c09d820 tx=0x7f9c840091d0).stop 2020-11-20T12:18:00.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.124+0000 7f9c92c17700 1 -- 172.21.15.187:0/3943097440 shutdown_connections 2020-11-20T12:18:00.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.124+0000 7f9c92c17700 1 --2- 172.21.15.187:0/3943097440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065570 0x7f9c8c09d1a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.124+0000 7f9c92c17700 1 -- 172.21.15.187:0/3943097440 >> 172.21.15.187:0/3943097440 conn(0x7f9c8c0bab30 msgr2=0x7f9c8c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:00.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.124+0000 7f9c92c17700 1 -- 172.21.15.187:0/3943097440 shutdown_connections 2020-11-20T12:18:00.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.124+0000 7f9c92c17700 1 -- 172.21.15.187:0/3943097440 wait complete. 2020-11-20T12:18:00.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.126+0000 7f9c92c17700 1 Processor -- start 2020-11-20T12:18:00.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.126+0000 7f9c92c17700 1 -- start start 2020-11-20T12:18:00.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.127+0000 7f9c92c17700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065190 0x7f9c8c152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.127+0000 7f9c92c17700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c8c065570 0x7f9c8c0b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.127+0000 7f9c92c17700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c8c065f90 0x7f9c8c15b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.127+0000 7f9c92c17700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c148580 con 0x7f9c8c065190 2020-11-20T12:18:00.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.127+0000 7f9c92c17700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c148820 con 0x7f9c8c065f90 2020-11-20T12:18:00.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.127+0000 7f9c92c17700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8c148ac0 con 0x7f9c8c065570 2020-11-20T12:18:00.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c92416700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c8c065f90 0x7f9c8c15b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c92416700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c8c065f90 0x7f9c8c15b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51500/0 (socket says 172.21.15.187:51500) 2020-11-20T12:18:00.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c92416700 1 -- 172.21.15.187:0/1090761809 learned_addr learned my addr 172.21.15.187:0/1090761809 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:00.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c91c15700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065190 0x7f9c8c152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c91414700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c8c065570 0x7f9c8c0b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c92416700 1 -- 172.21.15.187:0/1090761809 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c8c065570 msgr2=0x7f9c8c0b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c92416700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c8c065570 0x7f9c8c0b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c92416700 1 -- 172.21.15.187:0/1090761809 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065190 msgr2=0x7f9c8c152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c92416700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065190 0x7f9c8c152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.128+0000 7f9c92416700 1 -- 172.21.15.187:0/1090761809 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c152610 con 0x7f9c8c065f90 2020-11-20T12:18:00.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.129+0000 7f9c92416700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c8c065f90 0x7f9c8c15b760 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7f9c84009210 tx=0x7f9c84016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:00.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.129+0000 7f9c79ffb700 1 -- 172.21.15.187:0/1090761809 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84010800 con 0x7f9c8c065f90 2020-11-20T12:18:00.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.129+0000 7f9c79ffb700 1 -- 172.21.15.187:0/1090761809 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c84010960 con 0x7f9c8c065f90 2020-11-20T12:18:00.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.129+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8c15c290 con 0x7f9c8c065f90 2020-11-20T12:18:00.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.129+0000 7f9c79ffb700 1 -- 172.21.15.187:0/1090761809 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84010ce0 con 0x7f9c8c065f90 2020-11-20T12:18:00.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.129+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8c15bc50 con 0x7f9c8c065f90 2020-11-20T12:18:00.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.130+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8c15bf00 con 0x7f9c8c065f90 2020-11-20T12:18:00.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.131+0000 7f9c79ffb700 1 -- 172.21.15.187:0/1090761809 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9c84024030 con 0x7f9c8c065f90 2020-11-20T12:18:00.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.132+0000 7f9c79ffb700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c64056d20 0x7f9c640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.132+0000 7f9c79ffb700 1 -- 172.21.15.187:0/1090761809 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9c84071d70 con 0x7f9c8c065f90 2020-11-20T12:18:00.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.132+0000 7f9c91c15700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c64056d20 0x7f9c640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.133+0000 7f9c91c15700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c64056d20 0x7f9c640591e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f9c880055a0 tx=0x7f9c88009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:00.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.136+0000 7f9c79ffb700 1 -- 172.21.15.187:0/1090761809 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9c840471a0 con 0x7f9c8c065f90 2020-11-20T12:18:00.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.380+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c8c148c80 con 0x7f9c8c065f90 2020-11-20T12:18:00.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.381+0000 7f9c79ffb700 1 -- 172.21.15.187:0/1090761809 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f9c84022030 con 0x7f9c8c065f90 2020-11-20T12:18:00.384 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:00.384 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":108,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2047602760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2047602760},{"type":"v1","addr":"172.21.15.5:6833","nonce":2047602760}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":104}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:16:03.867608+0000","modified":"2020-11-20T12:16:26.456436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8647":{"gid":8647,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3515729375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3515729375},{"type":"v1","addr":"172.21.15.187:6835","nonce":3515729375}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2020-11-20T12:18:00.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.383+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c64056d20 msgr2=0x7f9c640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.383+0000 7f9c92c17700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c64056d20 0x7f9c640591e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f9c880055a0 tx=0x7f9c88009040).stop 2020-11-20T12:18:00.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.383+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c8c065f90 msgr2=0x7f9c8c15b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.383+0000 7f9c92c17700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c8c065f90 0x7f9c8c15b760 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7f9c84009210 tx=0x7f9c84016000).stop 2020-11-20T12:18:00.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.384+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 shutdown_connections 2020-11-20T12:18:00.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.384+0000 7f9c92c17700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9c64056d20 0x7f9c640591e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.384+0000 7f9c92c17700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c8c065190 0x7f9c8c152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.384+0000 7f9c92c17700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c8c065f90 0x7f9c8c15b760 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.384+0000 7f9c92c17700 1 --2- 172.21.15.187:0/1090761809 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c8c065570 0x7f9c8c0b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.384+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 >> 172.21.15.187:0/1090761809 conn(0x7f9c8c0bab30 msgr2=0x7f9c8c09ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:00.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.384+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 shutdown_connections 2020-11-20T12:18:00.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.384+0000 7f9c92c17700 1 -- 172.21.15.187:0/1090761809 wait complete. 2020-11-20T12:18:00.389 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 108 2020-11-20T12:18:00.408 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:18:00.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.647+0000 7f2e50a57700 1 Processor -- start 2020-11-20T12:18:00.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.647+0000 7f2e50a57700 1 -- start start 2020-11-20T12:18:00.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.648+0000 7f2e50a57700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2e4c145cc0 0x7f2e4c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.648+0000 7f2e50a57700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2e4c0683f0 0x7f2e4c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.648+0000 7f2e50a57700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c14c170 con 0x7f2e4c065680 2020-11-20T12:18:00.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.648+0000 7f2e50a57700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4c1497f0 con 0x7f2e4c0683f0 2020-11-20T12:18:00.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.648+0000 7f2e50a57700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4c14c410 con 0x7f2e4c145cc0 2020-11-20T12:18:00.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.649+0000 7f2e4bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2e4c065680 0x7f2e4c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48114/0 (socket says 172.21.15.187:48114) 2020-11-20T12:18:00.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.649+0000 7f2e4bfff700 1 -- 172.21.15.187:0/2581621948 learned_addr learned my addr 172.21.15.187:0/2581621948 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:00.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.649+0000 7f2e4b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2e4c145cc0 0x7f2e4c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.649+0000 7f2e43fff700 1 --2- 172.21.15.187:0/2581621948 >> v2:172.21.15.5:3300/0 conn(0x7f2e4c0683f0 0x7f2e4c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.649+0000 7f2e4b7fe700 1 -- 172.21.15.187:0/2581621948 >> v2:172.21.15.5:3300/0 conn(0x7f2e4c0683f0 msgr2=0x7f2e4c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:00.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.649+0000 7f2e4affd700 1 -- 172.21.15.187:0/2581621948 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159981706 0 0) 0x7f2e4c14c170 con 0x7f2e4c065680 2020-11-20T12:18:00.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.650+0000 7f2e4b7fe700 1 --2- 172.21.15.187:0/2581621948 >> v2:172.21.15.5:3300/0 conn(0x7f2e4c0683f0 0x7f2e4c065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.650+0000 7f2e4b7fe700 1 -- 172.21.15.187:0/2581621948 >> v1:172.21.15.187:6789/0 conn(0x7f2e4c065680 legacy=0x7f2e4c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.650+0000 7f2e4b7fe700 1 -- 172.21.15.187:0/2581621948 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c14c6b0 con 0x7f2e4c145cc0 2020-11-20T12:18:00.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.650+0000 7f2e43fff700 1 --2- 172.21.15.187:0/2581621948 >> v2:172.21.15.5:3300/0 conn(0x7f2e4c0683f0 0x7f2e4c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:00.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.650+0000 7f2e4b7fe700 1 --2- 172.21.15.187:0/2581621948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c145cc0 0x7f2e4c066040 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 rx=0x7f2e4c0b85e0 tx=0x7f2e3c006df0).ready entity=mon.1 client_cookie=58639fa88780c178 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:00.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.650+0000 7f2e4affd700 1 -- 172.21.15.187:0/2581621948 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c0097b0 con 0x7f2e4c145cc0 2020-11-20T12:18:00.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.650+0000 7f2e4affd700 1 -- 172.21.15.187:0/2581621948 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c009910 con 0x7f2e4c145cc0 2020-11-20T12:18:00.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.650+0000 7f2e4affd700 1 -- 172.21.15.187:0/2581621948 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c009c90 con 0x7f2e4c145cc0 2020-11-20T12:18:00.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.651+0000 7f2e50a57700 1 -- 172.21.15.187:0/2581621948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c145cc0 msgr2=0x7f2e4c066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.651+0000 7f2e50a57700 1 --2- 172.21.15.187:0/2581621948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c145cc0 0x7f2e4c066040 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 rx=0x7f2e4c0b85e0 tx=0x7f2e3c006df0).stop 2020-11-20T12:18:00.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.651+0000 7f2e50a57700 1 -- 172.21.15.187:0/2581621948 shutdown_connections 2020-11-20T12:18:00.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.651+0000 7f2e50a57700 1 --2- 172.21.15.187:0/2581621948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c145cc0 0x7f2e4c066040 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.651+0000 7f2e50a57700 1 --2- 172.21.15.187:0/2581621948 >> v2:172.21.15.5:3300/0 conn(0x7f2e4c0683f0 0x7f2e4c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.651+0000 7f2e50a57700 1 -- 172.21.15.187:0/2581621948 >> 172.21.15.187:0/2581621948 conn(0x7f2e4c0bab30 msgr2=0x7f2e4c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:00.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.651+0000 7f2e50a57700 1 -- 172.21.15.187:0/2581621948 shutdown_connections 2020-11-20T12:18:00.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.651+0000 7f2e50a57700 1 -- 172.21.15.187:0/2581621948 wait complete. 2020-11-20T12:18:00.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.653+0000 7f2e50a57700 1 Processor -- start 2020-11-20T12:18:00.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.653+0000 7f2e50a57700 1 -- start start 2020-11-20T12:18:00.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.654+0000 7f2e50a57700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c065680 0x7f2e4c144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.654+0000 7f2e50a57700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e4c0683f0 0x7f2e4c14e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.654+0000 7f2e50a57700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e4c145cc0 0x7f2e4c0b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.654+0000 7f2e50a57700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c145370 con 0x7f2e4c145cc0 2020-11-20T12:18:00.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.654+0000 7f2e50a57700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c15c2e0 con 0x7f2e4c065680 2020-11-20T12:18:00.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.654+0000 7f2e50a57700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c14d960 con 0x7f2e4c0683f0 2020-11-20T12:18:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.654+0000 7f2e4b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c065680 0x7f2e4c144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e4b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c065680 0x7f2e4c144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51510/0 (socket says 172.21.15.187:51510) 2020-11-20T12:18:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e4b7fe700 1 -- 172.21.15.187:0/3758608467 learned_addr learned my addr 172.21.15.187:0/3758608467 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e43fff700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e4c0683f0 0x7f2e4c14e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e4b7fe700 1 -- 172.21.15.187:0/3758608467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e4c0683f0 msgr2=0x7f2e4c14e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e4b7fe700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e4c0683f0 0x7f2e4c14e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e4b7fe700 1 -- 172.21.15.187:0/3758608467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e4c145cc0 msgr2=0x7f2e4c0b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e4b7fe700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e4c145cc0 0x7f2e4c0b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e4b7fe700 1 -- 172.21.15.187:0/3758608467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c14dc00 con 0x7f2e4c065680 2020-11-20T12:18:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e43fff700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e4c0683f0 0x7f2e4c14e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e4b7fe700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c065680 0x7f2e4c144e80 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 rx=0x7f2e3c006e30 tx=0x7f2e3c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.655+0000 7f2e437fe700 1 -- 172.21.15.187:0/3758608467 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c01e460 con 0x7f2e4c065680 2020-11-20T12:18:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.656+0000 7f2e437fe700 1 -- 172.21.15.187:0/3758608467 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c01e5c0 con 0x7f2e4c065680 2020-11-20T12:18:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.656+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c14eb00 con 0x7f2e4c065680 2020-11-20T12:18:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.656+0000 7f2e437fe700 1 -- 172.21.15.187:0/3758608467 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c01e960 con 0x7f2e4c065680 2020-11-20T12:18:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.656+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c09a860 con 0x7f2e4c065680 2020-11-20T12:18:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.657+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c0662f0 con 0x7f2e4c065680 2020-11-20T12:18:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.657+0000 7f2e437fe700 1 -- 172.21.15.187:0/3758608467 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2e3c025030 con 0x7f2e4c065680 2020-11-20T12:18:00.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.658+0000 7f2e437fe700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2e28056d20 0x7f2e280591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:00.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.658+0000 7f2e437fe700 1 -- 172.21.15.187:0/3758608467 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2e3c071a10 con 0x7f2e4c065680 2020-11-20T12:18:00.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.658+0000 7f2e43fff700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2e28056d20 0x7f2e280591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:00.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.659+0000 7f2e43fff700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2e28056d20 0x7f2e280591e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f2e4c09c1a0 tx=0x7f2e34008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:00.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.662+0000 7f2e437fe700 1 -- 172.21.15.187:0/3758608467 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2e3c046db0 con 0x7f2e4c065680 2020-11-20T12:18:00.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.846+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e4c14dd90 con 0x7f2e4c065680 2020-11-20T12:18:00.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.848+0000 7f2e437fe700 1 -- 172.21.15.187:0/3758608467 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16922 (secure 0 0 0) 0x7f2e3c031230 con 0x7f2e4c065680 2020-11-20T12:18:00.851 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:00.851 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":121,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:16:17.809383+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:16:00.509740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2020-11-20T12:16:02.486990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/3143922581":"2020-11-21T12:16:17.798582+0000","172.21.15.187:6834/3143922581":"2020-11-21T12:16:17.798582+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:18:00.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.850+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2e28056d20 msgr2=0x7f2e280591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.850+0000 7f2e50a57700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2e28056d20 0x7f2e280591e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f2e4c09c1a0 tx=0x7f2e34008040).stop 2020-11-20T12:18:00.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c065680 msgr2=0x7f2e4c144e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:00.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c065680 0x7f2e4c144e80 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 rx=0x7f2e3c006e30 tx=0x7f2e3c015000).stop 2020-11-20T12:18:00.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 shutdown_connections 2020-11-20T12:18:00.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2e28056d20 0x7f2e280591e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e4c145cc0 0x7f2e4c0b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:00.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e4c065680 0x7f2e4c144e80 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 --2- 172.21.15.187:0/3758608467 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e4c0683f0 0x7f2e4c14e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:00.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 >> 172.21.15.187:0/3758608467 conn(0x7f2e4c0bab30 msgr2=0x7f2e4c0b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:00.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 shutdown_connections 2020-11-20T12:18:00.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:00.851+0000 7f2e50a57700 1 -- 172.21.15.187:0/3758608467 wait complete. 2020-11-20T12:18:00.875 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:18:00.875 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:18:01.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.106+0000 7ff743fff700 1 Processor -- start 2020-11-20T12:18:01.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.106+0000 7ff743fff700 1 -- start start 2020-11-20T12:18:01.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.108+0000 7ff743fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7440675d0 0x7ff74414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.108+0000 7ff743fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff7440679b0 0x7ff744147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.108+0000 7ff743fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74414be70 con 0x7ff7440ba9b0 2020-11-20T12:18:01.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.108+0000 7ff743fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74414c110 con 0x7ff7440679b0 2020-11-20T12:18:01.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.108+0000 7ff743fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74414c3b0 con 0x7ff7440675d0 2020-11-20T12:18:01.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.108+0000 7ff7437fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff7440ba9b0 0x7ff7441481e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48126/0 (socket says 172.21.15.187:48126) 2020-11-20T12:18:01.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.108+0000 7ff7437fe700 1 -- 172.21.15.187:0/1390205880 learned_addr learned my addr 172.21.15.187:0/1390205880 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:01.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.108+0000 7ff742ffd700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7440675d0 0x7ff74414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.109+0000 7ff73bfff700 1 --2- 172.21.15.187:0/1390205880 >> v2:172.21.15.5:3300/0 conn(0x7ff7440679b0 0x7ff744147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.109+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634098035 0 0) 0x7ff74414be70 con 0x7ff7440ba9b0 2020-11-20T12:18:01.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.109+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff724003240 con 0x7ff7440ba9b0 2020-11-20T12:18:01.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.109+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1064106788 0 0) 0x7ff724003240 con 0x7ff7440ba9b0 2020-11-20T12:18:01.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.109+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7440675d0 msgr2=0x7ff74414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:01.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.109+0000 7ff7427fc700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7440675d0 0x7ff74414c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.109+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 >> v2:172.21.15.5:3300/0 conn(0x7ff7440679b0 msgr2=0x7ff744147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:01.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 --2- 172.21.15.187:0/1390205880 >> v2:172.21.15.5:3300/0 conn(0x7ff7440679b0 0x7ff744147cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74414cd40 con 0x7ff7440ba9b0 2020-11-20T12:18:01.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff73c001860 con 0x7ff7440ba9b0 2020-11-20T12:18:01.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 >> v1:172.21.15.187:6789/0 conn(0x7ff7440ba9b0 legacy=0x7ff7441481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7440675d0 0x7ff724003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff724003c40 0x7ff7240044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7240049d0 0x7ff724004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74414c110 con 0x7ff724003c40 2020-11-20T12:18:01.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74414c3b0 con 0x7ff7240049d0 2020-11-20T12:18:01.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff724004200 con 0x7ff7440675d0 2020-11-20T12:18:01.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.110+0000 7ff7437fe700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7240049d0 0x7ff724004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff742ffd700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7440675d0 0x7ff724003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff73bfff700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff724003c40 0x7ff7240044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7437fe700 1 -- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7440675d0 msgr2=0x7ff724003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:01.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7437fe700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff7440675d0 0x7ff724003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7437fe700 1 -- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff724003c40 msgr2=0x7ff7240044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:01.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7437fe700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff724003c40 0x7ff7240044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7437fe700 1 -- 172.21.15.187:0/1390205880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff724003240 con 0x7ff7240049d0 2020-11-20T12:18:01.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7437fe700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7240049d0 0x7ff724004e50 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7ff73c00c810 tx=0x7ff7440bde60).ready entity=mon.1 client_cookie=2b71d0de4cc8aa08 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:01.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c005b80 con 0x7ff7240049d0 2020-11-20T12:18:01.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73c005ce0 con 0x7ff7240049d0 2020-11-20T12:18:01.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.111+0000 7ff7427fc700 1 -- 172.21.15.187:0/1390205880 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c007320 con 0x7ff7240049d0 2020-11-20T12:18:01.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.112+0000 7ff743fff700 1 -- 172.21.15.187:0/1390205880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7240049d0 msgr2=0x7ff724004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.112+0000 7ff743fff700 1 --2- 172.21.15.187:0/1390205880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7240049d0 0x7ff724004e50 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7ff73c00c810 tx=0x7ff7440bde60).stop 2020-11-20T12:18:01.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.112+0000 7ff7437fe700 1 -- 172.21.15.187:0/1390205880 reap_dead start 2020-11-20T12:18:01.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.112+0000 7ff743fff700 1 -- 172.21.15.187:0/1390205880 shutdown_connections 2020-11-20T12:18:01.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.112+0000 7ff743fff700 1 -- 172.21.15.187:0/1390205880 >> 172.21.15.187:0/1390205880 conn(0x7ff7440bce50 msgr2=0x7ff7440bd260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:01.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.112+0000 7ff743fff700 1 -- 172.21.15.187:0/1390205880 shutdown_connections 2020-11-20T12:18:01.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.112+0000 7ff743fff700 1 -- 172.21.15.187:0/1390205880 wait complete. 2020-11-20T12:18:01.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.114+0000 7ff743fff700 1 Processor -- start 2020-11-20T12:18:01.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.114+0000 7ff743fff700 1 -- start start 2020-11-20T12:18:01.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.115+0000 7ff743fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff744062ad0 0x7ff74415bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.115+0000 7ff743fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff744062eb0 0x7ff74415c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.115+0000 7ff743fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7440a1390 0x7ff74409fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.115+0000 7ff743fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440b6690 con 0x7ff7440a1390 2020-11-20T12:18:01.129 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.115+0000 7ff743fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440b6090 con 0x7ff744062ad0 2020-11-20T12:18:01.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.115+0000 7ff743fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74415b840 con 0x7ff744062eb0 2020-11-20T12:18:01.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.115+0000 7ff742ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff744062ad0 0x7ff74415bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff742ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff744062ad0 0x7ff74415bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51530/0 (socket says 172.21.15.187:51530) 2020-11-20T12:18:01.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff742ffd700 1 -- 172.21.15.187:0/3453183202 learned_addr learned my addr 172.21.15.187:0/3453183202 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:01.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff742ffd700 1 -- 172.21.15.187:0/3453183202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff744062eb0 msgr2=0x7ff74415c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff742ffd700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff744062eb0 0x7ff74415c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:01.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff742ffd700 1 -- 172.21.15.187:0/3453183202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7440a1390 msgr2=0x7ff74409fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff742ffd700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7440a1390 0x7ff74409fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:01.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff742ffd700 1 -- 172.21.15.187:0/3453183202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74409d210 con 0x7ff744062ad0 2020-11-20T12:18:01.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff742ffd700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff744062ad0 0x7ff74415bce0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 rx=0x7ff734010c20 tx=0x7ff734008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:01.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff73affd700 1 -- 172.21.15.187:0/3453183202 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73400e9f0 con 0x7ff744062ad0 2020-11-20T12:18:01.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff73affd700 1 -- 172.21.15.187:0/3453183202 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73400eb50 con 0x7ff744062ad0 2020-11-20T12:18:01.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff73affd700 1 -- 172.21.15.187:0/3453183202 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73400eed0 con 0x7ff744062ad0 2020-11-20T12:18:01.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.116+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74409d4c0 con 0x7ff744062ad0 2020-11-20T12:18:01.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.117+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7440a04b0 con 0x7ff744062ad0 2020-11-20T12:18:01.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.118+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74415ba00 con 0x7ff744062ad0 2020-11-20T12:18:01.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.118+0000 7ff73affd700 1 -- 172.21.15.187:0/3453183202 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff734024070 con 0x7ff744062ad0 2020-11-20T12:18:01.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.119+0000 7ff73affd700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff720056d20 0x7ff7200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.119+0000 7ff73affd700 1 -- 172.21.15.187:0/3453183202 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff7340709f0 con 0x7ff744062ad0 2020-11-20T12:18:01.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.119+0000 7ff73bfff700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff720056d20 0x7ff7200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.120+0000 7ff73bfff700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff720056d20 0x7ff7200591e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7ff72c006d00 tx=0x7ff72c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:01.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.123+0000 7ff73affd700 1 -- 172.21.15.187:0/3453183202 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff734045d70 con 0x7ff744062ad0 2020-11-20T12:18:01.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.350+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff7440bcb60 con 0x7ff744062ad0 2020-11-20T12:18:01.602 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:18:01.599+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:18:01.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.616+0000 7ff73affd700 1 -- 172.21.15.187:0/3453183202 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v109) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff734049390 con 0x7ff744062ad0 2020-11-20T12:18:01.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff720056d20 msgr2=0x7ff7200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff720056d20 0x7ff7200591e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7ff72c006d00 tx=0x7ff72c008040).stop 2020-11-20T12:18:01.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff744062ad0 msgr2=0x7ff74415bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff744062ad0 0x7ff74415bce0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 rx=0x7ff734010c20 tx=0x7ff734008040).stop 2020-11-20T12:18:01.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 shutdown_connections 2020-11-20T12:18:01.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff720056d20 0x7ff7200591e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7440a1390 0x7ff74409fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:01.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff744062ad0 0x7ff74415bce0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 --2- 172.21.15.187:0/3453183202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff744062eb0 0x7ff74415c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:01.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.619+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 >> 172.21.15.187:0/3453183202 conn(0x7ff7440bce50 msgr2=0x7ff744147bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:01.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.620+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 shutdown_connections 2020-11-20T12:18:01.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.620+0000 7ff743fff700 1 -- 172.21.15.187:0/3453183202 wait complete. 2020-11-20T12:18:01.624 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:18:01.649 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:18:01.740 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:18:01.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.872+0000 7fd533344700 1 Processor -- start 2020-11-20T12:18:01.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.873+0000 7fd533344700 1 -- start start 2020-11-20T12:18:01.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.874+0000 7fd533344700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd52c0b1380 0x7fd52c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.874+0000 7fd533344700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd52c0b1760 0x7fd52c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.874+0000 7fd533344700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52c150390 con 0x7fd52c065680 2020-11-20T12:18:01.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.874+0000 7fd533344700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52c150630 con 0x7fd52c0b1380 2020-11-20T12:18:01.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.874+0000 7fd533344700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52c068510 con 0x7fd52c0b1760 2020-11-20T12:18:01.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.874+0000 7fd531b41700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd52c0b1760 0x7fd52c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.874+0000 7fd531b41700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd52c0b1760 0x7fd52c065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51552/0 (socket says 172.21.15.187:51552) 2020-11-20T12:18:01.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.874+0000 7fd531b41700 1 -- 172.21.15.187:0/1992601404 learned_addr learned my addr 172.21.15.187:0/1992601404 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:01.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd532342700 1 --2- 172.21.15.187:0/1992601404 >> v2:172.21.15.5:3300/0 conn(0x7fd52c0b1380 0x7fd52c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd531b41700 1 -- 172.21.15.187:0/1992601404 >> v2:172.21.15.5:3300/0 conn(0x7fd52c0b1380 msgr2=0x7fd52c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:01.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd531b41700 1 --2- 172.21.15.187:0/1992601404 >> v2:172.21.15.5:3300/0 conn(0x7fd52c0b1380 0x7fd52c150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd531b41700 1 -- 172.21.15.187:0/1992601404 >> v1:172.21.15.5:6789/0 conn(0x7fd52c065680 legacy=0x7fd52c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:01.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd531b41700 1 -- 172.21.15.187:0/1992601404 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c150fc0 con 0x7fd52c0b1760 2020-11-20T12:18:01.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd531b41700 1 --2- 172.21.15.187:0/1992601404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c065190 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 rx=0x7fd52c063a20 tx=0x7fd528006eb0).ready entity=mon.1 client_cookie=342d6384cf0c8c44 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:01.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd531340700 1 -- 172.21.15.187:0/1992601404 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528009890 con 0x7fd52c0b1760 2020-11-20T12:18:01.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd531340700 1 -- 172.21.15.187:0/1992601404 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5280099f0 con 0x7fd52c0b1760 2020-11-20T12:18:01.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.875+0000 7fd531340700 1 -- 172.21.15.187:0/1992601404 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528009d70 con 0x7fd52c0b1760 2020-11-20T12:18:01.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.876+0000 7fd533344700 1 -- 172.21.15.187:0/1992601404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 msgr2=0x7fd52c065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.876+0000 7fd533344700 1 --2- 172.21.15.187:0/1992601404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c065190 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 rx=0x7fd52c063a20 tx=0x7fd528006eb0).stop 2020-11-20T12:18:01.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.876+0000 7fd533344700 1 -- 172.21.15.187:0/1992601404 shutdown_connections 2020-11-20T12:18:01.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.876+0000 7fd533344700 1 --2- 172.21.15.187:0/1992601404 >> v2:172.21.15.5:3300/0 conn(0x7fd52c0b1380 0x7fd52c150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.876+0000 7fd533344700 1 --2- 172.21.15.187:0/1992601404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c065190 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.876+0000 7fd533344700 1 -- 172.21.15.187:0/1992601404 >> 172.21.15.187:0/1992601404 conn(0x7fd52c0bab30 msgr2=0x7fd52c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:01.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.876+0000 7fd533344700 1 -- 172.21.15.187:0/1992601404 shutdown_connections 2020-11-20T12:18:01.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.876+0000 7fd533344700 1 -- 172.21.15.187:0/1992601404 wait complete. 2020-11-20T12:18:01.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.878+0000 7fd533344700 1 Processor -- start 2020-11-20T12:18:01.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.878+0000 7fd533344700 1 -- start start 2020-11-20T12:18:01.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.879+0000 7fd533344700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd52c0b1380 0x7fd52c0b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.879+0000 7fd533344700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c1497c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.879+0000 7fd533344700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd52c149cb0 0x7fd52c15fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.879+0000 7fd533344700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c14a110 con 0x7fd52c149cb0 2020-11-20T12:18:01.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.879+0000 7fd533344700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c0b7400 con 0x7fd52c0b1760 2020-11-20T12:18:01.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.879+0000 7fd533344700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c0b7650 con 0x7fd52c0b1380 2020-11-20T12:18:01.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c1497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c1497c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51560/0 (socket says 172.21.15.187:51560) 2020-11-20T12:18:01.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 -- 172.21.15.187:0/734576082 learned_addr learned my addr 172.21.15.187:0/734576082 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:01.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd532342700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd52c0b1380 0x7fd52c0b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd532b43700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd52c149cb0 0x7fd52c15fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 -- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd52c0b1380 msgr2=0x7fd52c0b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd52c0b1380 0x7fd52c0b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 -- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd52c149cb0 msgr2=0x7fd52c15fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:01.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd52c149cb0 0x7fd52c15fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:01.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 -- 172.21.15.187:0/734576082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c0b78f0 con 0x7fd52c0b1760 2020-11-20T12:18:01.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd532b43700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd52c149cb0 0x7fd52c15fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:01.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.880+0000 7fd531b41700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c1497c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7fd528006fd0 tx=0x7fd528015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:01.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.881+0000 7fd51e7fc700 1 -- 172.21.15.187:0/734576082 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528011dd0 con 0x7fd52c0b1760 2020-11-20T12:18:01.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.881+0000 7fd51e7fc700 1 -- 172.21.15.187:0/734576082 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52801e460 con 0x7fd52c0b1760 2020-11-20T12:18:01.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.881+0000 7fd51e7fc700 1 -- 172.21.15.187:0/734576082 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52801e800 con 0x7fd52c0b1760 2020-11-20T12:18:01.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.881+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c0b7b90 con 0x7fd52c0b1760 2020-11-20T12:18:01.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.881+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52c160410 con 0x7fd52c0b1760 2020-11-20T12:18:01.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.882+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52c0b7d20 con 0x7fd52c0b1760 2020-11-20T12:18:01.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.883+0000 7fd51e7fc700 1 -- 172.21.15.187:0/734576082 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd528025030 con 0x7fd52c0b1760 2020-11-20T12:18:01.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.883+0000 7fd51e7fc700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd504056d20 0x7fd5040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:01.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.883+0000 7fd51e7fc700 1 -- 172.21.15.187:0/734576082 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fd5280718e0 con 0x7fd52c0b1760 2020-11-20T12:18:01.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.884+0000 7fd532342700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd504056d20 0x7fd5040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:01.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.884+0000 7fd532342700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd504056d20 0x7fd5040591e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fd52c0a0bb0 tx=0x7fd520008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:01.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:01.887+0000 7fd51e7fc700 1 -- 172.21.15.187:0/734576082 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd528046c80 con 0x7fd52c0b1760 2020-11-20T12:18:02.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.115+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd52c061a20 con 0x7fd52c0b1760 2020-11-20T12:18:02.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.629+0000 7fd51e7fc700 1 -- 172.21.15.187:0/734576082 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v110) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd5280230a0 con 0x7fd52c0b1760 2020-11-20T12:18:02.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.632+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd504056d20 msgr2=0x7fd5040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:02.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.632+0000 7fd533344700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd504056d20 0x7fd5040591e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fd52c0a0bb0 tx=0x7fd520008040).stop 2020-11-20T12:18:02.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.632+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 msgr2=0x7fd52c1497c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:02.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.632+0000 7fd533344700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c1497c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7fd528006fd0 tx=0x7fd528015000).stop 2020-11-20T12:18:02.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.633+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 shutdown_connections 2020-11-20T12:18:02.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.633+0000 7fd533344700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd504056d20 0x7fd5040591e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.633+0000 7fd533344700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd52c149cb0 0x7fd52c15fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.633+0000 7fd533344700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd52c0b1380 0x7fd52c0b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.633+0000 7fd533344700 1 --2- 172.21.15.187:0/734576082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd52c0b1760 0x7fd52c1497c0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.633+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 >> 172.21.15.187:0/734576082 conn(0x7fd52c0bab30 msgr2=0x7fd52c14a960 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:02.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.633+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 shutdown_connections 2020-11-20T12:18:02.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.633+0000 7fd533344700 1 -- 172.21.15.187:0/734576082 wait complete. 2020-11-20T12:18:02.662 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:18:02.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.919+0000 7fa0218da700 1 Processor -- start 2020-11-20T12:18:02.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.919+0000 7fa0218da700 1 -- start start 2020-11-20T12:18:02.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.920+0000 7fa0218da700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa01c0683f0 0x7fa01c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:02.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.920+0000 7fa0218da700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c1487e0 0x7fa01c148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:02.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.920+0000 7fa0218da700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01c14c170 con 0x7fa01c145cc0 2020-11-20T12:18:02.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.920+0000 7fa0218da700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c14c410 con 0x7fa01c0683f0 2020-11-20T12:18:02.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.920+0000 7fa0218da700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c14c6b0 con 0x7fa01c1487e0 2020-11-20T12:18:02.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.921+0000 7fa01bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa01c0683f0 0x7fa01c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:02.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.922+0000 7fa01bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa01c0683f0 0x7fa01c066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:43828/0 (socket says 172.21.15.187:43828) 2020-11-20T12:18:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.922+0000 7fa01bfff700 1 -- 172.21.15.187:0/2397420655 learned_addr learned my addr 172.21.15.187:0/2397420655 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.922+0000 7fa0210d9700 1 --2- 172.21.15.187:0/2397420655 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c1487e0 0x7fa01c148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.923+0000 7fa01bfff700 1 -- 172.21.15.187:0/2397420655 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c1487e0 msgr2=0x7fa01c148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.923+0000 7fa01bfff700 1 --2- 172.21.15.187:0/2397420655 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c1487e0 0x7fa01c148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.923+0000 7fa01bfff700 1 -- 172.21.15.187:0/2397420655 >> v1:172.21.15.5:6789/0 conn(0x7fa01c145cc0 legacy=0x7fa01c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:02.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.923+0000 7fa01bfff700 1 -- 172.21.15.187:0/2397420655 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c0658a0 con 0x7fa01c0683f0 2020-11-20T12:18:02.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.923+0000 7fa01bfff700 1 --2- 172.21.15.187:0/2397420655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0683f0 0x7fa01c066040 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fa008006fd0 tx=0x7fa008006eb0).ready entity=mon.0 client_cookie=667e8eea63b8bf04 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:02.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa01b7fe700 1 -- 172.21.15.187:0/2397420655 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008009810 con 0x7fa01c0683f0 2020-11-20T12:18:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa01b7fe700 1 -- 172.21.15.187:0/2397420655 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa008009970 con 0x7fa01c0683f0 2020-11-20T12:18:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa01b7fe700 1 -- 172.21.15.187:0/2397420655 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008009cf0 con 0x7fa01c0683f0 2020-11-20T12:18:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa0218da700 1 -- 172.21.15.187:0/2397420655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0683f0 msgr2=0x7fa01c066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa0218da700 1 --2- 172.21.15.187:0/2397420655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0683f0 0x7fa01c066040 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fa008006fd0 tx=0x7fa008006eb0).stop 2020-11-20T12:18:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa0218da700 1 -- 172.21.15.187:0/2397420655 shutdown_connections 2020-11-20T12:18:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa0218da700 1 --2- 172.21.15.187:0/2397420655 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c1487e0 0x7fa01c148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa0218da700 1 --2- 172.21.15.187:0/2397420655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0683f0 0x7fa01c066040 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.924+0000 7fa0218da700 1 -- 172.21.15.187:0/2397420655 >> 172.21.15.187:0/2397420655 conn(0x7fa01c0bab30 msgr2=0x7fa01c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.925+0000 7fa0218da700 1 -- 172.21.15.187:0/2397420655 shutdown_connections 2020-11-20T12:18:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.925+0000 7fa0218da700 1 -- 172.21.15.187:0/2397420655 wait complete. 2020-11-20T12:18:02.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.927+0000 7fa0218da700 1 Processor -- start 2020-11-20T12:18:02.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.927+0000 7fa0218da700 1 -- start start 2020-11-20T12:18:02.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.928+0000 7fa0218da700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0683f0 0x7fa01c09dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:02.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.928+0000 7fa0218da700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa01c145cc0 0x7fa01c0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:02.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.928+0000 7fa0218da700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c1487e0 0x7fa01c1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:02.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.928+0000 7fa0218da700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c09dfd0 con 0x7fa01c1487e0 2020-11-20T12:18:02.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.928+0000 7fa0218da700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c09e260 con 0x7fa01c145cc0 2020-11-20T12:18:02.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.928+0000 7fa0218da700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c09e500 con 0x7fa01c0683f0 2020-11-20T12:18:02.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa01bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa01c145cc0 0x7fa01c0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:02.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa01bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa01c145cc0 0x7fa01c0b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51572/0 (socket says 172.21.15.187:51572) 2020-11-20T12:18:02.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa01bfff700 1 -- 172.21.15.187:0/887898637 learned_addr learned my addr 172.21.15.187:0/887898637 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:02.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa0210d9700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c1487e0 0x7fa01c1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:02.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa0208d8700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0683f0 0x7fa01c09dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:02.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa01bfff700 1 -- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0683f0 msgr2=0x7fa01c09dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:02.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa01bfff700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0683f0 0x7fa01c09dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa01bfff700 1 -- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c1487e0 msgr2=0x7fa01c1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:02.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa01bfff700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c1487e0 0x7fa01c1584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:02.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa01bfff700 1 -- 172.21.15.187:0/887898637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c09e7a0 con 0x7fa01c145cc0 2020-11-20T12:18:02.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa0208d8700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0683f0 0x7fa01c09dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:02.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.929+0000 7fa0210d9700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c1487e0 0x7fa01c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:02.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.930+0000 7fa01bfff700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa01c145cc0 0x7fa01c0b66b0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 rx=0x7fa008009fd0 tx=0x7fa008015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:02.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.930+0000 7fa018ff9700 1 -- 172.21.15.187:0/887898637 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008011dd0 con 0x7fa01c145cc0 2020-11-20T12:18:02.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.930+0000 7fa018ff9700 1 -- 172.21.15.187:0/887898637 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00801e460 con 0x7fa01c145cc0 2020-11-20T12:18:02.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.930+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c0b6ba0 con 0x7fa01c145cc0 2020-11-20T12:18:02.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.930+0000 7fa018ff9700 1 -- 172.21.15.187:0/887898637 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00801e7e0 con 0x7fa01c145cc0 2020-11-20T12:18:02.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.930+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c09b610 con 0x7fa01c145cc0 2020-11-20T12:18:02.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.931+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c09b8c0 con 0x7fa01c145cc0 2020-11-20T12:18:02.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.932+0000 7fa018ff9700 1 -- 172.21.15.187:0/887898637 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa008025030 con 0x7fa01c145cc0 2020-11-20T12:18:02.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.933+0000 7fa018ff9700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ff4056d20 0x7f9ff40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:02.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.933+0000 7fa018ff9700 1 -- 172.21.15.187:0/887898637 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa0080718c0 con 0x7fa01c145cc0 2020-11-20T12:18:02.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.933+0000 7fa0208d8700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ff4056d20 0x7f9ff40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:02.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.934+0000 7fa0208d8700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ff4056d20 0x7f9ff40591e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fa010001570 tx=0x7fa010009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:02.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:02.937+0000 7fa018ff9700 1 -- 172.21.15.187:0/887898637 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa008046c60 con 0x7fa01c145cc0 2020-11-20T12:18:03.127 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.125+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa01c09e930 con 0x7fa01c145cc0 2020-11-20T12:18:03.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.632+0000 7fa018ff9700 1 -- 172.21.15.187:0/887898637 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v123) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa008023020 con 0x7fa01c145cc0 2020-11-20T12:18:03.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.635+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ff4056d20 msgr2=0x7f9ff40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:03.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.635+0000 7fa0218da700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ff4056d20 0x7f9ff40591e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fa010001570 tx=0x7fa010009040).stop 2020-11-20T12:18:03.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.635+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa01c145cc0 msgr2=0x7fa01c0b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:03.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.635+0000 7fa0218da700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa01c145cc0 0x7fa01c0b66b0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 rx=0x7fa008009fd0 tx=0x7fa008015000).stop 2020-11-20T12:18:03.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.636+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 shutdown_connections 2020-11-20T12:18:03.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.636+0000 7fa0218da700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ff4056d20 0x7f9ff40591e0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:03.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.636+0000 7fa0218da700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c1487e0 0x7fa01c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:03.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.636+0000 7fa0218da700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0683f0 0x7fa01c09dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:03.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.636+0000 7fa0218da700 1 --2- 172.21.15.187:0/887898637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa01c145cc0 0x7fa01c0b66b0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:03.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.636+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 >> 172.21.15.187:0/887898637 conn(0x7fa01c0bab30 msgr2=0x7fa01c065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:03.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.636+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 shutdown_connections 2020-11-20T12:18:03.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.636+0000 7fa0218da700 1 -- 172.21.15.187:0/887898637 wait complete. 2020-11-20T12:18:03.642 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:18:03.668 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:18:03.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.902+0000 7fc45bca8700 1 Processor -- start 2020-11-20T12:18:03.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.902+0000 7fc45bca8700 1 -- start start 2020-11-20T12:18:03.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.903+0000 7fc45bca8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc4541483e0 0x7fc454065300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:03.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.903+0000 7fc45bca8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc454061de0 0x7fc45414c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:03.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.903+0000 7fc45bca8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc454067710 con 0x7fc4541487c0 2020-11-20T12:18:03.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.903+0000 7fc45bca8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc454062280 con 0x7fc4541483e0 2020-11-20T12:18:03.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.903+0000 7fc45bca8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc454062520 con 0x7fc454061de0 2020-11-20T12:18:03.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.904+0000 7fc459243700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc454061de0 0x7fc45414c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:03.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.904+0000 7fc459243700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc454061de0 0x7fc45414c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:52470/0 (socket says 172.21.15.187:52470) 2020-11-20T12:18:03.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.904+0000 7fc459243700 1 -- 172.21.15.187:0/2129721231 learned_addr learned my addr 172.21.15.187:0/2129721231 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:03.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.904+0000 7fc459a44700 1 --2- 172.21.15.187:0/2129721231 >> v2:172.21.15.5:3300/0 conn(0x7fc4541483e0 0x7fc454065300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:03.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.905+0000 7fc459243700 1 -- 172.21.15.187:0/2129721231 >> v2:172.21.15.5:3300/0 conn(0x7fc4541483e0 msgr2=0x7fc454065300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:03.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.905+0000 7fc459243700 1 --2- 172.21.15.187:0/2129721231 >> v2:172.21.15.5:3300/0 conn(0x7fc4541483e0 0x7fc454065300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:03.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.905+0000 7fc459243700 1 -- 172.21.15.187:0/2129721231 >> v1:172.21.15.5:6789/0 conn(0x7fc4541487c0 legacy=0x7fc45414c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:03.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.905+0000 7fc459243700 1 -- 172.21.15.187:0/2129721231 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540627c0 con 0x7fc454061de0 2020-11-20T12:18:03.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.906+0000 7fc459243700 1 --2- 172.21.15.187:0/2129721231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc454061de0 0x7fc45414c150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc448006fd0 tx=0x7fc448006eb0).ready entity=mon.2 client_cookie=799570119d8d602a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:03.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc458a42700 1 -- 172.21.15.187:0/2129721231 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448009810 con 0x7fc454061de0 2020-11-20T12:18:03.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc458a42700 1 -- 172.21.15.187:0/2129721231 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc448009970 con 0x7fc454061de0 2020-11-20T12:18:03.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc458a42700 1 -- 172.21.15.187:0/2129721231 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448009cf0 con 0x7fc454061de0 2020-11-20T12:18:03.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc45bca8700 1 -- 172.21.15.187:0/2129721231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc454061de0 msgr2=0x7fc45414c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:03.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2129721231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc454061de0 0x7fc45414c150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc448006fd0 tx=0x7fc448006eb0).stop 2020-11-20T12:18:03.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc45bca8700 1 -- 172.21.15.187:0/2129721231 shutdown_connections 2020-11-20T12:18:03.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2129721231 >> v2:172.21.15.5:3300/0 conn(0x7fc4541483e0 0x7fc454065300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:03.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2129721231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc454061de0 0x7fc45414c150 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:03.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.907+0000 7fc45bca8700 1 -- 172.21.15.187:0/2129721231 >> 172.21.15.187:0/2129721231 conn(0x7fc4540bacf0 msgr2=0x7fc4540bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:03.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.908+0000 7fc45bca8700 1 -- 172.21.15.187:0/2129721231 shutdown_connections 2020-11-20T12:18:03.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.908+0000 7fc45bca8700 1 -- 172.21.15.187:0/2129721231 wait complete. 2020-11-20T12:18:03.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.910+0000 7fc45bca8700 1 Processor -- start 2020-11-20T12:18:03.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.910+0000 7fc45bca8700 1 -- start start 2020-11-20T12:18:03.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.911+0000 7fc45bca8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc454061de0 0x7fc454143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:03.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.911+0000 7fc45bca8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4541483e0 0x7fc4540b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:03.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.911+0000 7fc45bca8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4541487c0 0x7fc4540b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:03.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.911+0000 7fc45bca8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b6910 con 0x7fc454061de0 2020-11-20T12:18:03.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.911+0000 7fc45bca8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4541439f0 con 0x7fc4541483e0 2020-11-20T12:18:03.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.911+0000 7fc45bca8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b7c90 con 0x7fc4541487c0 2020-11-20T12:18:03.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.911+0000 7fc459243700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4541483e0 0x7fc4540b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:03.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459243700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4541483e0 0x7fc4540b77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51586/0 (socket says 172.21.15.187:51586) 2020-11-20T12:18:03.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459243700 1 -- 172.21.15.187:0/2942572110 learned_addr learned my addr 172.21.15.187:0/2942572110 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:03.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459243700 1 -- 172.21.15.187:0/2942572110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4541487c0 msgr2=0x7fc4540b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:03.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459a44700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc454061de0 0x7fc454143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:03.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459243700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4541487c0 0x7fc4540b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:03.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459243700 1 -- 172.21.15.187:0/2942572110 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc454061de0 msgr2=0x7fc454143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:03.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459243700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc454061de0 0x7fc454143500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:03.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459243700 1 -- 172.21.15.187:0/2942572110 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540b44b0 con 0x7fc4541483e0 2020-11-20T12:18:03.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.912+0000 7fc459243700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4541483e0 0x7fc4540b77a0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7fc454064b50 tx=0x7fc448015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:03.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.913+0000 7fc445ffb700 1 -- 172.21.15.187:0/2942572110 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448011d90 con 0x7fc4541483e0 2020-11-20T12:18:03.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.913+0000 7fc445ffb700 1 -- 172.21.15.187:0/2942572110 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44801e460 con 0x7fc4541483e0 2020-11-20T12:18:03.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.913+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4540b61e0 con 0x7fc4541483e0 2020-11-20T12:18:03.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.913+0000 7fc445ffb700 1 -- 172.21.15.187:0/2942572110 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44801e820 con 0x7fc4541483e0 2020-11-20T12:18:03.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.913+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4540b6670 con 0x7fc4541483e0 2020-11-20T12:18:03.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.914+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4540b7e50 con 0x7fc4541483e0 2020-11-20T12:18:03.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.914+0000 7fc445ffb700 1 -- 172.21.15.187:0/2942572110 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc448025030 con 0x7fc4541483e0 2020-11-20T12:18:03.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.915+0000 7fc445ffb700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc42c056d20 0x7fc42c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:03.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.915+0000 7fc445ffb700 1 -- 172.21.15.187:0/2942572110 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5807+0+0 (secure 0 0 0) 0x7fc4480717a0 con 0x7fc4541483e0 2020-11-20T12:18:03.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.915+0000 7fc459a44700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc42c056d20 0x7fc42c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:03.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.916+0000 7fc459a44700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc42c056d20 0x7fc42c0591e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fc450006bd0 tx=0x7fc450008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:03.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:03.919+0000 7fc445ffb700 1 -- 172.21.15.187:0/2942572110 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc448046c70 con 0x7fc4541483e0 2020-11-20T12:18:04.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.092+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc454044f60 con 0x7fc4541483e0 2020-11-20T12:18:04.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.645+0000 7fc445ffb700 1 -- 172.21.15.187:0/2942572110 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v124) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc44804a290 con 0x7fc4541483e0 2020-11-20T12:18:04.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc42c056d20 msgr2=0x7fc42c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:04.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc42c056d20 0x7fc42c0591e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fc450006bd0 tx=0x7fc450008040).stop 2020-11-20T12:18:04.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4541483e0 msgr2=0x7fc4540b77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:04.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4541483e0 0x7fc4540b77a0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7fc454064b50 tx=0x7fc448015000).stop 2020-11-20T12:18:04.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 shutdown_connections 2020-11-20T12:18:04.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc42c056d20 0x7fc42c0591e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:04.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc454061de0 0x7fc454143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:04.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc4541487c0 0x7fc4540b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:04.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 --2- 172.21.15.187:0/2942572110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4541483e0 0x7fc4540b77a0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:04.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 >> 172.21.15.187:0/2942572110 conn(0x7fc4540bacf0 msgr2=0x7fc454147510 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:04.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.649+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 shutdown_connections 2020-11-20T12:18:04.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.650+0000 7fc45bca8700 1 -- 172.21.15.187:0/2942572110 wait complete. 2020-11-20T12:18:04.656 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:18:04.681 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:18:04.682 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:18:04.682 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:18:04.683 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:18:04.683 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:18:04.684 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:18:04.684 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:18:04.685 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:18:04.710 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:18:04.711 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:18:04.722 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:18:04.723 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:18:04.724 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:18:04.726 DEBUG:teuthology.parallel:result is None 2020-11-20T12:18:04.733 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:18:04.734 DEBUG:teuthology.parallel:result is None 2020-11-20T12:18:04.735 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:18:04.905 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:18:04.939 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:18:04.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.967+0000 7efc919b6700 1 Processor -- start 2020-11-20T12:18:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.968+0000 7efc919b6700 1 -- start start 2020-11-20T12:18:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc919b6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efc8c065190 0x7efc8c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc919b6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c065f90 0x7efc8c143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc919b6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c0b13e0 con 0x7efc8c143c70 2020-11-20T12:18:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc919b6700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8c14ab30 con 0x7efc8c065190 2020-11-20T12:18:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc919b6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c062c10 con 0x7efc8c065f90 2020-11-20T12:18:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc909b4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efc8c065190 0x7efc8c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc909b4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efc8c065190 0x7efc8c065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51604/0 (socket says 172.21.15.187:51604) 2020-11-20T12:18:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc909b4700 1 -- 172.21.15.187:0/1794664360 learned_addr learned my addr 172.21.15.187:0/1794664360 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc8bfff700 1 --2- 172.21.15.187:0/1794664360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c065f90 0x7efc8c143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc909b4700 1 -- 172.21.15.187:0/1794664360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c065f90 msgr2=0x7efc8c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:04.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc909b4700 1 --2- 172.21.15.187:0/1794664360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c065f90 0x7efc8c143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc909b4700 1 -- 172.21.15.187:0/1794664360 >> v1:172.21.15.5:6789/0 conn(0x7efc8c143c70 legacy=0x7efc8c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.969+0000 7efc909b4700 1 -- 172.21.15.187:0/1794664360 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0b1640 con 0x7efc8c065190 2020-11-20T12:18:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.970+0000 7efc8bfff700 1 --2- 172.21.15.187:0/1794664360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c065f90 0x7efc8c143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.970+0000 7efc909b4700 1 --2- 172.21.15.187:0/1794664360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065190 0x7efc8c065570 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7efc8c14ac70 tx=0x7efc80006eb0).ready entity=mon.1 client_cookie=6d0a832a662a61e1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.970+0000 7efc8b7fe700 1 -- 172.21.15.187:0/1794664360 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc80009920 con 0x7efc8c065190 2020-11-20T12:18:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.970+0000 7efc8b7fe700 1 -- 172.21.15.187:0/1794664360 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc80009a80 con 0x7efc8c065190 2020-11-20T12:18:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.970+0000 7efc8b7fe700 1 -- 172.21.15.187:0/1794664360 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc80009e00 con 0x7efc8c065190 2020-11-20T12:18:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.971+0000 7efc919b6700 1 -- 172.21.15.187:0/1794664360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065190 msgr2=0x7efc8c065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.971+0000 7efc919b6700 1 --2- 172.21.15.187:0/1794664360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065190 0x7efc8c065570 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7efc8c14ac70 tx=0x7efc80006eb0).stop 2020-11-20T12:18:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.971+0000 7efc919b6700 1 -- 172.21.15.187:0/1794664360 shutdown_connections 2020-11-20T12:18:04.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.971+0000 7efc919b6700 1 --2- 172.21.15.187:0/1794664360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065190 0x7efc8c065570 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:04.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.971+0000 7efc919b6700 1 --2- 172.21.15.187:0/1794664360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c065f90 0x7efc8c143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:04.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.971+0000 7efc919b6700 1 -- 172.21.15.187:0/1794664360 >> 172.21.15.187:0/1794664360 conn(0x7efc8c0bab30 msgr2=0x7efc8c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:04.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.971+0000 7efc919b6700 1 -- 172.21.15.187:0/1794664360 shutdown_connections 2020-11-20T12:18:04.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.971+0000 7efc919b6700 1 -- 172.21.15.187:0/1794664360 wait complete. 2020-11-20T12:18:04.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.972+0000 7efc919b6700 1 Processor -- start 2020-11-20T12:18:04.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.972+0000 7efc919b6700 1 -- start start 2020-11-20T12:18:04.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.973+0000 7efc919b6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc8c065190 0x7efc8c0b9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:04.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.973+0000 7efc919b6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065f90 0x7efc8c142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:04.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.973+0000 7efc919b6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c143c70 0x7efc8c147640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:04.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.973+0000 7efc919b6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c1564b0 con 0x7efc8c065190 2020-11-20T12:18:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.973+0000 7efc919b6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c156750 con 0x7efc8c065f90 2020-11-20T12:18:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.973+0000 7efc919b6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c1569f0 con 0x7efc8c143c70 2020-11-20T12:18:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.973+0000 7efc8bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065f90 0x7efc8c142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc8bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065f90 0x7efc8c142e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51612/0 (socket says 172.21.15.187:51612) 2020-11-20T12:18:04.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc8bfff700 1 -- 172.21.15.187:0/1413225832 learned_addr learned my addr 172.21.15.187:0/1413225832 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:04.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc909b4700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc8c065190 0x7efc8c0b9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:04.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc911b5700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c143c70 0x7efc8c147640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:04.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc8bfff700 1 -- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c143c70 msgr2=0x7efc8c147640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:04.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc8bfff700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c143c70 0x7efc8c147640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:04.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc8bfff700 1 -- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc8c065190 msgr2=0x7efc8c0b9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:04.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc8bfff700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc8c065190 0x7efc8c0b9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:04.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc8bfff700 1 -- 172.21.15.187:0/1413225832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0ba430 con 0x7efc8c065f90 2020-11-20T12:18:04.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc909b4700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc8c065190 0x7efc8c0b9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:04.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc8bfff700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065f90 0x7efc8c142e00 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7efc7800cb50 tx=0x7efc78008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:04.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.974+0000 7efc88ff9700 1 -- 172.21.15.187:0/1413225832 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc780077d0 con 0x7efc8c065f90 2020-11-20T12:18:04.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.975+0000 7efc88ff9700 1 -- 172.21.15.187:0/1413225832 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc78007930 con 0x7efc8c065f90 2020-11-20T12:18:04.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.975+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c156e90 con 0x7efc8c065f90 2020-11-20T12:18:04.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.975+0000 7efc88ff9700 1 -- 172.21.15.187:0/1413225832 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc78007cb0 con 0x7efc8c065f90 2020-11-20T12:18:04.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.975+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8c0a0010 con 0x7efc8c065f90 2020-11-20T12:18:04.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.976+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8c156300 con 0x7efc8c065f90 2020-11-20T12:18:04.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.976+0000 7efc88ff9700 1 -- 172.21.15.187:0/1413225832 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efc78011620 con 0x7efc8c065f90 2020-11-20T12:18:04.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.977+0000 7efc88ff9700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc64056d20 0x7efc640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:04.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.977+0000 7efc88ff9700 1 -- 172.21.15.187:0/1413225832 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5425+0+0 (secure 0 0 0) 0x7efc7806ffc0 con 0x7efc8c065f90 2020-11-20T12:18:04.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.977+0000 7efc909b4700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc64056d20 0x7efc640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:04.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.978+0000 7efc909b4700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc64056d20 0x7efc640591e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7efc8c0b1e20 tx=0x7efc80015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:04.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:04.981+0000 7efc88ff9700 1 -- 172.21.15.187:0/1413225832 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efc78045690 con 0x7efc8c065f90 2020-11-20T12:18:05.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.187+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7efc8c156bb0 con 0x7efc8c065f90 2020-11-20T12:18:05.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.650+0000 7efc88ff9700 1 -- 172.21.15.187:0/1413225832 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v125) v1 ==== 97+0+0 (secure 0 0 0) 0x7efc78048cb0 con 0x7efc8c065f90 2020-11-20T12:18:05.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc64056d20 msgr2=0x7efc640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:05.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc64056d20 0x7efc640591e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7efc8c0b1e20 tx=0x7efc80015000).stop 2020-11-20T12:18:05.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065f90 msgr2=0x7efc8c142e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:05.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065f90 0x7efc8c142e00 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7efc7800cb50 tx=0x7efc78008040).stop 2020-11-20T12:18:05.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 shutdown_connections 2020-11-20T12:18:05.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc64056d20 0x7efc640591e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:05.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc8c065190 0x7efc8c0b9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:05.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc8c143c70 0x7efc8c147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:05.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 --2- 172.21.15.187:0/1413225832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc8c065f90 0x7efc8c142e00 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:05.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 >> 172.21.15.187:0/1413225832 conn(0x7efc8c0bab30 msgr2=0x7efc8c0b67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:05.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.654+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 shutdown_connections 2020-11-20T12:18:05.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.655+0000 7efc919b6700 1 -- 172.21.15.187:0/1413225832 wait complete. 2020-11-20T12:18:05.660 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:18:05.685 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:18:05.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.931+0000 7f591e466700 1 Processor -- start 2020-11-20T12:18:05.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.932+0000 7f591e466700 1 -- start start 2020-11-20T12:18:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.933+0000 7f591e466700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f59180b6650 0x7f59180afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:05.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.933+0000 7f591e466700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59180b1300 con 0x7f59181484d0 2020-11-20T12:18:05.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.933+0000 7f591e466700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59180b15a0 con 0x7f5918066f70 2020-11-20T12:18:05.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.933+0000 7f591e466700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5918067730 con 0x7f59180b6650 2020-11-20T12:18:05.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.934+0000 7f591dc65700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5918066f70 0x7f59180aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48228/0 (socket says 172.21.15.187:48228) 2020-11-20T12:18:05.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.934+0000 7f591dc65700 1 -- 172.21.15.187:0/2308187399 learned_addr learned my addr 172.21.15.187:0/2308187399 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:05.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.934+0000 7f591cc63700 1 --2- 172.21.15.187:0/2308187399 >> v2:172.21.15.5:3300/0 conn(0x7f59180b6650 0x7f59180afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:05.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.934+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397116550 0 0) 0x7f59180b15a0 con 0x7f5918066f70 2020-11-20T12:18:05.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.934+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5900003240 con 0x7f5918066f70 2020-11-20T12:18:05.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1355764073 0 0) 0x7f5900003240 con 0x7f5918066f70 2020-11-20T12:18:05.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 >> v2:172.21.15.5:3300/0 conn(0x7f59180b6650 msgr2=0x7f59180afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:05.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 --2- 172.21.15.187:0/2308187399 >> v2:172.21.15.5:3300/0 conn(0x7f59180b6650 0x7f59180afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:05.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 >> v1:172.21.15.5:6789/0 conn(0x7f59181484d0 legacy=0x7f5918061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:05.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59180b1840 con 0x7f5918066f70 2020-11-20T12:18:05.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5910001860 con 0x7f5918066f70 2020-11-20T12:18:05.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 >> v1:172.21.15.187:6789/0 conn(0x7f5918066f70 legacy=0x7f59180aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:05.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5900003800 0x7f5900003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:05.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59000044e0 0x7f5900004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:05.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5900004e50 0x7f590000d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:05.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5918067730 con 0x7f5900004e50 2020-11-20T12:18:05.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5900005300 con 0x7f59000044e0 2020-11-20T12:18:05.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590000de30 con 0x7f5900003800 2020-11-20T12:18:05.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.935+0000 7f591cc63700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59000044e0 0x7f5900004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:05.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f591cc63700 1 -- 172.21.15.187:0/2308187399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5900003800 msgr2=0x7f5900003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:05.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f591cc63700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5900003800 0x7f5900003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:05.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f591cc63700 1 -- 172.21.15.187:0/2308187399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5900004e50 msgr2=0x7f590000d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:05.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f591cc63700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5900004e50 0x7f590000d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:05.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f591cc63700 1 -- 172.21.15.187:0/2308187399 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900003240 con 0x7f59000044e0 2020-11-20T12:18:05.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f591dc65700 1 -- 172.21.15.187:0/2308187399 reap_dead start 2020-11-20T12:18:05.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f591cc63700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59000044e0 0x7f5900004960 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 rx=0x7f590800bc00 tx=0x7f5908006df0).ready entity=mon.1 client_cookie=424e9e3a65b2b560 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:05.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590800b310 con 0x7f59000044e0 2020-11-20T12:18:05.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590800eae0 con 0x7f59000044e0 2020-11-20T12:18:05.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.936+0000 7f5907fff700 1 -- 172.21.15.187:0/2308187399 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590800ee60 con 0x7f59000044e0 2020-11-20T12:18:05.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.937+0000 7f591e466700 1 -- 172.21.15.187:0/2308187399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59000044e0 msgr2=0x7f5900004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:05.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.937+0000 7f591e466700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59000044e0 0x7f5900004960 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 rx=0x7f590800bc00 tx=0x7f5908006df0).stop 2020-11-20T12:18:05.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.937+0000 7f591e466700 1 -- 172.21.15.187:0/2308187399 shutdown_connections 2020-11-20T12:18:05.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.937+0000 7f591e466700 1 --2- 172.21.15.187:0/2308187399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59000044e0 0x7f5900004960 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:05.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.937+0000 7f591e466700 1 -- 172.21.15.187:0/2308187399 >> 172.21.15.187:0/2308187399 conn(0x7f59180bab30 msgr2=0x7f59180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:05.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.937+0000 7f591e466700 1 -- 172.21.15.187:0/2308187399 shutdown_connections 2020-11-20T12:18:05.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.937+0000 7f591e466700 1 -- 172.21.15.187:0/2308187399 wait complete. 2020-11-20T12:18:05.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.939+0000 7f591e466700 1 Processor -- start 2020-11-20T12:18:05.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.939+0000 7f591e466700 1 -- start start 2020-11-20T12:18:05.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591e466700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f591809a830 0x7f59180b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:05.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591e466700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f591809ac10 0x7f5918144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:05.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591e466700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59180b3eb0 0x7f5918145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:05.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591e466700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59180b43c0 con 0x7f59180b3eb0 2020-11-20T12:18:05.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591e466700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59180b4660 con 0x7f591809a830 2020-11-20T12:18:05.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591e466700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5918145970 con 0x7f591809ac10 2020-11-20T12:18:05.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591d464700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f591809a830 0x7f59180b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:05.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591d464700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f591809a830 0x7f59180b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51630/0 (socket says 172.21.15.187:51630) 2020-11-20T12:18:05.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591d464700 1 -- 172.21.15.187:0/2235733959 learned_addr learned my addr 172.21.15.187:0/2235733959 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:05.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591dc65700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59180b3eb0 0x7f5918145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:05.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.940+0000 7f591cc63700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f591809ac10 0x7f5918144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:05.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f591d464700 1 -- 172.21.15.187:0/2235733959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f591809ac10 msgr2=0x7f5918144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:05.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f591d464700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f591809ac10 0x7f5918144f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:05.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f591d464700 1 -- 172.21.15.187:0/2235733959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59180b3eb0 msgr2=0x7f5918145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:05.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f591d464700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59180b3eb0 0x7f5918145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:05.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f591d464700 1 -- 172.21.15.187:0/2235733959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591809c120 con 0x7f591809a830 2020-11-20T12:18:05.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f591d464700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f591809a830 0x7f59180b1a40 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7f591400c710 tx=0x7f5914007bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:05.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f59057fa700 1 -- 172.21.15.187:0/2235733959 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400e490 con 0x7f591809a830 2020-11-20T12:18:05.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f59057fa700 1 -- 172.21.15.187:0/2235733959 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591400e610 con 0x7f591809a830 2020-11-20T12:18:05.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591809b270 con 0x7f591809a830 2020-11-20T12:18:05.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.941+0000 7f59057fa700 1 -- 172.21.15.187:0/2235733959 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400e9d0 con 0x7f591809a830 2020-11-20T12:18:05.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.942+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59181490a0 con 0x7f591809a830 2020-11-20T12:18:05.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.943+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5918063ef0 con 0x7f591809a830 2020-11-20T12:18:05.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.943+0000 7f59057fa700 1 -- 172.21.15.187:0/2235733959 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5914010070 con 0x7f591809a830 2020-11-20T12:18:05.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.944+0000 7f59057fa700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f0056d20 0x7f58f00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:05.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.944+0000 7f59057fa700 1 -- 172.21.15.187:0/2235733959 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f591406f2a0 con 0x7f591809a830 2020-11-20T12:18:05.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.944+0000 7f591cc63700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f0056d20 0x7f58f00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:05.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.945+0000 7f591cc63700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f0056d20 0x7f58f00591e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f5918045400 tx=0x7f5908019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:05.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:05.948+0000 7f59057fa700 1 -- 172.21.15.187:0/2235733959 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5914044a20 con 0x7f591809a830 2020-11-20T12:18:06.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.199+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5918061a20 con 0x7f591809a830 2020-11-20T12:18:06.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.200+0000 7f59057fa700 1 -- 172.21.15.187:0/2235733959 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f5914007cf0 con 0x7f591809a830 2020-11-20T12:18:06.204 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:06.204 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:18:06.205 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:18:06.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.205 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:18:06.206 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:18:06.206 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.206 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:18:06.206 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.206 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.207 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.207 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.207 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:18:06.208 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:18:06.208 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.208 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:18:06.208 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.209 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.209 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.209 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.210 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:18:06.210 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:18:06.210 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.211 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.211 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.211 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.211 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.212 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:18:06.212 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:18:06.212 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.212 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.213 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.213 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.213 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.214 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:18:06.214 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:18:06.214 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.214 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.215 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.215 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.215 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.216 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:18:06.216 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:18:06.216 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.216 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.217 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.217 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.218 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:18:06.218 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:18:06.218 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.218 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.218 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.219 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.219 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.220 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:18:06.220 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:18:06.220 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.220 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.220 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.221 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.221 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.221 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:18:06.222 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:18:06.222 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.222 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.222 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.223 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.223 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.223 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:18:06.224 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:18:06.224 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.224 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.224 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.224 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.225 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.225 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:18:06.226 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:18:06.226 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.226 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:18:06.226 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:18:06.226 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:18:06.227 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.227 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:18:06.227 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:18:06.228 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.228 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:18:06.228 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:18:06.228 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:18:06.228 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:18:06.229 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.229 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:18:06.230 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:18:06.230 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.230 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:18:06.230 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:18:06.230 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:18:06.231 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:18:06.231 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.231 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:18:06.232 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:18:06.232 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.232 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:18:06.232 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:18:06.232 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:18:06.233 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:18:06.233 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.234 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:18:06.234 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:18:06.234 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.234 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:18:06.234 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:18:06.235 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:18:06.235 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.235 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.236 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:18:06.236 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:18:06.236 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.236 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:18:06.236 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:18:06.237 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.237 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:06.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:06.237 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:18:06.238 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:18:06.238 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:18:06.238 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:18:06.238 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:18:06.239 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:18:06.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:06.239 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:18:06.239 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:18:06.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f0056d20 msgr2=0x7f58f00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f0056d20 0x7f58f00591e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f5918045400 tx=0x7f5908019000).stop 2020-11-20T12:18:06.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f591809a830 msgr2=0x7f59180b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f591809a830 0x7f59180b1a40 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7f591400c710 tx=0x7f5914007bd0).stop 2020-11-20T12:18:06.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 shutdown_connections 2020-11-20T12:18:06.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f58f0056d20 0x7f58f00591e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59180b3eb0 0x7f5918145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f591809a830 0x7f59180b1a40 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.202+0000 7f591e466700 1 --2- 172.21.15.187:0/2235733959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f591809ac10 0x7f5918144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.203+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 >> 172.21.15.187:0/2235733959 conn(0x7f59180bab30 msgr2=0x7f5918065ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:06.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.203+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 shutdown_connections 2020-11-20T12:18:06.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.203+0000 7f591e466700 1 -- 172.21.15.187:0/2235733959 wait complete. 2020-11-20T12:18:06.244 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:18:06.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.487+0000 7f34954d3700 1 Processor -- start 2020-11-20T12:18:06.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.487+0000 7f34954d3700 1 -- start start 2020-11-20T12:18:06.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.488+0000 7f34954d3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3490065190 0x7f3490065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.489+0000 7f34954d3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f349009ca90 0x7f349009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.489+0000 7f34954d3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34900b13e0 con 0x7f3490065f90 2020-11-20T12:18:06.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.489+0000 7f34954d3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349014ab30 con 0x7f349009ca90 2020-11-20T12:18:06.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.489+0000 7f34954d3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3490062c10 con 0x7f3490065190 2020-11-20T12:18:06.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.489+0000 7f348ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3490065190 0x7f3490065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.489+0000 7f348ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3490065190 0x7f3490065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51638/0 (socket says 172.21.15.187:51638) 2020-11-20T12:18:06.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.489+0000 7f348ffff700 1 -- 172.21.15.187:0/1807925603 learned_addr learned my addr 172.21.15.187:0/1807925603 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:06.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.489+0000 7f3494cd2700 1 --2- 172.21.15.187:0/1807925603 >> v2:172.21.15.5:3300/0 conn(0x7f349009ca90 0x7f349009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f348ffff700 1 -- 172.21.15.187:0/1807925603 >> v2:172.21.15.5:3300/0 conn(0x7f349009ca90 msgr2=0x7f349009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:06.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f348ffff700 1 --2- 172.21.15.187:0/1807925603 >> v2:172.21.15.5:3300/0 conn(0x7f349009ca90 0x7f349009cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f348ffff700 1 -- 172.21.15.187:0/1807925603 >> v1:172.21.15.5:6789/0 conn(0x7f3490065f90 legacy=0x7f3490143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:06.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f348ffff700 1 -- 172.21.15.187:0/1807925603 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34900b1640 con 0x7f3490065190 2020-11-20T12:18:06.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f3494cd2700 1 --2- 172.21.15.187:0/1807925603 >> v2:172.21.15.5:3300/0 conn(0x7f349009ca90 0x7f349009cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:06.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f348ffff700 1 --2- 172.21.15.187:0/1807925603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3490065190 0x7f3490065570 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 rx=0x7f349005c8b0 tx=0x7f3480006eb0).ready entity=mon.1 client_cookie=cc1b6a1b873c204d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:06.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f348f7fe700 1 -- 172.21.15.187:0/1807925603 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3480009920 con 0x7f3490065190 2020-11-20T12:18:06.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f348f7fe700 1 -- 172.21.15.187:0/1807925603 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3480009a80 con 0x7f3490065190 2020-11-20T12:18:06.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.490+0000 7f348f7fe700 1 -- 172.21.15.187:0/1807925603 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3480009e00 con 0x7f3490065190 2020-11-20T12:18:06.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.491+0000 7f34954d3700 1 -- 172.21.15.187:0/1807925603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3490065190 msgr2=0x7f3490065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.491+0000 7f34954d3700 1 --2- 172.21.15.187:0/1807925603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3490065190 0x7f3490065570 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 rx=0x7f349005c8b0 tx=0x7f3480006eb0).stop 2020-11-20T12:18:06.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.491+0000 7f34954d3700 1 -- 172.21.15.187:0/1807925603 shutdown_connections 2020-11-20T12:18:06.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.491+0000 7f34954d3700 1 --2- 172.21.15.187:0/1807925603 >> v2:172.21.15.5:3300/0 conn(0x7f349009ca90 0x7f349009cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.491+0000 7f34954d3700 1 --2- 172.21.15.187:0/1807925603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3490065190 0x7f3490065570 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.491+0000 7f34954d3700 1 -- 172.21.15.187:0/1807925603 >> 172.21.15.187:0/1807925603 conn(0x7f34900bab30 msgr2=0x7f34900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:06.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.491+0000 7f34954d3700 1 -- 172.21.15.187:0/1807925603 shutdown_connections 2020-11-20T12:18:06.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.491+0000 7f34954d3700 1 -- 172.21.15.187:0/1807925603 wait complete. 2020-11-20T12:18:06.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.493+0000 7f34954d3700 1 Processor -- start 2020-11-20T12:18:06.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.493+0000 7f34954d3700 1 -- start start 2020-11-20T12:18:06.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f34954d3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3490065190 0x7f34900680a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f34954d3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3490065f90 0x7f3490142b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f34954d3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f349009ca90 0x7f349009bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f34954d3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3490068590 con 0x7f3490065190 2020-11-20T12:18:06.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f34954d3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349009f760 con 0x7f349009ca90 2020-11-20T12:18:06.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f34954d3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349009fa00 con 0x7f3490065f90 2020-11-20T12:18:06.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f3494cd2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f349009ca90 0x7f349009bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f3494cd2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f349009ca90 0x7f349009bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51648/0 (socket says 172.21.15.187:51648) 2020-11-20T12:18:06.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f3494cd2700 1 -- 172.21.15.187:0/1351557115 learned_addr learned my addr 172.21.15.187:0/1351557115 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:06.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.494+0000 7f348ffff700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3490065190 0x7f34900680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f3494cd2700 1 -- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3490065f90 msgr2=0x7f3490142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f34877fe700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3490065f90 0x7f3490142b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f3494cd2700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3490065f90 0x7f3490142b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f3494cd2700 1 -- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3490065190 msgr2=0x7f34900680a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f3494cd2700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3490065190 0x7f34900680a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f3494cd2700 1 -- 172.21.15.187:0/1351557115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34900b2f60 con 0x7f349009ca90 2020-11-20T12:18:06.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f348ffff700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3490065190 0x7f34900680a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:06.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f3494cd2700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f349009ca90 0x7f349009bdf0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f348800cb50 tx=0x7f3488008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:06.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f348cff9700 1 -- 172.21.15.187:0/1351557115 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488007900 con 0x7f349009ca90 2020-11-20T12:18:06.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f348cff9700 1 -- 172.21.15.187:0/1351557115 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3488007a60 con 0x7f349009ca90 2020-11-20T12:18:06.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f348cff9700 1 -- 172.21.15.187:0/1351557115 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488007de0 con 0x7f349009ca90 2020-11-20T12:18:06.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349009f490 con 0x7f349009ca90 2020-11-20T12:18:06.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.495+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3490143050 con 0x7f349009ca90 2020-11-20T12:18:06.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.496+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34900b30f0 con 0x7f349009ca90 2020-11-20T12:18:06.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.497+0000 7f348cff9700 1 -- 172.21.15.187:0/1351557115 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3488011620 con 0x7f349009ca90 2020-11-20T12:18:06.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.497+0000 7f348cff9700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f346c056d20 0x7f346c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.498+0000 7f348cff9700 1 -- 172.21.15.187:0/1351557115 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3488070090 con 0x7f349009ca90 2020-11-20T12:18:06.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.498+0000 7f348ffff700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f346c056d20 0x7f346c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.498+0000 7f348ffff700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f346c056d20 0x7f346c0591e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f348000b1b0 tx=0x7f3480015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:06.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.501+0000 7f348cff9700 1 -- 172.21.15.187:0/1351557115 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3488045700 con 0x7f349009ca90 2020-11-20T12:18:06.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.689+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3490061a20 con 0x7f349009ca90 2020-11-20T12:18:06.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.691+0000 7f348cff9700 1 -- 172.21.15.187:0/1351557115 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+13636 (secure 0 0 0) 0x7f3488048d20 con 0x7f349009ca90 2020-11-20T12:18:06.695 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:06.695 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":125,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:18:05.639149+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:18:06.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f346c056d20 msgr2=0x7f346c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f346c056d20 0x7f346c0591e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f348000b1b0 tx=0x7f3480015000).stop 2020-11-20T12:18:06.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f349009ca90 msgr2=0x7f349009bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f349009ca90 0x7f349009bdf0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f348800cb50 tx=0x7f3488008040).stop 2020-11-20T12:18:06.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 shutdown_connections 2020-11-20T12:18:06.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f346c056d20 0x7f346c0591e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3490065190 0x7f34900680a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f349009ca90 0x7f349009bdf0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 --2- 172.21.15.187:0/1351557115 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3490065f90 0x7f3490142b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 >> 172.21.15.187:0/1351557115 conn(0x7f34900bab30 msgr2=0x7f34900b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:06.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.694+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 shutdown_connections 2020-11-20T12:18:06.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.695+0000 7f34954d3700 1 -- 172.21.15.187:0/1351557115 wait complete. 2020-11-20T12:18:06.719 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:18:06.720 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:18:06.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.956+0000 7efdeffff700 1 Processor -- start 2020-11-20T12:18:06.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.956+0000 7efdeffff700 1 -- start start 2020-11-20T12:18:06.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.958+0000 7efdeffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf0065190 0x7efdf0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.958+0000 7efdeffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efdf0065f90 0x7efdf0143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.958+0000 7efdeffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf00b13e0 con 0x7efdf0143c70 2020-11-20T12:18:06.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.958+0000 7efdeffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf014ab30 con 0x7efdf0065f90 2020-11-20T12:18:06.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.958+0000 7efdeffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0062c10 con 0x7efdf0065190 2020-11-20T12:18:06.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.958+0000 7efdef7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7efdf0143c70 0x7efdf009ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48262/0 (socket says 172.21.15.187:48262) 2020-11-20T12:18:06.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.958+0000 7efdef7fe700 1 -- 172.21.15.187:0/3311603538 learned_addr learned my addr 172.21.15.187:0/3311603538 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:06.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.959+0000 7efdeeffd700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf0065190 0x7efdf0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.959+0000 7efde67fc700 1 --2- 172.21.15.187:0/3311603538 >> v2:172.21.15.5:3300/0 conn(0x7efdf0065f90 0x7efdf0143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.959+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491192728 0 0) 0x7efdf00b13e0 con 0x7efdf0143c70 2020-11-20T12:18:06.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.959+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd0003240 con 0x7efdf0143c70 2020-11-20T12:18:06.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.959+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2305734852 0 0) 0x7efdd0003240 con 0x7efdf0143c70 2020-11-20T12:18:06.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf0065190 msgr2=0x7efdf0065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:06.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf0065190 0x7efdf0065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 >> v2:172.21.15.5:3300/0 conn(0x7efdf0065f90 msgr2=0x7efdf0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:06.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 --2- 172.21.15.187:0/3311603538 >> v2:172.21.15.5:3300/0 conn(0x7efdf0065f90 0x7efdf0143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00b1640 con 0x7efdf0143c70 2020-11-20T12:18:06.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7efde8001860 con 0x7efdf0143c70 2020-11-20T12:18:06.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 >> v1:172.21.15.187:6789/0 conn(0x7efdf0143c70 legacy=0x7efdf009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf0065190 0x7efdd00037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd0003d30 0x7efdd00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd00049d0 0x7efdd0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf014ab30 con 0x7efdd0003d30 2020-11-20T12:18:06.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0062c10 con 0x7efdd00049d0 2020-11-20T12:18:06.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd00042f0 con 0x7efdf0065190 2020-11-20T12:18:06.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.960+0000 7efdef7fe700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd00049d0 0x7efdd0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efde67fc700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd0003d30 0x7efdd00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdef7fe700 1 -- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf0065190 msgr2=0x7efdd00037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:06.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdef7fe700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf0065190 0x7efdd00037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:06.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdef7fe700 1 -- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd0003d30 msgr2=0x7efdd00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:06.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdef7fe700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdd0003d30 0x7efdd00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdef7fe700 1 -- 172.21.15.187:0/3311603538 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd0003240 con 0x7efdd00049d0 2020-11-20T12:18:06.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdef7fe700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd00049d0 0x7efdd0004e50 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7efde800c600 tx=0x7efdf00bb650).ready entity=mon.1 client_cookie=782a78be78122612 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:06.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8007a70 con 0x7efdd00049d0 2020-11-20T12:18:06.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde8005b80 con 0x7efdd00049d0 2020-11-20T12:18:06.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.961+0000 7efdee7fc700 1 -- 172.21.15.187:0/3311603538 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde80070d0 con 0x7efdd00049d0 2020-11-20T12:18:06.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.962+0000 7efdeffff700 1 -- 172.21.15.187:0/3311603538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd00049d0 msgr2=0x7efdd0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.962+0000 7efdeffff700 1 --2- 172.21.15.187:0/3311603538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdd00049d0 0x7efdd0004e50 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7efde800c600 tx=0x7efdf00bb650).stop 2020-11-20T12:18:06.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.962+0000 7efdef7fe700 1 -- 172.21.15.187:0/3311603538 reap_dead start 2020-11-20T12:18:06.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.962+0000 7efdeffff700 1 -- 172.21.15.187:0/3311603538 shutdown_connections 2020-11-20T12:18:06.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.962+0000 7efdeffff700 1 -- 172.21.15.187:0/3311603538 >> 172.21.15.187:0/3311603538 conn(0x7efdf00bab30 msgr2=0x7efdf00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:06.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.963+0000 7efdeffff700 1 -- 172.21.15.187:0/3311603538 shutdown_connections 2020-11-20T12:18:06.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.963+0000 7efdeffff700 1 -- 172.21.15.187:0/3311603538 wait complete. 2020-11-20T12:18:06.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.964+0000 7efdeffff700 1 Processor -- start 2020-11-20T12:18:06.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.965+0000 7efdeffff700 1 -- start start 2020-11-20T12:18:06.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdeffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf00b9ac0 0x7efdf00b9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdeffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf009dae0 0x7efdf009f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdeffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf00682a0 0x7efdf009f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdeffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0157e80 con 0x7efdf00b9ac0 2020-11-20T12:18:06.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdeffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0158070 con 0x7efdf00682a0 2020-11-20T12:18:06.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdeffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf009bbc0 con 0x7efdf009dae0 2020-11-20T12:18:06.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdef7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf00682a0 0x7efdf009f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdef7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf00682a0 0x7efdf009f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51668/0 (socket says 172.21.15.187:51668) 2020-11-20T12:18:06.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdef7fe700 1 -- 172.21.15.187:0/98539437 learned_addr learned my addr 172.21.15.187:0/98539437 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:06.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.966+0000 7efdeeffd700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf00b9ac0 0x7efdf00b9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efde67fc700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf009dae0 0x7efdf009f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efdef7fe700 1 -- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf009dae0 msgr2=0x7efdf009f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efdef7fe700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf009dae0 0x7efdf009f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efdef7fe700 1 -- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf00b9ac0 msgr2=0x7efdf00b9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:06.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efdef7fe700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf00b9ac0 0x7efdf00b9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:06.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efdef7fe700 1 -- 172.21.15.187:0/98539437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf009c060 con 0x7efdf00682a0 2020-11-20T12:18:06.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efde67fc700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf009dae0 0x7efdf009f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:06.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efdef7fe700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf00682a0 0x7efdf009f6c0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7efde800a000 tx=0x7efde8013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:06.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efde77fe700 1 -- 172.21.15.187:0/98539437 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8007a70 con 0x7efdf00682a0 2020-11-20T12:18:06.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efde77fe700 1 -- 172.21.15.187:0/98539437 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde8018070 con 0x7efdf00682a0 2020-11-20T12:18:06.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf009c2b0 con 0x7efdf00682a0 2020-11-20T12:18:06.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.967+0000 7efde77fe700 1 -- 172.21.15.187:0/98539437 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde800ced0 con 0x7efdf00682a0 2020-11-20T12:18:06.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.968+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf009c6f0 con 0x7efdf00682a0 2020-11-20T12:18:06.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.969+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf009c970 con 0x7efdf00682a0 2020-11-20T12:18:06.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.969+0000 7efde77fe700 1 -- 172.21.15.187:0/98539437 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efde8010070 con 0x7efdf00682a0 2020-11-20T12:18:06.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.970+0000 7efde77fe700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d20 0x7efdcc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:06.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.970+0000 7efde77fe700 1 -- 172.21.15.187:0/98539437 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efde8071270 con 0x7efdf00682a0 2020-11-20T12:18:06.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.970+0000 7efdeeffd700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d20 0x7efdcc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:06.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.971+0000 7efdeeffd700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d20 0x7efdcc0591e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7efde0006a90 tx=0x7efde0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:06.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:06.974+0000 7efde77fe700 1 -- 172.21.15.187:0/98539437 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efdf009c970 con 0x7efdf00682a0 2020-11-20T12:18:07.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.157+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7efdf009c970 con 0x7efdf00682a0 2020-11-20T12:18:07.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.158+0000 7efde77fe700 1 -- 172.21.15.187:0/98539437 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v125) v1 ==== 110+0+10 (secure 0 0 0) 0x7efdf009c970 con 0x7efdf00682a0 2020-11-20T12:18:07.162 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:18:07.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d20 msgr2=0x7efdcc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d20 0x7efdcc0591e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7efde0006a90 tx=0x7efde0008040).stop 2020-11-20T12:18:07.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf00682a0 msgr2=0x7efdf009f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf00682a0 0x7efdf009f6c0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7efde800a000 tx=0x7efde8013000).stop 2020-11-20T12:18:07.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 shutdown_connections 2020-11-20T12:18:07.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdcc056d20 0x7efdcc0591e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efdf00b9ac0 0x7efdf00b9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efdf00682a0 0x7efdf009f6c0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 --2- 172.21.15.187:0/98539437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efdf009dae0 0x7efdf009f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 >> 172.21.15.187:0/98539437 conn(0x7efdf00bab30 msgr2=0x7efdf0061d00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:07.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 shutdown_connections 2020-11-20T12:18:07.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.161+0000 7efdeffff700 1 -- 172.21.15.187:0/98539437 wait complete. 2020-11-20T12:18:07.190 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:18:07.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.440+0000 7fa33f59e700 1 Processor -- start 2020-11-20T12:18:07.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.441+0000 7fa33f59e700 1 -- start start 2020-11-20T12:18:07.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.442+0000 7fa33f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa340065f90 0x7fa340065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.442+0000 7fa33f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa34009ca90 0x7fa34009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.442+0000 7fa33f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3400b13e0 con 0x7fa340065190 2020-11-20T12:18:07.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.442+0000 7fa33f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34014ab30 con 0x7fa34009ca90 2020-11-20T12:18:07.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.442+0000 7fa33f59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa340062c10 con 0x7fa340065f90 2020-11-20T12:18:07.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.442+0000 7fa33dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa340065f90 0x7fa340065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.443+0000 7fa33dd9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa340065f90 0x7fa340065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51672/0 (socket says 172.21.15.187:51672) 2020-11-20T12:18:07.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.443+0000 7fa33dd9b700 1 -- 172.21.15.187:0/2265162749 learned_addr learned my addr 172.21.15.187:0/2265162749 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:07.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.443+0000 7fa33ed9d700 1 --2- 172.21.15.187:0/2265162749 >> v2:172.21.15.5:3300/0 conn(0x7fa34009ca90 0x7fa34009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.443+0000 7fa33dd9b700 1 -- 172.21.15.187:0/2265162749 >> v2:172.21.15.5:3300/0 conn(0x7fa34009ca90 msgr2=0x7fa34009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.443+0000 7fa33dd9b700 1 --2- 172.21.15.187:0/2265162749 >> v2:172.21.15.5:3300/0 conn(0x7fa34009ca90 0x7fa34009cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.443+0000 7fa33dd9b700 1 -- 172.21.15.187:0/2265162749 >> v1:172.21.15.5:6789/0 conn(0x7fa340065190 legacy=0x7fa340143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.443+0000 7fa33dd9b700 1 -- 172.21.15.187:0/2265162749 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400b1640 con 0x7fa340065f90 2020-11-20T12:18:07.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.443+0000 7fa33dd9b700 1 --2- 172.21.15.187:0/2265162749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065f90 0x7fa340065570 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 rx=0x7fa3400b5fb0 tx=0x7fa334006eb0).ready entity=mon.1 client_cookie=c42b8856fb5d47dc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.444+0000 7fa33d59a700 1 -- 172.21.15.187:0/2265162749 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334009920 con 0x7fa340065f90 2020-11-20T12:18:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.444+0000 7fa33d59a700 1 -- 172.21.15.187:0/2265162749 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa334009a80 con 0x7fa340065f90 2020-11-20T12:18:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.444+0000 7fa33d59a700 1 -- 172.21.15.187:0/2265162749 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334009e00 con 0x7fa340065f90 2020-11-20T12:18:07.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.444+0000 7fa33f59e700 1 -- 172.21.15.187:0/2265162749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065f90 msgr2=0x7fa340065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.444+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2265162749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065f90 0x7fa340065570 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 rx=0x7fa3400b5fb0 tx=0x7fa334006eb0).stop 2020-11-20T12:18:07.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.444+0000 7fa33f59e700 1 -- 172.21.15.187:0/2265162749 shutdown_connections 2020-11-20T12:18:07.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.445+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2265162749 >> v2:172.21.15.5:3300/0 conn(0x7fa34009ca90 0x7fa34009cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.445+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2265162749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065f90 0x7fa340065570 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.445+0000 7fa33f59e700 1 -- 172.21.15.187:0/2265162749 >> 172.21.15.187:0/2265162749 conn(0x7fa3400bab30 msgr2=0x7fa3400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:07.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.445+0000 7fa33f59e700 1 -- 172.21.15.187:0/2265162749 shutdown_connections 2020-11-20T12:18:07.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.445+0000 7fa33f59e700 1 -- 172.21.15.187:0/2265162749 wait complete. 2020-11-20T12:18:07.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.446+0000 7fa33f59e700 1 Processor -- start 2020-11-20T12:18:07.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.446+0000 7fa33f59e700 1 -- start start 2020-11-20T12:18:07.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.447+0000 7fa33f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065190 0x7fa340148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.447+0000 7fa33f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa340065f90 0x7fa340142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.447+0000 7fa33f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa34009ca90 0x7fa340143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.447+0000 7fa33f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa340148380 con 0x7fa34009ca90 2020-11-20T12:18:07.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.447+0000 7fa33f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa340148ac0 con 0x7fa340065190 2020-11-20T12:18:07.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.447+0000 7fa33f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa340148580 con 0x7fa340065f90 2020-11-20T12:18:07.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065190 0x7fa340148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065190 0x7fa340148cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51678/0 (socket says 172.21.15.187:51678) 2020-11-20T12:18:07.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 -- 172.21.15.187:0/2132002143 learned_addr learned my addr 172.21.15.187:0/2132002143 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:07.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 -- 172.21.15.187:0/2132002143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa340065f90 msgr2=0x7fa340142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33dd9b700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa340065f90 0x7fa340142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa340065f90 0x7fa340142d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 -- 172.21.15.187:0/2132002143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa34009ca90 msgr2=0x7fa340143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa34009ca90 0x7fa340143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:07.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 -- 172.21.15.187:0/2132002143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa340148820 con 0x7fa340065190 2020-11-20T12:18:07.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.448+0000 7fa33e59c700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065190 0x7fa340148cb0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 rx=0x7fa330009710 tx=0x7fa330006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.449+0000 7fa32e7fc700 1 -- 172.21.15.187:0/2132002143 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33000d5c0 con 0x7fa340065190 2020-11-20T12:18:07.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.449+0000 7fa32e7fc700 1 -- 172.21.15.187:0/2132002143 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33000d740 con 0x7fa340065190 2020-11-20T12:18:07.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.449+0000 7fa32e7fc700 1 -- 172.21.15.187:0/2132002143 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33000db00 con 0x7fa340065190 2020-11-20T12:18:07.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.449+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34009eab0 con 0x7fa340065190 2020-11-20T12:18:07.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.449+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3400b41f0 con 0x7fa340065190 2020-11-20T12:18:07.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.450+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa340061a20 con 0x7fa340065190 2020-11-20T12:18:07.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.451+0000 7fa32e7fc700 1 -- 172.21.15.187:0/2132002143 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa330012420 con 0x7fa340065190 2020-11-20T12:18:07.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.452+0000 7fa32e7fc700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa314056d20 0x7fa3140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.452+0000 7fa32e7fc700 1 -- 172.21.15.187:0/2132002143 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa33006fc60 con 0x7fa340065190 2020-11-20T12:18:07.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.452+0000 7fa33dd9b700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa314056d20 0x7fa3140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.453+0000 7fa33dd9b700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa314056d20 0x7fa3140591e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fa3400b5fb0 tx=0x7fa334015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.456+0000 7fa32e7fc700 1 -- 172.21.15.187:0/2132002143 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa330045380 con 0x7fa340065190 2020-11-20T12:18:07.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.641+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa3400afc60 con 0x7fa340065190 2020-11-20T12:18:07.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.642+0000 7fa32e7fc700 1 -- 172.21.15.187:0/2132002143 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v125) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa3300489a0 con 0x7fa340065190 2020-11-20T12:18:07.646 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:18:07.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa314056d20 msgr2=0x7fa3140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa314056d20 0x7fa3140591e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fa3400b5fb0 tx=0x7fa334015000).stop 2020-11-20T12:18:07.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065190 msgr2=0x7fa340148cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065190 0x7fa340148cb0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 rx=0x7fa330009710 tx=0x7fa330006df0).stop 2020-11-20T12:18:07.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 shutdown_connections 2020-11-20T12:18:07.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa314056d20 0x7fa3140591e0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa34009ca90 0x7fa340143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:07.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa340065190 0x7fa340148cb0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 --2- 172.21.15.187:0/2132002143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa340065f90 0x7fa340142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.644+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 >> 172.21.15.187:0/2132002143 conn(0x7fa3400bab30 msgr2=0x7fa34009bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:07.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.645+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 shutdown_connections 2020-11-20T12:18:07.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.645+0000 7fa33f59e700 1 -- 172.21.15.187:0/2132002143 wait complete. 2020-11-20T12:18:07.689 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:18:07.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.937+0000 7f95e12eb700 1 Processor -- start 2020-11-20T12:18:07.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.937+0000 7f95e12eb700 1 -- start start 2020-11-20T12:18:07.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.939+0000 7f95e12eb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f95dc0654a0 0x7f95dc0b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.939+0000 7f95e12eb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc0b6c10 0x7f95dc0666e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.939+0000 7f95e12eb700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95dc0b08e0 con 0x7f95dc065880 2020-11-20T12:18:07.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.939+0000 7f95e12eb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95dc0b0b80 con 0x7f95dc0654a0 2020-11-20T12:18:07.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.939+0000 7f95e12eb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0b05f0 con 0x7f95dc0b6c10 2020-11-20T12:18:07.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.939+0000 7f95d27fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f95dc065880 0x7f95dc144e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48294/0 (socket says 172.21.15.187:48294) 2020-11-20T12:18:07.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.939+0000 7f95d27fc700 1 -- 172.21.15.187:0/11750569 learned_addr learned my addr 172.21.15.187:0/11750569 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:07.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95daffd700 1 --2- 172.21.15.187:0/11750569 >> v2:172.21.15.5:3300/0 conn(0x7f95dc0654a0 0x7f95dc0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95db7fe700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc0b6c10 0x7f95dc0666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948930652 0 0) 0x7f95dc0b08e0 con 0x7f95dc065880 2020-11-20T12:18:07.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95bc003240 con 0x7f95dc065880 2020-11-20T12:18:07.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3496641856 0 0) 0x7f95bc003240 con 0x7f95dc065880 2020-11-20T12:18:07.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc0b6c10 msgr2=0x7f95dc0666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:07.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc0b6c10 0x7f95dc0666e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 >> v2:172.21.15.5:3300/0 conn(0x7f95dc0654a0 msgr2=0x7f95dc0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:07.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 --2- 172.21.15.187:0/11750569 >> v2:172.21.15.5:3300/0 conn(0x7f95dc0654a0 0x7f95dc0b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc145570 con 0x7f95dc065880 2020-11-20T12:18:07.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f95c4001660 con 0x7f95dc065880 2020-11-20T12:18:07.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95daffd700 1 --2- 172.21.15.187:0/11750569 >> v2:172.21.15.5:3300/0 conn(0x7f95dc0654a0 0x7f95dc0b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:07.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 >> v1:172.21.15.187:6789/0 conn(0x7f95dc065880 legacy=0x7f95dc144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95bc0037b0 0x7f95bc003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95bc0044e0 0x7f95bc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc0b6c10 0x7f95bc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0b0b80 con 0x7f95bc0037b0 2020-11-20T12:18:07.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0b05f0 con 0x7f95bc0044e0 2020-11-20T12:18:07.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.940+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc005340 con 0x7f95dc0b6c10 2020-11-20T12:18:07.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.941+0000 7f95d27fc700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95bc0044e0 0x7f95bc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.941+0000 7f95daffd700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95bc0037b0 0x7f95bc003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.941+0000 7f95d27fc700 1 -- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc0b6c10 msgr2=0x7f95bc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:07.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.941+0000 7f95db7fe700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc0b6c10 0x7f95bc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.941+0000 7f95d27fc700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc0b6c10 0x7f95bc004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.941+0000 7f95d27fc700 1 -- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95bc0037b0 msgr2=0x7f95bc003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:07.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.941+0000 7f95d27fc700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95bc0037b0 0x7f95bc003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.941+0000 7f95d27fc700 1 -- 172.21.15.187:0/11750569 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc003240 con 0x7f95bc0044e0 2020-11-20T12:18:07.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.942+0000 7f95d27fc700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95bc0044e0 0x7f95bc004960 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 rx=0x7f95c400c200 tx=0x7f95c4007770).ready entity=mon.1 client_cookie=5cde067df581c047 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.942+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4007800 con 0x7f95bc0044e0 2020-11-20T12:18:07.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.942+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c40070d0 con 0x7f95bc0044e0 2020-11-20T12:18:07.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.942+0000 7f95da7fc700 1 -- 172.21.15.187:0/11750569 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c40054c0 con 0x7f95bc0044e0 2020-11-20T12:18:07.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.942+0000 7f95e12eb700 1 -- 172.21.15.187:0/11750569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95bc0044e0 msgr2=0x7f95bc004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.942+0000 7f95e12eb700 1 --2- 172.21.15.187:0/11750569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95bc0044e0 0x7f95bc004960 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 rx=0x7f95c400c200 tx=0x7f95c4007770).stop 2020-11-20T12:18:07.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.942+0000 7f95db7fe700 1 -- 172.21.15.187:0/11750569 reap_dead start 2020-11-20T12:18:07.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.943+0000 7f95e12eb700 1 -- 172.21.15.187:0/11750569 shutdown_connections 2020-11-20T12:18:07.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.943+0000 7f95e12eb700 1 -- 172.21.15.187:0/11750569 >> 172.21.15.187:0/11750569 conn(0x7f95dc0bacd0 msgr2=0x7f95dc0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:07.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.943+0000 7f95e12eb700 1 -- 172.21.15.187:0/11750569 shutdown_connections 2020-11-20T12:18:07.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.943+0000 7f95e12eb700 1 -- 172.21.15.187:0/11750569 wait complete. 2020-11-20T12:18:07.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.944+0000 7f95e12eb700 1 Processor -- start 2020-11-20T12:18:07.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.944+0000 7f95e12eb700 1 -- start start 2020-11-20T12:18:07.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.945+0000 7f95e12eb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc09c060 0x7f95dc13c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.945+0000 7f95e12eb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc142010 0x7f95dc13c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.945+0000 7f95e12eb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc09c440 0x7f95dc09c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.945+0000 7f95e12eb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0b2a00 con 0x7f95dc142010 2020-11-20T12:18:07.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.945+0000 7f95e12eb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0b2c30 con 0x7f95dc09c440 2020-11-20T12:18:07.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.945+0000 7f95e12eb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc09cf30 con 0x7f95dc09c060 2020-11-20T12:18:07.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.945+0000 7f95db7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc09c440 0x7f95dc09c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95db7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc09c440 0x7f95dc09c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51702/0 (socket says 172.21.15.187:51702) 2020-11-20T12:18:07.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95db7fe700 1 -- 172.21.15.187:0/3596545643 learned_addr learned my addr 172.21.15.187:0/3596545643 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:07.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95d27fc700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc142010 0x7f95dc13c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95daffd700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc09c060 0x7f95dc13c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95db7fe700 1 -- 172.21.15.187:0/3596545643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc09c060 msgr2=0x7f95dc13c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95db7fe700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc09c060 0x7f95dc13c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95db7fe700 1 -- 172.21.15.187:0/3596545643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc142010 msgr2=0x7f95dc13c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95db7fe700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc142010 0x7f95dc13c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.946+0000 7f95db7fe700 1 -- 172.21.15.187:0/3596545643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc149b50 con 0x7f95dc09c440 2020-11-20T12:18:07.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.947+0000 7f95db7fe700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc09c440 0x7f95dc09c8c0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f95dc063240 tx=0x7f95d4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.947+0000 7f95d37fe700 1 -- 172.21.15.187:0/3596545643 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4012570 con 0x7f95dc09c440 2020-11-20T12:18:07.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.947+0000 7f95d37fe700 1 -- 172.21.15.187:0/3596545643 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d400c070 con 0x7f95dc09c440 2020-11-20T12:18:07.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.947+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95dc149d40 con 0x7f95dc09c440 2020-11-20T12:18:07.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.947+0000 7f95d37fe700 1 -- 172.21.15.187:0/3596545643 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4012920 con 0x7f95dc09c440 2020-11-20T12:18:07.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.947+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95dc14a230 con 0x7f95dc09c440 2020-11-20T12:18:07.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.948+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95dc063600 con 0x7f95dc09c440 2020-11-20T12:18:07.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.949+0000 7f95d37fe700 1 -- 172.21.15.187:0/3596545643 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f95d4012d10 con 0x7f95dc09c440 2020-11-20T12:18:07.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.949+0000 7f95d37fe700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b8056d20 0x7f95b80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.949+0000 7f95d37fe700 1 -- 172.21.15.187:0/3596545643 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f95d406f690 con 0x7f95dc09c440 2020-11-20T12:18:07.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.950+0000 7f95daffd700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b8056d20 0x7f95b80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.950+0000 7f95daffd700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b8056d20 0x7f95b80591e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f95cc006fd0 tx=0x7f95cc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:07.953+0000 7f95d37fe700 1 -- 172.21.15.187:0/3596545643 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f95d4044d90 con 0x7f95dc09c440 2020-11-20T12:18:08.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.135+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f95dc09a5d0 con 0x7f95dc09c440 2020-11-20T12:18:08.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.136+0000 7f95d37fe700 1 -- 172.21.15.187:0/3596545643 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v125) v1 ==== 84+0+143 (secure 0 0 0) 0x7f95d40483b0 con 0x7f95dc09c440 2020-11-20T12:18:08.139 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:08.139 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:18:08.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:08.140 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:18:08.140 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:18:08.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:18:08.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:18:08.141 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:18:08.141 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:18:08.141 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:18:08.141 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:18:08.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.138+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b8056d20 msgr2=0x7f95b80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:08.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.138+0000 7f95e12eb700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b8056d20 0x7f95b80591e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f95cc006fd0 tx=0x7f95cc008040).stop 2020-11-20T12:18:08.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.138+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc09c440 msgr2=0x7f95dc09c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:08.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.138+0000 7f95e12eb700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc09c440 0x7f95dc09c8c0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f95dc063240 tx=0x7f95d4008040).stop 2020-11-20T12:18:08.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.139+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 shutdown_connections 2020-11-20T12:18:08.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.139+0000 7f95e12eb700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f95b8056d20 0x7f95b80591e0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.139+0000 7f95e12eb700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95dc09c440 0x7f95dc09c8c0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.139+0000 7f95e12eb700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95dc09c060 0x7f95dc13c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.139+0000 7f95e12eb700 1 --2- 172.21.15.187:0/3596545643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95dc142010 0x7f95dc13c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.139+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 >> 172.21.15.187:0/3596545643 conn(0x7f95dc0bacd0 msgr2=0x7f95dc0a0380 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:08.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.139+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 shutdown_connections 2020-11-20T12:18:08.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.139+0000 7f95e12eb700 1 -- 172.21.15.187:0/3596545643 wait complete. 2020-11-20T12:18:08.167 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:18:08.171 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:18:08.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.409+0000 7f3c5d053700 1 Processor -- start 2020-11-20T12:18:08.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.409+0000 7f3c5d053700 1 -- start start 2020-11-20T12:18:08.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c5d053700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3c58060330 0x7f3c580662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:08.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c5d053700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 0x7f3c580b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:08.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c5d053700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5809b540 con 0x7f3c58060710 2020-11-20T12:18:08.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c5d053700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c580b7da0 con 0x7f3c58060330 2020-11-20T12:18:08.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c5d053700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580b8040 con 0x7f3c580676a0 2020-11-20T12:18:08.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c57fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3c58060330 0x7f3c580662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c57fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3c58060330 0x7f3c580662a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51706/0 (socket says 172.21.15.187:51706) 2020-11-20T12:18:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c57fff700 1 -- 172.21.15.187:0/1811721398 learned_addr learned my addr 172.21.15.187:0/1811721398 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c57fff700 1 -- 172.21.15.187:0/1811721398 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 msgr2=0x7f3c580b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c57fff700 1 --2- 172.21.15.187:0/1811721398 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 0x7f3c580b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:08.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c57fff700 1 -- 172.21.15.187:0/1811721398 >> v1:172.21.15.5:6789/0 conn(0x7f3c58060710 legacy=0x7f3c581480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.411+0000 7f3c57fff700 1 -- 172.21.15.187:0/1811721398 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c580b82e0 con 0x7f3c58060330 2020-11-20T12:18:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.412+0000 7f3c57fff700 1 --2- 172.21.15.187:0/1811721398 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c580662a0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f3c48006fd0 tx=0x7f3c48006eb0).ready entity=mon.1 client_cookie=af96b7b6e1968133 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.412+0000 7f3c56ffd700 1 -- 172.21.15.187:0/1811721398 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48009920 con 0x7f3c58060330 2020-11-20T12:18:08.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.412+0000 7f3c56ffd700 1 -- 172.21.15.187:0/1811721398 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c48009a80 con 0x7f3c58060330 2020-11-20T12:18:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.412+0000 7f3c56ffd700 1 -- 172.21.15.187:0/1811721398 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48009e00 con 0x7f3c58060330 2020-11-20T12:18:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.413+0000 7f3c5d053700 1 -- 172.21.15.187:0/1811721398 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 msgr2=0x7f3c580662a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.413+0000 7f3c5d053700 1 --2- 172.21.15.187:0/1811721398 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c580662a0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f3c48006fd0 tx=0x7f3c48006eb0).stop 2020-11-20T12:18:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.413+0000 7f3c5d053700 1 -- 172.21.15.187:0/1811721398 shutdown_connections 2020-11-20T12:18:08.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.413+0000 7f3c5d053700 1 --2- 172.21.15.187:0/1811721398 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 0x7f3c580b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:08.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.413+0000 7f3c5d053700 1 --2- 172.21.15.187:0/1811721398 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c580662a0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.413+0000 7f3c5d053700 1 -- 172.21.15.187:0/1811721398 >> 172.21.15.187:0/1811721398 conn(0x7f3c580bbc20 msgr2=0x7f3c580bc030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.413+0000 7f3c5d053700 1 -- 172.21.15.187:0/1811721398 shutdown_connections 2020-11-20T12:18:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.413+0000 7f3c5d053700 1 -- 172.21.15.187:0/1811721398 wait complete. 2020-11-20T12:18:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.415+0000 7f3c5d053700 1 Processor -- start 2020-11-20T12:18:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.415+0000 7f3c5d053700 1 -- start start 2020-11-20T12:18:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.416+0000 7f3c5d053700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c581492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.416+0000 7f3c5d053700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c58060710 0x7f3c581582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.416+0000 7f3c5d053700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 0x7f3c5814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.416+0000 7f3c5d053700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581497c0 con 0x7f3c58060710 2020-11-20T12:18:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.416+0000 7f3c5d053700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58148b90 con 0x7f3c58060330 2020-11-20T12:18:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.416+0000 7f3c5d053700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58148e30 con 0x7f3c580676a0 2020-11-20T12:18:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.416+0000 7f3c57fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c581492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.416+0000 7f3c57fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c581492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51712/0 (socket says 172.21.15.187:51712) 2020-11-20T12:18:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c57fff700 1 -- 172.21.15.187:0/2329915642 learned_addr learned my addr 172.21.15.187:0/2329915642 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c577fe700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c58060710 0x7f3c581582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c5c852700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 0x7f3c5814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c57fff700 1 -- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 msgr2=0x7f3c5814cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:08.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c57fff700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 0x7f3c5814cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c57fff700 1 -- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c58060710 msgr2=0x7f3c581582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:08.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c57fff700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c58060710 0x7f3c581582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c57fff700 1 -- 172.21.15.187:0/2329915642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c581488b0 con 0x7f3c58060330 2020-11-20T12:18:08.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c577fe700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c58060710 0x7f3c581582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:08.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c57fff700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c581492d0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 rx=0x7f3c580641b0 tx=0x7f3c48015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:08.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c37fff700 1 -- 172.21.15.187:0/2329915642 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4801d5d0 con 0x7f3c58060330 2020-11-20T12:18:08.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c37fff700 1 -- 172.21.15.187:0/2329915642 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4801d730 con 0x7f3c58060330 2020-11-20T12:18:08.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c37fff700 1 -- 172.21.15.187:0/2329915642 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4801daf0 con 0x7f3c58060330 2020-11-20T12:18:08.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.417+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c581587a0 con 0x7f3c58060330 2020-11-20T12:18:08.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.418+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5808eb40 con 0x7f3c58060330 2020-11-20T12:18:08.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.419+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c58148ff0 con 0x7f3c58060330 2020-11-20T12:18:08.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.419+0000 7f3c37fff700 1 -- 172.21.15.187:0/2329915642 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3c48025030 con 0x7f3c58060330 2020-11-20T12:18:08.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.420+0000 7f3c37fff700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c30056d50 0x7f3c30059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:08.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.420+0000 7f3c37fff700 1 -- 172.21.15.187:0/2329915642 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3c48071610 con 0x7f3c58060330 2020-11-20T12:18:08.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.420+0000 7f3c577fe700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c30056d50 0x7f3c30059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:08.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.421+0000 7f3c577fe700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c30056d50 0x7f3c30059210 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f3c5809dd80 tx=0x7f3c4c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:08.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.424+0000 7f3c37fff700 1 -- 172.21.15.187:0/2329915642 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3c48046dc0 con 0x7f3c58060330 2020-11-20T12:18:08.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.612+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f3c580446d0 con 0x7f3c58060330 2020-11-20T12:18:08.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.754+0000 7f3c37fff700 1 -- 172.21.15.187:0/2329915642 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v126) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3c48023030 con 0x7f3c58060330 2020-11-20T12:18:08.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.757+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c30056d50 msgr2=0x7f3c30059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:08.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.757+0000 7f3c5d053700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c30056d50 0x7f3c30059210 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f3c5809dd80 tx=0x7f3c4c008040).stop 2020-11-20T12:18:08.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.757+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 msgr2=0x7f3c581492d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:08.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.757+0000 7f3c5d053700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c581492d0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 rx=0x7f3c580641b0 tx=0x7f3c48015000).stop 2020-11-20T12:18:08.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.758+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 shutdown_connections 2020-11-20T12:18:08.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.758+0000 7f3c5d053700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3c30056d50 0x7f3c30059210 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.758+0000 7f3c5d053700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c580676a0 0x7f3c5814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.758+0000 7f3c5d053700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c58060330 0x7f3c581492d0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.758+0000 7f3c5d053700 1 --2- 172.21.15.187:0/2329915642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c58060710 0x7f3c581582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:08.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.758+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 >> 172.21.15.187:0/2329915642 conn(0x7f3c580bbc20 msgr2=0x7f3c580b9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.758+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 shutdown_connections 2020-11-20T12:18:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:08.758+0000 7f3c5d053700 1 -- 172.21.15.187:0/2329915642 wait complete. 2020-11-20T12:18:08.765 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:18:08.786 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:18:09.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.045+0000 7f088b59e700 1 Processor -- start 2020-11-20T12:18:09.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.045+0000 7f088b59e700 1 -- start start 2020-11-20T12:18:09.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.046+0000 7f088b59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f088c0b57b0 0x7f088c068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.046+0000 7f088b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f088c1482d0 0x7f088c062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.046+0000 7f088b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c062b50 0x7f088c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.046+0000 7f088b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c144b80 con 0x7f088c1482d0 2020-11-20T12:18:09.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.046+0000 7f088b59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c1489e0 con 0x7f088c0b57b0 2020-11-20T12:18:09.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.046+0000 7f088b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c148c80 con 0x7f088c062b50 2020-11-20T12:18:09.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088a59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f088c0b57b0 0x7f088c068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088a59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f088c0b57b0 0x7f088c068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51720/0 (socket says 172.21.15.187:51720) 2020-11-20T12:18:09.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088a59c700 1 -- 172.21.15.187:0/2960754764 learned_addr learned my addr 172.21.15.187:0/2960754764 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:09.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088ad9d700 1 --2- 172.21.15.187:0/2960754764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c062b50 0x7f088c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f0889d9b700 1 --2- 172.21.15.187:0/2960754764 >> v2:172.21.15.5:3300/0 conn(0x7f088c1482d0 0x7f088c062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088a59c700 1 -- 172.21.15.187:0/2960754764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c062b50 msgr2=0x7f088c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:09.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088a59c700 1 --2- 172.21.15.187:0/2960754764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c062b50 0x7f088c065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088a59c700 1 -- 172.21.15.187:0/2960754764 >> v2:172.21.15.5:3300/0 conn(0x7f088c1482d0 msgr2=0x7f088c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:09.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088a59c700 1 --2- 172.21.15.187:0/2960754764 >> v2:172.21.15.5:3300/0 conn(0x7f088c1482d0 0x7f088c062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.047+0000 7f088a59c700 1 -- 172.21.15.187:0/2960754764 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c148f20 con 0x7f088c0b57b0 2020-11-20T12:18:09.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.048+0000 7f088a59c700 1 --2- 172.21.15.187:0/2960754764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c068230 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 rx=0x7f088c0b85e0 tx=0x7f0884006eb0).ready entity=mon.1 client_cookie=7d924c68a9659a67 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:09.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.048+0000 7f088959a700 1 -- 172.21.15.187:0/2960754764 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08840098d0 con 0x7f088c0b57b0 2020-11-20T12:18:09.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.048+0000 7f088959a700 1 -- 172.21.15.187:0/2960754764 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0884009a30 con 0x7f088c0b57b0 2020-11-20T12:18:09.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.048+0000 7f088959a700 1 -- 172.21.15.187:0/2960754764 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0884009db0 con 0x7f088c0b57b0 2020-11-20T12:18:09.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 -- 172.21.15.187:0/2960754764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 msgr2=0x7f088c068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 --2- 172.21.15.187:0/2960754764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c068230 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 rx=0x7f088c0b85e0 tx=0x7f0884006eb0).stop 2020-11-20T12:18:09.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 -- 172.21.15.187:0/2960754764 shutdown_connections 2020-11-20T12:18:09.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 --2- 172.21.15.187:0/2960754764 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c062b50 0x7f088c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 --2- 172.21.15.187:0/2960754764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c068230 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 --2- 172.21.15.187:0/2960754764 >> v2:172.21.15.5:3300/0 conn(0x7f088c1482d0 0x7f088c062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 -- 172.21.15.187:0/2960754764 >> 172.21.15.187:0/2960754764 conn(0x7f088c0bab30 msgr2=0x7f088c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:09.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 -- 172.21.15.187:0/2960754764 shutdown_connections 2020-11-20T12:18:09.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.049+0000 7f088b59e700 1 -- 172.21.15.187:0/2960754764 wait complete. 2020-11-20T12:18:09.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.051+0000 7f088b59e700 1 Processor -- start 2020-11-20T12:18:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.051+0000 7f088b59e700 1 -- start start 2020-11-20T12:18:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f088b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c062b50 0x7f088c09c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f088b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c09fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f088b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c1482d0 0x7f088c15b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f088b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c09b820 con 0x7f088c062b50 2020-11-20T12:18:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f088b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c09bac0 con 0x7f088c0b57b0 2020-11-20T12:18:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f088b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c09bd60 con 0x7f088c1482d0 2020-11-20T12:18:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f0889d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c09fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f0889d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c09fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51724/0 (socket says 172.21.15.187:51724) 2020-11-20T12:18:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.052+0000 7f0889d9b700 1 -- 172.21.15.187:0/2026208125 learned_addr learned my addr 172.21.15.187:0/2026208125 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f0889d9b700 1 -- 172.21.15.187:0/2026208125 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c1482d0 msgr2=0x7f088c15b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f088a59c700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c062b50 0x7f088c09c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f088ad9d700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c1482d0 0x7f088c15b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f0889d9b700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c1482d0 0x7f088c15b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f0889d9b700 1 -- 172.21.15.187:0/2026208125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c062b50 msgr2=0x7f088c09c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f0889d9b700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c062b50 0x7f088c09c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f0889d9b700 1 -- 172.21.15.187:0/2026208125 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c143d30 con 0x7f088c0b57b0 2020-11-20T12:18:09.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f0889d9b700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c09fac0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f087400dcb0 tx=0x7f0874008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f08827fc700 1 -- 172.21.15.187:0/2026208125 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874009b40 con 0x7f088c0b57b0 2020-11-20T12:18:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f08827fc700 1 -- 172.21.15.187:0/2026208125 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0874009ca0 con 0x7f088c0b57b0 2020-11-20T12:18:09.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f08827fc700 1 -- 172.21.15.187:0/2026208125 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874012630 con 0x7f088c0b57b0 2020-11-20T12:18:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088c09c6f0 con 0x7f088c0b57b0 2020-11-20T12:18:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.053+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088c0a06e0 con 0x7f088c0b57b0 2020-11-20T12:18:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.054+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088c061a20 con 0x7f088c0b57b0 2020-11-20T12:18:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.055+0000 7f08827fc700 1 -- 172.21.15.187:0/2026208125 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f08740129a0 con 0x7f088c0b57b0 2020-11-20T12:18:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.056+0000 7f08827fc700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0868056d20 0x7f08680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.056+0000 7f08827fc700 1 -- 172.21.15.187:0/2026208125 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f0874070240 con 0x7f088c0b57b0 2020-11-20T12:18:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.056+0000 7f088a59c700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0868056d20 0x7f08680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.057+0000 7f088a59c700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0868056d20 0x7f08680591e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f088400b1b0 tx=0x7f0884015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:09.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.060+0000 7f08827fc700 1 -- 172.21.15.187:0/2026208125 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0874045750 con 0x7f088c0b57b0 2020-11-20T12:18:09.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.244+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f088c1486b0 con 0x7f088c0b57b0 2020-11-20T12:18:10.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:09.999+0000 7f08827fc700 1 -- 172.21.15.187:0/2026208125 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v128) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0874048d70 con 0x7f088c0b57b0 2020-11-20T12:18:10.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0868056d20 msgr2=0x7f08680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0868056d20 0x7f08680591e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f088400b1b0 tx=0x7f0884015000).stop 2020-11-20T12:18:10.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 msgr2=0x7f088c09fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c09fac0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f087400dcb0 tx=0x7f0874008040).stop 2020-11-20T12:18:10.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 shutdown_connections 2020-11-20T12:18:10.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0868056d20 0x7f08680591e0 secure :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0x7f088400b1b0 tx=0x7f0884015000).stop 2020-11-20T12:18:10.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f088c062b50 0x7f088c09c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f088c1482d0 0x7f088c15b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 --2- 172.21.15.187:0/2026208125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f088c0b57b0 0x7f088c09fac0 secure :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 rx=0x7f087400dcb0 tx=0x7f0874008040).stop 2020-11-20T12:18:10.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.003+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 >> 172.21.15.187:0/2026208125 conn(0x7f088c0bab30 msgr2=0x7f088c146d80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:10.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.004+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 shutdown_connections 2020-11-20T12:18:10.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.005+0000 7f088b59e700 1 -- 172.21.15.187:0/2026208125 wait complete. 2020-11-20T12:18:10.010 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:18:10.037 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:18:10.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.287+0000 7f0caf59e700 1 Processor -- start 2020-11-20T12:18:10.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.287+0000 7f0caf59e700 1 -- start start 2020-11-20T12:18:10.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.288+0000 7f0caf59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0cb00bfbe0 0x7f0cb01480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.288+0000 7f0caf59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0cb009f560 0x7f0cb01485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.288+0000 7f0caf59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb014d110 con 0x7f0cb00bf800 2020-11-20T12:18:10.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.288+0000 7f0caf59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb014c9d0 con 0x7f0cb009f560 2020-11-20T12:18:10.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.288+0000 7f0caf59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb0148aa0 con 0x7f0cb00bfbe0 2020-11-20T12:18:10.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0cae59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0cb00bf800 0x7f0cb014c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48340/0 (socket says 172.21.15.187:48340) 2020-11-20T12:18:10.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0cae59c700 1 -- 172.21.15.187:0/12540762 learned_addr learned my addr 172.21.15.187:0/12540762 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:10.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0ca5d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0cb00bfbe0 0x7f0cb01480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0caed9d700 1 --2- 172.21.15.187:0/12540762 >> v2:172.21.15.5:3300/0 conn(0x7f0cb009f560 0x7f0cb01485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0ca5d9b700 1 -- 172.21.15.187:0/12540762 >> v2:172.21.15.5:3300/0 conn(0x7f0cb009f560 msgr2=0x7f0cb01485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:10.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0ca5d9b700 1 --2- 172.21.15.187:0/12540762 >> v2:172.21.15.5:3300/0 conn(0x7f0cb009f560 0x7f0cb01485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0ca5d9b700 1 -- 172.21.15.187:0/12540762 >> v1:172.21.15.187:6789/0 conn(0x7f0cb00bf800 legacy=0x7f0cb014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0cadd9b700 1 -- 172.21.15.187:0/12540762 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991195689 0 0) 0x7f0cb014d110 con 0x7f0cb00bf800 2020-11-20T12:18:10.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.289+0000 7f0ca5d9b700 1 -- 172.21.15.187:0/12540762 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb014cbd0 con 0x7f0cb00bfbe0 2020-11-20T12:18:10.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.290+0000 7f0ca5d9b700 1 --2- 172.21.15.187:0/12540762 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb00bfbe0 0x7f0cb01480c0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 rx=0x7f0cb00687b0 tx=0x7f0c98006df0).ready entity=mon.1 client_cookie=dd8760fc4fc6b236 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:10.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.290+0000 7f0cadd9b700 1 -- 172.21.15.187:0/12540762 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98009840 con 0x7f0cb00bfbe0 2020-11-20T12:18:10.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.290+0000 7f0cadd9b700 1 -- 172.21.15.187:0/12540762 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c980099a0 con 0x7f0cb00bfbe0 2020-11-20T12:18:10.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.290+0000 7f0cadd9b700 1 -- 172.21.15.187:0/12540762 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98009d20 con 0x7f0cb00bfbe0 2020-11-20T12:18:10.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.291+0000 7f0caf59e700 1 -- 172.21.15.187:0/12540762 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb00bfbe0 msgr2=0x7f0cb01480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.291+0000 7f0caf59e700 1 --2- 172.21.15.187:0/12540762 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb00bfbe0 0x7f0cb01480c0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 rx=0x7f0cb00687b0 tx=0x7f0c98006df0).stop 2020-11-20T12:18:10.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.291+0000 7f0caf59e700 1 -- 172.21.15.187:0/12540762 shutdown_connections 2020-11-20T12:18:10.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.291+0000 7f0caf59e700 1 --2- 172.21.15.187:0/12540762 >> v2:172.21.15.5:3300/0 conn(0x7f0cb009f560 0x7f0cb01485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.291+0000 7f0caf59e700 1 --2- 172.21.15.187:0/12540762 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb00bfbe0 0x7f0cb01480c0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.291+0000 7f0caf59e700 1 -- 172.21.15.187:0/12540762 >> 172.21.15.187:0/12540762 conn(0x7f0cb0068370 msgr2=0x7f0cb00bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:10.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.291+0000 7f0caf59e700 1 -- 172.21.15.187:0/12540762 shutdown_connections 2020-11-20T12:18:10.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.291+0000 7f0caf59e700 1 -- 172.21.15.187:0/12540762 wait complete. 2020-11-20T12:18:10.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.293+0000 7f0caf59e700 1 Processor -- start 2020-11-20T12:18:10.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.293+0000 7f0caf59e700 1 -- start start 2020-11-20T12:18:10.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.294+0000 7f0caf59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb009f560 0x7f0cb0134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.294+0000 7f0caf59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0cb00bf800 0x7f0cb0143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.294+0000 7f0caf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0cb00bb700 0x7f0cb00bbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.294+0000 7f0caf59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0144440 con 0x7f0cb00bf800 2020-11-20T12:18:10.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.294+0000 7f0caf59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb01446e0 con 0x7f0cb009f560 2020-11-20T12:18:10.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.294+0000 7f0caf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0144980 con 0x7f0cb00bb700 2020-11-20T12:18:10.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb009f560 0x7f0cb0134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb009f560 0x7f0cb0134c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51740/0 (socket says 172.21.15.187:51740) 2020-11-20T12:18:10.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 -- 172.21.15.187:0/1273492754 learned_addr learned my addr 172.21.15.187:0/1273492754 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:10.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0ca5d9b700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0cb00bf800 0x7f0cb0143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0caed9d700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0cb00bb700 0x7f0cb00bbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 -- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0cb00bb700 msgr2=0x7f0cb00bbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0cb00bb700 0x7f0cb00bbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 -- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0cb00bf800 msgr2=0x7f0cb0143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0cb00bf800 0x7f0cb0143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 -- 172.21.15.187:0/1273492754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0135170 con 0x7f0cb009f560 2020-11-20T12:18:10.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0caed9d700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0cb00bb700 0x7f0cb00bbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:10.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.295+0000 7f0cae59c700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb009f560 0x7f0cb0134c80 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 rx=0x7f0ca000cae0 tx=0x7f0ca0007de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:10.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.296+0000 7f0ca6ffd700 1 -- 172.21.15.187:0/1273492754 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0005c50 con 0x7f0cb009f560 2020-11-20T12:18:10.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.296+0000 7f0ca6ffd700 1 -- 172.21.15.187:0/1273492754 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0005db0 con 0x7f0cb009f560 2020-11-20T12:18:10.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.296+0000 7f0ca6ffd700 1 -- 172.21.15.187:0/1273492754 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0007340 con 0x7f0cb009f560 2020-11-20T12:18:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.296+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb00b4010 con 0x7f0cb009f560 2020-11-20T12:18:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.296+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb00b5600 con 0x7f0cb009f560 2020-11-20T12:18:10.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.297+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb0080050 con 0x7f0cb009f560 2020-11-20T12:18:10.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.298+0000 7f0ca6ffd700 1 -- 172.21.15.187:0/1273492754 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0ca0010030 con 0x7f0cb009f560 2020-11-20T12:18:10.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.298+0000 7f0ca6ffd700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c8c056d20 0x7f0c8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.299+0000 7f0ca6ffd700 1 -- 172.21.15.187:0/1273492754 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f0ca0070040 con 0x7f0cb009f560 2020-11-20T12:18:10.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.299+0000 7f0ca5d9b700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c8c056d20 0x7f0c8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.300+0000 7f0ca5d9b700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c8c056d20 0x7f0c8c0591e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f0c9800c010 tx=0x7f0c98015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:10.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.303+0000 7f0ca6ffd700 1 -- 172.21.15.187:0/1273492754 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0ca0045500 con 0x7f0cb009f560 2020-11-20T12:18:10.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.554+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f0cb0144b40 con 0x7f0cb009f560 2020-11-20T12:18:10.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.569+0000 7f0ca6ffd700 1 -- 172.21.15.187:0/1273492754 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 25 and data pool 26 v112) v1 ==== 189+0+0 (secure 0 0 0) 0x7f0ca0048b20 con 0x7f0cb009f560 2020-11-20T12:18:10.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.572+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c8c056d20 msgr2=0x7f0c8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.572+0000 7f0caf59e700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c8c056d20 0x7f0c8c0591e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f0c9800c010 tx=0x7f0c98015000).stop 2020-11-20T12:18:10.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.572+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb009f560 msgr2=0x7f0cb0134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.572+0000 7f0caf59e700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb009f560 0x7f0cb0134c80 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 rx=0x7f0ca000cae0 tx=0x7f0ca0007de0).stop 2020-11-20T12:18:10.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.574+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 shutdown_connections 2020-11-20T12:18:10.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.574+0000 7f0caf59e700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0c8c056d20 0x7f0c8c0591e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.574+0000 7f0caf59e700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0cb00bb700 0x7f0cb00bbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.574+0000 7f0caf59e700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb009f560 0x7f0cb0134c80 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.574+0000 7f0caf59e700 1 --2- 172.21.15.187:0/1273492754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0cb00bf800 0x7f0cb0143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.574+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 >> 172.21.15.187:0/1273492754 conn(0x7f0cb0068370 msgr2=0x7f0cb0141ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:10.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.577+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 shutdown_connections 2020-11-20T12:18:10.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.577+0000 7f0caf59e700 1 -- 172.21.15.187:0/1273492754 wait complete. 2020-11-20T12:18:10.582 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 25 and data pool 26 2020-11-20T12:18:10.607 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:18:10.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.820+0000 7f1ef1e54700 1 Processor -- start 2020-11-20T12:18:10.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.821+0000 7f1ef1e54700 1 -- start start 2020-11-20T12:18:10.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.822+0000 7f1ef1e54700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.822+0000 7f1ef1e54700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1eec0683f0 0x7f1eec065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.822+0000 7f1ef1e54700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec14c170 con 0x7f1eec065680 2020-11-20T12:18:10.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.822+0000 7f1ef1e54700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec1497f0 con 0x7f1eec0683f0 2020-11-20T12:18:10.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.822+0000 7f1ef1e54700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec14c410 con 0x7f1eec145cc0 2020-11-20T12:18:10.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.822+0000 7f1eebfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1eec0683f0 0x7f1eec065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.822+0000 7f1eebfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1eec0683f0 0x7f1eec065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51756/0 (socket says 172.21.15.187:51756) 2020-11-20T12:18:10.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.822+0000 7f1eebfff700 1 -- 172.21.15.187:0/2762489152 learned_addr learned my addr 172.21.15.187:0/2762489152 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:10.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.823+0000 7f1ef0e52700 1 --2- 172.21.15.187:0/2762489152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.823+0000 7f1eebfff700 1 -- 172.21.15.187:0/2762489152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 msgr2=0x7f1eec066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:10.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.823+0000 7f1eebfff700 1 --2- 172.21.15.187:0/2762489152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.823+0000 7f1eebfff700 1 -- 172.21.15.187:0/2762489152 >> v1:172.21.15.5:6789/0 conn(0x7f1eec065680 legacy=0x7f1eec061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:10.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.823+0000 7f1eebfff700 1 -- 172.21.15.187:0/2762489152 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec14c6b0 con 0x7f1eec0683f0 2020-11-20T12:18:10.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.823+0000 7f1eebfff700 1 --2- 172.21.15.187:0/2762489152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec0683f0 0x7f1eec065190 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 rx=0x7f1ed8006fd0 tx=0x7f1ed8006eb0).ready entity=mon.1 client_cookie=6891e81b63f4ddf5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:10.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1eeb7fe700 1 -- 172.21.15.187:0/2762489152 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8009890 con 0x7f1eec0683f0 2020-11-20T12:18:10.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1eeb7fe700 1 -- 172.21.15.187:0/2762489152 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed80099f0 con 0x7f1eec0683f0 2020-11-20T12:18:10.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1eeb7fe700 1 -- 172.21.15.187:0/2762489152 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8009d70 con 0x7f1eec0683f0 2020-11-20T12:18:10.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1ef1e54700 1 -- 172.21.15.187:0/2762489152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec0683f0 msgr2=0x7f1eec065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/2762489152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec0683f0 0x7f1eec065190 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 rx=0x7f1ed8006fd0 tx=0x7f1ed8006eb0).stop 2020-11-20T12:18:10.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1ef1e54700 1 -- 172.21.15.187:0/2762489152 shutdown_connections 2020-11-20T12:18:10.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/2762489152 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/2762489152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec0683f0 0x7f1eec065190 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1ef1e54700 1 -- 172.21.15.187:0/2762489152 >> 172.21.15.187:0/2762489152 conn(0x7f1eec0bab30 msgr2=0x7f1eec0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:10.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1ef1e54700 1 -- 172.21.15.187:0/2762489152 shutdown_connections 2020-11-20T12:18:10.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.824+0000 7f1ef1e54700 1 -- 172.21.15.187:0/2762489152 wait complete. 2020-11-20T12:18:10.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.826+0000 7f1ef1e54700 1 Processor -- start 2020-11-20T12:18:10.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.826+0000 7f1ef1e54700 1 -- start start 2020-11-20T12:18:10.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1e54700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1eec065680 0x7f1eec09c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1e54700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec09cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1e54700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec09d3b0 0x7f1eec143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1e54700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec0a0990 con 0x7f1eec065680 2020-11-20T12:18:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1e54700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec09c7c0 con 0x7f1eec09d3b0 2020-11-20T12:18:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1e54700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec09e510 con 0x7f1eec145cc0 2020-11-20T12:18:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1653700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec09d3b0 0x7f1eec143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1653700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec09d3b0 0x7f1eec143d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51766/0 (socket says 172.21.15.187:51766) 2020-11-20T12:18:10.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.827+0000 7f1ef1653700 1 -- 172.21.15.187:0/3152520746 learned_addr learned my addr 172.21.15.187:0/3152520746 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:10.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef0e52700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1eec065680 0x7f1eec09c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1eebfff700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec09cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef1653700 1 -- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 msgr2=0x7f1eec09cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef1653700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec09cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef1653700 1 -- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1eec065680 msgr2=0x7f1eec09c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:10.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef1653700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1eec065680 0x7f1eec09c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:10.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef1653700 1 -- 172.21.15.187:0/3152520746 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec09e7b0 con 0x7f1eec09d3b0 2020-11-20T12:18:10.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1eebfff700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec09cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:10.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef1653700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec09d3b0 0x7f1eec143d10 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f1ee400cab0 tx=0x7f1ee40023b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:10.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ee8ff9700 1 -- 172.21.15.187:0/3152520746 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee400a830 con 0x7f1eec09d3b0 2020-11-20T12:18:10.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ee8ff9700 1 -- 172.21.15.187:0/3152520746 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee400a990 con 0x7f1eec09d3b0 2020-11-20T12:18:10.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec09c500 con 0x7f1eec09d3b0 2020-11-20T12:18:10.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ee8ff9700 1 -- 172.21.15.187:0/3152520746 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee400ad30 con 0x7f1eec09d3b0 2020-11-20T12:18:10.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.828+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec0b5040 con 0x7f1eec09d3b0 2020-11-20T12:18:10.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.830+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec061a20 con 0x7f1eec09d3b0 2020-11-20T12:18:10.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.830+0000 7f1ee8ff9700 1 -- 172.21.15.187:0/3152520746 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1ee4019030 con 0x7f1eec09d3b0 2020-11-20T12:18:10.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.831+0000 7f1ee8ff9700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ec4056d20 0x7f1ec40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:10.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.831+0000 7f1ee8ff9700 1 -- 172.21.15.187:0/3152520746 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ee406f830 con 0x7f1eec09d3b0 2020-11-20T12:18:10.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.831+0000 7f1ef0e52700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ec4056d20 0x7f1ec40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:10.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.832+0000 7f1ef0e52700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ec4056d20 0x7f1ec40591e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7f1eec1457c0 tx=0x7f1ee0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:10.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:10.835+0000 7f1ee8ff9700 1 -- 172.21.15.187:0/3152520746 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1ee4043fd0 con 0x7f1eec09d3b0 2020-11-20T12:18:11.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.022+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1eec14bdf0 con 0x7f1eec09d3b0 2020-11-20T12:18:11.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.024+0000 7f1ee8ff9700 1 -- 172.21.15.187:0/3152520746 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16805 (secure 0 0 0) 0x7f1ee40251c0 con 0x7f1eec09d3b0 2020-11-20T12:18:11.028 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:11.028 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":129,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:18:10.556875+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:18:08.692656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2020-11-20T12:18:09.325174+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:18:11.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ec4056d20 msgr2=0x7f1ec40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ec4056d20 0x7f1ec40591e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7f1eec1457c0 tx=0x7f1ee0008040).stop 2020-11-20T12:18:11.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec09d3b0 msgr2=0x7f1eec143d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec09d3b0 0x7f1eec143d10 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f1ee400cab0 tx=0x7f1ee40023b0).stop 2020-11-20T12:18:11.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 shutdown_connections 2020-11-20T12:18:11.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ec4056d20 0x7f1ec40591e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1eec065680 0x7f1eec09c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1eec09d3b0 0x7f1eec143d10 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 --2- 172.21.15.187:0/3152520746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1eec145cc0 0x7f1eec09cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.027+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 >> 172.21.15.187:0/3152520746 conn(0x7f1eec0bab30 msgr2=0x7f1eec061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:11.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.028+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 shutdown_connections 2020-11-20T12:18:11.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.028+0000 7f1ef1e54700 1 -- 172.21.15.187:0/3152520746 wait complete. 2020-11-20T12:18:11.052 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:18:11.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.299+0000 7faa81bf8700 1 Processor -- start 2020-11-20T12:18:11.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.299+0000 7faa81bf8700 1 -- start start 2020-11-20T12:18:11.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.300+0000 7faa81bf8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faa7c1484d0 0x7faa7c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.300+0000 7faa81bf8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c0622c0 0x7faa7c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.300+0000 7faa81bf8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c0b1300 con 0x7faa7c066f70 2020-11-20T12:18:11.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.300+0000 7faa81bf8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c067730 con 0x7faa7c1484d0 2020-11-20T12:18:11.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.300+0000 7faa81bf8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c0b15a0 con 0x7faa7c0622c0 2020-11-20T12:18:11.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.301+0000 7faa7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c0622c0 0x7faa7c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.301+0000 7faa7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c0622c0 0x7faa7c0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:52660/0 (socket says 172.21.15.187:52660) 2020-11-20T12:18:11.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.301+0000 7faa7bfff700 1 -- 172.21.15.187:0/137095728 learned_addr learned my addr 172.21.15.187:0/137095728 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:11.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.301+0000 7faa80bf6700 1 --2- 172.21.15.187:0/137095728 >> v2:172.21.15.5:3300/0 conn(0x7faa7c1484d0 0x7faa7c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.302+0000 7faa7bfff700 1 -- 172.21.15.187:0/137095728 >> v2:172.21.15.5:3300/0 conn(0x7faa7c1484d0 msgr2=0x7faa7c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:11.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.302+0000 7faa7bfff700 1 --2- 172.21.15.187:0/137095728 >> v2:172.21.15.5:3300/0 conn(0x7faa7c1484d0 0x7faa7c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.302+0000 7faa7bfff700 1 -- 172.21.15.187:0/137095728 >> v1:172.21.15.5:6789/0 conn(0x7faa7c066f70 legacy=0x7faa7c0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:11.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.302+0000 7faa7bfff700 1 -- 172.21.15.187:0/137095728 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c0b1840 con 0x7faa7c0622c0 2020-11-20T12:18:11.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.303+0000 7faa7bfff700 1 --2- 172.21.15.187:0/137095728 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c0622c0 0x7faa7c0b5640 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7faa68006fd0 tx=0x7faa68006eb0).ready entity=mon.2 client_cookie=a87acc3d0a04faa4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:11.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.303+0000 7faa7b7fe700 1 -- 172.21.15.187:0/137095728 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa680097d0 con 0x7faa7c0622c0 2020-11-20T12:18:11.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.303+0000 7faa7b7fe700 1 -- 172.21.15.187:0/137095728 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa68009930 con 0x7faa7c0622c0 2020-11-20T12:18:11.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.303+0000 7faa7b7fe700 1 -- 172.21.15.187:0/137095728 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68009cb0 con 0x7faa7c0622c0 2020-11-20T12:18:11.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.304+0000 7faa81bf8700 1 -- 172.21.15.187:0/137095728 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c0622c0 msgr2=0x7faa7c0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.304+0000 7faa81bf8700 1 --2- 172.21.15.187:0/137095728 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c0622c0 0x7faa7c0b5640 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7faa68006fd0 tx=0x7faa68006eb0).stop 2020-11-20T12:18:11.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.304+0000 7faa81bf8700 1 -- 172.21.15.187:0/137095728 shutdown_connections 2020-11-20T12:18:11.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.304+0000 7faa81bf8700 1 --2- 172.21.15.187:0/137095728 >> v2:172.21.15.5:3300/0 conn(0x7faa7c1484d0 0x7faa7c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.304+0000 7faa81bf8700 1 --2- 172.21.15.187:0/137095728 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c0622c0 0x7faa7c0b5640 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.304+0000 7faa81bf8700 1 -- 172.21.15.187:0/137095728 >> 172.21.15.187:0/137095728 conn(0x7faa7c0bab30 msgr2=0x7faa7c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:11.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.304+0000 7faa81bf8700 1 -- 172.21.15.187:0/137095728 shutdown_connections 2020-11-20T12:18:11.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.304+0000 7faa81bf8700 1 -- 172.21.15.187:0/137095728 wait complete. 2020-11-20T12:18:11.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.306+0000 7faa81bf8700 1 Processor -- start 2020-11-20T12:18:11.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.306+0000 7faa81bf8700 1 -- start start 2020-11-20T12:18:11.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa81bf8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa7c066f70 0x7faa7c142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa81bf8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c1484d0 0x7faa7c142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa81bf8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa7c09f5e0 0x7faa7c09dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa81bf8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c143320 con 0x7faa7c066f70 2020-11-20T12:18:11.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa81bf8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c0a0040 con 0x7faa7c09f5e0 2020-11-20T12:18:11.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa81bf8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c09ea90 con 0x7faa7c1484d0 2020-11-20T12:18:11.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa813f7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa7c09f5e0 0x7faa7c09dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa813f7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa7c09f5e0 0x7faa7c09dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51780/0 (socket says 172.21.15.187:51780) 2020-11-20T12:18:11.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa813f7700 1 -- 172.21.15.187:0/2086059854 learned_addr learned my addr 172.21.15.187:0/2086059854 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:11.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.307+0000 7faa7bfff700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c1484d0 0x7faa7c142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa80bf6700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa7c066f70 0x7faa7c142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa813f7700 1 -- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c1484d0 msgr2=0x7faa7c142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa813f7700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c1484d0 0x7faa7c142e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa813f7700 1 -- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa7c066f70 msgr2=0x7faa7c142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa813f7700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa7c066f70 0x7faa7c142940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa813f7700 1 -- 172.21.15.187:0/2086059854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c09ed30 con 0x7faa7c09f5e0 2020-11-20T12:18:11.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa80bf6700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa7c066f70 0x7faa7c142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:11.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa813f7700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa7c09f5e0 0x7faa7c09dca0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 rx=0x7faa7400cab0 tx=0x7faa740023b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa78ff9700 1 -- 172.21.15.187:0/2086059854 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7400a830 con 0x7faa7c09f5e0 2020-11-20T12:18:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa78ff9700 1 -- 172.21.15.187:0/2086059854 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7400a990 con 0x7faa7c09f5e0 2020-11-20T12:18:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c09efd0 con 0x7faa7c09f5e0 2020-11-20T12:18:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa78ff9700 1 -- 172.21.15.187:0/2086059854 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7400ad30 con 0x7faa7c09f5e0 2020-11-20T12:18:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.308+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c09e480 con 0x7faa7c09f5e0 2020-11-20T12:18:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.309+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c061a20 con 0x7faa7c09f5e0 2020-11-20T12:18:11.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.310+0000 7faa78ff9700 1 -- 172.21.15.187:0/2086059854 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faa74019030 con 0x7faa7c09f5e0 2020-11-20T12:18:11.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.311+0000 7faa78ff9700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa54056d20 0x7faa540591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.311+0000 7faa78ff9700 1 -- 172.21.15.187:0/2086059854 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faa7406f800 con 0x7faa7c09f5e0 2020-11-20T12:18:11.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.311+0000 7faa80bf6700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa54056d20 0x7faa540591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.312+0000 7faa80bf6700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa54056d20 0x7faa540591e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7faa70006bd0 tx=0x7faa70008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:11.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.314+0000 7faa78ff9700 1 -- 172.21.15.187:0/2086059854 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faa74043fd0 con 0x7faa7c09f5e0 2020-11-20T12:18:11.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.501+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa7c1488b0 con 0x7faa7c09f5e0 2020-11-20T12:18:11.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.503+0000 7faa78ff9700 1 -- 172.21.15.187:0/2086059854 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16805 (secure 0 0 0) 0x7faa740251c0 con 0x7faa7c09f5e0 2020-11-20T12:18:11.506 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:11.507 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":129,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:18:10.556875+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:18:08.692656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2020-11-20T12:18:09.325174+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:18:11.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.505+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa54056d20 msgr2=0x7faa540591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.505+0000 7faa81bf8700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa54056d20 0x7faa540591e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7faa70006bd0 tx=0x7faa70008040).stop 2020-11-20T12:18:11.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.505+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa7c09f5e0 msgr2=0x7faa7c09dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.505+0000 7faa81bf8700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa7c09f5e0 0x7faa7c09dca0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 rx=0x7faa7400cab0 tx=0x7faa740023b0).stop 2020-11-20T12:18:11.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.506+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 shutdown_connections 2020-11-20T12:18:11.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.506+0000 7faa81bf8700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa54056d20 0x7faa540591e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.506+0000 7faa81bf8700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa7c066f70 0x7faa7c142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.506+0000 7faa81bf8700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa7c09f5e0 0x7faa7c09dca0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.506+0000 7faa81bf8700 1 --2- 172.21.15.187:0/2086059854 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faa7c1484d0 0x7faa7c142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.506+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 >> 172.21.15.187:0/2086059854 conn(0x7faa7c0bab30 msgr2=0x7faa7c0b9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:11.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.506+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 shutdown_connections 2020-11-20T12:18:11.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.506+0000 7faa81bf8700 1 -- 172.21.15.187:0/2086059854 wait complete. 2020-11-20T12:18:11.530 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:18:11.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.767+0000 7f57dadb8700 1 Processor -- start 2020-11-20T12:18:11.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.767+0000 7f57dadb8700 1 -- start start 2020-11-20T12:18:11.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.768+0000 7f57dadb8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f57d40bf800 0x7f57d414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.768+0000 7f57dadb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d409f560 0x7f57d40a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.768+0000 7f57dadb8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d414d110 con 0x7f57d40bfbe0 2020-11-20T12:18:11.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.768+0000 7f57dadb8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d414cbd0 con 0x7f57d40bf800 2020-11-20T12:18:11.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.768+0000 7f57dadb8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d414ce70 con 0x7f57d409f560 2020-11-20T12:18:11.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.769+0000 7f57d9db6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f57d40bf800 0x7f57d414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.769+0000 7f57d9db6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f57d40bf800 0x7f57d414c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51784/0 (socket says 172.21.15.187:51784) 2020-11-20T12:18:11.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.769+0000 7f57d9db6700 1 -- 172.21.15.187:0/66067113 learned_addr learned my addr 172.21.15.187:0/66067113 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:11.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.769+0000 7f57da5b7700 1 --2- 172.21.15.187:0/66067113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d409f560 0x7f57d40a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.769+0000 7f57d9db6700 1 -- 172.21.15.187:0/66067113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d409f560 msgr2=0x7f57d40a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:11.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.769+0000 7f57d9db6700 1 --2- 172.21.15.187:0/66067113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d409f560 0x7f57d40a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.769+0000 7f57d9db6700 1 -- 172.21.15.187:0/66067113 >> v1:172.21.15.5:6789/0 conn(0x7f57d40bfbe0 legacy=0x7f57d41480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:11.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.769+0000 7f57d9db6700 1 -- 172.21.15.187:0/66067113 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d40a4cc0 con 0x7f57d40bf800 2020-11-20T12:18:11.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.770+0000 7f57d9db6700 1 --2- 172.21.15.187:0/66067113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40bf800 0x7f57d414c2c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 rx=0x7f57d0006fd0 tx=0x7f57d0006eb0).ready entity=mon.1 client_cookie=d6533001059fcf9f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:11.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57d8db4700 1 -- 172.21.15.187:0/66067113 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0009920 con 0x7f57d40bf800 2020-11-20T12:18:11.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57d8db4700 1 -- 172.21.15.187:0/66067113 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d0009a80 con 0x7f57d40bf800 2020-11-20T12:18:11.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57d8db4700 1 -- 172.21.15.187:0/66067113 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0009e00 con 0x7f57d40bf800 2020-11-20T12:18:11.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57dadb8700 1 -- 172.21.15.187:0/66067113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40bf800 msgr2=0x7f57d414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57dadb8700 1 --2- 172.21.15.187:0/66067113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40bf800 0x7f57d414c2c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 rx=0x7f57d0006fd0 tx=0x7f57d0006eb0).stop 2020-11-20T12:18:11.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57dadb8700 1 -- 172.21.15.187:0/66067113 shutdown_connections 2020-11-20T12:18:11.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57dadb8700 1 --2- 172.21.15.187:0/66067113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d409f560 0x7f57d40a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57dadb8700 1 --2- 172.21.15.187:0/66067113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d40bf800 0x7f57d414c2c0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:11.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57dadb8700 1 -- 172.21.15.187:0/66067113 >> 172.21.15.187:0/66067113 conn(0x7f57d4068370 msgr2=0x7f57d40bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:11.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.771+0000 7f57dadb8700 1 -- 172.21.15.187:0/66067113 shutdown_connections 2020-11-20T12:18:11.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.772+0000 7f57dadb8700 1 -- 172.21.15.187:0/66067113 wait complete. 2020-11-20T12:18:11.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.773+0000 7f57dadb8700 1 Processor -- start 2020-11-20T12:18:11.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.773+0000 7f57dadb8700 1 -- start start 2020-11-20T12:18:11.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57dadb8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d409f560 0x7f57d41413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57dadb8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40bf800 0x7f57d415bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57dadb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40ba580 0x7f57d415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57dadb8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d415c6e0 con 0x7f57d40bf800 2020-11-20T12:18:11.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57dadb8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d415c980 con 0x7f57d409f560 2020-11-20T12:18:11.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57dadb8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d4143d40 con 0x7f57d40ba580 2020-11-20T12:18:11.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57d9db6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d409f560 0x7f57d41413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57d9db6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d409f560 0x7f57d41413d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51790/0 (socket says 172.21.15.187:51790) 2020-11-20T12:18:11.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.774+0000 7f57d9db6700 1 -- 172.21.15.187:0/1489602088 learned_addr learned my addr 172.21.15.187:0/1489602088 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:11.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57d9db6700 1 -- 172.21.15.187:0/1489602088 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40ba580 msgr2=0x7f57d415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57d9db6700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40ba580 0x7f57d415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:11.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57d9db6700 1 -- 172.21.15.187:0/1489602088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40bf800 msgr2=0x7f57d415bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:11.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57d9db6700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40bf800 0x7f57d415bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:11.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57d9db6700 1 -- 172.21.15.187:0/1489602088 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d41418c0 con 0x7f57d409f560 2020-11-20T12:18:11.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57d9db6700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d409f560 0x7f57d41413d0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7f57d0006ef0 tx=0x7f57d0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:11.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57c1ffb700 1 -- 172.21.15.187:0/1489602088 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d001d5d0 con 0x7f57d409f560 2020-11-20T12:18:11.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57c1ffb700 1 -- 172.21.15.187:0/1489602088 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d001d730 con 0x7f57d409f560 2020-11-20T12:18:11.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d40b49f0 con 0x7f57d409f560 2020-11-20T12:18:11.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57c1ffb700 1 -- 172.21.15.187:0/1489602088 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d001daf0 con 0x7f57d409f560 2020-11-20T12:18:11.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.775+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d4145fc0 con 0x7f57d409f560 2020-11-20T12:18:11.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.777+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d4080050 con 0x7f57d409f560 2020-11-20T12:18:11.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.777+0000 7f57c1ffb700 1 -- 172.21.15.187:0/1489602088 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f57d0025030 con 0x7f57d409f560 2020-11-20T12:18:11.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.777+0000 7f57c1ffb700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57ac056d50 0x7f57ac059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:11.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.778+0000 7f57c1ffb700 1 -- 172.21.15.187:0/1489602088 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f57d00719c0 con 0x7f57d409f560 2020-11-20T12:18:11.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.778+0000 7f57d95b5700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57ac056d50 0x7f57ac059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:11.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.778+0000 7f57d95b5700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57ac056d50 0x7f57ac059210 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f57c4006fd0 tx=0x7f57c4006930).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:11.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:11.782+0000 7f57c1ffb700 1 -- 172.21.15.187:0/1489602088 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f57d0046d00 con 0x7f57d409f560 2020-11-20T12:18:12.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.040+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f57d415c510 con 0x7f57d409f560 2020-11-20T12:18:12.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.783+0000 7f57c1ffb700 1 -- 172.21.15.187:0/1489602088 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v115) v1 ==== 118+0+0 (secure 0 0 0) 0x7f57d0023070 con 0x7f57d409f560 2020-11-20T12:18:12.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57ac056d50 msgr2=0x7f57ac059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:12.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57ac056d50 0x7f57ac059210 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f57c4006fd0 tx=0x7f57c4006930).stop 2020-11-20T12:18:12.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d409f560 msgr2=0x7f57d41413d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:12.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d409f560 0x7f57d41413d0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7f57d0006ef0 tx=0x7f57d0015000).stop 2020-11-20T12:18:12.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 shutdown_connections 2020-11-20T12:18:12.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57ac056d50 0x7f57ac059210 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:12.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57d40ba580 0x7f57d415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:12.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57d409f560 0x7f57d41413d0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:12.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.786+0000 7f57dadb8700 1 --2- 172.21.15.187:0/1489602088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57d40bf800 0x7f57d415bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:12.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.787+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 >> 172.21.15.187:0/1489602088 conn(0x7f57d4068370 msgr2=0x7f57d40a4010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:12.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.787+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 shutdown_connections 2020-11-20T12:18:12.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:12.787+0000 7f57dadb8700 1 -- 172.21.15.187:0/1489602088 wait complete. 2020-11-20T12:18:12.817 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:13.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.054+0000 7f8535cd0700 1 Processor -- start 2020-11-20T12:18:13.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.054+0000 7f8535cd0700 1 -- start start 2020-11-20T12:18:13.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.056+0000 7f8535cd0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f85300b57b0 0x7f8530068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.056+0000 7f8535cd0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 0x7f8530062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.056+0000 7f8535cd0700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8530144b80 con 0x7f8530062b50 2020-11-20T12:18:13.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.056+0000 7f8535cd0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85301489e0 con 0x7f85300b57b0 2020-11-20T12:18:13.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.056+0000 7f8535cd0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8530148c80 con 0x7f85301482d0 2020-11-20T12:18:13.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.056+0000 7f85354cf700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8530062b50 0x7f8530065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48406/0 (socket says 172.21.15.187:48406) 2020-11-20T12:18:13.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.056+0000 7f85354cf700 1 -- 172.21.15.187:0/1036520685 learned_addr learned my addr 172.21.15.187:0/1036520685 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:13.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f8534cce700 1 --2- 172.21.15.187:0/1036520685 >> v2:172.21.15.5:3300/0 conn(0x7f85300b57b0 0x7f8530068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852ffff700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 0x7f8530062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039001198 0 0) 0x7f8530144b80 con 0x7f8530062b50 2020-11-20T12:18:13.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8514007650 con 0x7f8530062b50 2020-11-20T12:18:13.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 675153171 0 0) 0x7f8514007650 con 0x7f8530062b50 2020-11-20T12:18:13.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 msgr2=0x7f8530062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:13.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 0x7f8530062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 >> v2:172.21.15.5:3300/0 conn(0x7f85300b57b0 msgr2=0x7f8530068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:13.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 --2- 172.21.15.187:0/1036520685 >> v2:172.21.15.5:3300/0 conn(0x7f85300b57b0 0x7f8530068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530148f20 con 0x7f8530062b50 2020-11-20T12:18:13.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8528001860 con 0x7f8530062b50 2020-11-20T12:18:13.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852ffff700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 0x7f8530062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:13.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 >> v1:172.21.15.187:6789/0 conn(0x7f8530062b50 legacy=0x7f8530065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f8514007fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 0x7f85140088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8514008de0 0x7f8514009260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85301489e0 con 0x7f8514008de0 2020-11-20T12:18:13.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8530148c80 con 0x7f8514007bc0 2020-11-20T12:18:13.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.057+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8514008670 con 0x7f85301482d0 2020-11-20T12:18:13.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f8534cce700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f8514007fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f852ffff700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 0x7f85140088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f85354cf700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8514008de0 0x7f8514009260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f8534cce700 1 -- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 msgr2=0x7f85140088f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:13.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f8534cce700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85301482d0 0x7f85140088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f8534cce700 1 -- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8514008de0 msgr2=0x7f8514009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:13.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f8534cce700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8514008de0 0x7f8514009260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f8534cce700 1 -- 172.21.15.187:0/1036520685 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8514007650 con 0x7f8514007bc0 2020-11-20T12:18:13.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.058+0000 7f85354cf700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8514008de0 0x7f8514009260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:13.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.059+0000 7f8534cce700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f8514007fe0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 rx=0x7f852400bc00 tx=0x7f8524006df0).ready entity=mon.1 client_cookie=5f6079817011cc5d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:13.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.059+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852400b310 con 0x7f8514007bc0 2020-11-20T12:18:13.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.059+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852400eb40 con 0x7f8514007bc0 2020-11-20T12:18:13.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.059+0000 7f852f7fe700 1 -- 172.21.15.187:0/1036520685 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852400eec0 con 0x7f8514007bc0 2020-11-20T12:18:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.060+0000 7f8535cd0700 1 -- 172.21.15.187:0/1036520685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 msgr2=0x7f8514007fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.060+0000 7f8535cd0700 1 --2- 172.21.15.187:0/1036520685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f8514007fe0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 rx=0x7f852400bc00 tx=0x7f8524006df0).stop 2020-11-20T12:18:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.060+0000 7f85354cf700 1 -- 172.21.15.187:0/1036520685 reap_dead start 2020-11-20T12:18:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.060+0000 7f8535cd0700 1 -- 172.21.15.187:0/1036520685 shutdown_connections 2020-11-20T12:18:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.060+0000 7f8535cd0700 1 -- 172.21.15.187:0/1036520685 >> 172.21.15.187:0/1036520685 conn(0x7f85300bab30 msgr2=0x7f85300baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:13.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.060+0000 7f8535cd0700 1 -- 172.21.15.187:0/1036520685 shutdown_connections 2020-11-20T12:18:13.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.060+0000 7f8535cd0700 1 -- 172.21.15.187:0/1036520685 wait complete. 2020-11-20T12:18:13.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.062+0000 7f8535cd0700 1 Processor -- start 2020-11-20T12:18:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.062+0000 7f8535cd0700 1 -- start start 2020-11-20T12:18:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8535cd0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f85300b2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8535cd0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853009deb0 0x7f8530154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8535cd0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8530147080 0x7f85301535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8535cd0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8530154f60 con 0x7f853009deb0 2020-11-20T12:18:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8535cd0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85300b1b60 con 0x7f8514007bc0 2020-11-20T12:18:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8535cd0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85300b1e00 con 0x7f8530147080 2020-11-20T12:18:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8534cce700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f85300b2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8534cce700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f85300b2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51810/0 (socket says 172.21.15.187:51810) 2020-11-20T12:18:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f8534cce700 1 -- 172.21.15.187:0/3307869773 learned_addr learned my addr 172.21.15.187:0/3307869773 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.064+0000 7f852ffff700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853009deb0 0x7f8530154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f8534cce700 1 -- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8530147080 msgr2=0x7f85301535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f85354cf700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8530147080 0x7f85301535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f8534cce700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8530147080 0x7f85301535b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f8534cce700 1 -- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853009deb0 msgr2=0x7f8530154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f8534cce700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853009deb0 0x7f8530154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f8534cce700 1 -- 172.21.15.187:0/3307869773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85300b20a0 con 0x7f8514007bc0 2020-11-20T12:18:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f852ffff700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853009deb0 0x7f8530154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:13.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f8534cce700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f85300b2540 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7f8514009890 tx=0x7f8524019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:13.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f852cff9700 1 -- 172.21.15.187:0/3307869773 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852401e630 con 0x7f8514007bc0 2020-11-20T12:18:13.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f852cff9700 1 -- 172.21.15.187:0/3307869773 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852400a070 con 0x7f8514007bc0 2020-11-20T12:18:13.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8530147500 con 0x7f8514007bc0 2020-11-20T12:18:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.065+0000 7f852cff9700 1 -- 172.21.15.187:0/3307869773 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852401e7b0 con 0x7f8514007bc0 2020-11-20T12:18:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.066+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8530153ce0 con 0x7f8514007bc0 2020-11-20T12:18:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.067+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85300b5ec0 con 0x7f8514007bc0 2020-11-20T12:18:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.067+0000 7f852cff9700 1 -- 172.21.15.187:0/3307869773 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f852401eb60 con 0x7f8514007bc0 2020-11-20T12:18:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.068+0000 7f852cff9700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8508056d20 0x7f85080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.068+0000 7f852cff9700 1 -- 172.21.15.187:0/3307869773 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8524072560 con 0x7f8514007bc0 2020-11-20T12:18:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.068+0000 7f852ffff700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8508056d20 0x7f85080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.069+0000 7f852ffff700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8508056d20 0x7f85080591e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f851c006c50 tx=0x7f851c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.072+0000 7f852cff9700 1 -- 172.21.15.187:0/3307869773 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8524047950 con 0x7f8514007bc0 2020-11-20T12:18:13.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.312+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8530153ed0 con 0x7f8514007bc0 2020-11-20T12:18:13.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.313+0000 7f852cff9700 1 -- 172.21.15.187:0/3307869773 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+2142 (secure 0 0 0) 0x7f852404af70 con 0x7f8514007bc0 2020-11-20T12:18:13.316 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:13.316 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":115,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5900,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3549193191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3549193191},{"type":"v1","addr":"172.21.15.5:6833","nonce":3549193191}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":111}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:12.767651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:13.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8508056d20 msgr2=0x7f85080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8508056d20 0x7f85080591e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f851c006c50 tx=0x7f851c008040).stop 2020-11-20T12:18:13.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 msgr2=0x7f85300b2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f85300b2540 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7f8514009890 tx=0x7f8524019000).stop 2020-11-20T12:18:13.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 shutdown_connections 2020-11-20T12:18:13.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8508056d20 0x7f85080591e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8530147080 0x7f85301535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8514007bc0 0x7f85300b2540 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 --2- 172.21.15.187:0/3307869773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853009deb0 0x7f8530154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 >> 172.21.15.187:0/3307869773 conn(0x7f85300bab30 msgr2=0x7f85300afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:13.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 shutdown_connections 2020-11-20T12:18:13.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.315+0000 7f8535cd0700 1 -- 172.21.15.187:0/3307869773 wait complete. 2020-11-20T12:18:13.323 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 115 2020-11-20T12:18:13.340 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:18:13.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.556+0000 7f3660ef6700 1 Processor -- start 2020-11-20T12:18:13.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.556+0000 7f3660ef6700 1 -- start start 2020-11-20T12:18:13.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f3660ef6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c066f70 0x7f365c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f3660ef6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c0b1300 con 0x7f365c1484d0 2020-11-20T12:18:13.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f3660ef6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c0b15a0 con 0x7f365c0b6650 2020-11-20T12:18:13.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f3660ef6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365c067730 con 0x7f365c066f70 2020-11-20T12:18:13.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f3652ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f365c0b6650 0x7f365c0afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48426/0 (socket says 172.21.15.187:48426) 2020-11-20T12:18:13.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f3652ffd700 1 -- 172.21.15.187:0/1291226952 learned_addr learned my addr 172.21.15.187:0/1291226952 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:13.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331880682 0 0) 0x7f365c0b15a0 con 0x7f365c0b6650 2020-11-20T12:18:13.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c003240 con 0x7f365c0b6650 2020-11-20T12:18:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.558+0000 7f365bfff700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c066f70 0x7f365c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1353005388 0 0) 0x7f363c003240 con 0x7f365c0b6650 2020-11-20T12:18:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c066f70 msgr2=0x7f365c0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c066f70 0x7f365c0b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 >> v1:172.21.15.5:6789/0 conn(0x7f365c1484d0 legacy=0x7f365c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c0b1840 con 0x7f365c0b6650 2020-11-20T12:18:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f3644001520 con 0x7f365c0b6650 2020-11-20T12:18:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 >> v1:172.21.15.187:6789/0 conn(0x7f365c0b6650 legacy=0x7f365c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c066f70 0x7f363c003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f363c003c90 0x7f363c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f363c0049d0 0x7f363c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c067730 con 0x7f363c0049d0 2020-11-20T12:18:13.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c004220 con 0x7f363c003c90 2020-11-20T12:18:13.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.559+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c00d940 con 0x7f365c066f70 2020-11-20T12:18:13.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.560+0000 7f3652ffd700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f363c003c90 0x7f363c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.560+0000 7f3652ffd700 1 -- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c066f70 msgr2=0x7f363c003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:13.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.560+0000 7f3652ffd700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c066f70 0x7f363c003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:13.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.560+0000 7f3652ffd700 1 -- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f363c0049d0 msgr2=0x7f363c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:13.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.560+0000 7f3652ffd700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f363c0049d0 0x7f363c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:13.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.560+0000 7f3652ffd700 1 -- 172.21.15.187:0/1291226952 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c003240 con 0x7f363c003c90 2020-11-20T12:18:13.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.560+0000 7f3652ffd700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f363c003c90 0x7f363c0044e0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 rx=0x7f364400c210 tx=0x7f3644007750).ready entity=mon.1 client_cookie=f887010dec94491c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:13.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36440070d0 con 0x7f363c003c90 2020-11-20T12:18:13.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3644007230 con 0x7f363c003c90 2020-11-20T12:18:13.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f365affd700 1 -- 172.21.15.187:0/1291226952 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36440055f0 con 0x7f363c003c90 2020-11-20T12:18:13.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f3660ef6700 1 -- 172.21.15.187:0/1291226952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f363c003c90 msgr2=0x7f363c0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f3660ef6700 1 --2- 172.21.15.187:0/1291226952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f363c003c90 0x7f363c0044e0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 rx=0x7f364400c210 tx=0x7f3644007750).stop 2020-11-20T12:18:13.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f365bfff700 1 -- 172.21.15.187:0/1291226952 reap_dead start 2020-11-20T12:18:13.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f3660ef6700 1 -- 172.21.15.187:0/1291226952 shutdown_connections 2020-11-20T12:18:13.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f3660ef6700 1 -- 172.21.15.187:0/1291226952 >> 172.21.15.187:0/1291226952 conn(0x7f365c0bab30 msgr2=0x7f365c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:13.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f3660ef6700 1 -- 172.21.15.187:0/1291226952 shutdown_connections 2020-11-20T12:18:13.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.561+0000 7f3660ef6700 1 -- 172.21.15.187:0/1291226952 wait complete. 2020-11-20T12:18:13.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.563+0000 7f3660ef6700 1 Processor -- start 2020-11-20T12:18:13.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.563+0000 7f3660ef6700 1 -- start start 2020-11-20T12:18:13.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f3660ef6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c09cd10 0x7f365c141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f3660ef6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f365c09d0f0 0x7f365c1423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f3660ef6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f365c09dae0 0x7f365c0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f3660ef6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c09e960 con 0x7f365c09d0f0 2020-11-20T12:18:13.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f3660ef6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c09df80 con 0x7f365c09dae0 2020-11-20T12:18:13.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f3660ef6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365c09e220 con 0x7f365c09cd10 2020-11-20T12:18:13.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f365bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f365c09dae0 0x7f365c0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f365bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f365c09dae0 0x7f365c0b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51834/0 (socket says 172.21.15.187:51834) 2020-11-20T12:18:13.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.564+0000 7f365bfff700 1 -- 172.21.15.187:0/3776974648 learned_addr learned my addr 172.21.15.187:0/3776974648 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:13.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f3652ffd700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f365c09d0f0 0x7f365c1423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f365b7fe700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c09cd10 0x7f365c141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f365bfff700 1 -- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c09cd10 msgr2=0x7f365c141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f365bfff700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c09cd10 0x7f365c141ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f365bfff700 1 -- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f365c09d0f0 msgr2=0x7f365c1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f365bfff700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f365c09d0f0 0x7f365c1423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f365bfff700 1 -- 172.21.15.187:0/3776974648 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c09e4c0 con 0x7f365c09dae0 2020-11-20T12:18:13.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f3652ffd700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f365c09d0f0 0x7f365c1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:13.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f365b7fe700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c09cd10 0x7f365c141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:13.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f365bfff700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f365c09dae0 0x7f365c0b1a20 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 rx=0x7f365c099f10 tx=0x7f3654008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:13.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f3653fff700 1 -- 172.21.15.187:0/3776974648 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365400c4f0 con 0x7f365c09dae0 2020-11-20T12:18:13.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365c1428a0 con 0x7f365c09dae0 2020-11-20T12:18:13.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f3653fff700 1 -- 172.21.15.187:0/3776974648 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365400c650 con 0x7f365c09dae0 2020-11-20T12:18:13.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.565+0000 7f3653fff700 1 -- 172.21.15.187:0/3776974648 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365400c9f0 con 0x7f365c09dae0 2020-11-20T12:18:13.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.566+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c157270 con 0x7f365c09dae0 2020-11-20T12:18:13.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.567+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365c061a20 con 0x7f365c09dae0 2020-11-20T12:18:13.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.567+0000 7f3653fff700 1 -- 172.21.15.187:0/3776974648 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f365401b030 con 0x7f365c09dae0 2020-11-20T12:18:13.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.568+0000 7f3653fff700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3638056d50 0x7f3638059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:13.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.568+0000 7f3653fff700 1 -- 172.21.15.187:0/3776974648 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f365406f570 con 0x7f365c09dae0 2020-11-20T12:18:13.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.568+0000 7f365b7fe700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3638056d50 0x7f3638059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:13.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.569+0000 7f365b7fe700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3638056d50 0x7f3638059210 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f364c000f20 tx=0x7f364c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:13.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.572+0000 7f3653fff700 1 -- 172.21.15.187:0/3776974648 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f36540449c0 con 0x7f365c09dae0 2020-11-20T12:18:13.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.754+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f365c0624e0 con 0x7f365c09dae0 2020-11-20T12:18:13.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.757+0000 7f3653fff700 1 -- 172.21.15.187:0/3776974648 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f3654047fe0 con 0x7f365c09dae0 2020-11-20T12:18:13.760 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:13.761 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":130,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:18:11.562810+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:18:08.692656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2020-11-20T12:18:09.325174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:18:13.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3638056d50 msgr2=0x7f3638059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3638056d50 0x7f3638059210 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f364c000f20 tx=0x7f364c009040).stop 2020-11-20T12:18:13.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f365c09dae0 msgr2=0x7f365c0b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:13.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f365c09dae0 0x7f365c0b1a20 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 rx=0x7f365c099f10 tx=0x7f3654008040).stop 2020-11-20T12:18:13.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 shutdown_connections 2020-11-20T12:18:13.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3638056d50 0x7f3638059210 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f365c09dae0 0x7f365c0b1a20 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f365c09cd10 0x7f365c141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 --2- 172.21.15.187:0/3776974648 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f365c09d0f0 0x7f365c1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:13.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 >> 172.21.15.187:0/3776974648 conn(0x7f365c0bab30 msgr2=0x7f365c142c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:13.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 shutdown_connections 2020-11-20T12:18:13.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:13.760+0000 7f3660ef6700 1 -- 172.21.15.187:0/3776974648 wait complete. 2020-11-20T12:18:13.788 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:18:14.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.023+0000 7fecd4adf700 1 Processor -- start 2020-11-20T12:18:14.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.024+0000 7fecd4adf700 1 -- start start 2020-11-20T12:18:14.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.025+0000 7fecd4adf700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fecd00b1760 0x7fecd0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.025+0000 7fecd4adf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd0065680 0x7fecd014c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.025+0000 7fecd4adf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd0150390 con 0x7fecd00b1380 2020-11-20T12:18:14.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.025+0000 7fecd4adf700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd01511e0 con 0x7fecd00b1760 2020-11-20T12:18:14.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.025+0000 7fecd4adf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd014d760 con 0x7fecd0065680 2020-11-20T12:18:14.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.026+0000 7feccffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd0065680 0x7fecd014c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.026+0000 7fecceffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fecd00b1760 0x7fecd0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.026+0000 7feccffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd0065680 0x7fecd014c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:52730/0 (socket says 172.21.15.187:52730) 2020-11-20T12:18:14.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.026+0000 7fecceffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fecd00b1760 0x7fecd0065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:44104/0 (socket says 172.21.15.187:44104) 2020-11-20T12:18:14.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.026+0000 7feccffff700 1 -- 172.21.15.187:0/51208902 learned_addr learned my addr 172.21.15.187:0/51208902 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:14.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.027+0000 7fecceffd700 1 -- 172.21.15.187:0/51208902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd0065680 msgr2=0x7fecd014c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:14.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.027+0000 7fecceffd700 1 --2- 172.21.15.187:0/51208902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd0065680 0x7fecd014c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.027+0000 7fecceffd700 1 -- 172.21.15.187:0/51208902 >> v1:172.21.15.5:6789/0 conn(0x7fecd00b1380 legacy=0x7fecd0150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.027+0000 7fecceffd700 1 -- 172.21.15.187:0/51208902 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd0150630 con 0x7fecd00b1760 2020-11-20T12:18:14.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.028+0000 7fecceffd700 1 --2- 172.21.15.187:0/51208902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1760 0x7fecd0065190 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fecb800b9d0 tx=0x7fecb8006df0).ready entity=mon.0 client_cookie=f9b31d8655e62ea5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:14.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.028+0000 7fecce7fc700 1 -- 172.21.15.187:0/51208902 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8009770 con 0x7fecd00b1760 2020-11-20T12:18:14.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.028+0000 7fecce7fc700 1 -- 172.21.15.187:0/51208902 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb80098d0 con 0x7fecd00b1760 2020-11-20T12:18:14.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.028+0000 7fecce7fc700 1 -- 172.21.15.187:0/51208902 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8009c50 con 0x7fecd00b1760 2020-11-20T12:18:14.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.029+0000 7fecd4adf700 1 -- 172.21.15.187:0/51208902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1760 msgr2=0x7fecd0065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.029+0000 7fecd4adf700 1 --2- 172.21.15.187:0/51208902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1760 0x7fecd0065190 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fecb800b9d0 tx=0x7fecb8006df0).stop 2020-11-20T12:18:14.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.029+0000 7fecd4adf700 1 -- 172.21.15.187:0/51208902 shutdown_connections 2020-11-20T12:18:14.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.029+0000 7fecd4adf700 1 --2- 172.21.15.187:0/51208902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd0065680 0x7fecd014c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.029+0000 7fecd4adf700 1 --2- 172.21.15.187:0/51208902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1760 0x7fecd0065190 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.029+0000 7fecd4adf700 1 -- 172.21.15.187:0/51208902 >> 172.21.15.187:0/51208902 conn(0x7fecd00bab30 msgr2=0x7fecd00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:14.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.029+0000 7fecd4adf700 1 -- 172.21.15.187:0/51208902 shutdown_connections 2020-11-20T12:18:14.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.029+0000 7fecd4adf700 1 -- 172.21.15.187:0/51208902 wait complete. 2020-11-20T12:18:14.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.031+0000 7fecd4adf700 1 Processor -- start 2020-11-20T12:18:14.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.031+0000 7fecd4adf700 1 -- start start 2020-11-20T12:18:14.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.032+0000 7fecd4adf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1380 0x7fecd00ba1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.032+0000 7fecd4adf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd00b1760 0x7fecd00b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.032+0000 7fecd4adf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd00b3fe0 0x7fecd009ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.032+0000 7fecd4adf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd00b4710 con 0x7fecd00b1380 2020-11-20T12:18:14.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.032+0000 7fecd4adf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd00b4470 con 0x7fecd00b1760 2020-11-20T12:18:14.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.032+0000 7fecd4adf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015fb60 con 0x7fecd00b3fe0 2020-11-20T12:18:14.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7fecceffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd00b1760 0x7fecd00b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7fecceffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd00b1760 0x7fecd00b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51844/0 (socket says 172.21.15.187:51844) 2020-11-20T12:18:14.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7fecceffd700 1 -- 172.21.15.187:0/727805101 learned_addr learned my addr 172.21.15.187:0/727805101 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:14.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7feccffff700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd00b3fe0 0x7fecd009ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7feccf7fe700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1380 0x7fecd00ba1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7fecceffd700 1 -- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd00b3fe0 msgr2=0x7fecd009ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7fecceffd700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd00b3fe0 0x7fecd009ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7fecceffd700 1 -- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1380 msgr2=0x7fecd00ba1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7fecceffd700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1380 0x7fecd00ba1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7fecceffd700 1 -- 172.21.15.187:0/727805101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd009e6b0 con 0x7fecd00b1760 2020-11-20T12:18:14.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.033+0000 7feccf7fe700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1380 0x7fecd00ba1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:14.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.034+0000 7fecceffd700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd00b1760 0x7fecd00b3af0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 rx=0x7fecb800b920 tx=0x7fecb8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:14.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.034+0000 7fecc77fe700 1 -- 172.21.15.187:0/727805101 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8011dd0 con 0x7fecd00b1760 2020-11-20T12:18:14.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.034+0000 7fecc77fe700 1 -- 172.21.15.187:0/727805101 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb801e460 con 0x7fecd00b1760 2020-11-20T12:18:14.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.034+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd00ba6e0 con 0x7fecd00b1760 2020-11-20T12:18:14.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.034+0000 7fecc77fe700 1 -- 172.21.15.187:0/727805101 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb801e800 con 0x7fecd00b1760 2020-11-20T12:18:14.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.034+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd009b430 con 0x7fecd00b1760 2020-11-20T12:18:14.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.035+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd015fd20 con 0x7fecd00b1760 2020-11-20T12:18:14.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.036+0000 7fecc77fe700 1 -- 172.21.15.187:0/727805101 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fecb8025030 con 0x7fecd00b1760 2020-11-20T12:18:14.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.036+0000 7fecc77fe700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecac056d20 0x7fecac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.037+0000 7fecc77fe700 1 -- 172.21.15.187:0/727805101 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fecb8071890 con 0x7fecd00b1760 2020-11-20T12:18:14.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.037+0000 7feccf7fe700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecac056d20 0x7fecac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.037+0000 7feccf7fe700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecac056d20 0x7fecac0591e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7fecc00014d0 tx=0x7fecc0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:14.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.040+0000 7fecc77fe700 1 -- 172.21.15.187:0/727805101 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fecb8046d60 con 0x7fecd00b1760 2020-11-20T12:18:14.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.292+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fecd0061a20 con 0x7fecd00b1760 2020-11-20T12:18:14.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.302+0000 7fecc77fe700 1 -- 172.21.15.187:0/727805101 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 184+0+0 (secure 0 0 0) 0x7fecb80230e0 con 0x7fecd00b1760 2020-11-20T12:18:14.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.304+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecac056d20 msgr2=0x7fecac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.304+0000 7fecd4adf700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecac056d20 0x7fecac0591e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7fecc00014d0 tx=0x7fecc0009040).stop 2020-11-20T12:18:14.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.304+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd00b1760 msgr2=0x7fecd00b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.304+0000 7fecd4adf700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd00b1760 0x7fecd00b3af0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 rx=0x7fecb800b920 tx=0x7fecb8015000).stop 2020-11-20T12:18:14.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.305+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 shutdown_connections 2020-11-20T12:18:14.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.305+0000 7fecd4adf700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fecac056d20 0x7fecac0591e0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.305+0000 7fecd4adf700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fecd00b1380 0x7fecd00ba1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.305+0000 7fecd4adf700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fecd00b3fe0 0x7fecd009ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.305+0000 7fecd4adf700 1 --2- 172.21.15.187:0/727805101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecd00b1760 0x7fecd00b3af0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.305+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 >> 172.21.15.187:0/727805101 conn(0x7fecd00bab30 msgr2=0x7fecd014c940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:14.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.305+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 shutdown_connections 2020-11-20T12:18:14.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.305+0000 7fecd4adf700 1 -- 172.21.15.187:0/727805101 wait complete. 2020-11-20T12:18:14.310 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:18:14.329 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:14.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.560+0000 7f83eb401700 1 Processor -- start 2020-11-20T12:18:14.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.561+0000 7f83eb401700 1 -- start start 2020-11-20T12:18:14.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.562+0000 7f83eb401700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e4066040 0x7f83e4148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.562+0000 7f83eb401700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e414c170 con 0x7f83e4145cc0 2020-11-20T12:18:14.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.562+0000 7f83eb401700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e414c410 con 0x7f83e40683f0 2020-11-20T12:18:14.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.562+0000 7f83eb401700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e414c6b0 con 0x7f83e4066040 2020-11-20T12:18:14.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.562+0000 7f83e9bfe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f83e40683f0 0x7f83e41487e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48460/0 (socket says 172.21.15.187:48460) 2020-11-20T12:18:14.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.562+0000 7f83e9bfe700 1 -- 172.21.15.187:0/821488753 learned_addr learned my addr 172.21.15.187:0/821488753 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:14.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.563+0000 7f83eac00700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e4066040 0x7f83e4148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.563+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013283541 0 0) 0x7f83e414c410 con 0x7f83e40683f0 2020-11-20T12:18:14.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.563+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83cc003240 con 0x7f83e40683f0 2020-11-20T12:18:14.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.563+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1418920755 0 0) 0x7f83cc003240 con 0x7f83e40683f0 2020-11-20T12:18:14.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e4066040 msgr2=0x7f83e4148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:14.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e4066040 0x7f83e4148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 >> v1:172.21.15.5:6789/0 conn(0x7f83e4145cc0 legacy=0x7f83e4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e40658a0 con 0x7f83e40683f0 2020-11-20T12:18:14.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f83d40014e0 con 0x7f83e40683f0 2020-11-20T12:18:14.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 >> v1:172.21.15.187:6789/0 conn(0x7f83e40683f0 legacy=0x7f83e41487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e4066040 0x7f83cc003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83cc003c40 0x7f83cc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83cc0049d0 0x7f83cc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e414c6b0 con 0x7f83cc0049d0 2020-11-20T12:18:14.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83cc0041d0 con 0x7f83cc003c40 2020-11-20T12:18:14.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83cc00b730 con 0x7f83e4066040 2020-11-20T12:18:14.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.564+0000 7f83e9bfe700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83cc003c40 0x7f83cc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83ea3ff700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e4066040 0x7f83cc003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83eac00700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83cc0049d0 0x7f83cc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83e9bfe700 1 -- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e4066040 msgr2=0x7f83cc003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:14.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83e9bfe700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e4066040 0x7f83cc003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83e9bfe700 1 -- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83cc0049d0 msgr2=0x7f83cc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:14.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83e9bfe700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83cc0049d0 0x7f83cc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83e9bfe700 1 -- 172.21.15.187:0/821488753 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc003240 con 0x7f83cc003c40 2020-11-20T12:18:14.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83e9bfe700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83cc003c40 0x7f83cc0044e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 rx=0x7f83d400c410 tx=0x7f83d40077d0).ready entity=mon.1 client_cookie=1167af199692fd60 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:14.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.565+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d400ce20 con 0x7f83cc003c40 2020-11-20T12:18:14.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d40072e0 con 0x7f83cc003c40 2020-11-20T12:18:14.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83e93fd700 1 -- 172.21.15.187:0/821488753 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4005800 con 0x7f83cc003c40 2020-11-20T12:18:14.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83eb401700 1 -- 172.21.15.187:0/821488753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83cc003c40 msgr2=0x7f83cc0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83eb401700 1 --2- 172.21.15.187:0/821488753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83cc003c40 0x7f83cc0044e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 rx=0x7f83d400c410 tx=0x7f83d40077d0).stop 2020-11-20T12:18:14.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83eac00700 1 -- 172.21.15.187:0/821488753 reap_dead start 2020-11-20T12:18:14.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83eb401700 1 -- 172.21.15.187:0/821488753 shutdown_connections 2020-11-20T12:18:14.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83eb401700 1 -- 172.21.15.187:0/821488753 >> 172.21.15.187:0/821488753 conn(0x7f83e40bab30 msgr2=0x7f83e40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:14.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83eb401700 1 -- 172.21.15.187:0/821488753 shutdown_connections 2020-11-20T12:18:14.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.566+0000 7f83eb401700 1 -- 172.21.15.187:0/821488753 wait complete. 2020-11-20T12:18:14.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.568+0000 7f83eb401700 1 Processor -- start 2020-11-20T12:18:14.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.568+0000 7f83eb401700 1 -- start start 2020-11-20T12:18:14.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83eb401700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83e40b3580 0x7f83e409c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83eb401700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e40b3960 0x7f83e4143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83eb401700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83e409eab0 0x7f83e4144200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83eb401700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40b19b0 con 0x7f83e409eab0 2020-11-20T12:18:14.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83eb401700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40b1c50 con 0x7f83e40b3580 2020-11-20T12:18:14.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83eb401700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40b1ef0 con 0x7f83e40b3960 2020-11-20T12:18:14.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83ea3ff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83e40b3580 0x7f83e409c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83ea3ff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83e40b3580 0x7f83e409c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51864/0 (socket says 172.21.15.187:51864) 2020-11-20T12:18:14.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83ea3ff700 1 -- 172.21.15.187:0/2436439919 learned_addr learned my addr 172.21.15.187:0/2436439919 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:14.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.569+0000 7f83e9bfe700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e40b3960 0x7f83e4143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83ea3ff700 1 -- 172.21.15.187:0/2436439919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e40b3960 msgr2=0x7f83e4143d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83ea3ff700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e40b3960 0x7f83e4143d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83ea3ff700 1 -- 172.21.15.187:0/2436439919 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83e409eab0 msgr2=0x7f83e4144200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83ea3ff700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83e409eab0 0x7f83e4144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:14.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83ea3ff700 1 -- 172.21.15.187:0/2436439919 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e40b3f40 con 0x7f83e40b3580 2020-11-20T12:18:14.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83e9bfe700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e40b3960 0x7f83e4143d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:14.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83ea3ff700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83e40b3580 0x7f83e409c190 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7f83e000e8e0 tx=0x7f83e0009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:14.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83d27fc700 1 -- 172.21.15.187:0/2436439919 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e000eee0 con 0x7f83e40b3580 2020-11-20T12:18:14.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83d27fc700 1 -- 172.21.15.187:0/2436439919 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e000c840 con 0x7f83e40b3580 2020-11-20T12:18:14.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83d27fc700 1 -- 172.21.15.187:0/2436439919 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e000cbc0 con 0x7f83e40b3580 2020-11-20T12:18:14.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e409c680 con 0x7f83e40b3580 2020-11-20T12:18:14.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.570+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e40b25d0 con 0x7f83e40b3580 2020-11-20T12:18:14.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.571+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e40b20b0 con 0x7f83e40b3580 2020-11-20T12:18:14.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.572+0000 7f83d27fc700 1 -- 172.21.15.187:0/2436439919 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f83e001b030 con 0x7f83e40b3580 2020-11-20T12:18:14.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.573+0000 7f83d27fc700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f83bc056d50 0x7f83bc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:14.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.573+0000 7f83d27fc700 1 -- 172.21.15.187:0/2436439919 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f83e006fbb0 con 0x7f83e40b3580 2020-11-20T12:18:14.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.573+0000 7f83e9bfe700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f83bc056d50 0x7f83bc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:14.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.573+0000 7f83e9bfe700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f83bc056d50 0x7f83bc059210 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f83e405f200 tx=0x7f83d4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:14.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.576+0000 7f83d27fc700 1 -- 172.21.15.187:0/2436439919 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f83e0076080 con 0x7f83e40b3580 2020-11-20T12:18:14.767 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:18:14.764+0000 7f357cee0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:18:14.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.803+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83e4061a20 con 0x7f83e40b3580 2020-11-20T12:18:14.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.804+0000 7f83d27fc700 1 -- 172.21.15.187:0/2436439919 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+2142 (secure 0 0 0) 0x7f83e00450f0 con 0x7f83e40b3580 2020-11-20T12:18:14.807 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:14.808 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":115,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5900,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3549193191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3549193191},{"type":"v1","addr":"172.21.15.5:6833","nonce":3549193191}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":111}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:12.767651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:14.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.806+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f83bc056d50 msgr2=0x7f83bc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.806+0000 7f83eb401700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f83bc056d50 0x7f83bc059210 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f83e405f200 tx=0x7f83d4015000).stop 2020-11-20T12:18:14.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.806+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83e40b3580 msgr2=0x7f83e409c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:14.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.806+0000 7f83eb401700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83e40b3580 0x7f83e409c190 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7f83e000e8e0 tx=0x7f83e0009040).stop 2020-11-20T12:18:14.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.806+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 shutdown_connections 2020-11-20T12:18:14.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.806+0000 7f83eb401700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f83bc056d50 0x7f83bc059210 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.807+0000 7f83eb401700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83e409eab0 0x7f83e4144200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:14.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.807+0000 7f83eb401700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83e40b3580 0x7f83e409c190 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.807+0000 7f83eb401700 1 --2- 172.21.15.187:0/2436439919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f83e40b3960 0x7f83e4143d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:14.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.807+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 >> 172.21.15.187:0/2436439919 conn(0x7f83e40bab30 msgr2=0x7f83e40b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:14.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.807+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 shutdown_connections 2020-11-20T12:18:14.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:14.807+0000 7f83eb401700 1 -- 172.21.15.187:0/2436439919 wait complete. 2020-11-20T12:18:14.812 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 115 2020-11-20T12:18:15.038 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:18:12.767651+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:18:15.039 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:18:15.039 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:18:15.318 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8932,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.7291619010000003,"mdsmap_epoch":115,"osdmap_epoch":130,"osdmap_epoch_barrier":130,"uptime":10.384034024} 2020-11-20T12:18:15.318 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8932, "mdsmap_epoch": 115, "osdmap_epoch": 130, "osdmap_epoch_barrier": 130, "rank_uptime": 4.729161901, "state": "up:active", "uptime": 10.384034024, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:18:15.319 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:18:15.319 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:18:15.319 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:18:15.320 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:18:15.320 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:18:15.320 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:18:15.321 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:18:15.321 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:18:15.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:15.322 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:18:15.380 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:18:15.381 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:18:15.381 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:18:15.382 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:18:15.382 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:18:15.383 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:18:15.383 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:18:15.383 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:18:15.384 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:18:15.384 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:18:15.385 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:18:15.385 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:18:15.385 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:18:15.386 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:18:15.386 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:18:15.387 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 41943sec preferred_lft 41943sec 2020-11-20T12:18:15.387 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:18:15.387 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:18:15.388 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:18:15.388 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:18:15.388 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:18:15.389 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:18:15.389 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:18:15.390 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:18:15.390 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:18:15.390 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:18:15.391 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:18:15.391 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:18:15.391 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:18:15.392 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:18:15.392 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:18:15.393 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:18:15.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:15.395 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:18:15.497 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:18:15.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:15.503 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:18:15.561 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:18:15.562 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:18:15.562 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:18:15.562 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:18:15.562 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:18:15.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:15.564 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:18:15.564 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:18:15.565 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:18:15.565 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:18:15.566 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:18:15.566 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:18:15.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:15.821 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:18:15.879 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:18:15.879 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:18:15.880 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:18:15.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:15.882 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:18:15.984 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:18:16.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.226+0000 7fb28e8d0700 1 Processor -- start 2020-11-20T12:18:16.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.226+0000 7fb28e8d0700 1 -- start start 2020-11-20T12:18:16.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.227+0000 7fb28e8d0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 0x7fb288068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.227+0000 7fb28e8d0700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb2881482d0 0x7fb288062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.227+0000 7fb28e8d0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb288062b50 0x7fb288065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.227+0000 7fb28e8d0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb288144b80 con 0x7fb288062b50 2020-11-20T12:18:16.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.227+0000 7fb28e8d0700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2881489e0 con 0x7fb2881482d0 2020-11-20T12:18:16.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.227+0000 7fb28e8d0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb288148c80 con 0x7fb2880b57b0 2020-11-20T12:18:16.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.228+0000 7fb28d0cd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb2881482d0 0x7fb288062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.228+0000 7fb28d0cd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb2881482d0 0x7fb288062660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51876/0 (socket says 172.21.15.187:51876) 2020-11-20T12:18:16.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.228+0000 7fb28d0cd700 1 -- 172.21.15.187:0/1471038459 learned_addr learned my addr 172.21.15.187:0/1471038459 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:16.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.228+0000 7fb28d8ce700 1 --2- 172.21.15.187:0/1471038459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 0x7fb288068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.228+0000 7fb28e0cf700 1 --2- 172.21.15.187:0/1471038459 >> v2:172.21.15.5:3300/0 conn(0x7fb288062b50 0x7fb288065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.229+0000 7fb28d0cd700 1 -- 172.21.15.187:0/1471038459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 msgr2=0x7fb288068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:16.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.229+0000 7fb28d0cd700 1 --2- 172.21.15.187:0/1471038459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 0x7fb288068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.229+0000 7fb28d0cd700 1 -- 172.21.15.187:0/1471038459 >> v2:172.21.15.5:3300/0 conn(0x7fb288062b50 msgr2=0x7fb288065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:16.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.229+0000 7fb28d0cd700 1 --2- 172.21.15.187:0/1471038459 >> v2:172.21.15.5:3300/0 conn(0x7fb288062b50 0x7fb288065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.229+0000 7fb28d0cd700 1 -- 172.21.15.187:0/1471038459 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288148f20 con 0x7fb2881482d0 2020-11-20T12:18:16.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.229+0000 7fb28d0cd700 1 --2- 172.21.15.187:0/1471038459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb288062660 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 rx=0x7fb288062fe0 tx=0x7fb278006eb0).ready entity=mon.1 client_cookie=b6625d7b0e47bed3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:16.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.229+0000 7fb28c8cc700 1 -- 172.21.15.187:0/1471038459 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2780098d0 con 0x7fb2881482d0 2020-11-20T12:18:16.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28c8cc700 1 -- 172.21.15.187:0/1471038459 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb278009a30 con 0x7fb2881482d0 2020-11-20T12:18:16.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28c8cc700 1 -- 172.21.15.187:0/1471038459 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278009db0 con 0x7fb2881482d0 2020-11-20T12:18:16.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 -- 172.21.15.187:0/1471038459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 msgr2=0x7fb288062660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/1471038459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb288062660 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 rx=0x7fb288062fe0 tx=0x7fb278006eb0).stop 2020-11-20T12:18:16.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 -- 172.21.15.187:0/1471038459 shutdown_connections 2020-11-20T12:18:16.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/1471038459 >> v2:172.21.15.5:3300/0 conn(0x7fb288062b50 0x7fb288065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/1471038459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 0x7fb288068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/1471038459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb288062660 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 -- 172.21.15.187:0/1471038459 >> 172.21.15.187:0/1471038459 conn(0x7fb2880bab30 msgr2=0x7fb2880baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 -- 172.21.15.187:0/1471038459 shutdown_connections 2020-11-20T12:18:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.230+0000 7fb28e8d0700 1 -- 172.21.15.187:0/1471038459 wait complete. 2020-11-20T12:18:16.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.232+0000 7fb28e8d0700 1 Processor -- start 2020-11-20T12:18:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.232+0000 7fb28e8d0700 1 -- start start 2020-11-20T12:18:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.233+0000 7fb28e8d0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb288062b50 0x7fb288144d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.233+0000 7fb28e8d0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 0x7fb28809f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.233+0000 7fb28e8d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb28815bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.233+0000 7fb28e8d0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28809b500 con 0x7fb288062b50 2020-11-20T12:18:16.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.233+0000 7fb28e8d0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28815b830 con 0x7fb2881482d0 2020-11-20T12:18:16.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.233+0000 7fb28e8d0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28815bad0 con 0x7fb2880b57b0 2020-11-20T12:18:16.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb28815bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb28815bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51882/0 (socket says 172.21.15.187:51882) 2020-11-20T12:18:16.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 -- 172.21.15.187:0/2567328874 learned_addr learned my addr 172.21.15.187:0/2567328874 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:16.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28d8ce700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb288062b50 0x7fb288144d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 -- 172.21.15.187:0/2567328874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 msgr2=0x7fb28809f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 0x7fb28809f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:16.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 -- 172.21.15.187:0/2567328874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb288062b50 msgr2=0x7fb288144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb288062b50 0x7fb288144d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 -- 172.21.15.187:0/2567328874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28809b1f0 con 0x7fb2881482d0 2020-11-20T12:18:16.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28d8ce700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb288062b50 0x7fb288144d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:16.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.234+0000 7fb28e0cf700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb28815bf70 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 rx=0x7fb28000eb40 tx=0x7fb280008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:16.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.235+0000 7fb275ffb700 1 -- 172.21.15.187:0/2567328874 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28000c7a0 con 0x7fb2881482d0 2020-11-20T12:18:16.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.235+0000 7fb275ffb700 1 -- 172.21.15.187:0/2567328874 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28000c900 con 0x7fb2881482d0 2020-11-20T12:18:16.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.235+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28809fe60 con 0x7fb2881482d0 2020-11-20T12:18:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.235+0000 7fb275ffb700 1 -- 172.21.15.187:0/2567328874 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28000cca0 con 0x7fb2881482d0 2020-11-20T12:18:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.235+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28815c460 con 0x7fb2881482d0 2020-11-20T12:18:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.236+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2880b6220 con 0x7fb2881482d0 2020-11-20T12:18:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.236+0000 7fb275ffb700 1 -- 172.21.15.187:0/2567328874 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb280013640 con 0x7fb2881482d0 2020-11-20T12:18:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.237+0000 7fb275ffb700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb260056d20 0x7fb2600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.237+0000 7fb275ffb700 1 -- 172.21.15.187:0/2567328874 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb28006fcd0 con 0x7fb2881482d0 2020-11-20T12:18:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.237+0000 7fb28d8ce700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb260056d20 0x7fb2600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.238+0000 7fb28d8ce700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb260056d20 0x7fb2600591e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7fb284009710 tx=0x7fb284008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:16.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.241+0000 7fb275ffb700 1 -- 172.21.15.187:0/2567328874 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb2800450a0 con 0x7fb2881482d0 2020-11-20T12:18:16.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.426+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb28815bc90 con 0x7fb2881482d0 2020-11-20T12:18:16.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.427+0000 7fb275ffb700 1 -- 172.21.15.187:0/2567328874 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16792 (secure 0 0 0) 0x7fb2800486c0 con 0x7fb2881482d0 2020-11-20T12:18:16.430 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:16.431 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":130,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:18:11.562810+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:18:08.692656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2020-11-20T12:18:09.325174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:18:16.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb260056d20 msgr2=0x7fb2600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb260056d20 0x7fb2600591e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7fb284009710 tx=0x7fb284008040).stop 2020-11-20T12:18:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 msgr2=0x7fb28815bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb28815bf70 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 rx=0x7fb28000eb40 tx=0x7fb280008040).stop 2020-11-20T12:18:16.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 shutdown_connections 2020-11-20T12:18:16.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb260056d20 0x7fb2600591e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb288062b50 0x7fb288144d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2881482d0 0x7fb28815bf70 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 --2- 172.21.15.187:0/2567328874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb2880b57b0 0x7fb28809f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:16.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 >> 172.21.15.187:0/2567328874 conn(0x7fb2880bab30 msgr2=0x7fb288061ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:16.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 shutdown_connections 2020-11-20T12:18:16.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.430+0000 7fb28e8d0700 1 -- 172.21.15.187:0/2567328874 wait complete. 2020-11-20T12:18:16.456 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:18:08.692656+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:18:09.325174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:18:16.457 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:18:16.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.696+0000 7f33469a3700 1 Processor -- start 2020-11-20T12:18:16.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.697+0000 7f33469a3700 1 -- start start 2020-11-20T12:18:16.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.698+0000 7f33469a3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33400b1380 0x7f3340150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.698+0000 7f33469a3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f33400b1760 0x7f3340065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.698+0000 7f33469a3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3340150390 con 0x7f3340065680 2020-11-20T12:18:16.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.698+0000 7f33469a3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3340150630 con 0x7f33400b1760 2020-11-20T12:18:16.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.698+0000 7f33469a3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3340068510 con 0x7f33400b1380 2020-11-20T12:18:16.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.699+0000 7f33459a1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33400b1380 0x7f3340150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.699+0000 7f33459a1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33400b1380 0x7f3340150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51888/0 (socket says 172.21.15.187:51888) 2020-11-20T12:18:16.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.699+0000 7f33459a1700 1 -- 172.21.15.187:0/1479448695 learned_addr learned my addr 172.21.15.187:0/1479448695 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:16.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.699+0000 7f33451a0700 1 --2- 172.21.15.187:0/1479448695 >> v2:172.21.15.5:3300/0 conn(0x7f33400b1760 0x7f3340065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.699+0000 7f33459a1700 1 -- 172.21.15.187:0/1479448695 >> v2:172.21.15.5:3300/0 conn(0x7f33400b1760 msgr2=0x7f3340065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:16.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.699+0000 7f33459a1700 1 --2- 172.21.15.187:0/1479448695 >> v2:172.21.15.5:3300/0 conn(0x7f33400b1760 0x7f3340065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.699+0000 7f33459a1700 1 -- 172.21.15.187:0/1479448695 >> v1:172.21.15.5:6789/0 conn(0x7f3340065680 legacy=0x7f3340061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:16.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.700+0000 7f33459a1700 1 -- 172.21.15.187:0/1479448695 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340150fc0 con 0x7f33400b1380 2020-11-20T12:18:16.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.700+0000 7f33459a1700 1 --2- 172.21.15.187:0/1479448695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f3340150ad0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7f33400b85e0 tx=0x7f333c006eb0).ready entity=mon.1 client_cookie=871972ce931d3150 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:16.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.700+0000 7f334499f700 1 -- 172.21.15.187:0/1479448695 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0098d0 con 0x7f33400b1380 2020-11-20T12:18:16.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.700+0000 7f334499f700 1 -- 172.21.15.187:0/1479448695 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c009a30 con 0x7f33400b1380 2020-11-20T12:18:16.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.700+0000 7f334499f700 1 -- 172.21.15.187:0/1479448695 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c009db0 con 0x7f33400b1380 2020-11-20T12:18:16.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.701+0000 7f33469a3700 1 -- 172.21.15.187:0/1479448695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 msgr2=0x7f3340150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.701+0000 7f33469a3700 1 --2- 172.21.15.187:0/1479448695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f3340150ad0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7f33400b85e0 tx=0x7f333c006eb0).stop 2020-11-20T12:18:16.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.701+0000 7f33469a3700 1 -- 172.21.15.187:0/1479448695 shutdown_connections 2020-11-20T12:18:16.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.701+0000 7f33469a3700 1 --2- 172.21.15.187:0/1479448695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f3340150ad0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.701+0000 7f33469a3700 1 --2- 172.21.15.187:0/1479448695 >> v2:172.21.15.5:3300/0 conn(0x7f33400b1760 0x7f3340065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.701+0000 7f33469a3700 1 -- 172.21.15.187:0/1479448695 >> 172.21.15.187:0/1479448695 conn(0x7f33400bab30 msgr2=0x7f33400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:16.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.701+0000 7f33469a3700 1 -- 172.21.15.187:0/1479448695 shutdown_connections 2020-11-20T12:18:16.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.701+0000 7f33469a3700 1 -- 172.21.15.187:0/1479448695 wait complete. 2020-11-20T12:18:16.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.703+0000 7f33469a3700 1 Processor -- start 2020-11-20T12:18:16.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.703+0000 7f33469a3700 1 -- start start 2020-11-20T12:18:16.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33469a3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f334013f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33469a3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33400b1760 0x7f334013e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33469a3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f334009c800 0x7f334013ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33469a3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334009ccd0 con 0x7f334009c800 2020-11-20T12:18:16.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33469a3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334009cf30 con 0x7f33400b1380 2020-11-20T12:18:16.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33469a3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33400b5ab0 con 0x7f33400b1760 2020-11-20T12:18:16.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33459a1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f334013f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33459a1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f334013f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51892/0 (socket says 172.21.15.187:51892) 2020-11-20T12:18:16.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.704+0000 7f33459a1700 1 -- 172.21.15.187:0/2840829913 learned_addr learned my addr 172.21.15.187:0/2840829913 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:16.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33451a0700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33400b1760 0x7f334013e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33461a2700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f334009c800 0x7f334013ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33459a1700 1 -- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33400b1760 msgr2=0x7f334013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33459a1700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33400b1760 0x7f334013e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33459a1700 1 -- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f334009c800 msgr2=0x7f334013ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33459a1700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f334009c800 0x7f334013ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33459a1700 1 -- 172.21.15.187:0/2840829913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334013f830 con 0x7f33400b1380 2020-11-20T12:18:16.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33461a2700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f334009c800 0x7f334013ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:16.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33459a1700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f334013f340 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7f3340068650 tx=0x7f333c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:16.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f332dffb700 1 -- 172.21.15.187:0/2840829913 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c011dd0 con 0x7f33400b1380 2020-11-20T12:18:16.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f332dffb700 1 -- 172.21.15.187:0/2840829913 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c01e460 con 0x7f33400b1380 2020-11-20T12:18:16.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334009f230 con 0x7f33400b1380 2020-11-20T12:18:16.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f332dffb700 1 -- 172.21.15.187:0/2840829913 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c01e7e0 con 0x7f33400b1380 2020-11-20T12:18:16.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.705+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334009d4d0 con 0x7f33400b1380 2020-11-20T12:18:16.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.707+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33400b5c70 con 0x7f33400b1380 2020-11-20T12:18:16.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.707+0000 7f332dffb700 1 -- 172.21.15.187:0/2840829913 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f333c025030 con 0x7f33400b1380 2020-11-20T12:18:16.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.708+0000 7f332dffb700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3318056d20 0x7f33180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:16.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.708+0000 7f332dffb700 1 -- 172.21.15.187:0/2840829913 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f333c071710 con 0x7f33400b1380 2020-11-20T12:18:16.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.708+0000 7f33451a0700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3318056d20 0x7f33180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:16.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.709+0000 7f33451a0700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3318056d20 0x7f33180591e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f334015fb00 tx=0x7f3330008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:16.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.712+0000 7f332dffb700 1 -- 172.21.15.187:0/2840829913 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f333c046c10 con 0x7f33400b1380 2020-11-20T12:18:16.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.899+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3340061a20 con 0x7f33400b1380 2020-11-20T12:18:16.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.900+0000 7f332dffb700 1 -- 172.21.15.187:0/2840829913 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v130) v1 ==== 110+0+10 (secure 0 0 0) 0x7f333c023020 con 0x7f33400b1380 2020-11-20T12:18:16.903 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:18:16.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.902+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3318056d20 msgr2=0x7f33180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.902+0000 7f33469a3700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3318056d20 0x7f33180591e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f334015fb00 tx=0x7f3330008040).stop 2020-11-20T12:18:16.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.902+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 msgr2=0x7f334013f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:16.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.902+0000 7f33469a3700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f334013f340 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7f3340068650 tx=0x7f333c015000).stop 2020-11-20T12:18:16.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.903+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 shutdown_connections 2020-11-20T12:18:16.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.903+0000 7f33469a3700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3318056d20 0x7f33180591e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.903+0000 7f33469a3700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f334009c800 0x7f334013ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.903+0000 7f33469a3700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33400b1380 0x7f334013f340 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.903+0000 7f33469a3700 1 --2- 172.21.15.187:0/2840829913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f33400b1760 0x7f334013e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:16.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.903+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 >> 172.21.15.187:0/2840829913 conn(0x7f33400bab30 msgr2=0x7f334014f300 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:16.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.903+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 shutdown_connections 2020-11-20T12:18:16.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:16.903+0000 7f33469a3700 1 -- 172.21.15.187:0/2840829913 wait complete. 2020-11-20T12:18:16.928 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:18:17.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.174+0000 7f887b59e700 1 Processor -- start 2020-11-20T12:18:17.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.174+0000 7f887b59e700 1 -- start start 2020-11-20T12:18:17.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.175+0000 7f887b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c0bfbe0 0x7f887c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.175+0000 7f887b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f887c09f560 0x7f887c1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.175+0000 7f887b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887c14d110 con 0x7f887c0bf800 2020-11-20T12:18:17.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.175+0000 7f887b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887c14c9d0 con 0x7f887c09f560 2020-11-20T12:18:17.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.175+0000 7f887b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c148aa0 con 0x7f887c0bfbe0 2020-11-20T12:18:17.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.176+0000 7f887a59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f887c0bf800 0x7f887c14c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48508/0 (socket says 172.21.15.187:48508) 2020-11-20T12:18:17.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.176+0000 7f887a59c700 1 -- 172.21.15.187:0/882379445 learned_addr learned my addr 172.21.15.187:0/882379445 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:17.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.177+0000 7f887ad9d700 1 --2- 172.21.15.187:0/882379445 >> v2:172.21.15.5:3300/0 conn(0x7f887c09f560 0x7f887c1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.177+0000 7f8879d9b700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c0bfbe0 0x7f887c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.177+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144410686 0 0) 0x7f887c14d110 con 0x7f887c0bf800 2020-11-20T12:18:17.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.177+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8860003240 con 0x7f887c0bf800 2020-11-20T12:18:17.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 351103675 0 0) 0x7f8860003240 con 0x7f887c0bf800 2020-11-20T12:18:17.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c0bfbe0 msgr2=0x7f887c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:17.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c0bfbe0 0x7f887c1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 >> v2:172.21.15.5:3300/0 conn(0x7f887c09f560 msgr2=0x7f887c1485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:17.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 --2- 172.21.15.187:0/882379445 >> v2:172.21.15.5:3300/0 conn(0x7f887c09f560 0x7f887c1485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c14cbd0 con 0x7f887c0bf800 2020-11-20T12:18:17.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f886c0014e0 con 0x7f887c0bf800 2020-11-20T12:18:17.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 >> v1:172.21.15.187:6789/0 conn(0x7f887c0bf800 legacy=0x7f887c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88600037b0 0x7f8860003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88600044e0 0x7f8860004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8860004e50 0x7f886000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c14c9d0 con 0x7f88600044e0 2020-11-20T12:18:17.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c148aa0 con 0x7f8860004e50 2020-11-20T12:18:17.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8860004200 con 0x7f88600037b0 2020-11-20T12:18:17.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.178+0000 7f887ad9d700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8860004e50 0x7f886000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f887a59c700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88600037b0 0x7f8860003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f8879d9b700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88600044e0 0x7f8860004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f887ad9d700 1 -- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88600037b0 msgr2=0x7f8860003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:17.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f887ad9d700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88600037b0 0x7f8860003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f887ad9d700 1 -- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88600044e0 msgr2=0x7f8860004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:17.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f887ad9d700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88600044e0 0x7f8860004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f887ad9d700 1 -- 172.21.15.187:0/882379445 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8860003240 con 0x7f8860004e50 2020-11-20T12:18:17.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f887a59c700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88600037b0 0x7f8860003b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:17.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.179+0000 7f887ad9d700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8860004e50 0x7f886000b730 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7f8874009ea0 tx=0x7f88740091d0).ready entity=mon.1 client_cookie=fab388e656f7596c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:17.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88740095c0 con 0x7f8860004e50 2020-11-20T12:18:17.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887400eda0 con 0x7f8860004e50 2020-11-20T12:18:17.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887959a700 1 -- 172.21.15.187:0/882379445 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874012600 con 0x7f8860004e50 2020-11-20T12:18:17.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887b59e700 1 -- 172.21.15.187:0/882379445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8860004e50 msgr2=0x7f886000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887b59e700 1 --2- 172.21.15.187:0/882379445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8860004e50 0x7f886000b730 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7f8874009ea0 tx=0x7f88740091d0).stop 2020-11-20T12:18:17.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887ad9d700 1 -- 172.21.15.187:0/882379445 reap_dead start 2020-11-20T12:18:17.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887b59e700 1 -- 172.21.15.187:0/882379445 shutdown_connections 2020-11-20T12:18:17.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887b59e700 1 -- 172.21.15.187:0/882379445 >> 172.21.15.187:0/882379445 conn(0x7f887c068370 msgr2=0x7f887c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:17.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.180+0000 7f887b59e700 1 -- 172.21.15.187:0/882379445 shutdown_connections 2020-11-20T12:18:17.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.181+0000 7f887b59e700 1 -- 172.21.15.187:0/882379445 wait complete. 2020-11-20T12:18:17.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.182+0000 7f887b59e700 1 Processor -- start 2020-11-20T12:18:17.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.182+0000 7f887b59e700 1 -- start start 2020-11-20T12:18:17.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.183+0000 7f887b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f887c15faf0 0x7f887c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.183+0000 7f887b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c15fed0 0x7f887c159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.183+0000 7f887b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f887c0b51f0 0x7f887c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.183+0000 7f887b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0b70d0 con 0x7f887c15faf0 2020-11-20T12:18:17.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.183+0000 7f887b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0b72f0 con 0x7f887c0b51f0 2020-11-20T12:18:17.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.183+0000 7f887b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0b7520 con 0x7f887c15fed0 2020-11-20T12:18:17.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f887c0b51f0 0x7f887c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f887c0b51f0 0x7f887c15c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51916/0 (socket says 172.21.15.187:51916) 2020-11-20T12:18:17.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 -- 172.21.15.187:0/2275200922 learned_addr learned my addr 172.21.15.187:0/2275200922 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:17.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887a59c700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f887c15faf0 0x7f887c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f8879d9b700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c15fed0 0x7f887c159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 -- 172.21.15.187:0/2275200922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c15fed0 msgr2=0x7f887c159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c15fed0 0x7f887c159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 -- 172.21.15.187:0/2275200922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f887c15faf0 msgr2=0x7f887c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f887c15faf0 0x7f887c0b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 -- 172.21.15.187:0/2275200922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c0b77c0 con 0x7f887c0b51f0 2020-11-20T12:18:17.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.184+0000 7f887ad9d700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f887c0b51f0 0x7f887c15c330 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 rx=0x7f8860005470 tx=0x7f887401a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:17.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.185+0000 7f886a7fc700 1 -- 172.21.15.187:0/2275200922 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88740095c0 con 0x7f887c0b51f0 2020-11-20T12:18:17.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.185+0000 7f886a7fc700 1 -- 172.21.15.187:0/2275200922 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887401cba0 con 0x7f887c0b51f0 2020-11-20T12:18:17.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.185+0000 7f886a7fc700 1 -- 172.21.15.187:0/2275200922 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88740243f0 con 0x7f887c0b51f0 2020-11-20T12:18:17.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.186+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887c160440 con 0x7f887c0b51f0 2020-11-20T12:18:17.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.186+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887c159d40 con 0x7f887c0b51f0 2020-11-20T12:18:17.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.187+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c080050 con 0x7f887c0b51f0 2020-11-20T12:18:17.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.188+0000 7f886a7fc700 1 -- 172.21.15.187:0/2275200922 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8874024550 con 0x7f887c0b51f0 2020-11-20T12:18:17.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.188+0000 7f886a7fc700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8850056d70 0x7f8850059230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.188+0000 7f886a7fc700 1 -- 172.21.15.187:0/2275200922 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8874071db0 con 0x7f887c0b51f0 2020-11-20T12:18:17.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.189+0000 7f887a59c700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8850056d70 0x7f8850059230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.189+0000 7f887a59c700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8850056d70 0x7f8850059230 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f886c000fb0 tx=0x7f886c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:17.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.192+0000 7f886a7fc700 1 -- 172.21.15.187:0/2275200922 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8874047220 con 0x7f887c0b51f0 2020-11-20T12:18:17.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.379+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f887c0b7950 con 0x7f887c0b51f0 2020-11-20T12:18:17.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.380+0000 7f886a7fc700 1 -- 172.21.15.187:0/2275200922 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v130) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8874016030 con 0x7f887c0b51f0 2020-11-20T12:18:17.383 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:18:17.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8850056d70 msgr2=0x7f8850059230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8850056d70 0x7f8850059230 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f886c000fb0 tx=0x7f886c009040).stop 2020-11-20T12:18:17.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f887c0b51f0 msgr2=0x7f887c15c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f887c0b51f0 0x7f887c15c330 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 rx=0x7f8860005470 tx=0x7f887401a000).stop 2020-11-20T12:18:17.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 shutdown_connections 2020-11-20T12:18:17.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8850056d70 0x7f8850059230 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f887c15faf0 0x7f887c0b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f887c0b51f0 0x7f887c15c330 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 --2- 172.21.15.187:0/2275200922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f887c15fed0 0x7f887c159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.382+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 >> 172.21.15.187:0/2275200922 conn(0x7f887c068370 msgr2=0x7f887c0bdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:17.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.383+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 shutdown_connections 2020-11-20T12:18:17.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.383+0000 7f887b59e700 1 -- 172.21.15.187:0/2275200922 wait complete. 2020-11-20T12:18:17.408 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:18:17.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.649+0000 7efe1ecf9700 1 Processor -- start 2020-11-20T12:18:17.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.649+0000 7efe1ecf9700 1 -- start start 2020-11-20T12:18:17.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.651+0000 7efe1ecf9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efe180b1760 0x7efe18065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.651+0000 7efe1ecf9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe18065680 0x7efe1814c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.651+0000 7efe1ecf9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe18150390 con 0x7efe180b1380 2020-11-20T12:18:17.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.651+0000 7efe1ecf9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe181511e0 con 0x7efe180b1760 2020-11-20T12:18:17.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.651+0000 7efe1ecf9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1814d760 con 0x7efe18065680 2020-11-20T12:18:17.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.652+0000 7efe1d4f6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efe180b1760 0x7efe18065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.652+0000 7efe1d4f6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efe180b1760 0x7efe18065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:44186/0 (socket says 172.21.15.187:44186) 2020-11-20T12:18:17.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.652+0000 7efe1d4f6700 1 -- 172.21.15.187:0/1038867602 learned_addr learned my addr 172.21.15.187:0/1038867602 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:17.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.652+0000 7efe1e4f8700 1 --2- 172.21.15.187:0/1038867602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe18065680 0x7efe1814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.652+0000 7efe1d4f6700 1 -- 172.21.15.187:0/1038867602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe18065680 msgr2=0x7efe1814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:17.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.652+0000 7efe1d4f6700 1 --2- 172.21.15.187:0/1038867602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe18065680 0x7efe1814c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.652+0000 7efe1d4f6700 1 -- 172.21.15.187:0/1038867602 >> v1:172.21.15.5:6789/0 conn(0x7efe180b1380 legacy=0x7efe18150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:17.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.652+0000 7efe1d4f6700 1 -- 172.21.15.187:0/1038867602 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18150630 con 0x7efe180b1760 2020-11-20T12:18:17.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.653+0000 7efe1d4f6700 1 --2- 172.21.15.187:0/1038867602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe180b1760 0x7efe18065190 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7efe0800ba20 tx=0x7efe08006eb0).ready entity=mon.0 client_cookie=587e3df4ae33a931 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:17.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.653+0000 7efe1ccf5700 1 -- 172.21.15.187:0/1038867602 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08009850 con 0x7efe180b1760 2020-11-20T12:18:17.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.653+0000 7efe1ccf5700 1 -- 172.21.15.187:0/1038867602 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe080099b0 con 0x7efe180b1760 2020-11-20T12:18:17.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.653+0000 7efe1ccf5700 1 -- 172.21.15.187:0/1038867602 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08009d30 con 0x7efe180b1760 2020-11-20T12:18:17.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.654+0000 7efe1ecf9700 1 -- 172.21.15.187:0/1038867602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe180b1760 msgr2=0x7efe18065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.654+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/1038867602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe180b1760 0x7efe18065190 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7efe0800ba20 tx=0x7efe08006eb0).stop 2020-11-20T12:18:17.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.654+0000 7efe1ecf9700 1 -- 172.21.15.187:0/1038867602 shutdown_connections 2020-11-20T12:18:17.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.654+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/1038867602 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe18065680 0x7efe1814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.654+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/1038867602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe180b1760 0x7efe18065190 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.654+0000 7efe1ecf9700 1 -- 172.21.15.187:0/1038867602 >> 172.21.15.187:0/1038867602 conn(0x7efe180bab30 msgr2=0x7efe180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:17.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.654+0000 7efe1ecf9700 1 -- 172.21.15.187:0/1038867602 shutdown_connections 2020-11-20T12:18:17.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.654+0000 7efe1ecf9700 1 -- 172.21.15.187:0/1038867602 wait complete. 2020-11-20T12:18:17.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.656+0000 7efe1ecf9700 1 Processor -- start 2020-11-20T12:18:17.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.656+0000 7efe1ecf9700 1 -- start start 2020-11-20T12:18:17.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1ecf9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe180b1380 0x7efe1813f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1ecf9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe180b1760 0x7efe1813e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1ecf9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe1809f2c0 0x7efe1813ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1ecf9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe180b4640 con 0x7efe1809f2c0 2020-11-20T12:18:17.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1ecf9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1809e590 con 0x7efe180b1380 2020-11-20T12:18:17.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1ecf9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1809e830 con 0x7efe180b1760 2020-11-20T12:18:17.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1dcf7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe180b1380 0x7efe1813f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1dcf7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe180b1380 0x7efe1813f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51926/0 (socket says 172.21.15.187:51926) 2020-11-20T12:18:17.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.657+0000 7efe1dcf7700 1 -- 172.21.15.187:0/701676640 learned_addr learned my addr 172.21.15.187:0/701676640 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:17.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1d4f6700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe180b1760 0x7efe1813e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1dcf7700 1 -- 172.21.15.187:0/701676640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe180b1760 msgr2=0x7efe1813e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1dcf7700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe180b1760 0x7efe1813e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1dcf7700 1 -- 172.21.15.187:0/701676640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe1809f2c0 msgr2=0x7efe1813ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1e4f8700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe1809f2c0 0x7efe1813ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1dcf7700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe1809f2c0 0x7efe1813ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1dcf7700 1 -- 172.21.15.187:0/701676640 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe180b4330 con 0x7efe180b1380 2020-11-20T12:18:17.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1dcf7700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe180b1380 0x7efe1813f340 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7efe1400c730 tx=0x7efe14007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:17.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe05ffb700 1 -- 172.21.15.187:0/701676640 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1400e560 con 0x7efe180b1380 2020-11-20T12:18:17.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe05ffb700 1 -- 172.21.15.187:0/701676640 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1400e6e0 con 0x7efe180b1380 2020-11-20T12:18:17.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1813f830 con 0x7efe180b1380 2020-11-20T12:18:17.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.658+0000 7efe05ffb700 1 -- 172.21.15.187:0/701676640 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1400eaa0 con 0x7efe180b1380 2020-11-20T12:18:17.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.659+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe180b3fb0 con 0x7efe180b1380 2020-11-20T12:18:17.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.660+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe18061a20 con 0x7efe180b1380 2020-11-20T12:18:17.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.660+0000 7efe05ffb700 1 -- 172.21.15.187:0/701676640 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efe14019030 con 0x7efe180b1380 2020-11-20T12:18:17.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.661+0000 7efe05ffb700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdf0056d50 0x7efdf0059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:17.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.661+0000 7efe05ffb700 1 -- 172.21.15.187:0/701676640 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efe1406f420 con 0x7efe180b1380 2020-11-20T12:18:17.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.661+0000 7efe1d4f6700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdf0056d50 0x7efdf0059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:17.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.662+0000 7efe1d4f6700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdf0056d50 0x7efdf0059210 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7efe0800b1b0 tx=0x7efe08015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:17.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.665+0000 7efe05ffb700 1 -- 172.21.15.187:0/701676640 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efe14043fd0 con 0x7efe180b1380 2020-11-20T12:18:17.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.858+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe18068510 con 0x7efe180b1380 2020-11-20T12:18:17.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.859+0000 7efe05ffb700 1 -- 172.21.15.187:0/701676640 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+10 (secure 0 0 0) 0x7efe14044920 con 0x7efe180b1380 2020-11-20T12:18:17.862 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:18:17.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdf0056d50 msgr2=0x7efdf0059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdf0056d50 0x7efdf0059210 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7efe0800b1b0 tx=0x7efe08015000).stop 2020-11-20T12:18:17.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe180b1380 msgr2=0x7efe1813f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:17.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe180b1380 0x7efe1813f340 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7efe1400c730 tx=0x7efe14007c30).stop 2020-11-20T12:18:17.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 shutdown_connections 2020-11-20T12:18:17.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efdf0056d50 0x7efdf0059210 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efe1809f2c0 0x7efe1813ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe180b1380 0x7efe1813f340 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 --2- 172.21.15.187:0/701676640 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efe180b1760 0x7efe1813e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:17.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.861+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 >> 172.21.15.187:0/701676640 conn(0x7efe180bab30 msgr2=0x7efe1814ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:17.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.862+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 shutdown_connections 2020-11-20T12:18:17.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:17.862+0000 7efe1ecf9700 1 -- 172.21.15.187:0/701676640 wait complete. 2020-11-20T12:18:17.889 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:18:18.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.132+0000 7f05dad55700 1 Processor -- start 2020-11-20T12:18:18.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.132+0000 7f05dad55700 1 -- start start 2020-11-20T12:18:18.136 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.134+0000 7f05dad55700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f05d4065f90 0x7f05d4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:18.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.134+0000 7f05dad55700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d409ca90 0x7f05d409cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:18.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.134+0000 7f05dad55700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d40b13e0 con 0x7f05d4065190 2020-11-20T12:18:18.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.134+0000 7f05dad55700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d414ab30 con 0x7f05d4065f90 2020-11-20T12:18:18.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.134+0000 7f05dad55700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4062c10 con 0x7f05d409ca90 2020-11-20T12:18:18.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.135+0000 7f05d9d53700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f05d4065190 0x7f05d4143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:42050/0 (socket says 172.21.15.187:42050) 2020-11-20T12:18:18.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.135+0000 7f05d9d53700 1 -- 172.21.15.187:0/158479618 learned_addr learned my addr 172.21.15.187:0/158479618 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:18.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.135+0000 7f05da554700 1 --2- 172.21.15.187:0/158479618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d409ca90 0x7f05d409cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:18.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.135+0000 7f05d9552700 1 --2- 172.21.15.187:0/158479618 >> v2:172.21.15.5:3300/0 conn(0x7f05d4065f90 0x7f05d4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:18.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.136+0000 7f05da554700 1 -- 172.21.15.187:0/158479618 >> v2:172.21.15.5:3300/0 conn(0x7f05d4065f90 msgr2=0x7f05d4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:18.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.136+0000 7f05da554700 1 --2- 172.21.15.187:0/158479618 >> v2:172.21.15.5:3300/0 conn(0x7f05d4065f90 0x7f05d4065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:18.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.136+0000 7f05da554700 1 -- 172.21.15.187:0/158479618 >> v1:172.21.15.5:6789/0 conn(0x7f05d4065190 legacy=0x7f05d4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:18.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.136+0000 7f05da554700 1 -- 172.21.15.187:0/158479618 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d40b1640 con 0x7f05d409ca90 2020-11-20T12:18:18.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.136+0000 7f05da554700 1 --2- 172.21.15.187:0/158479618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d409ca90 0x7f05d409cf40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f05cc009c90 tx=0x7f05cc0091d0).ready entity=mon.2 client_cookie=b32d39ba6f9fe464 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:18.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.137+0000 7f05d8d51700 1 -- 172.21.15.187:0/158479618 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc00bb00 con 0x7f05d409ca90 2020-11-20T12:18:18.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.137+0000 7f05d8d51700 1 -- 172.21.15.187:0/158479618 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc00bc60 con 0x7f05d409ca90 2020-11-20T12:18:18.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.137+0000 7f05d8d51700 1 -- 172.21.15.187:0/158479618 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc011400 con 0x7f05d409ca90 2020-11-20T12:18:18.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.138+0000 7f05dad55700 1 -- 172.21.15.187:0/158479618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d409ca90 msgr2=0x7f05d409cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:18.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.138+0000 7f05dad55700 1 --2- 172.21.15.187:0/158479618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d409ca90 0x7f05d409cf40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f05cc009c90 tx=0x7f05cc0091d0).stop 2020-11-20T12:18:18.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.138+0000 7f05dad55700 1 -- 172.21.15.187:0/158479618 shutdown_connections 2020-11-20T12:18:18.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.138+0000 7f05dad55700 1 --2- 172.21.15.187:0/158479618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d409ca90 0x7f05d409cf40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:18.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.138+0000 7f05dad55700 1 --2- 172.21.15.187:0/158479618 >> v2:172.21.15.5:3300/0 conn(0x7f05d4065f90 0x7f05d4065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:18.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.138+0000 7f05dad55700 1 -- 172.21.15.187:0/158479618 >> 172.21.15.187:0/158479618 conn(0x7f05d40bab30 msgr2=0x7f05d40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:18.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.138+0000 7f05dad55700 1 -- 172.21.15.187:0/158479618 shutdown_connections 2020-11-20T12:18:18.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.138+0000 7f05dad55700 1 -- 172.21.15.187:0/158479618 wait complete. 2020-11-20T12:18:18.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.140+0000 7f05dad55700 1 Processor -- start 2020-11-20T12:18:18.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.140+0000 7f05dad55700 1 -- start start 2020-11-20T12:18:18.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.141+0000 7f05dad55700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d4065190 0x7f05d40b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:18.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.141+0000 7f05dad55700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05d4065f90 0x7f05d4158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:18.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.141+0000 7f05dad55700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05d409ca90 0x7f05d40990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:18.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.141+0000 7f05dad55700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4067a40 con 0x7f05d409ca90 2020-11-20T12:18:18.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.141+0000 7f05dad55700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d41581d0 con 0x7f05d4065f90 2020-11-20T12:18:18.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.141+0000 7f05dad55700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4158470 con 0x7f05d4065190 2020-11-20T12:18:18.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9552700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05d4065f90 0x7f05d4158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:18.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9552700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05d4065f90 0x7f05d4158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51942/0 (socket says 172.21.15.187:51942) 2020-11-20T12:18:18.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9552700 1 -- 172.21.15.187:0/42494171 learned_addr learned my addr 172.21.15.187:0/42494171 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:18.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05da554700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05d409ca90 0x7f05d40990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:18.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9d53700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d4065190 0x7f05d40b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:18.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9552700 1 -- 172.21.15.187:0/42494171 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d4065190 msgr2=0x7f05d40b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:18.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9552700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d4065190 0x7f05d40b7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:18.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9552700 1 -- 172.21.15.187:0/42494171 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05d409ca90 msgr2=0x7f05d40990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:18.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9552700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05d409ca90 0x7f05d40990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:18.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.142+0000 7f05d9552700 1 -- 172.21.15.187:0/42494171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d4067730 con 0x7f05d4065f90 2020-11-20T12:18:18.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.143+0000 7f05d9552700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05d4065f90 0x7f05d4158910 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 rx=0x7f05c400eba0 tx=0x7f05c4006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:18.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.143+0000 7f05c1ffb700 1 -- 172.21.15.187:0/42494171 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c400c950 con 0x7f05d4065f90 2020-11-20T12:18:18.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.143+0000 7f05c1ffb700 1 -- 172.21.15.187:0/42494171 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c400cab0 con 0x7f05d4065f90 2020-11-20T12:18:18.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.143+0000 7f05c1ffb700 1 -- 172.21.15.187:0/42494171 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c400ce30 con 0x7f05d4065f90 2020-11-20T12:18:18.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.143+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d40b7a40 con 0x7f05d4065f90 2020-11-20T12:18:18.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.143+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d4159040 con 0x7f05d4065f90 2020-11-20T12:18:18.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.144+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d4158630 con 0x7f05d4065f90 2020-11-20T12:18:18.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.145+0000 7f05c1ffb700 1 -- 172.21.15.187:0/42494171 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f05c401a030 con 0x7f05d4065f90 2020-11-20T12:18:18.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.145+0000 7f05c1ffb700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f05ac056d20 0x7f05ac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:18.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.146+0000 7f05c1ffb700 1 -- 172.21.15.187:0/42494171 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f05c406fc60 con 0x7f05d4065f90 2020-11-20T12:18:18.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.146+0000 7f05d9d53700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f05ac056d20 0x7f05ac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:18.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.146+0000 7f05d9d53700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f05ac056d20 0x7f05ac0591e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f05d0000bc0 tx=0x7f05d0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:18.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.149+0000 7f05c1ffb700 1 -- 172.21.15.187:0/42494171 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f05c4045050 con 0x7f05d4065f90 2020-11-20T12:18:18.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.337+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05d4061a20 con 0x7f05d4065f90 2020-11-20T12:18:19.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.338+0000 7f05c1ffb700 1 -- 172.21.15.187:0/42494171 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+10 (secure 0 0 0) 0x7f05c4048670 con 0x7f05d4065f90 2020-11-20T12:18:19.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.340+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f05ac056d20 msgr2=0x7f05ac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:19.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.340+0000 7f05dad55700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f05ac056d20 0x7f05ac0591e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f05d0000bc0 tx=0x7f05d0009040).stop 2020-11-20T12:18:19.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.340+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05d4065f90 msgr2=0x7f05d4158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:19.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.340+0000 7f05dad55700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05d4065f90 0x7f05d4158910 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 rx=0x7f05c400eba0 tx=0x7f05c4006df0).stop 2020-11-20T12:18:19.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.341+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 shutdown_connections 2020-11-20T12:18:19.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.341+0000 7f05dad55700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f05ac056d20 0x7f05ac0591e0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.341+0000 7f05dad55700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05d409ca90 0x7f05d40990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.341+0000 7f05dad55700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f05d4065190 0x7f05d40b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.341+0000 7f05dad55700 1 --2- 172.21.15.187:0/42494171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05d4065f90 0x7f05d4158910 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.341+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 >> 172.21.15.187:0/42494171 conn(0x7f05d40bab30 msgr2=0x7f05d40b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:19.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.341+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 shutdown_connections 2020-11-20T12:18:19.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:18.341+0000 7f05dad55700 1 -- 172.21.15.187:0/42494171 wait complete. 2020-11-20T12:18:19.083 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:18:19.084 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:18:19.085 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:19.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.338+0000 7f42f3fff700 1 Processor -- start 2020-11-20T12:18:19.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.338+0000 7f42f3fff700 1 -- start start 2020-11-20T12:18:19.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.339+0000 7f42f3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b57b0 0x7f42f4068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:19.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.339+0000 7f42f3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f42f40b5f50 0x7f42f40b6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:19.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.340+0000 7f42f3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f4144b80 con 0x7f42f41482d0 2020-11-20T12:18:19.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.340+0000 7f42f3fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f41489e0 con 0x7f42f40b5f50 2020-11-20T12:18:19.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.340+0000 7f42f3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f4148c80 con 0x7f42f40b57b0 2020-11-20T12:18:19.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.340+0000 7f42f2ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b57b0 0x7f42f4068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:19.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.341+0000 7f42ea7fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f42f41482d0 0x7f42f4062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:42064/0 (socket says 172.21.15.187:42064) 2020-11-20T12:18:19.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.341+0000 7f42ea7fc700 1 -- 172.21.15.187:0/2793979237 learned_addr learned my addr 172.21.15.187:0/2793979237 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:19.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.341+0000 7f42f37fe700 1 --2- 172.21.15.187:0/2793979237 >> v2:172.21.15.187:3300/0 conn(0x7f42f40b5f50 0x7f42f40b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:19.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.341+0000 7f42f37fe700 1 -- 172.21.15.187:0/2793979237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b57b0 msgr2=0x7f42f4068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:19.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.341+0000 7f42f37fe700 1 --2- 172.21.15.187:0/2793979237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b57b0 0x7f42f4068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.341+0000 7f42f37fe700 1 -- 172.21.15.187:0/2793979237 >> v1:172.21.15.5:6789/0 conn(0x7f42f41482d0 legacy=0x7f42f4062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:19.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.341+0000 7f42f37fe700 1 -- 172.21.15.187:0/2793979237 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f4148f20 con 0x7f42f40b5f50 2020-11-20T12:18:19.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.342+0000 7f42f37fe700 1 --2- 172.21.15.187:0/2793979237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f40b5f50 0x7f42f40b6400 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 rx=0x7f42ec009c90 tx=0x7f42ec0091d0).ready entity=mon.1 client_cookie=e12e16a3acebeee0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:19.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.342+0000 7f42f27fc700 1 -- 172.21.15.187:0/2793979237 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec00ba70 con 0x7f42f40b5f50 2020-11-20T12:18:19.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.342+0000 7f42f27fc700 1 -- 172.21.15.187:0/2793979237 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec00bbd0 con 0x7f42f40b5f50 2020-11-20T12:18:19.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.342+0000 7f42f27fc700 1 -- 172.21.15.187:0/2793979237 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec011400 con 0x7f42f40b5f50 2020-11-20T12:18:19.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.343+0000 7f42f3fff700 1 -- 172.21.15.187:0/2793979237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f40b5f50 msgr2=0x7f42f40b6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:19.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.343+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2793979237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f40b5f50 0x7f42f40b6400 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 rx=0x7f42ec009c90 tx=0x7f42ec0091d0).stop 2020-11-20T12:18:19.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.343+0000 7f42f3fff700 1 -- 172.21.15.187:0/2793979237 shutdown_connections 2020-11-20T12:18:19.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.343+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2793979237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f40b5f50 0x7f42f40b6400 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.343+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2793979237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b57b0 0x7f42f4068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.343+0000 7f42f3fff700 1 -- 172.21.15.187:0/2793979237 >> 172.21.15.187:0/2793979237 conn(0x7f42f40bab30 msgr2=0x7f42f40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:19.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.343+0000 7f42f3fff700 1 -- 172.21.15.187:0/2793979237 shutdown_connections 2020-11-20T12:18:19.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.343+0000 7f42f3fff700 1 -- 172.21.15.187:0/2793979237 wait complete. 2020-11-20T12:18:19.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.345+0000 7f42f3fff700 1 Processor -- start 2020-11-20T12:18:19.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.345+0000 7f42f3fff700 1 -- start start 2020-11-20T12:18:19.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.346+0000 7f42f3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42f40b57b0 0x7f42f4147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:19.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.346+0000 7f42f3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b5f50 0x7f42f4139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:19.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.346+0000 7f42f3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f41482d0 0x7f42f413ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:19.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.346+0000 7f42f3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f4146860 con 0x7f42f40b57b0 2020-11-20T12:18:19.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.346+0000 7f42f3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f4146b00 con 0x7f42f41482d0 2020-11-20T12:18:19.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.346+0000 7f42f3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f41475c0 con 0x7f42f40b5f50 2020-11-20T12:18:19.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f41482d0 0x7f42f413ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:19.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f41482d0 0x7f42f413ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51958/0 (socket says 172.21.15.187:51958) 2020-11-20T12:18:19.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f37fe700 1 -- 172.21.15.187:0/2771071767 learned_addr learned my addr 172.21.15.187:0/2771071767 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:19.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42ea7fc700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b5f50 0x7f42f4139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:19.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f2ffd700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42f40b57b0 0x7f42f4147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:19.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f37fe700 1 -- 172.21.15.187:0/2771071767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b5f50 msgr2=0x7f42f4139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:19.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f37fe700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b5f50 0x7f42f4139ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f37fe700 1 -- 172.21.15.187:0/2771071767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42f40b57b0 msgr2=0x7f42f4147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:19.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f37fe700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42f40b57b0 0x7f42f4147000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.347+0000 7f42f37fe700 1 -- 172.21.15.187:0/2771071767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f409f5d0 con 0x7f42f41482d0 2020-11-20T12:18:19.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.348+0000 7f42f37fe700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f41482d0 0x7f42f413ccd0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7f42ec009210 tx=0x7f42ec016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:19.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.348+0000 7f42eb7fe700 1 -- 172.21.15.187:0/2771071767 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec010780 con 0x7f42f41482d0 2020-11-20T12:18:19.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.348+0000 7f42eb7fe700 1 -- 172.21.15.187:0/2771071767 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec0108e0 con 0x7f42f41482d0 2020-11-20T12:18:19.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.348+0000 7f42eb7fe700 1 -- 172.21.15.187:0/2771071767 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec010c80 con 0x7f42f41482d0 2020-11-20T12:18:19.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.348+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f413d1c0 con 0x7f42f41482d0 2020-11-20T12:18:19.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.348+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f409dae0 con 0x7f42f41482d0 2020-11-20T12:18:19.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.349+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f4147780 con 0x7f42f41482d0 2020-11-20T12:18:19.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.350+0000 7f42eb7fe700 1 -- 172.21.15.187:0/2771071767 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f42ec017070 con 0x7f42f41482d0 2020-11-20T12:18:19.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.350+0000 7f42eb7fe700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42d0056d20 0x7f42d00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:19.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.350+0000 7f42eb7fe700 1 -- 172.21.15.187:0/2771071767 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f42ec072660 con 0x7f42f41482d0 2020-11-20T12:18:19.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.351+0000 7f42f2ffd700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42d0056d20 0x7f42d00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:19.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.352+0000 7f42f2ffd700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42d0056d20 0x7f42d00591e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7f42e4006fd0 tx=0x7f42e4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:19.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.356+0000 7f42eb7fe700 1 -- 172.21.15.187:0/2771071767 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f42ec0479a0 con 0x7f42f41482d0 2020-11-20T12:18:19.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.604+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42f4061a20 con 0x7f42f41482d0 2020-11-20T12:18:19.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.606+0000 7f42eb7fe700 1 -- 172.21.15.187:0/2771071767 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+2142 (secure 0 0 0) 0x7f42ec015020 con 0x7f42f41482d0 2020-11-20T12:18:19.608 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:19.609 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":115,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5900,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3549193191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3549193191},{"type":"v1","addr":"172.21.15.5:6833","nonce":3549193191}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":111}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:12.767651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:19.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42d0056d20 msgr2=0x7f42d00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:19.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42d0056d20 0x7f42d00591e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7f42e4006fd0 tx=0x7f42e4008040).stop 2020-11-20T12:18:19.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f41482d0 msgr2=0x7f42f413ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:19.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f41482d0 0x7f42f413ccd0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7f42ec009210 tx=0x7f42ec016000).stop 2020-11-20T12:18:19.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 shutdown_connections 2020-11-20T12:18:19.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42d0056d20 0x7f42d00591e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42f40b57b0 0x7f42f4147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f41482d0 0x7f42f413ccd0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 --2- 172.21.15.187:0/2771071767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42f40b5f50 0x7f42f4139ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:19.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.608+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 >> 172.21.15.187:0/2771071767 conn(0x7f42f40bab30 msgr2=0x7f42f4099c00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:19.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.609+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 shutdown_connections 2020-11-20T12:18:19.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:19.609+0000 7f42f3fff700 1 -- 172.21.15.187:0/2771071767 wait complete. 2020-11-20T12:18:19.615 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 115 2020-11-20T12:18:19.837 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:18:12.767651+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:18:19.838 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:18:19.839 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:18:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8932,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":9.537669996,"mdsmap_epoch":115,"osdmap_epoch":130,"osdmap_epoch_barrier":130,"uptime":15.192546700999999} 2020-11-20T12:18:20.124 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8932, "mdsmap_epoch": 115, "osdmap_epoch": 130, "osdmap_epoch_barrier": 130, "rank_uptime": 9.537669996, "state": "up:active", "uptime": 15.192546701, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:18:20.124 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:18:20.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:20.125 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:18:20.183 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:18:20.184 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:18:20.323 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:18:20.323 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:18:20.362 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:18:20.363 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:18:20.374 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:18:20.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:20.375 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:18:20.475 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2020-11-20T12:18:20.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.711+0000 7ff58134a700 1 Processor -- start 2020-11-20T12:18:20.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.712+0000 7ff58134a700 1 -- start start 2020-11-20T12:18:20.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.713+0000 7ff58134a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff57c065190 0x7ff57c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:20.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.713+0000 7ff58134a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff57c09ca90 0x7ff57c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:20.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.713+0000 7ff58134a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c0b13e0 con 0x7ff57c065f90 2020-11-20T12:18:20.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.713+0000 7ff58134a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57c14ab30 con 0x7ff57c09ca90 2020-11-20T12:18:20.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.713+0000 7ff58134a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57c062c10 con 0x7ff57c065190 2020-11-20T12:18:20.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.714+0000 7ff57bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff57c065190 0x7ff57c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:20.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.714+0000 7ff5737fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff57c065f90 0x7ff57c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48574/0 (socket says 172.21.15.187:48574) 2020-11-20T12:18:20.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.714+0000 7ff5737fe700 1 -- 172.21.15.187:0/4260081959 learned_addr learned my addr 172.21.15.187:0/4260081959 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:20.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.714+0000 7ff580b49700 1 --2- 172.21.15.187:0/4260081959 >> v2:172.21.15.5:3300/0 conn(0x7ff57c09ca90 0x7ff57c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:20.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.715+0000 7ff57bfff700 1 -- 172.21.15.187:0/4260081959 >> v2:172.21.15.5:3300/0 conn(0x7ff57c09ca90 msgr2=0x7ff57c09cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:20.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.715+0000 7ff57bfff700 1 --2- 172.21.15.187:0/4260081959 >> v2:172.21.15.5:3300/0 conn(0x7ff57c09ca90 0x7ff57c09cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:20.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.715+0000 7ff57bfff700 1 -- 172.21.15.187:0/4260081959 >> v1:172.21.15.187:6789/0 conn(0x7ff57c065f90 legacy=0x7ff57c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:20.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.715+0000 7ff57b7fe700 1 -- 172.21.15.187:0/4260081959 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907659258 0 0) 0x7ff57c0b13e0 con 0x7ff57c065f90 2020-11-20T12:18:20.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.715+0000 7ff57bfff700 1 -- 172.21.15.187:0/4260081959 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c0b1640 con 0x7ff57c065190 2020-11-20T12:18:20.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.715+0000 7ff57bfff700 1 --2- 172.21.15.187:0/4260081959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c065190 0x7ff57c065570 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7ff57c05c8b0 tx=0x7ff56c006df0).ready entity=mon.1 client_cookie=19280a27c2c5c305 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:20.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.716+0000 7ff57b7fe700 1 -- 172.21.15.187:0/4260081959 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c009840 con 0x7ff57c065190 2020-11-20T12:18:20.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.716+0000 7ff57b7fe700 1 -- 172.21.15.187:0/4260081959 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c0099a0 con 0x7ff57c065190 2020-11-20T12:18:20.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.716+0000 7ff57b7fe700 1 -- 172.21.15.187:0/4260081959 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c009d20 con 0x7ff57c065190 2020-11-20T12:18:20.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.717+0000 7ff58134a700 1 -- 172.21.15.187:0/4260081959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c065190 msgr2=0x7ff57c065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:20.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.717+0000 7ff58134a700 1 --2- 172.21.15.187:0/4260081959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c065190 0x7ff57c065570 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7ff57c05c8b0 tx=0x7ff56c006df0).stop 2020-11-20T12:18:20.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.717+0000 7ff58134a700 1 -- 172.21.15.187:0/4260081959 shutdown_connections 2020-11-20T12:18:20.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.717+0000 7ff58134a700 1 --2- 172.21.15.187:0/4260081959 >> v2:172.21.15.5:3300/0 conn(0x7ff57c09ca90 0x7ff57c09cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:20.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.717+0000 7ff58134a700 1 --2- 172.21.15.187:0/4260081959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c065190 0x7ff57c065570 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:20.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.717+0000 7ff58134a700 1 -- 172.21.15.187:0/4260081959 >> 172.21.15.187:0/4260081959 conn(0x7ff57c0bab30 msgr2=0x7ff57c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:20.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.717+0000 7ff58134a700 1 -- 172.21.15.187:0/4260081959 shutdown_connections 2020-11-20T12:18:20.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.717+0000 7ff58134a700 1 -- 172.21.15.187:0/4260081959 wait complete. 2020-11-20T12:18:20.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.719+0000 7ff58134a700 1 Processor -- start 2020-11-20T12:18:20.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.719+0000 7ff58134a700 1 -- start start 2020-11-20T12:18:20.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff58134a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff57c065190 0x7ff57c099320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:20.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff58134a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c065f90 0x7ff57c09fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:20.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff58134a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c09ca90 0x7ff57c0a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:20.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff58134a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0b2b10 con 0x7ff57c065f90 2020-11-20T12:18:20.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff58134a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c15c450 con 0x7ff57c09ca90 2020-11-20T12:18:20.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff58134a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c15c6f0 con 0x7ff57c065190 2020-11-20T12:18:20.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff580b49700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c09ca90 0x7ff57c0a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:20.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff580b49700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c09ca90 0x7ff57c0a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51976/0 (socket says 172.21.15.187:51976) 2020-11-20T12:18:20.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff580b49700 1 -- 172.21.15.187:0/2019981884 learned_addr learned my addr 172.21.15.187:0/2019981884 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:20.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff5737fe700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c065f90 0x7ff57c09fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:20.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff57bfff700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff57c065190 0x7ff57c099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:20.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.720+0000 7ff580b49700 1 -- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff57c065190 msgr2=0x7ff57c099320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:20.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff580b49700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff57c065190 0x7ff57c099320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:20.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff580b49700 1 -- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c065f90 msgr2=0x7ff57c09fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:20.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff580b49700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c065f90 0x7ff57c09fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:20.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff580b49700 1 -- 172.21.15.187:0/2019981884 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c099810 con 0x7ff57c09ca90 2020-11-20T12:18:20.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff5737fe700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c065f90 0x7ff57c09fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:20.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff57bfff700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff57c065190 0x7ff57c099320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:20.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff580b49700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c09ca90 0x7ff57c0a01e0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7ff57400ed20 tx=0x7ff5740091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:20.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff578ff9700 1 -- 172.21.15.187:0/2019981884 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57400cbb0 con 0x7ff57c09ca90 2020-11-20T12:18:20.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff578ff9700 1 -- 172.21.15.187:0/2019981884 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57400cd10 con 0x7ff57c09ca90 2020-11-20T12:18:20.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c066d20 con 0x7ff57c09ca90 2020-11-20T12:18:20.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff578ff9700 1 -- 172.21.15.187:0/2019981884 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574012620 con 0x7ff57c09ca90 2020-11-20T12:18:20.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.721+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c0a06d0 con 0x7ff57c09ca90 2020-11-20T12:18:20.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.723+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c067040 con 0x7ff57c09ca90 2020-11-20T12:18:20.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.723+0000 7ff578ff9700 1 -- 172.21.15.187:0/2019981884 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff574022070 con 0x7ff57c09ca90 2020-11-20T12:18:20.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.724+0000 7ff578ff9700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff558056d20 0x7ff5580591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:20.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.724+0000 7ff578ff9700 1 -- 172.21.15.187:0/2019981884 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff5740707a0 con 0x7ff57c09ca90 2020-11-20T12:18:20.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.724+0000 7ff57bfff700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff558056d20 0x7ff5580591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:20.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.725+0000 7ff57bfff700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff558056d20 0x7ff5580591e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7ff57c09f550 tx=0x7ff56c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:20.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.728+0000 7ff578ff9700 1 -- 172.21.15.187:0/2019981884 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff574045ae0 con 0x7ff57c09ca90 2020-11-20T12:18:20.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:20.974+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7ff57c15c290 con 0x7ff57c09ca90 2020-11-20T12:18:21.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.380+0000 7ff578ff9700 1 -- 172.21.15.187:0/2019981884 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v116) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff574016020 con 0x7ff57c09ca90 2020-11-20T12:18:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.384+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff558056d20 msgr2=0x7ff5580591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.384+0000 7ff58134a700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff558056d20 0x7ff5580591e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7ff57c09f550 tx=0x7ff56c015000).stop 2020-11-20T12:18:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.384+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c09ca90 msgr2=0x7ff57c0a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.384+0000 7ff58134a700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c09ca90 0x7ff57c0a01e0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7ff57400ed20 tx=0x7ff5740091d0).stop 2020-11-20T12:18:21.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.385+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 shutdown_connections 2020-11-20T12:18:21.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.385+0000 7ff58134a700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff558056d20 0x7ff5580591e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.385+0000 7ff58134a700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff57c09ca90 0x7ff57c0a01e0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.385+0000 7ff58134a700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff57c065190 0x7ff57c099320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.385+0000 7ff58134a700 1 --2- 172.21.15.187:0/2019981884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c065f90 0x7ff57c09fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.385+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 >> 172.21.15.187:0/2019981884 conn(0x7ff57c0bab30 msgr2=0x7ff57c0aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:21.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.386+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 shutdown_connections 2020-11-20T12:18:21.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.386+0000 7ff58134a700 1 -- 172.21.15.187:0/2019981884 wait complete. 2020-11-20T12:18:21.428 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:21.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.656+0000 7f7ce933e700 1 Processor -- start 2020-11-20T12:18:21.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.657+0000 7f7ce933e700 1 -- start start 2020-11-20T12:18:21.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.658+0000 7f7ce933e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7ce40b1760 0x7f7ce4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:21.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.658+0000 7f7ce933e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce4150390 con 0x7f7ce4065680 2020-11-20T12:18:21.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.658+0000 7f7ce933e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce4150630 con 0x7f7ce40b1380 2020-11-20T12:18:21.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.658+0000 7f7ce933e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce41511e0 con 0x7f7ce40b1760 2020-11-20T12:18:21.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.659+0000 7f7ce3fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7ce40b1380 0x7f7ce4150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48586/0 (socket says 172.21.15.187:48586) 2020-11-20T12:18:21.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.659+0000 7f7ce3fff700 1 -- 172.21.15.187:0/389494638 learned_addr learned my addr 172.21.15.187:0/389494638 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:21.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.659+0000 7f7ce37fe700 1 --2- 172.21.15.187:0/389494638 >> v2:172.21.15.187:3300/0 conn(0x7f7ce40b1760 0x7f7ce4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:21.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.659+0000 7f7ce37fe700 1 -- 172.21.15.187:0/389494638 >> v1:172.21.15.187:6789/0 conn(0x7f7ce40b1380 legacy=0x7f7ce4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:21.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.659+0000 7f7ce37fe700 1 -- 172.21.15.187:0/389494638 >> v1:172.21.15.5:6789/0 conn(0x7f7ce4065680 legacy=0x7f7ce414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:21.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.659+0000 7f7ce2ffd700 1 -- 172.21.15.187:0/389494638 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36623783 0 0) 0x7f7ce4150630 con 0x7f7ce40b1380 2020-11-20T12:18:21.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.659+0000 7f7ce37fe700 1 -- 172.21.15.187:0/389494638 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce4068510 con 0x7f7ce40b1760 2020-11-20T12:18:21.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce37fe700 1 --2- 172.21.15.187:0/389494638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1760 0x7f7ce4065190 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7f7cd800b9a0 tx=0x7f7cd8006df0).ready entity=mon.1 client_cookie=d4a31b85ee3ef156 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:21.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce2ffd700 1 -- 172.21.15.187:0/389494638 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8009770 con 0x7f7ce40b1760 2020-11-20T12:18:21.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce2ffd700 1 -- 172.21.15.187:0/389494638 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd80098d0 con 0x7f7ce40b1760 2020-11-20T12:18:21.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce2ffd700 1 -- 172.21.15.187:0/389494638 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8009c50 con 0x7f7ce40b1760 2020-11-20T12:18:21.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce933e700 1 -- 172.21.15.187:0/389494638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1760 msgr2=0x7f7ce4065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:21.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce933e700 1 --2- 172.21.15.187:0/389494638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1760 0x7f7ce4065190 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7f7cd800b9a0 tx=0x7f7cd8006df0).stop 2020-11-20T12:18:21.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce933e700 1 -- 172.21.15.187:0/389494638 shutdown_connections 2020-11-20T12:18:21.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce933e700 1 --2- 172.21.15.187:0/389494638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1760 0x7f7ce4065190 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.660+0000 7f7ce933e700 1 -- 172.21.15.187:0/389494638 >> 172.21.15.187:0/389494638 conn(0x7f7ce40bab30 msgr2=0x7f7ce40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:21.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.661+0000 7f7ce933e700 1 -- 172.21.15.187:0/389494638 shutdown_connections 2020-11-20T12:18:21.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.661+0000 7f7ce933e700 1 -- 172.21.15.187:0/389494638 wait complete. 2020-11-20T12:18:21.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.662+0000 7f7ce933e700 1 Processor -- start 2020-11-20T12:18:21.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.663+0000 7f7ce933e700 1 -- start start 2020-11-20T12:18:21.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce933e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1380 0x7f7ce40b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:21.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce933e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ce409a670 0x7f7ce4142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:21.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce933e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ce4142970 0x7f7ce41431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:21.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce933e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce40b27d0 con 0x7f7ce409a670 2020-11-20T12:18:21.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce933e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce40b2a70 con 0x7f7ce40b1380 2020-11-20T12:18:21.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce933e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4142dd0 con 0x7f7ce4142970 2020-11-20T12:18:21.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1380 0x7f7ce40b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:21.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1380 0x7f7ce40b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:51986/0 (socket says 172.21.15.187:51986) 2020-11-20T12:18:21.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce3fff700 1 -- 172.21.15.187:0/370251672 learned_addr learned my addr 172.21.15.187:0/370251672 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:21.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.664+0000 7f7ce37fe700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ce409a670 0x7f7ce4142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:21.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce8b3d700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ce4142970 0x7f7ce41431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:21.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce3fff700 1 -- 172.21.15.187:0/370251672 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ce4142970 msgr2=0x7f7ce41431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:21.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce3fff700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ce4142970 0x7f7ce41431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce3fff700 1 -- 172.21.15.187:0/370251672 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ce409a670 msgr2=0x7f7ce4142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:21.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce3fff700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ce409a670 0x7f7ce4142480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce3fff700 1 -- 172.21.15.187:0/370251672 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce41498c0 con 0x7f7ce40b1380 2020-11-20T12:18:21.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce3fff700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1380 0x7f7ce40b2f60 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 rx=0x7f7cd400c760 tx=0x7f7cd4007ce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:21.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7cc3fff700 1 -- 172.21.15.187:0/370251672 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40072e0 con 0x7f7ce40b1380 2020-11-20T12:18:21.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7cc3fff700 1 -- 172.21.15.187:0/370251672 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd4007440 con 0x7f7ce40b1380 2020-11-20T12:18:21.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce41436e0 con 0x7f7ce40b1380 2020-11-20T12:18:21.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7cc3fff700 1 -- 172.21.15.187:0/370251672 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4005800 con 0x7f7ce40b1380 2020-11-20T12:18:21.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.665+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce4143c20 con 0x7f7ce40b1380 2020-11-20T12:18:21.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.667+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce4143db0 con 0x7f7ce40b1380 2020-11-20T12:18:21.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.667+0000 7f7cc3fff700 1 -- 172.21.15.187:0/370251672 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7cd4010070 con 0x7f7ce40b1380 2020-11-20T12:18:21.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.668+0000 7f7cc3fff700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cbc056d20 0x7f7cbc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:21.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.668+0000 7f7cc3fff700 1 -- 172.21.15.187:0/370251672 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7cd406f720 con 0x7f7ce40b1380 2020-11-20T12:18:21.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.668+0000 7f7ce37fe700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cbc056d20 0x7f7cbc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:21.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.669+0000 7f7ce37fe700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cbc056d20 0x7f7cbc0591e0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f7ce4067750 tx=0x7f7cd8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:21.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.672+0000 7f7cc3fff700 1 -- 172.21.15.187:0/370251672 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7cd4044bf0 con 0x7f7ce40b1380 2020-11-20T12:18:21.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.922+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ce40b2c00 con 0x7f7ce40b1380 2020-11-20T12:18:21.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.924+0000 7f7cc3fff700 1 -- 172.21.15.187:0/370251672 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2159 (secure 0 0 0) 0x7f7cd4048210 con 0x7f7ce40b1380 2020-11-20T12:18:21.926 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:21.927 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":117,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:21.379401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5900},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5900":{"gid":5900,"name":"b","rank":1,"incarnation":117,"state":"up:creating","state_seq":2,"addr":"172.21.15.5:6833/3549193191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3549193191},{"type":"v1","addr":"172.21.15.5:6833","nonce":3549193191}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:21.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cbc056d20 msgr2=0x7f7cbc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:21.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cbc056d20 0x7f7cbc0591e0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f7ce4067750 tx=0x7f7cd8015000).stop 2020-11-20T12:18:21.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1380 msgr2=0x7f7ce40b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:21.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1380 0x7f7ce40b2f60 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 rx=0x7f7cd400c760 tx=0x7f7cd4007ce0).stop 2020-11-20T12:18:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 shutdown_connections 2020-11-20T12:18:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7cbc056d20 0x7f7cbc0591e0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ce4142970 0x7f7ce41431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce40b1380 0x7f7ce40b2f60 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 --2- 172.21.15.187:0/370251672 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ce409a670 0x7f7ce4142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.926+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 >> 172.21.15.187:0/370251672 conn(0x7f7ce40bab30 msgr2=0x7f7ce40baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.927+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 shutdown_connections 2020-11-20T12:18:21.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:21.927+0000 7f7ce933e700 1 -- 172.21.15.187:0/370251672 wait complete. 2020-11-20T12:18:21.932 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 117 2020-11-20T12:18:21.952 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2020-11-20T12:18:23.200 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:18:23.210 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:18:24.768 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:18:24.765+0000 7f357cee0700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2020-11-20T12:18:26.954 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:27.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.206+0000 7fa69abca700 1 Processor -- start 2020-11-20T12:18:27.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.206+0000 7fa69abca700 1 -- start start 2020-11-20T12:18:27.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.207+0000 7fa69abca700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa6941484d0 0x7fa694061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.207+0000 7fa69abca700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa694066f70 0x7fa6940b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.207+0000 7fa69abca700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6940b1300 con 0x7fa6940622c0 2020-11-20T12:18:27.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.207+0000 7fa69abca700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa694067730 con 0x7fa6941484d0 2020-11-20T12:18:27.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.207+0000 7fa69abca700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6940b15a0 con 0x7fa694066f70 2020-11-20T12:18:27.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.208+0000 7fa69a3c9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa694066f70 0x7fa6940b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.208+0000 7fa69a3c9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa694066f70 0x7fa6940b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52000/0 (socket says 172.21.15.187:52000) 2020-11-20T12:18:27.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.208+0000 7fa69a3c9700 1 -- 172.21.15.187:0/3773716694 learned_addr learned my addr 172.21.15.187:0/3773716694 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:27.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.208+0000 7fa699bc8700 1 --2- 172.21.15.187:0/3773716694 >> v2:172.21.15.5:3300/0 conn(0x7fa6941484d0 0x7fa694061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa69a3c9700 1 -- 172.21.15.187:0/3773716694 >> v2:172.21.15.5:3300/0 conn(0x7fa6941484d0 msgr2=0x7fa694061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:27.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa69a3c9700 1 --2- 172.21.15.187:0/3773716694 >> v2:172.21.15.5:3300/0 conn(0x7fa6941484d0 0x7fa694061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa69a3c9700 1 -- 172.21.15.187:0/3773716694 >> v1:172.21.15.5:6789/0 conn(0x7fa6940622c0 legacy=0x7fa6940b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:27.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa69a3c9700 1 -- 172.21.15.187:0/3773716694 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940b1840 con 0x7fa694066f70 2020-11-20T12:18:27.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa699bc8700 1 --2- 172.21.15.187:0/3773716694 >> v2:172.21.15.5:3300/0 conn(0x7fa6941484d0 0x7fa694061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:27.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa69a3c9700 1 --2- 172.21.15.187:0/3773716694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa694066f70 0x7fa6940b5d50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 rx=0x7fa694045000 tx=0x7fa68c009290).ready entity=mon.1 client_cookie=72e592a330678a37 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:27.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa698bc6700 1 -- 172.21.15.187:0/3773716694 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c00bb80 con 0x7fa694066f70 2020-11-20T12:18:27.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa698bc6700 1 -- 172.21.15.187:0/3773716694 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c00bce0 con 0x7fa694066f70 2020-11-20T12:18:27.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.209+0000 7fa698bc6700 1 -- 172.21.15.187:0/3773716694 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c011610 con 0x7fa694066f70 2020-11-20T12:18:27.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.210+0000 7fa69abca700 1 -- 172.21.15.187:0/3773716694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa694066f70 msgr2=0x7fa6940b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.210+0000 7fa69abca700 1 --2- 172.21.15.187:0/3773716694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa694066f70 0x7fa6940b5d50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 rx=0x7fa694045000 tx=0x7fa68c009290).stop 2020-11-20T12:18:27.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.210+0000 7fa69abca700 1 -- 172.21.15.187:0/3773716694 shutdown_connections 2020-11-20T12:18:27.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.210+0000 7fa69abca700 1 --2- 172.21.15.187:0/3773716694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa694066f70 0x7fa6940b5d50 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.210+0000 7fa69abca700 1 --2- 172.21.15.187:0/3773716694 >> v2:172.21.15.5:3300/0 conn(0x7fa6941484d0 0x7fa694061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.210+0000 7fa69abca700 1 -- 172.21.15.187:0/3773716694 >> 172.21.15.187:0/3773716694 conn(0x7fa6940bab30 msgr2=0x7fa6940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:27.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.210+0000 7fa69abca700 1 -- 172.21.15.187:0/3773716694 shutdown_connections 2020-11-20T12:18:27.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.210+0000 7fa69abca700 1 -- 172.21.15.187:0/3773716694 wait complete. 2020-11-20T12:18:27.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.212+0000 7fa69abca700 1 Processor -- start 2020-11-20T12:18:27.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.212+0000 7fa69abca700 1 -- start start 2020-11-20T12:18:27.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa69abca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6940622c0 0x7fa69409c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa69abca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa694066f70 0x7fa69409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa69abca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6941484d0 0x7fa6940b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa69abca700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69409b040 con 0x7fa6941484d0 2020-11-20T12:18:27.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa69abca700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69409ef80 con 0x7fa6940622c0 2020-11-20T12:18:27.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa69abca700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69409f220 con 0x7fa694066f70 2020-11-20T12:18:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa699bc8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6940622c0 0x7fa69409c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa699bc8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6940622c0 0x7fa69409c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52002/0 (socket says 172.21.15.187:52002) 2020-11-20T12:18:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa699bc8700 1 -- 172.21.15.187:0/3049908520 learned_addr learned my addr 172.21.15.187:0/3049908520 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa6993c7700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa694066f70 0x7fa69409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa699bc8700 1 -- 172.21.15.187:0/3049908520 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa694066f70 msgr2=0x7fa69409b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.213+0000 7fa69a3c9700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6941484d0 0x7fa6940b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa699bc8700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa694066f70 0x7fa69409b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa699bc8700 1 -- 172.21.15.187:0/3049908520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6941484d0 msgr2=0x7fa6940b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa699bc8700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6941484d0 0x7fa6940b2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa699bc8700 1 -- 172.21.15.187:0/3049908520 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940a0770 con 0x7fa6940622c0 2020-11-20T12:18:27.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa699bc8700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6940622c0 0x7fa69409c830 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7fa68400db60 tx=0x7fa684008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:27.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa691ffb700 1 -- 172.21.15.187:0/3049908520 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684009820 con 0x7fa6940622c0 2020-11-20T12:18:27.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa691ffb700 1 -- 172.21.15.187:0/3049908520 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa684009980 con 0x7fa6940622c0 2020-11-20T12:18:27.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6940b32a0 con 0x7fa6940622c0 2020-11-20T12:18:27.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa691ffb700 1 -- 172.21.15.187:0/3049908520 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684009d40 con 0x7fa6940622c0 2020-11-20T12:18:27.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.214+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69409bb60 con 0x7fa6940622c0 2020-11-20T12:18:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.216+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69414dd90 con 0x7fa6940622c0 2020-11-20T12:18:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.216+0000 7fa691ffb700 1 -- 172.21.15.187:0/3049908520 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa684013690 con 0x7fa6940622c0 2020-11-20T12:18:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.217+0000 7fa691ffb700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa66c056d20 0x7fa66c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.217+0000 7fa691ffb700 1 -- 172.21.15.187:0/3049908520 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa684070230 con 0x7fa6940622c0 2020-11-20T12:18:27.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.217+0000 7fa6993c7700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa66c056d20 0x7fa66c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.217+0000 7fa6993c7700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa66c056d20 0x7fa66c0591e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7fa6940b3fd0 tx=0x7fa688009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:27.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.221+0000 7fa691ffb700 1 -- 172.21.15.187:0/3049908520 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa684045620 con 0x7fa6940622c0 2020-11-20T12:18:27.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.466+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa694061a20 con 0x7fa6940622c0 2020-11-20T12:18:27.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.467+0000 7fa691ffb700 1 -- 172.21.15.187:0/3049908520 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2157 (secure 0 0 0) 0x7fa684048c40 con 0x7fa6940622c0 2020-11-20T12:18:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:27.471 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":118,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:22.384267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5900},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5900":{"gid":5900,"name":"b","rank":1,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/3549193191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3549193191},{"type":"v1","addr":"172.21.15.5:6833","nonce":3549193191}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:27.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa66c056d20 msgr2=0x7fa66c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa66c056d20 0x7fa66c0591e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7fa6940b3fd0 tx=0x7fa688009040).stop 2020-11-20T12:18:27.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6940622c0 msgr2=0x7fa69409c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6940622c0 0x7fa69409c830 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7fa68400db60 tx=0x7fa684008040).stop 2020-11-20T12:18:27.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 shutdown_connections 2020-11-20T12:18:27.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa66c056d20 0x7fa66c0591e0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6941484d0 0x7fa6940b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6940622c0 0x7fa69409c830 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 --2- 172.21.15.187:0/3049908520 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa694066f70 0x7fa69409b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.469+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 >> 172.21.15.187:0/3049908520 conn(0x7fa6940bab30 msgr2=0x7fa694144e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:27.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.470+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 shutdown_connections 2020-11-20T12:18:27.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.470+0000 7fa69abca700 1 -- 172.21.15.187:0/3049908520 wait complete. 2020-11-20T12:18:27.476 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 118 2020-11-20T12:18:27.497 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:27.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.726+0000 7f71c8804700 1 Processor -- start 2020-11-20T12:18:27.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.726+0000 7f71c8804700 1 -- start start 2020-11-20T12:18:27.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.727+0000 7f71c8804700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f71c41484d0 0x7f71c4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.727+0000 7f71c8804700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f71c40622c0 0x7f71c40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.727+0000 7f71c8804700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c4066f70 0x7f71c40b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.727+0000 7f71c8804700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c4067730 con 0x7f71c40622c0 2020-11-20T12:18:27.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.727+0000 7f71c8804700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c40b1300 con 0x7f71c41484d0 2020-11-20T12:18:27.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.727+0000 7f71c8804700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40b15a0 con 0x7f71c4066f70 2020-11-20T12:18:27.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.728+0000 7f71c2d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f71c41484d0 0x7f71c4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.728+0000 7f71c2d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f71c41484d0 0x7f71c4061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52012/0 (socket says 172.21.15.187:52012) 2020-11-20T12:18:27.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.728+0000 7f71c2d9d700 1 -- 172.21.15.187:0/3454677921 learned_addr learned my addr 172.21.15.187:0/3454677921 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:27.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.728+0000 7f71bbfff700 1 --2- 172.21.15.187:0/3454677921 >> v2:172.21.15.5:3300/0 conn(0x7f71c40622c0 0x7f71c40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.728+0000 7f71c359e700 1 --2- 172.21.15.187:0/3454677921 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c4066f70 0x7f71c40b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.729+0000 7f71c2d9d700 1 -- 172.21.15.187:0/3454677921 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c4066f70 msgr2=0x7f71c40b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:27.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.729+0000 7f71c2d9d700 1 --2- 172.21.15.187:0/3454677921 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c4066f70 0x7f71c40b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.729+0000 7f71c2d9d700 1 -- 172.21.15.187:0/3454677921 >> v2:172.21.15.5:3300/0 conn(0x7f71c40622c0 msgr2=0x7f71c40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:27.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.729+0000 7f71c2d9d700 1 --2- 172.21.15.187:0/3454677921 >> v2:172.21.15.5:3300/0 conn(0x7f71c40622c0 0x7f71c40b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.729+0000 7f71c2d9d700 1 -- 172.21.15.187:0/3454677921 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c40b1840 con 0x7f71c41484d0 2020-11-20T12:18:27.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.729+0000 7f71c2d9d700 1 --2- 172.21.15.187:0/3454677921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c41484d0 0x7f71c4061dd0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7f71c40b85e0 tx=0x7f71b4006eb0).ready entity=mon.1 client_cookie=f6334da5eb9432a9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:27.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c259c700 1 -- 172.21.15.187:0/3454677921 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b40098d0 con 0x7f71c41484d0 2020-11-20T12:18:27.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c259c700 1 -- 172.21.15.187:0/3454677921 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b4009a30 con 0x7f71c41484d0 2020-11-20T12:18:27.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c259c700 1 -- 172.21.15.187:0/3454677921 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4009db0 con 0x7f71c41484d0 2020-11-20T12:18:27.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c8804700 1 -- 172.21.15.187:0/3454677921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c41484d0 msgr2=0x7f71c4061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c8804700 1 --2- 172.21.15.187:0/3454677921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c41484d0 0x7f71c4061dd0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7f71c40b85e0 tx=0x7f71b4006eb0).stop 2020-11-20T12:18:27.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c8804700 1 -- 172.21.15.187:0/3454677921 shutdown_connections 2020-11-20T12:18:27.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c8804700 1 --2- 172.21.15.187:0/3454677921 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c4066f70 0x7f71c40b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c8804700 1 --2- 172.21.15.187:0/3454677921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c41484d0 0x7f71c4061dd0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c8804700 1 --2- 172.21.15.187:0/3454677921 >> v2:172.21.15.5:3300/0 conn(0x7f71c40622c0 0x7f71c40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.730+0000 7f71c8804700 1 -- 172.21.15.187:0/3454677921 >> 172.21.15.187:0/3454677921 conn(0x7f71c40bab30 msgr2=0x7f71c40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:27.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.731+0000 7f71c8804700 1 -- 172.21.15.187:0/3454677921 shutdown_connections 2020-11-20T12:18:27.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.731+0000 7f71c8804700 1 -- 172.21.15.187:0/3454677921 wait complete. 2020-11-20T12:18:27.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.732+0000 7f71c8804700 1 Processor -- start 2020-11-20T12:18:27.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.732+0000 7f71c8804700 1 -- start start 2020-11-20T12:18:27.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.733+0000 7f71c8804700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c40622c0 0x7f71c409d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.733+0000 7f71c8804700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c4066f70 0x7f71c409fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.733+0000 7f71c8804700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71c41484d0 0x7f71c40b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.733+0000 7f71c8804700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c409a880 con 0x7f71c41484d0 2020-11-20T12:18:27.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.733+0000 7f71c8804700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c409ab20 con 0x7f71c4066f70 2020-11-20T12:18:27.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.733+0000 7f71c8804700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c409adc0 con 0x7f71c40622c0 2020-11-20T12:18:27.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71bbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c4066f70 0x7f71c409fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71bbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c4066f70 0x7f71c409fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52018/0 (socket says 172.21.15.187:52018) 2020-11-20T12:18:27.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71bbfff700 1 -- 172.21.15.187:0/2264417806 learned_addr learned my addr 172.21.15.187:0/2264417806 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:27.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71c2d9d700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c40622c0 0x7f71c409d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71c359e700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71c41484d0 0x7f71c40b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71bbfff700 1 -- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c40622c0 msgr2=0x7f71c409d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71bbfff700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c40622c0 0x7f71c409d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71bbfff700 1 -- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71c41484d0 msgr2=0x7f71c40b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71bbfff700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71c41484d0 0x7f71c40b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71bbfff700 1 -- 172.21.15.187:0/2264417806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c40b4580 con 0x7f71c4066f70 2020-11-20T12:18:27.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71c2d9d700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c40622c0 0x7f71c409d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:27.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.734+0000 7f71c359e700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71c41484d0 0x7f71c40b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:27.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.735+0000 7f71bbfff700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c4066f70 0x7f71c409fed0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 rx=0x7f71ac00db60 tx=0x7f71ac008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:27.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.735+0000 7f71baffd700 1 -- 172.21.15.187:0/2264417806 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac009790 con 0x7f71c4066f70 2020-11-20T12:18:27.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.735+0000 7f71baffd700 1 -- 172.21.15.187:0/2264417806 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ac0098f0 con 0x7f71c4066f70 2020-11-20T12:18:27.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.735+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c41580d0 con 0x7f71c4066f70 2020-11-20T12:18:27.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.735+0000 7f71baffd700 1 -- 172.21.15.187:0/2264417806 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac009cb0 con 0x7f71c4066f70 2020-11-20T12:18:27.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.735+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c4158610 con 0x7f71c4066f70 2020-11-20T12:18:27.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.736+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c40a03c0 con 0x7f71c4066f70 2020-11-20T12:18:27.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.737+0000 7f71baffd700 1 -- 172.21.15.187:0/2264417806 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f71ac012630 con 0x7f71c4066f70 2020-11-20T12:18:27.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.738+0000 7f71baffd700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71a0056d20 0x7f71a00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:27.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.738+0000 7f71baffd700 1 -- 172.21.15.187:0/2264417806 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f71ac0702c0 con 0x7f71c4066f70 2020-11-20T12:18:27.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.738+0000 7f71c2d9d700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71a0056d20 0x7f71a00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:27.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.738+0000 7f71c2d9d700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71a0056d20 0x7f71a00591e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f71b400b1b0 tx=0x7f71b4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:27.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.741+0000 7f71baffd700 1 -- 172.21.15.187:0/2264417806 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f71ac045710 con 0x7f71c4066f70 2020-11-20T12:18:27.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.986+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71c4061a20 con 0x7f71c4066f70 2020-11-20T12:18:27.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.987+0000 7f71baffd700 1 -- 172.21.15.187:0/2264417806 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2157 (secure 0 0 0) 0x7f71ac048d30 con 0x7f71c4066f70 2020-11-20T12:18:27.993 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:27.994 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":118,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:22.384267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5900},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5900":{"gid":5900,"name":"b","rank":1,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/3549193191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3549193191},{"type":"v1","addr":"172.21.15.5:6833","nonce":3549193191}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:27.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71a0056d20 msgr2=0x7f71a00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71a0056d20 0x7f71a00591e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f71b400b1b0 tx=0x7f71b4015000).stop 2020-11-20T12:18:27.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c4066f70 msgr2=0x7f71c409fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:27.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c4066f70 0x7f71c409fed0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 rx=0x7f71ac00db60 tx=0x7f71ac008040).stop 2020-11-20T12:18:27.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 shutdown_connections 2020-11-20T12:18:27.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71a0056d20 0x7f71a00591e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71c41484d0 0x7f71c40b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71c40622c0 0x7f71c409d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 --2- 172.21.15.187:0/2264417806 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71c4066f70 0x7f71c409fed0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 >> 172.21.15.187:0/2264417806 conn(0x7f71c40bab30 msgr2=0x7f71c414cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 shutdown_connections 2020-11-20T12:18:27.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:27.989+0000 7f71c8804700 1 -- 172.21.15.187:0/2264417806 wait complete. 2020-11-20T12:18:27.999 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 118 2020-11-20T12:18:28.015 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and b 2020-11-20T12:18:28.016 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:18:28.026 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:18:28.192 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:18:28.192 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:18:28.192 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:18:28.193 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:18:28.193 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.193 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:18:28.193 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.193 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:18:28.194 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.194 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:18:28.194 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:18:28.194 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.195 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:18:28.195 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:18:28.195 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.195 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:18:28.195 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:18:28.196 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.196 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:18:28.196 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:18:28.196 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.196 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:18:28.197 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.197 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:18:28.197 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:18:28.197 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:18:28.197 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:18:28.198 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:18:28.198 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:18:28.198 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:18:28.198 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:18:28.198 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:18:28.199 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:18:28.199 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.199 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:18:28.199 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.199 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:18:28.200 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.200 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:18:28.200 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.200 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:18:28.201 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:18:28.201 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:18:28.201 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:18:28.201 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:18:28.202 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:18:28.202 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.202 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:18:28.202 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:18:28.202 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.203 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:18:28.203 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:18:28.203 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.203 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:18:28.203 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:18:28.204 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.204 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:18:28.204 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:18:28.204 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:18:28.204 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.205 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:18:28.205 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:18:28.205 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.205 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:18:28.205 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.206 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:18:28.206 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.206 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:18:28.206 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.207 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:18:28.207 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:18:28.207 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.207 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:18:28.207 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:18:28.208 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:18:28.208 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:18:28.208 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:18:28.208 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:18:28.209 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:18:28.209 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:18:28.209 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:18:28.209 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:18:28.209 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:18:28.210 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:18:28.210 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:18:28.210 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:18:28.210 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:18:28.210 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:18:28.211 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:18:28.211 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:18:28.211 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:18:28.211 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:18:28.211 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.212 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:18:28.212 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:18:28.212 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:18:28.212 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:18:28.212 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:18:28.213 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:18:28.213 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:18:28.213 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:18:28.213 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.213 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:18:28.214 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:18:28.214 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:18:28.214 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:18:28.214 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:18:28.214 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:18:28.215 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:18:28.215 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:18:28.215 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:18:28.215 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:18:28.215 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:18:28.216 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:18:28.216 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:28.216 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:18:28.216 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:18:28.216 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:18:28.217 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:18:28.217 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:18:28.217 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:18:28.217 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:18:28.217 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:18:28.218 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:18:28.218 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:18:28.218 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:18:28.218 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:18:28.218 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:18:28.219 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:18:28.219 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:18:28.219 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:18:28.219 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:18:28.219 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 0 2020-11-20T12:18:28.220 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:18:28.220 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:18:28.220 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:18:28.220 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:18:28.220 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:18:28.221 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:18:28.221 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:18:28.221 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:18:28.221 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:18:28.221 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:18:28.222 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:18:28.222 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:18:28.222 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:18:28.222 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:18:28.222 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:18:28.228 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:18:28.229 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:18:28.229 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:18:28.229 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:18:28.230 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.230 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:18:28.230 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.230 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:18:28.230 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.231 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:18:28.231 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:18:28.231 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.231 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:18:28.232 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:18:28.232 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.232 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:18:28.232 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:18:28.232 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.233 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:18:28.233 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:18:28.233 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.233 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:18:28.233 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.234 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:18:28.234 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:18:28.234 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:18:28.234 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:18:28.235 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:18:28.235 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:18:28.235 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:18:28.235 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:18:28.236 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:18:28.236 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:18:28.236 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.236 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:18:28.237 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.237 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:18:28.237 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.237 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:18:28.237 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.238 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:18:28.238 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:18:28.238 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:18:28.239 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:18:28.239 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:18:28.240 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:18:28.240 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.240 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:18:28.241 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:18:28.241 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.242 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:18:28.242 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:18:28.242 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.243 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:18:28.243 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:18:28.244 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.244 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:18:28.244 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:18:28.245 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:18:28.245 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.246 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:18:28.246 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:18:28.246 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.247 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:18:28.247 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.248 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:18:28.248 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.248 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:18:28.248 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.249 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:18:28.249 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:18:28.249 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.250 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:18:28.250 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:18:28.250 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:18:28.250 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:18:28.251 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:18:28.251 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:18:28.251 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:18:28.251 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:18:28.251 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:18:28.252 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:18:28.252 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:18:28.252 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:18:28.252 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:18:28.252 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:18:28.253 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:18:28.253 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:18:28.253 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:18:28.253 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:18:28.254 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:18:28.254 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:18:28.254 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.254 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:18:28.254 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:18:28.255 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:18:28.255 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:18:28.255 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:18:28.255 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:18:28.255 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:18:28.256 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:18:28.256 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.256 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:18:28.256 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:18:28.256 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:18:28.257 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:18:28.257 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:18:28.257 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:18:28.258 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:18:28.258 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:18:28.258 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:18:28.258 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:18:28.258 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:18:28.259 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:18:28.259 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:18:28.259 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:18:28.259 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:18:28.259 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:18:28.260 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:18:28.260 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:18:28.260 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:18:28.260 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:18:28.260 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:18:28.261 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:18:28.261 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:18:28.261 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:18:28.261 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:18:28.261 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:18:28.262 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:18:28.262 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:18:28.262 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:18:28.263 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:18:28.263 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 0 2020-11-20T12:18:28.263 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:18:28.264 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:18:28.264 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:18:28.265 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:18:28.265 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:18:28.265 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:18:28.266 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:18:28.266 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:18:28.266 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:18:28.266 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:18:28.267 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:18:28.267 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:18:28.267 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:18:28.268 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:18:28.268 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:18:28.285 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:18:28.285 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:18:28.304 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:18:28.305 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2020-11-20T12:18:28.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.555+0000 7f037adf4700 1 Processor -- start 2020-11-20T12:18:28.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.555+0000 7f037adf4700 1 -- start start 2020-11-20T12:18:28.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.557+0000 7f037adf4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374099fb0 0x7f0374148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:28.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.557+0000 7f037adf4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03740683b0 0x7f0374145640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:28.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.557+0000 7f037adf4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0374144f00 con 0x7f0374065cc0 2020-11-20T12:18:28.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.557+0000 7f037adf4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037409afb0 con 0x7f03740683b0 2020-11-20T12:18:28.561 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.557+0000 7f037adf4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0374144d10 con 0x7f0374099fb0 2020-11-20T12:18:28.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.558+0000 7f03795f1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03740683b0 0x7f0374145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:28.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.558+0000 7f0379df2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374099fb0 0x7f0374148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:28.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.558+0000 7f03795f1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03740683b0 0x7f0374145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:44288/0 (socket says 172.21.15.187:44288) 2020-11-20T12:18:28.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.558+0000 7f03795f1700 1 -- 172.21.15.187:0/4145898836 learned_addr learned my addr 172.21.15.187:0/4145898836 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:28.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.558+0000 7f037a5f3700 1 --1- 172.21.15.187:0/4145898836 >> v1:172.21.15.5:6789/0 conn(0x7f0374065cc0 0x7f037409c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:42140/0 (socket says 172.21.15.187:42140) 2020-11-20T12:18:28.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.559+0000 7f0379df2700 1 -- 172.21.15.187:0/4145898836 >> v2:172.21.15.5:3300/0 conn(0x7f03740683b0 msgr2=0x7f0374145640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:28.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.559+0000 7f0379df2700 1 --2- 172.21.15.187:0/4145898836 >> v2:172.21.15.5:3300/0 conn(0x7f03740683b0 0x7f0374145640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.559+0000 7f0379df2700 1 -- 172.21.15.187:0/4145898836 >> v1:172.21.15.5:6789/0 conn(0x7f0374065cc0 legacy=0x7f037409c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:28.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.559+0000 7f0379df2700 1 -- 172.21.15.187:0/4145898836 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03741451a0 con 0x7f0374099fb0 2020-11-20T12:18:28.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.559+0000 7f03795f1700 1 --2- 172.21.15.187:0/4145898836 >> v2:172.21.15.5:3300/0 conn(0x7f03740683b0 0x7f0374145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:18:28.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.560+0000 7f0379df2700 1 --2- 172.21.15.187:0/4145898836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374099fb0 0x7f0374148cf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f036800b9a0 tx=0x7f0368006df0).ready entity=mon.2 client_cookie=92192d0c1d568adf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:28.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f0378df0700 1 -- 172.21.15.187:0/4145898836 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368009840 con 0x7f0374099fb0 2020-11-20T12:18:28.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f0378df0700 1 -- 172.21.15.187:0/4145898836 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03680099a0 con 0x7f0374099fb0 2020-11-20T12:18:28.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f0378df0700 1 -- 172.21.15.187:0/4145898836 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368009d20 con 0x7f0374099fb0 2020-11-20T12:18:28.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f037adf4700 1 -- 172.21.15.187:0/4145898836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374099fb0 msgr2=0x7f0374148cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:28.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f037adf4700 1 --2- 172.21.15.187:0/4145898836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374099fb0 0x7f0374148cf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f036800b9a0 tx=0x7f0368006df0).stop 2020-11-20T12:18:28.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f037adf4700 1 -- 172.21.15.187:0/4145898836 shutdown_connections 2020-11-20T12:18:28.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f037adf4700 1 --2- 172.21.15.187:0/4145898836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374099fb0 0x7f0374148cf0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f037adf4700 1 --2- 172.21.15.187:0/4145898836 >> v2:172.21.15.5:3300/0 conn(0x7f03740683b0 0x7f0374145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f037adf4700 1 -- 172.21.15.187:0/4145898836 >> 172.21.15.187:0/4145898836 conn(0x7f03740babc0 msgr2=0x7f03740bafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:28.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.561+0000 7f037adf4700 1 -- 172.21.15.187:0/4145898836 shutdown_connections 2020-11-20T12:18:28.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.562+0000 7f037adf4700 1 -- 172.21.15.187:0/4145898836 wait complete. 2020-11-20T12:18:28.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.563+0000 7f037adf4700 1 Processor -- start 2020-11-20T12:18:28.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.563+0000 7f037adf4700 1 -- start start 2020-11-20T12:18:28.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037adf4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374065cc0 0x7f0374138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:28.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037adf4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03740683b0 0x7f037409c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:28.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037adf4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037409fe20 0x7f0374136e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:28.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037adf4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037413a9e0 con 0x7f03740683b0 2020-11-20T12:18:28.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037adf4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0374137c50 con 0x7f037409fe20 2020-11-20T12:18:28.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037adf4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037413abd0 con 0x7f0374065cc0 2020-11-20T12:18:28.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037a5f3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037409fe20 0x7f0374136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:28.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037a5f3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037409fe20 0x7f0374136e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52034/0 (socket says 172.21.15.187:52034) 2020-11-20T12:18:28.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.564+0000 7f037a5f3700 1 -- 172.21.15.187:0/3643269050 learned_addr learned my addr 172.21.15.187:0/3643269050 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:28.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f0379df2700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374065cc0 0x7f0374138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:28.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f03795f1700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03740683b0 0x7f037409c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:28.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f037a5f3700 1 -- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374065cc0 msgr2=0x7f0374138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:28.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f037a5f3700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374065cc0 0x7f0374138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f037a5f3700 1 -- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03740683b0 msgr2=0x7f037409c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:28.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f037a5f3700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03740683b0 0x7f037409c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f037a5f3700 1 -- 172.21.15.187:0/3643269050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037409c9d0 con 0x7f037409fe20 2020-11-20T12:18:28.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f03795f1700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03740683b0 0x7f037409c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:28.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f037a5f3700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037409fe20 0x7f0374136e40 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f036c00ca60 tx=0x7f036c007f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:28.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f0365ffb700 1 -- 172.21.15.187:0/3643269050 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c005ac0 con 0x7f037409fe20 2020-11-20T12:18:28.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f0365ffb700 1 -- 172.21.15.187:0/3643269050 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036c005c20 con 0x7f037409fe20 2020-11-20T12:18:28.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03740ba280 con 0x7f037409fe20 2020-11-20T12:18:28.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.565+0000 7f0365ffb700 1 -- 172.21.15.187:0/3643269050 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c007110 con 0x7f037409fe20 2020-11-20T12:18:28.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.566+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03741374f0 con 0x7f037409fe20 2020-11-20T12:18:28.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.567+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0374145b30 con 0x7f037409fe20 2020-11-20T12:18:28.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.567+0000 7f0365ffb700 1 -- 172.21.15.187:0/3643269050 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f036c010030 con 0x7f037409fe20 2020-11-20T12:18:28.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.568+0000 7f0365ffb700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f034c056d50 0x7f034c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:28.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.568+0000 7f0365ffb700 1 -- 172.21.15.187:0/3643269050 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f036c06ff30 con 0x7f037409fe20 2020-11-20T12:18:28.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.568+0000 7f0379df2700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f034c056d50 0x7f034c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:28.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.569+0000 7f0379df2700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f034c056d50 0x7f034c059210 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7f037409b1d0 tx=0x7f0368015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:28.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.572+0000 7f0365ffb700 1 -- 172.21.15.187:0/3643269050 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f036c045380 con 0x7f037409fe20 2020-11-20T12:18:28.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.831+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f0374142650 con 0x7f037409fe20 2020-11-20T12:18:28.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.855+0000 7f0365ffb700 1 -- 172.21.15.187:0/3643269050 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5900 v119) v1 ==== 95+0+0 (secure 0 0 0) 0x7f036c0489a0 con 0x7f037409fe20 2020-11-20T12:18:28.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.857+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f034c056d50 msgr2=0x7f034c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:28.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.857+0000 7f037adf4700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f034c056d50 0x7f034c059210 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7f037409b1d0 tx=0x7f0368015000).stop 2020-11-20T12:18:28.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.857+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037409fe20 msgr2=0x7f0374136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:28.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.857+0000 7f037adf4700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037409fe20 0x7f0374136e40 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f036c00ca60 tx=0x7f036c007f40).stop 2020-11-20T12:18:28.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.858+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 shutdown_connections 2020-11-20T12:18:28.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.858+0000 7f037adf4700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f034c056d50 0x7f034c059210 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.858+0000 7f037adf4700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037409fe20 0x7f0374136e40 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.858+0000 7f037adf4700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0374065cc0 0x7f0374138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.858+0000 7f037adf4700 1 --2- 172.21.15.187:0/3643269050 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03740683b0 0x7f037409c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.858+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 >> 172.21.15.187:0/3643269050 conn(0x7f03740babc0 msgr2=0x7f0374144590 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.858+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 shutdown_connections 2020-11-20T12:18:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:28.858+0000 7f037adf4700 1 -- 172.21.15.187:0/3643269050 wait complete. 2020-11-20T12:18:28.867 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 5900 2020-11-20T12:18:28.887 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:18:28.887 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:18:28.897 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:18:28.898 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:29.094 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:18:29.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.136+0000 7fc0c4322700 1 Processor -- start 2020-11-20T12:18:29.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.136+0000 7fc0c4322700 1 -- start start 2020-11-20T12:18:29.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.137+0000 7fc0c4322700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc0bc062c50 0x7fc0bc064160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:29.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.137+0000 7fc0c4322700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc0b2460 0x7fc0bc0b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:29.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.137+0000 7fc0c4322700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc065090 con 0x7fc0bc063030 2020-11-20T12:18:29.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.137+0000 7fc0c4322700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0bc064a70 con 0x7fc0bc062c50 2020-11-20T12:18:29.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.137+0000 7fc0c4322700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc064d10 con 0x7fc0bc0b2460 2020-11-20T12:18:29.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.138+0000 7fc0c28bf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc0b2460 0x7fc0bc0b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:29.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.138+0000 7fc0c28bf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc0b2460 0x7fc0bc0b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:52926/0 (socket says 172.21.15.187:52926) 2020-11-20T12:18:29.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.138+0000 7fc0c28bf700 1 -- 172.21.15.187:0/742801522 learned_addr learned my addr 172.21.15.187:0/742801522 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:29.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.138+0000 7fc0c20be700 1 --2- 172.21.15.187:0/742801522 >> v2:172.21.15.5:3300/0 conn(0x7fc0bc062c50 0x7fc0bc064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:29.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c28bf700 1 -- 172.21.15.187:0/742801522 >> v2:172.21.15.5:3300/0 conn(0x7fc0bc062c50 msgr2=0x7fc0bc064160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:29.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c28bf700 1 --2- 172.21.15.187:0/742801522 >> v2:172.21.15.5:3300/0 conn(0x7fc0bc062c50 0x7fc0bc064160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c28bf700 1 -- 172.21.15.187:0/742801522 >> v1:172.21.15.5:6789/0 conn(0x7fc0bc063030 legacy=0x7fc0bc0678b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:29.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c10bc700 1 -- 172.21.15.187:0/742801522 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043262311 0 0) 0x7fc0bc065090 con 0x7fc0bc063030 2020-11-20T12:18:29.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c28bf700 1 -- 172.21.15.187:0/742801522 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc0617e0 con 0x7fc0bc0b2460 2020-11-20T12:18:29.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c20be700 1 --2- 172.21.15.187:0/742801522 >> v2:172.21.15.5:3300/0 conn(0x7fc0bc062c50 0x7fc0bc064160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:18:29.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c28bf700 1 --2- 172.21.15.187:0/742801522 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc0b2460 0x7fc0bc0b28f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc0b4009d50 tx=0x7fc0b4009290).ready entity=mon.2 client_cookie=7e5c3161b3ed3630 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:29.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c10bc700 1 -- 172.21.15.187:0/742801522 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b400bbc0 con 0x7fc0bc0b2460 2020-11-20T12:18:29.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c10bc700 1 -- 172.21.15.187:0/742801522 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b400bd20 con 0x7fc0bc0b2460 2020-11-20T12:18:29.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.139+0000 7fc0c10bc700 1 -- 172.21.15.187:0/742801522 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4011610 con 0x7fc0bc0b2460 2020-11-20T12:18:29.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.140+0000 7fc0c4322700 1 -- 172.21.15.187:0/742801522 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc0b2460 msgr2=0x7fc0bc0b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:29.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.140+0000 7fc0c4322700 1 --2- 172.21.15.187:0/742801522 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc0b2460 0x7fc0bc0b28f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc0b4009d50 tx=0x7fc0b4009290).stop 2020-11-20T12:18:29.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.140+0000 7fc0c4322700 1 -- 172.21.15.187:0/742801522 shutdown_connections 2020-11-20T12:18:29.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.140+0000 7fc0c4322700 1 --2- 172.21.15.187:0/742801522 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc0b2460 0x7fc0bc0b28f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.140+0000 7fc0c4322700 1 --2- 172.21.15.187:0/742801522 >> v2:172.21.15.5:3300/0 conn(0x7fc0bc062c50 0x7fc0bc064160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.140+0000 7fc0c4322700 1 -- 172.21.15.187:0/742801522 >> 172.21.15.187:0/742801522 conn(0x7fc0bc0bcfb0 msgr2=0x7fc0bc0bd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:29.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.141+0000 7fc0c4322700 1 -- 172.21.15.187:0/742801522 shutdown_connections 2020-11-20T12:18:29.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.141+0000 7fc0c4322700 1 -- 172.21.15.187:0/742801522 wait complete. 2020-11-20T12:18:29.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.143+0000 7fc0c4322700 1 Processor -- start 2020-11-20T12:18:29.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.143+0000 7fc0c4322700 1 -- start start 2020-11-20T12:18:29.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c4322700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0bc062c50 0x7fc0bc15b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:29.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c4322700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc063030 0x7fc0bc0a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:29.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c4322700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0bc0b2460 0x7fc0bc0b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:29.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c4322700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc15bc70 con 0x7fc0bc062c50 2020-11-20T12:18:29.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c4322700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc15be90 con 0x7fc0bc0b2460 2020-11-20T12:18:29.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c4322700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc15c130 con 0x7fc0bc063030 2020-11-20T12:18:29.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c28bf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0bc0b2460 0x7fc0bc0b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:29.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c28bf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0bc0b2460 0x7fc0bc0b4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52048/0 (socket says 172.21.15.187:52048) 2020-11-20T12:18:29.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.144+0000 7fc0c28bf700 1 -- 172.21.15.187:0/1721675953 learned_addr learned my addr 172.21.15.187:0/1721675953 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:29.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c20be700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0bc062c50 0x7fc0bc15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:29.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c18bd700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc063030 0x7fc0bc0a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:29.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c28bf700 1 -- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc063030 msgr2=0x7fc0bc0a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:29.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c28bf700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc063030 0x7fc0bc0a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c28bf700 1 -- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0bc062c50 msgr2=0x7fc0bc15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:29.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c28bf700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0bc062c50 0x7fc0bc15b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c28bf700 1 -- 172.21.15.187:0/1721675953 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc15c3d0 con 0x7fc0bc0b2460 2020-11-20T12:18:29.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c18bd700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc063030 0x7fc0bc0a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:29.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0c28bf700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0bc0b2460 0x7fc0bc0b4770 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 rx=0x7fc0bc063d60 tx=0x7fc0b4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:29.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0aa7fc700 1 -- 172.21.15.187:0/1721675953 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b400be90 con 0x7fc0bc0b2460 2020-11-20T12:18:29.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.145+0000 7fc0aa7fc700 1 -- 172.21.15.187:0/1721675953 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b4011e30 con 0x7fc0bc0b2460 2020-11-20T12:18:29.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.146+0000 7fc0aa7fc700 1 -- 172.21.15.187:0/1721675953 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4010d40 con 0x7fc0bc0b2460 2020-11-20T12:18:29.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.146+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0bc0a2710 con 0x7fc0bc0b2460 2020-11-20T12:18:29.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.146+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc09f270 con 0x7fc0bc0b2460 2020-11-20T12:18:29.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.147+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0bc063a40 con 0x7fc0bc0b2460 2020-11-20T12:18:29.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.147+0000 7fc0aa7fc700 1 -- 172.21.15.187:0/1721675953 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc0b4017070 con 0x7fc0bc0b2460 2020-11-20T12:18:29.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.148+0000 7fc0aa7fc700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc094056d50 0x7fc094059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:29.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.148+0000 7fc0aa7fc700 1 -- 172.21.15.187:0/1721675953 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc0b4071bd0 con 0x7fc0bc0b2460 2020-11-20T12:18:29.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.148+0000 7fc0c20be700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc094056d50 0x7fc094059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:29.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.149+0000 7fc0c20be700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc094056d50 0x7fc094059210 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7fc0b8006fd0 tx=0x7fc0b8006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:29.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.152+0000 7fc0aa7fc700 1 -- 172.21.15.187:0/1721675953 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc0b4046fd0 con 0x7fc0bc0b2460 2020-11-20T12:18:29.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.410+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0bc0a28a0 con 0x7fc0bc0b2460 2020-11-20T12:18:29.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.412+0000 7fc0aa7fc700 1 -- 172.21.15.187:0/1721675953 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+1797 (secure 0 0 0) 0x7fc0bc0a28a0 con 0x7fc0bc0b2460 2020-11-20T12:18:29.415 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:29.416 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":119,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:28.847001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932},"failed":[1],"damaged":[],"stopped":[],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:29.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.414+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc094056d50 msgr2=0x7fc094059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:29.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.414+0000 7fc0c4322700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc094056d50 0x7fc094059210 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7fc0b8006fd0 tx=0x7fc0b8006df0).stop 2020-11-20T12:18:29.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.414+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0bc0b2460 msgr2=0x7fc0bc0b4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:29.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.414+0000 7fc0c4322700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0bc0b2460 0x7fc0bc0b4770 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 rx=0x7fc0bc063d60 tx=0x7fc0b4016000).stop 2020-11-20T12:18:29.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.415+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 shutdown_connections 2020-11-20T12:18:29.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.415+0000 7fc0c4322700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc094056d50 0x7fc094059210 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.415+0000 7fc0c4322700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0bc062c50 0x7fc0bc15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.415+0000 7fc0c4322700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0bc0b2460 0x7fc0bc0b4770 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.415+0000 7fc0c4322700 1 --2- 172.21.15.187:0/1721675953 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0bc063030 0x7fc0bc0a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:29.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.415+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 >> 172.21.15.187:0/1721675953 conn(0x7fc0bc0bcfb0 msgr2=0x7fc0bc0bbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:29.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.415+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 shutdown_connections 2020-11-20T12:18:29.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:29.415+0000 7fc0c4322700 1 -- 172.21.15.187:0/1721675953 wait complete. 2020-11-20T12:18:29.422 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 119 2020-11-20T12:18:29.640 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 119, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:18:28.847001+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8932}, 'failed': [1], 'damaged': [], 'stopped': [], 'info': {'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:18:29.641 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2020-11-20T12:18:30.643 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:30.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.896+0000 7f7567fff700 1 Processor -- start 2020-11-20T12:18:30.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.896+0000 7f7567fff700 1 -- start start 2020-11-20T12:18:30.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.897+0000 7f7567fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75680654a0 0x7f7568149290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:30.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.897+0000 7f7567fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f75680b1110 0x7f7568149780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:30.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.897+0000 7f7567fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756809a930 con 0x7f7568065880 2020-11-20T12:18:30.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.897+0000 7f7567fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75680b0be0 con 0x7f75680b1110 2020-11-20T12:18:30.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.897+0000 7f7567fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75680b71b0 con 0x7f75680654a0 2020-11-20T12:18:30.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.898+0000 7f755e7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7568065880 0x7f7568061d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48662/0 (socket says 172.21.15.187:48662) 2020-11-20T12:18:30.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.898+0000 7f755e7fc700 1 -- 172.21.15.187:0/283676794 learned_addr learned my addr 172.21.15.187:0/283676794 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:30.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.898+0000 7f7566ffd700 1 --2- 172.21.15.187:0/283676794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75680654a0 0x7f7568149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:30.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.898+0000 7f75677fe700 1 --2- 172.21.15.187:0/283676794 >> v2:172.21.15.5:3300/0 conn(0x7f75680b1110 0x7f7568149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:30.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.899+0000 7f75667fc700 1 -- 172.21.15.187:0/283676794 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007938495 0 0) 0x7f756809a930 con 0x7f7568065880 2020-11-20T12:18:30.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.899+0000 7f75667fc700 1 -- 172.21.15.187:0/283676794 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7548003240 con 0x7f7568065880 2020-11-20T12:18:30.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.899+0000 7f7566ffd700 1 -- 172.21.15.187:0/283676794 >> v2:172.21.15.5:3300/0 conn(0x7f75680b1110 msgr2=0x7f7568149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:30.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.899+0000 7f7566ffd700 1 --2- 172.21.15.187:0/283676794 >> v2:172.21.15.5:3300/0 conn(0x7f75680b1110 0x7f7568149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:30.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.899+0000 7f7566ffd700 1 -- 172.21.15.187:0/283676794 >> v1:172.21.15.187:6789/0 conn(0x7f7568065880 legacy=0x7f7568061d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:30.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.899+0000 7f75667fc700 1 -- 172.21.15.187:0/283676794 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2717407232 0 0) 0x7f7548003240 con 0x7f7568065880 2020-11-20T12:18:30.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.899+0000 7f7566ffd700 1 -- 172.21.15.187:0/283676794 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75680662d0 con 0x7f75680654a0 2020-11-20T12:18:30.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.899+0000 7f75677fe700 1 --2- 172.21.15.187:0/283676794 >> v2:172.21.15.5:3300/0 conn(0x7f75680b1110 0x7f7568149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:18:30.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.900+0000 7f7566ffd700 1 --2- 172.21.15.187:0/283676794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75680654a0 0x7f7568149290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f755800b9a0 tx=0x7f7558006df0).ready entity=mon.2 client_cookie=c215f0e7d4a170fd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:30.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.900+0000 7f75667fc700 1 -- 172.21.15.187:0/283676794 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75500014e0 con 0x7f75680654a0 2020-11-20T12:18:30.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.900+0000 7f75667fc700 1 -- 172.21.15.187:0/283676794 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558011400 con 0x7f75680654a0 2020-11-20T12:18:30.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.900+0000 7f75667fc700 1 -- 172.21.15.187:0/283676794 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558011780 con 0x7f75680654a0 2020-11-20T12:18:30.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.901+0000 7f7567fff700 1 -- 172.21.15.187:0/283676794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75680654a0 msgr2=0x7f7568149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:30.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.901+0000 7f7567fff700 1 --2- 172.21.15.187:0/283676794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75680654a0 0x7f7568149290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f755800b9a0 tx=0x7f7558006df0).stop 2020-11-20T12:18:30.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.901+0000 7f7567fff700 1 -- 172.21.15.187:0/283676794 shutdown_connections 2020-11-20T12:18:30.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.901+0000 7f7567fff700 1 --2- 172.21.15.187:0/283676794 >> v2:172.21.15.5:3300/0 conn(0x7f75680b1110 0x7f7568149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:30.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.901+0000 7f7567fff700 1 --2- 172.21.15.187:0/283676794 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75680654a0 0x7f7568149290 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0x7f755800b9a0 tx=0x7f7558006df0).stop 2020-11-20T12:18:30.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.901+0000 7f7567fff700 1 -- 172.21.15.187:0/283676794 >> 172.21.15.187:0/283676794 conn(0x7f75680bab90 msgr2=0x7f75680bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:30.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.901+0000 7f7567fff700 1 -- 172.21.15.187:0/283676794 shutdown_connections 2020-11-20T12:18:30.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.901+0000 7f7567fff700 1 -- 172.21.15.187:0/283676794 wait complete. 2020-11-20T12:18:30.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.903+0000 7f7567fff700 1 Processor -- start 2020-11-20T12:18:30.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.903+0000 7f7567fff700 1 -- start start 2020-11-20T12:18:30.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f7567fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7568065880 0x7f7568150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:30.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f7567fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75680b1110 0x7f7568156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:30.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f7567fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7568149c60 0x7f75680b3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:30.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f7567fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756809f6e0 con 0x7f75680b1110 2020-11-20T12:18:30.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f7567fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756809f980 con 0x7f7568149c60 2020-11-20T12:18:30.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f7567fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756809fc20 con 0x7f7568065880 2020-11-20T12:18:30.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f75677fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7568149c60 0x7f75680b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:30.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f75677fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7568149c60 0x7f75680b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52062/0 (socket says 172.21.15.187:52062) 2020-11-20T12:18:30.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f75677fe700 1 -- 172.21.15.187:0/524117466 learned_addr learned my addr 172.21.15.187:0/524117466 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.904+0000 7f7566ffd700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7568065880 0x7f7568150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f75677fe700 1 -- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7568065880 msgr2=0x7f7568150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f75677fe700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7568065880 0x7f7568150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f75677fe700 1 -- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75680b1110 msgr2=0x7f7568156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f755e7fc700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75680b1110 0x7f7568156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:30.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f75677fe700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75680b1110 0x7f7568156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:30.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f75677fe700 1 -- 172.21.15.187:0/524117466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75681512b0 con 0x7f7568149c60 2020-11-20T12:18:30.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f7566ffd700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7568065880 0x7f7568150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:30.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f75677fe700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7568149c60 0x7f75680b3890 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7f756000ef20 tx=0x7f75600091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:30.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f755f7fe700 1 -- 172.21.15.187:0/524117466 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756000cd40 con 0x7f7568149c60 2020-11-20T12:18:30.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f755f7fe700 1 -- 172.21.15.187:0/524117466 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75600123f0 con 0x7f7568149c60 2020-11-20T12:18:30.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756814c2b0 con 0x7f7568149c60 2020-11-20T12:18:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f755f7fe700 1 -- 172.21.15.187:0/524117466 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756000cea0 con 0x7f7568149c60 2020-11-20T12:18:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.905+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75680b3d80 con 0x7f7568149c60 2020-11-20T12:18:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.907+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756809fde0 con 0x7f7568149c60 2020-11-20T12:18:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.907+0000 7f755f7fe700 1 -- 172.21.15.187:0/524117466 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7560022070 con 0x7f7568149c60 2020-11-20T12:18:30.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.908+0000 7f755f7fe700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7544056d20 0x7f75440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:30.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.908+0000 7f755f7fe700 1 -- 172.21.15.187:0/524117466 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7560070960 con 0x7f7568149c60 2020-11-20T12:18:30.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.908+0000 7f7566ffd700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7544056d20 0x7f75440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:30.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.909+0000 7f7566ffd700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7544056d20 0x7f75440591e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f75680b0dc0 tx=0x7f7558013000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:30.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:30.912+0000 7f755f7fe700 1 -- 172.21.15.187:0/524117466 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7560045ce0 con 0x7f7568149c60 2020-11-20T12:18:31.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.163+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7568061ad0 con 0x7f7568149c60 2020-11-20T12:18:31.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.164+0000 7f755f7fe700 1 -- 172.21.15.187:0/524117466 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+1797 (secure 0 0 0) 0x7f7560016090 con 0x7f7568149c60 2020-11-20T12:18:31.169 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:31.169 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":119,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:28.847001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932},"failed":[1],"damaged":[],"stopped":[],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:31.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.166+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7544056d20 msgr2=0x7f75440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:31.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.166+0000 7f7567fff700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7544056d20 0x7f75440591e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f75680b0dc0 tx=0x7f7558013000).stop 2020-11-20T12:18:31.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.166+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7568149c60 msgr2=0x7f75680b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:31.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.166+0000 7f7567fff700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7568149c60 0x7f75680b3890 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7f756000ef20 tx=0x7f75600091d0).stop 2020-11-20T12:18:31.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.167+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 shutdown_connections 2020-11-20T12:18:31.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.167+0000 7f7567fff700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7544056d20 0x7f75440591e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:31.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.167+0000 7f7567fff700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7568149c60 0x7f75680b3890 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:31.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.167+0000 7f7567fff700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7568065880 0x7f7568150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:31.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.167+0000 7f7567fff700 1 --2- 172.21.15.187:0/524117466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75680b1110 0x7f7568156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:31.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.167+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 >> 172.21.15.187:0/524117466 conn(0x7f75680bab90 msgr2=0x7f75680b7610 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:31.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.167+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 shutdown_connections 2020-11-20T12:18:31.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:31.167+0000 7f7567fff700 1 -- 172.21.15.187:0/524117466 wait complete. 2020-11-20T12:18:31.174 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 119 2020-11-20T12:18:31.397 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 119, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:18:28.847001+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8932}, 'failed': [1], 'damaged': [], 'stopped': [], 'info': {'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:18:31.398 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2020-11-20T12:18:32.399 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:32.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.643+0000 7f426d3bb700 1 Processor -- start 2020-11-20T12:18:32.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.643+0000 7f426d3bb700 1 -- start start 2020-11-20T12:18:32.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f426d3bb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f42680654a0 0x7f42680675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f426d3bb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42680670c0 0x7f42680afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f426d3bb700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4268099f70 con 0x7f4268065880 2020-11-20T12:18:32.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f426d3bb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42680b6a20 con 0x7f42680654a0 2020-11-20T12:18:32.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f426d3bb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4268148990 con 0x7f42680670c0 2020-11-20T12:18:32.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f425f7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4268065880 0x7f4268148280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48672/0 (socket says 172.21.15.187:48672) 2020-11-20T12:18:32.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f425f7fe700 1 -- 172.21.15.187:0/1347585852 learned_addr learned my addr 172.21.15.187:0/1347585852 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:32.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286052960 0 0) 0x7f4268099f70 con 0x7f4268065880 2020-11-20T12:18:32.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.645+0000 7f4267fff700 1 --2- 172.21.15.187:0/1347585852 >> v2:172.21.15.5:3300/0 conn(0x7f42680654a0 0x7f42680675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:32.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4248003240 con 0x7f4268065880 2020-11-20T12:18:32.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f426cbba700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42680670c0 0x7f42680afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:32.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1499130075 0 0) 0x7f4248003240 con 0x7f4268065880 2020-11-20T12:18:32.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42680670c0 msgr2=0x7f42680afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:32.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42680670c0 0x7f42680afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:32.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 >> v2:172.21.15.5:3300/0 conn(0x7f42680654a0 msgr2=0x7f42680675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:32.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 --2- 172.21.15.187:0/1347585852 >> v2:172.21.15.5:3300/0 conn(0x7f42680654a0 0x7f42680675e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:32.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42680affe0 con 0x7f4268065880 2020-11-20T12:18:32.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f42500014e0 con 0x7f4268065880 2020-11-20T12:18:32.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 >> v1:172.21.15.187:6789/0 conn(0x7f4268065880 legacy=0x7f4268148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:32.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42480037b0 0x7f4248003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42680670c0 0x7f42480044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42480049d0 0x7f4248004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42680b6a20 con 0x7f42480049d0 2020-11-20T12:18:32.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4268148990 con 0x7f42480037b0 2020-11-20T12:18:32.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.646+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42480042a0 con 0x7f42680670c0 2020-11-20T12:18:32.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.647+0000 7f4267fff700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42480037b0 0x7f4248003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:32.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.647+0000 7f4267fff700 1 -- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42680670c0 msgr2=0x7f42480044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:32.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.647+0000 7f4267fff700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42680670c0 0x7f42480044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:32.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.647+0000 7f4267fff700 1 -- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42480049d0 msgr2=0x7f4248004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:32.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.647+0000 7f4267fff700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42480049d0 0x7f4248004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:32.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.647+0000 7f4267fff700 1 -- 172.21.15.187:0/1347585852 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4248003240 con 0x7f42480037b0 2020-11-20T12:18:32.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.647+0000 7f4267fff700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42480037b0 0x7f4248003bd0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 rx=0x7f425800bc00 tx=0x7f4258006df0).ready entity=mon.1 client_cookie=c87858c6083a595b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:32.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425800b310 con 0x7f42480037b0 2020-11-20T12:18:32.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425800eb40 con 0x7f42480037b0 2020-11-20T12:18:32.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f42677fe700 1 -- 172.21.15.187:0/1347585852 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425800eec0 con 0x7f42480037b0 2020-11-20T12:18:32.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f426d3bb700 1 -- 172.21.15.187:0/1347585852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42480037b0 msgr2=0x7f4248003bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:32.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f426d3bb700 1 --2- 172.21.15.187:0/1347585852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42480037b0 0x7f4248003bd0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 rx=0x7f425800bc00 tx=0x7f4258006df0).stop 2020-11-20T12:18:32.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f426cbba700 1 -- 172.21.15.187:0/1347585852 reap_dead start 2020-11-20T12:18:32.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f426d3bb700 1 -- 172.21.15.187:0/1347585852 shutdown_connections 2020-11-20T12:18:32.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f426d3bb700 1 -- 172.21.15.187:0/1347585852 >> 172.21.15.187:0/1347585852 conn(0x7f42680bab90 msgr2=0x7f42680bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:32.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f426d3bb700 1 -- 172.21.15.187:0/1347585852 shutdown_connections 2020-11-20T12:18:32.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.648+0000 7f426d3bb700 1 -- 172.21.15.187:0/1347585852 wait complete. 2020-11-20T12:18:32.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.650+0000 7f426d3bb700 1 Processor -- start 2020-11-20T12:18:32.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.650+0000 7f426d3bb700 1 -- start start 2020-11-20T12:18:32.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426d3bb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f426809c1b0 0x7f4268157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426d3bb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f426809c590 0x7f42680b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426d3bb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4268145790 0x7f4268141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426d3bb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42681582a0 con 0x7f426809c1b0 2020-11-20T12:18:32.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426d3bb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42680b1ab0 con 0x7f4268145790 2020-11-20T12:18:32.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426d3bb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42680b1d50 con 0x7f426809c590 2020-11-20T12:18:32.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426cbba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4268145790 0x7f4268141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:32.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426cbba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4268145790 0x7f4268141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52080/0 (socket says 172.21.15.187:52080) 2020-11-20T12:18:32.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.651+0000 7f426cbba700 1 -- 172.21.15.187:0/2855885129 learned_addr learned my addr 172.21.15.187:0/2855885129 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:32.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f426cbba700 1 -- 172.21.15.187:0/2855885129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f426809c590 msgr2=0x7f42680b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:32.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f426cbba700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f426809c590 0x7f42680b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:32.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f426cbba700 1 -- 172.21.15.187:0/2855885129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f426809c1b0 msgr2=0x7f4268157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:32.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f426cbba700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f426809c1b0 0x7f4268157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:32.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f426cbba700 1 -- 172.21.15.187:0/2855885129 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42680b17a0 con 0x7f4268145790 2020-11-20T12:18:32.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f426cbba700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4268145790 0x7f4268141be0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 rx=0x7f42680b87e0 tx=0x7f4260008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:32.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f4264ff9700 1 -- 172.21.15.187:0/2855885129 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42600133f0 con 0x7f4268145790 2020-11-20T12:18:32.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f4264ff9700 1 -- 172.21.15.187:0/2855885129 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4260013550 con 0x7f4268145790 2020-11-20T12:18:32.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42680b26e0 con 0x7f4268145790 2020-11-20T12:18:32.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f4264ff9700 1 -- 172.21.15.187:0/2855885129 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4260013910 con 0x7f4268145790 2020-11-20T12:18:32.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.652+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4268142290 con 0x7f4268145790 2020-11-20T12:18:32.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.653+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4268142510 con 0x7f4268145790 2020-11-20T12:18:32.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.654+0000 7f4264ff9700 1 -- 172.21.15.187:0/2855885129 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4260013d00 con 0x7f4268145790 2020-11-20T12:18:32.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.654+0000 7f4264ff9700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4244056d20 0x7f42440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:32.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.655+0000 7f4264ff9700 1 -- 172.21.15.187:0/2855885129 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f426006ff10 con 0x7f4268145790 2020-11-20T12:18:32.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.655+0000 7f4267fff700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4244056d20 0x7f42440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:32.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.656+0000 7f4267fff700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4244056d20 0x7f42440591e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f4268045400 tx=0x7f4258019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:32.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.659+0000 7f4264ff9700 1 -- 172.21.15.187:0/2855885129 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4260045200 con 0x7f4268145790 2020-11-20T12:18:32.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.906+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42680b1f10 con 0x7f4268145790 2020-11-20T12:18:32.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.907+0000 7f4264ff9700 1 -- 172.21.15.187:0/2855885129 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+1797 (secure 0 0 0) 0x7f4260048820 con 0x7f4268145790 2020-11-20T12:18:32.911 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:32.912 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":119,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:28.847001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932},"failed":[1],"damaged":[],"stopped":[],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:32.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.909+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4244056d20 msgr2=0x7f42440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:32.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.909+0000 7f426d3bb700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4244056d20 0x7f42440591e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f4268045400 tx=0x7f4258019000).stop 2020-11-20T12:18:32.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.909+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4268145790 msgr2=0x7f4268141be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:32.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.909+0000 7f426d3bb700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4268145790 0x7f4268141be0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 rx=0x7f42680b87e0 tx=0x7f4260008040).stop 2020-11-20T12:18:32.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.910+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 shutdown_connections 2020-11-20T12:18:32.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.910+0000 7f426d3bb700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4244056d20 0x7f42440591e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:32.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.910+0000 7f426d3bb700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f426809c1b0 0x7f4268157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:32.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.910+0000 7f426d3bb700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4268145790 0x7f4268141be0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:32.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.910+0000 7f426d3bb700 1 --2- 172.21.15.187:0/2855885129 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f426809c590 0x7f42680b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:32.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.910+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 >> 172.21.15.187:0/2855885129 conn(0x7f42680bab90 msgr2=0x7f42680b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:32.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.910+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 shutdown_connections 2020-11-20T12:18:32.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:32.910+0000 7f426d3bb700 1 -- 172.21.15.187:0/2855885129 wait complete. 2020-11-20T12:18:32.917 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 119 2020-11-20T12:18:33.140 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 119, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:18:28.847001+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8932}, 'failed': [1], 'damaged': [], 'stopped': [], 'info': {'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:18:33.141 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2020-11-20T12:18:34.141 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:34.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.388+0000 7f5dc759e700 1 Processor -- start 2020-11-20T12:18:34.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.389+0000 7f5dc759e700 1 -- start start 2020-11-20T12:18:34.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.390+0000 7f5dc759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5dc80654a0 0x7f5dc8149290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.390+0000 7f5dc759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 0x7f5dc8149780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.390+0000 7f5dc759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5dc80b1110 0x7f5dc8061d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.390+0000 7f5dc759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc809a930 con 0x7f5dc80654a0 2020-11-20T12:18:34.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.390+0000 7f5dc759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc80b71b0 con 0x7f5dc80b1110 2020-11-20T12:18:34.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.390+0000 7f5dc759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc80662d0 con 0x7f5dc8065880 2020-11-20T12:18:34.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc6d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5dc80b1110 0x7f5dc8061d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:34.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc6d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5dc80b1110 0x7f5dc8061d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52086/0 (socket says 172.21.15.187:52086) 2020-11-20T12:18:34.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc6d9d700 1 -- 172.21.15.187:0/3521339051 learned_addr learned my addr 172.21.15.187:0/3521339051 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:34.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc659c700 1 --2- 172.21.15.187:0/3521339051 >> v2:172.21.15.5:3300/0 conn(0x7f5dc80654a0 0x7f5dc8149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:34.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dbdd9b700 1 --2- 172.21.15.187:0/3521339051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 0x7f5dc8149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:34.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc6d9d700 1 -- 172.21.15.187:0/3521339051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 msgr2=0x7f5dc8149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:34.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc6d9d700 1 --2- 172.21.15.187:0/3521339051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 0x7f5dc8149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc6d9d700 1 -- 172.21.15.187:0/3521339051 >> v2:172.21.15.5:3300/0 conn(0x7f5dc80654a0 msgr2=0x7f5dc8149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:34.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc6d9d700 1 --2- 172.21.15.187:0/3521339051 >> v2:172.21.15.5:3300/0 conn(0x7f5dc80654a0 0x7f5dc8149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.391+0000 7f5dc6d9d700 1 -- 172.21.15.187:0/3521339051 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc809ab30 con 0x7f5dc80b1110 2020-11-20T12:18:34.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.392+0000 7f5dc6d9d700 1 --2- 172.21.15.187:0/3521339051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80b1110 0x7f5dc8061d50 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 rx=0x7f5dc80b60b0 tx=0x7f5dc0009290).ready entity=mon.1 client_cookie=5177344f8c3b27a4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:34.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.392+0000 7f5dc5d9b700 1 -- 172.21.15.187:0/3521339051 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc000bbc0 con 0x7f5dc80b1110 2020-11-20T12:18:34.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.392+0000 7f5dc5d9b700 1 -- 172.21.15.187:0/3521339051 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc000bd20 con 0x7f5dc80b1110 2020-11-20T12:18:34.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.392+0000 7f5dc5d9b700 1 -- 172.21.15.187:0/3521339051 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0011610 con 0x7f5dc80b1110 2020-11-20T12:18:34.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 -- 172.21.15.187:0/3521339051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80b1110 msgr2=0x7f5dc8061d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:34.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 --2- 172.21.15.187:0/3521339051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80b1110 0x7f5dc8061d50 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 rx=0x7f5dc80b60b0 tx=0x7f5dc0009290).stop 2020-11-20T12:18:34.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 -- 172.21.15.187:0/3521339051 shutdown_connections 2020-11-20T12:18:34.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 --2- 172.21.15.187:0/3521339051 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80b1110 0x7f5dc8061d50 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 --2- 172.21.15.187:0/3521339051 >> v2:172.21.15.5:3300/0 conn(0x7f5dc80654a0 0x7f5dc8149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 --2- 172.21.15.187:0/3521339051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 0x7f5dc8149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 -- 172.21.15.187:0/3521339051 >> 172.21.15.187:0/3521339051 conn(0x7f5dc80bab90 msgr2=0x7f5dc80bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:34.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 -- 172.21.15.187:0/3521339051 shutdown_connections 2020-11-20T12:18:34.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.393+0000 7f5dc759e700 1 -- 172.21.15.187:0/3521339051 wait complete. 2020-11-20T12:18:34.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.395+0000 7f5dc759e700 1 Processor -- start 2020-11-20T12:18:34.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.395+0000 7f5dc759e700 1 -- start start 2020-11-20T12:18:34.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.396+0000 7f5dc759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80654a0 0x7f5dc809a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:34.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.396+0000 7f5dc759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 0x7f5dc80a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:34.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.396+0000 7f5dc759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5dc80b1110 0x7f5dc80a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:34.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.396+0000 7f5dc759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8092eb0 con 0x7f5dc80b1110 2020-11-20T12:18:34.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.396+0000 7f5dc759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8093150 con 0x7f5dc80654a0 2020-11-20T12:18:34.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.396+0000 7f5dc759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc80b2d60 con 0x7f5dc8065880 2020-11-20T12:18:34.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80654a0 0x7f5dc809a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:34.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80654a0 0x7f5dc809a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52090/0 (socket says 172.21.15.187:52090) 2020-11-20T12:18:34.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 -- 172.21.15.187:0/2456646064 learned_addr learned my addr 172.21.15.187:0/2456646064 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:34.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 -- 172.21.15.187:0/2456646064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 msgr2=0x7f5dc80a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:34.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 0x7f5dc80a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:34.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 -- 172.21.15.187:0/2456646064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5dc80b1110 msgr2=0x7f5dc80a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:34.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc6d9d700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5dc80b1110 0x7f5dc80a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:34.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5dc80b1110 0x7f5dc80a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 -- 172.21.15.187:0/2456646064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8158970 con 0x7f5dc80654a0 2020-11-20T12:18:34.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.397+0000 7f5dc659c700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80654a0 0x7f5dc809a0b0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 rx=0x7f5db800db60 tx=0x7f5db8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:34.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.398+0000 7f5dbeffd700 1 -- 172.21.15.187:0/2456646064 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db800b170 con 0x7f5dc80654a0 2020-11-20T12:18:34.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.398+0000 7f5dbeffd700 1 -- 172.21.15.187:0/2456646064 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db8009790 con 0x7f5dc80654a0 2020-11-20T12:18:34.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.398+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc809edf0 con 0x7f5dc80654a0 2020-11-20T12:18:34.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.398+0000 7f5dbeffd700 1 -- 172.21.15.187:0/2456646064 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db8009b50 con 0x7f5dc80654a0 2020-11-20T12:18:34.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.398+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc81586d0 con 0x7f5dc80654a0 2020-11-20T12:18:34.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.399+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc8093310 con 0x7f5dc80654a0 2020-11-20T12:18:34.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.400+0000 7f5dbeffd700 1 -- 172.21.15.187:0/2456646064 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5db8013480 con 0x7f5dc80654a0 2020-11-20T12:18:34.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.400+0000 7f5dbeffd700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5da4056d20 0x7f5da40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:34.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.400+0000 7f5dbeffd700 1 -- 172.21.15.187:0/2456646064 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f5db8070600 con 0x7f5dc80654a0 2020-11-20T12:18:34.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.400+0000 7f5dbdd9b700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5da4056d20 0x7f5da40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:34.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.401+0000 7f5dbdd9b700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5da4056d20 0x7f5da40591e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f5dc809a5a0 tx=0x7f5db0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:34.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.404+0000 7f5dbeffd700 1 -- 172.21.15.187:0/2456646064 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5db8045890 con 0x7f5dc80654a0 2020-11-20T12:18:34.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.650+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc8061ad0 con 0x7f5dc80654a0 2020-11-20T12:18:34.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.652+0000 7f5dbeffd700 1 -- 172.21.15.187:0/2456646064 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+2159 (secure 0 0 0) 0x7f5db8048eb0 con 0x7f5dc80654a0 2020-11-20T12:18:34.655 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:34.655 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":121,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:33.903048+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:replay","state_seq":2,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:34.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5da4056d20 msgr2=0x7f5da40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:34.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5da4056d20 0x7f5da40591e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f5dc809a5a0 tx=0x7f5db0008040).stop 2020-11-20T12:18:34.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80654a0 msgr2=0x7f5dc809a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:34.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80654a0 0x7f5dc809a0b0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 rx=0x7f5db800db60 tx=0x7f5db8008040).stop 2020-11-20T12:18:34.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 shutdown_connections 2020-11-20T12:18:34.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5da4056d20 0x7f5da40591e0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5dc80b1110 0x7f5dc80a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dc80654a0 0x7f5dc809a0b0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:34.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 --2- 172.21.15.187:0/2456646064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5dc8065880 0x7f5dc80a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:34.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 >> 172.21.15.187:0/2456646064 conn(0x7f5dc80bab90 msgr2=0x7f5dc8148270 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:34.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.654+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 shutdown_connections 2020-11-20T12:18:34.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:34.655+0000 7f5dc759e700 1 -- 172.21.15.187:0/2456646064 wait complete. 2020-11-20T12:18:34.661 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 121 2020-11-20T12:18:34.883 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:18:33.903048+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8932, 'mds_1': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'b', 'rank': 1, 'incarnation': 121, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.5:6833/4181204645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 4181204645}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 4181204645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:18:34.883 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5981:up:replay 2020-11-20T12:18:35.885 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:36.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.136+0000 7f6908e2e700 1 Processor -- start 2020-11-20T12:18:36.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.136+0000 7f6908e2e700 1 -- start start 2020-11-20T12:18:36.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.137+0000 7f6908e2e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f69040654a0 0x7f69040675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:36.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.138+0000 7f6908e2e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6904099f70 con 0x7f6904065880 2020-11-20T12:18:36.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.138+0000 7f6908e2e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69040b6a20 con 0x7f69040670c0 2020-11-20T12:18:36.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.138+0000 7f6908e2e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6904148990 con 0x7f69040654a0 2020-11-20T12:18:36.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.138+0000 7f6903fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f69040670c0 0x7f69040afaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48710/0 (socket says 172.21.15.187:48710) 2020-11-20T12:18:36.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.138+0000 7f6903fff700 1 -- 172.21.15.187:0/1460380248 learned_addr learned my addr 172.21.15.187:0/1460380248 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:36.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.138+0000 7f69037fe700 1 --2- 172.21.15.187:0/1460380248 >> v2:172.21.15.5:3300/0 conn(0x7f69040654a0 0x7f69040675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:36.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.138+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769595907 0 0) 0x7f69040b6a20 con 0x7f69040670c0 2020-11-20T12:18:36.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f0003240 con 0x7f69040670c0 2020-11-20T12:18:36.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 575550483 0 0) 0x7f68f0003240 con 0x7f69040670c0 2020-11-20T12:18:36.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 >> v2:172.21.15.5:3300/0 conn(0x7f69040654a0 msgr2=0x7f69040675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:36.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 --2- 172.21.15.187:0/1460380248 >> v2:172.21.15.5:3300/0 conn(0x7f69040654a0 0x7f69040675e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 >> v1:172.21.15.5:6789/0 conn(0x7f6904065880 legacy=0x7f6904148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:36.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040b0200 con 0x7f69040670c0 2020-11-20T12:18:36.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f68fc001860 con 0x7f69040670c0 2020-11-20T12:18:36.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 >> v1:172.21.15.187:6789/0 conn(0x7f69040670c0 legacy=0x7f69040afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:36.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f68f0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:36.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68f00044e0 0x7f68f0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:36.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68f0004e50 0x7f68f000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:36.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904148990 con 0x7f68f00044e0 2020-11-20T12:18:36.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f0004260 con 0x7f68f0003730 2020-11-20T12:18:36.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.139+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f000bc20 con 0x7f68f0004e50 2020-11-20T12:18:36.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.140+0000 7f69037fe700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f68f0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:36.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.140+0000 7f6902ffd700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68f00044e0 0x7f68f0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:36.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.140+0000 7f6903fff700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68f0004e50 0x7f68f000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:36.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.140+0000 7f69037fe700 1 -- 172.21.15.187:0/1460380248 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68f0004e50 msgr2=0x7f68f000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:36.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.140+0000 7f69037fe700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f68f0004e50 0x7f68f000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.140+0000 7f69037fe700 1 -- 172.21.15.187:0/1460380248 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68f00044e0 msgr2=0x7f68f0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:36.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.140+0000 7f69037fe700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68f00044e0 0x7f68f0004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.140+0000 7f69037fe700 1 -- 172.21.15.187:0/1460380248 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f0003240 con 0x7f68f0003730 2020-11-20T12:18:36.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.141+0000 7f6903fff700 1 -- 172.21.15.187:0/1460380248 reap_dead start 2020-11-20T12:18:36.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.141+0000 7f69037fe700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f68f0003b50 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 rx=0x7f68f400bc00 tx=0x7f68f4006df0).ready entity=mon.1 client_cookie=5540a24537b9d6a3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:36.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.141+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f400b310 con 0x7f68f0003730 2020-11-20T12:18:36.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.141+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f400eb40 con 0x7f68f0003730 2020-11-20T12:18:36.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.141+0000 7f69027fc700 1 -- 172.21.15.187:0/1460380248 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f400eec0 con 0x7f68f0003730 2020-11-20T12:18:36.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.142+0000 7f6908e2e700 1 -- 172.21.15.187:0/1460380248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 msgr2=0x7f68f0003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:36.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.142+0000 7f6908e2e700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f68f0003b50 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 rx=0x7f68f400bc00 tx=0x7f68f4006df0).stop 2020-11-20T12:18:36.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.142+0000 7f6908e2e700 1 -- 172.21.15.187:0/1460380248 shutdown_connections 2020-11-20T12:18:36.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.142+0000 7f6908e2e700 1 --2- 172.21.15.187:0/1460380248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f68f0003b50 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.142+0000 7f6908e2e700 1 -- 172.21.15.187:0/1460380248 >> 172.21.15.187:0/1460380248 conn(0x7f69040bab90 msgr2=0x7f69040bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:36.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.142+0000 7f6908e2e700 1 -- 172.21.15.187:0/1460380248 shutdown_connections 2020-11-20T12:18:36.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.142+0000 7f6908e2e700 1 -- 172.21.15.187:0/1460380248 wait complete. 2020-11-20T12:18:36.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.144+0000 7f6908e2e700 1 Processor -- start 2020-11-20T12:18:36.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.144+0000 7f6908e2e700 1 -- start start 2020-11-20T12:18:36.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f6908e2e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f69040b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:36.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f6908e2e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6904159ca0 0x7f69040b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:36.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f6908e2e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6904138150 0x7f69041385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:36.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f6908e2e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904130aa0 con 0x7f6904138150 2020-11-20T12:18:36.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f6908e2e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690415a120 con 0x7f68f0003730 2020-11-20T12:18:36.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f6908e2e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040b12c0 con 0x7f6904159ca0 2020-11-20T12:18:36.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f69037fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f69040b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:36.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f69037fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f69040b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52112/0 (socket says 172.21.15.187:52112) 2020-11-20T12:18:36.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.145+0000 7f69037fe700 1 -- 172.21.15.187:0/3369161130 learned_addr learned my addr 172.21.15.187:0/3369161130 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:36.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f6902ffd700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6904159ca0 0x7f69040b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:36.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f69037fe700 1 -- 172.21.15.187:0/3369161130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6904159ca0 msgr2=0x7f69040b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:36.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f6903fff700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6904138150 0x7f69041385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:36.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f69037fe700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6904159ca0 0x7f69040b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f69037fe700 1 -- 172.21.15.187:0/3369161130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6904138150 msgr2=0x7f69041385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:36.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f69037fe700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6904138150 0x7f69041385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f69037fe700 1 -- 172.21.15.187:0/3369161130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040b1560 con 0x7f68f0003730 2020-11-20T12:18:36.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f69037fe700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f69040b1ca0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7f68f4006e30 tx=0x7f68f4019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:36.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f68e37fe700 1 -- 172.21.15.187:0/3369161130 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f401d620 con 0x7f68f0003730 2020-11-20T12:18:36.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f68e37fe700 1 -- 172.21.15.187:0/3369161130 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f4021070 con 0x7f68f0003730 2020-11-20T12:18:36.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f68e37fe700 1 -- 172.21.15.187:0/3369161130 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f401d780 con 0x7f68f0003730 2020-11-20T12:18:36.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.146+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69040b1800 con 0x7f68f0003730 2020-11-20T12:18:36.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.147+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69040b9210 con 0x7f68f0003730 2020-11-20T12:18:36.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.148+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6904061ad0 con 0x7f68f0003730 2020-11-20T12:18:36.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.148+0000 7f68e37fe700 1 -- 172.21.15.187:0/3369161130 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f68f402d070 con 0x7f68f0003730 2020-11-20T12:18:36.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.149+0000 7f68e37fe700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f68dc056d20 0x7f68dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:36.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.149+0000 7f68e37fe700 1 -- 172.21.15.187:0/3369161130 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f68f40723c0 con 0x7f68f0003730 2020-11-20T12:18:36.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.149+0000 7f6902ffd700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f68dc056d20 0x7f68dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:36.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.150+0000 7f6902ffd700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f68dc056d20 0x7f68dc0591e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f68f8000bc0 tx=0x7f68f8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:36.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.153+0000 7f68e37fe700 1 -- 172.21.15.187:0/3369161130 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f68f4047870 con 0x7f68f0003730 2020-11-20T12:18:36.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.401+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f690409ffe0 con 0x7f68f0003730 2020-11-20T12:18:36.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.402+0000 7f68e37fe700 1 -- 172.21.15.187:0/3369161130 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2162 (secure 0 0 0) 0x7f68f404ae90 con 0x7f68f0003730 2020-11-20T12:18:36.406 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:36.407 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":123,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:35.910810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:36.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f68dc056d20 msgr2=0x7f68dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f68dc056d20 0x7f68dc0591e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f68f8000bc0 tx=0x7f68f8009040).stop 2020-11-20T12:18:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 msgr2=0x7f69040b1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:36.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f69040b1ca0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7f68f4006e30 tx=0x7f68f4019000).stop 2020-11-20T12:18:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 shutdown_connections 2020-11-20T12:18:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f68dc056d20 0x7f68dc0591e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6904138150 0x7f69041385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f0003730 0x7f69040b1ca0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 --2- 172.21.15.187:0/3369161130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6904159ca0 0x7f69040b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:36.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.404+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 >> 172.21.15.187:0/3369161130 conn(0x7f69040bab90 msgr2=0x7f6904061d50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.405+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 shutdown_connections 2020-11-20T12:18:36.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:36.405+0000 7f6908e2e700 1 -- 172.21.15.187:0/3369161130 wait complete. 2020-11-20T12:18:36.413 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 123 2020-11-20T12:18:36.631 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:18:35.910810+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8932, 'mds_1': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'b', 'rank': 1, 'incarnation': 121, 'state': 'up:reconnect', 'state_seq': 4, 'addr': '172.21.15.5:6833/4181204645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 4181204645}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 4181204645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:18:36.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5981:up:reconnect 2020-11-20T12:18:37.634 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:37.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.885+0000 7f25a6cac700 1 Processor -- start 2020-11-20T12:18:37.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.886+0000 7f25a6cac700 1 -- start start 2020-11-20T12:18:37.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.887+0000 7f25a6cac700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a0065880 0x7f25a00afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.887+0000 7f25a6cac700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f25a00b1110 0x7f25a00b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.887+0000 7f25a6cac700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a009a930 con 0x7f25a00654a0 2020-11-20T12:18:37.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.887+0000 7f25a6cac700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a00b71b0 con 0x7f25a00b1110 2020-11-20T12:18:37.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.887+0000 7f25a6cac700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00662d0 con 0x7f25a0065880 2020-11-20T12:18:37.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.888+0000 7f25a5caa700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f25a00654a0 0x7f25a0149290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48726/0 (socket says 172.21.15.187:48726) 2020-11-20T12:18:37.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.888+0000 7f25a5caa700 1 -- 172.21.15.187:0/2151002770 learned_addr learned my addr 172.21.15.187:0/2151002770 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:37.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.888+0000 7f25a64ab700 1 --2- 172.21.15.187:0/2151002770 >> v2:172.21.15.5:3300/0 conn(0x7f25a00b1110 0x7f25a00b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:37.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.888+0000 7f25a54a9700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a0065880 0x7f25a00afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:37.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.888+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027406560 0 0) 0x7f25a009a930 con 0x7f25a00654a0 2020-11-20T12:18:37.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2588003240 con 0x7f25a00654a0 2020-11-20T12:18:37.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3435234946 0 0) 0x7f2588003240 con 0x7f25a00654a0 2020-11-20T12:18:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a0065880 msgr2=0x7f25a00afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a0065880 0x7f25a00afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 >> v2:172.21.15.5:3300/0 conn(0x7f25a00b1110 msgr2=0x7f25a00b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 --2- 172.21.15.187:0/2151002770 >> v2:172.21.15.5:3300/0 conn(0x7f25a00b1110 0x7f25a00b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a009ab30 con 0x7f25a00654a0 2020-11-20T12:18:37.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f259c0014e0 con 0x7f25a00654a0 2020-11-20T12:18:37.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 >> v1:172.21.15.187:6789/0 conn(0x7f25a00654a0 legacy=0x7f25a0149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:37.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588003730 0x7f2588003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a0065880 0x7f25880044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880049d0 0x7f2588004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00b71b0 con 0x7f2588003730 2020-11-20T12:18:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00662d0 con 0x7f25880049d0 2020-11-20T12:18:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.889+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25880041d0 con 0x7f25a0065880 2020-11-20T12:18:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.890+0000 7f25a64ab700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880049d0 0x7f2588004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:37.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.890+0000 7f25a5caa700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588003730 0x7f2588003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:37.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.890+0000 7f25a64ab700 1 -- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a0065880 msgr2=0x7f25880044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:37.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.890+0000 7f25a64ab700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a0065880 0x7f25880044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:37.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.890+0000 7f25a64ab700 1 -- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588003730 msgr2=0x7f2588003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:37.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.890+0000 7f25a64ab700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588003730 0x7f2588003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:37.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.890+0000 7f25a64ab700 1 -- 172.21.15.187:0/2151002770 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2588003240 con 0x7f25880049d0 2020-11-20T12:18:37.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.890+0000 7f25a5caa700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588003730 0x7f2588003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:37.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.891+0000 7f25a64ab700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880049d0 0x7f2588004e50 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7f25a00a0da0 tx=0x7f25980091d0).ready entity=mon.1 client_cookie=afdca36fbd6c0e4f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:37.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.891+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25980095c0 con 0x7f25880049d0 2020-11-20T12:18:37.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.891+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259800edc0 con 0x7f25880049d0 2020-11-20T12:18:37.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.891+0000 7f25a4ca8700 1 -- 172.21.15.187:0/2151002770 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2598012600 con 0x7f25880049d0 2020-11-20T12:18:37.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.891+0000 7f25a6cac700 1 -- 172.21.15.187:0/2151002770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880049d0 msgr2=0x7f2588004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.891+0000 7f25a6cac700 1 --2- 172.21.15.187:0/2151002770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880049d0 0x7f2588004e50 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7f25a00a0da0 tx=0x7f25980091d0).stop 2020-11-20T12:18:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.891+0000 7f25a64ab700 1 -- 172.21.15.187:0/2151002770 reap_dead start 2020-11-20T12:18:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.892+0000 7f25a6cac700 1 -- 172.21.15.187:0/2151002770 shutdown_connections 2020-11-20T12:18:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.892+0000 7f25a6cac700 1 -- 172.21.15.187:0/2151002770 >> 172.21.15.187:0/2151002770 conn(0x7f25a00bab90 msgr2=0x7f25a00bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.892+0000 7f25a6cac700 1 -- 172.21.15.187:0/2151002770 shutdown_connections 2020-11-20T12:18:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.892+0000 7f25a6cac700 1 -- 172.21.15.187:0/2151002770 wait complete. 2020-11-20T12:18:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.893+0000 7f25a6cac700 1 Processor -- start 2020-11-20T12:18:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.893+0000 7f25a6cac700 1 -- start start 2020-11-20T12:18:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.894+0000 7f25a6cac700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25a009f9f0 0x7f25a014c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.894+0000 7f25a6cac700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a009fdd0 0x7f25a0148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.894+0000 7f25a6cac700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a01349b0 0x7f25a0148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.894+0000 7f25a6cac700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a014c6f0 con 0x7f25a009f9f0 2020-11-20T12:18:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.894+0000 7f25a6cac700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a014c8d0 con 0x7f25a009fdd0 2020-11-20T12:18:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.894+0000 7f25a6cac700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00b3030 con 0x7f25a01349b0 2020-11-20T12:18:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a009fdd0 0x7f25a0148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a009fdd0 0x7f25a0148510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52132/0 (socket says 172.21.15.187:52132) 2020-11-20T12:18:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 -- 172.21.15.187:0/3868569355 learned_addr learned my addr 172.21.15.187:0/3868569355 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a5caa700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25a009f9f0 0x7f25a014c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:37.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a64ab700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a01349b0 0x7f25a0148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:37.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 -- 172.21.15.187:0/3868569355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a01349b0 msgr2=0x7f25a0148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:37.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a01349b0 0x7f25a0148a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:37.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 -- 172.21.15.187:0/3868569355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25a009f9f0 msgr2=0x7f25a014c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25a009f9f0 0x7f25a014c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 -- 172.21.15.187:0/3868569355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a00b32d0 con 0x7f25a009fdd0 2020-11-20T12:18:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.895+0000 7f25a54a9700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a009fdd0 0x7f25a0148510 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7f2590006d00 tx=0x7f2590006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.896+0000 7f258dffb700 1 -- 172.21.15.187:0/3868569355 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590016070 con 0x7f25a009fdd0 2020-11-20T12:18:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.896+0000 7f258dffb700 1 -- 172.21.15.187:0/3868569355 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259000c820 con 0x7f25a009fdd0 2020-11-20T12:18:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.896+0000 7f258dffb700 1 -- 172.21.15.187:0/3868569355 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259000cba0 con 0x7f25a009fdd0 2020-11-20T12:18:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.896+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a00b1740 con 0x7f25a009fdd0 2020-11-20T12:18:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.896+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a00b1cb0 con 0x7f25a009fdd0 2020-11-20T12:18:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.897+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a00b3460 con 0x7f25a009fdd0 2020-11-20T12:18:37.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.898+0000 7f258dffb700 1 -- 172.21.15.187:0/3868569355 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f25900123f0 con 0x7f25a009fdd0 2020-11-20T12:18:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.898+0000 7f258dffb700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2578056d20 0x7f25780591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.898+0000 7f258dffb700 1 -- 172.21.15.187:0/3868569355 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2590070020 con 0x7f25a009fdd0 2020-11-20T12:18:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.899+0000 7f25a5caa700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2578056d20 0x7f25780591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.899+0000 7f25a5caa700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2578056d20 0x7f25780591e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f259c001e90 tx=0x7f259c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:37.903+0000 7f258dffb700 1 -- 172.21.15.187:0/3868569355 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f25900453c0 con 0x7f25a009fdd0 2020-11-20T12:18:38.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.148+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a0061ad0 con 0x7f25a009fdd0 2020-11-20T12:18:38.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.149+0000 7f258dffb700 1 -- 172.21.15.187:0/3868569355 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+2159 (secure 0 0 0) 0x7f25900489e0 con 0x7f25a009fdd0 2020-11-20T12:18:38.153 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:38.153 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":125,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:37.935677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:38.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.151+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2578056d20 msgr2=0x7f25780591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:38.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.151+0000 7f25a6cac700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2578056d20 0x7f25780591e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f259c001e90 tx=0x7f259c009040).stop 2020-11-20T12:18:38.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.151+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a009fdd0 msgr2=0x7f25a0148510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:38.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.151+0000 7f25a6cac700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a009fdd0 0x7f25a0148510 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7f2590006d00 tx=0x7f2590006df0).stop 2020-11-20T12:18:38.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.152+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 shutdown_connections 2020-11-20T12:18:38.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.152+0000 7f25a6cac700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2578056d20 0x7f25780591e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:38.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.152+0000 7f25a6cac700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25a009f9f0 0x7f25a014c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:38.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.152+0000 7f25a6cac700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25a01349b0 0x7f25a0148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:38.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.152+0000 7f25a6cac700 1 --2- 172.21.15.187:0/3868569355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a009fdd0 0x7f25a0148510 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:38.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.152+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 >> 172.21.15.187:0/3868569355 conn(0x7f25a00bab90 msgr2=0x7f25a009a270 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:38.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.152+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 shutdown_connections 2020-11-20T12:18:38.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:38.152+0000 7f25a6cac700 1 -- 172.21.15.187:0/3868569355 wait complete. 2020-11-20T12:18:38.158 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 125 2020-11-20T12:18:38.380 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 125, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:18:37.935677+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8932, 'mds_1': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'b', 'rank': 1, 'incarnation': 121, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.5:6833/4181204645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 4181204645}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 4181204645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:18:38.380 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2020-11-20T12:18:38.381 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2020-11-20T12:18:38.677 INFO:teuthology.orchestra.run.smithi005.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":1,"id":5981,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.7411086080000002,"mdsmap_epoch":125,"osdmap_epoch":131,"osdmap_epoch_barrier":131,"uptime":9.5591373169999994} 2020-11-20T12:18:38.724 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 5981, "mdsmap_epoch": 125, "osdmap_epoch": 131, "osdmap_epoch_barrier": 131, "rank_uptime": 4.741108608, "state": "up:active", "uptime": 9.559137317, "want_state": "up:active", "whoami": 1 } 2020-11-20T12:18:38.725 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:18:39.009 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8932,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":28.422567377,"mdsmap_epoch":125,"osdmap_epoch":131,"osdmap_epoch_barrier":131,"uptime":34.077445818000001} 2020-11-20T12:18:39.011 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8932, "mdsmap_epoch": 125, "osdmap_epoch": 131, "osdmap_epoch_barrier": 131, "rank_uptime": 28.422567377, "state": "up:active", "uptime": 34.077445818, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:18:39.011 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:18:39.012 INFO:teuthology.orchestra.run.smithi187:> import sys 2020-11-20T12:18:39.012 INFO:teuthology.orchestra.run.smithi187:> import time 2020-11-20T12:18:39.012 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:18:39.013 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:18:39.013 INFO:teuthology.orchestra.run.smithi187:> n = 5 2020-11-20T12:18:39.014 INFO:teuthology.orchestra.run.smithi187:> abs_path = "/home/ubuntu/cephtest/mnt.0/delete_me/file" 2020-11-20T12:18:39.014 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:18:39.014 INFO:teuthology.orchestra.run.smithi187:> if not os.path.exists(os.path.dirname(abs_path)): 2020-11-20T12:18:39.015 INFO:teuthology.orchestra.run.smithi187:> os.makedirs(os.path.dirname(abs_path)) 2020-11-20T12:18:39.015 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:18:39.015 INFO:teuthology.orchestra.run.smithi187:> for i in range(0, n): 2020-11-20T12:18:39.016 INFO:teuthology.orchestra.run.smithi187:> fname = "{0}_{1}".format(abs_path, i) 2020-11-20T12:18:39.016 INFO:teuthology.orchestra.run.smithi187:> with open(fname, '"'"'w'"'"') as f: 2020-11-20T12:18:39.017 INFO:teuthology.orchestra.run.smithi187:> f.write('"'"'content'"'"') 2020-11-20T12:18:39.017 INFO:teuthology.orchestra.run.smithi187:> if False: 2020-11-20T12:18:39.017 INFO:teuthology.orchestra.run.smithi187:> f.flush() 2020-11-20T12:18:39.018 INFO:teuthology.orchestra.run.smithi187:> os.fsync(f.fileno()) 2020-11-20T12:18:39.018 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:18:39.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:18:39.391 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.pin -v 1 delete_me) 2020-11-20T12:18:39.559 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:39.769 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:18:39.765+0000 7f357cee0700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2020-11-20T12:18:39.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.801+0000 7f473309c700 1 Processor -- start 2020-11-20T12:18:39.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.801+0000 7f473309c700 1 -- start start 2020-11-20T12:18:39.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.803+0000 7f473309c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f472c148490 0x7f472c0bf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.803+0000 7f473309c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c148870 0x7f472c060c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.803+0000 7f473309c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c14cf50 con 0x7f472c0bfba0 2020-11-20T12:18:39.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.803+0000 7f473309c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c0b9bc0 con 0x7f472c148490 2020-11-20T12:18:39.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.803+0000 7f473309c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b9720 con 0x7f472c148870 2020-11-20T12:18:39.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.803+0000 7f473289b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f472c0bfba0 0x7f472c061160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48750/0 (socket says 172.21.15.187:48750) 2020-11-20T12:18:39.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.803+0000 7f473289b700 1 -- 172.21.15.187:0/4185534242 learned_addr learned my addr 172.21.15.187:0/4185534242 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:39.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.804+0000 7f473209a700 1 --2- 172.21.15.187:0/4185534242 >> v2:172.21.15.5:3300/0 conn(0x7f472c148490 0x7f472c0bf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:39.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.804+0000 7f4731899700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c148870 0x7f472c060c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:39.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.804+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202677862 0 0) 0x7f472c14cf50 con 0x7f472c0bfba0 2020-11-20T12:18:39.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.804+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4714003240 con 0x7f472c0bfba0 2020-11-20T12:18:39.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2774114114 0 0) 0x7f4714003240 con 0x7f472c0bfba0 2020-11-20T12:18:39.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c148870 msgr2=0x7f472c060c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:39.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c148870 0x7f472c060c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:39.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 >> v2:172.21.15.5:3300/0 conn(0x7f472c148490 msgr2=0x7f472c0bf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:39.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 --2- 172.21.15.187:0/4185534242 >> v2:172.21.15.5:3300/0 conn(0x7f472c148490 0x7f472c0bf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:39.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c14cbd0 con 0x7f472c0bfba0 2020-11-20T12:18:39.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4724001860 con 0x7f472c0bfba0 2020-11-20T12:18:39.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 >> v1:172.21.15.187:6789/0 conn(0x7f472c0bfba0 legacy=0x7f472c061160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:39.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47140037b0 0x7f4714003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47140044e0 0x7f4714004920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4714004e10 0x7f471400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b9bc0 con 0x7f47140037b0 2020-11-20T12:18:39.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b9720 con 0x7f4714004e10 2020-11-20T12:18:39.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4714005290 con 0x7f47140044e0 2020-11-20T12:18:39.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.805+0000 7f473289b700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4714004e10 0x7f471400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:39.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f473209a700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47140037b0 0x7f4714003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:39.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f473289b700 1 -- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47140044e0 msgr2=0x7f4714004920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:39.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f473289b700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47140044e0 0x7f4714004920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:39.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f473289b700 1 -- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47140037b0 msgr2=0x7f4714003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:39.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f473289b700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47140037b0 0x7f4714003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:39.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f473289b700 1 -- 172.21.15.187:0/4185534242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714003240 con 0x7f4714004e10 2020-11-20T12:18:39.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f473209a700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47140037b0 0x7f4714003bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:39.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f473289b700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4714004e10 0x7f471400b730 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f472400c5d0 tx=0x7f4724007ad0).ready entity=mon.1 client_cookie=cbd1a87eb3d6d19f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:39.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724005b80 con 0x7f4714004e10 2020-11-20T12:18:39.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4724005ce0 con 0x7f4714004e10 2020-11-20T12:18:39.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.806+0000 7f4731098700 1 -- 172.21.15.187:0/4185534242 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724007320 con 0x7f4714004e10 2020-11-20T12:18:39.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.807+0000 7f473309c700 1 -- 172.21.15.187:0/4185534242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4714004e10 msgr2=0x7f471400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:39.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.807+0000 7f473309c700 1 --2- 172.21.15.187:0/4185534242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4714004e10 0x7f471400b730 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f472400c5d0 tx=0x7f4724007ad0).stop 2020-11-20T12:18:39.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.807+0000 7f473289b700 1 -- 172.21.15.187:0/4185534242 reap_dead start 2020-11-20T12:18:39.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.807+0000 7f473309c700 1 -- 172.21.15.187:0/4185534242 shutdown_connections 2020-11-20T12:18:39.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.807+0000 7f473309c700 1 -- 172.21.15.187:0/4185534242 >> 172.21.15.187:0/4185534242 conn(0x7f472c068370 msgr2=0x7f472c0befa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:39.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.807+0000 7f473309c700 1 -- 172.21.15.187:0/4185534242 shutdown_connections 2020-11-20T12:18:39.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.807+0000 7f473309c700 1 -- 172.21.15.187:0/4185534242 wait complete. 2020-11-20T12:18:39.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.809+0000 7f473309c700 1 Processor -- start 2020-11-20T12:18:39.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.809+0000 7f473309c700 1 -- start start 2020-11-20T12:18:39.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473309c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c143b70 0x7f472c0a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473309c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c0a3d30 0x7f472c0a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473309c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f472c0a3810 0x7f472c14a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473309c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0a4e60 con 0x7f472c0a3d30 2020-11-20T12:18:39.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473309c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0a4920 con 0x7f472c0a3810 2020-11-20T12:18:39.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473309c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0a4bc0 con 0x7f472c143b70 2020-11-20T12:18:39.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473289b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f472c0a3810 0x7f472c14a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:39.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473289b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f472c0a3810 0x7f472c14a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52158/0 (socket says 172.21.15.187:52158) 2020-11-20T12:18:39.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473289b700 1 -- 172.21.15.187:0/1181093070 learned_addr learned my addr 172.21.15.187:0/1181093070 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:39.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473209a700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c143b70 0x7f472c0a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:39.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f4731899700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c0a3d30 0x7f472c0a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:39.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473289b700 1 -- 172.21.15.187:0/1181093070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c143b70 msgr2=0x7f472c0a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:39.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473289b700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c143b70 0x7f472c0a2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:39.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473289b700 1 -- 172.21.15.187:0/1181093070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c0a3d30 msgr2=0x7f472c0a3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:39.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473289b700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c0a3d30 0x7f472c0a3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:39.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.811+0000 7f473289b700 1 -- 172.21.15.187:0/1181093070 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0b8830 con 0x7f472c0a3810 2020-11-20T12:18:39.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.812+0000 7f473289b700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f472c0a3810 0x7f472c14a210 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7f47240048e0 tx=0x7f4724013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:39.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.812+0000 7f471a7fc700 1 -- 172.21.15.187:0/1181093070 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240138b0 con 0x7f472c0a3810 2020-11-20T12:18:39.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.812+0000 7f471a7fc700 1 -- 172.21.15.187:0/1181093070 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472400b070 con 0x7f472c0a3810 2020-11-20T12:18:39.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.812+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c0a41b0 con 0x7f472c0a3810 2020-11-20T12:18:39.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.812+0000 7f471a7fc700 1 -- 172.21.15.187:0/1181093070 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724013c60 con 0x7f472c0a3810 2020-11-20T12:18:39.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.812+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c14a700 con 0x7f472c0a3810 2020-11-20T12:18:39.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.813+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c0a4370 con 0x7f472c0a3810 2020-11-20T12:18:39.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.814+0000 7f471a7fc700 1 -- 172.21.15.187:0/1181093070 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f472401c020 con 0x7f472c0a3810 2020-11-20T12:18:39.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.814+0000 7f471a7fc700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4704056d20 0x7f47040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:39.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.815+0000 7f471a7fc700 1 -- 172.21.15.187:0/1181093070 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4724071710 con 0x7f472c0a3810 2020-11-20T12:18:39.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.815+0000 7f473209a700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4704056d20 0x7f47040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:39.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.815+0000 7f473209a700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4704056d20 0x7f47040591e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f4728006fd0 tx=0x7f4728008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:39.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:39.818+0000 7f471a7fc700 1 -- 172.21.15.187:0/1181093070 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4724046a00 con 0x7f472c0a3810 2020-11-20T12:18:40.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.066+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f472c060640 con 0x7f472c0a3810 2020-11-20T12:18:40.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.067+0000 7f471a7fc700 1 -- 172.21.15.187:0/1181093070 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+2159 (secure 0 0 0) 0x7f472404a020 con 0x7f472c0a3810 2020-11-20T12:18:40.071 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:40.071 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":125,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:37.935677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:40.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4704056d20 msgr2=0x7f47040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4704056d20 0x7f47040591e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f4728006fd0 tx=0x7f4728008040).stop 2020-11-20T12:18:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f472c0a3810 msgr2=0x7f472c14a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f472c0a3810 0x7f472c14a210 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7f47240048e0 tx=0x7f4724013000).stop 2020-11-20T12:18:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 shutdown_connections 2020-11-20T12:18:40.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4704056d20 0x7f47040591e0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f472c0a3810 0x7f472c14a210 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f472c143b70 0x7f472c0a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 --2- 172.21.15.187:0/1181093070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c0a3d30 0x7f472c0a3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:40.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 >> 172.21.15.187:0/1181093070 conn(0x7f472c068370 msgr2=0x7f472c0bd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:40.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 shutdown_connections 2020-11-20T12:18:40.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:40.069+0000 7f473309c700 1 -- 172.21.15.187:0/1181093070 wait complete. 2020-11-20T12:18:40.075 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 125 2020-11-20T12:18:40.094 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:18:40.379 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2020-11-20T12:18:40.380 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:18:40.380 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2020-11-20T12:18:42.382 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:42.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.635+0000 7f647359e700 1 Processor -- start 2020-11-20T12:18:42.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.636+0000 7f647359e700 1 -- start start 2020-11-20T12:18:42.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.637+0000 7f647359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f647414c650 0x7f64740652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:42.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.637+0000 7f647359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f64740657d0 0x7f6474062460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:42.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.637+0000 7f647359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6474099e70 con 0x7f6474066270 2020-11-20T12:18:42.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.637+0000 7f647359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647409a110 con 0x7f64740657d0 2020-11-20T12:18:42.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.637+0000 7f647359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647409a3b0 con 0x7f647414c650 2020-11-20T12:18:42.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.638+0000 7f647259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f647414c650 0x7f64740652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:42.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.638+0000 7f647259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f647414c650 0x7f64740652e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:53050/0 (socket says 172.21.15.187:53050) 2020-11-20T12:18:42.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.638+0000 7f647259c700 1 -- 172.21.15.187:0/1764939643 learned_addr learned my addr 172.21.15.187:0/1764939643 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:42.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.638+0000 7f6472d9d700 1 --2- 172.21.15.187:0/1764939643 >> v2:172.21.15.5:3300/0 conn(0x7f64740657d0 0x7f6474062460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:42.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.639+0000 7f647259c700 1 -- 172.21.15.187:0/1764939643 >> v2:172.21.15.5:3300/0 conn(0x7f64740657d0 msgr2=0x7f6474062460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:42.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.639+0000 7f647259c700 1 --2- 172.21.15.187:0/1764939643 >> v2:172.21.15.5:3300/0 conn(0x7f64740657d0 0x7f6474062460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:42.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.639+0000 7f647259c700 1 -- 172.21.15.187:0/1764939643 >> v1:172.21.15.5:6789/0 conn(0x7f6474066270 legacy=0x7f6474061d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:42.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.639+0000 7f647259c700 1 -- 172.21.15.187:0/1764939643 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474066750 con 0x7f647414c650 2020-11-20T12:18:42.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.639+0000 7f647259c700 1 --2- 172.21.15.187:0/1764939643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f647414c650 0x7f64740652e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f6464009d50 tx=0x7f6464009290).ready entity=mon.2 client_cookie=f9ba943929cc176b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:42.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.640+0000 7f6471d9b700 1 -- 172.21.15.187:0/1764939643 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646400bc10 con 0x7f647414c650 2020-11-20T12:18:42.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.640+0000 7f6471d9b700 1 -- 172.21.15.187:0/1764939643 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646400bd70 con 0x7f647414c650 2020-11-20T12:18:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.640+0000 7f6471d9b700 1 -- 172.21.15.187:0/1764939643 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464011610 con 0x7f647414c650 2020-11-20T12:18:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.641+0000 7f647359e700 1 -- 172.21.15.187:0/1764939643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f647414c650 msgr2=0x7f64740652e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.641+0000 7f647359e700 1 --2- 172.21.15.187:0/1764939643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f647414c650 0x7f64740652e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f6464009d50 tx=0x7f6464009290).stop 2020-11-20T12:18:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.641+0000 7f647359e700 1 -- 172.21.15.187:0/1764939643 shutdown_connections 2020-11-20T12:18:42.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.641+0000 7f647359e700 1 --2- 172.21.15.187:0/1764939643 >> v2:172.21.15.5:3300/0 conn(0x7f64740657d0 0x7f6474062460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:42.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.641+0000 7f647359e700 1 --2- 172.21.15.187:0/1764939643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f647414c650 0x7f64740652e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:42.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.641+0000 7f647359e700 1 -- 172.21.15.187:0/1764939643 >> 172.21.15.187:0/1764939643 conn(0x7f64740bab90 msgr2=0x7f64740bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:42.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.641+0000 7f647359e700 1 -- 172.21.15.187:0/1764939643 shutdown_connections 2020-11-20T12:18:42.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.641+0000 7f647359e700 1 -- 172.21.15.187:0/1764939643 wait complete. 2020-11-20T12:18:42.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.643+0000 7f647359e700 1 Processor -- start 2020-11-20T12:18:42.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.643+0000 7f647359e700 1 -- start start 2020-11-20T12:18:42.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f647359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f64740657d0 0x7f647414cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:42.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f647359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6474066270 0x7f6474149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:42.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f647359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f647414c650 0x7f647409ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:42.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f647359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647409c2d0 con 0x7f64740657d0 2020-11-20T12:18:42.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f647359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647414a160 con 0x7f647414c650 2020-11-20T12:18:42.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f647359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647413f150 con 0x7f6474066270 2020-11-20T12:18:42.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f6472d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f647414c650 0x7f647409ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:42.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f6472d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f647414c650 0x7f647409ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52172/0 (socket says 172.21.15.187:52172) 2020-11-20T12:18:42.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f6472d9d700 1 -- 172.21.15.187:0/2298172558 learned_addr learned my addr 172.21.15.187:0/2298172558 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:42.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.645+0000 7f6469d9b700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6474066270 0x7f6474149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:42.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f6472d9d700 1 -- 172.21.15.187:0/2298172558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6474066270 msgr2=0x7f6474149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:42.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f6472d9d700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6474066270 0x7f6474149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:42.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f6472d9d700 1 -- 172.21.15.187:0/2298172558 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f64740657d0 msgr2=0x7f647414cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:42.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f6472d9d700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f64740657d0 0x7f647414cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:42.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f6472d9d700 1 -- 172.21.15.187:0/2298172558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64740a0490 con 0x7f647414c650 2020-11-20T12:18:42.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f6469d9b700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6474066270 0x7f6474149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:42.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f6472d9d700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f647414c650 0x7f647409ffa0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 rx=0x7f646c00ec30 tx=0x7f646c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:42.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f646affd700 1 -- 172.21.15.187:0/2298172558 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c00c9b0 con 0x7f647414c650 2020-11-20T12:18:42.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f646affd700 1 -- 172.21.15.187:0/2298172558 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c00cb10 con 0x7f647414c650 2020-11-20T12:18:42.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f646affd700 1 -- 172.21.15.187:0/2298172558 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c00ceb0 con 0x7f647414c650 2020-11-20T12:18:42.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64740a0710 con 0x7f647414c650 2020-11-20T12:18:42.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.646+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647409ed60 con 0x7f647414c650 2020-11-20T12:18:42.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.648+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647409c680 con 0x7f647414c650 2020-11-20T12:18:42.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.648+0000 7f646affd700 1 -- 172.21.15.187:0/2298172558 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f646c012600 con 0x7f647414c650 2020-11-20T12:18:42.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.649+0000 7f646affd700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6450056d20 0x7f64500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:42.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.649+0000 7f646affd700 1 -- 172.21.15.187:0/2298172558 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f646c070430 con 0x7f647414c650 2020-11-20T12:18:42.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.649+0000 7f647259c700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6450056d20 0x7f64500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:42.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.650+0000 7f647259c700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6450056d20 0x7f64500591e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f64640092d0 tx=0x7f6464016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:42.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.653+0000 7f646affd700 1 -- 172.21.15.187:0/2298172558 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f646c045830 con 0x7f647414c650 2020-11-20T12:18:42.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.903+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f647409efb0 con 0x7f647414c650 2020-11-20T12:18:42.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.904+0000 7f646affd700 1 -- 172.21.15.187:0/2298172558 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+2159 (secure 0 0 0) 0x7f646c048e50 con 0x7f647414c650 2020-11-20T12:18:42.908 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:42.908 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":125,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:37.935677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:42.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.906+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6450056d20 msgr2=0x7f64500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:42.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6450056d20 0x7f64500591e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f64640092d0 tx=0x7f6464016000).stop 2020-11-20T12:18:42.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f647414c650 msgr2=0x7f647409ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:42.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f647414c650 0x7f647409ffa0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 rx=0x7f646c00ec30 tx=0x7f646c006df0).stop 2020-11-20T12:18:42.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 shutdown_connections 2020-11-20T12:18:42.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6450056d20 0x7f64500591e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:42.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f64740657d0 0x7f647414cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:18:42.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f647414c650 0x7f647409ffa0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:42.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 --2- 172.21.15.187:0/2298172558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6474066270 0x7f6474149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:42.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 >> 172.21.15.187:0/2298172558 conn(0x7f64740bab90 msgr2=0x7f64740b7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:42.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 shutdown_connections 2020-11-20T12:18:42.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:42.907+0000 7f647359e700 1 -- 172.21.15.187:0/2298172558 wait complete. 2020-11-20T12:18:42.914 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 125 2020-11-20T12:18:42.936 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:18:43.223 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2020-11-20T12:18:43.224 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:18:43.225 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2020-11-20T12:18:45.227 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:45.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.473+0000 7f46b6df4700 1 Processor -- start 2020-11-20T12:18:45.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.474+0000 7f46b6df4700 1 -- start start 2020-11-20T12:18:45.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.475+0000 7f46b6df4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 0x7f46b0062d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:45.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.475+0000 7f46b6df4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b014caa0 con 0x7f46b0149550 2020-11-20T12:18:45.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.475+0000 7f46b6df4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b01489a0 con 0x7f46b00679d0 2020-11-20T12:18:45.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.475+0000 7f46b6df4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b014c750 con 0x7f46b00675f0 2020-11-20T12:18:45.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.475+0000 7f46b55f1700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f46b00679d0 0x7f46b0148e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48782/0 (socket says 172.21.15.187:48782) 2020-11-20T12:18:45.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.475+0000 7f46b55f1700 1 -- 172.21.15.187:0/1607536473 learned_addr learned my addr 172.21.15.187:0/1607536473 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:18:45.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878593940 0 0) 0x7f46b01489a0 con 0x7f46b00679d0 2020-11-20T12:18:45.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4698003240 con 0x7f46b00679d0 2020-11-20T12:18:45.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b5df2700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 0x7f46b0062d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:45.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3748385277 0 0) 0x7f4698003240 con 0x7f46b00679d0 2020-11-20T12:18:45.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 msgr2=0x7f46b0062d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:45.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 0x7f46b0062d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 >> v1:172.21.15.5:6789/0 conn(0x7f46b0149550 legacy=0x7f46b01479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:45.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b0068430 con 0x7f46b00679d0 2020-11-20T12:18:45.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f46a00014e0 con 0x7f46b00679d0 2020-11-20T12:18:45.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 >> v1:172.21.15.187:6789/0 conn(0x7f46b00679d0 legacy=0x7f46b0148e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:45.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.476+0000 7f46b4df0700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46980037b0 0x7f4698003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:45.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b4df0700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46980044e0 0x7f4698004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:45.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b4df0700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 0x7f4698004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:45.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b014c750 con 0x7f46980044e0 2020-11-20T12:18:45.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4698005340 con 0x7f46980037b0 2020-11-20T12:18:45.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469800b730 con 0x7f46b00675f0 2020-11-20T12:18:45.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b5df2700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46980037b0 0x7f4698003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:45.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b55f1700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46980044e0 0x7f4698004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:45.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b65f3700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 0x7f4698004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:45.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.477+0000 7f46b5df2700 1 -- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 msgr2=0x7f4698004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:45.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b5df2700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 0x7f4698004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b5df2700 1 -- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46980044e0 msgr2=0x7f4698004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:45.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b5df2700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46980044e0 0x7f4698004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b5df2700 1 -- 172.21.15.187:0/1607536473 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4698003240 con 0x7f46980037b0 2020-11-20T12:18:45.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b65f3700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00675f0 0x7f4698004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:18:45.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b5df2700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46980037b0 0x7f4698003bb0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7f46ac009d80 tx=0x7f46ac008040).ready entity=mon.1 client_cookie=4775a4105af6ec85 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:45.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00c070 con 0x7f46980037b0 2020-11-20T12:18:45.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac00ea80 con 0x7f46980037b0 2020-11-20T12:18:45.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.478+0000 7f46b4df0700 1 -- 172.21.15.187:0/1607536473 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00ee40 con 0x7f46980037b0 2020-11-20T12:18:45.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.479+0000 7f46b6df4700 1 -- 172.21.15.187:0/1607536473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46980037b0 msgr2=0x7f4698003bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:45.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.479+0000 7f46b6df4700 1 --2- 172.21.15.187:0/1607536473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46980037b0 0x7f4698003bb0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7f46ac009d80 tx=0x7f46ac008040).stop 2020-11-20T12:18:45.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.479+0000 7f46b65f3700 1 -- 172.21.15.187:0/1607536473 reap_dead start 2020-11-20T12:18:45.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.479+0000 7f46b6df4700 1 -- 172.21.15.187:0/1607536473 shutdown_connections 2020-11-20T12:18:45.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.479+0000 7f46b6df4700 1 -- 172.21.15.187:0/1607536473 >> 172.21.15.187:0/1607536473 conn(0x7f46b00bcd80 msgr2=0x7f46b00bd190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:45.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.479+0000 7f46b6df4700 1 -- 172.21.15.187:0/1607536473 shutdown_connections 2020-11-20T12:18:45.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.479+0000 7f46b6df4700 1 -- 172.21.15.187:0/1607536473 wait complete. 2020-11-20T12:18:45.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.481+0000 7f46b6df4700 1 Processor -- start 2020-11-20T12:18:45.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.481+0000 7f46b6df4700 1 -- start start 2020-11-20T12:18:45.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.481+0000 7f46b6df4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46b009b1b0 0x7f46b009b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:45.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b6df4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00b7680 0x7f46b008ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:45.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b6df4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46b00a1de0 0x7f46b00a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:45.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b6df4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b008eef0 con 0x7f46b009b1b0 2020-11-20T12:18:45.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b6df4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b008f110 con 0x7f46b00a1de0 2020-11-20T12:18:45.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b6df4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b008f3b0 con 0x7f46b00b7680 2020-11-20T12:18:45.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b65f3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46b00a1de0 0x7f46b00a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:45.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b65f3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46b00a1de0 0x7f46b00a2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52190/0 (socket says 172.21.15.187:52190) 2020-11-20T12:18:45.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b65f3700 1 -- 172.21.15.187:0/92251233 learned_addr learned my addr 172.21.15.187:0/92251233 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:18:45.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b65f3700 1 -- 172.21.15.187:0/92251233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00b7680 msgr2=0x7f46b008ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:45.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b55f1700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00b7680 0x7f46b008ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:45.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b5df2700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46b009b1b0 0x7f46b009b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:45.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b65f3700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00b7680 0x7f46b008ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b65f3700 1 -- 172.21.15.187:0/92251233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46b009b1b0 msgr2=0x7f46b009b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:45.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b65f3700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46b009b1b0 0x7f46b009b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.482+0000 7f46b65f3700 1 -- 172.21.15.187:0/92251233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b008f650 con 0x7f46b00a1de0 2020-11-20T12:18:45.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.483+0000 7f46b65f3700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46b00a1de0 0x7f46b00a2260 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7f46b00bcc50 tx=0x7f46a8009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:45.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.483+0000 7f469dffb700 1 -- 172.21.15.187:0/92251233 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a80104e0 con 0x7f46b00a1de0 2020-11-20T12:18:45.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.483+0000 7f469dffb700 1 -- 172.21.15.187:0/92251233 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a8010640 con 0x7f46b00a1de0 2020-11-20T12:18:45.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.483+0000 7f469dffb700 1 -- 172.21.15.187:0/92251233 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a80109e0 con 0x7f46b00a1de0 2020-11-20T12:18:45.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.483+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b0145ff0 con 0x7f46b00a1de0 2020-11-20T12:18:45.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.483+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b0158290 con 0x7f46b00a1de0 2020-11-20T12:18:45.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.484+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b0065710 con 0x7f46b00a1de0 2020-11-20T12:18:45.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.485+0000 7f469dffb700 1 -- 172.21.15.187:0/92251233 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f46a801a030 con 0x7f46b00a1de0 2020-11-20T12:18:45.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.486+0000 7f469dffb700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4688056d20 0x7f46880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:45.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.486+0000 7f469dffb700 1 -- 172.21.15.187:0/92251233 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f46a806f680 con 0x7f46b00a1de0 2020-11-20T12:18:45.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.486+0000 7f46b5df2700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4688056d20 0x7f46880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:45.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.487+0000 7f46b5df2700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4688056d20 0x7f46880591e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f46b00bdae0 tx=0x7f46ac008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:45.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.490+0000 7f469dffb700 1 -- 172.21.15.187:0/92251233 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f46a8044b30 con 0x7f46b00a1de0 2020-11-20T12:18:45.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.734+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46b0069cd0 con 0x7f46b00a1de0 2020-11-20T12:18:45.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.736+0000 7f469dffb700 1 -- 172.21.15.187:0/92251233 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+2159 (secure 0 0 0) 0x7f46a8048150 con 0x7f46b00a1de0 2020-11-20T12:18:45.740 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:18:45.740 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":125,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:18:37.935677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:18:45.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4688056d20 msgr2=0x7f46880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:45.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4688056d20 0x7f46880591e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f46b00bdae0 tx=0x7f46ac008040).stop 2020-11-20T12:18:45.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46b00a1de0 msgr2=0x7f46b00a2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:45.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46b00a1de0 0x7f46b00a2260 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7f46b00bcc50 tx=0x7f46a8009040).stop 2020-11-20T12:18:45.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 shutdown_connections 2020-11-20T12:18:45.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4688056d20 0x7f46880591e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46b009b1b0 0x7f46b009b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46b00a1de0 0x7f46b00a2260 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 --2- 172.21.15.187:0/92251233 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f46b00b7680 0x7f46b008ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:45.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 >> 172.21.15.187:0/92251233 conn(0x7f46b00bcd80 msgr2=0x7f46b009dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:45.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.738+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 shutdown_connections 2020-11-20T12:18:45.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:18:45.739+0000 7f46b6df4700 1 -- 172.21.15.187:0/92251233 wait complete. 2020-11-20T12:18:45.745 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 125 2020-11-20T12:18:45.763 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:18:46.065 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"16","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/delete_me","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"11","version":"11","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2020-11-20T12:18:46.066 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/delete_me", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "11", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "11" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:18:46.066 INFO:tasks.cephfs.cephfs_test_case:[('/delete_me', 1)] =?= [('/delete_me', 1)] 2020-11-20T12:18:46.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:18:46.067 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -rf delete_me/*) 2020-11-20T12:18:46.213 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:18:46.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:18:46.214 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:18:46.391 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:18:46.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:18:46.393 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:18:46.449 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:18:46.747 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_created":5}} 2020-11-20T12:18:46.748 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 5 } } 2020-11-20T12:18:46.748 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:18:47.022 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_enqueued":0}} 2020-11-20T12:18:47.023 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2020-11-20T12:18:47.023 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 5, waiting (timeout=60)... 2020-11-20T12:18:52.025 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:18:52.349 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"strays_enqueued":5}} 2020-11-20T12:18:52.350 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 5 } } 2020-11-20T12:18:52.350 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache num_strays 2020-11-20T12:18:52.647 INFO:teuthology.orchestra.run.smithi005.stdout:{"mds_cache":{"num_strays":0}} 2020-11-20T12:18:52.648 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2020-11-20T12:18:53.356 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:18:53.378 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:02.649 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump purge_queue pq_executed 2020-11-20T12:19:02.944 INFO:teuthology.orchestra.run.smithi005.stdout:{"purge_queue":{"pq_executed":0}} 2020-11-20T12:19:02.945 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 0 } } 2020-11-20T12:19:02.946 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2020-11-20T12:19:03.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.201+0000 7f8b5c957700 1 Processor -- start 2020-11-20T12:19:03.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.202+0000 7f8b5c957700 1 -- start start 2020-11-20T12:19:03.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.203+0000 7f8b5c957700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b58148a50 0x7f8b58148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:03.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.203+0000 7f8b5c957700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b58148620 con 0x7f8b580b5bc0 2020-11-20T12:19:03.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.203+0000 7f8b5c957700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b58062560 con 0x7f8b58099fe0 2020-11-20T12:19:03.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.203+0000 7f8b5c957700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58144e50 con 0x7f8b58148a50 2020-11-20T12:19:03.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.204+0000 7f8b577fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8b58099fe0 0x7f8b58061e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48812/0 (socket says 172.21.15.187:48812) 2020-11-20T12:19:03.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.204+0000 7f8b577fe700 1 -- 172.21.15.187:0/347784458 learned_addr learned my addr 172.21.15.187:0/347784458 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:03.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.204+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822861961 0 0) 0x7f8b58062560 con 0x7f8b58099fe0 2020-11-20T12:19:03.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.204+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b38004350 con 0x7f8b58099fe0 2020-11-20T12:19:03.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.204+0000 7f8b57fff700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b58148a50 0x7f8b58148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:03.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4130484950 0 0) 0x7f8b38004350 con 0x7f8b58099fe0 2020-11-20T12:19:03.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b58148a50 msgr2=0x7f8b58148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:03.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b58148a50 0x7f8b58148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:03.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 >> v1:172.21.15.5:6789/0 conn(0x7f8b580b5bc0 legacy=0x7f8b58065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:03.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b580b00d0 con 0x7f8b58099fe0 2020-11-20T12:19:03.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8b480014e0 con 0x7f8b58099fe0 2020-11-20T12:19:03.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 >> v1:172.21.15.187:6789/0 conn(0x7f8b58099fe0 legacy=0x7f8b58061e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:03.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b58148a50 0x7f8b38004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:03.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b38004d50 0x7f8b380055f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:03.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b38005ae0 0x7f8b38005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:03.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58144e50 con 0x7f8b38005ae0 2020-11-20T12:19:03.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b380052e0 con 0x7f8b38004d50 2020-11-20T12:19:03.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3800c840 con 0x7f8b58148a50 2020-11-20T12:19:03.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.205+0000 7f8b4effd700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b38004d50 0x7f8b380055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:03.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b57fff700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b38005ae0 0x7f8b38005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:03.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b4effd700 1 -- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b58148a50 msgr2=0x7f8b38004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:03.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b4effd700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b58148a50 0x7f8b38004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:03.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b4effd700 1 -- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b38005ae0 msgr2=0x7f8b38005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:03.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b4effd700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b38005ae0 0x7f8b38005f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:03.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b4effd700 1 -- 172.21.15.187:0/347784458 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b38004350 con 0x7f8b38004d50 2020-11-20T12:19:03.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b4effd700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b38004d50 0x7f8b380055f0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 rx=0x7f8b40001920 tx=0x7f8b40007b90).ready entity=mon.1 client_cookie=229f1ff3ef597db2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:03.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40007020 con 0x7f8b38004d50 2020-11-20T12:19:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40007180 con 0x7f8b38004d50 2020-11-20T12:19:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.206+0000 7f8b56ffd700 1 -- 172.21.15.187:0/347784458 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4000c5c0 con 0x7f8b38004d50 2020-11-20T12:19:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.207+0000 7f8b5c957700 1 -- 172.21.15.187:0/347784458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b38004d50 msgr2=0x7f8b380055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.207+0000 7f8b5c957700 1 --2- 172.21.15.187:0/347784458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b38004d50 0x7f8b380055f0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 rx=0x7f8b40001920 tx=0x7f8b40007b90).stop 2020-11-20T12:19:03.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.207+0000 7f8b57fff700 1 -- 172.21.15.187:0/347784458 reap_dead start 2020-11-20T12:19:03.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.207+0000 7f8b5c957700 1 -- 172.21.15.187:0/347784458 shutdown_connections 2020-11-20T12:19:03.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.207+0000 7f8b5c957700 1 -- 172.21.15.187:0/347784458 >> 172.21.15.187:0/347784458 conn(0x7f8b580babf0 msgr2=0x7f8b580bb000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:03.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.207+0000 7f8b5c957700 1 -- 172.21.15.187:0/347784458 shutdown_connections 2020-11-20T12:19:03.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.207+0000 7f8b5c957700 1 -- 172.21.15.187:0/347784458 wait complete. 2020-11-20T12:19:03.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.209+0000 7f8b5c957700 1 Processor -- start 2020-11-20T12:19:03.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.209+0000 7f8b5c957700 1 -- start start 2020-11-20T12:19:03.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b5c957700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b5814ba40 0x7f8b580a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:03.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b5c957700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b5814be20 0x7f8b5814ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:03.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b5c957700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b5814b130 0x7f8b58159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:03.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b5c957700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580b25b0 con 0x7f8b5814ba40 2020-11-20T12:19:03.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b5c957700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580b2850 con 0x7f8b5814be20 2020-11-20T12:19:03.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b5c957700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58149060 con 0x7f8b5814b130 2020-11-20T12:19:03.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b4effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b5814be20 0x7f8b5814ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:03.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b4effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b5814be20 0x7f8b5814ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52218/0 (socket says 172.21.15.187:52218) 2020-11-20T12:19:03.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.210+0000 7f8b4effd700 1 -- 172.21.15.187:0/1376681119 learned_addr learned my addr 172.21.15.187:0/1376681119 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:03.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b577fe700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b5814ba40 0x7f8b580a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:03.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b57fff700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b5814b130 0x7f8b58159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:03.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4effd700 1 -- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b5814b130 msgr2=0x7f8b58159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:03.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4effd700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b5814b130 0x7f8b58159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:03.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4effd700 1 -- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b5814ba40 msgr2=0x7f8b580a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:03.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4effd700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b5814ba40 0x7f8b580a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:03.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4effd700 1 -- 172.21.15.187:0/1376681119 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b580b22a0 con 0x7f8b5814be20 2020-11-20T12:19:03.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b577fe700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b5814ba40 0x7f8b580a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:03.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4effd700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b5814be20 0x7f8b5814ac40 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f8b40000fd0 tx=0x7f8b40015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:03.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4ffff700 1 -- 172.21.15.187:0/1376681119 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40013a90 con 0x7f8b5814be20 2020-11-20T12:19:03.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4ffff700 1 -- 172.21.15.187:0/1376681119 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40013bf0 con 0x7f8b5814be20 2020-11-20T12:19:03.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5814b590 con 0x7f8b5814be20 2020-11-20T12:19:03.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b4ffff700 1 -- 172.21.15.187:0/1376681119 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40020440 con 0x7f8b5814be20 2020-11-20T12:19:03.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.211+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5815a270 con 0x7f8b5814be20 2020-11-20T12:19:03.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.213+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b580b36e0 con 0x7f8b5814be20 2020-11-20T12:19:03.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.213+0000 7f8b4ffff700 1 -- 172.21.15.187:0/1376681119 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8b40026020 con 0x7f8b5814be20 2020-11-20T12:19:03.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.214+0000 7f8b4ffff700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d20 0x7f8b340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:03.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.214+0000 7f8b4ffff700 1 -- 172.21.15.187:0/1376681119 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8b400714e0 con 0x7f8b5814be20 2020-11-20T12:19:03.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.214+0000 7f8b577fe700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d20 0x7f8b340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:03.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.215+0000 7f8b577fe700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d20 0x7f8b340591e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f8b48007840 tx=0x7f8b48009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:03.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.218+0000 7f8b4ffff700 1 -- 172.21.15.187:0/1376681119 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8b400468e0 con 0x7f8b5814be20 2020-11-20T12:19:03.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.465+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f8b5805c2b0 con 0x7f8b5814be20 2020-11-20T12:19:03.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.863+0000 7f8b4ffff700 1 -- 172.21.15.187:0/1376681119 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v128) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8b40049f00 con 0x7f8b5814be20 2020-11-20T12:19:03.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d20 msgr2=0x7f8b340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:03.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d20 0x7f8b340591e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f8b48007840 tx=0x7f8b48009040).stop 2020-11-20T12:19:03.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b5814be20 msgr2=0x7f8b5814ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:03.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b5814be20 0x7f8b5814ac40 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f8b40000fd0 tx=0x7f8b40015000).stop 2020-11-20T12:19:03.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 shutdown_connections 2020-11-20T12:19:03.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8b34056d20 0x7f8b340591e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:03.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b5814ba40 0x7f8b580a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:03.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8b5814b130 0x7f8b58159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:03.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 --2- 172.21.15.187:0/1376681119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b5814be20 0x7f8b5814ac40 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:03.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 >> 172.21.15.187:0/1376681119 conn(0x7f8b580babf0 msgr2=0x7f8b580bb000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:03.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 shutdown_connections 2020-11-20T12:19:03.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:03.867+0000 7f8b5c957700 1 -- 172.21.15.187:0/1376681119 wait complete. 2020-11-20T12:19:13.902 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump purge_queue pq_executed 2020-11-20T12:19:14.197 INFO:teuthology.orchestra.run.smithi005.stdout:{"purge_queue":{"pq_executed":0}} 2020-11-20T12:19:14.198 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 0 } } 2020-11-20T12:19:14.199 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:14.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.448+0000 7f73121d3700 1 Processor -- start 2020-11-20T12:19:14.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.448+0000 7f73121d3700 1 -- start start 2020-11-20T12:19:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.449+0000 7f73121d3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f730c0654a0 0x7f730c0675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.449+0000 7f73121d3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 0x7f730c0afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.449+0000 7f73121d3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c099f70 con 0x7f730c065880 2020-11-20T12:19:14.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.449+0000 7f73121d3700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c0b6a20 con 0x7f730c0654a0 2020-11-20T12:19:14.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.449+0000 7f73121d3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c148990 con 0x7f730c0670c0 2020-11-20T12:19:14.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.449+0000 7f73111d1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f730c0654a0 0x7f730c0675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:14.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.449+0000 7f73109d0700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f730c065880 0x7f730c148280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48830/0 (socket says 172.21.15.187:48830) 2020-11-20T12:19:14.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.449+0000 7f73109d0700 1 -- 172.21.15.187:0/2195013006 learned_addr learned my addr 172.21.15.187:0/2195013006 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f73119d2700 1 --2- 172.21.15.187:0/2195013006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 0x7f730c0afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:14.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f73111d1700 1 -- 172.21.15.187:0/2195013006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 msgr2=0x7f730c0afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f73111d1700 1 --2- 172.21.15.187:0/2195013006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 0x7f730c0afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:14.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f72fbfff700 1 -- 172.21.15.187:0/2195013006 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705358701 0 0) 0x7f730c099f70 con 0x7f730c065880 2020-11-20T12:19:14.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f73111d1700 1 -- 172.21.15.187:0/2195013006 >> v1:172.21.15.187:6789/0 conn(0x7f730c065880 legacy=0x7f730c148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:14.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f73111d1700 1 -- 172.21.15.187:0/2195013006 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0affe0 con 0x7f730c0654a0 2020-11-20T12:19:14.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f73119d2700 1 --2- 172.21.15.187:0/2195013006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 0x7f730c0afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:14.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f73111d1700 1 --2- 172.21.15.187:0/2195013006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c0654a0 0x7f730c0675e0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 rx=0x7f730800b9a0 tx=0x7f7308006df0).ready entity=mon.1 client_cookie=d5fbf82f4ad97837 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:14.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f72fbfff700 1 -- 172.21.15.187:0/2195013006 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73080097b0 con 0x7f730c0654a0 2020-11-20T12:19:14.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.450+0000 7f72fbfff700 1 -- 172.21.15.187:0/2195013006 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7308009910 con 0x7f730c0654a0 2020-11-20T12:19:14.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f72fbfff700 1 -- 172.21.15.187:0/2195013006 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308009c90 con 0x7f730c0654a0 2020-11-20T12:19:14.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f73121d3700 1 -- 172.21.15.187:0/2195013006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c0654a0 msgr2=0x7f730c0675e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:14.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f73121d3700 1 --2- 172.21.15.187:0/2195013006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c0654a0 0x7f730c0675e0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 rx=0x7f730800b9a0 tx=0x7f7308006df0).stop 2020-11-20T12:19:14.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f73121d3700 1 -- 172.21.15.187:0/2195013006 shutdown_connections 2020-11-20T12:19:14.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f73121d3700 1 --2- 172.21.15.187:0/2195013006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 0x7f730c0afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:14.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f73121d3700 1 --2- 172.21.15.187:0/2195013006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c0654a0 0x7f730c0675e0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:14.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f73121d3700 1 -- 172.21.15.187:0/2195013006 >> 172.21.15.187:0/2195013006 conn(0x7f730c0bab90 msgr2=0x7f730c0bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:14.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f73121d3700 1 -- 172.21.15.187:0/2195013006 shutdown_connections 2020-11-20T12:19:14.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.451+0000 7f73121d3700 1 -- 172.21.15.187:0/2195013006 wait complete. 2020-11-20T12:19:14.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.453+0000 7f73121d3700 1 Processor -- start 2020-11-20T12:19:14.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.453+0000 7f73121d3700 1 -- start start 2020-11-20T12:19:14.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.454+0000 7f73121d3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f730c0654a0 0x7f730c0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:14.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.454+0000 7f73121d3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c065880 0x7f730c13d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:14.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.454+0000 7f73121d3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 0x7f730c13dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:14.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.454+0000 7f73121d3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0b1c90 con 0x7f730c0654a0 2020-11-20T12:19:14.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.454+0000 7f73121d3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0b2580 con 0x7f730c065880 2020-11-20T12:19:14.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.454+0000 7f73121d3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0b19f0 con 0x7f730c0670c0 2020-11-20T12:19:14.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c065880 0x7f730c13d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:14.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c065880 0x7f730c13d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52232/0 (socket says 172.21.15.187:52232) 2020-11-20T12:19:14.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 -- 172.21.15.187:0/760429199 learned_addr learned my addr 172.21.15.187:0/760429199 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:14.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73111d1700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f730c0654a0 0x7f730c0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:14.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 -- 172.21.15.187:0/760429199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 msgr2=0x7f730c13dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:14.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 0x7f730c13dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:14.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 -- 172.21.15.187:0/760429199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f730c0654a0 msgr2=0x7f730c0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:14.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f730c0654a0 0x7f730c0b3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:14.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 -- 172.21.15.187:0/760429199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0a0ac0 con 0x7f730c065880 2020-11-20T12:19:14.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.455+0000 7f73109d0700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c065880 0x7f730c13d390 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 rx=0x7f72fc00c720 tx=0x7f72fc007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:14.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.456+0000 7f72f97fa700 1 -- 172.21.15.187:0/760429199 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc007320 con 0x7f730c065880 2020-11-20T12:19:14.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.456+0000 7f72f97fa700 1 -- 172.21.15.187:0/760429199 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc007480 con 0x7f730c065880 2020-11-20T12:19:14.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.456+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c13d880 con 0x7f730c065880 2020-11-20T12:19:14.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.456+0000 7f72f97fa700 1 -- 172.21.15.187:0/760429199 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc005800 con 0x7f730c065880 2020-11-20T12:19:14.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.456+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c13e210 con 0x7f730c065880 2020-11-20T12:19:14.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.457+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c061ad0 con 0x7f730c065880 2020-11-20T12:19:14.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.457+0000 7f72f97fa700 1 -- 172.21.15.187:0/760429199 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f72fc010030 con 0x7f730c065880 2020-11-20T12:19:14.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.458+0000 7f72f97fa700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f72e4056d20 0x7f72e40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:14.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.458+0000 7f72f97fa700 1 -- 172.21.15.187:0/760429199 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f72fc06fd50 con 0x7f730c065880 2020-11-20T12:19:14.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.459+0000 7f73111d1700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f72e4056d20 0x7f72e40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:14.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.459+0000 7f73111d1700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f72e4056d20 0x7f72e40591e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f7308006e30 tx=0x7f7308015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:14.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.463+0000 7f72f97fa700 1 -- 172.21.15.187:0/760429199 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f72fc045150 con 0x7f730c065880 2020-11-20T12:19:14.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.712+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f730c0b2710 con 0x7f730c065880 2020-11-20T12:19:14.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.713+0000 7f72f97fa700 1 -- 172.21.15.187:0/760429199 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+2163 (secure 0 0 0) 0x7f72fc007c00 con 0x7f730c065880 2020-11-20T12:19:14.716 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:14.717 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":129,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:19:03.863741+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:19:14.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f72e4056d20 msgr2=0x7f72e40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:14.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f72e4056d20 0x7f72e40591e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f7308006e30 tx=0x7f7308015000).stop 2020-11-20T12:19:14.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c065880 msgr2=0x7f730c13d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:14.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c065880 0x7f730c13d390 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 rx=0x7f72fc00c720 tx=0x7f72fc007ac0).stop 2020-11-20T12:19:14.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 shutdown_connections 2020-11-20T12:19:14.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f72e4056d20 0x7f72e40591e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:14.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f730c0654a0 0x7f730c0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:14.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f730c0670c0 0x7f730c13dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:14.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 --2- 172.21.15.187:0/760429199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f730c065880 0x7f730c13d390 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:14.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 >> 172.21.15.187:0/760429199 conn(0x7f730c0bab90 msgr2=0x7f730c14a130 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:14.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 shutdown_connections 2020-11-20T12:19:14.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.716+0000 7f73121d3700 1 -- 172.21.15.187:0/760429199 wait complete. 2020-11-20T12:19:14.722 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 129 2020-11-20T12:19:14.743 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b injectargs '--mds_max_purge_files 100' 2020-11-20T12:19:14.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.987+0000 7fe34f59e700 1 Processor -- start 2020-11-20T12:19:14.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.988+0000 7fe34f59e700 1 -- start start 2020-11-20T12:19:14.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.989+0000 7fe34f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe3500654a0 0x7fe350149290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:14.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.989+0000 7fe34f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35009a930 con 0x7fe350065880 2020-11-20T12:19:14.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.989+0000 7fe34f59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3500b0be0 con 0x7fe3500b1110 2020-11-20T12:19:14.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.989+0000 7fe34f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe350062b70 con 0x7fe3500654a0 2020-11-20T12:19:14.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.989+0000 7fe34ed9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe3500b1110 0x7fe350062460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48848/0 (socket says 172.21.15.187:48848) 2020-11-20T12:19:14.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.990+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3867998570 learned_addr learned my addr 172.21.15.187:0/3867998570 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:14.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.990+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3867998570 >> v2:172.21.15.5:3300/0 conn(0x7fe3500654a0 0x7fe350149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:14.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.990+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866652556 0 0) 0x7fe3500b0be0 con 0x7fe3500b1110 2020-11-20T12:19:14.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.990+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe330003240 con 0x7fe3500b1110 2020-11-20T12:19:14.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.990+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 306848601 0 0) 0x7fe330003240 con 0x7fe3500b1110 2020-11-20T12:19:14.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.990+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 >> v2:172.21.15.5:3300/0 conn(0x7fe3500654a0 msgr2=0x7fe350149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:14.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.990+0000 7fe34dd9b700 1 --2- 172.21.15.187:0/3867998570 >> v2:172.21.15.5:3300/0 conn(0x7fe3500654a0 0x7fe350149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:14.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 >> v1:172.21.15.5:6789/0 conn(0x7fe350065880 legacy=0x7fe350061d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:14.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3500b71b0 con 0x7fe3500b1110 2020-11-20T12:19:14.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fe3480018a0 con 0x7fe3500b1110 2020-11-20T12:19:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 >> v1:172.21.15.187:6789/0 conn(0x7fe3500b1110 legacy=0x7fe350062460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe330003800 0x7fe330003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3300044e0 0x7fe330004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe33000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe350062b70 con 0x7fe330003800 2020-11-20T12:19:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330005300 con 0x7fe330004e50 2020-11-20T12:19:15.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33000bc20 con 0x7fe3300044e0 2020-11-20T12:19:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.991+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe33000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe330003800 0x7fe330003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3867998570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3300044e0 msgr2=0x7fe330004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3300044e0 0x7fe330004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3867998570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe330003800 msgr2=0x7fe330003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe330003800 0x7fe330003c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3867998570 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe330003240 con 0x7fe330004e50 2020-11-20T12:19:15.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3867998570 reap_dead start 2020-11-20T12:19:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe33000b730 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 rx=0x7fe34800c810 tx=0x7fe3500b0df0).ready entity=mon.1 client_cookie=377c4c75250ef6ae server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348005b80 con 0x7fe330004e50 2020-11-20T12:19:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348005ce0 con 0x7fe330004e50 2020-11-20T12:19:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.992+0000 7fe34dd9b700 1 -- 172.21.15.187:0/3867998570 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348007320 con 0x7fe330004e50 2020-11-20T12:19:15.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.993+0000 7fe34f59e700 1 -- 172.21.15.187:0/3867998570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 msgr2=0x7fe33000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.993+0000 7fe34f59e700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe33000b730 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 rx=0x7fe34800c810 tx=0x7fe3500b0df0).stop 2020-11-20T12:19:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.993+0000 7fe34f59e700 1 -- 172.21.15.187:0/3867998570 shutdown_connections 2020-11-20T12:19:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.993+0000 7fe34f59e700 1 --2- 172.21.15.187:0/3867998570 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe33000b730 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.993+0000 7fe34f59e700 1 -- 172.21.15.187:0/3867998570 >> 172.21.15.187:0/3867998570 conn(0x7fe3500bab90 msgr2=0x7fe3500bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.993+0000 7fe34f59e700 1 -- 172.21.15.187:0/3867998570 shutdown_connections 2020-11-20T12:19:15.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.994+0000 7fe34f59e700 1 -- 172.21.15.187:0/3867998570 wait complete. 2020-11-20T12:19:15.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.995+0000 7fe34f59e700 1 Processor -- start 2020-11-20T12:19:15.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.995+0000 7fe34f59e700 1 -- start start 2020-11-20T12:19:15.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.996+0000 7fe34f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe350096790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.996+0000 7fe34f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35009a3c0 0x7fe35009ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.996+0000 7fe34f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe350092e10 0x7fe350093230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.996+0000 7fe34f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe350096c80 con 0x7fe35009a3c0 2020-11-20T12:19:15.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.996+0000 7fe34f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35009f430 con 0x7fe330004e50 2020-11-20T12:19:15.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.996+0000 7fe34f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35009fe20 con 0x7fe350092e10 2020-11-20T12:19:15.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe350096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe350096790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52250/0 (socket says 172.21.15.187:52250) 2020-11-20T12:19:15.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 -- 172.21.15.187:0/3104274650 learned_addr learned my addr 172.21.15.187:0/3104274650 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:15.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 -- 172.21.15.187:0/3104274650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe350092e10 msgr2=0x7fe350093230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe350092e10 0x7fe350093230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 -- 172.21.15.187:0/3104274650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35009a3c0 msgr2=0x7fe35009ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35009a3c0 0x7fe35009ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 -- 172.21.15.187:0/3104274650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3500a00c0 con 0x7fe330004e50 2020-11-20T12:19:15.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.997+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe350096790 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7fe34000e760 tx=0x7fe340008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.998+0000 7fe346ffd700 1 -- 172.21.15.187:0/3104274650 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340010560 con 0x7fe330004e50 2020-11-20T12:19:15.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.998+0000 7fe346ffd700 1 -- 172.21.15.187:0/3104274650 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3400106c0 con 0x7fe330004e50 2020-11-20T12:19:15.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.998+0000 7fe34f59e700 1 -- 172.21.15.187:0/3104274650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3500a0360 con 0x7fe330004e50 2020-11-20T12:19:15.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.998+0000 7fe346ffd700 1 -- 172.21.15.187:0/3104274650 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340010a40 con 0x7fe330004e50 2020-11-20T12:19:15.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:14.998+0000 7fe34f59e700 1 -- 172.21.15.187:0/3104274650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe350093a10 con 0x7fe330004e50 2020-11-20T12:19:15.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.000+0000 7fe346ffd700 1 -- 172.21.15.187:0/3104274650 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe340012070 con 0x7fe330004e50 2020-11-20T12:19:15.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.001+0000 7fe346ffd700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe32c056d20 0x7fe32c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.001+0000 7fe346ffd700 1 -- 172.21.15.187:0/3104274650 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe34006f900 con 0x7fe330004e50 2020-11-20T12:19:15.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.001+0000 7fe345d9b700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe32c056d20 0x7fe32c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.002+0000 7fe345d9b700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe32c056d20 0x7fe32c0591e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fe350095340 tx=0x7fe3380077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.004+0000 7fe366654640 1 Processor -- start 2020-11-20T12:19:15.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.004+0000 7fe366654640 1 -- start start 2020-11-20T12:19:15.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.004+0000 7fe366654640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 0x5643e62b6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.004+0000 7fe366654640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e6002080 0x5643e6296ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.004+0000 7fe366654640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 0x5643e6290aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe366654640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e620f800 con 0x5643e62459a0 2020-11-20T12:19:15.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe366654640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e61bd8e0 con 0x5643e6002080 2020-11-20T12:19:15.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe366654640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5643e61bdb60 con 0x5643e6290620 2020-11-20T12:19:15.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe345d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e6002080 0x5643e6296ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe345d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e6002080 0x5643e6296ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52260/0 (socket says 172.21.15.187:52260) 2020-11-20T12:19:15.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe345d9b700 1 -- 172.21.15.187:0/1694642047 learned_addr learned my addr 172.21.15.187:0/1694642047 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:15.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe34e59c700 1 --2- 172.21.15.187:0/1694642047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 0x5643e62b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe345d9b700 1 -- 172.21.15.187:0/1694642047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 msgr2=0x5643e6290aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe345d9b700 1 --2- 172.21.15.187:0/1694642047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 0x5643e6290aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe345d9b700 1 -- 172.21.15.187:0/1694642047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 msgr2=0x5643e62b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe345d9b700 1 --2- 172.21.15.187:0/1694642047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 0x5643e62b6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.005+0000 7fe345d9b700 1 -- 172.21.15.187:0/1694642047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643e61add10 con 0x5643e6002080 2020-11-20T12:19:15.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.006+0000 7fe345d9b700 1 --2- 172.21.15.187:0/1694642047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e6002080 0x5643e6296ed0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7fe338016ba0 tx=0x7fe338010000).ready entity=mon.1 client_cookie=5a23c1755a566f86 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.006+0000 7fe3277fe700 1 -- 172.21.15.187:0/1694642047 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338016240 con 0x5643e6002080 2020-11-20T12:19:15.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.006+0000 7fe3277fe700 1 -- 172.21.15.187:0/1694642047 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3380163a0 con 0x5643e6002080 2020-11-20T12:19:15.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.006+0000 7fe3277fe700 1 -- 172.21.15.187:0/1694642047 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338014bf0 con 0x5643e6002080 2020-11-20T12:19:15.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.006+0000 7fe366654640 1 -- 172.21.15.187:0/1694642047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e6002080 msgr2=0x5643e6296ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.006+0000 7fe366654640 1 --2- 172.21.15.187:0/1694642047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e6002080 0x5643e6296ed0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7fe338016ba0 tx=0x7fe338010000).stop 2020-11-20T12:19:15.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.007+0000 7fe366654640 1 -- 172.21.15.187:0/1694642047 shutdown_connections 2020-11-20T12:19:15.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.007+0000 7fe366654640 1 --2- 172.21.15.187:0/1694642047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 0x5643e62b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.007+0000 7fe366654640 1 --2- 172.21.15.187:0/1694642047 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 0x5643e6290aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.007+0000 7fe366654640 1 --2- 172.21.15.187:0/1694642047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e6002080 0x5643e6296ed0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.007+0000 7fe366654640 1 -- 172.21.15.187:0/1694642047 >> 172.21.15.187:0/1694642047 conn(0x5643e625e2f0 msgr2=0x5643e6294440 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:15.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.007+0000 7fe366654640 1 -- 172.21.15.187:0/1694642047 shutdown_connections 2020-11-20T12:19:15.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.007+0000 7fe366654640 1 -- 172.21.15.187:0/1694642047 wait complete. 2020-11-20T12:19:15.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.007+0000 7fe366654640 1 -- start start 2020-11-20T12:19:15.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.008+0000 7fe366654640 1 Processor -- start 2020-11-20T12:19:15.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe366654640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe366654640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e62b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe366654640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 0x5643e62d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe366654640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e61bdca0 con 0x5643e6002080 2020-11-20T12:19:15.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe366654640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e61bead0 con 0x5643e62459a0 2020-11-20T12:19:15.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe366654640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5643e618e810 con 0x5643e6290620 2020-11-20T12:19:15.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe366654640 10 client.0 ll_register_callbacks cb 0x5643e6201b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2020-11-20T12:19:15.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe345d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e62b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe345d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e62b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52266/0 (socket says 172.21.15.187:52266) 2020-11-20T12:19:15.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe345d9b700 1 -- 172.21.15.187:0/3687570793 learned_addr learned my addr 172.21.15.187:0/3687570793 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:15.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 0x5643e62d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.009+0000 7fe345d9b700 1 -- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 msgr2=0x5643e62d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe345d9b700 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 0x5643e62d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe345d9b700 1 -- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 msgr2=0x5643e62b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe345d9b700 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe345d9b700 1 -- 172.21.15.187:0/3687570793 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643e61b60e0 con 0x5643e62459a0 2020-11-20T12:19:15.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:15.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe345d9b700 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e62b5270 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7fe338016510 tx=0x7fe33801b000).ready entity=mon.1 client_cookie=fd15cd9ed089a2b3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe324ff9700 10 client.0 ms_handle_connect on v2:172.21.15.187:3300/0 2020-11-20T12:19:15.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe324ff9700 1 -- 172.21.15.187:0/3687570793 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338016240 con 0x5643e62459a0 2020-11-20T12:19:15.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe366654640 1 -- 172.21.15.187:0/3687570793 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643e625d6e0 con 0x5643e62459a0 2020-11-20T12:19:15.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe324ff9700 1 -- 172.21.15.187:0/3687570793 <== mon.1 v2:172.21.15.187:3300/0 2 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe338000ba0 con 0x5643e62459a0 2020-11-20T12:19:15.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe324ff9700 1 -- 172.21.15.187:0/3687570793 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe33802b660 con 0x5643e62459a0 2020-11-20T12:19:15.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe366654640 10 client.9178 fetch_fsmap learned FSMap version 129 2020-11-20T12:19:15.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.010+0000 7fe366654640 1 -- 172.21.15.187:0/3687570793 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) v3 -- 0x5643e61b7500 con 0x5643e62459a0 2020-11-20T12:19:15.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe324ff9700 1 -- 172.21.15.187:0/3687570793 <== mon.1 v2:172.21.15.187:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5643e625d6e0 con 0x5643e62459a0 2020-11-20T12:19:15.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe324ff9700 1 -- 172.21.15.187:0/3687570793 <== mon.1 v2:172.21.15.187:3300/0 5 ==== fsmap(e 129) v1 ==== 1282+0+0 (secure 0 0 0) 0x7fe33801d030 con 0x5643e62459a0 2020-11-20T12:19:15.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe366654640 10 client.9178 fetch_fsmap finished waiting for FSMap version 129 2020-11-20T12:19:15.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe366654640 10 client.9178 resolve_mds: resolved ID 'b' to GID 5981 2020-11-20T12:19:15.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe366654640 20 client.9178 populate_metadata read hostname 'smithi187' 2020-11-20T12:19:15.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe366654640 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] conn(0x5643e62b2480 0x5643e62e2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe366654640 4 client.9178 mds_command: new command op to 5981 tid=0[{,",p,r,e,f,i,x,",:, ,",g,e,t,_,c,o,m,m,a,n,d,_,d,e,s,c,r,i,p,t,i,o,n,s,",}] 2020-11-20T12:19:15.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe366654640 1 -- 172.21.15.187:0/3687570793 --> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] -- command(tid 0: { " p r e f i x " : " g e t _ c o m m a n d _ d e s c r i p t i o n s " }) v1 -- 0x5643e605b9e0 con 0x5643e62b2480 2020-11-20T12:19:15.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.011+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] conn(0x5643e62b2480 0x5643e62e2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.012+0000 7fe324ff9700 10 client.9178 ms_handle_connect on v2:172.21.15.5:6832/4181204645 2020-11-20T12:19:15.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.012+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] conn(0x5643e62b2480 0x5643e62e2c00 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.1 client_cookie=2137635171d5cdc8 server_cookie=e10405cd1a254d34 in_seq=0 out_seq=0 2020-11-20T12:19:15.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.014+0000 7fe324ff9700 1 -- 172.21.15.187:0/3687570793 <== mds.1 v2:172.21.15.5:6832/4181204645 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x5643e605b9e0 con 0x5643e62b2480 2020-11-20T12:19:15.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.014+0000 7fe324ff9700 10 client.9178 handle_command_reply: tid=0 2020-11-20T12:19:15.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.014+0000 7fe366654640 1 client.9178 shutdown 2020-11-20T12:19:15.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.014+0000 7fe366654640 1 -- 172.21.15.187:0/3687570793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 msgr2=0x5643e62b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.014+0000 7fe366654640 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e62b5270 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7fe338016510 tx=0x7fe33801b000).stop 2020-11-20T12:19:15.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.014+0000 7fe366654640 1 -- 172.21.15.187:0/3687570793 shutdown_connections 2020-11-20T12:19:15.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe366654640 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] conn(0x5643e62b2480 0x5643e62e2c00 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe366654640 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe366654640 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6290620 0x5643e62d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe366654640 1 --2- 172.21.15.187:0/3687570793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e62b5270 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe324ff9700 0 client.9178 ms_handle_reset on v2:172.21.15.5:6832/4181204645 2020-11-20T12:19:15.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe366654640 1 -- 172.21.15.187:0/3687570793 >> 172.21.15.187:0/3687570793 conn(0x5643e625e2f0 msgr2=0x5643e62ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:15.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe366654640 1 -- 172.21.15.187:0/3687570793 shutdown_connections 2020-11-20T12:19:15.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe366654640 1 -- 172.21.15.187:0/3687570793 wait complete. 2020-11-20T12:19:15.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.015+0000 7fe366654640 20 client.9178 trim_cache size 0 max 16384 2020-11-20T12:19:15.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.033+0000 7fe366654640 1 Processor -- start 2020-11-20T12:19:15.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.034+0000 7fe366654640 1 -- start start 2020-11-20T12:19:15.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.034+0000 7fe366654640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e625e2f0 0x5643e6294570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.034+0000 7fe366654640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.034+0000 7fe366654640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e628a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.034+0000 7fe366654640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e62d3740 con 0x5643e6002080 2020-11-20T12:19:15.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.034+0000 7fe366654640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e625d6c0 con 0x5643e62459a0 2020-11-20T12:19:15.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.034+0000 7fe366654640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5643e62b3030 con 0x5643e625e2f0 2020-11-20T12:19:15.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe34ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e628a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe34ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e628a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52276/0 (socket says 172.21.15.187:52276) 2020-11-20T12:19:15.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe34ed9d700 1 -- 172.21.15.187:0/2302261444 learned_addr learned my addr 172.21.15.187:0/2302261444 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:15.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe345d9b700 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe34ed9d700 1 -- 172.21.15.187:0/2302261444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e625e2f0 msgr2=0x5643e6294570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e625e2f0 0x5643e6294570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe34ed9d700 1 -- 172.21.15.187:0/2302261444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 msgr2=0x5643e62d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62d1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe34ed9d700 1 -- 172.21.15.187:0/2302261444 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318008270 con 0x5643e62459a0 2020-11-20T12:19:15.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.035+0000 7fe345d9b700 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:15.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.036+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e628a9f0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7fe348006000 tx=0x7fe348013000).ready entity=mon.1 client_cookie=6af542826bf1412 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.036+0000 7fe327fff700 1 -- 172.21.15.187:0/2302261444 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348007690 con 0x5643e62459a0 2020-11-20T12:19:15.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.036+0000 7fe327fff700 1 -- 172.21.15.187:0/2302261444 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348001450 con 0x5643e62459a0 2020-11-20T12:19:15.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.036+0000 7fe327fff700 1 -- 172.21.15.187:0/2302261444 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348007830 con 0x5643e62459a0 2020-11-20T12:19:15.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.036+0000 7fe366654640 1 -- 172.21.15.187:0/2302261444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 msgr2=0x5643e628a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.036+0000 7fe366654640 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e628a9f0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7fe348006000 tx=0x7fe348013000).stop 2020-11-20T12:19:15.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.037+0000 7fe366654640 1 -- 172.21.15.187:0/2302261444 shutdown_connections 2020-11-20T12:19:15.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.037+0000 7fe366654640 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e62459a0 0x5643e628a9f0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.037+0000 7fe366654640 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e625e2f0 0x5643e6294570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.037+0000 7fe366654640 1 --2- 172.21.15.187:0/2302261444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e6002080 0x5643e62d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.037+0000 7fe366654640 1 -- 172.21.15.187:0/2302261444 >> 172.21.15.187:0/2302261444 conn(0x7fe318002320 msgr2=0x5643e62d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:15.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.037+0000 7fe366654640 1 -- 172.21.15.187:0/2302261444 shutdown_connections 2020-11-20T12:19:15.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.037+0000 7fe366654640 1 -- 172.21.15.187:0/2302261444 wait complete. 2020-11-20T12:19:15.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.037+0000 7fe366654640 1 -- start start 2020-11-20T12:19:15.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.038+0000 7fe366654640 1 Processor -- start 2020-11-20T12:19:15.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe366654640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6002080 0x5643e62d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe366654640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 0x5643e6337550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe366654640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e625e2f0 0x5643e6337a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe366654640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e628b640 con 0x5643e62459a0 2020-11-20T12:19:15.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe366654640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e62cf070 con 0x5643e625e2f0 2020-11-20T12:19:15.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe366654640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5643e62cf310 con 0x5643e6002080 2020-11-20T12:19:15.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe366654640 10 client.0 ll_register_callbacks cb 0x5643e62b3ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2020-11-20T12:19:15.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe34ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e625e2f0 0x5643e6337a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe34ed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e625e2f0 0x5643e6337a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52282/0 (socket says 172.21.15.187:52282) 2020-11-20T12:19:15.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3992524091 learned_addr learned my addr 172.21.15.187:0/3992524091 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:15.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6002080 0x5643e62d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.039+0000 7fe345d9b700 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 0x5643e6337550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6002080 msgr2=0x5643e62d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6002080 0x5643e62d1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 msgr2=0x5643e6337550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 0x5643e6337550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe34ed9d700 1 -- 172.21.15.187:0/3992524091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643e61b2ee0 con 0x5643e625e2f0 2020-11-20T12:19:15.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe34ed9d700 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e625e2f0 0x5643e6337a40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 rx=0x7fe348007da0 tx=0x7fe34801c000).ready entity=mon.1 client_cookie=af0df2acbd392c1d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe324ff9700 10 client.0 ms_handle_connect on v2:172.21.15.187:3300/0 2020-11-20T12:19:15.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe324ff9700 1 -- 172.21.15.187:0/3992524091 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348025070 con 0x5643e625e2f0 2020-11-20T12:19:15.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe366654640 1 -- 172.21.15.187:0/3992524091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3380163a0 con 0x5643e625e2f0 2020-11-20T12:19:15.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.040+0000 7fe324ff9700 1 -- 172.21.15.187:0/3992524091 <== mon.1 v2:172.21.15.187:3300/0 2 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe348020450 con 0x5643e625e2f0 2020-11-20T12:19:15.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe324ff9700 1 -- 172.21.15.187:0/3992524091 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe348020820 con 0x5643e625e2f0 2020-11-20T12:19:15.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe366654640 10 client.9184 fetch_fsmap learned FSMap version 129 2020-11-20T12:19:15.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe366654640 1 -- 172.21.15.187:0/3992524091 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) v3 -- 0x5643e5fe59c0 con 0x5643e625e2f0 2020-11-20T12:19:15.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe324ff9700 1 -- 172.21.15.187:0/3992524091 <== mon.1 v2:172.21.15.187:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3380163a0 con 0x5643e625e2f0 2020-11-20T12:19:15.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe324ff9700 1 -- 172.21.15.187:0/3992524091 <== mon.1 v2:172.21.15.187:3300/0 5 ==== fsmap(e 129) v1 ==== 1282+0+0 (secure 0 0 0) 0x7fe348013450 con 0x5643e625e2f0 2020-11-20T12:19:15.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe366654640 10 client.9184 fetch_fsmap finished waiting for FSMap version 129 2020-11-20T12:19:15.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe366654640 10 client.9184 resolve_mds: resolved ID 'b' to GID 5981 2020-11-20T12:19:15.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe366654640 20 client.9184 populate_metadata read hostname 'smithi187' 2020-11-20T12:19:15.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe366654640 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] conn(0x5643e6290620 0x5643e62d3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe366654640 4 client.9184 mds_command: new command op to 5981 tid=0[{,",p,r,e,f,i,x,",:, ,",i,n,j,e,c,t,a,r,g,s,",,, ,",i,n,j,e,c,t,e,d,_,a,r,g,s,",:, ,[,",-,-,m,d,s,_,m,a,x,_,p,u,r,g,e,_,f,i,l,e,s, ,1,0,0,",],}] 2020-11-20T12:19:15.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.041+0000 7fe366654640 1 -- 172.21.15.187:0/3992524091 --> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] -- command(tid 0: { " p r e f i x " : " i n j e c t a r g s " , " i n j e c t e d _ a r g s " : [ " - - m d s _ m a x _ p u r g e _ f i l e s 1 0 0 " ] }) v1 -- 0x5643e61bcf90 con 0x5643e6290620 2020-11-20T12:19:15.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.042+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] conn(0x5643e6290620 0x5643e62d3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.043+0000 7fe34e59c700 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] conn(0x5643e6290620 0x5643e62d3e20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.1 client_cookie=d284613418051d75 server_cookie=b3d1c56352a7bee7 in_seq=0 out_seq=0 2020-11-20T12:19:15.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.043+0000 7fe324ff9700 10 client.9184 ms_handle_connect on v2:172.21.15.5:6832/4181204645 2020-11-20T12:19:15.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe324ff9700 1 -- 172.21.15.187:0/3992524091 <== mds.1 v2:172.21.15.5:6832/4181204645 1 ==== command_reply(tid 0: 0 mds_max_purge_files = '100' ) v1 ==== 36+0+3 (crc 0 0 0) 0x5643e61bcf90 con 0x5643e6290620 2020-11-20T12:19:15.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe324ff9700 10 client.9184 handle_command_reply: tid=0 2020-11-20T12:19:15.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 client.9184 shutdown 2020-11-20T12:19:15.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 -- 172.21.15.187:0/3992524091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e625e2f0 msgr2=0x5643e6337a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e625e2f0 0x5643e6337a40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 rx=0x7fe348007da0 tx=0x7fe34801c000).stop 2020-11-20T12:19:15.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 -- 172.21.15.187:0/3992524091 shutdown_connections 2020-11-20T12:19:15.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:6832/4181204645,v1:172.21.15.5:6833/4181204645] conn(0x5643e6290620 0x5643e62d3e20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5643e625e2f0 0x5643e6337a40 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5643e6002080 0x5643e62d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe324ff9700 0 client.9184 ms_handle_reset on v2:172.21.15.5:6832/4181204645 2020-11-20T12:19:15.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 --2- 172.21.15.187:0/3992524091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5643e62459a0 0x5643e6337550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 -- 172.21.15.187:0/3992524091 >> 172.21.15.187:0/3992524091 conn(0x7fe318002320 msgr2=0x5643e62b63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:15.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 -- 172.21.15.187:0/3992524091 shutdown_connections 2020-11-20T12:19:15.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 1 -- 172.21.15.187:0/3992524091 wait complete. 2020-11-20T12:19:15.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.046+0000 7fe366654640 20 client.9184 trim_cache size 0 max 16384 2020-11-20T12:19:15.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 -- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe32c056d20 msgr2=0x7fe32c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe32c056d20 0x7fe32c0591e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fe350095340 tx=0x7fe3380077f0).stop 2020-11-20T12:19:15.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 -- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 msgr2=0x7fe350096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe350096790 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7fe34000e760 tx=0x7fe340008040).stop 2020-11-20T12:19:15.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 -- 172.21.15.187:0/3104274650 shutdown_connections 2020-11-20T12:19:15.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe32c056d20 0x7fe32c0591e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe350092e10 0x7fe350093230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe330004e50 0x7fe350096790 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 --2- 172.21.15.187:0/3104274650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35009a3c0 0x7fe35009ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:15.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 -- 172.21.15.187:0/3104274650 >> 172.21.15.187:0/3104274650 conn(0x7fe3500bab90 msgr2=0x7fe350061d20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:15.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 -- 172.21.15.187:0/3104274650 shutdown_connections 2020-11-20T12:19:15.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.047+0000 7fe34f59e700 1 -- 172.21.15.187:0/3104274650 wait complete. 2020-11-20T12:19:15.069 INFO:teuthology.orchestra.run.smithi187.stderr:mds_max_purge_files = '100' 2020-11-20T12:19:15.070 INFO:teuthology.orchestra.run.smithi187.stdout:{} 2020-11-20T12:19:15.075 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:15.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.310+0000 7f1e6ad7d700 1 Processor -- start 2020-11-20T12:19:15.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.310+0000 7f1e6ad7d700 1 -- start start 2020-11-20T12:19:15.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.311+0000 7f1e6ad7d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1e640654a0 0x7f1e640b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.311+0000 7f1e6ad7d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e64065880 0x7f1e6414c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.312+0000 7f1e6ad7d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6414c050 con 0x7f1e640b5190 2020-11-20T12:19:15.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.312+0000 7f1e6ad7d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e6414c2f0 con 0x7f1e640654a0 2020-11-20T12:19:15.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.312+0000 7f1e6ad7d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e64066400 con 0x7f1e64065880 2020-11-20T12:19:15.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.312+0000 7f1e6a57c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1e640b5190 0x7f1e640b6ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48894/0 (socket says 172.21.15.187:48894) 2020-11-20T12:19:15.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.312+0000 7f1e6a57c700 1 -- 172.21.15.187:0/2774577286 learned_addr learned my addr 172.21.15.187:0/2774577286 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:15.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.312+0000 7f1e6957a700 1 --2- 172.21.15.187:0/2774577286 >> v2:172.21.15.187:3300/0 conn(0x7f1e64065880 0x7f1e6414c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.312+0000 7f1e69d7b700 1 --2- 172.21.15.187:0/2774577286 >> v2:172.21.15.5:3300/0 conn(0x7f1e640654a0 0x7f1e640b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.313+0000 7f1e6957a700 1 -- 172.21.15.187:0/2774577286 >> v2:172.21.15.5:3300/0 conn(0x7f1e640654a0 msgr2=0x7f1e640b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:15.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.313+0000 7f1e6957a700 1 --2- 172.21.15.187:0/2774577286 >> v2:172.21.15.5:3300/0 conn(0x7f1e640654a0 0x7f1e640b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.313+0000 7f1e6957a700 1 -- 172.21.15.187:0/2774577286 >> v1:172.21.15.187:6789/0 conn(0x7f1e640b5190 legacy=0x7f1e640b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.313+0000 7f1e6957a700 1 -- 172.21.15.187:0/2774577286 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6414cc80 con 0x7f1e64065880 2020-11-20T12:19:15.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.313+0000 7f1e68d79700 1 -- 172.21.15.187:0/2774577286 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704186941 0 0) 0x7f1e6414c050 con 0x7f1e640b5190 2020-11-20T12:19:15.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.313+0000 7f1e69d7b700 1 --2- 172.21.15.187:0/2774577286 >> v2:172.21.15.5:3300/0 conn(0x7f1e640654a0 0x7f1e640b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.313+0000 7f1e6957a700 1 --2- 172.21.15.187:0/2774577286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e64065880 0x7f1e6414c790 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 rx=0x7f1e64065270 tx=0x7f1e54006df0).ready entity=mon.1 client_cookie=274e7329819c321d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.313+0000 7f1e68d79700 1 -- 172.21.15.187:0/2774577286 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54009840 con 0x7f1e64065880 2020-11-20T12:19:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e68d79700 1 -- 172.21.15.187:0/2774577286 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e540099a0 con 0x7f1e64065880 2020-11-20T12:19:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e68d79700 1 -- 172.21.15.187:0/2774577286 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54009d20 con 0x7f1e64065880 2020-11-20T12:19:15.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/2774577286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e64065880 msgr2=0x7f1e6414c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/2774577286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e64065880 0x7f1e6414c790 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 rx=0x7f1e64065270 tx=0x7f1e54006df0).stop 2020-11-20T12:19:15.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/2774577286 shutdown_connections 2020-11-20T12:19:15.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/2774577286 >> v2:172.21.15.5:3300/0 conn(0x7f1e640654a0 0x7f1e640b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/2774577286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e64065880 0x7f1e6414c790 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/2774577286 >> 172.21.15.187:0/2774577286 conn(0x7f1e640bab90 msgr2=0x7f1e640bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/2774577286 shutdown_connections 2020-11-20T12:19:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.314+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/2774577286 wait complete. 2020-11-20T12:19:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.316+0000 7f1e6ad7d700 1 Processor -- start 2020-11-20T12:19:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.316+0000 7f1e6ad7d700 1 -- start start 2020-11-20T12:19:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.317+0000 7f1e6ad7d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e640654a0 0x7f1e640a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.317+0000 7f1e6ad7d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e64065880 0x7f1e6409eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e6ad7d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e640b5190 0x7f1e640b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e6ad7d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640b2480 con 0x7f1e640b5190 2020-11-20T12:19:15.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e6ad7d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6409e0c0 con 0x7f1e640654a0 2020-11-20T12:19:15.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e6ad7d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6409e360 con 0x7f1e64065880 2020-11-20T12:19:15.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e69d7b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e640654a0 0x7f1e640a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e69d7b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e640654a0 0x7f1e640a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52292/0 (socket says 172.21.15.187:52292) 2020-11-20T12:19:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e69d7b700 1 -- 172.21.15.187:0/830295008 learned_addr learned my addr 172.21.15.187:0/830295008 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e6a57c700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e640b5190 0x7f1e640b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e6957a700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e64065880 0x7f1e6409eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e69d7b700 1 -- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e64065880 msgr2=0x7f1e6409eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.318+0000 7f1e69d7b700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e64065880 0x7f1e6409eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e69d7b700 1 -- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e640b5190 msgr2=0x7f1e640b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e69d7b700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e640b5190 0x7f1e640b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e69d7b700 1 -- 172.21.15.187:0/830295008 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6409e600 con 0x7f1e640654a0 2020-11-20T12:19:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e6a57c700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e640b5190 0x7f1e640b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e6957a700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e64065880 0x7f1e6409eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:15.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e69d7b700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e640654a0 0x7f1e640a0320 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7f1e6000cb50 tx=0x7f1e60008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e51ffb700 1 -- 172.21.15.187:0/830295008 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e600077d0 con 0x7f1e640654a0 2020-11-20T12:19:15.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e51ffb700 1 -- 172.21.15.187:0/830295008 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60007930 con 0x7f1e640654a0 2020-11-20T12:19:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e640a0810 con 0x7f1e640654a0 2020-11-20T12:19:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e51ffb700 1 -- 172.21.15.187:0/830295008 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60007cb0 con 0x7f1e640654a0 2020-11-20T12:19:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.319+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6414a5f0 con 0x7f1e640654a0 2020-11-20T12:19:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.320+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e64061ad0 con 0x7f1e640654a0 2020-11-20T12:19:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.321+0000 7f1e51ffb700 1 -- 172.21.15.187:0/830295008 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1e60011620 con 0x7f1e640654a0 2020-11-20T12:19:15.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.322+0000 7f1e51ffb700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e3c056d20 0x7f1e3c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:15.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.322+0000 7f1e51ffb700 1 -- 172.21.15.187:0/830295008 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1e60070250 con 0x7f1e640654a0 2020-11-20T12:19:15.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.322+0000 7f1e6957a700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e3c056d20 0x7f1e3c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:15.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.323+0000 7f1e6957a700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e3c056d20 0x7f1e3c0591e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f1e6414b450 tx=0x7f1e54015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:15.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.326+0000 7f1e51ffb700 1 -- 172.21.15.187:0/830295008 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1e600455d0 con 0x7f1e640654a0 2020-11-20T12:19:15.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.570+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e6409a110 con 0x7f1e640654a0 2020-11-20T12:19:15.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.571+0000 7f1e51ffb700 1 -- 172.21.15.187:0/830295008 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+2163 (secure 0 0 0) 0x7f1e60048bf0 con 0x7f1e640654a0 2020-11-20T12:19:15.574 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:15.574 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":129,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:19:03.863741+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:19:15.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.573+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e3c056d20 msgr2=0x7f1e3c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.573+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e3c056d20 0x7f1e3c0591e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f1e6414b450 tx=0x7f1e54015000).stop 2020-11-20T12:19:15.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.573+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e640654a0 msgr2=0x7f1e640a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:15.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.573+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e640654a0 0x7f1e640a0320 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7f1e6000cb50 tx=0x7f1e60008040).stop 2020-11-20T12:19:15.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.574+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 shutdown_connections 2020-11-20T12:19:15.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.574+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e3c056d20 0x7f1e3c0591e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.574+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e640b5190 0x7f1e640b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.574+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e640654a0 0x7f1e640a0320 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.574+0000 7f1e6ad7d700 1 --2- 172.21.15.187:0/830295008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e64065880 0x7f1e6409eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:15.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.574+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 >> 172.21.15.187:0/830295008 conn(0x7f1e640bab90 msgr2=0x7f1e640b9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:15.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.574+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 shutdown_connections 2020-11-20T12:19:15.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:15.574+0000 7f1e6ad7d700 1 -- 172.21.15.187:0/830295008 wait complete. 2020-11-20T12:19:15.579 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 129 2020-11-20T12:19:15.800 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:19:03.863741+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8932, 'mds_1': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'b', 'rank': 1, 'incarnation': 121, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/4181204645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 4181204645}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 4181204645}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:15.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5981:up:stopping 2020-11-20T12:19:16.801 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:17.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.053+0000 7f1c4e260700 1 Processor -- start 2020-11-20T12:19:17.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.053+0000 7f1c4e260700 1 -- start start 2020-11-20T12:19:17.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.054+0000 7f1c4e260700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1c48065c50 0x7f1c48066030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:17.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.055+0000 7f1c4e260700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c48145c60 con 0x7f1c480aff00 2020-11-20T12:19:17.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.055+0000 7f1c4e260700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c481459c0 con 0x7f1c48068400 2020-11-20T12:19:17.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.055+0000 7f1c4e260700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c48061e60 con 0x7f1c48065c50 2020-11-20T12:19:17.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.055+0000 7f1c4da5f700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1c48068400 0x7f1c48062300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48910/0 (socket says 172.21.15.187:48910) 2020-11-20T12:19:17.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.055+0000 7f1c4da5f700 1 -- 172.21.15.187:0/10867760 learned_addr learned my addr 172.21.15.187:0/10867760 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:17.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.055+0000 7f1c4d25e700 1 --2- 172.21.15.187:0/10867760 >> v2:172.21.15.5:3300/0 conn(0x7f1c48065c50 0x7f1c48066030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:17.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966918616 0 0) 0x7f1c481459c0 con 0x7f1c48068400 2020-11-20T12:19:17.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c30003240 con 0x7f1c48068400 2020-11-20T12:19:17.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1630570936 0 0) 0x7f1c30003240 con 0x7f1c48068400 2020-11-20T12:19:17.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 >> v2:172.21.15.5:3300/0 conn(0x7f1c48065c50 msgr2=0x7f1c48066030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:17.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 --2- 172.21.15.187:0/10867760 >> v2:172.21.15.5:3300/0 conn(0x7f1c48065c50 0x7f1c48066030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:17.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 >> v1:172.21.15.5:6789/0 conn(0x7f1c480aff00 legacy=0x7f1c480b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:17.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c481456d0 con 0x7f1c48068400 2020-11-20T12:19:17.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1c40001860 con 0x7f1c48068400 2020-11-20T12:19:17.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 >> v1:172.21.15.187:6789/0 conn(0x7f1c48068400 legacy=0x7f1c48062300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c30003800 0x7f1c30003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c30004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c30004e50 0x7f1c30005700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48061e60 con 0x7f1c30003800 2020-11-20T12:19:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c30005300 con 0x7f1c300044e0 2020-11-20T12:19:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.056+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c30005bf0 con 0x7f1c30004e50 2020-11-20T12:19:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.057+0000 7f1c4ca5d700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c30004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.057+0000 7f1c4da5f700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c30004e50 0x7f1c30005700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.057+0000 7f1c4ca5d700 1 -- 172.21.15.187:0/10867760 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c30004e50 msgr2=0x7f1c30005700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.057+0000 7f1c4ca5d700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c30004e50 0x7f1c30005700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.057+0000 7f1c4ca5d700 1 -- 172.21.15.187:0/10867760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c30003800 msgr2=0x7f1c30003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.057+0000 7f1c4ca5d700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c30003800 0x7f1c30003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.057+0000 7f1c4ca5d700 1 -- 172.21.15.187:0/10867760 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c30003240 con 0x7f1c300044e0 2020-11-20T12:19:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.057+0000 7f1c4da5f700 1 -- 172.21.15.187:0/10867760 reap_dead start 2020-11-20T12:19:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.058+0000 7f1c4ca5d700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c30004960 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7f1c38002830 tx=0x7f1c38007b90).ready entity=mon.1 client_cookie=7347e23b1b8d548c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.058+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38007020 con 0x7f1c300044e0 2020-11-20T12:19:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.058+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c380071c0 con 0x7f1c300044e0 2020-11-20T12:19:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.058+0000 7f1c37fff700 1 -- 172.21.15.187:0/10867760 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3800c830 con 0x7f1c300044e0 2020-11-20T12:19:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.058+0000 7f1c4e260700 1 -- 172.21.15.187:0/10867760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 msgr2=0x7f1c30004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.058+0000 7f1c4e260700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c30004960 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7f1c38002830 tx=0x7f1c38007b90).stop 2020-11-20T12:19:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.059+0000 7f1c4e260700 1 -- 172.21.15.187:0/10867760 shutdown_connections 2020-11-20T12:19:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.059+0000 7f1c4e260700 1 --2- 172.21.15.187:0/10867760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c30004960 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.059+0000 7f1c4e260700 1 -- 172.21.15.187:0/10867760 >> 172.21.15.187:0/10867760 conn(0x7f1c480bab90 msgr2=0x7f1c480bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:17.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.059+0000 7f1c4e260700 1 -- 172.21.15.187:0/10867760 shutdown_connections 2020-11-20T12:19:17.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.059+0000 7f1c4e260700 1 -- 172.21.15.187:0/10867760 wait complete. 2020-11-20T12:19:17.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.061+0000 7f1c4e260700 1 Processor -- start 2020-11-20T12:19:17.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.061+0000 7f1c4e260700 1 -- start start 2020-11-20T12:19:17.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4e260700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c480a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:17.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4e260700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c481587a0 0x7f1c4815b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:17.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4e260700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c4809e6d0 0x7f1c4809ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:17.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4e260700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48157dc0 con 0x7f1c4809e6d0 2020-11-20T12:19:17.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4e260700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48158060 con 0x7f1c300044e0 2020-11-20T12:19:17.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4e260700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48158300 con 0x7f1c481587a0 2020-11-20T12:19:17.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4d25e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c480a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:17.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4d25e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c480a0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52312/0 (socket says 172.21.15.187:52312) 2020-11-20T12:19:17.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.062+0000 7f1c4d25e700 1 -- 172.21.15.187:0/3212400167 learned_addr learned my addr 172.21.15.187:0/3212400167 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:17.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c4d25e700 1 -- 172.21.15.187:0/3212400167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c481587a0 msgr2=0x7f1c4815b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:17.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c4d25e700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c481587a0 0x7f1c4815b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:17.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c4d25e700 1 -- 172.21.15.187:0/3212400167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c4809e6d0 msgr2=0x7f1c4809ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:17.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c4d25e700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c4809e6d0 0x7f1c4809ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:17.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c4d25e700 1 -- 172.21.15.187:0/3212400167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48157a70 con 0x7f1c300044e0 2020-11-20T12:19:17.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c4d25e700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c480a0480 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 rx=0x7f1c440067d0 tx=0x7f1c44006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:17.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c357fa700 1 -- 172.21.15.187:0/3212400167 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c44016070 con 0x7f1c300044e0 2020-11-20T12:19:17.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c357fa700 1 -- 172.21.15.187:0/3212400167 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4400c820 con 0x7f1c300044e0 2020-11-20T12:19:17.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c480a0970 con 0x7f1c300044e0 2020-11-20T12:19:17.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c357fa700 1 -- 172.21.15.187:0/3212400167 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4400cbc0 con 0x7f1c300044e0 2020-11-20T12:19:17.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.063+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4815bec0 con 0x7f1c300044e0 2020-11-20T12:19:17.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.065+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48157c00 con 0x7f1c300044e0 2020-11-20T12:19:17.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.065+0000 7f1c357fa700 1 -- 172.21.15.187:0/3212400167 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1c4401b410 con 0x7f1c300044e0 2020-11-20T12:19:17.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.066+0000 7f1c357fa700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c20056d20 0x7f1c200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:17.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.066+0000 7f1c357fa700 1 -- 172.21.15.187:0/3212400167 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1c4406ffc0 con 0x7f1c300044e0 2020-11-20T12:19:17.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.066+0000 7f1c4ca5d700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c20056d20 0x7f1c200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:17.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.067+0000 7f1c4ca5d700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c20056d20 0x7f1c200591e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7f1c48062a70 tx=0x7f1c38015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:17.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.069+0000 7f1c357fa700 1 -- 172.21.15.187:0/3212400167 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1c44045360 con 0x7f1c300044e0 2020-11-20T12:19:17.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.310+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c481584c0 con 0x7f1c300044e0 2020-11-20T12:19:17.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.311+0000 7f1c357fa700 1 -- 172.21.15.187:0/3212400167 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+2163 (secure 0 0 0) 0x7f1c44048980 con 0x7f1c300044e0 2020-11-20T12:19:17.314 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:17.314 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":129,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:19:03.863741+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:19:17.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c20056d20 msgr2=0x7f1c200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:17.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c20056d20 0x7f1c200591e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7f1c48062a70 tx=0x7f1c38015000).stop 2020-11-20T12:19:17.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 msgr2=0x7f1c480a0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:17.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c480a0480 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 rx=0x7f1c440067d0 tx=0x7f1c44006df0).stop 2020-11-20T12:19:17.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 shutdown_connections 2020-11-20T12:19:17.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1c20056d20 0x7f1c200591e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:17.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c4809e6d0 0x7f1c4809ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:17.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c300044e0 0x7f1c480a0480 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:17.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 --2- 172.21.15.187:0/3212400167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c481587a0 0x7f1c4815b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:17.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 >> 172.21.15.187:0/3212400167 conn(0x7f1c480bab90 msgr2=0x7f1c480baf70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:17.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 shutdown_connections 2020-11-20T12:19:17.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:17.313+0000 7f1c4e260700 1 -- 172.21.15.187:0/3212400167 wait complete. 2020-11-20T12:19:17.319 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 129 2020-11-20T12:19:17.539 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:19:03.863741+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8932, 'mds_1': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'b', 'rank': 1, 'incarnation': 121, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/4181204645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 4181204645}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 4181204645}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:17.539 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5981:up:stopping 2020-11-20T12:19:18.541 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:18.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.794+0000 7fc52d90a700 1 Processor -- start 2020-11-20T12:19:18.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.794+0000 7fc52d90a700 1 -- start start 2020-11-20T12:19:18.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc52d90a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280654a0 0x7fc5280b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:18.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc52d90a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc528065880 0x7fc52814c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:18.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc52d90a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc5280b5190 0x7fc5280b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:18.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc52d90a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52814c050 con 0x7fc5280b5190 2020-11-20T12:19:18.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc52d90a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52814c2f0 con 0x7fc528065880 2020-11-20T12:19:18.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc52d90a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc528066400 con 0x7fc5280654a0 2020-11-20T12:19:18.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc51ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc528065880 0x7fc52814c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:18.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc51ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc528065880 0x7fc52814c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52322/0 (socket says 172.21.15.187:52322) 2020-11-20T12:19:18.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.796+0000 7fc51ffff700 1 -- 172.21.15.187:0/3457384427 learned_addr learned my addr 172.21.15.187:0/3457384427 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:18.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.797+0000 7fc52c908700 1 --2- 172.21.15.187:0/3457384427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280654a0 0x7fc5280b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:18.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.797+0000 7fc52d109700 1 --2- 172.21.15.187:0/3457384427 >> v2:172.21.15.5:3300/0 conn(0x7fc5280b5190 0x7fc5280b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:18.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.797+0000 7fc51ffff700 1 -- 172.21.15.187:0/3457384427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280654a0 msgr2=0x7fc5280b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:18.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.797+0000 7fc51ffff700 1 --2- 172.21.15.187:0/3457384427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280654a0 0x7fc5280b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:18.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.797+0000 7fc51ffff700 1 -- 172.21.15.187:0/3457384427 >> v2:172.21.15.5:3300/0 conn(0x7fc5280b5190 msgr2=0x7fc5280b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:18.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.797+0000 7fc51ffff700 1 --2- 172.21.15.187:0/3457384427 >> v2:172.21.15.5:3300/0 conn(0x7fc5280b5190 0x7fc5280b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:18.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.797+0000 7fc51ffff700 1 -- 172.21.15.187:0/3457384427 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52814cc80 con 0x7fc528065880 2020-11-20T12:19:18.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.797+0000 7fc51ffff700 1 --2- 172.21.15.187:0/3457384427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc528065880 0x7fc52814c790 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7fc5280648e0 tx=0x7fc518006eb0).ready entity=mon.1 client_cookie=ab903f7de8329198 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:18.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc51f7fe700 1 -- 172.21.15.187:0/3457384427 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc518009960 con 0x7fc528065880 2020-11-20T12:19:18.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc51f7fe700 1 -- 172.21.15.187:0/3457384427 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc518009ac0 con 0x7fc528065880 2020-11-20T12:19:18.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc51f7fe700 1 -- 172.21.15.187:0/3457384427 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc518009e40 con 0x7fc528065880 2020-11-20T12:19:18.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 -- 172.21.15.187:0/3457384427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc528065880 msgr2=0x7fc52814c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:18.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3457384427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc528065880 0x7fc52814c790 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7fc5280648e0 tx=0x7fc518006eb0).stop 2020-11-20T12:19:18.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 -- 172.21.15.187:0/3457384427 shutdown_connections 2020-11-20T12:19:18.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3457384427 >> v2:172.21.15.5:3300/0 conn(0x7fc5280b5190 0x7fc5280b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:18.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3457384427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280654a0 0x7fc5280b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:18.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3457384427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc528065880 0x7fc52814c790 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:18.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 -- 172.21.15.187:0/3457384427 >> 172.21.15.187:0/3457384427 conn(0x7fc5280bab90 msgr2=0x7fc5280bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:18.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 -- 172.21.15.187:0/3457384427 shutdown_connections 2020-11-20T12:19:18.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.798+0000 7fc52d90a700 1 -- 172.21.15.187:0/3457384427 wait complete. 2020-11-20T12:19:18.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.800+0000 7fc52d90a700 1 Processor -- start 2020-11-20T12:19:18.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.800+0000 7fc52d90a700 1 -- start start 2020-11-20T12:19:18.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d90a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5280654a0 0x7fc5280b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:18.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d90a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280b5190 0x7fc5281463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:18.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d90a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5280b2070 0x7fc52809b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:18.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d90a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b2de0 con 0x7fc5280654a0 2020-11-20T12:19:18.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d90a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b1340 con 0x7fc5280b2070 2020-11-20T12:19:18.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d90a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280a03c0 con 0x7fc5280b5190 2020-11-20T12:19:18.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d109700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5280b2070 0x7fc52809b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:18.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d109700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5280b2070 0x7fc52809b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52330/0 (socket says 172.21.15.187:52330) 2020-11-20T12:19:18.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.801+0000 7fc52d109700 1 -- 172.21.15.187:0/3444062974 learned_addr learned my addr 172.21.15.187:0/3444062974 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:18.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc51ffff700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280b5190 0x7fc5281463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:18.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52c908700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5280654a0 0x7fc5280b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:18.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52d109700 1 -- 172.21.15.187:0/3444062974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280b5190 msgr2=0x7fc5281463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:18.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52d109700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280b5190 0x7fc5281463b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:18.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52d109700 1 -- 172.21.15.187:0/3444062974 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5280654a0 msgr2=0x7fc5280b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:18.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52d109700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5280654a0 0x7fc5280b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:18.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52d109700 1 -- 172.21.15.187:0/3444062974 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280a0660 con 0x7fc5280b2070 2020-11-20T12:19:18.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52d109700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5280b2070 0x7fc52809b0c0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7fc52000cb80 tx=0x7fc520008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:18.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc51cff9700 1 -- 172.21.15.187:0/3444062974 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5200078a0 con 0x7fc5280b2070 2020-11-20T12:19:18.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc51cff9700 1 -- 172.21.15.187:0/3444062974 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc520007a00 con 0x7fc5280b2070 2020-11-20T12:19:18.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5280b15e0 con 0x7fc5280b2070 2020-11-20T12:19:18.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc51cff9700 1 -- 172.21.15.187:0/3444062974 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc520007da0 con 0x7fc5280b2070 2020-11-20T12:19:18.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.802+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52809fd80 con 0x7fc5280b2070 2020-11-20T12:19:18.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.803+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc528068420 con 0x7fc5280b2070 2020-11-20T12:19:18.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.804+0000 7fc51cff9700 1 -- 172.21.15.187:0/3444062974 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc520011620 con 0x7fc5280b2070 2020-11-20T12:19:18.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.805+0000 7fc51cff9700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc500056d50 0x7fc500059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:18.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.805+0000 7fc51cff9700 1 -- 172.21.15.187:0/3444062974 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc520070510 con 0x7fc5280b2070 2020-11-20T12:19:18.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.805+0000 7fc52c908700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc500056d50 0x7fc500059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:18.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.806+0000 7fc52c908700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc500056d50 0x7fc500059210 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fc524009650 tx=0x7fc524008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:18.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:18.809+0000 7fc51cff9700 1 -- 172.21.15.187:0/3444062974 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc520045890 con 0x7fc5280b2070 2020-11-20T12:19:19.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.071+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc52814c6b0 con 0x7fc5280b2070 2020-11-20T12:19:19.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.072+0000 7fc51cff9700 1 -- 172.21.15.187:0/3444062974 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+2163 (secure 0 0 0) 0x7fc520048eb0 con 0x7fc5280b2070 2020-11-20T12:19:19.075 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:19.075 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":129,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:19:03.863741+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8932,"mds_1":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"b","rank":1,"incarnation":121,"state":"up:stopping","state_seq":6,"addr":"172.21.15.5:6833/4181204645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":4181204645},{"type":"v1","addr":"172.21.15.5:6833","nonce":4181204645}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:19:19.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc500056d50 msgr2=0x7fc500059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:19.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc500056d50 0x7fc500059210 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fc524009650 tx=0x7fc524008040).stop 2020-11-20T12:19:19.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5280b2070 msgr2=0x7fc52809b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:19.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5280b2070 0x7fc52809b0c0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7fc52000cb80 tx=0x7fc520008040).stop 2020-11-20T12:19:19.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 shutdown_connections 2020-11-20T12:19:19.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc500056d50 0x7fc500059210 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:19.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5280654a0 0x7fc5280b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:19.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5280b2070 0x7fc52809b0c0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:19.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 --2- 172.21.15.187:0/3444062974 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc5280b5190 0x7fc5281463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:19.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 >> 172.21.15.187:0/3444062974 conn(0x7fc5280bab90 msgr2=0x7fc528061ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:19.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 shutdown_connections 2020-11-20T12:19:19.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:19.075+0000 7fc52d90a700 1 -- 172.21.15.187:0/3444062974 wait complete. 2020-11-20T12:19:19.081 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 129 2020-11-20T12:19:19.304 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:19:03.863741+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8932, 'mds_1': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'b', 'rank': 1, 'incarnation': 121, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.5:6833/4181204645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 4181204645}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 4181204645}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540138297136906239, 'flags': 0}, 'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:19.305 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5981:up:stopping 2020-11-20T12:19:20.029 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:19:20.309 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:20.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.565+0000 7f4f82511700 1 Processor -- start 2020-11-20T12:19:20.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.565+0000 7f4f82511700 1 -- start start 2020-11-20T12:19:20.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.566+0000 7f4f82511700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4f7c0654a0 0x7f4f7c0675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.566+0000 7f4f82511700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0670c0 0x7f4f7c0afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.566+0000 7f4f82511700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c099f70 con 0x7f4f7c065880 2020-11-20T12:19:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.566+0000 7f4f82511700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7c0b6a20 con 0x7f4f7c0670c0 2020-11-20T12:19:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.566+0000 7f4f82511700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7c148990 con 0x7f4f7c0654a0 2020-11-20T12:19:20.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.567+0000 7f4f8150f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4f7c0654a0 0x7f4f7c0675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:20.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.567+0000 7f4f80d0e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4f7c065880 0x7f4f7c148280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48944/0 (socket says 172.21.15.187:48944) 2020-11-20T12:19:20.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.567+0000 7f4f80d0e700 1 -- 172.21.15.187:0/2219961820 learned_addr learned my addr 172.21.15.187:0/2219961820 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:20.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.567+0000 7f4f81d10700 1 --2- 172.21.15.187:0/2219961820 >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0670c0 0x7f4f7c0afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:20.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f8150f700 1 -- 172.21.15.187:0/2219961820 >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0670c0 msgr2=0x7f4f7c0afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:20.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f8150f700 1 --2- 172.21.15.187:0/2219961820 >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0670c0 0x7f4f7c0afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:20.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f8150f700 1 -- 172.21.15.187:0/2219961820 >> v1:172.21.15.187:6789/0 conn(0x7f4f7c065880 legacy=0x7f4f7c148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:20.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f8150f700 1 -- 172.21.15.187:0/2219961820 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0affe0 con 0x7f4f7c0654a0 2020-11-20T12:19:20.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f6bfff700 1 -- 172.21.15.187:0/2219961820 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409491161 0 0) 0x7f4f7c099f70 con 0x7f4f7c065880 2020-11-20T12:19:20.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f8150f700 1 --2- 172.21.15.187:0/2219961820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c0675e0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7f4f7c0692c0 tx=0x7f4f78006df0).ready entity=mon.1 client_cookie=8b87abc3421eca08 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:20.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f6bfff700 1 -- 172.21.15.187:0/2219961820 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f780097b0 con 0x7f4f7c0654a0 2020-11-20T12:19:20.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f6bfff700 1 -- 172.21.15.187:0/2219961820 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f78009910 con 0x7f4f7c0654a0 2020-11-20T12:19:20.574 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.568+0000 7f4f6bfff700 1 -- 172.21.15.187:0/2219961820 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78009c90 con 0x7f4f7c0654a0 2020-11-20T12:19:20.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.569+0000 7f4f82511700 1 -- 172.21.15.187:0/2219961820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 msgr2=0x7f4f7c0675e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:20.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.569+0000 7f4f82511700 1 --2- 172.21.15.187:0/2219961820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c0675e0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7f4f7c0692c0 tx=0x7f4f78006df0).stop 2020-11-20T12:19:20.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.569+0000 7f4f82511700 1 -- 172.21.15.187:0/2219961820 shutdown_connections 2020-11-20T12:19:20.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.569+0000 7f4f82511700 1 --2- 172.21.15.187:0/2219961820 >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0670c0 0x7f4f7c0afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:20.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.569+0000 7f4f82511700 1 --2- 172.21.15.187:0/2219961820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c0675e0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:20.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.569+0000 7f4f82511700 1 -- 172.21.15.187:0/2219961820 >> 172.21.15.187:0/2219961820 conn(0x7f4f7c0bab90 msgr2=0x7f4f7c0bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:20.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.569+0000 7f4f82511700 1 -- 172.21.15.187:0/2219961820 shutdown_connections 2020-11-20T12:19:20.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.569+0000 7f4f82511700 1 -- 172.21.15.187:0/2219961820 wait complete. 2020-11-20T12:19:20.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.570+0000 7f4f82511700 1 Processor -- start 2020-11-20T12:19:20.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.571+0000 7f4f82511700 1 -- start start 2020-11-20T12:19:20.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.571+0000 7f4f82511700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:20.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.571+0000 7f4f82511700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f7c065880 0x7f4f7c09c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f82511700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f7c0670c0 0x7f4f7c151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f82511700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c15bf20 con 0x7f4f7c065880 2020-11-20T12:19:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f82511700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c15c1c0 con 0x7f4f7c0654a0 2020-11-20T12:19:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f82511700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c15c460 con 0x7f4f7c0670c0 2020-11-20T12:19:20.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f8150f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f8150f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c15b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52342/0 (socket says 172.21.15.187:52342) 2020-11-20T12:19:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f8150f700 1 -- 172.21.15.187:0/4235986892 learned_addr learned my addr 172.21.15.187:0/4235986892 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f8150f700 1 -- 172.21.15.187:0/4235986892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f7c0670c0 msgr2=0x7f4f7c151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f8150f700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f7c0670c0 0x7f4f7c151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f8150f700 1 -- 172.21.15.187:0/4235986892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f7c065880 msgr2=0x7f4f7c09c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f8150f700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f7c065880 0x7f4f7c09c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.572+0000 7f4f8150f700 1 -- 172.21.15.187:0/4235986892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c15bbe0 con 0x7f4f7c0654a0 2020-11-20T12:19:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.573+0000 7f4f8150f700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c15b6f0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f4f7800bee0 tx=0x7f4f78015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:20.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.573+0000 7f4f697fa700 1 -- 172.21.15.187:0/4235986892 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78011e50 con 0x7f4f7c0654a0 2020-11-20T12:19:20.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.573+0000 7f4f697fa700 1 -- 172.21.15.187:0/4235986892 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7801e460 con 0x7f4f7c0654a0 2020-11-20T12:19:20.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.573+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c09c970 con 0x7f4f7c0654a0 2020-11-20T12:19:20.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.573+0000 7f4f697fa700 1 -- 172.21.15.187:0/4235986892 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7801e820 con 0x7f4f7c0654a0 2020-11-20T12:19:20.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.573+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c13cba0 con 0x7f4f7c0654a0 2020-11-20T12:19:20.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.575+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c15c620 con 0x7f4f7c0654a0 2020-11-20T12:19:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.575+0000 7f4f697fa700 1 -- 172.21.15.187:0/4235986892 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4f7801a040 con 0x7f4f7c0654a0 2020-11-20T12:19:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.576+0000 7f4f697fa700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.576+0000 7f4f697fa700 1 -- 172.21.15.187:0/4235986892 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4f78071490 con 0x7f4f7c0654a0 2020-11-20T12:19:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.576+0000 7f4f80d0e700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:20.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.577+0000 7f4f80d0e700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f4f6c0011a0 tx=0x7f4f6c0076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:20.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.580+0000 7f4f697fa700 1 -- 172.21.15.187:0/4235986892 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4f780469c0 con 0x7f4f7c0654a0 2020-11-20T12:19:20.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.820+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f7c061ad0 con 0x7f4f7c0654a0 2020-11-20T12:19:20.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.821+0000 7f4f697fa700 1 -- 172.21.15.187:0/4235986892 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+1796 (secure 0 0 0) 0x7f4f78023070 con 0x7f4f7c0654a0 2020-11-20T12:19:20.824 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:20.824 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":130,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:19:19.889320+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8932},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:19:20.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 msgr2=0x7f4f540591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:20.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f4f6c0011a0 tx=0x7f4f6c0076a0).stop 2020-11-20T12:19:20.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 msgr2=0x7f4f7c15b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:20.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c15b6f0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f4f7800bee0 tx=0x7f4f78015000).stop 2020-11-20T12:19:20.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 shutdown_connections 2020-11-20T12:19:20.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:20.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f7c0670c0 0x7f4f7c151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:20.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0654a0 0x7f4f7c15b6f0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:20.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.823+0000 7f4f82511700 1 --2- 172.21.15.187:0/4235986892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f7c065880 0x7f4f7c09c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:20.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.824+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 >> 172.21.15.187:0/4235986892 conn(0x7f4f7c0bab90 msgr2=0x7f4f7c0666a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:20.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.824+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 shutdown_connections 2020-11-20T12:19:20.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:20.824+0000 7f4f82511700 1 -- 172.21.15.187:0/4235986892 wait complete. 2020-11-20T12:19:20.830 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 130 2020-11-20T12:19:21.053 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:18:10.557467+0000', 'modified': '2020-11-20T12:19:19.889320+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 131, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8932}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_8932': {'gid': 8932, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3451092437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3451092437}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3451092437}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:21.053 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:19:21.054 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:19:21.341 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":8932,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":70.754446530999999,"mdsmap_epoch":130,"osdmap_epoch":131,"osdmap_epoch_barrier":131,"uptime":76.409364014999994} 2020-11-20T12:19:21.342 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 8932, "mdsmap_epoch": 130, "osdmap_epoch": 131, "osdmap_epoch_barrier": 131, "rank_uptime": 70.754446531, "state": "up:active", "uptime": 76.409364015, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:19:21.343 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2020-11-20T12:19:21.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.584+0000 7f3f4aaa5700 1 Processor -- start 2020-11-20T12:19:21.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.584+0000 7f3f4aaa5700 1 -- start start 2020-11-20T12:19:21.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f4aaa5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3f440654a0 0x7f3f440b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:21.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f4aaa5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3f44065880 0x7f3f4414c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:21.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f4aaa5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4414c050 con 0x7f3f440b5190 2020-11-20T12:19:21.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f4aaa5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4414c2f0 con 0x7f3f440654a0 2020-11-20T12:19:21.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f4aaa5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f44066400 con 0x7f3f44065880 2020-11-20T12:19:21.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f4a2a4700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3f440b5190 0x7f3f440b6ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48958/0 (socket says 172.21.15.187:48958) 2020-11-20T12:19:21.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f4a2a4700 1 -- 172.21.15.187:0/4006864005 learned_addr learned my addr 172.21.15.187:0/4006864005 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:21.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f492a2700 1 --2- 172.21.15.187:0/4006864005 >> v2:172.21.15.187:3300/0 conn(0x7f3f44065880 0x7f3f4414c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:21.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.586+0000 7f3f49aa3700 1 --2- 172.21.15.187:0/4006864005 >> v2:172.21.15.5:3300/0 conn(0x7f3f440654a0 0x7f3f440b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:21.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f492a2700 1 -- 172.21.15.187:0/4006864005 >> v2:172.21.15.5:3300/0 conn(0x7f3f440654a0 msgr2=0x7f3f440b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f492a2700 1 --2- 172.21.15.187:0/4006864005 >> v2:172.21.15.5:3300/0 conn(0x7f3f440654a0 0x7f3f440b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f492a2700 1 -- 172.21.15.187:0/4006864005 >> v1:172.21.15.187:6789/0 conn(0x7f3f440b5190 legacy=0x7f3f440b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f492a2700 1 -- 172.21.15.187:0/4006864005 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4414cc80 con 0x7f3f44065880 2020-11-20T12:19:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f48aa1700 1 -- 172.21.15.187:0/4006864005 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688755480 0 0) 0x7f3f4414c050 con 0x7f3f440b5190 2020-11-20T12:19:21.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f492a2700 1 --2- 172.21.15.187:0/4006864005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f44065880 0x7f3f4414c790 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7f3f44065270 tx=0x7f3f34006df0).ready entity=mon.1 client_cookie=95d9f340587cb766 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:21.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f48aa1700 1 -- 172.21.15.187:0/4006864005 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34009840 con 0x7f3f44065880 2020-11-20T12:19:21.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f48aa1700 1 -- 172.21.15.187:0/4006864005 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f340099a0 con 0x7f3f44065880 2020-11-20T12:19:21.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.587+0000 7f3f48aa1700 1 -- 172.21.15.187:0/4006864005 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34009d20 con 0x7f3f44065880 2020-11-20T12:19:21.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.588+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/4006864005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f44065880 msgr2=0x7f3f4414c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:21.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.588+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/4006864005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f44065880 0x7f3f4414c790 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7f3f44065270 tx=0x7f3f34006df0).stop 2020-11-20T12:19:21.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.588+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/4006864005 shutdown_connections 2020-11-20T12:19:21.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.588+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/4006864005 >> v2:172.21.15.5:3300/0 conn(0x7f3f440654a0 0x7f3f440b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.588+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/4006864005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f44065880 0x7f3f4414c790 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.588+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/4006864005 >> 172.21.15.187:0/4006864005 conn(0x7f3f440bab90 msgr2=0x7f3f440bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:21.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.588+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/4006864005 shutdown_connections 2020-11-20T12:19:21.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.588+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/4006864005 wait complete. 2020-11-20T12:19:21.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.589+0000 7f3f4aaa5700 1 Processor -- start 2020-11-20T12:19:21.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.589+0000 7f3f4aaa5700 1 -- start start 2020-11-20T12:19:21.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.590+0000 7f3f4aaa5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f440654a0 0x7f3f4415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:21.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.590+0000 7f3f4aaa5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3f44065880 0x7f3f4409cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:21.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.590+0000 7f3f4aaa5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f440b5190 0x7f3f4409b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:21.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.590+0000 7f3f4aaa5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f44157d80 con 0x7f3f440654a0 2020-11-20T12:19:21.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.590+0000 7f3f4aaa5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f440b1d70 con 0x7f3f440b5190 2020-11-20T12:19:21.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.590+0000 7f3f4aaa5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f440b2010 con 0x7f3f44065880 2020-11-20T12:19:21.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f440b5190 0x7f3f4409b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:21.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f440b5190 0x7f3f4409b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52358/0 (socket says 172.21.15.187:52358) 2020-11-20T12:19:21.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 -- 172.21.15.187:0/1003486779 learned_addr learned my addr 172.21.15.187:0/1003486779 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:21.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f49aa3700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f440654a0 0x7f3f4415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:21.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f492a2700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3f44065880 0x7f3f4409cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:21.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 -- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3f44065880 msgr2=0x7f3f4409cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:21.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3f44065880 0x7f3f4409cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 -- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f440654a0 msgr2=0x7f3f4415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:21.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f440654a0 0x7f3f4415bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 -- 172.21.15.187:0/1003486779 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4409d400 con 0x7f3f440b5190 2020-11-20T12:19:21.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f49aa3700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f440654a0 0x7f3f4415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:21.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.591+0000 7f3f4a2a4700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f440b5190 0x7f3f4409b9c0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 rx=0x7f3f3c00cae0 tx=0x7f3f3c001320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:21.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.592+0000 7f3f31ffb700 1 -- 172.21.15.187:0/1003486779 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c005b80 con 0x7f3f440b5190 2020-11-20T12:19:21.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.592+0000 7f3f31ffb700 1 -- 172.21.15.187:0/1003486779 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c005ce0 con 0x7f3f440b5190 2020-11-20T12:19:21.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.592+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f440b3d50 con 0x7f3f440b5190 2020-11-20T12:19:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.592+0000 7f3f31ffb700 1 -- 172.21.15.187:0/1003486779 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c007360 con 0x7f3f440b5190 2020-11-20T12:19:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.592+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4409beb0 con 0x7f3f440b5190 2020-11-20T12:19:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.593+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f44157ec0 con 0x7f3f440b5190 2020-11-20T12:19:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.593+0000 7f3f31ffb700 1 -- 172.21.15.187:0/1003486779 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3f3c010030 con 0x7f3f440b5190 2020-11-20T12:19:21.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.594+0000 7f3f31ffb700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f1c056d20 0x7f3f1c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:21.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.594+0000 7f3f31ffb700 1 -- 172.21.15.187:0/1003486779 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3f3c06ff50 con 0x7f3f440b5190 2020-11-20T12:19:21.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.594+0000 7f3f49aa3700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f1c056d20 0x7f3f1c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:21.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.595+0000 7f3f49aa3700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f1c056d20 0x7f3f1c0591e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f3f40009710 tx=0x7f3f40008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:21.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.598+0000 7f3f31ffb700 1 -- 172.21.15.187:0/1003486779 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3f3c045350 con 0x7f3f440b5190 2020-11-20T12:19:21.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.845+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f3f44061ad0 con 0x7f3f440b5190 2020-11-20T12:19:21.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.846+0000 7f3f31ffb700 1 -- 172.21.15.187:0/1003486779 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1943 (secure 0 0 0) 0x7f3f3c048970 con 0x7f3f440b5190 2020-11-20T12:19:21.849 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:21.849 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:19:21.849 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:19:21.850 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:19:21.850 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 772882190336, 2020-11-20T12:19:21.850 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 211922944, 2020-11-20T12:19:21.850 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 211922944, 2020-11-20T12:19:21.850 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 0.00027412310009822249, 2020-11-20T12:19:21.851 INFO:teuthology.orchestra.run.smithi187.stdout: "num_osds": 8, 2020-11-20T12:19:21.851 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_osds": 8, 2020-11-20T12:19:21.851 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_omap_osds": 8 2020-11-20T12:19:21.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:21.852 INFO:teuthology.orchestra.run.smithi187.stdout: "stats_by_class": { 2020-11-20T12:19:21.852 INFO:teuthology.orchestra.run.smithi187.stdout: "ssd": { 2020-11-20T12:19:21.852 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:19:21.852 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 772882190336, 2020-11-20T12:19:21.852 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 211922944, 2020-11-20T12:19:21.853 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 211922944, 2020-11-20T12:19:21.853 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 0.00027412310009822249 2020-11-20T12:19:21.853 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:21.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:21.853 INFO:teuthology.orchestra.run.smithi187.stdout: "pools": [ 2020-11-20T12:19:21.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:21.854 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "device_health_metrics", 2020-11-20T12:19:21.854 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 1, 2020-11-20T12:19:21.854 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:19:21.855 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:19:21.855 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 2, 2020-11-20T12:19:21.855 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:19:21.855 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:19:21.855 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:19:21.856 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367090302976 2020-11-20T12:19:21.856 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:21.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:21.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:21.856 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "rbd", 2020-11-20T12:19:21.857 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 2, 2020-11-20T12:19:21.857 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:19:21.857 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:19:21.857 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:19:21.857 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:19:21.858 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:19:21.858 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:19:21.858 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367090302976 2020-11-20T12:19:21.858 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:21.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:21.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:21.859 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_metadata", 2020-11-20T12:19:21.859 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 25, 2020-11-20T12:19:21.859 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:19:21.859 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 69876, 2020-11-20T12:19:21.860 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 42, 2020-11-20T12:19:21.860 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 240, 2020-11-20T12:19:21.860 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 245760, 2020-11-20T12:19:21.860 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 3.3474040606051858e-07, 2020-11-20T12:19:21.861 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367090302976 2020-11-20T12:19:21.861 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:21.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:21.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:21.861 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_data", 2020-11-20T12:19:21.862 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 26, 2020-11-20T12:19:21.862 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:19:21.862 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:19:21.862 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:19:21.862 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:19:21.863 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:19:21.863 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:19:21.863 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367090302976 2020-11-20T12:19:21.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:21.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:21.864 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:19:21.864 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:19:21.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.848+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f1c056d20 msgr2=0x7f3f1c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:21.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.848+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f1c056d20 0x7f3f1c0591e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f3f40009710 tx=0x7f3f40008040).stop 2020-11-20T12:19:21.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.848+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f440b5190 msgr2=0x7f3f4409b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:21.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.848+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f440b5190 0x7f3f4409b9c0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 rx=0x7f3f3c00cae0 tx=0x7f3f3c001320).stop 2020-11-20T12:19:21.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.849+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 shutdown_connections 2020-11-20T12:19:21.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.849+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3f1c056d20 0x7f3f1c0591e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.849+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f440654a0 0x7f3f4415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.849+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f440b5190 0x7f3f4409b9c0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.849+0000 7f3f4aaa5700 1 --2- 172.21.15.187:0/1003486779 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3f44065880 0x7f3f4409cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:21.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.849+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 >> 172.21.15.187:0/1003486779 conn(0x7f3f440bab90 msgr2=0x7f3f4409fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:21.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.849+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 shutdown_connections 2020-11-20T12:19:21.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:21.849+0000 7f3f4aaa5700 1 -- 172.21.15.187:0/1003486779 wait complete. 2020-11-20T12:19:21.873 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2020-11-20T12:19:21.873 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:19:21.874 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:19:21.996 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:19:22.010 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:19:22.127 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:19:22.146 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:19:22.148 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:19:22.163 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:19:22.357 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:19:22.358 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:19:22.360 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:19:22.362 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:19:22.363 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.364 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:19:22.366 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.367 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:19:22.368 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.368 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:19:22.368 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:19:22.368 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.368 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:19:22.369 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:19:22.369 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:19:22.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:19:22.370 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.370 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:19:22.370 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:19:22.370 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.370 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:19:22.371 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.371 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:19:22.371 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:19:22.371 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:19:22.371 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:19:22.372 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:19:22.372 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:19:22.372 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:19:22.372 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:19:22.372 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:19:22.373 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:19:22.373 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.373 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:19:22.373 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.374 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:19:22.374 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.374 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:19:22.374 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.374 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:19:22.375 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:19:22.375 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:19:22.375 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:19:22.376 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:19:22.376 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:19:22.376 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.377 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:19:22.377 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:19:22.377 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.377 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:19:22.387 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:19:22.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.387 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:19:22.387 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:19:22.388 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.388 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:19:22.388 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:19:22.388 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:19:22.389 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.389 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:19:22.389 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:19:22.389 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.390 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:19:22.390 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.390 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:19:22.390 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.390 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:19:22.391 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.391 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:19:22.391 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:19:22.391 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.391 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:19:22.392 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:19:22.392 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:19:22.392 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:19:22.393 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:19:22.393 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:19:22.393 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:19:22.393 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:19:22.393 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:19:22.394 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:19:22.394 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:19:22.394 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:19:22.394 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:19:22.394 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:19:22.395 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:19:22.395 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:19:22.395 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:19:22.395 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:19:22.395 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:19:22.396 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:19:22.396 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.396 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:19:22.396 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:19:22.396 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:19:22.397 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:19:22.397 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:19:22.397 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:19:22.397 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:19:22.398 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:19:22.398 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.398 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:19:22.398 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:19:22.398 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:19:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:19:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:19:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:19:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:19:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:19:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:19:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:19:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:19:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:19:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:19:22.401 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:19:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:19:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:19:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:19:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:19:22.402 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:19:22.402 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:19:22.402 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:19:22.402 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:19:22.402 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:19:22.403 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:19:22.403 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:19:22.403 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:19:22.403 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:19:22.403 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:19:22.404 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:19:22.404 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:19:22.404 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:19:22.404 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:19:22.404 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:19:22.405 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:19:22.405 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:19:22.405 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:19:22.405 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:19:22.406 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:19:22.406 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:19:22.406 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:19:22.406 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:19:22.406 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:19:22.407 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:19:22.407 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:19:22.407 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:19:22.408 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:19:22.408 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:19:22.408 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:19:22.408 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:19:22.409 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.409 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:19:22.409 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.409 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:19:22.409 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.410 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:19:22.410 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:19:22.410 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.410 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:19:22.410 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:19:22.411 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.411 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:19:22.411 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:19:22.411 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.411 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:19:22.412 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:19:22.412 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.412 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:19:22.412 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.412 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:19:22.413 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:19:22.413 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:19:22.413 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:19:22.413 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:19:22.413 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:19:22.414 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:19:22.414 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:19:22.414 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:19:22.414 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:19:22.414 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.415 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:19:22.415 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.415 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:19:22.415 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.415 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:19:22.416 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.416 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:19:22.416 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:19:22.416 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:19:22.416 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:19:22.417 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:19:22.417 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:19:22.417 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.417 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:19:22.418 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:19:22.418 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.418 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:19:22.418 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:19:22.418 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.419 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:19:22.419 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:19:22.419 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.419 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:19:22.419 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:19:22.420 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:19:22.420 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.420 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:19:22.420 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:19:22.420 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.421 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:19:22.421 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.421 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:19:22.421 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.421 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:19:22.422 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.422 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:19:22.422 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:19:22.422 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.422 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:19:22.423 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:19:22.423 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:19:22.423 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:19:22.423 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:19:22.423 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:19:22.424 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:19:22.424 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:19:22.424 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:19:22.424 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:19:22.424 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:19:22.425 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:19:22.425 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:19:22.425 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:19:22.425 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:19:22.426 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:19:22.426 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:19:22.426 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:19:22.426 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:19:22.426 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:19:22.427 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.427 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:19:22.427 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:19:22.427 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:19:22.427 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:19:22.428 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:19:22.428 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:19:22.428 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:19:22.428 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:19:22.428 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.429 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:19:22.429 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:19:22.429 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:19:22.429 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:19:22.429 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:19:22.430 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:19:22.430 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:19:22.430 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:19:22.430 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:19:22.430 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:19:22.431 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:19:22.431 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:19:22.431 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:22.431 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:19:22.431 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:19:22.432 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:19:22.432 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:19:22.432 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:19:22.432 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:19:22.432 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:19:22.433 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:19:22.433 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:19:22.433 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:19:22.433 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:19:22.433 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:19:22.434 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:19:22.434 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:19:22.434 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:19:22.434 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:19:22.434 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:19:22.435 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:19:22.435 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:19:22.435 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:19:22.435 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:19:22.435 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:19:22.436 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:19:22.436 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:19:22.436 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:19:22.436 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:19:22.437 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:19:22.437 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:19:22.437 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:19:22.437 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:19:22.437 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:19:22.438 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:19:22.439 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:19:22.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.690+0000 7fbf1bfff700 1 Processor -- start 2020-11-20T12:19:22.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.691+0000 7fbf1bfff700 1 -- start start 2020-11-20T12:19:22.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.692+0000 7fbf1bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbf1c065190 0x7fbf1c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:22.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.692+0000 7fbf1bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbf1c09ca90 0x7fbf1c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:22.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.692+0000 7fbf1bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1c0b13e0 con 0x7fbf1c065f90 2020-11-20T12:19:22.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.692+0000 7fbf1bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1c14ab30 con 0x7fbf1c065190 2020-11-20T12:19:22.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.692+0000 7fbf1bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1c062c10 con 0x7fbf1c09ca90 2020-11-20T12:19:22.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.692+0000 7fbf1b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbf1c09ca90 0x7fbf1c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:22.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.692+0000 7fbf1b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbf1c09ca90 0x7fbf1c09cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52364/0 (socket says 172.21.15.187:52364) 2020-11-20T12:19:22.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.692+0000 7fbf1b7fe700 1 -- 172.21.15.187:0/3667185520 learned_addr learned my addr 172.21.15.187:0/3667185520 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:22.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.693+0000 7fbf1affd700 1 --2- 172.21.15.187:0/3667185520 >> v2:172.21.15.5:3300/0 conn(0x7fbf1c065190 0x7fbf1c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:22.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.693+0000 7fbf1b7fe700 1 -- 172.21.15.187:0/3667185520 >> v2:172.21.15.5:3300/0 conn(0x7fbf1c065190 msgr2=0x7fbf1c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:22.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.693+0000 7fbf1b7fe700 1 --2- 172.21.15.187:0/3667185520 >> v2:172.21.15.5:3300/0 conn(0x7fbf1c065190 0x7fbf1c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.693+0000 7fbf1b7fe700 1 -- 172.21.15.187:0/3667185520 >> v1:172.21.15.5:6789/0 conn(0x7fbf1c065f90 legacy=0x7fbf1c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:22.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.693+0000 7fbf1b7fe700 1 -- 172.21.15.187:0/3667185520 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c0b1640 con 0x7fbf1c09ca90 2020-11-20T12:19:22.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.693+0000 7fbf1affd700 1 --2- 172.21.15.187:0/3667185520 >> v2:172.21.15.5:3300/0 conn(0x7fbf1c065190 0x7fbf1c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:22.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.693+0000 7fbf1b7fe700 1 --2- 172.21.15.187:0/3667185520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c09ca90 0x7fbf1c09cf20 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7fbf1c099f90 tx=0x7fbf14009290).ready entity=mon.1 client_cookie=25d817c584a199b5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:22.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf19ffb700 1 -- 172.21.15.187:0/3667185520 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1400bc10 con 0x7fbf1c09ca90 2020-11-20T12:19:22.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf19ffb700 1 -- 172.21.15.187:0/3667185520 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1400bd70 con 0x7fbf1c09ca90 2020-11-20T12:19:22.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf19ffb700 1 -- 172.21.15.187:0/3667185520 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14011610 con 0x7fbf1c09ca90 2020-11-20T12:19:22.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf1bfff700 1 -- 172.21.15.187:0/3667185520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c09ca90 msgr2=0x7fbf1c09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:22.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/3667185520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c09ca90 0x7fbf1c09cf20 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7fbf1c099f90 tx=0x7fbf14009290).stop 2020-11-20T12:19:22.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf1bfff700 1 -- 172.21.15.187:0/3667185520 shutdown_connections 2020-11-20T12:19:22.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/3667185520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c09ca90 0x7fbf1c09cf20 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/3667185520 >> v2:172.21.15.5:3300/0 conn(0x7fbf1c065190 0x7fbf1c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf1bfff700 1 -- 172.21.15.187:0/3667185520 >> 172.21.15.187:0/3667185520 conn(0x7fbf1c0bab30 msgr2=0x7fbf1c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:22.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf1bfff700 1 -- 172.21.15.187:0/3667185520 shutdown_connections 2020-11-20T12:19:22.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.694+0000 7fbf1bfff700 1 -- 172.21.15.187:0/3667185520 wait complete. 2020-11-20T12:19:22.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.696+0000 7fbf1bfff700 1 Processor -- start 2020-11-20T12:19:22.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.696+0000 7fbf1bfff700 1 -- start start 2020-11-20T12:19:22.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.697+0000 7fbf1bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c065190 0x7fbf1c149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:22.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.697+0000 7fbf1bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf1c065f90 0x7fbf1c09ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:22.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.697+0000 7fbf1bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf1c09ca90 0x7fbf1c09e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:22.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.697+0000 7fbf1bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1475c0 con 0x7fbf1c09ca90 2020-11-20T12:19:22.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.697+0000 7fbf1bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c147820 con 0x7fbf1c065190 2020-11-20T12:19:22.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.697+0000 7fbf1bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c09f540 con 0x7fbf1c065f90 2020-11-20T12:19:22.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.697+0000 7fbf1affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c065190 0x7fbf1c149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:22.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.697+0000 7fbf1affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c065190 0x7fbf1c149770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52372/0 (socket says 172.21.15.187:52372) 2020-11-20T12:19:22.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1affd700 1 -- 172.21.15.187:0/2312629271 learned_addr learned my addr 172.21.15.187:0/2312629271 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:22.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1a7fc700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf1c065f90 0x7fbf1c09ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:22.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1b7fe700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf1c09ca90 0x7fbf1c09e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:22.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1affd700 1 -- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf1c065f90 msgr2=0x7fbf1c09ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:22.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1affd700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf1c065f90 0x7fbf1c09ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1affd700 1 -- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf1c09ca90 msgr2=0x7fbf1c09e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:22.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1affd700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf1c09ca90 0x7fbf1c09e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1affd700 1 -- 172.21.15.187:0/2312629271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c0a02a0 con 0x7fbf1c065190 2020-11-20T12:19:22.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1b7fe700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf1c09ca90 0x7fbf1c09e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:22.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.698+0000 7fbf1affd700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c065190 0x7fbf1c149770 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 rx=0x7fbf0c00cb50 tx=0x7fbf0c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:22.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.699+0000 7fbf02ffd700 1 -- 172.21.15.187:0/2312629271 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c0077d0 con 0x7fbf1c065190 2020-11-20T12:19:22.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.699+0000 7fbf02ffd700 1 -- 172.21.15.187:0/2312629271 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0c007930 con 0x7fbf1c065190 2020-11-20T12:19:22.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.699+0000 7fbf02ffd700 1 -- 172.21.15.187:0/2312629271 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c007cb0 con 0x7fbf1c065190 2020-11-20T12:19:22.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.699+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1c09f120 con 0x7fbf1c065190 2020-11-20T12:19:22.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.699+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1c09e810 con 0x7fbf1c065190 2020-11-20T12:19:22.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.700+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1c061a20 con 0x7fbf1c065190 2020-11-20T12:19:22.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.701+0000 7fbf02ffd700 1 -- 172.21.15.187:0/2312629271 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbf0c011620 con 0x7fbf1c065190 2020-11-20T12:19:22.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.701+0000 7fbf02ffd700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbef8056d20 0x7fbef80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:22.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.701+0000 7fbf02ffd700 1 -- 172.21.15.187:0/2312629271 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbf0c0703a0 con 0x7fbf1c065190 2020-11-20T12:19:22.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.701+0000 7fbf1a7fc700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbef8056d20 0x7fbef80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:22.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.702+0000 7fbf1a7fc700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbef8056d20 0x7fbef80591e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7fbf1c0b3df0 tx=0x7fbf10009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:22.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.705+0000 7fbf02ffd700 1 -- 172.21.15.187:0/2312629271 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbf0c045690 con 0x7fbf1c065190 2020-11-20T12:19:22.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.894+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbf1c0afc60 con 0x7fbf1c065190 2020-11-20T12:19:22.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.904+0000 7fbf02ffd700 1 -- 172.21.15.187:0/2312629271 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbf0c048cb0 con 0x7fbf1c065190 2020-11-20T12:19:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbef8056d20 msgr2=0x7fbef80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbef8056d20 0x7fbef80591e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7fbf1c0b3df0 tx=0x7fbf10009040).stop 2020-11-20T12:19:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c065190 msgr2=0x7fbf1c149770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c065190 0x7fbf1c149770 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 rx=0x7fbf0c00cb50 tx=0x7fbf0c008040).stop 2020-11-20T12:19:22.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 shutdown_connections 2020-11-20T12:19:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbef8056d20 0x7fbef80591e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf1c09ca90 0x7fbf1c09e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c065190 0x7fbf1c149770 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 --2- 172.21.15.187:0/2312629271 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbf1c065f90 0x7fbf1c09ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:22.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 >> 172.21.15.187:0/2312629271 conn(0x7fbf1c0bab30 msgr2=0x7fbf1c149ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:22.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 shutdown_connections 2020-11-20T12:19:22.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:22.907+0000 7fbf1bfff700 1 -- 172.21.15.187:0/2312629271 wait complete. 2020-11-20T12:19:22.935 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown' 2020-11-20T12:19:23.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.177+0000 7f93123c7700 1 Processor -- start 2020-11-20T12:19:23.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.177+0000 7f93123c7700 1 -- start start 2020-11-20T12:19:23.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.178+0000 7f93123c7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c0bf800 0x7f930c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:23.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.178+0000 7f93123c7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c14d110 con 0x7f930c09f560 2020-11-20T12:19:23.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.178+0000 7f93123c7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c14cbd0 con 0x7f930c0bfbe0 2020-11-20T12:19:23.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.179+0000 7f93123c7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c14ce70 con 0x7f930c0bf800 2020-11-20T12:19:23.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.179+0000 7f9310bc4700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f930c0bfbe0 0x7f930c1480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:48986/0 (socket says 172.21.15.187:48986) 2020-11-20T12:19:23.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.179+0000 7f9310bc4700 1 -- 172.21.15.187:0/2528267362 learned_addr learned my addr 172.21.15.187:0/2528267362 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:23.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.179+0000 7f93113c5700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c0bf800 0x7f930c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:23.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.179+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034792741 0 0) 0x7f930c14cbd0 con 0x7f930c0bfbe0 2020-11-20T12:19:23.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.179+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0003240 con 0x7f930c0bfbe0 2020-11-20T12:19:23.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3342688531 0 0) 0x7f92f0003240 con 0x7f930c0bfbe0 2020-11-20T12:19:23.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c0bf800 msgr2=0x7f930c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:23.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c0bf800 0x7f930c14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:23.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 >> v1:172.21.15.5:6789/0 conn(0x7f930c09f560 legacy=0x7f930c0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:23.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0a4cc0 con 0x7f930c0bfbe0 2020-11-20T12:19:23.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f92f80014e0 con 0x7f930c0bfbe0 2020-11-20T12:19:23.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 >> v1:172.21.15.187:6789/0 conn(0x7f930c0bfbe0 legacy=0x7f930c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:23.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c0bf800 0x7f92f0003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:23.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92f0003c20 0x7f92f00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:23.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f00049d0 0x7f92f0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:23.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c14ce70 con 0x7f92f0003c20 2020-11-20T12:19:23.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00041b0 con 0x7f92f00049d0 2020-11-20T12:19:23.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f000b730 con 0x7f930c0bf800 2020-11-20T12:19:23.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.180+0000 7f9311bc6700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f00049d0 0x7f92f0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:23.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f93113c5700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c0bf800 0x7f92f0003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:23.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f9310bc4700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92f0003c20 0x7f92f00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:23.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f9311bc6700 1 -- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c0bf800 msgr2=0x7f92f0003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:23.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f9311bc6700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c0bf800 0x7f92f0003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:23.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f9311bc6700 1 -- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92f0003c20 msgr2=0x7f92f00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:23.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f9311bc6700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92f0003c20 0x7f92f00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:23.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f9311bc6700 1 -- 172.21.15.187:0/2528267362 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f0003240 con 0x7f92f00049d0 2020-11-20T12:19:23.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f9311bc6700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f00049d0 0x7f92f0004e50 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7f9304002860 tx=0x7f9304007b90).ready entity=mon.1 client_cookie=9b4dbb9f2b7d4574 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:23.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304007020 con 0x7f92f00049d0 2020-11-20T12:19:23.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.181+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93040071c0 con 0x7f92f00049d0 2020-11-20T12:19:23.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.182+0000 7f930bfff700 1 -- 172.21.15.187:0/2528267362 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930400c7d0 con 0x7f92f00049d0 2020-11-20T12:19:23.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.182+0000 7f93123c7700 1 -- 172.21.15.187:0/2528267362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f00049d0 msgr2=0x7f92f0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:23.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.182+0000 7f93123c7700 1 --2- 172.21.15.187:0/2528267362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f00049d0 0x7f92f0004e50 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7f9304002860 tx=0x7f9304007b90).stop 2020-11-20T12:19:23.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.182+0000 7f9311bc6700 1 -- 172.21.15.187:0/2528267362 reap_dead start 2020-11-20T12:19:23.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.182+0000 7f93123c7700 1 -- 172.21.15.187:0/2528267362 shutdown_connections 2020-11-20T12:19:23.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.182+0000 7f93123c7700 1 -- 172.21.15.187:0/2528267362 >> 172.21.15.187:0/2528267362 conn(0x7f930c068370 msgr2=0x7f930c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:23.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.182+0000 7f93123c7700 1 -- 172.21.15.187:0/2528267362 shutdown_connections 2020-11-20T12:19:23.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.182+0000 7f93123c7700 1 -- 172.21.15.187:0/2528267362 wait complete. 2020-11-20T12:19:23.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.184+0000 7f93123c7700 1 Processor -- start 2020-11-20T12:19:23.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.184+0000 7f93123c7700 1 -- start start 2020-11-20T12:19:23.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f93123c7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f930c142bc0 0x7f930c145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:23.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f93123c7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f930c142fa0 0x7f930c0b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:23.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f93123c7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c1461a0 0x7f930c0b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:23.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f93123c7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0b6cc0 con 0x7f930c142bc0 2020-11-20T12:19:23.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f93123c7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c15cab0 con 0x7f930c142fa0 2020-11-20T12:19:23.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f93123c7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0b5ba0 con 0x7f930c1461a0 2020-11-20T12:19:23.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f9310bc4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f930c142fa0 0x7f930c0b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:23.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f9310bc4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f930c142fa0 0x7f930c0b62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52390/0 (socket says 172.21.15.187:52390) 2020-11-20T12:19:23.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.185+0000 7f9310bc4700 1 -- 172.21.15.187:0/1319315452 learned_addr learned my addr 172.21.15.187:0/1319315452 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:23.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f93113c5700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f930c142bc0 0x7f930c145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:23.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f9310bc4700 1 -- 172.21.15.187:0/1319315452 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c1461a0 msgr2=0x7f930c0b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:23.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f9310bc4700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c1461a0 0x7f930c0b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:23.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f9310bc4700 1 -- 172.21.15.187:0/1319315452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f930c142bc0 msgr2=0x7f930c145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:23.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f9310bc4700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f930c142bc0 0x7f930c145cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:23.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f9310bc4700 1 -- 172.21.15.187:0/1319315452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0b5e40 con 0x7f930c142fa0 2020-11-20T12:19:23.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f9310bc4700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f930c142fa0 0x7f930c0b62e0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f92f800eab0 tx=0x7f92f8001560).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:23.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f93097fa700 1 -- 172.21.15.187:0/1319315452 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f800a7f0 con 0x7f930c142fa0 2020-11-20T12:19:23.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f93097fa700 1 -- 172.21.15.187:0/1319315452 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f8010040 con 0x7f930c142fa0 2020-11-20T12:19:23.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c0bac50 con 0x7f930c142fa0 2020-11-20T12:19:23.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f93097fa700 1 -- 172.21.15.187:0/1319315452 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f800aba0 con 0x7f930c142fa0 2020-11-20T12:19:23.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.186+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c15c860 con 0x7f930c142fa0 2020-11-20T12:19:23.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.187+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c080050 con 0x7f930c142fa0 2020-11-20T12:19:23.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.188+0000 7f93097fa700 1 -- 172.21.15.187:0/1319315452 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f92f801b020 con 0x7f930c142fa0 2020-11-20T12:19:23.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.189+0000 7f93097fa700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e4056d20 0x7f92e40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:23.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.189+0000 7f93097fa700 1 -- 172.21.15.187:0/1319315452 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f92f806f6b0 con 0x7f930c142fa0 2020-11-20T12:19:23.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.189+0000 7f93113c5700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e4056d20 0x7f92e40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:23.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.190+0000 7f93113c5700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e4056d20 0x7f92e40591e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f930000ca50 tx=0x7f9300008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:23.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.192+0000 7f93097fa700 1 -- 172.21.15.187:0/1319315452 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f92f8044ab0 con 0x7f930c142fa0 2020-11-20T12:19:23.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.449+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown"]} v 0) v1 -- 0x7f930c0b5fd0 con 0x7f930c142fa0 2020-11-20T12:19:23.582 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:23.593 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:23.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.935+0000 7f93097fa700 1 -- 172.21.15.187:0/1319315452 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown"]}]=0 v719) v1 ==== 137+0+0 (secure 0 0 0) 0x7f92f80480d0 con 0x7f930c142fa0 2020-11-20T12:19:23.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e4056d20 msgr2=0x7f92e40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:23.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e4056d20 0x7f92e40591e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f930000ca50 tx=0x7f9300008040).stop 2020-11-20T12:19:23.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f930c142fa0 msgr2=0x7f930c0b62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:23.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f930c142fa0 0x7f930c0b62e0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f92f800eab0 tx=0x7f92f8001560).stop 2020-11-20T12:19:23.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 shutdown_connections 2020-11-20T12:19:23.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92e4056d20 0x7f92e40591e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:23.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f930c142bc0 0x7f930c145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:23.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f930c1461a0 0x7f930c0b67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:23.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 --2- 172.21.15.187:0/1319315452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f930c142fa0 0x7f930c0b62e0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:23.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 >> 172.21.15.187:0/1319315452 conn(0x7f930c068370 msgr2=0x7f930c0be2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:23.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 shutdown_connections 2020-11-20T12:19:23.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:23.939+0000 7f93123c7700 1 -- 172.21.15.187:0/1319315452 wait complete. 2020-11-20T12:19:23.972 INFO:tasks.cephfs_test_runner:test_purge_on_shutdown (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:19:23.974 INFO:tasks.cephfs_test_runner:Starting test: test_purge_queue_upgrade (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:19:23.975 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade' 2020-11-20T12:19:24.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.225+0000 7f66c8f8e700 1 Processor -- start 2020-11-20T12:19:24.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.225+0000 7f66c8f8e700 1 -- start start 2020-11-20T12:19:24.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.226+0000 7f66c8f8e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c41482d0 0x7f66c40b6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:24.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.226+0000 7f66c8f8e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c4144b80 con 0x7f66c4068230 2020-11-20T12:19:24.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.227+0000 7f66c8f8e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c41489e0 con 0x7f66c40b57b0 2020-11-20T12:19:24.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.227+0000 7f66c8f8e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c4148c80 con 0x7f66c41482d0 2020-11-20T12:19:24.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.227+0000 7f66c37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f66c40b57b0 0x7f66c4062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49004/0 (socket says 172.21.15.187:49004) 2020-11-20T12:19:24.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.227+0000 7f66c37fe700 1 -- 172.21.15.187:0/1256628784 learned_addr learned my addr 172.21.15.187:0/1256628784 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:24.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.227+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008980916 0 0) 0x7f66c41489e0 con 0x7f66c40b57b0 2020-11-20T12:19:24.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.227+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b0003240 con 0x7f66c40b57b0 2020-11-20T12:19:24.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c3fff700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c41482d0 0x7f66c40b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:24.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1075951455 0 0) 0x7f66b0003240 con 0x7f66c40b57b0 2020-11-20T12:19:24.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c41482d0 msgr2=0x7f66c40b6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:24.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c27fc700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c41482d0 0x7f66c40b6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 >> v1:172.21.15.5:6789/0 conn(0x7f66c4068230 legacy=0x7f66c40b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:24.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c4148f20 con 0x7f66c40b57b0 2020-11-20T12:19:24.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f66b40014e0 con 0x7f66c40b57b0 2020-11-20T12:19:24.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 >> v1:172.21.15.187:6789/0 conn(0x7f66c40b57b0 legacy=0x7f66c4062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:24.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.228+0000 7f66c27fc700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b0003730 0x7f66b0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:24.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.229+0000 7f66c27fc700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c41482d0 0x7f66b00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:24.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.229+0000 7f66c27fc700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b00049d0 0x7f66b0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:24.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.229+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c4148c80 con 0x7f66b0003730 2020-11-20T12:19:24.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.229+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b0004220 con 0x7f66b00049d0 2020-11-20T12:19:24.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.229+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b000b730 con 0x7f66c41482d0 2020-11-20T12:19:24.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.229+0000 7f66c3fff700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b00049d0 0x7f66b0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:24.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.229+0000 7f66c37fe700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b0003730 0x7f66b0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:24.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.229+0000 7f66c2ffd700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c41482d0 0x7f66b00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:24.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c3fff700 1 -- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c41482d0 msgr2=0x7f66b00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:24.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c3fff700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c41482d0 0x7f66b00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c3fff700 1 -- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b0003730 msgr2=0x7f66b0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:24.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c3fff700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b0003730 0x7f66b0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c3fff700 1 -- 172.21.15.187:0/1256628784 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b0003240 con 0x7f66b00049d0 2020-11-20T12:19:24.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c3fff700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b00049d0 0x7f66b0004e50 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 rx=0x7f66bc00ae30 tx=0x7f66bc00a370).ready entity=mon.1 client_cookie=b352eb92c48716a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:24.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc00a760 con 0x7f66b00049d0 2020-11-20T12:19:24.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc00fe00 con 0x7f66b00049d0 2020-11-20T12:19:24.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.230+0000 7f66c27fc700 1 -- 172.21.15.187:0/1256628784 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc012770 con 0x7f66b00049d0 2020-11-20T12:19:24.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.231+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1256628784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b00049d0 msgr2=0x7f66b0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:24.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.231+0000 7f66c8f8e700 1 --2- 172.21.15.187:0/1256628784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b00049d0 0x7f66b0004e50 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 rx=0x7f66bc00ae30 tx=0x7f66bc00a370).stop 2020-11-20T12:19:24.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.231+0000 7f66c3fff700 1 -- 172.21.15.187:0/1256628784 reap_dead start 2020-11-20T12:19:24.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.231+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1256628784 shutdown_connections 2020-11-20T12:19:24.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.231+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1256628784 >> 172.21.15.187:0/1256628784 conn(0x7f66c40bab30 msgr2=0x7f66c40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:24.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.231+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1256628784 shutdown_connections 2020-11-20T12:19:24.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.231+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1256628784 wait complete. 2020-11-20T12:19:24.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.233+0000 7f66c8f8e700 1 Processor -- start 2020-11-20T12:19:24.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.233+0000 7f66c8f8e700 1 -- start start 2020-11-20T12:19:24.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.234+0000 7f66c8f8e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c40b3080 0x7f66c40b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:24.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.234+0000 7f66c8f8e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66c4147320 0x7f66c409fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:24.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.234+0000 7f66c8f8e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c41477a0 0x7f66c409ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:24.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.235+0000 7f66c8f8e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c40b2930 con 0x7f66c4147320 2020-11-20T12:19:24.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.235+0000 7f66c8f8e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c40b2bd0 con 0x7f66c41477a0 2020-11-20T12:19:24.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.235+0000 7f66c8f8e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c4146e40 con 0x7f66c40b3080 2020-11-20T12:19:24.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.235+0000 7f66c3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c41477a0 0x7f66c409ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:24.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.235+0000 7f66c3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c41477a0 0x7f66c409ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52412/0 (socket says 172.21.15.187:52412) 2020-11-20T12:19:24.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.235+0000 7f66c3fff700 1 -- 172.21.15.187:0/1460806176 learned_addr learned my addr 172.21.15.187:0/1460806176 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:24.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.235+0000 7f66c2ffd700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66c4147320 0x7f66c409fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:24.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.235+0000 7f66c37fe700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c40b3080 0x7f66c40b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:24.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66c3fff700 1 -- 172.21.15.187:0/1460806176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c40b3080 msgr2=0x7f66c40b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:24.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66c3fff700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c40b3080 0x7f66c40b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66c3fff700 1 -- 172.21.15.187:0/1460806176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66c4147320 msgr2=0x7f66c409fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:24.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66c3fff700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66c4147320 0x7f66c409fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66c3fff700 1 -- 172.21.15.187:0/1460806176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c40b25b0 con 0x7f66c41477a0 2020-11-20T12:19:24.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66c3fff700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c41477a0 0x7f66c409ff40 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f66bc0011a0 tx=0x7f66bc01c000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:24.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66a37fe700 1 -- 172.21.15.187:0/1460806176 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc00a760 con 0x7f66c41477a0 2020-11-20T12:19:24.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66a37fe700 1 -- 172.21.15.187:0/1460806176 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc01ac00 con 0x7f66c41477a0 2020-11-20T12:19:24.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c4144e50 con 0x7f66c41477a0 2020-11-20T12:19:24.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66a37fe700 1 -- 172.21.15.187:0/1460806176 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc024420 con 0x7f66c41477a0 2020-11-20T12:19:24.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.236+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c40b3b10 con 0x7f66c41477a0 2020-11-20T12:19:24.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.238+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c4145040 con 0x7f66c41477a0 2020-11-20T12:19:24.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.238+0000 7f66a37fe700 1 -- 172.21.15.187:0/1460806176 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f66bc02b070 con 0x7f66c41477a0 2020-11-20T12:19:24.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.239+0000 7f66a37fe700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f669c056d50 0x7f669c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:24.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.239+0000 7f66a37fe700 1 -- 172.21.15.187:0/1460806176 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f66bc020070 con 0x7f66c41477a0 2020-11-20T12:19:24.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.239+0000 7f66c37fe700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f669c056d50 0x7f669c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:24.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.240+0000 7f66c37fe700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f669c056d50 0x7f669c059210 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f66b4000dd0 tx=0x7f66b4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:24.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.243+0000 7f66a37fe700 1 -- 172.21.15.187:0/1460806176 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f66c4145040 con 0x7f66c41477a0 2020-11-20T12:19:24.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.493+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade"]} v 0) v1 -- 0x7f66c4145040 con 0x7f66c41477a0 2020-11-20T12:19:24.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.948+0000 7f66a37fe700 1 -- 172.21.15.187:0/1460806176 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade"]}]=0 v720) v1 ==== 142+0+0 (secure 0 0 0) 0x7f66c4145040 con 0x7f66c41477a0 2020-11-20T12:19:24.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f669c056d50 msgr2=0x7f669c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:24.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f669c056d50 0x7f669c059210 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f66b4000dd0 tx=0x7f66b4009040).stop 2020-11-20T12:19:24.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c41477a0 msgr2=0x7f66c409ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:24.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c41477a0 0x7f66c409ff40 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f66bc0011a0 tx=0x7f66bc01c000).stop 2020-11-20T12:19:24.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 shutdown_connections 2020-11-20T12:19:24.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f669c056d50 0x7f669c059210 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c41477a0 0x7f66c409ff40 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66c40b3080 0x7f66c40b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.951+0000 7f66c8f8e700 1 --2- 172.21.15.187:0/1460806176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66c4147320 0x7f66c409fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:24.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.952+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 >> 172.21.15.187:0/1460806176 conn(0x7f66c40bab30 msgr2=0x7f66c4065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:24.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.952+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 shutdown_connections 2020-11-20T12:19:24.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:24.952+0000 7f66c8f8e700 1 -- 172.21.15.187:0/1460806176 wait complete. 2020-11-20T12:19:24.979 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:19:25.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.216+0000 7f3e4520c700 1 Processor -- start 2020-11-20T12:19:25.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.216+0000 7f3e4520c700 1 -- start start 2020-11-20T12:19:25.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.217+0000 7f3e4520c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e401484d0 0x7f3e40061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:25.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.217+0000 7f3e4520c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3e400622c0 0x7f3e400b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:25.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.217+0000 7f3e4520c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400b1300 con 0x7f3e40066f70 2020-11-20T12:19:25.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.217+0000 7f3e4520c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e40067730 con 0x7f3e400622c0 2020-11-20T12:19:25.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.217+0000 7f3e4520c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e400b15a0 con 0x7f3e401484d0 2020-11-20T12:19:25.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.218+0000 7f3e3ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e401484d0 0x7f3e40061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:25.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.218+0000 7f3e3ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e401484d0 0x7f3e40061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52416/0 (socket says 172.21.15.187:52416) 2020-11-20T12:19:25.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.218+0000 7f3e3ffff700 1 -- 172.21.15.187:0/1310007298 learned_addr learned my addr 172.21.15.187:0/1310007298 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:25.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.218+0000 7f3e377fe700 1 --2- 172.21.15.187:0/1310007298 >> v2:172.21.15.5:3300/0 conn(0x7f3e400622c0 0x7f3e400b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:25.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e3ffff700 1 -- 172.21.15.187:0/1310007298 >> v2:172.21.15.5:3300/0 conn(0x7f3e400622c0 msgr2=0x7f3e400b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:25.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e3ffff700 1 --2- 172.21.15.187:0/1310007298 >> v2:172.21.15.5:3300/0 conn(0x7f3e400622c0 0x7f3e400b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:25.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e3ffff700 1 -- 172.21.15.187:0/1310007298 >> v1:172.21.15.187:6789/0 conn(0x7f3e40066f70 legacy=0x7f3e400b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:25.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e3ffff700 1 -- 172.21.15.187:0/1310007298 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400b1840 con 0x7f3e401484d0 2020-11-20T12:19:25.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e377fe700 1 --2- 172.21.15.187:0/1310007298 >> v2:172.21.15.5:3300/0 conn(0x7f3e400622c0 0x7f3e400b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:25.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e3ffff700 1 --2- 172.21.15.187:0/1310007298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e401484d0 0x7f3e40061dd0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 rx=0x7f3e400b85e0 tx=0x7f3e30006eb0).ready entity=mon.1 client_cookie=e642e3b949e13ccd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:25.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e3f7fe700 1 -- 172.21.15.187:0/1310007298 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e300097d0 con 0x7f3e401484d0 2020-11-20T12:19:25.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e3f7fe700 1 -- 172.21.15.187:0/1310007298 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e30009930 con 0x7f3e401484d0 2020-11-20T12:19:25.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.219+0000 7f3e3f7fe700 1 -- 172.21.15.187:0/1310007298 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30009cb0 con 0x7f3e401484d0 2020-11-20T12:19:25.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.220+0000 7f3e4520c700 1 -- 172.21.15.187:0/1310007298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e401484d0 msgr2=0x7f3e40061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:25.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.220+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1310007298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e401484d0 0x7f3e40061dd0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 rx=0x7f3e400b85e0 tx=0x7f3e30006eb0).stop 2020-11-20T12:19:25.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.220+0000 7f3e4520c700 1 -- 172.21.15.187:0/1310007298 shutdown_connections 2020-11-20T12:19:25.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.220+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1310007298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e401484d0 0x7f3e40061dd0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:25.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.220+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1310007298 >> v2:172.21.15.5:3300/0 conn(0x7f3e400622c0 0x7f3e400b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:25.226 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.220+0000 7f3e4520c700 1 -- 172.21.15.187:0/1310007298 >> 172.21.15.187:0/1310007298 conn(0x7f3e400bab30 msgr2=0x7f3e400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:25.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.220+0000 7f3e4520c700 1 -- 172.21.15.187:0/1310007298 shutdown_connections 2020-11-20T12:19:25.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.220+0000 7f3e4520c700 1 -- 172.21.15.187:0/1310007298 wait complete. 2020-11-20T12:19:25.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.221+0000 7f3e4520c700 1 Processor -- start 2020-11-20T12:19:25.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.221+0000 7f3e4520c700 1 -- start start 2020-11-20T12:19:25.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.222+0000 7f3e4520c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e400622c0 0x7f3e40158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:25.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.222+0000 7f3e4520c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e40066f70 0x7f3e400b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:25.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.222+0000 7f3e4520c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e401484d0 0x7f3e40131bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:25.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.222+0000 7f3e4520c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4009fde0 con 0x7f3e400622c0 2020-11-20T12:19:25.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.222+0000 7f3e4520c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4009c040 con 0x7f3e40066f70 2020-11-20T12:19:25.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.222+0000 7f3e4520c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4009c2e0 con 0x7f3e401484d0 2020-11-20T12:19:25.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e40066f70 0x7f3e400b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:25.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e40066f70 0x7f3e400b3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52424/0 (socket says 172.21.15.187:52424) 2020-11-20T12:19:25.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 -- 172.21.15.187:0/1417909836 learned_addr learned my addr 172.21.15.187:0/1417909836 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:25.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 -- 172.21.15.187:0/1417909836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e401484d0 msgr2=0x7f3e40131bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:25.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e401484d0 0x7f3e40131bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:25.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e3ffff700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e400622c0 0x7f3e40158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:25.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 -- 172.21.15.187:0/1417909836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e400622c0 msgr2=0x7f3e40158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:25.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e400622c0 0x7f3e40158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:25.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 -- 172.21.15.187:0/1417909836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40158700 con 0x7f3e40066f70 2020-11-20T12:19:25.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.223+0000 7f3e377fe700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e40066f70 0x7f3e400b3d50 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f3e2800cb50 tx=0x7f3e28008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:25.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.224+0000 7f3e3cff9700 1 -- 172.21.15.187:0/1417909836 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28007860 con 0x7f3e40066f70 2020-11-20T12:19:25.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.224+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e400b4240 con 0x7f3e40066f70 2020-11-20T12:19:25.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.224+0000 7f3e3cff9700 1 -- 172.21.15.187:0/1417909836 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e280079c0 con 0x7f3e40066f70 2020-11-20T12:19:25.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.224+0000 7f3e3cff9700 1 -- 172.21.15.187:0/1417909836 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28007d40 con 0x7f3e40066f70 2020-11-20T12:19:25.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.224+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4009ba80 con 0x7f3e40066f70 2020-11-20T12:19:25.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.225+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4009fc30 con 0x7f3e40066f70 2020-11-20T12:19:25.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.225+0000 7f3e3cff9700 1 -- 172.21.15.187:0/1417909836 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3e28011620 con 0x7f3e40066f70 2020-11-20T12:19:25.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.226+0000 7f3e3cff9700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e1c056d50 0x7f3e1c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:25.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.226+0000 7f3e3cff9700 1 -- 172.21.15.187:0/1417909836 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3e280700a0 con 0x7f3e40066f70 2020-11-20T12:19:25.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.226+0000 7f3e3ffff700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e1c056d50 0x7f3e1c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:25.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.227+0000 7f3e3ffff700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e1c056d50 0x7f3e1c059210 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f3e3000b1b0 tx=0x7f3e30017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:25.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.230+0000 7f3e3cff9700 1 -- 172.21.15.187:0/1417909836 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3e28045440 con 0x7f3e40066f70 2020-11-20T12:19:25.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.418+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7f3e400b43d0 con 0x7f3e40066f70 2020-11-20T12:19:25.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.428+0000 7f3e3cff9700 1 -- 172.21.15.187:0/1417909836 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3e400b43d0 con 0x7f3e40066f70 2020-11-20T12:19:25.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e1c056d50 msgr2=0x7f3e1c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:25.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e1c056d50 0x7f3e1c059210 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f3e3000b1b0 tx=0x7f3e30017000).stop 2020-11-20T12:19:25.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e40066f70 msgr2=0x7f3e400b3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:25.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e40066f70 0x7f3e400b3d50 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f3e2800cb50 tx=0x7f3e28008040).stop 2020-11-20T12:19:25.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 shutdown_connections 2020-11-20T12:19:25.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e1c056d50 0x7f3e1c059210 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:25.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e400622c0 0x7f3e40158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:25.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e401484d0 0x7f3e40131bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:25.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 --2- 172.21.15.187:0/1417909836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e40066f70 0x7f3e400b3d50 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:25.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 >> 172.21.15.187:0/1417909836 conn(0x7f3e400bab30 msgr2=0x7f3e40144f10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:25.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.430+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 shutdown_connections 2020-11-20T12:19:25.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.431+0000 7f3e4520c700 1 -- 172.21.15.187:0/1417909836 wait complete. 2020-11-20T12:19:25.457 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:19:25.457 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:19:25.591 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:19:25.608 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:19:25.722 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:19:25.745 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:19:25.746 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:19:25.746 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:26.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.998+0000 7f55316e5700 1 Processor -- start 2020-11-20T12:19:26.001 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.998+0000 7f55316e5700 1 -- start start 2020-11-20T12:19:26.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.999+0000 7f55316e5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f552c1484d0 0x7f552c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.999+0000 7f55316e5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 0x7f552c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.999+0000 7f55316e5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c0b1300 con 0x7f552c0622c0 2020-11-20T12:19:26.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.999+0000 7f55316e5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f552c067730 con 0x7f552c1484d0 2020-11-20T12:19:26.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:25.999+0000 7f55316e5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552c0b15a0 con 0x7f552c066f70 2020-11-20T12:19:26.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.000+0000 7f552bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f552c1484d0 0x7f552c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.000+0000 7f552b7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f552c0622c0 0x7f552c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49038/0 (socket says 172.21.15.187:49038) 2020-11-20T12:19:26.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.000+0000 7f552b7fe700 1 -- 172.21.15.187:0/544223375 learned_addr learned my addr 172.21.15.187:0/544223375 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:26.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.000+0000 7f552affd700 1 -- 172.21.15.187:0/544223375 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386774074 0 0) 0x7f552c0b1300 con 0x7f552c0622c0 2020-11-20T12:19:26.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.000+0000 7f552affd700 1 -- 172.21.15.187:0/544223375 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5518003240 con 0x7f552c0622c0 2020-11-20T12:19:26.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.000+0000 7f5530ee4700 1 --2- 172.21.15.187:0/544223375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 0x7f552c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.001+0000 7f552bfff700 1 -- 172.21.15.187:0/544223375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 msgr2=0x7f552c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:26.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.001+0000 7f552bfff700 1 --2- 172.21.15.187:0/544223375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 0x7f552c0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.001+0000 7f552bfff700 1 -- 172.21.15.187:0/544223375 >> v1:172.21.15.187:6789/0 conn(0x7f552c0622c0 legacy=0x7f552c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.001+0000 7f552affd700 1 -- 172.21.15.187:0/544223375 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1749735826 0 0) 0x7f5518003240 con 0x7f552c0622c0 2020-11-20T12:19:26.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.001+0000 7f552bfff700 1 -- 172.21.15.187:0/544223375 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c0b1840 con 0x7f552c1484d0 2020-11-20T12:19:26.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.001+0000 7f552bfff700 1 --2- 172.21.15.187:0/544223375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c1484d0 0x7f552c061dd0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7f551c00b9a0 tx=0x7f551c006df0).ready entity=mon.1 client_cookie=d5df52151991e4f0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:26.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.002+0000 7f552affd700 1 -- 172.21.15.187:0/544223375 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520001860 con 0x7f552c1484d0 2020-11-20T12:19:26.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.002+0000 7f552affd700 1 -- 172.21.15.187:0/544223375 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c0094a0 con 0x7f552c1484d0 2020-11-20T12:19:26.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.002+0000 7f552affd700 1 -- 172.21.15.187:0/544223375 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c009820 con 0x7f552c1484d0 2020-11-20T12:19:26.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.002+0000 7f55316e5700 1 -- 172.21.15.187:0/544223375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c1484d0 msgr2=0x7f552c061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.002+0000 7f55316e5700 1 --2- 172.21.15.187:0/544223375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c1484d0 0x7f552c061dd0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7f551c00b9a0 tx=0x7f551c006df0).stop 2020-11-20T12:19:26.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.003+0000 7f55316e5700 1 -- 172.21.15.187:0/544223375 shutdown_connections 2020-11-20T12:19:26.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.003+0000 7f55316e5700 1 --2- 172.21.15.187:0/544223375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 0x7f552c0b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.003+0000 7f55316e5700 1 --2- 172.21.15.187:0/544223375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c1484d0 0x7f552c061dd0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.003+0000 7f55316e5700 1 -- 172.21.15.187:0/544223375 >> 172.21.15.187:0/544223375 conn(0x7f552c0bab30 msgr2=0x7f552c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:26.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.003+0000 7f55316e5700 1 -- 172.21.15.187:0/544223375 shutdown_connections 2020-11-20T12:19:26.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.003+0000 7f55316e5700 1 -- 172.21.15.187:0/544223375 wait complete. 2020-11-20T12:19:26.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.004+0000 7f55316e5700 1 Processor -- start 2020-11-20T12:19:26.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.004+0000 7f55316e5700 1 -- start start 2020-11-20T12:19:26.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.005+0000 7f55316e5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c0622c0 0x7f552c158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.005+0000 7f55316e5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 0x7f552c09c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.005+0000 7f55316e5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f552c1484d0 0x7f552c09fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.005+0000 7f55316e5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c09b920 con 0x7f552c1484d0 2020-11-20T12:19:26.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.005+0000 7f55316e5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c09bbc0 con 0x7f552c0622c0 2020-11-20T12:19:26.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.005+0000 7f55316e5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552c09be60 con 0x7f552c066f70 2020-11-20T12:19:26.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c0622c0 0x7f552c158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c0622c0 0x7f552c158840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52436/0 (socket says 172.21.15.187:52436) 2020-11-20T12:19:26.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 -- 172.21.15.187:0/2221505006 learned_addr learned my addr 172.21.15.187:0/2221505006 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:26.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552b7fe700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 0x7f552c09c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 -- 172.21.15.187:0/2221505006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 msgr2=0x7f552c09c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 0x7f552c09c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 -- 172.21.15.187:0/2221505006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f552c1484d0 msgr2=0x7f552c09fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f552c1484d0 0x7f552c09fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 -- 172.21.15.187:0/2221505006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c158d30 con 0x7f552c0622c0 2020-11-20T12:19:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.006+0000 7f552bfff700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c0622c0 0x7f552c158840 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7f551c006e30 tx=0x7f551c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.007+0000 7f550bfff700 1 -- 172.21.15.187:0/2221505006 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c011ca0 con 0x7f552c0622c0 2020-11-20T12:19:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.007+0000 7f550bfff700 1 -- 172.21.15.187:0/2221505006 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c011e00 con 0x7f552c0622c0 2020-11-20T12:19:26.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.007+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c09c7f0 con 0x7f552c0622c0 2020-11-20T12:19:26.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.007+0000 7f550bfff700 1 -- 172.21.15.187:0/2221505006 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c01e670 con 0x7f552c0622c0 2020-11-20T12:19:26.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.007+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552c09cde0 con 0x7f552c0622c0 2020-11-20T12:19:26.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.008+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552c09c020 con 0x7f552c0622c0 2020-11-20T12:19:26.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.008+0000 7f550bfff700 1 -- 172.21.15.187:0/2221505006 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f551c025030 con 0x7f552c0622c0 2020-11-20T12:19:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.009+0000 7f550bfff700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5504056d20 0x7f55040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.009+0000 7f550bfff700 1 -- 172.21.15.187:0/2221505006 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f551c0716c0 con 0x7f552c0622c0 2020-11-20T12:19:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.009+0000 7f552b7fe700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5504056d20 0x7f55040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.010+0000 7f552b7fe700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5504056d20 0x7f55040591e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7f55200012f0 tx=0x7f5520009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:26.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.013+0000 7f550bfff700 1 -- 172.21.15.187:0/2221505006 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f551c046a60 con 0x7f552c0622c0 2020-11-20T12:19:26.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.252+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f552c061a20 con 0x7f552c0622c0 2020-11-20T12:19:26.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.252+0000 7f550bfff700 1 -- 172.21.15.187:0/2221505006 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+2146 (secure 0 0 0) 0x7f551c023070 con 0x7f552c0622c0 2020-11-20T12:19:26.255 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:26.256 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":131,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3102570506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3102570506},{"type":"v1","addr":"172.21.15.5:6833","nonce":3102570506}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:19:19.889320+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8932},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:19:26.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.254+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5504056d20 msgr2=0x7f55040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.254+0000 7f55316e5700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5504056d20 0x7f55040591e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7f55200012f0 tx=0x7f5520009040).stop 2020-11-20T12:19:26.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.254+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c0622c0 msgr2=0x7f552c158840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.254+0000 7f55316e5700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c0622c0 0x7f552c158840 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7f551c006e30 tx=0x7f551c015000).stop 2020-11-20T12:19:26.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.255+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 shutdown_connections 2020-11-20T12:19:26.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.255+0000 7f55316e5700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5504056d20 0x7f55040591e0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.255+0000 7f55316e5700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f552c1484d0 0x7f552c09fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:26.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.255+0000 7f55316e5700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c0622c0 0x7f552c158840 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.255+0000 7f55316e5700 1 --2- 172.21.15.187:0/2221505006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f552c066f70 0x7f552c09c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.255+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 >> 172.21.15.187:0/2221505006 conn(0x7f552c0bab30 msgr2=0x7f552c0b5500 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:26.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.255+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 shutdown_connections 2020-11-20T12:19:26.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.255+0000 7f55316e5700 1 -- 172.21.15.187:0/2221505006 wait complete. 2020-11-20T12:19:26.260 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 131 2020-11-20T12:19:26.281 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:19:26.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.520+0000 7f48e0d96700 1 Processor -- start 2020-11-20T12:19:26.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.520+0000 7f48e0d96700 1 -- start start 2020-11-20T12:19:26.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.521+0000 7f48e0d96700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc0b57b0 0x7f48dc068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.521+0000 7f48e0d96700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc144b80 con 0x7f48dc0b5f50 2020-11-20T12:19:26.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.521+0000 7f48e0d96700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc1489e0 con 0x7f48dc1482d0 2020-11-20T12:19:26.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.521+0000 7f48e0d96700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc148c80 con 0x7f48dc0b57b0 2020-11-20T12:19:26.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.522+0000 7f48daffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f48dc1482d0 0x7f48dc062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49052/0 (socket says 172.21.15.187:49052) 2020-11-20T12:19:26.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.522+0000 7f48daffd700 1 -- 172.21.15.187:0/3939208870 learned_addr learned my addr 172.21.15.187:0/3939208870 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:26.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.522+0000 7f48db7fe700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc0b57b0 0x7f48dc068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.522+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957602989 0 0) 0x7f48dc1489e0 con 0x7f48dc1482d0 2020-11-20T12:19:26.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.522+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c8004350 con 0x7f48dc1482d0 2020-11-20T12:19:26.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2349778375 0 0) 0x7f48c8004350 con 0x7f48dc1482d0 2020-11-20T12:19:26.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc0b57b0 msgr2=0x7f48dc068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:26.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc0b57b0 0x7f48dc068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 >> v1:172.21.15.5:6789/0 conn(0x7f48dc0b5f50 legacy=0x7f48dc0b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc148f20 con 0x7f48dc1482d0 2020-11-20T12:19:26.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f48d00014e0 con 0x7f48dc1482d0 2020-11-20T12:19:26.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 >> v1:172.21.15.187:6789/0 conn(0x7f48dc1482d0 legacy=0x7f48dc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48c8004840 0x7f48c8004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48c80055f0 0x7f48c8005a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc0b57b0 0x7f48c8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc148c80 con 0x7f48c8004840 2020-11-20T12:19:26.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c80064a0 con 0x7f48c80055f0 2020-11-20T12:19:26.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c8005450 con 0x7f48dc0b57b0 2020-11-20T12:19:26.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.523+0000 7f48daffd700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48c80055f0 0x7f48c8005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48daffd700 1 -- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc0b57b0 msgr2=0x7f48c8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:26.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48daffd700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc0b57b0 0x7f48c8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:26.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48daffd700 1 -- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48c8004840 msgr2=0x7f48c8004c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:26.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48daffd700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48c8004840 0x7f48c8004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:26.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48daffd700 1 -- 172.21.15.187:0/3939208870 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8004350 con 0x7f48c80055f0 2020-11-20T12:19:26.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48daffd700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48c80055f0 0x7f48c8005a70 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7f48d000c1e0 tx=0x7f48d00076d0).ready entity=mon.1 client_cookie=c89427fa4b4bda7f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:26.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00070d0 con 0x7f48c80055f0 2020-11-20T12:19:26.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d0007230 con 0x7f48c80055f0 2020-11-20T12:19:26.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.524+0000 7f48da7fc700 1 -- 172.21.15.187:0/3939208870 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00055f0 con 0x7f48c80055f0 2020-11-20T12:19:26.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.525+0000 7f48e0d96700 1 -- 172.21.15.187:0/3939208870 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48c80055f0 msgr2=0x7f48c8005a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.525+0000 7f48e0d96700 1 --2- 172.21.15.187:0/3939208870 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48c80055f0 0x7f48c8005a70 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7f48d000c1e0 tx=0x7f48d00076d0).stop 2020-11-20T12:19:26.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.525+0000 7f48dbfff700 1 -- 172.21.15.187:0/3939208870 reap_dead start 2020-11-20T12:19:26.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.525+0000 7f48e0d96700 1 -- 172.21.15.187:0/3939208870 shutdown_connections 2020-11-20T12:19:26.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.525+0000 7f48e0d96700 1 -- 172.21.15.187:0/3939208870 >> 172.21.15.187:0/3939208870 conn(0x7f48dc0bab30 msgr2=0x7f48dc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:26.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.525+0000 7f48e0d96700 1 -- 172.21.15.187:0/3939208870 shutdown_connections 2020-11-20T12:19:26.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.525+0000 7f48e0d96700 1 -- 172.21.15.187:0/3939208870 wait complete. 2020-11-20T12:19:26.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.527+0000 7f48e0d96700 1 Processor -- start 2020-11-20T12:19:26.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.527+0000 7f48e0d96700 1 -- start start 2020-11-20T12:19:26.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.528+0000 7f48e0d96700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc09d000 0x7f48dc1451f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.528+0000 7f48e0d96700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09d3e0 0x7f48dc153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.528+0000 7f48e0d96700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc1544c0 0x7f48dc09f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.528+0000 7f48e0d96700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0b28d0 con 0x7f48dc09d000 2020-11-20T12:19:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.528+0000 7f48e0d96700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc154950 con 0x7f48dc09d3e0 2020-11-20T12:19:26.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.528+0000 7f48e0d96700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc154bf0 con 0x7f48dc1544c0 2020-11-20T12:19:26.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.528+0000 7f48daffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09d3e0 0x7f48dc153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48daffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09d3e0 0x7f48dc153fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52458/0 (socket says 172.21.15.187:52458) 2020-11-20T12:19:26.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48daffd700 1 -- 172.21.15.187:0/3932919275 learned_addr learned my addr 172.21.15.187:0/3932919275 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:26.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48db7fe700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc09d000 0x7f48dc1451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48daffd700 1 -- 172.21.15.187:0/3932919275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc1544c0 msgr2=0x7f48dc09f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48daffd700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc1544c0 0x7f48dc09f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:26.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48daffd700 1 -- 172.21.15.187:0/3932919275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc09d000 msgr2=0x7f48dc1451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48daffd700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc09d000 0x7f48dc1451f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48daffd700 1 -- 172.21.15.187:0/3932919275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc0b3e30 con 0x7f48dc09d3e0 2020-11-20T12:19:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48daffd700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09d3e0 0x7f48dc153fd0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 rx=0x7f48dc063440 tx=0x7f48d0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48bb7fe700 1 -- 172.21.15.187:0/3932919275 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d0007420 con 0x7f48dc09d3e0 2020-11-20T12:19:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.529+0000 7f48bb7fe700 1 -- 172.21.15.187:0/3932919275 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d0013660 con 0x7f48dc09d3e0 2020-11-20T12:19:26.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.530+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48dc09f520 con 0x7f48dc09d3e0 2020-11-20T12:19:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.530+0000 7f48bb7fe700 1 -- 172.21.15.187:0/3932919275 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d0013a20 con 0x7f48dc09d3e0 2020-11-20T12:19:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.530+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc09fa10 con 0x7f48dc09d3e0 2020-11-20T12:19:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.531+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48dc154db0 con 0x7f48dc09d3e0 2020-11-20T12:19:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.531+0000 7f48bb7fe700 1 -- 172.21.15.187:0/3932919275 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f48d0010070 con 0x7f48dc09d3e0 2020-11-20T12:19:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.532+0000 7f48bb7fe700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b4056d20 0x7f48b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.532+0000 7f48bb7fe700 1 -- 172.21.15.187:0/3932919275 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f48d0071770 con 0x7f48dc09d3e0 2020-11-20T12:19:26.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.532+0000 7f48db7fe700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b4056d20 0x7f48b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:26.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.533+0000 7f48db7fe700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b4056d20 0x7f48b40591e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f48cc006fd0 tx=0x7f48cc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:26.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.535+0000 7f48bb7fe700 1 -- 172.21.15.187:0/3932919275 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f48d0046b20 con 0x7f48dc09d3e0 2020-11-20T12:19:26.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.722+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48dc061a20 con 0x7f48dc09d3e0 2020-11-20T12:19:26.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.724+0000 7f48bb7fe700 1 -- 172.21.15.187:0/3932919275 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16919 (secure 0 0 0) 0x7f48d0025020 con 0x7f48dc09d3e0 2020-11-20T12:19:26.728 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:26.729 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":131,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:18:28.833768+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:18:08.692656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2020-11-20T12:18:09.325174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:6833/3549193191":"2020-11-21T12:18:28.833712+0000","172.21.15.5:6832/3549193191":"2020-11-21T12:18:28.833712+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:19:26.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.726+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b4056d20 msgr2=0x7f48b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.726+0000 7f48e0d96700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b4056d20 0x7f48b40591e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f48cc006fd0 tx=0x7f48cc008040).stop 2020-11-20T12:19:26.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.726+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09d3e0 msgr2=0x7f48dc153fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:26.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.726+0000 7f48e0d96700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09d3e0 0x7f48dc153fd0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 rx=0x7f48dc063440 tx=0x7f48d0015000).stop 2020-11-20T12:19:26.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.727+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 shutdown_connections 2020-11-20T12:19:26.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.727+0000 7f48e0d96700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f48b4056d20 0x7f48b40591e0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.727+0000 7f48e0d96700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48dc09d000 0x7f48dc1451f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.727+0000 7f48e0d96700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48dc1544c0 0x7f48dc09f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:26.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.727+0000 7f48e0d96700 1 --2- 172.21.15.187:0/3932919275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48dc09d3e0 0x7f48dc153fd0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:26.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.727+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 >> 172.21.15.187:0/3932919275 conn(0x7f48dc0bab30 msgr2=0x7f48dc0b7270 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:26.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.727+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 shutdown_connections 2020-11-20T12:19:26.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:26.727+0000 7f48e0d96700 1 -- 172.21.15.187:0/3932919275 wait complete. 2020-11-20T12:19:26.757 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:18:08.692656+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:18:09.325174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:19:26.762 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:19:27.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.016+0000 7f1c1d7b4700 1 Processor -- start 2020-11-20T12:19:27.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.016+0000 7f1c1d7b4700 1 -- start start 2020-11-20T12:19:27.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.017+0000 7f1c1d7b4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1c180bf800 0x7f1c1814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.017+0000 7f1c1d7b4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1c1809f560 0x7f1c180a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.017+0000 7f1c1d7b4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1814d110 con 0x7f1c180bfbe0 2020-11-20T12:19:27.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.017+0000 7f1c1d7b4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1814cbd0 con 0x7f1c1809f560 2020-11-20T12:19:27.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.017+0000 7f1c1d7b4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1814ce70 con 0x7f1c180bf800 2020-11-20T12:19:27.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.018+0000 7f1c17fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1c180bf800 0x7f1c1814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.018+0000 7f1c17fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1c180bf800 0x7f1c1814c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52464/0 (socket says 172.21.15.187:52464) 2020-11-20T12:19:27.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.018+0000 7f1c17fff700 1 -- 172.21.15.187:0/3659395276 learned_addr learned my addr 172.21.15.187:0/3659395276 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:27.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.018+0000 7f1c1cfb3700 1 --2- 172.21.15.187:0/3659395276 >> v2:172.21.15.5:3300/0 conn(0x7f1c1809f560 0x7f1c180a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.018+0000 7f1c17fff700 1 -- 172.21.15.187:0/3659395276 >> v2:172.21.15.5:3300/0 conn(0x7f1c1809f560 msgr2=0x7f1c180a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:27.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.019+0000 7f1c17fff700 1 --2- 172.21.15.187:0/3659395276 >> v2:172.21.15.5:3300/0 conn(0x7f1c1809f560 0x7f1c180a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.019+0000 7f1c17fff700 1 -- 172.21.15.187:0/3659395276 >> v1:172.21.15.5:6789/0 conn(0x7f1c180bfbe0 legacy=0x7f1c181480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:27.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.019+0000 7f1c17fff700 1 -- 172.21.15.187:0/3659395276 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c180a4cc0 con 0x7f1c180bf800 2020-11-20T12:19:27.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.019+0000 7f1c17fff700 1 --2- 172.21.15.187:0/3659395276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c180bf800 0x7f1c1814c2c0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7f1c18068780 tx=0x7f1c08009290).ready entity=mon.1 client_cookie=9af34489466696bb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:27.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.019+0000 7f1c16ffd700 1 -- 172.21.15.187:0/3659395276 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0800bc10 con 0x7f1c180bf800 2020-11-20T12:19:27.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.019+0000 7f1c16ffd700 1 -- 172.21.15.187:0/3659395276 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0800bd70 con 0x7f1c180bf800 2020-11-20T12:19:27.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.019+0000 7f1c16ffd700 1 -- 172.21.15.187:0/3659395276 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08011610 con 0x7f1c180bf800 2020-11-20T12:19:27.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.020+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/3659395276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c180bf800 msgr2=0x7f1c1814c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.020+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/3659395276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c180bf800 0x7f1c1814c2c0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7f1c18068780 tx=0x7f1c08009290).stop 2020-11-20T12:19:27.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.020+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/3659395276 shutdown_connections 2020-11-20T12:19:27.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.020+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/3659395276 >> v2:172.21.15.5:3300/0 conn(0x7f1c1809f560 0x7f1c180a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.020+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/3659395276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c180bf800 0x7f1c1814c2c0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.020+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/3659395276 >> 172.21.15.187:0/3659395276 conn(0x7f1c18068370 msgr2=0x7f1c180bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:27.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.020+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/3659395276 shutdown_connections 2020-11-20T12:19:27.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.020+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/3659395276 wait complete. 2020-11-20T12:19:27.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.022+0000 7f1c1d7b4700 1 Processor -- start 2020-11-20T12:19:27.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.022+0000 7f1c1d7b4700 1 -- start start 2020-11-20T12:19:27.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.023+0000 7f1c1d7b4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c1809f560 0x7f1c1813f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.023+0000 7f1c1d7b4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c180bf800 0x7f1c18145070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.023+0000 7f1c1d7b4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c18144160 0x7f1c180ba740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.023+0000 7f1c1d7b4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c18145e20 con 0x7f1c1809f560 2020-11-20T12:19:27.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.023+0000 7f1c1d7b4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c181458e0 con 0x7f1c18144160 2020-11-20T12:19:27.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.023+0000 7f1c1d7b4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c18145b80 con 0x7f1c180bf800 2020-11-20T12:19:27.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c1cfb3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c18144160 0x7f1c180ba740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c1cfb3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c18144160 0x7f1c180ba740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52474/0 (socket says 172.21.15.187:52474) 2020-11-20T12:19:27.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c1cfb3700 1 -- 172.21.15.187:0/2779158618 learned_addr learned my addr 172.21.15.187:0/2779158618 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:27.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c17fff700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c1809f560 0x7f1c1813f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c177fe700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c180bf800 0x7f1c18145070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c1cfb3700 1 -- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c180bf800 msgr2=0x7f1c18145070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c1cfb3700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c180bf800 0x7f1c18145070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c1cfb3700 1 -- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c1809f560 msgr2=0x7f1c1813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c1cfb3700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c1809f560 0x7f1c1813f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c1cfb3700 1 -- 172.21.15.187:0/2779158618 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1813f5e0 con 0x7f1c18144160 2020-11-20T12:19:27.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.024+0000 7f1c17fff700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c1809f560 0x7f1c1813f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:19:27.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.025+0000 7f1c1cfb3700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c18144160 0x7f1c180ba740 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f1c1000d830 tx=0x7f1c10008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:27.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.025+0000 7f1bf7fff700 1 -- 172.21.15.187:0/2779158618 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10009620 con 0x7f1c18144160 2020-11-20T12:19:27.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.025+0000 7f1bf7fff700 1 -- 172.21.15.187:0/2779158618 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c10009780 con 0x7f1c18144160 2020-11-20T12:19:27.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.025+0000 7f1bf7fff700 1 -- 172.21.15.187:0/2779158618 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10009b40 con 0x7f1c18144160 2020-11-20T12:19:27.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.025+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c18145560 con 0x7f1c18144160 2020-11-20T12:19:27.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.025+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c180badf0 con 0x7f1c18144160 2020-11-20T12:19:27.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.026+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c18080050 con 0x7f1c18144160 2020-11-20T12:19:27.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.027+0000 7f1bf7fff700 1 -- 172.21.15.187:0/2779158618 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1c10012420 con 0x7f1c18144160 2020-11-20T12:19:27.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.027+0000 7f1bf7fff700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bf0056d20 0x7f1bf00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.027+0000 7f1bf7fff700 1 -- 172.21.15.187:0/2779158618 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1c1006fc50 con 0x7f1c18144160 2020-11-20T12:19:27.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.028+0000 7f1c17fff700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bf0056d20 0x7f1bf00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.028+0000 7f1c17fff700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bf0056d20 0x7f1bf00591e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f1c080090f0 tx=0x7f1c08016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:27.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.032+0000 7f1bf7fff700 1 -- 172.21.15.187:0/2779158618 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1c10044ff0 con 0x7f1c18144160 2020-11-20T12:19:27.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.214+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f1c18145750 con 0x7f1c18144160 2020-11-20T12:19:27.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.215+0000 7f1bf7fff700 1 -- 172.21.15.187:0/2779158618 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v131) v1 ==== 110+0+10 (secure 0 0 0) 0x7f1c10048610 con 0x7f1c18144160 2020-11-20T12:19:27.218 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:19:27.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bf0056d20 msgr2=0x7f1bf00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bf0056d20 0x7f1bf00591e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f1c080090f0 tx=0x7f1c08016000).stop 2020-11-20T12:19:27.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c18144160 msgr2=0x7f1c180ba740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c18144160 0x7f1c180ba740 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f1c1000d830 tx=0x7f1c10008040).stop 2020-11-20T12:19:27.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 shutdown_connections 2020-11-20T12:19:27.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bf0056d20 0x7f1bf00591e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c1809f560 0x7f1c1813f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c18144160 0x7f1c180ba740 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 --2- 172.21.15.187:0/2779158618 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c180bf800 0x7f1c18145070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 >> 172.21.15.187:0/2779158618 conn(0x7f1c18068370 msgr2=0x7f1c180bc140 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:27.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 shutdown_connections 2020-11-20T12:19:27.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.218+0000 7f1c1d7b4700 1 -- 172.21.15.187:0/2779158618 wait complete. 2020-11-20T12:19:27.242 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:19:27.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.503+0000 7f6d01fa8700 1 Processor -- start 2020-11-20T12:19:27.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.504+0000 7f6d01fa8700 1 -- start start 2020-11-20T12:19:27.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.505+0000 7f6d01fa8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6cfc0bf800 0x7f6cfc14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.505+0000 7f6d01fa8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cfc14d110 con 0x7f6cfc09f560 2020-11-20T12:19:27.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.505+0000 7f6d01fa8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cfc14cbd0 con 0x7f6cfc0bfbe0 2020-11-20T12:19:27.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.505+0000 7f6d01fa8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cfc14ce70 con 0x7f6cfc0bf800 2020-11-20T12:19:27.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.505+0000 7f6cfbfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6cfc0bfbe0 0x7f6cfc1480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49086/0 (socket says 172.21.15.187:49086) 2020-11-20T12:19:27.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.505+0000 7f6cfbfff700 1 -- 172.21.15.187:0/3496661985 learned_addr learned my addr 172.21.15.187:0/3496661985 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:27.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.506+0000 7f6d00fa6700 1 --2- 172.21.15.187:0/3496661985 >> v2:172.21.15.5:3300/0 conn(0x7f6cfc0bf800 0x7f6cfc14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.506+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171530769 0 0) 0x7f6cfc14cbd0 con 0x7f6cfc0bfbe0 2020-11-20T12:19:27.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.506+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ce0003240 con 0x7f6cfc0bfbe0 2020-11-20T12:19:27.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.506+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 660875822 0 0) 0x7f6ce0003240 con 0x7f6cfc0bfbe0 2020-11-20T12:19:27.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.506+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 >> v2:172.21.15.5:3300/0 conn(0x7f6cfc0bf800 msgr2=0x7f6cfc14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:27.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.506+0000 7f6cfb7fe700 1 --2- 172.21.15.187:0/3496661985 >> v2:172.21.15.5:3300/0 conn(0x7f6cfc0bf800 0x7f6cfc14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.506+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 >> v1:172.21.15.5:6789/0 conn(0x7f6cfc09f560 legacy=0x7f6cfc0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc0a4cc0 con 0x7f6cfc0bfbe0 2020-11-20T12:19:27.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6ce8001520 con 0x7f6cfc0bfbe0 2020-11-20T12:19:27.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 >> v1:172.21.15.187:6789/0 conn(0x7f6cfc0bfbe0 legacy=0x7f6cfc1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ce0003730 0x7f6ce0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ce00044e0 0x7f6ce0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6ce000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc14ce70 con 0x7f6ce0003730 2020-11-20T12:19:27.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0004260 con 0x7f6ce0004e50 2020-11-20T12:19:27.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce000bc20 con 0x7f6ce00044e0 2020-11-20T12:19:27.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.507+0000 7f6d017a7700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6ce000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6cfbfff700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ce00044e0 0x7f6ce0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d00fa6700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ce0003730 0x7f6ce0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d017a7700 1 -- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ce00044e0 msgr2=0x7f6ce0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:27.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d017a7700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6ce00044e0 0x7f6ce0004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d017a7700 1 -- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ce0003730 msgr2=0x7f6ce0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:27.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d017a7700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ce0003730 0x7f6ce0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d017a7700 1 -- 172.21.15.187:0/3496661985 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0003240 con 0x7f6ce0004e50 2020-11-20T12:19:27.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d00fa6700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ce0003730 0x7f6ce0003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:27.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d017a7700 1 -- 172.21.15.187:0/3496661985 reap_dead start 2020-11-20T12:19:27.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6d017a7700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6ce000b730 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 rx=0x7f6cf4001e40 tx=0x7f6cf4007f10).ready entity=mon.1 client_cookie=c4c0da2c8f6730ac server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:27.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4001540 con 0x7f6ce0004e50 2020-11-20T12:19:27.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.508+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf400cad0 con 0x7f6ce0004e50 2020-11-20T12:19:27.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.509+0000 7f6cfb7fe700 1 -- 172.21.15.187:0/3496661985 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400ce70 con 0x7f6ce0004e50 2020-11-20T12:19:27.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.509+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3496661985 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 msgr2=0x7f6ce000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.509+0000 7f6d01fa8700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6ce000b730 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 rx=0x7f6cf4001e40 tx=0x7f6cf4007f10).stop 2020-11-20T12:19:27.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.509+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3496661985 shutdown_connections 2020-11-20T12:19:27.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.509+0000 7f6d01fa8700 1 --2- 172.21.15.187:0/3496661985 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6ce000b730 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.509+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3496661985 >> 172.21.15.187:0/3496661985 conn(0x7f6cfc068370 msgr2=0x7f6cfc0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:27.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.509+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3496661985 shutdown_connections 2020-11-20T12:19:27.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.509+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3496661985 wait complete. 2020-11-20T12:19:27.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.511+0000 7f6d01fa8700 1 Processor -- start 2020-11-20T12:19:27.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.511+0000 7f6d01fa8700 1 -- start start 2020-11-20T12:19:27.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.512+0000 7f6d01fa8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6cfc149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.512+0000 7f6d01fa8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc1437f0 0x7f6cfc1424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.512+0000 7f6d01fa8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc1463e0 0x7f6cfc146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.512+0000 7f6d01fa8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc1496b0 con 0x7f6cfc1437f0 2020-11-20T12:19:27.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.512+0000 7f6d01fa8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc149950 con 0x7f6ce0004e50 2020-11-20T12:19:27.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.512+0000 7f6d01fa8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc149bf0 con 0x7f6cfc1463e0 2020-11-20T12:19:27.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6cfc149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6cfc149010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52492/0 (socket says 172.21.15.187:52492) 2020-11-20T12:19:27.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 -- 172.21.15.187:0/3159787892 learned_addr learned my addr 172.21.15.187:0/3159787892 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:27.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 -- 172.21.15.187:0/3159787892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc1463e0 msgr2=0x7f6cfc146860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6cfbfff700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc1437f0 0x7f6cfc1424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc1463e0 0x7f6cfc146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:27.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 -- 172.21.15.187:0/3159787892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc1437f0 msgr2=0x7f6cfc1424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc1437f0 0x7f6cfc1424a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 -- 172.21.15.187:0/3159787892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc142990 con 0x7f6ce0004e50 2020-11-20T12:19:27.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.513+0000 7f6d00fa6700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6cfc149010 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 rx=0x7f6cf000f830 tx=0x7f6cf0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:27.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.514+0000 7f6cf8ff9700 1 -- 172.21.15.187:0/3159787892 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0017070 con 0x7f6ce0004e50 2020-11-20T12:19:27.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.514+0000 7f6cf8ff9700 1 -- 172.21.15.187:0/3159787892 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf0011660 con 0x7f6ce0004e50 2020-11-20T12:19:27.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.514+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cfc145ac0 con 0x7f6ce0004e50 2020-11-20T12:19:27.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.514+0000 7f6cf8ff9700 1 -- 172.21.15.187:0/3159787892 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf00119e0 con 0x7f6ce0004e50 2020-11-20T12:19:27.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.514+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cfc15c080 con 0x7f6ce0004e50 2020-11-20T12:19:27.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.515+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cfc149500 con 0x7f6ce0004e50 2020-11-20T12:19:27.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.516+0000 7f6cf8ff9700 1 -- 172.21.15.187:0/3159787892 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6cf001c410 con 0x7f6ce0004e50 2020-11-20T12:19:27.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.516+0000 7f6cf8ff9700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6cd4056d50 0x7f6cd4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:27.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.516+0000 7f6cf8ff9700 1 -- 172.21.15.187:0/3159787892 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6cf006fe50 con 0x7f6ce0004e50 2020-11-20T12:19:27.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.516+0000 7f6cfbfff700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6cd4056d50 0x7f6cd4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:27.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.517+0000 7f6cfbfff700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6cd4056d50 0x7f6cd4059210 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f6ce80013f0 tx=0x7f6ce8001460).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:27.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.520+0000 7f6cf8ff9700 1 -- 172.21.15.187:0/3159787892 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6cf00451f0 con 0x7f6ce0004e50 2020-11-20T12:19:27.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.706+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6cfc080050 con 0x7f6ce0004e50 2020-11-20T12:19:27.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.707+0000 7f6cf8ff9700 1 -- 172.21.15.187:0/3159787892 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v131) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6cf0048810 con 0x7f6ce0004e50 2020-11-20T12:19:27.713 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:19:27.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.709+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6cd4056d50 msgr2=0x7f6cd4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.709+0000 7f6d01fa8700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6cd4056d50 0x7f6cd4059210 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f6ce80013f0 tx=0x7f6ce8001460).stop 2020-11-20T12:19:27.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 msgr2=0x7f6cfc149010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:27.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6cfc149010 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 rx=0x7f6cf000f830 tx=0x7f6cf0008040).stop 2020-11-20T12:19:27.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 shutdown_connections 2020-11-20T12:19:27.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6cd4056d50 0x7f6cd4059210 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc1463e0 0x7f6cfc146860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:27.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004e50 0x7f6cfc149010 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 --2- 172.21.15.187:0/3159787892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc1437f0 0x7f6cfc1424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:27.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 >> 172.21.15.187:0/3159787892 conn(0x7f6cfc068370 msgr2=0x7f6cfc0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:27.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 shutdown_connections 2020-11-20T12:19:27.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:27.710+0000 7f6d01fa8700 1 -- 172.21.15.187:0/3159787892 wait complete. 2020-11-20T12:19:27.738 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:19:28.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.009+0000 7fd94a5e2700 1 Processor -- start 2020-11-20T12:19:28.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.009+0000 7fd94a5e2700 1 -- start start 2020-11-20T12:19:28.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.010+0000 7fd94a5e2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9440bf800 0x7fd94414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.010+0000 7fd94a5e2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94414d110 con 0x7fd94409f560 2020-11-20T12:19:28.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.010+0000 7fd94a5e2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94414cbd0 con 0x7fd9440bfbe0 2020-11-20T12:19:28.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.010+0000 7fd94a5e2700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd94414ce70 con 0x7fd9440bf800 2020-11-20T12:19:28.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.011+0000 7fd948ddf700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd9440bfbe0 0x7fd9441480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49106/0 (socket says 172.21.15.187:49106) 2020-11-20T12:19:28.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.011+0000 7fd948ddf700 1 -- 172.21.15.187:0/1331216633 learned_addr learned my addr 172.21.15.187:0/1331216633 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:28.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.011+0000 7fd9495e0700 1 --2- 172.21.15.187:0/1331216633 >> v2:172.21.15.5:3300/0 conn(0x7fd9440bf800 0x7fd94414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.011+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503915448 0 0) 0x7fd94414cbd0 con 0x7fd9440bfbe0 2020-11-20T12:19:28.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.011+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd92c003240 con 0x7fd9440bfbe0 2020-11-20T12:19:28.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1784168861 0 0) 0x7fd92c003240 con 0x7fd9440bfbe0 2020-11-20T12:19:28.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 >> v2:172.21.15.5:3300/0 conn(0x7fd9440bf800 msgr2=0x7fd94414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:28.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 --2- 172.21.15.187:0/1331216633 >> v2:172.21.15.5:3300/0 conn(0x7fd9440bf800 0x7fd94414c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 >> v1:172.21.15.5:6789/0 conn(0x7fd94409f560 legacy=0x7fd9440a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9440a4cc0 con 0x7fd9440bfbe0 2020-11-20T12:19:28.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fd940001520 con 0x7fd9440bfbe0 2020-11-20T12:19:28.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 >> v1:172.21.15.187:6789/0 conn(0x7fd9440bfbe0 legacy=0x7fd9441480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd92c003730 0x7fd92c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c0044e0 0x7fd92c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004e50 0x7fd92c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94414ce70 con 0x7fd92c0044e0 2020-11-20T12:19:28.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c004260 con 0x7fd92c004e50 2020-11-20T12:19:28.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.012+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c00bc20 con 0x7fd92c003730 2020-11-20T12:19:28.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd949de1700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004e50 0x7fd92c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd948ddf700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c0044e0 0x7fd92c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd949de1700 1 -- 172.21.15.187:0/1331216633 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd92c003730 msgr2=0x7fd92c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:28.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd949de1700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd92c003730 0x7fd92c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:28.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd949de1700 1 -- 172.21.15.187:0/1331216633 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c0044e0 msgr2=0x7fd92c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:28.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd949de1700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c0044e0 0x7fd92c004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd949de1700 1 -- 172.21.15.187:0/1331216633 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c003240 con 0x7fd92c004e50 2020-11-20T12:19:28.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd949de1700 1 -- 172.21.15.187:0/1331216633 reap_dead start 2020-11-20T12:19:28.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.013+0000 7fd949de1700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004e50 0x7fd92c00b730 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 rx=0x7fd93c001e40 tx=0x7fd93c007f10).ready entity=mon.1 client_cookie=79945efc11802dcc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c001540 con 0x7fd92c004e50 2020-11-20T12:19:28.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c00cad0 con 0x7fd92c004e50 2020-11-20T12:19:28.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd933fff700 1 -- 172.21.15.187:0/1331216633 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00ce70 con 0x7fd92c004e50 2020-11-20T12:19:28.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1331216633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004e50 msgr2=0x7fd92c00b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd94a5e2700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004e50 0x7fd92c00b730 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 rx=0x7fd93c001e40 tx=0x7fd93c007f10).stop 2020-11-20T12:19:28.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1331216633 shutdown_connections 2020-11-20T12:19:28.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd94a5e2700 1 --2- 172.21.15.187:0/1331216633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd92c004e50 0x7fd92c00b730 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1331216633 >> 172.21.15.187:0/1331216633 conn(0x7fd944068370 msgr2=0x7fd9440bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:28.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1331216633 shutdown_connections 2020-11-20T12:19:28.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.014+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1331216633 wait complete. 2020-11-20T12:19:28.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.016+0000 7fd94a5e2700 1 Processor -- start 2020-11-20T12:19:28.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.016+0000 7fd94a5e2700 1 -- start start 2020-11-20T12:19:28.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd94a5e2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c004e50 0x7fd944135930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd94a5e2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94414beb0 0x7fd944158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd94a5e2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9440b5a90 0x7fd9440b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd94a5e2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd944148860 con 0x7fd92c004e50 2020-11-20T12:19:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd94a5e2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440b6800 con 0x7fd94414beb0 2020-11-20T12:19:28.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd94a5e2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440b6aa0 con 0x7fd9440b5a90 2020-11-20T12:19:28.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd948ddf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94414beb0 0x7fd944158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd948ddf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94414beb0 0x7fd944158600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52514/0 (socket says 172.21.15.187:52514) 2020-11-20T12:19:28.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.017+0000 7fd948ddf700 1 -- 172.21.15.187:0/1288737251 learned_addr learned my addr 172.21.15.187:0/1288737251 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:28.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd9495e0700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c004e50 0x7fd944135930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd949de1700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9440b5a90 0x7fd9440b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd948ddf700 1 -- 172.21.15.187:0/1288737251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9440b5a90 msgr2=0x7fd9440b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd948ddf700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9440b5a90 0x7fd9440b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd948ddf700 1 -- 172.21.15.187:0/1288737251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c004e50 msgr2=0x7fd944135930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd948ddf700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c004e50 0x7fd944135930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd948ddf700 1 -- 172.21.15.187:0/1288737251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd944135e20 con 0x7fd94414beb0 2020-11-20T12:19:28.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd948ddf700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94414beb0 0x7fd944158600 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 rx=0x7fd94000dad0 tx=0x7fd940009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd9317fa700 1 -- 172.21.15.187:0/1288737251 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940007170 con 0x7fd94414beb0 2020-11-20T12:19:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.018+0000 7fd9317fa700 1 -- 172.21.15.187:0/1288737251 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9400072d0 con 0x7fd94414beb0 2020-11-20T12:19:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.019+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd944158af0 con 0x7fd94414beb0 2020-11-20T12:19:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.019+0000 7fd9317fa700 1 -- 172.21.15.187:0/1288737251 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000ba80 con 0x7fd94414beb0 2020-11-20T12:19:28.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.019+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd944148cd0 con 0x7fd94414beb0 2020-11-20T12:19:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.020+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd944080050 con 0x7fd94414beb0 2020-11-20T12:19:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.020+0000 7fd9317fa700 1 -- 172.21.15.187:0/1288737251 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd940004500 con 0x7fd94414beb0 2020-11-20T12:19:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.021+0000 7fd9317fa700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.021+0000 7fd9317fa700 1 -- 172.21.15.187:0/1288737251 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd940070380 con 0x7fd94414beb0 2020-11-20T12:19:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.022+0000 7fd9495e0700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.022+0000 7fd9495e0700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7fd934006dc0 tx=0x7fd934008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.025+0000 7fd9317fa700 1 -- 172.21.15.187:0/1288737251 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd940045610 con 0x7fd94414beb0 2020-11-20T12:19:28.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.212+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9440b6600 con 0x7fd94414beb0 2020-11-20T12:19:28.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.213+0000 7fd9317fa700 1 -- 172.21.15.187:0/1288737251 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd9400011d0 con 0x7fd94414beb0 2020-11-20T12:19:28.216 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:19:28.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 msgr2=0x7fd91c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7fd934006dc0 tx=0x7fd934008040).stop 2020-11-20T12:19:28.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94414beb0 msgr2=0x7fd944158600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94414beb0 0x7fd944158600 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 rx=0x7fd94000dad0 tx=0x7fd940009040).stop 2020-11-20T12:19:28.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 shutdown_connections 2020-11-20T12:19:28.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd91c056d20 0x7fd91c0591e0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd92c004e50 0x7fd944135930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9440b5a90 0x7fd9440b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 --2- 172.21.15.187:0/1288737251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd94414beb0 0x7fd944158600 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 >> 172.21.15.187:0/1288737251 conn(0x7fd944068370 msgr2=0x7fd944141b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:28.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 shutdown_connections 2020-11-20T12:19:28.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.215+0000 7fd94a5e2700 1 -- 172.21.15.187:0/1288737251 wait complete. 2020-11-20T12:19:28.240 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:19:28.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.484+0000 7f4b42d33700 1 Processor -- start 2020-11-20T12:19:28.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.484+0000 7f4b42d33700 1 -- start start 2020-11-20T12:19:28.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.485+0000 7f4b42d33700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4b3c1484d0 0x7f4b3c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.485+0000 7f4b42d33700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4b3c0622c0 0x7f4b3c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.485+0000 7f4b42d33700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0b1300 con 0x7f4b3c066f70 2020-11-20T12:19:28.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.485+0000 7f4b42d33700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c067730 con 0x7f4b3c1484d0 2020-11-20T12:19:28.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.485+0000 7f4b42d33700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c0b15a0 con 0x7f4b3c0622c0 2020-11-20T12:19:28.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.486+0000 7f4b41530700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4b3c0622c0 0x7f4b3c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.486+0000 7f4b41530700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4b3c0622c0 0x7f4b3c0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52522/0 (socket says 172.21.15.187:52522) 2020-11-20T12:19:28.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.486+0000 7f4b41530700 1 -- 172.21.15.187:0/4220880774 learned_addr learned my addr 172.21.15.187:0/4220880774 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:28.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.486+0000 7f4b41d31700 1 --2- 172.21.15.187:0/4220880774 >> v2:172.21.15.5:3300/0 conn(0x7f4b3c1484d0 0x7f4b3c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.487+0000 7f4b41530700 1 -- 172.21.15.187:0/4220880774 >> v2:172.21.15.5:3300/0 conn(0x7f4b3c1484d0 msgr2=0x7f4b3c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:28.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.487+0000 7f4b41530700 1 --2- 172.21.15.187:0/4220880774 >> v2:172.21.15.5:3300/0 conn(0x7f4b3c1484d0 0x7f4b3c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.487+0000 7f4b41530700 1 -- 172.21.15.187:0/4220880774 >> v1:172.21.15.5:6789/0 conn(0x7f4b3c066f70 legacy=0x7f4b3c0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.487+0000 7f4b41530700 1 -- 172.21.15.187:0/4220880774 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0b1840 con 0x7f4b3c0622c0 2020-11-20T12:19:28.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.487+0000 7f4b41530700 1 --2- 172.21.15.187:0/4220880774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c0622c0 0x7f4b3c0b5640 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f4b3c068600 tx=0x7f4b38006eb0).ready entity=mon.1 client_cookie=a6978525b0f16569 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.487+0000 7f4b40d2f700 1 -- 172.21.15.187:0/4220880774 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b380098d0 con 0x7f4b3c0622c0 2020-11-20T12:19:28.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.487+0000 7f4b40d2f700 1 -- 172.21.15.187:0/4220880774 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b38009a30 con 0x7f4b3c0622c0 2020-11-20T12:19:28.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.487+0000 7f4b40d2f700 1 -- 172.21.15.187:0/4220880774 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38009db0 con 0x7f4b3c0622c0 2020-11-20T12:19:28.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.488+0000 7f4b42d33700 1 -- 172.21.15.187:0/4220880774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c0622c0 msgr2=0x7f4b3c0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.488+0000 7f4b42d33700 1 --2- 172.21.15.187:0/4220880774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c0622c0 0x7f4b3c0b5640 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f4b3c068600 tx=0x7f4b38006eb0).stop 2020-11-20T12:19:28.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.488+0000 7f4b42d33700 1 -- 172.21.15.187:0/4220880774 shutdown_connections 2020-11-20T12:19:28.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.488+0000 7f4b42d33700 1 --2- 172.21.15.187:0/4220880774 >> v2:172.21.15.5:3300/0 conn(0x7f4b3c1484d0 0x7f4b3c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.488+0000 7f4b42d33700 1 --2- 172.21.15.187:0/4220880774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c0622c0 0x7f4b3c0b5640 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.488+0000 7f4b42d33700 1 -- 172.21.15.187:0/4220880774 >> 172.21.15.187:0/4220880774 conn(0x7f4b3c0bab30 msgr2=0x7f4b3c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:28.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.488+0000 7f4b42d33700 1 -- 172.21.15.187:0/4220880774 shutdown_connections 2020-11-20T12:19:28.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.488+0000 7f4b42d33700 1 -- 172.21.15.187:0/4220880774 wait complete. 2020-11-20T12:19:28.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.490+0000 7f4b42d33700 1 Processor -- start 2020-11-20T12:19:28.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.490+0000 7f4b42d33700 1 -- start start 2020-11-20T12:19:28.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b42d33700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c066f70 0x7f4b3c09e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b42d33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b3c1484d0 0x7f4b3c160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b42d33700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b3c160500 0x7f4b3c09b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b42d33700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c09ef80 con 0x7f4b3c160500 2020-11-20T12:19:28.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b42d33700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c09ed30 con 0x7f4b3c066f70 2020-11-20T12:19:28.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b42d33700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c097340 con 0x7f4b3c1484d0 2020-11-20T12:19:28.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b41d31700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c066f70 0x7f4b3c09e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b41d31700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c066f70 0x7f4b3c09e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52524/0 (socket says 172.21.15.187:52524) 2020-11-20T12:19:28.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b41d31700 1 -- 172.21.15.187:0/3975016707 learned_addr learned my addr 172.21.15.187:0/3975016707 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:28.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b41d31700 1 -- 172.21.15.187:0/3975016707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b3c1484d0 msgr2=0x7f4b3c160010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b41d31700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b3c1484d0 0x7f4b3c160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:28.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b41d31700 1 -- 172.21.15.187:0/3975016707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b3c160500 msgr2=0x7f4b3c09b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b41d31700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b3c160500 0x7f4b3c09b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:28.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.491+0000 7f4b41d31700 1 -- 172.21.15.187:0/3975016707 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0975e0 con 0x7f4b3c066f70 2020-11-20T12:19:28.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.492+0000 7f4b41d31700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c066f70 0x7f4b3c09e840 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7f4b3000d910 tx=0x7f4b30008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.492+0000 7f4b2dffb700 1 -- 172.21.15.187:0/3975016707 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3000b130 con 0x7f4b3c066f70 2020-11-20T12:19:28.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.492+0000 7f4b2dffb700 1 -- 172.21.15.187:0/3975016707 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b30009790 con 0x7f4b3c066f70 2020-11-20T12:19:28.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.492+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c09f160 con 0x7f4b3c066f70 2020-11-20T12:19:28.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.492+0000 7f4b2dffb700 1 -- 172.21.15.187:0/3975016707 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30009b10 con 0x7f4b3c066f70 2020-11-20T12:19:28.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.492+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c160b20 con 0x7f4b3c066f70 2020-11-20T12:19:28.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.493+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c061a20 con 0x7f4b3c066f70 2020-11-20T12:19:28.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.494+0000 7f4b2dffb700 1 -- 172.21.15.187:0/3975016707 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4b3001c460 con 0x7f4b3c066f70 2020-11-20T12:19:28.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.494+0000 7f4b2dffb700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4b14056d20 0x7f4b140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.494+0000 7f4b2dffb700 1 -- 172.21.15.187:0/3975016707 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4b3006ff30 con 0x7f4b3c066f70 2020-11-20T12:19:28.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.495+0000 7f4b41530700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4b14056d20 0x7f4b140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.495+0000 7f4b41530700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4b14056d20 0x7f4b140591e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7f4b3c068600 tx=0x7f4b38015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.498+0000 7f4b2dffb700 1 -- 172.21.15.187:0/3975016707 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4b300452d0 con 0x7f4b3c066f70 2020-11-20T12:19:28.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.688+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4b3c1488b0 con 0x7f4b3c066f70 2020-11-20T12:19:28.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.689+0000 7f4b2dffb700 1 -- 172.21.15.187:0/3975016707 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4b300488f0 con 0x7f4b3c066f70 2020-11-20T12:19:28.693 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:19:28.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.691+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4b14056d20 msgr2=0x7f4b140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.691+0000 7f4b42d33700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4b14056d20 0x7f4b140591e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7f4b3c068600 tx=0x7f4b38015000).stop 2020-11-20T12:19:28.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.691+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c066f70 msgr2=0x7f4b3c09e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.691+0000 7f4b42d33700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c066f70 0x7f4b3c09e840 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7f4b3000d910 tx=0x7f4b30008040).stop 2020-11-20T12:19:28.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.692+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 shutdown_connections 2020-11-20T12:19:28.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.692+0000 7f4b42d33700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4b14056d20 0x7f4b140591e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.692+0000 7f4b42d33700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b3c160500 0x7f4b3c09b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:28.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.692+0000 7f4b42d33700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b3c066f70 0x7f4b3c09e840 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.692+0000 7f4b42d33700 1 --2- 172.21.15.187:0/3975016707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b3c1484d0 0x7f4b3c160010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:28.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.692+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 >> 172.21.15.187:0/3975016707 conn(0x7f4b3c0bab30 msgr2=0x7f4b3c13ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:28.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.692+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 shutdown_connections 2020-11-20T12:19:28.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.692+0000 7f4b42d33700 1 -- 172.21.15.187:0/3975016707 wait complete. 2020-11-20T12:19:28.718 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:28.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.958+0000 7ff41ac3f700 1 Processor -- start 2020-11-20T12:19:28.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.958+0000 7ff41ac3f700 1 -- start start 2020-11-20T12:19:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.959+0000 7ff41ac3f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff414065190 0x7ff414065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.959+0000 7ff41ac3f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4140b13e0 con 0x7ff41409ca90 2020-11-20T12:19:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.959+0000 7ff41ac3f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff414062c10 con 0x7ff414065f90 2020-11-20T12:19:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.959+0000 7ff41ac3f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41414ab30 con 0x7ff414065190 2020-11-20T12:19:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.960+0000 7ff41943c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff414065f90 0x7ff414143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49138/0 (socket says 172.21.15.187:49138) 2020-11-20T12:19:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.960+0000 7ff41943c700 1 -- 172.21.15.187:0/1529819470 learned_addr learned my addr 172.21.15.187:0/1529819470 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.960+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179312346 0 0) 0x7ff414062c10 con 0x7ff414065f90 2020-11-20T12:19:28.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.960+0000 7ff419c3d700 1 --2- 172.21.15.187:0/1529819470 >> v2:172.21.15.5:3300/0 conn(0x7ff414065190 0x7ff414065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.960+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc003240 con 0x7ff414065f90 2020-11-20T12:19:28.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2459610618 0 0) 0x7ff3fc003240 con 0x7ff414065f90 2020-11-20T12:19:28.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 >> v2:172.21.15.5:3300/0 conn(0x7ff414065190 msgr2=0x7ff414065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:28.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 --2- 172.21.15.187:0/1529819470 >> v2:172.21.15.5:3300/0 conn(0x7ff414065190 0x7ff414065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 >> v1:172.21.15.5:6789/0 conn(0x7ff41409ca90 legacy=0x7ff41409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:28.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4140b1640 con 0x7ff414065f90 2020-11-20T12:19:28.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff4040014e0 con 0x7ff414065f90 2020-11-20T12:19:28.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 >> v1:172.21.15.187:6789/0 conn(0x7ff414065f90 legacy=0x7ff414143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3fc0037b0 0x7ff3fc003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3fc0044e0 0x7ff3fc004940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.961+0000 7ff418c3b700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3fc004e30 0x7ff3fc00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41414ab30 con 0x7ff3fc004e30 2020-11-20T12:19:28.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc0052e0 con 0x7ff3fc0044e0 2020-11-20T12:19:28.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc00bc20 con 0x7ff3fc0037b0 2020-11-20T12:19:28.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff41943c700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3fc0044e0 0x7ff3fc004940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff41943c700 1 -- 172.21.15.187:0/1529819470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3fc0037b0 msgr2=0x7ff3fc003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:28.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff41a43e700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3fc004e30 0x7ff3fc00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff41943c700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3fc0037b0 0x7ff3fc003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:28.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff41943c700 1 -- 172.21.15.187:0/1529819470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3fc004e30 msgr2=0x7ff3fc00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:28.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.962+0000 7ff41943c700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3fc004e30 0x7ff3fc00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.963+0000 7ff41943c700 1 -- 172.21.15.187:0/1529819470 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc003240 con 0x7ff3fc0044e0 2020-11-20T12:19:28.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.963+0000 7ff41a43e700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3fc004e30 0x7ff3fc00b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:28.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.963+0000 7ff41a43e700 1 -- 172.21.15.187:0/1529819470 reap_dead start 2020-11-20T12:19:28.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.963+0000 7ff41943c700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3fc0044e0 0x7ff3fc004940 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 rx=0x7ff40400c4c0 tx=0x7ff4040079a0).ready entity=mon.1 client_cookie=b5f6e0666e3418f8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.963+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040072e0 con 0x7ff3fc0044e0 2020-11-20T12:19:28.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.963+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff404007440 con 0x7ff3fc0044e0 2020-11-20T12:19:28.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.963+0000 7ff418c3b700 1 -- 172.21.15.187:0/1529819470 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404005820 con 0x7ff3fc0044e0 2020-11-20T12:19:28.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.964+0000 7ff41ac3f700 1 -- 172.21.15.187:0/1529819470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3fc0044e0 msgr2=0x7ff3fc004940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.964+0000 7ff41ac3f700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3fc0044e0 0x7ff3fc004940 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 rx=0x7ff40400c4c0 tx=0x7ff4040079a0).stop 2020-11-20T12:19:28.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.964+0000 7ff41ac3f700 1 -- 172.21.15.187:0/1529819470 shutdown_connections 2020-11-20T12:19:28.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.964+0000 7ff41ac3f700 1 --2- 172.21.15.187:0/1529819470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3fc0044e0 0x7ff3fc004940 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.964+0000 7ff41ac3f700 1 -- 172.21.15.187:0/1529819470 >> 172.21.15.187:0/1529819470 conn(0x7ff4140bab30 msgr2=0x7ff4140baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:28.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.964+0000 7ff41ac3f700 1 -- 172.21.15.187:0/1529819470 shutdown_connections 2020-11-20T12:19:28.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.964+0000 7ff41ac3f700 1 -- 172.21.15.187:0/1529819470 wait complete. 2020-11-20T12:19:28.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.966+0000 7ff41ac3f700 1 Processor -- start 2020-11-20T12:19:28.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.966+0000 7ff41ac3f700 1 -- start start 2020-11-20T12:19:28.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41ac3f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3fc0044e0 0x7ff414148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41ac3f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4140b3500 0x7ff4140b7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41ac3f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff41409dae0 0x7ff41409df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41ac3f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff414159f70 con 0x7ff4140b3500 2020-11-20T12:19:28.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41ac3f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41415a200 con 0x7ff41409dae0 2020-11-20T12:19:28.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41ac3f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff414147780 con 0x7ff3fc0044e0 2020-11-20T12:19:28.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41a43e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff41409dae0 0x7ff41409df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41a43e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff41409dae0 0x7ff41409df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52548/0 (socket says 172.21.15.187:52548) 2020-11-20T12:19:28.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.967+0000 7ff41a43e700 1 -- 172.21.15.187:0/564292137 learned_addr learned my addr 172.21.15.187:0/564292137 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:28.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.968+0000 7ff419c3d700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3fc0044e0 0x7ff414148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.968+0000 7ff41943c700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4140b3500 0x7ff4140b7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.968+0000 7ff41a43e700 1 -- 172.21.15.187:0/564292137 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3fc0044e0 msgr2=0x7ff414148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.968+0000 7ff41a43e700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3fc0044e0 0x7ff414148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.968+0000 7ff41a43e700 1 -- 172.21.15.187:0/564292137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4140b3500 msgr2=0x7ff4140b7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:28.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.968+0000 7ff41a43e700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4140b3500 0x7ff4140b7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:28.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.968+0000 7ff41a43e700 1 -- 172.21.15.187:0/564292137 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4140b7c70 con 0x7ff41409dae0 2020-11-20T12:19:28.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.968+0000 7ff41a43e700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff41409dae0 0x7ff41409df40 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 rx=0x7ff41409ce70 tx=0x7ff40c009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.969+0000 7ff401ffb700 1 -- 172.21.15.187:0/564292137 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c00de40 con 0x7ff41409dae0 2020-11-20T12:19:28.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.969+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41413cfa0 con 0x7ff41409dae0 2020-11-20T12:19:28.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.969+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41413d440 con 0x7ff41409dae0 2020-11-20T12:19:28.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.969+0000 7ff401ffb700 1 -- 172.21.15.187:0/564292137 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c007020 con 0x7ff41409dae0 2020-11-20T12:19:28.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.971+0000 7ff401ffb700 1 -- 172.21.15.187:0/564292137 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c007190 con 0x7ff41409dae0 2020-11-20T12:19:28.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.971+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41415a3c0 con 0x7ff41409dae0 2020-11-20T12:19:28.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.971+0000 7ff401ffb700 1 -- 172.21.15.187:0/564292137 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff40c010900 con 0x7ff41409dae0 2020-11-20T12:19:28.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.972+0000 7ff401ffb700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3ec056d20 0x7ff3ec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:28.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.972+0000 7ff401ffb700 1 -- 172.21.15.187:0/564292137 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff40c06f4d0 con 0x7ff41409dae0 2020-11-20T12:19:28.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.973+0000 7ff419c3d700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3ec056d20 0x7ff3ec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:28.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.973+0000 7ff419c3d700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3ec056d20 0x7ff3ec0591e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7ff410006dd0 tx=0x7ff410008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:28.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:28.976+0000 7ff401ffb700 1 -- 172.21.15.187:0/564292137 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff40c043fd0 con 0x7ff41409dae0 2020-11-20T12:19:29.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.225+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff414061a20 con 0x7ff41409dae0 2020-11-20T12:19:29.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.226+0000 7ff401ffb700 1 -- 172.21.15.187:0/564292137 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+2146 (secure 0 0 0) 0x7ff40c044920 con 0x7ff41409dae0 2020-11-20T12:19:29.230 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:29.230 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":131,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3102570506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3102570506},{"type":"v1","addr":"172.21.15.5:6833","nonce":3102570506}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:18:10.557467+0000","modified":"2020-11-20T12:19:19.889320+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8932},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8932":{"gid":8932,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3451092437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3451092437},{"type":"v1","addr":"172.21.15.187:6835","nonce":3451092437}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2020-11-20T12:19:29.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3ec056d20 msgr2=0x7ff3ec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3ec056d20 0x7ff3ec0591e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7ff410006dd0 tx=0x7ff410008040).stop 2020-11-20T12:19:29.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff41409dae0 msgr2=0x7ff41409df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff41409dae0 0x7ff41409df40 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 rx=0x7ff41409ce70 tx=0x7ff40c009040).stop 2020-11-20T12:19:29.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 shutdown_connections 2020-11-20T12:19:29.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff3ec056d20 0x7ff3ec0591e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff41409dae0 0x7ff41409df40 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3fc0044e0 0x7ff414148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 --2- 172.21.15.187:0/564292137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4140b3500 0x7ff4140b7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.228+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 >> 172.21.15.187:0/564292137 conn(0x7ff4140bab30 msgr2=0x7ff4140b9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:29.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.229+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 shutdown_connections 2020-11-20T12:19:29.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.229+0000 7ff41ac3f700 1 -- 172.21.15.187:0/564292137 wait complete. 2020-11-20T12:19:29.235 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 131 2020-11-20T12:19:29.254 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:19:29.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.493+0000 7f97f506a700 1 Processor -- start 2020-11-20T12:19:29.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.493+0000 7f97f506a700 1 -- start start 2020-11-20T12:19:29.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.494+0000 7f97f506a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065190 0x7f97f0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.494+0000 7f97f506a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97f0065f90 0x7f97f0143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.494+0000 7f97f506a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f00b13e0 con 0x7f97f0143c70 2020-11-20T12:19:29.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.494+0000 7f97f506a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f014ab30 con 0x7f97f0065f90 2020-11-20T12:19:29.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.494+0000 7f97f506a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f0062c10 con 0x7f97f0065190 2020-11-20T12:19:29.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.495+0000 7f97e77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97f0065f90 0x7f97f0143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.495+0000 7f97e77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97f0065f90 0x7f97f0143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52554/0 (socket says 172.21.15.187:52554) 2020-11-20T12:19:29.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.495+0000 7f97e77fe700 1 -- 172.21.15.187:0/4294723940 learned_addr learned my addr 172.21.15.187:0/4294723940 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:29.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.495+0000 7f97effff700 1 --2- 172.21.15.187:0/4294723940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065190 0x7f97f0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.496+0000 7f97e77fe700 1 -- 172.21.15.187:0/4294723940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065190 msgr2=0x7f97f0065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:29.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.496+0000 7f97e77fe700 1 --2- 172.21.15.187:0/4294723940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065190 0x7f97f0065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.496+0000 7f97e77fe700 1 -- 172.21.15.187:0/4294723940 >> v1:172.21.15.5:6789/0 conn(0x7f97f0143c70 legacy=0x7f97f009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:29.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.496+0000 7f97e77fe700 1 -- 172.21.15.187:0/4294723940 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f00b1640 con 0x7f97f0065f90 2020-11-20T12:19:29.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.496+0000 7f97e77fe700 1 --2- 172.21.15.187:0/4294723940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0065f90 0x7f97f0143780 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 rx=0x7f97f014ac70 tx=0x7f97d8006eb0).ready entity=mon.1 client_cookie=2ae1a5005ca9a2ee server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:29.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.496+0000 7f97ef7fe700 1 -- 172.21.15.187:0/4294723940 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8009920 con 0x7f97f0065f90 2020-11-20T12:19:29.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.496+0000 7f97ef7fe700 1 -- 172.21.15.187:0/4294723940 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d8009a80 con 0x7f97f0065f90 2020-11-20T12:19:29.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.496+0000 7f97ef7fe700 1 -- 172.21.15.187:0/4294723940 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8009e00 con 0x7f97f0065f90 2020-11-20T12:19:29.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.497+0000 7f97f506a700 1 -- 172.21.15.187:0/4294723940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0065f90 msgr2=0x7f97f0143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.497+0000 7f97f506a700 1 --2- 172.21.15.187:0/4294723940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0065f90 0x7f97f0143780 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 rx=0x7f97f014ac70 tx=0x7f97d8006eb0).stop 2020-11-20T12:19:29.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.497+0000 7f97f506a700 1 -- 172.21.15.187:0/4294723940 shutdown_connections 2020-11-20T12:19:29.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.497+0000 7f97f506a700 1 --2- 172.21.15.187:0/4294723940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065190 0x7f97f0065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.497+0000 7f97f506a700 1 --2- 172.21.15.187:0/4294723940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0065f90 0x7f97f0143780 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.497+0000 7f97f506a700 1 -- 172.21.15.187:0/4294723940 >> 172.21.15.187:0/4294723940 conn(0x7f97f00bab30 msgr2=0x7f97f00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:29.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.497+0000 7f97f506a700 1 -- 172.21.15.187:0/4294723940 shutdown_connections 2020-11-20T12:19:29.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.497+0000 7f97f506a700 1 -- 172.21.15.187:0/4294723940 wait complete. 2020-11-20T12:19:29.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.500+0000 7f97f506a700 1 Processor -- start 2020-11-20T12:19:29.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.501+0000 7f97f506a700 1 -- start start 2020-11-20T12:19:29.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.502+0000 7f97f506a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f0065190 0x7f97f01590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.502+0000 7f97f506a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065f90 0x7f97f00b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.502+0000 7f97f506a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0143c70 0x7f97f00990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.502+0000 7f97f506a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f0142b60 con 0x7f97f0065190 2020-11-20T12:19:29.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.502+0000 7f97f506a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f0148c10 con 0x7f97f0143c70 2020-11-20T12:19:29.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.502+0000 7f97f506a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f0148eb0 con 0x7f97f0065f90 2020-11-20T12:19:29.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0143c70 0x7f97f00990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0143c70 0x7f97f00990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52562/0 (socket says 172.21.15.187:52562) 2020-11-20T12:19:29.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 -- 172.21.15.187:0/3006479830 learned_addr learned my addr 172.21.15.187:0/3006479830 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:29.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97effff700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f0065190 0x7f97f01590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97e77fe700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065f90 0x7f97f00b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 -- 172.21.15.187:0/3006479830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065f90 msgr2=0x7f97f00b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065f90 0x7f97f00b6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 -- 172.21.15.187:0/3006479830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f0065190 msgr2=0x7f97f01590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f0065190 0x7f97f01590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 -- 172.21.15.187:0/3006479830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f0149150 con 0x7f97f0143c70 2020-11-20T12:19:29.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.503+0000 7f97f4869700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0143c70 0x7f97f00990a0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 rx=0x7f97e8009690 tx=0x7f97e8006cf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:29.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.504+0000 7f97ecff9700 1 -- 172.21.15.187:0/3006479830 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e800d560 con 0x7f97f0143c70 2020-11-20T12:19:29.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.504+0000 7f97ecff9700 1 -- 172.21.15.187:0/3006479830 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e8000e90 con 0x7f97f0143c70 2020-11-20T12:19:29.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.504+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f01584e0 con 0x7f97f0143c70 2020-11-20T12:19:29.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.504+0000 7f97ecff9700 1 -- 172.21.15.187:0/3006479830 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e800d930 con 0x7f97f0143c70 2020-11-20T12:19:29.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.504+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f00b6dc0 con 0x7f97f0143c70 2020-11-20T12:19:29.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.505+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f0149510 con 0x7f97f0143c70 2020-11-20T12:19:29.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.506+0000 7f97ecff9700 1 -- 172.21.15.187:0/3006479830 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f97e800f030 con 0x7f97f0143c70 2020-11-20T12:19:29.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.506+0000 7f97ecff9700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f97cc056d20 0x7f97cc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.507+0000 7f97ecff9700 1 -- 172.21.15.187:0/3006479830 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f97e806fb90 con 0x7f97f0143c70 2020-11-20T12:19:29.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.507+0000 7f97effff700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f97cc056d20 0x7f97cc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.507+0000 7f97effff700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f97cc056d20 0x7f97cc0591e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f97e0006fd0 tx=0x7f97e0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:29.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.510+0000 7f97ecff9700 1 -- 172.21.15.187:0/3006479830 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f97e8044f90 con 0x7f97f0143c70 2020-11-20T12:19:29.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.697+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97f01492e0 con 0x7f97f0143c70 2020-11-20T12:19:29.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.699+0000 7f97ecff9700 1 -- 172.21.15.187:0/3006479830 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16919 (secure 0 0 0) 0x7f97e8006e10 con 0x7f97f0143c70 2020-11-20T12:19:29.703 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:29.703 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":131,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:18:28.833768+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:18:08.692656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2020-11-20T12:18:09.325174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:6833/3549193191":"2020-11-21T12:18:28.833712+0000","172.21.15.5:6832/3549193191":"2020-11-21T12:18:28.833712+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:19:29.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.701+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f97cc056d20 msgr2=0x7f97cc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.701+0000 7f97f506a700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f97cc056d20 0x7f97cc0591e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f97e0006fd0 tx=0x7f97e0008040).stop 2020-11-20T12:19:29.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.701+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0143c70 msgr2=0x7f97f00990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.701+0000 7f97f506a700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0143c70 0x7f97f00990a0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 rx=0x7f97e8009690 tx=0x7f97e8006cf0).stop 2020-11-20T12:19:29.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.702+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 shutdown_connections 2020-11-20T12:19:29.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.702+0000 7f97f506a700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f97cc056d20 0x7f97cc0591e0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.702+0000 7f97f506a700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f0065190 0x7f97f01590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.702+0000 7f97f506a700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97f0143c70 0x7f97f00990a0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.702+0000 7f97f506a700 1 --2- 172.21.15.187:0/3006479830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97f0065f90 0x7f97f00b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.702+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 >> 172.21.15.187:0/3006479830 conn(0x7f97f00bab30 msgr2=0x7f97f009cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:29.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.702+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 shutdown_connections 2020-11-20T12:19:29.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.702+0000 7f97f506a700 1 -- 172.21.15.187:0/3006479830 wait complete. 2020-11-20T12:19:29.728 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:19:29.729 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:19:29.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.966+0000 7faaa8ad2700 1 Processor -- start 2020-11-20T12:19:29.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.966+0000 7faaa8ad2700 1 -- start start 2020-11-20T12:19:29.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.967+0000 7faaa8ad2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa41482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.967+0000 7faaa8ad2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faaa40b5f50 0x7faaa40b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.967+0000 7faaa8ad2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa4144b80 con 0x7faaa40b57b0 2020-11-20T12:19:29.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.967+0000 7faaa8ad2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa41487c0 con 0x7faaa40b5f50 2020-11-20T12:19:29.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.967+0000 7faaa8ad2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa41489e0 con 0x7faaa4068230 2020-11-20T12:19:29.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.968+0000 7faaa37fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7faaa40b57b0 0x7faaa4062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49172/0 (socket says 172.21.15.187:49172) 2020-11-20T12:19:29.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.968+0000 7faaa37fe700 1 -- 172.21.15.187:0/1343310097 learned_addr learned my addr 172.21.15.187:0/1343310097 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:29.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.968+0000 7faaa3fff700 1 --2- 172.21.15.187:0/1343310097 >> v2:172.21.15.187:3300/0 conn(0x7faaa40b5f50 0x7faaa40b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.968+0000 7faaa2ffd700 1 --2- 172.21.15.187:0/1343310097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa41482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.969+0000 7faaa3fff700 1 -- 172.21.15.187:0/1343310097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 msgr2=0x7faaa41482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:29.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.969+0000 7faaa3fff700 1 --2- 172.21.15.187:0/1343310097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa41482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.969+0000 7faaa3fff700 1 -- 172.21.15.187:0/1343310097 >> v1:172.21.15.187:6789/0 conn(0x7faaa40b57b0 legacy=0x7faaa4062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.969+0000 7faaa3fff700 1 -- 172.21.15.187:0/1343310097 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa4148c80 con 0x7faaa40b5f50 2020-11-20T12:19:29.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.969+0000 7faaa3fff700 1 --2- 172.21.15.187:0/1343310097 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b5f50 0x7faaa40b63e0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7faa9c009c90 tx=0x7faa9c0091d0).ready entity=mon.1 client_cookie=84420879856ef2d4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:29.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.971+0000 7faaa27fc700 1 -- 172.21.15.187:0/1343310097 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9c00bbc0 con 0x7faaa40b5f50 2020-11-20T12:19:29.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.972+0000 7faaa27fc700 1 -- 172.21.15.187:0/1343310097 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9c00bd20 con 0x7faaa40b5f50 2020-11-20T12:19:29.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.972+0000 7faaa27fc700 1 -- 172.21.15.187:0/1343310097 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9c011610 con 0x7faaa40b5f50 2020-11-20T12:19:29.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.972+0000 7faaa8ad2700 1 -- 172.21.15.187:0/1343310097 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b5f50 msgr2=0x7faaa40b63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.972+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/1343310097 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b5f50 0x7faaa40b63e0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7faa9c009c90 tx=0x7faa9c0091d0).stop 2020-11-20T12:19:29.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.973+0000 7faaa8ad2700 1 -- 172.21.15.187:0/1343310097 shutdown_connections 2020-11-20T12:19:29.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.973+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/1343310097 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b5f50 0x7faaa40b63e0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.973+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/1343310097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa41482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.973+0000 7faaa8ad2700 1 -- 172.21.15.187:0/1343310097 >> 172.21.15.187:0/1343310097 conn(0x7faaa40bab30 msgr2=0x7faaa40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:29.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.973+0000 7faaa8ad2700 1 -- 172.21.15.187:0/1343310097 shutdown_connections 2020-11-20T12:19:29.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.973+0000 7faaa8ad2700 1 -- 172.21.15.187:0/1343310097 wait complete. 2020-11-20T12:19:29.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.974+0000 7faaa8ad2700 1 Processor -- start 2020-11-20T12:19:29.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.975+0000 7faaa8ad2700 1 -- start start 2020-11-20T12:19:29.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.975+0000 7faaa8ad2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa4090530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.975+0000 7faaa8ad2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b57b0 0x7faaa408ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.975+0000 7faaa8ad2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa40b5f50 0x7faaa41594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.975+0000 7faaa8ad2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa4147db0 con 0x7faaa40b5f50 2020-11-20T12:19:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.975+0000 7faaa8ad2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa4148050 con 0x7faaa40b57b0 2020-11-20T12:19:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.975+0000 7faaa8ad2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa408f0d0 con 0x7faaa4068230 2020-11-20T12:19:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b57b0 0x7faaa408ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b57b0 0x7faaa408ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52574/0 (socket says 172.21.15.187:52574) 2020-11-20T12:19:29.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa2ffd700 1 -- 172.21.15.187:0/2448809319 learned_addr learned my addr 172.21.15.187:0/2448809319 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:29.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa37fe700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa4090530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa3fff700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa40b5f50 0x7faaa41594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa2ffd700 1 -- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 msgr2=0x7faaa4090530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa2ffd700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa4090530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa2ffd700 1 -- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa40b5f50 msgr2=0x7faaa41594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:29.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa2ffd700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa40b5f50 0x7faaa41594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:29.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa2ffd700 1 -- 172.21.15.187:0/2448809319 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa408f410 con 0x7faaa40b57b0 2020-11-20T12:19:29.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.976+0000 7faaa37fe700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa4090530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:29.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.977+0000 7faaa2ffd700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b57b0 0x7faaa408ebe0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 rx=0x7faa9400d8e0 tx=0x7faa94008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:29.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.977+0000 7faa9b7fe700 1 -- 172.21.15.187:0/2448809319 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94009790 con 0x7faaa40b57b0 2020-11-20T12:19:29.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.977+0000 7faa9b7fe700 1 -- 172.21.15.187:0/2448809319 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa940098f0 con 0x7faaa40b57b0 2020-11-20T12:19:29.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.977+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa408f6c0 con 0x7faaa40b57b0 2020-11-20T12:19:29.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.977+0000 7faa9b7fe700 1 -- 172.21.15.187:0/2448809319 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94009c90 con 0x7faaa40b57b0 2020-11-20T12:19:29.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.977+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa4159bf0 con 0x7faaa40b57b0 2020-11-20T12:19:29.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.978+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa4148e60 con 0x7faaa40b57b0 2020-11-20T12:19:29.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.979+0000 7faa9b7fe700 1 -- 172.21.15.187:0/2448809319 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faa94013480 con 0x7faaa40b57b0 2020-11-20T12:19:29.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.979+0000 7faa9b7fe700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa80056d20 0x7faa800591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:29.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.979+0000 7faa9b7fe700 1 -- 172.21.15.187:0/2448809319 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7faa9406ff00 con 0x7faaa40b57b0 2020-11-20T12:19:29.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.980+0000 7faaa37fe700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa80056d20 0x7faa800591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:29.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.980+0000 7faaa37fe700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa80056d20 0x7faa800591e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7faa8c0014b0 tx=0x7faa8c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:29.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:29.983+0000 7faa9b7fe700 1 -- 172.21.15.187:0/2448809319 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faa940452a0 con 0x7faaa40b57b0 2020-11-20T12:19:30.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.229+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7faaa4061a20 con 0x7faaa40b57b0 2020-11-20T12:19:30.974 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:19:30.970+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:19:30.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.986+0000 7faa9b7fe700 1 -- 172.21.15.187:0/2448809319 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v132) v1 ==== 161+0+0 (secure 0 0 0) 0x7faa940488c0 con 0x7faaa40b57b0 2020-11-20T12:19:30.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa80056d20 msgr2=0x7faa800591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:30.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa80056d20 0x7faa800591e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7faa8c0014b0 tx=0x7faa8c009040).stop 2020-11-20T12:19:30.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b57b0 msgr2=0x7faaa408ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:30.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b57b0 0x7faaa408ebe0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 rx=0x7faa9400d8e0 tx=0x7faa94008040).stop 2020-11-20T12:19:30.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 shutdown_connections 2020-11-20T12:19:30.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faa80056d20 0x7faa800591e0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:30.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa40b5f50 0x7faaa41594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:30.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa4068230 0x7faaa4090530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:30.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 --2- 172.21.15.187:0/2448809319 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faaa40b57b0 0x7faaa408ebe0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:30.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 >> 172.21.15.187:0/2448809319 conn(0x7faaa40bab30 msgr2=0x7faaa40afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:30.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.991+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 shutdown_connections 2020-11-20T12:19:30.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:30.992+0000 7faaa8ad2700 1 -- 172.21.15.187:0/2448809319 wait complete. 2020-11-20T12:19:30.998 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:19:31.011 INFO:tasks.ceph.mds.a.smithi187.stderr: -5598> 2020-11-20T12:18:14.764+0000 7f357cee0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:19:31.012 INFO:tasks.ceph.mds.a.smithi187.stderr: -5130> 2020-11-20T12:18:24.765+0000 7f357cee0700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2020-11-20T12:19:31.024 INFO:tasks.ceph.mds.a.smithi187.stderr: -2766> 2020-11-20T12:18:39.765+0000 7f357cee0700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2020-11-20T12:19:31.031 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:19:31.075 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:19:31.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.253+0000 7f20c9d8d700 1 Processor -- start 2020-11-20T12:19:31.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.253+0000 7f20c9d8d700 1 -- start start 2020-11-20T12:19:31.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.255+0000 7f20c9d8d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20c4065190 0x7f20c4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:31.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.255+0000 7f20c9d8d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c40b13e0 con 0x7f20c4065f90 2020-11-20T12:19:31.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.255+0000 7f20c9d8d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c4062c10 con 0x7f20c409ca90 2020-11-20T12:19:31.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.255+0000 7f20c9d8d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c414ab30 con 0x7f20c4065190 2020-11-20T12:19:31.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.255+0000 7f20c958c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f20c409ca90 0x7f20c409cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49208/0 (socket says 172.21.15.187:49208) 2020-11-20T12:19:31.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.255+0000 7f20c958c700 1 -- 172.21.15.187:0/850942397 learned_addr learned my addr 172.21.15.187:0/850942397 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:31.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/850942397 >> v2:172.21.15.5:3300/0 conn(0x7f20c4065190 0x7f20c4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:31.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365977455 0 0) 0x7f20c4062c10 con 0x7f20c409ca90 2020-11-20T12:19:31.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a8003240 con 0x7f20c409ca90 2020-11-20T12:19:31.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3286526249 0 0) 0x7f20a8003240 con 0x7f20c409ca90 2020-11-20T12:19:31.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 >> v2:172.21.15.5:3300/0 conn(0x7f20c4065190 msgr2=0x7f20c4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:31.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 --2- 172.21.15.187:0/850942397 >> v2:172.21.15.5:3300/0 conn(0x7f20c4065190 0x7f20c4065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:31.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 >> v1:172.21.15.5:6789/0 conn(0x7f20c4065f90 legacy=0x7f20c4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:31.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40b1640 con 0x7f20c409ca90 2020-11-20T12:19:31.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f20bc001860 con 0x7f20c409ca90 2020-11-20T12:19:31.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/850942397 >> v2:172.21.15.5:3300/0 conn(0x7f20c4065190 0x7f20c4065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:19:31.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 >> v1:172.21.15.187:6789/0 conn(0x7f20c409ca90 legacy=0x7f20c409cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:31.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a8003730 0x7f20a8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:31.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a80044e0 0x7f20a8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:31.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a8004e50 0x7f20a800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:31.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c414ab30 con 0x7f20a8004e50 2020-11-20T12:19:31.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a8004280 con 0x7f20a8003730 2020-11-20T12:19:31.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.256+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a800bc20 con 0x7f20a80044e0 2020-11-20T12:19:31.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a8003730 0x7f20a8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:31.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c958c700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a8004e50 0x7f20a800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:31.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c3fff700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a80044e0 0x7f20a8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:31.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c8d8b700 1 -- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a80044e0 msgr2=0x7f20a8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:31.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a80044e0 0x7f20a8004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:31.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c8d8b700 1 -- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a8004e50 msgr2=0x7f20a800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:31.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a8004e50 0x7f20a800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:31.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c8d8b700 1 -- 172.21.15.187:0/850942397 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8003240 con 0x7f20a8003730 2020-11-20T12:19:31.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c958c700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20a8004e50 0x7f20a800b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:31.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c958c700 1 -- 172.21.15.187:0/850942397 reap_dead start 2020-11-20T12:19:31.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.257+0000 7f20c3fff700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a80044e0 0x7f20a8004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:31.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a8003730 0x7f20a8003b70 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 rx=0x7f20b800bb40 tx=0x7f20b8008040).ready entity=mon.1 client_cookie=df9f52432b29c7e2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:31.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b800da20 con 0x7f20a8003730 2020-11-20T12:19:31.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b800db80 con 0x7f20a8003730 2020-11-20T12:19:31.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c37fe700 1 -- 172.21.15.187:0/850942397 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b8012420 con 0x7f20a8003730 2020-11-20T12:19:31.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c9d8d700 1 -- 172.21.15.187:0/850942397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a8003730 msgr2=0x7f20a8003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:31.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c9d8d700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a8003730 0x7f20a8003b70 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 rx=0x7f20b800bb40 tx=0x7f20b8008040).stop 2020-11-20T12:19:31.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c9d8d700 1 -- 172.21.15.187:0/850942397 shutdown_connections 2020-11-20T12:19:31.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c9d8d700 1 --2- 172.21.15.187:0/850942397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20a8003730 0x7f20a8003b70 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:31.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.258+0000 7f20c9d8d700 1 -- 172.21.15.187:0/850942397 >> 172.21.15.187:0/850942397 conn(0x7f20c40bab30 msgr2=0x7f20c40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:31.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.259+0000 7f20c9d8d700 1 -- 172.21.15.187:0/850942397 shutdown_connections 2020-11-20T12:19:31.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.259+0000 7f20c9d8d700 1 -- 172.21.15.187:0/850942397 wait complete. 2020-11-20T12:19:31.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.260+0000 7f20c9d8d700 1 Processor -- start 2020-11-20T12:19:31.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.260+0000 7f20c9d8d700 1 -- start start 2020-11-20T12:19:31.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c9d8d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a8003730 0x7f20c40b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:31.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c9d8d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20c409dae0 0x7f20c4148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:31.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c9d8d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20c40b3140 0x7f20c41410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:31.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c9d8d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b20a0 con 0x7f20c409dae0 2020-11-20T12:19:31.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c9d8d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b22c0 con 0x7f20c40b3140 2020-11-20T12:19:31.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c9d8d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b2560 con 0x7f20a8003730 2020-11-20T12:19:31.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c958c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20c40b3140 0x7f20c41410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:31.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c958c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20c40b3140 0x7f20c41410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52614/0 (socket says 172.21.15.187:52614) 2020-11-20T12:19:31.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.261+0000 7f20c958c700 1 -- 172.21.15.187:0/2789567425 learned_addr learned my addr 172.21.15.187:0/2789567425 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:31.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a8003730 0x7f20c40b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:31.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c3fff700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20c409dae0 0x7f20c4148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:31.281 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c958c700 1 -- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a8003730 msgr2=0x7f20c40b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:31.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c958c700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a8003730 0x7f20c40b1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:31.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c958c700 1 -- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20c409dae0 msgr2=0x7f20c4148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:31.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c958c700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20c409dae0 0x7f20c4148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:31.282 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c958c700 1 -- 172.21.15.187:0/2789567425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40b2800 con 0x7f20c40b3140 2020-11-20T12:19:31.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c3fff700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20c409dae0 0x7f20c4148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:31.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c958c700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20c40b3140 0x7f20c41410e0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 rx=0x7f20c409ce70 tx=0x7f20bc009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:31.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c0ff9700 1 -- 172.21.15.187:0/2789567425 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc00bb30 con 0x7f20c40b3140 2020-11-20T12:19:31.283 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c0ff9700 1 -- 172.21.15.187:0/2789567425 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc00bc90 con 0x7f20c40b3140 2020-11-20T12:19:31.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.262+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c41492a0 con 0x7f20c40b3140 2020-11-20T12:19:31.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.263+0000 7f20c0ff9700 1 -- 172.21.15.187:0/2789567425 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc004880 con 0x7f20c40b3140 2020-11-20T12:19:31.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.263+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c4141810 con 0x7f20c40b3140 2020-11-20T12:19:31.284 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.264+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c4061a20 con 0x7f20c40b3140 2020-11-20T12:19:31.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.264+0000 7f20c0ff9700 1 -- 172.21.15.187:0/2789567425 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f20bc01e070 con 0x7f20c40b3140 2020-11-20T12:19:31.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.265+0000 7f20c0ff9700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f209c056d20 0x7f209c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:31.285 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.265+0000 7f20c0ff9700 1 -- 172.21.15.187:0/2789567425 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f20bc06f740 con 0x7f20c40b3140 2020-11-20T12:19:31.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.265+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f209c056d20 0x7f209c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:31.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.266+0000 7f20c8d8b700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f209c056d20 0x7f209c0591e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f20b800b210 tx=0x7f20b8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:31.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.269+0000 7f20c0ff9700 1 -- 172.21.15.187:0/2789567425 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f20bc0449d0 con 0x7f20c40b3140 2020-11-20T12:19:31.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.517+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f20c40afc60 con 0x7f20c40b3140 2020-11-20T12:19:31.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.991+0000 7f20c0ff9700 1 -- 172.21.15.187:0/2789567425 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v133) v1 ==== 104+0+0 (secure 0 0 0) 0x7f20bc01c0a0 con 0x7f20c40b3140 2020-11-20T12:19:32.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.994+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f209c056d20 msgr2=0x7f209c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:32.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.994+0000 7f20c9d8d700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f209c056d20 0x7f209c0591e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f20b800b210 tx=0x7f20b8008040).stop 2020-11-20T12:19:32.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.994+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20c40b3140 msgr2=0x7f20c41410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:32.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.994+0000 7f20c9d8d700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20c40b3140 0x7f20c41410e0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 rx=0x7f20c409ce70 tx=0x7f20bc009040).stop 2020-11-20T12:19:32.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.995+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 shutdown_connections 2020-11-20T12:19:32.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.995+0000 7f20c9d8d700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f209c056d20 0x7f209c0591e0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.995+0000 7f20c9d8d700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20c40b3140 0x7f20c41410e0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.995+0000 7f20c9d8d700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20a8003730 0x7f20c40b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.995+0000 7f20c9d8d700 1 --2- 172.21.15.187:0/2789567425 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20c409dae0 0x7f20c4148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.995+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 >> 172.21.15.187:0/2789567425 conn(0x7f20c40bab30 msgr2=0x7f20c41459f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:32.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.995+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 shutdown_connections 2020-11-20T12:19:32.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:31.995+0000 7f20c9d8d700 1 -- 172.21.15.187:0/2789567425 wait complete. 2020-11-20T12:19:32.031 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:19:32.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.319+0000 7f1c02bb7700 1 Processor -- start 2020-11-20T12:19:32.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.319+0000 7f1c02bb7700 1 -- start start 2020-11-20T12:19:32.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.321+0000 7f1c02bb7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1bfc065190 0x7f1bfc065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:32.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.321+0000 7f1c02bb7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc09ca90 0x7f1bfc09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:32.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.321+0000 7f1c02bb7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bfc0b13e0 con 0x7f1bfc065f90 2020-11-20T12:19:32.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.321+0000 7f1c02bb7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc14ab30 con 0x7f1bfc065190 2020-11-20T12:19:32.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.321+0000 7f1c02bb7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc062c10 con 0x7f1bfc09ca90 2020-11-20T12:19:32.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.321+0000 7f1c01bb5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1bfc065190 0x7f1bfc065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:32.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.322+0000 7f1c01bb5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1bfc065190 0x7f1bfc065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52620/0 (socket says 172.21.15.187:52620) 2020-11-20T12:19:32.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.322+0000 7f1c01bb5700 1 -- 172.21.15.187:0/3621114834 learned_addr learned my addr 172.21.15.187:0/3621114834 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:32.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.322+0000 7f1c023b6700 1 --2- 172.21.15.187:0/3621114834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc09ca90 0x7f1bfc09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:32.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.322+0000 7f1c01bb5700 1 -- 172.21.15.187:0/3621114834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc09ca90 msgr2=0x7f1bfc09cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:32.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.322+0000 7f1c01bb5700 1 --2- 172.21.15.187:0/3621114834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc09ca90 0x7f1bfc09cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.322+0000 7f1c01bb5700 1 -- 172.21.15.187:0/3621114834 >> v1:172.21.15.5:6789/0 conn(0x7f1bfc065f90 legacy=0x7f1bfc143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:32.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.322+0000 7f1c01bb5700 1 -- 172.21.15.187:0/3621114834 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0b1640 con 0x7f1bfc065190 2020-11-20T12:19:32.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.322+0000 7f1c01bb5700 1 --2- 172.21.15.187:0/3621114834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc065570 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7f1bf8006fd0 tx=0x7f1bf8006eb0).ready entity=mon.1 client_cookie=26916acd8fb0b2c3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:32.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c00bb3700 1 -- 172.21.15.187:0/3621114834 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8009920 con 0x7f1bfc065190 2020-11-20T12:19:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c00bb3700 1 -- 172.21.15.187:0/3621114834 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf8009a80 con 0x7f1bfc065190 2020-11-20T12:19:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c00bb3700 1 -- 172.21.15.187:0/3621114834 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8009e00 con 0x7f1bfc065190 2020-11-20T12:19:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3621114834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 msgr2=0x7f1bfc065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3621114834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc065570 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7f1bf8006fd0 tx=0x7f1bf8006eb0).stop 2020-11-20T12:19:32.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3621114834 shutdown_connections 2020-11-20T12:19:32.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3621114834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc09ca90 0x7f1bfc09cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3621114834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc065570 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3621114834 >> 172.21.15.187:0/3621114834 conn(0x7f1bfc0bab30 msgr2=0x7f1bfc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:32.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3621114834 shutdown_connections 2020-11-20T12:19:32.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.323+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3621114834 wait complete. 2020-11-20T12:19:32.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.325+0000 7f1c02bb7700 1 Processor -- start 2020-11-20T12:19:32.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.325+0000 7f1c02bb7700 1 -- start start 2020-11-20T12:19:32.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c02bb7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:32.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c02bb7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc065f90 0x7f1bfc0b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:32.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c02bb7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bfc09ca90 0x7f1bfc09fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:32.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c02bb7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc0b4a40 con 0x7f1bfc09ca90 2020-11-20T12:19:32.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c02bb7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc09f670 con 0x7f1bfc065190 2020-11-20T12:19:32.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c02bb7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc09f910 con 0x7f1bfc065f90 2020-11-20T12:19:32.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c01bb5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:32.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c01bb5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52626/0 (socket says 172.21.15.187:52626) 2020-11-20T12:19:32.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.326+0000 7f1c01bb5700 1 -- 172.21.15.187:0/3876082754 learned_addr learned my addr 172.21.15.187:0/3876082754 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:32.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.327+0000 7f1c023b6700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bfc09ca90 0x7f1bfc09fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:32.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.327+0000 7f1c013b4700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc065f90 0x7f1bfc0b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:32.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.327+0000 7f1c01bb5700 1 -- 172.21.15.187:0/3876082754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc065f90 msgr2=0x7f1bfc0b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:32.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.327+0000 7f1c01bb5700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc065f90 0x7f1bfc0b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.327+0000 7f1c01bb5700 1 -- 172.21.15.187:0/3876082754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bfc09ca90 msgr2=0x7f1bfc09fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:32.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.327+0000 7f1c01bb5700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bfc09ca90 0x7f1bfc09fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:32.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.327+0000 7f1c01bb5700 1 -- 172.21.15.187:0/3876082754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0b7af0 con 0x7f1bfc065190 2020-11-20T12:19:32.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.327+0000 7f1c01bb5700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc146680 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 rx=0x7f1bf800b1b0 tx=0x7f1bf8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:32.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.328+0000 7f1be9ffb700 1 -- 172.21.15.187:0/3876082754 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf801e460 con 0x7f1bfc065190 2020-11-20T12:19:32.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.328+0000 7f1be9ffb700 1 -- 172.21.15.187:0/3876082754 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf801e5c0 con 0x7f1bfc065190 2020-11-20T12:19:32.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.328+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc0b7d40 con 0x7f1bfc065190 2020-11-20T12:19:32.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.328+0000 7f1be9ffb700 1 -- 172.21.15.187:0/3876082754 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf801e940 con 0x7f1bfc065190 2020-11-20T12:19:32.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.328+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc142ca0 con 0x7f1bfc065190 2020-11-20T12:19:32.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.329+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc0b4890 con 0x7f1bfc065190 2020-11-20T12:19:32.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.329+0000 7f1be9ffb700 1 -- 172.21.15.187:0/3876082754 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1bf8025030 con 0x7f1bfc065190 2020-11-20T12:19:32.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.330+0000 7f1be9ffb700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bd4056d20 0x7f1bd40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:32.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.330+0000 7f1be9ffb700 1 -- 172.21.15.187:0/3876082754 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1bf8071900 con 0x7f1bfc065190 2020-11-20T12:19:32.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.330+0000 7f1c013b4700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bd4056d20 0x7f1bd40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:32.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.331+0000 7f1c013b4700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bd4056d20 0x7f1bd40591e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f1bfc0b3a00 tx=0x7f1bec009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:32.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.334+0000 7f1be9ffb700 1 -- 172.21.15.187:0/3876082754 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1bf8046d50 con 0x7f1bfc065190 2020-11-20T12:19:32.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:32.520+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1bfc061a20 con 0x7f1bfc065190 2020-11-20T12:19:33.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.097+0000 7f1be9ffb700 1 -- 172.21.15.187:0/3876082754 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1bf8023030 con 0x7f1bfc065190 2020-11-20T12:19:33.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bd4056d20 msgr2=0x7f1bd40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:33.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bd4056d20 0x7f1bd40591e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f1bfc0b3a00 tx=0x7f1bec009040).stop 2020-11-20T12:19:33.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 msgr2=0x7f1bfc146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:33.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc146680 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 rx=0x7f1bf800b1b0 tx=0x7f1bf8015000).stop 2020-11-20T12:19:33.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 shutdown_connections 2020-11-20T12:19:33.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1bd4056d20 0x7f1bd40591e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:33.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bfc09ca90 0x7f1bfc09fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:33.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bfc065190 0x7f1bfc146680 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:33.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.100+0000 7f1c02bb7700 1 --2- 172.21.15.187:0/3876082754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bfc065f90 0x7f1bfc0b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:33.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.101+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 >> 172.21.15.187:0/3876082754 conn(0x7f1bfc0bab30 msgr2=0x7f1bfc09aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:33.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.101+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 shutdown_connections 2020-11-20T12:19:33.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.101+0000 7f1c02bb7700 1 -- 172.21.15.187:0/3876082754 wait complete. 2020-11-20T12:19:33.108 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:19:33.137 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:19:33.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.393+0000 7fb7ac95d700 1 Processor -- start 2020-11-20T12:19:33.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.393+0000 7fb7ac95d700 1 -- start start 2020-11-20T12:19:33.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.395+0000 7fb7ac95d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb7a8131ab0 0x7fb7a8149290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:33.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.395+0000 7fb7ac95d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7a80b51c0 0x7fb7a814c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:33.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.395+0000 7fb7ac95d700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a8149780 con 0x7fb7a80bf600 2020-11-20T12:19:33.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.395+0000 7fb7ac95d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a814cd50 con 0x7fb7a8131ab0 2020-11-20T12:19:33.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.395+0000 7fb7ac95d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a814be80 con 0x7fb7a80b51c0 2020-11-20T12:19:33.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.395+0000 7fb7a77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb7a8131ab0 0x7fb7a8149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:33.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.395+0000 7fb7a77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb7a8131ab0 0x7fb7a8149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52634/0 (socket says 172.21.15.187:52634) 2020-11-20T12:19:33.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.395+0000 7fb7a77fe700 1 -- 172.21.15.187:0/3061025811 learned_addr learned my addr 172.21.15.187:0/3061025811 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:33.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.396+0000 7fb7a77fe700 1 -- 172.21.15.187:0/3061025811 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7a80b51c0 msgr2=0x7fb7a814c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:33.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.396+0000 7fb7a77fe700 1 --2- 172.21.15.187:0/3061025811 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7a80b51c0 0x7fb7a814c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:33.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.396+0000 7fb7a77fe700 1 -- 172.21.15.187:0/3061025811 >> v1:172.21.15.5:6789/0 conn(0x7fb7a80bf600 legacy=0x7fb7a80b9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:33.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.396+0000 7fb7a77fe700 1 -- 172.21.15.187:0/3061025811 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a814c120 con 0x7fb7a8131ab0 2020-11-20T12:19:33.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.396+0000 7fb7a77fe700 1 --2- 172.21.15.187:0/3061025811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a8131ab0 0x7fb7a8149290 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 rx=0x7fb798009d40 tx=0x7fb798009250).ready entity=mon.1 client_cookie=3d2273584cfc6317 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:33.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7a6ffd700 1 -- 172.21.15.187:0/3061025811 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79800bc40 con 0x7fb7a8131ab0 2020-11-20T12:19:33.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7a6ffd700 1 -- 172.21.15.187:0/3061025811 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79800bda0 con 0x7fb7a8131ab0 2020-11-20T12:19:33.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7a6ffd700 1 -- 172.21.15.187:0/3061025811 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798005c60 con 0x7fb7a8131ab0 2020-11-20T12:19:33.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3061025811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a8131ab0 msgr2=0x7fb7a8149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:33.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3061025811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a8131ab0 0x7fb7a8149290 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 rx=0x7fb798009d40 tx=0x7fb798009250).stop 2020-11-20T12:19:33.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3061025811 shutdown_connections 2020-11-20T12:19:33.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3061025811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a8131ab0 0x7fb7a8149290 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:33.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3061025811 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7a80b51c0 0x7fb7a814c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:33.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.397+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3061025811 >> 172.21.15.187:0/3061025811 conn(0x7fb7a8068370 msgr2=0x7fb7a80beef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:33.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.398+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3061025811 shutdown_connections 2020-11-20T12:19:33.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.398+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3061025811 wait complete. 2020-11-20T12:19:33.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.399+0000 7fb7ac95d700 1 Processor -- start 2020-11-20T12:19:33.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.399+0000 7fb7ac95d700 1 -- start start 2020-11-20T12:19:33.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.400+0000 7fb7ac95d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a80b51c0 0x7fb7a815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:33.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.400+0000 7fb7ac95d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a80bf600 0x7fb7a80a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:33.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.400+0000 7fb7ac95d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7a812e9f0 0x7fb7a81307c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:33.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.400+0000 7fb7ac95d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80a4bb0 con 0x7fb7a80bf600 2020-11-20T12:19:33.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.400+0000 7fb7ac95d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80a3ce0 con 0x7fb7a80b51c0 2020-11-20T12:19:33.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.400+0000 7fb7ac95d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80a3f80 con 0x7fb7a812e9f0 2020-11-20T12:19:33.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.400+0000 7fb7a77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a80b51c0 0x7fb7a815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:33.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb7a77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a80b51c0 0x7fb7a815c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52640/0 (socket says 172.21.15.187:52640) 2020-11-20T12:19:33.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb7a77fe700 1 -- 172.21.15.187:0/3557859096 learned_addr learned my addr 172.21.15.187:0/3557859096 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:33.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb79effd700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a80bf600 0x7fb7a80a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:33.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb7a77fe700 1 -- 172.21.15.187:0/3557859096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7a812e9f0 msgr2=0x7fb7a81307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:33.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb7a77fe700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7a812e9f0 0x7fb7a81307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:33.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb7a77fe700 1 -- 172.21.15.187:0/3557859096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a80bf600 msgr2=0x7fb7a80a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:33.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb7a77fe700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a80bf600 0x7fb7a80a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:33.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb7a77fe700 1 -- 172.21.15.187:0/3557859096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a80a4220 con 0x7fb7a80b51c0 2020-11-20T12:19:33.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb7a77fe700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a80b51c0 0x7fb7a815c260 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7fb7980090b0 tx=0x7fb798015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:33.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.401+0000 7fb79ffff700 1 -- 172.21.15.187:0/3557859096 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7980068b0 con 0x7fb7a80b51c0 2020-11-20T12:19:33.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.402+0000 7fb79ffff700 1 -- 172.21.15.187:0/3557859096 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79801d440 con 0x7fb7a80b51c0 2020-11-20T12:19:33.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.402+0000 7fb79ffff700 1 -- 172.21.15.187:0/3557859096 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79801d800 con 0x7fb7a80b51c0 2020-11-20T12:19:33.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.402+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a812ee50 con 0x7fb7a80b51c0 2020-11-20T12:19:33.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.402+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a80a4eb0 con 0x7fb7a80b51c0 2020-11-20T12:19:33.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.403+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a80a4400 con 0x7fb7a80b51c0 2020-11-20T12:19:33.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.403+0000 7fb79ffff700 1 -- 172.21.15.187:0/3557859096 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb798017070 con 0x7fb7a80b51c0 2020-11-20T12:19:33.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.404+0000 7fb79ffff700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb784056d20 0x7fb7840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:33.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.404+0000 7fb79ffff700 1 -- 172.21.15.187:0/3557859096 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5807+0+0 (secure 0 0 0) 0x7fb798071250 con 0x7fb7a80b51c0 2020-11-20T12:19:33.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.404+0000 7fb79effd700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb784056d20 0x7fb7840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:33.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.405+0000 7fb79effd700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb784056d20 0x7fb7840591e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fb7a80a31c0 tx=0x7fb790008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:33.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.408+0000 7fb79ffff700 1 -- 172.21.15.187:0/3557859096 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb7a80a4400 con 0x7fb7a80b51c0 2020-11-20T12:19:33.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:33.601+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb7a80a4400 con 0x7fb7a80b51c0 2020-11-20T12:19:34.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.109+0000 7fb79ffff700 1 -- 172.21.15.187:0/3557859096 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v134) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb7a80a4400 con 0x7fb7a80b51c0 2020-11-20T12:19:34.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.112+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb784056d20 msgr2=0x7fb7840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:34.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.112+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb784056d20 0x7fb7840591e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fb7a80a31c0 tx=0x7fb790008040).stop 2020-11-20T12:19:34.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.112+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a80b51c0 msgr2=0x7fb7a815c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:34.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.112+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a80b51c0 0x7fb7a815c260 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7fb7980090b0 tx=0x7fb798015000).stop 2020-11-20T12:19:34.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.113+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 shutdown_connections 2020-11-20T12:19:34.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.113+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb784056d20 0x7fb7840591e0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:34.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.113+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7a812e9f0 0x7fb7a81307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:34.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.113+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7a80b51c0 0x7fb7a815c260 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:34.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.113+0000 7fb7ac95d700 1 --2- 172.21.15.187:0/3557859096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a80bf600 0x7fb7a80a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:34.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.113+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 >> 172.21.15.187:0/3557859096 conn(0x7fb7a8068370 msgr2=0x7fb7a80b9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:34.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.113+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 shutdown_connections 2020-11-20T12:19:34.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.113+0000 7fb7ac95d700 1 -- 172.21.15.187:0/3557859096 wait complete. 2020-11-20T12:19:34.122 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:19:34.149 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:19:34.149 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:19:34.150 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:19:34.150 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:19:34.151 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:19:34.151 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:19:34.152 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:19:34.152 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:19:34.174 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:19:34.175 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:19:34.186 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:19:34.187 DEBUG:teuthology.parallel:result is None 2020-11-20T12:19:34.195 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:19:34.196 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:19:34.244 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:19:34.245 DEBUG:teuthology.parallel:result is None 2020-11-20T12:19:34.245 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:19:34.362 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:19:34.447 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:19:34.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.471+0000 7f5e11b04700 1 Processor -- start 2020-11-20T12:19:34.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.471+0000 7f5e11b04700 1 -- start start 2020-11-20T12:19:34.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.472+0000 7f5e11b04700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b1840 0x7f5e0c0b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:34.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.472+0000 7f5e11b04700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5e0c0b11d0 0x7f5e0c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:34.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.472+0000 7f5e11b04700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c149790 con 0x7f5e0c0b1c20 2020-11-20T12:19:34.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.472+0000 7f5e11b04700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c065ba0 con 0x7f5e0c0b11d0 2020-11-20T12:19:34.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.472+0000 7f5e11b04700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0b8ca0 con 0x7f5e0c0b1840 2020-11-20T12:19:34.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.473+0000 7f5e03fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5e0c0b1c20 0x7f5e0c0b8590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49268/0 (socket says 172.21.15.187:49268) 2020-11-20T12:19:34.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.473+0000 7f5e03fff700 1 -- 172.21.15.187:0/2679014445 learned_addr learned my addr 172.21.15.187:0/2679014445 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:34.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.473+0000 7f5e10b02700 1 --2- 172.21.15.187:0/2679014445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b1840 0x7f5e0c0b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:34.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.473+0000 7f5e11303700 1 --2- 172.21.15.187:0/2679014445 >> v2:172.21.15.5:3300/0 conn(0x7f5e0c0b11d0 0x7f5e0c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:34.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.474+0000 7f5e10b02700 1 -- 172.21.15.187:0/2679014445 >> v2:172.21.15.5:3300/0 conn(0x7f5e0c0b11d0 msgr2=0x7f5e0c066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:34.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.474+0000 7f5e10b02700 1 --2- 172.21.15.187:0/2679014445 >> v2:172.21.15.5:3300/0 conn(0x7f5e0c0b11d0 0x7f5e0c066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:34.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.474+0000 7f5e10b02700 1 -- 172.21.15.187:0/2679014445 >> v1:172.21.15.187:6789/0 conn(0x7f5e0c0b1c20 legacy=0x7f5e0c0b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:34.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.474+0000 7f5e10b02700 1 -- 172.21.15.187:0/2679014445 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c149450 con 0x7f5e0c0b1840 2020-11-20T12:19:34.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.474+0000 7f5e10b02700 1 --2- 172.21.15.187:0/2679014445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b1840 0x7f5e0c0b0ce0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5dfc00b9a0 tx=0x7f5dfc006df0).ready entity=mon.2 client_cookie=7af99a5b46253756 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:34.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.474+0000 7f5e037fe700 1 -- 172.21.15.187:0/2679014445 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc009840 con 0x7f5e0c0b1840 2020-11-20T12:19:34.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.474+0000 7f5e037fe700 1 -- 172.21.15.187:0/2679014445 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc0099a0 con 0x7f5e0c0b1840 2020-11-20T12:19:34.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.474+0000 7f5e037fe700 1 -- 172.21.15.187:0/2679014445 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc009d20 con 0x7f5e0c0b1840 2020-11-20T12:19:34.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.475+0000 7f5e11b04700 1 -- 172.21.15.187:0/2679014445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b1840 msgr2=0x7f5e0c0b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:34.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.475+0000 7f5e11b04700 1 --2- 172.21.15.187:0/2679014445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b1840 0x7f5e0c0b0ce0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5dfc00b9a0 tx=0x7f5dfc006df0).stop 2020-11-20T12:19:34.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.475+0000 7f5e11b04700 1 -- 172.21.15.187:0/2679014445 shutdown_connections 2020-11-20T12:19:34.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.475+0000 7f5e11b04700 1 --2- 172.21.15.187:0/2679014445 >> v2:172.21.15.5:3300/0 conn(0x7f5e0c0b11d0 0x7f5e0c066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:34.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.475+0000 7f5e11b04700 1 --2- 172.21.15.187:0/2679014445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b1840 0x7f5e0c0b0ce0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:34.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.475+0000 7f5e11b04700 1 -- 172.21.15.187:0/2679014445 >> 172.21.15.187:0/2679014445 conn(0x7f5e0c0bbeb0 msgr2=0x7f5e0c0b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:34.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.475+0000 7f5e11b04700 1 -- 172.21.15.187:0/2679014445 shutdown_connections 2020-11-20T12:19:34.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.475+0000 7f5e11b04700 1 -- 172.21.15.187:0/2679014445 wait complete. 2020-11-20T12:19:34.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.477+0000 7f5e11b04700 1 Processor -- start 2020-11-20T12:19:34.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.477+0000 7f5e11b04700 1 -- start start 2020-11-20T12:19:34.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e11b04700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b11d0 0x7f5e0c14aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:34.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e11b04700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0b1840 0x7f5e0c1361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:34.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e11b04700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c0b1c20 0x7f5e0c136690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:34.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e11b04700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c136b80 con 0x7f5e0c0b1c20 2020-11-20T12:19:34.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e11b04700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0b4600 con 0x7f5e0c0b1840 2020-11-20T12:19:34.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e11b04700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0b48a0 con 0x7f5e0c0b11d0 2020-11-20T12:19:34.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e03fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0b1840 0x7f5e0c1361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:34.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e03fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0b1840 0x7f5e0c1361a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52668/0 (socket says 172.21.15.187:52668) 2020-11-20T12:19:34.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e03fff700 1 -- 172.21.15.187:0/3366641431 learned_addr learned my addr 172.21.15.187:0/3366641431 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:34.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e11303700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c0b1c20 0x7f5e0c136690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:34.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e10b02700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b11d0 0x7f5e0c14aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:34.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e03fff700 1 -- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b11d0 msgr2=0x7f5e0c14aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:34.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.478+0000 7f5e03fff700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b11d0 0x7f5e0c14aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:34.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e03fff700 1 -- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c0b1c20 msgr2=0x7f5e0c136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:34.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e03fff700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c0b1c20 0x7f5e0c136690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:34.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e03fff700 1 -- 172.21.15.187:0/3366641431 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c0b4b40 con 0x7f5e0c0b1840 2020-11-20T12:19:34.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e10b02700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b11d0 0x7f5e0c14aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:34.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e03fff700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0b1840 0x7f5e0c1361a0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f5e0800c6e0 tx=0x7f5e08007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:34.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e00ff9700 1 -- 172.21.15.187:0/3366641431 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e080072e0 con 0x7f5e0c0b1840 2020-11-20T12:19:34.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e00ff9700 1 -- 172.21.15.187:0/3366641431 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e08007440 con 0x7f5e0c0b1840 2020-11-20T12:19:34.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c14b250 con 0x7f5e0c0b1840 2020-11-20T12:19:34.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e00ff9700 1 -- 172.21.15.187:0/3366641431 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08005800 con 0x7f5e0c0b1840 2020-11-20T12:19:34.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.479+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c0b2500 con 0x7f5e0c0b1840 2020-11-20T12:19:34.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.480+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c0b4cd0 con 0x7f5e0c0b1840 2020-11-20T12:19:34.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.481+0000 7f5e00ff9700 1 -- 172.21.15.187:0/3366641431 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5e08010070 con 0x7f5e0c0b1840 2020-11-20T12:19:34.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.481+0000 7f5e00ff9700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5de4056d20 0x7f5de40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:34.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.482+0000 7f5e00ff9700 1 -- 172.21.15.187:0/3366641431 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5425+0+0 (secure 0 0 0) 0x7f5e0806f540 con 0x7f5e0c0b1840 2020-11-20T12:19:34.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.482+0000 7f5e10b02700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5de4056d20 0x7f5de40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:34.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.482+0000 7f5e10b02700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5de4056d20 0x7f5de40591e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7f5dfc009fd0 tx=0x7f5dfc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:34.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.485+0000 7f5e00ff9700 1 -- 172.21.15.187:0/3366641431 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5e08044bb0 con 0x7f5e0c0b1840 2020-11-20T12:19:34.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:34.687+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5e0c061050 con 0x7f5e0c0b1840 2020-11-20T12:19:35.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.398+0000 7f5e00ff9700 1 -- 172.21.15.187:0/3366641431 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v135) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5e08007d50 con 0x7f5e0c0b1840 2020-11-20T12:19:35.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.401+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5de4056d20 msgr2=0x7f5de40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:35.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5de4056d20 0x7f5de40591e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7f5dfc009fd0 tx=0x7f5dfc015000).stop 2020-11-20T12:19:35.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0b1840 msgr2=0x7f5e0c1361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:35.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0b1840 0x7f5e0c1361a0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f5e0800c6e0 tx=0x7f5e08007c10).stop 2020-11-20T12:19:35.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 shutdown_connections 2020-11-20T12:19:35.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5de4056d20 0x7f5de40591e0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5e0c0b1c20 0x7f5e0c136690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5e0c0b11d0 0x7f5e0c14aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 --2- 172.21.15.187:0/3366641431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e0c0b1840 0x7f5e0c1361a0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 >> 172.21.15.187:0/3366641431 conn(0x7f5e0c0bbeb0 msgr2=0x7f5e0c0631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:35.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 shutdown_connections 2020-11-20T12:19:35.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.402+0000 7f5e11b04700 1 -- 172.21.15.187:0/3366641431 wait complete. 2020-11-20T12:19:35.410 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:19:35.437 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:19:35.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.682+0000 7efca201a700 1 Processor -- start 2020-11-20T12:19:35.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.682+0000 7efca201a700 1 -- start start 2020-11-20T12:19:35.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.683+0000 7efca201a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efc9c0652e0 0x7efc9c0656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:35.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.683+0000 7efca201a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efc9c0b1590 0x7efc9c14a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:35.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.683+0000 7efca201a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c149dd0 con 0x7efc9c0b7300 2020-11-20T12:19:35.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.684+0000 7efca201a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9c14a070 con 0x7efc9c0b1590 2020-11-20T12:19:35.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.684+0000 7efca201a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9c14a310 con 0x7efc9c0652e0 2020-11-20T12:19:35.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.684+0000 7efc9b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efc9c0652e0 0x7efc9c0656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:35.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.684+0000 7efc9bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7efc9c0b7300 0x7efc9c143920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49284/0 (socket says 172.21.15.187:49284) 2020-11-20T12:19:35.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.684+0000 7efc9b7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efc9c0652e0 0x7efc9c0656c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52674/0 (socket says 172.21.15.187:52674) 2020-11-20T12:19:35.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.684+0000 7efc9bfff700 1 -- 172.21.15.187:0/639917718 learned_addr learned my addr 172.21.15.187:0/639917718 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:35.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.684+0000 7efc92ffd700 1 --2- 172.21.15.187:0/639917718 >> v2:172.21.15.5:3300/0 conn(0x7efc9c0b1590 0x7efc9c14a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:35.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9b7fe700 1 -- 172.21.15.187:0/639917718 >> v2:172.21.15.5:3300/0 conn(0x7efc9c0b1590 msgr2=0x7efc9c14a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:35.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9b7fe700 1 --2- 172.21.15.187:0/639917718 >> v2:172.21.15.5:3300/0 conn(0x7efc9c0b1590 0x7efc9c14a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9b7fe700 1 -- 172.21.15.187:0/639917718 >> v1:172.21.15.187:6789/0 conn(0x7efc9c0b7300 legacy=0x7efc9c143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:35.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9affd700 1 -- 172.21.15.187:0/639917718 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811383581 0 0) 0x7efc9c149dd0 con 0x7efc9c0b7300 2020-11-20T12:19:35.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9b7fe700 1 -- 172.21.15.187:0/639917718 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c14aca0 con 0x7efc9c0652e0 2020-11-20T12:19:35.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9b7fe700 1 --2- 172.21.15.187:0/639917718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c0656c0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 rx=0x7efc9c0b87c0 tx=0x7efc8c006df0).ready entity=mon.1 client_cookie=b8e52a9a370e2b02 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:35.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9affd700 1 -- 172.21.15.187:0/639917718 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0097b0 con 0x7efc9c0652e0 2020-11-20T12:19:35.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9affd700 1 -- 172.21.15.187:0/639917718 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c009910 con 0x7efc9c0652e0 2020-11-20T12:19:35.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.685+0000 7efc9affd700 1 -- 172.21.15.187:0/639917718 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c009c90 con 0x7efc9c0652e0 2020-11-20T12:19:35.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.686+0000 7efca201a700 1 -- 172.21.15.187:0/639917718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 msgr2=0x7efc9c0656c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:35.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.686+0000 7efca201a700 1 --2- 172.21.15.187:0/639917718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c0656c0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 rx=0x7efc9c0b87c0 tx=0x7efc8c006df0).stop 2020-11-20T12:19:35.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.686+0000 7efca201a700 1 -- 172.21.15.187:0/639917718 shutdown_connections 2020-11-20T12:19:35.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.686+0000 7efca201a700 1 --2- 172.21.15.187:0/639917718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c0656c0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.686+0000 7efca201a700 1 --2- 172.21.15.187:0/639917718 >> v2:172.21.15.5:3300/0 conn(0x7efc9c0b1590 0x7efc9c14a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.686+0000 7efca201a700 1 -- 172.21.15.187:0/639917718 >> 172.21.15.187:0/639917718 conn(0x7efc9c0bacd0 msgr2=0x7efc9c0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:35.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.686+0000 7efca201a700 1 -- 172.21.15.187:0/639917718 shutdown_connections 2020-11-20T12:19:35.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.686+0000 7efca201a700 1 -- 172.21.15.187:0/639917718 wait complete. 2020-11-20T12:19:35.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.688+0000 7efca201a700 1 Processor -- start 2020-11-20T12:19:35.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.688+0000 7efca201a700 1 -- start start 2020-11-20T12:19:35.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.689+0000 7efca201a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:35.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.689+0000 7efca201a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc9c0b1590 0x7efc9c0b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:35.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.689+0000 7efca201a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9c0b7300 0x7efc9c144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:35.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.689+0000 7efca201a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c136c20 con 0x7efc9c0b1590 2020-11-20T12:19:35.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.689+0000 7efca201a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c136e80 con 0x7efc9c0652e0 2020-11-20T12:19:35.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.689+0000 7efca201a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c137120 con 0x7efc9c0b7300 2020-11-20T12:19:35.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.689+0000 7efc9b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:35.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc9b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c149920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52680/0 (socket says 172.21.15.187:52680) 2020-11-20T12:19:35.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc9b7fe700 1 -- 172.21.15.187:0/3671496549 learned_addr learned my addr 172.21.15.187:0/3671496549 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:35.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc92ffd700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc9c0b1590 0x7efc9c0b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:35.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc9b7fe700 1 -- 172.21.15.187:0/3671496549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9c0b7300 msgr2=0x7efc9c144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:35.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc9b7fe700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9c0b7300 0x7efc9c144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:35.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc9b7fe700 1 -- 172.21.15.187:0/3671496549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc9c0b1590 msgr2=0x7efc9c0b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:35.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc9b7fe700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc9c0b1590 0x7efc9c0b4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc9b7fe700 1 -- 172.21.15.187:0/3671496549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c1375c0 con 0x7efc9c0652e0 2020-11-20T12:19:35.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc9b7fe700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c149920 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 rx=0x7efc8c0098e0 tx=0x7efc8c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:35.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.690+0000 7efc93fff700 1 -- 172.21.15.187:0/3671496549 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c01e460 con 0x7efc9c0652e0 2020-11-20T12:19:35.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.691+0000 7efc93fff700 1 -- 172.21.15.187:0/3671496549 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c01e5c0 con 0x7efc9c0652e0 2020-11-20T12:19:35.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.691+0000 7efc93fff700 1 -- 172.21.15.187:0/3671496549 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c01e980 con 0x7efc9c0652e0 2020-11-20T12:19:35.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.691+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c08f5c0 con 0x7efc9c0652e0 2020-11-20T12:19:35.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.691+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c1482d0 con 0x7efc9c0652e0 2020-11-20T12:19:35.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.692+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c148580 con 0x7efc9c0652e0 2020-11-20T12:19:35.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.693+0000 7efc93fff700 1 -- 172.21.15.187:0/3671496549 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efc8c01a040 con 0x7efc9c0652e0 2020-11-20T12:19:35.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.694+0000 7efc93fff700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc78056d50 0x7efc78059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:35.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.694+0000 7efc93fff700 1 -- 172.21.15.187:0/3671496549 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efc8c071540 con 0x7efc9c0652e0 2020-11-20T12:19:35.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.694+0000 7efc92ffd700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc78056d50 0x7efc78059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:35.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.695+0000 7efc92ffd700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc78056d50 0x7efc78059210 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7efc9c0b19d0 tx=0x7efc84008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:35.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.697+0000 7efc93fff700 1 -- 172.21.15.187:0/3671496549 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efc8c046bb0 con 0x7efc9c0652e0 2020-11-20T12:19:35.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.947+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efc9c1372e0 con 0x7efc9c0652e0 2020-11-20T12:19:35.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.948+0000 7efc93fff700 1 -- 172.21.15.187:0/3671496549 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+4570 (secure 0 0 0) 0x7efc8c023090 con 0x7efc9c0652e0 2020-11-20T12:19:35.952 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:35.952 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:19:35.953 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:19:35.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.953 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:19:35.953 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:19:35.953 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.954 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:19:35.954 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.954 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.954 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.955 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.955 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:19:35.955 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:19:35.956 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.956 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:19:35.956 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.956 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.956 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.957 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.957 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:19:35.957 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:19:35.958 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.958 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.958 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.958 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.958 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.959 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:19:35.959 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:19:35.959 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.960 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.960 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.960 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.960 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.961 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:19:35.961 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:19:35.961 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.961 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.962 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.962 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.962 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.963 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:19:35.963 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:19:35.963 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.963 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.963 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.964 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.964 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.965 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:19:35.965 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:19:35.965 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.965 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.965 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.966 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.966 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.966 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:19:35.967 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:19:35.967 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.967 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.967 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.967 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.968 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.968 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:19:35.968 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:19:35.969 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.969 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.969 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.969 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.969 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.970 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:19:35.970 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:19:35.970 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.971 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.971 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.971 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.971 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.972 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:19:35.972 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:19:35.972 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.972 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:19:35.973 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:19:35.973 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:19:35.973 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.974 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:19:35.974 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:19:35.974 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.974 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:19:35.975 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:19:35.975 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:19:35.975 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:19:35.975 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.976 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:19:35.976 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:19:35.976 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.976 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:19:35.977 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:19:35.977 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:19:35.977 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:19:35.977 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.978 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:19:35.978 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:19:35.978 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.978 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:19:35.979 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:19:35.979 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:19:35.979 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:19:35.979 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.980 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:19:35.980 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:19:35.980 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.980 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:19:35.981 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:19:35.981 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:19:35.981 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.981 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.981 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.982 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:19:35.982 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:19:35.982 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.982 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:19:35.983 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:19:35.983 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.983 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:35.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:35.984 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:19:35.984 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:19:35.984 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:19:35.984 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:19:35.984 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:19:35.985 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:19:35.985 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.985 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:35.985 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:19:35.986 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:19:35.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc78056d50 msgr2=0x7efc78059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:35.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc78056d50 0x7efc78059210 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7efc9c0b19d0 tx=0x7efc84008040).stop 2020-11-20T12:19:35.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 msgr2=0x7efc9c149920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:35.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c149920 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 rx=0x7efc8c0098e0 tx=0x7efc8c015000).stop 2020-11-20T12:19:35.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 shutdown_connections 2020-11-20T12:19:35.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc78056d50 0x7efc78059210 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9c0b7300 0x7efc9c144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:35.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc9c0652e0 0x7efc9c149920 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 --2- 172.21.15.187:0/3671496549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc9c0b1590 0x7efc9c0b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:35.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 >> 172.21.15.187:0/3671496549 conn(0x7efc9c0bacd0 msgr2=0x7efc9c0b8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:35.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 shutdown_connections 2020-11-20T12:19:35.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:35.951+0000 7efca201a700 1 -- 172.21.15.187:0/3671496549 wait complete. 2020-11-20T12:19:35.990 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:19:36.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.230+0000 7f96b359e700 1 Processor -- start 2020-11-20T12:19:36.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.231+0000 7f96b359e700 1 -- start start 2020-11-20T12:19:36.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.232+0000 7f96b359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065f90 0x7f96b4065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.232+0000 7f96b359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f96b409ca90 0x7f96b409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.232+0000 7f96b359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b40b13e0 con 0x7f96b4065190 2020-11-20T12:19:36.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.232+0000 7f96b359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b414ab30 con 0x7f96b409ca90 2020-11-20T12:19:36.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.232+0000 7f96b359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b4062c10 con 0x7f96b4065f90 2020-11-20T12:19:36.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.233+0000 7f96b259c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f96b4065190 0x7f96b4143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:42804/0 (socket says 172.21.15.187:42804) 2020-11-20T12:19:36.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.233+0000 7f96b259c700 1 -- 172.21.15.187:0/3999998695 learned_addr learned my addr 172.21.15.187:0/3999998695 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:36.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.233+0000 7f96a9d9b700 1 --2- 172.21.15.187:0/3999998695 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065f90 0x7f96b4065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.233+0000 7f96b2d9d700 1 --2- 172.21.15.187:0/3999998695 >> v2:172.21.15.5:3300/0 conn(0x7f96b409ca90 0x7f96b409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.234+0000 7f96b1d9b700 1 -- 172.21.15.187:0/3999998695 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756472173 0 0) 0x7f96b40b13e0 con 0x7f96b4065190 2020-11-20T12:19:36.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.234+0000 7f96b1d9b700 1 -- 172.21.15.187:0/3999998695 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9694003240 con 0x7f96b4065190 2020-11-20T12:19:36.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.234+0000 7f96b2d9d700 1 -- 172.21.15.187:0/3999998695 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065f90 msgr2=0x7f96b4065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:36.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.234+0000 7f96b2d9d700 1 --2- 172.21.15.187:0/3999998695 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065f90 0x7f96b4065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.234+0000 7f96b2d9d700 1 -- 172.21.15.187:0/3999998695 >> v1:172.21.15.5:6789/0 conn(0x7f96b4065190 legacy=0x7f96b4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.234+0000 7f96b2d9d700 1 -- 172.21.15.187:0/3999998695 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40b1640 con 0x7f96b409ca90 2020-11-20T12:19:36.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.235+0000 7f96b2d9d700 1 --2- 172.21.15.187:0/3999998695 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b409ca90 0x7f96b409cf20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f96b414ac70 tx=0x7f96ac0091d0).ready entity=mon.0 client_cookie=af4da9246a03530f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:36.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.235+0000 7f96b1d9b700 1 -- 172.21.15.187:0/3999998695 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00ba20 con 0x7f96b409ca90 2020-11-20T12:19:36.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.236+0000 7f96b1d9b700 1 -- 172.21.15.187:0/3999998695 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac00bb80 con 0x7f96b409ca90 2020-11-20T12:19:36.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.236+0000 7f96b1d9b700 1 -- 172.21.15.187:0/3999998695 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac011400 con 0x7f96b409ca90 2020-11-20T12:19:36.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.236+0000 7f96b359e700 1 -- 172.21.15.187:0/3999998695 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b409ca90 msgr2=0x7f96b409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.237+0000 7f96b359e700 1 --2- 172.21.15.187:0/3999998695 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b409ca90 0x7f96b409cf20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f96b414ac70 tx=0x7f96ac0091d0).stop 2020-11-20T12:19:36.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.237+0000 7f96b359e700 1 -- 172.21.15.187:0/3999998695 shutdown_connections 2020-11-20T12:19:36.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.237+0000 7f96b359e700 1 --2- 172.21.15.187:0/3999998695 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b409ca90 0x7f96b409cf20 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.237+0000 7f96b359e700 1 --2- 172.21.15.187:0/3999998695 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065f90 0x7f96b4065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.237+0000 7f96b359e700 1 -- 172.21.15.187:0/3999998695 >> 172.21.15.187:0/3999998695 conn(0x7f96b40bab30 msgr2=0x7f96b40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:36.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.237+0000 7f96b359e700 1 -- 172.21.15.187:0/3999998695 shutdown_connections 2020-11-20T12:19:36.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.237+0000 7f96b359e700 1 -- 172.21.15.187:0/3999998695 wait complete. 2020-11-20T12:19:36.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.239+0000 7f96b359e700 1 Processor -- start 2020-11-20T12:19:36.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.239+0000 7f96b359e700 1 -- start start 2020-11-20T12:19:36.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.240+0000 7f96b359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065190 0x7f96b40a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.240+0000 7f96b359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b4065f90 0x7f96b41412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.240+0000 7f96b359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96b409ca90 0x7f96b409f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.240+0000 7f96b359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b4145080 con 0x7f96b4065f90 2020-11-20T12:19:36.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.240+0000 7f96b359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b41417c0 con 0x7f96b409ca90 2020-11-20T12:19:36.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.240+0000 7f96b359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b4067990 con 0x7f96b4065190 2020-11-20T12:19:36.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b2d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96b409ca90 0x7f96b409f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b2d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96b409ca90 0x7f96b409f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52698/0 (socket says 172.21.15.187:52698) 2020-11-20T12:19:36.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b2d9d700 1 -- 172.21.15.187:0/2416550813 learned_addr learned my addr 172.21.15.187:0/2416550813 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:36.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96a9d9b700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b4065f90 0x7f96b41412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b259c700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065190 0x7f96b40a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b2d9d700 1 -- 172.21.15.187:0/2416550813 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065190 msgr2=0x7f96b40a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b2d9d700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065190 0x7f96b40a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b2d9d700 1 -- 172.21.15.187:0/2416550813 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b4065f90 msgr2=0x7f96b41412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b2d9d700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b4065f90 0x7f96b41412d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.241+0000 7f96b2d9d700 1 -- 172.21.15.187:0/2416550813 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b4067c30 con 0x7f96b409ca90 2020-11-20T12:19:36.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.242+0000 7f96b2d9d700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96b409ca90 0x7f96b409f910 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7f96b40b1570 tx=0x7f96ac016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:36.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.242+0000 7f96aaffd700 1 -- 172.21.15.187:0/2416550813 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac010700 con 0x7f96b409ca90 2020-11-20T12:19:36.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.242+0000 7f96aaffd700 1 -- 172.21.15.187:0/2416550813 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac010860 con 0x7f96b409ca90 2020-11-20T12:19:36.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.242+0000 7f96aaffd700 1 -- 172.21.15.187:0/2416550813 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac010be0 con 0x7f96b409ca90 2020-11-20T12:19:36.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.242+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b409e070 con 0x7f96b409ca90 2020-11-20T12:19:36.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.242+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b409fe00 con 0x7f96b409ca90 2020-11-20T12:19:36.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.244+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b41451f0 con 0x7f96b409ca90 2020-11-20T12:19:36.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.244+0000 7f96aaffd700 1 -- 172.21.15.187:0/2416550813 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f96ac024030 con 0x7f96b409ca90 2020-11-20T12:19:36.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.245+0000 7f96aaffd700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9690056d50 0x7f9690059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.245+0000 7f96aaffd700 1 -- 172.21.15.187:0/2416550813 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f96ac071640 con 0x7f96b409ca90 2020-11-20T12:19:36.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.245+0000 7f96b259c700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9690056d50 0x7f9690059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.246+0000 7f96b259c700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9690056d50 0x7f9690059210 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f96a40055a0 tx=0x7f96a4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:36.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.249+0000 7f96aaffd700 1 -- 172.21.15.187:0/2416550813 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f96b41451f0 con 0x7f96b409ca90 2020-11-20T12:19:36.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.436+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96b41451f0 con 0x7f96b409ca90 2020-11-20T12:19:36.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.438+0000 7f96aaffd700 1 -- 172.21.15.187:0/2416550813 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+13637 (secure 0 0 0) 0x7f96ac02f560 con 0x7f96b409ca90 2020-11-20T12:19:36.442 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:36.443 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":135,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:19:35.389290+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:19:36.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9690056d50 msgr2=0x7f9690059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.445 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9690056d50 0x7f9690059210 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f96a40055a0 tx=0x7f96a4009040).stop 2020-11-20T12:19:36.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96b409ca90 msgr2=0x7f96b409f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96b409ca90 0x7f96b409f910 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7f96b40b1570 tx=0x7f96ac016000).stop 2020-11-20T12:19:36.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 shutdown_connections 2020-11-20T12:19:36.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9690056d50 0x7f9690059210 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96b409ca90 0x7f96b409f910 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96b4065190 0x7f96b40a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 --2- 172.21.15.187:0/2416550813 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96b4065f90 0x7f96b41412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 >> 172.21.15.187:0/2416550813 conn(0x7f96b40bab30 msgr2=0x7f96b40aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:36.447 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 shutdown_connections 2020-11-20T12:19:36.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.441+0000 7f96b359e700 1 -- 172.21.15.187:0/2416550813 wait complete. 2020-11-20T12:19:36.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:19:36.473 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:19:36.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.725+0000 7f721727f700 1 Processor -- start 2020-11-20T12:19:36.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.725+0000 7f721727f700 1 -- start start 2020-11-20T12:19:36.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.726+0000 7f721727f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7210065190 0x7f7210065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.726+0000 7f721727f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f721009ca90 0x7f721009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.726+0000 7f721727f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100b13e0 con 0x7f7210065f90 2020-11-20T12:19:36.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.726+0000 7f721727f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721014ab30 con 0x7f721009ca90 2020-11-20T12:19:36.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.726+0000 7f721727f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7210062c10 con 0x7f7210065190 2020-11-20T12:19:36.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.727+0000 7f721627d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7210065190 0x7f7210065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.727+0000 7f721627d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7210065190 0x7f7210065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52704/0 (socket says 172.21.15.187:52704) 2020-11-20T12:19:36.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.727+0000 7f721627d700 1 -- 172.21.15.187:0/2659098960 learned_addr learned my addr 172.21.15.187:0/2659098960 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:36.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.727+0000 7f7216a7e700 1 --2- 172.21.15.187:0/2659098960 >> v2:172.21.15.5:3300/0 conn(0x7f721009ca90 0x7f721009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.728+0000 7f721627d700 1 -- 172.21.15.187:0/2659098960 >> v2:172.21.15.5:3300/0 conn(0x7f721009ca90 msgr2=0x7f721009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:36.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.728+0000 7f721627d700 1 --2- 172.21.15.187:0/2659098960 >> v2:172.21.15.5:3300/0 conn(0x7f721009ca90 0x7f721009cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.728+0000 7f721627d700 1 -- 172.21.15.187:0/2659098960 >> v1:172.21.15.5:6789/0 conn(0x7f7210065f90 legacy=0x7f7210143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:36.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.728+0000 7f721627d700 1 -- 172.21.15.187:0/2659098960 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100b1640 con 0x7f7210065190 2020-11-20T12:19:36.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.728+0000 7f721627d700 1 --2- 172.21.15.187:0/2659098960 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065190 0x7f7210065570 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f721005c8b0 tx=0x7f720c006eb0).ready entity=mon.1 client_cookie=eb66a96994623bf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:36.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721527b700 1 -- 172.21.15.187:0/2659098960 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c009860 con 0x7f7210065190 2020-11-20T12:19:36.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721527b700 1 -- 172.21.15.187:0/2659098960 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c0099c0 con 0x7f7210065190 2020-11-20T12:19:36.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721527b700 1 -- 172.21.15.187:0/2659098960 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c009d40 con 0x7f7210065190 2020-11-20T12:19:36.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721727f700 1 -- 172.21.15.187:0/2659098960 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065190 msgr2=0x7f7210065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721727f700 1 --2- 172.21.15.187:0/2659098960 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065190 0x7f7210065570 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f721005c8b0 tx=0x7f720c006eb0).stop 2020-11-20T12:19:36.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721727f700 1 -- 172.21.15.187:0/2659098960 shutdown_connections 2020-11-20T12:19:36.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721727f700 1 --2- 172.21.15.187:0/2659098960 >> v2:172.21.15.5:3300/0 conn(0x7f721009ca90 0x7f721009cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721727f700 1 --2- 172.21.15.187:0/2659098960 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065190 0x7f7210065570 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.729+0000 7f721727f700 1 -- 172.21.15.187:0/2659098960 >> 172.21.15.187:0/2659098960 conn(0x7f72100bab30 msgr2=0x7f72100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:36.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.730+0000 7f721727f700 1 -- 172.21.15.187:0/2659098960 shutdown_connections 2020-11-20T12:19:36.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.730+0000 7f721727f700 1 -- 172.21.15.187:0/2659098960 wait complete. 2020-11-20T12:19:36.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.731+0000 7f721727f700 1 Processor -- start 2020-11-20T12:19:36.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.731+0000 7f721727f700 1 -- start start 2020-11-20T12:19:36.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721727f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065f90 0x7f72100b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721727f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f721009ca90 0x7f721013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721727f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f721013f4b0 0x7f7210159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721727f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721013f910 con 0x7f721009ca90 2020-11-20T12:19:36.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721727f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210149880 con 0x7f7210065f90 2020-11-20T12:19:36.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721727f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7210149b20 con 0x7f721013f4b0 2020-11-20T12:19:36.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721627d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065f90 0x7f72100b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721627d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065f90 0x7f72100b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52706/0 (socket says 172.21.15.187:52706) 2020-11-20T12:19:36.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.732+0000 7f721627d700 1 -- 172.21.15.187:0/2378594185 learned_addr learned my addr 172.21.15.187:0/2378594185 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:36.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f7215a7c700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f721009ca90 0x7f721013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f721627d700 1 -- 172.21.15.187:0/2378594185 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f721013f4b0 msgr2=0x7f7210159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f721627d700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f721013f4b0 0x7f7210159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:36.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f721627d700 1 -- 172.21.15.187:0/2378594185 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f721009ca90 msgr2=0x7f721013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f721627d700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f721009ca90 0x7f721013efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f721627d700 1 -- 172.21.15.187:0/2378594185 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7210149dc0 con 0x7f7210065f90 2020-11-20T12:19:36.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f721627d700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065f90 0x7f72100b28f0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 rx=0x7f720c006ef0 tx=0x7f720c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:36.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f71fe7fc700 1 -- 172.21.15.187:0/2378594185 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c01e460 con 0x7f7210065f90 2020-11-20T12:19:36.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f71fe7fc700 1 -- 172.21.15.187:0/2378594185 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c01e5c0 con 0x7f7210065f90 2020-11-20T12:19:36.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f71fe7fc700 1 -- 172.21.15.187:0/2378594185 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c01e940 con 0x7f7210065f90 2020-11-20T12:19:36.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.733+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721015a030 con 0x7f7210065f90 2020-11-20T12:19:36.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.734+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721015a520 con 0x7f7210065f90 2020-11-20T12:19:36.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.735+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7210149f50 con 0x7f7210065f90 2020-11-20T12:19:36.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.735+0000 7f71fe7fc700 1 -- 172.21.15.187:0/2378594185 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f720c025030 con 0x7f7210065f90 2020-11-20T12:19:36.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.736+0000 7f71fe7fc700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71e8056d50 0x7f71e8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:36.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.736+0000 7f71fe7fc700 1 -- 172.21.15.187:0/2378594185 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f720c071400 con 0x7f7210065f90 2020-11-20T12:19:36.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.736+0000 7f7215a7c700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71e8056d50 0x7f71e8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:36.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.737+0000 7f7215a7c700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71e8056d50 0x7f71e8059210 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f72100a08e0 tx=0x7f7200009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:36.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.740+0000 7f71fe7fc700 1 -- 172.21.15.187:0/2378594185 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f720c046c30 con 0x7f7210065f90 2020-11-20T12:19:36.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.926+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7210061a20 con 0x7f7210065f90 2020-11-20T12:19:36.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.927+0000 7f71fe7fc700 1 -- 172.21.15.187:0/2378594185 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v135) v1 ==== 110+0+10 (secure 0 0 0) 0x7f720c023020 con 0x7f7210065f90 2020-11-20T12:19:36.931 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:19:36.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71e8056d50 msgr2=0x7f71e8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71e8056d50 0x7f71e8059210 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f72100a08e0 tx=0x7f7200009040).stop 2020-11-20T12:19:36.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065f90 msgr2=0x7f72100b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:36.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065f90 0x7f72100b28f0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 rx=0x7f720c006ef0 tx=0x7f720c015000).stop 2020-11-20T12:19:36.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 shutdown_connections 2020-11-20T12:19:36.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f71e8056d50 0x7f71e8059210 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f721013f4b0 0x7f7210159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:36.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7210065f90 0x7f72100b28f0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 --2- 172.21.15.187:0/2378594185 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f721009ca90 0x7f721013efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:36.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 >> 172.21.15.187:0/2378594185 conn(0x7f72100bab30 msgr2=0x7f7210061d00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:36.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.930+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 shutdown_connections 2020-11-20T12:19:36.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:36.931+0000 7f721727f700 1 -- 172.21.15.187:0/2378594185 wait complete. 2020-11-20T12:19:36.958 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:19:37.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.202+0000 7f8d73fff700 1 Processor -- start 2020-11-20T12:19:37.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.203+0000 7f8d73fff700 1 -- start start 2020-11-20T12:19:37.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.204+0000 7f8d73fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8d74145cc0 0x7f8d74066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.204+0000 7f8d73fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d74148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.204+0000 7f8d73fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7414c170 con 0x7f8d740683f0 2020-11-20T12:19:37.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.204+0000 7f8d73fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7414c410 con 0x7f8d74145cc0 2020-11-20T12:19:37.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.204+0000 7f8d73fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7414c6b0 con 0x7f8d741487e0 2020-11-20T12:19:37.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.205+0000 7f8d737fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d74148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.205+0000 7f8d72ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8d74145cc0 0x7f8d74066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.205+0000 7f8d737fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d74148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:53606/0 (socket says 172.21.15.187:53606) 2020-11-20T12:19:37.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.205+0000 7f8d737fe700 1 -- 172.21.15.187:0/1245489402 learned_addr learned my addr 172.21.15.187:0/1245489402 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:37.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.206+0000 7f8d737fe700 1 -- 172.21.15.187:0/1245489402 >> v2:172.21.15.5:3300/0 conn(0x7f8d74145cc0 msgr2=0x7f8d74066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:37.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.206+0000 7f8d737fe700 1 --2- 172.21.15.187:0/1245489402 >> v2:172.21.15.5:3300/0 conn(0x7f8d74145cc0 0x7f8d74066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.206+0000 7f8d737fe700 1 -- 172.21.15.187:0/1245489402 >> v1:172.21.15.5:6789/0 conn(0x7f8d740683f0 legacy=0x7f8d74065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.206+0000 7f8d737fe700 1 -- 172.21.15.187:0/1245489402 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d740658a0 con 0x7f8d741487e0 2020-11-20T12:19:37.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.206+0000 7f8d737fe700 1 --2- 172.21.15.187:0/1245489402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d74148c90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f8d6c009d50 tx=0x7f8d6c009290).ready entity=mon.2 client_cookie=cceed9fb769a17e6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:37.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.207+0000 7f8d727fc700 1 -- 172.21.15.187:0/1245489402 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c00bbc0 con 0x7f8d741487e0 2020-11-20T12:19:37.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.207+0000 7f8d727fc700 1 -- 172.21.15.187:0/1245489402 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c00bd20 con 0x7f8d741487e0 2020-11-20T12:19:37.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.207+0000 7f8d727fc700 1 -- 172.21.15.187:0/1245489402 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c011610 con 0x7f8d741487e0 2020-11-20T12:19:37.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.208+0000 7f8d73fff700 1 -- 172.21.15.187:0/1245489402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 msgr2=0x7f8d74148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.208+0000 7f8d73fff700 1 --2- 172.21.15.187:0/1245489402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d74148c90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f8d6c009d50 tx=0x7f8d6c009290).stop 2020-11-20T12:19:37.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.208+0000 7f8d73fff700 1 -- 172.21.15.187:0/1245489402 shutdown_connections 2020-11-20T12:19:37.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.208+0000 7f8d73fff700 1 --2- 172.21.15.187:0/1245489402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d74148c90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.208+0000 7f8d73fff700 1 --2- 172.21.15.187:0/1245489402 >> v2:172.21.15.5:3300/0 conn(0x7f8d74145cc0 0x7f8d74066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.208+0000 7f8d73fff700 1 -- 172.21.15.187:0/1245489402 >> 172.21.15.187:0/1245489402 conn(0x7f8d740bab30 msgr2=0x7f8d740baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:37.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.208+0000 7f8d73fff700 1 -- 172.21.15.187:0/1245489402 shutdown_connections 2020-11-20T12:19:37.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.208+0000 7f8d73fff700 1 -- 172.21.15.187:0/1245489402 wait complete. 2020-11-20T12:19:37.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.210+0000 7f8d73fff700 1 Processor -- start 2020-11-20T12:19:37.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.210+0000 7f8d73fff700 1 -- start start 2020-11-20T12:19:37.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.211+0000 7f8d73fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d740683f0 0x7f8d740a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.211+0000 7f8d73fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d74145cc0 0x7f8d7415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.211+0000 7f8d73fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d741477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.211+0000 7f8d73fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7415b480 con 0x7f8d740683f0 2020-11-20T12:19:37.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.211+0000 7f8d73fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7415b720 con 0x7f8d74145cc0 2020-11-20T12:19:37.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.211+0000 7f8d73fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7415c0b0 con 0x7f8d741487e0 2020-11-20T12:19:37.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d6a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d74145cc0 0x7f8d7415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d6a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d74145cc0 0x7f8d7415bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52724/0 (socket says 172.21.15.187:52724) 2020-11-20T12:19:37.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d6a7fc700 1 -- 172.21.15.187:0/2958155758 learned_addr learned my addr 172.21.15.187:0/2958155758 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:37.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d72ffd700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d740683f0 0x7f8d740a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d737fe700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d741477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d6a7fc700 1 -- 172.21.15.187:0/2958155758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 msgr2=0x7f8d741477f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d6a7fc700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d741477f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d6a7fc700 1 -- 172.21.15.187:0/2958155758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d740683f0 msgr2=0x7f8d740a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d6a7fc700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d740683f0 0x7f8d740a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.212+0000 7f8d6a7fc700 1 -- 172.21.15.187:0/2958155758 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7415c270 con 0x7f8d74145cc0 2020-11-20T12:19:37.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.213+0000 7f8d6a7fc700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d74145cc0 0x7f8d7415bbc0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7f8d5c00c710 tx=0x7f8d5c007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:37.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.213+0000 7f8d6b7fe700 1 -- 172.21.15.187:0/2958155758 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c00e560 con 0x7f8d74145cc0 2020-11-20T12:19:37.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.213+0000 7f8d6b7fe700 1 -- 172.21.15.187:0/2958155758 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c00e6e0 con 0x7f8d74145cc0 2020-11-20T12:19:37.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.213+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7409b6e0 con 0x7f8d74145cc0 2020-11-20T12:19:37.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.213+0000 7f8d6b7fe700 1 -- 172.21.15.187:0/2958155758 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c00eaa0 con 0x7f8d74145cc0 2020-11-20T12:19:37.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.213+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d740afce0 con 0x7f8d74145cc0 2020-11-20T12:19:37.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.214+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d740afe70 con 0x7f8d74145cc0 2020-11-20T12:19:37.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.215+0000 7f8d6b7fe700 1 -- 172.21.15.187:0/2958155758 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8d5c019020 con 0x7f8d74145cc0 2020-11-20T12:19:37.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.216+0000 7f8d6b7fe700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d50056d20 0x7f8d500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.216+0000 7f8d6b7fe700 1 -- 172.21.15.187:0/2958155758 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8d5c06f280 con 0x7f8d74145cc0 2020-11-20T12:19:37.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.216+0000 7f8d72ffd700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d50056d20 0x7f8d500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.216+0000 7f8d72ffd700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d50056d20 0x7f8d500591e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f8d64006fd0 tx=0x7f8d64006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:37.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.221+0000 7f8d6b7fe700 1 -- 172.21.15.187:0/2958155758 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8d5c044a00 con 0x7f8d74145cc0 2020-11-20T12:19:37.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.412+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8d7415c270 con 0x7f8d74145cc0 2020-11-20T12:19:37.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.413+0000 7f8d6b7fe700 1 -- 172.21.15.187:0/2958155758 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v135) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8d7415c270 con 0x7f8d74145cc0 2020-11-20T12:19:37.417 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:19:37.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d50056d20 msgr2=0x7f8d500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d50056d20 0x7f8d500591e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f8d64006fd0 tx=0x7f8d64006df0).stop 2020-11-20T12:19:37.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d74145cc0 msgr2=0x7f8d7415bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d74145cc0 0x7f8d7415bbc0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7f8d5c00c710 tx=0x7f8d5c007c10).stop 2020-11-20T12:19:37.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 shutdown_connections 2020-11-20T12:19:37.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d50056d20 0x7f8d500591e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d740683f0 0x7f8d740a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d741487e0 0x7f8d741477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 --2- 172.21.15.187:0/2958155758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d74145cc0 0x7f8d7415bbc0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.415+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 >> 172.21.15.187:0/2958155758 conn(0x7f8d740bab30 msgr2=0x7f8d741450b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:37.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.416+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 shutdown_connections 2020-11-20T12:19:37.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.416+0000 7f8d73fff700 1 -- 172.21.15.187:0/2958155758 wait complete. 2020-11-20T12:19:37.442 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:19:37.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.681+0000 7f67e2764700 1 Processor -- start 2020-11-20T12:19:37.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.682+0000 7f67e2764700 1 -- start start 2020-11-20T12:19:37.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.683+0000 7f67e2764700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f67dc0bb840 0x7f67dc0655c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.683+0000 7f67e2764700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc0b7580 0x7f67dc065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.683+0000 7f67e2764700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc066b40 con 0x7f67dc0bbc20 2020-11-20T12:19:37.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.683+0000 7f67e2764700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67dc148980 con 0x7f67dc0bb840 2020-11-20T12:19:37.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.683+0000 7f67e2764700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc149530 con 0x7f67dc0b7580 2020-11-20T12:19:37.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.684+0000 7f67e1f63700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc0b7580 0x7f67dc065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.684+0000 7f67e1f63700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc0b7580 0x7f67dc065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:53620/0 (socket says 172.21.15.187:53620) 2020-11-20T12:19:37.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.684+0000 7f67e1f63700 1 -- 172.21.15.187:0/4028327840 learned_addr learned my addr 172.21.15.187:0/4028327840 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:37.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.684+0000 7f67e1762700 1 --2- 172.21.15.187:0/4028327840 >> v2:172.21.15.5:3300/0 conn(0x7f67dc0bb840 0x7f67dc0655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.685+0000 7f67e1762700 1 -- 172.21.15.187:0/4028327840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc0b7580 msgr2=0x7f67dc065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:37.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.685+0000 7f67e1762700 1 --2- 172.21.15.187:0/4028327840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc0b7580 0x7f67dc065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.685+0000 7f67e1762700 1 -- 172.21.15.187:0/4028327840 >> v1:172.21.15.5:6789/0 conn(0x7f67dc0bbc20 legacy=0x7f67dc148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.685+0000 7f67e1762700 1 -- 172.21.15.187:0/4028327840 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc1496a0 con 0x7f67dc0bb840 2020-11-20T12:19:37.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.685+0000 7f67e1762700 1 --2- 172.21.15.187:0/4028327840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0bb840 0x7f67dc0655c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f67d800b9a0 tx=0x7f67d8006df0).ready entity=mon.0 client_cookie=fe360c1f23d77680 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:37.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.686+0000 7f67cbfff700 1 -- 172.21.15.187:0/4028327840 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d80097b0 con 0x7f67dc0bb840 2020-11-20T12:19:37.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.686+0000 7f67cbfff700 1 -- 172.21.15.187:0/4028327840 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d8009910 con 0x7f67dc0bb840 2020-11-20T12:19:37.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.686+0000 7f67cbfff700 1 -- 172.21.15.187:0/4028327840 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8009c90 con 0x7f67dc0bb840 2020-11-20T12:19:37.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.687+0000 7f67e2764700 1 -- 172.21.15.187:0/4028327840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0bb840 msgr2=0x7f67dc0655c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.687+0000 7f67e2764700 1 --2- 172.21.15.187:0/4028327840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0bb840 0x7f67dc0655c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f67d800b9a0 tx=0x7f67d8006df0).stop 2020-11-20T12:19:37.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.687+0000 7f67e2764700 1 -- 172.21.15.187:0/4028327840 shutdown_connections 2020-11-20T12:19:37.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.687+0000 7f67e2764700 1 --2- 172.21.15.187:0/4028327840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc0b7580 0x7f67dc065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.687+0000 7f67e2764700 1 --2- 172.21.15.187:0/4028327840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0bb840 0x7f67dc0655c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.687+0000 7f67e2764700 1 -- 172.21.15.187:0/4028327840 >> 172.21.15.187:0/4028327840 conn(0x7f67dc060bf0 msgr2=0x7f67dc0baf70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:37.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.687+0000 7f67e2764700 1 -- 172.21.15.187:0/4028327840 shutdown_connections 2020-11-20T12:19:37.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.687+0000 7f67e2764700 1 -- 172.21.15.187:0/4028327840 wait complete. 2020-11-20T12:19:37.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.688+0000 7f67e2764700 1 Processor -- start 2020-11-20T12:19:37.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.689+0000 7f67e2764700 1 -- start start 2020-11-20T12:19:37.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.689+0000 7f67e2764700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0b7580 0x7f67dc15bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.689+0000 7f67e2764700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67dc0bb840 0x7f67dc0b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.689+0000 7f67e2764700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc146ae0 0x7f67dc14a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.689+0000 7f67e2764700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc157d70 con 0x7f67dc0b7580 2020-11-20T12:19:37.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.689+0000 7f67e2764700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc144360 con 0x7f67dc0bb840 2020-11-20T12:19:37.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.689+0000 7f67e2764700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc1499b0 con 0x7f67dc146ae0 2020-11-20T12:19:37.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.690+0000 7f67e0f61700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67dc0bb840 0x7f67dc0b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.690+0000 7f67e0f61700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67dc0bb840 0x7f67dc0b96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52738/0 (socket says 172.21.15.187:52738) 2020-11-20T12:19:37.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.690+0000 7f67e0f61700 1 -- 172.21.15.187:0/4068126532 learned_addr learned my addr 172.21.15.187:0/4068126532 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:37.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.690+0000 7f67e1762700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0b7580 0x7f67dc15bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.690+0000 7f67e1f63700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc146ae0 0x7f67dc14a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67e0f61700 1 -- 172.21.15.187:0/4068126532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc146ae0 msgr2=0x7f67dc14a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67e0f61700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc146ae0 0x7f67dc14a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67e0f61700 1 -- 172.21.15.187:0/4068126532 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0b7580 msgr2=0x7f67dc15bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67e0f61700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0b7580 0x7f67dc15bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67e0f61700 1 -- 172.21.15.187:0/4068126532 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc149c50 con 0x7f67dc0bb840 2020-11-20T12:19:37.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67e0f61700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67dc0bb840 0x7f67dc0b96e0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7f67cc00c710 tx=0x7f67cc007bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:37.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67c97fa700 1 -- 172.21.15.187:0/4068126532 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc00e4a0 con 0x7f67dc0bb840 2020-11-20T12:19:37.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67c97fa700 1 -- 172.21.15.187:0/4068126532 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67cc00e600 con 0x7f67dc0bb840 2020-11-20T12:19:37.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67c97fa700 1 -- 172.21.15.187:0/4068126532 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc00e9c0 con 0x7f67dc0bb840 2020-11-20T12:19:37.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.691+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc0b9bd0 con 0x7f67dc0bb840 2020-11-20T12:19:37.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.692+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc14a820 con 0x7f67dc0bb840 2020-11-20T12:19:37.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.693+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67dc15c340 con 0x7f67dc0bb840 2020-11-20T12:19:37.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.693+0000 7f67c97fa700 1 -- 172.21.15.187:0/4068126532 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f67cc019030 con 0x7f67dc0bb840 2020-11-20T12:19:37.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.694+0000 7f67c97fa700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f67b4056d20 0x7f67b40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:37.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.694+0000 7f67c97fa700 1 -- 172.21.15.187:0/4068126532 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f67cc06f0b0 con 0x7f67dc0bb840 2020-11-20T12:19:37.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.694+0000 7f67e1762700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f67b4056d20 0x7f67b40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:37.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.695+0000 7f67e1762700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f67b4056d20 0x7f67b40591e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f67d8006e30 tx=0x7f67d8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:37.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.698+0000 7f67c97fa700 1 -- 172.21.15.187:0/4068126532 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f67cc044830 con 0x7f67dc0bb840 2020-11-20T12:19:37.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.889+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f67dc149de0 con 0x7f67dc0bb840 2020-11-20T12:19:37.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.890+0000 7f67c97fa700 1 -- 172.21.15.187:0/4068126532 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) v1 ==== 84+0+143 (secure 0 0 0) 0x7f67cc047e50 con 0x7f67dc0bb840 2020-11-20T12:19:37.894 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:37.894 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:19:37.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:37.895 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:19:37.895 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:19:37.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:19:37.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:19:37.896 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:19:37.896 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:19:37.896 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:19:37.896 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:19:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.892+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f67b4056d20 msgr2=0x7f67b40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.892+0000 7f67e2764700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f67b4056d20 0x7f67b40591e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f67d8006e30 tx=0x7f67d8015000).stop 2020-11-20T12:19:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.892+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67dc0bb840 msgr2=0x7f67dc0b96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.892+0000 7f67e2764700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67dc0bb840 0x7f67dc0b96e0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7f67cc00c710 tx=0x7f67cc007bd0).stop 2020-11-20T12:19:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.893+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 shutdown_connections 2020-11-20T12:19:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.893+0000 7f67e2764700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f67b4056d20 0x7f67b40591e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.893+0000 7f67e2764700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67dc0b7580 0x7f67dc15bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.893+0000 7f67e2764700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67dc146ae0 0x7f67dc14a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.893+0000 7f67e2764700 1 --2- 172.21.15.187:0/4068126532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67dc0bb840 0x7f67dc0b96e0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.893+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 >> 172.21.15.187:0/4068126532 conn(0x7f67dc060bf0 msgr2=0x7f67dc0655c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.893+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 shutdown_connections 2020-11-20T12:19:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:37.893+0000 7f67e2764700 1 -- 172.21.15.187:0/4068126532 wait complete. 2020-11-20T12:19:37.921 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:19:37.922 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:19:38.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.170+0000 7f85bf59e700 1 Processor -- start 2020-11-20T12:19:38.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.171+0000 7f85bf59e700 1 -- start start 2020-11-20T12:19:38.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.172+0000 7f85bf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c00b6650 0x7f85c00afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.172+0000 7f85bf59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f85c0066f70 0x7f85c00aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.172+0000 7f85bf59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00b1300 con 0x7f85c01484d0 2020-11-20T12:19:38.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.172+0000 7f85bf59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c0067730 con 0x7f85c0066f70 2020-11-20T12:19:38.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.172+0000 7f85bf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00b15a0 con 0x7f85c00b6650 2020-11-20T12:19:38.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.173+0000 7f85b5d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c00b6650 0x7f85c00afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.173+0000 7f85bed9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f85c0066f70 0x7f85c00aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.173+0000 7f85bed9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f85c0066f70 0x7f85c00aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:45010/0 (socket says 172.21.15.187:45010) 2020-11-20T12:19:38.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.173+0000 7f85b5d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c00b6650 0x7f85c00afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:53632/0 (socket says 172.21.15.187:53632) 2020-11-20T12:19:38.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.173+0000 7f85bed9d700 1 -- 172.21.15.187:0/2050804312 learned_addr learned my addr 172.21.15.187:0/2050804312 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:38.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.174+0000 7f85b5d9b700 1 -- 172.21.15.187:0/2050804312 >> v2:172.21.15.5:3300/0 conn(0x7f85c0066f70 msgr2=0x7f85c00aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:38.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.174+0000 7f85b5d9b700 1 --2- 172.21.15.187:0/2050804312 >> v2:172.21.15.5:3300/0 conn(0x7f85c0066f70 0x7f85c00aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.174+0000 7f85b5d9b700 1 -- 172.21.15.187:0/2050804312 >> v1:172.21.15.5:6789/0 conn(0x7f85c01484d0 legacy=0x7f85c0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.174+0000 7f85b5d9b700 1 -- 172.21.15.187:0/2050804312 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00b1840 con 0x7f85c00b6650 2020-11-20T12:19:38.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.174+0000 7f85b5d9b700 1 --2- 172.21.15.187:0/2050804312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c00b6650 0x7f85c00afa90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f85a800b9a0 tx=0x7f85a8006df0).ready entity=mon.2 client_cookie=184bc2ddca9f0dc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:38.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.175+0000 7f85bdd9b700 1 -- 172.21.15.187:0/2050804312 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a80097b0 con 0x7f85c00b6650 2020-11-20T12:19:38.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.175+0000 7f85bdd9b700 1 -- 172.21.15.187:0/2050804312 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a8009910 con 0x7f85c00b6650 2020-11-20T12:19:38.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.175+0000 7f85bdd9b700 1 -- 172.21.15.187:0/2050804312 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8009c90 con 0x7f85c00b6650 2020-11-20T12:19:38.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.175+0000 7f85bf59e700 1 -- 172.21.15.187:0/2050804312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c00b6650 msgr2=0x7f85c00afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.175+0000 7f85bf59e700 1 --2- 172.21.15.187:0/2050804312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c00b6650 0x7f85c00afa90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f85a800b9a0 tx=0x7f85a8006df0).stop 2020-11-20T12:19:38.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.176+0000 7f85bf59e700 1 -- 172.21.15.187:0/2050804312 shutdown_connections 2020-11-20T12:19:38.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.176+0000 7f85bf59e700 1 --2- 172.21.15.187:0/2050804312 >> v2:172.21.15.5:3300/0 conn(0x7f85c0066f70 0x7f85c00aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.176+0000 7f85bf59e700 1 --2- 172.21.15.187:0/2050804312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c00b6650 0x7f85c00afa90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.176+0000 7f85bf59e700 1 -- 172.21.15.187:0/2050804312 >> 172.21.15.187:0/2050804312 conn(0x7f85c00bab30 msgr2=0x7f85c00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:38.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.176+0000 7f85bf59e700 1 -- 172.21.15.187:0/2050804312 shutdown_connections 2020-11-20T12:19:38.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.176+0000 7f85bf59e700 1 -- 172.21.15.187:0/2050804312 wait complete. 2020-11-20T12:19:38.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.178+0000 7f85bf59e700 1 Processor -- start 2020-11-20T12:19:38.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.178+0000 7f85bf59e700 1 -- start start 2020-11-20T12:19:38.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85bf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c0066f70 0x7f85c009dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85bf59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c00b6650 0x7f85c0065350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85bf59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c01484d0 0x7f85c0135080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85bf59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c009c130 con 0x7f85c01484d0 2020-11-20T12:19:38.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85bf59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00b20a0 con 0x7f85c00b6650 2020-11-20T12:19:38.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85bf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00b2340 con 0x7f85c0066f70 2020-11-20T12:19:38.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85b5d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c00b6650 0x7f85c0065350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85b5d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c00b6650 0x7f85c0065350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52750/0 (socket says 172.21.15.187:52750) 2020-11-20T12:19:38.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.179+0000 7f85b5d9b700 1 -- 172.21.15.187:0/1419288010 learned_addr learned my addr 172.21.15.187:0/1419288010 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:38.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85bed9d700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c01484d0 0x7f85c0135080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85be59c700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c0066f70 0x7f85c009dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85b5d9b700 1 -- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c0066f70 msgr2=0x7f85c009dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85b5d9b700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c0066f70 0x7f85c009dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85b5d9b700 1 -- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c01484d0 msgr2=0x7f85c0135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85b5d9b700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c01484d0 0x7f85c0135080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85b5d9b700 1 -- 172.21.15.187:0/1419288010 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00b1d90 con 0x7f85c00b6650 2020-11-20T12:19:38.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85be59c700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c0066f70 0x7f85c009dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:38.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85bed9d700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c01484d0 0x7f85c0135080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:38.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85b5d9b700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c00b6650 0x7f85c0065350 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 rx=0x7f85a8006e30 tx=0x7f85a8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:38.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.180+0000 7f85b6ffd700 1 -- 172.21.15.187:0/1419288010 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8011d50 con 0x7f85c00b6650 2020-11-20T12:19:38.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.181+0000 7f85b6ffd700 1 -- 172.21.15.187:0/1419288010 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a801e460 con 0x7f85c00b6650 2020-11-20T12:19:38.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.181+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c0065840 con 0x7f85c00b6650 2020-11-20T12:19:38.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.181+0000 7f85b6ffd700 1 -- 172.21.15.187:0/1419288010 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a801e7e0 con 0x7f85c00b6650 2020-11-20T12:19:38.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.181+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c00b4780 con 0x7f85c00b6650 2020-11-20T12:19:38.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.182+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c009c270 con 0x7f85c00b6650 2020-11-20T12:19:38.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.182+0000 7f85b6ffd700 1 -- 172.21.15.187:0/1419288010 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f85a8025030 con 0x7f85c00b6650 2020-11-20T12:19:38.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.183+0000 7f85b6ffd700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f859c056d20 0x7f859c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.183+0000 7f85b6ffd700 1 -- 172.21.15.187:0/1419288010 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f85a80713c0 con 0x7f85c00b6650 2020-11-20T12:19:38.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.183+0000 7f85be59c700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f859c056d20 0x7f859c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.184+0000 7f85be59c700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f859c056d20 0x7f859c0591e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f85c00a05a0 tx=0x7f85b0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:38.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.187+0000 7f85b6ffd700 1 -- 172.21.15.187:0/1419288010 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f85a8046bf0 con 0x7f85c00b6650 2020-11-20T12:19:38.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.373+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f85c0061a20 con 0x7f85c00b6650 2020-11-20T12:19:38.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.474+0000 7f85b6ffd700 1 -- 172.21.15.187:0/1419288010 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v136) v1 ==== 133+0+0 (secure 0 0 0) 0x7f85a8023020 con 0x7f85c00b6650 2020-11-20T12:19:38.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.477+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f859c056d20 msgr2=0x7f859c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.477+0000 7f85bf59e700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f859c056d20 0x7f859c0591e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f85c00a05a0 tx=0x7f85b0009040).stop 2020-11-20T12:19:38.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.477+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c00b6650 msgr2=0x7f85c0065350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.477+0000 7f85bf59e700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c00b6650 0x7f85c0065350 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 rx=0x7f85a8006e30 tx=0x7f85a8015000).stop 2020-11-20T12:19:38.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.478+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 shutdown_connections 2020-11-20T12:19:38.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.478+0000 7f85bf59e700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f859c056d20 0x7f859c0591e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.478+0000 7f85bf59e700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c01484d0 0x7f85c0135080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.478+0000 7f85bf59e700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85c0066f70 0x7f85c009dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.478+0000 7f85bf59e700 1 --2- 172.21.15.187:0/1419288010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c00b6650 0x7f85c0065350 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.478+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 >> 172.21.15.187:0/1419288010 conn(0x7f85c00bab30 msgr2=0x7f85c013e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:38.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.479+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 shutdown_connections 2020-11-20T12:19:38.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.479+0000 7f85bf59e700 1 -- 172.21.15.187:0/1419288010 wait complete. 2020-11-20T12:19:38.487 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:19:38.513 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:19:38.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.749+0000 7fad1759e700 1 Processor -- start 2020-11-20T12:19:38.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.749+0000 7fad1759e700 1 -- start start 2020-11-20T12:19:38.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.750+0000 7fad1759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 0x7fad181482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.757 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.751+0000 7fad1759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad18144b80 con 0x7fad180b5f50 2020-11-20T12:19:38.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.751+0000 7fad1759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad181489e0 con 0x7fad180b57b0 2020-11-20T12:19:38.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.751+0000 7fad1759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad181487c0 con 0x7fad18068230 2020-11-20T12:19:38.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.751+0000 7fad1659c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fad180b57b0 0x7fad18062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49364/0 (socket says 172.21.15.187:49364) 2020-11-20T12:19:38.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.751+0000 7fad1659c700 1 -- 172.21.15.187:0/3497101286 learned_addr learned my addr 172.21.15.187:0/3497101286 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:38.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.751+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078270582 0 0) 0x7fad181489e0 con 0x7fad180b57b0 2020-11-20T12:19:38.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.751+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf8003240 con 0x7fad180b57b0 2020-11-20T12:19:38.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.751+0000 7fad0dd9b700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 0x7fad181482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3392178550 0 0) 0x7facf8003240 con 0x7fad180b57b0 2020-11-20T12:19:38.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 msgr2=0x7fad181482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:38.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 0x7fad181482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 >> v1:172.21.15.5:6789/0 conn(0x7fad180b5f50 legacy=0x7fad180b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:38.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad18148c80 con 0x7fad180b57b0 2020-11-20T12:19:38.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fad08001520 con 0x7fad180b57b0 2020-11-20T12:19:38.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 >> v1:172.21.15.187:6789/0 conn(0x7fad180b57b0 legacy=0x7fad18062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7facf8003730 0x7facf8003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 0x7facf80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf80049d0 0x7facf8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad181487c0 con 0x7facf8003730 2020-11-20T12:19:38.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80041b0 con 0x7facf80049d0 2020-11-20T12:19:38.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf800c840 con 0x7fad18068230 2020-11-20T12:19:38.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.752+0000 7fad16d9d700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf80049d0 0x7facf8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad1659c700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7facf8003730 0x7facf8003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad16d9d700 1 -- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 msgr2=0x7facf80044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:38.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad0dd9b700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 0x7facf80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad16d9d700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 0x7facf80044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad16d9d700 1 -- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7facf8003730 msgr2=0x7facf8003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:38.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad16d9d700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7facf8003730 0x7facf8003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad16d9d700 1 -- 172.21.15.187:0/3497101286 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8003240 con 0x7facf80049d0 2020-11-20T12:19:38.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad0dd9b700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad18068230 0x7facf80044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:38.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.753+0000 7fad16d9d700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf80049d0 0x7facf8004e50 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7fad10002db0 tx=0x7fad10007f10).ready entity=mon.1 client_cookie=e8eb569241577d41 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:38.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10002430 con 0x7facf80049d0 2020-11-20T12:19:38.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1000ca20 con 0x7facf80049d0 2020-11-20T12:19:38.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad15d9b700 1 -- 172.21.15.187:0/3497101286 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1000cde0 con 0x7facf80049d0 2020-11-20T12:19:38.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad1759e700 1 -- 172.21.15.187:0/3497101286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf80049d0 msgr2=0x7facf8004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad1759e700 1 --2- 172.21.15.187:0/3497101286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf80049d0 0x7facf8004e50 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7fad10002db0 tx=0x7fad10007f10).stop 2020-11-20T12:19:38.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad16d9d700 1 -- 172.21.15.187:0/3497101286 reap_dead start 2020-11-20T12:19:38.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad1759e700 1 -- 172.21.15.187:0/3497101286 shutdown_connections 2020-11-20T12:19:38.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad1759e700 1 -- 172.21.15.187:0/3497101286 >> 172.21.15.187:0/3497101286 conn(0x7fad180bab30 msgr2=0x7fad180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:38.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.754+0000 7fad1759e700 1 -- 172.21.15.187:0/3497101286 shutdown_connections 2020-11-20T12:19:38.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.755+0000 7fad1759e700 1 -- 172.21.15.187:0/3497101286 wait complete. 2020-11-20T12:19:38.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.756+0000 7fad1759e700 1 Processor -- start 2020-11-20T12:19:38.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.756+0000 7fad1759e700 1 -- start start 2020-11-20T12:19:38.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.757+0000 7fad1759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad18158490 0x7fad181575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.757+0000 7fad1759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad18158870 0x7fad18157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.757+0000 7fad1759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad181479b0 0x7fad1815b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.757+0000 7fad1759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad180b3290 con 0x7fad18158490 2020-11-20T12:19:38.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.757+0000 7fad1759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad18147f30 con 0x7fad18158870 2020-11-20T12:19:38.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.757+0000 7fad1759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad18158d60 con 0x7fad181479b0 2020-11-20T12:19:38.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.757+0000 7fad0dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad18158870 0x7fad18157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad18158870 0x7fad18157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52772/0 (socket says 172.21.15.187:52772) 2020-11-20T12:19:38.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0dd9b700 1 -- 172.21.15.187:0/1385205343 learned_addr learned my addr 172.21.15.187:0/1385205343 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:38.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad1659c700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad18158490 0x7fad181575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0dd9b700 1 -- 172.21.15.187:0/1385205343 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad181479b0 msgr2=0x7fad1815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0dd9b700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad181479b0 0x7fad1815b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:38.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0dd9b700 1 -- 172.21.15.187:0/1385205343 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad18158490 msgr2=0x7fad181575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:38.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0dd9b700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad18158490 0x7fad181575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:38.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0dd9b700 1 -- 172.21.15.187:0/1385205343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad18159000 con 0x7fad18158870 2020-11-20T12:19:38.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0dd9b700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad18158870 0x7fad18157ae0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 rx=0x7fad000097a0 tx=0x7fad00008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:38.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0effd700 1 -- 172.21.15.187:0/1385205343 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000133f0 con 0x7fad18158870 2020-11-20T12:19:38.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad0effd700 1 -- 172.21.15.187:0/1385205343 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad00013550 con 0x7fad18158870 2020-11-20T12:19:38.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.758+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad18157fd0 con 0x7fad18158870 2020-11-20T12:19:38.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.759+0000 7fad0effd700 1 -- 172.21.15.187:0/1385205343 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000138d0 con 0x7fad18158870 2020-11-20T12:19:38.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.759+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad181458f0 con 0x7fad18158870 2020-11-20T12:19:38.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.760+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad18147d90 con 0x7fad18158870 2020-11-20T12:19:38.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.760+0000 7fad0effd700 1 -- 172.21.15.187:0/1385205343 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fad00019020 con 0x7fad18158870 2020-11-20T12:19:38.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.761+0000 7fad0effd700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d20 0x7facf40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:38.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.761+0000 7fad0effd700 1 -- 172.21.15.187:0/1385205343 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fad0006f940 con 0x7fad18158870 2020-11-20T12:19:38.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.761+0000 7fad1659c700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d20 0x7facf40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:38.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.762+0000 7fad1659c700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d20 0x7facf40591e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fad080013f0 tx=0x7fad08000f20).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:38.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.765+0000 7fad0effd700 1 -- 172.21.15.187:0/1385205343 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fad18147d90 con 0x7fad18158870 2020-11-20T12:19:38.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:38.956+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fad18147d90 con 0x7fad18158870 2020-11-20T12:19:39.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.487+0000 7fad0effd700 1 -- 172.21.15.187:0/1385205343 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v138) v1 ==== 125+0+0 (secure 0 0 0) 0x7fad18147d90 con 0x7fad18158870 2020-11-20T12:19:39.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d20 msgr2=0x7facf40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:39.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d20 0x7facf40591e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fad080013f0 tx=0x7fad08000f20).stop 2020-11-20T12:19:39.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad18158870 msgr2=0x7fad18157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:39.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad18158870 0x7fad18157ae0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 rx=0x7fad000097a0 tx=0x7fad00008040).stop 2020-11-20T12:19:39.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 shutdown_connections 2020-11-20T12:19:39.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d20 0x7facf40591e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:39.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad18158490 0x7fad181575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:39.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad181479b0 0x7fad1815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:39.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 --2- 172.21.15.187:0/1385205343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad18158870 0x7fad18157ae0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:39.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.496+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 >> 172.21.15.187:0/1385205343 conn(0x7fad180bab30 msgr2=0x7fad180b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:39.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.497+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 shutdown_connections 2020-11-20T12:19:39.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.497+0000 7fad1759e700 1 -- 172.21.15.187:0/1385205343 wait complete. 2020-11-20T12:19:39.507 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:19:39.534 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:19:39.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.767+0000 7f910b4a7700 1 Processor -- start 2020-11-20T12:19:39.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.767+0000 7f910b4a7700 1 -- start start 2020-11-20T12:19:39.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.768+0000 7f910b4a7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f91040b6650 0x7f91040afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:39.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.768+0000 7f910b4a7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9104066f70 0x7f91040aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:39.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.768+0000 7f910b4a7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91040b1300 con 0x7f91041484d0 2020-11-20T12:19:39.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.768+0000 7f910b4a7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9104067730 con 0x7f91040b6650 2020-11-20T12:19:39.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.768+0000 7f910b4a7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91040b15a0 con 0x7f9104066f70 2020-11-20T12:19:39.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.769+0000 7f910a4a5700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f91041484d0 0x7f9104061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49384/0 (socket says 172.21.15.187:49384) 2020-11-20T12:19:39.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.769+0000 7f910a4a5700 1 -- 172.21.15.187:0/3149585231 learned_addr learned my addr 172.21.15.187:0/3149585231 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:39.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.769+0000 7f9109ca4700 1 --2- 172.21.15.187:0/3149585231 >> v2:172.21.15.187:3300/0 conn(0x7f91040b6650 0x7f91040afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:39.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.769+0000 7f910aca6700 1 --2- 172.21.15.187:0/3149585231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9104066f70 0x7f91040aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:39.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.770+0000 7f91094a3700 1 -- 172.21.15.187:0/3149585231 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863752298 0 0) 0x7f91040b1300 con 0x7f91041484d0 2020-11-20T12:19:39.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.770+0000 7f91094a3700 1 -- 172.21.15.187:0/3149585231 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90ec003240 con 0x7f91041484d0 2020-11-20T12:19:39.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.770+0000 7f9109ca4700 1 -- 172.21.15.187:0/3149585231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9104066f70 msgr2=0x7f91040aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:39.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.770+0000 7f9109ca4700 1 --2- 172.21.15.187:0/3149585231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9104066f70 0x7f91040aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:39.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.770+0000 7f9109ca4700 1 -- 172.21.15.187:0/3149585231 >> v1:172.21.15.187:6789/0 conn(0x7f91041484d0 legacy=0x7f9104061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:39.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.770+0000 7f9109ca4700 1 -- 172.21.15.187:0/3149585231 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91040b1840 con 0x7f91040b6650 2020-11-20T12:19:39.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.770+0000 7f9109ca4700 1 --2- 172.21.15.187:0/3149585231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91040b6650 0x7f91040afa90 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7f90f400b9a0 tx=0x7f90f4006df0).ready entity=mon.1 client_cookie=587709bffc6772d7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:39.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f91094a3700 1 -- 172.21.15.187:0/3149585231 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4009680 con 0x7f91040b6650 2020-11-20T12:19:39.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f91094a3700 1 -- 172.21.15.187:0/3149585231 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f40097e0 con 0x7f91040b6650 2020-11-20T12:19:39.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f91094a3700 1 -- 172.21.15.187:0/3149585231 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4009b60 con 0x7f91040b6650 2020-11-20T12:19:39.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f910b4a7700 1 -- 172.21.15.187:0/3149585231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91040b6650 msgr2=0x7f91040afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:39.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f910b4a7700 1 --2- 172.21.15.187:0/3149585231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91040b6650 0x7f91040afa90 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7f90f400b9a0 tx=0x7f90f4006df0).stop 2020-11-20T12:19:39.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f910b4a7700 1 -- 172.21.15.187:0/3149585231 shutdown_connections 2020-11-20T12:19:39.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f910b4a7700 1 --2- 172.21.15.187:0/3149585231 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9104066f70 0x7f91040aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:39.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f910b4a7700 1 --2- 172.21.15.187:0/3149585231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91040b6650 0x7f91040afa90 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:39.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f910b4a7700 1 -- 172.21.15.187:0/3149585231 >> 172.21.15.187:0/3149585231 conn(0x7f91040bab30 msgr2=0x7f91040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:39.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f910b4a7700 1 -- 172.21.15.187:0/3149585231 shutdown_connections 2020-11-20T12:19:39.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.771+0000 7f910b4a7700 1 -- 172.21.15.187:0/3149585231 wait complete. 2020-11-20T12:19:39.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.773+0000 7f910b4a7700 1 Processor -- start 2020-11-20T12:19:39.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.773+0000 7f910b4a7700 1 -- start start 2020-11-20T12:19:39.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910b4a7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9104066f70 0x7f91040b9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:39.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910b4a7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91040b6650 0x7f91041435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:39.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910b4a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91041484d0 0x7f9104143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:39.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910b4a7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91040b26f0 con 0x7f9104066f70 2020-11-20T12:19:39.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910b4a7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91040b2990 con 0x7f91041484d0 2020-11-20T12:19:39.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910b4a7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91040b2c80 con 0x7f91040b6650 2020-11-20T12:19:39.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910aca6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91041484d0 0x7f9104143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:39.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910aca6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91041484d0 0x7f9104143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52788/0 (socket says 172.21.15.187:52788) 2020-11-20T12:19:39.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.774+0000 7f910aca6700 1 -- 172.21.15.187:0/4272983597 learned_addr learned my addr 172.21.15.187:0/4272983597 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:39.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f9109ca4700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91040b6650 0x7f91041435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:39.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f910a4a5700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9104066f70 0x7f91040b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:39.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f910aca6700 1 -- 172.21.15.187:0/4272983597 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91040b6650 msgr2=0x7f91041435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:39.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f910aca6700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91040b6650 0x7f91041435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:39.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f910aca6700 1 -- 172.21.15.187:0/4272983597 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9104066f70 msgr2=0x7f91040b9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:39.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f910aca6700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9104066f70 0x7f91040b9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f910aca6700 1 -- 172.21.15.187:0/4272983597 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91040b2410 con 0x7f91041484d0 2020-11-20T12:19:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f910aca6700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91041484d0 0x7f9104143ab0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 rx=0x7f90fc00ef50 tx=0x7f90fc0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f90f27fc700 1 -- 172.21.15.187:0/4272983597 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc00ccc0 con 0x7f91041484d0 2020-11-20T12:19:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f90f27fc700 1 -- 172.21.15.187:0/4272983597 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc00ce20 con 0x7f91041484d0 2020-11-20T12:19:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91040b9610 con 0x7f91041484d0 2020-11-20T12:19:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.775+0000 7f90f27fc700 1 -- 172.21.15.187:0/4272983597 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc012620 con 0x7f91041484d0 2020-11-20T12:19:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.776+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910414a340 con 0x7f91041484d0 2020-11-20T12:19:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.777+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91040b2e40 con 0x7f91041484d0 2020-11-20T12:19:39.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.777+0000 7f90f27fc700 1 -- 172.21.15.187:0/4272983597 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f90fc022070 con 0x7f91041484d0 2020-11-20T12:19:39.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.778+0000 7f90f27fc700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f90dc056d20 0x7f90dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:39.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.778+0000 7f90f27fc700 1 -- 172.21.15.187:0/4272983597 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f90fc070880 con 0x7f91041484d0 2020-11-20T12:19:39.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.778+0000 7f910a4a5700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f90dc056d20 0x7f90dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:39.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.779+0000 7f910a4a5700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f90dc056d20 0x7f90dc0591e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f91000055a0 tx=0x7f9100009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:39.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:39.782+0000 7f90f27fc700 1 -- 172.21.15.187:0/4272983597 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f90fc045c30 con 0x7f91041484d0 2020-11-20T12:19:40.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.029+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f9104061a20 con 0x7f91041484d0 2020-11-20T12:19:40.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.044+0000 7f90f27fc700 1 -- 172.21.15.187:0/4272983597 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 27 and data pool 28 v135) v1 ==== 189+0+0 (secure 0 0 0) 0x7f90fc0160d0 con 0x7f91041484d0 2020-11-20T12:19:40.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.051+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f90dc056d20 msgr2=0x7f90dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.051+0000 7f910b4a7700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f90dc056d20 0x7f90dc0591e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f91000055a0 tx=0x7f9100009040).stop 2020-11-20T12:19:40.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.051+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91041484d0 msgr2=0x7f9104143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.051+0000 7f910b4a7700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91041484d0 0x7f9104143ab0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 rx=0x7f90fc00ef50 tx=0x7f90fc0091d0).stop 2020-11-20T12:19:40.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.054+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 shutdown_connections 2020-11-20T12:19:40.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.054+0000 7f910b4a7700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f90dc056d20 0x7f90dc0591e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.054+0000 7f910b4a7700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9104066f70 0x7f91040b9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.054+0000 7f910b4a7700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91041484d0 0x7f9104143ab0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.054+0000 7f910b4a7700 1 --2- 172.21.15.187:0/4272983597 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f91040b6650 0x7f91041435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.060 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.054+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 >> 172.21.15.187:0/4272983597 conn(0x7f91040bab30 msgr2=0x7f91040ba0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:40.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.056+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 shutdown_connections 2020-11-20T12:19:40.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.057+0000 7f910b4a7700 1 -- 172.21.15.187:0/4272983597 wait complete. 2020-11-20T12:19:40.062 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 27 and data pool 28 2020-11-20T12:19:40.086 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:19:40.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.295+0000 7f0845607700 1 Processor -- start 2020-11-20T12:19:40.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.296+0000 7f0845607700 1 -- start start 2020-11-20T12:19:40.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.297+0000 7f0845607700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f08401484d0 0x7f0840061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.297+0000 7f0845607700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08400b1300 con 0x7f0840066f70 2020-11-20T12:19:40.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.297+0000 7f0845607700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08400b15a0 con 0x7f08400622c0 2020-11-20T12:19:40.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.297+0000 7f0845607700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0840067730 con 0x7f08401484d0 2020-11-20T12:19:40.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.297+0000 7f083ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f08401484d0 0x7f0840061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.297+0000 7f08377fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f08400622c0 0x7f08400b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49408/0 (socket says 172.21.15.187:49408) 2020-11-20T12:19:40.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.298+0000 7f08377fe700 1 -- 172.21.15.187:0/685798384 learned_addr learned my addr 172.21.15.187:0/685798384 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:40.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.298+0000 7f083ffff700 1 -- 172.21.15.187:0/685798384 >> v1:172.21.15.187:6789/0 conn(0x7f08400622c0 legacy=0x7f08400b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.298+0000 7f083ffff700 1 -- 172.21.15.187:0/685798384 >> v1:172.21.15.5:6789/0 conn(0x7f0840066f70 legacy=0x7f08400b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:40.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.298+0000 7f083ffff700 1 -- 172.21.15.187:0/685798384 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08400b1840 con 0x7f08401484d0 2020-11-20T12:19:40.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.298+0000 7f083ffff700 1 --2- 172.21.15.187:0/685798384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f0840061dd0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f083000b9a0 tx=0x7f0830006df0).ready entity=mon.1 client_cookie=ae83cf7f654af98e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:40.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.299+0000 7f083f7fe700 1 -- 172.21.15.187:0/685798384 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830009770 con 0x7f08401484d0 2020-11-20T12:19:40.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.299+0000 7f083f7fe700 1 -- 172.21.15.187:0/685798384 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08300098d0 con 0x7f08401484d0 2020-11-20T12:19:40.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.299+0000 7f083f7fe700 1 -- 172.21.15.187:0/685798384 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830009c50 con 0x7f08401484d0 2020-11-20T12:19:40.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.299+0000 7f0845607700 1 -- 172.21.15.187:0/685798384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 msgr2=0x7f0840061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.299+0000 7f0845607700 1 --2- 172.21.15.187:0/685798384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f0840061dd0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f083000b9a0 tx=0x7f0830006df0).stop 2020-11-20T12:19:40.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.299+0000 7f0845607700 1 -- 172.21.15.187:0/685798384 shutdown_connections 2020-11-20T12:19:40.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.299+0000 7f0845607700 1 --2- 172.21.15.187:0/685798384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f0840061dd0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.299+0000 7f0845607700 1 -- 172.21.15.187:0/685798384 >> 172.21.15.187:0/685798384 conn(0x7f08400bab30 msgr2=0x7f08400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:40.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.300+0000 7f0845607700 1 -- 172.21.15.187:0/685798384 shutdown_connections 2020-11-20T12:19:40.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.300+0000 7f0845607700 1 -- 172.21.15.187:0/685798384 wait complete. 2020-11-20T12:19:40.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.301+0000 7f0845607700 1 Processor -- start 2020-11-20T12:19:40.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.301+0000 7f0845607700 1 -- start start 2020-11-20T12:19:40.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.302+0000 7f0845607700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08400622c0 0x7f08401421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.302+0000 7f0845607700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0840066f70 0x7f0840149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.302+0000 7f0845607700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f08400b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.302+0000 7f0845607700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400b9960 con 0x7f08400622c0 2020-11-20T12:19:40.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.302+0000 7f0845607700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0840148cb0 con 0x7f08401484d0 2020-11-20T12:19:40.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.302+0000 7f0845607700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0840148f50 con 0x7f0840066f70 2020-11-20T12:19:40.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.302+0000 7f0844e06700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f08400b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f0844e06700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f08400b76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52810/0 (socket says 172.21.15.187:52810) 2020-11-20T12:19:40.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f0844e06700 1 -- 172.21.15.187:0/3465857941 learned_addr learned my addr 172.21.15.187:0/3465857941 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:40.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f0844e06700 1 -- 172.21.15.187:0/3465857941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0840066f70 msgr2=0x7f0840149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f0844e06700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0840066f70 0x7f0840149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f0844e06700 1 -- 172.21.15.187:0/3465857941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08400622c0 msgr2=0x7f08401421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f083ffff700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08400622c0 0x7f08401421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f0844e06700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08400622c0 0x7f08401421b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f0844e06700 1 -- 172.21.15.187:0/3465857941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08401491f0 con 0x7f08401484d0 2020-11-20T12:19:40.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.303+0000 7f0844e06700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f08400b76e0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f083800c860 tx=0x7f083800cc20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:40.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.304+0000 7f083cff9700 1 -- 172.21.15.187:0/3465857941 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083800a7b0 con 0x7f08401484d0 2020-11-20T12:19:40.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.304+0000 7f083cff9700 1 -- 172.21.15.187:0/3465857941 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083800a930 con 0x7f08401484d0 2020-11-20T12:19:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.304+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08401426a0 con 0x7f08401484d0 2020-11-20T12:19:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.304+0000 7f083cff9700 1 -- 172.21.15.187:0/3465857941 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083800acf0 con 0x7f08401484d0 2020-11-20T12:19:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.304+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084014a170 con 0x7f08401484d0 2020-11-20T12:19:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.305+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0840149380 con 0x7f08401484d0 2020-11-20T12:19:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.305+0000 7f083cff9700 1 -- 172.21.15.187:0/3465857941 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0838010070 con 0x7f08401484d0 2020-11-20T12:19:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.306+0000 7f083cff9700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f081c056d50 0x7f081c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.306+0000 7f083cff9700 1 -- 172.21.15.187:0/3465857941 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f083806fd90 con 0x7f08401484d0 2020-11-20T12:19:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.306+0000 7f083ffff700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f081c056d50 0x7f081c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.307+0000 7f083ffff700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f081c056d50 0x7f081c059210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f083000bf10 tx=0x7f0830015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:40.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.310+0000 7f083cff9700 1 -- 172.21.15.187:0/3465857941 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f08380451e0 con 0x7f08401484d0 2020-11-20T12:19:40.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.497+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0840061a20 con 0x7f08401484d0 2020-11-20T12:19:40.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.498+0000 7f083cff9700 1 -- 172.21.15.187:0/3465857941 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16805 (secure 0 0 0) 0x7f0838048800 con 0x7f08401484d0 2020-11-20T12:19:40.502 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:40.502 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":139,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:19:40.032529+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:19:38.454721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2020-11-20T12:19:39.038543+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:19:40.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.500+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f081c056d50 msgr2=0x7f081c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.500+0000 7f0845607700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f081c056d50 0x7f081c059210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f083000bf10 tx=0x7f0830015000).stop 2020-11-20T12:19:40.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.500+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 msgr2=0x7f08400b76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.500+0000 7f0845607700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f08400b76e0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f083800c860 tx=0x7f083800cc20).stop 2020-11-20T12:19:40.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.501+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 shutdown_connections 2020-11-20T12:19:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.501+0000 7f0845607700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f081c056d50 0x7f081c059210 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.501+0000 7f0845607700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08400622c0 0x7f08401421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.501+0000 7f0845607700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08401484d0 0x7f08400b76e0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.501+0000 7f0845607700 1 --2- 172.21.15.187:0/3465857941 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0840066f70 0x7f0840149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:40.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.501+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 >> 172.21.15.187:0/3465857941 conn(0x7f08400bab30 msgr2=0x7f08400ba170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:40.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.501+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 shutdown_connections 2020-11-20T12:19:40.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.501+0000 7f0845607700 1 -- 172.21.15.187:0/3465857941 wait complete. 2020-11-20T12:19:40.528 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:19:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.764+0000 7f7dec1d7700 1 Processor -- start 2020-11-20T12:19:40.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.764+0000 7f7dec1d7700 1 -- start start 2020-11-20T12:19:40.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.765+0000 7f7dec1d7700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7de4065e10 0x7f7de4144470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.766+0000 7f7dec1d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4099ce0 0x7f7de4144960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.766+0000 7f7dec1d7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de40b5df0 con 0x7f7de40661f0 2020-11-20T12:19:40.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.766+0000 7f7dec1d7700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de40b72f0 con 0x7f7de4065e10 2020-11-20T12:19:40.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.766+0000 7f7dec1d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de409a440 con 0x7f7de4099ce0 2020-11-20T12:19:40.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.766+0000 7f7de9f73700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7de4065e10 0x7f7de4144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.766+0000 7f7de9772700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7de40661f0 0x7f7de4061de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49420/0 (socket says 172.21.15.187:49420) 2020-11-20T12:19:40.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.766+0000 7f7de9772700 1 -- 172.21.15.187:0/824683832 learned_addr learned my addr 172.21.15.187:0/824683832 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:40.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.766+0000 7f7dea774700 1 --2- 172.21.15.187:0/824683832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4099ce0 0x7f7de4144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.767+0000 7f7de9f73700 1 -- 172.21.15.187:0/824683832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4099ce0 msgr2=0x7f7de4144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:40.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.767+0000 7f7de9f73700 1 --2- 172.21.15.187:0/824683832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4099ce0 0x7f7de4144960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.767+0000 7f7de9f73700 1 -- 172.21.15.187:0/824683832 >> v1:172.21.15.187:6789/0 conn(0x7f7de40661f0 legacy=0x7f7de4061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.767+0000 7f7de9f73700 1 -- 172.21.15.187:0/824683832 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de409a1a0 con 0x7f7de4065e10 2020-11-20T12:19:40.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.767+0000 7f7de9f73700 1 --2- 172.21.15.187:0/824683832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4065e10 0x7f7de4144470 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 rx=0x7f7de40b7430 tx=0x7f7de0006df0).ready entity=mon.1 client_cookie=adda8ae796435cf0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:40.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.767+0000 7f7de8f71700 1 -- 172.21.15.187:0/824683832 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de0009840 con 0x7f7de4065e10 2020-11-20T12:19:40.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.767+0000 7f7de8f71700 1 -- 172.21.15.187:0/824683832 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de00099a0 con 0x7f7de4065e10 2020-11-20T12:19:40.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.767+0000 7f7de8f71700 1 -- 172.21.15.187:0/824683832 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de0009d20 con 0x7f7de4065e10 2020-11-20T12:19:40.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.768+0000 7f7dec1d7700 1 -- 172.21.15.187:0/824683832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4065e10 msgr2=0x7f7de4144470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.768+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/824683832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4065e10 0x7f7de4144470 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 rx=0x7f7de40b7430 tx=0x7f7de0006df0).stop 2020-11-20T12:19:40.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.768+0000 7f7dec1d7700 1 -- 172.21.15.187:0/824683832 shutdown_connections 2020-11-20T12:19:40.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.768+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/824683832 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4099ce0 0x7f7de4144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.768+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/824683832 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4065e10 0x7f7de4144470 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.768+0000 7f7dec1d7700 1 -- 172.21.15.187:0/824683832 >> 172.21.15.187:0/824683832 conn(0x7f7de40bacd0 msgr2=0x7f7de40bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:40.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.768+0000 7f7dec1d7700 1 -- 172.21.15.187:0/824683832 shutdown_connections 2020-11-20T12:19:40.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.768+0000 7f7dec1d7700 1 -- 172.21.15.187:0/824683832 wait complete. 2020-11-20T12:19:40.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.770+0000 7f7dec1d7700 1 Processor -- start 2020-11-20T12:19:40.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.770+0000 7f7dec1d7700 1 -- start start 2020-11-20T12:19:40.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dec1d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4065e10 0x7f7de4158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dec1d7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de40661f0 0x7f7de4157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dec1d7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4099ce0 0x7f7de4142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dec1d7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de41581f0 con 0x7f7de40661f0 2020-11-20T12:19:40.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dec1d7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de4158420 con 0x7f7de4099ce0 2020-11-20T12:19:40.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dec1d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de41586c0 con 0x7f7de4065e10 2020-11-20T12:19:40.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dea774700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4099ce0 0x7f7de4142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dea774700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4099ce0 0x7f7de4142430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52824/0 (socket says 172.21.15.187:52824) 2020-11-20T12:19:40.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7dea774700 1 -- 172.21.15.187:0/2214788864 learned_addr learned my addr 172.21.15.187:0/2214788864 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:40.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.771+0000 7f7de9772700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de40661f0 0x7f7de4157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7de9f73700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4065e10 0x7f7de4158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dea774700 1 -- 172.21.15.187:0/2214788864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4065e10 msgr2=0x7f7de4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dea774700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4065e10 0x7f7de4158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dea774700 1 -- 172.21.15.187:0/2214788864 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de40661f0 msgr2=0x7f7de4157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dea774700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de40661f0 0x7f7de4157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dea774700 1 -- 172.21.15.187:0/2214788864 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de4159050 con 0x7f7de4099ce0 2020-11-20T12:19:40.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dea774700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4099ce0 0x7f7de4142430 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f7ddc00db10 tx=0x7f7ddc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:40.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dd1ffb700 1 -- 172.21.15.187:0/2214788864 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc009900 con 0x7f7de4099ce0 2020-11-20T12:19:40.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dd1ffb700 1 -- 172.21.15.187:0/2214788864 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc009a60 con 0x7f7de4099ce0 2020-11-20T12:19:40.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de4157e30 con 0x7f7de4099ce0 2020-11-20T12:19:40.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.772+0000 7f7dd1ffb700 1 -- 172.21.15.187:0/2214788864 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc009e00 con 0x7f7de4099ce0 2020-11-20T12:19:40.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.773+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de4142ae0 con 0x7f7de4099ce0 2020-11-20T12:19:40.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.774+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de4158020 con 0x7f7de4099ce0 2020-11-20T12:19:40.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.774+0000 7f7dd1ffb700 1 -- 172.21.15.187:0/2214788864 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7ddc012630 con 0x7f7de4099ce0 2020-11-20T12:19:40.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.775+0000 7f7dd1ffb700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dbc056d20 0x7f7dbc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:40.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.775+0000 7f7dd1ffb700 1 -- 172.21.15.187:0/2214788864 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7ddc06ff50 con 0x7f7de4099ce0 2020-11-20T12:19:40.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.775+0000 7f7de9f73700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dbc056d20 0x7f7dbc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:40.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.776+0000 7f7de9f73700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dbc056d20 0x7f7dbc0591e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f7de000c010 tx=0x7f7de0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:40.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.779+0000 7f7dd1ffb700 1 -- 172.21.15.187:0/2214788864 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7ddc0453a0 con 0x7f7de4099ce0 2020-11-20T12:19:40.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.967+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7de4063600 con 0x7f7de4099ce0 2020-11-20T12:19:40.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.968+0000 7f7dd1ffb700 1 -- 172.21.15.187:0/2214788864 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16805 (secure 0 0 0) 0x7f7ddc0489c0 con 0x7f7de4099ce0 2020-11-20T12:19:40.972 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:40.973 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":139,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:19:40.032529+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:19:38.454721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2020-11-20T12:19:39.038543+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:19:40.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.970+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dbc056d20 msgr2=0x7f7dbc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.970+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dbc056d20 0x7f7dbc0591e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f7de000c010 tx=0x7f7de0015000).stop 2020-11-20T12:19:40.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4099ce0 msgr2=0x7f7de4142430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:40.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4099ce0 0x7f7de4142430 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f7ddc00db10 tx=0x7f7ddc008040).stop 2020-11-20T12:19:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 shutdown_connections 2020-11-20T12:19:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7dbc056d20 0x7f7dbc0591e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7de4099ce0 0x7f7de4142430 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7de4065e10 0x7f7de4158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 --2- 172.21.15.187:0/2214788864 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7de40661f0 0x7f7de4157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:40.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 >> 172.21.15.187:0/2214788864 conn(0x7f7de40bacd0 msgr2=0x7f7de4144440 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:40.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 shutdown_connections 2020-11-20T12:19:40.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:40.971+0000 7f7dec1d7700 1 -- 172.21.15.187:0/2214788864 wait complete. 2020-11-20T12:19:40.997 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:19:41.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.232+0000 7fa06bfff700 1 Processor -- start 2020-11-20T12:19:41.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.233+0000 7fa06bfff700 1 -- start start 2020-11-20T12:19:41.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.234+0000 7fa06bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa06c14d270 0x7fa06c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:41.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.234+0000 7fa06bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06c0658a0 con 0x7fa06c0b1380 2020-11-20T12:19:41.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.234+0000 7fa06bfff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06c150390 con 0x7fa06c0b1760 2020-11-20T12:19:41.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.234+0000 7fa06bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06c1511e0 con 0x7fa06c14d270 2020-11-20T12:19:41.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.234+0000 7fa0627fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa06c0b1760 0x7fa06c065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49436/0 (socket says 172.21.15.187:49436) 2020-11-20T12:19:41.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.234+0000 7fa0627fc700 1 -- 172.21.15.187:0/1380543175 learned_addr learned my addr 172.21.15.187:0/1380543175 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:41.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.235+0000 7fa06b7fe700 1 --2- 172.21.15.187:0/1380543175 >> v2:172.21.15.187:3300/0 conn(0x7fa06c14d270 0x7fa06c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:41.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.235+0000 7fa06a7fc700 1 -- 172.21.15.187:0/1380543175 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516662025 0 0) 0x7fa06c150390 con 0x7fa06c0b1760 2020-11-20T12:19:41.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.235+0000 7fa06a7fc700 1 -- 172.21.15.187:0/1380543175 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa04c003240 con 0x7fa06c0b1760 2020-11-20T12:19:41.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.235+0000 7fa06b7fe700 1 -- 172.21.15.187:0/1380543175 >> v1:172.21.15.187:6789/0 conn(0x7fa06c0b1760 legacy=0x7fa06c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:41.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.235+0000 7fa06b7fe700 1 -- 172.21.15.187:0/1380543175 >> v1:172.21.15.5:6789/0 conn(0x7fa06c0b1380 legacy=0x7fa06c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:41.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.236+0000 7fa06b7fe700 1 -- 172.21.15.187:0/1380543175 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c150630 con 0x7fa06c14d270 2020-11-20T12:19:41.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.236+0000 7fa06b7fe700 1 --2- 172.21.15.187:0/1380543175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c14d270 0x7fa06c1573a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7fa064009c90 tx=0x7fa0640091d0).ready entity=mon.1 client_cookie=a520eb1193d09e21 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:41.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.236+0000 7fa06a7fc700 1 -- 172.21.15.187:0/1380543175 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06400ba70 con 0x7fa06c14d270 2020-11-20T12:19:41.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.236+0000 7fa06a7fc700 1 -- 172.21.15.187:0/1380543175 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06400bbd0 con 0x7fa06c14d270 2020-11-20T12:19:41.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.236+0000 7fa06a7fc700 1 -- 172.21.15.187:0/1380543175 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064011400 con 0x7fa06c14d270 2020-11-20T12:19:41.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.237+0000 7fa06bfff700 1 -- 172.21.15.187:0/1380543175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c14d270 msgr2=0x7fa06c1573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:41.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.237+0000 7fa06bfff700 1 --2- 172.21.15.187:0/1380543175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c14d270 0x7fa06c1573a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7fa064009c90 tx=0x7fa0640091d0).stop 2020-11-20T12:19:41.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.237+0000 7fa06bfff700 1 -- 172.21.15.187:0/1380543175 shutdown_connections 2020-11-20T12:19:41.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.237+0000 7fa06bfff700 1 --2- 172.21.15.187:0/1380543175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c14d270 0x7fa06c1573a0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:41.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.237+0000 7fa06bfff700 1 -- 172.21.15.187:0/1380543175 >> 172.21.15.187:0/1380543175 conn(0x7fa06c0bab30 msgr2=0x7fa06c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:41.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.237+0000 7fa06bfff700 1 -- 172.21.15.187:0/1380543175 shutdown_connections 2020-11-20T12:19:41.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.237+0000 7fa06bfff700 1 -- 172.21.15.187:0/1380543175 wait complete. 2020-11-20T12:19:41.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.238+0000 7fa06bfff700 1 Processor -- start 2020-11-20T12:19:41.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.239+0000 7fa06bfff700 1 -- start start 2020-11-20T12:19:41.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.239+0000 7fa06bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c0b1380 0x7fa06c0b9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:41.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.239+0000 7fa06bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa06c0b1760 0x7fa06c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:41.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.239+0000 7fa06bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa06c14d270 0x7fa06c09b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:41.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.239+0000 7fa06bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c160610 con 0x7fa06c0b1760 2020-11-20T12:19:41.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.239+0000 7fa06bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c0b4210 con 0x7fa06c0b1380 2020-11-20T12:19:41.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.239+0000 7fa06bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c15c230 con 0x7fa06c14d270 2020-11-20T12:19:41.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c0b1380 0x7fa06c0b9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:41.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c0b1380 0x7fa06c0b9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52834/0 (socket says 172.21.15.187:52834) 2020-11-20T12:19:41.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 -- 172.21.15.187:0/2134547037 learned_addr learned my addr 172.21.15.187:0/2134547037 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:41.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 -- 172.21.15.187:0/2134547037 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa06c14d270 msgr2=0x7fa06c09b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:41.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa06c14d270 0x7fa06c09b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:41.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 -- 172.21.15.187:0/2134547037 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa06c0b1760 msgr2=0x7fa06c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:41.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa06c0b1760 0x7fa06c15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:41.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 -- 172.21.15.187:0/2134547037 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c15c4d0 con 0x7fa06c0b1380 2020-11-20T12:19:41.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.240+0000 7fa06affd700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c0b1380 0x7fa06c0b9fa0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7fa05c00c6e0 tx=0x7fa05c007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:41.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.241+0000 7fa0637fe700 1 -- 172.21.15.187:0/2134547037 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c007020 con 0x7fa06c0b1380 2020-11-20T12:19:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.241+0000 7fa0637fe700 1 -- 172.21.15.187:0/2134547037 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c007180 con 0x7fa06c0b1380 2020-11-20T12:19:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.241+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06c0ba490 con 0x7fa06c0b1380 2020-11-20T12:19:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.241+0000 7fa0637fe700 1 -- 172.21.15.187:0/2134547037 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c00e560 con 0x7fa06c0b1380 2020-11-20T12:19:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.241+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06c15d0a0 con 0x7fa06c0b1380 2020-11-20T12:19:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.242+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06c15c660 con 0x7fa06c0b1380 2020-11-20T12:19:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.243+0000 7fa0637fe700 1 -- 172.21.15.187:0/2134547037 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa05c018020 con 0x7fa06c0b1380 2020-11-20T12:19:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.243+0000 7fa0637fe700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa048056d20 0x7fa0480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.243+0000 7fa0637fe700 1 -- 172.21.15.187:0/2134547037 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa05c06f940 con 0x7fa06c0b1380 2020-11-20T12:19:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.244+0000 7fa0627fc700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa048056d20 0x7fa0480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:41.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.244+0000 7fa0627fc700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa048056d20 0x7fa0480591e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7fa0540055a0 tx=0x7fa0540076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:41.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.247+0000 7fa0637fe700 1 -- 172.21.15.187:0/2134547037 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa05c044d90 con 0x7fa06c0b1380 2020-11-20T12:19:41.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:41.495+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa06c061a20 con 0x7fa06c0b1380 2020-11-20T12:19:42.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.514+0000 7fa0637fe700 1 -- 172.21.15.187:0/2134547037 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v138) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa05c016030 con 0x7fa06c0b1380 2020-11-20T12:19:42.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa048056d20 msgr2=0x7fa0480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:42.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa048056d20 0x7fa0480591e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7fa0540055a0 tx=0x7fa0540076a0).stop 2020-11-20T12:19:42.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c0b1380 msgr2=0x7fa06c0b9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:42.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c0b1380 0x7fa06c0b9fa0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7fa05c00c6e0 tx=0x7fa05c007c50).stop 2020-11-20T12:19:42.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 shutdown_connections 2020-11-20T12:19:42.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa048056d20 0x7fa0480591e0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:42.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa06c14d270 0x7fa06c09b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:42.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c0b1380 0x7fa06c0b9fa0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:42.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 --2- 172.21.15.187:0/2134547037 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa06c0b1760 0x7fa06c15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:42.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.517+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 >> 172.21.15.187:0/2134547037 conn(0x7fa06c0bab30 msgr2=0x7fa06c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:42.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.518+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 shutdown_connections 2020-11-20T12:19:42.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.518+0000 7fa06bfff700 1 -- 172.21.15.187:0/2134547037 wait complete. 2020-11-20T12:19:42.544 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:42.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.792+0000 7f5b98d16700 1 Processor -- start 2020-11-20T12:19:42.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.792+0000 7f5b98d16700 1 -- start start 2020-11-20T12:19:42.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.794+0000 7f5b98d16700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5b940b1760 0x7f5b94065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:42.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.794+0000 7f5b98d16700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b94150390 con 0x7f5b94065680 2020-11-20T12:19:42.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.794+0000 7f5b98d16700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b94150630 con 0x7f5b940b1380 2020-11-20T12:19:42.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.794+0000 7f5b98d16700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b941511e0 con 0x7f5b940b1760 2020-11-20T12:19:42.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.794+0000 7f5b92ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5b940b1760 0x7f5b94065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:42.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.794+0000 7f5b937fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5b940b1380 0x7f5b94150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49448/0 (socket says 172.21.15.187:49448) 2020-11-20T12:19:42.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.794+0000 7f5b937fe700 1 -- 172.21.15.187:0/214068729 learned_addr learned my addr 172.21.15.187:0/214068729 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:42.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.795+0000 7f5b92ffd700 1 -- 172.21.15.187:0/214068729 >> v1:172.21.15.187:6789/0 conn(0x7f5b940b1380 legacy=0x7f5b94150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:42.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.795+0000 7f5b927fc700 1 -- 172.21.15.187:0/214068729 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318517866 0 0) 0x7f5b94150630 con 0x7f5b940b1380 2020-11-20T12:19:42.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.795+0000 7f5b92ffd700 1 -- 172.21.15.187:0/214068729 >> v1:172.21.15.5:6789/0 conn(0x7f5b94065680 legacy=0x7f5b9414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:42.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.795+0000 7f5b92ffd700 1 -- 172.21.15.187:0/214068729 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94068510 con 0x7f5b940b1760 2020-11-20T12:19:42.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.795+0000 7f5b92ffd700 1 --2- 172.21.15.187:0/214068729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940b1760 0x7f5b94065190 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7f5b8800b9a0 tx=0x7f5b88006df0).ready entity=mon.1 client_cookie=685092d6626fe8c0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:42.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b927fc700 1 -- 172.21.15.187:0/214068729 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b880097b0 con 0x7f5b940b1760 2020-11-20T12:19:42.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b927fc700 1 -- 172.21.15.187:0/214068729 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b88009910 con 0x7f5b940b1760 2020-11-20T12:19:42.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b927fc700 1 -- 172.21.15.187:0/214068729 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88009c90 con 0x7f5b940b1760 2020-11-20T12:19:42.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b98d16700 1 -- 172.21.15.187:0/214068729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940b1760 msgr2=0x7f5b94065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:42.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b98d16700 1 --2- 172.21.15.187:0/214068729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940b1760 0x7f5b94065190 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7f5b8800b9a0 tx=0x7f5b88006df0).stop 2020-11-20T12:19:42.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b98d16700 1 -- 172.21.15.187:0/214068729 shutdown_connections 2020-11-20T12:19:42.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b98d16700 1 --2- 172.21.15.187:0/214068729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940b1760 0x7f5b94065190 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:42.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b98d16700 1 -- 172.21.15.187:0/214068729 >> 172.21.15.187:0/214068729 conn(0x7f5b940bab30 msgr2=0x7f5b940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:42.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.796+0000 7f5b98d16700 1 -- 172.21.15.187:0/214068729 shutdown_connections 2020-11-20T12:19:42.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.797+0000 7f5b98d16700 1 -- 172.21.15.187:0/214068729 wait complete. 2020-11-20T12:19:42.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.798+0000 7f5b98d16700 1 Processor -- start 2020-11-20T12:19:42.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.798+0000 7f5b98d16700 1 -- start start 2020-11-20T12:19:42.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.799+0000 7f5b98d16700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b940b1380 0x7f5b9409fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:42.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.799+0000 7f5b98d16700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b940b1760 0x7f5b9413f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:42.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.799+0000 7f5b98d16700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940a0580 0x7f5b940b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:42.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.799+0000 7f5b98d16700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9409e4b0 con 0x7f5b940b1760 2020-11-20T12:19:42.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.799+0000 7f5b98d16700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9413fcf0 con 0x7f5b940a0580 2020-11-20T12:19:42.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.799+0000 7f5b98d16700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940b5c90 con 0x7f5b940b1380 2020-11-20T12:19:42.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.799+0000 7f5b93fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940a0580 0x7f5b940b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:42.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b93fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940a0580 0x7f5b940b6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52852/0 (socket says 172.21.15.187:52852) 2020-11-20T12:19:42.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b93fff700 1 -- 172.21.15.187:0/841540927 learned_addr learned my addr 172.21.15.187:0/841540927 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:42.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b937fe700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b940b1380 0x7f5b9409fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:42.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b92ffd700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b940b1760 0x7f5b9413f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:42.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b93fff700 1 -- 172.21.15.187:0/841540927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b940b1380 msgr2=0x7f5b9409fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:42.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b93fff700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b940b1380 0x7f5b9409fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:42.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b93fff700 1 -- 172.21.15.187:0/841540927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b940b1760 msgr2=0x7f5b9413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:42.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b93fff700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b940b1760 0x7f5b9413f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:42.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b93fff700 1 -- 172.21.15.187:0/841540927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b940b5f30 con 0x7f5b940a0580 2020-11-20T12:19:42.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.800+0000 7f5b93fff700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940a0580 0x7f5b940b6260 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 rx=0x7f5b8c00ca60 tx=0x7f5b8c007f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:42.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.801+0000 7f5b737fe700 1 -- 172.21.15.187:0/841540927 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c00a7d0 con 0x7f5b940a0580 2020-11-20T12:19:42.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.801+0000 7f5b737fe700 1 -- 172.21.15.187:0/841540927 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c00a930 con 0x7f5b940a0580 2020-11-20T12:19:42.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.801+0000 7f5b737fe700 1 -- 172.21.15.187:0/841540927 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c00acb0 con 0x7f5b940a0580 2020-11-20T12:19:42.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.801+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b940b6750 con 0x7f5b940a0580 2020-11-20T12:19:42.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.801+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b940b5320 con 0x7f5b940a0580 2020-11-20T12:19:42.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.802+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b94061a20 con 0x7f5b940a0580 2020-11-20T12:19:42.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.803+0000 7f5b737fe700 1 -- 172.21.15.187:0/841540927 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5b8c019030 con 0x7f5b940a0580 2020-11-20T12:19:42.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.804+0000 7f5b737fe700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b6c056d20 0x7f5b6c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:42.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.804+0000 7f5b737fe700 1 -- 172.21.15.187:0/841540927 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5b8c06f750 con 0x7f5b940a0580 2020-11-20T12:19:42.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.804+0000 7f5b937fe700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b6c056d20 0x7f5b6c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:42.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.805+0000 7f5b937fe700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b6c056d20 0x7f5b6c0591e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f5b840011a0 tx=0x7f5b84009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:42.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:42.808+0000 7f5b737fe700 1 -- 172.21.15.187:0/841540927 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5b8c043fd0 con 0x7f5b940a0580 2020-11-20T12:19:43.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.058+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b940b8470 con 0x7f5b940a0580 2020-11-20T12:19:43.062 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.058+0000 7f5b737fe700 1 -- 172.21.15.187:0/841540927 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2142 (secure 0 0 0) 0x7f5b8c044c50 con 0x7f5b940a0580 2020-11-20T12:19:43.062 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:43.063 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":138,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1775892026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1775892026},{"type":"v1","addr":"172.21.15.5:6833","nonce":1775892026}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":134}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:42.498695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9343":{"gid":9343,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1878527443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1878527443},{"type":"v1","addr":"172.21.15.187:6835","nonce":1878527443}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:19:43.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b6c056d20 msgr2=0x7f5b6c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b6c056d20 0x7f5b6c0591e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f5b840011a0 tx=0x7f5b84009040).stop 2020-11-20T12:19:43.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940a0580 msgr2=0x7f5b940b6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940a0580 0x7f5b940b6260 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 rx=0x7f5b8c00ca60 tx=0x7f5b8c007f40).stop 2020-11-20T12:19:43.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 shutdown_connections 2020-11-20T12:19:43.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b6c056d20 0x7f5b6c0591e0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b940a0580 0x7f5b940b6260 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b940b1380 0x7f5b9409fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 --2- 172.21.15.187:0/841540927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b940b1760 0x7f5b9413f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.061+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 >> 172.21.15.187:0/841540927 conn(0x7f5b940bab30 msgr2=0x7f5b940b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:43.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.062+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 shutdown_connections 2020-11-20T12:19:43.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.062+0000 7f5b98d16700 1 -- 172.21.15.187:0/841540927 wait complete. 2020-11-20T12:19:43.068 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 138 2020-11-20T12:19:43.087 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:19:43.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.308+0000 7fb55da43700 1 Processor -- start 2020-11-20T12:19:43.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.308+0000 7fb55da43700 1 -- start start 2020-11-20T12:19:43.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55da43700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 0x7fb558150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55da43700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb5580b1760 0x7fb558065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55da43700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb558065680 0x7fb558061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55da43700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb558150390 con 0x7fb5580b1760 2020-11-20T12:19:43.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55da43700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb558150630 con 0x7fb558065680 2020-11-20T12:19:43.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55da43700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb558068510 con 0x7fb5580b1380 2020-11-20T12:19:43.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55d242700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb558065680 0x7fb558061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55d242700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb558065680 0x7fb558061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52858/0 (socket says 172.21.15.187:52858) 2020-11-20T12:19:43.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.310+0000 7fb55d242700 1 -- 172.21.15.187:0/846029536 learned_addr learned my addr 172.21.15.187:0/846029536 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:43.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.311+0000 7fb557fff700 1 --2- 172.21.15.187:0/846029536 >> v2:172.21.15.5:3300/0 conn(0x7fb5580b1760 0x7fb558065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.311+0000 7fb55ca41700 1 --2- 172.21.15.187:0/846029536 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 0x7fb558150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.311+0000 7fb55d242700 1 -- 172.21.15.187:0/846029536 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 msgr2=0x7fb558150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:43.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.311+0000 7fb55d242700 1 --2- 172.21.15.187:0/846029536 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 0x7fb558150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.311+0000 7fb55d242700 1 -- 172.21.15.187:0/846029536 >> v2:172.21.15.5:3300/0 conn(0x7fb5580b1760 msgr2=0x7fb558065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:43.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.311+0000 7fb55d242700 1 --2- 172.21.15.187:0/846029536 >> v2:172.21.15.5:3300/0 conn(0x7fb5580b1760 0x7fb558065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.311+0000 7fb55d242700 1 -- 172.21.15.187:0/846029536 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb558150fc0 con 0x7fb558065680 2020-11-20T12:19:43.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.311+0000 7fb55d242700 1 --2- 172.21.15.187:0/846029536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb558065680 0x7fb558061dd0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 rx=0x7fb5580b50e0 tx=0x7fb550009290).ready entity=mon.1 client_cookie=29f0f568ed3c59ac server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:43.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb5577fe700 1 -- 172.21.15.187:0/846029536 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55000bbc0 con 0x7fb558065680 2020-11-20T12:19:43.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb5577fe700 1 -- 172.21.15.187:0/846029536 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55000bd20 con 0x7fb558065680 2020-11-20T12:19:43.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb5577fe700 1 -- 172.21.15.187:0/846029536 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550011610 con 0x7fb558065680 2020-11-20T12:19:43.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 -- 172.21.15.187:0/846029536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb558065680 msgr2=0x7fb558061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 --2- 172.21.15.187:0/846029536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb558065680 0x7fb558061dd0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 rx=0x7fb5580b50e0 tx=0x7fb550009290).stop 2020-11-20T12:19:43.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 -- 172.21.15.187:0/846029536 shutdown_connections 2020-11-20T12:19:43.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 --2- 172.21.15.187:0/846029536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb558065680 0x7fb558061dd0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 --2- 172.21.15.187:0/846029536 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 0x7fb558150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 --2- 172.21.15.187:0/846029536 >> v2:172.21.15.5:3300/0 conn(0x7fb5580b1760 0x7fb558065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 -- 172.21.15.187:0/846029536 >> 172.21.15.187:0/846029536 conn(0x7fb5580bab30 msgr2=0x7fb5580baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:43.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 -- 172.21.15.187:0/846029536 shutdown_connections 2020-11-20T12:19:43.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.312+0000 7fb55da43700 1 -- 172.21.15.187:0/846029536 wait complete. 2020-11-20T12:19:43.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.314+0000 7fb55da43700 1 Processor -- start 2020-11-20T12:19:43.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.314+0000 7fb55da43700 1 -- start start 2020-11-20T12:19:43.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55da43700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 0x7fb5580b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55da43700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5580b1760 0x7fb5580b9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55da43700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55809ee70 0x7fb55809b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55da43700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580a05e0 con 0x7fb5580b1760 2020-11-20T12:19:43.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55da43700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55809dbf0 con 0x7fb55809ee70 2020-11-20T12:19:43.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55da43700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580b2e30 con 0x7fb5580b1380 2020-11-20T12:19:43.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55d242700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55809ee70 0x7fb55809b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55d242700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55809ee70 0x7fb55809b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52866/0 (socket says 172.21.15.187:52866) 2020-11-20T12:19:43.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.315+0000 7fb55d242700 1 -- 172.21.15.187:0/1341283539 learned_addr learned my addr 172.21.15.187:0/1341283539 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:43.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb55d242700 1 -- 172.21.15.187:0/1341283539 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 msgr2=0x7fb5580b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb55d242700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 0x7fb5580b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:43.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb55d242700 1 -- 172.21.15.187:0/1341283539 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5580b1760 msgr2=0x7fb5580b9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb557fff700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5580b1760 0x7fb5580b9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb55d242700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5580b1760 0x7fb5580b9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb55d242700 1 -- 172.21.15.187:0/1341283539 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580ba4d0 con 0x7fb55809ee70 2020-11-20T12:19:43.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb55d242700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55809ee70 0x7fb55809b130 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7fb5500092d0 tx=0x7fb550016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:43.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb554ff9700 1 -- 172.21.15.187:0/1341283539 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55000bbc0 con 0x7fb55809ee70 2020-11-20T12:19:43.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb554ff9700 1 -- 172.21.15.187:0/1341283539 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55000be90 con 0x7fb55809ee70 2020-11-20T12:19:43.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb554ff9700 1 -- 172.21.15.187:0/1341283539 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550010e50 con 0x7fb55809ee70 2020-11-20T12:19:43.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55809b620 con 0x7fb55809ee70 2020-11-20T12:19:43.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.316+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55814a760 con 0x7fb55809ee70 2020-11-20T12:19:43.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.318+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5580ba660 con 0x7fb55809ee70 2020-11-20T12:19:43.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.318+0000 7fb554ff9700 1 -- 172.21.15.187:0/1341283539 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb550017070 con 0x7fb55809ee70 2020-11-20T12:19:43.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.319+0000 7fb554ff9700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb530056d20 0x7fb5300591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.319+0000 7fb554ff9700 1 -- 172.21.15.187:0/1341283539 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb550072160 con 0x7fb55809ee70 2020-11-20T12:19:43.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.319+0000 7fb55ca41700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb530056d20 0x7fb5300591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.320+0000 7fb55ca41700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb530056d20 0x7fb5300591e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fb55809ff60 tx=0x7fb54c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:43.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.323+0000 7fb554ff9700 1 -- 172.21.15.187:0/1341283539 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb5580ba660 con 0x7fb55809ee70 2020-11-20T12:19:43.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.507+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5580ba660 con 0x7fb55809ee70 2020-11-20T12:19:43.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.510+0000 7fb554ff9700 1 -- 172.21.15.187:0/1341283539 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16792 (secure 0 0 0) 0x7fb5580ba660 con 0x7fb55809ee70 2020-11-20T12:19:43.514 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:43.514 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":140,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:19:41.038077+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:19:38.454721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2020-11-20T12:19:39.038543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:19:43.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.512+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb530056d20 msgr2=0x7fb5300591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.512+0000 7fb55da43700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb530056d20 0x7fb5300591e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fb55809ff60 tx=0x7fb54c008040).stop 2020-11-20T12:19:43.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.512+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55809ee70 msgr2=0x7fb55809b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.512+0000 7fb55da43700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55809ee70 0x7fb55809b130 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7fb5500092d0 tx=0x7fb550016000).stop 2020-11-20T12:19:43.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.513+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 shutdown_connections 2020-11-20T12:19:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.513+0000 7fb55da43700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb530056d20 0x7fb5300591e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.513+0000 7fb55da43700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55809ee70 0x7fb55809b130 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.513+0000 7fb55da43700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5580b1380 0x7fb5580b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.513+0000 7fb55da43700 1 --2- 172.21.15.187:0/1341283539 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5580b1760 0x7fb5580b9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.513+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 >> 172.21.15.187:0/1341283539 conn(0x7fb5580bab30 msgr2=0x7fb55814b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:43.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.513+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 shutdown_connections 2020-11-20T12:19:43.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.513+0000 7fb55da43700 1 -- 172.21.15.187:0/1341283539 wait complete. 2020-11-20T12:19:43.522 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:19:43.519+0000 7fc6b8c49700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:19:43.541 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:19:43.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.782+0000 7f8cc228c700 1 Processor -- start 2020-11-20T12:19:43.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.782+0000 7f8cc228c700 1 -- start start 2020-11-20T12:19:43.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.783+0000 7f8cc228c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cbc060330 0x7f8cbc066fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.783+0000 7f8cc228c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cbc060710 0x7f8cbc145af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.783+0000 7f8cc228c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc146490 con 0x7f8cbc145fe0 2020-11-20T12:19:43.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.783+0000 7f8cc228c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cbc148e40 con 0x7f8cbc060710 2020-11-20T12:19:43.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.783+0000 7f8cc228c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc1490e0 con 0x7f8cbc060330 2020-11-20T12:19:43.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.784+0000 7f8cc0a89700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cbc060710 0x7f8cbc145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.784+0000 7f8cc0a89700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cbc060710 0x7f8cbc145af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52872/0 (socket says 172.21.15.187:52872) 2020-11-20T12:19:43.789 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.784+0000 7f8cc0a89700 1 -- 172.21.15.187:0/3073809523 learned_addr learned my addr 172.21.15.187:0/3073809523 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:43.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.784+0000 7f8cc0a89700 1 -- 172.21.15.187:0/3073809523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cbc060330 msgr2=0x7f8cbc066fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:43.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.784+0000 7f8cc0a89700 1 --2- 172.21.15.187:0/3073809523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cbc060330 0x7f8cbc066fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:43.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.784+0000 7f8cc0a89700 1 -- 172.21.15.187:0/3073809523 >> v1:172.21.15.187:6789/0 conn(0x7f8cbc145fe0 legacy=0x7f8cbc14c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:43.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.784+0000 7f8cc0a89700 1 -- 172.21.15.187:0/3073809523 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc149380 con 0x7f8cbc060710 2020-11-20T12:19:43.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.785+0000 7f8cc0a89700 1 --2- 172.21.15.187:0/3073809523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc060710 0x7f8cbc145af0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7f8cac00ba20 tx=0x7f8cac006eb0).ready entity=mon.1 client_cookie=ccff960f0bae701a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:43.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.785+0000 7f8cabfff700 1 -- 172.21.15.187:0/3073809523 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac009850 con 0x7f8cbc060710 2020-11-20T12:19:43.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.785+0000 7f8cabfff700 1 -- 172.21.15.187:0/3073809523 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac0099b0 con 0x7f8cbc060710 2020-11-20T12:19:43.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.785+0000 7f8cabfff700 1 -- 172.21.15.187:0/3073809523 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac009d30 con 0x7f8cbc060710 2020-11-20T12:19:43.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.785+0000 7f8cc228c700 1 -- 172.21.15.187:0/3073809523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc060710 msgr2=0x7f8cbc145af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.785+0000 7f8cc228c700 1 --2- 172.21.15.187:0/3073809523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc060710 0x7f8cbc145af0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7f8cac00ba20 tx=0x7f8cac006eb0).stop 2020-11-20T12:19:43.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.786+0000 7f8cc228c700 1 -- 172.21.15.187:0/3073809523 shutdown_connections 2020-11-20T12:19:43.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.786+0000 7f8cc228c700 1 --2- 172.21.15.187:0/3073809523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cbc060330 0x7f8cbc066fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:43.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.786+0000 7f8cc228c700 1 --2- 172.21.15.187:0/3073809523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc060710 0x7f8cbc145af0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.786+0000 7f8cc228c700 1 -- 172.21.15.187:0/3073809523 >> 172.21.15.187:0/3073809523 conn(0x7f8cbc0bbc80 msgr2=0x7f8cbc0bc090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:43.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.786+0000 7f8cc228c700 1 -- 172.21.15.187:0/3073809523 shutdown_connections 2020-11-20T12:19:43.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.786+0000 7f8cc228c700 1 -- 172.21.15.187:0/3073809523 wait complete. 2020-11-20T12:19:43.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.787+0000 7f8cc228c700 1 Processor -- start 2020-11-20T12:19:43.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.787+0000 7f8cc228c700 1 -- start start 2020-11-20T12:19:43.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc228c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cbc060330 0x7f8cbc1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc228c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cbc060710 0x7f8cbc1448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc228c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc145fe0 0x7f8cbc09c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc228c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc09e4a0 con 0x7f8cbc060330 2020-11-20T12:19:43.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc228c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc09d360 con 0x7f8cbc145fe0 2020-11-20T12:19:43.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc228c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc09d600 con 0x7f8cbc060710 2020-11-20T12:19:43.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc1a8b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc145fe0 0x7f8cbc09c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc1a8b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc145fe0 0x7f8cbc09c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52880/0 (socket says 172.21.15.187:52880) 2020-11-20T12:19:43.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.788+0000 7f8cc1a8b700 1 -- 172.21.15.187:0/1462259959 learned_addr learned my addr 172.21.15.187:0/1462259959 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:43.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8cc128a700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cbc060330 0x7f8cbc1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8cc1a8b700 1 -- 172.21.15.187:0/1462259959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cbc060710 msgr2=0x7f8cbc1448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8cc1a8b700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cbc060710 0x7f8cbc1448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:43.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8cc1a8b700 1 -- 172.21.15.187:0/1462259959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cbc060330 msgr2=0x7f8cbc1587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:43.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8cc1a8b700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cbc060330 0x7f8cbc1587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:43.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8cc1a8b700 1 -- 172.21.15.187:0/1462259959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc144dc0 con 0x7f8cbc145fe0 2020-11-20T12:19:43.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8cc1a8b700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc145fe0 0x7f8cbc09c8d0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 rx=0x7f8cb4009690 tx=0x7f8cb4006cf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:43.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8ca97fa700 1 -- 172.21.15.187:0/1462259959 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb400d560 con 0x7f8cbc145fe0 2020-11-20T12:19:43.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc09e120 con 0x7f8cbc145fe0 2020-11-20T12:19:43.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8ca97fa700 1 -- 172.21.15.187:0/1462259959 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb4000e90 con 0x7f8cbc145fe0 2020-11-20T12:19:43.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.789+0000 7f8ca97fa700 1 -- 172.21.15.187:0/1462259959 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb400d930 con 0x7f8cbc145fe0 2020-11-20T12:19:43.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.790+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc09c3a0 con 0x7f8cbc145fe0 2020-11-20T12:19:43.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.791+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cbc09d7c0 con 0x7f8cbc145fe0 2020-11-20T12:19:43.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.791+0000 7f8ca97fa700 1 -- 172.21.15.187:0/1462259959 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8cb400f030 con 0x7f8cbc145fe0 2020-11-20T12:19:43.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.792+0000 7f8ca97fa700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c94056d20 0x7f8c940591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:43.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.792+0000 7f8ca97fa700 1 -- 172.21.15.187:0/1462259959 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8cb406fb60 con 0x7f8cbc145fe0 2020-11-20T12:19:43.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.792+0000 7f8cc128a700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c94056d20 0x7f8c940591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:43.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.793+0000 7f8cc128a700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c94056d20 0x7f8c940591e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f8cbc0b8a30 tx=0x7f8cb8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:43.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:43.795+0000 7f8ca97fa700 1 -- 172.21.15.187:0/1462259959 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8cb4044fb0 con 0x7f8cbc145fe0 2020-11-20T12:19:44.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.047+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f8cbc09c650 con 0x7f8cbc145fe0 2020-11-20T12:19:44.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.059+0000 7f8ca97fa700 1 -- 172.21.15.187:0/1462259959 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8cb4006e10 con 0x7f8cbc145fe0 2020-11-20T12:19:44.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.061+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c94056d20 msgr2=0x7f8c940591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:44.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.061+0000 7f8cc228c700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c94056d20 0x7f8c940591e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f8cbc0b8a30 tx=0x7f8cb8008040).stop 2020-11-20T12:19:44.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.061+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc145fe0 msgr2=0x7f8cbc09c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:44.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.061+0000 7f8cc228c700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc145fe0 0x7f8cbc09c8d0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 rx=0x7f8cb4009690 tx=0x7f8cb4006cf0).stop 2020-11-20T12:19:44.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.062+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 shutdown_connections 2020-11-20T12:19:44.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.062+0000 7f8cc228c700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c94056d20 0x7f8c940591e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.062+0000 7f8cc228c700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cbc060330 0x7f8cbc1587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.062+0000 7f8cc228c700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cbc145fe0 0x7f8cbc09c8d0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.062+0000 7f8cc228c700 1 --2- 172.21.15.187:0/1462259959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cbc060710 0x7f8cbc1448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:44.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.062+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 >> 172.21.15.187:0/1462259959 conn(0x7f8cbc0bbc80 msgr2=0x7f8cbc0ba220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:44.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.062+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 shutdown_connections 2020-11-20T12:19:44.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.062+0000 7f8cc228c700 1 -- 172.21.15.187:0/1462259959 wait complete. 2020-11-20T12:19:44.077 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:19:44.089 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:44.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.335+0000 7ff6f4a1e700 1 Processor -- start 2020-11-20T12:19:44.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.336+0000 7ff6f4a1e700 1 -- start start 2020-11-20T12:19:44.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.337+0000 7ff6f4a1e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6f0065190 0x7ff6f0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:44.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.337+0000 7ff6f4a1e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f009ca90 0x7ff6f009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:44.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.337+0000 7ff6f4a1e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00b13e0 con 0x7ff6f0065f90 2020-11-20T12:19:44.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.337+0000 7ff6f4a1e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f014ab30 con 0x7ff6f0065190 2020-11-20T12:19:44.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.337+0000 7ff6f4a1e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f0062c10 con 0x7ff6f009ca90 2020-11-20T12:19:44.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.337+0000 7ff6eed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6f0065190 0x7ff6f0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:44.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.337+0000 7ff6eed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6f0065190 0x7ff6f0065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52884/0 (socket says 172.21.15.187:52884) 2020-11-20T12:19:44.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.338+0000 7ff6eed9d700 1 -- 172.21.15.187:0/1329933956 learned_addr learned my addr 172.21.15.187:0/1329933956 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:44.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.338+0000 7ff6ef59e700 1 --2- 172.21.15.187:0/1329933956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f009ca90 0x7ff6f009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:44.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.338+0000 7ff6eed9d700 1 -- 172.21.15.187:0/1329933956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f009ca90 msgr2=0x7ff6f009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:44.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.338+0000 7ff6eed9d700 1 --2- 172.21.15.187:0/1329933956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f009ca90 0x7ff6f009cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.338+0000 7ff6eed9d700 1 -- 172.21.15.187:0/1329933956 >> v1:172.21.15.5:6789/0 conn(0x7ff6f0065f90 legacy=0x7ff6f0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:44.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.338+0000 7ff6eed9d700 1 -- 172.21.15.187:0/1329933956 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f00b1640 con 0x7ff6f0065190 2020-11-20T12:19:44.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.338+0000 7ff6eed9d700 1 --2- 172.21.15.187:0/1329933956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f0065570 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 rx=0x7ff6e0006fd0 tx=0x7ff6e0006eb0).ready entity=mon.1 client_cookie=cf7cb67ecb732308 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:44.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.339+0000 7ff6ee59c700 1 -- 172.21.15.187:0/1329933956 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0009920 con 0x7ff6f0065190 2020-11-20T12:19:44.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.339+0000 7ff6ee59c700 1 -- 172.21.15.187:0/1329933956 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e0009a80 con 0x7ff6f0065190 2020-11-20T12:19:44.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.339+0000 7ff6ee59c700 1 -- 172.21.15.187:0/1329933956 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0009e00 con 0x7ff6f0065190 2020-11-20T12:19:44.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.339+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/1329933956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 msgr2=0x7ff6f0065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:44.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.339+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/1329933956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f0065570 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 rx=0x7ff6e0006fd0 tx=0x7ff6e0006eb0).stop 2020-11-20T12:19:44.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.339+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/1329933956 shutdown_connections 2020-11-20T12:19:44.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.339+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/1329933956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f009ca90 0x7ff6f009cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.339+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/1329933956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f0065570 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.340+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/1329933956 >> 172.21.15.187:0/1329933956 conn(0x7ff6f00bab30 msgr2=0x7ff6f00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:44.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.340+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/1329933956 shutdown_connections 2020-11-20T12:19:44.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.340+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/1329933956 wait complete. 2020-11-20T12:19:44.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.342+0000 7ff6f4a1e700 1 Processor -- start 2020-11-20T12:19:44.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.342+0000 7ff6f4a1e700 1 -- start start 2020-11-20T12:19:44.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6f4a1e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f00b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:44.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6f4a1e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f0065f90 0x7ff6f0099260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:44.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6f4a1e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6f009ca90 0x7ff6f009eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:44.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6f4a1e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f0067240 con 0x7ff6f009ca90 2020-11-20T12:19:44.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6f4a1e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00674e0 con 0x7ff6f0065190 2020-11-20T12:19:44.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6f4a1e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f0067780 con 0x7ff6f0065f90 2020-11-20T12:19:44.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6eed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f00b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:44.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6eed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f00b39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52890/0 (socket says 172.21.15.187:52890) 2020-11-20T12:19:44.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.343+0000 7ff6eed9d700 1 -- 172.21.15.187:0/3266424086 learned_addr learned my addr 172.21.15.187:0/3266424086 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:44.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6e659c700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f0065f90 0x7ff6f0099260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:44.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6eed9d700 1 -- 172.21.15.187:0/3266424086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f0065f90 msgr2=0x7ff6f0099260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:44.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6eed9d700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f0065f90 0x7ff6f0099260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6eed9d700 1 -- 172.21.15.187:0/3266424086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6f009ca90 msgr2=0x7ff6f009eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:44.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6eed9d700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6f009ca90 0x7ff6f009eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:44.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6eed9d700 1 -- 172.21.15.187:0/3266424086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0099750 con 0x7ff6f0065190 2020-11-20T12:19:44.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6e659c700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f0065f90 0x7ff6f0099260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:44.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6eed9d700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f00b39e0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7ff6e000b1b0 tx=0x7ff6e0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:44.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.344+0000 7ff6e77fe700 1 -- 172.21.15.187:0/3266424086 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e001e460 con 0x7ff6f0065190 2020-11-20T12:19:44.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.345+0000 7ff6e77fe700 1 -- 172.21.15.187:0/3266424086 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e001e5c0 con 0x7ff6f0065190 2020-11-20T12:19:44.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.345+0000 7ff6e77fe700 1 -- 172.21.15.187:0/3266424086 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e001e940 con 0x7ff6f0065190 2020-11-20T12:19:44.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.345+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f0066e10 con 0x7ff6f0065190 2020-11-20T12:19:44.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.345+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f009efa0 con 0x7ff6f0065190 2020-11-20T12:19:44.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.346+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f0067940 con 0x7ff6f0065190 2020-11-20T12:19:44.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.347+0000 7ff6e77fe700 1 -- 172.21.15.187:0/3266424086 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff6e0025030 con 0x7ff6f0065190 2020-11-20T12:19:44.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.347+0000 7ff6e77fe700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6cc056d50 0x7ff6cc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:44.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.348+0000 7ff6e77fe700 1 -- 172.21.15.187:0/3266424086 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff6e0071770 con 0x7ff6f0065190 2020-11-20T12:19:44.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.348+0000 7ff6e659c700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6cc056d50 0x7ff6cc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:44.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.348+0000 7ff6e659c700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6cc056d50 0x7ff6cc059210 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7ff6f00b0de0 tx=0x7ff6d8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:44.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.352+0000 7ff6e77fe700 1 -- 172.21.15.187:0/3266424086 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff6e0046c10 con 0x7ff6f0065190 2020-11-20T12:19:44.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.601+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f0061a20 con 0x7ff6f0065190 2020-11-20T12:19:44.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.602+0000 7ff6e77fe700 1 -- 172.21.15.187:0/3266424086 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2142 (secure 0 0 0) 0x7ff6e0023070 con 0x7ff6f0065190 2020-11-20T12:19:44.607 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:44.607 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":138,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1775892026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1775892026},{"type":"v1","addr":"172.21.15.5:6833","nonce":1775892026}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":134}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:42.498695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9343":{"gid":9343,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1878527443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1878527443},{"type":"v1","addr":"172.21.15.187:6835","nonce":1878527443}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:19:44.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.604+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6cc056d50 msgr2=0x7ff6cc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:44.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.604+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6cc056d50 0x7ff6cc059210 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7ff6f00b0de0 tx=0x7ff6d8009040).stop 2020-11-20T12:19:44.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.604+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 msgr2=0x7ff6f00b39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:44.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.604+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f00b39e0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7ff6e000b1b0 tx=0x7ff6e0015000).stop 2020-11-20T12:19:44.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.604+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 shutdown_connections 2020-11-20T12:19:44.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.604+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff6cc056d50 0x7ff6cc059210 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.604+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6f009ca90 0x7ff6f009eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:44.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.604+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6f0065190 0x7ff6f00b39e0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.605+0000 7ff6f4a1e700 1 --2- 172.21.15.187:0/3266424086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff6f0065f90 0x7ff6f0099260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:44.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.605+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 >> 172.21.15.187:0/3266424086 conn(0x7ff6f00bab30 msgr2=0x7ff6f00ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:44.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.605+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 shutdown_connections 2020-11-20T12:19:44.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:44.605+0000 7ff6f4a1e700 1 -- 172.21.15.187:0/3266424086 wait complete. 2020-11-20T12:19:44.613 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 138 2020-11-20T12:19:44.834 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 138, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:19:40.033030+0000', 'modified': '2020-11-20T12:19:42.498695+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9343}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9343': {'gid': 9343, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1878527443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1878527443}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1878527443}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:44.834 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:19:44.835 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:19:45.110 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":9343,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":5.048316764,"mdsmap_epoch":138,"osdmap_epoch":140,"osdmap_epoch_barrier":140,"uptime":10.724249632999999} 2020-11-20T12:19:45.111 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 9343, "mdsmap_epoch": 138, "osdmap_epoch": 140, "osdmap_epoch_barrier": 140, "rank_uptime": 5.048316764, "state": "up:active", "uptime": 10.724249633, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:19:45.111 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:19:45.112 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:19:45.112 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:19:45.113 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:19:45.113 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:19:45.114 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:19:45.114 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:19:45.115 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:19:45.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:45.116 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:19:45.171 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:19:45.172 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:19:45.172 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:19:45.172 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:19:45.173 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:19:45.173 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:19:45.173 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:19:45.174 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:19:45.174 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:19:45.174 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:19:45.174 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:19:45.175 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:19:45.175 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:19:45.175 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:19:45.176 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:19:45.176 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 41853sec preferred_lft 41853sec 2020-11-20T12:19:45.176 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:19:45.176 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:19:45.177 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:19:45.177 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:19:45.177 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:19:45.177 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:19:45.177 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:19:45.178 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:19:45.178 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:19:45.178 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:19:45.178 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:19:45.179 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:19:45.179 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:19:45.179 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:19:45.179 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:19:45.179 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:19:45.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:45.182 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:19:45.279 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:19:45.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:45.284 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:19:45.344 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:19:45.344 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:19:45.345 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:19:45.345 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:19:45.345 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:19:45.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:45.347 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:19:45.347 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:19:45.348 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:19:45.348 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:19:45.349 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:19:45.349 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:19:45.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:45.611 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:19:45.663 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:19:45.664 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:19:45.665 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:19:45.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:45.667 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:19:45.768 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:19:46.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.009+0000 7f1e8248f700 1 Processor -- start 2020-11-20T12:19:46.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.010+0000 7f1e8248f700 1 -- start start 2020-11-20T12:19:46.014 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.011+0000 7f1e8248f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1e7c0b1380 0x7f1e7c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.011+0000 7f1e8248f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e7c0b1760 0x7f1e7c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.011+0000 7f1e8248f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c150390 con 0x7f1e7c065680 2020-11-20T12:19:46.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.011+0000 7f1e8248f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c150630 con 0x7f1e7c0b1380 2020-11-20T12:19:46.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.011+0000 7f1e8248f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c068510 con 0x7f1e7c0b1760 2020-11-20T12:19:46.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.011+0000 7f1e80c8c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e7c0b1760 0x7f1e7c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.011+0000 7f1e80c8c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e7c0b1760 0x7f1e7c065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52904/0 (socket says 172.21.15.187:52904) 2020-11-20T12:19:46.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.012+0000 7f1e80c8c700 1 -- 172.21.15.187:0/1611796914 learned_addr learned my addr 172.21.15.187:0/1611796914 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:46.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.012+0000 7f1e80c8c700 1 -- 172.21.15.187:0/1611796914 >> v2:172.21.15.5:3300/0 conn(0x7f1e7c0b1380 msgr2=0x7f1e7c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:46.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.012+0000 7f1e80c8c700 1 --2- 172.21.15.187:0/1611796914 >> v2:172.21.15.5:3300/0 conn(0x7f1e7c0b1380 0x7f1e7c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:46.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.012+0000 7f1e80c8c700 1 -- 172.21.15.187:0/1611796914 >> v1:172.21.15.5:6789/0 conn(0x7f1e7c065680 legacy=0x7f1e7c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:46.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.012+0000 7f1e80c8c700 1 -- 172.21.15.187:0/1611796914 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c150fc0 con 0x7f1e7c0b1760 2020-11-20T12:19:46.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.012+0000 7f1e80c8c700 1 --2- 172.21.15.187:0/1611796914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c065190 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 rx=0x7f1e7c063a20 tx=0x7f1e70006eb0).ready entity=mon.1 client_cookie=7f03144b4a49518 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:46.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.013+0000 7f1e6ffff700 1 -- 172.21.15.187:0/1611796914 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70009890 con 0x7f1e7c0b1760 2020-11-20T12:19:46.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.013+0000 7f1e6ffff700 1 -- 172.21.15.187:0/1611796914 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e700099f0 con 0x7f1e7c0b1760 2020-11-20T12:19:46.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.013+0000 7f1e6ffff700 1 -- 172.21.15.187:0/1611796914 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70009d70 con 0x7f1e7c0b1760 2020-11-20T12:19:46.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.014+0000 7f1e8248f700 1 -- 172.21.15.187:0/1611796914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 msgr2=0x7f1e7c065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.014+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1611796914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c065190 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 rx=0x7f1e7c063a20 tx=0x7f1e70006eb0).stop 2020-11-20T12:19:46.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.014+0000 7f1e8248f700 1 -- 172.21.15.187:0/1611796914 shutdown_connections 2020-11-20T12:19:46.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.014+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1611796914 >> v2:172.21.15.5:3300/0 conn(0x7f1e7c0b1380 0x7f1e7c150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:46.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.014+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1611796914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c065190 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.014+0000 7f1e8248f700 1 -- 172.21.15.187:0/1611796914 >> 172.21.15.187:0/1611796914 conn(0x7f1e7c0bab30 msgr2=0x7f1e7c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:46.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.014+0000 7f1e8248f700 1 -- 172.21.15.187:0/1611796914 shutdown_connections 2020-11-20T12:19:46.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.014+0000 7f1e8248f700 1 -- 172.21.15.187:0/1611796914 wait complete. 2020-11-20T12:19:46.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.016+0000 7f1e8248f700 1 Processor -- start 2020-11-20T12:19:46.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.016+0000 7f1e8248f700 1 -- start start 2020-11-20T12:19:46.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.017+0000 7f1e8248f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e7c0b1380 0x7f1e7c14f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.017+0000 7f1e8248f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c09f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.017+0000 7f1e8248f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e7c09fbd0 0x7f1e7c149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.017+0000 7f1e8248f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0a0060 con 0x7f1e7c0b1380 2020-11-20T12:19:46.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.017+0000 7f1e8248f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0a0300 con 0x7f1e7c0b1760 2020-11-20T12:19:46.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.017+0000 7f1e8248f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c14f9b0 con 0x7f1e7c09fbd0 2020-11-20T12:19:46.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.017+0000 7f1e80c8c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c09f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e80c8c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c09f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52912/0 (socket says 172.21.15.187:52912) 2020-11-20T12:19:46.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e80c8c700 1 -- 172.21.15.187:0/1536486683 learned_addr learned my addr 172.21.15.187:0/1536486683 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:46.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e80c8c700 1 -- 172.21.15.187:0/1536486683 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e7c09fbd0 msgr2=0x7f1e7c149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e81c8e700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e7c09fbd0 0x7f1e7c149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e80c8c700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e7c09fbd0 0x7f1e7c149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e80c8c700 1 -- 172.21.15.187:0/1536486683 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e7c0b1380 msgr2=0x7f1e7c14f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e8148d700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e7c0b1380 0x7f1e7c14f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e80c8c700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e7c0b1380 0x7f1e7c14f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e80c8c700 1 -- 172.21.15.187:0/1536486683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c15fcc0 con 0x7f1e7c0b1760 2020-11-20T12:19:46.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.018+0000 7f1e80c8c700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c09f6e0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7f1e70006ef0 tx=0x7f1e70015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:46.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.019+0000 7f1e6d7fa700 1 -- 172.21.15.187:0/1536486683 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7001e460 con 0x7f1e7c0b1760 2020-11-20T12:19:46.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.019+0000 7f1e6d7fa700 1 -- 172.21.15.187:0/1536486683 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7001e5c0 con 0x7f1e7c0b1760 2020-11-20T12:19:46.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.019+0000 7f1e6d7fa700 1 -- 172.21.15.187:0/1536486683 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7001e940 con 0x7f1e7c0b1760 2020-11-20T12:19:46.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.019+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7c149a70 con 0x7f1e7c0b1760 2020-11-20T12:19:46.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.019+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c149eb0 con 0x7f1e7c0b1760 2020-11-20T12:19:46.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.020+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7c15fe50 con 0x7f1e7c0b1760 2020-11-20T12:19:46.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.021+0000 7f1e6d7fa700 1 -- 172.21.15.187:0/1536486683 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1e70025030 con 0x7f1e7c0b1760 2020-11-20T12:19:46.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.021+0000 7f1e6d7fa700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e54056d20 0x7f1e540591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.021+0000 7f1e6d7fa700 1 -- 172.21.15.187:0/1536486683 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1e70071870 con 0x7f1e7c0b1760 2020-11-20T12:19:46.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.022+0000 7f1e8148d700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e54056d20 0x7f1e540591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.022+0000 7f1e8148d700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e54056d20 0x7f1e540591e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f1e7800a6d0 tx=0x7f1e78008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:46.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.026+0000 7f1e6d7fa700 1 -- 172.21.15.187:0/1536486683 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1e70046c60 con 0x7f1e7c0b1760 2020-11-20T12:19:46.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.212+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e7c061a20 con 0x7f1e7c0b1760 2020-11-20T12:19:46.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.214+0000 7f1e6d7fa700 1 -- 172.21.15.187:0/1536486683 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f1e700311b0 con 0x7f1e7c0b1760 2020-11-20T12:19:46.219 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:46.220 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":140,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:19:41.038077+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:19:38.454721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2020-11-20T12:19:39.038543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:19:46.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.216+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e54056d20 msgr2=0x7f1e540591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.216+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e54056d20 0x7f1e540591e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f1e7800a6d0 tx=0x7f1e78008040).stop 2020-11-20T12:19:46.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.216+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 msgr2=0x7f1e7c09f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.216+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c09f6e0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7f1e70006ef0 tx=0x7f1e70015000).stop 2020-11-20T12:19:46.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.217+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 shutdown_connections 2020-11-20T12:19:46.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.217+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1e54056d20 0x7f1e540591e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.217+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1e7c0b1380 0x7f1e7c14f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.217+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1e7c09fbd0 0x7f1e7c149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.217+0000 7f1e8248f700 1 --2- 172.21.15.187:0/1536486683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e7c0b1760 0x7f1e7c09f6e0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.223 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.217+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 >> 172.21.15.187:0/1536486683 conn(0x7f1e7c0bab30 msgr2=0x7f1e7c0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:46.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.217+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 shutdown_connections 2020-11-20T12:19:46.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.217+0000 7f1e8248f700 1 -- 172.21.15.187:0/1536486683 wait complete. 2020-11-20T12:19:46.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:19:38.454721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:19:39.038543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:19:46.246 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:19:46.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.488+0000 7fc16ffff700 1 Processor -- start 2020-11-20T12:19:46.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.489+0000 7fc16ffff700 1 -- start start 2020-11-20T12:19:46.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.490+0000 7fc16ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc170145cc0 0x7fc170066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.490+0000 7fc16ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc1701487e0 0x7fc170148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.490+0000 7fc16ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17014c170 con 0x7fc1700683f0 2020-11-20T12:19:46.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.490+0000 7fc16ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17014c410 con 0x7fc1701487e0 2020-11-20T12:19:46.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.490+0000 7fc16ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17014c6b0 con 0x7fc170145cc0 2020-11-20T12:19:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.490+0000 7fc1667fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc1700683f0 0x7fc170065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49526/0 (socket says 172.21.15.187:49526) 2020-11-20T12:19:46.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.490+0000 7fc1667fc700 1 -- 172.21.15.187:0/3090510299 learned_addr learned my addr 172.21.15.187:0/3090510299 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.491+0000 7fc16effd700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc170145cc0 0x7fc170066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.491+0000 7fc16f7fe700 1 --2- 172.21.15.187:0/3090510299 >> v2:172.21.15.5:3300/0 conn(0x7fc1701487e0 0x7fc170148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.491+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298540167 0 0) 0x7fc17014c170 con 0x7fc1700683f0 2020-11-20T12:19:46.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.491+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc150003240 con 0x7fc1700683f0 2020-11-20T12:19:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1431689723 0 0) 0x7fc150003240 con 0x7fc1700683f0 2020-11-20T12:19:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc170145cc0 msgr2=0x7fc170066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc170145cc0 0x7fc170066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 >> v2:172.21.15.5:3300/0 conn(0x7fc1701487e0 msgr2=0x7fc170148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:46.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 --2- 172.21.15.187:0/3090510299 >> v2:172.21.15.5:3300/0 conn(0x7fc1701487e0 0x7fc170148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700658a0 con 0x7fc1700683f0 2020-11-20T12:19:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fc1580014e0 con 0x7fc1700683f0 2020-11-20T12:19:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 >> v1:172.21.15.187:6789/0 conn(0x7fc1700683f0 legacy=0x7fc170065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc170145cc0 0x7fc1500037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc1500044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1500049d0 0x7fc150004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17014c410 con 0x7fc1500049d0 2020-11-20T12:19:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17014c6b0 con 0x7fc150003d10 2020-11-20T12:19:46.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1500042d0 con 0x7fc170145cc0 2020-11-20T12:19:46.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.492+0000 7fc1667fc700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc1500044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc16f7fe700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1500049d0 0x7fc150004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc16effd700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc170145cc0 0x7fc1500037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc1667fc700 1 -- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc170145cc0 msgr2=0x7fc1500037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:46.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc1667fc700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc170145cc0 0x7fc1500037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc1667fc700 1 -- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1500049d0 msgr2=0x7fc150004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:46.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc1667fc700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1500049d0 0x7fc150004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc1667fc700 1 -- 172.21.15.187:0/3090510299 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc150003240 con 0x7fc150003d10 2020-11-20T12:19:46.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc1667fc700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc1500044e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 rx=0x7fc170063a20 tx=0x7fc158007750).ready entity=mon.1 client_cookie=9d2fa32cd0475d3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:46.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1580070d0 con 0x7fc150003d10 2020-11-20T12:19:46.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158007230 con 0x7fc150003d10 2020-11-20T12:19:46.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.493+0000 7fc16e7fc700 1 -- 172.21.15.187:0/3090510299 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1580055f0 con 0x7fc150003d10 2020-11-20T12:19:46.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.494+0000 7fc16ffff700 1 -- 172.21.15.187:0/3090510299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 msgr2=0x7fc1500044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.494+0000 7fc16ffff700 1 --2- 172.21.15.187:0/3090510299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc1500044e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 rx=0x7fc170063a20 tx=0x7fc158007750).stop 2020-11-20T12:19:46.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.494+0000 7fc16f7fe700 1 -- 172.21.15.187:0/3090510299 reap_dead start 2020-11-20T12:19:46.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.494+0000 7fc16ffff700 1 -- 172.21.15.187:0/3090510299 shutdown_connections 2020-11-20T12:19:46.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.494+0000 7fc16ffff700 1 -- 172.21.15.187:0/3090510299 >> 172.21.15.187:0/3090510299 conn(0x7fc1700bab30 msgr2=0x7fc1700baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:46.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.494+0000 7fc16ffff700 1 -- 172.21.15.187:0/3090510299 shutdown_connections 2020-11-20T12:19:46.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.494+0000 7fc16ffff700 1 -- 172.21.15.187:0/3090510299 wait complete. 2020-11-20T12:19:46.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.496+0000 7fc16ffff700 1 Processor -- start 2020-11-20T12:19:46.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.496+0000 7fc16ffff700 1 -- start start 2020-11-20T12:19:46.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc17009c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17009ca20 0x7fc170145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc17015b340 0x7fc17014af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170145650 con 0x7fc17009ca20 2020-11-20T12:19:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b1260 con 0x7fc150003d10 2020-11-20T12:19:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b1500 con 0x7fc17015b340 2020-11-20T12:19:46.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc17009c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc17009c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52930/0 (socket says 172.21.15.187:52930) 2020-11-20T12:19:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16effd700 1 -- 172.21.15.187:0/4152645447 learned_addr learned my addr 172.21.15.187:0/4152645447 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16effd700 1 -- 172.21.15.187:0/4152645447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc17015b340 msgr2=0x7fc17014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc1667fc700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17009ca20 0x7fc170145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16effd700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc17015b340 0x7fc17014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:46.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16effd700 1 -- 172.21.15.187:0/4152645447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17009ca20 msgr2=0x7fc170145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.497+0000 7fc16effd700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17009ca20 0x7fc170145160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.498+0000 7fc16effd700 1 -- 172.21.15.187:0/4152645447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700b17a0 con 0x7fc150003d10 2020-11-20T12:19:46.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.498+0000 7fc16effd700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc17009c460 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7fc160006d00 tx=0x7fc160008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:46.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.498+0000 7fc1677fe700 1 -- 172.21.15.187:0/4152645447 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16000b2b0 con 0x7fc150003d10 2020-11-20T12:19:46.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.498+0000 7fc1677fe700 1 -- 172.21.15.187:0/4152645447 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc160011660 con 0x7fc150003d10 2020-11-20T12:19:46.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.498+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17015b7a0 con 0x7fc150003d10 2020-11-20T12:19:46.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.498+0000 7fc1677fe700 1 -- 172.21.15.187:0/4152645447 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1600119e0 con 0x7fc150003d10 2020-11-20T12:19:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.498+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17014b640 con 0x7fc150003d10 2020-11-20T12:19:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.499+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17009ce00 con 0x7fc150003d10 2020-11-20T12:19:46.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.500+0000 7fc1677fe700 1 -- 172.21.15.187:0/4152645447 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc160013400 con 0x7fc150003d10 2020-11-20T12:19:46.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.501+0000 7fc1677fe700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc14c056d20 0x7fc14c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.501+0000 7fc1677fe700 1 -- 172.21.15.187:0/4152645447 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc160070050 con 0x7fc150003d10 2020-11-20T12:19:46.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.501+0000 7fc1667fc700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc14c056d20 0x7fc14c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.501+0000 7fc1667fc700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc14c056d20 0x7fc14c0591e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7fc170063a20 tx=0x7fc158015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:46.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.504+0000 7fc1677fe700 1 -- 172.21.15.187:0/4152645447 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc160045440 con 0x7fc150003d10 2020-11-20T12:19:46.695 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.691+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc170061a20 con 0x7fc150003d10 2020-11-20T12:19:46.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.692+0000 7fc1677fe700 1 -- 172.21.15.187:0/4152645447 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc160017070 con 0x7fc150003d10 2020-11-20T12:19:46.696 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:19:46.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.694+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc14c056d20 msgr2=0x7fc14c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.694+0000 7fc16ffff700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc14c056d20 0x7fc14c0591e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7fc170063a20 tx=0x7fc158015000).stop 2020-11-20T12:19:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 msgr2=0x7fc17009c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc17009c460 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7fc160006d00 tx=0x7fc160008040).stop 2020-11-20T12:19:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 shutdown_connections 2020-11-20T12:19:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc14c056d20 0x7fc14c0591e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc17015b340 0x7fc17014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc150003d10 0x7fc17009c460 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 --2- 172.21.15.187:0/4152645447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17009ca20 0x7fc170145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 >> 172.21.15.187:0/4152645447 conn(0x7fc1700bab30 msgr2=0x7fc170065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 shutdown_connections 2020-11-20T12:19:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.695+0000 7fc16ffff700 1 -- 172.21.15.187:0/4152645447 wait complete. 2020-11-20T12:19:46.721 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:19:46.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.963+0000 7f71ba0c2700 1 Processor -- start 2020-11-20T12:19:46.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.963+0000 7f71ba0c2700 1 -- start start 2020-11-20T12:19:46.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.964+0000 7f71ba0c2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b40bfbe0 0x7f71b41480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.964+0000 7f71ba0c2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f71b409f560 0x7f71b41485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:46.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.964+0000 7f71ba0c2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b414d110 con 0x7f71b40bf800 2020-11-20T12:19:46.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.964+0000 7f71ba0c2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b414c9d0 con 0x7f71b409f560 2020-11-20T12:19:46.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.964+0000 7f71ba0c2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4148aa0 con 0x7f71b40bfbe0 2020-11-20T12:19:46.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.965+0000 7f71b98c1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f71b409f560 0x7f71b41485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.965+0000 7f71b98c1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f71b409f560 0x7f71b41485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52940/0 (socket says 172.21.15.187:52940) 2020-11-20T12:19:46.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.965+0000 7f71b98c1700 1 -- 172.21.15.187:0/2411335777 learned_addr learned my addr 172.21.15.187:0/2411335777 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:46.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.965+0000 7f71b98c1700 1 -- 172.21.15.187:0/2411335777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b40bfbe0 msgr2=0x7f71b41480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:46.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.965+0000 7f71b88bf700 1 --2- 172.21.15.187:0/2411335777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b40bfbe0 0x7f71b41480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:46.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.966+0000 7f71b98c1700 1 --2- 172.21.15.187:0/2411335777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b40bfbe0 0x7f71b41480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.966+0000 7f71b98c1700 1 -- 172.21.15.187:0/2411335777 >> v1:172.21.15.5:6789/0 conn(0x7f71b40bf800 legacy=0x7f71b414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.966+0000 7f71b98c1700 1 -- 172.21.15.187:0/2411335777 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b414cbd0 con 0x7f71b409f560 2020-11-20T12:19:46.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.966+0000 7f71b88bf700 1 --2- 172.21.15.187:0/2411335777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b40bfbe0 0x7f71b41480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:46.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.966+0000 7f71b98c1700 1 --2- 172.21.15.187:0/2411335777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b409f560 0x7f71b41485b0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7f71ac009d80 tx=0x7f71ac009290).ready entity=mon.1 client_cookie=c5d0536a48eab609 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:46.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.966+0000 7f71a3fff700 1 -- 172.21.15.187:0/2411335777 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac00bcd0 con 0x7f71b409f560 2020-11-20T12:19:46.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.966+0000 7f71a3fff700 1 -- 172.21.15.187:0/2411335777 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ac00be30 con 0x7f71b409f560 2020-11-20T12:19:46.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.966+0000 7f71a3fff700 1 -- 172.21.15.187:0/2411335777 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac011610 con 0x7f71b409f560 2020-11-20T12:19:46.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.967+0000 7f71ba0c2700 1 -- 172.21.15.187:0/2411335777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b409f560 msgr2=0x7f71b41485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:46.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.967+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/2411335777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b409f560 0x7f71b41485b0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7f71ac009d80 tx=0x7f71ac009290).stop 2020-11-20T12:19:46.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.967+0000 7f71ba0c2700 1 -- 172.21.15.187:0/2411335777 shutdown_connections 2020-11-20T12:19:46.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.967+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/2411335777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b409f560 0x7f71b41485b0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.967+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/2411335777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b40bfbe0 0x7f71b41480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:46.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.967+0000 7f71ba0c2700 1 -- 172.21.15.187:0/2411335777 >> 172.21.15.187:0/2411335777 conn(0x7f71b4068370 msgr2=0x7f71b40bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:46.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.967+0000 7f71ba0c2700 1 -- 172.21.15.187:0/2411335777 shutdown_connections 2020-11-20T12:19:46.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.967+0000 7f71ba0c2700 1 -- 172.21.15.187:0/2411335777 wait complete. 2020-11-20T12:19:46.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.969+0000 7f71ba0c2700 1 Processor -- start 2020-11-20T12:19:46.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.969+0000 7f71ba0c2700 1 -- start start 2020-11-20T12:19:46.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.970+0000 7f71ba0c2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b409f560 0x7f71b4158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.002 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.970+0000 7f71ba0c2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b40bf800 0x7f71b4147520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.970+0000 7f71ba0c2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71b4144540 0x7f71b41466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.003 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.970+0000 7f71ba0c2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4145d10 con 0x7f71b4144540 2020-11-20T12:19:47.004 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.970+0000 7f71ba0c2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4145fb0 con 0x7f71b40bf800 2020-11-20T12:19:47.005 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.970+0000 7f71ba0c2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4146250 con 0x7f71b409f560 2020-11-20T12:19:47.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b88bf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b40bf800 0x7f71b4147520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.006 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b88bf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b40bf800 0x7f71b4147520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52944/0 (socket says 172.21.15.187:52944) 2020-11-20T12:19:47.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b88bf700 1 -- 172.21.15.187:0/262038717 learned_addr learned my addr 172.21.15.187:0/262038717 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:47.008 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b98c1700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71b4144540 0x7f71b41466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.009 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b88bf700 1 -- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b409f560 msgr2=0x7f71b4158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b90c0700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b409f560 0x7f71b4158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.010 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b88bf700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b409f560 0x7f71b4158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b88bf700 1 -- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71b4144540 msgr2=0x7f71b41466f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b88bf700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71b4144540 0x7f71b41466f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b88bf700 1 -- 172.21.15.187:0/262038717 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b4159230 con 0x7f71b40bf800 2020-11-20T12:19:47.011 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b98c1700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71b4144540 0x7f71b41466f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:47.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.971+0000 7f71b90c0700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b409f560 0x7f71b4158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:47.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.972+0000 7f71b88bf700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b40bf800 0x7f71b4147520 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 rx=0x7f71a400c8f0 tx=0x7f71a4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:47.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.972+0000 7f71a17fa700 1 -- 172.21.15.187:0/262038717 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a40077a0 con 0x7f71b40bf800 2020-11-20T12:19:47.012 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.972+0000 7f71a17fa700 1 -- 172.21.15.187:0/262038717 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a4007900 con 0x7f71b40bf800 2020-11-20T12:19:47.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.972+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b4147a10 con 0x7f71b40bf800 2020-11-20T12:19:47.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.972+0000 7f71a17fa700 1 -- 172.21.15.187:0/262038717 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4007c80 con 0x7f71b40bf800 2020-11-20T12:19:47.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.972+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b414be10 con 0x7f71b40bf800 2020-11-20T12:19:47.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.973+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b4080050 con 0x7f71b40bf800 2020-11-20T12:19:47.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.974+0000 7f71a17fa700 1 -- 172.21.15.187:0/262038717 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f71a4011620 con 0x7f71b40bf800 2020-11-20T12:19:47.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.974+0000 7f71a17fa700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f718c056d20 0x7f718c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.974+0000 7f71a17fa700 1 -- 172.21.15.187:0/262038717 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f71a4070270 con 0x7f71b40bf800 2020-11-20T12:19:47.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.975+0000 7f71b90c0700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f718c056d20 0x7f718c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.975+0000 7f71b90c0700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f718c056d20 0x7f718c0591e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f71b00014f0 tx=0x7f71b0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:47.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:46.978+0000 7f71a17fa700 1 -- 172.21.15.187:0/262038717 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f71a40455b0 con 0x7f71b40bf800 2020-11-20T12:19:47.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.168+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f71b4146410 con 0x7f71b40bf800 2020-11-20T12:19:47.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.169+0000 7f71a17fa700 1 -- 172.21.15.187:0/262038717 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7f71a4048bd0 con 0x7f71b40bf800 2020-11-20T12:19:47.173 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:19:47.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f718c056d20 msgr2=0x7f718c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f718c056d20 0x7f718c0591e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f71b00014f0 tx=0x7f71b0009040).stop 2020-11-20T12:19:47.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b40bf800 msgr2=0x7f71b4147520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b40bf800 0x7f71b4147520 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 rx=0x7f71a400c8f0 tx=0x7f71a4008040).stop 2020-11-20T12:19:47.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 shutdown_connections 2020-11-20T12:19:47.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f718c056d20 0x7f718c0591e0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f71b4144540 0x7f71b41466f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f71b409f560 0x7f71b4158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 --2- 172.21.15.187:0/262038717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71b40bf800 0x7f71b4147520 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 >> 172.21.15.187:0/262038717 conn(0x7f71b4068370 msgr2=0x7f71b40a0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:47.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 shutdown_connections 2020-11-20T12:19:47.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.172+0000 7f71ba0c2700 1 -- 172.21.15.187:0/262038717 wait complete. 2020-11-20T12:19:47.197 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:19:47.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.450+0000 7f13bf59e700 1 Processor -- start 2020-11-20T12:19:47.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.450+0000 7f13bf59e700 1 -- start start 2020-11-20T12:19:47.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.452+0000 7f13bf59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13c00b6650 0x7f13c00afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.452+0000 7f13bf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 0x7f13c00aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.452+0000 7f13bf59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c00b1300 con 0x7f13c01484d0 2020-11-20T12:19:47.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.452+0000 7f13bf59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c0067730 con 0x7f13c00b6650 2020-11-20T12:19:47.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.452+0000 7f13bf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c00b15a0 con 0x7f13c0066f70 2020-11-20T12:19:47.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.452+0000 7f13be59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f13c01484d0 0x7f13c0061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49558/0 (socket says 172.21.15.187:49558) 2020-11-20T12:19:47.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.452+0000 7f13be59c700 1 -- 172.21.15.187:0/2321739916 learned_addr learned my addr 172.21.15.187:0/2321739916 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:47.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.452+0000 7f13b5d9b700 1 --2- 172.21.15.187:0/2321739916 >> v2:172.21.15.187:3300/0 conn(0x7f13c00b6650 0x7f13c00afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.453+0000 7f13bed9d700 1 --2- 172.21.15.187:0/2321739916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 0x7f13c00aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.453+0000 7f13bdd9b700 1 -- 172.21.15.187:0/2321739916 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003476173 0 0) 0x7f13c00b1300 con 0x7f13c01484d0 2020-11-20T12:19:47.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.453+0000 7f13bdd9b700 1 -- 172.21.15.187:0/2321739916 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a0003240 con 0x7f13c01484d0 2020-11-20T12:19:47.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.453+0000 7f13b5d9b700 1 -- 172.21.15.187:0/2321739916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 msgr2=0x7f13c00aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:47.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.453+0000 7f13b5d9b700 1 --2- 172.21.15.187:0/2321739916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 0x7f13c00aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.453+0000 7f13b5d9b700 1 -- 172.21.15.187:0/2321739916 >> v1:172.21.15.187:6789/0 conn(0x7f13c01484d0 legacy=0x7f13c0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.453+0000 7f13b5d9b700 1 -- 172.21.15.187:0/2321739916 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c00b1840 con 0x7f13c00b6650 2020-11-20T12:19:47.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.454+0000 7f13b5d9b700 1 --2- 172.21.15.187:0/2321739916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c00b6650 0x7f13c00afa90 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7f13a800b9a0 tx=0x7f13a8006df0).ready entity=mon.1 client_cookie=b17c9e3812ff276f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:47.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.454+0000 7f13bdd9b700 1 -- 172.21.15.187:0/2321739916 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a80097b0 con 0x7f13c00b6650 2020-11-20T12:19:47.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.454+0000 7f13bdd9b700 1 -- 172.21.15.187:0/2321739916 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a8009910 con 0x7f13c00b6650 2020-11-20T12:19:47.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.454+0000 7f13bdd9b700 1 -- 172.21.15.187:0/2321739916 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8009c90 con 0x7f13c00b6650 2020-11-20T12:19:47.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.455+0000 7f13bf59e700 1 -- 172.21.15.187:0/2321739916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c00b6650 msgr2=0x7f13c00afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.455+0000 7f13bf59e700 1 --2- 172.21.15.187:0/2321739916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c00b6650 0x7f13c00afa90 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7f13a800b9a0 tx=0x7f13a8006df0).stop 2020-11-20T12:19:47.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.455+0000 7f13bf59e700 1 -- 172.21.15.187:0/2321739916 shutdown_connections 2020-11-20T12:19:47.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.455+0000 7f13bf59e700 1 --2- 172.21.15.187:0/2321739916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 0x7f13c00aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.455+0000 7f13bf59e700 1 --2- 172.21.15.187:0/2321739916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c00b6650 0x7f13c00afa90 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.455+0000 7f13bf59e700 1 -- 172.21.15.187:0/2321739916 >> 172.21.15.187:0/2321739916 conn(0x7f13c00bab30 msgr2=0x7f13c00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:47.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.455+0000 7f13bf59e700 1 -- 172.21.15.187:0/2321739916 shutdown_connections 2020-11-20T12:19:47.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.455+0000 7f13bf59e700 1 -- 172.21.15.187:0/2321739916 wait complete. 2020-11-20T12:19:47.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.457+0000 7f13bf59e700 1 Processor -- start 2020-11-20T12:19:47.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.457+0000 7f13bf59e700 1 -- start start 2020-11-20T12:19:47.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.458+0000 7f13bf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 0x7f13c014b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.458+0000 7f13bf59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13c00b6650 0x7f13c0142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.458+0000 7f13bf59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c01484d0 0x7f13c01431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.458+0000 7f13bf59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c015c550 con 0x7f13c00b6650 2020-11-20T12:19:47.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.458+0000 7f13bf59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c0099a10 con 0x7f13c01484d0 2020-11-20T12:19:47.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.458+0000 7f13bf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c01436e0 con 0x7f13c0066f70 2020-11-20T12:19:47.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13bed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c01484d0 0x7f13c01431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13bed9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c01484d0 0x7f13c01431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52962/0 (socket says 172.21.15.187:52962) 2020-11-20T12:19:47.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13bed9d700 1 -- 172.21.15.187:0/3811233558 learned_addr learned my addr 172.21.15.187:0/3811233558 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:47.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13b5d9b700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13c00b6650 0x7f13c0142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13be59c700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 0x7f13c014b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13bed9d700 1 -- 172.21.15.187:0/3811233558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 msgr2=0x7f13c014b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13bed9d700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 0x7f13c014b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13bed9d700 1 -- 172.21.15.187:0/3811233558 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13c00b6650 msgr2=0x7f13c0142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13bed9d700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13c00b6650 0x7f13c0142d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.459+0000 7f13bed9d700 1 -- 172.21.15.187:0/3811233558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c015bb70 con 0x7f13c01484d0 2020-11-20T12:19:47.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.460+0000 7f13bed9d700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c01484d0 0x7f13c01431f0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f13b800eb70 tx=0x7f13b8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:47.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.460+0000 7f13b6ffd700 1 -- 172.21.15.187:0/3811233558 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b800c830 con 0x7f13c01484d0 2020-11-20T12:19:47.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.460+0000 7f13b6ffd700 1 -- 172.21.15.187:0/3811233558 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b800c990 con 0x7f13c01484d0 2020-11-20T12:19:47.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.460+0000 7f13b6ffd700 1 -- 172.21.15.187:0/3811233558 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b800cd30 con 0x7f13c01484d0 2020-11-20T12:19:47.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.460+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c015be70 con 0x7f13c01484d0 2020-11-20T12:19:47.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.460+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c015b5f0 con 0x7f13c01484d0 2020-11-20T12:19:47.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.461+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c0061a20 con 0x7f13c01484d0 2020-11-20T12:19:47.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.462+0000 7f13b6ffd700 1 -- 172.21.15.187:0/3811233558 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f13b8013640 con 0x7f13c01484d0 2020-11-20T12:19:47.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.462+0000 7f13b6ffd700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f139c056d20 0x7f139c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:47.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.463+0000 7f13b6ffd700 1 -- 172.21.15.187:0/3811233558 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13b806ff80 con 0x7f13c01484d0 2020-11-20T12:19:47.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.463+0000 7f13be59c700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f139c056d20 0x7f139c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:47.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.464+0000 7f13be59c700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f139c056d20 0x7f139c0591e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f13b00055a0 tx=0x7f13b0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:47.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.467+0000 7f13b6ffd700 1 -- 172.21.15.187:0/3811233558 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f13b8045350 con 0x7f13c01484d0 2020-11-20T12:19:47.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.660+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f13c00624e0 con 0x7f13c01484d0 2020-11-20T12:19:47.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.661+0000 7f13b6ffd700 1 -- 172.21.15.187:0/3811233558 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7f13b8048970 con 0x7f13c01484d0 2020-11-20T12:19:47.665 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:19:47.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f139c056d20 msgr2=0x7f139c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f139c056d20 0x7f139c0591e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f13b00055a0 tx=0x7f13b0009040).stop 2020-11-20T12:19:47.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c01484d0 msgr2=0x7f13c01431f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:47.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c01484d0 0x7f13c01431f0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f13b800eb70 tx=0x7f13b8008040).stop 2020-11-20T12:19:47.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 shutdown_connections 2020-11-20T12:19:47.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f139c056d20 0x7f139c0591e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c01484d0 0x7f13c01431f0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f13c0066f70 0x7f13c014b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 --2- 172.21.15.187:0/3811233558 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13c00b6650 0x7f13c0142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:47.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 >> 172.21.15.187:0/3811233558 conn(0x7f13c00bab30 msgr2=0x7f13c00afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:47.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 shutdown_connections 2020-11-20T12:19:47.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.664+0000 7f13bf59e700 1 -- 172.21.15.187:0/3811233558 wait complete. 2020-11-20T12:19:47.689 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:19:47.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.917+0000 7fe79caf3700 1 Processor -- start 2020-11-20T12:19:48.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.918+0000 7fe79caf3700 1 -- start start 2020-11-20T12:19:48.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.919+0000 7fe79caf3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe7980b6650 0x7fe7980afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.919+0000 7fe79caf3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980b1300 con 0x7fe7981484d0 2020-11-20T12:19:48.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.919+0000 7fe79caf3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980b15a0 con 0x7fe798066f70 2020-11-20T12:19:48.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.919+0000 7fe79caf3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe798067730 con 0x7fe7980b6650 2020-11-20T12:19:48.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.920+0000 7fe78effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe7980b6650 0x7fe7980afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.920+0000 7fe78effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe7980b6650 0x7fe7980afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:45232/0 (socket says 172.21.15.187:45232) 2020-11-20T12:19:48.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.920+0000 7fe797fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe798066f70 0x7fe7980aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49576/0 (socket says 172.21.15.187:49576) 2020-11-20T12:19:48.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.920+0000 7fe78effd700 1 -- 172.21.15.187:0/3542044876 learned_addr learned my addr 172.21.15.187:0/3542044876 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:48.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.920+0000 7fe796ffd700 1 -- 172.21.15.187:0/3542044876 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756311782 0 0) 0x7fe7980b15a0 con 0x7fe798066f70 2020-11-20T12:19:48.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.920+0000 7fe796ffd700 1 -- 172.21.15.187:0/3542044876 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe778003240 con 0x7fe798066f70 2020-11-20T12:19:48.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.921+0000 7fe78effd700 1 -- 172.21.15.187:0/3542044876 >> v1:172.21.15.187:6789/0 conn(0x7fe798066f70 legacy=0x7fe7980aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.921+0000 7fe78effd700 1 -- 172.21.15.187:0/3542044876 >> v1:172.21.15.5:6789/0 conn(0x7fe7981484d0 legacy=0x7fe798061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.921+0000 7fe78effd700 1 -- 172.21.15.187:0/3542044876 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7980b1840 con 0x7fe7980b6650 2020-11-20T12:19:48.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.921+0000 7fe78effd700 1 --2- 172.21.15.187:0/3542044876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7980b6650 0x7fe7980afa90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7fe780006fd0 tx=0x7fe780006eb0).ready entity=mon.0 client_cookie=7806a25296fce6b5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:48.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe796ffd700 1 -- 172.21.15.187:0/3542044876 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780009810 con 0x7fe7980b6650 2020-11-20T12:19:48.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe796ffd700 1 -- 172.21.15.187:0/3542044876 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe780009970 con 0x7fe7980b6650 2020-11-20T12:19:48.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe796ffd700 1 -- 172.21.15.187:0/3542044876 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780009cf0 con 0x7fe7980b6650 2020-11-20T12:19:48.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe79caf3700 1 -- 172.21.15.187:0/3542044876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7980b6650 msgr2=0x7fe7980afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe79caf3700 1 --2- 172.21.15.187:0/3542044876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7980b6650 0x7fe7980afa90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7fe780006fd0 tx=0x7fe780006eb0).stop 2020-11-20T12:19:48.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe79caf3700 1 -- 172.21.15.187:0/3542044876 shutdown_connections 2020-11-20T12:19:48.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe79caf3700 1 --2- 172.21.15.187:0/3542044876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7980b6650 0x7fe7980afa90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.150 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe79caf3700 1 -- 172.21.15.187:0/3542044876 >> 172.21.15.187:0/3542044876 conn(0x7fe7980bab30 msgr2=0x7fe7980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:48.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe79caf3700 1 -- 172.21.15.187:0/3542044876 shutdown_connections 2020-11-20T12:19:48.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.922+0000 7fe79caf3700 1 -- 172.21.15.187:0/3542044876 wait complete. 2020-11-20T12:19:48.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.924+0000 7fe79caf3700 1 Processor -- start 2020-11-20T12:19:48.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.924+0000 7fe79caf3700 1 -- start start 2020-11-20T12:19:48.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe79caf3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe798066f70 0x7fe7980b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe79caf3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7980b6650 0x7fe798138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe79caf3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7981484d0 0x7fe7980b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe79caf3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe798138b60 con 0x7fe798066f70 2020-11-20T12:19:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe79caf3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79809d350 con 0x7fe7980b6650 2020-11-20T12:19:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe79caf3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981391c0 con 0x7fe7981484d0 2020-11-20T12:19:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe78effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7980b6650 0x7fe798138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe78effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7980b6650 0x7fe798138670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52974/0 (socket says 172.21.15.187:52974) 2020-11-20T12:19:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.925+0000 7fe78effd700 1 -- 172.21.15.187:0/913038507 learned_addr learned my addr 172.21.15.187:0/913038507 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe797fff700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7981484d0 0x7fe7980b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe7977fe700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe798066f70 0x7fe7980b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78effd700 1 -- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7981484d0 msgr2=0x7fe7980b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78effd700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7981484d0 0x7fe7980b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78effd700 1 -- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe798066f70 msgr2=0x7fe7980b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78effd700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe798066f70 0x7fe7980b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78effd700 1 -- 172.21.15.187:0/913038507 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79809d0b0 con 0x7fe7980b6650 2020-11-20T12:19:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe7977fe700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe798066f70 0x7fe7980b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78effd700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7980b6650 0x7fe798138670 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7fe780006ef0 tx=0x7fe780015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78ffff700 1 -- 172.21.15.187:0/913038507 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780011e50 con 0x7fe7980b6650 2020-11-20T12:19:48.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78ffff700 1 -- 172.21.15.187:0/913038507 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78001e460 con 0x7fe7980b6650 2020-11-20T12:19:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe798138eb0 con 0x7fe7980b6650 2020-11-20T12:19:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.926+0000 7fe78ffff700 1 -- 172.21.15.187:0/913038507 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78001e800 con 0x7fe7980b6650 2020-11-20T12:19:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.927+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7980b2270 con 0x7fe7980b6650 2020-11-20T12:19:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.928+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe798061a20 con 0x7fe7980b6650 2020-11-20T12:19:48.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.928+0000 7fe78ffff700 1 -- 172.21.15.187:0/913038507 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe780025030 con 0x7fe7980b6650 2020-11-20T12:19:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.929+0000 7fe78ffff700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe774056d20 0x7fe7740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.929+0000 7fe78ffff700 1 -- 172.21.15.187:0/913038507 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe7800716c0 con 0x7fe7980b6650 2020-11-20T12:19:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.929+0000 7fe7977fe700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe774056d20 0x7fe7740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.930+0000 7fe7977fe700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe774056d20 0x7fe7740591e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fe788000bc0 tx=0x7fe788009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:48.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:47.933+0000 7fe78ffff700 1 -- 172.21.15.187:0/913038507 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe780046b60 con 0x7fe7980b6650 2020-11-20T12:19:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.121+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe7980624e0 con 0x7fe7980b6650 2020-11-20T12:19:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.122+0000 7fe78ffff700 1 -- 172.21.15.187:0/913038507 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe780023070 con 0x7fe7980b6650 2020-11-20T12:19:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe774056d20 msgr2=0x7fe7740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe774056d20 0x7fe7740591e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fe788000bc0 tx=0x7fe788009040).stop 2020-11-20T12:19:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7980b6650 msgr2=0x7fe798138670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7980b6650 0x7fe798138670 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7fe780006ef0 tx=0x7fe780015000).stop 2020-11-20T12:19:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 shutdown_connections 2020-11-20T12:19:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe774056d20 0x7fe7740591e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe798066f70 0x7fe7980b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe7981484d0 0x7fe7980b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 --2- 172.21.15.187:0/913038507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7980b6650 0x7fe798138670 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 >> 172.21.15.187:0/913038507 conn(0x7fe7980bab30 msgr2=0x7fe798066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:48.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.125+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 shutdown_connections 2020-11-20T12:19:48.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.126+0000 7fe79caf3700 1 -- 172.21.15.187:0/913038507 wait complete. 2020-11-20T12:19:48.162 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:19:48.163 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:19:48.164 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:48.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.402+0000 7f347ae9d700 1 Processor -- start 2020-11-20T12:19:48.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.403+0000 7f347ae9d700 1 -- start start 2020-11-20T12:19:48.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.404+0000 7f347ae9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f3474061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.404+0000 7f347ae9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f34740622c0 0x7f34740b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.404+0000 7f347ae9d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34740b1300 con 0x7f3474066f70 2020-11-20T12:19:48.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.404+0000 7f347ae9d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3474067730 con 0x7f34740622c0 2020-11-20T12:19:48.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.404+0000 7f347ae9d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34740b15a0 con 0x7f34741484d0 2020-11-20T12:19:48.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.405+0000 7f3479e9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f3474061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.405+0000 7f347969a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f34740622c0 0x7f34740b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.405+0000 7f3479e9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f3474061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:53868/0 (socket says 172.21.15.187:53868) 2020-11-20T12:19:48.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.405+0000 7f3479e9b700 1 -- 172.21.15.187:0/3692912843 learned_addr learned my addr 172.21.15.187:0/3692912843 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:48.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.405+0000 7f347a69c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3474066f70 0x7f34740b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49590/0 (socket says 172.21.15.187:49590) 2020-11-20T12:19:48.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f347969a700 1 -- 172.21.15.187:0/3692912843 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 msgr2=0x7f3474061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:48.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f347969a700 1 --2- 172.21.15.187:0/3692912843 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f3474061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f347969a700 1 -- 172.21.15.187:0/3692912843 >> v1:172.21.15.187:6789/0 conn(0x7f3474066f70 legacy=0x7f34740b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f347969a700 1 -- 172.21.15.187:0/3692912843 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740b1840 con 0x7f34740622c0 2020-11-20T12:19:48.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f3478e99700 1 -- 172.21.15.187:0/3692912843 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410541685 0 0) 0x7f34740b1300 con 0x7f3474066f70 2020-11-20T12:19:48.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f347969a700 1 --2- 172.21.15.187:0/3692912843 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34740622c0 0x7f34740b5640 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f347000b9a0 tx=0x7f3470006df0).ready entity=mon.1 client_cookie=97c03db0bd25cf5e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:48.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f3478e99700 1 -- 172.21.15.187:0/3692912843 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700097b0 con 0x7f34740622c0 2020-11-20T12:19:48.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f3478e99700 1 -- 172.21.15.187:0/3692912843 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3470009910 con 0x7f34740622c0 2020-11-20T12:19:48.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.406+0000 7f3478e99700 1 -- 172.21.15.187:0/3692912843 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470009c90 con 0x7f34740622c0 2020-11-20T12:19:48.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.407+0000 7f347ae9d700 1 -- 172.21.15.187:0/3692912843 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34740622c0 msgr2=0x7f34740b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.407+0000 7f347ae9d700 1 --2- 172.21.15.187:0/3692912843 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34740622c0 0x7f34740b5640 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f347000b9a0 tx=0x7f3470006df0).stop 2020-11-20T12:19:48.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.407+0000 7f347ae9d700 1 -- 172.21.15.187:0/3692912843 shutdown_connections 2020-11-20T12:19:48.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.407+0000 7f347ae9d700 1 --2- 172.21.15.187:0/3692912843 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f3474061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.407+0000 7f347ae9d700 1 --2- 172.21.15.187:0/3692912843 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34740622c0 0x7f34740b5640 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.407+0000 7f347ae9d700 1 -- 172.21.15.187:0/3692912843 >> 172.21.15.187:0/3692912843 conn(0x7f34740bab30 msgr2=0x7f34740baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:48.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.407+0000 7f347ae9d700 1 -- 172.21.15.187:0/3692912843 shutdown_connections 2020-11-20T12:19:48.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.407+0000 7f347ae9d700 1 -- 172.21.15.187:0/3692912843 wait complete. 2020-11-20T12:19:48.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.409+0000 7f347ae9d700 1 Processor -- start 2020-11-20T12:19:48.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.409+0000 7f347ae9d700 1 -- start start 2020-11-20T12:19:48.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.409+0000 7f347ae9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3474066f70 0x7f34740b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f347ae9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f347413c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f347ae9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f347415b660 0x7f347413d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f347ae9d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347413c160 con 0x7f347415b660 2020-11-20T12:19:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f347ae9d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347413c400 con 0x7f3474066f70 2020-11-20T12:19:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f347ae9d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347413be50 con 0x7f34741484d0 2020-11-20T12:19:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f3479e9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3474066f70 0x7f34740b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f3479e9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3474066f70 0x7f34740b7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52986/0 (socket says 172.21.15.187:52986) 2020-11-20T12:19:48.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f3479e9b700 1 -- 172.21.15.187:0/2184287093 learned_addr learned my addr 172.21.15.187:0/2184287093 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:48.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f347969a700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f347413c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f3479e9b700 1 -- 172.21.15.187:0/2184287093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 msgr2=0x7f347413c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f3479e9b700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f347413c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f3479e9b700 1 -- 172.21.15.187:0/2184287093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f347415b660 msgr2=0x7f347413d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f3479e9b700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f347415b660 0x7f347413d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.410+0000 7f3479e9b700 1 -- 172.21.15.187:0/2184287093 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740b80e0 con 0x7f3474066f70 2020-11-20T12:19:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.411+0000 7f3479e9b700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3474066f70 0x7f34740b7bf0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7f346800f860 tx=0x7f3468008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.411+0000 7f3465ffb700 1 -- 172.21.15.187:0/2184287093 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346800d6c0 con 0x7f3474066f70 2020-11-20T12:19:48.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.411+0000 7f3465ffb700 1 -- 172.21.15.187:0/2184287093 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346800d820 con 0x7f3474066f70 2020-11-20T12:19:48.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.411+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34740b4080 con 0x7f3474066f70 2020-11-20T12:19:48.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.411+0000 7f3465ffb700 1 -- 172.21.15.187:0/2184287093 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346800dba0 con 0x7f3474066f70 2020-11-20T12:19:48.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.411+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347413cfd0 con 0x7f3474066f70 2020-11-20T12:19:48.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.412+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347413c5c0 con 0x7f3474066f70 2020-11-20T12:19:48.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.413+0000 7f3465ffb700 1 -- 172.21.15.187:0/2184287093 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3468024070 con 0x7f3474066f70 2020-11-20T12:19:48.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.413+0000 7f3465ffb700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f344c056d20 0x7f344c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:48.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.413+0000 7f3465ffb700 1 -- 172.21.15.187:0/2184287093 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3468070550 con 0x7f3474066f70 2020-11-20T12:19:48.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.414+0000 7f347969a700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f344c056d20 0x7f344c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:48.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.414+0000 7f347969a700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f344c056d20 0x7f344c0591e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f347000b9a0 tx=0x7f3470015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:48.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.417+0000 7f3465ffb700 1 -- 172.21.15.187:0/2184287093 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3468045890 con 0x7f3474066f70 2020-11-20T12:19:48.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.662+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3474061a20 con 0x7f3474066f70 2020-11-20T12:19:48.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.663+0000 7f3465ffb700 1 -- 172.21.15.187:0/2184287093 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2142 (secure 0 0 0) 0x7f3468018020 con 0x7f3474066f70 2020-11-20T12:19:48.667 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:48.667 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":138,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1775892026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1775892026},{"type":"v1","addr":"172.21.15.5:6833","nonce":1775892026}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":134}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:42.498695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9343":{"gid":9343,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1878527443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1878527443},{"type":"v1","addr":"172.21.15.187:6835","nonce":1878527443}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:19:48.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.665+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f344c056d20 msgr2=0x7f344c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.665+0000 7f347ae9d700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f344c056d20 0x7f344c0591e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f347000b9a0 tx=0x7f3470015000).stop 2020-11-20T12:19:48.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.665+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3474066f70 msgr2=0x7f34740b7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:48.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.665+0000 7f347ae9d700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3474066f70 0x7f34740b7bf0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7f346800f860 tx=0x7f3468008040).stop 2020-11-20T12:19:48.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.666+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 shutdown_connections 2020-11-20T12:19:48.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.666+0000 7f347ae9d700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f344c056d20 0x7f344c0591e0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.666+0000 7f347ae9d700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f347415b660 0x7f347413d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:48.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.666+0000 7f347ae9d700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3474066f70 0x7f34740b7bf0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.666+0000 7f347ae9d700 1 --2- 172.21.15.187:0/2184287093 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34741484d0 0x7f347413c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:48.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.666+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 >> 172.21.15.187:0/2184287093 conn(0x7f34740bab30 msgr2=0x7f3474065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:48.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.666+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 shutdown_connections 2020-11-20T12:19:48.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:48.666+0000 7f347ae9d700 1 -- 172.21.15.187:0/2184287093 wait complete. 2020-11-20T12:19:48.672 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 138 2020-11-20T12:19:48.893 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 138, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:19:40.033030+0000', 'modified': '2020-11-20T12:19:42.498695+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9343}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9343': {'gid': 9343, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1878527443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1878527443}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1878527443}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:48.894 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:19:48.894 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:19:49.175 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":9343,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":9.1127086800000008,"mdsmap_epoch":138,"osdmap_epoch":140,"osdmap_epoch_barrier":140,"uptime":14.788656588} 2020-11-20T12:19:49.176 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 9343, "mdsmap_epoch": 138, "osdmap_epoch": 140, "osdmap_epoch_barrier": 140, "rank_uptime": 9.11270868, "state": "up:active", "uptime": 14.788656588, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:19:49.176 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:19:49.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:49.177 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:19:49.238 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:19:49.239 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:19:49.387 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:19:49.387 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:19:49.428 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:19:49.428 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:19:49.439 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:19:49.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:49.441 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:19:49.546 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:19:49.547 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:19:49.548 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:19:49.550 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:19:49.572 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:19:49.572 DEBUG:teuthology.parallel:result is None 2020-11-20T12:19:49.574 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:19:49.574 DEBUG:teuthology.parallel:result is None 2020-11-20T12:19:49.575 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2020-11-20T12:19:49.576 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2020-11-20T12:19:49.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.792+0000 7fbb2759e700 1 Processor -- start 2020-11-20T12:19:49.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.792+0000 7fbb2759e700 1 -- start start 2020-11-20T12:19:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.793+0000 7fbb2759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb280bf800 0x7fbb2814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.793+0000 7fbb2759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2814d110 con 0x7fbb280bfbe0 2020-11-20T12:19:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.794+0000 7fbb2759e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2814cbd0 con 0x7fbb2809f560 2020-11-20T12:19:49.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.794+0000 7fbb2759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb2814ce70 con 0x7fbb280bf800 2020-11-20T12:19:49.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.794+0000 7fbb26d9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbb2809f560 0x7fbb280a36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49608/0 (socket says 172.21.15.187:49608) 2020-11-20T12:19:49.799 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.794+0000 7fbb26d9d700 1 -- 172.21.15.187:0/2783770963 learned_addr learned my addr 172.21.15.187:0/2783770963 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:49.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.794+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221563034 0 0) 0x7fbb2814cbd0 con 0x7fbb2809f560 2020-11-20T12:19:49.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb0c003240 con 0x7fbb2809f560 2020-11-20T12:19:49.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2659c700 1 --2- 172.21.15.187:0/2783770963 >> v2:172.21.15.5:3300/0 conn(0x7fbb280bf800 0x7fbb2814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3243828495 0 0) 0x7fbb0c003240 con 0x7fbb2809f560 2020-11-20T12:19:49.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 >> v2:172.21.15.5:3300/0 conn(0x7fbb280bf800 msgr2=0x7fbb2814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:49.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 --2- 172.21.15.187:0/2783770963 >> v2:172.21.15.5:3300/0 conn(0x7fbb280bf800 0x7fbb2814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 >> v1:172.21.15.5:6789/0 conn(0x7fbb280bfbe0 legacy=0x7fbb281480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb280a4cc0 con 0x7fbb2809f560 2020-11-20T12:19:49.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fbb200018a0 con 0x7fbb2809f560 2020-11-20T12:19:49.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 >> v1:172.21.15.187:6789/0 conn(0x7fbb2809f560 legacy=0x7fbb280a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb0c003730 0x7fbb0c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb0c0044e0 0x7fbb0c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c004e50 0x7fbb0c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2814ce70 con 0x7fbb0c003730 2020-11-20T12:19:49.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c004260 con 0x7fbb0c0044e0 2020-11-20T12:19:49.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.795+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c00bc20 con 0x7fbb0c004e50 2020-11-20T12:19:49.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb25d9b700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb0c0044e0 0x7fbb0c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb26d9d700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c004e50 0x7fbb0c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb2659c700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb0c003730 0x7fbb0c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb25d9b700 1 -- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c004e50 msgr2=0x7fbb0c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:49.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb25d9b700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c004e50 0x7fbb0c00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb25d9b700 1 -- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb0c003730 msgr2=0x7fbb0c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:49.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb25d9b700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb0c003730 0x7fbb0c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb26d9d700 1 -- 172.21.15.187:0/2783770963 reap_dead start 2020-11-20T12:19:49.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb25d9b700 1 -- 172.21.15.187:0/2783770963 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0c003240 con 0x7fbb0c0044e0 2020-11-20T12:19:49.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7fbb2659c700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb0c003730 0x7fbb0c003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:49.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.797+0000 7fbb25d9b700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb0c0044e0 0x7fbb0c004960 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 rx=0x7fbb1c001920 tx=0x7fbb1c007b90).ready entity=mon.1 client_cookie=cae92ecd7a5e1087 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:49.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.797+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c007020 con 0x7fbb0c0044e0 2020-11-20T12:19:49.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.797+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c007180 con 0x7fbb0c0044e0 2020-11-20T12:19:49.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.797+0000 7fbb2559a700 1 -- 172.21.15.187:0/2783770963 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c00c620 con 0x7fbb0c0044e0 2020-11-20T12:19:49.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7fbb2759e700 1 -- 172.21.15.187:0/2783770963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb0c0044e0 msgr2=0x7fbb0c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7fbb2759e700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb0c0044e0 0x7fbb0c004960 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 rx=0x7fbb1c001920 tx=0x7fbb1c007b90).stop 2020-11-20T12:19:49.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7fbb2759e700 1 -- 172.21.15.187:0/2783770963 shutdown_connections 2020-11-20T12:19:49.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7fbb2759e700 1 --2- 172.21.15.187:0/2783770963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb0c0044e0 0x7fbb0c004960 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7fbb2759e700 1 -- 172.21.15.187:0/2783770963 >> 172.21.15.187:0/2783770963 conn(0x7fbb28068370 msgr2=0x7fbb280bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:49.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7fbb2759e700 1 -- 172.21.15.187:0/2783770963 shutdown_connections 2020-11-20T12:19:49.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7fbb2759e700 1 -- 172.21.15.187:0/2783770963 wait complete. 2020-11-20T12:19:49.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.800+0000 7fbb2759e700 1 Processor -- start 2020-11-20T12:19:49.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.800+0000 7fbb2759e700 1 -- start start 2020-11-20T12:19:49.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb2759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c0044e0 0x7fbb2814b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb2759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb28145f90 0x7fbb281335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb2759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb28133ae0 0x7fbb281422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb2759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb28133f60 con 0x7fbb28133ae0 2020-11-20T12:19:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb2759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2815c0f0 con 0x7fbb28145f90 2020-11-20T12:19:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb2759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2815c390 con 0x7fbb0c0044e0 2020-11-20T12:19:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb25d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb28145f90 0x7fbb281335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb25d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb28145f90 0x7fbb281335f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53018/0 (socket says 172.21.15.187:53018) 2020-11-20T12:19:49.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7fbb25d9b700 1 -- 172.21.15.187:0/4087911824 learned_addr learned my addr 172.21.15.187:0/4087911824 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:49.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7fbb2659c700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c0044e0 0x7fbb2814b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7fbb25d9b700 1 -- 172.21.15.187:0/4087911824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c0044e0 msgr2=0x7fbb2814b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7fbb25d9b700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c0044e0 0x7fbb2814b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7fbb25d9b700 1 -- 172.21.15.187:0/4087911824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb28133ae0 msgr2=0x7fbb281422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7fbb25d9b700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb28133ae0 0x7fbb281422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:49.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7fbb25d9b700 1 -- 172.21.15.187:0/4087911824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2815c630 con 0x7fbb28145f90 2020-11-20T12:19:49.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7fbb25d9b700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb28145f90 0x7fbb281335f0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 rx=0x7fbb1c00c750 tx=0x7fbb1c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:49.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7fbb167fc700 1 -- 172.21.15.187:0/4087911824 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c0139f0 con 0x7fbb28145f90 2020-11-20T12:19:49.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.803+0000 7fbb167fc700 1 -- 172.21.15.187:0/4087911824 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c013b50 con 0x7fbb28145f90 2020-11-20T12:19:49.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.803+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2814b520 con 0x7fbb28145f90 2020-11-20T12:19:49.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.803+0000 7fbb167fc700 1 -- 172.21.15.187:0/4087911824 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c020440 con 0x7fbb28145f90 2020-11-20T12:19:49.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.803+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2815cad0 con 0x7fbb28145f90 2020-11-20T12:19:49.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.804+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb2809ef10 con 0x7fbb28145f90 2020-11-20T12:19:49.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.804+0000 7fbb167fc700 1 -- 172.21.15.187:0/4087911824 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbb1c026020 con 0x7fbb28145f90 2020-11-20T12:19:49.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7fbb167fc700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbafc056d50 0x7fbafc059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7fbb167fc700 1 -- 172.21.15.187:0/4087911824 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbb1c0713b0 con 0x7fbb28145f90 2020-11-20T12:19:49.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7fbb2659c700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbafc056d50 0x7fbafc059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7fbb2659c700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbafc056d50 0x7fbafc059210 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fbb18006d00 tx=0x7fbb18008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:49.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7f9b0a248700 1 Processor -- start 2020-11-20T12:19:49.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.796+0000 7f9b0a248700 1 -- start start 2020-11-20T12:19:49.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7f9b0a248700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b040bf800 0x7f9b0414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7f9b0a248700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9b040bfbe0 0x7f9b041480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7f9b0a248700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0414d110 con 0x7f9b0409f560 2020-11-20T12:19:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7f9b0a248700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0414cbd0 con 0x7f9b040bfbe0 2020-11-20T12:19:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.798+0000 7f9b0a248700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0414ce70 con 0x7f9b040bf800 2020-11-20T12:19:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.799+0000 7f9b09246700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b040bf800 0x7f9b0414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.799+0000 7f9b09246700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b040bf800 0x7f9b0414c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:53898/0 (socket says 172.21.15.187:53898) 2020-11-20T12:19:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.799+0000 7f9b09246700 1 -- 172.21.15.187:0/2012767202 learned_addr learned my addr 172.21.15.187:0/2012767202 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.799+0000 7f9b08a45700 1 --2- 172.21.15.187:0/2012767202 >> v2:172.21.15.5:3300/0 conn(0x7f9b040bfbe0 0x7f9b041480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.800+0000 7f9b09246700 1 -- 172.21.15.187:0/2012767202 >> v2:172.21.15.5:3300/0 conn(0x7f9b040bfbe0 msgr2=0x7f9b041480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:49.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.800+0000 7f9b09246700 1 --2- 172.21.15.187:0/2012767202 >> v2:172.21.15.5:3300/0 conn(0x7f9b040bfbe0 0x7f9b041480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.800+0000 7f9b09246700 1 -- 172.21.15.187:0/2012767202 >> v1:172.21.15.5:6789/0 conn(0x7f9b0409f560 legacy=0x7f9b04060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.800+0000 7f9b09246700 1 -- 172.21.15.187:0/2012767202 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040a4cc0 con 0x7f9b040bf800 2020-11-20T12:19:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.800+0000 7f9b09246700 1 --2- 172.21.15.187:0/2012767202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b040bf800 0x7f9b0414c2c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9b00006fd0 tx=0x7f9b00006eb0).ready entity=mon.2 client_cookie=db25564373f2eb4a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9af3fff700 1 -- 172.21.15.187:0/2012767202 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b000098a0 con 0x7f9b040bf800 2020-11-20T12:19:49.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9af3fff700 1 -- 172.21.15.187:0/2012767202 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b00009a00 con 0x7f9b040bf800 2020-11-20T12:19:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9af3fff700 1 -- 172.21.15.187:0/2012767202 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00009d80 con 0x7f9b040bf800 2020-11-20T12:19:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9b0a248700 1 -- 172.21.15.187:0/2012767202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b040bf800 msgr2=0x7f9b0414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9b0a248700 1 --2- 172.21.15.187:0/2012767202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b040bf800 0x7f9b0414c2c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9b00006fd0 tx=0x7f9b00006eb0).stop 2020-11-20T12:19:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9b0a248700 1 -- 172.21.15.187:0/2012767202 shutdown_connections 2020-11-20T12:19:49.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9b0a248700 1 --2- 172.21.15.187:0/2012767202 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b040bf800 0x7f9b0414c2c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9b0a248700 1 --2- 172.21.15.187:0/2012767202 >> v2:172.21.15.5:3300/0 conn(0x7f9b040bfbe0 0x7f9b041480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.801+0000 7f9b0a248700 1 -- 172.21.15.187:0/2012767202 >> 172.21.15.187:0/2012767202 conn(0x7f9b04068370 msgr2=0x7f9b040bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7f9b0a248700 1 -- 172.21.15.187:0/2012767202 shutdown_connections 2020-11-20T12:19:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.802+0000 7f9b0a248700 1 -- 172.21.15.187:0/2012767202 wait complete. 2020-11-20T12:19:49.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.803+0000 7f9b0a248700 1 Processor -- start 2020-11-20T12:19:49.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.804+0000 7f9b0a248700 1 -- start start 2020-11-20T12:19:49.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b0a248700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b0409f560 0x7f9b0415ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b0a248700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b040bf800 0x7f9b04066220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b0a248700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b04063e40 0x7f9b04066e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b0a248700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b04065840 con 0x7f9b040bf800 2020-11-20T12:19:49.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b0a248700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b04065ae0 con 0x7f9b0409f560 2020-11-20T12:19:49.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b0a248700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b04065d80 con 0x7f9b04063e40 2020-11-20T12:19:49.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b09246700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b0409f560 0x7f9b0415ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b09246700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b0409f560 0x7f9b0415ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53022/0 (socket says 172.21.15.187:53022) 2020-11-20T12:19:49.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.805+0000 7f9b09246700 1 -- 172.21.15.187:0/1272021177 learned_addr learned my addr 172.21.15.187:0/1272021177 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:49.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7f9b09a47700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b04063e40 0x7f9b04066e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7f9b08a45700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b040bf800 0x7f9b04066220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7f9b09246700 1 -- 172.21.15.187:0/1272021177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b04063e40 msgr2=0x7f9b04066e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7f9b09246700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b04063e40 0x7f9b04066e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7f9b09246700 1 -- 172.21.15.187:0/1272021177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b040bf800 msgr2=0x7f9b04066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:49.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7f9b09246700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b040bf800 0x7f9b04066220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:49.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7f9b09246700 1 -- 172.21.15.187:0/1272021177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b04067360 con 0x7f9b0409f560 2020-11-20T12:19:49.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.806+0000 7f9b09246700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b0409f560 0x7f9b0415ca80 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f9b0000b1b0 tx=0x7f9b00015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:49.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.809+0000 7f9af17fa700 1 -- 172.21.15.187:0/1272021177 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0001d5d0 con 0x7f9b0409f560 2020-11-20T12:19:49.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.809+0000 7f9af17fa700 1 -- 172.21.15.187:0/1272021177 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0001d750 con 0x7f9b0409f560 2020-11-20T12:19:49.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.809+0000 7f9af17fa700 1 -- 172.21.15.187:0/1272021177 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0001db10 con 0x7f9b0409f560 2020-11-20T12:19:49.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.809+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b04065450 con 0x7f9b0409f560 2020-11-20T12:19:49.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.810+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b04156570 con 0x7f9b0409f560 2020-11-20T12:19:49.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.811+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b04080050 con 0x7f9b0409f560 2020-11-20T12:19:49.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.811+0000 7f9af17fa700 1 -- 172.21.15.187:0/1272021177 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9b00025030 con 0x7f9b0409f560 2020-11-20T12:19:49.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.812+0000 7f9af17fa700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9adc056d20 0x7f9adc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:49.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.812+0000 7f9af17fa700 1 -- 172.21.15.187:0/1272021177 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9b00071970 con 0x7f9b0409f560 2020-11-20T12:19:49.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.812+0000 7f9b08a45700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9adc056d20 0x7f9adc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:49.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.813+0000 7f9b08a45700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9adc056d20 0x7f9adc0591e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f9b04141aa0 tx=0x7f9af4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:49.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.816+0000 7f9af17fa700 1 -- 172.21.15.187:0/1272021177 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9b00046dc0 con 0x7f9b0409f560 2020-11-20T12:19:49.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:49.810+0000 7fbb167fc700 1 -- 172.21.15.187:0/4087911824 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbb1c0466f0 con 0x7fbb28145f90 2020-11-20T12:19:50.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.067+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fbb28080050 con 0x7fbb28145f90 2020-11-20T12:19:50.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.075+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f9b04065f40 con 0x7f9b0409f560 2020-11-20T12:19:50.567 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:19:50.563+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:19:50.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.579+0000 7fbb167fc700 1 -- 172.21.15.187:0/4087911824 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 9343 v139) v1 ==== 95+0+0 (secure 0 0 0) 0x7fbb1c049d10 con 0x7fbb28145f90 2020-11-20T12:19:50.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.579+0000 7f9af17fa700 1 -- 172.21.15.187:0/1272021177 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6071 v139) v1 ==== 95+0+0 (secure 0 0 0) 0x7f9b0004a3e0 con 0x7f9b0409f560 2020-11-20T12:19:50.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbafc056d50 msgr2=0x7fbafc059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbafc056d50 0x7fbafc059210 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fbb18006d00 tx=0x7fbb18008040).stop 2020-11-20T12:19:50.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb28145f90 msgr2=0x7fbb281335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb28145f90 0x7fbb281335f0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 rx=0x7fbb1c00c750 tx=0x7fbb1c015000).stop 2020-11-20T12:19:50.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 shutdown_connections 2020-11-20T12:19:50.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fbafc056d50 0x7fbafc059210 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb28133ae0 0x7fbb281422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:50.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb0c0044e0 0x7fbb2814b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7fbb2759e700 1 --2- 172.21.15.187:0/4087911824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb28145f90 0x7fbb281335f0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.583+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 >> 172.21.15.187:0/4087911824 conn(0x7fbb28068370 msgr2=0x7fbb280bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.583+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 shutdown_connections 2020-11-20T12:19:50.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.583+0000 7fbb2759e700 1 -- 172.21.15.187:0/4087911824 wait complete. 2020-11-20T12:19:50.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9adc056d20 msgr2=0x7f9adc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9adc056d20 0x7f9adc0591e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f9b04141aa0 tx=0x7f9af4008040).stop 2020-11-20T12:19:50.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b0409f560 msgr2=0x7f9b0415ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b0409f560 0x7f9b0415ca80 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f9b0000b1b0 tx=0x7f9b00015000).stop 2020-11-20T12:19:50.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 shutdown_connections 2020-11-20T12:19:50.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9adc056d20 0x7f9adc0591e0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b04063e40 0x7f9b04066e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b0409f560 0x7f9b0415ca80 secure :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 rx=0x7f9b0000b1b0 tx=0x7f9b00015000).stop 2020-11-20T12:19:50.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 --2- 172.21.15.187:0/1272021177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b040bf800 0x7f9b04066220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.582+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 >> 172.21.15.187:0/1272021177 conn(0x7f9b04068370 msgr2=0x7f9b040bb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.583+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 shutdown_connections 2020-11-20T12:19:50.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.583+0000 7f9b0a248700 1 -- 172.21.15.187:0/1272021177 wait complete. 2020-11-20T12:19:50.592 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 6071 2020-11-20T12:19:50.592 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 9343 2020-11-20T12:19:50.624 DEBUG:teuthology.parallel:result is '' 2020-11-20T12:19:50.624 DEBUG:teuthology.parallel:result is '' 2020-11-20T12:19:50.625 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_metadata rm 500.00000000 2020-11-20T12:19:50.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.709+0000 7fb03a303d00 1 Processor -- start 2020-11-20T12:19:50.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.709+0000 7fb03a303d00 1 -- start start 2020-11-20T12:19:50.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.710+0000 7fb03a303d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55dc0ace87a0 0x55dc0ad915e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.710+0000 7fb03a303d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad90bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.710+0000 7fb03a303d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55dc0ad933d0 0x55dc0ad05980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.710+0000 7fb03a303d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0ac3f810 con 0x55dc0ace87a0 2020-11-20T12:19:50.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.710+0000 7fb03a303d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0acc5720 con 0x55dc0ad063f0 2020-11-20T12:19:50.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.710+0000 7fb03a303d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc0ad910b0 con 0x55dc0ad933d0 2020-11-20T12:19:50.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.711+0000 7fb0223a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad90bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.711+0000 7fb0223a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad90bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53040/0 (socket says 172.21.15.187:53040) 2020-11-20T12:19:50.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.711+0000 7fb0223a7700 1 -- 172.21.15.187:0/2975259656 learned_addr learned my addr 172.21.15.187:0/2975259656 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:50.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.711+0000 7fb0223a7700 1 -- 172.21.15.187:0/2975259656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55dc0ad933d0 msgr2=0x55dc0ad05980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:50.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.711+0000 7fb0223a7700 1 --2- 172.21.15.187:0/2975259656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55dc0ad933d0 0x55dc0ad05980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:50.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.711+0000 7fb0223a7700 1 -- 172.21.15.187:0/2975259656 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55dc0ace87a0 msgr2=0x55dc0ad915e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:50.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.711+0000 7fb0223a7700 1 --2- 172.21.15.187:0/2975259656 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55dc0ace87a0 0x55dc0ad915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:50.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.711+0000 7fb0223a7700 1 -- 172.21.15.187:0/2975259656 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc0ac694e0 con 0x55dc0ad063f0 2020-11-20T12:19:50.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb0223a7700 1 --2- 172.21.15.187:0/2975259656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad90bc0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7fb01400ab30 tx=0x7fb014005ba0).ready entity=mon.1 client_cookie=29ad884558e560fe server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb029ba6700 1 -- 172.21.15.187:0/2975259656 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01400f070 con 0x55dc0ad063f0 2020-11-20T12:19:50.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb029ba6700 1 -- 172.21.15.187:0/2975259656 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb014008ba0 con 0x55dc0ad063f0 2020-11-20T12:19:50.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb029ba6700 1 -- 172.21.15.187:0/2975259656 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014013410 con 0x55dc0ad063f0 2020-11-20T12:19:50.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb03a303d00 1 -- 172.21.15.187:0/2975259656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 msgr2=0x55dc0ad90bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb03a303d00 1 --2- 172.21.15.187:0/2975259656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad90bc0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7fb01400ab30 tx=0x7fb014005ba0).stop 2020-11-20T12:19:50.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb03a303d00 1 -- 172.21.15.187:0/2975259656 shutdown_connections 2020-11-20T12:19:50.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb03a303d00 1 --2- 172.21.15.187:0/2975259656 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55dc0ace87a0 0x55dc0ad915e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:50.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb03a303d00 1 --2- 172.21.15.187:0/2975259656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55dc0ad933d0 0x55dc0ad05980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:50.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb03a303d00 1 --2- 172.21.15.187:0/2975259656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad90bc0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.712+0000 7fb03a303d00 1 -- 172.21.15.187:0/2975259656 >> 172.21.15.187:0/2975259656 conn(0x55dc0acc2f00 msgr2=0x55dc0ace6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.713+0000 7fb03a303d00 1 -- 172.21.15.187:0/2975259656 shutdown_connections 2020-11-20T12:19:50.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.713+0000 7fb03a303d00 1 -- 172.21.15.187:0/2975259656 wait complete. 2020-11-20T12:19:50.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.713+0000 7fb03a303d00 1 Processor -- start 2020-11-20T12:19:50.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.714+0000 7fb03a303d00 1 -- start start 2020-11-20T12:19:50.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.714+0000 7fb03a303d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55dc0ace87a0 0x55dc0ad8eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.714+0000 7fb03a303d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad8d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.714+0000 7fb03a303d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55dc0ad933d0 0x55dc0ad8dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.714+0000 7fb03a303d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0ada5ec0 con 0x55dc0ace87a0 2020-11-20T12:19:50.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.714+0000 7fb03a303d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0ada2510 con 0x55dc0ad063f0 2020-11-20T12:19:50.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.714+0000 7fb03a303d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc0ada2730 con 0x55dc0ad933d0 2020-11-20T12:19:50.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb0223a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad8d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb0223a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad8d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53046/0 (socket says 172.21.15.187:53046) 2020-11-20T12:19:50.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb0223a7700 1 -- 172.21.15.187:0/3546964150 learned_addr learned my addr 172.21.15.187:0/3546964150 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:50.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb0223a7700 1 -- 172.21.15.187:0/3546964150 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55dc0ad933d0 msgr2=0x55dc0ad8dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb0223a7700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55dc0ad933d0 0x55dc0ad8dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:50.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb02aba8700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55dc0ace87a0 0x55dc0ad8eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb0223a7700 1 -- 172.21.15.187:0/3546964150 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55dc0ace87a0 msgr2=0x55dc0ad8eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb0223a7700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55dc0ace87a0 0x55dc0ad8eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.715+0000 7fb0223a7700 1 -- 172.21.15.187:0/3546964150 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc0ac64ec0 con 0x55dc0ad063f0 2020-11-20T12:19:50.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.716+0000 7fb0223a7700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad8d880 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 rx=0x7fb014012000 tx=0x7fb014018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.716+0000 7fb023fff700 1 -- 172.21.15.187:0/3546964150 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01401c650 con 0x55dc0ad063f0 2020-11-20T12:19:50.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.716+0000 7fb023fff700 1 -- 172.21.15.187:0/3546964150 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01401c7b0 con 0x55dc0ad063f0 2020-11-20T12:19:50.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.716+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55dc0ac43250 con 0x55dc0ad063f0 2020-11-20T12:19:50.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.716+0000 7fb023fff700 1 -- 172.21.15.187:0/3546964150 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01401cb50 con 0x55dc0ad063f0 2020-11-20T12:19:50.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.716+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc0ad8f3d0 con 0x55dc0ad063f0 2020-11-20T12:19:50.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.718+0000 7fb023fff700 1 -- 172.21.15.187:0/3546964150 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb01402c070 con 0x55dc0ad063f0 2020-11-20T12:19:50.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.718+0000 7fb023fff700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb00c0559a0 0x7fb00c057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.718+0000 7fb023fff700 1 -- 172.21.15.187:0/3546964150 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb0140710c0 con 0x55dc0ad063f0 2020-11-20T12:19:50.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.719+0000 7fb02aba8700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb00c0559a0 0x7fb00c057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.719+0000 7fb03a303d00 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x55dc0ad90c60 0x55dc0ad8ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.719+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- osd_op(unknown.0.0:1 27.5 27:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e142) v8 -- 0x55dc0ace6be0 con 0x55dc0ad90c60 2020-11-20T12:19:50.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.719+0000 7fb02b3a9700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x55dc0ad90c60 0x55dc0ad8ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.719+0000 7fb02aba8700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb00c0559a0 0x7fb00c057e60 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7fb01c00c2e0 tx=0x7fb01c001440).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.720+0000 7fb02b3a9700 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x55dc0ad90c60 0x55dc0ad8ba50 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.724+0000 7fb02b3a9700 1 -- 172.21.15.187:0/3546964150 <== osd.0 v2:172.21.15.187:6802/19903 1 ==== osd_op_reply(1 500.00000000 [delete] v142'7 uv7 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb024005830 con 0x55dc0ad90c60 2020-11-20T12:19:50.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.724+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x55dc0ad90c60 msgr2=0x55dc0ad8ba50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.724+0000 7fb03a303d00 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x55dc0ad90c60 0x55dc0ad8ba50 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb00c0559a0 msgr2=0x7fb00c057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb03a303d00 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb00c0559a0 0x7fb00c057e60 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7fb01c00c2e0 tx=0x7fb01c001440).stop 2020-11-20T12:19:50.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 msgr2=0x55dc0ad8d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb03a303d00 1 --2- 172.21.15.187:0/3546964150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55dc0ad063f0 0x55dc0ad8d880 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 rx=0x7fb014012000 tx=0x7fb014018000).stop 2020-11-20T12:19:50.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb02b3a9700 1 -- 172.21.15.187:0/3546964150 reap_dead start 2020-11-20T12:19:50.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 shutdown_connections 2020-11-20T12:19:50.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 >> 172.21.15.187:0/3546964150 conn(0x55dc0acc2f00 msgr2=0x55dc0ad94e50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 shutdown_connections 2020-11-20T12:19:50.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.725+0000 7fb03a303d00 1 -- 172.21.15.187:0/3546964150 wait complete. 2020-11-20T12:19:50.736 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:19:50.736 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:19:50.737 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:19:50.737 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:19:50.746 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:19:50.746 DEBUG:teuthology.parallel:result is None 2020-11-20T12:19:50.749 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:19:50.749 DEBUG:teuthology.parallel:result is None 2020-11-20T12:19:50.750 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:50.905 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:19:50.951 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:19:50.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.962+0000 7f7b038f2700 1 Processor -- start 2020-11-20T12:19:50.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.962+0000 7f7b038f2700 1 -- start start 2020-11-20T12:19:50.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.963+0000 7f7b038f2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7afc0654a0 0x7f7afc0b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.963+0000 7f7b038f2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 0x7f7afc144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.963+0000 7f7b038f2700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc0b08e0 con 0x7f7afc0b6c10 2020-11-20T12:19:50.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.963+0000 7f7b038f2700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc0b0b80 con 0x7f7afc0654a0 2020-11-20T12:19:50.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.963+0000 7f7b038f2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b05f0 con 0x7f7afc065880 2020-11-20T12:19:50.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.963+0000 7f7b0168e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7afc0654a0 0x7f7afc0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.963+0000 7f7b0168e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7afc0654a0 0x7f7afc0b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53066/0 (socket says 172.21.15.187:53066) 2020-11-20T12:19:50.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.963+0000 7f7b0168e700 1 -- 172.21.15.187:0/2903791546 learned_addr learned my addr 172.21.15.187:0/2903791546 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:50.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.964+0000 7f7b00e8d700 1 --2- 172.21.15.187:0/2903791546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 0x7f7afc144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.964+0000 7f7b0168e700 1 -- 172.21.15.187:0/2903791546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 msgr2=0x7f7afc144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:50.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.964+0000 7f7b0168e700 1 --2- 172.21.15.187:0/2903791546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 0x7f7afc144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.964+0000 7f7b0168e700 1 -- 172.21.15.187:0/2903791546 >> v1:172.21.15.187:6789/0 conn(0x7f7afc0b6c10 legacy=0x7f7afc0b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.964+0000 7f7aebfff700 1 -- 172.21.15.187:0/2903791546 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621764507 0 0) 0x7f7afc0b08e0 con 0x7f7afc0b6c10 2020-11-20T12:19:50.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.964+0000 7f7b0168e700 1 -- 172.21.15.187:0/2903791546 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0676f0 con 0x7f7afc0654a0 2020-11-20T12:19:50.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.965+0000 7f7b0168e700 1 --2- 172.21.15.187:0/2903791546 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc0b1650 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f7af8006fd0 tx=0x7f7af8006eb0).ready entity=mon.1 client_cookie=15487fb9c0517140 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.965+0000 7f7aebfff700 1 -- 172.21.15.187:0/2903791546 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af80098d0 con 0x7f7afc0654a0 2020-11-20T12:19:50.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.965+0000 7f7aebfff700 1 -- 172.21.15.187:0/2903791546 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8009a30 con 0x7f7afc0654a0 2020-11-20T12:19:50.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.965+0000 7f7aebfff700 1 -- 172.21.15.187:0/2903791546 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8009db0 con 0x7f7afc0654a0 2020-11-20T12:19:50.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.965+0000 7f7b038f2700 1 -- 172.21.15.187:0/2903791546 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 msgr2=0x7f7afc0b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.965+0000 7f7b038f2700 1 --2- 172.21.15.187:0/2903791546 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc0b1650 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f7af8006fd0 tx=0x7f7af8006eb0).stop 2020-11-20T12:19:50.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.966+0000 7f7b038f2700 1 -- 172.21.15.187:0/2903791546 shutdown_connections 2020-11-20T12:19:50.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.966+0000 7f7b038f2700 1 --2- 172.21.15.187:0/2903791546 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc0b1650 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.966+0000 7f7b038f2700 1 --2- 172.21.15.187:0/2903791546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 0x7f7afc144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.966+0000 7f7b038f2700 1 -- 172.21.15.187:0/2903791546 >> 172.21.15.187:0/2903791546 conn(0x7f7afc0bacd0 msgr2=0x7f7afc0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.966+0000 7f7b038f2700 1 -- 172.21.15.187:0/2903791546 shutdown_connections 2020-11-20T12:19:50.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.966+0000 7f7b038f2700 1 -- 172.21.15.187:0/2903791546 wait complete. 2020-11-20T12:19:50.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.967+0000 7f7b038f2700 1 Processor -- start 2020-11-20T12:19:50.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.967+0000 7f7b038f2700 1 -- start start 2020-11-20T12:19:50.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.968+0000 7f7b038f2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc09a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.968+0000 7f7b038f2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 0x7f7afc0b96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.968+0000 7f7b038f2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc0b6c10 0x7f7afc140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.968+0000 7f7b038f2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b38c0 con 0x7f7afc0b6c10 2020-11-20T12:19:50.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.968+0000 7f7b038f2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b3620 con 0x7f7afc0654a0 2020-11-20T12:19:50.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.968+0000 7f7b038f2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b3ba0 con 0x7f7afc065880 2020-11-20T12:19:50.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.968+0000 7f7b0168e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc09a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b0168e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc09a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53072/0 (socket says 172.21.15.187:53072) 2020-11-20T12:19:50.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b0168e700 1 -- 172.21.15.187:0/887568730 learned_addr learned my addr 172.21.15.187:0/887568730 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:50.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b00e8d700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 0x7f7afc0b96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b01e8f700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc0b6c10 0x7f7afc140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b0168e700 1 -- 172.21.15.187:0/887568730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 msgr2=0x7f7afc0b96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b0168e700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 0x7f7afc0b96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b0168e700 1 -- 172.21.15.187:0/887568730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc0b6c10 msgr2=0x7f7afc140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b0168e700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc0b6c10 0x7f7afc140f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b0168e700 1 -- 172.21.15.187:0/887568730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0a0ab0 con 0x7f7afc0654a0 2020-11-20T12:19:50.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.969+0000 7f7b0168e700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc09a7f0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f7afc064380 tx=0x7f7af8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.970+0000 7f7ae97fa700 1 -- 172.21.15.187:0/887568730 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8011c80 con 0x7f7afc0654a0 2020-11-20T12:19:50.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.970+0000 7f7ae97fa700 1 -- 172.21.15.187:0/887568730 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8011de0 con 0x7f7afc0654a0 2020-11-20T12:19:50.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.970+0000 7f7ae97fa700 1 -- 172.21.15.187:0/887568730 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af801e690 con 0x7f7afc0654a0 2020-11-20T12:19:50.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.970+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc09ace0 con 0x7f7afc0654a0 2020-11-20T12:19:50.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.970+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc0b3180 con 0x7f7afc0654a0 2020-11-20T12:19:50.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.971+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc063600 con 0x7f7afc0654a0 2020-11-20T12:19:50.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.971+0000 7f7ae97fa700 1 -- 172.21.15.187:0/887568730 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7af8025030 con 0x7f7afc0654a0 2020-11-20T12:19:50.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.972+0000 7f7ae97fa700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad4056d20 0x7f7ad40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.972+0000 7f7ae97fa700 1 -- 172.21.15.187:0/887568730 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7af8071750 con 0x7f7afc0654a0 2020-11-20T12:19:50.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.972+0000 7f7b00e8d700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad4056d20 0x7f7ad40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.973+0000 7f7b00e8d700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad4056d20 0x7f7ad40591e0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f7afc09f5b0 tx=0x7f7aec008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:50.976+0000 7f7ae97fa700 1 -- 172.21.15.187:0/887568730 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7af8046b50 con 0x7f7afc0654a0 2020-11-20T12:19:51.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.221+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7afc0b85a0 con 0x7f7afc0654a0 2020-11-20T12:19:51.225 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.222+0000 7f7ae97fa700 1 -- 172.21.15.187:0/887568730 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+1428 (secure 0 0 0) 0x7f7af804a170 con 0x7f7afc0654a0 2020-11-20T12:19:51.226 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:51.226 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":139,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:50.564372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:19:51.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad4056d20 msgr2=0x7f7ad40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:51.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad4056d20 0x7f7ad40591e0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f7afc09f5b0 tx=0x7f7aec008040).stop 2020-11-20T12:19:51.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 msgr2=0x7f7afc09a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:51.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc09a7f0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f7afc064380 tx=0x7f7af8015000).stop 2020-11-20T12:19:51.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 shutdown_connections 2020-11-20T12:19:51.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad4056d20 0x7f7ad40591e0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:51.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc0b6c10 0x7f7afc140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:51.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0654a0 0x7f7afc09a7f0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:51.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 --2- 172.21.15.187:0/887568730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc065880 0x7f7afc0b96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:51.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 >> 172.21.15.187:0/887568730 conn(0x7f7afc0bacd0 msgr2=0x7f7afc0b7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:51.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.224+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 shutdown_connections 2020-11-20T12:19:51.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:51.225+0000 7f7b038f2700 1 -- 172.21.15.187:0/887568730 wait complete. 2020-11-20T12:19:51.231 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 139 2020-11-20T12:19:51.451 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 139, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:19:40.033030+0000', 'modified': '2020-11-20T12:19:50.564372+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 141, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:51.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2020-11-20T12:19:52.453 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:52.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.709+0000 7f8ad62a8700 1 Processor -- start 2020-11-20T12:19:52.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.709+0000 7f8ad62a8700 1 -- start start 2020-11-20T12:19:52.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.710+0000 7f8ad62a8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8ad00b57b0 0x7f8ad0068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.710+0000 7f8ad62a8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8ad01482d0 0x7f8ad0062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.710+0000 7f8ad62a8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad0062b50 0x7f8ad0065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.710+0000 7f8ad62a8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad0144b80 con 0x7f8ad00b57b0 2020-11-20T12:19:52.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.710+0000 7f8ad62a8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad01489e0 con 0x7f8ad01482d0 2020-11-20T12:19:52.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.710+0000 7f8ad62a8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad0148c80 con 0x7f8ad0062b50 2020-11-20T12:19:52.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad4aa5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8ad01482d0 0x7f8ad0062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad4aa5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8ad01482d0 0x7f8ad0062660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53092/0 (socket says 172.21.15.187:53092) 2020-11-20T12:19:52.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad4aa5700 1 -- 172.21.15.187:0/3069694867 learned_addr learned my addr 172.21.15.187:0/3069694867 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:52.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad52a6700 1 --2- 172.21.15.187:0/3069694867 >> v2:172.21.15.5:3300/0 conn(0x7f8ad00b57b0 0x7f8ad0068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad4aa5700 1 -- 172.21.15.187:0/3069694867 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad0062b50 msgr2=0x7f8ad0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:52.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad4aa5700 1 --2- 172.21.15.187:0/3069694867 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad0062b50 0x7f8ad0065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:52.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad4aa5700 1 -- 172.21.15.187:0/3069694867 >> v2:172.21.15.5:3300/0 conn(0x7f8ad00b57b0 msgr2=0x7f8ad0068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:52.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad4aa5700 1 --2- 172.21.15.187:0/3069694867 >> v2:172.21.15.5:3300/0 conn(0x7f8ad00b57b0 0x7f8ad0068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad4aa5700 1 -- 172.21.15.187:0/3069694867 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad0148f20 con 0x7f8ad01482d0 2020-11-20T12:19:52.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.711+0000 7f8ad52a6700 1 --2- 172.21.15.187:0/3069694867 >> v2:172.21.15.5:3300/0 conn(0x7f8ad00b57b0 0x7f8ad0068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:52.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.712+0000 7f8ad4aa5700 1 --2- 172.21.15.187:0/3069694867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad01482d0 0x7f8ad0062660 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 rx=0x7f8ad009d820 tx=0x7f8ac0006eb0).ready entity=mon.1 client_cookie=7b873a7068003cf9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:52.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.712+0000 7f8abffff700 1 -- 172.21.15.187:0/3069694867 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac00098d0 con 0x7f8ad01482d0 2020-11-20T12:19:52.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.712+0000 7f8abffff700 1 -- 172.21.15.187:0/3069694867 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac0009a30 con 0x7f8ad01482d0 2020-11-20T12:19:52.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.712+0000 7f8abffff700 1 -- 172.21.15.187:0/3069694867 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0009db0 con 0x7f8ad01482d0 2020-11-20T12:19:52.720 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.712+0000 7f8ad62a8700 1 -- 172.21.15.187:0/3069694867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad01482d0 msgr2=0x7f8ad0062660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:52.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.712+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/3069694867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad01482d0 0x7f8ad0062660 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 rx=0x7f8ad009d820 tx=0x7f8ac0006eb0).stop 2020-11-20T12:19:52.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.712+0000 7f8ad62a8700 1 -- 172.21.15.187:0/3069694867 shutdown_connections 2020-11-20T12:19:52.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.713+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/3069694867 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad0062b50 0x7f8ad0065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:52.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.713+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/3069694867 >> v2:172.21.15.5:3300/0 conn(0x7f8ad00b57b0 0x7f8ad0068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.713+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/3069694867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad01482d0 0x7f8ad0062660 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.713+0000 7f8ad62a8700 1 -- 172.21.15.187:0/3069694867 >> 172.21.15.187:0/3069694867 conn(0x7f8ad00bab30 msgr2=0x7f8ad00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:52.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.713+0000 7f8ad62a8700 1 -- 172.21.15.187:0/3069694867 shutdown_connections 2020-11-20T12:19:52.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.713+0000 7f8ad62a8700 1 -- 172.21.15.187:0/3069694867 wait complete. 2020-11-20T12:19:52.722 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.714+0000 7f8ad62a8700 1 Processor -- start 2020-11-20T12:19:52.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.714+0000 7f8ad62a8700 1 -- start start 2020-11-20T12:19:52.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad62a8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad0062b50 0x7f8ad009ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad62a8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad00b57b0 0x7f8ad00b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad62a8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ad01482d0 0x7f8ad00b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.728 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad62a8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad0139c90 con 0x7f8ad01482d0 2020-11-20T12:19:52.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad62a8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad013c840 con 0x7f8ad0062b50 2020-11-20T12:19:52.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad62a8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad01395f0 con 0x7f8ad00b57b0 2020-11-20T12:19:52.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad52a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad0062b50 0x7f8ad009ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad52a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad0062b50 0x7f8ad009ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53096/0 (socket says 172.21.15.187:53096) 2020-11-20T12:19:52.729 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.715+0000 7f8ad52a6700 1 -- 172.21.15.187:0/4193414526 learned_addr learned my addr 172.21.15.187:0/4193414526 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:52.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.716+0000 7f8ad4aa5700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad00b57b0 0x7f8ad00b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.716+0000 7f8ad52a6700 1 -- 172.21.15.187:0/4193414526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad00b57b0 msgr2=0x7f8ad00b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:52.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.716+0000 7f8ad52a6700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad00b57b0 0x7f8ad00b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.716+0000 7f8ad5aa7700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ad01482d0 0x7f8ad00b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.716+0000 7f8ad52a6700 1 -- 172.21.15.187:0/4193414526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ad01482d0 msgr2=0x7f8ad00b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:52.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.716+0000 7f8ad52a6700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ad01482d0 0x7f8ad00b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.716+0000 7f8ad52a6700 1 -- 172.21.15.187:0/4193414526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad013ca30 con 0x7f8ad0062b50 2020-11-20T12:19:52.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.716+0000 7f8ad52a6700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad0062b50 0x7f8ad009ed70 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f8acc00cb50 tx=0x7f8acc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:52.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.717+0000 7f8abd7fa700 1 -- 172.21.15.187:0/4193414526 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc0077d0 con 0x7f8ad0062b50 2020-11-20T12:19:52.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.717+0000 7f8abd7fa700 1 -- 172.21.15.187:0/4193414526 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc007930 con 0x7f8ad0062b50 2020-11-20T12:19:52.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.717+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad00b4780 con 0x7f8ad0062b50 2020-11-20T12:19:52.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.717+0000 7f8abd7fa700 1 -- 172.21.15.187:0/4193414526 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc007cb0 con 0x7f8ad0062b50 2020-11-20T12:19:52.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.717+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad00b1bb0 con 0x7f8ad0062b50 2020-11-20T12:19:52.732 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.718+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad0139ad0 con 0x7f8ad0062b50 2020-11-20T12:19:52.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.718+0000 7f8abd7fa700 1 -- 172.21.15.187:0/4193414526 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8acc011620 con 0x7f8ad0062b50 2020-11-20T12:19:52.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.719+0000 7f8abd7fa700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8aa8056d20 0x7f8aa80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.719+0000 7f8abd7fa700 1 -- 172.21.15.187:0/4193414526 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8acc0702c0 con 0x7f8ad0062b50 2020-11-20T12:19:52.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.720+0000 7f8ad4aa5700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8aa8056d20 0x7f8aa80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.733 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.720+0000 7f8ad4aa5700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8aa8056d20 0x7f8aa80591e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f8ad0147d70 tx=0x7f8ac0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:52.734 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.723+0000 7f8abd7fa700 1 -- 172.21.15.187:0/4193414526 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8acc045640 con 0x7f8ad0062b50 2020-11-20T12:19:52.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.973+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ad0139050 con 0x7f8ad0062b50 2020-11-20T12:19:53.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.974+0000 7f8abd7fa700 1 -- 172.21.15.187:0/4193414526 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+1797 (secure 0 0 0) 0x7f8acc048c60 con 0x7f8ad0062b50 2020-11-20T12:19:53.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8aa8056d20 msgr2=0x7f8aa80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:53.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8aa8056d20 0x7f8aa80591e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f8ad0147d70 tx=0x7f8ac0015000).stop 2020-11-20T12:19:53.130 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad0062b50 msgr2=0x7f8ad009ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:53.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad0062b50 0x7f8ad009ed70 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f8acc00cb50 tx=0x7f8acc008040).stop 2020-11-20T12:19:53.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 shutdown_connections 2020-11-20T12:19:53.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8aa8056d20 0x7f8aa80591e0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:53.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ad01482d0 0x7f8ad00b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:53.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ad0062b50 0x7f8ad009ed70 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:53.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 --2- 172.21.15.187:0/4193414526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ad00b57b0 0x7f8ad00b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:53.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 >> 172.21.15.187:0/4193414526 conn(0x7f8ad00bab30 msgr2=0x7f8ad0066070 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:53.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 shutdown_connections 2020-11-20T12:19:53.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:52.976+0000 7f8ad62a8700 1 -- 172.21.15.187:0/4193414526 wait complete. 2020-11-20T12:19:53.133 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 142 2020-11-20T12:19:53.133 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:53.133 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":142,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:52.588361+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9511":{"gid":9511,"name":"a","rank":0,"incarnation":141,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.187:6835/3910175723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3910175723},{"type":"v1","addr":"172.21.15.187:6835","nonce":3910175723}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:19:53.335 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 142, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:19:40.033030+0000', 'modified': '2020-11-20T12:19:52.588361+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 141, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9511}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9511': {'gid': 9511, 'name': 'a', 'rank': 0, 'incarnation': 141, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.187:6835/3910175723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3910175723}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3910175723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:53.336 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9511:up:reconnect 2020-11-20T12:19:53.780 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:53.793 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:54.337 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:54.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.607+0000 7f22d52fa700 1 Processor -- start 2020-11-20T12:19:54.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.607+0000 7f22d52fa700 1 -- start start 2020-11-20T12:19:54.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.609+0000 7f22d52fa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f22d00bf800 0x7f22d014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:54.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.609+0000 7f22d52fa700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f22d009f560 0x7f22d00a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:54.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.609+0000 7f22d52fa700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d014d110 con 0x7f22d00bfbe0 2020-11-20T12:19:54.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.609+0000 7f22d52fa700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d014cbd0 con 0x7f22d009f560 2020-11-20T12:19:54.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.609+0000 7f22d52fa700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d014ce70 con 0x7f22d00bf800 2020-11-20T12:19:54.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.609+0000 7f22cf7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f22d00bfbe0 0x7f22d01480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49710/0 (socket says 172.21.15.187:49710) 2020-11-20T12:19:54.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.609+0000 7f22cf7fe700 1 -- 172.21.15.187:0/1290824333 learned_addr learned my addr 172.21.15.187:0/1290824333 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:54.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.609+0000 7f22cffff700 1 --2- 172.21.15.187:0/1290824333 >> v2:172.21.15.187:3300/0 conn(0x7f22d00bf800 0x7f22d014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:54.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.610+0000 7f22d4af9700 1 --2- 172.21.15.187:0/1290824333 >> v2:172.21.15.5:3300/0 conn(0x7f22d009f560 0x7f22d00a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:54.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.610+0000 7f22cffff700 1 -- 172.21.15.187:0/1290824333 >> v2:172.21.15.5:3300/0 conn(0x7f22d009f560 msgr2=0x7f22d00a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:54.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.610+0000 7f22cffff700 1 --2- 172.21.15.187:0/1290824333 >> v2:172.21.15.5:3300/0 conn(0x7f22d009f560 0x7f22d00a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.610+0000 7f22cffff700 1 -- 172.21.15.187:0/1290824333 >> v1:172.21.15.187:6789/0 conn(0x7f22d00bfbe0 legacy=0x7f22d01480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:54.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.610+0000 7f22cffff700 1 -- 172.21.15.187:0/1290824333 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d00a4cc0 con 0x7f22d00bf800 2020-11-20T12:19:54.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.610+0000 7f22cffff700 1 --2- 172.21.15.187:0/1290824333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d014c2c0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7f22d0068780 tx=0x7f22c0006df0).ready entity=mon.1 client_cookie=4b546e61fae76f7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:54.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22ceffd700 1 -- 172.21.15.187:0/1290824333 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0009840 con 0x7f22d00bf800 2020-11-20T12:19:54.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22ceffd700 1 -- 172.21.15.187:0/1290824333 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c00099a0 con 0x7f22d00bf800 2020-11-20T12:19:54.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22ceffd700 1 -- 172.21.15.187:0/1290824333 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0009d20 con 0x7f22d00bf800 2020-11-20T12:19:54.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22d52fa700 1 -- 172.21.15.187:0/1290824333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 msgr2=0x7f22d014c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:54.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22d52fa700 1 --2- 172.21.15.187:0/1290824333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d014c2c0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7f22d0068780 tx=0x7f22c0006df0).stop 2020-11-20T12:19:54.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22d52fa700 1 -- 172.21.15.187:0/1290824333 shutdown_connections 2020-11-20T12:19:54.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22d52fa700 1 --2- 172.21.15.187:0/1290824333 >> v2:172.21.15.5:3300/0 conn(0x7f22d009f560 0x7f22d00a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22d52fa700 1 --2- 172.21.15.187:0/1290824333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d014c2c0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22d52fa700 1 -- 172.21.15.187:0/1290824333 >> 172.21.15.187:0/1290824333 conn(0x7f22d0068370 msgr2=0x7f22d00bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:54.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22d52fa700 1 -- 172.21.15.187:0/1290824333 shutdown_connections 2020-11-20T12:19:54.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.611+0000 7f22d52fa700 1 -- 172.21.15.187:0/1290824333 wait complete. 2020-11-20T12:19:54.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.613+0000 7f22d52fa700 1 Processor -- start 2020-11-20T12:19:54.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.613+0000 7f22d52fa700 1 -- start start 2020-11-20T12:19:54.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22d52fa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22d009f560 0x7f22d00bb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:54.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22d52fa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d0144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:54.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22d52fa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22d00b4b20 0x7f22d0145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:54.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22d52fa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d0146f80 con 0x7f22d00b4b20 2020-11-20T12:19:54.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22d52fa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d015c1c0 con 0x7f22d00bf800 2020-11-20T12:19:54.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22d52fa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d00bbe80 con 0x7f22d009f560 2020-11-20T12:19:54.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22cf7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d0144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:54.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22cf7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d0144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53112/0 (socket says 172.21.15.187:53112) 2020-11-20T12:19:54.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.614+0000 7f22cf7fe700 1 -- 172.21.15.187:0/3269421923 learned_addr learned my addr 172.21.15.187:0/3269421923 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:54.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22d4af9700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22d00b4b20 0x7f22d0145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:54.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22cffff700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22d009f560 0x7f22d00bb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:54.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22cf7fe700 1 -- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22d009f560 msgr2=0x7f22d00bb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:54.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22cf7fe700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22d009f560 0x7f22d00bb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22cf7fe700 1 -- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22d00b4b20 msgr2=0x7f22d0145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:54.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22cf7fe700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22d00b4b20 0x7f22d0145320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22cf7fe700 1 -- 172.21.15.187:0/3269421923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d0146bc0 con 0x7f22d00bf800 2020-11-20T12:19:54.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22d4af9700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22d00b4b20 0x7f22d0145320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:19:54.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.615+0000 7f22cf7fe700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d0144e30 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 rx=0x7f22c400c710 tx=0x7f22c4007c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:54.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.616+0000 7f22affff700 1 -- 172.21.15.187:0/3269421923 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c40072e0 con 0x7f22d00bf800 2020-11-20T12:19:54.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.616+0000 7f22affff700 1 -- 172.21.15.187:0/3269421923 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c4007440 con 0x7f22d00bf800 2020-11-20T12:19:54.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.616+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d00b4f30 con 0x7f22d00bf800 2020-11-20T12:19:54.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.616+0000 7f22affff700 1 -- 172.21.15.187:0/3269421923 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c4005820 con 0x7f22d00bf800 2020-11-20T12:19:54.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.616+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d015c820 con 0x7f22d00bf800 2020-11-20T12:19:54.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.617+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d0080050 con 0x7f22d00bf800 2020-11-20T12:19:54.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.617+0000 7f22affff700 1 -- 172.21.15.187:0/3269421923 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f22c4010070 con 0x7f22d00bf800 2020-11-20T12:19:54.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.618+0000 7f22affff700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22a8056d20 0x7f22a80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:54.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.618+0000 7f22affff700 1 -- 172.21.15.187:0/3269421923 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f22c406f610 con 0x7f22d00bf800 2020-11-20T12:19:54.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.619+0000 7f22cffff700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22a8056d20 0x7f22a80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:54.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.619+0000 7f22cffff700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22a8056d20 0x7f22a80591e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f22d0147d10 tx=0x7f22c0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:54.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.622+0000 7f22affff700 1 -- 172.21.15.187:0/3269421923 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f22c4044a10 con 0x7f22d00bf800 2020-11-20T12:19:54.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.870+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22d015c350 con 0x7f22d00bf800 2020-11-20T12:19:54.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.871+0000 7f22affff700 1 -- 172.21.15.187:0/3269421923 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+1800 (secure 0 0 0) 0x7f22c4018020 con 0x7f22d00bf800 2020-11-20T12:19:54.874 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:54.875 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":144,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:54.611519+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9511":{"gid":9511,"name":"a","rank":0,"incarnation":141,"state":"up:clientreplay","state_seq":4,"addr":"172.21.15.187:6835/3910175723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3910175723},{"type":"v1","addr":"172.21.15.187:6835","nonce":3910175723}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:19:54.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.873+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22a8056d20 msgr2=0x7f22a80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:54.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.873+0000 7f22d52fa700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22a8056d20 0x7f22a80591e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f22d0147d10 tx=0x7f22c0015000).stop 2020-11-20T12:19:54.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.873+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 msgr2=0x7f22d0144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:54.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.873+0000 7f22d52fa700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d0144e30 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 rx=0x7f22c400c710 tx=0x7f22c4007c60).stop 2020-11-20T12:19:54.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.874+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 shutdown_connections 2020-11-20T12:19:54.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.874+0000 7f22d52fa700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22a8056d20 0x7f22a80591e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.874+0000 7f22d52fa700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22d00b4b20 0x7f22d0145320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.874+0000 7f22d52fa700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22d009f560 0x7f22d00bb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.874+0000 7f22d52fa700 1 --2- 172.21.15.187:0/3269421923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22d00bf800 0x7f22d0144e30 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:54.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.874+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 >> 172.21.15.187:0/3269421923 conn(0x7f22d0068370 msgr2=0x7f22d01433e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:54.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.874+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 shutdown_connections 2020-11-20T12:19:54.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:54.874+0000 7f22d52fa700 1 -- 172.21.15.187:0/3269421923 wait complete. 2020-11-20T12:19:54.881 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 144 2020-11-20T12:19:55.104 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:19:40.033030+0000', 'modified': '2020-11-20T12:19:54.611519+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 141, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9511}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9511': {'gid': 9511, 'name': 'a', 'rank': 0, 'incarnation': 141, 'state': 'up:clientreplay', 'state_seq': 4, 'addr': '172.21.15.187:6835/3910175723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3910175723}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3910175723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:55.105 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9511:up:clientreplay 2020-11-20T12:19:56.106 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:56.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.353+0000 7f52131a5700 1 Processor -- start 2020-11-20T12:19:56.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.353+0000 7f52131a5700 1 -- start start 2020-11-20T12:19:56.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.355+0000 7f52131a5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f520c0bf800 0x7f520c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.355+0000 7f52131a5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f520c0bfbe0 0x7f520c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.355+0000 7f52131a5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c14d110 con 0x7f520c09f560 2020-11-20T12:19:56.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.355+0000 7f52131a5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c14cbd0 con 0x7f520c0bf800 2020-11-20T12:19:56.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.355+0000 7f52131a5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c14ce70 con 0x7f520c0bfbe0 2020-11-20T12:19:56.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.355+0000 7f52129a4700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f520c09f560 0x7f520c060c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49728/0 (socket says 172.21.15.187:49728) 2020-11-20T12:19:56.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.355+0000 7f52129a4700 1 -- 172.21.15.187:0/938105113 learned_addr learned my addr 172.21.15.187:0/938105113 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:56.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.356+0000 7f52119a2700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f520c0bfbe0 0x7f520c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:56.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.356+0000 7f52121a3700 1 --2- 172.21.15.187:0/938105113 >> v2:172.21.15.5:3300/0 conn(0x7f520c0bf800 0x7f520c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:56.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.356+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408936880 0 0) 0x7f520c14d110 con 0x7f520c09f560 2020-11-20T12:19:56.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.356+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f4003240 con 0x7f520c09f560 2020-11-20T12:19:56.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.356+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 620469356 0 0) 0x7f51f4003240 con 0x7f520c09f560 2020-11-20T12:19:56.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.356+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f520c0bfbe0 msgr2=0x7f520c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:56.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f520c0bfbe0 0x7f520c1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:56.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 >> v2:172.21.15.5:3300/0 conn(0x7f520c0bf800 msgr2=0x7f520c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:56.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 --2- 172.21.15.187:0/938105113 >> v2:172.21.15.5:3300/0 conn(0x7f520c0bf800 0x7f520c14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:56.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c0a4cc0 con 0x7f520c09f560 2020-11-20T12:19:56.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f52040014e0 con 0x7f520c09f560 2020-11-20T12:19:56.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 >> v1:172.21.15.187:6789/0 conn(0x7f520c09f560 legacy=0x7f520c060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:56.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51f40037b0 0x7f51f4003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f51f40044e0 0x7f51f4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51f4004e50 0x7f51f400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c14cbd0 con 0x7f51f40044e0 2020-11-20T12:19:56.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c14ce70 con 0x7f51f4004e50 2020-11-20T12:19:56.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f4004200 con 0x7f51f40037b0 2020-11-20T12:19:56.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.357+0000 7f52129a4700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51f4004e50 0x7f51f400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:56.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52119a2700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f51f40044e0 0x7f51f4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:56.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52121a3700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51f40037b0 0x7f51f4003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:56.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52129a4700 1 -- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51f40037b0 msgr2=0x7f51f4003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:56.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52129a4700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51f40037b0 0x7f51f4003b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:56.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52129a4700 1 -- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f51f40044e0 msgr2=0x7f51f4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:56.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52129a4700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f51f40044e0 0x7f51f4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:56.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52129a4700 1 -- 172.21.15.187:0/938105113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f4003240 con 0x7f51f4004e50 2020-11-20T12:19:56.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52129a4700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51f4004e50 0x7f51f400b730 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7f520400c210 tx=0x7f5204007750).ready entity=mon.1 client_cookie=ce76cc817900c634 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:56.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52040070d0 con 0x7f51f4004e50 2020-11-20T12:19:56.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5204007230 con 0x7f51f4004e50 2020-11-20T12:19:56.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.358+0000 7f52111a1700 1 -- 172.21.15.187:0/938105113 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52040055f0 con 0x7f51f4004e50 2020-11-20T12:19:56.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.359+0000 7f52131a5700 1 -- 172.21.15.187:0/938105113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51f4004e50 msgr2=0x7f51f400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:56.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.359+0000 7f52131a5700 1 --2- 172.21.15.187:0/938105113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51f4004e50 0x7f51f400b730 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7f520400c210 tx=0x7f5204007750).stop 2020-11-20T12:19:56.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.359+0000 7f52129a4700 1 -- 172.21.15.187:0/938105113 reap_dead start 2020-11-20T12:19:56.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.359+0000 7f52131a5700 1 -- 172.21.15.187:0/938105113 shutdown_connections 2020-11-20T12:19:56.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.359+0000 7f52131a5700 1 -- 172.21.15.187:0/938105113 >> 172.21.15.187:0/938105113 conn(0x7f520c068370 msgr2=0x7f520c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:56.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.359+0000 7f52131a5700 1 -- 172.21.15.187:0/938105113 shutdown_connections 2020-11-20T12:19:56.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.359+0000 7f52131a5700 1 -- 172.21.15.187:0/938105113 wait complete. 2020-11-20T12:19:56.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.361+0000 7f52131a5700 1 Processor -- start 2020-11-20T12:19:56.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.361+0000 7f52131a5700 1 -- start start 2020-11-20T12:19:56.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.361+0000 7f52131a5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51f4004e50 0x7f520c158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52131a5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c130520 0x7f520c1430e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52131a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f520c1435d0 0x7f520c1440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52131a5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c158d20 con 0x7f520c130520 2020-11-20T12:19:56.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52131a5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c14b3d0 con 0x7f520c1435d0 2020-11-20T12:19:56.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52131a5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c14b670 con 0x7f51f4004e50 2020-11-20T12:19:56.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52129a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f520c1435d0 0x7f520c1440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:56.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52129a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f520c1435d0 0x7f520c1440e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53136/0 (socket says 172.21.15.187:53136) 2020-11-20T12:19:56.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52129a4700 1 -- 172.21.15.187:0/4068100869 learned_addr learned my addr 172.21.15.187:0/4068100869 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:56.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52129a4700 1 -- 172.21.15.187:0/4068100869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51f4004e50 msgr2=0x7f520c158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:56.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52129a4700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51f4004e50 0x7f520c158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:56.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52129a4700 1 -- 172.21.15.187:0/4068100869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c130520 msgr2=0x7f520c1430e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:56.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52129a4700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c130520 0x7f520c1430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:56.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.362+0000 7f52129a4700 1 -- 172.21.15.187:0/4068100869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c14b910 con 0x7f520c1435d0 2020-11-20T12:19:56.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.363+0000 7f52129a4700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f520c1435d0 0x7f520c1440e0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7f520400c1e0 tx=0x7f5204015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:56.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.363+0000 7f51fa7fc700 1 -- 172.21.15.187:0/4068100869 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204013450 con 0x7f520c1435d0 2020-11-20T12:19:56.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.363+0000 7f51fa7fc700 1 -- 172.21.15.187:0/4068100869 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52040135b0 con 0x7f520c1435d0 2020-11-20T12:19:56.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.363+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c145300 con 0x7f520c1435d0 2020-11-20T12:19:56.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.363+0000 7f51fa7fc700 1 -- 172.21.15.187:0/4068100869 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204013950 con 0x7f520c1435d0 2020-11-20T12:19:56.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.363+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c1445d0 con 0x7f520c1435d0 2020-11-20T12:19:56.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.364+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c080050 con 0x7f520c1435d0 2020-11-20T12:19:56.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.365+0000 7f51fa7fc700 1 -- 172.21.15.187:0/4068100869 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5204010070 con 0x7f520c1435d0 2020-11-20T12:19:56.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.365+0000 7f51fa7fc700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e4056d20 0x7f51e40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:56.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.365+0000 7f51fa7fc700 1 -- 172.21.15.187:0/4068100869 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f5204070e80 con 0x7f520c1435d0 2020-11-20T12:19:56.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.366+0000 7f52121a3700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e4056d20 0x7f51e40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:56.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.366+0000 7f52121a3700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e4056d20 0x7f51e40591e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f5208006df0 tx=0x7f520800b040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:56.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.369+0000 7f51fa7fc700 1 -- 172.21.15.187:0/4068100869 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5204046170 con 0x7f520c1435d0 2020-11-20T12:19:56.600 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:19:56.597+0000 7f7d6550a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:19:56.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.614+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f520c14baa0 con 0x7f520c1435d0 2020-11-20T12:19:56.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.615+0000 7f51fa7fc700 1 -- 172.21.15.187:0/4068100869 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f5204049790 con 0x7f520c1435d0 2020-11-20T12:19:56.620 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:19:56.620 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":145,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7032,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3018710892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3018710892},{"type":"v1","addr":"172.21.15.5:6833","nonce":3018710892}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:55.627158+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9511":{"gid":9511,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3910175723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3910175723},{"type":"v1","addr":"172.21.15.187:6835","nonce":3910175723}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:19:56.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.617+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e4056d20 msgr2=0x7f51e40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:56.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.617+0000 7f52131a5700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e4056d20 0x7f51e40591e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f5208006df0 tx=0x7f520800b040).stop 2020-11-20T12:19:56.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.617+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f520c1435d0 msgr2=0x7f520c1440e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:56.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.617+0000 7f52131a5700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f520c1435d0 0x7f520c1440e0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7f520400c1e0 tx=0x7f5204015000).stop 2020-11-20T12:19:56.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.618+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 shutdown_connections 2020-11-20T12:19:56.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.618+0000 7f52131a5700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51e4056d20 0x7f51e40591e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:56.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.618+0000 7f52131a5700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f520c1435d0 0x7f520c1440e0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:56.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.618+0000 7f52131a5700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51f4004e50 0x7f520c158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:56.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.618+0000 7f52131a5700 1 --2- 172.21.15.187:0/4068100869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c130520 0x7f520c1430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:56.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.618+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 >> 172.21.15.187:0/4068100869 conn(0x7f520c068370 msgr2=0x7f520c060c90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:56.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.618+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 shutdown_connections 2020-11-20T12:19:56.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:56.618+0000 7f52131a5700 1 -- 172.21.15.187:0/4068100869 wait complete. 2020-11-20T12:19:56.629 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 145 2020-11-20T12:19:56.850 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 145, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:19:40.033030+0000', 'modified': '2020-11-20T12:19:55.627158+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 141, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9511}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9511': {'gid': 9511, 'name': 'a', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.187:6835/3910175723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3910175723}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3910175723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:19:56.850 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:19:56.851 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:19:57.136 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":9511,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":5.5183228809999996,"mdsmap_epoch":145,"osdmap_epoch":142,"osdmap_epoch_barrier":142,"uptime":6.2057315380000002} 2020-11-20T12:19:57.137 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 9511, "mdsmap_epoch": 145, "osdmap_epoch": 142, "osdmap_epoch_barrier": 142, "rank_uptime": 5.518322881, "state": "up:active", "uptime": 6.205731538, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:19:57.137 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:19:57.138 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:19:57.252 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:19:57.270 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:19:57.374 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:19:57.398 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:19:57.399 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:19:57.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:19:57.400 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:19:57.527 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:19:57.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:19:57.528 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:19:57.584 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:19:57.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.822+0000 7f12694fe700 1 Processor -- start 2020-11-20T12:19:57.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.822+0000 7f12694fe700 1 -- start start 2020-11-20T12:19:57.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.823+0000 7f12694fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f12641484d0 0x7f1264061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.823+0000 7f12694fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1264066f70 0x7f12640b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.823+0000 7f12694fe700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b1300 con 0x7f12640622c0 2020-11-20T12:19:57.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.823+0000 7f12694fe700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1264067730 con 0x7f1264066f70 2020-11-20T12:19:57.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.823+0000 7f12694fe700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12640b15a0 con 0x7f12641484d0 2020-11-20T12:19:57.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.824+0000 7f12637fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f12640622c0 0x7f12640b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49748/0 (socket says 172.21.15.187:49748) 2020-11-20T12:19:57.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.824+0000 7f12637fe700 1 -- 172.21.15.187:0/765311872 learned_addr learned my addr 172.21.15.187:0/765311872 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:19:57.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.824+0000 7f1263fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f12641484d0 0x7f1264061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.824+0000 7f1268cfd700 1 --2- 172.21.15.187:0/765311872 >> v2:172.21.15.5:3300/0 conn(0x7f1264066f70 0x7f12640b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.824+0000 7f1262ffd700 1 -- 172.21.15.187:0/765311872 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884600991 0 0) 0x7f12640b1300 con 0x7f12640622c0 2020-11-20T12:19:57.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.825+0000 7f1262ffd700 1 -- 172.21.15.187:0/765311872 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1250003240 con 0x7f12640622c0 2020-11-20T12:19:57.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.825+0000 7f1263fff700 1 -- 172.21.15.187:0/765311872 >> v2:172.21.15.5:3300/0 conn(0x7f1264066f70 msgr2=0x7f12640b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:57.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.825+0000 7f1263fff700 1 --2- 172.21.15.187:0/765311872 >> v2:172.21.15.5:3300/0 conn(0x7f1264066f70 0x7f12640b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.825+0000 7f1263fff700 1 -- 172.21.15.187:0/765311872 >> v1:172.21.15.187:6789/0 conn(0x7f12640622c0 legacy=0x7f12640b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:57.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.825+0000 7f1263fff700 1 -- 172.21.15.187:0/765311872 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640b1840 con 0x7f12641484d0 2020-11-20T12:19:57.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.825+0000 7f1263fff700 1 --2- 172.21.15.187:0/765311872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12641484d0 0x7f1264061dd0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7f12640b85e0 tx=0x7f1254006df0).ready entity=mon.1 client_cookie=2a2451f74471c4ec server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:57.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.825+0000 7f1262ffd700 1 -- 172.21.15.187:0/765311872 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254009640 con 0x7f12641484d0 2020-11-20T12:19:57.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f1262ffd700 1 -- 172.21.15.187:0/765311872 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12540097a0 con 0x7f12641484d0 2020-11-20T12:19:57.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f1262ffd700 1 -- 172.21.15.187:0/765311872 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254009b20 con 0x7f12641484d0 2020-11-20T12:19:57.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f12694fe700 1 -- 172.21.15.187:0/765311872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12641484d0 msgr2=0x7f1264061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:57.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f12694fe700 1 --2- 172.21.15.187:0/765311872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12641484d0 0x7f1264061dd0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7f12640b85e0 tx=0x7f1254006df0).stop 2020-11-20T12:19:57.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f12694fe700 1 -- 172.21.15.187:0/765311872 shutdown_connections 2020-11-20T12:19:57.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f12694fe700 1 --2- 172.21.15.187:0/765311872 >> v2:172.21.15.5:3300/0 conn(0x7f1264066f70 0x7f12640b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f12694fe700 1 --2- 172.21.15.187:0/765311872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12641484d0 0x7f1264061dd0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f12694fe700 1 -- 172.21.15.187:0/765311872 >> 172.21.15.187:0/765311872 conn(0x7f12640bab30 msgr2=0x7f12640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:57.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f12694fe700 1 -- 172.21.15.187:0/765311872 shutdown_connections 2020-11-20T12:19:57.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.826+0000 7f12694fe700 1 -- 172.21.15.187:0/765311872 wait complete. 2020-11-20T12:19:57.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.828+0000 7f12694fe700 1 Processor -- start 2020-11-20T12:19:57.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.828+0000 7f12694fe700 1 -- start start 2020-11-20T12:19:57.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12694fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12640622c0 0x7f1264158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12694fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1264066f70 0x7f12640b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12694fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12641484d0 0x7f126413f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12694fe700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126413e970 con 0x7f12640622c0 2020-11-20T12:19:57.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12694fe700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126413ec10 con 0x7f1264066f70 2020-11-20T12:19:57.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12694fe700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126413eeb0 con 0x7f12641484d0 2020-11-20T12:19:57.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12637fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1264066f70 0x7f12640b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12637fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1264066f70 0x7f12640b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53148/0 (socket says 172.21.15.187:53148) 2020-11-20T12:19:57.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.829+0000 7f12637fe700 1 -- 172.21.15.187:0/3912277401 learned_addr learned my addr 172.21.15.187:0/3912277401 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:57.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f12637fe700 1 -- 172.21.15.187:0/3912277401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12641484d0 msgr2=0x7f126413f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:57.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f12637fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12641484d0 0x7f126413f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:57.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f12637fe700 1 -- 172.21.15.187:0/3912277401 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12640622c0 msgr2=0x7f1264158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:57.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f1263fff700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12640622c0 0x7f1264158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f12637fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12640622c0 0x7f1264158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f12637fe700 1 -- 172.21.15.187:0/3912277401 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640b4710 con 0x7f1264066f70 2020-11-20T12:19:57.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f12637fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1264066f70 0x7f12640b4220 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7f125800c710 tx=0x7f1258007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:57.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f1243fff700 1 -- 172.21.15.187:0/3912277401 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12580072e0 con 0x7f1264066f70 2020-11-20T12:19:57.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f1243fff700 1 -- 172.21.15.187:0/3912277401 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1258007440 con 0x7f1264066f70 2020-11-20T12:19:57.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126413f840 con 0x7f1264066f70 2020-11-20T12:19:57.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f1243fff700 1 -- 172.21.15.187:0/3912277401 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258005820 con 0x7f1264066f70 2020-11-20T12:19:57.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.830+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126409c280 con 0x7f1264066f70 2020-11-20T12:19:57.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.832+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1264159020 con 0x7f1264066f70 2020-11-20T12:19:57.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.832+0000 7f1243fff700 1 -- 172.21.15.187:0/3912277401 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1258010070 con 0x7f1264066f70 2020-11-20T12:19:57.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.833+0000 7f1243fff700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.833+0000 7f1243fff700 1 -- 172.21.15.187:0/3912277401 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f125806f670 con 0x7f1264066f70 2020-11-20T12:19:57.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.833+0000 7f1263fff700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.834+0000 7f1263fff700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f1254009340 tx=0x7f1254015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:57.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:57.838+0000 7f1243fff700 1 -- 172.21.15.187:0/3912277401 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1264159020 con 0x7f1264066f70 2020-11-20T12:19:58.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.027+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1264159020 con 0x7f1264066f70 2020-11-20T12:19:58.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.039+0000 7f1243fff700 1 -- 172.21.15.187:0/3912277401 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1264159020 con 0x7f1264066f70 2020-11-20T12:19:58.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 msgr2=0x7f123c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f1254009340 tx=0x7f1254015000).stop 2020-11-20T12:19:58.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1264066f70 msgr2=0x7f12640b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1264066f70 0x7f12640b4220 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7f125800c710 tx=0x7f1258007a60).stop 2020-11-20T12:19:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 shutdown_connections 2020-11-20T12:19:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f123c056d20 0x7f123c0591e0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12640622c0 0x7f1264158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12641484d0 0x7f126413f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:58.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 --2- 172.21.15.187:0/3912277401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1264066f70 0x7f12640b4220 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.042+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 >> 172.21.15.187:0/3912277401 conn(0x7f12640bab30 msgr2=0x7f1264066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:58.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.043+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 shutdown_connections 2020-11-20T12:19:58.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.043+0000 7f12694fe700 1 -- 172.21.15.187:0/3912277401 wait complete. 2020-11-20T12:19:58.070 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade' 2020-11-20T12:19:58.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.315+0000 7fdf71dfa700 1 Processor -- start 2020-11-20T12:19:58.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.315+0000 7fdf71dfa700 1 -- start start 2020-11-20T12:19:58.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.316+0000 7fdf71dfa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c0683f0 0x7fdf6c066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:58.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.316+0000 7fdf71dfa700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdf6c1487e0 0x7fdf6c148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:58.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.316+0000 7fdf71dfa700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c14c170 con 0x7fdf6c145cc0 2020-11-20T12:19:58.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.316+0000 7fdf71dfa700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c14c410 con 0x7fdf6c1487e0 2020-11-20T12:19:58.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.316+0000 7fdf71dfa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c14c6b0 con 0x7fdf6c0683f0 2020-11-20T12:19:58.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.317+0000 7fdf715f9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdf6c1487e0 0x7fdf6c148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:58.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.317+0000 7fdf715f9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdf6c1487e0 0x7fdf6c148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53158/0 (socket says 172.21.15.187:53158) 2020-11-20T12:19:58.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.317+0000 7fdf715f9700 1 -- 172.21.15.187:0/3906271736 learned_addr learned my addr 172.21.15.187:0/3906271736 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:58.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.317+0000 7fdf6bfff700 1 --2- 172.21.15.187:0/3906271736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c0683f0 0x7fdf6c066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:58.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.318+0000 7fdf715f9700 1 -- 172.21.15.187:0/3906271736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c0683f0 msgr2=0x7fdf6c066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:58.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.318+0000 7fdf715f9700 1 --2- 172.21.15.187:0/3906271736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c0683f0 0x7fdf6c066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.318+0000 7fdf715f9700 1 -- 172.21.15.187:0/3906271736 >> v1:172.21.15.5:6789/0 conn(0x7fdf6c145cc0 legacy=0x7fdf6c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:58.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.318+0000 7fdf715f9700 1 -- 172.21.15.187:0/3906271736 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c0658a0 con 0x7fdf6c1487e0 2020-11-20T12:19:58.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.318+0000 7fdf715f9700 1 --2- 172.21.15.187:0/3906271736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c148c90 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 rx=0x7fdf64009d50 tx=0x7fdf64009290).ready entity=mon.1 client_cookie=f0a6c2ec65f726d9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:58.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.318+0000 7fdf6b7fe700 1 -- 172.21.15.187:0/3906271736 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400bc40 con 0x7fdf6c1487e0 2020-11-20T12:19:58.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.318+0000 7fdf6b7fe700 1 -- 172.21.15.187:0/3906271736 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6400bda0 con 0x7fdf6c1487e0 2020-11-20T12:19:58.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.318+0000 7fdf6b7fe700 1 -- 172.21.15.187:0/3906271736 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64011610 con 0x7fdf6c1487e0 2020-11-20T12:19:58.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.319+0000 7fdf71dfa700 1 -- 172.21.15.187:0/3906271736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 msgr2=0x7fdf6c148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.319+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/3906271736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c148c90 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 rx=0x7fdf64009d50 tx=0x7fdf64009290).stop 2020-11-20T12:19:58.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.319+0000 7fdf71dfa700 1 -- 172.21.15.187:0/3906271736 shutdown_connections 2020-11-20T12:19:58.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.319+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/3906271736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c148c90 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.319+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/3906271736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c0683f0 0x7fdf6c066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.319+0000 7fdf71dfa700 1 -- 172.21.15.187:0/3906271736 >> 172.21.15.187:0/3906271736 conn(0x7fdf6c0bab30 msgr2=0x7fdf6c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:58.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.319+0000 7fdf71dfa700 1 -- 172.21.15.187:0/3906271736 shutdown_connections 2020-11-20T12:19:58.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.319+0000 7fdf71dfa700 1 -- 172.21.15.187:0/3906271736 wait complete. 2020-11-20T12:19:58.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.321+0000 7fdf71dfa700 1 Processor -- start 2020-11-20T12:19:58.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.321+0000 7fdf71dfa700 1 -- start start 2020-11-20T12:19:58.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.322+0000 7fdf71dfa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf6c0683f0 0x7fdf6c09f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:58.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.322+0000 7fdf71dfa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c145cc0 0x7fdf6c0a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:58.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.322+0000 7fdf71dfa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c09a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:58.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.322+0000 7fdf71dfa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c09dbf0 con 0x7fdf6c0683f0 2020-11-20T12:19:58.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.322+0000 7fdf71dfa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c09fdd0 con 0x7fdf6c1487e0 2020-11-20T12:19:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.322+0000 7fdf71dfa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c14ba90 con 0x7fdf6c145cc0 2020-11-20T12:19:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.322+0000 7fdf715f9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c09a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf715f9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c09a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53162/0 (socket says 172.21.15.187:53162) 2020-11-20T12:19:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf715f9700 1 -- 172.21.15.187:0/1324544726 learned_addr learned my addr 172.21.15.187:0/1324544726 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:58.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf6bfff700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c145cc0 0x7fdf6c0a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:58.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf715f9700 1 -- 172.21.15.187:0/1324544726 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c145cc0 msgr2=0x7fdf6c0a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf715f9700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c145cc0 0x7fdf6c0a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf715f9700 1 -- 172.21.15.187:0/1324544726 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf6c0683f0 msgr2=0x7fdf6c09f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf715f9700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf6c0683f0 0x7fdf6c09f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf715f9700 1 -- 172.21.15.187:0/1324544726 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c09de50 con 0x7fdf6c1487e0 2020-11-20T12:19:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.323+0000 7fdf715f9700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c09a500 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7fdf64007fd0 tx=0x7fdf64016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.324+0000 7fdf68ff9700 1 -- 172.21.15.187:0/1324544726 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400bc40 con 0x7fdf6c1487e0 2020-11-20T12:19:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.324+0000 7fdf68ff9700 1 -- 172.21.15.187:0/1324544726 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf64009e90 con 0x7fdf6c1487e0 2020-11-20T12:19:58.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.324+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c0a0900 con 0x7fdf6c1487e0 2020-11-20T12:19:58.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.324+0000 7fdf68ff9700 1 -- 172.21.15.187:0/1324544726 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64010e30 con 0x7fdf6c1487e0 2020-11-20T12:19:58.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.324+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c09ac30 con 0x7fdf6c1487e0 2020-11-20T12:19:58.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.325+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c09dfe0 con 0x7fdf6c1487e0 2020-11-20T12:19:58.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.326+0000 7fdf68ff9700 1 -- 172.21.15.187:0/1324544726 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdf64017070 con 0x7fdf6c1487e0 2020-11-20T12:19:58.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.327+0000 7fdf68ff9700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf44056d20 0x7fdf440591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:58.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.327+0000 7fdf68ff9700 1 -- 172.21.15.187:0/1324544726 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fdf64071ef0 con 0x7fdf6c1487e0 2020-11-20T12:19:58.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.327+0000 7fdf70df8700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf44056d20 0x7fdf440591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:58.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.328+0000 7fdf70df8700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf44056d20 0x7fdf440591e0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7fdf6c09d240 tx=0x7fdf600077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:58.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.330+0000 7fdf68ff9700 1 -- 172.21.15.187:0/1324544726 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdf640472f0 con 0x7fdf6c1487e0 2020-11-20T12:19:58.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.585+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade"]} v 0) v1 -- 0x7fdf6c061a20 con 0x7fdf6c1487e0 2020-11-20T12:19:58.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.869+0000 7fdf68ff9700 1 -- 172.21.15.187:0/1324544726 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade"]}]=0 v752) v1 ==== 139+0+0 (secure 0 0 0) 0x7fdf64015030 con 0x7fdf6c1487e0 2020-11-20T12:19:58.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.873+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf44056d20 msgr2=0x7fdf440591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.873+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf44056d20 0x7fdf440591e0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7fdf6c09d240 tx=0x7fdf600077f0).stop 2020-11-20T12:19:58.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.873+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 msgr2=0x7fdf6c09a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.873+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c09a500 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7fdf64007fd0 tx=0x7fdf64016000).stop 2020-11-20T12:19:58.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.874+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 shutdown_connections 2020-11-20T12:19:58.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.874+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf44056d20 0x7fdf440591e0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.874+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdf6c0683f0 0x7fdf6c09f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:58.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.874+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf6c1487e0 0x7fdf6c09a500 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.874+0000 7fdf71dfa700 1 --2- 172.21.15.187:0/1324544726 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdf6c145cc0 0x7fdf6c0a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.874+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 >> 172.21.15.187:0/1324544726 conn(0x7fdf6c0bab30 msgr2=0x7fdf6c061ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:58.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.874+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 shutdown_connections 2020-11-20T12:19:58.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:58.874+0000 7fdf71dfa700 1 -- 172.21.15.187:0/1324544726 wait complete. 2020-11-20T12:19:58.904 INFO:tasks.cephfs_test_runner:test_purge_queue_upgrade (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:19:58.905 INFO:tasks.cephfs_test_runner:Starting test: test_replicated_delete_speed (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:19:58.905 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed' 2020-11-20T12:19:59.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.172+0000 7efeb359e700 1 Processor -- start 2020-11-20T12:19:59.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.172+0000 7efeb359e700 1 -- start start 2020-11-20T12:19:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.173+0000 7efeb359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 0x7efeb4065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.173+0000 7efeb359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efeb4065680 0x7efeb414c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.173+0000 7efeb359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb4150390 con 0x7efeb40b1380 2020-11-20T12:19:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.173+0000 7efeb359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb41511e0 con 0x7efeb4065680 2020-11-20T12:19:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.173+0000 7efeb359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb414d760 con 0x7efeb40b1760 2020-11-20T12:19:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.174+0000 7efeb2d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efeb4065680 0x7efeb414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.174+0000 7efeb2d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efeb4065680 0x7efeb414c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53172/0 (socket says 172.21.15.187:53172) 2020-11-20T12:19:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.174+0000 7efeb2d9d700 1 -- 172.21.15.187:0/1956858919 learned_addr learned my addr 172.21.15.187:0/1956858919 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:59.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeabfff700 1 --2- 172.21.15.187:0/1956858919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 0x7efeb4065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:59.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeb2d9d700 1 -- 172.21.15.187:0/1956858919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 msgr2=0x7efeb4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:59.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeb2d9d700 1 --2- 172.21.15.187:0/1956858919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 0x7efeb4065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:59.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeb2d9d700 1 -- 172.21.15.187:0/1956858919 >> v1:172.21.15.5:6789/0 conn(0x7efeb40b1380 legacy=0x7efeb4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:59.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeb2d9d700 1 -- 172.21.15.187:0/1956858919 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb4150630 con 0x7efeb4065680 2020-11-20T12:19:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeb2d9d700 1 --2- 172.21.15.187:0/1956858919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb4065680 0x7efeb414c260 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 rx=0x7efeac009d50 tx=0x7efeac009290).ready entity=mon.1 client_cookie=1a9f146a196ec33f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeb1d9b700 1 -- 172.21.15.187:0/1956858919 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac00bc80 con 0x7efeb4065680 2020-11-20T12:19:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeb1d9b700 1 -- 172.21.15.187:0/1956858919 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeac00bde0 con 0x7efeb4065680 2020-11-20T12:19:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.175+0000 7efeb1d9b700 1 -- 172.21.15.187:0/1956858919 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac011610 con 0x7efeb4065680 2020-11-20T12:19:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.176+0000 7efeb359e700 1 -- 172.21.15.187:0/1956858919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb4065680 msgr2=0x7efeb414c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.176+0000 7efeb359e700 1 --2- 172.21.15.187:0/1956858919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb4065680 0x7efeb414c260 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 rx=0x7efeac009d50 tx=0x7efeac009290).stop 2020-11-20T12:19:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.176+0000 7efeb359e700 1 -- 172.21.15.187:0/1956858919 shutdown_connections 2020-11-20T12:19:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.176+0000 7efeb359e700 1 --2- 172.21.15.187:0/1956858919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb4065680 0x7efeb414c260 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.176+0000 7efeb359e700 1 --2- 172.21.15.187:0/1956858919 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 0x7efeb4065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:59.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.176+0000 7efeb359e700 1 -- 172.21.15.187:0/1956858919 >> 172.21.15.187:0/1956858919 conn(0x7efeb40bab30 msgr2=0x7efeb40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.176+0000 7efeb359e700 1 -- 172.21.15.187:0/1956858919 shutdown_connections 2020-11-20T12:19:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.176+0000 7efeb359e700 1 -- 172.21.15.187:0/1956858919 wait complete. 2020-11-20T12:19:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.178+0000 7efeb359e700 1 Processor -- start 2020-11-20T12:19:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.178+0000 7efeb359e700 1 -- start start 2020-11-20T12:19:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40b1380 0x7efeb409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 0x7efeb40b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efeb414a650 0x7efeb414aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb41420b0 con 0x7efeb414a650 2020-11-20T12:19:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb415fbe0 con 0x7efeb40b1380 2020-11-20T12:19:59.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb415fe80 con 0x7efeb40b1760 2020-11-20T12:19:59.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40b1380 0x7efeb409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:59.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40b1380 0x7efeb409cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53174/0 (socket says 172.21.15.187:53174) 2020-11-20T12:19:59.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.179+0000 7efeb259c700 1 -- 172.21.15.187:0/3537205357 learned_addr learned my addr 172.21.15.187:0/3537205357 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:19:59.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeb259c700 1 -- 172.21.15.187:0/3537205357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 msgr2=0x7efeb40b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:59.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeb259c700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 0x7efeb40b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:59.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeb259c700 1 -- 172.21.15.187:0/3537205357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efeb414a650 msgr2=0x7efeb414aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:59.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeb259c700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efeb414a650 0x7efeb414aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:59.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeb259c700 1 -- 172.21.15.187:0/3537205357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb4160120 con 0x7efeb40b1380 2020-11-20T12:19:59.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeb259c700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40b1380 0x7efeb409cf20 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7efea400c6e0 tx=0x7efea4007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:59.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeaa7fc700 1 -- 172.21.15.187:0/3537205357 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea4007020 con 0x7efeb40b1380 2020-11-20T12:19:59.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeaa7fc700 1 -- 172.21.15.187:0/3537205357 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea4007180 con 0x7efeb40b1380 2020-11-20T12:19:59.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeaa7fc700 1 -- 172.21.15.187:0/3537205357 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea400e560 con 0x7efeb40b1380 2020-11-20T12:19:59.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb409d410 con 0x7efeb40b1380 2020-11-20T12:19:59.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.180+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb409ff20 con 0x7efeb40b1380 2020-11-20T12:19:59.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.181+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb41602b0 con 0x7efeb40b1380 2020-11-20T12:19:59.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.182+0000 7efeaa7fc700 1 -- 172.21.15.187:0/3537205357 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efea4018020 con 0x7efeb40b1380 2020-11-20T12:19:59.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.182+0000 7efeaa7fc700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efe90056d20 0x7efe900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:59.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.183+0000 7efeaa7fc700 1 -- 172.21.15.187:0/3537205357 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7efea406f350 con 0x7efeb40b1380 2020-11-20T12:19:59.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.183+0000 7efeabfff700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efe90056d20 0x7efe900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:59.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.183+0000 7efeabfff700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efe90056d20 0x7efe900591e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7efeb4150310 tx=0x7efe9c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:59.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.187+0000 7efeaa7fc700 1 -- 172.21.15.187:0/3537205357 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efea4043fd0 con 0x7efeb40b1380 2020-11-20T12:19:59.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.442+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed"]} v 0) v1 -- 0x7efeb4061a20 con 0x7efeb40b1380 2020-11-20T12:19:59.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.874+0000 7efeaa7fc700 1 -- 172.21.15.187:0/3537205357 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed"]}]=0 v753) v1 ==== 146+0+0 (secure 0 0 0) 0x7efea4044750 con 0x7efeb40b1380 2020-11-20T12:19:59.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efe90056d20 msgr2=0x7efe900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:59.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efe90056d20 0x7efe900591e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7efeb4150310 tx=0x7efe9c008040).stop 2020-11-20T12:19:59.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40b1380 msgr2=0x7efeb409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:59.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40b1380 0x7efeb409cf20 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7efea400c6e0 tx=0x7efea4007c50).stop 2020-11-20T12:19:59.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 shutdown_connections 2020-11-20T12:19:59.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efe90056d20 0x7efe900591e0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:59.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efeb414a650 0x7efeb414aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:59.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40b1380 0x7efeb409cf20 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:59.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 --2- 172.21.15.187:0/3537205357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efeb40b1760 0x7efeb40b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:19:59.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 >> 172.21.15.187:0/3537205357 conn(0x7efeb40bab30 msgr2=0x7efeb414c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:59.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.878+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 shutdown_connections 2020-11-20T12:19:59.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:19:59.879+0000 7efeb359e700 1 -- 172.21.15.187:0/3537205357 wait complete. 2020-11-20T12:19:59.909 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:20:00.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.178+0000 7fdc30c9f700 1 Processor -- start 2020-11-20T12:20:00.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.178+0000 7fdc30c9f700 1 -- start start 2020-11-20T12:20:00.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.179+0000 7fdc30c9f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdc2c0b6650 0x7fdc2c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.179+0000 7fdc30c9f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c066f70 0x7fdc2c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.179+0000 7fdc30c9f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c0b1300 con 0x7fdc2c1484d0 2020-11-20T12:20:00.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.180+0000 7fdc30c9f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2c067730 con 0x7fdc2c0b6650 2020-11-20T12:20:00.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.180+0000 7fdc30c9f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0b15a0 con 0x7fdc2c066f70 2020-11-20T12:20:00.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.180+0000 7fdc2b7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fdc2c1484d0 0x7fdc2c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49788/0 (socket says 172.21.15.187:49788) 2020-11-20T12:20:00.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.180+0000 7fdc2b7fe700 1 -- 172.21.15.187:0/4283253450 learned_addr learned my addr 172.21.15.187:0/4283253450 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:00.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.180+0000 7fdc2affd700 1 --2- 172.21.15.187:0/4283253450 >> v2:172.21.15.5:3300/0 conn(0x7fdc2c0b6650 0x7fdc2c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2bfff700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c066f70 0x7fdc2c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268433329 0 0) 0x7fdc2c0b1300 con 0x7fdc2c1484d0 2020-11-20T12:20:00.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc0c003240 con 0x7fdc2c1484d0 2020-11-20T12:20:00.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1445466885 0 0) 0x7fdc0c003240 con 0x7fdc2c1484d0 2020-11-20T12:20:00.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c066f70 msgr2=0x7fdc2c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:00.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c066f70 0x7fdc2c0aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 >> v2:172.21.15.5:3300/0 conn(0x7fdc2c0b6650 msgr2=0x7fdc2c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:00.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 --2- 172.21.15.187:0/4283253450 >> v2:172.21.15.5:3300/0 conn(0x7fdc2c0b6650 0x7fdc2c0afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c0b1840 con 0x7fdc2c1484d0 2020-11-20T12:20:00.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.181+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fdc140014e0 con 0x7fdc2c1484d0 2020-11-20T12:20:00.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.182+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 >> v1:172.21.15.187:6789/0 conn(0x7fdc2c1484d0 legacy=0x7fdc2c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.182+0000 7fdc2a7fc700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc0c003730 0x7fdc0c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.182+0000 7fdc2a7fc700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc0c0044e0 0x7fdc0c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.182+0000 7fdc2a7fc700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c066f70 0x7fdc0c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.182+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c067730 con 0x7fdc0c003730 2020-11-20T12:20:00.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.182+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0b15a0 con 0x7fdc0c0044e0 2020-11-20T12:20:00.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.182+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c004270 con 0x7fdc2c066f70 2020-11-20T12:20:00.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.182+0000 7fdc2affd700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc0c0044e0 0x7fdc0c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2b7fe700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc0c003730 0x7fdc0c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2affd700 1 -- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c066f70 msgr2=0x7fdc0c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:00.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2affd700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c066f70 0x7fdc0c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:00.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2affd700 1 -- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc0c003730 msgr2=0x7fdc0c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:00.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2affd700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc0c003730 0x7fdc0c003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2affd700 1 -- 172.21.15.187:0/4283253450 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c003240 con 0x7fdc0c0044e0 2020-11-20T12:20:00.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2affd700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc0c0044e0 0x7fdc0c004960 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fdc1c009ea0 tx=0x7fdc1c0091d0).ready entity=mon.1 client_cookie=e3be0f2116b1fe48 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c0095c0 con 0x7fdc0c0044e0 2020-11-20T12:20:00.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c00eda0 con 0x7fdc0c0044e0 2020-11-20T12:20:00.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.183+0000 7fdc2a7fc700 1 -- 172.21.15.187:0/4283253450 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c012600 con 0x7fdc0c0044e0 2020-11-20T12:20:00.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.184+0000 7fdc30c9f700 1 -- 172.21.15.187:0/4283253450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc0c0044e0 msgr2=0x7fdc0c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.184+0000 7fdc30c9f700 1 --2- 172.21.15.187:0/4283253450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc0c0044e0 0x7fdc0c004960 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fdc1c009ea0 tx=0x7fdc1c0091d0).stop 2020-11-20T12:20:00.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.184+0000 7fdc2bfff700 1 -- 172.21.15.187:0/4283253450 reap_dead start 2020-11-20T12:20:00.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.184+0000 7fdc30c9f700 1 -- 172.21.15.187:0/4283253450 shutdown_connections 2020-11-20T12:20:00.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.184+0000 7fdc30c9f700 1 -- 172.21.15.187:0/4283253450 >> 172.21.15.187:0/4283253450 conn(0x7fdc2c0bab30 msgr2=0x7fdc2c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:00.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.184+0000 7fdc30c9f700 1 -- 172.21.15.187:0/4283253450 shutdown_connections 2020-11-20T12:20:00.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.184+0000 7fdc30c9f700 1 -- 172.21.15.187:0/4283253450 wait complete. 2020-11-20T12:20:00.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.186+0000 7fdc30c9f700 1 Processor -- start 2020-11-20T12:20:00.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.186+0000 7fdc30c9f700 1 -- start start 2020-11-20T12:20:00.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc30c9f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c142fc0 0x7fdc2c0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc30c9f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c09c930 0x7fdc2c1574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc30c9f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc2c157990 0x7fdc2c09e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc30c9f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0b36a0 con 0x7fdc2c157990 2020-11-20T12:20:00.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc30c9f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0b3940 con 0x7fdc2c09c930 2020-11-20T12:20:00.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc30c9f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0b33c0 con 0x7fdc2c142fc0 2020-11-20T12:20:00.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc2affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c09c930 0x7fdc2c1574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc2affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c09c930 0x7fdc2c1574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53196/0 (socket says 172.21.15.187:53196) 2020-11-20T12:20:00.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.187+0000 7fdc2affd700 1 -- 172.21.15.187:0/3905368887 learned_addr learned my addr 172.21.15.187:0/3905368887 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:00.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc2affd700 1 -- 172.21.15.187:0/3905368887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c142fc0 msgr2=0x7fdc2c0b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc2affd700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c142fc0 0x7fdc2c0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:00.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc2affd700 1 -- 172.21.15.187:0/3905368887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc2c157990 msgr2=0x7fdc2c09e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc2affd700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc2c157990 0x7fdc2c09e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:00.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc2affd700 1 -- 172.21.15.187:0/3905368887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c1433a0 con 0x7fdc2c09c930 2020-11-20T12:20:00.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc2affd700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c09c930 0x7fdc2c1574a0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7fdc1c009e70 tx=0x7fdc1c01a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc237fe700 1 -- 172.21.15.187:0/3905368887 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c000f80 con 0x7fdc2c09c930 2020-11-20T12:20:00.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc237fe700 1 -- 172.21.15.187:0/3905368887 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c01dc60 con 0x7fdc2c09c930 2020-11-20T12:20:00.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c09df90 con 0x7fdc2c09c930 2020-11-20T12:20:00.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.188+0000 7fdc237fe700 1 -- 172.21.15.187:0/3905368887 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c01b410 con 0x7fdc2c09c930 2020-11-20T12:20:00.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.189+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c158050 con 0x7fdc2c09c930 2020-11-20T12:20:00.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.190+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c143590 con 0x7fdc2c09c930 2020-11-20T12:20:00.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.190+0000 7fdc237fe700 1 -- 172.21.15.187:0/3905368887 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdc1c02b020 con 0x7fdc2c09c930 2020-11-20T12:20:00.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.191+0000 7fdc237fe700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdc08056d20 0x7fdc080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.191+0000 7fdc237fe700 1 -- 172.21.15.187:0/3905368887 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fdc1c0725e0 con 0x7fdc2c09c930 2020-11-20T12:20:00.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.191+0000 7fdc2b7fe700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdc08056d20 0x7fdc080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.192+0000 7fdc2b7fe700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdc08056d20 0x7fdc080591e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7fdc14000f20 tx=0x7fdc14009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.195+0000 7fdc237fe700 1 -- 172.21.15.187:0/3905368887 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdc1c0478d0 con 0x7fdc2c09c930 2020-11-20T12:20:00.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.384+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7fdc2c061a20 con 0x7fdc2c09c930 2020-11-20T12:20:00.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.394+0000 7fdc237fe700 1 -- 172.21.15.187:0/3905368887 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdc1c0160c0 con 0x7fdc2c09c930 2020-11-20T12:20:00.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdc08056d20 msgr2=0x7fdc080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdc08056d20 0x7fdc080591e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7fdc14000f20 tx=0x7fdc14009040).stop 2020-11-20T12:20:00.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c09c930 msgr2=0x7fdc2c1574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c09c930 0x7fdc2c1574a0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7fdc1c009e70 tx=0x7fdc1c01a000).stop 2020-11-20T12:20:00.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 shutdown_connections 2020-11-20T12:20:00.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdc08056d20 0x7fdc080591e0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc2c157990 0x7fdc2c09e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:00.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc2c142fc0 0x7fdc2c0b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:00.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 --2- 172.21.15.187:0/3905368887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c09c930 0x7fdc2c1574a0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 >> 172.21.15.187:0/3905368887 conn(0x7fdc2c0bab30 msgr2=0x7fdc2c144460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:00.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.397+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 shutdown_connections 2020-11-20T12:20:00.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.398+0000 7fdc30c9f700 1 -- 172.21.15.187:0/3905368887 wait complete. 2020-11-20T12:20:00.427 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:20:00.428 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:20:00.544 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:20:00.563 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:20:00.678 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:20:00.696 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:20:00.697 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:20:00.697 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:00.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.934+0000 7f1a0ffff700 1 Processor -- start 2020-11-20T12:20:00.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.934+0000 7f1a0ffff700 1 -- start start 2020-11-20T12:20:00.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.935+0000 7f1a0ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1a10065190 0x7f1a10065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.935+0000 7f1a0ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1a1009ca90 0x7f1a1009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.935+0000 7f1a0ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100b13e0 con 0x7f1a10065f90 2020-11-20T12:20:00.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.935+0000 7f1a0ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1014ab30 con 0x7f1a1009ca90 2020-11-20T12:20:00.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.935+0000 7f1a0ffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a10062c10 con 0x7f1a10065190 2020-11-20T12:20:00.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1a10065190 0x7f1a10065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0effd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1a10065190 0x7f1a10065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53202/0 (socket says 172.21.15.187:53202) 2020-11-20T12:20:00.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0effd700 1 -- 172.21.15.187:0/1942402140 learned_addr learned my addr 172.21.15.187:0/1942402140 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:00.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0f7fe700 1 --2- 172.21.15.187:0/1942402140 >> v2:172.21.15.5:3300/0 conn(0x7f1a1009ca90 0x7f1a1009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0effd700 1 -- 172.21.15.187:0/1942402140 >> v2:172.21.15.5:3300/0 conn(0x7f1a1009ca90 msgr2=0x7f1a1009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:00.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0effd700 1 --2- 172.21.15.187:0/1942402140 >> v2:172.21.15.5:3300/0 conn(0x7f1a1009ca90 0x7f1a1009cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0e7fc700 1 -- 172.21.15.187:0/1942402140 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473717959 0 0) 0x7f1a100b13e0 con 0x7f1a10065f90 2020-11-20T12:20:00.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0effd700 1 -- 172.21.15.187:0/1942402140 >> v1:172.21.15.187:6789/0 conn(0x7f1a10065f90 legacy=0x7f1a10143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.936+0000 7f1a0effd700 1 -- 172.21.15.187:0/1942402140 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100b1640 con 0x7f1a10065190 2020-11-20T12:20:00.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.937+0000 7f1a0effd700 1 --2- 172.21.15.187:0/1942402140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a10065190 0x7f1a10065570 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f1a1005c8b0 tx=0x7f1a00006eb0).ready entity=mon.1 client_cookie=5ef599e4a0c01333 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.937+0000 7f1a0e7fc700 1 -- 172.21.15.187:0/1942402140 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00009960 con 0x7f1a10065190 2020-11-20T12:20:00.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.937+0000 7f1a0e7fc700 1 -- 172.21.15.187:0/1942402140 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a00009ac0 con 0x7f1a10065190 2020-11-20T12:20:00.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.937+0000 7f1a0e7fc700 1 -- 172.21.15.187:0/1942402140 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00009e40 con 0x7f1a10065190 2020-11-20T12:20:00.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.938+0000 7f1a0ffff700 1 -- 172.21.15.187:0/1942402140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a10065190 msgr2=0x7f1a10065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.938+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/1942402140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a10065190 0x7f1a10065570 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f1a1005c8b0 tx=0x7f1a00006eb0).stop 2020-11-20T12:20:00.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.938+0000 7f1a0ffff700 1 -- 172.21.15.187:0/1942402140 shutdown_connections 2020-11-20T12:20:00.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.938+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/1942402140 >> v2:172.21.15.5:3300/0 conn(0x7f1a1009ca90 0x7f1a1009cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.938+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/1942402140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a10065190 0x7f1a10065570 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.938+0000 7f1a0ffff700 1 -- 172.21.15.187:0/1942402140 >> 172.21.15.187:0/1942402140 conn(0x7f1a100bab30 msgr2=0x7f1a100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:00.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.938+0000 7f1a0ffff700 1 -- 172.21.15.187:0/1942402140 shutdown_connections 2020-11-20T12:20:00.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.938+0000 7f1a0ffff700 1 -- 172.21.15.187:0/1942402140 wait complete. 2020-11-20T12:20:00.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.940+0000 7f1a0ffff700 1 Processor -- start 2020-11-20T12:20:00.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.940+0000 7f1a0ffff700 1 -- start start 2020-11-20T12:20:00.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a0ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a10065f90 0x7f1a1015c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a0ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1009ca90 0x7f1a101450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a0ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a10149c30 0x7f1a10142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a0ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1015c9d0 con 0x7f1a10149c30 2020-11-20T12:20:00.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a0ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a100b26b0 con 0x7f1a1009ca90 2020-11-20T12:20:00.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a0ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1009eff0 con 0x7f1a10065f90 2020-11-20T12:20:00.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a067fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1009ca90 0x7f1a101450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a067fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1009ca90 0x7f1a101450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53210/0 (socket says 172.21.15.187:53210) 2020-11-20T12:20:00.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a067fc700 1 -- 172.21.15.187:0/581901107 learned_addr learned my addr 172.21.15.187:0/581901107 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:00.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a0f7fe700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a10149c30 0x7f1a10142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a067fc700 1 -- 172.21.15.187:0/581901107 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a10065f90 msgr2=0x7f1a1015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a067fc700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a10065f90 0x7f1a1015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.941+0000 7f1a067fc700 1 -- 172.21.15.187:0/581901107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a10149c30 msgr2=0x7f1a10142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a067fc700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a10149c30 0x7f1a10142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a067fc700 1 -- 172.21.15.187:0/581901107 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1009f290 con 0x7f1a1009ca90 2020-11-20T12:20:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a0f7fe700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a10149c30 0x7f1a10142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a067fc700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1009ca90 0x7f1a101450b0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f19f800c710 tx=0x7f19f8007a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a077fe700 1 -- 172.21.15.187:0/581901107 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f800e560 con 0x7f1a1009ca90 2020-11-20T12:20:00.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a077fe700 1 -- 172.21.15.187:0/581901107 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f800e6c0 con 0x7f1a1009ca90 2020-11-20T12:20:00.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a101455a0 con 0x7f1a1009ca90 2020-11-20T12:20:00.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a077fe700 1 -- 172.21.15.187:0/581901107 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f800ea80 con 0x7f1a1009ca90 2020-11-20T12:20:00.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.942+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a10143460 con 0x7f1a1009ca90 2020-11-20T12:20:00.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.943+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1014a460 con 0x7f1a1009ca90 2020-11-20T12:20:00.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.944+0000 7f1a077fe700 1 -- 172.21.15.187:0/581901107 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f19f8019030 con 0x7f1a1009ca90 2020-11-20T12:20:00.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.944+0000 7f1a077fe700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f19ec056d20 0x7f19ec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.945+0000 7f1a077fe700 1 -- 172.21.15.187:0/581901107 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f19f806f7a0 con 0x7f1a1009ca90 2020-11-20T12:20:00.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.945+0000 7f1a0effd700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f19ec056d20 0x7f19ec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.945+0000 7f1a0effd700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f19ec056d20 0x7f19ec0591e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7f1a10063480 tx=0x7f1a00015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:00.948+0000 7f1a077fe700 1 -- 172.21.15.187:0/581901107 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f19f8044a90 con 0x7f1a1009ca90 2020-11-20T12:20:01.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.196+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a1009f420 con 0x7f1a1009ca90 2020-11-20T12:20:01.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.197+0000 7f1a077fe700 1 -- 172.21.15.187:0/581901107 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+2144 (secure 0 0 0) 0x7f19f806de60 con 0x7f1a1009ca90 2020-11-20T12:20:01.201 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:01.201 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":145,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7032,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3018710892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3018710892},{"type":"v1","addr":"172.21.15.5:6833","nonce":3018710892}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:55.627158+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9511":{"gid":9511,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3910175723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3910175723},{"type":"v1","addr":"172.21.15.187:6835","nonce":3910175723}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:20:01.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f19ec056d20 msgr2=0x7f19ec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f19ec056d20 0x7f19ec0591e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7f1a10063480 tx=0x7f1a00015000).stop 2020-11-20T12:20:01.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1009ca90 msgr2=0x7f1a101450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1009ca90 0x7f1a101450b0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f19f800c710 tx=0x7f19f8007a90).stop 2020-11-20T12:20:01.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 shutdown_connections 2020-11-20T12:20:01.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f19ec056d20 0x7f19ec0591e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1a10149c30 0x7f1a10142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1a10065f90 0x7f1a1015c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:01.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 --2- 172.21.15.187:0/581901107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1009ca90 0x7f1a101450b0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 >> 172.21.15.187:0/581901107 conn(0x7f1a100bab30 msgr2=0x7f1a10061d00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:01.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 shutdown_connections 2020-11-20T12:20:01.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.200+0000 7f1a0ffff700 1 -- 172.21.15.187:0/581901107 wait complete. 2020-11-20T12:20:01.207 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 145 2020-11-20T12:20:01.225 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:20:01.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.471+0000 7f42cb59e700 1 Processor -- start 2020-11-20T12:20:01.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.471+0000 7f42cb59e700 1 -- start start 2020-11-20T12:20:01.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.473+0000 7f42cb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc1484d0 0x7f42cc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.473+0000 7f42cb59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f42cc0622c0 0x7f42cc0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.473+0000 7f42cb59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc0b1300 con 0x7f42cc066f70 2020-11-20T12:20:01.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.473+0000 7f42cb59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42cc067730 con 0x7f42cc0622c0 2020-11-20T12:20:01.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.473+0000 7f42cb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc0b15a0 con 0x7f42cc1484d0 2020-11-20T12:20:01.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.474+0000 7f42c1d9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f42cc0622c0 0x7f42cc0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.474+0000 7f42c1d9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f42cc0622c0 0x7f42cc0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:45480/0 (socket says 172.21.15.187:45480) 2020-11-20T12:20:01.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.474+0000 7f42c1d9b700 1 -- 172.21.15.187:0/1442737740 learned_addr learned my addr 172.21.15.187:0/1442737740 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:01.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.474+0000 7f42ca59c700 1 --2- 172.21.15.187:0/1442737740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc1484d0 0x7f42cc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.475+0000 7f42c1d9b700 1 -- 172.21.15.187:0/1442737740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc1484d0 msgr2=0x7f42cc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:01.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.475+0000 7f42c1d9b700 1 --2- 172.21.15.187:0/1442737740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc1484d0 0x7f42cc061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.475+0000 7f42c1d9b700 1 -- 172.21.15.187:0/1442737740 >> v1:172.21.15.5:6789/0 conn(0x7f42cc066f70 legacy=0x7f42cc0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.475+0000 7f42c1d9b700 1 -- 172.21.15.187:0/1442737740 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc0b1840 con 0x7f42cc0622c0 2020-11-20T12:20:01.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.475+0000 7f42c1d9b700 1 --2- 172.21.15.187:0/1442737740 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc0622c0 0x7f42cc0b5640 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f42b4006fd0 tx=0x7f42b4006eb0).ready entity=mon.0 client_cookie=f6eaf5011b4e24b3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:01.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.476+0000 7f42c9d9b700 1 -- 172.21.15.187:0/1442737740 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4009810 con 0x7f42cc0622c0 2020-11-20T12:20:01.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.476+0000 7f42c9d9b700 1 -- 172.21.15.187:0/1442737740 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b4009970 con 0x7f42cc0622c0 2020-11-20T12:20:01.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.476+0000 7f42c9d9b700 1 -- 172.21.15.187:0/1442737740 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4009cf0 con 0x7f42cc0622c0 2020-11-20T12:20:01.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.476+0000 7f42cb59e700 1 -- 172.21.15.187:0/1442737740 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc0622c0 msgr2=0x7f42cc0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.477+0000 7f42cb59e700 1 --2- 172.21.15.187:0/1442737740 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc0622c0 0x7f42cc0b5640 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f42b4006fd0 tx=0x7f42b4006eb0).stop 2020-11-20T12:20:01.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.477+0000 7f42cb59e700 1 -- 172.21.15.187:0/1442737740 shutdown_connections 2020-11-20T12:20:01.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.477+0000 7f42cb59e700 1 --2- 172.21.15.187:0/1442737740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc1484d0 0x7f42cc061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.477+0000 7f42cb59e700 1 --2- 172.21.15.187:0/1442737740 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc0622c0 0x7f42cc0b5640 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.477+0000 7f42cb59e700 1 -- 172.21.15.187:0/1442737740 >> 172.21.15.187:0/1442737740 conn(0x7f42cc0bab30 msgr2=0x7f42cc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:01.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.477+0000 7f42cb59e700 1 -- 172.21.15.187:0/1442737740 shutdown_connections 2020-11-20T12:20:01.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.477+0000 7f42cb59e700 1 -- 172.21.15.187:0/1442737740 wait complete. 2020-11-20T12:20:01.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.479+0000 7f42cb59e700 1 Processor -- start 2020-11-20T12:20:01.484 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.479+0000 7f42cb59e700 1 -- start start 2020-11-20T12:20:01.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42cb59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42cc0622c0 0x7f42cc13dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42cb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc066f70 0x7f42cc13e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42cb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc1484d0 0x7f42cc15b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42cb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc13d280 con 0x7f42cc1484d0 2020-11-20T12:20:01.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42cb59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc13d520 con 0x7f42cc0622c0 2020-11-20T12:20:01.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42cb59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc13cfa0 con 0x7f42cc066f70 2020-11-20T12:20:01.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42ca59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42cc0622c0 0x7f42cc13dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42ca59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42cc0622c0 0x7f42cc13dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53224/0 (socket says 172.21.15.187:53224) 2020-11-20T12:20:01.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.480+0000 7f42ca59c700 1 -- 172.21.15.187:0/2833778323 learned_addr learned my addr 172.21.15.187:0/2833778323 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:01.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.481+0000 7f42cad9d700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc1484d0 0x7f42cc15b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.481+0000 7f42c1d9b700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc066f70 0x7f42cc13e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.481+0000 7f42ca59c700 1 -- 172.21.15.187:0/2833778323 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc066f70 msgr2=0x7f42cc13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.481+0000 7f42ca59c700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc066f70 0x7f42cc13e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.481+0000 7f42ca59c700 1 -- 172.21.15.187:0/2833778323 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc1484d0 msgr2=0x7f42cc15b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.481+0000 7f42ca59c700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc1484d0 0x7f42cc15b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.481+0000 7f42ca59c700 1 -- 172.21.15.187:0/2833778323 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc09b770 con 0x7f42cc0622c0 2020-11-20T12:20:01.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.481+0000 7f42ca59c700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42cc0622c0 0x7f42cc13dce0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f42bc00ecf0 tx=0x7f42bc0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:01.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.482+0000 7f42c2ffd700 1 -- 172.21.15.187:0/2833778323 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00cbb0 con 0x7f42cc0622c0 2020-11-20T12:20:01.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.482+0000 7f42c2ffd700 1 -- 172.21.15.187:0/2833778323 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc00cd10 con 0x7f42cc0622c0 2020-11-20T12:20:01.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.482+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc13b030 con 0x7f42cc0622c0 2020-11-20T12:20:01.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.482+0000 7f42c2ffd700 1 -- 172.21.15.187:0/2833778323 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc012600 con 0x7f42cc0622c0 2020-11-20T12:20:01.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.482+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42cc0a0960 con 0x7f42cc0622c0 2020-11-20T12:20:01.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.483+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42cc13d6e0 con 0x7f42cc0622c0 2020-11-20T12:20:01.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.484+0000 7f42c2ffd700 1 -- 172.21.15.187:0/2833778323 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f42bc022070 con 0x7f42cc0622c0 2020-11-20T12:20:01.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.484+0000 7f42c2ffd700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42a8056d20 0x7f42a80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.485+0000 7f42c2ffd700 1 -- 172.21.15.187:0/2833778323 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f42bc070730 con 0x7f42cc0622c0 2020-11-20T12:20:01.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.485+0000 7f42c1d9b700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42a8056d20 0x7f42a80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.485+0000 7f42c1d9b700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42a8056d20 0x7f42a80591e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f42b4006ef0 tx=0x7f42b4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:01.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.490+0000 7f42c2ffd700 1 -- 172.21.15.187:0/2833778323 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f42bc045b30 con 0x7f42cc0622c0 2020-11-20T12:20:01.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.676+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42cc061a20 con 0x7f42cc0622c0 2020-11-20T12:20:01.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.678+0000 7f42c2ffd700 1 -- 172.21.15.187:0/2833778323 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+16923 (secure 0 0 0) 0x7f42bc016090 con 0x7f42cc0622c0 2020-11-20T12:20:01.686 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:01.686 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":142,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:19:50.083281+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:19:38.454721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2020-11-20T12:19:39.038543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1878527443":"2020-11-21T12:19:50.070388+0000","172.21.15.187:6834/1878527443":"2020-11-21T12:19:50.070388+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:20:01.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42a8056d20 msgr2=0x7f42a80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42a8056d20 0x7f42a80591e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f42b4006ef0 tx=0x7f42b4015000).stop 2020-11-20T12:20:01.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42cc0622c0 msgr2=0x7f42cc13dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42cc0622c0 0x7f42cc13dce0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f42bc00ecf0 tx=0x7f42bc0091d0).stop 2020-11-20T12:20:01.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 shutdown_connections 2020-11-20T12:20:01.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f42a8056d20 0x7f42a80591e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42cc1484d0 0x7f42cc15b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42cc0622c0 0x7f42cc13dce0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 --2- 172.21.15.187:0/2833778323 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f42cc066f70 0x7f42cc13e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 >> 172.21.15.187:0/2833778323 conn(0x7f42cc0bab30 msgr2=0x7f42cc147190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:01.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 shutdown_connections 2020-11-20T12:20:01.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.681+0000 7f42cb59e700 1 -- 172.21.15.187:0/2833778323 wait complete. 2020-11-20T12:20:01.708 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:19:38.454721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:19:39.038543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:20:01.709 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:20:01.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.961+0000 7f2d2ee93700 1 Processor -- start 2020-11-20T12:20:01.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.962+0000 7f2d2ee93700 1 -- start start 2020-11-20T12:20:01.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.963+0000 7f2d2ee93700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d28145360 0x7f2d28148890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.963+0000 7f2d2ee93700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280685f0 con 0x7f2d280afca0 2020-11-20T12:20:01.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.963+0000 7f2d2ee93700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d28067e10 con 0x7f2d28063dd0 2020-11-20T12:20:01.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.963+0000 7f2d2ee93700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d280b0150 con 0x7f2d28145360 2020-11-20T12:20:01.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.963+0000 7f2d2de91700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2d28063dd0 0x7f2d28061f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49836/0 (socket says 172.21.15.187:49836) 2020-11-20T12:20:01.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.963+0000 7f2d2de91700 1 -- 172.21.15.187:0/377969055 learned_addr learned my addr 172.21.15.187:0/377969055 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:01.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.964+0000 7f2d2e692700 1 --2- 172.21.15.187:0/377969055 >> v2:172.21.15.187:3300/0 conn(0x7f2d28145360 0x7f2d28148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.964+0000 7f2d2ce8f700 1 -- 172.21.15.187:0/377969055 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599440267 0 0) 0x7f2d28067e10 con 0x7f2d28063dd0 2020-11-20T12:20:01.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.964+0000 7f2d2ce8f700 1 -- 172.21.15.187:0/377969055 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d10003240 con 0x7f2d28063dd0 2020-11-20T12:20:01.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.964+0000 7f2d2e692700 1 -- 172.21.15.187:0/377969055 >> v1:172.21.15.187:6789/0 conn(0x7f2d28063dd0 legacy=0x7f2d28061f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.964+0000 7f2d2e692700 1 -- 172.21.15.187:0/377969055 >> v1:172.21.15.5:6789/0 conn(0x7f2d280afca0 legacy=0x7f2d28148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.964+0000 7f2d2e692700 1 -- 172.21.15.187:0/377969055 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d280680b0 con 0x7f2d28145360 2020-11-20T12:20:01.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.965+0000 7f2d2e692700 1 --2- 172.21.15.187:0/377969055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28145360 0x7f2d28148890 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 rx=0x7f2d20009c90 tx=0x7f2d200091d0).ready entity=mon.1 client_cookie=fecc42122cde6858 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:01.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.965+0000 7f2d2ce8f700 1 -- 172.21.15.187:0/377969055 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2000ba70 con 0x7f2d28145360 2020-11-20T12:20:01.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.965+0000 7f2d2ce8f700 1 -- 172.21.15.187:0/377969055 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2000bbd0 con 0x7f2d28145360 2020-11-20T12:20:01.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.965+0000 7f2d2ce8f700 1 -- 172.21.15.187:0/377969055 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20011400 con 0x7f2d28145360 2020-11-20T12:20:01.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.966+0000 7f2d2ee93700 1 -- 172.21.15.187:0/377969055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28145360 msgr2=0x7f2d28148890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.966+0000 7f2d2ee93700 1 --2- 172.21.15.187:0/377969055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28145360 0x7f2d28148890 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 rx=0x7f2d20009c90 tx=0x7f2d200091d0).stop 2020-11-20T12:20:01.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.966+0000 7f2d2ee93700 1 -- 172.21.15.187:0/377969055 shutdown_connections 2020-11-20T12:20:01.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.966+0000 7f2d2ee93700 1 --2- 172.21.15.187:0/377969055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28145360 0x7f2d28148890 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.966+0000 7f2d2ee93700 1 -- 172.21.15.187:0/377969055 >> 172.21.15.187:0/377969055 conn(0x7f2d280bab30 msgr2=0x7f2d280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:01.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.966+0000 7f2d2ee93700 1 -- 172.21.15.187:0/377969055 shutdown_connections 2020-11-20T12:20:01.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.966+0000 7f2d2ee93700 1 -- 172.21.15.187:0/377969055 wait complete. 2020-11-20T12:20:01.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.968+0000 7f2d2ee93700 1 Processor -- start 2020-11-20T12:20:01.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.968+0000 7f2d2ee93700 1 -- start start 2020-11-20T12:20:01.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2ee93700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28063dd0 0x7f2d281440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2ee93700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d280afca0 0x7f2d281472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2ee93700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d28145360 0x7f2d2809c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2ee93700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28147ad0 con 0x7f2d280afca0 2020-11-20T12:20:01.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2ee93700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d280b1c80 con 0x7f2d28063dd0 2020-11-20T12:20:01.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2ee93700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d280b1f20 con 0x7f2d28145360 2020-11-20T12:20:01.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2de91700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28063dd0 0x7f2d281440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2de91700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28063dd0 0x7f2d281440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53236/0 (socket says 172.21.15.187:53236) 2020-11-20T12:20:01.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.969+0000 7f2d2de91700 1 -- 172.21.15.187:0/1632426979 learned_addr learned my addr 172.21.15.187:0/1632426979 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:01.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2d690700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d280afca0 0x7f2d281472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2e692700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d28145360 0x7f2d2809c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2de91700 1 -- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d28145360 msgr2=0x7f2d2809c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2de91700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d28145360 0x7f2d2809c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2de91700 1 -- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d280afca0 msgr2=0x7f2d281472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2de91700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d280afca0 0x7f2d281472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2de91700 1 -- 172.21.15.187:0/1632426979 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28144590 con 0x7f2d28063dd0 2020-11-20T12:20:01.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2d690700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d280afca0 0x7f2d281472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:01.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.970+0000 7f2d2de91700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28063dd0 0x7f2d281440a0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7f2d2400c6e0 tx=0x7f2d24007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:01.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.971+0000 7f2d15ffb700 1 -- 172.21.15.187:0/1632426979 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d240072e0 con 0x7f2d28063dd0 2020-11-20T12:20:01.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.971+0000 7f2d15ffb700 1 -- 172.21.15.187:0/1632426979 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24007440 con 0x7f2d28063dd0 2020-11-20T12:20:01.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.971+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d281477c0 con 0x7f2d28063dd0 2020-11-20T12:20:01.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.971+0000 7f2d15ffb700 1 -- 172.21.15.187:0/1632426979 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24005820 con 0x7f2d28063dd0 2020-11-20T12:20:01.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.971+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d280b6250 con 0x7f2d28063dd0 2020-11-20T12:20:01.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.972+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d280b8a50 con 0x7f2d28063dd0 2020-11-20T12:20:01.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.972+0000 7f2d15ffb700 1 -- 172.21.15.187:0/1632426979 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2d24010070 con 0x7f2d28063dd0 2020-11-20T12:20:01.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.973+0000 7f2d15ffb700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d00056d20 0x7f2d000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:01.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.973+0000 7f2d15ffb700 1 -- 172.21.15.187:0/1632426979 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2d2406f830 con 0x7f2d28063dd0 2020-11-20T12:20:01.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.973+0000 7f2d2d690700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d00056d20 0x7f2d000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:01.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.974+0000 7f2d2d690700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d00056d20 0x7f2d000591e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f2d280b5ff0 tx=0x7f2d18009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:01.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:01.977+0000 7f2d15ffb700 1 -- 172.21.15.187:0/1632426979 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2d24044c30 con 0x7f2d28063dd0 2020-11-20T12:20:02.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.164+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f2d280b20e0 con 0x7f2d28063dd0 2020-11-20T12:20:02.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.165+0000 7f2d15ffb700 1 -- 172.21.15.187:0/1632426979 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v142) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2d24048250 con 0x7f2d28063dd0 2020-11-20T12:20:02.170 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:20:02.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d00056d20 msgr2=0x7f2d000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d00056d20 0x7f2d000591e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f2d280b5ff0 tx=0x7f2d18009040).stop 2020-11-20T12:20:02.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28063dd0 msgr2=0x7f2d281440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28063dd0 0x7f2d281440a0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7f2d2400c6e0 tx=0x7f2d24007ac0).stop 2020-11-20T12:20:02.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 shutdown_connections 2020-11-20T12:20:02.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2d00056d20 0x7f2d000591e0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d28145360 0x7f2d2809c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d28063dd0 0x7f2d281440a0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 --2- 172.21.15.187:0/1632426979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d280afca0 0x7f2d281472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 >> 172.21.15.187:0/1632426979 conn(0x7f2d280bab30 msgr2=0x7f2d281480a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:02.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 shutdown_connections 2020-11-20T12:20:02.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.168+0000 7f2d2ee93700 1 -- 172.21.15.187:0/1632426979 wait complete. 2020-11-20T12:20:02.194 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:20:02.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.438+0000 7fb86359e700 1 Processor -- start 2020-11-20T12:20:02.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.438+0000 7fb86359e700 1 -- start start 2020-11-20T12:20:02.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.440+0000 7fb86359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb8640a4bf0 0x7fb8640a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.440+0000 7fb86359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb864149930 con 0x7fb8640bbe80 2020-11-20T12:20:02.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.440+0000 7fb86359e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8640a3560 con 0x7fb8640bbaa0 2020-11-20T12:20:02.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.440+0000 7fb86359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86414a330 con 0x7fb8640a4bf0 2020-11-20T12:20:02.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.440+0000 7fb86259c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb8640bbaa0 0x7fb864149c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49850/0 (socket says 172.21.15.187:49850) 2020-11-20T12:20:02.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.440+0000 7fb86259c700 1 -- 172.21.15.187:0/3650609754 learned_addr learned my addr 172.21.15.187:0/3650609754 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:02.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.441+0000 7fb862d9d700 1 --2- 172.21.15.187:0/3650609754 >> v2:172.21.15.187:3300/0 conn(0x7fb8640a4bf0 0x7fb8640a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.441+0000 7fb861d9b700 1 -- 172.21.15.187:0/3650609754 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554459379 0 0) 0x7fb8640a3560 con 0x7fb8640bbaa0 2020-11-20T12:20:02.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.441+0000 7fb861d9b700 1 -- 172.21.15.187:0/3650609754 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb844003240 con 0x7fb8640bbaa0 2020-11-20T12:20:02.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.441+0000 7fb862d9d700 1 -- 172.21.15.187:0/3650609754 >> v1:172.21.15.187:6789/0 conn(0x7fb8640bbaa0 legacy=0x7fb864149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.441+0000 7fb861d9b700 1 -- 172.21.15.187:0/3650609754 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1112241655 0 0) 0x7fb844003240 con 0x7fb8640bbaa0 2020-11-20T12:20:02.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.441+0000 7fb862d9d700 1 -- 172.21.15.187:0/3650609754 >> v1:172.21.15.5:6789/0 conn(0x7fb8640bbe80 legacy=0x7fb8641426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.441+0000 7fb862d9d700 1 -- 172.21.15.187:0/3650609754 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8640a3250 con 0x7fb8640a4bf0 2020-11-20T12:20:02.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.442+0000 7fb862d9d700 1 --2- 172.21.15.187:0/3650609754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640a4bf0 0x7fb8640a3a00 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 rx=0x7fb85c009c10 tx=0x7fb85c0091d0).ready entity=mon.1 client_cookie=ebf42889a555d869 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:02.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.442+0000 7fb861d9b700 1 -- 172.21.15.187:0/3650609754 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8540014e0 con 0x7fb8640a4bf0 2020-11-20T12:20:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.442+0000 7fb861d9b700 1 -- 172.21.15.187:0/3650609754 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c00b790 con 0x7fb8640a4bf0 2020-11-20T12:20:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.442+0000 7fb861d9b700 1 -- 172.21.15.187:0/3650609754 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c00bb10 con 0x7fb8640a4bf0 2020-11-20T12:20:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.443+0000 7fb86359e700 1 -- 172.21.15.187:0/3650609754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640a4bf0 msgr2=0x7fb8640a3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.443+0000 7fb86359e700 1 --2- 172.21.15.187:0/3650609754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640a4bf0 0x7fb8640a3a00 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 rx=0x7fb85c009c10 tx=0x7fb85c0091d0).stop 2020-11-20T12:20:02.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.443+0000 7fb86359e700 1 -- 172.21.15.187:0/3650609754 shutdown_connections 2020-11-20T12:20:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.443+0000 7fb86359e700 1 --2- 172.21.15.187:0/3650609754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640a4bf0 0x7fb8640a3a00 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.443+0000 7fb86359e700 1 -- 172.21.15.187:0/3650609754 >> 172.21.15.187:0/3650609754 conn(0x7fb864061bd0 msgr2=0x7fb8640678a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.443+0000 7fb86359e700 1 -- 172.21.15.187:0/3650609754 shutdown_connections 2020-11-20T12:20:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.443+0000 7fb86359e700 1 -- 172.21.15.187:0/3650609754 wait complete. 2020-11-20T12:20:02.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.444+0000 7fb86359e700 1 Processor -- start 2020-11-20T12:20:02.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.444+0000 7fb86359e700 1 -- start start 2020-11-20T12:20:02.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.445+0000 7fb86359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8640a4bf0 0x7fb8641588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.445+0000 7fb86359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8640bbaa0 0x7fb86413e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.445+0000 7fb86359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640bbe80 0x7fb864143ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.445+0000 7fb86359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864158db0 con 0x7fb8640a4bf0 2020-11-20T12:20:02.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.445+0000 7fb86359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86413e830 con 0x7fb8640bbe80 2020-11-20T12:20:02.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.445+0000 7fb86359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86409ebe0 con 0x7fb8640bbaa0 2020-11-20T12:20:02.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640bbe80 0x7fb864143ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640bbe80 0x7fb864143ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53254/0 (socket says 172.21.15.187:53254) 2020-11-20T12:20:02.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 -- 172.21.15.187:0/2818007770 learned_addr learned my addr 172.21.15.187:0/2818007770 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:02.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb86259c700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8640a4bf0 0x7fb8641588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 -- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8640bbaa0 msgr2=0x7fb86413e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb859d9b700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8640bbaa0 0x7fb86413e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8640bbaa0 0x7fb86413e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 -- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8640a4bf0 msgr2=0x7fb8641588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8640a4bf0 0x7fb8641588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 -- 172.21.15.187:0/2818007770 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86409ee80 con 0x7fb8640bbe80 2020-11-20T12:20:02.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb859d9b700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8640bbaa0 0x7fb86413e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:02.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb86259c700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8640a4bf0 0x7fb8641588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:02.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.446+0000 7fb862d9d700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640bbe80 0x7fb864143ca0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7fb85c00b6a0 tx=0x7fb85c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:02.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.447+0000 7fb85affd700 1 -- 172.21.15.187:0/2818007770 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c01d730 con 0x7fb8640bbe80 2020-11-20T12:20:02.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.447+0000 7fb85affd700 1 -- 172.21.15.187:0/2818007770 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c01d890 con 0x7fb8640bbe80 2020-11-20T12:20:02.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.447+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86409f120 con 0x7fb8640bbe80 2020-11-20T12:20:02.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.447+0000 7fb85affd700 1 -- 172.21.15.187:0/2818007770 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c01dc30 con 0x7fb8640bbe80 2020-11-20T12:20:02.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.447+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb864145b80 con 0x7fb8640bbe80 2020-11-20T12:20:02.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.448+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86409ea20 con 0x7fb8640bbe80 2020-11-20T12:20:02.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.449+0000 7fb85affd700 1 -- 172.21.15.187:0/2818007770 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb85c017040 con 0x7fb8640bbe80 2020-11-20T12:20:02.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.450+0000 7fb85affd700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb840056d20 0x7fb8400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.450+0000 7fb85affd700 1 -- 172.21.15.187:0/2818007770 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb85c0720f0 con 0x7fb8640bbe80 2020-11-20T12:20:02.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.450+0000 7fb86259c700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb840056d20 0x7fb8400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.450+0000 7fb86259c700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb840056d20 0x7fb8400591e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fb854007990 tx=0x7fb854009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:02.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.453+0000 7fb85affd700 1 -- 172.21.15.187:0/2818007770 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb85c0474f0 con 0x7fb8640bbe80 2020-11-20T12:20:02.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.640+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb8640a4790 con 0x7fb8640bbe80 2020-11-20T12:20:02.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.641+0000 7fb85affd700 1 -- 172.21.15.187:0/2818007770 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v142) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb85c0210a0 con 0x7fb8640bbe80 2020-11-20T12:20:02.645 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:20:02.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.643+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb840056d20 msgr2=0x7fb8400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.643+0000 7fb86359e700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb840056d20 0x7fb8400591e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fb854007990 tx=0x7fb854009040).stop 2020-11-20T12:20:02.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.643+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640bbe80 msgr2=0x7fb864143ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.643+0000 7fb86359e700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640bbe80 0x7fb864143ca0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7fb85c00b6a0 tx=0x7fb85c016000).stop 2020-11-20T12:20:02.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.644+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 shutdown_connections 2020-11-20T12:20:02.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.644+0000 7fb86359e700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb840056d20 0x7fb8400591e0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.644+0000 7fb86359e700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8640a4bf0 0x7fb8641588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.644+0000 7fb86359e700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8640bbe80 0x7fb864143ca0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.644+0000 7fb86359e700 1 --2- 172.21.15.187:0/2818007770 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8640bbaa0 0x7fb86413e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.644+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 >> 172.21.15.187:0/2818007770 conn(0x7fb864061bd0 msgr2=0x7fb8640a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:02.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.644+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 shutdown_connections 2020-11-20T12:20:02.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.644+0000 7fb86359e700 1 -- 172.21.15.187:0/2818007770 wait complete. 2020-11-20T12:20:02.671 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:20:02.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.907+0000 7fa1a7fff700 1 Processor -- start 2020-11-20T12:20:02.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.908+0000 7fa1a7fff700 1 -- start start 2020-11-20T12:20:02.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80bf800 0x7fa1a814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa1a80bfbe0 0x7fa1a81480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a7fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a814d110 con 0x7fa1a809f560 2020-11-20T12:20:02.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a814cbd0 con 0x7fa1a80bfbe0 2020-11-20T12:20:02.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a814ce70 con 0x7fa1a80bf800 2020-11-20T12:20:02.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a77fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa1a809f560 0x7fa1a8060c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49868/0 (socket says 172.21.15.187:49868) 2020-11-20T12:20:02.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a77fe700 1 -- 172.21.15.187:0/2412024676 learned_addr learned my addr 172.21.15.187:0/2412024676 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:02.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa19ffff700 1 --2- 172.21.15.187:0/2412024676 >> v2:172.21.15.5:3300/0 conn(0x7fa1a80bfbe0 0x7fa1a81480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a6ffd700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80bf800 0x7fa1a814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235638422 0 0) 0x7fa1a814d110 con 0x7fa1a809f560 2020-11-20T12:20:02.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.910+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa188003240 con 0x7fa1a809f560 2020-11-20T12:20:02.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 534584749 0 0) 0x7fa188003240 con 0x7fa1a809f560 2020-11-20T12:20:02.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80bf800 msgr2=0x7fa1a814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:02.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80bf800 0x7fa1a814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 >> v2:172.21.15.5:3300/0 conn(0x7fa1a80bfbe0 msgr2=0x7fa1a81480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:02.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 --2- 172.21.15.187:0/2412024676 >> v2:172.21.15.5:3300/0 conn(0x7fa1a80bfbe0 0x7fa1a81480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a80a4cc0 con 0x7fa1a809f560 2020-11-20T12:20:02.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fa1a0001860 con 0x7fa1a809f560 2020-11-20T12:20:02.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 >> v1:172.21.15.187:6789/0 conn(0x7fa1a809f560 legacy=0x7fa1a8060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa188003730 0x7fa188003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80bf800 0x7fa1880044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1880049d0 0x7fa188004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a814cbd0 con 0x7fa1880049d0 2020-11-20T12:20:02.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a814ce70 con 0x7fa188003730 2020-11-20T12:20:02.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.911+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1880041d0 con 0x7fa1a80bf800 2020-11-20T12:20:02.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa1a6ffd700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa188003730 0x7fa188003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa19ffff700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80bf800 0x7fa1880044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa1a77fe700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1880049d0 0x7fa188004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa1a6ffd700 1 -- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80bf800 msgr2=0x7fa1880044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:02.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa1a6ffd700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80bf800 0x7fa1880044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa1a6ffd700 1 -- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1880049d0 msgr2=0x7fa188004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:02.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa1a6ffd700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1880049d0 0x7fa188004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa1a6ffd700 1 -- 172.21.15.187:0/2412024676 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa188003240 con 0x7fa188003730 2020-11-20T12:20:02.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.912+0000 7fa1a77fe700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1880049d0 0x7fa188004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.913+0000 7fa1a6ffd700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa188003730 0x7fa188003b50 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7fa19800ba80 tx=0x7fa198008040).ready entity=mon.1 client_cookie=839056a4641149e6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.913+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19800d8f0 con 0x7fa188003730 2020-11-20T12:20:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.913+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19800da50 con 0x7fa188003730 2020-11-20T12:20:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.913+0000 7fa1a67fc700 1 -- 172.21.15.187:0/2412024676 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19800ddd0 con 0x7fa188003730 2020-11-20T12:20:02.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.914+0000 7fa1a7fff700 1 -- 172.21.15.187:0/2412024676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa188003730 msgr2=0x7fa188003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.914+0000 7fa1a7fff700 1 --2- 172.21.15.187:0/2412024676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa188003730 0x7fa188003b50 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7fa19800ba80 tx=0x7fa198008040).stop 2020-11-20T12:20:02.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.914+0000 7fa1a77fe700 1 -- 172.21.15.187:0/2412024676 reap_dead start 2020-11-20T12:20:02.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.914+0000 7fa1a7fff700 1 -- 172.21.15.187:0/2412024676 shutdown_connections 2020-11-20T12:20:02.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.914+0000 7fa1a7fff700 1 -- 172.21.15.187:0/2412024676 >> 172.21.15.187:0/2412024676 conn(0x7fa1a8068370 msgr2=0x7fa1a80bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:02.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.914+0000 7fa1a7fff700 1 -- 172.21.15.187:0/2412024676 shutdown_connections 2020-11-20T12:20:02.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.914+0000 7fa1a7fff700 1 -- 172.21.15.187:0/2412024676 wait complete. 2020-11-20T12:20:02.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.915+0000 7fa1a7fff700 1 Processor -- start 2020-11-20T12:20:02.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.915+0000 7fa1a7fff700 1 -- start start 2020-11-20T12:20:02.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.916+0000 7fa1a7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1a80b4820 0x7fa1a815c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.916+0000 7fa1a7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80b4c00 0x7fa1a8141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.916+0000 7fa1a7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1a80b3e90 0x7fa1a8141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.916+0000 7fa1a7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a815cd00 con 0x7fa1a80b3e90 2020-11-20T12:20:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.916+0000 7fa1a7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8145f30 con 0x7fa1a80b4820 2020-11-20T12:20:02.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.916+0000 7fa1a7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a81461d0 con 0x7fa1a80b4c00 2020-11-20T12:20:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1a80b4820 0x7fa1a815c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1a80b4820 0x7fa1a815c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53270/0 (socket says 172.21.15.187:53270) 2020-11-20T12:20:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 -- 172.21.15.187:0/3163526631 learned_addr learned my addr 172.21.15.187:0/3163526631 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa19ffff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80b4c00 0x7fa1a8141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 -- 172.21.15.187:0/3163526631 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80b4c00 msgr2=0x7fa1a8141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80b4c00 0x7fa1a8141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 -- 172.21.15.187:0/3163526631 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1a80b3e90 msgr2=0x7fa1a8141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1a80b3e90 0x7fa1a8141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 -- 172.21.15.187:0/3163526631 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a8146470 con 0x7fa1a80b4820 2020-11-20T12:20:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa1a6ffd700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1a80b4820 0x7fa1a815c810 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7fa19800ba50 tx=0x7fa198008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.917+0000 7fa19effd700 1 -- 172.21.15.187:0/3163526631 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19801d410 con 0x7fa1a80b4820 2020-11-20T12:20:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.918+0000 7fa19effd700 1 -- 172.21.15.187:0/3163526631 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19801d570 con 0x7fa1a80b4820 2020-11-20T12:20:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.918+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a8143a90 con 0x7fa1a80b4820 2020-11-20T12:20:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.918+0000 7fa19effd700 1 -- 172.21.15.187:0/3163526631 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19801d910 con 0x7fa1a80b4820 2020-11-20T12:20:02.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.918+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a8158ac0 con 0x7fa1a80b4820 2020-11-20T12:20:02.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.919+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a8143c80 con 0x7fa1a80b4820 2020-11-20T12:20:02.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.919+0000 7fa19effd700 1 -- 172.21.15.187:0/3163526631 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa19800b650 con 0x7fa1a80b4820 2020-11-20T12:20:02.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.920+0000 7fa19effd700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa184056d20 0x7fa1840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:02.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.920+0000 7fa19effd700 1 -- 172.21.15.187:0/3163526631 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa1980717f0 con 0x7fa1a80b4820 2020-11-20T12:20:02.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.920+0000 7fa19ffff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa184056d20 0x7fa1840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:02.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.921+0000 7fa19ffff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa184056d20 0x7fa1840591e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7fa190006d00 tx=0x7fa190008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:02.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:02.925+0000 7fa19effd700 1 -- 172.21.15.187:0/3163526631 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa198046bf0 con 0x7fa1a80b4820 2020-11-20T12:20:03.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.111+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1a8080050 con 0x7fa1a80b4820 2020-11-20T12:20:03.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.112+0000 7fa19effd700 1 -- 172.21.15.187:0/3163526631 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v142) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa19804a210 con 0x7fa1a80b4820 2020-11-20T12:20:03.116 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:20:03.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa184056d20 msgr2=0x7fa1840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa184056d20 0x7fa1840591e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7fa190006d00 tx=0x7fa190008040).stop 2020-11-20T12:20:03.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1a80b4820 msgr2=0x7fa1a815c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1a80b4820 0x7fa1a815c810 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7fa19800ba50 tx=0x7fa198008040).stop 2020-11-20T12:20:03.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 shutdown_connections 2020-11-20T12:20:03.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa184056d20 0x7fa1840591e0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa1a80b3e90 0x7fa1a8141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:03.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1a80b4820 0x7fa1a815c810 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 --2- 172.21.15.187:0/3163526631 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1a80b4c00 0x7fa1a8141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 >> 172.21.15.187:0/3163526631 conn(0x7fa1a8068370 msgr2=0x7fa1a80a3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:03.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 shutdown_connections 2020-11-20T12:20:03.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.115+0000 7fa1a7fff700 1 -- 172.21.15.187:0/3163526631 wait complete. 2020-11-20T12:20:03.140 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:20:03.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.382+0000 7fafbafba700 1 Processor -- start 2020-11-20T12:20:03.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.382+0000 7fafbafba700 1 -- start start 2020-11-20T12:20:03.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.384+0000 7fafbafba700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 0x7fafb4066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.384+0000 7fafbafba700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fafb41487e0 0x7fafb4148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.384+0000 7fafbafba700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb414c170 con 0x7fafb40683f0 2020-11-20T12:20:03.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.384+0000 7fafbafba700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb414c410 con 0x7fafb41487e0 2020-11-20T12:20:03.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.384+0000 7fafbafba700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb414c6b0 con 0x7fafb4145cc0 2020-11-20T12:20:03.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafb97b7700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fafb40683f0 0x7fafb4065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49888/0 (socket says 172.21.15.187:49888) 2020-11-20T12:20:03.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafb97b7700 1 -- 172.21.15.187:0/2767693989 learned_addr learned my addr 172.21.15.187:0/2767693989 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:03.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafb9fb8700 1 --2- 172.21.15.187:0/2767693989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 0x7fafb4066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafba7b9700 1 --2- 172.21.15.187:0/2767693989 >> v2:172.21.15.187:3300/0 conn(0x7fafb41487e0 0x7fafb4148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafb8fb6700 1 -- 172.21.15.187:0/2767693989 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644104618 0 0) 0x7fafb414c170 con 0x7fafb40683f0 2020-11-20T12:20:03.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafb8fb6700 1 -- 172.21.15.187:0/2767693989 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9c003240 con 0x7fafb40683f0 2020-11-20T12:20:03.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafba7b9700 1 -- 172.21.15.187:0/2767693989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 msgr2=0x7fafb4066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:03.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafba7b9700 1 --2- 172.21.15.187:0/2767693989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 0x7fafb4066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.385+0000 7fafba7b9700 1 -- 172.21.15.187:0/2767693989 >> v1:172.21.15.187:6789/0 conn(0x7fafb40683f0 legacy=0x7fafb4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.386+0000 7fafba7b9700 1 -- 172.21.15.187:0/2767693989 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb40658a0 con 0x7fafb41487e0 2020-11-20T12:20:03.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.386+0000 7fafba7b9700 1 --2- 172.21.15.187:0/2767693989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb41487e0 0x7fafb4148c90 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7fafac009c90 tx=0x7fafac0091d0).ready entity=mon.1 client_cookie=9d848f475da56e49 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.386+0000 7fafb8fb6700 1 -- 172.21.15.187:0/2767693989 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac00b9c0 con 0x7fafb41487e0 2020-11-20T12:20:03.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.386+0000 7fafb8fb6700 1 -- 172.21.15.187:0/2767693989 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac00bb20 con 0x7fafb41487e0 2020-11-20T12:20:03.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.386+0000 7fafb8fb6700 1 -- 172.21.15.187:0/2767693989 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac00bea0 con 0x7fafb41487e0 2020-11-20T12:20:03.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.387+0000 7fafbafba700 1 -- 172.21.15.187:0/2767693989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb41487e0 msgr2=0x7fafb4148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.387+0000 7fafbafba700 1 --2- 172.21.15.187:0/2767693989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb41487e0 0x7fafb4148c90 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7fafac009c90 tx=0x7fafac0091d0).stop 2020-11-20T12:20:03.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.387+0000 7fafbafba700 1 -- 172.21.15.187:0/2767693989 shutdown_connections 2020-11-20T12:20:03.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.387+0000 7fafbafba700 1 --2- 172.21.15.187:0/2767693989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb41487e0 0x7fafb4148c90 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.387+0000 7fafbafba700 1 --2- 172.21.15.187:0/2767693989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 0x7fafb4066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.387+0000 7fafbafba700 1 -- 172.21.15.187:0/2767693989 >> 172.21.15.187:0/2767693989 conn(0x7fafb40bab30 msgr2=0x7fafb40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:03.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.387+0000 7fafbafba700 1 -- 172.21.15.187:0/2767693989 shutdown_connections 2020-11-20T12:20:03.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.387+0000 7fafbafba700 1 -- 172.21.15.187:0/2767693989 wait complete. 2020-11-20T12:20:03.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.389+0000 7fafbafba700 1 Processor -- start 2020-11-20T12:20:03.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.389+0000 7fafbafba700 1 -- start start 2020-11-20T12:20:03.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.390+0000 7fafbafba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb40683f0 0x7fafb4158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.390+0000 7fafbafba700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 0x7fafb414ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.390+0000 7fafbafba700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafb41487e0 0x7fafb415b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.390+0000 7fafbafba700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb4157b00 con 0x7fafb41487e0 2020-11-20T12:20:03.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.390+0000 7fafbafba700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb41555b0 con 0x7fafb40683f0 2020-11-20T12:20:03.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.390+0000 7fafbafba700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb415bfe0 con 0x7fafb4145cc0 2020-11-20T12:20:03.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.390+0000 7fafb9fb8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb40683f0 0x7fafb4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafb9fb8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb40683f0 0x7fafb4158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53288/0 (socket says 172.21.15.187:53288) 2020-11-20T12:20:03.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafb9fb8700 1 -- 172.21.15.187:0/1636317724 learned_addr learned my addr 172.21.15.187:0/1636317724 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:03.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafb97b7700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 0x7fafb414ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafba7b9700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafb41487e0 0x7fafb415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafb9fb8700 1 -- 172.21.15.187:0/1636317724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 msgr2=0x7fafb414ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafb9fb8700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 0x7fafb414ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafb9fb8700 1 -- 172.21.15.187:0/1636317724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafb41487e0 msgr2=0x7fafb415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafb9fb8700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafb41487e0 0x7fafb415b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.391+0000 7fafb9fb8700 1 -- 172.21.15.187:0/1636317724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4157ce0 con 0x7fafb40683f0 2020-11-20T12:20:03.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.392+0000 7fafb9fb8700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb40683f0 0x7fafb4158420 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7fafb000ec00 tx=0x7fafb0006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.392+0000 7fafa1ffb700 1 -- 172.21.15.187:0/1636317724 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb000ca10 con 0x7fafb40683f0 2020-11-20T12:20:03.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.392+0000 7fafa1ffb700 1 -- 172.21.15.187:0/1636317724 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb000cb70 con 0x7fafb40683f0 2020-11-20T12:20:03.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.392+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb409d430 con 0x7fafb40683f0 2020-11-20T12:20:03.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.392+0000 7fafa1ffb700 1 -- 172.21.15.187:0/1636317724 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb00123f0 con 0x7fafb40683f0 2020-11-20T12:20:03.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.392+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb414b0f0 con 0x7fafb40683f0 2020-11-20T12:20:03.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.393+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb414b370 con 0x7fafb40683f0 2020-11-20T12:20:03.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.394+0000 7fafa1ffb700 1 -- 172.21.15.187:0/1636317724 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fafb00127a0 con 0x7fafb40683f0 2020-11-20T12:20:03.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.394+0000 7fafa1ffb700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf8c056d20 0x7faf8c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.395+0000 7fafa1ffb700 1 -- 172.21.15.187:0/1636317724 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fafb00704f0 con 0x7fafb40683f0 2020-11-20T12:20:03.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.395+0000 7fafb97b7700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf8c056d20 0x7faf8c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.395+0000 7fafb97b7700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf8c056d20 0x7faf8c0591e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fafb40b5c20 tx=0x7fafa4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.398+0000 7fafa1ffb700 1 -- 172.21.15.187:0/1636317724 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fafb0045870 con 0x7fafb40683f0 2020-11-20T12:20:03.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.590+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fafb4061a20 con 0x7fafb40683f0 2020-11-20T12:20:03.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.591+0000 7fafa1ffb700 1 -- 172.21.15.187:0/1636317724 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v142) v1 ==== 100+0+10 (secure 0 0 0) 0x7fafb0048e90 con 0x7fafb40683f0 2020-11-20T12:20:03.594 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:20:03.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf8c056d20 msgr2=0x7faf8c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf8c056d20 0x7faf8c0591e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fafb40b5c20 tx=0x7fafa4009040).stop 2020-11-20T12:20:03.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb40683f0 msgr2=0x7fafb4158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb40683f0 0x7fafb4158420 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7fafb000ec00 tx=0x7fafb0006df0).stop 2020-11-20T12:20:03.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 shutdown_connections 2020-11-20T12:20:03.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faf8c056d20 0x7faf8c0591e0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafb41487e0 0x7fafb415b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafb40683f0 0x7fafb4158420 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 --2- 172.21.15.187:0/1636317724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafb4145cc0 0x7fafb414ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 >> 172.21.15.187:0/1636317724 conn(0x7fafb40bab30 msgr2=0x7fafb4065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:03.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 shutdown_connections 2020-11-20T12:20:03.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.594+0000 7fafbafba700 1 -- 172.21.15.187:0/1636317724 wait complete. 2020-11-20T12:20:03.618 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:03.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.865+0000 7fe99dd68700 1 Processor -- start 2020-11-20T12:20:03.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.866+0000 7fe99dd68700 1 -- start start 2020-11-20T12:20:03.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.867+0000 7fe99dd68700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe99814c750 0x7fe998062580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.867+0000 7fe99dd68700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe9980666e0 0x7fe998067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.867+0000 7fe99dd68700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe998061e80 con 0x7fe998062a70 2020-11-20T12:20:03.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.867+0000 7fe99dd68700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9980620e0 con 0x7fe9980666e0 2020-11-20T12:20:03.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.867+0000 7fe99dd68700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980b1500 con 0x7fe99814c750 2020-11-20T12:20:03.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.868+0000 7fe98effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe9980666e0 0x7fe998067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.868+0000 7fe9977fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe99814c750 0x7fe998062580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.868+0000 7fe98effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe9980666e0 0x7fe998067d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:45560/0 (socket says 172.21.15.187:45560) 2020-11-20T12:20:03.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.868+0000 7fe98effd700 1 -- 172.21.15.187:0/2398510133 learned_addr learned my addr 172.21.15.187:0/2398510133 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:03.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.869+0000 7fe996ffd700 1 -- 172.21.15.187:0/2398510133 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879138099 0 0) 0x7fe998061e80 con 0x7fe998062a70 2020-11-20T12:20:03.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.869+0000 7fe9977fe700 1 -- 172.21.15.187:0/2398510133 >> v2:172.21.15.5:3300/0 conn(0x7fe9980666e0 msgr2=0x7fe998067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:03.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.869+0000 7fe9977fe700 1 --2- 172.21.15.187:0/2398510133 >> v2:172.21.15.5:3300/0 conn(0x7fe9980666e0 0x7fe998067d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.869+0000 7fe9977fe700 1 -- 172.21.15.187:0/2398510133 >> v1:172.21.15.5:6789/0 conn(0x7fe998062a70 legacy=0x7fe9981503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.869+0000 7fe9977fe700 1 -- 172.21.15.187:0/2398510133 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9980b17d0 con 0x7fe99814c750 2020-11-20T12:20:03.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.869+0000 7fe98effd700 1 --2- 172.21.15.187:0/2398510133 >> v2:172.21.15.5:3300/0 conn(0x7fe9980666e0 0x7fe998067d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:20:03.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.870+0000 7fe9977fe700 1 --2- 172.21.15.187:0/2398510133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe99814c750 0x7fe998062580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fe98800b9a0 tx=0x7fe988006df0).ready entity=mon.2 client_cookie=1d94d88237d1bd38 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.871+0000 7fe996ffd700 1 -- 172.21.15.187:0/2398510133 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9880097b0 con 0x7fe99814c750 2020-11-20T12:20:03.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.871+0000 7fe996ffd700 1 -- 172.21.15.187:0/2398510133 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe988009910 con 0x7fe99814c750 2020-11-20T12:20:03.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.871+0000 7fe996ffd700 1 -- 172.21.15.187:0/2398510133 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988009c90 con 0x7fe99814c750 2020-11-20T12:20:03.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.872+0000 7fe99dd68700 1 -- 172.21.15.187:0/2398510133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe99814c750 msgr2=0x7fe998062580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.872+0000 7fe99dd68700 1 --2- 172.21.15.187:0/2398510133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe99814c750 0x7fe998062580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fe98800b9a0 tx=0x7fe988006df0).stop 2020-11-20T12:20:03.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.872+0000 7fe99dd68700 1 -- 172.21.15.187:0/2398510133 shutdown_connections 2020-11-20T12:20:03.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.872+0000 7fe99dd68700 1 --2- 172.21.15.187:0/2398510133 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe99814c750 0x7fe998062580 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.872+0000 7fe99dd68700 1 --2- 172.21.15.187:0/2398510133 >> v2:172.21.15.5:3300/0 conn(0x7fe9980666e0 0x7fe998067d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.872+0000 7fe99dd68700 1 -- 172.21.15.187:0/2398510133 >> 172.21.15.187:0/2398510133 conn(0x7fe9980bacd0 msgr2=0x7fe9980bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:03.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.872+0000 7fe99dd68700 1 -- 172.21.15.187:0/2398510133 shutdown_connections 2020-11-20T12:20:03.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.872+0000 7fe99dd68700 1 -- 172.21.15.187:0/2398510133 wait complete. 2020-11-20T12:20:03.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.874+0000 7fe99dd68700 1 Processor -- start 2020-11-20T12:20:03.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.874+0000 7fe99dd68700 1 -- start start 2020-11-20T12:20:03.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.875+0000 7fe99dd68700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe998062a70 0x7fe99815fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.875+0000 7fe99dd68700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9980666e0 0x7fe998149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.875+0000 7fe99dd68700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe99809d1f0 0x7fe998149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.875+0000 7fe99dd68700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99809a810 con 0x7fe9980666e0 2020-11-20T12:20:03.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.875+0000 7fe99dd68700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe998160b80 con 0x7fe99809d1f0 2020-11-20T12:20:03.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.875+0000 7fe99dd68700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980b1cf0 con 0x7fe998062a70 2020-11-20T12:20:03.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.876+0000 7fe997fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe99809d1f0 0x7fe998149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.876+0000 7fe997fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe99809d1f0 0x7fe998149c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53298/0 (socket says 172.21.15.187:53298) 2020-11-20T12:20:03.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.876+0000 7fe997fff700 1 -- 172.21.15.187:0/3002117927 learned_addr learned my addr 172.21.15.187:0/3002117927 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:03.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.876+0000 7fe9977fe700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe998062a70 0x7fe99815fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.876+0000 7fe98effd700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9980666e0 0x7fe998149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe997fff700 1 -- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe998062a70 msgr2=0x7fe99815fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe997fff700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe998062a70 0x7fe99815fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe997fff700 1 -- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9980666e0 msgr2=0x7fe998149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe997fff700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9980666e0 0x7fe998149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe997fff700 1 -- 172.21.15.187:0/3002117927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99814e1e0 con 0x7fe99809d1f0 2020-11-20T12:20:03.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe98effd700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9980666e0 0x7fe998149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:03.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe997fff700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe99809d1f0 0x7fe998149c60 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 rx=0x7fe99000cab0 tx=0x7fe990007d90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe98ffff700 1 -- 172.21.15.187:0/3002117927 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99000a950 con 0x7fe99809d1f0 2020-11-20T12:20:03.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe98ffff700 1 -- 172.21.15.187:0/3002117927 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99000aab0 con 0x7fe99809d1f0 2020-11-20T12:20:03.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99809aa40 con 0x7fe99809d1f0 2020-11-20T12:20:03.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.877+0000 7fe98ffff700 1 -- 172.21.15.187:0/3002117927 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99000ae50 con 0x7fe99809d1f0 2020-11-20T12:20:03.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.878+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99814dc60 con 0x7fe99809d1f0 2020-11-20T12:20:03.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.878+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99814e370 con 0x7fe99809d1f0 2020-11-20T12:20:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.879+0000 7fe98ffff700 1 -- 172.21.15.187:0/3002117927 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe990019030 con 0x7fe99809d1f0 2020-11-20T12:20:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.880+0000 7fe98ffff700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe974056d20 0x7fe9740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.880+0000 7fe98ffff700 1 -- 172.21.15.187:0/3002117927 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe99006fde0 con 0x7fe99809d1f0 2020-11-20T12:20:03.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.880+0000 7fe9977fe700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe974056d20 0x7fe9740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.881+0000 7fe9977fe700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe974056d20 0x7fe9740591e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7fe99809d9c0 tx=0x7fe988015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:03.884+0000 7fe98ffff700 1 -- 172.21.15.187:0/3002117927 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe990045260 con 0x7fe99809d1f0 2020-11-20T12:20:04.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.131+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe998063600 con 0x7fe99809d1f0 2020-11-20T12:20:04.135 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.132+0000 7fe98ffff700 1 -- 172.21.15.187:0/3002117927 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+2144 (secure 0 0 0) 0x7fe990048880 con 0x7fe99809d1f0 2020-11-20T12:20:04.135 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:04.135 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":145,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7032,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/3018710892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3018710892},{"type":"v1","addr":"172.21.15.5:6833","nonce":3018710892}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:19:40.033030+0000","modified":"2020-11-20T12:19:55.627158+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9511":{"gid":9511,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/3910175723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3910175723},{"type":"v1","addr":"172.21.15.187:6835","nonce":3910175723}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2020-11-20T12:20:04.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe974056d20 msgr2=0x7fe9740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe974056d20 0x7fe9740591e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7fe99809d9c0 tx=0x7fe988015000).stop 2020-11-20T12:20:04.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe99809d1f0 msgr2=0x7fe998149c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe99809d1f0 0x7fe998149c60 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 rx=0x7fe99000cab0 tx=0x7fe990007d90).stop 2020-11-20T12:20:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 shutdown_connections 2020-11-20T12:20:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe974056d20 0x7fe9740591e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe99809d1f0 0x7fe998149c60 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe998062a70 0x7fe99815fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 --2- 172.21.15.187:0/3002117927 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9980666e0 0x7fe998149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.134+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 >> 172.21.15.187:0/3002117927 conn(0x7fe9980bacd0 msgr2=0x7fe9980b9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:04.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.135+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 shutdown_connections 2020-11-20T12:20:04.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.135+0000 7fe99dd68700 1 -- 172.21.15.187:0/3002117927 wait complete. 2020-11-20T12:20:04.141 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 145 2020-11-20T12:20:04.162 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:20:04.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.405+0000 7f25466f4700 1 Processor -- start 2020-11-20T12:20:04.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.405+0000 7f25466f4700 1 -- start start 2020-11-20T12:20:04.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.407+0000 7f25466f4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f254009f560 0x7f2540153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.407+0000 7f25466f4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254014d110 con 0x7f25400bf800 2020-11-20T12:20:04.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.407+0000 7f25466f4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254014cbd0 con 0x7f25400bfbe0 2020-11-20T12:20:04.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.407+0000 7f25466f4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254014c9d0 con 0x7f254009f560 2020-11-20T12:20:04.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.407+0000 7f2544ef1700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f25400bfbe0 0x7f25401480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49914/0 (socket says 172.21.15.187:49914) 2020-11-20T12:20:04.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.407+0000 7f2544ef1700 1 -- 172.21.15.187:0/4027143564 learned_addr learned my addr 172.21.15.187:0/4027143564 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:04.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f2545ef3700 1 --2- 172.21.15.187:0/4027143564 >> v2:172.21.15.5:3300/0 conn(0x7f254009f560 0x7f2540153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422302292 0 0) 0x7f254014cbd0 con 0x7f25400bfbe0 2020-11-20T12:20:04.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2528003240 con 0x7f25400bfbe0 2020-11-20T12:20:04.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 574472871 0 0) 0x7f2528003240 con 0x7f25400bfbe0 2020-11-20T12:20:04.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 >> v2:172.21.15.5:3300/0 conn(0x7f254009f560 msgr2=0x7f2540153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:04.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 --2- 172.21.15.187:0/4027143564 >> v2:172.21.15.5:3300/0 conn(0x7f254009f560 0x7f2540153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 >> v1:172.21.15.5:6789/0 conn(0x7f25400bf800 legacy=0x7f254014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:04.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254014ce70 con 0x7f25400bfbe0 2020-11-20T12:20:04.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f25300014e0 con 0x7f25400bfbe0 2020-11-20T12:20:04.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 >> v1:172.21.15.187:6789/0 conn(0x7f25400bfbe0 legacy=0x7f25401480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2528003730 0x7f2528003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25280044e0 0x7f2528004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.408+0000 7f252ffff700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f252800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254014c9d0 con 0x7f2528003730 2020-11-20T12:20:04.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2528004240 con 0x7f2528004e50 2020-11-20T12:20:04.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252800bc20 con 0x7f25280044e0 2020-11-20T12:20:04.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f2545ef3700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f252800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f2545ef3700 1 -- 172.21.15.187:0/4027143564 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25280044e0 msgr2=0x7f2528004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:04.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f2545ef3700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25280044e0 0x7f2528004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:04.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f2545ef3700 1 -- 172.21.15.187:0/4027143564 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2528003730 msgr2=0x7f2528003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:04.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f2545ef3700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2528003730 0x7f2528003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:04.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f2545ef3700 1 -- 172.21.15.187:0/4027143564 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2528003240 con 0x7f2528004e50 2020-11-20T12:20:04.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.409+0000 7f2545ef3700 1 -- 172.21.15.187:0/4027143564 reap_dead start 2020-11-20T12:20:04.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f2545ef3700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f252800b730 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7f2538009db0 tx=0x7f2538008040).ready entity=mon.1 client_cookie=e2dd2b70412c9937 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:04.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800c070 con 0x7f2528004e50 2020-11-20T12:20:04.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253800ea80 con 0x7f2528004e50 2020-11-20T12:20:04.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f252ffff700 1 -- 172.21.15.187:0/4027143564 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800ee60 con 0x7f2528004e50 2020-11-20T12:20:04.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f25466f4700 1 -- 172.21.15.187:0/4027143564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 msgr2=0x7f252800b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f25466f4700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f252800b730 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7f2538009db0 tx=0x7f2538008040).stop 2020-11-20T12:20:04.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f25466f4700 1 -- 172.21.15.187:0/4027143564 shutdown_connections 2020-11-20T12:20:04.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f25466f4700 1 --2- 172.21.15.187:0/4027143564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f252800b730 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.410+0000 7f25466f4700 1 -- 172.21.15.187:0/4027143564 >> 172.21.15.187:0/4027143564 conn(0x7f2540068370 msgr2=0x7f25400bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:04.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.411+0000 7f25466f4700 1 -- 172.21.15.187:0/4027143564 shutdown_connections 2020-11-20T12:20:04.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.411+0000 7f25466f4700 1 -- 172.21.15.187:0/4027143564 wait complete. 2020-11-20T12:20:04.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.412+0000 7f25466f4700 1 Processor -- start 2020-11-20T12:20:04.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.412+0000 7f25466f4700 1 -- start start 2020-11-20T12:20:04.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.413+0000 7f25466f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f25400b7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.413+0000 7f25466f4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25400b5960 0x7f254013dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.413+0000 7f25466f4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25401474a0 0x7f25400b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.413+0000 7f25466f4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25400b8030 con 0x7f25401474a0 2020-11-20T12:20:04.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.413+0000 7f25466f4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25400ba8b0 con 0x7f2528004e50 2020-11-20T12:20:04.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.413+0000 7f25466f4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25400bab50 con 0x7f25400b5960 2020-11-20T12:20:04.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.413+0000 7f25456f2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f25400b7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f25456f2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f25400b7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53318/0 (socket says 172.21.15.187:53318) 2020-11-20T12:20:04.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f25456f2700 1 -- 172.21.15.187:0/337223310 learned_addr learned my addr 172.21.15.187:0/337223310 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:04.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f2545ef3700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25401474a0 0x7f25400b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f25456f2700 1 -- 172.21.15.187:0/337223310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25400b5960 msgr2=0x7f254013dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f25456f2700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25400b5960 0x7f254013dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:04.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f25456f2700 1 -- 172.21.15.187:0/337223310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25401474a0 msgr2=0x7f25400b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f25456f2700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25401474a0 0x7f25400b6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f25456f2700 1 -- 172.21.15.187:0/337223310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25400badf0 con 0x7f2528004e50 2020-11-20T12:20:04.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.414+0000 7f25456f2700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f25400b7b40 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7f253c00c730 tx=0x7f253c007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:04.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.415+0000 7f252d7fa700 1 -- 172.21.15.187:0/337223310 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c00e560 con 0x7f2528004e50 2020-11-20T12:20:04.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.415+0000 7f252d7fa700 1 -- 172.21.15.187:0/337223310 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c00e700 con 0x7f2528004e50 2020-11-20T12:20:04.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.415+0000 7f252d7fa700 1 -- 172.21.15.187:0/337223310 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c00eac0 con 0x7f2528004e50 2020-11-20T12:20:04.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.415+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254013e200 con 0x7f2528004e50 2020-11-20T12:20:04.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.415+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25400b73a0 con 0x7f2528004e50 2020-11-20T12:20:04.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.416+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2540080050 con 0x7f2528004e50 2020-11-20T12:20:04.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.416+0000 7f252d7fa700 1 -- 172.21.15.187:0/337223310 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f253c010070 con 0x7f2528004e50 2020-11-20T12:20:04.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.417+0000 7f252d7fa700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2518056d20 0x7f25180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.417+0000 7f252d7fa700 1 -- 172.21.15.187:0/337223310 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f253c06f5e0 con 0x7f2528004e50 2020-11-20T12:20:04.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.417+0000 7f2544ef1700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2518056d20 0x7f25180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.418+0000 7f2544ef1700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2518056d20 0x7f25180591e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f2530000dd0 tx=0x7f254014d000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:04.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.421+0000 7f252d7fa700 1 -- 172.21.15.187:0/337223310 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f253c0449e0 con 0x7f2528004e50 2020-11-20T12:20:04.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.609+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25400a4d20 con 0x7f2528004e50 2020-11-20T12:20:04.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.611+0000 7f252d7fa700 1 -- 172.21.15.187:0/337223310 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+16923 (secure 0 0 0) 0x7f253c048000 con 0x7f2528004e50 2020-11-20T12:20:04.614 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:04.615 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":142,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:19:50.083281+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:19:38.454721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2020-11-20T12:19:39.038543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1878527443":"2020-11-21T12:19:50.070388+0000","172.21.15.187:6834/1878527443":"2020-11-21T12:19:50.070388+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:20:04.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2518056d20 msgr2=0x7f25180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2518056d20 0x7f25180591e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f2530000dd0 tx=0x7f254014d000).stop 2020-11-20T12:20:04.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 msgr2=0x7f25400b7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f25400b7b40 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7f253c00c730 tx=0x7f253c007c30).stop 2020-11-20T12:20:04.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 shutdown_connections 2020-11-20T12:20:04.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2518056d20 0x7f25180591e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25401474a0 0x7f25400b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2528004e50 0x7f25400b7b40 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 --2- 172.21.15.187:0/337223310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25400b5960 0x7f254013dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 >> 172.21.15.187:0/337223310 conn(0x7f2540068370 msgr2=0x7f25400a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.614+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 shutdown_connections 2020-11-20T12:20:04.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.615+0000 7f25466f4700 1 -- 172.21.15.187:0/337223310 wait complete. 2020-11-20T12:20:04.640 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:20:04.640 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:20:04.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.879+0000 7fced96b9700 1 Processor -- start 2020-11-20T12:20:04.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.879+0000 7fced96b9700 1 -- start start 2020-11-20T12:20:04.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.880+0000 7fced96b9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fced41484d0 0x7fced4061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.880+0000 7fced96b9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fced40622c0 0x7fced40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.881+0000 7fced96b9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced40b1300 con 0x7fced4066f70 2020-11-20T12:20:04.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.881+0000 7fced96b9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced4067730 con 0x7fced40622c0 2020-11-20T12:20:04.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.881+0000 7fced96b9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced40b15a0 con 0x7fced41484d0 2020-11-20T12:20:04.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.881+0000 7fced3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fced41484d0 0x7fced4061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.881+0000 7fced3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fced41484d0 0x7fced4061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53326/0 (socket says 172.21.15.187:53326) 2020-11-20T12:20:04.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.881+0000 7fced3fff700 1 -- 172.21.15.187:0/1703374552 learned_addr learned my addr 172.21.15.187:0/1703374552 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:04.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.881+0000 7fced37fe700 1 --2- 172.21.15.187:0/1703374552 >> v2:172.21.15.5:3300/0 conn(0x7fced40622c0 0x7fced40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.882+0000 7fced3fff700 1 -- 172.21.15.187:0/1703374552 >> v2:172.21.15.5:3300/0 conn(0x7fced40622c0 msgr2=0x7fced40b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:04.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.882+0000 7fced3fff700 1 --2- 172.21.15.187:0/1703374552 >> v2:172.21.15.5:3300/0 conn(0x7fced40622c0 0x7fced40b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.882+0000 7fced3fff700 1 -- 172.21.15.187:0/1703374552 >> v1:172.21.15.5:6789/0 conn(0x7fced4066f70 legacy=0x7fced40b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:04.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.882+0000 7fced3fff700 1 -- 172.21.15.187:0/1703374552 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced40b1840 con 0x7fced41484d0 2020-11-20T12:20:04.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.882+0000 7fced3fff700 1 --2- 172.21.15.187:0/1703374552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced41484d0 0x7fced4061dd0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7fced40b85e0 tx=0x7fcec4006eb0).ready entity=mon.1 client_cookie=dfa1f0f403203a0f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:04.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.882+0000 7fced2ffd700 1 -- 172.21.15.187:0/1703374552 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4009890 con 0x7fced41484d0 2020-11-20T12:20:04.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.882+0000 7fced2ffd700 1 -- 172.21.15.187:0/1703374552 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec40099f0 con 0x7fced41484d0 2020-11-20T12:20:04.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.882+0000 7fced2ffd700 1 -- 172.21.15.187:0/1703374552 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4009d70 con 0x7fced41484d0 2020-11-20T12:20:04.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.883+0000 7fced96b9700 1 -- 172.21.15.187:0/1703374552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced41484d0 msgr2=0x7fced4061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.883+0000 7fced96b9700 1 --2- 172.21.15.187:0/1703374552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced41484d0 0x7fced4061dd0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7fced40b85e0 tx=0x7fcec4006eb0).stop 2020-11-20T12:20:04.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.883+0000 7fced96b9700 1 -- 172.21.15.187:0/1703374552 shutdown_connections 2020-11-20T12:20:04.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.883+0000 7fced96b9700 1 --2- 172.21.15.187:0/1703374552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced41484d0 0x7fced4061dd0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.883+0000 7fced96b9700 1 --2- 172.21.15.187:0/1703374552 >> v2:172.21.15.5:3300/0 conn(0x7fced40622c0 0x7fced40b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.883+0000 7fced96b9700 1 -- 172.21.15.187:0/1703374552 >> 172.21.15.187:0/1703374552 conn(0x7fced40bab30 msgr2=0x7fced40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:04.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.883+0000 7fced96b9700 1 -- 172.21.15.187:0/1703374552 shutdown_connections 2020-11-20T12:20:04.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.883+0000 7fced96b9700 1 -- 172.21.15.187:0/1703374552 wait complete. 2020-11-20T12:20:04.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.885+0000 7fced96b9700 1 Processor -- start 2020-11-20T12:20:04.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.885+0000 7fced96b9700 1 -- start start 2020-11-20T12:20:04.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced96b9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fced40622c0 0x7fced4147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced96b9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced4066f70 0x7fced413df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced96b9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fced41484d0 0x7fced4139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced96b9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced413e7e0 con 0x7fced40622c0 2020-11-20T12:20:04.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced96b9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced409ad30 con 0x7fced4066f70 2020-11-20T12:20:04.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced96b9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced409afd0 con 0x7fced41484d0 2020-11-20T12:20:04.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced4066f70 0x7fced413df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced4066f70 0x7fced413df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53334/0 (socket says 172.21.15.187:53334) 2020-11-20T12:20:04.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced37fe700 1 -- 172.21.15.187:0/1870719911 learned_addr learned my addr 172.21.15.187:0/1870719911 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:04.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.886+0000 7fced3fff700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fced40622c0 0x7fced4147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fced37fe700 1 -- 172.21.15.187:0/1870719911 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fced41484d0 msgr2=0x7fced4139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fced37fe700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fced41484d0 0x7fced4139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:04.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fced37fe700 1 -- 172.21.15.187:0/1870719911 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fced40622c0 msgr2=0x7fced4147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:04.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fced37fe700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fced40622c0 0x7fced4147130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:04.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fced37fe700 1 -- 172.21.15.187:0/1870719911 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4147620 con 0x7fced4066f70 2020-11-20T12:20:04.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fced37fe700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced4066f70 0x7fced413df10 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7fcec800db60 tx=0x7fcec8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:04.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fceb3fff700 1 -- 172.21.15.187:0/1870719911 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8009790 con 0x7fced4066f70 2020-11-20T12:20:04.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fceb3fff700 1 -- 172.21.15.187:0/1870719911 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec80098f0 con 0x7fced4066f70 2020-11-20T12:20:04.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced409db90 con 0x7fced4066f70 2020-11-20T12:20:04.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fceb3fff700 1 -- 172.21.15.187:0/1870719911 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8009cb0 con 0x7fced4066f70 2020-11-20T12:20:04.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.887+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced40a0300 con 0x7fced4066f70 2020-11-20T12:20:04.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.888+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced413e640 con 0x7fced4066f70 2020-11-20T12:20:04.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.889+0000 7fceb3fff700 1 -- 172.21.15.187:0/1870719911 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcec8012630 con 0x7fced4066f70 2020-11-20T12:20:04.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.890+0000 7fceb3fff700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fceac056d20 0x7fceac0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:04.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.890+0000 7fceb3fff700 1 -- 172.21.15.187:0/1870719911 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcec80703d0 con 0x7fced4066f70 2020-11-20T12:20:04.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.890+0000 7fced3fff700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fceac056d20 0x7fceac0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:04.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.891+0000 7fced3fff700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fceac056d20 0x7fceac0591e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7fcec4006ef0 tx=0x7fcec4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:04.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:04.893+0000 7fceb3fff700 1 -- 172.21.15.187:0/1870719911 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fced413e640 con 0x7fced4066f70 2020-11-20T12:20:05.143 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.141+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fced413e640 con 0x7fced4066f70 2020-11-20T12:20:05.906 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:20:05.903+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:20:05.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.918+0000 7fceb3fff700 1 -- 172.21.15.187:0/1870719911 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v146) v1 ==== 161+0+0 (secure 0 0 0) 0x7fced413e640 con 0x7fced4066f70 2020-11-20T12:20:05.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.921+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fceac056d20 msgr2=0x7fceac0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:05.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.921+0000 7fced96b9700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fceac056d20 0x7fceac0591e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7fcec4006ef0 tx=0x7fcec4015000).stop 2020-11-20T12:20:05.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.921+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced4066f70 msgr2=0x7fced413df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:05.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.921+0000 7fced96b9700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced4066f70 0x7fced413df10 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7fcec800db60 tx=0x7fcec8008040).stop 2020-11-20T12:20:05.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.922+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 shutdown_connections 2020-11-20T12:20:05.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.922+0000 7fced96b9700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fceac056d20 0x7fceac0591e0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.922+0000 7fced96b9700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fced40622c0 0x7fced4147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.922+0000 7fced96b9700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fced41484d0 0x7fced4139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.922+0000 7fced96b9700 1 --2- 172.21.15.187:0/1870719911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced4066f70 0x7fced413df10 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.922+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 >> 172.21.15.187:0/1870719911 conn(0x7fced40bab30 msgr2=0x7fced40b6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:05.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.922+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 shutdown_connections 2020-11-20T12:20:05.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:05.922+0000 7fced96b9700 1 -- 172.21.15.187:0/1870719911 wait complete. 2020-11-20T12:20:05.927 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:20:05.938 INFO:tasks.ceph.mds.a.smithi187.stderr: -324> 2020-11-20T12:19:56.597+0000 7f7d6550a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:20:05.950 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:20:05.980 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:20:06.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.168+0000 7fa0cc8f4700 1 Processor -- start 2020-11-20T12:20:06.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.168+0000 7fa0cc8f4700 1 -- start start 2020-11-20T12:20:06.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.169+0000 7fa0cc8f4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa0c81484d0 0x7fa0c8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:06.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.170+0000 7fa0cc8f4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c8066f70 0x7fa0c80b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.170+0000 7fa0cc8f4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c80b1300 con 0x7fa0c80622c0 2020-11-20T12:20:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.170+0000 7fa0cc8f4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c8067730 con 0x7fa0c81484d0 2020-11-20T12:20:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.170+0000 7fa0cc8f4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80b15a0 con 0x7fa0c8066f70 2020-11-20T12:20:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.170+0000 7fa0c6d9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa0c81484d0 0x7fa0c8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:06.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.171+0000 7fa0c6d9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa0c81484d0 0x7fa0c8061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:45618/0 (socket says 172.21.15.187:45618) 2020-11-20T12:20:06.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.171+0000 7fa0c6d9d700 1 -- 172.21.15.187:0/812814913 learned_addr learned my addr 172.21.15.187:0/812814913 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:06.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.171+0000 7fa0c759e700 1 --2- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c8066f70 0x7fa0c80b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:06.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.172+0000 7fa0c6d9d700 1 -- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c8066f70 msgr2=0x7fa0c80b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:06.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.172+0000 7fa0c6d9d700 1 --2- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c8066f70 0x7fa0c80b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.172+0000 7fa0c6d9d700 1 -- 172.21.15.187:0/812814913 >> v1:172.21.15.5:6789/0 conn(0x7fa0c80622c0 legacy=0x7fa0c80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:06.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.172+0000 7fa0c6d9d700 1 -- 172.21.15.187:0/812814913 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c80b1840 con 0x7fa0c81484d0 2020-11-20T12:20:06.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.172+0000 7fa0c759e700 1 --2- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c8066f70 0x7fa0c80b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:20:06.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.172+0000 7fa0c6d9d700 1 --2- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c81484d0 0x7fa0c8061dd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fa0b8009d50 tx=0x7fa0b8009290).ready entity=mon.0 client_cookie=5ce8ccbc7c1109e3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:06.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.173+0000 7fa0c659c700 1 -- 172.21.15.187:0/812814913 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800bbc0 con 0x7fa0c81484d0 2020-11-20T12:20:06.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.173+0000 7fa0c659c700 1 -- 172.21.15.187:0/812814913 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b800bd20 con 0x7fa0c81484d0 2020-11-20T12:20:06.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.173+0000 7fa0c659c700 1 -- 172.21.15.187:0/812814913 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8011610 con 0x7fa0c81484d0 2020-11-20T12:20:06.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.173+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c81484d0 msgr2=0x7fa0c8061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:06.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.173+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c81484d0 0x7fa0c8061dd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fa0b8009d50 tx=0x7fa0b8009290).stop 2020-11-20T12:20:06.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.174+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/812814913 shutdown_connections 2020-11-20T12:20:06.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.174+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c8066f70 0x7fa0c80b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.174+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/812814913 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c81484d0 0x7fa0c8061dd0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.174+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/812814913 >> 172.21.15.187:0/812814913 conn(0x7fa0c80bab30 msgr2=0x7fa0c80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:06.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.174+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/812814913 shutdown_connections 2020-11-20T12:20:06.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.174+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/812814913 wait complete. 2020-11-20T12:20:06.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.176+0000 7fa0cc8f4700 1 Processor -- start 2020-11-20T12:20:06.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.176+0000 7fa0cc8f4700 1 -- start start 2020-11-20T12:20:06.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.177+0000 7fa0cc8f4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c80622c0 0x7fa0c8157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:06.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.177+0000 7fa0cc8f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0c8066f70 0x7fa0c809dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:06.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.177+0000 7fa0cc8f4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c81484d0 0x7fa0c80b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:06.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.177+0000 7fa0cc8f4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c81582d0 con 0x7fa0c80622c0 2020-11-20T12:20:06.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.177+0000 7fa0cc8f4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80b1a40 con 0x7fa0c8066f70 2020-11-20T12:20:06.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.177+0000 7fa0cc8f4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80b1ce0 con 0x7fa0c81484d0 2020-11-20T12:20:06.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.177+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0c8066f70 0x7fa0c809dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:06.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0c8066f70 0x7fa0c809dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53360/0 (socket says 172.21.15.187:53360) 2020-11-20T12:20:06.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0be59c700 1 -- 172.21.15.187:0/3101538251 learned_addr learned my addr 172.21.15.187:0/3101538251 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:06.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0c6d9d700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c80622c0 0x7fa0c8157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:06.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0c759e700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c81484d0 0x7fa0c80b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:06.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0be59c700 1 -- 172.21.15.187:0/3101538251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c81484d0 msgr2=0x7fa0c80b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:06.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0be59c700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c81484d0 0x7fa0c80b4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0be59c700 1 -- 172.21.15.187:0/3101538251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c80622c0 msgr2=0x7fa0c8157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:06.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0be59c700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c80622c0 0x7fa0c8157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0be59c700 1 -- 172.21.15.187:0/3101538251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c80b1f80 con 0x7fa0c8066f70 2020-11-20T12:20:06.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.178+0000 7fa0be59c700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0c8066f70 0x7fa0c809dae0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7fa0b000c710 tx=0x7fa0b0007be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:06.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.179+0000 7fa0bf7fe700 1 -- 172.21.15.187:0/3101538251 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b000e560 con 0x7fa0c8066f70 2020-11-20T12:20:06.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.179+0000 7fa0bf7fe700 1 -- 172.21.15.187:0/3101538251 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b000e6c0 con 0x7fa0c8066f70 2020-11-20T12:20:06.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.179+0000 7fa0bf7fe700 1 -- 172.21.15.187:0/3101538251 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b000ea40 con 0x7fa0c8066f70 2020-11-20T12:20:06.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.179+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c80b2220 con 0x7fa0c8066f70 2020-11-20T12:20:06.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.179+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c80b4770 con 0x7fa0c8066f70 2020-11-20T12:20:06.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.180+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c80b23b0 con 0x7fa0c8066f70 2020-11-20T12:20:06.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.180+0000 7fa0bf7fe700 1 -- 172.21.15.187:0/3101538251 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa0b0019030 con 0x7fa0c8066f70 2020-11-20T12:20:06.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.181+0000 7fa0bf7fe700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0a4056d20 0x7fa0a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:06.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.181+0000 7fa0bf7fe700 1 -- 172.21.15.187:0/3101538251 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa0b006f610 con 0x7fa0c8066f70 2020-11-20T12:20:06.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.182+0000 7fa0c6d9d700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0a4056d20 0x7fa0a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:06.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.182+0000 7fa0c6d9d700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0a4056d20 0x7fa0a40591e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fa0b8009fd0 tx=0x7fa0b8016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:06.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.185+0000 7fa0bf7fe700 1 -- 172.21.15.187:0/3101538251 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa0b00448a0 con 0x7fa0c8066f70 2020-11-20T12:20:06.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:06.432+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa0c8061a20 con 0x7fa0c8066f70 2020-11-20T12:20:07.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.159+0000 7fa0bf7fe700 1 -- 172.21.15.187:0/3101538251 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v147) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa0b00170a0 con 0x7fa0c8066f70 2020-11-20T12:20:07.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.162+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0a4056d20 msgr2=0x7fa0a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:07.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.162+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0a4056d20 0x7fa0a40591e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fa0b8009fd0 tx=0x7fa0b8016000).stop 2020-11-20T12:20:07.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.162+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0c8066f70 msgr2=0x7fa0c809dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:07.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.162+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0c8066f70 0x7fa0c809dae0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7fa0b000c710 tx=0x7fa0b0007be0).stop 2020-11-20T12:20:07.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.163+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 shutdown_connections 2020-11-20T12:20:07.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.163+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa0a4056d20 0x7fa0a40591e0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:07.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.163+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa0c80622c0 0x7fa0c8157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:07.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.163+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa0c81484d0 0x7fa0c80b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:07.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.163+0000 7fa0cc8f4700 1 --2- 172.21.15.187:0/3101538251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0c8066f70 0x7fa0c809dae0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:07.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.163+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 >> 172.21.15.187:0/3101538251 conn(0x7fa0c80bab30 msgr2=0x7fa0c8065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:07.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.163+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 shutdown_connections 2020-11-20T12:20:07.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.163+0000 7fa0cc8f4700 1 -- 172.21.15.187:0/3101538251 wait complete. 2020-11-20T12:20:07.195 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:20:07.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.491+0000 7f03e33f8700 1 Processor -- start 2020-11-20T12:20:07.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.491+0000 7f03e33f8700 1 -- start start 2020-11-20T12:20:07.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.492+0000 7f03e33f8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03dc066f70 0x7f03dc0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:07.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.492+0000 7f03e33f8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc0b1300 con 0x7f03dc0b6650 2020-11-20T12:20:07.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.492+0000 7f03e33f8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc0b15a0 con 0x7f03dc1484d0 2020-11-20T12:20:07.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.493+0000 7f03e33f8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03dc067730 con 0x7f03dc066f70 2020-11-20T12:20:07.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.493+0000 7f03e23f6700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f03dc1484d0 0x7f03dc061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:49972/0 (socket says 172.21.15.187:49972) 2020-11-20T12:20:07.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.493+0000 7f03e23f6700 1 -- 172.21.15.187:0/3627220380 learned_addr learned my addr 172.21.15.187:0/3627220380 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:07.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.493+0000 7f03e2bf7700 1 --2- 172.21.15.187:0/3627220380 >> v2:172.21.15.5:3300/0 conn(0x7f03dc066f70 0x7f03dc0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178014722 0 0) 0x7f03dc0b15a0 con 0x7f03dc1484d0 2020-11-20T12:20:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c4003240 con 0x7f03dc1484d0 2020-11-20T12:20:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 175405440 0 0) 0x7f03c4003240 con 0x7f03dc1484d0 2020-11-20T12:20:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 >> v2:172.21.15.5:3300/0 conn(0x7f03dc066f70 msgr2=0x7f03dc0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:07.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 --2- 172.21.15.187:0/3627220380 >> v2:172.21.15.5:3300/0 conn(0x7f03dc066f70 0x7f03dc0b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:07.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 >> v1:172.21.15.5:6789/0 conn(0x7f03dc0b6650 legacy=0x7f03dc0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:07.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0b1840 con 0x7f03dc1484d0 2020-11-20T12:20:07.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f03d80014e0 con 0x7f03dc1484d0 2020-11-20T12:20:07.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 >> v1:172.21.15.187:6789/0 conn(0x7f03dc1484d0 legacy=0x7f03dc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:07.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.494+0000 7f03e13f4700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03c4003730 0x7f03c4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:07.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.495+0000 7f03e13f4700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03c40044e0 0x7f03c4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:07.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.495+0000 7f03e13f4700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03c400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:07.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.495+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc067730 con 0x7f03c40044e0 2020-11-20T12:20:07.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.495+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c4004260 con 0x7f03c4004e50 2020-11-20T12:20:07.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.495+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c400bc20 con 0x7f03c4003730 2020-11-20T12:20:07.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.495+0000 7f03e2bf7700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03c400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:07.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.495+0000 7f03e1bf5700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03c40044e0 0x7f03c4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:07.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.495+0000 7f03e23f6700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03c4003730 0x7f03c4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:07.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e2bf7700 1 -- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03c4003730 msgr2=0x7f03c4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:07.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e2bf7700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03c4003730 0x7f03c4003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:07.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e2bf7700 1 -- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03c40044e0 msgr2=0x7f03c4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:07.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e2bf7700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03c40044e0 0x7f03c4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:07.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e2bf7700 1 -- 172.21.15.187:0/3627220380 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c4003240 con 0x7f03c4004e50 2020-11-20T12:20:07.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e1bf5700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03c40044e0 0x7f03c4004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:07.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e2bf7700 1 -- 172.21.15.187:0/3627220380 reap_dead start 2020-11-20T12:20:07.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e2bf7700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03c400b730 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7f03d4009e70 tx=0x7f03d40091d0).ready entity=mon.1 client_cookie=dd5a4fae32e4c77c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:07.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d40095c0 con 0x7f03c4004e50 2020-11-20T12:20:07.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d400ed70 con 0x7f03c4004e50 2020-11-20T12:20:07.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.496+0000 7f03e13f4700 1 -- 172.21.15.187:0/3627220380 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d400eee0 con 0x7f03c4004e50 2020-11-20T12:20:07.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.497+0000 7f03e33f8700 1 -- 172.21.15.187:0/3627220380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 msgr2=0x7f03c400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:07.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.497+0000 7f03e33f8700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03c400b730 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7f03d4009e70 tx=0x7f03d40091d0).stop 2020-11-20T12:20:07.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.497+0000 7f03e33f8700 1 -- 172.21.15.187:0/3627220380 shutdown_connections 2020-11-20T12:20:07.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.497+0000 7f03e33f8700 1 --2- 172.21.15.187:0/3627220380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03c400b730 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:07.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.497+0000 7f03e33f8700 1 -- 172.21.15.187:0/3627220380 >> 172.21.15.187:0/3627220380 conn(0x7f03dc0bab30 msgr2=0x7f03dc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:07.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.497+0000 7f03e33f8700 1 -- 172.21.15.187:0/3627220380 shutdown_connections 2020-11-20T12:20:07.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.497+0000 7f03e33f8700 1 -- 172.21.15.187:0/3627220380 wait complete. 2020-11-20T12:20:07.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.499+0000 7f03e33f8700 1 Processor -- start 2020-11-20T12:20:07.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.499+0000 7f03e33f8700 1 -- start start 2020-11-20T12:20:07.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e33f8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03dc142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:07.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e33f8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03dc08ea20 0x7f03dc09e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:07.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e33f8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc09ead0 0x7f03dc1444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:07.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e33f8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc142960 con 0x7f03dc08ea20 2020-11-20T12:20:07.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e33f8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc142c50 con 0x7f03c4004e50 2020-11-20T12:20:07.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e33f8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc142ef0 con 0x7f03dc09ead0 2020-11-20T12:20:07.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e23f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03dc142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e23f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03dc142470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53378/0 (socket says 172.21.15.187:53378) 2020-11-20T12:20:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.500+0000 7f03e23f6700 1 -- 172.21.15.187:0/4036989158 learned_addr learned my addr 172.21.15.187:0/4036989158 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03e23f6700 1 -- 172.21.15.187:0/4036989158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc09ead0 msgr2=0x7f03dc1444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03e23f6700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc09ead0 0x7f03dc1444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03e23f6700 1 -- 172.21.15.187:0/4036989158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03dc08ea20 msgr2=0x7f03dc09e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:07.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03e23f6700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03dc08ea20 0x7f03dc09e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:07.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03e23f6700 1 -- 172.21.15.187:0/4036989158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc09e340 con 0x7f03c4004e50 2020-11-20T12:20:07.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03e23f6700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03dc142470 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f03d800eab0 tx=0x7f03d8001420).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:07.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03ca7fc700 1 -- 172.21.15.187:0/4036989158 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d8005240 con 0x7f03c4004e50 2020-11-20T12:20:07.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03ca7fc700 1 -- 172.21.15.187:0/4036989158 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d8010040 con 0x7f03c4004e50 2020-11-20T12:20:07.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03dc1449e0 con 0x7f03c4004e50 2020-11-20T12:20:07.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03ca7fc700 1 -- 172.21.15.187:0/4036989158 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d800a8e0 con 0x7f03c4004e50 2020-11-20T12:20:07.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.501+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03dc0b36f0 con 0x7f03c4004e50 2020-11-20T12:20:07.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.502+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03dc144cd0 con 0x7f03c4004e50 2020-11-20T12:20:07.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.503+0000 7f03ca7fc700 1 -- 172.21.15.187:0/4036989158 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f03d800ac90 con 0x7f03c4004e50 2020-11-20T12:20:07.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.504+0000 7f03ca7fc700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d50 0x7f03b4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:07.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.504+0000 7f03ca7fc700 1 -- 172.21.15.187:0/4036989158 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f03d806fc00 con 0x7f03c4004e50 2020-11-20T12:20:07.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.504+0000 7f03e1bf5700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d50 0x7f03b4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:07.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.505+0000 7f03e1bf5700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d50 0x7f03b4059210 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f03cc007fd0 tx=0x7f03cc009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:07.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.508+0000 7f03ca7fc700 1 -- 172.21.15.187:0/4036989158 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f03dc144cd0 con 0x7f03c4004e50 2020-11-20T12:20:07.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:07.694+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f03dc142ab0 con 0x7f03c4004e50 2020-11-20T12:20:08.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.170+0000 7f03ca7fc700 1 -- 172.21.15.187:0/4036989158 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v144) v1 ==== 181+0+0 (secure 0 0 0) 0x7f03dc142ab0 con 0x7f03c4004e50 2020-11-20T12:20:08.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d50 msgr2=0x7f03b4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:08.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d50 0x7f03b4059210 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f03cc007fd0 tx=0x7f03cc009040).stop 2020-11-20T12:20:08.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 msgr2=0x7f03dc142470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:08.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03dc142470 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f03d800eab0 tx=0x7f03d8001420).stop 2020-11-20T12:20:08.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 shutdown_connections 2020-11-20T12:20:08.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f03b4056d50 0x7f03b4059210 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:08.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03dc09ead0 0x7f03dc1444f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:08.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c4004e50 0x7f03dc142470 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:08.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 --2- 172.21.15.187:0/4036989158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03dc08ea20 0x7f03dc09e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:08.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.173+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 >> 172.21.15.187:0/4036989158 conn(0x7f03dc0bab30 msgr2=0x7f03dc143a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:08.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.174+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 shutdown_connections 2020-11-20T12:20:08.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.174+0000 7f03e33f8700 1 -- 172.21.15.187:0/4036989158 wait complete. 2020-11-20T12:20:08.180 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:20:08.202 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:20:08.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.443+0000 7f0c108d5700 1 Processor -- start 2020-11-20T12:20:08.446 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.443+0000 7f0c108d5700 1 -- start start 2020-11-20T12:20:08.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.444+0000 7f0c108d5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0c0c065f90 0x7f0c0c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:08.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.444+0000 7f0c108d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c09ca90 0x7f0c0c09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:08.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.444+0000 7f0c108d5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c0b13e0 con 0x7f0c0c065190 2020-11-20T12:20:08.448 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.444+0000 7f0c108d5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c14ab30 con 0x7f0c0c065f90 2020-11-20T12:20:08.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.445+0000 7f0c108d5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c062c10 con 0x7f0c0c09ca90 2020-11-20T12:20:08.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.445+0000 7f0c02ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0c0c065f90 0x7f0c0c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:08.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.445+0000 7f0c02ffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0c0c065f90 0x7f0c0c065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53388/0 (socket says 172.21.15.187:53388) 2020-11-20T12:20:08.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.445+0000 7f0c02ffd700 1 -- 172.21.15.187:0/3998758962 learned_addr learned my addr 172.21.15.187:0/3998758962 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:08.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c0bfff700 1 --2- 172.21.15.187:0/3998758962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c09ca90 0x7f0c0c09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:08.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c02ffd700 1 -- 172.21.15.187:0/3998758962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c09ca90 msgr2=0x7f0c0c09cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:08.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c02ffd700 1 --2- 172.21.15.187:0/3998758962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c09ca90 0x7f0c0c09cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:08.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c02ffd700 1 -- 172.21.15.187:0/3998758962 >> v1:172.21.15.5:6789/0 conn(0x7f0c0c065190 legacy=0x7f0c0c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:08.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c02ffd700 1 -- 172.21.15.187:0/3998758962 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c0b1640 con 0x7f0c0c065f90 2020-11-20T12:20:08.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c02ffd700 1 --2- 172.21.15.187:0/3998758962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065f90 0x7f0c0c065570 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7f0bf4006fd0 tx=0x7f0bf4006eb0).ready entity=mon.1 client_cookie=74a611dd3327426d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:08.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c0affd700 1 -- 172.21.15.187:0/3998758962 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4009920 con 0x7f0c0c065f90 2020-11-20T12:20:08.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c0affd700 1 -- 172.21.15.187:0/3998758962 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf4009a80 con 0x7f0c0c065f90 2020-11-20T12:20:08.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.446+0000 7f0c0affd700 1 -- 172.21.15.187:0/3998758962 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4009e00 con 0x7f0c0c065f90 2020-11-20T12:20:08.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.447+0000 7f0c108d5700 1 -- 172.21.15.187:0/3998758962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065f90 msgr2=0x7f0c0c065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:08.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.447+0000 7f0c108d5700 1 --2- 172.21.15.187:0/3998758962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065f90 0x7f0c0c065570 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7f0bf4006fd0 tx=0x7f0bf4006eb0).stop 2020-11-20T12:20:08.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.447+0000 7f0c108d5700 1 -- 172.21.15.187:0/3998758962 shutdown_connections 2020-11-20T12:20:08.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.447+0000 7f0c108d5700 1 --2- 172.21.15.187:0/3998758962 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c09ca90 0x7f0c0c09cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:08.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.447+0000 7f0c108d5700 1 --2- 172.21.15.187:0/3998758962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065f90 0x7f0c0c065570 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:08.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.447+0000 7f0c108d5700 1 -- 172.21.15.187:0/3998758962 >> 172.21.15.187:0/3998758962 conn(0x7f0c0c0bab30 msgr2=0x7f0c0c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:08.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.447+0000 7f0c108d5700 1 -- 172.21.15.187:0/3998758962 shutdown_connections 2020-11-20T12:20:08.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.447+0000 7f0c108d5700 1 -- 172.21.15.187:0/3998758962 wait complete. 2020-11-20T12:20:08.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.449+0000 7f0c108d5700 1 Processor -- start 2020-11-20T12:20:08.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.449+0000 7f0c108d5700 1 -- start start 2020-11-20T12:20:08.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c108d5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065190 0x7f0c0c0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:08.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c108d5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c0c09ca90 0x7f0c0c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:08.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c108d5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c149c40 0x7f0c0c145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:08.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c108d5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0a0060 con 0x7f0c0c09ca90 2020-11-20T12:20:08.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c108d5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0a0300 con 0x7f0c0c065190 2020-11-20T12:20:08.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c108d5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c146010 con 0x7f0c0c149c40 2020-11-20T12:20:08.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c0b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065190 0x7f0c0c0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:08.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c0b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065190 0x7f0c0c0b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53394/0 (socket says 172.21.15.187:53394) 2020-11-20T12:20:08.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.450+0000 7f0c0b7fe700 1 -- 172.21.15.187:0/2037560521 learned_addr learned my addr 172.21.15.187:0/2037560521 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:08.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c02ffd700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c0c09ca90 0x7f0c0c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:08.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c0b7fe700 1 -- 172.21.15.187:0/2037560521 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c149c40 msgr2=0x7f0c0c145b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:08.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c0b7fe700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c149c40 0x7f0c0c145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:08.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c0b7fe700 1 -- 172.21.15.187:0/2037560521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c0c09ca90 msgr2=0x7f0c0c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:08.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c0b7fe700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c0c09ca90 0x7f0c0c0b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:08.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c0b7fe700 1 -- 172.21.15.187:0/2037560521 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c142d90 con 0x7f0c0c065190 2020-11-20T12:20:08.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c0b7fe700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065190 0x7f0c0c0b3020 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 rx=0x7f0bfc00c730 tx=0x7f0bfc007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:08.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c03fff700 1 -- 172.21.15.187:0/2037560521 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00e560 con 0x7f0c0c065190 2020-11-20T12:20:08.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c03fff700 1 -- 172.21.15.187:0/2037560521 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc00e6e0 con 0x7f0c0c065190 2020-11-20T12:20:08.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c143040 con 0x7f0c0c065190 2020-11-20T12:20:08.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c03fff700 1 -- 172.21.15.187:0/2037560521 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00eaa0 con 0x7f0c0c065190 2020-11-20T12:20:08.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.451+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c143500 con 0x7f0c0c065190 2020-11-20T12:20:08.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.452+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c09fea0 con 0x7f0c0c065190 2020-11-20T12:20:08.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.453+0000 7f0c03fff700 1 -- 172.21.15.187:0/2037560521 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0bfc019030 con 0x7f0c0c065190 2020-11-20T12:20:08.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.454+0000 7f0c03fff700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0be8056d20 0x7f0be80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:08.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.454+0000 7f0c03fff700 1 -- 172.21.15.187:0/2037560521 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5807+0+0 (secure 0 0 0) 0x7f0bfc06f3e0 con 0x7f0c0c065190 2020-11-20T12:20:08.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.454+0000 7f0c02ffd700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0be8056d20 0x7f0be80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:08.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.454+0000 7f0c02ffd700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0be8056d20 0x7f0be80591e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f0c0c066370 tx=0x7f0bf4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:08.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.457+0000 7f0c03fff700 1 -- 172.21.15.187:0/2037560521 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0bfc043fd0 con 0x7f0c0c065190 2020-11-20T12:20:08.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:08.649+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0c0c061a20 con 0x7f0c0c065190 2020-11-20T12:20:09.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.175+0000 7f0c03fff700 1 -- 172.21.15.187:0/2037560521 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v145) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0bfc044970 con 0x7f0c0c065190 2020-11-20T12:20:09.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0be8056d20 msgr2=0x7f0be80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:09.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0be8056d20 0x7f0be80591e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f0c0c066370 tx=0x7f0bf4015000).stop 2020-11-20T12:20:09.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065190 msgr2=0x7f0c0c0b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:09.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065190 0x7f0c0c0b3020 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 rx=0x7f0bfc00c730 tx=0x7f0bfc007c30).stop 2020-11-20T12:20:09.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 shutdown_connections 2020-11-20T12:20:09.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0be8056d20 0x7f0be80591e0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:09.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c0c149c40 0x7f0c0c145b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:09.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c065190 0x7f0c0c0b3020 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:09.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 --2- 172.21.15.187:0/2037560521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c0c09ca90 0x7f0c0c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:09.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 >> 172.21.15.187:0/2037560521 conn(0x7f0c0c0bab30 msgr2=0x7f0c0c0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:09.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 shutdown_connections 2020-11-20T12:20:09.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:09.179+0000 7f0c108d5700 1 -- 172.21.15.187:0/2037560521 wait complete. 2020-11-20T12:20:09.185 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:20:09.211 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:20:09.211 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:20:09.211 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:20:09.212 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:20:09.212 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:20:09.212 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:20:09.213 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:20:09.213 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:20:09.231 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:20:09.231 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:20:09.238 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:20:09.238 DEBUG:teuthology.parallel:result is None 2020-11-20T12:20:09.423 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:20:11.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2020-11-20T12:20:15.237 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:20:15.237 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:20:15.246 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:20:15.246 DEBUG:teuthology.parallel:result is None 2020-11-20T12:20:15.247 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:20:15.446 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:20:15.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.495+0000 7f1b7da60700 1 Processor -- start 2020-11-20T12:20:15.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.496+0000 7f1b7da60700 1 -- start start 2020-11-20T12:20:15.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.497+0000 7f1b7da60700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1b780b57b0 0x7f1b78068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.497+0000 7f1b7da60700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.497+0000 7f1b7da60700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b78144b80 con 0x7f1b78062b50 2020-11-20T12:20:15.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.497+0000 7f1b7da60700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b781489e0 con 0x7f1b780b57b0 2020-11-20T12:20:15.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.497+0000 7f1b7da60700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b78148c80 con 0x7f1b781482d0 2020-11-20T12:20:15.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b7d25f700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1b78062b50 0x7f1b78065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50024/0 (socket says 172.21.15.187:50024) 2020-11-20T12:20:15.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b7d25f700 1 -- 172.21.15.187:0/1914707978 learned_addr learned my addr 172.21.15.187:0/1914707978 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:15.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b77fff700 1 --2- 172.21.15.187:0/1914707978 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b7ca5e700 1 --2- 172.21.15.187:0/1914707978 >> v2:172.21.15.5:3300/0 conn(0x7f1b780b57b0 0x7f1b78068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b777fe700 1 -- 172.21.15.187:0/1914707978 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114736862 0 0) 0x7f1b78144b80 con 0x7f1b78062b50 2020-11-20T12:20:15.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b777fe700 1 -- 172.21.15.187:0/1914707978 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b5c003240 con 0x7f1b78062b50 2020-11-20T12:20:15.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b77fff700 1 -- 172.21.15.187:0/1914707978 >> v2:172.21.15.5:3300/0 conn(0x7f1b780b57b0 msgr2=0x7f1b78068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:15.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b77fff700 1 --2- 172.21.15.187:0/1914707978 >> v2:172.21.15.5:3300/0 conn(0x7f1b780b57b0 0x7f1b78068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b77fff700 1 -- 172.21.15.187:0/1914707978 >> v1:172.21.15.187:6789/0 conn(0x7f1b78062b50 legacy=0x7f1b78065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.498+0000 7f1b77fff700 1 -- 172.21.15.187:0/1914707978 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b78148f20 con 0x7f1b781482d0 2020-11-20T12:20:15.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.499+0000 7f1b77fff700 1 --2- 172.21.15.187:0/1914707978 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78062660 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1b6400b9a0 tx=0x7f1b64006df0).ready entity=mon.2 client_cookie=4923ea670555c865 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:15.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.499+0000 7f1b777fe700 1 -- 172.21.15.187:0/1914707978 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64009680 con 0x7f1b781482d0 2020-11-20T12:20:15.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.499+0000 7f1b777fe700 1 -- 172.21.15.187:0/1914707978 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b640097e0 con 0x7f1b781482d0 2020-11-20T12:20:15.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.499+0000 7f1b777fe700 1 -- 172.21.15.187:0/1914707978 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64009b60 con 0x7f1b781482d0 2020-11-20T12:20:15.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.500+0000 7f1b7da60700 1 -- 172.21.15.187:0/1914707978 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 msgr2=0x7f1b78062660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.500+0000 7f1b7da60700 1 --2- 172.21.15.187:0/1914707978 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78062660 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1b6400b9a0 tx=0x7f1b64006df0).stop 2020-11-20T12:20:15.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.500+0000 7f1b7da60700 1 -- 172.21.15.187:0/1914707978 shutdown_connections 2020-11-20T12:20:15.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.500+0000 7f1b7da60700 1 --2- 172.21.15.187:0/1914707978 >> v2:172.21.15.5:3300/0 conn(0x7f1b780b57b0 0x7f1b78068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.500+0000 7f1b7da60700 1 --2- 172.21.15.187:0/1914707978 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78062660 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.500+0000 7f1b7da60700 1 -- 172.21.15.187:0/1914707978 >> 172.21.15.187:0/1914707978 conn(0x7f1b780bab30 msgr2=0x7f1b780baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:15.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.500+0000 7f1b7da60700 1 -- 172.21.15.187:0/1914707978 shutdown_connections 2020-11-20T12:20:15.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.500+0000 7f1b7da60700 1 -- 172.21.15.187:0/1914707978 wait complete. 2020-11-20T12:20:15.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.502+0000 7f1b7da60700 1 Processor -- start 2020-11-20T12:20:15.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.502+0000 7f1b7da60700 1 -- start start 2020-11-20T12:20:15.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.502+0000 7f1b7da60700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b78062b50 0x7f1b780b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b7da60700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b780b57b0 0x7f1b7813bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b7da60700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78139d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b7da60700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7813d3a0 con 0x7f1b78062b50 2020-11-20T12:20:15.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b7da60700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7813d640 con 0x7f1b780b57b0 2020-11-20T12:20:15.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b7da60700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7813d020 con 0x7f1b781482d0 2020-11-20T12:20:15.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b77fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b780b57b0 0x7f1b7813bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b77fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b780b57b0 0x7f1b7813bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53424/0 (socket says 172.21.15.187:53424) 2020-11-20T12:20:15.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b77fff700 1 -- 172.21.15.187:0/2346037108 learned_addr learned my addr 172.21.15.187:0/2346037108 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:15.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b7d25f700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78139d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b7ca5e700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b78062b50 0x7f1b780b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b77fff700 1 -- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 msgr2=0x7f1b78139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.503+0000 7f1b77fff700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78139d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b77fff700 1 -- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b78062b50 msgr2=0x7f1b780b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b77fff700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b78062b50 0x7f1b780b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b77fff700 1 -- 172.21.15.187:0/2346037108 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7813da30 con 0x7f1b780b57b0 2020-11-20T12:20:15.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b7ca5e700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b78062b50 0x7f1b780b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:15.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b77fff700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b780b57b0 0x7f1b7813bf10 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f1b64009380 tx=0x7f1b64015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:15.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b74ff9700 1 -- 172.21.15.187:0/2346037108 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64011e20 con 0x7f1b780b57b0 2020-11-20T12:20:15.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b74ff9700 1 -- 172.21.15.187:0/2346037108 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6401d450 con 0x7f1b780b57b0 2020-11-20T12:20:15.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7813cb70 con 0x7f1b780b57b0 2020-11-20T12:20:15.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b74ff9700 1 -- 172.21.15.187:0/2346037108 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6401d7f0 con 0x7f1b780b57b0 2020-11-20T12:20:15.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.504+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7813c400 con 0x7f1b780b57b0 2020-11-20T12:20:15.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.505+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7813d1e0 con 0x7f1b780b57b0 2020-11-20T12:20:15.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.506+0000 7f1b74ff9700 1 -- 172.21.15.187:0/2346037108 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1b64024030 con 0x7f1b780b57b0 2020-11-20T12:20:15.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.506+0000 7f1b74ff9700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b50056d20 0x7f1b500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.507+0000 7f1b74ff9700 1 -- 172.21.15.187:0/2346037108 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5425+0+0 (secure 0 0 0) 0x7f1b64071390 con 0x7f1b780b57b0 2020-11-20T12:20:15.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.507+0000 7f1b7ca5e700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b50056d20 0x7f1b500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.507+0000 7f1b7ca5e700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b50056d20 0x7f1b500591e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7f1b6c006bd0 tx=0x7f1b6c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:15.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.510+0000 7f1b74ff9700 1 -- 172.21.15.187:0/2346037108 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1b64046a60 con 0x7f1b780b57b0 2020-11-20T12:20:15.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:15.714+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1b7813dbc0 con 0x7f1b780b57b0 2020-11-20T12:20:16.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.222+0000 7f1b74ff9700 1 -- 172.21.15.187:0/2346037108 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v146) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1b64022020 con 0x7f1b780b57b0 2020-11-20T12:20:16.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.225+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b50056d20 msgr2=0x7f1b500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:16.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.225+0000 7f1b7da60700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b50056d20 0x7f1b500591e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7f1b6c006bd0 tx=0x7f1b6c008040).stop 2020-11-20T12:20:16.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.225+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b780b57b0 msgr2=0x7f1b7813bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:16.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.225+0000 7f1b7da60700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b780b57b0 0x7f1b7813bf10 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f1b64009380 tx=0x7f1b64015000).stop 2020-11-20T12:20:16.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.226+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 shutdown_connections 2020-11-20T12:20:16.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.226+0000 7f1b7da60700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1b50056d20 0x7f1b500591e0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.226+0000 7f1b7da60700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b78062b50 0x7f1b780b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.226+0000 7f1b7da60700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1b781482d0 0x7f1b78139d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.226+0000 7f1b7da60700 1 --2- 172.21.15.187:0/2346037108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b780b57b0 0x7f1b7813bf10 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.226+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 >> 172.21.15.187:0/2346037108 conn(0x7f1b780bab30 msgr2=0x7f1b78061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:16.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.226+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 shutdown_connections 2020-11-20T12:20:16.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.226+0000 7f1b7da60700 1 -- 172.21.15.187:0/2346037108 wait complete. 2020-11-20T12:20:16.231 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:20:16.258 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:20:16.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.498+0000 7f899db44700 1 Processor -- start 2020-11-20T12:20:16.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.499+0000 7f899db44700 1 -- start start 2020-11-20T12:20:16.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.500+0000 7f899db44700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8998065190 0x7f8998065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:16.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.500+0000 7f899db44700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998065f90 0x7f8998143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:16.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.500+0000 7f899db44700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980b13e0 con 0x7f8998143c70 2020-11-20T12:20:16.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.500+0000 7f899db44700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f899814ab30 con 0x7f8998065190 2020-11-20T12:20:16.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.500+0000 7f899db44700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8998062c10 con 0x7f8998065f90 2020-11-20T12:20:16.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.501+0000 7f899cb42700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8998065190 0x7f8998065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:16.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.501+0000 7f899cb42700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8998065190 0x7f8998065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53428/0 (socket says 172.21.15.187:53428) 2020-11-20T12:20:16.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.501+0000 7f899cb42700 1 -- 172.21.15.187:0/204544869 learned_addr learned my addr 172.21.15.187:0/204544869 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:16.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.501+0000 7f898ffff700 1 --2- 172.21.15.187:0/204544869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998065f90 0x7f8998143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:16.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.502+0000 7f899cb42700 1 -- 172.21.15.187:0/204544869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998065f90 msgr2=0x7f8998143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:16.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.502+0000 7f899cb42700 1 --2- 172.21.15.187:0/204544869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998065f90 0x7f8998143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.502+0000 7f899cb42700 1 -- 172.21.15.187:0/204544869 >> v1:172.21.15.5:6789/0 conn(0x7f8998143c70 legacy=0x7f899809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:16.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.502+0000 7f899cb42700 1 -- 172.21.15.187:0/204544869 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89980b1640 con 0x7f8998065190 2020-11-20T12:20:16.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.502+0000 7f898ffff700 1 --2- 172.21.15.187:0/204544869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998065f90 0x7f8998143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:16.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.502+0000 7f899cb42700 1 --2- 172.21.15.187:0/204544869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8998065190 0x7f8998065570 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f899814ac70 tx=0x7f8994006eb0).ready entity=mon.1 client_cookie=f5eaeac4cc0e7a55 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:16.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.503+0000 7f898f7fe700 1 -- 172.21.15.187:0/204544869 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994009960 con 0x7f8998065190 2020-11-20T12:20:16.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.503+0000 7f898f7fe700 1 -- 172.21.15.187:0/204544869 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8994009ac0 con 0x7f8998065190 2020-11-20T12:20:16.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.503+0000 7f898f7fe700 1 -- 172.21.15.187:0/204544869 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994009e40 con 0x7f8998065190 2020-11-20T12:20:16.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.503+0000 7f899db44700 1 -- 172.21.15.187:0/204544869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8998065190 msgr2=0x7f8998065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:16.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.504+0000 7f899db44700 1 --2- 172.21.15.187:0/204544869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8998065190 0x7f8998065570 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f899814ac70 tx=0x7f8994006eb0).stop 2020-11-20T12:20:16.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.504+0000 7f899db44700 1 -- 172.21.15.187:0/204544869 shutdown_connections 2020-11-20T12:20:16.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.504+0000 7f899db44700 1 --2- 172.21.15.187:0/204544869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8998065190 0x7f8998065570 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.504+0000 7f899db44700 1 --2- 172.21.15.187:0/204544869 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998065f90 0x7f8998143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.504+0000 7f899db44700 1 -- 172.21.15.187:0/204544869 >> 172.21.15.187:0/204544869 conn(0x7f89980bab30 msgr2=0x7f89980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:16.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.504+0000 7f899db44700 1 -- 172.21.15.187:0/204544869 shutdown_connections 2020-11-20T12:20:16.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.504+0000 7f899db44700 1 -- 172.21.15.187:0/204544869 wait complete. 2020-11-20T12:20:16.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.506+0000 7f899db44700 1 Processor -- start 2020-11-20T12:20:16.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.506+0000 7f899db44700 1 -- start start 2020-11-20T12:20:16.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.507+0000 7f899db44700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998065f90 0x7f89980b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:16.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.507+0000 7f899db44700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998143c70 0x7f8998146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:16.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.507+0000 7f899db44700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f899809bc00 0x7f899809f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:16.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.507+0000 7f899db44700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899809dae0 con 0x7f8998065f90 2020-11-20T12:20:16.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.507+0000 7f899db44700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89981473a0 con 0x7f899809bc00 2020-11-20T12:20:16.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.507+0000 7f899db44700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899809dd10 con 0x7f8998143c70 2020-11-20T12:20:16.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.507+0000 7f899d343700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f899809bc00 0x7f899809f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:16.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899d343700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f899809bc00 0x7f899809f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53438/0 (socket says 172.21.15.187:53438) 2020-11-20T12:20:16.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899d343700 1 -- 172.21.15.187:0/285347363 learned_addr learned my addr 172.21.15.187:0/285347363 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:16.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899cb42700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998065f90 0x7f89980b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:16.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f898ffff700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998143c70 0x7f8998146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:16.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899d343700 1 -- 172.21.15.187:0/285347363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998143c70 msgr2=0x7f8998146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:16.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899d343700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998143c70 0x7f8998146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899d343700 1 -- 172.21.15.187:0/285347363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998065f90 msgr2=0x7f89980b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:16.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899d343700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998065f90 0x7f89980b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899d343700 1 -- 172.21.15.187:0/285347363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899809c060 con 0x7f899809bc00 2020-11-20T12:20:16.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.508+0000 7f899d343700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f899809bc00 0x7f899809f590 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f899000ca90 tx=0x7f8990001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:16.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.509+0000 7f898cff9700 1 -- 172.21.15.187:0/285347363 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899000a730 con 0x7f899809bc00 2020-11-20T12:20:16.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.509+0000 7f898cff9700 1 -- 172.21.15.187:0/285347363 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899000a890 con 0x7f899809bc00 2020-11-20T12:20:16.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.509+0000 7f898cff9700 1 -- 172.21.15.187:0/285347363 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899000ac30 con 0x7f899809bc00 2020-11-20T12:20:16.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.509+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899809fa80 con 0x7f899809bc00 2020-11-20T12:20:16.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.509+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899809e3f0 con 0x7f899809bc00 2020-11-20T12:20:16.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.510+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89981471f0 con 0x7f899809bc00 2020-11-20T12:20:16.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.511+0000 7f898cff9700 1 -- 172.21.15.187:0/285347363 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8990010070 con 0x7f899809bc00 2020-11-20T12:20:16.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.511+0000 7f898cff9700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8970056d50 0x7f8970059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:16.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.511+0000 7f898cff9700 1 -- 172.21.15.187:0/285347363 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f899006f990 con 0x7f899809bc00 2020-11-20T12:20:16.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.512+0000 7f899cb42700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8970056d50 0x7f8970059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:16.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.512+0000 7f899cb42700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8970056d50 0x7f8970059210 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f8998063480 tx=0x7f8994015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:16.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.515+0000 7f898cff9700 1 -- 172.21.15.187:0/285347363 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8990045110 con 0x7f899809bc00 2020-11-20T12:20:16.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.770+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f899809ded0 con 0x7f899809bc00 2020-11-20T12:20:16.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.771+0000 7f898cff9700 1 -- 172.21.15.187:0/285347363 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f8990048730 con 0x7f899809bc00 2020-11-20T12:20:16.773 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:16.774 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:20:16.774 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:20:16.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.774 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:20:16.775 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:20:16.775 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.775 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:20:16.775 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.775 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.776 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:20:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:20:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:20:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.777 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.778 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.778 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:20:16.779 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:20:16.779 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.779 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.779 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.780 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:20:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:20:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.781 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.782 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.782 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:20:16.782 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:20:16.783 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.783 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.783 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.783 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.783 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.784 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:20:16.784 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:20:16.784 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.785 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.785 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.785 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.785 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.786 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:20:16.786 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:20:16.786 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.787 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.787 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.787 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.787 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.789 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:20:16.789 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:20:16.789 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.789 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.790 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.790 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.790 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.791 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:20:16.791 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:20:16.791 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.791 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.791 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.792 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.792 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.792 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:20:16.793 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:20:16.793 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.793 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.793 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.793 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.794 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.794 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:20:16.795 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:20:16.795 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.795 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:20:16.795 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:20:16.795 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:20:16.796 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.796 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:20:16.796 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:20:16.797 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.797 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:20:16.797 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:20:16.797 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:20:16.797 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:20:16.798 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.798 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:20:16.798 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:20:16.799 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.799 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:20:16.799 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:20:16.799 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:20:16.799 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:20:16.800 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.800 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:20:16.800 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:20:16.801 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.801 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:20:16.801 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:20:16.801 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:20:16.802 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:20:16.802 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.802 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:20:16.803 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:20:16.803 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.803 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:20:16.803 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:20:16.803 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:20:16.804 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.804 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.804 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:20:16.805 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:20:16.805 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.805 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:20:16.805 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:20:16.805 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.806 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:16.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:16.806 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:20:16.806 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:20:16.807 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:20:16.807 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:20:16.807 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:20:16.807 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:20:16.807 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.808 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:16.808 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:20:16.808 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:20:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8970056d50 msgr2=0x7f8970059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8970056d50 0x7f8970059210 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f8998063480 tx=0x7f8994015000).stop 2020-11-20T12:20:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f899809bc00 msgr2=0x7f899809f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:16.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f899809bc00 0x7f899809f590 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f899000ca90 tx=0x7f8990001360).stop 2020-11-20T12:20:16.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 shutdown_connections 2020-11-20T12:20:16.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8970056d50 0x7f8970059210 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8998065f90 0x7f89980b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f899809bc00 0x7f899809f590 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 --2- 172.21.15.187:0/285347363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8998143c70 0x7f8998146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:16.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 >> 172.21.15.187:0/285347363 conn(0x7f89980bab30 msgr2=0x7f8998066590 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:16.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.773+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 shutdown_connections 2020-11-20T12:20:16.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:16.774+0000 7f899db44700 1 -- 172.21.15.187:0/285347363 wait complete. 2020-11-20T12:20:16.813 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:20:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.062+0000 7f3ba759e700 1 Processor -- start 2020-11-20T12:20:17.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.062+0000 7f3ba759e700 1 -- start start 2020-11-20T12:20:17.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.064+0000 7f3ba759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ba8065190 0x7f3ba8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.064+0000 7f3ba759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 0x7f3ba8143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.064+0000 7f3ba759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3ba8143c70 0x7f3ba809ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.064+0000 7f3ba759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba814ab30 con 0x7f3ba8143c70 2020-11-20T12:20:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.064+0000 7f3ba759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba80b13e0 con 0x7f3ba8065190 2020-11-20T12:20:17.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.064+0000 7f3ba759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8062c10 con 0x7f3ba8065f90 2020-11-20T12:20:17.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.064+0000 7f3ba659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ba8065190 0x7f3ba8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3ba659c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ba8065190 0x7f3ba8065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53446/0 (socket says 172.21.15.187:53446) 2020-11-20T12:20:17.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3ba659c700 1 -- 172.21.15.187:0/2661284397 learned_addr learned my addr 172.21.15.187:0/2661284397 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3ba6d9d700 1 --2- 172.21.15.187:0/2661284397 >> v2:172.21.15.5:3300/0 conn(0x7f3ba8143c70 0x7f3ba809ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3b9dd9b700 1 --2- 172.21.15.187:0/2661284397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 0x7f3ba8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3ba659c700 1 -- 172.21.15.187:0/2661284397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 msgr2=0x7f3ba8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3ba659c700 1 --2- 172.21.15.187:0/2661284397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 0x7f3ba8143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3ba659c700 1 -- 172.21.15.187:0/2661284397 >> v2:172.21.15.5:3300/0 conn(0x7f3ba8143c70 msgr2=0x7f3ba809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3ba659c700 1 --2- 172.21.15.187:0/2661284397 >> v2:172.21.15.5:3300/0 conn(0x7f3ba8143c70 0x7f3ba809ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.065+0000 7f3ba659c700 1 -- 172.21.15.187:0/2661284397 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba80b1640 con 0x7f3ba8065190 2020-11-20T12:20:17.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.066+0000 7f3ba659c700 1 --2- 172.21.15.187:0/2661284397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba8065570 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7f3ba805c8b0 tx=0x7f3b98006eb0).ready entity=mon.1 client_cookie=d437b588da02fec6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:17.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.066+0000 7f3ba5d9b700 1 -- 172.21.15.187:0/2661284397 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b980097b0 con 0x7f3ba8065190 2020-11-20T12:20:17.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.066+0000 7f3ba5d9b700 1 -- 172.21.15.187:0/2661284397 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b98009910 con 0x7f3ba8065190 2020-11-20T12:20:17.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.066+0000 7f3ba5d9b700 1 -- 172.21.15.187:0/2661284397 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b98009c90 con 0x7f3ba8065190 2020-11-20T12:20:17.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 -- 172.21.15.187:0/2661284397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 msgr2=0x7f3ba8065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 --2- 172.21.15.187:0/2661284397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba8065570 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7f3ba805c8b0 tx=0x7f3b98006eb0).stop 2020-11-20T12:20:17.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 -- 172.21.15.187:0/2661284397 shutdown_connections 2020-11-20T12:20:17.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 --2- 172.21.15.187:0/2661284397 >> v2:172.21.15.5:3300/0 conn(0x7f3ba8143c70 0x7f3ba809ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 --2- 172.21.15.187:0/2661284397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba8065570 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 --2- 172.21.15.187:0/2661284397 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 0x7f3ba8143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 -- 172.21.15.187:0/2661284397 >> 172.21.15.187:0/2661284397 conn(0x7f3ba80bab30 msgr2=0x7f3ba80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 -- 172.21.15.187:0/2661284397 shutdown_connections 2020-11-20T12:20:17.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.067+0000 7f3ba759e700 1 -- 172.21.15.187:0/2661284397 wait complete. 2020-11-20T12:20:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.068+0000 7f3ba759e700 1 Processor -- start 2020-11-20T12:20:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.069+0000 7f3ba759e700 1 -- start start 2020-11-20T12:20:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.069+0000 7f3ba759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba80b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.069+0000 7f3ba759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 0x7f3ba8158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.069+0000 7f3ba759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ba8143c70 0x7f3ba80a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.069+0000 7f3ba759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba80b3510 con 0x7f3ba8143c70 2020-11-20T12:20:17.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.069+0000 7f3ba759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba80b37b0 con 0x7f3ba8065190 2020-11-20T12:20:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.069+0000 7f3ba759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba80a0850 con 0x7f3ba8065f90 2020-11-20T12:20:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba80b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba80b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53448/0 (socket says 172.21.15.187:53448) 2020-11-20T12:20:17.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 -- 172.21.15.187:0/3729296650 learned_addr learned my addr 172.21.15.187:0/3729296650 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:17.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 -- 172.21.15.187:0/3729296650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 msgr2=0x7f3ba8158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 0x7f3ba8158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:17.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 -- 172.21.15.187:0/3729296650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ba8143c70 msgr2=0x7f3ba80a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ba8143c70 0x7f3ba80a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:17.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 -- 172.21.15.187:0/3729296650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba80b3120 con 0x7f3ba8065190 2020-11-20T12:20:17.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.070+0000 7f3ba659c700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba80b3ef0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 rx=0x7f3b98006ef0 tx=0x7f3b98015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:17.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.071+0000 7f3b9effd700 1 -- 172.21.15.187:0/3729296650 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b98011e50 con 0x7f3ba8065190 2020-11-20T12:20:17.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.071+0000 7f3b9effd700 1 -- 172.21.15.187:0/3729296650 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9801e460 con 0x7f3ba8065190 2020-11-20T12:20:17.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.071+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba80b4520 con 0x7f3ba8065190 2020-11-20T12:20:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.071+0000 7f3b9effd700 1 -- 172.21.15.187:0/3729296650 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9801e7e0 con 0x7f3ba8065190 2020-11-20T12:20:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.071+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba8158cb0 con 0x7f3ba8065190 2020-11-20T12:20:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.072+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba8061a20 con 0x7f3ba8065190 2020-11-20T12:20:17.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.072+0000 7f3b9effd700 1 -- 172.21.15.187:0/3729296650 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3b98025030 con 0x7f3ba8065190 2020-11-20T12:20:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.074+0000 7f3b9effd700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b84056d20 0x7f3b840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.074+0000 7f3b9effd700 1 -- 172.21.15.187:0/3729296650 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3b98071470 con 0x7f3ba8065190 2020-11-20T12:20:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.074+0000 7f3b9dd9b700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b84056d20 0x7f3b840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.075+0000 7f3b9dd9b700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b84056d20 0x7f3b840591e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f3ba80b2ec0 tx=0x7f3b90006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.079+0000 7f3b9effd700 1 -- 172.21.15.187:0/3729296650 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3b98046b90 con 0x7f3ba8065190 2020-11-20T12:20:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.270+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ba80afc60 con 0x7f3ba8065190 2020-11-20T12:20:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.272+0000 7f3b9effd700 1 -- 172.21.15.187:0/3729296650 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+13637 (secure 0 0 0) 0x7f3b98030560 con 0x7f3ba8065190 2020-11-20T12:20:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b84056d20 msgr2=0x7f3b840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b84056d20 0x7f3b840591e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f3ba80b2ec0 tx=0x7f3b90006df0).stop 2020-11-20T12:20:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 msgr2=0x7f3ba80b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba80b3ef0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 rx=0x7f3b98006ef0 tx=0x7f3b98015000).stop 2020-11-20T12:20:17.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 shutdown_connections 2020-11-20T12:20:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b84056d20 0x7f3b840591e0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ba8143c70 0x7f3ba80a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ba8065190 0x7f3ba80b3ef0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 --2- 172.21.15.187:0/3729296650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ba8065f90 0x7f3ba8158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:17.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 >> 172.21.15.187:0/3729296650 conn(0x7f3ba80bab30 msgr2=0x7f3ba80aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:17.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 shutdown_connections 2020-11-20T12:20:17.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.275+0000 7f3ba759e700 1 -- 172.21.15.187:0/3729296650 wait complete. 2020-11-20T12:20:17.507 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:17.508 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":146,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:20:16.205233+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:20:17.509 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:20:17.509 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:20:17.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.759+0000 7f09991f9700 1 Processor -- start 2020-11-20T12:20:17.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.760+0000 7f09991f9700 1 -- start start 2020-11-20T12:20:17.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.761+0000 7f09991f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09940683f0 0x7f0994066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.761+0000 7f09991f9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099414c170 con 0x7f0994145cc0 2020-11-20T12:20:17.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.761+0000 7f09991f9700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099414c410 con 0x7f09941487e0 2020-11-20T12:20:17.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.761+0000 7f09991f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099414c6b0 con 0x7f09940683f0 2020-11-20T12:20:17.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.761+0000 7f09989f8700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f09941487e0 0x7f0994148c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50066/0 (socket says 172.21.15.187:50066) 2020-11-20T12:20:17.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.761+0000 7f09989f8700 1 -- 172.21.15.187:0/3895459338 learned_addr learned my addr 172.21.15.187:0/3895459338 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:17.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.762+0000 7f098b7fe700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09940683f0 0x7f0994066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.762+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633662099 0 0) 0x7f099414c410 con 0x7f09941487e0 2020-11-20T12:20:17.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.762+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0974003240 con 0x7f09941487e0 2020-11-20T12:20:17.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3953394602 0 0) 0x7f0974003240 con 0x7f09941487e0 2020-11-20T12:20:17.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09940683f0 msgr2=0x7f0994066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:17.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09940683f0 0x7f0994066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 >> v1:172.21.15.5:6789/0 conn(0x7f0994145cc0 legacy=0x7f0994065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09940658a0 con 0x7f09941487e0 2020-11-20T12:20:17.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f098c001860 con 0x7f09941487e0 2020-11-20T12:20:17.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 >> v1:172.21.15.187:6789/0 conn(0x7f09941487e0 legacy=0x7f0994148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0974003730 0x7f0974003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09940683f0 0x7f09740044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09740049d0 0x7f0974004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099414c6b0 con 0x7f09740049d0 2020-11-20T12:20:17.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0974004220 con 0x7f0974003730 2020-11-20T12:20:17.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097400b730 con 0x7f09940683f0 2020-11-20T12:20:17.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.763+0000 7f0993fff700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0974003730 0x7f0974003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.764+0000 7f098b7fe700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09940683f0 0x7f09740044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.764+0000 7f09989f8700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09740049d0 0x7f0974004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.764+0000 7f0993fff700 1 -- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09940683f0 msgr2=0x7f09740044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:17.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.764+0000 7f0993fff700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09940683f0 0x7f09740044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.764+0000 7f0993fff700 1 -- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09740049d0 msgr2=0x7f0974004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:17.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.764+0000 7f0993fff700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09740049d0 0x7f0974004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.764+0000 7f0993fff700 1 -- 172.21.15.187:0/3895459338 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0974003240 con 0x7f0974003730 2020-11-20T12:20:17.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.764+0000 7f0993fff700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0974003730 0x7f0974003b50 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f0984001920 tx=0x7f0984007b90).ready entity=mon.1 client_cookie=3789e99e206a1df0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:17.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984007020 con 0x7f0974003730 2020-11-20T12:20:17.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0984007180 con 0x7f0974003730 2020-11-20T12:20:17.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09937fe700 1 -- 172.21.15.187:0/3895459338 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098400c620 con 0x7f0974003730 2020-11-20T12:20:17.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09991f9700 1 -- 172.21.15.187:0/3895459338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0974003730 msgr2=0x7f0974003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09991f9700 1 --2- 172.21.15.187:0/3895459338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0974003730 0x7f0974003b50 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f0984001920 tx=0x7f0984007b90).stop 2020-11-20T12:20:17.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09989f8700 1 -- 172.21.15.187:0/3895459338 reap_dead start 2020-11-20T12:20:17.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09991f9700 1 -- 172.21.15.187:0/3895459338 shutdown_connections 2020-11-20T12:20:17.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09991f9700 1 -- 172.21.15.187:0/3895459338 >> 172.21.15.187:0/3895459338 conn(0x7f09940bab30 msgr2=0x7f09940baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:17.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09991f9700 1 -- 172.21.15.187:0/3895459338 shutdown_connections 2020-11-20T12:20:17.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.765+0000 7f09991f9700 1 -- 172.21.15.187:0/3895459338 wait complete. 2020-11-20T12:20:17.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.767+0000 7f09991f9700 1 Processor -- start 2020-11-20T12:20:17.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.767+0000 7f09991f9700 1 -- start start 2020-11-20T12:20:17.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09991f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09941392f0 0x7f099409f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09991f9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f099409fc00 0x7f0994149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09991f9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099414a150 0x7f099409a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09991f9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09940a0100 con 0x7f099409fc00 2020-11-20T12:20:17.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09991f9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09940a03a0 con 0x7f099414a150 2020-11-20T12:20:17.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09991f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099409d3e0 con 0x7f09941392f0 2020-11-20T12:20:17.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09989f8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099414a150 0x7f099409a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09989f8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099414a150 0x7f099409a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53472/0 (socket says 172.21.15.187:53472) 2020-11-20T12:20:17.782 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.768+0000 7f09989f8700 1 -- 172.21.15.187:0/4027567569 learned_addr learned my addr 172.21.15.187:0/4027567569 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:17.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f0993fff700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09941392f0 0x7f099409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f09989f8700 1 -- 172.21.15.187:0/4027567569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09941392f0 msgr2=0x7f099409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f09989f8700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09941392f0 0x7f099409f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f09989f8700 1 -- 172.21.15.187:0/4027567569 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f099409fc00 msgr2=0x7f0994149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f09989f8700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f099409fc00 0x7f0994149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:17.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f09989f8700 1 -- 172.21.15.187:0/4027567569 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09941396d0 con 0x7f099414a150 2020-11-20T12:20:17.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f09989f8700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099414a150 0x7f099409a670 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 rx=0x7f0994148bc0 tx=0x7f098c009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:17.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f0990ff9700 1 -- 172.21.15.187:0/4027567569 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c00bc80 con 0x7f099414a150 2020-11-20T12:20:17.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f0990ff9700 1 -- 172.21.15.187:0/4027567569 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098c00be00 con 0x7f099414a150 2020-11-20T12:20:17.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099409ab60 con 0x7f099414a150 2020-11-20T12:20:17.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f0990ff9700 1 -- 172.21.15.187:0/4027567569 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c004b90 con 0x7f099414a150 2020-11-20T12:20:17.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.769+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099409b000 con 0x7f099414a150 2020-11-20T12:20:17.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.771+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0994061a20 con 0x7f099414a150 2020-11-20T12:20:17.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.771+0000 7f0990ff9700 1 -- 172.21.15.187:0/4027567569 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f098c01e070 con 0x7f099414a150 2020-11-20T12:20:17.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.772+0000 7f0990ff9700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0970056d20 0x7f09700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:17.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.772+0000 7f0990ff9700 1 -- 172.21.15.187:0/4027567569 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f098c06f3f0 con 0x7f099414a150 2020-11-20T12:20:17.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.772+0000 7f0993fff700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0970056d20 0x7f09700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:17.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.773+0000 7f0993fff700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0970056d20 0x7f09700591e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f0994063a20 tx=0x7f0984015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:17.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.776+0000 7f0990ff9700 1 -- 172.21.15.187:0/4027567569 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f098c044b70 con 0x7f099414a150 2020-11-20T12:20:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.966+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f099414bdf0 con 0x7f099414a150 2020-11-20T12:20:17.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.967+0000 7f0990ff9700 1 -- 172.21.15.187:0/4027567569 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v146) v1 ==== 110+0+10 (secure 0 0 0) 0x7f098c048190 con 0x7f099414a150 2020-11-20T12:20:17.970 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:20:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.969+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0970056d20 msgr2=0x7f09700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.969+0000 7f09991f9700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0970056d20 0x7f09700591e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f0994063a20 tx=0x7f0984015000).stop 2020-11-20T12:20:17.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.969+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099414a150 msgr2=0x7f099409a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:17.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.969+0000 7f09991f9700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099414a150 0x7f099409a670 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 rx=0x7f0994148bc0 tx=0x7f098c009040).stop 2020-11-20T12:20:17.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.970+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 shutdown_connections 2020-11-20T12:20:17.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.970+0000 7f09991f9700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0970056d20 0x7f09700591e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.970+0000 7f09991f9700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099414a150 0x7f099409a670 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.970+0000 7f09991f9700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f09941392f0 0x7f099409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:17.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.970+0000 7f09991f9700 1 --2- 172.21.15.187:0/4027567569 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f099409fc00 0x7f0994149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:17.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.970+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 >> 172.21.15.187:0/4027567569 conn(0x7f09940bab30 msgr2=0x7f0994065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:17.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.970+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 shutdown_connections 2020-11-20T12:20:17.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:17.970+0000 7f09991f9700 1 -- 172.21.15.187:0/4027567569 wait complete. 2020-11-20T12:20:17.999 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:20:18.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2020-11-20T12:20:18.287 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.263+0000 7fc1b73a4700 1 Processor -- start 2020-11-20T12:20:18.288 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.264+0000 7fc1b73a4700 1 -- start start 2020-11-20T12:20:18.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.265+0000 7fc1b73a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b01484d0 0x7fc1b0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.265+0000 7fc1b73a4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1b0066f70 0x7fc1b00b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.265+0000 7fc1b73a4700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00b1300 con 0x7fc1b00622c0 2020-11-20T12:20:18.289 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.265+0000 7fc1b73a4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b0067730 con 0x7fc1b0066f70 2020-11-20T12:20:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.265+0000 7fc1b73a4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00b15a0 con 0x7fc1b01484d0 2020-11-20T12:20:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b5ba1700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc1b00622c0 0x7fc1b00b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50084/0 (socket says 172.21.15.187:50084) 2020-11-20T12:20:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b5ba1700 1 -- 172.21.15.187:0/202370685 learned_addr learned my addr 172.21.15.187:0/202370685 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b6ba3700 1 --2- 172.21.15.187:0/202370685 >> v2:172.21.15.187:3300/0 conn(0x7fc1b0066f70 0x7fc1b00b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.290 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b63a2700 1 --2- 172.21.15.187:0/202370685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b01484d0 0x7fc1b0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b53a0700 1 -- 172.21.15.187:0/202370685 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418757549 0 0) 0x7fc1b00b1300 con 0x7fc1b00622c0 2020-11-20T12:20:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b53a0700 1 -- 172.21.15.187:0/202370685 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc198003240 con 0x7fc1b00622c0 2020-11-20T12:20:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b6ba3700 1 -- 172.21.15.187:0/202370685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b01484d0 msgr2=0x7fc1b0061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b6ba3700 1 --2- 172.21.15.187:0/202370685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b01484d0 0x7fc1b0061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.291 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b6ba3700 1 -- 172.21.15.187:0/202370685 >> v1:172.21.15.187:6789/0 conn(0x7fc1b00622c0 legacy=0x7fc1b00b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.266+0000 7fc1b6ba3700 1 -- 172.21.15.187:0/202370685 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00b1840 con 0x7fc1b0066f70 2020-11-20T12:20:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.267+0000 7fc1b6ba3700 1 --2- 172.21.15.187:0/202370685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b0066f70 0x7fc1b00b5d50 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7fc1a8009c90 tx=0x7fc1a80091d0).ready entity=mon.1 client_cookie=c0a7856a69b524be server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.267+0000 7fc1b53a0700 1 -- 172.21.15.187:0/202370685 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800b9c0 con 0x7fc1b0066f70 2020-11-20T12:20:18.292 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.267+0000 7fc1b53a0700 1 -- 172.21.15.187:0/202370685 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a800bb20 con 0x7fc1b0066f70 2020-11-20T12:20:18.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.267+0000 7fc1b53a0700 1 -- 172.21.15.187:0/202370685 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800bea0 con 0x7fc1b0066f70 2020-11-20T12:20:18.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.268+0000 7fc1b73a4700 1 -- 172.21.15.187:0/202370685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b0066f70 msgr2=0x7fc1b00b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.293 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.268+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/202370685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b0066f70 0x7fc1b00b5d50 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7fc1a8009c90 tx=0x7fc1a80091d0).stop 2020-11-20T12:20:18.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.268+0000 7fc1b73a4700 1 -- 172.21.15.187:0/202370685 shutdown_connections 2020-11-20T12:20:18.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.268+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/202370685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b0066f70 0x7fc1b00b5d50 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.268+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/202370685 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b01484d0 0x7fc1b0061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.268+0000 7fc1b73a4700 1 -- 172.21.15.187:0/202370685 >> 172.21.15.187:0/202370685 conn(0x7fc1b00bab30 msgr2=0x7fc1b00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:18.294 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.268+0000 7fc1b73a4700 1 -- 172.21.15.187:0/202370685 shutdown_connections 2020-11-20T12:20:18.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.268+0000 7fc1b73a4700 1 -- 172.21.15.187:0/202370685 wait complete. 2020-11-20T12:20:18.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.269+0000 7fc1b73a4700 1 Processor -- start 2020-11-20T12:20:18.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.269+0000 7fc1b73a4700 1 -- start start 2020-11-20T12:20:18.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.270+0000 7fc1b73a4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1b00622c0 0x7fc1b00b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.295 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.270+0000 7fc1b73a4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b0066f70 0x7fc1b00b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.270+0000 7fc1b73a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b01484d0 0x7fc1b013a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.270+0000 7fc1b73a4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b009d2f0 con 0x7fc1b00622c0 2020-11-20T12:20:18.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.270+0000 7fc1b73a4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00a05e0 con 0x7fc1b01484d0 2020-11-20T12:20:18.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.270+0000 7fc1b73a4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00b2220 con 0x7fc1b0066f70 2020-11-20T12:20:18.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b6ba3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b01484d0 0x7fc1b013a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b6ba3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b01484d0 0x7fc1b013a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53486/0 (socket says 172.21.15.187:53486) 2020-11-20T12:20:18.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b6ba3700 1 -- 172.21.15.187:0/2284450308 learned_addr learned my addr 172.21.15.187:0/2284450308 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:18.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b5ba1700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b0066f70 0x7fc1b00b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b6ba3700 1 -- 172.21.15.187:0/2284450308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b0066f70 msgr2=0x7fc1b00b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b6ba3700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b0066f70 0x7fc1b00b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b6ba3700 1 -- 172.21.15.187:0/2284450308 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1b00622c0 msgr2=0x7fc1b00b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b6ba3700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1b00622c0 0x7fc1b00b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b6ba3700 1 -- 172.21.15.187:0/2284450308 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00b34a0 con 0x7fc1b01484d0 2020-11-20T12:20:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.271+0000 7fc1b5ba1700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b0066f70 0x7fc1b00b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.272+0000 7fc1b6ba3700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b01484d0 0x7fc1b013a300 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7fc1a8009210 tx=0x7fc1a8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.272+0000 7fc1a27fc700 1 -- 172.21.15.187:0/2284450308 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a801c5f0 con 0x7fc1b01484d0 2020-11-20T12:20:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.272+0000 7fc1a27fc700 1 -- 172.21.15.187:0/2284450308 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a801c750 con 0x7fc1b01484d0 2020-11-20T12:20:18.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.272+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b00a0a80 con 0x7fc1b01484d0 2020-11-20T12:20:18.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.272+0000 7fc1a27fc700 1 -- 172.21.15.187:0/2284450308 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a801cb10 con 0x7fc1b01484d0 2020-11-20T12:20:18.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.272+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b009d090 con 0x7fc1b01484d0 2020-11-20T12:20:18.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.273+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b00a07b0 con 0x7fc1b01484d0 2020-11-20T12:20:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.274+0000 7fc1a27fc700 1 -- 172.21.15.187:0/2284450308 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc1a8017070 con 0x7fc1b01484d0 2020-11-20T12:20:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.275+0000 7fc1a27fc700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc188056d20 0x7fc1880591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.275+0000 7fc1a27fc700 1 -- 172.21.15.187:0/2284450308 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc1a8072280 con 0x7fc1b01484d0 2020-11-20T12:20:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.276+0000 7fc1b63a2700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc188056d20 0x7fc1880591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.276+0000 7fc1b63a2700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc188056d20 0x7fc1880591e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fc1ac006fd0 tx=0x7fc1ac006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.279+0000 7fc1a27fc700 1 -- 172.21.15.187:0/2284450308 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc1a8047980 con 0x7fc1b01484d0 2020-11-20T12:20:18.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.465+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc1b009d4c0 con 0x7fc1b01484d0 2020-11-20T12:20:18.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.465+0000 7fc1a27fc700 1 -- 172.21.15.187:0/2284450308 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v146) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc1a804afa0 con 0x7fc1b01484d0 2020-11-20T12:20:18.469 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:20:18.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.467+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc188056d20 msgr2=0x7fc1880591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.467+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc188056d20 0x7fc1880591e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fc1ac006fd0 tx=0x7fc1ac006df0).stop 2020-11-20T12:20:18.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.467+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b01484d0 msgr2=0x7fc1b013a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b01484d0 0x7fc1b013a300 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7fc1a8009210 tx=0x7fc1a8016000).stop 2020-11-20T12:20:18.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 shutdown_connections 2020-11-20T12:20:18.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc188056d20 0x7fc1880591e0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1b00622c0 0x7fc1b00b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:18.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1b01484d0 0x7fc1b013a300 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 --2- 172.21.15.187:0/2284450308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc1b0066f70 0x7fc1b00b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 >> 172.21.15.187:0/2284450308 conn(0x7fc1b00bab30 msgr2=0x7fc1b013b810 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:18.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 shutdown_connections 2020-11-20T12:20:18.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.468+0000 7fc1b73a4700 1 -- 172.21.15.187:0/2284450308 wait complete. 2020-11-20T12:20:18.496 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:20:18.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.740+0000 7f931c9c5700 1 Processor -- start 2020-11-20T12:20:18.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.741+0000 7f931c9c5700 1 -- start start 2020-11-20T12:20:18.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.742+0000 7f931c9c5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9318065190 0x7f9318065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.745 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.742+0000 7f931c9c5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.742+0000 7f931c9c5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93180b13e0 con 0x7f9318143c70 2020-11-20T12:20:18.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.742+0000 7f931c9c5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931814ab30 con 0x7f9318065190 2020-11-20T12:20:18.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.742+0000 7f931c9c5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9318062c10 con 0x7f9318065f90 2020-11-20T12:20:18.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.743+0000 7f930effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.743+0000 7f930effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:54378/0 (socket says 172.21.15.187:54378) 2020-11-20T12:20:18.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.743+0000 7f930effd700 1 -- 172.21.15.187:0/3377685398 learned_addr learned my addr 172.21.15.187:0/3377685398 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:18.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.743+0000 7f93177fe700 1 --2- 172.21.15.187:0/3377685398 >> v2:172.21.15.5:3300/0 conn(0x7f9318065190 0x7f9318065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.744+0000 7f930effd700 1 -- 172.21.15.187:0/3377685398 >> v2:172.21.15.5:3300/0 conn(0x7f9318065190 msgr2=0x7f9318065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:18.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.744+0000 7f930effd700 1 --2- 172.21.15.187:0/3377685398 >> v2:172.21.15.5:3300/0 conn(0x7f9318065190 0x7f9318065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.744+0000 7f930effd700 1 -- 172.21.15.187:0/3377685398 >> v1:172.21.15.5:6789/0 conn(0x7f9318143c70 legacy=0x7f931809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:18.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.744+0000 7f930effd700 1 -- 172.21.15.187:0/3377685398 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93180b1640 con 0x7f9318065f90 2020-11-20T12:20:18.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.745+0000 7f930effd700 1 --2- 172.21.15.187:0/3377685398 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318143780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f931814ac70 tx=0x7f9300006eb0).ready entity=mon.2 client_cookie=b3951b2ab603a0d4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.745+0000 7f9316ffd700 1 -- 172.21.15.187:0/3377685398 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300009860 con 0x7f9318065f90 2020-11-20T12:20:18.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.745+0000 7f9316ffd700 1 -- 172.21.15.187:0/3377685398 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93000099c0 con 0x7f9318065f90 2020-11-20T12:20:18.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.745+0000 7f9316ffd700 1 -- 172.21.15.187:0/3377685398 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300009d40 con 0x7f9318065f90 2020-11-20T12:20:18.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.746+0000 7f931c9c5700 1 -- 172.21.15.187:0/3377685398 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 msgr2=0x7f9318143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.746+0000 7f931c9c5700 1 --2- 172.21.15.187:0/3377685398 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318143780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f931814ac70 tx=0x7f9300006eb0).stop 2020-11-20T12:20:18.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.746+0000 7f931c9c5700 1 -- 172.21.15.187:0/3377685398 shutdown_connections 2020-11-20T12:20:18.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.746+0000 7f931c9c5700 1 --2- 172.21.15.187:0/3377685398 >> v2:172.21.15.5:3300/0 conn(0x7f9318065190 0x7f9318065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.746+0000 7f931c9c5700 1 --2- 172.21.15.187:0/3377685398 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318143780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.746+0000 7f931c9c5700 1 -- 172.21.15.187:0/3377685398 >> 172.21.15.187:0/3377685398 conn(0x7f93180bab30 msgr2=0x7f93180baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:18.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.746+0000 7f931c9c5700 1 -- 172.21.15.187:0/3377685398 shutdown_connections 2020-11-20T12:20:18.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.746+0000 7f931c9c5700 1 -- 172.21.15.187:0/3377685398 wait complete. 2020-11-20T12:20:18.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.748+0000 7f931c9c5700 1 Processor -- start 2020-11-20T12:20:18.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.748+0000 7f931c9c5700 1 -- start start 2020-11-20T12:20:18.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.749+0000 7f931c9c5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9318065190 0x7f93180b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.749+0000 7f931c9c5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.749+0000 7f931c9c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9318143c70 0x7f931809cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.749+0000 7f931c9c5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93180b2870 con 0x7f9318065190 2020-11-20T12:20:18.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.749+0000 7f931c9c5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93180b2b10 con 0x7f9318143c70 2020-11-20T12:20:18.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.749+0000 7f931c9c5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93180b2db0 con 0x7f9318065f90 2020-11-20T12:20:18.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f9317fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9318143c70 0x7f931809cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f9317fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9318143c70 0x7f931809cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53498/0 (socket says 172.21.15.187:53498) 2020-11-20T12:20:18.761 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f9317fff700 1 -- 172.21.15.187:0/1029597448 learned_addr learned my addr 172.21.15.187:0/1029597448 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:18.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f930effd700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f9317fff700 1 -- 172.21.15.187:0/1029597448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 msgr2=0x7f9318156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f9317fff700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f9317fff700 1 -- 172.21.15.187:0/1029597448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9318065190 msgr2=0x7f93180b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f9317fff700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9318065190 0x7f93180b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:18.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f9317fff700 1 -- 172.21.15.187:0/1029597448 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931809d110 con 0x7f9318143c70 2020-11-20T12:20:18.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.750+0000 7f930effd700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:18.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.751+0000 7f9317fff700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9318143c70 0x7f931809cc20 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 rx=0x7f9310009690 tx=0x7f9310006cf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.751+0000 7f930ffff700 1 -- 172.21.15.187:0/1029597448 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931000d560 con 0x7f9318143c70 2020-11-20T12:20:18.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.751+0000 7f930ffff700 1 -- 172.21.15.187:0/1029597448 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9310000e90 con 0x7f9318143c70 2020-11-20T12:20:18.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.751+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93181487c0 con 0x7f9318143c70 2020-11-20T12:20:18.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.751+0000 7f930ffff700 1 -- 172.21.15.187:0/1029597448 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931000d930 con 0x7f9318143c70 2020-11-20T12:20:18.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.751+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9318142e00 con 0x7f9318143c70 2020-11-20T12:20:18.765 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.752+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93180b2f70 con 0x7f9318143c70 2020-11-20T12:20:18.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.753+0000 7f930ffff700 1 -- 172.21.15.187:0/1029597448 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f931000f030 con 0x7f9318143c70 2020-11-20T12:20:18.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.754+0000 7f930ffff700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92f4056d20 0x7f92f40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.754+0000 7f930ffff700 1 -- 172.21.15.187:0/1029597448 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f931006f840 con 0x7f9318143c70 2020-11-20T12:20:18.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.754+0000 7f93177fe700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92f4056d20 0x7f92f40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.755+0000 7f93177fe700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92f4056d20 0x7f92f40591e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f93180b1f40 tx=0x7f93080091d0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.758+0000 7f930ffff700 1 -- 172.21.15.187:0/1029597448 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9310044fc0 con 0x7f9318143c70 2020-11-20T12:20:18.946 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.943+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f93180b2680 con 0x7f9318143c70 2020-11-20T12:20:18.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.944+0000 7f930ffff700 1 -- 172.21.15.187:0/1029597448 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v146) v1 ==== 84+0+143 (secure 0 0 0) 0x7f93100485e0 con 0x7f9318143c70 2020-11-20T12:20:18.950 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:18.950 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:20:18.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:18.951 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:20:18.951 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:20:18.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:20:18.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:20:18.952 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:20:18.952 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:20:18.952 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:20:18.952 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:20:18.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.946+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92f4056d20 msgr2=0x7f92f40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.946+0000 7f931c9c5700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92f4056d20 0x7f92f40591e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f93180b1f40 tx=0x7f93080091d0).stop 2020-11-20T12:20:18.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.946+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9318143c70 msgr2=0x7f931809cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.946+0000 7f931c9c5700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9318143c70 0x7f931809cc20 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 rx=0x7f9310009690 tx=0x7f9310006cf0).stop 2020-11-20T12:20:18.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.947+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 shutdown_connections 2020-11-20T12:20:18.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.947+0000 7f931c9c5700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92f4056d20 0x7f92f40591e0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.947+0000 7f931c9c5700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9318065190 0x7f93180b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:18.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.947+0000 7f931c9c5700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9318143c70 0x7f931809cc20 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.947+0000 7f931c9c5700 1 --2- 172.21.15.187:0/1029597448 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9318065f90 0x7f9318156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.947+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 >> 172.21.15.187:0/1029597448 conn(0x7f93180bab30 msgr2=0x7f93181458d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:18.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.947+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 shutdown_connections 2020-11-20T12:20:18.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:18.947+0000 7f931c9c5700 1 -- 172.21.15.187:0/1029597448 wait complete. 2020-11-20T12:20:18.974 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:20:18.975 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:20:19.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.219+0000 7f348933b700 1 Processor -- start 2020-11-20T12:20:19.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.219+0000 7f348933b700 1 -- start start 2020-11-20T12:20:19.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.220+0000 7f348933b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 0x7f3484061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.220+0000 7f348933b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3484066f70 0x7f34840b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.227 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.220+0000 7f348933b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840b1300 con 0x7f34840622c0 2020-11-20T12:20:19.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.220+0000 7f348933b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3484067730 con 0x7f3484066f70 2020-11-20T12:20:19.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.220+0000 7f348933b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34840b15a0 con 0x7f34841484d0 2020-11-20T12:20:19.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.221+0000 7f3488b3a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3484066f70 0x7f34840b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.221+0000 7f3483fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 0x7f3484061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.228 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.221+0000 7f3488b3a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3484066f70 0x7f34840b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:45770/0 (socket says 172.21.15.187:45770) 2020-11-20T12:20:19.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.221+0000 7f3488b3a700 1 -- 172.21.15.187:0/2434020138 learned_addr learned my addr 172.21.15.187:0/2434020138 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:19.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.222+0000 7f3488b3a700 1 -- 172.21.15.187:0/2434020138 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 msgr2=0x7f3484061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:19.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.222+0000 7f3488b3a700 1 --2- 172.21.15.187:0/2434020138 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 0x7f3484061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.222+0000 7f3488b3a700 1 -- 172.21.15.187:0/2434020138 >> v1:172.21.15.5:6789/0 conn(0x7f34840622c0 legacy=0x7f34840b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.222+0000 7f3488b3a700 1 -- 172.21.15.187:0/2434020138 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34840b1840 con 0x7f3484066f70 2020-11-20T12:20:19.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.222+0000 7f3483fff700 1 --2- 172.21.15.187:0/2434020138 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 0x7f3484061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:20:19.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.223+0000 7f3488b3a700 1 --2- 172.21.15.187:0/2434020138 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 0x7f34840b5d50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f347c009d50 tx=0x7f347c009290).ready entity=mon.0 client_cookie=eb2c0f0606702d19 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:19.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.223+0000 7f34837fe700 1 -- 172.21.15.187:0/2434020138 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c00bbc0 con 0x7f3484066f70 2020-11-20T12:20:19.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f34837fe700 1 -- 172.21.15.187:0/2434020138 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347c00bd20 con 0x7f3484066f70 2020-11-20T12:20:19.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f34837fe700 1 -- 172.21.15.187:0/2434020138 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c011610 con 0x7f3484066f70 2020-11-20T12:20:19.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f348933b700 1 -- 172.21.15.187:0/2434020138 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 msgr2=0x7f34840b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f348933b700 1 --2- 172.21.15.187:0/2434020138 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 0x7f34840b5d50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f347c009d50 tx=0x7f347c009290).stop 2020-11-20T12:20:19.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f348933b700 1 -- 172.21.15.187:0/2434020138 shutdown_connections 2020-11-20T12:20:19.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f348933b700 1 --2- 172.21.15.187:0/2434020138 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 0x7f34840b5d50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f348933b700 1 --2- 172.21.15.187:0/2434020138 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 0x7f3484061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f348933b700 1 -- 172.21.15.187:0/2434020138 >> 172.21.15.187:0/2434020138 conn(0x7f34840bab30 msgr2=0x7f34840baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:19.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f348933b700 1 -- 172.21.15.187:0/2434020138 shutdown_connections 2020-11-20T12:20:19.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.224+0000 7f348933b700 1 -- 172.21.15.187:0/2434020138 wait complete. 2020-11-20T12:20:19.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.226+0000 7f348933b700 1 Processor -- start 2020-11-20T12:20:19.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.226+0000 7f348933b700 1 -- start start 2020-11-20T12:20:19.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.227+0000 7f348933b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34840622c0 0x7f348409f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f348933b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 0x7f348413d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f348933b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 0x7f3484136a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f348933b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348413d200 con 0x7f3484066f70 2020-11-20T12:20:19.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f348933b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348413d4a0 con 0x7f34840622c0 2020-11-20T12:20:19.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f348933b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348409fcc0 con 0x7f34841484d0 2020-11-20T12:20:19.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f3483fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34840622c0 0x7f348409f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f3483fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34840622c0 0x7f348409f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53510/0 (socket says 172.21.15.187:53510) 2020-11-20T12:20:19.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f3483fff700 1 -- 172.21.15.187:0/100150092 learned_addr learned my addr 172.21.15.187:0/100150092 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:19.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f3483fff700 1 -- 172.21.15.187:0/100150092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 msgr2=0x7f3484136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f3483fff700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 0x7f3484136a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:19.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f3483fff700 1 -- 172.21.15.187:0/100150092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 msgr2=0x7f348413d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.228+0000 7f347bfff700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 0x7f348413d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.229+0000 7f3483fff700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 0x7f348413d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.229+0000 7f3483fff700 1 -- 172.21.15.187:0/100150092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348409ec80 con 0x7f34840622c0 2020-11-20T12:20:19.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.229+0000 7f3483fff700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34840622c0 0x7f348409f7d0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f347400ed20 tx=0x7f3474006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:19.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.229+0000 7f3480ff9700 1 -- 172.21.15.187:0/100150092 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400cb50 con 0x7f34840622c0 2020-11-20T12:20:19.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.229+0000 7f3480ff9700 1 -- 172.21.15.187:0/100150092 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400ccb0 con 0x7f34840622c0 2020-11-20T12:20:19.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.229+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348413dd60 con 0x7f34840622c0 2020-11-20T12:20:19.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.229+0000 7f3480ff9700 1 -- 172.21.15.187:0/100150092 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474012600 con 0x7f34840622c0 2020-11-20T12:20:19.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.229+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3484137140 con 0x7f34840622c0 2020-11-20T12:20:19.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.230+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3484061a20 con 0x7f34840622c0 2020-11-20T12:20:19.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.231+0000 7f3480ff9700 1 -- 172.21.15.187:0/100150092 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3474012970 con 0x7f34840622c0 2020-11-20T12:20:19.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.231+0000 7f3480ff9700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3460056d20 0x7f34600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.232+0000 7f3480ff9700 1 -- 172.21.15.187:0/100150092 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f34740703b0 con 0x7f34840622c0 2020-11-20T12:20:19.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.232+0000 7f347bfff700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3460056d20 0x7f34600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.232+0000 7f347bfff700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3460056d20 0x7f34600591e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f3484065120 tx=0x7f346c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:19.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.236+0000 7f3480ff9700 1 -- 172.21.15.187:0/100150092 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3474045a20 con 0x7f34840622c0 2020-11-20T12:20:19.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.421+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f348409b110 con 0x7f34840622c0 2020-11-20T12:20:19.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.520+0000 7f3480ff9700 1 -- 172.21.15.187:0/100150092 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v147) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3474049040 con 0x7f34840622c0 2020-11-20T12:20:19.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.523+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3460056d20 msgr2=0x7f34600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.523+0000 7f348933b700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3460056d20 0x7f34600591e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f3484065120 tx=0x7f346c009040).stop 2020-11-20T12:20:19.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.523+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34840622c0 msgr2=0x7f348409f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.523+0000 7f348933b700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34840622c0 0x7f348409f7d0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f347400ed20 tx=0x7f3474006df0).stop 2020-11-20T12:20:19.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.524+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 shutdown_connections 2020-11-20T12:20:19.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.524+0000 7f348933b700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3460056d20 0x7f34600591e0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.524+0000 7f348933b700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f34841484d0 0x7f3484136a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:19.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.524+0000 7f348933b700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34840622c0 0x7f348409f7d0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.524+0000 7f348933b700 1 --2- 172.21.15.187:0/100150092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3484066f70 0x7f348413d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.524+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 >> 172.21.15.187:0/100150092 conn(0x7f34840bab30 msgr2=0x7f3484065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:19.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.524+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 shutdown_connections 2020-11-20T12:20:19.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.524+0000 7f348933b700 1 -- 172.21.15.187:0/100150092 wait complete. 2020-11-20T12:20:19.531 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:20:19.557 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:20:19.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.814+0000 7f21db59e700 1 Processor -- start 2020-11-20T12:20:19.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.814+0000 7f21db59e700 1 -- start start 2020-11-20T12:20:19.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.815+0000 7f21db59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 0x7f21dc14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.815+0000 7f21db59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f21dc0bfbe0 0x7f21dc1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.815+0000 7f21db59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21dc14d110 con 0x7f21dc09f560 2020-11-20T12:20:19.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.815+0000 7f21db59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc14cbd0 con 0x7f21dc0bfbe0 2020-11-20T12:20:19.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.815+0000 7f21db59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc14ce70 con 0x7f21dc0bf800 2020-11-20T12:20:19.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.816+0000 7f21d9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f21dc0bfbe0 0x7f21dc1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.816+0000 7f21d9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f21dc0bfbe0 0x7f21dc1480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53520/0 (socket says 172.21.15.187:53520) 2020-11-20T12:20:19.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.816+0000 7f21d9d9b700 1 -- 172.21.15.187:0/278913185 learned_addr learned my addr 172.21.15.187:0/278913185 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:19.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.816+0000 7f21da59c700 1 --2- 172.21.15.187:0/278913185 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 0x7f21dc14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.817+0000 7f21d9d9b700 1 -- 172.21.15.187:0/278913185 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 msgr2=0x7f21dc14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:19.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.817+0000 7f21d9d9b700 1 --2- 172.21.15.187:0/278913185 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 0x7f21dc14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.817+0000 7f21d9d9b700 1 -- 172.21.15.187:0/278913185 >> v1:172.21.15.5:6789/0 conn(0x7f21dc09f560 legacy=0x7f21dc060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.817+0000 7f21d9d9b700 1 -- 172.21.15.187:0/278913185 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc0a4cc0 con 0x7f21dc0bfbe0 2020-11-20T12:20:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.817+0000 7f21d9d9b700 1 --2- 172.21.15.187:0/278913185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc0bfbe0 0x7f21dc1480c0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f21c4006fd0 tx=0x7f21c4006eb0).ready entity=mon.1 client_cookie=3be8d7d0f7bf162a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.817+0000 7f21d959a700 1 -- 172.21.15.187:0/278913185 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4009920 con 0x7f21dc0bfbe0 2020-11-20T12:20:19.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.817+0000 7f21d959a700 1 -- 172.21.15.187:0/278913185 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c4009a80 con 0x7f21dc0bfbe0 2020-11-20T12:20:19.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.817+0000 7f21d959a700 1 -- 172.21.15.187:0/278913185 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4009e00 con 0x7f21dc0bfbe0 2020-11-20T12:20:19.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.818+0000 7f21db59e700 1 -- 172.21.15.187:0/278913185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc0bfbe0 msgr2=0x7f21dc1480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.818+0000 7f21db59e700 1 --2- 172.21.15.187:0/278913185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc0bfbe0 0x7f21dc1480c0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f21c4006fd0 tx=0x7f21c4006eb0).stop 2020-11-20T12:20:19.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.818+0000 7f21db59e700 1 -- 172.21.15.187:0/278913185 shutdown_connections 2020-11-20T12:20:19.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.818+0000 7f21db59e700 1 --2- 172.21.15.187:0/278913185 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 0x7f21dc14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.818+0000 7f21db59e700 1 --2- 172.21.15.187:0/278913185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc0bfbe0 0x7f21dc1480c0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.818+0000 7f21db59e700 1 -- 172.21.15.187:0/278913185 >> 172.21.15.187:0/278913185 conn(0x7f21dc068370 msgr2=0x7f21dc0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:19.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.818+0000 7f21db59e700 1 -- 172.21.15.187:0/278913185 shutdown_connections 2020-11-20T12:20:19.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.818+0000 7f21db59e700 1 -- 172.21.15.187:0/278913185 wait complete. 2020-11-20T12:20:19.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.820+0000 7f21db59e700 1 Processor -- start 2020-11-20T12:20:19.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.820+0000 7f21db59e700 1 -- start start 2020-11-20T12:20:19.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.821+0000 7f21db59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21dc09f560 0x7f21dc0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.821+0000 7f21db59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 0x7f21dc0ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.821+0000 7f21db59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc15bfd0 0x7f21dc145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.821+0000 7f21db59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0b4380 con 0x7f21dc09f560 2020-11-20T12:20:19.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.821+0000 7f21db59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0b4610 con 0x7f21dc15bfd0 2020-11-20T12:20:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.821+0000 7f21db59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0b48b0 con 0x7f21dc0bf800 2020-11-20T12:20:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21dad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc15bfd0 0x7f21dc145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21dad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc15bfd0 0x7f21dc145330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53528/0 (socket says 172.21.15.187:53528) 2020-11-20T12:20:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21dad9d700 1 -- 172.21.15.187:0/2707467044 learned_addr learned my addr 172.21.15.187:0/2707467044 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21da59c700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21dc09f560 0x7f21dc0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21d9d9b700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 0x7f21dc0ba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21dad9d700 1 -- 172.21.15.187:0/2707467044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 msgr2=0x7f21dc0ba6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21dad9d700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 0x7f21dc0ba6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21dad9d700 1 -- 172.21.15.187:0/2707467044 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21dc09f560 msgr2=0x7f21dc0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:19.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21dad9d700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21dc09f560 0x7f21dc0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.822+0000 7f21dad9d700 1 -- 172.21.15.187:0/2707467044 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc0b4b50 con 0x7f21dc15bfd0 2020-11-20T12:20:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.823+0000 7f21dad9d700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc15bfd0 0x7f21dc145330 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f21cc00cab0 tx=0x7f21cc0023b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.823+0000 7f21d27fc700 1 -- 172.21.15.187:0/2707467044 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc00a830 con 0x7f21dc15bfd0 2020-11-20T12:20:19.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.823+0000 7f21d27fc700 1 -- 172.21.15.187:0/2707467044 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc00a990 con 0x7f21dc15bfd0 2020-11-20T12:20:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.823+0000 7f21d27fc700 1 -- 172.21.15.187:0/2707467044 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc00ad30 con 0x7f21dc15bfd0 2020-11-20T12:20:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.823+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc0babe0 con 0x7f21dc15bfd0 2020-11-20T12:20:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.823+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc1459e0 con 0x7f21dc15bfd0 2020-11-20T12:20:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.824+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21dc145c90 con 0x7f21dc15bfd0 2020-11-20T12:20:19.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.825+0000 7f21d27fc700 1 -- 172.21.15.187:0/2707467044 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f21cc019030 con 0x7f21dc15bfd0 2020-11-20T12:20:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.826+0000 7f21d27fc700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f21b8056d50 0x7f21b8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.826+0000 7f21d27fc700 1 -- 172.21.15.187:0/2707467044 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f21cc06f680 con 0x7f21dc15bfd0 2020-11-20T12:20:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.826+0000 7f21da59c700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f21b8056d50 0x7f21b8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.827+0000 7f21da59c700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f21b8056d50 0x7f21b8059210 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f21d400d990 tx=0x7f21d4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:19.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:19.830+0000 7f21d27fc700 1 -- 172.21.15.187:0/2707467044 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f21cc043fd0 con 0x7f21dc15bfd0 2020-11-20T12:20:20.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.015+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f21dc0bd970 con 0x7f21dc15bfd0 2020-11-20T12:20:20.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.533+0000 7f21d27fc700 1 -- 172.21.15.187:0/2707467044 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v148) v1 ==== 125+0+0 (secure 0 0 0) 0x7f21dc145c90 con 0x7f21dc15bfd0 2020-11-20T12:20:20.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.543+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f21b8056d50 msgr2=0x7f21b8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:20.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.543+0000 7f21db59e700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f21b8056d50 0x7f21b8059210 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f21d400d990 tx=0x7f21d4008040).stop 2020-11-20T12:20:20.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.543+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc15bfd0 msgr2=0x7f21dc145330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:20.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.543+0000 7f21db59e700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc15bfd0 0x7f21dc145330 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f21cc00cab0 tx=0x7f21cc0023b0).stop 2020-11-20T12:20:20.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.544+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 shutdown_connections 2020-11-20T12:20:20.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.544+0000 7f21db59e700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f21b8056d50 0x7f21b8059210 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:20.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.544+0000 7f21db59e700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21dc09f560 0x7f21dc0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:20.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.544+0000 7f21db59e700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21dc15bfd0 0x7f21dc145330 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:20.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.544+0000 7f21db59e700 1 --2- 172.21.15.187:0/2707467044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f21dc0bf800 0x7f21dc0ba6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:20.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.544+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 >> 172.21.15.187:0/2707467044 conn(0x7f21dc068370 msgr2=0x7f21dc0a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:20.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.545+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 shutdown_connections 2020-11-20T12:20:20.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.545+0000 7f21db59e700 1 -- 172.21.15.187:0/2707467044 wait complete. 2020-11-20T12:20:20.550 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:20:20.575 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:20:20.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.807+0000 7f1bd1ada700 1 Processor -- start 2020-11-20T12:20:20.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.807+0000 7f1bd1ada700 1 -- start start 2020-11-20T12:20:20.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.809+0000 7f1bd1ada700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc1482d0 0x7f1bcc0b6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:20.811 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.809+0000 7f1bd1ada700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc144b80 con 0x7f1bcc068230 2020-11-20T12:20:20.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.809+0000 7f1bd1ada700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc1489e0 con 0x7f1bcc0b57b0 2020-11-20T12:20:20.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.809+0000 7f1bd1ada700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc148c80 con 0x7f1bcc1482d0 2020-11-20T12:20:20.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.810+0000 7f1bd0ad8700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1bcc0b57b0 0x7f1bcc062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50140/0 (socket says 172.21.15.187:50140) 2020-11-20T12:20:20.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.810+0000 7f1bd0ad8700 1 -- 172.21.15.187:0/2971351965 learned_addr learned my addr 172.21.15.187:0/2971351965 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:20.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.810+0000 7f1bd12d9700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc1482d0 0x7f1bcc0b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:20.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.810+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955862195 0 0) 0x7f1bcc1489e0 con 0x7f1bcc0b57b0 2020-11-20T12:20:20.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.810+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb8003240 con 0x7f1bcc0b57b0 2020-11-20T12:20:20.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650110917 0 0) 0x7f1bcc144b80 con 0x7f1bcc068230 2020-11-20T12:20:20.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bcc1489e0 con 0x7f1bcc068230 2020-11-20T12:20:20.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1743929662 0 0) 0x7f1bb8003240 con 0x7f1bcc0b57b0 2020-11-20T12:20:20.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc1482d0 msgr2=0x7f1bcc0b6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:20.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc1482d0 0x7f1bcc0b6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:20.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 >> v1:172.21.15.5:6789/0 conn(0x7f1bcc068230 legacy=0x7f1bcc0b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:20.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc148f20 con 0x7f1bcc0b57b0 2020-11-20T12:20:20.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f1bbc0014e0 con 0x7f1bcc0b57b0 2020-11-20T12:20:20.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 >> v1:172.21.15.187:6789/0 conn(0x7f1bcc0b57b0 legacy=0x7f1bcc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:20.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb80038c0 0x7f1bb8003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:20.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80044e0 0x7f1bb8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:20.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc1482d0 0x7f1bb8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:20.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc148c80 con 0x7f1bb80038c0 2020-11-20T12:20:20.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8005340 con 0x7f1bb80044e0 2020-11-20T12:20:20.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.811+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb800c840 con 0x7f1bcc1482d0 2020-11-20T12:20:20.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.812+0000 7f1bcbfff700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80044e0 0x7f1bb8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:20.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.812+0000 7f1bd12d9700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc1482d0 0x7f1bb8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:20.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.812+0000 7f1bcbfff700 1 -- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc1482d0 msgr2=0x7f1bb8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:20.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.812+0000 7f1bcbfff700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc1482d0 0x7f1bb8004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:20.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.812+0000 7f1bcbfff700 1 -- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb80038c0 msgr2=0x7f1bb8003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:20.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.812+0000 7f1bcbfff700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb80038c0 0x7f1bb8003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:20.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.812+0000 7f1bcbfff700 1 -- 172.21.15.187:0/2971351965 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8003240 con 0x7f1bb80044e0 2020-11-20T12:20:20.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.812+0000 7f1bcbfff700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80044e0 0x7f1bb8004960 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f1bc0001ab0 tx=0x7f1bc0007b90).ready entity=mon.1 client_cookie=b13f56b725b178e6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:20.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0007020 con 0x7f1bb80044e0 2020-11-20T12:20:20.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc0007180 con 0x7f1bb80044e0 2020-11-20T12:20:20.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bcb7fe700 1 -- 172.21.15.187:0/2971351965 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc000c980 con 0x7f1bb80044e0 2020-11-20T12:20:20.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2971351965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80044e0 msgr2=0x7f1bb8004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:20.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bd1ada700 1 --2- 172.21.15.187:0/2971351965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80044e0 0x7f1bb8004960 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f1bc0001ab0 tx=0x7f1bc0007b90).stop 2020-11-20T12:20:20.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bd12d9700 1 -- 172.21.15.187:0/2971351965 reap_dead start 2020-11-20T12:20:20.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2971351965 shutdown_connections 2020-11-20T12:20:20.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2971351965 >> 172.21.15.187:0/2971351965 conn(0x7f1bcc0bab30 msgr2=0x7f1bcc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:20.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.813+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2971351965 shutdown_connections 2020-11-20T12:20:20.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.814+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2971351965 wait complete. 2020-11-20T12:20:20.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.815+0000 7f1bd1ada700 1 Processor -- start 2020-11-20T12:20:20.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.815+0000 7f1bd1ada700 1 -- start start 2020-11-20T12:20:20.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.816+0000 7f1bd1ada700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15bbd0 0x7f1bcc094c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:20.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.816+0000 7f1bd1ada700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bcc15bfb0 0x7f1bcc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:20.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.816+0000 7f1bd1ada700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc15c470 0x7f1bcc155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:20.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.816+0000 7f1bd1ada700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc145d40 con 0x7f1bcc15bfb0 2020-11-20T12:20:20.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.816+0000 7f1bd1ada700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc145f70 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.816+0000 7f1bd1ada700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc146210 con 0x7f1bcc15c470 2020-11-20T12:20:20.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.816+0000 7f1bd0ad8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15bbd0 0x7f1bcc094c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:20.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd0ad8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15bbd0 0x7f1bcc094c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53546/0 (socket says 172.21.15.187:53546) 2020-11-20T12:20:20.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd0ad8700 1 -- 172.21.15.187:0/2820782502 learned_addr learned my addr 172.21.15.187:0/2820782502 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:20.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd12d9700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc15c470 0x7f1bcc155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:20.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bcbfff700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bcc15bfb0 0x7f1bcc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:20.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd0ad8700 1 -- 172.21.15.187:0/2820782502 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc15c470 msgr2=0x7f1bcc155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:20.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd0ad8700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc15c470 0x7f1bcc155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:20.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd0ad8700 1 -- 172.21.15.187:0/2820782502 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bcc15bfb0 msgr2=0x7f1bcc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:20.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd0ad8700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bcc15bfb0 0x7f1bcc1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:20.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd0ad8700 1 -- 172.21.15.187:0/2820782502 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc155d80 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.817+0000 7f1bd0ad8700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15bbd0 0x7f1bcc094c30 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7f1bbc00ca90 tx=0x7f1bbc000d90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:20.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.818+0000 7f1bc8ff9700 1 -- 172.21.15.187:0/2820782502 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc007170 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.818+0000 7f1bc8ff9700 1 -- 172.21.15.187:0/2820782502 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc0072d0 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.818+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc095120 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.818+0000 7f1bc8ff9700 1 -- 172.21.15.187:0/2820782502 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0049e0 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.818+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc13b700 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.819+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bcc0666d0 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.819+0000 7f1bc8ff9700 1 -- 172.21.15.187:0/2820782502 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1bbc01b060 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.820+0000 7f1bc8ff9700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ba4056d20 0x7f1ba40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:20.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.820+0000 7f1bc8ff9700 1 -- 172.21.15.187:0/2820782502 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f1bbc06fa20 con 0x7f1bcc15bbd0 2020-11-20T12:20:20.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.821+0000 7f1bcbfff700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ba4056d20 0x7f1ba40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:20.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.821+0000 7f1bcbfff700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ba4056d20 0x7f1ba40591e0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f1bcc05f200 tx=0x7f1bc0016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:20.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:20.824+0000 7f1bc8ff9700 1 -- 172.21.15.187:0/2820782502 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1bbc044dd0 con 0x7f1bcc15bbd0 2020-11-20T12:20:21.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.069+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1bcc0b1210 con 0x7f1bcc15bbd0 2020-11-20T12:20:21.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.085+0000 7f1bc8ff9700 1 -- 172.21.15.187:0/2820782502 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 29 and data pool 30 v150) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1bbc019030 con 0x7f1bcc15bbd0 2020-11-20T12:20:21.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.092+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ba4056d20 msgr2=0x7f1ba40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.092+0000 7f1bd1ada700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ba4056d20 0x7f1ba40591e0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f1bcc05f200 tx=0x7f1bc0016000).stop 2020-11-20T12:20:21.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.092+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15bbd0 msgr2=0x7f1bcc094c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.092+0000 7f1bd1ada700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15bbd0 0x7f1bcc094c30 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7f1bbc00ca90 tx=0x7f1bbc000d90).stop 2020-11-20T12:20:21.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.093+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 shutdown_connections 2020-11-20T12:20:21.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.093+0000 7f1bd1ada700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1ba4056d20 0x7f1ba40591e0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.093+0000 7f1bd1ada700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bcc15c470 0x7f1bcc155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.093+0000 7f1bd1ada700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15bbd0 0x7f1bcc094c30 secure :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 rx=0x7f1bbc00ca90 tx=0x7f1bbc000d90).stop 2020-11-20T12:20:21.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.093+0000 7f1bd1ada700 1 --2- 172.21.15.187:0/2820782502 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bcc15bfb0 0x7f1bcc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.093+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 >> 172.21.15.187:0/2820782502 conn(0x7f1bcc0bab30 msgr2=0x7f1bcc09fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:21.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.097+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 shutdown_connections 2020-11-20T12:20:21.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.098+0000 7f1bd1ada700 1 -- 172.21.15.187:0/2820782502 wait complete. 2020-11-20T12:20:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 29 and data pool 30 2020-11-20T12:20:21.125 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:20:21.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.350+0000 7fcae1027700 1 Processor -- start 2020-11-20T12:20:21.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.350+0000 7fcae1027700 1 -- start start 2020-11-20T12:20:21.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.352+0000 7fcae1027700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcadc1484d0 0x7fcadc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.352+0000 7fcae1027700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 0x7fcadc0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.352+0000 7fcae1027700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc0b1300 con 0x7fcadc066f70 2020-11-20T12:20:21.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.352+0000 7fcae1027700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcadc067730 con 0x7fcadc1484d0 2020-11-20T12:20:21.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.352+0000 7fcae1027700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0b15a0 con 0x7fcadc0622c0 2020-11-20T12:20:21.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.352+0000 7fcae0826700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcadc066f70 0x7fcadc0b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50168/0 (socket says 172.21.15.187:50168) 2020-11-20T12:20:21.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.352+0000 7fcae0826700 1 -- 172.21.15.187:0/3182373007 learned_addr learned my addr 172.21.15.187:0/3182373007 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:21.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadbfff700 1 --2- 172.21.15.187:0/3182373007 >> v2:172.21.15.5:3300/0 conn(0x7fcadc1484d0 0x7fcadc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275749300 0 0) 0x7fcadc0b1300 con 0x7fcadc066f70 2020-11-20T12:20:21.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac8003240 con 0x7fcadc066f70 2020-11-20T12:20:21.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadb7fe700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 0x7fcadc0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1259546821 0 0) 0x7fcac8003240 con 0x7fcadc066f70 2020-11-20T12:20:21.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 msgr2=0x7fcadc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:21.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 0x7fcadc0b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 >> v2:172.21.15.5:3300/0 conn(0x7fcadc1484d0 msgr2=0x7fcadc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:21.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 --2- 172.21.15.187:0/3182373007 >> v2:172.21.15.5:3300/0 conn(0x7fcadc1484d0 0x7fcadc061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc0b1840 con 0x7fcadc066f70 2020-11-20T12:20:21.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fcad4001860 con 0x7fcadc066f70 2020-11-20T12:20:21.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 >> v1:172.21.15.187:6789/0 conn(0x7fcadc066f70 legacy=0x7fcadc0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 0x7fcac80037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac8003d10 0x7fcac80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcac80049d0 0x7fcac8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc067730 con 0x7fcac80049d0 2020-11-20T12:20:21.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0b15a0 con 0x7fcac8003d10 2020-11-20T12:20:21.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.353+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80042d0 con 0x7fcadc0622c0 2020-11-20T12:20:21.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcadb7fe700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac8003d10 0x7fcac80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcadbfff700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 0x7fcac80037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcae0826700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcac80049d0 0x7fcac8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcadb7fe700 1 -- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 msgr2=0x7fcac80037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:21.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcadb7fe700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 0x7fcac80037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcadb7fe700 1 -- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcac80049d0 msgr2=0x7fcac8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:21.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcadb7fe700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcac80049d0 0x7fcac8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcadb7fe700 1 -- 172.21.15.187:0/3182373007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8003240 con 0x7fcac8003d10 2020-11-20T12:20:21.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.354+0000 7fcadbfff700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc0622c0 0x7fcac80037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:21.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.355+0000 7fcadb7fe700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac8003d10 0x7fcac80044e0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7fcad000ba50 tx=0x7fcad0008040).ready entity=mon.1 client_cookie=a458850ecdaf9775 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:21.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.355+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad000d9c0 con 0x7fcac8003d10 2020-11-20T12:20:21.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.355+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad000db20 con 0x7fcac8003d10 2020-11-20T12:20:21.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.355+0000 7fcadaffd700 1 -- 172.21.15.187:0/3182373007 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad000dea0 con 0x7fcac8003d10 2020-11-20T12:20:21.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.355+0000 7fcae1027700 1 -- 172.21.15.187:0/3182373007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac8003d10 msgr2=0x7fcac80044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.355+0000 7fcae1027700 1 --2- 172.21.15.187:0/3182373007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac8003d10 0x7fcac80044e0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7fcad000ba50 tx=0x7fcad0008040).stop 2020-11-20T12:20:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.355+0000 7fcae0826700 1 -- 172.21.15.187:0/3182373007 reap_dead start 2020-11-20T12:20:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.355+0000 7fcae1027700 1 -- 172.21.15.187:0/3182373007 shutdown_connections 2020-11-20T12:20:21.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.356+0000 7fcae1027700 1 -- 172.21.15.187:0/3182373007 >> 172.21.15.187:0/3182373007 conn(0x7fcadc0bab30 msgr2=0x7fcadc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:21.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.356+0000 7fcae1027700 1 -- 172.21.15.187:0/3182373007 shutdown_connections 2020-11-20T12:20:21.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.356+0000 7fcae1027700 1 -- 172.21.15.187:0/3182373007 wait complete. 2020-11-20T12:20:21.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.357+0000 7fcae1027700 1 Processor -- start 2020-11-20T12:20:21.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.357+0000 7fcae1027700 1 -- start start 2020-11-20T12:20:21.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.358+0000 7fcae1027700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcadc09efa0 0x7fcadc09f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.358+0000 7fcae1027700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcadc0b38f0 0x7fcadc0a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.358+0000 7fcae1027700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc13ee20 0x7fcadc13f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.358+0000 7fcae1027700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0a05f0 con 0x7fcadc0b38f0 2020-11-20T12:20:21.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.358+0000 7fcae1027700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0b3d70 con 0x7fcadc09efa0 2020-11-20T12:20:21.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.358+0000 7fcae1027700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc13f790 con 0x7fcadc13ee20 2020-11-20T12:20:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.358+0000 7fcadbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcadc09efa0 0x7fcadc09f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcadc09efa0 0x7fcadc09f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53574/0 (socket says 172.21.15.187:53574) 2020-11-20T12:20:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadbfff700 1 -- 172.21.15.187:0/2120324834 learned_addr learned my addr 172.21.15.187:0/2120324834 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:21.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadb7fe700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcadc0b38f0 0x7fcadc0a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcae0826700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc13ee20 0x7fcadc13f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadbfff700 1 -- 172.21.15.187:0/2120324834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc13ee20 msgr2=0x7fcadc13f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadbfff700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc13ee20 0x7fcadc13f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadbfff700 1 -- 172.21.15.187:0/2120324834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcadc0b38f0 msgr2=0x7fcadc0a0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadbfff700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcadc0b38f0 0x7fcadc0a0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadbfff700 1 -- 172.21.15.187:0/2120324834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc09fa00 con 0x7fcadc09efa0 2020-11-20T12:20:21.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.359+0000 7fcadbfff700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcadc09efa0 0x7fcadc09f380 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7fcacc00f860 tx=0x7fcacc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:21.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.360+0000 7fcabbfff700 1 -- 172.21.15.187:0/2120324834 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc017070 con 0x7fcadc09efa0 2020-11-20T12:20:21.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.360+0000 7fcabbfff700 1 -- 172.21.15.187:0/2120324834 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcacc00d740 con 0x7fcadc09efa0 2020-11-20T12:20:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.360+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcadc09fca0 con 0x7fcadc09efa0 2020-11-20T12:20:21.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.360+0000 7fcabbfff700 1 -- 172.21.15.187:0/2120324834 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc00dac0 con 0x7fcadc09efa0 2020-11-20T12:20:21.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.360+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcadc09b720 con 0x7fcadc09efa0 2020-11-20T12:20:21.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.361+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcadc09f870 con 0x7fcadc09efa0 2020-11-20T12:20:21.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.361+0000 7fcabbfff700 1 -- 172.21.15.187:0/2120324834 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcacc01c410 con 0x7fcadc09efa0 2020-11-20T12:20:21.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.362+0000 7fcabbfff700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcab4056d20 0x7fcab40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.362+0000 7fcabbfff700 1 -- 172.21.15.187:0/2120324834 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcacc070030 con 0x7fcadc09efa0 2020-11-20T12:20:21.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.362+0000 7fcadb7fe700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcab4056d20 0x7fcab40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.363+0000 7fcadb7fe700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcab4056d20 0x7fcab40591e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fcadc05ec00 tx=0x7fcad0008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:21.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.366+0000 7fcabbfff700 1 -- 172.21.15.187:0/2120324834 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcacc045420 con 0x7fcadc09efa0 2020-11-20T12:20:21.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.537+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcadc0b8940 con 0x7fcadc09efa0 2020-11-20T12:20:21.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.540+0000 7fcabbfff700 1 -- 172.21.15.187:0/2120324834 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16805 (secure 0 0 0) 0x7fcacc048a40 con 0x7fcadc09efa0 2020-11-20T12:20:21.543 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:21.543 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":149,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:20:21.071415+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:20:19.501212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2020-11-20T12:20:20.096442+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:20:21.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcab4056d20 msgr2=0x7fcab40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcab4056d20 0x7fcab40591e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fcadc05ec00 tx=0x7fcad0008040).stop 2020-11-20T12:20:21.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcadc09efa0 msgr2=0x7fcadc09f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcadc09efa0 0x7fcadc09f380 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7fcacc00f860 tx=0x7fcacc008040).stop 2020-11-20T12:20:21.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 shutdown_connections 2020-11-20T12:20:21.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcab4056d20 0x7fcab40591e0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcadc13ee20 0x7fcadc13f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcadc09efa0 0x7fcadc09f380 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 --2- 172.21.15.187:0/2120324834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcadc0b38f0 0x7fcadc0a0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 >> 172.21.15.187:0/2120324834 conn(0x7fcadc0bab30 msgr2=0x7fcadc0afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:21.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 shutdown_connections 2020-11-20T12:20:21.547 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.542+0000 7fcae1027700 1 -- 172.21.15.187:0/2120324834 wait complete. 2020-11-20T12:20:21.566 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:20:21.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.788+0000 7f0fc7fff700 1 Processor -- start 2020-11-20T12:20:21.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.788+0000 7f0fc7fff700 1 -- start start 2020-11-20T12:20:21.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.790+0000 7f0fc7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc8061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.790+0000 7f0fc7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0fc8066f70 0x7f0fc80b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.790+0000 7f0fc7fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc80b1300 con 0x7f0fc80622c0 2020-11-20T12:20:21.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.790+0000 7f0fc7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc8067730 con 0x7f0fc8066f70 2020-11-20T12:20:21.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.790+0000 7f0fc7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc80b15a0 con 0x7f0fc81484d0 2020-11-20T12:20:21.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.790+0000 7f0fc6ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc8061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.791+0000 7f0fc77fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0fc8066f70 0x7f0fc80b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.791+0000 7f0fc6ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc8061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:54470/0 (socket says 172.21.15.187:54470) 2020-11-20T12:20:21.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.791+0000 7f0fbe7fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0fc80622c0 0x7f0fc80b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:43696/0 (socket says 172.21.15.187:43696) 2020-11-20T12:20:21.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.791+0000 7f0fc6ffd700 1 -- 172.21.15.187:0/2126754380 learned_addr learned my addr 172.21.15.187:0/2126754380 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:21.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.792+0000 7f0fc67fc700 1 -- 172.21.15.187:0/2126754380 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074531199 0 0) 0x7f0fc80b1300 con 0x7f0fc80622c0 2020-11-20T12:20:21.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.792+0000 7f0fc67fc700 1 -- 172.21.15.187:0/2126754380 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa8003240 con 0x7f0fc80622c0 2020-11-20T12:20:21.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.792+0000 7f0fc6ffd700 1 -- 172.21.15.187:0/2126754380 >> v2:172.21.15.5:3300/0 conn(0x7f0fc8066f70 msgr2=0x7f0fc80b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:21.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.792+0000 7f0fc6ffd700 1 --2- 172.21.15.187:0/2126754380 >> v2:172.21.15.5:3300/0 conn(0x7f0fc8066f70 0x7f0fc80b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.792+0000 7f0fc6ffd700 1 -- 172.21.15.187:0/2126754380 >> v1:172.21.15.5:6789/0 conn(0x7f0fc80622c0 legacy=0x7f0fc80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.792+0000 7f0fc6ffd700 1 -- 172.21.15.187:0/2126754380 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc80b1840 con 0x7f0fc81484d0 2020-11-20T12:20:21.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.792+0000 7f0fc77fe700 1 --2- 172.21.15.187:0/2126754380 >> v2:172.21.15.5:3300/0 conn(0x7f0fc8066f70 0x7f0fc80b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:20:21.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.792+0000 7f0fc6ffd700 1 --2- 172.21.15.187:0/2126754380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc8061dd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f0fb8006fd0 tx=0x7f0fb8006eb0).ready entity=mon.2 client_cookie=73a578ad1a9727e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:21.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.793+0000 7f0fc67fc700 1 -- 172.21.15.187:0/2126754380 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8009720 con 0x7f0fc81484d0 2020-11-20T12:20:21.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.793+0000 7f0fc67fc700 1 -- 172.21.15.187:0/2126754380 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb8009880 con 0x7f0fc81484d0 2020-11-20T12:20:21.800 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.793+0000 7f0fc67fc700 1 -- 172.21.15.187:0/2126754380 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8009c00 con 0x7f0fc81484d0 2020-11-20T12:20:21.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.794+0000 7f0fc7fff700 1 -- 172.21.15.187:0/2126754380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 msgr2=0x7f0fc8061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.794+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/2126754380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc8061dd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f0fb8006fd0 tx=0x7f0fb8006eb0).stop 2020-11-20T12:20:21.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.794+0000 7f0fc7fff700 1 -- 172.21.15.187:0/2126754380 shutdown_connections 2020-11-20T12:20:21.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.794+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/2126754380 >> v2:172.21.15.5:3300/0 conn(0x7f0fc8066f70 0x7f0fc80b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.801 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.794+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/2126754380 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc8061dd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.794+0000 7f0fc7fff700 1 -- 172.21.15.187:0/2126754380 >> 172.21.15.187:0/2126754380 conn(0x7f0fc80bab30 msgr2=0x7f0fc80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:21.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.794+0000 7f0fc7fff700 1 -- 172.21.15.187:0/2126754380 shutdown_connections 2020-11-20T12:20:21.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.794+0000 7f0fc7fff700 1 -- 172.21.15.187:0/2126754380 wait complete. 2020-11-20T12:20:21.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.796+0000 7f0fc7fff700 1 Processor -- start 2020-11-20T12:20:21.802 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.796+0000 7f0fc7fff700 1 -- start start 2020-11-20T12:20:21.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fc7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fc80622c0 0x7f0fc80a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fc7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fc8066f70 0x7f0fc8135480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fc7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc81348c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fc7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc8158c60 con 0x7f0fc80622c0 2020-11-20T12:20:21.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fc7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc813e120 con 0x7f0fc8066f70 2020-11-20T12:20:21.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fc7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc813e5c0 con 0x7f0fc81484d0 2020-11-20T12:20:21.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fbe7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fc8066f70 0x7f0fc8135480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.804 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fbe7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fc8066f70 0x7f0fc8135480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53588/0 (socket says 172.21.15.187:53588) 2020-11-20T12:20:21.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.797+0000 7f0fbe7fc700 1 -- 172.21.15.187:0/39486830 learned_addr learned my addr 172.21.15.187:0/39486830 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:21.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fc6ffd700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fc80622c0 0x7f0fc80a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbe7fc700 1 -- 172.21.15.187:0/39486830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 msgr2=0x7f0fc81348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbe7fc700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc81348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:21.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbe7fc700 1 -- 172.21.15.187:0/39486830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fc80622c0 msgr2=0x7f0fc80a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbe7fc700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fc80622c0 0x7f0fc80a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbe7fc700 1 -- 172.21.15.187:0/39486830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc81588c0 con 0x7f0fc8066f70 2020-11-20T12:20:21.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbe7fc700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fc8066f70 0x7f0fc8135480 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7f0fb000c710 tx=0x7f0fb0007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:21.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbf7fe700 1 -- 172.21.15.187:0/39486830 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00072e0 con 0x7f0fc8066f70 2020-11-20T12:20:21.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbf7fe700 1 -- 172.21.15.187:0/39486830 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb0005530 con 0x7f0fc8066f70 2020-11-20T12:20:21.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc809f130 con 0x7f0fc8066f70 2020-11-20T12:20:21.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fbf7fe700 1 -- 172.21.15.187:0/39486830 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00058d0 con 0x7f0fc8066f70 2020-11-20T12:20:21.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.798+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc8135b30 con 0x7f0fc8066f70 2020-11-20T12:20:21.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.799+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc8158e30 con 0x7f0fc8066f70 2020-11-20T12:20:21.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.800+0000 7f0fbf7fe700 1 -- 172.21.15.187:0/39486830 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0fb0010070 con 0x7f0fc8066f70 2020-11-20T12:20:21.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.801+0000 7f0fbf7fe700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fa4056d20 0x7f0fa40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:21.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.801+0000 7f0fbf7fe700 1 -- 172.21.15.187:0/39486830 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0fb006f700 con 0x7f0fc8066f70 2020-11-20T12:20:21.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.801+0000 7f0fc6ffd700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fa4056d20 0x7f0fa40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:21.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.802+0000 7f0fc6ffd700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fa4056d20 0x7f0fa40591e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f0fb8009420 tx=0x7f0fb8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:21.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.804+0000 7f0fbf7fe700 1 -- 172.21.15.187:0/39486830 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0fb0044a40 con 0x7f0fc8066f70 2020-11-20T12:20:21.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.973+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fc813e2b0 con 0x7f0fc8066f70 2020-11-20T12:20:21.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.975+0000 7f0fbf7fe700 1 -- 172.21.15.187:0/39486830 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16805 (secure 0 0 0) 0x7f0fb0048060 con 0x7f0fc8066f70 2020-11-20T12:20:21.978 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:21.978 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":149,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:20:21.071415+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:20:19.501212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2020-11-20T12:20:20.096442+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:20:21.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fa4056d20 msgr2=0x7f0fa40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fa4056d20 0x7f0fa40591e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f0fb8009420 tx=0x7f0fb8015000).stop 2020-11-20T12:20:21.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fc8066f70 msgr2=0x7f0fc8135480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:21.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fc8066f70 0x7f0fc8135480 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7f0fb000c710 tx=0x7f0fb0007a60).stop 2020-11-20T12:20:21.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 shutdown_connections 2020-11-20T12:20:21.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0fa4056d20 0x7f0fa40591e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fc80622c0 0x7f0fc80a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fc81484d0 0x7f0fc81348c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:21.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 --2- 172.21.15.187:0/39486830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fc8066f70 0x7f0fc8135480 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:21.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.977+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 >> 172.21.15.187:0/39486830 conn(0x7f0fc80bab30 msgr2=0x7f0fc80b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:21.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.978+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 shutdown_connections 2020-11-20T12:20:21.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:21.978+0000 7f0fc7fff700 1 -- 172.21.15.187:0/39486830 wait complete. 2020-11-20T12:20:22.005 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:20:22.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.247+0000 7f2d1a91c700 1 Processor -- start 2020-11-20T12:20:22.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.247+0000 7f2d1a91c700 1 -- start start 2020-11-20T12:20:22.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.249+0000 7f2d1a91c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d140b8750 0x7f2d14147420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:22.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.249+0000 7f2d1a91c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2d140b8b30 0x7f2d14066510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:22.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.249+0000 7f2d1a91c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d14065b30 con 0x7f2d1405bda0 2020-11-20T12:20:22.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.249+0000 7f2d1a91c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d14065dd0 con 0x7f2d140b8b30 2020-11-20T12:20:22.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.249+0000 7f2d1a91c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d14066070 con 0x7f2d140b8750 2020-11-20T12:20:22.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.249+0000 7f2d1991a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d140b8750 0x7f2d14147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:22.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.250+0000 7f2d1991a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d140b8750 0x7f2d14147420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53594/0 (socket says 172.21.15.187:53594) 2020-11-20T12:20:22.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.250+0000 7f2d1991a700 1 -- 172.21.15.187:0/2402973860 learned_addr learned my addr 172.21.15.187:0/2402973860 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:22.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.250+0000 7f2d19119700 1 --2- 172.21.15.187:0/2402973860 >> v2:172.21.15.5:3300/0 conn(0x7f2d140b8b30 0x7f2d14066510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:22.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.250+0000 7f2d1991a700 1 -- 172.21.15.187:0/2402973860 >> v2:172.21.15.5:3300/0 conn(0x7f2d140b8b30 msgr2=0x7f2d14066510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:22.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.250+0000 7f2d1991a700 1 --2- 172.21.15.187:0/2402973860 >> v2:172.21.15.5:3300/0 conn(0x7f2d140b8b30 0x7f2d14066510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:22.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.250+0000 7f2d1991a700 1 -- 172.21.15.187:0/2402973860 >> v1:172.21.15.5:6789/0 conn(0x7f2d1405bda0 legacy=0x7f2d14140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:22.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.250+0000 7f2d1991a700 1 -- 172.21.15.187:0/2402973860 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14147910 con 0x7f2d140b8750 2020-11-20T12:20:22.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.250+0000 7f2d1991a700 1 --2- 172.21.15.187:0/2402973860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d140b8750 0x7f2d14147420 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7f2d140688f0 tx=0x7f2d10006eb0).ready entity=mon.1 client_cookie=fa0f273789768f2d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:22.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.251+0000 7f2d18918700 1 -- 172.21.15.187:0/2402973860 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d10009890 con 0x7f2d140b8750 2020-11-20T12:20:22.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.251+0000 7f2d18918700 1 -- 172.21.15.187:0/2402973860 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d100099f0 con 0x7f2d140b8750 2020-11-20T12:20:22.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.251+0000 7f2d18918700 1 -- 172.21.15.187:0/2402973860 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d10009d70 con 0x7f2d140b8750 2020-11-20T12:20:22.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.251+0000 7f2d1a91c700 1 -- 172.21.15.187:0/2402973860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d140b8750 msgr2=0x7f2d14147420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:22.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.251+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/2402973860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d140b8750 0x7f2d14147420 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7f2d140688f0 tx=0x7f2d10006eb0).stop 2020-11-20T12:20:22.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.252+0000 7f2d1a91c700 1 -- 172.21.15.187:0/2402973860 shutdown_connections 2020-11-20T12:20:22.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.252+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/2402973860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d140b8750 0x7f2d14147420 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:22.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.252+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/2402973860 >> v2:172.21.15.5:3300/0 conn(0x7f2d140b8b30 0x7f2d14066510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:22.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.252+0000 7f2d1a91c700 1 -- 172.21.15.187:0/2402973860 >> 172.21.15.187:0/2402973860 conn(0x7f2d140bbeb0 msgr2=0x7f2d140b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:22.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.252+0000 7f2d1a91c700 1 -- 172.21.15.187:0/2402973860 shutdown_connections 2020-11-20T12:20:22.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.252+0000 7f2d1a91c700 1 -- 172.21.15.187:0/2402973860 wait complete. 2020-11-20T12:20:22.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.253+0000 7f2d1a91c700 1 Processor -- start 2020-11-20T12:20:22.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.253+0000 7f2d1a91c700 1 -- start start 2020-11-20T12:20:22.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.254+0000 7f2d1a91c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d1405bda0 0x7f2d1413fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:22.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.254+0000 7f2d1a91c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d140b8750 0x7f2d1415ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:22.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.254+0000 7f2d1a91c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1409f4d0 0x7f2d1415bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:22.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.254+0000 7f2d1a91c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14143cd0 con 0x7f2d140b8750 2020-11-20T12:20:22.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.254+0000 7f2d1a91c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d141435d0 con 0x7f2d1409f4d0 2020-11-20T12:20:22.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.254+0000 7f2d1a91c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14144230 con 0x7f2d1405bda0 2020-11-20T12:20:22.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1409f4d0 0x7f2d1415bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:22.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1409f4d0 0x7f2d1415bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53604/0 (socket says 172.21.15.187:53604) 2020-11-20T12:20:22.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 -- 172.21.15.187:0/430209751 learned_addr learned my addr 172.21.15.187:0/430209751 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:22.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 -- 172.21.15.187:0/430209751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d1405bda0 msgr2=0x7f2d1413fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:22.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d1405bda0 0x7f2d1413fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:22.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 -- 172.21.15.187:0/430209751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d140b8750 msgr2=0x7f2d1415ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:22.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d140b8750 0x7f2d1415ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:22.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 -- 172.21.15.187:0/430209751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14143830 con 0x7f2d1409f4d0 2020-11-20T12:20:22.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d1a11b700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1409f4d0 0x7f2d1415bef0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7f2d0c00cae0 tx=0x7f2d0c0023b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:22.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.255+0000 7f2d05ffb700 1 -- 172.21.15.187:0/430209751 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c00a7f0 con 0x7f2d1409f4d0 2020-11-20T12:20:22.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.256+0000 7f2d05ffb700 1 -- 172.21.15.187:0/430209751 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0c00a950 con 0x7f2d1409f4d0 2020-11-20T12:20:22.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.256+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1415c900 con 0x7f2d1409f4d0 2020-11-20T12:20:22.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.256+0000 7f2d05ffb700 1 -- 172.21.15.187:0/430209751 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c00acf0 con 0x7f2d1409f4d0 2020-11-20T12:20:22.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.256+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1415c3e0 con 0x7f2d1409f4d0 2020-11-20T12:20:22.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.257+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d141439f0 con 0x7f2d1409f4d0 2020-11-20T12:20:22.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.257+0000 7f2d05ffb700 1 -- 172.21.15.187:0/430209751 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2d0c019030 con 0x7f2d1409f4d0 2020-11-20T12:20:22.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.258+0000 7f2d05ffb700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d20 0x7f2cec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:22.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.258+0000 7f2d05ffb700 1 -- 172.21.15.187:0/430209751 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2d0c06f840 con 0x7f2d1409f4d0 2020-11-20T12:20:22.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.259+0000 7f2d1991a700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d20 0x7f2cec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:22.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.259+0000 7f2d1991a700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d20 0x7f2cec0591e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f2d10006ef0 tx=0x7f2d10015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:22.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.262+0000 7f2d05ffb700 1 -- 172.21.15.187:0/430209751 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2d0c043fd0 con 0x7f2d1409f4d0 2020-11-20T12:20:22.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:22.491+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2d14066a00 con 0x7f2d1409f4d0 2020-11-20T12:20:23.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.296+0000 7f2d05ffb700 1 -- 172.21.15.187:0/430209751 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v153) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2d0c044d40 con 0x7f2d1409f4d0 2020-11-20T12:20:23.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.299+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d20 msgr2=0x7f2cec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:23.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.299+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d20 0x7f2cec0591e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f2d10006ef0 tx=0x7f2d10015000).stop 2020-11-20T12:20:23.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.299+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1409f4d0 msgr2=0x7f2d1415bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:23.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.299+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1409f4d0 0x7f2d1415bef0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7f2d0c00cae0 tx=0x7f2d0c0023b0).stop 2020-11-20T12:20:23.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.300+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 shutdown_connections 2020-11-20T12:20:23.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.300+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2cec056d20 0x7f2cec0591e0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.300+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1409f4d0 0x7f2d1415bef0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.300+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d1405bda0 0x7f2d1413fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:23.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.300+0000 7f2d1a91c700 1 --2- 172.21.15.187:0/430209751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d140b8750 0x7f2d1415ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:23.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.300+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 >> 172.21.15.187:0/430209751 conn(0x7f2d140bbeb0 msgr2=0x7f2d141471f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:23.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.300+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 shutdown_connections 2020-11-20T12:20:23.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.300+0000 7f2d1a91c700 1 -- 172.21.15.187:0/430209751 wait complete. 2020-11-20T12:20:23.331 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:23.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.570+0000 7fe08c63c700 1 Processor -- start 2020-11-20T12:20:23.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.571+0000 7fe08c63c700 1 -- start start 2020-11-20T12:20:23.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.572+0000 7fe08c63c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe08414c750 0x7fe084062580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:23.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.572+0000 7fe08c63c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840666e0 0x7fe084067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:23.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.572+0000 7fe08c63c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe084062a70 0x7fe084068220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:23.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.572+0000 7fe08c63c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe084061e80 con 0x7fe084062a70 2020-11-20T12:20:23.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.572+0000 7fe08c63c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0840620e0 con 0x7fe08414c750 2020-11-20T12:20:23.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.572+0000 7fe08c63c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840b1500 con 0x7fe0840666e0 2020-11-20T12:20:23.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.572+0000 7fe08a3d8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe08414c750 0x7fe084062580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:23.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.573+0000 7fe08a3d8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe08414c750 0x7fe084062580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53608/0 (socket says 172.21.15.187:53608) 2020-11-20T12:20:23.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.573+0000 7fe08a3d8700 1 -- 172.21.15.187:0/2295893694 learned_addr learned my addr 172.21.15.187:0/2295893694 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:23.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.573+0000 7fe089bd7700 1 --2- 172.21.15.187:0/2295893694 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840666e0 0x7fe084067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:23.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.573+0000 7fe08abd9700 1 --2- 172.21.15.187:0/2295893694 >> v2:172.21.15.5:3300/0 conn(0x7fe084062a70 0x7fe084068220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:23.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.573+0000 7fe08a3d8700 1 -- 172.21.15.187:0/2295893694 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840666e0 msgr2=0x7fe084067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:23.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.573+0000 7fe08a3d8700 1 --2- 172.21.15.187:0/2295893694 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840666e0 0x7fe084067d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.573+0000 7fe08a3d8700 1 -- 172.21.15.187:0/2295893694 >> v2:172.21.15.5:3300/0 conn(0x7fe084062a70 msgr2=0x7fe084068220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:23.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.573+0000 7fe08a3d8700 1 --2- 172.21.15.187:0/2295893694 >> v2:172.21.15.5:3300/0 conn(0x7fe084062a70 0x7fe084068220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.574+0000 7fe08a3d8700 1 -- 172.21.15.187:0/2295893694 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0840b17d0 con 0x7fe08414c750 2020-11-20T12:20:23.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.574+0000 7fe08abd9700 1 --2- 172.21.15.187:0/2295893694 >> v2:172.21.15.5:3300/0 conn(0x7fe084062a70 0x7fe084068220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:23.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.574+0000 7fe08a3d8700 1 --2- 172.21.15.187:0/2295893694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08414c750 0x7fe084062580 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fe0840b8ab0 tx=0x7fe080006eb0).ready entity=mon.1 client_cookie=c13f560845584d06 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:23.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.574+0000 7fe0893d6700 1 -- 172.21.15.187:0/2295893694 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080009960 con 0x7fe08414c750 2020-11-20T12:20:23.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.574+0000 7fe0893d6700 1 -- 172.21.15.187:0/2295893694 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe080009ac0 con 0x7fe08414c750 2020-11-20T12:20:23.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.574+0000 7fe0893d6700 1 -- 172.21.15.187:0/2295893694 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080009e40 con 0x7fe08414c750 2020-11-20T12:20:23.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 -- 172.21.15.187:0/2295893694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08414c750 msgr2=0x7fe084062580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:23.583 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2295893694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08414c750 0x7fe084062580 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fe0840b8ab0 tx=0x7fe080006eb0).stop 2020-11-20T12:20:23.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 -- 172.21.15.187:0/2295893694 shutdown_connections 2020-11-20T12:20:23.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2295893694 >> v2:172.21.15.5:3300/0 conn(0x7fe084062a70 0x7fe084068220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2295893694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08414c750 0x7fe084062580 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2295893694 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840666e0 0x7fe084067d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 -- 172.21.15.187:0/2295893694 >> 172.21.15.187:0/2295893694 conn(0x7fe0840bacd0 msgr2=0x7fe0840bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:23.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 -- 172.21.15.187:0/2295893694 shutdown_connections 2020-11-20T12:20:23.585 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.575+0000 7fe08c63c700 1 -- 172.21.15.187:0/2295893694 wait complete. 2020-11-20T12:20:23.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.577+0000 7fe08c63c700 1 Processor -- start 2020-11-20T12:20:23.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.577+0000 7fe08c63c700 1 -- start start 2020-11-20T12:20:23.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.578+0000 7fe08c63c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe084062a70 0x7fe08414dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:23.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.578+0000 7fe08c63c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0840666e0 0x7fe0840b9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:23.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.578+0000 7fe08c63c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840a01d0 0x7fe084149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:23.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08c63c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08409dfa0 con 0x7fe0840666e0 2020-11-20T12:20:23.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08c63c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08414e220 con 0x7fe084062a70 2020-11-20T12:20:23.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08c63c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08414e4c0 con 0x7fe0840a01d0 2020-11-20T12:20:23.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08a3d8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe084062a70 0x7fe08414dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:23.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08a3d8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe084062a70 0x7fe08414dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53618/0 (socket says 172.21.15.187:53618) 2020-11-20T12:20:23.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08a3d8700 1 -- 172.21.15.187:0/2391435780 learned_addr learned my addr 172.21.15.187:0/2391435780 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:23.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe089bd7700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0840666e0 0x7fe0840b9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:23.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08abd9700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840a01d0 0x7fe084149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:23.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08a3d8700 1 -- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840a01d0 msgr2=0x7fe084149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:23.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08a3d8700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840a01d0 0x7fe084149690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08a3d8700 1 -- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0840666e0 msgr2=0x7fe0840b9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:23.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.579+0000 7fe08a3d8700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0840666e0 0x7fe0840b9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.580+0000 7fe08a3d8700 1 -- 172.21.15.187:0/2391435780 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08414e760 con 0x7fe084062a70 2020-11-20T12:20:23.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.580+0000 7fe08abd9700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840a01d0 0x7fe084149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:23.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.580+0000 7fe08a3d8700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe084062a70 0x7fe08414dc30 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7fe080010000 tx=0x7fe080015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:23.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.580+0000 7fe0727fc700 1 -- 172.21.15.187:0/2391435780 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08001e460 con 0x7fe084062a70 2020-11-20T12:20:23.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.580+0000 7fe0727fc700 1 -- 172.21.15.187:0/2391435780 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08001e5c0 con 0x7fe084062a70 2020-11-20T12:20:23.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.580+0000 7fe0727fc700 1 -- 172.21.15.187:0/2391435780 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08001e940 con 0x7fe084062a70 2020-11-20T12:20:23.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.580+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0840ba270 con 0x7fe084062a70 2020-11-20T12:20:23.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.580+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe084149b80 con 0x7fe084062a70 2020-11-20T12:20:23.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.581+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe084063600 con 0x7fe084062a70 2020-11-20T12:20:23.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.582+0000 7fe0727fc700 1 -- 172.21.15.187:0/2391435780 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe080025030 con 0x7fe084062a70 2020-11-20T12:20:23.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.582+0000 7fe0727fc700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c056d20 0x7fe05c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:23.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.583+0000 7fe0727fc700 1 -- 172.21.15.187:0/2391435780 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe080071880 con 0x7fe084062a70 2020-11-20T12:20:23.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.583+0000 7fe089bd7700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c056d20 0x7fe05c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:23.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.584+0000 7fe089bd7700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c056d20 0x7fe05c0591e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7fe08414d490 tx=0x7fe074008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:23.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.586+0000 7fe0727fc700 1 -- 172.21.15.187:0/2391435780 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe080046c70 con 0x7fe084062a70 2020-11-20T12:20:23.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.833+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe08414cb30 con 0x7fe084062a70 2020-11-20T12:20:23.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.834+0000 7fe0727fc700 1 -- 172.21.15.187:0/2391435780 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2142 (secure 0 0 0) 0x7fe080023070 con 0x7fe084062a70 2020-11-20T12:20:23.836 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:23.837 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":153,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1287724648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1287724648},{"type":"v1","addr":"172.21.15.5:6833","nonce":1287724648}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:23.278959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:23.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c056d20 msgr2=0x7fe05c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:23.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c056d20 0x7fe05c0591e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7fe08414d490 tx=0x7fe074008040).stop 2020-11-20T12:20:23.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe084062a70 msgr2=0x7fe08414dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:23.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe084062a70 0x7fe08414dc30 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7fe080010000 tx=0x7fe080015000).stop 2020-11-20T12:20:23.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 shutdown_connections 2020-11-20T12:20:23.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe05c056d20 0x7fe05c0591e0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe0840a01d0 0x7fe084149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe084062a70 0x7fe08414dc30 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 --2- 172.21.15.187:0/2391435780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0840666e0 0x7fe0840b9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:23.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 >> 172.21.15.187:0/2391435780 conn(0x7fe0840bacd0 msgr2=0x7fe0840b92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:23.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.836+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 shutdown_connections 2020-11-20T12:20:23.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:23.837+0000 7fe08c63c700 1 -- 172.21.15.187:0/2391435780 wait complete. 2020-11-20T12:20:23.842 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 153 2020-11-20T12:20:23.862 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:20:23.973 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:20:23.982 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:20:24.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.098+0000 7f4d2c81e700 1 Processor -- start 2020-11-20T12:20:24.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.098+0000 7f4d2c81e700 1 -- start start 2020-11-20T12:20:24.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.100+0000 7f4d2c81e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d28065f90 0x7f4d28065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.100+0000 7f4d2c81e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d2809ca90 0x7f4d2809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.100+0000 7f4d2c81e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d280b13e0 con 0x7f4d28065190 2020-11-20T12:20:24.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.100+0000 7f4d2c81e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d2814ab30 con 0x7f4d28065f90 2020-11-20T12:20:24.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.100+0000 7f4d2c81e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28062c10 con 0x7f4d2809ca90 2020-11-20T12:20:24.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.101+0000 7f4d27fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d2809ca90 0x7f4d2809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.101+0000 7f4d26ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d28065f90 0x7f4d28065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.101+0000 7f4d277fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4d28065190 0x7f4d28143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:43738/0 (socket says 172.21.15.187:43738) 2020-11-20T12:20:24.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.101+0000 7f4d277fe700 1 -- 172.21.15.187:0/3935377621 learned_addr learned my addr 172.21.15.187:0/3935377621 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:24.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.101+0000 7f4d27fff700 1 -- 172.21.15.187:0/3935377621 >> v2:172.21.15.5:3300/0 conn(0x7f4d28065f90 msgr2=0x7f4d28065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:24.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.101+0000 7f4d27fff700 1 --2- 172.21.15.187:0/3935377621 >> v2:172.21.15.5:3300/0 conn(0x7f4d28065f90 0x7f4d28065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.101+0000 7f4d27fff700 1 -- 172.21.15.187:0/3935377621 >> v1:172.21.15.5:6789/0 conn(0x7f4d28065190 legacy=0x7f4d28143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.101+0000 7f4d27fff700 1 -- 172.21.15.187:0/3935377621 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d280b1640 con 0x7f4d2809ca90 2020-11-20T12:20:24.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.102+0000 7f4d27fff700 1 --2- 172.21.15.187:0/3935377621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d2809ca90 0x7f4d2809cf40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4d20009c90 tx=0x7f4d200091d0).ready entity=mon.2 client_cookie=198763b529290ece server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:24.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.102+0000 7f4d267fc700 1 -- 172.21.15.187:0/3935377621 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2000bb00 con 0x7f4d2809ca90 2020-11-20T12:20:24.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.102+0000 7f4d267fc700 1 -- 172.21.15.187:0/3935377621 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2000bc60 con 0x7f4d2809ca90 2020-11-20T12:20:24.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.102+0000 7f4d267fc700 1 -- 172.21.15.187:0/3935377621 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d20011400 con 0x7f4d2809ca90 2020-11-20T12:20:24.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.103+0000 7f4d2c81e700 1 -- 172.21.15.187:0/3935377621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d2809ca90 msgr2=0x7f4d2809cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.103+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/3935377621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d2809ca90 0x7f4d2809cf40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4d20009c90 tx=0x7f4d200091d0).stop 2020-11-20T12:20:24.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.103+0000 7f4d2c81e700 1 -- 172.21.15.187:0/3935377621 shutdown_connections 2020-11-20T12:20:24.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.103+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/3935377621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d2809ca90 0x7f4d2809cf40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.103+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/3935377621 >> v2:172.21.15.5:3300/0 conn(0x7f4d28065f90 0x7f4d28065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.103+0000 7f4d2c81e700 1 -- 172.21.15.187:0/3935377621 >> 172.21.15.187:0/3935377621 conn(0x7f4d280bab30 msgr2=0x7f4d280baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:24.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.103+0000 7f4d2c81e700 1 -- 172.21.15.187:0/3935377621 shutdown_connections 2020-11-20T12:20:24.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.103+0000 7f4d2c81e700 1 -- 172.21.15.187:0/3935377621 wait complete. 2020-11-20T12:20:24.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.105+0000 7f4d2c81e700 1 Processor -- start 2020-11-20T12:20:24.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.105+0000 7f4d2c81e700 1 -- start start 2020-11-20T12:20:24.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d2c81e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d28065190 0x7f4d280991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d2c81e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d28065f90 0x7f4d28149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d2c81e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d2809ca90 0x7f4d281452d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d2c81e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28141bd0 con 0x7f4d2809ca90 2020-11-20T12:20:24.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d2c81e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28141e70 con 0x7f4d28065f90 2020-11-20T12:20:24.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d2c81e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2814a350 con 0x7f4d28065190 2020-11-20T12:20:24.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d26ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d28065f90 0x7f4d28149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d26ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d28065f90 0x7f4d28149e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53630/0 (socket says 172.21.15.187:53630) 2020-11-20T12:20:24.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d26ffd700 1 -- 172.21.15.187:0/2258523167 learned_addr learned my addr 172.21.15.187:0/2258523167 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:24.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d277fe700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d28065190 0x7f4d280991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.106+0000 7f4d27fff700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d2809ca90 0x7f4d281452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d26ffd700 1 -- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d28065190 msgr2=0x7f4d280991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d26ffd700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d28065190 0x7f4d280991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d26ffd700 1 -- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d2809ca90 msgr2=0x7f4d281452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d26ffd700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d2809ca90 0x7f4d281452d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d26ffd700 1 -- 172.21.15.187:0/2258523167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d281417e0 con 0x7f4d28065f90 2020-11-20T12:20:24.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d277fe700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d28065190 0x7f4d280991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:24.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d26ffd700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d28065f90 0x7f4d28149e60 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f4d1c00eba0 tx=0x7f4d1c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:24.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d077fe700 1 -- 172.21.15.187:0/2258523167 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00c9b0 con 0x7f4d28065f90 2020-11-20T12:20:24.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d077fe700 1 -- 172.21.15.187:0/2258523167 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c00cb10 con 0x7f4d28065f90 2020-11-20T12:20:24.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d077fe700 1 -- 172.21.15.187:0/2258523167 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00ce90 con 0x7f4d28065f90 2020-11-20T12:20:24.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d28145da0 con 0x7f4d28065f90 2020-11-20T12:20:24.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.107+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d28145a00 con 0x7f4d28065f90 2020-11-20T12:20:24.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.109+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d28141a30 con 0x7f4d28065f90 2020-11-20T12:20:24.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.109+0000 7f4d077fe700 1 -- 172.21.15.187:0/2258523167 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4d1c01a030 con 0x7f4d28065f90 2020-11-20T12:20:24.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.110+0000 7f4d077fe700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d00056d20 0x7f4d000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.110+0000 7f4d077fe700 1 -- 172.21.15.187:0/2258523167 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4d1c06fbc0 con 0x7f4d28065f90 2020-11-20T12:20:24.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.110+0000 7f4d277fe700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d00056d20 0x7f4d000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.111+0000 7f4d277fe700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d00056d20 0x7f4d000591e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f4d18001550 tx=0x7f4d18009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:24.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.114+0000 7f4d077fe700 1 -- 172.21.15.187:0/2258523167 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4d28141a30 con 0x7f4d28065f90 2020-11-20T12:20:24.216 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:20:24.214+0000 7f4976072700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:20:24.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.301+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d28141a30 con 0x7f4d28065f90 2020-11-20T12:20:24.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.308+0000 7f4d077fe700 1 -- 172.21.15.187:0/2258523167 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f4d1c0261b0 con 0x7f4d28065f90 2020-11-20T12:20:24.311 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:24.312 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":150,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:20:22.077535+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:20:19.501212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2020-11-20T12:20:20.096442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:20:24.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d00056d20 msgr2=0x7f4d000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d00056d20 0x7f4d000591e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f4d18001550 tx=0x7f4d18009040).stop 2020-11-20T12:20:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d28065f90 msgr2=0x7f4d28149e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d28065f90 0x7f4d28149e60 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f4d1c00eba0 tx=0x7f4d1c006df0).stop 2020-11-20T12:20:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 shutdown_connections 2020-11-20T12:20:24.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4d00056d20 0x7f4d000591e0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d2809ca90 0x7f4d281452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d28065190 0x7f4d280991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 --2- 172.21.15.187:0/2258523167 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d28065f90 0x7f4d28149e60 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 >> 172.21.15.187:0/2258523167 conn(0x7f4d280bab30 msgr2=0x7f4d281422f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:24.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 shutdown_connections 2020-11-20T12:20:24.316 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.311+0000 7f4d2c81e700 1 -- 172.21.15.187:0/2258523167 wait complete. 2020-11-20T12:20:24.338 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:20:24.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.634+0000 7fb8cf59e700 1 Processor -- start 2020-11-20T12:20:24.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.634+0000 7fb8cf59e700 1 -- start start 2020-11-20T12:20:24.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.635+0000 7fb8cf59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb8d01484d0 0x7fb8d0061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.635+0000 7fb8cf59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8d00622c0 0x7fb8d00b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.635+0000 7fb8cf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 0x7fb8d00b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8cf59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d0067730 con 0x7fb8d00622c0 2020-11-20T12:20:24.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8cf59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d00b1300 con 0x7fb8d01484d0 2020-11-20T12:20:24.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8cf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d00b15a0 con 0x7fb8d0066f70 2020-11-20T12:20:24.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8ce59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb8d01484d0 0x7fb8d0061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8ce59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb8d01484d0 0x7fb8d0061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53636/0 (socket says 172.21.15.187:53636) 2020-11-20T12:20:24.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8ce59c700 1 -- 172.21.15.187:0/74024066 learned_addr learned my addr 172.21.15.187:0/74024066 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:24.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8ce59c700 1 -- 172.21.15.187:0/74024066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 msgr2=0x7fb8d00b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:24.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8ce59c700 1 --2- 172.21.15.187:0/74024066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 0x7fb8d00b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:24.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8ce59c700 1 -- 172.21.15.187:0/74024066 >> v2:172.21.15.5:3300/0 conn(0x7fb8d00622c0 msgr2=0x7fb8d00b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:24.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8ce59c700 1 --2- 172.21.15.187:0/74024066 >> v2:172.21.15.5:3300/0 conn(0x7fb8d00622c0 0x7fb8d00b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:24.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.636+0000 7fb8ce59c700 1 -- 172.21.15.187:0/74024066 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d00b1840 con 0x7fb8d01484d0 2020-11-20T12:20:24.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.637+0000 7fb8ce59c700 1 --2- 172.21.15.187:0/74024066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d01484d0 0x7fb8d0061dd0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 rx=0x7fb8d00b85e0 tx=0x7fb8c0006eb0).ready entity=mon.1 client_cookie=e0f717d778a3a20a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:24.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.637+0000 7fb8cd59a700 1 -- 172.21.15.187:0/74024066 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c00098d0 con 0x7fb8d01484d0 2020-11-20T12:20:24.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.637+0000 7fb8cd59a700 1 -- 172.21.15.187:0/74024066 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c0009a30 con 0x7fb8d01484d0 2020-11-20T12:20:24.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.637+0000 7fb8cd59a700 1 -- 172.21.15.187:0/74024066 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0009db0 con 0x7fb8d01484d0 2020-11-20T12:20:24.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 -- 172.21.15.187:0/74024066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d01484d0 msgr2=0x7fb8d0061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/74024066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d01484d0 0x7fb8d0061dd0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 rx=0x7fb8d00b85e0 tx=0x7fb8c0006eb0).stop 2020-11-20T12:20:24.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 -- 172.21.15.187:0/74024066 shutdown_connections 2020-11-20T12:20:24.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/74024066 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 0x7fb8d00b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:24.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/74024066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d01484d0 0x7fb8d0061dd0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/74024066 >> v2:172.21.15.5:3300/0 conn(0x7fb8d00622c0 0x7fb8d00b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:24.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 -- 172.21.15.187:0/74024066 >> 172.21.15.187:0/74024066 conn(0x7fb8d00bab30 msgr2=0x7fb8d00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:24.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 -- 172.21.15.187:0/74024066 shutdown_connections 2020-11-20T12:20:24.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.638+0000 7fb8cf59e700 1 -- 172.21.15.187:0/74024066 wait complete. 2020-11-20T12:20:24.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.640+0000 7fb8cf59e700 1 Processor -- start 2020-11-20T12:20:24.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.640+0000 7fb8cf59e700 1 -- start start 2020-11-20T12:20:24.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8cf59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d00622c0 0x7fb8d013a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8cf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 0x7fb8d01593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8cf59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8d01484d0 0x7fb8d0159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8cf59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d0137860 con 0x7fb8d01484d0 2020-11-20T12:20:24.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8cf59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d015a150 con 0x7fb8d00622c0 2020-11-20T12:20:24.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8cf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d009e670 con 0x7fb8d0066f70 2020-11-20T12:20:24.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8ce59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d00622c0 0x7fb8d013a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8ce59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d00622c0 0x7fb8d013a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53642/0 (socket says 172.21.15.187:53642) 2020-11-20T12:20:24.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.641+0000 7fb8ce59c700 1 -- 172.21.15.187:0/944180934 learned_addr learned my addr 172.21.15.187:0/944180934 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:24.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8cdd9b700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 0x7fb8d01593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8ced9d700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8d01484d0 0x7fb8d0159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8ce59c700 1 -- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 msgr2=0x7fb8d01593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8ce59c700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 0x7fb8d01593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8ce59c700 1 -- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8d01484d0 msgr2=0x7fb8d0159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8ce59c700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8d01484d0 0x7fb8d0159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8ce59c700 1 -- 172.21.15.187:0/944180934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d013ad80 con 0x7fb8d00622c0 2020-11-20T12:20:24.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8cdd9b700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 0x7fb8d01593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:24.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8ced9d700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8d01484d0 0x7fb8d0159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:24.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8ce59c700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d00622c0 0x7fb8d013a890 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7fb8c0006ef0 tx=0x7fb8c0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:24.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8be7fc700 1 -- 172.21.15.187:0/944180934 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c001e460 con 0x7fb8d00622c0 2020-11-20T12:20:24.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8be7fc700 1 -- 172.21.15.187:0/944180934 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c001e5c0 con 0x7fb8d00622c0 2020-11-20T12:20:24.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d009fb00 con 0x7fb8d00622c0 2020-11-20T12:20:24.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.642+0000 7fb8be7fc700 1 -- 172.21.15.187:0/944180934 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c001e940 con 0x7fb8d00622c0 2020-11-20T12:20:24.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.643+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d009eb10 con 0x7fb8d00622c0 2020-11-20T12:20:24.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.644+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d0159f40 con 0x7fb8d00622c0 2020-11-20T12:20:24.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.644+0000 7fb8be7fc700 1 -- 172.21.15.187:0/944180934 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb8c0025030 con 0x7fb8d00622c0 2020-11-20T12:20:24.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.645+0000 7fb8be7fc700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8a4056d20 0x7fb8a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:24.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.645+0000 7fb8be7fc700 1 -- 172.21.15.187:0/944180934 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb8c00718d0 con 0x7fb8d00622c0 2020-11-20T12:20:24.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.645+0000 7fb8cdd9b700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8a4056d20 0x7fb8a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:24.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.646+0000 7fb8cdd9b700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8a4056d20 0x7fb8a40591e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7fb8c4006fd0 tx=0x7fb8c4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:24.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.649+0000 7fb8be7fc700 1 -- 172.21.15.187:0/944180934 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb8c0046dd0 con 0x7fb8d00622c0 2020-11-20T12:20:24.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.900+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fb8d0061a20 con 0x7fb8d00622c0 2020-11-20T12:20:24.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.910+0000 7fb8be7fc700 1 -- 172.21.15.187:0/944180934 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb8c0023030 con 0x7fb8d00622c0 2020-11-20T12:20:24.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8a4056d20 msgr2=0x7fb8a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8a4056d20 0x7fb8a40591e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7fb8c4006fd0 tx=0x7fb8c4008040).stop 2020-11-20T12:20:24.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d00622c0 msgr2=0x7fb8d013a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:24.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d00622c0 0x7fb8d013a890 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7fb8c0006ef0 tx=0x7fb8c0015000).stop 2020-11-20T12:20:24.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 shutdown_connections 2020-11-20T12:20:24.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb8a4056d20 0x7fb8a40591e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8d01484d0 0x7fb8d0159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8d00622c0 0x7fb8d013a890 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 --2- 172.21.15.187:0/944180934 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb8d0066f70 0x7fb8d01593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:24.923 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 >> 172.21.15.187:0/944180934 conn(0x7fb8d00bab30 msgr2=0x7fb8d00b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:24.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.913+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 shutdown_connections 2020-11-20T12:20:24.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:24.914+0000 7fb8cf59e700 1 -- 172.21.15.187:0/944180934 wait complete. 2020-11-20T12:20:24.931 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:20:24.941 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:25.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.192+0000 7f206b59e700 1 Processor -- start 2020-11-20T12:20:25.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.192+0000 7f206b59e700 1 -- start start 2020-11-20T12:20:25.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.193+0000 7f206b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c066f70 0x7f206c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:25.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.193+0000 7f206b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c0b1300 con 0x7f206c0b6650 2020-11-20T12:20:25.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.193+0000 7f206b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c0b15a0 con 0x7f206c1484d0 2020-11-20T12:20:25.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.193+0000 7f206b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206c067730 con 0x7f206c066f70 2020-11-20T12:20:25.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.194+0000 7f206a59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f206c1484d0 0x7f206c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50256/0 (socket says 172.21.15.187:50256) 2020-11-20T12:20:25.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.194+0000 7f206a59c700 1 -- 172.21.15.187:0/4265579846 learned_addr learned my addr 172.21.15.187:0/4265579846 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:25.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.194+0000 7f206ad9d700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c066f70 0x7f206c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:25.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417654900 0 0) 0x7f206c0b15a0 con 0x7f206c1484d0 2020-11-20T12:20:25.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2050003240 con 0x7f206c1484d0 2020-11-20T12:20:25.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2424590649 0 0) 0x7f2050003240 con 0x7f206c1484d0 2020-11-20T12:20:25.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c066f70 msgr2=0x7f206c0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:25.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c066f70 0x7f206c0b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:25.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 >> v1:172.21.15.5:6789/0 conn(0x7f206c0b6650 legacy=0x7f206c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:25.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c0b1840 con 0x7f206c1484d0 2020-11-20T12:20:25.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f205c001520 con 0x7f206c1484d0 2020-11-20T12:20:25.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 >> v1:172.21.15.187:6789/0 conn(0x7f206c1484d0 legacy=0x7f206c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:25.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2050003730 0x7f2050003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:25.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c066f70 0x7f20500044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:25.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20500049d0 0x7f2050004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:25.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c067730 con 0x7f2050003730 2020-11-20T12:20:25.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500041d0 con 0x7f20500049d0 2020-11-20T12:20:25.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.195+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205000d940 con 0x7f206c066f70 2020-11-20T12:20:25.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.196+0000 7f206ad9d700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20500049d0 0x7f2050004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:25.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.196+0000 7f206ad9d700 1 -- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c066f70 msgr2=0x7f20500044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:25.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.196+0000 7f206ad9d700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c066f70 0x7f20500044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:25.208 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.196+0000 7f206ad9d700 1 -- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2050003730 msgr2=0x7f2050003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:25.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.196+0000 7f206ad9d700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2050003730 0x7f2050003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:25.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.196+0000 7f206ad9d700 1 -- 172.21.15.187:0/4265579846 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2050003240 con 0x7f20500049d0 2020-11-20T12:20:25.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.196+0000 7f206ad9d700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20500049d0 0x7f2050004e50 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f2064009e70 tx=0x7f20640091d0).ready entity=mon.1 client_cookie=2bbe8ed8282e829b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:25.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20640095c0 con 0x7f20500049d0 2020-11-20T12:20:25.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206400ed10 con 0x7f20500049d0 2020-11-20T12:20:25.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206959a700 1 -- 172.21.15.187:0/4265579846 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2064012600 con 0x7f20500049d0 2020-11-20T12:20:25.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206b59e700 1 -- 172.21.15.187:0/4265579846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20500049d0 msgr2=0x7f2050004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:25.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206b59e700 1 --2- 172.21.15.187:0/4265579846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20500049d0 0x7f2050004e50 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f2064009e70 tx=0x7f20640091d0).stop 2020-11-20T12:20:25.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206ad9d700 1 -- 172.21.15.187:0/4265579846 reap_dead start 2020-11-20T12:20:25.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206b59e700 1 -- 172.21.15.187:0/4265579846 shutdown_connections 2020-11-20T12:20:25.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206b59e700 1 -- 172.21.15.187:0/4265579846 >> 172.21.15.187:0/4265579846 conn(0x7f206c0bab30 msgr2=0x7f206c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:25.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206b59e700 1 -- 172.21.15.187:0/4265579846 shutdown_connections 2020-11-20T12:20:25.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.197+0000 7f206b59e700 1 -- 172.21.15.187:0/4265579846 wait complete. 2020-11-20T12:20:25.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.199+0000 7f206b59e700 1 Processor -- start 2020-11-20T12:20:25.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.200+0000 7f206b59e700 1 -- start start 2020-11-20T12:20:25.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f206c142120 0x7f206c0b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:25.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f206c0b2900 0x7f206c157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:25.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c09d100 0x7f206c157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:25.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0b2260 con 0x7f206c0b2900 2020-11-20T12:20:25.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0b1fc0 con 0x7f206c142120 2020-11-20T12:20:25.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206c144200 con 0x7f206c09d100 2020-11-20T12:20:25.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f206c142120 0x7f206c0b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:25.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f206c142120 0x7f206c0b2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53666/0 (socket says 172.21.15.187:53666) 2020-11-20T12:20:25.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.201+0000 7f206a59c700 1 -- 172.21.15.187:0/1757084769 learned_addr learned my addr 172.21.15.187:0/1757084769 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:25.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f206a59c700 1 -- 172.21.15.187:0/1757084769 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c09d100 msgr2=0x7f206c157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:25.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f206a59c700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c09d100 0x7f206c157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:25.218 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f206a59c700 1 -- 172.21.15.187:0/1757084769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f206c0b2900 msgr2=0x7f206c157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:25.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f2069d9b700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f206c0b2900 0x7f206c157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:25.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f206a59c700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f206c0b2900 0x7f206c157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:25.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f206a59c700 1 -- 172.21.15.187:0/1757084769 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c0b1c40 con 0x7f206c142120 2020-11-20T12:20:25.219 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f206a59c700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f206c142120 0x7f206c0b2410 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f205c00ca70 tx=0x7f205c001b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:25.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f205a7fc700 1 -- 172.21.15.187:0/1757084769 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c007170 con 0x7f206c142120 2020-11-20T12:20:25.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f205a7fc700 1 -- 172.21.15.187:0/1757084769 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c0072d0 con 0x7f206c142120 2020-11-20T12:20:25.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c1446a0 con 0x7f206c142120 2020-11-20T12:20:25.220 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f205a7fc700 1 -- 172.21.15.187:0/1757084769 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00aa80 con 0x7f206c142120 2020-11-20T12:20:25.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.202+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c142500 con 0x7f206c142120 2020-11-20T12:20:25.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.204+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206c0b1e30 con 0x7f206c142120 2020-11-20T12:20:25.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.204+0000 7f205a7fc700 1 -- 172.21.15.187:0/1757084769 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f205c01b060 con 0x7f206c142120 2020-11-20T12:20:25.221 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.205+0000 7f205a7fc700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2040056d20 0x7f20400591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:25.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.205+0000 7f205a7fc700 1 -- 172.21.15.187:0/1757084769 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f205c06fc80 con 0x7f206c142120 2020-11-20T12:20:25.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.205+0000 7f2069d9b700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2040056d20 0x7f20400591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:25.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.206+0000 7f2069d9b700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2040056d20 0x7f20400591e0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f206c09cc20 tx=0x7f2060009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:25.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.208+0000 7f205a7fc700 1 -- 172.21.15.187:0/1757084769 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f205c044fc0 con 0x7f206c142120 2020-11-20T12:20:25.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.457+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f206c0b7180 con 0x7f206c142120 2020-11-20T12:20:25.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.458+0000 7f205a7fc700 1 -- 172.21.15.187:0/1757084769 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2142 (secure 0 0 0) 0x7f205c019030 con 0x7f206c142120 2020-11-20T12:20:25.461 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:25.461 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":153,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1287724648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1287724648},{"type":"v1","addr":"172.21.15.5:6833","nonce":1287724648}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:23.278959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:25.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2040056d20 msgr2=0x7f20400591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:25.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2040056d20 0x7f20400591e0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f206c09cc20 tx=0x7f2060009040).stop 2020-11-20T12:20:25.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f206c142120 msgr2=0x7f206c0b2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:25.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f206c142120 0x7f206c0b2410 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f205c00ca70 tx=0x7f205c001b00).stop 2020-11-20T12:20:25.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 shutdown_connections 2020-11-20T12:20:25.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2040056d20 0x7f20400591e0 secure :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0x7f206c09cc20 tx=0x7f2060009040).stop 2020-11-20T12:20:25.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f206c09d100 0x7f206c157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:25.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f206c142120 0x7f206c0b2410 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:25.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 --2- 172.21.15.187:0/1757084769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f206c0b2900 0x7f206c157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:25.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 >> 172.21.15.187:0/1757084769 conn(0x7f206c0bab30 msgr2=0x7f206c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:25.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 shutdown_connections 2020-11-20T12:20:25.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:25.460+0000 7f206b59e700 1 -- 172.21.15.187:0/1757084769 wait complete. 2020-11-20T12:20:25.466 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 153 2020-11-20T12:20:25.688 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 153, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:20:21.071872+0000', 'modified': '2020-11-20T12:20:23.278959+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9661}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9661': {'gid': 9661, 'name': 'a', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3618925546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3618925546}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3618925546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:20:25.688 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:20:25.689 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:20:25.972 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":9661,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":4.8662750829999997,"mdsmap_epoch":153,"osdmap_epoch":150,"osdmap_epoch_barrier":150,"uptime":16.519587711} 2020-11-20T12:20:25.973 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 9661, "mdsmap_epoch": 153, "osdmap_epoch": 150, "osdmap_epoch_barrier": 150, "rank_uptime": 4.866275083, "state": "up:active", "uptime": 16.519587711, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:20:25.974 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:20:25.975 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:20:25.976 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:20:25.977 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:20:25.978 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:20:25.978 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:20:25.979 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:20:25.980 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:20:25.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:20:25.982 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:20:26.043 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:20:26.043 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:20:26.044 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:20:26.044 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:20:26.047 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:20:26.048 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:20:26.048 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:20:26.049 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:20:26.049 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:20:26.050 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:20:26.050 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:20:26.051 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:20:26.051 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:20:26.052 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:20:26.053 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:20:26.054 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 41812sec preferred_lft 41812sec 2020-11-20T12:20:26.054 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:20:26.055 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:20:26.055 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:20:26.056 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:20:26.056 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:20:26.057 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:20:26.058 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:20:26.058 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:20:26.059 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:20:26.059 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:20:26.060 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:20:26.060 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:20:26.061 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:20:26.062 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:20:26.063 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:20:26.063 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:20:26.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:20:26.066 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:20:26.168 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:20:26.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:20:26.173 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:20:26.236 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:20:26.237 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:20:26.237 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:20:26.238 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:20:26.238 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:20:26.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:20:26.241 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:20:26.242 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:20:26.245 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:20:26.246 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:20:26.247 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:20:26.248 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:20:26.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:20:26.515 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:20:26.566 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:20:26.566 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:20:26.567 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:20:26.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:20:26.568 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:20:26.666 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:20:26.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.900+0000 7f4fdffff700 1 Processor -- start 2020-11-20T12:20:26.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.901+0000 7f4fdffff700 1 -- start start 2020-11-20T12:20:26.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.902+0000 7f4fdffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4fe0065190 0x7f4fe0065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:26.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.902+0000 7f4fdffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4fe009ca90 0x7f4fe009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:26.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.902+0000 7f4fdffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe00b13e0 con 0x7f4fe0065f90 2020-11-20T12:20:26.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.902+0000 7f4fdffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe014ab30 con 0x7f4fe009ca90 2020-11-20T12:20:26.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.902+0000 7f4fdffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe0062c10 con 0x7f4fe0065190 2020-11-20T12:20:26.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.902+0000 7f4fdeffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4fe0065190 0x7f4fe0065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:26.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.902+0000 7f4fd67fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4fe0065f90 0x7f4fe0143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50276/0 (socket says 172.21.15.187:50276) 2020-11-20T12:20:26.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.902+0000 7f4fd67fc700 1 -- 172.21.15.187:0/3481281671 learned_addr learned my addr 172.21.15.187:0/3481281671 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:26.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.903+0000 7f4fdf7fe700 1 --2- 172.21.15.187:0/3481281671 >> v2:172.21.15.5:3300/0 conn(0x7f4fe009ca90 0x7f4fe009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:26.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.903+0000 7f4fde7fc700 1 -- 172.21.15.187:0/3481281671 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441428335 0 0) 0x7f4fe00b13e0 con 0x7f4fe0065f90 2020-11-20T12:20:26.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.903+0000 7f4fde7fc700 1 -- 172.21.15.187:0/3481281671 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc0003240 con 0x7f4fe0065f90 2020-11-20T12:20:26.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.903+0000 7f4fdeffd700 1 -- 172.21.15.187:0/3481281671 >> v2:172.21.15.5:3300/0 conn(0x7f4fe009ca90 msgr2=0x7f4fe009cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:26.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.903+0000 7f4fdeffd700 1 --2- 172.21.15.187:0/3481281671 >> v2:172.21.15.5:3300/0 conn(0x7f4fe009ca90 0x7f4fe009cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:26.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.903+0000 7f4fdeffd700 1 -- 172.21.15.187:0/3481281671 >> v1:172.21.15.187:6789/0 conn(0x7f4fe0065f90 legacy=0x7f4fe0143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.903+0000 7f4fdeffd700 1 -- 172.21.15.187:0/3481281671 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe00b1640 con 0x7f4fe0065190 2020-11-20T12:20:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.904+0000 7f4fdeffd700 1 --2- 172.21.15.187:0/3481281671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe0065190 0x7f4fe0065570 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7f4fe005c8b0 tx=0x7f4fd0006df0).ready entity=mon.1 client_cookie=22858f97d316af00 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.904+0000 7f4fde7fc700 1 -- 172.21.15.187:0/3481281671 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0009840 con 0x7f4fe0065190 2020-11-20T12:20:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.904+0000 7f4fde7fc700 1 -- 172.21.15.187:0/3481281671 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd00099a0 con 0x7f4fe0065190 2020-11-20T12:20:26.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.904+0000 7f4fde7fc700 1 -- 172.21.15.187:0/3481281671 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0009d20 con 0x7f4fe0065190 2020-11-20T12:20:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.905+0000 7f4fdffff700 1 -- 172.21.15.187:0/3481281671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe0065190 msgr2=0x7f4fe0065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.905+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3481281671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe0065190 0x7f4fe0065570 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7f4fe005c8b0 tx=0x7f4fd0006df0).stop 2020-11-20T12:20:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.905+0000 7f4fdffff700 1 -- 172.21.15.187:0/3481281671 shutdown_connections 2020-11-20T12:20:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.905+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3481281671 >> v2:172.21.15.5:3300/0 conn(0x7f4fe009ca90 0x7f4fe009cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:26.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.905+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3481281671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe0065190 0x7f4fe0065570 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:26.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.905+0000 7f4fdffff700 1 -- 172.21.15.187:0/3481281671 >> 172.21.15.187:0/3481281671 conn(0x7f4fe00bab30 msgr2=0x7f4fe00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:26.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.905+0000 7f4fdffff700 1 -- 172.21.15.187:0/3481281671 shutdown_connections 2020-11-20T12:20:26.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.905+0000 7f4fdffff700 1 -- 172.21.15.187:0/3481281671 wait complete. 2020-11-20T12:20:26.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.907+0000 7f4fdffff700 1 Processor -- start 2020-11-20T12:20:26.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.907+0000 7f4fdffff700 1 -- start start 2020-11-20T12:20:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.908+0000 7f4fdffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fe0065190 0x7f4fe00b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.908+0000 7f4fdffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fe0065f90 0x7f4fe00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.908+0000 7f4fdffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe009ca90 0x7f4fe015b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:26.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.908+0000 7f4fdffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe00b4a00 con 0x7f4fe0065f90 2020-11-20T12:20:26.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.908+0000 7f4fdffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe01587f0 con 0x7f4fe009ca90 2020-11-20T12:20:26.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.908+0000 7f4fdffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe0158a90 con 0x7f4fe0065190 2020-11-20T12:20:26.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdf7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe009ca90 0x7f4fe015b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:26.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdf7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe009ca90 0x7f4fe015b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53680/0 (socket says 172.21.15.187:53680) 2020-11-20T12:20:26.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdf7fe700 1 -- 172.21.15.187:0/3533034469 learned_addr learned my addr 172.21.15.187:0/3533034469 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:26.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdeffd700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fe0065190 0x7f4fe00b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:26.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fd67fc700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fe0065f90 0x7f4fe00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:26.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdf7fe700 1 -- 172.21.15.187:0/3533034469 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fe0065190 msgr2=0x7f4fe00b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:26.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdf7fe700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fe0065190 0x7f4fe00b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:26.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdf7fe700 1 -- 172.21.15.187:0/3533034469 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fe0065f90 msgr2=0x7f4fe00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:26.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdf7fe700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fe0065f90 0x7f4fe00b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:26.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.909+0000 7f4fdf7fe700 1 -- 172.21.15.187:0/3533034469 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0158d30 con 0x7f4fe009ca90 2020-11-20T12:20:26.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.910+0000 7f4fdf7fe700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe009ca90 0x7f4fe015b920 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 rx=0x7f4fd800db40 tx=0x7f4fd8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:26.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.910+0000 7f4fd77fe700 1 -- 172.21.15.187:0/3533034469 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8009980 con 0x7f4fe009ca90 2020-11-20T12:20:26.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.910+0000 7f4fd77fe700 1 -- 172.21.15.187:0/3533034469 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd8009ae0 con 0x7f4fe009ca90 2020-11-20T12:20:26.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.910+0000 7f4fd77fe700 1 -- 172.21.15.187:0/3533034469 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8009e80 con 0x7f4fe009ca90 2020-11-20T12:20:26.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.910+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe01591d0 con 0x7f4fe009ca90 2020-11-20T12:20:26.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.910+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe0158350 con 0x7f4fe009ca90 2020-11-20T12:20:26.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.911+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe0158630 con 0x7f4fe009ca90 2020-11-20T12:20:26.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.912+0000 7f4fd77fe700 1 -- 172.21.15.187:0/3533034469 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4fd8013690 con 0x7f4fe009ca90 2020-11-20T12:20:26.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.912+0000 7f4fd77fe700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fbc056d20 0x7f4fbc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:26.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.912+0000 7f4fd77fe700 1 -- 172.21.15.187:0/3533034469 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4fd8070710 con 0x7f4fe009ca90 2020-11-20T12:20:26.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.913+0000 7f4fdeffd700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fbc056d20 0x7f4fbc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:26.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.913+0000 7f4fdeffd700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fbc056d20 0x7f4fbc0591e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f4fd0006e30 tx=0x7f4fd0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:26.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:26.917+0000 7f4fd77fe700 1 -- 172.21.15.187:0/3533034469 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4fd8045a50 con 0x7f4fe009ca90 2020-11-20T12:20:27.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.103+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fe0061a20 con 0x7f4fe009ca90 2020-11-20T12:20:27.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.105+0000 7f4fd77fe700 1 -- 172.21.15.187:0/3533034469 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f4fd8049070 con 0x7f4fe009ca90 2020-11-20T12:20:27.108 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:27.109 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":150,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:20:22.077535+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:20:19.501212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2020-11-20T12:20:20.096442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:20:27.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fbc056d20 msgr2=0x7f4fbc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fbc056d20 0x7f4fbc0591e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f4fd0006e30 tx=0x7f4fd0015000).stop 2020-11-20T12:20:27.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe009ca90 msgr2=0x7f4fe015b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe009ca90 0x7f4fe015b920 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 rx=0x7f4fd800db40 tx=0x7f4fd8008040).stop 2020-11-20T12:20:27.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 shutdown_connections 2020-11-20T12:20:27.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4fbc056d20 0x7f4fbc0591e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe009ca90 0x7f4fe015b920 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fe0065190 0x7f4fe00b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.107+0000 7f4fdffff700 1 --2- 172.21.15.187:0/3533034469 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fe0065f90 0x7f4fe00b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.108+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 >> 172.21.15.187:0/3533034469 conn(0x7f4fe00bab30 msgr2=0x7f4fe00b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:27.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.108+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 shutdown_connections 2020-11-20T12:20:27.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.108+0000 7f4fdffff700 1 -- 172.21.15.187:0/3533034469 wait complete. 2020-11-20T12:20:27.133 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:20:19.501212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:20:20.096442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:20:27.134 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:20:27.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.377+0000 7f854122a700 1 Processor -- start 2020-11-20T12:20:27.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.377+0000 7f854122a700 1 -- start start 2020-11-20T12:20:27.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.379+0000 7f854122a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c065190 0x7f853c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.379+0000 7f854122a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c0b13e0 con 0x7f853c09ca90 2020-11-20T12:20:27.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.379+0000 7f854122a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c062c10 con 0x7f853c065f90 2020-11-20T12:20:27.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.379+0000 7f854122a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c14ab30 con 0x7f853c065190 2020-11-20T12:20:27.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.379+0000 7f85337fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f853c065f90 0x7f853c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50292/0 (socket says 172.21.15.187:50292) 2020-11-20T12:20:27.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.379+0000 7f85337fe700 1 -- 172.21.15.187:0/1711686243 learned_addr learned my addr 172.21.15.187:0/1711686243 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:27.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853bfff700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c065190 0x7f853c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140591114 0 0) 0x7f853c062c10 con 0x7f853c065f90 2020-11-20T12:20:27.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851c003240 con 0x7f853c065f90 2020-11-20T12:20:27.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3614147865 0 0) 0x7f851c003240 con 0x7f853c065f90 2020-11-20T12:20:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c065190 msgr2=0x7f853c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c065190 0x7f853c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 >> v1:172.21.15.5:6789/0 conn(0x7f853c09ca90 legacy=0x7f853c09cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0b1640 con 0x7f853c065f90 2020-11-20T12:20:27.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f85240014e0 con 0x7f853c065f90 2020-11-20T12:20:27.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 >> v1:172.21.15.187:6789/0 conn(0x7f853c065f90 legacy=0x7f853c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f851c003730 0x7f851c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.380+0000 7f853b7fe700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c065190 0x7f851c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853b7fe700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f851c0049d0 0x7f851c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c14ab30 con 0x7f851c0049d0 2020-11-20T12:20:27.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c0041d0 con 0x7f851c003730 2020-11-20T12:20:27.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851c00fb40 con 0x7f853c065190 2020-11-20T12:20:27.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853bfff700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f851c003730 0x7f851c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853bfff700 1 -- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c065190 msgr2=0x7f851c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:27.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853bfff700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c065190 0x7f851c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:27.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853bfff700 1 -- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f851c0049d0 msgr2=0x7f851c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:27.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f8540a29700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f851c0049d0 0x7f851c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853bfff700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f851c0049d0 0x7f851c004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.381+0000 7f853bfff700 1 -- 172.21.15.187:0/1711686243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c003240 con 0x7f851c003730 2020-11-20T12:20:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.382+0000 7f853bfff700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f851c003730 0x7f851c003b50 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7f852c00bc30 tx=0x7f852c006df0).ready entity=mon.1 client_cookie=2b2490282e5a6a4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.382+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00b310 con 0x7f851c003730 2020-11-20T12:20:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.382+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c00eb40 con 0x7f851c003730 2020-11-20T12:20:27.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.382+0000 7f853b7fe700 1 -- 172.21.15.187:0/1711686243 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00eec0 con 0x7f851c003730 2020-11-20T12:20:27.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.382+0000 7f854122a700 1 -- 172.21.15.187:0/1711686243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f851c003730 msgr2=0x7f851c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.382+0000 7f854122a700 1 --2- 172.21.15.187:0/1711686243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f851c003730 0x7f851c003b50 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7f852c00bc30 tx=0x7f852c006df0).stop 2020-11-20T12:20:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.382+0000 7f8540a29700 1 -- 172.21.15.187:0/1711686243 reap_dead start 2020-11-20T12:20:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.382+0000 7f854122a700 1 -- 172.21.15.187:0/1711686243 shutdown_connections 2020-11-20T12:20:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.383+0000 7f854122a700 1 -- 172.21.15.187:0/1711686243 >> 172.21.15.187:0/1711686243 conn(0x7f853c0bab30 msgr2=0x7f853c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:27.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.383+0000 7f854122a700 1 -- 172.21.15.187:0/1711686243 shutdown_connections 2020-11-20T12:20:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.383+0000 7f854122a700 1 -- 172.21.15.187:0/1711686243 wait complete. 2020-11-20T12:20:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.384+0000 7f854122a700 1 Processor -- start 2020-11-20T12:20:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.384+0000 7f854122a700 1 -- start start 2020-11-20T12:20:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f854122a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c0b4100 0x7f853c0b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f854122a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f853c0b44e0 0x7f853c146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f854122a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853c0a0590 0x7f853c143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f854122a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c09e2d0 con 0x7f853c0a0590 2020-11-20T12:20:27.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f854122a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c09e570 con 0x7f853c0b44e0 2020-11-20T12:20:27.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f854122a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c09e810 con 0x7f853c0b4100 2020-11-20T12:20:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f85337fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f853c0b44e0 0x7f853c146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f85337fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f853c0b44e0 0x7f853c146670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53698/0 (socket says 172.21.15.187:53698) 2020-11-20T12:20:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.385+0000 7f85337fe700 1 -- 172.21.15.187:0/4020100075 learned_addr learned my addr 172.21.15.187:0/4020100075 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f85337fe700 1 -- 172.21.15.187:0/4020100075 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c0b4100 msgr2=0x7f853c0b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f853bfff700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c0b4100 0x7f853c0b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f85337fe700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c0b4100 0x7f853c0b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f85337fe700 1 -- 172.21.15.187:0/4020100075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853c0a0590 msgr2=0x7f853c143d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f85337fe700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853c0a0590 0x7f853c143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:27.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f85337fe700 1 -- 172.21.15.187:0/4020100075 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0b3c90 con 0x7f853c0b44e0 2020-11-20T12:20:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f85337fe700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f853c0b44e0 0x7f853c146670 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f852400ca40 tx=0x7f853c0bb650).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f8538ff9700 1 -- 172.21.15.187:0/4020100075 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524007170 con 0x7f853c0b44e0 2020-11-20T12:20:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f8538ff9700 1 -- 172.21.15.187:0/4020100075 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85240072d0 con 0x7f853c0b44e0 2020-11-20T12:20:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.386+0000 7f8538ff9700 1 -- 172.21.15.187:0/4020100075 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852400ab10 con 0x7f853c0b44e0 2020-11-20T12:20:27.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.387+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c144200 con 0x7f853c0b44e0 2020-11-20T12:20:27.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.387+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c09eef0 con 0x7f853c0b44e0 2020-11-20T12:20:27.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.388+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c061a20 con 0x7f853c0b44e0 2020-11-20T12:20:27.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.388+0000 7f8538ff9700 1 -- 172.21.15.187:0/4020100075 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8524010070 con 0x7f853c0b44e0 2020-11-20T12:20:27.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.389+0000 7f8538ff9700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8518056d20 0x7f85180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.389+0000 7f8538ff9700 1 -- 172.21.15.187:0/4020100075 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f852406fa50 con 0x7f853c0b44e0 2020-11-20T12:20:27.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.389+0000 7f853bfff700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8518056d20 0x7f85180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.390+0000 7f853bfff700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8518056d20 0x7f85180591e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f853c05e630 tx=0x7f852c019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:27.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.393+0000 7f8538ff9700 1 -- 172.21.15.187:0/4020100075 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8524044ea0 con 0x7f853c0b44e0 2020-11-20T12:20:27.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.578+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f853c144390 con 0x7f853c0b44e0 2020-11-20T12:20:27.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.579+0000 7f8538ff9700 1 -- 172.21.15.187:0/4020100075 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v150) v1 ==== 110+0+10 (secure 0 0 0) 0x7f853c144390 con 0x7f853c0b44e0 2020-11-20T12:20:27.582 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:20:27.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.581+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8518056d20 msgr2=0x7f85180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.581+0000 7f854122a700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8518056d20 0x7f85180591e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f853c05e630 tx=0x7f852c019000).stop 2020-11-20T12:20:27.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.581+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f853c0b44e0 msgr2=0x7f853c146670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.584 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.581+0000 7f854122a700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f853c0b44e0 0x7f853c146670 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f852400ca40 tx=0x7f853c0bb650).stop 2020-11-20T12:20:27.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.582+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 shutdown_connections 2020-11-20T12:20:27.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.582+0000 7f854122a700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8518056d20 0x7f85180591e0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.582+0000 7f854122a700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f853c0a0590 0x7f853c143d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:27.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.582+0000 7f854122a700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f853c0b4100 0x7f853c0b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.582+0000 7f854122a700 1 --2- 172.21.15.187:0/4020100075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f853c0b44e0 0x7f853c146670 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.582+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 >> 172.21.15.187:0/4020100075 conn(0x7f853c0bab30 msgr2=0x7f853c09adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:27.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.582+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 shutdown_connections 2020-11-20T12:20:27.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.582+0000 7f854122a700 1 -- 172.21.15.187:0/4020100075 wait complete. 2020-11-20T12:20:27.609 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:20:27.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.855+0000 7f56bda53700 1 Processor -- start 2020-11-20T12:20:27.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.855+0000 7f56bda53700 1 -- start start 2020-11-20T12:20:27.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.856+0000 7f56bda53700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f56b809f560 0x7f56b8153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.856+0000 7f56bda53700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b814d110 con 0x7f56b80bfbe0 2020-11-20T12:20:27.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.857+0000 7f56bda53700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b814cbd0 con 0x7f56b80bf800 2020-11-20T12:20:27.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.857+0000 7f56bda53700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b814c9d0 con 0x7f56b809f560 2020-11-20T12:20:27.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.857+0000 7f56bca51700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f56b80bf800 0x7f56b814c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50310/0 (socket says 172.21.15.187:50310) 2020-11-20T12:20:27.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.857+0000 7f56bca51700 1 -- 172.21.15.187:0/2652046335 learned_addr learned my addr 172.21.15.187:0/2652046335 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:27.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.857+0000 7f56bd252700 1 --2- 172.21.15.187:0/2652046335 >> v2:172.21.15.5:3300/0 conn(0x7f56b809f560 0x7f56b8153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982056391 0 0) 0x7f56b814cbd0 con 0x7f56b80bf800 2020-11-20T12:20:27.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a4003240 con 0x7f56b80bf800 2020-11-20T12:20:27.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1153060982 0 0) 0x7f56a4003240 con 0x7f56b80bf800 2020-11-20T12:20:27.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 >> v2:172.21.15.5:3300/0 conn(0x7f56b809f560 msgr2=0x7f56b8153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:27.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 --2- 172.21.15.187:0/2652046335 >> v2:172.21.15.5:3300/0 conn(0x7f56b809f560 0x7f56b8153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 >> v1:172.21.15.5:6789/0 conn(0x7f56b80bfbe0 legacy=0x7f56b81480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b814ce70 con 0x7f56b80bf800 2020-11-20T12:20:27.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f56b4001860 con 0x7f56b80bf800 2020-11-20T12:20:27.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 >> v1:172.21.15.187:6789/0 conn(0x7f56b80bf800 legacy=0x7f56b814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56a4003730 0x7f56a4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56a40044e0 0x7f56a4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a4004e50 0x7f56a400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b814c9d0 con 0x7f56a4004e50 2020-11-20T12:20:27.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a4004260 con 0x7f56a40044e0 2020-11-20T12:20:27.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.858+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a400bc20 con 0x7f56a4003730 2020-11-20T12:20:27.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56affff700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56a40044e0 0x7f56a4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56bca51700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56a4003730 0x7f56a4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56bd252700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a4004e50 0x7f56a400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56affff700 1 -- 172.21.15.187:0/2652046335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56a4003730 msgr2=0x7f56a4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:27.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56affff700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56a4003730 0x7f56a4003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56affff700 1 -- 172.21.15.187:0/2652046335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a4004e50 msgr2=0x7f56a400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:27.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56affff700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a4004e50 0x7f56a400b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56affff700 1 -- 172.21.15.187:0/2652046335 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a4003240 con 0x7f56a40044e0 2020-11-20T12:20:27.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.859+0000 7f56bd252700 1 -- 172.21.15.187:0/2652046335 reap_dead start 2020-11-20T12:20:27.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.860+0000 7f56affff700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56a40044e0 0x7f56a4004960 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f56a8001920 tx=0x7f56a8007b90).ready entity=mon.1 client_cookie=c85cd173ea06458b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:27.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.860+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8007020 con 0x7f56a40044e0 2020-11-20T12:20:27.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.860+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a8007180 con 0x7f56a40044e0 2020-11-20T12:20:27.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.860+0000 7f56af7fe700 1 -- 172.21.15.187:0/2652046335 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a800c620 con 0x7f56a40044e0 2020-11-20T12:20:27.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.861+0000 7f56bda53700 1 -- 172.21.15.187:0/2652046335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56a40044e0 msgr2=0x7f56a4004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.861+0000 7f56bda53700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56a40044e0 0x7f56a4004960 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f56a8001920 tx=0x7f56a8007b90).stop 2020-11-20T12:20:27.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.861+0000 7f56bda53700 1 -- 172.21.15.187:0/2652046335 shutdown_connections 2020-11-20T12:20:27.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.861+0000 7f56bda53700 1 --2- 172.21.15.187:0/2652046335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56a40044e0 0x7f56a4004960 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.861+0000 7f56bda53700 1 -- 172.21.15.187:0/2652046335 >> 172.21.15.187:0/2652046335 conn(0x7f56b8068370 msgr2=0x7f56b80bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:27.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.861+0000 7f56bda53700 1 -- 172.21.15.187:0/2652046335 shutdown_connections 2020-11-20T12:20:27.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.861+0000 7f56bda53700 1 -- 172.21.15.187:0/2652046335 wait complete. 2020-11-20T12:20:27.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.862+0000 7f56bda53700 1 Processor -- start 2020-11-20T12:20:27.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.863+0000 7f56bda53700 1 -- start start 2020-11-20T12:20:27.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.863+0000 7f56bda53700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a40044e0 0x7f56b80ba680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.863+0000 7f56bda53700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56b8149180 0x7f56b8147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.863+0000 7f56bda53700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56b8149600 0x7f56b8140ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.863+0000 7f56bda53700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b81487a0 con 0x7f56a40044e0 2020-11-20T12:20:27.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.863+0000 7f56bda53700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8148a40 con 0x7f56b8149600 2020-11-20T12:20:27.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.863+0000 7f56bda53700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8148ce0 con 0x7f56b8149180 2020-11-20T12:20:27.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56b8149600 0x7f56b8140ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56b8149600 0x7f56b8140ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53720/0 (socket says 172.21.15.187:53720) 2020-11-20T12:20:27.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 -- 172.21.15.187:0/358664257 learned_addr learned my addr 172.21.15.187:0/358664257 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:27.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bca51700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a40044e0 0x7f56b80ba680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56affff700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56b8149180 0x7f56b8147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 -- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56b8149180 msgr2=0x7f56b8147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56b8149180 0x7f56b8147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 -- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a40044e0 msgr2=0x7f56b80ba680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:27.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a40044e0 0x7f56b80ba680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:27.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 -- 172.21.15.187:0/358664257 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80bab70 con 0x7f56b8149600 2020-11-20T12:20:27.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bca51700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a40044e0 0x7f56b80ba680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:27.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.864+0000 7f56bd252700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56b8149600 0x7f56b8140ce0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f56b8062a00 tx=0x7f56b0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:27.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.865+0000 7f56acff9700 1 -- 172.21.15.187:0/358664257 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0018070 con 0x7f56b8149600 2020-11-20T12:20:27.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.865+0000 7f56acff9700 1 -- 172.21.15.187:0/358664257 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b00126d0 con 0x7f56b8149600 2020-11-20T12:20:27.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.865+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b815be40 con 0x7f56b8149600 2020-11-20T12:20:27.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.865+0000 7f56acff9700 1 -- 172.21.15.187:0/358664257 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0012a50 con 0x7f56b8149600 2020-11-20T12:20:27.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.865+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b8148540 con 0x7f56b8149600 2020-11-20T12:20:27.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.866+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b8080050 con 0x7f56b8149600 2020-11-20T12:20:27.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.867+0000 7f56acff9700 1 -- 172.21.15.187:0/358664257 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f56b0014470 con 0x7f56b8149600 2020-11-20T12:20:27.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.867+0000 7f56acff9700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5690056d20 0x7f56900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:27.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.867+0000 7f56acff9700 1 -- 172.21.15.187:0/358664257 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f56b0070410 con 0x7f56b8149600 2020-11-20T12:20:27.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.868+0000 7f56bca51700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5690056d20 0x7f56900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:27.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.868+0000 7f56bca51700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5690056d20 0x7f56900591e0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f56b4001150 tx=0x7f56b814d000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:27.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:27.871+0000 7f56acff9700 1 -- 172.21.15.187:0/358664257 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f56b0045800 con 0x7f56b8149600 2020-11-20T12:20:28.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.062+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f56b8148ea0 con 0x7f56b8149600 2020-11-20T12:20:28.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.063+0000 7f56acff9700 1 -- 172.21.15.187:0/358664257 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v150) v1 ==== 92+0+10 (secure 0 0 0) 0x7f56b0048e20 con 0x7f56b8149600 2020-11-20T12:20:28.066 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:20:28.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5690056d20 msgr2=0x7f56900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5690056d20 0x7f56900591e0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f56b4001150 tx=0x7f56b814d000).stop 2020-11-20T12:20:28.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56b8149600 msgr2=0x7f56b8140ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56b8149600 0x7f56b8140ce0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f56b8062a00 tx=0x7f56b0008040).stop 2020-11-20T12:20:28.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 shutdown_connections 2020-11-20T12:20:28.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5690056d20 0x7f56900591e0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56a40044e0 0x7f56b80ba680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56b8149600 0x7f56b8140ce0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 --2- 172.21.15.187:0/358664257 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f56b8149180 0x7f56b8147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 >> 172.21.15.187:0/358664257 conn(0x7f56b8068370 msgr2=0x7f56b80bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:28.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 shutdown_connections 2020-11-20T12:20:28.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.066+0000 7f56bda53700 1 -- 172.21.15.187:0/358664257 wait complete. 2020-11-20T12:20:28.092 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:20:28.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.342+0000 7efc1b59e700 1 Processor -- start 2020-11-20T12:20:28.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.342+0000 7efc1b59e700 1 -- start start 2020-11-20T12:20:28.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.343+0000 7efc1b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c09dba0 0x7efc1c0676a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.343+0000 7efc1b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efc1c09df80 0x7efc1c145f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.343+0000 7efc1b59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c063520 con 0x7efc1c0b8190 2020-11-20T12:20:28.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.343+0000 7efc1b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1c0637c0 con 0x7efc1c09df80 2020-11-20T12:20:28.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.343+0000 7efc1b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c063a60 con 0x7efc1c09dba0 2020-11-20T12:20:28.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.344+0000 7efc1ad9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7efc1c0b8190 0x7efc1c060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50334/0 (socket says 172.21.15.187:50334) 2020-11-20T12:20:28.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.344+0000 7efc1ad9d700 1 -- 172.21.15.187:0/1367336132 learned_addr learned my addr 172.21.15.187:0/1367336132 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:28.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.344+0000 7efc19d9b700 1 --2- 172.21.15.187:0/1367336132 >> v2:172.21.15.5:3300/0 conn(0x7efc1c09df80 0x7efc1c145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1a59c700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c09dba0 0x7efc1c0676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854581042 0 0) 0x7efc1c063520 con 0x7efc1c0b8190 2020-11-20T12:20:28.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc00003240 con 0x7efc1c0b8190 2020-11-20T12:20:28.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1491126973 0 0) 0x7efc00003240 con 0x7efc1c0b8190 2020-11-20T12:20:28.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c09dba0 msgr2=0x7efc1c0676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:28.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c09dba0 0x7efc1c0676a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 >> v2:172.21.15.5:3300/0 conn(0x7efc1c09df80 msgr2=0x7efc1c145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:28.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 --2- 172.21.15.187:0/1367336132 >> v2:172.21.15.5:3300/0 conn(0x7efc1c09df80 0x7efc1c145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c063f00 con 0x7efc1c0b8190 2020-11-20T12:20:28.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7efc14001860 con 0x7efc1c0b8190 2020-11-20T12:20:28.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 >> v1:172.21.15.187:6789/0 conn(0x7efc1c0b8190 legacy=0x7efc1c060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc000037b0 0x7efc00003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc000044e0 0x7efc00004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c09dba0 0x7efc00004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0637c0 con 0x7efc000044e0 2020-11-20T12:20:28.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c063a60 con 0x7efc000037b0 2020-11-20T12:20:28.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.345+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc00005340 con 0x7efc1c09dba0 2020-11-20T12:20:28.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.346+0000 7efc1a59c700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc000037b0 0x7efc00003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.346+0000 7efc1ad9d700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c09dba0 0x7efc00004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.346+0000 7efc19d9b700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc000044e0 0x7efc00004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1a59c700 1 -- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c09dba0 msgr2=0x7efc00004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:28.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1a59c700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c09dba0 0x7efc00004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1a59c700 1 -- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc000044e0 msgr2=0x7efc00004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:28.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1a59c700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc000044e0 0x7efc00004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1a59c700 1 -- 172.21.15.187:0/1367336132 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00003240 con 0x7efc000037b0 2020-11-20T12:20:28.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc19d9b700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc000044e0 0x7efc00004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:28.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1a59c700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc000037b0 0x7efc00003bd0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7efc0c00bc00 tx=0x7efc0c006df0).ready entity=mon.1 client_cookie=1bfcfb0b8b97d22f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:28.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00b310 con 0x7efc000037b0 2020-11-20T12:20:28.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c00eb40 con 0x7efc000037b0 2020-11-20T12:20:28.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.347+0000 7efc1959a700 1 -- 172.21.15.187:0/1367336132 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00eec0 con 0x7efc000037b0 2020-11-20T12:20:28.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.348+0000 7efc1b59e700 1 -- 172.21.15.187:0/1367336132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc000037b0 msgr2=0x7efc00003bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.348+0000 7efc1b59e700 1 --2- 172.21.15.187:0/1367336132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc000037b0 0x7efc00003bd0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7efc0c00bc00 tx=0x7efc0c006df0).stop 2020-11-20T12:20:28.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.348+0000 7efc1ad9d700 1 -- 172.21.15.187:0/1367336132 reap_dead start 2020-11-20T12:20:28.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.348+0000 7efc1b59e700 1 -- 172.21.15.187:0/1367336132 shutdown_connections 2020-11-20T12:20:28.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.348+0000 7efc1b59e700 1 -- 172.21.15.187:0/1367336132 >> 172.21.15.187:0/1367336132 conn(0x7efc1c0bbc20 msgr2=0x7efc1c0bc030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:28.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.348+0000 7efc1b59e700 1 -- 172.21.15.187:0/1367336132 shutdown_connections 2020-11-20T12:20:28.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.348+0000 7efc1b59e700 1 -- 172.21.15.187:0/1367336132 wait complete. 2020-11-20T12:20:28.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.350+0000 7efc1b59e700 1 Processor -- start 2020-11-20T12:20:28.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.350+0000 7efc1b59e700 1 -- start start 2020-11-20T12:20:28.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc1c13cb70 0x7efc1c13cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c13d440 0x7efc1c0b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc1c0b5810 0x7efc1c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c139b40 con 0x7efc1c13cb70 2020-11-20T12:20:28.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0b2760 con 0x7efc1c0b5810 2020-11-20T12:20:28.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0b24c0 con 0x7efc1c13d440 2020-11-20T12:20:28.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc1c0b5810 0x7efc1c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1ad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc1c0b5810 0x7efc1c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53740/0 (socket says 172.21.15.187:53740) 2020-11-20T12:20:28.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1ad9d700 1 -- 172.21.15.187:0/4274128063 learned_addr learned my addr 172.21.15.187:0/4274128063 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:28.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc19d9b700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c13d440 0x7efc1c0b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.351+0000 7efc1a59c700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc1c13cb70 0x7efc1c13cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1ad9d700 1 -- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c13d440 msgr2=0x7efc1c0b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1ad9d700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c13d440 0x7efc1c0b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1ad9d700 1 -- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc1c13cb70 msgr2=0x7efc1c13cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1ad9d700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc1c13cb70 0x7efc1c13cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1ad9d700 1 -- 172.21.15.187:0/4274128063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c1397a0 con 0x7efc1c0b5810 2020-11-20T12:20:28.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc19d9b700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c13d440 0x7efc1c0b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:28.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1a59c700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc1c13cb70 0x7efc1c13cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:28.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1ad9d700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc1c0b5810 0x7efc1c0b1070 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7efc1c0b8570 tx=0x7efc14001850).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:28.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc0a7fc700 1 -- 172.21.15.187:0/4274128063 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1400c9a0 con 0x7efc1c0b5810 2020-11-20T12:20:28.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc0a7fc700 1 -- 172.21.15.187:0/4274128063 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1400cb20 con 0x7efc1c0b5810 2020-11-20T12:20:28.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c13a530 con 0x7efc1c0b5810 2020-11-20T12:20:28.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc0a7fc700 1 -- 172.21.15.187:0/4274128063 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1400cee0 con 0x7efc1c0b5810 2020-11-20T12:20:28.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.352+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c0b1720 con 0x7efc1c0b5810 2020-11-20T12:20:28.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.354+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c13a720 con 0x7efc1c0b5810 2020-11-20T12:20:28.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.354+0000 7efc0a7fc700 1 -- 172.21.15.187:0/4274128063 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efc14014640 con 0x7efc1c0b5810 2020-11-20T12:20:28.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.355+0000 7efc0a7fc700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efbf0056d20 0x7efbf00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.355+0000 7efc0a7fc700 1 -- 172.21.15.187:0/4274128063 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efc1406ff80 con 0x7efc1c0b5810 2020-11-20T12:20:28.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.355+0000 7efc1a59c700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efbf0056d20 0x7efbf00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.355+0000 7efc1a59c700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efbf0056d20 0x7efbf00591e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7efc1c05e630 tx=0x7efc0c019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:28.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.358+0000 7efc0a7fc700 1 -- 172.21.15.187:0/4274128063 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efc14045370 con 0x7efc1c0b5810 2020-11-20T12:20:28.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.549+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc1c139990 con 0x7efc1c0b5810 2020-11-20T12:20:28.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.550+0000 7efc0a7fc700 1 -- 172.21.15.187:0/4274128063 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+10 (secure 0 0 0) 0x7efc14048990 con 0x7efc1c0b5810 2020-11-20T12:20:28.553 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:20:28.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efbf0056d20 msgr2=0x7efbf00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efbf0056d20 0x7efbf00591e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7efc1c05e630 tx=0x7efc0c019000).stop 2020-11-20T12:20:28.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc1c0b5810 msgr2=0x7efc1c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc1c0b5810 0x7efc1c0b1070 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7efc1c0b8570 tx=0x7efc14001850).stop 2020-11-20T12:20:28.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 shutdown_connections 2020-11-20T12:20:28.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efbf0056d20 0x7efbf00591e0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc1c13cb70 0x7efc1c13cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc1c0b5810 0x7efc1c0b1070 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 --2- 172.21.15.187:0/4274128063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc1c13d440 0x7efc1c0b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.553+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 >> 172.21.15.187:0/4274128063 conn(0x7efc1c0bbc20 msgr2=0x7efc1c145040 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:28.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.554+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 shutdown_connections 2020-11-20T12:20:28.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.554+0000 7efc1b59e700 1 -- 172.21.15.187:0/4274128063 wait complete. 2020-11-20T12:20:28.579 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:20:28.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.815+0000 7f9b00adf700 1 Processor -- start 2020-11-20T12:20:28.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.816+0000 7f9b00adf700 1 -- start start 2020-11-20T12:20:28.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.817+0000 7f9b00adf700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9afc065300 0x7f9afc0656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.817+0000 7f9b00adf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc09a8a0 0x7f9afc062400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.817+0000 7f9b00adf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9afc066420 con 0x7f9afc068030 2020-11-20T12:20:28.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.817+0000 7f9b00adf700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9afc0b7380 con 0x7f9afc065300 2020-11-20T12:20:28.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.817+0000 7f9b00adf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc061f60 con 0x7f9afc09a8a0 2020-11-20T12:20:28.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.818+0000 7f9afa59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9afc065300 0x7f9afc0656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.818+0000 7f9af1d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc09a8a0 0x7f9afc062400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.818+0000 7f9afa59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9afc065300 0x7f9afc0656e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:46008/0 (socket says 172.21.15.187:46008) 2020-11-20T12:20:28.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.818+0000 7f9afa59c700 1 -- 172.21.15.187:0/456228031 learned_addr learned my addr 172.21.15.187:0/456228031 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:28.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.819+0000 7f9afa59c700 1 -- 172.21.15.187:0/456228031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc09a8a0 msgr2=0x7f9afc062400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:28.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.819+0000 7f9afa59c700 1 --2- 172.21.15.187:0/456228031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc09a8a0 0x7f9afc062400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.819+0000 7f9afa59c700 1 -- 172.21.15.187:0/456228031 >> v1:172.21.15.5:6789/0 conn(0x7f9afc068030 legacy=0x7f9afc146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.819+0000 7f9afa59c700 1 -- 172.21.15.187:0/456228031 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc0b6fd0 con 0x7f9afc065300 2020-11-20T12:20:28.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.820+0000 7f9afa59c700 1 --2- 172.21.15.187:0/456228031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc065300 0x7f9afc0656e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f9aec006fd0 tx=0x7f9aec006eb0).ready entity=mon.0 client_cookie=16fa473c36cce636 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:28.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.820+0000 7f9af9d9b700 1 -- 172.21.15.187:0/456228031 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec0098a0 con 0x7f9afc065300 2020-11-20T12:20:28.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9af9d9b700 1 -- 172.21.15.187:0/456228031 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec009a00 con 0x7f9afc065300 2020-11-20T12:20:28.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9af9d9b700 1 -- 172.21.15.187:0/456228031 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec009d80 con 0x7f9afc065300 2020-11-20T12:20:28.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9b00adf700 1 -- 172.21.15.187:0/456228031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc065300 msgr2=0x7f9afc0656e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9b00adf700 1 --2- 172.21.15.187:0/456228031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc065300 0x7f9afc0656e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f9aec006fd0 tx=0x7f9aec006eb0).stop 2020-11-20T12:20:28.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9b00adf700 1 -- 172.21.15.187:0/456228031 shutdown_connections 2020-11-20T12:20:28.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9b00adf700 1 --2- 172.21.15.187:0/456228031 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc065300 0x7f9afc0656e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9b00adf700 1 --2- 172.21.15.187:0/456228031 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc09a8a0 0x7f9afc062400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9b00adf700 1 -- 172.21.15.187:0/456228031 >> 172.21.15.187:0/456228031 conn(0x7f9afc0bacf0 msgr2=0x7f9afc0bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:28.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9b00adf700 1 -- 172.21.15.187:0/456228031 shutdown_connections 2020-11-20T12:20:28.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.821+0000 7f9b00adf700 1 -- 172.21.15.187:0/456228031 wait complete. 2020-11-20T12:20:28.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.823+0000 7f9b00adf700 1 Processor -- start 2020-11-20T12:20:28.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.823+0000 7f9b00adf700 1 -- start start 2020-11-20T12:20:28.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.824+0000 7f9b00adf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9afc065300 0x7f9afc158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.824+0000 7f9b00adf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc068030 0x7f9afc0a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.824+0000 7f9b00adf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc09a8a0 0x7f9afc15bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.824+0000 7f9b00adf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc158330 con 0x7f9afc09a8a0 2020-11-20T12:20:28.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.824+0000 7f9b00adf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc158520 con 0x7f9afc065300 2020-11-20T12:20:28.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.824+0000 7f9b00adf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc1587c0 con 0x7f9afc068030 2020-11-20T12:20:28.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.824+0000 7f9afa59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9afc065300 0x7f9afc158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afa59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9afc065300 0x7f9afc158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53750/0 (socket says 172.21.15.187:53750) 2020-11-20T12:20:28.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afa59c700 1 -- 172.21.15.187:0/3821846792 learned_addr learned my addr 172.21.15.187:0/3821846792 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:28.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9af1d9b700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc068030 0x7f9afc0a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afad9d700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc09a8a0 0x7f9afc15bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afa59c700 1 -- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc068030 msgr2=0x7f9afc0a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afa59c700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc068030 0x7f9afc0a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afa59c700 1 -- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc09a8a0 msgr2=0x7f9afc15bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:28.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afa59c700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc09a8a0 0x7f9afc15bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:28.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afa59c700 1 -- 172.21.15.187:0/3821846792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc158a60 con 0x7f9afc065300 2020-11-20T12:20:28.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afad9d700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc09a8a0 0x7f9afc15bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:28.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.825+0000 7f9afa59c700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9afc065300 0x7f9afc158f00 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f9afc064b50 tx=0x7f9aec015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:28.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.826+0000 7f9af2ffd700 1 -- 172.21.15.187:0/3821846792 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec011e50 con 0x7f9afc065300 2020-11-20T12:20:28.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.826+0000 7f9af2ffd700 1 -- 172.21.15.187:0/3821846792 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec01e460 con 0x7f9afc065300 2020-11-20T12:20:28.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.826+0000 7f9af2ffd700 1 -- 172.21.15.187:0/3821846792 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec01e800 con 0x7f9afc065300 2020-11-20T12:20:28.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.826+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9afc0b4140 con 0x7f9afc065300 2020-11-20T12:20:28.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.826+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9afc0a0bc0 con 0x7f9afc065300 2020-11-20T12:20:28.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.827+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9afc0b4360 con 0x7f9afc065300 2020-11-20T12:20:28.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.828+0000 7f9af2ffd700 1 -- 172.21.15.187:0/3821846792 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9aec025030 con 0x7f9afc065300 2020-11-20T12:20:28.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.828+0000 7f9af2ffd700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ad8056d20 0x7f9ad80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:28.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.828+0000 7f9af2ffd700 1 -- 172.21.15.187:0/3821846792 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9aec071790 con 0x7f9afc065300 2020-11-20T12:20:28.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.829+0000 7f9af1d9b700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ad8056d20 0x7f9ad80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:28.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.829+0000 7f9af1d9b700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ad8056d20 0x7f9ad80591e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f9ae4006fd0 tx=0x7f9ae4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:28.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:28.832+0000 7f9af2ffd700 1 -- 172.21.15.187:0/3821846792 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9aec046c90 con 0x7f9afc065300 2020-11-20T12:20:29.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.016+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9afc044f60 con 0x7f9afc065300 2020-11-20T12:20:29.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.017+0000 7f9af2ffd700 1 -- 172.21.15.187:0/3821846792 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9aec023070 con 0x7f9afc065300 2020-11-20T12:20:29.019 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:20:29.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ad8056d20 msgr2=0x7f9ad80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:29.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ad8056d20 0x7f9ad80591e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f9ae4006fd0 tx=0x7f9ae4008040).stop 2020-11-20T12:20:29.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9afc065300 msgr2=0x7f9afc158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:29.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9afc065300 0x7f9afc158f00 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f9afc064b50 tx=0x7f9aec015000).stop 2020-11-20T12:20:29.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 shutdown_connections 2020-11-20T12:20:29.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9ad8056d20 0x7f9ad80591e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:29.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9afc09a8a0 0x7f9afc15bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:29.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9afc065300 0x7f9afc158f00 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:29.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 --2- 172.21.15.187:0/3821846792 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9afc068030 0x7f9afc0a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:29.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.019+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 >> 172.21.15.187:0/3821846792 conn(0x7f9afc0bacf0 msgr2=0x7f9afc146800 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:29.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.020+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 shutdown_connections 2020-11-20T12:20:29.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.020+0000 7f9b00adf700 1 -- 172.21.15.187:0/3821846792 wait complete. 2020-11-20T12:20:29.045 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:20:29.046 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:29.286 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.283+0000 7fa83abe1700 1 Processor -- start 2020-11-20T12:20:29.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.283+0000 7fa83abe1700 1 -- start start 2020-11-20T12:20:29.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.285+0000 7fa83abe1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa834145cc0 0x7fa834066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:29.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.285+0000 7fa83abe1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa8340683f0 0x7fa834065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:29.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.285+0000 7fa83abe1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83414c170 con 0x7fa834065680 2020-11-20T12:20:29.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.285+0000 7fa83abe1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8341497f0 con 0x7fa834145cc0 2020-11-20T12:20:29.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.285+0000 7fa83abe1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83414c410 con 0x7fa8340683f0 2020-11-20T12:20:29.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.285+0000 7fa8393de700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa8340683f0 0x7fa834065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:29.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.285+0000 7fa8393de700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa8340683f0 0x7fa834065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53758/0 (socket says 172.21.15.187:53758) 2020-11-20T12:20:29.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.285+0000 7fa8393de700 1 -- 172.21.15.187:0/1885726724 learned_addr learned my addr 172.21.15.187:0/1885726724 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:29.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.286+0000 7fa839bdf700 1 --2- 172.21.15.187:0/1885726724 >> v2:172.21.15.5:3300/0 conn(0x7fa834145cc0 0x7fa834066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:29.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.286+0000 7fa8393de700 1 -- 172.21.15.187:0/1885726724 >> v2:172.21.15.5:3300/0 conn(0x7fa834145cc0 msgr2=0x7fa834066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:29.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.286+0000 7fa8393de700 1 --2- 172.21.15.187:0/1885726724 >> v2:172.21.15.5:3300/0 conn(0x7fa834145cc0 0x7fa834066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:29.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.286+0000 7fa8393de700 1 -- 172.21.15.187:0/1885726724 >> v1:172.21.15.5:6789/0 conn(0x7fa834065680 legacy=0x7fa834061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:29.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.286+0000 7fa8393de700 1 -- 172.21.15.187:0/1885726724 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83414c6b0 con 0x7fa8340683f0 2020-11-20T12:20:29.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.286+0000 7fa8393de700 1 --2- 172.21.15.187:0/1885726724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8340683f0 0x7fa834065190 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7fa8340674b0 tx=0x7fa824006eb0).ready entity=mon.1 client_cookie=1153b9c6da44dbf3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:29.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.286+0000 7fa838bdd700 1 -- 172.21.15.187:0/1885726724 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa824009890 con 0x7fa8340683f0 2020-11-20T12:20:29.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa838bdd700 1 -- 172.21.15.187:0/1885726724 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8240099f0 con 0x7fa8340683f0 2020-11-20T12:20:29.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa838bdd700 1 -- 172.21.15.187:0/1885726724 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa824009d70 con 0x7fa8340683f0 2020-11-20T12:20:29.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa83abe1700 1 -- 172.21.15.187:0/1885726724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8340683f0 msgr2=0x7fa834065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:29.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa83abe1700 1 --2- 172.21.15.187:0/1885726724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8340683f0 0x7fa834065190 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7fa8340674b0 tx=0x7fa824006eb0).stop 2020-11-20T12:20:29.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa83abe1700 1 -- 172.21.15.187:0/1885726724 shutdown_connections 2020-11-20T12:20:29.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa83abe1700 1 --2- 172.21.15.187:0/1885726724 >> v2:172.21.15.5:3300/0 conn(0x7fa834145cc0 0x7fa834066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:29.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa83abe1700 1 --2- 172.21.15.187:0/1885726724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8340683f0 0x7fa834065190 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:29.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa83abe1700 1 -- 172.21.15.187:0/1885726724 >> 172.21.15.187:0/1885726724 conn(0x7fa8340bab30 msgr2=0x7fa8340baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:29.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa83abe1700 1 -- 172.21.15.187:0/1885726724 shutdown_connections 2020-11-20T12:20:29.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.287+0000 7fa83abe1700 1 -- 172.21.15.187:0/1885726724 wait complete. 2020-11-20T12:20:29.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.289+0000 7fa83abe1700 1 Processor -- start 2020-11-20T12:20:29.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.289+0000 7fa83abe1700 1 -- start start 2020-11-20T12:20:29.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.290+0000 7fa83abe1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa834065680 0x7fa8340b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:29.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.290+0000 7fa83abe1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa834145cc0 0x7fa8340b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:29.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.290+0000 7fa83abe1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8341468e0 0x7fa834141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:29.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.290+0000 7fa83abe1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340b49b0 con 0x7fa834145cc0 2020-11-20T12:20:29.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.290+0000 7fa83abe1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340a03d0 con 0x7fa8341468e0 2020-11-20T12:20:29.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.290+0000 7fa83abe1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340a0670 con 0x7fa834065680 2020-11-20T12:20:29.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8341468e0 0x7fa834141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:29.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8341468e0 0x7fa834141e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53768/0 (socket says 172.21.15.187:53768) 2020-11-20T12:20:29.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 -- 172.21.15.187:0/2847933909 learned_addr learned my addr 172.21.15.187:0/2847933909 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:29.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 -- 172.21.15.187:0/2847933909 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa834065680 msgr2=0x7fa8340b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:29.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa834065680 0x7fa8340b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:29.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 -- 172.21.15.187:0/2847933909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa834145cc0 msgr2=0x7fa8340b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:29.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa834145cc0 0x7fa8340b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:29.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 -- 172.21.15.187:0/2847933909 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8340a0910 con 0x7fa8341468e0 2020-11-20T12:20:29.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.291+0000 7fa83a3e0700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8341468e0 0x7fa834141e50 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7fa82c00bae0 tx=0x7fa82c009120).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:29.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.292+0000 7fa821ffb700 1 -- 172.21.15.187:0/2847933909 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00c890 con 0x7fa8341468e0 2020-11-20T12:20:29.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.292+0000 7fa821ffb700 1 -- 172.21.15.187:0/2847933909 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c00ca10 con 0x7fa8341468e0 2020-11-20T12:20:29.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.292+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8340a0140 con 0x7fa8341468e0 2020-11-20T12:20:29.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.292+0000 7fa821ffb700 1 -- 172.21.15.187:0/2847933909 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00cbc0 con 0x7fa8341468e0 2020-11-20T12:20:29.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.292+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa834142340 con 0x7fa8341468e0 2020-11-20T12:20:29.481 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.293+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa834061a20 con 0x7fa8341468e0 2020-11-20T12:20:29.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.294+0000 7fa821ffb700 1 -- 172.21.15.187:0/2847933909 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa82c01c070 con 0x7fa8341468e0 2020-11-20T12:20:29.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.294+0000 7fa821ffb700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa80c056d50 0x7fa80c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:29.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.294+0000 7fa821ffb700 1 -- 172.21.15.187:0/2847933909 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa82c06f760 con 0x7fa8341468e0 2020-11-20T12:20:29.482 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.295+0000 7fa839bdf700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa80c056d50 0x7fa80c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:29.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.295+0000 7fa839bdf700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa80c056d50 0x7fa80c059210 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fa83000b690 tx=0x7fa830008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:29.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.299+0000 7fa821ffb700 1 -- 172.21.15.187:0/2847933909 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa82c044bb0 con 0x7fa8341468e0 2020-11-20T12:20:29.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.547+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa83414bdf0 con 0x7fa8341468e0 2020-11-20T12:20:29.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.548+0000 7fa821ffb700 1 -- 172.21.15.187:0/2847933909 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2142 (secure 0 0 0) 0x7fa82c01a870 con 0x7fa8341468e0 2020-11-20T12:20:29.551 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:29.551 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":153,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/1287724648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1287724648},{"type":"v1","addr":"172.21.15.5:6833","nonce":1287724648}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:23.278959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:29.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa80c056d50 msgr2=0x7fa80c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:29.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa80c056d50 0x7fa80c059210 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fa83000b690 tx=0x7fa830008040).stop 2020-11-20T12:20:29.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8341468e0 msgr2=0x7fa834141e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:29.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8341468e0 0x7fa834141e50 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7fa82c00bae0 tx=0x7fa82c009120).stop 2020-11-20T12:20:29.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 shutdown_connections 2020-11-20T12:20:29.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa80c056d50 0x7fa80c059210 secure :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0x7fa83000b690 tx=0x7fa830008040).stop 2020-11-20T12:20:29.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8341468e0 0x7fa834141e50 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:29.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa834065680 0x7fa8340b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:29.556 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 --2- 172.21.15.187:0/2847933909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa834145cc0 0x7fa8340b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:29.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.552+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 >> 172.21.15.187:0/2847933909 conn(0x7fa8340bab30 msgr2=0x7fa8340baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:29.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.553+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 shutdown_connections 2020-11-20T12:20:29.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:29.553+0000 7fa83abe1700 1 -- 172.21.15.187:0/2847933909 wait complete. 2020-11-20T12:20:29.558 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 153 2020-11-20T12:20:29.782 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 153, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:20:21.071872+0000', 'modified': '2020-11-20T12:20:23.278959+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9661}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9661': {'gid': 9661, 'name': 'a', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3618925546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3618925546}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3618925546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:20:29.783 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:20:29.784 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:20:30.063 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":9661,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":8.9615865209999992,"mdsmap_epoch":153,"osdmap_epoch":150,"osdmap_epoch_barrier":150,"uptime":20.614908763999999} 2020-11-20T12:20:30.064 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 9661, "mdsmap_epoch": 153, "osdmap_epoch": 150, "osdmap_epoch_barrier": 150, "rank_uptime": 8.961586521, "state": "up:active", "uptime": 20.614908764, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:20:30.065 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:20:30.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:20:30.066 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:20:30.118 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:20:30.119 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:20:30.256 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:20:30.256 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:20:30.294 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:20:30.295 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:20:30.306 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:20:30.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:20:30.307 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:20:30.401 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2020-11-20T12:20:30.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.646+0000 7f2c5a1a9700 1 Processor -- start 2020-11-20T12:20:30.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.646+0000 7f2c5a1a9700 1 -- start start 2020-11-20T12:20:30.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.647+0000 7f2c5a1a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c541484d0 0x7f2c54061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:30.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.647+0000 7f2c5a1a9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c540622c0 0x7f2c540b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:30.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.647+0000 7f2c5a1a9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2c54066f70 0x7f2c540b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:30.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.647+0000 7f2c5a1a9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c54067730 con 0x7f2c54066f70 2020-11-20T12:20:30.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.647+0000 7f2c5a1a9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c540b1300 con 0x7f2c540622c0 2020-11-20T12:20:30.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.647+0000 7f2c5a1a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c540b15a0 con 0x7f2c541484d0 2020-11-20T12:20:30.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.648+0000 7f2c589a6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c540622c0 0x7f2c540b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:30.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.648+0000 7f2c589a6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c540622c0 0x7f2c540b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53780/0 (socket says 172.21.15.187:53780) 2020-11-20T12:20:30.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.648+0000 7f2c589a6700 1 -- 172.21.15.187:0/3538942120 learned_addr learned my addr 172.21.15.187:0/3538942120 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:30.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.648+0000 7f2c599a8700 1 --2- 172.21.15.187:0/3538942120 >> v2:172.21.15.5:3300/0 conn(0x7f2c54066f70 0x7f2c540b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:30.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.648+0000 7f2c591a7700 1 --2- 172.21.15.187:0/3538942120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c541484d0 0x7f2c54061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:30.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.648+0000 7f2c589a6700 1 -- 172.21.15.187:0/3538942120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c541484d0 msgr2=0x7f2c54061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:30.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.648+0000 7f2c589a6700 1 --2- 172.21.15.187:0/3538942120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c541484d0 0x7f2c54061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:30.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.649+0000 7f2c589a6700 1 -- 172.21.15.187:0/3538942120 >> v2:172.21.15.5:3300/0 conn(0x7f2c54066f70 msgr2=0x7f2c540b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:30.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.649+0000 7f2c589a6700 1 --2- 172.21.15.187:0/3538942120 >> v2:172.21.15.5:3300/0 conn(0x7f2c54066f70 0x7f2c540b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:30.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.649+0000 7f2c589a6700 1 -- 172.21.15.187:0/3538942120 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c540b1840 con 0x7f2c540622c0 2020-11-20T12:20:30.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.649+0000 7f2c589a6700 1 --2- 172.21.15.187:0/3538942120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c540622c0 0x7f2c540b5640 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 rx=0x7f2c54066380 tx=0x7f2c44006eb0).ready entity=mon.1 client_cookie=83bb8c318cf62d7a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:30.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.649+0000 7f2c43fff700 1 -- 172.21.15.187:0/3538942120 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c440098d0 con 0x7f2c540622c0 2020-11-20T12:20:30.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.649+0000 7f2c43fff700 1 -- 172.21.15.187:0/3538942120 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c44009a30 con 0x7f2c540622c0 2020-11-20T12:20:30.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c43fff700 1 -- 172.21.15.187:0/3538942120 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44009db0 con 0x7f2c540622c0 2020-11-20T12:20:30.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/3538942120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c540622c0 msgr2=0x7f2c540b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:30.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/3538942120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c540622c0 0x7f2c540b5640 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 rx=0x7f2c54066380 tx=0x7f2c44006eb0).stop 2020-11-20T12:20:30.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/3538942120 shutdown_connections 2020-11-20T12:20:30.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/3538942120 >> v2:172.21.15.5:3300/0 conn(0x7f2c54066f70 0x7f2c540b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:30.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/3538942120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c541484d0 0x7f2c54061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:30.668 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/3538942120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c540622c0 0x7f2c540b5640 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:30.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/3538942120 >> 172.21.15.187:0/3538942120 conn(0x7f2c540bab30 msgr2=0x7f2c540baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:30.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/3538942120 shutdown_connections 2020-11-20T12:20:30.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.650+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/3538942120 wait complete. 2020-11-20T12:20:30.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.652+0000 7f2c5a1a9700 1 Processor -- start 2020-11-20T12:20:30.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.652+0000 7f2c5a1a9700 1 -- start start 2020-11-20T12:20:30.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.653+0000 7f2c5a1a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c540622c0 0x7f2c54139c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:30.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.653+0000 7f2c5a1a9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c54066f70 0x7f2c5415c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:30.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.653+0000 7f2c5a1a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c541484d0 0x7f2c5409f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:30.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.653+0000 7f2c5a1a9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5415c8b0 con 0x7f2c54066f70 2020-11-20T12:20:30.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.654+0000 7f2c5a1a9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c540a0880 con 0x7f2c541484d0 2020-11-20T12:20:30.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.654+0000 7f2c5a1a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5409e000 con 0x7f2c540622c0 2020-11-20T12:20:30.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.654+0000 7f2c599a8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c541484d0 0x7f2c5409f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:30.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.654+0000 7f2c599a8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c541484d0 0x7f2c5409f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53786/0 (socket says 172.21.15.187:53786) 2020-11-20T12:20:30.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.654+0000 7f2c599a8700 1 -- 172.21.15.187:0/1171147345 learned_addr learned my addr 172.21.15.187:0/1171147345 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:30.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.654+0000 7f2c591a7700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c540622c0 0x7f2c54139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:30.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.654+0000 7f2c589a6700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c54066f70 0x7f2c5415c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:30.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c599a8700 1 -- 172.21.15.187:0/1171147345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c540622c0 msgr2=0x7f2c54139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:30.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c599a8700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c540622c0 0x7f2c54139c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:30.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c599a8700 1 -- 172.21.15.187:0/1171147345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c54066f70 msgr2=0x7f2c5415c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:30.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c599a8700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c54066f70 0x7f2c5415c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:30.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c599a8700 1 -- 172.21.15.187:0/1171147345 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5409e2a0 con 0x7f2c541484d0 2020-11-20T12:20:30.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c599a8700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c541484d0 0x7f2c5409f670 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7f2c4c00d860 tx=0x7f2c4c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:30.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c417fa700 1 -- 172.21.15.187:0/1171147345 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c0096a0 con 0x7f2c541484d0 2020-11-20T12:20:30.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c417fa700 1 -- 172.21.15.187:0/1171147345 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c009800 con 0x7f2c541484d0 2020-11-20T12:20:30.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c417fa700 1 -- 172.21.15.187:0/1171147345 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c009ba0 con 0x7f2c541484d0 2020-11-20T12:20:30.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c540b1ae0 con 0x7f2c541484d0 2020-11-20T12:20:30.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.655+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5409ddb0 con 0x7f2c541484d0 2020-11-20T12:20:30.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.657+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c540b1c70 con 0x7f2c541484d0 2020-11-20T12:20:30.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.657+0000 7f2c417fa700 1 -- 172.21.15.187:0/1171147345 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2c4c013480 con 0x7f2c541484d0 2020-11-20T12:20:30.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.658+0000 7f2c417fa700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c2c056d20 0x7f2c2c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:30.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.658+0000 7f2c417fa700 1 -- 172.21.15.187:0/1171147345 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2c4c0702b0 con 0x7f2c541484d0 2020-11-20T12:20:30.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.658+0000 7f2c591a7700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c2c056d20 0x7f2c2c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:30.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.659+0000 7f2c591a7700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c2c056d20 0x7f2c2c0591e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f2c50002cd0 tx=0x7f2c50008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:30.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.662+0000 7f2c417fa700 1 -- 172.21.15.187:0/1171147345 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2c4c0456a0 con 0x7f2c541484d0 2020-11-20T12:20:30.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:30.908+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f2c54061a20 con 0x7f2c541484d0 2020-11-20T12:20:31.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.537+0000 7f2c417fa700 1 -- 172.21.15.187:0/1171147345 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v154) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2c4c048cc0 con 0x7f2c541484d0 2020-11-20T12:20:31.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.539+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c2c056d20 msgr2=0x7f2c2c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:31.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.540+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c2c056d20 0x7f2c2c0591e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f2c50002cd0 tx=0x7f2c50008040).stop 2020-11-20T12:20:31.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.540+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c541484d0 msgr2=0x7f2c5409f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:31.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.540+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c541484d0 0x7f2c5409f670 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7f2c4c00d860 tx=0x7f2c4c008040).stop 2020-11-20T12:20:31.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.541+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 shutdown_connections 2020-11-20T12:20:31.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.541+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2c2c056d20 0x7f2c2c0591e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.541+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c541484d0 0x7f2c5409f670 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.541+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c540622c0 0x7f2c54139c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.541+0000 7f2c5a1a9700 1 --2- 172.21.15.187:0/1171147345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c54066f70 0x7f2c5415c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.541+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 >> 172.21.15.187:0/1171147345 conn(0x7f2c540bab30 msgr2=0x7f2c540b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:31.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.541+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 shutdown_connections 2020-11-20T12:20:31.546 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.541+0000 7f2c5a1a9700 1 -- 172.21.15.187:0/1171147345 wait complete. 2020-11-20T12:20:31.578 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:31.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.801+0000 7f5ba9322700 1 Processor -- start 2020-11-20T12:20:31.806 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.801+0000 7f5ba9322700 1 -- start start 2020-11-20T12:20:31.812 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.802+0000 7f5ba9322700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5ba40b6650 0x7f5ba40afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.803+0000 7f5ba9322700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5ba4066f70 0x7f5ba40aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.803+0000 7f5ba9322700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba40b1300 con 0x7f5ba41484d0 2020-11-20T12:20:31.813 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.803+0000 7f5ba9322700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba4067730 con 0x7f5ba40b6650 2020-11-20T12:20:31.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.803+0000 7f5ba9322700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba40b15a0 con 0x7f5ba4066f70 2020-11-20T12:20:31.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.803+0000 7f5ba3fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5ba41484d0 0x7f5ba4061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50396/0 (socket says 172.21.15.187:50396) 2020-11-20T12:20:31.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.803+0000 7f5ba3fff700 1 -- 172.21.15.187:0/3206316118 learned_addr learned my addr 172.21.15.187:0/3206316118 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:31.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.803+0000 7f5ba8b21700 1 --2- 172.21.15.187:0/3206316118 >> v2:172.21.15.187:3300/0 conn(0x7f5ba4066f70 0x7f5ba40aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220174679 0 0) 0x7f5ba40b1300 con 0x7f5ba41484d0 2020-11-20T12:20:31.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b88003240 con 0x7f5ba41484d0 2020-11-20T12:20:31.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5b9b7fe700 1 --2- 172.21.15.187:0/3206316118 >> v2:172.21.15.5:3300/0 conn(0x7f5ba40b6650 0x7f5ba40afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2783853271 0 0) 0x7f5b88003240 con 0x7f5ba41484d0 2020-11-20T12:20:31.815 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 >> v2:172.21.15.187:3300/0 conn(0x7f5ba4066f70 msgr2=0x7f5ba40aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:31.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 --2- 172.21.15.187:0/3206316118 >> v2:172.21.15.187:3300/0 conn(0x7f5ba4066f70 0x7f5ba40aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 >> v2:172.21.15.5:3300/0 conn(0x7f5ba40b6650 msgr2=0x7f5ba40afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:31.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 --2- 172.21.15.187:0/3206316118 >> v2:172.21.15.5:3300/0 conn(0x7f5ba40b6650 0x7f5ba40afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba40b1840 con 0x7f5ba41484d0 2020-11-20T12:20:31.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5b940014e0 con 0x7f5ba41484d0 2020-11-20T12:20:31.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 >> v1:172.21.15.187:6789/0 conn(0x7f5ba41484d0 legacy=0x7f5ba4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:31.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88003800 0x7f5b88003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b880044e0 0x7f5b88004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.817 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b88004e50 0x7f5b8800d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba4067730 con 0x7f5b88003800 2020-11-20T12:20:31.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40b15a0 con 0x7f5b88004e50 2020-11-20T12:20:31.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.804+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88005380 con 0x7f5b880044e0 2020-11-20T12:20:31.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba8b21700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b88004e50 0x7f5b8800d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.818 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba3fff700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88003800 0x7f5b88003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba8b21700 1 -- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b880044e0 msgr2=0x7f5b88004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:31.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5b9b7fe700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b880044e0 0x7f5b88004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba8b21700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b880044e0 0x7f5b88004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.819 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba8b21700 1 -- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88003800 msgr2=0x7f5b88003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:31.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba8b21700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88003800 0x7f5b88003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba8b21700 1 -- 172.21.15.187:0/3206316118 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b88003240 con 0x7f5b88004e50 2020-11-20T12:20:31.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba3fff700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88003800 0x7f5b88003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:31.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.805+0000 7f5ba8b21700 1 -- 172.21.15.187:0/3206316118 reap_dead start 2020-11-20T12:20:31.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba8b21700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b88004e50 0x7f5b8800d940 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f5b9c009d80 tx=0x7f5b9c008040).ready entity=mon.1 client_cookie=99334b5008601dd0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:31.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00c070 con 0x7f5b88004e50 2020-11-20T12:20:31.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c00ea80 con 0x7f5b88004e50 2020-11-20T12:20:31.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba37fe700 1 -- 172.21.15.187:0/3206316118 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00ee20 con 0x7f5b88004e50 2020-11-20T12:20:31.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba9322700 1 -- 172.21.15.187:0/3206316118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b88004e50 msgr2=0x7f5b8800d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:31.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba9322700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b88004e50 0x7f5b8800d940 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f5b9c009d80 tx=0x7f5b9c008040).stop 2020-11-20T12:20:31.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba9322700 1 -- 172.21.15.187:0/3206316118 shutdown_connections 2020-11-20T12:20:31.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba9322700 1 --2- 172.21.15.187:0/3206316118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b88004e50 0x7f5b8800d940 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.806+0000 7f5ba9322700 1 -- 172.21.15.187:0/3206316118 >> 172.21.15.187:0/3206316118 conn(0x7f5ba40bab30 msgr2=0x7f5ba40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:31.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.807+0000 7f5ba9322700 1 -- 172.21.15.187:0/3206316118 shutdown_connections 2020-11-20T12:20:31.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.807+0000 7f5ba9322700 1 -- 172.21.15.187:0/3206316118 wait complete. 2020-11-20T12:20:31.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.808+0000 7f5ba9322700 1 Processor -- start 2020-11-20T12:20:31.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.808+0000 7f5ba9322700 1 -- start start 2020-11-20T12:20:31.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5ba9322700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88004e50 0x7f5ba40a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5ba9322700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ba409a670 0x7f5ba41577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5ba9322700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ba4065190 0x7f5ba4065610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5ba9322700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba409cf00 con 0x7f5b88004e50 2020-11-20T12:20:31.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5ba9322700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40a0960 con 0x7f5ba409a670 2020-11-20T12:20:31.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5ba9322700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba4142930 con 0x7f5ba4065190 2020-11-20T12:20:31.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5b9b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ba409a670 0x7f5ba41577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5b9b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ba409a670 0x7f5ba41577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53804/0 (socket says 172.21.15.187:53804) 2020-11-20T12:20:31.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.809+0000 7f5b9b7fe700 1 -- 172.21.15.187:0/287123309 learned_addr learned my addr 172.21.15.187:0/287123309 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:31.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5ba3fff700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88004e50 0x7f5ba40a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5ba8b21700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ba4065190 0x7f5ba4065610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5b9b7fe700 1 -- 172.21.15.187:0/287123309 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ba4065190 msgr2=0x7f5ba4065610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:31.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5b9b7fe700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ba4065190 0x7f5ba4065610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5b9b7fe700 1 -- 172.21.15.187:0/287123309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88004e50 msgr2=0x7f5ba40a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:31.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5b9b7fe700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88004e50 0x7f5ba40a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:31.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5b9b7fe700 1 -- 172.21.15.187:0/287123309 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba4157cc0 con 0x7f5ba409a670 2020-11-20T12:20:31.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5b9b7fe700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ba409a670 0x7f5ba41577d0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f5b8c00e7c0 tx=0x7f5b8c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:31.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.810+0000 7f5ba0ff9700 1 -- 172.21.15.187:0/287123309 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c010560 con 0x7f5ba409a670 2020-11-20T12:20:31.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.811+0000 7f5ba0ff9700 1 -- 172.21.15.187:0/287123309 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c0106c0 con 0x7f5ba409a670 2020-11-20T12:20:31.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.811+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba409d390 con 0x7f5ba409a670 2020-11-20T12:20:31.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.811+0000 7f5ba0ff9700 1 -- 172.21.15.187:0/287123309 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c010a40 con 0x7f5ba409a670 2020-11-20T12:20:31.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.811+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba40b9ce0 con 0x7f5ba409a670 2020-11-20T12:20:31.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.812+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba409d050 con 0x7f5ba409a670 2020-11-20T12:20:31.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.812+0000 7f5ba0ff9700 1 -- 172.21.15.187:0/287123309 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5b8c012070 con 0x7f5ba409a670 2020-11-20T12:20:31.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.813+0000 7f5ba0ff9700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b74056d20 0x7f5b740591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:31.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.813+0000 7f5ba0ff9700 1 -- 172.21.15.187:0/287123309 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5b8c06f6f0 con 0x7f5ba409a670 2020-11-20T12:20:31.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.813+0000 7f5ba3fff700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b74056d20 0x7f5b740591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:31.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.814+0000 7f5ba3fff700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b74056d20 0x7f5b740591e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f5b940013b0 tx=0x7f5b94009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:31.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:31.817+0000 7f5ba0ff9700 1 -- 172.21.15.187:0/287123309 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5b8c044ae0 con 0x7f5ba409a670 2020-11-20T12:20:32.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.064+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ba409d1e0 con 0x7f5ba409a670 2020-11-20T12:20:32.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.065+0000 7f5ba0ff9700 1 -- 172.21.15.187:0/287123309 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+2159 (secure 0 0 0) 0x7f5b8c018070 con 0x7f5ba409a670 2020-11-20T12:20:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:32.070 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":155,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:31.537179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6197":{"gid":6197,"name":"b","rank":1,"incarnation":155,"state":"up:creating","state_seq":2,"addr":"172.21.15.5:6833/1287724648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1287724648},{"type":"v1","addr":"172.21.15.5:6833","nonce":1287724648}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:32.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.067+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b74056d20 msgr2=0x7f5b740591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:32.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.067+0000 7f5ba9322700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b74056d20 0x7f5b740591e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f5b940013b0 tx=0x7f5b94009040).stop 2020-11-20T12:20:32.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ba409a670 msgr2=0x7f5ba41577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:32.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ba409a670 0x7f5ba41577d0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f5b8c00e7c0 tx=0x7f5b8c008040).stop 2020-11-20T12:20:32.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 shutdown_connections 2020-11-20T12:20:32.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5b74056d20 0x7f5b740591e0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:32.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b88004e50 0x7f5ba40a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:32.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ba4065190 0x7f5ba4065610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:32.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 --2- 172.21.15.187:0/287123309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ba409a670 0x7f5ba41577d0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:32.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 >> 172.21.15.187:0/287123309 conn(0x7f5ba40bab30 msgr2=0x7f5ba4142ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:32.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 shutdown_connections 2020-11-20T12:20:32.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:32.068+0000 7f5ba9322700 1 -- 172.21.15.187:0/287123309 wait complete. 2020-11-20T12:20:32.075 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 155 2020-11-20T12:20:32.095 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2020-11-20T12:20:34.217 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:20:34.214+0000 7f4976072700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2020-11-20T12:20:37.096 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:37.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.344+0000 7f51d3fff700 1 Processor -- start 2020-11-20T12:20:37.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.345+0000 7f51d3fff700 1 -- start start 2020-11-20T12:20:37.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.346+0000 7f51d3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f51d40b6650 0x7f51d40afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.346+0000 7f51d3fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f51d4066f70 0x7f51d40aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.346+0000 7f51d3fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40b1300 con 0x7f51d41484d0 2020-11-20T12:20:37.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.346+0000 7f51d3fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d4067730 con 0x7f51d4066f70 2020-11-20T12:20:37.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.346+0000 7f51d3fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d40b15a0 con 0x7f51d40b6650 2020-11-20T12:20:37.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.346+0000 7f51d2ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f51d41484d0 0x7f51d4061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50418/0 (socket says 172.21.15.187:50418) 2020-11-20T12:20:37.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.346+0000 7f51d2ffd700 1 -- 172.21.15.187:0/2181752523 learned_addr learned my addr 172.21.15.187:0/2181752523 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:37.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.347+0000 7f51d27fc700 1 --2- 172.21.15.187:0/2181752523 >> v2:172.21.15.187:3300/0 conn(0x7f51d40b6650 0x7f51d40afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.347+0000 7f51d1ffb700 1 -- 172.21.15.187:0/2181752523 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506934757 0 0) 0x7f51d40b1300 con 0x7f51d41484d0 2020-11-20T12:20:37.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.347+0000 7f51d1ffb700 1 -- 172.21.15.187:0/2181752523 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c0003240 con 0x7f51d41484d0 2020-11-20T12:20:37.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.347+0000 7f51d37fe700 1 --2- 172.21.15.187:0/2181752523 >> v2:172.21.15.5:3300/0 conn(0x7f51d4066f70 0x7f51d40aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.347+0000 7f51d27fc700 1 -- 172.21.15.187:0/2181752523 >> v2:172.21.15.5:3300/0 conn(0x7f51d4066f70 msgr2=0x7f51d40aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:37.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.347+0000 7f51d27fc700 1 --2- 172.21.15.187:0/2181752523 >> v2:172.21.15.5:3300/0 conn(0x7f51d4066f70 0x7f51d40aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.347+0000 7f51d27fc700 1 -- 172.21.15.187:0/2181752523 >> v1:172.21.15.187:6789/0 conn(0x7f51d41484d0 legacy=0x7f51d4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.347+0000 7f51d27fc700 1 -- 172.21.15.187:0/2181752523 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40b1840 con 0x7f51d40b6650 2020-11-20T12:20:37.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.348+0000 7f51d27fc700 1 --2- 172.21.15.187:0/2181752523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40afa90 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f51d4068600 tx=0x7f51c8006df0).ready entity=mon.1 client_cookie=83adcc8d7febdc04 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:37.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.348+0000 7f51d1ffb700 1 -- 172.21.15.187:0/2181752523 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c80097b0 con 0x7f51d40b6650 2020-11-20T12:20:37.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.348+0000 7f51d1ffb700 1 -- 172.21.15.187:0/2181752523 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8009910 con 0x7f51d40b6650 2020-11-20T12:20:37.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.348+0000 7f51d1ffb700 1 -- 172.21.15.187:0/2181752523 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8009c90 con 0x7f51d40b6650 2020-11-20T12:20:37.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.349+0000 7f51d3fff700 1 -- 172.21.15.187:0/2181752523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 msgr2=0x7f51d40afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.349+0000 7f51d3fff700 1 --2- 172.21.15.187:0/2181752523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40afa90 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f51d4068600 tx=0x7f51c8006df0).stop 2020-11-20T12:20:37.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.349+0000 7f51d3fff700 1 -- 172.21.15.187:0/2181752523 shutdown_connections 2020-11-20T12:20:37.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.349+0000 7f51d3fff700 1 --2- 172.21.15.187:0/2181752523 >> v2:172.21.15.5:3300/0 conn(0x7f51d4066f70 0x7f51d40aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.349+0000 7f51d3fff700 1 --2- 172.21.15.187:0/2181752523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40afa90 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.349+0000 7f51d3fff700 1 -- 172.21.15.187:0/2181752523 >> 172.21.15.187:0/2181752523 conn(0x7f51d40bab30 msgr2=0x7f51d40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:37.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.349+0000 7f51d3fff700 1 -- 172.21.15.187:0/2181752523 shutdown_connections 2020-11-20T12:20:37.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.349+0000 7f51d3fff700 1 -- 172.21.15.187:0/2181752523 wait complete. 2020-11-20T12:20:37.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.350+0000 7f51d3fff700 1 Processor -- start 2020-11-20T12:20:37.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.350+0000 7f51d3fff700 1 -- start start 2020-11-20T12:20:37.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.351+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51d4066f70 0x7f51d409bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.351+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.351+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f51d41484d0 0x7f51d409f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.351+0000 7f51d3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d409e740 con 0x7f51d41484d0 2020-11-20T12:20:37.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.351+0000 7f51d3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d409c490 con 0x7f51d40b6650 2020-11-20T12:20:37.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.351+0000 7f51d3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d409e950 con 0x7f51d4066f70 2020-11-20T12:20:37.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53820/0 (socket says 172.21.15.187:53820) 2020-11-20T12:20:37.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 -- 172.21.15.187:0/1674141963 learned_addr learned my addr 172.21.15.187:0/1674141963 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:37.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 -- 172.21.15.187:0/1674141963 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51d4066f70 msgr2=0x7f51d409bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51d4066f70 0x7f51d409bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:37.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 -- 172.21.15.187:0/1674141963 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f51d41484d0 msgr2=0x7f51d409f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f51d41484d0 0x7f51d409f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:37.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 -- 172.21.15.187:0/1674141963 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d409ebf0 con 0x7f51d40b6650 2020-11-20T12:20:37.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51d27fc700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40b42d0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f51c800b150 tx=0x7f51c8017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:37.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.352+0000 7f51baffd700 1 -- 172.21.15.187:0/1674141963 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8020070 con 0x7f51d40b6650 2020-11-20T12:20:37.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.353+0000 7f51baffd700 1 -- 172.21.15.187:0/1674141963 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8010dc0 con 0x7f51d40b6650 2020-11-20T12:20:37.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.353+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d40b47c0 con 0x7f51d40b6650 2020-11-20T12:20:37.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.353+0000 7f51baffd700 1 -- 172.21.15.187:0/1674141963 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c801a690 con 0x7f51d40b6650 2020-11-20T12:20:37.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.353+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d409bce0 con 0x7f51d40b6650 2020-11-20T12:20:37.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.354+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d409ed80 con 0x7f51d40b6650 2020-11-20T12:20:37.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.354+0000 7f51baffd700 1 -- 172.21.15.187:0/1674141963 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f51c801aa40 con 0x7f51d40b6650 2020-11-20T12:20:37.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.355+0000 7f51baffd700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51b0056d20 0x7f51b00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.355+0000 7f51baffd700 1 -- 172.21.15.187:0/1674141963 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f51c8073600 con 0x7f51d40b6650 2020-11-20T12:20:37.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.355+0000 7f51d2ffd700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51b0056d20 0x7f51b00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.356+0000 7f51d2ffd700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51b0056d20 0x7f51b00591e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f51c40055a0 tx=0x7f51c40076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:37.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.359+0000 7f51baffd700 1 -- 172.21.15.187:0/1674141963 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f51c8048a50 con 0x7f51d40b6650 2020-11-20T12:20:37.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.608+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51d4061a20 con 0x7f51d40b6650 2020-11-20T12:20:37.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.610+0000 7f51baffd700 1 -- 172.21.15.187:0/1674141963 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+2157 (secure 0 0 0) 0x7f51c804c070 con 0x7f51d40b6650 2020-11-20T12:20:37.614 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:37.614 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":156,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:32.541389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6197":{"gid":6197,"name":"b","rank":1,"incarnation":155,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1287724648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1287724648},{"type":"v1","addr":"172.21.15.5:6833","nonce":1287724648}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:37.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51b0056d20 msgr2=0x7f51b00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51b0056d20 0x7f51b00591e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f51c40055a0 tx=0x7f51c40076a0).stop 2020-11-20T12:20:37.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 msgr2=0x7f51d40b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40b42d0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f51c800b150 tx=0x7f51c8017000).stop 2020-11-20T12:20:37.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 shutdown_connections 2020-11-20T12:20:37.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f51b0056d20 0x7f51b00591e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f51d41484d0 0x7f51d409f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:37.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f51d4066f70 0x7f51d409bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:37.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 --2- 172.21.15.187:0/1674141963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51d40b6650 0x7f51d40b42d0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.612+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 >> 172.21.15.187:0/1674141963 conn(0x7f51d40bab30 msgr2=0x7f51d4065190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:37.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.613+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 shutdown_connections 2020-11-20T12:20:37.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.613+0000 7f51d3fff700 1 -- 172.21.15.187:0/1674141963 wait complete. 2020-11-20T12:20:37.619 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 156 2020-11-20T12:20:37.639 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:37.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.890+0000 7f8ced60f700 1 Processor -- start 2020-11-20T12:20:37.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.890+0000 7f8ced60f700 1 -- start start 2020-11-20T12:20:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.891+0000 7f8ced60f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8ce8065f90 0x7f8ce8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.891+0000 7f8ced60f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8ce809ca90 0x7f8ce809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.891+0000 7f8ced60f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce80b13e0 con 0x7f8ce8065190 2020-11-20T12:20:37.895 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.891+0000 7f8ced60f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce814ab30 con 0x7f8ce809ca90 2020-11-20T12:20:37.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.891+0000 7f8ced60f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce8062c10 con 0x7f8ce8065f90 2020-11-20T12:20:37.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.892+0000 7f8ce7fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8ce8065190 0x7f8ce8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50432/0 (socket says 172.21.15.187:50432) 2020-11-20T12:20:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.892+0000 7f8ce7fff700 1 -- 172.21.15.187:0/3184213820 learned_addr learned my addr 172.21.15.187:0/3184213820 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.892+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/3184213820 >> v2:172.21.15.187:3300/0 conn(0x7f8ce8065f90 0x7f8ce8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.892+0000 7f8cece0e700 1 --2- 172.21.15.187:0/3184213820 >> v2:172.21.15.5:3300/0 conn(0x7f8ce809ca90 0x7f8ce809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.892+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607134866 0 0) 0x7f8ce80b13e0 con 0x7f8ce8065190 2020-11-20T12:20:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.892+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd4003240 con 0x7f8ce8065190 2020-11-20T12:20:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1009706299 0 0) 0x7f8cd4003240 con 0x7f8ce8065190 2020-11-20T12:20:37.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 >> v2:172.21.15.187:3300/0 conn(0x7f8ce8065f90 msgr2=0x7f8ce8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 --2- 172.21.15.187:0/3184213820 >> v2:172.21.15.187:3300/0 conn(0x7f8ce8065f90 0x7f8ce8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 >> v2:172.21.15.5:3300/0 conn(0x7f8ce809ca90 msgr2=0x7f8ce809cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 --2- 172.21.15.187:0/3184213820 >> v2:172.21.15.5:3300/0 conn(0x7f8ce809ca90 0x7f8ce809cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce80b1640 con 0x7f8ce8065190 2020-11-20T12:20:37.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8cd80014e0 con 0x7f8ce8065190 2020-11-20T12:20:37.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/3184213820 >> v2:172.21.15.187:3300/0 conn(0x7f8ce8065f90 0x7f8ce8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:20:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 >> v1:172.21.15.187:6789/0 conn(0x7f8ce8065190 legacy=0x7f8ce8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4003800 0x7f8cd4003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cd40044e0 0x7f8cd4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cd4004e50 0x7f8cd400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce814ab30 con 0x7f8cd40044e0 2020-11-20T12:20:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce8062c10 con 0x7f8cd4004e50 2020-11-20T12:20:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd4005380 con 0x7f8cd4003800 2020-11-20T12:20:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.893+0000 7f8cece0e700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cd4004e50 0x7f8cd400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8ce7fff700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4003800 0x7f8cd4003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cd40044e0 0x7f8cd4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8cece0e700 1 -- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4003800 msgr2=0x7f8cd4003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8cece0e700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4003800 0x7f8cd4003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8cece0e700 1 -- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cd40044e0 msgr2=0x7f8cd4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:37.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8cece0e700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8cd40044e0 0x7f8cd4004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8cece0e700 1 -- 172.21.15.187:0/3184213820 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd4003240 con 0x7f8cd4004e50 2020-11-20T12:20:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8ce7fff700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4003800 0x7f8cd4003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8cece0e700 1 -- 172.21.15.187:0/3184213820 reap_dead start 2020-11-20T12:20:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8cece0e700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cd4004e50 0x7f8cd400b730 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f8ce0009ed0 tx=0x7f8ce00091d0).ready entity=mon.1 client_cookie=bf0b0fe1e257152 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:37.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce00095c0 con 0x7f8cd4004e50 2020-11-20T12:20:37.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce000ee20 con 0x7f8cd4004e50 2020-11-20T12:20:37.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.894+0000 7f8ce6ffd700 1 -- 172.21.15.187:0/3184213820 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0012600 con 0x7f8cd4004e50 2020-11-20T12:20:37.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.895+0000 7f8ced60f700 1 -- 172.21.15.187:0/3184213820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cd4004e50 msgr2=0x7f8cd400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.895+0000 7f8ced60f700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cd4004e50 0x7f8cd400b730 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f8ce0009ed0 tx=0x7f8ce00091d0).stop 2020-11-20T12:20:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.895+0000 7f8ced60f700 1 -- 172.21.15.187:0/3184213820 shutdown_connections 2020-11-20T12:20:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.895+0000 7f8ced60f700 1 --2- 172.21.15.187:0/3184213820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cd4004e50 0x7f8cd400b730 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.895+0000 7f8ced60f700 1 -- 172.21.15.187:0/3184213820 >> 172.21.15.187:0/3184213820 conn(0x7f8ce80bab30 msgr2=0x7f8ce80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.895+0000 7f8ced60f700 1 -- 172.21.15.187:0/3184213820 shutdown_connections 2020-11-20T12:20:37.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.895+0000 7f8ced60f700 1 -- 172.21.15.187:0/3184213820 wait complete. 2020-11-20T12:20:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.897+0000 7f8ced60f700 1 Processor -- start 2020-11-20T12:20:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.897+0000 7f8ced60f700 1 -- start start 2020-11-20T12:20:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ced60f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4004e50 0x7f8ce80b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ced60f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce80b2ef0 0x7f8ce80b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ced60f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80a00c0 0x7f8ce80a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ced60f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce809df10 con 0x7f8ce80a00c0 2020-11-20T12:20:37.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ced60f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce809e1b0 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ced60f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce809e450 con 0x7f8cd4004e50 2020-11-20T12:20:37.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ce77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce80b2ef0 0x7f8ce80b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ce77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce80b2ef0 0x7f8ce80b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53840/0 (socket says 172.21.15.187:53840) 2020-11-20T12:20:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.898+0000 7f8ce77fe700 1 -- 172.21.15.187:0/235665466 learned_addr learned my addr 172.21.15.187:0/235665466 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ce7fff700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4004e50 0x7f8ce80b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8cece0e700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80a00c0 0x7f8ce80a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ce77fe700 1 -- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4004e50 msgr2=0x7f8ce80b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4004e50 0x7f8ce80b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ce77fe700 1 -- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80a00c0 msgr2=0x7f8ce80a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:37.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80a00c0 0x7f8ce80a0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:37.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ce77fe700 1 -- 172.21.15.187:0/235665466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce813a860 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8cece0e700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80a00c0 0x7f8ce80a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:37.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ce77fe700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce80b2ef0 0x7f8ce80b3370 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f8cdc0067d0 tx=0x7f8cdc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:37.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8cc7fff700 1 -- 172.21.15.187:0/235665466 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc00d740 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8cc7fff700 1 -- 172.21.15.187:0/235665466 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc00d8a0 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8cc7fff700 1 -- 172.21.15.187:0/235665466 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc00dc40 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce809dae0 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.899+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce8147e60 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.900+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce809e610 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.901+0000 7f8cc7fff700 1 -- 172.21.15.187:0/235665466 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8cdc00dda0 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.901+0000 7f8cc7fff700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:37.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.902+0000 7f8cc7fff700 1 -- 172.21.15.187:0/235665466 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8cdc070290 con 0x7f8ce80b2ef0 2020-11-20T12:20:37.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.902+0000 7f8ce7fff700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:37.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.902+0000 7f8ce7fff700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f8cd8001e90 tx=0x7f8cd8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:37.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:37.905+0000 7f8cc7fff700 1 -- 172.21.15.187:0/235665466 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8cdc045660 con 0x7f8ce80b2ef0 2020-11-20T12:20:38.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.153+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ce8061a20 con 0x7f8ce80b2ef0 2020-11-20T12:20:38.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.154+0000 7f8cc7fff700 1 -- 172.21.15.187:0/235665466 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+2157 (secure 0 0 0) 0x7f8cdc017090 con 0x7f8ce80b2ef0 2020-11-20T12:20:38.158 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.158 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":156,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:32.541389+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6197":{"gid":6197,"name":"b","rank":1,"incarnation":155,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/1287724648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1287724648},{"type":"v1","addr":"172.21.15.5:6833","nonce":1287724648}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:38.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 msgr2=0x7f8cc00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:38.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f8cd8001e90 tx=0x7f8cd8009040).stop 2020-11-20T12:20:38.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce80b2ef0 msgr2=0x7f8ce80b3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:38.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce80b2ef0 0x7f8ce80b3370 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f8cdc0067d0 tx=0x7f8cdc008040).stop 2020-11-20T12:20:38.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 shutdown_connections 2020-11-20T12:20:38.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8cc0056d20 0x7f8cc00591e0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:38.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ce80a00c0 0x7f8ce80a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:38.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8cd4004e50 0x7f8ce80b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:38.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 --2- 172.21.15.187:0/235665466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce80b2ef0 0x7f8ce80b3370 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:38.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.156+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 >> 172.21.15.187:0/235665466 conn(0x7f8ce80bab30 msgr2=0x7f8ce80b9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:38.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.157+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 shutdown_connections 2020-11-20T12:20:38.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.157+0000 7f8ced60f700 1 -- 172.21.15.187:0/235665466 wait complete. 2020-11-20T12:20:38.163 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 156 2020-11-20T12:20:38.184 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and b 2020-11-20T12:20:38.185 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:20:38.198 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:20:38.400 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:20:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:20:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:20:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:20:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:20:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:20:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:20:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:20:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:20:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:20:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:20:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:20:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:20:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:20:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:20:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:20:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:20:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:20:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:20:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:20:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:20:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:20:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:20:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:20:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:20:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:20:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:20:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:20:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:20:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:20:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:20:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:20:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:20:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:20:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.410 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:20:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:20:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:20:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:20:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:20:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:20:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:20:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:20:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:20:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:20:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:20:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:20:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:20:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:20:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:20:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:20:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:20:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:20:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:20:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:20:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:20:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:20:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:20:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:20:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:20:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:20:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:20:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:20:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:20:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:20:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:20:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:20:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:20:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:20:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:20:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:20:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.420 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:20:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:20:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:20:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:20:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:20:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:20:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:20:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:20:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.422 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:20:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:20:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:20:38.423 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:20:38.423 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:20:38.423 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:20:38.423 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:20:38.423 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:20:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:20:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:20:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:20:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:20:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:38.425 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:20:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:20:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:20:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:20:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:20:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:20:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:20:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:20:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:20:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:20:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:20:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:20:38.427 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:20:38.427 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:20:38.427 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:20:38.428 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:20:38.428 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:20:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: mds_max_purge_files = 0 2020-11-20T12:20:38.428 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:20:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:20:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:20:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:20:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:20:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:20:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:20:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:20:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:20:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:20:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:20:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:20:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:20:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:20:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:20:38.432 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:20:38.432 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:20:38.432 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:20:38.432 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:20:38.433 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.433 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:20:38.433 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.433 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:20:38.433 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.434 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:20:38.434 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:20:38.434 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.434 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:20:38.434 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:20:38.435 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.435 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:20:38.435 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:20:38.435 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.435 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:20:38.436 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:20:38.436 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.436 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:20:38.436 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.436 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:20:38.437 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:20:38.437 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:20:38.437 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:20:38.437 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:20:38.437 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:20:38.438 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:20:38.438 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:20:38.438 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:20:38.438 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:20:38.438 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.439 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:20:38.439 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.439 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:20:38.439 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.439 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:20:38.440 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.440 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:20:38.440 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:20:38.440 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:20:38.440 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:20:38.441 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:20:38.441 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:20:38.441 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.441 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:20:38.441 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:20:38.442 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.442 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:20:38.442 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:20:38.442 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.442 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:20:38.443 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:20:38.443 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.443 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:20:38.443 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:20:38.443 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:20:38.444 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.444 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:20:38.444 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:20:38.444 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.445 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:20:38.445 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.445 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:20:38.445 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.445 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:20:38.446 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.446 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:20:38.446 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:20:38.446 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.446 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:20:38.447 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:20:38.447 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:20:38.447 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:20:38.447 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:20:38.447 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:20:38.448 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:20:38.448 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:20:38.448 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:20:38.448 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:20:38.448 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:20:38.449 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:20:38.449 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:20:38.449 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:20:38.449 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:20:38.449 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:20:38.450 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:20:38.450 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:20:38.450 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:20:38.450 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:20:38.450 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.451 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:20:38.451 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:20:38.451 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:20:38.451 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:20:38.451 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:20:38.452 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:20:38.452 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:20:38.452 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:20:38.452 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.452 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:20:38.453 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:20:38.453 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:20:38.453 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:20:38.453 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:20:38.453 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:20:38.454 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:20:38.454 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:20:38.454 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:20:38.454 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:20:38.454 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:20:38.455 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:20:38.455 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:20:38.455 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:20:38.455 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:20:38.455 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:20:38.455 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:20:38.456 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:20:38.456 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:20:38.456 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:20:38.456 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:20:38.456 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:20:38.457 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:20:38.457 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:20:38.457 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:20:38.457 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:20:38.457 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:20:38.458 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:20:38.458 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:20:38.458 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:20:38.458 INFO:teuthology.orchestra.run.smithi005.stdout: mds_max_purge_files = 0 2020-11-20T12:20:38.458 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:20:38.459 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:20:38.459 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:20:38.459 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:20:38.459 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:20:38.459 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:20:38.460 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:20:38.460 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:20:38.460 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:20:38.460 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:20:38.460 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:20:38.461 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:20:38.461 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:20:38.461 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:20:38.461 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:20:38.463 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:20:38.463 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:20:38.483 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:20:38.484 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2020-11-20T12:20:38.735 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.731+0000 7f3bf9e4d700 1 Processor -- start 2020-11-20T12:20:38.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.732+0000 7f3bf9e4d700 1 -- start start 2020-11-20T12:20:38.736 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf9e4d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bf4067fa0 0x7f3bf40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:38.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf9e4d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3bf4068380 0x7f3bf409c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:38.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf9e4d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3bf409cab0 0x7f3bf414c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:38.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf9e4d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf409be80 con 0x7f3bf4068380 2020-11-20T12:20:38.737 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf9e4d700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf409c120 con 0x7f3bf409cab0 2020-11-20T12:20:38.738 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf9e4d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf409bb70 con 0x7f3bf4067fa0 2020-11-20T12:20:38.739 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf964c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3bf409cab0 0x7f3bf414c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:38.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf964c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3bf409cab0 0x7f3bf414c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53850/0 (socket says 172.21.15.187:53850) 2020-11-20T12:20:38.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.733+0000 7f3bf964c700 1 -- 172.21.15.187:0/2542131711 learned_addr learned my addr 172.21.15.187:0/2542131711 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:38.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.734+0000 7f3bf964c700 1 -- 172.21.15.187:0/2542131711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bf4067fa0 msgr2=0x7f3bf40661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:38.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.734+0000 7f3bf964c700 1 --2- 172.21.15.187:0/2542131711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bf4067fa0 0x7f3bf40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:38.740 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.734+0000 7f3bf964c700 1 -- 172.21.15.187:0/2542131711 >> v2:172.21.15.5:3300/0 conn(0x7f3bf4068380 msgr2=0x7f3bf409c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:38.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.734+0000 7f3bebfff700 1 --2- 172.21.15.187:0/2542131711 >> v2:172.21.15.5:3300/0 conn(0x7f3bf4068380 0x7f3bf409c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:38.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.734+0000 7f3bf964c700 1 --2- 172.21.15.187:0/2542131711 >> v2:172.21.15.5:3300/0 conn(0x7f3bf4068380 0x7f3bf409c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:38.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.734+0000 7f3bf964c700 1 -- 172.21.15.187:0/2542131711 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf40b70c0 con 0x7f3bf409cab0 2020-11-20T12:20:38.741 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.734+0000 7f3bebfff700 1 --2- 172.21.15.187:0/2542131711 >> v2:172.21.15.5:3300/0 conn(0x7f3bf4068380 0x7f3bf409c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:38.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.734+0000 7f3bf964c700 1 --2- 172.21.15.187:0/2542131711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf409cab0 0x7f3bf414c280 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7f3bf405c370 tx=0x7f3bec009290).ready entity=mon.1 client_cookie=6a3689279bf4911c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:38.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3beb7fe700 1 -- 172.21.15.187:0/2542131711 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00bbc0 con 0x7f3bf409cab0 2020-11-20T12:20:38.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3beb7fe700 1 -- 172.21.15.187:0/2542131711 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec00bd20 con 0x7f3bf409cab0 2020-11-20T12:20:38.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3beb7fe700 1 -- 172.21.15.187:0/2542131711 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec011610 con 0x7f3bf409cab0 2020-11-20T12:20:38.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2542131711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf409cab0 msgr2=0x7f3bf414c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:38.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2542131711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf409cab0 0x7f3bf414c280 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7f3bf405c370 tx=0x7f3bec009290).stop 2020-11-20T12:20:38.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2542131711 shutdown_connections 2020-11-20T12:20:38.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2542131711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf409cab0 0x7f3bf414c280 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:38.743 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2542131711 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bf4067fa0 0x7f3bf40661c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:38.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2542131711 >> v2:172.21.15.5:3300/0 conn(0x7f3bf4068380 0x7f3bf409c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:38.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2542131711 >> 172.21.15.187:0/2542131711 conn(0x7f3bf40babc0 msgr2=0x7f3bf40bafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:38.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2542131711 shutdown_connections 2020-11-20T12:20:38.744 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.735+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2542131711 wait complete. 2020-11-20T12:20:38.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.738+0000 7f3bf9e4d700 1 Processor -- start 2020-11-20T12:20:38.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.738+0000 7f3bf9e4d700 1 -- start start 2020-11-20T12:20:38.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf9e4d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf4067fa0 0x7f3bf4099520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:38.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf9e4d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bf4068380 0x7f3bf414cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:38.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf9e4d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bf409cab0 0x7f3bf41474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:38.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf9e4d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf414d5b0 con 0x7f3bf4068380 2020-11-20T12:20:38.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf9e4d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf414d850 con 0x7f3bf4067fa0 2020-11-20T12:20:38.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf9e4d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf414daf0 con 0x7f3bf409cab0 2020-11-20T12:20:38.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf8e4b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf4067fa0 0x7f3bf4099520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:38.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf8e4b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf4067fa0 0x7f3bf4099520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53852/0 (socket says 172.21.15.187:53852) 2020-11-20T12:20:38.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.739+0000 7f3bf8e4b700 1 -- 172.21.15.187:0/2616249836 learned_addr learned my addr 172.21.15.187:0/2616249836 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:38.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3bebfff700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bf4068380 0x7f3bf414cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:38.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3bf8e4b700 1 -- 172.21.15.187:0/2616249836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bf409cab0 msgr2=0x7f3bf41474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:38.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3bf8e4b700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bf409cab0 0x7f3bf41474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:38.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3bf8e4b700 1 -- 172.21.15.187:0/2616249836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bf4068380 msgr2=0x7f3bf414cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:38.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3bf8e4b700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bf4068380 0x7f3bf414cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:38.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3bf8e4b700 1 -- 172.21.15.187:0/2616249836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4099a10 con 0x7f3bf4067fa0 2020-11-20T12:20:38.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3bf8e4b700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf4067fa0 0x7f3bf4099520 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7f3bf0009770 tx=0x7f3bf0006ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:38.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3be8ff9700 1 -- 172.21.15.187:0/2616249836 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000d640 con 0x7f3bf4067fa0 2020-11-20T12:20:38.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.740+0000 7f3be8ff9700 1 -- 172.21.15.187:0/2616249836 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0000e90 con 0x7f3bf4067fa0 2020-11-20T12:20:38.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.741+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf409e690 con 0x7f3bf4067fa0 2020-11-20T12:20:38.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.741+0000 7f3be8ff9700 1 -- 172.21.15.187:0/2616249836 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000da10 con 0x7f3bf4067fa0 2020-11-20T12:20:38.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.741+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf41479e0 con 0x7f3bf4067fa0 2020-11-20T12:20:38.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.742+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf414d3d0 con 0x7f3bf4067fa0 2020-11-20T12:20:38.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.742+0000 7f3be8ff9700 1 -- 172.21.15.187:0/2616249836 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3bf0017030 con 0x7f3bf4067fa0 2020-11-20T12:20:38.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.743+0000 7f3be8ff9700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bcc056d20 0x7f3bcc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:38.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.743+0000 7f3be8ff9700 1 -- 172.21.15.187:0/2616249836 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3bf006f100 con 0x7f3bf4067fa0 2020-11-20T12:20:38.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.744+0000 7f3bebfff700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bcc056d20 0x7f3bcc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:38.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.744+0000 7f3bebfff700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bcc056d20 0x7f3bcc0591e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f3bf415fb10 tx=0x7f3be4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:38.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:38.747+0000 7f3be8ff9700 1 -- 172.21.15.187:0/2616249836 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3bf0044600 con 0x7f3bf4067fa0 2020-11-20T12:20:39.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.003+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f3bf414dcb0 con 0x7f3bf4067fa0 2020-11-20T12:20:39.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.905+0000 7f3be8ff9700 1 -- 172.21.15.187:0/2616249836 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6197 v157) v1 ==== 95+0+0 (secure 0 0 0) 0x7f3bf0047c20 con 0x7f3bf4067fa0 2020-11-20T12:20:39.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.908+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bcc056d20 msgr2=0x7f3bcc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:39.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bcc056d20 0x7f3bcc0591e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f3bf415fb10 tx=0x7f3be4008040).stop 2020-11-20T12:20:39.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf4067fa0 msgr2=0x7f3bf4099520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:39.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf4067fa0 0x7f3bf4099520 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7f3bf0009770 tx=0x7f3bf0006ec0).stop 2020-11-20T12:20:39.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 shutdown_connections 2020-11-20T12:20:39.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3bcc056d20 0x7f3bcc0591e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:39.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3bf409cab0 0x7f3bf41474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:39.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3bf4067fa0 0x7f3bf4099520 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:39.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 --2- 172.21.15.187:0/2616249836 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3bf4068380 0x7f3bf414cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:39.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 >> 172.21.15.187:0/2616249836 conn(0x7f3bf40babc0 msgr2=0x7f3bf40b8980 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:39.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 shutdown_connections 2020-11-20T12:20:39.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:39.909+0000 7f3bf9e4d700 1 -- 172.21.15.187:0/2616249836 wait complete. 2020-11-20T12:20:39.915 INFO:teuthology.orchestra.run.smithi187.stderr:failed mds gid 6197 2020-11-20T12:20:39.937 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:20:39.938 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:20:39.946 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:20:39.947 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:40.144 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:20:40.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.185+0000 7ff612621700 1 Processor -- start 2020-11-20T12:20:40.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.185+0000 7ff612621700 1 -- start start 2020-11-20T12:20:40.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.186+0000 7ff612621700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff60c065880 0x7ff60c0b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:40.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.186+0000 7ff612621700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c0b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:40.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.186+0000 7ff612621700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60c14c050 con 0x7ff60c0654a0 2020-11-20T12:20:40.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.186+0000 7ff612621700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c14cea0 con 0x7ff60c065880 2020-11-20T12:20:40.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.186+0000 7ff612621700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c14c2f0 con 0x7ff60c0b5190 2020-11-20T12:20:40.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.187+0000 7ff61161f700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff60c0654a0 0x7ff60c14c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:43980/0 (socket says 172.21.15.187:43980) 2020-11-20T12:20:40.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.187+0000 7ff61161f700 1 -- 172.21.15.187:0/4178491708 learned_addr learned my addr 172.21.15.187:0/4178491708 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:40.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.187+0000 7ff611e20700 1 --2- 172.21.15.187:0/4178491708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c0b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:40.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.187+0000 7ff610e1e700 1 --2- 172.21.15.187:0/4178491708 >> v2:172.21.15.5:3300/0 conn(0x7ff60c065880 0x7ff60c0b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:40.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.188+0000 7ff611e20700 1 -- 172.21.15.187:0/4178491708 >> v2:172.21.15.5:3300/0 conn(0x7ff60c065880 msgr2=0x7ff60c0b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:40.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.188+0000 7ff611e20700 1 --2- 172.21.15.187:0/4178491708 >> v2:172.21.15.5:3300/0 conn(0x7ff60c065880 0x7ff60c0b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.188+0000 7ff611e20700 1 -- 172.21.15.187:0/4178491708 >> v1:172.21.15.5:6789/0 conn(0x7ff60c0654a0 legacy=0x7ff60c14c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:40.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.188+0000 7ff611e20700 1 -- 172.21.15.187:0/4178491708 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c066400 con 0x7ff60c0b5190 2020-11-20T12:20:40.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.188+0000 7ff611e20700 1 --2- 172.21.15.187:0/4178491708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c0b6ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff604009c90 tx=0x7ff6040091d0).ready entity=mon.2 client_cookie=678ec9c780891ca3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:40.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff5fbfff700 1 -- 172.21.15.187:0/4178491708 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60400bc50 con 0x7ff60c0b5190 2020-11-20T12:20:40.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff5fbfff700 1 -- 172.21.15.187:0/4178491708 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60400bdb0 con 0x7ff60c0b5190 2020-11-20T12:20:40.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff5fbfff700 1 -- 172.21.15.187:0/4178491708 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604011610 con 0x7ff60c0b5190 2020-11-20T12:20:40.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff612621700 1 -- 172.21.15.187:0/4178491708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 msgr2=0x7ff60c0b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:40.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff612621700 1 --2- 172.21.15.187:0/4178491708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c0b6ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff604009c90 tx=0x7ff6040091d0).stop 2020-11-20T12:20:40.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff612621700 1 -- 172.21.15.187:0/4178491708 shutdown_connections 2020-11-20T12:20:40.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff612621700 1 --2- 172.21.15.187:0/4178491708 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c0b6ab0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff612621700 1 --2- 172.21.15.187:0/4178491708 >> v2:172.21.15.5:3300/0 conn(0x7ff60c065880 0x7ff60c0b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.189+0000 7ff612621700 1 -- 172.21.15.187:0/4178491708 >> 172.21.15.187:0/4178491708 conn(0x7ff60c0bab90 msgr2=0x7ff60c0bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:40.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.190+0000 7ff612621700 1 -- 172.21.15.187:0/4178491708 shutdown_connections 2020-11-20T12:20:40.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.190+0000 7ff612621700 1 -- 172.21.15.187:0/4178491708 wait complete. 2020-11-20T12:20:40.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.192+0000 7ff612621700 1 Processor -- start 2020-11-20T12:20:40.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.192+0000 7ff612621700 1 -- start start 2020-11-20T12:20:40.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff612621700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff60c0654a0 0x7ff60c09f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:40.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff612621700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff60c065880 0x7ff60c09a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:40.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff612621700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:40.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff612621700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c09cc50 con 0x7ff60c0654a0 2020-11-20T12:20:40.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff612621700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c09ce40 con 0x7ff60c065880 2020-11-20T12:20:40.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff612621700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c09d0a0 con 0x7ff60c0b5190 2020-11-20T12:20:40.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff610e1e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff60c065880 0x7ff60c09a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:40.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff610e1e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff60c065880 0x7ff60c09a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53872/0 (socket says 172.21.15.187:53872) 2020-11-20T12:20:40.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff610e1e700 1 -- 172.21.15.187:0/2626221207 learned_addr learned my addr 172.21.15.187:0/2626221207 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:40.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.193+0000 7ff61161f700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff60c0654a0 0x7ff60c09f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:40.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff611e20700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:40.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff610e1e700 1 -- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 msgr2=0x7ff60c147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:40.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff610e1e700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff610e1e700 1 -- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff60c0654a0 msgr2=0x7ff60c09f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:40.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff610e1e700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff60c0654a0 0x7ff60c09f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff610e1e700 1 -- 172.21.15.187:0/2626221207 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c09d340 con 0x7ff60c065880 2020-11-20T12:20:40.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff611e20700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:40.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff610e1e700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff60c065880 0x7ff60c09a5e0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7ff5fc00db60 tx=0x7ff5fc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:40.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff5f97fa700 1 -- 172.21.15.187:0/2626221207 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc009820 con 0x7ff60c065880 2020-11-20T12:20:40.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff5f97fa700 1 -- 172.21.15.187:0/2626221207 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5fc009980 con 0x7ff60c065880 2020-11-20T12:20:40.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c09fed0 con 0x7ff60c065880 2020-11-20T12:20:40.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.194+0000 7ff5f97fa700 1 -- 172.21.15.187:0/2626221207 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc009d20 con 0x7ff60c065880 2020-11-20T12:20:40.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.195+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c0b3450 con 0x7ff60c065880 2020-11-20T12:20:40.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.196+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c061ad0 con 0x7ff60c065880 2020-11-20T12:20:40.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.196+0000 7ff5f97fa700 1 -- 172.21.15.187:0/2626221207 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff5fc013690 con 0x7ff60c065880 2020-11-20T12:20:40.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.197+0000 7ff5f97fa700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5e4056d20 0x7ff5e40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:40.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.197+0000 7ff5f97fa700 1 -- 172.21.15.187:0/2626221207 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff5fc070790 con 0x7ff60c065880 2020-11-20T12:20:40.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.197+0000 7ff61161f700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5e4056d20 0x7ff5e40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:40.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.198+0000 7ff61161f700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5e4056d20 0x7ff5e40591e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7ff6080014b0 tx=0x7ff608009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:40.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.201+0000 7ff5f97fa700 1 -- 172.21.15.187:0/2626221207 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff5fc045a20 con 0x7ff60c065880 2020-11-20T12:20:40.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.450+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff60c0ba8f0 con 0x7ff60c065880 2020-11-20T12:20:40.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.452+0000 7ff5f97fa700 1 -- 172.21.15.187:0/2626221207 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+1797 (secure 0 0 0) 0x7ff5fc049040 con 0x7ff60c065880 2020-11-20T12:20:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:40.456 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":157,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:39.891429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661},"failed":[1],"damaged":[],"stopped":[],"info":{"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:40.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5e4056d20 msgr2=0x7ff5e40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:40.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5e4056d20 0x7ff5e40591e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7ff6080014b0 tx=0x7ff608009040).stop 2020-11-20T12:20:40.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff60c065880 msgr2=0x7ff60c09a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:40.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff60c065880 0x7ff60c09a5e0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7ff5fc00db60 tx=0x7ff5fc008040).stop 2020-11-20T12:20:40.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 shutdown_connections 2020-11-20T12:20:40.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff5e4056d20 0x7ff5e40591e0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff60c0654a0 0x7ff60c09f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff60c0b5190 0x7ff60c147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 --2- 172.21.15.187:0/2626221207 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff60c065880 0x7ff60c09a5e0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:40.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 >> 172.21.15.187:0/2626221207 conn(0x7ff60c0bab90 msgr2=0x7ff60c062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:40.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 shutdown_connections 2020-11-20T12:20:40.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:40.454+0000 7ff612621700 1 -- 172.21.15.187:0/2626221207 wait complete. 2020-11-20T12:20:40.461 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 157 2020-11-20T12:20:40.683 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 157, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:20:21.071872+0000', 'modified': '2020-11-20T12:20:39.891429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 151, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9661}, 'failed': [1], 'damaged': [], 'stopped': [], 'info': {'gid_9661': {'gid': 9661, 'name': 'a', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3618925546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3618925546}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3618925546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:20:40.684 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2020-11-20T12:20:41.685 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:41.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.948+0000 7fca15767700 1 Processor -- start 2020-11-20T12:20:41.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.948+0000 7fca15767700 1 -- start start 2020-11-20T12:20:41.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca15767700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10062c50 0x7fca10064160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:41.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca15767700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fca10063030 0x7fca100678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:41.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca15767700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fca10067da0 0x7fca1014c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:41.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca15767700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca10068210 con 0x7fca10067da0 2020-11-20T12:20:41.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca15767700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca10065090 con 0x7fca10063030 2020-11-20T12:20:41.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca15767700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca10064a70 con 0x7fca10062c50 2020-11-20T12:20:41.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca0e7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fca10063030 0x7fca100678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:41.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca0e7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fca10063030 0x7fca100678b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53880/0 (socket says 172.21.15.187:53880) 2020-11-20T12:20:41.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca0e7fc700 1 -- 172.21.15.187:0/2390097486 learned_addr learned my addr 172.21.15.187:0/2390097486 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:41.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca0effd700 1 --2- 172.21.15.187:0/2390097486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10062c50 0x7fca10064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:41.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.950+0000 7fca0f7fe700 1 --2- 172.21.15.187:0/2390097486 >> v2:172.21.15.5:3300/0 conn(0x7fca10067da0 0x7fca1014c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:41.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0e7fc700 1 -- 172.21.15.187:0/2390097486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10062c50 msgr2=0x7fca10064160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:41.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0e7fc700 1 --2- 172.21.15.187:0/2390097486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10062c50 0x7fca10064160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:41.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0e7fc700 1 -- 172.21.15.187:0/2390097486 >> v2:172.21.15.5:3300/0 conn(0x7fca10067da0 msgr2=0x7fca1014c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:41.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0e7fc700 1 --2- 172.21.15.187:0/2390097486 >> v2:172.21.15.5:3300/0 conn(0x7fca10067da0 0x7fca1014c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:41.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0e7fc700 1 -- 172.21.15.187:0/2390097486 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10064d10 con 0x7fca10063030 2020-11-20T12:20:41.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0e7fc700 1 --2- 172.21.15.187:0/2390097486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10063030 0x7fca100678b0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7fca1005c3d0 tx=0x7fca04006eb0).ready entity=mon.1 client_cookie=2617ba102712a2ff server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:41.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0dffb700 1 -- 172.21.15.187:0/2390097486 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca040098d0 con 0x7fca10063030 2020-11-20T12:20:41.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0dffb700 1 -- 172.21.15.187:0/2390097486 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04009a30 con 0x7fca10063030 2020-11-20T12:20:41.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.951+0000 7fca0dffb700 1 -- 172.21.15.187:0/2390097486 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04009db0 con 0x7fca10063030 2020-11-20T12:20:41.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 -- 172.21.15.187:0/2390097486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10063030 msgr2=0x7fca100678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:41.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 --2- 172.21.15.187:0/2390097486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10063030 0x7fca100678b0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7fca1005c3d0 tx=0x7fca04006eb0).stop 2020-11-20T12:20:41.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 -- 172.21.15.187:0/2390097486 shutdown_connections 2020-11-20T12:20:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 --2- 172.21.15.187:0/2390097486 >> v2:172.21.15.5:3300/0 conn(0x7fca10067da0 0x7fca1014c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 --2- 172.21.15.187:0/2390097486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10062c50 0x7fca10064160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 --2- 172.21.15.187:0/2390097486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10063030 0x7fca100678b0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:41.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 -- 172.21.15.187:0/2390097486 >> 172.21.15.187:0/2390097486 conn(0x7fca100bcfb0 msgr2=0x7fca100bd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:41.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 -- 172.21.15.187:0/2390097486 shutdown_connections 2020-11-20T12:20:41.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.952+0000 7fca15767700 1 -- 172.21.15.187:0/2390097486 wait complete. 2020-11-20T12:20:41.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.954+0000 7fca15767700 1 Processor -- start 2020-11-20T12:20:41.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.954+0000 7fca15767700 1 -- start start 2020-11-20T12:20:41.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.955+0000 7fca15767700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10062c50 0x7fca1015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:41.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca15767700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca10063030 0x7fca10147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:41.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca15767700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10067da0 0x7fca10157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:41.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca15767700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1015c870 con 0x7fca10063030 2020-11-20T12:20:41.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca15767700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca101586b0 con 0x7fca10062c50 2020-11-20T12:20:41.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca15767700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca10158950 con 0x7fca10067da0 2020-11-20T12:20:41.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10062c50 0x7fca1015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:41.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0effd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10062c50 0x7fca1015c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53884/0 (socket says 172.21.15.187:53884) 2020-11-20T12:20:41.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0effd700 1 -- 172.21.15.187:0/1020824134 learned_addr learned my addr 172.21.15.187:0/1020824134 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:41.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0effd700 1 -- 172.21.15.187:0/1020824134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10067da0 msgr2=0x7fca10157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:41.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0effd700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10067da0 0x7fca10157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:41.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0effd700 1 -- 172.21.15.187:0/1020824134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca10063030 msgr2=0x7fca10147860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:41.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0e7fc700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca10063030 0x7fca10147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:41.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0effd700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca10063030 0x7fca10147860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:41.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.956+0000 7fca0effd700 1 -- 172.21.15.187:0/1020824134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10158bf0 con 0x7fca10062c50 2020-11-20T12:20:41.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.957+0000 7fca0effd700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10062c50 0x7fca1015c380 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7fca0000db60 tx=0x7fca00008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:41.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.957+0000 7fc9f6ffd700 1 -- 172.21.15.187:0/1020824134 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca00009790 con 0x7fca10062c50 2020-11-20T12:20:41.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.957+0000 7fc9f6ffd700 1 -- 172.21.15.187:0/1020824134 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca000098f0 con 0x7fca10062c50 2020-11-20T12:20:41.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.957+0000 7fc9f6ffd700 1 -- 172.21.15.187:0/1020824134 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca00009c90 con 0x7fca10062c50 2020-11-20T12:20:41.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.957+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1009ed60 con 0x7fca10062c50 2020-11-20T12:20:41.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.957+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca10159250 con 0x7fca10062c50 2020-11-20T12:20:41.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.958+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca10063a40 con 0x7fca10062c50 2020-11-20T12:20:41.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.959+0000 7fc9f6ffd700 1 -- 172.21.15.187:0/1020824134 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fca00013480 con 0x7fca10062c50 2020-11-20T12:20:41.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.960+0000 7fc9f6ffd700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9ec056d20 0x7fc9ec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:41.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.960+0000 7fc9f6ffd700 1 -- 172.21.15.187:0/1020824134 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fca00070620 con 0x7fca10062c50 2020-11-20T12:20:41.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.960+0000 7fca0e7fc700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9ec056d20 0x7fc9ec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:41.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.961+0000 7fca0e7fc700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9ec056d20 0x7fc9ec0591e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7fca100627c0 tx=0x7fca04015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:41.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:41.963+0000 7fc9f6ffd700 1 -- 172.21.15.187:0/1020824134 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fca000459c0 con 0x7fca10062c50 2020-11-20T12:20:42.209 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.206+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca10065e30 con 0x7fca10062c50 2020-11-20T12:20:42.210 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.207+0000 7fc9f6ffd700 1 -- 172.21.15.187:0/1020824134 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+1797 (secure 0 0 0) 0x7fca00048fe0 con 0x7fca10062c50 2020-11-20T12:20:42.210 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:42.211 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":157,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:39.891429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661},"failed":[1],"damaged":[],"stopped":[],"info":{"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:42.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9ec056d20 msgr2=0x7fc9ec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:42.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9ec056d20 0x7fc9ec0591e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7fca100627c0 tx=0x7fca04015000).stop 2020-11-20T12:20:42.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10062c50 msgr2=0x7fca1015c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:42.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10062c50 0x7fca1015c380 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7fca0000db60 tx=0x7fca00008040).stop 2020-11-20T12:20:42.213 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 shutdown_connections 2020-11-20T12:20:42.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc9ec056d20 0x7fc9ec0591e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:42.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca10067da0 0x7fca10157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:42.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca10062c50 0x7fca1015c380 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:42.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 --2- 172.21.15.187:0/1020824134 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca10063030 0x7fca10147860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:42.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 >> 172.21.15.187:0/1020824134 conn(0x7fca100bcfb0 msgr2=0x7fca100bb180 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:42.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 shutdown_connections 2020-11-20T12:20:42.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:42.209+0000 7fca15767700 1 -- 172.21.15.187:0/1020824134 wait complete. 2020-11-20T12:20:42.215 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 157 2020-11-20T12:20:42.438 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 157, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:20:21.071872+0000', 'modified': '2020-11-20T12:20:39.891429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 151, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9661}, 'failed': [1], 'damaged': [], 'stopped': [], 'info': {'gid_9661': {'gid': 9661, 'name': 'a', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3618925546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3618925546}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3618925546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:20:42.439 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2020-11-20T12:20:43.440 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:44.947 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.944+0000 7f3264a28700 1 Processor -- start 2020-11-20T12:20:44.948 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.944+0000 7f3264a28700 1 -- start start 2020-11-20T12:20:44.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.945+0000 7f3264a28700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32600654a0 0x7f3260149290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.945+0000 7f3264a28700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3260065880 0x7f3260149780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.945+0000 7f3264a28700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326009a930 con 0x7f32600b1110 2020-11-20T12:20:44.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.945+0000 7f3264a28700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32600b0be0 con 0x7f3260065880 2020-11-20T12:20:44.950 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.945+0000 7f3264a28700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32600b71b0 con 0x7f32600654a0 2020-11-20T12:20:44.951 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.946+0000 7f325ed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32600654a0 0x7f3260149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.946+0000 7f325f59e700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f32600b1110 0x7f3260061d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50502/0 (socket says 172.21.15.187:50502) 2020-11-20T12:20:44.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.946+0000 7f325f59e700 1 -- 172.21.15.187:0/3793463951 learned_addr learned my addr 172.21.15.187:0/3793463951 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:44.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.946+0000 7f325e59c700 1 --2- 172.21.15.187:0/3793463951 >> v2:172.21.15.5:3300/0 conn(0x7f3260065880 0x7f3260149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.947+0000 7f325ed9d700 1 -- 172.21.15.187:0/3793463951 >> v2:172.21.15.5:3300/0 conn(0x7f3260065880 msgr2=0x7f3260149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:44.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.947+0000 7f325ed9d700 1 --2- 172.21.15.187:0/3793463951 >> v2:172.21.15.5:3300/0 conn(0x7f3260065880 0x7f3260149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.947+0000 7f325ed9d700 1 -- 172.21.15.187:0/3793463951 >> v1:172.21.15.187:6789/0 conn(0x7f32600b1110 legacy=0x7f3260061d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.947+0000 7f325ed9d700 1 -- 172.21.15.187:0/3793463951 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600662d0 con 0x7f32600654a0 2020-11-20T12:20:44.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.947+0000 7f325dd9b700 1 -- 172.21.15.187:0/3793463951 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772311413 0 0) 0x7f326009a930 con 0x7f32600b1110 2020-11-20T12:20:44.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.947+0000 7f325ed9d700 1 --2- 172.21.15.187:0/3793463951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32600654a0 0x7f3260149290 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7f32600692c0 tx=0x7f3250006df0).ready entity=mon.1 client_cookie=530153e62c524e39 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:44.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.947+0000 7f325dd9b700 1 -- 172.21.15.187:0/3793463951 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250014b70 con 0x7f32600654a0 2020-11-20T12:20:44.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f325dd9b700 1 -- 172.21.15.187:0/3793463951 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3250014cd0 con 0x7f32600654a0 2020-11-20T12:20:44.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f325dd9b700 1 -- 172.21.15.187:0/3793463951 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250014e40 con 0x7f32600654a0 2020-11-20T12:20:44.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f3264a28700 1 -- 172.21.15.187:0/3793463951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32600654a0 msgr2=0x7f3260149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f3264a28700 1 --2- 172.21.15.187:0/3793463951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32600654a0 0x7f3260149290 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7f32600692c0 tx=0x7f3250006df0).stop 2020-11-20T12:20:44.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f3264a28700 1 -- 172.21.15.187:0/3793463951 shutdown_connections 2020-11-20T12:20:44.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f3264a28700 1 --2- 172.21.15.187:0/3793463951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32600654a0 0x7f3260149290 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.955 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f3264a28700 1 --2- 172.21.15.187:0/3793463951 >> v2:172.21.15.5:3300/0 conn(0x7f3260065880 0x7f3260149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f3264a28700 1 -- 172.21.15.187:0/3793463951 >> 172.21.15.187:0/3793463951 conn(0x7f32600bab90 msgr2=0x7f32600bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:44.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f3264a28700 1 -- 172.21.15.187:0/3793463951 shutdown_connections 2020-11-20T12:20:44.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.948+0000 7f3264a28700 1 -- 172.21.15.187:0/3793463951 wait complete. 2020-11-20T12:20:44.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.950+0000 7f3264a28700 1 Processor -- start 2020-11-20T12:20:44.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.950+0000 7f3264a28700 1 -- start start 2020-11-20T12:20:44.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f3264a28700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32600654a0 0x7f32600b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f3264a28700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3260065880 0x7f3260153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f3264a28700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32600b1110 0x7f3260154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f3264a28700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32600b2900 con 0x7f32600654a0 2020-11-20T12:20:44.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f3264a28700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32600b2ba0 con 0x7f3260065880 2020-11-20T12:20:44.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f3264a28700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32600b2e40 con 0x7f32600b1110 2020-11-20T12:20:44.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f325e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3260065880 0x7f3260153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f325e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3260065880 0x7f3260153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53900/0 (socket says 172.21.15.187:53900) 2020-11-20T12:20:44.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.951+0000 7f325e59c700 1 -- 172.21.15.187:0/289447658 learned_addr learned my addr 172.21.15.187:0/289447658 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:44.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325ed9d700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32600654a0 0x7f32600b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325f59e700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32600b1110 0x7f3260154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325e59c700 1 -- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32600b1110 msgr2=0x7f3260154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325e59c700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32600b1110 0x7f3260154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325e59c700 1 -- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32600654a0 msgr2=0x7f32600b39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325e59c700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32600654a0 0x7f32600b39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325e59c700 1 -- 172.21.15.187:0/289447658 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600b25f0 con 0x7f3260065880 2020-11-20T12:20:44.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325ed9d700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32600654a0 0x7f32600b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:44.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f325e59c700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3260065880 0x7f3260153c80 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f325400be60 tx=0x7f325400f040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:44.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f3246ffd700 1 -- 172.21.15.187:0/289447658 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325400d800 con 0x7f3260065880 2020-11-20T12:20:44.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f3246ffd700 1 -- 172.21.15.187:0/289447658 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325400f900 con 0x7f3260065880 2020-11-20T12:20:44.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32600b1e70 con 0x7f3260065880 2020-11-20T12:20:44.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f3246ffd700 1 -- 172.21.15.187:0/289447658 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325400fc80 con 0x7f3260065880 2020-11-20T12:20:44.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.952+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32601548a0 con 0x7f3260065880 2020-11-20T12:20:44.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.953+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3260061ad0 con 0x7f3260065880 2020-11-20T12:20:44.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.954+0000 7f3246ffd700 1 -- 172.21.15.187:0/289447658 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3254014070 con 0x7f3260065880 2020-11-20T12:20:44.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.955+0000 7f3246ffd700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f323c056d20 0x7f323c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.955+0000 7f3246ffd700 1 -- 172.21.15.187:0/289447658 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f325406fc60 con 0x7f3260065880 2020-11-20T12:20:44.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.955+0000 7f325ed9d700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f323c056d20 0x7f323c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.956+0000 7f325ed9d700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f323c056d20 0x7f323c0591e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f3250006e30 tx=0x7f325001b000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:44.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:44.959+0000 7f3246ffd700 1 -- 172.21.15.187:0/289447658 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3254045060 con 0x7f3260065880 2020-11-20T12:20:45.211 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.207+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f326009ab30 con 0x7f3260065880 2020-11-20T12:20:45.212 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.208+0000 7f3246ffd700 1 -- 172.21.15.187:0/289447658 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2156 (secure 0 0 0) 0x7f3254048680 con 0x7f3260065880 2020-11-20T12:20:45.212 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:45.212 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":158,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:44.513184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:replay","state_seq":2,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:45.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f323c056d20 msgr2=0x7f323c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:45.214 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f323c056d20 0x7f323c0591e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f3250006e30 tx=0x7f325001b000).stop 2020-11-20T12:20:45.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3260065880 msgr2=0x7f3260153c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:45.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3260065880 0x7f3260153c80 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f325400be60 tx=0x7f325400f040).stop 2020-11-20T12:20:45.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 shutdown_connections 2020-11-20T12:20:45.215 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f323c056d20 0x7f323c0591e0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:45.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f32600654a0 0x7f32600b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:45.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f32600b1110 0x7f3260154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:45.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 --2- 172.21.15.187:0/289447658 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3260065880 0x7f3260153c80 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:45.216 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 >> 172.21.15.187:0/289447658 conn(0x7f32600bab90 msgr2=0x7f32600b7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:45.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 shutdown_connections 2020-11-20T12:20:45.217 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:45.211+0000 7f3264a28700 1 -- 172.21.15.187:0/289447658 wait complete. 2020-11-20T12:20:45.217 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 158 2020-11-20T12:20:45.442 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 158, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:20:21.071872+0000', 'modified': '2020-11-20T12:20:44.513184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 151, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9661, 'mds_1': 6281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6281': {'gid': 6281, 'name': 'b', 'rank': 1, 'incarnation': 158, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.5:6833/191681417', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 191681417}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 191681417}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_9661': {'gid': 9661, 'name': 'a', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3618925546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3618925546}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3618925546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:20:45.443 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6281:up:replay 2020-11-20T12:20:46.444 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:46.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.693+0000 7f0885a3a700 1 Processor -- start 2020-11-20T12:20:46.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.694+0000 7f0885a3a700 1 -- start start 2020-11-20T12:20:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.695+0000 7f0885a3a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0880065880 0x7f08800675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.695+0000 7f0885a3a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0880099f70 con 0x7f08800670c0 2020-11-20T12:20:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.695+0000 7f0885a3a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08800b6a20 con 0x7f08800654a0 2020-11-20T12:20:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.695+0000 7f0885a3a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0880148990 con 0x7f0880065880 2020-11-20T12:20:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.695+0000 7f0884a38700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f08800654a0 0x7f0880148280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50514/0 (socket says 172.21.15.187:50514) 2020-11-20T12:20:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.695+0000 7f0884a38700 1 -- 172.21.15.187:0/1631015273 learned_addr learned my addr 172.21.15.187:0/1631015273 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.696+0000 7f0877fff700 1 --2- 172.21.15.187:0/1631015273 >> v2:172.21.15.187:3300/0 conn(0x7f0880065880 0x7f08800675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.696+0000 7f08777fe700 1 -- 172.21.15.187:0/1631015273 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070255790 0 0) 0x7f08800b6a20 con 0x7f08800654a0 2020-11-20T12:20:46.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.696+0000 7f08777fe700 1 -- 172.21.15.187:0/1631015273 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c003240 con 0x7f08800654a0 2020-11-20T12:20:46.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.697+0000 7f0877fff700 1 -- 172.21.15.187:0/1631015273 >> v1:172.21.15.187:6789/0 conn(0x7f08800654a0 legacy=0x7f0880148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.697+0000 7f08777fe700 1 -- 172.21.15.187:0/1631015273 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4251528507 0 0) 0x7f086c003240 con 0x7f08800654a0 2020-11-20T12:20:46.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.697+0000 7f0877fff700 1 -- 172.21.15.187:0/1631015273 >> v1:172.21.15.5:6789/0 conn(0x7f08800670c0 legacy=0x7f08800afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.697+0000 7f0877fff700 1 -- 172.21.15.187:0/1631015273 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08800b0200 con 0x7f0880065880 2020-11-20T12:20:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.697+0000 7f0877fff700 1 --2- 172.21.15.187:0/1631015273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880065880 0x7f08800675e0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f087000b920 tx=0x7f0870006df0).ready entity=mon.1 client_cookie=17ce6eae844fa403 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.697+0000 7f08777fe700 1 -- 172.21.15.187:0/1631015273 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c001860 con 0x7f0880065880 2020-11-20T12:20:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.697+0000 7f08777fe700 1 -- 172.21.15.187:0/1631015273 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0870011400 con 0x7f0880065880 2020-11-20T12:20:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.697+0000 7f08777fe700 1 -- 172.21.15.187:0/1631015273 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870011780 con 0x7f0880065880 2020-11-20T12:20:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.698+0000 7f0885a3a700 1 -- 172.21.15.187:0/1631015273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880065880 msgr2=0x7f08800675e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.698+0000 7f0885a3a700 1 --2- 172.21.15.187:0/1631015273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880065880 0x7f08800675e0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f087000b920 tx=0x7f0870006df0).stop 2020-11-20T12:20:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.698+0000 7f0885a3a700 1 -- 172.21.15.187:0/1631015273 shutdown_connections 2020-11-20T12:20:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.698+0000 7f0885a3a700 1 --2- 172.21.15.187:0/1631015273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880065880 0x7f08800675e0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.698+0000 7f0885a3a700 1 -- 172.21.15.187:0/1631015273 >> 172.21.15.187:0/1631015273 conn(0x7f08800bab90 msgr2=0x7f08800bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:46.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.698+0000 7f0885a3a700 1 -- 172.21.15.187:0/1631015273 shutdown_connections 2020-11-20T12:20:46.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.698+0000 7f0885a3a700 1 -- 172.21.15.187:0/1631015273 wait complete. 2020-11-20T12:20:46.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.700+0000 7f0885a3a700 1 Processor -- start 2020-11-20T12:20:46.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.700+0000 7f0885a3a700 1 -- start start 2020-11-20T12:20:46.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.701+0000 7f0885a3a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08800654a0 0x7f0880138340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.701+0000 7f0885a3a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08800670c0 0x7f08800b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.701+0000 7f0885a3a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880146c60 0x7f08801470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.701+0000 7f0885a3a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088009bbd0 con 0x7f08800654a0 2020-11-20T12:20:46.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.701+0000 7f0885a3a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0880138830 con 0x7f0880146c60 2020-11-20T12:20:46.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.701+0000 7f0885a3a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0880138f40 con 0x7f08800670c0 2020-11-20T12:20:46.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0885239700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880146c60 0x7f08801470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0885239700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880146c60 0x7f08801470c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53918/0 (socket says 172.21.15.187:53918) 2020-11-20T12:20:46.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0885239700 1 -- 172.21.15.187:0/2896900552 learned_addr learned my addr 172.21.15.187:0/2896900552 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:46.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0877fff700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08800670c0 0x7f08800b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0884a38700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08800654a0 0x7f0880138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0885239700 1 -- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08800670c0 msgr2=0x7f08800b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0885239700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08800670c0 0x7f08800b0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0885239700 1 -- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08800654a0 msgr2=0x7f0880138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.702+0000 7f0885239700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08800654a0 0x7f0880138340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0885239700 1 -- 172.21.15.187:0/2896900552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0880138ca0 con 0x7f0880146c60 2020-11-20T12:20:46.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0877fff700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08800670c0 0x7f08800b0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:46.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0884a38700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08800654a0 0x7f0880138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:46.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0885239700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880146c60 0x7f08801470c0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f087800c6e0 tx=0x7f0878007b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:46.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0874ff9700 1 -- 172.21.15.187:0/2896900552 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087800e560 con 0x7f0880146c60 2020-11-20T12:20:46.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0874ff9700 1 -- 172.21.15.187:0/2896900552 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087800e6c0 con 0x7f0880146c60 2020-11-20T12:20:46.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0874ff9700 1 -- 172.21.15.187:0/2896900552 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087800ea60 con 0x7f0880146c60 2020-11-20T12:20:46.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088009c030 con 0x7f0880146c60 2020-11-20T12:20:46.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.703+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08800b0b30 con 0x7f0880146c60 2020-11-20T12:20:46.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.705+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0880061ad0 con 0x7f0880146c60 2020-11-20T12:20:46.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.705+0000 7f0874ff9700 1 -- 172.21.15.187:0/2896900552 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0878010030 con 0x7f0880146c60 2020-11-20T12:20:46.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.706+0000 7f0874ff9700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0858056d50 0x7f0858059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.706+0000 7f0874ff9700 1 -- 172.21.15.187:0/2896900552 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f087806fe50 con 0x7f0880146c60 2020-11-20T12:20:46.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.706+0000 7f0884a38700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0858056d50 0x7f0858059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.707+0000 7f0884a38700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0858056d50 0x7f0858059210 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f087c007cb0 tx=0x7f087c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:46.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.710+0000 7f0874ff9700 1 -- 172.21.15.187:0/2896900552 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f08780451d0 con 0x7f0880146c60 2020-11-20T12:20:46.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.956+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0880069cd0 con 0x7f0880146c60 2020-11-20T12:20:46.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.956+0000 7f0874ff9700 1 -- 172.21.15.187:0/2896900552 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+2159 (secure 0 0 0) 0x7f08780487f0 con 0x7f0880146c60 2020-11-20T12:20:46.960 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:46.961 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":160,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:46.529050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:46.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0858056d50 msgr2=0x7f0858059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0858056d50 0x7f0858059210 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f087c007cb0 tx=0x7f087c009040).stop 2020-11-20T12:20:46.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880146c60 msgr2=0x7f08801470c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880146c60 0x7f08801470c0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f087800c6e0 tx=0x7f0878007b50).stop 2020-11-20T12:20:46.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 shutdown_connections 2020-11-20T12:20:46.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0858056d50 0x7f0858059210 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08800654a0 0x7f0880138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0880146c60 0x7f08801470c0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 --2- 172.21.15.187:0/2896900552 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08800670c0 0x7f08800b0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 >> 172.21.15.187:0/2896900552 conn(0x7f08800bab90 msgr2=0x7f0880146510 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:46.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 shutdown_connections 2020-11-20T12:20:46.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:46.959+0000 7f0885a3a700 1 -- 172.21.15.187:0/2896900552 wait complete. 2020-11-20T12:20:46.966 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 160 2020-11-20T12:20:47.188 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:20:21.071872+0000', 'modified': '2020-11-20T12:20:46.529050+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 151, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9661, 'mds_1': 6281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6281': {'gid': 6281, 'name': 'b', 'rank': 1, 'incarnation': 158, 'state': 'up:reconnect', 'state_seq': 4, 'addr': '172.21.15.5:6833/191681417', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 191681417}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 191681417}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_9661': {'gid': 9661, 'name': 'a', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3618925546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3618925546}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3618925546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:20:47.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6281:up:reconnect 2020-11-20T12:20:48.190 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:48.444 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.441+0000 7f7e65498700 1 Processor -- start 2020-11-20T12:20:49.949 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.441+0000 7f7e65498700 1 -- start start 2020-11-20T12:20:49.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.443+0000 7f7e65498700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e60148870 0x7f7e600b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:49.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.443+0000 7f7e65498700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e6014cf50 con 0x7f7e60148490 2020-11-20T12:20:49.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.443+0000 7f7e65498700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e600b9720 con 0x7f7e600a1360 2020-11-20T12:20:49.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.443+0000 7f7e65498700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e600b9bc0 con 0x7f7e60148870 2020-11-20T12:20:49.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.443+0000 7f7e64c97700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7e600a1360 0x7f7e600b5110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50532/0 (socket says 172.21.15.187:50532) 2020-11-20T12:20:49.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.443+0000 7f7e64c97700 1 -- 172.21.15.187:0/2201887773 learned_addr learned my addr 172.21.15.187:0/2201887773 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:49.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.443+0000 7f7e577fe700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e60148870 0x7f7e600b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:49.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.444+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707311176 0 0) 0x7f7e600b9720 con 0x7f7e600a1360 2020-11-20T12:20:49.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.444+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e40003240 con 0x7f7e600a1360 2020-11-20T12:20:49.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.444+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2265390847 0 0) 0x7f7e40003240 con 0x7f7e600a1360 2020-11-20T12:20:49.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.444+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e60148870 msgr2=0x7f7e600b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:49.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.444+0000 7f7e5f7fe700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e60148870 0x7f7e600b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.444+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 >> v1:172.21.15.5:6789/0 conn(0x7f7e60148490 legacy=0x7f7e600bf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.444+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e600bfdc0 con 0x7f7e600a1360 2020-11-20T12:20:49.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.444+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f7e58001860 con 0x7f7e600a1360 2020-11-20T12:20:49.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 >> v1:172.21.15.187:6789/0 conn(0x7f7e600a1360 legacy=0x7f7e600b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e5f7fe700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e40003730 0x7f7e40003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:49.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e5f7fe700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e400044e0 0x7f7e40004920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:49.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e5f7fe700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e40004e10 0x7f7e4000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:49.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e600b9bc0 con 0x7f7e40004e10 2020-11-20T12:20:49.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e40005380 con 0x7f7e40003730 2020-11-20T12:20:49.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e400041c0 con 0x7f7e400044e0 2020-11-20T12:20:49.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e5ffff700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e40003730 0x7f7e40003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:49.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.445+0000 7f7e577fe700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e400044e0 0x7f7e40004920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:49.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e64c97700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e40004e10 0x7f7e4000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:49.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e5ffff700 1 -- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e400044e0 msgr2=0x7f7e40004920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:49.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e5ffff700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e400044e0 0x7f7e40004920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e5ffff700 1 -- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e40004e10 msgr2=0x7f7e4000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:49.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e5ffff700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e40004e10 0x7f7e4000b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e5ffff700 1 -- 172.21.15.187:0/2201887773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40003240 con 0x7f7e40003730 2020-11-20T12:20:49.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e5ffff700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e40003730 0x7f7e40003b50 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7f7e50001920 tx=0x7f7e50007b90).ready entity=mon.1 client_cookie=15e43c9f1eff3b11 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:49.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e50007020 con 0x7f7e40003730 2020-11-20T12:20:49.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.446+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e50007180 con 0x7f7e40003730 2020-11-20T12:20:49.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.447+0000 7f7e5f7fe700 1 -- 172.21.15.187:0/2201887773 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000c830 con 0x7f7e40003730 2020-11-20T12:20:49.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.447+0000 7f7e65498700 1 -- 172.21.15.187:0/2201887773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e40003730 msgr2=0x7f7e40003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.447+0000 7f7e65498700 1 --2- 172.21.15.187:0/2201887773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e40003730 0x7f7e40003b50 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7f7e50001920 tx=0x7f7e50007b90).stop 2020-11-20T12:20:49.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.447+0000 7f7e64c97700 1 -- 172.21.15.187:0/2201887773 reap_dead start 2020-11-20T12:20:49.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.447+0000 7f7e65498700 1 -- 172.21.15.187:0/2201887773 shutdown_connections 2020-11-20T12:20:49.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.447+0000 7f7e65498700 1 -- 172.21.15.187:0/2201887773 >> 172.21.15.187:0/2201887773 conn(0x7f7e60068370 msgr2=0x7f7e600befa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:49.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.447+0000 7f7e65498700 1 -- 172.21.15.187:0/2201887773 shutdown_connections 2020-11-20T12:20:49.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.447+0000 7f7e65498700 1 -- 172.21.15.187:0/2201887773 wait complete. 2020-11-20T12:20:49.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.449+0000 7f7e65498700 1 Processor -- start 2020-11-20T12:20:49.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.449+0000 7f7e65498700 1 -- start start 2020-11-20T12:20:49.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e65498700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e600a4670 0x7f7e60158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:49.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e65498700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e600a28e0 0x7f7e600a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:49.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e65498700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e6014a7c0 0x7f7e601585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:49.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e65498700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6014a080 con 0x7f7e600a28e0 2020-11-20T12:20:49.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e65498700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6014a320 con 0x7f7e6014a7c0 2020-11-20T12:20:49.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e65498700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e60149d70 con 0x7f7e600a4670 2020-11-20T12:20:49.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e64c97700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e6014a7c0 0x7f7e601585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:49.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e64c97700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e6014a7c0 0x7f7e601585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53938/0 (socket says 172.21.15.187:53938) 2020-11-20T12:20:49.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.450+0000 7f7e64c97700 1 -- 172.21.15.187:0/1238012899 learned_addr learned my addr 172.21.15.187:0/1238012899 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:49.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e5ffff700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e600a4670 0x7f7e60158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:49.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e577fe700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e600a28e0 0x7f7e600a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:49.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e64c97700 1 -- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e600a4670 msgr2=0x7f7e60158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e64c97700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e600a4670 0x7f7e60158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e64c97700 1 -- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e600a28e0 msgr2=0x7f7e600a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e64c97700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e600a28e0 0x7f7e600a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e64c97700 1 -- 172.21.15.187:0/1238012899 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6014b1e0 con 0x7f7e6014a7c0 2020-11-20T12:20:49.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e5ffff700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e600a4670 0x7f7e60158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:49.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e64c97700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e6014a7c0 0x7f7e601585e0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7f7e6005c640 tx=0x7f7e580017a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:49.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e5cff9700 1 -- 172.21.15.187:0/1238012899 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5800cc70 con 0x7f7e6014a7c0 2020-11-20T12:20:49.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e5cff9700 1 -- 172.21.15.187:0/1238012899 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5800cdd0 con 0x7f7e6014a7c0 2020-11-20T12:20:49.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6014ac40 con 0x7f7e6014a7c0 2020-11-20T12:20:49.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e5cff9700 1 -- 172.21.15.187:0/1238012899 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58004b50 con 0x7f7e6014a7c0 2020-11-20T12:20:49.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.451+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e60158d10 con 0x7f7e6014a7c0 2020-11-20T12:20:49.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.452+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e60060640 con 0x7f7e6014a7c0 2020-11-20T12:20:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.453+0000 7f7e5cff9700 1 -- 172.21.15.187:0/1238012899 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7e5801c070 con 0x7f7e6014a7c0 2020-11-20T12:20:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.454+0000 7f7e5cff9700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e3c056d20 0x7f7e3c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.454+0000 7f7e5cff9700 1 -- 172.21.15.187:0/1238012899 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7e5806f9f0 con 0x7f7e6014a7c0 2020-11-20T12:20:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.454+0000 7f7e5ffff700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e3c056d20 0x7f7e3c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:49.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.455+0000 7f7e5ffff700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e3c056d20 0x7f7e3c0591e0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f7e600b5ff0 tx=0x7f7e50015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:49.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.457+0000 7f7e5cff9700 1 -- 172.21.15.187:0/1238012899 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7e58044d70 con 0x7f7e6014a7c0 2020-11-20T12:20:49.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.704+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e600801f0 con 0x7f7e6014a7c0 2020-11-20T12:20:49.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.705+0000 7f7e5cff9700 1 -- 172.21.15.187:0/1238012899 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2156 (secure 0 0 0) 0x7f7e58048390 con 0x7f7e6014a7c0 2020-11-20T12:20:49.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e3c056d20 msgr2=0x7f7e3c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e3c056d20 0x7f7e3c0591e0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f7e600b5ff0 tx=0x7f7e50015000).stop 2020-11-20T12:20:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e6014a7c0 msgr2=0x7f7e601585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e6014a7c0 0x7f7e601585e0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7f7e6005c640 tx=0x7f7e580017a0).stop 2020-11-20T12:20:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 shutdown_connections 2020-11-20T12:20:49.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7e3c056d20 0x7f7e3c0591e0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e6014a7c0 0x7f7e601585e0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e600a4670 0x7f7e60158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 --2- 172.21.15.187:0/1238012899 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e600a28e0 0x7f7e600a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 >> 172.21.15.187:0/1238012899 conn(0x7f7e60068370 msgr2=0x7f7e60060ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:49.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 shutdown_connections 2020-11-20T12:20:49.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:48.707+0000 7f7e65498700 1 -- 172.21.15.187:0/1238012899 wait complete. 2020-11-20T12:20:49.979 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 162 2020-11-20T12:20:49.979 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:49.979 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":162,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:48.542585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:49.980 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:20:49.215+0000 7f4976072700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2020-11-20T12:20:50.182 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 162, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:20:21.071872+0000', 'modified': '2020-11-20T12:20:48.542585+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 151, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9661, 'mds_1': 6281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6281': {'gid': 6281, 'name': 'b', 'rank': 1, 'incarnation': 158, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.5:6833/191681417', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6832', 'nonce': 191681417}, {'type': 'v1', 'addr': '172.21.15.5:6833', 'nonce': 191681417}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_9661': {'gid': 9661, 'name': 'a', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.187:6835/3618925546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3618925546}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3618925546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:20:50.182 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2020-11-20T12:20:50.183 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2020-11-20T12:20:50.460 INFO:teuthology.orchestra.run.smithi005.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":1,"id":6281,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":5.917239092,"mdsmap_epoch":162,"osdmap_epoch":151,"osdmap_epoch_barrier":151,"uptime":10.289484085} 2020-11-20T12:20:50.461 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 6281, "mdsmap_epoch": 162, "osdmap_epoch": 151, "osdmap_epoch_barrier": 151, "rank_uptime": 5.917239092, "state": "up:active", "uptime": 10.289484085, "want_state": "up:active", "whoami": 1 } 2020-11-20T12:20:50.461 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:20:50.746 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":9661,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":29.641788701999999,"mdsmap_epoch":162,"osdmap_epoch":151,"osdmap_epoch_barrier":151,"uptime":41.295122401999997} 2020-11-20T12:20:50.746 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 9661, "mdsmap_epoch": 162, "osdmap_epoch": 151, "osdmap_epoch_barrier": 151, "rank_uptime": 29.641788702, "state": "up:active", "uptime": 41.295122402, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:20:50.747 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:20:50.748 INFO:teuthology.orchestra.run.smithi187:> import sys 2020-11-20T12:20:50.748 INFO:teuthology.orchestra.run.smithi187:> import time 2020-11-20T12:20:50.748 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:20:50.749 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:20:50.749 INFO:teuthology.orchestra.run.smithi187:> n = 10 2020-11-20T12:20:50.750 INFO:teuthology.orchestra.run.smithi187:> abs_path = "/home/ubuntu/cephtest/mnt.0/delete_me/file" 2020-11-20T12:20:50.750 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:20:50.751 INFO:teuthology.orchestra.run.smithi187:> if not os.path.exists(os.path.dirname(abs_path)): 2020-11-20T12:20:50.751 INFO:teuthology.orchestra.run.smithi187:> os.makedirs(os.path.dirname(abs_path)) 2020-11-20T12:20:50.751 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:20:50.752 INFO:teuthology.orchestra.run.smithi187:> for i in range(0, n): 2020-11-20T12:20:50.752 INFO:teuthology.orchestra.run.smithi187:> fname = "{0}_{1}".format(abs_path, i) 2020-11-20T12:20:50.752 INFO:teuthology.orchestra.run.smithi187:> with open(fname, '"'"'w'"'"') as f: 2020-11-20T12:20:50.753 INFO:teuthology.orchestra.run.smithi187:> f.write('"'"'content'"'"') 2020-11-20T12:20:50.753 INFO:teuthology.orchestra.run.smithi187:> if False: 2020-11-20T12:20:50.754 INFO:teuthology.orchestra.run.smithi187:> f.flush() 2020-11-20T12:20:50.754 INFO:teuthology.orchestra.run.smithi187:> os.fsync(f.fileno()) 2020-11-20T12:20:50.754 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:20:51.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:20:51.127 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.pin -v 1 delete_me) 2020-11-20T12:20:51.248 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:51.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.486+0000 7f96e3fff700 1 Processor -- start 2020-11-20T12:20:51.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.486+0000 7f96e3fff700 1 -- start start 2020-11-20T12:20:51.490 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.487+0000 7f96e3fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f96e414c650 0x7f96e40652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.487+0000 7f96e3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f96e40657d0 0x7f96e4062460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.487+0000 7f96e3fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e4099e70 con 0x7f96e4066270 2020-11-20T12:20:51.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.487+0000 7f96e3fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e409a110 con 0x7f96e414c650 2020-11-20T12:20:51.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.487+0000 7f96e3fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e409a3b0 con 0x7f96e40657d0 2020-11-20T12:20:51.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96da7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f96e4066270 0x7f96e4061d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50550/0 (socket says 172.21.15.187:50550) 2020-11-20T12:20:51.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96e37fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f96e40657d0 0x7f96e4062460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96da7fc700 1 -- 172.21.15.187:0/2370076344 learned_addr learned my addr 172.21.15.187:0/2370076344 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:51.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96e2ffd700 1 --2- 172.21.15.187:0/2370076344 >> v2:172.21.15.5:3300/0 conn(0x7f96e414c650 0x7f96e40652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96e37fe700 1 -- 172.21.15.187:0/2370076344 >> v2:172.21.15.5:3300/0 conn(0x7f96e414c650 msgr2=0x7f96e40652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:51.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96e37fe700 1 --2- 172.21.15.187:0/2370076344 >> v2:172.21.15.5:3300/0 conn(0x7f96e414c650 0x7f96e40652e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96e37fe700 1 -- 172.21.15.187:0/2370076344 >> v1:172.21.15.187:6789/0 conn(0x7f96e4066270 legacy=0x7f96e4061d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96e27fc700 1 -- 172.21.15.187:0/2370076344 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958592186 0 0) 0x7f96e4099e70 con 0x7f96e4066270 2020-11-20T12:20:51.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.488+0000 7f96e37fe700 1 -- 172.21.15.187:0/2370076344 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e4066750 con 0x7f96e40657d0 2020-11-20T12:20:51.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.489+0000 7f96e37fe700 1 --2- 172.21.15.187:0/2370076344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e4062460 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f96e409a680 tx=0x7f96dc0091d0).ready entity=mon.1 client_cookie=cacaaae3d40b30c3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:51.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.489+0000 7f96e27fc700 1 -- 172.21.15.187:0/2370076344 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc00bc50 con 0x7f96e40657d0 2020-11-20T12:20:51.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.489+0000 7f96e27fc700 1 -- 172.21.15.187:0/2370076344 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96dc00bdb0 con 0x7f96e40657d0 2020-11-20T12:20:51.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.489+0000 7f96e27fc700 1 -- 172.21.15.187:0/2370076344 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc011610 con 0x7f96e40657d0 2020-11-20T12:20:51.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.490+0000 7f96e3fff700 1 -- 172.21.15.187:0/2370076344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 msgr2=0x7f96e4062460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.490+0000 7f96e3fff700 1 --2- 172.21.15.187:0/2370076344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e4062460 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f96e409a680 tx=0x7f96dc0091d0).stop 2020-11-20T12:20:51.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.490+0000 7f96e3fff700 1 -- 172.21.15.187:0/2370076344 shutdown_connections 2020-11-20T12:20:51.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.490+0000 7f96e3fff700 1 --2- 172.21.15.187:0/2370076344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e4062460 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.490+0000 7f96e3fff700 1 --2- 172.21.15.187:0/2370076344 >> v2:172.21.15.5:3300/0 conn(0x7f96e414c650 0x7f96e40652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.490+0000 7f96e3fff700 1 -- 172.21.15.187:0/2370076344 >> 172.21.15.187:0/2370076344 conn(0x7f96e40bab90 msgr2=0x7f96e40bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:51.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.490+0000 7f96e3fff700 1 -- 172.21.15.187:0/2370076344 shutdown_connections 2020-11-20T12:20:51.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.490+0000 7f96e3fff700 1 -- 172.21.15.187:0/2370076344 wait complete. 2020-11-20T12:20:51.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.491+0000 7f96e3fff700 1 Processor -- start 2020-11-20T12:20:51.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.492+0000 7f96e3fff700 1 -- start start 2020-11-20T12:20:51.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.492+0000 7f96e3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e415c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.492+0000 7f96e3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96e4066270 0x7f96e41493b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96e414c650 0x7f96e409f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e40a04b0 con 0x7f96e414c650 2020-11-20T12:20:51.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e414d110 con 0x7f96e40657d0 2020-11-20T12:20:51.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e414ce70 con 0x7f96e4066270 2020-11-20T12:20:51.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e415c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e415c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53950/0 (socket says 172.21.15.187:53950) 2020-11-20T12:20:51.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 -- 172.21.15.187:0/2150306466 learned_addr learned my addr 172.21.15.187:0/2150306466 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:51.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 -- 172.21.15.187:0/2150306466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96e4066270 msgr2=0x7f96e41493b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96e4066270 0x7f96e41493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:51.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 -- 172.21.15.187:0/2150306466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96e414c650 msgr2=0x7f96e409f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96e414c650 0x7f96e409f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:51.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 -- 172.21.15.187:0/2150306466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e41498a0 con 0x7f96e40657d0 2020-11-20T12:20:51.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.493+0000 7f96e2ffd700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e415c140 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f96d400d8e0 tx=0x7f96d4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:51.505 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.494+0000 7f96db7fe700 1 -- 172.21.15.187:0/2150306466 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4009790 con 0x7f96e40657d0 2020-11-20T12:20:51.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.494+0000 7f96db7fe700 1 -- 172.21.15.187:0/2150306466 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d40098f0 con 0x7f96e40657d0 2020-11-20T12:20:51.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.494+0000 7f96db7fe700 1 -- 172.21.15.187:0/2150306466 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4009c90 con 0x7f96e40657d0 2020-11-20T12:20:51.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.494+0000 7f96e3fff700 1 -- 172.21.15.187:0/2150306466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e409fa10 con 0x7f96e40657d0 2020-11-20T12:20:51.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.494+0000 7f96e3fff700 1 -- 172.21.15.187:0/2150306466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e40b2a40 con 0x7f96e40657d0 2020-11-20T12:20:51.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.495+0000 7f96d8ff9700 1 -- 172.21.15.187:0/2150306466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e414a530 con 0x7f96e40657d0 2020-11-20T12:20:51.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.496+0000 7f96db7fe700 1 -- 172.21.15.187:0/2150306466 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f96d4013480 con 0x7f96e40657d0 2020-11-20T12:20:51.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.496+0000 7f96db7fe700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96c0056d20 0x7f96c00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.496+0000 7f96db7fe700 1 -- 172.21.15.187:0/2150306466 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f96d406ffe0 con 0x7f96e40657d0 2020-11-20T12:20:51.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.497+0000 7f96da7fc700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96c0056d20 0x7f96c00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.497+0000 7f96da7fc700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96c0056d20 0x7f96c00591e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f96cc0011a0 tx=0x7f96cc0076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:51.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.500+0000 7f96db7fe700 1 -- 172.21.15.187:0/2150306466 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f96d40452b0 con 0x7f96e40657d0 2020-11-20T12:20:51.750 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.746+0000 7f96d8ff9700 1 -- 172.21.15.187:0/2150306466 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96e40b2c60 con 0x7f96e40657d0 2020-11-20T12:20:51.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.747+0000 7f96db7fe700 1 -- 172.21.15.187:0/2150306466 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2156 (secure 0 0 0) 0x7f96d40488d0 con 0x7f96e40657d0 2020-11-20T12:20:51.751 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:51.751 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":162,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:48.542585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:51.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 -- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96c0056d20 msgr2=0x7f96c00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96c0056d20 0x7f96c00591e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f96cc0011a0 tx=0x7f96cc0076a0).stop 2020-11-20T12:20:51.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 -- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 msgr2=0x7f96e415c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e415c140 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f96d400d8e0 tx=0x7f96d4008040).stop 2020-11-20T12:20:51.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 -- 172.21.15.187:0/2150306466 shutdown_connections 2020-11-20T12:20:51.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f96c0056d20 0x7f96c00591e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.754 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f96e414c650 0x7f96e409f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:51.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96e40657d0 0x7f96e415c140 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 --2- 172.21.15.187:0/2150306466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f96e4066270 0x7f96e41493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:51.755 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 -- 172.21.15.187:0/2150306466 >> 172.21.15.187:0/2150306466 conn(0x7f96e40bab90 msgr2=0x7f96e40b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:51.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 -- 172.21.15.187:0/2150306466 shutdown_connections 2020-11-20T12:20:51.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:51.750+0000 7f96d8ff9700 1 -- 172.21.15.187:0/2150306466 wait complete. 2020-11-20T12:20:51.756 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 162 2020-11-20T12:20:51.779 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:20:52.079 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2020-11-20T12:20:52.080 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:20:52.081 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2020-11-20T12:20:54.083 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:54.146 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:20:54.161 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:20:54.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.335+0000 7fa287fff700 1 Processor -- start 2020-11-20T12:20:54.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.335+0000 7fa287fff700 1 -- start start 2020-11-20T12:20:54.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.337+0000 7fa287fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa2880b5190 0x7fa2880b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:54.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.337+0000 7fa287fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa288099e70 con 0x7fa28814c650 2020-11-20T12:20:54.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.337+0000 7fa287fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28809a110 con 0x7fa288066270 2020-11-20T12:20:54.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.337+0000 7fa287fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28809a3b0 con 0x7fa2880b5190 2020-11-20T12:20:54.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.337+0000 7fa2867fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa288066270 0x7fa2880afcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50564/0 (socket says 172.21.15.187:50564) 2020-11-20T12:20:54.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.337+0000 7fa2867fc700 1 -- 172.21.15.187:0/2643900541 learned_addr learned my addr 172.21.15.187:0/2643900541 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:20:54.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.337+0000 7fa2877fe700 1 --2- 172.21.15.187:0/2643900541 >> v2:172.21.15.187:3300/0 conn(0x7fa2880b5190 0x7fa2880b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:54.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.338+0000 7fa285ffb700 1 -- 172.21.15.187:0/2643900541 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749308026 0 0) 0x7fa28809a110 con 0x7fa288066270 2020-11-20T12:20:54.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.338+0000 7fa285ffb700 1 -- 172.21.15.187:0/2643900541 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa274003240 con 0x7fa288066270 2020-11-20T12:20:54.344 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.338+0000 7fa2877fe700 1 -- 172.21.15.187:0/2643900541 >> v1:172.21.15.187:6789/0 conn(0x7fa288066270 legacy=0x7fa2880afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:54.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.338+0000 7fa2877fe700 1 -- 172.21.15.187:0/2643900541 >> v1:172.21.15.5:6789/0 conn(0x7fa28814c650 legacy=0x7fa2880652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:54.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.338+0000 7fa2877fe700 1 -- 172.21.15.187:0/2643900541 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288066750 con 0x7fa2880b5190 2020-11-20T12:20:54.349 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa2877fe700 1 --2- 172.21.15.187:0/2643900541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2880b5190 0x7fa2880b0400 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7fa280009c90 tx=0x7fa2800091d0).ready entity=mon.1 client_cookie=92b98ae75bbb900 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:54.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa285ffb700 1 -- 172.21.15.187:0/2643900541 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28000bb00 con 0x7fa2880b5190 2020-11-20T12:20:54.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa285ffb700 1 -- 172.21.15.187:0/2643900541 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28000bc60 con 0x7fa2880b5190 2020-11-20T12:20:54.350 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa285ffb700 1 -- 172.21.15.187:0/2643900541 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280011400 con 0x7fa2880b5190 2020-11-20T12:20:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa287fff700 1 -- 172.21.15.187:0/2643900541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2880b5190 msgr2=0x7fa2880b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa287fff700 1 --2- 172.21.15.187:0/2643900541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2880b5190 0x7fa2880b0400 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7fa280009c90 tx=0x7fa2800091d0).stop 2020-11-20T12:20:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa287fff700 1 -- 172.21.15.187:0/2643900541 shutdown_connections 2020-11-20T12:20:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa287fff700 1 --2- 172.21.15.187:0/2643900541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2880b5190 0x7fa2880b0400 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:54.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.339+0000 7fa287fff700 1 -- 172.21.15.187:0/2643900541 >> 172.21.15.187:0/2643900541 conn(0x7fa2880bab90 msgr2=0x7fa2880bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:54.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.340+0000 7fa287fff700 1 -- 172.21.15.187:0/2643900541 shutdown_connections 2020-11-20T12:20:54.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.340+0000 7fa287fff700 1 -- 172.21.15.187:0/2643900541 wait complete. 2020-11-20T12:20:54.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.341+0000 7fa287fff700 1 Processor -- start 2020-11-20T12:20:54.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.341+0000 7fa287fff700 1 -- start start 2020-11-20T12:20:54.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.342+0000 7fa287fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa288066270 0x7fa28809ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:54.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.342+0000 7fa287fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880b5190 0x7fa288090030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:54.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.342+0000 7fa287fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa28814c650 0x7fa28809b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:54.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.342+0000 7fa287fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28809a5b0 con 0x7fa2880b5190 2020-11-20T12:20:54.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.342+0000 7fa287fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2880906f0 con 0x7fa28814c650 2020-11-20T12:20:54.355 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.342+0000 7fa287fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2880b2030 con 0x7fa288066270 2020-11-20T12:20:54.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2877fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa28814c650 0x7fa28809b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:54.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2877fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa28814c650 0x7fa28809b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53968/0 (socket says 172.21.15.187:53968) 2020-11-20T12:20:54.356 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2877fe700 1 -- 172.21.15.187:0/1637173500 learned_addr learned my addr 172.21.15.187:0/1637173500 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:54.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2867fc700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880b5190 0x7fa288090030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:54.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa286ffd700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa288066270 0x7fa28809ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:54.357 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2877fe700 1 -- 172.21.15.187:0/1637173500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa288066270 msgr2=0x7fa28809ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2877fe700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa288066270 0x7fa28809ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2877fe700 1 -- 172.21.15.187:0/1637173500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880b5190 msgr2=0x7fa288090030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2877fe700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880b5190 0x7fa288090030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:54.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.343+0000 7fa2877fe700 1 -- 172.21.15.187:0/1637173500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2880b22d0 con 0x7fa28814c650 2020-11-20T12:20:54.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.344+0000 7fa2877fe700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa28814c650 0x7fa28809b9f0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7fa2880b8740 tx=0x7fa280016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:54.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.344+0000 7fa26effd700 1 -- 172.21.15.187:0/1637173500 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280010780 con 0x7fa28814c650 2020-11-20T12:20:54.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.344+0000 7fa26effd700 1 -- 172.21.15.187:0/1637173500 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2800108e0 con 0x7fa28814c650 2020-11-20T12:20:54.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.344+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28809bee0 con 0x7fa28814c650 2020-11-20T12:20:54.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.344+0000 7fa26effd700 1 -- 172.21.15.187:0/1637173500 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280010c60 con 0x7fa28814c650 2020-11-20T12:20:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.344+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28815f8c0 con 0x7fa28814c650 2020-11-20T12:20:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.345+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa288090520 con 0x7fa28814c650 2020-11-20T12:20:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.346+0000 7fa26effd700 1 -- 172.21.15.187:0/1637173500 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa280024030 con 0x7fa28814c650 2020-11-20T12:20:54.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.346+0000 7fa26effd700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa264056d20 0x7fa2640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:54.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.346+0000 7fa26effd700 1 -- 172.21.15.187:0/1637173500 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa280071e80 con 0x7fa28814c650 2020-11-20T12:20:54.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.347+0000 7fa286ffd700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa264056d20 0x7fa2640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:54.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.347+0000 7fa286ffd700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa264056d20 0x7fa2640591e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7fa278003450 tx=0x7fa278009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:54.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.350+0000 7fa26effd700 1 -- 172.21.15.187:0/1637173500 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa280047220 con 0x7fa28814c650 2020-11-20T12:20:54.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.599+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2880b2460 con 0x7fa28814c650 2020-11-20T12:20:54.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.600+0000 7fa26effd700 1 -- 172.21.15.187:0/1637173500 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2156 (secure 0 0 0) 0x7fa280022070 con 0x7fa28814c650 2020-11-20T12:20:54.604 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:54.604 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":162,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:48.542585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:54.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.602+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa264056d20 msgr2=0x7fa2640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:54.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.602+0000 7fa287fff700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa264056d20 0x7fa2640591e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7fa278003450 tx=0x7fa278009040).stop 2020-11-20T12:20:54.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.602+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa28814c650 msgr2=0x7fa28809b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:54.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.602+0000 7fa287fff700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa28814c650 0x7fa28809b9f0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7fa2880b8740 tx=0x7fa280016000).stop 2020-11-20T12:20:54.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.603+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 shutdown_connections 2020-11-20T12:20:54.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.603+0000 7fa287fff700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa264056d20 0x7fa2640591e0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:54.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.603+0000 7fa287fff700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa28814c650 0x7fa28809b9f0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:54.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.603+0000 7fa287fff700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa288066270 0x7fa28809ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:54.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.603+0000 7fa287fff700 1 --2- 172.21.15.187:0/1637173500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880b5190 0x7fa288090030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:54.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.603+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 >> 172.21.15.187:0/1637173500 conn(0x7fa2880bab90 msgr2=0x7fa2880b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:54.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.603+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 shutdown_connections 2020-11-20T12:20:54.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:54.603+0000 7fa287fff700 1 -- 172.21.15.187:0/1637173500 wait complete. 2020-11-20T12:20:54.610 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 162 2020-11-20T12:20:54.629 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:20:54.929 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2020-11-20T12:20:54.929 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:20:54.930 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2020-11-20T12:20:56.931 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:57.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.185+0000 7f75872f5700 1 Processor -- start 2020-11-20T12:20:57.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.185+0000 7f75872f5700 1 -- start start 2020-11-20T12:20:57.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.187+0000 7f75872f5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75800654a0 0x7f7580149290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:57.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.187+0000 7f75872f5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7580065880 0x7f7580149780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:57.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.187+0000 7f75872f5700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758009a930 con 0x7f75800b1110 2020-11-20T12:20:57.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.187+0000 7f75872f5700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75800b0be0 con 0x7f7580065880 2020-11-20T12:20:57.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.187+0000 7f75872f5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75800b71b0 con 0x7f75800654a0 2020-11-20T12:20:57.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f75862f3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75800654a0 0x7f7580149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:57.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f75862f3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75800654a0 0x7f7580149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:54860/0 (socket says 172.21.15.187:54860) 2020-11-20T12:20:57.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f7586af4700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f75800b1110 0x7f7580061d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50580/0 (socket says 172.21.15.187:50580) 2020-11-20T12:20:57.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f7585af2700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7580065880 0x7f7580149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:57.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f75862f3700 1 -- 172.21.15.187:0/3755426951 learned_addr learned my addr 172.21.15.187:0/3755426951 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:57.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f7585af2700 1 -- 172.21.15.187:0/3755426951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75800654a0 msgr2=0x7f7580149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:57.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f7585af2700 1 --2- 172.21.15.187:0/3755426951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75800654a0 0x7f7580149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f7585af2700 1 -- 172.21.15.187:0/3755426951 >> v1:172.21.15.187:6789/0 conn(0x7f75800b1110 legacy=0x7f7580061d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:57.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.188+0000 7f7585af2700 1 -- 172.21.15.187:0/3755426951 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800662d0 con 0x7f7580065880 2020-11-20T12:20:57.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.189+0000 7f7585af2700 1 --2- 172.21.15.187:0/3755426951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7580065880 0x7f7580149780 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f757000b9a0 tx=0x7f7570006df0).ready entity=mon.1 client_cookie=468ec239f2a921bd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:57.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.189+0000 7f75852f1700 1 -- 172.21.15.187:0/3755426951 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75700097b0 con 0x7f7580065880 2020-11-20T12:20:57.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.189+0000 7f75852f1700 1 -- 172.21.15.187:0/3755426951 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7570009910 con 0x7f7580065880 2020-11-20T12:20:57.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.189+0000 7f75852f1700 1 -- 172.21.15.187:0/3755426951 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570009c90 con 0x7f7580065880 2020-11-20T12:20:57.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.190+0000 7f75872f5700 1 -- 172.21.15.187:0/3755426951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7580065880 msgr2=0x7f7580149780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:57.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.190+0000 7f75872f5700 1 --2- 172.21.15.187:0/3755426951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7580065880 0x7f7580149780 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f757000b9a0 tx=0x7f7570006df0).stop 2020-11-20T12:20:57.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.190+0000 7f75872f5700 1 -- 172.21.15.187:0/3755426951 shutdown_connections 2020-11-20T12:20:57.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.190+0000 7f75872f5700 1 --2- 172.21.15.187:0/3755426951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f75800654a0 0x7f7580149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.190+0000 7f75872f5700 1 --2- 172.21.15.187:0/3755426951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7580065880 0x7f7580149780 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.190+0000 7f75872f5700 1 -- 172.21.15.187:0/3755426951 >> 172.21.15.187:0/3755426951 conn(0x7f75800bab90 msgr2=0x7f75800bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.190+0000 7f75872f5700 1 -- 172.21.15.187:0/3755426951 shutdown_connections 2020-11-20T12:20:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.190+0000 7f75872f5700 1 -- 172.21.15.187:0/3755426951 wait complete. 2020-11-20T12:20:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.192+0000 7f75872f5700 1 Processor -- start 2020-11-20T12:20:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.192+0000 7f75872f5700 1 -- start start 2020-11-20T12:20:57.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75872f5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75800654a0 0x7f75800b2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:57.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75872f5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7580065880 0x7f7580159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:57.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75872f5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75800b1110 0x7f75800903d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:57.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75872f5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758015a4b0 con 0x7f75800b1110 2020-11-20T12:20:57.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75872f5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758009f260 con 0x7f75800654a0 2020-11-20T12:20:57.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75872f5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7580145ad0 con 0x7f7580065880 2020-11-20T12:20:57.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75862f3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75800654a0 0x7f75800b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:57.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75862f3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75800654a0 0x7f75800b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53980/0 (socket says 172.21.15.187:53980) 2020-11-20T12:20:57.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.193+0000 7f75862f3700 1 -- 172.21.15.187:0/478894006 learned_addr learned my addr 172.21.15.187:0/478894006 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:20:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.194+0000 7f7585af2700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7580065880 0x7f7580159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.194+0000 7f7586af4700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75800b1110 0x7f75800903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.194+0000 7f75862f3700 1 -- 172.21.15.187:0/478894006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7580065880 msgr2=0x7f7580159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.194+0000 7f75862f3700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7580065880 0x7f7580159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.194+0000 7f75862f3700 1 -- 172.21.15.187:0/478894006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75800b1110 msgr2=0x7f75800903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.194+0000 7f75862f3700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75800b1110 0x7f75800903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.194+0000 7f75862f3700 1 -- 172.21.15.187:0/478894006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800913d0 con 0x7f75800654a0 2020-11-20T12:20:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.194+0000 7f75862f3700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75800654a0 0x7f75800b2950 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f757c00f860 tx=0x7f757c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:57.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.195+0000 7f756e7fc700 1 -- 172.21.15.187:0/478894006 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c00d6c0 con 0x7f75800654a0 2020-11-20T12:20:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.195+0000 7f756e7fc700 1 -- 172.21.15.187:0/478894006 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c00d820 con 0x7f75800654a0 2020-11-20T12:20:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.195+0000 7f756e7fc700 1 -- 172.21.15.187:0/478894006 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c00dba0 con 0x7f75800654a0 2020-11-20T12:20:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.195+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758009f490 con 0x7f75800654a0 2020-11-20T12:20:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.195+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7580090940 con 0x7f75800654a0 2020-11-20T12:20:57.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.196+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7580061ad0 con 0x7f75800654a0 2020-11-20T12:20:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.196+0000 7f756e7fc700 1 -- 172.21.15.187:0/478894006 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f757c024070 con 0x7f75800654a0 2020-11-20T12:20:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.197+0000 7f756e7fc700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755805b130 0x7f755805d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.197+0000 7f756e7fc700 1 -- 172.21.15.187:0/478894006 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f757c070530 con 0x7f75800654a0 2020-11-20T12:20:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.197+0000 7f7585af2700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755805b130 0x7f755805d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:57.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.198+0000 7f7585af2700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755805b130 0x7f755805d5f0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f7580064380 tx=0x7f7570015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:57.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.201+0000 7f756e7fc700 1 -- 172.21.15.187:0/478894006 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f757c045930 con 0x7f75800654a0 2020-11-20T12:20:57.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.447+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f758009ab30 con 0x7f75800654a0 2020-11-20T12:20:57.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.447+0000 7f756e7fc700 1 -- 172.21.15.187:0/478894006 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2156 (secure 0 0 0) 0x7f757c018070 con 0x7f75800654a0 2020-11-20T12:20:57.451 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:20:57.451 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":162,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:48.542585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:20:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.449+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755805b130 msgr2=0x7f755805d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.449+0000 7f75872f5700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755805b130 0x7f755805d5f0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f7580064380 tx=0x7f7570015000).stop 2020-11-20T12:20:57.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.449+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75800654a0 msgr2=0x7f75800b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.449+0000 7f75872f5700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75800654a0 0x7f75800b2950 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f757c00f860 tx=0x7f757c008040).stop 2020-11-20T12:20:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.450+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 shutdown_connections 2020-11-20T12:20:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.450+0000 7f75872f5700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f755805b130 0x7f755805d5f0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.450+0000 7f75872f5700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75800b1110 0x7f75800903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.450+0000 7f75872f5700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75800654a0 0x7f75800b2950 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.450+0000 7f75872f5700 1 --2- 172.21.15.187:0/478894006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7580065880 0x7f7580159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:57.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.450+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 >> 172.21.15.187:0/478894006 conn(0x7f75800bab90 msgr2=0x7f7580099f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:57.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.450+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 shutdown_connections 2020-11-20T12:20:57.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:20:57.450+0000 7f75872f5700 1 -- 172.21.15.187:0/478894006 wait complete. 2020-11-20T12:20:57.457 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 162 2020-11-20T12:20:57.478 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:20:57.770 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2020-11-20T12:20:57.771 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:20:57.771 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2020-11-20T12:20:59.772 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:00.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.854+0000 7fd0d0e85700 1 Processor -- start 2020-11-20T12:21:00.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.854+0000 7fd0d0e85700 1 -- start start 2020-11-20T12:21:00.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.855+0000 7fd0d0e85700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd0cc0654a0 0x7fd0cc0675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.856+0000 7fd0d0e85700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0cc0670c0 0x7fd0cc0afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.856+0000 7fd0d0e85700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0cc099f70 con 0x7fd0cc065880 2020-11-20T12:21:00.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.856+0000 7fd0d0e85700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0cc0b6a20 con 0x7fd0cc0654a0 2020-11-20T12:21:00.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.856+0000 7fd0d0e85700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0cc148990 con 0x7fd0cc0670c0 2020-11-20T12:21:00.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.856+0000 7fd0cbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0cc0670c0 0x7fd0cc0afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:00.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.856+0000 7fd0cbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0cc0670c0 0x7fd0cc0afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53992/0 (socket says 172.21.15.187:53992) 2020-11-20T12:21:00.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.856+0000 7fd0cbfff700 1 -- 172.21.15.187:0/757213157 learned_addr learned my addr 172.21.15.187:0/757213157 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:00.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.856+0000 7fd0cb7fe700 1 --2- 172.21.15.187:0/757213157 >> v2:172.21.15.5:3300/0 conn(0x7fd0cc0654a0 0x7fd0cc0675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.857+0000 7fd0cbfff700 1 -- 172.21.15.187:0/757213157 >> v2:172.21.15.5:3300/0 conn(0x7fd0cc0654a0 msgr2=0x7fd0cc0675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.857+0000 7fd0cbfff700 1 --2- 172.21.15.187:0/757213157 >> v2:172.21.15.5:3300/0 conn(0x7fd0cc0654a0 0x7fd0cc0675e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.857+0000 7fd0cbfff700 1 -- 172.21.15.187:0/757213157 >> v1:172.21.15.5:6789/0 conn(0x7fd0cc065880 legacy=0x7fd0cc148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:00.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.857+0000 7fd0cbfff700 1 -- 172.21.15.187:0/757213157 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc0affe0 con 0x7fd0cc0670c0 2020-11-20T12:21:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.857+0000 7fd0cbfff700 1 --2- 172.21.15.187:0/757213157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0670c0 0x7fd0cc0afaf0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7fd0cc068310 tx=0x7fd0c4009290).ready entity=mon.1 client_cookie=e307fb963610365f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.857+0000 7fd0ca7fc700 1 -- 172.21.15.187:0/757213157 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c400bb80 con 0x7fd0cc0670c0 2020-11-20T12:21:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.857+0000 7fd0ca7fc700 1 -- 172.21.15.187:0/757213157 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c400bce0 con 0x7fd0cc0670c0 2020-11-20T12:21:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.857+0000 7fd0ca7fc700 1 -- 172.21.15.187:0/757213157 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4011610 con 0x7fd0cc0670c0 2020-11-20T12:21:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.858+0000 7fd0d0e85700 1 -- 172.21.15.187:0/757213157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0670c0 msgr2=0x7fd0cc0afaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:00.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.858+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/757213157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0670c0 0x7fd0cc0afaf0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7fd0cc068310 tx=0x7fd0c4009290).stop 2020-11-20T12:21:00.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.858+0000 7fd0d0e85700 1 -- 172.21.15.187:0/757213157 shutdown_connections 2020-11-20T12:21:00.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.858+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/757213157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0670c0 0x7fd0cc0afaf0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:00.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.858+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/757213157 >> v2:172.21.15.5:3300/0 conn(0x7fd0cc0654a0 0x7fd0cc0675e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.858+0000 7fd0d0e85700 1 -- 172.21.15.187:0/757213157 >> 172.21.15.187:0/757213157 conn(0x7fd0cc0bab90 msgr2=0x7fd0cc0bafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.858+0000 7fd0d0e85700 1 -- 172.21.15.187:0/757213157 shutdown_connections 2020-11-20T12:21:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.858+0000 7fd0d0e85700 1 -- 172.21.15.187:0/757213157 wait complete. 2020-11-20T12:21:00.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.860+0000 7fd0d0e85700 1 Processor -- start 2020-11-20T12:21:00.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.860+0000 7fd0d0e85700 1 -- start start 2020-11-20T12:21:00.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0d0e85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0654a0 0x7fd0cc146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:00.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0d0e85700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0cc065880 0x7fd0cc153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:00.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0d0e85700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0cc0670c0 0x7fd0cc09cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:00.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0d0e85700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc143460 con 0x7fd0cc065880 2020-11-20T12:21:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0d0e85700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc143690 con 0x7fd0cc0654a0 2020-11-20T12:21:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0d0e85700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc143930 con 0x7fd0cc0670c0 2020-11-20T12:21:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0cb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0654a0 0x7fd0cc146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0cb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0654a0 0x7fd0cc146ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53994/0 (socket says 172.21.15.187:53994) 2020-11-20T12:21:00.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.861+0000 7fd0cb7fe700 1 -- 172.21.15.187:0/4051933936 learned_addr learned my addr 172.21.15.187:0/4051933936 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0cbfff700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0cc0670c0 0x7fd0cc09cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0caffd700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0cc065880 0x7fd0cc153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0cb7fe700 1 -- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0cc0670c0 msgr2=0x7fd0cc09cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:00.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0cb7fe700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0cc0670c0 0x7fd0cc09cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0cb7fe700 1 -- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0cc065880 msgr2=0x7fd0cc153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0cb7fe700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0cc065880 0x7fd0cc153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0cb7fe700 1 -- 172.21.15.187:0/4051933936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc143bd0 con 0x7fd0cc0654a0 2020-11-20T12:21:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0caffd700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0cc065880 0x7fd0cc153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:00.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0cb7fe700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0654a0 0x7fd0cc146ed0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7fd0bc00d8e0 tx=0x7fd0bc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:00.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0ab7fe700 1 -- 172.21.15.187:0/4051933936 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00b130 con 0x7fd0cc0654a0 2020-11-20T12:21:00.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.862+0000 7fd0ab7fe700 1 -- 172.21.15.187:0/4051933936 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc009790 con 0x7fd0cc0654a0 2020-11-20T12:21:00.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.863+0000 7fd0ab7fe700 1 -- 172.21.15.187:0/4051933936 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc009b10 con 0x7fd0cc0654a0 2020-11-20T12:21:00.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.863+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc154270 con 0x7fd0cc0654a0 2020-11-20T12:21:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.863+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc1458b0 con 0x7fd0cc0654a0 2020-11-20T12:21:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.864+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc061ad0 con 0x7fd0cc0654a0 2020-11-20T12:21:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.864+0000 7fd0ab7fe700 1 -- 172.21.15.187:0/4051933936 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd0bc01c460 con 0x7fd0cc0654a0 2020-11-20T12:21:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.865+0000 7fd0ab7fe700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0a4056d20 0x7fd0a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:00.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.865+0000 7fd0ab7fe700 1 -- 172.21.15.187:0/4051933936 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd0bc06ff40 con 0x7fd0cc0654a0 2020-11-20T12:21:00.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.865+0000 7fd0caffd700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0a4056d20 0x7fd0a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:00.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.866+0000 7fd0caffd700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0a4056d20 0x7fd0a40591e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7fd0cc0b10c0 tx=0x7fd0c0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:00.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:00.869+0000 7fd0ab7fe700 1 -- 172.21.15.187:0/4051933936 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd0bc0452e0 con 0x7fd0cc0654a0 2020-11-20T12:21:01.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.114+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0cc069cd0 con 0x7fd0cc0654a0 2020-11-20T12:21:01.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.115+0000 7fd0ab7fe700 1 -- 172.21.15.187:0/4051933936 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2157 (secure 0 0 0) 0x7fd0bc048900 con 0x7fd0cc0654a0 2020-11-20T12:21:01.119 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:01.119 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":163,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:20:59.904093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:21:01.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0a4056d20 msgr2=0x7fd0a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:01.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0a4056d20 0x7fd0a40591e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7fd0cc0b10c0 tx=0x7fd0c0009040).stop 2020-11-20T12:21:01.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0654a0 msgr2=0x7fd0cc146ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:01.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0654a0 0x7fd0cc146ed0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7fd0bc00d8e0 tx=0x7fd0bc008040).stop 2020-11-20T12:21:01.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 shutdown_connections 2020-11-20T12:21:01.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd0a4056d20 0x7fd0a40591e0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd0cc0670c0 0x7fd0cc09cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc0654a0 0x7fd0cc146ed0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 --2- 172.21.15.187:0/4051933936 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0cc065880 0x7fd0cc153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.124 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 >> 172.21.15.187:0/4051933936 conn(0x7fd0cc0bab90 msgr2=0x7fd0cc142990 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:01.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 shutdown_connections 2020-11-20T12:21:01.125 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:01.118+0000 7fd0d0e85700 1 -- 172.21.15.187:0/4051933936 wait complete. 2020-11-20T12:21:01.126 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 163 2020-11-20T12:21:01.146 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2020-11-20T12:21:01.434 INFO:teuthology.orchestra.run.smithi005.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"26","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/delete_me","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"21","version":"21","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2020-11-20T12:21:01.435 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "26" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/delete_me", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "21", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "21" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2020-11-20T12:21:01.436 INFO:tasks.cephfs.cephfs_test_case:[('/delete_me', 1)] =?= [('/delete_me', 1)] 2020-11-20T12:21:01.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:01.437 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -rf delete_me/*) 2020-11-20T12:21:01.601 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_tick_interval 2020-11-20T12:21:01.877 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_tick_interval":"5.000000"} 2020-11-20T12:21:01.878 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_tick_interval": "5.000000" } 2020-11-20T12:21:01.879 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:21:01.879 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:21:01.996 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:21:02.018 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:21:02.126 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:21:02.150 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:21:02.151 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:21:02.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:02.152 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:21:02.354 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:21:02.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:02.355 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:21:02.424 INFO:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:21:02.436 INFO:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:21:02.639 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2020-11-20T12:21:02.639 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2020-11-20T12:21:02.640 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:21:02.640 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2020-11-20T12:21:02.640 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.640 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2020-11-20T12:21:02.640 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.641 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2020-11-20T12:21:02.641 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.641 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2020-11-20T12:21:02.641 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2020-11-20T12:21:02.641 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.642 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2020-11-20T12:21:02.642 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2020-11-20T12:21:02.642 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.642 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:21:02.643 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2020-11-20T12:21:02.643 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.643 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:21:02.643 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:21:02.643 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.644 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2020-11-20T12:21:02.644 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.644 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2020-11-20T12:21:02.644 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2020-11-20T12:21:02.644 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:21:02.645 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:21:02.645 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2020-11-20T12:21:02.645 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:21:02.645 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2020-11-20T12:21:02.645 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:21:02.646 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:21:02.646 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2020-11-20T12:21:02.646 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.646 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:21:02.646 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.647 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2020-11-20T12:21:02.647 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.647 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2020-11-20T12:21:02.647 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.647 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2020-11-20T12:21:02.648 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2020-11-20T12:21:02.648 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2020-11-20T12:21:02.648 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:21:02.648 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2020-11-20T12:21:02.648 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:21:02.649 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.649 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2020-11-20T12:21:02.649 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2020-11-20T12:21:02.649 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.649 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2020-11-20T12:21:02.650 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2020-11-20T12:21:02.650 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.650 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2020-11-20T12:21:02.650 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2020-11-20T12:21:02.650 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.651 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2020-11-20T12:21:02.651 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2020-11-20T12:21:02.651 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2020-11-20T12:21:02.651 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.651 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2020-11-20T12:21:02.652 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2020-11-20T12:21:02.652 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.652 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:21:02.652 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.652 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2020-11-20T12:21:02.653 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.653 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2020-11-20T12:21:02.653 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.653 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:21:02.653 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2020-11-20T12:21:02.654 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.654 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2020-11-20T12:21:02.654 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2020-11-20T12:21:02.654 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2020-11-20T12:21:02.654 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2020-11-20T12:21:02.655 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2020-11-20T12:21:02.655 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2020-11-20T12:21:02.655 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2020-11-20T12:21:02.655 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2020-11-20T12:21:02.656 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2020-11-20T12:21:02.656 INFO:teuthology.orchestra.run.smithi187.stdout: debug filestore = 20 2020-11-20T12:21:02.656 INFO:teuthology.orchestra.run.smithi187.stdout: debug journal = 20 2020-11-20T12:21:02.656 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 20 2020-11-20T12:21:02.657 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 25 2020-11-20T12:21:02.657 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2020-11-20T12:21:02.657 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:21:02.657 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2020-11-20T12:21:02.658 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:21:02.658 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:21:02.658 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2020-11-20T12:21:02.658 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:21:02.658 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.659 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2020-11-20T12:21:02.659 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:21:02.659 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2020-11-20T12:21:02.659 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:21:02.659 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:21:02.660 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:21:02.660 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:21:02.660 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2020-11-20T12:21:02.660 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.660 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2020-11-20T12:21:02.661 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:21:02.661 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2020-11-20T12:21:02.661 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2020-11-20T12:21:02.661 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2020-11-20T12:21:02.661 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2020-11-20T12:21:02.662 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2020-11-20T12:21:02.662 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:21:02.662 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2020-11-20T12:21:02.662 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2020-11-20T12:21:02.662 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:21:02.663 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2020-11-20T12:21:02.663 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:02.663 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2020-11-20T12:21:02.663 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2020-11-20T12:21:02.663 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2020-11-20T12:21:02.664 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2020-11-20T12:21:02.664 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:21:02.664 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:21:02.664 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2020-11-20T12:21:02.664 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2020-11-20T12:21:02.665 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:21:02.665 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2020-11-20T12:21:02.665 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:21:02.665 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:21:02.665 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2020-11-20T12:21:02.666 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2020-11-20T12:21:02.666 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2020-11-20T12:21:02.666 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2020-11-20T12:21:02.666 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2020-11-20T12:21:02.666 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2020-11-20T12:21:02.667 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2020-11-20T12:21:02.667 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2020-11-20T12:21:02.667 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal frag = True 2020-11-20T12:21:02.667 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal fragment size max = 10000 2020-11-20T12:21:02.667 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal merge size = 5 2020-11-20T12:21:02.668 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split bits = 3 2020-11-20T12:21:02.668 INFO:teuthology.orchestra.run.smithi187.stdout: mds bal split size = 100 2020-11-20T12:21:02.668 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2020-11-20T12:21:02.668 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2020-11-20T12:21:02.669 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2020-11-20T12:21:02.670 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2020-11-20T12:21:02.670 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2020-11-20T12:21:02.670 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 15m 2020-11-20T12:21:02.671 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 15m 2020-11-20T12:21:02.671 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2020-11-20T12:21:02.672 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2020-11-20T12:21:02.672 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:21:02.672 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2020-11-20T12:21:02.673 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.673 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2020-11-20T12:21:02.674 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.674 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2020-11-20T12:21:02.674 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.674 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2020-11-20T12:21:02.675 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2020-11-20T12:21:02.675 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.675 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2020-11-20T12:21:02.676 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2020-11-20T12:21:02.676 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.676 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:21:02.676 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2020-11-20T12:21:02.677 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.677 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:21:02.677 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:21:02.678 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.678 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2020-11-20T12:21:02.678 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.679 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2020-11-20T12:21:02.679 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2020-11-20T12:21:02.679 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:21:02.680 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:21:02.680 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2020-11-20T12:21:02.680 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:21:02.681 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2020-11-20T12:21:02.681 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:21:02.681 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:21:02.681 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2020-11-20T12:21:02.682 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.682 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:21:02.682 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.683 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2020-11-20T12:21:02.683 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.683 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2020-11-20T12:21:02.684 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.684 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2020-11-20T12:21:02.684 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2020-11-20T12:21:02.685 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2020-11-20T12:21:02.685 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.187,172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790]" 2020-11-20T12:21:02.685 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on skipped message = False 2020-11-20T12:21:02.686 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 099efcab-1650-4909-9b03-145da903b26d 2020-11-20T12:21:02.686 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.686 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2020-11-20T12:21:02.686 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2020-11-20T12:21:02.687 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.687 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2020-11-20T12:21:02.687 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2020-11-20T12:21:02.688 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.688 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2020-11-20T12:21:02.688 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2020-11-20T12:21:02.688 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.689 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2020-11-20T12:21:02.689 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2020-11-20T12:21:02.689 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2020-11-20T12:21:02.690 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.690 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2020-11-20T12:21:02.690 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2020-11-20T12:21:02.690 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.691 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:21:02.691 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.691 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2020-11-20T12:21:02.692 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.692 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2020-11-20T12:21:02.692 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.693 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:21:02.693 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2020-11-20T12:21:02.693 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.693 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2020-11-20T12:21:02.694 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2020-11-20T12:21:02.694 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2020-11-20T12:21:02.694 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2020-11-20T12:21:02.694 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2020-11-20T12:21:02.695 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2020-11-20T12:21:02.695 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2020-11-20T12:21:02.695 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2020-11-20T12:21:02.696 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2020-11-20T12:21:02.696 INFO:teuthology.orchestra.run.smithi005.stdout: debug filestore = 20 2020-11-20T12:21:02.696 INFO:teuthology.orchestra.run.smithi005.stdout: debug journal = 20 2020-11-20T12:21:02.697 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 20 2020-11-20T12:21:02.697 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 25 2020-11-20T12:21:02.697 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2020-11-20T12:21:02.697 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T12:21:02.698 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2020-11-20T12:21:02.698 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2020-11-20T12:21:02.698 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2020-11-20T12:21:02.698 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2020-11-20T12:21:02.699 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:21:02.699 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.699 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2020-11-20T12:21:02.700 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:21:02.700 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2020-11-20T12:21:02.700 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:21:02.701 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:21:02.701 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:21:02.701 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:21:02.702 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2020-11-20T12:21:02.702 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.702 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2020-11-20T12:21:02.702 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:21:02.703 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2020-11-20T12:21:02.703 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2020-11-20T12:21:02.703 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2020-11-20T12:21:02.703 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2020-11-20T12:21:02.704 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2020-11-20T12:21:02.704 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:21:02.704 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2020-11-20T12:21:02.705 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2020-11-20T12:21:02.705 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:21:02.705 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2020-11-20T12:21:02.705 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-11-20T12:21:02.706 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2020-11-20T12:21:02.706 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2020-11-20T12:21:02.706 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2020-11-20T12:21:02.707 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2020-11-20T12:21:02.707 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:21:02.707 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:21:02.707 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2020-11-20T12:21:02.707 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2020-11-20T12:21:02.708 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:21:02.708 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = "" 2020-11-20T12:21:02.708 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:21:02.708 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:21:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2020-11-20T12:21:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2020-11-20T12:21:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2020-11-20T12:21:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2020-11-20T12:21:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2020-11-20T12:21:02.710 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2020-11-20T12:21:02.710 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2020-11-20T12:21:02.710 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2020-11-20T12:21:02.710 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal frag = True 2020-11-20T12:21:02.711 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal fragment size max = 10000 2020-11-20T12:21:02.711 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal merge size = 5 2020-11-20T12:21:02.711 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split bits = 3 2020-11-20T12:21:02.711 INFO:teuthology.orchestra.run.smithi005.stdout: mds bal split size = 100 2020-11-20T12:21:02.712 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2020-11-20T12:21:02.712 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2020-11-20T12:21:02.712 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2020-11-20T12:21:02.712 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2020-11-20T12:21:02.713 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2020-11-20T12:21:02.713 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 15m 2020-11-20T12:21:02.713 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 15m 2020-11-20T12:21:02.715 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:21:02.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.971+0000 7fe29db61700 1 Processor -- start 2020-11-20T12:21:02.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.971+0000 7fe29db61700 1 -- start start 2020-11-20T12:21:02.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.972+0000 7fe29db61700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298065190 0x7fe298065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:02.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.972+0000 7fe29db61700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe298065f90 0x7fe298143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:02.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.972+0000 7fe29db61700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe298143c70 0x7fe29809ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:02.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.972+0000 7fe29db61700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29814ab30 con 0x7fe298065f90 2020-11-20T12:21:02.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.972+0000 7fe29db61700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2980b13e0 con 0x7fe298143c70 2020-11-20T12:21:02.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.972+0000 7fe29db61700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe298062c10 con 0x7fe298065190 2020-11-20T12:21:02.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29d360700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe298143c70 0x7fe29809ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:02.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29d360700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe298143c70 0x7fe29809ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54016/0 (socket says 172.21.15.187:54016) 2020-11-20T12:21:02.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29d360700 1 -- 172.21.15.187:0/1453858123 learned_addr learned my addr 172.21.15.187:0/1453858123 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:02.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29cb5f700 1 --2- 172.21.15.187:0/1453858123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298065190 0x7fe298065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:02.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe28ffff700 1 --2- 172.21.15.187:0/1453858123 >> v2:172.21.15.5:3300/0 conn(0x7fe298065f90 0x7fe298143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:02.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29d360700 1 -- 172.21.15.187:0/1453858123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298065190 msgr2=0x7fe298065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:02.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29d360700 1 --2- 172.21.15.187:0/1453858123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298065190 0x7fe298065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:02.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29d360700 1 -- 172.21.15.187:0/1453858123 >> v2:172.21.15.5:3300/0 conn(0x7fe298065f90 msgr2=0x7fe298143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:02.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29d360700 1 --2- 172.21.15.187:0/1453858123 >> v2:172.21.15.5:3300/0 conn(0x7fe298065f90 0x7fe298143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:02.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe29d360700 1 -- 172.21.15.187:0/1453858123 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2980b1640 con 0x7fe298143c70 2020-11-20T12:21:02.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.973+0000 7fe28ffff700 1 --2- 172.21.15.187:0/1453858123 >> v2:172.21.15.5:3300/0 conn(0x7fe298065f90 0x7fe298143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:02.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.974+0000 7fe29d360700 1 --2- 172.21.15.187:0/1453858123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298143c70 0x7fe29809ca90 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7fe2980b6390 tx=0x7fe290009290).ready entity=mon.1 client_cookie=d653e20be0eee7ce server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:02.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.974+0000 7fe28f7fe700 1 -- 172.21.15.187:0/1453858123 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29000bc50 con 0x7fe298143c70 2020-11-20T12:21:02.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.974+0000 7fe28f7fe700 1 -- 172.21.15.187:0/1453858123 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29000bdb0 con 0x7fe298143c70 2020-11-20T12:21:02.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.974+0000 7fe28f7fe700 1 -- 172.21.15.187:0/1453858123 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290011610 con 0x7fe298143c70 2020-11-20T12:21:02.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 -- 172.21.15.187:0/1453858123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298143c70 msgr2=0x7fe29809ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:02.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 --2- 172.21.15.187:0/1453858123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298143c70 0x7fe29809ca90 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7fe2980b6390 tx=0x7fe290009290).stop 2020-11-20T12:21:02.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 -- 172.21.15.187:0/1453858123 shutdown_connections 2020-11-20T12:21:02.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 --2- 172.21.15.187:0/1453858123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298143c70 0x7fe29809ca90 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:02.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 --2- 172.21.15.187:0/1453858123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298065190 0x7fe298065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:02.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 --2- 172.21.15.187:0/1453858123 >> v2:172.21.15.5:3300/0 conn(0x7fe298065f90 0x7fe298143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:02.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 -- 172.21.15.187:0/1453858123 >> 172.21.15.187:0/1453858123 conn(0x7fe2980bab30 msgr2=0x7fe2980baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:02.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 -- 172.21.15.187:0/1453858123 shutdown_connections 2020-11-20T12:21:02.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.975+0000 7fe29db61700 1 -- 172.21.15.187:0/1453858123 wait complete. 2020-11-20T12:21:02.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.976+0000 7fe29db61700 1 Processor -- start 2020-11-20T12:21:02.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.977+0000 7fe29db61700 1 -- start start 2020-11-20T12:21:02.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.977+0000 7fe29db61700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298065190 0x7fe2980b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:02.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29db61700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe298065f90 0x7fe29809bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:02.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29db61700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298143c70 0x7fe298140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:02.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29db61700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29809f190 con 0x7fe298065f90 2020-11-20T12:21:02.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29db61700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29809f430 con 0x7fe298065190 2020-11-20T12:21:02.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29db61700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29809f6d0 con 0x7fe298143c70 2020-11-20T12:21:02.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29cb5f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298065190 0x7fe2980b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:02.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29cb5f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298065190 0x7fe2980b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54018/0 (socket says 172.21.15.187:54018) 2020-11-20T12:21:02.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29cb5f700 1 -- 172.21.15.187:0/260278069 learned_addr learned my addr 172.21.15.187:0/260278069 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:02.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe28ffff700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe298065f90 0x7fe29809bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:02.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29cb5f700 1 -- 172.21.15.187:0/260278069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298143c70 msgr2=0x7fe298140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:02.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29cb5f700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298143c70 0x7fe298140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:02.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29cb5f700 1 -- 172.21.15.187:0/260278069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe298065f90 msgr2=0x7fe29809bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:02.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29cb5f700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe298065f90 0x7fe29809bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:02.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.978+0000 7fe29cb5f700 1 -- 172.21.15.187:0/260278069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2980b23f0 con 0x7fe298065190 2020-11-20T12:21:02.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.979+0000 7fe29cb5f700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298065190 0x7fe2980b45c0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7fe294006cf0 tx=0x7fe294008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:02.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.979+0000 7fe28cff9700 1 -- 172.21.15.187:0/260278069 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2940117e0 con 0x7fe298065190 2020-11-20T12:21:02.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.979+0000 7fe28cff9700 1 -- 172.21.15.187:0/260278069 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe294011940 con 0x7fe298065190 2020-11-20T12:21:02.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.979+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29809c0f0 con 0x7fe298065190 2020-11-20T12:21:02.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.979+0000 7fe28cff9700 1 -- 172.21.15.187:0/260278069 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294011ce0 con 0x7fe298065190 2020-11-20T12:21:02.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.979+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe298146670 con 0x7fe298065190 2020-11-20T12:21:02.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.980+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe298146920 con 0x7fe298065190 2020-11-20T12:21:02.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.981+0000 7fe28cff9700 1 -- 172.21.15.187:0/260278069 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe294021060 con 0x7fe298065190 2020-11-20T12:21:02.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.982+0000 7fe28cff9700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe270056d20 0x7fe2700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:02.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.982+0000 7fe28cff9700 1 -- 172.21.15.187:0/260278069 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe294074030 con 0x7fe298065190 2020-11-20T12:21:02.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.982+0000 7fe28ffff700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe270056d20 0x7fe2700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:02.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.982+0000 7fe28ffff700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe270056d20 0x7fe2700591e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7fe298142c10 tx=0x7fe288008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:02.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:02.985+0000 7fe28cff9700 1 -- 172.21.15.187:0/260278069 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe294049430 con 0x7fe298065190 2020-11-20T12:21:03.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.172+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe298061a20 con 0x7fe298065190 2020-11-20T12:21:03.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.182+0000 7fe28cff9700 1 -- 172.21.15.187:0/260278069 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe29401f070 con 0x7fe298065190 2020-11-20T12:21:03.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe270056d20 msgr2=0x7fe2700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:03.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe270056d20 0x7fe2700591e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7fe298142c10 tx=0x7fe288008040).stop 2020-11-20T12:21:03.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298065190 msgr2=0x7fe2980b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:03.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298065190 0x7fe2980b45c0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7fe294006cf0 tx=0x7fe294008040).stop 2020-11-20T12:21:03.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 shutdown_connections 2020-11-20T12:21:03.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe270056d20 0x7fe2700591e0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe298143c70 0x7fe298140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:03.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe298065190 0x7fe2980b45c0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 --2- 172.21.15.187:0/260278069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe298065f90 0x7fe29809bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 >> 172.21.15.187:0/260278069 conn(0x7fe2980bab30 msgr2=0x7fe29809cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:03.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 shutdown_connections 2020-11-20T12:21:03.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.185+0000 7fe29db61700 1 -- 172.21.15.187:0/260278069 wait complete. 2020-11-20T12:21:03.210 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed' 2020-11-20T12:21:03.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.437+0000 7ff8f65c8700 1 Processor -- start 2020-11-20T12:21:03.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.438+0000 7ff8f65c8700 1 -- start start 2020-11-20T12:21:03.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.439+0000 7ff8f65c8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8f0068340 0x7ff8f00afc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:03.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.439+0000 7ff8f65c8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f0061e80 con 0x7ff8f00b7180 2020-11-20T12:21:03.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.439+0000 7ff8f65c8700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f00620b0 con 0x7ff8f0067f60 2020-11-20T12:21:03.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.439+0000 7ff8f65c8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f0062350 con 0x7ff8f0068340 2020-11-20T12:21:03.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.440+0000 7ff8e77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8f0068340 0x7ff8f00afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:03.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.440+0000 7ff8e77fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8f0068340 0x7ff8f00afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54028/0 (socket says 172.21.15.187:54028) 2020-11-20T12:21:03.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.440+0000 7ff8e77fe700 1 -- 172.21.15.187:0/3370991174 learned_addr learned my addr 172.21.15.187:0/3370991174 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:03.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.440+0000 7ff8e77fe700 1 -- 172.21.15.187:0/3370991174 >> v1:172.21.15.187:6789/0 conn(0x7ff8f0067f60 legacy=0x7ff8f0065510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:03.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.440+0000 7ff8e77fe700 1 -- 172.21.15.187:0/3370991174 >> v1:172.21.15.5:6789/0 conn(0x7ff8f00b7180 legacy=0x7ff8f00b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:03.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.440+0000 7ff8e77fe700 1 -- 172.21.15.187:0/3370991174 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00668f0 con 0x7ff8f0068340 2020-11-20T12:21:03.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.441+0000 7ff8e77fe700 1 --2- 172.21.15.187:0/3370991174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0068340 0x7ff8f00afc30 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7ff8d8006fd0 tx=0x7ff8d8006e50).ready entity=mon.1 client_cookie=82bcb3a85a7d3835 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:03.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.441+0000 7ff8ef7fe700 1 -- 172.21.15.187:0/3370991174 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8006030 con 0x7ff8f0068340 2020-11-20T12:21:03.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.441+0000 7ff8ef7fe700 1 -- 172.21.15.187:0/3370991174 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8006190 con 0x7ff8f0068340 2020-11-20T12:21:03.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.441+0000 7ff8ef7fe700 1 -- 172.21.15.187:0/3370991174 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8006510 con 0x7ff8f0068340 2020-11-20T12:21:03.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.442+0000 7ff8f65c8700 1 -- 172.21.15.187:0/3370991174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0068340 msgr2=0x7ff8f00afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:03.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.442+0000 7ff8f65c8700 1 --2- 172.21.15.187:0/3370991174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0068340 0x7ff8f00afc30 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7ff8d8006fd0 tx=0x7ff8d8006e50).stop 2020-11-20T12:21:03.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.442+0000 7ff8f65c8700 1 -- 172.21.15.187:0/3370991174 shutdown_connections 2020-11-20T12:21:03.453 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.442+0000 7ff8f65c8700 1 --2- 172.21.15.187:0/3370991174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0068340 0x7ff8f00afc30 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.442+0000 7ff8f65c8700 1 -- 172.21.15.187:0/3370991174 >> 172.21.15.187:0/3370991174 conn(0x7ff8f00bacd0 msgr2=0x7ff8f00bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:03.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.442+0000 7ff8f65c8700 1 -- 172.21.15.187:0/3370991174 shutdown_connections 2020-11-20T12:21:03.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.442+0000 7ff8f65c8700 1 -- 172.21.15.187:0/3370991174 wait complete. 2020-11-20T12:21:03.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.443+0000 7ff8f65c8700 1 Processor -- start 2020-11-20T12:21:03.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.443+0000 7ff8f65c8700 1 -- start start 2020-11-20T12:21:03.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.444+0000 7ff8f65c8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0067f60 0x7ff8f009a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:03.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.444+0000 7ff8f65c8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8f0068340 0x7ff8f00a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:03.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.444+0000 7ff8f65c8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8f00b7180 0x7ff8f01460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:03.455 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.444+0000 7ff8f65c8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f009d1a0 con 0x7ff8f00b7180 2020-11-20T12:21:03.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.444+0000 7ff8f65c8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0149020 con 0x7ff8f0067f60 2020-11-20T12:21:03.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.444+0000 7ff8f65c8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f01492c0 con 0x7ff8f0068340 2020-11-20T12:21:03.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.444+0000 7ff8effff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0067f60 0x7ff8f009a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:03.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.445+0000 7ff8effff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0067f60 0x7ff8f009a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54032/0 (socket says 172.21.15.187:54032) 2020-11-20T12:21:03.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.445+0000 7ff8effff700 1 -- 172.21.15.187:0/4037413903 learned_addr learned my addr 172.21.15.187:0/4037413903 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:03.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.445+0000 7ff8f4b65700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8f00b7180 0x7ff8f01460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:03.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.445+0000 7ff8e77fe700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8f0068340 0x7ff8f00a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:03.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.446+0000 7ff8effff700 1 -- 172.21.15.187:0/4037413903 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8f0068340 msgr2=0x7ff8f00a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:03.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.446+0000 7ff8effff700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8f0068340 0x7ff8f00a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.446+0000 7ff8effff700 1 -- 172.21.15.187:0/4037413903 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8f00b7180 msgr2=0x7ff8f01460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:03.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.446+0000 7ff8effff700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8f00b7180 0x7ff8f01460c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.446+0000 7ff8effff700 1 -- 172.21.15.187:0/4037413903 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f0149560 con 0x7ff8f0067f60 2020-11-20T12:21:03.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.446+0000 7ff8effff700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0067f60 0x7ff8f009a670 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7ff8e000c760 tx=0x7ff8e0007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:03.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.446+0000 7ff8ecff9700 1 -- 172.21.15.187:0/4037413903 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e000e560 con 0x7ff8f0067f60 2020-11-20T12:21:03.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.447+0000 7ff8ecff9700 1 -- 172.21.15.187:0/4037413903 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e000e6c0 con 0x7ff8f0067f60 2020-11-20T12:21:03.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.447+0000 7ff8ecff9700 1 -- 172.21.15.187:0/4037413903 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e000ea40 con 0x7ff8f0067f60 2020-11-20T12:21:03.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.447+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f009abc0 con 0x7ff8f0067f60 2020-11-20T12:21:03.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.447+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f0148b10 con 0x7ff8f0067f60 2020-11-20T12:21:03.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.448+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f0063600 con 0x7ff8f0067f60 2020-11-20T12:21:03.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.448+0000 7ff8ecff9700 1 -- 172.21.15.187:0/4037413903 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff8e0010070 con 0x7ff8f0067f60 2020-11-20T12:21:03.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.449+0000 7ff8ecff9700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff8cc056d20 0x7ff8cc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:03.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.449+0000 7ff8ecff9700 1 -- 172.21.15.187:0/4037413903 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff8e006fb40 con 0x7ff8f0067f60 2020-11-20T12:21:03.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.449+0000 7ff8e77fe700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff8cc056d20 0x7ff8cc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:03.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.450+0000 7ff8e77fe700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff8cc056d20 0x7ff8cc0591e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7ff8d800b1b0 tx=0x7ff8d8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:03.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.453+0000 7ff8ecff9700 1 -- 172.21.15.187:0/4037413903 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff8e0044f40 con 0x7ff8f0067f60 2020-11-20T12:21:03.717 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.714+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed"]} v 0) v1 -- 0x7ff8f0065c20 con 0x7ff8f0067f60 2020-11-20T12:21:03.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.935+0000 7ff8ecff9700 1 -- 172.21.15.187:0/4037413903 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed"]}]=0 v808) v1 ==== 143+0+0 (secure 0 0 0) 0x7ff8e0048560 con 0x7ff8f0067f60 2020-11-20T12:21:03.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.938+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff8cc056d20 msgr2=0x7ff8cc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.938+0000 7ff8f65c8700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff8cc056d20 0x7ff8cc0591e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7ff8d800b1b0 tx=0x7ff8d8015000).stop 2020-11-20T12:21:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.938+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0067f60 msgr2=0x7ff8f009a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.938+0000 7ff8f65c8700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0067f60 0x7ff8f009a670 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7ff8e000c760 tx=0x7ff8e0007c30).stop 2020-11-20T12:21:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.939+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 shutdown_connections 2020-11-20T12:21:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.939+0000 7ff8f65c8700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff8cc056d20 0x7ff8cc0591e0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.939+0000 7ff8f65c8700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8f00b7180 0x7ff8f01460c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.939+0000 7ff8f65c8700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8f0067f60 0x7ff8f009a670 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.939+0000 7ff8f65c8700 1 --2- 172.21.15.187:0/4037413903 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff8f0068340 0x7ff8f00a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:03.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.939+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 >> 172.21.15.187:0/4037413903 conn(0x7ff8f00bacd0 msgr2=0x7ff8f00bb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:03.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.939+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 shutdown_connections 2020-11-20T12:21:03.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:03.939+0000 7ff8f65c8700 1 -- 172.21.15.187:0/4037413903 wait complete. 2020-11-20T12:21:03.972 INFO:tasks.cephfs_test_runner:test_replicated_delete_speed (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:21:03.973 INFO:tasks.cephfs_test_runner:Starting test: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2020-11-20T12:21:03.974 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove' 2020-11-20T12:21:04.222 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.218+0000 7f940ddd1700 1 Processor -- start 2020-11-20T12:21:04.224 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.218+0000 7f940ddd1700 1 -- start start 2020-11-20T12:21:04.229 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.220+0000 7f940ddd1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9408068230 0x7f94081482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.220+0000 7f940ddd1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9408144b80 con 0x7f94080b5f50 2020-11-20T12:21:04.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.220+0000 7f940ddd1700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94081489e0 con 0x7f94080b57b0 2020-11-20T12:21:04.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.220+0000 7f940ddd1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94081487c0 con 0x7f9408068230 2020-11-20T12:21:04.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.220+0000 7f940cdcf700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f94080b57b0 0x7f9408062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50648/0 (socket says 172.21.15.187:50648) 2020-11-20T12:21:04.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.220+0000 7f940cdcf700 1 -- 172.21.15.187:0/735489929 learned_addr learned my addr 172.21.15.187:0/735489929 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:04.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.220+0000 7f93fffff700 1 --2- 172.21.15.187:0/735489929 >> v2:172.21.15.187:3300/0 conn(0x7f9408068230 0x7f94081482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93ff7fe700 1 -- 172.21.15.187:0/735489929 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211355668 0 0) 0x7f94081489e0 con 0x7f94080b57b0 2020-11-20T12:21:04.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93ff7fe700 1 -- 172.21.15.187:0/735489929 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f4003240 con 0x7f94080b57b0 2020-11-20T12:21:04.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93fffff700 1 -- 172.21.15.187:0/735489929 >> v1:172.21.15.187:6789/0 conn(0x7f94080b57b0 legacy=0x7f9408062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93fffff700 1 -- 172.21.15.187:0/735489929 >> v1:172.21.15.5:6789/0 conn(0x7f94080b5f50 legacy=0x7f94080b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:04.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93fffff700 1 -- 172.21.15.187:0/735489929 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408148c80 con 0x7f9408068230 2020-11-20T12:21:04.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93fffff700 1 --2- 172.21.15.187:0/735489929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f94081482d0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f93f800b9a0 tx=0x7f93f8006df0).ready entity=mon.1 client_cookie=272dadaab3131e45 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:04.239 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93ff7fe700 1 -- 172.21.15.187:0/735489929 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f80097b0 con 0x7f9408068230 2020-11-20T12:21:04.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93ff7fe700 1 -- 172.21.15.187:0/735489929 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f8009910 con 0x7f9408068230 2020-11-20T12:21:04.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.221+0000 7f93ff7fe700 1 -- 172.21.15.187:0/735489929 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8009c90 con 0x7f9408068230 2020-11-20T12:21:04.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.222+0000 7f940ddd1700 1 -- 172.21.15.187:0/735489929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 msgr2=0x7f94081482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.222+0000 7f940ddd1700 1 --2- 172.21.15.187:0/735489929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f94081482d0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f93f800b9a0 tx=0x7f93f8006df0).stop 2020-11-20T12:21:04.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.222+0000 7f940ddd1700 1 -- 172.21.15.187:0/735489929 shutdown_connections 2020-11-20T12:21:04.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.222+0000 7f940ddd1700 1 --2- 172.21.15.187:0/735489929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f94081482d0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.222+0000 7f940ddd1700 1 -- 172.21.15.187:0/735489929 >> 172.21.15.187:0/735489929 conn(0x7f94080bab30 msgr2=0x7f94080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:04.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.222+0000 7f940ddd1700 1 -- 172.21.15.187:0/735489929 shutdown_connections 2020-11-20T12:21:04.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.222+0000 7f940ddd1700 1 -- 172.21.15.187:0/735489929 wait complete. 2020-11-20T12:21:04.256 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.224+0000 7f940ddd1700 1 Processor -- start 2020-11-20T12:21:04.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.224+0000 7f940ddd1700 1 -- start start 2020-11-20T12:21:04.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940ddd1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f940809f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940ddd1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94080b57b0 0x7f940809cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940ddd1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94080b5f50 0x7f940809d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940ddd1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9408141970 con 0x7f94080b57b0 2020-11-20T12:21:04.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940ddd1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940809d630 con 0x7f9408068230 2020-11-20T12:21:04.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940ddd1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9408158670 con 0x7f94080b5f50 2020-11-20T12:21:04.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940cdcf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f940809f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940cdcf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f940809f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54046/0 (socket says 172.21.15.187:54046) 2020-11-20T12:21:04.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f940cdcf700 1 -- 172.21.15.187:0/37885462 learned_addr learned my addr 172.21.15.187:0/37885462 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:04.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.225+0000 7f93fffff700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94080b57b0 0x7f940809cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f940cdcf700 1 -- 172.21.15.187:0/37885462 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94080b5f50 msgr2=0x7f940809d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f940cdcf700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94080b5f50 0x7f940809d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:04.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f940cdcf700 1 -- 172.21.15.187:0/37885462 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94080b57b0 msgr2=0x7f940809cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f940cdcf700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94080b57b0 0x7f940809cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f940cdcf700 1 -- 172.21.15.187:0/37885462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408158910 con 0x7f9408068230 2020-11-20T12:21:04.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f940cdcf700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f940809f560 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7f940400ca60 tx=0x7f9404007e50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:04.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f93fcff9700 1 -- 172.21.15.187:0/37885462 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404005b80 con 0x7f9408068230 2020-11-20T12:21:04.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f93fcff9700 1 -- 172.21.15.187:0/37885462 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9404005ce0 con 0x7f9408068230 2020-11-20T12:21:04.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f93fcff9700 1 -- 172.21.15.187:0/37885462 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404007340 con 0x7f9408068230 2020-11-20T12:21:04.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9408158bb0 con 0x7f9408068230 2020-11-20T12:21:04.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.226+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940809fa50 con 0x7f9408068230 2020-11-20T12:21:04.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.227+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9408061a20 con 0x7f9408068230 2020-11-20T12:21:04.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.228+0000 7f93fcff9700 1 -- 172.21.15.187:0/37885462 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9404010030 con 0x7f9408068230 2020-11-20T12:21:04.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.229+0000 7f93fcff9700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f93e0056d20 0x7f93e00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.229+0000 7f93fcff9700 1 -- 172.21.15.187:0/37885462 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f94040701d0 con 0x7f9408068230 2020-11-20T12:21:04.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.229+0000 7f93fffff700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f93e0056d20 0x7f93e00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.230+0000 7f93fffff700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f93e0056d20 0x7f93e00591e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f93f800be60 tx=0x7f93f8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:04.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.233+0000 7f93fcff9700 1 -- 172.21.15.187:0/37885462 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9404045550 con 0x7f9408068230 2020-11-20T12:21:04.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.488+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove"]} v 0) v1 -- 0x7f9408158d40 con 0x7f9408068230 2020-11-20T12:21:05.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.937+0000 7f93fcff9700 1 -- 172.21.15.187:0/37885462 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove"]}]=0 v809) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9404048b70 con 0x7f9408068230 2020-11-20T12:21:05.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f93e0056d20 msgr2=0x7f93e00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:05.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f93e0056d20 0x7f93e00591e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f93f800be60 tx=0x7f93f8015000).stop 2020-11-20T12:21:05.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 msgr2=0x7f940809f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:05.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f940809f560 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7f940400ca60 tx=0x7f9404007e50).stop 2020-11-20T12:21:05.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 shutdown_connections 2020-11-20T12:21:05.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f93e0056d20 0x7f93e00591e0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:05.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94080b5f50 0x7f940809d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:05.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9408068230 0x7f940809f560 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:05.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 --2- 172.21.15.187:0/37885462 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94080b57b0 0x7f940809cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:05.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 >> 172.21.15.187:0/37885462 conn(0x7f94080bab30 msgr2=0x7f9408099c00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:05.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 shutdown_connections 2020-11-20T12:21:05.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:04.942+0000 7f940ddd1700 1 -- 172.21.15.187:0/37885462 wait complete. 2020-11-20T12:21:05.891 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2020-11-20T12:21:06.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.149+0000 7fa0316e8700 1 Processor -- start 2020-11-20T12:21:06.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.149+0000 7fa0316e8700 1 -- start start 2020-11-20T12:21:06.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.150+0000 7fa0316e8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.150+0000 7fa0316e8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa02c065f90 0x7fa02c143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.150+0000 7fa0316e8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02c0b13e0 con 0x7fa02c143c70 2020-11-20T12:21:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.150+0000 7fa0316e8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02c14ab30 con 0x7fa02c065f90 2020-11-20T12:21:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.150+0000 7fa0316e8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c062c10 con 0x7fa02c065190 2020-11-20T12:21:06.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.151+0000 7fa0237fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa02c065f90 0x7fa02c143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.151+0000 7fa0237fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa02c065f90 0x7fa02c143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54058/0 (socket says 172.21.15.187:54058) 2020-11-20T12:21:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.151+0000 7fa0237fe700 1 -- 172.21.15.187:0/3585734645 learned_addr learned my addr 172.21.15.187:0/3585734645 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.151+0000 7fa02bfff700 1 --2- 172.21.15.187:0/3585734645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.152+0000 7fa0237fe700 1 -- 172.21.15.187:0/3585734645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 msgr2=0x7fa02c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:06.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.152+0000 7fa0237fe700 1 --2- 172.21.15.187:0/3585734645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.152+0000 7fa0237fe700 1 -- 172.21.15.187:0/3585734645 >> v1:172.21.15.5:6789/0 conn(0x7fa02c143c70 legacy=0x7fa02c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.152+0000 7fa0237fe700 1 -- 172.21.15.187:0/3585734645 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c0b1640 con 0x7fa02c065f90 2020-11-20T12:21:06.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.152+0000 7fa02bfff700 1 --2- 172.21.15.187:0/3585734645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:06.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.152+0000 7fa0237fe700 1 --2- 172.21.15.187:0/3585734645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c065f90 0x7fa02c143780 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7fa02c14ac70 tx=0x7fa014006eb0).ready entity=mon.1 client_cookie=b73f43e15808ccd7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa02b7fe700 1 -- 172.21.15.187:0/3585734645 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014009960 con 0x7fa02c065f90 2020-11-20T12:21:06.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa02b7fe700 1 -- 172.21.15.187:0/3585734645 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa014009ac0 con 0x7fa02c065f90 2020-11-20T12:21:06.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa02b7fe700 1 -- 172.21.15.187:0/3585734645 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014009e40 con 0x7fa02c065f90 2020-11-20T12:21:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa0316e8700 1 -- 172.21.15.187:0/3585734645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c065f90 msgr2=0x7fa02c143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa0316e8700 1 --2- 172.21.15.187:0/3585734645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c065f90 0x7fa02c143780 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7fa02c14ac70 tx=0x7fa014006eb0).stop 2020-11-20T12:21:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa0316e8700 1 -- 172.21.15.187:0/3585734645 shutdown_connections 2020-11-20T12:21:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa0316e8700 1 --2- 172.21.15.187:0/3585734645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa0316e8700 1 --2- 172.21.15.187:0/3585734645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c065f90 0x7fa02c143780 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa0316e8700 1 -- 172.21.15.187:0/3585734645 >> 172.21.15.187:0/3585734645 conn(0x7fa02c0bab30 msgr2=0x7fa02c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa0316e8700 1 -- 172.21.15.187:0/3585734645 shutdown_connections 2020-11-20T12:21:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.153+0000 7fa0316e8700 1 -- 172.21.15.187:0/3585734645 wait complete. 2020-11-20T12:21:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.155+0000 7fa0316e8700 1 Processor -- start 2020-11-20T12:21:06.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.155+0000 7fa0316e8700 1 -- start start 2020-11-20T12:21:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.156+0000 7fa0316e8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c1479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.156+0000 7fa0316e8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa02c065f90 0x7fa02c09c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.156+0000 7fa0316e8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c143c70 0x7fa02c1466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.156+0000 7fa0316e8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c146fb0 con 0x7fa02c065f90 2020-11-20T12:21:06.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.156+0000 7fa0316e8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c147250 con 0x7fa02c143c70 2020-11-20T12:21:06.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.156+0000 7fa0316e8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1474f0 con 0x7fa02c065190 2020-11-20T12:21:06.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa030ee7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c143c70 0x7fa02c1466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa030ee7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c143c70 0x7fa02c1466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54064/0 (socket says 172.21.15.187:54064) 2020-11-20T12:21:06.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa030ee7700 1 -- 172.21.15.187:0/1018556063 learned_addr learned my addr 172.21.15.187:0/1018556063 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:06.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa0237fe700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa02c065f90 0x7fa02c09c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa02bfff700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c1479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa030ee7700 1 -- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 msgr2=0x7fa02c1479a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa030ee7700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c1479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa030ee7700 1 -- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa02c065f90 msgr2=0x7fa02c09c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa030ee7700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa02c065f90 0x7fa02c09c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa030ee7700 1 -- 172.21.15.187:0/1018556063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c09c800 con 0x7fa02c143c70 2020-11-20T12:21:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.157+0000 7fa0237fe700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa02c065f90 0x7fa02c09c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.158+0000 7fa030ee7700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c143c70 0x7fa02c1466e0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7fa02400ca90 tx=0x7fa024001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.158+0000 7fa028ff9700 1 -- 172.21.15.187:0/1018556063 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02400a730 con 0x7fa02c143c70 2020-11-20T12:21:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.158+0000 7fa028ff9700 1 -- 172.21.15.187:0/1018556063 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02400a890 con 0x7fa02c143c70 2020-11-20T12:21:06.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.158+0000 7fa028ff9700 1 -- 172.21.15.187:0/1018556063 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02400ac30 con 0x7fa02c143c70 2020-11-20T12:21:06.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.158+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02c146bd0 con 0x7fa02c143c70 2020-11-20T12:21:06.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.158+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02c09e740 con 0x7fa02c143c70 2020-11-20T12:21:06.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.159+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02c061a20 con 0x7fa02c143c70 2020-11-20T12:21:06.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.160+0000 7fa028ff9700 1 -- 172.21.15.187:0/1018556063 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa024010070 con 0x7fa02c143c70 2020-11-20T12:21:06.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.160+0000 7fa028ff9700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa008056d20 0x7fa0080591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.161+0000 7fa028ff9700 1 -- 172.21.15.187:0/1018556063 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa02406fde0 con 0x7fa02c143c70 2020-11-20T12:21:06.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.161+0000 7fa02bfff700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa008056d20 0x7fa0080591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.161+0000 7fa02bfff700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa008056d20 0x7fa0080591e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7fa01c006bd0 tx=0x7fa01c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.164+0000 7fa028ff9700 1 -- 172.21.15.187:0/1018556063 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa024045160 con 0x7fa02c143c70 2020-11-20T12:21:06.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.348+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"} v 0) v1 -- 0x7fa02c0afc60 con 0x7fa02c143c70 2020-11-20T12:21:06.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.360+0000 7fa028ff9700 1 -- 172.21.15.187:0/1018556063 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config set", "who": "mon", "name": "mon_allow_pool_delete", "value": "True"}]=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa024048780 con 0x7fa02c143c70 2020-11-20T12:21:06.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa008056d20 msgr2=0x7fa0080591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa008056d20 0x7fa0080591e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7fa01c006bd0 tx=0x7fa01c008040).stop 2020-11-20T12:21:06.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c143c70 msgr2=0x7fa02c1466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c143c70 0x7fa02c1466e0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7fa02400ca90 tx=0x7fa024001360).stop 2020-11-20T12:21:06.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 shutdown_connections 2020-11-20T12:21:06.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa008056d20 0x7fa0080591e0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa02c143c70 0x7fa02c1466e0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa02c065190 0x7fa02c1479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 --2- 172.21.15.187:0/1018556063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa02c065f90 0x7fa02c09c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.362+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 >> 172.21.15.187:0/1018556063 conn(0x7fa02c0bab30 msgr2=0x7fa02c0b9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:06.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.363+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 shutdown_connections 2020-11-20T12:21:06.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.363+0000 7fa0316e8700 1 -- 172.21.15.187:0/1018556063 wait complete. 2020-11-20T12:21:06.389 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:21:06.390 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:21:06.510 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:21:06.527 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:21:06.637 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:21:06.654 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:21:06.654 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2020-11-20T12:21:06.655 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:06.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.888+0000 7f4f8265b700 1 Processor -- start 2020-11-20T12:21:06.891 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.888+0000 7f4f8265b700 1 -- start start 2020-11-20T12:21:06.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.889+0000 7f4f8265b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0b6650 0x7f4f7c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.889+0000 7f4f8265b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c0b1300 con 0x7f4f7c066f70 2020-11-20T12:21:06.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.889+0000 7f4f8265b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c0b15a0 con 0x7f4f7c1484d0 2020-11-20T12:21:06.892 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.889+0000 7f4f8265b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7c067730 con 0x7f4f7c0b6650 2020-11-20T12:21:06.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.890+0000 7f4f81659700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4f7c1484d0 0x7f4f7c061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50674/0 (socket says 172.21.15.187:50674) 2020-11-20T12:21:06.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.890+0000 7f4f81659700 1 -- 172.21.15.187:0/562759035 learned_addr learned my addr 172.21.15.187:0/562759035 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:06.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.890+0000 7f4f80e58700 1 --2- 172.21.15.187:0/562759035 >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0b6650 0x7f4f7c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599028759 0 0) 0x7f4f7c0b15a0 con 0x7f4f7c1484d0 2020-11-20T12:21:06.894 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f64003240 con 0x7f4f7c1484d0 2020-11-20T12:21:06.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1697981535 0 0) 0x7f4f64003240 con 0x7f4f7c1484d0 2020-11-20T12:21:06.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0b6650 msgr2=0x7f4f7c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:06.896 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 --2- 172.21.15.187:0/562759035 >> v2:172.21.15.5:3300/0 conn(0x7f4f7c0b6650 0x7f4f7c0afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 >> v1:172.21.15.5:6789/0 conn(0x7f4f7c066f70 legacy=0x7f4f7c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0b1840 con 0x7f4f7c1484d0 2020-11-20T12:21:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f4f780014e0 con 0x7f4f7c1484d0 2020-11-20T12:21:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 >> v1:172.21.15.187:6789/0 conn(0x7f4f7c1484d0 legacy=0x7f4f7c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f64003730 0x7f4f64003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f64004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f64004e50 0x7f4f6400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c067730 con 0x7f4f64004e50 2020-11-20T12:21:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f64004260 con 0x7f4f640044e0 2020-11-20T12:21:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.891+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6400bc20 con 0x7f4f64003730 2020-11-20T12:21:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.892+0000 7f4f80e58700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f64004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.892+0000 7f4f80e58700 1 -- 172.21.15.187:0/562759035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f64003730 msgr2=0x7f4f64003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T12:21:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.892+0000 7f4f80e58700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f64003730 0x7f4f64003b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:06.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.892+0000 7f4f80e58700 1 -- 172.21.15.187:0/562759035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f64004e50 msgr2=0x7f4f6400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:06.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.892+0000 7f4f80e58700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f64004e50 0x7f4f6400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:06.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.892+0000 7f4f80e58700 1 -- 172.21.15.187:0/562759035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f64003240 con 0x7f4f640044e0 2020-11-20T12:21:06.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.892+0000 7f4f81e5a700 1 -- 172.21.15.187:0/562759035 reap_dead start 2020-11-20T12:21:06.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.892+0000 7f4f80e58700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f64004960 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f4f7000bc00 tx=0x7f4f70006df0).ready entity=mon.1 client_cookie=e430c5af6b456dc9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.893+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000b310 con 0x7f4f640044e0 2020-11-20T12:21:06.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.893+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7000eb40 con 0x7f4f640044e0 2020-11-20T12:21:06.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.893+0000 7f4f6ffff700 1 -- 172.21.15.187:0/562759035 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000ecb0 con 0x7f4f640044e0 2020-11-20T12:21:06.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.893+0000 7f4f8265b700 1 -- 172.21.15.187:0/562759035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 msgr2=0x7f4f64004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.893+0000 7f4f8265b700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f64004960 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f4f7000bc00 tx=0x7f4f70006df0).stop 2020-11-20T12:21:06.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.893+0000 7f4f8265b700 1 -- 172.21.15.187:0/562759035 shutdown_connections 2020-11-20T12:21:06.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.893+0000 7f4f8265b700 1 --2- 172.21.15.187:0/562759035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f64004960 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.893+0000 7f4f8265b700 1 -- 172.21.15.187:0/562759035 >> 172.21.15.187:0/562759035 conn(0x7f4f7c0bab30 msgr2=0x7f4f7c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:06.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.894+0000 7f4f8265b700 1 -- 172.21.15.187:0/562759035 shutdown_connections 2020-11-20T12:21:06.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.894+0000 7f4f8265b700 1 -- 172.21.15.187:0/562759035 wait complete. 2020-11-20T12:21:06.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.895+0000 7f4f8265b700 1 Processor -- start 2020-11-20T12:21:06.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.895+0000 7f4f8265b700 1 -- start start 2020-11-20T12:21:06.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.896+0000 7f4f8265b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f7c09e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.896+0000 7f4f8265b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f7c09c900 0x7f4f7c136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.896+0000 7f4f8265b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f7c137430 0x7f4f7c15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.896+0000 7f4f8265b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c0b1cf0 con 0x7f4f7c09c900 2020-11-20T12:21:06.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.896+0000 7f4f8265b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c09f1a0 con 0x7f4f640044e0 2020-11-20T12:21:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.896+0000 7f4f8265b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c137d90 con 0x7f4f7c137430 2020-11-20T12:21:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.896+0000 7f4f81659700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f7c09e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f81659700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f7c09e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54080/0 (socket says 172.21.15.187:54080) 2020-11-20T12:21:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f81659700 1 -- 172.21.15.187:0/817904519 learned_addr learned my addr 172.21.15.187:0/817904519 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:06.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f81659700 1 -- 172.21.15.187:0/817904519 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f7c137430 msgr2=0x7f4f7c15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f81659700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f7c137430 0x7f4f7c15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f81659700 1 -- 172.21.15.187:0/817904519 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f7c09c900 msgr2=0x7f4f7c136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f81659700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f7c09c900 0x7f4f7c136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f81659700 1 -- 172.21.15.187:0/817904519 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c138030 con 0x7f4f640044e0 2020-11-20T12:21:06.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f81659700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f7c09e4b0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f4f7800ca60 tx=0x7f4f78001b10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.897+0000 7f4f6d7fa700 1 -- 172.21.15.187:0/817904519 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78007170 con 0x7f4f640044e0 2020-11-20T12:21:06.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.898+0000 7f4f6d7fa700 1 -- 172.21.15.187:0/817904519 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f780072d0 con 0x7f4f640044e0 2020-11-20T12:21:06.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.898+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c1382d0 con 0x7f4f640044e0 2020-11-20T12:21:06.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.898+0000 7f4f6d7fa700 1 -- 172.21.15.187:0/817904519 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7800aa40 con 0x7f4f640044e0 2020-11-20T12:21:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.898+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c09cd80 con 0x7f4f640044e0 2020-11-20T12:21:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.899+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c061a20 con 0x7f4f640044e0 2020-11-20T12:21:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.899+0000 7f4f6d7fa700 1 -- 172.21.15.187:0/817904519 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4f7801b060 con 0x7f4f640044e0 2020-11-20T12:21:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.900+0000 7f4f6d7fa700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.900+0000 7f4f6d7fa700 1 -- 172.21.15.187:0/817904519 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4f7806fbc0 con 0x7f4f640044e0 2020-11-20T12:21:06.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.900+0000 7f4f80e58700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.901+0000 7f4f80e58700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f4f70006e30 tx=0x7f4f70019000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:06.904+0000 7f4f6d7fa700 1 -- 172.21.15.187:0/817904519 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4f78044eb0 con 0x7f4f640044e0 2020-11-20T12:21:07.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.150+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f7c1488b0 con 0x7f4f640044e0 2020-11-20T12:21:07.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.151+0000 7f4f6d7fa700 1 -- 172.21.15.187:0/817904519 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+2158 (secure 0 0 0) 0x7f4f78019030 con 0x7f4f640044e0 2020-11-20T12:21:07.154 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:07.155 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":164,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:21:05.934493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:21:07.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 msgr2=0x7f4f540591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f4f70006e30 tx=0x7f4f70019000).stop 2020-11-20T12:21:07.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 msgr2=0x7f4f7c09e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f7c09e4b0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f4f7800ca60 tx=0x7f4f78001b10).stop 2020-11-20T12:21:07.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 shutdown_connections 2020-11-20T12:21:07.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4f54056d20 0x7f4f540591e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4f7c137430 0x7f4f7c15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:07.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f640044e0 0x7f4f7c09e4b0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 --2- 172.21.15.187:0/817904519 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f7c09c900 0x7f4f7c136f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:07.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 >> 172.21.15.187:0/817904519 conn(0x7f4f7c0bab30 msgr2=0x7f4f7c066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:07.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 shutdown_connections 2020-11-20T12:21:07.159 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.154+0000 7f4f8265b700 1 -- 172.21.15.187:0/817904519 wait complete. 2020-11-20T12:21:07.160 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 164 2020-11-20T12:21:07.181 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:21:07.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.418+0000 7f7ff2cc6700 1 Processor -- start 2020-11-20T12:21:07.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.418+0000 7f7ff2cc6700 1 -- start start 2020-11-20T12:21:07.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.419+0000 7f7ff2cc6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7fec14c750 0x7f7fec062580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.419+0000 7f7ff2cc6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.419+0000 7f7ff2cc6700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7fec062a70 0x7f7fec068220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.420+0000 7f7ff2cc6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec061e80 con 0x7f7fec14c750 2020-11-20T12:21:07.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.420+0000 7f7ff2cc6700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec0620e0 con 0x7f7fec062a70 2020-11-20T12:21:07.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.420+0000 7f7ff2cc6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0b1500 con 0x7f7fec0666e0 2020-11-20T12:21:07.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.420+0000 7f7ff1263700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7fec062a70 0x7f7fec068220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.420+0000 7f7ff1263700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7fec062a70 0x7f7fec068220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54092/0 (socket says 172.21.15.187:54092) 2020-11-20T12:21:07.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.420+0000 7f7ff1263700 1 -- 172.21.15.187:0/3261165065 learned_addr learned my addr 172.21.15.187:0/3261165065 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:07.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.420+0000 7f7fe3fff700 1 --2- 172.21.15.187:0/3261165065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.420+0000 7f7ff0a62700 1 --2- 172.21.15.187:0/3261165065 >> v2:172.21.15.5:3300/0 conn(0x7f7fec14c750 0x7f7fec062580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.421+0000 7f7ff1263700 1 -- 172.21.15.187:0/3261165065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 msgr2=0x7f7fec067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:07.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.421+0000 7f7ff1263700 1 --2- 172.21.15.187:0/3261165065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec067d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.421+0000 7f7ff1263700 1 -- 172.21.15.187:0/3261165065 >> v2:172.21.15.5:3300/0 conn(0x7f7fec14c750 msgr2=0x7f7fec062580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:07.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.421+0000 7f7ff1263700 1 --2- 172.21.15.187:0/3261165065 >> v2:172.21.15.5:3300/0 conn(0x7f7fec14c750 0x7f7fec062580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.421+0000 7f7ff1263700 1 -- 172.21.15.187:0/3261165065 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec0b17d0 con 0x7f7fec062a70 2020-11-20T12:21:07.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.421+0000 7f7ff1263700 1 --2- 172.21.15.187:0/3261165065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec062a70 0x7f7fec068220 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f7fec065ed0 tx=0x7f7fe4009290).ready entity=mon.1 client_cookie=b63b6e8216bd3fb1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:07.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.421+0000 7f7fe37fe700 1 -- 172.21.15.187:0/3261165065 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe400bc50 con 0x7f7fec062a70 2020-11-20T12:21:07.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.421+0000 7f7fe37fe700 1 -- 172.21.15.187:0/3261165065 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe400bdb0 con 0x7f7fec062a70 2020-11-20T12:21:07.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7fe37fe700 1 -- 172.21.15.187:0/3261165065 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4011610 con 0x7f7fec062a70 2020-11-20T12:21:07.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/3261165065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec062a70 msgr2=0x7f7fec068220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/3261165065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec062a70 0x7f7fec068220 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f7fec065ed0 tx=0x7f7fe4009290).stop 2020-11-20T12:21:07.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/3261165065 shutdown_connections 2020-11-20T12:21:07.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/3261165065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec062a70 0x7f7fec068220 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/3261165065 >> v2:172.21.15.5:3300/0 conn(0x7f7fec14c750 0x7f7fec062580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/3261165065 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec067d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/3261165065 >> 172.21.15.187:0/3261165065 conn(0x7f7fec0bacd0 msgr2=0x7f7fec0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:07.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/3261165065 shutdown_connections 2020-11-20T12:21:07.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.422+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/3261165065 wait complete. 2020-11-20T12:21:07.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.424+0000 7f7ff2cc6700 1 Processor -- start 2020-11-20T12:21:07.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.425+0000 7f7ff2cc6700 1 -- start start 2020-11-20T12:21:07.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff2cc6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fec062a70 0x7f7fec0b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff2cc6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec15caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff2cc6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec14c750 0x7f7fec09dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff2cc6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec15c3b0 con 0x7f7fec062a70 2020-11-20T12:21:07.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff2cc6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec15c650 con 0x7f7fec14c750 2020-11-20T12:21:07.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff2cc6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec09f5b0 con 0x7f7fec0666e0 2020-11-20T12:21:07.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff1263700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec14c750 0x7f7fec09dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff1263700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec14c750 0x7f7fec09dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54098/0 (socket says 172.21.15.187:54098) 2020-11-20T12:21:07.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff1263700 1 -- 172.21.15.187:0/2393280885 learned_addr learned my addr 172.21.15.187:0/2393280885 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:07.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff1263700 1 -- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 msgr2=0x7f7fec15caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7fe3fff700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec15caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.426+0000 7f7ff0a62700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fec062a70 0x7f7fec0b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7ff1263700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec15caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7ff1263700 1 -- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fec062a70 msgr2=0x7f7fec0b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7ff1263700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fec062a70 0x7f7fec0b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7ff1263700 1 -- 172.21.15.187:0/2393280885 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec0b4580 con 0x7f7fec14c750 2020-11-20T12:21:07.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7fe3fff700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec15caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:07.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7ff0a62700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fec062a70 0x7f7fec0b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:07.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7ff1263700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec14c750 0x7f7fec09dc80 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7f7fe40092d0 tx=0x7f7fe4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:07.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7fe0ff9700 1 -- 172.21.15.187:0/2393280885 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe400bc50 con 0x7f7fec14c750 2020-11-20T12:21:07.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7fe0ff9700 1 -- 172.21.15.187:0/2393280885 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe4010c40 con 0x7f7fec14c750 2020-11-20T12:21:07.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fec15cfe0 con 0x7f7fec14c750 2020-11-20T12:21:07.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7fe0ff9700 1 -- 172.21.15.187:0/2393280885 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe401c470 con 0x7f7fec14c750 2020-11-20T12:21:07.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.427+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fec09d070 con 0x7f7fec14c750 2020-11-20T12:21:07.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.429+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fec063600 con 0x7f7fec14c750 2020-11-20T12:21:07.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.429+0000 7f7fe0ff9700 1 -- 172.21.15.187:0/2393280885 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7fe4023030 con 0x7f7fec14c750 2020-11-20T12:21:07.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.430+0000 7f7fe0ff9700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc4056d20 0x7f7fc40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.430+0000 7f7fe0ff9700 1 -- 172.21.15.187:0/2393280885 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7fe40718a0 con 0x7f7fec14c750 2020-11-20T12:21:07.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.430+0000 7f7ff0a62700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc4056d20 0x7f7fc40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.431+0000 7f7ff0a62700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc4056d20 0x7f7fc40591e0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f7fdc009670 tx=0x7f7fdc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:07.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.433+0000 7f7fe0ff9700 1 -- 172.21.15.187:0/2393280885 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7fe4046d50 con 0x7f7fec14c750 2020-11-20T12:21:07.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.621+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fec14cb30 con 0x7f7fec14c750 2020-11-20T12:21:07.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.623+0000 7f7fe0ff9700 1 -- 172.21.15.187:0/2393280885 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16919 (secure 0 0 0) 0x7f7fe402f230 con 0x7f7fec14c750 2020-11-20T12:21:07.626 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:07.626 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":151,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:20:39.006799+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:20:19.501212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2020-11-20T12:20:20.096442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:6833/1287724648":"2020-11-21T12:20:39.006748+0000","172.21.15.5:6832/1287724648":"2020-11-21T12:20:39.006748+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:21:07.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc4056d20 msgr2=0x7f7fc40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc4056d20 0x7f7fc40591e0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f7fdc009670 tx=0x7f7fdc008040).stop 2020-11-20T12:21:07.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec14c750 msgr2=0x7f7fec09dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec14c750 0x7f7fec09dc80 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7f7fe40092d0 tx=0x7f7fe4016000).stop 2020-11-20T12:21:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 shutdown_connections 2020-11-20T12:21:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7fc4056d20 0x7f7fc40591e0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fec062a70 0x7f7fec0b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fec14c750 0x7f7fec09dc80 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 --2- 172.21.15.187:0/2393280885 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7fec0666e0 0x7f7fec15caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.625+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 >> 172.21.15.187:0/2393280885 conn(0x7f7fec0bacd0 msgr2=0x7f7fec0b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.626+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 shutdown_connections 2020-11-20T12:21:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.626+0000 7f7ff2cc6700 1 -- 172.21.15.187:0/2393280885 wait complete. 2020-11-20T12:21:07.650 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:20:19.501212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:20:20.096442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:21:07.651 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:21:07.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.894+0000 7f3a422a3700 1 Processor -- start 2020-11-20T12:21:07.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.895+0000 7f3a422a3700 1 -- start start 2020-11-20T12:21:07.898 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.896+0000 7f3a422a3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 0x7f3a3c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.896+0000 7f3a422a3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3c0b1300 con 0x7f3a3c1484d0 2020-11-20T12:21:07.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.896+0000 7f3a422a3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3c0b15a0 con 0x7f3a3c0b6650 2020-11-20T12:21:07.899 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.896+0000 7f3a422a3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c067730 con 0x7f3a3c066f70 2020-11-20T12:21:07.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.896+0000 7f3a40aa0700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3a3c0b6650 0x7f3a3c0afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50710/0 (socket says 172.21.15.187:50710) 2020-11-20T12:21:07.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.896+0000 7f3a40aa0700 1 -- 172.21.15.187:0/2995418984 learned_addr learned my addr 172.21.15.187:0/2995418984 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:07.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.897+0000 7f3a41aa2700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 0x7f3a3c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.897+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581388267 0 0) 0x7f3a3c0b15a0 con 0x7f3a3c0b6650 2020-11-20T12:21:07.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.897+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a24003240 con 0x7f3a3c0b6650 2020-11-20T12:21:07.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.897+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 982894696 0 0) 0x7f3a24003240 con 0x7f3a3c0b6650 2020-11-20T12:21:07.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 msgr2=0x7f3a3c0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:07.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 0x7f3a3c0b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 >> v1:172.21.15.5:6789/0 conn(0x7f3a3c1484d0 legacy=0x7f3a3c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c0b1840 con 0x7f3a3c0b6650 2020-11-20T12:21:07.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f3a2c001520 con 0x7f3a3c0b6650 2020-11-20T12:21:07.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 >> v1:172.21.15.187:6789/0 conn(0x7f3a3c0b6650 legacy=0x7f3a3c0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a24003730 0x7f3a24003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 0x7f3a240044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a240049d0 0x7f3a24004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c067730 con 0x7f3a240049d0 2020-11-20T12:21:07.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240041d0 con 0x7f3a24003730 2020-11-20T12:21:07.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2400d940 con 0x7f3a3c066f70 2020-11-20T12:21:07.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.898+0000 7f3a412a1700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a24003730 0x7f3a24003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a40aa0700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 0x7f3a240044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a41aa2700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a240049d0 0x7f3a24004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a412a1700 1 -- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 msgr2=0x7f3a240044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:07.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a412a1700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 0x7f3a240044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a412a1700 1 -- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a240049d0 msgr2=0x7f3a24004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:07.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a412a1700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a240049d0 0x7f3a24004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a412a1700 1 -- 172.21.15.187:0/2995418984 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a24003240 con 0x7f3a24003730 2020-11-20T12:21:07.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a40aa0700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c066f70 0x7f3a240044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:07.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a412a1700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a24003730 0x7f3a24003b50 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f3a38001920 tx=0x7f3a38007b90).ready entity=mon.1 client_cookie=fa27592a7204589 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:07.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.899+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38007020 con 0x7f3a24003730 2020-11-20T12:21:07.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a38007180 con 0x7f3a24003730 2020-11-20T12:21:07.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a2bfff700 1 -- 172.21.15.187:0/2995418984 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3800c5c0 con 0x7f3a24003730 2020-11-20T12:21:07.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a422a3700 1 -- 172.21.15.187:0/2995418984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a24003730 msgr2=0x7f3a24003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a422a3700 1 --2- 172.21.15.187:0/2995418984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a24003730 0x7f3a24003b50 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f3a38001920 tx=0x7f3a38007b90).stop 2020-11-20T12:21:07.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a41aa2700 1 -- 172.21.15.187:0/2995418984 reap_dead start 2020-11-20T12:21:07.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a422a3700 1 -- 172.21.15.187:0/2995418984 shutdown_connections 2020-11-20T12:21:07.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a422a3700 1 -- 172.21.15.187:0/2995418984 >> 172.21.15.187:0/2995418984 conn(0x7f3a3c0bab30 msgr2=0x7f3a3c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:07.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a422a3700 1 -- 172.21.15.187:0/2995418984 shutdown_connections 2020-11-20T12:21:07.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.900+0000 7f3a422a3700 1 -- 172.21.15.187:0/2995418984 wait complete. 2020-11-20T12:21:07.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.902+0000 7f3a422a3700 1 Processor -- start 2020-11-20T12:21:07.913 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.902+0000 7f3a422a3700 1 -- start start 2020-11-20T12:21:07.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a422a3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c0b9e40 0x7f3a3c158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a422a3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0ba220 0x7f3a3c09b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a422a3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a3c09cba0 0x7f3a3c09bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a422a3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0b37f0 con 0x7f3a3c09cba0 2020-11-20T12:21:07.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a422a3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c158820 con 0x7f3a3c0ba220 2020-11-20T12:21:07.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a422a3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c158ac0 con 0x7f3a3c0b9e40 2020-11-20T12:21:07.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a40aa0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0ba220 0x7f3a3c09b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a40aa0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0ba220 0x7f3a3c09b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54116/0 (socket says 172.21.15.187:54116) 2020-11-20T12:21:07.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.903+0000 7f3a40aa0700 1 -- 172.21.15.187:0/576118440 learned_addr learned my addr 172.21.15.187:0/576118440 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:07.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a41aa2700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a3c09cba0 0x7f3a3c09bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a412a1700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c0b9e40 0x7f3a3c158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a40aa0700 1 -- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c0b9e40 msgr2=0x7f3a3c158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a40aa0700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c0b9e40 0x7f3a3c158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a40aa0700 1 -- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a3c09cba0 msgr2=0x7f3a3c09bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a40aa0700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a3c09cba0 0x7f3a3c09bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a40aa0700 1 -- 172.21.15.187:0/576118440 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c158d60 con 0x7f3a3c0ba220 2020-11-20T12:21:07.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a412a1700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c0b9e40 0x7f3a3c158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:07.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a41aa2700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a3c09cba0 0x7f3a3c09bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:07.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a40aa0700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0ba220 0x7f3a3c09b8b0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f3a2c00dad0 tx=0x7f3a2c009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:07.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a297fa700 1 -- 172.21.15.187:0/576118440 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c007170 con 0x7f3a3c0ba220 2020-11-20T12:21:07.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3c158560 con 0x7f3a3c0ba220 2020-11-20T12:21:07.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.904+0000 7f3a297fa700 1 -- 172.21.15.187:0/576118440 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c0072d0 con 0x7f3a3c0ba220 2020-11-20T12:21:07.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.905+0000 7f3a297fa700 1 -- 172.21.15.187:0/576118440 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c00bb40 con 0x7f3a3c0ba220 2020-11-20T12:21:07.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.905+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3c09ecf0 con 0x7f3a3c0ba220 2020-11-20T12:21:07.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.906+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3c061a20 con 0x7f3a3c0ba220 2020-11-20T12:21:07.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.906+0000 7f3a297fa700 1 -- 172.21.15.187:0/576118440 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3a2c0045c0 con 0x7f3a3c0ba220 2020-11-20T12:21:07.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.907+0000 7f3a297fa700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3a14056d20 0x7f3a140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:07.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.907+0000 7f3a297fa700 1 -- 172.21.15.187:0/576118440 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3a2c0704c0 con 0x7f3a3c0ba220 2020-11-20T12:21:07.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.907+0000 7f3a412a1700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3a14056d20 0x7f3a140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:07.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.908+0000 7f3a412a1700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3a14056d20 0x7f3a140591e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f3a3c05e630 tx=0x7f3a38015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:07.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:07.911+0000 7f3a297fa700 1 -- 172.21.15.187:0/576118440 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3a2c045750 con 0x7f3a3c0ba220 2020-11-20T12:21:08.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.100+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3a3c0624e0 con 0x7f3a3c0ba220 2020-11-20T12:21:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.102+0000 7f3a297fa700 1 -- 172.21.15.187:0/576118440 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v151) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3a2c048d70 con 0x7f3a3c0ba220 2020-11-20T12:21:08.105 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:21:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3a14056d20 msgr2=0x7f3a140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3a14056d20 0x7f3a140591e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f3a3c05e630 tx=0x7f3a38015000).stop 2020-11-20T12:21:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0ba220 msgr2=0x7f3a3c09b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0ba220 0x7f3a3c09b8b0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f3a2c00dad0 tx=0x7f3a2c009040).stop 2020-11-20T12:21:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 shutdown_connections 2020-11-20T12:21:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3a14056d20 0x7f3a140591e0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3a3c09cba0 0x7f3a3c09bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3a3c0b9e40 0x7f3a3c158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 --2- 172.21.15.187:0/576118440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0ba220 0x7f3a3c09b8b0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.104+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 >> 172.21.15.187:0/576118440 conn(0x7f3a3c0bab30 msgr2=0x7f3a3c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:08.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.105+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 shutdown_connections 2020-11-20T12:21:08.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.105+0000 7f3a422a3700 1 -- 172.21.15.187:0/576118440 wait complete. 2020-11-20T12:21:08.131 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:21:08.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.398+0000 7f1703fff700 1 Processor -- start 2020-11-20T12:21:08.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.398+0000 7f1703fff700 1 -- start start 2020-11-20T12:21:08.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.399+0000 7f1703fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1380 0x7f1704150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.400+0000 7f1703fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f17040b1760 0x7f1704065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.400+0000 7f1703fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1704150390 con 0x7f1704065680 2020-11-20T12:21:08.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.400+0000 7f1703fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1704150630 con 0x7f17040b1760 2020-11-20T12:21:08.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.400+0000 7f1703fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1704068510 con 0x7f17040b1380 2020-11-20T12:21:08.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.400+0000 7f17027fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f17040b1760 0x7f1704065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.400+0000 7f17027fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f17040b1760 0x7f1704065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54122/0 (socket says 172.21.15.187:54122) 2020-11-20T12:21:08.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.400+0000 7f17027fc700 1 -- 172.21.15.187:0/1561197192 learned_addr learned my addr 172.21.15.187:0/1561197192 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:08.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.400+0000 7f1702ffd700 1 --2- 172.21.15.187:0/1561197192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1380 0x7f1704150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.401+0000 7f17027fc700 1 -- 172.21.15.187:0/1561197192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1380 msgr2=0x7f1704150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:08.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.401+0000 7f17027fc700 1 --2- 172.21.15.187:0/1561197192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1380 0x7f1704150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.401+0000 7f17027fc700 1 -- 172.21.15.187:0/1561197192 >> v1:172.21.15.5:6789/0 conn(0x7f1704065680 legacy=0x7f1704061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:08.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.401+0000 7f17027fc700 1 -- 172.21.15.187:0/1561197192 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704150fc0 con 0x7f17040b1760 2020-11-20T12:21:08.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.401+0000 7f17027fc700 1 --2- 172.21.15.187:0/1561197192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1760 0x7f1704065190 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7f16f8006fd0 tx=0x7f16f8006eb0).ready entity=mon.1 client_cookie=7082080a8fb56e2f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:08.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.401+0000 7f1701ffb700 1 -- 172.21.15.187:0/1561197192 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8009890 con 0x7f17040b1760 2020-11-20T12:21:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.401+0000 7f1701ffb700 1 -- 172.21.15.187:0/1561197192 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f80099f0 con 0x7f17040b1760 2020-11-20T12:21:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.401+0000 7f1701ffb700 1 -- 172.21.15.187:0/1561197192 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8009d70 con 0x7f17040b1760 2020-11-20T12:21:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.402+0000 7f1703fff700 1 -- 172.21.15.187:0/1561197192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1760 msgr2=0x7f1704065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.402+0000 7f1703fff700 1 --2- 172.21.15.187:0/1561197192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1760 0x7f1704065190 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7f16f8006fd0 tx=0x7f16f8006eb0).stop 2020-11-20T12:21:08.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.402+0000 7f1703fff700 1 -- 172.21.15.187:0/1561197192 shutdown_connections 2020-11-20T12:21:08.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.402+0000 7f1703fff700 1 --2- 172.21.15.187:0/1561197192 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1380 0x7f1704150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.402+0000 7f1703fff700 1 --2- 172.21.15.187:0/1561197192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1760 0x7f1704065190 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.402+0000 7f1703fff700 1 -- 172.21.15.187:0/1561197192 >> 172.21.15.187:0/1561197192 conn(0x7f17040bab30 msgr2=0x7f17040baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:08.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.402+0000 7f1703fff700 1 -- 172.21.15.187:0/1561197192 shutdown_connections 2020-11-20T12:21:08.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.402+0000 7f1703fff700 1 -- 172.21.15.187:0/1561197192 wait complete. 2020-11-20T12:21:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.404+0000 7f1703fff700 1 Processor -- start 2020-11-20T12:21:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.404+0000 7f1703fff700 1 -- start start 2020-11-20T12:21:08.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1703fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1380 0x7f170409e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1703fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1760 0x7f1704149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1703fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f170414b4d0 0x7f170414b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1703fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17040b7fb0 con 0x7f170414b4d0 2020-11-20T12:21:08.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1703fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17040b8250 con 0x7f17040b1380 2020-11-20T12:21:08.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1703fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170414a140 con 0x7f17040b1760 2020-11-20T12:21:08.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1702ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1380 0x7f170409e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1702ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1380 0x7f170409e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54128/0 (socket says 172.21.15.187:54128) 2020-11-20T12:21:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f1702ffd700 1 -- 172.21.15.187:0/3683123291 learned_addr learned my addr 172.21.15.187:0/3683123291 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.406+0000 7f17037fe700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f170414b4d0 0x7f170414b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f17027fc700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1760 0x7f1704149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f1702ffd700 1 -- 172.21.15.187:0/3683123291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1760 msgr2=0x7f1704149c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f1702ffd700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1760 0x7f1704149c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f1702ffd700 1 -- 172.21.15.187:0/3683123291 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f170414b4d0 msgr2=0x7f170414b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f1702ffd700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f170414b4d0 0x7f170414b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f1702ffd700 1 -- 172.21.15.187:0/3683123291 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040b7c30 con 0x7f17040b1380 2020-11-20T12:21:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f1702ffd700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1380 0x7f170409e7d0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7f16f400db60 tx=0x7f16f4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f16eaffd700 1 -- 172.21.15.187:0/3683123291 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400b170 con 0x7f17040b1380 2020-11-20T12:21:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f16eaffd700 1 -- 172.21.15.187:0/3683123291 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f4009790 con 0x7f17040b1380 2020-11-20T12:21:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f16eaffd700 1 -- 172.21.15.187:0/3683123291 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4009b50 con 0x7f17040b1380 2020-11-20T12:21:08.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.407+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17040a0900 con 0x7f17040b1380 2020-11-20T12:21:08.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.408+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17040b1d80 con 0x7f17040b1380 2020-11-20T12:21:08.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.409+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17040a0af0 con 0x7f17040b1380 2020-11-20T12:21:08.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.409+0000 7f16eaffd700 1 -- 172.21.15.187:0/3683123291 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f16f4013480 con 0x7f17040b1380 2020-11-20T12:21:08.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.410+0000 7f16eaffd700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.410+0000 7f16eaffd700 1 -- 172.21.15.187:0/3683123291 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f16f4070600 con 0x7f17040b1380 2020-11-20T12:21:08.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.410+0000 7f17027fc700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.411+0000 7f17027fc700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f1704150170 tx=0x7f16f8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:08.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.413+0000 7f16eaffd700 1 -- 172.21.15.187:0/3683123291 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f16f4045890 con 0x7f17040b1380 2020-11-20T12:21:08.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.602+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1704061a20 con 0x7f17040b1380 2020-11-20T12:21:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.603+0000 7f16eaffd700 1 -- 172.21.15.187:0/3683123291 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v151) v1 ==== 92+0+10 (secure 0 0 0) 0x7f16f4048eb0 con 0x7f17040b1380 2020-11-20T12:21:08.606 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.605+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 msgr2=0x7f16e00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.605+0000 7f1703fff700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f1704150170 tx=0x7f16f8015000).stop 2020-11-20T12:21:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1380 msgr2=0x7f170409e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1380 0x7f170409e7d0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7f16f400db60 tx=0x7f16f4008040).stop 2020-11-20T12:21:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 shutdown_connections 2020-11-20T12:21:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f16e0056d20 0x7f16e00591e0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f170414b4d0 0x7f170414b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17040b1380 0x7f170409e7d0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 --2- 172.21.15.187:0/3683123291 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17040b1760 0x7f1704149c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 >> 172.21.15.187:0/3683123291 conn(0x7f17040bab30 msgr2=0x7f17040afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:08.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 shutdown_connections 2020-11-20T12:21:08.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.606+0000 7f1703fff700 1 -- 172.21.15.187:0/3683123291 wait complete. 2020-11-20T12:21:08.630 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:21:08.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.863+0000 7fb1c0b4a700 1 Processor -- start 2020-11-20T12:21:08.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.863+0000 7fb1c0b4a700 1 -- start start 2020-11-20T12:21:08.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.864+0000 7fb1c0b4a700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb1bc066f70 0x7fb1bc0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.864+0000 7fb1c0b4a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1bc0b1300 con 0x7fb1bc0b6650 2020-11-20T12:21:08.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.864+0000 7fb1c0b4a700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1bc0b15a0 con 0x7fb1bc1484d0 2020-11-20T12:21:08.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.864+0000 7fb1c0b4a700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1bc067730 con 0x7fb1bc066f70 2020-11-20T12:21:08.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.865+0000 7fb1bb7fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb1bc1484d0 0x7fb1bc061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50742/0 (socket says 172.21.15.187:50742) 2020-11-20T12:21:08.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.865+0000 7fb1bb7fe700 1 -- 172.21.15.187:0/1292991980 learned_addr learned my addr 172.21.15.187:0/1292991980 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:08.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.865+0000 7fb1bbfff700 1 --2- 172.21.15.187:0/1292991980 >> v2:172.21.15.187:3300/0 conn(0x7fb1bc066f70 0x7fb1bc0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.865+0000 7fb1baffd700 1 -- 172.21.15.187:0/1292991980 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44769491 0 0) 0x7fb1bc0b15a0 con 0x7fb1bc1484d0 2020-11-20T12:21:08.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.865+0000 7fb1baffd700 1 -- 172.21.15.187:0/1292991980 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19c003240 con 0x7fb1bc1484d0 2020-11-20T12:21:08.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.865+0000 7fb1bbfff700 1 -- 172.21.15.187:0/1292991980 >> v1:172.21.15.187:6789/0 conn(0x7fb1bc1484d0 legacy=0x7fb1bc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.865+0000 7fb1baffd700 1 -- 172.21.15.187:0/1292991980 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3051708181 0 0) 0x7fb19c003240 con 0x7fb1bc1484d0 2020-11-20T12:21:08.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.866+0000 7fb1bbfff700 1 -- 172.21.15.187:0/1292991980 >> v1:172.21.15.5:6789/0 conn(0x7fb1bc0b6650 legacy=0x7fb1bc0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.866+0000 7fb1bbfff700 1 -- 172.21.15.187:0/1292991980 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc0b1840 con 0x7fb1bc066f70 2020-11-20T12:21:08.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.866+0000 7fb1bbfff700 1 --2- 172.21.15.187:0/1292991980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc066f70 0x7fb1bc0b01a0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7fb1b400b920 tx=0x7fb1b4006df0).ready entity=mon.1 client_cookie=5fdbe2263a4a7b8e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:08.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.866+0000 7fb1baffd700 1 -- 172.21.15.187:0/1292991980 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac001860 con 0x7fb1bc066f70 2020-11-20T12:21:08.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.866+0000 7fb1baffd700 1 -- 172.21.15.187:0/1292991980 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b4011400 con 0x7fb1bc066f70 2020-11-20T12:21:08.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.866+0000 7fb1baffd700 1 -- 172.21.15.187:0/1292991980 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4011780 con 0x7fb1bc066f70 2020-11-20T12:21:08.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.867+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/1292991980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc066f70 msgr2=0x7fb1bc0b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.867+0000 7fb1c0b4a700 1 --2- 172.21.15.187:0/1292991980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc066f70 0x7fb1bc0b01a0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7fb1b400b920 tx=0x7fb1b4006df0).stop 2020-11-20T12:21:08.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.867+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/1292991980 shutdown_connections 2020-11-20T12:21:08.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.867+0000 7fb1c0b4a700 1 --2- 172.21.15.187:0/1292991980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc066f70 0x7fb1bc0b01a0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.867+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/1292991980 >> 172.21.15.187:0/1292991980 conn(0x7fb1bc0bab30 msgr2=0x7fb1bc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:08.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.867+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/1292991980 shutdown_connections 2020-11-20T12:21:08.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.867+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/1292991980 wait complete. 2020-11-20T12:21:08.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.868+0000 7fb1c0b4a700 1 Processor -- start 2020-11-20T12:21:08.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.869+0000 7fb1c0b4a700 1 -- start start 2020-11-20T12:21:08.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.869+0000 7fb1c0b4a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1bc066f70 0x7fb1bc139b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.869+0000 7fb1c0b4a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc0b6650 0x7fb1bc09bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.869+0000 7fb1c0b4a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1bc1484d0 0x7fb1bc1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.869+0000 7fb1c0b4a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc09b200 con 0x7fb1bc066f70 2020-11-20T12:21:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.869+0000 7fb1c0b4a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc09b4a0 con 0x7fb1bc0b6650 2020-11-20T12:21:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.869+0000 7fb1c0b4a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc09b740 con 0x7fb1bc1484d0 2020-11-20T12:21:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.870+0000 7fb1b2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc0b6650 0x7fb1bc09bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.870+0000 7fb1b2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc0b6650 0x7fb1bc09bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54146/0 (socket says 172.21.15.187:54146) 2020-11-20T12:21:08.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.870+0000 7fb1b2ffd700 1 -- 172.21.15.187:0/284076259 learned_addr learned my addr 172.21.15.187:0/284076259 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:08.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.870+0000 7fb1bb7fe700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1bc066f70 0x7fb1bc139b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.870+0000 7fb1bbfff700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1bc1484d0 0x7fb1bc1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b2ffd700 1 -- 172.21.15.187:0/284076259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1bc1484d0 msgr2=0x7fb1bc1499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b2ffd700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1bc1484d0 0x7fb1bc1499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b2ffd700 1 -- 172.21.15.187:0/284076259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1bc066f70 msgr2=0x7fb1bc139b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:08.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b2ffd700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1bc066f70 0x7fb1bc139b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:08.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b2ffd700 1 -- 172.21.15.187:0/284076259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc09d520 con 0x7fb1bc0b6650 2020-11-20T12:21:08.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b2ffd700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc0b6650 0x7fb1bc09bbe0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7fb1a4001e70 tx=0x7fb1a4007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:08.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b3fff700 1 -- 172.21.15.187:0/284076259 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4007020 con 0x7fb1bc0b6650 2020-11-20T12:21:08.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b3fff700 1 -- 172.21.15.187:0/284076259 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a4007180 con 0x7fb1bc0b6650 2020-11-20T12:21:08.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1b3fff700 1 -- 172.21.15.187:0/284076259 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a400e560 con 0x7fb1bc0b6650 2020-11-20T12:21:08.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1bc13a050 con 0x7fb1bc0b6650 2020-11-20T12:21:08.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.871+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1bc09c7b0 con 0x7fb1bc0b6650 2020-11-20T12:21:08.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.872+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1bc09b900 con 0x7fb1bc0b6650 2020-11-20T12:21:08.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.873+0000 7fb1b3fff700 1 -- 172.21.15.187:0/284076259 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb1a4010070 con 0x7fb1bc0b6650 2020-11-20T12:21:08.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.874+0000 7fb1b3fff700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb198056d20 0x7fb1980591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:08.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.874+0000 7fb1b3fff700 1 -- 172.21.15.187:0/284076259 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb1a406f560 con 0x7fb1bc0b6650 2020-11-20T12:21:08.889 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.874+0000 7fb1bb7fe700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb198056d20 0x7fb1980591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:08.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.875+0000 7fb1bb7fe700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb198056d20 0x7fb1980591e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7fb1ac002210 tx=0x7fb1ac001820).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:08.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:08.877+0000 7fb1b3fff700 1 -- 172.21.15.187:0/284076259 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb1a4044850 con 0x7fb1bc0b6650 2020-11-20T12:21:09.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.064+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1bc061a20 con 0x7fb1bc0b6650 2020-11-20T12:21:09.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.065+0000 7fb1b3fff700 1 -- 172.21.15.187:0/284076259 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v151) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb1a4047e70 con 0x7fb1bc0b6650 2020-11-20T12:21:09.067 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:09.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb198056d20 msgr2=0x7fb1980591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb198056d20 0x7fb1980591e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7fb1ac002210 tx=0x7fb1ac001820).stop 2020-11-20T12:21:09.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc0b6650 msgr2=0x7fb1bc09bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc0b6650 0x7fb1bc09bbe0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7fb1a4001e70 tx=0x7fb1a4007c50).stop 2020-11-20T12:21:09.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 shutdown_connections 2020-11-20T12:21:09.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb198056d20 0x7fb1980591e0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.071 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1bc066f70 0x7fb1bc139b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb1bc1484d0 0x7fb1bc1499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 --2- 172.21.15.187:0/284076259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1bc0b6650 0x7fb1bc09bbe0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.067+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 >> 172.21.15.187:0/284076259 conn(0x7fb1bc0bab30 msgr2=0x7fb1bc066570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:09.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.068+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 shutdown_connections 2020-11-20T12:21:09.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.068+0000 7fb1c0b4a700 1 -- 172.21.15.187:0/284076259 wait complete. 2020-11-20T12:21:09.092 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:21:09.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.366+0000 7f4841ab3700 1 Processor -- start 2020-11-20T12:21:09.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.366+0000 7f4841ab3700 1 -- start start 2020-11-20T12:21:09.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.367+0000 7f4841ab3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c14d270 0x7f483c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.367+0000 7f4841ab3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0658a0 con 0x7f483c0b1380 2020-11-20T12:21:09.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.367+0000 7f4841ab3700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c150390 con 0x7f483c0b1760 2020-11-20T12:21:09.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.367+0000 7f4841ab3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1511e0 con 0x7f483c14d270 2020-11-20T12:21:09.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.368+0000 7f483bfff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f483c0b1760 0x7f483c065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50756/0 (socket says 172.21.15.187:50756) 2020-11-20T12:21:09.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.368+0000 7f483bfff700 1 -- 172.21.15.187:0/2971800955 learned_addr learned my addr 172.21.15.187:0/2971800955 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:09.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.368+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942735303 0 0) 0x7f483c150390 con 0x7f483c0b1760 2020-11-20T12:21:09.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.368+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4820003240 con 0x7f483c0b1760 2020-11-20T12:21:09.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f48412b2700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c14d270 0x7f483c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 195168424 0 0) 0x7f4820003240 con 0x7f483c0b1760 2020-11-20T12:21:09.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c14d270 msgr2=0x7f483c1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:09.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c14d270 0x7f483c1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 >> v1:172.21.15.5:6789/0 conn(0x7f483c0b1380 legacy=0x7f483c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:09.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c150630 con 0x7f483c0b1760 2020-11-20T12:21:09.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f48280014e0 con 0x7f483c0b1760 2020-11-20T12:21:09.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 >> v1:172.21.15.187:6789/0 conn(0x7f483c0b1760 legacy=0x7f483c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c14d270 0x7f48200037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4820003cf0 0x7f48200044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200049d0 0x7f4820004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1511e0 con 0x7f4820003cf0 2020-11-20T12:21:09.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4820004280 con 0x7f48200049d0 2020-11-20T12:21:09.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482000b730 con 0x7f483c14d270 2020-11-20T12:21:09.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.369+0000 7f48412b2700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200049d0 0x7f4820004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f483bfff700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4820003cf0 0x7f48200044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f48412b2700 1 -- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c14d270 msgr2=0x7f48200037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:09.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f48412b2700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c14d270 0x7f48200037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:09.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f48412b2700 1 -- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4820003cf0 msgr2=0x7f48200044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:09.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f48412b2700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4820003cf0 0x7f48200044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f48412b2700 1 -- 172.21.15.187:0/2971800955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820003240 con 0x7f48200049d0 2020-11-20T12:21:09.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f483bfff700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4820003cf0 0x7f48200044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:09.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f48412b2700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200049d0 0x7f4820004e50 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f4834009d80 tx=0x7f4834008040).ready entity=mon.1 client_cookie=18bc009a5aa7972f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483400c070 con 0x7f48200049d0 2020-11-20T12:21:09.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483400ea80 con 0x7f48200049d0 2020-11-20T12:21:09.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.370+0000 7f483b7fe700 1 -- 172.21.15.187:0/2971800955 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483400ee40 con 0x7f48200049d0 2020-11-20T12:21:09.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.371+0000 7f4841ab3700 1 -- 172.21.15.187:0/2971800955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200049d0 msgr2=0x7f4820004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.371+0000 7f4841ab3700 1 --2- 172.21.15.187:0/2971800955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200049d0 0x7f4820004e50 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f4834009d80 tx=0x7f4834008040).stop 2020-11-20T12:21:09.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.371+0000 7f48412b2700 1 -- 172.21.15.187:0/2971800955 reap_dead start 2020-11-20T12:21:09.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.371+0000 7f4841ab3700 1 -- 172.21.15.187:0/2971800955 shutdown_connections 2020-11-20T12:21:09.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.371+0000 7f4841ab3700 1 -- 172.21.15.187:0/2971800955 >> 172.21.15.187:0/2971800955 conn(0x7f483c0bab30 msgr2=0x7f483c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:09.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.371+0000 7f4841ab3700 1 -- 172.21.15.187:0/2971800955 shutdown_connections 2020-11-20T12:21:09.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.371+0000 7f4841ab3700 1 -- 172.21.15.187:0/2971800955 wait complete. 2020-11-20T12:21:09.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.373+0000 7f4841ab3700 1 Processor -- start 2020-11-20T12:21:09.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.373+0000 7f4841ab3700 1 -- start start 2020-11-20T12:21:09.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.374+0000 7f4841ab3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c15bb20 0x7f483c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.374+0000 7f4841ab3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f483c15bf00 0x7f483c09fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.374+0000 7f4841ab3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f483c0a0000 0x7f483c15f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.374+0000 7f4841ab3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0a0480 con 0x7f483c0a0000 2020-11-20T12:21:09.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.374+0000 7f4841ab3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c09f370 con 0x7f483c15bf00 2020-11-20T12:21:09.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.374+0000 7f4841ab3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0b46a0 con 0x7f483c15bb20 2020-11-20T12:21:09.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f483bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f483c15bf00 0x7f483c09fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f483bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f483c15bf00 0x7f483c09fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54162/0 (socket says 172.21.15.187:54162) 2020-11-20T12:21:09.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f483bfff700 1 -- 172.21.15.187:0/1221167671 learned_addr learned my addr 172.21.15.187:0/1221167671 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:09.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f483bfff700 1 -- 172.21.15.187:0/1221167671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c15bb20 msgr2=0x7f483c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f483bfff700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c15bb20 0x7f483c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:09.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f48412b2700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f483c0a0000 0x7f483c15f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f483bfff700 1 -- 172.21.15.187:0/1221167671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f483c0a0000 msgr2=0x7f483c15f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f483bfff700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f483c0a0000 0x7f483c15f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f483bfff700 1 -- 172.21.15.187:0/1221167671 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c09f5d0 con 0x7f483c15bf00 2020-11-20T12:21:09.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.375+0000 7f48412b2700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f483c0a0000 0x7f483c15f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:09.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.376+0000 7f483bfff700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f483c15bf00 0x7f483c09fb10 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f482800dad0 tx=0x7f4828009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.376+0000 7f4838ff9700 1 -- 172.21.15.187:0/1221167671 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4828007170 con 0x7f483c15bf00 2020-11-20T12:21:09.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.376+0000 7f4838ff9700 1 -- 172.21.15.187:0/1221167671 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48280072d0 con 0x7f483c15bf00 2020-11-20T12:21:09.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.376+0000 7f4838ff9700 1 -- 172.21.15.187:0/1221167671 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482800ba10 con 0x7f483c15bf00 2020-11-20T12:21:09.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.376+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c09f870 con 0x7f483c15bf00 2020-11-20T12:21:09.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.377+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c15fce0 con 0x7f483c15bf00 2020-11-20T12:21:09.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.378+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c061a20 con 0x7f483c15bf00 2020-11-20T12:21:09.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.378+0000 7f4838ff9700 1 -- 172.21.15.187:0/1221167671 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f48280045c0 con 0x7f483c15bf00 2020-11-20T12:21:09.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.379+0000 7f4838ff9700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4814056d20 0x7f48140591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.379+0000 7f4838ff9700 1 -- 172.21.15.187:0/1221167671 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4828070260 con 0x7f483c15bf00 2020-11-20T12:21:09.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.379+0000 7f4840ab1700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4814056d20 0x7f48140591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.380+0000 7f4840ab1700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4814056d20 0x7f48140591e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f483c0b09e0 tx=0x7f48300077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.383+0000 7f4838ff9700 1 -- 172.21.15.187:0/1221167671 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4828045660 con 0x7f483c15bf00 2020-11-20T12:21:09.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.573+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f483c066200 con 0x7f483c15bf00 2020-11-20T12:21:09.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.573+0000 7f4838ff9700 1 -- 172.21.15.187:0/1221167671 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v151) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4828001b20 con 0x7f483c15bf00 2020-11-20T12:21:09.576 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:09.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4814056d20 msgr2=0x7f48140591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4814056d20 0x7f48140591e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f483c0b09e0 tx=0x7f48300077f0).stop 2020-11-20T12:21:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f483c15bf00 msgr2=0x7f483c09fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f483c15bf00 0x7f483c09fb10 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f482800dad0 tx=0x7f4828009040).stop 2020-11-20T12:21:09.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 shutdown_connections 2020-11-20T12:21:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4814056d20 0x7f48140591e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f483c0a0000 0x7f483c15f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f483c15bb20 0x7f483c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:09.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 --2- 172.21.15.187:0/1221167671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f483c15bf00 0x7f483c09fb10 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 >> 172.21.15.187:0/1221167671 conn(0x7f483c0bab30 msgr2=0x7f483c0afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 shutdown_connections 2020-11-20T12:21:09.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.576+0000 7f4841ab3700 1 -- 172.21.15.187:0/1221167671 wait complete. 2020-11-20T12:21:09.603 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:09.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.848+0000 7f6579b50700 1 Processor -- start 2020-11-20T12:21:09.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.848+0000 7f6579b50700 1 -- start start 2020-11-20T12:21:09.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.849+0000 7f6579b50700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6574065f90 0x7f6574065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.849+0000 7f6579b50700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f657409ca90 0x7f657409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.849+0000 7f6579b50700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65740b13e0 con 0x7f6574065190 2020-11-20T12:21:09.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.849+0000 7f6579b50700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657414ab30 con 0x7f6574065f90 2020-11-20T12:21:09.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.849+0000 7f6579b50700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6574062c10 con 0x7f657409ca90 2020-11-20T12:21:09.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.850+0000 7f657934f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f657409ca90 0x7f657409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.850+0000 7f657934f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f657409ca90 0x7f657409cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54174/0 (socket says 172.21.15.187:54174) 2020-11-20T12:21:09.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.850+0000 7f657934f700 1 -- 172.21.15.187:0/2677547484 learned_addr learned my addr 172.21.15.187:0/2677547484 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:09.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.850+0000 7f656bfff700 1 --2- 172.21.15.187:0/2677547484 >> v2:172.21.15.5:3300/0 conn(0x7f6574065f90 0x7f6574065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.850+0000 7f657934f700 1 -- 172.21.15.187:0/2677547484 >> v2:172.21.15.5:3300/0 conn(0x7f6574065f90 msgr2=0x7f6574065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:09.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.850+0000 7f657934f700 1 --2- 172.21.15.187:0/2677547484 >> v2:172.21.15.5:3300/0 conn(0x7f6574065f90 0x7f6574065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.851+0000 7f657934f700 1 -- 172.21.15.187:0/2677547484 >> v1:172.21.15.5:6789/0 conn(0x7f6574065190 legacy=0x7f6574143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:09.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.851+0000 7f657934f700 1 -- 172.21.15.187:0/2677547484 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65740b1640 con 0x7f657409ca90 2020-11-20T12:21:09.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.851+0000 7f657934f700 1 --2- 172.21.15.187:0/2677547484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657409ca90 0x7f657409cf20 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f6574099f90 tx=0x7f656c006eb0).ready entity=mon.1 client_cookie=d55482aae94eb1be server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.851+0000 7f656b7fe700 1 -- 172.21.15.187:0/2677547484 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c009920 con 0x7f657409ca90 2020-11-20T12:21:09.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.851+0000 7f656b7fe700 1 -- 172.21.15.187:0/2677547484 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c009a80 con 0x7f657409ca90 2020-11-20T12:21:09.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.851+0000 7f656b7fe700 1 -- 172.21.15.187:0/2677547484 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c009e00 con 0x7f657409ca90 2020-11-20T12:21:09.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.852+0000 7f6579b50700 1 -- 172.21.15.187:0/2677547484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657409ca90 msgr2=0x7f657409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.852+0000 7f6579b50700 1 --2- 172.21.15.187:0/2677547484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657409ca90 0x7f657409cf20 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f6574099f90 tx=0x7f656c006eb0).stop 2020-11-20T12:21:09.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.852+0000 7f6579b50700 1 -- 172.21.15.187:0/2677547484 shutdown_connections 2020-11-20T12:21:09.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.852+0000 7f6579b50700 1 --2- 172.21.15.187:0/2677547484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f657409ca90 0x7f657409cf20 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.852+0000 7f6579b50700 1 --2- 172.21.15.187:0/2677547484 >> v2:172.21.15.5:3300/0 conn(0x7f6574065f90 0x7f6574065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.852+0000 7f6579b50700 1 -- 172.21.15.187:0/2677547484 >> 172.21.15.187:0/2677547484 conn(0x7f65740bab30 msgr2=0x7f65740baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:09.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.852+0000 7f6579b50700 1 -- 172.21.15.187:0/2677547484 shutdown_connections 2020-11-20T12:21:09.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.852+0000 7f6579b50700 1 -- 172.21.15.187:0/2677547484 wait complete. 2020-11-20T12:21:09.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.854+0000 7f6579b50700 1 Processor -- start 2020-11-20T12:21:09.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.854+0000 7f6579b50700 1 -- start start 2020-11-20T12:21:09.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f6579b50700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6574065190 0x7f6574066590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f6579b50700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6574065f90 0x7f65740aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f6579b50700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657409ca90 0x7f65740b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f6579b50700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65740a0a60 con 0x7f657409ca90 2020-11-20T12:21:09.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f6579b50700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65740b4390 con 0x7f6574065f90 2020-11-20T12:21:09.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f6579b50700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65740b4630 con 0x7f6574065190 2020-11-20T12:21:09.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f656bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6574065f90 0x7f65740aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f656bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6574065f90 0x7f65740aff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54178/0 (socket says 172.21.15.187:54178) 2020-11-20T12:21:09.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.855+0000 7f656bfff700 1 -- 172.21.15.187:0/61551256 learned_addr learned my addr 172.21.15.187:0/61551256 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:09.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f657934f700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657409ca90 0x7f65740b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f6578b4e700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6574065190 0x7f6574066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f656bfff700 1 -- 172.21.15.187:0/61551256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6574065190 msgr2=0x7f6574066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f656bfff700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6574065190 0x7f6574066590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f656bfff700 1 -- 172.21.15.187:0/61551256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657409ca90 msgr2=0x7f65740b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f656bfff700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657409ca90 0x7f65740b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f656bfff700 1 -- 172.21.15.187:0/61551256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65740a05c0 con 0x7f6574065f90 2020-11-20T12:21:09.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f656bfff700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6574065f90 0x7f65740aff00 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f656400db90 tx=0x7f6564008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.856+0000 7f6568ff9700 1 -- 172.21.15.187:0/61551256 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564009790 con 0x7f6574065f90 2020-11-20T12:21:09.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.857+0000 7f6568ff9700 1 -- 172.21.15.187:0/61551256 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65640098f0 con 0x7f6574065f90 2020-11-20T12:21:09.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.857+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6574067150 con 0x7f6574065f90 2020-11-20T12:21:09.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.857+0000 7f6568ff9700 1 -- 172.21.15.187:0/61551256 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564009cb0 con 0x7f6574065f90 2020-11-20T12:21:09.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.857+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65740b03f0 con 0x7f6574065f90 2020-11-20T12:21:09.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.858+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65740b06d0 con 0x7f6574065f90 2020-11-20T12:21:09.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.858+0000 7f6568ff9700 1 -- 172.21.15.187:0/61551256 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6564012630 con 0x7f6574065f90 2020-11-20T12:21:09.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.859+0000 7f6568ff9700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f654c056d50 0x7f654c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.859+0000 7f6568ff9700 1 -- 172.21.15.187:0/61551256 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6564070270 con 0x7f6574065f90 2020-11-20T12:21:09.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.859+0000 7f6578b4e700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f654c056d50 0x7f654c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.860+0000 7f6578b4e700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f654c056d50 0x7f654c059210 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7f6570003bf0 tx=0x7f6570009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:09.863+0000 7f6568ff9700 1 -- 172.21.15.187:0/61551256 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6564045670 con 0x7f6574065f90 2020-11-20T12:21:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.107+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65740a0750 con 0x7f6574065f90 2020-11-20T12:21:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.108+0000 7f6568ff9700 1 -- 172.21.15.187:0/61551256 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+2158 (secure 0 0 0) 0x7f6564048c90 con 0x7f6574065f90 2020-11-20T12:21:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:10.111 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":164,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:20:21.071872+0000","modified":"2020-11-20T12:21:05.934493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9661,"mds_1":6281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6281":{"gid":6281,"name":"b","rank":1,"incarnation":158,"state":"up:active","state_seq":6,"addr":"172.21.15.5:6833/191681417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":191681417},{"type":"v1","addr":"172.21.15.5:6833","nonce":191681417}]},"join_fscid":-1,"export_targets":[0],"features":4540138297136906239,"flags":0},"gid_9661":{"gid":9661,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":4,"addr":"172.21.15.187:6835/3618925546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3618925546},{"type":"v1","addr":"172.21.15.187:6835","nonce":3618925546}]},"join_fscid":-1,"export_targets":[1],"features":4540138297136906239,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2020-11-20T12:21:10.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f654c056d50 msgr2=0x7f654c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:10.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f654c056d50 0x7f654c059210 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7f6570003bf0 tx=0x7f6570009040).stop 2020-11-20T12:21:10.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6574065f90 msgr2=0x7f65740aff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:10.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6574065f90 0x7f65740aff00 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f656400db90 tx=0x7f6564008040).stop 2020-11-20T12:21:10.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 shutdown_connections 2020-11-20T12:21:10.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f654c056d50 0x7f654c059210 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:10.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f657409ca90 0x7f65740b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:10.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6574065190 0x7f6574066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:10.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 --2- 172.21.15.187:0/61551256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6574065f90 0x7f65740aff00 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:10.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 >> 172.21.15.187:0/61551256 conn(0x7f65740bab30 msgr2=0x7f657409c180 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:10.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 shutdown_connections 2020-11-20T12:21:10.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.111+0000 7f6579b50700 1 -- 172.21.15.187:0/61551256 wait complete. 2020-11-20T12:21:10.117 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 164 2020-11-20T12:21:10.137 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:21:10.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.415+0000 7f512ffff700 1 Processor -- start 2020-11-20T12:21:11.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.415+0000 7f512ffff700 1 -- start start 2020-11-20T12:21:11.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.416+0000 7f512ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065190 0x7f5130065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:11.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.416+0000 7f512ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f513009ca90 0x7f513009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:11.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.416+0000 7f512ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51300b13e0 con 0x7f5130065f90 2020-11-20T12:21:11.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.416+0000 7f512ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513014ab30 con 0x7f513009ca90 2020-11-20T12:21:11.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.416+0000 7f512ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5130062c10 con 0x7f5130065190 2020-11-20T12:21:11.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.417+0000 7f512effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065190 0x7f5130065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:11.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.417+0000 7f512f7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f513009ca90 0x7f513009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:11.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.417+0000 7f512effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065190 0x7f5130065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:55072/0 (socket says 172.21.15.187:55072) 2020-11-20T12:21:11.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.417+0000 7f512effd700 1 -- 172.21.15.187:0/869657335 learned_addr learned my addr 172.21.15.187:0/869657335 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:11.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.418+0000 7f512effd700 1 -- 172.21.15.187:0/869657335 >> v2:172.21.15.5:3300/0 conn(0x7f513009ca90 msgr2=0x7f513009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:11.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.418+0000 7f512effd700 1 --2- 172.21.15.187:0/869657335 >> v2:172.21.15.5:3300/0 conn(0x7f513009ca90 0x7f513009cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.418+0000 7f512effd700 1 -- 172.21.15.187:0/869657335 >> v1:172.21.15.5:6789/0 conn(0x7f5130065f90 legacy=0x7f5130143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:11.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.418+0000 7f512effd700 1 -- 172.21.15.187:0/869657335 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51300b1640 con 0x7f5130065190 2020-11-20T12:21:11.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.419+0000 7f512effd700 1 --2- 172.21.15.187:0/869657335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065190 0x7f5130065570 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f513014ac70 tx=0x7f5120006eb0).ready entity=mon.2 client_cookie=190b576e8b052cc8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:11.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.419+0000 7f512e7fc700 1 -- 172.21.15.187:0/869657335 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51200098a0 con 0x7f5130065190 2020-11-20T12:21:11.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512e7fc700 1 -- 172.21.15.187:0/869657335 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5120009a00 con 0x7f5130065190 2020-11-20T12:21:11.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512e7fc700 1 -- 172.21.15.187:0/869657335 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5120009d80 con 0x7f5130065190 2020-11-20T12:21:11.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512ffff700 1 -- 172.21.15.187:0/869657335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065190 msgr2=0x7f5130065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:11.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512ffff700 1 --2- 172.21.15.187:0/869657335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065190 0x7f5130065570 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f513014ac70 tx=0x7f5120006eb0).stop 2020-11-20T12:21:11.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512ffff700 1 -- 172.21.15.187:0/869657335 shutdown_connections 2020-11-20T12:21:11.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512ffff700 1 --2- 172.21.15.187:0/869657335 >> v2:172.21.15.5:3300/0 conn(0x7f513009ca90 0x7f513009cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512ffff700 1 --2- 172.21.15.187:0/869657335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065190 0x7f5130065570 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512ffff700 1 -- 172.21.15.187:0/869657335 >> 172.21.15.187:0/869657335 conn(0x7f51300bab30 msgr2=0x7f51300baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:11.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512ffff700 1 -- 172.21.15.187:0/869657335 shutdown_connections 2020-11-20T12:21:11.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.420+0000 7f512ffff700 1 -- 172.21.15.187:0/869657335 wait complete. 2020-11-20T12:21:11.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.422+0000 7f512ffff700 1 Processor -- start 2020-11-20T12:21:11.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.422+0000 7f512ffff700 1 -- start start 2020-11-20T12:21:11.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.423+0000 7f512ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5130065190 0x7f51300670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:11.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.423+0000 7f512ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065f90 0x7f51300b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:11.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.423+0000 7f512ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513009ca90 0x7f5130099160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:11.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.423+0000 7f512ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300685f0 con 0x7f5130065190 2020-11-20T12:21:11.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.423+0000 7f512ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5130068350 con 0x7f513009ca90 2020-11-20T12:21:11.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.423+0000 7f512ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5130146780 con 0x7f5130065f90 2020-11-20T12:21:11.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513009ca90 0x7f5130099160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:11.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513009ca90 0x7f5130099160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54194/0 (socket says 172.21.15.187:54194) 2020-11-20T12:21:11.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512f7fe700 1 -- 172.21.15.187:0/549531893 learned_addr learned my addr 172.21.15.187:0/549531893 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:11.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512effd700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5130065190 0x7f51300670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:11.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f51267fc700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065f90 0x7f51300b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:11.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512f7fe700 1 -- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065f90 msgr2=0x7f51300b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:11.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512f7fe700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065f90 0x7f51300b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.982 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512f7fe700 1 -- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5130065190 msgr2=0x7f51300670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:11.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512f7fe700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5130065190 0x7f51300670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f512f7fe700 1 -- 172.21.15.187:0/549531893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5130068070 con 0x7f513009ca90 2020-11-20T12:21:11.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.424+0000 7f51267fc700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065f90 0x7f51300b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:11.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.425+0000 7f512f7fe700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513009ca90 0x7f5130099160 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f512800ed20 tx=0x7f51280091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:11.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.425+0000 7f51277fe700 1 -- 172.21.15.187:0/549531893 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512800cbb0 con 0x7f513009ca90 2020-11-20T12:21:11.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.425+0000 7f51277fe700 1 -- 172.21.15.187:0/549531893 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512800cd10 con 0x7f513009ca90 2020-11-20T12:21:11.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.425+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5130146c20 con 0x7f513009ca90 2020-11-20T12:21:11.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.425+0000 7f51277fe700 1 -- 172.21.15.187:0/549531893 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128012620 con 0x7f513009ca90 2020-11-20T12:21:11.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.425+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51300b2af0 con 0x7f513009ca90 2020-11-20T12:21:11.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.426+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51300b2d40 con 0x7f513009ca90 2020-11-20T12:21:11.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.427+0000 7f51277fe700 1 -- 172.21.15.187:0/549531893 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5128022070 con 0x7f513009ca90 2020-11-20T12:21:11.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.428+0000 7f51277fe700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f510c056d20 0x7f510c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:11.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.428+0000 7f51277fe700 1 -- 172.21.15.187:0/549531893 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f5128070720 con 0x7f513009ca90 2020-11-20T12:21:11.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.428+0000 7f512effd700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f510c056d20 0x7f510c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:11.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.429+0000 7f512effd700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f510c056d20 0x7f510c0591e0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f5120006ef0 tx=0x7f5120015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:11.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.431+0000 7f51277fe700 1 -- 172.21.15.187:0/549531893 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5128045b20 con 0x7f513009ca90 2020-11-20T12:21:11.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.619+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5130061a20 con 0x7f513009ca90 2020-11-20T12:21:11.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.621+0000 7f51277fe700 1 -- 172.21.15.187:0/549531893 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16919 (secure 0 0 0) 0x7f5128016020 con 0x7f513009ca90 2020-11-20T12:21:11.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.623+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f510c056d20 msgr2=0x7f510c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:11.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.623+0000 7f512ffff700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f510c056d20 0x7f510c0591e0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f5120006ef0 tx=0x7f5120015000).stop 2020-11-20T12:21:11.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.623+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513009ca90 msgr2=0x7f5130099160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:11.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.623+0000 7f512ffff700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513009ca90 0x7f5130099160 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f512800ed20 tx=0x7f51280091d0).stop 2020-11-20T12:21:11.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.624+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 shutdown_connections 2020-11-20T12:21:11.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.624+0000 7f512ffff700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f510c056d20 0x7f510c0591e0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.624+0000 7f512ffff700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5130065190 0x7f51300670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.624+0000 7f512ffff700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513009ca90 0x7f5130099160 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.624+0000 7f512ffff700 1 --2- 172.21.15.187:0/549531893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5130065f90 0x7f51300b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:11.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.624+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 >> 172.21.15.187:0/549531893 conn(0x7f51300bab30 msgr2=0x7f513009aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:11.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.624+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 shutdown_connections 2020-11-20T12:21:11.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:10.624+0000 7f512ffff700 1 -- 172.21.15.187:0/549531893 wait complete. 2020-11-20T12:21:11.992 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:11.993 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":151,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:20:39.006799+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:20:19.501212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2020-11-20T12:20:20.096442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:6833/1287724648":"2020-11-21T12:20:39.006748+0000","172.21.15.5:6832/1287724648":"2020-11-21T12:20:39.006748+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:21:11.994 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2020-11-20T12:21:11.995 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2020-11-20T12:21:12.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.244+0000 7f03ba4f7700 1 Processor -- start 2020-11-20T12:21:12.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.244+0000 7f03ba4f7700 1 -- start start 2020-11-20T12:21:12.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.246+0000 7f03ba4f7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03b40b57b0 0x7f03b4068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:12.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.246+0000 7f03ba4f7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b41482d0 0x7f03b4062660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:12.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.246+0000 7f03ba4f7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b4144b80 con 0x7f03b4062b50 2020-11-20T12:21:12.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.246+0000 7f03ba4f7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b41489e0 con 0x7f03b40b57b0 2020-11-20T12:21:12.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.246+0000 7f03ba4f7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4148c80 con 0x7f03b41482d0 2020-11-20T12:21:12.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.247+0000 7f03b94f5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03b40b57b0 0x7f03b4068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:12.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.247+0000 7f03b94f5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03b40b57b0 0x7f03b4068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:46462/0 (socket says 172.21.15.187:46462) 2020-11-20T12:21:12.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.247+0000 7f03b94f5700 1 -- 172.21.15.187:0/3348043784 learned_addr learned my addr 172.21.15.187:0/3348043784 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:12.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.247+0000 7f03b8cf4700 1 --2- 172.21.15.187:0/3348043784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b41482d0 0x7f03b4062660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:12.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.248+0000 7f03b94f5700 1 -- 172.21.15.187:0/3348043784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b41482d0 msgr2=0x7f03b4062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:12.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.248+0000 7f03b94f5700 1 --2- 172.21.15.187:0/3348043784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b41482d0 0x7f03b4062660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:12.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.248+0000 7f03b94f5700 1 -- 172.21.15.187:0/3348043784 >> v1:172.21.15.5:6789/0 conn(0x7f03b4062b50 legacy=0x7f03b4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:12.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.248+0000 7f03b94f5700 1 -- 172.21.15.187:0/3348043784 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b4148f20 con 0x7f03b40b57b0 2020-11-20T12:21:12.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.248+0000 7f03b94f5700 1 --2- 172.21.15.187:0/3348043784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03b40b57b0 0x7f03b4068230 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f03b0006fd0 tx=0x7f03b0006eb0).ready entity=mon.0 client_cookie=5a84b59e134dbd1e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:12.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.249+0000 7f03a3fff700 1 -- 172.21.15.187:0/3348043784 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0009810 con 0x7f03b40b57b0 2020-11-20T12:21:12.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.249+0000 7f03a3fff700 1 -- 172.21.15.187:0/3348043784 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b0009970 con 0x7f03b40b57b0 2020-11-20T12:21:12.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.249+0000 7f03a3fff700 1 -- 172.21.15.187:0/3348043784 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0009cf0 con 0x7f03b40b57b0 2020-11-20T12:21:12.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.249+0000 7f03ba4f7700 1 -- 172.21.15.187:0/3348043784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03b40b57b0 msgr2=0x7f03b4068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:12.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.249+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/3348043784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03b40b57b0 0x7f03b4068230 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f03b0006fd0 tx=0x7f03b0006eb0).stop 2020-11-20T12:21:12.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.250+0000 7f03ba4f7700 1 -- 172.21.15.187:0/3348043784 shutdown_connections 2020-11-20T12:21:12.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.250+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/3348043784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03b40b57b0 0x7f03b4068230 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:12.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.250+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/3348043784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b41482d0 0x7f03b4062660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:12.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.250+0000 7f03ba4f7700 1 -- 172.21.15.187:0/3348043784 >> 172.21.15.187:0/3348043784 conn(0x7f03b40bab30 msgr2=0x7f03b40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:12.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.250+0000 7f03ba4f7700 1 -- 172.21.15.187:0/3348043784 shutdown_connections 2020-11-20T12:21:12.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.250+0000 7f03ba4f7700 1 -- 172.21.15.187:0/3348043784 wait complete. 2020-11-20T12:21:12.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.251+0000 7f03ba4f7700 1 Processor -- start 2020-11-20T12:21:12.257 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.251+0000 7f03ba4f7700 1 -- start start 2020-11-20T12:21:12.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.252+0000 7f03ba4f7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b4062b50 0x7f03b4146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:12.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.252+0000 7f03ba4f7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40b57b0 0x7f03b41585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:12.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.252+0000 7f03ba4f7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03b41482d0 0x7f03b41476e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:12.258 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.252+0000 7f03ba4f7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4143b70 con 0x7f03b41482d0 2020-11-20T12:21:12.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.252+0000 7f03ba4f7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b41480b0 con 0x7f03b40b57b0 2020-11-20T12:21:12.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.252+0000 7f03ba4f7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4146780 con 0x7f03b4062b50 2020-11-20T12:21:12.259 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40b57b0 0x7f03b41585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:12.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40b57b0 0x7f03b41585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54204/0 (socket says 172.21.15.187:54204) 2020-11-20T12:21:12.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 -- 172.21.15.187:0/1333990199 learned_addr learned my addr 172.21.15.187:0/1333990199 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:12.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b94f5700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b4062b50 0x7f03b4146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:12.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 -- 172.21.15.187:0/1333990199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b4062b50 msgr2=0x7f03b4146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:12.260 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b4062b50 0x7f03b4146190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:12.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 -- 172.21.15.187:0/1333990199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03b41482d0 msgr2=0x7f03b41476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:12.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03b41482d0 0x7f03b41476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:12.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 -- 172.21.15.187:0/1333990199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b4146a20 con 0x7f03b40b57b0 2020-11-20T12:21:12.261 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.253+0000 7f03b8cf4700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40b57b0 0x7f03b41585e0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 rx=0x7f03a400ec70 tx=0x7f03a4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:12.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.254+0000 7f03a17fa700 1 -- 172.21.15.187:0/1333990199 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a400ab90 con 0x7f03b40b57b0 2020-11-20T12:21:12.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.254+0000 7f03a17fa700 1 -- 172.21.15.187:0/1333990199 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a400acf0 con 0x7f03b40b57b0 2020-11-20T12:21:12.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.254+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b4146cc0 con 0x7f03b40b57b0 2020-11-20T12:21:12.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.254+0000 7f03a17fa700 1 -- 172.21.15.187:0/1333990199 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a400ae60 con 0x7f03b40b57b0 2020-11-20T12:21:12.262 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.254+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b4147160 con 0x7f03b40b57b0 2020-11-20T12:21:12.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.255+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b4146e50 con 0x7f03b40b57b0 2020-11-20T12:21:12.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.255+0000 7f03a17fa700 1 -- 172.21.15.187:0/1333990199 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f03a4020070 con 0x7f03b40b57b0 2020-11-20T12:21:12.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.256+0000 7f03a17fa700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f038c056d20 0x7f038c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:12.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.256+0000 7f03a17fa700 1 -- 172.21.15.187:0/1333990199 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f03a406fea0 con 0x7f03b40b57b0 2020-11-20T12:21:12.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.257+0000 7f03b94f5700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f038c056d20 0x7f038c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:12.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.257+0000 7f03b94f5700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f038c056d20 0x7f038c0591e0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f03b000beb0 tx=0x7f03b0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:12.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.260+0000 7f03a17fa700 1 -- 172.21.15.187:0/1333990199 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f03a40452a0 con 0x7f03b40b57b0 2020-11-20T12:21:12.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.509+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f03b4061a20 con 0x7f03b40b57b0 2020-11-20T12:21:12.980 INFO:tasks.ceph.mon.b.smithi005.stderr:2020-11-20T12:21:12.976+0000 7f8d071a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-11-20T12:21:12.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.992+0000 7f03a17fa700 1 -- 172.21.15.187:0/1333990199 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v165) v1 ==== 161+0+0 (secure 0 0 0) 0x7f03a401e070 con 0x7f03b40b57b0 2020-11-20T12:21:12.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f038c056d20 msgr2=0x7f038c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:12.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f038c056d20 0x7f038c0591e0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f03b000beb0 tx=0x7f03b0015000).stop 2020-11-20T12:21:12.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40b57b0 msgr2=0x7f03b41585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:12.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40b57b0 0x7f03b41585e0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 rx=0x7f03a400ec70 tx=0x7f03a4008040).stop 2020-11-20T12:21:12.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 shutdown_connections 2020-11-20T12:21:12.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f038c056d20 0x7f038c0591e0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:12.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03b41482d0 0x7f03b41476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:12.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03b4062b50 0x7f03b4146190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:12.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 --2- 172.21.15.187:0/1333990199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40b57b0 0x7f03b41585e0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:12.999 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 >> 172.21.15.187:0/1333990199 conn(0x7f03b40bab30 msgr2=0x7f03b40b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:13.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 shutdown_connections 2020-11-20T12:21:13.000 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:12.995+0000 7f03ba4f7700 1 -- 172.21.15.187:0/1333990199 wait complete. 2020-11-20T12:21:13.001 INFO:teuthology.orchestra.run.smithi187.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2020-11-20T12:21:13.022 INFO:tasks.ceph.mds.a.smithi187.stderr: -5910> 2020-11-20T12:20:24.214+0000 7f4976072700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:21:13.027 INFO:tasks.ceph.mds.a.smithi187.stderr: -5448> 2020-11-20T12:20:34.214+0000 7f4976072700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2020-11-20T12:21:13.029 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2020-11-20T12:21:13.035 INFO:tasks.ceph.mds.a.smithi187.stderr: -4719> 2020-11-20T12:20:49.215+0000 7f4976072700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2020-11-20T12:21:13.099 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:21:13.127 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:21:13.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.260+0000 7fc18d035700 1 Processor -- start 2020-11-20T12:21:13.263 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.260+0000 7fc18d035700 1 -- start start 2020-11-20T12:21:13.264 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc18d035700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188065e30 0x7fc188099da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:13.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc18d035700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1880b52f0 0x7fc18809a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:13.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc18d035700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc188148b60 con 0x7fc188066210 2020-11-20T12:21:13.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc18d035700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc188146020 con 0x7fc1880b52f0 2020-11-20T12:21:13.265 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc18d035700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc188149540 con 0x7fc188065e30 2020-11-20T12:21:13.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc18759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1880b52f0 0x7fc18809a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:13.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc18759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1880b52f0 0x7fc18809a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54232/0 (socket says 172.21.15.187:54232) 2020-11-20T12:21:13.266 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc18759e700 1 -- 172.21.15.187:0/3461924401 learned_addr learned my addr 172.21.15.187:0/3461924401 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:13.267 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.262+0000 7fc186d9d700 1 --2- 172.21.15.187:0/3461924401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188065e30 0x7fc188099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:13.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.263+0000 7fc18759e700 1 -- 172.21.15.187:0/3461924401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188065e30 msgr2=0x7fc188099da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:13.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.263+0000 7fc18759e700 1 --2- 172.21.15.187:0/3461924401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188065e30 0x7fc188099da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:13.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.263+0000 7fc18759e700 1 -- 172.21.15.187:0/3461924401 >> v1:172.21.15.5:6789/0 conn(0x7fc188066210 legacy=0x7fc188061de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:13.268 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.263+0000 7fc18759e700 1 -- 172.21.15.187:0/3461924401 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1880b72c0 con 0x7fc1880b52f0 2020-11-20T12:21:13.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.263+0000 7fc18759e700 1 --2- 172.21.15.187:0/3461924401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18809a290 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7fc180009d50 tx=0x7fc180009290).ready entity=mon.1 client_cookie=5994f0911c2baff server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:13.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.263+0000 7fc185d9b700 1 -- 172.21.15.187:0/3461924401 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18000bc80 con 0x7fc1880b52f0 2020-11-20T12:21:13.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc185d9b700 1 -- 172.21.15.187:0/3461924401 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18000bde0 con 0x7fc1880b52f0 2020-11-20T12:21:13.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc185d9b700 1 -- 172.21.15.187:0/3461924401 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180011610 con 0x7fc1880b52f0 2020-11-20T12:21:13.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc18d035700 1 -- 172.21.15.187:0/3461924401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 msgr2=0x7fc18809a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:13.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc18d035700 1 --2- 172.21.15.187:0/3461924401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18809a290 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7fc180009d50 tx=0x7fc180009290).stop 2020-11-20T12:21:13.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc18d035700 1 -- 172.21.15.187:0/3461924401 shutdown_connections 2020-11-20T12:21:13.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc18d035700 1 --2- 172.21.15.187:0/3461924401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18809a290 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:13.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc18d035700 1 --2- 172.21.15.187:0/3461924401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188065e30 0x7fc188099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:13.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc18d035700 1 -- 172.21.15.187:0/3461924401 >> 172.21.15.187:0/3461924401 conn(0x7fc1880bacf0 msgr2=0x7fc1880bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:13.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc18d035700 1 -- 172.21.15.187:0/3461924401 shutdown_connections 2020-11-20T12:21:13.271 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.264+0000 7fc18d035700 1 -- 172.21.15.187:0/3461924401 wait complete. 2020-11-20T12:21:13.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.266+0000 7fc18d035700 1 Processor -- start 2020-11-20T12:21:13.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.266+0000 7fc18d035700 1 -- start start 2020-11-20T12:21:13.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18d035700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc188065e30 0x7fc188157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:13.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18d035700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188066210 0x7fc18815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:13.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18d035700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18815c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:13.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18d035700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1880b4630 con 0x7fc188065e30 2020-11-20T12:21:13.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18d035700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18815c500 con 0x7fc1880b52f0 2020-11-20T12:21:13.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18d035700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18809cc60 con 0x7fc188066210 2020-11-20T12:21:13.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18815c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:13.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18815c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54236/0 (socket says 172.21.15.187:54236) 2020-11-20T12:21:13.275 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc18759e700 1 -- 172.21.15.187:0/3988611305 learned_addr learned my addr 172.21.15.187:0/3988611305 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:13.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.267+0000 7fc186d9d700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc188065e30 0x7fc188157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:13.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc18759e700 1 -- 172.21.15.187:0/3988611305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188066210 msgr2=0x7fc18815bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:13.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc18759e700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188066210 0x7fc18815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:13.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc18759e700 1 -- 172.21.15.187:0/3988611305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc188065e30 msgr2=0x7fc188157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:13.276 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc18759e700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc188065e30 0x7fc188157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:13.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc18759e700 1 -- 172.21.15.187:0/3988611305 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1880b4d30 con 0x7fc1880b52f0 2020-11-20T12:21:13.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc18759e700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18815c010 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7fc1880643a0 tx=0x7fc180016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:13.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc16effd700 1 -- 172.21.15.187:0/3988611305 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18000bc80 con 0x7fc1880b52f0 2020-11-20T12:21:13.277 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc16effd700 1 -- 172.21.15.187:0/3988611305 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc180010c40 con 0x7fc1880b52f0 2020-11-20T12:21:13.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18815c740 con 0x7fc1880b52f0 2020-11-20T12:21:13.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc16effd700 1 -- 172.21.15.187:0/3988611305 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18001c470 con 0x7fc1880b52f0 2020-11-20T12:21:13.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.268+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18809de60 con 0x7fc1880b52f0 2020-11-20T12:21:13.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.269+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1880b4a40 con 0x7fc1880b52f0 2020-11-20T12:21:13.278 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.270+0000 7fc16effd700 1 -- 172.21.15.187:0/3988611305 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc180017070 con 0x7fc1880b52f0 2020-11-20T12:21:13.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.270+0000 7fc16effd700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc164056d20 0x7fc1640591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:13.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.270+0000 7fc16effd700 1 -- 172.21.15.187:0/3988611305 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fc180072350 con 0x7fc1880b52f0 2020-11-20T12:21:13.279 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.271+0000 7fc186d9d700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc164056d20 0x7fc1640591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:13.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.271+0000 7fc186d9d700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc164056d20 0x7fc1640591e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fc1780095b0 tx=0x7fc178008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:13.280 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.274+0000 7fc16effd700 1 -- 172.21.15.187:0/3988611305 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc1880b4a40 con 0x7fc1880b52f0 2020-11-20T12:21:13.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:13.531+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc1880b4a40 con 0x7fc1880b52f0 2020-11-20T12:21:14.007 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.005+0000 7fc16effd700 1 -- 172.21.15.187:0/3988611305 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v166) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc1880b4a40 con 0x7fc1880b52f0 2020-11-20T12:21:14.015 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc164056d20 msgr2=0x7fc1640591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:14.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc164056d20 0x7fc1640591e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fc1780095b0 tx=0x7fc178008040).stop 2020-11-20T12:21:14.016 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 msgr2=0x7fc18815c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:14.017 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18815c010 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7fc1880643a0 tx=0x7fc180016000).stop 2020-11-20T12:21:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 shutdown_connections 2020-11-20T12:21:14.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc164056d20 0x7fc1640591e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc188065e30 0x7fc188157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:14.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1880b52f0 0x7fc18815c010 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:14.021 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.008+0000 7fc18d035700 1 --2- 172.21.15.187:0/3988611305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc188066210 0x7fc18815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:14.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.009+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 >> 172.21.15.187:0/3988611305 conn(0x7fc1880bacf0 msgr2=0x7fc188066dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:14.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.009+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 shutdown_connections 2020-11-20T12:21:14.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.009+0000 7fc18d035700 1 -- 172.21.15.187:0/3988611305 wait complete. 2020-11-20T12:21:14.039 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2020-11-20T12:21:14.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.316+0000 7f3e66874700 1 Processor -- start 2020-11-20T12:21:14.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.316+0000 7f3e66874700 1 -- start start 2020-11-20T12:21:14.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.318+0000 7f3e66874700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e60065190 0x7f3e60065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:14.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.318+0000 7f3e66874700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e600b13e0 con 0x7f3e6009ca90 2020-11-20T12:21:14.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.318+0000 7f3e66874700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e60062c10 con 0x7f3e60065f90 2020-11-20T12:21:14.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.318+0000 7f3e66874700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6014ab30 con 0x7f3e60065190 2020-11-20T12:21:14.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.318+0000 7f3e65071700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3e60065f90 0x7f3e60143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50850/0 (socket says 172.21.15.187:50850) 2020-11-20T12:21:14.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.318+0000 7f3e65071700 1 -- 172.21.15.187:0/116911375 learned_addr learned my addr 172.21.15.187:0/116911375 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:14.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.318+0000 7f3e65872700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e60065190 0x7f3e60065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:14.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487853255 0 0) 0x7f3e60062c10 con 0x7f3e60065f90 2020-11-20T12:21:14.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e48003240 con 0x7f3e60065f90 2020-11-20T12:21:14.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3809681128 0 0) 0x7f3e48003240 con 0x7f3e60065f90 2020-11-20T12:21:14.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e60065190 msgr2=0x7f3e60065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:14.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e60065190 0x7f3e60065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:14.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 >> v1:172.21.15.5:6789/0 conn(0x7f3e6009ca90 legacy=0x7f3e6009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:14.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600b1640 con 0x7f3e60065f90 2020-11-20T12:21:14.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f3e500014e0 con 0x7f3e60065f90 2020-11-20T12:21:14.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 >> v1:172.21.15.187:6789/0 conn(0x7f3e60065f90 legacy=0x7f3e60143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:14.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e48003730 0x7f3e48003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:14.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e60065190 0x7f3e480044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:14.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e480049d0 0x7f3e48004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:14.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6014ab30 con 0x7f3e480049d0 2020-11-20T12:21:14.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480041d0 con 0x7f3e48003730 2020-11-20T12:21:14.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.319+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4800b730 con 0x7f3e60065190 2020-11-20T12:21:14.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.320+0000 7f3e65872700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e48003730 0x7f3e48003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:14.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.320+0000 7f3e65071700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e60065190 0x7f3e480044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:14.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.320+0000 7f3e65872700 1 -- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e60065190 msgr2=0x7f3e480044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:14.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.320+0000 7f3e65872700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e60065190 0x7f3e480044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:14.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.320+0000 7f3e65872700 1 -- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e480049d0 msgr2=0x7f3e48004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:14.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.320+0000 7f3e65872700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e480049d0 0x7f3e48004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:14.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.320+0000 7f3e65872700 1 -- 172.21.15.187:0/116911375 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e48003240 con 0x7f3e48003730 2020-11-20T12:21:14.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.321+0000 7f3e65872700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e48003730 0x7f3e48003b50 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7f3e5c00bc30 tx=0x7f3e5c006df0).ready entity=mon.1 client_cookie=c7dc4e8681e0e3e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.321+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c00b310 con 0x7f3e48003730 2020-11-20T12:21:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.321+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c00eb40 con 0x7f3e48003730 2020-11-20T12:21:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.321+0000 7f3e64870700 1 -- 172.21.15.187:0/116911375 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c00eec0 con 0x7f3e48003730 2020-11-20T12:21:14.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.322+0000 7f3e66874700 1 -- 172.21.15.187:0/116911375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e48003730 msgr2=0x7f3e48003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:14.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.322+0000 7f3e66874700 1 --2- 172.21.15.187:0/116911375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e48003730 0x7f3e48003b50 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7f3e5c00bc30 tx=0x7f3e5c006df0).stop 2020-11-20T12:21:14.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.322+0000 7f3e66073700 1 -- 172.21.15.187:0/116911375 reap_dead start 2020-11-20T12:21:14.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.322+0000 7f3e66874700 1 -- 172.21.15.187:0/116911375 shutdown_connections 2020-11-20T12:21:14.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.322+0000 7f3e66874700 1 -- 172.21.15.187:0/116911375 >> 172.21.15.187:0/116911375 conn(0x7f3e600bab30 msgr2=0x7f3e600baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:14.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.322+0000 7f3e66874700 1 -- 172.21.15.187:0/116911375 shutdown_connections 2020-11-20T12:21:14.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.322+0000 7f3e66874700 1 -- 172.21.15.187:0/116911375 wait complete. 2020-11-20T12:21:14.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.323+0000 7f3e66874700 1 Processor -- start 2020-11-20T12:21:14.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.324+0000 7f3e66874700 1 -- start start 2020-11-20T12:21:14.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e66874700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e600b41a0 0x7f3e601482f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:14.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e66874700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e600b4580 0x7f3e601487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:14.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e66874700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e600b2710 0x7f3e600b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:14.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e66874700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e60145940 con 0x7f3e600b2710 2020-11-20T12:21:14.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e66874700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e60149020 con 0x7f3e600b41a0 2020-11-20T12:21:14.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e66874700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e601590f0 con 0x7f3e600b4580 2020-11-20T12:21:14.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65872700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e600b41a0 0x7f3e601482f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:14.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65872700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e600b41a0 0x7f3e601482f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54254/0 (socket says 172.21.15.187:54254) 2020-11-20T12:21:14.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65872700 1 -- 172.21.15.187:0/756494430 learned_addr learned my addr 172.21.15.187:0/756494430 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:14.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65071700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e600b4580 0x7f3e601487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:14.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65872700 1 -- 172.21.15.187:0/756494430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e600b4580 msgr2=0x7f3e601487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:14.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65872700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e600b4580 0x7f3e601487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:14.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65872700 1 -- 172.21.15.187:0/756494430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e600b2710 msgr2=0x7f3e600b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:14.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65872700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e600b2710 0x7f3e600b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:14.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.325+0000 7f3e65872700 1 -- 172.21.15.187:0/756494430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e60159390 con 0x7f3e600b41a0 2020-11-20T12:21:14.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.326+0000 7f3e65872700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e600b41a0 0x7f3e601482f0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f3e5c00bc00 tx=0x7f3e5c019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:14.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.326+0000 7f3e4dffb700 1 -- 172.21.15.187:0/756494430 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c01e630 con 0x7f3e600b41a0 2020-11-20T12:21:14.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.326+0000 7f3e4dffb700 1 -- 172.21.15.187:0/756494430 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c00a070 con 0x7f3e600b41a0 2020-11-20T12:21:14.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.326+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e60148cd0 con 0x7f3e600b41a0 2020-11-20T12:21:14.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.326+0000 7f3e4dffb700 1 -- 172.21.15.187:0/756494430 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c01e790 con 0x7f3e600b41a0 2020-11-20T12:21:14.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.326+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e60147980 con 0x7f3e600b41a0 2020-11-20T12:21:14.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.328+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e60148e60 con 0x7f3e600b41a0 2020-11-20T12:21:14.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.328+0000 7f3e4dffb700 1 -- 172.21.15.187:0/756494430 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3e5c01eb40 con 0x7f3e600b41a0 2020-11-20T12:21:14.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.329+0000 7f3e4dffb700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e38056d20 0x7f3e380591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:14.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.329+0000 7f3e4dffb700 1 -- 172.21.15.187:0/756494430 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f3e5c072580 con 0x7f3e600b41a0 2020-11-20T12:21:14.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.329+0000 7f3e65071700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e38056d20 0x7f3e380591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:14.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.330+0000 7f3e65071700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e38056d20 0x7f3e380591e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f3e50001610 tx=0x7f3e600bb650).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:14.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.333+0000 7f3e4dffb700 1 -- 172.21.15.187:0/756494430 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3e5c047950 con 0x7f3e600b41a0 2020-11-20T12:21:14.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:14.521+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3e60147c30 con 0x7f3e600b41a0 2020-11-20T12:21:15.013 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.010+0000 7f3e4dffb700 1 -- 172.21.15.187:0/756494430 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v153) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3e5c04af70 con 0x7f3e600b41a0 2020-11-20T12:21:15.018 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.013+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e38056d20 msgr2=0x7f3e380591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:15.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.013+0000 7f3e66874700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e38056d20 0x7f3e380591e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f3e50001610 tx=0x7f3e600bb650).stop 2020-11-20T12:21:15.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.013+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e600b41a0 msgr2=0x7f3e601482f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:15.022 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.013+0000 7f3e66874700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e600b41a0 0x7f3e601482f0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f3e5c00bc00 tx=0x7f3e5c019000).stop 2020-11-20T12:21:15.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.014+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 shutdown_connections 2020-11-20T12:21:15.023 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.014+0000 7f3e66874700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3e38056d20 0x7f3e380591e0 secure :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0x7f3e50001610 tx=0x7f3e600bb650).stop 2020-11-20T12:21:15.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.014+0000 7f3e66874700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e600b2710 0x7f3e600b2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:15.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.014+0000 7f3e66874700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e600b41a0 0x7f3e601482f0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:15.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.014+0000 7f3e66874700 1 --2- 172.21.15.187:0/756494430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e600b4580 0x7f3e601487e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:15.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.014+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 >> 172.21.15.187:0/756494430 conn(0x7f3e600bab30 msgr2=0x7f3e600affa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:15.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.014+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 shutdown_connections 2020-11-20T12:21:15.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.014+0000 7f3e66874700 1 -- 172.21.15.187:0/756494430 wait complete. 2020-11-20T12:21:15.028 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' removed 2020-11-20T12:21:15.044 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2020-11-20T12:21:15.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.301+0000 7fd86acc9700 1 Processor -- start 2020-11-20T12:21:15.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.301+0000 7fd86acc9700 1 -- start start 2020-11-20T12:21:15.318 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.302+0000 7fd86acc9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b57b0 0x7fd864068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.302+0000 7fd86acc9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd8640b5f50 0x7fd8640b6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.302+0000 7fd86acc9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd864144b80 con 0x7fd8641482d0 2020-11-20T12:21:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.302+0000 7fd86acc9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8641489e0 con 0x7fd8640b5f50 2020-11-20T12:21:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.302+0000 7fd86acc9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd864148c80 con 0x7fd8640b57b0 2020-11-20T12:21:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.303+0000 7fd869cc7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b57b0 0x7fd864068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:15.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.303+0000 7fd8694c6700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd8641482d0 0x7fd864062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:44378/0 (socket says 172.21.15.187:44378) 2020-11-20T12:21:15.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.303+0000 7fd8694c6700 1 -- 172.21.15.187:0/304422730 learned_addr learned my addr 172.21.15.187:0/304422730 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:15.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.304+0000 7fd86a4c8700 1 --2- 172.21.15.187:0/304422730 >> v2:172.21.15.5:3300/0 conn(0x7fd8640b5f50 0x7fd8640b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:15.320 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.304+0000 7fd869cc7700 1 -- 172.21.15.187:0/304422730 >> v2:172.21.15.5:3300/0 conn(0x7fd8640b5f50 msgr2=0x7fd8640b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:15.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.304+0000 7fd869cc7700 1 --2- 172.21.15.187:0/304422730 >> v2:172.21.15.5:3300/0 conn(0x7fd8640b5f50 0x7fd8640b6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:15.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.305+0000 7fd869cc7700 1 -- 172.21.15.187:0/304422730 >> v1:172.21.15.5:6789/0 conn(0x7fd8641482d0 legacy=0x7fd864062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:15.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.305+0000 7fd869cc7700 1 -- 172.21.15.187:0/304422730 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd864148f20 con 0x7fd8640b57b0 2020-11-20T12:21:15.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.304+0000 7fd868cc5700 1 -- 172.21.15.187:0/304422730 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992863290 0 0) 0x7fd864144b80 con 0x7fd8641482d0 2020-11-20T12:21:15.321 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.305+0000 7fd869cc7700 1 --2- 172.21.15.187:0/304422730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b57b0 0x7fd864068230 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fd86000b9a0 tx=0x7fd860006df0).ready entity=mon.2 client_cookie=8fe33bd55f4feb5a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:15.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.305+0000 7fd868cc5700 1 -- 172.21.15.187:0/304422730 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600097b0 con 0x7fd8640b57b0 2020-11-20T12:21:15.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd868cc5700 1 -- 172.21.15.187:0/304422730 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860009910 con 0x7fd8640b57b0 2020-11-20T12:21:15.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd868cc5700 1 -- 172.21.15.187:0/304422730 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860009c90 con 0x7fd8640b57b0 2020-11-20T12:21:15.322 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd86acc9700 1 -- 172.21.15.187:0/304422730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b57b0 msgr2=0x7fd864068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd86acc9700 1 --2- 172.21.15.187:0/304422730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b57b0 0x7fd864068230 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fd86000b9a0 tx=0x7fd860006df0).stop 2020-11-20T12:21:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd86acc9700 1 -- 172.21.15.187:0/304422730 shutdown_connections 2020-11-20T12:21:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd86acc9700 1 --2- 172.21.15.187:0/304422730 >> v2:172.21.15.5:3300/0 conn(0x7fd8640b5f50 0x7fd8640b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd86acc9700 1 --2- 172.21.15.187:0/304422730 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b57b0 0x7fd864068230 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:15.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd86acc9700 1 -- 172.21.15.187:0/304422730 >> 172.21.15.187:0/304422730 conn(0x7fd8640bab30 msgr2=0x7fd8640baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd86acc9700 1 -- 172.21.15.187:0/304422730 shutdown_connections 2020-11-20T12:21:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.306+0000 7fd86acc9700 1 -- 172.21.15.187:0/304422730 wait complete. 2020-11-20T12:21:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.308+0000 7fd86acc9700 1 Processor -- start 2020-11-20T12:21:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.308+0000 7fd86acc9700 1 -- start start 2020-11-20T12:21:15.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86acc9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8640b57b0 0x7fd864132ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:15.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86acc9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b5f50 0x7fd86413c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:15.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86acc9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8641482d0 0x7fd8641339e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:15.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86acc9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640a0850 con 0x7fd8640b57b0 2020-11-20T12:21:15.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86acc9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86413c6d0 con 0x7fd8641482d0 2020-11-20T12:21:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86acc9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86413baa0 con 0x7fd8640b5f50 2020-11-20T12:21:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86a4c8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8641482d0 0x7fd8641339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86a4c8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8641482d0 0x7fd8641339e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54272/0 (socket says 172.21.15.187:54272) 2020-11-20T12:21:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd86a4c8700 1 -- 172.21.15.187:0/1017758995 learned_addr learned my addr 172.21.15.187:0/1017758995 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:15.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd869cc7700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8640b57b0 0x7fd864132ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.309+0000 7fd8694c6700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b5f50 0x7fd86413c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd86a4c8700 1 -- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b5f50 msgr2=0x7fd86413c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd86a4c8700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b5f50 0x7fd86413c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd86a4c8700 1 -- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8640b57b0 msgr2=0x7fd864132ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd86a4c8700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8640b57b0 0x7fd864132ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:15.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd86a4c8700 1 -- 172.21.15.187:0/1017758995 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86413bd40 con 0x7fd8641482d0 2020-11-20T12:21:15.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd869cc7700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8640b57b0 0x7fd864132ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:15.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd8694c6700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b5f50 0x7fd86413c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:15.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd86a4c8700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8641482d0 0x7fd8641339e0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7fd85c00ef50 tx=0x7fd85c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:15.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.310+0000 7fd855ffb700 1 -- 172.21.15.187:0/1017758995 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00cd20 con 0x7fd8641482d0 2020-11-20T12:21:15.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.311+0000 7fd855ffb700 1 -- 172.21.15.187:0/1017758995 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c00ce80 con 0x7fd8641482d0 2020-11-20T12:21:15.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.311+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8640a0460 con 0x7fd8641482d0 2020-11-20T12:21:15.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.311+0000 7fd855ffb700 1 -- 172.21.15.187:0/1017758995 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c012620 con 0x7fd8641482d0 2020-11-20T12:21:15.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.311+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd864133ed0 con 0x7fd8641482d0 2020-11-20T12:21:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.312+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86413bed0 con 0x7fd8641482d0 2020-11-20T12:21:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.312+0000 7fd855ffb700 1 -- 172.21.15.187:0/1017758995 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd85c022070 con 0x7fd8641482d0 2020-11-20T12:21:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.313+0000 7fd855ffb700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd83c056d50 0x7fd83c059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.313+0000 7fd855ffb700 1 -- 172.21.15.187:0/1017758995 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fd85c0707e0 con 0x7fd8641482d0 2020-11-20T12:21:15.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.313+0000 7fd869cc7700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd83c056d50 0x7fd83c059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:15.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.314+0000 7fd869cc7700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd83c056d50 0x7fd83c059210 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7fd86000be60 tx=0x7fd860015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:15.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.317+0000 7fd855ffb700 1 -- 172.21.15.187:0/1017758995 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd85c045ce0 con 0x7fd8641482d0 2020-11-20T12:21:15.506 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:15.503+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd864134150 con 0x7fd8641482d0 2020-11-20T12:21:16.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.022+0000 7fd855ffb700 1 -- 172.21.15.187:0/1017758995 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v154) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd85c016030 con 0x7fd8641482d0 2020-11-20T12:21:16.028 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd83c056d50 msgr2=0x7fd83c059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:16.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd83c056d50 0x7fd83c059210 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7fd86000be60 tx=0x7fd860015000).stop 2020-11-20T12:21:16.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8641482d0 msgr2=0x7fd8641339e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:16.029 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8641482d0 0x7fd8641339e0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7fd85c00ef50 tx=0x7fd85c0091d0).stop 2020-11-20T12:21:16.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 shutdown_connections 2020-11-20T12:21:16.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd83c056d50 0x7fd83c059210 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:16.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8640b57b0 0x7fd864132ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:16.030 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8641482d0 0x7fd8641339e0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:16.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 --2- 172.21.15.187:0/1017758995 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8640b5f50 0x7fd86413c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:16.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.026+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 >> 172.21.15.187:0/1017758995 conn(0x7fd8640bab30 msgr2=0x7fd864146d80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:16.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.027+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 shutdown_connections 2020-11-20T12:21:16.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.027+0000 7fd86acc9700 1 -- 172.21.15.187:0/1017758995 wait complete. 2020-11-20T12:21:16.032 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' removed 2020-11-20T12:21:16.054 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:21:16.054 INFO:tasks.ceph.mds.a:Stopping old one... 2020-11-20T12:21:16.055 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:21:16.055 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:21:16.055 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:21:16.056 INFO:tasks.ceph.mds.b:Stopping old one... 2020-11-20T12:21:16.056 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:21:16.056 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:21:16.075 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:21:16.075 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:21:16.083 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:21:16.084 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:21:16.101 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:21:16.101 DEBUG:teuthology.parallel:result is None 2020-11-20T12:21:16.132 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:21:16.132 DEBUG:teuthology.parallel:result is None 2020-11-20T12:21:16.133 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2020-11-20T12:21:16.306 INFO:tasks.ceph.mds.a.smithi187.stdout:starting mds.a at 2020-11-20T12:21:17.310 INFO:tasks.ceph.mds.b.smithi005.stdout:starting mds.b at 2020-11-20T12:21:17.323 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.360+0000 7f5a1e15d700 1 Processor -- start 2020-11-20T12:21:17.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.360+0000 7f5a1e15d700 1 -- start start 2020-11-20T12:21:17.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.361+0000 7f5a1e15d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180bf800 0x7f5a1814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.361+0000 7f5a1e15d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5a1809f560 0x7f5a180a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.362+0000 7f5a1e15d700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1814d110 con 0x7f5a180bfbe0 2020-11-20T12:21:17.324 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.362+0000 7f5a1e15d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1814cbd0 con 0x7f5a1809f560 2020-11-20T12:21:17.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.362+0000 7f5a1e15d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1814ce70 con 0x7f5a180bf800 2020-11-20T12:21:17.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.362+0000 7f5a1c95a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5a180bfbe0 0x7f5a181480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50894/0 (socket says 172.21.15.187:50894) 2020-11-20T12:21:17.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.362+0000 7f5a1c95a700 1 -- 172.21.15.187:0/174850491 learned_addr learned my addr 172.21.15.187:0/174850491 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:17.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.362+0000 7f5a1d95c700 1 --2- 172.21.15.187:0/174850491 >> v2:172.21.15.5:3300/0 conn(0x7f5a1809f560 0x7f5a180a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.362+0000 7f5a1d15b700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180bf800 0x7f5a1814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.362+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895073584 0 0) 0x7f5a1814d110 con 0x7f5a180bfbe0 2020-11-20T12:21:17.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00003240 con 0x7f5a180bfbe0 2020-11-20T12:21:17.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2449001196 0 0) 0x7f5a00003240 con 0x7f5a180bfbe0 2020-11-20T12:21:17.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180bf800 msgr2=0x7f5a1814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:17.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180bf800 0x7f5a1814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 >> v2:172.21.15.5:3300/0 conn(0x7f5a1809f560 msgr2=0x7f5a180a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:17.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 --2- 172.21.15.187:0/174850491 >> v2:172.21.15.5:3300/0 conn(0x7f5a1809f560 0x7f5a180a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180a4cc0 con 0x7f5a180bfbe0 2020-11-20T12:21:17.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5a080014e0 con 0x7f5a180bfbe0 2020-11-20T12:21:17.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 >> v1:172.21.15.187:6789/0 conn(0x7f5a180bfbe0 legacy=0x7f5a181480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a000037b0 0x7f5a00003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a000044e0 0x7f5a00004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.328 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180bf800 0x7f5a00004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1814cbd0 con 0x7f5a000037b0 2020-11-20T12:21:17.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1814ce70 con 0x7f5a000044e0 2020-11-20T12:21:17.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a00005340 con 0x7f5a180bf800 2020-11-20T12:21:17.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.363+0000 7f5a1c95a700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a000044e0 0x7f5a00004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.329 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a1d15b700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a000037b0 0x7f5a00003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a1d95c700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180bf800 0x7f5a00004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a1c95a700 1 -- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180bf800 msgr2=0x7f5a00004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:17.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a1c95a700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180bf800 0x7f5a00004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a1c95a700 1 -- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a000037b0 msgr2=0x7f5a00003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:17.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a1c95a700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a000037b0 0x7f5a00003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a1c95a700 1 -- 172.21.15.187:0/174850491 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a00003240 con 0x7f5a000044e0 2020-11-20T12:21:17.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a1c95a700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a000044e0 0x7f5a00004960 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7f5a180bcaa0 tx=0x7f5a08007750).ready entity=mon.1 client_cookie=57df8879c9754200 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:17.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a080070d0 con 0x7f5a000044e0 2020-11-20T12:21:17.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.364+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08007230 con 0x7f5a000044e0 2020-11-20T12:21:17.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.365+0000 7f5a07fff700 1 -- 172.21.15.187:0/174850491 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a080055f0 con 0x7f5a000044e0 2020-11-20T12:21:17.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.365+0000 7f5a1e15d700 1 -- 172.21.15.187:0/174850491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a000044e0 msgr2=0x7f5a00004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.365+0000 7f5a1e15d700 1 --2- 172.21.15.187:0/174850491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a000044e0 0x7f5a00004960 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7f5a180bcaa0 tx=0x7f5a08007750).stop 2020-11-20T12:21:17.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.365+0000 7f5a1d95c700 1 -- 172.21.15.187:0/174850491 reap_dead start 2020-11-20T12:21:17.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.365+0000 7f5a1e15d700 1 -- 172.21.15.187:0/174850491 shutdown_connections 2020-11-20T12:21:17.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.365+0000 7f5a1e15d700 1 -- 172.21.15.187:0/174850491 >> 172.21.15.187:0/174850491 conn(0x7f5a18068370 msgr2=0x7f5a180bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:17.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.365+0000 7f5a1e15d700 1 -- 172.21.15.187:0/174850491 shutdown_connections 2020-11-20T12:21:17.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.365+0000 7f5a1e15d700 1 -- 172.21.15.187:0/174850491 wait complete. 2020-11-20T12:21:17.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.367+0000 7f5a1e15d700 1 Processor -- start 2020-11-20T12:21:17.333 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.367+0000 7f5a1e15d700 1 -- start start 2020-11-20T12:21:17.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1e15d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180b4050 0x7f5a180b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1e15d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a181474f0 0x7f5a1813e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1e15d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a1813eb70 0x7f5a1815bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.334 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1e15d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18149c50 con 0x7f5a181474f0 2020-11-20T12:21:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1e15d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18145e30 con 0x7f5a1813eb70 2020-11-20T12:21:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1e15d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181460d0 con 0x7f5a180b4050 2020-11-20T12:21:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1d95c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a1813eb70 0x7f5a1815bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1d95c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a1813eb70 0x7f5a1815bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54304/0 (socket says 172.21.15.187:54304) 2020-11-20T12:21:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1d95c700 1 -- 172.21.15.187:0/20487537 learned_addr learned my addr 172.21.15.187:0/20487537 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:17.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1c95a700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a181474f0 0x7f5a1813e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1d95c700 1 -- 172.21.15.187:0/20487537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180b4050 msgr2=0x7f5a180b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1d95c700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180b4050 0x7f5a180b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:17.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1d95c700 1 -- 172.21.15.187:0/20487537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a181474f0 msgr2=0x7f5a1813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.336 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1d95c700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a181474f0 0x7f5a1813e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1d95c700 1 -- 172.21.15.187:0/20487537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18143ef0 con 0x7f5a1813eb70 2020-11-20T12:21:17.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.368+0000 7f5a1c95a700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a181474f0 0x7f5a1813e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:17.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.369+0000 7f5a1d95c700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a1813eb70 0x7f5a1815bcf0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7f5a18044fd0 tx=0x7f5a10008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:17.337 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.369+0000 7f5a057fa700 1 -- 172.21.15.187:0/20487537 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000e9d0 con 0x7f5a1813eb70 2020-11-20T12:21:17.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.369+0000 7f5a057fa700 1 -- 172.21.15.187:0/20487537 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1000eb30 con 0x7f5a1813eb70 2020-11-20T12:21:17.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.369+0000 7f5a057fa700 1 -- 172.21.15.187:0/20487537 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10014430 con 0x7f5a1813eb70 2020-11-20T12:21:17.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.369+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a180b4920 con 0x7f5a1813eb70 2020-11-20T12:21:17.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.369+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a18133da0 con 0x7f5a1813eb70 2020-11-20T12:21:17.338 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.370+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18080050 con 0x7f5a1813eb70 2020-11-20T12:21:17.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.371+0000 7f5a057fa700 1 -- 172.21.15.187:0/20487537 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5a10014820 con 0x7f5a1813eb70 2020-11-20T12:21:17.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.371+0000 7f5a057fa700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59f0056d20 0x7f59f00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.371+0000 7f5a057fa700 1 -- 172.21.15.187:0/20487537 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f5a10070480 con 0x7f5a1813eb70 2020-11-20T12:21:17.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.372+0000 7f5a1d15b700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59f0056d20 0x7f59f00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.339 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.372+0000 7f5a1d15b700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59f0056d20 0x7f59f00591e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f5a14006fd0 tx=0x7f5a14006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:17.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.376+0000 7f5a057fa700 1 -- 172.21.15.187:0/20487537 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5a10045b00 con 0x7f5a1813eb70 2020-11-20T12:21:17.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:16.584+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5a18144080 con 0x7f5a1813eb70 2020-11-20T12:21:17.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.026+0000 7f5a057fa700 1 -- 172.21.15.187:0/20487537 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v155) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5a10049120 con 0x7f5a1813eb70 2020-11-20T12:21:17.340 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59f0056d20 msgr2=0x7f59f00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59f0056d20 0x7f59f00591e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f5a14006fd0 tx=0x7f5a14006df0).stop 2020-11-20T12:21:17.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a1813eb70 msgr2=0x7f5a1815bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a1813eb70 0x7f5a1815bcf0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7f5a18044fd0 tx=0x7f5a10008040).stop 2020-11-20T12:21:17.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 shutdown_connections 2020-11-20T12:21:17.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f59f0056d20 0x7f59f00591e0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a1813eb70 0x7f5a1815bcf0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a180b4050 0x7f5a180b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:17.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 --2- 172.21.15.187:0/20487537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a181474f0 0x7f5a1813e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.029+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 >> 172.21.15.187:0/20487537 conn(0x7f5a18068370 msgr2=0x7f5a1814c130 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:17.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.030+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 shutdown_connections 2020-11-20T12:21:17.343 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.030+0000 7f5a1e15d700 1 -- 172.21.15.187:0/20487537 wait complete. 2020-11-20T12:21:17.343 INFO:teuthology.orchestra.run.smithi187.stderr: removed all blocklist entries 2020-11-20T12:21:17.343 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2020-11-20T12:21:17.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.603+0000 7f9b1925b700 1 Processor -- start 2020-11-20T12:21:17.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.604+0000 7f9b1925b700 1 -- start start 2020-11-20T12:21:17.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.605+0000 7f9b1925b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9b14063dd0 0x7f9b140afca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.605+0000 7f9b1925b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b14145360 0x7f9b14061f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.605+0000 7f9b1925b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b140685f0 con 0x7f9b14148180 2020-11-20T12:21:17.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.605+0000 7f9b1925b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b141457e0 con 0x7f9b14063dd0 2020-11-20T12:21:17.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.605+0000 7f9b1925b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b14067e10 con 0x7f9b14145360 2020-11-20T12:21:17.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.606+0000 7f9b18a5a700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9b14148180 0x7f9b141485a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50918/0 (socket says 172.21.15.187:50918) 2020-11-20T12:21:17.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.606+0000 7f9b18a5a700 1 -- 172.21.15.187:0/2599525643 learned_addr learned my addr 172.21.15.187:0/2599525643 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:17.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.606+0000 7f9b0b7fe700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b14145360 0x7f9b14061f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.606+0000 7f9b13fff700 1 --2- 172.21.15.187:0/2599525643 >> v2:172.21.15.5:3300/0 conn(0x7f9b14063dd0 0x7f9b140afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.606+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948529836 0 0) 0x7f9b140685f0 con 0x7f9b14148180 2020-11-20T12:21:17.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af4003240 con 0x7f9b14148180 2020-11-20T12:21:17.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2430543979 0 0) 0x7f9af4003240 con 0x7f9b14148180 2020-11-20T12:21:17.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b14145360 msgr2=0x7f9b14061f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:17.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b14145360 0x7f9b14061f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 >> v2:172.21.15.5:3300/0 conn(0x7f9b14063dd0 msgr2=0x7f9b140afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:17.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 --2- 172.21.15.187:0/2599525643 >> v2:172.21.15.5:3300/0 conn(0x7f9b14063dd0 0x7f9b140afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b140680b0 con 0x7f9b14148180 2020-11-20T12:21:17.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f9b0c001860 con 0x7f9b14148180 2020-11-20T12:21:17.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 >> v1:172.21.15.187:6789/0 conn(0x7f9b14148180 legacy=0x7f9b141485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af40037b0 0x7f9af4003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9af40044e0 0x7f9af4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af4004e50 0x7f9af400d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b141457e0 con 0x7f9af40044e0 2020-11-20T12:21:17.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b14067e10 con 0x7f9af4004e50 2020-11-20T12:21:17.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.607+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40052d0 con 0x7f9af40037b0 2020-11-20T12:21:17.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.608+0000 7f9b18a5a700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af4004e50 0x7f9af400d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.608+0000 7f9b13fff700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af40037b0 0x7f9af4003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.608+0000 7f9b0b7fe700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9af40044e0 0x7f9af4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.609+0000 7f9b18a5a700 1 -- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af40037b0 msgr2=0x7f9af4003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:17.616 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.609+0000 7f9b18a5a700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af40037b0 0x7f9af4003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.609+0000 7f9b18a5a700 1 -- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9af40044e0 msgr2=0x7f9af4004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:17.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.609+0000 7f9b18a5a700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9af40044e0 0x7f9af4004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.609+0000 7f9b18a5a700 1 -- 172.21.15.187:0/2599525643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af4003240 con 0x7f9af4004e50 2020-11-20T12:21:17.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.609+0000 7f9b18a5a700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af4004e50 0x7f9af400d940 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7f9b0c00c600 tx=0x7f9b14147e70).ready entity=mon.1 client_cookie=f7954e0178172107 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:17.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.610+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c007a70 con 0x7f9af4004e50 2020-11-20T12:21:17.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.610+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0c005b80 con 0x7f9af4004e50 2020-11-20T12:21:17.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.610+0000 7f9b137fe700 1 -- 172.21.15.187:0/2599525643 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c0070d0 con 0x7f9af4004e50 2020-11-20T12:21:17.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.610+0000 7f9b1925b700 1 -- 172.21.15.187:0/2599525643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af4004e50 msgr2=0x7f9af400d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.610+0000 7f9b1925b700 1 --2- 172.21.15.187:0/2599525643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af4004e50 0x7f9af400d940 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7f9b0c00c600 tx=0x7f9b14147e70).stop 2020-11-20T12:21:17.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.610+0000 7f9b18a5a700 1 -- 172.21.15.187:0/2599525643 reap_dead start 2020-11-20T12:21:17.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.611+0000 7f9b1925b700 1 -- 172.21.15.187:0/2599525643 shutdown_connections 2020-11-20T12:21:17.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.611+0000 7f9b1925b700 1 -- 172.21.15.187:0/2599525643 >> 172.21.15.187:0/2599525643 conn(0x7f9b140bab30 msgr2=0x7f9b140baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:17.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.611+0000 7f9b1925b700 1 -- 172.21.15.187:0/2599525643 shutdown_connections 2020-11-20T12:21:17.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.611+0000 7f9b1925b700 1 -- 172.21.15.187:0/2599525643 wait complete. 2020-11-20T12:21:17.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.613+0000 7f9b1925b700 1 Processor -- start 2020-11-20T12:21:17.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.614+0000 7f9b1925b700 1 -- start start 2020-11-20T12:21:17.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b1925b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af4004e50 0x7f9b14154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b1925b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b140a0680 0x7f9b1409c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b1925b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b140b2ba0 0x7f9b14146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b1925b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b14141900 con 0x7f9b140b2ba0 2020-11-20T12:21:17.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b1925b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b14154730 con 0x7f9b140a0680 2020-11-20T12:21:17.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b1925b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b140b21c0 con 0x7f9af4004e50 2020-11-20T12:21:17.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b0b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b140a0680 0x7f9b1409c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b0b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b140a0680 0x7f9b1409c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54322/0 (socket says 172.21.15.187:54322) 2020-11-20T12:21:17.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.615+0000 7f9b0b7fe700 1 -- 172.21.15.187:0/3697123701 learned_addr learned my addr 172.21.15.187:0/3697123701 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:17.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b13fff700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af4004e50 0x7f9b14154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b18a5a700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b140b2ba0 0x7f9b14146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b0b7fe700 1 -- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af4004e50 msgr2=0x7f9b14154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b0b7fe700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af4004e50 0x7f9b14154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b0b7fe700 1 -- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b140b2ba0 msgr2=0x7f9b14146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b0b7fe700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b140b2ba0 0x7f9b14146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b0b7fe700 1 -- 172.21.15.187:0/3697123701 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b140b2460 con 0x7f9b140a0680 2020-11-20T12:21:17.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b13fff700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af4004e50 0x7f9b14154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:17.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b0b7fe700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b140a0680 0x7f9b1409c2c0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7f9afc010a00 tx=0x7f9afc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:17.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b10ff9700 1 -- 172.21.15.187:0/3697123701 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc00e930 con 0x7f9b140a0680 2020-11-20T12:21:17.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.616+0000 7f9b10ff9700 1 -- 172.21.15.187:0/3697123701 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc00ea90 con 0x7f9b140a0680 2020-11-20T12:21:17.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.617+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b140b2700 con 0x7f9b140a0680 2020-11-20T12:21:17.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.617+0000 7f9b10ff9700 1 -- 172.21.15.187:0/3697123701 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc00ee10 con 0x7f9b140a0680 2020-11-20T12:21:17.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.617+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b140b3020 con 0x7f9b140a0680 2020-11-20T12:21:17.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.618+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b14061a20 con 0x7f9b140a0680 2020-11-20T12:21:17.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.619+0000 7f9b10ff9700 1 -- 172.21.15.187:0/3697123701 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9afc024070 con 0x7f9b140a0680 2020-11-20T12:21:17.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.620+0000 7f9b10ff9700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9af0056d20 0x7f9af00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:17.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.620+0000 7f9b10ff9700 1 -- 172.21.15.187:0/3697123701 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f9afc0704b0 con 0x7f9b140a0680 2020-11-20T12:21:17.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.620+0000 7f9b13fff700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9af0056d20 0x7f9af00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:17.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.620+0000 7f9b13fff700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9af0056d20 0x7f9af00591e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f9b04006fd0 tx=0x7f9b04008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:17.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.624+0000 7f9b10ff9700 1 -- 172.21.15.187:0/3697123701 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9afc045bd0 con 0x7f9b140a0680 2020-11-20T12:21:17.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.875+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9b140446d0 con 0x7f9b140a0680 2020-11-20T12:21:17.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.876+0000 7f9b10ff9700 1 -- 172.21.15.187:0/3697123701 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+4570 (secure 0 0 0) 0x7f9afc018020 con 0x7f9b140a0680 2020-11-20T12:21:17.879 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:17.880 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:21:17.880 INFO:teuthology.orchestra.run.smithi187.stdout: "auth_dump": [ 2020-11-20T12:21:17.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.880 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.a", 2020-11-20T12:21:17.880 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfRb2ACRAAqmjgnYux7baz2MF/2XP98w==", 2020-11-20T12:21:17.881 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.881 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:21:17.881 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.881 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.882 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.882 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.882 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mds.b", 2020-11-20T12:21:17.883 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf/xjxGBAAGeDPS76JVdZemY7RmABEbQ==", 2020-11-20T12:21:17.883 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.883 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:21:17.883 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.883 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.884 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.884 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.884 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.0", 2020-11-20T12:21:17.885 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCssLdfI7ZAEBAAQWTbSTfn7AYBjac6xvPosA==", 2020-11-20T12:21:17.885 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.885 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.885 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.885 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.886 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.886 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.1", 2020-11-20T12:21:17.886 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCvsLdfRxvjGhAAtDFak4hmHxIxkaNQAxTDYQ==", 2020-11-20T12:21:17.887 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.887 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.887 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.887 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.888 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.888 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.2", 2020-11-20T12:21:17.888 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCysLdfnQbiIxAApYMIv7ZID6fy9agATQFd/A==", 2020-11-20T12:21:17.889 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.889 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.889 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.889 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.889 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.890 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.3", 2020-11-20T12:21:17.890 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC1sLdf6wxRLhAAY9RVDSuZOihB0PfQ2rp5zg==", 2020-11-20T12:21:17.890 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.891 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.891 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.891 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.891 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.892 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.4", 2020-11-20T12:21:17.892 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC6sLdfCIhdNRAADWkld7H54/mYFbeB7UqrPQ==", 2020-11-20T12:21:17.892 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.892 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.893 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.893 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.893 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.894 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.5", 2020-11-20T12:21:17.894 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQC+sLdf8q2vAxAA9B0AeeFv1SxqAwg4jyozjQ==", 2020-11-20T12:21:17.894 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.894 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.895 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.895 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.895 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.896 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.6", 2020-11-20T12:21:17.896 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDBsLdfS1RGDBAAvIJDDL1q09kJW83yn8RYRg==", 2020-11-20T12:21:17.896 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.896 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.896 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.897 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.897 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.897 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "osd.7", 2020-11-20T12:21:17.898 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQDEsLdfGCtaFhAA9hVul9F+vkmi+dhJe25fFQ==", 2020-11-20T12:21:17.898 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.898 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.898 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.898 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.899 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.899 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.0", 2020-11-20T12:21:17.899 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfScLFJRAA6lAACSuIlkmr2z4j0TEcRA==", 2020-11-20T12:21:17.900 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.900 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:21:17.900 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow r", 2020-11-20T12:21:17.900 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rw pool=cephfs_data" 2020-11-20T12:21:17.900 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.901 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.1", 2020-11-20T12:21:17.901 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdf78RIMBAAipbwy9QzThFn1rMQSx4rvw==", 2020-11-20T12:21:17.901 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.902 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:21:17.902 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:21:17.902 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:21:17.902 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:21:17.902 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.903 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.2", 2020-11-20T12:21:17.903 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCnsLdfk/R4ORAAdy3RjxKJR8aBM3J+83UDrw==", 2020-11-20T12:21:17.903 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.904 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:21:17.904 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:21:17.904 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:21:17.904 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:21:17.904 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.905 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.3", 2020-11-20T12:21:17.905 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCosLdf/xkcCBAACswo+L23lO+i/1y3DnwDeg==", 2020-11-20T12:21:17.905 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.906 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow", 2020-11-20T12:21:17.906 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow r", 2020-11-20T12:21:17.906 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow rw", 2020-11-20T12:21:17.906 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow rwx" 2020-11-20T12:21:17.906 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.907 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "client.admin", 2020-11-20T12:21:17.907 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfhKIcARAAWTLvwgBH1bmTn64BCug1Rw==", 2020-11-20T12:21:17.907 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.908 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:21:17.908 INFO:teuthology.orchestra.run.smithi187.stdout: "mgr": "allow *", 2020-11-20T12:21:17.908 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow *", 2020-11-20T12:21:17.908 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.908 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.909 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.x", 2020-11-20T12:21:17.909 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdfaonXOBAAEiFy8kIhL5+Cm/58LPM9wg==", 2020-11-20T12:21:17.909 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.910 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:21:17.910 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:21:17.910 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:17.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:17.911 INFO:teuthology.orchestra.run.smithi187.stdout: "entity": "mgr.y", 2020-11-20T12:21:17.911 INFO:teuthology.orchestra.run.smithi187.stdout: "key": "AQCmsLdf8OJgKhAA7B4TZXwJw+PeXIkevBdJTQ==", 2020-11-20T12:21:17.911 INFO:teuthology.orchestra.run.smithi187.stdout: "caps": { 2020-11-20T12:21:17.912 INFO:teuthology.orchestra.run.smithi187.stdout: "mds": "allow *", 2020-11-20T12:21:17.912 INFO:teuthology.orchestra.run.smithi187.stdout: "mon": "allow profile mgr", 2020-11-20T12:21:17.912 INFO:teuthology.orchestra.run.smithi187.stdout: "osd": "allow *" 2020-11-20T12:21:17.912 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.912 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:17.913 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:21:17.913 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:21:17.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.878+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9af0056d20 msgr2=0x7f9af00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.878+0000 7f9b1925b700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9af0056d20 0x7f9af00591e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f9b04006fd0 tx=0x7f9b04008040).stop 2020-11-20T12:21:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.878+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b140a0680 msgr2=0x7f9b1409c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.878+0000 7f9b1925b700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b140a0680 0x7f9b1409c2c0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7f9afc010a00 tx=0x7f9afc008040).stop 2020-11-20T12:21:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.879+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 shutdown_connections 2020-11-20T12:21:17.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.879+0000 7f9b1925b700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9af0056d20 0x7f9af00591e0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.879+0000 7f9b1925b700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b140b2ba0 0x7f9b14146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.879+0000 7f9b1925b700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9af4004e50 0x7f9b14154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.879+0000 7f9b1925b700 1 --2- 172.21.15.187:0/3697123701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b140a0680 0x7f9b1409c2c0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.879+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 >> 172.21.15.187:0/3697123701 conn(0x7f9b140bab30 msgr2=0x7f9b14144950 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:17.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.879+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 shutdown_connections 2020-11-20T12:21:17.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:17.879+0000 7f9b1925b700 1 -- 172.21.15.187:0/3697123701 wait complete. 2020-11-20T12:21:17.917 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:21:18.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.157+0000 7f60b2a3f700 1 Processor -- start 2020-11-20T12:21:18.160 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.158+0000 7f60b2a3f700 1 -- start start 2020-11-20T12:21:18.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.159+0000 7f60b2a3f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f60ac065f90 0x7f60ac065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.159+0000 7f60b2a3f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f60ac09ca90 0x7f60ac09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.159+0000 7f60b2a3f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ac0b13e0 con 0x7f60ac065190 2020-11-20T12:21:18.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.159+0000 7f60b2a3f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac14ab30 con 0x7f60ac09ca90 2020-11-20T12:21:18.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.159+0000 7f60b2a3f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac062c10 con 0x7f60ac065f90 2020-11-20T12:21:18.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.159+0000 7f60b123c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f60ac065f90 0x7f60ac065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b123c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f60ac065f90 0x7f60ac065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54330/0 (socket says 172.21.15.187:54330) 2020-11-20T12:21:18.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b123c700 1 -- 172.21.15.187:0/226679892 learned_addr learned my addr 172.21.15.187:0/226679892 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:18.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.159+0000 7f60b1a3d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f60ac065190 0x7f60ac143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50934/0 (socket says 172.21.15.187:50934) 2020-11-20T12:21:18.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b123c700 1 -- 172.21.15.187:0/226679892 >> v2:172.21.15.5:3300/0 conn(0x7f60ac09ca90 msgr2=0x7f60ac09cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:18.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b223e700 1 --2- 172.21.15.187:0/226679892 >> v2:172.21.15.5:3300/0 conn(0x7f60ac09ca90 0x7f60ac09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b123c700 1 --2- 172.21.15.187:0/226679892 >> v2:172.21.15.5:3300/0 conn(0x7f60ac09ca90 0x7f60ac09cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b123c700 1 -- 172.21.15.187:0/226679892 >> v1:172.21.15.187:6789/0 conn(0x7f60ac065190 legacy=0x7f60ac143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b123c700 1 -- 172.21.15.187:0/226679892 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac0b1640 con 0x7f60ac065f90 2020-11-20T12:21:18.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b223e700 1 --2- 172.21.15.187:0/226679892 >> v2:172.21.15.5:3300/0 conn(0x7f60ac09ca90 0x7f60ac09cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:18.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.160+0000 7f60b123c700 1 --2- 172.21.15.187:0/226679892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065f90 0x7f60ac065570 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7f60ac0b5fb0 tx=0x7f609c006eb0).ready entity=mon.1 client_cookie=2c762310ce18452c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:18.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.161+0000 7f60b0a3b700 1 -- 172.21.15.187:0/226679892 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c009960 con 0x7f60ac065f90 2020-11-20T12:21:18.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.161+0000 7f60b0a3b700 1 -- 172.21.15.187:0/226679892 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c009ac0 con 0x7f60ac065f90 2020-11-20T12:21:18.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.161+0000 7f60b0a3b700 1 -- 172.21.15.187:0/226679892 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c009e40 con 0x7f60ac065f90 2020-11-20T12:21:18.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.162+0000 7f60b2a3f700 1 -- 172.21.15.187:0/226679892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065f90 msgr2=0x7f60ac065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.162+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/226679892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065f90 0x7f60ac065570 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7f60ac0b5fb0 tx=0x7f609c006eb0).stop 2020-11-20T12:21:18.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.162+0000 7f60b2a3f700 1 -- 172.21.15.187:0/226679892 shutdown_connections 2020-11-20T12:21:18.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.162+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/226679892 >> v2:172.21.15.5:3300/0 conn(0x7f60ac09ca90 0x7f60ac09cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.162+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/226679892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065f90 0x7f60ac065570 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.162+0000 7f60b2a3f700 1 -- 172.21.15.187:0/226679892 >> 172.21.15.187:0/226679892 conn(0x7f60ac0bab30 msgr2=0x7f60ac0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:18.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.162+0000 7f60b2a3f700 1 -- 172.21.15.187:0/226679892 shutdown_connections 2020-11-20T12:21:18.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.162+0000 7f60b2a3f700 1 -- 172.21.15.187:0/226679892 wait complete. 2020-11-20T12:21:18.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.163+0000 7f60b2a3f700 1 Processor -- start 2020-11-20T12:21:18.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.164+0000 7f60b2a3f700 1 -- start start 2020-11-20T12:21:18.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.164+0000 7f60b2a3f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065190 0x7f60ac0b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.164+0000 7f60b2a3f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60ac065f90 0x7f60ac09fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.164+0000 7f60b2a3f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60ac09ca90 0x7f60ac067d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.171 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.164+0000 7f60b2a3f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac0b1f70 con 0x7f60ac09ca90 2020-11-20T12:21:18.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.164+0000 7f60b2a3f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac09f760 con 0x7f60ac065190 2020-11-20T12:21:18.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.164+0000 7f60b2a3f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac09f490 con 0x7f60ac065f90 2020-11-20T12:21:18.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065190 0x7f60ac0b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.172 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065190 0x7f60ac0b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54336/0 (socket says 172.21.15.187:54336) 2020-11-20T12:21:18.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 -- 172.21.15.187:0/2162357447 learned_addr learned my addr 172.21.15.187:0/2162357447 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:18.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 -- 172.21.15.187:0/2162357447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60ac065f90 msgr2=0x7f60ac09fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60ac065f90 0x7f60ac09fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:18.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 -- 172.21.15.187:0/2162357447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60ac09ca90 msgr2=0x7f60ac067d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.173 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60ac09ca90 0x7f60ac067d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:18.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 -- 172.21.15.187:0/2162357447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac0b44e0 con 0x7f60ac065190 2020-11-20T12:21:18.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.165+0000 7f60b1a3d700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065190 0x7f60ac0b3ff0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7f60a800ca90 tx=0x7f60a800ce50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:18.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.166+0000 7f6099ffb700 1 -- 172.21.15.187:0/2162357447 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8005ac0 con 0x7f60ac065190 2020-11-20T12:21:18.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.166+0000 7f6099ffb700 1 -- 172.21.15.187:0/2162357447 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a8005c20 con 0x7f60ac065190 2020-11-20T12:21:18.174 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.166+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ac0b4970 con 0x7f60ac065190 2020-11-20T12:21:18.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.166+0000 7f6099ffb700 1 -- 172.21.15.187:0/2162357447 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8007110 con 0x7f60ac065190 2020-11-20T12:21:18.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.166+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ac0b1d00 con 0x7f60ac065190 2020-11-20T12:21:18.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.167+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ac099ce0 con 0x7f60ac065190 2020-11-20T12:21:18.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.167+0000 7f6099ffb700 1 -- 172.21.15.187:0/2162357447 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f60a8010030 con 0x7f60ac065190 2020-11-20T12:21:18.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.168+0000 7f6099ffb700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6084056d20 0x7f60840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.168+0000 7f6099ffb700 1 -- 172.21.15.187:0/2162357447 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f60a806fcb0 con 0x7f60ac065190 2020-11-20T12:21:18.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.168+0000 7f60b123c700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6084056d20 0x7f60840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.169+0000 7f60b123c700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6084056d20 0x7f60840591e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f60ac0b5fb0 tx=0x7f609c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:18.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.172+0000 7f6099ffb700 1 -- 172.21.15.187:0/2162357447 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f60a8045320 con 0x7f60ac065190 2020-11-20T12:21:18.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.358+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60ac09f8f0 con 0x7f60ac065190 2020-11-20T12:21:18.362 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.360+0000 7f6099ffb700 1 -- 172.21.15.187:0/2162357447 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+13637 (secure 0 0 0) 0x7f60a8048940 con 0x7f60ac065190 2020-11-20T12:21:18.363 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:18.363 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":155,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:21:17.015970+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:21:18.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6084056d20 msgr2=0x7f60840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6084056d20 0x7f60840591e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f60ac0b5fb0 tx=0x7f609c015000).stop 2020-11-20T12:21:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065190 msgr2=0x7f60ac0b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065190 0x7f60ac0b3ff0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7f60a800ca90 tx=0x7f60a800ce50).stop 2020-11-20T12:21:18.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 shutdown_connections 2020-11-20T12:21:18.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6084056d20 0x7f60840591e0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60ac09ca90 0x7f60ac067d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:18.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60ac065190 0x7f60ac0b3ff0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 --2- 172.21.15.187:0/2162357447 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60ac065f90 0x7f60ac09fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:18.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 >> 172.21.15.187:0/2162357447 conn(0x7f60ac0bab30 msgr2=0x7f60ac09bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:18.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.362+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 shutdown_connections 2020-11-20T12:21:18.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.363+0000 7f60b2a3f700 1 -- 172.21.15.187:0/2162357447 wait complete. 2020-11-20T12:21:18.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:21:18.391 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:21:18.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.631+0000 7fdfaefb1700 1 Processor -- start 2020-11-20T12:21:18.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.631+0000 7fdfaefb1700 1 -- start start 2020-11-20T12:21:18.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.632+0000 7fdfaefb1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdfa8065190 0x7fdfa8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.632+0000 7fdfaefb1700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdfa8065f90 0x7fdfa8143780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.632+0000 7fdfaefb1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa80b13e0 con 0x7fdfa8143c70 2020-11-20T12:21:18.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.632+0000 7fdfaefb1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa814ab30 con 0x7fdfa8065190 2020-11-20T12:21:18.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.632+0000 7fdfaefb1700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa8062c10 con 0x7fdfa8065f90 2020-11-20T12:21:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.633+0000 7fdfad7ae700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdfa8065f90 0x7fdfa8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.637 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.633+0000 7fdfad7ae700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdfa8065f90 0x7fdfa8143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54344/0 (socket says 172.21.15.187:54344) 2020-11-20T12:21:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.633+0000 7fdfad7ae700 1 -- 172.21.15.187:0/1568006170 learned_addr learned my addr 172.21.15.187:0/1568006170 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.633+0000 7fdfadfaf700 1 --2- 172.21.15.187:0/1568006170 >> v2:172.21.15.5:3300/0 conn(0x7fdfa8065190 0x7fdfa8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.633+0000 7fdfad7ae700 1 -- 172.21.15.187:0/1568006170 >> v2:172.21.15.5:3300/0 conn(0x7fdfa8065190 msgr2=0x7fdfa8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:18.638 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.634+0000 7fdfad7ae700 1 --2- 172.21.15.187:0/1568006170 >> v2:172.21.15.5:3300/0 conn(0x7fdfa8065190 0x7fdfa8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.634+0000 7fdfad7ae700 1 -- 172.21.15.187:0/1568006170 >> v1:172.21.15.5:6789/0 conn(0x7fdfa8143c70 legacy=0x7fdfa809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.634+0000 7fdfad7ae700 1 -- 172.21.15.187:0/1568006170 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa80b1640 con 0x7fdfa8065f90 2020-11-20T12:21:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.634+0000 7fdfadfaf700 1 --2- 172.21.15.187:0/1568006170 >> v2:172.21.15.5:3300/0 conn(0x7fdfa8065190 0x7fdfa8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.634+0000 7fdfad7ae700 1 --2- 172.21.15.187:0/1568006170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa8143780 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7fdfa80b5fb0 tx=0x7fdfa4006eb0).ready entity=mon.1 client_cookie=3f754cc5de3247a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:18.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.634+0000 7fdfacfad700 1 -- 172.21.15.187:0/1568006170 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4009920 con 0x7fdfa8065f90 2020-11-20T12:21:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.634+0000 7fdfacfad700 1 -- 172.21.15.187:0/1568006170 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa4009a80 con 0x7fdfa8065f90 2020-11-20T12:21:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.634+0000 7fdfacfad700 1 -- 172.21.15.187:0/1568006170 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4009e00 con 0x7fdfa8065f90 2020-11-20T12:21:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.635+0000 7fdfaefb1700 1 -- 172.21.15.187:0/1568006170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 msgr2=0x7fdfa8143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.635+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/1568006170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa8143780 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7fdfa80b5fb0 tx=0x7fdfa4006eb0).stop 2020-11-20T12:21:18.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.635+0000 7fdfaefb1700 1 -- 172.21.15.187:0/1568006170 shutdown_connections 2020-11-20T12:21:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.635+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/1568006170 >> v2:172.21.15.5:3300/0 conn(0x7fdfa8065190 0x7fdfa8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.635+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/1568006170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa8143780 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.635+0000 7fdfaefb1700 1 -- 172.21.15.187:0/1568006170 >> 172.21.15.187:0/1568006170 conn(0x7fdfa80bab30 msgr2=0x7fdfa80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.635+0000 7fdfaefb1700 1 -- 172.21.15.187:0/1568006170 shutdown_connections 2020-11-20T12:21:18.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.635+0000 7fdfaefb1700 1 -- 172.21.15.187:0/1568006170 wait complete. 2020-11-20T12:21:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.637+0000 7fdfaefb1700 1 Processor -- start 2020-11-20T12:21:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.637+0000 7fdfaefb1700 1 -- start start 2020-11-20T12:21:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.638+0000 7fdfaefb1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdfa8065190 0x7fdfa809e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.638+0000 7fdfaefb1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa80993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfaefb1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdfa8143c70 0x7fdfa815b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfaefb1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa815c310 con 0x7fdfa8065190 2020-11-20T12:21:18.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfaefb1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa815c5b0 con 0x7fdfa8065f90 2020-11-20T12:21:18.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfaefb1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80a0960 con 0x7fdfa8143c70 2020-11-20T12:21:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfad7ae700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa80993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfad7ae700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa80993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54350/0 (socket says 172.21.15.187:54350) 2020-11-20T12:21:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfad7ae700 1 -- 172.21.15.187:0/795861095 learned_addr learned my addr 172.21.15.187:0/795861095 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfad7ae700 1 -- 172.21.15.187:0/795861095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdfa8143c70 msgr2=0x7fdfa815b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfad7ae700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdfa8143c70 0x7fdfa815b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfadfaf700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdfa8065190 0x7fdfa809e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfad7ae700 1 -- 172.21.15.187:0/795861095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdfa8065190 msgr2=0x7fdfa809e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfad7ae700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdfa8065190 0x7fdfa809e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.639+0000 7fdfad7ae700 1 -- 172.21.15.187:0/795861095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa80998d0 con 0x7fdfa8065f90 2020-11-20T12:21:18.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.640+0000 7fdfad7ae700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa80993e0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 rx=0x7fdfa4006ef0 tx=0x7fdfa4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.640+0000 7fdf99ffb700 1 -- 172.21.15.187:0/795861095 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4011e20 con 0x7fdfa8065f90 2020-11-20T12:21:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.640+0000 7fdf99ffb700 1 -- 172.21.15.187:0/795861095 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa401d5d0 con 0x7fdfa8065f90 2020-11-20T12:21:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.640+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa8066d20 con 0x7fdfa8065f90 2020-11-20T12:21:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.640+0000 7fdf99ffb700 1 -- 172.21.15.187:0/795861095 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa401d950 con 0x7fdfa8065f90 2020-11-20T12:21:18.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.640+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa815bf90 con 0x7fdfa8065f90 2020-11-20T12:21:18.649 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.641+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa815c180 con 0x7fdfa8065f90 2020-11-20T12:21:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.642+0000 7fdf99ffb700 1 -- 172.21.15.187:0/795861095 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdfa4025030 con 0x7fdfa8065f90 2020-11-20T12:21:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.643+0000 7fdf99ffb700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf80056d20 0x7fdf800591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.643+0000 7fdf99ffb700 1 -- 172.21.15.187:0/795861095 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fdfa40717b0 con 0x7fdfa8065f90 2020-11-20T12:21:18.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.643+0000 7fdfadfaf700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf80056d20 0x7fdf800591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:18.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.644+0000 7fdfadfaf700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf80056d20 0x7fdf800591e0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7fdf9c00b7b0 tx=0x7fdf9c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:18.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.647+0000 7fdf99ffb700 1 -- 172.21.15.187:0/795861095 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdfa4046e20 con 0x7fdfa8065f90 2020-11-20T12:21:18.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.835+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fdfa815c6f0 con 0x7fdfa8065f90 2020-11-20T12:21:18.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.836+0000 7fdf99ffb700 1 -- 172.21.15.187:0/795861095 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v155) v1 ==== 110+0+10 (secure 0 0 0) 0x7fdfa4023070 con 0x7fdfa8065f90 2020-11-20T12:21:18.838 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:21:18.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf80056d20 msgr2=0x7fdf800591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf80056d20 0x7fdf800591e0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7fdf9c00b7b0 tx=0x7fdf9c008040).stop 2020-11-20T12:21:18.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 msgr2=0x7fdfa80993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:18.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa80993e0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 rx=0x7fdfa4006ef0 tx=0x7fdfa4015000).stop 2020-11-20T12:21:18.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 shutdown_connections 2020-11-20T12:21:18.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fdf80056d20 0x7fdf800591e0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdfa8065190 0x7fdfa809e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdfa8143c70 0x7fdfa815b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:18.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 --2- 172.21.15.187:0/795861095 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa8065f90 0x7fdfa80993e0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:18.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 >> 172.21.15.187:0/795861095 conn(0x7fdfa80bab30 msgr2=0x7fdfa809ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:18.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 shutdown_connections 2020-11-20T12:21:18.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:18.838+0000 7fdfaefb1700 1 -- 172.21.15.187:0/795861095 wait complete. 2020-11-20T12:21:18.862 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:21:19.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.101+0000 7f8d7091b700 1 Processor -- start 2020-11-20T12:21:19.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.101+0000 7f8d7091b700 1 -- start start 2020-11-20T12:21:19.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d7091b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8d6c0bfbe0 0x7f8d6c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d7091b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8d6c09f560 0x7f8d6c1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d7091b700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6c14d110 con 0x7f8d6c0bf800 2020-11-20T12:21:19.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d7091b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6c14c9d0 con 0x7f8d6c0bfbe0 2020-11-20T12:21:19.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d7091b700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6c148aa0 con 0x7f8d6c09f560 2020-11-20T12:21:19.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d6ad9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8d6c0bf800 0x7f8d6c14c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50962/0 (socket says 172.21.15.187:50962) 2020-11-20T12:21:19.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d6ad9d700 1 -- 172.21.15.187:0/941701241 learned_addr learned my addr 172.21.15.187:0/941701241 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:19.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d6b59e700 1 --2- 172.21.15.187:0/941701241 >> v2:172.21.15.187:3300/0 conn(0x7f8d6c09f560 0x7f8d6c1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d69d9b700 1 -- 172.21.15.187:0/941701241 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199257524 0 0) 0x7f8d6c14d110 con 0x7f8d6c0bf800 2020-11-20T12:21:19.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.103+0000 7f8d6a59c700 1 --2- 172.21.15.187:0/941701241 >> v2:172.21.15.5:3300/0 conn(0x7f8d6c0bfbe0 0x7f8d6c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.104+0000 7f8d69d9b700 1 -- 172.21.15.187:0/941701241 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d58003240 con 0x7f8d6c0bf800 2020-11-20T12:21:19.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.104+0000 7f8d6b59e700 1 -- 172.21.15.187:0/941701241 >> v2:172.21.15.5:3300/0 conn(0x7f8d6c0bfbe0 msgr2=0x7f8d6c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:19.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.104+0000 7f8d6b59e700 1 --2- 172.21.15.187:0/941701241 >> v2:172.21.15.5:3300/0 conn(0x7f8d6c0bfbe0 0x7f8d6c1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.104+0000 7f8d69d9b700 1 -- 172.21.15.187:0/941701241 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2245712808 0 0) 0x7f8d58003240 con 0x7f8d6c0bf800 2020-11-20T12:21:19.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.104+0000 7f8d6b59e700 1 -- 172.21.15.187:0/941701241 >> v1:172.21.15.187:6789/0 conn(0x7f8d6c0bf800 legacy=0x7f8d6c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.104+0000 7f8d6b59e700 1 -- 172.21.15.187:0/941701241 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c14cbd0 con 0x7f8d6c09f560 2020-11-20T12:21:19.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.104+0000 7f8d6b59e700 1 --2- 172.21.15.187:0/941701241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c09f560 0x7f8d6c1485b0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7f8d6c09f2b0 tx=0x7f8d640091d0).ready entity=mon.1 client_cookie=e60d801de87a20c9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:19.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d69d9b700 1 -- 172.21.15.187:0/941701241 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c0014e0 con 0x7f8d6c09f560 2020-11-20T12:21:19.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d69d9b700 1 -- 172.21.15.187:0/941701241 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6400b7c0 con 0x7f8d6c09f560 2020-11-20T12:21:19.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d69d9b700 1 -- 172.21.15.187:0/941701241 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6400bb40 con 0x7f8d6c09f560 2020-11-20T12:21:19.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d7091b700 1 -- 172.21.15.187:0/941701241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c09f560 msgr2=0x7f8d6c1485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d7091b700 1 --2- 172.21.15.187:0/941701241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c09f560 0x7f8d6c1485b0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7f8d6c09f2b0 tx=0x7f8d640091d0).stop 2020-11-20T12:21:19.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d7091b700 1 -- 172.21.15.187:0/941701241 shutdown_connections 2020-11-20T12:21:19.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d7091b700 1 --2- 172.21.15.187:0/941701241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c09f560 0x7f8d6c1485b0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d7091b700 1 --2- 172.21.15.187:0/941701241 >> v2:172.21.15.5:3300/0 conn(0x7f8d6c0bfbe0 0x7f8d6c1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d7091b700 1 -- 172.21.15.187:0/941701241 >> 172.21.15.187:0/941701241 conn(0x7f8d6c068370 msgr2=0x7f8d6c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:19.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d7091b700 1 -- 172.21.15.187:0/941701241 shutdown_connections 2020-11-20T12:21:19.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.105+0000 7f8d7091b700 1 -- 172.21.15.187:0/941701241 wait complete. 2020-11-20T12:21:19.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.107+0000 7f8d7091b700 1 Processor -- start 2020-11-20T12:21:19.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.107+0000 7f8d7091b700 1 -- start start 2020-11-20T12:21:19.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.108+0000 7f8d7091b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d6c09f560 0x7f8d6c0b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.108+0000 7f8d7091b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d6c0bf800 0x7f8d6c138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.108+0000 7f8d7091b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c145c50 0x7f8d6c136300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.108+0000 7f8d7091b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c138550 con 0x7f8d6c0bf800 2020-11-20T12:21:19.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.108+0000 7f8d7091b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c0b68b0 con 0x7f8d6c145c50 2020-11-20T12:21:19.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.108+0000 7f8d7091b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c1367f0 con 0x7f8d6c09f560 2020-11-20T12:21:19.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c145c50 0x7f8d6c136300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c145c50 0x7f8d6c136300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54366/0 (socket says 172.21.15.187:54366) 2020-11-20T12:21:19.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 -- 172.21.15.187:0/704439888 learned_addr learned my addr 172.21.15.187:0/704439888 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:19.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6ad9d700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d6c09f560 0x7f8d6c0b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 -- 172.21.15.187:0/704439888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d6c09f560 msgr2=0x7f8d6c0b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d6c09f560 0x7f8d6c0b5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 -- 172.21.15.187:0/704439888 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d6c0bf800 msgr2=0x7f8d6c138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d6c0bf800 0x7f8d6c138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:19.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 -- 172.21.15.187:0/704439888 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c1461d0 con 0x7f8d6c145c50 2020-11-20T12:21:19.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.109+0000 7f8d6b59e700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c145c50 0x7f8d6c136300 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7f8d6c148be0 tx=0x7f8d64016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:19.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.110+0000 7f8d52ffd700 1 -- 172.21.15.187:0/704439888 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6401d770 con 0x7f8d6c145c50 2020-11-20T12:21:19.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.110+0000 7f8d52ffd700 1 -- 172.21.15.187:0/704439888 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6401d8d0 con 0x7f8d6c145c50 2020-11-20T12:21:19.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.110+0000 7f8d52ffd700 1 -- 172.21.15.187:0/704439888 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6401dc70 con 0x7f8d6c145c50 2020-11-20T12:21:19.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.110+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6c146470 con 0x7f8d6c145c50 2020-11-20T12:21:19.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.110+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6c14beb0 con 0x7f8d6c145c50 2020-11-20T12:21:19.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.111+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6c0bef10 con 0x7f8d6c145c50 2020-11-20T12:21:19.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.112+0000 7f8d52ffd700 1 -- 172.21.15.187:0/704439888 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8d64017040 con 0x7f8d6c145c50 2020-11-20T12:21:19.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.112+0000 7f8d52ffd700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d48056d20 0x7f8d480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.112+0000 7f8d52ffd700 1 -- 172.21.15.187:0/704439888 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8d640721a0 con 0x7f8d6c145c50 2020-11-20T12:21:19.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.113+0000 7f8d6ad9d700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d48056d20 0x7f8d480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.113+0000 7f8d6ad9d700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d48056d20 0x7f8d480591e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f8d5c001e90 tx=0x7f8d5c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:19.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.116+0000 7f8d52ffd700 1 -- 172.21.15.187:0/704439888 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8d640479e0 con 0x7f8d6c145c50 2020-11-20T12:21:19.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.300+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8d6c080050 con 0x7f8d6c145c50 2020-11-20T12:21:19.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.301+0000 7f8d52ffd700 1 -- 172.21.15.187:0/704439888 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v155) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8d64015020 con 0x7f8d6c145c50 2020-11-20T12:21:19.303 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:19.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d48056d20 msgr2=0x7f8d480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d48056d20 0x7f8d480591e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f8d5c001e90 tx=0x7f8d5c009040).stop 2020-11-20T12:21:19.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c145c50 msgr2=0x7f8d6c136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c145c50 0x7f8d6c136300 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7f8d6c148be0 tx=0x7f8d64016000).stop 2020-11-20T12:21:19.306 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 shutdown_connections 2020-11-20T12:21:19.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8d48056d20 0x7f8d480591e0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6c145c50 0x7f8d6c136300 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8d6c09f560 0x7f8d6c0b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 --2- 172.21.15.187:0/704439888 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8d6c0bf800 0x7f8d6c138060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:19.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 >> 172.21.15.187:0/704439888 conn(0x7f8d6c068370 msgr2=0x7f8d6c141b90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:19.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.303+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 shutdown_connections 2020-11-20T12:21:19.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.304+0000 7f8d7091b700 1 -- 172.21.15.187:0/704439888 wait complete. 2020-11-20T12:21:19.328 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:21:19.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.562+0000 7f4957fff700 1 Processor -- start 2020-11-20T12:21:19.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.562+0000 7f4957fff700 1 -- start start 2020-11-20T12:21:19.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.563+0000 7f4957fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f49580bfbe0 0x7f49581480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.563+0000 7f4957fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f495809f560 0x7f49581485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.563+0000 7f4957fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495814d110 con 0x7f49580bf800 2020-11-20T12:21:19.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.563+0000 7f4957fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495814c9d0 con 0x7f49580bfbe0 2020-11-20T12:21:19.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.563+0000 7f4957fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4958148aa0 con 0x7f495809f560 2020-11-20T12:21:19.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.564+0000 7f49577fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f495809f560 0x7f49581485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.564+0000 7f49577fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f495809f560 0x7f49581485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54374/0 (socket says 172.21.15.187:54374) 2020-11-20T12:21:19.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.564+0000 7f49577fe700 1 -- 172.21.15.187:0/3554052466 learned_addr learned my addr 172.21.15.187:0/3554052466 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:19.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.564+0000 7f494e7fc700 1 --2- 172.21.15.187:0/3554052466 >> v2:172.21.15.5:3300/0 conn(0x7f49580bfbe0 0x7f49581480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.565+0000 7f49577fe700 1 -- 172.21.15.187:0/3554052466 >> v2:172.21.15.5:3300/0 conn(0x7f49580bfbe0 msgr2=0x7f49581480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:19.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.565+0000 7f49577fe700 1 --2- 172.21.15.187:0/3554052466 >> v2:172.21.15.5:3300/0 conn(0x7f49580bfbe0 0x7f49581480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.565+0000 7f49577fe700 1 -- 172.21.15.187:0/3554052466 >> v1:172.21.15.5:6789/0 conn(0x7f49580bf800 legacy=0x7f495814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:19.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.565+0000 7f49577fe700 1 -- 172.21.15.187:0/3554052466 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495814cbd0 con 0x7f495809f560 2020-11-20T12:21:19.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.565+0000 7f49577fe700 1 --2- 172.21.15.187:0/3554052466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f495809f560 0x7f49581485b0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7f495809f2b0 tx=0x7f4950009290).ready entity=mon.1 client_cookie=edb38bc996724ab4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:19.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.565+0000 7f49567fc700 1 -- 172.21.15.187:0/3554052466 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495000bc50 con 0x7f495809f560 2020-11-20T12:21:19.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.565+0000 7f49567fc700 1 -- 172.21.15.187:0/3554052466 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495000bdb0 con 0x7f495809f560 2020-11-20T12:21:19.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.565+0000 7f49567fc700 1 -- 172.21.15.187:0/3554052466 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950011610 con 0x7f495809f560 2020-11-20T12:21:19.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.566+0000 7f4957fff700 1 -- 172.21.15.187:0/3554052466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f495809f560 msgr2=0x7f49581485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.566+0000 7f4957fff700 1 --2- 172.21.15.187:0/3554052466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f495809f560 0x7f49581485b0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7f495809f2b0 tx=0x7f4950009290).stop 2020-11-20T12:21:19.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.566+0000 7f4957fff700 1 -- 172.21.15.187:0/3554052466 shutdown_connections 2020-11-20T12:21:19.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.566+0000 7f4957fff700 1 --2- 172.21.15.187:0/3554052466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f495809f560 0x7f49581485b0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.566+0000 7f4957fff700 1 --2- 172.21.15.187:0/3554052466 >> v2:172.21.15.5:3300/0 conn(0x7f49580bfbe0 0x7f49581480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.566+0000 7f4957fff700 1 -- 172.21.15.187:0/3554052466 >> 172.21.15.187:0/3554052466 conn(0x7f4958068370 msgr2=0x7f49580bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:19.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.566+0000 7f4957fff700 1 -- 172.21.15.187:0/3554052466 shutdown_connections 2020-11-20T12:21:19.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.566+0000 7f4957fff700 1 -- 172.21.15.187:0/3554052466 wait complete. 2020-11-20T12:21:19.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.568+0000 7f4957fff700 1 Processor -- start 2020-11-20T12:21:19.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.568+0000 7f4957fff700 1 -- start start 2020-11-20T12:21:19.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f4957fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f495809f560 0x7f495814bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f4957fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49580bf800 0x7f49581582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f4957fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f495815b9d0 0x7f495815c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f4957fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4958158bc0 con 0x7f495809f560 2020-11-20T12:21:19.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f4957fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4958158e60 con 0x7f49580bf800 2020-11-20T12:21:19.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f4957fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4958159100 con 0x7f495815b9d0 2020-11-20T12:21:19.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f494e7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49580bf800 0x7f49581582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f494e7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49580bf800 0x7f49581582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54378/0 (socket says 172.21.15.187:54378) 2020-11-20T12:21:19.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f494e7fc700 1 -- 172.21.15.187:0/817643918 learned_addr learned my addr 172.21.15.187:0/817643918 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:19.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f494e7fc700 1 -- 172.21.15.187:0/817643918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f495815b9d0 msgr2=0x7f495815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f494e7fc700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f495815b9d0 0x7f495815c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:19.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f494e7fc700 1 -- 172.21.15.187:0/817643918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f495809f560 msgr2=0x7f495814bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.570+0000 7f4956ffd700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f495809f560 0x7f495814bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f494e7fc700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f495809f560 0x7f495814bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f494e7fc700 1 -- 172.21.15.187:0/817643918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495814c2a0 con 0x7f49580bf800 2020-11-20T12:21:19.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f4956ffd700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f495809f560 0x7f495814bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:19.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f494e7fc700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49580bf800 0x7f49581582e0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f494000d8e0 tx=0x7f4940008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:19.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f494f7fe700 1 -- 172.21.15.187:0/817643918 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940009820 con 0x7f49580bf800 2020-11-20T12:21:19.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f494f7fe700 1 -- 172.21.15.187:0/817643918 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4940009980 con 0x7f49580bf800 2020-11-20T12:21:19.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49581587d0 con 0x7f49580bf800 2020-11-20T12:21:19.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f494f7fe700 1 -- 172.21.15.187:0/817643918 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940009d20 con 0x7f49580bf800 2020-11-20T12:21:19.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.571+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495815c4f0 con 0x7f49580bf800 2020-11-20T12:21:19.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.572+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4958080050 con 0x7f49580bf800 2020-11-20T12:21:19.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.573+0000 7f494f7fe700 1 -- 172.21.15.187:0/817643918 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4940013690 con 0x7f49580bf800 2020-11-20T12:21:19.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.574+0000 7f494f7fe700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4934056d20 0x7f49340591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:19.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.574+0000 7f494f7fe700 1 -- 172.21.15.187:0/817643918 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f494006fc40 con 0x7f49580bf800 2020-11-20T12:21:19.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.574+0000 7f4956ffd700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4934056d20 0x7f49340591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:19.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.575+0000 7f4956ffd700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4934056d20 0x7f49340591e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f49480014d0 tx=0x7f4948009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:19.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.578+0000 7f494f7fe700 1 -- 172.21.15.187:0/817643918 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4940045360 con 0x7f49580bf800 2020-11-20T12:21:19.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.764+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f49580a4cc0 con 0x7f49580bf800 2020-11-20T12:21:19.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.765+0000 7f494f7fe700 1 -- 172.21.15.187:0/817643918 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v155) v1 ==== 84+0+143 (secure 0 0 0) 0x7f4940048980 con 0x7f49580bf800 2020-11-20T12:21:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:19.768 INFO:teuthology.orchestra.run.smithi187.stdout:[ 2020-11-20T12:21:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 1, 2020-11-20T12:21:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "device_health_metrics" 2020-11-20T12:21:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:21:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:21:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "poolnum": 2, 2020-11-20T12:21:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: "poolname": "rbd" 2020-11-20T12:21:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:21:19.770 INFO:teuthology.orchestra.run.smithi187.stdout:] 2020-11-20T12:21:19.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4934056d20 msgr2=0x7f49340591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4934056d20 0x7f49340591e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f49480014d0 tx=0x7f4948009040).stop 2020-11-20T12:21:19.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49580bf800 msgr2=0x7f49581582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:19.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49580bf800 0x7f49581582e0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f494000d8e0 tx=0x7f4940008040).stop 2020-11-20T12:21:19.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 shutdown_connections 2020-11-20T12:21:19.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4934056d20 0x7f49340591e0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f495809f560 0x7f495814bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f495815b9d0 0x7f495815c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:19.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 --2- 172.21.15.187:0/817643918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49580bf800 0x7f49581582e0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:19.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 >> 172.21.15.187:0/817643918 conn(0x7f4958068370 msgr2=0x7f49581419d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:19.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 shutdown_connections 2020-11-20T12:21:19.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:19.768+0000 7f4957fff700 1 -- 172.21.15.187:0/817643918 wait complete. 2020-11-20T12:21:19.792 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:21:19.793 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:21:20.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.030+0000 7fe2224b6700 1 Processor -- start 2020-11-20T12:21:20.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.030+0000 7fe2224b6700 1 -- start start 2020-11-20T12:21:20.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.031+0000 7fe2224b6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1484d0 0x7fe21c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.031+0000 7fe2224b6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe21c066f70 0x7fe21c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.031+0000 7fe2224b6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c0b1300 con 0x7fe21c0622c0 2020-11-20T12:21:20.035 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.031+0000 7fe2224b6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c067730 con 0x7fe21c066f70 2020-11-20T12:21:20.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.031+0000 7fe2224b6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b15a0 con 0x7fe21c1484d0 2020-11-20T12:21:20.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.032+0000 7fe220cb3700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe21c0622c0 0x7fe21c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:50994/0 (socket says 172.21.15.187:50994) 2020-11-20T12:21:20.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.032+0000 7fe220cb3700 1 -- 172.21.15.187:0/4205428149 learned_addr learned my addr 172.21.15.187:0/4205428149 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:20.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.032+0000 7fe2214b4700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1484d0 0x7fe21c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:20.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.032+0000 7fe221cb5700 1 --2- 172.21.15.187:0/4205428149 >> v2:172.21.15.5:3300/0 conn(0x7fe21c066f70 0x7fe21c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:20.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.032+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623753782 0 0) 0x7fe21c0b1300 con 0x7fe21c0622c0 2020-11-20T12:21:20.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.032+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe204003240 con 0x7fe21c0622c0 2020-11-20T12:21:20.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1680124380 0 0) 0x7fe204003240 con 0x7fe21c0622c0 2020-11-20T12:21:20.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1484d0 msgr2=0x7fe21c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:20.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1484d0 0x7fe21c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:20.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 >> v2:172.21.15.5:3300/0 conn(0x7fe21c066f70 msgr2=0x7fe21c0b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:20.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 --2- 172.21.15.187:0/4205428149 >> v2:172.21.15.5:3300/0 conn(0x7fe21c066f70 0x7fe21c0b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:20.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c0b1840 con 0x7fe21c0622c0 2020-11-20T12:21:20.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fe20c0014e0 con 0x7fe21c0622c0 2020-11-20T12:21:20.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 >> v1:172.21.15.187:6789/0 conn(0x7fe21c0622c0 legacy=0x7fe21c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:20.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2040037b0 0x7fe204003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1484d0 0x7fe2040044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2040049d0 0x7fe204004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c067730 con 0x7fe2040037b0 2020-11-20T12:21:20.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b15a0 con 0x7fe2040049d0 2020-11-20T12:21:20.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.033+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040042a0 con 0x7fe21c1484d0 2020-11-20T12:21:20.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.034+0000 7fe221cb5700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2040049d0 0x7fe204004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:20.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.034+0000 7fe221cb5700 1 -- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1484d0 msgr2=0x7fe2040044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:20.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.034+0000 7fe221cb5700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1484d0 0x7fe2040044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:20.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.034+0000 7fe221cb5700 1 -- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2040037b0 msgr2=0x7fe204003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:20.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.034+0000 7fe221cb5700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2040037b0 0x7fe204003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:20.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.034+0000 7fe221cb5700 1 -- 172.21.15.187:0/4205428149 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204003240 con 0x7fe2040049d0 2020-11-20T12:21:20.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.034+0000 7fe221cb5700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2040049d0 0x7fe204004e50 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7fe21400bc00 tx=0x7fe214006df0).ready entity=mon.1 client_cookie=63e49ea2dd44b12e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:20.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.035+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21400b310 con 0x7fe2040049d0 2020-11-20T12:21:20.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.035+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21400eb40 con 0x7fe2040049d0 2020-11-20T12:21:20.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.035+0000 7fe20bfff700 1 -- 172.21.15.187:0/4205428149 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21400eec0 con 0x7fe2040049d0 2020-11-20T12:21:20.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.035+0000 7fe2224b6700 1 -- 172.21.15.187:0/4205428149 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2040049d0 msgr2=0x7fe204004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:20.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.035+0000 7fe2224b6700 1 --2- 172.21.15.187:0/4205428149 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2040049d0 0x7fe204004e50 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7fe21400bc00 tx=0x7fe214006df0).stop 2020-11-20T12:21:20.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.035+0000 7fe221cb5700 1 -- 172.21.15.187:0/4205428149 reap_dead start 2020-11-20T12:21:20.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.036+0000 7fe2224b6700 1 -- 172.21.15.187:0/4205428149 shutdown_connections 2020-11-20T12:21:20.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.036+0000 7fe2224b6700 1 -- 172.21.15.187:0/4205428149 >> 172.21.15.187:0/4205428149 conn(0x7fe21c0bab30 msgr2=0x7fe21c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:20.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.036+0000 7fe2224b6700 1 -- 172.21.15.187:0/4205428149 shutdown_connections 2020-11-20T12:21:20.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.036+0000 7fe2224b6700 1 -- 172.21.15.187:0/4205428149 wait complete. 2020-11-20T12:21:20.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.037+0000 7fe2224b6700 1 Processor -- start 2020-11-20T12:21:20.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.037+0000 7fe2224b6700 1 -- start start 2020-11-20T12:21:20.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.038+0000 7fe2224b6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe21c09f4e0 0x7fe21c09afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.038+0000 7fe2224b6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe21c09f8c0 0x7fe21c136eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.038+0000 7fe2224b6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1373a0 0x7fe21c15b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.038+0000 7fe2224b6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c137800 con 0x7fe21c09f8c0 2020-11-20T12:21:20.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.038+0000 7fe2224b6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c137bc0 con 0x7fe21c09f4e0 2020-11-20T12:21:20.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.038+0000 7fe2224b6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c137e60 con 0x7fe21c1373a0 2020-11-20T12:21:20.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.038+0000 7fe2214b4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe21c09f4e0 0x7fe21c09afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:20.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2214b4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe21c09f4e0 0x7fe21c09afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54398/0 (socket says 172.21.15.187:54398) 2020-11-20T12:21:20.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2214b4700 1 -- 172.21.15.187:0/4181222415 learned_addr learned my addr 172.21.15.187:0/4181222415 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:20.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2214b4700 1 -- 172.21.15.187:0/4181222415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1373a0 msgr2=0x7fe21c15b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:20.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2214b4700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1373a0 0x7fe21c15b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:20.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2214b4700 1 -- 172.21.15.187:0/4181222415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe21c09f8c0 msgr2=0x7fe21c136eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:20.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2214b4700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe21c09f8c0 0x7fe21c136eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:20.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2214b4700 1 -- 172.21.15.187:0/4181222415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c138100 con 0x7fe21c09f4e0 2020-11-20T12:21:20.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2214b4700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe21c09f4e0 0x7fe21c09afd0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7fe21800ecf0 tx=0x7fe2180091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:20.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2097fa700 1 -- 172.21.15.187:0/4181222415 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218016070 con 0x7fe21c09f4e0 2020-11-20T12:21:20.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2097fa700 1 -- 172.21.15.187:0/4181222415 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21800cbb0 con 0x7fe21c09f4e0 2020-11-20T12:21:20.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.039+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c09b4c0 con 0x7fe21c09f4e0 2020-11-20T12:21:20.052 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.040+0000 7fe2097fa700 1 -- 172.21.15.187:0/4181222415 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2180123f0 con 0x7fe21c09f4e0 2020-11-20T12:21:20.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.040+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c15ba10 con 0x7fe21c09f4e0 2020-11-20T12:21:20.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.041+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c0b4c70 con 0x7fe21c09f4e0 2020-11-20T12:21:20.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.041+0000 7fe2097fa700 1 -- 172.21.15.187:0/4181222415 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe2180127a0 con 0x7fe21c09f4e0 2020-11-20T12:21:20.053 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.042+0000 7fe2097fa700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1f4056d20 0x7fe1f40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:20.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.042+0000 7fe2097fa700 1 -- 172.21.15.187:0/4181222415 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe218070230 con 0x7fe21c09f4e0 2020-11-20T12:21:20.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.042+0000 7fe220cb3700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1f4056d20 0x7fe1f40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:20.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.043+0000 7fe220cb3700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1f4056d20 0x7fe1f40591e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7fe20c0013b0 tx=0x7fe20c0011f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:20.054 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.046+0000 7fe2097fa700 1 -- 172.21.15.187:0/4181222415 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe2180459b0 con 0x7fe21c09f4e0 2020-11-20T12:21:20.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:20.233+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fe21c138290 con 0x7fe21c09f4e0 2020-11-20T12:21:21.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.070+0000 7fe2097fa700 1 -- 172.21.15.187:0/4181222415 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v156) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe218048fd0 con 0x7fe21c09f4e0 2020-11-20T12:21:21.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.073+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1f4056d20 msgr2=0x7fe1f40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:21.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.073+0000 7fe2224b6700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1f4056d20 0x7fe1f40591e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7fe20c0013b0 tx=0x7fe20c0011f0).stop 2020-11-20T12:21:21.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.073+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe21c09f4e0 msgr2=0x7fe21c09afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:21.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.073+0000 7fe2224b6700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe21c09f4e0 0x7fe21c09afd0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7fe21800ecf0 tx=0x7fe2180091d0).stop 2020-11-20T12:21:21.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.074+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 shutdown_connections 2020-11-20T12:21:21.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.074+0000 7fe2224b6700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe1f4056d20 0x7fe1f40591e0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:21.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.074+0000 7fe2224b6700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe21c1373a0 0x7fe21c15b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:21.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.074+0000 7fe2224b6700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe21c09f4e0 0x7fe21c09afd0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:21.078 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.074+0000 7fe2224b6700 1 --2- 172.21.15.187:0/4181222415 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe21c09f8c0 0x7fe21c136eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:21.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.074+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 >> 172.21.15.187:0/4181222415 conn(0x7fe21c0bab30 msgr2=0x7fe21c144c50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:21.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.074+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 shutdown_connections 2020-11-20T12:21:21.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.074+0000 7fe2224b6700 1 -- 172.21.15.187:0/4181222415 wait complete. 2020-11-20T12:21:21.080 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_metadata' created 2020-11-20T12:21:21.103 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:21:21.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.361+0000 7efeec81e700 1 Processor -- start 2020-11-20T12:21:21.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.361+0000 7efeec81e700 1 -- start start 2020-11-20T12:21:21.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efeec81e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efee8068230 0x7efee81482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:21.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efeec81e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b5f50 0x7efee80b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:21.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efeec81e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee8144b80 con 0x7efee80b57b0 2020-11-20T12:21:21.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efeec81e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee81487c0 con 0x7efee8068230 2020-11-20T12:21:21.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efeec81e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee81489e0 con 0x7efee80b5f50 2020-11-20T12:21:21.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efede59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efee8068230 0x7efee81482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:21.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efede59c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efee8068230 0x7efee81482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54408/0 (socket says 172.21.15.187:54408) 2020-11-20T12:21:21.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efede59c700 1 -- 172.21.15.187:0/479645227 learned_addr learned my addr 172.21.15.187:0/479645227 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:21.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.363+0000 7efee6d9d700 1 --1- 172.21.15.187:0/479645227 >> v1:172.21.15.187:6789/0 conn(0x7efee80b57b0 0x7efee8062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51012/0 (socket says 172.21.15.187:51012) 2020-11-20T12:21:21.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.364+0000 7efede59c700 1 -- 172.21.15.187:0/479645227 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b5f50 msgr2=0x7efee80b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:21.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.364+0000 7efee759e700 1 --2- 172.21.15.187:0/479645227 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b5f50 0x7efee80b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:21.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.364+0000 7efede59c700 1 --2- 172.21.15.187:0/479645227 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b5f50 0x7efee80b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:21.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.364+0000 7efede59c700 1 -- 172.21.15.187:0/479645227 >> v1:172.21.15.187:6789/0 conn(0x7efee80b57b0 legacy=0x7efee8062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:21.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.364+0000 7efede59c700 1 -- 172.21.15.187:0/479645227 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8148c80 con 0x7efee8068230 2020-11-20T12:21:21.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.364+0000 7efee759e700 1 --2- 172.21.15.187:0/479645227 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b5f50 0x7efee80b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:21.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.364+0000 7efede59c700 1 --2- 172.21.15.187:0/479645227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee81482d0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7efed0006fd0 tx=0x7efed0006eb0).ready entity=mon.1 client_cookie=d3f27324d504a197 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:21.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.364+0000 7efee659c700 1 -- 172.21.15.187:0/479645227 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed00098d0 con 0x7efee8068230 2020-11-20T12:21:21.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.365+0000 7efee659c700 1 -- 172.21.15.187:0/479645227 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed0009a30 con 0x7efee8068230 2020-11-20T12:21:21.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.365+0000 7efee659c700 1 -- 172.21.15.187:0/479645227 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0009db0 con 0x7efee8068230 2020-11-20T12:21:21.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.365+0000 7efeec81e700 1 -- 172.21.15.187:0/479645227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 msgr2=0x7efee81482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.365+0000 7efeec81e700 1 --2- 172.21.15.187:0/479645227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee81482d0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7efed0006fd0 tx=0x7efed0006eb0).stop 2020-11-20T12:21:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.365+0000 7efeec81e700 1 -- 172.21.15.187:0/479645227 shutdown_connections 2020-11-20T12:21:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.365+0000 7efeec81e700 1 --2- 172.21.15.187:0/479645227 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b5f50 0x7efee80b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.365+0000 7efeec81e700 1 --2- 172.21.15.187:0/479645227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee81482d0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:21.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.365+0000 7efeec81e700 1 -- 172.21.15.187:0/479645227 >> 172.21.15.187:0/479645227 conn(0x7efee80bab30 msgr2=0x7efee80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:21.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.366+0000 7efeec81e700 1 -- 172.21.15.187:0/479645227 shutdown_connections 2020-11-20T12:21:21.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.366+0000 7efeec81e700 1 -- 172.21.15.187:0/479645227 wait complete. 2020-11-20T12:21:21.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.367+0000 7efeec81e700 1 Processor -- start 2020-11-20T12:21:21.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.367+0000 7efeec81e700 1 -- start start 2020-11-20T12:21:21.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.368+0000 7efeec81e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee80b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:21.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.368+0000 7efeec81e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b57b0 0x7efee80b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:21.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.368+0000 7efeec81e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efee80b5f50 0x7efee80b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:21.376 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.368+0000 7efeec81e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee81472d0 con 0x7efee80b5f50 2020-11-20T12:21:21.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.368+0000 7efeec81e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee80b4af0 con 0x7efee8068230 2020-11-20T12:21:21.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.368+0000 7efeec81e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee81474c0 con 0x7efee80b57b0 2020-11-20T12:21:21.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.368+0000 7efee6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee80b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:21.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.368+0000 7efee6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee80b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54412/0 (socket says 172.21.15.187:54412) 2020-11-20T12:21:21.377 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efee6d9d700 1 -- 172.21.15.187:0/3668267874 learned_addr learned my addr 172.21.15.187:0/3668267874 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:21.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efede59c700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b57b0 0x7efee80b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:21.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efee6d9d700 1 -- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b57b0 msgr2=0x7efee80b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:21.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efee759e700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efee80b5f50 0x7efee80b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:21.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efee6d9d700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b57b0 0x7efee80b6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:21.378 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efee6d9d700 1 -- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efee80b5f50 msgr2=0x7efee80b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:21.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efee6d9d700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efee80b5f50 0x7efee80b1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:21.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efee6d9d700 1 -- 172.21.15.187:0/3668267874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8147760 con 0x7efee8068230 2020-11-20T12:21:21.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efede59c700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b57b0 0x7efee80b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:21.379 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efee6d9d700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee80b4600 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7efed800c710 tx=0x7efed8007c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:21.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.369+0000 7efedf7fe700 1 -- 172.21.15.187:0/3668267874 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed8007320 con 0x7efee8068230 2020-11-20T12:21:21.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.370+0000 7efedf7fe700 1 -- 172.21.15.187:0/3668267874 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed8005530 con 0x7efee8068230 2020-11-20T12:21:21.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.370+0000 7efedf7fe700 1 -- 172.21.15.187:0/3668267874 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed80058b0 con 0x7efee8068230 2020-11-20T12:21:21.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.370+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee80b7180 con 0x7efee8068230 2020-11-20T12:21:21.380 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.370+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee8145ab0 con 0x7efee8068230 2020-11-20T12:21:21.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.371+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee8061a20 con 0x7efee8068230 2020-11-20T12:21:21.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.371+0000 7efedf7fe700 1 -- 172.21.15.187:0/3668267874 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efed8010030 con 0x7efee8068230 2020-11-20T12:21:21.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.372+0000 7efedf7fe700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efec4056d20 0x7efec40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:21.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.372+0000 7efedf7fe700 1 -- 172.21.15.187:0/3668267874 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5607+0+0 (secure 0 0 0) 0x7efed806fa70 con 0x7efee8068230 2020-11-20T12:21:21.381 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.372+0000 7efede59c700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efec4056d20 0x7efec40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.373+0000 7efede59c700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efec4056d20 0x7efec40591e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7efed000b1b0 tx=0x7efed0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.376+0000 7efedf7fe700 1 -- 172.21.15.187:0/3668267874 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efed8045090 con 0x7efee8068230 2020-11-20T12:21:21.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:21.562+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7efee80446d0 con 0x7efee8068230 2020-11-20T12:21:22.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.087+0000 7efedf7fe700 1 -- 172.21.15.187:0/3668267874 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v157) v1 ==== 125+0+0 (secure 0 0 0) 0x7efed806fe60 con 0x7efee8068230 2020-11-20T12:21:22.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.093+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efec4056d20 msgr2=0x7efec40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.093+0000 7efeec81e700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efec4056d20 0x7efec40591e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7efed000b1b0 tx=0x7efed0015000).stop 2020-11-20T12:21:22.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.093+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 msgr2=0x7efee80b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.093+0000 7efeec81e700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee80b4600 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7efed800c710 tx=0x7efed8007c40).stop 2020-11-20T12:21:22.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.095+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 shutdown_connections 2020-11-20T12:21:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.095+0000 7efeec81e700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efec4056d20 0x7efec40591e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.095+0000 7efeec81e700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efee80b5f50 0x7efee80b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.095+0000 7efeec81e700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efee8068230 0x7efee80b4600 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.095+0000 7efeec81e700 1 --2- 172.21.15.187:0/3668267874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efee80b57b0 0x7efee80b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.095+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 >> 172.21.15.187:0/3668267874 conn(0x7efee80bab30 msgr2=0x7efee80b7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:22.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.096+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 shutdown_connections 2020-11-20T12:21:22.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.097+0000 7efeec81e700 1 -- 172.21.15.187:0/3668267874 wait complete. 2020-11-20T12:21:22.103 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'cephfs_data' created 2020-11-20T12:21:22.130 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:21:22.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.379+0000 7ff3b2a00700 1 Processor -- start 2020-11-20T12:21:22.382 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.380+0000 7ff3b2a00700 1 -- start start 2020-11-20T12:21:22.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.381+0000 7ff3b2a00700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac068230 0x7ff3ac1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.381+0000 7ff3b2a00700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ac144b80 con 0x7ff3ac0b5f50 2020-11-20T12:21:22.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.381+0000 7ff3b2a00700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ac1489e0 con 0x7ff3ac0b57b0 2020-11-20T12:21:22.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.381+0000 7ff3b2a00700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac1487c0 con 0x7ff3ac068230 2020-11-20T12:21:22.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.382+0000 7ff3b19fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff3ac0b57b0 0x7ff3ac062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51026/0 (socket says 172.21.15.187:51026) 2020-11-20T12:21:22.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.382+0000 7ff3b19fe700 1 -- 172.21.15.187:0/2512792712 learned_addr learned my addr 172.21.15.187:0/2512792712 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:22.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.382+0000 7ff3b11fd700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac068230 0x7ff3ac1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.382+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539655949 0 0) 0x7ff3ac1489e0 con 0x7ff3ac0b57b0 2020-11-20T12:21:22.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.382+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff394003240 con 0x7ff3ac0b57b0 2020-11-20T12:21:22.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 129456377 0 0) 0x7ff394003240 con 0x7ff3ac0b57b0 2020-11-20T12:21:22.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac068230 msgr2=0x7ff3ac1482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:22.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac068230 0x7ff3ac1482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 >> v1:172.21.15.5:6789/0 conn(0x7ff3ac0b5f50 legacy=0x7ff3ac0b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac148c80 con 0x7ff3ac0b57b0 2020-11-20T12:21:22.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff3a8001520 con 0x7ff3ac0b57b0 2020-11-20T12:21:22.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 >> v1:172.21.15.187:6789/0 conn(0x7ff3ac0b57b0 legacy=0x7ff3ac062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff394003730 0x7ff394003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac068230 0x7ff3940044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3940049d0 0x7ff394004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac1487c0 con 0x7ff394003730 2020-11-20T12:21:22.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff394004220 con 0x7ff3940049d0 2020-11-20T12:21:22.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.383+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff394005700 con 0x7ff3ac068230 2020-11-20T12:21:22.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b21ff700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3940049d0 0x7ff394004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b11fd700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac068230 0x7ff3940044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b19fe700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff394003730 0x7ff394003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b21ff700 1 -- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac068230 msgr2=0x7ff3940044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:22.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b21ff700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac068230 0x7ff3940044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b21ff700 1 -- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff394003730 msgr2=0x7ff394003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:22.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b21ff700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff394003730 0x7ff394003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b21ff700 1 -- 172.21.15.187:0/2512792712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff394003240 con 0x7ff3940049d0 2020-11-20T12:21:22.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.384+0000 7ff3b21ff700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3940049d0 0x7ff394004e50 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7ff3a4001e70 tx=0x7ff3a4007f10).ready entity=mon.1 client_cookie=f7e3e0ccc3ebb2ab server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:22.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.385+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4001540 con 0x7ff3940049d0 2020-11-20T12:21:22.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.385+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a400c9c0 con 0x7ff3940049d0 2020-11-20T12:21:22.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.385+0000 7ff3b09fc700 1 -- 172.21.15.187:0/2512792712 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a400cd60 con 0x7ff3940049d0 2020-11-20T12:21:22.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.385+0000 7ff3b2a00700 1 -- 172.21.15.187:0/2512792712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3940049d0 msgr2=0x7ff394004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.385+0000 7ff3b2a00700 1 --2- 172.21.15.187:0/2512792712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3940049d0 0x7ff394004e50 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7ff3a4001e70 tx=0x7ff3a4007f10).stop 2020-11-20T12:21:22.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.385+0000 7ff3b21ff700 1 -- 172.21.15.187:0/2512792712 reap_dead start 2020-11-20T12:21:22.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.385+0000 7ff3b2a00700 1 -- 172.21.15.187:0/2512792712 shutdown_connections 2020-11-20T12:21:22.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.385+0000 7ff3b2a00700 1 -- 172.21.15.187:0/2512792712 >> 172.21.15.187:0/2512792712 conn(0x7ff3ac0bab30 msgr2=0x7ff3ac0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:22.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.386+0000 7ff3b2a00700 1 -- 172.21.15.187:0/2512792712 shutdown_connections 2020-11-20T12:21:22.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.386+0000 7ff3b2a00700 1 -- 172.21.15.187:0/2512792712 wait complete. 2020-11-20T12:21:22.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.387+0000 7ff3b2a00700 1 Processor -- start 2020-11-20T12:21:22.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.388+0000 7ff3b2a00700 1 -- start start 2020-11-20T12:21:22.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b2a00700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac0a0000 0x7ff3ac147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b2a00700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac0a03e0 0x7ff3ac1481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b2a00700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ac0b3bc0 0x7ff3ac0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b2a00700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac147660 con 0x7ff3ac0b3bc0 2020-11-20T12:21:22.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b2a00700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac1461f0 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b2a00700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0b4060 con 0x7ff3ac0a0000 2020-11-20T12:21:22.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b11fd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac0a03e0 0x7ff3ac1481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b11fd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac0a03e0 0x7ff3ac1481e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54434/0 (socket says 172.21.15.187:54434) 2020-11-20T12:21:22.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b11fd700 1 -- 172.21.15.187:0/914852741 learned_addr learned my addr 172.21.15.187:0/914852741 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:22.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.389+0000 7ff3b19fe700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac0a0000 0x7ff3ac147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b21ff700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ac0b3bc0 0x7ff3ac0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b11fd700 1 -- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac0a0000 msgr2=0x7ff3ac147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b11fd700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac0a0000 0x7ff3ac147cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b11fd700 1 -- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ac0b3bc0 msgr2=0x7ff3ac0b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b11fd700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ac0b3bc0 0x7ff3ac0b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b11fd700 1 -- 172.21.15.187:0/914852741 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac147850 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b21ff700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ac0b3bc0 0x7ff3ac0b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:22.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b11fd700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac0a03e0 0x7ff3ac1481e0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7ff39c006860 tx=0x7ff39c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:22.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff399ffb700 1 -- 172.21.15.187:0/914852741 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c018070 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff399ffb700 1 -- 172.21.15.187:0/914852741 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c00fb20 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac0b46b0 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff399ffb700 1 -- 172.21.15.187:0/914852741 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c00fea0 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.390+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac0b2230 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.391+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac0b4220 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.392+0000 7ff399ffb700 1 -- 172.21.15.187:0/914852741 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff39c014650 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.393+0000 7ff399ffb700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff384056d50 0x7ff384059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.393+0000 7ff399ffb700 1 -- 172.21.15.187:0/914852741 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ff39c070970 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.393+0000 7ff3b19fe700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff384056d50 0x7ff384059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.394+0000 7ff3b19fe700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff384056d50 0x7ff384059210 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7ff3a8007140 tx=0x7ff3a8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:22.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.398+0000 7ff399ffb700 1 -- 172.21.15.187:0/914852741 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff39c045d20 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.644+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ff3ac0b63d0 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.659+0000 7ff399ffb700 1 -- 172.21.15.187:0/914852741 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 31 and data pool 32 v168) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff39c049340 con 0x7ff3ac0a03e0 2020-11-20T12:21:22.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.663+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff384056d50 msgr2=0x7ff384059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.663+0000 7ff3b2a00700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff384056d50 0x7ff384059210 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7ff3a8007140 tx=0x7ff3a8009040).stop 2020-11-20T12:21:22.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.664+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac0a03e0 msgr2=0x7ff3ac1481e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.664+0000 7ff3b2a00700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac0a03e0 0x7ff3ac1481e0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7ff39c006860 tx=0x7ff39c006df0).stop 2020-11-20T12:21:22.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.665+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 shutdown_connections 2020-11-20T12:21:22.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.665+0000 7ff3b2a00700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff384056d50 0x7ff384059210 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.665+0000 7ff3b2a00700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ac0b3bc0 0x7ff3ac0b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.665+0000 7ff3b2a00700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ac0a0000 0x7ff3ac147cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.665+0000 7ff3b2a00700 1 --2- 172.21.15.187:0/914852741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac0a03e0 0x7ff3ac1481e0 secure :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 rx=0x7ff39c006860 tx=0x7ff39c006df0).stop 2020-11-20T12:21:22.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.665+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 >> 172.21.15.187:0/914852741 conn(0x7ff3ac0bab30 msgr2=0x7ff3ac0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:22.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.666+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 shutdown_connections 2020-11-20T12:21:22.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.666+0000 7ff3b2a00700 1 -- 172.21.15.187:0/914852741 wait complete. 2020-11-20T12:21:22.673 INFO:teuthology.orchestra.run.smithi187.stderr:new fs with metadata pool 31 and data pool 32 2020-11-20T12:21:22.698 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:21:22.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.916+0000 7f85e7fff700 1 Processor -- start 2020-11-20T12:21:22.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.917+0000 7f85e7fff700 1 -- start start 2020-11-20T12:21:22.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.918+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e8065190 0x7f85e8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.918+0000 7f85e7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f85e809ca90 0x7f85e809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.918+0000 7f85e7fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80b13e0 con 0x7f85e8065f90 2020-11-20T12:21:22.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.918+0000 7f85e7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e814ab30 con 0x7f85e809ca90 2020-11-20T12:21:22.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.918+0000 7f85e7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e8062c10 con 0x7f85e8065190 2020-11-20T12:21:22.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.918+0000 7f85de7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f85e8065f90 0x7f85e8143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51060/0 (socket says 172.21.15.187:51060) 2020-11-20T12:21:22.922 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.918+0000 7f85de7fc700 1 -- 172.21.15.187:0/1930105537 learned_addr learned my addr 172.21.15.187:0/1930105537 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:22.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223502463 0 0) 0x7f85e80b13e0 con 0x7f85e8065f90 2020-11-20T12:21:22.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c8003240 con 0x7f85e8065f90 2020-11-20T12:21:22.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e6ffd700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e8065190 0x7f85e8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e77fe700 1 --2- 172.21.15.187:0/1930105537 >> v2:172.21.15.5:3300/0 conn(0x7f85e809ca90 0x7f85e809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3612921614 0 0) 0x7f85c8003240 con 0x7f85e8065f90 2020-11-20T12:21:22.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e8065190 msgr2=0x7f85e8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:22.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e8065190 0x7f85e8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 >> v2:172.21.15.5:3300/0 conn(0x7f85e809ca90 msgr2=0x7f85e809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:22.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 --2- 172.21.15.187:0/1930105537 >> v2:172.21.15.5:3300/0 conn(0x7f85e809ca90 0x7f85e809cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:22.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80b1640 con 0x7f85e8065f90 2020-11-20T12:21:22.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f85d00014e0 con 0x7f85e8065f90 2020-11-20T12:21:22.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.919+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 >> v1:172.21.15.187:6789/0 conn(0x7f85e8065f90 legacy=0x7f85e8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e67fc700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c80037b0 0x7f85c8003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e67fc700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e8065190 0x7f85c80044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e67fc700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c80049d0 0x7f85c8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e814ab30 con 0x7f85c80037b0 2020-11-20T12:21:22.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e8062c10 con 0x7f85c80049d0 2020-11-20T12:21:22.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c80042c0 con 0x7f85e8065190 2020-11-20T12:21:22.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e77fe700 1 --2- 172.21.15.187:0/1930105537 >> v2:172.21.15.5:3300/0 conn(0x7f85e809ca90 0x7f85e809cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:22.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e77fe700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c80049d0 0x7f85c8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e77fe700 1 -- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e8065190 msgr2=0x7f85c80044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:22.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e77fe700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e8065190 0x7f85c80044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:22.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e77fe700 1 -- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c80037b0 msgr2=0x7f85c8003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:22.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e77fe700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85c80037b0 0x7f85c8003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:22.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.920+0000 7f85e77fe700 1 -- 172.21.15.187:0/1930105537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8003240 con 0x7f85c80049d0 2020-11-20T12:21:22.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.921+0000 7f85e77fe700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c80049d0 0x7f85c8004e50 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f85e0007df0 tx=0x7f85e0008040).ready entity=mon.1 client_cookie=6f59a715906c072d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:22.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.921+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e000cc70 con 0x7f85c80049d0 2020-11-20T12:21:22.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.921+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e000cdd0 con 0x7f85c80049d0 2020-11-20T12:21:22.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.921+0000 7f85e67fc700 1 -- 172.21.15.187:0/1930105537 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e0011620 con 0x7f85c80049d0 2020-11-20T12:21:22.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.921+0000 7f85e7fff700 1 -- 172.21.15.187:0/1930105537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c80049d0 msgr2=0x7f85c8004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.921+0000 7f85e7fff700 1 --2- 172.21.15.187:0/1930105537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85c80049d0 0x7f85c8004e50 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f85e0007df0 tx=0x7f85e0008040).stop 2020-11-20T12:21:22.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.922+0000 7f85e77fe700 1 -- 172.21.15.187:0/1930105537 reap_dead start 2020-11-20T12:21:22.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.922+0000 7f85e7fff700 1 -- 172.21.15.187:0/1930105537 shutdown_connections 2020-11-20T12:21:22.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.922+0000 7f85e7fff700 1 -- 172.21.15.187:0/1930105537 >> 172.21.15.187:0/1930105537 conn(0x7f85e80bab30 msgr2=0x7f85e80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:22.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.922+0000 7f85e7fff700 1 -- 172.21.15.187:0/1930105537 shutdown_connections 2020-11-20T12:21:22.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.922+0000 7f85e7fff700 1 -- 172.21.15.187:0/1930105537 wait complete. 2020-11-20T12:21:22.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.923+0000 7f85e7fff700 1 Processor -- start 2020-11-20T12:21:22.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.923+0000 7f85e7fff700 1 -- start start 2020-11-20T12:21:22.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.924+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85e80b42e0 0x7f85e80990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.924+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e80b46c0 0x7f85e80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.924+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e809f090 0x7f85e8142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.935 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.924+0000 7f85e7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e80b2e00 con 0x7f85e80b42e0 2020-11-20T12:21:22.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.924+0000 7f85e7fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e80b1d00 con 0x7f85e80b46c0 2020-11-20T12:21:22.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.924+0000 7f85e7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e80b1fa0 con 0x7f85e809f090 2020-11-20T12:21:22.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e80b46c0 0x7f85e80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e80b46c0 0x7f85e80b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54466/0 (socket says 172.21.15.187:54466) 2020-11-20T12:21:22.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 -- 172.21.15.187:0/1238728959 learned_addr learned my addr 172.21.15.187:0/1238728959 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:22.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 -- 172.21.15.187:0/1238728959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e809f090 msgr2=0x7f85e8142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e809f090 0x7f85e8142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:22.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 -- 172.21.15.187:0/1238728959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85e80b42e0 msgr2=0x7f85e80990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:22.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85e80b42e0 0x7f85e80990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:22.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 -- 172.21.15.187:0/1238728959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e81466e0 con 0x7f85e80b46c0 2020-11-20T12:21:22.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85de7fc700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e80b46c0 0x7f85e80b2910 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7f85d000ca90 tx=0x7f85e80bb650).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:22.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.925+0000 7f85df7fe700 1 -- 172.21.15.187:0/1238728959 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0007170 con 0x7f85e80b46c0 2020-11-20T12:21:22.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.926+0000 7f85df7fe700 1 -- 172.21.15.187:0/1238728959 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d00072f0 con 0x7f85e80b46c0 2020-11-20T12:21:22.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.926+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e8146b80 con 0x7f85e80b46c0 2020-11-20T12:21:22.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.926+0000 7f85df7fe700 1 -- 172.21.15.187:0/1238728959 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d000aa40 con 0x7f85e80b46c0 2020-11-20T12:21:22.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.926+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e8143430 con 0x7f85e80b46c0 2020-11-20T12:21:22.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.927+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e80b1b40 con 0x7f85e80b46c0 2020-11-20T12:21:22.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.927+0000 7f85df7fe700 1 -- 172.21.15.187:0/1238728959 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f85d001b060 con 0x7f85e80b46c0 2020-11-20T12:21:22.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.928+0000 7f85df7fe700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85c4056d20 0x7f85c40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:22.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.928+0000 7f85df7fe700 1 -- 172.21.15.187:0/1238728959 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f85d006fab0 con 0x7f85e80b46c0 2020-11-20T12:21:22.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.928+0000 7f85e6ffd700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85c4056d20 0x7f85c40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:22.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.929+0000 7f85e6ffd700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85c4056d20 0x7f85c40591e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f85d8006a30 tx=0x7f85d8008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:22.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:22.932+0000 7f85df7fe700 1 -- 172.21.15.187:0/1238728959 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f85d0044e80 con 0x7f85e80b46c0 2020-11-20T12:21:23.126 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.123+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85e8061a20 con 0x7f85e80b46c0 2020-11-20T12:21:23.128 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.126+0000 7f85df7fe700 1 -- 172.21.15.187:0/1238728959 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+16805 (secure 0 0 0) 0x7f85d0019030 con 0x7f85e80b46c0 2020-11-20T12:21:23.129 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:23.129 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":158,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:21:22.646265+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:21:20.315283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2020-11-20T12:21:21.644244+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:21:23.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.128+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85c4056d20 msgr2=0x7f85c40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.128+0000 7f85e7fff700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85c4056d20 0x7f85c40591e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f85d8006a30 tx=0x7f85d8008040).stop 2020-11-20T12:21:23.131 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.128+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e80b46c0 msgr2=0x7f85e80b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.128+0000 7f85e7fff700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e80b46c0 0x7f85e80b2910 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7f85d000ca90 tx=0x7f85e80bb650).stop 2020-11-20T12:21:23.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.128+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 shutdown_connections 2020-11-20T12:21:23.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.129+0000 7f85e7fff700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f85c4056d20 0x7f85c40591e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.129+0000 7f85e7fff700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f85e80b42e0 0x7f85e80990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:23.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.129+0000 7f85e7fff700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85e809f090 0x7f85e8142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:23.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.129+0000 7f85e7fff700 1 --2- 172.21.15.187:0/1238728959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e80b46c0 0x7f85e80b2910 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.129+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 >> 172.21.15.187:0/1238728959 conn(0x7f85e80bab30 msgr2=0x7f85e809baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:23.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.129+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 shutdown_connections 2020-11-20T12:21:23.133 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.129+0000 7f85e7fff700 1 -- 172.21.15.187:0/1238728959 wait complete. 2020-11-20T12:21:23.152 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:21:23.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.394+0000 7fbaa759e700 1 Processor -- start 2020-11-20T12:21:23.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.395+0000 7fbaa759e700 1 -- start start 2020-11-20T12:21:23.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.396+0000 7fbaa759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbaa8065f90 0x7fbaa8065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.396+0000 7fbaa759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbaa809ca90 0x7fbaa809cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.396+0000 7fbaa759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa80b13e0 con 0x7fbaa8065190 2020-11-20T12:21:23.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.396+0000 7fbaa759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa814ab30 con 0x7fbaa8065f90 2020-11-20T12:21:23.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.396+0000 7fbaa759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa8062c10 con 0x7fbaa809ca90 2020-11-20T12:21:23.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.397+0000 7fbaa6d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbaa809ca90 0x7fbaa809cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.397+0000 7fbaa6d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbaa809ca90 0x7fbaa809cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54476/0 (socket says 172.21.15.187:54476) 2020-11-20T12:21:23.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.397+0000 7fbaa6d9d700 1 -- 172.21.15.187:0/3392958770 learned_addr learned my addr 172.21.15.187:0/3392958770 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:23.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.397+0000 7fba9dd9b700 1 --2- 172.21.15.187:0/3392958770 >> v2:172.21.15.5:3300/0 conn(0x7fbaa8065f90 0x7fbaa8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.397+0000 7fbaa6d9d700 1 -- 172.21.15.187:0/3392958770 >> v2:172.21.15.5:3300/0 conn(0x7fbaa8065f90 msgr2=0x7fbaa8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:23.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.397+0000 7fbaa6d9d700 1 --2- 172.21.15.187:0/3392958770 >> v2:172.21.15.5:3300/0 conn(0x7fbaa8065f90 0x7fbaa8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.397+0000 7fbaa6d9d700 1 -- 172.21.15.187:0/3392958770 >> v1:172.21.15.5:6789/0 conn(0x7fbaa8065190 legacy=0x7fbaa8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:23.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.397+0000 7fbaa6d9d700 1 -- 172.21.15.187:0/3392958770 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa80b1640 con 0x7fbaa809ca90 2020-11-20T12:21:23.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.398+0000 7fbaa6d9d700 1 --2- 172.21.15.187:0/3392958770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa809ca90 0x7fbaa809cf20 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7fbaa8099f90 tx=0x7fbaa0009290).ready entity=mon.1 client_cookie=606c3067bc0c9f91 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:23.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.398+0000 7fbaa5d9b700 1 -- 172.21.15.187:0/3392958770 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa000bc10 con 0x7fbaa809ca90 2020-11-20T12:21:23.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.398+0000 7fbaa5d9b700 1 -- 172.21.15.187:0/3392958770 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa000bd70 con 0x7fbaa809ca90 2020-11-20T12:21:23.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.398+0000 7fbaa5d9b700 1 -- 172.21.15.187:0/3392958770 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0011610 con 0x7fbaa809ca90 2020-11-20T12:21:23.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.398+0000 7fbaa759e700 1 -- 172.21.15.187:0/3392958770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa809ca90 msgr2=0x7fbaa809cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.398+0000 7fbaa759e700 1 --2- 172.21.15.187:0/3392958770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa809ca90 0x7fbaa809cf20 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7fbaa8099f90 tx=0x7fbaa0009290).stop 2020-11-20T12:21:23.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.399+0000 7fbaa759e700 1 -- 172.21.15.187:0/3392958770 shutdown_connections 2020-11-20T12:21:23.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.399+0000 7fbaa759e700 1 --2- 172.21.15.187:0/3392958770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa809ca90 0x7fbaa809cf20 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.399+0000 7fbaa759e700 1 --2- 172.21.15.187:0/3392958770 >> v2:172.21.15.5:3300/0 conn(0x7fbaa8065f90 0x7fbaa8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.399+0000 7fbaa759e700 1 -- 172.21.15.187:0/3392958770 >> 172.21.15.187:0/3392958770 conn(0x7fbaa80bab30 msgr2=0x7fbaa80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:23.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.399+0000 7fbaa759e700 1 -- 172.21.15.187:0/3392958770 shutdown_connections 2020-11-20T12:21:23.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.399+0000 7fbaa759e700 1 -- 172.21.15.187:0/3392958770 wait complete. 2020-11-20T12:21:23.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.401+0000 7fbaa759e700 1 Processor -- start 2020-11-20T12:21:23.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.401+0000 7fbaa759e700 1 -- start start 2020-11-20T12:21:23.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.402+0000 7fbaa759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbaa8065190 0x7fbaa8149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.402+0000 7fbaa759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa8065f90 0x7fbaa81495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.402+0000 7fbaa759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbaa809ca90 0x7fbaa8151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.402+0000 7fbaa759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa8143d30 con 0x7fbaa809ca90 2020-11-20T12:21:23.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.402+0000 7fbaa759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa81467f0 con 0x7fbaa8065f90 2020-11-20T12:21:23.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.402+0000 7fbaa759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa80b1be0 con 0x7fbaa8065190 2020-11-20T12:21:23.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fba9dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa8065f90 0x7fbaa81495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fba9dd9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa8065f90 0x7fbaa81495e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54482/0 (socket says 172.21.15.187:54482) 2020-11-20T12:21:23.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fba9dd9b700 1 -- 172.21.15.187:0/668031459 learned_addr learned my addr 172.21.15.187:0/668031459 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:23.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fbaa6d9d700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbaa809ca90 0x7fbaa8151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fbaa659c700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbaa8065190 0x7fbaa8149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fba9dd9b700 1 -- 172.21.15.187:0/668031459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbaa8065190 msgr2=0x7fbaa8149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fba9dd9b700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbaa8065190 0x7fbaa8149070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fba9dd9b700 1 -- 172.21.15.187:0/668031459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbaa809ca90 msgr2=0x7fbaa8151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fba9dd9b700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbaa809ca90 0x7fbaa8151490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.403+0000 7fba9dd9b700 1 -- 172.21.15.187:0/668031459 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa8146a50 con 0x7fbaa8065f90 2020-11-20T12:21:23.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.404+0000 7fba9dd9b700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa8065f90 0x7fbaa81495e0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7fba9000db60 tx=0x7fba90008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:23.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.404+0000 7fba9effd700 1 -- 172.21.15.187:0/668031459 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9000b170 con 0x7fbaa8065f90 2020-11-20T12:21:23.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.404+0000 7fba9effd700 1 -- 172.21.15.187:0/668031459 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba90009790 con 0x7fbaa8065f90 2020-11-20T12:21:23.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.404+0000 7fba9effd700 1 -- 172.21.15.187:0/668031459 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90009b50 con 0x7fbaa8065f90 2020-11-20T12:21:23.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.404+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa809f210 con 0x7fbaa8065f90 2020-11-20T12:21:23.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.404+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa8151bc0 con 0x7fbaa8065f90 2020-11-20T12:21:23.413 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.405+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa809a8b0 con 0x7fbaa8065f90 2020-11-20T12:21:23.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.406+0000 7fba9effd700 1 -- 172.21.15.187:0/668031459 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fba9001c460 con 0x7fbaa8065f90 2020-11-20T12:21:23.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.406+0000 7fba9effd700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba84056d20 0x7fba840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.406+0000 7fba9effd700 1 -- 172.21.15.187:0/668031459 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fba900702e0 con 0x7fbaa8065f90 2020-11-20T12:21:23.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.407+0000 7fbaa659c700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba84056d20 0x7fba840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.407+0000 7fbaa659c700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba84056d20 0x7fba840591e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7fbaa809f1a0 tx=0x7fba98009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:23.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.410+0000 7fba9effd700 1 -- 172.21.15.187:0/668031459 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbaa809a8b0 con 0x7fbaa8065f90 2020-11-20T12:21:23.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.599+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaa809a8b0 con 0x7fbaa8065f90 2020-11-20T12:21:23.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.601+0000 7fba9effd700 1 -- 172.21.15.187:0/668031459 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+16805 (secure 0 0 0) 0x7fbaa809a8b0 con 0x7fbaa8065f90 2020-11-20T12:21:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:23.604 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":158,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:21:22.646265+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:21:20.315283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2020-11-20T12:21:21.644244+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:21:23.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba84056d20 msgr2=0x7fba840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba84056d20 0x7fba840591e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7fbaa809f1a0 tx=0x7fba98009040).stop 2020-11-20T12:21:23.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa8065f90 msgr2=0x7fbaa81495e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa8065f90 0x7fbaa81495e0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7fba9000db60 tx=0x7fba90008040).stop 2020-11-20T12:21:23.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 shutdown_connections 2020-11-20T12:21:23.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fba84056d20 0x7fba840591e0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbaa809ca90 0x7fbaa8151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbaa8065190 0x7fbaa8149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 --2- 172.21.15.187:0/668031459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa8065f90 0x7fbaa81495e0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 >> 172.21.15.187:0/668031459 conn(0x7fbaa80bab30 msgr2=0x7fbaa80b89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:23.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.604+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 shutdown_connections 2020-11-20T12:21:23.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.605+0000 7fbaa759e700 1 -- 172.21.15.187:0/668031459 wait complete. 2020-11-20T12:21:23.631 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:21:23.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.868+0000 7fd14f162700 1 Processor -- start 2020-11-20T12:21:23.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.869+0000 7fd14f162700 1 -- start start 2020-11-20T12:21:23.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.870+0000 7fd14f162700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd1480b1760 0x7fd148065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.870+0000 7fd14f162700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd148065680 0x7fd14814c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.870+0000 7fd14f162700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd148150390 con 0x7fd1480b1380 2020-11-20T12:21:23.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.870+0000 7fd14f162700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1481511e0 con 0x7fd1480b1760 2020-11-20T12:21:23.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.870+0000 7fd14f162700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14814d760 con 0x7fd148065680 2020-11-20T12:21:23.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.871+0000 7fd14e961700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd148065680 0x7fd14814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.871+0000 7fd14e160700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd1480b1380 0x7fd148150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:44604/0 (socket says 172.21.15.187:44604) 2020-11-20T12:21:23.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.871+0000 7fd14e160700 1 -- 172.21.15.187:0/1715957940 learned_addr learned my addr 172.21.15.187:0/1715957940 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:23.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.871+0000 7fd14d95f700 1 --2- 172.21.15.187:0/1715957940 >> v2:172.21.15.5:3300/0 conn(0x7fd1480b1760 0x7fd148065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.872+0000 7fd14e961700 1 -- 172.21.15.187:0/1715957940 >> v2:172.21.15.5:3300/0 conn(0x7fd1480b1760 msgr2=0x7fd148065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:23.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.872+0000 7fd14e961700 1 --2- 172.21.15.187:0/1715957940 >> v2:172.21.15.5:3300/0 conn(0x7fd1480b1760 0x7fd148065190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.872+0000 7fd14e961700 1 -- 172.21.15.187:0/1715957940 >> v1:172.21.15.5:6789/0 conn(0x7fd1480b1380 legacy=0x7fd148150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.872+0000 7fd14e961700 1 -- 172.21.15.187:0/1715957940 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148150630 con 0x7fd148065680 2020-11-20T12:21:23.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.872+0000 7fd14e961700 1 --2- 172.21.15.187:0/1715957940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd148065680 0x7fd14814c260 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fd140009c90 tx=0x7fd1400091d0).ready entity=mon.2 client_cookie=8fa94d1ca9683ea4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:23.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.873+0000 7fd14d15e700 1 -- 172.21.15.187:0/1715957940 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14000ba70 con 0x7fd148065680 2020-11-20T12:21:23.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.873+0000 7fd14d15e700 1 -- 172.21.15.187:0/1715957940 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14000bbd0 con 0x7fd148065680 2020-11-20T12:21:23.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.873+0000 7fd14d15e700 1 -- 172.21.15.187:0/1715957940 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140011400 con 0x7fd148065680 2020-11-20T12:21:23.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.873+0000 7fd14f162700 1 -- 172.21.15.187:0/1715957940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd148065680 msgr2=0x7fd14814c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.873+0000 7fd14f162700 1 --2- 172.21.15.187:0/1715957940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd148065680 0x7fd14814c260 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fd140009c90 tx=0x7fd1400091d0).stop 2020-11-20T12:21:23.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.874+0000 7fd14f162700 1 -- 172.21.15.187:0/1715957940 shutdown_connections 2020-11-20T12:21:23.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.874+0000 7fd14f162700 1 --2- 172.21.15.187:0/1715957940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd148065680 0x7fd14814c260 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.874+0000 7fd14f162700 1 --2- 172.21.15.187:0/1715957940 >> v2:172.21.15.5:3300/0 conn(0x7fd1480b1760 0x7fd148065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.874+0000 7fd14f162700 1 -- 172.21.15.187:0/1715957940 >> 172.21.15.187:0/1715957940 conn(0x7fd1480bab30 msgr2=0x7fd1480baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:23.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.874+0000 7fd14f162700 1 -- 172.21.15.187:0/1715957940 shutdown_connections 2020-11-20T12:21:23.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.874+0000 7fd14f162700 1 -- 172.21.15.187:0/1715957940 wait complete. 2020-11-20T12:21:23.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.876+0000 7fd14f162700 1 Processor -- start 2020-11-20T12:21:23.880 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.876+0000 7fd14f162700 1 -- start start 2020-11-20T12:21:23.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.877+0000 7fd14f162700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1480b1380 0x7fd1480b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.877+0000 7fd14f162700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1480b1760 0x7fd1480b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.877+0000 7fd14f162700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd14815fd10 0x7fd14809a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.881 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.877+0000 7fd14f162700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480b2170 con 0x7fd1480b1760 2020-11-20T12:21:23.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.877+0000 7fd14f162700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480b2410 con 0x7fd1480b1380 2020-11-20T12:21:23.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.877+0000 7fd14f162700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480b26b0 con 0x7fd14815fd10 2020-11-20T12:21:23.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.878+0000 7fd14e160700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1480b1380 0x7fd1480b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.878+0000 7fd14e961700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd14815fd10 0x7fd14809a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.882 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.878+0000 7fd14d95f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1480b1760 0x7fd1480b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.878+0000 7fd14d95f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1480b1760 0x7fd1480b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:46762/0 (socket says 172.21.15.187:46762) 2020-11-20T12:21:23.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.878+0000 7fd14d95f700 1 -- 172.21.15.187:0/3421357123 learned_addr learned my addr 172.21.15.187:0/3421357123 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:23.883 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.878+0000 7fd14e160700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1480b1380 0x7fd1480b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54496/0 (socket says 172.21.15.187:54496) 2020-11-20T12:21:23.884 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd14e160700 1 -- 172.21.15.187:0/3421357123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd14815fd10 msgr2=0x7fd14809a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd14e160700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd14815fd10 0x7fd14809a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd14e160700 1 -- 172.21.15.187:0/3421357123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1480b1760 msgr2=0x7fd1480b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:23.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd14e160700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1480b1760 0x7fd1480b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:23.885 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd14e160700 1 -- 172.21.15.187:0/3421357123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480b4580 con 0x7fd1480b1380 2020-11-20T12:21:23.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd14e160700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1480b1380 0x7fd1480b1b40 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7fd144007fd0 tx=0x7fd144007b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:23.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd1367fc700 1 -- 172.21.15.187:0/3421357123 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1440072e0 con 0x7fd1480b1380 2020-11-20T12:21:23.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd1367fc700 1 -- 172.21.15.187:0/3421357123 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd144007440 con 0x7fd1480b1380 2020-11-20T12:21:23.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14809aa60 con 0x7fd1480b1380 2020-11-20T12:21:23.886 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd1367fc700 1 -- 172.21.15.187:0/3421357123 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144005820 con 0x7fd1480b1380 2020-11-20T12:21:23.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.879+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14814a660 con 0x7fd1480b1380 2020-11-20T12:21:23.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.881+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd148061a20 con 0x7fd1480b1380 2020-11-20T12:21:23.887 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.882+0000 7fd1367fc700 1 -- 172.21.15.187:0/3421357123 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd14401a060 con 0x7fd1480b1380 2020-11-20T12:21:23.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.883+0000 7fd1367fc700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd120056d20 0x7fd1200591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:23.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.883+0000 7fd1367fc700 1 -- 172.21.15.187:0/3421357123 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd14406f630 con 0x7fd1480b1380 2020-11-20T12:21:23.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.883+0000 7fd14d95f700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd120056d20 0x7fd1200591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:23.888 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.884+0000 7fd14d95f700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd120056d20 0x7fd1200591e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7fd1480b8160 tx=0x7fd138006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:23.890 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:23.887+0000 7fd1367fc700 1 -- 172.21.15.187:0/3421357123 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd144044a80 con 0x7fd1480b1380 2020-11-20T12:21:24.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:24.140+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd14809abf0 con 0x7fd1480b1380 2020-11-20T12:21:24.303 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:21:24.313 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:21:25.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.101+0000 7fd1367fc700 1 -- 172.21.15.187:0/3421357123 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v171) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd14809abf0 con 0x7fd1480b1380 2020-11-20T12:21:25.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd120056d20 msgr2=0x7fd1200591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd120056d20 0x7fd1200591e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7fd1480b8160 tx=0x7fd138006df0).stop 2020-11-20T12:21:25.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1480b1380 msgr2=0x7fd1480b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1480b1380 0x7fd1480b1b40 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7fd144007fd0 tx=0x7fd144007b50).stop 2020-11-20T12:21:25.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 shutdown_connections 2020-11-20T12:21:25.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fd120056d20 0x7fd1200591e0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd14815fd10 0x7fd14809a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1480b1380 0x7fd1480b1b40 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 --2- 172.21.15.187:0/3421357123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1480b1760 0x7fd1480b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.105+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 >> 172.21.15.187:0/3421357123 conn(0x7fd1480bab30 msgr2=0x7fd14814c940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:25.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.106+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 shutdown_connections 2020-11-20T12:21:25.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.107+0000 7fd14f162700 1 -- 172.21.15.187:0/3421357123 wait complete. 2020-11-20T12:21:25.142 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:25.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.381+0000 7f5bf62be700 1 Processor -- start 2020-11-20T12:21:25.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.381+0000 7f5bf62be700 1 -- start start 2020-11-20T12:21:25.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.383+0000 7f5bf62be700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf0099ce0 0x7f5bf00b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.385 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.383+0000 7f5bf62be700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf00b5df0 con 0x7f5bf0065e10 2020-11-20T12:21:25.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.383+0000 7f5bf62be700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf009a440 con 0x7f5bf00661f0 2020-11-20T12:21:25.386 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.383+0000 7f5bf62be700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf009a1a0 con 0x7f5bf0099ce0 2020-11-20T12:21:25.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.383+0000 7f5be77fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5bf00661f0 0x7f5bf00b62e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51110/0 (socket says 172.21.15.187:51110) 2020-11-20T12:21:25.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.383+0000 7f5be77fe700 1 -- 172.21.15.187:0/941691951 learned_addr learned my addr 172.21.15.187:0/941691951 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:25.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bf485b700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf0099ce0 0x7f5bf00b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211354795 0 0) 0x7f5bf009a440 con 0x7f5bf00661f0 2020-11-20T12:21:25.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd0004350 con 0x7f5bf00661f0 2020-11-20T12:21:25.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3634810213 0 0) 0x7f5bd0004350 con 0x7f5bf00661f0 2020-11-20T12:21:25.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf0099ce0 msgr2=0x7f5bf00b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:25.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf0099ce0 0x7f5bf00b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 >> v1:172.21.15.5:6789/0 conn(0x7f5bf0065e10 legacy=0x7f5bf0144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf00b0160 con 0x7f5bf00661f0 2020-11-20T12:21:25.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f5bd8001660 con 0x7f5bf00661f0 2020-11-20T12:21:25.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.384+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 >> v1:172.21.15.187:6789/0 conn(0x7f5bf00661f0 legacy=0x7f5bf00b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5bef7fe700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bd0004840 0x7f5bd0004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5bef7fe700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bd00055f0 0x7f5bd0005a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5bef7fe700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf0099ce0 0x7f5bd0005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf009a1a0 con 0x7f5bd0004840 2020-11-20T12:21:25.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd00064a0 con 0x7f5bd00055f0 2020-11-20T12:21:25.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0005430 con 0x7f5bf0099ce0 2020-11-20T12:21:25.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5be77fe700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bd00055f0 0x7f5bd0005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5bf485b700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf0099ce0 0x7f5bd0005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.385+0000 7f5beffff700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bd0004840 0x7f5bd0004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.386+0000 7f5be77fe700 1 -- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf0099ce0 msgr2=0x7f5bd0005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:25.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.386+0000 7f5be77fe700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf0099ce0 0x7f5bd0005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.386+0000 7f5be77fe700 1 -- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bd0004840 msgr2=0x7f5bd0004c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:25.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.386+0000 7f5be77fe700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bd0004840 0x7f5bd0004c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.386+0000 7f5be77fe700 1 -- 172.21.15.187:0/941691951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0004350 con 0x7f5bd00055f0 2020-11-20T12:21:25.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.386+0000 7f5be77fe700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bd00055f0 0x7f5bd0005a70 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f5bd800c410 tx=0x7f5bd80077f0).ready entity=mon.1 client_cookie=59b887a85cc3d2d6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:25.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd800ce80 con 0x7f5bd00055f0 2020-11-20T12:21:25.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd8007860 con 0x7f5bd00055f0 2020-11-20T12:21:25.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bef7fe700 1 -- 172.21.15.187:0/941691951 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80056d0 con 0x7f5bd00055f0 2020-11-20T12:21:25.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bf62be700 1 -- 172.21.15.187:0/941691951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bd00055f0 msgr2=0x7f5bd0005a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bf62be700 1 --2- 172.21.15.187:0/941691951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bd00055f0 0x7f5bd0005a70 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f5bd800c410 tx=0x7f5bd80077f0).stop 2020-11-20T12:21:25.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bf485b700 1 -- 172.21.15.187:0/941691951 reap_dead start 2020-11-20T12:21:25.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bf62be700 1 -- 172.21.15.187:0/941691951 shutdown_connections 2020-11-20T12:21:25.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bf62be700 1 -- 172.21.15.187:0/941691951 >> 172.21.15.187:0/941691951 conn(0x7f5bf00bacd0 msgr2=0x7f5bf00bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:25.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.387+0000 7f5bf62be700 1 -- 172.21.15.187:0/941691951 shutdown_connections 2020-11-20T12:21:25.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.388+0000 7f5bf62be700 1 -- 172.21.15.187:0/941691951 wait complete. 2020-11-20T12:21:25.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.390+0000 7f5bf62be700 1 Processor -- start 2020-11-20T12:21:25.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.390+0000 7f5bf62be700 1 -- start start 2020-11-20T12:21:25.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf62be700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bf00b3750 0x7f5bf0153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf62be700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf00b3b30 0x7f5bf01545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf62be700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf00b2c70 0x7f5bf0154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf62be700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf009f8a0 con 0x7f5bf00b3750 2020-11-20T12:21:25.399 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf62be700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf00b2290 con 0x7f5bf00b2c70 2020-11-20T12:21:25.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf62be700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf00b2530 con 0x7f5bf00b3b30 2020-11-20T12:21:25.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf485b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf00b2c70 0x7f5bf0154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf485b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf00b2c70 0x7f5bf0154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54518/0 (socket says 172.21.15.187:54518) 2020-11-20T12:21:25.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.391+0000 7f5bf485b700 1 -- 172.21.15.187:0/999331378 learned_addr learned my addr 172.21.15.187:0/999331378 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:25.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5bf485b700 1 -- 172.21.15.187:0/999331378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf00b3b30 msgr2=0x7f5bf01545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5bf485b700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf00b3b30 0x7f5bf01545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:25.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5bf485b700 1 -- 172.21.15.187:0/999331378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bf00b3750 msgr2=0x7f5bf0153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5bf485b700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bf00b3750 0x7f5bf0153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:25.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5bf485b700 1 -- 172.21.15.187:0/999331378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf00b27d0 con 0x7f5bf00b2c70 2020-11-20T12:21:25.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5bf485b700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf00b2c70 0x7f5bf0154ac0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7f5bf0067de0 tx=0x7f5be8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:25.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5becff9700 1 -- 172.21.15.187:0/999331378 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be8017070 con 0x7f5bf00b2c70 2020-11-20T12:21:25.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5becff9700 1 -- 172.21.15.187:0/999331378 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be8011590 con 0x7f5bf00b2c70 2020-11-20T12:21:25.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5becff9700 1 -- 172.21.15.187:0/999331378 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be8011930 con 0x7f5bf00b2c70 2020-11-20T12:21:25.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.392+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf00a0960 con 0x7f5bf00b2c70 2020-11-20T12:21:25.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.393+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf00b2020 con 0x7f5bf00b2c70 2020-11-20T12:21:25.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.394+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf0063600 con 0x7f5bf00b2c70 2020-11-20T12:21:25.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.394+0000 7f5becff9700 1 -- 172.21.15.187:0/999331378 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5be8011ca0 con 0x7f5bf00b2c70 2020-11-20T12:21:25.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.395+0000 7f5becff9700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5bcc056d20 0x7f5bcc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.395+0000 7f5becff9700 1 -- 172.21.15.187:0/999331378 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5be806fe30 con 0x7f5bf00b2c70 2020-11-20T12:21:25.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.396+0000 7f5beffff700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5bcc056d20 0x7f5bcc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.396+0000 7f5beffff700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5bcc056d20 0x7f5bcc0591e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f5be0003c00 tx=0x7f5be0009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:25.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.400+0000 7f5becff9700 1 -- 172.21.15.187:0/999331378 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5be8045170 con 0x7f5bf00b2c70 2020-11-20T12:21:25.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.644+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bf00666e0 con 0x7f5bf00b2c70 2020-11-20T12:21:25.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.645+0000 7f5becff9700 1 -- 172.21.15.187:0/999331378 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+2145 (secure 0 0 0) 0x7f5be8048790 con 0x7f5bf00b2c70 2020-11-20T12:21:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:25.648 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":171,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:25.085806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:21:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5bcc056d20 msgr2=0x7f5bcc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5bcc056d20 0x7f5bcc0591e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f5be0003c00 tx=0x7f5be0009040).stop 2020-11-20T12:21:25.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf00b2c70 msgr2=0x7f5bf0154ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf00b2c70 0x7f5bf0154ac0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7f5bf0067de0 tx=0x7f5be8008040).stop 2020-11-20T12:21:25.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 shutdown_connections 2020-11-20T12:21:25.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5bcc056d20 0x7f5bcc0591e0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bf00b3750 0x7f5bf0153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:25.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf00b2c70 0x7f5bf0154ac0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 --2- 172.21.15.187:0/999331378 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bf00b3b30 0x7f5bf01545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:25.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 >> 172.21.15.187:0/999331378 conn(0x7f5bf00bacd0 msgr2=0x7f5bf00b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:25.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 shutdown_connections 2020-11-20T12:21:25.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.648+0000 7f5bf62be700 1 -- 172.21.15.187:0/999331378 wait complete. 2020-11-20T12:21:25.654 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 171 2020-11-20T12:21:25.675 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:21:25.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.921+0000 7f58023c9700 1 Processor -- start 2020-11-20T12:21:25.924 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.922+0000 7f58023c9700 1 -- start start 2020-11-20T12:21:25.925 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.923+0000 7f58023c9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f57fc065f90 0x7f57fc065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.923+0000 7f58023c9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 0x7f57fc09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.923+0000 7f58023c9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57fc0b13e0 con 0x7f57fc065190 2020-11-20T12:21:25.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.923+0000 7f58023c9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57fc14ab30 con 0x7f57fc065f90 2020-11-20T12:21:25.926 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.923+0000 7f58023c9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc062c10 con 0x7f57fc09ca90 2020-11-20T12:21:25.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.924+0000 7f5801bc8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 0x7f57fc09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.927 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.924+0000 7f58013c7700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f57fc065190 0x7f57fc143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:44638/0 (socket says 172.21.15.187:44638) 2020-11-20T12:21:25.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.924+0000 7f58013c7700 1 -- 172.21.15.187:0/3804993972 learned_addr learned my addr 172.21.15.187:0/3804993972 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:25.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.924+0000 7f5801bc8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 0x7f57fc09cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.187:55414/0 (socket says 172.21.15.187:55414) 2020-11-20T12:21:25.928 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.924+0000 7f5800bc6700 1 --2- 172.21.15.187:0/3804993972 >> v2:172.21.15.5:3300/0 conn(0x7f57fc065f90 0x7f57fc065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.929 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.925+0000 7f57ebfff700 1 -- 172.21.15.187:0/3804993972 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477315681 0 0) 0x7f57fc0b13e0 con 0x7f57fc065190 2020-11-20T12:21:25.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.925+0000 7f57ebfff700 1 -- 172.21.15.187:0/3804993972 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e4003240 con 0x7f57fc065190 2020-11-20T12:21:25.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.925+0000 7f5800bc6700 1 -- 172.21.15.187:0/3804993972 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 msgr2=0x7f57fc09cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:25.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.925+0000 7f5800bc6700 1 --2- 172.21.15.187:0/3804993972 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 0x7f57fc09cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.925+0000 7f5800bc6700 1 -- 172.21.15.187:0/3804993972 >> v1:172.21.15.5:6789/0 conn(0x7f57fc065190 legacy=0x7f57fc143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.926+0000 7f5800bc6700 1 -- 172.21.15.187:0/3804993972 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc0b1640 con 0x7f57fc065f90 2020-11-20T12:21:25.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.926+0000 7f5800bc6700 1 --2- 172.21.15.187:0/3804993972 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065f90 0x7f57fc065570 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f57ec00b9a0 tx=0x7f57ec006df0).ready entity=mon.0 client_cookie=3651f6dba5169f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:25.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.927+0000 7f57ebfff700 1 -- 172.21.15.187:0/3804993972 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec009710 con 0x7f57fc065f90 2020-11-20T12:21:25.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.927+0000 7f57ebfff700 1 -- 172.21.15.187:0/3804993972 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec009870 con 0x7f57fc065f90 2020-11-20T12:21:25.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.927+0000 7f57ebfff700 1 -- 172.21.15.187:0/3804993972 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec009bf0 con 0x7f57fc065f90 2020-11-20T12:21:25.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.927+0000 7f58023c9700 1 -- 172.21.15.187:0/3804993972 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065f90 msgr2=0x7f57fc065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.927+0000 7f58023c9700 1 --2- 172.21.15.187:0/3804993972 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065f90 0x7f57fc065570 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f57ec00b9a0 tx=0x7f57ec006df0).stop 2020-11-20T12:21:25.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.928+0000 7f58023c9700 1 -- 172.21.15.187:0/3804993972 shutdown_connections 2020-11-20T12:21:25.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.928+0000 7f58023c9700 1 --2- 172.21.15.187:0/3804993972 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 0x7f57fc09cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.928+0000 7f58023c9700 1 --2- 172.21.15.187:0/3804993972 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065f90 0x7f57fc065570 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.928+0000 7f58023c9700 1 -- 172.21.15.187:0/3804993972 >> 172.21.15.187:0/3804993972 conn(0x7f57fc0bab30 msgr2=0x7f57fc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:25.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.928+0000 7f58023c9700 1 -- 172.21.15.187:0/3804993972 shutdown_connections 2020-11-20T12:21:25.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.928+0000 7f58023c9700 1 -- 172.21.15.187:0/3804993972 wait complete. 2020-11-20T12:21:25.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.930+0000 7f58023c9700 1 Processor -- start 2020-11-20T12:21:25.934 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.930+0000 7f58023c9700 1 -- start start 2020-11-20T12:21:25.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f58023c9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065190 0x7f57fc0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.936 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f58023c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57fc065f90 0x7f57fc146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f58023c9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 0x7f57fc067560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f58023c9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0b4680 con 0x7f57fc065190 2020-11-20T12:21:25.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f58023c9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0b4860 con 0x7f57fc065f90 2020-11-20T12:21:25.937 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f58023c9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc09efd0 con 0x7f57fc09ca90 2020-11-20T12:21:25.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f5800bc6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57fc065f90 0x7f57fc146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f5800bc6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57fc065f90 0x7f57fc146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54530/0 (socket says 172.21.15.187:54530) 2020-11-20T12:21:25.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.931+0000 7f5800bc6700 1 -- 172.21.15.187:0/1780563315 learned_addr learned my addr 172.21.15.187:0/1780563315 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:25.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f58013c7700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065190 0x7f57fc0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f5800bc6700 1 -- 172.21.15.187:0/1780563315 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 msgr2=0x7f57fc067560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f5800bc6700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 0x7f57fc067560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:25.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f5800bc6700 1 -- 172.21.15.187:0/1780563315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065190 msgr2=0x7f57fc0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:25.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f5800bc6700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065190 0x7f57fc0b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:25.939 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f5800bc6700 1 -- 172.21.15.187:0/1780563315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc09f270 con 0x7f57fc065f90 2020-11-20T12:21:25.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f5800bc6700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57fc065f90 0x7f57fc146680 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f57ec006e30 tx=0x7f57ec015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:25.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f57e97fa700 1 -- 172.21.15.187:0/1780563315 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec01d5d0 con 0x7f57fc065f90 2020-11-20T12:21:25.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f57e97fa700 1 -- 172.21.15.187:0/1780563315 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec01d730 con 0x7f57fc065f90 2020-11-20T12:21:25.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.932+0000 7f57e97fa700 1 -- 172.21.15.187:0/1780563315 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec01dad0 con 0x7f57fc065f90 2020-11-20T12:21:25.940 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.933+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc146b70 con 0x7f57fc065f90 2020-11-20T12:21:25.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.933+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc143590 con 0x7f57fc065f90 2020-11-20T12:21:25.941 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.934+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57fc09f400 con 0x7f57fc065f90 2020-11-20T12:21:25.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.935+0000 7f57e97fa700 1 -- 172.21.15.187:0/1780563315 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f57ec01a040 con 0x7f57fc065f90 2020-11-20T12:21:25.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.936+0000 7f57e97fa700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57d4056d20 0x7f57d40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:25.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.936+0000 7f57e97fa700 1 -- 172.21.15.187:0/1780563315 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f57ec071690 con 0x7f57fc065f90 2020-11-20T12:21:25.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.936+0000 7f58013c7700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57d4056d20 0x7f57d40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:25.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.937+0000 7f58013c7700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57d4056d20 0x7f57d40591e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f57f8005920 tx=0x7f57f8009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:25.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:25.940+0000 7f57e97fa700 1 -- 172.21.15.187:0/1780563315 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f57ec046ae0 con 0x7f57fc065f90 2020-11-20T12:21:26.084 INFO:tasks.ceph.mds.a.smithi187.stderr:2020-11-20T12:21:26.082+0000 7fdf5d747700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:21:26.132 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.130+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57fc09ee00 con 0x7f57fc065f90 2020-11-20T12:21:26.134 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.132+0000 7f57e97fa700 1 -- 172.21.15.187:0/1780563315 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f57ec023020 con 0x7f57fc065f90 2020-11-20T12:21:26.135 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:26.135 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":159,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:21:23.651529+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:21:20.315283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2020-11-20T12:21:21.644244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:21:26.137 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57d4056d20 msgr2=0x7f57d40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57d4056d20 0x7f57d40591e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f57f8005920 tx=0x7f57f8009040).stop 2020-11-20T12:21:26.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57fc065f90 msgr2=0x7f57fc146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.138 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57fc065f90 0x7f57fc146680 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f57ec006e30 tx=0x7f57ec015000).stop 2020-11-20T12:21:26.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 shutdown_connections 2020-11-20T12:21:26.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f57d4056d20 0x7f57d40591e0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57fc065190 0x7f57fc0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57fc09ca90 0x7f57fc067560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:26.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 --2- 172.21.15.187:0/1780563315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57fc065f90 0x7f57fc146680 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.134+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 >> 172.21.15.187:0/1780563315 conn(0x7f57fc0bab30 msgr2=0x7f57fc0b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:26.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.135+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 shutdown_connections 2020-11-20T12:21:26.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.135+0000 7f58023c9700 1 -- 172.21.15.187:0/1780563315 wait complete. 2020-11-20T12:21:26.161 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2020-11-20T12:21:26.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.415+0000 7fad792ff700 1 Processor -- start 2020-11-20T12:21:26.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.415+0000 7fad792ff700 1 -- start start 2020-11-20T12:21:26.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.417+0000 7fad792ff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7405bd20 0x7fad740b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.417+0000 7fad792ff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad74148cb0 con 0x7fad740b5e80 2020-11-20T12:21:26.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.417+0000 7fad792ff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad74148ee0 con 0x7fad740b8590 2020-11-20T12:21:26.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.417+0000 7fad792ff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad740b2230 con 0x7fad7405bd20 2020-11-20T12:21:26.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.417+0000 7fad78afe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fad740b8590 0x7fad741456e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51146/0 (socket says 172.21.15.187:51146) 2020-11-20T12:21:26.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.417+0000 7fad78afe700 1 -- 172.21.15.187:0/3390546123 learned_addr learned my addr 172.21.15.187:0/3390546123 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:26.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.418+0000 7fad6b7fe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7405bd20 0x7fad740b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.418+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792885358 0 0) 0x7fad74148ee0 con 0x7fad740b8590 2020-11-20T12:21:26.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.418+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad54003240 con 0x7fad740b8590 2020-11-20T12:21:26.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 124838333 0 0) 0x7fad54003240 con 0x7fad740b8590 2020-11-20T12:21:26.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7405bd20 msgr2=0x7fad740b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:26.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7405bd20 0x7fad740b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 >> v1:172.21.15.5:6789/0 conn(0x7fad740b5e80 legacy=0x7fad74149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad74149180 con 0x7fad740b8590 2020-11-20T12:21:26.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fad6c001860 con 0x7fad740b8590 2020-11-20T12:21:26.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 >> v1:172.21.15.187:6789/0 conn(0x7fad740b8590 legacy=0x7fad741456e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7405bd20 0x7fad54003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad54003c20 0x7fad540044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad540049d0 0x7fad54004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740b2230 con 0x7fad54003c20 2020-11-20T12:21:26.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad54004140 con 0x7fad540049d0 2020-11-20T12:21:26.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.419+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad54004300 con 0x7fad7405bd20 2020-11-20T12:21:26.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.420+0000 7fad78afe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad540049d0 0x7fad54004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.420+0000 7fad6b7fe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad54003c20 0x7fad540044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.420+0000 7fad78afe700 1 -- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7405bd20 msgr2=0x7fad54003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:26.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.420+0000 7fad78afe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7405bd20 0x7fad54003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:26.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.420+0000 7fad78afe700 1 -- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad54003c20 msgr2=0x7fad540044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:26.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.420+0000 7fad78afe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad54003c20 0x7fad540044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.420+0000 7fad78afe700 1 -- 172.21.15.187:0/3390546123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad54003240 con 0x7fad540049d0 2020-11-20T12:21:26.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.420+0000 7fad6b7fe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad54003c20 0x7fad540044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:26.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.421+0000 7fad78afe700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad540049d0 0x7fad54004e50 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7fad6c00c840 tx=0x7fad6c007b50).ready entity=mon.1 client_cookie=ba20c2ef113b7232 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:26.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.421+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c005b80 con 0x7fad540049d0 2020-11-20T12:21:26.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.421+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c005ce0 con 0x7fad540049d0 2020-11-20T12:21:26.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.421+0000 7fad737fe700 1 -- 172.21.15.187:0/3390546123 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c007360 con 0x7fad540049d0 2020-11-20T12:21:26.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.421+0000 7fad792ff700 1 -- 172.21.15.187:0/3390546123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad540049d0 msgr2=0x7fad54004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.421+0000 7fad792ff700 1 --2- 172.21.15.187:0/3390546123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad540049d0 0x7fad54004e50 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7fad6c00c840 tx=0x7fad6c007b50).stop 2020-11-20T12:21:26.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.421+0000 7fad78afe700 1 -- 172.21.15.187:0/3390546123 reap_dead start 2020-11-20T12:21:26.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.422+0000 7fad792ff700 1 -- 172.21.15.187:0/3390546123 shutdown_connections 2020-11-20T12:21:26.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.422+0000 7fad792ff700 1 -- 172.21.15.187:0/3390546123 >> 172.21.15.187:0/3390546123 conn(0x7fad740bbeb0 msgr2=0x7fad740b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:26.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.422+0000 7fad792ff700 1 -- 172.21.15.187:0/3390546123 shutdown_connections 2020-11-20T12:21:26.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.422+0000 7fad792ff700 1 -- 172.21.15.187:0/3390546123 wait complete. 2020-11-20T12:21:26.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.423+0000 7fad792ff700 1 Processor -- start 2020-11-20T12:21:26.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.423+0000 7fad792ff700 1 -- start start 2020-11-20T12:21:26.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.424+0000 7fad792ff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad7409f4c0 0x7fad740a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.424+0000 7fad792ff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad74148390 0x7fad740b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.424+0000 7fad792ff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7409e810 0x7fad74153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.424+0000 7fad792ff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740b3f90 con 0x7fad74148390 2020-11-20T12:21:26.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.424+0000 7fad792ff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740b4230 con 0x7fad7409f4c0 2020-11-20T12:21:26.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.424+0000 7fad792ff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7409f8a0 con 0x7fad7409e810 2020-11-20T12:21:26.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad7409f4c0 0x7fad740a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad7409f4c0 0x7fad740a1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54548/0 (socket says 172.21.15.187:54548) 2020-11-20T12:21:26.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 -- 172.21.15.187:0/901341396 learned_addr learned my addr 172.21.15.187:0/901341396 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:26.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad78afe700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7409e810 0x7fad74153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 -- 172.21.15.187:0/901341396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7409e810 msgr2=0x7fad74153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad6b7fe700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad74148390 0x7fad740b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7409e810 0x7fad74153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 -- 172.21.15.187:0/901341396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad74148390 msgr2=0x7fad740b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad74148390 0x7fad740b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 -- 172.21.15.187:0/901341396 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad740b34a0 con 0x7fad7409f4c0 2020-11-20T12:21:26.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.425+0000 7fad73fff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad7409f4c0 0x7fad740a1000 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7fad6400c880 tx=0x7fad64007bf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:26.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.426+0000 7fad70ff9700 1 -- 172.21.15.187:0/901341396 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64001ea0 con 0x7fad7409f4c0 2020-11-20T12:21:26.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.426+0000 7fad70ff9700 1 -- 172.21.15.187:0/901341396 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6400a5e0 con 0x7fad7409f4c0 2020-11-20T12:21:26.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.426+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad740b3630 con 0x7fad7409f4c0 2020-11-20T12:21:26.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.426+0000 7fad70ff9700 1 -- 172.21.15.187:0/901341396 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6400a9a0 con 0x7fad7409f4c0 2020-11-20T12:21:26.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.426+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad741548e0 con 0x7fad7409f4c0 2020-11-20T12:21:26.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.427+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad740b43f0 con 0x7fad7409f4c0 2020-11-20T12:21:26.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.427+0000 7fad70ff9700 1 -- 172.21.15.187:0/901341396 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fad64010070 con 0x7fad7409f4c0 2020-11-20T12:21:26.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.428+0000 7fad70ff9700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad50056d20 0x7fad500591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.428+0000 7fad70ff9700 1 -- 172.21.15.187:0/901341396 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fad6406f680 con 0x7fad7409f4c0 2020-11-20T12:21:26.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.428+0000 7fad6b7fe700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad50056d20 0x7fad500591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.429+0000 7fad6b7fe700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad50056d20 0x7fad500591e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fad74141780 tx=0x7fad5c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:26.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.432+0000 7fad70ff9700 1 -- 172.21.15.187:0/901341396 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fad640449c0 con 0x7fad7409f4c0 2020-11-20T12:21:26.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.686+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fad740446d0 con 0x7fad7409f4c0 2020-11-20T12:21:26.701 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.698+0000 7fad70ff9700 1 -- 172.21.15.187:0/901341396 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7fad64047fe0 con 0x7fad7409f4c0 2020-11-20T12:21:26.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.700+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad50056d20 msgr2=0x7fad500591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.700+0000 7fad792ff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad50056d20 0x7fad500591e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fad74141780 tx=0x7fad5c008040).stop 2020-11-20T12:21:26.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.700+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad7409f4c0 msgr2=0x7fad740a1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad7409f4c0 0x7fad740a1000 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7fad6400c880 tx=0x7fad64007bf0).stop 2020-11-20T12:21:26.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 shutdown_connections 2020-11-20T12:21:26.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fad50056d20 0x7fad500591e0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad7409e810 0x7fad74153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad7409f4c0 0x7fad740a1000 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 --2- 172.21.15.187:0/901341396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad74148390 0x7fad740b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 >> 172.21.15.187:0/901341396 conn(0x7fad740bbeb0 msgr2=0x7fad74062190 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:26.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 shutdown_connections 2020-11-20T12:21:26.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.701+0000 7fad792ff700 1 -- 172.21.15.187:0/901341396 wait complete. 2020-11-20T12:21:26.715 INFO:teuthology.orchestra.run.smithi187.stderr:updated caps for client.0 2020-11-20T12:21:26.727 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:26.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.976+0000 7f79e0938700 1 Processor -- start 2020-11-20T12:21:26.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.977+0000 7f79e0938700 1 -- start start 2020-11-20T12:21:26.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.978+0000 7f79e0938700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.978+0000 7f79e0938700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f79dc0b5f50 0x7f79dc0b6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.978+0000 7f79e0938700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc144b80 con 0x7f79dc1482d0 2020-11-20T12:21:26.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.978+0000 7f79e0938700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc1489e0 con 0x7f79dc0b5f50 2020-11-20T12:21:26.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.978+0000 7f79e0938700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc148c80 con 0x7f79dc0b57b0 2020-11-20T12:21:26.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79db59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f79dc0b5f50 0x7f79dc0b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79da59c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f79dc1482d0 0x7f79dc062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51164/0 (socket says 172.21.15.187:51164) 2020-11-20T12:21:26.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79da59c700 1 -- 172.21.15.187:0/1522546673 learned_addr learned my addr 172.21.15.187:0/1522546673 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:26.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79dad9d700 1 --2- 172.21.15.187:0/1522546673 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79db59e700 1 -- 172.21.15.187:0/1522546673 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 msgr2=0x7f79dc068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:26.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79db59e700 1 --2- 172.21.15.187:0/1522546673 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79db59e700 1 -- 172.21.15.187:0/1522546673 >> v1:172.21.15.187:6789/0 conn(0x7f79dc1482d0 legacy=0x7f79dc062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79d9d9b700 1 -- 172.21.15.187:0/1522546673 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744338697 0 0) 0x7f79dc144b80 con 0x7f79dc1482d0 2020-11-20T12:21:26.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.979+0000 7f79db59e700 1 -- 172.21.15.187:0/1522546673 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc148f20 con 0x7f79dc0b5f50 2020-11-20T12:21:26.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.980+0000 7f79db59e700 1 --2- 172.21.15.187:0/1522546673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc0b5f50 0x7f79dc0b6400 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 rx=0x7f79d4009c90 tx=0x7f79d40091d0).ready entity=mon.1 client_cookie=6a9a4441c186f506 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:26.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.980+0000 7f79d9d9b700 1 -- 172.21.15.187:0/1522546673 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d400bbc0 con 0x7f79dc0b5f50 2020-11-20T12:21:26.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.980+0000 7f79d9d9b700 1 -- 172.21.15.187:0/1522546673 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d400bd20 con 0x7f79dc0b5f50 2020-11-20T12:21:26.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.980+0000 7f79d9d9b700 1 -- 172.21.15.187:0/1522546673 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4010680 con 0x7f79dc0b5f50 2020-11-20T12:21:26.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.981+0000 7f79e0938700 1 -- 172.21.15.187:0/1522546673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc0b5f50 msgr2=0x7f79dc0b6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.981+0000 7f79e0938700 1 --2- 172.21.15.187:0/1522546673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc0b5f50 0x7f79dc0b6400 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 rx=0x7f79d4009c90 tx=0x7f79d40091d0).stop 2020-11-20T12:21:26.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.981+0000 7f79e0938700 1 -- 172.21.15.187:0/1522546673 shutdown_connections 2020-11-20T12:21:26.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.981+0000 7f79e0938700 1 --2- 172.21.15.187:0/1522546673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc0b5f50 0x7f79dc0b6400 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.981+0000 7f79e0938700 1 --2- 172.21.15.187:0/1522546673 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.981+0000 7f79e0938700 1 -- 172.21.15.187:0/1522546673 >> 172.21.15.187:0/1522546673 conn(0x7f79dc0bab30 msgr2=0x7f79dc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:26.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.981+0000 7f79e0938700 1 -- 172.21.15.187:0/1522546673 shutdown_connections 2020-11-20T12:21:26.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.981+0000 7f79e0938700 1 -- 172.21.15.187:0/1522546673 wait complete. 2020-11-20T12:21:26.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.983+0000 7f79e0938700 1 Processor -- start 2020-11-20T12:21:26.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.983+0000 7f79e0938700 1 -- start start 2020-11-20T12:21:26.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79e0938700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc1590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79e0938700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc0b5f50 0x7f79dc09f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79e0938700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc1482d0 0x7f79dc15aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79e0938700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc094870 con 0x7f79dc0b5f50 2020-11-20T12:21:26.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79e0938700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc158cd0 con 0x7f79dc1482d0 2020-11-20T12:21:26.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79e0938700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc094a60 con 0x7f79dc0b57b0 2020-11-20T12:21:26.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79db59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc1482d0 0x7f79dc15aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79db59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc1482d0 0x7f79dc15aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54566/0 (socket says 172.21.15.187:54566) 2020-11-20T12:21:26.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.984+0000 7f79db59e700 1 -- 172.21.15.187:0/3454641916 learned_addr learned my addr 172.21.15.187:0/3454641916 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:26.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79dad9d700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc1590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79db59e700 1 -- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 msgr2=0x7f79dc1590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79da59c700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc0b5f50 0x7f79dc09f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79db59e700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc1590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79db59e700 1 -- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc0b5f50 msgr2=0x7f79dc09f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:26.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79db59e700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc0b5f50 0x7f79dc09f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:26.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79db59e700 1 -- 172.21.15.187:0/3454641916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc1589d0 con 0x7f79dc1482d0 2020-11-20T12:21:26.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79dad9d700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc1590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:26.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.985+0000 7f79db59e700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc1482d0 0x7f79dc15aad0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f79d4009f80 tx=0x7f79d4018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:26.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.986+0000 7f79c2ffd700 1 -- 172.21.15.187:0/3454641916 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d401d070 con 0x7f79dc1482d0 2020-11-20T12:21:26.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.986+0000 7f79c2ffd700 1 -- 172.21.15.187:0/3454641916 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d4010e50 con 0x7f79dc1482d0 2020-11-20T12:21:26.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.986+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc1595d0 con 0x7f79dc1482d0 2020-11-20T12:21:26.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.986+0000 7f79c2ffd700 1 -- 172.21.15.187:0/3454641916 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4018e00 con 0x7f79dc1482d0 2020-11-20T12:21:26.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.986+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc09fc10 con 0x7f79dc1482d0 2020-11-20T12:21:26.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.987+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc094c20 con 0x7f79dc1482d0 2020-11-20T12:21:26.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.988+0000 7f79c2ffd700 1 -- 172.21.15.187:0/3454641916 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f79d4022630 con 0x7f79dc1482d0 2020-11-20T12:21:26.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.988+0000 7f79c2ffd700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f79b8056d20 0x7f79b80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:26.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.988+0000 7f79c2ffd700 1 -- 172.21.15.187:0/3454641916 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f79d4071db0 con 0x7f79dc1482d0 2020-11-20T12:21:26.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.989+0000 7f79dad9d700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f79b8056d20 0x7f79b80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:26.997 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.989+0000 7f79dad9d700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f79b8056d20 0x7f79b80591e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f79dc0b8ce0 tx=0x7f79cc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:26.998 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:26.992+0000 7f79c2ffd700 1 -- 172.21.15.187:0/3454641916 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f79d4047200 con 0x7f79dc1482d0 2020-11-20T12:21:27.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.242+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79dc061a20 con 0x7f79dc1482d0 2020-11-20T12:21:27.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.244+0000 7f79c2ffd700 1 -- 172.21.15.187:0/3454641916 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+2145 (secure 0 0 0) 0x7f79d4027030 con 0x7f79dc1482d0 2020-11-20T12:21:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:27.246 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":171,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:25.085806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:21:27.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f79b8056d20 msgr2=0x7f79b80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:27.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f79b8056d20 0x7f79b80591e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f79dc0b8ce0 tx=0x7f79cc008040).stop 2020-11-20T12:21:27.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc1482d0 msgr2=0x7f79dc15aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:27.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc1482d0 0x7f79dc15aad0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f79d4009f80 tx=0x7f79d4018000).stop 2020-11-20T12:21:27.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 shutdown_connections 2020-11-20T12:21:27.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f79b8056d20 0x7f79b80591e0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:27.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79dc1482d0 0x7f79dc15aad0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:27.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79dc0b57b0 0x7f79dc1590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:27.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 --2- 172.21.15.187:0/3454641916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc0b5f50 0x7f79dc09f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:27.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 >> 172.21.15.187:0/3454641916 conn(0x7f79dc0bab30 msgr2=0x7f79dc146d80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:27.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.246+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 shutdown_connections 2020-11-20T12:21:27.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:27.247+0000 7f79e0938700 1 -- 172.21.15.187:0/3454641916 wait complete. 2020-11-20T12:21:27.252 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 171 2020-11-20T12:21:27.474 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 171, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:21:22.646683+0000', 'modified': '2020-11-20T12:21:25.085806+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10018}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10018': {'gid': 10018, 'name': 'a', 'rank': 0, 'incarnation': 168, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1915143367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1915143367}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1915143367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:21:27.475 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:21:27.476 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:21:27.756 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":10018,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":5.0819974720000003,"mdsmap_epoch":171,"osdmap_epoch":159,"osdmap_epoch_barrier":159,"uptime":11.43018436} 2020-11-20T12:21:27.757 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 10018, "mdsmap_epoch": 171, "osdmap_epoch": 159, "osdmap_epoch_barrier": 159, "rank_uptime": 5.081997472, "state": "up:active", "uptime": 11.43018436, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:21:27.757 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:21:27.758 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:21:27.758 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:21:27.758 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:21:27.759 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:21:27.759 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:21:27.759 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:21:27.760 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:21:27.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:27.761 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:21:27.815 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:21:27.815 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:21:27.816 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:21:27.816 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:27.817 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:21:27.819 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:27.819 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:21:27.819 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:27.819 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:21:27.820 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:27.820 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:21:27.820 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:27.820 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:21:27.822 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:27.822 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:21:27.823 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 41751sec preferred_lft 41751sec 2020-11-20T12:21:27.823 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:21:27.823 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:27.824 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:21:27.824 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:27.825 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:21:27.825 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:27.826 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:21:27.826 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:27.826 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:21:27.827 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:21:27.827 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:21:27.828 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:27.828 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:21:27.828 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:21:27.829 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:21:27.830 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:27.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:27.831 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:21:27.936 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:21:27.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:27.947 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:21:28.001 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:21:28.001 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:21:28.002 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:21:28.002 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:21:28.002 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:21:28.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:28.004 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:21:28.004 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:21:28.004 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:21:28.005 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:21:28.005 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:21:28.006 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:21:28.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:28.271 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:21:28.334 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:21:28.335 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:21:28.335 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:21:28.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:28.337 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:21:28.442 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:21:28.693 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.690+0000 7fe096fae700 1 Processor -- start 2020-11-20T12:21:28.694 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.691+0000 7fe096fae700 1 -- start start 2020-11-20T12:21:28.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.692+0000 7fe096fae700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe090066f70 0x7fe0900b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:28.696 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.692+0000 7fe096fae700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0900b1300 con 0x7fe0901484d0 2020-11-20T12:21:28.697 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.692+0000 7fe096fae700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0900b15a0 con 0x7fe0900b6650 2020-11-20T12:21:28.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.692+0000 7fe096fae700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe090067730 con 0x7fe090066f70 2020-11-20T12:21:28.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.692+0000 7fe0957ab700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe0900b6650 0x7fe0900afa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51178/0 (socket says 172.21.15.187:51178) 2020-11-20T12:21:28.698 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.692+0000 7fe0957ab700 1 -- 172.21.15.187:0/3585788949 learned_addr learned my addr 172.21.15.187:0/3585788949 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:28.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.693+0000 7fe0967ad700 1 --2- 172.21.15.187:0/3585788949 >> v2:172.21.15.5:3300/0 conn(0x7fe090066f70 0x7fe0900b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:28.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.693+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055955851 0 0) 0x7fe0900b15a0 con 0x7fe0900b6650 2020-11-20T12:21:28.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.693+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe078003240 con 0x7fe0900b6650 2020-11-20T12:21:28.699 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.693+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3313111292 0 0) 0x7fe078003240 con 0x7fe0900b6650 2020-11-20T12:21:28.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 >> v2:172.21.15.5:3300/0 conn(0x7fe090066f70 msgr2=0x7fe0900b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:28.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 --2- 172.21.15.187:0/3585788949 >> v2:172.21.15.5:3300/0 conn(0x7fe090066f70 0x7fe0900b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:28.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 >> v1:172.21.15.5:6789/0 conn(0x7fe0901484d0 legacy=0x7fe090061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:28.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0900b1840 con 0x7fe0900b6650 2020-11-20T12:21:28.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fe0840014e0 con 0x7fe0900b6650 2020-11-20T12:21:28.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 >> v1:172.21.15.187:6789/0 conn(0x7fe0900b6650 legacy=0x7fe0900afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:28.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe078003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:28.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0780044e0 0x7fe078004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:28.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe078004e50 0x7fe07800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:28.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe090067730 con 0x7fe0780044e0 2020-11-20T12:21:28.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe078004260 con 0x7fe078003730 2020-11-20T12:21:28.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07800bc20 con 0x7fe078004e50 2020-11-20T12:21:28.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.694+0000 7fe095fac700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe078003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:28.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe095fac700 1 -- 172.21.15.187:0/3585788949 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe078004e50 msgr2=0x7fe07800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:28.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe095fac700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe078004e50 0x7fe07800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:28.707 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe095fac700 1 -- 172.21.15.187:0/3585788949 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0780044e0 msgr2=0x7fe078004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:28.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe0957ab700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0780044e0 0x7fe078004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:28.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe095fac700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0780044e0 0x7fe078004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:28.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe095fac700 1 -- 172.21.15.187:0/3585788949 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe078003240 con 0x7fe078003730 2020-11-20T12:21:28.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe0967ad700 1 -- 172.21.15.187:0/3585788949 reap_dead start 2020-11-20T12:21:28.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe095fac700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe078003b50 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 rx=0x7fe08c001920 tx=0x7fe08c007b90).ready entity=mon.1 client_cookie=b7a55105ae42474d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:28.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c007020 con 0x7fe078003730 2020-11-20T12:21:28.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c007180 con 0x7fe078003730 2020-11-20T12:21:28.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.695+0000 7fe094faa700 1 -- 172.21.15.187:0/3585788949 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00c620 con 0x7fe078003730 2020-11-20T12:21:28.709 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.696+0000 7fe096fae700 1 -- 172.21.15.187:0/3585788949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 msgr2=0x7fe078003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:28.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.696+0000 7fe096fae700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe078003b50 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 rx=0x7fe08c001920 tx=0x7fe08c007b90).stop 2020-11-20T12:21:28.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.696+0000 7fe096fae700 1 -- 172.21.15.187:0/3585788949 shutdown_connections 2020-11-20T12:21:28.710 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.696+0000 7fe096fae700 1 --2- 172.21.15.187:0/3585788949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe078003b50 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:28.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.696+0000 7fe096fae700 1 -- 172.21.15.187:0/3585788949 >> 172.21.15.187:0/3585788949 conn(0x7fe0900bab30 msgr2=0x7fe0900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:28.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.696+0000 7fe096fae700 1 -- 172.21.15.187:0/3585788949 shutdown_connections 2020-11-20T12:21:28.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.696+0000 7fe096fae700 1 -- 172.21.15.187:0/3585788949 wait complete. 2020-11-20T12:21:28.711 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.698+0000 7fe096fae700 1 Processor -- start 2020-11-20T12:21:28.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.698+0000 7fe096fae700 1 -- start start 2020-11-20T12:21:28.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe096fae700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe0900b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:28.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe096fae700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe09009f6b0 0x7fe090131b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:28.712 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe096fae700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe09009ce20 0x7fe090130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:28.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe096fae700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09009dc60 con 0x7fe09009ce20 2020-11-20T12:21:28.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe096fae700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09009df00 con 0x7fe078003730 2020-11-20T12:21:28.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe096fae700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900b3a80 con 0x7fe09009f6b0 2020-11-20T12:21:28.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe095fac700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe0900b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:28.713 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe095fac700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe0900b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54582/0 (socket says 172.21.15.187:54582) 2020-11-20T12:21:28.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe095fac700 1 -- 172.21.15.187:0/1714668495 learned_addr learned my addr 172.21.15.187:0/1714668495 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:28.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe095fac700 1 -- 172.21.15.187:0/1714668495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe09009f6b0 msgr2=0x7fe090131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:28.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe095fac700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe09009f6b0 0x7fe090131b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:28.714 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe095fac700 1 -- 172.21.15.187:0/1714668495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe09009ce20 msgr2=0x7fe090130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:28.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe095fac700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe09009ce20 0x7fe090130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:28.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.699+0000 7fe095fac700 1 -- 172.21.15.187:0/1714668495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090131ff0 con 0x7fe078003730 2020-11-20T12:21:28.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.700+0000 7fe095fac700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe0900b3590 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7fe08c007ed0 tx=0x7fe08c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:28.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.700+0000 7fe081ffb700 1 -- 172.21.15.187:0/1714668495 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c013ae0 con 0x7fe078003730 2020-11-20T12:21:28.715 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.700+0000 7fe081ffb700 1 -- 172.21.15.187:0/1714668495 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c013c40 con 0x7fe078003730 2020-11-20T12:21:28.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.700+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09009d2a0 con 0x7fe078003730 2020-11-20T12:21:28.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.700+0000 7fe081ffb700 1 -- 172.21.15.187:0/1714668495 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c020440 con 0x7fe078003730 2020-11-20T12:21:28.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.700+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0901313a0 con 0x7fe078003730 2020-11-20T12:21:28.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.701+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09009d430 con 0x7fe078003730 2020-11-20T12:21:28.716 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.702+0000 7fe081ffb700 1 -- 172.21.15.187:0/1714668495 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe08c026020 con 0x7fe078003730 2020-11-20T12:21:28.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.702+0000 7fe081ffb700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe068056d50 0x7fe068059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:28.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.702+0000 7fe081ffb700 1 -- 172.21.15.187:0/1714668495 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe08c0716b0 con 0x7fe078003730 2020-11-20T12:21:28.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.703+0000 7fe0957ab700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe068056d50 0x7fe068059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:28.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.703+0000 7fe0957ab700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe068056d50 0x7fe068059210 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7fe084001610 tx=0x7fe084009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:28.719 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.706+0000 7fe081ffb700 1 -- 172.21.15.187:0/1714668495 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe08c046a80 con 0x7fe078003730 2020-11-20T12:21:28.893 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.890+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe090061a20 con 0x7fe078003730 2020-11-20T12:21:28.897 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.893+0000 7fe081ffb700 1 -- 172.21.15.187:0/1714668495 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+16792 (secure 0 0 0) 0x7fe08c04a0a0 con 0x7fe078003730 2020-11-20T12:21:28.898 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:28.899 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":159,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:21:23.651529+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:21:20.315283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2020-11-20T12:21:21.644244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:21:28.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe068056d50 msgr2=0x7fe068059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:28.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe068056d50 0x7fe068059210 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7fe084001610 tx=0x7fe084009040).stop 2020-11-20T12:21:28.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 msgr2=0x7fe0900b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:28.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe0900b3590 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7fe08c007ed0 tx=0x7fe08c015000).stop 2020-11-20T12:21:28.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 shutdown_connections 2020-11-20T12:21:28.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe068056d50 0x7fe068059210 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:28.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe09009ce20 0x7fe090130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:28.905 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe078003730 0x7fe0900b3590 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:28.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 --2- 172.21.15.187:0/1714668495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe09009f6b0 0x7fe090131b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:28.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.895+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 >> 172.21.15.187:0/1714668495 conn(0x7fe0900bab30 msgr2=0x7fe0900afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:28.906 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.896+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 shutdown_connections 2020-11-20T12:21:28.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:28.896+0000 7fe096fae700 1 -- 172.21.15.187:0/1714668495 wait complete. 2020-11-20T12:21:28.922 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:21:20.315283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:21:21.644244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:21:28.923 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:21:29.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.175+0000 7f315ac56700 1 Processor -- start 2020-11-20T12:21:29.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.175+0000 7f315ac56700 1 -- start start 2020-11-20T12:21:29.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.177+0000 7f315ac56700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3154060330 0x7f31540662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.177+0000 7f315ac56700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f31540676a0 0x7f31540b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.195 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.177+0000 7f315ac56700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315409b540 con 0x7f3154060710 2020-11-20T12:21:29.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.177+0000 7f315ac56700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31540b7da0 con 0x7f3154060330 2020-11-20T12:21:29.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.177+0000 7f315ac56700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540b8040 con 0x7f31540676a0 2020-11-20T12:21:29.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.177+0000 7f3159c54700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3154060330 0x7f31540662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.177+0000 7f3159c54700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3154060330 0x7f31540662a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54590/0 (socket says 172.21.15.187:54590) 2020-11-20T12:21:29.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.177+0000 7f3159c54700 1 -- 172.21.15.187:0/324615630 learned_addr learned my addr 172.21.15.187:0/324615630 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:29.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f315a455700 1 --2- 172.21.15.187:0/324615630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f31540676a0 0x7f31540b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f3159c54700 1 -- 172.21.15.187:0/324615630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f31540676a0 msgr2=0x7f31540b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:29.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f3159c54700 1 --2- 172.21.15.187:0/324615630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f31540676a0 0x7f31540b6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f3159c54700 1 -- 172.21.15.187:0/324615630 >> v1:172.21.15.5:6789/0 conn(0x7f3154060710 legacy=0x7f31541480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:29.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f3159c54700 1 -- 172.21.15.187:0/324615630 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540b82e0 con 0x7f3154060330 2020-11-20T12:21:29.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f315a455700 1 --2- 172.21.15.187:0/324615630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f31540676a0 0x7f31540b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:29.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f3159c54700 1 --2- 172.21.15.187:0/324615630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3154060330 0x7f31540662a0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f314c009d50 tx=0x7f314c009290).ready entity=mon.1 client_cookie=155a20dc4adeaccb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:29.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f3158c52700 1 -- 172.21.15.187:0/324615630 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c00bcd0 con 0x7f3154060330 2020-11-20T12:21:29.198 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f3158c52700 1 -- 172.21.15.187:0/324615630 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c00be30 con 0x7f3154060330 2020-11-20T12:21:29.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.178+0000 7f3158c52700 1 -- 172.21.15.187:0/324615630 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c011610 con 0x7f3154060330 2020-11-20T12:21:29.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.179+0000 7f315ac56700 1 -- 172.21.15.187:0/324615630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3154060330 msgr2=0x7f31540662a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.179+0000 7f315ac56700 1 --2- 172.21.15.187:0/324615630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3154060330 0x7f31540662a0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f314c009d50 tx=0x7f314c009290).stop 2020-11-20T12:21:29.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.179+0000 7f315ac56700 1 -- 172.21.15.187:0/324615630 shutdown_connections 2020-11-20T12:21:29.199 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.179+0000 7f315ac56700 1 --2- 172.21.15.187:0/324615630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f31540676a0 0x7f31540b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.179+0000 7f315ac56700 1 --2- 172.21.15.187:0/324615630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3154060330 0x7f31540662a0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.179+0000 7f315ac56700 1 -- 172.21.15.187:0/324615630 >> 172.21.15.187:0/324615630 conn(0x7f31540bbc20 msgr2=0x7f31540bc030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:29.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.179+0000 7f315ac56700 1 -- 172.21.15.187:0/324615630 shutdown_connections 2020-11-20T12:21:29.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.179+0000 7f315ac56700 1 -- 172.21.15.187:0/324615630 wait complete. 2020-11-20T12:21:29.200 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.181+0000 7f315ac56700 1 Processor -- start 2020-11-20T12:21:29.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.181+0000 7f315ac56700 1 -- start start 2020-11-20T12:21:29.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.181+0000 7f315ac56700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3154060330 0x7f3154159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.181+0000 7f315ac56700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3154060710 0x7f3154148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.201 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315ac56700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31540676a0 0x7f3154148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315ac56700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315409a5e0 con 0x7f3154060330 2020-11-20T12:21:29.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315ac56700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315409a810 con 0x7f31540676a0 2020-11-20T12:21:29.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315ac56700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315409aab0 con 0x7f3154060710 2020-11-20T12:21:29.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315a455700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31540676a0 0x7f3154148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.202 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315a455700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31540676a0 0x7f3154148e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54598/0 (socket says 172.21.15.187:54598) 2020-11-20T12:21:29.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315a455700 1 -- 172.21.15.187:0/4118158705 learned_addr learned my addr 172.21.15.187:0/4118158705 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:29.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315a455700 1 -- 172.21.15.187:0/4118158705 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3154060710 msgr2=0x7f3154148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315a455700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3154060710 0x7f3154148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:29.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315a455700 1 -- 172.21.15.187:0/4118158705 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3154060330 msgr2=0x7f3154159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.203 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f3159c54700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3154060330 0x7f3154159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315a455700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3154060330 0x7f3154159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.182+0000 7f315a455700 1 -- 172.21.15.187:0/4118158705 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154159570 con 0x7f31540676a0 2020-11-20T12:21:29.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.183+0000 7f315a455700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31540676a0 0x7f3154148e50 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 rx=0x7f314800c7d0 tx=0x7f3148008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:29.204 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.183+0000 7f3145ffb700 1 -- 172.21.15.187:0/4118158705 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314800e5d0 con 0x7f31540676a0 2020-11-20T12:21:29.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.183+0000 7f3145ffb700 1 -- 172.21.15.187:0/4118158705 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314800e730 con 0x7f31540676a0 2020-11-20T12:21:29.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.183+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3154149340 con 0x7f31540676a0 2020-11-20T12:21:29.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.183+0000 7f3145ffb700 1 -- 172.21.15.187:0/4118158705 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314800eab0 con 0x7f31540676a0 2020-11-20T12:21:29.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.183+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315408ebe0 con 0x7f31540676a0 2020-11-20T12:21:29.205 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.184+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315409ac70 con 0x7f31540676a0 2020-11-20T12:21:29.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.185+0000 7f3145ffb700 1 -- 172.21.15.187:0/4118158705 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f314801b460 con 0x7f31540676a0 2020-11-20T12:21:29.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.185+0000 7f3145ffb700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f312c056d20 0x7f312c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.186+0000 7f3145ffb700 1 -- 172.21.15.187:0/4118158705 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3148070150 con 0x7f31540676a0 2020-11-20T12:21:29.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.186+0000 7f3159c54700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f312c056d20 0x7f312c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.206 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.187+0000 7f3159c54700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f312c056d20 0x7f312c0591e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f314c009fd0 tx=0x7f314c016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:29.207 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.190+0000 7f3145ffb700 1 -- 172.21.15.187:0/4118158705 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3148045540 con 0x7f31540676a0 2020-11-20T12:21:29.383 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.380+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f31540446d0 con 0x7f31540676a0 2020-11-20T12:21:29.384 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.380+0000 7f3145ffb700 1 -- 172.21.15.187:0/4118158705 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v159) v1 ==== 110+0+10 (secure 0 0 0) 0x7f314800ee60 con 0x7f31540676a0 2020-11-20T12:21:29.384 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:21:29.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.382+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f312c056d20 msgr2=0x7f312c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.382+0000 7f315ac56700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f312c056d20 0x7f312c0591e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f314c009fd0 tx=0x7f314c016000).stop 2020-11-20T12:21:29.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.382+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31540676a0 msgr2=0x7f3154148e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.382+0000 7f315ac56700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31540676a0 0x7f3154148e50 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 rx=0x7f314800c7d0 tx=0x7f3148008040).stop 2020-11-20T12:21:29.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.383+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 shutdown_connections 2020-11-20T12:21:29.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.383+0000 7f315ac56700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f312c056d20 0x7f312c0591e0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.383+0000 7f315ac56700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3154060330 0x7f3154159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.383+0000 7f315ac56700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31540676a0 0x7f3154148e50 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.383+0000 7f315ac56700 1 --2- 172.21.15.187:0/4118158705 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3154060710 0x7f3154148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.383+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 >> 172.21.15.187:0/4118158705 conn(0x7f31540bbc20 msgr2=0x7f31540b9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:29.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.383+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 shutdown_connections 2020-11-20T12:21:29.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.383+0000 7f315ac56700 1 -- 172.21.15.187:0/4118158705 wait complete. 2020-11-20T12:21:29.409 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:21:29.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.659+0000 7f0dbffff700 1 Processor -- start 2020-11-20T12:21:29.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.659+0000 7f0dbffff700 1 -- start start 2020-11-20T12:21:29.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.660+0000 7f0dbffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0dc0145cc0 0x7f0dc0066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.660+0000 7f0dbffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0dc01487e0 0x7f0dc0148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.660+0000 7f0dbffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc014c170 con 0x7f0dc00683f0 2020-11-20T12:21:29.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.660+0000 7f0dbffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc014c410 con 0x7f0dc01487e0 2020-11-20T12:21:29.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.660+0000 7f0dbffff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc014c6b0 con 0x7f0dc0145cc0 2020-11-20T12:21:29.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.661+0000 7f0dbeffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0dc0145cc0 0x7f0dc0066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.661+0000 7f0dbeffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0dc0145cc0 0x7f0dc0066040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54604/0 (socket says 172.21.15.187:54604) 2020-11-20T12:21:29.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.661+0000 7f0dbeffd700 1 -- 172.21.15.187:0/1311884431 learned_addr learned my addr 172.21.15.187:0/1311884431 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:29.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.661+0000 7f0dbeffd700 1 -- 172.21.15.187:0/1311884431 >> v2:172.21.15.5:3300/0 conn(0x7f0dc01487e0 msgr2=0x7f0dc0148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:29.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.661+0000 7f0dbeffd700 1 --2- 172.21.15.187:0/1311884431 >> v2:172.21.15.5:3300/0 conn(0x7f0dc01487e0 0x7f0dc0148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:29.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.661+0000 7f0dbeffd700 1 -- 172.21.15.187:0/1311884431 >> v1:172.21.15.5:6789/0 conn(0x7f0dc00683f0 legacy=0x7f0dc0065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:29.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.661+0000 7f0dbeffd700 1 -- 172.21.15.187:0/1311884431 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc00658a0 con 0x7f0dc0145cc0 2020-11-20T12:21:29.667 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.661+0000 7f0dbeffd700 1 --2- 172.21.15.187:0/1311884431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc0145cc0 0x7f0dc0066040 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f0dc00b85e0 tx=0x7f0db0006eb0).ready entity=mon.1 client_cookie=2823444ba6250ccb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:29.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.662+0000 7f0dbdffb700 1 -- 172.21.15.187:0/1311884431 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0009890 con 0x7f0dc0145cc0 2020-11-20T12:21:29.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.662+0000 7f0dbdffb700 1 -- 172.21.15.187:0/1311884431 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db00099f0 con 0x7f0dc0145cc0 2020-11-20T12:21:29.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.662+0000 7f0dbdffb700 1 -- 172.21.15.187:0/1311884431 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0009d70 con 0x7f0dc0145cc0 2020-11-20T12:21:29.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.662+0000 7f0dbffff700 1 -- 172.21.15.187:0/1311884431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc0145cc0 msgr2=0x7f0dc0066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.662+0000 7f0dbffff700 1 --2- 172.21.15.187:0/1311884431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc0145cc0 0x7f0dc0066040 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f0dc00b85e0 tx=0x7f0db0006eb0).stop 2020-11-20T12:21:29.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.662+0000 7f0dbffff700 1 -- 172.21.15.187:0/1311884431 shutdown_connections 2020-11-20T12:21:29.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.663+0000 7f0dbffff700 1 --2- 172.21.15.187:0/1311884431 >> v2:172.21.15.5:3300/0 conn(0x7f0dc01487e0 0x7f0dc0148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:29.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.663+0000 7f0dbffff700 1 --2- 172.21.15.187:0/1311884431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc0145cc0 0x7f0dc0066040 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.663+0000 7f0dbffff700 1 -- 172.21.15.187:0/1311884431 >> 172.21.15.187:0/1311884431 conn(0x7f0dc00bab30 msgr2=0x7f0dc00baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:29.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.663+0000 7f0dbffff700 1 -- 172.21.15.187:0/1311884431 shutdown_connections 2020-11-20T12:21:29.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.663+0000 7f0dbffff700 1 -- 172.21.15.187:0/1311884431 wait complete. 2020-11-20T12:21:29.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.664+0000 7f0dbffff700 1 Processor -- start 2020-11-20T12:21:29.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.664+0000 7f0dbffff700 1 -- start start 2020-11-20T12:21:29.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.665+0000 7f0dbffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0dc00683f0 0x7f0dc009e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.665+0000 7f0dbffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0dc0145cc0 0x7f0dc009e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.665+0000 7f0dbffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc01487e0 0x7f0dc015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.665+0000 7f0dbffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc009f130 con 0x7f0dc0145cc0 2020-11-20T12:21:29.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.665+0000 7f0dbffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc009f3d0 con 0x7f0dc01487e0 2020-11-20T12:21:29.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.665+0000 7f0dbffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc014a670 con 0x7f0dc00683f0 2020-11-20T12:21:29.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbf7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc01487e0 0x7f0dc015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbf7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc01487e0 0x7f0dc015b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54614/0 (socket says 172.21.15.187:54614) 2020-11-20T12:21:29.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbf7fe700 1 -- 172.21.15.187:0/2538178455 learned_addr learned my addr 172.21.15.187:0/2538178455 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:29.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbeffd700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0dc00683f0 0x7f0dc009e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbe7fc700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0dc0145cc0 0x7f0dc009e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbf7fe700 1 -- 172.21.15.187:0/2538178455 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0dc00683f0 msgr2=0x7f0dc009e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbf7fe700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0dc00683f0 0x7f0dc009e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbf7fe700 1 -- 172.21.15.187:0/2538178455 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0dc0145cc0 msgr2=0x7f0dc009e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbf7fe700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0dc0145cc0 0x7f0dc009e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.666+0000 7f0dbf7fe700 1 -- 172.21.15.187:0/2538178455 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc009ed10 con 0x7f0dc01487e0 2020-11-20T12:21:29.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.667+0000 7f0dbf7fe700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc01487e0 0x7f0dc015b970 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f0db800bae0 tx=0x7f0db80091a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:29.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.667+0000 7f0da6ffd700 1 -- 172.21.15.187:0/2538178455 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db800c890 con 0x7f0dc01487e0 2020-11-20T12:21:29.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.667+0000 7f0da6ffd700 1 -- 172.21.15.187:0/2538178455 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db800c9f0 con 0x7f0dc01487e0 2020-11-20T12:21:29.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.667+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc014b000 con 0x7f0dc01487e0 2020-11-20T12:21:29.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.667+0000 7f0da6ffd700 1 -- 172.21.15.187:0/2538178455 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db800cba0 con 0x7f0dc01487e0 2020-11-20T12:21:29.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.667+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc014ad50 con 0x7f0dc01487e0 2020-11-20T12:21:29.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.668+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc014a830 con 0x7f0dc01487e0 2020-11-20T12:21:29.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.669+0000 7f0da6ffd700 1 -- 172.21.15.187:0/2538178455 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0db801c070 con 0x7f0dc01487e0 2020-11-20T12:21:29.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.669+0000 7f0da6ffd700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d9c056d20 0x7f0d9c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:29.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.670+0000 7f0da6ffd700 1 -- 172.21.15.187:0/2538178455 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0db806f760 con 0x7f0dc01487e0 2020-11-20T12:21:29.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.670+0000 7f0dbeffd700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d9c056d20 0x7f0d9c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:29.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.670+0000 7f0dbeffd700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d9c056d20 0x7f0d9c0591e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f0db0006ef0 tx=0x7f0db0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:29.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.673+0000 7f0da6ffd700 1 -- 172.21.15.187:0/2538178455 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0db8044bb0 con 0x7f0dc01487e0 2020-11-20T12:21:29.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.863+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0dc009ef60 con 0x7f0dc01487e0 2020-11-20T12:21:29.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.864+0000 7f0da6ffd700 1 -- 172.21.15.187:0/2538178455 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v159) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0db80481d0 con 0x7f0dc01487e0 2020-11-20T12:21:29.866 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:29.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.866+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d9c056d20 msgr2=0x7f0d9c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.866+0000 7f0dbffff700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d9c056d20 0x7f0d9c0591e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f0db0006ef0 tx=0x7f0db0015000).stop 2020-11-20T12:21:29.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.866+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc01487e0 msgr2=0x7f0dc015b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:29.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.866+0000 7f0dbffff700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc01487e0 0x7f0dc015b970 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f0db800bae0 tx=0x7f0db80091a0).stop 2020-11-20T12:21:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.867+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 shutdown_connections 2020-11-20T12:21:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.867+0000 7f0dbffff700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f0d9c056d20 0x7f0d9c0591e0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.867+0000 7f0dbffff700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc01487e0 0x7f0dc015b970 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.867+0000 7f0dbffff700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0dc00683f0 0x7f0dc009e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.867+0000 7f0dbffff700 1 --2- 172.21.15.187:0/2538178455 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0dc0145cc0 0x7f0dc009e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:29.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.867+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 >> 172.21.15.187:0/2538178455 conn(0x7f0dc00bab30 msgr2=0x7f0dc00b8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:29.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.867+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 shutdown_connections 2020-11-20T12:21:29.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:29.867+0000 7f0dbffff700 1 -- 172.21.15.187:0/2538178455 wait complete. 2020-11-20T12:21:29.891 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:21:30.139 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.137+0000 7f9e23fff700 1 Processor -- start 2020-11-20T12:21:30.140 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.137+0000 7f9e23fff700 1 -- start start 2020-11-20T12:21:30.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.139+0000 7f9e23fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9e240b1c20 0x7f9e240b8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.141 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.139+0000 7f9e23fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b8a80 0x7f9e24066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.139+0000 7f9e23fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e24149790 con 0x7f9e240b1840 2020-11-20T12:21:30.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.139+0000 7f9e23fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e24065ba0 con 0x7f9e240b1c20 2020-11-20T12:21:30.142 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.139+0000 7f9e23fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240b13f0 con 0x7f9e240b8a80 2020-11-20T12:21:30.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.139+0000 7f9e227fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9e240b1c20 0x7f9e240b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.139+0000 7f9e22ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f9e240b1840 0x7f9e240b0ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51224/0 (socket says 172.21.15.187:51224) 2020-11-20T12:21:30.144 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.139+0000 7f9e22ffd700 1 -- 172.21.15.187:0/4286490288 learned_addr learned my addr 172.21.15.187:0/4286490288 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:30.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.140+0000 7f9e237fe700 1 --2- 172.21.15.187:0/4286490288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b8a80 0x7f9e24066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.140+0000 7f9e227fc700 1 -- 172.21.15.187:0/4286490288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b8a80 msgr2=0x7f9e24066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:30.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.140+0000 7f9e227fc700 1 --2- 172.21.15.187:0/4286490288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b8a80 0x7f9e24066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.145 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.140+0000 7f9e227fc700 1 -- 172.21.15.187:0/4286490288 >> v1:172.21.15.187:6789/0 conn(0x7f9e240b1840 legacy=0x7f9e240b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.140+0000 7f9e227fc700 1 -- 172.21.15.187:0/4286490288 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e24149450 con 0x7f9e240b1c20 2020-11-20T12:21:30.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.140+0000 7f9e227fc700 1 --2- 172.21.15.187:0/4286490288 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b1c20 0x7f9e240b8590 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f9e1800b9a0 tx=0x7f9e18006df0).ready entity=mon.1 client_cookie=3f8863bba876ccd0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e21ffb700 1 -- 172.21.15.187:0/4286490288 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18009840 con 0x7f9e240b1c20 2020-11-20T12:21:30.146 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e21ffb700 1 -- 172.21.15.187:0/4286490288 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e180099a0 con 0x7f9e240b1c20 2020-11-20T12:21:30.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e21ffb700 1 -- 172.21.15.187:0/4286490288 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18009d20 con 0x7f9e240b1c20 2020-11-20T12:21:30.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e23fff700 1 -- 172.21.15.187:0/4286490288 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b1c20 msgr2=0x7f9e240b8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e23fff700 1 --2- 172.21.15.187:0/4286490288 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b1c20 0x7f9e240b8590 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f9e1800b9a0 tx=0x7f9e18006df0).stop 2020-11-20T12:21:30.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e23fff700 1 -- 172.21.15.187:0/4286490288 shutdown_connections 2020-11-20T12:21:30.147 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e23fff700 1 --2- 172.21.15.187:0/4286490288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b8a80 0x7f9e24066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e23fff700 1 --2- 172.21.15.187:0/4286490288 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b1c20 0x7f9e240b8590 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.141+0000 7f9e23fff700 1 -- 172.21.15.187:0/4286490288 >> 172.21.15.187:0/4286490288 conn(0x7f9e240bbeb0 msgr2=0x7f9e240b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:30.148 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.142+0000 7f9e23fff700 1 -- 172.21.15.187:0/4286490288 shutdown_connections 2020-11-20T12:21:30.149 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.142+0000 7f9e23fff700 1 -- 172.21.15.187:0/4286490288 wait complete. 2020-11-20T12:21:30.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.143+0000 7f9e23fff700 1 Processor -- start 2020-11-20T12:21:30.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.143+0000 7f9e23fff700 1 -- start start 2020-11-20T12:21:30.151 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.144+0000 7f9e23fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e240b1840 0x7f9e240b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.144+0000 7f9e23fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b1c20 0x7f9e2413f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.144+0000 7f9e23fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b8a80 0x7f9e240b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.144+0000 7f9e23fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240b52f0 con 0x7f9e240b1840 2020-11-20T12:21:30.152 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.144+0000 7f9e23fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240b2d50 con 0x7f9e240b8a80 2020-11-20T12:21:30.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.144+0000 7f9e23fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240b3220 con 0x7f9e240b1c20 2020-11-20T12:21:30.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.144+0000 7f9e237fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b8a80 0x7f9e240b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e237fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b8a80 0x7f9e240b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54628/0 (socket says 172.21.15.187:54628) 2020-11-20T12:21:30.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e237fe700 1 -- 172.21.15.187:0/1896882623 learned_addr learned my addr 172.21.15.187:0/1896882623 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:30.153 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e22ffd700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e240b1840 0x7f9e240b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e227fc700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b1c20 0x7f9e2413f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e237fe700 1 -- 172.21.15.187:0/1896882623 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b1c20 msgr2=0x7f9e2413f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e237fe700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b1c20 0x7f9e2413f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.154 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e237fe700 1 -- 172.21.15.187:0/1896882623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e240b1840 msgr2=0x7f9e240b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e237fe700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e240b1840 0x7f9e240b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e237fe700 1 -- 172.21.15.187:0/1896882623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240b2f80 con 0x7f9e240b8a80 2020-11-20T12:21:30.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.145+0000 7f9e237fe700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b8a80 0x7f9e240b4600 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7f9e1c00db40 tx=0x7f9e1c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.155 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.146+0000 7f9e0affd700 1 -- 172.21.15.187:0/1896882623 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c009980 con 0x7f9e240b8a80 2020-11-20T12:21:30.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.146+0000 7f9e0affd700 1 -- 172.21.15.187:0/1896882623 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c009ae0 con 0x7f9e240b8a80 2020-11-20T12:21:30.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.146+0000 7f9e0affd700 1 -- 172.21.15.187:0/1896882623 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c009e80 con 0x7f9e240b8a80 2020-11-20T12:21:30.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.146+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e240a0ff0 con 0x7f9e240b8a80 2020-11-20T12:21:30.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.146+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e240a1490 con 0x7f9e240b8a80 2020-11-20T12:21:30.156 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.147+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e240a1180 con 0x7f9e240b8a80 2020-11-20T12:21:30.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.148+0000 7f9e0affd700 1 -- 172.21.15.187:0/1896882623 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9e1c013690 con 0x7f9e240b8a80 2020-11-20T12:21:30.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.148+0000 7f9e0affd700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e00056d20 0x7f9e000591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.148+0000 7f9e0affd700 1 -- 172.21.15.187:0/1896882623 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9e1c0705c0 con 0x7f9e240b8a80 2020-11-20T12:21:30.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.149+0000 7f9e22ffd700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e00056d20 0x7f9e000591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.149+0000 7f9e22ffd700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e00056d20 0x7f9e000591e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f9e14000bc0 tx=0x7f9e14009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.152+0000 7f9e0affd700 1 -- 172.21.15.187:0/1896882623 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9e1c0459b0 con 0x7f9e240b8a80 2020-11-20T12:21:30.341 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.339+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e24061050 con 0x7f9e240b8a80 2020-11-20T12:21:30.342 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.339+0000 7f9e0affd700 1 -- 172.21.15.187:0/1896882623 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v159) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9e1c048fd0 con 0x7f9e240b8a80 2020-11-20T12:21:30.342 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:30.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e00056d20 msgr2=0x7f9e000591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e00056d20 0x7f9e000591e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f9e14000bc0 tx=0x7f9e14009040).stop 2020-11-20T12:21:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b8a80 msgr2=0x7f9e240b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b8a80 0x7f9e240b4600 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7f9e1c00db40 tx=0x7f9e1c008040).stop 2020-11-20T12:21:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 shutdown_connections 2020-11-20T12:21:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f9e00056d20 0x7f9e000591e0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.346 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e240b1840 0x7f9e240b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e240b8a80 0x7f9e240b4600 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 --2- 172.21.15.187:0/1896882623 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e240b1c20 0x7f9e2413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.342+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 >> 172.21.15.187:0/1896882623 conn(0x7f9e240bbeb0 msgr2=0x7f9e24142020 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:30.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.343+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 shutdown_connections 2020-11-20T12:21:30.348 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.343+0000 7f9e23fff700 1 -- 172.21.15.187:0/1896882623 wait complete. 2020-11-20T12:21:30.369 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:21:30.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.610+0000 7fc115888700 1 Processor -- start 2020-11-20T12:21:30.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.610+0000 7fc115888700 1 -- start start 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.611+0000 7fc115888700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc1101484d0 0x7fc110061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.611+0000 7fc115888700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1100622c0 0x7fc1100b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.611+0000 7fc115888700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc1100b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.611+0000 7fc115888700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc110067730 con 0x7fc1101484d0 2020-11-20T12:21:30.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.611+0000 7fc115888700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100b1300 con 0x7fc1100622c0 2020-11-20T12:21:30.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.611+0000 7fc115888700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100b15a0 con 0x7fc110066f70 2020-11-20T12:21:30.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.612+0000 7fc10ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1100622c0 0x7fc1100b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.617 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.612+0000 7fc10ffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1100622c0 0x7fc1100b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54632/0 (socket says 172.21.15.187:54632) 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.612+0000 7fc10ffff700 1 -- 172.21.15.187:0/3154081684 learned_addr learned my addr 172.21.15.187:0/3154081684 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.612+0000 7fc115087700 1 --2- 172.21.15.187:0/3154081684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc1100b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.612+0000 7fc114886700 1 --2- 172.21.15.187:0/3154081684 >> v2:172.21.15.5:3300/0 conn(0x7fc1101484d0 0x7fc110061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc10ffff700 1 -- 172.21.15.187:0/3154081684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 msgr2=0x7fc1100b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc10ffff700 1 --2- 172.21.15.187:0/3154081684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc1100b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc10ffff700 1 -- 172.21.15.187:0/3154081684 >> v2:172.21.15.5:3300/0 conn(0x7fc1101484d0 msgr2=0x7fc110061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc10ffff700 1 --2- 172.21.15.187:0/3154081684 >> v2:172.21.15.5:3300/0 conn(0x7fc1101484d0 0x7fc110061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc10ffff700 1 -- 172.21.15.187:0/3154081684 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1100b1840 con 0x7fc1100622c0 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc114886700 1 --2- 172.21.15.187:0/3154081684 >> v2:172.21.15.5:3300/0 conn(0x7fc1101484d0 0x7fc110061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc10ffff700 1 --2- 172.21.15.187:0/3154081684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc1100b5640 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7fc110068600 tx=0x7fc0fc006eb0).ready entity=mon.1 client_cookie=5270982472680336 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc10f7fe700 1 -- 172.21.15.187:0/3154081684 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0098d0 con 0x7fc1100622c0 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.613+0000 7fc10f7fe700 1 -- 172.21.15.187:0/3154081684 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0fc009a30 con 0x7fc1100622c0 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc10f7fe700 1 -- 172.21.15.187:0/3154081684 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc009db0 con 0x7fc1100622c0 2020-11-20T12:21:30.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 -- 172.21.15.187:0/3154081684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 msgr2=0x7fc1100b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 --2- 172.21.15.187:0/3154081684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc1100b5640 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7fc110068600 tx=0x7fc0fc006eb0).stop 2020-11-20T12:21:30.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 -- 172.21.15.187:0/3154081684 shutdown_connections 2020-11-20T12:21:30.621 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 --2- 172.21.15.187:0/3154081684 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc1100b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 --2- 172.21.15.187:0/3154081684 >> v2:172.21.15.5:3300/0 conn(0x7fc1101484d0 0x7fc110061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 --2- 172.21.15.187:0/3154081684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc1100b5640 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 -- 172.21.15.187:0/3154081684 >> 172.21.15.187:0/3154081684 conn(0x7fc1100bab30 msgr2=0x7fc1100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 -- 172.21.15.187:0/3154081684 shutdown_connections 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.614+0000 7fc115888700 1 -- 172.21.15.187:0/3154081684 wait complete. 2020-11-20T12:21:30.625 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.616+0000 7fc115888700 1 Processor -- start 2020-11-20T12:21:30.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.616+0000 7fc115888700 1 -- start start 2020-11-20T12:21:30.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.617+0000 7fc115888700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc11009cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.617+0000 7fc115888700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc11013aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.626 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc115888700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1101484d0 0x7fc11013b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc115888700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100a05a0 con 0x7fc1101484d0 2020-11-20T12:21:30.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc115888700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11009fb70 con 0x7fc1100622c0 2020-11-20T12:21:30.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc115888700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11009fe10 con 0x7fc110066f70 2020-11-20T12:21:30.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc114886700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc11009cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc114886700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc11009cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54638/0 (socket says 172.21.15.187:54638) 2020-11-20T12:21:30.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc114886700 1 -- 172.21.15.187:0/1576667782 learned_addr learned my addr 172.21.15.187:0/1576667782 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:30.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc10ffff700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc11013aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc114886700 1 -- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 msgr2=0x7fc11013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc114886700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc11013aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc115087700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1101484d0 0x7fc11013b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc114886700 1 -- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1101484d0 msgr2=0x7fc11013b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.618+0000 7fc114886700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1101484d0 0x7fc11013b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.619+0000 7fc114886700 1 -- 172.21.15.187:0/1576667782 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1100a00b0 con 0x7fc1100622c0 2020-11-20T12:21:30.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.619+0000 7fc10ffff700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc11013aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:30.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.619+0000 7fc114886700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc11009cf90 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7fc10400c8f0 tx=0x7fc104008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.619+0000 7fc10cff9700 1 -- 172.21.15.187:0/1576667782 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1040077a0 con 0x7fc1100622c0 2020-11-20T12:21:30.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.619+0000 7fc10cff9700 1 -- 172.21.15.187:0/1576667782 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc104007900 con 0x7fc1100622c0 2020-11-20T12:21:30.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.619+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11009d480 con 0x7fc1100622c0 2020-11-20T12:21:30.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.619+0000 7fc10cff9700 1 -- 172.21.15.187:0/1576667782 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc104007c80 con 0x7fc1100622c0 2020-11-20T12:21:30.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.619+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11009b250 con 0x7fc1100622c0 2020-11-20T12:21:30.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.620+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc110061a20 con 0x7fc1100622c0 2020-11-20T12:21:30.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.621+0000 7fc10cff9700 1 -- 172.21.15.187:0/1576667782 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc104011620 con 0x7fc1100622c0 2020-11-20T12:21:30.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.622+0000 7fc10cff9700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc0e8056d20 0x7fc0e80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.622+0000 7fc10cff9700 1 -- 172.21.15.187:0/1576667782 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc1040701e0 con 0x7fc1100622c0 2020-11-20T12:21:30.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.622+0000 7fc10ffff700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc0e8056d20 0x7fc0e80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.623+0000 7fc10ffff700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc0e8056d20 0x7fc0e80591e0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fc110068600 tx=0x7fc0fc015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.626+0000 7fc10cff9700 1 -- 172.21.15.187:0/1576667782 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc104045630 con 0x7fc1100622c0 2020-11-20T12:21:30.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.813+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc1101488b0 con 0x7fc1100622c0 2020-11-20T12:21:30.816 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.814+0000 7fc10cff9700 1 -- 172.21.15.187:0/1576667782 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v159) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc104048c50 con 0x7fc1100622c0 2020-11-20T12:21:30.817 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:30.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.816+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc0e8056d20 msgr2=0x7fc0e80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.816+0000 7fc115888700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc0e8056d20 0x7fc0e80591e0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fc110068600 tx=0x7fc0fc015000).stop 2020-11-20T12:21:30.820 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.816+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 msgr2=0x7fc11009cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.816+0000 7fc115888700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc11009cf90 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7fc10400c8f0 tx=0x7fc104008040).stop 2020-11-20T12:21:30.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.817+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 shutdown_connections 2020-11-20T12:21:30.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.817+0000 7fc115888700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc0e8056d20 0x7fc0e80591e0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.821 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.817+0000 7fc115888700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc1101484d0 0x7fc11013b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.817+0000 7fc115888700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1100622c0 0x7fc11009cf90 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.817+0000 7fc115888700 1 --2- 172.21.15.187:0/1576667782 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc110066f70 0x7fc11013aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.817+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 >> 172.21.15.187:0/1576667782 conn(0x7fc1100bab30 msgr2=0x7fc110142a60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:30.822 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.817+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 shutdown_connections 2020-11-20T12:21:30.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:30.817+0000 7fc115888700 1 -- 172.21.15.187:0/1576667782 wait complete. 2020-11-20T12:21:30.841 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:21:30.842 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:31.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.076+0000 7faf0c886700 1 Processor -- start 2020-11-20T12:21:31.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.076+0000 7faf0c886700 1 -- start start 2020-11-20T12:21:31.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.078+0000 7faf0c886700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf081484d0 0x7faf08061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.078+0000 7faf0c886700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faf080622c0 0x7faf080b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.078+0000 7faf0c886700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf080b1300 con 0x7faf08066f70 2020-11-20T12:21:31.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.078+0000 7faf0c886700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf08067730 con 0x7faf081484d0 2020-11-20T12:21:31.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.078+0000 7faf0c886700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf080b15a0 con 0x7faf080622c0 2020-11-20T12:21:31.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.078+0000 7faefeffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faf080622c0 0x7faf080b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.078+0000 7faefeffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faf080622c0 0x7faf080b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54646/0 (socket says 172.21.15.187:54646) 2020-11-20T12:21:31.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.078+0000 7faefeffd700 1 -- 172.21.15.187:0/3970488962 learned_addr learned my addr 172.21.15.187:0/3970488962 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:31.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.079+0000 7faefeffd700 1 -- 172.21.15.187:0/3970488962 >> v2:172.21.15.5:3300/0 conn(0x7faf081484d0 msgr2=0x7faf08061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:31.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.079+0000 7faefeffd700 1 --2- 172.21.15.187:0/3970488962 >> v2:172.21.15.5:3300/0 conn(0x7faf081484d0 0x7faf08061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:31.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.079+0000 7faefeffd700 1 -- 172.21.15.187:0/3970488962 >> v1:172.21.15.5:6789/0 conn(0x7faf08066f70 legacy=0x7faf080b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:31.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.079+0000 7faefeffd700 1 -- 172.21.15.187:0/3970488962 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf080b1840 con 0x7faf080622c0 2020-11-20T12:21:31.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.079+0000 7faefeffd700 1 --2- 172.21.15.187:0/3970488962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf080622c0 0x7faf080b5640 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7faf08068600 tx=0x7faef0006eb0).ready entity=mon.1 client_cookie=a718f9fd9527b45e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:31.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.079+0000 7faf06ffd700 1 -- 172.21.15.187:0/3970488962 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0009890 con 0x7faf080622c0 2020-11-20T12:21:31.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.079+0000 7faf06ffd700 1 -- 172.21.15.187:0/3970488962 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef00099f0 con 0x7faf080622c0 2020-11-20T12:21:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.079+0000 7faf06ffd700 1 -- 172.21.15.187:0/3970488962 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0009d70 con 0x7faf080622c0 2020-11-20T12:21:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.080+0000 7faf0c886700 1 -- 172.21.15.187:0/3970488962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf080622c0 msgr2=0x7faf080b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.080+0000 7faf0c886700 1 --2- 172.21.15.187:0/3970488962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf080622c0 0x7faf080b5640 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7faf08068600 tx=0x7faef0006eb0).stop 2020-11-20T12:21:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.080+0000 7faf0c886700 1 -- 172.21.15.187:0/3970488962 shutdown_connections 2020-11-20T12:21:31.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.080+0000 7faf0c886700 1 --2- 172.21.15.187:0/3970488962 >> v2:172.21.15.5:3300/0 conn(0x7faf081484d0 0x7faf08061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.080+0000 7faf0c886700 1 --2- 172.21.15.187:0/3970488962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf080622c0 0x7faf080b5640 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.080+0000 7faf0c886700 1 -- 172.21.15.187:0/3970488962 >> 172.21.15.187:0/3970488962 conn(0x7faf080bab30 msgr2=0x7faf080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.080+0000 7faf0c886700 1 -- 172.21.15.187:0/3970488962 shutdown_connections 2020-11-20T12:21:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.080+0000 7faf0c886700 1 -- 172.21.15.187:0/3970488962 wait complete. 2020-11-20T12:21:31.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.082+0000 7faf0c886700 1 Processor -- start 2020-11-20T12:21:31.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.083+0000 7faf0c886700 1 -- start start 2020-11-20T12:21:31.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faf0c886700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf080622c0 0x7faf0809fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faf0c886700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf08066f70 0x7faf0809eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faf0c886700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf081484d0 0x7faf080b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faf0c886700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08145b50 con 0x7faf080622c0 2020-11-20T12:21:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faf0c886700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08145df0 con 0x7faf08066f70 2020-11-20T12:21:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faf0c886700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf08146090 con 0x7faf081484d0 2020-11-20T12:21:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faefeffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf08066f70 0x7faf0809eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faefeffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf08066f70 0x7faf0809eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54652/0 (socket says 172.21.15.187:54652) 2020-11-20T12:21:31.090 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.084+0000 7faefeffd700 1 -- 172.21.15.187:0/3435569530 learned_addr learned my addr 172.21.15.187:0/3435569530 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:31.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefeffd700 1 -- 172.21.15.187:0/3435569530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf081484d0 msgr2=0x7faf080b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefeffd700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf081484d0 0x7faf080b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:31.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefeffd700 1 -- 172.21.15.187:0/3435569530 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf080622c0 msgr2=0x7faf0809fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefeffd700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf080622c0 0x7faf0809fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:31.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefeffd700 1 -- 172.21.15.187:0/3435569530 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf080a0370 con 0x7faf08066f70 2020-11-20T12:21:31.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefeffd700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf08066f70 0x7faf0809eaa0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7faf080b16e0 tx=0x7faef0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:31.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefffff700 1 -- 172.21.15.187:0/3435569530 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0011e20 con 0x7faf08066f70 2020-11-20T12:21:31.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefffff700 1 -- 172.21.15.187:0/3435569530 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef001d5d0 con 0x7faf08066f70 2020-11-20T12:21:31.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faefffff700 1 -- 172.21.15.187:0/3435569530 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef001d970 con 0x7faf08066f70 2020-11-20T12:21:31.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf080a0560 con 0x7faf08066f70 2020-11-20T12:21:31.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.085+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf08141f60 con 0x7faf08066f70 2020-11-20T12:21:31.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.087+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf08061a20 con 0x7faf08066f70 2020-11-20T12:21:31.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.087+0000 7faefffff700 1 -- 172.21.15.187:0/3435569530 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faef001a040 con 0x7faf08066f70 2020-11-20T12:21:31.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.088+0000 7faefffff700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee4056d50 0x7faee4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.088+0000 7faefffff700 1 -- 172.21.15.187:0/3435569530 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faef0071480 con 0x7faf08066f70 2020-11-20T12:21:31.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.088+0000 7faf077fe700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee4056d50 0x7faee4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.089+0000 7faf077fe700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee4056d50 0x7faee4059210 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7faef80098b0 tx=0x7faef8006d60).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:31.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.092+0000 7faefffff700 1 -- 172.21.15.187:0/3435569530 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faef0046960 con 0x7faf08066f70 2020-11-20T12:21:31.345 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.338+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf081488b0 con 0x7faf08066f70 2020-11-20T12:21:31.347 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.340+0000 7faefffff700 1 -- 172.21.15.187:0/3435569530 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+2145 (secure 0 0 0) 0x7faef0023070 con 0x7faf08066f70 2020-11-20T12:21:31.348 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:31.350 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":171,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:25.085806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:21:31.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee4056d50 msgr2=0x7faee4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.351 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee4056d50 0x7faee4059210 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7faef80098b0 tx=0x7faef8006d60).stop 2020-11-20T12:21:31.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf08066f70 msgr2=0x7faf0809eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf08066f70 0x7faf0809eaa0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7faf080b16e0 tx=0x7faef0015000).stop 2020-11-20T12:21:31.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 shutdown_connections 2020-11-20T12:21:31.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7faee4056d50 0x7faee4059210 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.352 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf080622c0 0x7faf0809fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:31.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf081484d0 0x7faf080b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:31.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 --2- 172.21.15.187:0/3435569530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf08066f70 0x7faf0809eaa0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 >> 172.21.15.187:0/3435569530 conn(0x7faf080bab30 msgr2=0x7faf0814ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:31.353 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 shutdown_connections 2020-11-20T12:21:31.354 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:31.343+0000 7faf0c886700 1 -- 172.21.15.187:0/3435569530 wait complete. 2020-11-20T12:21:31.354 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 171 2020-11-20T12:21:31.571 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 171, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:21:22.646683+0000', 'modified': '2020-11-20T12:21:25.085806+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10018}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10018': {'gid': 10018, 'name': 'a', 'rank': 0, 'incarnation': 168, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1915143367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1915143367}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1915143367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:21:31.571 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:21:31.572 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:21:31.853 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":10018,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":9.1792527269999997,"mdsmap_epoch":171,"osdmap_epoch":159,"osdmap_epoch_barrier":159,"uptime":15.527436183000001} 2020-11-20T12:21:31.854 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 10018, "mdsmap_epoch": 171, "osdmap_epoch": 159, "osdmap_epoch_barrier": 159, "rank_uptime": 9.179252727, "state": "up:active", "uptime": 15.527436183, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:21:31.854 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:21:31.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:31.855 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:21:31.909 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:21:31.909 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:21:32.047 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:21:32.047 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:21:32.095 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:21:32.095 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:21:32.102 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:21:32.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:32.103 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:21:32.235 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2020-11-20T12:21:32.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.488+0000 7f2ba024f700 1 Processor -- start 2020-11-20T12:21:32.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.489+0000 7f2ba024f700 1 -- start start 2020-11-20T12:21:32.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.490+0000 7f2ba024f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2b980b73b0 0x7f2b98066e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:32.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.490+0000 7f2ba024f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b98146010 con 0x7f2b980b6fd0 2020-11-20T12:21:32.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.490+0000 7f2ba024f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b981462b0 con 0x7f2b98067300 2020-11-20T12:21:32.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.490+0000 7f2ba024f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b98146550 con 0x7f2b980b73b0 2020-11-20T12:21:32.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.491+0000 7f2b9e7ec700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f2b98067300 0x7f2b98144d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51270/0 (socket says 172.21.15.187:51270) 2020-11-20T12:21:32.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.491+0000 7f2b9e7ec700 1 -- 172.21.15.187:0/2058346526 learned_addr learned my addr 172.21.15.187:0/2058346526 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:32.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.491+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300263089 0 0) 0x7f2b981462b0 con 0x7f2b98067300 2020-11-20T12:21:32.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.491+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b80003240 con 0x7f2b98067300 2020-11-20T12:21:32.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.491+0000 7f2b9d7ea700 1 --2- 172.21.15.187:0/2058346526 >> v2:172.21.15.5:3300/0 conn(0x7f2b980b73b0 0x7f2b98066e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:32.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2783594222 0 0) 0x7f2b80003240 con 0x7f2b98067300 2020-11-20T12:21:32.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 >> v2:172.21.15.5:3300/0 conn(0x7f2b980b73b0 msgr2=0x7f2b98066e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:32.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 --2- 172.21.15.187:0/2058346526 >> v2:172.21.15.5:3300/0 conn(0x7f2b980b73b0 0x7f2b98066e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:32.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 >> v1:172.21.15.5:6789/0 conn(0x7f2b980b6fd0 legacy=0x7f2b98066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:32.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b980b6650 con 0x7f2b98067300 2020-11-20T12:21:32.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f2b90001860 con 0x7f2b98067300 2020-11-20T12:21:32.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 >> v1:172.21.15.187:6789/0 conn(0x7f2b98067300 legacy=0x7f2b98144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:32.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b80003800 0x7f2b80003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:32.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b800044e0 0x7f2b80004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:32.497 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b80004e50 0x7f2b8000d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:32.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b98146550 con 0x7f2b80004e50 2020-11-20T12:21:32.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b800053b0 con 0x7f2b80003800 2020-11-20T12:21:32.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.492+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8000de80 con 0x7f2b800044e0 2020-11-20T12:21:32.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.493+0000 7f2b9dfeb700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b80003800 0x7f2b80003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:32.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.493+0000 7f2b9d7ea700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b800044e0 0x7f2b80004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:32.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.493+0000 7f2b9e7ec700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b80004e50 0x7f2b8000d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:32.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.493+0000 7f2b9dfeb700 1 -- 172.21.15.187:0/2058346526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b800044e0 msgr2=0x7f2b80004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:32.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.494+0000 7f2b9dfeb700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b800044e0 0x7f2b80004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:32.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.494+0000 7f2b9dfeb700 1 -- 172.21.15.187:0/2058346526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b80004e50 msgr2=0x7f2b8000d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:32.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.494+0000 7f2b9dfeb700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b80004e50 0x7f2b8000d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:32.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.494+0000 7f2b9e7ec700 1 -- 172.21.15.187:0/2058346526 reap_dead start 2020-11-20T12:21:32.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.494+0000 7f2b9dfeb700 1 -- 172.21.15.187:0/2058346526 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b80003240 con 0x7f2b80003800 2020-11-20T12:21:32.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.494+0000 7f2b9dfeb700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b80003800 0x7f2b80003c00 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7f2b94002860 tx=0x7f2b94007b90).ready entity=mon.1 client_cookie=728c51ee6a7e782f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:32.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.495+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94007020 con 0x7f2b80003800 2020-11-20T12:21:32.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.495+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b940071a0 con 0x7f2b80003800 2020-11-20T12:21:32.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.495+0000 7f2b9cfe9700 1 -- 172.21.15.187:0/2058346526 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400e560 con 0x7f2b80003800 2020-11-20T12:21:32.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.495+0000 7f2ba024f700 1 -- 172.21.15.187:0/2058346526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b80003800 msgr2=0x7f2b80003c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:32.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.495+0000 7f2ba024f700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b80003800 0x7f2b80003c00 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7f2b94002860 tx=0x7f2b94007b90).stop 2020-11-20T12:21:32.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.495+0000 7f2ba024f700 1 -- 172.21.15.187:0/2058346526 shutdown_connections 2020-11-20T12:21:32.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.495+0000 7f2ba024f700 1 --2- 172.21.15.187:0/2058346526 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b80003800 0x7f2b80003c00 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:32.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.495+0000 7f2ba024f700 1 -- 172.21.15.187:0/2058346526 >> 172.21.15.187:0/2058346526 conn(0x7f2b980bacf0 msgr2=0x7f2b980bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:32.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.496+0000 7f2ba024f700 1 -- 172.21.15.187:0/2058346526 shutdown_connections 2020-11-20T12:21:32.504 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.496+0000 7f2ba024f700 1 -- 172.21.15.187:0/2058346526 wait complete. 2020-11-20T12:21:32.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.497+0000 7f2ba024f700 1 Processor -- start 2020-11-20T12:21:32.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.498+0000 7f2ba024f700 1 -- start start 2020-11-20T12:21:32.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2ba024f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b980b4250 0x7f2b9815bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:32.507 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2ba024f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b980b4630 0x7f2b98157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:32.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2ba024f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b98157800 0x7f2b98157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:32.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2ba024f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b98158310 con 0x7f2b98157800 2020-11-20T12:21:32.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2ba024f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b981585b0 con 0x7f2b980b4630 2020-11-20T12:21:32.508 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2ba024f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b98158850 con 0x7f2b980b4250 2020-11-20T12:21:32.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2b9d7ea700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b980b4630 0x7f2b98157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:32.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2b9d7ea700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b980b4630 0x7f2b98157310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54678/0 (socket says 172.21.15.187:54678) 2020-11-20T12:21:32.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2b9d7ea700 1 -- 172.21.15.187:0/834904773 learned_addr learned my addr 172.21.15.187:0/834904773 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:32.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.499+0000 7f2b9dfeb700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b980b4250 0x7f2b9815bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:32.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b9d7ea700 1 -- 172.21.15.187:0/834904773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b980b4250 msgr2=0x7f2b9815bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:32.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b9d7ea700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b980b4250 0x7f2b9815bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:32.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b9d7ea700 1 -- 172.21.15.187:0/834904773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b98157800 msgr2=0x7f2b98157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:32.510 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b9d7ea700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b98157800 0x7f2b98157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:32.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b9d7ea700 1 -- 172.21.15.187:0/834904773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9809d370 con 0x7f2b980b4630 2020-11-20T12:21:32.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b9dfeb700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b980b4250 0x7f2b9815bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:32.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b9d7ea700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b980b4630 0x7f2b98157310 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f2b8c010c50 tx=0x7f2b8c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:32.511 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b89ffb700 1 -- 172.21.15.187:0/834904773 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c00c9e0 con 0x7f2b980b4630 2020-11-20T12:21:32.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b89ffb700 1 -- 172.21.15.187:0/834904773 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8c00c020 con 0x7f2b980b4630 2020-11-20T12:21:32.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b98158cf0 con 0x7f2b980b4630 2020-11-20T12:21:32.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2b89ffb700 1 -- 172.21.15.187:0/834904773 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c00cd50 con 0x7f2b980b4630 2020-11-20T12:21:32.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.500+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b980b82f0 con 0x7f2b980b4630 2020-11-20T12:21:32.512 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.502+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b98158150 con 0x7f2b980b4630 2020-11-20T12:21:32.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.502+0000 7f2b89ffb700 1 -- 172.21.15.187:0/834904773 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2b8c023070 con 0x7f2b980b4630 2020-11-20T12:21:32.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.503+0000 7f2b89ffb700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2b70056d20 0x7f2b700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:32.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.503+0000 7f2b89ffb700 1 -- 172.21.15.187:0/834904773 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2b8c070650 con 0x7f2b980b4630 2020-11-20T12:21:32.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.503+0000 7f2b9dfeb700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2b70056d20 0x7f2b700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:32.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.504+0000 7f2b9dfeb700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2b70056d20 0x7f2b700591e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f2b9805f730 tx=0x7f2b94016000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:32.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.507+0000 7f2b89ffb700 1 -- 172.21.15.187:0/834904773 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2b8c045aa0 con 0x7f2b980b4630 2020-11-20T12:21:32.760 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:32.757+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2b98158a10 con 0x7f2b980b4630 2020-11-20T12:21:33.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.159+0000 7f2b89ffb700 1 -- 172.21.15.187:0/834904773 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v172) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2b8c0490c0 con 0x7f2b980b4630 2020-11-20T12:21:33.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2b70056d20 msgr2=0x7f2b700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:33.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2b70056d20 0x7f2b700591e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f2b9805f730 tx=0x7f2b94016000).stop 2020-11-20T12:21:33.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b980b4630 msgr2=0x7f2b98157310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:33.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b980b4630 0x7f2b98157310 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f2b8c010c50 tx=0x7f2b8c008040).stop 2020-11-20T12:21:33.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 shutdown_connections 2020-11-20T12:21:33.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2b70056d20 0x7f2b700591e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b98157800 0x7f2b98157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:33.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2b980b4250 0x7f2b9815bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 --2- 172.21.15.187:0/834904773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b980b4630 0x7f2b98157310 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 >> 172.21.15.187:0/834904773 conn(0x7f2b980bacf0 msgr2=0x7f2b98147ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:33.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.162+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 shutdown_connections 2020-11-20T12:21:33.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:33.163+0000 7f2ba024f700 1 -- 172.21.15.187:0/834904773 wait complete. 2020-11-20T12:21:33.170 INFO:teuthology.orchestra.run.smithi187.stderr:enabled new snapshots 2020-11-20T12:21:33.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:33.197 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir snapdir) 2020-11-20T12:21:33.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:33.316 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir snapdir/subdir) 2020-11-20T12:21:33.428 INFO:tasks.cephfs.mount:Writing 8388608 bytes to snapdir/subdir/file_a 2020-11-20T12:21:33.429 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:21:33.429 INFO:teuthology.orchestra.run.smithi187:> import zlib 2020-11-20T12:21:33.430 INFO:teuthology.orchestra.run.smithi187:> path = "/home/ubuntu/cephtest/mnt.0/snapdir/subdir/file_a" 2020-11-20T12:21:33.430 INFO:teuthology.orchestra.run.smithi187:> with open(path, '"'"'w'"'"') as f: 2020-11-20T12:21:33.430 INFO:teuthology.orchestra.run.smithi187:> for i in range(0, 8388608): 2020-11-20T12:21:33.431 INFO:teuthology.orchestra.run.smithi187:> val = zlib.crc32(str(i).encode('"'"'utf-8'"'"')) & 7 2020-11-20T12:21:33.431 INFO:teuthology.orchestra.run.smithi187:> f.write(chr(val)) 2020-11-20T12:21:33.432 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:21:40.431 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:21:40.431 INFO:teuthology.orchestra.run.smithi187:> import os 2020-11-20T12:21:40.432 INFO:teuthology.orchestra.run.smithi187:> import stat 2020-11-20T12:21:40.432 INFO:teuthology.orchestra.run.smithi187:> 2020-11-20T12:21:40.432 INFO:teuthology.orchestra.run.smithi187:> print(os.stat("/home/ubuntu/cephtest/mnt.0/snapdir/subdir/file_a").st_ino) 2020-11-20T12:21:40.433 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:21:40.625 INFO:teuthology.orchestra.run.smithi187.stdout:1099511627778 2020-11-20T12:21:40.813 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:40.813 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir snapdir/.snap/snap1) 2020-11-20T12:21:40.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:40.978 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo dd if=/dev/urandom of=snapdir/subdir/file_a bs=1M conv=fdatasync count=8 seek=0) 2020-11-20T12:21:41.320 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records in 2020-11-20T12:21:41.321 INFO:teuthology.orchestra.run.smithi187.stderr:8+0 records out 2020-11-20T12:21:41.321 INFO:teuthology.orchestra.run.smithi187.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.146602 s, 57.2 MB/s 2020-11-20T12:21:41.328 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:41.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.560+0000 7f258d931700 1 Processor -- start 2020-11-20T12:21:41.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.561+0000 7f258d931700 1 -- start start 2020-11-20T12:21:41.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.562+0000 7f258d931700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2588142740 0x7f258814c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:41.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.562+0000 7f258d931700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25880bd8e0 0x7f2588067a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:41.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.562+0000 7f258d931700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25880629d0 con 0x7f25880bd460 2020-11-20T12:21:41.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.562+0000 7f258d931700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2588149780 con 0x7f25880bd8e0 2020-11-20T12:21:41.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.562+0000 7f258d931700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2588062c70 con 0x7f2588142740 2020-11-20T12:21:41.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.563+0000 7f258d130700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25880bd8e0 0x7f2588067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:41.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.563+0000 7f2587fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f25880bd460 0x7f2588067320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:44804/0 (socket says 172.21.15.187:44804) 2020-11-20T12:21:41.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.563+0000 7f2587fff700 1 -- 172.21.15.187:0/4147590559 learned_addr learned my addr 172.21.15.187:0/4147590559 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:41.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.563+0000 7f258c92f700 1 --2- 172.21.15.187:0/4147590559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2588142740 0x7f258814c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:41.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.564+0000 7f258d130700 1 -- 172.21.15.187:0/4147590559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2588142740 msgr2=0x7f258814c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:41.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.564+0000 7f258d130700 1 --2- 172.21.15.187:0/4147590559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2588142740 0x7f258814c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.564+0000 7f258d130700 1 -- 172.21.15.187:0/4147590559 >> v1:172.21.15.5:6789/0 conn(0x7f25880bd460 legacy=0x7f2588067320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:41.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.564+0000 7f258d130700 1 -- 172.21.15.187:0/4147590559 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2588062f10 con 0x7f25880bd8e0 2020-11-20T12:21:41.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.564+0000 7f258d130700 1 --2- 172.21.15.187:0/4147590559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588067a30 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7f2580009c90 tx=0x7f25800091d0).ready entity=mon.1 client_cookie=7b1e4ac917088148 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:41.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.564+0000 7f25877fe700 1 -- 172.21.15.187:0/4147590559 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258000ba70 con 0x7f25880bd8e0 2020-11-20T12:21:41.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.564+0000 7f25877fe700 1 -- 172.21.15.187:0/4147590559 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258000bbd0 con 0x7f25880bd8e0 2020-11-20T12:21:41.570 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.564+0000 7f25877fe700 1 -- 172.21.15.187:0/4147590559 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580011400 con 0x7f25880bd8e0 2020-11-20T12:21:41.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.565+0000 7f258d931700 1 -- 172.21.15.187:0/4147590559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 msgr2=0x7f2588067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:41.571 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.565+0000 7f258d931700 1 --2- 172.21.15.187:0/4147590559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588067a30 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7f2580009c90 tx=0x7f25800091d0).stop 2020-11-20T12:21:41.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.565+0000 7f258d931700 1 -- 172.21.15.187:0/4147590559 shutdown_connections 2020-11-20T12:21:41.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.565+0000 7f258d931700 1 --2- 172.21.15.187:0/4147590559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588067a30 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.565+0000 7f258d931700 1 --2- 172.21.15.187:0/4147590559 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2588142740 0x7f258814c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.565+0000 7f258d931700 1 -- 172.21.15.187:0/4147590559 >> 172.21.15.187:0/4147590559 conn(0x7f2588060c40 msgr2=0x7f25880bcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:41.572 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.565+0000 7f258d931700 1 -- 172.21.15.187:0/4147590559 shutdown_connections 2020-11-20T12:21:41.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.565+0000 7f258d931700 1 -- 172.21.15.187:0/4147590559 wait complete. 2020-11-20T12:21:41.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.567+0000 7f258d931700 1 Processor -- start 2020-11-20T12:21:41.573 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.568+0000 7f258d931700 1 -- start start 2020-11-20T12:21:41.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f258d931700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25880bd460 0x7f25880bbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:41.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f258d931700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:41.575 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f258d931700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588142740 0x7f258813f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:41.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f258d931700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2588158680 con 0x7f2588142740 2020-11-20T12:21:41.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f258d931700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2588158920 con 0x7f25880bd8e0 2020-11-20T12:21:41.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f258d931700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2588158bc0 con 0x7f25880bd460 2020-11-20T12:21:41.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f2587fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:41.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f2587fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54696/0 (socket says 172.21.15.187:54696) 2020-11-20T12:21:41.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.569+0000 7f2587fff700 1 -- 172.21.15.187:0/1334511607 learned_addr learned my addr 172.21.15.187:0/1334511607 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:41.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f258c92f700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25880bd460 0x7f25880bbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:41.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f258d130700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588142740 0x7f258813f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:41.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2587fff700 1 -- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25880bd460 msgr2=0x7f25880bbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:41.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2587fff700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25880bd460 0x7f25880bbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2587fff700 1 -- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588142740 msgr2=0x7f258813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:41.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2587fff700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588142740 0x7f258813f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2587fff700 1 -- 172.21.15.187:0/1334511607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880bc2a0 con 0x7f25880bd8e0 2020-11-20T12:21:41.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f258d130700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588142740 0x7f258813f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:41.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2587fff700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588157d00 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7f257400c710 tx=0x7f2574007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:41.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2584ff9700 1 -- 172.21.15.187:0/1334511607 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574007320 con 0x7f25880bd8e0 2020-11-20T12:21:41.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2584ff9700 1 -- 172.21.15.187:0/1334511607 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2574007480 con 0x7f25880bd8e0 2020-11-20T12:21:41.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258815b460 con 0x7f25880bd8e0 2020-11-20T12:21:41.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.570+0000 7f2584ff9700 1 -- 172.21.15.187:0/1334511607 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574005800 con 0x7f25880bd8e0 2020-11-20T12:21:41.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.571+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2588158430 con 0x7f25880bd8e0 2020-11-20T12:21:41.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.572+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2588062710 con 0x7f25880bd8e0 2020-11-20T12:21:41.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.572+0000 7f2584ff9700 1 -- 172.21.15.187:0/1334511607 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2574010030 con 0x7f25880bd8e0 2020-11-20T12:21:41.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.573+0000 7f2584ff9700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2560056d20 0x7f25600591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:41.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.573+0000 7f2584ff9700 1 -- 172.21.15.187:0/1334511607 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f257406fc70 con 0x7f25880bd8e0 2020-11-20T12:21:41.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.573+0000 7f258c92f700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2560056d20 0x7f25600591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:41.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.574+0000 7f258c92f700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2560056d20 0x7f25600591e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f257c006bd0 tx=0x7f257c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:41.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.577+0000 7f2584ff9700 1 -- 172.21.15.187:0/1334511607 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f25740450c0 con 0x7f25880bd8e0 2020-11-20T12:21:41.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.822+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25880446d0 con 0x7f25880bd8e0 2020-11-20T12:21:41.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.823+0000 7f2584ff9700 1 -- 172.21.15.187:0/1334511607 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2145 (secure 0 0 0) 0x7f25740486e0 con 0x7f25880bd8e0 2020-11-20T12:21:41.827 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:41.828 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":172,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:33.149592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:21:41.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2560056d20 msgr2=0x7f25600591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:41.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2560056d20 0x7f25600591e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f257c006bd0 tx=0x7f257c008040).stop 2020-11-20T12:21:41.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 msgr2=0x7f2588157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:41.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588157d00 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7f257400c710 tx=0x7f2574007c10).stop 2020-11-20T12:21:41.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 shutdown_connections 2020-11-20T12:21:41.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f2560056d20 0x7f25600591e0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2588142740 0x7f258813f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25880bd460 0x7f25880bbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 --2- 172.21.15.187:0/1334511607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25880bd8e0 0x7f2588157d00 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:41.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 >> 172.21.15.187:0/1334511607 conn(0x7f2588060c40 msgr2=0x7f25880bcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:41.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 shutdown_connections 2020-11-20T12:21:41.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:41.826+0000 7f258d931700 1 -- 172.21.15.187:0/1334511607 wait complete. 2020-11-20T12:21:41.833 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 172 2020-11-20T12:21:41.854 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:21:42.163 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:21:42.164 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:21:42.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:42.165 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rm -f snapdir/subdir/file_a) 2020-11-20T12:21:42.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:42.271 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rmdir snapdir/subdir) 2020-11-20T12:21:42.389 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:21:42.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:42.390 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:21:42.535 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:21:42.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:42.536 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:21:42.595 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:42.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.835+0000 7f340b59e700 1 Processor -- start 2020-11-20T12:21:42.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.835+0000 7f340b59e700 1 -- start start 2020-11-20T12:21:42.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.836+0000 7f340b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c065f90 0x7f340c065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.836+0000 7f340b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f340c09ca90 0x7f340c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.836+0000 7f340b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c0b13e0 con 0x7f340c065190 2020-11-20T12:21:42.841 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.836+0000 7f340b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f340c14ab30 con 0x7f340c09ca90 2020-11-20T12:21:42.842 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.837+0000 7f340b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340c062c10 con 0x7f340c065f90 2020-11-20T12:21:42.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.837+0000 7f340ad9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f340c09ca90 0x7f340c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.837+0000 7f340a59c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f340c065190 0x7f340c143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:44818/0 (socket says 172.21.15.187:44818) 2020-11-20T12:21:42.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.838+0000 7f340a59c700 1 -- 172.21.15.187:0/3168938690 learned_addr learned my addr 172.21.15.187:0/3168938690 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:42.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.838+0000 7f3403fff700 1 --2- 172.21.15.187:0/3168938690 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c065f90 0x7f340c065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.839+0000 7f340ad9d700 1 -- 172.21.15.187:0/3168938690 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c065f90 msgr2=0x7f340c065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:42.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.839+0000 7f340ad9d700 1 --2- 172.21.15.187:0/3168938690 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c065f90 0x7f340c065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:42.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.839+0000 7f340ad9d700 1 -- 172.21.15.187:0/3168938690 >> v1:172.21.15.5:6789/0 conn(0x7f340c065190 legacy=0x7f340c143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.839+0000 7f340ad9d700 1 -- 172.21.15.187:0/3168938690 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c0b1640 con 0x7f340c09ca90 2020-11-20T12:21:42.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.839+0000 7f340ad9d700 1 --2- 172.21.15.187:0/3168938690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c09ca90 0x7f340c09cf20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f340c14ac70 tx=0x7f34040091d0).ready entity=mon.0 client_cookie=a24cdeb0402f228b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:42.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f3409d9b700 1 -- 172.21.15.187:0/3168938690 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340400bad0 con 0x7f340c09ca90 2020-11-20T12:21:42.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f3409d9b700 1 -- 172.21.15.187:0/3168938690 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340400bc30 con 0x7f340c09ca90 2020-11-20T12:21:42.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f3409d9b700 1 -- 172.21.15.187:0/3168938690 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404011400 con 0x7f340c09ca90 2020-11-20T12:21:42.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f340b59e700 1 -- 172.21.15.187:0/3168938690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c09ca90 msgr2=0x7f340c09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f340b59e700 1 --2- 172.21.15.187:0/3168938690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c09ca90 0x7f340c09cf20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f340c14ac70 tx=0x7f34040091d0).stop 2020-11-20T12:21:42.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f340b59e700 1 -- 172.21.15.187:0/3168938690 shutdown_connections 2020-11-20T12:21:42.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f340b59e700 1 --2- 172.21.15.187:0/3168938690 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c09ca90 0x7f340c09cf20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:42.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f340b59e700 1 --2- 172.21.15.187:0/3168938690 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c065f90 0x7f340c065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:42.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.840+0000 7f340b59e700 1 -- 172.21.15.187:0/3168938690 >> 172.21.15.187:0/3168938690 conn(0x7f340c0bab30 msgr2=0x7f340c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:42.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.841+0000 7f340b59e700 1 -- 172.21.15.187:0/3168938690 shutdown_connections 2020-11-20T12:21:42.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.841+0000 7f340b59e700 1 -- 172.21.15.187:0/3168938690 wait complete. 2020-11-20T12:21:42.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.842+0000 7f340b59e700 1 Processor -- start 2020-11-20T12:21:42.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.842+0000 7f340b59e700 1 -- start start 2020-11-20T12:21:42.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f340c065190 0x7f340c099350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c065f90 0x7f340c0b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c09ca90 0x7f340c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c09dae0 con 0x7f340c065f90 2020-11-20T12:21:42.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c141920 con 0x7f340c065190 2020-11-20T12:21:42.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340c0b2b90 con 0x7f340c09ca90 2020-11-20T12:21:42.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f340c065190 0x7f340c099350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340a59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f340c065190 0x7f340c099350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54708/0 (socket says 172.21.15.187:54708) 2020-11-20T12:21:42.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.843+0000 7f340a59c700 1 -- 172.21.15.187:0/2583927697 learned_addr learned my addr 172.21.15.187:0/2583927697 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:42.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.844+0000 7f3403fff700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c065f90 0x7f340c0b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.844+0000 7f340ad9d700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c09ca90 0x7f340c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.844+0000 7f340a59c700 1 -- 172.21.15.187:0/2583927697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c09ca90 msgr2=0x7f340c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.844+0000 7f340a59c700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c09ca90 0x7f340c0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:42.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.844+0000 7f340a59c700 1 -- 172.21.15.187:0/2583927697 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c065f90 msgr2=0x7f340c0b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.844+0000 7f340a59c700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c065f90 0x7f340c0b2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:42.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.844+0000 7f340a59c700 1 -- 172.21.15.187:0/2583927697 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c0b2e30 con 0x7f340c065190 2020-11-20T12:21:42.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.844+0000 7f340a59c700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f340c065190 0x7f340c099350 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7f33fc00c710 tx=0x7f33fc007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:42.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.845+0000 7f34027fc700 1 -- 172.21.15.187:0/2583927697 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc007320 con 0x7f340c065190 2020-11-20T12:21:42.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.845+0000 7f34027fc700 1 -- 172.21.15.187:0/2583927697 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33fc007480 con 0x7f340c065190 2020-11-20T12:21:42.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.845+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340c099840 con 0x7f340c065190 2020-11-20T12:21:42.869 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.845+0000 7f34027fc700 1 -- 172.21.15.187:0/2583927697 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc005800 con 0x7f340c065190 2020-11-20T12:21:42.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.845+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340c0684f0 con 0x7f340c065190 2020-11-20T12:21:42.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.846+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340c061a20 con 0x7f340c065190 2020-11-20T12:21:42.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.846+0000 7f34027fc700 1 -- 172.21.15.187:0/2583927697 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f33fc010030 con 0x7f340c065190 2020-11-20T12:21:42.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.847+0000 7f34027fc700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33e8056d50 0x7f33e8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.847+0000 7f34027fc700 1 -- 172.21.15.187:0/2583927697 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f33fc06fc40 con 0x7f340c065190 2020-11-20T12:21:42.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.848+0000 7f3403fff700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33e8056d50 0x7f33e8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.848+0000 7f3403fff700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33e8056d50 0x7f33e8059210 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f33f4006bd0 tx=0x7f33f4008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:42.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:42.851+0000 7f34027fc700 1 -- 172.21.15.187:0/2583927697 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f33fc045090 con 0x7f340c065190 2020-11-20T12:21:43.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.096+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f340c0afc60 con 0x7f340c065190 2020-11-20T12:21:43.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.097+0000 7f34027fc700 1 -- 172.21.15.187:0/2583927697 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2145 (secure 0 0 0) 0x7f33fc0486b0 con 0x7f340c065190 2020-11-20T12:21:43.101 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:43.101 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":172,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:33.149592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:21:43.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.099+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33e8056d50 msgr2=0x7f33e8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:43.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.099+0000 7f340b59e700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33e8056d50 0x7f33e8059210 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f33f4006bd0 tx=0x7f33f4008040).stop 2020-11-20T12:21:43.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.099+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f340c065190 msgr2=0x7f340c099350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:43.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.099+0000 7f340b59e700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f340c065190 0x7f340c099350 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7f33fc00c710 tx=0x7f33fc007c00).stop 2020-11-20T12:21:43.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.100+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 shutdown_connections 2020-11-20T12:21:43.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.100+0000 7f340b59e700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f33e8056d50 0x7f33e8059210 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:43.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.100+0000 7f340b59e700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f340c09ca90 0x7f340c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:43.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.100+0000 7f340b59e700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f340c065190 0x7f340c099350 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:43.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.100+0000 7f340b59e700 1 --2- 172.21.15.187:0/2583927697 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f340c065f90 0x7f340c0b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:43.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.100+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 >> 172.21.15.187:0/2583927697 conn(0x7f340c0bab30 msgr2=0x7f340c09c120 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:43.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.100+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 shutdown_connections 2020-11-20T12:21:43.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:43.100+0000 7f340b59e700 1 -- 172.21.15.187:0/2583927697 wait complete. 2020-11-20T12:21:43.106 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 172 2020-11-20T12:21:43.124 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2020-11-20T12:21:43.401 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_created":2}} 2020-11-20T12:21:43.402 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 2 } } 2020-11-20T12:21:43.403 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:21:43.404 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2020-11-20T12:21:43.404 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:21:43.406 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:21:43.406 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:21:43.407 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2020-11-20T12:21:43.407 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:21:43.407 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:21:43.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:43.408 INFO:teuthology.orchestra.run.smithi187:> ip addr 2020-11-20T12:21:43.463 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:21:43.464 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:21:43.464 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:21:43.464 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:43.464 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2020-11-20T12:21:43.464 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:43.465 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:21:43.465 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:43.465 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:21:43.465 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:43.466 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T12:21:43.466 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:43.466 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:21:43.466 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:43.467 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2020-11-20T12:21:43.467 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 41735sec preferred_lft 41735sec 2020-11-20T12:21:43.467 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2020-11-20T12:21:43.467 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:43.467 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2020-11-20T12:21:43.468 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff 2020-11-20T12:21:43.468 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2020-11-20T12:21:43.468 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:43.468 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::9caf:efff:fe0d:50cc/64 scope link 2020-11-20T12:21:43.468 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:43.469 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:21:43.469 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether fa:1e:64:83:49:3f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:21:43.469 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f81e:64ff:fe83:493f/64 scope link 2020-11-20T12:21:43.469 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:43.470 INFO:teuthology.orchestra.run.smithi187.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2020-11-20T12:21:43.470 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether b6:dd:a8:85:2e:ef brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:21:43.470 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::b4dd:a8ff:fe85:2eef/64 scope link 2020-11-20T12:21:43.470 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:21:43.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:43.472 INFO:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:21:43.572 INFO:teuthology.orchestra.run.smithi187.stdout:1 2020-11-20T12:21:43.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:43.577 INFO:teuthology.orchestra.run.smithi187:> route 2020-11-20T12:21:43.632 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2020-11-20T12:21:43.633 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:21:43.633 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 103 0 0 enp3s0f1 2020-11-20T12:21:43.633 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 103 0 0 enp3s0f1 2020-11-20T12:21:43.634 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:21:43.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:43.635 INFO:teuthology.orchestra.run.smithi187:> (cd / && exec sudo bash -c ' 2020-11-20T12:21:43.636 INFO:teuthology.orchestra.run.smithi187:> set -e 2020-11-20T12:21:43.636 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:21:43.636 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:21:43.637 INFO:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:21:43.637 INFO:teuthology.orchestra.run.smithi187:> ') 2020-11-20T12:21:43.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:43.891 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:21:43.946 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:21:43.946 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:21:43.947 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:21:43.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:43.948 INFO:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2020-11-20T12:21:44.047 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:21:44.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.311+0000 7f3b5f59e700 1 Processor -- start 2020-11-20T12:21:45.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.312+0000 7f3b5f59e700 1 -- start start 2020-11-20T12:21:45.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.313+0000 7f3b5f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b60145cc0 0x7f3b60066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.313+0000 7f3b5f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3b601487e0 0x7f3b60148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.313+0000 7f3b5f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6014c170 con 0x7f3b600683f0 2020-11-20T12:21:45.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.313+0000 7f3b5f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6014c410 con 0x7f3b601487e0 2020-11-20T12:21:45.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.313+0000 7f3b5f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6014c6b0 con 0x7f3b60145cc0 2020-11-20T12:21:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.314+0000 7f3b5e59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b60145cc0 0x7f3b60066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.314+0000 7f3b55d9b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3b600683f0 0x7f3b60065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:44832/0 (socket says 172.21.15.187:44832) 2020-11-20T12:21:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.314+0000 7f3b55d9b700 1 -- 172.21.15.187:0/1424525912 learned_addr learned my addr 172.21.15.187:0/1424525912 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.314+0000 7f3b5ed9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3b601487e0 0x7f3b60148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.315+0000 7f3b5e59c700 1 -- 172.21.15.187:0/1424525912 >> v2:172.21.15.5:3300/0 conn(0x7f3b601487e0 msgr2=0x7f3b60148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.315+0000 7f3b5e59c700 1 --2- 172.21.15.187:0/1424525912 >> v2:172.21.15.5:3300/0 conn(0x7f3b601487e0 0x7f3b60148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.315+0000 7f3b5e59c700 1 -- 172.21.15.187:0/1424525912 >> v1:172.21.15.5:6789/0 conn(0x7f3b600683f0 legacy=0x7f3b60065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.315+0000 7f3b5e59c700 1 -- 172.21.15.187:0/1424525912 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600658a0 con 0x7f3b60145cc0 2020-11-20T12:21:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.315+0000 7f3b5dd9b700 1 -- 172.21.15.187:0/1424525912 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734907504 0 0) 0x7f3b6014c170 con 0x7f3b600683f0 2020-11-20T12:21:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.316+0000 7f3b5ed9d700 1 --2- 172.21.15.187:0/1424525912 >> v2:172.21.15.5:3300/0 conn(0x7f3b601487e0 0x7f3b60148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:21:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.316+0000 7f3b5e59c700 1 --2- 172.21.15.187:0/1424525912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b60145cc0 0x7f3b60066040 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f3b5000b9a0 tx=0x7f3b50006df0).ready entity=mon.2 client_cookie=9ec355f55975e7dc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5dd9b700 1 -- 172.21.15.187:0/1424525912 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b500097b0 con 0x7f3b60145cc0 2020-11-20T12:21:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5dd9b700 1 -- 172.21.15.187:0/1424525912 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b50009910 con 0x7f3b60145cc0 2020-11-20T12:21:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5dd9b700 1 -- 172.21.15.187:0/1424525912 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50009c90 con 0x7f3b60145cc0 2020-11-20T12:21:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5f59e700 1 -- 172.21.15.187:0/1424525912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b60145cc0 msgr2=0x7f3b60066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/1424525912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b60145cc0 0x7f3b60066040 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f3b5000b9a0 tx=0x7f3b50006df0).stop 2020-11-20T12:21:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5f59e700 1 -- 172.21.15.187:0/1424525912 shutdown_connections 2020-11-20T12:21:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/1424525912 >> v2:172.21.15.5:3300/0 conn(0x7f3b601487e0 0x7f3b60148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/1424525912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b60145cc0 0x7f3b60066040 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5f59e700 1 -- 172.21.15.187:0/1424525912 >> 172.21.15.187:0/1424525912 conn(0x7f3b600bab30 msgr2=0x7f3b600baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:45.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5f59e700 1 -- 172.21.15.187:0/1424525912 shutdown_connections 2020-11-20T12:21:45.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.317+0000 7f3b5f59e700 1 -- 172.21.15.187:0/1424525912 wait complete. 2020-11-20T12:21:45.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.319+0000 7f3b5f59e700 1 Processor -- start 2020-11-20T12:21:45.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.319+0000 7f3b5f59e700 1 -- start start 2020-11-20T12:21:45.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.320+0000 7f3b5f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b600683f0 0x7f3b6009fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.320+0000 7f3b5f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b60145cc0 0x7f3b60158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.320+0000 7f3b5f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b601487e0 0x7f3b6014b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.320+0000 7f3b5f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b60158cf0 con 0x7f3b600683f0 2020-11-20T12:21:45.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.320+0000 7f3b5f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6009ec30 con 0x7f3b60145cc0 2020-11-20T12:21:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.320+0000 7f3b5f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6009eed0 con 0x7f3b601487e0 2020-11-20T12:21:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b60145cc0 0x7f3b60158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b60145cc0 0x7f3b60158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54724/0 (socket says 172.21.15.187:54724) 2020-11-20T12:21:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 -- 172.21.15.187:0/3007067686 learned_addr learned my addr 172.21.15.187:0/3007067686 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:45.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b5e59c700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b600683f0 0x7f3b6009fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b5ed9d700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b601487e0 0x7f3b6014b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 -- 172.21.15.187:0/3007067686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b601487e0 msgr2=0x7f3b6014b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b601487e0 0x7f3b6014b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 -- 172.21.15.187:0/3007067686 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b600683f0 msgr2=0x7f3b6009fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b600683f0 0x7f3b6009fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 -- 172.21.15.187:0/3007067686 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6009e920 con 0x7f3b60145cc0 2020-11-20T12:21:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.321+0000 7f3b55d9b700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b60145cc0 0x7f3b60158800 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f3b48007fd0 tx=0x7f3b48007a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.322+0000 7f3b56ffd700 1 -- 172.21.15.187:0/3007067686 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b480072e0 con 0x7f3b60145cc0 2020-11-20T12:21:45.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.322+0000 7f3b56ffd700 1 -- 172.21.15.187:0/3007067686 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b48007440 con 0x7f3b60145cc0 2020-11-20T12:21:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.322+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6014b6a0 con 0x7f3b60145cc0 2020-11-20T12:21:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.322+0000 7f3b56ffd700 1 -- 172.21.15.187:0/3007067686 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48005800 con 0x7f3b60145cc0 2020-11-20T12:21:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.322+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6009f5b0 con 0x7f3b60145cc0 2020-11-20T12:21:45.688 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.323+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b60061a20 con 0x7f3b60145cc0 2020-11-20T12:21:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.324+0000 7f3b56ffd700 1 -- 172.21.15.187:0/3007067686 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3b48010070 con 0x7f3b60145cc0 2020-11-20T12:21:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.324+0000 7f3b56ffd700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b3c056d20 0x7f3b3c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.324+0000 7f3b56ffd700 1 -- 172.21.15.187:0/3007067686 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3b4806f790 con 0x7f3b60145cc0 2020-11-20T12:21:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.325+0000 7f3b5e59c700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b3c056d20 0x7f3b3c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.325+0000 7f3b5e59c700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b3c056d20 0x7f3b3c0591e0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f3b50006e30 tx=0x7f3b50015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.328+0000 7f3b56ffd700 1 -- 172.21.15.187:0/3007067686 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3b48044ad0 con 0x7f3b60145cc0 2020-11-20T12:21:45.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.516+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b600446d0 con 0x7f3b60145cc0 2020-11-20T12:21:45.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.519+0000 7f3b56ffd700 1 -- 172.21.15.187:0/3007067686 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f3b48018070 con 0x7f3b60145cc0 2020-11-20T12:21:45.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b3c056d20 msgr2=0x7f3b3c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.690 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b3c056d20 0x7f3b3c0591e0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f3b50006e30 tx=0x7f3b50015000).stop 2020-11-20T12:21:45.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b60145cc0 msgr2=0x7f3b60158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b60145cc0 0x7f3b60158800 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f3b48007fd0 tx=0x7f3b48007a20).stop 2020-11-20T12:21:45.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 shutdown_connections 2020-11-20T12:21:45.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3b3c056d20 0x7f3b3c0591e0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.691 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b600683f0 0x7f3b6009fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3b601487e0 0x7f3b6014b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 --2- 172.21.15.187:0/3007067686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b60145cc0 0x7f3b60158800 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.521+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 >> 172.21.15.187:0/3007067686 conn(0x7f3b600bab30 msgr2=0x7f3b600b9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:45.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.522+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 shutdown_connections 2020-11-20T12:21:45.692 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:44.522+0000 7f3b5f59e700 1 -- 172.21.15.187:0/3007067686 wait complete. 2020-11-20T12:21:45.693 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:45.693 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":159,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:21:23.651529+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:21:20.315283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2020-11-20T12:21:21.644244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:21:45.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:43.604166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:51.942528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:21:20.315283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:21:21.644244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:21:45.695 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:21:45.952 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.949+0000 7f894f59e700 1 Processor -- start 2020-11-20T12:21:45.953 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.949+0000 7f894f59e700 1 -- start start 2020-11-20T12:21:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.950+0000 7f894f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f89501484d0 0x7f8950061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.950+0000 7f894f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89500b1300 con 0x7f8950066f70 2020-11-20T12:21:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.950+0000 7f894f59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89500b15a0 con 0x7f89500622c0 2020-11-20T12:21:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.950+0000 7f894f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8950067730 con 0x7f89501484d0 2020-11-20T12:21:45.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.951+0000 7f894dd9b700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f89500622c0 0x7f89500b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51340/0 (socket says 172.21.15.187:51340) 2020-11-20T12:21:45.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.951+0000 7f894dd9b700 1 -- 172.21.15.187:0/1244226912 learned_addr learned my addr 172.21.15.187:0/1244226912 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:45.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.951+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694242757 0 0) 0x7f89500b15a0 con 0x7f89500622c0 2020-11-20T12:21:45.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.951+0000 7f894e59c700 1 --2- 172.21.15.187:0/1244226912 >> v2:172.21.15.5:3300/0 conn(0x7f89501484d0 0x7f8950061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934003240 con 0x7f89500622c0 2020-11-20T12:21:45.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2910248180 0 0) 0x7f8934003240 con 0x7f89500622c0 2020-11-20T12:21:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 >> v2:172.21.15.5:3300/0 conn(0x7f89501484d0 msgr2=0x7f8950061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 --2- 172.21.15.187:0/1244226912 >> v2:172.21.15.5:3300/0 conn(0x7f89501484d0 0x7f8950061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 >> v1:172.21.15.5:6789/0 conn(0x7f8950066f70 legacy=0x7f89500b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89500b1840 con 0x7f89500622c0 2020-11-20T12:21:45.958 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f89440014e0 con 0x7f89500622c0 2020-11-20T12:21:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894e59c700 1 --2- 172.21.15.187:0/1244226912 >> v2:172.21.15.5:3300/0 conn(0x7f89501484d0 0x7f8950061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 >> v1:172.21.15.187:6789/0 conn(0x7f89500622c0 legacy=0x7f89500b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8934003730 0x7f8934003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89340044e0 0x7f8934004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.959 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f893400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8950067730 con 0x7f89340044e0 2020-11-20T12:21:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934004260 con 0x7f8934004e50 2020-11-20T12:21:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.952+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893400bc20 con 0x7f8934003730 2020-11-20T12:21:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.953+0000 7f894dd9b700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89340044e0 0x7f8934004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.953+0000 7f894ed9d700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f893400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.953+0000 7f894e59c700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8934003730 0x7f8934003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894ed9d700 1 -- 172.21.15.187:0/1244226912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8934003730 msgr2=0x7f8934003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:45.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894ed9d700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8934003730 0x7f8934003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894ed9d700 1 -- 172.21.15.187:0/1244226912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89340044e0 msgr2=0x7f8934004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:45.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894ed9d700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89340044e0 0x7f8934004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894ed9d700 1 -- 172.21.15.187:0/1244226912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934003240 con 0x7f8934004e50 2020-11-20T12:21:45.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894ed9d700 1 -- 172.21.15.187:0/1244226912 reap_dead start 2020-11-20T12:21:45.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894ed9d700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f893400b730 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f8948001e40 tx=0x7f8948007f10).ready entity=mon.1 client_cookie=3091c1f920653218 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948001540 con 0x7f8934004e50 2020-11-20T12:21:45.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894800ca20 con 0x7f8934004e50 2020-11-20T12:21:45.966 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.954+0000 7f894d59a700 1 -- 172.21.15.187:0/1244226912 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894800cdc0 con 0x7f8934004e50 2020-11-20T12:21:45.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.955+0000 7f894f59e700 1 -- 172.21.15.187:0/1244226912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 msgr2=0x7f893400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.955+0000 7f894f59e700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f893400b730 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f8948001e40 tx=0x7f8948007f10).stop 2020-11-20T12:21:45.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.955+0000 7f894f59e700 1 -- 172.21.15.187:0/1244226912 shutdown_connections 2020-11-20T12:21:45.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.955+0000 7f894f59e700 1 --2- 172.21.15.187:0/1244226912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f893400b730 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.967 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.955+0000 7f894f59e700 1 -- 172.21.15.187:0/1244226912 >> 172.21.15.187:0/1244226912 conn(0x7f89500bab30 msgr2=0x7f89500baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:45.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.955+0000 7f894f59e700 1 -- 172.21.15.187:0/1244226912 shutdown_connections 2020-11-20T12:21:45.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.955+0000 7f894f59e700 1 -- 172.21.15.187:0/1244226912 wait complete. 2020-11-20T12:21:45.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.956+0000 7f894f59e700 1 Processor -- start 2020-11-20T12:21:45.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.956+0000 7f894f59e700 1 -- start start 2020-11-20T12:21:45.968 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.957+0000 7f894f59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f895009adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.957+0000 7f894f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f895009db50 0x7f8950143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.957+0000 7f894f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89501428b0 0x7f8950142d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.957+0000 7f894f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89500b34c0 con 0x7f89501428b0 2020-11-20T12:21:45.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.957+0000 7f894f59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895009b2b0 con 0x7f8934004e50 2020-11-20T12:21:45.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.957+0000 7f894f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895009b540 con 0x7f895009db50 2020-11-20T12:21:45.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f895009adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f895009adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54744/0 (socket says 172.21.15.187:54744) 2020-11-20T12:21:45.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894e59c700 1 -- 172.21.15.187:0/888698980 learned_addr learned my addr 172.21.15.187:0/888698980 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:45.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894ed9d700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89501428b0 0x7f8950142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894dd9b700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f895009db50 0x7f8950143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894e59c700 1 -- 172.21.15.187:0/888698980 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f895009db50 msgr2=0x7f8950143990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894e59c700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f895009db50 0x7f8950143990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894e59c700 1 -- 172.21.15.187:0/888698980 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89501428b0 msgr2=0x7f8950142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894e59c700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89501428b0 0x7f8950142d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.958+0000 7f894e59c700 1 -- 172.21.15.187:0/888698980 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895009b7e0 con 0x7f8934004e50 2020-11-20T12:21:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.959+0000 7f894e59c700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f895009adc0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f8940006fd0 tx=0x7f8940008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.959+0000 7f893e7fc700 1 -- 172.21.15.187:0/888698980 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940011660 con 0x7f8934004e50 2020-11-20T12:21:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.959+0000 7f893e7fc700 1 -- 172.21.15.187:0/888698980 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89400117c0 con 0x7f8934004e50 2020-11-20T12:21:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.959+0000 7f893e7fc700 1 -- 172.21.15.187:0/888698980 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940011b40 con 0x7f8934004e50 2020-11-20T12:21:45.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.959+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895009e2a0 con 0x7f8934004e50 2020-11-20T12:21:45.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.959+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89500b1a40 con 0x7f8934004e50 2020-11-20T12:21:45.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.960+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8950061a20 con 0x7f8934004e50 2020-11-20T12:21:45.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.961+0000 7f893e7fc700 1 -- 172.21.15.187:0/888698980 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8940013400 con 0x7f8934004e50 2020-11-20T12:21:45.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.961+0000 7f893e7fc700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8924056d20 0x7f89240591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.961+0000 7f893e7fc700 1 -- 172.21.15.187:0/888698980 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f894006ffa0 con 0x7f8934004e50 2020-11-20T12:21:45.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.962+0000 7f894dd9b700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8924056d20 0x7f89240591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.962+0000 7f894dd9b700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8924056d20 0x7f89240591e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f89440013b0 tx=0x7f8944009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:45.965+0000 7f893e7fc700 1 -- 172.21.15.187:0/888698980 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8940045310 con 0x7f8934004e50 2020-11-20T12:21:46.157 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.154+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f89501488b0 con 0x7f8934004e50 2020-11-20T12:21:46.158 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.155+0000 7f893e7fc700 1 -- 172.21.15.187:0/888698980 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v159) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8940048930 con 0x7f8934004e50 2020-11-20T12:21:46.159 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 1 2020-11-20T12:21:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.157+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8924056d20 msgr2=0x7f89240591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.161 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.157+0000 7f894f59e700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8924056d20 0x7f89240591e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f89440013b0 tx=0x7f8944009040).stop 2020-11-20T12:21:46.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.157+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 msgr2=0x7f895009adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.157+0000 7f894f59e700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f895009adc0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f8940006fd0 tx=0x7f8940008040).stop 2020-11-20T12:21:46.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.158+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 shutdown_connections 2020-11-20T12:21:46.162 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.158+0000 7f894f59e700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8924056d20 0x7f89240591e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.158+0000 7f894f59e700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89501428b0 0x7f8950142d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.158+0000 7f894f59e700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8934004e50 0x7f895009adc0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.158+0000 7f894f59e700 1 --2- 172.21.15.187:0/888698980 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f895009db50 0x7f8950143990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.158+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 >> 172.21.15.187:0/888698980 conn(0x7f89500bab30 msgr2=0x7f89501449a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:46.163 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.158+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 shutdown_connections 2020-11-20T12:21:46.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.158+0000 7f894f59e700 1 -- 172.21.15.187:0/888698980 wait complete. 2020-11-20T12:21:46.183 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:21:46.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.421+0000 7f4a5f586700 1 Processor -- start 2020-11-20T12:21:46.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.421+0000 7f4a5f586700 1 -- start start 2020-11-20T12:21:46.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.422+0000 7f4a5f586700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4a58145cc0 0x7f4a58066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.422+0000 7f4a5f586700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a581487e0 0x7f4a58148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.422+0000 7f4a5f586700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5814c170 con 0x7f4a580683f0 2020-11-20T12:21:46.426 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.422+0000 7f4a5f586700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5814c410 con 0x7f4a58145cc0 2020-11-20T12:21:46.427 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.422+0000 7f4a5f586700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5814c6b0 con 0x7f4a581487e0 2020-11-20T12:21:46.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.423+0000 7f4a5dd83700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4a580683f0 0x7f4a58065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51358/0 (socket says 172.21.15.187:51358) 2020-11-20T12:21:46.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.423+0000 7f4a5dd83700 1 -- 172.21.15.187:0/3260706979 learned_addr learned my addr 172.21.15.187:0/3260706979 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:21:46.428 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.423+0000 7f4a5e584700 1 --2- 172.21.15.187:0/3260706979 >> v2:172.21.15.187:3300/0 conn(0x7f4a58145cc0 0x7f4a58066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.423+0000 7f4a5ed85700 1 --2- 172.21.15.187:0/3260706979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a581487e0 0x7f4a58148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5e584700 1 -- 172.21.15.187:0/3260706979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a581487e0 msgr2=0x7f4a58148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:46.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5e584700 1 --2- 172.21.15.187:0/3260706979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a581487e0 0x7f4a58148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5e584700 1 -- 172.21.15.187:0/3260706979 >> v1:172.21.15.187:6789/0 conn(0x7f4a580683f0 legacy=0x7f4a58065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.429 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5e584700 1 -- 172.21.15.187:0/3260706979 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a580658a0 con 0x7f4a58145cc0 2020-11-20T12:21:46.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5d582700 1 -- 172.21.15.187:0/3260706979 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771140569 0 0) 0x7f4a5814c170 con 0x7f4a580683f0 2020-11-20T12:21:46.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5e584700 1 --2- 172.21.15.187:0/3260706979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a58066040 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7f4a5400b9a0 tx=0x7f4a54006df0).ready entity=mon.1 client_cookie=d7ecd91e9d94cc4a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:46.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5d582700 1 -- 172.21.15.187:0/3260706979 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a540097b0 con 0x7f4a58145cc0 2020-11-20T12:21:46.430 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5d582700 1 -- 172.21.15.187:0/3260706979 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a54009910 con 0x7f4a58145cc0 2020-11-20T12:21:46.431 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.424+0000 7f4a5d582700 1 -- 172.21.15.187:0/3260706979 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54009c90 con 0x7f4a58145cc0 2020-11-20T12:21:46.432 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.425+0000 7f4a5f586700 1 -- 172.21.15.187:0/3260706979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 msgr2=0x7f4a58066040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.425+0000 7f4a5f586700 1 --2- 172.21.15.187:0/3260706979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a58066040 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7f4a5400b9a0 tx=0x7f4a54006df0).stop 2020-11-20T12:21:46.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.425+0000 7f4a5f586700 1 -- 172.21.15.187:0/3260706979 shutdown_connections 2020-11-20T12:21:46.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.425+0000 7f4a5f586700 1 --2- 172.21.15.187:0/3260706979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a581487e0 0x7f4a58148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.433 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.425+0000 7f4a5f586700 1 --2- 172.21.15.187:0/3260706979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a58066040 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.425+0000 7f4a5f586700 1 -- 172.21.15.187:0/3260706979 >> 172.21.15.187:0/3260706979 conn(0x7f4a580bab30 msgr2=0x7f4a580baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:46.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.425+0000 7f4a5f586700 1 -- 172.21.15.187:0/3260706979 shutdown_connections 2020-11-20T12:21:46.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.425+0000 7f4a5f586700 1 -- 172.21.15.187:0/3260706979 wait complete. 2020-11-20T12:21:46.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.427+0000 7f4a5f586700 1 Processor -- start 2020-11-20T12:21:46.434 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.427+0000 7f4a5f586700 1 -- start start 2020-11-20T12:21:46.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.428+0000 7f4a5f586700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a580683f0 0x7f4a58148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.428+0000 7f4a5f586700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a5809f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.428+0000 7f4a5f586700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a581487e0 0x7f4a5809fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.428+0000 7f4a5f586700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5814b860 con 0x7f4a581487e0 2020-11-20T12:21:46.435 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.428+0000 7f4a5f586700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5814bb00 con 0x7f4a58145cc0 2020-11-20T12:21:46.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.428+0000 7f4a5f586700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5814ddf0 con 0x7f4a580683f0 2020-11-20T12:21:46.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.428+0000 7f4a5dd83700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a5809f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5dd83700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a5809f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54758/0 (socket says 172.21.15.187:54758) 2020-11-20T12:21:46.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5dd83700 1 -- 172.21.15.187:0/250844590 learned_addr learned my addr 172.21.15.187:0/250844590 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:46.436 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5ed85700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a581487e0 0x7f4a5809fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5e584700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a580683f0 0x7f4a58148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5dd83700 1 -- 172.21.15.187:0/250844590 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a580683f0 msgr2=0x7f4a58148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5dd83700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a580683f0 0x7f4a58148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.437 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5dd83700 1 -- 172.21.15.187:0/250844590 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a581487e0 msgr2=0x7f4a5809fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5dd83700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a581487e0 0x7f4a5809fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.438 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5dd83700 1 -- 172.21.15.187:0/250844590 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5814e290 con 0x7f4a58145cc0 2020-11-20T12:21:46.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.429+0000 7f4a5dd83700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a5809f3a0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f4a4800c720 tx=0x7f4a48007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:46.439 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.430+0000 7f4a467fc700 1 -- 172.21.15.187:0/250844590 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a480072e0 con 0x7f4a58145cc0 2020-11-20T12:21:46.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.430+0000 7f4a467fc700 1 -- 172.21.15.187:0/250844590 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a48007440 con 0x7f4a58145cc0 2020-11-20T12:21:46.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.430+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5814d590 con 0x7f4a58145cc0 2020-11-20T12:21:46.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.430+0000 7f4a467fc700 1 -- 172.21.15.187:0/250844590 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48005800 con 0x7f4a58145cc0 2020-11-20T12:21:46.440 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.430+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5814db50 con 0x7f4a58145cc0 2020-11-20T12:21:46.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.431+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a58061a20 con 0x7f4a58145cc0 2020-11-20T12:21:46.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.432+0000 7f4a467fc700 1 -- 172.21.15.187:0/250844590 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4a48010070 con 0x7f4a58145cc0 2020-11-20T12:21:46.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.432+0000 7f4a467fc700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a30056d50 0x7f4a30059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.432+0000 7f4a467fc700 1 -- 172.21.15.187:0/250844590 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4a4806f600 con 0x7f4a58145cc0 2020-11-20T12:21:46.441 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.433+0000 7f4a5e584700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a30056d50 0x7f4a30059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.433+0000 7f4a5e584700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a30056d50 0x7f4a30059210 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f4a5400b150 tx=0x7f4a54015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:46.442 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.436+0000 7f4a467fc700 1 -- 172.21.15.187:0/250844590 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4a48044a50 con 0x7f4a58145cc0 2020-11-20T12:21:46.627 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.623+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4a5814d720 con 0x7f4a58145cc0 2020-11-20T12:21:46.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.624+0000 7f4a467fc700 1 -- 172.21.15.187:0/250844590 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v159) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4a5814d720 con 0x7f4a58145cc0 2020-11-20T12:21:46.628 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:46.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a30056d50 msgr2=0x7f4a30059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a30056d50 0x7f4a30059210 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f4a5400b150 tx=0x7f4a54015000).stop 2020-11-20T12:21:46.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 msgr2=0x7f4a5809f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a5809f3a0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f4a4800c720 tx=0x7f4a48007ac0).stop 2020-11-20T12:21:46.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 shutdown_connections 2020-11-20T12:21:46.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4a30056d50 0x7f4a30059210 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a581487e0 0x7f4a5809fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4a580683f0 0x7f4a58148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 --2- 172.21.15.187:0/250844590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a58145cc0 0x7f4a5809f3a0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 >> 172.21.15.187:0/250844590 conn(0x7f4a580bab30 msgr2=0x7f4a58147b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:46.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 shutdown_connections 2020-11-20T12:21:46.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.627+0000 7f4a5f586700 1 -- 172.21.15.187:0/250844590 wait complete. 2020-11-20T12:21:46.655 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:21:46.900 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.897+0000 7f3d6bfff700 1 Processor -- start 2020-11-20T12:21:46.901 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.897+0000 7f3d6bfff700 1 -- start start 2020-11-20T12:21:46.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.898+0000 7f3d6bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3d6c0b1380 0x7f3d6c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.898+0000 7f3d6bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1760 0x7f3d6c065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.902 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.898+0000 7f3d6bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3d6c065680 0x7f3d6c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.898+0000 7f3d6bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6c150390 con 0x7f3d6c065680 2020-11-20T12:21:46.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.898+0000 7f3d6bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6c150630 con 0x7f3d6c0b1380 2020-11-20T12:21:46.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.898+0000 7f3d6bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c068510 con 0x7f3d6c0b1760 2020-11-20T12:21:46.903 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3d6c0b1380 0x7f3d6c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3d6c0b1380 0x7f3d6c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54764/0 (socket says 172.21.15.187:54764) 2020-11-20T12:21:46.904 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 -- 172.21.15.187:0/842923990 learned_addr learned my addr 172.21.15.187:0/842923990 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:46.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 -- 172.21.15.187:0/842923990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1760 msgr2=0x7f3d6c065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:46.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6b7fe700 1 --2- 172.21.15.187:0/842923990 >> v2:172.21.15.5:3300/0 conn(0x7f3d6c065680 0x7f3d6c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6a7fc700 1 --2- 172.21.15.187:0/842923990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1760 0x7f3d6c065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.907 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 --2- 172.21.15.187:0/842923990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1760 0x7f3d6c065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 -- 172.21.15.187:0/842923990 >> v2:172.21.15.5:3300/0 conn(0x7f3d6c065680 msgr2=0x7f3d6c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:46.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 --2- 172.21.15.187:0/842923990 >> v2:172.21.15.5:3300/0 conn(0x7f3d6c065680 0x7f3d6c061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 -- 172.21.15.187:0/842923990 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c150fc0 con 0x7f3d6c0b1380 2020-11-20T12:21:46.908 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.899+0000 7f3d6affd700 1 --2- 172.21.15.187:0/842923990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c0b1380 0x7f3d6c150ad0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f3d6c0b85e0 tx=0x7f3d5c006eb0).ready entity=mon.1 client_cookie=c49a993df11e6014 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:46.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.900+0000 7f3d69ffb700 1 -- 172.21.15.187:0/842923990 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c0098d0 con 0x7f3d6c0b1380 2020-11-20T12:21:46.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.900+0000 7f3d69ffb700 1 -- 172.21.15.187:0/842923990 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5c009a30 con 0x7f3d6c0b1380 2020-11-20T12:21:46.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.900+0000 7f3d69ffb700 1 -- 172.21.15.187:0/842923990 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c009db0 con 0x7f3d6c0b1380 2020-11-20T12:21:46.909 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.900+0000 7f3d6bfff700 1 -- 172.21.15.187:0/842923990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c0b1380 msgr2=0x7f3d6c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.900+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/842923990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c0b1380 0x7f3d6c150ad0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f3d6c0b85e0 tx=0x7f3d5c006eb0).stop 2020-11-20T12:21:46.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.900+0000 7f3d6bfff700 1 -- 172.21.15.187:0/842923990 shutdown_connections 2020-11-20T12:21:46.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.901+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/842923990 >> v2:172.21.15.5:3300/0 conn(0x7f3d6c065680 0x7f3d6c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.910 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.901+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/842923990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c0b1380 0x7f3d6c150ad0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.901+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/842923990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1760 0x7f3d6c065190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:46.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.901+0000 7f3d6bfff700 1 -- 172.21.15.187:0/842923990 >> 172.21.15.187:0/842923990 conn(0x7f3d6c0bab30 msgr2=0x7f3d6c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:46.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.901+0000 7f3d6bfff700 1 -- 172.21.15.187:0/842923990 shutdown_connections 2020-11-20T12:21:46.911 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.901+0000 7f3d6bfff700 1 -- 172.21.15.187:0/842923990 wait complete. 2020-11-20T12:21:46.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.903+0000 7f3d6bfff700 1 Processor -- start 2020-11-20T12:21:46.912 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.903+0000 7f3d6bfff700 1 -- start start 2020-11-20T12:21:46.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1380 0x7f3d6c0b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d6c0b1760 0x7f3d6c09cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.914 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c09d380 0x7f3d6c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c09f9f0 con 0x7f3d6c0b1760 2020-11-20T12:21:46.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c0b3360 con 0x7f3d6c09d380 2020-11-20T12:21:46.915 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c09d1d0 con 0x7f3d6c0b1380 2020-11-20T12:21:46.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c09d380 0x7f3d6c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6b7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c09d380 0x7f3d6c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54774/0 (socket says 172.21.15.187:54774) 2020-11-20T12:21:46.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.904+0000 7f3d6b7fe700 1 -- 172.21.15.187:0/3732787060 learned_addr learned my addr 172.21.15.187:0/3732787060 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:46.916 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d6b7fe700 1 -- 172.21.15.187:0/3732787060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1380 msgr2=0x7f3d6c0b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d6b7fe700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1380 0x7f3d6c0b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:46.917 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d6b7fe700 1 -- 172.21.15.187:0/3732787060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d6c0b1760 msgr2=0x7f3d6c09cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:46.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d6b7fe700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d6c0b1760 0x7f3d6c09cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:46.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d6b7fe700 1 -- 172.21.15.187:0/3732787060 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c160880 con 0x7f3d6c09d380 2020-11-20T12:21:46.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d6b7fe700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c09d380 0x7f3d6c0b6a90 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7f3d6400cb50 tx=0x7f3d64008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:46.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d52ffd700 1 -- 172.21.15.187:0/3732787060 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64007900 con 0x7f3d6c09d380 2020-11-20T12:21:46.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d52ffd700 1 -- 172.21.15.187:0/3732787060 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d64007a60 con 0x7f3d6c09d380 2020-11-20T12:21:46.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c160b20 con 0x7f3d6c09d380 2020-11-20T12:21:46.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d52ffd700 1 -- 172.21.15.187:0/3732787060 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64007de0 con 0x7f3d6c09d380 2020-11-20T12:21:46.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.905+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c149760 con 0x7f3d6c09d380 2020-11-20T12:21:46.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.906+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6c0b7140 con 0x7f3d6c09d380 2020-11-20T12:21:46.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.907+0000 7f3d52ffd700 1 -- 172.21.15.187:0/3732787060 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3d64011620 con 0x7f3d6c09d380 2020-11-20T12:21:46.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.907+0000 7f3d52ffd700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3d48056d20 0x7f3d480591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:46.920 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.908+0000 7f3d52ffd700 1 -- 172.21.15.187:0/3732787060 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3d64070410 con 0x7f3d6c09d380 2020-11-20T12:21:46.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.908+0000 7f3d6affd700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3d48056d20 0x7f3d480591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:46.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.908+0000 7f3d6affd700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3d48056d20 0x7f3d480591e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f3d6c0b85e0 tx=0x7f3d5c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:46.921 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:46.911+0000 7f3d52ffd700 1 -- 172.21.15.187:0/3732787060 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3d6c0b7140 con 0x7f3d6c09d380 2020-11-20T12:21:47.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.103+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3d6c0b7140 con 0x7f3d6c09d380 2020-11-20T12:21:47.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.104+0000 7f3d52ffd700 1 -- 172.21.15.187:0/3732787060 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v159) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3d6c0b7140 con 0x7f3d6c09d380 2020-11-20T12:21:47.108 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:47.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3d48056d20 msgr2=0x7f3d480591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3d48056d20 0x7f3d480591e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f3d6c0b85e0 tx=0x7f3d5c015000).stop 2020-11-20T12:21:47.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c09d380 msgr2=0x7f3d6c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c09d380 0x7f3d6c0b6a90 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7f3d6400cb50 tx=0x7f3d64008040).stop 2020-11-20T12:21:47.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 shutdown_connections 2020-11-20T12:21:47.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3d48056d20 0x7f3d480591e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d6c09d380 0x7f3d6c0b6a90 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d6c0b1380 0x7f3d6c0b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:47.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 --2- 172.21.15.187:0/3732787060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d6c0b1760 0x7f3d6c09cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:47.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.106+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 >> 172.21.15.187:0/3732787060 conn(0x7f3d6c0bab30 msgr2=0x7f3d6c14a820 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:47.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.107+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 shutdown_connections 2020-11-20T12:21:47.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.107+0000 7f3d6bfff700 1 -- 172.21.15.187:0/3732787060 wait complete. 2020-11-20T12:21:47.135 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:21:47.389 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.386+0000 7f3ddbfff700 1 Processor -- start 2020-11-20T12:21:47.390 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.386+0000 7f3ddbfff700 1 -- start start 2020-11-20T12:21:47.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.387+0000 7f3ddbfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3ddc145cc0 0x7f3ddc066040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.387+0000 7f3ddbfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ddc0683f0 0x7f3ddc065190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.387+0000 7f3ddbfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ddc14c170 con 0x7f3ddc065680 2020-11-20T12:21:47.391 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.387+0000 7f3ddbfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ddc1497f0 con 0x7f3ddc145cc0 2020-11-20T12:21:47.392 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.387+0000 7f3ddbfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ddc14c410 con 0x7f3ddc0683f0 2020-11-20T12:21:47.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.388+0000 7f3dd27fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ddc0683f0 0x7f3ddc065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.388+0000 7f3dd27fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3ddc0683f0 0x7f3ddc065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54782/0 (socket says 172.21.15.187:54782) 2020-11-20T12:21:47.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.388+0000 7f3dd27fc700 1 -- 172.21.15.187:0/1217361359 learned_addr learned my addr 172.21.15.187:0/1217361359 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:47.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.388+0000 7f3ddaffd700 1 --2- 172.21.15.187:0/1217361359 >> v2:172.21.15.5:3300/0 conn(0x7f3ddc145cc0 0x7f3ddc066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.393 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.388+0000 7f3dd27fc700 1 -- 172.21.15.187:0/1217361359 >> v2:172.21.15.5:3300/0 conn(0x7f3ddc145cc0 msgr2=0x7f3ddc066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:47.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.389+0000 7f3dd27fc700 1 --2- 172.21.15.187:0/1217361359 >> v2:172.21.15.5:3300/0 conn(0x7f3ddc145cc0 0x7f3ddc066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.389+0000 7f3dd27fc700 1 -- 172.21.15.187:0/1217361359 >> v1:172.21.15.5:6789/0 conn(0x7f3ddc065680 legacy=0x7f3ddc061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.389+0000 7f3dd27fc700 1 -- 172.21.15.187:0/1217361359 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc14c6b0 con 0x7f3ddc0683f0 2020-11-20T12:21:47.394 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.389+0000 7f3dd27fc700 1 --2- 172.21.15.187:0/1217361359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc0683f0 0x7f3ddc065190 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7f3ddc0674b0 tx=0x7f3dc4006eb0).ready entity=mon.1 client_cookie=c144ac2c01f79f74 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.395 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3dda7fc700 1 -- 172.21.15.187:0/1217361359 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc40098d0 con 0x7f3ddc0683f0 2020-11-20T12:21:47.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3dda7fc700 1 -- 172.21.15.187:0/1217361359 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc4009a30 con 0x7f3ddc0683f0 2020-11-20T12:21:47.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3dda7fc700 1 -- 172.21.15.187:0/1217361359 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4009db0 con 0x7f3ddc0683f0 2020-11-20T12:21:47.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3ddbfff700 1 -- 172.21.15.187:0/1217361359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc0683f0 msgr2=0x7f3ddc065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.396 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/1217361359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc0683f0 0x7f3ddc065190 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7f3ddc0674b0 tx=0x7f3dc4006eb0).stop 2020-11-20T12:21:47.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3ddbfff700 1 -- 172.21.15.187:0/1217361359 shutdown_connections 2020-11-20T12:21:47.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/1217361359 >> v2:172.21.15.5:3300/0 conn(0x7f3ddc145cc0 0x7f3ddc066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/1217361359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc0683f0 0x7f3ddc065190 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.390+0000 7f3ddbfff700 1 -- 172.21.15.187:0/1217361359 >> 172.21.15.187:0/1217361359 conn(0x7f3ddc0bab30 msgr2=0x7f3ddc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:47.397 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.391+0000 7f3ddbfff700 1 -- 172.21.15.187:0/1217361359 shutdown_connections 2020-11-20T12:21:47.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.391+0000 7f3ddbfff700 1 -- 172.21.15.187:0/1217361359 wait complete. 2020-11-20T12:21:47.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.392+0000 7f3ddbfff700 1 Processor -- start 2020-11-20T12:21:47.398 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.392+0000 7f3ddbfff700 1 -- start start 2020-11-20T12:21:47.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.393+0000 7f3ddbfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc065680 0x7f3ddc142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.393+0000 7f3ddbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ddc145cc0 0x7f3ddc09ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.393+0000 7f3ddbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ddc14a0f0 0x7f3ddc0a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.400 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.393+0000 7f3ddbfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc14b0d0 con 0x7f3ddc145cc0 2020-11-20T12:21:47.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.393+0000 7f3ddbfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc14b330 con 0x7f3ddc065680 2020-11-20T12:21:47.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.393+0000 7f3ddbfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc14b5d0 con 0x7f3ddc14a0f0 2020-11-20T12:21:47.401 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc065680 0x7f3ddc142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddaffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc065680 0x7f3ddc142120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54788/0 (socket says 172.21.15.187:54788) 2020-11-20T12:21:47.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddaffd700 1 -- 172.21.15.187:0/2083382436 learned_addr learned my addr 172.21.15.187:0/2083382436 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:47.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3dd27fc700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ddc145cc0 0x7f3ddc09ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddb7fe700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ddc14a0f0 0x7f3ddc0a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.402 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddaffd700 1 -- 172.21.15.187:0/2083382436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ddc14a0f0 msgr2=0x7f3ddc0a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddaffd700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ddc14a0f0 0x7f3ddc0a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddaffd700 1 -- 172.21.15.187:0/2083382436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ddc145cc0 msgr2=0x7f3ddc09ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddaffd700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ddc145cc0 0x7f3ddc09ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.394+0000 7f3ddaffd700 1 -- 172.21.15.187:0/2083382436 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc0b24d0 con 0x7f3ddc065680 2020-11-20T12:21:47.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.395+0000 7f3ddaffd700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc065680 0x7f3ddc142120 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7f3dcc00d8e0 tx=0x7f3dcc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.395+0000 7f3dd37fe700 1 -- 172.21.15.187:0/2083382436 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc0125f0 con 0x7f3ddc065680 2020-11-20T12:21:47.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.395+0000 7f3dd37fe700 1 -- 172.21.15.187:0/2083382436 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc012750 con 0x7f3ddc065680 2020-11-20T12:21:47.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.395+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ddc142610 con 0x7f3ddc065680 2020-11-20T12:21:47.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.395+0000 7f3dd37fe700 1 -- 172.21.15.187:0/2083382436 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc012af0 con 0x7f3ddc065680 2020-11-20T12:21:47.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.395+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ddc0b3940 con 0x7f3ddc065680 2020-11-20T12:21:47.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.396+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ddc061a20 con 0x7f3ddc065680 2020-11-20T12:21:47.405 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.397+0000 7f3dd37fe700 1 -- 172.21.15.187:0/2083382436 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3dcc013480 con 0x7f3ddc065680 2020-11-20T12:21:47.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.397+0000 7f3dd37fe700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3db8056d20 0x7f3db80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.397+0000 7f3dd37fe700 1 -- 172.21.15.187:0/2083382436 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3dcc06fe00 con 0x7f3ddc065680 2020-11-20T12:21:47.406 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.398+0000 7f3dd27fc700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3db8056d20 0x7f3db80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.398+0000 7f3dd27fc700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3db8056d20 0x7f3db80591e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f3ddc063a20 tx=0x7f3dc4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.401+0000 7f3dd37fe700 1 -- 172.21.15.187:0/2083382436 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3dcc0451f0 con 0x7f3ddc065680 2020-11-20T12:21:47.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.587+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ddc14bdf0 con 0x7f3ddc065680 2020-11-20T12:21:47.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.588+0000 7f3dd37fe700 1 -- 172.21.15.187:0/2083382436 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v159) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3dcc012e60 con 0x7f3ddc065680 2020-11-20T12:21:47.592 INFO:teuthology.orchestra.run.smithi187.stdout:pg_num: 8 2020-11-20T12:21:47.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3db8056d20 msgr2=0x7f3db80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3db8056d20 0x7f3db80591e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f3ddc063a20 tx=0x7f3dc4015000).stop 2020-11-20T12:21:47.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc065680 msgr2=0x7f3ddc142120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc065680 0x7f3ddc142120 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7f3dcc00d8e0 tx=0x7f3dcc008040).stop 2020-11-20T12:21:47.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 shutdown_connections 2020-11-20T12:21:47.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3db8056d20 0x7f3db80591e0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3ddc14a0f0 0x7f3ddc0a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ddc065680 0x7f3ddc142120 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 --2- 172.21.15.187:0/2083382436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ddc145cc0 0x7f3ddc09ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.590+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 >> 172.21.15.187:0/2083382436 conn(0x7f3ddc0bab30 msgr2=0x7f3ddc0baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:47.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.591+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 shutdown_connections 2020-11-20T12:21:47.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.591+0000 7f3ddbfff700 1 -- 172.21.15.187:0/2083382436 wait complete. 2020-11-20T12:21:47.618 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:21:47.618 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:47.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.855+0000 7fb494937700 1 Processor -- start 2020-11-20T12:21:47.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.855+0000 7fb494937700 1 -- start start 2020-11-20T12:21:47.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.856+0000 7fb494937700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb4901484d0 0x7fb490061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.856+0000 7fb494937700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.856+0000 7fb494937700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb490066f70 0x7fb4900b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.856+0000 7fb494937700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb490067730 con 0x7fb490066f70 2020-11-20T12:21:47.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.856+0000 7fb494937700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4900b1300 con 0x7fb4901484d0 2020-11-20T12:21:47.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.856+0000 7fb494937700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900b15a0 con 0x7fb4900622c0 2020-11-20T12:21:47.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48ed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb4901484d0 0x7fb490061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48ed9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb4901484d0 0x7fb490061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54792/0 (socket says 172.21.15.187:54792) 2020-11-20T12:21:47.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48ed9d700 1 -- 172.21.15.187:0/53876584 learned_addr learned my addr 172.21.15.187:0/53876584 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:47.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48f59e700 1 --2- 172.21.15.187:0/53876584 >> v2:172.21.15.5:3300/0 conn(0x7fb490066f70 0x7fb4900b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48ed9d700 1 -- 172.21.15.187:0/53876584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 msgr2=0x7fb4900b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:47.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48e59c700 1 --2- 172.21.15.187:0/53876584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48ed9d700 1 --2- 172.21.15.187:0/53876584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48ed9d700 1 -- 172.21.15.187:0/53876584 >> v2:172.21.15.5:3300/0 conn(0x7fb490066f70 msgr2=0x7fb4900b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:47.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48ed9d700 1 --2- 172.21.15.187:0/53876584 >> v2:172.21.15.5:3300/0 conn(0x7fb490066f70 0x7fb4900b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48ed9d700 1 -- 172.21.15.187:0/53876584 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900b1840 con 0x7fb4901484d0 2020-11-20T12:21:47.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.857+0000 7fb48f59e700 1 --2- 172.21.15.187:0/53876584 >> v2:172.21.15.5:3300/0 conn(0x7fb490066f70 0x7fb4900b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:47.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.858+0000 7fb48ed9d700 1 --2- 172.21.15.187:0/53876584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4901484d0 0x7fb490061dd0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7fb4900b85e0 tx=0x7fb480006eb0).ready entity=mon.1 client_cookie=faaf31fa9d3bb6d5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.858+0000 7fb48dd9b700 1 -- 172.21.15.187:0/53876584 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4800098d0 con 0x7fb4901484d0 2020-11-20T12:21:47.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.858+0000 7fb48dd9b700 1 -- 172.21.15.187:0/53876584 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb480009a30 con 0x7fb4901484d0 2020-11-20T12:21:47.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.858+0000 7fb48dd9b700 1 -- 172.21.15.187:0/53876584 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480009db0 con 0x7fb4901484d0 2020-11-20T12:21:47.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 -- 172.21.15.187:0/53876584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4901484d0 msgr2=0x7fb490061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 --2- 172.21.15.187:0/53876584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4901484d0 0x7fb490061dd0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7fb4900b85e0 tx=0x7fb480006eb0).stop 2020-11-20T12:21:47.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 -- 172.21.15.187:0/53876584 shutdown_connections 2020-11-20T12:21:47.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 --2- 172.21.15.187:0/53876584 >> v2:172.21.15.5:3300/0 conn(0x7fb490066f70 0x7fb4900b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 --2- 172.21.15.187:0/53876584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4901484d0 0x7fb490061dd0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 --2- 172.21.15.187:0/53876584 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 -- 172.21.15.187:0/53876584 >> 172.21.15.187:0/53876584 conn(0x7fb4900bab30 msgr2=0x7fb4900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:47.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 -- 172.21.15.187:0/53876584 shutdown_connections 2020-11-20T12:21:47.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.859+0000 7fb494937700 1 -- 172.21.15.187:0/53876584 wait complete. 2020-11-20T12:21:47.868 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.860+0000 7fb494937700 1 Processor -- start 2020-11-20T12:21:47.870 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.861+0000 7fb494937700 1 -- start start 2020-11-20T12:21:47.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.861+0000 7fb494937700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.861+0000 7fb494937700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb490066f70 0x7fb49009e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.861+0000 7fb494937700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4901484d0 0x7fb49009dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.861+0000 7fb494937700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49009eca0 con 0x7fb4901484d0 2020-11-20T12:21:47.871 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.861+0000 7fb494937700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb490158830 con 0x7fb490066f70 2020-11-20T12:21:47.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.861+0000 7fb494937700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb490158ab0 con 0x7fb4900622c0 2020-11-20T12:21:47.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.862+0000 7fb48e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb490066f70 0x7fb49009e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.862+0000 7fb48e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb490066f70 0x7fb49009e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54802/0 (socket says 172.21.15.187:54802) 2020-11-20T12:21:47.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.862+0000 7fb48e59c700 1 -- 172.21.15.187:0/3938616361 learned_addr learned my addr 172.21.15.187:0/3938616361 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:47.872 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.862+0000 7fb48f59e700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4901484d0 0x7fb49009dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.862+0000 7fb48ed9d700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb48e59c700 1 -- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 msgr2=0x7fb4900a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb48e59c700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.873 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb48e59c700 1 -- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4901484d0 msgr2=0x7fb49009dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb48e59c700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4901484d0 0x7fb49009dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb48e59c700 1 -- 172.21.15.187:0/3938616361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49009ef10 con 0x7fb490066f70 2020-11-20T12:21:47.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb48ed9d700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:47.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb48e59c700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb490066f70 0x7fb49009e7b0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7fb48400c920 tx=0x7fb484008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb476ffd700 1 -- 172.21.15.187:0/3938616361 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb484007710 con 0x7fb490066f70 2020-11-20T12:21:47.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb476ffd700 1 -- 172.21.15.187:0/3938616361 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb484007870 con 0x7fb490066f70 2020-11-20T12:21:47.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb490158d50 con 0x7fb490066f70 2020-11-20T12:21:47.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb476ffd700 1 -- 172.21.15.187:0/3938616361 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb484007bf0 con 0x7fb490066f70 2020-11-20T12:21:47.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.863+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49009e210 con 0x7fb490066f70 2020-11-20T12:21:47.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.864+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49009f0a0 con 0x7fb490066f70 2020-11-20T12:21:47.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.865+0000 7fb476ffd700 1 -- 172.21.15.187:0/3938616361 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb48401b460 con 0x7fb490066f70 2020-11-20T12:21:47.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.866+0000 7fb476ffd700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb46c056d20 0x7fb46c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.866+0000 7fb476ffd700 1 -- 172.21.15.187:0/3938616361 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb484070130 con 0x7fb490066f70 2020-11-20T12:21:47.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.866+0000 7fb48ed9d700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb46c056d20 0x7fb46c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.876 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.867+0000 7fb48ed9d700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb46c056d20 0x7fb46c0591e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7fb48000b1b0 tx=0x7fb480015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:47.869+0000 7fb476ffd700 1 -- 172.21.15.187:0/3938616361 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb484045520 con 0x7fb490066f70 2020-11-20T12:21:48.114 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.110+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb490061a20 con 0x7fb490066f70 2020-11-20T12:21:48.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.111+0000 7fb476ffd700 1 -- 172.21.15.187:0/3938616361 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2145 (secure 0 0 0) 0x7fb484048b40 con 0x7fb490066f70 2020-11-20T12:21:48.115 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:48.116 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":172,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:33.149592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:21:48.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb46c056d20 msgr2=0x7fb46c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb46c056d20 0x7fb46c0591e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7fb48000b1b0 tx=0x7fb480015000).stop 2020-11-20T12:21:48.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb490066f70 msgr2=0x7fb49009e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb490066f70 0x7fb49009e7b0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7fb48400c920 tx=0x7fb484008040).stop 2020-11-20T12:21:48.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 shutdown_connections 2020-11-20T12:21:48.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb46c056d20 0x7fb46c0591e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4901484d0 0x7fb49009dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4900622c0 0x7fb4900a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 --2- 172.21.15.187:0/3938616361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb490066f70 0x7fb49009e7b0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 >> 172.21.15.187:0/3938616361 conn(0x7fb4900bab30 msgr2=0x7fb4900b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:48.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 shutdown_connections 2020-11-20T12:21:48.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:48.114+0000 7fb494937700 1 -- 172.21.15.187:0/3938616361 wait complete. 2020-11-20T12:21:48.121 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 172 2020-11-20T12:21:48.344 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'ever_allowed_features': 2, 'explicitly_allowed_features': 2, 'created': '2020-11-20T12:21:22.646683+0000', 'modified': '2020-11-20T12:21:33.149592+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10018}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10018': {'gid': 10018, 'name': 'a', 'rank': 0, 'incarnation': 168, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/1915143367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 1915143367}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1915143367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:21:48.344 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:21:48.345 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:21:48.625 INFO:teuthology.orchestra.run.smithi187.stdout:{"cluster_fsid":"099efcab-1650-4909-9b03-145da903b26d","whoami":0,"id":10018,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":25.951152240999999,"mdsmap_epoch":172,"osdmap_epoch":159,"osdmap_epoch_barrier":159,"uptime":32.299333392000001} 2020-11-20T12:21:48.626 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "099efcab-1650-4909-9b03-145da903b26d", "fs_name": "cephfs", "id": 10018, "mdsmap_epoch": 172, "osdmap_epoch": 159, "osdmap_epoch_barrier": 159, "rank_uptime": 25.951152241, "state": "up:active", "uptime": 32.299333392, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:21:48.627 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2020-11-20T12:21:48.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:48.627 INFO:teuthology.orchestra.run.smithi187:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-11-20T12:21:48.678 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T12:21:48.678 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:21:48.861 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:21:48.862 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:21:48.900 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2020-11-20T12:21:48.900 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2020-11-20T12:21:48.907 INFO:tasks.cephfs.kernel_mount:mount command passed 2020-11-20T12:21:48.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:21:48.909 INFO:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-11-20T12:21:48.999 INFO:tasks.cephfs.mount:Validating 8388608 bytes from snapdir/.snap/snap1/subdir/file_a 2020-11-20T12:21:49.000 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:21:49.001 INFO:teuthology.orchestra.run.smithi187:> import zlib 2020-11-20T12:21:49.001 INFO:teuthology.orchestra.run.smithi187:> path = "/home/ubuntu/cephtest/mnt.0/snapdir/.snap/snap1/subdir/file_a" 2020-11-20T12:21:49.002 INFO:teuthology.orchestra.run.smithi187:> with open(path, '"'"'r'"'"') as f: 2020-11-20T12:21:49.002 INFO:teuthology.orchestra.run.smithi187:> bytes = f.read() 2020-11-20T12:21:49.002 INFO:teuthology.orchestra.run.smithi187:> if len(bytes) != 8388608: 2020-11-20T12:21:49.003 INFO:teuthology.orchestra.run.smithi187:> raise RuntimeError("Bad length {0} vs. expected {1}".format( 2020-11-20T12:21:49.004 INFO:teuthology.orchestra.run.smithi187:> len(bytes), 8388608 2020-11-20T12:21:49.005 INFO:teuthology.orchestra.run.smithi187:> )) 2020-11-20T12:21:49.005 INFO:teuthology.orchestra.run.smithi187:> for i, b in enumerate(bytes): 2020-11-20T12:21:49.006 INFO:teuthology.orchestra.run.smithi187:> val = zlib.crc32(str(i).encode('"'"'utf-8'"'"')) & 7 2020-11-20T12:21:49.007 INFO:teuthology.orchestra.run.smithi187:> if b != chr(val): 2020-11-20T12:21:49.007 INFO:teuthology.orchestra.run.smithi187:> raise RuntimeError("Bad data at offset {0}".format(i)) 2020-11-20T12:21:49.007 INFO:teuthology.orchestra.run.smithi187:> ' 2020-11-20T12:21:54.501 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:21:54.515 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:21:55.396 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:21:55.397 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rmdir snapdir/.snap/snap1) 2020-11-20T12:21:55.529 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:55.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.774+0000 7fabebfff700 1 Processor -- start 2020-11-20T12:21:55.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.774+0000 7fabebfff700 1 -- start start 2020-11-20T12:21:55.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabebfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fabec1484d0 0x7fabec061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabebfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabebfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fabec066f70 0x7fabec0b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabebfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabec067730 con 0x7fabec066f70 2020-11-20T12:21:55.780 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabebfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabec0b1300 con 0x7fabec1484d0 2020-11-20T12:21:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabebfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0b15a0 con 0x7fabec0622c0 2020-11-20T12:21:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabeaffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fabec1484d0 0x7fabec061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabeaffd700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fabec1484d0 0x7fabec061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54814/0 (socket says 172.21.15.187:54814) 2020-11-20T12:21:55.781 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.776+0000 7fabeaffd700 1 -- 172.21.15.187:0/3046623718 learned_addr learned my addr 172.21.15.187:0/3046623718 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:55.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabeb7fe700 1 --2- 172.21.15.187:0/3046623718 >> v2:172.21.15.5:3300/0 conn(0x7fabec066f70 0x7fabec0b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabea7fc700 1 --2- 172.21.15.187:0/3046623718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.783 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabeaffd700 1 -- 172.21.15.187:0/3046623718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 msgr2=0x7fabec0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:55.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabeaffd700 1 --2- 172.21.15.187:0/3046623718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabeaffd700 1 -- 172.21.15.187:0/3046623718 >> v2:172.21.15.5:3300/0 conn(0x7fabec066f70 msgr2=0x7fabec0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:55.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabeaffd700 1 --2- 172.21.15.187:0/3046623718 >> v2:172.21.15.5:3300/0 conn(0x7fabec066f70 0x7fabec0b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.784 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabeaffd700 1 -- 172.21.15.187:0/3046623718 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec0b1840 con 0x7fabec1484d0 2020-11-20T12:21:55.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabeb7fe700 1 --2- 172.21.15.187:0/3046623718 >> v2:172.21.15.5:3300/0 conn(0x7fabec066f70 0x7fabec0b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:55.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabeaffd700 1 --2- 172.21.15.187:0/3046623718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec061dd0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7fabec0b85e0 tx=0x7fabdc006eb0).ready entity=mon.1 client_cookie=b5ab89facb341b18 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:55.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabe9ffb700 1 -- 172.21.15.187:0/3046623718 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabdc0098d0 con 0x7fabec1484d0 2020-11-20T12:21:55.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabe9ffb700 1 -- 172.21.15.187:0/3046623718 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabdc009a30 con 0x7fabec1484d0 2020-11-20T12:21:55.785 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.777+0000 7fabe9ffb700 1 -- 172.21.15.187:0/3046623718 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabdc009db0 con 0x7fabec1484d0 2020-11-20T12:21:55.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 -- 172.21.15.187:0/3046623718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 msgr2=0x7fabec061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 --2- 172.21.15.187:0/3046623718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec061dd0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7fabec0b85e0 tx=0x7fabdc006eb0).stop 2020-11-20T12:21:55.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 -- 172.21.15.187:0/3046623718 shutdown_connections 2020-11-20T12:21:55.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 --2- 172.21.15.187:0/3046623718 >> v2:172.21.15.5:3300/0 conn(0x7fabec066f70 0x7fabec0b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.786 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 --2- 172.21.15.187:0/3046623718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec061dd0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 --2- 172.21.15.187:0/3046623718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 -- 172.21.15.187:0/3046623718 >> 172.21.15.187:0/3046623718 conn(0x7fabec0bab30 msgr2=0x7fabec0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:55.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 -- 172.21.15.187:0/3046623718 shutdown_connections 2020-11-20T12:21:55.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.778+0000 7fabebfff700 1 -- 172.21.15.187:0/3046623718 wait complete. 2020-11-20T12:21:55.787 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.780+0000 7fabebfff700 1 Processor -- start 2020-11-20T12:21:55.788 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.780+0000 7fabebfff700 1 -- start start 2020-11-20T12:21:55.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabebfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabebfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec066f70 0x7fabec0b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.790 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabebfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabebfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0b3f30 con 0x7fabec066f70 2020-11-20T12:21:55.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabebfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0b41d0 con 0x7fabec1484d0 2020-11-20T12:21:55.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabebfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0b4470 con 0x7fabec0622c0 2020-11-20T12:21:55.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabeb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.791 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabeb7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec141ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54824/0 (socket says 172.21.15.187:54824) 2020-11-20T12:21:55.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.781+0000 7fabeb7fe700 1 -- 172.21.15.187:0/1053054754 learned_addr learned my addr 172.21.15.187:0/1053054754 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:55.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabeaffd700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabeb7fe700 1 -- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 msgr2=0x7fabec0b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabea7fc700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec066f70 0x7fabec0b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabeb7fe700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabeb7fe700 1 -- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec066f70 msgr2=0x7fabec0b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabeb7fe700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec066f70 0x7fabec0b31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabeb7fe700 1 -- 172.21.15.187:0/1053054754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec0b4910 con 0x7fabec1484d0 2020-11-20T12:21:55.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabeaffd700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:55.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabeb7fe700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec141ec0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7fabe400cb50 tx=0x7fabe4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:55.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabd2ffd700 1 -- 172.21.15.187:0/1053054754 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe400a990 con 0x7fabec1484d0 2020-11-20T12:21:55.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabd2ffd700 1 -- 172.21.15.187:0/1053054754 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe400aaf0 con 0x7fabec1484d0 2020-11-20T12:21:55.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.782+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabec0b0800 con 0x7fabec1484d0 2020-11-20T12:21:55.794 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.783+0000 7fabd2ffd700 1 -- 172.21.15.187:0/1053054754 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe400ae70 con 0x7fabec1484d0 2020-11-20T12:21:55.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.783+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabec0b36e0 con 0x7fabec1484d0 2020-11-20T12:21:55.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.784+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabec0b4630 con 0x7fabec1484d0 2020-11-20T12:21:55.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.784+0000 7fabd2ffd700 1 -- 172.21.15.187:0/1053054754 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fabe4011620 con 0x7fabec1484d0 2020-11-20T12:21:55.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.785+0000 7fabd2ffd700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabc8056d50 0x7fabc8059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.785+0000 7fabd2ffd700 1 -- 172.21.15.187:0/1053054754 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fabe40704e0 con 0x7fabec1484d0 2020-11-20T12:21:55.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.785+0000 7fabeaffd700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabc8056d50 0x7fabc8059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.786+0000 7fabeaffd700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabc8056d50 0x7fabc8059210 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7fabdc00b1b0 tx=0x7fabdc017000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:55.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:55.789+0000 7fabd2ffd700 1 -- 172.21.15.187:0/1053054754 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fabe4045930 con 0x7fabec1484d0 2020-11-20T12:21:56.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.036+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabec061a20 con 0x7fabec1484d0 2020-11-20T12:21:56.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.037+0000 7fabd2ffd700 1 -- 172.21.15.187:0/1053054754 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2145 (secure 0 0 0) 0x7fabe4048f50 con 0x7fabec1484d0 2020-11-20T12:21:56.041 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:21:56.041 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":172,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:33.149592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:21:56.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabc8056d50 msgr2=0x7fabc8059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabc8056d50 0x7fabc8059210 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7fabdc00b1b0 tx=0x7fabdc017000).stop 2020-11-20T12:21:56.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 msgr2=0x7fabec141ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec141ec0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7fabe400cb50 tx=0x7fabe4008040).stop 2020-11-20T12:21:56.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 shutdown_connections 2020-11-20T12:21:56.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fabc8056d50 0x7fabc8059210 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabec1484d0 0x7fabec141ec0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fabec0622c0 0x7fabec0b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 --2- 172.21.15.187:0/1053054754 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec066f70 0x7fabec0b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 >> 172.21.15.187:0/1053054754 conn(0x7fabec0bab30 msgr2=0x7fabec14a340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:56.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.039+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 shutdown_connections 2020-11-20T12:21:56.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.040+0000 7fabebfff700 1 -- 172.21.15.187:0/1053054754 wait complete. 2020-11-20T12:21:56.048 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 172 2020-11-20T12:21:56.063 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2020-11-20T12:21:56.358 INFO:teuthology.orchestra.run.smithi187.stdout:{"message":"","return_code":0} 2020-11-20T12:21:56.359 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2020-11-20T12:21:56.359 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:21:56.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.444+0000 7f53ebf16d00 1 Processor -- start 2020-11-20T12:21:56.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.444+0000 7f53ebf16d00 1 -- start start 2020-11-20T12:21:56.449 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53ebf16d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53c80022c0 0x55e1a107bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53ebf16d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e1a1126750 0x55e1a1127500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53ebf16d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55e1a110d560 0x55e1a1109eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53ebf16d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a1002e30 con 0x55e1a110d560 2020-11-20T12:21:56.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53ebf16d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a0fd1810 con 0x7f53c80022c0 2020-11-20T12:21:56.450 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53ebf16d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e1a1126e90 con 0x55e1a1126750 2020-11-20T12:21:56.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53dc7bb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53c80022c0 0x55e1a107bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:56.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53dc7bb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53c80022c0 0x55e1a107bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54840/0 (socket says 172.21.15.187:54840) 2020-11-20T12:21:56.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53dc7bb700 1 -- 172.21.15.187:0/1755826420 learned_addr learned my addr 172.21.15.187:0/1755826420 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:56.451 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53dc7bb700 1 -- 172.21.15.187:0/1755826420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e1a1126750 msgr2=0x55e1a1127500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:56.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/1755826420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e1a1126750 0x55e1a1127500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:56.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53dc7bb700 1 -- 172.21.15.187:0/1755826420 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55e1a110d560 msgr2=0x55e1a1109eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:56.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/1755826420 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55e1a110d560 0x55e1a1109eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:56.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.445+0000 7f53dc7bb700 1 -- 172.21.15.187:0/1755826420 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1a0ff6ec0 con 0x7f53c80022c0 2020-11-20T12:21:56.452 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/1755826420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53c80022c0 0x55e1a107bed0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7f53cc00c010 tx=0x7f53cc015000).ready entity=mon.1 client_cookie=e994c100a3c6cd60 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:56.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53db7b9700 1 -- 172.21.15.187:0/1755826420 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc00e430 con 0x7f53c80022c0 2020-11-20T12:21:56.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53db7b9700 1 -- 172.21.15.187:0/1755826420 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53cc001780 con 0x7f53c80022c0 2020-11-20T12:21:56.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53db7b9700 1 -- 172.21.15.187:0/1755826420 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc00e9f0 con 0x7f53c80022c0 2020-11-20T12:21:56.456 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53ebf16d00 1 -- 172.21.15.187:0/1755826420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53c80022c0 msgr2=0x55e1a107bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/1755826420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53c80022c0 0x55e1a107bed0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7f53cc00c010 tx=0x7f53cc015000).stop 2020-11-20T12:21:56.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53ebf16d00 1 -- 172.21.15.187:0/1755826420 shutdown_connections 2020-11-20T12:21:56.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/1755826420 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55e1a110d560 0x55e1a1109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:56.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/1755826420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53c80022c0 0x55e1a107bed0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/1755826420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e1a1126750 0x55e1a1127500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:56.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53ebf16d00 1 -- 172.21.15.187:0/1755826420 >> 172.21.15.187:0/1755826420 conn(0x55e1a1056a90 msgr2=0x55e1a1096760 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:56.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.446+0000 7f53ebf16d00 1 -- 172.21.15.187:0/1755826420 shutdown_connections 2020-11-20T12:21:56.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.447+0000 7f53ebf16d00 1 -- 172.21.15.187:0/1755826420 wait complete. 2020-11-20T12:21:56.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.447+0000 7f53ebf16d00 1 Processor -- start 2020-11-20T12:21:56.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.447+0000 7f53ebf16d00 1 -- start start 2020-11-20T12:21:56.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.448+0000 7f53ebf16d00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55e1a110d560 0x55e1a1125da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.448+0000 7f53ebf16d00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e1a1126750 0x55e1a1078640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.448+0000 7f53ebf16d00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53c80022c0 0x55e1a107a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.448+0000 7f53ebf16d00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a112dcd0 con 0x55e1a110d560 2020-11-20T12:21:56.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.448+0000 7f53ebf16d00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a112fa30 con 0x55e1a1126750 2020-11-20T12:21:56.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.448+0000 7f53ebf16d00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e1a112fca0 con 0x7f53c80022c0 2020-11-20T12:21:56.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.448+0000 7f53d3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e1a1126750 0x55e1a1078640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:56.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e1a1126750 0x55e1a1078640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54848/0 (socket says 172.21.15.187:54848) 2020-11-20T12:21:56.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d3fff700 1 -- 172.21.15.187:0/3250032857 learned_addr learned my addr 172.21.15.187:0/3250032857 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:21:56.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d3fff700 1 -- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53c80022c0 msgr2=0x55e1a107a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d3fff700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53c80022c0 0x55e1a107a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:56.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d3fff700 1 -- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55e1a110d560 msgr2=0x55e1a1125da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d3fff700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55e1a110d560 0x55e1a1125da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:21:56.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d3fff700 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1a0fd5250 con 0x55e1a1126750 2020-11-20T12:21:56.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d3fff700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e1a1126750 0x55e1a1078640 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7f53c4007a50 tx=0x7f53c4005b10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:56.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d9fb6700 1 -- 172.21.15.187:0/3250032857 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c400f070 con 0x55e1a1126750 2020-11-20T12:21:56.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.449+0000 7f53d9fb6700 1 -- 172.21.15.187:0/3250032857 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c40068e0 con 0x55e1a1126750 2020-11-20T12:21:56.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.450+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e1a112ff40 con 0x55e1a1126750 2020-11-20T12:21:56.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.450+0000 7f53d9fb6700 1 -- 172.21.15.187:0/3250032857 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c4006c60 con 0x55e1a1126750 2020-11-20T12:21:56.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.450+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1a1123ac0 con 0x55e1a1126750 2020-11-20T12:21:56.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.451+0000 7f53d9fb6700 1 -- 172.21.15.187:0/3250032857 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f53c401b060 con 0x55e1a1126750 2020-11-20T12:21:56.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.452+0000 7f53d9fb6700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53bc0559a0 0x7f53bc057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.452+0000 7f53d9fb6700 1 -- 172.21.15.187:0/3250032857 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f53c406eb40 con 0x55e1a1126750 2020-11-20T12:21:56.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.452+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55e1a107d2e0 0x55e1a112e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.452+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53bc0559a0 0x7f53bc057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:56.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.452+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:1 32.0 32:00000000::::head [pgnls start_epoch 159 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e159) v8 -- 0x55e1a112eac0 con 0x55e1a107d2e0 2020-11-20T12:21:56.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.453+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53bc0559a0 0x7f53bc057e60 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f53cc020000 tx=0x7f53cc01c040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:56.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.453+0000 7f53dcfbc700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55e1a107d2e0 0x55e1a112e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:56.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.456+0000 7f53dcfbc700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55e1a107d2e0 0x55e1a112e5d0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:56.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.457+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== osd_op_reply(1 [pgnls start_epoch 159 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53d40043f0 con 0x55e1a107d2e0 2020-11-20T12:21:56.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.457+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:2 32.4 32:20000000::::head [pgnls start_epoch 159 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e159) v8 -- 0x55e1a112eac0 con 0x55e1a107d2e0 2020-11-20T12:21:56.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.459+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 <== osd.4 v2:172.21.15.5:6808/20156 2 ==== osd_op_reply(2 [pgnls start_epoch 159 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53d40043f0 con 0x55e1a107d2e0 2020-11-20T12:21:56.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.459+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:3 32.2 32:40000000::::head [pgnls start_epoch 159 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e159) v8 -- 0x55e1a112eac0 con 0x55e1a107d2e0 2020-11-20T12:21:56.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.460+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 <== osd.4 v2:172.21.15.5:6808/20156 3 ==== osd_op_reply(3 [pgnls start_epoch 159 out=81b] v159'8 uv7 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f53d40043f0 con 0x55e1a107d2e0 2020-11-20T12:21:56.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.460+0000 7f53dcfbc700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f53c80039e0 0x7f53d40107c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.460+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:4 32.6 32:60000000::::head [pgnls start_epoch 159 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e159) v8 -- 0x55e1a112eac0 con 0x7f53c80039e0 2020-11-20T12:21:56.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.461+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f53c80039e0 0x7f53d40107c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:56.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.461+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f53c80039e0 0x7f53d40107c0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:56.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.462+0000 7f53dc7bb700 1 -- 172.21.15.187:0/3250032857 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(4 [pgnls start_epoch 159 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53cc014020 con 0x7f53c80039e0 2020-11-20T12:21:56.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.462+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f53cc017420 0x7f53cc017860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.462+0000 7f53dc7bb700 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:5 32.1 32:80000000::::head [pgnls start_epoch 159 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e159) v8 -- 0x55e1a112eac0 con 0x7f53cc017420 2020-11-20T12:21:56.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.463+0000 7f53d3fff700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f53cc017420 0x7f53cc017860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:56.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.464+0000 7f53d3fff700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f53cc017420 0x7f53cc017860 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:56.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.465+0000 7f53d3fff700 1 -- 172.21.15.187:0/3250032857 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(5 [pgnls start_epoch 159 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53c4045ff0 con 0x7f53cc017420 2020-11-20T12:21:56.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.465+0000 7f53d3fff700 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:6 32.5 32:a0000000::::head [pgnls start_epoch 159 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e159) v8 -- 0x55e1a112eac0 con 0x7f53c80039e0 2020-11-20T12:21:56.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.466+0000 7f53dc7bb700 1 -- 172.21.15.187:0/3250032857 <== osd.1 v2:172.21.15.187:6818/19920 2 ==== osd_op_reply(6 [pgnls start_epoch 159 out=81b] v159'6 uv5 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f53cc014020 con 0x7f53c80039e0 2020-11-20T12:21:56.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.466+0000 7f53dc7bb700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f53cc019920 0x7f53cc0313f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:56.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.466+0000 7f53dc7bb700 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:7 32.3 32:c0000000::::head [pgnls start_epoch 159 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e159) v8 -- 0x55e1a112eac0 con 0x7f53cc019920 2020-11-20T12:21:56.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.467+0000 7f53dcfbc700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f53cc019920 0x7f53cc0313f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:56.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.469+0000 7f53dcfbc700 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f53cc019920 0x7f53cc0313f0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:56.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.470+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(7 [pgnls start_epoch 159 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53d40043f0 con 0x7f53cc019920 2020-11-20T12:21:56.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.470+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:8 32.7 32:e0000000::::head [pgnls start_epoch 159 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e159) v8 -- 0x55e1a112eac0 con 0x7f53cc017420 2020-11-20T12:21:56.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53d3fff700 1 -- 172.21.15.187:0/3250032857 <== osd.6 v2:172.21.15.5:6800/20157 2 ==== osd_op_reply(8 [pgnls start_epoch 159 out=49b] v159'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53c4045ff0 con 0x7f53cc017420 2020-11-20T12:21:56.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f53c80039e0 msgr2=0x7f53d40107c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f53c80039e0 0x7f53d40107c0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55e1a107d2e0 msgr2=0x55e1a112e5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55e1a107d2e0 0x55e1a112e5d0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f53cc017420 msgr2=0x7f53cc017860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f53cc017420 0x7f53cc017860 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f53cc019920 msgr2=0x7f53cc0313f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f53cc019920 0x7f53cc0313f0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 reap_dead start 2020-11-20T12:21:56.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.471+0000 7f53dcfbc700 1 -- 172.21.15.187:0/3250032857 reap_dead start 2020-11-20T12:21:56.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53bc0559a0 msgr2=0x7f53bc057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53bc0559a0 0x7f53bc057e60 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f53cc020000 tx=0x7f53cc01c040).stop 2020-11-20T12:21:56.478 INFO:teuthology.orchestra.run.smithi187.stdout:10000000002.00000000 2020-11-20T12:21:56.478 INFO:teuthology.orchestra.run.smithi187.stdout:10000000002.00000001 2020-11-20T12:21:56.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e1a1126750 msgr2=0x55e1a1078640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:56.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e1a1126750 0x55e1a1078640 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7f53c4007a50 tx=0x7f53c4005b10).stop 2020-11-20T12:21:56.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 shutdown_connections 2020-11-20T12:21:56.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e1a1126750 0x55e1a1078640 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 --2- 172.21.15.187:0/3250032857 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f53bc0559a0 0x7f53bc057e60 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:56.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 >> 172.21.15.187:0/3250032857 conn(0x55e1a1056a90 msgr2=0x55e1a1096760 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:56.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 shutdown_connections 2020-11-20T12:21:56.480 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:21:56.472+0000 7f53ebf16d00 1 -- 172.21.15.187:0/3250032857 wait complete. 2020-11-20T12:21:56.481 INFO:tasks.cephfs.filesystem:Objects not absent (ino 1099511627778, size 8388608): {'10000000002.00000001', '10000000002.00000000'} 2020-11-20T12:21:56.482 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60)... 2020-11-20T12:21:56.938 INFO:tasks.ceph.osd.4.smithi005.stderr:2020-11-20T12:21:56.933+0000 7f2be6e12700 -1 osd.4 pg_epoch: 160 pg[32.2( v 159'8 (0'0,159'8] local-lis/les=157/158 n=2 ec=157/157 lis/c=157/157 les/c/f=158/158/0 sis=157) [4,0] r=0 lpr=157 crt=159'7 lcod 159'7 mlcod 159'7 active+clean+snaptrim trimq=[2~2]] removing snap head 2020-11-20T12:21:57.114 INFO:tasks.ceph.osd.1.smithi187.stderr:2020-11-20T12:21:57.110+0000 7fa04e696700 -1 osd.1 pg_epoch: 160 pg[32.5( v 159'6 (0'0,159'6] local-lis/les=157/158 n=2 ec=157/157 lis/c=157/157 les/c/f=158/158/0 sis=157) [1,3] r=0 lpr=157 crt=159'5 lcod 159'5 mlcod 159'5 active+clean+snaptrim trimq=[2~2]] removing snap head 2020-11-20T12:22:01.483 INFO:teuthology.orchestra.run.smithi187:> rados -p cephfs_data ls 2020-11-20T12:22:01.576 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.573+0000 7f929ab6bd00 1 Processor -- start 2020-11-20T12:22:01.577 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.573+0000 7f929ab6bd00 1 -- start start 2020-11-20T12:22:01.578 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f929ab6bd00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c60481dce0 0x55c6048a7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f929ab6bd00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048acab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f929ab6bd00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c6048ab3b0 0x55c6047fece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f929ab6bd00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c604757810 con 0x55c6048ab3b0 2020-11-20T12:22:01.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f929ab6bd00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60481b480 con 0x55c60481e3d0 2020-11-20T12:22:01.579 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f929ab6bd00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6048a9c50 con 0x55c60481dce0 2020-11-20T12:22:01.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f928ac0f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f928ac0f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048acab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54870/0 (socket says 172.21.15.187:54870) 2020-11-20T12:22:01.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.574+0000 7f928ac0f700 1 -- 172.21.15.187:0/2017118807 learned_addr learned my addr 172.21.15.187:0/2017118807 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:01.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f928ac0f700 1 -- 172.21.15.187:0/2017118807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c60481dce0 msgr2=0x55c6048a7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:01.580 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f928ac0f700 1 --2- 172.21.15.187:0/2017118807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c60481dce0 0x55c6048a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:01.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f928ac0f700 1 -- 172.21.15.187:0/2017118807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c6048ab3b0 msgr2=0x55c6047fece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:01.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f928ac0f700 1 --2- 172.21.15.187:0/2017118807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c6048ab3b0 0x55c6047fece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:01.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f928ac0f700 1 -- 172.21.15.187:0/2017118807 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6047814e0 con 0x55c60481e3d0 2020-11-20T12:22:01.581 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f928ac0f700 1 --2- 172.21.15.187:0/2017118807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048acab0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f928000a860 tx=0x7f9280005a40).ready entity=mon.1 client_cookie=cfe0a3930aa6e736 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f9289c0d700 1 -- 172.21.15.187:0/2017118807 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280008960 con 0x55c60481e3d0 2020-11-20T12:22:01.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f9289c0d700 1 -- 172.21.15.187:0/2017118807 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9280000e90 con 0x55c60481e3d0 2020-11-20T12:22:01.582 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.575+0000 7f9289c0d700 1 -- 172.21.15.187:0/2017118807 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280008cf0 con 0x55c60481e3d0 2020-11-20T12:22:01.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 -- 172.21.15.187:0/2017118807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 msgr2=0x55c6048acab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/2017118807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048acab0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f928000a860 tx=0x7f9280005a40).stop 2020-11-20T12:22:01.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 -- 172.21.15.187:0/2017118807 shutdown_connections 2020-11-20T12:22:01.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/2017118807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c6048ab3b0 0x55c6047fece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:01.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/2017118807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c60481dce0 0x55c6048a7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:01.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/2017118807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048acab0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 -- 172.21.15.187:0/2017118807 >> 172.21.15.187:0/2017118807 conn(0x55c6047daee0 msgr2=0x55c6048aa950 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:01.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 -- 172.21.15.187:0/2017118807 shutdown_connections 2020-11-20T12:22:01.587 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.576+0000 7f929ab6bd00 1 -- 172.21.15.187:0/2017118807 wait complete. 2020-11-20T12:22:01.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.577+0000 7f929ab6bd00 1 Processor -- start 2020-11-20T12:22:01.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.577+0000 7f929ab6bd00 1 -- start start 2020-11-20T12:22:01.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f929ab6bd00 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c60481dce0 0x55c6048a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f929ab6bd00 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048be7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.588 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f929ab6bd00 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c6048ab3b0 0x55c6048bab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f929ab6bd00 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6048be100 con 0x55c60481dce0 2020-11-20T12:22:01.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f929ab6bd00 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60481a150 con 0x55c60481e3d0 2020-11-20T12:22:01.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f929ab6bd00 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6048be330 con 0x55c6048ab3b0 2020-11-20T12:22:01.589 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f928ac0f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048be7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f928ac0f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048be7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54876/0 (socket says 172.21.15.187:54876) 2020-11-20T12:22:01.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f928ac0f700 1 -- 172.21.15.187:0/1626155150 learned_addr learned my addr 172.21.15.187:0/1626155150 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:01.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f928bc11700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c6048ab3b0 0x55c6048bab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.578+0000 7f928b410700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c60481dce0 0x55c6048a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f928ac0f700 1 -- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c6048ab3b0 msgr2=0x55c6048bab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f928ac0f700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c6048ab3b0 0x55c6048bab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f928ac0f700 1 -- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c60481dce0 msgr2=0x55c6048a86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f928ac0f700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c60481dce0 0x55c6048a86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f928ac0f700 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c60477cec0 con 0x55c60481e3d0 2020-11-20T12:22:01.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f928bc11700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55c6048ab3b0 0x55c6048bab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:01.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f928b410700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x55c60481dce0 0x55c6048a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:01.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f928ac0f700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048be7d0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f928000a830 tx=0x7f9280005a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f9273fff700 1 -- 172.21.15.187:0/1626155150 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928001d070 con 0x55c60481e3d0 2020-11-20T12:22:01.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f9273fff700 1 -- 172.21.15.187:0/1626155150 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928000ce40 con 0x55c60481e3d0 2020-11-20T12:22:01.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.579+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c60475b250 con 0x55c60481e3d0 2020-11-20T12:22:01.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.580+0000 7f9273fff700 1 -- 172.21.15.187:0/1626155150 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280016870 con 0x55c60481e3d0 2020-11-20T12:22:01.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.580+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c604819dd0 con 0x55c60481e3d0 2020-11-20T12:22:01.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.581+0000 7f9273fff700 1 -- 172.21.15.187:0/1626155150 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9280006040 con 0x55c60481e3d0 2020-11-20T12:22:01.593 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.582+0000 7f9273fff700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92780559a0 0x7f9278057e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.582+0000 7f9273fff700 1 -- 172.21.15.187:0/1626155150 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f9280070210 con 0x55c60481e3d0 2020-11-20T12:22:01.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.582+0000 7f928b410700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92780559a0 0x7f9278057e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.582+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55c604878c10 0x55c6048a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.582+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:1 32.0 32:00000000::::head [pgnls start_epoch 160 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e160) v8 -- 0x55c6047dcc90 con 0x55c604878c10 2020-11-20T12:22:01.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.583+0000 7f928b410700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92780559a0 0x7f9278057e60 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f927c006010 tx=0x7f927c015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.583+0000 7f928bc11700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55c604878c10 0x55c6048a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.584+0000 7f928bc11700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55c604878c10 0x55c6048a6d60 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.585+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== osd_op_reply(1 [pgnls start_epoch 160 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92840017b0 con 0x55c604878c10 2020-11-20T12:22:01.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.585+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:2 32.4 32:20000000::::head [pgnls start_epoch 160 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e160) v8 -- 0x55c6047dcc90 con 0x55c604878c10 2020-11-20T12:22:01.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.587+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 <== osd.4 v2:172.21.15.5:6808/20156 2 ==== osd_op_reply(2 [pgnls start_epoch 160 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92840017b0 con 0x55c604878c10 2020-11-20T12:22:01.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.587+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- osd_op(unknown.0.0:3 32.2 32:40000000::::head [pgnls start_epoch 160 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e160) v8 -- 0x55c6047dcc90 con 0x55c604878c10 2020-11-20T12:22:01.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.588+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 <== osd.4 v2:172.21.15.5:6808/20156 3 ==== osd_op_reply(3 [pgnls start_epoch 160 out=49b] v160'10 uv7 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92840017b0 con 0x55c604878c10 2020-11-20T12:22:01.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.588+0000 7f928bc11700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f9284010310 0x7f9284010730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.588+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:4 32.6 32:60000000::::head [pgnls start_epoch 160 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e160) v8 -- 0x55c6047dcc90 con 0x7f9284010310 2020-11-20T12:22:01.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.588+0000 7f928b410700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f9284010310 0x7f9284010730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.589+0000 7f928b410700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f9284010310 0x7f9284010730 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.589+0000 7f928b410700 1 -- 172.21.15.187:0/1626155150 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== osd_op_reply(4 [pgnls start_epoch 160 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f927c00a5e0 con 0x7f9284010310 2020-11-20T12:22:01.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.590+0000 7f928b410700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f927c01d990 0x7f927c011390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.590+0000 7f928b410700 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:5 32.1 32:80000000::::head [pgnls start_epoch 160 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e160) v8 -- 0x55c6047dcc90 con 0x7f927c01d990 2020-11-20T12:22:01.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.590+0000 7f928ac0f700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f927c01d990 0x7f927c011390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.592+0000 7f928ac0f700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f927c01d990 0x7f927c011390 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.593+0000 7f928ac0f700 1 -- 172.21.15.187:0/1626155150 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== osd_op_reply(5 [pgnls start_epoch 160 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9280012070 con 0x7f927c01d990 2020-11-20T12:22:01.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.593+0000 7f928ac0f700 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- osd_op(unknown.0.0:6 32.5 32:a0000000::::head [pgnls start_epoch 160 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e160) v8 -- 0x55c6047dcc90 con 0x7f9284010310 2020-11-20T12:22:01.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.594+0000 7f928b410700 1 -- 172.21.15.187:0/1626155150 <== osd.1 v2:172.21.15.187:6818/19920 2 ==== osd_op_reply(6 [pgnls start_epoch 160 out=49b] v160'8 uv5 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f927c00a5e0 con 0x7f9284010310 2020-11-20T12:22:01.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.594+0000 7f928b410700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f927c01c3e0 0x7f927c01c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.594+0000 7f928b410700 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- osd_op(unknown.0.0:7 32.3 32:c0000000::::head [pgnls start_epoch 160 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e160) v8 -- 0x55c6047dcc90 con 0x7f927c01c3e0 2020-11-20T12:22:01.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.596+0000 7f928bc11700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f927c01c3e0 0x7f927c01c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.598+0000 7f928bc11700 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f927c01c3e0 0x7f927c01c820 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.599+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== osd_op_reply(7 [pgnls start_epoch 160 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92840017b0 con 0x7f927c01c3e0 2020-11-20T12:22:01.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.599+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- osd_op(unknown.0.0:8 32.7 32:e0000000::::head [pgnls start_epoch 160 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e160) v8 -- 0x55c6047dcc90 con 0x7f927c01d990 2020-11-20T12:22:01.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f928ac0f700 1 -- 172.21.15.187:0/1626155150 <== osd.6 v2:172.21.15.5:6800/20157 2 ==== osd_op_reply(8 [pgnls start_epoch 160 out=49b] v159'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9280012070 con 0x7f927c01d990 2020-11-20T12:22:01.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f9284010310 msgr2=0x7f9284010730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f9284010310 0x7f9284010730 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55c604878c10 msgr2=0x55c6048a6d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x55c604878c10 0x55c6048a6d60 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f927c01d990 msgr2=0x7f927c011390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f927c01d990 0x7f927c011390 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f927c01c3e0 msgr2=0x7f927c01c820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f927c01c3e0 0x7f927c01c820 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 reap_dead start 2020-11-20T12:22:01.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.600+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92780559a0 msgr2=0x7f9278057e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.601+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92780559a0 0x7f9278057e60 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f927c006010 tx=0x7f927c015000).stop 2020-11-20T12:22:01.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.601+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 msgr2=0x55c6048be7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.601+0000 7f929ab6bd00 1 --2- 172.21.15.187:0/1626155150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c60481e3d0 0x55c6048be7d0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f928000a830 tx=0x7f9280005a40).stop 2020-11-20T12:22:01.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.601+0000 7f928bc11700 1 -- 172.21.15.187:0/1626155150 reap_dead start 2020-11-20T12:22:01.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.601+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 shutdown_connections 2020-11-20T12:22:01.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.601+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 >> 172.21.15.187:0/1626155150 conn(0x55c6047daee0 msgr2=0x55c6048aa950 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:01.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.601+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 shutdown_connections 2020-11-20T12:22:01.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.601+0000 7f929ab6bd00 1 -- 172.21.15.187:0/1626155150 wait complete. 2020-11-20T12:22:01.609 INFO:tasks.cephfs.filesystem:All objects for ino 1099511627778 size 8388608 are absent 2020-11-20T12:22:01.609 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2020-11-20T12:22:01.610 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:22:01.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.842+0000 7ff33338f700 1 Processor -- start 2020-11-20T12:22:01.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.842+0000 7ff33338f700 1 -- start start 2020-11-20T12:22:01.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.843+0000 7ff33338f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c1482d0 0x7ff32c0b6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.843+0000 7ff33338f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c144b80 con 0x7ff32c0b57b0 2020-11-20T12:22:01.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.844+0000 7ff33338f700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c1489e0 con 0x7ff32c068230 2020-11-20T12:22:01.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.844+0000 7ff33338f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c148c80 con 0x7ff32c1482d0 2020-11-20T12:22:01.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.844+0000 7ff331b8c700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff32c068230 0x7ff32c0b5f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51498/0 (socket says 172.21.15.187:51498) 2020-11-20T12:22:01.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.844+0000 7ff331b8c700 1 -- 172.21.15.187:0/1837070989 learned_addr learned my addr 172.21.15.187:0/1837070989 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.844+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194018302 0 0) 0x7ff32c1489e0 con 0x7ff32c068230 2020-11-20T12:22:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.844+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff314003240 con 0x7ff32c068230 2020-11-20T12:22:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.844+0000 7ff332b8e700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c1482d0 0x7ff32c0b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1724420684 0 0) 0x7ff314003240 con 0x7ff32c068230 2020-11-20T12:22:01.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c1482d0 msgr2=0x7ff32c0b6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:01.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c1482d0 0x7ff32c0b6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 >> v1:172.21.15.5:6789/0 conn(0x7ff32c0b57b0 legacy=0x7ff32c062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:01.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c148f20 con 0x7ff32c068230 2020-11-20T12:22:01.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7ff31c0014e0 con 0x7ff32c068230 2020-11-20T12:22:01.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 >> v1:172.21.15.187:6789/0 conn(0x7ff32c068230 legacy=0x7ff32c0b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff314003730 0x7ff314003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c1482d0 0x7ff3140044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.853 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3140049d0 0x7ff314004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c148c80 con 0x7ff314003730 2020-11-20T12:22:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140041d0 con 0x7ff3140049d0 2020-11-20T12:22:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff314004310 con 0x7ff32c1482d0 2020-11-20T12:22:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.845+0000 7ff332b8e700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3140049d0 0x7ff314004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.854 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.846+0000 7ff33238d700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff314003730 0x7ff314003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.846+0000 7ff332b8e700 1 -- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c1482d0 msgr2=0x7ff3140044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.846+0000 7ff332b8e700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c1482d0 0x7ff3140044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.846+0000 7ff332b8e700 1 -- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff314003730 msgr2=0x7ff314003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.846+0000 7ff332b8e700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff314003730 0x7ff314003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.846+0000 7ff332b8e700 1 -- 172.21.15.187:0/1837070989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff314003240 con 0x7ff3140049d0 2020-11-20T12:22:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.846+0000 7ff332b8e700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3140049d0 0x7ff314004e50 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7ff32400ac90 tx=0x7ff324008040).ready entity=mon.1 client_cookie=8849d7fbaee0788a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.847+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32400dcc0 con 0x7ff3140049d0 2020-11-20T12:22:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.847+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32400de20 con 0x7ff3140049d0 2020-11-20T12:22:01.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.847+0000 7ff33138b700 1 -- 172.21.15.187:0/1837070989 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3240116d0 con 0x7ff3140049d0 2020-11-20T12:22:01.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.847+0000 7ff33338f700 1 -- 172.21.15.187:0/1837070989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3140049d0 msgr2=0x7ff314004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.847+0000 7ff33338f700 1 --2- 172.21.15.187:0/1837070989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3140049d0 0x7ff314004e50 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7ff32400ac90 tx=0x7ff324008040).stop 2020-11-20T12:22:01.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.847+0000 7ff332b8e700 1 -- 172.21.15.187:0/1837070989 reap_dead start 2020-11-20T12:22:01.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.847+0000 7ff33338f700 1 -- 172.21.15.187:0/1837070989 shutdown_connections 2020-11-20T12:22:01.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.847+0000 7ff33338f700 1 -- 172.21.15.187:0/1837070989 >> 172.21.15.187:0/1837070989 conn(0x7ff32c0bab30 msgr2=0x7ff32c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:01.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.848+0000 7ff33338f700 1 -- 172.21.15.187:0/1837070989 shutdown_connections 2020-11-20T12:22:01.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.848+0000 7ff33338f700 1 -- 172.21.15.187:0/1837070989 wait complete. 2020-11-20T12:22:01.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.849+0000 7ff33338f700 1 Processor -- start 2020-11-20T12:22:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.849+0000 7ff33338f700 1 -- start start 2020-11-20T12:22:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.850+0000 7ff33338f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff32c143780 0x7ff32c15b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.850+0000 7ff33338f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c14b780 0x7ff32c15be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.850+0000 7ff33338f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff32c15c380 0x7ff32c1583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.850+0000 7ff33338f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c14af10 con 0x7ff32c15c380 2020-11-20T12:22:01.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.850+0000 7ff33338f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c14b1b0 con 0x7ff32c143780 2020-11-20T12:22:01.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.850+0000 7ff33338f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c14b450 con 0x7ff32c14b780 2020-11-20T12:22:01.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff32c143780 0x7ff32c15b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff32c143780 0x7ff32c15b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54902/0 (socket says 172.21.15.187:54902) 2020-11-20T12:22:01.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 -- 172.21.15.187:0/3753936605 learned_addr learned my addr 172.21.15.187:0/3753936605 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff332b8e700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff32c15c380 0x7ff32c1583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff331b8c700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c14b780 0x7ff32c15be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 -- 172.21.15.187:0/3753936605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c14b780 msgr2=0x7ff32c15be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.863 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c14b780 0x7ff32c15be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 -- 172.21.15.187:0/3753936605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff32c15c380 msgr2=0x7ff32c1583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff32c15c380 0x7ff32c1583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 -- 172.21.15.187:0/3753936605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c09ad70 con 0x7ff32c143780 2020-11-20T12:22:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.851+0000 7ff33238d700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff32c143780 0x7ff32c15b9a0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7ff32800e680 tx=0x7ff328009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.864 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.852+0000 7ff31a7fc700 1 -- 172.21.15.187:0/3753936605 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff328010550 con 0x7ff32c143780 2020-11-20T12:22:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.852+0000 7ff31a7fc700 1 -- 172.21.15.187:0/3753936605 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff328010040 con 0x7ff32c143780 2020-11-20T12:22:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.852+0000 7ff31a7fc700 1 -- 172.21.15.187:0/3753936605 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3280108c0 con 0x7ff32c143780 2020-11-20T12:22:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.852+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c143b60 con 0x7ff32c143780 2020-11-20T12:22:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.852+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c158890 con 0x7ff32c143780 2020-11-20T12:22:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.853+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32c0b1130 con 0x7ff32c143780 2020-11-20T12:22:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.854+0000 7ff31a7fc700 1 -- 172.21.15.187:0/3753936605 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff32801b020 con 0x7ff32c143780 2020-11-20T12:22:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.854+0000 7ff31a7fc700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff304056d50 0x7ff304059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.854+0000 7ff31a7fc700 1 -- 172.21.15.187:0/3753936605 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6129+0+0 (secure 0 0 0) 0x7ff32806faa0 con 0x7ff32c143780 2020-11-20T12:22:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.855+0000 7ff331b8c700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff304056d50 0x7ff304059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:01.866 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.855+0000 7ff331b8c700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff304056d50 0x7ff304059210 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7ff31c001e90 tx=0x7ff31c001300).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:01.867 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:01.859+0000 7ff31a7fc700 1 -- 172.21.15.187:0/3753936605 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff328044eb0 con 0x7ff32c143780 2020-11-20T12:22:02.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.112+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff32c061a20 con 0x7ff32c143780 2020-11-20T12:22:02.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.113+0000 7ff31a7fc700 1 -- 172.21.15.187:0/3753936605 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2145 (secure 0 0 0) 0x7ff3280484d0 con 0x7ff32c143780 2020-11-20T12:22:02.117 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:02.117 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":172,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:33.149592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:22:02.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.116+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff304056d50 msgr2=0x7ff304059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:02.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.116+0000 7ff33338f700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff304056d50 0x7ff304059210 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7ff31c001e90 tx=0x7ff31c001300).stop 2020-11-20T12:22:02.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.116+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff32c143780 msgr2=0x7ff32c15b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:02.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.116+0000 7ff33338f700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff32c143780 0x7ff32c15b9a0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7ff32800e680 tx=0x7ff328009040).stop 2020-11-20T12:22:02.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.116+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 shutdown_connections 2020-11-20T12:22:02.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.116+0000 7ff33338f700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7ff304056d50 0x7ff304059210 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.117+0000 7ff33338f700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff32c15c380 0x7ff32c1583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.117+0000 7ff33338f700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff32c143780 0x7ff32c15b9a0 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.117+0000 7ff33338f700 1 --2- 172.21.15.187:0/3753936605 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff32c14b780 0x7ff32c15be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.117+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 >> 172.21.15.187:0/3753936605 conn(0x7ff32c0bab30 msgr2=0x7ff32c0afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:02.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.117+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 shutdown_connections 2020-11-20T12:22:02.123 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.117+0000 7ff33338f700 1 -- 172.21.15.187:0/3753936605 wait complete. 2020-11-20T12:22:02.123 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 172 2020-11-20T12:22:02.144 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2020-11-20T12:22:02.409 INFO:teuthology.orchestra.run.smithi187.stdout:{"mds_cache":{"strays_enqueued":2}} 2020-11-20T12:22:02.410 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 2 } } 2020-11-20T12:22:02.411 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:22:02.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.647+0000 7fddf8360700 1 Processor -- start 2020-11-20T12:22:02.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.647+0000 7fddf8360700 1 -- start start 2020-11-20T12:22:02.655 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.649+0000 7fddf8360700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fddf014c750 0x7fddf0062580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.649+0000 7fddf8360700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 0x7fddf00b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.649+0000 7fddf8360700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0061e80 con 0x7fddf00666e0 2020-11-20T12:22:02.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.649+0000 7fddf8360700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf00620e0 con 0x7fddf014c750 2020-11-20T12:22:02.656 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.649+0000 7fddf8360700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf00b1500 con 0x7fddf0062a70 2020-11-20T12:22:02.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.649+0000 7fddf58fb700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fddf00666e0 0x7fddf0067d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51518/0 (socket says 172.21.15.187:51518) 2020-11-20T12:22:02.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.649+0000 7fddf58fb700 1 -- 172.21.15.187:0/2914896487 learned_addr learned my addr 172.21.15.187:0/2914896487 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:02.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf68fd700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 0x7fddf00b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:02.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063676683 0 0) 0x7fddf0061e80 con 0x7fddf00666e0 2020-11-20T12:22:02.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd8003240 con 0x7fddf00666e0 2020-11-20T12:22:02.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf60fc700 1 --2- 172.21.15.187:0/2914896487 >> v2:172.21.15.5:3300/0 conn(0x7fddf014c750 0x7fddf0062580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:02.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3426787159 0 0) 0x7fddd8003240 con 0x7fddf00666e0 2020-11-20T12:22:02.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 msgr2=0x7fddf00b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:02.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 0x7fddf00b0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 >> v2:172.21.15.5:3300/0 conn(0x7fddf014c750 msgr2=0x7fddf0062580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:02.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 --2- 172.21.15.187:0/2914896487 >> v2:172.21.15.5:3300/0 conn(0x7fddf014c750 0x7fddf0062580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf00b17d0 con 0x7fddf00666e0 2020-11-20T12:22:02.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7fdde0001660 con 0x7fddf00666e0 2020-11-20T12:22:02.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 >> v1:172.21.15.187:6789/0 conn(0x7fddf00666e0 legacy=0x7fddf0067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:02.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddd80037b0 0x7fddd8003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd80044e0 0x7fddd8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 0x7fddd8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf00620e0 con 0x7fddd80037b0 2020-11-20T12:22:02.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf00b1500 con 0x7fddd80044e0 2020-11-20T12:22:02.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.650+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd8005340 con 0x7fddf0062a70 2020-11-20T12:22:02.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf58fb700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd80044e0 0x7fddd8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:02.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf68fd700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 0x7fddd8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:02.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf58fb700 1 -- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 msgr2=0x7fddd8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:02.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf58fb700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 0x7fddd8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf60fc700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddd80037b0 0x7fddd8003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:02.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf58fb700 1 -- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddd80037b0 msgr2=0x7fddd8003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:02.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf58fb700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddd80037b0 0x7fddd8003bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf58fb700 1 -- 172.21.15.187:0/2914896487 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8003240 con 0x7fddd80044e0 2020-11-20T12:22:02.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.651+0000 7fddf68fd700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddf0062a70 0x7fddd8004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:02.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.652+0000 7fddf58fb700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd80044e0 0x7fddd8004960 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7fddf0066ae0 tx=0x7fdde0007790).ready entity=mon.1 client_cookie=423f208af3e79215 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:02.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.652+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0007820 con 0x7fddd80044e0 2020-11-20T12:22:02.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.652+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde00070d0 con 0x7fddd80044e0 2020-11-20T12:22:02.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.652+0000 7fddf50fa700 1 -- 172.21.15.187:0/2914896487 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00054c0 con 0x7fddd80044e0 2020-11-20T12:22:02.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.653+0000 7fddf8360700 1 -- 172.21.15.187:0/2914896487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd80044e0 msgr2=0x7fddd8004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:02.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.653+0000 7fddf8360700 1 --2- 172.21.15.187:0/2914896487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd80044e0 0x7fddd8004960 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7fddf0066ae0 tx=0x7fdde0007790).stop 2020-11-20T12:22:02.669 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.653+0000 7fddf68fd700 1 -- 172.21.15.187:0/2914896487 reap_dead start 2020-11-20T12:22:02.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.653+0000 7fddf8360700 1 -- 172.21.15.187:0/2914896487 shutdown_connections 2020-11-20T12:22:02.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.653+0000 7fddf8360700 1 -- 172.21.15.187:0/2914896487 >> 172.21.15.187:0/2914896487 conn(0x7fddf00bacd0 msgr2=0x7fddf00bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:02.670 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.653+0000 7fddf8360700 1 -- 172.21.15.187:0/2914896487 shutdown_connections 2020-11-20T12:22:02.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.653+0000 7fddf8360700 1 -- 172.21.15.187:0/2914896487 wait complete. 2020-11-20T12:22:02.671 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.654+0000 7fddf8360700 1 Processor -- start 2020-11-20T12:22:02.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.655+0000 7fddf8360700 1 -- start start 2020-11-20T12:22:02.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.655+0000 7fddf8360700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddd80044e0 0x7fddf0150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.672 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.655+0000 7fddf8360700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddf014f300 0x7fddf0151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.673 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.655+0000 7fddf8360700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddf014aa30 0x7fddf014aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.674 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.655+0000 7fddf8360700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0151b70 con 0x7fddf014aa30 2020-11-20T12:22:02.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.655+0000 7fddf8360700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0150ad0 con 0x7fddf014f300 2020-11-20T12:22:02.675 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.655+0000 7fddf8360700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf014cdc0 con 0x7fddd80044e0 2020-11-20T12:22:02.676 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddf014f300 0x7fddf0151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:02.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddf014f300 0x7fddf0151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54922/0 (socket says 172.21.15.187:54922) 2020-11-20T12:22:02.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 -- 172.21.15.187:0/2995170612 learned_addr learned my addr 172.21.15.187:0/2995170612 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:02.677 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 -- 172.21.15.187:0/2995170612 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddd80044e0 msgr2=0x7fddf0150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:02.678 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddd80044e0 0x7fddf0150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:02.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 -- 172.21.15.187:0/2995170612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddf014aa30 msgr2=0x7fddf014aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:02.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddf014aa30 0x7fddf014aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:02.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 -- 172.21.15.187:0/2995170612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf0150d70 con 0x7fddf014f300 2020-11-20T12:22:02.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.656+0000 7fddf58fb700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddf014f300 0x7fddf0151180 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7fdde000cc80 tx=0x7fdde0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:02.681 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.657+0000 7fddde7fc700 1 -- 172.21.15.187:0/2995170612 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0005e60 con 0x7fddf014f300 2020-11-20T12:22:02.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.657+0000 7fddde7fc700 1 -- 172.21.15.187:0/2995170612 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde0013660 con 0x7fddf014f300 2020-11-20T12:22:02.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.657+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf0151670 con 0x7fddf014f300 2020-11-20T12:22:02.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.657+0000 7fddde7fc700 1 -- 172.21.15.187:0/2995170612 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0013a00 con 0x7fddf014f300 2020-11-20T12:22:02.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.657+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf00b3050 con 0x7fddf014f300 2020-11-20T12:22:02.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.658+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf0063600 con 0x7fddf014f300 2020-11-20T12:22:02.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.659+0000 7fddde7fc700 1 -- 172.21.15.187:0/2995170612 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdde001d070 con 0x7fddf014f300 2020-11-20T12:22:02.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.659+0000 7fddde7fc700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddc8056d20 0x7fddc80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:02.685 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.660+0000 7fddde7fc700 1 -- 172.21.15.187:0/2995170612 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fdde00714d0 con 0x7fddf014f300 2020-11-20T12:22:02.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.660+0000 7fddf60fc700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddc8056d20 0x7fddc80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:02.686 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.660+0000 7fddf60fc700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddc8056d20 0x7fddc80591e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fddec006fd0 tx=0x7fddec008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:02.687 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.663+0000 7fddde7fc700 1 -- 172.21.15.187:0/2995170612 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdde0046810 con 0x7fddf014f300 2020-11-20T12:22:02.918 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.912+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddf0068440 con 0x7fddf014f300 2020-11-20T12:22:02.919 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.913+0000 7fddde7fc700 1 -- 172.21.15.187:0/2995170612 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2145 (secure 0 0 0) 0x7fdde001a070 con 0x7fddf014f300 2020-11-20T12:22:02.920 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:02.924 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":172,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.5:6833/2701465500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":2701465500},{"type":"v1","addr":"172.21.15.5:6833","nonce":2701465500}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2020-11-20T12:21:22.646683+0000","modified":"2020-11-20T12:21:33.149592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10018},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10018":{"gid":10018,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/1915143367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1915143367},{"type":"v1","addr":"172.21.15.187:6835","nonce":1915143367}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2020-11-20T12:22:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.915+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddc8056d20 msgr2=0x7fddc80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:02.930 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.915+0000 7fddf8360700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddc8056d20 0x7fddc80591e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fddec006fd0 tx=0x7fddec008040).stop 2020-11-20T12:22:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.915+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddf014f300 msgr2=0x7fddf0151180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.915+0000 7fddf8360700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddf014f300 0x7fddf0151180 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7fdde000cc80 tx=0x7fdde0015000).stop 2020-11-20T12:22:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.916+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 shutdown_connections 2020-11-20T12:22:02.931 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.916+0000 7fddf8360700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fddc8056d20 0x7fddc80591e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.916+0000 7fddf8360700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fddf014aa30 0x7fddf014aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.916+0000 7fddf8360700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fddd80044e0 0x7fddf0150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.916+0000 7fddf8360700 1 --2- 172.21.15.187:0/2995170612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddf014f300 0x7fddf0151180 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.916+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 >> 172.21.15.187:0/2995170612 conn(0x7fddf00bacd0 msgr2=0x7fddf014b580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.916+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 shutdown_connections 2020-11-20T12:22:02.933 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:02.916+0000 7fddf8360700 1 -- 172.21.15.187:0/2995170612 wait complete. 2020-11-20T12:22:02.933 INFO:teuthology.orchestra.run.smithi187.stderr:dumped fsmap epoch 172 2020-11-20T12:22:02.941 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2020-11-20T12:22:03.212 INFO:teuthology.orchestra.run.smithi187.stdout:{"purge_queue":{"pq_executed":2}} 2020-11-20T12:22:03.214 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 2 } } 2020-11-20T12:22:03.216 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2020-11-20T12:22:03.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.458+0000 7fc3d3fff700 1 Processor -- start 2020-11-20T12:22:03.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.459+0000 7fc3d3fff700 1 -- start start 2020-11-20T12:22:03.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.460+0000 7fc3d3fff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc3d4065730 0x7fc3d40624e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:03.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.460+0000 7fc3d3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d414c140 con 0x7fc3d40b1790 2020-11-20T12:22:03.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.460+0000 7fc3d3fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d414c3b0 con 0x7fc3d4065350 2020-11-20T12:22:03.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.460+0000 7fc3d3fff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d414c650 con 0x7fc3d4065730 2020-11-20T12:22:03.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.460+0000 7fc3d2ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc3d4065350 0x7fc3d4061dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51536/0 (socket says 172.21.15.187:51536) 2020-11-20T12:22:03.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.460+0000 7fc3ca7fc700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc3d4065730 0x7fc3d40624e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:03.465 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.461+0000 7fc3d2ffd700 1 -- 172.21.15.187:0/2666427947 learned_addr learned my addr 172.21.15.187:0/2666427947 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:03.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.462+0000 7fc3d27fc700 1 -- 172.21.15.187:0/2666427947 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111178949 0 0) 0x7fc3d414c3b0 con 0x7fc3d4065350 2020-11-20T12:22:03.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.462+0000 7fc3ca7fc700 1 -- 172.21.15.187:0/2666427947 >> v1:172.21.15.187:6789/0 conn(0x7fc3d4065350 legacy=0x7fc3d4061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:03.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.462+0000 7fc3ca7fc700 1 -- 172.21.15.187:0/2666427947 >> v1:172.21.15.5:6789/0 conn(0x7fc3d40b1790 legacy=0x7fc3d4148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:03.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.462+0000 7fc3ca7fc700 1 -- 172.21.15.187:0/2666427947 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d414be00 con 0x7fc3d4065730 2020-11-20T12:22:03.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.462+0000 7fc3ca7fc700 1 --2- 172.21.15.187:0/2666427947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d4065730 0x7fc3d40624e0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7fc3bc00b9a0 tx=0x7fc3bc006df0).ready entity=mon.1 client_cookie=8936cd1534cb5f5c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:03.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.462+0000 7fc3d27fc700 1 -- 172.21.15.187:0/2666427947 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc0096b0 con 0x7fc3d4065730 2020-11-20T12:22:03.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.463+0000 7fc3d27fc700 1 -- 172.21.15.187:0/2666427947 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3bc009810 con 0x7fc3d4065730 2020-11-20T12:22:03.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.463+0000 7fc3d27fc700 1 -- 172.21.15.187:0/2666427947 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc009b90 con 0x7fc3d4065730 2020-11-20T12:22:03.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.463+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2666427947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d4065730 msgr2=0x7fc3d40624e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:03.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.463+0000 7fc3d3fff700 1 --2- 172.21.15.187:0/2666427947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d4065730 0x7fc3d40624e0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7fc3bc00b9a0 tx=0x7fc3bc006df0).stop 2020-11-20T12:22:03.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.463+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2666427947 shutdown_connections 2020-11-20T12:22:03.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.463+0000 7fc3d3fff700 1 --2- 172.21.15.187:0/2666427947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d4065730 0x7fc3d40624e0 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:03.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.463+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2666427947 >> 172.21.15.187:0/2666427947 conn(0x7fc3d40bab10 msgr2=0x7fc3d40baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:03.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.463+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2666427947 shutdown_connections 2020-11-20T12:22:03.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.464+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2666427947 wait complete. 2020-11-20T12:22:03.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.465+0000 7fc3d3fff700 1 Processor -- start 2020-11-20T12:22:03.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.465+0000 7fc3d3fff700 1 -- start start 2020-11-20T12:22:03.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3d4065350 0x7fc3d4155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:03.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3d4065730 0x7fc3d40b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:03.472 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d40b1790 0x7fc3d415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:03.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d40b1b70 con 0x7fc3d4065730 2020-11-20T12:22:03.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d4148f50 con 0x7fc3d40b1790 2020-11-20T12:22:03.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41491f0 con 0x7fc3d4065350 2020-11-20T12:22:03.473 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d40b1790 0x7fc3d415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:03.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d37fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d40b1790 0x7fc3d415b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54938/0 (socket says 172.21.15.187:54938) 2020-11-20T12:22:03.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.466+0000 7fc3d37fe700 1 -- 172.21.15.187:0/2904302649 learned_addr learned my addr 172.21.15.187:0/2904302649 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:03.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d2ffd700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3d4065350 0x7fc3d4155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:03.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d37fe700 1 -- 172.21.15.187:0/2904302649 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3d4065350 msgr2=0x7fc3d4155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:03.474 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3ca7fc700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3d4065730 0x7fc3d40b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:03.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d37fe700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3d4065350 0x7fc3d4155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:03.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d37fe700 1 -- 172.21.15.187:0/2904302649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3d4065730 msgr2=0x7fc3d40b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:03.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d37fe700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3d4065730 0x7fc3d40b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:03.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d37fe700 1 -- 172.21.15.187:0/2904302649 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d4149490 con 0x7fc3d40b1790 2020-11-20T12:22:03.475 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d37fe700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d40b1790 0x7fc3d415b3f0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7fc3cc00cae0 tx=0x7fc3cc007ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:03.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3cb7fe700 1 -- 172.21.15.187:0/2904302649 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc00a7f0 con 0x7fc3d40b1790 2020-11-20T12:22:03.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3cb7fe700 1 -- 172.21.15.187:0/2904302649 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3cc00a950 con 0x7fc3d40b1790 2020-11-20T12:22:03.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d4155820 con 0x7fc3d40b1790 2020-11-20T12:22:03.476 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3cb7fe700 1 -- 172.21.15.187:0/2904302649 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc00ad10 con 0x7fc3d40b1790 2020-11-20T12:22:03.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.467+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d40b3eb0 con 0x7fc3d40b1790 2020-11-20T12:22:03.477 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.469+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d4148d70 con 0x7fc3d40b1790 2020-11-20T12:22:03.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.469+0000 7fc3cb7fe700 1 -- 172.21.15.187:0/2904302649 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc3cc019030 con 0x7fc3d40b1790 2020-11-20T12:22:03.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.470+0000 7fc3cb7fe700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3b0056d20 0x7fc3b00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:03.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.470+0000 7fc3cb7fe700 1 -- 172.21.15.187:0/2904302649 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fc3cc06f880 con 0x7fc3d40b1790 2020-11-20T12:22:03.478 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.470+0000 7fc3d2ffd700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3b0056d20 0x7fc3b00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:03.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.471+0000 7fc3d2ffd700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3b0056d20 0x7fc3b00591e0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7fc3c4004920 tx=0x7fc3c4009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:03.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.474+0000 7fc3cb7fe700 1 -- 172.21.15.187:0/2904302649 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc3cc043fd0 con 0x7fc3d40b1790 2020-11-20T12:22:03.730 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.726+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fc3d4061a20 con 0x7fc3d40b1790 2020-11-20T12:22:03.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.727+0000 7fc3cb7fe700 1 -- 172.21.15.187:0/2904302649 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1942 (secure 0 0 0) 0x7fc3cc044d40 con 0x7fc3d40b1790 2020-11-20T12:22:03.731 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:03.731 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:03.732 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:22:03.732 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:22:03.732 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 772875018240, 2020-11-20T12:22:03.732 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 219095040, 2020-11-20T12:22:03.733 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 219095040, 2020-11-20T12:22:03.733 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 0.00028340020799078047, 2020-11-20T12:22:03.733 INFO:teuthology.orchestra.run.smithi187.stdout: "num_osds": 8, 2020-11-20T12:22:03.734 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_osds": 8, 2020-11-20T12:22:03.734 INFO:teuthology.orchestra.run.smithi187.stdout: "num_per_pool_omap_osds": 8 2020-11-20T12:22:03.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:22:03.735 INFO:teuthology.orchestra.run.smithi187.stdout: "stats_by_class": { 2020-11-20T12:22:03.735 INFO:teuthology.orchestra.run.smithi187.stdout: "ssd": { 2020-11-20T12:22:03.735 INFO:teuthology.orchestra.run.smithi187.stdout: "total_bytes": 773094113280, 2020-11-20T12:22:03.736 INFO:teuthology.orchestra.run.smithi187.stdout: "total_avail_bytes": 772875018240, 2020-11-20T12:22:03.736 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_bytes": 219095040, 2020-11-20T12:22:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_bytes": 219095040, 2020-11-20T12:22:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: "total_used_raw_ratio": 0.00028340020799078047 2020-11-20T12:22:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:22:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:22:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: "pools": [ 2020-11-20T12:22:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:22:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "device_health_metrics", 2020-11-20T12:22:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 1, 2020-11-20T12:22:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:22:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:22:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 2, 2020-11-20T12:22:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:22:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:22:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:22:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367086174208 2020-11-20T12:22:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:22:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:22:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:22:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "rbd", 2020-11-20T12:22:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 2, 2020-11-20T12:22:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:22:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:22:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:22:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:22:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:22:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:22:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367086174208 2020-11-20T12:22:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:22:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:22:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:22:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_metadata", 2020-11-20T12:22:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 31, 2020-11-20T12:22:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:22:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 60397, 2020-11-20T12:22:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 24, 2020-11-20T12:22:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 184, 2020-11-20T12:22:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 188416, 2020-11-20T12:22:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 2.566372074852552e-07, 2020-11-20T12:22:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367086174208 2020-11-20T12:22:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:22:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2020-11-20T12:22:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2020-11-20T12:22:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: "name": "cephfs_data", 2020-11-20T12:22:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: "id": 32, 2020-11-20T12:22:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: "stats": { 2020-11-20T12:22:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: "stored": 0, 2020-11-20T12:22:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: "objects": 0, 2020-11-20T12:22:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: "kb_used": 0, 2020-11-20T12:22:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: "bytes_used": 0, 2020-11-20T12:22:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: "percent_used": 0, 2020-11-20T12:22:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: "max_avail": 367086174208 2020-11-20T12:22:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:22:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: } 2020-11-20T12:22:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2020-11-20T12:22:03.750 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:03.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3b0056d20 msgr2=0x7fc3b00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:03.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3b0056d20 0x7fc3b00591e0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7fc3c4004920 tx=0x7fc3c4009040).stop 2020-11-20T12:22:03.751 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d40b1790 msgr2=0x7fc3d415b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:03.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d40b1790 0x7fc3d415b3f0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7fc3cc00cae0 tx=0x7fc3cc007ec0).stop 2020-11-20T12:22:03.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 shutdown_connections 2020-11-20T12:22:03.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fc3b0056d20 0x7fc3b00591e0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:03.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3d40b1790 0x7fc3d415b3f0 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:03.752 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3d4065350 0x7fc3d4155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:03.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 --2- 172.21.15.187:0/2904302649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3d4065730 0x7fc3d40b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:03.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.730+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 >> 172.21.15.187:0/2904302649 conn(0x7fc3d40bab10 msgr2=0x7fc3d414b060 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:03.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.731+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 shutdown_connections 2020-11-20T12:22:03.753 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:03.731+0000 7fc3d3fff700 1 -- 172.21.15.187:0/2904302649 wait complete. 2020-11-20T12:22:03.757 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2020-11-20T12:22:03.758 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-11-20T12:22:03.759 INFO:teuthology.orchestra.run.smithi005:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:22:03.877 INFO:teuthology.orchestra.run.smithi005.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:22:03.895 INFO:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2020-11-20T12:22:04.018 INFO:teuthology.orchestra.run.smithi187.stderr:# Warning: iptables-legacy tables present, use iptables-legacy-save to see them 2020-11-20T12:22:04.036 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2020-11-20T12:22:04.036 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2020-11-20T12:22:04.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:22:04.037 INFO:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.0 2020-11-20T12:22:04.182 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2020-11-20T12:22:04.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:04.183 INFO:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:22:04.238 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2020-11-20T12:22:04.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.479+0000 7f5fa759e700 1 Processor -- start 2020-11-20T12:22:04.483 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.479+0000 7f5fa759e700 1 -- start start 2020-11-20T12:22:04.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.481+0000 7f5fa759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5fa80b1380 0x7f5fa8150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.481+0000 7f5fa759e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5fa814c260 0x7f5fa814c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.481+0000 7f5fa759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa80658a0 con 0x7f5fa80b1760 2020-11-20T12:22:04.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.481+0000 7f5fa759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa8150390 con 0x7f5fa80b1380 2020-11-20T12:22:04.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.481+0000 7f5fa759e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa8150630 con 0x7f5fa814c260 2020-11-20T12:22:04.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5fa6d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5fa814c260 0x7f5fa814c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5fa6d9d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5fa814c260 0x7f5fa814c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54948/0 (socket says 172.21.15.187:54948) 2020-11-20T12:22:04.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5fa6d9d700 1 -- 172.21.15.187:0/4014164514 learned_addr learned my addr 172.21.15.187:0/4014164514 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:04.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5f9dd9b700 1 --1- 172.21.15.187:0/4014164514 >> v1:172.21.15.5:6789/0 conn(0x7f5fa80b1760 0x7f5fa8065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:45062/0 (socket says 172.21.15.187:45062) 2020-11-20T12:22:04.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5fa659c700 1 --2- 172.21.15.187:0/4014164514 >> v2:172.21.15.5:3300/0 conn(0x7f5fa80b1380 0x7f5fa8150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5fa6d9d700 1 -- 172.21.15.187:0/4014164514 >> v2:172.21.15.5:3300/0 conn(0x7f5fa80b1380 msgr2=0x7f5fa8150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:04.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5fa6d9d700 1 --2- 172.21.15.187:0/4014164514 >> v2:172.21.15.5:3300/0 conn(0x7f5fa80b1380 0x7f5fa8150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5fa6d9d700 1 -- 172.21.15.187:0/4014164514 >> v1:172.21.15.5:6789/0 conn(0x7f5fa80b1760 legacy=0x7f5fa8065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.489 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.482+0000 7f5fa6d9d700 1 -- 172.21.15.187:0/4014164514 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8068510 con 0x7f5fa814c260 2020-11-20T12:22:04.491 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.483+0000 7f5fa6d9d700 1 --2- 172.21.15.187:0/4014164514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa814c6c0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f5fa8061740 tx=0x7f5fa0009290).ready entity=mon.1 client_cookie=5b185e17cd35f2d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:04.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.483+0000 7f5fa5d9b700 1 -- 172.21.15.187:0/4014164514 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000bbc0 con 0x7f5fa814c260 2020-11-20T12:22:04.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.483+0000 7f5fa5d9b700 1 -- 172.21.15.187:0/4014164514 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa000bd20 con 0x7f5fa814c260 2020-11-20T12:22:04.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.483+0000 7f5fa5d9b700 1 -- 172.21.15.187:0/4014164514 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0011610 con 0x7f5fa814c260 2020-11-20T12:22:04.492 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.484+0000 7f5fa759e700 1 -- 172.21.15.187:0/4014164514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 msgr2=0x7f5fa814c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.484+0000 7f5fa759e700 1 --2- 172.21.15.187:0/4014164514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa814c6c0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f5fa8061740 tx=0x7f5fa0009290).stop 2020-11-20T12:22:04.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.484+0000 7f5fa759e700 1 -- 172.21.15.187:0/4014164514 shutdown_connections 2020-11-20T12:22:04.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.484+0000 7f5fa759e700 1 --2- 172.21.15.187:0/4014164514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa814c6c0 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.493 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.484+0000 7f5fa759e700 1 --2- 172.21.15.187:0/4014164514 >> v2:172.21.15.5:3300/0 conn(0x7f5fa80b1380 0x7f5fa8150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.484+0000 7f5fa759e700 1 -- 172.21.15.187:0/4014164514 >> 172.21.15.187:0/4014164514 conn(0x7f5fa80bab30 msgr2=0x7f5fa80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:04.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.484+0000 7f5fa759e700 1 -- 172.21.15.187:0/4014164514 shutdown_connections 2020-11-20T12:22:04.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.484+0000 7f5fa759e700 1 -- 172.21.15.187:0/4014164514 wait complete. 2020-11-20T12:22:04.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.486+0000 7f5fa759e700 1 Processor -- start 2020-11-20T12:22:04.494 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.486+0000 7f5fa759e700 1 -- start start 2020-11-20T12:22:04.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fa80b1380 0x7f5fa80b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fa80b1760 0x7f5fa809e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa759e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa80b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa809dae0 con 0x7f5fa80b1760 2020-11-20T12:22:04.495 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa759e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa809dd80 con 0x7f5fa814c260 2020-11-20T12:22:04.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa809e020 con 0x7f5fa80b1380 2020-11-20T12:22:04.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa80b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa6d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa80b92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54954/0 (socket says 172.21.15.187:54954) 2020-11-20T12:22:04.496 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa6d9d700 1 -- 172.21.15.187:0/1311595256 learned_addr learned my addr 172.21.15.187:0/1311595256 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:04.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa6d9d700 1 -- 172.21.15.187:0/1311595256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fa80b1380 msgr2=0x7f5fa80b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.498 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa6d9d700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fa80b1380 0x7f5fa80b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:04.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5f9dd9b700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fa80b1760 0x7f5fa809e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.487+0000 7f5fa6d9d700 1 -- 172.21.15.187:0/1311595256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fa80b1760 msgr2=0x7f5fa809e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.488+0000 7f5fa6d9d700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fa80b1760 0x7f5fa809e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.499 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.488+0000 7f5fa6d9d700 1 -- 172.21.15.187:0/1311595256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa80b8330 con 0x7f5fa814c260 2020-11-20T12:22:04.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.488+0000 7f5fa6d9d700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa80b92b0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 rx=0x7f5fa00092d0 tx=0x7f5fa0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:04.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.488+0000 7f5f9effd700 1 -- 172.21.15.187:0/1311595256 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000bbc0 con 0x7f5fa814c260 2020-11-20T12:22:04.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.488+0000 7f5f9effd700 1 -- 172.21.15.187:0/1311595256 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa000be90 con 0x7f5fa814c260 2020-11-20T12:22:04.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.488+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa80b48a0 con 0x7f5fa814c260 2020-11-20T12:22:04.500 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.488+0000 7f5f9effd700 1 -- 172.21.15.187:0/1311595256 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0010e50 con 0x7f5fa814c260 2020-11-20T12:22:04.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.488+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa809ebf0 con 0x7f5fa814c260 2020-11-20T12:22:04.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.489+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa809e1e0 con 0x7f5fa814c260 2020-11-20T12:22:04.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.490+0000 7f5f9effd700 1 -- 172.21.15.187:0/1311595256 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5fa0017070 con 0x7f5fa814c260 2020-11-20T12:22:04.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.491+0000 7f5f9effd700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f84056d20 0x7f5f840591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.501 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.491+0000 7f5f9effd700 1 -- 172.21.15.187:0/1311595256 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f5fa0072240 con 0x7f5fa814c260 2020-11-20T12:22:04.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.491+0000 7f5fa659c700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f84056d20 0x7f5f840591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.491+0000 7f5fa659c700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f84056d20 0x7f5f840591e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f5f98006fd0 tx=0x7f5f98006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:04.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.495+0000 7f5f9effd700 1 -- 172.21.15.187:0/1311595256 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5fa00475d0 con 0x7f5fa814c260 2020-11-20T12:22:04.689 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.686+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5fa8061a20 con 0x7f5fa814c260 2020-11-20T12:22:04.700 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.697+0000 7f5f9effd700 1 -- 172.21.15.187:0/1311595256 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5fa00150a0 con 0x7f5fa814c260 2020-11-20T12:22:04.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f84056d20 msgr2=0x7f5f840591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f84056d20 0x7f5f840591e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f5f98006fd0 tx=0x7f5f98006df0).stop 2020-11-20T12:22:04.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 msgr2=0x7f5fa80b92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.703 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa80b92b0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 rx=0x7f5fa00092d0 tx=0x7f5fa0016000).stop 2020-11-20T12:22:04.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 shutdown_connections 2020-11-20T12:22:04.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5f84056d20 0x7f5f840591e0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fa814c260 0x7f5fa80b92b0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.704 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5fa80b1380 0x7f5fa80b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:04.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 --2- 172.21.15.187:0/1311595256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5fa80b1760 0x7f5fa809e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.705 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.699+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 >> 172.21.15.187:0/1311595256 conn(0x7f5fa80bab30 msgr2=0x7f5fa80b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:04.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.700+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 shutdown_connections 2020-11-20T12:22:04.706 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.700+0000 7f5fa759e700 1 -- 172.21.15.187:0/1311595256 wait complete. 2020-11-20T12:22:04.728 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove' 2020-11-20T12:22:04.969 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.966+0000 7fad226dd700 1 Processor -- start 2020-11-20T12:22:04.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.966+0000 7fad226dd700 1 -- start start 2020-11-20T12:22:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.967+0000 7fad226dd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fad1c1484d0 0x7fad1c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.967+0000 7fad226dd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.967+0000 7fad226dd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0b1300 con 0x7fad1c0622c0 2020-11-20T12:22:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.967+0000 7fad226dd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1c067730 con 0x7fad1c1484d0 2020-11-20T12:22:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.967+0000 7fad226dd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0b15a0 con 0x7fad1c066f70 2020-11-20T12:22:04.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.968+0000 7fad21edc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.969+0000 7fad20eda700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fad1c0622c0 0x7fad1c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:45076/0 (socket says 172.21.15.187:45076) 2020-11-20T12:22:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.969+0000 7fad20eda700 1 -- 172.21.15.187:0/1981421785 learned_addr learned my addr 172.21.15.187:0/1981421785 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:04.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.969+0000 7fad216db700 1 --2- 172.21.15.187:0/1981421785 >> v2:172.21.15.5:3300/0 conn(0x7fad1c1484d0 0x7fad1c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.970+0000 7fad21edc700 1 -- 172.21.15.187:0/1981421785 >> v2:172.21.15.5:3300/0 conn(0x7fad1c1484d0 msgr2=0x7fad1c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.970+0000 7fad21edc700 1 --2- 172.21.15.187:0/1981421785 >> v2:172.21.15.5:3300/0 conn(0x7fad1c1484d0 0x7fad1c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.970+0000 7fad21edc700 1 -- 172.21.15.187:0/1981421785 >> v1:172.21.15.5:6789/0 conn(0x7fad1c0622c0 legacy=0x7fad1c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.970+0000 7fad21edc700 1 -- 172.21.15.187:0/1981421785 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0b1840 con 0x7fad1c066f70 2020-11-20T12:22:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.971+0000 7fad21edc700 1 --2- 172.21.15.187:0/1981421785 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c0b5d50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fad14009c90 tx=0x7fad140091d0).ready entity=mon.2 client_cookie=7c1f6127efce275 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.971+0000 7fad0bfff700 1 -- 172.21.15.187:0/1981421785 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1400ba70 con 0x7fad1c066f70 2020-11-20T12:22:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad0bfff700 1 -- 172.21.15.187:0/1981421785 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1400bbd0 con 0x7fad1c066f70 2020-11-20T12:22:04.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad0bfff700 1 -- 172.21.15.187:0/1981421785 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14011400 con 0x7fad1c066f70 2020-11-20T12:22:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad226dd700 1 -- 172.21.15.187:0/1981421785 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 msgr2=0x7fad1c0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad226dd700 1 --2- 172.21.15.187:0/1981421785 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c0b5d50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fad14009c90 tx=0x7fad140091d0).stop 2020-11-20T12:22:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad226dd700 1 -- 172.21.15.187:0/1981421785 shutdown_connections 2020-11-20T12:22:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad226dd700 1 --2- 172.21.15.187:0/1981421785 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c0b5d50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad226dd700 1 --2- 172.21.15.187:0/1981421785 >> v2:172.21.15.5:3300/0 conn(0x7fad1c1484d0 0x7fad1c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad226dd700 1 -- 172.21.15.187:0/1981421785 >> 172.21.15.187:0/1981421785 conn(0x7fad1c0bab30 msgr2=0x7fad1c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:04.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad226dd700 1 -- 172.21.15.187:0/1981421785 shutdown_connections 2020-11-20T12:22:04.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.972+0000 7fad226dd700 1 -- 172.21.15.187:0/1981421785 wait complete. 2020-11-20T12:22:04.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.974+0000 7fad226dd700 1 Processor -- start 2020-11-20T12:22:04.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.974+0000 7fad226dd700 1 -- start start 2020-11-20T12:22:04.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.975+0000 7fad226dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad1c0622c0 0x7fad1c0b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.975+0000 7fad226dd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c09dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad226dd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad1c1484d0 0x7fad1c09a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.980 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad226dd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c09e380 con 0x7fad1c1484d0 2020-11-20T12:22:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad226dd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c156330 con 0x7fad1c0622c0 2020-11-20T12:22:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad226dd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0b8050 con 0x7fad1c066f70 2020-11-20T12:22:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad216db700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad1c0622c0 0x7fad1c0b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad216db700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad1c0622c0 0x7fad1c0b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54964/0 (socket says 172.21.15.187:54964) 2020-11-20T12:22:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad216db700 1 -- 172.21.15.187:0/3180758821 learned_addr learned my addr 172.21.15.187:0/3180758821 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:04.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad216db700 1 -- 172.21.15.187:0/3180758821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 msgr2=0x7fad1c09dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.983 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad21edc700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad1c1484d0 0x7fad1c09a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.976+0000 7fad20eda700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c09dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad216db700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c09dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad216db700 1 -- 172.21.15.187:0/3180758821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad1c1484d0 msgr2=0x7fad1c09a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:04.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad216db700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad1c1484d0 0x7fad1c09a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad216db700 1 -- 172.21.15.187:0/3180758821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c09dfd0 con 0x7fad1c0622c0 2020-11-20T12:22:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad216db700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad1c0622c0 0x7fad1c0b7b60 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7fad1800eba0 tx=0x7fad18006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad097fa700 1 -- 172.21.15.187:0/3180758821 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1800ca60 con 0x7fad1c0622c0 2020-11-20T12:22:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad097fa700 1 -- 172.21.15.187:0/3180758821 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1800cbc0 con 0x7fad1c0622c0 2020-11-20T12:22:04.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad097fa700 1 -- 172.21.15.187:0/3180758821 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad180123f0 con 0x7fad1c0622c0 2020-11-20T12:22:04.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c09d320 con 0x7fad1c0622c0 2020-11-20T12:22:04.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.977+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c09b050 con 0x7fad1c0622c0 2020-11-20T12:22:04.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.979+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1c061a20 con 0x7fad1c0622c0 2020-11-20T12:22:04.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.979+0000 7fad097fa700 1 -- 172.21.15.187:0/3180758821 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fad180127a0 con 0x7fad1c0622c0 2020-11-20T12:22:04.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.980+0000 7fad097fa700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d50 0x7facf4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:04.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.980+0000 7fad097fa700 1 -- 172.21.15.187:0/3180758821 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fad180703f0 con 0x7fad1c0622c0 2020-11-20T12:22:04.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.980+0000 7fad20eda700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d50 0x7facf4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:04.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.981+0000 7fad20eda700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d50 0x7facf4059210 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7fad0c007fd0 tx=0x7fad0c009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:04.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:04.985+0000 7fad097fa700 1 -- 172.21.15.187:0/3180758821 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fad18045820 con 0x7fad1c0622c0 2020-11-20T12:22:05.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.240+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove"]} v 0) v1 -- 0x7fad1c1488b0 con 0x7fad1c0622c0 2020-11-20T12:22:05.938 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.935+0000 7fad097fa700 1 -- 172.21.15.187:0/3180758821 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove"]}]=0 v859) v1 ==== 135+0+0 (secure 0 0 0) 0x7fad18048e40 con 0x7fad1c0622c0 2020-11-20T12:22:05.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d50 msgr2=0x7facf4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:05.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d50 0x7facf4059210 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7fad0c007fd0 tx=0x7fad0c009040).stop 2020-11-20T12:22:05.942 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad1c0622c0 msgr2=0x7fad1c0b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad1c0622c0 0x7fad1c0b7b60 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7fad1800eba0 tx=0x7fad18006df0).stop 2020-11-20T12:22:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 shutdown_connections 2020-11-20T12:22:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7facf4056d50 0x7facf4059210 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad1c1484d0 0x7fad1c09a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:05.943 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad1c0622c0 0x7fad1c0b7b60 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:05.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 --2- 172.21.15.187:0/3180758821 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad1c066f70 0x7fad1c09dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:05.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.938+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 >> 172.21.15.187:0/3180758821 conn(0x7fad1c0bab30 msgr2=0x7fad1c143c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:05.944 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.939+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 shutdown_connections 2020-11-20T12:22:05.945 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:05.939+0000 7fad226dd700 1 -- 172.21.15.187:0/3180758821 wait complete. 2020-11-20T12:22:05.971 INFO:tasks.cephfs_test_runner:test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) ... ok 2020-11-20T12:22:05.972 INFO:tasks.cephfs_test_runner: 2020-11-20T12:22:05.972 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2020-11-20T12:22:05.972 INFO:tasks.cephfs_test_runner:Ran 14 tests in 974.329s 2020-11-20T12:22:05.973 INFO:tasks.cephfs_test_runner: 2020-11-20T12:22:05.973 INFO:tasks.cephfs_test_runner:OK 2020-11-20T12:22:05.973 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2020-11-20T12:22:05.989 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2020-11-20T12:22:06.052 INFO:tasks.kclient:Unmounting kernel clients... 2020-11-20T12:22:06.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.056 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:22:06.116 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 1) 2020-11-20T12:22:06.117 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T12:22:06.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.119 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2020-11-20T12:22:06.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.227 INFO:teuthology.orchestra.run.smithi187:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:22:06.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.357 INFO:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2020-11-20T12:22:06.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.541 INFO:teuthology.orchestra.run.smithi187:> ip netns list 2020-11-20T12:22:06.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.586 INFO:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2020-11-20T12:22:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2020-11-20T12:22:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:22:06.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.733 INFO:teuthology.orchestra.run.smithi005:> ip netns list 2020-11-20T12:22:06.795 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2020-11-20T12:22:06.796 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 0) 2020-11-20T12:22:06.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.797 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2020-11-20T12:22:06.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:06.913 INFO:teuthology.orchestra.run.smithi005:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2020-11-20T12:22:07.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:07.021 INFO:teuthology.orchestra.run.smithi005:> sudo ip link delete ceph-brx 2020-11-20T12:22:07.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:07.221 INFO:teuthology.orchestra.run.smithi005:> ip netns list 2020-11-20T12:22:07.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:22:07.266 INFO:teuthology.orchestra.run.smithi005:> sudo ip link delete ceph-brx 2020-11-20T12:22:07.354 INFO:teuthology.orchestra.run.smithi005.stderr:Cannot find device "ceph-brx" 2020-11-20T12:22:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:22:07.355 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2020-11-20T12:22:07.365 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2020-11-20T12:22:07.366 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:22:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.618+0000 7efc617d7700 1 Processor -- start 2020-11-20T12:22:07.622 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.619+0000 7efc617d7700 1 -- start start 2020-11-20T12:22:07.623 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.620+0000 7efc617d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c0bf800 0x7efc5c14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:07.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.620+0000 7efc617d7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5c14d110 con 0x7efc5c09f560 2020-11-20T12:22:07.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.620+0000 7efc617d7700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5c14cbd0 con 0x7efc5c0bfbe0 2020-11-20T12:22:07.624 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.620+0000 7efc617d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c14ce70 con 0x7efc5c0bf800 2020-11-20T12:22:07.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.621+0000 7efc537fe700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7efc5c0bfbe0 0x7efc5c1480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51580/0 (socket says 172.21.15.187:51580) 2020-11-20T12:22:07.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.621+0000 7efc537fe700 1 -- 172.21.15.187:0/779701063 learned_addr learned my addr 172.21.15.187:0/779701063 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:07.628 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.621+0000 7efc5bfff700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c0bf800 0x7efc5c14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.621+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318667735 0 0) 0x7efc5c14cbd0 con 0x7efc5c0bfbe0 2020-11-20T12:22:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.621+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3c003240 con 0x7efc5c0bfbe0 2020-11-20T12:22:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299675185 0 0) 0x7efc5c14d110 con 0x7efc5c09f560 2020-11-20T12:22:07.629 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc5c14cbd0 con 0x7efc5c09f560 2020-11-20T12:22:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1231432720 0 0) 0x7efc3c003240 con 0x7efc5c0bfbe0 2020-11-20T12:22:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c0bf800 msgr2=0x7efc5c14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c0bf800 0x7efc5c14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:07.630 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 >> v1:172.21.15.5:6789/0 conn(0x7efc5c09f560 legacy=0x7efc5c0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c0a4cc0 con 0x7efc5c0bfbe0 2020-11-20T12:22:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7efc440014e0 con 0x7efc5c0bfbe0 2020-11-20T12:22:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 >> v1:172.21.15.187:6789/0 conn(0x7efc5c0bfbe0 legacy=0x7efc5c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c0bf800 0x7efc3c0038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:07.631 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c003e00 0x7efc3c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc3c0049d0 0x7efc3c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c14ce70 con 0x7efc3c0049d0 2020-11-20T12:22:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0053a0 con 0x7efc3c003e00 2020-11-20T12:22:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c00b7b0 con 0x7efc5c0bf800 2020-11-20T12:22:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.622+0000 7efc537fe700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c003e00 0x7efc3c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc5bfff700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c0bf800 0x7efc3c0038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc537fe700 1 -- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c0bf800 msgr2=0x7efc3c0038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc537fe700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c0bf800 0x7efc3c0038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:07.633 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc537fe700 1 -- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc3c0049d0 msgr2=0x7efc3c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:07.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc537fe700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc3c0049d0 0x7efc3c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:07.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc537fe700 1 -- 172.21.15.187:0/779701063 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c003240 con 0x7efc3c003e00 2020-11-20T12:22:07.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc537fe700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c003e00 0x7efc3c0044e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 rx=0x7efc5c0bcaa0 tx=0x7efc440076d0).ready entity=mon.1 client_cookie=a46321d13ef86f39 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:07.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc440070d0 con 0x7efc3c003e00 2020-11-20T12:22:07.634 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc44007230 con 0x7efc3c003e00 2020-11-20T12:22:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.623+0000 7efc5b7fe700 1 -- 172.21.15.187:0/779701063 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc440055f0 con 0x7efc3c003e00 2020-11-20T12:22:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.624+0000 7efc617d7700 1 -- 172.21.15.187:0/779701063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c003e00 msgr2=0x7efc3c0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.624+0000 7efc617d7700 1 --2- 172.21.15.187:0/779701063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc3c003e00 0x7efc3c0044e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 rx=0x7efc5c0bcaa0 tx=0x7efc440076d0).stop 2020-11-20T12:22:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.624+0000 7efc60fd6700 1 -- 172.21.15.187:0/779701063 reap_dead start 2020-11-20T12:22:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.624+0000 7efc617d7700 1 -- 172.21.15.187:0/779701063 shutdown_connections 2020-11-20T12:22:07.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.624+0000 7efc617d7700 1 -- 172.21.15.187:0/779701063 >> 172.21.15.187:0/779701063 conn(0x7efc5c068370 msgr2=0x7efc5c0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:07.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.624+0000 7efc617d7700 1 -- 172.21.15.187:0/779701063 shutdown_connections 2020-11-20T12:22:07.636 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.624+0000 7efc617d7700 1 -- 172.21.15.187:0/779701063 wait complete. 2020-11-20T12:22:07.639 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.626+0000 7efc617d7700 1 Processor -- start 2020-11-20T12:22:07.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.627+0000 7efc617d7700 1 -- start start 2020-11-20T12:22:07.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc617d7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5c1431c0 0x7efc5c15c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:07.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc617d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c1435a0 0x7efc5c146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:07.640 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc617d7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc5c146b70 0x7efc5c0b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:07.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc617d7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c0b8220 con 0x7efc5c146b70 2020-11-20T12:22:07.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc617d7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c0baa50 con 0x7efc5c1431c0 2020-11-20T12:22:07.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc617d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c0b8410 con 0x7efc5c1435a0 2020-11-20T12:22:07.641 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc5bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5c1431c0 0x7efc5c15c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:07.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc5bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5c1431c0 0x7efc5c15c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54984/0 (socket says 172.21.15.187:54984) 2020-11-20T12:22:07.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.628+0000 7efc5bfff700 1 -- 172.21.15.187:0/838147768 learned_addr learned my addr 172.21.15.187:0/838147768 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:07.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc537fe700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c1435a0 0x7efc5c146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:07.642 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc60fd6700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc5c146b70 0x7efc5c0b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:07.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc5bfff700 1 -- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c1435a0 msgr2=0x7efc5c146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:07.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc5bfff700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c1435a0 0x7efc5c146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:07.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc5bfff700 1 -- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc5c146b70 msgr2=0x7efc5c0b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:07.643 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc5bfff700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc5c146b70 0x7efc5c0b8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:07.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc5bfff700 1 -- 172.21.15.187:0/838147768 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c0b86b0 con 0x7efc5c1431c0 2020-11-20T12:22:07.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc537fe700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c1435a0 0x7efc5c146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:07.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc60fd6700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc5c146b70 0x7efc5c0b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:07.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc5bfff700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5c1431c0 0x7efc5c15c900 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7efc4c006d00 tx=0x7efc4c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:07.644 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc58ff9700 1 -- 172.21.15.187:0/838147768 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c00d6d0 con 0x7efc5c1431c0 2020-11-20T12:22:07.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc58ff9700 1 -- 172.21.15.187:0/838147768 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c00d830 con 0x7efc5c1431c0 2020-11-20T12:22:07.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.629+0000 7efc58ff9700 1 -- 172.21.15.187:0/838147768 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c00dbb0 con 0x7efc5c1431c0 2020-11-20T12:22:07.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.630+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5c0b65b0 con 0x7efc5c1431c0 2020-11-20T12:22:07.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.630+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5c0b9040 con 0x7efc5c1431c0 2020-11-20T12:22:07.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.631+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5c080050 con 0x7efc5c1431c0 2020-11-20T12:22:07.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.632+0000 7efc58ff9700 1 -- 172.21.15.187:0/838147768 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efc4c013400 con 0x7efc5c1431c0 2020-11-20T12:22:07.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.632+0000 7efc58ff9700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc38056d50 0x7efc38059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:07.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.632+0000 7efc58ff9700 1 -- 172.21.15.187:0/838147768 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efc4c0701c0 con 0x7efc5c1431c0 2020-11-20T12:22:07.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.633+0000 7efc537fe700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc38056d50 0x7efc38059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:07.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.633+0000 7efc537fe700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc38056d50 0x7efc38059210 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7efc5c0bcaa0 tx=0x7efc44015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:07.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.639+0000 7efc58ff9700 1 -- 172.21.15.187:0/838147768 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efc4c045590 con 0x7efc5c1431c0 2020-11-20T12:22:07.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.827+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc5c0b8840 con 0x7efc5c1431c0 2020-11-20T12:22:07.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.830+0000 7efc58ff9700 1 -- 172.21.15.187:0/838147768 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16859 (secure 0 0 0) 0x7efc4c017020 con 0x7efc5c1431c0 2020-11-20T12:22:07.833 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:07.834 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":161,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:22:04.549389+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:21:20.315283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2020-11-20T12:21:21.644244+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":160,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":32,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:22:07.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc38056d50 msgr2=0x7efc38059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:07.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc38056d50 0x7efc38059210 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7efc5c0bcaa0 tx=0x7efc44015000).stop 2020-11-20T12:22:07.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5c1431c0 msgr2=0x7efc5c15c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:07.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5c1431c0 0x7efc5c15c900 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7efc4c006d00 tx=0x7efc4c008040).stop 2020-11-20T12:22:07.837 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 shutdown_connections 2020-11-20T12:22:07.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7efc38056d50 0x7efc38059210 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:07.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc5c146b70 0x7efc5c0b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:07.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc5c1431c0 0x7efc5c15c900 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:07.838 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 --2- 172.21.15.187:0/838147768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc5c1435a0 0x7efc5c146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:07.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 >> 172.21.15.187:0/838147768 conn(0x7efc5c068370 msgr2=0x7efc5c0bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:07.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 shutdown_connections 2020-11-20T12:22:07.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:07.832+0000 7efc617d7700 1 -- 172.21.15.187:0/838147768 wait complete. 2020-11-20T12:22:07.860 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:22:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.101+0000 7fcc258f6700 1 Processor -- start 2020-11-20T12:22:08.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.101+0000 7fcc258f6700 1 -- start start 2020-11-20T12:22:08.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.103+0000 7fcc258f6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc2014d270 0x7fcc201573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.103+0000 7fcc258f6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc200658a0 con 0x7fcc200b1380 2020-11-20T12:22:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.103+0000 7fcc258f6700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc20150390 con 0x7fcc200b1760 2020-11-20T12:22:08.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.103+0000 7fcc258f6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201511e0 con 0x7fcc2014d270 2020-11-20T12:22:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.104+0000 7fcc17fff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fcc200b1760 0x7fcc20065190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51600/0 (socket says 172.21.15.187:51600) 2020-11-20T12:22:08.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.104+0000 7fcc17fff700 1 -- 172.21.15.187:0/2388876158 learned_addr learned my addr 172.21.15.187:0/2388876158 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.104+0000 7fcc250f5700 1 --2- 172.21.15.187:0/2388876158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc2014d270 0x7fcc201573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.104+0000 7fcc177fe700 1 -- 172.21.15.187:0/2388876158 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977329329 0 0) 0x7fcc20150390 con 0x7fcc200b1760 2020-11-20T12:22:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.105+0000 7fcc177fe700 1 -- 172.21.15.187:0/2388876158 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c003240 con 0x7fcc200b1760 2020-11-20T12:22:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.105+0000 7fcc250f5700 1 -- 172.21.15.187:0/2388876158 >> v1:172.21.15.187:6789/0 conn(0x7fcc200b1760 legacy=0x7fcc20065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.105+0000 7fcc177fe700 1 -- 172.21.15.187:0/2388876158 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263113855 0 0) 0x7fcc200658a0 con 0x7fcc200b1380 2020-11-20T12:22:08.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.105+0000 7fcc250f5700 1 -- 172.21.15.187:0/2388876158 >> v1:172.21.15.5:6789/0 conn(0x7fcc200b1380 legacy=0x7fcc20150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.105+0000 7fcc250f5700 1 -- 172.21.15.187:0/2388876158 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc20150630 con 0x7fcc2014d270 2020-11-20T12:22:08.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.105+0000 7fcc250f5700 1 --2- 172.21.15.187:0/2388876158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc2014d270 0x7fcc201573a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcc1800b9d0 tx=0x7fcc18006df0).ready entity=mon.2 client_cookie=b4c2a42ada483c09 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:08.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.106+0000 7fcc177fe700 1 -- 172.21.15.187:0/2388876158 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc180097b0 con 0x7fcc2014d270 2020-11-20T12:22:08.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.106+0000 7fcc177fe700 1 -- 172.21.15.187:0/2388876158 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18009910 con 0x7fcc2014d270 2020-11-20T12:22:08.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.106+0000 7fcc177fe700 1 -- 172.21.15.187:0/2388876158 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18009c90 con 0x7fcc2014d270 2020-11-20T12:22:08.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.106+0000 7fcc258f6700 1 -- 172.21.15.187:0/2388876158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc2014d270 msgr2=0x7fcc201573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.106+0000 7fcc258f6700 1 --2- 172.21.15.187:0/2388876158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc2014d270 0x7fcc201573a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcc1800b9d0 tx=0x7fcc18006df0).stop 2020-11-20T12:22:08.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.107+0000 7fcc258f6700 1 -- 172.21.15.187:0/2388876158 shutdown_connections 2020-11-20T12:22:08.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.107+0000 7fcc258f6700 1 --2- 172.21.15.187:0/2388876158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc2014d270 0x7fcc201573a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.107+0000 7fcc258f6700 1 -- 172.21.15.187:0/2388876158 >> 172.21.15.187:0/2388876158 conn(0x7fcc200bab30 msgr2=0x7fcc200baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:08.112 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.107+0000 7fcc258f6700 1 -- 172.21.15.187:0/2388876158 shutdown_connections 2020-11-20T12:22:08.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.107+0000 7fcc258f6700 1 -- 172.21.15.187:0/2388876158 wait complete. 2020-11-20T12:22:08.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.108+0000 7fcc258f6700 1 Processor -- start 2020-11-20T12:22:08.113 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.108+0000 7fcc258f6700 1 -- start start 2020-11-20T12:22:08.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.109+0000 7fcc258f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc200b1380 0x7fcc2015cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.109+0000 7fcc258f6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc200b1760 0x7fcc2015ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.109+0000 7fcc258f6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc2014d270 0x7fcc2014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.115 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.109+0000 7fcc258f6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc20160420 con 0x7fcc2014d270 2020-11-20T12:22:08.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.109+0000 7fcc258f6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2015c120 con 0x7fcc200b1380 2020-11-20T12:22:08.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.109+0000 7fcc258f6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2015c3c0 con 0x7fcc200b1760 2020-11-20T12:22:08.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.110+0000 7fcc248f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc200b1380 0x7fcc2015cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.116 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.110+0000 7fcc248f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc200b1380 0x7fcc2015cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54998/0 (socket says 172.21.15.187:54998) 2020-11-20T12:22:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.110+0000 7fcc248f4700 1 -- 172.21.15.187:0/711626430 learned_addr learned my addr 172.21.15.187:0/711626430 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.110+0000 7fcc248f4700 1 -- 172.21.15.187:0/711626430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc200b1760 msgr2=0x7fcc2015ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.110+0000 7fcc248f4700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc200b1760 0x7fcc2015ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.110+0000 7fcc248f4700 1 -- 172.21.15.187:0/711626430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc2014d270 msgr2=0x7fcc2014b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.110+0000 7fcc248f4700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc2014d270 0x7fcc2014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.110+0000 7fcc248f4700 1 -- 172.21.15.187:0/711626430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2015c660 con 0x7fcc200b1380 2020-11-20T12:22:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.111+0000 7fcc248f4700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc200b1380 0x7fcc2015cb00 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7fcc1c00ca30 tx=0x7fcc1c007e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.111+0000 7fcc14ff9700 1 -- 172.21.15.187:0/711626430 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c00a830 con 0x7fcc200b1380 2020-11-20T12:22:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.111+0000 7fcc14ff9700 1 -- 172.21.15.187:0/711626430 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c00a9d0 con 0x7fcc200b1380 2020-11-20T12:22:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.111+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2014b9b0 con 0x7fcc200b1380 2020-11-20T12:22:08.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.111+0000 7fcc14ff9700 1 -- 172.21.15.187:0/711626430 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c00ad90 con 0x7fcc200b1380 2020-11-20T12:22:08.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.111+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2009c880 con 0x7fcc200b1380 2020-11-20T12:22:08.119 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.112+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2015c7f0 con 0x7fcc200b1380 2020-11-20T12:22:08.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.113+0000 7fcc14ff9700 1 -- 172.21.15.187:0/711626430 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcc1c010070 con 0x7fcc200b1380 2020-11-20T12:22:08.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.113+0000 7fcc14ff9700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbf8056d20 0x7fcbf80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.114+0000 7fcc14ff9700 1 -- 172.21.15.187:0/711626430 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcc1c06ff60 con 0x7fcc200b1380 2020-11-20T12:22:08.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.114+0000 7fcc17fff700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbf8056d20 0x7fcbf80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.114+0000 7fcc17fff700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbf8056d20 0x7fcbf80591e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7fcc100055a0 tx=0x7fcc100076a0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:08.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.117+0000 7fcc14ff9700 1 -- 172.21.15.187:0/711626430 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc1c06cfd0 con 0x7fcc200b1380 2020-11-20T12:22:08.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.301+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc20061a20 con 0x7fcbf8056d20 2020-11-20T12:22:08.317 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.314+0000 7fcc14ff9700 1 -- 172.21.15.187:0/711626430 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+104784 (secure 0 0 0) 0x7fcc20061a20 con 0x7fcbf8056d20 2020-11-20T12:22:08.317 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:08.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbf8056d20 msgr2=0x7fcbf80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbf8056d20 0x7fcbf80591e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7fcc100055a0 tx=0x7fcc100076a0).stop 2020-11-20T12:22:08.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc200b1380 msgr2=0x7fcc2015cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.325 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc200b1380 0x7fcc2015cb00 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7fcc1c00ca30 tx=0x7fcc1c007e40).stop 2020-11-20T12:22:08.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 shutdown_connections 2020-11-20T12:22:08.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcbf8056d20 0x7fcbf80591e0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc2014d270 0x7fcc2014b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:08.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc200b1380 0x7fcc2015cb00 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 --2- 172.21.15.187:0/711626430 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcc200b1760 0x7fcc2015ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:08.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 >> 172.21.15.187:0/711626430 conn(0x7fcc200bab30 msgr2=0x7fcc200baf10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:08.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 shutdown_connections 2020-11-20T12:22:08.327 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.317+0000 7fcc258f6700 1 -- 172.21.15.187:0/711626430 wait complete. 2020-11-20T12:22:08.327 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2020-11-20T12:22:08.342 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2020-11-20T12:22:07.845504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62081,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":8194,"num_write":148,"num_write_kb":16514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":214056,"kb_used_data":16168,"kb_used_omap":0,"kb_used_meta":197824,"kb_avail":754760664,"statfs":{"total":773094113280,"available":772874919936,"internally_reserved":0,"allocated":16556032,"data_stored":7783670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":202571776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4192620,"num_objects":-2,"num_object_clones":-1,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":-1,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003301"},"pg_stats":[{"pgid":"32.4","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.563482+0000","last_change":"2020-11-20T12:21:56.934847+0000","last_active":"2020-11-20T12:22:04.563482+0000","last_peered":"2020-11-20T12:22:04.563482+0000","last_clean":"2020-11-20T12:22:04.563482+0000","last_became_active":"2020-11-20T12:21:22.693729+0000","last_became_peered":"2020-11-20T12:21:22.693729+0000","last_unstale":"2020-11-20T12:22:04.563482+0000","last_undegraded":"2020-11-20T12:22:04.563482+0000","last_fullsized":"2020-11-20T12:22:04.563482+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.826440+0000","last_change":"2020-11-20T12:21:57.112403+0000","last_active":"2020-11-20T12:22:04.826440+0000","last_peered":"2020-11-20T12:22:04.826440+0000","last_clean":"2020-11-20T12:22:04.826440+0000","last_became_active":"2020-11-20T12:21:22.685927+0000","last_became_peered":"2020-11-20T12:21:22.685927+0000","last_unstale":"2020-11-20T12:22:04.826440+0000","last_undegraded":"2020-11-20T12:22:04.826440+0000","last_fullsized":"2020-11-20T12:22:04.826440+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2","version":"160'10","reported_seq":"30","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.563406+0000","last_change":"2020-11-20T12:21:56.945182+0000","last_active":"2020-11-20T12:22:04.563406+0000","last_peered":"2020-11-20T12:22:04.563406+0000","last_clean":"2020-11-20T12:22:04.563406+0000","last_became_active":"2020-11-20T12:21:22.693767+0000","last_became_peered":"2020-11-20T12:21:22.693767+0000","last_unstale":"2020-11-20T12:22:04.563406+0000","last_undegraded":"2020-11-20T12:22:04.563406+0000","last_fullsized":"2020-11-20T12:22:04.563406+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":6,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.567518+0000","last_change":"2020-11-20T12:21:56.936178+0000","last_active":"2020-11-20T12:22:04.567518+0000","last_peered":"2020-11-20T12:22:04.567518+0000","last_clean":"2020-11-20T12:22:04.567518+0000","last_became_active":"2020-11-20T12:21:22.684395+0000","last_became_peered":"2020-11-20T12:21:22.684395+0000","last_unstale":"2020-11-20T12:22:04.567518+0000","last_undegraded":"2020-11-20T12:22:04.567518+0000","last_fullsized":"2020-11-20T12:22:04.567518+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.563331+0000","last_change":"2020-11-20T12:21:56.937043+0000","last_active":"2020-11-20T12:22:04.563331+0000","last_peered":"2020-11-20T12:22:04.563331+0000","last_clean":"2020-11-20T12:22:04.563331+0000","last_became_active":"2020-11-20T12:21:22.692492+0000","last_became_peered":"2020-11-20T12:21:22.692492+0000","last_unstale":"2020-11-20T12:22:04.563331+0000","last_undegraded":"2020-11-20T12:22:04.563331+0000","last_fullsized":"2020-11-20T12:22:04.563331+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"159'16","reported_seq":"28","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.110707+0000","last_change":"2020-11-20T12:21:22.109003+0000","last_active":"2020-11-20T12:21:57.110707+0000","last_peered":"2020-11-20T12:21:57.110707+0000","last_clean":"2020-11-20T12:21:57.110707+0000","last_became_active":"2020-11-20T12:21:22.108194+0000","last_became_peered":"2020-11-20T12:21:22.108194+0000","last_unstale":"2020-11-20T12:21:57.110707+0000","last_undegraded":"2020-11-20T12:21:57.110707+0000","last_fullsized":"2020-11-20T12:21:57.110707+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"161'12","reported_seq":"24","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:05.191362+0000","last_change":"2020-11-20T12:21:22.114586+0000","last_active":"2020-11-20T12:22:05.191362+0000","last_peered":"2020-11-20T12:22:05.191362+0000","last_clean":"2020-11-20T12:22:05.191362+0000","last_became_active":"2020-11-20T12:21:22.113651+0000","last_became_peered":"2020-11-20T12:21:22.113651+0000","last_unstale":"2020-11-20T12:22:05.191362+0000","last_undegraded":"2020-11-20T12:22:05.191362+0000","last_fullsized":"2020-11-20T12:22:05.191362+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"159'12","reported_seq":"24","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:56.931064+0000","last_change":"2020-11-20T12:21:22.113496+0000","last_active":"2020-11-20T12:21:56.931064+0000","last_peered":"2020-11-20T12:21:56.931064+0000","last_clean":"2020-11-20T12:21:56.931064+0000","last_became_active":"2020-11-20T12:21:22.111838+0000","last_became_peered":"2020-11-20T12:21:22.111838+0000","last_unstale":"2020-11-20T12:21:56.931064+0000","last_undegraded":"2020-11-20T12:21:56.931064+0000","last_fullsized":"2020-11-20T12:21:56.931064+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":184,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.4","version":"160'44","reported_seq":"56","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:22:04.157405+0000","last_change":"2020-11-20T12:21:22.108311+0000","last_active":"2020-11-20T12:22:04.157405+0000","last_peered":"2020-11-20T12:22:04.157405+0000","last_clean":"2020-11-20T12:22:04.157405+0000","last_became_active":"2020-11-20T12:21:22.107044+0000","last_became_peered":"2020-11-20T12:21:22.107044+0000","last_unstale":"2020-11-20T12:22:04.157405+0000","last_undegraded":"2020-11-20T12:22:04.157405+0000","last_fullsized":"2020-11-20T12:22:04.157405+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61078,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"159'12","reported_seq":"23","reported_epoch":"159","state":"active+clean","last_fresh":"2020-11-20T12:21:56.326269+0000","last_change":"2020-11-20T12:21:22.105792+0000","last_active":"2020-11-20T12:21:56.326269+0000","last_peered":"2020-11-20T12:21:56.326269+0000","last_clean":"2020-11-20T12:21:56.326269+0000","last_became_active":"2020-11-20T12:21:22.105399+0000","last_became_peered":"2020-11-20T12:21:22.105399+0000","last_unstale":"2020-11-20T12:21:56.326269+0000","last_undegraded":"2020-11-20T12:21:56.326269+0000","last_fullsized":"2020-11-20T12:21:56.326269+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"158'1","reported_seq":"13","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.111049+0000","last_change":"2020-11-20T12:21:22.098417+0000","last_active":"2020-11-20T12:21:57.111049+0000","last_peered":"2020-11-20T12:21:57.111049+0000","last_clean":"2020-11-20T12:21:57.111049+0000","last_became_active":"2020-11-20T12:21:22.098007+0000","last_became_peered":"2020-11-20T12:21:22.098007+0000","last_unstale":"2020-11-20T12:21:57.111049+0000","last_undegraded":"2020-11-20T12:21:57.111049+0000","last_fullsized":"2020-11-20T12:21:57.111049+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"159'3","reported_seq":"17","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.110530+0000","last_change":"2020-11-20T12:21:22.103551+0000","last_active":"2020-11-20T12:21:57.110530+0000","last_peered":"2020-11-20T12:21:57.110530+0000","last_clean":"2020-11-20T12:21:57.110530+0000","last_became_active":"2020-11-20T12:21:22.102371+0000","last_became_peered":"2020-11-20T12:21:22.102371+0000","last_unstale":"2020-11-20T12:21:57.110530+0000","last_undegraded":"2020-11-20T12:21:57.110530+0000","last_fullsized":"2020-11-20T12:21:57.110530+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":210,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"159'13","reported_seq":"25","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.111610+0000","last_change":"2020-11-20T12:21:22.094228+0000","last_active":"2020-11-20T12:21:57.111610+0000","last_peered":"2020-11-20T12:21:57.111610+0000","last_clean":"2020-11-20T12:21:57.111610+0000","last_became_active":"2020-11-20T12:21:22.093851+0000","last_became_peered":"2020-11-20T12:21:22.093851+0000","last_unstale":"2020-11-20T12:21:57.111610+0000","last_undegraded":"2020-11-20T12:21:57.111610+0000","last_fullsized":"2020-11-20T12:21:57.111610+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"146","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.111417+0000","last_change":"2020-11-20T12:04:53.296291+0000","last_active":"2020-11-20T12:21:57.111417+0000","last_peered":"2020-11-20T12:21:57.111417+0000","last_clean":"2020-11-20T12:21:57.111417+0000","last_became_active":"2020-11-20T12:04:53.294169+0000","last_became_peered":"2020-11-20T12:04:53.294169+0000","last_unstale":"2020-11-20T12:21:57.111417+0000","last_undegraded":"2020-11-20T12:21:57.111417+0000","last_fullsized":"2020-11-20T12:21:57.111417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"160'8","reported_seq":"26","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.122565+0000","last_change":"2020-11-20T12:21:57.123189+0000","last_active":"2020-11-20T12:21:57.122565+0000","last_peered":"2020-11-20T12:21:57.122565+0000","last_clean":"2020-11-20T12:21:57.122565+0000","last_became_active":"2020-11-20T12:21:22.676940+0000","last_became_peered":"2020-11-20T12:21:22.676940+0000","last_unstale":"2020-11-20T12:21:57.122565+0000","last_undegraded":"2020-11-20T12:21:57.122565+0000","last_fullsized":"2020-11-20T12:21:57.122565+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":4,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.112069+0000","last_change":"2020-11-20T12:04:53.294740+0000","last_active":"2020-11-20T12:21:57.112069+0000","last_peered":"2020-11-20T12:21:57.112069+0000","last_clean":"2020-11-20T12:21:57.112069+0000","last_became_active":"2020-11-20T12:04:53.293882+0000","last_became_peered":"2020-11-20T12:04:53.293882+0000","last_unstale":"2020-11-20T12:21:57.112069+0000","last_undegraded":"2020-11-20T12:21:57.112069+0000","last_fullsized":"2020-11-20T12:21:57.112069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"145","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.110499+0000","last_change":"2020-11-20T12:04:53.299007+0000","last_active":"2020-11-20T12:21:57.110499+0000","last_peered":"2020-11-20T12:21:57.110499+0000","last_clean":"2020-11-20T12:21:57.110499+0000","last_became_active":"2020-11-20T12:04:53.297849+0000","last_became_peered":"2020-11-20T12:04:53.297849+0000","last_unstale":"2020-11-20T12:21:57.110499+0000","last_undegraded":"2020-11-20T12:21:57.110499+0000","last_fullsized":"2020-11-20T12:21:57.110499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":"18","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.113626+0000","last_change":"2020-11-20T12:21:57.114223+0000","last_active":"2020-11-20T12:21:57.113626+0000","last_peered":"2020-11-20T12:21:57.113626+0000","last_clean":"2020-11-20T12:21:57.113626+0000","last_became_active":"2020-11-20T12:21:22.675260+0000","last_became_peered":"2020-11-20T12:21:22.675260+0000","last_unstale":"2020-11-20T12:21:57.113626+0000","last_undegraded":"2020-11-20T12:21:57.113626+0000","last_fullsized":"2020-11-20T12:21:57.113626+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.112390+0000","last_change":"2020-11-20T12:04:54.004976+0000","last_active":"2020-11-20T12:21:57.112390+0000","last_peered":"2020-11-20T12:21:57.112390+0000","last_clean":"2020-11-20T12:21:57.112390+0000","last_became_active":"2020-11-20T12:04:54.004416+0000","last_became_peered":"2020-11-20T12:04:54.004416+0000","last_unstale":"2020-11-20T12:21:57.112390+0000","last_undegraded":"2020-11-20T12:21:57.112390+0000","last_fullsized":"2020-11-20T12:21:57.112390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'2","reported_seq":"145","reported_epoch":"159","state":"active+clean","last_fresh":"2020-11-20T12:21:24.791179+0000","last_change":"2020-11-20T12:04:49.157393+0000","last_active":"2020-11-20T12:21:24.791179+0000","last_peered":"2020-11-20T12:21:24.791179+0000","last_clean":"2020-11-20T12:21:24.791179+0000","last_became_active":"2020-11-20T12:04:49.156248+0000","last_became_peered":"2020-11-20T12:04:49.156248+0000","last_unstale":"2020-11-20T12:21:24.791179+0000","last_undegraded":"2020-11-20T12:21:24.791179+0000","last_fullsized":"2020-11-20T12:21:24.791179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:44.517280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:44.517280+0000","last_clean_scrub_stamp":"2020-11-20T12:04:44.517280+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.7","version":"159'1","reported_seq":"21","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.568502+0000","last_change":"2020-11-20T12:21:56.937079+0000","last_active":"2020-11-20T12:22:04.568502+0000","last_peered":"2020-11-20T12:22:04.568502+0000","last_clean":"2020-11-20T12:22:04.568502+0000","last_became_active":"2020-11-20T12:21:22.697706+0000","last_became_peered":"2020-11-20T12:21:22.697706+0000","last_unstale":"2020-11-20T12:22:04.568502+0000","last_undegraded":"2020-11-20T12:22:04.568502+0000","last_fullsized":"2020-11-20T12:22:04.568502+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.111741+0000","last_change":"2020-11-20T12:04:53.297146+0000","last_active":"2020-11-20T12:21:57.111741+0000","last_peered":"2020-11-20T12:21:57.111741+0000","last_clean":"2020-11-20T12:21:57.111741+0000","last_became_active":"2020-11-20T12:04:53.296376+0000","last_became_peered":"2020-11-20T12:04:53.296376+0000","last_unstale":"2020-11-20T12:21:57.111741+0000","last_undegraded":"2020-11-20T12:21:57.111741+0000","last_fullsized":"2020-11-20T12:21:57.111741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.113106+0000","last_change":"2020-11-20T12:04:53.303192+0000","last_active":"2020-11-20T12:21:57.113106+0000","last_peered":"2020-11-20T12:21:57.113106+0000","last_clean":"2020-11-20T12:21:57.113106+0000","last_became_active":"2020-11-20T12:04:53.302386+0000","last_became_peered":"2020-11-20T12:04:53.302386+0000","last_unstale":"2020-11-20T12:21:57.113106+0000","last_undegraded":"2020-11-20T12:21:57.113106+0000","last_fullsized":"2020-11-20T12:21:57.113106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"146","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:56.931724+0000","last_change":"2020-11-20T12:04:54.003353+0000","last_active":"2020-11-20T12:21:56.931724+0000","last_peered":"2020-11-20T12:21:56.931724+0000","last_clean":"2020-11-20T12:21:56.931724+0000","last_became_active":"2020-11-20T12:04:54.002137+0000","last_became_peered":"2020-11-20T12:04:54.002137+0000","last_unstale":"2020-11-20T12:21:56.931724+0000","last_undegraded":"2020-11-20T12:21:56.931724+0000","last_fullsized":"2020-11-20T12:21:56.931724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.113362+0000","last_change":"2020-11-20T12:04:54.001880+0000","last_active":"2020-11-20T12:21:57.113362+0000","last_peered":"2020-11-20T12:21:57.113362+0000","last_clean":"2020-11-20T12:21:57.113362+0000","last_became_active":"2020-11-20T12:04:54.001419+0000","last_became_peered":"2020-11-20T12:04:54.001419+0000","last_unstale":"2020-11-20T12:21:57.113362+0000","last_undegraded":"2020-11-20T12:21:57.113362+0000","last_fullsized":"2020-11-20T12:21:57.113362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":10,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":16,"acting":16,"num_store_stats":7},{"poolid":31,"num_pg":8,"stat_sum":{"num_bytes":62081,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":136,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":124162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542354,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20144,"kb_used_data":2088,"kb_used_omap":0,"kb_used_meta":18048,"kb_avail":94351696,"statfs":{"total":96636764160,"available":96616136704,"internally_reserved":0,"allocated":2138112,"data_stored":1020351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18481152},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.3050000000000002,"5min":2.7290000000000001,"15min":2.4750000000000001},"min":{"1min":1.508,"5min":0.63,"15min":0.63},"max":{"1min":3.4500000000000002,"5min":37.622,"15min":37.622},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.881,"5min":2.7149999999999999,"15min":2.4489999999999998},"min":{"1min":1.2949999999999999,"5min":1.206,"15min":1.206},"max":{"1min":2.8980000000000001,"5min":36.816000000000003,"15min":36.816000000000003},"last":1.891}]},{"osd":1,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":2.5089999999999999,"5min":3.4500000000000002,"15min":3.1259999999999999},"min":{"1min":1.411,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":3.6859999999999999,"5min":42.296999999999997,"15min":42.296999999999997},"last":1.869},{"interface":"front","average":{"1min":2.431,"5min":16.081,"15min":7.0220000000000002},"min":{"1min":1.627,"5min":1.0600000000000001,"15min":1.0600000000000001},"max":{"1min":3.6360000000000001,"5min":633.35299999999995,"15min":633.35299999999995},"last":1.4179999999999999}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.903,"15min":2.798},"min":{"1min":1.861,"5min":1.073,"15min":1.073},"max":{"1min":3.9790000000000001,"5min":7.359,"15min":7.359},"last":1.9199999999999999},{"interface":"front","average":{"1min":2.6800000000000002,"5min":3.2690000000000001,"15min":3.0259999999999998},"min":{"1min":1.7769999999999999,"5min":1.6000000000000001,"15min":1.4259999999999999},"max":{"1min":3.9649999999999999,"5min":36.509,"15min":36.509},"last":1.4990000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.2410000000000001,"5min":3.762,"15min":3.6000000000000001},"min":{"1min":2.2000000000000002,"5min":1.129,"15min":1.129},"max":{"1min":4.2439999999999998,"5min":37.859000000000002,"15min":37.859000000000002},"last":1.633},{"interface":"front","average":{"1min":3.218,"5min":3.3599999999999999,"15min":3.306},"min":{"1min":1.968,"5min":1.516,"15min":1.45},"max":{"1min":4.3730000000000002,"5min":21.952999999999999,"15min":21.952999999999999},"last":2.0390000000000001}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.2320000000000002,"5min":4.0490000000000004,"15min":3.6699999999999999},"min":{"1min":1.411,"5min":1.3160000000000001,"15min":1.2809999999999999},"max":{"1min":4.4640000000000004,"5min":36.979999999999997,"15min":36.979999999999997},"last":1.0429999999999999},{"interface":"front","average":{"1min":3.79,"5min":4.4329999999999998,"15min":4.2199999999999998},"min":{"1min":1.9530000000000001,"5min":1.2649999999999999,"15min":1.2649999999999999},"max":{"1min":4.9729999999999999,"5min":38.177,"15min":38.177},"last":1.1879999999999999}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.8319999999999999,"5min":3.6629999999999998,"15min":3.694},"min":{"1min":2.7080000000000002,"5min":1.351,"15min":1.351},"max":{"1min":4.9950000000000001,"5min":5.2590000000000003,"15min":5.8129999999999997},"last":1.0920000000000001},{"interface":"front","average":{"1min":4.1699999999999999,"5min":4.6589999999999998,"15min":4.4710000000000001},"min":{"1min":3.2450000000000001,"5min":2.1619999999999999,"15min":1.7809999999999999},"max":{"1min":5.3040000000000003,"5min":37.438000000000002,"15min":37.438000000000002},"last":1.4770000000000001}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.9870000000000001,"5min":4.3250000000000002,"15min":4.117},"min":{"1min":2.6589999999999998,"5min":1.5389999999999999,"15min":1.5389999999999999},"max":{"1min":7.1710000000000003,"5min":36.674999999999997,"15min":36.674999999999997},"last":1.218},{"interface":"front","average":{"1min":4.4420000000000002,"5min":4.9039999999999999,"15min":4.7629999999999999},"min":{"1min":2.7589999999999999,"5min":1.401,"15min":1.401},"max":{"1min":6.5069999999999997,"5min":38.027000000000001,"15min":38.027000000000001},"last":1.3380000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542354,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30048,"kb_used_data":2072,"kb_used_omap":0,"kb_used_meta":27968,"kb_avail":94341792,"statfs":{"total":96636764160,"available":96605995008,"internally_reserved":0,"allocated":2121728,"data_stored":1019416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28639232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":2.1659999999999999,"5min":2.2010000000000001,"15min":2.1779999999999999},"min":{"1min":1.052,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":5.0380000000000003,"5min":5.6600000000000001,"15min":5.6600000000000001},"last":1.121},{"interface":"front","average":{"1min":2.5670000000000002,"5min":2.3809999999999998,"15min":2.4039999999999999},"min":{"1min":1.6839999999999999,"5min":0.86799999999999999,"15min":0.79500000000000004},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":7.1630000000000003},"last":1.5109999999999999}]},{"osd":1,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":2.9809999999999999,"5min":2.6859999999999999,"15min":2.5800000000000001},"min":{"1min":1.1639999999999999,"5min":0.78800000000000003,"15min":0.70599999999999996},"max":{"1min":12.022,"5min":12.022,"15min":12.022},"last":1.627},{"interface":"front","average":{"1min":3.4239999999999999,"5min":5.3120000000000003,"15min":3.7869999999999999},"min":{"1min":2.274,"5min":1.252,"15min":0.84699999999999998},"max":{"1min":6.3410000000000002,"5min":140.53299999999999,"15min":140.53299999999999},"last":1.982}]},{"osd":2,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":3.5419999999999998,"5min":3.0259999999999998,"15min":3.0539999999999998},"min":{"1min":1.804,"5min":1.3,"15min":1.02},"max":{"1min":12.430999999999999,"5min":12.430999999999999,"15min":12.430999999999999},"last":2.02},{"interface":"front","average":{"1min":3.0830000000000002,"5min":3.0739999999999998,"15min":3.04},"min":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.2370000000000001},"max":{"1min":5.6600000000000001,"5min":5.7619999999999996,"15min":7.5140000000000002},"last":1.5620000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":3.3879999999999999,"5min":3.3919999999999999,"15min":3.4460000000000002},"min":{"1min":2.2330000000000001,"5min":1.6140000000000001,"15min":1.121},"max":{"1min":5,"5min":7.274,"15min":8.2249999999999996},"last":1.6559999999999999},{"interface":"front","average":{"1min":4.0490000000000004,"5min":3.5419999999999998,"15min":3.5840000000000001},"min":{"1min":2.3679999999999999,"5min":1.105,"15min":1.105},"max":{"1min":13.329000000000001,"5min":13.329000000000001,"15min":13.329000000000001},"last":1.8700000000000001}]},{"osd":4,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":3.8279999999999998,"5min":3.6480000000000001,"15min":3.621},"min":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.0900000000000001},"max":{"1min":11.247,"5min":11.247,"15min":11.247},"last":0.97799999999999998},{"interface":"front","average":{"1min":4.3710000000000004,"5min":3.9350000000000001,"15min":4.0049999999999999},"min":{"1min":3.0640000000000001,"5min":1.349,"15min":1.2809999999999999},"max":{"1min":10.82,"5min":10.82,"15min":11.474},"last":1.335}]},{"osd":5,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":4.4059999999999997,"5min":4.1539999999999999,"15min":4.1310000000000002},"min":{"1min":2.7719999999999998,"5min":1.4319999999999999,"15min":1.3400000000000001},"max":{"1min":11.568,"5min":11.568,"15min":14.435},"last":1.468},{"interface":"front","average":{"1min":4.0579999999999998,"5min":4.1319999999999997,"15min":4.1600000000000001},"min":{"1min":2.2120000000000002,"5min":1.7789999999999999,"15min":1.599},"max":{"1min":6.8140000000000001,"5min":9.5850000000000009,"15min":22.896000000000001},"last":1.278}]},{"osd":7,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":4.5960000000000001,"5min":4.3529999999999998,"15min":4.3659999999999997},"min":{"1min":3.145,"5min":2.0049999999999999,"15min":1.738},"max":{"1min":7.6139999999999999,"5min":8.2379999999999995,"15min":11.917},"last":1.732},{"interface":"front","average":{"1min":4.9329999999999998,"5min":4.4189999999999996,"15min":4.5359999999999996},"min":{"1min":2.9399999999999999,"5min":1.5620000000000001,"15min":1.5069999999999999},"max":{"1min":16.263000000000002,"5min":16.263000000000002,"15min":23.207999999999998},"last":2.1850000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542353,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":30592,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":2043904,"data_stored":952227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31326208},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":1.7669999999999999,"5min":11.311999999999999,"15min":5.0890000000000004},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.78100000000000003},"max":{"1min":2.6640000000000001,"5min":868.745,"15min":868.745},"last":1.198},{"interface":"front","average":{"1min":1.9119999999999999,"5min":2.093,"15min":2.0910000000000002},"min":{"1min":0.71799999999999997,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":3.1869999999999998,"5min":7.6580000000000004,"15min":10.042},"last":1.1299999999999999}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.0089999999999999,"5min":2.4540000000000002,"15min":2.6659999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.68100000000000005},"max":{"1min":2.7400000000000002,"5min":20.43,"15min":142.95500000000001},"last":1.155},{"interface":"front","average":{"1min":2.5609999999999999,"5min":12.111000000000001,"15min":6.3460000000000001},"min":{"1min":1.04,"5min":1.04,"15min":0.95999999999999996},"max":{"1min":3.827,"5min":868.82299999999998,"15min":868.82299999999998},"last":1.6830000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":12.082000000000001,"15min":5.891},"min":{"1min":1.4299999999999999,"5min":1.2669999999999999,"15min":1.0409999999999999},"max":{"1min":3.2719999999999998,"5min":869.04300000000001,"15min":869.04300000000001},"last":1.385},{"interface":"front","average":{"1min":2.5339999999999998,"5min":2.9990000000000001,"15min":2.8300000000000001},"min":{"1min":1.4059999999999999,"5min":1.3029999999999999,"15min":0.94799999999999995},"max":{"1min":3.7599999999999998,"5min":11.404,"15min":11.404},"last":1.7909999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.915,"5min":3.2589999999999999,"15min":3.0169999999999999},"min":{"1min":1.5469999999999999,"5min":1.3580000000000001,"15min":0.96899999999999997},"max":{"1min":4.1870000000000003,"5min":20.577999999999999,"15min":20.577999999999999},"last":2.2999999999999998},{"interface":"front","average":{"1min":3.077,"5min":12.645,"15min":6.4370000000000003},"min":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.022},"max":{"1min":4.3280000000000003,"5min":869.22299999999996,"15min":869.22299999999996},"last":3.4969999999999999}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.1789999999999998,"5min":3.6589999999999998,"15min":3.5379999999999998},"min":{"1min":1.875,"5min":1.875,"15min":1.2649999999999999},"max":{"1min":5.1319999999999997,"5min":13.064,"15min":13.064},"last":3.1469999999999998},{"interface":"front","average":{"1min":3.2029999999999998,"5min":4.0309999999999997,"15min":3.5470000000000002},"min":{"1min":1.944,"5min":1.8440000000000001,"15min":1.1299999999999999},"max":{"1min":5.0730000000000004,"5min":62.292000000000002,"15min":62.292000000000002},"last":3.419}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.5190000000000001,"5min":13.275,"15min":6.8369999999999997},"min":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.3879999999999999},"max":{"1min":5.0110000000000001,"5min":869.15300000000002,"15min":869.15300000000002},"last":2.9830000000000001},{"interface":"front","average":{"1min":3.2490000000000001,"5min":3.6829999999999998,"15min":3.6989999999999998},"min":{"1min":2.012,"5min":2.012,"15min":1.573},"max":{"1min":4.8550000000000004,"5min":11.545,"15min":17.042999999999999},"last":2.5920000000000001}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.484,"5min":4.0129999999999999,"15min":3.984},"min":{"1min":1.944,"5min":1.944,"15min":1.411},"max":{"1min":4.8609999999999998,"5min":11.667,"15min":11.667},"last":3.3399999999999999},{"interface":"front","average":{"1min":3.6099999999999999,"5min":3.9510000000000001,"15min":3.7549999999999999},"min":{"1min":2.0760000000000001,"5min":1.222,"15min":1.222},"max":{"1min":5.3029999999999999,"5min":20.710000000000001,"15min":20.710000000000001},"last":2.6379999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21720,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":19712,"kb_avail":94350120,"statfs":{"total":96636764160,"available":96614522880,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20185088},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":1.845,"5min":2.1640000000000001,"15min":2.0870000000000002},"min":{"1min":0.93200000000000005,"5min":0.91200000000000003,"15min":0.71299999999999997},"max":{"1min":2.7160000000000002,"5min":7.3330000000000002,"15min":10.723000000000001},"last":2.681},{"interface":"front","average":{"1min":2.2669999999999999,"5min":2.6509999999999998,"15min":2.5299999999999998},"min":{"1min":1.1140000000000001,"5min":1.0589999999999999,"15min":0.98699999999999999},"max":{"1min":3.7210000000000001,"5min":7.8929999999999998,"15min":7.8929999999999998},"last":1.9159999999999999}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":1.9370000000000001,"5min":2.4550000000000001,"15min":2.1379999999999999},"min":{"1min":0.75,"5min":0.66400000000000003,"15min":0.66100000000000003},"max":{"1min":3.4980000000000002,"5min":42.066000000000003,"15min":42.066000000000003},"last":2.4350000000000001},{"interface":"front","average":{"1min":1.9750000000000001,"5min":2.2200000000000002,"15min":2.2269999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":3.2010000000000001,"5min":4.7990000000000004,"15min":4.7990000000000004},"last":2.3479999999999999}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.4700000000000002,"5min":3.1299999999999999,"15min":2.8149999999999999},"min":{"1min":1.095,"5min":0.76400000000000001,"15min":0.753},"max":{"1min":3.5310000000000001,"5min":38.067,"15min":38.067},"last":1.621},{"interface":"front","average":{"1min":2.7290000000000001,"5min":3.306,"15min":3.0710000000000002},"min":{"1min":1.242,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":4.2089999999999996,"5min":38.463000000000001,"15min":38.463000000000001},"last":3.048}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.827,"5min":3.1219999999999999,"15min":3.1000000000000001},"min":{"1min":1.444,"5min":1.05,"15min":1.05},"max":{"1min":4.4189999999999996,"5min":14.467000000000001,"15min":14.467000000000001},"last":3.1379999999999999},{"interface":"front","average":{"1min":2.8170000000000002,"5min":2.9609999999999999,"15min":3.0339999999999998},"min":{"1min":1.54,"5min":1.1579999999999999,"15min":1.1579999999999999},"max":{"1min":5.0090000000000003,"5min":5.0090000000000003,"15min":5.0679999999999996},"last":3.3530000000000002}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.222,"5min":3.387,"15min":3.3759999999999999},"min":{"1min":1.623,"5min":1.5589999999999999,"15min":1.4790000000000001},"max":{"1min":4.8929999999999998,"5min":7.8120000000000003,"15min":7.8120000000000003},"last":4.0759999999999996},{"interface":"front","average":{"1min":3.2759999999999998,"5min":3.6520000000000001,"15min":3.6579999999999999},"min":{"1min":1.6599999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"max":{"1min":4.2039999999999997,"5min":6.5780000000000003,"15min":11.58},"last":3.5369999999999999}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.254,"5min":3.444,"15min":3.4580000000000002},"min":{"1min":1.8049999999999999,"5min":1.29,"15min":1.29},"max":{"1min":4.8710000000000004,"5min":6.6230000000000002,"15min":6.6230000000000002},"last":3.609},{"interface":"front","average":{"1min":3.29,"5min":3.4849999999999999,"15min":3.536},"min":{"1min":1.9199999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"max":{"1min":4.4660000000000002,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":3.863}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.5329999999999999,"5min":4.0339999999999998,"15min":3.8490000000000002},"min":{"1min":2.161,"5min":1.357,"15min":1.357},"max":{"1min":4.6050000000000004,"5min":37.826999999999998,"15min":37.826999999999998},"last":4.298},{"interface":"front","average":{"1min":3.5419999999999998,"5min":3.7850000000000001,"15min":3.738},"min":{"1min":2.1219999999999999,"5min":1.7370000000000001,"15min":1.665},"max":{"1min":4.8200000000000003,"5min":6.7329999999999997,"15min":12.154},"last":4.0179999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":27136,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27787264},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":2.0489999999999999,"5min":1.994,"15min":2.1030000000000002},"min":{"1min":1.232,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":3.7400000000000002,"5min":3.8380000000000001,"15min":14.422000000000001},"last":3.5310000000000001},{"interface":"front","average":{"1min":2.5499999999999998,"5min":2.4140000000000001,"15min":2.516},"min":{"1min":1.2949999999999999,"5min":0.88100000000000001,"15min":0.871},"max":{"1min":3.472,"5min":4.6429999999999998,"15min":18.148},"last":3.2250000000000001}]},{"osd":1,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":2.6230000000000002,"15min":2.4500000000000002},"min":{"1min":1.587,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":4.0620000000000003,"5min":24.391999999999999,"15min":24.391999999999999},"last":2.875},{"interface":"front","average":{"1min":2.1829999999999998,"5min":11.826000000000001,"15min":5.4880000000000004},"min":{"1min":0.91300000000000003,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":3.8879999999999999,"5min":820.03499999999997,"15min":820.03499999999997},"last":2.0259999999999998}]},{"osd":3,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":2.2519999999999998,"5min":2.1269999999999998,"15min":2.6040000000000001},"min":{"1min":1.2569999999999999,"5min":0.97499999999999998,"15min":0.748},"max":{"1min":3.1480000000000001,"5min":4.1050000000000004,"15min":134.77600000000001},"last":1.5509999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":2.8700000000000001,"15min":2.9340000000000002},"min":{"1min":2.0510000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":3.8370000000000002,"5min":5.601,"15min":10.574999999999999},"last":2.2869999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":3.2610000000000001,"15min":3.2400000000000002},"min":{"1min":2.0569999999999999,"5min":1.462,"15min":1.462},"max":{"1min":4.2530000000000001,"5min":6.1589999999999998,"15min":10.132},"last":3.3170000000000002},{"interface":"front","average":{"1min":2.9940000000000002,"5min":3.1059999999999999,"15min":3.0950000000000002},"min":{"1min":1.8480000000000001,"5min":1.377,"15min":1.274},"max":{"1min":3.855,"5min":5.0650000000000004,"15min":11.913},"last":3.621}]},{"osd":5,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":3.1699999999999999,"5min":3.0499999999999998,"15min":3.5529999999999999},"min":{"1min":1.9039999999999999,"5min":1.8069999999999999,"15min":1.554},"max":{"1min":4.3319999999999999,"5min":5.6319999999999997,"15min":141.113},"last":3.024},{"interface":"front","average":{"1min":3.6619999999999999,"5min":3.5640000000000001,"15min":3.593},"min":{"1min":2.774,"5min":1.4379999999999999,"15min":1.4330000000000001},"max":{"1min":4.7199999999999998,"5min":5.8259999999999996,"15min":10.968999999999999},"last":3.1190000000000002}]},{"osd":6,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":3.4969999999999999,"5min":3.423,"15min":3.4950000000000001},"min":{"1min":2.048,"5min":1.3069999999999999,"15min":1.137},"max":{"1min":5.4790000000000001,"5min":5.532,"15min":16.738},"last":3.9940000000000002},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.4420000000000002,"15min":3.4860000000000002},"min":{"1min":2.7480000000000002,"5min":1.7789999999999999,"15min":1.702},"max":{"1min":4.8810000000000002,"5min":5.7930000000000001,"15min":6.7709999999999999},"last":3.4049999999999998}]},{"osd":7,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":3.6960000000000002,"5min":3.7450000000000001,"15min":3.7610000000000001},"min":{"1min":2.5990000000000002,"5min":1.7649999999999999,"15min":1.5860000000000001},"max":{"1min":4.7729999999999997,"5min":10.839,"15min":12.412000000000001},"last":4.1840000000000002},{"interface":"front","average":{"1min":3.8290000000000002,"5min":3.6240000000000001,"15min":3.915},"min":{"1min":2.621,"5min":2.0219999999999998,"15min":1.7030000000000001},"max":{"1min":4.8159999999999998,"5min":5.4880000000000004,"15min":73.125},"last":3.786}]}]},{"osd":3,"up_from":14,"seq":60129542354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":2.1000000000000001,"15min":1.9690000000000001},"min":{"1min":1.304,"5min":0.73399999999999999,"15min":0.72499999999999998},"max":{"1min":3.71,"5min":14.436999999999999,"15min":14.436999999999999},"last":2.77},{"interface":"front","average":{"1min":2.1469999999999998,"5min":2.0350000000000001,"15min":1.974},"min":{"1min":1.141,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":3.3100000000000001,"5min":3.4940000000000002,"15min":4.9870000000000001},"last":2.1200000000000001}]},{"osd":1,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.1920000000000002,"5min":3.8399999999999999,"15min":2.6869999999999998},"min":{"1min":1.5720000000000001,"5min":1.079,"15min":0.88800000000000001},"max":{"1min":3.8999999999999999,"5min":102.953,"15min":102.953},"last":3.21},{"interface":"front","average":{"1min":2.4380000000000002,"5min":2.6309999999999998,"15min":2.4969999999999999},"min":{"1min":1.1850000000000001,"5min":1.139,"15min":0.91700000000000004},"max":{"1min":3.3980000000000001,"5min":16.895,"15min":16.895},"last":2.3690000000000002}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.6150000000000002,"5min":2.6419999999999999,"15min":2.7189999999999999},"min":{"1min":1.2090000000000001,"5min":0.755,"15min":0.755},"max":{"1min":3.3580000000000001,"5min":5.085,"15min":6.2469999999999999},"last":3.8359999999999999},{"interface":"front","average":{"1min":2.992,"5min":3.1379999999999999,"15min":3.0019999999999998},"min":{"1min":1.8600000000000001,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":4.0389999999999997,"5min":14.715999999999999,"15min":14.715999999999999},"last":3.512}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.9180000000000001,"5min":2.964,"15min":3.2269999999999999},"min":{"1min":2.0449999999999999,"5min":1.454,"15min":1.0960000000000001},"max":{"1min":4.2279999999999998,"5min":5.0199999999999996,"15min":70.090999999999994},"last":3.3849999999999998},{"interface":"front","average":{"1min":3.2490000000000001,"5min":3.2189999999999999,"15min":3.444},"min":{"1min":2.0339999999999998,"5min":1.28,"15min":1.28},"max":{"1min":5.5199999999999996,"5min":10.496,"15min":71.367999999999995},"last":2.4689999999999999}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.355,"5min":3.4100000000000001,"15min":3.395},"min":{"1min":2.2509999999999999,"5min":1.659,"15min":1.5589999999999999},"max":{"1min":5.0279999999999996,"5min":10.621,"15min":10.621},"last":2.863},{"interface":"front","average":{"1min":3.677,"5min":3.7559999999999998,"15min":3.5190000000000001},"min":{"1min":2.6539999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":5.1760000000000002,"5min":17.579999999999998,"15min":17.579999999999998},"last":3.0699999999999998}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.5800000000000001,"5min":3.456,"15min":3.6459999999999999},"min":{"1min":2.4580000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"max":{"1min":4.6529999999999996,"5min":4.7889999999999997,"15min":59.569000000000003},"last":4.4119999999999999},{"interface":"front","average":{"1min":3.9940000000000002,"5min":3.9199999999999999,"15min":4.0910000000000002},"min":{"1min":2.6120000000000001,"5min":1.79,"15min":1.79},"max":{"1min":6.3890000000000002,"5min":10.726000000000001,"15min":60.023000000000003},"last":4.2889999999999997}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.851,"5min":3.9729999999999999,"15min":3.7690000000000001},"min":{"1min":2.762,"5min":1.595,"15min":1.595},"max":{"1min":4.9909999999999997,"5min":22.126000000000001,"15min":22.126000000000001},"last":4.0679999999999996},{"interface":"front","average":{"1min":3.7050000000000001,"5min":3.6669999999999998,"15min":3.5979999999999999},"min":{"1min":2.2370000000000001,"5min":1.7270000000000001,"15min":1.615},"max":{"1min":5.4489999999999998,"5min":8.2690000000000001,"15min":8.2690000000000001},"last":4}]}]},{"osd":4,"up_from":14,"seq":60129542354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29032,"kb_used_data":2016,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94342808,"statfs":{"total":96636764160,"available":96607035392,"internally_reserved":0,"allocated":2064384,"data_stored":959063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":2.133,"15min":2.3639999999999999},"min":{"1min":1.5569999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.5059999999999998,"5min":4.1429999999999998,"15min":57.734999999999999},"last":1.8839999999999999},{"interface":"front","average":{"1min":2.23,"5min":2.7309999999999999,"15min":2.431},"min":{"1min":1.581,"5min":1.3069999999999999,"15min":1.1539999999999999},"max":{"1min":3.6539999999999999,"5min":26.75,"15min":26.75},"last":1.964}]},{"osd":1,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.4319999999999999,"5min":2.879,"15min":2.6619999999999999},"min":{"1min":1.4330000000000001,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":3.923,"5min":18.234999999999999,"15min":18.234999999999999},"last":2.9980000000000002},{"interface":"front","average":{"1min":2.9849999999999999,"5min":3.2770000000000001,"15min":3.0939999999999999},"min":{"1min":2,"5min":1.696,"15min":1.397},"max":{"1min":4.4509999999999996,"5min":14.981,"15min":14.981},"last":2.6619999999999999}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.012,"5min":4.0350000000000001,"15min":3.323},"min":{"1min":1.9590000000000001,"5min":1.3009999999999999,"15min":1.3009999999999999},"max":{"1min":4.1550000000000002,"5min":63.558999999999997,"15min":63.558999999999997},"last":3.427},{"interface":"front","average":{"1min":2.911,"5min":3.0699999999999998,"15min":2.964},"min":{"1min":1.776,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":4.0519999999999996,"5min":12.754,"15min":12.754},"last":4.5190000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.3849999999999998,"5min":3.8929999999999998,"15min":3.5630000000000002},"min":{"1min":2.6579999999999999,"5min":1.9650000000000001,"15min":1.5609999999999999},"max":{"1min":4.0720000000000001,"5min":27.239999999999998,"15min":27.239999999999998},"last":3.9689999999999999},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.665,"15min":3.492},"min":{"1min":2.1619999999999999,"5min":1.6859999999999999,"15min":1.113},"max":{"1min":4.5,"5min":12.881,"15min":12.881},"last":4.0899999999999999}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":3.3610000000000002,"15min":3.4220000000000002},"min":{"1min":2.3279999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":4.8479999999999999,"5min":7.0590000000000002,"15min":7.0590000000000002},"last":3.6970000000000001},{"interface":"front","average":{"1min":4.0640000000000001,"5min":4.4089999999999998,"15min":4.0810000000000004},"min":{"1min":2.4750000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"max":{"1min":4.9480000000000004,"5min":26.917000000000002,"15min":26.917000000000002},"last":4.4199999999999999}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.9569999999999999,"5min":4.0259999999999998,"15min":3.9849999999999999},"min":{"1min":1.597,"5min":1.109,"15min":1.109},"max":{"1min":4.8380000000000001,"5min":12.606,"15min":12.606},"last":5.048},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.9540000000000002,"15min":3.9580000000000002},"min":{"1min":2.2370000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":5.298,"5min":13.57,"15min":13.57},"last":4.2599999999999998}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":4.3109999999999999,"5min":4.79,"15min":4.5039999999999996},"min":{"1min":3.0630000000000002,"5min":2.0379999999999998,"15min":2.0379999999999998},"max":{"1min":5.9720000000000004,"5min":27.085000000000001,"15min":27.085000000000001},"last":4.7220000000000004},{"interface":"front","average":{"1min":4.444,"5min":4.4580000000000002,"15min":4.4989999999999997},"min":{"1min":2.456,"5min":1.323,"15min":1.323},"max":{"1min":5.6719999999999997,"5min":8.798,"15min":8.798},"last":5.2629999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542354,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23124,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":21120,"kb_avail":94348716,"statfs":{"total":96636764160,"available":96613085184,"internally_reserved":0,"allocated":2043904,"data_stored":957701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21626880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.1669999999999998,"5min":2.2919999999999998,"15min":3.1619999999999999},"min":{"1min":1.5049999999999999,"5min":1.2589999999999999,"15min":1.123},"max":{"1min":3.3340000000000001,"5min":11.779999999999999,"15min":194.91},"last":1.5880000000000001},{"interface":"front","average":{"1min":2.0600000000000001,"5min":2.2170000000000001,"15min":2.246},"min":{"1min":1.3,"5min":1.0309999999999999,"15min":1.024},"max":{"1min":3.694,"5min":10.419,"15min":15.971},"last":1.8109999999999999}]},{"osd":1,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.0739999999999998,"5min":2.3580000000000001,"15min":3.0539999999999998},"min":{"1min":1.345,"5min":1.288,"15min":1.23},"max":{"1min":2.984,"5min":8.3659999999999997,"15min":206.43600000000001},"last":1.6839999999999999},{"interface":"front","average":{"1min":3.1520000000000001,"5min":3.3380000000000001,"15min":3.371},"min":{"1min":2.0329999999999999,"5min":1.4019999999999999,"15min":1.401},"max":{"1min":3.8849999999999998,"5min":15.067,"15min":36.784999999999997},"last":3.726}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.1829999999999998,"5min":2.794,"15min":3.7010000000000001},"min":{"1min":1.831,"5min":1.405,"15min":1.0469999999999999},"max":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":209.839},"last":2.8820000000000001},{"interface":"front","average":{"1min":3.3700000000000001,"5min":3.286,"15min":3.387},"min":{"1min":2.6669999999999998,"5min":1.637,"15min":1.478},"max":{"1min":4.1950000000000003,"5min":12.802,"15min":51.009},"last":4.1929999999999996}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.1059999999999999,"5min":3.2839999999999998,"15min":3.214},"min":{"1min":2.2240000000000002,"5min":1.379,"15min":1.3069999999999999},"max":{"1min":3.9700000000000002,"5min":11.237,"15min":21.593},"last":3.306},{"interface":"front","average":{"1min":3.7909999999999999,"5min":3.5830000000000002,"15min":4.1180000000000003},"min":{"1min":2.2970000000000002,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":5.1260000000000003,"5min":7.5270000000000001,"15min":210.172},"last":3.3919999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.9100000000000001,"5min":3.7189999999999999,"15min":4.3490000000000002},"min":{"1min":2.5950000000000002,"5min":1.536,"15min":1.526},"max":{"1min":4.944,"5min":5.5350000000000001,"15min":206.27799999999999},"last":4.0220000000000002},{"interface":"front","average":{"1min":4.1269999999999998,"5min":4.0380000000000003,"15min":4.0800000000000001},"min":{"1min":3.4729999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"max":{"1min":4.6630000000000003,"5min":15.467000000000001,"15min":51.384},"last":4.5990000000000002}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.6709999999999998,"5min":3.673,"15min":3.7210000000000001},"min":{"1min":2.444,"5min":1.3560000000000001,"15min":1.341},"max":{"1min":4.6299999999999999,"5min":10.332000000000001,"15min":17.254000000000001},"last":4.3840000000000003},{"interface":"front","average":{"1min":4.3380000000000001,"5min":4.3520000000000003,"15min":4.4500000000000002},"min":{"1min":3.2130000000000001,"5min":1.607,"15min":1.607},"max":{"1min":5.1369999999999996,"5min":15.326000000000001,"15min":51.531999999999996},"last":4.8970000000000002}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":4.1390000000000002,"5min":4.165,"15min":4.5060000000000002},"min":{"1min":2.6629999999999998,"5min":1.4810000000000001,"15min":1.4810000000000001},"max":{"1min":5.0730000000000004,"5min":10.488,"15min":142.23699999999999},"last":3.956},{"interface":"front","average":{"1min":4.359,"5min":4.3070000000000004,"15min":4.8550000000000004},"min":{"1min":2.8410000000000002,"5min":1.675,"15min":1.675},"max":{"1min":5.1740000000000004,"5min":12.441000000000001,"15min":210.02500000000001},"last":4.4729999999999999}]}]}]}} 2020-11-20T12:22:08.346 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:22:08.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.593+0000 7f828bfff700 1 Processor -- start 2020-11-20T12:22:08.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.593+0000 7f828bfff700 1 -- start start 2020-11-20T12:22:08.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.595+0000 7f828bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 0x7f828c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.595+0000 7f828bfff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f828c066f70 0x7f828c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.595+0000 7f828bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c0b1300 con 0x7f828c0622c0 2020-11-20T12:22:08.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.595+0000 7f828bfff700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828c067730 con 0x7f828c066f70 2020-11-20T12:22:08.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.595+0000 7f828bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0b15a0 con 0x7f828c1484d0 2020-11-20T12:22:08.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.596+0000 7f82827fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f828c0622c0 0x7f828c0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.187:45122/0 (socket says 172.21.15.187:45122) 2020-11-20T12:22:08.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.596+0000 7f82827fc700 1 -- 172.21.15.187:0/3850362625 learned_addr learned my addr 172.21.15.187:0/3850362625 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:08.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.596+0000 7f828b7fe700 1 --2- 172.21.15.187:0/3850362625 >> v2:172.21.15.187:3300/0 conn(0x7f828c066f70 0x7f828c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.596+0000 7f828affd700 1 --2- 172.21.15.187:0/3850362625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 0x7f828c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.597+0000 7f828b7fe700 1 -- 172.21.15.187:0/3850362625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 msgr2=0x7f828c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:08.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.597+0000 7f828b7fe700 1 --2- 172.21.15.187:0/3850362625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 0x7f828c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.600 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.597+0000 7f828b7fe700 1 -- 172.21.15.187:0/3850362625 >> v1:172.21.15.5:6789/0 conn(0x7f828c0622c0 legacy=0x7f828c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.601 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.597+0000 7f828b7fe700 1 -- 172.21.15.187:0/3850362625 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c0b1840 con 0x7f828c066f70 2020-11-20T12:22:08.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.597+0000 7f828b7fe700 1 --2- 172.21.15.187:0/3850362625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c0b5d50 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f8284009c90 tx=0x7f82840091d0).ready entity=mon.1 client_cookie=bdc202a3a1717f3e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:08.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828a7fc700 1 -- 172.21.15.187:0/3850362625 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828400ba70 con 0x7f828c066f70 2020-11-20T12:22:08.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828a7fc700 1 -- 172.21.15.187:0/3850362625 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828400bbd0 con 0x7f828c066f70 2020-11-20T12:22:08.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828a7fc700 1 -- 172.21.15.187:0/3850362625 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284011400 con 0x7f828c066f70 2020-11-20T12:22:08.602 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828bfff700 1 -- 172.21.15.187:0/3850362625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 msgr2=0x7f828c0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828bfff700 1 --2- 172.21.15.187:0/3850362625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c0b5d50 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f8284009c90 tx=0x7f82840091d0).stop 2020-11-20T12:22:08.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828bfff700 1 -- 172.21.15.187:0/3850362625 shutdown_connections 2020-11-20T12:22:08.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828bfff700 1 --2- 172.21.15.187:0/3850362625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c0b5d50 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828bfff700 1 --2- 172.21.15.187:0/3850362625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 0x7f828c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.598+0000 7f828bfff700 1 -- 172.21.15.187:0/3850362625 >> 172.21.15.187:0/3850362625 conn(0x7f828c0bab30 msgr2=0x7f828c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:08.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.599+0000 7f828bfff700 1 -- 172.21.15.187:0/3850362625 shutdown_connections 2020-11-20T12:22:08.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.599+0000 7f828bfff700 1 -- 172.21.15.187:0/3850362625 wait complete. 2020-11-20T12:22:08.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.601+0000 7f828bfff700 1 Processor -- start 2020-11-20T12:22:08.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.601+0000 7f828bfff700 1 -- start start 2020-11-20T12:22:08.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.602+0000 7f828bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f828c0622c0 0x7f828c146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.602+0000 7f828bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c09f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.602+0000 7f828bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 0x7f828c09b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.602+0000 7f828bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c1468d0 con 0x7f828c0622c0 2020-11-20T12:22:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.602+0000 7f828bfff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c146670 con 0x7f828c066f70 2020-11-20T12:22:08.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.602+0000 7f828bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c146be0 con 0x7f828c1484d0 2020-11-20T12:22:08.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.603+0000 7f82827fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c09f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.603+0000 7f828affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f828c0622c0 0x7f828c146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.603+0000 7f828b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 0x7f828c09b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.603+0000 7f828affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f828c0622c0 0x7f828c146180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.187:47276/0 (socket says 172.21.15.187:47276) 2020-11-20T12:22:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.603+0000 7f82827fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c09f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55014/0 (socket says 172.21.15.187:55014) 2020-11-20T12:22:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.603+0000 7f828affd700 1 -- 172.21.15.187:0/3847319825 learned_addr learned my addr 172.21.15.187:0/3847319825 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.604+0000 7f82827fc700 1 -- 172.21.15.187:0/3847319825 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 msgr2=0x7f828c09b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.604+0000 7f82827fc700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 0x7f828c09b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.604+0000 7f82827fc700 1 -- 172.21.15.187:0/3847319825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f828c0622c0 msgr2=0x7f828c146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.604+0000 7f82827fc700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f828c0622c0 0x7f828c146180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.604+0000 7f82827fc700 1 -- 172.21.15.187:0/3847319825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c146e80 con 0x7f828c066f70 2020-11-20T12:22:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.604+0000 7f82827fc700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c09f720 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7f827400c6e0 tx=0x7f8274007be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.605+0000 7f82837fe700 1 -- 172.21.15.187:0/3847319825 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82740072e0 con 0x7f828c066f70 2020-11-20T12:22:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.605+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f828c09fc10 con 0x7f828c066f70 2020-11-20T12:22:08.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.605+0000 7f82837fe700 1 -- 172.21.15.187:0/3847319825 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8274007440 con 0x7f828c066f70 2020-11-20T12:22:08.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.605+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828c09bf20 con 0x7f828c066f70 2020-11-20T12:22:08.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.605+0000 7f82837fe700 1 -- 172.21.15.187:0/3847319825 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274005800 con 0x7f828c066f70 2020-11-20T12:22:08.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.607+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828c061a20 con 0x7f828c066f70 2020-11-20T12:22:08.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.607+0000 7f82837fe700 1 -- 172.21.15.187:0/3847319825 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8274010070 con 0x7f828c066f70 2020-11-20T12:22:08.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.608+0000 7f82837fe700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8268056d20 0x7f82680591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:08.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.608+0000 7f82837fe700 1 -- 172.21.15.187:0/3847319825 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f827406f4a0 con 0x7f828c066f70 2020-11-20T12:22:08.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.608+0000 7f828affd700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8268056d20 0x7f82680591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:08.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.609+0000 7f828affd700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8268056d20 0x7f82680591e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f827c006a90 tx=0x7f827c006df0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:08.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.612+0000 7f82837fe700 1 -- 172.21.15.187:0/3847319825 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f82740447c0 con 0x7f828c066f70 2020-11-20T12:22:08.803 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.800+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f828c1488b0 con 0x7f828c066f70 2020-11-20T12:22:08.805 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.802+0000 7f82837fe700 1 -- 172.21.15.187:0/3847319825 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16859 (secure 0 0 0) 0x7f8274007d20 con 0x7f828c066f70 2020-11-20T12:22:08.805 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:08.806 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":161,"fsid":"099efcab-1650-4909-9b03-145da903b26d","created":"2020-11-20T12:04:39.066270+0000","modified":"2020-11-20T12:22:04.549389+0000","last_up_change":"2020-11-20T12:04:47.548535+0000","last_in_change":"2020-11-20T12:04:47.548535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:43.604166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:51.942528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:21:20.315283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2020-11-20T12:21:21.644244+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":160,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"59e4fe7b-bbb5-405f-900c-7b2b145087c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":19903},{"type":"v1","addr":"172.21.15.187:6803","nonce":19903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":19903},{"type":"v1","addr":"172.21.15.187:6805","nonce":19903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":19903},{"type":"v1","addr":"172.21.15.187:6809","nonce":19903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":19903},{"type":"v1","addr":"172.21.15.187:6807","nonce":19903}]},"public_addr":"172.21.15.187:6803/19903","cluster_addr":"172.21.15.187:6805/19903","heartbeat_back_addr":"172.21.15.187:6809/19903","heartbeat_front_addr":"172.21.15.187:6807/19903","state":["exists","up"]},{"osd":1,"uuid":"3051fdba-a9b2-4e83-9bc7-acfd0e1036fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":19920},{"type":"v1","addr":"172.21.15.187:6819","nonce":19920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":19920},{"type":"v1","addr":"172.21.15.187:6821","nonce":19920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":19920},{"type":"v1","addr":"172.21.15.187:6825","nonce":19920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":19920},{"type":"v1","addr":"172.21.15.187:6823","nonce":19920}]},"public_addr":"172.21.15.187:6819/19920","cluster_addr":"172.21.15.187:6821/19920","heartbeat_back_addr":"172.21.15.187:6825/19920","heartbeat_front_addr":"172.21.15.187:6823/19920","state":["exists","up"]},{"osd":2,"uuid":"588f2d2b-cca4-4d3c-b599-ea8d32a04b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":19904},{"type":"v1","addr":"172.21.15.187:6811","nonce":19904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":19904},{"type":"v1","addr":"172.21.15.187:6813","nonce":19904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":19904},{"type":"v1","addr":"172.21.15.187:6817","nonce":19904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":19904},{"type":"v1","addr":"172.21.15.187:6815","nonce":19904}]},"public_addr":"172.21.15.187:6811/19904","cluster_addr":"172.21.15.187:6813/19904","heartbeat_back_addr":"172.21.15.187:6817/19904","heartbeat_front_addr":"172.21.15.187:6815/19904","state":["exists","up"]},{"osd":3,"uuid":"e041ab46-c4c7-48ae-8144-caac199f94f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":19935},{"type":"v1","addr":"172.21.15.187:6827","nonce":19935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":19935},{"type":"v1","addr":"172.21.15.187:6829","nonce":19935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":19935},{"type":"v1","addr":"172.21.15.187:6833","nonce":19935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":19935},{"type":"v1","addr":"172.21.15.187:6831","nonce":19935}]},"public_addr":"172.21.15.187:6827/19935","cluster_addr":"172.21.15.187:6829/19935","heartbeat_back_addr":"172.21.15.187:6833/19935","heartbeat_front_addr":"172.21.15.187:6831/19935","state":["exists","up"]},{"osd":4,"uuid":"6adcc4eb-3632-47cf-a500-e134866924da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":20156},{"type":"v1","addr":"172.21.15.5:6810","nonce":20156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":20156},{"type":"v1","addr":"172.21.15.5:6815","nonce":20156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":20156},{"type":"v1","addr":"172.21.15.5:6829","nonce":20156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6819","nonce":20156},{"type":"v1","addr":"172.21.15.5:6823","nonce":20156}]},"public_addr":"172.21.15.5:6810/20156","cluster_addr":"172.21.15.5:6815/20156","heartbeat_back_addr":"172.21.15.5:6829/20156","heartbeat_front_addr":"172.21.15.5:6823/20156","state":["exists","up"]},{"osd":5,"uuid":"77c9ee1b-3f40-4a15-84e0-8b89b96e5237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":20155},{"type":"v1","addr":"172.21.15.5:6811","nonce":20155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":20155},{"type":"v1","addr":"172.21.15.5:6816","nonce":20155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6825","nonce":20155},{"type":"v1","addr":"172.21.15.5:6828","nonce":20155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":20155},{"type":"v1","addr":"172.21.15.5:6821","nonce":20155}]},"public_addr":"172.21.15.5:6811/20155","cluster_addr":"172.21.15.5:6816/20155","heartbeat_back_addr":"172.21.15.5:6828/20155","heartbeat_front_addr":"172.21.15.5:6821/20155","state":["exists","up"]},{"osd":6,"uuid":"d79cbb7e-2a05-4688-bd3d-1289b4b231bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":20157},{"type":"v1","addr":"172.21.15.5:6801","nonce":20157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":20157},{"type":"v1","addr":"172.21.15.5:6803","nonce":20157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":20157},{"type":"v1","addr":"172.21.15.5:6807","nonce":20157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":20157},{"type":"v1","addr":"172.21.15.5:6805","nonce":20157}]},"public_addr":"172.21.15.5:6801/20157","cluster_addr":"172.21.15.5:6803/20157","heartbeat_back_addr":"172.21.15.5:6807/20157","heartbeat_front_addr":"172.21.15.5:6805/20157","state":["exists","up"]},{"osd":7,"uuid":"65c46353-4546-48a6-813e-3968184574c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":20158},{"type":"v1","addr":"172.21.15.5:6817","nonce":20158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":20158},{"type":"v1","addr":"172.21.15.5:6822","nonce":20158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":20158},{"type":"v1","addr":"172.21.15.5:6831","nonce":20158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":20158},{"type":"v1","addr":"172.21.15.5:6827","nonce":20158}]},"public_addr":"172.21.15.5:6817/20158","cluster_addr":"172.21.15.5:6822/20158","heartbeat_back_addr":"172.21.15.5:6831/20158","heartbeat_front_addr":"172.21.15.5:6827/20158","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.013153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.049795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.110394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.090082+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.166666+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.191960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.168202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:46.128082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":32,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:22:08.807 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8268056d20 msgr2=0x7f82680591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8268056d20 0x7f82680591e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f827c006a90 tx=0x7f827c006df0).stop 2020-11-20T12:22:08.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 msgr2=0x7f828c09f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:08.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c09f720 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7f827400c6e0 tx=0x7f8274007be0).stop 2020-11-20T12:22:08.808 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 shutdown_connections 2020-11-20T12:22:08.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8268056d20 0x7f82680591e0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f828c0622c0 0x7f828c146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f828c1484d0 0x7f828c09b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 --2- 172.21.15.187:0/3847319825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f828c066f70 0x7f828c09f720 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:08.809 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 >> 172.21.15.187:0/3847319825 conn(0x7f828c0bab30 msgr2=0x7f828c143de0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:08.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 shutdown_connections 2020-11-20T12:22:08.810 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:08.805+0000 7f828bfff700 1 -- 172.21.15.187:0/3847319825 wait complete. 2020-11-20T12:22:09.831 INFO:tasks.ceph:Scrubbing osd.0 2020-11-20T12:22:09.832 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2020-11-20T12:22:10.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.084+0000 7f8699485700 1 Processor -- start 2020-11-20T12:22:10.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.084+0000 7f8699485700 1 -- start start 2020-11-20T12:22:10.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.085+0000 7f8699485700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86940b6fd0 0x7f8694066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.085+0000 7f8699485700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8694146010 con 0x7f8694067300 2020-11-20T12:22:10.088 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.085+0000 7f8699485700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86941462b0 con 0x7f86940b73b0 2020-11-20T12:22:10.089 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.085+0000 7f8699485700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8694146550 con 0x7f86940b6fd0 2020-11-20T12:22:10.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.086+0000 7f86927fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f86940b73b0 0x7f8694066bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51626/0 (socket says 172.21.15.187:51626) 2020-11-20T12:22:10.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.086+0000 7f86927fc700 1 -- 172.21.15.187:0/1273682035 learned_addr learned my addr 172.21.15.187:0/1273682035 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:10.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.086+0000 7f8692ffd700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86940b6fd0 0x7f8694066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.091 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885145285 0 0) 0x7f86941462b0 con 0x7f86940b73b0 2020-11-20T12:22:10.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8680003240 con 0x7f86940b73b0 2020-11-20T12:22:10.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2007658795 0 0) 0x7f8680003240 con 0x7f86940b73b0 2020-11-20T12:22:10.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86940b6fd0 msgr2=0x7f8694066700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:10.092 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86940b6fd0 0x7f8694066700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 >> v1:172.21.15.5:6789/0 conn(0x7f8694067300 legacy=0x7f8694144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86940b6650 con 0x7f86940b73b0 2020-11-20T12:22:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f86880014e0 con 0x7f86940b73b0 2020-11-20T12:22:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 >> v1:172.21.15.187:6789/0 conn(0x7f86940b73b0 legacy=0x7f8694066bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86940b6fd0 0x7f8680003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8680003c40 0x7f86800044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.087+0000 7f8691ffb700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86800049d0 0x7f8680004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.094 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8694146550 con 0x7f86800049d0 2020-11-20T12:22:10.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86800041d0 con 0x7f8680003c40 2020-11-20T12:22:10.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868000b730 con 0x7f86940b6fd0 2020-11-20T12:22:10.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f86927fc700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8680003c40 0x7f86800044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.095 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f8692ffd700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86940b6fd0 0x7f8680003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f86927fc700 1 -- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86940b6fd0 msgr2=0x7f8680003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f86927fc700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86940b6fd0 0x7f8680003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f86927fc700 1 -- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86800049d0 msgr2=0x7f8680004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f86927fc700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86800049d0 0x7f8680004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:10.096 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.088+0000 7f86927fc700 1 -- 172.21.15.187:0/1273682035 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680003240 con 0x7f8680003c40 2020-11-20T12:22:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.089+0000 7f86927fc700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8680003c40 0x7f86800044e0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f869405d390 tx=0x7f86880077d0).ready entity=mon.1 client_cookie=c7539ab9665bc81 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:10.100 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.089+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868800ce10 con 0x7f8680003c40 2020-11-20T12:22:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.089+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86880072e0 con 0x7f8680003c40 2020-11-20T12:22:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.089+0000 7f8691ffb700 1 -- 172.21.15.187:0/1273682035 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688005820 con 0x7f8680003c40 2020-11-20T12:22:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.089+0000 7f8699485700 1 -- 172.21.15.187:0/1273682035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8680003c40 msgr2=0x7f86800044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.089+0000 7f8699485700 1 --2- 172.21.15.187:0/1273682035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8680003c40 0x7f86800044e0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f869405d390 tx=0x7f86880077d0).stop 2020-11-20T12:22:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.089+0000 7f86937fe700 1 -- 172.21.15.187:0/1273682035 reap_dead start 2020-11-20T12:22:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.090+0000 7f8699485700 1 -- 172.21.15.187:0/1273682035 shutdown_connections 2020-11-20T12:22:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.090+0000 7f8699485700 1 -- 172.21.15.187:0/1273682035 >> 172.21.15.187:0/1273682035 conn(0x7f86940bacf0 msgr2=0x7f86940bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.090+0000 7f8699485700 1 -- 172.21.15.187:0/1273682035 shutdown_connections 2020-11-20T12:22:10.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.090+0000 7f8699485700 1 -- 172.21.15.187:0/1273682035 wait complete. 2020-11-20T12:22:10.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.091+0000 7f8699485700 1 Processor -- start 2020-11-20T12:22:10.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.091+0000 7f8699485700 1 -- start start 2020-11-20T12:22:10.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.092+0000 7f8699485700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8694145710 0x7f869408f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.092+0000 7f8699485700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8694145af0 0x7f8694158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.092+0000 7f8699485700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8694141ff0 0x7f8694142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.092+0000 7f8699485700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8694142960 con 0x7f8694141ff0 2020-11-20T12:22:10.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.092+0000 7f8699485700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869409f8e0 con 0x7f8694145af0 2020-11-20T12:22:10.104 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.092+0000 7f8699485700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869409d180 con 0x7f8694145710 2020-11-20T12:22:10.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8694145af0 0x7f8694158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8694145af0 0x7f8694158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55034/0 (socket says 172.21.15.187:55034) 2020-11-20T12:22:10.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 -- 172.21.15.187:0/1448682311 learned_addr learned my addr 172.21.15.187:0/1448682311 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:10.105 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f8692ffd700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8694145710 0x7f869408f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86937fe700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8694141ff0 0x7f8694142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 -- 172.21.15.187:0/1448682311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8694145710 msgr2=0x7f869408f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8694145710 0x7f869408f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 -- 172.21.15.187:0/1448682311 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8694141ff0 msgr2=0x7f8694142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8694141ff0 0x7f8694142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 -- 172.21.15.187:0/1448682311 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869409d420 con 0x7f8694145af0 2020-11-20T12:22:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f8692ffd700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8694145710 0x7f869408f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:22:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.093+0000 7f86927fc700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8694145af0 0x7f8694158450 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f8680004360 tx=0x7f8688017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:10.107 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.094+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688005240 con 0x7f8694145af0 2020-11-20T12:22:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.094+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86880053a0 con 0x7f8694145af0 2020-11-20T12:22:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.094+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688005740 con 0x7f8694145af0 2020-11-20T12:22:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.094+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869409d6c0 con 0x7f8694145af0 2020-11-20T12:22:10.108 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.094+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869409ccd0 con 0x7f8694145af0 2020-11-20T12:22:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.095+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f869409d8c0 con 0x7f8694145af0 2020-11-20T12:22:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.096+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8688021020 con 0x7f8694145af0 2020-11-20T12:22:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.097+0000 7f867affd700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8670056d20 0x7f86700591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.097+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8688071960 con 0x7f8694145af0 2020-11-20T12:22:10.109 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.097+0000 7f867affd700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f867005e2d0 0x7f86700606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.097+0000 7f8692ffd700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8670056d20 0x7f86700591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.097+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8670060d70 con 0x7f867005e2d0 2020-11-20T12:22:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.097+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=161) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8688076220 con 0x7f8694145af0 2020-11-20T12:22:10.110 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.097+0000 7f86937fe700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f867005e2d0 0x7f86700606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.097+0000 7f8692ffd700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8670056d20 0x7f86700591e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7f86840090e0 tx=0x7f8684008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:10.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.098+0000 7f86937fe700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f867005e2d0 0x7f86700606f0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:10.111 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.099+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 <== osd.0 v2:172.21.15.187:6802/19903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f8670060d70 con 0x7f867005e2d0 2020-11-20T12:22:10.118 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.116+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 --> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f869409a0f0 con 0x7f867005e2d0 2020-11-20T12:22:10.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f867affd700 1 -- 172.21.15.187:0/1448682311 <== osd.0 v2:172.21.15.187:6802/19903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f869409a0f0 con 0x7f867005e2d0 2020-11-20T12:22:10.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f867005e2d0 msgr2=0x7f86700606f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.120 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f8699485700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6802/19903,v1:172.21.15.187:6803/19903] conn(0x7f867005e2d0 0x7f86700606f0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8670056d20 msgr2=0x7f86700591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f8699485700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8670056d20 0x7f86700591e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7f86840090e0 tx=0x7f8684008040).stop 2020-11-20T12:22:10.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8694145af0 msgr2=0x7f8694158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.121 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f8699485700 1 --2- 172.21.15.187:0/1448682311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8694145af0 0x7f8694158450 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f8680004360 tx=0x7f8688017000).stop 2020-11-20T12:22:10.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f86937fe700 1 -- 172.21.15.187:0/1448682311 reap_dead start 2020-11-20T12:22:10.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 shutdown_connections 2020-11-20T12:22:10.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.117+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 >> 172.21.15.187:0/1448682311 conn(0x7f86940bacf0 msgr2=0x7f86940b9770 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:10.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.118+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 shutdown_connections 2020-11-20T12:22:10.122 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.118+0000 7f8699485700 1 -- 172.21.15.187:0/1448682311 wait complete. 2020-11-20T12:22:10.123 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T12:22:10.123 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:10.140 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2020-11-20T12:22:10.387 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.383+0000 7f89ab59e700 1 Processor -- start 2020-11-20T12:22:10.388 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.384+0000 7f89ab59e700 1 -- start start 2020-11-20T12:22:10.403 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.385+0000 7f89ab59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.385+0000 7f89ab59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89ac0622c0 0x7f89ac0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.404 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.385+0000 7f89ab59e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ac0b1300 con 0x7f89ac066f70 2020-11-20T12:22:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.385+0000 7f89ab59e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac067730 con 0x7f89ac0622c0 2020-11-20T12:22:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.385+0000 7f89ab59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac0b15a0 con 0x7f89ac1484d0 2020-11-20T12:22:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.385+0000 7f89a9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89ac0622c0 0x7f89ac0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.407 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.385+0000 7f89a9d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89ac0622c0 0x7f89ac0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55044/0 (socket says 172.21.15.187:55044) 2020-11-20T12:22:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.385+0000 7f89a9d9b700 1 -- 172.21.15.187:0/2377611241 learned_addr learned my addr 172.21.15.187:0/2377611241 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.386+0000 7f89aa59c700 1 --2- 172.21.15.187:0/2377611241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.386+0000 7f89a9d9b700 1 -- 172.21.15.187:0/2377611241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 msgr2=0x7f89ac061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:10.408 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.386+0000 7f89a9d9b700 1 --2- 172.21.15.187:0/2377611241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.386+0000 7f89a9d9b700 1 -- 172.21.15.187:0/2377611241 >> v1:172.21.15.187:6789/0 conn(0x7f89ac066f70 legacy=0x7f89ac0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.386+0000 7f89a9d9b700 1 -- 172.21.15.187:0/2377611241 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac0b1840 con 0x7f89ac0622c0 2020-11-20T12:22:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.386+0000 7f89aa59c700 1 --2- 172.21.15.187:0/2377611241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:10.409 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.386+0000 7f89a9d9b700 1 --2- 172.21.15.187:0/2377611241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac0622c0 0x7f89ac0b5640 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f8998006fd0 tx=0x7f8998006eb0).ready entity=mon.1 client_cookie=5bc9aa6a3a812709 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:10.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89a959a700 1 -- 172.21.15.187:0/2377611241 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89980098d0 con 0x7f89ac0622c0 2020-11-20T12:22:10.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89a959a700 1 -- 172.21.15.187:0/2377611241 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8998009a30 con 0x7f89ac0622c0 2020-11-20T12:22:10.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89a959a700 1 -- 172.21.15.187:0/2377611241 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998009db0 con 0x7f89ac0622c0 2020-11-20T12:22:10.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89ab59e700 1 -- 172.21.15.187:0/2377611241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac0622c0 msgr2=0x7f89ac0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.410 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89ab59e700 1 --2- 172.21.15.187:0/2377611241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac0622c0 0x7f89ac0b5640 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f8998006fd0 tx=0x7f8998006eb0).stop 2020-11-20T12:22:10.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89ab59e700 1 -- 172.21.15.187:0/2377611241 shutdown_connections 2020-11-20T12:22:10.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89ab59e700 1 --2- 172.21.15.187:0/2377611241 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89ab59e700 1 --2- 172.21.15.187:0/2377611241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac0622c0 0x7f89ac0b5640 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89ab59e700 1 -- 172.21.15.187:0/2377611241 >> 172.21.15.187:0/2377611241 conn(0x7f89ac0bab30 msgr2=0x7f89ac0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:10.411 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89ab59e700 1 -- 172.21.15.187:0/2377611241 shutdown_connections 2020-11-20T12:22:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.387+0000 7f89ab59e700 1 -- 172.21.15.187:0/2377611241 wait complete. 2020-11-20T12:22:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.389+0000 7f89ab59e700 1 Processor -- start 2020-11-20T12:22:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.390+0000 7f89ab59e700 1 -- start start 2020-11-20T12:22:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.391+0000 7f89ab59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89ac0622c0 0x7f89ac09b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.412 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.391+0000 7f89ab59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac066f70 0x7f89ac140b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.391+0000 7f89ab59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac0b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.414 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.391+0000 7f89ab59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac09c330 con 0x7f89ac0622c0 2020-11-20T12:22:10.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.391+0000 7f89ab59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1455d0 con 0x7f89ac066f70 2020-11-20T12:22:10.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.391+0000 7f89ab59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac09c590 con 0x7f89ac1484d0 2020-11-20T12:22:10.415 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.391+0000 7f89a9d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac066f70 0x7f89ac140b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89a9d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac066f70 0x7f89ac140b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55050/0 (socket says 172.21.15.187:55050) 2020-11-20T12:22:10.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89a9d9b700 1 -- 172.21.15.187:0/594221320 learned_addr learned my addr 172.21.15.187:0/594221320 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:10.416 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89aa59c700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89ac0622c0 0x7f89ac09b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89aad9d700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac0b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89a9d9b700 1 -- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 msgr2=0x7f89ac0b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.417 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89a9d9b700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac0b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89a9d9b700 1 -- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89ac0622c0 msgr2=0x7f89ac09b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:10.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89a9d9b700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89ac0622c0 0x7f89ac09b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:10.418 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89a9d9b700 1 -- 172.21.15.187:0/594221320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac145250 con 0x7f89ac066f70 2020-11-20T12:22:10.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89aa59c700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89ac0622c0 0x7f89ac09b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:22:10.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.392+0000 7f89a9d9b700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac066f70 0x7f89ac140b60 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f8998010000 tx=0x7f8998015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:10.419 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.393+0000 7f89a27fc700 1 -- 172.21.15.187:0/594221320 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998011e50 con 0x7f89ac066f70 2020-11-20T12:22:10.420 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.393+0000 7f89a27fc700 1 -- 172.21.15.187:0/594221320 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899801e460 con 0x7f89ac066f70 2020-11-20T12:22:10.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.393+0000 7f89a27fc700 1 -- 172.21.15.187:0/594221320 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899801e7e0 con 0x7f89ac066f70 2020-11-20T12:22:10.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.393+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac141050 con 0x7f89ac066f70 2020-11-20T12:22:10.421 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.393+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac0b7f20 con 0x7f89ac066f70 2020-11-20T12:22:10.422 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.394+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac1453e0 con 0x7f89ac066f70 2020-11-20T12:22:10.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.395+0000 7f89a27fc700 1 -- 172.21.15.187:0/594221320 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8998025030 con 0x7f89ac066f70 2020-11-20T12:22:10.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.395+0000 7f89a27fc700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8980056d20 0x7f89800591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:10.423 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.396+0000 7f89a27fc700 1 -- 172.21.15.187:0/594221320 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8998071830 con 0x7f89ac066f70 2020-11-20T12:22:10.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.396+0000 7f89aa59c700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8980056d20 0x7f89800591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:10.424 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.396+0000 7f89aa59c700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8980056d20 0x7f89800591e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f899c006fd0 tx=0x7f899c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:10.425 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.399+0000 7f89a27fc700 1 -- 172.21.15.187:0/594221320 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8998046cb0 con 0x7f89ac066f70 2020-11-20T12:22:10.586 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.584+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f89ac1411e0 con 0x7f8980056d20 2020-11-20T12:22:11.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.585+0000 7f89a27fc700 1 -- 172.21.15.187:0/594221320 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f899c00c030 con 0x7f8980056d20 2020-11-20T12:22:11.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8980056d20 msgr2=0x7f89800591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8980056d20 0x7f89800591e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f899c006fd0 tx=0x7f899c008040).stop 2020-11-20T12:22:11.232 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac066f70 msgr2=0x7f89ac140b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.233 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac066f70 0x7f89ac140b60 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f8998010000 tx=0x7f8998015000).stop 2020-11-20T12:22:11.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 shutdown_connections 2020-11-20T12:22:11.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8980056d20 0x7f89800591e0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.234 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89ac0622c0 0x7f89ac09b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89ac1484d0 0x7f89ac0b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 --2- 172.21.15.187:0/594221320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89ac066f70 0x7f89ac140b60 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 >> 172.21.15.187:0/594221320 conn(0x7f89ac0bab30 msgr2=0x7f89ac142c20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:11.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 shutdown_connections 2020-11-20T12:22:11.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:10.587+0000 7f89ab59e700 1 -- 172.21.15.187:0/594221320 wait complete. 2020-11-20T12:22:11.236 INFO:teuthology.orchestra.run.smithi187.stderr:instructed osd(s) 0 to deep-scrub 2020-11-20T12:22:11.237 INFO:tasks.ceph:Scrubbing osd.1 2020-11-20T12:22:11.238 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2020-11-20T12:22:11.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.506+0000 7f014ffff700 1 Processor -- start 2020-11-20T12:22:11.509 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.506+0000 7f014ffff700 1 -- start start 2020-11-20T12:22:11.513 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.508+0000 7f014ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f01500bfbe0 0x7f01501480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.508+0000 7f014ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015014d110 con 0x7f015009f560 2020-11-20T12:22:11.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.508+0000 7f014ffff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015014cbd0 con 0x7f01500bf800 2020-11-20T12:22:11.514 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.508+0000 7f014ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015014c9d0 con 0x7f01500bfbe0 2020-11-20T12:22:11.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.508+0000 7f014effd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f01500bf800 0x7f015014c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51662/0 (socket says 172.21.15.187:51662) 2020-11-20T12:22:11.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.508+0000 7f014effd700 1 -- 172.21.15.187:0/1737682968 learned_addr learned my addr 172.21.15.187:0/1737682968 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:11.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272662861 0 0) 0x7f015014cbd0 con 0x7f01500bf800 2020-11-20T12:22:11.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0130003240 con 0x7f01500bf800 2020-11-20T12:22:11.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f01467fc700 1 --2- 172.21.15.187:0/1737682968 >> v2:172.21.15.5:3300/0 conn(0x7f01500bfbe0 0x7f01501480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 623433448 0 0) 0x7f0130003240 con 0x7f01500bf800 2020-11-20T12:22:11.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 >> v2:172.21.15.5:3300/0 conn(0x7f01500bfbe0 msgr2=0x7f01501480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:11.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 --2- 172.21.15.187:0/1737682968 >> v2:172.21.15.5:3300/0 conn(0x7f01500bfbe0 0x7f01501480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 >> v1:172.21.15.5:6789/0 conn(0x7f015009f560 legacy=0x7f01500a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015014ce70 con 0x7f01500bf800 2020-11-20T12:22:11.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f0140001520 con 0x7f01500bf800 2020-11-20T12:22:11.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.509+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 >> v1:172.21.15.187:6789/0 conn(0x7f01500bf800 legacy=0x7f015014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f014e7fc700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0130003730 0x7f0130003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.526 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f014e7fc700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01300044e0 0x7f0130004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f014e7fc700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0130004e50 0x7f013000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015014c9d0 con 0x7f0130003730 2020-11-20T12:22:11.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0130004260 con 0x7f01300044e0 2020-11-20T12:22:11.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013000bc20 con 0x7f0130004e50 2020-11-20T12:22:11.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f01467fc700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01300044e0 0x7f0130004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f014effd700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0130003730 0x7f0130003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f01467fc700 1 -- 172.21.15.187:0/1737682968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0130004e50 msgr2=0x7f013000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:11.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f014f7fe700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0130004e50 0x7f013000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.510+0000 7f01467fc700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0130004e50 0x7f013000b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.511+0000 7f01467fc700 1 -- 172.21.15.187:0/1737682968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0130003730 msgr2=0x7f0130003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:11.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.511+0000 7f01467fc700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0130003730 0x7f0130003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.511+0000 7f01467fc700 1 -- 172.21.15.187:0/1737682968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0130003240 con 0x7f01300044e0 2020-11-20T12:22:11.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.511+0000 7f014f7fe700 1 -- 172.21.15.187:0/1737682968 reap_dead start 2020-11-20T12:22:11.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.511+0000 7f01467fc700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01300044e0 0x7f0130004960 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f013800bc00 tx=0x7f0138006df0).ready entity=mon.1 client_cookie=244c9f11338c5672 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:11.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.511+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013800b310 con 0x7f01300044e0 2020-11-20T12:22:11.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.511+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013800eb40 con 0x7f01300044e0 2020-11-20T12:22:11.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.511+0000 7f014e7fc700 1 -- 172.21.15.187:0/1737682968 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013800eec0 con 0x7f01300044e0 2020-11-20T12:22:11.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.512+0000 7f014ffff700 1 -- 172.21.15.187:0/1737682968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01300044e0 msgr2=0x7f0130004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.512+0000 7f014ffff700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01300044e0 0x7f0130004960 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f013800bc00 tx=0x7f0138006df0).stop 2020-11-20T12:22:11.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.512+0000 7f014ffff700 1 -- 172.21.15.187:0/1737682968 shutdown_connections 2020-11-20T12:22:11.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.512+0000 7f014ffff700 1 --2- 172.21.15.187:0/1737682968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01300044e0 0x7f0130004960 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.512+0000 7f014ffff700 1 -- 172.21.15.187:0/1737682968 >> 172.21.15.187:0/1737682968 conn(0x7f0150068370 msgr2=0x7f01500bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:11.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.512+0000 7f014ffff700 1 -- 172.21.15.187:0/1737682968 shutdown_connections 2020-11-20T12:22:11.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.512+0000 7f014ffff700 1 -- 172.21.15.187:0/1737682968 wait complete. 2020-11-20T12:22:11.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.513+0000 7f014ffff700 1 Processor -- start 2020-11-20T12:22:11.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.514+0000 7f014ffff700 1 -- start start 2020-11-20T12:22:11.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.514+0000 7f014ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01300044e0 0x7f015015c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.514+0000 7f014ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01500ba680 0x7f0150142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.514+0000 7f014ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f015013ca70 0x7f015013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.514+0000 7f014ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500b5d60 con 0x7f01300044e0 2020-11-20T12:22:11.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.514+0000 7f014ffff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500b4090 con 0x7f01500ba680 2020-11-20T12:22:11.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.514+0000 7f014ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01500b5f50 con 0x7f015013ca70 2020-11-20T12:22:11.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f01467fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01500ba680 0x7f0150142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f01467fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01500ba680 0x7f0150142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55070/0 (socket says 172.21.15.187:55070) 2020-11-20T12:22:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f01467fc700 1 -- 172.21.15.187:0/708641915 learned_addr learned my addr 172.21.15.187:0/708641915 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f014effd700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01300044e0 0x7f015015c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f014f7fe700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f015013ca70 0x7f015013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f01467fc700 1 -- 172.21.15.187:0/708641915 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f015013ca70 msgr2=0x7f015013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f01467fc700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f015013ca70 0x7f015013cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f01467fc700 1 -- 172.21.15.187:0/708641915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01300044e0 msgr2=0x7f015015c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f01467fc700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01300044e0 0x7f015015c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.515+0000 7f01467fc700 1 -- 172.21.15.187:0/708641915 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01500b61f0 con 0x7f01500ba680 2020-11-20T12:22:11.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.516+0000 7f01467fc700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01500ba680 0x7f0150142b20 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7f0138012dc0 tx=0x7f0138019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:11.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.516+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013801d620 con 0x7f01500ba680 2020-11-20T12:22:11.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.516+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138021070 con 0x7f01500ba680 2020-11-20T12:22:11.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.516+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0150143010 con 0x7f01500ba680 2020-11-20T12:22:11.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.516+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013801d780 con 0x7f01500ba680 2020-11-20T12:22:11.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.516+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01500b6850 con 0x7f01500ba680 2020-11-20T12:22:11.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.517+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0150080050 con 0x7f01500ba680 2020-11-20T12:22:11.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.518+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f013802d070 con 0x7f01500ba680 2020-11-20T12:22:11.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.518+0000 7f01477fe700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f012c056d20 0x7f012c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.518+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0138071fb0 con 0x7f01500ba680 2020-11-20T12:22:11.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.519+0000 7f01477fe700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f012c05e2d0 0x7f012c0606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.519+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f012c060d70 con 0x7f012c05e2d0 2020-11-20T12:22:11.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.519+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=161) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0138074620 con 0x7f01500ba680 2020-11-20T12:22:11.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.519+0000 7f014effd700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f012c056d20 0x7f012c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.519+0000 7f014f7fe700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f012c05e2d0 0x7f012c0606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.520+0000 7f014effd700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f012c056d20 0x7f012c0591e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f0140001e90 tx=0x7f0140009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.520+0000 7f014f7fe700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f012c05e2d0 0x7f012c0606f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.521+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 <== osd.1 v2:172.21.15.187:6818/19920 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f012c060d70 con 0x7f012c05e2d0 2020-11-20T12:22:11.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.537+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 --> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f015015cc00 con 0x7f012c05e2d0 2020-11-20T12:22:11.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.538+0000 7f01477fe700 1 -- 172.21.15.187:0/708641915 <== osd.1 v2:172.21.15.187:6818/19920 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f015015cc00 con 0x7f012c05e2d0 2020-11-20T12:22:11.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.538+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f012c05e2d0 msgr2=0x7f012c0606f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.538+0000 7f014ffff700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6818/19920,v1:172.21.15.187:6819/19920] conn(0x7f012c05e2d0 0x7f012c0606f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f012c056d20 msgr2=0x7f012c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014ffff700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f012c056d20 0x7f012c0591e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f0140001e90 tx=0x7f0140009040).stop 2020-11-20T12:22:11.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01500ba680 msgr2=0x7f0150142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014ffff700 1 --2- 172.21.15.187:0/708641915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01500ba680 0x7f0150142b20 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7f0138012dc0 tx=0x7f0138019000).stop 2020-11-20T12:22:11.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014f7fe700 1 -- 172.21.15.187:0/708641915 reap_dead start 2020-11-20T12:22:11.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 shutdown_connections 2020-11-20T12:22:11.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 >> 172.21.15.187:0/708641915 conn(0x7f0150068370 msgr2=0x7f01500bef10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:11.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 shutdown_connections 2020-11-20T12:22:11.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.539+0000 7f014ffff700 1 -- 172.21.15.187:0/708641915 wait complete. 2020-11-20T12:22:11.552 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:11.552 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T12:22:11.553 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:11.564 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2020-11-20T12:22:11.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.810+0000 7fa30d30c700 1 Processor -- start 2020-11-20T12:22:11.814 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.811+0000 7fa30d30c700 1 -- start start 2020-11-20T12:22:11.823 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.812+0000 7fa30d30c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa308065f90 0x7fa308065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.812+0000 7fa30d30c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa30809ca90 0x7fa30809cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.812+0000 7fa30d30c700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3080b13e0 con 0x7fa308065190 2020-11-20T12:22:11.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.812+0000 7fa30d30c700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30814ab30 con 0x7fa308065f90 2020-11-20T12:22:11.824 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.812+0000 7fa30d30c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa308062c10 con 0x7fa30809ca90 2020-11-20T12:22:11.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.812+0000 7fa2ff7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa308065f90 0x7fa308065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.813+0000 7fa2ff7fe700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa308065f90 0x7fa308065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55078/0 (socket says 172.21.15.187:55078) 2020-11-20T12:22:11.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.813+0000 7fa2ff7fe700 1 -- 172.21.15.187:0/2663724892 learned_addr learned my addr 172.21.15.187:0/2663724892 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:11.825 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.813+0000 7fa30cb0b700 1 --2- 172.21.15.187:0/2663724892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa30809ca90 0x7fa30809cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.813+0000 7fa2ff7fe700 1 -- 172.21.15.187:0/2663724892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa30809ca90 msgr2=0x7fa30809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:11.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.813+0000 7fa2ff7fe700 1 --2- 172.21.15.187:0/2663724892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa30809ca90 0x7fa30809cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.813+0000 7fa2ff7fe700 1 -- 172.21.15.187:0/2663724892 >> v1:172.21.15.187:6789/0 conn(0x7fa308065190 legacy=0x7fa308143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:11.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.813+0000 7fa2ff7fe700 1 -- 172.21.15.187:0/2663724892 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080b1640 con 0x7fa308065f90 2020-11-20T12:22:11.826 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.814+0000 7fa2ff7fe700 1 --2- 172.21.15.187:0/2663724892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065f90 0x7fa308065570 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fa2f8006fd0 tx=0x7fa2f8006eb0).ready entity=mon.1 client_cookie=be5a647dc79056de server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:11.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.814+0000 7fa3077fe700 1 -- 172.21.15.187:0/2663724892 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8009960 con 0x7fa308065f90 2020-11-20T12:22:11.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.814+0000 7fa3077fe700 1 -- 172.21.15.187:0/2663724892 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f8009ac0 con 0x7fa308065f90 2020-11-20T12:22:11.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.814+0000 7fa3077fe700 1 -- 172.21.15.187:0/2663724892 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8009e40 con 0x7fa308065f90 2020-11-20T12:22:11.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.814+0000 7fa30d30c700 1 -- 172.21.15.187:0/2663724892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065f90 msgr2=0x7fa308065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.827 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.814+0000 7fa30d30c700 1 --2- 172.21.15.187:0/2663724892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065f90 0x7fa308065570 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fa2f8006fd0 tx=0x7fa2f8006eb0).stop 2020-11-20T12:22:11.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.815+0000 7fa30d30c700 1 -- 172.21.15.187:0/2663724892 shutdown_connections 2020-11-20T12:22:11.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.815+0000 7fa30d30c700 1 --2- 172.21.15.187:0/2663724892 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa30809ca90 0x7fa30809cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.815+0000 7fa30d30c700 1 --2- 172.21.15.187:0/2663724892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065f90 0x7fa308065570 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:11.828 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.815+0000 7fa30d30c700 1 -- 172.21.15.187:0/2663724892 >> 172.21.15.187:0/2663724892 conn(0x7fa3080bab30 msgr2=0x7fa3080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:11.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.815+0000 7fa30d30c700 1 -- 172.21.15.187:0/2663724892 shutdown_connections 2020-11-20T12:22:11.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.815+0000 7fa30d30c700 1 -- 172.21.15.187:0/2663724892 wait complete. 2020-11-20T12:22:11.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.817+0000 7fa30d30c700 1 Processor -- start 2020-11-20T12:22:11.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.817+0000 7fa30d30c700 1 -- start start 2020-11-20T12:22:11.829 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.818+0000 7fa30d30c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065190 0x7fa308151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.818+0000 7fa30d30c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa308065f90 0x7fa308143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.818+0000 7fa30d30c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa30809ca90 0x7fa308143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.818+0000 7fa30d30c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080a0240 con 0x7fa30809ca90 2020-11-20T12:22:11.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.818+0000 7fa30d30c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080a04e0 con 0x7fa308065190 2020-11-20T12:22:11.830 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.818+0000 7fa30d30c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30809ff60 con 0x7fa308065f90 2020-11-20T12:22:11.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.818+0000 7fa307fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065190 0x7fa308151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.819+0000 7fa307fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065190 0x7fa308151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55084/0 (socket says 172.21.15.187:55084) 2020-11-20T12:22:11.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.819+0000 7fa307fff700 1 -- 172.21.15.187:0/1871176265 learned_addr learned my addr 172.21.15.187:0/1871176265 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:11.831 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.819+0000 7fa307fff700 1 -- 172.21.15.187:0/1871176265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa308065f90 msgr2=0x7fa308143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.819+0000 7fa307fff700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa308065f90 0x7fa308143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:11.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.819+0000 7fa307fff700 1 -- 172.21.15.187:0/1871176265 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa30809ca90 msgr2=0x7fa308143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:11.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.819+0000 7fa307fff700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa30809ca90 0x7fa308143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:11.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.819+0000 7fa307fff700 1 -- 172.21.15.187:0/1871176265 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308151970 con 0x7fa308065190 2020-11-20T12:22:11.832 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.819+0000 7fa307fff700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065190 0x7fa308151480 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7fa2f0001e70 tx=0x7fa2f0007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:11.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.820+0000 7fa304ff9700 1 -- 172.21.15.187:0/1871176265 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0007020 con 0x7fa308065190 2020-11-20T12:22:11.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.820+0000 7fa304ff9700 1 -- 172.21.15.187:0/1871176265 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f0007180 con 0x7fa308065190 2020-11-20T12:22:11.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.820+0000 7fa304ff9700 1 -- 172.21.15.187:0/1871176265 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f000e560 con 0x7fa308065190 2020-11-20T12:22:11.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.820+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3080b1dd0 con 0x7fa308065190 2020-11-20T12:22:11.833 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.820+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa308149a80 con 0x7fa308065190 2020-11-20T12:22:11.834 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.821+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3080a06a0 con 0x7fa308065190 2020-11-20T12:22:11.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.822+0000 7fa304ff9700 1 -- 172.21.15.187:0/1871176265 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa2f0010070 con 0x7fa308065190 2020-11-20T12:22:11.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.822+0000 7fa304ff9700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa2e4056d50 0x7fa2e4059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:11.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.822+0000 7fa304ff9700 1 -- 172.21.15.187:0/1871176265 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa2f006f4c0 con 0x7fa308065190 2020-11-20T12:22:11.835 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.823+0000 7fa2ff7fe700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa2e4056d50 0x7fa2e4059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:11.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.823+0000 7fa2ff7fe700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa2e4056d50 0x7fa2e4059210 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7fa308145fb0 tx=0x7fa2f8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:11.836 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:11.826+0000 7fa304ff9700 1 -- 172.21.15.187:0/1871176265 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa2f00447e0 con 0x7fa308065190 2020-11-20T12:22:12.019 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.017+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa308061a20 con 0x7fa2e4056d50 2020-11-20T12:22:12.020 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.017+0000 7fa304ff9700 1 -- 172.21.15.187:0/1871176265 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa308061a20 con 0x7fa2e4056d50 2020-11-20T12:22:12.024 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa2e4056d50 msgr2=0x7fa2e4059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.025 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa2e4056d50 0x7fa2e4059210 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7fa308145fb0 tx=0x7fa2f8015000).stop 2020-11-20T12:22:12.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065190 msgr2=0x7fa308151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065190 0x7fa308151480 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7fa2f0001e70 tx=0x7fa2f0007c50).stop 2020-11-20T12:22:12.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 shutdown_connections 2020-11-20T12:22:12.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fa2e4056d50 0x7fa2e4059210 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.026 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa30809ca90 0x7fa308143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:12.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa308065190 0x7fa308151480 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 --2- 172.21.15.187:0/1871176265 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa308065f90 0x7fa308143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:12.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 >> 172.21.15.187:0/1871176265 conn(0x7fa3080bab30 msgr2=0x7fa3080aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:12.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 shutdown_connections 2020-11-20T12:22:12.027 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.020+0000 7fa30d30c700 1 -- 172.21.15.187:0/1871176265 wait complete. 2020-11-20T12:22:12.028 INFO:teuthology.orchestra.run.smithi187.stderr:instructed osd(s) 1 to deep-scrub 2020-11-20T12:22:12.045 INFO:tasks.ceph:Scrubbing osd.2 2020-11-20T12:22:12.045 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2020-11-20T12:22:12.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.293+0000 7f1265d72700 1 Processor -- start 2020-11-20T12:22:12.296 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.294+0000 7f1265d72700 1 -- start start 2020-11-20T12:22:12.297 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.295+0000 7f1265d72700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 0x7f1260065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.295+0000 7f1265d72700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f126009ca90 0x7f126009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.295+0000 7f1265d72700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12600b13e0 con 0x7f1260065f90 2020-11-20T12:22:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.295+0000 7f1265d72700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126014ab30 con 0x7f126009ca90 2020-11-20T12:22:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.295+0000 7f1265d72700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1260062c10 con 0x7f1260065190 2020-11-20T12:22:12.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.295+0000 7f1265571700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f126009ca90 0x7f126009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.299 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.296+0000 7f1265571700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f126009ca90 0x7f126009cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55096/0 (socket says 172.21.15.187:55096) 2020-11-20T12:22:12.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.296+0000 7f1265571700 1 -- 172.21.15.187:0/2989964466 learned_addr learned my addr 172.21.15.187:0/2989964466 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:12.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.296+0000 7f1264d70700 1 --2- 172.21.15.187:0/2989964466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 0x7f1260065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.296+0000 7f1265571700 1 -- 172.21.15.187:0/2989964466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 msgr2=0x7f1260065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:12.300 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.296+0000 7f1265571700 1 --2- 172.21.15.187:0/2989964466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 0x7f1260065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.296+0000 7f1265571700 1 -- 172.21.15.187:0/2989964466 >> v1:172.21.15.5:6789/0 conn(0x7f1260065f90 legacy=0x7f1260143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:12.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.296+0000 7f1265571700 1 -- 172.21.15.187:0/2989964466 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600b1640 con 0x7f126009ca90 2020-11-20T12:22:12.301 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.297+0000 7f1265571700 1 --2- 172.21.15.187:0/2989964466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 0x7f126009cf40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7f126014ac70 tx=0x7f1258009290).ready entity=mon.1 client_cookie=ed3920fc4e6883f5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:12.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.297+0000 7f12577fe700 1 -- 172.21.15.187:0/2989964466 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800bc10 con 0x7f126009ca90 2020-11-20T12:22:12.302 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.297+0000 7f12577fe700 1 -- 172.21.15.187:0/2989964466 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800bd70 con 0x7f126009ca90 2020-11-20T12:22:12.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.297+0000 7f12577fe700 1 -- 172.21.15.187:0/2989964466 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258011610 con 0x7f126009ca90 2020-11-20T12:22:12.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.298+0000 7f1265d72700 1 -- 172.21.15.187:0/2989964466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 msgr2=0x7f126009cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.298+0000 7f1265d72700 1 --2- 172.21.15.187:0/2989964466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 0x7f126009cf40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7f126014ac70 tx=0x7f1258009290).stop 2020-11-20T12:22:12.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.298+0000 7f1265d72700 1 -- 172.21.15.187:0/2989964466 shutdown_connections 2020-11-20T12:22:12.303 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.298+0000 7f1265d72700 1 --2- 172.21.15.187:0/2989964466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 0x7f126009cf40 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.298+0000 7f1265d72700 1 --2- 172.21.15.187:0/2989964466 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 0x7f1260065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.298+0000 7f1265d72700 1 -- 172.21.15.187:0/2989964466 >> 172.21.15.187:0/2989964466 conn(0x7f12600bab30 msgr2=0x7f12600baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:12.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.298+0000 7f1265d72700 1 -- 172.21.15.187:0/2989964466 shutdown_connections 2020-11-20T12:22:12.304 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.298+0000 7f1265d72700 1 -- 172.21.15.187:0/2989964466 wait complete. 2020-11-20T12:22:12.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.300+0000 7f1265d72700 1 Processor -- start 2020-11-20T12:22:12.305 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.300+0000 7f1265d72700 1 -- start start 2020-11-20T12:22:12.307 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265d72700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 0x7f12600b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265d72700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1260065f90 0x7f1260158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265d72700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 0x7f12601454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265d72700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1260158b10 con 0x7f1260065f90 2020-11-20T12:22:12.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265d72700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1260158db0 con 0x7f126009ca90 2020-11-20T12:22:12.308 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265d72700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600b4930 con 0x7f1260065190 2020-11-20T12:22:12.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265571700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 0x7f12601454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265571700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 0x7f12601454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55102/0 (socket says 172.21.15.187:55102) 2020-11-20T12:22:12.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1265571700 1 -- 172.21.15.187:0/2964177736 learned_addr learned my addr 172.21.15.187:0/2964177736 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:12.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.301+0000 7f1257fff700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1260065f90 0x7f1260158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.309 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1264d70700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 0x7f12600b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1265571700 1 -- 172.21.15.187:0/2964177736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 msgr2=0x7f12600b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1265571700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1260065190 0x7f12600b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1265571700 1 -- 172.21.15.187:0/2964177736 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1260065f90 msgr2=0x7f1260158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.310 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1265571700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1260065f90 0x7f1260158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1265571700 1 -- 172.21.15.187:0/2964177736 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126014ae80 con 0x7f126009ca90 2020-11-20T12:22:12.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1265571700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 0x7f12601454e0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7f1258009fd0 tx=0x7f1258016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:12.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258020070 con 0x7f126009ca90 2020-11-20T12:22:12.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1258010c00 con 0x7f126009ca90 2020-11-20T12:22:12.311 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12601586f0 con 0x7f126009ca90 2020-11-20T12:22:12.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800bc10 con 0x7f126009ca90 2020-11-20T12:22:12.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.302+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1260145c10 con 0x7f126009ca90 2020-11-20T12:22:12.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.303+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f126009dae0 con 0x7f126009ca90 2020-11-20T12:22:12.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.304+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1258017070 con 0x7f126009ca90 2020-11-20T12:22:12.312 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.305+0000 7f1254ff9700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1238056d50 0x7f1238059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.305+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f12580720f0 con 0x7f126009ca90 2020-11-20T12:22:12.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.305+0000 7f1254ff9700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f123805e280 0x7f12380606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.305+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1238060d20 con 0x7f123805e280 2020-11-20T12:22:12.313 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.305+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=161) v2 ==== 24+0+0 (secure 0 0 0) 0x7f125800bee0 con 0x7f126009ca90 2020-11-20T12:22:12.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.305+0000 7f1264d70700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1238056d50 0x7f1238059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.314 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.305+0000 7f1257fff700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f123805e280 0x7f12380606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.306+0000 7f1264d70700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1238056d50 0x7f1238059210 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f125c009650 tx=0x7f125c008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:12.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.306+0000 7f1257fff700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f123805e280 0x7f12380606a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:12.315 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.307+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 <== osd.2 v2:172.21.15.187:6810/19904 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f1238060d20 con 0x7f123805e280 2020-11-20T12:22:12.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.323+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 --> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1260158970 con 0x7f123805e280 2020-11-20T12:22:12.326 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.324+0000 7f1254ff9700 1 -- 172.21.15.187:0/2964177736 <== osd.2 v2:172.21.15.187:6810/19904 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1260158970 con 0x7f123805e280 2020-11-20T12:22:12.327 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:12.327 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T12:22:12.327 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:12.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f123805e280 msgr2=0x7f12380606a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265d72700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6810/19904,v1:172.21.15.187:6811/19904] conn(0x7f123805e280 0x7f12380606a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.330 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1238056d50 msgr2=0x7f1238059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265d72700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f1238056d50 0x7f1238059210 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f125c009650 tx=0x7f125c008040).stop 2020-11-20T12:22:12.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 msgr2=0x7f12601454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265d72700 1 --2- 172.21.15.187:0/2964177736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f126009ca90 0x7f12601454e0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7f1258009fd0 tx=0x7f1258016000).stop 2020-11-20T12:22:12.331 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265571700 1 -- 172.21.15.187:0/2964177736 reap_dead start 2020-11-20T12:22:12.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 shutdown_connections 2020-11-20T12:22:12.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.325+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 >> 172.21.15.187:0/2964177736 conn(0x7f12600bab30 msgr2=0x7f1260067940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:12.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.326+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 shutdown_connections 2020-11-20T12:22:12.332 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.326+0000 7f1265d72700 1 -- 172.21.15.187:0/2964177736 wait complete. 2020-11-20T12:22:12.352 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2020-11-20T12:22:12.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.587+0000 7f6d3b457700 1 Processor -- start 2020-11-20T12:22:12.590 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.588+0000 7f6d3b457700 1 -- start start 2020-11-20T12:22:12.591 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.589+0000 7f6d3b457700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6d340652e0 0x7f6d340656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.589+0000 7f6d3b457700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d34149dd0 con 0x7f6d340b1590 2020-11-20T12:22:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.589+0000 7f6d3b457700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3414a070 con 0x7f6d340b7300 2020-11-20T12:22:12.592 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.589+0000 7f6d3b457700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3414a310 con 0x7f6d340652e0 2020-11-20T12:22:12.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.590+0000 7f6d399f4700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6d340b7300 0x7f6d34143920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51718/0 (socket says 172.21.15.187:51718) 2020-11-20T12:22:12.594 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.590+0000 7f6d399f4700 1 -- 172.21.15.187:0/3613633591 learned_addr learned my addr 172.21.15.187:0/3613633591 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:12.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.590+0000 7f6d391f3700 1 --2- 172.21.15.187:0/3613633591 >> v2:172.21.15.5:3300/0 conn(0x7f6d340652e0 0x7f6d340656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.595 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.590+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467583689 0 0) 0x7f6d3414a070 con 0x7f6d340b7300 2020-11-20T12:22:12.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.590+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d1c003240 con 0x7f6d340b7300 2020-11-20T12:22:12.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2644978349 0 0) 0x7f6d1c003240 con 0x7f6d340b7300 2020-11-20T12:22:12.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 >> v2:172.21.15.5:3300/0 conn(0x7f6d340652e0 msgr2=0x7f6d340656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:12.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 --2- 172.21.15.187:0/3613633591 >> v2:172.21.15.5:3300/0 conn(0x7f6d340652e0 0x7f6d340656c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.596 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 >> v1:172.21.15.5:6789/0 conn(0x7f6d340b1590 legacy=0x7f6d3414a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d340b5380 con 0x7f6d340b7300 2020-11-20T12:22:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f6d2c001860 con 0x7f6d340b7300 2020-11-20T12:22:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 >> v1:172.21.15.187:6789/0 conn(0x7f6d340b7300 legacy=0x7f6d34143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c003730 0x7f6d1c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1c0044e0 0x7f6d1c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d1c004e50 0x7f6d1c00fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3414a310 con 0x7f6d1c004e50 2020-11-20T12:22:12.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c004260 con 0x7f6d1c0044e0 2020-11-20T12:22:12.598 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.591+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c010030 con 0x7f6d1c003730 2020-11-20T12:22:12.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d389f2700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1c0044e0 0x7f6d1c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d391f3700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c003730 0x7f6d1c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.599 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d399f4700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d1c004e50 0x7f6d1c00fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d389f2700 1 -- 172.21.15.187:0/3613633591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c003730 msgr2=0x7f6d1c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:12.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d389f2700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c003730 0x7f6d1c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.603 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d389f2700 1 -- 172.21.15.187:0/3613633591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d1c004e50 msgr2=0x7f6d1c00fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:12.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d389f2700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d1c004e50 0x7f6d1c00fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d389f2700 1 -- 172.21.15.187:0/3613633591 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c003240 con 0x7f6d1c0044e0 2020-11-20T12:22:12.604 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d399f4700 1 -- 172.21.15.187:0/3613633591 reap_dead start 2020-11-20T12:22:12.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.592+0000 7f6d389f2700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1c0044e0 0x7f6d1c004960 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f6d240018e0 tx=0x7f6d24007be0).ready entity=mon.1 client_cookie=10501ed736f14188 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:12.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.593+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24007020 con 0x7f6d1c0044e0 2020-11-20T12:22:12.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.593+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d24007180 con 0x7f6d1c0044e0 2020-11-20T12:22:12.605 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.593+0000 7f6d23fff700 1 -- 172.21.15.187:0/3613633591 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2400c830 con 0x7f6d1c0044e0 2020-11-20T12:22:12.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.593+0000 7f6d3b457700 1 -- 172.21.15.187:0/3613633591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1c0044e0 msgr2=0x7f6d1c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.593+0000 7f6d3b457700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1c0044e0 0x7f6d1c004960 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f6d240018e0 tx=0x7f6d24007be0).stop 2020-11-20T12:22:12.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.593+0000 7f6d3b457700 1 -- 172.21.15.187:0/3613633591 shutdown_connections 2020-11-20T12:22:12.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.593+0000 7f6d3b457700 1 --2- 172.21.15.187:0/3613633591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1c0044e0 0x7f6d1c004960 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.606 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.593+0000 7f6d3b457700 1 -- 172.21.15.187:0/3613633591 >> 172.21.15.187:0/3613633591 conn(0x7f6d340bacd0 msgr2=0x7f6d340bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:12.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.594+0000 7f6d3b457700 1 -- 172.21.15.187:0/3613633591 shutdown_connections 2020-11-20T12:22:12.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.594+0000 7f6d3b457700 1 -- 172.21.15.187:0/3613633591 wait complete. 2020-11-20T12:22:12.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.595+0000 7f6d3b457700 1 Processor -- start 2020-11-20T12:22:12.607 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.596+0000 7f6d3b457700 1 -- start start 2020-11-20T12:22:12.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d3b457700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c0044e0 0x7f6d34092450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d3b457700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d34153b60 0x7f6d34090a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d3b457700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d341540a0 0x7f6d34154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d3b457700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d34092cf0 con 0x7f6d34153b60 2020-11-20T12:22:12.608 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d3b457700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d34092f90 con 0x7f6d341540a0 2020-11-20T12:22:12.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d3b457700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d34093230 con 0x7f6d1c0044e0 2020-11-20T12:22:12.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d399f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d341540a0 0x7f6d34154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d399f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d341540a0 0x7f6d34154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55124/0 (socket says 172.21.15.187:55124) 2020-11-20T12:22:12.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.597+0000 7f6d399f4700 1 -- 172.21.15.187:0/3292334387 learned_addr learned my addr 172.21.15.187:0/3292334387 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:12.609 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d389f2700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d34153b60 0x7f6d34090a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d391f3700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c0044e0 0x7f6d34092450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d399f4700 1 -- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c0044e0 msgr2=0x7f6d34092450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d399f4700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c0044e0 0x7f6d34092450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d399f4700 1 -- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d34153b60 msgr2=0x7f6d34090a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.610 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d399f4700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d34153b60 0x7f6d34090a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d399f4700 1 -- 172.21.15.187:0/3292334387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d34092940 con 0x7f6d341540a0 2020-11-20T12:22:12.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d389f2700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d34153b60 0x7f6d34090a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:12.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d399f4700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d341540a0 0x7f6d34154520 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7f6d340b76e0 tx=0x7f6d2c001200).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:12.611 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d217fa700 1 -- 172.21.15.187:0/3292334387 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c00cc90 con 0x7f6d341540a0 2020-11-20T12:22:12.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d217fa700 1 -- 172.21.15.187:0/3292334387 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2c00cdf0 con 0x7f6d341540a0 2020-11-20T12:22:12.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d34090f80 con 0x7f6d341540a0 2020-11-20T12:22:12.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d217fa700 1 -- 172.21.15.187:0/3292334387 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c014660 con 0x7f6d341540a0 2020-11-20T12:22:12.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.598+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d34157220 con 0x7f6d341540a0 2020-11-20T12:22:12.612 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.600+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d34091170 con 0x7f6d341540a0 2020-11-20T12:22:12.613 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.600+0000 7f6d217fa700 1 -- 172.21.15.187:0/3292334387 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6d2c014a10 con 0x7f6d341540a0 2020-11-20T12:22:12.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.601+0000 7f6d217fa700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d0c056d20 0x7f6d0c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:12.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.601+0000 7f6d217fa700 1 -- 172.21.15.187:0/3292334387 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6d2c06fff0 con 0x7f6d341540a0 2020-11-20T12:22:12.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.601+0000 7f6d391f3700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d0c056d20 0x7f6d0c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:12.614 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.602+0000 7f6d391f3700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d0c056d20 0x7f6d0c0591e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f6d3405f730 tx=0x7f6d30008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:12.615 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.605+0000 7f6d217fa700 1 -- 172.21.15.187:0/3292334387 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6d2c045420 con 0x7f6d341540a0 2020-11-20T12:22:12.792 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.790+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d340b4e20 con 0x7f6d0c056d20 2020-11-20T12:22:12.793 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.790+0000 7f6d217fa700 1 -- 172.21.15.187:0/3292334387 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6d340b4e20 con 0x7f6d0c056d20 2020-11-20T12:22:12.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.792+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d0c056d20 msgr2=0x7f6d0c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.792+0000 7f6d3b457700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d0c056d20 0x7f6d0c0591e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f6d3405f730 tx=0x7f6d30008040).stop 2020-11-20T12:22:12.795 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.792+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d341540a0 msgr2=0x7f6d34154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:12.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.792+0000 7f6d3b457700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d341540a0 0x7f6d34154520 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7f6d340b76e0 tx=0x7f6d2c001200).stop 2020-11-20T12:22:12.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.793+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 shutdown_connections 2020-11-20T12:22:12.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.793+0000 7f6d3b457700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f6d0c056d20 0x7f6d0c0591e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.793+0000 7f6d3b457700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d341540a0 0x7f6d34154520 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.796 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.793+0000 7f6d3b457700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d1c0044e0 0x7f6d34092450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.793+0000 7f6d3b457700 1 --2- 172.21.15.187:0/3292334387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d34153b60 0x7f6d34090a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:12.797 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.793+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 >> 172.21.15.187:0/3292334387 conn(0x7f6d340bacd0 msgr2=0x7f6d3414a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:12.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.793+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 shutdown_connections 2020-11-20T12:22:12.798 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:12.793+0000 7f6d3b457700 1 -- 172.21.15.187:0/3292334387 wait complete. 2020-11-20T12:22:12.798 INFO:teuthology.orchestra.run.smithi187.stderr:instructed osd(s) 2 to deep-scrub 2020-11-20T12:22:12.819 INFO:tasks.ceph:Scrubbing osd.3 2020-11-20T12:22:12.819 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2020-11-20T12:22:13.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.065+0000 7f484359e700 1 Processor -- start 2020-11-20T12:22:13.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.065+0000 7f484359e700 1 -- start start 2020-11-20T12:22:13.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f484359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 0x7f4844061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.069 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f484359e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48440622c0 0x7f48440b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f484359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4844066f70 0x7f48440b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f484359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4844067730 con 0x7f4844066f70 2020-11-20T12:22:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f484359e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48440b1300 con 0x7f48440622c0 2020-11-20T12:22:13.070 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f484359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48440b15a0 con 0x7f48441484d0 2020-11-20T12:22:13.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f4841d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48440622c0 0x7f48440b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f4841d9b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48440622c0 0x7f48440b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55128/0 (socket says 172.21.15.187:55128) 2020-11-20T12:22:13.072 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.067+0000 7f4841d9b700 1 -- 172.21.15.187:0/996028489 learned_addr learned my addr 172.21.15.187:0/996028489 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:13.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f4842d9d700 1 --2- 172.21.15.187:0/996028489 >> v2:172.21.15.5:3300/0 conn(0x7f4844066f70 0x7f48440b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f4841d9b700 1 -- 172.21.15.187:0/996028489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 msgr2=0x7f4844061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:13.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f484259c700 1 --2- 172.21.15.187:0/996028489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 0x7f4844061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.073 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f4841d9b700 1 --2- 172.21.15.187:0/996028489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 0x7f4844061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f4841d9b700 1 -- 172.21.15.187:0/996028489 >> v2:172.21.15.5:3300/0 conn(0x7f4844066f70 msgr2=0x7f48440b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f4841d9b700 1 --2- 172.21.15.187:0/996028489 >> v2:172.21.15.5:3300/0 conn(0x7f4844066f70 0x7f48440b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f4841d9b700 1 -- 172.21.15.187:0/996028489 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48440b1840 con 0x7f48440622c0 2020-11-20T12:22:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f484259c700 1 --2- 172.21.15.187:0/996028489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 0x7f4844061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:13.074 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.068+0000 7f4841d9b700 1 --2- 172.21.15.187:0/996028489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 0x7f48440b5640 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f4844066380 tx=0x7f4838006eb0).ready entity=mon.1 client_cookie=b5627a14d2f2d051 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.069+0000 7f484159a700 1 -- 172.21.15.187:0/996028489 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48380098d0 con 0x7f48440622c0 2020-11-20T12:22:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.069+0000 7f484159a700 1 -- 172.21.15.187:0/996028489 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4838009a30 con 0x7f48440622c0 2020-11-20T12:22:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.069+0000 7f484159a700 1 -- 172.21.15.187:0/996028489 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838009db0 con 0x7f48440622c0 2020-11-20T12:22:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.069+0000 7f484359e700 1 -- 172.21.15.187:0/996028489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 msgr2=0x7f48440b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.075 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.069+0000 7f484359e700 1 --2- 172.21.15.187:0/996028489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 0x7f48440b5640 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f4844066380 tx=0x7f4838006eb0).stop 2020-11-20T12:22:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.070+0000 7f484359e700 1 -- 172.21.15.187:0/996028489 shutdown_connections 2020-11-20T12:22:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.070+0000 7f484359e700 1 --2- 172.21.15.187:0/996028489 >> v2:172.21.15.5:3300/0 conn(0x7f4844066f70 0x7f48440b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.070+0000 7f484359e700 1 --2- 172.21.15.187:0/996028489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 0x7f4844061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.076 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.070+0000 7f484359e700 1 --2- 172.21.15.187:0/996028489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 0x7f48440b5640 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.070+0000 7f484359e700 1 -- 172.21.15.187:0/996028489 >> 172.21.15.187:0/996028489 conn(0x7f48440bab30 msgr2=0x7f48440baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.070+0000 7f484359e700 1 -- 172.21.15.187:0/996028489 shutdown_connections 2020-11-20T12:22:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.070+0000 7f484359e700 1 -- 172.21.15.187:0/996028489 wait complete. 2020-11-20T12:22:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.072+0000 7f484359e700 1 Processor -- start 2020-11-20T12:22:13.079 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.072+0000 7f484359e700 1 -- start start 2020-11-20T12:22:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484359e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 0x7f484413ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4844066f70 0x7f4844137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 0x7f48441383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484413c240 con 0x7f4844066f70 2020-11-20T12:22:13.080 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484359e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484413f1a0 con 0x7f48440622c0 2020-11-20T12:22:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484409ac60 con 0x7f48441484d0 2020-11-20T12:22:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 0x7f484413ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 0x7f484413ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55134/0 (socket says 172.21.15.187:55134) 2020-11-20T12:22:13.081 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.073+0000 7f484259c700 1 -- 172.21.15.187:0/1696992880 learned_addr learned my addr 172.21.15.187:0/1696992880 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f4841d9b700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4844066f70 0x7f4844137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f484259c700 1 -- 172.21.15.187:0/1696992880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 msgr2=0x7f48441383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f484259c700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48441484d0 0x7f48441383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f484259c700 1 -- 172.21.15.187:0/1696992880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4844066f70 msgr2=0x7f4844137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f484259c700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4844066f70 0x7f4844137ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f484259c700 1 -- 172.21.15.187:0/1696992880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484415b8e0 con 0x7f48440622c0 2020-11-20T12:22:13.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f4841d9b700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4844066f70 0x7f4844137ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:13.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f484259c700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 0x7f484413ebd0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7f483400c8f0 tx=0x7f4834008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834007710 con 0x7f48440622c0 2020-11-20T12:22:13.083 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4834007870 con 0x7f48440622c0 2020-11-20T12:22:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834007bf0 con 0x7f48440622c0 2020-11-20T12:22:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.074+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484415bb30 con 0x7f48440622c0 2020-11-20T12:22:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.075+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484415bfa0 con 0x7f48440622c0 2020-11-20T12:22:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.076+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f484413f360 con 0x7f48440622c0 2020-11-20T12:22:13.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.076+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f483401b460 con 0x7f48440622c0 2020-11-20T12:22:13.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.077+0000 7f48327fc700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4818056d20 0x7f48180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.085 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.077+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f48340701d0 con 0x7f48440622c0 2020-11-20T12:22:13.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.077+0000 7f48327fc700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f481805e2d0 0x7f48180606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.077+0000 7f4841d9b700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4818056d20 0x7f48180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.086 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.077+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 --> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4818060d70 con 0x7f481805e2d0 2020-11-20T12:22:13.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.077+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=161) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4834072840 con 0x7f48440622c0 2020-11-20T12:22:13.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.078+0000 7f4842d9d700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f481805e2d0 0x7f48180606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.078+0000 7f4841d9b700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4818056d20 0x7f48180591e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f4844066380 tx=0x7f4838015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.078+0000 7f4842d9d700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f481805e2d0 0x7f48180606f0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.087 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.079+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 <== osd.3 v2:172.21.15.187:6826/19935 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f4818060d70 con 0x7f481805e2d0 2020-11-20T12:22:13.098 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.096+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 --> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4844061a20 con 0x7f481805e2d0 2020-11-20T12:22:13.099 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.096+0000 7f48327fc700 1 -- 172.21.15.187:0/1696992880 <== osd.3 v2:172.21.15.187:6826/19935 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4844061a20 con 0x7f481805e2d0 2020-11-20T12:22:13.099 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:13.099 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T12:22:13.099 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:13.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f481805e2d0 msgr2=0x7f48180606f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6826/19935,v1:172.21.15.187:6827/19935] conn(0x7f481805e2d0 0x7f48180606f0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4818056d20 msgr2=0x7f48180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.101 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4818056d20 0x7f48180591e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f4844066380 tx=0x7f4838015000).stop 2020-11-20T12:22:13.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 msgr2=0x7f484413ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 --2- 172.21.15.187:0/1696992880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48440622c0 0x7f484413ebd0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7f483400c8f0 tx=0x7f4834008040).stop 2020-11-20T12:22:13.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f4842d9d700 1 -- 172.21.15.187:0/1696992880 reap_dead start 2020-11-20T12:22:13.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 shutdown_connections 2020-11-20T12:22:13.102 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 >> 172.21.15.187:0/1696992880 conn(0x7f48440bab30 msgr2=0x7f4844146e00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:13.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.097+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 shutdown_connections 2020-11-20T12:22:13.103 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.098+0000 7f484359e700 1 -- 172.21.15.187:0/1696992880 wait complete. 2020-11-20T12:22:13.120 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2020-11-20T12:22:13.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.356+0000 7f53bedc8700 1 Processor -- start 2020-11-20T12:22:13.358 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.356+0000 7f53bedc8700 1 -- start start 2020-11-20T12:22:13.359 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.357+0000 7f53bedc8700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53b80b6650 0x7f53b80afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.357+0000 7f53bedc8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f53b8066f70 0x7f53b80aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.357+0000 7f53bedc8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b80b1300 con 0x7f53b81484d0 2020-11-20T12:22:13.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.357+0000 7f53bedc8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b8067730 con 0x7f53b8066f70 2020-11-20T12:22:13.360 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.357+0000 7f53bedc8700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b80b15a0 con 0x7f53b80b6650 2020-11-20T12:22:13.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.357+0000 7f53bd5c5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53b80b6650 0x7f53b80afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.357+0000 7f53bd5c5700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53b80b6650 0x7f53b80afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55146/0 (socket says 172.21.15.187:55146) 2020-11-20T12:22:13.361 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.357+0000 7f53bd5c5700 1 -- 172.21.15.187:0/137493270 learned_addr learned my addr 172.21.15.187:0/137493270 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:13.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.358+0000 7f53be5c7700 1 --2- 172.21.15.187:0/137493270 >> v2:172.21.15.5:3300/0 conn(0x7f53b8066f70 0x7f53b80aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.358+0000 7f53bd5c5700 1 -- 172.21.15.187:0/137493270 >> v2:172.21.15.5:3300/0 conn(0x7f53b8066f70 msgr2=0x7f53b80aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:13.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.358+0000 7f53bd5c5700 1 --2- 172.21.15.187:0/137493270 >> v2:172.21.15.5:3300/0 conn(0x7f53b8066f70 0x7f53b80aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.358+0000 7f53bd5c5700 1 -- 172.21.15.187:0/137493270 >> v1:172.21.15.5:6789/0 conn(0x7f53b81484d0 legacy=0x7f53b8061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:13.363 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.358+0000 7f53bd5c5700 1 -- 172.21.15.187:0/137493270 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b80b1840 con 0x7f53b80b6650 2020-11-20T12:22:13.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.358+0000 7f53bd5c5700 1 --2- 172.21.15.187:0/137493270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b80afa90 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7f53b8068600 tx=0x7f53a8006eb0).ready entity=mon.1 client_cookie=bf6af2920f0de763 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.358+0000 7f53bcdc4700 1 -- 172.21.15.187:0/137493270 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8009890 con 0x7f53b80b6650 2020-11-20T12:22:13.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.358+0000 7f53bcdc4700 1 -- 172.21.15.187:0/137493270 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a80099f0 con 0x7f53b80b6650 2020-11-20T12:22:13.364 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.359+0000 7f53bcdc4700 1 -- 172.21.15.187:0/137493270 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8009d70 con 0x7f53b80b6650 2020-11-20T12:22:13.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.359+0000 7f53bedc8700 1 -- 172.21.15.187:0/137493270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 msgr2=0x7f53b80afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.359+0000 7f53bedc8700 1 --2- 172.21.15.187:0/137493270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b80afa90 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7f53b8068600 tx=0x7f53a8006eb0).stop 2020-11-20T12:22:13.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.359+0000 7f53bedc8700 1 -- 172.21.15.187:0/137493270 shutdown_connections 2020-11-20T12:22:13.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.359+0000 7f53bedc8700 1 --2- 172.21.15.187:0/137493270 >> v2:172.21.15.5:3300/0 conn(0x7f53b8066f70 0x7f53b80aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.365 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.359+0000 7f53bedc8700 1 --2- 172.21.15.187:0/137493270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b80afa90 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.359+0000 7f53bedc8700 1 -- 172.21.15.187:0/137493270 >> 172.21.15.187:0/137493270 conn(0x7f53b80bab30 msgr2=0x7f53b80baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:13.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.359+0000 7f53bedc8700 1 -- 172.21.15.187:0/137493270 shutdown_connections 2020-11-20T12:22:13.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.360+0000 7f53bedc8700 1 -- 172.21.15.187:0/137493270 wait complete. 2020-11-20T12:22:13.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.361+0000 7f53bedc8700 1 Processor -- start 2020-11-20T12:22:13.368 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.362+0000 7f53bedc8700 1 -- start start 2020-11-20T12:22:13.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.362+0000 7f53bedc8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b8066f70 0x7f53b8146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.362+0000 7f53bedc8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b8142e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bedc8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53b81484d0 0x7f53b8143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bedc8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b809c930 con 0x7f53b8066f70 2020-11-20T12:22:13.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bedc8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b809cbd0 con 0x7f53b80b6650 2020-11-20T12:22:13.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bedc8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b809ce70 con 0x7f53b81484d0 2020-11-20T12:22:13.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bd5c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b8142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bd5c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b8142e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55152/0 (socket says 172.21.15.187:55152) 2020-11-20T12:22:13.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bd5c5700 1 -- 172.21.15.187:0/1621628275 learned_addr learned my addr 172.21.15.187:0/1621628275 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:13.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bd5c5700 1 -- 172.21.15.187:0/1621628275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53b81484d0 msgr2=0x7f53b8143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bd5c5700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53b81484d0 0x7f53b8143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bd5c5700 1 -- 172.21.15.187:0/1621628275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b8066f70 msgr2=0x7f53b8146b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bd5c5700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b8066f70 0x7f53b8146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:13.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.363+0000 7f53bd5c5700 1 -- 172.21.15.187:0/1621628275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b8147510 con 0x7f53b80b6650 2020-11-20T12:22:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.364+0000 7f53bd5c5700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b8142e40 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7f53b80b16e0 tx=0x7f53a8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.364+0000 7f53a5ffb700 1 -- 172.21.15.187:0/1621628275 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8011dd0 con 0x7f53b80b6650 2020-11-20T12:22:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.364+0000 7f53a5ffb700 1 -- 172.21.15.187:0/1621628275 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a801e460 con 0x7f53b80b6650 2020-11-20T12:22:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.364+0000 7f53a5ffb700 1 -- 172.21.15.187:0/1621628275 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a801e800 con 0x7f53b80b6650 2020-11-20T12:22:13.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.364+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b8147050 con 0x7f53b80b6650 2020-11-20T12:22:13.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.364+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b809d4d0 con 0x7f53b80b6650 2020-11-20T12:22:13.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.365+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b809d030 con 0x7f53b80b6650 2020-11-20T12:22:13.373 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.366+0000 7f53a5ffb700 1 -- 172.21.15.187:0/1621628275 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f53a8025030 con 0x7f53b80b6650 2020-11-20T12:22:13.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.366+0000 7f53a5ffb700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5390056d20 0x7f53900591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.366+0000 7f53a5ffb700 1 -- 172.21.15.187:0/1621628275 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f53a8071950 con 0x7f53b80b6650 2020-11-20T12:22:13.374 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.367+0000 7f53bddc6700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5390056d20 0x7f53900591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.367+0000 7f53bddc6700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5390056d20 0x7f53900591e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f53b4006fd0 tx=0x7f53b4006930).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.375 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.370+0000 7f53a5ffb700 1 -- 172.21.15.187:0/1621628275 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f53a8046c50 con 0x7f53b80b6650 2020-11-20T12:22:13.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.555+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f53b8061a20 con 0x7f5390056d20 2020-11-20T12:22:13.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.556+0000 7f53a5ffb700 1 -- 172.21.15.187:0/1621628275 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f53b8061a20 con 0x7f5390056d20 2020-11-20T12:22:13.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5390056d20 msgr2=0x7f53900591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5390056d20 0x7f53900591e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f53b4006fd0 tx=0x7f53b4006930).stop 2020-11-20T12:22:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 msgr2=0x7f53b8142e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b8142e40 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7f53b80b16e0 tx=0x7f53a8015000).stop 2020-11-20T12:22:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 shutdown_connections 2020-11-20T12:22:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f5390056d20 0x7f53900591e0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b8066f70 0x7f53b8146b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f53b81484d0 0x7f53b8143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:13.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 --2- 172.21.15.187:0/1621628275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b80b6650 0x7f53b8142e40 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 >> 172.21.15.187:0/1621628275 conn(0x7f53b80bab30 msgr2=0x7f53b80b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:13.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.559+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 shutdown_connections 2020-11-20T12:22:13.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.560+0000 7f53bedc8700 1 -- 172.21.15.187:0/1621628275 wait complete. 2020-11-20T12:22:13.567 INFO:teuthology.orchestra.run.smithi187.stderr:instructed osd(s) 3 to deep-scrub 2020-11-20T12:22:13.583 INFO:tasks.ceph:Scrubbing osd.4 2020-11-20T12:22:13.584 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2020-11-20T12:22:13.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.836+0000 7f8214c1e700 1 Processor -- start 2020-11-20T12:22:13.839 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.836+0000 7f8214c1e700 1 -- start start 2020-11-20T12:22:13.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.837+0000 7f8214c1e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f82100b57b0 0x7f8210068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.838+0000 7f8214c1e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8210144b80 con 0x7f82100b5f50 2020-11-20T12:22:13.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.838+0000 7f8214c1e700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82101489e0 con 0x7f82101482d0 2020-11-20T12:22:13.840 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.838+0000 7f8214c1e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8210148c80 con 0x7f82100b57b0 2020-11-20T12:22:13.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.838+0000 7f8206ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f82101482d0 0x7f8210062660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51766/0 (socket says 172.21.15.187:51766) 2020-11-20T12:22:13.843 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.838+0000 7f8206ffd700 1 -- 172.21.15.187:0/374961085 learned_addr learned my addr 172.21.15.187:0/374961085 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:13.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820f7fe700 1 --2- 172.21.15.187:0/374961085 >> v2:172.21.15.5:3300/0 conn(0x7f82100b57b0 0x7f8210068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173397691 0 0) 0x7f82101489e0 con 0x7f82101482d0 2020-11-20T12:22:13.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f0003240 con 0x7f82101482d0 2020-11-20T12:22:13.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 144644826 0 0) 0x7f81f0003240 con 0x7f82101482d0 2020-11-20T12:22:13.844 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 >> v2:172.21.15.5:3300/0 conn(0x7f82100b57b0 msgr2=0x7f8210068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:13.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 --2- 172.21.15.187:0/374961085 >> v2:172.21.15.5:3300/0 conn(0x7f82100b57b0 0x7f8210068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 >> v1:172.21.15.5:6789/0 conn(0x7f82100b5f50 legacy=0x7f82100b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8210148f20 con 0x7f82101482d0 2020-11-20T12:22:13.845 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f81f80014e0 con 0x7f82101482d0 2020-11-20T12:22:13.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 >> v1:172.21.15.187:6789/0 conn(0x7f82101482d0 legacy=0x7f8210062660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81f0003780 0x7f81f0003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81f00044e0 0x7f81f0004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f0004e50 0x7f81f000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.846 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.839+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8210148c80 con 0x7f81f00044e0 2020-11-20T12:22:13.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.840+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f0005300 con 0x7f81f0004e50 2020-11-20T12:22:13.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.840+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f000bc20 con 0x7f81f0003780 2020-11-20T12:22:13.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.840+0000 7f820ffff700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f0004e50 0x7f81f000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.840+0000 7f820f7fe700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81f0003780 0x7f81f0003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.847 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.840+0000 7f8206ffd700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81f00044e0 0x7f81f0004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.840+0000 7f820ffff700 1 -- 172.21.15.187:0/374961085 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81f0003780 msgr2=0x7f81f0003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:13.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820ffff700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81f0003780 0x7f81f0003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820ffff700 1 -- 172.21.15.187:0/374961085 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81f00044e0 msgr2=0x7f81f0004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:13.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820ffff700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81f00044e0 0x7f81f0004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.848 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820ffff700 1 -- 172.21.15.187:0/374961085 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f0003240 con 0x7f81f0004e50 2020-11-20T12:22:13.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820ffff700 1 -- 172.21.15.187:0/374961085 reap_dead start 2020-11-20T12:22:13.849 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820ffff700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f0004e50 0x7f81f000b730 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f8208001e70 tx=0x7f8208007f10).ready entity=mon.1 client_cookie=5947d3178fc96d78 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208001540 con 0x7f81f0004e50 2020-11-20T12:22:13.850 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8208003030 con 0x7f81f0004e50 2020-11-20T12:22:13.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.841+0000 7f820effd700 1 -- 172.21.15.187:0/374961085 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820800cdf0 con 0x7f81f0004e50 2020-11-20T12:22:13.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.842+0000 7f8214c1e700 1 -- 172.21.15.187:0/374961085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f0004e50 msgr2=0x7f81f000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.842+0000 7f8214c1e700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f0004e50 0x7f81f000b730 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f8208001e70 tx=0x7f8208007f10).stop 2020-11-20T12:22:13.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.842+0000 7f8214c1e700 1 -- 172.21.15.187:0/374961085 shutdown_connections 2020-11-20T12:22:13.851 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.842+0000 7f8214c1e700 1 --2- 172.21.15.187:0/374961085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f0004e50 0x7f81f000b730 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.842+0000 7f8214c1e700 1 -- 172.21.15.187:0/374961085 >> 172.21.15.187:0/374961085 conn(0x7f82100bab30 msgr2=0x7f82100baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:13.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.842+0000 7f8214c1e700 1 -- 172.21.15.187:0/374961085 shutdown_connections 2020-11-20T12:22:13.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.842+0000 7f8214c1e700 1 -- 172.21.15.187:0/374961085 wait complete. 2020-11-20T12:22:13.852 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.844+0000 7f8214c1e700 1 Processor -- start 2020-11-20T12:22:13.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.844+0000 7f8214c1e700 1 -- start start 2020-11-20T12:22:13.855 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.845+0000 7f8214c1e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81f0004e50 0x7f821009f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.845+0000 7f8214c1e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100b2060 0x7f82100b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.845+0000 7f8214c1e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82100991f0 0x7f8210099650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.845+0000 7f8214c1e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8210145ef0 con 0x7f81f0004e50 2020-11-20T12:22:13.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.845+0000 7f8214c1e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821009fcc0 con 0x7f82100b2060 2020-11-20T12:22:13.856 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.845+0000 7f8214c1e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8210146120 con 0x7f82100991f0 2020-11-20T12:22:13.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f8206ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100b2060 0x7f82100b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f8206ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100b2060 0x7f82100b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55172/0 (socket says 172.21.15.187:55172) 2020-11-20T12:22:13.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f8206ffd700 1 -- 172.21.15.187:0/376256917 learned_addr learned my addr 172.21.15.187:0/376256917 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:13.857 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f820f7fe700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81f0004e50 0x7f821009f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f8206ffd700 1 -- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82100991f0 msgr2=0x7f8210099650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f8206ffd700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82100991f0 0x7f8210099650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:13.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f8206ffd700 1 -- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81f0004e50 msgr2=0x7f821009f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f8206ffd700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81f0004e50 0x7f821009f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.858 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.846+0000 7f8206ffd700 1 -- 172.21.15.187:0/376256917 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82101463c0 con 0x7f82100b2060 2020-11-20T12:22:13.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.847+0000 7f8206ffd700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100b2060 0x7f82100b40c0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7f81f800ea80 tx=0x7f8210068110).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.847+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f800a730 con 0x7f82100b2060 2020-11-20T12:22:13.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.847+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f8010040 con 0x7f82100b2060 2020-11-20T12:22:13.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.847+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8210146660 con 0x7f82100b2060 2020-11-20T12:22:13.859 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.847+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f800aae0 con 0x7f82100b2060 2020-11-20T12:22:13.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.847+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82100b24e0 con 0x7f82100b2060 2020-11-20T12:22:13.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.848+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8210061a20 con 0x7f82100b2060 2020-11-20T12:22:13.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.849+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f81f801b020 con 0x7f82100b2060 2020-11-20T12:22:13.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.849+0000 7f8207fff700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ec056d20 0x7f81ec0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.860 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.849+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f81f806f6c0 con 0x7f82100b2060 2020-11-20T12:22:13.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.850+0000 7f820f7fe700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ec056d20 0x7f81ec0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.850+0000 7f8207fff700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f81ec05e2d0 0x7f81ec0606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:13.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.850+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81ec060d70 con 0x7f81ec05e2d0 2020-11-20T12:22:13.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.850+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=161) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81f8071cb0 con 0x7f82100b2060 2020-11-20T12:22:13.861 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.850+0000 7f820f7fe700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ec056d20 0x7f81ec0591e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f8200006fd0 tx=0x7f8200008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.850+0000 7f820ffff700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f81ec05e2d0 0x7f81ec0606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:13.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.852+0000 7f820ffff700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f81ec05e2d0 0x7f81ec0606f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:13.862 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.854+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 <== osd.4 v2:172.21.15.5:6808/20156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f81ec060d70 con 0x7f81ec05e2d0 2020-11-20T12:22:13.874 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.871+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 --> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f82101467f0 con 0x7f81ec05e2d0 2020-11-20T12:22:13.875 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8207fff700 1 -- 172.21.15.187:0/376256917 <== osd.4 v2:172.21.15.5:6808/20156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f82101467f0 con 0x7f81ec05e2d0 2020-11-20T12:22:13.876 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:13.876 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T12:22:13.876 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:13.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f81ec05e2d0 msgr2=0x7f81ec0606f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.877 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8214c1e700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.5:6808/20156,v1:172.21.15.5:6810/20156] conn(0x7f81ec05e2d0 0x7f81ec0606f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:13.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ec056d20 msgr2=0x7f81ec0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8214c1e700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f81ec056d20 0x7f81ec0591e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f8200006fd0 tx=0x7f8200008040).stop 2020-11-20T12:22:13.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100b2060 msgr2=0x7f82100b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:13.878 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8214c1e700 1 --2- 172.21.15.187:0/376256917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100b2060 0x7f82100b40c0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7f81f800ea80 tx=0x7f8210068110).stop 2020-11-20T12:22:13.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f820ffff700 1 -- 172.21.15.187:0/376256917 reap_dead start 2020-11-20T12:22:13.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 shutdown_connections 2020-11-20T12:22:13.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.873+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 >> 172.21.15.187:0/376256917 conn(0x7f82100bab30 msgr2=0x7f82100b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:13.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.874+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 shutdown_connections 2020-11-20T12:22:13.879 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:13.874+0000 7f8214c1e700 1 -- 172.21.15.187:0/376256917 wait complete. 2020-11-20T12:22:13.902 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2020-11-20T12:22:14.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.161+0000 7f3846ffe700 1 Processor -- start 2020-11-20T12:22:14.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.161+0000 7f3846ffe700 1 -- start start 2020-11-20T12:22:14.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.162+0000 7f3846ffe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014d270 0x7f38401573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.166 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.162+0000 7f3846ffe700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400658a0 con 0x7f38400b1760 2020-11-20T12:22:14.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.162+0000 7f3846ffe700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3840150390 con 0x7f38400b1380 2020-11-20T12:22:14.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.162+0000 7f3846ffe700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38401511e0 con 0x7f384014d270 2020-11-20T12:22:14.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.163+0000 7f3845ffc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f38400b1380 0x7f3840150ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51786/0 (socket says 172.21.15.187:51786) 2020-11-20T12:22:14.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.163+0000 7f3845ffc700 1 -- 172.21.15.187:0/2028666236 learned_addr learned my addr 172.21.15.187:0/2028666236 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:14.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.163+0000 7f38467fd700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014d270 0x7f38401573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908159491 0 0) 0x7f3840150390 con 0x7f38400b1380 2020-11-20T12:22:14.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3828003240 con 0x7f38400b1380 2020-11-20T12:22:14.175 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2321121414 0 0) 0x7f3828003240 con 0x7f38400b1380 2020-11-20T12:22:14.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014d270 msgr2=0x7f38401573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:14.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014d270 0x7f38401573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.176 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 >> v1:172.21.15.5:6789/0 conn(0x7f38400b1760 legacy=0x7f3840065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3840150630 con 0x7f38400b1380 2020-11-20T12:22:14.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f383c0014e0 con 0x7f38400b1380 2020-11-20T12:22:14.177 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 >> v1:172.21.15.187:6789/0 conn(0x7f38400b1380 legacy=0x7f3840150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3828003730 0x7f3828003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014d270 0x7f38280044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.178 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38280049d0 0x7f3828004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38401511e0 con 0x7f3828003730 2020-11-20T12:22:14.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828004140 con 0x7f38280049d0 2020-11-20T12:22:14.179 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.164+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3828004300 con 0x7f384014d270 2020-11-20T12:22:14.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.165+0000 7f38467fd700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38280049d0 0x7f3828004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.165+0000 7f38467fd700 1 -- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014d270 msgr2=0x7f38280044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:14.180 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.165+0000 7f38467fd700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014d270 0x7f38280044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:14.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.165+0000 7f38467fd700 1 -- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3828003730 msgr2=0x7f3828003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:14.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.165+0000 7f38467fd700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3828003730 0x7f3828003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:14.181 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.165+0000 7f38467fd700 1 -- 172.21.15.187:0/2028666236 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828003240 con 0x7f38280049d0 2020-11-20T12:22:14.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.165+0000 7f38467fd700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38280049d0 0x7f3828004e50 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7f3838009ea0 tx=0x7f38380091d0).ready entity=mon.1 client_cookie=acb54fb2b4341f71 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.166+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380095c0 con 0x7f38280049d0 2020-11-20T12:22:14.182 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.166+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383800eda0 con 0x7f38280049d0 2020-11-20T12:22:14.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.166+0000 7f3844ffa700 1 -- 172.21.15.187:0/2028666236 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838012600 con 0x7f38280049d0 2020-11-20T12:22:14.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.167+0000 7f3846ffe700 1 -- 172.21.15.187:0/2028666236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38280049d0 msgr2=0x7f3828004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.183 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.167+0000 7f3846ffe700 1 --2- 172.21.15.187:0/2028666236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38280049d0 0x7f3828004e50 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7f3838009ea0 tx=0x7f38380091d0).stop 2020-11-20T12:22:14.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.167+0000 7f38467fd700 1 -- 172.21.15.187:0/2028666236 reap_dead start 2020-11-20T12:22:14.184 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.167+0000 7f3846ffe700 1 -- 172.21.15.187:0/2028666236 shutdown_connections 2020-11-20T12:22:14.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.167+0000 7f3846ffe700 1 -- 172.21.15.187:0/2028666236 >> 172.21.15.187:0/2028666236 conn(0x7f38400bab30 msgr2=0x7f38400baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:14.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.167+0000 7f3846ffe700 1 -- 172.21.15.187:0/2028666236 shutdown_connections 2020-11-20T12:22:14.185 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.167+0000 7f3846ffe700 1 -- 172.21.15.187:0/2028666236 wait complete. 2020-11-20T12:22:14.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.168+0000 7f3846ffe700 1 Processor -- start 2020-11-20T12:22:14.186 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.169+0000 7f3846ffe700 1 -- start start 2020-11-20T12:22:14.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.169+0000 7f3846ffe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f384014c1f0 0x7f384009b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.187 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.169+0000 7f3846ffe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014c5d0 0x7f38400b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.169+0000 7f3846ffe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38400b3f90 0x7f38400b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.169+0000 7f3846ffe700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384009b630 con 0x7f38400b3f90 2020-11-20T12:22:14.188 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.169+0000 7f3846ffe700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400b3aa0 con 0x7f384014c1f0 2020-11-20T12:22:14.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.169+0000 7f3846ffe700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384009b390 con 0x7f384014c5d0 2020-11-20T12:22:14.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f3845ffc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f384014c1f0 0x7f384009b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.189 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f3845ffc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f384014c1f0 0x7f384009b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55192/0 (socket says 172.21.15.187:55192) 2020-11-20T12:22:14.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f3845ffc700 1 -- 172.21.15.187:0/645299098 learned_addr learned my addr 172.21.15.187:0/645299098 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:14.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f38467fd700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38400b3f90 0x7f38400b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.190 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f3845ffc700 1 -- 172.21.15.187:0/645299098 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014c5d0 msgr2=0x7f38400b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f3845ffc700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014c5d0 0x7f38400b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:14.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f3845ffc700 1 -- 172.21.15.187:0/645299098 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38400b3f90 msgr2=0x7f38400b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.191 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f3845ffc700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38400b3f90 0x7f38400b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.170+0000 7f3845ffc700 1 -- 172.21.15.187:0/645299098 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400b2c20 con 0x7f384014c1f0 2020-11-20T12:22:14.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.171+0000 7f3845ffc700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f384014c1f0 0x7f384009b810 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7f383c00cac0 tx=0x7f383c001a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.192 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.171+0000 7f382dffb700 1 -- 172.21.15.187:0/645299098 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c007170 con 0x7f384014c1f0 2020-11-20T12:22:14.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.171+0000 7f382dffb700 1 -- 172.21.15.187:0/645299098 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383c0072f0 con 0x7f384014c1f0 2020-11-20T12:22:14.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.171+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384009b040 con 0x7f384014c1f0 2020-11-20T12:22:14.193 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.171+0000 7f382dffb700 1 -- 172.21.15.187:0/645299098 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c004a20 con 0x7f384014c1f0 2020-11-20T12:22:14.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.171+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384009bf40 con 0x7f384014c1f0 2020-11-20T12:22:14.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.172+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384009b1d0 con 0x7f384014c1f0 2020-11-20T12:22:14.194 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.173+0000 7f382dffb700 1 -- 172.21.15.187:0/645299098 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f383c010070 con 0x7f384014c1f0 2020-11-20T12:22:14.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.174+0000 7f382dffb700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3818056d20 0x7f38180591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.196 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.174+0000 7f382dffb700 1 -- 172.21.15.187:0/645299098 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f383c06fb10 con 0x7f384014c1f0 2020-11-20T12:22:14.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.174+0000 7f38457fb700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3818056d20 0x7f38180591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.175+0000 7f38457fb700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3818056d20 0x7f38180591e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f3830000f20 tx=0x7f38300077f0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.197 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.178+0000 7f382dffb700 1 -- 172.21.15.187:0/645299098 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f383c044e30 con 0x7f384014c1f0 2020-11-20T12:22:14.366 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.364+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f38401507f0 con 0x7f3818056d20 2020-11-20T12:22:14.367 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.364+0000 7f382dffb700 1 -- 172.21.15.187:0/645299098 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f38401507f0 con 0x7f3818056d20 2020-11-20T12:22:14.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3818056d20 msgr2=0x7f38180591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3818056d20 0x7f38180591e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f3830000f20 tx=0x7f38300077f0).stop 2020-11-20T12:22:14.369 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f384014c1f0 msgr2=0x7f384009b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f384014c1f0 0x7f384009b810 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7f383c00cac0 tx=0x7f383c001a90).stop 2020-11-20T12:22:14.370 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 shutdown_connections 2020-11-20T12:22:14.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f3818056d20 0x7f38180591e0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38400b3f90 0x7f38400b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f384014c1f0 0x7f384009b810 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.371 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 --2- 172.21.15.187:0/645299098 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f384014c5d0 0x7f38400b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:14.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 >> 172.21.15.187:0/645299098 conn(0x7f38400bab30 msgr2=0x7f384014ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:14.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 shutdown_connections 2020-11-20T12:22:14.372 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.367+0000 7f3846ffe700 1 -- 172.21.15.187:0/645299098 wait complete. 2020-11-20T12:22:14.372 INFO:teuthology.orchestra.run.smithi187.stderr:instructed osd(s) 4 to deep-scrub 2020-11-20T12:22:14.393 INFO:tasks.ceph:Scrubbing osd.5 2020-11-20T12:22:14.394 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2020-11-20T12:22:14.645 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.643+0000 7fe40e35e700 1 Processor -- start 2020-11-20T12:22:14.646 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.643+0000 7fe40e35e700 1 -- start start 2020-11-20T12:22:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.644+0000 7fe40e35e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe4081484d0 0x7fe408061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.644+0000 7fe40e35e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe4080622c0 0x7fe4080b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.644+0000 7fe40e35e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4080b1300 con 0x7fe408066f70 2020-11-20T12:22:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.644+0000 7fe40e35e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe408067730 con 0x7fe4081484d0 2020-11-20T12:22:14.648 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.644+0000 7fe40e35e700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4080b15a0 con 0x7fe4080622c0 2020-11-20T12:22:14.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.645+0000 7fe40cb5b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe4080622c0 0x7fe4080b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.645+0000 7fe40cb5b700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe4080622c0 0x7fe4080b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55200/0 (socket says 172.21.15.187:55200) 2020-11-20T12:22:14.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.645+0000 7fe40cb5b700 1 -- 172.21.15.187:0/124756501 learned_addr learned my addr 172.21.15.187:0/124756501 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:14.650 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.645+0000 7fe40cb5b700 1 -- 172.21.15.187:0/124756501 >> v2:172.21.15.5:3300/0 conn(0x7fe4081484d0 msgr2=0x7fe408061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:14.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.645+0000 7fe40cb5b700 1 --2- 172.21.15.187:0/124756501 >> v2:172.21.15.5:3300/0 conn(0x7fe4081484d0 0x7fe408061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:14.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.645+0000 7fe40cb5b700 1 -- 172.21.15.187:0/124756501 >> v1:172.21.15.5:6789/0 conn(0x7fe408066f70 legacy=0x7fe4080b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:14.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.645+0000 7fe40cb5b700 1 -- 172.21.15.187:0/124756501 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080b1840 con 0x7fe4080622c0 2020-11-20T12:22:14.651 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.646+0000 7fe40cb5b700 1 --2- 172.21.15.187:0/124756501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4080622c0 0x7fe4080b5640 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7fe408068600 tx=0x7fe3f8006eb0).ready entity=mon.1 client_cookie=8b7d7b9c88892b5d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.646+0000 7fe3f7fff700 1 -- 172.21.15.187:0/124756501 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8009890 con 0x7fe4080622c0 2020-11-20T12:22:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.646+0000 7fe3f7fff700 1 -- 172.21.15.187:0/124756501 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f80099f0 con 0x7fe4080622c0 2020-11-20T12:22:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.646+0000 7fe3f7fff700 1 -- 172.21.15.187:0/124756501 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8009d70 con 0x7fe4080622c0 2020-11-20T12:22:14.652 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.646+0000 7fe40e35e700 1 -- 172.21.15.187:0/124756501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4080622c0 msgr2=0x7fe4080b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.647+0000 7fe40e35e700 1 --2- 172.21.15.187:0/124756501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4080622c0 0x7fe4080b5640 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7fe408068600 tx=0x7fe3f8006eb0).stop 2020-11-20T12:22:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.647+0000 7fe40e35e700 1 -- 172.21.15.187:0/124756501 shutdown_connections 2020-11-20T12:22:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.647+0000 7fe40e35e700 1 --2- 172.21.15.187:0/124756501 >> v2:172.21.15.5:3300/0 conn(0x7fe4081484d0 0x7fe408061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:14.653 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.647+0000 7fe40e35e700 1 --2- 172.21.15.187:0/124756501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4080622c0 0x7fe4080b5640 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.647+0000 7fe40e35e700 1 -- 172.21.15.187:0/124756501 >> 172.21.15.187:0/124756501 conn(0x7fe4080bab30 msgr2=0x7fe4080baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.647+0000 7fe40e35e700 1 -- 172.21.15.187:0/124756501 shutdown_connections 2020-11-20T12:22:14.654 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.647+0000 7fe40e35e700 1 -- 172.21.15.187:0/124756501 wait complete. 2020-11-20T12:22:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.648+0000 7fe40e35e700 1 Processor -- start 2020-11-20T12:22:14.657 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.649+0000 7fe40e35e700 1 -- start start 2020-11-20T12:22:14.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.649+0000 7fe40e35e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4080622c0 0x7fe40809ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.649+0000 7fe40e35e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe408066f70 0x7fe4081468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40e35e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4081484d0 0x7fe4081456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40e35e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408144cc0 con 0x7fe408066f70 2020-11-20T12:22:14.658 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40e35e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408144f60 con 0x7fe4081484d0 2020-11-20T12:22:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40e35e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe408145200 con 0x7fe4080622c0 2020-11-20T12:22:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40db5d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4081484d0 0x7fe4081456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40db5d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4081484d0 0x7fe4081456a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55210/0 (socket says 172.21.15.187:55210) 2020-11-20T12:22:14.659 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40db5d700 1 -- 172.21.15.187:0/1337494363 learned_addr learned my addr 172.21.15.187:0/1337494363 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40cb5b700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe408066f70 0x7fe4081468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.650+0000 7fe40d35c700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4080622c0 0x7fe40809ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40db5d700 1 -- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4080622c0 msgr2=0x7fe40809ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40db5d700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4080622c0 0x7fe40809ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40db5d700 1 -- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe408066f70 msgr2=0x7fe4081468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40db5d700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe408066f70 0x7fe4081468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40db5d700 1 -- 172.21.15.187:0/1337494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe408146da0 con 0x7fe4081484d0 2020-11-20T12:22:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40cb5b700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe408066f70 0x7fe4081468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:22:14.661 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40d35c700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe4080622c0 0x7fe40809ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:22:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40db5d700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4081484d0 0x7fe4081456a0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7fe4000096e0 tx=0x7fe400006d40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40000d560 con 0x7fe4081484d0 2020-11-20T12:22:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40000d6e0 con 0x7fe4081484d0 2020-11-20T12:22:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.651+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe408145b90 con 0x7fe4081484d0 2020-11-20T12:22:14.662 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.652+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40000daa0 con 0x7fe4081484d0 2020-11-20T12:22:14.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.652+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4081473b0 con 0x7fe4081484d0 2020-11-20T12:22:14.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.652+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe408144b40 con 0x7fe4081484d0 2020-11-20T12:22:14.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.654+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe40000f030 con 0x7fe4081484d0 2020-11-20T12:22:14.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.654+0000 7fe3f57fa700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe3e0056d20 0x7fe3e00591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.663 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.654+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe400070570 con 0x7fe4081484d0 2020-11-20T12:22:14.664 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.655+0000 7fe3f57fa700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe3e005e2d0 0x7fe3e00606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.655+0000 7fe40d35c700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe3e0056d20 0x7fe3e00591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.655+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3e0060d70 con 0x7fe3e005e2d0 2020-11-20T12:22:14.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.655+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=161) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe400070970 con 0x7fe4081484d0 2020-11-20T12:22:14.665 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.655+0000 7fe40d35c700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe3e0056d20 0x7fe3e00591e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7fe404006fd0 tx=0x7fe404008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.656+0000 7fe40cb5b700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe3e005e2d0 0x7fe3e00606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.657+0000 7fe40cb5b700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe3e005e2d0 0x7fe3e00606f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.666 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.660+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 <== osd.5 v2:172.21.15.5:6809/20155 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fe3e0060d70 con 0x7fe3e005e2d0 2020-11-20T12:22:14.679 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.676+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 --> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe408061a20 con 0x7fe3e005e2d0 2020-11-20T12:22:14.680 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.678+0000 7fe3f57fa700 1 -- 172.21.15.187:0/1337494363 <== osd.5 v2:172.21.15.5:6809/20155 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe408061a20 con 0x7fe3e005e2d0 2020-11-20T12:22:14.680 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:14.681 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T12:22:14.681 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:14.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.678+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe3e005e2d0 msgr2=0x7fe3e00606f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.678+0000 7fe40e35e700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.5:6809/20155,v1:172.21.15.5:6811/20155] conn(0x7fe3e005e2d0 0x7fe3e00606f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.682 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.678+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe3e0056d20 msgr2=0x7fe3e00591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.678+0000 7fe40e35e700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fe3e0056d20 0x7fe3e00591e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7fe404006fd0 tx=0x7fe404008040).stop 2020-11-20T12:22:14.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.678+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4081484d0 msgr2=0x7fe4081456a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.678+0000 7fe40e35e700 1 --2- 172.21.15.187:0/1337494363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4081484d0 0x7fe4081456a0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7fe4000096e0 tx=0x7fe400006d40).stop 2020-11-20T12:22:14.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.679+0000 7fe40db5d700 1 -- 172.21.15.187:0/1337494363 reap_dead start 2020-11-20T12:22:14.683 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.679+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 shutdown_connections 2020-11-20T12:22:14.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.679+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 >> 172.21.15.187:0/1337494363 conn(0x7fe4080bab30 msgr2=0x7fe408142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:14.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.679+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 shutdown_connections 2020-11-20T12:22:14.684 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.679+0000 7fe40e35e700 1 -- 172.21.15.187:0/1337494363 wait complete. 2020-11-20T12:22:14.704 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2020-11-20T12:22:14.964 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.962+0000 7f92d207f700 1 Processor -- start 2020-11-20T12:22:14.965 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.962+0000 7f92d207f700 1 -- start start 2020-11-20T12:22:14.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.964+0000 7f92d207f700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f92cc1484d0 0x7f92cc061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.970 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.964+0000 7f92d207f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc0622c0 0x7f92cc0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.964+0000 7f92d207f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0b1300 con 0x7f92cc066f70 2020-11-20T12:22:14.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.964+0000 7f92d207f700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc067730 con 0x7f92cc1484d0 2020-11-20T12:22:14.971 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.964+0000 7f92d207f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0b15a0 con 0x7f92cc0622c0 2020-11-20T12:22:14.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.964+0000 7f92d107d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f92cc1484d0 0x7f92cc061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.964+0000 7f92d107d700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f92cc1484d0 0x7f92cc061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55216/0 (socket says 172.21.15.187:55216) 2020-11-20T12:22:14.972 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.964+0000 7f92d107d700 1 -- 172.21.15.187:0/1599478525 learned_addr learned my addr 172.21.15.187:0/1599478525 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:14.973 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.965+0000 7f92d107d700 1 -- 172.21.15.187:0/1599478525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc0622c0 msgr2=0x7f92cc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:14.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.965+0000 7f92d087c700 1 --2- 172.21.15.187:0/1599478525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc0622c0 0x7f92cc0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.965+0000 7f92d107d700 1 --2- 172.21.15.187:0/1599478525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc0622c0 0x7f92cc0b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.974 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.965+0000 7f92d107d700 1 -- 172.21.15.187:0/1599478525 >> v1:172.21.15.5:6789/0 conn(0x7f92cc066f70 legacy=0x7f92cc0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.965+0000 7f92d107d700 1 -- 172.21.15.187:0/1599478525 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0b1840 con 0x7f92cc1484d0 2020-11-20T12:22:14.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.965+0000 7f92d087c700 1 --2- 172.21.15.187:0/1599478525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc0622c0 0x7f92cc0b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:14.975 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.965+0000 7f92d107d700 1 --2- 172.21.15.187:0/1599478525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc1484d0 0x7f92cc061dd0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7f92c8006fd0 tx=0x7f92c8006eb0).ready entity=mon.1 client_cookie=d20cad84b4785dbf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.966+0000 7f92bbfff700 1 -- 172.21.15.187:0/1599478525 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c80098d0 con 0x7f92cc1484d0 2020-11-20T12:22:14.976 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.966+0000 7f92bbfff700 1 -- 172.21.15.187:0/1599478525 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c8009a30 con 0x7f92cc1484d0 2020-11-20T12:22:14.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.966+0000 7f92bbfff700 1 -- 172.21.15.187:0/1599478525 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8009db0 con 0x7f92cc1484d0 2020-11-20T12:22:14.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.966+0000 7f92d207f700 1 -- 172.21.15.187:0/1599478525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc1484d0 msgr2=0x7f92cc061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.977 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.966+0000 7f92d207f700 1 --2- 172.21.15.187:0/1599478525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc1484d0 0x7f92cc061dd0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7f92c8006fd0 tx=0x7f92c8006eb0).stop 2020-11-20T12:22:14.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.967+0000 7f92d207f700 1 -- 172.21.15.187:0/1599478525 shutdown_connections 2020-11-20T12:22:14.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.967+0000 7f92d207f700 1 --2- 172.21.15.187:0/1599478525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc1484d0 0x7f92cc061dd0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.978 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.967+0000 7f92d207f700 1 --2- 172.21.15.187:0/1599478525 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc0622c0 0x7f92cc0b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.967+0000 7f92d207f700 1 -- 172.21.15.187:0/1599478525 >> 172.21.15.187:0/1599478525 conn(0x7f92cc0bab30 msgr2=0x7f92cc0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:14.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.967+0000 7f92d207f700 1 -- 172.21.15.187:0/1599478525 shutdown_connections 2020-11-20T12:22:14.979 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.967+0000 7f92d207f700 1 -- 172.21.15.187:0/1599478525 wait complete. 2020-11-20T12:22:14.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.968+0000 7f92d207f700 1 Processor -- start 2020-11-20T12:22:14.984 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.968+0000 7f92d207f700 1 -- start start 2020-11-20T12:22:14.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.969+0000 7f92d207f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92cc0622c0 0x7f92cc09c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.985 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.969+0000 7f92d207f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc066f70 0x7f92cc09efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.986 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.969+0000 7f92d207f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc1484d0 0x7f92cc09aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.969+0000 7f92d207f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc09eb40 con 0x7f92cc0622c0 2020-11-20T12:22:14.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.969+0000 7f92d207f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc13e670 con 0x7f92cc066f70 2020-11-20T12:22:14.987 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.969+0000 7f92d207f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc13cfd0 con 0x7f92cc1484d0 2020-11-20T12:22:14.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc066f70 0x7f92cc09efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc066f70 0x7f92cc09efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55224/0 (socket says 172.21.15.187:55224) 2020-11-20T12:22:14.988 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 -- 172.21.15.187:0/2061252616 learned_addr learned my addr 172.21.15.187:0/2061252616 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:14.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d107d700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92cc0622c0 0x7f92cc09c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 -- 172.21.15.187:0/2061252616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc1484d0 msgr2=0x7f92cc09aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.989 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc1484d0 0x7f92cc09aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:14.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 -- 172.21.15.187:0/2061252616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92cc0622c0 msgr2=0x7f92cc09c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:14.990 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92cc0622c0 0x7f92cc09c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:14.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 -- 172.21.15.187:0/2061252616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc09c650 con 0x7f92cc066f70 2020-11-20T12:22:14.991 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.970+0000 7f92d087c700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc066f70 0x7f92cc09efa0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7f92bc00cb50 tx=0x7f92bc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.971+0000 7f92b97fa700 1 -- 172.21.15.187:0/2061252616 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc0077d0 con 0x7f92cc066f70 2020-11-20T12:22:14.992 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.971+0000 7f92b97fa700 1 -- 172.21.15.187:0/2061252616 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc007930 con 0x7f92cc066f70 2020-11-20T12:22:14.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.971+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc09f490 con 0x7f92cc066f70 2020-11-20T12:22:14.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.971+0000 7f92b97fa700 1 -- 172.21.15.187:0/2061252616 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc007cb0 con 0x7f92cc066f70 2020-11-20T12:22:14.993 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.971+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc0a0010 con 0x7f92cc066f70 2020-11-20T12:22:14.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.972+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc061ce0 con 0x7f92cc066f70 2020-11-20T12:22:14.994 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.973+0000 7f92b97fa700 1 -- 172.21.15.187:0/2061252616 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f92bc011620 con 0x7f92cc066f70 2020-11-20T12:22:14.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.973+0000 7f92b97fa700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92a4056d20 0x7f92a40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:14.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.973+0000 7f92b97fa700 1 -- 172.21.15.187:0/2061252616 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f92bc0700e0 con 0x7f92cc066f70 2020-11-20T12:22:14.995 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.974+0000 7f92d107d700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92a4056d20 0x7f92a40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:14.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.974+0000 7f92d107d700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92a4056d20 0x7f92a40591e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f92c800b1b0 tx=0x7f92c8015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:14.996 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:14.977+0000 7f92b97fa700 1 -- 172.21.15.187:0/2061252616 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f92bc0453a0 con 0x7f92cc066f70 2020-11-20T12:22:15.164 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.162+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f92cc09ecd0 con 0x7f92a4056d20 2020-11-20T12:22:15.165 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.162+0000 7f92b97fa700 1 -- 172.21.15.187:0/2061252616 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f92cc09ecd0 con 0x7f92a4056d20 2020-11-20T12:22:15.167 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92a4056d20 msgr2=0x7f92a40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92a4056d20 0x7f92a40591e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f92c800b1b0 tx=0x7f92c8015000).stop 2020-11-20T12:22:15.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc066f70 msgr2=0x7f92cc09efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc066f70 0x7f92cc09efa0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7f92bc00cb50 tx=0x7f92bc008040).stop 2020-11-20T12:22:15.168 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 shutdown_connections 2020-11-20T12:22:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f92a4056d20 0x7f92a40591e0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92cc0622c0 0x7f92cc09c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92cc1484d0 0x7f92cc09aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 --2- 172.21.15.187:0/2061252616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92cc066f70 0x7f92cc09efa0 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.169 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 >> 172.21.15.187:0/2061252616 conn(0x7f92cc0bab30 msgr2=0x7f92cc0b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:15.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 shutdown_connections 2020-11-20T12:22:15.170 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.165+0000 7f92d207f700 1 -- 172.21.15.187:0/2061252616 wait complete. 2020-11-20T12:22:15.170 INFO:teuthology.orchestra.run.smithi187.stderr:instructed osd(s) 5 to deep-scrub 2020-11-20T12:22:15.191 INFO:tasks.ceph:Scrubbing osd.6 2020-11-20T12:22:15.191 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2020-11-20T12:22:15.443 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.440+0000 7f4494bcc700 1 Processor -- start 2020-11-20T12:22:15.454 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.440+0000 7f4494bcc700 1 -- start start 2020-11-20T12:22:15.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.442+0000 7f4494bcc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4490065570 0x7f449009d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.442+0000 7f4494bcc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44900b13e0 con 0x7f4490065190 2020-11-20T12:22:15.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.442+0000 7f4494bcc700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449009d690 con 0x7f4490065f90 2020-11-20T12:22:15.457 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.442+0000 7f4494bcc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449014ab30 con 0x7f4490065570 2020-11-20T12:22:15.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.443+0000 7f4486ffd700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4490065f90 0x7f449009ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51840/0 (socket says 172.21.15.187:51840) 2020-11-20T12:22:15.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.443+0000 7f4486ffd700 1 -- 172.21.15.187:0/3667402579 learned_addr learned my addr 172.21.15.187:0/3667402579 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:15.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.443+0000 7f448ffff700 1 --2- 172.21.15.187:0/3667402579 >> v2:172.21.15.5:3300/0 conn(0x7f4490065570 0x7f449009d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.444+0000 7f448ffff700 1 -- 172.21.15.187:0/3667402579 >> v1:172.21.15.187:6789/0 conn(0x7f4490065f90 legacy=0x7f449009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.458 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.444+0000 7f448effd700 1 -- 172.21.15.187:0/3667402579 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877323018 0 0) 0x7f449009d690 con 0x7f4490065f90 2020-11-20T12:22:15.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.444+0000 7f448ffff700 1 -- 172.21.15.187:0/3667402579 >> v1:172.21.15.5:6789/0 conn(0x7f4490065190 legacy=0x7f4490143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.444+0000 7f448ffff700 1 -- 172.21.15.187:0/3667402579 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490062c10 con 0x7f4490065570 2020-11-20T12:22:15.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.444+0000 7f448ffff700 1 --2- 172.21.15.187:0/3667402579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 0x7f449009d1a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f449009d820 tx=0x7f4488006df0).ready entity=mon.0 client_cookie=187132a77da4b5a4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:15.459 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.445+0000 7f448effd700 1 -- 172.21.15.187:0/3667402579 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4488009840 con 0x7f4490065570 2020-11-20T12:22:15.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.445+0000 7f448effd700 1 -- 172.21.15.187:0/3667402579 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44880099a0 con 0x7f4490065570 2020-11-20T12:22:15.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.445+0000 7f448effd700 1 -- 172.21.15.187:0/3667402579 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4488009d20 con 0x7f4490065570 2020-11-20T12:22:15.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.446+0000 7f4494bcc700 1 -- 172.21.15.187:0/3667402579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 msgr2=0x7f449009d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.446+0000 7f4494bcc700 1 --2- 172.21.15.187:0/3667402579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 0x7f449009d1a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f449009d820 tx=0x7f4488006df0).stop 2020-11-20T12:22:15.460 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.446+0000 7f4494bcc700 1 -- 172.21.15.187:0/3667402579 shutdown_connections 2020-11-20T12:22:15.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.446+0000 7f4494bcc700 1 --2- 172.21.15.187:0/3667402579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 0x7f449009d1a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.446+0000 7f4494bcc700 1 -- 172.21.15.187:0/3667402579 >> 172.21.15.187:0/3667402579 conn(0x7f44900bab30 msgr2=0x7f44900baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:15.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.446+0000 7f4494bcc700 1 -- 172.21.15.187:0/3667402579 shutdown_connections 2020-11-20T12:22:15.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.446+0000 7f4494bcc700 1 -- 172.21.15.187:0/3667402579 wait complete. 2020-11-20T12:22:15.461 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.448+0000 7f4494bcc700 1 Processor -- start 2020-11-20T12:22:15.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.448+0000 7f4494bcc700 1 -- start start 2020-11-20T12:22:15.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f4494bcc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4490065190 0x7f4490136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f4494bcc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 0x7f4490137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f4494bcc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4490065f90 0x7f4490137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.462 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f4494bcc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4490148c60 con 0x7f4490065570 2020-11-20T12:22:15.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f4494bcc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449009e7a0 con 0x7f4490065190 2020-11-20T12:22:15.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f4494bcc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4490148e50 con 0x7f4490065f90 2020-11-20T12:22:15.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f448f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4490065190 0x7f4490136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.463 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f448f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4490065190 0x7f4490136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55236/0 (socket says 172.21.15.187:55236) 2020-11-20T12:22:15.464 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.449+0000 7f448f7fe700 1 -- 172.21.15.187:0/3748897187 learned_addr learned my addr 172.21.15.187:0/3748897187 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:15.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f448ffff700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4490065f90 0x7f4490137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f448f7fe700 1 -- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4490065f90 msgr2=0x7f4490137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.466 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f4486ffd700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 0x7f4490137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f448f7fe700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4490065f90 0x7f4490137b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f448f7fe700 1 -- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 msgr2=0x7f4490137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f448f7fe700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 0x7f4490137430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.467 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f448f7fe700 1 -- 172.21.15.187:0/3748897187 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44900b2690 con 0x7f4490065190 2020-11-20T12:22:15.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f4486ffd700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4490065570 0x7f4490137430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:15.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.450+0000 7f448f7fe700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4490065190 0x7f4490136f40 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 rx=0x7f448000ca90 tx=0x7f4480007f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:15.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.451+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448000a7f0 con 0x7f4490065190 2020-11-20T12:22:15.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.451+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448000a950 con 0x7f4490065190 2020-11-20T12:22:15.468 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.451+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4490149bb0 con 0x7f4490065190 2020-11-20T12:22:15.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.451+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448000acf0 con 0x7f4490065190 2020-11-20T12:22:15.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.451+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44901381f0 con 0x7f4490065190 2020-11-20T12:22:15.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.452+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f44900b2be0 con 0x7f4490065190 2020-11-20T12:22:15.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.452+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4480019030 con 0x7f4490065190 2020-11-20T12:22:15.469 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.453+0000 7f4487fff700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f446c056d20 0x7f446c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.454+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f448006f990 con 0x7f4490065190 2020-11-20T12:22:15.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.454+0000 7f4486ffd700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f446c056d20 0x7f446c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.454+0000 7f4487fff700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f446c05e2c0 0x7f446c0606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.470 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.454+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f446c060d60 con 0x7f446c05e2c0 2020-11-20T12:22:15.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.454+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=161) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4480074220 con 0x7f4490065190 2020-11-20T12:22:15.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.455+0000 7f4486ffd700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f446c056d20 0x7f446c0591e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f44900a0300 tx=0x7f4478009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:15.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.455+0000 7f448ffff700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f446c05e2c0 0x7f446c0606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.457+0000 7f448ffff700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f446c05e2c0 0x7f446c0606e0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:15.471 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.459+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 <== osd.6 v2:172.21.15.5:6800/20157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f446c060d60 con 0x7f446c05e2c0 2020-11-20T12:22:15.479 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.476+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 --> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4490138470 con 0x7f446c05e2c0 2020-11-20T12:22:15.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.478+0000 7f4487fff700 1 -- 172.21.15.187:0/3748897187 <== osd.6 v2:172.21.15.5:6800/20157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4490138470 con 0x7f446c05e2c0 2020-11-20T12:22:15.485 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f446c05e2c0 msgr2=0x7f446c0606e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.5:6800/20157,v1:172.21.15.5:6801/20157] conn(0x7f446c05e2c0 0x7f446c0606e0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f446c056d20 msgr2=0x7f446c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f446c056d20 0x7f446c0591e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f44900a0300 tx=0x7f4478009040).stop 2020-11-20T12:22:15.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4490065190 msgr2=0x7f4490136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.486 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 --2- 172.21.15.187:0/3748897187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4490065190 0x7f4490136f40 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 rx=0x7f448000ca90 tx=0x7f4480007f90).stop 2020-11-20T12:22:15.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f448ffff700 1 -- 172.21.15.187:0/3748897187 reap_dead start 2020-11-20T12:22:15.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 shutdown_connections 2020-11-20T12:22:15.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 >> 172.21.15.187:0/3748897187 conn(0x7f44900bab30 msgr2=0x7f4490145d70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:15.487 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 shutdown_connections 2020-11-20T12:22:15.488 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.479+0000 7f4494bcc700 1 -- 172.21.15.187:0/3748897187 wait complete. 2020-11-20T12:22:15.488 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:15.488 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T12:22:15.488 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:15.503 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2020-11-20T12:22:15.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.753+0000 7fb534832700 1 Processor -- start 2020-11-20T12:22:15.756 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.754+0000 7fb534832700 1 -- start start 2020-11-20T12:22:15.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.755+0000 7fb534832700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb53009f560 0x7fb530153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.755+0000 7fb534832700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53014d110 con 0x7fb5300bf800 2020-11-20T12:22:15.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.755+0000 7fb534832700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53014cbd0 con 0x7fb5300bfbe0 2020-11-20T12:22:15.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.755+0000 7fb534832700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53014c9d0 con 0x7fb53009f560 2020-11-20T12:22:15.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.756+0000 7fb52f59e700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb53009f560 0x7fb530153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.756+0000 7fb51ffff700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb5300bfbe0 0x7fb5301480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51856/0 (socket says 172.21.15.187:51856) 2020-11-20T12:22:15.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.756+0000 7fb51ffff700 1 -- 172.21.15.187:0/1293991864 learned_addr learned my addr 172.21.15.187:0/1293991864 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:15.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.756+0000 7fb52f59e700 1 -- 172.21.15.187:0/1293991864 >> v1:172.21.15.187:6789/0 conn(0x7fb5300bfbe0 legacy=0x7fb5301480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.756+0000 7fb52f59e700 1 -- 172.21.15.187:0/1293991864 >> v1:172.21.15.5:6789/0 conn(0x7fb5300bf800 legacy=0x7fb53014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:15.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.756+0000 7fb52f59e700 1 -- 172.21.15.187:0/1293991864 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53014ce70 con 0x7fb53009f560 2020-11-20T12:22:15.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.756+0000 7fb52f59e700 1 --2- 172.21.15.187:0/1293991864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb53009f560 0x7fb530153580 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7fb52800b9a0 tx=0x7fb528006df0).ready entity=mon.1 client_cookie=78d2b1273d903019 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:15.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb52e59c700 1 -- 172.21.15.187:0/1293991864 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528009800 con 0x7fb53009f560 2020-11-20T12:22:15.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb52e59c700 1 -- 172.21.15.187:0/1293991864 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb528009960 con 0x7fb53009f560 2020-11-20T12:22:15.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb52e59c700 1 -- 172.21.15.187:0/1293991864 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528009ce0 con 0x7fb53009f560 2020-11-20T12:22:15.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb534832700 1 -- 172.21.15.187:0/1293991864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb53009f560 msgr2=0x7fb530153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb534832700 1 --2- 172.21.15.187:0/1293991864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb53009f560 0x7fb530153580 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7fb52800b9a0 tx=0x7fb528006df0).stop 2020-11-20T12:22:15.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb534832700 1 -- 172.21.15.187:0/1293991864 shutdown_connections 2020-11-20T12:22:15.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb534832700 1 --2- 172.21.15.187:0/1293991864 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb53009f560 0x7fb530153580 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb534832700 1 -- 172.21.15.187:0/1293991864 >> 172.21.15.187:0/1293991864 conn(0x7fb530068370 msgr2=0x7fb5300bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:15.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb534832700 1 -- 172.21.15.187:0/1293991864 shutdown_connections 2020-11-20T12:22:15.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.757+0000 7fb534832700 1 -- 172.21.15.187:0/1293991864 wait complete. 2020-11-20T12:22:15.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.759+0000 7fb534832700 1 Processor -- start 2020-11-20T12:22:15.771 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.759+0000 7fb534832700 1 -- start start 2020-11-20T12:22:15.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb534832700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb53009f560 0x7fb5300b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb534832700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5300bf800 0x7fb5301585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.772 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb534832700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53014bfe0 0x7fb530135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb534832700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb530146200 con 0x7fb53009f560 2020-11-20T12:22:15.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb534832700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301464a0 con 0x7fb5300bf800 2020-11-20T12:22:15.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb534832700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb530146740 con 0x7fb53014bfe0 2020-11-20T12:22:15.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb51ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5300bf800 0x7fb5301585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.773 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb51ffff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5300bf800 0x7fb5301585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55254/0 (socket says 172.21.15.187:55254) 2020-11-20T12:22:15.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.760+0000 7fb51ffff700 1 -- 172.21.15.187:0/1057855661 learned_addr learned my addr 172.21.15.187:0/1057855661 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:15.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb52ed9d700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb53009f560 0x7fb5300b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51ffff700 1 -- 172.21.15.187:0/1057855661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53014bfe0 msgr2=0x7fb530135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51ffff700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53014bfe0 0x7fb530135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:15.774 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51ffff700 1 -- 172.21.15.187:0/1057855661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb53009f560 msgr2=0x7fb5300b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51ffff700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb53009f560 0x7fb5300b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51ffff700 1 -- 172.21.15.187:0/1057855661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53014c460 con 0x7fb5300bf800 2020-11-20T12:22:15.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb52ed9d700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb53009f560 0x7fb5300b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:15.775 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51ffff700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5300bf800 0x7fb5301585e0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7fb52400c710 tx=0x7fb524007c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:15.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51effd700 1 -- 172.21.15.187:0/1057855661 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5240072e0 con 0x7fb5300bf800 2020-11-20T12:22:15.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51effd700 1 -- 172.21.15.187:0/1057855661 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb524007440 con 0x7fb5300bf800 2020-11-20T12:22:15.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb534832700 1 -- 172.21.15.187:0/1057855661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5300b67c0 con 0x7fb5300bf800 2020-11-20T12:22:15.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb51effd700 1 -- 172.21.15.187:0/1057855661 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb524005820 con 0x7fb5300bf800 2020-11-20T12:22:15.776 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.761+0000 7fb534832700 1 -- 172.21.15.187:0/1057855661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb530135b00 con 0x7fb5300bf800 2020-11-20T12:22:15.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.763+0000 7fb51cff9700 1 -- 172.21.15.187:0/1057855661 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb530080050 con 0x7fb5300bf800 2020-11-20T12:22:15.777 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.763+0000 7fb51effd700 1 -- 172.21.15.187:0/1057855661 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb524010070 con 0x7fb5300bf800 2020-11-20T12:22:15.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.764+0000 7fb51effd700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb50c056d20 0x7fb50c0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:15.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.764+0000 7fb51effd700 1 -- 172.21.15.187:0/1057855661 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb52406f540 con 0x7fb5300bf800 2020-11-20T12:22:15.778 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.764+0000 7fb52ed9d700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb50c056d20 0x7fb50c0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:15.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.765+0000 7fb52ed9d700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb50c056d20 0x7fb50c0591e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7fb520002c00 tx=0x7fb520009040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:15.779 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.768+0000 7fb51effd700 1 -- 172.21.15.187:0/1057855661 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb524044970 con 0x7fb5300bf800 2020-11-20T12:22:15.956 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.954+0000 7fb51cff9700 1 -- 172.21.15.187:0/1057855661 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fb530146900 con 0x7fb50c056d20 2020-11-20T12:22:15.957 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.954+0000 7fb51effd700 1 -- 172.21.15.187:0/1057855661 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb530146900 con 0x7fb50c056d20 2020-11-20T12:22:15.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 -- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb50c056d20 msgr2=0x7fb50c0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb50c056d20 0x7fb50c0591e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7fb520002c00 tx=0x7fb520009040).stop 2020-11-20T12:22:15.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 -- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5300bf800 msgr2=0x7fb5301585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:15.960 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5300bf800 0x7fb5301585e0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7fb52400c710 tx=0x7fb524007c60).stop 2020-11-20T12:22:15.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 -- 172.21.15.187:0/1057855661 shutdown_connections 2020-11-20T12:22:15.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fb50c056d20 0x7fb50c0591e0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb53009f560 0x7fb5300b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.961 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb53014bfe0 0x7fb530135610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:15.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 --2- 172.21.15.187:0/1057855661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5300bf800 0x7fb5301585e0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:15.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.957+0000 7fb51cff9700 1 -- 172.21.15.187:0/1057855661 >> 172.21.15.187:0/1057855661 conn(0x7fb530068370 msgr2=0x7fb530141b90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:15.962 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.958+0000 7fb51cff9700 1 -- 172.21.15.187:0/1057855661 shutdown_connections 2020-11-20T12:22:15.963 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:15.958+0000 7fb51cff9700 1 -- 172.21.15.187:0/1057855661 wait complete. 2020-11-20T12:22:15.963 INFO:teuthology.orchestra.run.smithi187.stderr:instructed osd(s) 6 to deep-scrub 2020-11-20T12:22:15.985 INFO:tasks.ceph:Scrubbing osd.7 2020-11-20T12:22:15.985 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2020-11-20T12:22:16.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.233+0000 7f4232762700 1 Processor -- start 2020-11-20T12:22:16.236 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.233+0000 7f4232762700 1 -- start start 2020-11-20T12:22:16.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.234+0000 7f4232762700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f422c1484d0 0x7f422c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.234+0000 7f4232762700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c0622c0 0x7f422c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.234+0000 7f4232762700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f422c066f70 0x7f422c0b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.237 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.235+0000 7f4232762700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c067730 con 0x7f422c066f70 2020-11-20T12:22:16.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.235+0000 7f4232762700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0b1300 con 0x7f422c1484d0 2020-11-20T12:22:16.238 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.235+0000 7f4232762700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0b15a0 con 0x7f422c0622c0 2020-11-20T12:22:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.235+0000 7f4231760700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f422c1484d0 0x7f422c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.235+0000 7f4231760700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f422c1484d0 0x7f422c061dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55260/0 (socket says 172.21.15.187:55260) 2020-11-20T12:22:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.235+0000 7f4231760700 1 -- 172.21.15.187:0/2296447512 learned_addr learned my addr 172.21.15.187:0/2296447512 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.235+0000 7f4230f5f700 1 --2- 172.21.15.187:0/2296447512 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c0622c0 0x7f422c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f4231760700 1 -- 172.21.15.187:0/2296447512 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c0622c0 msgr2=0x7f422c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:16.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f4231f61700 1 --2- 172.21.15.187:0/2296447512 >> v2:172.21.15.5:3300/0 conn(0x7f422c066f70 0x7f422c0b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f4231760700 1 --2- 172.21.15.187:0/2296447512 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c0622c0 0x7f422c0b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f4231760700 1 -- 172.21.15.187:0/2296447512 >> v2:172.21.15.5:3300/0 conn(0x7f422c066f70 msgr2=0x7f422c0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:16.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f4231760700 1 --2- 172.21.15.187:0/2296447512 >> v2:172.21.15.5:3300/0 conn(0x7f422c066f70 0x7f422c0b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f4231760700 1 -- 172.21.15.187:0/2296447512 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c0b1840 con 0x7f422c1484d0 2020-11-20T12:22:16.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f4231f61700 1 --2- 172.21.15.187:0/2296447512 >> v2:172.21.15.5:3300/0 conn(0x7f422c066f70 0x7f422c0b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:16.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f4231760700 1 --2- 172.21.15.187:0/2296447512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 0x7f422c061dd0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7f422c0b85e0 tx=0x7f4228006eb0).ready entity=mon.1 client_cookie=345563a552ad3752 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:16.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f421ffff700 1 -- 172.21.15.187:0/2296447512 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280098d0 con 0x7f422c1484d0 2020-11-20T12:22:16.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.236+0000 7f421ffff700 1 -- 172.21.15.187:0/2296447512 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228009a30 con 0x7f422c1484d0 2020-11-20T12:22:16.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f421ffff700 1 -- 172.21.15.187:0/2296447512 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228009db0 con 0x7f422c1484d0 2020-11-20T12:22:16.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 -- 172.21.15.187:0/2296447512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 msgr2=0x7f422c061dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 --2- 172.21.15.187:0/2296447512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 0x7f422c061dd0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7f422c0b85e0 tx=0x7f4228006eb0).stop 2020-11-20T12:22:16.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 -- 172.21.15.187:0/2296447512 shutdown_connections 2020-11-20T12:22:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 --2- 172.21.15.187:0/2296447512 >> v2:172.21.15.5:3300/0 conn(0x7f422c066f70 0x7f422c0b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 --2- 172.21.15.187:0/2296447512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 0x7f422c061dd0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 --2- 172.21.15.187:0/2296447512 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c0622c0 0x7f422c0b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 -- 172.21.15.187:0/2296447512 >> 172.21.15.187:0/2296447512 conn(0x7f422c0bab30 msgr2=0x7f422c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:16.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 -- 172.21.15.187:0/2296447512 shutdown_connections 2020-11-20T12:22:16.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.237+0000 7f4232762700 1 -- 172.21.15.187:0/2296447512 wait complete. 2020-11-20T12:22:16.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.239+0000 7f4232762700 1 Processor -- start 2020-11-20T12:22:16.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.240+0000 7f4232762700 1 -- start start 2020-11-20T12:22:16.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4232762700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f422c0622c0 0x7f422c1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4232762700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c066f70 0x7f422c09f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.248 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4232762700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 0x7f422c145aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4232762700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c15c2f0 con 0x7f422c0622c0 2020-11-20T12:22:16.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4232762700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c1607c0 con 0x7f422c1484d0 2020-11-20T12:22:16.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4232762700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c09b720 con 0x7f422c066f70 2020-11-20T12:22:16.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4231f61700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 0x7f422c145aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.249 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4231f61700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 0x7f422c145aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55270/0 (socket says 172.21.15.187:55270) 2020-11-20T12:22:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4231f61700 1 -- 172.21.15.187:0/1497938112 learned_addr learned my addr 172.21.15.187:0/1497938112 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4231760700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f422c0622c0 0x7f422c1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.241+0000 7f4230f5f700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c066f70 0x7f422c09f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f4231f61700 1 -- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c066f70 msgr2=0x7f422c09f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.250 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f4231f61700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f422c066f70 0x7f422c09f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f4231f61700 1 -- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f422c0622c0 msgr2=0x7f422c1602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f4231f61700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f422c0622c0 0x7f422c1602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f4231f61700 1 -- 172.21.15.187:0/1497938112 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c09b9c0 con 0x7f422c1484d0 2020-11-20T12:22:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f4231f61700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 0x7f422c145aa0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7f422400cb50 tx=0x7f4224008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:16.251 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422400a990 con 0x7f422c1484d0 2020-11-20T12:22:16.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422400aaf0 con 0x7f422c1484d0 2020-11-20T12:22:16.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422400ae70 con 0x7f422c1484d0 2020-11-20T12:22:16.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c09bc60 con 0x7f422c1484d0 2020-11-20T12:22:16.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.242+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c09fcc0 con 0x7f422c1484d0 2020-11-20T12:22:16.252 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.243+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f422c061a20 con 0x7f422c1484d0 2020-11-20T12:22:16.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.244+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4224011620 con 0x7f422c1484d0 2020-11-20T12:22:16.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.245+0000 7f421d7fa700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4204056d20 0x7f42040591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.245+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4224070480 con 0x7f422c1484d0 2020-11-20T12:22:16.253 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.245+0000 7f421d7fa700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f420405e2c0 0x7f42040606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.245+0000 7f4231760700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4204056d20 0x7f42040591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.245+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4204060d60 con 0x7f420405e2c0 2020-11-20T12:22:16.254 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.245+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=161) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4224072af0 con 0x7f422c1484d0 2020-11-20T12:22:16.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.246+0000 7f4231760700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4204056d20 0x7f42040591e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f4228006ef0 tx=0x7f4228015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:16.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.246+0000 7f4230f5f700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f420405e2c0 0x7f42040606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.247+0000 7f4230f5f700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f420405e2c0 0x7f42040606e0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:16.255 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.250+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 <== osd.7 v2:172.21.15.5:6814/20158 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f4204060d60 con 0x7f420405e2c0 2020-11-20T12:22:16.269 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.266+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 --> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f422c1488b0 con 0x7f420405e2c0 2020-11-20T12:22:16.270 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.267+0000 7f421d7fa700 1 -- 172.21.15.187:0/1497938112 <== osd.7 v2:172.21.15.5:6814/20158 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f422c1488b0 con 0x7f420405e2c0 2020-11-20T12:22:16.270 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2020-11-20T12:22:16.270 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T12:22:16.271 INFO:teuthology.orchestra.run.smithi187.stdout:} 2020-11-20T12:22:16.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f420405e2c0 msgr2=0x7f42040606e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4232762700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.5:6814/20158,v1:172.21.15.5:6817/20158] conn(0x7f420405e2c0 0x7f42040606e0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4204056d20 msgr2=0x7f42040591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.272 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4232762700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f4204056d20 0x7f42040591e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f4228006ef0 tx=0x7f4228015000).stop 2020-11-20T12:22:16.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 msgr2=0x7f422c145aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4232762700 1 --2- 172.21.15.187:0/1497938112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1484d0 0x7f422c145aa0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7f422400cb50 tx=0x7f4224008040).stop 2020-11-20T12:22:16.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4231f61700 1 -- 172.21.15.187:0/1497938112 reap_dead start 2020-11-20T12:22:16.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 shutdown_connections 2020-11-20T12:22:16.273 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.268+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 >> 172.21.15.187:0/1497938112 conn(0x7f422c0bab30 msgr2=0x7f422c14d640 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:16.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.269+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 shutdown_connections 2020-11-20T12:22:16.274 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.269+0000 7f4232762700 1 -- 172.21.15.187:0/1497938112 wait complete. 2020-11-20T12:22:16.294 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2020-11-20T12:22:16.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.550+0000 7f7b01230700 1 Processor -- start 2020-11-20T12:22:16.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.550+0000 7f7b01230700 1 -- start start 2020-11-20T12:22:16.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.552+0000 7f7b01230700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7afc0666e0 0x7f7afc0b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.552+0000 7f7b01230700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc061e80 con 0x7f7afc067d30 2020-11-20T12:22:16.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.552+0000 7f7b01230700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc0620e0 con 0x7f7afc14c750 2020-11-20T12:22:16.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.552+0000 7f7b01230700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc062c90 con 0x7f7afc0666e0 2020-11-20T12:22:16.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.552+0000 7f7af259c700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7afc0666e0 0x7f7afc0b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.552+0000 7f7afad9d700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7afc14c750 0x7f7afc062580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51882/0 (socket says 172.21.15.187:51882) 2020-11-20T12:22:16.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.552+0000 7f7afad9d700 1 -- 172.21.15.187:0/3223010228 learned_addr learned my addr 172.21.15.187:0/3223010228 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:16.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.553+0000 7f7af259c700 1 -- 172.21.15.187:0/3223010228 >> v1:172.21.15.187:6789/0 conn(0x7f7afc14c750 legacy=0x7f7afc062580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.557 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.553+0000 7f7af259c700 1 -- 172.21.15.187:0/3223010228 >> v1:172.21.15.5:6789/0 conn(0x7f7afc067d30 legacy=0x7f7afc1503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:16.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.553+0000 7f7af259c700 1 -- 172.21.15.187:0/3223010228 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0b1500 con 0x7f7afc0666e0 2020-11-20T12:22:16.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.553+0000 7f7af259c700 1 --2- 172.21.15.187:0/3223010228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc0b0c40 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7f7ae400b990 tx=0x7f7ae4006d90).ready entity=mon.1 client_cookie=887df30967c874cc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:16.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.553+0000 7f7afa59c700 1 -- 172.21.15.187:0/3223010228 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4006030 con 0x7f7afc0666e0 2020-11-20T12:22:16.558 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.554+0000 7f7afa59c700 1 -- 172.21.15.187:0/3223010228 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae4006190 con 0x7f7afc0666e0 2020-11-20T12:22:16.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.554+0000 7f7afa59c700 1 -- 172.21.15.187:0/3223010228 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4006510 con 0x7f7afc0666e0 2020-11-20T12:22:16.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.554+0000 7f7b01230700 1 -- 172.21.15.187:0/3223010228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 msgr2=0x7f7afc0b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.554+0000 7f7b01230700 1 --2- 172.21.15.187:0/3223010228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc0b0c40 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7f7ae400b990 tx=0x7f7ae4006d90).stop 2020-11-20T12:22:16.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.554+0000 7f7b01230700 1 -- 172.21.15.187:0/3223010228 shutdown_connections 2020-11-20T12:22:16.559 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.554+0000 7f7b01230700 1 --2- 172.21.15.187:0/3223010228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc0b0c40 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.554+0000 7f7b01230700 1 -- 172.21.15.187:0/3223010228 >> 172.21.15.187:0/3223010228 conn(0x7f7afc0bacd0 msgr2=0x7f7afc0bb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:16.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.554+0000 7f7b01230700 1 -- 172.21.15.187:0/3223010228 shutdown_connections 2020-11-20T12:22:16.560 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.555+0000 7f7b01230700 1 -- 172.21.15.187:0/3223010228 wait complete. 2020-11-20T12:22:16.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.556+0000 7f7b01230700 1 Processor -- start 2020-11-20T12:22:16.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.557+0000 7f7b01230700 1 -- start start 2020-11-20T12:22:16.562 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7b01230700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc147bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7b01230700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc067d30 0x7f7afc0b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7b01230700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc14c750 0x7f7afc0b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7b01230700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc1480b0 con 0x7f7afc14c750 2020-11-20T12:22:16.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7b01230700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b4d00 con 0x7f7afc0666e0 2020-11-20T12:22:16.563 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7b01230700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b4a60 con 0x7f7afc067d30 2020-11-20T12:22:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7afad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc147bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7afad9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc147bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55282/0 (socket says 172.21.15.187:55282) 2020-11-20T12:22:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.558+0000 7f7afad9d700 1 -- 172.21.15.187:0/3074654722 learned_addr learned my addr 172.21.15.187:0/3074654722 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7af259c700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc067d30 0x7f7afc0b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7afad9d700 1 -- 172.21.15.187:0/3074654722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc067d30 msgr2=0x7f7afc0b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7afad9d700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc067d30 0x7f7afc0b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7afad9d700 1 -- 172.21.15.187:0/3074654722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc14c750 msgr2=0x7f7afc0b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7afad9d700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc14c750 0x7f7afc0b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:16.565 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7afad9d700 1 -- 172.21.15.187:0/3074654722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc09ea90 con 0x7f7afc0666e0 2020-11-20T12:22:16.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7afad9d700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc147bc0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f7aec00c710 tx=0x7f7aec007c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:16.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7af37fe700 1 -- 172.21.15.187:0/3074654722 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0072e0 con 0x7f7afc0666e0 2020-11-20T12:22:16.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7af37fe700 1 -- 172.21.15.187:0/3074654722 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec007440 con 0x7f7afc0666e0 2020-11-20T12:22:16.566 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc0b3370 con 0x7f7afc0666e0 2020-11-20T12:22:16.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7af37fe700 1 -- 172.21.15.187:0/3074654722 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec005820 con 0x7f7afc0666e0 2020-11-20T12:22:16.567 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.559+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc0b4790 con 0x7f7afc0666e0 2020-11-20T12:22:16.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.561+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc09ec20 con 0x7f7afc0666e0 2020-11-20T12:22:16.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.561+0000 7f7af37fe700 1 -- 172.21.15.187:0/3074654722 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7aec010070 con 0x7f7afc0666e0 2020-11-20T12:22:16.568 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.562+0000 7f7af37fe700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad8056d20 0x7f7ad80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:16.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.562+0000 7f7af37fe700 1 -- 172.21.15.187:0/3074654722 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7aec06f570 con 0x7f7afc0666e0 2020-11-20T12:22:16.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.562+0000 7f7af259c700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad8056d20 0x7f7ad80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:16.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.563+0000 7f7af259c700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad8056d20 0x7f7ad80591e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f7afc066ae0 tx=0x7f7ae4015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:16.569 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.566+0000 7f7af37fe700 1 -- 172.21.15.187:0/3074654722 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7aec044a50 con 0x7f7afc0666e0 2020-11-20T12:22:16.758 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.755+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f7afc0b3500 con 0x7f7ad8056d20 2020-11-20T12:22:16.759 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.756+0000 7f7af37fe700 1 -- 172.21.15.187:0/3074654722 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7afc0b3500 con 0x7f7ad8056d20 2020-11-20T12:22:16.762 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad8056d20 msgr2=0x7f7ad80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad8056d20 0x7f7ad80591e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f7afc066ae0 tx=0x7f7ae4015000).stop 2020-11-20T12:22:16.763 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 msgr2=0x7f7afc147bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:16.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc147bc0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f7aec00c710 tx=0x7f7aec007c60).stop 2020-11-20T12:22:16.764 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 shutdown_connections 2020-11-20T12:22:16.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f7ad8056d20 0x7f7ad80591e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.766 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7afc14c750 0x7f7afc0b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:16.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7afc0666e0 0x7f7afc147bc0 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.767 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 --2- 172.21.15.187:0/3074654722 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7afc067d30 0x7f7afc0b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:16.768 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 >> 172.21.15.187:0/3074654722 conn(0x7f7afc0bacd0 msgr2=0x7f7afc0bb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:16.769 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 shutdown_connections 2020-11-20T12:22:16.770 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:16.759+0000 7f7b01230700 1 -- 172.21.15.187:0/3074654722 wait complete. 2020-11-20T12:22:16.772 INFO:teuthology.orchestra.run.smithi187.stderr:instructed osd(s) 7 to deep-scrub 2020-11-20T12:22:16.785 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:22:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.028+0000 7fcb114f4700 1 Processor -- start 2020-11-20T12:22:17.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.029+0000 7fcb114f4700 1 -- start start 2020-11-20T12:22:17.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.030+0000 7fcb114f4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcb0c1484d0 0x7fcb0c061dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.030+0000 7fcb114f4700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcb0c066f70 0x7fcb0c0b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:17.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.030+0000 7fcb114f4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c0b1300 con 0x7fcb0c0622c0 2020-11-20T12:22:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.030+0000 7fcb114f4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0c067730 con 0x7fcb0c1484d0 2020-11-20T12:22:17.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.030+0000 7fcb114f4700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0c0b15a0 con 0x7fcb0c066f70 2020-11-20T12:22:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.030+0000 7fcb10cf3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcb0c066f70 0x7fcb0c0b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:17.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.031+0000 7fcb10cf3700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcb0c066f70 0x7fcb0c0b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55294/0 (socket says 172.21.15.187:55294) 2020-11-20T12:22:17.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.031+0000 7fcb10cf3700 1 -- 172.21.15.187:0/2632848412 learned_addr learned my addr 172.21.15.187:0/2632848412 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:17.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.031+0000 7fcb0bfff700 1 --2- 172.21.15.187:0/2632848412 >> v2:172.21.15.5:3300/0 conn(0x7fcb0c1484d0 0x7fcb0c061dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:17.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.031+0000 7fcb10cf3700 1 -- 172.21.15.187:0/2632848412 >> v2:172.21.15.5:3300/0 conn(0x7fcb0c1484d0 msgr2=0x7fcb0c061dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:17.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.031+0000 7fcb10cf3700 1 --2- 172.21.15.187:0/2632848412 >> v2:172.21.15.5:3300/0 conn(0x7fcb0c1484d0 0x7fcb0c061dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.031+0000 7fcb10cf3700 1 -- 172.21.15.187:0/2632848412 >> v1:172.21.15.5:6789/0 conn(0x7fcb0c0622c0 legacy=0x7fcb0c0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:17.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.032+0000 7fcb10cf3700 1 -- 172.21.15.187:0/2632848412 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0b1840 con 0x7fcb0c066f70 2020-11-20T12:22:17.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.032+0000 7fcb0bfff700 1 --2- 172.21.15.187:0/2632848412 >> v2:172.21.15.5:3300/0 conn(0x7fcb0c1484d0 0x7fcb0c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:17.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.032+0000 7fcb10cf3700 1 --2- 172.21.15.187:0/2632848412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c066f70 0x7fcb0c0b5d50 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7fcb0c045000 tx=0x7fcb04009290).ready entity=mon.1 client_cookie=c2b7dd775f7d419e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.032+0000 7fcb0b7fe700 1 -- 172.21.15.187:0/2632848412 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400bbc0 con 0x7fcb0c066f70 2020-11-20T12:22:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.032+0000 7fcb0b7fe700 1 -- 172.21.15.187:0/2632848412 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0400bd20 con 0x7fcb0c066f70 2020-11-20T12:22:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.032+0000 7fcb0b7fe700 1 -- 172.21.15.187:0/2632848412 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04011610 con 0x7fcb0c066f70 2020-11-20T12:22:17.041 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.033+0000 7fcb114f4700 1 -- 172.21.15.187:0/2632848412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c066f70 msgr2=0x7fcb0c0b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.033+0000 7fcb114f4700 1 --2- 172.21.15.187:0/2632848412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c066f70 0x7fcb0c0b5d50 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7fcb0c045000 tx=0x7fcb04009290).stop 2020-11-20T12:22:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.033+0000 7fcb114f4700 1 -- 172.21.15.187:0/2632848412 shutdown_connections 2020-11-20T12:22:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.033+0000 7fcb114f4700 1 --2- 172.21.15.187:0/2632848412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c066f70 0x7fcb0c0b5d50 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.033+0000 7fcb114f4700 1 --2- 172.21.15.187:0/2632848412 >> v2:172.21.15.5:3300/0 conn(0x7fcb0c1484d0 0x7fcb0c061dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.033+0000 7fcb114f4700 1 -- 172.21.15.187:0/2632848412 >> 172.21.15.187:0/2632848412 conn(0x7fcb0c0bab30 msgr2=0x7fcb0c0baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.033+0000 7fcb114f4700 1 -- 172.21.15.187:0/2632848412 shutdown_connections 2020-11-20T12:22:17.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.033+0000 7fcb114f4700 1 -- 172.21.15.187:0/2632848412 wait complete. 2020-11-20T12:22:17.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.035+0000 7fcb114f4700 1 Processor -- start 2020-11-20T12:22:17.055 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.035+0000 7fcb114f4700 1 -- start start 2020-11-20T12:22:17.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.036+0000 7fcb114f4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb0c0622c0 0x7fcb0c09c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:17.056 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.036+0000 7fcb114f4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb0c066f70 0x7fcb0c09a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:17.057 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.036+0000 7fcb114f4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c1484d0 0x7fcb0c09ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:17.058 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.036+0000 7fcb114f4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0b4800 con 0x7fcb0c066f70 2020-11-20T12:22:17.059 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.036+0000 7fcb114f4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0b25d0 con 0x7fcb0c1484d0 2020-11-20T12:22:17.061 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.036+0000 7fcb114f4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c09aeb0 con 0x7fcb0c0622c0 2020-11-20T12:22:17.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb10cf3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c1484d0 0x7fcb0c09ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:17.063 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb10cf3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c1484d0 0x7fcb0c09ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55300/0 (socket says 172.21.15.187:55300) 2020-11-20T12:22:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb10cf3700 1 -- 172.21.15.187:0/1555033343 learned_addr learned my addr 172.21.15.187:0/1555033343 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb037fe700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb0c066f70 0x7fcb0c09a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb0bfff700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb0c0622c0 0x7fcb0c09c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb10cf3700 1 -- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb0c0622c0 msgr2=0x7fcb0c09c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:17.064 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb10cf3700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb0c0622c0 0x7fcb0c09c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb10cf3700 1 -- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb0c066f70 msgr2=0x7fcb0c09a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb10cf3700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb0c066f70 0x7fcb0c09a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb10cf3700 1 -- 172.21.15.187:0/1555033343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0b4000 con 0x7fcb0c1484d0 2020-11-20T12:22:17.065 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.037+0000 7fcb0bfff700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb0c0622c0 0x7fcb0c09c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.038+0000 7fcb10cf3700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c1484d0 0x7fcb0c09ecd0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7fcb04011000 tx=0x7fcb04016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.038+0000 7fcb08ff9700 1 -- 172.21.15.187:0/1555033343 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400be90 con 0x7fcb0c1484d0 2020-11-20T12:22:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.038+0000 7fcb08ff9700 1 -- 172.21.15.187:0/1555033343 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb04010aa0 con 0x7fcb0c1484d0 2020-11-20T12:22:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.038+0000 7fcb08ff9700 1 -- 172.21.15.187:0/1555033343 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04010e20 con 0x7fcb0c1484d0 2020-11-20T12:22:17.066 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.038+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0c09f1c0 con 0x7fcb0c1484d0 2020-11-20T12:22:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.038+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c0b45a0 con 0x7fcb0c1484d0 2020-11-20T12:22:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.039+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c0b4190 con 0x7fcb0c1484d0 2020-11-20T12:22:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.040+0000 7fcb08ff9700 1 -- 172.21.15.187:0/1555033343 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcb04017070 con 0x7fcb0c1484d0 2020-11-20T12:22:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.040+0000 7fcb08ff9700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcae8056d20 0x7fcae80591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.040+0000 7fcb08ff9700 1 -- 172.21.15.187:0/1555033343 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcb04072250 con 0x7fcb0c1484d0 2020-11-20T12:22:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.041+0000 7fcb0bfff700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcae8056d20 0x7fcae80591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.041+0000 7fcb0bfff700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcae8056d20 0x7fcae80591e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fcafc00b7d0 tx=0x7fcafc008040).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.044+0000 7fcb08ff9700 1 -- 172.21.15.187:0/1555033343 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcb04047680 con 0x7fcb0c1484d0 2020-11-20T12:22:17.231 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.228+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb0c0b2450 con 0x7fcae8056d20 2020-11-20T12:22:17.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.238+0000 7fcb08ff9700 1 -- 172.21.15.187:0/1555033343 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+104705 (secure 0 0 0) 0x7fcafc00c040 con 0x7fcae8056d20 2020-11-20T12:22:17.243 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:17.244 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcae8056d20 msgr2=0x7fcae80591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:17.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcae8056d20 0x7fcae80591e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fcafc00b7d0 tx=0x7fcafc008040).stop 2020-11-20T12:22:17.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c1484d0 msgr2=0x7fcb0c09ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:17.245 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c1484d0 0x7fcb0c09ecd0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7fcb04011000 tx=0x7fcb04016000).stop 2020-11-20T12:22:17.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 shutdown_connections 2020-11-20T12:22:17.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7fcae8056d20 0x7fcae80591e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0c1484d0 0x7fcb0c09ecd0 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.246 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb0c0622c0 0x7fcb0c09c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 --2- 172.21.15.187:0/1555033343 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb0c066f70 0x7fcb0c09a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:17.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 >> 172.21.15.187:0/1555033343 conn(0x7fcb0c0bab30 msgr2=0x7fcb0c142a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:17.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 shutdown_connections 2020-11-20T12:22:17.247 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:17.242+0000 7fcb114f4700 1 -- 172.21.15.187:0/1555033343 wait complete. 2020-11-20T12:22:17.248 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2020-11-20T12:22:17.267 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2020-11-20T12:22:15.847419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62081,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":8194,"num_write":148,"num_write_kb":16514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":214068,"kb_used_data":16180,"kb_used_omap":0,"kb_used_meta":197824,"kb_avail":754760652,"statfs":{"total":773094113280,"available":772874907648,"internally_reserved":0,"allocated":16568320,"data_stored":7789991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":202571776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.290096"},"pg_stats":[{"pgid":"32.4","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.563482+0000","last_change":"2020-11-20T12:21:56.934847+0000","last_active":"2020-11-20T12:22:04.563482+0000","last_peered":"2020-11-20T12:22:04.563482+0000","last_clean":"2020-11-20T12:22:04.563482+0000","last_became_active":"2020-11-20T12:21:22.693729+0000","last_became_peered":"2020-11-20T12:21:22.693729+0000","last_unstale":"2020-11-20T12:22:04.563482+0000","last_undegraded":"2020-11-20T12:22:04.563482+0000","last_fullsized":"2020-11-20T12:22:04.563482+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.826440+0000","last_change":"2020-11-20T12:21:57.112403+0000","last_active":"2020-11-20T12:22:04.826440+0000","last_peered":"2020-11-20T12:22:04.826440+0000","last_clean":"2020-11-20T12:22:04.826440+0000","last_became_active":"2020-11-20T12:21:22.685927+0000","last_became_peered":"2020-11-20T12:21:22.685927+0000","last_unstale":"2020-11-20T12:22:04.826440+0000","last_undegraded":"2020-11-20T12:22:04.826440+0000","last_fullsized":"2020-11-20T12:22:04.826440+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2","version":"160'10","reported_seq":"30","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.563406+0000","last_change":"2020-11-20T12:21:56.945182+0000","last_active":"2020-11-20T12:22:04.563406+0000","last_peered":"2020-11-20T12:22:04.563406+0000","last_clean":"2020-11-20T12:22:04.563406+0000","last_became_active":"2020-11-20T12:21:22.693767+0000","last_became_peered":"2020-11-20T12:21:22.693767+0000","last_unstale":"2020-11-20T12:22:04.563406+0000","last_undegraded":"2020-11-20T12:22:04.563406+0000","last_fullsized":"2020-11-20T12:22:04.563406+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":6,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.567518+0000","last_change":"2020-11-20T12:21:56.936178+0000","last_active":"2020-11-20T12:22:04.567518+0000","last_peered":"2020-11-20T12:22:04.567518+0000","last_clean":"2020-11-20T12:22:04.567518+0000","last_became_active":"2020-11-20T12:21:22.684395+0000","last_became_peered":"2020-11-20T12:21:22.684395+0000","last_unstale":"2020-11-20T12:22:04.567518+0000","last_undegraded":"2020-11-20T12:22:04.567518+0000","last_fullsized":"2020-11-20T12:22:04.567518+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.563331+0000","last_change":"2020-11-20T12:21:56.937043+0000","last_active":"2020-11-20T12:22:04.563331+0000","last_peered":"2020-11-20T12:22:04.563331+0000","last_clean":"2020-11-20T12:22:04.563331+0000","last_became_active":"2020-11-20T12:21:22.692492+0000","last_became_peered":"2020-11-20T12:21:22.692492+0000","last_unstale":"2020-11-20T12:22:04.563331+0000","last_undegraded":"2020-11-20T12:22:04.563331+0000","last_fullsized":"2020-11-20T12:22:04.563331+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"159'16","reported_seq":"28","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.110707+0000","last_change":"2020-11-20T12:21:22.109003+0000","last_active":"2020-11-20T12:21:57.110707+0000","last_peered":"2020-11-20T12:21:57.110707+0000","last_clean":"2020-11-20T12:21:57.110707+0000","last_became_active":"2020-11-20T12:21:22.108194+0000","last_became_peered":"2020-11-20T12:21:22.108194+0000","last_unstale":"2020-11-20T12:21:57.110707+0000","last_undegraded":"2020-11-20T12:21:57.110707+0000","last_fullsized":"2020-11-20T12:21:57.110707+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"161'13","reported_seq":"28","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:12.028222+0000","last_change":"2020-11-20T12:22:12.028348+0000","last_active":"2020-11-20T12:22:12.028222+0000","last_peered":"2020-11-20T12:22:12.028222+0000","last_clean":"2020-11-20T12:22:12.028222+0000","last_became_active":"2020-11-20T12:21:22.113651+0000","last_became_peered":"2020-11-20T12:21:22.113651+0000","last_unstale":"2020-11-20T12:22:12.028222+0000","last_undegraded":"2020-11-20T12:22:12.028222+0000","last_fullsized":"2020-11-20T12:22:12.028222+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'13","last_scrub_stamp":"2020-11-20T12:22:12.028215+0000","last_deep_scrub":"161'13","last_deep_scrub_stamp":"2020-11-20T12:22:12.028215+0000","last_clean_scrub_stamp":"2020-11-20T12:22:12.028215+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"159'12","reported_seq":"24","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:56.931064+0000","last_change":"2020-11-20T12:21:22.113496+0000","last_active":"2020-11-20T12:21:56.931064+0000","last_peered":"2020-11-20T12:21:56.931064+0000","last_clean":"2020-11-20T12:21:56.931064+0000","last_became_active":"2020-11-20T12:21:22.111838+0000","last_became_peered":"2020-11-20T12:21:22.111838+0000","last_unstale":"2020-11-20T12:21:56.931064+0000","last_undegraded":"2020-11-20T12:21:56.931064+0000","last_fullsized":"2020-11-20T12:21:56.931064+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":184,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.4","version":"160'44","reported_seq":"56","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:22:04.157405+0000","last_change":"2020-11-20T12:21:22.108311+0000","last_active":"2020-11-20T12:22:04.157405+0000","last_peered":"2020-11-20T12:22:04.157405+0000","last_clean":"2020-11-20T12:22:04.157405+0000","last_became_active":"2020-11-20T12:21:22.107044+0000","last_became_peered":"2020-11-20T12:21:22.107044+0000","last_unstale":"2020-11-20T12:22:04.157405+0000","last_undegraded":"2020-11-20T12:22:04.157405+0000","last_fullsized":"2020-11-20T12:22:04.157405+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61078,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"159'12","reported_seq":"24","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.110489+0000","last_change":"2020-11-20T12:21:22.105792+0000","last_active":"2020-11-20T12:21:57.110489+0000","last_peered":"2020-11-20T12:21:57.110489+0000","last_clean":"2020-11-20T12:21:57.110489+0000","last_became_active":"2020-11-20T12:21:22.105399+0000","last_became_peered":"2020-11-20T12:21:22.105399+0000","last_unstale":"2020-11-20T12:21:57.110489+0000","last_undegraded":"2020-11-20T12:21:57.110489+0000","last_fullsized":"2020-11-20T12:21:57.110489+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"158'1","reported_seq":"13","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.111049+0000","last_change":"2020-11-20T12:21:22.098417+0000","last_active":"2020-11-20T12:21:57.111049+0000","last_peered":"2020-11-20T12:21:57.111049+0000","last_clean":"2020-11-20T12:21:57.111049+0000","last_became_active":"2020-11-20T12:21:22.098007+0000","last_became_peered":"2020-11-20T12:21:22.098007+0000","last_unstale":"2020-11-20T12:21:57.111049+0000","last_undegraded":"2020-11-20T12:21:57.111049+0000","last_fullsized":"2020-11-20T12:21:57.111049+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"159'3","reported_seq":"17","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.110530+0000","last_change":"2020-11-20T12:21:22.103551+0000","last_active":"2020-11-20T12:21:57.110530+0000","last_peered":"2020-11-20T12:21:57.110530+0000","last_clean":"2020-11-20T12:21:57.110530+0000","last_became_active":"2020-11-20T12:21:22.102371+0000","last_became_peered":"2020-11-20T12:21:22.102371+0000","last_unstale":"2020-11-20T12:21:57.110530+0000","last_undegraded":"2020-11-20T12:21:57.110530+0000","last_fullsized":"2020-11-20T12:21:57.110530+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":210,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"159'13","reported_seq":"25","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.111610+0000","last_change":"2020-11-20T12:21:22.094228+0000","last_active":"2020-11-20T12:21:57.111610+0000","last_peered":"2020-11-20T12:21:57.111610+0000","last_clean":"2020-11-20T12:21:57.111610+0000","last_became_active":"2020-11-20T12:21:22.093851+0000","last_became_peered":"2020-11-20T12:21:22.093851+0000","last_unstale":"2020-11-20T12:21:57.111610+0000","last_undegraded":"2020-11-20T12:21:57.111610+0000","last_fullsized":"2020-11-20T12:21:57.111610+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:21.051038+0000","last_clean_scrub_stamp":"2020-11-20T12:21:21.051038+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"146","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.111417+0000","last_change":"2020-11-20T12:04:53.296291+0000","last_active":"2020-11-20T12:21:57.111417+0000","last_peered":"2020-11-20T12:21:57.111417+0000","last_clean":"2020-11-20T12:21:57.111417+0000","last_became_active":"2020-11-20T12:04:53.294169+0000","last_became_peered":"2020-11-20T12:04:53.294169+0000","last_unstale":"2020-11-20T12:21:57.111417+0000","last_undegraded":"2020-11-20T12:21:57.111417+0000","last_fullsized":"2020-11-20T12:21:57.111417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"160'8","reported_seq":"28","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:05.215608+0000","last_change":"2020-11-20T12:21:57.123189+0000","last_active":"2020-11-20T12:22:05.215608+0000","last_peered":"2020-11-20T12:22:05.215608+0000","last_clean":"2020-11-20T12:22:05.215608+0000","last_became_active":"2020-11-20T12:21:22.676940+0000","last_became_peered":"2020-11-20T12:21:22.676940+0000","last_unstale":"2020-11-20T12:22:05.215608+0000","last_undegraded":"2020-11-20T12:22:05.215608+0000","last_fullsized":"2020-11-20T12:22:05.215608+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":4,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.112069+0000","last_change":"2020-11-20T12:04:53.294740+0000","last_active":"2020-11-20T12:21:57.112069+0000","last_peered":"2020-11-20T12:21:57.112069+0000","last_clean":"2020-11-20T12:21:57.112069+0000","last_became_active":"2020-11-20T12:04:53.293882+0000","last_became_peered":"2020-11-20T12:04:53.293882+0000","last_unstale":"2020-11-20T12:21:57.112069+0000","last_undegraded":"2020-11-20T12:21:57.112069+0000","last_fullsized":"2020-11-20T12:21:57.112069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"145","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.110499+0000","last_change":"2020-11-20T12:04:53.299007+0000","last_active":"2020-11-20T12:21:57.110499+0000","last_peered":"2020-11-20T12:21:57.110499+0000","last_clean":"2020-11-20T12:21:57.110499+0000","last_became_active":"2020-11-20T12:04:53.297849+0000","last_became_peered":"2020-11-20T12:04:53.297849+0000","last_unstale":"2020-11-20T12:21:57.110499+0000","last_undegraded":"2020-11-20T12:21:57.110499+0000","last_fullsized":"2020-11-20T12:21:57.110499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:05.215761+0000","last_change":"2020-11-20T12:21:57.114223+0000","last_active":"2020-11-20T12:22:05.215761+0000","last_peered":"2020-11-20T12:22:05.215761+0000","last_clean":"2020-11-20T12:22:05.215761+0000","last_became_active":"2020-11-20T12:21:22.675260+0000","last_became_peered":"2020-11-20T12:21:22.675260+0000","last_unstale":"2020-11-20T12:22:05.215761+0000","last_undegraded":"2020-11-20T12:22:05.215761+0000","last_fullsized":"2020-11-20T12:22:05.215761+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.112390+0000","last_change":"2020-11-20T12:04:54.004976+0000","last_active":"2020-11-20T12:21:57.112390+0000","last_peered":"2020-11-20T12:21:57.112390+0000","last_clean":"2020-11-20T12:21:57.112390+0000","last_became_active":"2020-11-20T12:04:54.004416+0000","last_became_peered":"2020-11-20T12:04:54.004416+0000","last_unstale":"2020-11-20T12:21:57.112390+0000","last_undegraded":"2020-11-20T12:21:57.112390+0000","last_fullsized":"2020-11-20T12:21:57.112390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'2","reported_seq":"146","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.110323+0000","last_change":"2020-11-20T12:04:49.157393+0000","last_active":"2020-11-20T12:21:57.110323+0000","last_peered":"2020-11-20T12:21:57.110323+0000","last_clean":"2020-11-20T12:21:57.110323+0000","last_became_active":"2020-11-20T12:04:49.156248+0000","last_became_peered":"2020-11-20T12:04:49.156248+0000","last_unstale":"2020-11-20T12:21:57.110323+0000","last_undegraded":"2020-11-20T12:21:57.110323+0000","last_fullsized":"2020-11-20T12:21:57.110323+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:44.517280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:44.517280+0000","last_clean_scrub_stamp":"2020-11-20T12:04:44.517280+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.7","version":"159'1","reported_seq":"21","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.568502+0000","last_change":"2020-11-20T12:21:56.937079+0000","last_active":"2020-11-20T12:22:04.568502+0000","last_peered":"2020-11-20T12:22:04.568502+0000","last_clean":"2020-11-20T12:22:04.568502+0000","last_became_active":"2020-11-20T12:21:22.697706+0000","last_became_peered":"2020-11-20T12:21:22.697706+0000","last_unstale":"2020-11-20T12:22:04.568502+0000","last_undegraded":"2020-11-20T12:22:04.568502+0000","last_fullsized":"2020-11-20T12:22:04.568502+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.111741+0000","last_change":"2020-11-20T12:04:53.297146+0000","last_active":"2020-11-20T12:21:57.111741+0000","last_peered":"2020-11-20T12:21:57.111741+0000","last_clean":"2020-11-20T12:21:57.111741+0000","last_became_active":"2020-11-20T12:04:53.296376+0000","last_became_peered":"2020-11-20T12:04:53.296376+0000","last_unstale":"2020-11-20T12:21:57.111741+0000","last_undegraded":"2020-11-20T12:21:57.111741+0000","last_fullsized":"2020-11-20T12:21:57.111741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.113106+0000","last_change":"2020-11-20T12:04:53.303192+0000","last_active":"2020-11-20T12:21:57.113106+0000","last_peered":"2020-11-20T12:21:57.113106+0000","last_clean":"2020-11-20T12:21:57.113106+0000","last_became_active":"2020-11-20T12:04:53.302386+0000","last_became_peered":"2020-11-20T12:04:53.302386+0000","last_unstale":"2020-11-20T12:21:57.113106+0000","last_undegraded":"2020-11-20T12:21:57.113106+0000","last_fullsized":"2020-11-20T12:21:57.113106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"146","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:56.931724+0000","last_change":"2020-11-20T12:04:54.003353+0000","last_active":"2020-11-20T12:21:56.931724+0000","last_peered":"2020-11-20T12:21:56.931724+0000","last_clean":"2020-11-20T12:21:56.931724+0000","last_became_active":"2020-11-20T12:04:54.002137+0000","last_became_peered":"2020-11-20T12:04:54.002137+0000","last_unstale":"2020-11-20T12:21:56.931724+0000","last_undegraded":"2020-11-20T12:21:56.931724+0000","last_fullsized":"2020-11-20T12:21:56.931724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"144","reported_epoch":"160","state":"active+clean","last_fresh":"2020-11-20T12:21:57.113362+0000","last_change":"2020-11-20T12:04:54.001880+0000","last_active":"2020-11-20T12:21:57.113362+0000","last_peered":"2020-11-20T12:21:57.113362+0000","last_clean":"2020-11-20T12:21:57.113362+0000","last_became_active":"2020-11-20T12:04:54.001419+0000","last_became_peered":"2020-11-20T12:04:54.001419+0000","last_unstale":"2020-11-20T12:21:57.113362+0000","last_undegraded":"2020-11-20T12:21:57.113362+0000","last_fullsized":"2020-11-20T12:21:57.113362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.257420+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.257420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":10,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":16,"acting":16,"num_store_stats":7},{"poolid":31,"num_pg":8,"stat_sum":{"num_bytes":62081,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":136,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":124162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542355,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20144,"kb_used_data":2088,"kb_used_omap":0,"kb_used_meta":18048,"kb_avail":94351696,"statfs":{"total":96636764160,"available":96616136704,"internally_reserved":0,"allocated":2138112,"data_stored":1020351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18481152},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.3050000000000002,"5min":2.7290000000000001,"15min":2.4750000000000001},"min":{"1min":1.508,"5min":0.63,"15min":0.63},"max":{"1min":3.4500000000000002,"5min":37.622,"15min":37.622},"last":1.637},{"interface":"front","average":{"1min":1.881,"5min":2.7149999999999999,"15min":2.4489999999999998},"min":{"1min":1.2949999999999999,"5min":1.206,"15min":1.206},"max":{"1min":2.8980000000000001,"5min":36.816000000000003,"15min":36.816000000000003},"last":1.857}]},{"osd":1,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":2.5089999999999999,"5min":3.4500000000000002,"15min":3.1259999999999999},"min":{"1min":1.411,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":3.6859999999999999,"5min":42.296999999999997,"15min":42.296999999999997},"last":3.702},{"interface":"front","average":{"1min":2.431,"5min":16.081,"15min":7.0220000000000002},"min":{"1min":1.627,"5min":1.0600000000000001,"15min":1.0600000000000001},"max":{"1min":3.6360000000000001,"5min":633.35299999999995,"15min":633.35299999999995},"last":1.9850000000000001}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.903,"15min":2.798},"min":{"1min":1.861,"5min":1.073,"15min":1.073},"max":{"1min":3.9790000000000001,"5min":7.359,"15min":7.359},"last":2.7290000000000001},{"interface":"front","average":{"1min":2.6800000000000002,"5min":3.2690000000000001,"15min":3.0259999999999998},"min":{"1min":1.7769999999999999,"5min":1.6000000000000001,"15min":1.4259999999999999},"max":{"1min":3.9649999999999999,"5min":36.509,"15min":36.509},"last":2.9279999999999999}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.2410000000000001,"5min":3.762,"15min":3.6000000000000001},"min":{"1min":2.2000000000000002,"5min":1.129,"15min":1.129},"max":{"1min":4.2439999999999998,"5min":37.859000000000002,"15min":37.859000000000002},"last":4.1959999999999997},{"interface":"front","average":{"1min":3.218,"5min":3.3599999999999999,"15min":3.306},"min":{"1min":1.968,"5min":1.516,"15min":1.45},"max":{"1min":4.3730000000000002,"5min":21.952999999999999,"15min":21.952999999999999},"last":3.1890000000000001}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.2320000000000002,"5min":4.0490000000000004,"15min":3.6699999999999999},"min":{"1min":1.411,"5min":1.3160000000000001,"15min":1.2809999999999999},"max":{"1min":4.4640000000000004,"5min":36.979999999999997,"15min":36.979999999999997},"last":4.1020000000000003},{"interface":"front","average":{"1min":3.79,"5min":4.4329999999999998,"15min":4.2199999999999998},"min":{"1min":1.9530000000000001,"5min":1.2649999999999999,"15min":1.2649999999999999},"max":{"1min":4.9729999999999999,"5min":38.177,"15min":38.177},"last":4.8810000000000002}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.8319999999999999,"5min":3.6629999999999998,"15min":3.694},"min":{"1min":2.7080000000000002,"5min":1.351,"15min":1.351},"max":{"1min":4.9950000000000001,"5min":5.2590000000000003,"15min":5.8129999999999997},"last":4.5049999999999999},{"interface":"front","average":{"1min":4.1699999999999999,"5min":4.6589999999999998,"15min":4.4710000000000001},"min":{"1min":3.2450000000000001,"5min":2.1619999999999999,"15min":1.7809999999999999},"max":{"1min":5.3040000000000003,"5min":37.438000000000002,"15min":37.438000000000002},"last":4.6369999999999996}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.9870000000000001,"5min":4.3250000000000002,"15min":4.117},"min":{"1min":2.6589999999999998,"5min":1.5389999999999999,"15min":1.5389999999999999},"max":{"1min":7.1710000000000003,"5min":36.674999999999997,"15min":36.674999999999997},"last":4.585},{"interface":"front","average":{"1min":4.4420000000000002,"5min":4.9039999999999999,"15min":4.7629999999999999},"min":{"1min":2.7589999999999999,"5min":1.401,"15min":1.401},"max":{"1min":6.5069999999999997,"5min":38.027000000000001,"15min":38.027000000000001},"last":5.1429999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542355,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30048,"kb_used_data":2072,"kb_used_omap":0,"kb_used_meta":27968,"kb_avail":94341792,"statfs":{"total":96636764160,"available":96605995008,"internally_reserved":0,"allocated":2121728,"data_stored":1019416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28639232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":2.1659999999999999,"5min":2.2010000000000001,"15min":2.1779999999999999},"min":{"1min":1.052,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":5.0380000000000003,"5min":5.6600000000000001,"15min":5.6600000000000001},"last":2.1949999999999998},{"interface":"front","average":{"1min":2.5670000000000002,"5min":2.3809999999999998,"15min":2.4039999999999999},"min":{"1min":1.6839999999999999,"5min":0.86799999999999999,"15min":0.79500000000000004},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":7.1630000000000003},"last":2.3119999999999998}]},{"osd":1,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":2.9809999999999999,"5min":2.6859999999999999,"15min":2.5800000000000001},"min":{"1min":1.1639999999999999,"5min":0.78800000000000003,"15min":0.70599999999999996},"max":{"1min":12.022,"5min":12.022,"15min":12.022},"last":2.427},{"interface":"front","average":{"1min":3.4239999999999999,"5min":5.3120000000000003,"15min":3.7869999999999999},"min":{"1min":2.274,"5min":1.252,"15min":0.84699999999999998},"max":{"1min":6.3410000000000002,"5min":140.53299999999999,"15min":140.53299999999999},"last":3.278}]},{"osd":2,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":3.5419999999999998,"5min":3.0259999999999998,"15min":3.0539999999999998},"min":{"1min":1.804,"5min":1.3,"15min":1.02},"max":{"1min":12.430999999999999,"5min":12.430999999999999,"15min":12.430999999999999},"last":4.1829999999999998},{"interface":"front","average":{"1min":3.0830000000000002,"5min":3.0739999999999998,"15min":3.04},"min":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.2370000000000001},"max":{"1min":5.6600000000000001,"5min":5.7619999999999996,"15min":7.5140000000000002},"last":3.149}]},{"osd":3,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":3.3879999999999999,"5min":3.3919999999999999,"15min":3.4460000000000002},"min":{"1min":2.2330000000000001,"5min":1.6140000000000001,"15min":1.121},"max":{"1min":5,"5min":7.274,"15min":8.2249999999999996},"last":3.7069999999999999},{"interface":"front","average":{"1min":4.0490000000000004,"5min":3.5419999999999998,"15min":3.5840000000000001},"min":{"1min":2.3679999999999999,"5min":1.105,"15min":1.105},"max":{"1min":13.329000000000001,"5min":13.329000000000001,"15min":13.329000000000001},"last":4.7240000000000002}]},{"osd":4,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":3.8279999999999998,"5min":3.6480000000000001,"15min":3.621},"min":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.0900000000000001},"max":{"1min":11.247,"5min":11.247,"15min":11.247},"last":4.4219999999999997},{"interface":"front","average":{"1min":4.3710000000000004,"5min":3.9350000000000001,"15min":4.0049999999999999},"min":{"1min":3.0640000000000001,"5min":1.349,"15min":1.2809999999999999},"max":{"1min":10.82,"5min":10.82,"15min":11.474},"last":4.4710000000000001}]},{"osd":5,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":4.4059999999999997,"5min":4.1539999999999999,"15min":4.1310000000000002},"min":{"1min":2.7719999999999998,"5min":1.4319999999999999,"15min":1.3400000000000001},"max":{"1min":11.568,"5min":11.568,"15min":14.435},"last":4.9660000000000002},{"interface":"front","average":{"1min":4.0579999999999998,"5min":4.1319999999999997,"15min":4.1600000000000001},"min":{"1min":2.2120000000000002,"5min":1.7789999999999999,"15min":1.599},"max":{"1min":6.8140000000000001,"5min":9.5850000000000009,"15min":22.896000000000001},"last":4.641}]},{"osd":7,"last update":"Fri Nov 20 12:21:12 2020","interfaces":[{"interface":"back","average":{"1min":4.5960000000000001,"5min":4.3529999999999998,"15min":4.3659999999999997},"min":{"1min":3.145,"5min":2.0049999999999999,"15min":1.738},"max":{"1min":7.6139999999999999,"5min":8.2379999999999995,"15min":11.917},"last":4.1159999999999997},{"interface":"front","average":{"1min":4.9329999999999998,"5min":4.4189999999999996,"15min":4.5359999999999996},"min":{"1min":2.9399999999999999,"5min":1.5620000000000001,"15min":1.5069999999999999},"max":{"1min":16.263000000000002,"5min":16.263000000000002,"15min":23.207999999999998},"last":5.1609999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542355,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":30592,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":2056192,"data_stored":958548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31326208},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":1.7669999999999999,"5min":11.311999999999999,"15min":5.0890000000000004},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.78100000000000003},"max":{"1min":2.6640000000000001,"5min":868.745,"15min":868.745},"last":1.9199999999999999},{"interface":"front","average":{"1min":1.9119999999999999,"5min":2.093,"15min":2.0910000000000002},"min":{"1min":0.71799999999999997,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":3.1869999999999998,"5min":7.6580000000000004,"15min":10.042},"last":1.9850000000000001}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.0089999999999999,"5min":2.4540000000000002,"15min":2.6659999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.68100000000000005},"max":{"1min":2.7400000000000002,"5min":20.43,"15min":142.95500000000001},"last":3.165},{"interface":"front","average":{"1min":2.5609999999999999,"5min":12.111000000000001,"15min":6.3460000000000001},"min":{"1min":1.04,"5min":1.04,"15min":0.95999999999999996},"max":{"1min":3.827,"5min":868.82299999999998,"15min":868.82299999999998},"last":2.7069999999999999}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":12.082000000000001,"15min":5.891},"min":{"1min":1.4299999999999999,"5min":1.2669999999999999,"15min":1.0409999999999999},"max":{"1min":3.2719999999999998,"5min":869.04300000000001,"15min":869.04300000000001},"last":3.0830000000000002},{"interface":"front","average":{"1min":2.5339999999999998,"5min":2.9990000000000001,"15min":2.8300000000000001},"min":{"1min":1.4059999999999999,"5min":1.3029999999999999,"15min":0.94799999999999995},"max":{"1min":3.7599999999999998,"5min":11.404,"15min":11.404},"last":3.3610000000000002}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.915,"5min":3.2589999999999999,"15min":3.0169999999999999},"min":{"1min":1.5469999999999999,"5min":1.3580000000000001,"15min":0.96899999999999997},"max":{"1min":4.1870000000000003,"5min":20.577999999999999,"15min":20.577999999999999},"last":2.254},{"interface":"front","average":{"1min":3.077,"5min":12.645,"15min":6.4370000000000003},"min":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.022},"max":{"1min":4.3280000000000003,"5min":869.22299999999996,"15min":869.22299999999996},"last":3.4390000000000001}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.1789999999999998,"5min":3.6589999999999998,"15min":3.5379999999999998},"min":{"1min":1.875,"5min":1.875,"15min":1.2649999999999999},"max":{"1min":5.1319999999999997,"5min":13.064,"15min":13.064},"last":2.79},{"interface":"front","average":{"1min":3.2029999999999998,"5min":4.0309999999999997,"15min":3.5470000000000002},"min":{"1min":1.944,"5min":1.8440000000000001,"15min":1.1299999999999999},"max":{"1min":5.0730000000000004,"5min":62.292000000000002,"15min":62.292000000000002},"last":3.9409999999999998}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.5190000000000001,"5min":13.275,"15min":6.8369999999999997},"min":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.3879999999999999},"max":{"1min":5.0110000000000001,"5min":869.15300000000002,"15min":869.15300000000002},"last":3.8580000000000001},{"interface":"front","average":{"1min":3.2490000000000001,"5min":3.6829999999999998,"15min":3.6989999999999998},"min":{"1min":2.012,"5min":2.012,"15min":1.573},"max":{"1min":4.8550000000000004,"5min":11.545,"15min":17.042999999999999},"last":4.157}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.484,"5min":4.0129999999999999,"15min":3.984},"min":{"1min":1.944,"5min":1.944,"15min":1.411},"max":{"1min":4.8609999999999998,"5min":11.667,"15min":11.667},"last":3.782},{"interface":"front","average":{"1min":3.6099999999999999,"5min":3.9510000000000001,"15min":3.7549999999999999},"min":{"1min":2.0760000000000001,"5min":1.222,"15min":1.222},"max":{"1min":5.3029999999999999,"5min":20.710000000000001,"15min":20.710000000000001},"last":4.4909999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21720,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":19712,"kb_avail":94350120,"statfs":{"total":96636764160,"available":96614522880,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20185088},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":1.845,"5min":2.1640000000000001,"15min":2.0870000000000002},"min":{"1min":0.93200000000000005,"5min":0.91200000000000003,"15min":0.71299999999999997},"max":{"1min":2.7160000000000002,"5min":7.3330000000000002,"15min":10.723000000000001},"last":2.1629999999999998},{"interface":"front","average":{"1min":2.2669999999999999,"5min":2.6509999999999998,"15min":2.5299999999999998},"min":{"1min":1.1140000000000001,"5min":1.0589999999999999,"15min":0.98699999999999999},"max":{"1min":3.7210000000000001,"5min":7.8929999999999998,"15min":7.8929999999999998},"last":2.609}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":1.9370000000000001,"5min":2.4550000000000001,"15min":2.1379999999999999},"min":{"1min":0.75,"5min":0.66400000000000003,"15min":0.66100000000000003},"max":{"1min":3.4980000000000002,"5min":42.066000000000003,"15min":42.066000000000003},"last":2.02},{"interface":"front","average":{"1min":1.9750000000000001,"5min":2.2200000000000002,"15min":2.2269999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":3.2010000000000001,"5min":4.7990000000000004,"15min":4.7990000000000004},"last":4.1520000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.4700000000000002,"5min":3.1299999999999999,"15min":2.8149999999999999},"min":{"1min":1.095,"5min":0.76400000000000001,"15min":0.753},"max":{"1min":3.5310000000000001,"5min":38.067,"15min":38.067},"last":2.7149999999999999},{"interface":"front","average":{"1min":2.7290000000000001,"5min":3.306,"15min":3.0710000000000002},"min":{"1min":1.242,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":4.2089999999999996,"5min":38.463000000000001,"15min":38.463000000000001},"last":2.923}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.827,"5min":3.1219999999999999,"15min":3.1000000000000001},"min":{"1min":1.444,"5min":1.05,"15min":1.05},"max":{"1min":4.4189999999999996,"5min":14.467000000000001,"15min":14.467000000000001},"last":3.6030000000000002},{"interface":"front","average":{"1min":2.8170000000000002,"5min":2.9609999999999999,"15min":3.0339999999999998},"min":{"1min":1.54,"5min":1.1579999999999999,"15min":1.1579999999999999},"max":{"1min":5.0090000000000003,"5min":5.0090000000000003,"15min":5.0679999999999996},"last":3.444}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.222,"5min":3.387,"15min":3.3759999999999999},"min":{"1min":1.623,"5min":1.5589999999999999,"15min":1.4790000000000001},"max":{"1min":4.8929999999999998,"5min":7.8120000000000003,"15min":7.8120000000000003},"last":3.1579999999999999},{"interface":"front","average":{"1min":3.2759999999999998,"5min":3.6520000000000001,"15min":3.6579999999999999},"min":{"1min":1.6599999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"max":{"1min":4.2039999999999997,"5min":6.5780000000000003,"15min":11.58},"last":4.2489999999999997}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.254,"5min":3.444,"15min":3.4580000000000002},"min":{"1min":1.8049999999999999,"5min":1.29,"15min":1.29},"max":{"1min":4.8710000000000004,"5min":6.6230000000000002,"15min":6.6230000000000002},"last":3.835},{"interface":"front","average":{"1min":3.29,"5min":3.4849999999999999,"15min":3.536},"min":{"1min":1.9199999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"max":{"1min":4.4660000000000002,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":3.7320000000000002}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.5329999999999999,"5min":4.0339999999999998,"15min":3.8490000000000002},"min":{"1min":2.161,"5min":1.357,"15min":1.357},"max":{"1min":4.6050000000000004,"5min":37.826999999999998,"15min":37.826999999999998},"last":4.4699999999999998},{"interface":"front","average":{"1min":3.5419999999999998,"5min":3.7850000000000001,"15min":3.738},"min":{"1min":2.1219999999999999,"5min":1.7370000000000001,"15min":1.665},"max":{"1min":4.8200000000000003,"5min":6.7329999999999997,"15min":12.154},"last":4.5890000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":27136,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27787264},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":2.0489999999999999,"5min":1.994,"15min":2.1030000000000002},"min":{"1min":1.232,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":3.7400000000000002,"5min":3.8380000000000001,"15min":14.422000000000001},"last":1.581},{"interface":"front","average":{"1min":2.5499999999999998,"5min":2.4140000000000001,"15min":2.516},"min":{"1min":1.2949999999999999,"5min":0.88100000000000001,"15min":0.871},"max":{"1min":3.472,"5min":4.6429999999999998,"15min":18.148},"last":3.2410000000000001}]},{"osd":1,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":2.6230000000000002,"15min":2.4500000000000002},"min":{"1min":1.587,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":4.0620000000000003,"5min":24.391999999999999,"15min":24.391999999999999},"last":1.9770000000000001},{"interface":"front","average":{"1min":2.1829999999999998,"5min":11.826000000000001,"15min":5.4880000000000004},"min":{"1min":0.91300000000000003,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":3.8879999999999999,"5min":820.03499999999997,"15min":820.03499999999997},"last":3.8119999999999998}]},{"osd":3,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":2.2519999999999998,"5min":2.1269999999999998,"15min":2.6040000000000001},"min":{"1min":1.2569999999999999,"5min":0.97499999999999998,"15min":0.748},"max":{"1min":3.1480000000000001,"5min":4.1050000000000004,"15min":134.77600000000001},"last":2.556},{"interface":"front","average":{"1min":2.9820000000000002,"5min":2.8700000000000001,"15min":2.9340000000000002},"min":{"1min":2.0510000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":3.8370000000000002,"5min":5.601,"15min":10.574999999999999},"last":2.9169999999999998}]},{"osd":4,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":3.3029999999999999,"5min":3.2610000000000001,"15min":3.2400000000000002},"min":{"1min":2.0569999999999999,"5min":1.462,"15min":1.462},"max":{"1min":4.2530000000000001,"5min":6.1589999999999998,"15min":10.132},"last":2.7000000000000002},{"interface":"front","average":{"1min":2.9940000000000002,"5min":3.1059999999999999,"15min":3.0950000000000002},"min":{"1min":1.8480000000000001,"5min":1.377,"15min":1.274},"max":{"1min":3.855,"5min":5.0650000000000004,"15min":11.913},"last":2.758}]},{"osd":5,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":3.1699999999999999,"5min":3.0499999999999998,"15min":3.5529999999999999},"min":{"1min":1.9039999999999999,"5min":1.8069999999999999,"15min":1.554},"max":{"1min":4.3319999999999999,"5min":5.6319999999999997,"15min":141.113},"last":3.016},{"interface":"front","average":{"1min":3.6619999999999999,"5min":3.5640000000000001,"15min":3.593},"min":{"1min":2.774,"5min":1.4379999999999999,"15min":1.4330000000000001},"max":{"1min":4.7199999999999998,"5min":5.8259999999999996,"15min":10.968999999999999},"last":2.4140000000000001}]},{"osd":6,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":3.4969999999999999,"5min":3.423,"15min":3.4950000000000001},"min":{"1min":2.048,"5min":1.3069999999999999,"15min":1.137},"max":{"1min":5.4790000000000001,"5min":5.532,"15min":16.738},"last":4.3959999999999999},{"interface":"front","average":{"1min":3.8069999999999999,"5min":3.4420000000000002,"15min":3.4860000000000002},"min":{"1min":2.7480000000000002,"5min":1.7789999999999999,"15min":1.702},"max":{"1min":4.8810000000000002,"5min":5.7930000000000001,"15min":6.7709999999999999},"last":3.3599999999999999}]},{"osd":7,"last update":"Fri Nov 20 12:21:22 2020","interfaces":[{"interface":"back","average":{"1min":3.6960000000000002,"5min":3.7450000000000001,"15min":3.7610000000000001},"min":{"1min":2.5990000000000002,"5min":1.7649999999999999,"15min":1.5860000000000001},"max":{"1min":4.7729999999999997,"5min":10.839,"15min":12.412000000000001},"last":4.5880000000000001},{"interface":"front","average":{"1min":3.8290000000000002,"5min":3.6240000000000001,"15min":3.915},"min":{"1min":2.621,"5min":2.0219999999999998,"15min":1.7030000000000001},"max":{"1min":4.8159999999999998,"5min":5.4880000000000004,"15min":73.125},"last":3.621}]}]},{"osd":3,"up_from":14,"seq":60129542355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":2.1000000000000001,"15min":1.9690000000000001},"min":{"1min":1.304,"5min":0.73399999999999999,"15min":0.72499999999999998},"max":{"1min":3.71,"5min":14.436999999999999,"15min":14.436999999999999},"last":2.9590000000000001},{"interface":"front","average":{"1min":2.1469999999999998,"5min":2.0350000000000001,"15min":1.974},"min":{"1min":1.141,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":3.3100000000000001,"5min":3.4940000000000002,"15min":4.9870000000000001},"last":1.5960000000000001}]},{"osd":1,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.1920000000000002,"5min":3.8399999999999999,"15min":2.6869999999999998},"min":{"1min":1.5720000000000001,"5min":1.079,"15min":0.88800000000000001},"max":{"1min":3.8999999999999999,"5min":102.953,"15min":102.953},"last":1.8460000000000001},{"interface":"front","average":{"1min":2.4380000000000002,"5min":2.6309999999999998,"15min":2.4969999999999999},"min":{"1min":1.1850000000000001,"5min":1.139,"15min":0.91700000000000004},"max":{"1min":3.3980000000000001,"5min":16.895,"15min":16.895},"last":2.0920000000000001}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.6150000000000002,"5min":2.6419999999999999,"15min":2.7189999999999999},"min":{"1min":1.2090000000000001,"5min":0.755,"15min":0.755},"max":{"1min":3.3580000000000001,"5min":5.085,"15min":6.2469999999999999},"last":3.4910000000000001},{"interface":"front","average":{"1min":2.992,"5min":3.1379999999999999,"15min":3.0019999999999998},"min":{"1min":1.8600000000000001,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":4.0389999999999997,"5min":14.715999999999999,"15min":14.715999999999999},"last":3.3660000000000001}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.9180000000000001,"5min":2.964,"15min":3.2269999999999999},"min":{"1min":2.0449999999999999,"5min":1.454,"15min":1.0960000000000001},"max":{"1min":4.2279999999999998,"5min":5.0199999999999996,"15min":70.090999999999994},"last":1.9470000000000001},{"interface":"front","average":{"1min":3.2490000000000001,"5min":3.2189999999999999,"15min":3.444},"min":{"1min":2.0339999999999998,"5min":1.28,"15min":1.28},"max":{"1min":5.5199999999999996,"5min":10.496,"15min":71.367999999999995},"last":2.4710000000000001}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.355,"5min":3.4100000000000001,"15min":3.395},"min":{"1min":2.2509999999999999,"5min":1.659,"15min":1.5589999999999999},"max":{"1min":5.0279999999999996,"5min":10.621,"15min":10.621},"last":3.0289999999999999},{"interface":"front","average":{"1min":3.677,"5min":3.7559999999999998,"15min":3.5190000000000001},"min":{"1min":2.6539999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":5.1760000000000002,"5min":17.579999999999998,"15min":17.579999999999998},"last":3.6469999999999998}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.5800000000000001,"5min":3.456,"15min":3.6459999999999999},"min":{"1min":2.4580000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"max":{"1min":4.6529999999999996,"5min":4.7889999999999997,"15min":59.569000000000003},"last":2.6869999999999998},{"interface":"front","average":{"1min":3.9940000000000002,"5min":3.9199999999999999,"15min":4.0910000000000002},"min":{"1min":2.6120000000000001,"5min":1.79,"15min":1.79},"max":{"1min":6.3890000000000002,"5min":10.726000000000001,"15min":60.023000000000003},"last":3.7160000000000002}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.851,"5min":3.9729999999999999,"15min":3.7690000000000001},"min":{"1min":2.762,"5min":1.595,"15min":1.595},"max":{"1min":4.9909999999999997,"5min":22.126000000000001,"15min":22.126000000000001},"last":3.859},{"interface":"front","average":{"1min":3.7050000000000001,"5min":3.6669999999999998,"15min":3.5979999999999999},"min":{"1min":2.2370000000000001,"5min":1.7270000000000001,"15min":1.615},"max":{"1min":5.4489999999999998,"5min":8.2690000000000001,"15min":8.2690000000000001},"last":3.0979999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29032,"kb_used_data":2016,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94342808,"statfs":{"total":96636764160,"available":96607035392,"internally_reserved":0,"allocated":2064384,"data_stored":959063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":2.133,"15min":2.3639999999999999},"min":{"1min":1.5569999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.5059999999999998,"5min":4.1429999999999998,"15min":57.734999999999999},"last":2.1299999999999999},{"interface":"front","average":{"1min":2.23,"5min":2.7309999999999999,"15min":2.431},"min":{"1min":1.581,"5min":1.3069999999999999,"15min":1.1539999999999999},"max":{"1min":3.6539999999999999,"5min":26.75,"15min":26.75},"last":2.2240000000000002}]},{"osd":1,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.4319999999999999,"5min":2.879,"15min":2.6619999999999999},"min":{"1min":1.4330000000000001,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":3.923,"5min":18.234999999999999,"15min":18.234999999999999},"last":2.347},{"interface":"front","average":{"1min":2.9849999999999999,"5min":3.2770000000000001,"15min":3.0939999999999999},"min":{"1min":2,"5min":1.696,"15min":1.397},"max":{"1min":4.4509999999999996,"5min":14.981,"15min":14.981},"last":3.218}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.012,"5min":4.0350000000000001,"15min":3.323},"min":{"1min":1.9590000000000001,"5min":1.3009999999999999,"15min":1.3009999999999999},"max":{"1min":4.1550000000000002,"5min":63.558999999999997,"15min":63.558999999999997},"last":3.4460000000000002},{"interface":"front","average":{"1min":2.911,"5min":3.0699999999999998,"15min":2.964},"min":{"1min":1.776,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":4.0519999999999996,"5min":12.754,"15min":12.754},"last":3.802}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.3849999999999998,"5min":3.8929999999999998,"15min":3.5630000000000002},"min":{"1min":2.6579999999999999,"5min":1.9650000000000001,"15min":1.5609999999999999},"max":{"1min":4.0720000000000001,"5min":27.239999999999998,"15min":27.239999999999998},"last":3.879},{"interface":"front","average":{"1min":3.4449999999999998,"5min":3.665,"15min":3.492},"min":{"1min":2.1619999999999999,"5min":1.6859999999999999,"15min":1.113},"max":{"1min":4.5,"5min":12.881,"15min":12.881},"last":4.2240000000000002}]},{"osd":5,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.3759999999999999,"5min":3.3610000000000002,"15min":3.4220000000000002},"min":{"1min":2.3279999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":4.8479999999999999,"5min":7.0590000000000002,"15min":7.0590000000000002},"last":4.5469999999999997},{"interface":"front","average":{"1min":4.0640000000000001,"5min":4.4089999999999998,"15min":4.0810000000000004},"min":{"1min":2.4750000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"max":{"1min":4.9480000000000004,"5min":26.917000000000002,"15min":26.917000000000002},"last":5.4400000000000004}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.9569999999999999,"5min":4.0259999999999998,"15min":3.9849999999999999},"min":{"1min":1.597,"5min":1.109,"15min":1.109},"max":{"1min":4.8380000000000001,"5min":12.606,"15min":12.606},"last":5.0449999999999999},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.9540000000000002,"15min":3.9580000000000002},"min":{"1min":2.2370000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":5.298,"5min":13.57,"15min":13.57},"last":4.9790000000000001}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":4.3109999999999999,"5min":4.79,"15min":4.5039999999999996},"min":{"1min":3.0630000000000002,"5min":2.0379999999999998,"15min":2.0379999999999998},"max":{"1min":5.9720000000000004,"5min":27.085000000000001,"15min":27.085000000000001},"last":5.1609999999999996},{"interface":"front","average":{"1min":4.444,"5min":4.4580000000000002,"15min":4.4989999999999997},"min":{"1min":2.456,"5min":1.323,"15min":1.323},"max":{"1min":5.6719999999999997,"5min":8.798,"15min":8.798},"last":4.75}]}]},{"osd":5,"up_from":14,"seq":60129542355,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23124,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":21120,"kb_avail":94348716,"statfs":{"total":96636764160,"available":96613085184,"internally_reserved":0,"allocated":2043904,"data_stored":957701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21626880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.1669999999999998,"5min":2.2919999999999998,"15min":3.1619999999999999},"min":{"1min":1.5049999999999999,"5min":1.2589999999999999,"15min":1.123},"max":{"1min":3.3340000000000001,"5min":11.779999999999999,"15min":194.91},"last":1.46},{"interface":"front","average":{"1min":2.0600000000000001,"5min":2.2170000000000001,"15min":2.246},"min":{"1min":1.3,"5min":1.0309999999999999,"15min":1.024},"max":{"1min":3.694,"5min":10.419,"15min":15.971},"last":1.593}]},{"osd":1,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":2.0739999999999998,"5min":2.3580000000000001,"15min":3.0539999999999998},"min":{"1min":1.345,"5min":1.288,"15min":1.23},"max":{"1min":2.984,"5min":8.3659999999999997,"15min":206.43600000000001},"last":2.1629999999999998},{"interface":"front","average":{"1min":3.1520000000000001,"5min":3.3380000000000001,"15min":3.371},"min":{"1min":2.0329999999999999,"5min":1.4019999999999999,"15min":1.401},"max":{"1min":3.8849999999999998,"5min":15.067,"15min":36.784999999999997},"last":3.157}]},{"osd":2,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.1829999999999998,"5min":2.794,"15min":3.7010000000000001},"min":{"1min":1.831,"5min":1.405,"15min":1.0469999999999999},"max":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":209.839},"last":1.7629999999999999},{"interface":"front","average":{"1min":3.3700000000000001,"5min":3.286,"15min":3.387},"min":{"1min":2.6669999999999998,"5min":1.637,"15min":1.478},"max":{"1min":4.1950000000000003,"5min":12.802,"15min":51.009},"last":3.6280000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.1059999999999999,"5min":3.2839999999999998,"15min":3.214},"min":{"1min":2.2240000000000002,"5min":1.379,"15min":1.3069999999999999},"max":{"1min":3.9700000000000002,"5min":11.237,"15min":21.593},"last":2.5649999999999999},{"interface":"front","average":{"1min":3.7909999999999999,"5min":3.5830000000000002,"15min":4.1180000000000003},"min":{"1min":2.2970000000000002,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":5.1260000000000003,"5min":7.5270000000000001,"15min":210.172},"last":2.8839999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.9100000000000001,"5min":3.7189999999999999,"15min":4.3490000000000002},"min":{"1min":2.5950000000000002,"5min":1.536,"15min":1.526},"max":{"1min":4.944,"5min":5.5350000000000001,"15min":206.27799999999999},"last":3.3809999999999998},{"interface":"front","average":{"1min":4.1269999999999998,"5min":4.0380000000000003,"15min":4.0800000000000001},"min":{"1min":3.4729999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"max":{"1min":4.6630000000000003,"5min":15.467000000000001,"15min":51.384},"last":4.2249999999999996}]},{"osd":6,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":3.6709999999999998,"5min":3.673,"15min":3.7210000000000001},"min":{"1min":2.444,"5min":1.3560000000000001,"15min":1.341},"max":{"1min":4.6299999999999999,"5min":10.332000000000001,"15min":17.254000000000001},"last":3.7599999999999998},{"interface":"front","average":{"1min":4.3380000000000001,"5min":4.3520000000000003,"15min":4.4500000000000002},"min":{"1min":3.2130000000000001,"5min":1.607,"15min":1.607},"max":{"1min":5.1369999999999996,"5min":15.326000000000001,"15min":51.531999999999996},"last":3.996}]},{"osd":7,"last update":"Fri Nov 20 12:21:18 2020","interfaces":[{"interface":"back","average":{"1min":4.1390000000000002,"5min":4.165,"15min":4.5060000000000002},"min":{"1min":2.6629999999999998,"5min":1.4810000000000001,"15min":1.4810000000000001},"max":{"1min":5.0730000000000004,"5min":10.488,"15min":142.23699999999999},"last":3.2930000000000001},{"interface":"front","average":{"1min":4.359,"5min":4.3070000000000004,"15min":4.8550000000000004},"min":{"1min":2.8410000000000002,"5min":1.675,"15min":1.675},"max":{"1min":5.1740000000000004,"5min":12.441000000000001,"15min":210.02500000000001},"last":4.4020000000000001}]}]}]}} 2020-11-20T12:22:17.272 INFO:tasks.ceph:pgid 32.4 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.272 INFO:tasks.ceph:pgid 32.3 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.272 INFO:tasks.ceph:pgid 32.2 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.273 INFO:tasks.ceph:pgid 32.1 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.273 INFO:tasks.ceph:pgid 32.0 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.273 INFO:tasks.ceph:pgid 31.7 last_scrub_stamp 2020-11-20T12:21:21.051038+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.274 INFO:tasks.ceph:pgid 31.5 last_scrub_stamp 2020-11-20T12:21:21.051038+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.274 INFO:tasks.ceph:pgid 31.4 last_scrub_stamp 2020-11-20T12:21:21.051038+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.275 INFO:tasks.ceph:pgid 31.3 last_scrub_stamp 2020-11-20T12:21:21.051038+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.275 INFO:tasks.ceph:pgid 31.2 last_scrub_stamp 2020-11-20T12:21:21.051038+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.275 INFO:tasks.ceph:pgid 31.1 last_scrub_stamp 2020-11-20T12:21:21.051038+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.275 INFO:tasks.ceph:pgid 31.0 last_scrub_stamp 2020-11-20T12:21:21.051038+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.276 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-11-20T12:04:52.257420+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=52, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.276 INFO:tasks.ceph:pgid 32.5 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.277 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-11-20T12:04:52.257420+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=52, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.277 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-20T12:04:52.257420+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=52, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.277 INFO:tasks.ceph:pgid 32.6 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.278 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-11-20T12:04:52.257420+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=52, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.278 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-11-20T12:04:44.517280+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.278 INFO:tasks.ceph:pgid 32.7 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.278 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-11-20T12:04:52.257420+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=52, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.279 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-11-20T12:04:52.257420+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=52, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.279 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-11-20T12:04:52.257420+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=52, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.279 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-11-20T12:04:52.257420+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=4, tm_sec=52, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:17.280 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-20T12:22:24.662 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:22:24.678 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:22:37.282 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:22:37.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.530+0000 7f23053f9700 1 Processor -- start 2020-11-20T12:22:37.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.530+0000 7f23053f9700 1 -- start start 2020-11-20T12:22:37.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.531+0000 7f23053f9700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2300065190 0x7f2300065570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:37.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.532+0000 7f23053f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230009ca90 0x7f230009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:37.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.532+0000 7f23053f9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23000b13e0 con 0x7f2300065f90 2020-11-20T12:22:37.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.532+0000 7f23053f9700 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230014ab30 con 0x7f2300065190 2020-11-20T12:22:37.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.532+0000 7f23053f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300062c10 con 0x7f230009ca90 2020-11-20T12:22:37.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.532+0000 7f22fffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2300065190 0x7f2300065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:37.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.532+0000 7f22fffff700 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2300065190 0x7f2300065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55304/0 (socket says 172.21.15.187:55304) 2020-11-20T12:22:37.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.532+0000 7f22fffff700 1 -- 172.21.15.187:0/4175082807 learned_addr learned my addr 172.21.15.187:0/4175082807 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:37.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.532+0000 7f2304bf8700 1 --2- 172.21.15.187:0/4175082807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230009ca90 0x7f230009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:37.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.533+0000 7f22fffff700 1 -- 172.21.15.187:0/4175082807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230009ca90 msgr2=0x7f230009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:37.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.533+0000 7f22fffff700 1 --2- 172.21.15.187:0/4175082807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230009ca90 0x7f230009cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.533+0000 7f22fffff700 1 -- 172.21.15.187:0/4175082807 >> v1:172.21.15.5:6789/0 conn(0x7f2300065f90 legacy=0x7f2300143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:37.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.533+0000 7f22fffff700 1 -- 172.21.15.187:0/4175082807 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000b1640 con 0x7f2300065190 2020-11-20T12:22:37.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.534+0000 7f22fffff700 1 --2- 172.21.15.187:0/4175082807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065190 0x7f2300065570 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7f22f0006fd0 tx=0x7f22f0006eb0).ready entity=mon.1 client_cookie=62a0e621584715f3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:37.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.534+0000 7f22ff7fe700 1 -- 172.21.15.187:0/4175082807 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f00098a0 con 0x7f2300065190 2020-11-20T12:22:37.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.534+0000 7f22ff7fe700 1 -- 172.21.15.187:0/4175082807 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f0009a00 con 0x7f2300065190 2020-11-20T12:22:37.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.534+0000 7f22ff7fe700 1 -- 172.21.15.187:0/4175082807 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0009d80 con 0x7f2300065190 2020-11-20T12:22:37.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.535+0000 7f23053f9700 1 -- 172.21.15.187:0/4175082807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065190 msgr2=0x7f2300065570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:37.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.535+0000 7f23053f9700 1 --2- 172.21.15.187:0/4175082807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065190 0x7f2300065570 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7f22f0006fd0 tx=0x7f22f0006eb0).stop 2020-11-20T12:22:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.535+0000 7f23053f9700 1 -- 172.21.15.187:0/4175082807 shutdown_connections 2020-11-20T12:22:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.535+0000 7f23053f9700 1 --2- 172.21.15.187:0/4175082807 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230009ca90 0x7f230009cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.535+0000 7f23053f9700 1 --2- 172.21.15.187:0/4175082807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065190 0x7f2300065570 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.535+0000 7f23053f9700 1 -- 172.21.15.187:0/4175082807 >> 172.21.15.187:0/4175082807 conn(0x7f23000bab30 msgr2=0x7f23000baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:37.543 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.535+0000 7f23053f9700 1 -- 172.21.15.187:0/4175082807 shutdown_connections 2020-11-20T12:22:37.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.535+0000 7f23053f9700 1 -- 172.21.15.187:0/4175082807 wait complete. 2020-11-20T12:22:37.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.537+0000 7f23053f9700 1 Processor -- start 2020-11-20T12:22:37.544 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.537+0000 7f23053f9700 1 -- start start 2020-11-20T12:22:37.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.537+0000 7f23053f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2300065190 0x7f2300142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:37.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.537+0000 7f23053f9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065f90 0x7f23001431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:37.548 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f23053f9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f230009ca90 0x7f23001599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:37.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f23053f9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2300149750 con 0x7f230009ca90 2020-11-20T12:22:37.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f23053f9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000a0840 con 0x7f2300065f90 2020-11-20T12:22:37.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f23053f9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23000b27e0 con 0x7f2300065190 2020-11-20T12:22:37.549 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22f77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065f90 0x7f23001431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:37.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22f77fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065f90 0x7f23001431d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55312/0 (socket says 172.21.15.187:55312) 2020-11-20T12:22:37.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22f77fe700 1 -- 172.21.15.187:0/2807540153 learned_addr learned my addr 172.21.15.187:0/2807540153 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:37.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22fffff700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2300065190 0x7f2300142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:37.550 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f2304bf8700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f230009ca90 0x7f23001599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:37.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22f77fe700 1 -- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2300065190 msgr2=0x7f2300142ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:37.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22f77fe700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2300065190 0x7f2300142ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22f77fe700 1 -- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f230009ca90 msgr2=0x7f23001599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:37.551 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22f77fe700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f230009ca90 0x7f23001599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22f77fe700 1 -- 172.21.15.187:0/2807540153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23001499c0 con 0x7f2300065f90 2020-11-20T12:22:37.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.538+0000 7f22fffff700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2300065190 0x7f2300142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:22:37.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.539+0000 7f22f77fe700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065f90 0x7f23001431d0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7f22e800c710 tx=0x7f22e8007be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:37.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.539+0000 7f22fcff9700 1 -- 172.21.15.187:0/2807540153 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800e560 con 0x7f2300065f90 2020-11-20T12:22:37.552 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.539+0000 7f22fcff9700 1 -- 172.21.15.187:0/2807540153 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e800e6c0 con 0x7f2300065f90 2020-11-20T12:22:37.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.539+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2300149e60 con 0x7f2300065f90 2020-11-20T12:22:37.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.539+0000 7f22fcff9700 1 -- 172.21.15.187:0/2807540153 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800ea60 con 0x7f2300065f90 2020-11-20T12:22:37.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.539+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23000b24a0 con 0x7f2300065f90 2020-11-20T12:22:37.553 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.540+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2300149b50 con 0x7f2300065f90 2020-11-20T12:22:37.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.541+0000 7f22fcff9700 1 -- 172.21.15.187:0/2807540153 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f22e8019030 con 0x7f2300065f90 2020-11-20T12:22:37.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.541+0000 7f22fcff9700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22dc056d20 0x7f22dc0591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:37.554 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.542+0000 7f22fcff9700 1 -- 172.21.15.187:0/2807540153 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f22e806f370 con 0x7f2300065f90 2020-11-20T12:22:37.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.542+0000 7f22fffff700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22dc056d20 0x7f22dc0591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:37.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.542+0000 7f22fffff700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22dc056d20 0x7f22dc0591e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f22f0002fc0 tx=0x7f22f0015000).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:37.555 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.545+0000 7f22fcff9700 1 -- 172.21.15.187:0/2807540153 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f22e8043fd0 con 0x7f2300065f90 2020-11-20T12:22:37.731 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.727+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2300061a20 con 0x7f22dc056d20 2020-11-20T12:22:37.742 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.739+0000 7f22fcff9700 1 -- 172.21.15.187:0/2807540153 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+105180 (secure 0 0 0) 0x7f2300061a20 con 0x7f22dc056d20 2020-11-20T12:22:37.744 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:37.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22dc056d20 msgr2=0x7f22dc0591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:37.746 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22dc056d20 0x7f22dc0591e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f22f0002fc0 tx=0x7f22f0015000).stop 2020-11-20T12:22:37.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065f90 msgr2=0x7f23001431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:37.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065f90 0x7f23001431d0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7f22e800c710 tx=0x7f22e8007be0).stop 2020-11-20T12:22:37.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 shutdown_connections 2020-11-20T12:22:37.747 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f22dc056d20 0x7f22dc0591e0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f230009ca90 0x7f23001599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2300065190 0x7f2300142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.742+0000 7f23053f9700 1 --2- 172.21.15.187:0/2807540153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2300065f90 0x7f23001431d0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:37.748 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.743+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 >> 172.21.15.187:0/2807540153 conn(0x7f23000bab30 msgr2=0x7f23000aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:37.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.743+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 shutdown_connections 2020-11-20T12:22:37.749 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:37.743+0000 7f23053f9700 1 -- 172.21.15.187:0/2807540153 wait complete. 2020-11-20T12:22:37.749 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2020-11-20T12:22:37.770 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2020-11-20T12:22:35.854338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62081,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":8194,"num_write":148,"num_write_kb":16514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14224,"num_omap_keys":15,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":214324,"kb_used_data":16180,"kb_used_omap":0,"kb_used_meta":198080,"kb_avail":754760396,"statfs":{"total":773094113280,"available":772874645504,"internally_reserved":0,"allocated":16568320,"data_stored":7789991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":202833920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1205,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004223"},"pg_stats":[{"pgid":"32.4","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:28.494810+0000","last_change":"2020-11-20T12:22:28.495011+0000","last_active":"2020-11-20T12:22:28.494810+0000","last_peered":"2020-11-20T12:22:28.494810+0000","last_clean":"2020-11-20T12:22:28.494810+0000","last_became_active":"2020-11-20T12:21:22.693729+0000","last_became_peered":"2020-11-20T12:21:22.693729+0000","last_unstale":"2020-11-20T12:22:28.494810+0000","last_undegraded":"2020-11-20T12:22:28.494810+0000","last_fullsized":"2020-11-20T12:22:28.494810+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:28.494800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:28.494800+0000","last_clean_scrub_stamp":"2020-11-20T12:22:28.494800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":"20","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:04.826440+0000","last_change":"2020-11-20T12:21:57.112403+0000","last_active":"2020-11-20T12:22:04.826440+0000","last_peered":"2020-11-20T12:22:04.826440+0000","last_clean":"2020-11-20T12:22:04.826440+0000","last_became_active":"2020-11-20T12:21:22.685927+0000","last_became_peered":"2020-11-20T12:21:22.685927+0000","last_unstale":"2020-11-20T12:22:04.826440+0000","last_undegraded":"2020-11-20T12:22:04.826440+0000","last_fullsized":"2020-11-20T12:22:04.826440+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:21:22.063495+0000","last_clean_scrub_stamp":"2020-11-20T12:21:22.063495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2","version":"160'10","reported_seq":"33","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:27.499185+0000","last_change":"2020-11-20T12:22:27.499347+0000","last_active":"2020-11-20T12:22:27.499185+0000","last_peered":"2020-11-20T12:22:27.499185+0000","last_clean":"2020-11-20T12:22:27.499185+0000","last_became_active":"2020-11-20T12:21:22.693767+0000","last_became_peered":"2020-11-20T12:21:22.693767+0000","last_unstale":"2020-11-20T12:22:27.499185+0000","last_undegraded":"2020-11-20T12:22:27.499185+0000","last_fullsized":"2020-11-20T12:22:27.499185+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'10","last_scrub_stamp":"2020-11-20T12:22:27.499175+0000","last_deep_scrub":"160'10","last_deep_scrub_stamp":"2020-11-20T12:22:27.499175+0000","last_clean_scrub_stamp":"2020-11-20T12:22:27.499175+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":6,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:17.379608+0000","last_change":"2020-11-20T12:22:17.379764+0000","last_active":"2020-11-20T12:22:17.379608+0000","last_peered":"2020-11-20T12:22:17.379608+0000","last_clean":"2020-11-20T12:22:17.379608+0000","last_became_active":"2020-11-20T12:21:22.684395+0000","last_became_peered":"2020-11-20T12:21:22.684395+0000","last_unstale":"2020-11-20T12:22:17.379608+0000","last_undegraded":"2020-11-20T12:22:17.379608+0000","last_fullsized":"2020-11-20T12:22:17.379608+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:17.379599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:17.379599+0000","last_clean_scrub_stamp":"2020-11-20T12:22:17.379599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:26.469813+0000","last_change":"2020-11-20T12:22:26.470009+0000","last_active":"2020-11-20T12:22:26.469813+0000","last_peered":"2020-11-20T12:22:26.469813+0000","last_clean":"2020-11-20T12:22:26.469813+0000","last_became_active":"2020-11-20T12:21:22.692492+0000","last_became_peered":"2020-11-20T12:21:22.692492+0000","last_unstale":"2020-11-20T12:22:26.469813+0000","last_undegraded":"2020-11-20T12:22:26.469813+0000","last_fullsized":"2020-11-20T12:22:26.469813+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:26.469803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:26.469803+0000","last_clean_scrub_stamp":"2020-11-20T12:22:26.469803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"161'20","reported_seq":"35","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:31.744034+0000","last_change":"2020-11-20T12:22:31.744195+0000","last_active":"2020-11-20T12:22:31.744034+0000","last_peered":"2020-11-20T12:22:31.744034+0000","last_clean":"2020-11-20T12:22:31.744034+0000","last_became_active":"2020-11-20T12:21:22.108194+0000","last_became_peered":"2020-11-20T12:21:22.108194+0000","last_unstale":"2020-11-20T12:22:31.744034+0000","last_undegraded":"2020-11-20T12:22:31.744034+0000","last_fullsized":"2020-11-20T12:22:31.744034+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'20","last_scrub_stamp":"2020-11-20T12:22:31.744020+0000","last_deep_scrub":"161'20","last_deep_scrub_stamp":"2020-11-20T12:22:31.744020+0000","last_clean_scrub_stamp":"2020-11-20T12:22:31.744020+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":591,"num_omap_keys":1,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"161'13","reported_seq":"28","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:12.028222+0000","last_change":"2020-11-20T12:22:12.028348+0000","last_active":"2020-11-20T12:22:12.028222+0000","last_peered":"2020-11-20T12:22:12.028222+0000","last_clean":"2020-11-20T12:22:12.028222+0000","last_became_active":"2020-11-20T12:21:22.113651+0000","last_became_peered":"2020-11-20T12:21:22.113651+0000","last_unstale":"2020-11-20T12:22:12.028222+0000","last_undegraded":"2020-11-20T12:22:12.028222+0000","last_fullsized":"2020-11-20T12:22:12.028222+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'13","last_scrub_stamp":"2020-11-20T12:22:12.028215+0000","last_deep_scrub":"161'13","last_deep_scrub_stamp":"2020-11-20T12:22:12.028215+0000","last_clean_scrub_stamp":"2020-11-20T12:22:12.028215+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"161'14","reported_seq":"29","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:21.423283+0000","last_change":"2020-11-20T12:22:21.423449+0000","last_active":"2020-11-20T12:22:21.423283+0000","last_peered":"2020-11-20T12:22:21.423283+0000","last_clean":"2020-11-20T12:22:21.423283+0000","last_became_active":"2020-11-20T12:21:22.111838+0000","last_became_peered":"2020-11-20T12:21:22.111838+0000","last_unstale":"2020-11-20T12:22:21.423283+0000","last_undegraded":"2020-11-20T12:22:21.423283+0000","last_fullsized":"2020-11-20T12:22:21.423283+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'14","last_scrub_stamp":"2020-11-20T12:22:21.423270+0000","last_deep_scrub":"161'14","last_deep_scrub_stamp":"2020-11-20T12:22:21.423270+0000","last_clean_scrub_stamp":"2020-11-20T12:22:21.423270+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":184,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.4","version":"161'46","reported_seq":"61","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:16.428786+0000","last_change":"2020-11-20T12:22:16.428983+0000","last_active":"2020-11-20T12:22:16.428786+0000","last_peered":"2020-11-20T12:22:16.428786+0000","last_clean":"2020-11-20T12:22:16.428786+0000","last_became_active":"2020-11-20T12:21:22.107044+0000","last_became_peered":"2020-11-20T12:21:22.107044+0000","last_unstale":"2020-11-20T12:22:16.428786+0000","last_undegraded":"2020-11-20T12:22:16.428786+0000","last_fullsized":"2020-11-20T12:22:16.428786+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'46","last_scrub_stamp":"2020-11-20T12:22:16.428774+0000","last_deep_scrub":"161'46","last_deep_scrub_stamp":"2020-11-20T12:22:16.428774+0000","last_clean_scrub_stamp":"2020-11-20T12:22:16.428774+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61078,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"159'12","reported_seq":"27","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:18.385337+0000","last_change":"2020-11-20T12:22:18.385540+0000","last_active":"2020-11-20T12:22:18.385337+0000","last_peered":"2020-11-20T12:22:18.385337+0000","last_clean":"2020-11-20T12:22:18.385337+0000","last_became_active":"2020-11-20T12:21:22.105399+0000","last_became_peered":"2020-11-20T12:21:22.105399+0000","last_unstale":"2020-11-20T12:22:18.385337+0000","last_undegraded":"2020-11-20T12:22:18.385337+0000","last_fullsized":"2020-11-20T12:22:18.385337+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'12","last_scrub_stamp":"2020-11-20T12:22:18.385325+0000","last_deep_scrub":"159'12","last_deep_scrub_stamp":"2020-11-20T12:22:18.385325+0000","last_clean_scrub_stamp":"2020-11-20T12:22:18.385325+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4948,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"158'1","reported_seq":"16","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:14.715943+0000","last_change":"2020-11-20T12:22:14.716078+0000","last_active":"2020-11-20T12:22:14.715943+0000","last_peered":"2020-11-20T12:22:14.715943+0000","last_clean":"2020-11-20T12:22:14.715943+0000","last_became_active":"2020-11-20T12:21:22.098007+0000","last_became_peered":"2020-11-20T12:21:22.098007+0000","last_unstale":"2020-11-20T12:22:14.715943+0000","last_undegraded":"2020-11-20T12:22:14.715943+0000","last_fullsized":"2020-11-20T12:22:14.715943+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'1","last_scrub_stamp":"2020-11-20T12:22:14.715932+0000","last_deep_scrub":"158'1","last_deep_scrub_stamp":"2020-11-20T12:22:14.715932+0000","last_clean_scrub_stamp":"2020-11-20T12:22:14.715932+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"161'4","reported_seq":"21","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:26.725780+0000","last_change":"2020-11-20T12:22:26.725966+0000","last_active":"2020-11-20T12:22:26.725780+0000","last_peered":"2020-11-20T12:22:26.725780+0000","last_clean":"2020-11-20T12:22:26.725780+0000","last_became_active":"2020-11-20T12:21:22.102371+0000","last_became_peered":"2020-11-20T12:21:22.102371+0000","last_unstale":"2020-11-20T12:22:26.725780+0000","last_undegraded":"2020-11-20T12:22:26.725780+0000","last_fullsized":"2020-11-20T12:22:26.725780+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'4","last_scrub_stamp":"2020-11-20T12:22:26.725767+0000","last_deep_scrub":"161'4","last_deep_scrub_stamp":"2020-11-20T12:22:26.725767+0000","last_clean_scrub_stamp":"2020-11-20T12:22:26.725767+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":210,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"161'18","reported_seq":"33","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:30.806708+0000","last_change":"2020-11-20T12:22:30.806858+0000","last_active":"2020-11-20T12:22:30.806708+0000","last_peered":"2020-11-20T12:22:30.806708+0000","last_clean":"2020-11-20T12:22:30.806708+0000","last_became_active":"2020-11-20T12:21:22.093851+0000","last_became_peered":"2020-11-20T12:21:22.093851+0000","last_unstale":"2020-11-20T12:22:30.806708+0000","last_undegraded":"2020-11-20T12:22:30.806708+0000","last_fullsized":"2020-11-20T12:22:30.806708+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'18","last_scrub_stamp":"2020-11-20T12:22:30.806695+0000","last_deep_scrub":"161'18","last_deep_scrub_stamp":"2020-11-20T12:22:30.806695+0000","last_clean_scrub_stamp":"2020-11-20T12:22:30.806695+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":614,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"149","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:16.840688+0000","last_change":"2020-11-20T12:22:16.840862+0000","last_active":"2020-11-20T12:22:16.840688+0000","last_peered":"2020-11-20T12:22:16.840688+0000","last_clean":"2020-11-20T12:22:16.840688+0000","last_became_active":"2020-11-20T12:04:53.294169+0000","last_became_peered":"2020-11-20T12:04:53.294169+0000","last_unstale":"2020-11-20T12:22:16.840688+0000","last_undegraded":"2020-11-20T12:22:16.840688+0000","last_fullsized":"2020-11-20T12:22:16.840688+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:16.840678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:16.840678+0000","last_clean_scrub_stamp":"2020-11-20T12:22:16.840678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"160'8","reported_seq":"31","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:19.421879+0000","last_change":"2020-11-20T12:22:19.422107+0000","last_active":"2020-11-20T12:22:19.421879+0000","last_peered":"2020-11-20T12:22:19.421879+0000","last_clean":"2020-11-20T12:22:19.421879+0000","last_became_active":"2020-11-20T12:21:22.676940+0000","last_became_peered":"2020-11-20T12:21:22.676940+0000","last_unstale":"2020-11-20T12:22:19.421879+0000","last_undegraded":"2020-11-20T12:22:19.421879+0000","last_fullsized":"2020-11-20T12:22:19.421879+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'8","last_scrub_stamp":"2020-11-20T12:22:19.421858+0000","last_deep_scrub":"160'8","last_deep_scrub_stamp":"2020-11-20T12:22:19.421858+0000","last_clean_scrub_stamp":"2020-11-20T12:22:19.421858+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":4,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:20.737373+0000","last_change":"2020-11-20T12:22:20.737575+0000","last_active":"2020-11-20T12:22:20.737373+0000","last_peered":"2020-11-20T12:22:20.737373+0000","last_clean":"2020-11-20T12:22:20.737373+0000","last_became_active":"2020-11-20T12:04:53.293882+0000","last_became_peered":"2020-11-20T12:04:53.293882+0000","last_unstale":"2020-11-20T12:22:20.737373+0000","last_undegraded":"2020-11-20T12:22:20.737373+0000","last_fullsized":"2020-11-20T12:22:20.737373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:20.737363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:20.737363+0000","last_clean_scrub_stamp":"2020-11-20T12:22:20.737363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"148","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:22.743332+0000","last_change":"2020-11-20T12:22:22.743517+0000","last_active":"2020-11-20T12:22:22.743332+0000","last_peered":"2020-11-20T12:22:22.743332+0000","last_clean":"2020-11-20T12:22:22.743332+0000","last_became_active":"2020-11-20T12:04:53.297849+0000","last_became_peered":"2020-11-20T12:04:53.297849+0000","last_unstale":"2020-11-20T12:22:22.743332+0000","last_undegraded":"2020-11-20T12:22:22.743332+0000","last_fullsized":"2020-11-20T12:22:22.743332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:22.743322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:22.743322+0000","last_clean_scrub_stamp":"2020-11-20T12:22:22.743322+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:21.425127+0000","last_change":"2020-11-20T12:22:21.425275+0000","last_active":"2020-11-20T12:22:21.425127+0000","last_peered":"2020-11-20T12:22:21.425127+0000","last_clean":"2020-11-20T12:22:21.425127+0000","last_became_active":"2020-11-20T12:21:22.675260+0000","last_became_peered":"2020-11-20T12:21:22.675260+0000","last_unstale":"2020-11-20T12:22:21.425127+0000","last_undegraded":"2020-11-20T12:22:21.425127+0000","last_fullsized":"2020-11-20T12:22:21.425127+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:21.425117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:21.425117+0000","last_clean_scrub_stamp":"2020-11-20T12:22:21.425117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:22.756356+0000","last_change":"2020-11-20T12:22:22.756572+0000","last_active":"2020-11-20T12:22:22.756356+0000","last_peered":"2020-11-20T12:22:22.756356+0000","last_clean":"2020-11-20T12:22:22.756356+0000","last_became_active":"2020-11-20T12:04:54.004416+0000","last_became_peered":"2020-11-20T12:04:54.004416+0000","last_unstale":"2020-11-20T12:22:22.756356+0000","last_undegraded":"2020-11-20T12:22:22.756356+0000","last_fullsized":"2020-11-20T12:22:22.756356+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:22.756347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:22.756347+0000","last_clean_scrub_stamp":"2020-11-20T12:22:22.756347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"161'4","reported_seq":"151","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:17.427670+0000","last_change":"2020-11-20T12:22:17.427902+0000","last_active":"2020-11-20T12:22:17.427670+0000","last_peered":"2020-11-20T12:22:17.427670+0000","last_clean":"2020-11-20T12:22:17.427670+0000","last_became_active":"2020-11-20T12:04:49.156248+0000","last_became_peered":"2020-11-20T12:04:49.156248+0000","last_unstale":"2020-11-20T12:22:17.427670+0000","last_undegraded":"2020-11-20T12:22:17.427670+0000","last_fullsized":"2020-11-20T12:22:17.427670+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'4","last_scrub_stamp":"2020-11-20T12:22:17.427649+0000","last_deep_scrub":"161'4","last_deep_scrub_stamp":"2020-11-20T12:22:17.427649+0000","last_clean_scrub_stamp":"2020-11-20T12:22:17.427649+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8071,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.7","version":"159'1","reported_seq":"24","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:19.297099+0000","last_change":"2020-11-20T12:22:19.297210+0000","last_active":"2020-11-20T12:22:19.297099+0000","last_peered":"2020-11-20T12:22:19.297099+0000","last_clean":"2020-11-20T12:22:19.297099+0000","last_became_active":"2020-11-20T12:21:22.697706+0000","last_became_peered":"2020-11-20T12:21:22.697706+0000","last_unstale":"2020-11-20T12:22:19.297099+0000","last_undegraded":"2020-11-20T12:22:19.297099+0000","last_fullsized":"2020-11-20T12:22:19.297099+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'1","last_scrub_stamp":"2020-11-20T12:22:19.297090+0000","last_deep_scrub":"159'1","last_deep_scrub_stamp":"2020-11-20T12:22:19.297090+0000","last_clean_scrub_stamp":"2020-11-20T12:22:19.297090+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:13.714096+0000","last_change":"2020-11-20T12:22:13.714198+0000","last_active":"2020-11-20T12:22:13.714096+0000","last_peered":"2020-11-20T12:22:13.714096+0000","last_clean":"2020-11-20T12:22:13.714096+0000","last_became_active":"2020-11-20T12:04:53.296376+0000","last_became_peered":"2020-11-20T12:04:53.296376+0000","last_unstale":"2020-11-20T12:22:13.714096+0000","last_undegraded":"2020-11-20T12:22:13.714096+0000","last_fullsized":"2020-11-20T12:22:13.714096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:13.714091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:13.714091+0000","last_clean_scrub_stamp":"2020-11-20T12:22:13.714091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:28.789256+0000","last_change":"2020-11-20T12:22:28.789413+0000","last_active":"2020-11-20T12:22:28.789256+0000","last_peered":"2020-11-20T12:22:28.789256+0000","last_clean":"2020-11-20T12:22:28.789256+0000","last_became_active":"2020-11-20T12:04:53.302386+0000","last_became_peered":"2020-11-20T12:04:53.302386+0000","last_unstale":"2020-11-20T12:22:28.789256+0000","last_undegraded":"2020-11-20T12:22:28.789256+0000","last_fullsized":"2020-11-20T12:22:28.789256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:28.789247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:28.789247+0000","last_clean_scrub_stamp":"2020-11-20T12:22:28.789247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"149","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:20.433611+0000","last_change":"2020-11-20T12:22:20.433794+0000","last_active":"2020-11-20T12:22:20.433611+0000","last_peered":"2020-11-20T12:22:20.433611+0000","last_clean":"2020-11-20T12:22:20.433611+0000","last_became_active":"2020-11-20T12:04:54.002137+0000","last_became_peered":"2020-11-20T12:04:54.002137+0000","last_unstale":"2020-11-20T12:22:20.433611+0000","last_undegraded":"2020-11-20T12:22:20.433611+0000","last_fullsized":"2020-11-20T12:22:20.433611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:20.433600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:20.433600+0000","last_clean_scrub_stamp":"2020-11-20T12:22:20.433600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:29.757483+0000","last_change":"2020-11-20T12:22:29.757654+0000","last_active":"2020-11-20T12:22:29.757483+0000","last_peered":"2020-11-20T12:22:29.757483+0000","last_clean":"2020-11-20T12:22:29.757483+0000","last_became_active":"2020-11-20T12:04:54.001419+0000","last_became_peered":"2020-11-20T12:04:54.001419+0000","last_unstale":"2020-11-20T12:22:29.757483+0000","last_undegraded":"2020-11-20T12:22:29.757483+0000","last_fullsized":"2020-11-20T12:22:29.757483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:29.757472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:29.757472+0000","last_clean_scrub_stamp":"2020-11-20T12:22:29.757472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":10,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":16,"acting":16,"num_store_stats":7},{"poolid":31,"num_pg":8,"stat_sum":{"num_bytes":62081,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":136,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6153,"num_omap_keys":13,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":124162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8071,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542359,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20208,"kb_used_data":2088,"kb_used_omap":0,"kb_used_meta":18112,"kb_avail":94351632,"statfs":{"total":96636764160,"available":96616071168,"internally_reserved":0,"allocated":2138112,"data_stored":1020351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18546688},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":1.869,"5min":2.274,"15min":2.431},"min":{"1min":1.1319999999999999,"5min":0.63,"15min":0.63},"max":{"1min":3.0299999999999998,"5min":8.2889999999999997,"15min":37.622},"last":2.0169999999999999},{"interface":"front","average":{"1min":1.9590000000000001,"5min":2.2360000000000002,"15min":2.4159999999999999},"min":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"max":{"1min":3.0179999999999998,"5min":12.936999999999999,"15min":36.816000000000003},"last":2.109}]},{"osd":1,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":2.6840000000000002,"5min":2.8700000000000001,"15min":3.097},"min":{"1min":1.4710000000000001,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":4.3739999999999997,"5min":11.837999999999999,"15min":42.296999999999997},"last":3.4470000000000001},{"interface":"front","average":{"1min":2.093,"5min":7.1820000000000004,"15min":7.0019999999999998},"min":{"1min":1.4179999999999999,"5min":1.0600000000000001,"15min":1.0600000000000001},"max":{"1min":3.7189999999999999,"5min":498.23500000000001,"15min":633.35299999999995},"last":2.7639999999999998}]},{"osd":2,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.9009999999999998,"15min":2.7999999999999998},"min":{"1min":1.8720000000000001,"5min":1.2090000000000001,"15min":1.073},"max":{"1min":3.8410000000000002,"5min":7.359,"15min":7.359},"last":2.3690000000000002},{"interface":"front","average":{"1min":2.5659999999999998,"5min":2.7709999999999999,"15min":2.988},"min":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4259999999999999},"max":{"1min":3.6400000000000001,"5min":4.4009999999999998,"15min":36.509},"last":4.1760000000000002}]},{"osd":3,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.5899999999999999,"5min":3.4100000000000001,"15min":3.585},"min":{"1min":1.633,"5min":1.129,"15min":1.129},"max":{"1min":4.7649999999999997,"5min":8.3819999999999997,"15min":37.859000000000002},"last":3.8620000000000001},{"interface":"front","average":{"1min":3.02,"5min":3.1640000000000001,"15min":3.2799999999999998},"min":{"1min":2.0390000000000001,"5min":1.702,"15min":1.45},"max":{"1min":4.3369999999999997,"5min":5.0359999999999996,"15min":21.952999999999999},"last":3.1970000000000001}]},{"osd":4,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.2999999999999998,"5min":3.4500000000000002,"15min":3.6379999999999999},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":5.8499999999999996,"5min":13.117000000000001,"15min":36.979999999999997},"last":3.7669999999999999},{"interface":"front","average":{"1min":3.96,"5min":3.9700000000000002,"15min":4.1900000000000004},"min":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"max":{"1min":5.4390000000000001,"5min":9.1329999999999991,"15min":38.177},"last":4.7359999999999998}]},{"osd":5,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.589,"5min":3.633,"15min":3.6819999999999999},"min":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"max":{"1min":4.5049999999999999,"5min":5.2549999999999999,"15min":5.4480000000000004},"last":3.6019999999999999},{"interface":"front","average":{"1min":4.3689999999999998,"5min":4.2750000000000004,"15min":4.4550000000000001},"min":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"max":{"1min":5.1909999999999998,"5min":7.6890000000000001,"15min":37.438000000000002},"last":4.3899999999999997}]},{"osd":6,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.6680000000000001,"5min":3.8519999999999999,"15min":4.077},"min":{"1min":1.218,"5min":1.218,"15min":1.218},"max":{"1min":4.585,"5min":7.1710000000000003,"15min":36.674999999999997},"last":4.5819999999999999},{"interface":"front","average":{"1min":4.5819999999999999,"5min":4.5069999999999997,"15min":4.7450000000000001},"min":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"max":{"1min":5.9329999999999998,"5min":7.79,"15min":38.027000000000001},"last":4.9480000000000004}]}]},{"osd":6,"up_from":14,"seq":60129542359,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30048,"kb_used_data":2072,"kb_used_omap":0,"kb_used_meta":27968,"kb_avail":94341792,"statfs":{"total":96636764160,"available":96605995008,"internally_reserved":0,"allocated":2121728,"data_stored":1019416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28639232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":2.2509999999999999,"5min":2.1829999999999998,"15min":2.1739999999999999},"min":{"1min":1.121,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":3.6869999999999998,"5min":5.0380000000000003,"15min":5.6600000000000001},"last":1.9490000000000001},{"interface":"front","average":{"1min":2.3420000000000001,"5min":2.3900000000000001,"15min":2.4089999999999998},"min":{"1min":1.5109999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.8599999999999999,"5min":4.5179999999999998,"15min":7.1630000000000003},"last":3.3639999999999999}]},{"osd":1,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":2.6459999999999999,"15min":2.5750000000000002},"min":{"1min":1.627,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":4.0739999999999998,"5min":12.022,"15min":12.022},"last":3.657},{"interface":"front","average":{"1min":3.149,"5min":3.8599999999999999,"15min":3.8140000000000001},"min":{"1min":1.982,"5min":1.5629999999999999,"15min":1.252},"max":{"1min":3.891,"5min":74.932000000000002,"15min":140.53299999999999},"last":2.085}]},{"osd":2,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":2.944,"5min":3.0489999999999999,"15min":3.0419999999999998},"min":{"1min":1.6299999999999999,"5min":1.3,"15min":1.3},"max":{"1min":4.2240000000000002,"5min":12.430999999999999,"15min":12.430999999999999},"last":2.2229999999999999},{"interface":"front","average":{"1min":3.0299999999999998,"5min":3.0449999999999999,"15min":3.0459999999999998},"min":{"1min":1.5620000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"max":{"1min":4.5490000000000004,"5min":5.6600000000000001,"15min":7.5140000000000002},"last":3.0339999999999998}]},{"osd":3,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.4060000000000001,"15min":3.4569999999999999},"min":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6140000000000001},"max":{"1min":4.3579999999999997,"5min":5,"15min":8.2249999999999996},"last":2.9409999999999998},{"interface":"front","average":{"1min":3.6640000000000001,"5min":3.5979999999999999,"15min":3.597},"min":{"1min":1.8700000000000001,"5min":1.105,"15min":1.105},"max":{"1min":4.7530000000000001,"5min":13.329000000000001,"15min":13.329000000000001},"last":3.1659999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":3.7909999999999999,"5min":3.6549999999999998,"15min":3.6440000000000001},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":4.9409999999999998,"5min":11.247,"15min":11.247},"last":3.875},{"interface":"front","average":{"1min":4.1390000000000002,"5min":4.0069999999999997,"15min":4.016},"min":{"1min":1.335,"5min":1.335,"15min":1.335},"max":{"1min":5.1200000000000001,"5min":10.82,"15min":11.474},"last":4.2380000000000004}]},{"osd":5,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":4.117,"5min":4.1200000000000001,"15min":4.1349999999999998},"min":{"1min":1.468,"5min":1.4319999999999999,"15min":1.4319999999999999},"max":{"1min":5.2080000000000002,"5min":11.568,"15min":14.435},"last":4.0750000000000002},{"interface":"front","average":{"1min":4.1399999999999997,"5min":4.1210000000000004,"15min":4.1159999999999997},"min":{"1min":1.278,"5min":1.278,"15min":1.278},"max":{"1min":5.2149999999999999,"5min":6.9089999999999998,"15min":11.827999999999999},"last":4.3570000000000002}]},{"osd":7,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":4.4550000000000001,"5min":4.3710000000000004,"15min":4.3869999999999996},"min":{"1min":1.732,"5min":1.732,"15min":1.732},"max":{"1min":5.5270000000000001,"5min":8.2379999999999995,"15min":11.917},"last":4.7409999999999997},{"interface":"front","average":{"1min":4.5979999999999999,"5min":4.5190000000000001,"15min":4.484},"min":{"1min":2.1850000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"max":{"1min":5.569,"5min":16.263000000000002,"15min":16.263000000000002},"last":4.4500000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542359,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":30592,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":2056192,"data_stored":958548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31326208},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":2.194,"15min":5.0750000000000002},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":3.0830000000000002,"5min":32.231999999999999,"15min":868.745},"last":2.423},{"interface":"front","average":{"1min":1.667,"5min":1.9630000000000001,"15min":2.0710000000000002},"min":{"1min":1.0029999999999999,"5min":0.71799999999999997,"15min":0.70399999999999996},"max":{"1min":2.427,"5min":5.7370000000000001,"15min":10.042},"last":2.1179999999999999}]},{"osd":2,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":2.0739999999999998,"15min":2.645},"min":{"1min":0.97499999999999998,"5min":0.69199999999999995,"15min":0.68100000000000005},"max":{"1min":3.165,"5min":3.9689999999999999,"15min":142.95500000000001},"last":2.536},{"interface":"front","average":{"1min":2.4950000000000001,"5min":3.0190000000000001,"15min":6.3310000000000004},"min":{"1min":1.044,"5min":1.04,"15min":0.95999999999999996},"max":{"1min":3.5299999999999998,"5min":32.755000000000003,"15min":868.82299999999998},"last":2.8809999999999998}]},{"osd":3,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.984,"15min":5.8710000000000004},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.0409999999999999},"max":{"1min":4.1040000000000001,"5min":32.377000000000002,"15min":869.04300000000001},"last":2.0390000000000001},{"interface":"front","average":{"1min":2.4199999999999999,"5min":2.7599999999999998,"15min":2.8050000000000002},"min":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":0.94799999999999995},"max":{"1min":3.6880000000000002,"5min":5.593,"15min":11.404},"last":3.766}]},{"osd":4,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.681,"5min":2.8690000000000002,"15min":3.0009999999999999},"min":{"1min":1.4379999999999999,"5min":1.3580000000000001,"15min":0.96899999999999997},"max":{"1min":4.0609999999999999,"5min":4.5979999999999999,"15min":20.577999999999999},"last":3.1789999999999998},{"interface":"front","average":{"1min":3.1200000000000001,"5min":3.5099999999999998,"15min":6.4039999999999999},"min":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.246},"max":{"1min":4.4329999999999998,"5min":32.622,"15min":869.22299999999996},"last":3.6899999999999999}]},{"osd":5,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.2970000000000002,"5min":3.4129999999999998,"15min":3.5289999999999999},"min":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.327},"max":{"1min":4.54,"5min":5.9400000000000004,"15min":13.064},"last":4.4480000000000004},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.2850000000000001,"15min":3.5329999999999999},"min":{"1min":1.387,"5min":1.387,"15min":1.1299999999999999},"max":{"1min":4.1879999999999997,"5min":5.2009999999999996,"15min":62.292000000000002},"last":3.992}]},{"osd":6,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.2789999999999999,"5min":3.8330000000000002,"15min":6.8040000000000003},"min":{"1min":1.587,"5min":1.587,"15min":1.5329999999999999},"max":{"1min":4.4909999999999997,"5min":33.268999999999998,"15min":869.15300000000002},"last":3.2610000000000001},{"interface":"front","average":{"1min":3.2789999999999999,"5min":3.448,"15min":3.681},"min":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"max":{"1min":4.3259999999999996,"5min":6.0759999999999996,"15min":17.042999999999999},"last":4.0499999999999998}]},{"osd":7,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.5640000000000001,"5min":3.7709999999999999,"15min":3.9620000000000002},"min":{"1min":1.798,"5min":1.798,"15min":1.7010000000000001},"max":{"1min":4.7290000000000001,"5min":6.2000000000000002,"15min":11.667},"last":4.2539999999999996},{"interface":"front","average":{"1min":3.5219999999999998,"5min":3.5950000000000002,"15min":3.7450000000000001},"min":{"1min":1.871,"5min":1.222,"15min":1.222},"max":{"1min":4.593,"5min":5.3029999999999999,"15min":20.710000000000001},"last":3.6139999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21784,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":19776,"kb_avail":94350056,"statfs":{"total":96636764160,"available":96614457344,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20250624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.0990000000000002},"min":{"1min":1.3049999999999999,"5min":0.93200000000000005,"15min":0.71299999999999997},"max":{"1min":3.1160000000000001,"5min":6.569,"15min":10.723000000000001},"last":2.0720000000000001},{"interface":"front","average":{"1min":2.4380000000000002,"5min":2.5750000000000002,"15min":2.552},"min":{"1min":1.6419999999999999,"5min":1.0589999999999999,"15min":1.05},"max":{"1min":3.1000000000000001,"5min":7.8929999999999998,"15min":7.8929999999999998},"last":2.3210000000000002}]},{"osd":2,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":2.073,"15min":2.149},"min":{"1min":1.165,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":3.1789999999999998,"5min":3.8929999999999998,"15min":42.066000000000003},"last":1.794},{"interface":"front","average":{"1min":2.0779999999999998,"5min":2.1389999999999998,"15min":2.2330000000000001},"min":{"1min":1.1339999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.7990000000000004},"last":2.7320000000000002}]},{"osd":3,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":3.0529999999999999,"15min":2.8170000000000002},"min":{"1min":1.2989999999999999,"5min":1.095,"15min":0.753},"max":{"1min":4.3659999999999997,"5min":38.067,"15min":38.067},"last":3.008},{"interface":"front","average":{"1min":2.7250000000000001,"5min":3.3100000000000001,"15min":3.0710000000000002},"min":{"1min":1.4890000000000001,"5min":1.242,"15min":1.0149999999999999},"max":{"1min":3.8140000000000001,"5min":38.463000000000001,"15min":38.463000000000001},"last":3.105}]},{"osd":4,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":3,"15min":3.093},"min":{"1min":1.774,"5min":1.2390000000000001,"15min":1.05},"max":{"1min":4.2880000000000003,"5min":9.0749999999999993,"15min":14.467000000000001},"last":2.4359999999999999},{"interface":"front","average":{"1min":2.903,"5min":2.9700000000000002,"15min":3.0219999999999998},"min":{"1min":1.704,"5min":1.329,"15min":1.1579999999999999},"max":{"1min":3.871,"5min":5.0090000000000003,"15min":5.0679999999999996},"last":3.5209999999999999}]},{"osd":5,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.306,"15min":3.3940000000000001},"min":{"1min":1.996,"5min":1.5589999999999999,"15min":1.5589999999999999},"max":{"1min":4.2949999999999999,"5min":6.4539999999999997,"15min":7.8120000000000003},"last":3.1909999999999998},{"interface":"front","average":{"1min":3.427,"5min":3.5659999999999998,"15min":3.6629999999999998},"min":{"1min":2.302,"5min":1.6579999999999999,"15min":1.6579999999999999},"max":{"1min":4.6159999999999997,"5min":6.1959999999999997,"15min":11.58},"last":3.403}]},{"osd":6,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.4089999999999998,"15min":3.4580000000000002},"min":{"1min":1.9199999999999999,"5min":1.5369999999999999,"15min":1.29},"max":{"1min":4.218,"5min":6.6230000000000002,"15min":6.6230000000000002},"last":3.7909999999999999},{"interface":"front","average":{"1min":3.452,"5min":3.4820000000000002,"15min":3.5430000000000001},"min":{"1min":2.1869999999999998,"5min":1.2090000000000001,"15min":1.2090000000000001},"max":{"1min":4.4950000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":3.6240000000000001}]},{"osd":7,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.7320000000000002,"5min":3.7450000000000001,"15min":3.867},"min":{"1min":2.1000000000000001,"5min":1.357,"15min":1.357},"max":{"1min":5.0220000000000002,"5min":7.4770000000000003,"15min":37.826999999999998},"last":4.0599999999999996},{"interface":"front","average":{"1min":3.5779999999999998,"5min":3.738,"15min":3.7389999999999999},"min":{"1min":2.214,"5min":1.7370000000000001,"15min":1.665},"max":{"1min":4.8529999999999998,"5min":6.7329999999999997,"15min":12.154},"last":3.992}]}]},{"osd":2,"up_from":14,"seq":60129542359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":27136,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27787264},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":1.9650000000000001,"5min":1.978,"15min":2.0510000000000002},"min":{"1min":1.016,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":3.9950000000000001,"5min":3.9950000000000001,"15min":9.0239999999999991},"last":1.645},{"interface":"front","average":{"1min":2.4220000000000002,"5min":2.4119999999999999,"15min":2.431},"min":{"1min":0.873,"5min":0.873,"15min":0.871},"max":{"1min":3.8370000000000002,"5min":3.8370000000000002,"15min":6.0800000000000001},"last":3.363}]},{"osd":1,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":2.3620000000000001,"15min":2.452},"min":{"1min":1.2569999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":3.7869999999999999,"5min":4.782,"15min":24.391999999999999},"last":2.0259999999999998},{"interface":"front","average":{"1min":2.3130000000000002,"5min":3,"15min":5.4770000000000003},"min":{"1min":0.79900000000000004,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":3.8580000000000001,"5min":66.977999999999994,"15min":820.03499999999997},"last":1.9410000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":2.0049999999999999,"5min":2.089,"15min":2.161},"min":{"1min":1.4350000000000001,"5min":0.97499999999999998,"15min":0.748},"max":{"1min":3.1629999999999998,"5min":3.2229999999999999,"15min":6.1429999999999998},"last":1.7529999999999999},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.859,"15min":2.9249999999999998},"min":{"1min":1.514,"5min":1.0569999999999999,"15min":0.84599999999999997},"max":{"1min":3.79,"5min":5.601,"15min":10.574999999999999},"last":2.173}]},{"osd":4,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.1819999999999999,"15min":3.2290000000000001},"min":{"1min":1.782,"5min":1.462,"15min":1.462},"max":{"1min":4.1989999999999998,"5min":5.0129999999999999,"15min":10.132},"last":2.403},{"interface":"front","average":{"1min":2.8839999999999999,"5min":3.012,"15min":3.0790000000000002},"min":{"1min":1.762,"5min":1.377,"15min":1.377},"max":{"1min":4.4740000000000002,"5min":4.609,"15min":6.4800000000000004},"last":2.7610000000000001}]},{"osd":5,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.1150000000000002,"5min":3.073,"15min":3.105},"min":{"1min":1.8400000000000001,"5min":1.8069999999999999,"15min":1.554},"max":{"1min":4.9969999999999999,"5min":4.9969999999999999,"15min":7.827},"last":2.9060000000000001},{"interface":"front","average":{"1min":3.2509999999999999,"5min":3.5150000000000001,"15min":3.5800000000000001},"min":{"1min":1.631,"5min":1.4379999999999999,"15min":1.4330000000000001},"max":{"1min":4.4279999999999999,"5min":5.2850000000000001,"15min":10.968999999999999},"last":3.4609999999999999}]},{"osd":6,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.4489999999999998,"5min":3.391,"15min":3.4689999999999999},"min":{"1min":1.5409999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"max":{"1min":4.5970000000000004,"5min":5.532,"15min":8.0549999999999997},"last":4.3849999999999998},{"interface":"front","average":{"1min":3.3359999999999999,"5min":3.4569999999999999,"15min":3.4780000000000002},"min":{"1min":2.3599999999999999,"5min":1.944,"15min":1.702},"max":{"1min":4.8799999999999999,"5min":4.8810000000000002,"15min":6.7709999999999999},"last":3.7029999999999998}]},{"osd":7,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":3.6509999999999998,"15min":3.7149999999999999},"min":{"1min":2.1160000000000001,"5min":1.7649999999999999,"15min":1.5860000000000001},"max":{"1min":4.7439999999999998,"5min":5.0350000000000001,"15min":10.839},"last":3.8980000000000001},{"interface":"front","average":{"1min":3.5699999999999998,"5min":3.6320000000000001,"15min":3.6920000000000002},"min":{"1min":2.0419999999999998,"5min":2.0219999999999998,"15min":1.7030000000000001},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":8.6609999999999996},"last":4.1619999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28312,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":26304,"kb_avail":94343528,"statfs":{"total":96636764160,"available":96607772672,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26935296},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.9550000000000001,"15min":1.966},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.9590000000000001,"5min":3.71,"15min":14.436999999999999},"last":2.0710000000000002},{"interface":"front","average":{"1min":1.802,"5min":2.0550000000000002,"15min":1.968},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.8109999999999999,"5min":3.4940000000000002,"15min":4.9870000000000001},"last":2.968}]},{"osd":1,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":2.1629999999999998,"15min":2.6949999999999998},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":3.524,"5min":4.1699999999999999,"15min":102.953},"last":2.8860000000000001},{"interface":"front","average":{"1min":2.2040000000000002,"5min":2.4180000000000001,"15min":2.4809999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.4849999999999999,"5min":4.2210000000000001,"15min":16.895},"last":3.6840000000000002}]},{"osd":2,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.4209999999999998,"5min":2.6309999999999998,"15min":2.7069999999999999},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":0.755},"max":{"1min":3.8359999999999999,"5min":3.96,"15min":6.2469999999999999},"last":4.0350000000000001},{"interface":"front","average":{"1min":2.8090000000000002,"5min":3.0209999999999999,"15min":3.0030000000000001},"min":{"1min":1.2649999999999999,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":4.1929999999999996,"5min":6.4850000000000003,"15min":14.715999999999999},"last":4.1059999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.899,"5min":2.9580000000000002,"15min":3.222},"min":{"1min":1.855,"5min":1.534,"15min":1.0960000000000001},"max":{"1min":4.4039999999999999,"5min":5.0199999999999996,"15min":70.090999999999994},"last":3.4060000000000001},{"interface":"front","average":{"1min":3.0089999999999999,"5min":3.1640000000000001,"15min":3.4460000000000002},"min":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.28},"max":{"1min":4.6459999999999999,"5min":5.5330000000000004,"15min":71.367999999999995},"last":3.2429999999999999}]},{"osd":5,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.335,"15min":3.399},"min":{"1min":1.8740000000000001,"5min":1.782,"15min":1.5589999999999999},"max":{"1min":4.9329999999999998,"5min":6.1470000000000002,"15min":10.621},"last":3.794},{"interface":"front","average":{"1min":3.2250000000000001,"5min":3.4710000000000001,"15min":3.5070000000000001},"min":{"1min":1.383,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":4.7880000000000003,"5min":5.9379999999999997,"15min":17.579999999999998},"last":4.5099999999999998}]},{"osd":6,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.2970000000000002,"5min":3.4609999999999999,"15min":3.6400000000000001},"min":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6140000000000001},"max":{"1min":4.984,"5min":4.984,"15min":59.569000000000003},"last":3.8879999999999999},{"interface":"front","average":{"1min":3.698,"5min":3.851,"15min":4.0860000000000003},"min":{"1min":2.4630000000000001,"5min":1.8879999999999999,"15min":1.79},"max":{"1min":5.2060000000000004,"5min":6.3890000000000002,"15min":60.023000000000003},"last":4.367}]},{"osd":7,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.7170000000000001,"15min":3.7429999999999999},"min":{"1min":1.6040000000000001,"5min":1.595,"15min":1.595},"max":{"1min":4.8620000000000001,"5min":6.8620000000000001,"15min":22.126000000000001},"last":4.718},{"interface":"front","average":{"1min":3.532,"5min":3.6150000000000002,"15min":3.5859999999999999},"min":{"1min":2.0939999999999999,"5min":1.9450000000000001,"15min":1.7270000000000001},"max":{"1min":5.2400000000000002,"5min":5.4489999999999998,"15min":8.2690000000000001},"last":4.2619999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29032,"kb_used_data":2016,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94342808,"statfs":{"total":96636764160,"available":96607035392,"internally_reserved":0,"allocated":2064384,"data_stored":959063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":2.109,"15min":2.3740000000000001},"min":{"1min":1.401,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.4809999999999999,"5min":3.6709999999999998,"15min":57.734999999999999},"last":1.6240000000000001},{"interface":"front","average":{"1min":2.2629999999999999,"5min":2.5939999999999999,"15min":2.4220000000000002},"min":{"1min":1.4370000000000001,"5min":1.3069999999999999,"15min":1.1539999999999999},"max":{"1min":3.2069999999999999,"5min":26.75,"15min":26.75},"last":2.7799999999999998}]},{"osd":1,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":2.496,"5min":2.7160000000000002,"15min":2.6669999999999998},"min":{"1min":1.5569999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":4.7439999999999998,"5min":18.234999999999999,"15min":18.234999999999999},"last":2.669},{"interface":"front","average":{"1min":2.9769999999999999,"5min":3.2530000000000001,"15min":3.0779999999999998},"min":{"1min":1.833,"5min":1.7210000000000001,"15min":1.397},"max":{"1min":3.9289999999999998,"5min":14.981,"15min":14.981},"last":2.5819999999999999}]},{"osd":2,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":2.9460000000000002,"5min":3.9239999999999999,"15min":3.3260000000000001},"min":{"1min":2.1259999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"max":{"1min":4.0380000000000003,"5min":63.558999999999997,"15min":63.558999999999997},"last":3.218},{"interface":"front","average":{"1min":3.1669999999999998,"5min":3.0110000000000001,"15min":2.9670000000000001},"min":{"1min":1.5209999999999999,"5min":1.1779999999999999,"15min":0.76000000000000001},"max":{"1min":4.5300000000000002,"5min":4.944,"15min":12.754},"last":2.9710000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":3.5310000000000001,"5min":3.794,"15min":3.5640000000000001},"min":{"1min":2.5449999999999999,"5min":2.0099999999999998,"15min":1.5609999999999999},"max":{"1min":4.5049999999999999,"5min":27.239999999999998,"15min":27.239999999999998},"last":3.8690000000000002},{"interface":"front","average":{"1min":3.7069999999999999,"5min":3.5680000000000001,"15min":3.5129999999999999},"min":{"1min":2.4460000000000002,"5min":1.6859999999999999,"15min":1.113},"max":{"1min":5.1520000000000001,"5min":5.1749999999999998,"15min":12.881},"last":3.1800000000000002}]},{"osd":5,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":3.4830000000000001,"5min":3.4550000000000001,"15min":3.4239999999999999},"min":{"1min":2.3809999999999998,"5min":1.6140000000000001,"15min":0.92200000000000004},"max":{"1min":4.8719999999999999,"5min":7.0590000000000002,"15min":7.0590000000000002},"last":2.169},{"interface":"front","average":{"1min":4.1440000000000001,"5min":4.3479999999999999,"15min":4.0940000000000003},"min":{"1min":3.3260000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"max":{"1min":5.4400000000000004,"5min":26.917000000000002,"15min":26.917000000000002},"last":3.4390000000000001}]},{"osd":6,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":4.3769999999999998,"5min":4.0789999999999997,"15min":4.0060000000000002},"min":{"1min":3.4239999999999999,"5min":1.288,"15min":1.109},"max":{"1min":5.4740000000000002,"5min":5.6189999999999998,"15min":12.606},"last":3.5920000000000001},{"interface":"front","average":{"1min":4.1989999999999998,"5min":3.956,"15min":3.9710000000000001},"min":{"1min":3.2839999999999998,"5min":1.4319999999999999,"15min":1.2050000000000001},"max":{"1min":4.9790000000000001,"5min":5.298,"15min":13.57},"last":3.0670000000000002}]},{"osd":7,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":4.7080000000000002,"5min":4.7850000000000001,"15min":4.524},"min":{"1min":3.7200000000000002,"5min":2.577,"15min":2.0379999999999998},"max":{"1min":5.8470000000000004,"5min":27.085000000000001,"15min":27.085000000000001},"last":4.2880000000000003},{"interface":"front","average":{"1min":4.8150000000000004,"5min":4.5380000000000003,"15min":4.5190000000000001},"min":{"1min":3.1080000000000001,"5min":1.421,"15min":1.323},"max":{"1min":7.5119999999999996,"5min":7.5119999999999996,"15min":8.798},"last":3.391}]}]},{"osd":5,"up_from":14,"seq":60129542359,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23188,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":21184,"kb_avail":94348652,"statfs":{"total":96636764160,"available":96613019648,"internally_reserved":0,"allocated":2043904,"data_stored":957701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21692416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":2.1659999999999999,"15min":2.9809999999999999},"min":{"1min":1.288,"5min":1.2589999999999999,"15min":1.123},"max":{"1min":3.633,"5min":3.863,"15min":194.91},"last":2.1619999999999999},{"interface":"front","average":{"1min":1.9490000000000001,"5min":2.1890000000000001,"15min":2.2200000000000002},"min":{"1min":1.4890000000000001,"5min":1.05,"15min":1.024},"max":{"1min":2.375,"5min":10.419,"15min":15.971},"last":2.2280000000000002}]},{"osd":1,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.363,"5min":2.3519999999999999,"15min":2.5089999999999999},"min":{"1min":1.617,"5min":1.306,"15min":1.23},"max":{"1min":3.9489999999999998,"5min":4.8319999999999999,"15min":8.3659999999999997},"last":2.048},{"interface":"front","average":{"1min":3.2269999999999999,"5min":3.1880000000000002,"15min":3.2789999999999999},"min":{"1min":2.0030000000000001,"5min":1.4019999999999999,"15min":1.4019999999999999},"max":{"1min":4.0099999999999998,"5min":4.2629999999999999,"15min":15.067},"last":3.52}]},{"osd":2,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.8090000000000002,"15min":3.1459999999999999},"min":{"1min":1.7629999999999999,"5min":1.5469999999999999,"15min":1.0469999999999999},"max":{"1min":4.056,"5min":4.5129999999999999,"15min":91.242999999999995},"last":2.919},{"interface":"front","average":{"1min":3.0880000000000001,"5min":3.149,"15min":3.254},"min":{"1min":2.1709999999999998,"5min":1.7090000000000001,"15min":1.478},"max":{"1min":4.1929999999999996,"5min":4.8550000000000004,"15min":12.802},"last":3.0760000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.1829999999999998,"5min":3.2330000000000001,"15min":3.1720000000000002},"min":{"1min":2.4009999999999998,"5min":1.5900000000000001,"15min":1.3069999999999999},"max":{"1min":3.9460000000000002,"5min":10.143000000000001,"15min":14.003},"last":3.2109999999999999},{"interface":"front","average":{"1min":3.6000000000000001,"5min":3.5649999999999999,"15min":3.5760000000000001},"min":{"1min":2.5430000000000001,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":5.194,"5min":5.194,"15min":7.5270000000000001},"last":3.3490000000000002}]},{"osd":4,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.9489999999999998,"5min":3.8290000000000002,"15min":3.8260000000000001},"min":{"1min":2.7730000000000001,"5min":1.827,"15min":1.526},"max":{"1min":4.9729999999999999,"5min":5.5350000000000001,"15min":9.3119999999999994},"last":4.2469999999999999},{"interface":"front","average":{"1min":3.8639999999999999,"5min":3.8879999999999999,"15min":3.9510000000000001},"min":{"1min":2.9159999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"max":{"1min":4.5990000000000002,"5min":5.2720000000000002,"15min":15.467000000000001},"last":4.0289999999999999}]},{"osd":6,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.887,"5min":3.7909999999999999,"15min":3.7280000000000002},"min":{"1min":2.5019999999999998,"5min":1.3560000000000001,"15min":1.341},"max":{"1min":4.835,"5min":10.332000000000001,"15min":17.254000000000001},"last":4.4900000000000002},{"interface":"front","average":{"1min":4.3289999999999997,"5min":4.226,"15min":4.3319999999999999},"min":{"1min":3.4860000000000002,"5min":1.607,"15min":1.607},"max":{"1min":5.3120000000000003,"5min":5.6520000000000001,"15min":15.326000000000001},"last":4.3780000000000001}]},{"osd":7,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":4.1559999999999997,"5min":4.1520000000000001,"15min":4.1379999999999999},"min":{"1min":3.2930000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"max":{"1min":4.6020000000000003,"5min":10.488,"15min":14.505000000000001},"last":4.7519999999999998},{"interface":"front","average":{"1min":4.6020000000000003,"5min":4.3339999999999996,"15min":4.3220000000000001},"min":{"1min":4.0279999999999996,"5min":2.4340000000000002,"15min":1.675},"max":{"1min":5.577,"5min":5.8579999999999997,"15min":12.441000000000001},"last":3.7890000000000001}]}]}]}} 2020-11-20T12:22:37.774 INFO:tasks.ceph:pgid 32.3 last_scrub_stamp 2020-11-20T12:21:22.063495+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=21, tm_sec=22, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=12, tm_min=22, tm_sec=8, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T12:22:37.775 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-20T12:22:54.846 INFO:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:22:54.855 INFO:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:22:57.777 INFO:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:22:58.031 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.027+0000 7f70f29be700 1 Processor -- start 2020-11-20T12:22:58.032 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.028+0000 7f70f29be700 1 -- start start 2020-11-20T12:22:58.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.029+0000 7f70f29be700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.029+0000 7f70f29be700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec14d110 con 0x7f70ec09f560 2020-11-20T12:22:58.033 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.029+0000 7f70f29be700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec14cbd0 con 0x7f70ec0bfbe0 2020-11-20T12:22:58.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.029+0000 7f70f29be700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec14ce70 con 0x7f70ec0bf800 2020-11-20T12:22:58.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.030+0000 7f70f11bb700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f70ec0bfbe0 0x7f70ec1480c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51928/0 (socket says 172.21.15.187:51928) 2020-11-20T12:22:58.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.030+0000 7f70f11bb700 1 -- 172.21.15.187:0/3172185069 learned_addr learned my addr 172.21.15.187:0/3172185069 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:58.034 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.030+0000 7f70f19bc700 1 --2- 172.21.15.187:0/3172185069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.030+0000 7f70f09ba700 1 -- 172.21.15.187:0/3172185069 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950009811 0 0) 0x7f70ec14cbd0 con 0x7f70ec0bfbe0 2020-11-20T12:22:58.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.030+0000 7f70f09ba700 1 -- 172.21.15.187:0/3172185069 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d4003240 con 0x7f70ec0bfbe0 2020-11-20T12:22:58.036 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.031+0000 7f70f09ba700 1 -- 172.21.15.187:0/3172185069 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1910880831 0 0) 0x7f70d4003240 con 0x7f70ec0bfbe0 2020-11-20T12:22:58.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.031+0000 7f70f19bc700 1 -- 172.21.15.187:0/3172185069 >> v1:172.21.15.187:6789/0 conn(0x7f70ec0bfbe0 legacy=0x7f70ec1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.031+0000 7f70f19bc700 1 -- 172.21.15.187:0/3172185069 >> v1:172.21.15.5:6789/0 conn(0x7f70ec09f560 legacy=0x7f70ec0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.037 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.031+0000 7f70f19bc700 1 -- 172.21.15.187:0/3172185069 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec0a4cc0 con 0x7f70ec0bf800 2020-11-20T12:22:58.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.031+0000 7f70f19bc700 1 --2- 172.21.15.187:0/3172185069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec14c2c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f70e8009c40 tx=0x7f70e80091d0).ready entity=mon.2 client_cookie=fba72e0e4ae88ff6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.032+0000 7f70f09ba700 1 -- 172.21.15.187:0/3172185069 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0014e0 con 0x7f70ec0bf800 2020-11-20T12:22:58.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.032+0000 7f70f09ba700 1 -- 172.21.15.187:0/3172185069 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e800b710 con 0x7f70ec0bf800 2020-11-20T12:22:58.038 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.032+0000 7f70f09ba700 1 -- 172.21.15.187:0/3172185069 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e800ba90 con 0x7f70ec0bf800 2020-11-20T12:22:58.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.032+0000 7f70f29be700 1 -- 172.21.15.187:0/3172185069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 msgr2=0x7f70ec14c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.032+0000 7f70f29be700 1 --2- 172.21.15.187:0/3172185069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec14c2c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f70e8009c40 tx=0x7f70e80091d0).stop 2020-11-20T12:22:58.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.032+0000 7f70f29be700 1 -- 172.21.15.187:0/3172185069 shutdown_connections 2020-11-20T12:22:58.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.032+0000 7f70f29be700 1 --2- 172.21.15.187:0/3172185069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec14c2c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.039 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.032+0000 7f70f29be700 1 -- 172.21.15.187:0/3172185069 >> 172.21.15.187:0/3172185069 conn(0x7f70ec068370 msgr2=0x7f70ec0bef30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:58.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.033+0000 7f70f29be700 1 -- 172.21.15.187:0/3172185069 shutdown_connections 2020-11-20T12:22:58.040 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.033+0000 7f70f29be700 1 -- 172.21.15.187:0/3172185069 wait complete. 2020-11-20T12:22:58.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.034+0000 7f70f29be700 1 Processor -- start 2020-11-20T12:22:58.042 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.034+0000 7f70f29be700 1 -- start start 2020-11-20T12:22:58.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.035+0000 7f70f29be700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec09f560 0x7f70ec0b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.035+0000 7f70f29be700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec0b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.035+0000 7f70f29be700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec146130 0x7f70ec0ba9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.043 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.035+0000 7f70f29be700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0b7190 con 0x7f70ec09f560 2020-11-20T12:22:58.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.035+0000 7f70f29be700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec146590 con 0x7f70ec146130 2020-11-20T12:22:58.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.035+0000 7f70f29be700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec143c90 con 0x7f70ec0bf800 2020-11-20T12:22:58.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.035+0000 7f70f21bd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec146130 0x7f70ec0ba9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f21bd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec146130 0x7f70ec0ba9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55328/0 (socket says 172.21.15.187:55328) 2020-11-20T12:22:58.044 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f21bd700 1 -- 172.21.15.187:0/170029461 learned_addr learned my addr 172.21.15.187:0/170029461 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:58.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f19bc700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec09f560 0x7f70ec0b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f11bb700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec0b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f21bd700 1 -- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 msgr2=0x7f70ec0b48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f21bd700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec0b48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.045 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f21bd700 1 -- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec09f560 msgr2=0x7f70ec0b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.046 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f21bd700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec09f560 0x7f70ec0b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f21bd700 1 -- 172.21.15.187:0/170029461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec143f30 con 0x7f70ec146130 2020-11-20T12:22:58.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f19bc700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec09f560 0x7f70ec0b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:22:58.047 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.036+0000 7f70f21bd700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec146130 0x7f70ec0ba9d0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7f70e400c810 tx=0x7f70e4007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.037+0000 7f70d9ffb700 1 -- 172.21.15.187:0/170029461 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e400a5d0 con 0x7f70ec146130 2020-11-20T12:22:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.037+0000 7f70d9ffb700 1 -- 172.21.15.187:0/170029461 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e400a730 con 0x7f70ec146130 2020-11-20T12:22:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.037+0000 7f70d9ffb700 1 -- 172.21.15.187:0/170029461 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e400aad0 con 0x7f70ec146130 2020-11-20T12:22:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.037+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec1441d0 con 0x7f70ec146130 2020-11-20T12:22:58.048 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.037+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec140e20 con 0x7f70ec146130 2020-11-20T12:22:58.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.038+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ec080050 con 0x7f70ec146130 2020-11-20T12:22:58.049 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.039+0000 7f70d9ffb700 1 -- 172.21.15.187:0/170029461 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f70e4019030 con 0x7f70ec146130 2020-11-20T12:22:58.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.039+0000 7f70d9ffb700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c4056d20 0x7f70c40591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.039+0000 7f70d9ffb700 1 -- 172.21.15.187:0/170029461 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f70e406f590 con 0x7f70ec146130 2020-11-20T12:22:58.050 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.040+0000 7f70f19bc700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c4056d20 0x7f70c40591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.040+0000 7f70f19bc700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c4056d20 0x7f70c40591e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f70e8009440 tx=0x7f70e80091d0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.051 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.044+0000 7f70d9ffb700 1 -- 172.21.15.187:0/170029461 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f70e4044960 con 0x7f70ec146130 2020-11-20T12:22:58.230 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.227+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 --> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70ec144360 con 0x7f70c4056d20 2020-11-20T12:22:58.235 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.231+0000 7f70d9ffb700 1 -- 172.21.15.187:0/170029461 <== mgr.4102 v2:172.21.15.187:6800/19649 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+105130 (secure 0 0 0) 0x7f70ec144360 con 0x7f70c4056d20 2020-11-20T12:22:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: 2020-11-20T12:22:58.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.234+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c4056d20 msgr2=0x7f70c40591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.240 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.234+0000 7f70f29be700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c4056d20 0x7f70c40591e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f70e8009440 tx=0x7f70e80091d0).stop 2020-11-20T12:22:58.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.234+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec146130 msgr2=0x7f70ec0ba9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.234+0000 7f70f29be700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec146130 0x7f70ec0ba9d0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7f70e400c810 tx=0x7f70e4007b90).stop 2020-11-20T12:22:58.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.235+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 shutdown_connections 2020-11-20T12:22:58.241 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.235+0000 7f70f29be700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f70c4056d20 0x7f70c40591e0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.235+0000 7f70f29be700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70ec09f560 0x7f70ec0b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.235+0000 7f70f29be700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70ec146130 0x7f70ec0ba9d0 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.235+0000 7f70f29be700 1 --2- 172.21.15.187:0/170029461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f70ec0bf800 0x7f70ec0b48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.242 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.235+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 >> 172.21.15.187:0/170029461 conn(0x7f70ec068370 msgr2=0x7f70ec0bb330 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:58.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.235+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 shutdown_connections 2020-11-20T12:22:58.243 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.235+0000 7f70f29be700 1 -- 172.21.15.187:0/170029461 wait complete. 2020-11-20T12:22:58.243 INFO:teuthology.orchestra.run.smithi187.stderr:dumped all 2020-11-20T12:22:58.263 INFO:teuthology.orchestra.run.smithi187.stdout:{"pg_ready":true,"pg_map":{"version":699,"stamp":"2020-11-20T12:22:57.861907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":62081,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":10,"num_read_kb":8194,"num_write":148,"num_write_kb":16514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14224,"num_omap_keys":15,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":214324,"kb_used_data":16180,"kb_used_omap":0,"kb_used_meta":198080,"kb_avail":754760396,"statfs":{"total":773094113280,"available":772874645504,"internally_reserved":0,"allocated":16568320,"data_stored":7789991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":202833920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004106"},"pg_stats":[{"pgid":"32.4","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:28.494810+0000","last_change":"2020-11-20T12:22:28.495011+0000","last_active":"2020-11-20T12:22:28.494810+0000","last_peered":"2020-11-20T12:22:28.494810+0000","last_clean":"2020-11-20T12:22:28.494810+0000","last_became_active":"2020-11-20T12:21:22.693729+0000","last_became_peered":"2020-11-20T12:21:22.693729+0000","last_unstale":"2020-11-20T12:22:28.494810+0000","last_undegraded":"2020-11-20T12:22:28.494810+0000","last_fullsized":"2020-11-20T12:22:28.494810+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:28.494800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:28.494800+0000","last_clean_scrub_stamp":"2020-11-20T12:22:28.494800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:35.747672+0000","last_change":"2020-11-20T12:22:35.747824+0000","last_active":"2020-11-20T12:22:35.747672+0000","last_peered":"2020-11-20T12:22:35.747672+0000","last_clean":"2020-11-20T12:22:35.747672+0000","last_became_active":"2020-11-20T12:21:22.685927+0000","last_became_peered":"2020-11-20T12:21:22.685927+0000","last_unstale":"2020-11-20T12:22:35.747672+0000","last_undegraded":"2020-11-20T12:22:35.747672+0000","last_fullsized":"2020-11-20T12:22:35.747672+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:35.747662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:35.747662+0000","last_clean_scrub_stamp":"2020-11-20T12:22:35.747662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2","version":"160'10","reported_seq":"33","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:27.499185+0000","last_change":"2020-11-20T12:22:27.499347+0000","last_active":"2020-11-20T12:22:27.499185+0000","last_peered":"2020-11-20T12:22:27.499185+0000","last_clean":"2020-11-20T12:22:27.499185+0000","last_became_active":"2020-11-20T12:21:22.693767+0000","last_became_peered":"2020-11-20T12:21:22.693767+0000","last_unstale":"2020-11-20T12:22:27.499185+0000","last_undegraded":"2020-11-20T12:22:27.499185+0000","last_fullsized":"2020-11-20T12:22:27.499185+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'10","last_scrub_stamp":"2020-11-20T12:22:27.499175+0000","last_deep_scrub":"160'10","last_deep_scrub_stamp":"2020-11-20T12:22:27.499175+0000","last_clean_scrub_stamp":"2020-11-20T12:22:27.499175+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":6,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:17.379608+0000","last_change":"2020-11-20T12:22:17.379764+0000","last_active":"2020-11-20T12:22:17.379608+0000","last_peered":"2020-11-20T12:22:17.379608+0000","last_clean":"2020-11-20T12:22:17.379608+0000","last_became_active":"2020-11-20T12:21:22.684395+0000","last_became_peered":"2020-11-20T12:21:22.684395+0000","last_unstale":"2020-11-20T12:22:17.379608+0000","last_undegraded":"2020-11-20T12:22:17.379608+0000","last_fullsized":"2020-11-20T12:22:17.379608+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:17.379599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:17.379599+0000","last_clean_scrub_stamp":"2020-11-20T12:22:17.379599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:26.469813+0000","last_change":"2020-11-20T12:22:26.470009+0000","last_active":"2020-11-20T12:22:26.469813+0000","last_peered":"2020-11-20T12:22:26.469813+0000","last_clean":"2020-11-20T12:22:26.469813+0000","last_became_active":"2020-11-20T12:21:22.692492+0000","last_became_peered":"2020-11-20T12:21:22.692492+0000","last_unstale":"2020-11-20T12:22:26.469813+0000","last_undegraded":"2020-11-20T12:22:26.469813+0000","last_fullsized":"2020-11-20T12:22:26.469813+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:26.469803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:26.469803+0000","last_clean_scrub_stamp":"2020-11-20T12:22:26.469803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"161'20","reported_seq":"35","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:31.744034+0000","last_change":"2020-11-20T12:22:31.744195+0000","last_active":"2020-11-20T12:22:31.744034+0000","last_peered":"2020-11-20T12:22:31.744034+0000","last_clean":"2020-11-20T12:22:31.744034+0000","last_became_active":"2020-11-20T12:21:22.108194+0000","last_became_peered":"2020-11-20T12:21:22.108194+0000","last_unstale":"2020-11-20T12:22:31.744034+0000","last_undegraded":"2020-11-20T12:22:31.744034+0000","last_fullsized":"2020-11-20T12:22:31.744034+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'20","last_scrub_stamp":"2020-11-20T12:22:31.744020+0000","last_deep_scrub":"161'20","last_deep_scrub_stamp":"2020-11-20T12:22:31.744020+0000","last_clean_scrub_stamp":"2020-11-20T12:22:31.744020+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":591,"num_omap_keys":1,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"161'13","reported_seq":"28","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:12.028222+0000","last_change":"2020-11-20T12:22:12.028348+0000","last_active":"2020-11-20T12:22:12.028222+0000","last_peered":"2020-11-20T12:22:12.028222+0000","last_clean":"2020-11-20T12:22:12.028222+0000","last_became_active":"2020-11-20T12:21:22.113651+0000","last_became_peered":"2020-11-20T12:21:22.113651+0000","last_unstale":"2020-11-20T12:22:12.028222+0000","last_undegraded":"2020-11-20T12:22:12.028222+0000","last_fullsized":"2020-11-20T12:22:12.028222+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'13","last_scrub_stamp":"2020-11-20T12:22:12.028215+0000","last_deep_scrub":"161'13","last_deep_scrub_stamp":"2020-11-20T12:22:12.028215+0000","last_clean_scrub_stamp":"2020-11-20T12:22:12.028215+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"161'14","reported_seq":"29","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:21.423283+0000","last_change":"2020-11-20T12:22:21.423449+0000","last_active":"2020-11-20T12:22:21.423283+0000","last_peered":"2020-11-20T12:22:21.423283+0000","last_clean":"2020-11-20T12:22:21.423283+0000","last_became_active":"2020-11-20T12:21:22.111838+0000","last_became_peered":"2020-11-20T12:21:22.111838+0000","last_unstale":"2020-11-20T12:22:21.423283+0000","last_undegraded":"2020-11-20T12:22:21.423283+0000","last_fullsized":"2020-11-20T12:22:21.423283+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'14","last_scrub_stamp":"2020-11-20T12:22:21.423270+0000","last_deep_scrub":"161'14","last_deep_scrub_stamp":"2020-11-20T12:22:21.423270+0000","last_clean_scrub_stamp":"2020-11-20T12:22:21.423270+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":184,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.4","version":"161'46","reported_seq":"61","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:16.428786+0000","last_change":"2020-11-20T12:22:16.428983+0000","last_active":"2020-11-20T12:22:16.428786+0000","last_peered":"2020-11-20T12:22:16.428786+0000","last_clean":"2020-11-20T12:22:16.428786+0000","last_became_active":"2020-11-20T12:21:22.107044+0000","last_became_peered":"2020-11-20T12:21:22.107044+0000","last_unstale":"2020-11-20T12:22:16.428786+0000","last_undegraded":"2020-11-20T12:22:16.428786+0000","last_fullsized":"2020-11-20T12:22:16.428786+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'46","last_scrub_stamp":"2020-11-20T12:22:16.428774+0000","last_deep_scrub":"161'46","last_deep_scrub_stamp":"2020-11-20T12:22:16.428774+0000","last_clean_scrub_stamp":"2020-11-20T12:22:16.428774+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61078,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"159'12","reported_seq":"27","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:18.385337+0000","last_change":"2020-11-20T12:22:18.385540+0000","last_active":"2020-11-20T12:22:18.385337+0000","last_peered":"2020-11-20T12:22:18.385337+0000","last_clean":"2020-11-20T12:22:18.385337+0000","last_became_active":"2020-11-20T12:21:22.105399+0000","last_became_peered":"2020-11-20T12:21:22.105399+0000","last_unstale":"2020-11-20T12:22:18.385337+0000","last_undegraded":"2020-11-20T12:22:18.385337+0000","last_fullsized":"2020-11-20T12:22:18.385337+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'12","last_scrub_stamp":"2020-11-20T12:22:18.385325+0000","last_deep_scrub":"159'12","last_deep_scrub_stamp":"2020-11-20T12:22:18.385325+0000","last_clean_scrub_stamp":"2020-11-20T12:22:18.385325+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4948,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"158'1","reported_seq":"16","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:14.715943+0000","last_change":"2020-11-20T12:22:14.716078+0000","last_active":"2020-11-20T12:22:14.715943+0000","last_peered":"2020-11-20T12:22:14.715943+0000","last_clean":"2020-11-20T12:22:14.715943+0000","last_became_active":"2020-11-20T12:21:22.098007+0000","last_became_peered":"2020-11-20T12:21:22.098007+0000","last_unstale":"2020-11-20T12:22:14.715943+0000","last_undegraded":"2020-11-20T12:22:14.715943+0000","last_fullsized":"2020-11-20T12:22:14.715943+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'1","last_scrub_stamp":"2020-11-20T12:22:14.715932+0000","last_deep_scrub":"158'1","last_deep_scrub_stamp":"2020-11-20T12:22:14.715932+0000","last_clean_scrub_stamp":"2020-11-20T12:22:14.715932+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"161'4","reported_seq":"21","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:26.725780+0000","last_change":"2020-11-20T12:22:26.725966+0000","last_active":"2020-11-20T12:22:26.725780+0000","last_peered":"2020-11-20T12:22:26.725780+0000","last_clean":"2020-11-20T12:22:26.725780+0000","last_became_active":"2020-11-20T12:21:22.102371+0000","last_became_peered":"2020-11-20T12:21:22.102371+0000","last_unstale":"2020-11-20T12:22:26.725780+0000","last_undegraded":"2020-11-20T12:22:26.725780+0000","last_fullsized":"2020-11-20T12:22:26.725780+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'4","last_scrub_stamp":"2020-11-20T12:22:26.725767+0000","last_deep_scrub":"161'4","last_deep_scrub_stamp":"2020-11-20T12:22:26.725767+0000","last_clean_scrub_stamp":"2020-11-20T12:22:26.725767+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":210,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"161'18","reported_seq":"33","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:30.806708+0000","last_change":"2020-11-20T12:22:30.806858+0000","last_active":"2020-11-20T12:22:30.806708+0000","last_peered":"2020-11-20T12:22:30.806708+0000","last_clean":"2020-11-20T12:22:30.806708+0000","last_became_active":"2020-11-20T12:21:22.093851+0000","last_became_peered":"2020-11-20T12:21:22.093851+0000","last_unstale":"2020-11-20T12:22:30.806708+0000","last_undegraded":"2020-11-20T12:22:30.806708+0000","last_fullsized":"2020-11-20T12:22:30.806708+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'18","last_scrub_stamp":"2020-11-20T12:22:30.806695+0000","last_deep_scrub":"161'18","last_deep_scrub_stamp":"2020-11-20T12:22:30.806695+0000","last_clean_scrub_stamp":"2020-11-20T12:22:30.806695+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":614,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"149","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:16.840688+0000","last_change":"2020-11-20T12:22:16.840862+0000","last_active":"2020-11-20T12:22:16.840688+0000","last_peered":"2020-11-20T12:22:16.840688+0000","last_clean":"2020-11-20T12:22:16.840688+0000","last_became_active":"2020-11-20T12:04:53.294169+0000","last_became_peered":"2020-11-20T12:04:53.294169+0000","last_unstale":"2020-11-20T12:22:16.840688+0000","last_undegraded":"2020-11-20T12:22:16.840688+0000","last_fullsized":"2020-11-20T12:22:16.840688+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:16.840678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:16.840678+0000","last_clean_scrub_stamp":"2020-11-20T12:22:16.840678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"160'8","reported_seq":"31","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:19.421879+0000","last_change":"2020-11-20T12:22:19.422107+0000","last_active":"2020-11-20T12:22:19.421879+0000","last_peered":"2020-11-20T12:22:19.421879+0000","last_clean":"2020-11-20T12:22:19.421879+0000","last_became_active":"2020-11-20T12:21:22.676940+0000","last_became_peered":"2020-11-20T12:21:22.676940+0000","last_unstale":"2020-11-20T12:22:19.421879+0000","last_undegraded":"2020-11-20T12:22:19.421879+0000","last_fullsized":"2020-11-20T12:22:19.421879+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'8","last_scrub_stamp":"2020-11-20T12:22:19.421858+0000","last_deep_scrub":"160'8","last_deep_scrub_stamp":"2020-11-20T12:22:19.421858+0000","last_clean_scrub_stamp":"2020-11-20T12:22:19.421858+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":4,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:20.737373+0000","last_change":"2020-11-20T12:22:20.737575+0000","last_active":"2020-11-20T12:22:20.737373+0000","last_peered":"2020-11-20T12:22:20.737373+0000","last_clean":"2020-11-20T12:22:20.737373+0000","last_became_active":"2020-11-20T12:04:53.293882+0000","last_became_peered":"2020-11-20T12:04:53.293882+0000","last_unstale":"2020-11-20T12:22:20.737373+0000","last_undegraded":"2020-11-20T12:22:20.737373+0000","last_fullsized":"2020-11-20T12:22:20.737373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:20.737363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:20.737363+0000","last_clean_scrub_stamp":"2020-11-20T12:22:20.737363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"148","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:22.743332+0000","last_change":"2020-11-20T12:22:22.743517+0000","last_active":"2020-11-20T12:22:22.743332+0000","last_peered":"2020-11-20T12:22:22.743332+0000","last_clean":"2020-11-20T12:22:22.743332+0000","last_became_active":"2020-11-20T12:04:53.297849+0000","last_became_peered":"2020-11-20T12:04:53.297849+0000","last_unstale":"2020-11-20T12:22:22.743332+0000","last_undegraded":"2020-11-20T12:22:22.743332+0000","last_fullsized":"2020-11-20T12:22:22.743332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:22.743322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:22.743322+0000","last_clean_scrub_stamp":"2020-11-20T12:22:22.743322+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":"23","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:21.425127+0000","last_change":"2020-11-20T12:22:21.425275+0000","last_active":"2020-11-20T12:22:21.425127+0000","last_peered":"2020-11-20T12:22:21.425127+0000","last_clean":"2020-11-20T12:22:21.425127+0000","last_became_active":"2020-11-20T12:21:22.675260+0000","last_became_peered":"2020-11-20T12:21:22.675260+0000","last_unstale":"2020-11-20T12:22:21.425127+0000","last_undegraded":"2020-11-20T12:22:21.425127+0000","last_fullsized":"2020-11-20T12:22:21.425127+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:21.425117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:21.425117+0000","last_clean_scrub_stamp":"2020-11-20T12:22:21.425117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:22.756356+0000","last_change":"2020-11-20T12:22:22.756572+0000","last_active":"2020-11-20T12:22:22.756356+0000","last_peered":"2020-11-20T12:22:22.756356+0000","last_clean":"2020-11-20T12:22:22.756356+0000","last_became_active":"2020-11-20T12:04:54.004416+0000","last_became_peered":"2020-11-20T12:04:54.004416+0000","last_unstale":"2020-11-20T12:22:22.756356+0000","last_undegraded":"2020-11-20T12:22:22.756356+0000","last_fullsized":"2020-11-20T12:22:22.756356+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:22.756347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:22.756347+0000","last_clean_scrub_stamp":"2020-11-20T12:22:22.756347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"161'4","reported_seq":"151","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:17.427670+0000","last_change":"2020-11-20T12:22:17.427902+0000","last_active":"2020-11-20T12:22:17.427670+0000","last_peered":"2020-11-20T12:22:17.427670+0000","last_clean":"2020-11-20T12:22:17.427670+0000","last_became_active":"2020-11-20T12:04:49.156248+0000","last_became_peered":"2020-11-20T12:04:49.156248+0000","last_unstale":"2020-11-20T12:22:17.427670+0000","last_undegraded":"2020-11-20T12:22:17.427670+0000","last_fullsized":"2020-11-20T12:22:17.427670+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'4","last_scrub_stamp":"2020-11-20T12:22:17.427649+0000","last_deep_scrub":"161'4","last_deep_scrub_stamp":"2020-11-20T12:22:17.427649+0000","last_clean_scrub_stamp":"2020-11-20T12:22:17.427649+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8071,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.7","version":"159'1","reported_seq":"24","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:19.297099+0000","last_change":"2020-11-20T12:22:19.297210+0000","last_active":"2020-11-20T12:22:19.297099+0000","last_peered":"2020-11-20T12:22:19.297099+0000","last_clean":"2020-11-20T12:22:19.297099+0000","last_became_active":"2020-11-20T12:21:22.697706+0000","last_became_peered":"2020-11-20T12:21:22.697706+0000","last_unstale":"2020-11-20T12:22:19.297099+0000","last_undegraded":"2020-11-20T12:22:19.297099+0000","last_fullsized":"2020-11-20T12:22:19.297099+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'1","last_scrub_stamp":"2020-11-20T12:22:19.297090+0000","last_deep_scrub":"159'1","last_deep_scrub_stamp":"2020-11-20T12:22:19.297090+0000","last_clean_scrub_stamp":"2020-11-20T12:22:19.297090+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:13.714096+0000","last_change":"2020-11-20T12:22:13.714198+0000","last_active":"2020-11-20T12:22:13.714096+0000","last_peered":"2020-11-20T12:22:13.714096+0000","last_clean":"2020-11-20T12:22:13.714096+0000","last_became_active":"2020-11-20T12:04:53.296376+0000","last_became_peered":"2020-11-20T12:04:53.296376+0000","last_unstale":"2020-11-20T12:22:13.714096+0000","last_undegraded":"2020-11-20T12:22:13.714096+0000","last_fullsized":"2020-11-20T12:22:13.714096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:13.714091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:13.714091+0000","last_clean_scrub_stamp":"2020-11-20T12:22:13.714091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:28.789256+0000","last_change":"2020-11-20T12:22:28.789413+0000","last_active":"2020-11-20T12:22:28.789256+0000","last_peered":"2020-11-20T12:22:28.789256+0000","last_clean":"2020-11-20T12:22:28.789256+0000","last_became_active":"2020-11-20T12:04:53.302386+0000","last_became_peered":"2020-11-20T12:04:53.302386+0000","last_unstale":"2020-11-20T12:22:28.789256+0000","last_undegraded":"2020-11-20T12:22:28.789256+0000","last_fullsized":"2020-11-20T12:22:28.789256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:28.789247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:28.789247+0000","last_clean_scrub_stamp":"2020-11-20T12:22:28.789247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"149","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:20.433611+0000","last_change":"2020-11-20T12:22:20.433794+0000","last_active":"2020-11-20T12:22:20.433611+0000","last_peered":"2020-11-20T12:22:20.433611+0000","last_clean":"2020-11-20T12:22:20.433611+0000","last_became_active":"2020-11-20T12:04:54.002137+0000","last_became_peered":"2020-11-20T12:04:54.002137+0000","last_unstale":"2020-11-20T12:22:20.433611+0000","last_undegraded":"2020-11-20T12:22:20.433611+0000","last_fullsized":"2020-11-20T12:22:20.433611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:20.433600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:20.433600+0000","last_clean_scrub_stamp":"2020-11-20T12:22:20.433600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"147","reported_epoch":"161","state":"active+clean","last_fresh":"2020-11-20T12:22:29.757483+0000","last_change":"2020-11-20T12:22:29.757654+0000","last_active":"2020-11-20T12:22:29.757483+0000","last_peered":"2020-11-20T12:22:29.757483+0000","last_clean":"2020-11-20T12:22:29.757483+0000","last_became_active":"2020-11-20T12:04:54.001419+0000","last_became_peered":"2020-11-20T12:04:54.001419+0000","last_unstale":"2020-11-20T12:22:29.757483+0000","last_undegraded":"2020-11-20T12:22:29.757483+0000","last_fullsized":"2020-11-20T12:22:29.757483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:22:29.757472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:22:29.757472+0000","last_clean_scrub_stamp":"2020-11-20T12:22:29.757472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":10,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":16,"acting":16,"num_store_stats":7},{"poolid":31,"num_pg":8,"stat_sum":{"num_bytes":62081,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":136,"num_write_kb":121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6153,"num_omap_keys":13,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":124162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8071,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542364,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20208,"kb_used_data":2088,"kb_used_omap":0,"kb_used_meta":18112,"kb_avail":94351632,"statfs":{"total":96636764160,"available":96616071168,"internally_reserved":0,"allocated":2138112,"data_stored":1020351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18546688},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":1.869,"5min":2.274,"15min":2.431},"min":{"1min":1.1319999999999999,"5min":0.63,"15min":0.63},"max":{"1min":3.0299999999999998,"5min":8.2889999999999997,"15min":37.622},"last":2.1669999999999998},{"interface":"front","average":{"1min":1.9590000000000001,"5min":2.2360000000000002,"15min":2.4159999999999999},"min":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"max":{"1min":3.0179999999999998,"5min":12.936999999999999,"15min":36.816000000000003},"last":1.6910000000000001}]},{"osd":1,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":2.6840000000000002,"5min":2.8700000000000001,"15min":3.097},"min":{"1min":1.4710000000000001,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":4.3739999999999997,"5min":11.837999999999999,"15min":42.296999999999997},"last":2.9020000000000001},{"interface":"front","average":{"1min":2.093,"5min":7.1820000000000004,"15min":7.0019999999999998},"min":{"1min":1.4179999999999999,"5min":1.0600000000000001,"15min":1.0600000000000001},"max":{"1min":3.7189999999999999,"5min":498.23500000000001,"15min":633.35299999999995},"last":1.8680000000000001}]},{"osd":2,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.9009999999999998,"15min":2.7999999999999998},"min":{"1min":1.8720000000000001,"5min":1.2090000000000001,"15min":1.073},"max":{"1min":3.8410000000000002,"5min":7.359,"15min":7.359},"last":2.105},{"interface":"front","average":{"1min":2.5659999999999998,"5min":2.7709999999999999,"15min":2.988},"min":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4259999999999999},"max":{"1min":3.6400000000000001,"5min":4.4009999999999998,"15min":36.509},"last":2.7210000000000001}]},{"osd":3,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.5899999999999999,"5min":3.4100000000000001,"15min":3.585},"min":{"1min":1.633,"5min":1.129,"15min":1.129},"max":{"1min":4.7649999999999997,"5min":8.3819999999999997,"15min":37.859000000000002},"last":3.3239999999999998},{"interface":"front","average":{"1min":3.02,"5min":3.1640000000000001,"15min":3.2799999999999998},"min":{"1min":2.0390000000000001,"5min":1.702,"15min":1.45},"max":{"1min":4.3369999999999997,"5min":5.0359999999999996,"15min":21.952999999999999},"last":2.4630000000000001}]},{"osd":4,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.2999999999999998,"5min":3.4500000000000002,"15min":3.6379999999999999},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":5.8499999999999996,"5min":13.117000000000001,"15min":36.979999999999997},"last":3.2599999999999998},{"interface":"front","average":{"1min":3.96,"5min":3.9700000000000002,"15min":4.1900000000000004},"min":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"max":{"1min":5.4390000000000001,"5min":9.1329999999999991,"15min":38.177},"last":3.7130000000000001}]},{"osd":5,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.589,"5min":3.633,"15min":3.6819999999999999},"min":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"max":{"1min":4.5049999999999999,"5min":5.2549999999999999,"15min":5.4480000000000004},"last":4.5919999999999996},{"interface":"front","average":{"1min":4.3689999999999998,"5min":4.2750000000000004,"15min":4.4550000000000001},"min":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"max":{"1min":5.1909999999999998,"5min":7.6890000000000001,"15min":37.438000000000002},"last":4.7050000000000001}]},{"osd":6,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.6680000000000001,"5min":3.8519999999999999,"15min":4.077},"min":{"1min":1.218,"5min":1.218,"15min":1.218},"max":{"1min":4.585,"5min":7.1710000000000003,"15min":36.674999999999997},"last":3.9590000000000001},{"interface":"front","average":{"1min":4.5819999999999999,"5min":4.5069999999999997,"15min":4.7450000000000001},"min":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"max":{"1min":5.9329999999999998,"5min":7.79,"15min":38.027000000000001},"last":4.141}]}]},{"osd":6,"up_from":14,"seq":60129542364,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30048,"kb_used_data":2072,"kb_used_omap":0,"kb_used_meta":27968,"kb_avail":94341792,"statfs":{"total":96636764160,"available":96605995008,"internally_reserved":0,"allocated":2121728,"data_stored":1019416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28639232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":2.2509999999999999,"5min":2.1829999999999998,"15min":2.1739999999999999},"min":{"1min":1.121,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":3.6869999999999998,"5min":5.0380000000000003,"15min":5.6600000000000001},"last":3.048},{"interface":"front","average":{"1min":2.3420000000000001,"5min":2.3900000000000001,"15min":2.4089999999999998},"min":{"1min":1.5109999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.8599999999999999,"5min":4.5179999999999998,"15min":7.1630000000000003},"last":3.3279999999999998}]},{"osd":1,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":2.6459999999999999,"15min":2.5750000000000002},"min":{"1min":1.627,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":4.0739999999999998,"5min":12.022,"15min":12.022},"last":1.982},{"interface":"front","average":{"1min":3.149,"5min":3.8599999999999999,"15min":3.8140000000000001},"min":{"1min":1.982,"5min":1.5629999999999999,"15min":1.252},"max":{"1min":3.891,"5min":74.932000000000002,"15min":140.53299999999999},"last":2.8919999999999999}]},{"osd":2,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":2.944,"5min":3.0489999999999999,"15min":3.0419999999999998},"min":{"1min":1.6299999999999999,"5min":1.3,"15min":1.3},"max":{"1min":4.2240000000000002,"5min":12.430999999999999,"15min":12.430999999999999},"last":2.379},{"interface":"front","average":{"1min":3.0299999999999998,"5min":3.0449999999999999,"15min":3.0459999999999998},"min":{"1min":1.5620000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"max":{"1min":4.5490000000000004,"5min":5.6600000000000001,"15min":7.5140000000000002},"last":3.9780000000000002}]},{"osd":3,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":3.4060000000000001,"15min":3.4569999999999999},"min":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6140000000000001},"max":{"1min":4.3579999999999997,"5min":5,"15min":8.2249999999999996},"last":4.6589999999999998},{"interface":"front","average":{"1min":3.6640000000000001,"5min":3.5979999999999999,"15min":3.597},"min":{"1min":1.8700000000000001,"5min":1.105,"15min":1.105},"max":{"1min":4.7530000000000001,"5min":13.329000000000001,"15min":13.329000000000001},"last":2.7949999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":3.7909999999999999,"5min":3.6549999999999998,"15min":3.6440000000000001},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":4.9409999999999998,"5min":11.247,"15min":11.247},"last":3.504},{"interface":"front","average":{"1min":4.1390000000000002,"5min":4.0069999999999997,"15min":4.016},"min":{"1min":1.335,"5min":1.335,"15min":1.335},"max":{"1min":5.1200000000000001,"5min":10.82,"15min":11.474},"last":3.819}]},{"osd":5,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":4.117,"5min":4.1200000000000001,"15min":4.1349999999999998},"min":{"1min":1.468,"5min":1.4319999999999999,"15min":1.4319999999999999},"max":{"1min":5.2080000000000002,"5min":11.568,"15min":14.435},"last":3.1909999999999998},{"interface":"front","average":{"1min":4.1399999999999997,"5min":4.1210000000000004,"15min":4.1159999999999997},"min":{"1min":1.278,"5min":1.278,"15min":1.278},"max":{"1min":5.2149999999999999,"5min":6.9089999999999998,"15min":11.827999999999999},"last":3.7400000000000002}]},{"osd":7,"last update":"Fri Nov 20 12:22:15 2020","interfaces":[{"interface":"back","average":{"1min":4.4550000000000001,"5min":4.3710000000000004,"15min":4.3869999999999996},"min":{"1min":1.732,"5min":1.732,"15min":1.732},"max":{"1min":5.5270000000000001,"5min":8.2379999999999995,"15min":11.917},"last":4.2270000000000003},{"interface":"front","average":{"1min":4.5979999999999999,"5min":4.5190000000000001,"15min":4.484},"min":{"1min":2.1850000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"max":{"1min":5.569,"5min":16.263000000000002,"15min":16.263000000000002},"last":3.4470000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542363,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":2008,"kb_used_omap":0,"kb_used_meta":30592,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":2056192,"data_stored":958548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31326208},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":2.194,"15min":5.0750000000000002},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":3.0830000000000002,"5min":32.231999999999999,"15min":868.745},"last":1.782},{"interface":"front","average":{"1min":1.667,"5min":1.9630000000000001,"15min":2.0710000000000002},"min":{"1min":1.0029999999999999,"5min":0.71799999999999997,"15min":0.70399999999999996},"max":{"1min":2.427,"5min":5.7370000000000001,"15min":10.042},"last":2.3450000000000002}]},{"osd":2,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":2.0739999999999998,"15min":2.645},"min":{"1min":0.97499999999999998,"5min":0.69199999999999995,"15min":0.68100000000000005},"max":{"1min":3.165,"5min":3.9689999999999999,"15min":142.95500000000001},"last":1.6839999999999999},{"interface":"front","average":{"1min":2.4950000000000001,"5min":3.0190000000000001,"15min":6.3310000000000004},"min":{"1min":1.044,"5min":1.04,"15min":0.95999999999999996},"max":{"1min":3.5299999999999998,"5min":32.755000000000003,"15min":868.82299999999998},"last":2.911}]},{"osd":3,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.984,"15min":5.8710000000000004},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.0409999999999999},"max":{"1min":4.1040000000000001,"5min":32.377000000000002,"15min":869.04300000000001},"last":3.4609999999999999},{"interface":"front","average":{"1min":2.4199999999999999,"5min":2.7599999999999998,"15min":2.8050000000000002},"min":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":0.94799999999999995},"max":{"1min":3.6880000000000002,"5min":5.593,"15min":11.404},"last":3.9359999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.681,"5min":2.8690000000000002,"15min":3.0009999999999999},"min":{"1min":1.4379999999999999,"5min":1.3580000000000001,"15min":0.96899999999999997},"max":{"1min":4.0609999999999999,"5min":4.5979999999999999,"15min":20.577999999999999},"last":2.8279999999999998},{"interface":"front","average":{"1min":3.1200000000000001,"5min":3.5099999999999998,"15min":6.4039999999999999},"min":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.246},"max":{"1min":4.4329999999999998,"5min":32.622,"15min":869.22299999999996},"last":2.468}]},{"osd":5,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.2970000000000002,"5min":3.4129999999999998,"15min":3.5289999999999999},"min":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.327},"max":{"1min":4.54,"5min":5.9400000000000004,"15min":13.064},"last":3.2959999999999998},{"interface":"front","average":{"1min":3.2130000000000001,"5min":3.2850000000000001,"15min":3.5329999999999999},"min":{"1min":1.387,"5min":1.387,"15min":1.1299999999999999},"max":{"1min":4.1879999999999997,"5min":5.2009999999999996,"15min":62.292000000000002},"last":3.214}]},{"osd":6,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.2789999999999999,"5min":3.8330000000000002,"15min":6.8040000000000003},"min":{"1min":1.587,"5min":1.587,"15min":1.5329999999999999},"max":{"1min":4.4909999999999997,"5min":33.268999999999998,"15min":869.15300000000002},"last":3.8490000000000002},{"interface":"front","average":{"1min":3.2789999999999999,"5min":3.448,"15min":3.681},"min":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"max":{"1min":4.3259999999999996,"5min":6.0759999999999996,"15min":17.042999999999999},"last":4.1820000000000004}]},{"osd":7,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.5640000000000001,"5min":3.7709999999999999,"15min":3.9620000000000002},"min":{"1min":1.798,"5min":1.798,"15min":1.7010000000000001},"max":{"1min":4.7290000000000001,"5min":6.2000000000000002,"15min":11.667},"last":4.4160000000000004},{"interface":"front","average":{"1min":3.5219999999999998,"5min":3.5950000000000002,"15min":3.7450000000000001},"min":{"1min":1.871,"5min":1.222,"15min":1.222},"max":{"1min":4.593,"5min":5.3029999999999999,"15min":20.710000000000001},"last":3.657}]}]},{"osd":0,"up_from":14,"seq":60129542364,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21784,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":19776,"kb_avail":94350056,"statfs":{"total":96636764160,"available":96614457344,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20250624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.0990000000000002},"min":{"1min":1.3049999999999999,"5min":0.93200000000000005,"15min":0.71299999999999997},"max":{"1min":3.1160000000000001,"5min":6.569,"15min":10.723000000000001},"last":2.0790000000000002},{"interface":"front","average":{"1min":2.4380000000000002,"5min":2.5750000000000002,"15min":2.552},"min":{"1min":1.6419999999999999,"5min":1.0589999999999999,"15min":1.05},"max":{"1min":3.1000000000000001,"5min":7.8929999999999998,"15min":7.8929999999999998},"last":2.7389999999999999}]},{"osd":2,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":2.073,"15min":2.149},"min":{"1min":1.165,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":3.1789999999999998,"5min":3.8929999999999998,"15min":42.066000000000003},"last":2.5920000000000001},{"interface":"front","average":{"1min":2.0779999999999998,"5min":2.1389999999999998,"15min":2.2330000000000001},"min":{"1min":1.1339999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.7990000000000004},"last":3.0259999999999998}]},{"osd":3,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":3.0529999999999999,"15min":2.8170000000000002},"min":{"1min":1.2989999999999999,"5min":1.095,"15min":0.753},"max":{"1min":4.3659999999999997,"5min":38.067,"15min":38.067},"last":2.661},{"interface":"front","average":{"1min":2.7250000000000001,"5min":3.3100000000000001,"15min":3.0710000000000002},"min":{"1min":1.4890000000000001,"5min":1.242,"15min":1.0149999999999999},"max":{"1min":3.8140000000000001,"5min":38.463000000000001,"15min":38.463000000000001},"last":3.6699999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":3,"15min":3.093},"min":{"1min":1.774,"5min":1.2390000000000001,"15min":1.05},"max":{"1min":4.2880000000000003,"5min":9.0749999999999993,"15min":14.467000000000001},"last":3.1869999999999998},{"interface":"front","average":{"1min":2.903,"5min":2.9700000000000002,"15min":3.0219999999999998},"min":{"1min":1.704,"5min":1.329,"15min":1.1579999999999999},"max":{"1min":3.871,"5min":5.0090000000000003,"15min":5.0679999999999996},"last":3.0950000000000002}]},{"osd":5,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.306,"15min":3.3940000000000001},"min":{"1min":1.996,"5min":1.5589999999999999,"15min":1.5589999999999999},"max":{"1min":4.2949999999999999,"5min":6.4539999999999997,"15min":7.8120000000000003},"last":2.379},{"interface":"front","average":{"1min":3.427,"5min":3.5659999999999998,"15min":3.6629999999999998},"min":{"1min":2.302,"5min":1.6579999999999999,"15min":1.6579999999999999},"max":{"1min":4.6159999999999997,"5min":6.1959999999999997,"15min":11.58},"last":3.601}]},{"osd":6,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.4089999999999998,"15min":3.4580000000000002},"min":{"1min":1.9199999999999999,"5min":1.5369999999999999,"15min":1.29},"max":{"1min":4.218,"5min":6.6230000000000002,"15min":6.6230000000000002},"last":4.1619999999999999},{"interface":"front","average":{"1min":3.452,"5min":3.4820000000000002,"15min":3.5430000000000001},"min":{"1min":2.1869999999999998,"5min":1.2090000000000001,"15min":1.2090000000000001},"max":{"1min":4.4950000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":3.2749999999999999}]},{"osd":7,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.7320000000000002,"5min":3.7450000000000001,"15min":3.867},"min":{"1min":2.1000000000000001,"5min":1.357,"15min":1.357},"max":{"1min":5.0220000000000002,"5min":7.4770000000000003,"15min":37.826999999999998},"last":3.4790000000000001},{"interface":"front","average":{"1min":3.5779999999999998,"5min":3.738,"15min":3.7389999999999999},"min":{"1min":2.214,"5min":1.7370000000000001,"15min":1.665},"max":{"1min":4.8529999999999998,"5min":6.7329999999999997,"15min":12.154},"last":3.944}]}]},{"osd":2,"up_from":14,"seq":60129542364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29144,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":27136,"kb_avail":94342696,"statfs":{"total":96636764160,"available":96606920704,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27787264},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":1.9650000000000001,"5min":1.978,"15min":2.0510000000000002},"min":{"1min":1.016,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":3.9950000000000001,"5min":3.9950000000000001,"15min":9.0239999999999991},"last":2.617},{"interface":"front","average":{"1min":2.4220000000000002,"5min":2.4119999999999999,"15min":2.431},"min":{"1min":0.873,"5min":0.873,"15min":0.871},"max":{"1min":3.8370000000000002,"5min":3.8370000000000002,"15min":6.0800000000000001},"last":2.859}]},{"osd":1,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":2.3620000000000001,"15min":2.452},"min":{"1min":1.2569999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":3.7869999999999999,"5min":4.782,"15min":24.391999999999999},"last":2.2309999999999999},{"interface":"front","average":{"1min":2.3130000000000002,"5min":3,"15min":5.4770000000000003},"min":{"1min":0.79900000000000004,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":3.8580000000000001,"5min":66.977999999999994,"15min":820.03499999999997},"last":2.3029999999999999}]},{"osd":3,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":2.0049999999999999,"5min":2.089,"15min":2.161},"min":{"1min":1.4350000000000001,"5min":0.97499999999999998,"15min":0.748},"max":{"1min":3.1629999999999998,"5min":3.2229999999999999,"15min":6.1429999999999998},"last":3.0230000000000001},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.859,"15min":2.9249999999999998},"min":{"1min":1.514,"5min":1.0569999999999999,"15min":0.84599999999999997},"max":{"1min":3.79,"5min":5.601,"15min":10.574999999999999},"last":3.5609999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.1819999999999999,"15min":3.2290000000000001},"min":{"1min":1.782,"5min":1.462,"15min":1.462},"max":{"1min":4.1989999999999998,"5min":5.0129999999999999,"15min":10.132},"last":3.1509999999999998},{"interface":"front","average":{"1min":2.8839999999999999,"5min":3.012,"15min":3.0790000000000002},"min":{"1min":1.762,"5min":1.377,"15min":1.377},"max":{"1min":4.4740000000000002,"5min":4.609,"15min":6.4800000000000004},"last":3.0920000000000001}]},{"osd":5,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.1150000000000002,"5min":3.073,"15min":3.105},"min":{"1min":1.8400000000000001,"5min":1.8069999999999999,"15min":1.554},"max":{"1min":4.9969999999999999,"5min":4.9969999999999999,"15min":7.827},"last":3.4300000000000002},{"interface":"front","average":{"1min":3.2509999999999999,"5min":3.5150000000000001,"15min":3.5800000000000001},"min":{"1min":1.631,"5min":1.4379999999999999,"15min":1.4330000000000001},"max":{"1min":4.4279999999999999,"5min":5.2850000000000001,"15min":10.968999999999999},"last":4.0119999999999996}]},{"osd":6,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.4489999999999998,"5min":3.391,"15min":3.4689999999999999},"min":{"1min":1.5409999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"max":{"1min":4.5970000000000004,"5min":5.532,"15min":8.0549999999999997},"last":4.0750000000000002},{"interface":"front","average":{"1min":3.3359999999999999,"5min":3.4569999999999999,"15min":3.4780000000000002},"min":{"1min":2.3599999999999999,"5min":1.944,"15min":1.702},"max":{"1min":4.8799999999999999,"5min":4.8810000000000002,"15min":6.7709999999999999},"last":3.71}]},{"osd":7,"last update":"Fri Nov 20 12:22:23 2020","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":3.6509999999999998,"15min":3.7149999999999999},"min":{"1min":2.1160000000000001,"5min":1.7649999999999999,"15min":1.5860000000000001},"max":{"1min":4.7439999999999998,"5min":5.0350000000000001,"15min":10.839},"last":3.7909999999999999},{"interface":"front","average":{"1min":3.5699999999999998,"5min":3.6320000000000001,"15min":3.6920000000000002},"min":{"1min":2.0419999999999998,"5min":2.0219999999999998,"15min":1.7030000000000001},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":8.6609999999999996},"last":3.9620000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542364,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28312,"kb_used_data":2000,"kb_used_omap":0,"kb_used_meta":26304,"kb_avail":94343528,"statfs":{"total":96636764160,"available":96607772672,"internally_reserved":0,"allocated":2048000,"data_stored":958304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26935296},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.9550000000000001,"15min":1.966},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.9590000000000001,"5min":3.71,"15min":14.436999999999999},"last":2.3860000000000001},{"interface":"front","average":{"1min":1.802,"5min":2.0550000000000002,"15min":1.968},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.8109999999999999,"5min":3.4940000000000002,"15min":4.9870000000000001},"last":2.4550000000000001}]},{"osd":1,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":2.1629999999999998,"15min":2.6949999999999998},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":3.524,"5min":4.1699999999999999,"15min":102.953},"last":3.3330000000000002},{"interface":"front","average":{"1min":2.2040000000000002,"5min":2.4180000000000001,"15min":2.4809999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.4849999999999999,"5min":4.2210000000000001,"15min":16.895},"last":2.9700000000000002}]},{"osd":2,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.4209999999999998,"5min":2.6309999999999998,"15min":2.7069999999999999},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":0.755},"max":{"1min":3.8359999999999999,"5min":3.96,"15min":6.2469999999999999},"last":4.3810000000000002},{"interface":"front","average":{"1min":2.8090000000000002,"5min":3.0209999999999999,"15min":3.0030000000000001},"min":{"1min":1.2649999999999999,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":4.1929999999999996,"5min":6.4850000000000003,"15min":14.715999999999999},"last":3.476}]},{"osd":4,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":2.899,"5min":2.9580000000000002,"15min":3.222},"min":{"1min":1.855,"5min":1.534,"15min":1.0960000000000001},"max":{"1min":4.4039999999999999,"5min":5.0199999999999996,"15min":70.090999999999994},"last":4.0190000000000001},{"interface":"front","average":{"1min":3.0089999999999999,"5min":3.1640000000000001,"15min":3.4460000000000002},"min":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.28},"max":{"1min":4.6459999999999999,"5min":5.5330000000000004,"15min":71.367999999999995},"last":2.5630000000000002}]},{"osd":5,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.1909999999999998,"5min":3.335,"15min":3.399},"min":{"1min":1.8740000000000001,"5min":1.782,"15min":1.5589999999999999},"max":{"1min":4.9329999999999998,"5min":6.1470000000000002,"15min":10.621},"last":3.8450000000000002},{"interface":"front","average":{"1min":3.2250000000000001,"5min":3.4710000000000001,"15min":3.5070000000000001},"min":{"1min":1.383,"5min":1.2709999999999999,"15min":1.2709999999999999},"max":{"1min":4.7880000000000003,"5min":5.9379999999999997,"15min":17.579999999999998},"last":3.286}]},{"osd":6,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.2970000000000002,"5min":3.4609999999999999,"15min":3.6400000000000001},"min":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6140000000000001},"max":{"1min":4.984,"5min":4.984,"15min":59.569000000000003},"last":4.899},{"interface":"front","average":{"1min":3.698,"5min":3.851,"15min":4.0860000000000003},"min":{"1min":2.4630000000000001,"5min":1.8879999999999999,"15min":1.79},"max":{"1min":5.2060000000000004,"5min":6.3890000000000002,"15min":60.023000000000003},"last":4.7839999999999998}]},{"osd":7,"last update":"Fri Nov 20 12:22:21 2020","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.7170000000000001,"15min":3.7429999999999999},"min":{"1min":1.6040000000000001,"5min":1.595,"15min":1.595},"max":{"1min":4.8620000000000001,"5min":6.8620000000000001,"15min":22.126000000000001},"last":3.673},{"interface":"front","average":{"1min":3.532,"5min":3.6150000000000002,"15min":3.5859999999999999},"min":{"1min":2.0939999999999999,"5min":1.9450000000000001,"15min":1.7270000000000001},"max":{"1min":5.2400000000000002,"5min":5.4489999999999998,"15min":8.2690000000000001},"last":4.2880000000000003}]}]},{"osd":4,"up_from":14,"seq":60129542364,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29032,"kb_used_data":2016,"kb_used_omap":0,"kb_used_meta":27008,"kb_avail":94342808,"statfs":{"total":96636764160,"available":96607035392,"internally_reserved":0,"allocated":2064384,"data_stored":959063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27656192},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":2.109,"15min":2.3740000000000001},"min":{"1min":1.401,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.4809999999999999,"5min":3.6709999999999998,"15min":57.734999999999999},"last":2.1320000000000001},{"interface":"front","average":{"1min":2.2629999999999999,"5min":2.5939999999999999,"15min":2.4220000000000002},"min":{"1min":1.4370000000000001,"5min":1.3069999999999999,"15min":1.1539999999999999},"max":{"1min":3.2069999999999999,"5min":26.75,"15min":26.75},"last":2.0310000000000001}]},{"osd":1,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":2.496,"5min":2.7160000000000002,"15min":2.6669999999999998},"min":{"1min":1.5569999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":4.7439999999999998,"5min":18.234999999999999,"15min":18.234999999999999},"last":2.6320000000000001},{"interface":"front","average":{"1min":2.9769999999999999,"5min":3.2530000000000001,"15min":3.0779999999999998},"min":{"1min":1.833,"5min":1.7210000000000001,"15min":1.397},"max":{"1min":3.9289999999999998,"5min":14.981,"15min":14.981},"last":2.9329999999999998}]},{"osd":2,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":2.9460000000000002,"5min":3.9239999999999999,"15min":3.3260000000000001},"min":{"1min":2.1259999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"max":{"1min":4.0380000000000003,"5min":63.558999999999997,"15min":63.558999999999997},"last":3.3530000000000002},{"interface":"front","average":{"1min":3.1669999999999998,"5min":3.0110000000000001,"15min":2.9670000000000001},"min":{"1min":1.5209999999999999,"5min":1.1779999999999999,"15min":0.76000000000000001},"max":{"1min":4.5300000000000002,"5min":4.944,"15min":12.754},"last":3.0249999999999999}]},{"osd":3,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":3.5310000000000001,"5min":3.794,"15min":3.5640000000000001},"min":{"1min":2.5449999999999999,"5min":2.0099999999999998,"15min":1.5609999999999999},"max":{"1min":4.5049999999999999,"5min":27.239999999999998,"15min":27.239999999999998},"last":3.855},{"interface":"front","average":{"1min":3.7069999999999999,"5min":3.5680000000000001,"15min":3.5129999999999999},"min":{"1min":2.4460000000000002,"5min":1.6859999999999999,"15min":1.113},"max":{"1min":5.1520000000000001,"5min":5.1749999999999998,"15min":12.881},"last":3.5710000000000002}]},{"osd":5,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":3.4830000000000001,"5min":3.4550000000000001,"15min":3.4239999999999999},"min":{"1min":2.3809999999999998,"5min":1.6140000000000001,"15min":0.92200000000000004},"max":{"1min":4.8719999999999999,"5min":7.0590000000000002,"15min":7.0590000000000002},"last":3.4820000000000002},{"interface":"front","average":{"1min":4.1440000000000001,"5min":4.3479999999999999,"15min":4.0940000000000003},"min":{"1min":3.3260000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"max":{"1min":5.4400000000000004,"5min":26.917000000000002,"15min":26.917000000000002},"last":4.3300000000000001}]},{"osd":6,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":4.3769999999999998,"5min":4.0789999999999997,"15min":4.0060000000000002},"min":{"1min":3.4239999999999999,"5min":1.288,"15min":1.109},"max":{"1min":5.4740000000000002,"5min":5.6189999999999998,"15min":12.606},"last":4.0019999999999998},{"interface":"front","average":{"1min":4.1989999999999998,"5min":3.956,"15min":3.9710000000000001},"min":{"1min":3.2839999999999998,"5min":1.4319999999999999,"15min":1.2050000000000001},"max":{"1min":4.9790000000000001,"5min":5.298,"15min":13.57},"last":4.2690000000000001}]},{"osd":7,"last update":"Fri Nov 20 12:22:18 2020","interfaces":[{"interface":"back","average":{"1min":4.7080000000000002,"5min":4.7850000000000001,"15min":4.524},"min":{"1min":3.7200000000000002,"5min":2.577,"15min":2.0379999999999998},"max":{"1min":5.8470000000000004,"5min":27.085000000000001,"15min":27.085000000000001},"last":4.6360000000000001},{"interface":"front","average":{"1min":4.8150000000000004,"5min":4.5380000000000003,"15min":4.5190000000000001},"min":{"1min":3.1080000000000001,"5min":1.421,"15min":1.323},"max":{"1min":7.5119999999999996,"5min":7.5119999999999996,"15min":8.798},"last":4.3979999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542364,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23188,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":21184,"kb_avail":94348652,"statfs":{"total":96636764160,"available":96613019648,"internally_reserved":0,"allocated":2043904,"data_stored":957701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21692416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":2.1659999999999999,"15min":2.9809999999999999},"min":{"1min":1.288,"5min":1.2589999999999999,"15min":1.123},"max":{"1min":3.633,"5min":3.863,"15min":194.91},"last":1.869},{"interface":"front","average":{"1min":1.9490000000000001,"5min":2.1890000000000001,"15min":2.2200000000000002},"min":{"1min":1.4890000000000001,"5min":1.05,"15min":1.024},"max":{"1min":2.375,"5min":10.419,"15min":15.971},"last":1.9790000000000001}]},{"osd":1,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.363,"5min":2.3519999999999999,"15min":2.5089999999999999},"min":{"1min":1.617,"5min":1.306,"15min":1.23},"max":{"1min":3.9489999999999998,"5min":4.8319999999999999,"15min":8.3659999999999997},"last":2.0840000000000001},{"interface":"front","average":{"1min":3.2269999999999999,"5min":3.1880000000000002,"15min":3.2789999999999999},"min":{"1min":2.0030000000000001,"5min":1.4019999999999999,"15min":1.4019999999999999},"max":{"1min":4.0099999999999998,"5min":4.2629999999999999,"15min":15.067},"last":3.298}]},{"osd":2,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.8090000000000002,"15min":3.1459999999999999},"min":{"1min":1.7629999999999999,"5min":1.5469999999999999,"15min":1.0469999999999999},"max":{"1min":4.056,"5min":4.5129999999999999,"15min":91.242999999999995},"last":2.734},{"interface":"front","average":{"1min":3.0880000000000001,"5min":3.149,"15min":3.254},"min":{"1min":2.1709999999999998,"5min":1.7090000000000001,"15min":1.478},"max":{"1min":4.1929999999999996,"5min":4.8550000000000004,"15min":12.802},"last":2.8839999999999999}]},{"osd":3,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.1829999999999998,"5min":3.2330000000000001,"15min":3.1720000000000002},"min":{"1min":2.4009999999999998,"5min":1.5900000000000001,"15min":1.3069999999999999},"max":{"1min":3.9460000000000002,"5min":10.143000000000001,"15min":14.003},"last":3.0289999999999999},{"interface":"front","average":{"1min":3.6000000000000001,"5min":3.5649999999999999,"15min":3.5760000000000001},"min":{"1min":2.5430000000000001,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":5.194,"5min":5.194,"15min":7.5270000000000001},"last":3.6099999999999999}]},{"osd":4,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.9489999999999998,"5min":3.8290000000000002,"15min":3.8260000000000001},"min":{"1min":2.7730000000000001,"5min":1.827,"15min":1.526},"max":{"1min":4.9729999999999999,"5min":5.5350000000000001,"15min":9.3119999999999994},"last":3.1699999999999999},{"interface":"front","average":{"1min":3.8639999999999999,"5min":3.8879999999999999,"15min":3.9510000000000001},"min":{"1min":2.9159999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"max":{"1min":4.5990000000000002,"5min":5.2720000000000002,"15min":15.467000000000001},"last":3.7679999999999998}]},{"osd":6,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":3.887,"5min":3.7909999999999999,"15min":3.7280000000000002},"min":{"1min":2.5019999999999998,"5min":1.3560000000000001,"15min":1.341},"max":{"1min":4.835,"5min":10.332000000000001,"15min":17.254000000000001},"last":3.4569999999999999},{"interface":"front","average":{"1min":4.3289999999999997,"5min":4.226,"15min":4.3319999999999999},"min":{"1min":3.4860000000000002,"5min":1.607,"15min":1.607},"max":{"1min":5.3120000000000003,"5min":5.6520000000000001,"15min":15.326000000000001},"last":4.2300000000000004}]},{"osd":7,"last update":"Fri Nov 20 12:22:20 2020","interfaces":[{"interface":"back","average":{"1min":4.1559999999999997,"5min":4.1520000000000001,"15min":4.1379999999999999},"min":{"1min":3.2930000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"max":{"1min":4.6020000000000003,"5min":10.488,"15min":14.505000000000001},"last":4.1310000000000002},{"interface":"front","average":{"1min":4.6020000000000003,"5min":4.3339999999999996,"15min":4.3220000000000001},"min":{"1min":4.0279999999999996,"5min":2.4340000000000002,"15min":1.675},"max":{"1min":5.577,"5min":5.8579999999999997,"15min":12.441000000000001},"last":4.0380000000000003}]}]}]}} 2020-11-20T12:22:58.267 INFO:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2020-11-20T12:22:58.502 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.498+0000 7f8ca3fff700 1 Processor -- start 2020-11-20T12:22:58.503 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.498+0000 7f8ca3fff700 1 -- start start 2020-11-20T12:22:58.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.500+0000 7f8ca3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 0x7f8ca4148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.515 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.500+0000 7f8ca3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca414c170 con 0x7f8ca4145cc0 2020-11-20T12:22:58.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.500+0000 7f8ca3fff700 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca414c410 con 0x7f8ca40683f0 2020-11-20T12:22:58.516 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.500+0000 7f8ca3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca414c6b0 con 0x7f8ca4066040 2020-11-20T12:22:58.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.500+0000 7f8c9a7fc700 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f8ca40683f0 0x7f8ca41487e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:51938/0 (socket says 172.21.15.187:51938) 2020-11-20T12:22:58.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.500+0000 7f8c9a7fc700 1 -- 172.21.15.187:0/963155157 learned_addr learned my addr 172.21.15.187:0/963155157 (peer_addr_for_me v1:172.21.15.187:0/0) 2020-11-20T12:22:58.517 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca37fe700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 0x7f8ca4148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397947731 0 0) 0x7f8ca414c410 con 0x7f8ca40683f0 2020-11-20T12:22:58.518 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c84003240 con 0x7f8ca40683f0 2020-11-20T12:22:58.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2538879987 0 0) 0x7f8c84003240 con 0x7f8ca40683f0 2020-11-20T12:22:58.519 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 msgr2=0x7f8ca4148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:58.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca27fc700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 0x7f8ca4148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.520 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 >> v1:172.21.15.5:6789/0 conn(0x7f8ca4145cc0 legacy=0x7f8ca4065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40658a0 con 0x7f8ca40683f0 2020-11-20T12:22:58.521 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.501+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1488313692 0 0) 0x7f8c8c0014e0 con 0x7f8ca40683f0 2020-11-20T12:22:58.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 >> v1:172.21.15.187:6789/0 conn(0x7f8ca40683f0 legacy=0x7f8ca41487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.522 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca27fc700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c84003730 0x7f8c84003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca27fc700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c840044e0 0x7f8c84004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca27fc700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 0x7f8c84004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.523 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca414c6b0 con 0x7f8c840044e0 2020-11-20T12:22:58.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84005390 con 0x7f8c84003730 2020-11-20T12:22:58.524 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c840042f0 con 0x7f8ca4066040 2020-11-20T12:22:58.525 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca37fe700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 0x7f8c84004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8ca2ffd700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c84003730 0x7f8c84003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.502+0000 7f8c9a7fc700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c840044e0 0x7f8c84004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.503+0000 7f8ca37fe700 1 -- 172.21.15.187:0/963155157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c84003730 msgr2=0x7f8c84003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:58.527 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.503+0000 7f8ca37fe700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c84003730 0x7f8c84003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.503+0000 7f8ca37fe700 1 -- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c840044e0 msgr2=0x7f8c84004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:58.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.503+0000 7f8ca37fe700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8c840044e0 0x7f8c84004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.503+0000 7f8ca37fe700 1 -- 172.21.15.187:0/963155157 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c84003240 con 0x7f8ca4066040 2020-11-20T12:22:58.528 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.503+0000 7f8ca2ffd700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c84003730 0x7f8c84003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:22:58.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.504+0000 7f8ca37fe700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 0x7f8c84004e50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f8c9c009ea0 tx=0x7f8c9c0091d0).ready entity=mon.2 client_cookie=62fe31c5b0f42160 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.504+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c0095c0 con 0x7f8ca4066040 2020-11-20T12:22:58.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.504+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c00ec40 con 0x7f8ca4066040 2020-11-20T12:22:58.529 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.504+0000 7f8ca27fc700 1 -- 172.21.15.187:0/963155157 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c0123f0 con 0x7f8ca4066040 2020-11-20T12:22:58.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.505+0000 7f8ca3fff700 1 -- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 msgr2=0x7f8c84004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.530 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.505+0000 7f8ca3fff700 1 --2- 172.21.15.187:0/963155157 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca4066040 0x7f8c84004e50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f8c9c009ea0 tx=0x7f8c9c0091d0).stop 2020-11-20T12:22:58.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.505+0000 7f8ca37fe700 1 -- 172.21.15.187:0/963155157 reap_dead start 2020-11-20T12:22:58.531 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.505+0000 7f8ca3fff700 1 -- 172.21.15.187:0/963155157 shutdown_connections 2020-11-20T12:22:58.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.505+0000 7f8ca3fff700 1 -- 172.21.15.187:0/963155157 >> 172.21.15.187:0/963155157 conn(0x7f8ca40bab30 msgr2=0x7f8ca40baf40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:58.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.505+0000 7f8ca3fff700 1 -- 172.21.15.187:0/963155157 shutdown_connections 2020-11-20T12:22:58.532 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.505+0000 7f8ca3fff700 1 -- 172.21.15.187:0/963155157 wait complete. 2020-11-20T12:22:58.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.507+0000 7f8ca3fff700 1 Processor -- start 2020-11-20T12:22:58.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.507+0000 7f8ca3fff700 1 -- start start 2020-11-20T12:22:58.533 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca3fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ca4066040 0x7f8ca415bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca40b3f40 0x7f8ca409de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ca409e320 0x7f8ca40b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.534 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40a06c0 con 0x7f8ca409e320 2020-11-20T12:22:58.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca3fff700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40b2b00 con 0x7f8ca4066040 2020-11-20T12:22:58.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40b2da0 con 0x7f8ca40b3f40 2020-11-20T12:22:58.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ca4066040 0x7f8ca415bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ca4066040 0x7f8ca415bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:55344/0 (socket says 172.21.15.187:55344) 2020-11-20T12:22:58.535 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.508+0000 7f8ca2ffd700 1 -- 172.21.15.187:0/123358990 learned_addr learned my addr 172.21.15.187:0/123358990 (peer_addr_for_me v2:172.21.15.187:0/0) 2020-11-20T12:22:58.536 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8ca2ffd700 1 -- 172.21.15.187:0/123358990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca40b3f40 msgr2=0x7f8ca409de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8ca2ffd700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca40b3f40 0x7f8ca409de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8ca2ffd700 1 -- 172.21.15.187:0/123358990 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ca409e320 msgr2=0x7f8ca40b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8ca2ffd700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ca409e320 0x7f8ca40b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8ca2ffd700 1 -- 172.21.15.187:0/123358990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40b48c0 con 0x7f8ca4066040 2020-11-20T12:22:58.537 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8ca2ffd700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ca4066040 0x7f8ca415bdb0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7f8c94007b60 tx=0x7f8c94009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8c9b7fe700 1 -- 172.21.15.187:0/123358990 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9400d9b0 con 0x7f8ca4066040 2020-11-20T12:22:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8c9b7fe700 1 -- 172.21.15.187:0/123358990 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9400db10 con 0x7f8ca4066040 2020-11-20T12:22:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.509+0000 7f8c9b7fe700 1 -- 172.21.15.187:0/123358990 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9400de90 con 0x7f8ca4066040 2020-11-20T12:22:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.510+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca40b43c0 con 0x7f8ca4066040 2020-11-20T12:22:58.538 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.510+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca40b3730 con 0x7f8ca4066040 2020-11-20T12:22:58.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.511+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca40a04d0 con 0x7f8ca4066040 2020-11-20T12:22:58.539 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.511+0000 7f8c9b7fe700 1 -- 172.21.15.187:0/123358990 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8c9401a030 con 0x7f8ca4066040 2020-11-20T12:22:58.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.512+0000 7f8c9b7fe700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c80056d50 0x7f8c80059210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.512+0000 7f8c9b7fe700 1 -- 172.21.15.187:0/123358990 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8c9406fa20 con 0x7f8ca4066040 2020-11-20T12:22:58.540 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.512+0000 7f8c9a7fc700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c80056d50 0x7f8c80059210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.541 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.513+0000 7f8c9a7fc700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c80056d50 0x7f8c80059210 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f8c8c004560 tx=0x7f8c8c0077d0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.542 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.516+0000 7f8c9b7fe700 1 -- 172.21.15.187:0/123358990 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8c94044f00 con 0x7f8ca4066040 2020-11-20T12:22:58.708 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.705+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"} v 0) v1 -- 0x7f8ca4061a20 con 0x7f8ca4066040 2020-11-20T12:22:58.718 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.714+0000 7f8c9b7fe700 1 -- 172.21.15.187:0/123358990 <== mon.1 v2:172.21.15.187:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8c94071e90 con 0x7f8ca4066040 2020-11-20T12:22:58.721 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.717+0000 7f8c9b7fe700 1 -- 172.21.15.187:0/123358990 <== mon.1 v2:172.21.15.187:3300/0 8 ==== mon_command_ack([{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}]=0 v30) v1 ==== 123+0+0 (secure 0 0 0) 0x7f8c94018030 con 0x7f8ca4066040 2020-11-20T12:22:58.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c80056d50 msgr2=0x7f8c80059210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.723 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c80056d50 0x7f8c80059210 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f8c8c004560 tx=0x7f8c8c0077d0).stop 2020-11-20T12:22:58.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ca4066040 msgr2=0x7f8ca415bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.724 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ca4066040 0x7f8ca415bdb0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7f8c94007b60 tx=0x7f8c94009040).stop 2020-11-20T12:22:58.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 shutdown_connections 2020-11-20T12:22:58.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:6800/19649,v1:172.21.15.187:6801/19649] conn(0x7f8c80056d50 0x7f8c80059210 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.725 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ca409e320 0x7f8ca40b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:58.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ca4066040 0x7f8ca415bdb0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.726 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 --2- 172.21.15.187:0/123358990 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8ca40b3f40 0x7f8ca409de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:22:58.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 >> 172.21.15.187:0/123358990 conn(0x7f8ca40bab30 msgr2=0x7f8ca4147920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:58.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 shutdown_connections 2020-11-20T12:22:58.727 INFO:teuthology.orchestra.run.smithi187.stderr:2020-11-20T12:22:58.719+0000 7f8ca3fff700 1 -- 172.21.15.187:0/123358990 wait complete. 2020-11-20T12:22:58.746 INFO:teuthology.misc:Shutting down mds daemons... 2020-11-20T12:22:58.747 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:22:58.747 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:22:58.774 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:22:58.775 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:22:58.775 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:23:04.801 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:23:04.801 INFO:teuthology.misc:Shutting down osd daemons... 2020-11-20T12:23:04.802 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2020-11-20T12:23:04.802 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:23:04.885 INFO:tasks.ceph.osd.0:Stopped 2020-11-20T12:23:04.886 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2020-11-20T12:23:04.886 INFO:teuthology.orchestra.run:waiting for 300